2023-11-28T20:14:45.252 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5 2023-11-28T20:14:45.252 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637 --name vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --owner scheduled_vshankar@teuthology -- /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/orig.config.yaml 2023-11-28T20:14:45.375 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T20:14:45.464 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637 branch: wip-vshankar-testing-20231127.102914 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: vshankar@redhat.com first_in_suite: false job_id: '7469637' kernel: client: branch: testing flavor: default kdb: true sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20231127.102914 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 43dd94c190073cf16cff1e711dd7f5a7b1cbc883 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 43dd94c190073cf16cff1e711dd7f5a7b1cbc883 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20231127.102914 sha1: 43dd94c190073cf16cff1e711dd7f5a7b1cbc883 owner: scheduled_vshankar@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8377 sha1: 43dd94c190073cf16cff1e711dd7f5a7b1cbc883 sleep_before_teardown: 0 subset: 462/1024 suite: fs suite_branch: wip-vshankar-testing-20231127.102914 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 43dd94c190073cf16cff1e711dd7f5a7b1cbc883 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhPbRY6n/z7r1pF/uD1tRgmCEGcsmUX170GbKygNpTf9+0WWSUXUT7sNh8sneEl4yMj4WqUfMlgxQbC129zAy0= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/yvpBetb1Oc/9+gPKWVCU/1lcSer5fSZJTIBiZbE7JIHVrtJbCVMGxerg9tkEPspZTIz+fk48ivQh6MPM1vyc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 5ae09a5cc7064ca2bf994df19f3006b79794789a timestamp: 2023-11-28_06:29:14 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-28T20:14:45.464 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa; will attempt to use it 2023-11-28T20:14:45.465 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks 2023-11-28T20:14:45.465 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-28T20:14:45.467 INFO:teuthology.task.internal:Checking packages... 2023-11-28T20:14:45.490 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '43dd94c190073cf16cff1e711dd7f5a7b1cbc883' 2023-11-28T20:14:45.490 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-28T20:14:45.491 INFO:teuthology.packaging:ref: None 2023-11-28T20:14:45.491 INFO:teuthology.packaging:tag: None 2023-11-28T20:14:45.491 INFO:teuthology.packaging:branch: wip-vshankar-testing-20231127.102914 2023-11-28T20:14:45.491 INFO:teuthology.packaging:sha1: 43dd94c190073cf16cff1e711dd7f5a7b1cbc883 2023-11-28T20:14:45.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20231127.102914 2023-11-28T20:14:45.708 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7520.g43dd94c1 2023-11-28T20:14:45.708 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-28T20:14:45.714 INFO:teuthology.task.internal:no buildpackages task found 2023-11-28T20:14:45.714 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-28T20:14:45.768 INFO:teuthology.task.internal:Saving configuration 2023-11-28T20:14:45.784 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-28T20:14:45.798 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T20:14:45.823 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 20:09:45.812081', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhPbRY6n/z7r1pF/uD1tRgmCEGcsmUX170GbKygNpTf9+0WWSUXUT7sNh8sneEl4yMj4WqUfMlgxQbC129zAy0='} 2023-11-28T20:14:45.846 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 20:09:45.813540', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/yvpBetb1Oc/9+gPKWVCU/1lcSer5fSZJTIBiZbE7JIHVrtJbCVMGxerg9tkEPspZTIz+fk48ivQh6MPM1vyc='} 2023-11-28T20:14:45.847 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-28T20:14:45.857 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-11-28T20:14:45.857 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-11-28T20:14:45.857 INFO:teuthology.run_tasks:Running task console_log... 2023-11-28T20:14:45.947 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb275ff0a60>, signals=[15]) 2023-11-28T20:14:45.948 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-28T20:14:45.956 INFO:teuthology.task.internal:Opening connections... 2023-11-28T20:14:45.956 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:14:45.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:14:46.061 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T20:14:46.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:14:46.146 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-28T20:14:46.157 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2023-11-28T20:14:46.175 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2023-11-28T20:14:46.175 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2023-11-28T20:14:46.233 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2023-11-28T20:14:46.233 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2023-11-28T20:14:46.233 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2023-11-28T20:14:46.233 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2023-11-28T20:14:46.234 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2023-11-28T20:14:46.234 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2023-11-28T20:14:46.234 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-28T20:14:46.234 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2023-11-28T20:14:46.234 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2023-11-28T20:14:46.234 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-28T20:14:46.235 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2023-11-28T20:14:46.235 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T20:14:46.235 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-28T20:14:46.235 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-28T20:14:46.235 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2023-11-28T20:14:46.260 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2023-11-28T20:14:46.277 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2023-11-28T20:14:46.277 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2023-11-28T20:14:46.333 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2023-11-28T20:14:46.333 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T20:14:46.334 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-28T20:14:46.335 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-28T20:14:46.335 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2023-11-28T20:14:46.360 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-28T20:14:46.369 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-28T20:14:46.386 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-28T20:14:46.387 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2023-11-28T20:14:46.389 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2023-11-28T20:14:46.405 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-28T20:14:46.412 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-28T20:14:46.413 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2023-11-28T20:14:46.447 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2023-11-28T20:14:46.486 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-28T20:14:46.542 INFO:teuthology.run_tasks:Running task kernel... 2023-11-28T20:14:46.557 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'} 2023-11-28T20:14:46.557 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2023-11-28T20:14:46.557 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2023-11-28T20:14:46.557 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T20:14:46.558 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T20:14:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T20:14:46.573 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-11-28T20:14:46.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T20:14:46.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-11-28T20:14:46.728 DEBUG:teuthology.task.kernel:sha1 for client.1 is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T20:14:46.728 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-28T20:14:46.729 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2023-11-28T20:14:46.746 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-361.el9.x86_64 2023-11-28T20:14:46.746 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T20:14:46.746 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-28T20:14:46.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum upgrade -y linux-firmware 2023-11-28T20:14:46.810 DEBUG:teuthology.task.kernel:sha1 for client.0 is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T20:14:46.811 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-28T20:14:46.811 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2023-11-28T20:14:46.829 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-361.el9.x86_64 2023-11-28T20:14:46.829 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T20:14:46.830 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-28T20:14:46.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum upgrade -y linux-firmware 2023-11-28T20:14:49.124 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2023-11-28T20:14:49.281 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 8.0 MB/s | 6.4 MB 00:00 2023-11-28T20:14:50.549 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2023-11-28T20:14:50.898 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2023-11-28T20:15:02.723 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 683 kB/s | 7.9 MB 00:11 2023-11-28T20:15:03.396 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 666 kB/s | 7.9 MB 00:12 2023-11-28T20:15:10.273 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 18 MB 00:06 2023-11-28T20:15:16.648 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 18 MB 00:12 2023-11-28T20:15:25.972 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 486 kB/s | 5.8 MB 00:12 2023-11-28T20:15:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 15 kB 00:00 2023-11-28T20:15:29.277 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2023-11-28T20:15:32.031 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 503 kB/s | 5.8 MB 00:11 2023-11-28T20:15:33.618 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2023-11-28T20:15:33.960 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2023-11-28T20:15:36.328 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-28T20:15:36.336 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-28T20:15:36.336 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout: linux-firmware noarch 20231030-141.el9 CentOS-BaseOS 328 M 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 1 Package 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 328 M 2023-11-28T20:15:36.337 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-28T20:15:38.064 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 20 MB 00:04 2023-11-28T20:15:42.978 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2023-11-28T20:15:44.786 INFO:teuthology.orchestra.run.smithi190.stdout:linux-firmware-20231030-141.el9.noarch.rpm 39 MB/s | 328 MB 00:08 2023-11-28T20:15:44.787 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-28T20:15:44.787 INFO:teuthology.orchestra.run.smithi190.stdout:Total 39 MB/s | 328 MB 00:08 2023-11-28T20:15:44.788 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-28T20:15:44.878 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-28T20:15:44.878 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-28T20:15:45.472 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-11-28T20:15:45.482 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-11-28T20:15:45.482 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-11-28T20:15:45.482 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-11-28T20:15:45.483 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2023-11-28T20:15:45.483 INFO:teuthology.orchestra.run.smithi045.stdout: linux-firmware noarch 20231030-141.el9 CentOS-BaseOS 328 M 2023-11-28T20:15:45.483 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:15:45.483 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-11-28T20:15:45.483 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-11-28T20:15:45.483 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 1 Package 2023-11-28T20:15:45.483 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:15:45.483 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 328 M 2023-11-28T20:15:45.484 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-11-28T20:15:47.014 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-28T20:15:47.014 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-28T20:15:52.638 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-28T20:15:53.120 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T20:15:53.207 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T20:15:53.978 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T20:15:53.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T20:15:54.355 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T20:15:54.355 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:15:54.355 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2023-11-28T20:15:54.356 INFO:teuthology.orchestra.run.smithi190.stdout: linux-firmware-20231030-141.el9.noarch 2023-11-28T20:15:54.356 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:15:54.356 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-28T20:15:54.625 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on client.1... 2023-11-28T20:15:54.626 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T20:15:54.770 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ 2023-11-28T20:15:54.770 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ --input-file=- 2023-11-28T20:15:54.773 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi190.front.sepia.ceph.com to complete... 2023-11-28T20:15:55.669 INFO:teuthology.orchestra.run.smithi045.stdout:linux-firmware-20231030-141.el9.noarch.rpm 32 MB/s | 328 MB 00:10 2023-11-28T20:15:55.671 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-11-28T20:15:55.671 INFO:teuthology.orchestra.run.smithi045.stdout:Total 32 MB/s | 328 MB 00:10 2023-11-28T20:15:55.671 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-11-28T20:15:55.772 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-11-28T20:15:55.772 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-11-28T20:15:58.006 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-11-28T20:15:58.006 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-11-28T20:16:04.068 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-11-28T20:16:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T20:16:04.505 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T20:16:05.305 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T20:16:05.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T20:16:05.762 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T20:16:05.763 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:16:05.763 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2023-11-28T20:16:05.763 INFO:teuthology.orchestra.run.smithi045.stdout: linux-firmware-20231030-141.el9.noarch 2023-11-28T20:16:05.763 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:16:05.763 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-11-28T20:16:06.251 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on client.0... 2023-11-28T20:16:06.252 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T20:16:06.393 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ 2023-11-28T20:16:06.393 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ --input-file=- 2023-11-28T20:16:06.395 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi045.front.sepia.ceph.com to complete... 2023-11-28T20:16:08.674 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28 20:16:08 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm [320493731/320493731] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-28T20:16:08.674 INFO:teuthology.orchestra.run.smithi190.stderr:FINISHED --2023-11-28 20:16:08-- 2023-11-28T20:16:08.674 INFO:teuthology.orchestra.run.smithi190.stderr:Total wall clock time: 14s 2023-11-28T20:16:08.675 INFO:teuthology.orchestra.run.smithi190.stderr:Downloaded: 1 files, 306M in 14s (22.5 MB/s) 2023-11-28T20:16:08.676 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on client.1... 2023-11-28T20:16:08.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-28T20:16:10.522 INFO:teuthology.orchestra.run.smithi190.stdout:Verifying... ######################################## 2023-11-28T20:16:10.581 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing... ######################################## 2023-11-28T20:16:10.661 INFO:teuthology.orchestra.run.smithi190.stdout:Updating / installing... 2023-11-28T20:16:17.752 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-6.6.0_g8247bbb5b470-1 ######################################## 2023-11-28T20:16:32.568 INFO:teuthology.orchestra.run.smithi190.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-28T20:16:34.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28 20:16:34 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm [320493731/320493731] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-28T20:16:34.822 INFO:teuthology.orchestra.run.smithi045.stderr:FINISHED --2023-11-28 20:16:34-- 2023-11-28T20:16:34.822 INFO:teuthology.orchestra.run.smithi045.stderr:Total wall clock time: 28s 2023-11-28T20:16:34.822 INFO:teuthology.orchestra.run.smithi045.stderr:Downloaded: 1 files, 306M in 28s (11.0 MB/s) 2023-11-28T20:16:34.822 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on client.0... 2023-11-28T20:16:34.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-28T20:16:36.756 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying... ######################################## 2023-11-28T20:16:36.812 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing... ######################################## 2023-11-28T20:16:36.895 INFO:teuthology.orchestra.run.smithi045.stdout:Updating / installing... 2023-11-28T20:16:44.417 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-6.6.0_g8247bbb5b470-1 ######################################## 2023-11-28T20:16:53.745 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-28T20:16:53.746 DEBUG:teuthology.orchestra.run.smithi190:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-28T20:16:53.810 INFO:teuthology.orchestra.run.smithi190.stdout:/boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T20:16:53.810 INFO:teuthology.orchestra.run.smithi190.stdout:/boot/config-6.6.0-g8247bbb5b470 2023-11-28T20:16:53.810 INFO:teuthology.orchestra.run.smithi190.stdout:/boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T20:16:53.811 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470 2023-11-28T20:16:53.811 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel 2023-11-28T20:16:53.811 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch 2023-11-28T20:16:53.811 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86 2023-11-28T20:16:53.811 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto 2023-11-28T20:16:53.811 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-28T20:16:53.811 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-28T20:16:53.811 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-28T20:16:53.812 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-28T20:16:53.812 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-28T20:16:53.812 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-28T20:16:53.812 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-28T20:16:53.812 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-28T20:16:53.812 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-28T20:16:53.812 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-28T20:16:53.813 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-28T20:16:53.813 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-28T20:16:53.813 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-28T20:16:53.813 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-28T20:16:53.813 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-28T20:16:53.813 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-28T20:16:53.813 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-28T20:16:53.813 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-28T20:16:53.814 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-28T20:16:53.814 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-28T20:16:53.814 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel 2023-11-28T20:16:53.814 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu 2023-11-28T20:16:53.814 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce 2023-11-28T20:16:53.814 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-28T20:16:53.814 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm 2023-11-28T20:16:53.815 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-28T20:16:53.815 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-28T20:16:53.815 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-28T20:16:53.815 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto 2023-11-28T20:16:53.815 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-28T20:16:53.815 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-28T20:16:53.815 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-28T20:16:53.815 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx 2023-11-28T20:16:53.815 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-28T20:16:53.816 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-28T20:16:53.816 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-28T20:16:53.816 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-28T20:16:53.816 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-28T20:16:53.816 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-28T20:16:53.816 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-28T20:16:53.816 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-28T20:16:53.817 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-28T20:16:53.817 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-28T20:16:53.817 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-28T20:16:53.817 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-28T20:16:53.818 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-28T20:16:53.818 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-28T20:16:53.818 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-28T20:16:53.818 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-28T20:16:53.818 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-28T20:16:53.818 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-28T20:16:53.818 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-28T20:16:53.819 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-28T20:16:53.820 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/custom_method.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-28T20:16:53.821 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-28T20:16:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-28T20:16:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-28T20:16:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata 2023-11-28T20:16:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-28T20:16:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-28T20:16:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ata_generic.ko 2023-11-28T20:16:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-28T20:16:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-28T20:16:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-28T20:16:53.823 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-28T20:16:53.824 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sis.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-28T20:16:53.825 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-28T20:16:53.826 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-28T20:16:53.826 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-28T20:16:53.826 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-28T20:16:53.826 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-28T20:16:53.826 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-28T20:16:53.826 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-28T20:16:53.827 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/brd.ko 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/loop.ko 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-28T20:16:53.828 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/virtio_blk.ko 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkfront.ko 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-28T20:16:53.829 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-28T20:16:53.830 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-28T20:16:53.831 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-28T20:16:53.831 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-28T20:16:53.831 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave 2023-11-28T20:16:53.831 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-28T20:16:53.831 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-28T20:16:53.831 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-28T20:16:53.831 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm 2023-11-28T20:16:53.832 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm.ko 2023-11-28T20:16:53.832 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-28T20:16:53.832 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-28T20:16:53.832 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-28T20:16:53.832 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-28T20:16:53.832 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-28T20:16:53.833 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/uv_mmtimer.ko 2023-11-28T20:16:53.833 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/virtio_console.ko 2023-11-28T20:16:53.833 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq 2023-11-28T20:16:53.833 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-28T20:16:53.833 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-28T20:16:53.833 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-28T20:16:53.833 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-28T20:16:53.834 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-28T20:16:53.834 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-28T20:16:53.834 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto 2023-11-28T20:16:53.834 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp 2023-11-28T20:16:53.834 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-28T20:16:53.835 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-28T20:16:53.835 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel 2023-11-28T20:16:53.835 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat 2023-11-28T20:16:53.835 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common 2023-11-28T20:16:53.835 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-28T20:16:53.835 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-28T20:16:53.836 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-28T20:16:53.836 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-28T20:16:53.836 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-28T20:16:53.836 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio 2023-11-28T20:16:53.836 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-28T20:16:53.836 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca 2023-11-28T20:16:53.837 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-28T20:16:53.837 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq 2023-11-28T20:16:53.837 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-28T20:16:53.837 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma 2023-11-28T20:16:53.837 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw 2023-11-28T20:16:53.837 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-28T20:16:53.837 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat 2023-11-28T20:16:53.837 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-28T20:16:53.838 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac 2023-11-28T20:16:53.838 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-28T20:16:53.838 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-28T20:16:53.838 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-28T20:16:53.838 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-28T20:16:53.838 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-28T20:16:53.839 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-28T20:16:53.839 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-28T20:16:53.839 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-28T20:16:53.839 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-28T20:16:53.839 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-28T20:16:53.840 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-28T20:16:53.840 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-28T20:16:53.840 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-28T20:16:53.840 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware 2023-11-28T20:16:53.840 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/edd.ko 2023-11-28T20:16:53.840 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-28T20:16:53.841 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-28T20:16:53.842 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-28T20:16:53.843 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-28T20:16:53.843 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-28T20:16:53.843 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-28T20:16:53.843 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-28T20:16:53.843 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-28T20:16:53.844 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-28T20:16:53.845 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-28T20:16:53.846 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-28T20:16:53.847 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon 2023-11-28T20:16:53.848 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-28T20:16:53.848 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-28T20:16:53.848 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-28T20:16:53.848 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-28T20:16:53.848 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-28T20:16:53.848 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-28T20:16:53.848 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-28T20:16:53.849 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-28T20:16:53.849 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-28T20:16:53.849 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-28T20:16:53.849 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-28T20:16:53.849 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-28T20:16:53.849 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-28T20:16:53.849 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-28T20:16:53.850 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-28T20:16:53.850 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-28T20:16:53.850 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-28T20:16:53.850 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-28T20:16:53.850 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-28T20:16:53.851 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-28T20:16:53.851 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-28T20:16:53.851 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-28T20:16:53.851 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-28T20:16:53.851 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-28T20:16:53.851 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-28T20:16:53.851 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-28T20:16:53.852 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-28T20:16:53.852 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-28T20:16:53.852 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-28T20:16:53.852 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-28T20:16:53.852 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-28T20:16:53.852 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-28T20:16:53.852 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-28T20:16:53.852 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-28T20:16:53.853 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-28T20:16:53.853 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-28T20:16:53.853 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-28T20:16:53.853 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-28T20:16:53.853 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-28T20:16:53.853 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-28T20:16:53.853 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-28T20:16:53.854 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-28T20:16:53.855 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-28T20:16:53.856 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-28T20:16:53.857 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-28T20:16:53.858 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-28T20:16:53.859 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-28T20:16:53.859 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-28T20:16:53.859 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-28T20:16:53.859 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-28T20:16:53.859 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-28T20:16:53.859 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-28T20:16:53.860 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-28T20:16:53.860 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-28T20:16:53.860 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-28T20:16:53.860 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-28T20:16:53.860 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-28T20:16:53.860 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-28T20:16:53.861 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-28T20:16:53.861 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-28T20:16:53.861 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-28T20:16:53.861 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-28T20:16:53.861 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-28T20:16:53.861 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-28T20:16:53.862 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-28T20:16:53.862 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-28T20:16:53.862 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-28T20:16:53.862 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-28T20:16:53.862 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-28T20:16:53.862 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-28T20:16:53.862 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-28T20:16:53.862 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-28T20:16:53.862 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-28T20:16:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-28T20:16:53.864 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-28T20:16:53.864 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-28T20:16:53.864 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-28T20:16:53.864 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-28T20:16:53.864 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-28T20:16:53.864 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-28T20:16:53.864 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-28T20:16:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-dev.ko 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core 2023-11-28T20:16:53.866 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-28T20:16:53.867 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp 2023-11-28T20:16:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-28T20:16:53.869 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt 2023-11-28T20:16:53.869 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-28T20:16:53.869 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input 2023-11-28T20:16:53.869 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport 2023-11-28T20:16:53.869 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-28T20:16:53.869 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-28T20:16:53.869 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-28T20:16:53.869 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-28T20:16:53.869 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4 2023-11-28T20:16:53.870 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-28T20:16:53.871 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio 2023-11-28T20:16:53.871 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-28T20:16:53.871 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-28T20:16:53.871 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-28T20:16:53.871 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-28T20:16:53.871 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-28T20:16:53.871 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu 2023-11-28T20:16:53.871 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd 2023-11-28T20:16:53.871 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-28T20:16:53.872 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-28T20:16:53.873 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-28T20:16:53.873 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md 2023-11-28T20:16:53.873 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache 2023-11-28T20:16:53.873 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-28T20:16:53.873 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-28T20:16:53.873 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-28T20:16:53.873 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-28T20:16:53.874 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-28T20:16:53.874 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-28T20:16:53.874 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-28T20:16:53.874 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-28T20:16:53.874 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-28T20:16:53.875 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-28T20:16:53.875 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-28T20:16:53.875 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-28T20:16:53.875 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-28T20:16:53.875 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-28T20:16:53.875 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion 2023-11-28T20:16:53.876 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mfd-core.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-28T20:16:53.877 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-28T20:16:53.878 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d 2023-11-28T20:16:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-28T20:16:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-28T20:16:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei 2023-11-28T20:16:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-28T20:16:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-28T20:16:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-28T20:16:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru 2023-11-28T20:16:53.881 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-28T20:16:53.881 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp 2023-11-28T20:16:53.881 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-28T20:16:53.881 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-28T20:16:53.881 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-28T20:16:53.881 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-28T20:16:53.881 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-28T20:16:53.881 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-28T20:16:53.882 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-28T20:16:53.882 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci 2023-11-28T20:16:53.882 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-28T20:16:53.882 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net 2023-11-28T20:16:53.883 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding 2023-11-28T20:16:53.883 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-28T20:16:53.883 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa 2023-11-28T20:16:53.883 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53 2023-11-28T20:16:53.883 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-28T20:16:53.884 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-28T20:16:53.884 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-28T20:16:53.884 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx 2023-11-28T20:16:53.884 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-28T20:16:53.884 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-28T20:16:53.884 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-28T20:16:53.884 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet 2023-11-28T20:16:53.885 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com 2023-11-28T20:16:53.885 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-28T20:16:53.885 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-28T20:16:53.885 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-28T20:16:53.885 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-28T20:16:53.885 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390 2023-11-28T20:16:53.885 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-28T20:16:53.886 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-28T20:16:53.886 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-28T20:16:53.886 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-28T20:16:53.886 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec 2023-11-28T20:16:53.886 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-28T20:16:53.886 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere 2023-11-28T20:16:53.886 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-28T20:16:53.886 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon 2023-11-28T20:16:53.887 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-28T20:16:53.887 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera 2023-11-28T20:16:53.887 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-28T20:16:53.887 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd 2023-11-28T20:16:53.887 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-28T20:16:53.887 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-28T20:16:53.887 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-28T20:16:53.887 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros 2023-11-28T20:16:53.888 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx 2023-11-28T20:16:53.888 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-28T20:16:53.888 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-28T20:16:53.888 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-28T20:16:53.888 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-28T20:16:53.888 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-28T20:16:53.888 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx 2023-11-28T20:16:53.889 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-28T20:16:53.889 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-28T20:16:53.889 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom 2023-11-28T20:16:53.889 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-28T20:16:53.889 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-28T20:16:53.889 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-28T20:16:53.889 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-28T20:16:53.889 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-28T20:16:53.889 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-28T20:16:53.890 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip 2023-11-28T20:16:53.891 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-28T20:16:53.892 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-28T20:16:53.892 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-28T20:16:53.892 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-28T20:16:53.892 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-28T20:16:53.892 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-28T20:16:53.892 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink 2023-11-28T20:16:53.892 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-28T20:16:53.892 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-28T20:16:53.893 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-28T20:16:53.893 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex 2023-11-28T20:16:53.893 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet 2023-11-28T20:16:53.893 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-28T20:16:53.893 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-28T20:16:53.893 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-28T20:16:53.893 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel 2023-11-28T20:16:53.893 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf 2023-11-28T20:16:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-28T20:16:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-28T20:16:53.896 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic 2023-11-28T20:16:53.897 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-28T20:16:53.898 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-28T20:16:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti 2023-11-28T20:16:53.900 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154 2023-11-28T20:16:53.901 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-28T20:16:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/net_failover.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-28T20:16:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/dp83640.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-28T20:16:53.904 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-28T20:16:53.905 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slhc.ko 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-28T20:16:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tun.ko 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/virtio_net.ko 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netfront.ko 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme 2023-11-28T20:16:53.907 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-28T20:16:53.908 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-28T20:16:53.909 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-28T20:16:53.910 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap 2023-11-28T20:16:53.911 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-28T20:16:53.912 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-28T20:16:53.913 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-28T20:16:53.914 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-28T20:16:53.915 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-28T20:16:53.915 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-28T20:16:53.915 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-28T20:16:53.915 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-28T20:16:53.915 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi 2023-11-28T20:16:53.915 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-28T20:16:53.915 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-28T20:16:53.915 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-28T20:16:53.916 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-28T20:16:53.916 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-28T20:16:53.916 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid 2023-11-28T20:16:53.916 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-28T20:16:53.917 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-28T20:16:53.917 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx 2023-11-28T20:16:53.917 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-28T20:16:53.917 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-28T20:16:53.917 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-28T20:16:53.917 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr 2023-11-28T20:16:53.917 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-28T20:16:53.918 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-28T20:16:53.918 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi 2023-11-28T20:16:53.918 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-28T20:16:53.918 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa 2023-11-28T20:16:53.918 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-28T20:16:53.918 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc 2023-11-28T20:16:53.919 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-28T20:16:53.919 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i 2023-11-28T20:16:53.919 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-28T20:16:53.919 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-28T20:16:53.919 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor 2023-11-28T20:16:53.919 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-28T20:16:53.920 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi 2023-11-28T20:16:53.920 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-28T20:16:53.920 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-28T20:16:53.920 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-28T20:16:53.920 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-28T20:16:53.920 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-28T20:16:53.920 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-28T20:16:53.921 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler 2023-11-28T20:16:53.921 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-28T20:16:53.921 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-28T20:16:53.921 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-28T20:16:53.921 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-28T20:16:53.921 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-28T20:16:53.921 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r 2023-11-28T20:16:53.922 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-28T20:16:53.922 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-28T20:16:53.922 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe 2023-11-28T20:16:53.922 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-28T20:16:53.922 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-28T20:16:53.922 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic 2023-11-28T20:16:53.923 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-28T20:16:53.923 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-28T20:16:53.923 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-28T20:16:53.923 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-28T20:16:53.923 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-28T20:16:53.923 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-28T20:16:53.924 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-28T20:16:53.924 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci 2023-11-28T20:16:53.924 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-28T20:16:53.924 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-28T20:16:53.924 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-28T20:16:53.924 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-28T20:16:53.925 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-28T20:16:53.926 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-28T20:16:53.927 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb 2023-11-28T20:16:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-28T20:16:53.929 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-28T20:16:53.930 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-28T20:16:53.931 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-28T20:16:53.931 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-28T20:16:53.931 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-28T20:16:53.931 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-28T20:16:53.931 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-28T20:16:53.933 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty 2023-11-28T20:16:53.934 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless 2023-11-28T20:16:53.934 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-28T20:16:53.934 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-28T20:16:53.934 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-28T20:16:53.934 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-28T20:16:53.934 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-28T20:16:53.935 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image 2023-11-28T20:16:53.936 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga 2023-11-28T20:16:53.937 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-28T20:16:53.938 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-28T20:16:53.939 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-28T20:16:53.940 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-28T20:16:53.941 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-28T20:16:53.942 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip 2023-11-28T20:16:53.943 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost 2023-11-28T20:16:53.944 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-28T20:16:53.945 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-28T20:16:53.945 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-28T20:16:53.945 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-28T20:16:53.945 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video 2023-11-28T20:16:53.945 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight 2023-11-28T20:16:53.945 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-28T20:16:53.946 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-28T20:16:53.946 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-28T20:16:53.946 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio 2023-11-28T20:16:53.946 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio.ko 2023-11-28T20:16:53.946 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_balloon.ko 2023-11-28T20:16:53.946 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_mmio.ko 2023-11-28T20:16:53.947 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci.ko 2023-11-28T20:16:53.947 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-28T20:16:53.947 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-28T20:16:53.947 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_ring.ko 2023-11-28T20:16:53.947 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1 2023-11-28T20:16:53.947 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters 2023-11-28T20:16:53.947 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-28T20:16:53.947 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-28T20:16:53.947 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves 2023-11-28T20:16:53.948 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-28T20:16:53.948 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-28T20:16:53.948 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-28T20:16:53.948 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-28T20:16:53.948 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-28T20:16:53.948 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-28T20:16:53.948 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-28T20:16:53.948 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-28T20:16:53.949 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-28T20:16:53.949 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-28T20:16:53.949 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-28T20:16:53.949 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-28T20:16:53.949 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog 2023-11-28T20:16:53.949 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-28T20:16:53.949 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-28T20:16:53.949 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-28T20:16:53.949 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-28T20:16:53.950 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-28T20:16:53.950 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-28T20:16:53.950 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-28T20:16:53.950 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-28T20:16:53.950 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-28T20:16:53.950 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-28T20:16:53.950 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-28T20:16:53.950 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-28T20:16:53.951 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-28T20:16:53.951 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-28T20:16:53.951 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-28T20:16:53.951 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-28T20:16:53.951 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-28T20:16:53.951 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-28T20:16:53.951 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-28T20:16:53.952 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-28T20:16:53.952 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-28T20:16:53.952 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-28T20:16:53.952 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-28T20:16:53.952 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-28T20:16:53.952 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-28T20:16:53.952 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-28T20:16:53.952 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen 2023-11-28T20:16:53.953 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-28T20:16:53.953 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-28T20:16:53.953 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-28T20:16:53.953 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-28T20:16:53.953 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback 2023-11-28T20:16:53.953 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-28T20:16:53.953 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-28T20:16:53.953 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs 2023-11-28T20:16:53.954 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-28T20:16:53.954 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs 2023-11-28T20:16:53.954 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p 2023-11-28T20:16:53.954 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-28T20:16:53.954 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs 2023-11-28T20:16:53.954 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-28T20:16:53.954 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs 2023-11-28T20:16:53.954 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-28T20:16:53.955 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-28T20:16:53.955 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs 2023-11-28T20:16:53.955 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-28T20:16:53.955 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles 2023-11-28T20:16:53.955 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-28T20:16:53.955 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph 2023-11-28T20:16:53.955 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-28T20:16:53.956 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda 2023-11-28T20:16:53.956 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-28T20:16:53.956 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs 2023-11-28T20:16:53.956 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-28T20:16:53.956 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm 2023-11-28T20:16:53.956 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-28T20:16:53.956 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs 2023-11-28T20:16:53.956 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-28T20:16:53.957 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs 2023-11-28T20:16:53.957 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-28T20:16:53.957 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat 2023-11-28T20:16:53.957 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/fat.ko 2023-11-28T20:16:53.957 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-28T20:16:53.957 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/vfat.ko 2023-11-28T20:16:53.957 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache 2023-11-28T20:16:53.957 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-28T20:16:53.957 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse 2023-11-28T20:16:53.958 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-28T20:16:53.958 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/fuse.ko 2023-11-28T20:16:53.958 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2 2023-11-28T20:16:53.958 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-28T20:16:53.958 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs 2023-11-28T20:16:53.958 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-28T20:16:53.958 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus 2023-11-28T20:16:53.959 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-28T20:16:53.959 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs 2023-11-28T20:16:53.959 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-28T20:16:53.959 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs 2023-11-28T20:16:53.959 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-28T20:16:53.959 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd 2023-11-28T20:16:53.959 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-28T20:16:53.960 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix 2023-11-28T20:16:53.960 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-28T20:16:53.960 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs 2023-11-28T20:16:53.960 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-28T20:16:53.960 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs 2023-11-28T20:16:53.960 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout 2023-11-28T20:16:53.960 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-28T20:16:53.960 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout 2023-11-28T20:16:53.961 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-28T20:16:53.961 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout 2023-11-28T20:16:53.961 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-28T20:16:53.961 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-28T20:16:53.961 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-28T20:16:53.961 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-28T20:16:53.962 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common 2023-11-28T20:16:53.962 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-28T20:16:53.962 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-28T20:16:53.962 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd 2023-11-28T20:16:53.962 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-28T20:16:53.962 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2 2023-11-28T20:16:53.962 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-28T20:16:53.962 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-28T20:16:53.963 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-28T20:16:53.964 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-28T20:16:53.965 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-28T20:16:53.965 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-28T20:16:53.965 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-28T20:16:53.965 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-28T20:16:53.965 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-28T20:16:53.965 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-28T20:16:53.965 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-28T20:16:53.965 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-28T20:16:53.966 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-28T20:16:53.966 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-28T20:16:53.966 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-28T20:16:53.966 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-28T20:16:53.966 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-28T20:16:53.966 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-28T20:16:53.967 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-28T20:16:53.968 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-28T20:16:53.969 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-28T20:16:53.969 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-28T20:16:53.969 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-28T20:16:53.969 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs 2023-11-28T20:16:53.969 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-28T20:16:53.969 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore 2023-11-28T20:16:53.969 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-28T20:16:53.970 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs 2023-11-28T20:16:53.970 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-28T20:16:53.970 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs 2023-11-28T20:16:53.970 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-28T20:16:53.970 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb 2023-11-28T20:16:53.970 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client 2023-11-28T20:16:53.970 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-28T20:16:53.971 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common 2023-11-28T20:16:53.971 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-28T20:16:53.971 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-28T20:16:53.971 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs 2023-11-28T20:16:53.971 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-28T20:16:53.971 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv 2023-11-28T20:16:53.971 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-28T20:16:53.972 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf 2023-11-28T20:16:53.972 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-28T20:16:53.972 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs 2023-11-28T20:16:53.972 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-28T20:16:53.972 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs 2023-11-28T20:16:53.972 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-28T20:16:53.973 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel 2023-11-28T20:16:53.973 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu 2023-11-28T20:16:53.973 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu/rcutorture.ko 2023-11-28T20:16:53.973 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-28T20:16:53.973 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace 2023-11-28T20:16:53.973 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-28T20:16:53.974 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib 2023-11-28T20:16:53.974 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/asn1_encoder.ko 2023-11-28T20:16:53.974 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-28T20:16:53.974 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-28T20:16:53.974 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto 2023-11-28T20:16:53.974 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6 2023-11-28T20:16:53.975 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd/zstd_compress.ko 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-28T20:16:53.976 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-28T20:16:53.977 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-28T20:16:53.978 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-28T20:16:53.979 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-28T20:16:53.980 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/failover.ko 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver/dns_resolver.ko 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-28T20:16:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-28T20:16:53.982 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-28T20:16:53.983 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-28T20:16:53.984 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-28T20:16:53.985 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-28T20:16:53.986 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-28T20:16:53.987 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-28T20:16:53.988 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-28T20:16:53.989 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset 2023-11-28T20:16:53.990 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-28T20:16:53.991 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-28T20:16:53.992 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-28T20:16:53.993 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-28T20:16:53.994 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-28T20:16:53.995 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_quota.ko 2023-11-28T20:16:53.996 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tunnel.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-28T20:16:53.997 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-28T20:16:53.998 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-28T20:16:53.999 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-28T20:16:54.000 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-28T20:16:54.001 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh 2023-11-28T20:16:54.002 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-28T20:16:54.003 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-28T20:16:54.004 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-28T20:16:54.005 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-28T20:16:54.006 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-28T20:16:54.007 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-28T20:16:54.008 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys/trusted.ko 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-28T20:16:54.009 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-28T20:16:54.010 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-28T20:16:54.010 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-28T20:16:54.010 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-28T20:16:54.010 INFO:teuthology.orchestra.run.smithi190.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-28T20:16:54.010 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-28T20:16:54.011 DEBUG:teuthology.orchestra.run.smithi190:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-28T20:16:54.034 INFO:teuthology.orchestra.run.smithi190.stdout:postinstall scriptlet (using /bin/sh): 2023-11-28T20:16:54.034 INFO:teuthology.orchestra.run.smithi190.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-g8247bbb5b470 -a -r /boot/System.map-6.6.0-g8247bbb5b470 ]; then 2023-11-28T20:16:54.034 INFO:teuthology.orchestra.run.smithi190.stdout:cp /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm 2023-11-28T20:16:54.034 INFO:teuthology.orchestra.run.smithi190.stdout:cp /boot/System.map-6.6.0-g8247bbb5b470 /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:rm -f /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:/sbin/installkernel 6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:rm -f /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:fi 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:new-kernel-pkg --remove 6.6.0-g8247bbb5b470 --rminitrd --initrdfile=/boot/initramfs-6.6.0-g8247bbb5b470.img 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-install remove 6.6.0-g8247bbb5b470 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:fi 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-28T20:16:54.035 INFO:teuthology.orchestra.run.smithi190.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-28T20:16:54.036 INFO:teuthology.orchestra.run.smithi190.stdout:/sbin/update-bootloader --remove 6.6.0-g8247bbb5b470 2023-11-28T20:16:54.036 INFO:teuthology.orchestra.run.smithi190.stdout:fi 2023-11-28T20:16:54.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2023-11-28T20:16:54.110 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2023-11-28T20:16:54.110 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 68.el9 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8264254 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-28T20:16:54.111 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:16:54.112 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2023-11-28T20:16:54.113 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-28T20:16:54.113 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 6.6.0-g8247bbb5b470 2023-11-28T20:16:54.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-28T20:16:54.671 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2023-11-28T20:16:56.172 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-28T20:16:56.202 INFO:teuthology.orchestra.run.smithi190.stderr:done 2023-11-28T20:16:56.205 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2023-11-28T20:16:56.220 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.xADAqbr1cM 2023-11-28T20:16:56.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xADAqbr1cM 2023-11-28T20:16:56.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.xADAqbr1cM 2023-11-28T20:16:56.439 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.xADAqbr1cM is 6KB 2023-11-28T20:16:56.489 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.xADAqbr1cM 2023-11-28T20:16:56.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2023-11-28T20:16:56.568 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-28T20:16:56.568 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-28T20:16:56.568 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-28T20:16:56.569 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470.conf 2023-11-28T20:16:56.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470 2023-11-28T20:16:56.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2023-11-28T20:16:57.759 INFO:teuthology.orchestra.run.smithi045.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-28T20:17:19.691 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-28T20:17:19.692 DEBUG:teuthology.orchestra.run.smithi045:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-28T20:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T20:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/config-6.6.0-g8247bbb5b470 2023-11-28T20:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T20:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470 2023-11-28T20:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-28T20:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-28T20:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-28T20:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-28T20:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-28T20:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-28T20:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-28T20:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-28T20:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-28T20:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-28T20:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-28T20:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-28T20:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-28T20:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-28T20:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers 2023-11-28T20:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi 2023-11-28T20:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-28T20:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-28T20:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/custom_method.ko 2023-11-28T20:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-28T20:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-28T20:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-28T20:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-28T20:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-28T20:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata 2023-11-28T20:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ata_generic.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-28T20:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-28T20:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-28T20:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-28T20:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-28T20:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-28T20:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-28T20:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-28T20:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-28T20:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-28T20:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-28T20:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sis.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-28T20:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-28T20:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe 2023-11-28T20:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/brd.ko 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/loop.ko 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-28T20:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/virtio_blk.ko 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkfront.ko 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random 2023-11-28T20:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-28T20:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/uv_mmtimer.ko 2023-11-28T20:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/virtio_console.ko 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel 2023-11-28T20:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq 2023-11-28T20:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-28T20:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/edd.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-28T20:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-28T20:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-28T20:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-28T20:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-28T20:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-28T20:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon 2023-11-28T20:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-28T20:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-28T20:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-28T20:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-28T20:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-28T20:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-28T20:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-28T20:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-28T20:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-28T20:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-28T20:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-28T20:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-28T20:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-dev.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-28T20:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4 2023-11-28T20:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-28T20:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-28T20:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-28T20:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-28T20:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-28T20:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-28T20:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-28T20:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd 2023-11-28T20:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mfd-core.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-28T20:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-28T20:17:19.819 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-28T20:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-28T20:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-28T20:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-28T20:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom 2023-11-28T20:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio 2023-11-28T20:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic 2023-11-28T20:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-28T20:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k 2023-11-28T20:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-28T20:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel 2023-11-28T20:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-28T20:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-28T20:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-28T20:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via 2023-11-28T20:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan 2023-11-28T20:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/net_failover.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-28T20:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-28T20:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/dp83640.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-28T20:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slhc.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-28T20:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tun.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/virtio_net.ko 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3 2023-11-28T20:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-28T20:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan 2023-11-28T20:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-28T20:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback 2023-11-28T20:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-28T20:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netfront.ko 2023-11-28T20:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb 2023-11-28T20:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-28T20:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme 2023-11-28T20:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-28T20:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-28T20:17:19.843 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-28T20:17:19.844 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps 2023-11-28T20:17:19.845 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-28T20:17:19.846 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-28T20:17:19.847 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-28T20:17:19.848 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-28T20:17:19.849 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-28T20:17:19.850 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-28T20:17:19.851 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-28T20:17:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-28T20:17:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas 2023-11-28T20:17:19.854 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-28T20:17:19.855 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-28T20:17:19.856 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-28T20:17:19.857 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-28T20:17:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-28T20:17:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host 2023-11-28T20:17:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-28T20:17:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-28T20:17:19.862 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy 2023-11-28T20:17:19.863 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-28T20:17:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-28T20:17:19.865 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-28T20:17:19.866 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-28T20:17:19.867 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-28T20:17:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-28T20:17:19.869 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio.ko 2023-11-28T20:17:19.870 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_balloon.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_mmio.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_ring.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-28T20:17:19.871 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-28T20:17:19.872 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-28T20:17:19.873 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-28T20:17:19.874 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs 2023-11-28T20:17:19.875 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs 2023-11-28T20:17:19.876 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-28T20:17:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm 2023-11-28T20:17:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-28T20:17:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs 2023-11-28T20:17:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-28T20:17:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs 2023-11-28T20:17:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-28T20:17:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat 2023-11-28T20:17:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/fat.ko 2023-11-28T20:17:19.877 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/vfat.ko 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/fuse.ko 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-28T20:17:19.878 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-28T20:17:19.879 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd 2023-11-28T20:17:19.880 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-28T20:17:19.881 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-28T20:17:19.882 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-28T20:17:19.883 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-28T20:17:19.884 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-28T20:17:19.885 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-28T20:17:19.886 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel 2023-11-28T20:17:19.887 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu/rcutorture.ko 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/asn1_encoder.ko 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-28T20:17:19.888 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd 2023-11-28T20:17:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd/zstd_compress.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-28T20:17:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-28T20:17:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-28T20:17:19.892 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-28T20:17:19.893 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/failover.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-28T20:17:19.894 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver/dns_resolver.ko 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-28T20:17:19.895 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-28T20:17:19.896 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-28T20:17:19.897 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-28T20:17:19.898 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-28T20:17:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-28T20:17:19.900 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-28T20:17:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-28T20:17:19.902 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls 2023-11-28T20:17:19.903 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-28T20:17:19.904 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-28T20:17:19.905 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-28T20:17:19.906 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-28T20:17:19.907 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-28T20:17:19.908 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-28T20:17:19.909 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_quota.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tunnel.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-28T20:17:19.910 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-28T20:17:19.911 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-28T20:17:19.912 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-28T20:17:19.913 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-28T20:17:19.914 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-28T20:17:19.915 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-28T20:17:19.916 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-28T20:17:19.917 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-28T20:17:19.918 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-28T20:17:19.919 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-28T20:17:19.920 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-28T20:17:19.921 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys 2023-11-28T20:17:19.922 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys/trusted.ko 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-28T20:17:19.923 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-28T20:17:19.924 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-28T20:17:19.924 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-28T20:17:19.924 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-28T20:17:19.924 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-28T20:17:19.924 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-28T20:17:19.924 DEBUG:teuthology.orchestra.run.smithi045:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-28T20:17:19.947 INFO:teuthology.orchestra.run.smithi045.stdout:postinstall scriptlet (using /bin/sh): 2023-11-28T20:17:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-g8247bbb5b470 -a -r /boot/System.map-6.6.0-g8247bbb5b470 ]; then 2023-11-28T20:17:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:cp /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm 2023-11-28T20:17:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:cp /boot/System.map-6.6.0-g8247bbb5b470 /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T20:17:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:rm -f /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T20:17:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/installkernel 6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T20:17:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:rm -f /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T20:17:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2023-11-28T20:17:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-28T20:17:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-28T20:17:19.949 INFO:teuthology.orchestra.run.smithi045.stdout:new-kernel-pkg --remove 6.6.0-g8247bbb5b470 --rminitrd --initrdfile=/boot/initramfs-6.6.0-g8247bbb5b470.img 2023-11-28T20:17:19.949 INFO:teuthology.orchestra.run.smithi045.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-28T20:17:19.949 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-install remove 6.6.0-g8247bbb5b470 2023-11-28T20:17:19.949 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2023-11-28T20:17:19.949 INFO:teuthology.orchestra.run.smithi045.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-28T20:17:19.949 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-28T20:17:19.949 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/update-bootloader --remove 6.6.0-g8247bbb5b470 2023-11-28T20:17:19.949 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2023-11-28T20:17:19.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2023-11-28T20:17:20.029 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 68.el9 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8264254 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-28T20:17:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-28T20:17:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-28T20:17:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2023-11-28T20:17:20.032 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:17:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2023-11-28T20:17:20.033 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-28T20:17:20.033 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 6.6.0-g8247bbb5b470 2023-11-28T20:17:20.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-28T20:17:20.660 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2023-11-28T20:17:22.112 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-28T20:17:22.150 INFO:teuthology.orchestra.run.smithi045.stderr:done 2023-11-28T20:17:22.153 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2023-11-28T20:17:22.170 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.rjsFc1d2kC 2023-11-28T20:17:22.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rjsFc1d2kC 2023-11-28T20:17:22.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.rjsFc1d2kC 2023-11-28T20:17:22.382 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.rjsFc1d2kC is 6KB 2023-11-28T20:17:22.434 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.rjsFc1d2kC 2023-11-28T20:17:22.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2023-11-28T20:17:22.517 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-28T20:17:22.517 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-28T20:17:22.517 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-28T20:17:22.517 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470.conf 2023-11-28T20:17:22.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470 2023-11-28T20:17:22.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2023-11-28T20:17:26.710 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-11-28T20:17:26.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-28T20:17:26.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:17:45.058 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-11-28T20:17:52.682 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-11-28T20:17:52.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-11-28T20:17:52.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:17:54.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-28T20:17:54.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:17:57.122 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-11-28T20:18:09.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-28T20:18:09.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:18:11.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2023-11-28T20:18:20.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-11-28T20:18:20.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:19:09.177 DEBUG:teuthology.orchestra.remote:timed out 2023-11-28T20:19:20.190 DEBUG:teuthology.orchestra.remote:timed out 2023-11-28T20:19:24.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-28T20:19:24.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:19:24.639 DEBUG:teuthology.orchestra.run.smithi190:> true 2023-11-28T20:19:25.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-28T20:19:25.316 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.6.0-g8247bbb5b470"... 2023-11-28T20:19:25.316 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2023-11-28T20:19:25.369 INFO:teuthology.orchestra.run.smithi190.stdout:6.6.0-g8247bbb5b470 2023-11-28T20:19:25.369 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-28T20:19:25.369 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T20:19:25.369 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-28T20:19:26.371 DEBUG:teuthology.parallel:result is None 2023-11-28T20:19:32.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-11-28T20:19:32.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:19:32.560 DEBUG:teuthology.orchestra.run.smithi045:> true 2023-11-28T20:19:33.219 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-11-28T20:19:33.220 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-g8247bbb5b470"... 2023-11-28T20:19:33.220 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2023-11-28T20:19:33.279 INFO:teuthology.orchestra.run.smithi045.stdout:6.6.0-g8247bbb5b470 2023-11-28T20:19:33.280 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-28T20:19:33.280 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T20:19:33.280 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-28T20:19:34.281 DEBUG:teuthology.parallel:result is None 2023-11-28T20:19:34.281 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-28T20:19:34.289 INFO:teuthology.task.internal:Creating test directory... 2023-11-28T20:19:34.289 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T20:19:34.298 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T20:19:34.358 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-28T20:19:34.366 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-28T20:19:34.399 INFO:teuthology.task.internal:Creating archive directory... 2023-11-28T20:19:34.400 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T20:19:34.406 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T20:19:34.484 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-28T20:19:34.491 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-28T20:19:34.492 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T20:19:34.501 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T20:19:34.637 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T20:19:34.644 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T20:19:34.699 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T20:19:34.719 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T20:19:34.724 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-28T20:19:34.731 INFO:teuthology.task.internal:Configuring sudo... 2023-11-28T20:19:34.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T20:19:34.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T20:19:34.841 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-28T20:19:34.849 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-28T20:19:34.849 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T20:19:34.858 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T20:19:34.917 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T20:19:35.012 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T20:19:35.127 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T20:19:35.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T20:19:35.318 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:19:35.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T20:19:35.425 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T20:19:35.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T20:19:35.630 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T20:19:35.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T20:19:35.816 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:19:35.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T20:19:35.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2023-11-28T20:19:35.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2023-11-28T20:19:36.069 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T20:19:36.088 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T20:19:36.357 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-28T20:19:36.365 INFO:teuthology.task.internal:Starting timer... 2023-11-28T20:19:36.365 INFO:teuthology.run_tasks:Running task pcp... 2023-11-28T20:19:36.444 INFO:teuthology.run_tasks:Running task selinux... 2023-11-28T20:19:36.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2023-11-28T20:19:36.630 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2023-11-28T20:19:36.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2023-11-28T20:19:36.760 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2023-11-28T20:19:36.765 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-28T20:19:36.765 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2023-11-28T20:19:36.829 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2023-11-28T20:19:36.830 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2023-11-28T20:19:36.901 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2023-11-28T20:19:36.902 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2023-11-28T20:19:36.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T20:19:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T20:19:37.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T20:19:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T20:19:37.099 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-28T20:19:37.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2023-11-28T20:19:37.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2023-11-28T20:19:37.341 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-28T20:19:37.353 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-28T20:19:37.363 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-28T20:19:37.475 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-28T20:19:37.490 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-28T20:19:37.491 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-28T20:30:02.962 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2023-11-28T20:30:02.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-11-28T20:30:02.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:30:03.106 DEBUG:teuthology.orchestra.run.smithi045:> true 2023-11-28T20:30:03.275 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-11-28T20:30:03.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-28T20:30:03.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T20:30:03.408 DEBUG:teuthology.orchestra.run.smithi190:> true 2023-11-28T20:30:03.556 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-28T20:30:03.556 INFO:teuthology.run_tasks:Running task clock... 2023-11-28T20:30:03.570 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-28T20:30:03.571 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T20:30:03.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T20:30:03.581 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T20:30:03.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T20:30:03.687 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T20:30:03.691 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T20:30:03.741 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T20:30:03.743 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T20:30:03.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2023-11-28T20:30:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2023-11-28T20:30:03.880 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2023-11-28T20:30:03.893 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2023-11-28T20:30:03.931 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T20:30:03.948 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T20:30:03.980 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T20:30:04.003 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T20:30:04.109 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2023-11-28T20:30:04.119 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T20:30:04.119 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2023-11-28T20:30:04.119 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T20:30:04.119 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T20:30:04.119 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T20:30:04.119 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T20:30:04.139 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2023-11-28T20:30:04.142 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T20:30:04.142 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2023-11-28T20:30:04.142 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T20:30:04.143 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T20:30:04.143 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T20:30:04.143 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T20:30:04.147 INFO:teuthology.run_tasks:Running task install... 2023-11-28T20:30:04.156 DEBUG:teuthology.task.install:project ceph 2023-11-28T20:30:04.157 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883'}} 2023-11-28T20:30:04.157 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883'} 2023-11-28T20:30:04.157 INFO:teuthology.task.install:Using flavor: default 2023-11-28T20:30:04.163 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-28T20:30:04.163 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-28T20:30:04.164 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883', 'tag': None, 'wait_for_package': False} 2023-11-28T20:30:04.164 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=43dd94c190073cf16cff1e711dd7f5a7b1cbc883 2023-11-28T20:30:04.166 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883', 'tag': None, 'wait_for_package': False} 2023-11-28T20:30:04.166 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=43dd94c190073cf16cff1e711dd7f5a7b1cbc883 2023-11-28T20:30:04.310 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102914/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/centos/9/flavors/default/ 2023-11-28T20:30:04.311 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7520.g43dd94c1 2023-11-28T20:30:04.383 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102914/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/centos/9/flavors/default/ 2023-11-28T20:30:04.383 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7520.g43dd94c1 2023-11-28T20:30:04.637 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102914/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102914/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102914/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T20:30:04.637 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:30:04.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T20:30:04.705 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102914/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102914/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102914/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T20:30:04.705 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:30:04.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T20:30:04.747 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-28T20:30:04.747 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T20:30:04.807 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-28T20:30:04.807 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/43dd94c190073cf16cff1e711dd7f5a7b1cbc883/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T20:30:04.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T20:30:04.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T20:30:05.034 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T20:30:05.084 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T20:30:05.134 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2023-11-28T20:30:05.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2023-11-28T20:30:05.175 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2023-11-28T20:30:05.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2023-11-28T20:30:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:73 files removed 2023-11-28T20:30:05.531 INFO:teuthology.orchestra.run.smithi190.stdout:73 files removed 2023-11-28T20:30:05.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-28T20:30:05.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-28T20:30:06.869 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 100 kB/s | 78 kB 00:00 2023-11-28T20:30:06.982 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 87 kB/s | 78 kB 00:00 2023-11-28T20:30:07.770 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 12 kB/s | 9.5 kB 00:00 2023-11-28T20:30:07.873 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 12 kB/s | 9.5 kB 00:00 2023-11-28T20:30:08.657 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2023-11-28T20:30:08.745 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2023-11-28T20:30:09.927 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 5.3 MB/s | 6.4 MB 00:01 2023-11-28T20:30:10.041 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 5.2 MB/s | 6.4 MB 00:01 2023-11-28T20:30:11.485 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2023-11-28T20:30:11.733 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2023-11-28T20:30:21.980 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 823 kB/s | 7.9 MB 00:09 2023-11-28T20:30:25.558 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 591 kB/s | 7.9 MB 00:13 2023-11-28T20:30:37.582 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 18 MB 00:14 2023-11-28T20:30:40.671 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 18 MB 00:13 2023-11-28T20:30:49.980 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 5.8 MB 00:05 2023-11-28T20:30:51.499 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 15 kB 00:00 2023-11-28T20:30:54.189 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 465 kB/s | 5.8 MB 00:12 2023-11-28T20:30:56.015 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2023-11-28T20:30:58.623 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 20 MB 00:02 2023-11-28T20:31:01.722 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 2.0 MB/s | 20 MB 00:10 2023-11-28T20:31:04.515 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2023-11-28T20:31:07.334 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2023-11-28T20:31:07.458 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T20:31:07.460 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T20:31:07.482 INFO:teuthology.orchestra.run.smithi045.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-11-28T20:31:07.483 INFO:teuthology.orchestra.run.smithi045.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-11-28T20:31:07.485 INFO:teuthology.orchestra.run.smithi045.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-11-28T20:31:07.486 INFO:teuthology.orchestra.run.smithi045.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-11-28T20:31:07.489 INFO:teuthology.orchestra.run.smithi045.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-11-28T20:31:07.493 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-11-28T20:31:07.494 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-11-28T20:31:07.582 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-11-28T20:31:07.598 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2023-11-28T20:31:07.598 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-11-28T20:31:07.598 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 6.6 k 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 5.2 M 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 860 k 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 141 k 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 1.5 M 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 133 k 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 1.8 M 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 7.4 M 2023-11-28T20:31:07.599 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 50 k 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 8.6 M 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 41 M 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 262 k 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 774 k 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 217 k 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 18 k 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-11-28T20:31:07.600 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 31 k 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 719 k 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 126 k 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 162 k 2023-11-28T20:31:07.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 320 k 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 299 k 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 99 k 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 85 k 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 3.0 M 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 170 k 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager x86_64 1:1.45.7-1.el9 baseos 2.3 M 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm x86_64 1:1.45.7-1.el9 baseos 1.8 M 2023-11-28T20:31:07.602 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team x86_64 1:1.45.7-1.el9 baseos 43 k 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui x86_64 1:1.45.7-1.el9 baseos 246 k 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 3.3 M 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 3.0 M 2023-11-28T20:31:07.603 INFO:teuthology.orchestra.run.smithi045.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 19 M 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 28 k 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 2.1 M 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 243 k 2023-11-28T20:31:07.604 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 4.4 M 2023-11-28T20:31:07.605 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 18 M 2023-11-28T20:31:07.605 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 15 k 2023-11-28T20:31:07.605 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 24 k 2023-11-28T20:31:07.605 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-28T20:31:07.605 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-28T20:31:07.605 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-28T20:31:07.605 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-28T20:31:07.605 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-28T20:31:07.605 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-28T20:31:07.606 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-28T20:31:07.606 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-28T20:31:07.606 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 164 k 2023-11-28T20:31:07.606 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-28T20:31:07.606 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-28T20:31:07.606 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-28T20:31:07.607 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-28T20:31:07.607 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-28T20:31:07.607 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 476 k 2023-11-28T20:31:07.607 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-28T20:31:07.607 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 4.8 M 2023-11-28T20:31:07.607 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-28T20:31:07.607 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-11-28T20:31:07.607 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-28T20:31:07.607 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-28T20:31:07.608 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 45 k 2023-11-28T20:31:07.609 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 129 k 2023-11-28T20:31:07.609 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-28T20:31:07.609 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-28T20:31:07.609 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-28T20:31:07.610 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-28T20:31:07.610 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-28T20:31:07.610 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-28T20:31:07.611 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-28T20:31:07.611 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-28T20:31:07.611 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-28T20:31:07.611 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-28T20:31:07.611 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-28T20:31:07.611 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-28T20:31:07.611 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-28T20:31:07.611 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-28T20:31:07.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-28T20:31:07.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-28T20:31:07.614 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout:Install 135 Packages 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 15 Packages 2023-11-28T20:31:07.615 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:31:07.617 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 205 M 2023-11-28T20:31:07.617 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-11-28T20:31:08.733 INFO:teuthology.orchestra.run.smithi045.stdout:(1/150): ceph-18.0.0-7520.g43dd94c1.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2023-11-28T20:31:08.967 INFO:teuthology.orchestra.run.smithi045.stdout:(2/150): ceph-fuse-18.0.0-7520.g43dd94c1.el9.x8 3.6 MB/s | 860 kB 00:00 2023-11-28T20:31:09.017 INFO:teuthology.orchestra.run.smithi045.stdout:(3/150): ceph-immutable-object-cache-18.0.0-752 2.8 MB/s | 141 kB 00:00 2023-11-28T20:31:09.309 INFO:teuthology.orchestra.run.smithi045.stdout:(4/150): ceph-mds-18.0.0-7520.g43dd94c1.el9.x86 7.2 MB/s | 2.1 MB 00:00 2023-11-28T20:31:09.451 INFO:teuthology.orchestra.run.smithi045.stdout:(5/150): ceph-base-18.0.0-7520.g43dd94c1.el9.x8 5.4 MB/s | 5.2 MB 00:00 2023-11-28T20:31:09.584 INFO:teuthology.orchestra.run.smithi045.stdout:(6/150): ceph-mgr-18.0.0-7520.g43dd94c1.el9.x86 5.4 MB/s | 1.5 MB 00:00 2023-11-28T20:31:09.893 INFO:teuthology.orchestra.run.smithi045.stdout:(7/150): ceph-mon-18.0.0-7520.g43dd94c1.el9.x86 9.9 MB/s | 4.4 MB 00:00 2023-11-28T20:31:10.235 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T20:31:10.236 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T20:31:10.258 INFO:teuthology.orchestra.run.smithi190.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-11-28T20:31:10.259 INFO:teuthology.orchestra.run.smithi190.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-11-28T20:31:10.260 INFO:teuthology.orchestra.run.smithi190.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-11-28T20:31:10.262 INFO:teuthology.orchestra.run.smithi190.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-11-28T20:31:10.265 INFO:teuthology.orchestra.run.smithi190.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-11-28T20:31:10.269 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-11-28T20:31:10.270 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-11-28T20:31:10.349 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-11-28T20:31:10.385 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 6.6 k 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 5.2 M 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 860 k 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 141 k 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 1.5 M 2023-11-28T20:31:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 133 k 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 1.8 M 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 7.4 M 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 50 k 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 8.6 M 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 41 M 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 262 k 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 774 k 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 217 k 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 18 k 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-11-28T20:31:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 31 k 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 719 k 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 126 k 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 162 k 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 320 k 2023-11-28T20:31:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 299 k 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 99 k 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 85 k 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 3.0 M 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 170 k 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager x86_64 1:1.45.7-1.el9 baseos 2.3 M 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-libnm x86_64 1:1.45.7-1.el9 baseos 1.8 M 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-team x86_64 1:1.45.7-1.el9 baseos 43 k 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-tui x86_64 1:1.45.7-1.el9 baseos 246 k 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-11-28T20:31:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 3.3 M 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 3.0 M 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-11-28T20:31:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 19 M 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 28 k 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 2.1 M 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 243 k 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 4.4 M 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 18 M 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7520.g43dd94c1.el9 ceph-noarch 15 k 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 24 k 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-28T20:31:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 164 k 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-28T20:31:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-28T20:31:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-28T20:31:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 476 k 2023-11-28T20:31:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-28T20:31:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 4.8 M 2023-11-28T20:31:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-28T20:31:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-11-28T20:31:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-28T20:31:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-11-28T20:31:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 45 k 2023-11-28T20:31:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:18.0.0-7520.g43dd94c1.el9 ceph 129 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-28T20:31:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-28T20:31:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-28T20:31:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-28T20:31:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:31:10.399 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-11-28T20:31:10.400 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2023-11-28T20:31:10.400 INFO:teuthology.orchestra.run.smithi190.stdout:Install 135 Packages 2023-11-28T20:31:10.400 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 15 Packages 2023-11-28T20:31:10.400 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:31:10.400 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 205 M 2023-11-28T20:31:10.401 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-11-28T20:31:10.735 INFO:teuthology.orchestra.run.smithi045.stdout:(8/150): ceph-radosgw-18.0.0-7520.g43dd94c1.el9 10 MB/s | 8.6 MB 00:00 2023-11-28T20:31:10.819 INFO:teuthology.orchestra.run.smithi045.stdout:(9/150): ceph-selinux-18.0.0-7520.g43dd94c1.el9 292 kB/s | 24 kB 00:00 2023-11-28T20:31:11.294 INFO:teuthology.orchestra.run.smithi045.stdout:(10/150): ceph-osd-18.0.0-7520.g43dd94c1.el9.x8 10 MB/s | 18 MB 00:01 2023-11-28T20:31:11.370 INFO:teuthology.orchestra.run.smithi045.stdout:(11/150): cephfs-mirror-18.0.0-7520.g43dd94c1.e 2.9 MB/s | 217 kB 00:00 2023-11-28T20:31:11.428 INFO:teuthology.orchestra.run.smithi045.stdout:(12/150): libcephfs-devel-18.0.0-7520.g43dd94c1 545 kB/s | 31 kB 00:00 2023-11-28T20:31:11.576 INFO:teuthology.orchestra.run.smithi190.stdout:(1/150): ceph-18.0.0-7520.g43dd94c1.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2023-11-28T20:31:11.581 INFO:teuthology.orchestra.run.smithi045.stdout:(13/150): libcephfs2-18.0.0-7520.g43dd94c1.el9. 4.7 MB/s | 719 kB 00:00 2023-11-28T20:31:11.646 INFO:teuthology.orchestra.run.smithi045.stdout:(14/150): libcephsqlite-18.0.0-7520.g43dd94c1.e 2.5 MB/s | 164 kB 00:00 2023-11-28T20:31:11.713 INFO:teuthology.orchestra.run.smithi045.stdout:(15/150): librados-devel-18.0.0-7520.g43dd94c1. 1.9 MB/s | 126 kB 00:00 2023-11-28T20:31:11.813 INFO:teuthology.orchestra.run.smithi045.stdout:(16/150): libradosstriper1-18.0.0-7520.g43dd94c 4.7 MB/s | 476 kB 00:00 2023-11-28T20:31:12.080 INFO:teuthology.orchestra.run.smithi045.stdout:(17/150): ceph-common-18.0.0-7520.g43dd94c1.el9 5.3 MB/s | 19 MB 00:03 2023-11-28T20:31:12.144 INFO:teuthology.orchestra.run.smithi190.stdout:(2/150): ceph-fuse-18.0.0-7520.g43dd94c1.el9.x8 1.5 MB/s | 860 kB 00:00 2023-11-28T20:31:12.155 INFO:teuthology.orchestra.run.smithi045.stdout:(18/150): python3-ceph-argparse-18.0.0-7520.g43 612 kB/s | 45 kB 00:00 2023-11-28T20:31:12.256 INFO:teuthology.orchestra.run.smithi045.stdout:(19/150): python3-ceph-common-18.0.0-7520.g43dd 1.3 MB/s | 129 kB 00:00 2023-11-28T20:31:12.294 INFO:teuthology.orchestra.run.smithi190.stdout:(3/150): ceph-immutable-object-cache-18.0.0-752 948 kB/s | 141 kB 00:00 2023-11-28T20:31:12.506 INFO:teuthology.orchestra.run.smithi045.stdout:(20/150): librgw2-18.0.0-7520.g43dd94c1.el9.x86 6.9 MB/s | 4.8 MB 00:00 2023-11-28T20:31:12.565 INFO:teuthology.orchestra.run.smithi045.stdout:(21/150): python3-cephfs-18.0.0-7520.g43dd94c1. 526 kB/s | 162 kB 00:00 2023-11-28T20:31:12.607 INFO:teuthology.orchestra.run.smithi045.stdout:(22/150): python3-rados-18.0.0-7520.g43dd94c1.e 3.1 MB/s | 320 kB 00:00 2023-11-28T20:31:12.657 INFO:teuthology.orchestra.run.smithi045.stdout:(23/150): python3-rgw-18.0.0-7520.g43dd94c1.el9 2.0 MB/s | 99 kB 00:00 2023-11-28T20:31:12.724 INFO:teuthology.orchestra.run.smithi045.stdout:(24/150): rbd-fuse-18.0.0-7520.g43dd94c1.el9.x8 1.3 MB/s | 85 kB 00:00 2023-11-28T20:31:12.735 INFO:teuthology.orchestra.run.smithi190.stdout:(4/150): ceph-mds-18.0.0-7520.g43dd94c1.el9.x86 4.7 MB/s | 2.1 MB 00:00 2023-11-28T20:31:12.758 INFO:teuthology.orchestra.run.smithi045.stdout:(25/150): python3-rbd-18.0.0-7520.g43dd94c1.el9 1.5 MB/s | 299 kB 00:00 2023-11-28T20:31:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:(26/150): rbd-nbd-18.0.0-7520.g43dd94c1.el9.x86 1.1 MB/s | 170 kB 00:00 2023-11-28T20:31:13.042 INFO:teuthology.orchestra.run.smithi045.stdout:(27/150): ceph-grafana-dashboards-18.0.0-7520.g 212 kB/s | 28 kB 00:00 2023-11-28T20:31:13.125 INFO:teuthology.orchestra.run.smithi045.stdout:(28/150): rbd-mirror-18.0.0-7520.g43dd94c1.el9. 7.5 MB/s | 3.0 MB 00:00 2023-11-28T20:31:13.159 INFO:teuthology.orchestra.run.smithi045.stdout:(29/150): ceph-mgr-cephadm-18.0.0-7520.g43dd94c 1.1 MB/s | 133 kB 00:00 2023-11-28T20:31:13.236 INFO:teuthology.orchestra.run.smithi190.stdout:(5/150): ceph-mgr-18.0.0-7520.g43dd94c1.el9.x86 2.9 MB/s | 1.5 MB 00:00 2023-11-28T20:31:13.451 INFO:teuthology.orchestra.run.smithi045.stdout:(30/150): ceph-mgr-dashboard-18.0.0-7520.g43dd9 5.4 MB/s | 1.8 MB 00:00 2023-11-28T20:31:13.535 INFO:teuthology.orchestra.run.smithi045.stdout:(31/150): ceph-mgr-modules-core-18.0.0-7520.g43 2.9 MB/s | 243 kB 00:00 2023-11-28T20:31:13.618 INFO:teuthology.orchestra.run.smithi045.stdout:(32/150): ceph-mgr-rook-18.0.0-7520.g43dd94c1.e 609 kB/s | 50 kB 00:00 2023-11-28T20:31:13.702 INFO:teuthology.orchestra.run.smithi045.stdout:(33/150): ceph-prometheus-alerts-18.0.0-7520.g4 183 kB/s | 15 kB 00:00 2023-11-28T20:31:13.802 INFO:teuthology.orchestra.run.smithi045.stdout:(34/150): ceph-volume-18.0.0-7520.g43dd94c1.el9 2.6 MB/s | 262 kB 00:00 2023-11-28T20:31:13.987 INFO:teuthology.orchestra.run.smithi045.stdout:(35/150): cephadm-18.0.0-7520.g43dd94c1.el9.noa 4.1 MB/s | 774 kB 00:00 2023-11-28T20:31:14.061 INFO:teuthology.orchestra.run.smithi045.stdout:(36/150): cephfs-top-18.0.0-7520.g43dd94c1.el9. 249 kB/s | 18 kB 00:00 2023-11-28T20:31:14.277 INFO:teuthology.orchestra.run.smithi190.stdout:(6/150): ceph-mon-18.0.0-7520.g43dd94c1.el9.x86 4.2 MB/s | 4.4 MB 00:01 2023-11-28T20:31:14.469 INFO:teuthology.orchestra.run.smithi190.stdout:(7/150): ceph-base-18.0.0-7520.g43dd94c1.el9.x8 1.6 MB/s | 5.2 MB 00:03 2023-11-28T20:31:14.620 INFO:teuthology.orchestra.run.smithi045.stdout:(37/150): ceph-test-18.0.0-7520.g43dd94c1.el9.x 11 MB/s | 41 MB 00:03 2023-11-28T20:31:14.654 INFO:teuthology.orchestra.run.smithi045.stdout:(38/150): automake-1.16.2-8.el9.noarch.rpm 1.1 MB/s | 662 kB 00:00 2023-11-28T20:31:14.712 INFO:teuthology.orchestra.run.smithi045.stdout:(39/150): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 920 kB 00:00 2023-11-28T20:31:14.738 INFO:teuthology.orchestra.run.smithi045.stdout:(40/150): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2023-11-28T20:31:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:(41/150): elfutils-libelf-devel-0.190-1.el9.x86 444 kB/s | 21 kB 00:00 2023-11-28T20:31:14.796 INFO:teuthology.orchestra.run.smithi045.stdout:(42/150): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2023-11-28T20:31:14.822 INFO:teuthology.orchestra.run.smithi045.stdout:(43/150): flexiblas-3.0.4-8.el9.x86_64.rpm 575 kB/s | 32 kB 00:00 2023-11-28T20:31:14.922 INFO:teuthology.orchestra.run.smithi045.stdout:(44/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2023-11-28T20:31:14.939 INFO:teuthology.orchestra.run.smithi045.stdout:(45/150): flexiblas-openblas-openmp-3.0.4-8.el9 173 kB/s | 17 kB 00:00 2023-11-28T20:31:14.964 INFO:teuthology.orchestra.run.smithi045.stdout:(46/150): libaio-devel-0.3.111-13.el9.x86_64.rp 427 kB/s | 10 kB 00:00 2023-11-28T20:31:14.998 INFO:teuthology.orchestra.run.smithi045.stdout:(47/150): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2023-11-28T20:31:15.032 INFO:teuthology.orchestra.run.smithi045.stdout:(48/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2023-11-28T20:31:15.049 INFO:teuthology.orchestra.run.smithi045.stdout:(49/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 919 kB/s | 44 kB 00:00 2023-11-28T20:31:15.116 INFO:teuthology.orchestra.run.smithi045.stdout:(50/150): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2023-11-28T20:31:15.158 INFO:teuthology.orchestra.run.smithi045.stdout:(51/150): libtool-2.4.6-45.el9.x86_64.rpm 5.6 MB/s | 578 kB 00:00 2023-11-28T20:31:15.183 INFO:teuthology.orchestra.run.smithi045.stdout:(52/150): libuuid-devel-2.37.4-15.el9.x86_64.rp 401 kB/s | 24 kB 00:00 2023-11-28T20:31:15.208 INFO:teuthology.orchestra.run.smithi045.stdout:(53/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2023-11-28T20:31:15.242 INFO:teuthology.orchestra.run.smithi045.stdout:(54/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-11-28T20:31:15.276 INFO:teuthology.orchestra.run.smithi045.stdout:(55/150): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2023-11-28T20:31:15.309 INFO:teuthology.orchestra.run.smithi045.stdout:(56/150): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2023-11-28T20:31:15.335 INFO:teuthology.orchestra.run.smithi045.stdout:(57/150): openblas-0.3.21-2.el9.x86_64.rpm 612 kB/s | 34 kB 00:00 2023-11-28T20:31:15.485 INFO:teuthology.orchestra.run.smithi045.stdout:(58/150): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2023-11-28T20:31:15.536 INFO:teuthology.orchestra.run.smithi045.stdout:(59/150): postgresql-13.11-1.el9.x86_64.rpm 8.2 MB/s | 1.5 MB 00:00 2023-11-28T20:31:15.584 INFO:teuthology.orchestra.run.smithi190.stdout:(8/150): ceph-common-18.0.0-7520.g43dd94c1.el9. 4.4 MB/s | 19 MB 00:04 2023-11-28T20:31:15.603 INFO:teuthology.orchestra.run.smithi045.stdout:(60/150): postgresql-contrib-13.11-1.el9.x86_64 6.8 MB/s | 811 kB 00:00 2023-11-28T20:31:15.628 INFO:teuthology.orchestra.run.smithi045.stdout:(61/150): postgresql-private-libs-13.11-1.el9.x 1.6 MB/s | 134 kB 00:00 2023-11-28T20:31:15.636 INFO:teuthology.orchestra.run.smithi190.stdout:(9/150): ceph-selinux-18.0.0-7520.g43dd94c1.el9 473 kB/s | 24 kB 00:00 2023-11-28T20:31:15.778 INFO:teuthology.orchestra.run.smithi045.stdout:(62/150): postgresql-server-13.11-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2023-11-28T20:31:15.912 INFO:teuthology.orchestra.run.smithi045.stdout:(63/150): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2023-11-28T20:31:15.937 INFO:teuthology.orchestra.run.smithi045.stdout:(64/150): python3-jinja2-2.11.3-4.el9.noarch.rp 1.6 MB/s | 228 kB 00:00 2023-11-28T20:31:15.971 INFO:teuthology.orchestra.run.smithi045.stdout:(65/150): python3-jsonpatch-1.21-16.el9.noarch. 760 kB/s | 24 kB 00:00 2023-11-28T20:31:15.996 INFO:teuthology.orchestra.run.smithi045.stdout:(66/150): python3-jsonpointer-2.0-4.el9.noarch. 316 kB/s | 18 kB 00:00 2023-11-28T20:31:16.022 INFO:teuthology.orchestra.run.smithi045.stdout:(67/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2023-11-28T20:31:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:(68/150): python3-markupsafe-1.1.1-12.el9.x86_6 431 kB/s | 32 kB 00:00 2023-11-28T20:31:16.205 INFO:teuthology.orchestra.run.smithi045.stdout:(69/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-11-28T20:31:16.231 INFO:teuthology.orchestra.run.smithi045.stdout:(70/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2023-11-28T20:31:16.273 INFO:teuthology.orchestra.run.smithi045.stdout:(71/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2023-11-28T20:31:16.298 INFO:teuthology.orchestra.run.smithi045.stdout:(72/150): python3-prettytable-0.7.2-27.el9.noar 629 kB/s | 41 kB 00:00 2023-11-28T20:31:16.345 INFO:teuthology.orchestra.run.smithi190.stdout:(10/150): ceph-osd-18.0.0-7520.g43dd94c1.el9.x8 8.6 MB/s | 18 MB 00:02 2023-11-28T20:31:16.412 INFO:teuthology.orchestra.run.smithi190.stdout:(11/150): cephfs-mirror-18.0.0-7520.g43dd94c1.e 3.2 MB/s | 217 kB 00:00 2023-11-28T20:31:16.448 INFO:teuthology.orchestra.run.smithi045.stdout:(73/150): ceph-mgr-diskprediction-local-18.0.0- 2.2 MB/s | 7.4 MB 00:03 2023-11-28T20:31:16.471 INFO:teuthology.orchestra.run.smithi190.stdout:(12/150): libcephfs-devel-18.0.0-7520.g43dd94c1 542 kB/s | 31 kB 00:00 2023-11-28T20:31:16.481 INFO:teuthology.orchestra.run.smithi045.stdout:(74/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 725 kB/s | 132 kB 00:00 2023-11-28T20:31:16.535 INFO:teuthology.orchestra.run.smithi045.stdout:(75/150): python3-pyasn1-modules-0.4.8-6.el9.no 894 kB/s | 211 kB 00:00 2023-11-28T20:31:16.557 INFO:teuthology.orchestra.run.smithi045.stdout:(76/150): python3-pytz-2021.1-5.el9.noarch.rpm 440 kB/s | 47 kB 00:00 2023-11-28T20:31:16.579 INFO:teuthology.orchestra.run.smithi190.stdout:(13/150): libcephfs2-18.0.0-7520.g43dd94c1.el9. 6.5 MB/s | 719 kB 00:00 2023-11-28T20:31:16.591 INFO:teuthology.orchestra.run.smithi045.stdout:(77/150): python3-requests-oauthlib-1.3.0-12.el 793 kB/s | 43 kB 00:00 2023-11-28T20:31:16.629 INFO:teuthology.orchestra.run.smithi190.stdout:(14/150): libcephsqlite-18.0.0-7520.g43dd94c1.e 3.3 MB/s | 164 kB 00:00 2023-11-28T20:31:16.680 INFO:teuthology.orchestra.run.smithi190.stdout:(15/150): librados-devel-18.0.0-7520.g43dd94c1. 2.5 MB/s | 126 kB 00:00 2023-11-28T20:31:16.763 INFO:teuthology.orchestra.run.smithi190.stdout:(16/150): libradosstriper1-18.0.0-7520.g43dd94c 5.6 MB/s | 476 kB 00:00 2023-11-28T20:31:16.908 INFO:teuthology.orchestra.run.smithi045.stdout:(78/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2023-11-28T20:31:16.925 INFO:teuthology.orchestra.run.smithi045.stdout:(79/150): python3-toml-0.10.2-6.el9.noarch.rpm 110 kB/s | 39 kB 00:00 2023-11-28T20:31:16.958 INFO:teuthology.orchestra.run.smithi045.stdout:(80/150): socat-1.7.4.1-5.el9.x86_64.rpm 854 kB/s | 300 kB 00:00 2023-11-28T20:31:16.992 INFO:teuthology.orchestra.run.smithi045.stdout:(81/150): uuid-1.6.2-55.el9.x86_64.rpm 1.7 MB/s | 56 kB 00:00 2023-11-28T20:31:17.042 INFO:teuthology.orchestra.run.smithi045.stdout:(82/150): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 1.0 MB/s | 85 kB 00:00 2023-11-28T20:31:17.059 INFO:teuthology.orchestra.run.smithi045.stdout:(83/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 636 kB/s | 63 kB 00:00 2023-11-28T20:31:17.080 INFO:teuthology.orchestra.run.smithi190.stdout:(17/150): ceph-radosgw-18.0.0-7520.g43dd94c1.el 3.3 MB/s | 8.6 MB 00:02 2023-11-28T20:31:17.084 INFO:teuthology.orchestra.run.smithi045.stdout:(84/150): zlib-devel-1.2.11-40.el9.x86_64.rpm 487 kB/s | 44 kB 00:00 2023-11-28T20:31:17.118 INFO:teuthology.orchestra.run.smithi045.stdout:(85/150): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2023-11-28T20:31:17.156 INFO:teuthology.orchestra.run.smithi190.stdout:(18/150): python3-ceph-argparse-18.0.0-7520.g43 608 kB/s | 45 kB 00:00 2023-11-28T20:31:17.160 INFO:teuthology.orchestra.run.smithi045.stdout:(86/150): libgfortran-11.4.1-2.3.el9.x86_64.rpm 7.9 MB/s | 803 kB 00:00 2023-11-28T20:31:17.194 INFO:teuthology.orchestra.run.smithi045.stdout:(87/150): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2023-11-28T20:31:17.231 INFO:teuthology.orchestra.run.smithi190.stdout:(19/150): python3-ceph-common-18.0.0-7520.g43dd 1.7 MB/s | 129 kB 00:00 2023-11-28T20:31:17.236 INFO:teuthology.orchestra.run.smithi045.stdout:(88/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2023-11-28T20:31:17.278 INFO:teuthology.orchestra.run.smithi045.stdout:(89/150): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.1 MB 00:00 2023-11-28T20:31:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:(90/150): python3-ply-3.11-14.el9.noarch.rpm 958 kB/s | 103 kB 00:00 2023-11-28T20:31:17.345 INFO:teuthology.orchestra.run.smithi045.stdout:(91/150): python3-pycparser-2.20-6.el9.noarch.r 3.0 MB/s | 124 kB 00:00 2023-11-28T20:31:17.370 INFO:teuthology.orchestra.run.smithi045.stdout:(92/150): python3-requests-2.25.1-7.el9.noarch. 1.7 MB/s | 113 kB 00:00 2023-11-28T20:31:17.389 INFO:teuthology.orchestra.run.smithi190.stdout:(20/150): librgw2-18.0.0-7520.g43dd94c1.el9.x86 7.6 MB/s | 4.8 MB 00:00 2023-11-28T20:31:17.395 INFO:teuthology.orchestra.run.smithi045.stdout:(93/150): python3-urllib3-1.26.5-4.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2023-11-28T20:31:17.423 INFO:teuthology.orchestra.run.smithi190.stdout:(21/150): python3-cephfs-18.0.0-7520.g43dd94c1. 847 kB/s | 162 kB 00:00 2023-11-28T20:31:17.475 INFO:teuthology.orchestra.run.smithi190.stdout:(22/150): python3-rados-18.0.0-7520.g43dd94c1.e 3.7 MB/s | 320 kB 00:00 2023-11-28T20:31:17.526 INFO:teuthology.orchestra.run.smithi190.stdout:(23/150): python3-rgw-18.0.0-7520.g43dd94c1.el9 2.0 MB/s | 99 kB 00:00 2023-11-28T20:31:17.560 INFO:teuthology.orchestra.run.smithi190.stdout:(24/150): python3-rbd-18.0.0-7520.g43dd94c1.el9 2.2 MB/s | 299 kB 00:00 2023-11-28T20:31:17.570 INFO:teuthology.orchestra.run.smithi045.stdout:(94/150): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2023-11-28T20:31:17.585 INFO:teuthology.orchestra.run.smithi190.stdout:(25/150): rbd-fuse-18.0.0-7520.g43dd94c1.el9.x8 1.4 MB/s | 85 kB 00:00 2023-11-28T20:31:17.635 INFO:teuthology.orchestra.run.smithi190.stdout:(26/150): rbd-nbd-18.0.0-7520.g43dd94c1.el9.x86 3.4 MB/s | 170 kB 00:00 2023-11-28T20:31:17.685 INFO:teuthology.orchestra.run.smithi190.stdout:(27/150): ceph-grafana-dashboards-18.0.0-7520.g 566 kB/s | 28 kB 00:00 2023-11-28T20:31:17.721 INFO:teuthology.orchestra.run.smithi045.stdout:(95/150): lua-devel-5.4.4-4.el9.x86_64.rpm 69 kB/s | 22 kB 00:00 2023-11-28T20:31:17.744 INFO:teuthology.orchestra.run.smithi190.stdout:(28/150): ceph-mgr-cephadm-18.0.0-7520.g43dd94c 2.3 MB/s | 133 kB 00:00 2023-11-28T20:31:17.821 INFO:teuthology.orchestra.run.smithi045.stdout:(96/150): openssl-devel-3.0.7-25.el9.x86_64.rpm 8.7 MB/s | 4.1 MB 00:00 2023-11-28T20:31:17.953 INFO:teuthology.orchestra.run.smithi190.stdout:(29/150): ceph-mgr-dashboard-18.0.0-7520.g43dd9 8.5 MB/s | 1.8 MB 00:00 2023-11-28T20:31:18.072 INFO:teuthology.orchestra.run.smithi045.stdout:(97/150): gperftools-libs-2.9.1-2.el9.x86_64.rp 618 kB/s | 309 kB 00:00 2023-11-28T20:31:18.155 INFO:teuthology.orchestra.run.smithi045.stdout:(98/150): liboath-2.6.7-2.el9.x86_64.rpm 617 kB/s | 49 kB 00:00 2023-11-28T20:31:18.205 INFO:teuthology.orchestra.run.smithi045.stdout:(99/150): libarrow-doc-9.0.0-9.el9.noarch.rpm 77 kB/s | 24 kB 00:00 2023-11-28T20:31:18.239 INFO:teuthology.orchestra.run.smithi045.stdout:(100/150): libunwind-1.6.2-1.el9.x86_64.rpm 813 kB/s | 67 kB 00:00 2023-11-28T20:31:18.398 INFO:teuthology.orchestra.run.smithi045.stdout:(101/150): luarocks-3.9.2-1.el9.noarch.rpm 791 kB/s | 151 kB 00:00 2023-11-28T20:31:18.420 INFO:teuthology.orchestra.run.smithi190.stdout:(30/150): rbd-mirror-18.0.0-7520.g43dd94c1.el9. 3.5 MB/s | 3.0 MB 00:00 2023-11-28T20:31:18.466 INFO:teuthology.orchestra.run.smithi045.stdout:(102/150): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.6 MB/s | 837 kB 00:00 2023-11-28T20:31:18.547 INFO:teuthology.orchestra.run.smithi190.stdout:(31/150): ceph-mgr-modules-core-18.0.0-7520.g43 1.9 MB/s | 243 kB 00:00 2023-11-28T20:31:18.551 INFO:teuthology.orchestra.run.smithi045.stdout:(103/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 535 kB/s | 43 kB 00:00 2023-11-28T20:31:18.608 INFO:teuthology.orchestra.run.smithi045.stdout:(104/150): python3-asyncssh-2.13.2-3.el9.noarch 2.6 MB/s | 548 kB 00:00 2023-11-28T20:31:18.628 INFO:teuthology.orchestra.run.smithi190.stdout:(32/150): ceph-mgr-rook-18.0.0-7520.g43dd94c1.e 610 kB/s | 50 kB 00:00 2023-11-28T20:31:18.634 INFO:teuthology.orchestra.run.smithi045.stdout:(105/150): python3-cachetools-4.2.4-1.el9.noarc 389 kB/s | 32 kB 00:00 2023-11-28T20:31:18.733 INFO:teuthology.orchestra.run.smithi045.stdout:(106/150): libarrow-9.0.0-9.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2023-11-28T20:31:18.758 INFO:teuthology.orchestra.run.smithi045.stdout:(107/150): python3-certifi-2021.10.8-2.el9.noar 100 kB/s | 15 kB 00:00 2023-11-28T20:31:18.767 INFO:teuthology.orchestra.run.smithi190.stdout:(33/150): ceph-mgr-diskprediction-local-18.0.0- 9.1 MB/s | 7.4 MB 00:00 2023-11-28T20:31:18.787 INFO:teuthology.orchestra.run.smithi190.stdout:(34/150): ceph-prometheus-alerts-18.0.0-7520.g4 96 kB/s | 15 kB 00:00 2023-11-28T20:31:18.792 INFO:teuthology.orchestra.run.smithi045.stdout:(108/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2023-11-28T20:31:18.821 INFO:teuthology.orchestra.run.smithi190.stdout:(35/150): ceph-volume-18.0.0-7520.g43dd94c1.el9 4.5 MB/s | 262 kB 00:00 2023-11-28T20:31:18.826 INFO:teuthology.orchestra.run.smithi045.stdout:(109/150): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2023-11-28T20:31:18.859 INFO:teuthology.orchestra.run.smithi045.stdout:(110/150): python3-google-auth-2.23.4-1.el9.noa 1.9 MB/s | 197 kB 00:00 2023-11-28T20:31:18.871 INFO:teuthology.orchestra.run.smithi190.stdout:(36/150): cephfs-top-18.0.0-7520.g43dd94c1.el9. 370 kB/s | 18 kB 00:00 2023-11-28T20:31:18.876 INFO:teuthology.orchestra.run.smithi045.stdout:(111/150): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2023-11-28T20:31:18.893 INFO:teuthology.orchestra.run.smithi045.stdout:(112/150): python3-jaraco-classes-3.2.1-5.el9.n 265 kB/s | 18 kB 00:00 2023-11-28T20:31:18.960 INFO:teuthology.orchestra.run.smithi045.stdout:(113/150): python3-jaraco-collections-3.0.0-8.e 232 kB/s | 23 kB 00:00 2023-11-28T20:31:18.985 INFO:teuthology.orchestra.run.smithi045.stdout:(114/150): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2023-11-28T20:31:19.002 INFO:teuthology.orchestra.run.smithi045.stdout:(115/150): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2023-11-28T20:31:19.028 INFO:teuthology.orchestra.run.smithi045.stdout:(116/150): python3-jwt+crypto-2.4.0-1.el9.noarc 136 kB/s | 9.0 kB 00:00 2023-11-28T20:31:19.070 INFO:teuthology.orchestra.run.smithi045.stdout:(117/150): python3-jwt-2.4.0-1.el9.noarch.rpm 619 kB/s | 41 kB 00:00 2023-11-28T20:31:19.072 INFO:teuthology.orchestra.run.smithi190.stdout:(37/150): cephadm-18.0.0-7520.g43dd94c1.el9.noa 2.7 MB/s | 774 kB 00:00 2023-11-28T20:31:19.095 INFO:teuthology.orchestra.run.smithi045.stdout:(118/150): python3-logutils-0.3.5-21.el9.noarch 696 kB/s | 46 kB 00:00 2023-11-28T20:31:19.143 INFO:teuthology.orchestra.run.smithi190.stdout:(38/150): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2023-11-28T20:31:19.145 INFO:teuthology.orchestra.run.smithi045.stdout:(119/150): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2023-11-28T20:31:19.171 INFO:teuthology.orchestra.run.smithi045.stdout:(120/150): python3-natsort-7.1.1-5.el9.noarch.r 774 kB/s | 58 kB 00:00 2023-11-28T20:31:19.197 INFO:teuthology.orchestra.run.smithi190.stdout:(39/150): bison-3.7.4-5.el9.x86_64.rpm 7.5 MB/s | 920 kB 00:00 2023-11-28T20:31:19.204 INFO:teuthology.orchestra.run.smithi045.stdout:(121/150): python3-more-itertools-8.12.0-2.el9. 590 kB/s | 79 kB 00:00 2023-11-28T20:31:19.223 INFO:teuthology.orchestra.run.smithi190.stdout:(40/150): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2023-11-28T20:31:19.239 INFO:teuthology.orchestra.run.smithi045.stdout:(122/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2023-11-28T20:31:19.248 INFO:teuthology.orchestra.run.smithi190.stdout:(41/150): elfutils-libelf-devel-0.190-1.el9.x86 443 kB/s | 21 kB 00:00 2023-11-28T20:31:19.264 INFO:teuthology.orchestra.run.smithi045.stdout:(123/150): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2023-11-28T20:31:19.281 INFO:teuthology.orchestra.run.smithi190.stdout:(42/150): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2023-11-28T20:31:19.288 INFO:teuthology.orchestra.run.smithi045.stdout:(124/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2023-11-28T20:31:19.307 INFO:teuthology.orchestra.run.smithi190.stdout:(43/150): flexiblas-3.0.4-8.el9.x86_64.rpm 566 kB/s | 32 kB 00:00 2023-11-28T20:31:19.314 INFO:teuthology.orchestra.run.smithi045.stdout:(125/150): python3-repoze-lru-0.7-16.el9.noarch 411 kB/s | 31 kB 00:00 2023-11-28T20:31:19.347 INFO:teuthology.orchestra.run.smithi045.stdout:(126/150): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2023-11-28T20:31:19.373 INFO:teuthology.orchestra.run.smithi045.stdout:(127/150): python3-rsa-4.9-2.el9.noarch.rpm 711 kB/s | 59 kB 00:00 2023-11-28T20:31:19.398 INFO:teuthology.orchestra.run.smithi045.stdout:(128/150): python3-tempora-5.0.0-2.el9.noarch.r 430 kB/s | 36 kB 00:00 2023-11-28T20:31:19.449 INFO:teuthology.orchestra.run.smithi045.stdout:(129/150): python3-typing-extensions-4.4.0-2.el 511 kB/s | 51 kB 00:00 2023-11-28T20:31:19.457 INFO:teuthology.orchestra.run.smithi190.stdout:(44/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2023-11-28T20:31:19.473 INFO:teuthology.orchestra.run.smithi045.stdout:(130/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2023-11-28T20:31:19.482 INFO:teuthology.orchestra.run.smithi190.stdout:(45/150): flexiblas-openblas-openmp-3.0.4-8.el9 99 kB/s | 17 kB 00:00 2023-11-28T20:31:19.507 INFO:teuthology.orchestra.run.smithi045.stdout:(131/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2023-11-28T20:31:19.516 INFO:teuthology.orchestra.run.smithi190.stdout:(46/150): libaio-devel-0.3.111-13.el9.x86_64.rp 315 kB/s | 10 kB 00:00 2023-11-28T20:31:19.549 INFO:teuthology.orchestra.run.smithi045.stdout:(132/150): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2023-11-28T20:31:19.550 INFO:teuthology.orchestra.run.smithi190.stdout:(47/150): jq-1.6-16.el9.x86_64.rpm 2.8 MB/s | 186 kB 00:00 2023-11-28T20:31:19.574 INFO:teuthology.orchestra.run.smithi045.stdout:(133/150): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2023-11-28T20:31:19.575 INFO:teuthology.orchestra.run.smithi190.stdout:(48/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.8 MB/s | 159 kB 00:00 2023-11-28T20:31:19.608 INFO:teuthology.orchestra.run.smithi045.stdout:(134/150): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2023-11-28T20:31:19.608 INFO:teuthology.orchestra.run.smithi190.stdout:(49/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 780 kB/s | 44 kB 00:00 2023-11-28T20:31:19.651 INFO:teuthology.orchestra.run.smithi190.stdout:(50/150): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 661 kB 00:00 2023-11-28T20:31:19.701 INFO:teuthology.orchestra.run.smithi190.stdout:(51/150): libtool-2.4.6-45.el9.x86_64.rpm 6.6 MB/s | 578 kB 00:00 2023-11-28T20:31:19.708 INFO:teuthology.orchestra.run.smithi045.stdout:(135/150): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2023-11-28T20:31:19.726 INFO:teuthology.orchestra.run.smithi190.stdout:(52/150): libuuid-devel-2.37.4-15.el9.x86_64.rp 352 kB/s | 24 kB 00:00 2023-11-28T20:31:19.742 INFO:teuthology.orchestra.run.smithi045.stdout:(136/150): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2023-11-28T20:31:19.751 INFO:teuthology.orchestra.run.smithi190.stdout:(53/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 46 kB 00:00 2023-11-28T20:31:19.785 INFO:teuthology.orchestra.run.smithi190.stdout:(54/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-11-28T20:31:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:(137/150): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 69 kB 00:00 2023-11-28T20:31:19.819 INFO:teuthology.orchestra.run.smithi190.stdout:(55/150): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2023-11-28T20:31:19.826 INFO:teuthology.orchestra.run.smithi045.stdout:(138/150): elfutils-debuginfod-client-0.190-1.e 1.1 MB/s | 36 kB 00:00 2023-11-28T20:31:19.852 INFO:teuthology.orchestra.run.smithi190.stdout:(56/150): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2023-11-28T20:31:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:(139/150): elfutils-libelf-0.190-1.el9.x86_64.r 4.6 MB/s | 192 kB 00:00 2023-11-28T20:31:19.869 INFO:teuthology.orchestra.run.smithi190.stdout:(57/150): openblas-0.3.21-2.el9.x86_64.rpm 718 kB/s | 34 kB 00:00 2023-11-28T20:31:19.935 INFO:teuthology.orchestra.run.smithi045.stdout:(140/150): elfutils-libs-0.190-1.el9.x86_64.rpm 3.8 MB/s | 255 kB 00:00 2023-11-28T20:31:20.011 INFO:teuthology.orchestra.run.smithi045.stdout:(141/150): librbd1-18.0.0-7520.g43dd94c1.el9.x8 7.5 MB/s | 3.0 MB 00:00 2023-11-28T20:31:20.029 INFO:teuthology.orchestra.run.smithi190.stdout:(58/150): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2023-11-28T20:31:20.036 INFO:teuthology.orchestra.run.smithi045.stdout:(142/150): libacl-2.3.1-4.el9.x86_64.rpm 220 kB/s | 22 kB 00:00 2023-11-28T20:31:20.078 INFO:teuthology.orchestra.run.smithi045.stdout:(143/150): xfsdump-3.1.12-4.el9.x86_64.rpm 4.8 MB/s | 321 kB 00:00 2023-11-28T20:31:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:20.095 INFO:teuthology.orchestra.run.smithi190.stdout:(59/150): postgresql-13.11-1.el9.x86_64.rpm 7.1 MB/s | 1.5 MB 00:00 2023-11-28T20:31:20.145 INFO:teuthology.orchestra.run.smithi190.stdout:(60/150): postgresql-contrib-13.11-1.el9.x86_64 7.7 MB/s | 811 kB 00:00 2023-11-28T20:31:20.161 INFO:teuthology.orchestra.run.smithi045.stdout:(144/150): librados2-18.0.0-7520.g43dd94c1.el9. 5.7 MB/s | 3.3 MB 00:00 2023-11-28T20:31:20.162 INFO:teuthology.orchestra.run.smithi190.stdout:(61/150): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2023-11-28T20:31:20.230 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:20.250 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:20.353 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:20.514 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:20.536 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:20.538 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:20.629 INFO:teuthology.orchestra.run.smithi190.stdout:(62/150): ceph-test-18.0.0-7520.g43dd94c1.el9.x 8.3 MB/s | 41 MB 00:04 2023-11-28T20:31:20.669 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:20.755 INFO:teuthology.orchestra.run.smithi190.stdout:(63/150): postgresql-server-13.11-1.el9.x86_64. 9.4 MB/s | 5.7 MB 00:00 2023-11-28T20:31:20.821 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:20.827 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:20.905 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:20.906 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:20.924 INFO:teuthology.orchestra.run.smithi190.stdout:(64/150): python3-babel-2.9.1-2.el9.noarch.rpm 7.7 MB/s | 5.8 MB 00:00 2023-11-28T20:31:20.963 INFO:teuthology.orchestra.run.smithi190.stdout:(65/150): python3-jinja2-2.11.3-4.el9.noarch.rp 734 kB/s | 228 kB 00:00 2023-11-28T20:31:20.989 INFO:teuthology.orchestra.run.smithi190.stdout:(66/150): python3-jsonpatch-1.21-16.el9.noarch. 111 kB/s | 24 kB 00:00 2023-11-28T20:31:21.012 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:21.014 INFO:teuthology.orchestra.run.smithi190.stdout:(67/150): python3-jsonpointer-2.0-4.el9.noarch. 208 kB/s | 18 kB 00:00 2023-11-28T20:31:21.014 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:21.039 INFO:teuthology.orchestra.run.smithi190.stdout:(68/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2023-11-28T20:31:21.064 INFO:teuthology.orchestra.run.smithi190.stdout:(69/150): python3-markupsafe-1.1.1-12.el9.x86_6 427 kB/s | 32 kB 00:00 2023-11-28T20:31:21.080 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:21.210 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:21.225 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:21.248 INFO:teuthology.orchestra.run.smithi190.stdout:(70/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2023-11-28T20:31:21.259 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:21.267 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:21.270 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:21.282 INFO:teuthology.orchestra.run.smithi190.stdout:(71/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2023-11-28T20:31:21.307 INFO:teuthology.orchestra.run.smithi190.stdout:(72/150): python3-oauthlib-3.1.1-5.el9.noarch.r 767 kB/s | 174 kB 00:00 2023-11-28T20:31:21.332 INFO:teuthology.orchestra.run.smithi190.stdout:(73/150): python3-prettytable-0.7.2-27.el9.noar 515 kB/s | 41 kB 00:00 2023-11-28T20:31:21.366 INFO:teuthology.orchestra.run.smithi190.stdout:(74/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2023-11-28T20:31:21.391 INFO:teuthology.orchestra.run.smithi190.stdout:(75/150): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2023-11-28T20:31:21.398 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:21.416 INFO:teuthology.orchestra.run.smithi190.stdout:(76/150): python3-pytz-2021.1-5.el9.noarch.rpm 571 kB/s | 47 kB 00:00 2023-11-28T20:31:21.425 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:21.425 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:21.450 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:21.458 INFO:teuthology.orchestra.run.smithi190.stdout:(77/150): python3-requests-oauthlib-1.3.0-12.el 654 kB/s | 43 kB 00:00 2023-11-28T20:31:21.594 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:21.598 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:21.609 INFO:teuthology.orchestra.run.smithi190.stdout:(78/150): python3-toml-0.10.2-6.el9.noarch.rpm 202 kB/s | 39 kB 00:00 2023-11-28T20:31:21.621 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:21.842 INFO:teuthology.orchestra.run.smithi190.stdout:(79/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2023-11-28T20:31:21.867 INFO:teuthology.orchestra.run.smithi190.stdout:(80/150): socat-1.7.4.1-5.el9.x86_64.rpm 760 kB/s | 300 kB 00:00 2023-11-28T20:31:21.893 INFO:teuthology.orchestra.run.smithi190.stdout:(81/150): uuid-1.6.2-55.el9.x86_64.rpm 209 kB/s | 56 kB 00:00 2023-11-28T20:31:21.935 INFO:teuthology.orchestra.run.smithi190.stdout:(82/150): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 2.1 MB/s | 85 kB 00:00 2023-11-28T20:31:21.960 INFO:teuthology.orchestra.run.smithi190.stdout:(83/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 960 kB/s | 63 kB 00:00 2023-11-28T20:31:21.985 INFO:teuthology.orchestra.run.smithi190.stdout:(84/150): zlib-devel-1.2.11-40.el9.x86_64.rpm 488 kB/s | 44 kB 00:00 2023-11-28T20:31:22.014 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:22.027 INFO:teuthology.orchestra.run.smithi190.stdout:(85/150): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2023-11-28T20:31:22.052 INFO:teuthology.orchestra.run.smithi190.stdout:(86/150): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2023-11-28T20:31:22.053 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:22.094 INFO:teuthology.orchestra.run.smithi190.stdout:(87/150): libgfortran-11.4.1-2.3.el9.x86_64.rpm 7.3 MB/s | 803 kB 00:00 2023-11-28T20:31:22.108 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:22.119 INFO:teuthology.orchestra.run.smithi190.stdout:(88/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2023-11-28T20:31:22.169 INFO:teuthology.orchestra.run.smithi190.stdout:(89/150): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2023-11-28T20:31:22.195 INFO:teuthology.orchestra.run.smithi190.stdout:(90/150): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2023-11-28T20:31:22.220 INFO:teuthology.orchestra.run.smithi190.stdout:(91/150): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2023-11-28T20:31:22.245 INFO:teuthology.orchestra.run.smithi190.stdout:(92/150): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2023-11-28T20:31:22.279 INFO:teuthology.orchestra.run.smithi190.stdout:(93/150): python3-urllib3-1.26.5-4.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2023-11-28T20:31:22.554 INFO:teuthology.orchestra.run.smithi190.stdout:(94/150): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2023-11-28T20:31:22.612 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:22.616 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:22.824 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:22.846 INFO:teuthology.orchestra.run.smithi190.stdout:(95/150): python3-devel-3.9.17-2.el9.x86_64.rpm 410 kB/s | 245 kB 00:00 2023-11-28T20:31:22.963 INFO:teuthology.orchestra.run.smithi190.stdout:(96/150): gperftools-libs-2.9.1-2.el9.x86_64.rp 757 kB/s | 309 kB 00:00 2023-11-28T20:31:23.047 INFO:teuthology.orchestra.run.smithi190.stdout:(97/150): libarrow-doc-9.0.0-9.el9.noarch.rpm 295 kB/s | 24 kB 00:00 2023-11-28T20:31:23.051 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:23.122 INFO:teuthology.orchestra.run.smithi190.stdout:(98/150): liboath-2.6.7-2.el9.x86_64.rpm 654 kB/s | 49 kB 00:00 2023-11-28T20:31:23.127 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:23.128 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:23.206 INFO:teuthology.orchestra.run.smithi190.stdout:(99/150): libunwind-1.6.2-1.el9.x86_64.rpm 815 kB/s | 67 kB 00:00 2023-11-28T20:31:23.292 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:23.330 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:23.330 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:23.331 INFO:teuthology.orchestra.run.smithi190.stdout:(100/150): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2023-11-28T20:31:23.368 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:23.368 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:23.481 INFO:teuthology.orchestra.run.smithi190.stdout:(101/150): libarrow-9.0.0-9.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2023-11-28T20:31:23.623 INFO:teuthology.orchestra.run.smithi190.stdout:(102/150): python3-asyncssh-2.13.2-3.el9.noarch 3.8 MB/s | 548 kB 00:00 2023-11-28T20:31:23.657 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:23.688 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:23.731 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:23.757 INFO:teuthology.orchestra.run.smithi190.stdout:(103/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 327 kB/s | 43 kB 00:00 2023-11-28T20:31:23.812 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:23.816 INFO:teuthology.orchestra.run.smithi190.stdout:(104/150): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.7 MB/s | 837 kB 00:00 2023-11-28T20:31:23.841 INFO:teuthology.orchestra.run.smithi190.stdout:(105/150): python3-cachetools-4.2.4-1.el9.noarc 387 kB/s | 32 kB 00:00 2023-11-28T20:31:23.845 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:23.852 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 199.223.246.113) 2023-11-28T20:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 199.223.246.113) 2023-11-28T20:31:23.885 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:23.891 INFO:teuthology.orchestra.run.smithi190.stdout:(106/150): python3-certifi-2021.10.8-2.el9.noar 201 kB/s | 15 kB 00:00 2023-11-28T20:31:23.905 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 199.223.246.113) 2023-11-28T20:31:23.928 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:23.933 INFO:teuthology.orchestra.run.smithi190.stdout:(107/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2023-11-28T20:31:23.958 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:24.042 INFO:teuthology.orchestra.run.smithi190.stdout:(108/150): python3-google-auth-2.23.4-1.el9.noa 1.8 MB/s | 197 kB 00:00 2023-11-28T20:31:24.075 INFO:teuthology.orchestra.run.smithi190.stdout:(109/150): python3-cherrypy-18.6.1-2.el9.noarch 1.9 MB/s | 358 kB 00:00 2023-11-28T20:31:24.092 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:24.126 INFO:teuthology.orchestra.run.smithi190.stdout:(110/150): python3-jaraco-8.2.1-3.el9.noarch.rp 129 kB/s | 11 kB 00:00 2023-11-28T20:31:24.168 INFO:teuthology.orchestra.run.smithi190.stdout:(111/150): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2023-11-28T20:31:24.210 INFO:teuthology.orchestra.run.smithi190.stdout:(112/150): python3-jaraco-collections-3.0.0-8.e 279 kB/s | 23 kB 00:00 2023-11-28T20:31:24.220 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:24.221 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:24.268 INFO:teuthology.orchestra.run.smithi190.stdout:(113/150): python3-jaraco-functools-3.5.0-2.el9 195 kB/s | 19 kB 00:00 2023-11-28T20:31:24.286 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:24.286 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:24.293 INFO:teuthology.orchestra.run.smithi190.stdout:(114/150): python3-jaraco-text-3.2.0-6.el9.noar 237 kB/s | 20 kB 00:00 2023-11-28T20:31:24.296 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:24.352 INFO:teuthology.orchestra.run.smithi190.stdout:(115/150): python3-jwt+crypto-2.4.0-1.el9.noarc 109 kB/s | 9.0 kB 00:00 2023-11-28T20:31:24.427 INFO:teuthology.orchestra.run.smithi190.stdout:(116/150): python3-jwt-2.4.0-1.el9.noarch.rpm 307 kB/s | 41 kB 00:00 2023-11-28T20:31:24.438 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:24.470 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:24.471 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:24.494 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:24.494 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:24.498 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:24.544 INFO:teuthology.orchestra.run.smithi190.stdout:(117/150): python3-logutils-0.3.5-21.el9.noarch 399 kB/s | 46 kB 00:00 2023-11-28T20:31:24.628 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:24.632 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:24.639 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:24.645 INFO:teuthology.orchestra.run.smithi190.stdout:(118/150): python3-more-itertools-8.12.0-2.el9. 794 kB/s | 79 kB 00:00 2023-11-28T20:31:24.705 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:24.729 INFO:teuthology.orchestra.run.smithi190.stdout:(119/150): python3-kubernetes-26.1.0-2.el9.noar 2.7 MB/s | 1.0 MB 00:00 2023-11-28T20:31:24.754 INFO:teuthology.orchestra.run.smithi190.stdout:(120/150): python3-natsort-7.1.1-5.el9.noarch.r 534 kB/s | 58 kB 00:00 2023-11-28T20:31:24.769 INFO:teuthology.orchestra.run.smithi045.stdout:(145/150): NetworkManager-team-1.45.7-1.el9.x86 9.5 kB/s | 43 kB 00:04 2023-11-28T20:31:24.829 INFO:teuthology.orchestra.run.smithi190.stdout:(121/150): python3-portend-3.1.0-2.el9.noarch.r 221 kB/s | 16 kB 00:00 2023-11-28T20:31:24.871 INFO:teuthology.orchestra.run.smithi190.stdout:(122/150): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2023-11-28T20:31:24.913 INFO:teuthology.orchestra.run.smithi190.stdout:(123/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2023-11-28T20:31:24.945 INFO:teuthology.orchestra.run.smithi045.stdout:(146/150): NetworkManager-1.45.7-1.el9.x86_64.r 471 kB/s | 2.3 MB 00:04 2023-11-28T20:31:24.955 INFO:teuthology.orchestra.run.smithi190.stdout:(124/150): python3-repoze-lru-0.7-16.el9.noarch 370 kB/s | 31 kB 00:00 2023-11-28T20:31:25.005 INFO:teuthology.orchestra.run.smithi190.stdout:(125/150): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2023-11-28T20:31:25.012 INFO:teuthology.orchestra.run.smithi045.stdout:(147/150): NetworkManager-libnm-1.45.7-1.el9.x8 369 kB/s | 1.8 MB 00:04 2023-11-28T20:31:25.014 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:25.030 INFO:teuthology.orchestra.run.smithi190.stdout:(126/150): python3-rsa-4.9-2.el9.noarch.rpm 791 kB/s | 59 kB 00:00 2023-11-28T20:31:25.062 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:25.081 INFO:teuthology.orchestra.run.smithi190.stdout:(127/150): python3-tempora-5.0.0-2.el9.noarch.r 480 kB/s | 36 kB 00:00 2023-11-28T20:31:25.106 INFO:teuthology.orchestra.run.smithi190.stdout:(128/150): python3-typing-extensions-4.4.0-2.el 683 kB/s | 51 kB 00:00 2023-11-28T20:31:25.179 INFO:teuthology.orchestra.run.smithi045.stdout:(148/150): openssl-3.0.7-25.el9.x86_64.rpm 5.3 MB/s | 1.2 MB 00:00 2023-11-28T20:31:25.181 INFO:teuthology.orchestra.run.smithi190.stdout:(129/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.3 MB/s | 230 kB 00:00 2023-11-28T20:31:25.206 INFO:teuthology.orchestra.run.smithi190.stdout:(130/150): python3-websocket-client-1.2.3-2.el9 900 kB/s | 90 kB 00:00 2023-11-28T20:31:25.307 INFO:teuthology.orchestra.run.smithi190.stdout:(131/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2023-11-28T20:31:25.312 INFO:teuthology.orchestra.run.smithi045.stdout:(149/150): openssl-libs-3.0.7-25.el9.x86_64.rpm 7.2 MB/s | 2.2 MB 00:00 2023-11-28T20:31:25.332 INFO:teuthology.orchestra.run.smithi190.stdout:(132/150): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2023-11-28T20:31:25.413 INFO:teuthology.orchestra.run.smithi190.stdout:(133/150): re2-20211101-3.el9.x86_64.rpm 1.8 MB/s | 192 kB 00:00 2023-11-28T20:31:25.419 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:25.577 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:25.730 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:25.741 INFO:teuthology.orchestra.run.smithi190.stdout:(134/150): thrift-0.15.0-2.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2023-11-28T20:31:25.765 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:25.825 INFO:teuthology.orchestra.run.smithi190.stdout:(135/150): librados2-18.0.0-7520.g43dd94c1.el9. 8.1 MB/s | 3.3 MB 00:00 2023-11-28T20:31:25.867 INFO:teuthology.orchestra.run.smithi190.stdout:(136/150): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2023-11-28T20:31:25.901 INFO:teuthology.orchestra.run.smithi190.stdout:(137/150): acl-2.3.1-4.el9.x86_64.rpm 2.1 MB/s | 69 kB 00:00 2023-11-28T20:31:25.909 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:25.943 INFO:teuthology.orchestra.run.smithi190.stdout:(138/150): elfutils-debuginfod-client-0.190-1.e 893 kB/s | 36 kB 00:00 2023-11-28T20:31:25.959 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:25.993 INFO:teuthology.orchestra.run.smithi190.stdout:(139/150): elfutils-libelf-0.190-1.el9.x86_64.r 3.8 MB/s | 192 kB 00:00 2023-11-28T20:31:26.035 INFO:teuthology.orchestra.run.smithi190.stdout:(140/150): elfutils-libs-0.190-1.el9.x86_64.rpm 6.1 MB/s | 255 kB 00:00 2023-11-28T20:31:26.069 INFO:teuthology.orchestra.run.smithi190.stdout:(141/150): libacl-2.3.1-4.el9.x86_64.rpm 673 kB/s | 22 kB 00:00 2023-11-28T20:31:26.169 INFO:teuthology.orchestra.run.smithi190.stdout:(142/150): openssl-devel-3.0.7-25.el9.x86_64.rp 1.0 MB/s | 4.1 MB 00:03 2023-11-28T20:31:26.203 INFO:teuthology.orchestra.run.smithi190.stdout:(143/150): xfsdump-3.1.12-4.el9.x86_64.rpm 2.4 MB/s | 321 kB 00:00 2023-11-28T20:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:26.526 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:26.670 INFO:teuthology.orchestra.run.smithi190.stdout:(144/150): librbd1-18.0.0-7520.g43dd94c1.el9.x8 3.3 MB/s | 3.0 MB 00:00 2023-11-28T20:31:26.750 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:26.901 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:26.911 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:27.299 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:27.408 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:27.518 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:27.552 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:27.787 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:27.851 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:27.869 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:27.893 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:27.896 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:27.933 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:28.026 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:28.042 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:28.043 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:28.203 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:28.203 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:28.204 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:28.266 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:28.266 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:28.285 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:28.339 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:28.411 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:28.414 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:28.445 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:28.445 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:28.449 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 199.223.246.113) 2023-11-28T20:31:28.475 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:28.484 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 199.223.246.113) 2023-11-28T20:31:28.485 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 199.223.246.113) 2023-11-28T20:31:28.508 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:28.514 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 199.223.246.113) 2023-11-28T20:31:28.554 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:28.559 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:28.580 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:28.735 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:28.757 INFO:teuthology.orchestra.run.smithi045.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:28.787 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:28.788 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:28.820 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:28.920 INFO:teuthology.orchestra.run.smithi045.stdout:(150/150): NetworkManager-tui-1.45.7-1.el9.x86_ 59 kB/s | 246 kB 00:04 2023-11-28T20:31:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-11-28T20:31:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:Total 9.6 MB/s | 205 MB 00:21 2023-11-28T20:31:28.948 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:28.952 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:28.981 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:29.271 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:29.308 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:29.309 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:29.328 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:29.417 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-11-28T20:31:29.445 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:29.466 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:29.481 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:29.537 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:29.593 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:29.642 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:29.697 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:29.716 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-11-28T20:31:29.716 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-11-28T20:31:29.844 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:30.077 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:30.161 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:30.168 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 104.167.241.199) 2023-11-28T20:31:30.281 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 104.167.241.199) 2023-11-28T20:31:30.282 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:30.386 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 104.167.241.199) 2023-11-28T20:31:30.430 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 104.167.241.199) 2023-11-28T20:31:30.430 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 104.167.241.199) 2023-11-28T20:31:30.478 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:30.478 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:30.480 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 104.167.241.199) 2023-11-28T20:31:30.615 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:30.637 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:30.638 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:30.657 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:30.683 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:30.687 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:30.704 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:30.878 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:30.882 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:30.903 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:30.904 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:30.908 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:31.014 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:31.017 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:31.027 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:31.088 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:31.217 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.hoobly.com/centos/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 85.25.43.203) 2023-11-28T20:31:31.258 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.hoobly.com/centos/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 85.25.43.203) 2023-11-28T20:31:31.288 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.hoobly.com/centos/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 85.25.43.203) 2023-11-28T20:31:31.580 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.hoobly.com/centos/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 85.25.43.203) 2023-11-28T20:31:31.637 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.hoobly.com/centos/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 85.25.43.203) 2023-11-28T20:31:31.678 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.hoobly.com/centos/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 85.25.43.203) 2023-11-28T20:31:31.746 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 163.114.132.32) 2023-11-28T20:31:31.800 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 163.114.132.32) 2023-11-28T20:31:31.948 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 163.114.132.32) 2023-11-28T20:31:32.014 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-11-28T20:31:32.015 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-11-28T20:31:32.191 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.45.7-1.el9.x86_64.rpm (IP: 163.114.132.32) 2023-11-28T20:31:32.202 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-team-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.45.7-1.el9.x86_64.rpm (IP: 163.114.132.32) 2023-11-28T20:31:32.294 INFO:teuthology.orchestra.run.smithi190.stdout:(145/150): NetworkManager-team-1.45.7-1.el9.x86 7.7 kB/s | 43 kB 00:05 2023-11-28T20:31:32.345 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.45.7-1.el9.x86_64.rpm (IP: 163.114.132.32) 2023-11-28T20:31:32.478 INFO:teuthology.orchestra.run.smithi190.stdout:(146/150): NetworkManager-1.45.7-1.el9.x86_64.r 367 kB/s | 2.3 MB 00:06 2023-11-28T20:31:32.670 INFO:teuthology.orchestra.run.smithi190.stdout:(147/150): NetworkManager-libnm-1.45.7-1.el9.x8 281 kB/s | 1.8 MB 00:06 2023-11-28T20:31:32.700 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:33.274 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 104.251.122.5) 2023-11-28T20:31:33.496 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:33.570 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 64.50.236.52) 2023-11-28T20:31:33.778 INFO:teuthology.orchestra.run.smithi190.stdout:(148/150): openssl-libs-3.0.7-25.el9.x86_64.rpm 1.9 MB/s | 2.2 MB 00:01 2023-11-28T20:31:33.828 INFO:teuthology.orchestra.run.smithi190.stdout:(149/150): openssl-3.0.7-25.el9.x86_64.rpm 920 kB/s | 1.2 MB 00:01 2023-11-28T20:31:33.828 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:34.137 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.ocf.berkeley.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 169.229.200.70) 2023-11-28T20:31:34.301 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:34.364 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 163.237.219.7) 2023-11-28T20:31:34.435 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:34.473 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 199.223.246.113) 2023-11-28T20:31:34.634 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:34.816 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 216.31.0.2) 2023-11-28T20:31:34.854 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:35.168 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.timkevin.us/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 23.145.168.14) 2023-11-28T20:31:35.346 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:35.497 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 64.50.233.100) 2023-11-28T20:31:35.584 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 104.167.241.199) 2023-11-28T20:31:35.984 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 140.211.166.134) 2023-11-28T20:31:36.025 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:36.354 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 104.167.241.199) 2023-11-28T20:31:36.399 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:36.541 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 162.255.226.75) 2023-11-28T20:31:36.569 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:36.703 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 206.82.17.213) 2023-11-28T20:31:36.904 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.hoobly.com/centos/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 85.25.43.203) 2023-11-28T20:31:37.023 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 72.4.120.219) 2023-11-28T20:31:37.187 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] NetworkManager-tui-1.45.7-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-tui-1.45.7-1.el9.x86_64.rpm (IP: 163.114.132.32) 2023-11-28T20:31:37.503 INFO:teuthology.orchestra.run.smithi190.stdout:(150/150): NetworkManager-tui-1.45.7-1.el9.x86_ 47 kB/s | 246 kB 00:05 2023-11-28T20:31:37.516 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-11-28T20:31:37.516 INFO:teuthology.orchestra.run.smithi190.stdout:Total 7.5 MB/s | 205 MB 00:27 2023-11-28T20:31:37.970 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-11-28T20:31:38.254 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-11-28T20:31:38.254 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-11-28T20:31:38.589 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-11-28T20:31:38.750 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/165 2023-11-28T20:31:38.787 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/165 2023-11-28T20:31:38.900 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/165 2023-11-28T20:31:38.960 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/165 2023-11-28T20:31:38.996 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2023-11-28T20:31:39.105 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2023-11-28T20:31:39.518 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2023-11-28T20:31:39.560 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:18.0.0-7520.g43dd94c1.el9.x86_64 8/165 2023-11-28T20:31:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:18.0.0-7520.g43dd94c1.el9.x86_64 8/165 2023-11-28T20:31:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:18.0.0-7520.g43dd94c1.el9.x86_64 9/165 2023-11-28T20:31:39.867 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:18.0.0-7520.g43dd94c1.el9.x86_64 9/165 2023-11-28T20:31:39.897 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:18.0.0-7520.g43dd94c1.el9.x86_64 10/165 2023-11-28T20:31:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/165 2023-11-28T20:31:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/165 2023-11-28T20:31:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/165 2023-11-28T20:31:40.350 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:18.0.0-7520.g43dd94c1.el9.x86_64 14/165 2023-11-28T20:31:40.488 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-11-28T20:31:40.489 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-11-28T20:31:40.530 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7520.g43dd94c1.el9.x86_64 14/165 2023-11-28T20:31:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:18.0.0-7520.g43dd94c1.el9.x86 15/165 2023-11-28T20:31:40.909 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7520.g43dd94c1.el9.x86 15/165 2023-11-28T20:31:41.090 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/165 2023-11-28T20:31:41.179 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/165 2023-11-28T20:31:41.379 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/165 2023-11-28T20:31:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-1:1.45.7-1.el9.x86_64 17/165 2023-11-28T20:31:41.780 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/165 2023-11-28T20:31:41.990 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 18/165 2023-11-28T20:31:42.031 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/165 2023-11-28T20:31:42.162 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/165 2023-11-28T20:31:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/165 2023-11-28T20:31:42.286 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/165 2023-11-28T20:31:42.346 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/165 2023-11-28T20:31:42.398 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/165 2023-11-28T20:31:42.514 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/165 2023-11-28T20:31:42.555 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/165 2023-11-28T20:31:42.600 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/165 2023-11-28T20:31:42.640 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2023-11-28T20:31:42.801 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2023-11-28T20:31:42.844 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:18.0.0-7520.g43dd94c1.el9. 30/165 2023-11-28T20:31:42.898 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:18.0.0-7520.g43dd94c1.el 31/165 2023-11-28T20:31:42.934 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:18.0.0-7520.g43dd94c1.el9.x86_6 32/165 2023-11-28T20:31:43.056 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/165 2023-11-28T20:31:43.173 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/165 2023-11-28T20:31:43.356 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 35/165 2023-11-28T20:31:43.423 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 36/165 2023-11-28T20:31:44.149 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 37/165 2023-11-28T20:31:44.251 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 38/165 2023-11-28T20:31:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 39/165 2023-11-28T20:31:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 40/165 2023-11-28T20:31:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:18.0.0-7520.g43dd94c1.el9.x86_6 41/165 2023-11-28T20:31:44.469 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 42/165 2023-11-28T20:31:44.695 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 43/165 2023-11-28T20:31:44.740 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 44/165 2023-11-28T20:31:44.894 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-3.el9.x86_64 45/165 2023-11-28T20:31:44.935 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 46/165 2023-11-28T20:31:45.017 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/165 2023-11-28T20:31:45.118 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/165 2023-11-28T20:31:45.161 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/165 2023-11-28T20:31:45.201 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/165 2023-11-28T20:31:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/165 2023-11-28T20:31:45.335 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/165 2023-11-28T20:31:45.395 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 53/165 2023-11-28T20:31:45.590 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 54/165 2023-11-28T20:31:45.643 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 55/165 2023-11-28T20:31:45.691 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 56/165 2023-11-28T20:31:45.824 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 57/165 2023-11-28T20:31:45.868 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 58/165 2023-11-28T20:31:45.918 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 59/165 2023-11-28T20:31:45.950 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 60/165 2023-11-28T20:31:46.100 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 61/165 2023-11-28T20:31:46.165 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 62/165 2023-11-28T20:31:46.238 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 63/165 2023-11-28T20:31:46.317 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 64/165 2023-11-28T20:31:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 65/165 2023-11-28T20:31:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 66/165 2023-11-28T20:31:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-11-28T20:31:46.813 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 67/165 2023-11-28T20:31:46.871 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 68/165 2023-11-28T20:31:46.904 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 69/165 2023-11-28T20:31:46.920 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/165 2023-11-28T20:31:46.954 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 70/165 2023-11-28T20:31:46.960 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/165 2023-11-28T20:31:47.031 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 71/165 2023-11-28T20:31:47.033 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/165 2023-11-28T20:31:47.077 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 72/165 2023-11-28T20:31:47.087 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/165 2023-11-28T20:31:47.119 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2023-11-28T20:31:47.130 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 73/165 2023-11-28T20:31:47.196 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/165 2023-11-28T20:31:47.218 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2023-11-28T20:31:47.259 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : uuid-1.6.2-55.el9.x86_64 75/165 2023-11-28T20:31:47.309 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/165 2023-11-28T20:31:47.457 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/165 2023-11-28T20:31:47.528 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/165 2023-11-28T20:31:47.564 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/165 2023-11-28T20:31:47.632 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/165 2023-11-28T20:31:47.633 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2023-11-28T20:31:47.676 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:18.0.0-7520.g43dd94c1.el9.x86_64 8/165 2023-11-28T20:31:47.676 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/165 2023-11-28T20:31:47.825 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/165 2023-11-28T20:31:47.831 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:18.0.0-7520.g43dd94c1.el9.x86_64 8/165 2023-11-28T20:31:47.860 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:18.0.0-7520.g43dd94c1.el9.x86_64 9/165 2023-11-28T20:31:47.873 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/165 2023-11-28T20:31:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:18.0.0-7520.g43dd94c1.el9.x86_64 9/165 2023-11-28T20:31:48.017 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/165 2023-11-28T20:31:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:18.0.0-7520.g43dd94c1.el9.x86_64 10/165 2023-11-28T20:31:48.079 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/165 2023-11-28T20:31:48.088 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/165 2023-11-28T20:31:48.354 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/165 2023-11-28T20:31:48.416 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/165 2023-11-28T20:31:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:18.0.0-7520.g43dd94c1.el9.x86_64 14/165 2023-11-28T20:31:48.586 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7520.g43dd94c1.el9.x86_64 14/165 2023-11-28T20:31:48.608 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:18.0.0-7520.g43dd94c1.el9.x86 15/165 2023-11-28T20:31:48.946 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7520.g43dd94c1.el9.x86 15/165 2023-11-28T20:31:49.104 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/165 2023-11-28T20:31:49.121 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/165 2023-11-28T20:31:49.162 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/165 2023-11-28T20:31:49.196 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/165 2023-11-28T20:31:49.210 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/165 2023-11-28T20:31:49.229 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/165 2023-11-28T20:31:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/165 2023-11-28T20:31:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-1:1.45.7-1.el9.x86_64 17/165 2023-11-28T20:31:49.478 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 90/165 2023-11-28T20:31:49.508 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 91/165 2023-11-28T20:31:49.610 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 92/165 2023-11-28T20:31:49.777 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/165 2023-11-28T20:31:49.944 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 18/165 2023-11-28T20:31:49.983 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/165 2023-11-28T20:31:50.099 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/165 2023-11-28T20:31:50.165 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/165 2023-11-28T20:31:50.204 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/165 2023-11-28T20:31:50.268 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/165 2023-11-28T20:31:50.323 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/165 2023-11-28T20:31:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/165 2023-11-28T20:31:50.457 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 93/165 2023-11-28T20:31:50.484 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/165 2023-11-28T20:31:50.536 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/165 2023-11-28T20:31:50.576 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2023-11-28T20:31:50.595 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 94/165 2023-11-28T20:31:50.690 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2023-11-28T20:31:50.729 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:18.0.0-7520.g43dd94c1.el9. 30/165 2023-11-28T20:31:50.772 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:18.0.0-7520.g43dd94c1.el 31/165 2023-11-28T20:31:50.793 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:18.0.0-7520.g43dd94c1.el9.x86_6 32/165 2023-11-28T20:31:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/165 2023-11-28T20:31:50.998 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/165 2023-11-28T20:31:51.170 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 35/165 2023-11-28T20:31:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 36/165 2023-11-28T20:31:51.922 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 37/165 2023-11-28T20:31:52.022 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 38/165 2023-11-28T20:31:52.084 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 39/165 2023-11-28T20:31:52.140 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 40/165 2023-11-28T20:31:52.188 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:18.0.0-7520.g43dd94c1.el9.x86_6 41/165 2023-11-28T20:31:52.250 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 42/165 2023-11-28T20:31:52.269 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 95/165 2023-11-28T20:31:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 96/165 2023-11-28T20:31:52.403 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 97/165 2023-11-28T20:31:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-16.el9.x86_64 98/165 2023-11-28T20:31:52.469 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2023-11-28T20:31:52.475 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 43/165 2023-11-28T20:31:52.502 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 100/165 2023-11-28T20:31:52.524 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 44/165 2023-11-28T20:31:52.667 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 45/165 2023-11-28T20:31:52.710 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 46/165 2023-11-28T20:31:52.774 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/165 2023-11-28T20:31:52.862 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/165 2023-11-28T20:31:52.886 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2023-11-28T20:31:52.903 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/165 2023-11-28T20:31:52.914 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:18.0.0-7520.g43dd94c1.el9.x86_64 102/165 2023-11-28T20:31:52.942 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/165 2023-11-28T20:31:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/165 2023-11-28T20:31:53.036 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:18.0.0-7520.g43dd94c1.el9.x86_64 102/165 2023-11-28T20:31:53.067 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/165 2023-11-28T20:31:53.070 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 103/165 2023-11-28T20:31:53.125 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2023-11-28T20:31:53.128 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 53/165 2023-11-28T20:31:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 54/165 2023-11-28T20:31:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 55/165 2023-11-28T20:31:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 56/165 2023-11-28T20:31:53.548 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 57/165 2023-11-28T20:31:53.586 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 58/165 2023-11-28T20:31:53.629 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 105/165 2023-11-28T20:31:53.633 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 59/165 2023-11-28T20:31:53.659 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:18.0.0-7520.g43dd94c1.el9.x86_64 106/165 2023-11-28T20:31:53.667 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 60/165 2023-11-28T20:31:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:18.0.0-7520.g43dd94c1.el9.x86_64 106/165 2023-11-28T20:31:53.811 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 107/165 2023-11-28T20:31:53.817 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 61/165 2023-11-28T20:31:53.871 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 62/165 2023-11-28T20:31:53.941 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 63/165 2023-11-28T20:31:54.020 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 64/165 2023-11-28T20:31:54.278 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 65/165 2023-11-28T20:31:54.334 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 66/165 2023-11-28T20:31:54.505 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 67/165 2023-11-28T20:31:54.554 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 68/165 2023-11-28T20:31:54.586 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 69/165 2023-11-28T20:31:54.618 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 70/165 2023-11-28T20:31:54.660 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 71/165 2023-11-28T20:31:54.697 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 72/165 2023-11-28T20:31:54.739 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 73/165 2023-11-28T20:31:54.810 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/165 2023-11-28T20:31:54.873 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : uuid-1.6.2-55.el9.x86_64 75/165 2023-11-28T20:31:54.924 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/165 2023-11-28T20:31:55.094 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/165 2023-11-28T20:31:55.136 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/165 2023-11-28T20:31:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/165 2023-11-28T20:31:55.237 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/165 2023-11-28T20:31:55.273 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/165 2023-11-28T20:31:55.405 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/165 2023-11-28T20:31:55.454 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/165 2023-11-28T20:31:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/165 2023-11-28T20:31:55.660 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/165 2023-11-28T20:31:56.471 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 108/165 2023-11-28T20:31:56.517 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 108/165 2023-11-28T20:31:56.631 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/165 2023-11-28T20:31:56.693 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/165 2023-11-28T20:31:56.719 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/165 2023-11-28T20:31:56.744 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/165 2023-11-28T20:31:57.002 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.21-2.el9.x86_64 90/165 2023-11-28T20:31:57.032 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 91/165 2023-11-28T20:31:57.133 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 92/165 2023-11-28T20:31:57.183 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 108/165 2023-11-28T20:31:57.230 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:18.0.0-7520.g43dd94c1.el9.x86_64 109/165 2023-11-28T20:31:57.422 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:18.0.0-7520.g43dd94c1.el9.x86_64 109/165 2023-11-28T20:31:57.422 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-28T20:31:57.423 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-28T20:31:57.423 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:31:57.480 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:18.0.0-7520.g43dd94c1.el9.x86_64 110/165 2023-11-28T20:31:57.927 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 93/165 2023-11-28T20:31:58.060 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 94/165 2023-11-28T20:31:59.585 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 95/165 2023-11-28T20:31:59.677 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 96/165 2023-11-28T20:31:59.715 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 97/165 2023-11-28T20:31:59.746 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : jq-1.6-16.el9.x86_64 98/165 2023-11-28T20:31:59.781 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2023-11-28T20:31:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 100/165 2023-11-28T20:32:00.166 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2023-11-28T20:32:00.194 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:18.0.0-7520.g43dd94c1.el9.x86_64 102/165 2023-11-28T20:32:00.299 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:18.0.0-7520.g43dd94c1.el9.x86_64 102/165 2023-11-28T20:32:00.356 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 103/165 2023-11-28T20:32:00.414 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2023-11-28T20:32:00.926 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 105/165 2023-11-28T20:32:00.955 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:18.0.0-7520.g43dd94c1.el9.x86_64 106/165 2023-11-28T20:32:01.047 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:18.0.0-7520.g43dd94c1.el9.x86_64 106/165 2023-11-28T20:32:01.074 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 107/165 2023-11-28T20:32:03.582 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 108/165 2023-11-28T20:32:03.621 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 108/165 2023-11-28T20:32:04.247 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 108/165 2023-11-28T20:32:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:18.0.0-7520.g43dd94c1.el9.x86_64 109/165 2023-11-28T20:32:04.446 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:18.0.0-7520.g43dd94c1.el9.x86_64 109/165 2023-11-28T20:32:04.446 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-28T20:32:04.446 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-28T20:32:04.446 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:04.484 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:18.0.0-7520.g43dd94c1.el9.x86_64 110/165 2023-11-28T20:32:10.494 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7520.g43dd94c1.el9.x86_64 110/165 2023-11-28T20:32:10.495 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2023-11-28T20:32:10.495 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2023-11-28T20:32:10.495 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2023-11-28T20:32:10.495 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2023-11-28T20:32:10.495 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2023-11-28T20:32:10.495 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2023-11-28T20:32:10.495 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2023-11-28T20:32:10.495 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:10.716 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:18.0.0-7520.g43dd94c1.el9.noarch 111/165 2023-11-28T20:32:10.770 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:18.0.0-7520.g43dd94c1.el9.noarch 111/165 2023-11-28T20:32:10.770 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:10.770 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-28T20:32:10.770 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:18.0.0-7520.g43dd94c1.el9.x86_64 112/165 2023-11-28T20:32:12.622 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:18.0.0-7520.g43dd94c1.el9.x86_64 112/165 2023-11-28T20:32:12.622 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:12.622 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-28T20:32:12.622 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T20:32:12.623 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T20:32:12.623 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:18.0.0-7520.g43dd94c1.el9.x86_64 113/165 2023-11-28T20:32:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:18.0.0-7520.g43dd94c1.el9.x86_64 113/165 2023-11-28T20:32:12.937 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:12.937 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-28T20:32:12.937 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T20:32:12.937 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T20:32:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:13.470 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:18.0.0-7520.g43dd94c1.el9.x86_64 114/165 2023-11-28T20:32:13.560 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:18.0.0-7520.g43dd94c1.el9.x86_64 114/165 2023-11-28T20:32:13.560 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:13.560 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-28T20:32:13.560 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T20:32:13.560 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T20:32:13.560 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:13.689 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2023-11-28T20:32:14.339 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 116/165 2023-11-28T20:32:14.373 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 116/165 2023-11-28T20:32:14.418 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7520.g43dd94c1.e 117/165 2023-11-28T20:32:14.528 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7520.g43dd94c1. 118/165 2023-11-28T20:32:14.561 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7520.g43dd94c1.el9.noa 119/165 2023-11-28T20:32:14.882 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7520.g43dd94c1.el9.noa 119/165 2023-11-28T20:32:14.912 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7520.g43dd94c1.el9.n 120/165 2023-11-28T20:32:16.096 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7520.g43dd94c1.el9.n 120/165 2023-11-28T20:32:16.120 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7520.g43d 121/165 2023-11-28T20:32:16.270 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7520.g43d 121/165 2023-11-28T20:32:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7520.g43dd94c1.el 122/165 2023-11-28T20:32:16.481 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:18.0.0-7520.g43dd94c1.el9.x86_64 123/165 2023-11-28T20:32:16.567 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7520.g43dd94c1.el9.x86_64 123/165 2023-11-28T20:32:16.567 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:16.567 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-28T20:32:16.567 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T20:32:16.567 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T20:32:16.568 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:16.632 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:18.0.0-7520.g43dd94c1.el9.noarch 124/165 2023-11-28T20:32:16.673 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7520.g43dd94c1.el9.noarch 124/165 2023-11-28T20:32:16.767 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:18.0.0-7520.g43dd94c1.el9.x86_64 125/165 2023-11-28T20:32:17.019 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7520.g43dd94c1.el9.x86_64 110/165 2023-11-28T20:32:17.019 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2023-11-28T20:32:17.019 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2023-11-28T20:32:17.019 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2023-11-28T20:32:17.019 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2023-11-28T20:32:17.020 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2023-11-28T20:32:17.020 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2023-11-28T20:32:17.020 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2023-11-28T20:32:17.020 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:17.231 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:18.0.0-7520.g43dd94c1.el9.noarch 111/165 2023-11-28T20:32:17.276 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:18.0.0-7520.g43dd94c1.el9.noarch 111/165 2023-11-28T20:32:17.277 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:17.277 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-28T20:32:17.277 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:17.712 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libtool-2.4.6-45.el9.x86_64 126/165 2023-11-28T20:32:17.743 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 127/165 2023-11-28T20:32:17.827 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 127/165 2023-11-28T20:32:17.828 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:17.828 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-28T20:32:17.828 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T20:32:17.828 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T20:32:17.828 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:17.877 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7520.g43dd9 128/165 2023-11-28T20:32:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7520.g43dd9 128/165 2023-11-28T20:32:17.982 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:17.982 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-28T20:32:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 129/165 2023-11-28T20:32:18.180 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 129/165 2023-11-28T20:32:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-11-28T20:32:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T20:32:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T20:32:18.180 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:18.553 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 130/165 2023-11-28T20:32:18.651 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 130/165 2023-11-28T20:32:18.652 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:18.652 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-28T20:32:18.652 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T20:32:18.652 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T20:32:18.652 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:18.961 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:18.0.0-7520.g43dd94c1.el9.x86_64 112/165 2023-11-28T20:32:19.045 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:18.0.0-7520.g43dd94c1.el9.x86_64 112/165 2023-11-28T20:32:19.045 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:19.045 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-28T20:32:19.046 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T20:32:19.046 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T20:32:19.046 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:19.261 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:18.0.0-7520.g43dd94c1.el9.x86_64 113/165 2023-11-28T20:32:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:18.0.0-7520.g43dd94c1.el9.x86_64 113/165 2023-11-28T20:32:19.334 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:19.334 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-28T20:32:19.335 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T20:32:19.335 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T20:32:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:19.820 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:18.0.0-7520.g43dd94c1.el9.x86_64 114/165 2023-11-28T20:32:19.901 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:18.0.0-7520.g43dd94c1.el9.x86_64 114/165 2023-11-28T20:32:19.901 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:19.902 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-28T20:32:19.902 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T20:32:19.902 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T20:32:19.902 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:20.014 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2023-11-28T20:32:20.669 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 116/165 2023-11-28T20:32:20.706 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 116/165 2023-11-28T20:32:20.742 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7520.g43dd94c1.e 117/165 2023-11-28T20:32:20.822 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7520.g43dd94c1. 118/165 2023-11-28T20:32:20.844 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7520.g43dd94c1.el9.noa 119/165 2023-11-28T20:32:21.159 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7520.g43dd94c1.el9.noa 119/165 2023-11-28T20:32:21.195 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7520.g43dd94c1.el9.n 120/165 2023-11-28T20:32:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7520.g43dd94c1.el9.n 120/165 2023-11-28T20:32:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7520.g43d 121/165 2023-11-28T20:32:22.512 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7520.g43d 121/165 2023-11-28T20:32:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7520.g43dd94c1.el 122/165 2023-11-28T20:32:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:18.0.0-7520.g43dd94c1.el9.x86_64 123/165 2023-11-28T20:32:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7520.g43dd94c1.el9.x86_64 123/165 2023-11-28T20:32:22.780 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:22.780 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-28T20:32:22.780 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T20:32:22.780 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T20:32:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:22.840 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:18.0.0-7520.g43dd94c1.el9.noarch 124/165 2023-11-28T20:32:22.879 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7520.g43dd94c1.el9.noarch 124/165 2023-11-28T20:32:22.968 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:18.0.0-7520.g43dd94c1.el9.x86_64 125/165 2023-11-28T20:32:23.018 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:18.0.0-7520.g43dd94c1.el9.x86_64 131/165 2023-11-28T20:32:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 132/165 2023-11-28T20:32:23.119 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 133/165 2023-11-28T20:32:23.158 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 134/165 2023-11-28T20:32:23.460 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 135/165 2023-11-28T20:32:23.599 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 136/165 2023-11-28T20:32:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 137/165 2023-11-28T20:32:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:18.0.0-7520.g43dd94c1.el9.x86_ 138/165 2023-11-28T20:32:23.772 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 139/165 2023-11-28T20:32:23.817 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libtool-2.4.6-45.el9.x86_64 126/165 2023-11-28T20:32:23.842 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 127/165 2023-11-28T20:32:23.877 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 140/165 2023-11-28T20:32:23.914 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 127/165 2023-11-28T20:32:23.914 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:23.914 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-28T20:32:23.914 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T20:32:23.914 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T20:32:23.914 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:23.918 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 141/165 2023-11-28T20:32:23.972 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-team-1:1.45.7-1.el9.x86_64 142/165 2023-11-28T20:32:24.009 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7520.g43dd9 128/165 2023-11-28T20:32:24.016 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-tui-1:1.45.7-1.el9.x86_64 143/165 2023-11-28T20:32:24.090 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7520.g43dd9 128/165 2023-11-28T20:32:24.090 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:24.091 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-28T20:32:24.091 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:24.160 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 129/165 2023-11-28T20:32:24.239 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 129/165 2023-11-28T20:32:24.239 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:24.239 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-11-28T20:32:24.239 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T20:32:24.239 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T20:32:24.239 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:24.577 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 130/165 2023-11-28T20:32:24.676 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 130/165 2023-11-28T20:32:24.676 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T20:32:24.676 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-28T20:32:24.676 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T20:32:24.677 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T20:32:24.677 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:24.977 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T20:32:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T20:32:25.077 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T20:32:26.529 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-top-2:18.0.0-7520.g43dd94c1.el9.noarch 145/165 2023-11-28T20:32:26.680 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 146/165 2023-11-28T20:32:26.754 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 147/165 2023-11-28T20:32:26.809 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/165 2023-11-28T20:32:26.970 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/165 2023-11-28T20:32:27.008 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/165 2023-11-28T20:32:27.042 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 151/165 2023-11-28T20:32:27.076 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 152/165 2023-11-28T20:32:27.145 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 153/165 2023-11-28T20:32:27.145 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T20:32:27.292 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T20:32:27.343 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 155/165 2023-11-28T20:32:27.389 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 156/165 2023-11-28T20:32:27.422 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 157/165 2023-11-28T20:32:27.422 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T20:32:27.559 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T20:32:27.590 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2023-11-28T20:32:27.642 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T20:32:27.643 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T20:32:27.964 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T20:32:27.964 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T20:32:28.117 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T20:32:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 162/165 2023-11-28T20:32:28.193 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 163/165 2023-11-28T20:32:28.228 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 164/165 2023-11-28T20:32:28.360 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T20:32:28.620 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T20:32:28.621 INFO:teuthology.orchestra.run.smithi045.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-28T20:32:28.621 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:28.808 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:18.0.0-7520.g43dd94c1.el9.x86_64 131/165 2023-11-28T20:32:28.863 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 132/165 2023-11-28T20:32:28.918 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 133/165 2023-11-28T20:32:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 134/165 2023-11-28T20:32:29.238 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 135/165 2023-11-28T20:32:29.366 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 136/165 2023-11-28T20:32:29.402 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 137/165 2023-11-28T20:32:29.427 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:18.0.0-7520.g43dd94c1.el9.x86_ 138/165 2023-11-28T20:32:29.504 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 139/165 2023-11-28T20:32:29.566 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 140/165 2023-11-28T20:32:29.614 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 141/165 2023-11-28T20:32:29.666 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-team-1:1.45.7-1.el9.x86_64 142/165 2023-11-28T20:32:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-tui-1:1.45.7-1.el9.x86_64 143/165 2023-11-28T20:32:30.683 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T20:32:30.745 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T20:32:30.819 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T20:32:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-top-2:18.0.0-7520.g43dd94c1.el9.noarch 145/165 2023-11-28T20:32:32.225 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 146/165 2023-11-28T20:32:32.275 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 147/165 2023-11-28T20:32:32.344 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/165 2023-11-28T20:32:32.493 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/165 2023-11-28T20:32:32.529 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/165 2023-11-28T20:32:32.562 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 151/165 2023-11-28T20:32:32.596 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 152/165 2023-11-28T20:32:32.637 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 153/165 2023-11-28T20:32:32.638 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T20:32:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T20:32:32.822 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 155/165 2023-11-28T20:32:32.868 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 156/165 2023-11-28T20:32:32.900 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 157/165 2023-11-28T20:32:32.900 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T20:32:33.046 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T20:32:33.077 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2023-11-28T20:32:33.116 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T20:32:33.117 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T20:32:33.292 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:18.0.0-7520.g43dd94c1.el9.x86_64 1/165 2023-11-28T20:32:33.292 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:18.0.0-7520.g43dd94c1.el9.x86_64 2/165 2023-11-28T20:32:33.292 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 3/165 2023-11-28T20:32:33.292 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 4/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7520.g43dd9 5/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:18.0.0-7520.g43dd94c1.el9.x86_64 6/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:18.0.0-7520.g43dd94c1.el9.x86_64 7/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:18.0.0-7520.g43dd94c1.el9.x86_64 8/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:18.0.0-7520.g43dd94c1.el9.x86_64 9/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 10/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:18.0.0-7520.g43dd94c1.el9.x86_64 11/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:18.0.0-7520.g43dd94c1.el9.x86_64 12/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 13/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:18.0.0-7520.g43dd94c1.el9.x86_ 14/165 2023-11-28T20:32:33.293 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:18.0.0-7520.g43dd94c1.el9.x86_64 15/165 2023-11-28T20:32:33.294 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:18.0.0-7520.g43dd94c1.el9.x86_64 16/165 2023-11-28T20:32:33.294 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:18.0.0-7520.g43dd94c1.el9.x86_6 17/165 2023-11-28T20:32:33.294 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:18.0.0-7520.g43dd94c1.el9.x86 18/165 2023-11-28T20:32:33.294 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:18.0.0-7520.g43dd94c1.el9.x86_64 19/165 2023-11-28T20:32:33.294 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7520.g43dd94c1.el 20/165 2023-11-28T20:32:33.294 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:18.0.0-7520.g43dd94c1.el9. 21/165 2023-11-28T20:32:33.294 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:18.0.0-7520.g43dd94c1.el9.x86_6 22/165 2023-11-28T20:32:33.294 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:18.0.0-7520.g43dd94c1.el9.x86_64 23/165 2023-11-28T20:32:33.294 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 24/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 25/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 26/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 27/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 28/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7520.g43dd94c1. 29/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7520.g43dd94c1.el9.noa 30/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7520.g43dd94c1.el9.n 31/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7520.g43d 32/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7520.g43dd94c1.el 33/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7520.g43dd94c1.el9.noarch 34/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7520.g43dd94c1.e 35/165 2023-11-28T20:32:33.295 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:18.0.0-7520.g43dd94c1.el9.noarch 36/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 37/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-top-2:18.0.0-7520.g43dd94c1.el9.noarch 38/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 42/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-16.el9.x86_64 47/165 2023-11-28T20:32:33.296 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 53/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 57/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 58/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 59/165 2023-11-28T20:32:33.297 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 65/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/165 2023-11-28T20:32:33.298 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 74/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 75/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 76/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 77/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 78/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 79/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 81/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 82/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/165 2023-11-28T20:32:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 84/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 85/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 86/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 87/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 88/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 89/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 90/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 91/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 92/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 93/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 94/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/165 2023-11-28T20:32:33.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 98/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 99/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 103/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/165 2023-11-28T20:32:33.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 110/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/165 2023-11-28T20:32:33.302 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/165 2023-11-28T20:32:33.303 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:18.0.0-7520.g43dd94c1.el9.x86_64 136/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:18.0.0-7520.g43dd94c1.el9.x86_64 138/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : acl-2.3.1-4.el9.x86_64 142/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : acl-2.3.1-3.el9.x86_64 143/165 2023-11-28T20:32:33.304 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 144/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 145/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 146/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 147/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 148/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 149/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.45.7-1.el9.x86_64 154/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 155/165 2023-11-28T20:32:33.305 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 156/165 2023-11-28T20:32:33.306 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 157/165 2023-11-28T20:32:33.306 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.45.7-1.el9.x86_64 158/165 2023-11-28T20:32:33.306 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2023-11-28T20:32:33.306 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.45.7-1.el9.x86_64 160/165 2023-11-28T20:32:33.306 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T20:32:33.306 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 162/165 2023-11-28T20:32:33.306 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 163/165 2023-11-28T20:32:33.306 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 164/165 2023-11-28T20:32:33.425 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T20:32:33.472 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T20:32:33.560 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T20:32:33.597 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 162/165 2023-11-28T20:32:33.630 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 163/165 2023-11-28T20:32:33.665 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 164/165 2023-11-28T20:32:33.792 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T20:32:34.057 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T20:32:34.057 INFO:teuthology.orchestra.run.smithi190.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-28T20:32:34.057 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 165/165 2023-11-28T20:32:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:35.546 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2023-11-28T20:32:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-1:1.45.7-1.el9.x86_64 2023-11-28T20:32:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 2023-11-28T20:32:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team-1:1.45.7-1.el9.x86_64 2023-11-28T20:32:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui-1:1.45.7-1.el9.x86_64 2023-11-28T20:32:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: acl-2.3.1-4.el9.x86_64 2023-11-28T20:32:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-2.3.1-4.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: automake-1.16.2-8.el9.noarch 2023-11-28T20:32:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: bison-3.7.4-5.el9.x86_64 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: flex-2.6.4-9.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-2.9.9-15.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.6-16.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-28T20:32:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-28T20:32:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: libtool-2.4.6-45.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-1.el9.noarch 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-28T20:32:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-13.11-1.el9.x86_64 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server-13.11-1.el9.x86_64 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-28T20:32:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-28T20:32:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-28T20:32:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-28T20:32:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-3.el9.x86_64 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-28T20:32:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-28T20:32:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: uuid-1.6.2-55.el9.x86_64 2023-11-28T20:32:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-11-28T20:32:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-28T20:32:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-11-28T20:32:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:35.559 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-11-28T20:32:36.142 DEBUG:teuthology.parallel:result is None 2023-11-28T20:32:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:18.0.0-7520.g43dd94c1.el9.x86_64 1/165 2023-11-28T20:32:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:18.0.0-7520.g43dd94c1.el9.x86_64 2/165 2023-11-28T20:32:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 3/165 2023-11-28T20:32:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 4/165 2023-11-28T20:32:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7520.g43dd9 5/165 2023-11-28T20:32:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:18.0.0-7520.g43dd94c1.el9.x86_64 6/165 2023-11-28T20:32:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:18.0.0-7520.g43dd94c1.el9.x86_64 7/165 2023-11-28T20:32:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:18.0.0-7520.g43dd94c1.el9.x86_64 8/165 2023-11-28T20:32:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:18.0.0-7520.g43dd94c1.el9.x86_64 9/165 2023-11-28T20:32:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 10/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:18.0.0-7520.g43dd94c1.el9.x86_64 11/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:18.0.0-7520.g43dd94c1.el9.x86_64 12/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 13/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:18.0.0-7520.g43dd94c1.el9.x86_ 14/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:18.0.0-7520.g43dd94c1.el9.x86_64 15/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:18.0.0-7520.g43dd94c1.el9.x86_64 16/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:18.0.0-7520.g43dd94c1.el9.x86_6 17/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:18.0.0-7520.g43dd94c1.el9.x86 18/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:18.0.0-7520.g43dd94c1.el9.x86_64 19/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7520.g43dd94c1.el 20/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:18.0.0-7520.g43dd94c1.el9. 21/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:18.0.0-7520.g43dd94c1.el9.x86_6 22/165 2023-11-28T20:32:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:18.0.0-7520.g43dd94c1.el9.x86_64 23/165 2023-11-28T20:32:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 24/165 2023-11-28T20:32:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 25/165 2023-11-28T20:32:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 26/165 2023-11-28T20:32:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 27/165 2023-11-28T20:32:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 28/165 2023-11-28T20:32:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7520.g43dd94c1. 29/165 2023-11-28T20:32:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7520.g43dd94c1.el9.noa 30/165 2023-11-28T20:32:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7520.g43dd94c1.el9.n 31/165 2023-11-28T20:32:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7520.g43d 32/165 2023-11-28T20:32:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7520.g43dd94c1.el 33/165 2023-11-28T20:32:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7520.g43dd94c1.el9.noarch 34/165 2023-11-28T20:32:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7520.g43dd94c1.e 35/165 2023-11-28T20:32:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:18.0.0-7520.g43dd94c1.el9.noarch 36/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 37/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-top-2:18.0.0-7520.g43dd94c1.el9.noarch 38/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 42/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : jq-1.6-16.el9.x86_64 47/165 2023-11-28T20:32:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 53/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 57/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 58/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 59/165 2023-11-28T20:32:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 65/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/165 2023-11-28T20:32:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 74/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 75/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 76/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 77/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 78/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 79/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 81/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 82/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 84/165 2023-11-28T20:32:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 85/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 86/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 87/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 88/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 89/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 90/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 91/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 92/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 93/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 94/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/165 2023-11-28T20:32:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 98/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 99/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 103/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/165 2023-11-28T20:32:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 110/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/165 2023-11-28T20:32:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/165 2023-11-28T20:32:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:18.0.0-7520.g43dd94c1.el9.x86_64 136/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:18.0.0-7520.g43dd94c1.el9.x86_64 138/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : acl-2.3.1-4.el9.x86_64 142/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : acl-2.3.1-3.el9.x86_64 143/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 144/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 145/165 2023-11-28T20:32:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 146/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 147/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 148/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 149/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-1:1.45.7-1.el9.x86_64 154/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 155/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 156/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 157/165 2023-11-28T20:32:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-team-1:1.45.7-1.el9.x86_64 158/165 2023-11-28T20:32:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2023-11-28T20:32:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-tui-1:1.45.7-1.el9.x86_64 160/165 2023-11-28T20:32:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T20:32:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 162/165 2023-11-28T20:32:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 163/165 2023-11-28T20:32:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 164/165 2023-11-28T20:32:40.729 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 165/165 2023-11-28T20:32:40.729 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:40.729 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2023-11-28T20:32:40.729 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-1:1.45.7-1.el9.x86_64 2023-11-28T20:32:40.729 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 2023-11-28T20:32:40.729 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-team-1:1.45.7-1.el9.x86_64 2023-11-28T20:32:40.729 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-tui-1:1.45.7-1.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: acl-2.3.1-4.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: libacl-2.3.1-4.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-11-28T20:32:40.730 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: automake-1.16.2-8.el9.noarch 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: bison-3.7.4-5.el9.x86_64 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.731 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top-2:18.0.0-7520.g43dd94c1.el9.noarch 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: flex-2.6.4-9.el9.x86_64 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-28T20:32:40.733 INFO:teuthology.orchestra.run.smithi190.stdout: fuse-2.9.9-15.el9.x86_64 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: jq-1.6-16.el9.x86_64 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-28T20:32:40.734 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: libtool-2.4.6-45.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-11-28T20:32:40.735 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-1.el9.noarch 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-13.11-1.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-11-28T20:32:40.736 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server-13.11-1.el9.x86_64 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-28T20:32:40.737 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-28T20:32:40.738 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-28T20:32:40.739 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.740 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-28T20:32:40.741 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-28T20:32:40.741 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-28T20:32:40.741 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.741 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-28T20:32:40.741 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-28T20:32:40.741 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-28T20:32:40.741 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-28T20:32:40.741 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-28T20:32:40.741 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:18.0.0-7520.g43dd94c1.el9.x86_64 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-28T20:32:40.742 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-28T20:32:40.743 INFO:teuthology.orchestra.run.smithi190.stdout: uuid-1.6.2-55.el9.x86_64 2023-11-28T20:32:40.743 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-11-28T20:32:40.743 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-28T20:32:40.743 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-11-28T20:32:40.743 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:40.743 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-11-28T20:32:41.292 DEBUG:teuthology.parallel:result is None 2023-11-28T20:32:41.292 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=43dd94c190073cf16cff1e711dd7f5a7b1cbc883 2023-11-28T20:32:41.531 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T20:32:41.582 INFO:teuthology.orchestra.run.smithi045.stdout:18.0.0-7520.g43dd94c1.el9 2023-11-28T20:32:41.582 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7520.g43dd94c1.el9 2023-11-28T20:32:41.583 INFO:teuthology.task.install:The correct ceph version 18.0.0-7520.g43dd94c1 is installed. 2023-11-28T20:32:41.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=43dd94c190073cf16cff1e711dd7f5a7b1cbc883 2023-11-28T20:32:41.721 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T20:32:41.779 INFO:teuthology.orchestra.run.smithi190.stdout:18.0.0-7520.g43dd94c1.el9 2023-11-28T20:32:41.779 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7520.g43dd94c1.el9 2023-11-28T20:32:41.779 INFO:teuthology.task.install:The correct ceph version 18.0.0-7520.g43dd94c1 is installed. 2023-11-28T20:32:41.780 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-28T20:32:41.780 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:41.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T20:32:41.857 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:32:41.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T20:32:41.932 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-28T20:32:41.933 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:41.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T20:32:42.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T20:32:42.128 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:32:42.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T20:32:42.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T20:32:42.325 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-28T20:32:42.325 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:42.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T20:32:42.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T20:32:42.545 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:32:42.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T20:32:42.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T20:32:42.758 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-28T20:32:42.759 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:42.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T20:32:42.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T20:32:42.941 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:32:42.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T20:32:43.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T20:32:43.132 INFO:teuthology.run_tasks:Running task ceph... 2023-11-28T20:32:43.211 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-28T20:32:43.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /var/log/ceph 2023-11-28T20:32:43.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2023-11-28T20:32:43.305 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-28T20:32:43.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T20:32:43.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T20:32:43.387 INFO:tasks.ceph:Creating extra log directories... 2023-11-28T20:32:43.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T20:32:43.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T20:32:43.484 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-11-28T20:32:43.485 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-11-28T20:32:43.485 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-11-28T20:32:43.485 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:43.485 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T20:32:43.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T20:32:43.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-28T20:32:43.897 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:32:43.898 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T20:32:43.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T20:32:44.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-28T20:32:44.235 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-28T20:32:44.235 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-28T20:32:44.236 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637', 'branch': 'wip-vshankar-testing-20231127.102914', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7469637', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20231127.102914'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20231127.102914', 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8377, 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883', 'sleep_before_teardown': 0, 'subset': '462/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20231127.102914', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi045.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhPbRY6n/z7r1pF/uD1tRgmCEGcsmUX170GbKygNpTf9+0WWSUXUT7sNh8sneEl4yMj4WqUfMlgxQbC129zAy0=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/yvpBetb1Oc/9+gPKWVCU/1lcSer5fSZJTIBiZbE7JIHVrtJbCVMGxerg9tkEPspZTIz+fk48ivQh6MPM1vyc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '43dd94c190073cf16cff1e711dd7f5a7b1cbc883', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '5ae09a5cc7064ca2bf994df19f3006b79794789a', 'timestamp': '2023-11-28_06:29:14', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-28T20:32:44.236 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T20:32:44.246 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T20:32:44.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T20:32:44.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T20:32:44.409 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:44.410 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T20:32:44.468 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T20:32:44.468 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2023-11-28T20:32:44.521 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T20:32:44.522 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T20:32:44.522 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 632 Links: 1 2023-11-28T20:32:44.522 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T20:32:44.522 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T20:32:44.522 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2023-11-28 20:32:29.065899611 +0000 2023-11-28T20:32:44.522 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2023-11-28 20:29:13.045603343 +0000 2023-11-28T20:32:44.522 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2023-11-28 20:29:13.045603343 +0000 2023-11-28T20:32:44.522 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2023-11-28 20:29:13.045603343 +0000 2023-11-28T20:32:44.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T20:32:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2023-11-28T20:32:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2023-11-28T20:32:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000252338 s, 2.0 MB/s 2023-11-28T20:32:44.619 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T20:32:44.681 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2023-11-28T20:32:44.738 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T20:32:44.739 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T20:32:44.739 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 653 Links: 1 2023-11-28T20:32:44.739 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T20:32:44.739 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T20:32:44.739 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2023-11-28 20:32:29.066899595 +0000 2023-11-28T20:32:44.739 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2023-11-28 20:29:14.084587073 +0000 2023-11-28T20:32:44.739 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2023-11-28 20:29:14.084587073 +0000 2023-11-28T20:32:44.739 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2023-11-28 20:29:14.084587073 +0000 2023-11-28T20:32:44.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T20:32:44.832 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2023-11-28T20:32:44.832 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2023-11-28T20:32:44.832 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000208169 s, 2.5 MB/s 2023-11-28T20:32:44.837 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T20:32:44.893 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2023-11-28T20:32:44.945 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T20:32:44.946 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T20:32:44.946 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 670 Links: 1 2023-11-28T20:32:44.946 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T20:32:44.946 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T20:32:44.946 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2023-11-28 20:32:29.067899580 +0000 2023-11-28T20:32:44.946 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2023-11-28 20:29:15.256568720 +0000 2023-11-28T20:32:44.946 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2023-11-28 20:29:15.256568720 +0000 2023-11-28T20:32:44.946 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2023-11-28 20:29:15.256568720 +0000 2023-11-28T20:32:44.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T20:32:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2023-11-28T20:32:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2023-11-28T20:32:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000284108 s, 1.8 MB/s 2023-11-28T20:32:45.046 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T20:32:45.107 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2023-11-28T20:32:45.161 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T20:32:45.162 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T20:32:45.162 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 691 Links: 1 2023-11-28T20:32:45.162 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T20:32:45.162 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T20:32:45.162 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2023-11-28 20:32:29.067899580 +0000 2023-11-28T20:32:45.162 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2023-11-28 20:29:16.331551886 +0000 2023-11-28T20:32:45.162 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2023-11-28 20:29:16.331551886 +0000 2023-11-28T20:32:45.162 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2023-11-28 20:29:16.331551886 +0000 2023-11-28T20:32:45.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T20:32:45.260 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2023-11-28T20:32:45.260 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2023-11-28T20:32:45.260 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000269326 s, 1.9 MB/s 2023-11-28T20:32:45.264 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T20:32:45.329 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T20:32:45.329 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:32:45.329 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T20:32:45.374 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T20:32:45.374 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2023-11-28T20:32:45.467 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T20:32:45.467 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T20:32:45.467 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 632 Links: 1 2023-11-28T20:32:45.467 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T20:32:45.467 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T20:32:45.467 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-11-28 20:32:34.479970203 +0000 2023-11-28T20:32:45.467 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-11-28 20:28:48.185288546 +0000 2023-11-28T20:32:45.467 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-11-28 20:28:48.185288546 +0000 2023-11-28T20:32:45.467 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2023-11-28 20:28:48.185288546 +0000 2023-11-28T20:32:45.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T20:32:45.545 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-11-28T20:32:45.545 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-11-28T20:32:45.545 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000454087 s, 1.1 MB/s 2023-11-28T20:32:45.548 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T20:32:45.597 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2023-11-28T20:32:45.640 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T20:32:45.640 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T20:32:45.640 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 651 Links: 1 2023-11-28T20:32:45.640 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T20:32:45.641 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T20:32:45.641 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-11-28 20:32:34.481970145 +0000 2023-11-28T20:32:45.641 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-11-28 20:28:49.199259717 +0000 2023-11-28T20:32:45.641 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-11-28 20:28:49.199259717 +0000 2023-11-28T20:32:45.641 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2023-11-28 20:28:49.199259717 +0000 2023-11-28T20:32:45.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T20:32:45.758 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-11-28T20:32:45.758 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-11-28T20:32:45.759 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000306956 s, 1.7 MB/s 2023-11-28T20:32:45.764 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T20:32:45.808 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2023-11-28T20:32:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T20:32:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T20:32:45.889 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 672 Links: 1 2023-11-28T20:32:45.890 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T20:32:45.890 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T20:32:45.890 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-11-28 20:32:34.481970145 +0000 2023-11-28T20:32:45.890 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-11-28 20:28:50.213230889 +0000 2023-11-28T20:32:45.890 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-11-28 20:28:50.213230889 +0000 2023-11-28T20:32:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2023-11-28 20:28:50.213230889 +0000 2023-11-28T20:32:45.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T20:32:45.999 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-11-28T20:32:46.000 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-11-28T20:32:46.000 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000335079 s, 1.5 MB/s 2023-11-28T20:32:46.003 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T20:32:46.044 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2023-11-28T20:32:46.127 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T20:32:46.128 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T20:32:46.128 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 690 Links: 1 2023-11-28T20:32:46.128 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T20:32:46.128 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T20:32:46.128 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-11-28 20:32:34.482970117 +0000 2023-11-28T20:32:46.128 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-11-28 20:28:51.231201947 +0000 2023-11-28T20:32:46.128 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-11-28 20:28:51.231201947 +0000 2023-11-28T20:32:46.128 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2023-11-28 20:28:51.231201947 +0000 2023-11-28T20:32:46.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T20:32:46.261 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-11-28T20:32:46.261 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-11-28T20:32:46.261 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000354234 s, 1.4 MB/s 2023-11-28T20:32:46.264 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T20:32:46.319 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T20:32:46.319 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T20:32:46.319 INFO:tasks.ceph:Generating config... 2023-11-28T20:32:46.320 INFO:tasks.ceph:[client] client mount timeout = 600 2023-11-28T20:32:46.320 INFO:tasks.ceph:[client] debug client = 20 2023-11-28T20:32:46.320 INFO:tasks.ceph:[client] debug ms = 1 2023-11-28T20:32:46.320 INFO:tasks.ceph:[client] pid file = 2023-11-28T20:32:46.321 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-11-28T20:32:46.321 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-11-28T20:32:46.321 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-11-28T20:32:46.321 INFO:tasks.ceph:[mds] debug mds = 20 2023-11-28T20:32:46.321 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-11-28T20:32:46.321 INFO:tasks.ceph:[mds] debug ms = 1 2023-11-28T20:32:46.321 INFO:tasks.ceph:[mds] mds debug frag = True 2023-11-28T20:32:46.321 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-11-28T20:32:46.321 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mgr] debug client = 20 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-28T20:32:46.322 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-28T20:32:46.323 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-11-28T20:32:46.323 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-28T20:32:46.323 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-28T20:32:46.323 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-28T20:32:46.323 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-28T20:32:46.323 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-28T20:32:46.323 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-28T20:32:46.323 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-28T20:32:46.323 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-28T20:32:46.323 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-28T20:32:46.324 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-28T20:32:46.324 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-28T20:32:46.324 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-28T20:32:46.324 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-11-28T20:32:46.324 INFO:tasks.ceph:Setting up mon.a... 2023-11-28T20:32:46.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-28T20:32:46.483 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.keyring 2023-11-28T20:32:46.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-28T20:32:46.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T20:32:46.747 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.45'), ('mon.b', '172.21.15.190'), ('mon.c', '[v2:172.21.15.190:3301,v1:172.21.15.190:6790]')] 2023-11-28T20:32:46.747 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.45,172.21.15.190,[v2:172.21.15.190:3301,v1:172.21.15.190:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-11-28T20:32:46.748 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:46.748 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T20:32:46.808 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.45 --add b 172.21.15.190 --addv c '[v2:172.21.15.190:3301,v1:172.21.15.190:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:32:46.971 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:32:46.971 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: generated fsid ef8eac8d-57a5-4d94-ab39-848e2a7bfeef 2023-11-28T20:32:46.971 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = pacific 2023-11-28T20:32:46.971 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 0 2023-11-28T20:32:46.971 INFO:teuthology.orchestra.run.smithi045.stdout:fsid ef8eac8d-57a5-4d94-ab39-848e2a7bfeef 2023-11-28T20:32:46.971 INFO:teuthology.orchestra.run.smithi045.stdout:last_changed 2023-11-28T20:32:46.970356+0000 2023-11-28T20:32:46.971 INFO:teuthology.orchestra.run.smithi045.stdout:created 2023-11-28T20:32:46.970356+0000 2023-11-28T20:32:46.971 INFO:teuthology.orchestra.run.smithi045.stdout:min_mon_release 16 (pacific) 2023-11-28T20:32:46.971 INFO:teuthology.orchestra.run.smithi045.stdout:election_strategy: 1 2023-11-28T20:32:46.972 INFO:teuthology.orchestra.run.smithi045.stdout:0: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.a 2023-11-28T20:32:46.972 INFO:teuthology.orchestra.run.smithi045.stdout:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2023-11-28T20:32:46.972 INFO:teuthology.orchestra.run.smithi045.stdout:2: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.c 2023-11-28T20:32:46.972 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-28T20:32:46.974 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T20:32:47.029 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ef8eac8d-57a5-4d94-ab39-848e2a7bfeef... 2023-11-28T20:32:47.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T20:32:47.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T20:32:47.213 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2023-11-28T20:32:47.214 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.215 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.216 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = "172.21.15.45,172.21.15.190,[v2:172.21.15.190:3301,v1:172.21.15.190:6790]" 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on skipped message = False 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = ef8eac8d-57a5-4d94-ab39-848e2a7bfeef 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.217 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 600 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock profile = high_recovery_ops 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2023-11-28T20:32:47.218 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug trim objects = true 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.219 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore block size = 96636764160 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore fsck on mount = True 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluefs = 20 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluestore = 20 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: debug rocksdb = 10 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T20:32:47.220 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd full ratio = 0.9 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd nearfull ratio = 0.8 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: osd failsafe full ratio = 0.95 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: osd objectstore = bluestore 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: osd op complaint time = 180 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2023-11-28T20:32:47.221 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: debug client = 20 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: debug finisher = 20 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2023-11-28T20:32:47.222 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.223 INFO:teuthology.orchestra.run.smithi045.stdout: # 1m isn't quite enough 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: mon_down_mkfs_grace = 2m 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_filestore_osds = false 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: mon op complaint time = 120 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T20:32:47.224 INFO:teuthology.orchestra.run.smithi045.stdout: client mount timeout = 600 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout: debug client = 20 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = "" 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout: rados mon op timeout = 900 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout: rados osd op timeout = 900 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.b] 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.c] 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.a] 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.c] 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.b] 2023-11-28T20:32:47.225 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.d] 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout:[mds] 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout: debug mds = 20 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout: debug mds balancer = 20 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout: mds debug frag = True 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout: mds debug scatterstat = True 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout: mds op complaint time = 180 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout: mds verify scatter = True 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout: osd op complaint time = 180 2023-11-28T20:32:47.226 INFO:teuthology.orchestra.run.smithi045.stdout: rados mon op timeout = 900 2023-11-28T20:32:47.227 INFO:teuthology.orchestra.run.smithi045.stdout: rados osd op timeout = 900 2023-11-28T20:32:47.256 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2023-11-28T20:32:47.256 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2023-11-28T20:32:47.256 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T20:32:47.256 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2023-11-28T20:32:47.256 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.256 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2023-11-28T20:32:47.256 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T20:32:47.257 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T20:32:47.258 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = "172.21.15.45,172.21.15.190,[v2:172.21.15.190:3301,v1:172.21.15.190:6790]" 2023-11-28T20:32:47.259 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on skipped message = False 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = ef8eac8d-57a5-4d94-ab39-848e2a7bfeef 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.260 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.261 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 20 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 20 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 10 2023-11-28T20:32:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: osd op complaint time = 180 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2023-11-28T20:32:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: debug client = 20 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: debug finisher = 20 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2023-11-28T20:32:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2023-11-28T20:32:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: mon op complaint time = 120 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T20:32:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: client mount timeout = 600 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout: debug client = 20 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = "" 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout: rados mon op timeout = 900 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout: rados osd op timeout = 900 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.b] 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.c] 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.a] 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.c] 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.b] 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.d] 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout:[mds] 2023-11-28T20:32:47.267 INFO:teuthology.orchestra.run.smithi190.stdout: debug mds = 20 2023-11-28T20:32:47.268 INFO:teuthology.orchestra.run.smithi190.stdout: debug mds balancer = 20 2023-11-28T20:32:47.268 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-11-28T20:32:47.268 INFO:teuthology.orchestra.run.smithi190.stdout: mds debug frag = True 2023-11-28T20:32:47.268 INFO:teuthology.orchestra.run.smithi190.stdout: mds debug scatterstat = True 2023-11-28T20:32:47.268 INFO:teuthology.orchestra.run.smithi190.stdout: mds op complaint time = 180 2023-11-28T20:32:47.268 INFO:teuthology.orchestra.run.smithi190.stdout: mds verify scatter = True 2023-11-28T20:32:47.268 INFO:teuthology.orchestra.run.smithi190.stdout: osd op complaint time = 180 2023-11-28T20:32:47.268 INFO:teuthology.orchestra.run.smithi190.stdout: rados mon op timeout = 900 2023-11-28T20:32:47.268 INFO:teuthology.orchestra.run.smithi190.stdout: rados osd op timeout = 900 2023-11-28T20:32:47.302 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-28T20:32:47.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-28T20:32:47.464 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-28T20:32:47.464 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:47.464 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-28T20:32:47.522 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:47.522 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-28T20:32:47.577 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:32:47.577 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:47.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T20:32:47.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T20:32:47.741 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:32:47.741 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:32:47.845 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T20:32:47.845 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:32:47.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T20:32:47.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T20:32:47.990 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:32:47.990 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:32:48.088 INFO:tasks.ceph:Setting up mon nodes... 2023-11-28T20:32:48.088 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-28T20:32:48.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T20:32:48.292 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T20:32:48.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T20:32:48.500 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T20:32:48.512 INFO:tasks.ceph:Setting up mds nodes... 2023-11-28T20:32:48.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T20:32:48.713 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T20:32:48.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-11-28T20:32:48.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T20:32:49.018 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T20:32:49.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-11-28T20:32:49.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T20:32:49.319 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T20:32:49.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-11-28T20:32:49.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T20:32:49.600 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T20:32:49.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-11-28T20:32:49.693 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-28T20:32:49.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-28T20:32:49.837 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-28T20:32:49.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-11-28T20:32:50.051 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-11-28T20:32:50.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-28T20:32:50.223 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-28T20:32:50.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-11-28T20:32:50.382 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-11-28T20:32:50.441 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-28T20:32:50.441 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T20:32:50.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-28T20:32:50.551 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T20:32:50.551 INFO:tasks.ceph:role: osd.0 2023-11-28T20:32:50.551 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:32:50.551 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T20:32:50.652 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T20:32:50.652 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T20:32:50.652 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T20:32:50.653 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T20:32:50.653 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T20:32:50.653 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2023-11-28T20:32:50.653 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T20:32:50.653 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T20:32:50.653 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T20:32:50.653 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T20:32:50.686 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2023-11-28T20:32:50.692 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2023-11-28T20:32:50.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-28T20:32:50.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-28T20:32:50.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-28T20:32:50.966 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T20:32:50.966 INFO:tasks.ceph:role: osd.1 2023-11-28T20:32:50.966 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:32:50.966 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T20:32:51.060 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T20:32:51.060 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T20:32:51.061 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T20:32:51.061 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T20:32:51.061 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T20:32:51.061 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2023-11-28T20:32:51.061 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T20:32:51.061 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T20:32:51.061 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T20:32:51.061 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T20:32:51.095 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2023-11-28T20:32:51.101 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2023-11-28T20:32:51.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-28T20:32:51.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-28T20:32:51.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-28T20:32:51.413 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T20:32:51.414 INFO:tasks.ceph:role: osd.2 2023-11-28T20:32:51.414 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:32:51.414 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T20:32:51.510 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T20:32:51.510 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T20:32:51.510 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T20:32:51.510 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T20:32:51.510 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T20:32:51.510 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2023-11-28T20:32:51.510 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T20:32:51.510 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T20:32:51.511 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T20:32:51.511 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T20:32:51.545 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2023-11-28T20:32:51.550 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2023-11-28T20:32:51.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-28T20:32:51.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-28T20:32:51.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-28T20:32:51.865 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T20:32:51.865 INFO:tasks.ceph:role: osd.3 2023-11-28T20:32:51.865 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:32:51.865 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T20:32:51.966 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T20:32:51.967 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T20:32:51.967 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T20:32:51.967 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T20:32:51.967 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T20:32:51.967 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2023-11-28T20:32:51.967 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T20:32:51.967 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T20:32:51.967 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T20:32:51.967 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T20:32:52.001 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2023-11-28T20:32:52.007 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2023-11-28T20:32:52.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-28T20:32:52.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-28T20:32:52.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:32:52.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:52.377+0000 7f48b6bad500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-28T20:32:52.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:52.378+0000 7f48b6bad500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-28T20:32:52.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:52.378+0000 7f48b6bad500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T20:32:52.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:52.378+0000 7f48b6bad500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T20:32:52.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:52.378+0000 7f48b6bad500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T20:32:52.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:52.382+0000 7f48b6bad500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-28T20:32:55.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-28T20:32:55.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:32:55.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:55.805+0000 7f79ca249500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-28T20:32:55.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:55.806+0000 7f79ca249500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-28T20:32:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:55.806+0000 7f79ca249500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T20:32:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:55.806+0000 7f79ca249500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T20:32:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:55.806+0000 7f79ca249500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T20:32:55.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:55.809+0000 7f79ca249500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-28T20:32:58.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-28T20:32:59.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:32:59.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:59.236+0000 7f625200c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-28T20:32:59.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:59.236+0000 7f625200c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-28T20:32:59.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:59.237+0000 7f625200c500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T20:32:59.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:59.237+0000 7f625200c500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T20:32:59.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:59.237+0000 7f625200c500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T20:32:59.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:32:59.239+0000 7f625200c500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-28T20:33:02.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-28T20:33:02.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:33:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:02.678+0000 7f3418414500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-28T20:33:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:02.678+0000 7f3418414500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-28T20:33:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:02.678+0000 7f3418414500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T20:33:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:02.678+0000 7f3418414500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T20:33:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:02.678+0000 7f3418414500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T20:33:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:02.681+0000 7f3418414500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-28T20:33:05.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-28T20:33:05.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-28T20:33:06.063 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T20:33:06.064 INFO:tasks.ceph:role: osd.4 2023-11-28T20:33:06.064 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T20:33:06.064 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T20:33:06.167 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T20:33:06.167 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T20:33:06.167 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T20:33:06.169 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T20:33:06.170 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T20:33:06.170 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-11-28T20:33:06.170 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T20:33:06.170 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T20:33:06.170 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T20:33:06.170 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T20:33:06.688 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-11-28T20:33:06.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-11-28T20:33:06.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-28T20:33:06.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-28T20:33:06.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-28T20:33:07.057 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T20:33:07.057 INFO:tasks.ceph:role: osd.5 2023-11-28T20:33:07.057 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T20:33:07.057 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T20:33:07.152 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T20:33:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T20:33:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T20:33:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T20:33:07.152 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T20:33:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-11-28T20:33:07.152 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T20:33:07.153 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T20:33:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T20:33:07.153 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T20:33:07.625 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-11-28T20:33:07.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-11-28T20:33:07.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-28T20:33:07.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-28T20:33:07.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-28T20:33:07.961 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T20:33:07.961 INFO:tasks.ceph:role: osd.6 2023-11-28T20:33:07.962 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T20:33:07.962 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T20:33:08.058 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T20:33:08.058 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T20:33:08.059 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T20:33:08.059 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T20:33:08.059 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T20:33:08.059 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-11-28T20:33:08.059 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T20:33:08.059 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T20:33:08.060 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T20:33:08.060 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T20:33:08.570 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-11-28T20:33:08.579 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-11-28T20:33:08.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-28T20:33:08.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-28T20:33:08.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-28T20:33:08.963 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T20:33:08.963 INFO:tasks.ceph:role: osd.7 2023-11-28T20:33:08.963 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T20:33:08.963 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T20:33:09.064 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T20:33:09.065 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T20:33:09.065 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T20:33:09.065 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T20:33:09.065 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T20:33:09.065 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-11-28T20:33:09.065 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T20:33:09.066 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T20:33:09.066 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T20:33:09.066 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T20:33:09.520 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-11-28T20:33:09.527 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-11-28T20:33:09.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-28T20:33:09.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-28T20:33:09.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:33:09.905 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:09.903+0000 7f176001d500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-28T20:33:09.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:09.909+0000 7f176001d500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-28T20:33:09.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:09.909+0000 7f176001d500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T20:33:09.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:09.909+0000 7f176001d500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T20:33:09.912 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:09.910+0000 7f176001d500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T20:33:09.916 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:09.914+0000 7f176001d500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-28T20:33:12.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-28T20:33:12.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:33:13.080 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:13.078+0000 7fac5956c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-28T20:33:13.081 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:13.078+0000 7fac5956c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-28T20:33:13.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:13.078+0000 7fac5956c500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T20:33:13.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:13.078+0000 7fac5956c500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T20:33:13.082 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:13.078+0000 7fac5956c500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T20:33:13.083 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:13.081+0000 7fac5956c500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-28T20:33:14.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:33:14.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:33:16.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-28T20:33:16.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:33:16.487 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:16.485+0000 7f8cdb11c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-28T20:33:16.488 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:16.486+0000 7f8cdb11c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-28T20:33:16.488 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:16.486+0000 7f8cdb11c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T20:33:16.488 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:16.486+0000 7f8cdb11c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T20:33:16.489 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:16.486+0000 7f8cdb11c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T20:33:16.490 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:16.488+0000 7f8cdb11c500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-28T20:33:19.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-28T20:33:19.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:33:19.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:19.900+0000 7fb36fc14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-28T20:33:19.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:19.900+0000 7fb36fc14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-28T20:33:19.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:19.900+0000 7fb36fc14500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T20:33:19.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:19.900+0000 7fb36fc14500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T20:33:19.906 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:19.900+0000 7fb36fc14500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T20:33:19.907 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:19.902+0000 7fb36fc14500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-28T20:33:22.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-28T20:33:22.958 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-28T20:33:22.958 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:22.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-28T20:33:23.069 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:23.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-11-28T20:33:23.174 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:23.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-11-28T20:33:23.275 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:23.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-28T20:33:23.385 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:23.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-28T20:33:23.494 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:23.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-28T20:33:23.602 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:23.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-28T20:33:23.708 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:23.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-28T20:33:23.813 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:28.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-11-28T20:33:28.933 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:28.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-11-28T20:33:29.016 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:29.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-28T20:33:29.159 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:29.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-28T20:33:29.250 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:29.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-28T20:33:29.342 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:29.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-28T20:33:29.435 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:29.435 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-28T20:33:29.497 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:29.497 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-11-28T20:33:29.554 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:29.554 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-28T20:33:29.614 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:29.614 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-11-28T20:33:29.668 INFO:tasks.ceph:Adding keys to all mons... 2023-11-28T20:33:29.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T20:33:29.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T20:33:29.771 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.y] 2023-11-28T20:33:29.821 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w== 2023-11-28T20:33:29.821 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.a] 2023-11-28T20:33:29.821 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w== 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.c] 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw== 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA== 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q== 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw== 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.3] 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng== 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.x] 2023-11-28T20:33:29.822 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w== 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.b] 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ== 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.d] 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw== 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.4] 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg== 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.5] 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q== 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.6] 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw== 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.7] 2023-11-28T20:33:29.823 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ== 2023-11-28T20:33:29.824 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T20:33:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:33:29.824 INFO:teuthology.orchestra.run.smithi045.stdout:[client.2] 2023-11-28T20:33:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA== 2023-11-28T20:33:29.824 INFO:teuthology.orchestra.run.smithi045.stdout:[client.1] 2023-11-28T20:33:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw== 2023-11-28T20:33:29.824 INFO:teuthology.orchestra.run.smithi045.stdout:[client.3] 2023-11-28T20:33:29.824 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg== 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.y] 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w== 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.a] 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w== 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.c] 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw== 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA== 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2023-11-28T20:33:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q== 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw== 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.3] 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng== 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w== 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.b] 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ== 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.d] 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw== 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.4] 2023-11-28T20:33:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg== 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.5] 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q== 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.6] 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw== 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.7] 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ== 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout:[client.2] 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA== 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2023-11-28T20:33:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw== 2023-11-28T20:33:29.829 INFO:teuthology.orchestra.run.smithi190.stdout:[client.3] 2023-11-28T20:33:29.829 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg== 2023-11-28T20:33:29.829 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T20:33:29.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T20:33:29.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T20:33:30.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T20:33:30.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T20:33:30.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T20:33:30.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:30.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:30.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:30.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:30.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:30.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:30.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:30.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:30.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T20:33:30.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T20:33:31.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T20:33:31.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T20:33:31.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T20:33:31.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T20:33:31.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:31.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:31.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:31.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:31.668 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:31.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:31.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:31.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T20:33:31.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T20:33:31.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T20:33:32.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T20:33:32.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T20:33:32.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T20:33:32.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T20:33:32.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T20:33:32.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T20:33:32.591 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-28T20:33:32.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-28T20:33:32.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T20:33:32.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-28T20:33:32.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-28T20:33:33.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T20:33:33.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-28T20:33:33.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-28T20:33:33.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T20:33:33.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-28T20:33:33.640 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:33:33.649 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T20:33:33.706 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-28T20:33:33.707 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T20:33:33.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T20:33:33.713 INFO:tasks.ceph.mon.a:Started 2023-11-28T20:33:33.713 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-28T20:33:33.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-28T20:33:33.718 INFO:tasks.ceph.mon.b:Started 2023-11-28T20:33:33.718 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-28T20:33:33.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-28T20:33:33.723 INFO:tasks.ceph.mon.c:Started 2023-11-28T20:33:33.723 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-28T20:33:33.724 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-28T20:33:33.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-28T20:33:33.730 INFO:tasks.ceph.mgr.y:Started 2023-11-28T20:33:33.730 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-28T20:33:33.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-28T20:33:33.735 INFO:tasks.ceph.mgr.x:Started 2023-11-28T20:33:33.735 INFO:tasks.ceph:Setting crush tunables to default 2023-11-28T20:33:33.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd crush tunables default 2023-11-28T20:33:33.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.918+0000 7fafd6e64640 1 Processor -- start 2023-11-28T20:33:33.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.918+0000 7fafd6e64640 1 -- start start 2023-11-28T20:33:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.919+0000 7fafd6e64640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fafd015d7a0 0x7fafd00cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:33.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.919+0000 7fafd6e64640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00cf500 con 0x7fafd0066900 2023-11-28T20:33:33.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.919+0000 7fafd6e64640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00cf7e0 con 0x7fafd006d620 2023-11-28T20:33:33.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.919+0000 7fafd6e64640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd00cfda0 con 0x7fafd015d7a0 2023-11-28T20:33:33.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.920+0000 7fafd5401640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fafd0066900 0x7fafd006abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58796/0 (socket says 172.21.15.45:58796) 2023-11-28T20:33:33.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.920+0000 7fafd5401640 1 -- 172.21.15.45:0/4227683998 learned_addr learned my addr 172.21.15.45:0/4227683998 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:33:33.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.920+0000 7fafd4c00640 1 --2- 172.21.15.45:0/4227683998 >> v2:172.21.15.190:3300/0 conn(0x7fafd015d7a0 0x7fafd00cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:33.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.920+0000 7fafd4c00640 1 -- 172.21.15.45:0/4227683998 >> v2:172.21.15.190:3300/0 conn(0x7fafd015d7a0 msgr2=0x7fafd00cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T20:33:33.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.920+0000 7fafd4c00640 1 -- 172.21.15.45:0/4227683998 >> v2:172.21.15.190:3300/0 conn(0x7fafd015d7a0 msgr2=0x7fafd00cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T20:33:33.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.920+0000 7fafd4c00640 1 --2- 172.21.15.45:0/4227683998 >> v2:172.21.15.190:3300/0 conn(0x7fafd015d7a0 0x7fafd00cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T20:33:33.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.920+0000 7fafd4c00640 1 --2- 172.21.15.45:0/4227683998 >> v2:172.21.15.190:3300/0 conn(0x7fafd015d7a0 0x7fafd00cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:33:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.936+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856748778 0 0) 0x7fafd00cf7e0 con 0x7fafd006d620 2023-11-28T20:33:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.936+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc0003520 con 0x7fafd006d620 2023-11-28T20:33:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.936+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445056953 0 0) 0x7fafd00cf500 con 0x7fafd0066900 2023-11-28T20:33:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.936+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd00cf7e0 con 0x7fafd0066900 2023-11-28T20:33:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635024493 0 0) 0x7fafd00cf7e0 con 0x7fafd0066900 2023-11-28T20:33:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 >> v2:172.21.15.190:3300/0 conn(0x7fafd015d7a0 msgr2=0x7fafd00cd4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T20:33:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 --2- 172.21.15.45:0/4227683998 >> v2:172.21.15.190:3300/0 conn(0x7fafd015d7a0 0x7fafd00cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 >> v1:172.21.15.190:6789/0 conn(0x7fafd006d620 legacy=0x7fafd00661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00cfac0 con 0x7fafd0066900 2023-11-28T20:33:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fafc80019f0 con 0x7fafd0066900 2023-11-28T20:33:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 >> v1:172.21.15.45:6789/0 conn(0x7fafd0066900 legacy=0x7fafd006abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0003b00 0x7fafc0003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafc00047c0 0x7fafc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc0005180 0x7fafc000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00cfda0 con 0x7fafc0005180 2023-11-28T20:33:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc000c000 con 0x7fafc00047c0 2023-11-28T20:33:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc000c1c0 con 0x7fafc0003b00 2023-11-28T20:33:33.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafd5401640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc0005180 0x7fafc000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:33.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafd4c00640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0003b00 0x7fafc0003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:33.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafcffff640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafc00047c0 0x7fafc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:33.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafd5401640 1 -- 172.21.15.45:0/4227683998 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0003b00 msgr2=0x7fafc0003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:33.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafd5401640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0003b00 0x7fafc0003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:33.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafd5401640 1 -- 172.21.15.45:0/4227683998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafc00047c0 msgr2=0x7fafc0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:33.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafd5401640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafc00047c0 0x7fafc0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:33.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.937+0000 7fafd5401640 1 -- 172.21.15.45:0/4227683998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00cf7e0 con 0x7fafc0005180 2023-11-28T20:33:33.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.938+0000 7fafd5401640 1 -- 172.21.15.45:0/4227683998 reap_dead start 2023-11-28T20:33:33.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.939+0000 7fafd5401640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc0005180 0x7fafc000b980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fafd00712e0 tx=0x7fafc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba9896380526f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:33.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.940+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8034630 con 0x7fafc0005180 2023-11-28T20:33:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.940+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc802f070 con 0x7fafc0005180 2023-11-28T20:33:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.940+0000 7fafcf7fe640 1 -- 172.21.15.45:0/4227683998 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc80349d0 con 0x7fafc0005180 2023-11-28T20:33:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.941+0000 7fafd6e64640 1 -- 172.21.15.45:0/4227683998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc0005180 msgr2=0x7fafc000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.941+0000 7fafd6e64640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc0005180 0x7fafc000b980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fafd00712e0 tx=0x7fafc00057f0 comp rx=0 tx=0).stop 2023-11-28T20:33:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.941+0000 7fafd6e64640 1 -- 172.21.15.45:0/4227683998 shutdown_connections 2023-11-28T20:33:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.941+0000 7fafd6e64640 1 --2- 172.21.15.45:0/4227683998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc0005180 0x7fafc000b980 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.941+0000 7fafd6e64640 1 -- 172.21.15.45:0/4227683998 >> 172.21.15.45:0/4227683998 conn(0x7fafd00cc9c0 msgr2=0x7fafd00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.941+0000 7fafd6e64640 1 -- 172.21.15.45:0/4227683998 shutdown_connections 2023-11-28T20:33:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.941+0000 7fafd6e64640 1 -- 172.21.15.45:0/4227683998 wait complete. 2023-11-28T20:33:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.943+0000 7fafd6e64640 1 Processor -- start 2023-11-28T20:33:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.943+0000 7fafd6e64640 1 -- start start 2023-11-28T20:33:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.944+0000 7fafd6e64640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0005180 0x7fafd00c0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.944+0000 7fafd6e64640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd009ed20 0x7fafd009f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.944+0000 7fafd6e64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd00c91f0 0x7fafd00c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.944+0000 7fafd6e64640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd0157580 con 0x7fafd00c91f0 2023-11-28T20:33:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.944+0000 7fafd6e64640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00bde30 con 0x7fafd009ed20 2023-11-28T20:33:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.944+0000 7fafd6e64640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00c9b90 con 0x7fafc0005180 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd5401640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd00c91f0 0x7fafd00c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd4c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0005180 0x7fafd00c0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd4c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0005180 0x7fafd00c0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:57052/0 (socket says 172.21.15.45:57052) 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd4c00640 1 -- 172.21.15.45:0/340030324 learned_addr learned my addr 172.21.15.45:0/340030324 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd5401640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd00c91f0 0x7fafd00c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47214/0 (socket says 172.21.15.45:47214) 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafcffff640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd009ed20 0x7fafd009f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd5401640 1 -- 172.21.15.45:0/340030324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0005180 msgr2=0x7fafd00c0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd5401640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0005180 0x7fafd00c0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd5401640 1 -- 172.21.15.45:0/340030324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd009ed20 msgr2=0x7fafd009f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd5401640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd009ed20 0x7fafd009f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:33.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd5401640 1 -- 172.21.15.45:0/340030324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00c7660 con 0x7fafd00c91f0 2023-11-28T20:33:33.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafcffff640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd009ed20 0x7fafd009f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:33:33.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.945+0000 7fafd5401640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd00c91f0 0x7fafd00c9650 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fafd00ca520 tx=0x7fafd00ce700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:33.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.946+0000 7fafcd7fa640 1 -- 172.21.15.45:0/340030324 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8034430 con 0x7fafd00c91f0 2023-11-28T20:33:33.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.946+0000 7fafcd7fa640 1 -- 172.21.15.45:0/340030324 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc8034590 con 0x7fafd00c91f0 2023-11-28T20:33:33.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.946+0000 7fafcd7fa640 1 -- 172.21.15.45:0/340030324 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8034920 con 0x7fafd00c91f0 2023-11-28T20:33:33.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.946+0000 7fafd6e64640 1 -- 172.21.15.45:0/340030324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd00a1170 con 0x7fafd00c91f0 2023-11-28T20:33:33.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.946+0000 7fafd6e64640 1 -- 172.21.15.45:0/340030324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd009fb70 con 0x7fafd00c91f0 2023-11-28T20:33:33.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.946+0000 7fafcd7fa640 1 -- 172.21.15.45:0/340030324 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fafc8031030 con 0x7fafd00c91f0 2023-11-28T20:33:33.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.946+0000 7fafcd7fa640 1 -- 172.21.15.45:0/340030324 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fafc8049ad0 con 0x7fafd00c91f0 2023-11-28T20:33:33.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.947+0000 7fafd6e64640 1 -- 172.21.15.45:0/340030324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd00c77f0 con 0x7fafd00c91f0 2023-11-28T20:33:33.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:33.950+0000 7fafcd7fa640 1 -- 172.21.15.45:0/340030324 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fafc800a3d0 con 0x7fafd00c91f0 2023-11-28T20:33:34.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.022+0000 7fafd6e64640 1 -- 172.21.15.45:0/340030324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fafd00c77f0 con 0x7fafd00c91f0 2023-11-28T20:33:34.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.026+0000 7fafcd7fa640 1 -- 172.21.15.45:0/340030324 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fafc8057090 con 0x7fafd00c91f0 2023-11-28T20:33:34.028 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2023-11-28T20:33:34.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.027+0000 7fafd6e64640 1 -- 172.21.15.45:0/340030324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd00c91f0 msgr2=0x7fafd00c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:34.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.027+0000 7fafd6e64640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd00c91f0 0x7fafd00c9650 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fafd00ca520 tx=0x7fafd00ce700 comp rx=0 tx=0).stop 2023-11-28T20:33:34.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.027+0000 7fafd6e64640 1 -- 172.21.15.45:0/340030324 shutdown_connections 2023-11-28T20:33:34.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.027+0000 7fafd6e64640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd00c91f0 0x7fafd00c9650 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:34.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.027+0000 7fafd6e64640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd009ed20 0x7fafd009f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:34.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.027+0000 7fafd6e64640 1 --2- 172.21.15.45:0/340030324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafc0005180 0x7fafd00c0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:34.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.027+0000 7fafd6e64640 1 -- 172.21.15.45:0/340030324 >> 172.21.15.45:0/340030324 conn(0x7fafd00cc9c0 msgr2=0x7fafd00cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:34.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.027+0000 7fafd6e64640 1 -- 172.21.15.45:0/340030324 shutdown_connections 2023-11-28T20:33:34.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:34.027+0000 7fafd6e64640 1 -- 172.21.15.45:0/340030324 wait complete. 2023-11-28T20:33:34.047 INFO:tasks.ceph:check_enable_crimson: False 2023-11-28T20:33:34.047 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-28T20:33:34.048 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:34.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-28T20:33:34.140 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:34.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-28T20:33:34.266 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:34.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-28T20:33:34.400 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:33:34.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-28T20:33:34.491 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:34.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-28T20:33:34.588 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:34.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-28T20:33:34.676 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:34.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-28T20:33:34.693 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:34.688+0000 7f529be10040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:33:34.693 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:34.686+0000 7fb587222040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:33:34.755 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:34.752+0000 7f529be10040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:33:34.755 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:34.753+0000 7fb587222040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:33:34.760 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:33:34.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-28T20:33:34.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 83d93ee1-0e46-4bac-b972-2e148e343c9d 0 2023-11-28T20:33:35.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.096+0000 7fc039a02640 1 Processor -- start 2023-11-28T20:33:35.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.096+0000 7fc039a02640 1 -- start start 2023-11-28T20:33:35.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.098+0000 7fc039a02640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc03415af30 0x7fc0340cb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.098+0000 7fc039a02640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0340cf6e0 con 0x7fc0340ae220 2023-11-28T20:33:35.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.098+0000 7fc039a02640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0340ae6b0 con 0x7fc03415b310 2023-11-28T20:33:35.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.098+0000 7fc039a02640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0340ae990 con 0x7fc03415af30 2023-11-28T20:33:35.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.098+0000 7fc033fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fc03415b310 0x7fc0340adb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:33604/0 (socket says 172.21.15.190:33604) 2023-11-28T20:33:35.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.098+0000 7fc033fff640 1 -- 172.21.15.190:0/674712653 learned_addr learned my addr 172.21.15.190:0/674712653 (peer_addr_for_me v1:172.21.15.190:0/0) 2023-11-28T20:33:35.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.098+0000 7fc038a00640 1 --2- 172.21.15.190:0/674712653 >> v2:172.21.15.45:3300/0 conn(0x7fc03415af30 0x7fc0340cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.098+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116910923 0 0) 0x7fc0340ae6b0 con 0x7fc03415b310 2023-11-28T20:33:35.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.098+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc024003520 con 0x7fc03415b310 2023-11-28T20:33:35.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720961156 0 0) 0x7fc0340cf6e0 con 0x7fc0340ae220 2023-11-28T20:33:35.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0340ae6b0 con 0x7fc0340ae220 2023-11-28T20:33:35.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 175039271 0 0) 0x7fc024003520 con 0x7fc03415b310 2023-11-28T20:33:35.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 >> v2:172.21.15.45:3300/0 conn(0x7fc03415af30 msgr2=0x7fc0340cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:35.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 --2- 172.21.15.190:0/674712653 >> v2:172.21.15.45:3300/0 conn(0x7fc03415af30 0x7fc0340cb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 >> v1:172.21.15.45:6789/0 conn(0x7fc0340ae220 legacy=0x7fc0340d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0340aec70 con 0x7fc03415b310 2023-11-28T20:33:35.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc020001670 con 0x7fc03415b310 2023-11-28T20:33:35.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 >> v1:172.21.15.190:6789/0 conn(0x7fc03415b310 legacy=0x7fc0340adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc024003b50 0x7fc024003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0240047c0 0x7fc024004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc024005180 0x7fc02400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0340ae990 con 0x7fc0240047c0 2023-11-28T20:33:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02400c050 con 0x7fc024003b50 2023-11-28T20:33:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02400c210 con 0x7fc024005180 2023-11-28T20:33:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc038a00640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc024003b50 0x7fc024003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.099+0000 7fc039201640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc024005180 0x7fc02400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.100+0000 7fc038a00640 1 -- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc024005180 msgr2=0x7fc02400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:35.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.100+0000 7fc038a00640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc024005180 0x7fc02400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.100+0000 7fc033fff640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0240047c0 0x7fc024004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.100+0000 7fc038a00640 1 -- 172.21.15.190:0/674712653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0240047c0 msgr2=0x7fc024004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.100+0000 7fc038a00640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0240047c0 0x7fc024004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.100+0000 7fc038a00640 1 -- 172.21.15.190:0/674712653 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024003520 con 0x7fc024003b50 2023-11-28T20:33:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.100+0000 7fc039201640 1 -- 172.21.15.190:0/674712653 reap_dead start 2023-11-28T20:33:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.100+0000 7fc033fff640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0240047c0 0x7fc024004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.101+0000 7fc038a00640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc024003b50 0x7fc024003f50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc028009ac0 tx=0x7fc0240057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b3d85d8db481fea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.101+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02802c6b0 con 0x7fc024003b50 2023-11-28T20:33:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.101+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02802c810 con 0x7fc024003b50 2023-11-28T20:33:35.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.101+0000 7fc0337fe640 1 -- 172.21.15.190:0/674712653 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02802cba0 con 0x7fc024003b50 2023-11-28T20:33:35.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.102+0000 7fc039a02640 1 -- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc024003b50 msgr2=0x7fc024003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.102+0000 7fc039a02640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc024003b50 0x7fc024003f50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc028009ac0 tx=0x7fc0240057f0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.102+0000 7fc039a02640 1 -- 172.21.15.190:0/674712653 shutdown_connections 2023-11-28T20:33:35.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.102+0000 7fc039a02640 1 --2- 172.21.15.190:0/674712653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc024003b50 0x7fc024003f50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.102+0000 7fc039a02640 1 -- 172.21.15.190:0/674712653 >> 172.21.15.190:0/674712653 conn(0x7fc034099a40 msgr2=0x7fc034099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:35.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.102+0000 7fc039a02640 1 -- 172.21.15.190:0/674712653 shutdown_connections 2023-11-28T20:33:35.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.102+0000 7fc039a02640 1 -- 172.21.15.190:0/674712653 wait complete. 2023-11-28T20:33:35.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.104+0000 7fc039a02640 1 Processor -- start 2023-11-28T20:33:35.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.104+0000 7fc039a02640 1 -- start start 2023-11-28T20:33:35.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc039a02640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc024003b50 0x7fc0340b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc039a02640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0340b65b0 0x7fc034157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc039a02640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0340b6a30 0x7fc034158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc039a02640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0340b4000 con 0x7fc024003b50 2023-11-28T20:33:35.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc039a02640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03415e070 con 0x7fc0340b6a30 2023-11-28T20:33:35.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc039a02640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03415e350 con 0x7fc0340b65b0 2023-11-28T20:33:35.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc033fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0340b65b0 0x7fc034157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.115 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc033fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0340b65b0 0x7fc034157ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:49262/0 (socket says 172.21.15.190:49262) 2023-11-28T20:33:35.116 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc033fff640 1 -- 172.21.15.190:0/1677408165 learned_addr learned my addr 172.21.15.190:0/1677408165 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:35.116 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc039201640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0340b6a30 0x7fc034158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.116 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc038a00640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc024003b50 0x7fc0340b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.116 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc033fff640 1 -- 172.21.15.190:0/1677408165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0340b6a30 msgr2=0x7fc034158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.116 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc033fff640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0340b6a30 0x7fc034158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.116 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc033fff640 1 -- 172.21.15.190:0/1677408165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc024003b50 msgr2=0x7fc0340b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.116 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc033fff640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc024003b50 0x7fc0340b5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.117 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc033fff640 1 -- 172.21.15.190:0/1677408165 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03415e630 con 0x7fc0340b65b0 2023-11-28T20:33:35.117 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.105+0000 7fc039201640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0340b6a30 0x7fc034158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:33:35.117 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.106+0000 7fc033fff640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0340b65b0 0x7fc034157ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc02000d770 tx=0x7fc02000db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:35.117 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.106+0000 7fc0317fa640 1 -- 172.21.15.190:0/1677408165 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0200072d0 con 0x7fc0340b65b0 2023-11-28T20:33:35.117 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.106+0000 7fc0317fa640 1 -- 172.21.15.190:0/1677408165 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc020007430 con 0x7fc0340b65b0 2023-11-28T20:33:35.117 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.106+0000 7fc0317fa640 1 -- 172.21.15.190:0/1677408165 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0200047e0 con 0x7fc0340b65b0 2023-11-28T20:33:35.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.106+0000 7fc039a02640 1 -- 172.21.15.190:0/1677408165 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0340b3c90 con 0x7fc0340b65b0 2023-11-28T20:33:35.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.106+0000 7fc039a02640 1 -- 172.21.15.190:0/1677408165 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03415dc90 con 0x7fc0340b65b0 2023-11-28T20:33:35.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.106+0000 7fc0317fa640 1 -- 172.21.15.190:0/1677408165 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc020004940 con 0x7fc0340b65b0 2023-11-28T20:33:35.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.106+0000 7fc0317fa640 1 -- 172.21.15.190:0/1677408165 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc02000bb60 con 0x7fc0340b65b0 2023-11-28T20:33:35.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.107+0000 7fc039a02640 1 -- 172.21.15.190:0/1677408165 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03404e920 con 0x7fc0340b65b0 2023-11-28T20:33:35.118 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.110+0000 7fc0317fa640 1 -- 172.21.15.190:0/1677408165 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fc020019070 con 0x7fc0340b65b0 2023-11-28T20:33:35.213 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.211+0000 7fc039a02640 1 -- 172.21.15.190:0/1677408165 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "83d93ee1-0e46-4bac-b972-2e148e343c9d", "id": 0} v 0) v1 -- 0x7fc0340b0290 con 0x7fc0340b65b0 2023-11-28T20:33:35.218 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.217+0000 7fc0317fa640 1 -- 172.21.15.190:0/1677408165 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "83d93ee1-0e46-4bac-b972-2e148e343c9d", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc020004e20 con 0x7fc0340b65b0 2023-11-28T20:33:35.218 INFO:teuthology.orchestra.run.smithi190.stdout:0 2023-11-28T20:33:35.220 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.218+0000 7fc039a02640 1 -- 172.21.15.190:0/1677408165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0340b65b0 msgr2=0x7fc034157ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.221 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.218+0000 7fc039a02640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0340b65b0 0x7fc034157ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc02000d770 tx=0x7fc02000db70 comp rx=0 tx=0).stop 2023-11-28T20:33:35.221 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.218+0000 7fc039a02640 1 -- 172.21.15.190:0/1677408165 shutdown_connections 2023-11-28T20:33:35.221 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.218+0000 7fc039a02640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0340b6a30 0x7fc034158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.221 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.218+0000 7fc039a02640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0340b65b0 0x7fc034157ef0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.221 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.218+0000 7fc039a02640 1 --2- 172.21.15.190:0/1677408165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc024003b50 0x7fc0340b5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.222 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.218+0000 7fc039a02640 1 -- 172.21.15.190:0/1677408165 >> 172.21.15.190:0/1677408165 conn(0x7fc034099a40 msgr2=0x7fc0340af510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:35.222 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.219+0000 7fc039a02640 1 -- 172.21.15.190:0/1677408165 shutdown_connections 2023-11-28T20:33:35.222 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.219+0000 7fc039a02640 1 -- 172.21.15.190:0/1677408165 wait complete. 2023-11-28T20:33:35.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 4eaa39fb-6b40-4a4b-ad83-0a3427a81b13 1 2023-11-28T20:33:35.397 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:35.395+0000 7f529be10040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:33:35.418 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:35.416+0000 7fb587222040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:33:35.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.432+0000 7f09d6210640 1 Processor -- start 2023-11-28T20:33:35.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.433+0000 7f09d6210640 1 -- start start 2023-11-28T20:33:35.436 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.434+0000 7f09d6210640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f09d015d420 0x7f09d0161a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.436 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.434+0000 7f09d6210640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d00d0240 0x7f09d00d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.436 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.434+0000 7f09d6210640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d0161fd0 con 0x7f09d00cda70 2023-11-28T20:33:35.436 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.434+0000 7f09d6210640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00cdf00 con 0x7f09d015d420 2023-11-28T20:33:35.436 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.434+0000 7f09d6210640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00ce1e0 con 0x7f09d00d0240 2023-11-28T20:33:35.437 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.434+0000 7f09cf7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f09d015d420 0x7f09d0161a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.437 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.434+0000 7f09cf7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f09d015d420 0x7f09d0161a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:36034/0 (socket says 172.21.15.190:36034) 2023-11-28T20:33:35.437 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.434+0000 7f09cf7fe640 1 -- 172.21.15.190:0/3641487964 learned_addr learned my addr 172.21.15.190:0/3641487964 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:35.437 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.435+0000 7f09cffff640 1 --2- 172.21.15.190:0/3641487964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d00d0240 0x7f09d00d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.437 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.435+0000 7f09cf7fe640 1 -- 172.21.15.190:0/3641487964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d00d0240 msgr2=0x7f09d00d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:35.437 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.435+0000 7f09cf7fe640 1 --2- 172.21.15.190:0/3641487964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d00d0240 0x7f09d00d1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.437 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.435+0000 7f09cf7fe640 1 -- 172.21.15.190:0/3641487964 >> v1:172.21.15.190:6789/0 conn(0x7f09d00cda70 legacy=0x7f09d00cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.437 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.435+0000 7f09cf7fe640 1 -- 172.21.15.190:0/3641487964 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00ce4c0 con 0x7f09d015d420 2023-11-28T20:33:35.438 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.436+0000 7f09cf7fe640 1 --2- 172.21.15.190:0/3641487964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09d015d420 0x7f09d0161a90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009870 tx=0x7f09c002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6273044550a1b66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:35.439 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.436+0000 7f09ceffd640 1 -- 172.21.15.190:0/3641487964 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0030840 con 0x7f09d015d420 2023-11-28T20:33:35.439 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.436+0000 7f09ceffd640 1 -- 172.21.15.190:0/3641487964 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c00309a0 con 0x7f09d015d420 2023-11-28T20:33:35.439 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.436+0000 7f09ceffd640 1 -- 172.21.15.190:0/3641487964 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0030d30 con 0x7f09d015d420 2023-11-28T20:33:35.439 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.437+0000 7f09d6210640 1 -- 172.21.15.190:0/3641487964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09d015d420 msgr2=0x7f09d0161a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.439 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.437+0000 7f09d6210640 1 --2- 172.21.15.190:0/3641487964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09d015d420 0x7f09d0161a90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009870 tx=0x7f09c002b980 comp rx=0 tx=0).stop 2023-11-28T20:33:35.439 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.437+0000 7f09d6210640 1 -- 172.21.15.190:0/3641487964 shutdown_connections 2023-11-28T20:33:35.439 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.437+0000 7f09d6210640 1 --2- 172.21.15.190:0/3641487964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d00d0240 0x7f09d00d1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.439 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.437+0000 7f09d6210640 1 --2- 172.21.15.190:0/3641487964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09d015d420 0x7f09d0161a90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.440 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.437+0000 7f09d6210640 1 -- 172.21.15.190:0/3641487964 >> 172.21.15.190:0/3641487964 conn(0x7f09d0070b20 msgr2=0x7f09d0070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:35.440 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.437+0000 7f09d6210640 1 -- 172.21.15.190:0/3641487964 shutdown_connections 2023-11-28T20:33:35.440 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.437+0000 7f09d6210640 1 -- 172.21.15.190:0/3641487964 wait complete. 2023-11-28T20:33:35.440 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.439+0000 7f09d6210640 1 Processor -- start 2023-11-28T20:33:35.440 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.439+0000 7f09d6210640 1 -- start start 2023-11-28T20:33:35.442 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09d6210640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09d00d0240 0x7f09d015db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.443 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09d6210640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09d015d420 0x7f09d015e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.443 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09d6210640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d015fa60 0x7f09d015fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.443 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09d6210640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0160430 con 0x7f09d00d0240 2023-11-28T20:33:35.443 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09d6210640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01f2d80 con 0x7f09d015d420 2023-11-28T20:33:35.443 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09d6210640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d015f250 con 0x7f09d015fa60 2023-11-28T20:33:35.443 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09cffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d015fa60 0x7f09d015fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.443 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09c7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09d015d420 0x7f09d015e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09cffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d015fa60 0x7f09d015fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:49274/0 (socket says 172.21.15.190:49274) 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09cffff640 1 -- 172.21.15.190:0/2615356788 learned_addr learned my addr 172.21.15.190:0/2615356788 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.440+0000 7f09cf7fe640 1 --2- 172.21.15.190:0/2615356788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09d00d0240 0x7f09d015db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09cffff640 1 -- 172.21.15.190:0/2615356788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09d015d420 msgr2=0x7f09d015e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09cffff640 1 --2- 172.21.15.190:0/2615356788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09d015d420 0x7f09d015e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09cffff640 1 -- 172.21.15.190:0/2615356788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09d00d0240 msgr2=0x7f09d015db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09cffff640 1 --2- 172.21.15.190:0/2615356788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09d00d0240 0x7f09d015db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09cffff640 1 -- 172.21.15.190:0/2615356788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c00094d0 con 0x7f09d015fa60 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09cffff640 1 --2- 172.21.15.190:0/2615356788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d015fa60 0x7f09d015fef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f09c800efc0 tx=0x7f09c800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09ccff9640 1 -- 172.21.15.190:0/2615356788 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8009ca0 con 0x7f09d015fa60 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09ccff9640 1 -- 172.21.15.190:0/2615356788 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c800f040 con 0x7f09d015fa60 2023-11-28T20:33:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09d6210640 1 -- 172.21.15.190:0/2615356788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d0151620 con 0x7f09d015fa60 2023-11-28T20:33:35.445 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09ccff9640 1 -- 172.21.15.190:0/2615356788 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c80146c0 con 0x7f09d015fa60 2023-11-28T20:33:35.445 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.441+0000 7f09d6210640 1 -- 172.21.15.190:0/2615356788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d015e9b0 con 0x7f09d015fa60 2023-11-28T20:33:35.445 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.442+0000 7f09ccff9640 1 -- 172.21.15.190:0/2615356788 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f09c8014b20 con 0x7f09d015fa60 2023-11-28T20:33:35.445 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.442+0000 7f09ccff9640 1 -- 172.21.15.190:0/2615356788 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f09c8013790 con 0x7f09d015fa60 2023-11-28T20:33:35.445 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.442+0000 7f09d6210640 1 -- 172.21.15.190:0/2615356788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d00733d0 con 0x7f09d015fa60 2023-11-28T20:33:35.446 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.445+0000 7f09ccff9640 1 -- 172.21.15.190:0/2615356788 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f09c803bce0 con 0x7f09d015fa60 2023-11-28T20:33:35.461 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:35.460+0000 7f529be10040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:33:35.483 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:35.480+0000 7fb587222040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:33:35.514 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:35.513+0000 7f529be10040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:33:35.524 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.523+0000 7f09d6210640 1 -- 172.21.15.190:0/2615356788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4eaa39fb-6b40-4a4b-ad83-0a3427a81b13", "id": 1} v 0) v1 -- 0x7f09d015c440 con 0x7f09d015fa60 2023-11-28T20:33:35.529 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.528+0000 7f09ccff9640 1 -- 172.21.15.190:0/2615356788 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4eaa39fb-6b40-4a4b-ad83-0a3427a81b13", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f09c802a070 con 0x7f09d015fa60 2023-11-28T20:33:35.529 INFO:teuthology.orchestra.run.smithi190.stdout:1 2023-11-28T20:33:35.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.530+0000 7f09d6210640 1 -- 172.21.15.190:0/2615356788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d015fa60 msgr2=0x7f09d015fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.530+0000 7f09d6210640 1 --2- 172.21.15.190:0/2615356788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d015fa60 0x7f09d015fef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f09c800efc0 tx=0x7f09c800c490 comp rx=0 tx=0).stop 2023-11-28T20:33:35.531 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.530+0000 7f09d6210640 1 -- 172.21.15.190:0/2615356788 shutdown_connections 2023-11-28T20:33:35.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.530+0000 7f09d6210640 1 --2- 172.21.15.190:0/2615356788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09d015fa60 0x7f09d015fef0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.530+0000 7f09d6210640 1 --2- 172.21.15.190:0/2615356788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09d015d420 0x7f09d015e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.530+0000 7f09d6210640 1 --2- 172.21.15.190:0/2615356788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09d00d0240 0x7f09d015db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.530+0000 7f09d6210640 1 -- 172.21.15.190:0/2615356788 >> 172.21.15.190:0/2615356788 conn(0x7f09d0070b20 msgr2=0x7f09d00c6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:35.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.530+0000 7f09d6210640 1 -- 172.21.15.190:0/2615356788 shutdown_connections 2023-11-28T20:33:35.532 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.530+0000 7f09d6210640 1 -- 172.21.15.190:0/2615356788 wait complete. 2023-11-28T20:33:35.537 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:35.534+0000 7fb587222040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:33:35.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 92ec50eb-f647-4f17-9fd4-7437537ce096 2 2023-11-28T20:33:35.580 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:35.579+0000 7f529be10040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:33:35.603 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:35.600+0000 7fb587222040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:33:35.633 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:35.632+0000 7f529be10040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:33:35.654 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:35.652+0000 7fb587222040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:33:35.739 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.738+0000 7f9077fff640 1 Processor -- start 2023-11-28T20:33:35.739 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.738+0000 7f9077fff640 1 -- start start 2023-11-28T20:33:35.741 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.740+0000 7f9077fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f907815d230 0x7f90780b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.740+0000 7f9077fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f90780cddf0 0x7f90780ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.740+0000 7f9077fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90780ce740 con 0x7f90780b2640 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.740+0000 7f9077fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90780b90e0 con 0x7f907815d230 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.740+0000 7f9077fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90780d11d0 con 0x7f90780cddf0 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.740+0000 7f90777fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f90780cddf0 0x7f90780ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.740+0000 7f90777fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f90780cddf0 0x7f90780ce200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:36038/0 (socket says 172.21.15.190:36038) 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.740+0000 7f90777fe640 1 -- 172.21.15.190:0/1835912198 learned_addr learned my addr 172.21.15.190:0/1835912198 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.740+0000 7f9076ffd640 1 --2- 172.21.15.190:0/1835912198 >> v2:172.21.15.45:3300/0 conn(0x7f907815d230 0x7f90780b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.741+0000 7f90777fe640 1 -- 172.21.15.190:0/1835912198 >> v2:172.21.15.45:3300/0 conn(0x7f907815d230 msgr2=0x7f90780b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.741+0000 7f90777fe640 1 --2- 172.21.15.190:0/1835912198 >> v2:172.21.15.45:3300/0 conn(0x7f907815d230 0x7f90780b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.742 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.741+0000 7f90777fe640 1 -- 172.21.15.190:0/1835912198 >> v1:172.21.15.190:6789/0 conn(0x7f90780b2640 legacy=0x7f90780b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.743 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.741+0000 7f90777fe640 1 -- 172.21.15.190:0/1835912198 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90780d1490 con 0x7f90780cddf0 2023-11-28T20:33:35.743 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.742+0000 7f90777fe640 1 --2- 172.21.15.190:0/1835912198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 0x7f90780ce200 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f90780b5c00 tx=0x7f907002c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b513e0dd83f747f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:35.744 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9075ffb640 1 -- 172.21.15.190:0/1835912198 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907002cbb0 con 0x7f90780cddf0 2023-11-28T20:33:35.744 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9075ffb640 1 -- 172.21.15.190:0/1835912198 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907002cd10 con 0x7f90780cddf0 2023-11-28T20:33:35.744 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9075ffb640 1 -- 172.21.15.190:0/1835912198 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070034610 con 0x7f90780cddf0 2023-11-28T20:33:35.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9077fff640 1 -- 172.21.15.190:0/1835912198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 msgr2=0x7f90780ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9077fff640 1 --2- 172.21.15.190:0/1835912198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 0x7f90780ce200 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f90780b5c00 tx=0x7f907002c010 comp rx=0 tx=0).stop 2023-11-28T20:33:35.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9077fff640 1 -- 172.21.15.190:0/1835912198 shutdown_connections 2023-11-28T20:33:35.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9077fff640 1 --2- 172.21.15.190:0/1835912198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 0x7f90780ce200 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9077fff640 1 --2- 172.21.15.190:0/1835912198 >> v2:172.21.15.45:3300/0 conn(0x7f907815d230 0x7f90780b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9077fff640 1 -- 172.21.15.190:0/1835912198 >> 172.21.15.190:0/1835912198 conn(0x7f9078070f40 msgr2=0x7f9078071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:35.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9077fff640 1 -- 172.21.15.190:0/1835912198 shutdown_connections 2023-11-28T20:33:35.745 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.743+0000 7f9077fff640 1 -- 172.21.15.190:0/1835912198 wait complete. 2023-11-28T20:33:35.746 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.745+0000 7f9077fff640 1 Processor -- start 2023-11-28T20:33:35.746 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.745+0000 7f9077fff640 1 -- start start 2023-11-28T20:33:35.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f9077fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f90780b2640 0x7f9078152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f9077fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 0x7f907815d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f9077fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907815d230 0x7f90781f6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:35.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f9077fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90781f29b0 con 0x7f907815d230 2023-11-28T20:33:35.749 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f9077fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9078152990 con 0x7f90780cddf0 2023-11-28T20:33:35.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f9077fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9078160840 con 0x7f90780b2640 2023-11-28T20:33:35.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f90767fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 0x7f907815d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f9076ffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f90780b2640 0x7f9078152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f90767fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 0x7f907815d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:36048/0 (socket says 172.21.15.190:36048) 2023-11-28T20:33:35.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.746+0000 7f90767fc640 1 -- 172.21.15.190:0/672392436 learned_addr learned my addr 172.21.15.190:0/672392436 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:35.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f90777fe640 1 --2- 172.21.15.190:0/672392436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907815d230 0x7f90781f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:35.750 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f90767fc640 1 -- 172.21.15.190:0/672392436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f90780b2640 msgr2=0x7f9078152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f90767fc640 1 --2- 172.21.15.190:0/672392436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f90780b2640 0x7f9078152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f90767fc640 1 -- 172.21.15.190:0/672392436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907815d230 msgr2=0x7f90781f6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f90767fc640 1 --2- 172.21.15.190:0/672392436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907815d230 0x7f90781f6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f90767fc640 1 -- 172.21.15.190:0/672392436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9070009b30 con 0x7f90780cddf0 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f90767fc640 1 --2- 172.21.15.190:0/672392436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 0x7f907815d820 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f906c00c400 tx=0x7f906c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f905ffff640 1 -- 172.21.15.190:0/672392436 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c0052d0 con 0x7f90780cddf0 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f905ffff640 1 -- 172.21.15.190:0/672392436 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c005430 con 0x7f90780cddf0 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f905ffff640 1 -- 172.21.15.190:0/672392436 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c003830 con 0x7f90780cddf0 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.747+0000 7f9077fff640 1 -- 172.21.15.190:0/672392436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90780cb810 con 0x7f90780cddf0 2023-11-28T20:33:35.751 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.748+0000 7f9077fff640 1 -- 172.21.15.190:0/672392436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907815ffa0 con 0x7f90780cddf0 2023-11-28T20:33:35.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.748+0000 7f905ffff640 1 -- 172.21.15.190:0/672392436 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f906c003c90 con 0x7f90780cddf0 2023-11-28T20:33:35.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.748+0000 7f905ffff640 1 -- 172.21.15.190:0/672392436 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f906c0119b0 con 0x7f90780cddf0 2023-11-28T20:33:35.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.749+0000 7f9077fff640 1 -- 172.21.15.190:0/672392436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90780cb9a0 con 0x7f90780cddf0 2023-11-28T20:33:35.752 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.751+0000 7f905ffff640 1 -- 172.21.15.190:0/672392436 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f906c017070 con 0x7f90780cddf0 2023-11-28T20:33:35.753 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:35.753+0000 7f529be10040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:33:35.774 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:35.771+0000 7fb587222040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:33:35.830 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.829+0000 7f9077fff640 1 -- 172.21.15.190:0/672392436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "92ec50eb-f647-4f17-9fd4-7437537ce096", "id": 2} v 0) v1 -- 0x7f9078071b30 con 0x7f90780cddf0 2023-11-28T20:33:35.837 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:35.836+0000 7f529be10040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:33:35.837 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.836+0000 7f905ffff640 1 -- 172.21.15.190:0/672392436 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "92ec50eb-f647-4f17-9fd4-7437537ce096", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f906c011590 con 0x7f90780cddf0 2023-11-28T20:33:35.838 INFO:teuthology.orchestra.run.smithi190.stdout:2 2023-11-28T20:33:35.839 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.837+0000 7f9077fff640 1 -- 172.21.15.190:0/672392436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 msgr2=0x7f907815d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:35.839 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.837+0000 7f9077fff640 1 --2- 172.21.15.190:0/672392436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 0x7f907815d820 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f906c00c400 tx=0x7f906c00c8c0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.838+0000 7f9077fff640 1 -- 172.21.15.190:0/672392436 shutdown_connections 2023-11-28T20:33:35.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.838+0000 7f9077fff640 1 --2- 172.21.15.190:0/672392436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f907815d230 0x7f90781f6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.838+0000 7f9077fff640 1 --2- 172.21.15.190:0/672392436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90780cddf0 0x7f907815d820 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.838+0000 7f9077fff640 1 --2- 172.21.15.190:0/672392436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f90780b2640 0x7f9078152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:35.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.838+0000 7f9077fff640 1 -- 172.21.15.190:0/672392436 >> 172.21.15.190:0/672392436 conn(0x7f9078070f40 msgr2=0x7f90780b6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:35.841 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.838+0000 7f9077fff640 1 -- 172.21.15.190:0/672392436 shutdown_connections 2023-11-28T20:33:35.841 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:35.838+0000 7f9077fff640 1 -- 172.21.15.190:0/672392436 wait complete. 2023-11-28T20:33:35.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 52955585-272d-45b1-839d-e89bad4ed76a 3 2023-11-28T20:33:35.858 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:35.856+0000 7fb587222040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:33:35.893 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:35.891+0000 7f529be10040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:33:35.914 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:35.912+0000 7fb587222040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:33:35.957 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:35.956+0000 7f529be10040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:33:35.978 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:35.976+0000 7fb587222040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:33:36.015 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:36.013+0000 7f529be10040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:33:36.036 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:36.033+0000 7fb587222040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:33:36.094 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.093+0000 7f7778bb5640 1 Processor -- start 2023-11-28T20:33:36.094 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.093+0000 7f7778bb5640 1 -- start start 2023-11-28T20:33:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.094+0000 7f7778bb5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f77740cf940 0x7f7774161800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.094+0000 7f7778bb5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cfd20 0x7f77740b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f7778bb5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77740b2930 con 0x7f77740b5070 2023-11-28T20:33:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f7778bb5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7774161d40 con 0x7f77740cf940 2023-11-28T20:33:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f7778bb5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77740b2650 con 0x7f77740cfd20 2023-11-28T20:33:36.096 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f77737fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f77740cf940 0x7f7774161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f77737fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f77740cf940 0x7f7774161800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:36054/0 (socket says 172.21.15.190:36054) 2023-11-28T20:33:36.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f776bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cfd20 0x7f77740b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.097 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f77737fe640 1 -- 172.21.15.190:0/515621588 learned_addr learned my addr 172.21.15.190:0/515621588 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:36.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f77737fe640 1 -- 172.21.15.190:0/515621588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cfd20 msgr2=0x7f77740b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:36.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f77737fe640 1 --2- 172.21.15.190:0/515621588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cfd20 0x7f77740b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f77737fe640 1 -- 172.21.15.190:0/515621588 >> v1:172.21.15.45:6789/0 conn(0x7f77740b5070 legacy=0x7f77740d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.098 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f77737fe640 1 -- 172.21.15.190:0/515621588 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77740ce240 con 0x7f77740cf940 2023-11-28T20:33:36.099 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.095+0000 7f776bfff640 1 --2- 172.21.15.190:0/515621588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cfd20 0x7f77740b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:33:36.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.097+0000 7f77737fe640 1 --2- 172.21.15.190:0/515621588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77740cf940 0x7f7774161800 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7764009870 tx=0x7f776402b7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47cde0301b3cae20 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:36.100 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.097+0000 7f7772ffd640 1 -- 172.21.15.190:0/515621588 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764033620 con 0x7f77740cf940 2023-11-28T20:33:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.097+0000 7f7772ffd640 1 -- 172.21.15.190:0/515621588 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7764033780 con 0x7f77740cf940 2023-11-28T20:33:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.097+0000 7f7772ffd640 1 -- 172.21.15.190:0/515621588 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764033b10 con 0x7f77740cf940 2023-11-28T20:33:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.098+0000 7f7778bb5640 1 -- 172.21.15.190:0/515621588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77740cf940 msgr2=0x7f7774161800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.098+0000 7f7778bb5640 1 --2- 172.21.15.190:0/515621588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77740cf940 0x7f7774161800 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7764009870 tx=0x7f776402b7d0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.098+0000 7f7778bb5640 1 -- 172.21.15.190:0/515621588 shutdown_connections 2023-11-28T20:33:36.101 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.098+0000 7f7778bb5640 1 --2- 172.21.15.190:0/515621588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cfd20 0x7f77740b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.098+0000 7f7778bb5640 1 --2- 172.21.15.190:0/515621588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77740cf940 0x7f7774161800 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.098+0000 7f7778bb5640 1 -- 172.21.15.190:0/515621588 >> 172.21.15.190:0/515621588 conn(0x7f777406a500 msgr2=0x7f777406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:36.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.098+0000 7f7778bb5640 1 -- 172.21.15.190:0/515621588 shutdown_connections 2023-11-28T20:33:36.102 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.098+0000 7f7778bb5640 1 -- 172.21.15.190:0/515621588 wait complete. 2023-11-28T20:33:36.104 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.100+0000 7f7778bb5640 1 Processor -- start 2023-11-28T20:33:36.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.100+0000 7f7778bb5640 1 -- start start 2023-11-28T20:33:36.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f7778bb5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77740b5070 0x7f77740c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f7778bb5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cf940 0x7f77741565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f7778bb5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77740cfd20 0x7f7774156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f7778bb5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77741e82c0 con 0x7f77740b5070 2023-11-28T20:33:36.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f7778bb5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77740ccb50 con 0x7f77740cfd20 2023-11-28T20:33:36.105 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f7778bb5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77741e7990 con 0x7f77740cf940 2023-11-28T20:33:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f776bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cf940 0x7f77741565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f7773fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77740cfd20 0x7f7774156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f776bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cf940 0x7f77741565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:49290/0 (socket says 172.21.15.190:49290) 2023-11-28T20:33:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f776bfff640 1 -- 172.21.15.190:0/1982774296 learned_addr learned my addr 172.21.15.190:0/1982774296 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.101+0000 7f77737fe640 1 --2- 172.21.15.190:0/1982774296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77740b5070 0x7f77740c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f776bfff640 1 -- 172.21.15.190:0/1982774296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77740cfd20 msgr2=0x7f7774156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f776bfff640 1 --2- 172.21.15.190:0/1982774296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77740cfd20 0x7f7774156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f776bfff640 1 -- 172.21.15.190:0/1982774296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77740b5070 msgr2=0x7f77740c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f776bfff640 1 --2- 172.21.15.190:0/1982774296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77740b5070 0x7f77740c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f776bfff640 1 -- 172.21.15.190:0/1982774296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c009400 con 0x7f77740cf940 2023-11-28T20:33:36.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f776bfff640 1 --2- 172.21.15.190:0/1982774296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cf940 0x7f77741565a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f775c00ebe0 tx=0x7f775c00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:36.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f7770ff9640 1 -- 172.21.15.190:0/1982774296 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c009760 con 0x7f77740cf940 2023-11-28T20:33:36.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f7770ff9640 1 -- 172.21.15.190:0/1982774296 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775c00cc30 con 0x7f77740cf940 2023-11-28T20:33:36.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f7770ff9640 1 -- 172.21.15.190:0/1982774296 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c00eee0 con 0x7f77740cf940 2023-11-28T20:33:36.107 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f7778bb5640 1 -- 172.21.15.190:0/1982774296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77640094d0 con 0x7f77740cf940 2023-11-28T20:33:36.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.102+0000 7f7778bb5640 1 -- 172.21.15.190:0/1982774296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77741e7f50 con 0x7f77740cf940 2023-11-28T20:33:36.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.103+0000 7f7770ff9640 1 -- 172.21.15.190:0/1982774296 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f775c005c90 con 0x7f77740cf940 2023-11-28T20:33:36.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.103+0000 7f7770ff9640 1 -- 172.21.15.190:0/1982774296 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f775c0064b0 con 0x7f77740cf940 2023-11-28T20:33:36.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.103+0000 7f7778bb5640 1 -- 172.21.15.190:0/1982774296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77741e8430 con 0x7f77740cf940 2023-11-28T20:33:36.108 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.106+0000 7f7770ff9640 1 -- 172.21.15.190:0/1982774296 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f775c021060 con 0x7f77740cf940 2023-11-28T20:33:36.185 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.184+0000 7f7778bb5640 1 -- 172.21.15.190:0/1982774296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "52955585-272d-45b1-839d-e89bad4ed76a", "id": 3} v 0) v1 -- 0x7f7774071b30 con 0x7f77740cf940 2023-11-28T20:33:36.193 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.191+0000 7f7770ff9640 1 -- 172.21.15.190:0/1982774296 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "52955585-272d-45b1-839d-e89bad4ed76a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f775c029020 con 0x7f77740cf940 2023-11-28T20:33:36.193 INFO:teuthology.orchestra.run.smithi190.stdout:3 2023-11-28T20:33:36.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.193+0000 7f7778bb5640 1 -- 172.21.15.190:0/1982774296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cf940 msgr2=0x7f77741565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.193+0000 7f7778bb5640 1 --2- 172.21.15.190:0/1982774296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cf940 0x7f77741565a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f775c00ebe0 tx=0x7f775c00c520 comp rx=0 tx=0).stop 2023-11-28T20:33:36.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.193+0000 7f7778bb5640 1 -- 172.21.15.190:0/1982774296 shutdown_connections 2023-11-28T20:33:36.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.193+0000 7f7778bb5640 1 --2- 172.21.15.190:0/1982774296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77740cfd20 0x7f7774156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.193+0000 7f7778bb5640 1 --2- 172.21.15.190:0/1982774296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77740cf940 0x7f77741565a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.193+0000 7f7778bb5640 1 --2- 172.21.15.190:0/1982774296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77740b5070 0x7f77740c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.193+0000 7f7778bb5640 1 -- 172.21.15.190:0/1982774296 >> 172.21.15.190:0/1982774296 conn(0x7f777406a500 msgr2=0x7f77740b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:36.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.193+0000 7f7778bb5640 1 -- 172.21.15.190:0/1982774296 shutdown_connections 2023-11-28T20:33:36.195 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.193+0000 7f7778bb5640 1 -- 172.21.15.190:0/1982774296 wait complete. 2023-11-28T20:33:36.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 429ee08c-9684-47e8-afab-16671b7045fb 4 2023-11-28T20:33:36.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.407+0000 7f17cf4da640 1 Processor -- start 2023-11-28T20:33:36.408 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.407+0000 7f17cf4da640 1 -- start start 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.408+0000 7f17cf4da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.408+0000 7f17cf4da640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c8160e80 con 0x7f17c8160aa0 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17cf4da640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c80d1220 con 0x7f17c80ce0e0 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17cf4da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80d2e70 con 0x7f17c80d24f0 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17cda77640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f17c80ce0e0 0x7f17c80ce570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:33628/0 (socket says 172.21.15.190:33628) 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17cda77640 1 -- 172.21.15.190:0/1614422818 learned_addr learned my addr 172.21.15.190:0/1614422818 (peer_addr_for_me v1:172.21.15.190:0/0) 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17cca75640 1 --2- 172.21.15.190:0/1614422818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17bffff640 1 -- 172.21.15.190:0/1614422818 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179429163 0 0) 0x7f17c80d1220 con 0x7f17c80ce0e0 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17bffff640 1 -- 172.21.15.190:0/1614422818 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b4003520 con 0x7f17c80ce0e0 2023-11-28T20:33:36.411 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17cca75640 1 -- 172.21.15.190:0/1614422818 >> v1:172.21.15.190:6789/0 conn(0x7f17c80ce0e0 legacy=0x7f17c80ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.412 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17cca75640 1 -- 172.21.15.190:0/1614422818 >> v1:172.21.15.45:6789/0 conn(0x7f17c8160aa0 legacy=0x7f17c80b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.412 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.409+0000 7f17cca75640 1 -- 172.21.15.190:0/1614422818 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c80d1470 con 0x7f17c80d24f0 2023-11-28T20:33:36.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.410+0000 7f17cca75640 1 --2- 172.21.15.190:0/1614422818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80d2930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f17b80097d0 tx=0x7f17b802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdc6c8f71450947e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:36.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.411+0000 7f17bffff640 1 -- 172.21.15.190:0/1614422818 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8030880 con 0x7f17c80d24f0 2023-11-28T20:33:36.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.411+0000 7f17bffff640 1 -- 172.21.15.190:0/1614422818 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b80309e0 con 0x7f17c80d24f0 2023-11-28T20:33:36.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.411+0000 7f17bffff640 1 -- 172.21.15.190:0/1614422818 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8030d70 con 0x7f17c80d24f0 2023-11-28T20:33:36.413 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.411+0000 7f17cf4da640 1 -- 172.21.15.190:0/1614422818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 msgr2=0x7f17c80d2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.414 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.411+0000 7f17cf4da640 1 --2- 172.21.15.190:0/1614422818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80d2930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f17b80097d0 tx=0x7f17b802b9c0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.414 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.411+0000 7f17cf4da640 1 -- 172.21.15.190:0/1614422818 shutdown_connections 2023-11-28T20:33:36.414 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.411+0000 7f17cf4da640 1 --2- 172.21.15.190:0/1614422818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80d2930 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.415 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.411+0000 7f17cf4da640 1 -- 172.21.15.190:0/1614422818 >> 172.21.15.190:0/1614422818 conn(0x7f17c8070b20 msgr2=0x7f17c8070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:36.415 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.412+0000 7f17cf4da640 1 -- 172.21.15.190:0/1614422818 shutdown_connections 2023-11-28T20:33:36.415 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.412+0000 7f17cf4da640 1 -- 172.21.15.190:0/1614422818 wait complete. 2023-11-28T20:33:36.417 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.413+0000 7f17cf4da640 1 Processor -- start 2023-11-28T20:33:36.417 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.413+0000 7f17cf4da640 1 -- start start 2023-11-28T20:33:36.417 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.414+0000 7f17cf4da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17c80ce0e0 0x7f17c8145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.417 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.414+0000 7f17cf4da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.414+0000 7f17cf4da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c8160aa0 0x7f17c81f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.414+0000 7f17cf4da640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80c7280 con 0x7f17c80ce0e0 2023-11-28T20:33:36.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.414+0000 7f17cf4da640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81f2fc0 con 0x7f17c8160aa0 2023-11-28T20:33:36.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.414+0000 7f17cf4da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80c7470 con 0x7f17c80d24f0 2023-11-28T20:33:36.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.414+0000 7f17cca75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.414+0000 7f17cda77640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c8160aa0 0x7f17c81f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17cca75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:49318/0 (socket says 172.21.15.190:49318) 2023-11-28T20:33:36.418 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17cca75640 1 -- 172.21.15.190:0/2142867132 learned_addr learned my addr 172.21.15.190:0/2142867132 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17cd276640 1 --2- 172.21.15.190:0/2142867132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17c80ce0e0 0x7f17c8145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17cda77640 1 -- 172.21.15.190:0/2142867132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 msgr2=0x7f17c80c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17cda77640 1 --2- 172.21.15.190:0/2142867132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17cda77640 1 -- 172.21.15.190:0/2142867132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17c80ce0e0 msgr2=0x7f17c8145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17cda77640 1 --2- 172.21.15.190:0/2142867132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17c80ce0e0 0x7f17c8145c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17cda77640 1 -- 172.21.15.190:0/2142867132 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b8009400 con 0x7f17c8160aa0 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17cda77640 1 --2- 172.21.15.190:0/2142867132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c8160aa0 0x7f17c81f2a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f17c000ca80 tx=0x7f17c000cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17bdffb640 1 -- 172.21.15.190:0/2142867132 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c000ac30 con 0x7f17c8160aa0 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.415+0000 7f17bdffb640 1 -- 172.21.15.190:0/2142867132 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c000ad90 con 0x7f17c8160aa0 2023-11-28T20:33:36.419 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.416+0000 7f17bdffb640 1 -- 172.21.15.190:0/2142867132 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0012600 con 0x7f17c8160aa0 2023-11-28T20:33:36.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.416+0000 7f17cf4da640 1 -- 172.21.15.190:0/2142867132 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c80c8bc0 con 0x7f17c8160aa0 2023-11-28T20:33:36.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.416+0000 7f17cf4da640 1 -- 172.21.15.190:0/2142867132 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c80c9180 con 0x7f17c8160aa0 2023-11-28T20:33:36.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.416+0000 7f17bdffb640 1 -- 172.21.15.190:0/2142867132 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f17c0012a60 con 0x7f17c8160aa0 2023-11-28T20:33:36.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.416+0000 7f17bdffb640 1 -- 172.21.15.190:0/2142867132 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f17c0012d10 con 0x7f17c8160aa0 2023-11-28T20:33:36.420 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.417+0000 7f17cf4da640 1 -- 172.21.15.190:0/2142867132 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c80733d0 con 0x7f17c8160aa0 2023-11-28T20:33:36.421 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.419+0000 7f17bdffb640 1 -- 172.21.15.190:0/2142867132 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f17c0021060 con 0x7f17c8160aa0 2023-11-28T20:33:36.491 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:36.490+0000 7f529be10040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:33:36.499 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.498+0000 7f17cf4da640 1 -- 172.21.15.190:0/2142867132 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "429ee08c-9684-47e8-afab-16671b7045fb", "id": 4} v 0) v1 -- 0x7f17c80d1000 con 0x7f17c8160aa0 2023-11-28T20:33:36.507 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.504+0000 7f17bdffb640 1 -- 172.21.15.190:0/2142867132 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "429ee08c-9684-47e8-afab-16671b7045fb", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f17c0017070 con 0x7f17c8160aa0 2023-11-28T20:33:36.508 INFO:teuthology.orchestra.run.smithi190.stdout:4 2023-11-28T20:33:36.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.506+0000 7f17cf4da640 1 -- 172.21.15.190:0/2142867132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c8160aa0 msgr2=0x7f17c81f2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.506+0000 7f17cf4da640 1 --2- 172.21.15.190:0/2142867132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c8160aa0 0x7f17c81f2a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f17c000ca80 tx=0x7f17c000cf40 comp rx=0 tx=0).stop 2023-11-28T20:33:36.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.506+0000 7f17cf4da640 1 -- 172.21.15.190:0/2142867132 shutdown_connections 2023-11-28T20:33:36.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.506+0000 7f17cf4da640 1 --2- 172.21.15.190:0/2142867132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c8160aa0 0x7f17c81f2a80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.506+0000 7f17cf4da640 1 --2- 172.21.15.190:0/2142867132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17c80d24f0 0x7f17c80c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.506+0000 7f17cf4da640 1 --2- 172.21.15.190:0/2142867132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17c80ce0e0 0x7f17c8145c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.506+0000 7f17cf4da640 1 -- 172.21.15.190:0/2142867132 >> 172.21.15.190:0/2142867132 conn(0x7f17c8070b20 msgr2=0x7f17c80b6480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:36.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.506+0000 7f17cf4da640 1 -- 172.21.15.190:0/2142867132 shutdown_connections 2023-11-28T20:33:36.509 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.506+0000 7f17cf4da640 1 -- 172.21.15.190:0/2142867132 wait complete. 2023-11-28T20:33:36.512 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:36.510+0000 7fb587222040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:33:36.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 65bce9bc-01b3-40a1-aa05-9a0ad17e5de2 5 2023-11-28T20:33:36.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:36.659+0000 7f529be10040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:33:36.681 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:36.679+0000 7fb587222040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:33:36.711 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.710+0000 7fab56375640 1 Processor -- start 2023-11-28T20:33:36.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.710+0000 7fab56375640 1 -- start start 2023-11-28T20:33:36.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.712+0000 7fab56375640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab500af070 0x7fab500acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.712+0000 7fab56375640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5015db60 con 0x7fab5006e8e0 2023-11-28T20:33:36.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.712+0000 7fab56375640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500ad2c0 con 0x7fab5006b510 2023-11-28T20:33:36.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.712+0000 7fab56375640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5015e700 con 0x7fab500af070 2023-11-28T20:33:36.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.712+0000 7fab55b74640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fab5006b510 0x7fab500b3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:33630/0 (socket says 172.21.15.190:33630) 2023-11-28T20:33:36.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.712+0000 7fab55b74640 1 -- 172.21.15.190:0/2427060405 learned_addr learned my addr 172.21.15.190:0/2427060405 (peer_addr_for_me v1:172.21.15.190:0/0) 2023-11-28T20:33:36.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.712+0000 7fab54b72640 1 --2- 172.21.15.190:0/2427060405 >> v2:172.21.15.45:3300/0 conn(0x7fab500af070 0x7fab500acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764919933 0 0) 0x7fab500ad2c0 con 0x7fab5006b510 2023-11-28T20:33:36.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab3c003520 con 0x7fab5006b510 2023-11-28T20:33:36.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2837945373 0 0) 0x7fab3c003520 con 0x7fab5006b510 2023-11-28T20:33:36.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 >> v2:172.21.15.45:3300/0 conn(0x7fab500af070 msgr2=0x7fab500acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:36.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 --2- 172.21.15.190:0/2427060405 >> v2:172.21.15.45:3300/0 conn(0x7fab500af070 0x7fab500acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 >> v1:172.21.15.45:6789/0 conn(0x7fab5006e8e0 legacy=0x7fab5015dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500af4f0 con 0x7fab5006b510 2023-11-28T20:33:36.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fab480019f0 con 0x7fab5006b510 2023-11-28T20:33:36.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 >> v1:172.21.15.190:6789/0 conn(0x7fab5006b510 legacy=0x7fab500b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0039c0 0x7fab3c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab3c0047c0 0x7fab3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c005180 0x7fab3c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5015e700 con 0x7fab3c005180 2023-11-28T20:33:36.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c00c080 con 0x7fab3c0039c0 2023-11-28T20:33:36.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c00c240 con 0x7fab3c0047c0 2023-11-28T20:33:36.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab55373640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0039c0 0x7fab3c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.713+0000 7fab54b72640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab3c0047c0 0x7fab3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.714+0000 7fab55b74640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c005180 0x7fab3c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.714+0000 7fab54b72640 1 -- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0039c0 msgr2=0x7fab3c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:36.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.714+0000 7fab54b72640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0039c0 0x7fab3c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.714+0000 7fab54b72640 1 -- 172.21.15.190:0/2427060405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c005180 msgr2=0x7fab3c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:36.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.714+0000 7fab54b72640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c005180 0x7fab3c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.714+0000 7fab54b72640 1 -- 172.21.15.190:0/2427060405 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c003520 con 0x7fab3c0047c0 2023-11-28T20:33:36.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.714+0000 7fab55b74640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c005180 0x7fab3c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:36.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.714+0000 7fab55b74640 1 -- 172.21.15.190:0/2427060405 reap_dead start 2023-11-28T20:33:36.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.715+0000 7fab54b72640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab3c0047c0 0x7fab3c004c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fab40009ac0 tx=0x7fab3c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46f6b0523f8a3d17 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:36.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.715+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4002c6d0 con 0x7fab3c0047c0 2023-11-28T20:33:36.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.715+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4002c830 con 0x7fab3c0047c0 2023-11-28T20:33:36.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.715+0000 7fab47fff640 1 -- 172.21.15.190:0/2427060405 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4002cbc0 con 0x7fab3c0047c0 2023-11-28T20:33:36.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.716+0000 7fab56375640 1 -- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab3c0047c0 msgr2=0x7fab3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.716+0000 7fab56375640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab3c0047c0 0x7fab3c004c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fab40009ac0 tx=0x7fab3c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.716+0000 7fab56375640 1 -- 172.21.15.190:0/2427060405 shutdown_connections 2023-11-28T20:33:36.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.716+0000 7fab56375640 1 --2- 172.21.15.190:0/2427060405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab3c0047c0 0x7fab3c004c40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.716+0000 7fab56375640 1 -- 172.21.15.190:0/2427060405 >> 172.21.15.190:0/2427060405 conn(0x7fab50071240 msgr2=0x7fab50071650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:36.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.716+0000 7fab56375640 1 -- 172.21.15.190:0/2427060405 shutdown_connections 2023-11-28T20:33:36.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.716+0000 7fab56375640 1 -- 172.21.15.190:0/2427060405 wait complete. 2023-11-28T20:33:36.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.719+0000 7fab56375640 1 Processor -- start 2023-11-28T20:33:36.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.719+0000 7fab56375640 1 -- start start 2023-11-28T20:33:36.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.720+0000 7fab56375640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0047c0 0x7fab50160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab56375640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab50066d20 0x7fab5015e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab56375640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab501447c0 0x7fab50153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:36.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab56375640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5015fcc0 con 0x7fab50066d20 2023-11-28T20:33:36.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab56375640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5015f0b0 con 0x7fab3c0047c0 2023-11-28T20:33:36.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab56375640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab5015f390 con 0x7fab501447c0 2023-11-28T20:33:36.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab55373640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0047c0 0x7fab50160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab55373640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0047c0 0x7fab50160150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:36090/0 (socket says 172.21.15.190:36090) 2023-11-28T20:33:36.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab55373640 1 -- 172.21.15.190:0/3711621245 learned_addr learned my addr 172.21.15.190:0/3711621245 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:36.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab55b74640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab501447c0 0x7fab50153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab54b72640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab50066d20 0x7fab5015e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:36.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab55373640 1 -- 172.21.15.190:0/3711621245 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab501447c0 msgr2=0x7fab50153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab55373640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab501447c0 0x7fab50153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab55373640 1 -- 172.21.15.190:0/3711621245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab50066d20 msgr2=0x7fab5015e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab55373640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab50066d20 0x7fab5015e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab55373640 1 -- 172.21.15.190:0/3711621245 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5015f670 con 0x7fab3c0047c0 2023-11-28T20:33:36.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.721+0000 7fab54b72640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab50066d20 0x7fab5015e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:36.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.722+0000 7fab55373640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0047c0 0x7fab50160150 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00f590 tx=0x7fab4c00fa50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:36.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.722+0000 7fab45ffb640 1 -- 172.21.15.190:0/3711621245 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0052e0 con 0x7fab3c0047c0 2023-11-28T20:33:36.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.722+0000 7fab45ffb640 1 -- 172.21.15.190:0/3711621245 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4c005440 con 0x7fab3c0047c0 2023-11-28T20:33:36.735 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.722+0000 7fab56375640 1 -- 172.21.15.190:0/3711621245 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5015f950 con 0x7fab3c0047c0 2023-11-28T20:33:36.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.722+0000 7fab45ffb640 1 -- 172.21.15.190:0/3711621245 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c003830 con 0x7fab3c0047c0 2023-11-28T20:33:36.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.722+0000 7fab56375640 1 -- 172.21.15.190:0/3711621245 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab50144c40 con 0x7fab3c0047c0 2023-11-28T20:33:36.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.722+0000 7fab45ffb640 1 -- 172.21.15.190:0/3711621245 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fab4c003c90 con 0x7fab3c0047c0 2023-11-28T20:33:36.736 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.722+0000 7fab45ffb640 1 -- 172.21.15.190:0/3711621245 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fab4c0133f0 con 0x7fab3c0047c0 2023-11-28T20:33:36.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.723+0000 7fab56375640 1 -- 172.21.15.190:0/3711621245 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5009e100 con 0x7fab3c0047c0 2023-11-28T20:33:36.737 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.726+0000 7fab45ffb640 1 -- 172.21.15.190:0/3711621245 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fab4c01d460 con 0x7fab3c0047c0 2023-11-28T20:33:36.803 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:36.800+0000 7f529be10040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:33:36.804 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.803+0000 7fab56375640 1 -- 172.21.15.190:0/3711621245 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "65bce9bc-01b3-40a1-aa05-9a0ad17e5de2", "id": 5} v 0) v1 -- 0x7fab5004e920 con 0x7fab3c0047c0 2023-11-28T20:33:36.810 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.809+0000 7fab45ffb640 1 -- 172.21.15.190:0/3711621245 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "65bce9bc-01b3-40a1-aa05-9a0ad17e5de2", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fab4c02c0b0 con 0x7fab3c0047c0 2023-11-28T20:33:36.811 INFO:teuthology.orchestra.run.smithi190.stdout:5 2023-11-28T20:33:36.812 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.810+0000 7fab56375640 1 -- 172.21.15.190:0/3711621245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0047c0 msgr2=0x7fab50160150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:36.812 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.810+0000 7fab56375640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0047c0 0x7fab50160150 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00f590 tx=0x7fab4c00fa50 comp rx=0 tx=0).stop 2023-11-28T20:33:36.812 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.810+0000 7fab56375640 1 -- 172.21.15.190:0/3711621245 shutdown_connections 2023-11-28T20:33:36.813 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.810+0000 7fab56375640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab501447c0 0x7fab50153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.813 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.811+0000 7fab56375640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab50066d20 0x7fab5015e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.813 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.811+0000 7fab56375640 1 --2- 172.21.15.190:0/3711621245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab3c0047c0 0x7fab50160150 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:36.813 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.811+0000 7fab56375640 1 -- 172.21.15.190:0/3711621245 >> 172.21.15.190:0/3711621245 conn(0x7fab50071240 msgr2=0x7fab500b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:36.813 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.811+0000 7fab56375640 1 -- 172.21.15.190:0/3711621245 shutdown_connections 2023-11-28T20:33:36.813 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:36.811+0000 7fab56375640 1 -- 172.21.15.190:0/3711621245 wait complete. 2023-11-28T20:33:36.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new d414aa0f-ad7f-44dd-9072-6d3f07d1e44d 6 2023-11-28T20:33:36.832 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:36.830+0000 7fb587222040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:33:36.858 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:36.857+0000 7f529be10040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:33:36.889 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:36.887+0000 7fb587222040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:33:36.975 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:36.974+0000 7f529be10040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:33:37.007 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:37.005+0000 7fb587222040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:33:37.013 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.012+0000 7f679f548640 1 Processor -- start 2023-11-28T20:33:37.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.012+0000 7f679f548640 1 -- start start 2023-11-28T20:33:37.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679f548640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f679806a500 0x7f679806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679f548640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f679815ddd0 0x7f679815e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679f548640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679806ca60 con 0x7f6798157c10 2023-11-28T20:33:37.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679f548640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679815eb40 con 0x7f679815ddd0 2023-11-28T20:33:37.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679f548640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6798157ff0 con 0x7f679806a500 2023-11-28T20:33:37.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679cae3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f679806a500 0x7f679806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679d2e4640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f6798157c10 0x7f679806da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:33644/0 (socket says 172.21.15.190:33644) 2023-11-28T20:33:37.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679cae3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f679806a500 0x7f679806a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:49348/0 (socket says 172.21.15.190:49348) 2023-11-28T20:33:37.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679d2e4640 1 -- 172.21.15.190:0/3884241556 learned_addr learned my addr 172.21.15.190:0/3884241556 (peer_addr_for_me v1:172.21.15.190:0/0) 2023-11-28T20:33:37.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679dae5640 1 --2- 172.21.15.190:0/3884241556 >> v2:172.21.15.45:3300/0 conn(0x7f679815ddd0 0x7f679815e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679cae3640 1 -- 172.21.15.190:0/3884241556 >> v2:172.21.15.45:3300/0 conn(0x7f679815ddd0 msgr2=0x7f679815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:37.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679cae3640 1 --2- 172.21.15.190:0/3884241556 >> v2:172.21.15.45:3300/0 conn(0x7f679815ddd0 0x7f679815e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679cae3640 1 -- 172.21.15.190:0/3884241556 >> v1:172.21.15.190:6789/0 conn(0x7f6798157c10 legacy=0x7f679806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.017 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.014+0000 7f679cae3640 1 -- 172.21.15.190:0/3884241556 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679806aea0 con 0x7f679806a500 2023-11-28T20:33:37.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.015+0000 7f679cae3640 1 --2- 172.21.15.190:0/3884241556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f679806a500 0x7f679806a960 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f67880097a0 tx=0x7f678802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=516138cf462df48e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:37.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.016+0000 7f678ffff640 1 -- 172.21.15.190:0/3884241556 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678802d040 con 0x7f679806a500 2023-11-28T20:33:37.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.016+0000 7f678ffff640 1 -- 172.21.15.190:0/3884241556 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6788031810 con 0x7f679806a500 2023-11-28T20:33:37.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.016+0000 7f678ffff640 1 -- 172.21.15.190:0/3884241556 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788031ba0 con 0x7f679806a500 2023-11-28T20:33:37.018 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.016+0000 7f679f548640 1 -- 172.21.15.190:0/3884241556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f679806a500 msgr2=0x7f679806a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.016+0000 7f679f548640 1 --2- 172.21.15.190:0/3884241556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f679806a500 0x7f679806a960 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f67880097a0 tx=0x7f678802b8b0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.016+0000 7f679f548640 1 -- 172.21.15.190:0/3884241556 shutdown_connections 2023-11-28T20:33:37.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.016+0000 7f679f548640 1 --2- 172.21.15.190:0/3884241556 >> v2:172.21.15.45:3300/0 conn(0x7f679815ddd0 0x7f679815e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.016+0000 7f679f548640 1 --2- 172.21.15.190:0/3884241556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f679806a500 0x7f679806a960 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.016+0000 7f679f548640 1 -- 172.21.15.190:0/3884241556 >> 172.21.15.190:0/3884241556 conn(0x7f67980ced60 msgr2=0x7f67980cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:37.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.017+0000 7f679f548640 1 -- 172.21.15.190:0/3884241556 shutdown_connections 2023-11-28T20:33:37.019 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.017+0000 7f679f548640 1 -- 172.21.15.190:0/3884241556 wait complete. 2023-11-28T20:33:37.020 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.018+0000 7f679f548640 1 Processor -- start 2023-11-28T20:33:37.020 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.018+0000 7f679f548640 1 -- start start 2023-11-28T20:33:37.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.019+0000 7f679f548640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f679806a500 0x7f67981569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.019+0000 7f679f548640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6798157c10 0x7f679815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.019+0000 7f679f548640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f679815ddd0 0x7f679815c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.021 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.019+0000 7f679f548640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67980c4eb0 con 0x7f679815ddd0 2023-11-28T20:33:37.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.019+0000 7f679f548640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67980c4610 con 0x7f679806a500 2023-11-28T20:33:37.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.019+0000 7f679f548640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67980c48f0 con 0x7f6798157c10 2023-11-28T20:33:37.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679d2e4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f679806a500 0x7f67981569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679cae3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6798157c10 0x7f679815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679d2e4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f679806a500 0x7f67981569e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:36100/0 (socket says 172.21.15.190:36100) 2023-11-28T20:33:37.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679d2e4640 1 -- 172.21.15.190:0/2944685365 learned_addr learned my addr 172.21.15.190:0/2944685365 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:37.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679dae5640 1 --2- 172.21.15.190:0/2944685365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f679815ddd0 0x7f679815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.022 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679cae3640 1 -- 172.21.15.190:0/2944685365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f679806a500 msgr2=0x7f67981569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.023 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679cae3640 1 --2- 172.21.15.190:0/2944685365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f679806a500 0x7f67981569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.023 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679cae3640 1 -- 172.21.15.190:0/2944685365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f679815ddd0 msgr2=0x7f679815c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.023 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679cae3640 1 --2- 172.21.15.190:0/2944685365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f679815ddd0 0x7f679815c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.023 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679cae3640 1 -- 172.21.15.190:0/2944685365 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788009400 con 0x7f6798157c10 2023-11-28T20:33:37.023 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679dae5640 1 --2- 172.21.15.190:0/2944685365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f679815ddd0 0x7f679815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:33:37.024 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.020+0000 7f679cae3640 1 --2- 172.21.15.190:0/2944685365 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6798157c10 0x7f679815c1c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6788033000 tx=0x7f67880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:37.024 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.021+0000 7f678dffb640 1 -- 172.21.15.190:0/2944685365 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678803ace0 con 0x7f6798157c10 2023-11-28T20:33:37.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.021+0000 7f678dffb640 1 -- 172.21.15.190:0/2944685365 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678803ae80 con 0x7f6798157c10 2023-11-28T20:33:37.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.021+0000 7f678dffb640 1 -- 172.21.15.190:0/2944685365 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788044630 con 0x7f6798157c10 2023-11-28T20:33:37.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.021+0000 7f679f548640 1 -- 172.21.15.190:0/2944685365 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67980c4bd0 con 0x7f6798157c10 2023-11-28T20:33:37.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.021+0000 7f679f548640 1 -- 172.21.15.190:0/2944685365 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67980c37a0 con 0x7f6798157c10 2023-11-28T20:33:37.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.021+0000 7f678dffb640 1 -- 172.21.15.190:0/2944685365 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6788044a90 con 0x7f6798157c10 2023-11-28T20:33:37.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.021+0000 7f678dffb640 1 -- 172.21.15.190:0/2944685365 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f6788043990 con 0x7f6798157c10 2023-11-28T20:33:37.025 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.022+0000 7f679f548640 1 -- 172.21.15.190:0/2944685365 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6798099db0 con 0x7f6798157c10 2023-11-28T20:33:37.026 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.024+0000 7f678dffb640 1 -- 172.21.15.190:0/2944685365 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f6798099db0 con 0x7f6798157c10 2023-11-28T20:33:37.092 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:37.090+0000 7f529be10040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:33:37.103 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.102+0000 7f679f548640 1 -- 172.21.15.190:0/2944685365 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d414aa0f-ad7f-44dd-9072-6d3f07d1e44d", "id": 6} v 0) v1 -- 0x7f67980733d0 con 0x7f6798157c10 2023-11-28T20:33:37.110 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.109+0000 7f678dffb640 1 -- 172.21.15.190:0/2944685365 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d414aa0f-ad7f-44dd-9072-6d3f07d1e44d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6788035020 con 0x7f6798157c10 2023-11-28T20:33:37.111 INFO:teuthology.orchestra.run.smithi190.stdout:6 2023-11-28T20:33:37.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.110+0000 7f679f548640 1 -- 172.21.15.190:0/2944685365 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6798157c10 msgr2=0x7f679815c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.110+0000 7f679f548640 1 --2- 172.21.15.190:0/2944685365 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6798157c10 0x7f679815c1c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6788033000 tx=0x7f67880046c0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.110+0000 7f679f548640 1 -- 172.21.15.190:0/2944685365 shutdown_connections 2023-11-28T20:33:37.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.110+0000 7f679f548640 1 --2- 172.21.15.190:0/2944685365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f679815ddd0 0x7f679815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.110+0000 7f679f548640 1 --2- 172.21.15.190:0/2944685365 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6798157c10 0x7f679815c1c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.110+0000 7f679f548640 1 --2- 172.21.15.190:0/2944685365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f679806a500 0x7f67981569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.113 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.110+0000 7f679f548640 1 -- 172.21.15.190:0/2944685365 >> 172.21.15.190:0/2944685365 conn(0x7f67980ced60 msgr2=0x7f67980cad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:37.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.110+0000 7f679f548640 1 -- 172.21.15.190:0/2944685365 shutdown_connections 2023-11-28T20:33:37.114 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.110+0000 7f679f548640 1 -- 172.21.15.190:0/2944685365 wait complete. 2023-11-28T20:33:37.124 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:37.122+0000 7fb587222040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:33:37.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 5cfcd895-a0df-4b21-b568-5aa243d732e3 7 2023-11-28T20:33:37.262 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:37.259+0000 7f529be10040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:33:37.291 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:37.289+0000 7fb587222040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:33:37.313 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:37.311+0000 7f529be10040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:33:37.343 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:37.340+0000 7fb587222040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:33:37.352 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.350+0000 7feb8cfc1640 1 Processor -- start 2023-11-28T20:33:37.352 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.350+0000 7feb8cfc1640 1 -- start start 2023-11-28T20:33:37.356 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.352+0000 7feb8cfc1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb880cf940 0x7feb88161800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.352+0000 7feb8cfc1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb880b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.352+0000 7feb8cfc1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb880b2930 con 0x7feb880cfd20 2023-11-28T20:33:37.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.352+0000 7feb8cfc1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb88161d40 con 0x7feb880cf940 2023-11-28T20:33:37.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.352+0000 7feb8cfc1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb880b2650 con 0x7feb880d28e0 2023-11-28T20:33:37.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.352+0000 7feb877fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb880cf940 0x7feb88161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.357 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.352+0000 7feb87fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb880b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.352+0000 7feb877fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb880cf940 0x7feb88161800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:36106/0 (socket says 172.21.15.190:36106) 2023-11-28T20:33:37.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.352+0000 7feb877fe640 1 -- 172.21.15.190:0/299059035 learned_addr learned my addr 172.21.15.190:0/299059035 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:37.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.353+0000 7feb877fe640 1 -- 172.21.15.190:0/299059035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 msgr2=0x7feb880b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:37.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.353+0000 7feb877fe640 1 --2- 172.21.15.190:0/299059035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb880b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.353+0000 7feb877fe640 1 -- 172.21.15.190:0/299059035 >> v1:172.21.15.190:6789/0 conn(0x7feb880cfd20 legacy=0x7feb880b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.353+0000 7feb877fe640 1 -- 172.21.15.190:0/299059035 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb880ce240 con 0x7feb880cf940 2023-11-28T20:33:37.358 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.353+0000 7feb87fff640 1 --2- 172.21.15.190:0/299059035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb880b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:33:37.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.354+0000 7feb877fe640 1 --2- 172.21.15.190:0/299059035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 0x7feb88161800 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feb78009870 tx=0x7feb7802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c67fadd436df1761 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:37.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.354+0000 7feb86ffd640 1 -- 172.21.15.190:0/299059035 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb780317c0 con 0x7feb880cf940 2023-11-28T20:33:37.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.354+0000 7feb86ffd640 1 -- 172.21.15.190:0/299059035 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb78031920 con 0x7feb880cf940 2023-11-28T20:33:37.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.354+0000 7feb86ffd640 1 -- 172.21.15.190:0/299059035 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb78031cb0 con 0x7feb880cf940 2023-11-28T20:33:37.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.355+0000 7feb8cfc1640 1 -- 172.21.15.190:0/299059035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 msgr2=0x7feb88161800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.359 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.355+0000 7feb8cfc1640 1 --2- 172.21.15.190:0/299059035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 0x7feb88161800 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feb78009870 tx=0x7feb7802b860 comp rx=0 tx=0).stop 2023-11-28T20:33:37.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.355+0000 7feb8cfc1640 1 -- 172.21.15.190:0/299059035 shutdown_connections 2023-11-28T20:33:37.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.355+0000 7feb8cfc1640 1 --2- 172.21.15.190:0/299059035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb880b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.355+0000 7feb8cfc1640 1 --2- 172.21.15.190:0/299059035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 0x7feb88161800 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.355+0000 7feb8cfc1640 1 -- 172.21.15.190:0/299059035 >> 172.21.15.190:0/299059035 conn(0x7feb8806a500 msgr2=0x7feb8806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:37.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.355+0000 7feb8cfc1640 1 -- 172.21.15.190:0/299059035 shutdown_connections 2023-11-28T20:33:37.360 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.355+0000 7feb8cfc1640 1 -- 172.21.15.190:0/299059035 wait complete. 2023-11-28T20:33:37.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.356+0000 7feb8cfc1640 1 Processor -- start 2023-11-28T20:33:37.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.357+0000 7feb8cfc1640 1 -- start start 2023-11-28T20:33:37.364 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb8cfc1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 0x7feb88148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb8cfc1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb880cfd20 0x7feb8814b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb8cfc1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb8814bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb8cfc1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb881e7f10 con 0x7feb880cfd20 2023-11-28T20:33:37.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb8cfc1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb88160900 con 0x7feb880cf940 2023-11-28T20:33:37.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb8cfc1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb880c9a60 con 0x7feb880d28e0 2023-11-28T20:33:37.365 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb877fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 0x7feb88148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.366 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb877fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 0x7feb88148e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:36120/0 (socket says 172.21.15.190:36120) 2023-11-28T20:33:37.366 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb877fe640 1 -- 172.21.15.190:0/2544450683 learned_addr learned my addr 172.21.15.190:0/2544450683 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-11-28T20:33:37.366 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb87fff640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb8814bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.366 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb7ffff640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb880cfd20 0x7feb8814b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.366 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb877fe640 1 -- 172.21.15.190:0/2544450683 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 msgr2=0x7feb8814bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb877fe640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb8814bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb877fe640 1 -- 172.21.15.190:0/2544450683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb880cfd20 msgr2=0x7feb8814b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb877fe640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb880cfd20 0x7feb8814b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb877fe640 1 -- 172.21.15.190:0/2544450683 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb80009a70 con 0x7feb880cf940 2023-11-28T20:33:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.358+0000 7feb87fff640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb8814bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:33:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.359+0000 7feb877fe640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 0x7feb88148e30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb78000c00 tx=0x7feb780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.359+0000 7feb84ff9640 1 -- 172.21.15.190:0/2544450683 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb78032860 con 0x7feb880cf940 2023-11-28T20:33:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.359+0000 7feb84ff9640 1 -- 172.21.15.190:0/2544450683 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb78032a00 con 0x7feb880cf940 2023-11-28T20:33:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.359+0000 7feb84ff9640 1 -- 172.21.15.190:0/2544450683 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb78032db0 con 0x7feb880cf940 2023-11-28T20:33:37.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.359+0000 7feb8cfc1640 1 -- 172.21.15.190:0/2544450683 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb780094d0 con 0x7feb880cf940 2023-11-28T20:33:37.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.359+0000 7feb8cfc1640 1 -- 172.21.15.190:0/2544450683 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb88160340 con 0x7feb880cf940 2023-11-28T20:33:37.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.359+0000 7feb84ff9640 1 -- 172.21.15.190:0/2544450683 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feb7804e750 con 0x7feb880cf940 2023-11-28T20:33:37.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.359+0000 7feb84ff9640 1 -- 172.21.15.190:0/2544450683 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7feb78045b90 con 0x7feb880cf940 2023-11-28T20:33:37.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.360+0000 7feb8cfc1640 1 -- 172.21.15.190:0/2544450683 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb881e7c70 con 0x7feb880cf940 2023-11-28T20:33:37.368 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.362+0000 7feb84ff9640 1 -- 172.21.15.190:0/2544450683 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7feb881e7c70 con 0x7feb880cf940 2023-11-28T20:33:37.442 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.441+0000 7feb8cfc1640 1 -- 172.21.15.190:0/2544450683 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5cfcd895-a0df-4b21-b568-5aa243d732e3", "id": 7} v 0) v1 -- 0x7feb881e8050 con 0x7feb880cf940 2023-11-28T20:33:37.449 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.447+0000 7feb84ff9640 1 -- 172.21.15.190:0/2544450683 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5cfcd895-a0df-4b21-b568-5aa243d732e3", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7feb881e8050 con 0x7feb880cf940 2023-11-28T20:33:37.449 INFO:teuthology.orchestra.run.smithi190.stdout:7 2023-11-28T20:33:37.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.448+0000 7feb8cfc1640 1 -- 172.21.15.190:0/2544450683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 msgr2=0x7feb88148e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.448+0000 7feb8cfc1640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 0x7feb88148e30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb78000c00 tx=0x7feb780046c0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.448+0000 7feb8cfc1640 1 -- 172.21.15.190:0/2544450683 shutdown_connections 2023-11-28T20:33:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.448+0000 7feb8cfc1640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb880d28e0 0x7feb8814bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.448+0000 7feb8cfc1640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb880cfd20 0x7feb8814b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.448+0000 7feb8cfc1640 1 --2- 172.21.15.190:0/2544450683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb880cf940 0x7feb88148e30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.448+0000 7feb8cfc1640 1 -- 172.21.15.190:0/2544450683 >> 172.21.15.190:0/2544450683 conn(0x7feb8806a500 msgr2=0x7feb880b3140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.449+0000 7feb8cfc1640 1 -- 172.21.15.190:0/2544450683 shutdown_connections 2023-11-28T20:33:37.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-28T20:33:37.449+0000 7feb8cfc1640 1 -- 172.21.15.190:0/2544450683 wait complete. 2023-11-28T20:33:37.467 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-28T20:33:37.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-28T20:33:37.476 INFO:tasks.ceph.osd.0:Started 2023-11-28T20:33:37.476 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-28T20:33:37.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-28T20:33:37.480 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:37.479+0000 7f529be10040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:33:37.485 INFO:tasks.ceph.osd.1:Started 2023-11-28T20:33:37.485 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-28T20:33:37.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-28T20:33:37.493 INFO:tasks.ceph.osd.2:Started 2023-11-28T20:33:37.493 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-28T20:33:37.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-28T20:33:37.503 INFO:tasks.ceph.osd.3:Started 2023-11-28T20:33:37.503 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-28T20:33:37.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-28T20:33:37.512 INFO:tasks.ceph.osd.4:Started 2023-11-28T20:33:37.512 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-28T20:33:37.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-28T20:33:37.514 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:37.511+0000 7fb587222040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:33:37.520 INFO:tasks.ceph.osd.5:Started 2023-11-28T20:33:37.521 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-28T20:33:37.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-28T20:33:37.526 INFO:tasks.ceph.osd.6:Started 2023-11-28T20:33:37.526 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-28T20:33:37.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-28T20:33:37.535 INFO:tasks.ceph.osd.7:Started 2023-11-28T20:33:37.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:33:37.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.764+0000 7f346b143640 1 Processor -- start 2023-11-28T20:33:37.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.764+0000 7f346b143640 1 -- start start 2023-11-28T20:33:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.765+0000 7f346b143640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640661f0 0x7f34640665d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.765+0000 7f346b143640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f34640cdf10 0x7f34640cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.765+0000 7f346b143640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34640c3880 con 0x7f34640ce460 2023-11-28T20:33:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.765+0000 7f346b143640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640cd7c0 con 0x7f34640cdf10 2023-11-28T20:33:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.765+0000 7f346b143640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346415b960 con 0x7f34640661f0 2023-11-28T20:33:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f3468edf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640661f0 0x7f34640665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f345bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f34640cdf10 0x7f34640cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f3468edf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640661f0 0x7f34640665d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:57098/0 (socket says 172.21.15.45:57098) 2023-11-28T20:33:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f345bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f34640cdf10 0x7f34640cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:60190/0 (socket says 172.21.15.45:60190) 2023-11-28T20:33:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f3468edf640 1 -- 172.21.15.45:0/1693038818 learned_addr learned my addr 172.21.15.45:0/1693038818 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f345bfff640 1 -- 172.21.15.45:0/1693038818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640661f0 msgr2=0x7f34640665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f345bfff640 1 --2- 172.21.15.45:0/1693038818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640661f0 0x7f34640665d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f345bfff640 1 -- 172.21.15.45:0/1693038818 >> v1:172.21.15.190:6789/0 conn(0x7f34640ce460 legacy=0x7f34640cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f345bfff640 1 -- 172.21.15.45:0/1693038818 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640cd490 con 0x7f34640cdf10 2023-11-28T20:33:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.766+0000 7f3468edf640 1 --2- 172.21.15.45:0/1693038818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640661f0 0x7f34640665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:33:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.767+0000 7f345bfff640 1 --2- 172.21.15.45:0/1693038818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34640cdf10 0x7f34640cb2f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f34540097a0 tx=0x7f345402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f8bcd52c87b8710 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.767+0000 7f345b7fe640 1 -- 172.21.15.45:0/1693038818 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345402f050 con 0x7f34640cdf10 2023-11-28T20:33:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f345b7fe640 1 -- 172.21.15.45:0/1693038818 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3454031700 con 0x7f34640cdf10 2023-11-28T20:33:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f345b7fe640 1 -- 172.21.15.45:0/1693038818 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454031a90 con 0x7f34640cdf10 2023-11-28T20:33:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f346b143640 1 -- 172.21.15.45:0/1693038818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34640cdf10 msgr2=0x7f34640cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f346b143640 1 --2- 172.21.15.45:0/1693038818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34640cdf10 0x7f34640cb2f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f34540097a0 tx=0x7f345402b7a0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f346b143640 1 -- 172.21.15.45:0/1693038818 shutdown_connections 2023-11-28T20:33:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f346b143640 1 --2- 172.21.15.45:0/1693038818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34640cdf10 0x7f34640cb2f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f346b143640 1 --2- 172.21.15.45:0/1693038818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640661f0 0x7f34640665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f346b143640 1 -- 172.21.15.45:0/1693038818 >> 172.21.15.45:0/1693038818 conn(0x7f34640cc970 msgr2=0x7f34640ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f346b143640 1 -- 172.21.15.45:0/1693038818 shutdown_connections 2023-11-28T20:33:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.768+0000 7f346b143640 1 -- 172.21.15.45:0/1693038818 wait complete. 2023-11-28T20:33:37.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.770+0000 7f346b143640 1 Processor -- start 2023-11-28T20:33:37.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.770+0000 7f346b143640 1 -- start start 2023-11-28T20:33:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f346b143640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34640661f0 0x7f3464156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f346b143640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34640cdf10 0x7f34641f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f346b143640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640ce460 0x7f34640c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f346b143640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34640c9d40 con 0x7f34640661f0 2023-11-28T20:33:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f346b143640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34641f0750 con 0x7f34640cdf10 2023-11-28T20:33:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f346b143640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34640b5da0 con 0x7f34640ce460 2023-11-28T20:33:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f3468edf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34640661f0 0x7f3464156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f3468edf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34640661f0 0x7f3464156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47274/0 (socket says 172.21.15.45:47274) 2023-11-28T20:33:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f3468edf640 1 -- 172.21.15.45:0/64330646 learned_addr learned my addr 172.21.15.45:0/64330646 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.771+0000 7f34696e0640 1 --2- 172.21.15.45:0/64330646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640ce460 0x7f34640c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.772+0000 7f345bfff640 1 --2- 172.21.15.45:0/64330646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34640cdf10 0x7f34641f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.772+0000 7f3468edf640 1 -- 172.21.15.45:0/64330646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640ce460 msgr2=0x7f34640c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.772+0000 7f3468edf640 1 --2- 172.21.15.45:0/64330646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640ce460 0x7f34640c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.772+0000 7f3468edf640 1 -- 172.21.15.45:0/64330646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34640cdf10 msgr2=0x7f34641f0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.772+0000 7f3468edf640 1 --2- 172.21.15.45:0/64330646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34640cdf10 0x7f34641f0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.772+0000 7f3468edf640 1 -- 172.21.15.45:0/64330646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3460009b30 con 0x7f34640661f0 2023-11-28T20:33:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.772+0000 7f3468edf640 1 --2- 172.21.15.45:0/64330646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34640661f0 0x7f3464156580 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3460009f20 tx=0x7f346000c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.773+0000 7f34597fa640 1 -- 172.21.15.45:0/64330646 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3460014480 con 0x7f34640661f0 2023-11-28T20:33:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.773+0000 7f34597fa640 1 -- 172.21.15.45:0/64330646 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346000f040 con 0x7f34640661f0 2023-11-28T20:33:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.773+0000 7f34597fa640 1 -- 172.21.15.45:0/64330646 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34600147e0 con 0x7f34640661f0 2023-11-28T20:33:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.773+0000 7f346b143640 1 -- 172.21.15.45:0/64330646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3454009400 con 0x7f34640661f0 2023-11-28T20:33:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.773+0000 7f346b143640 1 -- 172.21.15.45:0/64330646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34640b5ac0 con 0x7f34640661f0 2023-11-28T20:33:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.773+0000 7f34597fa640 1 -- 172.21.15.45:0/64330646 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3460014c40 con 0x7f34640661f0 2023-11-28T20:33:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.773+0000 7f34597fa640 1 -- 172.21.15.45:0/64330646 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f346001d410 con 0x7f34640661f0 2023-11-28T20:33:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.774+0000 7f346b143640 1 -- 172.21.15.45:0/64330646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34640c9fb0 con 0x7f34640661f0 2023-11-28T20:33:37.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.776+0000 7f34597fa640 1 -- 172.21.15.45:0/64330646 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f346001d5d0 con 0x7f34640661f0 2023-11-28T20:33:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.842+0000 7f346b143640 1 -- 172.21.15.45:0/64330646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3464072bd0 con 0x7f34640661f0 2023-11-28T20:33:37.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.843+0000 7f34597fa640 1 -- 172.21.15.45:0/64330646 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f3460027480 con 0x7f34640661f0 2023-11-28T20:33:37.847 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:33:37.848 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":10,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:37.443264+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:33:37.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.844+0000 7f346b143640 1 -- 172.21.15.45:0/64330646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34640661f0 msgr2=0x7f3464156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.844+0000 7f346b143640 1 --2- 172.21.15.45:0/64330646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34640661f0 0x7f3464156580 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3460009f20 tx=0x7f346000c550 comp rx=0 tx=0).stop 2023-11-28T20:33:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.844+0000 7f346b143640 1 -- 172.21.15.45:0/64330646 shutdown_connections 2023-11-28T20:33:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.844+0000 7f346b143640 1 --2- 172.21.15.45:0/64330646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34640ce460 0x7f34640c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.844+0000 7f346b143640 1 --2- 172.21.15.45:0/64330646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34640cdf10 0x7f34641f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.844+0000 7f346b143640 1 --2- 172.21.15.45:0/64330646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34640661f0 0x7f3464156580 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.844+0000 7f346b143640 1 -- 172.21.15.45:0/64330646 >> 172.21.15.45:0/64330646 conn(0x7f34640cc970 msgr2=0x7f34640b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.844+0000 7f346b143640 1 -- 172.21.15.45:0/64330646 shutdown_connections 2023-11-28T20:33:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:37.844+0000 7f346b143640 1 -- 172.21.15.45:0/64330646 wait complete. 2023-11-28T20:33:37.863 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-28T20:33:37.864 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-28T20:33:38.197 INFO:tasks.ceph.osd.1.smithi045.stderr:2023-11-28T20:33:38.195+0000 7ff5093f9500 -1 Falling back to public interface 2023-11-28T20:33:38.203 INFO:tasks.ceph.osd.3.smithi045.stderr:2023-11-28T20:33:38.201+0000 7f2c62ab6500 -1 Falling back to public interface 2023-11-28T20:33:38.207 INFO:tasks.ceph.osd.4.smithi190.stderr:2023-11-28T20:33:38.205+0000 7fe88cd86500 -1 Falling back to public interface 2023-11-28T20:33:38.212 INFO:tasks.ceph.osd.0.smithi045.stderr:2023-11-28T20:33:38.210+0000 7f52a9a17500 -1 Falling back to public interface 2023-11-28T20:33:38.213 INFO:tasks.ceph.osd.5.smithi190.stderr:2023-11-28T20:33:38.211+0000 7f0b5e60e500 -1 Falling back to public interface 2023-11-28T20:33:38.227 INFO:tasks.ceph.osd.2.smithi045.stderr:2023-11-28T20:33:38.225+0000 7f499594a500 -1 Falling back to public interface 2023-11-28T20:33:38.239 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-28T20:33:38.238+0000 7f2a2b739500 -1 Falling back to public interface 2023-11-28T20:33:38.243 INFO:tasks.ceph.osd.6.smithi190.stderr:2023-11-28T20:33:38.242+0000 7f92b6937500 -1 Falling back to public interface 2023-11-28T20:33:38.615 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:38.613+0000 7fb587222040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:33:38.618 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:38.617+0000 7f529be10040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:33:38.669 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T20:33:38.829 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.827+0000 7f326fc00640 1 Processor -- start 2023-11-28T20:33:38.830 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.827+0000 7f326fc00640 1 -- start start 2023-11-28T20:33:38.831 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.828+0000 7f326fc00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f32680d2020 0x7f32680d0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:38.831 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.828+0000 7f326fc00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32680ce120 con 0x7f326815d970 2023-11-28T20:33:38.831 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.828+0000 7f326fc00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32680ce3c0 con 0x7f32680cf460 2023-11-28T20:33:38.831 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.828+0000 7f326fc00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32680cf890 con 0x7f32680d2020 2023-11-28T20:33:38.832 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.828+0000 7f326d99c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f326815d970 0x7f32680d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58828/0 (socket says 172.21.15.45:58828) 2023-11-28T20:33:38.832 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.828+0000 7f326d99c640 1 -- 172.21.15.45:0/2186468006 learned_addr learned my addr 172.21.15.45:0/2186468006 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:33:38.832 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.828+0000 7f326d19b640 1 --2- 172.21.15.45:0/2186468006 >> v2:172.21.15.45:3300/0 conn(0x7f32680d2020 0x7f32680d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:38.832 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.829+0000 7f326d19b640 1 -- 172.21.15.45:0/2186468006 >> v1:172.21.15.190:6789/0 conn(0x7f32680cf460 legacy=0x7f32680b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:38.832 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.829+0000 7f326d19b640 1 -- 172.21.15.45:0/2186468006 >> v1:172.21.15.45:6789/0 conn(0x7f326815d970 legacy=0x7f32680d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:38.833 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.829+0000 7f326d19b640 1 -- 172.21.15.45:0/2186468006 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32680ce6a0 con 0x7f32680d2020 2023-11-28T20:33:38.833 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.829+0000 7f326c99a640 1 -- 172.21.15.45:0/2186468006 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718887820 0 0) 0x7f32680ce120 con 0x7f326815d970 2023-11-28T20:33:38.834 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.830+0000 7f326d19b640 1 --2- 172.21.15.45:0/2186468006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32680d2020 0x7f32680d0570 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3264009d90 tx=0x7f326402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38abfea9650916ed server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.830+0000 7f326c99a640 1 -- 172.21.15.45:0/2186468006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326402cd00 con 0x7f32680d2020 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.830+0000 7f326c99a640 1 -- 172.21.15.45:0/2186468006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326402ce60 con 0x7f32680d2020 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.830+0000 7f326c99a640 1 -- 172.21.15.45:0/2186468006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264034660 con 0x7f32680d2020 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.830+0000 7f326fc00640 1 -- 172.21.15.45:0/2186468006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32680d2020 msgr2=0x7f32680d0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.830+0000 7f326fc00640 1 --2- 172.21.15.45:0/2186468006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32680d2020 0x7f32680d0570 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3264009d90 tx=0x7f326402bf50 comp rx=0 tx=0).stop 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.831+0000 7f326fc00640 1 -- 172.21.15.45:0/2186468006 shutdown_connections 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.831+0000 7f326fc00640 1 --2- 172.21.15.45:0/2186468006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32680d2020 0x7f32680d0570 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.831+0000 7f326fc00640 1 -- 172.21.15.45:0/2186468006 >> 172.21.15.45:0/2186468006 conn(0x7f326809c5e0 msgr2=0x7f32680d3330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.831+0000 7f326fc00640 1 -- 172.21.15.45:0/2186468006 shutdown_connections 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.831+0000 7f326fc00640 1 -- 172.21.15.45:0/2186468006 wait complete. 2023-11-28T20:33:38.835 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.832+0000 7f326fc00640 1 Processor -- start 2023-11-28T20:33:38.836 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.832+0000 7f326fc00640 1 -- start start 2023-11-28T20:33:38.837 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.833+0000 7f326fc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32680cf460 0x7f326815dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.833+0000 7f326fc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680d2020 0x7f32681e83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.833+0000 7f326fc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d970 0x7f32681433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.833+0000 7f326fc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326815e3a0 con 0x7f326815d970 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.833+0000 7f326fc00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268161020 con 0x7f32680d2020 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.833+0000 7f326fc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3268160a60 con 0x7f32680cf460 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326e19d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d970 0x7f32681433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326e19d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d970 0x7f32681433c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47296/0 (socket says 172.21.15.45:47296) 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326e19d640 1 -- 172.21.15.45:0/3407430825 learned_addr learned my addr 172.21.15.45:0/3407430825 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326d99c640 1 --2- 172.21.15.45:0/3407430825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32680cf460 0x7f326815dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326d19b640 1 --2- 172.21.15.45:0/3407430825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680d2020 0x7f32681e83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:38.838 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326e19d640 1 -- 172.21.15.45:0/3407430825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32680cf460 msgr2=0x7f326815dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326e19d640 1 --2- 172.21.15.45:0/3407430825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32680cf460 0x7f326815dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326e19d640 1 -- 172.21.15.45:0/3407430825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680d2020 msgr2=0x7f32681e83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326e19d640 1 --2- 172.21.15.45:0/3407430825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680d2020 0x7f32681e83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.834+0000 7f326e19d640 1 -- 172.21.15.45:0/3407430825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3264009a70 con 0x7f326815d970 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.835+0000 7f326e19d640 1 --2- 172.21.15.45:0/3407430825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d970 0x7f32681433c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3260001eb0 tx=0x7f326000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.835+0000 7f325e7fc640 1 -- 172.21.15.45:0/3407430825 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32600052d0 con 0x7f326815d970 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.835+0000 7f325e7fc640 1 -- 172.21.15.45:0/3407430825 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3260005430 con 0x7f326815d970 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.835+0000 7f325e7fc640 1 -- 172.21.15.45:0/3407430825 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260003810 con 0x7f326815d970 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.835+0000 7f326fc00640 1 -- 172.21.15.45:0/3407430825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3268160d40 con 0x7f326815d970 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.835+0000 7f326fc00640 1 -- 172.21.15.45:0/3407430825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32681e88f0 con 0x7f326815d970 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.835+0000 7f325e7fc640 1 -- 172.21.15.45:0/3407430825 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3260003c70 con 0x7f326815d970 2023-11-28T20:33:38.839 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.835+0000 7f325e7fc640 1 -- 172.21.15.45:0/3407430825 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f3260012420 con 0x7f326815d970 2023-11-28T20:33:38.840 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.836+0000 7f326fc00640 1 -- 172.21.15.45:0/3407430825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32681e8ee0 con 0x7f326815d970 2023-11-28T20:33:38.840 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.839+0000 7f325e7fc640 1 -- 172.21.15.45:0/3407430825 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f32681e8ee0 con 0x7f326815d970 2023-11-28T20:33:38.905 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.903+0000 7f326fc00640 1 -- 172.21.15.45:0/3407430825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32680d3da0 con 0x7f326815d970 2023-11-28T20:33:38.906 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.904+0000 7f325e7fc640 1 -- 172.21.15.45:0/3407430825 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f3260017070 con 0x7f326815d970 2023-11-28T20:33:38.906 INFO:teuthology.misc.health.smithi045.stdout: 2023-11-28T20:33:38.907 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:37.443264+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:33:38.908 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.905+0000 7f326fc00640 1 -- 172.21.15.45:0/3407430825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d970 msgr2=0x7f32681433c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:38.908 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.905+0000 7f326fc00640 1 --2- 172.21.15.45:0/3407430825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d970 0x7f32681433c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3260001eb0 tx=0x7f326000c890 comp rx=0 tx=0).stop 2023-11-28T20:33:38.908 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.905+0000 7f326fc00640 1 -- 172.21.15.45:0/3407430825 shutdown_connections 2023-11-28T20:33:38.908 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.905+0000 7f326fc00640 1 --2- 172.21.15.45:0/3407430825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d970 0x7f32681433c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:38.908 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.905+0000 7f326fc00640 1 --2- 172.21.15.45:0/3407430825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680d2020 0x7f32681e83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:38.909 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.905+0000 7f326fc00640 1 --2- 172.21.15.45:0/3407430825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32680cf460 0x7f326815dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:38.909 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.906+0000 7f326fc00640 1 -- 172.21.15.45:0/3407430825 >> 172.21.15.45:0/3407430825 conn(0x7f326809c5e0 msgr2=0x7f32680b76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:38.909 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.906+0000 7f326fc00640 1 -- 172.21.15.45:0/3407430825 shutdown_connections 2023-11-28T20:33:38.909 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:38.906+0000 7f326fc00640 1 -- 172.21.15.45:0/3407430825 wait complete. 2023-11-28T20:33:38.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-28T20:33:38.926 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:38.925+0000 7f529be10040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:33:38.943 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:38.941+0000 7fb587222040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:33:38.986 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:38.984+0000 7f529be10040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:33:39.006 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:39.003+0000 7fb587222040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:33:39.102 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:39.101+0000 7f529be10040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:33:39.126 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:33:39.123+0000 7fb587222040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:33:40.144 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:40.143+0000 7f523cef7640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-28T20:33:40.144 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:33:40.144+0000 7f523cef7640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-28T20:33:41.563 INFO:tasks.ceph.osd.4.smithi190.stderr:2023-11-28T20:33:41.562+0000 7fe88cd86500 -1 osd.4 0 log_to_monitors true 2023-11-28T20:33:41.740 INFO:tasks.ceph.osd.5.smithi190.stderr:2023-11-28T20:33:41.739+0000 7f0b5e60e500 -1 osd.5 0 log_to_monitors true 2023-11-28T20:33:41.792 INFO:tasks.ceph.osd.3.smithi045.stderr:2023-11-28T20:33:41.790+0000 7f2c62ab6500 -1 osd.3 0 log_to_monitors true 2023-11-28T20:33:41.794 INFO:tasks.ceph.osd.0.smithi045.stderr:2023-11-28T20:33:41.792+0000 7f52a9a17500 -1 osd.0 0 log_to_monitors true 2023-11-28T20:33:41.794 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-28T20:33:41.793+0000 7f2a2b739500 -1 osd.7 0 log_to_monitors true 2023-11-28T20:33:41.817 INFO:tasks.ceph.osd.2.smithi045.stderr:2023-11-28T20:33:41.815+0000 7f499594a500 -1 osd.2 0 log_to_monitors true 2023-11-28T20:33:41.832 INFO:tasks.ceph.osd.6.smithi190.stderr:2023-11-28T20:33:41.831+0000 7f92b6937500 -1 osd.6 0 log_to_monitors true 2023-11-28T20:33:41.976 INFO:tasks.ceph.osd.1.smithi045.stderr:2023-11-28T20:33:41.973+0000 7ff5093f9500 -1 osd.1 0 log_to_monitors true 2023-11-28T20:33:44.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:33:44.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:33:44.957 INFO:tasks.ceph.osd.2.smithi045.stderr:2023-11-28T20:33:44.954+0000 7f49918d8640 -1 osd.2 0 waiting for initial osdmap 2023-11-28T20:33:44.960 INFO:tasks.ceph.osd.1.smithi045.stderr:2023-11-28T20:33:44.958+0000 7ff505b9b640 -1 osd.1 0 waiting for initial osdmap 2023-11-28T20:33:44.965 INFO:tasks.ceph.osd.2.smithi045.stderr:2023-11-28T20:33:44.963+0000 7f49896fe640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T20:33:44.966 INFO:tasks.ceph.osd.1.smithi045.stderr:2023-11-28T20:33:44.964+0000 7ff4fd1ad640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T20:33:44.980 INFO:tasks.ceph.osd.0.smithi045.stderr:2023-11-28T20:33:44.978+0000 7f52a53e5640 -1 osd.0 0 waiting for initial osdmap 2023-11-28T20:33:44.987 INFO:tasks.ceph.osd.0.smithi045.stderr:2023-11-28T20:33:44.985+0000 7f529d20b640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T20:33:44.990 INFO:tasks.ceph.osd.3.smithi045.stderr:2023-11-28T20:33:44.989+0000 7f2c5ea46640 -1 osd.3 0 waiting for initial osdmap 2023-11-28T20:33:45.009 INFO:tasks.ceph.osd.3.smithi045.stderr:2023-11-28T20:33:45.007+0000 7f2c5686c640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T20:33:45.453 INFO:tasks.ceph.osd.6.smithi190.stderr:2023-11-28T20:33:45.451+0000 7f92b28c5640 -1 osd.6 0 waiting for initial osdmap 2023-11-28T20:33:45.472 INFO:tasks.ceph.osd.6.smithi190.stderr:2023-11-28T20:33:45.470+0000 7f92aa6eb640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T20:33:45.485 INFO:tasks.ceph.osd.4.smithi190.stderr:2023-11-28T20:33:45.483+0000 7fe88952a640 -1 osd.4 0 waiting for initial osdmap 2023-11-28T20:33:45.494 INFO:tasks.ceph.osd.4.smithi190.stderr:2023-11-28T20:33:45.492+0000 7fe880b3c640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T20:33:45.523 INFO:tasks.ceph.osd.5.smithi190.stderr:2023-11-28T20:33:45.521+0000 7f0b5adb0640 -1 osd.5 0 waiting for initial osdmap 2023-11-28T20:33:45.532 INFO:tasks.ceph.osd.5.smithi190.stderr:2023-11-28T20:33:45.530+0000 7f0b523c2640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T20:33:45.592 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-28T20:33:45.590+0000 7f2a276c7640 -1 osd.7 0 waiting for initial osdmap 2023-11-28T20:33:45.599 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-28T20:33:45.597+0000 7f2a1f4ed640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T20:33:45.727 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T20:33:45.925 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.923+0000 7fe1ea4e6640 1 Processor -- start 2023-11-28T20:33:45.925 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.923+0000 7fe1ea4e6640 1 -- start start 2023-11-28T20:33:45.927 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.924+0000 7fe1ea4e6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe1e4143e10 0x7fe1e41446f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:45.927 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.925+0000 7fe1ea4e6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e40b1b10 con 0x7fe1e40cde30 2023-11-28T20:33:45.927 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.925+0000 7fe1ea4e6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e40b23b0 con 0x7fe1e40cefe0 2023-11-28T20:33:45.927 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.925+0000 7fe1ea4e6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e40ce580 con 0x7fe1e4143e10 2023-11-28T20:33:45.927 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.925+0000 7fe1e8ce3640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe1e40cde30 0x7fe1e415b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50524/0 (socket says 172.21.15.45:50524) 2023-11-28T20:33:45.927 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.925+0000 7fe1e8ce3640 1 -- 172.21.15.45:0/1353739593 learned_addr learned my addr 172.21.15.45:0/1353739593 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:33:45.927 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.925+0000 7fe1e94e4640 1 --2- 172.21.15.45:0/1353739593 >> v2:172.21.15.45:3300/0 conn(0x7fe1e4143e10 0x7fe1e41446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:45.928 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.925+0000 7fe1dbfff640 1 -- 172.21.15.45:0/1353739593 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564606374 0 0) 0x7fe1e40b1b10 con 0x7fe1e40cde30 2023-11-28T20:33:45.928 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.925+0000 7fe1dbfff640 1 -- 172.21.15.45:0/1353739593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d0003520 con 0x7fe1e40cde30 2023-11-28T20:33:45.928 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.926+0000 7fe1e94e4640 1 -- 172.21.15.45:0/1353739593 >> v1:172.21.15.190:6789/0 conn(0x7fe1e40cefe0 legacy=0x7fe1e40b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:45.928 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.926+0000 7fe1e94e4640 1 -- 172.21.15.45:0/1353739593 >> v1:172.21.15.45:6789/0 conn(0x7fe1e40cde30 legacy=0x7fe1e415b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:45.928 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.926+0000 7fe1e94e4640 1 -- 172.21.15.45:0/1353739593 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e40b1d60 con 0x7fe1e4143e10 2023-11-28T20:33:45.929 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.927+0000 7fe1e94e4640 1 --2- 172.21.15.45:0/1353739593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e4143e10 0x7fe1e41446f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00097a0 tx=0x7fe1e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46724c1ad9fcde04 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:45.929 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.927+0000 7fe1dbfff640 1 -- 172.21.15.45:0/1353739593 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e002e070 con 0x7fe1e4143e10 2023-11-28T20:33:45.929 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.927+0000 7fe1dbfff640 1 -- 172.21.15.45:0/1353739593 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e00315a0 con 0x7fe1e4143e10 2023-11-28T20:33:45.929 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.927+0000 7fe1dbfff640 1 -- 172.21.15.45:0/1353739593 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0031930 con 0x7fe1e4143e10 2023-11-28T20:33:45.930 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.927+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/1353739593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e4143e10 msgr2=0x7fe1e41446f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:45.930 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.927+0000 7fe1ea4e6640 1 --2- 172.21.15.45:0/1353739593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e4143e10 0x7fe1e41446f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00097a0 tx=0x7fe1e002b840 comp rx=0 tx=0).stop 2023-11-28T20:33:45.930 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.928+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/1353739593 shutdown_connections 2023-11-28T20:33:45.930 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.928+0000 7fe1ea4e6640 1 --2- 172.21.15.45:0/1353739593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e4143e10 0x7fe1e41446f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:45.931 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.928+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/1353739593 >> 172.21.15.45:0/1353739593 conn(0x7fe1e40b0c00 msgr2=0x7fe1e40b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:45.931 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.928+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/1353739593 shutdown_connections 2023-11-28T20:33:45.931 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.928+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/1353739593 wait complete. 2023-11-28T20:33:45.931 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.929+0000 7fe1ea4e6640 1 Processor -- start 2023-11-28T20:33:45.931 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.930+0000 7fe1ea4e6640 1 -- start start 2023-11-28T20:33:45.934 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.930+0000 7fe1ea4e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40cde30 0x7fe1e41f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:45.934 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.930+0000 7fe1ea4e6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40cefe0 0x7fe1e4153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:45.934 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1ea4e6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e4143e10 0x7fe1e413c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:45.934 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1ea4e6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4154610 con 0x7fe1e40cefe0 2023-11-28T20:33:45.934 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1ea4e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4157610 con 0x7fe1e40cde30 2023-11-28T20:33:45.934 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1ea4e6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4153d70 con 0x7fe1e4143e10 2023-11-28T20:33:45.934 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e8ce3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40cefe0 0x7fe1e4153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:45.934 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e8ce3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40cefe0 0x7fe1e4153670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39728/0 (socket says 172.21.15.45:39728) 2023-11-28T20:33:45.934 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e8ce3640 1 -- 172.21.15.45:0/2298069887 learned_addr learned my addr 172.21.15.45:0/2298069887 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e94e4640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40cde30 0x7fe1e41f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e9ce5640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e4143e10 0x7fe1e413c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e8ce3640 1 -- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e4143e10 msgr2=0x7fe1e413c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e8ce3640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e4143e10 0x7fe1e413c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e8ce3640 1 -- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40cde30 msgr2=0x7fe1e41f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e8ce3640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40cde30 0x7fe1e41f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e8ce3640 1 -- 172.21.15.45:0/2298069887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e0009400 con 0x7fe1e40cefe0 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e94e4640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40cde30 0x7fe1e41f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.931+0000 7fe1e9ce5640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e4143e10 0x7fe1e413c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.932+0000 7fe1e8ce3640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40cefe0 0x7fe1e4153670 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400c500 tx=0x7fe1d400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.932+0000 7fe1d9ffb640 1 -- 172.21.15.45:0/2298069887 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4005ce0 con 0x7fe1e40cefe0 2023-11-28T20:33:45.935 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.932+0000 7fe1d9ffb640 1 -- 172.21.15.45:0/2298069887 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d4005e40 con 0x7fe1e40cefe0 2023-11-28T20:33:45.936 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.932+0000 7fe1d9ffb640 1 -- 172.21.15.45:0/2298069887 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4007330 con 0x7fe1e40cefe0 2023-11-28T20:33:45.936 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.932+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e4154050 con 0x7fe1e40cefe0 2023-11-28T20:33:45.936 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.932+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e413cd20 con 0x7fe1e40cefe0 2023-11-28T20:33:45.936 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.933+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e41573e0 con 0x7fe1e40cefe0 2023-11-28T20:33:45.936 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.934+0000 7fe1d9ffb640 1 -- 172.21.15.45:0/2298069887 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fe1d4007550 con 0x7fe1e40cefe0 2023-11-28T20:33:45.937 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.935+0000 7fe1d9ffb640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c806adb0 0x7fe1c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:45.937 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.935+0000 7fe1d9ffb640 1 -- 172.21.15.45:0/2298069887 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fe1d4087df0 con 0x7fe1e40cefe0 2023-11-28T20:33:45.937 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.935+0000 7fe1e94e4640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c806adb0 0x7fe1c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:45.939 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.936+0000 7fe1e94e4640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c806adb0 0x7fe1c806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0036000 tx=0x7fe1e003d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:45.942 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:45.939+0000 7fe1d9ffb640 1 -- 172.21.15.45:0/2298069887 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1d4053dc0 con 0x7fe1e40cefe0 2023-11-28T20:33:46.145 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.143+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1e406bfe0 con 0x7fe1e40cefe0 2023-11-28T20:33:46.146 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.144+0000 7fe1d9ffb640 1 -- 172.21.15.45:0/2298069887 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10896 (secure 0 0 0) 0x7fe1d4017070 con 0x7fe1e40cefe0 2023-11-28T20:33:46.147 INFO:teuthology.misc.health.smithi045.stdout: 2023-11-28T20:33:46.147 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":13,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:45.955837+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:33:46.149 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c806adb0 msgr2=0x7fe1c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.149 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c806adb0 0x7fe1c806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0036000 tx=0x7fe1e003d040 comp rx=0 tx=0).stop 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40cefe0 msgr2=0x7fe1e4153670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40cefe0 0x7fe1e4153670 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400c500 tx=0x7fe1d400c9c0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 shutdown_connections 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c806adb0 0x7fe1c806d270 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0036000 tx=0x7fe1e003d040 comp rx=0 tx=0).stop 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e4143e10 0x7fe1e413c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40cefe0 0x7fe1e4153670 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 --2- 172.21.15.45:0/2298069887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40cde30 0x7fe1e41f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 >> 172.21.15.45:0/2298069887 conn(0x7fe1e40b0c00 msgr2=0x7fe1e415a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 shutdown_connections 2023-11-28T20:33:46.150 INFO:teuthology.misc.health.smithi045.stderr:2023-11-28T20:33:46.147+0000 7fe1ea4e6640 1 -- 172.21.15.45:0/2298069887 wait complete. 2023-11-28T20:33:46.163 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-28T20:33:46.163 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-28T20:33:46.164 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T20:33:46.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T20:33:46.170 INFO:tasks.ceph.mds.a:Started 2023-11-28T20:33:46.170 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T20:33:46.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T20:33:46.176 INFO:tasks.ceph.mds.c:Started 2023-11-28T20:33:46.176 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T20:33:46.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T20:33:46.186 INFO:tasks.ceph.mds.b:Started 2023-11-28T20:33:46.186 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T20:33:46.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T20:33:46.194 INFO:tasks.ceph.mds.d:Started 2023-11-28T20:33:46.195 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-11-28T20:33:46.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:33:46.319 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:33:46.327 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:33:46.345 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T20:33:46.345 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:33:46.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.375+0000 7eff22c10640 1 Processor -- start 2023-11-28T20:33:46.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.375+0000 7eff22c10640 1 -- start start 2023-11-28T20:33:46.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff22c10640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff1c15d320 0x7eff1c0ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff22c10640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c0add40 0x7eff1c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff22c10640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1c0ce710 con 0x7eff1c0b1170 2023-11-28T20:33:46.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff22c10640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff1c0ae5a0 con 0x7eff1c15d320 2023-11-28T20:33:46.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff22c10640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c0b1600 con 0x7eff1c0add40 2023-11-28T20:33:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff1bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c0add40 0x7eff1c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff1bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c0add40 0x7eff1c0b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:38338/0 (socket says 172.21.15.45:38338) 2023-11-28T20:33:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff1bfff640 1 -- 172.21.15.45:0/2010610960 learned_addr learned my addr 172.21.15.45:0/2010610960 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff209ac640 1 --2- 172.21.15.45:0/2010610960 >> v2:172.21.15.190:3300/0 conn(0x7eff1c15d320 0x7eff1c0ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff1bfff640 1 -- 172.21.15.45:0/2010610960 >> v2:172.21.15.190:3300/0 conn(0x7eff1c15d320 msgr2=0x7eff1c0ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff1bfff640 1 --2- 172.21.15.45:0/2010610960 >> v2:172.21.15.190:3300/0 conn(0x7eff1c15d320 0x7eff1c0ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff1bfff640 1 -- 172.21.15.45:0/2010610960 >> v1:172.21.15.190:6789/0 conn(0x7eff1c0b1170 legacy=0x7eff1c0cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff1bfff640 1 -- 172.21.15.45:0/2010610960 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c0721a0 con 0x7eff1c0add40 2023-11-28T20:33:46.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.377+0000 7eff1b7fe640 1 -- 172.21.15.45:0/2010610960 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031947227 0 0) 0x7eff1c0ce710 con 0x7eff1c0b1170 2023-11-28T20:33:46.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff1bfff640 1 --2- 172.21.15.45:0/2010610960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c0add40 0x7eff1c0b0c30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7eff08009870 tx=0x7eff0802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=848b8983b7b65e03 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:46.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff1b7fe640 1 -- 172.21.15.45:0/2010610960 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff08030840 con 0x7eff1c0add40 2023-11-28T20:33:46.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff1b7fe640 1 -- 172.21.15.45:0/2010610960 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff080309a0 con 0x7eff1c0add40 2023-11-28T20:33:46.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff1b7fe640 1 -- 172.21.15.45:0/2010610960 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff08030d30 con 0x7eff1c0add40 2023-11-28T20:33:46.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff22c10640 1 -- 172.21.15.45:0/2010610960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c0add40 msgr2=0x7eff1c0b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff22c10640 1 --2- 172.21.15.45:0/2010610960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c0add40 0x7eff1c0b0c30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7eff08009870 tx=0x7eff0802b980 comp rx=0 tx=0).stop 2023-11-28T20:33:46.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff22c10640 1 -- 172.21.15.45:0/2010610960 shutdown_connections 2023-11-28T20:33:46.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff22c10640 1 --2- 172.21.15.45:0/2010610960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c0add40 0x7eff1c0b0c30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff22c10640 1 --2- 172.21.15.45:0/2010610960 >> v2:172.21.15.190:3300/0 conn(0x7eff1c15d320 0x7eff1c0ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.379+0000 7eff22c10640 1 -- 172.21.15.45:0/2010610960 >> 172.21.15.45:0/2010610960 conn(0x7eff1c06a500 msgr2=0x7eff1c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:46.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.380+0000 7eff22c10640 1 -- 172.21.15.45:0/2010610960 shutdown_connections 2023-11-28T20:33:46.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.380+0000 7eff22c10640 1 -- 172.21.15.45:0/2010610960 wait complete. 2023-11-28T20:33:46.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.381+0000 7eff22c10640 1 Processor -- start 2023-11-28T20:33:46.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.381+0000 7eff22c10640 1 -- start start 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.382+0000 7eff22c10640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff1c0add40 0x7eff1c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.382+0000 7eff22c10640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff1c0b1170 0x7eff1c1591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.382+0000 7eff22c10640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c15d320 0x7eff1c159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.382+0000 7eff22c10640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c159c60 con 0x7eff1c0add40 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.382+0000 7eff22c10640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c0b1d90 con 0x7eff1c0b1170 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.382+0000 7eff22c10640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c1e73e0 con 0x7eff1c15d320 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.382+0000 7eff209ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff1c0add40 0x7eff1c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff209ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff1c0add40 0x7eff1c14f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39770/0 (socket says 172.21.15.45:39770) 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff209ac640 1 -- 172.21.15.45:0/4092213138 learned_addr learned my addr 172.21.15.45:0/4092213138 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff211ad640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c15d320 0x7eff1c159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff1bfff640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff1c0b1170 0x7eff1c1591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff209ac640 1 -- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c15d320 msgr2=0x7eff1c159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff209ac640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c15d320 0x7eff1c159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff209ac640 1 -- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff1c0b1170 msgr2=0x7eff1c1591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff209ac640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff1c0b1170 0x7eff1c1591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff209ac640 1 -- 172.21.15.45:0/4092213138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff080094d0 con 0x7eff1c0add40 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff209ac640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff1c0add40 0x7eff1c14f850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff1000e870 tx=0x7eff1000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff197fa640 1 -- 172.21.15.45:0/4092213138 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff100094f0 con 0x7eff1c0add40 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff197fa640 1 -- 172.21.15.45:0/4092213138 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1000c940 con 0x7eff1c0add40 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.383+0000 7eff197fa640 1 -- 172.21.15.45:0/4092213138 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1000ccf0 con 0x7eff1c0add40 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.384+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1c159e80 con 0x7eff1c0add40 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.384+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1c1e7960 con 0x7eff1c0add40 2023-11-28T20:33:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.385+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1c0733d0 con 0x7eff1c0add40 2023-11-28T20:33:46.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.385+0000 7eff197fa640 1 -- 172.21.15.45:0/4092213138 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7eff10013470 con 0x7eff1c0add40 2023-11-28T20:33:46.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.387+0000 7eff197fa640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7eff0006ade0 0x7eff0006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.387+0000 7eff197fa640 1 -- 172.21.15.45:0/4092213138 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7eff10089050 con 0x7eff1c0add40 2023-11-28T20:33:46.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.387+0000 7eff1bfff640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7eff0006ade0 0x7eff0006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.388+0000 7eff1bfff640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7eff0006ade0 0x7eff0006d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7eff08036000 tx=0x7eff08008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:46.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.392+0000 7eff197fa640 1 -- 172.21.15.45:0/4092213138 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff10054760 con 0x7eff1c0add40 2023-11-28T20:33:46.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.584+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff1c0ad250 con 0x7eff1c0add40 2023-11-28T20:33:46.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.585+0000 7eff197fa640 1 -- 172.21.15.45:0/4092213138 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10896 (secure 0 0 0) 0x7eff10057d80 con 0x7eff1c0add40 2023-11-28T20:33:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:33:46.588 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":13,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:45.955837+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:33:46.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.587+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7eff0006ade0 msgr2=0x7eff0006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.587+0000 7eff22c10640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7eff0006ade0 0x7eff0006d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7eff08036000 tx=0x7eff08008030 comp rx=0 tx=0).stop 2023-11-28T20:33:46.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.587+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff1c0add40 msgr2=0x7eff1c14f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.587+0000 7eff22c10640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff1c0add40 0x7eff1c14f850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff1000e870 tx=0x7eff1000ed30 comp rx=0 tx=0).stop 2023-11-28T20:33:46.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.588+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 shutdown_connections 2023-11-28T20:33:46.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.588+0000 7eff22c10640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7eff0006ade0 0x7eff0006d2a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.588+0000 7eff22c10640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff1c15d320 0x7eff1c159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.588+0000 7eff22c10640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff1c0b1170 0x7eff1c1591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.588+0000 7eff22c10640 1 --2- 172.21.15.45:0/4092213138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff1c0add40 0x7eff1c14f850 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.588+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 >> 172.21.15.45:0/4092213138 conn(0x7eff1c06a500 msgr2=0x7eff1c15c530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:46.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.588+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 shutdown_connections 2023-11-28T20:33:46.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.588+0000 7eff22c10640 1 -- 172.21.15.45:0/4092213138 wait complete. 2023-11-28T20:33:46.607 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2023-11-28T20:33:46.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:33:46.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.816+0000 7fe6bffff640 1 Processor -- start 2023-11-28T20:33:46.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.816+0000 7fe6bffff640 1 -- start start 2023-11-28T20:33:46.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.818+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 0x7fe6c00abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.818+0000 7fe6bffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe6c00ce800 0x7fe6c00b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.818+0000 7fe6bffff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c015c3c0 con 0x7fe6c00ae540 2023-11-28T20:33:46.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.818+0000 7fe6bffff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c0155b60 con 0x7fe6c00ce800 2023-11-28T20:33:46.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.818+0000 7fe6bffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c015ba90 con 0x7fe6c0155780 2023-11-28T20:33:46.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.818+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe6c00ce800 0x7fe6c00b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.818+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe6c00ce800 0x7fe6c00b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39774/0 (socket says 172.21.15.45:39774) 2023-11-28T20:33:46.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.818+0000 7fe6bf7fe640 1 -- 172.21.15.45:0/2579553820 learned_addr learned my addr 172.21.15.45:0/2579553820 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:46.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.818+0000 7fe6beffd640 1 --2- 172.21.15.45:0/2579553820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 0x7fe6c00abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.819+0000 7fe6bf7fe640 1 -- 172.21.15.45:0/2579553820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 msgr2=0x7fe6c00abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:46.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.819+0000 7fe6bf7fe640 1 --2- 172.21.15.45:0/2579553820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 0x7fe6c00abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.819+0000 7fe6bf7fe640 1 -- 172.21.15.45:0/2579553820 >> v1:172.21.15.190:6789/0 conn(0x7fe6c00ae540 legacy=0x7fe6c00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.819+0000 7fe6bf7fe640 1 -- 172.21.15.45:0/2579553820 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c015bd70 con 0x7fe6c00ce800 2023-11-28T20:33:46.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.820+0000 7fe6bf7fe640 1 --2- 172.21.15.45:0/2579553820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 0x7fe6c00b20a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009e50 tx=0x7fe6b802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acf8e18b5b69dda server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:46.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.820+0000 7fe6bdffb640 1 -- 172.21.15.45:0/2579553820 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b802cb00 con 0x7fe6c00ce800 2023-11-28T20:33:46.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.820+0000 7fe6bdffb640 1 -- 172.21.15.45:0/2579553820 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b802cc60 con 0x7fe6c00ce800 2023-11-28T20:33:46.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.820+0000 7fe6bdffb640 1 -- 172.21.15.45:0/2579553820 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80343f0 con 0x7fe6c00ce800 2023-11-28T20:33:46.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.821+0000 7fe6bffff640 1 -- 172.21.15.45:0/2579553820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 msgr2=0x7fe6c00b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.821+0000 7fe6bffff640 1 --2- 172.21.15.45:0/2579553820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 0x7fe6c00b20a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009e50 tx=0x7fe6b802bf60 comp rx=0 tx=0).stop 2023-11-28T20:33:46.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.821+0000 7fe6bffff640 1 -- 172.21.15.45:0/2579553820 shutdown_connections 2023-11-28T20:33:46.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.821+0000 7fe6bffff640 1 --2- 172.21.15.45:0/2579553820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 0x7fe6c00b20a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.821+0000 7fe6bffff640 1 --2- 172.21.15.45:0/2579553820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 0x7fe6c00abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.821+0000 7fe6bffff640 1 -- 172.21.15.45:0/2579553820 >> 172.21.15.45:0/2579553820 conn(0x7fe6c006ab30 msgr2=0x7fe6c00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:46.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.821+0000 7fe6bffff640 1 -- 172.21.15.45:0/2579553820 shutdown_connections 2023-11-28T20:33:46.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.821+0000 7fe6bffff640 1 -- 172.21.15.45:0/2579553820 wait complete. 2023-11-28T20:33:46.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.823+0000 7fe6bffff640 1 Processor -- start 2023-11-28T20:33:46.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.823+0000 7fe6bffff640 1 -- start start 2023-11-28T20:33:46.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6c00ae540 0x7fe6c0147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 0x7fe6c01e7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 0x7fe6c0157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6bffff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c01e5170 con 0x7fe6c00ce800 2023-11-28T20:33:46.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6bffff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c01e5450 con 0x7fe6c00ae540 2023-11-28T20:33:46.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6bffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c01e5730 con 0x7fe6c0155780 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6be7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 0x7fe6c01e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6be7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 0x7fe6c01e7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39776/0 (socket says 172.21.15.45:39776) 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6be7fc640 1 -- 172.21.15.45:0/2032483571 learned_addr learned my addr 172.21.15.45:0/2032483571 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6beffd640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6c00ae540 0x7fe6c0147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6bf7fe640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 0x7fe6c0157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6be7fc640 1 -- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 msgr2=0x7fe6c0157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6be7fc640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 0x7fe6c0157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6be7fc640 1 -- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6c00ae540 msgr2=0x7fe6c0147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6be7fc640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6c00ae540 0x7fe6c0147410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:46.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.824+0000 7fe6be7fc640 1 -- 172.21.15.45:0/2032483571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b8009b30 con 0x7fe6c00ce800 2023-11-28T20:33:46.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.825+0000 7fe6bf7fe640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 0x7fe6c0157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:46.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.825+0000 7fe6beffd640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6c00ae540 0x7fe6c0147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:46.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.825+0000 7fe6be7fc640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 0x7fe6c01e7170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400c400 tx=0x7fe6b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:46.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.825+0000 7fe6a77fe640 1 -- 172.21.15.45:0/2032483571 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40052d0 con 0x7fe6c00ce800 2023-11-28T20:33:46.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.825+0000 7fe6a77fe640 1 -- 172.21.15.45:0/2032483571 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b4005450 con 0x7fe6c00ce800 2023-11-28T20:33:46.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.825+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c01e5a10 con 0x7fe6c00ce800 2023-11-28T20:33:46.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.825+0000 7fe6a77fe640 1 -- 172.21.15.45:0/2032483571 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4007260 con 0x7fe6c00ce800 2023-11-28T20:33:46.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.825+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c01e4dc0 con 0x7fe6c00ce800 2023-11-28T20:33:46.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.826+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c006d760 con 0x7fe6c00ce800 2023-11-28T20:33:46.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.827+0000 7fe6a77fe640 1 -- 172.21.15.45:0/2032483571 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fe6b40073c0 con 0x7fe6c00ce800 2023-11-28T20:33:46.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.828+0000 7fe6a77fe640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe6ac06ae00 0x7fe6ac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:46.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.828+0000 7fe6a77fe640 1 -- 172.21.15.45:0/2032483571 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fe6b4088300 con 0x7fe6c00ce800 2023-11-28T20:33:46.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.829+0000 7fe6beffd640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe6ac06ae00 0x7fe6ac06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:46.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.830+0000 7fe6beffd640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe6ac06ae00 0x7fe6ac06d2c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0006fd0 tx=0x7fe6b0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:46.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:46.833+0000 7fe6a77fe640 1 -- 172.21.15.45:0/2032483571 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6b4053ad0 con 0x7fe6c00ce800 2023-11-28T20:33:47.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.064+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6c004e920 con 0x7fe6c00ce800 2023-11-28T20:33:47.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.065+0000 7fe6a77fe640 1 -- 172.21.15.45:0/2032483571 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fe6b40570f0 con 0x7fe6c00ce800 2023-11-28T20:33:47.067 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:33:47.067 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-11-28T20:33:47.068 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 2 2023-11-28T20:33:47.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe6ac06ae00 msgr2=0x7fe6ac06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:47.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe6ac06ae00 0x7fe6ac06d2c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0006fd0 tx=0x7fe6b0008040 comp rx=0 tx=0).stop 2023-11-28T20:33:47.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 msgr2=0x7fe6c01e7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:47.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 0x7fe6c01e7170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400c400 tx=0x7fe6b400c8c0 comp rx=0 tx=0).stop 2023-11-28T20:33:47.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 shutdown_connections 2023-11-28T20:33:47.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe6ac06ae00 0x7fe6ac06d2c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:47.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6c0155780 0x7fe6c0157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:47.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6c00ce800 0x7fe6c01e7170 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:47.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 --2- 172.21.15.45:0/2032483571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6c00ae540 0x7fe6c0147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:47.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 >> 172.21.15.45:0/2032483571 conn(0x7fe6c006ab30 msgr2=0x7fe6c015df70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:47.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 shutdown_connections 2023-11-28T20:33:47.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:47.068+0000 7fe6bffff640 1 -- 172.21.15.45:0/2032483571 wait complete. 2023-11-28T20:33:49.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:33:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.317+0000 7f79ef181640 1 Processor -- start 2023-11-28T20:33:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.317+0000 7f79ef181640 1 -- start start 2023-11-28T20:33:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.319+0000 7f79ef181640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e8071e20 0x7f79e80b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.319+0000 7f79ef181640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f79e806b510 0x7f79e806b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.319+0000 7f79ef181640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e80b2ed0 con 0x7f79e80ae6e0 2023-11-28T20:33:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.319+0000 7f79ef181640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e80aeac0 con 0x7f79e806b510 2023-11-28T20:33:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.319+0000 7f79ef181640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e806bec0 con 0x7f79e8071e20 2023-11-28T20:33:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.319+0000 7f79ed97e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e8071e20 0x7f79e80b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.319+0000 7f79ed97e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e8071e20 0x7f79e80b49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:38376/0 (socket says 172.21.15.45:38376) 2023-11-28T20:33:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.319+0000 7f79ed97e640 1 -- 172.21.15.45:0/2423059441 learned_addr learned my addr 172.21.15.45:0/2423059441 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.320+0000 7f79ee980640 1 --2- 172.21.15.45:0/2423059441 >> v2:172.21.15.190:3300/0 conn(0x7f79e806b510 0x7f79e806b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.320+0000 7f79ed97e640 1 -- 172.21.15.45:0/2423059441 >> v2:172.21.15.190:3300/0 conn(0x7f79e806b510 msgr2=0x7f79e806b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.320+0000 7f79ed97e640 1 --2- 172.21.15.45:0/2423059441 >> v2:172.21.15.190:3300/0 conn(0x7f79e806b510 0x7f79e806b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.320+0000 7f79ed97e640 1 -- 172.21.15.45:0/2423059441 >> v1:172.21.15.190:6789/0 conn(0x7f79e80ae6e0 legacy=0x7f79e81618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.320+0000 7f79ed97e640 1 -- 172.21.15.45:0/2423059441 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e80b31b0 con 0x7f79e8071e20 2023-11-28T20:33:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.320+0000 7f79ee980640 1 --2- 172.21.15.45:0/2423059441 >> v2:172.21.15.190:3300/0 conn(0x7f79e806b510 0x7f79e806b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:33:49.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.321+0000 7f79ed97e640 1 --2- 172.21.15.45:0/2423059441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e8071e20 0x7f79e80b49c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009870 tx=0x7f79d402b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b9c0e38f663f782 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:49.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.322+0000 7f79ed17d640 1 -- 172.21.15.45:0/2423059441 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d40317d0 con 0x7f79e8071e20 2023-11-28T20:33:49.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.322+0000 7f79ed17d640 1 -- 172.21.15.45:0/2423059441 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d4031930 con 0x7f79e8071e20 2023-11-28T20:33:49.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.322+0000 7f79ed17d640 1 -- 172.21.15.45:0/2423059441 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4031cc0 con 0x7f79e8071e20 2023-11-28T20:33:49.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.323+0000 7f79ef181640 1 -- 172.21.15.45:0/2423059441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e8071e20 msgr2=0x7f79e80b49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.323+0000 7f79ef181640 1 --2- 172.21.15.45:0/2423059441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e8071e20 0x7f79e80b49c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009870 tx=0x7f79d402b870 comp rx=0 tx=0).stop 2023-11-28T20:33:49.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.323+0000 7f79ef181640 1 -- 172.21.15.45:0/2423059441 shutdown_connections 2023-11-28T20:33:49.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.323+0000 7f79ef181640 1 --2- 172.21.15.45:0/2423059441 >> v2:172.21.15.190:3300/0 conn(0x7f79e806b510 0x7f79e806b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.323+0000 7f79ef181640 1 --2- 172.21.15.45:0/2423059441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e8071e20 0x7f79e80b49c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.323+0000 7f79ef181640 1 -- 172.21.15.45:0/2423059441 >> 172.21.15.45:0/2423059441 conn(0x7f79e8071270 msgr2=0x7f79e8071680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:49.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.323+0000 7f79ef181640 1 -- 172.21.15.45:0/2423059441 shutdown_connections 2023-11-28T20:33:49.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.323+0000 7f79ef181640 1 -- 172.21.15.45:0/2423059441 wait complete. 2023-11-28T20:33:49.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.325+0000 7f79ef181640 1 Processor -- start 2023-11-28T20:33:49.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.325+0000 7f79ef181640 1 -- start start 2023-11-28T20:33:49.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ef181640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e806b510 0x7f79e814b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ef181640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e8071e20 0x7f79e8147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ef181640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e80ae6e0 0x7f79e81f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ef181640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81f42f0 con 0x7f79e806b510 2023-11-28T20:33:49.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ef181640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e814ae70 con 0x7f79e8071e20 2023-11-28T20:33:49.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ef181640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81f7410 con 0x7f79e80ae6e0 2023-11-28T20:33:49.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ee17f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e806b510 0x7f79e814b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ee17f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e806b510 0x7f79e814b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39782/0 (socket says 172.21.15.45:39782) 2023-11-28T20:33:49.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ee17f640 1 -- 172.21.15.45:0/2321193970 learned_addr learned my addr 172.21.15.45:0/2321193970 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:49.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ed97e640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e8071e20 0x7f79e8147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.326+0000 7f79ee980640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e80ae6e0 0x7f79e81f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79ee17f640 1 -- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e80ae6e0 msgr2=0x7f79e81f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79ee17f640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e80ae6e0 0x7f79e81f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79ee17f640 1 -- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e8071e20 msgr2=0x7f79e8147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79ee17f640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e8071e20 0x7f79e8147fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79ee17f640 1 -- 172.21.15.45:0/2321193970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d40094d0 con 0x7f79e806b510 2023-11-28T20:33:49.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79ee17f640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e806b510 0x7f79e814b300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f79dc00c7c0 tx=0x7f79dc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:49.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79e6ffd640 1 -- 172.21.15.45:0/2321193970 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc006d50 con 0x7f79e806b510 2023-11-28T20:33:49.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79e6ffd640 1 -- 172.21.15.45:0/2321193970 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc003970 con 0x7f79e806b510 2023-11-28T20:33:49.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e81f3d30 con 0x7f79e806b510 2023-11-28T20:33:49.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.327+0000 7f79e6ffd640 1 -- 172.21.15.45:0/2321193970 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc003d20 con 0x7f79e806b510 2023-11-28T20:33:49.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.328+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e814ab00 con 0x7f79e806b510 2023-11-28T20:33:49.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.329+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e814ac90 con 0x7f79e806b510 2023-11-28T20:33:49.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.330+0000 7f79e6ffd640 1 -- 172.21.15.45:0/2321193970 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f79dc00a850 con 0x7f79e806b510 2023-11-28T20:33:49.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.330+0000 7f79e6ffd640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f79cc06adb0 0x7f79cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.331+0000 7f79e6ffd640 1 -- 172.21.15.45:0/2321193970 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f79dc088e10 con 0x7f79e806b510 2023-11-28T20:33:49.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.331+0000 7f79ed97e640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f79cc06adb0 0x7f79cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.332+0000 7f79ed97e640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f79cc06adb0 0x7f79cc06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79d4036000 tx=0x7f79d403d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:49.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.336+0000 7f79e6ffd640 1 -- 172.21.15.45:0/2321193970 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79dc0542e0 con 0x7f79e806b510 2023-11-28T20:33:49.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.567+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79e814a960 con 0x7f79e806b510 2023-11-28T20:33:49.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.568+0000 7f79e6ffd640 1 -- 172.21.15.45:0/2321193970 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3700 (secure 0 0 0) 0x7f79e814a960 con 0x7f79e806b510 2023-11-28T20:33:49.570 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:33:49.570 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-11-28T20:33:49.571 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 3 2023-11-28T20:33:49.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.570+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f79cc06adb0 msgr2=0x7f79cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.570+0000 7f79ef181640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f79cc06adb0 0x7f79cc06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79d4036000 tx=0x7f79d403d040 comp rx=0 tx=0).stop 2023-11-28T20:33:49.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.570+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e806b510 msgr2=0x7f79e814b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.570+0000 7f79ef181640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e806b510 0x7f79e814b300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f79dc00c7c0 tx=0x7f79dc00cc80 comp rx=0 tx=0).stop 2023-11-28T20:33:49.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.571+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 shutdown_connections 2023-11-28T20:33:49.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.571+0000 7f79ef181640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f79cc06adb0 0x7f79cc06d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.571+0000 7f79ef181640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e80ae6e0 0x7f79e81f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.571+0000 7f79ef181640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e8071e20 0x7f79e8147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.571+0000 7f79ef181640 1 --2- 172.21.15.45:0/2321193970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e806b510 0x7f79e814b300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.571+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 >> 172.21.15.45:0/2321193970 conn(0x7f79e8071270 msgr2=0x7f79e806e580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:49.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.571+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 shutdown_connections 2023-11-28T20:33:49.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.571+0000 7f79ef181640 1 -- 172.21.15.45:0/2321193970 wait complete. 2023-11-28T20:33:49.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:33:49.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.816+0000 7f7e8af89640 1 Processor -- start 2023-11-28T20:33:49.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.816+0000 7f7e8af89640 1 -- start start 2023-11-28T20:33:49.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.819+0000 7f7e8af89640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e8415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.819+0000 7f7e8af89640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7e840b1340 0x7f7e8415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.819+0000 7f7e8af89640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8415bda0 con 0x7f7e8415b850 2023-11-28T20:33:49.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.819+0000 7f7e8af89640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8415ee40 con 0x7f7e840b1340 2023-11-28T20:33:49.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.819+0000 7f7e8af89640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8415c080 con 0x7f7e840b35b0 2023-11-28T20:33:49.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.819+0000 7f7e88d25640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e8415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.819+0000 7f7e88d25640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e8415ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:38382/0 (socket says 172.21.15.45:38382) 2023-11-28T20:33:49.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.819+0000 7f7e88d25640 1 -- 172.21.15.45:0/2726346695 learned_addr learned my addr 172.21.15.45:0/2726346695 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:49.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.820+0000 7f7e89526640 1 --2- 172.21.15.45:0/2726346695 >> v2:172.21.15.190:3300/0 conn(0x7f7e840b1340 0x7f7e8415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.820+0000 7f7e88d25640 1 -- 172.21.15.45:0/2726346695 >> v2:172.21.15.190:3300/0 conn(0x7f7e840b1340 msgr2=0x7f7e8415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.820+0000 7f7e88d25640 1 --2- 172.21.15.45:0/2726346695 >> v2:172.21.15.190:3300/0 conn(0x7f7e840b1340 0x7f7e8415f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.820+0000 7f7e88d25640 1 -- 172.21.15.45:0/2726346695 >> v1:172.21.15.190:6789/0 conn(0x7f7e8415b850 legacy=0x7f7e840b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.820+0000 7f7e88d25640 1 -- 172.21.15.45:0/2726346695 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8415c360 con 0x7f7e840b35b0 2023-11-28T20:33:49.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.822+0000 7f7e88d25640 1 --2- 172.21.15.45:0/2726346695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e8415ca10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009870 tx=0x7f7e7802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=438c9f4d7c2c00ca server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:49.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.823+0000 7f7e837fe640 1 -- 172.21.15.45:0/2726346695 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78004230 con 0x7f7e840b35b0 2023-11-28T20:33:49.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.823+0000 7f7e837fe640 1 -- 172.21.15.45:0/2726346695 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7802e050 con 0x7f7e840b35b0 2023-11-28T20:33:49.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.823+0000 7f7e837fe640 1 -- 172.21.15.45:0/2726346695 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78003610 con 0x7f7e840b35b0 2023-11-28T20:33:49.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.823+0000 7f7e8af89640 1 -- 172.21.15.45:0/2726346695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 msgr2=0x7f7e8415ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.823+0000 7f7e8af89640 1 --2- 172.21.15.45:0/2726346695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e8415ca10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009870 tx=0x7f7e7802b7e0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.824+0000 7f7e8af89640 1 -- 172.21.15.45:0/2726346695 shutdown_connections 2023-11-28T20:33:49.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.824+0000 7f7e8af89640 1 --2- 172.21.15.45:0/2726346695 >> v2:172.21.15.190:3300/0 conn(0x7f7e840b1340 0x7f7e8415f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.824+0000 7f7e8af89640 1 --2- 172.21.15.45:0/2726346695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e8415ca10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.824+0000 7f7e8af89640 1 -- 172.21.15.45:0/2726346695 >> 172.21.15.45:0/2726346695 conn(0x7f7e8406a500 msgr2=0x7f7e8406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:49.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.824+0000 7f7e8af89640 1 -- 172.21.15.45:0/2726346695 shutdown_connections 2023-11-28T20:33:49.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.824+0000 7f7e8af89640 1 -- 172.21.15.45:0/2726346695 wait complete. 2023-11-28T20:33:49.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.826+0000 7f7e8af89640 1 Processor -- start 2023-11-28T20:33:49.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.826+0000 7f7e8af89640 1 -- start start 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e8af89640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e840b1340 0x7f7e84136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e8af89640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e841f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e8af89640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e8415b850 0x7f7e840cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e8af89640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e841f7660 con 0x7f7e840b1340 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e8af89640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e841f3530 con 0x7f7e8415b850 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e8af89640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e841f3810 con 0x7f7e840b35b0 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e88d25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e840b1340 0x7f7e84136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e88d25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e840b1340 0x7f7e84136340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39784/0 (socket says 172.21.15.45:39784) 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e88d25640 1 -- 172.21.15.45:0/4147734418 learned_addr learned my addr 172.21.15.45:0/4147734418 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.828+0000 7f7e83fff640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e841f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.829+0000 7f7e89526640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e8415b850 0x7f7e840cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.829+0000 7f7e88d25640 1 -- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 msgr2=0x7f7e841f7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.829+0000 7f7e88d25640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e841f7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.829+0000 7f7e88d25640 1 -- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e8415b850 msgr2=0x7f7e840cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.829+0000 7f7e88d25640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e8415b850 0x7f7e840cb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.829+0000 7f7e88d25640 1 -- 172.21.15.45:0/4147734418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e780094d0 con 0x7f7e840b1340 2023-11-28T20:33:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.829+0000 7f7e88d25640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e840b1340 0x7f7e84136340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e78000c00 tx=0x7f7e780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.829+0000 7f7e817fa640 1 -- 172.21.15.45:0/4147734418 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78003e50 con 0x7f7e840b1340 2023-11-28T20:33:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.829+0000 7f7e817fa640 1 -- 172.21.15.45:0/4147734418 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e78030510 con 0x7f7e840b1340 2023-11-28T20:33:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.830+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e841f3af0 con 0x7f7e840b1340 2023-11-28T20:33:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.830+0000 7f7e817fa640 1 -- 172.21.15.45:0/4147734418 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e780308a0 con 0x7f7e840b1340 2023-11-28T20:33:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.830+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e840cbd30 con 0x7f7e840b1340 2023-11-28T20:33:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.831+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e841f3950 con 0x7f7e840b1340 2023-11-28T20:33:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.832+0000 7f7e817fa640 1 -- 172.21.15.45:0/4147734418 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f7e78039040 con 0x7f7e840b1340 2023-11-28T20:33:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.833+0000 7f7e817fa640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7e6806adb0 0x7f7e6806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.833+0000 7f7e817fa640 1 -- 172.21.15.45:0/4147734418 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e780aa940 con 0x7f7e840b1340 2023-11-28T20:33:49.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.833+0000 7f7e83fff640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7e6806adb0 0x7f7e6806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.835+0000 7f7e83fff640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7e6806adb0 0x7f7e6806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e840ab900 tx=0x7f7e70009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:49.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:49.838+0000 7f7e817fa640 1 -- 172.21.15.45:0/4147734418 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e841f3950 con 0x7f7e840b1340 2023-11-28T20:33:50.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.050+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e841f3950 con 0x7f7e840b1340 2023-11-28T20:33:50.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.051+0000 7f7e817fa640 1 -- 172.21.15.45:0/4147734418 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12838 (secure 0 0 0) 0x7f7e841f3950 con 0x7f7e840b1340 2023-11-28T20:33:50.053 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:33:50.054 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":17,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:49.985718+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:33:50.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7e6806adb0 msgr2=0x7f7e6806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7e6806adb0 0x7f7e6806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e840ab900 tx=0x7f7e70009040 comp rx=0 tx=0).stop 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e840b1340 msgr2=0x7f7e84136340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e840b1340 0x7f7e84136340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e78000c00 tx=0x7f7e780046c0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 shutdown_connections 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7e6806adb0 0x7f7e6806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e8415b850 0x7f7e840cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e840b35b0 0x7f7e841f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 --2- 172.21.15.45:0/4147734418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e840b1340 0x7f7e84136340 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 >> 172.21.15.45:0/4147734418 conn(0x7f7e8406a500 msgr2=0x7f7e840cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 shutdown_connections 2023-11-28T20:33:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.054+0000 7f7e8af89640 1 -- 172.21.15.45:0/4147734418 wait complete. 2023-11-28T20:33:50.076 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:33:50.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:33:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.271+0000 7fd1a5c72640 1 Processor -- start 2023-11-28T20:33:50.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.271+0000 7fd1a5c72640 1 -- start start 2023-11-28T20:33:50.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.273+0000 7fd1a5c72640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1a0157c10 0x7fd1a006da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.273+0000 7fd1a5c72640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a006ca60 con 0x7fd1a015e4e0 2023-11-28T20:33:50.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.273+0000 7fd1a5c72640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a015eb40 con 0x7fd1a006a500 2023-11-28T20:33:50.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.273+0000 7fd1a5c72640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a0157ff0 con 0x7fd1a0157c10 2023-11-28T20:33:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.274+0000 7fd19ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd1a015e4e0 0x7fd1a00c9e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50532/0 (socket says 172.21.15.45:50532) 2023-11-28T20:33:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.274+0000 7fd19ffff640 1 -- 172.21.15.45:0/529527618 learned_addr learned my addr 172.21.15.45:0/529527618 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:33:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.274+0000 7fd19f7fe640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1a0157c10 0x7fd1a006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.274+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092645281 0 0) 0x7fd1a006ca60 con 0x7fd1a015e4e0 2023-11-28T20:33:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.274+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd18c003520 con 0x7fd1a015e4e0 2023-11-28T20:33:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533246372 0 0) 0x7fd1a015eb40 con 0x7fd1a006a500 2023-11-28T20:33:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a006ca60 con 0x7fd1a006a500 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98874352 0 0) 0x7fd18c003520 con 0x7fd1a015e4e0 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1a0157c10 msgr2=0x7fd1a006da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1a0157c10 0x7fd1a006da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 >> v1:172.21.15.190:6789/0 conn(0x7fd1a006a500 legacy=0x7fd1a015ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a006cc90 con 0x7fd1a015e4e0 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd1980019f0 con 0x7fd1a015e4e0 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 >> v1:172.21.15.45:6789/0 conn(0x7fd1a015e4e0 legacy=0x7fd1a00c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd18c003b30 0x7fd18c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1a0157c10 0x7fd18c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c004d00 0x7fd18c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a0157ff0 con 0x7fd18c004d00 2023-11-28T20:33:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c00dc40 con 0x7fd18c003b30 2023-11-28T20:33:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c00de00 con 0x7fd1a0157c10 2023-11-28T20:33:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.275+0000 7fd19ffff640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c004d00 0x7fd18c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.276+0000 7fd197fff640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1a0157c10 0x7fd18c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.276+0000 7fd19f7fe640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd18c003b30 0x7fd18c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.276+0000 7fd19ffff640 1 -- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1a0157c10 msgr2=0x7fd18c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.276+0000 7fd19ffff640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1a0157c10 0x7fd18c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.276+0000 7fd19ffff640 1 -- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd18c003b30 msgr2=0x7fd18c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.276+0000 7fd19ffff640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd18c003b30 0x7fd18c003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.276+0000 7fd19ffff640 1 -- 172.21.15.45:0/529527618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c003520 con 0x7fd18c004d00 2023-11-28T20:33:50.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.277+0000 7fd19ffff640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c004d00 0x7fd18c005180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd19800a700 tx=0x7fd18c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66faff493531d7fa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:50.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.277+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198034630 con 0x7fd18c004d00 2023-11-28T20:33:50.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.277+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19802f070 con 0x7fd18c004d00 2023-11-28T20:33:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.277+0000 7fd19effd640 1 -- 172.21.15.45:0/529527618 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1980349f0 con 0x7fd18c004d00 2023-11-28T20:33:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.278+0000 7fd1a5c72640 1 -- 172.21.15.45:0/529527618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c004d00 msgr2=0x7fd18c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.278+0000 7fd1a5c72640 1 --2- 172.21.15.45:0/529527618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd18c004d00 0x7fd18c005180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd19800a700 tx=0x7fd18c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.278+0000 7fd19ffff640 1 -- 172.21.15.45:0/529527618 reap_dead start 2023-11-28T20:33:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.278+0000 7fd1a5c72640 1 -- 172.21.15.45:0/529527618 shutdown_connections 2023-11-28T20:33:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.278+0000 7fd1a5c72640 1 -- 172.21.15.45:0/529527618 >> 172.21.15.45:0/529527618 conn(0x7fd1a00ced60 msgr2=0x7fd1a00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.278+0000 7fd1a5c72640 1 -- 172.21.15.45:0/529527618 shutdown_connections 2023-11-28T20:33:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.278+0000 7fd1a5c72640 1 -- 172.21.15.45:0/529527618 wait complete. 2023-11-28T20:33:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.281+0000 7fd1a5c72640 1 Processor -- start 2023-11-28T20:33:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.281+0000 7fd1a5c72640 1 -- start start 2023-11-28T20:33:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd1a5c72640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd18c004d00 0x7fd1a0152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd1a5c72640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a015b960 0x7fd1a015bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd1a5c72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1a015c370 0x7fd1a015c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd1a5c72640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a00ce140 con 0x7fd1a015c370 2023-11-28T20:33:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd1a5c72640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a0153830 con 0x7fd1a015b960 2023-11-28T20:33:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd1a5c72640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a015cd30 con 0x7fd18c004d00 2023-11-28T20:33:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1a015c370 0x7fd1a015c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1a015c370 0x7fd1a015c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39796/0 (socket says 172.21.15.45:39796) 2023-11-28T20:33:50.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd19ffff640 1 -- 172.21.15.45:0/4029595330 learned_addr learned my addr 172.21.15.45:0/4029595330 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:50.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd197fff640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a015b960 0x7fd1a015bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd19f7fe640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd18c004d00 0x7fd1a0152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd19ffff640 1 -- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd18c004d00 msgr2=0x7fd1a0152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd19ffff640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd18c004d00 0x7fd1a0152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd19ffff640 1 -- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a015b960 msgr2=0x7fd1a015bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.282+0000 7fd19ffff640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a015b960 0x7fd1a015bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.283+0000 7fd19ffff640 1 -- 172.21.15.45:0/4029595330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a01f26c0 con 0x7fd1a015c370 2023-11-28T20:33:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.283+0000 7fd19ffff640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1a015c370 0x7fd1a015c7f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd19802caf0 tx=0x7fd198001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.283+0000 7fd19cff9640 1 -- 172.21.15.45:0/4029595330 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198034840 con 0x7fd1a015c370 2023-11-28T20:33:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.283+0000 7fd19cff9640 1 -- 172.21.15.45:0/4029595330 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1980349a0 con 0x7fd1a015c370 2023-11-28T20:33:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.283+0000 7fd19cff9640 1 -- 172.21.15.45:0/4029595330 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198034d30 con 0x7fd1a015c370 2023-11-28T20:33:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.283+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a00cd270 con 0x7fd1a015c370 2023-11-28T20:33:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.283+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a00cd7a0 con 0x7fd1a015c370 2023-11-28T20:33:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.285+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a00733d0 con 0x7fd1a015c370 2023-11-28T20:33:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.285+0000 7fd19cff9640 1 -- 172.21.15.45:0/4029595330 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fd198031030 con 0x7fd1a015c370 2023-11-28T20:33:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.286+0000 7fd19cff9640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd18006ade0 0x7fd18006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.286+0000 7fd19cff9640 1 -- 172.21.15.45:0/4029595330 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd1980aaeb0 con 0x7fd1a015c370 2023-11-28T20:33:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.287+0000 7fd19f7fe640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd18006ade0 0x7fd18006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.288+0000 7fd19f7fe640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd18006ade0 0x7fd18006d2a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd190009a10 tx=0x7fd190008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.293+0000 7fd19cff9640 1 -- 172.21.15.45:0/4029595330 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1980763a0 con 0x7fd1a015c370 2023-11-28T20:33:50.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.493+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1a01f26c0 con 0x7fd1a015c370 2023-11-28T20:33:50.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.494+0000 7fd19cff9640 1 -- 172.21.15.45:0/4029595330 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd1a01f26c0 con 0x7fd1a015c370 2023-11-28T20:33:50.496 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd18006ade0 msgr2=0x7fd18006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd18006ade0 0x7fd18006d2a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd190009a10 tx=0x7fd190008040 comp rx=0 tx=0).stop 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1a015c370 msgr2=0x7fd1a015c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1a015c370 0x7fd1a015c7f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd19802caf0 tx=0x7fd198001cb0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 shutdown_connections 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd18006ade0 0x7fd18006d2a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1a015c370 0x7fd1a015c7f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a015b960 0x7fd1a015bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 --2- 172.21.15.45:0/4029595330 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd18c004d00 0x7fd1a0152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.496+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 >> 172.21.15.45:0/4029595330 conn(0x7fd1a00ced60 msgr2=0x7fd1a0155a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.497+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 shutdown_connections 2023-11-28T20:33:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.497+0000 7fd1a5c72640 1 -- 172.21.15.45:0/4029595330 wait complete. 2023-11-28T20:33:50.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T20:33:50.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.718+0000 7f1b54590640 1 Processor -- start 2023-11-28T20:33:50.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.718+0000 7f1b54590640 1 -- start start 2023-11-28T20:33:50.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.720+0000 7f1b54590640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c160e70 0x7f1b4c0b6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.720+0000 7f1b54590640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4c06aa70 con 0x7f1b4c06a500 2023-11-28T20:33:50.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.720+0000 7f1b54590640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4c06ad50 con 0x7f1b4c071230 2023-11-28T20:33:50.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.720+0000 7f1b54590640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c06b030 con 0x7f1b4c160e70 2023-11-28T20:33:50.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.720+0000 7f1b51b2b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1b4c06a500 0x7f1b4c070b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50538/0 (socket says 172.21.15.45:50538) 2023-11-28T20:33:50.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.720+0000 7f1b51b2b640 1 -- 172.21.15.45:0/3716957397 learned_addr learned my addr 172.21.15.45:0/3716957397 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:33:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5232c640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c160e70 0x7f1b4c0b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732795071 0 0) 0x7f1b4c06aa70 con 0x7f1b4c06a500 2023-11-28T20:33:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b38003520 con 0x7f1b4c06a500 2023-11-28T20:33:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243035889 0 0) 0x7f1b38003520 con 0x7f1b4c06a500 2023-11-28T20:33:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c160e70 msgr2=0x7f1b4c0b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c160e70 0x7f1b4c0b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 >> v1:172.21.15.190:6789/0 conn(0x7f1b4c071230 legacy=0x7f1b4c0d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c06b310 con 0x7f1b4c06a500 2023-11-28T20:33:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f1b3c001670 con 0x7f1b4c06a500 2023-11-28T20:33:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 >> v1:172.21.15.45:6789/0 conn(0x7f1b4c06a500 legacy=0x7f1b4c070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c160e70 0x7f1b380039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38003f70 0x7f1b380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b38004d00 0x7f1b38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c06b030 con 0x7f1b38004d00 2023-11-28T20:33:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3800bac0 con 0x7f1b38003f70 2023-11-28T20:33:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.721+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3800bc80 con 0x7f1b4c160e70 2023-11-28T20:33:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.722+0000 7f1b52b2d640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b38004d00 0x7f1b38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.722+0000 7f1b51b2b640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38003f70 0x7f1b380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.722+0000 7f1b5232c640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c160e70 0x7f1b380039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.722+0000 7f1b52b2d640 1 -- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c160e70 msgr2=0x7f1b380039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.722+0000 7f1b52b2d640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c160e70 0x7f1b380039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.722+0000 7f1b52b2d640 1 -- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38003f70 msgr2=0x7f1b380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.722+0000 7f1b52b2d640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38003f70 0x7f1b380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.722+0000 7f1b52b2d640 1 -- 172.21.15.45:0/3716957397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b38003520 con 0x7f1b38004d00 2023-11-28T20:33:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.723+0000 7f1b52b2d640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b38004d00 0x7f1b38005180 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0d2e80 tx=0x7f1b380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738e7ae9d017046b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44003b70 con 0x7f1b38004d00 2023-11-28T20:33:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b44003cd0 con 0x7f1b38004d00 2023-11-28T20:33:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b5132a640 1 -- 172.21.15.45:0/3716957397 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44005310 con 0x7f1b38004d00 2023-11-28T20:33:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b54590640 1 -- 172.21.15.45:0/3716957397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b38004d00 msgr2=0x7f1b38005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b54590640 1 --2- 172.21.15.45:0/3716957397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b38004d00 0x7f1b38005180 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0d2e80 tx=0x7f1b380057f0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b52b2d640 1 -- 172.21.15.45:0/3716957397 reap_dead start 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b54590640 1 -- 172.21.15.45:0/3716957397 shutdown_connections 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b54590640 1 -- 172.21.15.45:0/3716957397 >> 172.21.15.45:0/3716957397 conn(0x7f1b4c0cfc60 msgr2=0x7f1b4c0d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b54590640 1 -- 172.21.15.45:0/3716957397 shutdown_connections 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.724+0000 7f1b54590640 1 -- 172.21.15.45:0/3716957397 wait complete. 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.726+0000 7f1b54590640 1 Processor -- start 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.726+0000 7f1b54590640 1 -- start start 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.727+0000 7f1b54590640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38004d00 0x7f1b4c0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.727+0000 7f1b54590640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c15e140 0x7f1b4c1f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.727+0000 7f1b54590640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4c15f740 0x7f1b4c15fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.727+0000 7f1b54590640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c0b0a50 con 0x7f1b4c15f740 2023-11-28T20:33:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.727+0000 7f1b54590640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c160100 con 0x7f1b38004d00 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.727+0000 7f1b54590640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c1f7970 con 0x7f1b4c15e140 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b52b2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4c15f740 0x7f1b4c15fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b52b2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4c15f740 0x7f1b4c15fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39818/0 (socket says 172.21.15.45:39818) 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b52b2d640 1 -- 172.21.15.45:0/2745731934 learned_addr learned my addr 172.21.15.45:0/2745731934 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b5232c640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38004d00 0x7f1b4c0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b51b2b640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c15e140 0x7f1b4c1f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b52b2d640 1 -- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c15e140 msgr2=0x7f1b4c1f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b52b2d640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c15e140 0x7f1b4c1f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b52b2d640 1 -- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38004d00 msgr2=0x7f1b4c0adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b52b2d640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38004d00 0x7f1b4c0adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b52b2d640 1 -- 172.21.15.45:0/2745731934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c1f7c50 con 0x7f1b4c15f740 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b5232c640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38004d00 0x7f1b4c0adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:33:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.728+0000 7f1b52b2d640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4c15f740 0x7f1b4c15fbc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b4402c020 tx=0x7f1b440014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:50.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.729+0000 7f1b42ffd640 1 -- 172.21.15.45:0/2745731934 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4403e070 con 0x7f1b4c15f740 2023-11-28T20:33:50.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.729+0000 7f1b42ffd640 1 -- 172.21.15.45:0/2745731934 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b44003e40 con 0x7f1b4c15f740 2023-11-28T20:33:50.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.729+0000 7f1b42ffd640 1 -- 172.21.15.45:0/2745731934 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44006d50 con 0x7f1b4c15f740 2023-11-28T20:33:50.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.729+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4c0b0670 con 0x7f1b4c15f740 2023-11-28T20:33:50.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.729+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4c1f75b0 con 0x7f1b4c15f740 2023-11-28T20:33:50.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.730+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4c0b04d0 con 0x7f1b4c15f740 2023-11-28T20:33:50.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.731+0000 7f1b42ffd640 1 -- 172.21.15.45:0/2745731934 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f1b44007030 con 0x7f1b4c15f740 2023-11-28T20:33:50.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.732+0000 7f1b42ffd640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1b3006ade0 0x7f1b3006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:50.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.732+0000 7f1b42ffd640 1 -- 172.21.15.45:0/2745731934 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1b440aa330 con 0x7f1b4c15f740 2023-11-28T20:33:50.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.733+0000 7f1b5232c640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1b3006ade0 0x7f1b3006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:50.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.734+0000 7f1b5232c640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1b3006ade0 0x7f1b3006d2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009ad0 tx=0x7f1b48008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:50.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.737+0000 7f1b42ffd640 1 -- 172.21.15.45:0/2745731934 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b4c0b04d0 con 0x7f1b4c15f740 2023-11-28T20:33:50.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.935+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1b4c0b04d0 con 0x7f1b4c15f740 2023-11-28T20:33:50.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.936+0000 7f1b42ffd640 1 -- 172.21.15.45:0/2745731934 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1b4c0b04d0 con 0x7f1b4c15f740 2023-11-28T20:33:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:33:50.938 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:33:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:33:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T20:33:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T20:33:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:33:50.939 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:33:50.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1b3006ade0 msgr2=0x7f1b3006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1b3006ade0 0x7f1b3006d2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009ad0 tx=0x7f1b48008040 comp rx=0 tx=0).stop 2023-11-28T20:33:50.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4c15f740 msgr2=0x7f1b4c15fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:50.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4c15f740 0x7f1b4c15fbc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b4402c020 tx=0x7f1b440014a0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 shutdown_connections 2023-11-28T20:33:50.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1b3006ade0 0x7f1b3006d2a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b4c15f740 0x7f1b4c15fbc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b4c15e140 0x7f1b4c1f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 --2- 172.21.15.45:0/2745731934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b38004d00 0x7f1b4c0adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:50.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 >> 172.21.15.45:0/2745731934 conn(0x7f1b4c0cfc60 msgr2=0x7f1b4c0ce240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:50.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 shutdown_connections 2023-11-28T20:33:50.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:50.938+0000 7f1b54590640 1 -- 172.21.15.45:0/2745731934 wait complete. 2023-11-28T20:33:50.958 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T20:33:50.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T20:33:51.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.150+0000 7fe3f3948640 1 Processor -- start 2023-11-28T20:33:51.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.150+0000 7fe3f3948640 1 -- start start 2023-11-28T20:33:51.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.151+0000 7fe3f3948640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe3ec066900 0x7fe3ec0cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:51.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.151+0000 7fe3f3948640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec06d620 0x7fe3ec06abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:51.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.152+0000 7fe3f3948640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ec0cf500 con 0x7fe3ec15d7a0 2023-11-28T20:33:51.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.152+0000 7fe3f3948640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec0cfda0 con 0x7fe3ec066900 2023-11-28T20:33:51.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.152+0000 7fe3f3948640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec0cf7e0 con 0x7fe3ec06d620 2023-11-28T20:33:51.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.152+0000 7fe3f0ee3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe3ec066900 0x7fe3ec0cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:51.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.152+0000 7fe3f0ee3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe3ec066900 0x7fe3ec0cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:34916/0 (socket says 172.21.15.45:34916) 2023-11-28T20:33:51.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.152+0000 7fe3f0ee3640 1 -- 172.21.15.45:0/794076050 learned_addr learned my addr 172.21.15.45:0/794076050 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:51.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.152+0000 7fe3f1ee5640 1 --2- 172.21.15.45:0/794076050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec06d620 0x7fe3ec06abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:51.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.153+0000 7fe3f0ee3640 1 -- 172.21.15.45:0/794076050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec06d620 msgr2=0x7fe3ec06abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:51.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.153+0000 7fe3f0ee3640 1 --2- 172.21.15.45:0/794076050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec06d620 0x7fe3ec06abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:51.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.153+0000 7fe3f0ee3640 1 -- 172.21.15.45:0/794076050 >> v1:172.21.15.190:6789/0 conn(0x7fe3ec15d7a0 legacy=0x7fe3ec0661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:51.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.153+0000 7fe3f0ee3640 1 -- 172.21.15.45:0/794076050 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec0cfac0 con 0x7fe3ec066900 2023-11-28T20:33:51.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.154+0000 7fe3f0ee3640 1 --2- 172.21.15.45:0/794076050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec066900 0x7fe3ec0cd4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009870 tx=0x7fe3dc02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7b8d9f412fa9c5b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:51.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.155+0000 7fe3e3fff640 1 -- 172.21.15.45:0/794076050 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc02f840 con 0x7fe3ec066900 2023-11-28T20:33:51.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.155+0000 7fe3e3fff640 1 -- 172.21.15.45:0/794076050 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc02f9a0 con 0x7fe3ec066900 2023-11-28T20:33:51.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.155+0000 7fe3e3fff640 1 -- 172.21.15.45:0/794076050 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc02fd30 con 0x7fe3ec066900 2023-11-28T20:33:51.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.156+0000 7fe3f3948640 1 -- 172.21.15.45:0/794076050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec066900 msgr2=0x7fe3ec0cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:51.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.156+0000 7fe3f3948640 1 --2- 172.21.15.45:0/794076050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec066900 0x7fe3ec0cd4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009870 tx=0x7fe3dc02ba20 comp rx=0 tx=0).stop 2023-11-28T20:33:51.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.156+0000 7fe3f3948640 1 -- 172.21.15.45:0/794076050 shutdown_connections 2023-11-28T20:33:51.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.156+0000 7fe3f3948640 1 --2- 172.21.15.45:0/794076050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec06d620 0x7fe3ec06abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:51.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.156+0000 7fe3f3948640 1 --2- 172.21.15.45:0/794076050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec066900 0x7fe3ec0cd4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.156+0000 7fe3f3948640 1 -- 172.21.15.45:0/794076050 >> 172.21.15.45:0/794076050 conn(0x7fe3ec0cc9c0 msgr2=0x7fe3ec0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.156+0000 7fe3f3948640 1 -- 172.21.15.45:0/794076050 shutdown_connections 2023-11-28T20:33:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.156+0000 7fe3f3948640 1 -- 172.21.15.45:0/794076050 wait complete. 2023-11-28T20:33:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.158+0000 7fe3f3948640 1 Processor -- start 2023-11-28T20:33:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.158+0000 7fe3f3948640 1 -- start start 2023-11-28T20:33:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f3948640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec066900 0x7fe3ec0c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f3948640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3ec06d620 0x7fe3ec1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f3948640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec15d7a0 0x7fe3ec15a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f3948640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec0c3270 con 0x7fe3ec06d620 2023-11-28T20:33:51.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f3948640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1595d0 con 0x7fe3ec066900 2023-11-28T20:33:51.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f3948640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1f4380 con 0x7fe3ec15d7a0 2023-11-28T20:33:51.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f0ee3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3ec06d620 0x7fe3ec1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:51.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f0ee3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3ec06d620 0x7fe3ec1f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39834/0 (socket says 172.21.15.45:39834) 2023-11-28T20:33:51.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f0ee3640 1 -- 172.21.15.45:0/2210050172 learned_addr learned my addr 172.21.15.45:0/2210050172 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:51.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f16e4640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec066900 0x7fe3ec0c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:51.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.160+0000 7fe3f1ee5640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec15d7a0 0x7fe3ec15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:51.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3f0ee3640 1 -- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec15d7a0 msgr2=0x7fe3ec15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:51.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3f0ee3640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec15d7a0 0x7fe3ec15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:51.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3f0ee3640 1 -- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec066900 msgr2=0x7fe3ec0c2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:51.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3f0ee3640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec066900 0x7fe3ec0c2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:51.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3f0ee3640 1 -- 172.21.15.45:0/2210050172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0094d0 con 0x7fe3ec06d620 2023-11-28T20:33:51.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3f0ee3640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3ec06d620 0x7fe3ec1f3e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc000c00 tx=0x7fe3dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:51.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3e1ffb640 1 -- 172.21.15.45:0/2210050172 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc03b040 con 0x7fe3ec06d620 2023-11-28T20:33:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3e1ffb640 1 -- 172.21.15.45:0/2210050172 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc037c60 con 0x7fe3ec06d620 2023-11-28T20:33:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3e1ffb640 1 -- 172.21.15.45:0/2210050172 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc042480 con 0x7fe3ec06d620 2023-11-28T20:33:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.161+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec1589c0 con 0x7fe3ec06d620 2023-11-28T20:33:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.162+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec158f80 con 0x7fe3ec06d620 2023-11-28T20:33:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.163+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ec0af870 con 0x7fe3ec06d620 2023-11-28T20:33:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.164+0000 7fe3e1ffb640 1 -- 172.21.15.45:0/2210050172 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fe3dc0425e0 con 0x7fe3ec06d620 2023-11-28T20:33:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.164+0000 7fe3e1ffb640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe3d006adb0 0x7fe3d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.165+0000 7fe3e1ffb640 1 -- 172.21.15.45:0/2210050172 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe3dc0ac230 con 0x7fe3ec06d620 2023-11-28T20:33:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.165+0000 7fe3f16e4640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe3d006adb0 0x7fe3d006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:51.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.166+0000 7fe3f16e4640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe3d006adb0 0x7fe3d006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0b0e80 tx=0x7fe3e8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:51.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.169+0000 7fe3e1ffb640 1 -- 172.21.15.45:0/2210050172 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3dc077670 con 0x7fe3ec06d620 2023-11-28T20:33:51.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:51.361+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe3ec0733d0 con 0x7fe3ec06d620 2023-11-28T20:33:52.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.008+0000 7fe3e1ffb640 1 -- 172.21.15.45:0/2210050172 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe3dc07ac90 con 0x7fe3ec06d620 2023-11-28T20:33:52.011 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T20:33:52.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe3d006adb0 msgr2=0x7fe3d006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:52.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe3d006adb0 0x7fe3d006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0b0e80 tx=0x7fe3e8009040 comp rx=0 tx=0).stop 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3ec06d620 msgr2=0x7fe3ec1f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3ec06d620 0x7fe3ec1f3e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc000c00 tx=0x7fe3dc0046c0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 shutdown_connections 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe3d006adb0 0x7fe3d006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec15d7a0 0x7fe3ec15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3ec06d620 0x7fe3ec1f3e40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 --2- 172.21.15.45:0/2210050172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec066900 0x7fe3ec0c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.012+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 >> 172.21.15.45:0/2210050172 conn(0x7fe3ec0cc9c0 msgr2=0x7fe3ec06aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.013+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 shutdown_connections 2023-11-28T20:33:52.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.013+0000 7fe3f3948640 1 -- 172.21.15.45:0/2210050172 wait complete. 2023-11-28T20:33:52.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-11-28T20:33:52.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.257+0000 7f284ec7c640 1 Processor -- start 2023-11-28T20:33:52.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.258+0000 7f284ec7c640 1 -- start start 2023-11-28T20:33:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.259+0000 7f284ec7c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2848157c10 0x7f284806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.259+0000 7f284ec7c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f284806a500 0x7f284806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.259+0000 7f284ec7c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f284815ddd0 0x7f284815e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.259+0000 7f284ec7c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f284806ca60 con 0x7f284815ddd0 2023-11-28T20:33:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.259+0000 7f284ec7c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f284806aea0 con 0x7f284806a500 2023-11-28T20:33:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.259+0000 7f284ec7c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284815eb40 con 0x7f2848157c10 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284d219640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f284815ddd0 0x7f284815e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284d219640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f284815ddd0 0x7f284815e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39840/0 (socket says 172.21.15.45:39840) 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284d219640 1 -- 172.21.15.45:0/83034159 learned_addr learned my addr 172.21.15.45:0/83034159 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284ca18640 1 --2- 172.21.15.45:0/83034159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2848157c10 0x7f284806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f2847fff640 1 --2- 172.21.15.45:0/83034159 >> v2:172.21.15.190:3300/0 conn(0x7f284806a500 0x7f284806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284d219640 1 -- 172.21.15.45:0/83034159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2848157c10 msgr2=0x7f284806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284d219640 1 --2- 172.21.15.45:0/83034159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2848157c10 0x7f284806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284d219640 1 -- 172.21.15.45:0/83034159 >> v2:172.21.15.190:3300/0 conn(0x7f284806a500 msgr2=0x7f284806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284d219640 1 --2- 172.21.15.45:0/83034159 >> v2:172.21.15.190:3300/0 conn(0x7f284806a500 0x7f284806a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284d219640 1 -- 172.21.15.45:0/83034159 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2848157ff0 con 0x7f284815ddd0 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f2847fff640 1 --2- 172.21.15.45:0/83034159 >> v2:172.21.15.190:3300/0 conn(0x7f284806a500 0x7f284806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:33:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.260+0000 7f284ca18640 1 --2- 172.21.15.45:0/83034159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2848157c10 0x7f284806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:33:52.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.261+0000 7f284d219640 1 --2- 172.21.15.45:0/83034159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 0x7f284815e260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28480cfd50 tx=0x7f284002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb3531f6561e2eb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:52.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f28477fe640 1 -- 172.21.15.45:0/83034159 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840031870 con 0x7f284815ddd0 2023-11-28T20:33:52.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f28477fe640 1 -- 172.21.15.45:0/83034159 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28400319d0 con 0x7f284815ddd0 2023-11-28T20:33:52.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f28477fe640 1 -- 172.21.15.45:0/83034159 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840031d60 con 0x7f284815ddd0 2023-11-28T20:33:52.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f284ec7c640 1 -- 172.21.15.45:0/83034159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 msgr2=0x7f284815e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:52.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f284ec7c640 1 --2- 172.21.15.45:0/83034159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 0x7f284815e260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28480cfd50 tx=0x7f284002b910 comp rx=0 tx=0).stop 2023-11-28T20:33:52.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f284ec7c640 1 -- 172.21.15.45:0/83034159 shutdown_connections 2023-11-28T20:33:52.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f284ec7c640 1 --2- 172.21.15.45:0/83034159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 0x7f284815e260 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f284ec7c640 1 --2- 172.21.15.45:0/83034159 >> v2:172.21.15.190:3300/0 conn(0x7f284806a500 0x7f284806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f284ec7c640 1 --2- 172.21.15.45:0/83034159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2848157c10 0x7f284806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f284ec7c640 1 -- 172.21.15.45:0/83034159 >> 172.21.15.45:0/83034159 conn(0x7f28480ced60 msgr2=0x7f28480cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:52.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f284ec7c640 1 -- 172.21.15.45:0/83034159 shutdown_connections 2023-11-28T20:33:52.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.262+0000 7f284ec7c640 1 -- 172.21.15.45:0/83034159 wait complete. 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.264+0000 7f284ec7c640 1 Processor -- start 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.264+0000 7f284ec7c640 1 -- start start 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.265+0000 7f284ec7c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f284806a500 0x7f284809a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.265+0000 7f284ec7c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2848157c10 0x7f2848155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.265+0000 7f284ec7c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 0x7f284815c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.265+0000 7f284ec7c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2848156080 con 0x7f284815ddd0 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.265+0000 7f284ec7c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284809b270 con 0x7f2848157c10 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.265+0000 7f284ec7c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284809acb0 con 0x7f284806a500 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284d219640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 0x7f284815c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284d219640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 0x7f284815c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39850/0 (socket says 172.21.15.45:39850) 2023-11-28T20:33:52.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284d219640 1 -- 172.21.15.45:0/935740951 learned_addr learned my addr 172.21.15.45:0/935740951 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:52.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f2847fff640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2848157c10 0x7f2848155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:52.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284ca18640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f284806a500 0x7f284809a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:52.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284d219640 1 -- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f284806a500 msgr2=0x7f284809a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:52.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284d219640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f284806a500 0x7f284809a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284d219640 1 -- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2848157c10 msgr2=0x7f2848155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284d219640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2848157c10 0x7f2848155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284d219640 1 -- 172.21.15.45:0/935740951 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28400094d0 con 0x7f284815ddd0 2023-11-28T20:33:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.266+0000 7f284d219640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 0x7f284815c8d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f28400090b0 tx=0x7f28400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.267+0000 7f28457fa640 1 -- 172.21.15.45:0/935740951 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284003abe0 con 0x7f284815ddd0 2023-11-28T20:33:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.267+0000 7f28457fa640 1 -- 172.21.15.45:0/935740951 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284003ad40 con 0x7f284815ddd0 2023-11-28T20:33:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.267+0000 7f28457fa640 1 -- 172.21.15.45:0/935740951 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840044630 con 0x7f284815ddd0 2023-11-28T20:33:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.267+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284809af90 con 0x7f284815ddd0 2023-11-28T20:33:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.267+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28480acc10 con 0x7f284815ddd0 2023-11-28T20:33:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.268+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28480ac790 con 0x7f284815ddd0 2023-11-28T20:33:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.269+0000 7f28457fa640 1 -- 172.21.15.45:0/935740951 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f284004d450 con 0x7f284815ddd0 2023-11-28T20:33:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.270+0000 7f28457fa640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f282806ae40 0x7f282806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.270+0000 7f28457fa640 1 -- 172.21.15.45:0/935740951 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f284002d040 con 0x7f284815ddd0 2023-11-28T20:33:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.270+0000 7f284ca18640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f282806ae40 0x7f282806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:52.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.272+0000 7f284ca18640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f282806ae40 0x7f282806d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28480ceaa0 tx=0x7f283800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:52.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.275+0000 7f28457fa640 1 -- 172.21.15.45:0/935740951 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28480ac790 con 0x7f284815ddd0 2023-11-28T20:33:52.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:52.473+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f28480ac790 con 0x7f284815ddd0 2023-11-28T20:33:53.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.024+0000 7f28457fa640 1 -- 172.21.15.45:0/935740951 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) v1 ==== 171+0+0 (secure 0 0 0) 0x7f28480ac790 con 0x7f284815ddd0 2023-11-28T20:33:53.026 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T20:33:53.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.028+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f282806ae40 msgr2=0x7f282806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:53.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.028+0000 7f284ec7c640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f282806ae40 0x7f282806d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28480ceaa0 tx=0x7f283800b040 comp rx=0 tx=0).stop 2023-11-28T20:33:53.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.028+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 msgr2=0x7f284815c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:53.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.028+0000 7f284ec7c640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 0x7f284815c8d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f28400090b0 tx=0x7f28400046c0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.028+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 shutdown_connections 2023-11-28T20:33:53.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.029+0000 7f284ec7c640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f282806ae40 0x7f282806d300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.029+0000 7f284ec7c640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f284815ddd0 0x7f284815c8d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.029+0000 7f284ec7c640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2848157c10 0x7f2848155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.029+0000 7f284ec7c640 1 --2- 172.21.15.45:0/935740951 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f284806a500 0x7f284809a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.029+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 >> 172.21.15.45:0/935740951 conn(0x7f28480ced60 msgr2=0x7f2848156b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:53.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.029+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 shutdown_connections 2023-11-28T20:33:53.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.031+0000 7f284ec7c640 1 -- 172.21.15.45:0/935740951 wait complete. 2023-11-28T20:33:53.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T20:33:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.264+0000 7f21df856640 1 Processor -- start 2023-11-28T20:33:53.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.264+0000 7f21df856640 1 -- start start 2023-11-28T20:33:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.266+0000 7f21df856640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f21d815e840 0x7f21d806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.266+0000 7f21df856640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21d815f720 0x7f21d80717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.266+0000 7f21df856640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d815ecd0 con 0x7f21d806a500 2023-11-28T20:33:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.266+0000 7f21df856640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d8071ce0 con 0x7f21d815f720 2023-11-28T20:33:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.266+0000 7f21df856640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d815f290 con 0x7f21d815e840 2023-11-28T20:33:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.266+0000 7f21dcdf1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21d815f720 0x7f21d80717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:53.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.266+0000 7f21dcdf1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21d815f720 0x7f21d80717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39854/0 (socket says 172.21.15.45:39854) 2023-11-28T20:33:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.266+0000 7f21dcdf1640 1 -- 172.21.15.45:0/3276088080 learned_addr learned my addr 172.21.15.45:0/3276088080 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.266+0000 7f21dd5f2640 1 --2- 172.21.15.45:0/3276088080 >> v2:172.21.15.190:3300/0 conn(0x7f21d815e840 0x7f21d806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.267+0000 7f21dcdf1640 1 -- 172.21.15.45:0/3276088080 >> v2:172.21.15.190:3300/0 conn(0x7f21d815e840 msgr2=0x7f21d806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.267+0000 7f21dcdf1640 1 --2- 172.21.15.45:0/3276088080 >> v2:172.21.15.190:3300/0 conn(0x7f21d815e840 0x7f21d806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.267+0000 7f21dcdf1640 1 -- 172.21.15.45:0/3276088080 >> v1:172.21.15.190:6789/0 conn(0x7f21d806a500 legacy=0x7f21d80cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.267+0000 7f21dcdf1640 1 -- 172.21.15.45:0/3276088080 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d815fb60 con 0x7f21d815f720 2023-11-28T20:33:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.268+0000 7f21dcdf1640 1 --2- 172.21.15.45:0/3276088080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815f720 0x7f21d80717a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21d80cb5c0 tx=0x7f21c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b532cfd5610d1b45 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.268+0000 7f21cffff640 1 -- 172.21.15.45:0/3276088080 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8031870 con 0x7f21d815f720 2023-11-28T20:33:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.268+0000 7f21cffff640 1 -- 172.21.15.45:0/3276088080 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c80319d0 con 0x7f21d815f720 2023-11-28T20:33:53.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.268+0000 7f21cffff640 1 -- 172.21.15.45:0/3276088080 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8031d60 con 0x7f21d815f720 2023-11-28T20:33:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.269+0000 7f21df856640 1 -- 172.21.15.45:0/3276088080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815f720 msgr2=0x7f21d80717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.269+0000 7f21df856640 1 --2- 172.21.15.45:0/3276088080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815f720 0x7f21d80717a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21d80cb5c0 tx=0x7f21c802b910 comp rx=0 tx=0).stop 2023-11-28T20:33:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.269+0000 7f21df856640 1 -- 172.21.15.45:0/3276088080 shutdown_connections 2023-11-28T20:33:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.269+0000 7f21df856640 1 --2- 172.21.15.45:0/3276088080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815f720 0x7f21d80717a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.269+0000 7f21df856640 1 --2- 172.21.15.45:0/3276088080 >> v2:172.21.15.190:3300/0 conn(0x7f21d815e840 0x7f21d806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.269+0000 7f21df856640 1 -- 172.21.15.45:0/3276088080 >> 172.21.15.45:0/3276088080 conn(0x7f21d80ced60 msgr2=0x7f21d80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.269+0000 7f21df856640 1 -- 172.21.15.45:0/3276088080 shutdown_connections 2023-11-28T20:33:53.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.269+0000 7f21df856640 1 -- 172.21.15.45:0/3276088080 wait complete. 2023-11-28T20:33:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.271+0000 7f21df856640 1 Processor -- start 2023-11-28T20:33:53.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.271+0000 7f21df856640 1 -- start start 2023-11-28T20:33:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.272+0000 7f21df856640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d806a500 0x7f21d81f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.272+0000 7f21df856640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815e840 0x7f21d81f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.272+0000 7f21df856640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21d815f720 0x7f21d80c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.272+0000 7f21df856640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80c57f0 con 0x7f21d815e840 2023-11-28T20:33:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.272+0000 7f21df856640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80c9d70 con 0x7f21d815f720 2023-11-28T20:33:53.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.272+0000 7f21df856640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8155fe0 con 0x7f21d806a500 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dcdf1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815e840 0x7f21d81f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dcdf1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815e840 0x7f21d81f33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39858/0 (socket says 172.21.15.45:39858) 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dcdf1640 1 -- 172.21.15.45:0/2443885006 learned_addr learned my addr 172.21.15.45:0/2443885006 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dcdf1640 1 -- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d806a500 msgr2=0x7f21d81f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dddf3640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21d815f720 0x7f21d80c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dcdf1640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d806a500 0x7f21d81f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dcdf1640 1 -- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21d815f720 msgr2=0x7f21d80c3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dcdf1640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21d815f720 0x7f21d80c3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dcdf1640 1 -- 172.21.15.45:0/2443885006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c80094d0 con 0x7f21d815e840 2023-11-28T20:33:53.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.273+0000 7f21dcdf1640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815e840 0x7f21d81f33d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000c00 tx=0x7f21c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:53.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.274+0000 7f21cdffb640 1 -- 172.21.15.45:0/2443885006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c803abe0 con 0x7f21d815e840 2023-11-28T20:33:53.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.274+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d80c9f60 con 0x7f21d815e840 2023-11-28T20:33:53.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.274+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d80ca520 con 0x7f21d815e840 2023-11-28T20:33:53.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.274+0000 7f21cdffb640 1 -- 172.21.15.45:0/2443885006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c803ad40 con 0x7f21d815e840 2023-11-28T20:33:53.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.274+0000 7f21cdffb640 1 -- 172.21.15.45:0/2443885006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c80445f0 con 0x7f21d815e840 2023-11-28T20:33:53.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.275+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b8005250 con 0x7f21d815e840 2023-11-28T20:33:53.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.276+0000 7f21cdffb640 1 -- 172.21.15.45:0/2443885006 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f21c8044750 con 0x7f21d815e840 2023-11-28T20:33:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.277+0000 7f21cdffb640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f21bc06ae40 0x7f21bc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.277+0000 7f21cdffb640 1 -- 172.21.15.45:0/2443885006 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f21c802d040 con 0x7f21d815e840 2023-11-28T20:33:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.278+0000 7f21dd5f2640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f21bc06ae40 0x7f21bc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:53.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.279+0000 7f21dd5f2640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f21bc06ae40 0x7f21bc06d300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21d40098f0 tx=0x7f21d4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:53.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.282+0000 7f21cdffb640 1 -- 172.21.15.45:0/2443885006 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21c8077020 con 0x7f21d815e840 2023-11-28T20:33:53.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:53.517+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f21b80057d0 con 0x7f21d815e840 2023-11-28T20:33:54.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.051+0000 7f21cdffb640 1 -- 172.21.15.45:0/2443885006 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f21c807a640 con 0x7f21d815e840 2023-11-28T20:33:54.054 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 2 and data pool 3 2023-11-28T20:33:54.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f21bc06ae40 msgr2=0x7f21bc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:54.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f21bc06ae40 0x7f21bc06d300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21d40098f0 tx=0x7f21d4008040 comp rx=0 tx=0).stop 2023-11-28T20:33:54.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815e840 msgr2=0x7f21d81f33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:54.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815e840 0x7f21d81f33d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000c00 tx=0x7f21c80046c0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 shutdown_connections 2023-11-28T20:33:54.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f21bc06ae40 0x7f21bc06d300 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21d815f720 0x7f21d80c3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d815e840 0x7f21d81f33d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 --2- 172.21.15.45:0/2443885006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d806a500 0x7f21d81f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.055+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 >> 172.21.15.45:0/2443885006 conn(0x7f21d80ced60 msgr2=0x7f21d80ac810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:54.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.056+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 shutdown_connections 2023-11-28T20:33:54.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.056+0000 7f21df856640 1 -- 172.21.15.45:0/2443885006 wait complete. 2023-11-28T20:33:54.085 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-11-28T20:33:54.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-11-28T20:33:54.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.276+0000 7fccdf27e640 1 Processor -- start 2023-11-28T20:33:54.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.277+0000 7fccdf27e640 1 -- start start 2023-11-28T20:33:54.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.278+0000 7fccdf27e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fccd815d3b0 0x7fccd80d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:54.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.278+0000 7fccdf27e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fccd80d26e0 0x7fccd80cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:54.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.278+0000 7fccdf27e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd80ce7f0 con 0x7fccd80ce340 2023-11-28T20:33:54.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.278+0000 7fccdf27e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd815d790 con 0x7fccd815d3b0 2023-11-28T20:33:54.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.278+0000 7fccdf27e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80d2bf0 con 0x7fccd80d26e0 2023-11-28T20:33:54.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.279+0000 7fccde27c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fccd815d3b0 0x7fccd80d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:54.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.279+0000 7fccdea7d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fccd80ce340 0x7fccd80cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46296/0 (socket says 172.21.15.45:46296) 2023-11-28T20:33:54.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.279+0000 7fccde27c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fccd815d3b0 0x7fccd80d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36814/0 (socket says 172.21.15.45:36814) 2023-11-28T20:33:54.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.279+0000 7fccdea7d640 1 -- 172.21.15.45:0/2469483598 learned_addr learned my addr 172.21.15.45:0/2469483598 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:33:54.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.279+0000 7fccdda7b640 1 --2- 172.21.15.45:0/2469483598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fccd80d26e0 0x7fccd80cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:54.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.279+0000 7fccde27c640 1 -- 172.21.15.45:0/2469483598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fccd80d26e0 msgr2=0x7fccd80cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:54.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.279+0000 7fccde27c640 1 --2- 172.21.15.45:0/2469483598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fccd80d26e0 0x7fccd80cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.279+0000 7fccde27c640 1 -- 172.21.15.45:0/2469483598 >> v1:172.21.15.45:6789/0 conn(0x7fccd80ce340 legacy=0x7fccd80cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:54.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.279+0000 7fccde27c640 1 -- 172.21.15.45:0/2469483598 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd80d2ed0 con 0x7fccd815d3b0 2023-11-28T20:33:54.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.280+0000 7fccde27c640 1 --2- 172.21.15.45:0/2469483598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd815d3b0 0x7fccd80d21a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fccd40097a0 tx=0x7fccd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664160cf8371db9d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.281+0000 7fccdd27a640 1 -- 172.21.15.45:0/2469483598 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4031830 con 0x7fccd815d3b0 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.281+0000 7fccdd27a640 1 -- 172.21.15.45:0/2469483598 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd4031990 con 0x7fccd815d3b0 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.281+0000 7fccdd27a640 1 -- 172.21.15.45:0/2469483598 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4031d20 con 0x7fccd815d3b0 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.281+0000 7fccdf27e640 1 -- 172.21.15.45:0/2469483598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd815d3b0 msgr2=0x7fccd80d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.281+0000 7fccdf27e640 1 --2- 172.21.15.45:0/2469483598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd815d3b0 0x7fccd80d21a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fccd40097a0 tx=0x7fccd402b8d0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.281+0000 7fccdf27e640 1 -- 172.21.15.45:0/2469483598 shutdown_connections 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.281+0000 7fccdf27e640 1 --2- 172.21.15.45:0/2469483598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fccd80d26e0 0x7fccd80cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.281+0000 7fccdf27e640 1 --2- 172.21.15.45:0/2469483598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd815d3b0 0x7fccd80d21a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.281+0000 7fccdf27e640 1 -- 172.21.15.45:0/2469483598 >> 172.21.15.45:0/2469483598 conn(0x7fccd8070f40 msgr2=0x7fccd8071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:54.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.282+0000 7fccdf27e640 1 -- 172.21.15.45:0/2469483598 shutdown_connections 2023-11-28T20:33:54.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.282+0000 7fccdf27e640 1 -- 172.21.15.45:0/2469483598 wait complete. 2023-11-28T20:33:54.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.283+0000 7fccdf27e640 1 Processor -- start 2023-11-28T20:33:54.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.284+0000 7fccdf27e640 1 -- start start 2023-11-28T20:33:54.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccdf27e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd80ce340 0x7fccd8150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:54.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccdf27e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fccd80d26e0 0x7fccd815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:54.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccdf27e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccd815d3b0 0x7fccd80b8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:54.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccdf27e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8150ea0 con 0x7fccd80ce340 2023-11-28T20:33:54.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccdf27e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80c8350 con 0x7fccd815d3b0 2023-11-28T20:33:54.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccdf27e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80c7740 con 0x7fccd80d26e0 2023-11-28T20:33:54.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccde27c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd80ce340 0x7fccd8150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:54.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccde27c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd80ce340 0x7fccd8150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36830/0 (socket says 172.21.15.45:36830) 2023-11-28T20:33:54.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccde27c640 1 -- 172.21.15.45:0/2456002757 learned_addr learned my addr 172.21.15.45:0/2456002757 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:54.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccde27c640 1 -- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fccd80d26e0 msgr2=0x7fccd815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:54.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccde27c640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fccd80d26e0 0x7fccd815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccde27c640 1 -- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccd815d3b0 msgr2=0x7fccd80b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:54.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccde27c640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccd815d3b0 0x7fccd80b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccde27c640 1 -- 172.21.15.45:0/2456002757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4009400 con 0x7fccd80ce340 2023-11-28T20:33:54.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.285+0000 7fccde27c640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd80ce340 0x7fccd8150960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fccd4034040 tx=0x7fccd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:54.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.286+0000 7fccceffd640 1 -- 172.21.15.45:0/2456002757 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd403aa60 con 0x7fccd80ce340 2023-11-28T20:33:54.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.286+0000 7fccceffd640 1 -- 172.21.15.45:0/2456002757 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd403abe0 con 0x7fccd80ce340 2023-11-28T20:33:54.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.286+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd80c7a20 con 0x7fccd80ce340 2023-11-28T20:33:54.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.286+0000 7fccceffd640 1 -- 172.21.15.45:0/2456002757 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4044400 con 0x7fccd80ce340 2023-11-28T20:33:54.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.286+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd80c7fe0 con 0x7fccd80ce340 2023-11-28T20:33:54.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.287+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd80c8170 con 0x7fccd80ce340 2023-11-28T20:33:54.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.288+0000 7fccceffd640 1 -- 172.21.15.45:0/2456002757 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fccd404d440 con 0x7fccd80ce340 2023-11-28T20:33:54.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.289+0000 7fccceffd640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fccbc06ae40 0x7fccbc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:54.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.289+0000 7fccceffd640 1 -- 172.21.15.45:0/2456002757 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccd40ab7b0 con 0x7fccd80ce340 2023-11-28T20:33:54.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.290+0000 7fccdda7b640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fccbc06ae40 0x7fccbc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:54.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.291+0000 7fccdda7b640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fccbc06ae40 0x7fccbc06d300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fccd80cde40 tx=0x7fccc8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:54.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.294+0000 7fccceffd640 1 -- 172.21.15.45:0/2456002757 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fccd4076980 con 0x7fccd80ce340 2023-11-28T20:33:54.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.507+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fccd80733d0 con 0x7fccd80ce340 2023-11-28T20:33:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.911+0000 7fccceffd640 1 -- 172.21.15.45:0/2456002757 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) v1 ==== 157+0+0 (secure 0 0 0) 0x7fccd40350e0 con 0x7fccd80ce340 2023-11-28T20:33:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fccbc06ae40 msgr2=0x7fccbc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fccbc06ae40 0x7fccbc06d300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fccd80cde40 tx=0x7fccc8008040 comp rx=0 tx=0).stop 2023-11-28T20:33:54.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd80ce340 msgr2=0x7fccd8150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd80ce340 0x7fccd8150960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fccd4034040 tx=0x7fccd40046c0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 shutdown_connections 2023-11-28T20:33:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fccbc06ae40 0x7fccbc06d300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fccd815d3b0 0x7fccd80b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fccd80d26e0 0x7fccd815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 --2- 172.21.15.45:0/2456002757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccd80ce340 0x7fccd8150960 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.915+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 >> 172.21.15.45:0/2456002757 conn(0x7fccd8070f40 msgr2=0x7fccd806a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.917+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 shutdown_connections 2023-11-28T20:33:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:54.917+0000 7fccdf27e640 1 -- 172.21.15.45:0/2456002757 wait complete. 2023-11-28T20:33:54.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T20:33:55.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.160+0000 7f1b1abf4640 1 Processor -- start 2023-11-28T20:33:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.160+0000 7f1b1abf4640 1 -- start start 2023-11-28T20:33:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.162+0000 7f1b1abf4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b1415b190 0x7f1b1406de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.162+0000 7f1b1abf4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1b14070030 0x7f1b140b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.162+0000 7f1b1abf4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1406e3a0 con 0x7f1b1406c090 2023-11-28T20:33:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.162+0000 7f1b1abf4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b1415b570 con 0x7f1b1415b190 2023-11-28T20:33:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.162+0000 7f1b1abf4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b140c9d20 con 0x7f1b14070030 2023-11-28T20:33:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.162+0000 7f1b19bf2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b1415b190 0x7f1b1406de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.162+0000 7f1b19bf2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b1415b190 0x7f1b1406de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36838/0 (socket says 172.21.15.45:36838) 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.162+0000 7f1b1a3f3640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1b1406c090 0x7f1b140ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46310/0 (socket says 172.21.15.45:46310) 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.162+0000 7f1b19bf2640 1 -- 172.21.15.45:0/2691443442 learned_addr learned my addr 172.21.15.45:0/2691443442 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.163+0000 7f1b193f1640 1 --2- 172.21.15.45:0/2691443442 >> v2:172.21.15.190:3300/0 conn(0x7f1b14070030 0x7f1b140b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.163+0000 7f1b19bf2640 1 -- 172.21.15.45:0/2691443442 >> v2:172.21.15.190:3300/0 conn(0x7f1b14070030 msgr2=0x7f1b140b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.163+0000 7f1b19bf2640 1 --2- 172.21.15.45:0/2691443442 >> v2:172.21.15.190:3300/0 conn(0x7f1b14070030 0x7f1b140b1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.163+0000 7f1b19bf2640 1 -- 172.21.15.45:0/2691443442 >> v1:172.21.15.45:6789/0 conn(0x7f1b1406c090 legacy=0x7f1b140ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.163+0000 7f1b18bf0640 1 -- 172.21.15.45:0/2691443442 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172312718 0 0) 0x7f1b0c001110 con 0x7f1b1406c090 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.163+0000 7f1b19bf2640 1 -- 172.21.15.45:0/2691443442 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b14071c50 con 0x7f1b1415b190 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.164+0000 7f1b19bf2640 1 --2- 172.21.15.45:0/2691443442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1415b190 0x7f1b1406de60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1b140cc750 tx=0x7f1b1002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866cc35183657f7d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.165+0000 7f1b18bf0640 1 -- 172.21.15.45:0/2691443442 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1002cb20 con 0x7f1b1415b190 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.165+0000 7f1b18bf0640 1 -- 172.21.15.45:0/2691443442 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1002cc80 con 0x7f1b1415b190 2023-11-28T20:33:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.165+0000 7f1b18bf0640 1 -- 172.21.15.45:0/2691443442 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b100343f0 con 0x7f1b1415b190 2023-11-28T20:33:55.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.165+0000 7f1b1abf4640 1 -- 172.21.15.45:0/2691443442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1415b190 msgr2=0x7f1b1406de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.165+0000 7f1b1abf4640 1 --2- 172.21.15.45:0/2691443442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1415b190 0x7f1b1406de60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1b140cc750 tx=0x7f1b1002bf80 comp rx=0 tx=0).stop 2023-11-28T20:33:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.165+0000 7f1b1abf4640 1 -- 172.21.15.45:0/2691443442 shutdown_connections 2023-11-28T20:33:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.165+0000 7f1b1abf4640 1 --2- 172.21.15.45:0/2691443442 >> v2:172.21.15.190:3300/0 conn(0x7f1b14070030 0x7f1b140b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.165+0000 7f1b1abf4640 1 --2- 172.21.15.45:0/2691443442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1415b190 0x7f1b1406de60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.165+0000 7f1b1abf4640 1 -- 172.21.15.45:0/2691443442 >> 172.21.15.45:0/2691443442 conn(0x7f1b140cec30 msgr2=0x7f1b140cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.166+0000 7f1b1abf4640 1 -- 172.21.15.45:0/2691443442 shutdown_connections 2023-11-28T20:33:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.166+0000 7f1b1abf4640 1 -- 172.21.15.45:0/2691443442 wait complete. 2023-11-28T20:33:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.167+0000 7f1b1abf4640 1 Processor -- start 2023-11-28T20:33:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.168+0000 7f1b1abf4640 1 -- start start 2023-11-28T20:33:55.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.168+0000 7f1b1abf4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1406c090 0x7f1b140c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:55.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.168+0000 7f1b1abf4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b14070030 0x7f1b140c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b1abf4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b1415b190 0x7f1b140b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b1abf4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140c8c30 con 0x7f1b1406c090 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b1abf4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140c7650 con 0x7f1b1415b190 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b1abf4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140c7930 con 0x7f1b14070030 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b19bf2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1406c090 0x7f1b140c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b19bf2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1406c090 0x7f1b140c86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36846/0 (socket says 172.21.15.45:36846) 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b19bf2640 1 -- 172.21.15.45:0/3925684707 learned_addr learned my addr 172.21.15.45:0/3925684707 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b1a3f3640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b1415b190 0x7f1b140b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b193f1640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b14070030 0x7f1b140c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b19bf2640 1 -- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b14070030 msgr2=0x7f1b140c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b19bf2640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b14070030 0x7f1b140c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b19bf2640 1 -- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b1415b190 msgr2=0x7f1b140b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b19bf2640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b1415b190 0x7f1b140b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b19bf2640 1 -- 172.21.15.45:0/3925684707 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b10009b30 con 0x7f1b1406c090 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b1a3f3640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b1415b190 0x7f1b140b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.169+0000 7f1b193f1640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b14070030 0x7f1b140c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.170+0000 7f1b19bf2640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1406c090 0x7f1b140c86f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1b10000f80 tx=0x7f1b10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.170+0000 7f1b0a7fc640 1 -- 172.21.15.45:0/3925684707 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b10007e90 con 0x7f1b1406c090 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.170+0000 7f1b0a7fc640 1 -- 172.21.15.45:0/3925684707 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b10009510 con 0x7f1b1406c090 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.170+0000 7f1b0a7fc640 1 -- 172.21.15.45:0/3925684707 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1003e690 con 0x7f1b1406c090 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.170+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b140c7c10 con 0x7f1b1406c090 2023-11-28T20:33:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.170+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b140c81d0 con 0x7f1b1406c090 2023-11-28T20:33:55.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.171+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b14071ed0 con 0x7f1b1406c090 2023-11-28T20:33:55.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.172+0000 7f1b0a7fc640 1 -- 172.21.15.45:0/3925684707 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1b1003e7f0 con 0x7f1b1406c090 2023-11-28T20:33:55.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.173+0000 7f1b0a7fc640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1af806ae40 0x7f1af806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:55.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.173+0000 7f1b0a7fc640 1 -- 172.21.15.45:0/3925684707 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f1b100ac480 con 0x7f1b1406c090 2023-11-28T20:33:55.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.174+0000 7f1b193f1640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1af806ae40 0x7f1af806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.175+0000 7f1b193f1640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1af806ae40 0x7f1af806d300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1b14158a30 tx=0x7f1b04008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.178+0000 7f1b0a7fc640 1 -- 172.21.15.45:0/3925684707 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b14071ed0 con 0x7f1b1406c090 2023-11-28T20:33:55.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:55.372+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1b14071ed0 con 0x7f1b1406c090 2023-11-28T20:33:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.082+0000 7f1b0a7fc640 1 -- 172.21.15.45:0/3925684707 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) v1 ==== 228+0+0 (secure 0 0 0) 0x7f1b14071ed0 con 0x7f1b1406c090 2023-11-28T20:33:57.085 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data_ec' created 2023-11-28T20:33:57.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.086+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1af806ae40 msgr2=0x7f1af806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:57.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1af806ae40 0x7f1af806d300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1b14158a30 tx=0x7f1b04008040 comp rx=0 tx=0).stop 2023-11-28T20:33:57.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1406c090 msgr2=0x7f1b140c86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:57.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1406c090 0x7f1b140c86f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1b10000f80 tx=0x7f1b10004a40 comp rx=0 tx=0).stop 2023-11-28T20:33:57.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 shutdown_connections 2023-11-28T20:33:57.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1af806ae40 0x7f1af806d300 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b1415b190 0x7f1b140b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b14070030 0x7f1b140c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 --2- 172.21.15.45:0/3925684707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b1406c090 0x7f1b140c86f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 >> 172.21.15.45:0/3925684707 conn(0x7f1b140cec30 msgr2=0x7f1b14070530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 shutdown_connections 2023-11-28T20:33:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.087+0000 7f1b1abf4640 1 -- 172.21.15.45:0/3925684707 wait complete. 2023-11-28T20:33:57.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-11-28T20:33:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.316+0000 7ff941bc9640 1 Processor -- start 2023-11-28T20:33:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.316+0000 7ff941bc9640 1 -- start start 2023-11-28T20:33:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.317+0000 7ff941bc9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff93c1607c0 0x7ff93c0cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.317+0000 7ff941bc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c070030 0x7ff93c070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.317+0000 7ff941bc9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93c0acad0 con 0x7ff93c0cc900 2023-11-28T20:33:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.317+0000 7ff941bc9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff93c0ac8c0 con 0x7ff93c1607c0 2023-11-28T20:33:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.317+0000 7ff941bc9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0acdb0 con 0x7ff93c070030 2023-11-28T20:33:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.318+0000 7ff9413c8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c070030 0x7ff93c070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.318+0000 7ff940bc7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff93c1607c0 0x7ff93c0cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.318+0000 7ff9413c8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c070030 0x7ff93c070460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:54806/0 (socket says 172.21.15.45:54806) 2023-11-28T20:33:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.318+0000 7ff9413c8640 1 -- 172.21.15.45:0/584463984 learned_addr learned my addr 172.21.15.45:0/584463984 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.319+0000 7ff940bc7640 1 -- 172.21.15.45:0/584463984 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c070030 msgr2=0x7ff93c070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:57.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.319+0000 7ff940bc7640 1 --2- 172.21.15.45:0/584463984 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c070030 0x7ff93c070460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:57.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.319+0000 7ff940bc7640 1 -- 172.21.15.45:0/584463984 >> v1:172.21.15.190:6789/0 conn(0x7ff93c0cc900 legacy=0x7ff93c0adb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:57.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.319+0000 7ff940bc7640 1 -- 172.21.15.45:0/584463984 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93c0ad090 con 0x7ff93c1607c0 2023-11-28T20:33:57.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.320+0000 7ff940bc7640 1 --2- 172.21.15.45:0/584463984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93c1607c0 0x7ff93c0cd720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9300097a0 tx=0x7ff93002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f21ef65fd521fb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:57.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff93b7fe640 1 -- 172.21.15.45:0/584463984 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930031780 con 0x7ff93c1607c0 2023-11-28T20:33:57.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff93b7fe640 1 -- 172.21.15.45:0/584463984 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9300318e0 con 0x7ff93c1607c0 2023-11-28T20:33:57.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff93b7fe640 1 -- 172.21.15.45:0/584463984 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930031c70 con 0x7ff93c1607c0 2023-11-28T20:33:57.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff941bc9640 1 -- 172.21.15.45:0/584463984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93c1607c0 msgr2=0x7ff93c0cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:57.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff941bc9640 1 --2- 172.21.15.45:0/584463984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93c1607c0 0x7ff93c0cd720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9300097a0 tx=0x7ff93002b820 comp rx=0 tx=0).stop 2023-11-28T20:33:57.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff941bc9640 1 -- 172.21.15.45:0/584463984 shutdown_connections 2023-11-28T20:33:57.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff941bc9640 1 --2- 172.21.15.45:0/584463984 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c070030 0x7ff93c070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:57.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff941bc9640 1 --2- 172.21.15.45:0/584463984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93c1607c0 0x7ff93c0cd720 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:57.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff941bc9640 1 -- 172.21.15.45:0/584463984 >> 172.21.15.45:0/584463984 conn(0x7ff93c0cec30 msgr2=0x7ff93c0cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:57.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff941bc9640 1 -- 172.21.15.45:0/584463984 shutdown_connections 2023-11-28T20:33:57.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.321+0000 7ff941bc9640 1 -- 172.21.15.45:0/584463984 wait complete. 2023-11-28T20:33:57.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.323+0000 7ff941bc9640 1 Processor -- start 2023-11-28T20:33:57.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.323+0000 7ff941bc9640 1 -- start start 2023-11-28T20:33:57.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.324+0000 7ff941bc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff93c070030 0x7ff93c1f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:57.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.324+0000 7ff941bc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c0cc900 0x7ff93c156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:57.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.324+0000 7ff941bc9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93c1607c0 0x7ff93c1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:57.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.324+0000 7ff941bc9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c159a10 con 0x7ff93c070030 2023-11-28T20:33:57.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.324+0000 7ff941bc9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c1561d0 con 0x7ff93c1607c0 2023-11-28T20:33:57.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.324+0000 7ff941bc9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c159c40 con 0x7ff93c0cc900 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff940bc7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff93c070030 0x7ff93c1f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff940bc7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff93c070030 0x7ff93c1f2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36848/0 (socket says 172.21.15.45:36848) 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff940bc7640 1 -- 172.21.15.45:0/399662649 learned_addr learned my addr 172.21.15.45:0/399662649 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff93bfff640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c0cc900 0x7ff93c156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff9413c8640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93c1607c0 0x7ff93c1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff940bc7640 1 -- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c0cc900 msgr2=0x7ff93c156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff940bc7640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c0cc900 0x7ff93c156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff940bc7640 1 -- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93c1607c0 msgr2=0x7ff93c1594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff940bc7640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93c1607c0 0x7ff93c1594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff940bc7640 1 -- 172.21.15.45:0/399662649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff930009400 con 0x7ff93c070030 2023-11-28T20:33:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.325+0000 7ff940bc7640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff93c070030 0x7ff93c1f2f80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff930034040 tx=0x7ff9300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.326+0000 7ff9397fa640 1 -- 172.21.15.45:0/399662649 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930032820 con 0x7ff93c070030 2023-11-28T20:33:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.326+0000 7ff9397fa640 1 -- 172.21.15.45:0/399662649 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff930032980 con 0x7ff93c070030 2023-11-28T20:33:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.326+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93c0c3e00 con 0x7ff93c070030 2023-11-28T20:33:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.326+0000 7ff9397fa640 1 -- 172.21.15.45:0/399662649 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930032d30 con 0x7ff93c070030 2023-11-28T20:33:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.326+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93c0c43c0 con 0x7ff93c070030 2023-11-28T20:33:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.327+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff93c0733d0 con 0x7ff93c070030 2023-11-28T20:33:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.328+0000 7ff9397fa640 1 -- 172.21.15.45:0/399662649 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff93004e440 con 0x7ff93c070030 2023-11-28T20:33:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.329+0000 7ff9397fa640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff92806ae70 0x7ff92806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.329+0000 7ff9397fa640 1 -- 172.21.15.45:0/399662649 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6428+0+0 (secure 0 0 0) 0x7ff9300aba40 con 0x7ff93c070030 2023-11-28T20:33:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.330+0000 7ff93bfff640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff92806ae70 0x7ff92806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.331+0000 7ff93bfff640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff92806ae70 0x7ff92806d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff93c1f6450 tx=0x7ff92c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.334+0000 7ff9397fa640 1 -- 172.21.15.45:0/399662649 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff930076950 con 0x7ff93c070030 2023-11-28T20:33:57.350 INFO:tasks.ceph.mds.c.smithi045.stderr:2023-11-28T20:33:57.347+0000 7f9540fa8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:33:57.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:57.527+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7ff93c0b4bb0 con 0x7ff93c070030 2023-11-28T20:33:58.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.092+0000 7ff9397fa640 1 -- 172.21.15.45:0/399662649 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff930079f70 con 0x7ff93c070030 2023-11-28T20:33:58.095 INFO:teuthology.orchestra.run.smithi045.stderr:set pool 4 allow_ec_overwrites to true 2023-11-28T20:33:58.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff92806ae70 msgr2=0x7ff92806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:58.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff92806ae70 0x7ff92806d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff93c1f6450 tx=0x7ff92c009040 comp rx=0 tx=0).stop 2023-11-28T20:33:58.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff93c070030 msgr2=0x7ff93c1f2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:58.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff93c070030 0x7ff93c1f2f80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff930034040 tx=0x7ff9300046c0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 shutdown_connections 2023-11-28T20:33:58.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff92806ae70 0x7ff92806d330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93c1607c0 0x7ff93c1594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff93c0cc900 0x7ff93c156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 --2- 172.21.15.45:0/399662649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff93c070030 0x7ff93c1f2f80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.096+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 >> 172.21.15.45:0/399662649 conn(0x7ff93c0cec30 msgr2=0x7ff93c0af7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:58.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.097+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 shutdown_connections 2023-11-28T20:33:58.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.097+0000 7ff941bc9640 1 -- 172.21.15.45:0/399662649 wait complete. 2023-11-28T20:33:58.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-11-28T20:33:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.330+0000 7ffa66487640 1 Processor -- start 2023-11-28T20:33:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.330+0000 7ffa66487640 1 -- start start 2023-11-28T20:33:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.331+0000 7ffa66487640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ffa6006dc50 0x7ffa600cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.331+0000 7ffa66487640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6006e030 con 0x7ffa6006ac10 2023-11-28T20:33:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.331+0000 7ffa66487640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa600ccf60 con 0x7ffa600ce600 2023-11-28T20:33:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.331+0000 7ffa66487640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa600cd930 con 0x7ffa6006dc50 2023-11-28T20:33:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.332+0000 7ffa64a24640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ffa6006ac10 0x7ffa600c93b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46320/0 (socket says 172.21.15.45:46320) 2023-11-28T20:33:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.332+0000 7ffa64a24640 1 -- 172.21.15.45:0/477527553 learned_addr learned my addr 172.21.15.45:0/477527553 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:33:58.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.332+0000 7ffa5ffff640 1 --2- 172.21.15.45:0/477527553 >> v2:172.21.15.190:3300/0 conn(0x7ffa6006dc50 0x7ffa600cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:58.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.332+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68369354 0 0) 0x7ffa58001490 con 0x7ffa6006ac10 2023-11-28T20:33:58.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.332+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa48003520 con 0x7ffa6006ac10 2023-11-28T20:33:58.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2611782913 0 0) 0x7ffa48003520 con 0x7ffa6006ac10 2023-11-28T20:33:58.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 >> v2:172.21.15.190:3300/0 conn(0x7ffa6006dc50 msgr2=0x7ffa600cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 --2- 172.21.15.45:0/477527553 >> v2:172.21.15.190:3300/0 conn(0x7ffa6006dc50 0x7ffa600cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 >> v1:172.21.15.190:6789/0 conn(0x7ffa600ce600 legacy=0x7ffa6006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa600cc930 con 0x7ffa6006ac10 2023-11-28T20:33:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ffa58001b80 con 0x7ffa6006ac10 2023-11-28T20:33:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 >> v1:172.21.15.45:6789/0 conn(0x7ffa6006ac10 legacy=0x7ffa600c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa48003970 0x7ffa48003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa480047c0 0x7ffa48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600cd930 con 0x7ffa48005180 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4800c020 con 0x7ffa48003970 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4800c1e0 con 0x7ffa480047c0 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa64a24640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.333+0000 7ffa5ffff640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa48003970 0x7ffa48003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.334+0000 7ffa5f7fe640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa480047c0 0x7ffa48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.334+0000 7ffa64a24640 1 -- 172.21.15.45:0/477527553 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa480047c0 msgr2=0x7ffa48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.334+0000 7ffa64a24640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa480047c0 0x7ffa48004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.334+0000 7ffa64a24640 1 -- 172.21.15.45:0/477527553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa48003970 msgr2=0x7ffa48003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.334+0000 7ffa64a24640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa48003970 0x7ffa48003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.334+0000 7ffa64a24640 1 -- 172.21.15.45:0/477527553 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa48003520 con 0x7ffa48005180 2023-11-28T20:33:58.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.335+0000 7ffa64a24640 1 -- 172.21.15.45:0/477527553 reap_dead start 2023-11-28T20:33:58.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.335+0000 7ffa64a24640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa4800b980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800a800 tx=0x7ffa480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35dcceaa0f9575b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:58.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.335+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58034630 con 0x7ffa48005180 2023-11-28T20:33:58.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.335+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5802f070 con 0x7ffa48005180 2023-11-28T20:33:58.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.335+0000 7ffa5effd640 1 -- 172.21.15.45:0/477527553 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58034990 con 0x7ffa48005180 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.336+0000 7ffa66487640 1 -- 172.21.15.45:0/477527553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 msgr2=0x7ffa4800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.336+0000 7ffa66487640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa4800b980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800a800 tx=0x7ffa480057f0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.336+0000 7ffa66487640 1 -- 172.21.15.45:0/477527553 shutdown_connections 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.336+0000 7ffa66487640 1 --2- 172.21.15.45:0/477527553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa4800b980 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.336+0000 7ffa66487640 1 -- 172.21.15.45:0/477527553 >> 172.21.15.45:0/477527553 conn(0x7ffa600ced60 msgr2=0x7ffa600cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.336+0000 7ffa66487640 1 -- 172.21.15.45:0/477527553 shutdown_connections 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.336+0000 7ffa66487640 1 -- 172.21.15.45:0/477527553 wait complete. 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.338+0000 7ffa66487640 1 Processor -- start 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.338+0000 7ffa66487640 1 -- start start 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa66487640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa600b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa66487640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa600c7f70 0x7ffa6015f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa66487640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6015b470 0x7ffa6015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa66487640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600c9140 con 0x7ffa48005180 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa66487640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600c8530 con 0x7ffa6015b470 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa66487640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600c8810 con 0x7ffa600c7f70 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa600b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa600b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36868/0 (socket says 172.21.15.45:36868) 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa5ffff640 1 -- 172.21.15.45:0/288769441 learned_addr learned my addr 172.21.15.45:0/288769441 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa64a24640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6015b470 0x7ffa6015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa5f7fe640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa600c7f70 0x7ffa6015f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa5ffff640 1 -- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa600c7f70 msgr2=0x7ffa6015f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa5ffff640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa600c7f70 0x7ffa6015f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa5ffff640 1 -- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6015b470 msgr2=0x7ffa6015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.339+0000 7ffa5ffff640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6015b470 0x7ffa6015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.340+0000 7ffa5ffff640 1 -- 172.21.15.45:0/288769441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa600c8af0 con 0x7ffa48005180 2023-11-28T20:33:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.340+0000 7ffa64a24640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6015b470 0x7ffa6015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.340+0000 7ffa5ffff640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa600b4580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa50011540 tx=0x7ffa50011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.340+0000 7ffa5cff9640 1 -- 172.21.15.45:0/288769441 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa50015930 con 0x7ffa48005180 2023-11-28T20:33:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.340+0000 7ffa5cff9640 1 -- 172.21.15.45:0/288769441 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa50015a90 con 0x7ffa48005180 2023-11-28T20:33:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.340+0000 7ffa5cff9640 1 -- 172.21.15.45:0/288769441 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa50015e60 con 0x7ffa48005180 2023-11-28T20:33:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.340+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa600c8dd0 con 0x7ffa48005180 2023-11-28T20:33:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.340+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6015c260 con 0x7ffa48005180 2023-11-28T20:33:58.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.342+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa600733d0 con 0x7ffa48005180 2023-11-28T20:33:58.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.342+0000 7ffa5cff9640 1 -- 172.21.15.45:0/288769441 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffa5000f460 con 0x7ffa48005180 2023-11-28T20:33:58.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.343+0000 7ffa5cff9640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa4c06ae40 0x7ffa4c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:58.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.343+0000 7ffa5cff9640 1 -- 172.21.15.45:0/288769441 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6428+0+0 (secure 0 0 0) 0x7ffa5008a190 con 0x7ffa48005180 2023-11-28T20:33:58.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.344+0000 7ffa5f7fe640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa4c06ae40 0x7ffa4c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:58.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.345+0000 7ffa5f7fe640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa4c06ae40 0x7ffa4c06d300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffa54000c00 tx=0x7ffa54009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:58.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.348+0000 7ffa5cff9640 1 -- 172.21.15.45:0/288769441 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa50054ed0 con 0x7ffa48005180 2023-11-28T20:33:58.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:58.581+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7ffa600caf40 con 0x7ffa48005180 2023-11-28T20:33:59.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.096+0000 7ffa5cff9640 1 -- 172.21.15.45:0/288769441 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7ffa500584f0 con 0x7ffa48005180 2023-11-28T20:33:59.099 INFO:teuthology.orchestra.run.smithi045.stderr:added data pool 4 to fsmap 2023-11-28T20:33:59.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.099+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa4c06ae40 msgr2=0x7ffa4c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.099+0000 7ffa66487640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa4c06ae40 0x7ffa4c06d300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffa54000c00 tx=0x7ffa54009040 comp rx=0 tx=0).stop 2023-11-28T20:33:59.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.099+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 msgr2=0x7ffa600b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.099+0000 7ffa66487640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa600b4580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa50011540 tx=0x7ffa50011a00 comp rx=0 tx=0).stop 2023-11-28T20:33:59.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.100+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 shutdown_connections 2023-11-28T20:33:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.100+0000 7ffa66487640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa4c06ae40 0x7ffa4c06d300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.100+0000 7ffa66487640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6015b470 0x7ffa6015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.100+0000 7ffa66487640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa600c7f70 0x7ffa6015f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.100+0000 7ffa66487640 1 --2- 172.21.15.45:0/288769441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa48005180 0x7ffa600b4580 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.100+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 >> 172.21.15.45:0/288769441 conn(0x7ffa600ced60 msgr2=0x7ffa6006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.100+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 shutdown_connections 2023-11-28T20:33:59.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.100+0000 7ffa66487640 1 -- 172.21.15.45:0/288769441 wait complete. 2023-11-28T20:33:59.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:33:59.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.347+0000 7fb748ba9640 1 Processor -- start 2023-11-28T20:33:59.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.347+0000 7fb748ba9640 1 -- start start 2023-11-28T20:33:59.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb748ba9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7441438b0 0x7fb7440b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb748ba9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb7440b24e0 0x7fb7440b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb748ba9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74415dcc0 con 0x7fb7440b1170 2023-11-28T20:33:59.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb748ba9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7440b1600 con 0x7fb7441438b0 2023-11-28T20:33:59.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb748ba9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74415e840 con 0x7fb7440b24e0 2023-11-28T20:33:59.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb74259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7441438b0 0x7fb7440b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb742d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb7440b1170 0x7fb7440ccdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46330/0 (socket says 172.21.15.45:46330) 2023-11-28T20:33:59.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb742d9d640 1 -- 172.21.15.45:0/1637592698 learned_addr learned my addr 172.21.15.45:0/1637592698 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:33:59.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb73bfff640 1 --2- 172.21.15.45:0/1637592698 >> v2:172.21.15.190:3300/0 conn(0x7fb7440b24e0 0x7fb7440b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb74259c640 1 -- 172.21.15.45:0/1637592698 >> v2:172.21.15.190:3300/0 conn(0x7fb7440b24e0 msgr2=0x7fb7440b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:59.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.349+0000 7fb74259c640 1 --2- 172.21.15.45:0/1637592698 >> v2:172.21.15.190:3300/0 conn(0x7fb7440b24e0 0x7fb7440b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.350+0000 7fb74259c640 1 -- 172.21.15.45:0/1637592698 >> v1:172.21.15.45:6789/0 conn(0x7fb7440b1170 legacy=0x7fb7440ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.350+0000 7fb741d9b640 1 -- 172.21.15.45:0/1637592698 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513993622 0 0) 0x7fb74415dcc0 con 0x7fb7440b1170 2023-11-28T20:33:59.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.350+0000 7fb74259c640 1 -- 172.21.15.45:0/1637592698 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74415df10 con 0x7fb7441438b0 2023-11-28T20:33:59.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.351+0000 7fb74259c640 1 --2- 172.21.15.45:0/1637592698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 0x7fb7440b1fa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb7440ce2b0 tx=0x7fb73402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0309af84da172 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:59.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.351+0000 7fb741d9b640 1 -- 172.21.15.45:0/1637592698 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734031710 con 0x7fb7441438b0 2023-11-28T20:33:59.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.351+0000 7fb741d9b640 1 -- 172.21.15.45:0/1637592698 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73402f050 con 0x7fb7441438b0 2023-11-28T20:33:59.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.351+0000 7fb741d9b640 1 -- 172.21.15.45:0/1637592698 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734031a90 con 0x7fb7441438b0 2023-11-28T20:33:59.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.352+0000 7fb748ba9640 1 -- 172.21.15.45:0/1637592698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 msgr2=0x7fb7440b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.352+0000 7fb748ba9640 1 --2- 172.21.15.45:0/1637592698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 0x7fb7440b1fa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb7440ce2b0 tx=0x7fb73402b7b0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.352+0000 7fb748ba9640 1 -- 172.21.15.45:0/1637592698 shutdown_connections 2023-11-28T20:33:59.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.352+0000 7fb748ba9640 1 --2- 172.21.15.45:0/1637592698 >> v2:172.21.15.190:3300/0 conn(0x7fb7440b24e0 0x7fb7440b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.352+0000 7fb748ba9640 1 --2- 172.21.15.45:0/1637592698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 0x7fb7440b1fa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.352+0000 7fb748ba9640 1 -- 172.21.15.45:0/1637592698 >> 172.21.15.45:0/1637592698 conn(0x7fb74406a500 msgr2=0x7fb74406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:59.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.352+0000 7fb748ba9640 1 -- 172.21.15.45:0/1637592698 shutdown_connections 2023-11-28T20:33:59.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.352+0000 7fb748ba9640 1 -- 172.21.15.45:0/1637592698 wait complete. 2023-11-28T20:33:59.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.354+0000 7fb748ba9640 1 Processor -- start 2023-11-28T20:33:59.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.354+0000 7fb748ba9640 1 -- start start 2023-11-28T20:33:59.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb748ba9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7440b1170 0x7fb7441582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb748ba9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7440b24e0 0x7fb744135640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb748ba9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 0x7fb7440ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb748ba9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744159750 con 0x7fb7441438b0 2023-11-28T20:33:59.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb748ba9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744158b40 con 0x7fb7440b24e0 2023-11-28T20:33:59.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb748ba9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb744158e20 con 0x7fb7440b1170 2023-11-28T20:33:59.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb742d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 0x7fb7440ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb742d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 0x7fb7440ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36882/0 (socket says 172.21.15.45:36882) 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb742d9d640 1 -- 172.21.15.45:0/3474734286 learned_addr learned my addr 172.21.15.45:0/3474734286 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb74259c640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7440b1170 0x7fb7441582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb73bfff640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7440b24e0 0x7fb744135640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb742d9d640 1 -- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7440b1170 msgr2=0x7fb7441582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.355+0000 7fb742d9d640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7440b1170 0x7fb7441582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.356+0000 7fb742d9d640 1 -- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7440b24e0 msgr2=0x7fb744135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.356+0000 7fb742d9d640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7440b24e0 0x7fb744135640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.356+0000 7fb742d9d640 1 -- 172.21.15.45:0/3474734286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734009400 con 0x7fb7441438b0 2023-11-28T20:33:59.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.356+0000 7fb742d9d640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 0x7fb7440ad350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00c780 tx=0x7fb73c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:59.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.356+0000 7fb73affd640 1 -- 172.21.15.45:0/3474734286 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00ce90 con 0x7fb7441438b0 2023-11-28T20:33:59.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.356+0000 7fb73affd640 1 -- 172.21.15.45:0/3474734286 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c00a640 con 0x7fb7441438b0 2023-11-28T20:33:59.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.356+0000 7fb73affd640 1 -- 172.21.15.45:0/3474734286 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00a9d0 con 0x7fb7441438b0 2023-11-28T20:33:59.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.356+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb744159100 con 0x7fb7441438b0 2023-11-28T20:33:59.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.356+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb744159be0 con 0x7fb7441438b0 2023-11-28T20:33:59.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.357+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7441598c0 con 0x7fb7441438b0 2023-11-28T20:33:59.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.358+0000 7fb73affd640 1 -- 172.21.15.45:0/3474734286 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb73c00ab30 con 0x7fb7441438b0 2023-11-28T20:33:59.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.359+0000 7fb73affd640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb72406ae40 0x7fb72406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.359+0000 7fb73affd640 1 -- 172.21.15.45:0/3474734286 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb73c088eb0 con 0x7fb7441438b0 2023-11-28T20:33:59.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.360+0000 7fb74259c640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb72406ae40 0x7fb72406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.361+0000 7fb74259c640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb72406ae40 0x7fb72406d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb734035000 tx=0x7fb734008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:59.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.364+0000 7fb73affd640 1 -- 172.21.15.45:0/3474734286 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb73c053bd0 con 0x7fb7441438b0 2023-11-28T20:33:59.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.596+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7440733d0 con 0x7fb7441438b0 2023-11-28T20:33:59.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.597+0000 7fb73affd640 1 -- 172.21.15.45:0/3474734286 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4915 (secure 0 0 0) 0x7fb73c018070 con 0x7fb7441438b0 2023-11-28T20:33:59.600 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:33:59.600 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:33:58.584368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-28T20:33:59.600 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 7 2023-11-28T20:33:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb72406ae40 msgr2=0x7fb72406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb72406ae40 0x7fb72406d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb734035000 tx=0x7fb734008030 comp rx=0 tx=0).stop 2023-11-28T20:33:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 msgr2=0x7fb7440ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 0x7fb7440ad350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00c780 tx=0x7fb73c00cc40 comp rx=0 tx=0).stop 2023-11-28T20:33:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 shutdown_connections 2023-11-28T20:33:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb72406ae40 0x7fb72406d300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7441438b0 0x7fb7440ad350 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7440b24e0 0x7fb744135640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 --2- 172.21.15.45:0/3474734286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7440b1170 0x7fb7441582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.600+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 >> 172.21.15.45:0/3474734286 conn(0x7fb74406a500 msgr2=0x7fb744142340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:59.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.601+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 shutdown_connections 2023-11-28T20:33:59.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.601+0000 7fb748ba9640 1 -- 172.21.15.45:0/3474734286 wait complete. 2023-11-28T20:33:59.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:33:59.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.839+0000 7f3909832640 1 Processor -- start 2023-11-28T20:33:59.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.839+0000 7f3909832640 1 -- start start 2023-11-28T20:33:59.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.841+0000 7f3909832640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f39040ae540 0x7f39040ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.841+0000 7f3909832640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f39040ce630 0x7f39040b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.841+0000 7f3909832640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390415c3c0 con 0x7f3904155780 2023-11-28T20:33:59.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.841+0000 7f3909832640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39040ceac0 con 0x7f39040ce630 2023-11-28T20:33:59.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.841+0000 7f3909832640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3904155b60 con 0x7f39040ae540 2023-11-28T20:33:59.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.842+0000 7f3909031640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f39040ce630 0x7f39040b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.842+0000 7f3909031640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f39040ce630 0x7f39040b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36898/0 (socket says 172.21.15.45:36898) 2023-11-28T20:33:59.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.842+0000 7f3909031640 1 -- 172.21.15.45:0/2027390479 learned_addr learned my addr 172.21.15.45:0/2027390479 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:59.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.842+0000 7f3903fff640 1 --2- 172.21.15.45:0/2027390479 >> v2:172.21.15.190:3300/0 conn(0x7f39040ae540 0x7f39040ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.842+0000 7f3909031640 1 -- 172.21.15.45:0/2027390479 >> v2:172.21.15.190:3300/0 conn(0x7f39040ae540 msgr2=0x7f39040ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:59.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.842+0000 7f3909031640 1 --2- 172.21.15.45:0/2027390479 >> v2:172.21.15.190:3300/0 conn(0x7f39040ae540 0x7f39040ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.842+0000 7f3909031640 1 -- 172.21.15.45:0/2027390479 >> v1:172.21.15.190:6789/0 conn(0x7f3904155780 legacy=0x7f39040abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:33:59.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.842+0000 7f3909031640 1 -- 172.21.15.45:0/2027390479 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390415ba90 con 0x7f39040ce630 2023-11-28T20:33:59.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.843+0000 7f3909031640 1 --2- 172.21.15.45:0/2027390479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 0x7f39040b20a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39040cfec0 tx=0x7f38fc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8faebff9f4fa70f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:59.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.844+0000 7f39037fe640 1 -- 172.21.15.45:0/2027390479 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc02ca80 con 0x7f39040ce630 2023-11-28T20:33:59.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.844+0000 7f39037fe640 1 -- 172.21.15.45:0/2027390479 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc02cbe0 con 0x7f39040ce630 2023-11-28T20:33:59.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.844+0000 7f39037fe640 1 -- 172.21.15.45:0/2027390479 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0343f0 con 0x7f39040ce630 2023-11-28T20:33:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.845+0000 7f3909832640 1 -- 172.21.15.45:0/2027390479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 msgr2=0x7f39040b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.845+0000 7f3909832640 1 --2- 172.21.15.45:0/2027390479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 0x7f39040b20a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39040cfec0 tx=0x7f38fc02bee0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.845+0000 7f3909832640 1 -- 172.21.15.45:0/2027390479 shutdown_connections 2023-11-28T20:33:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.845+0000 7f3909832640 1 --2- 172.21.15.45:0/2027390479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 0x7f39040b20a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.845+0000 7f3909832640 1 --2- 172.21.15.45:0/2027390479 >> v2:172.21.15.190:3300/0 conn(0x7f39040ae540 0x7f39040ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.845+0000 7f3909832640 1 -- 172.21.15.45:0/2027390479 >> 172.21.15.45:0/2027390479 conn(0x7f390406ab30 msgr2=0x7f39040b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:33:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.845+0000 7f3909832640 1 -- 172.21.15.45:0/2027390479 shutdown_connections 2023-11-28T20:33:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.845+0000 7f3909832640 1 -- 172.21.15.45:0/2027390479 wait complete. 2023-11-28T20:33:59.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.847+0000 7f3909832640 1 Processor -- start 2023-11-28T20:33:59.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.847+0000 7f3909832640 1 -- start start 2023-11-28T20:33:59.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3909832640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39040ae540 0x7f390409afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3909832640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 0x7f39041467e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3909832640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904155780 0x7f390414c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3909832640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390414b8d0 con 0x7f39040ce630 2023-11-28T20:33:59.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3909832640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904099cc0 con 0x7f3904155780 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3909832640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3904146d20 con 0x7f39040ae540 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3903fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 0x7f39041467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3903fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 0x7f39041467e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36914/0 (socket says 172.21.15.45:36914) 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3903fff640 1 -- 172.21.15.45:0/3792150997 learned_addr learned my addr 172.21.15.45:0/3792150997 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3909031640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904155780 0x7f390414c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3908830640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39040ae540 0x7f390409afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3903fff640 1 -- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39040ae540 msgr2=0x7f390409afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3903fff640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39040ae540 0x7f390409afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3903fff640 1 -- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904155780 msgr2=0x7f390414c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3903fff640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904155780 0x7f390414c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:33:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3903fff640 1 -- 172.21.15.45:0/3792150997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc009b30 con 0x7f39040ce630 2023-11-28T20:33:59.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3908830640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39040ae540 0x7f390409afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:59.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.848+0000 7f3909031640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904155780 0x7f390414c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:33:59.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.849+0000 7f3903fff640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 0x7f39041467e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38f000d800 tx=0x7f38f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:59.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.849+0000 7f39017fa640 1 -- 172.21.15.45:0/3792150997 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f000b6c0 con 0x7f39040ce630 2023-11-28T20:33:59.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.849+0000 7f39017fa640 1 -- 172.21.15.45:0/3792150997 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f000b820 con 0x7f39040ce630 2023-11-28T20:33:59.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.849+0000 7f39017fa640 1 -- 172.21.15.45:0/3792150997 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f000bbb0 con 0x7f39040ce630 2023-11-28T20:33:59.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.849+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3904099f10 con 0x7f39040ce630 2023-11-28T20:33:59.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.849+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390409a4d0 con 0x7f39040ce630 2023-11-28T20:33:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.850+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390404e920 con 0x7f39040ce630 2023-11-28T20:33:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.851+0000 7f39017fa640 1 -- 172.21.15.45:0/3792150997 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f38f000bd10 con 0x7f39040ce630 2023-11-28T20:33:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.852+0000 7f39017fa640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f38e806ae40 0x7f38e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:33:59.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.852+0000 7f39017fa640 1 -- 172.21.15.45:0/3792150997 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f38f0089f40 con 0x7f39040ce630 2023-11-28T20:33:59.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.853+0000 7f3908830640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f38e806ae40 0x7f38e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:33:59.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.854+0000 7f3908830640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f38e806ae40 0x7f38e806d300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38f800a830 tx=0x7f38f8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:33:59.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:33:59.857+0000 7f39017fa640 1 -- 172.21.15.45:0/3792150997 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38f0054c60 con 0x7f39040ce630 2023-11-28T20:34:00.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.055+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f390415e650 con 0x7f39040ce630 2023-11-28T20:34:00.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.058+0000 7f39017fa640 1 -- 172.21.15.45:0/3792150997 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7f38f0058280 con 0x7f39040ce630 2023-11-28T20:34:00.061 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:00.061 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f38e806ae40 msgr2=0x7f38e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f38e806ae40 0x7f38e806d300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38f800a830 tx=0x7f38f8008040 comp rx=0 tx=0).stop 2023-11-28T20:34:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 msgr2=0x7f39041467e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 0x7f39041467e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38f000d800 tx=0x7f38f000dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 shutdown_connections 2023-11-28T20:34:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f38e806ae40 0x7f38e806d300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904155780 0x7f390414c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39040ce630 0x7f39041467e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 --2- 172.21.15.45:0/3792150997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39040ae540 0x7f390409afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.061+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 >> 172.21.15.45:0/3792150997 conn(0x7f390406ab30 msgr2=0x7f390406cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.062+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 shutdown_connections 2023-11-28T20:34:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.062+0000 7f3909832640 1 -- 172.21.15.45:0/3792150997 wait complete. 2023-11-28T20:34:00.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:00.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.311+0000 7fee1ffff640 1 Processor -- start 2023-11-28T20:34:00.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.312+0000 7fee1ffff640 1 -- start start 2023-11-28T20:34:00.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.313+0000 7fee1ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fee20155780 0x7fee200abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.313+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee200b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.313+0000 7fee1ffff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2015c3c0 con 0x7fee200ae540 2023-11-28T20:34:00.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.313+0000 7fee1ffff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee20155b60 con 0x7fee20155780 2023-11-28T20:34:00.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.313+0000 7fee1ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2015ba90 con 0x7fee200ce800 2023-11-28T20:34:00.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.313+0000 7fee1effd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fee20155780 0x7fee200abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.313+0000 7fee1effd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fee20155780 0x7fee200abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36924/0 (socket says 172.21.15.45:36924) 2023-11-28T20:34:00.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.313+0000 7fee1effd640 1 -- 172.21.15.45:0/3036277703 learned_addr learned my addr 172.21.15.45:0/3036277703 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:00.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.313+0000 7fee1e7fc640 1 --1- 172.21.15.45:0/3036277703 >> v1:172.21.15.45:6789/0 conn(0x7fee200ae540 0x7fee200ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46336/0 (socket says 172.21.15.45:46336) 2023-11-28T20:34:00.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.314+0000 7fee1f7fe640 1 --2- 172.21.15.45:0/3036277703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee200b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.314+0000 7fee1effd640 1 -- 172.21.15.45:0/3036277703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 msgr2=0x7fee200b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:00.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.314+0000 7fee1effd640 1 --2- 172.21.15.45:0/3036277703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee200b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.314+0000 7fee1effd640 1 -- 172.21.15.45:0/3036277703 >> v1:172.21.15.45:6789/0 conn(0x7fee200ae540 legacy=0x7fee200ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.314+0000 7fee1effd640 1 -- 172.21.15.45:0/3036277703 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2015bd70 con 0x7fee20155780 2023-11-28T20:34:00.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.314+0000 7fee1f7fe640 1 --2- 172.21.15.45:0/3036277703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee200b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:00.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.315+0000 7fee1effd640 1 --2- 172.21.15.45:0/3036277703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 0x7fee200abd70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fee10009e50 tx=0x7fee1002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6fe3ee8ffb339d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:00.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.315+0000 7fee1dffb640 1 -- 172.21.15.45:0/3036277703 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1002cb00 con 0x7fee20155780 2023-11-28T20:34:00.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.315+0000 7fee1dffb640 1 -- 172.21.15.45:0/3036277703 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1002cc60 con 0x7fee20155780 2023-11-28T20:34:00.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.315+0000 7fee1dffb640 1 -- 172.21.15.45:0/3036277703 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100343f0 con 0x7fee20155780 2023-11-28T20:34:00.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.316+0000 7fee1ffff640 1 -- 172.21.15.45:0/3036277703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 msgr2=0x7fee200abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.316+0000 7fee1ffff640 1 --2- 172.21.15.45:0/3036277703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 0x7fee200abd70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fee10009e50 tx=0x7fee1002bf60 comp rx=0 tx=0).stop 2023-11-28T20:34:00.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.316+0000 7fee1ffff640 1 -- 172.21.15.45:0/3036277703 shutdown_connections 2023-11-28T20:34:00.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.316+0000 7fee1ffff640 1 --2- 172.21.15.45:0/3036277703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee200b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.316+0000 7fee1ffff640 1 --2- 172.21.15.45:0/3036277703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 0x7fee200abd70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.316+0000 7fee1ffff640 1 -- 172.21.15.45:0/3036277703 >> 172.21.15.45:0/3036277703 conn(0x7fee2006ab30 msgr2=0x7fee200b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.316+0000 7fee1ffff640 1 -- 172.21.15.45:0/3036277703 shutdown_connections 2023-11-28T20:34:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.316+0000 7fee1ffff640 1 -- 172.21.15.45:0/3036277703 wait complete. 2023-11-28T20:34:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.318+0000 7fee1ffff640 1 Processor -- start 2023-11-28T20:34:00.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.318+0000 7fee1ffff640 1 -- start start 2023-11-28T20:34:00.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee200ae540 0x7fee20159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee201578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 0x7fee200b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1ffff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2015abf0 con 0x7fee20155780 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1ffff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee20148110 con 0x7fee200ae540 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee20159890 con 0x7fee200ce800 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 0x7fee200b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 0x7fee200b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36938/0 (socket says 172.21.15.45:36938) 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1f7fe640 1 -- 172.21.15.45:0/1583236321 learned_addr learned my addr 172.21.15.45:0/1583236321 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.319+0000 7fee1effd640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee200ae540 0x7fee20159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1e7fc640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee201578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1f7fe640 1 -- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 msgr2=0x7fee201578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1f7fe640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee201578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1f7fe640 1 -- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee200ae540 msgr2=0x7fee20159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1f7fe640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee200ae540 0x7fee20159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1f7fe640 1 -- 172.21.15.45:0/1583236321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10009b30 con 0x7fee20155780 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1e7fc640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee201578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1f7fe640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 0x7fee200b2110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee1800c7f0 tx=0x7fee1800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee17fff640 1 -- 172.21.15.45:0/1583236321 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee18007820 con 0x7fee20155780 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee17fff640 1 -- 172.21.15.45:0/1583236321 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee18007980 con 0x7fee20155780 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee17fff640 1 -- 172.21.15.45:0/1583236321 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee18007d70 con 0x7fee20155780 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee20159b50 con 0x7fee20155780 2023-11-28T20:34:00.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.320+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2015a110 con 0x7fee20155780 2023-11-28T20:34:00.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.322+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2004e920 con 0x7fee20155780 2023-11-28T20:34:00.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.323+0000 7fee17fff640 1 -- 172.21.15.45:0/1583236321 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fee180113f0 con 0x7fee20155780 2023-11-28T20:34:00.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.323+0000 7fee17fff640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fee0006ae90 0x7fee0006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.324+0000 7fee17fff640 1 -- 172.21.15.45:0/1583236321 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fee18089c00 con 0x7fee20155780 2023-11-28T20:34:00.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.324+0000 7fee1effd640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fee0006ae90 0x7fee0006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.325+0000 7fee1effd640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fee0006ae90 0x7fee0006d350 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee10035000 tx=0x7fee10030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:00.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.328+0000 7fee17fff640 1 -- 172.21.15.45:0/1583236321 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee18054a30 con 0x7fee20155780 2023-11-28T20:34:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.521+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee200b3ef0 con 0x7fee20155780 2023-11-28T20:34:00.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.523+0000 7fee17fff640 1 -- 172.21.15.45:0/1583236321 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7fee18058050 con 0x7fee20155780 2023-11-28T20:34:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:00.527 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:00.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.525+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fee0006ae90 msgr2=0x7fee0006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.525+0000 7fee1ffff640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fee0006ae90 0x7fee0006d350 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee10035000 tx=0x7fee10030000 comp rx=0 tx=0).stop 2023-11-28T20:34:00.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.525+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 msgr2=0x7fee200b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.525+0000 7fee1ffff640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 0x7fee200b2110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee1800c7f0 tx=0x7fee1800ccb0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.525+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 shutdown_connections 2023-11-28T20:34:00.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.525+0000 7fee1ffff640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fee0006ae90 0x7fee0006d350 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.525+0000 7fee1ffff640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee20155780 0x7fee200b2110 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.525+0000 7fee1ffff640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee200ce800 0x7fee201578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.525+0000 7fee1ffff640 1 --2- 172.21.15.45:0/1583236321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee200ae540 0x7fee20159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.526+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 >> 172.21.15.45:0/1583236321 conn(0x7fee2006ab30 msgr2=0x7fee2015af50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:00.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.526+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 shutdown_connections 2023-11-28T20:34:00.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.526+0000 7fee1ffff640 1 -- 172.21.15.45:0/1583236321 wait complete. 2023-11-28T20:34:00.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:34:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.796+0000 7f2041090640 1 Processor -- start 2023-11-28T20:34:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.796+0000 7f2041090640 1 -- start start 2023-11-28T20:34:00.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.798+0000 7f2041090640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f203c161960 0x7f203c0b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.798+0000 7f2041090640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f203c0b4c40 0x7f203c070330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.798+0000 7f2041090640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0711c0 con 0x7f203c0b0c30 2023-11-28T20:34:00.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.798+0000 7f2041090640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f203c071e40 con 0x7f203c0b4c40 2023-11-28T20:34:00.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.798+0000 7f2041090640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c0714a0 con 0x7f203c161960 2023-11-28T20:34:00.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.798+0000 7f203259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f203c0b4c40 0x7f203c070330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.799+0000 7f203b59e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f203c0b0c30 0x7f203c165660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46348/0 (socket says 172.21.15.45:46348) 2023-11-28T20:34:00.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.799+0000 7f203259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f203c0b4c40 0x7f203c070330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36954/0 (socket says 172.21.15.45:36954) 2023-11-28T20:34:00.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.799+0000 7f203b59e640 1 -- 172.21.15.45:0/672824033 learned_addr learned my addr 172.21.15.45:0/672824033 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:00.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.799+0000 7f203ad9d640 1 --2- 172.21.15.45:0/672824033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f203c161960 0x7f203c0b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.799+0000 7f203259c640 1 -- 172.21.15.45:0/672824033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f203c161960 msgr2=0x7f203c0b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:00.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.799+0000 7f203259c640 1 --2- 172.21.15.45:0/672824033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f203c161960 0x7f203c0b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.799+0000 7f203259c640 1 -- 172.21.15.45:0/672824033 >> v1:172.21.15.45:6789/0 conn(0x7f203c0b0c30 legacy=0x7f203c165660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.799+0000 7f203259c640 1 -- 172.21.15.45:0/672824033 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c071780 con 0x7f203c0b4c40 2023-11-28T20:34:00.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.800+0000 7f203259c640 1 --2- 172.21.15.45:0/672824033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b4c40 0x7f203c070330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f20240097a0 tx=0x7f202402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa2e8c882378dcd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:00.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f203a59c640 1 -- 172.21.15.45:0/672824033 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024031780 con 0x7f203c0b4c40 2023-11-28T20:34:00.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f203a59c640 1 -- 172.21.15.45:0/672824033 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20240318e0 con 0x7f203c0b4c40 2023-11-28T20:34:00.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f203a59c640 1 -- 172.21.15.45:0/672824033 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024031c70 con 0x7f203c0b4c40 2023-11-28T20:34:00.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f2041090640 1 -- 172.21.15.45:0/672824033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b4c40 msgr2=0x7f203c070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f2041090640 1 --2- 172.21.15.45:0/672824033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b4c40 0x7f203c070330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f20240097a0 tx=0x7f202402b820 comp rx=0 tx=0).stop 2023-11-28T20:34:00.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f2041090640 1 -- 172.21.15.45:0/672824033 shutdown_connections 2023-11-28T20:34:00.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f2041090640 1 --2- 172.21.15.45:0/672824033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b4c40 0x7f203c070330 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f2041090640 1 --2- 172.21.15.45:0/672824033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f203c161960 0x7f203c0b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f2041090640 1 -- 172.21.15.45:0/672824033 >> 172.21.15.45:0/672824033 conn(0x7f203c06a500 msgr2=0x7f203c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:00.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f2041090640 1 -- 172.21.15.45:0/672824033 shutdown_connections 2023-11-28T20:34:00.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.801+0000 7f2041090640 1 -- 172.21.15.45:0/672824033 wait complete. 2023-11-28T20:34:00.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.803+0000 7f2041090640 1 Processor -- start 2023-11-28T20:34:00.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.803+0000 7f2041090640 1 -- start start 2023-11-28T20:34:00.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.804+0000 7f2041090640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b0c30 0x7f203c162460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.804+0000 7f2041090640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f203c0b4c40 0x7f203c15ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.804+0000 7f2041090640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f203c161960 0x7f203c15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.804+0000 7f2041090640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c163de0 con 0x7f203c0b0c30 2023-11-28T20:34:00.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.804+0000 7f2041090640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c162d40 con 0x7f203c161960 2023-11-28T20:34:00.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.804+0000 7f2041090640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c163020 con 0x7f203c0b4c40 2023-11-28T20:34:00.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b0c30 0x7f203c162460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b0c30 0x7f203c162460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36964/0 (socket says 172.21.15.45:36964) 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203ad9d640 1 -- 172.21.15.45:0/982473897 learned_addr learned my addr 172.21.15.45:0/982473897 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203ad9d640 1 -- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f203c0b4c40 msgr2=0x7f203c15ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203ad9d640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f203c0b4c40 0x7f203c15ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203ad9d640 1 -- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f203c161960 msgr2=0x7f203c15bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203b59e640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f203c161960 0x7f203c15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203ad9d640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f203c161960 0x7f203c15bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203ad9d640 1 -- 172.21.15.45:0/982473897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024009400 con 0x7f203c0b0c30 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.805+0000 7f203ad9d640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b0c30 0x7f203c162460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f202c00e870 tx=0x7f202c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.806+0000 7f2033fff640 1 -- 172.21.15.45:0/982473897 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0094f0 con 0x7f203c0b0c30 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.806+0000 7f2033fff640 1 -- 172.21.15.45:0/982473897 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c00c940 con 0x7f203c0b0c30 2023-11-28T20:34:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.806+0000 7f2033fff640 1 -- 172.21.15.45:0/982473897 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c00ccf0 con 0x7f203c0b0c30 2023-11-28T20:34:00.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.806+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203c163300 con 0x7f203c0b0c30 2023-11-28T20:34:00.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.806+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c1638c0 con 0x7f203c0b0c30 2023-11-28T20:34:00.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.807+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203c0cecf0 con 0x7f203c0b0c30 2023-11-28T20:34:00.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.808+0000 7f2033fff640 1 -- 172.21.15.45:0/982473897 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f202c013470 con 0x7f203c0b0c30 2023-11-28T20:34:00.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.809+0000 7f2033fff640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f201c06ae40 0x7f201c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:00.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.809+0000 7f2033fff640 1 -- 172.21.15.45:0/982473897 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f202c089a30 con 0x7f203c0b0c30 2023-11-28T20:34:00.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.809+0000 7f203259c640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f201c06ae40 0x7f201c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:00.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.810+0000 7f203259c640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f201c06ae40 0x7f201c06d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2024036000 tx=0x7f2024030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:00.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:00.813+0000 7f2033fff640 1 -- 172.21.15.45:0/982473897 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f202c0547e0 con 0x7f203c0b0c30 2023-11-28T20:34:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.044+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f203c0733d0 con 0x7f203c0b0c30 2023-11-28T20:34:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.045+0000 7f2033fff640 1 -- 172.21.15.45:0/982473897 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4915 (secure 0 0 0) 0x7f202c017090 con 0x7f203c0b0c30 2023-11-28T20:34:01.048 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:01.048 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:33:58.584368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-28T20:34:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 7 2023-11-28T20:34:01.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f201c06ae40 msgr2=0x7f201c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:01.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f201c06ae40 0x7f201c06d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2024036000 tx=0x7f2024030000 comp rx=0 tx=0).stop 2023-11-28T20:34:01.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b0c30 msgr2=0x7f203c162460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:01.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b0c30 0x7f203c162460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f202c00e870 tx=0x7f202c00ed30 comp rx=0 tx=0).stop 2023-11-28T20:34:01.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 shutdown_connections 2023-11-28T20:34:01.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f201c06ae40 0x7f201c06d300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f203c161960 0x7f203c15bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f203c0b4c40 0x7f203c15ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 --2- 172.21.15.45:0/982473897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f203c0b0c30 0x7f203c162460 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 >> 172.21.15.45:0/982473897 conn(0x7f203c06a500 msgr2=0x7f203c15b240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:01.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.048+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 shutdown_connections 2023-11-28T20:34:01.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.049+0000 7f2041090640 1 -- 172.21.15.45:0/982473897 wait complete. 2023-11-28T20:34:01.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:01.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.281+0000 7fe21b59e640 1 Processor -- start 2023-11-28T20:34:01.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.281+0000 7fe21b59e640 1 -- start start 2023-11-28T20:34:01.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.283+0000 7fe21b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe21c0b3870 0x7fe21c15db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:01.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.283+0000 7fe21b59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe21c0721e0 0x7fe21c15e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:01.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.283+0000 7fe21b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 0x7fe21c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:01.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.283+0000 7fe21b59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c15a460 con 0x7fe21c0721e0 2023-11-28T20:34:01.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.283+0000 7fe21b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c06db40 con 0x7fe21c0b3870 2023-11-28T20:34:01.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.283+0000 7fe21b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c15e680 con 0x7fe21c06d170 2023-11-28T20:34:01.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe219d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe21c0721e0 0x7fe21c15e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:01.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe219d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe21c0721e0 0x7fe21c15e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36966/0 (socket says 172.21.15.45:36966) 2023-11-28T20:34:01.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe219d9b640 1 -- 172.21.15.45:0/4070165508 learned_addr learned my addr 172.21.15.45:0/4070165508 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:01.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe21ad9d640 1 --2- 172.21.15.45:0/4070165508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 0x7fe21c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:01.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe21a59c640 1 --2- 172.21.15.45:0/4070165508 >> v2:172.21.15.190:3300/0 conn(0x7fe21c0b3870 0x7fe21c15db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:01.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe219d9b640 1 -- 172.21.15.45:0/4070165508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 msgr2=0x7fe21c06d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:01.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe219d9b640 1 --2- 172.21.15.45:0/4070165508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 0x7fe21c06d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe219d9b640 1 -- 172.21.15.45:0/4070165508 >> v2:172.21.15.190:3300/0 conn(0x7fe21c0b3870 msgr2=0x7fe21c15db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:01.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe219d9b640 1 --2- 172.21.15.45:0/4070165508 >> v2:172.21.15.190:3300/0 conn(0x7fe21c0b3870 0x7fe21c15db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.284+0000 7fe219d9b640 1 -- 172.21.15.45:0/4070165508 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c15e960 con 0x7fe21c0721e0 2023-11-28T20:34:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.286+0000 7fe219d9b640 1 --2- 172.21.15.45:0/4070165508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 0x7fe21c15e050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe21c06df50 tx=0x7fe21002db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec472aa8b79253e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.286+0000 7fe21959a640 1 -- 172.21.15.45:0/4070165508 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21002f8d0 con 0x7fe21c0721e0 2023-11-28T20:34:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.286+0000 7fe21959a640 1 -- 172.21.15.45:0/4070165508 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21002fa30 con 0x7fe21c0721e0 2023-11-28T20:34:01.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.286+0000 7fe21959a640 1 -- 172.21.15.45:0/4070165508 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21002fdc0 con 0x7fe21c0721e0 2023-11-28T20:34:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.287+0000 7fe21b59e640 1 -- 172.21.15.45:0/4070165508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 msgr2=0x7fe21c15e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.287+0000 7fe21b59e640 1 --2- 172.21.15.45:0/4070165508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 0x7fe21c15e050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe21c06df50 tx=0x7fe21002db10 comp rx=0 tx=0).stop 2023-11-28T20:34:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.287+0000 7fe21b59e640 1 -- 172.21.15.45:0/4070165508 shutdown_connections 2023-11-28T20:34:01.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.287+0000 7fe21b59e640 1 --2- 172.21.15.45:0/4070165508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 0x7fe21c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.287+0000 7fe21b59e640 1 --2- 172.21.15.45:0/4070165508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 0x7fe21c15e050 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.287+0000 7fe21b59e640 1 --2- 172.21.15.45:0/4070165508 >> v2:172.21.15.190:3300/0 conn(0x7fe21c0b3870 0x7fe21c15db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.287+0000 7fe21b59e640 1 -- 172.21.15.45:0/4070165508 >> 172.21.15.45:0/4070165508 conn(0x7fe21c06ab30 msgr2=0x7fe21c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:01.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.287+0000 7fe21b59e640 1 -- 172.21.15.45:0/4070165508 shutdown_connections 2023-11-28T20:34:01.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.287+0000 7fe21b59e640 1 -- 172.21.15.45:0/4070165508 wait complete. 2023-11-28T20:34:01.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.289+0000 7fe21b59e640 1 Processor -- start 2023-11-28T20:34:01.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.289+0000 7fe21b59e640 1 -- start start 2023-11-28T20:34:01.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.290+0000 7fe21b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 0x7fe21c099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:01.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.290+0000 7fe21b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 0x7fe21c0b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:01.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.290+0000 7fe21b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe21c0b3870 0x7fe21c14e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:01.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.290+0000 7fe21b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b0c60 con 0x7fe21c0721e0 2023-11-28T20:34:01.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.290+0000 7fe21b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c09b1c0 con 0x7fe21c0b3870 2023-11-28T20:34:01.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.290+0000 7fe21b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c14ee10 con 0x7fe21c06d170 2023-11-28T20:34:01.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.290+0000 7fe219d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 0x7fe21c0b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:01.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe21a59c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 0x7fe21c099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:01.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe219d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 0x7fe21c0b0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36968/0 (socket says 172.21.15.45:36968) 2023-11-28T20:34:01.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe219d9b640 1 -- 172.21.15.45:0/2178855196 learned_addr learned my addr 172.21.15.45:0/2178855196 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:01.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe21ad9d640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe21c0b3870 0x7fe21c14e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:01.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe219d9b640 1 -- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 msgr2=0x7fe21c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:01.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe219d9b640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 0x7fe21c099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe219d9b640 1 -- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe21c0b3870 msgr2=0x7fe21c14e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:01.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe219d9b640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe21c0b3870 0x7fe21c14e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe219d9b640 1 -- 172.21.15.45:0/2178855196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2100094d0 con 0x7fe21c0721e0 2023-11-28T20:34:01.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.291+0000 7fe219d9b640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 0x7fe21c0b0720 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe21002db50 tx=0x7fe2100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:01.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.292+0000 7fe20affd640 1 -- 172.21.15.45:0/2178855196 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210037c60 con 0x7fe21c0721e0 2023-11-28T20:34:01.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.292+0000 7fe20affd640 1 -- 172.21.15.45:0/2178855196 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe210037dc0 con 0x7fe21c0721e0 2023-11-28T20:34:01.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.292+0000 7fe20affd640 1 -- 172.21.15.45:0/2178855196 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2100426b0 con 0x7fe21c0721e0 2023-11-28T20:34:01.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.292+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c09a120 con 0x7fe21c0721e0 2023-11-28T20:34:01.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.292+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c09a6e0 con 0x7fe21c0721e0 2023-11-28T20:34:01.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.293+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c09d0f0 con 0x7fe21c0721e0 2023-11-28T20:34:01.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.294+0000 7fe20affd640 1 -- 172.21.15.45:0/2178855196 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe210042810 con 0x7fe21c0721e0 2023-11-28T20:34:01.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.295+0000 7fe20affd640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1fc06ae90 0x7fe1fc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:01.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.295+0000 7fe20affd640 1 -- 172.21.15.45:0/2178855196 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe210033070 con 0x7fe21c0721e0 2023-11-28T20:34:01.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.297+0000 7fe21a59c640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1fc06ae90 0x7fe1fc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:01.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.298+0000 7fe21a59c640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1fc06ae90 0x7fe1fc06d350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe20c006fd0 tx=0x7fe20c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:01.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.300+0000 7fe20affd640 1 -- 172.21.15.45:0/2178855196 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe210077600 con 0x7fe21c0721e0 2023-11-28T20:34:01.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.496+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe21c159ef0 con 0x7fe21c0721e0 2023-11-28T20:34:01.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.498+0000 7fe20affd640 1 -- 172.21.15.45:0/2178855196 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7fe21007ac20 con 0x7fe21c0721e0 2023-11-28T20:34:01.502 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:01.502 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:01.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1fc06ae90 msgr2=0x7fe1fc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:01.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1fc06ae90 0x7fe1fc06d350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe20c006fd0 tx=0x7fe20c008040 comp rx=0 tx=0).stop 2023-11-28T20:34:01.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 msgr2=0x7fe21c0b0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:01.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 0x7fe21c0b0720 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe21002db50 tx=0x7fe2100046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 shutdown_connections 2023-11-28T20:34:01.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1fc06ae90 0x7fe1fc06d350 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe21c0b3870 0x7fe21c14e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe21c0721e0 0x7fe21c0b0720 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 --2- 172.21.15.45:0/2178855196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe21c06d170 0x7fe21c099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:01.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 >> 172.21.15.45:0/2178855196 conn(0x7fe21c06ab30 msgr2=0x7fe21c15c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:01.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 shutdown_connections 2023-11-28T20:34:01.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:01.501+0000 7fe21b59e640 1 -- 172.21.15.45:0/2178855196 wait complete. 2023-11-28T20:34:01.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:33:53.520990+0000', 'modified': '2023-11-28T20:33:58.584368+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/2066232376', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6834', 'nonce': 2066232376}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 2066232376}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-11-28T20:34:01.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:34:01.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T20:34:02.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.162+0000 7f2a747b6640 1 Processor -- start 2023-11-28T20:34:02.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.162+0000 7f2a747b6640 1 -- start start 2023-11-28T20:34:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.164+0000 7f2a747b6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a6c15f4d0 0x7f2a6c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.164+0000 7f2a747b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6c15fa80 con 0x7f2a6c15b820 2023-11-28T20:34:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.164+0000 7f2a747b6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6c06cf40 con 0x7f2a6c0cf730 2023-11-28T20:34:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.164+0000 7f2a747b6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a6c15fd60 con 0x7f2a6c15f4d0 2023-11-28T20:34:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.164+0000 7f2a72552640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2a6c15b820 0x7f2a6c0cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46356/0 (socket says 172.21.15.45:46356) 2023-11-28T20:34:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.164+0000 7f2a72552640 1 -- 172.21.15.45:0/2008634523 learned_addr learned my addr 172.21.15.45:0/2008634523 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.164+0000 7f2a72d53640 1 --2- 172.21.15.45:0/2008634523 >> v2:172.21.15.45:3300/0 conn(0x7f2a6c15f4d0 0x7f2a6c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.165+0000 7f2a71550640 1 -- 172.21.15.45:0/2008634523 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212551156 0 0) 0x7f2a6c15fa80 con 0x7f2a6c15b820 2023-11-28T20:34:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.165+0000 7f2a72d53640 1 -- 172.21.15.45:0/2008634523 >> v1:172.21.15.190:6789/0 conn(0x7f2a6c0cf730 legacy=0x7f2a6c15edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.165+0000 7f2a72d53640 1 -- 172.21.15.45:0/2008634523 >> v1:172.21.15.45:6789/0 conn(0x7f2a6c15b820 legacy=0x7f2a6c0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.165+0000 7f2a72d53640 1 -- 172.21.15.45:0/2008634523 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6c06d220 con 0x7f2a6c15f4d0 2023-11-28T20:34:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.166+0000 7f2a72d53640 1 --2- 172.21.15.45:0/2008634523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15f4d0 0x7f2a6c06da20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009d90 tx=0x7f2a6402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a980929fca58e42 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.166+0000 7f2a71550640 1 -- 172.21.15.45:0/2008634523 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6402cc60 con 0x7f2a6c15f4d0 2023-11-28T20:34:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.166+0000 7f2a71550640 1 -- 172.21.15.45:0/2008634523 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6402cdc0 con 0x7f2a6c15f4d0 2023-11-28T20:34:02.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.167+0000 7f2a71550640 1 -- 172.21.15.45:0/2008634523 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64034660 con 0x7f2a6c15f4d0 2023-11-28T20:34:02.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.167+0000 7f2a747b6640 1 -- 172.21.15.45:0/2008634523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15f4d0 msgr2=0x7f2a6c06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.167+0000 7f2a747b6640 1 --2- 172.21.15.45:0/2008634523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15f4d0 0x7f2a6c06da20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009d90 tx=0x7f2a6402beb0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.167+0000 7f2a747b6640 1 -- 172.21.15.45:0/2008634523 shutdown_connections 2023-11-28T20:34:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.167+0000 7f2a747b6640 1 --2- 172.21.15.45:0/2008634523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15f4d0 0x7f2a6c06da20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.167+0000 7f2a747b6640 1 -- 172.21.15.45:0/2008634523 >> 172.21.15.45:0/2008634523 conn(0x7f2a6c06a6e0 msgr2=0x7f2a6c06aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.167+0000 7f2a747b6640 1 -- 172.21.15.45:0/2008634523 shutdown_connections 2023-11-28T20:34:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.167+0000 7f2a747b6640 1 -- 172.21.15.45:0/2008634523 wait complete. 2023-11-28T20:34:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.169+0000 7f2a747b6640 1 Processor -- start 2023-11-28T20:34:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.169+0000 7f2a747b6640 1 -- start start 2023-11-28T20:34:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.170+0000 7f2a747b6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a6c0cf730 0x7f2a6c159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.170+0000 7f2a747b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15b820 0x7f2a6c14fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.170+0000 7f2a747b6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a6c15f4d0 0x7f2a6c0b3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.170+0000 7f2a747b6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c15a3d0 con 0x7f2a6c15b820 2023-11-28T20:34:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.170+0000 7f2a747b6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c09a390 con 0x7f2a6c0cf730 2023-11-28T20:34:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.170+0000 7f2a747b6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c0b3970 con 0x7f2a6c15f4d0 2023-11-28T20:34:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.170+0000 7f2a71d51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15b820 0x7f2a6c14fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.170+0000 7f2a71d51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15b820 0x7f2a6c14fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36994/0 (socket says 172.21.15.45:36994) 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.170+0000 7f2a71d51640 1 -- 172.21.15.45:0/3413214911 learned_addr learned my addr 172.21.15.45:0/3413214911 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.171+0000 7f2a72552640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a6c0cf730 0x7f2a6c159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.171+0000 7f2a72d53640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a6c15f4d0 0x7f2a6c0b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.171+0000 7f2a71d51640 1 -- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a6c15f4d0 msgr2=0x7f2a6c0b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.171+0000 7f2a71d51640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a6c15f4d0 0x7f2a6c0b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.171+0000 7f2a71d51640 1 -- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a6c0cf730 msgr2=0x7f2a6c159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.171+0000 7f2a71d51640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a6c0cf730 0x7f2a6c159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.171+0000 7f2a71d51640 1 -- 172.21.15.45:0/3413214911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a64009a70 con 0x7f2a6c15b820 2023-11-28T20:34:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.171+0000 7f2a71d51640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15b820 0x7f2a6c14fb20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00c430 tx=0x7f2a5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.172+0000 7f2a62ffd640 1 -- 172.21.15.45:0/3413214911 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c011790 con 0x7f2a6c15b820 2023-11-28T20:34:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.172+0000 7f2a62ffd640 1 -- 172.21.15.45:0/3413214911 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5c0118f0 con 0x7f2a6c15b820 2023-11-28T20:34:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.172+0000 7f2a62ffd640 1 -- 172.21.15.45:0/3413214911 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c011ce0 con 0x7f2a6c15b820 2023-11-28T20:34:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.172+0000 7f2a747b6640 1 -- 172.21.15.45:0/3413214911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6c159b30 con 0x7f2a6c15b820 2023-11-28T20:34:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.172+0000 7f2a747b6640 1 -- 172.21.15.45:0/3413214911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6c15a0f0 con 0x7f2a6c15b820 2023-11-28T20:34:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.174+0000 7f2a62ffd640 1 -- 172.21.15.45:0/3413214911 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2a5c01c460 con 0x7f2a6c15b820 2023-11-28T20:34:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.175+0000 7f2a62ffd640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a5006ae40 0x7f2a5006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.175+0000 7f2a62ffd640 1 -- 172.21.15.45:0/3413214911 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2a5c089850 con 0x7f2a6c15b820 2023-11-28T20:34:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.176+0000 7f2a72552640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a5006ae40 0x7f2a5006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.176+0000 7f2a77421740 1 Processor -- start 2023-11-28T20:34:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.176+0000 7f2a77421740 1 -- start start 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a72552640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a5006ae40 0x7f2a5006d300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2a68000f60 tx=0x7f2a68007b50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a77421740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a77421740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a77421740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x563623415560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a77421740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563623330880 con 0x5636233d1ea0 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a77421740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362333fae0 con 0x5636233d15e0 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a77421740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x56362333d1d0 con 0x5636233ccba0 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a72552640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a72552640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233d2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37004/0 (socket says 172.21.15.45:37004) 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a72552640 1 -- 172.21.15.45:0/1119127518 learned_addr learned my addr 172.21.15.45:0/1119127518 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a71d51640 1 --2- 172.21.15.45:0/1119127518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.177+0000 7f2a72d53640 1 --2- 172.21.15.45:0/1119127518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x563623415560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.178+0000 7f2a72552640 1 -- 172.21.15.45:0/1119127518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 msgr2=0x5636233d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.178+0000 7f2a72552640 1 --2- 172.21.15.45:0/1119127518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.178+0000 7f2a72552640 1 -- 172.21.15.45:0/1119127518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 msgr2=0x563623415560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.178+0000 7f2a72552640 1 --2- 172.21.15.45:0/1119127518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x563623415560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.178+0000 7f2a72552640 1 -- 172.21.15.45:0/1119127518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563623300860 con 0x5636233d1ea0 2023-11-28T20:34:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.178+0000 7f2a72552640 1 --2- 172.21.15.45:0/1119127518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233d2280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2a68005de0 tx=0x7f2a68012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b6063ec3d26c38 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.178+0000 7f2a4ffff640 1 -- 172.21.15.45:0/1119127518 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68005e10 con 0x5636233d1ea0 2023-11-28T20:34:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.178+0000 7f2a4ffff640 1 -- 172.21.15.45:0/1119127518 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a680071a0 con 0x5636233d1ea0 2023-11-28T20:34:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.178+0000 7f2a4ffff640 1 -- 172.21.15.45:0/1119127518 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68015bc0 con 0x5636233d1ea0 2023-11-28T20:34:02.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.179+0000 7f2a77421740 1 -- 172.21.15.45:0/1119127518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 msgr2=0x5636233d2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.179+0000 7f2a77421740 1 --2- 172.21.15.45:0/1119127518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233d2280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2a68005de0 tx=0x7f2a68012000 comp rx=0 tx=0).stop 2023-11-28T20:34:02.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.179+0000 7f2a77421740 1 -- 172.21.15.45:0/1119127518 shutdown_connections 2023-11-28T20:34:02.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.179+0000 7f2a77421740 1 --2- 172.21.15.45:0/1119127518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x563623415560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.179+0000 7f2a77421740 1 --2- 172.21.15.45:0/1119127518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.179+0000 7f2a77421740 1 --2- 172.21.15.45:0/1119127518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233d2280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.179+0000 7f2a77421740 1 -- 172.21.15.45:0/1119127518 >> 172.21.15.45:0/1119127518 conn(0x563623393c70 msgr2=0x563623417990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.179+0000 7f2a77421740 1 -- 172.21.15.45:0/1119127518 shutdown_connections 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.179+0000 7f2a77421740 1 -- 172.21.15.45:0/1119127518 wait complete. 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.180+0000 7f2a77421740 1 -- start start 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.180+0000 7f2a77421740 1 Processor -- start 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a77421740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233ecb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a77421740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x5636233f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a77421740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233f4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a77421740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563623338ac0 con 0x5636233d1ea0 2023-11-28T20:34:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a77421740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636232d52f0 con 0x5636233d15e0 2023-11-28T20:34:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a77421740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636232f2c70 con 0x5636233ccba0 2023-11-28T20:34:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a77421740 10 client.0 _ll_register_callbacks cb 0x5636232f73b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:34:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a72d53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a72d53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37016/0 (socket says 172.21.15.45:37016) 2023-11-28T20:34:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.181+0000 7f2a72d53640 1 -- 172.21.15.45:0/1130608477 learned_addr learned my addr 172.21.15.45:0/1130608477 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a71d51640 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x5636233f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a72552640 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233ecb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a72d53640 1 -- 172.21.15.45:0/1130608477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 msgr2=0x5636233ecb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a72d53640 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233ecb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a72d53640 1 -- 172.21.15.45:0/1130608477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 msgr2=0x5636233f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a72d53640 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x5636233f3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a72d53640 1 -- 172.21.15.45:0/1130608477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6800f040 con 0x5636233d1ea0 2023-11-28T20:34:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a72d53640 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233f4470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009eb0 tx=0x7f2a64004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918273e65106d4ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:34:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1130608477 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64034e00 con 0x5636233d1ea0 2023-11-28T20:34:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1130608477 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2a6404be20 con 0x5636233d1ea0 2023-11-28T20:34:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.182+0000 7f2a77421740 1 -- 172.21.15.45:0/1130608477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563623331aa0 con 0x5636233d1ea0 2023-11-28T20:34:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1130608477 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a64041410 con 0x5636233d1ea0 2023-11-28T20:34:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a77421740 10 client.4398 fetch_fsmap learned FSMap version 7 2023-11-28T20:34:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a77421740 1 -- 172.21.15.45:0/1130608477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x563623337140 con 0x5636233d1ea0 2023-11-28T20:34:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1130608477 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563623331aa0 con 0x5636233d1ea0 2023-11-28T20:34:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1130608477 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f2a64003990 con 0x5636233d1ea0 2023-11-28T20:34:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a77421740 10 client.4398 fetch_fsmap finished waiting for FSMap version 7 2023-11-28T20:34:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a77421740 10 client.4398 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:34:02.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a77421740 20 client.4398 populate_metadata read hostname 'smithi045' 2023-11-28T20:34:02.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a77421740 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x5636233f2770 0x563623434db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a77421740 4 client.4398 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:34:02.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a77421740 1 -- 172.21.15.45:0/1130608477 --> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636233317c0 con 0x5636233f2770 2023-11-28T20:34:02.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.183+0000 7f2a72552640 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x5636233f2770 0x563623434db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.184+0000 7f2a72552640 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x5636233f2770 0x563623434db0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a723180cc57611be server_cookie=a3c124a3ab96f6ec in_seq=0 out_seq=0 2023-11-28T20:34:02.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.184+0000 7f2a4d7fa640 10 client.4398 ms_handle_connect on v2:172.21.15.45:6834/2066232376 2023-11-28T20:34:02.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1130608477 <== mds.0 v2:172.21.15.45:6834/2066232376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5636233317c0 con 0x5636233f2770 2023-11-28T20:34:02.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a4d7fa640 10 client.4398 handle_command_reply: tid=0 2023-11-28T20:34:02.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a77421740 1 client.4398 shutdown 2023-11-28T20:34:02.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a77421740 1 -- 172.21.15.45:0/1130608477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 msgr2=0x5636233f4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a77421740 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233f4470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009eb0 tx=0x7f2a64004ab0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a77421740 1 -- 172.21.15.45:0/1130608477 shutdown_connections 2023-11-28T20:34:02.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a77421740 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x5636233f2770 0x563623434db0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a77421740 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x5636233f4470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a77421740 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x5636233f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.185+0000 7f2a77421740 1 --2- 172.21.15.45:0/1130608477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233ecb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.186+0000 7f2a77421740 1 -- 172.21.15.45:0/1130608477 >> 172.21.15.45:0/1130608477 conn(0x563623393c70 msgr2=0x5636233cbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:02.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.186+0000 7f2a4d7fa640 0 client.4398 ms_handle_reset on v2:172.21.15.45:6834/2066232376 2023-11-28T20:34:02.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.186+0000 7f2a77421740 1 -- 172.21.15.45:0/1130608477 shutdown_connections 2023-11-28T20:34:02.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.186+0000 7f2a77421740 1 -- 172.21.15.45:0/1130608477 wait complete. 2023-11-28T20:34:02.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.186+0000 7f2a77421740 20 client.4398 trim_cache size 0 max 16384 2023-11-28T20:34:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.203+0000 7f2a77421740 1 Processor -- start 2023-11-28T20:34:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.203+0000 7f2a77421740 1 -- start start 2023-11-28T20:34:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.204+0000 7f2a77421740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233c8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.204+0000 7f2a77421740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x5636233d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.204+0000 7f2a77421740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x563623414720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.204+0000 7f2a77421740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362309bf20 con 0x5636233d1ea0 2023-11-28T20:34:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.204+0000 7f2a77421740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636233f7ca0 con 0x5636233d15e0 2023-11-28T20:34:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.204+0000 7f2a77421740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636233c89c0 con 0x5636233ccba0 2023-11-28T20:34:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72d53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x563623414720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72d53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x563623414720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37030/0 (socket says 172.21.15.45:37030) 2023-11-28T20:34:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72d53640 1 -- 172.21.15.45:0/442254894 learned_addr learned my addr 172.21.15.45:0/442254894 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72552640 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a71d51640 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x5636233d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72d53640 1 -- 172.21.15.45:0/442254894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 msgr2=0x5636233c8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72d53640 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233c8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72d53640 1 -- 172.21.15.45:0/442254894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 msgr2=0x5636233d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72d53640 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x5636233d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72d53640 1 -- 172.21.15.45:0/442254894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636232d4b80 con 0x5636233d1ea0 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.205+0000 7f2a72552640 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233c8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a72d53640 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x563623414720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2a64034fd0 tx=0x7f2a64043070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f826fd01af74bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a60ff9640 1 -- 172.21.15.45:0/442254894 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64034ae0 con 0x5636233d1ea0 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a60ff9640 1 -- 172.21.15.45:0/442254894 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a64045070 con 0x5636233d1ea0 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a60ff9640 1 -- 172.21.15.45:0/442254894 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6404a480 con 0x5636233d1ea0 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a77421740 1 -- 172.21.15.45:0/442254894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 msgr2=0x563623414720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a77421740 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x563623414720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2a64034fd0 tx=0x7f2a64043070 comp rx=0 tx=0).stop 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a77421740 1 -- 172.21.15.45:0/442254894 shutdown_connections 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a77421740 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233d1ea0 0x563623414720 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a77421740 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d15e0 0x5636233d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a77421740 1 --2- 172.21.15.45:0/442254894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233ccba0 0x5636233c8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.206+0000 7f2a77421740 1 -- 172.21.15.45:0/442254894 >> 172.21.15.45:0/442254894 conn(0x563623393c70 msgr2=0x5636233f3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.207+0000 7f2a77421740 1 -- 172.21.15.45:0/442254894 shutdown_connections 2023-11-28T20:34:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.207+0000 7f2a77421740 1 -- 172.21.15.45:0/442254894 wait complete. 2023-11-28T20:34:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.207+0000 7f2a77421740 1 -- start start 2023-11-28T20:34:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.207+0000 7f2a77421740 1 Processor -- start 2023-11-28T20:34:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.208+0000 7f2a77421740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233ccba0 0x5636233f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.208+0000 7f2a77421740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233d15e0 0x5636233f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a77421740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d1ea0 0x563623413590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a77421740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563623438f60 con 0x5636233ccba0 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a77421740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563623439a90 con 0x5636233d1ea0 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a77421740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636234392b0 con 0x5636233d15e0 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a77421740 10 client.0 _ll_register_callbacks cb 0x7f2a64037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72552640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233ccba0 0x5636233f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72552640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233ccba0 0x5636233f3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37040/0 (socket says 172.21.15.45:37040) 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72552640 1 -- 172.21.15.45:0/1922370802 learned_addr learned my addr 172.21.15.45:0/1922370802 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a71d51640 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233d15e0 0x5636233f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72552640 1 -- 172.21.15.45:0/1922370802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233d15e0 msgr2=0x5636233f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72d53640 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d1ea0 0x563623413590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72552640 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233d15e0 0x5636233f43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72552640 1 -- 172.21.15.45:0/1922370802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d1ea0 msgr2=0x563623413590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72552640 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d1ea0 0x563623413590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72552640 1 -- 172.21.15.45:0/1922370802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563623337140 con 0x5636233ccba0 2023-11-28T20:34:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a72552640 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233ccba0 0x5636233f3eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2a68008fd0 tx=0x7f2a68000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e47a785cb7cdfd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.209+0000 7f2a4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:34:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1922370802 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68007310 con 0x5636233ccba0 2023-11-28T20:34:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1922370802 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2a68015bd0 con 0x5636233ccba0 2023-11-28T20:34:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a77421740 1 -- 172.21.15.45:0/1922370802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a680071a0 con 0x5636233ccba0 2023-11-28T20:34:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1922370802 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a68015d90 con 0x5636233ccba0 2023-11-28T20:34:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a77421740 10 client.4404 fetch_fsmap learned FSMap version 7 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a77421740 1 -- 172.21.15.45:0/1922370802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5636232fa170 con 0x5636233ccba0 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1922370802 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a680071a0 con 0x5636233ccba0 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1922370802 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f2a6802e440 con 0x5636233ccba0 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a77421740 10 client.4404 fetch_fsmap finished waiting for FSMap version 7 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a77421740 10 client.4404 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a77421740 20 client.4404 populate_metadata read hostname 'smithi045' 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a77421740 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x563623413ad0 0x5636233e95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a77421740 4 client.4404 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.210+0000 7f2a77421740 1 -- 172.21.15.45:0/1922370802 --> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5636230396c0 con 0x563623413ad0 2023-11-28T20:34:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.211+0000 7f2a71d51640 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x563623413ad0 0x5636233e95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.211+0000 7f2a71d51640 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x563623413ad0 0x5636233e95b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d737a6d73079ee0b server_cookie=92d0f2b8dc9d369e in_seq=0 out_seq=0 2023-11-28T20:34:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.211+0000 7f2a4d7fa640 10 client.4404 ms_handle_connect on v2:172.21.15.45:6834/2066232376 2023-11-28T20:34:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.211+0000 7f2a4d7fa640 1 -- 172.21.15.45:0/1922370802 <== mds.0 v2:172.21.15.45:6834/2066232376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x5636230396c0 con 0x563623413ad0 2023-11-28T20:34:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.211+0000 7f2a4d7fa640 10 client.4404 handle_command_reply: tid=0 2023-11-28T20:34:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.211+0000 7f2a77421740 1 client.4404 shutdown 2023-11-28T20:34:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 -- 172.21.15.45:0/1922370802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233ccba0 msgr2=0x5636233f3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233ccba0 0x5636233f3eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2a68008fd0 tx=0x7f2a68000ed0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 -- 172.21.15.45:0/1922370802 shutdown_connections 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x563623413ad0 0x5636233e95b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636233d1ea0 0x563623413590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5636233d15e0 0x5636233f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 --2- 172.21.15.45:0/1922370802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636233ccba0 0x5636233f3eb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a4d7fa640 0 client.4404 ms_handle_reset on v2:172.21.15.45:6834/2066232376 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 -- 172.21.15.45:0/1922370802 >> 172.21.15.45:0/1922370802 conn(0x563623393c70 msgr2=0x5636233d2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 -- 172.21.15.45:0/1922370802 shutdown_connections 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 1 -- 172.21.15.45:0/1922370802 wait complete. 2023-11-28T20:34:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.212+0000 7f2a77421740 20 client.4404 trim_cache size 0 max 16384 2023-11-28T20:34:02.220 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:34:02.220 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:34:02.220 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:34:02.220 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 4257, 2023-11-28T20:34:02.220 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:34:02.220 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:34:02.220 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:34:02.221 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 8.1578217950000003, 2023-11-28T20:34:02.221 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 7, 2023-11-28T20:34:02.221 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 25, 2023-11-28T20:34:02.221 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 25, 2023-11-28T20:34:02.221 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 15.878886037999999 2023-11-28T20:34:02.221 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:34:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 -- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a5006ae40 msgr2=0x7f2a5006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a5006ae40 0x7f2a5006d300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2a68000f60 tx=0x7f2a68007b50 comp rx=0 tx=0).stop 2023-11-28T20:34:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 -- 172.21.15.45:0/3413214911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15b820 msgr2=0x7f2a6c14fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15b820 0x7f2a6c14fb20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00c430 tx=0x7f2a5c00c8f0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 -- 172.21.15.45:0/3413214911 shutdown_connections 2023-11-28T20:34:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a5006ae40 0x7f2a5006d300 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a6c15f4d0 0x7f2a6c0b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c15b820 0x7f2a6c14fb20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 --2- 172.21.15.45:0/3413214911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a6c0cf730 0x7f2a6c159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 -- 172.21.15.45:0/3413214911 >> 172.21.15.45:0/3413214911 conn(0x7f2a6c06a6e0 msgr2=0x7f2a6c15e230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 -- 172.21.15.45:0/3413214911 shutdown_connections 2023-11-28T20:34:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.213+0000 7f2a747b6640 1 -- 172.21.15.45:0/3413214911 wait complete. 2023-11-28T20:34:02.231 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T20:34:02.231 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-28T20:34:02.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:34:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.429+0000 7f7d58339640 1 Processor -- start 2023-11-28T20:34:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.429+0000 7f7d58339640 1 -- start start 2023-11-28T20:34:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.431+0000 7f7d58339640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7d500b0c30 0x7f7d500b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.431+0000 7f7d58339640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7d50071190 0x7f7d50071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.431+0000 7f7d58339640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d500ce710 con 0x7f7d5015d320 2023-11-28T20:34:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.431+0000 7f7d58339640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d500ae5a0 con 0x7f7d500b0c30 2023-11-28T20:34:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.431+0000 7f7d58339640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d50160fe0 con 0x7f7d50071190 2023-11-28T20:34:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.432+0000 7f7d558d4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7d500b0c30 0x7f7d500b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.432+0000 7f7d558d4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7d500b0c30 0x7f7d500b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37046/0 (socket says 172.21.15.45:37046) 2023-11-28T20:34:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.432+0000 7f7d558d4640 1 -- 172.21.15.45:0/319972039 learned_addr learned my addr 172.21.15.45:0/319972039 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.432+0000 7f7d560d5640 1 --1- 172.21.15.45:0/319972039 >> v1:172.21.15.45:6789/0 conn(0x7f7d5015d320 0x7f7d500ad800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46372/0 (socket says 172.21.15.45:46372) 2023-11-28T20:34:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.432+0000 7f7d568d6640 1 --2- 172.21.15.45:0/319972039 >> v2:172.21.15.190:3300/0 conn(0x7f7d50071190 0x7f7d50071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.432+0000 7f7d558d4640 1 -- 172.21.15.45:0/319972039 >> v2:172.21.15.190:3300/0 conn(0x7f7d50071190 msgr2=0x7f7d50071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.432+0000 7f7d558d4640 1 --2- 172.21.15.45:0/319972039 >> v2:172.21.15.190:3300/0 conn(0x7f7d50071190 0x7f7d50071600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.432+0000 7f7d558d4640 1 -- 172.21.15.45:0/319972039 >> v1:172.21.15.45:6789/0 conn(0x7f7d5015d320 legacy=0x7f7d500ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.432+0000 7f7d558d4640 1 -- 172.21.15.45:0/319972039 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d500ae2c0 con 0x7f7d500b0c30 2023-11-28T20:34:02.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.433+0000 7f7d558d4640 1 --2- 172.21.15.45:0/319972039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500b0c30 0x7f7d500b10b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d500b1d40 tx=0x7f7d4002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16499022162f351 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.434+0000 7f7d550d3640 1 -- 172.21.15.45:0/319972039 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40030860 con 0x7f7d500b0c30 2023-11-28T20:34:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.434+0000 7f7d550d3640 1 -- 172.21.15.45:0/319972039 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d400309c0 con 0x7f7d500b0c30 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.434+0000 7f7d550d3640 1 -- 172.21.15.45:0/319972039 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40030d50 con 0x7f7d500b0c30 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.434+0000 7f7d58339640 1 -- 172.21.15.45:0/319972039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500b0c30 msgr2=0x7f7d500b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.434+0000 7f7d58339640 1 --2- 172.21.15.45:0/319972039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500b0c30 0x7f7d500b10b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d500b1d40 tx=0x7f7d4002b9a0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.434+0000 7f7d58339640 1 -- 172.21.15.45:0/319972039 shutdown_connections 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.434+0000 7f7d58339640 1 --2- 172.21.15.45:0/319972039 >> v2:172.21.15.190:3300/0 conn(0x7f7d50071190 0x7f7d50071600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.434+0000 7f7d58339640 1 --2- 172.21.15.45:0/319972039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d500b0c30 0x7f7d500b10b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.434+0000 7f7d58339640 1 -- 172.21.15.45:0/319972039 >> 172.21.15.45:0/319972039 conn(0x7f7d5006a500 msgr2=0x7f7d5006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.435+0000 7f7d58339640 1 -- 172.21.15.45:0/319972039 shutdown_connections 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.435+0000 7f7d58339640 1 -- 172.21.15.45:0/319972039 wait complete. 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.436+0000 7f7d58339640 1 Processor -- start 2023-11-28T20:34:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.437+0000 7f7d58339640 1 -- start start 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.437+0000 7f7d58339640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d50071190 0x7f7d500b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.437+0000 7f7d58339640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d500b0c30 0x7f7d501e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d58339640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d5015d320 0x7f7d50156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d58339640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500b3490 con 0x7f7d50071190 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d58339640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500b3770 con 0x7f7d500b0c30 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d58339640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500b3a50 con 0x7f7d5015d320 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d560d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d50071190 0x7f7d500b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d560d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d50071190 0x7f7d500b4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37054/0 (socket says 172.21.15.45:37054) 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d560d5640 1 -- 172.21.15.45:0/2240946920 learned_addr learned my addr 172.21.15.45:0/2240946920 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d568d6640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d5015d320 0x7f7d50156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d558d4640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d500b0c30 0x7f7d501e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d560d5640 1 -- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d5015d320 msgr2=0x7f7d50156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d560d5640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d5015d320 0x7f7d50156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d560d5640 1 -- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d500b0c30 msgr2=0x7f7d501e6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d560d5640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d500b0c30 0x7f7d501e6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.438+0000 7f7d560d5640 1 -- 172.21.15.45:0/2240946920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d400094d0 con 0x7f7d50071190 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.439+0000 7f7d560d5640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d50071190 0x7f7d500b4530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c00c3e0 tx=0x7f7d4c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.439+0000 7f7d46ffd640 1 -- 172.21.15.45:0/2240946920 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c0074e0 con 0x7f7d50071190 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.439+0000 7f7d46ffd640 1 -- 172.21.15.45:0/2240946920 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c007640 con 0x7f7d50071190 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.439+0000 7f7d46ffd640 1 -- 172.21.15.45:0/2240946920 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c0049e0 con 0x7f7d50071190 2023-11-28T20:34:02.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.439+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d500b3d30 con 0x7f7d50071190 2023-11-28T20:34:02.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.439+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d500b4a70 con 0x7f7d50071190 2023-11-28T20:34:02.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.440+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d500733d0 con 0x7f7d50071190 2023-11-28T20:34:02.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.441+0000 7f7d46ffd640 1 -- 172.21.15.45:0/2240946920 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7d4c004b40 con 0x7f7d50071190 2023-11-28T20:34:02.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.442+0000 7f7d46ffd640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7d3406ae40 0x7f7d3406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:02.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.442+0000 7f7d46ffd640 1 -- 172.21.15.45:0/2240946920 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7d4c089390 con 0x7f7d50071190 2023-11-28T20:34:02.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.443+0000 7f7d558d4640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7d3406ae40 0x7f7d3406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:02.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.444+0000 7f7d558d4640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7d3406ae40 0x7f7d3406d300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7d40036000 tx=0x7f7d4003b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:02.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.446+0000 7f7d46ffd640 1 -- 172.21.15.45:0/2240946920 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d4c0540b0 con 0x7f7d50071190 2023-11-28T20:34:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.671+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7d500ad250 con 0x7f7d50071190 2023-11-28T20:34:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.672+0000 7f7d46ffd640 1 -- 172.21.15.45:0/2240946920 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f7d4c0576d0 con 0x7f7d50071190 2023-11-28T20:34:02.673 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T20:34:02.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7d3406ae40 msgr2=0x7f7d3406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7d3406ae40 0x7f7d3406d300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7d40036000 tx=0x7f7d4003b040 comp rx=0 tx=0).stop 2023-11-28T20:34:02.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d50071190 msgr2=0x7f7d500b4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:02.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d50071190 0x7f7d500b4530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c00c3e0 tx=0x7f7d4c00c8a0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 shutdown_connections 2023-11-28T20:34:02.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7d3406ae40 0x7f7d3406d300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d5015d320 0x7f7d50156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d500b0c30 0x7f7d501e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 --2- 172.21.15.45:0/2240946920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d50071190 0x7f7d500b4530 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:02.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.674+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 >> 172.21.15.45:0/2240946920 conn(0x7f7d5006a500 msgr2=0x7f7d500afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:02.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.675+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 shutdown_connections 2023-11-28T20:34:02.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:02.675+0000 7f7d58339640 1 -- 172.21.15.45:0/2240946920 wait complete. 2023-11-28T20:34:02.694 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T20:34:02.695 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:34:02.695 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:34:02.695 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-11-28T20:34:02.696 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-28T20:34:02.696 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T20:34:02.696 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T20:34:02.696 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:34:02.696 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-11-28T20:34:02.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:02.697 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:34:02.753 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:34:02.753 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:34:02.753 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:34:02.753 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:02.753 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:34:02.754 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:34:02.755 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42300sec preferred_lft 42300sec 2023-11-28T20:34:02.755 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:34:02.755 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:02.755 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:02.755 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:02.755 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:34:02.756 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T20:34:02.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:02.757 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:02.757 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:02.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add name ceph-brx type bridge 2023-11-28T20:34:02.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr flush dev ceph-brx 2023-11-28T20:34:02.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set ceph-brx up 2023-11-28T20:34:02.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T20:34:02.758 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:03.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:03.306 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:34:03.395 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:34:03.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:03.399 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:34:03.456 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:34:03.456 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:34:03.456 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:34:03.456 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:34:03.456 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:34:03.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:03.457 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:03.457 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:03.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:34:03.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:34:03.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:34:03.458 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:04.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:04.274 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:34:04.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:04.326 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2023-11-28T20:34:04.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:04.373 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:04.373 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:04.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T20:34:04.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-11-28T20:34:04.373 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:04.553 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-11-28T20:34:04.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:04.553 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:04.553 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:04.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-11-28T20:34:04.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T20:34:04.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-11-28T20:34:04.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-11-28T20:34:04.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-11-28T20:34:04.554 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:04.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:04.841 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:04.841 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:04.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:34:04.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T20:34:04.841 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:04.998 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-11-28T20:34:04.999 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:04.999 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-11-28T20:34:05.038 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-11-28T20:34:05.038 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:05.039 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-11-28T20:34:05.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2023-11-28T20:34:05.203 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T20:34:05.251 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2023-11-28T20:34:05.251 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2023-11-28T20:34:05.251 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2023-11-28T20:34:05.251 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2023-11-28T20:34:05.252 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T20:34:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2023-11-28T20:34:05.254 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T20:34:05.254 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T20:34:05.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:05.255 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2023-11-28T20:34:05.300 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T20:34:05.300 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-11-28T20:34:05.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2023-11-28T20:34:05.429 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T20:34:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2023-11-28T20:34:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2023-11-28T20:34:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2023-11-28T20:34:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2023-11-28T20:34:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2023-11-28T20:34:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2023-11-28T20:34:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2023-11-28T20:34:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2023-11-28T20:34:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2023-11-28T20:34:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T20:34:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T20:34:05.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:05.476 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2023-11-28T20:34:05.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi045.stderr:2023-11-28T20:34:05.567+0000 7fbe09580480 -1 init, newargv = 0x5610d3ed9820 newargc=15 2023-11-28T20:34:05.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi045.stderr:ceph-fuse[94005]: starting ceph client 2023-11-28T20:34:05.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi045.stderr:ceph-fuse[94005]: starting fuse 2023-11-28T20:34:06.666 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2023-11-28T20:34:06.783 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T20:34:06.843 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2023-11-28T20:34:06.844 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2023-11-28T20:34:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2023-11-28T20:34:06.846 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T20:34:06.846 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T20:34:06.846 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T20:34:06.846 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T20:34:06.846 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2023-11-28T20:34:06.846 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T20:34:06.846 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T20:34:06.846 INFO:teuthology.orchestra.run.smithi045.stdout:/home/ubuntu/cephtest/mnt.admin 2023-11-28T20:34:06.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:06.847 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2023-11-28T20:34:06.903 INFO:teuthology.orchestra.run.smithi045.stdout:41 2023-11-28T20:34:06.903 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-11-28T20:34:06.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo stdin-killer -- python3 -c ' 2023-11-28T20:34:06.903 DEBUG:teuthology.orchestra.run.smithi045:> import glob 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> import re 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> import os 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> import subprocess 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> def _find_admin_socket(client_name): 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> files = glob.glob(asok_path) 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> # Given a non-glob path, it better be there 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> if "*" not in asok_path: 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> assert(len(files) == 1) 2023-11-28T20:34:06.904 DEBUG:teuthology.orchestra.run.smithi045:> return files[0] 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> for f in files: 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> contents = proc_f.read() 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> if mountpoint in contents: 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> return f 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> print(_find_admin_socket("client.admin")) 2023-11-28T20:34:06.905 DEBUG:teuthology.orchestra.run.smithi045:> ' 2023-11-28T20:34:07.093 INFO:teuthology.orchestra.run.smithi045.stdout:/var/run/ceph/ceph-client.admin.94005.asok 2023-11-28T20:34:07.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T20:34:07.111 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.94005.asok 2023-11-28T20:34:07.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:07.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.94005.asok status 2023-11-28T20:34:07.362 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:34:07.362 INFO:teuthology.orchestra.run.smithi045.stdout: "metadata": { 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_sha1": "43dd94c190073cf16cff1e711dd7f5a7b1cbc883", 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 18.0.0-7520-g43dd94c1 (43dd94c190073cf16cff1e711dd7f5a7b1cbc883) reef (dev)", 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "entity_id": "admin", 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "hostname": "smithi045", 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "pid": "94005", 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "root": "/" 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "dentry_count": 0, 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "dentry_pinned_count": 0, 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 4419, 2023-11-28T20:34:07.363 INFO:teuthology.orchestra.run.smithi045.stdout: "inst": { 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: "name": { 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "client", 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 4419 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": { 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "192.168.0.1:0", 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3974227024 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:34:07.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": { 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "192.168.0.1:0", 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3974227024 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "inst_str": "client.4419 192.168.0.1:0/3974227024", 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "addr_str": "192.168.0.1:0/3974227024", 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "inode_count": 1, 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_epoch": 7, 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_epoch": 25, 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_epoch_barrier": 0, 2023-11-28T20:34:07.365 INFO:teuthology.orchestra.run.smithi045.stdout: "blocklisted": false, 2023-11-28T20:34:07.366 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs" 2023-11-28T20:34:07.366 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:34:07.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:07.378 DEBUG:teuthology.orchestra.run.smithi045:> stat --file-system '--printf=%T 2023-11-28T20:34:07.378 DEBUG:teuthology.orchestra.run.smithi045:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-11-28T20:34:07.434 INFO:teuthology.orchestra.run.smithi045.stdout:fuseblk 2023-11-28T20:34:07.435 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-11-28T20:34:07.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:07.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-11-28T20:34:07.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:07.539 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-11-28T20:34:07.632 INFO:teuthology.orchestra.run.smithi045.stdout:# file: . 2023-11-28T20:34:07.633 INFO:teuthology.orchestra.run.smithi045.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-11-28T20:34:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:07.633 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:34:07.633 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:34:07.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:07.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.921+0000 7f1a2f3c8640 1 Processor -- start 2023-11-28T20:34:07.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.921+0000 7f1a2f3c8640 1 -- start start 2023-11-28T20:34:07.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.923+0000 7f1a2f3c8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280cfbb0 0x7f1a280d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:07.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.923+0000 7f1a2f3c8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a280d1110 con 0x7f1a2815d6b0 2023-11-28T20:34:07.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.923+0000 7f1a2f3c8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a280d13f0 con 0x7f1a2815d2d0 2023-11-28T20:34:07.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.923+0000 7f1a2f3c8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280d2070 con 0x7f1a280cfbb0 2023-11-28T20:34:07.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.923+0000 7f1a2dbc5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1a2815d6b0 0x7f1a280ce0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51070/0 (socket says 172.21.15.45:51070) 2023-11-28T20:34:07.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.923+0000 7f1a2dbc5640 1 -- 172.21.15.45:0/2906561182 learned_addr learned my addr 172.21.15.45:0/2906561182 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:07.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2ebc7640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280cfbb0 0x7f1a280d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:07.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555244550 0 0) 0x7f1a280d1110 con 0x7f1a2815d6b0 2023-11-28T20:34:07.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a14003520 con 0x7f1a2815d6b0 2023-11-28T20:34:07.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2920448369 0 0) 0x7f1a14003520 con 0x7f1a2815d6b0 2023-11-28T20:34:07.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280cfbb0 msgr2=0x7f1a280d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280cfbb0 0x7f1a280d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 >> v1:172.21.15.190:6789/0 conn(0x7f1a2815d2d0 legacy=0x7f1a280b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a280d16d0 con 0x7f1a2815d6b0 2023-11-28T20:34:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f1a18001670 con 0x7f1a2815d6b0 2023-11-28T20:34:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 >> v1:172.21.15.45:6789/0 conn(0x7f1a2815d6b0 legacy=0x7f1a280ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:07.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a14003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280cfbb0 0x7f1a140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.924+0000 7f1a2d3c4640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a14004d00 0x7f1a14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280d2070 con 0x7f1a140039a0 2023-11-28T20:34:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1400ba60 con 0x7f1a14004d00 2023-11-28T20:34:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1400bc20 con 0x7f1a280cfbb0 2023-11-28T20:34:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2e3c6640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a14003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2dbc5640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280cfbb0 0x7f1a140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2ebc7640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a14004d00 0x7f1a14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:07.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2e3c6640 1 -- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280cfbb0 msgr2=0x7f1a140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2e3c6640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280cfbb0 0x7f1a140047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2e3c6640 1 -- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a14004d00 msgr2=0x7f1a14005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2e3c6640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a14004d00 0x7f1a14005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2e3c6640 1 -- 172.21.15.45:0/2906561182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a14003520 con 0x7f1a140039a0 2023-11-28T20:34:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.925+0000 7f1a2ebc7640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a14004d00 0x7f1a14005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.927+0000 7f1a2e3c6640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a14003dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1a24001720 tx=0x7f1a140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e03e7ddccce38a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.927+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240052d0 con 0x7f1a140039a0 2023-11-28T20:34:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.927+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2402e050 con 0x7f1a140039a0 2023-11-28T20:34:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.927+0000 7f1a2d3c4640 1 -- 172.21.15.45:0/2906561182 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240035f0 con 0x7f1a140039a0 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.927+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2906561182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 msgr2=0x7f1a14003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.927+0000 7f1a2f3c8640 1 --2- 172.21.15.45:0/2906561182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a14003dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1a24001720 tx=0x7f1a140057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.927+0000 7f1a2ebc7640 1 -- 172.21.15.45:0/2906561182 reap_dead start 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.928+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2906561182 shutdown_connections 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.928+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2906561182 >> 172.21.15.45:0/2906561182 conn(0x7f1a2806a500 msgr2=0x7f1a2806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.928+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2906561182 shutdown_connections 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.928+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2906561182 wait complete. 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.930+0000 7f1a2f3c8640 1 Processor -- start 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.930+0000 7f1a2f3c8640 1 -- start start 2023-11-28T20:34:07.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.931+0000 7f1a2f3c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a28153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.931+0000 7f1a2f3c8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a280b4070 0x7f1a280b2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.931+0000 7f1a2f3c8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280c8d30 0x7f1a280c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.931+0000 7f1a2f3c8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280c6e50 con 0x7f1a140039a0 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.931+0000 7f1a2f3c8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280c9700 con 0x7f1a280b4070 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.931+0000 7f1a2f3c8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280b34b0 con 0x7f1a280c8d30 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.931+0000 7f1a2e3c6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a28153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.931+0000 7f1a2e3c6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a28153810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56460/0 (socket says 172.21.15.45:56460) 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.931+0000 7f1a2e3c6640 1 -- 172.21.15.45:0/2377233842 learned_addr learned my addr 172.21.15.45:0/2377233842 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a2ebc7640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280c8d30 0x7f1a280c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a2dbc5640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a280b4070 0x7f1a280b2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:07.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a2e3c6640 1 -- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280c8d30 msgr2=0x7f1a280c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a2e3c6640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280c8d30 0x7f1a280c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a2e3c6640 1 -- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a280b4070 msgr2=0x7f1a280b2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a2e3c6640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a280b4070 0x7f1a280b2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a2e3c6640 1 -- 172.21.15.45:0/2377233842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a280b3790 con 0x7f1a140039a0 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a2e3c6640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a28153810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a24008fd0 tx=0x7f1a24001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a1effd640 1 -- 172.21.15.45:0/2377233842 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240076c0 con 0x7f1a140039a0 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a1effd640 1 -- 172.21.15.45:0/2377233842 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a24000ed0 con 0x7f1a140039a0 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a280b3a70 con 0x7f1a140039a0 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.932+0000 7f1a1effd640 1 -- 172.21.15.45:0/2377233842 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24006be0 con 0x7f1a140039a0 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.933+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a28153d50 con 0x7f1a140039a0 2023-11-28T20:34:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.934+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a28071ba0 con 0x7f1a140039a0 2023-11-28T20:34:07.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.935+0000 7f1a1effd640 1 -- 172.21.15.45:0/2377233842 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1a24006d40 con 0x7f1a140039a0 2023-11-28T20:34:07.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.936+0000 7f1a1effd640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a0c06ae40 0x7f1a0c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:07.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.936+0000 7f1a1effd640 1 -- 172.21.15.45:0/2377233842 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1a240ab5b0 con 0x7f1a140039a0 2023-11-28T20:34:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.936+0000 7f1a2dbc5640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a0c06ae40 0x7f1a0c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.937+0000 7f1a2dbc5640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a0c06ae40 0x7f1a0c06d300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a18001eb0 tx=0x7f1a18000ed0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:07.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:07.940+0000 7f1a1effd640 1 -- 172.21.15.45:0/2377233842 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a24076400 con 0x7f1a140039a0 2023-11-28T20:34:08.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.140+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a280b6900 con 0x7f1a140039a0 2023-11-28T20:34:08.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.143+0000 7f1a1effd640 1 -- 172.21.15.45:0/2377233842 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7f1a24079a20 con 0x7f1a140039a0 2023-11-28T20:34:08.144 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:08.144 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.145+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a0c06ae40 msgr2=0x7f1a0c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.145+0000 7f1a2f3c8640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a0c06ae40 0x7f1a0c06d300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a18001eb0 tx=0x7f1a18000ed0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.145+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 msgr2=0x7f1a28153810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.145+0000 7f1a2f3c8640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a28153810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a24008fd0 tx=0x7f1a24001120 comp rx=0 tx=0).stop 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.145+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 shutdown_connections 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.145+0000 7f1a2f3c8640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a0c06ae40 0x7f1a0c06d300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.145+0000 7f1a2f3c8640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a280c8d30 0x7f1a280c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.145+0000 7f1a2f3c8640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a280b4070 0x7f1a280b2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.145+0000 7f1a2f3c8640 1 --2- 172.21.15.45:0/2377233842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a140039a0 0x7f1a28153810 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.146+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 >> 172.21.15.45:0/2377233842 conn(0x7f1a2806a500 msgr2=0x7f1a280b6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.146+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 shutdown_connections 2023-11-28T20:34:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.146+0000 7f1a2f3c8640 1 -- 172.21.15.45:0/2377233842 wait complete. 2023-11-28T20:34:08.165 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:34:08.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:34:08.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.390+0000 7f97d3fff640 1 Processor -- start 2023-11-28T20:34:08.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.390+0000 7f97d3fff640 1 -- start start 2023-11-28T20:34:08.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.392+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 0x7f97d406cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.392+0000 7f97d3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97d415b270 0x7f97d415f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.392+0000 7f97d3fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d415faf0 con 0x7f97d40b3420 2023-11-28T20:34:08.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.392+0000 7f97d3fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d40ceaa0 con 0x7f97d415b270 2023-11-28T20:34:08.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.392+0000 7f97d3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d406d410 con 0x7f97d40b3040 2023-11-28T20:34:08.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.392+0000 7f97cbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97d415b270 0x7f97d415f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.392+0000 7f97d37fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f97d40b3420 0x7f97d40b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51080/0 (socket says 172.21.15.45:51080) 2023-11-28T20:34:08.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.393+0000 7f97d37fe640 1 -- 172.21.15.45:0/2049210874 learned_addr learned my addr 172.21.15.45:0/2049210874 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:08.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.393+0000 7f97d2ffd640 1 --2- 172.21.15.45:0/2049210874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 0x7f97d406cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.393+0000 7f97cbfff640 1 -- 172.21.15.45:0/2049210874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 msgr2=0x7f97d406cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:08.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.393+0000 7f97cbfff640 1 --2- 172.21.15.45:0/2049210874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 0x7f97d406cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.393+0000 7f97cbfff640 1 -- 172.21.15.45:0/2049210874 >> v1:172.21.15.45:6789/0 conn(0x7f97d40b3420 legacy=0x7f97d40b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.393+0000 7f97cbfff640 1 -- 172.21.15.45:0/2049210874 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d406d6f0 con 0x7f97d415b270 2023-11-28T20:34:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.393+0000 7f97d27fc640 1 -- 172.21.15.45:0/2049210874 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535815196 0 0) 0x7f97d415faf0 con 0x7f97d40b3420 2023-11-28T20:34:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.394+0000 7f97cbfff640 1 --2- 172.21.15.45:0/2049210874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 0x7f97d415f5b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097a0 tx=0x7f97bc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a296ac2399a6a0ba server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d27fc640 1 -- 172.21.15.45:0/2049210874 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0316f0 con 0x7f97d415b270 2023-11-28T20:34:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d27fc640 1 -- 172.21.15.45:0/2049210874 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc031850 con 0x7f97d415b270 2023-11-28T20:34:08.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d27fc640 1 -- 172.21.15.45:0/2049210874 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc031be0 con 0x7f97d415b270 2023-11-28T20:34:08.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d3fff640 1 -- 172.21.15.45:0/2049210874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 msgr2=0x7f97d415f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d3fff640 1 --2- 172.21.15.45:0/2049210874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 0x7f97d415f5b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097a0 tx=0x7f97bc02b790 comp rx=0 tx=0).stop 2023-11-28T20:34:08.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d3fff640 1 -- 172.21.15.45:0/2049210874 shutdown_connections 2023-11-28T20:34:08.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d3fff640 1 --2- 172.21.15.45:0/2049210874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 0x7f97d415f5b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d3fff640 1 --2- 172.21.15.45:0/2049210874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 0x7f97d406cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d3fff640 1 -- 172.21.15.45:0/2049210874 >> 172.21.15.45:0/2049210874 conn(0x7f97d40b1050 msgr2=0x7f97d40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.395+0000 7f97d3fff640 1 -- 172.21.15.45:0/2049210874 shutdown_connections 2023-11-28T20:34:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.396+0000 7f97d3fff640 1 -- 172.21.15.45:0/2049210874 wait complete. 2023-11-28T20:34:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.397+0000 7f97d3fff640 1 Processor -- start 2023-11-28T20:34:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.397+0000 7f97d3fff640 1 -- start start 2023-11-28T20:34:08.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.398+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 0x7f97d4099b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.398+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40b3420 0x7f97d409b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.398+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 0x7f97d409bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.398+0000 7f97d3fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d415d420 con 0x7f97d415b270 2023-11-28T20:34:08.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.398+0000 7f97d3fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d415aa40 con 0x7f97d40b3420 2023-11-28T20:34:08.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.398+0000 7f97d3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d4147070 con 0x7f97d40b3040 2023-11-28T20:34:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97d37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 0x7f97d409bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97d37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 0x7f97d409bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56474/0 (socket says 172.21.15.45:56474) 2023-11-28T20:34:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97d37fe640 1 -- 172.21.15.45:0/2396293535 learned_addr learned my addr 172.21.15.45:0/2396293535 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97d2ffd640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 0x7f97d4099b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97cbfff640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40b3420 0x7f97d409b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97d37fe640 1 -- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 msgr2=0x7f97d4099b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97d37fe640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 0x7f97d4099b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97d37fe640 1 -- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40b3420 msgr2=0x7f97d409b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97d37fe640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40b3420 0x7f97d409b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97d37fe640 1 -- 172.21.15.45:0/2396293535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc009400 con 0x7f97d415b270 2023-11-28T20:34:08.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.399+0000 7f97cbfff640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40b3420 0x7f97d409b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.400+0000 7f97d37fe640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 0x7f97d409bef0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00c810 tx=0x7f97cc00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.400+0000 7f97cb7fe640 1 -- 172.21.15.45:0/2396293535 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc00a640 con 0x7f97d415b270 2023-11-28T20:34:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.400+0000 7f97cb7fe640 1 -- 172.21.15.45:0/2396293535 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97cc00a7c0 con 0x7f97d415b270 2023-11-28T20:34:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.400+0000 7f97cb7fe640 1 -- 172.21.15.45:0/2396293535 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc00ab70 con 0x7f97d415b270 2023-11-28T20:34:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.400+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d415c8a0 con 0x7f97d415b270 2023-11-28T20:34:08.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.400+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d415ce60 con 0x7f97d415b270 2023-11-28T20:34:08.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.402+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d404e920 con 0x7f97d415b270 2023-11-28T20:34:08.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.403+0000 7f97cb7fe640 1 -- 172.21.15.45:0/2396293535 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f97cc00acd0 con 0x7f97d415b270 2023-11-28T20:34:08.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.403+0000 7f97cb7fe640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97b406ae70 0x7f97b406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.404+0000 7f97cb7fe640 1 -- 172.21.15.45:0/2396293535 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f97cc088c20 con 0x7f97d415b270 2023-11-28T20:34:08.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.404+0000 7f97d2ffd640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97b406ae70 0x7f97b406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.405+0000 7f97d2ffd640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97b406ae70 0x7f97b406d330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f97c40079a0 tx=0x7f97c4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:08.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.408+0000 7f97cb7fe640 1 -- 172.21.15.45:0/2396293535 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f97cc053b20 con 0x7f97d415b270 2023-11-28T20:34:08.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.601+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f97d40cf9a0 con 0x7f97d415b270 2023-11-28T20:34:08.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.602+0000 7f97cb7fe640 1 -- 172.21.15.45:0/2396293535 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97cc018020 con 0x7f97d415b270 2023-11-28T20:34:08.603 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:34:08.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97b406ae70 msgr2=0x7f97b406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97b406ae70 0x7f97b406d330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f97c40079a0 tx=0x7f97c4008040 comp rx=0 tx=0).stop 2023-11-28T20:34:08.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 msgr2=0x7f97d409bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 0x7f97d409bef0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00c810 tx=0x7f97cc00ccd0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 shutdown_connections 2023-11-28T20:34:08.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97b406ae70 0x7f97b406d330 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d415b270 0x7f97d409bef0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40b3420 0x7f97d409b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 --2- 172.21.15.45:0/2396293535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97d40b3040 0x7f97d4099b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 >> 172.21.15.45:0/2396293535 conn(0x7f97d40b1050 msgr2=0x7f97d40ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:08.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 shutdown_connections 2023-11-28T20:34:08.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.605+0000 7f97d3fff640 1 -- 172.21.15.45:0/2396293535 wait complete. 2023-11-28T20:34:08.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:34:08.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.853+0000 7faf174d1640 1 Processor -- start 2023-11-28T20:34:08.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.853+0000 7faf174d1640 1 -- start start 2023-11-28T20:34:08.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.855+0000 7faf174d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf1015b270 0x7faf1006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.855+0000 7faf174d1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf100b3420 0x7faf100ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.855+0000 7faf174d1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1006d410 con 0x7faf100b3040 2023-11-28T20:34:08.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.855+0000 7faf174d1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf100ceaa0 con 0x7faf100b3420 2023-11-28T20:34:08.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.855+0000 7faf174d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1006d6f0 con 0x7faf1015b270 2023-11-28T20:34:08.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.855+0000 7faf15cce640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf1015b270 0x7faf1006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.855+0000 7faf16cd0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf100b3420 0x7faf100ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.855+0000 7faf15cce640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf1015b270 0x7faf1006cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:35416/0 (socket says 172.21.15.45:35416) 2023-11-28T20:34:08.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.855+0000 7faf15cce640 1 -- 172.21.15.45:0/122034656 learned_addr learned my addr 172.21.15.45:0/122034656 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:08.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.856+0000 7faf15cce640 1 -- 172.21.15.45:0/122034656 >> v2:172.21.15.190:3300/0 conn(0x7faf100b3420 msgr2=0x7faf100ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:08.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.856+0000 7faf15cce640 1 --2- 172.21.15.45:0/122034656 >> v2:172.21.15.190:3300/0 conn(0x7faf100b3420 0x7faf100ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.856+0000 7faf15cce640 1 -- 172.21.15.45:0/122034656 >> v1:172.21.15.190:6789/0 conn(0x7faf100b3040 legacy=0x7faf1015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.856+0000 7faf15cce640 1 -- 172.21.15.45:0/122034656 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1006d9d0 con 0x7faf1015b270 2023-11-28T20:34:08.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.857+0000 7faf15cce640 1 --2- 172.21.15.45:0/122034656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf1015b270 0x7faf1006cda0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf00009870 tx=0x7faf0002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b920123e6c9e0ca6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:08.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.858+0000 7faf154cd640 1 -- 172.21.15.45:0/122034656 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf000317c0 con 0x7faf1015b270 2023-11-28T20:34:08.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.858+0000 7faf154cd640 1 -- 172.21.15.45:0/122034656 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf00031920 con 0x7faf1015b270 2023-11-28T20:34:08.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.858+0000 7faf154cd640 1 -- 172.21.15.45:0/122034656 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00031cb0 con 0x7faf1015b270 2023-11-28T20:34:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.858+0000 7faf174d1640 1 -- 172.21.15.45:0/122034656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf1015b270 msgr2=0x7faf1006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.858+0000 7faf174d1640 1 --2- 172.21.15.45:0/122034656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf1015b270 0x7faf1006cda0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf00009870 tx=0x7faf0002b860 comp rx=0 tx=0).stop 2023-11-28T20:34:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.859+0000 7faf174d1640 1 -- 172.21.15.45:0/122034656 shutdown_connections 2023-11-28T20:34:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.859+0000 7faf174d1640 1 --2- 172.21.15.45:0/122034656 >> v2:172.21.15.190:3300/0 conn(0x7faf100b3420 0x7faf100ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.859+0000 7faf174d1640 1 --2- 172.21.15.45:0/122034656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf1015b270 0x7faf1006cda0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.859+0000 7faf174d1640 1 -- 172.21.15.45:0/122034656 >> 172.21.15.45:0/122034656 conn(0x7faf100b1050 msgr2=0x7faf100cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.859+0000 7faf174d1640 1 -- 172.21.15.45:0/122034656 shutdown_connections 2023-11-28T20:34:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.859+0000 7faf174d1640 1 -- 172.21.15.45:0/122034656 wait complete. 2023-11-28T20:34:08.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.860+0000 7faf174d1640 1 Processor -- start 2023-11-28T20:34:08.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.861+0000 7faf174d1640 1 -- start start 2023-11-28T20:34:08.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf174d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf100b3040 0x7faf1014c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf174d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf100b3420 0x7faf101596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf174d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf1015b270 0x7faf10159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf174d1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf101f2f70 con 0x7faf100b3420 2023-11-28T20:34:08.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf174d1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1014e320 con 0x7faf1015b270 2023-11-28T20:34:08.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf174d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1014e600 con 0x7faf100b3040 2023-11-28T20:34:08.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf15cce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf100b3420 0x7faf101596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf15cce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf100b3420 0x7faf101596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56490/0 (socket says 172.21.15.45:56490) 2023-11-28T20:34:08.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf15cce640 1 -- 172.21.15.45:0/4267834992 learned_addr learned my addr 172.21.15.45:0/4267834992 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:08.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf164cf640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf100b3040 0x7faf1014c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.862+0000 7faf16cd0640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf1015b270 0x7faf10159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf15cce640 1 -- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf100b3040 msgr2=0x7faf1014c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf15cce640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf100b3040 0x7faf1014c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf15cce640 1 -- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf1015b270 msgr2=0x7faf10159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:08.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf15cce640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf1015b270 0x7faf10159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:08.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf15cce640 1 -- 172.21.15.45:0/4267834992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf000094d0 con 0x7faf100b3420 2023-11-28T20:34:08.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf15cce640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf100b3420 0x7faf101596f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faf000090b0 tx=0x7faf000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:08.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf06ffd640 1 -- 172.21.15.45:0/4267834992 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00032860 con 0x7faf100b3420 2023-11-28T20:34:08.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf06ffd640 1 -- 172.21.15.45:0/4267834992 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf000329c0 con 0x7faf100b3420 2023-11-28T20:34:08.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1014e8e0 con 0x7faf100b3420 2023-11-28T20:34:08.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf06ffd640 1 -- 172.21.15.45:0/4267834992 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00032d70 con 0x7faf100b3420 2023-11-28T20:34:08.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.863+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1014eea0 con 0x7faf100b3420 2023-11-28T20:34:08.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.864+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1009d0f0 con 0x7faf100b3420 2023-11-28T20:34:08.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.866+0000 7faf06ffd640 1 -- 172.21.15.45:0/4267834992 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faf0004d3f0 con 0x7faf100b3420 2023-11-28T20:34:08.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.867+0000 7faf06ffd640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faef406ae40 0x7faef406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:08.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.867+0000 7faf06ffd640 1 -- 172.21.15.45:0/4267834992 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faf000ab640 con 0x7faf100b3420 2023-11-28T20:34:08.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.867+0000 7faf164cf640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faef406ae40 0x7faef406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:08.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.868+0000 7faf164cf640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faef406ae40 0x7faef406d300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faf100ac900 tx=0x7faf0c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:08.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:08.871+0000 7faf06ffd640 1 -- 172.21.15.45:0/4267834992 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf00076360 con 0x7faf100b3420 2023-11-28T20:34:09.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.070+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faf1004e920 con 0x7faf100b3420 2023-11-28T20:34:09.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.071+0000 7faf06ffd640 1 -- 172.21.15.45:0/4267834992 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7faf00079980 con 0x7faf100b3420 2023-11-28T20:34:09.072 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faef406ae40 msgr2=0x7faef406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faef406ae40 0x7faef406d300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faf100ac900 tx=0x7faf0c009040 comp rx=0 tx=0).stop 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf100b3420 msgr2=0x7faf101596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf100b3420 0x7faf101596f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faf000090b0 tx=0x7faf000046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 shutdown_connections 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faef406ae40 0x7faef406d300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf1015b270 0x7faf10159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf100b3420 0x7faf101596f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 --2- 172.21.15.45:0/4267834992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf100b3040 0x7faf1014c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 >> 172.21.15.45:0/4267834992 conn(0x7faf100b1050 msgr2=0x7faf100abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.074+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 shutdown_connections 2023-11-28T20:34:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.075+0000 7faf174d1640 1 -- 172.21.15.45:0/4267834992 wait complete. 2023-11-28T20:34:09.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:34:09.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.320+0000 7ff02a087640 1 Processor -- start 2023-11-28T20:34:09.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.320+0000 7ff02a087640 1 -- start start 2023-11-28T20:34:09.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.322+0000 7ff02a087640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff02415e840 0x7ff02406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.322+0000 7ff02a087640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff02406ac10 0x7ff0240717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.322+0000 7ff02a087640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02415ecd0 con 0x7ff02415f720 2023-11-28T20:34:09.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.322+0000 7ff02a087640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff024071ce0 con 0x7ff02406ac10 2023-11-28T20:34:09.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.322+0000 7ff02a087640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff02415f290 con 0x7ff02415e840 2023-11-28T20:34:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.322+0000 7ff023fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff02406ac10 0x7ff0240717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.322+0000 7ff023fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff02406ac10 0x7ff0240717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56504/0 (socket says 172.21.15.45:56504) 2023-11-28T20:34:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.322+0000 7ff023fff640 1 -- 172.21.15.45:0/1197455842 learned_addr learned my addr 172.21.15.45:0/1197455842 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.322+0000 7ff0237fe640 1 --2- 172.21.15.45:0/1197455842 >> v2:172.21.15.190:3300/0 conn(0x7ff02415e840 0x7ff02406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.323+0000 7ff023fff640 1 -- 172.21.15.45:0/1197455842 >> v2:172.21.15.190:3300/0 conn(0x7ff02415e840 msgr2=0x7ff02406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.323+0000 7ff023fff640 1 --2- 172.21.15.45:0/1197455842 >> v2:172.21.15.190:3300/0 conn(0x7ff02415e840 0x7ff02406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.323+0000 7ff023fff640 1 -- 172.21.15.45:0/1197455842 >> v1:172.21.15.190:6789/0 conn(0x7ff02415f720 legacy=0x7ff02406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.323+0000 7ff023fff640 1 -- 172.21.15.45:0/1197455842 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02415fb60 con 0x7ff02406ac10 2023-11-28T20:34:09.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.324+0000 7ff023fff640 1 --2- 172.21.15.45:0/1197455842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 0x7ff0240717a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff0240cb5c0 tx=0x7ff01c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ddd869dfe22c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:09.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.324+0000 7ff0227fc640 1 -- 172.21.15.45:0/1197455842 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c02ca80 con 0x7ff02406ac10 2023-11-28T20:34:09.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.324+0000 7ff0227fc640 1 -- 172.21.15.45:0/1197455842 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c02cbe0 con 0x7ff02406ac10 2023-11-28T20:34:09.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.324+0000 7ff0227fc640 1 -- 172.21.15.45:0/1197455842 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c0343f0 con 0x7ff02406ac10 2023-11-28T20:34:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.325+0000 7ff02a087640 1 -- 172.21.15.45:0/1197455842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 msgr2=0x7ff0240717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.325+0000 7ff02a087640 1 --2- 172.21.15.45:0/1197455842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 0x7ff0240717a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff0240cb5c0 tx=0x7ff01c02bee0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.325+0000 7ff02a087640 1 -- 172.21.15.45:0/1197455842 shutdown_connections 2023-11-28T20:34:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.325+0000 7ff02a087640 1 --2- 172.21.15.45:0/1197455842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 0x7ff0240717a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.325+0000 7ff02a087640 1 --2- 172.21.15.45:0/1197455842 >> v2:172.21.15.190:3300/0 conn(0x7ff02415e840 0x7ff02406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.325+0000 7ff02a087640 1 -- 172.21.15.45:0/1197455842 >> 172.21.15.45:0/1197455842 conn(0x7ff0240ced60 msgr2=0x7ff0240cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.325+0000 7ff02a087640 1 -- 172.21.15.45:0/1197455842 shutdown_connections 2023-11-28T20:34:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.325+0000 7ff02a087640 1 -- 172.21.15.45:0/1197455842 wait complete. 2023-11-28T20:34:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.327+0000 7ff02a087640 1 Processor -- start 2023-11-28T20:34:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.327+0000 7ff02a087640 1 -- start start 2023-11-28T20:34:09.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff02a087640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 0x7ff02414d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff02a087640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02415e840 0x7ff02414c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff02a087640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff02415f720 0x7ff0240b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff02a087640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240c2bc0 con 0x7ff02406ac10 2023-11-28T20:34:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff02a087640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240c55c0 con 0x7ff02415f720 2023-11-28T20:34:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff02a087640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240c58a0 con 0x7ff02415e840 2023-11-28T20:34:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff0237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 0x7ff02414d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff0237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 0x7ff02414d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56520/0 (socket says 172.21.15.45:56520) 2023-11-28T20:34:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff0237fe640 1 -- 172.21.15.45:0/2237503666 learned_addr learned my addr 172.21.15.45:0/2237503666 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff0237fe640 1 -- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02415e840 msgr2=0x7ff02414c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff0237fe640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02415e840 0x7ff02414c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff0237fe640 1 -- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff02415f720 msgr2=0x7ff0240b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff023fff640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff02415f720 0x7ff0240b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff0237fe640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff02415f720 0x7ff0240b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.328+0000 7ff0237fe640 1 -- 172.21.15.45:0/2237503666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c009b30 con 0x7ff02406ac10 2023-11-28T20:34:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.329+0000 7ff0237fe640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 0x7ff02414d3b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff01400d800 tx=0x7ff01400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.329+0000 7ff00bfff640 1 -- 172.21.15.45:0/2237503666 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01400b6c0 con 0x7ff02406ac10 2023-11-28T20:34:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.329+0000 7ff00bfff640 1 -- 172.21.15.45:0/2237503666 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01400b820 con 0x7ff02406ac10 2023-11-28T20:34:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.329+0000 7ff00bfff640 1 -- 172.21.15.45:0/2237503666 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01400bbb0 con 0x7ff02406ac10 2023-11-28T20:34:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.329+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0240c2ea0 con 0x7ff02406ac10 2023-11-28T20:34:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.329+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0240c5b80 con 0x7ff02406ac10 2023-11-28T20:34:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.330+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0240733d0 con 0x7ff02406ac10 2023-11-28T20:34:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.331+0000 7ff00bfff640 1 -- 172.21.15.45:0/2237503666 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff01400bd10 con 0x7ff02406ac10 2023-11-28T20:34:09.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.332+0000 7ff00bfff640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff01006ae40 0x7ff01006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.332+0000 7ff00bfff640 1 -- 172.21.15.45:0/2237503666 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff014089fd0 con 0x7ff02406ac10 2023-11-28T20:34:09.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.333+0000 7ff022ffd640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff01006ae40 0x7ff01006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.334+0000 7ff022ffd640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff01006ae40 0x7ff01006d300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff018006fd0 tx=0x7ff018006bc0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.337+0000 7ff00bfff640 1 -- 172.21.15.45:0/2237503666 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff014054c60 con 0x7ff02406ac10 2023-11-28T20:34:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.533+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff0240ce2b0 con 0x7ff02406ac10 2023-11-28T20:34:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.534+0000 7ff00bfff640 1 -- 172.21.15.45:0/2237503666 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff014058280 con 0x7ff02406ac10 2023-11-28T20:34:09.535 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:34:09.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff01006ae40 msgr2=0x7ff01006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff01006ae40 0x7ff01006d300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff018006fd0 tx=0x7ff018006bc0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 msgr2=0x7ff02414d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 0x7ff02414d3b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff01400d800 tx=0x7ff01400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 shutdown_connections 2023-11-28T20:34:09.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff01006ae40 0x7ff01006d300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff02415f720 0x7ff0240b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02415e840 0x7ff02414c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 --2- 172.21.15.45:0/2237503666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff02406ac10 0x7ff02414d3b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.540+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 >> 172.21.15.45:0/2237503666 conn(0x7ff0240ced60 msgr2=0x7ff0240ad4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:09.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.541+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 shutdown_connections 2023-11-28T20:34:09.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.541+0000 7ff02a087640 1 -- 172.21.15.45:0/2237503666 wait complete. 2023-11-28T20:34:09.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:34:09.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.771+0000 7fbe7d86d640 1 Processor -- start 2023-11-28T20:34:09.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.771+0000 7fbe7d86d640 1 -- start start 2023-11-28T20:34:09.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.773+0000 7fbe7d86d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe7815b270 0x7fbe7806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.773+0000 7fbe7d86d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7806d410 con 0x7fbe780b3040 2023-11-28T20:34:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.773+0000 7fbe7d86d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7806d6f0 con 0x7fbe780b3420 2023-11-28T20:34:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.773+0000 7fbe7d86d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780ceaa0 con 0x7fbe7815b270 2023-11-28T20:34:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.773+0000 7fbe7c86b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbe780b3040 0x7fbe7815f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51090/0 (socket says 172.21.15.45:51090) 2023-11-28T20:34:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe7c86b640 1 -- 172.21.15.45:0/2551109877 learned_addr learned my addr 172.21.15.45:0/2551109877 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe77fff640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe7815b270 0x7fbe7806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832688113 0 0) 0x7fbe7806d410 con 0x7fbe780b3040 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe68003520 con 0x7fbe780b3040 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3205683762 0 0) 0x7fbe68003520 con 0x7fbe780b3040 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe7815b270 msgr2=0x7fbe7806cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe777fe640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe7815b270 0x7fbe7806cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 >> v1:172.21.15.190:6789/0 conn(0x7fbe780b3420 legacy=0x7fbe780ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7806d9d0 con 0x7fbe780b3040 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fbe6c001670 con 0x7fbe780b3040 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.774+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 >> v1:172.21.15.45:6789/0 conn(0x7fbe780b3040 legacy=0x7fbe7815f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe777fe640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe680039a0 0x7fbe68003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe777fe640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe7815b270 0x7fbe680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe777fe640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe68004d00 0x7fbe68005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6800bb10 con 0x7fbe680039a0 2023-11-28T20:34:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6800bc80 con 0x7fbe68004d00 2023-11-28T20:34:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6800be40 con 0x7fbe7815b270 2023-11-28T20:34:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe7c86b640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe680039a0 0x7fbe68003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe77fff640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe7815b270 0x7fbe680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe7d06c640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe68004d00 0x7fbe68005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe7c86b640 1 -- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe7815b270 msgr2=0x7fbe680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe7c86b640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe7815b270 0x7fbe680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe7c86b640 1 -- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe68004d00 msgr2=0x7fbe68005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe7c86b640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe68004d00 0x7fbe68005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe7c86b640 1 -- 172.21.15.45:0/2551109877 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe68003520 con 0x7fbe680039a0 2023-11-28T20:34:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.775+0000 7fbe7d06c640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe68004d00 0x7fbe68005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.776+0000 7fbe7c86b640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe680039a0 0x7fbe68003dc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00c240 tx=0x7fbe680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4195c8f09b4b443 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.777+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c005ad0 con 0x7fbe680039a0 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.777+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c005c30 con 0x7fbe680039a0 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.777+0000 7fbe777fe640 1 -- 172.21.15.45:0/2551109877 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c02c8b0 con 0x7fbe680039a0 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.777+0000 7fbe7d86d640 1 -- 172.21.15.45:0/2551109877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe680039a0 msgr2=0x7fbe68003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.777+0000 7fbe7d86d640 1 --2- 172.21.15.45:0/2551109877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe680039a0 0x7fbe68003dc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00c240 tx=0x7fbe680057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.777+0000 7fbe7d06c640 1 -- 172.21.15.45:0/2551109877 reap_dead start 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.777+0000 7fbe7d86d640 1 -- 172.21.15.45:0/2551109877 shutdown_connections 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.777+0000 7fbe7d86d640 1 -- 172.21.15.45:0/2551109877 >> 172.21.15.45:0/2551109877 conn(0x7fbe780b1050 msgr2=0x7fbe780cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.777+0000 7fbe7d86d640 1 -- 172.21.15.45:0/2551109877 shutdown_connections 2023-11-28T20:34:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.778+0000 7fbe7d86d640 1 -- 172.21.15.45:0/2551109877 wait complete. 2023-11-28T20:34:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.779+0000 7fbe7d86d640 1 Processor -- start 2023-11-28T20:34:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.779+0000 7fbe7d86d640 1 -- start start 2023-11-28T20:34:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.780+0000 7fbe7d86d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe680039a0 0x7fbe78147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.780+0000 7fbe7d86d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe78149ac0 0x7fbe780ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.780+0000 7fbe7d86d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78149f40 0x7fbe780ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.780+0000 7fbe7d86d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe78154c70 con 0x7fbe78149f40 2023-11-28T20:34:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.780+0000 7fbe7d86d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe781475b0 con 0x7fbe78149ac0 2023-11-28T20:34:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.780+0000 7fbe7d86d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe78147aa0 con 0x7fbe680039a0 2023-11-28T20:34:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe7d06c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78149f40 0x7fbe780ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe7d06c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78149f40 0x7fbe780ce9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56538/0 (socket says 172.21.15.45:56538) 2023-11-28T20:34:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe7d06c640 1 -- 172.21.15.45:0/497006658 learned_addr learned my addr 172.21.15.45:0/497006658 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe77fff640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe78149ac0 0x7fbe780ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe7c86b640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe680039a0 0x7fbe78147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe7d06c640 1 -- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe680039a0 msgr2=0x7fbe78147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe7d06c640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe680039a0 0x7fbe78147070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe7d06c640 1 -- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe78149ac0 msgr2=0x7fbe780ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe7d06c640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe78149ac0 0x7fbe780ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.781+0000 7fbe7d06c640 1 -- 172.21.15.45:0/497006658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7815d7c0 con 0x7fbe78149f40 2023-11-28T20:34:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.782+0000 7fbe7d06c640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78149f40 0x7fbe780ce9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbe780adfd0 tx=0x7fbe7000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.782+0000 7fbe757fa640 1 -- 172.21.15.45:0/497006658 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70006d50 con 0x7fbe78149f40 2023-11-28T20:34:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.782+0000 7fbe757fa640 1 -- 172.21.15.45:0/497006658 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70003970 con 0x7fbe78149f40 2023-11-28T20:34:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.782+0000 7fbe757fa640 1 -- 172.21.15.45:0/497006658 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70003d20 con 0x7fbe78149f40 2023-11-28T20:34:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.782+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe781546b0 con 0x7fbe78149f40 2023-11-28T20:34:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.782+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7815bd10 con 0x7fbe78149f40 2023-11-28T20:34:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.784+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7815bb70 con 0x7fbe78149f40 2023-11-28T20:34:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.784+0000 7fbe757fa640 1 -- 172.21.15.45:0/497006658 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbe70005070 con 0x7fbe78149f40 2023-11-28T20:34:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.785+0000 7fbe757fa640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbe5c06ae90 0x7fbe5c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.785+0000 7fbe757fa640 1 -- 172.21.15.45:0/497006658 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbe70089660 con 0x7fbe78149f40 2023-11-28T20:34:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.786+0000 7fbe7c86b640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbe5c06ae90 0x7fbe5c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.787+0000 7fbe7c86b640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbe5c06ae90 0x7fbe5c06d350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c034000 tx=0x7fbe6c009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.789+0000 7fbe757fa640 1 -- 172.21.15.45:0/497006658 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe7815bb70 con 0x7fbe78149f40 2023-11-28T20:34:09.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.994+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbe7815bb70 con 0x7fbe78149f40 2023-11-28T20:34:09.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.995+0000 7fbe757fa640 1 -- 172.21.15.45:0/497006658 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbe7815bb70 con 0x7fbe78149f40 2023-11-28T20:34:09.996 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:09.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.997+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbe5c06ae90 msgr2=0x7fbe5c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.997+0000 7fbe7d86d640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbe5c06ae90 0x7fbe5c06d350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c034000 tx=0x7fbe6c009030 comp rx=0 tx=0).stop 2023-11-28T20:34:09.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.997+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78149f40 msgr2=0x7fbe780ce9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:09.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.997+0000 7fbe7d86d640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78149f40 0x7fbe780ce9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbe780adfd0 tx=0x7fbe7000dc00 comp rx=0 tx=0).stop 2023-11-28T20:34:09.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.998+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 shutdown_connections 2023-11-28T20:34:10.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.998+0000 7fbe7d86d640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbe5c06ae90 0x7fbe5c06d350 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.998+0000 7fbe7d86d640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78149f40 0x7fbe780ce9d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.998+0000 7fbe7d86d640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe78149ac0 0x7fbe780ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.998+0000 7fbe7d86d640 1 --2- 172.21.15.45:0/497006658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe680039a0 0x7fbe78147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.998+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 >> 172.21.15.45:0/497006658 conn(0x7fbe780b1050 msgr2=0x7fbe780ad190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:10.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.998+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 shutdown_connections 2023-11-28T20:34:10.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:09.998+0000 7fbe7d86d640 1 -- 172.21.15.45:0/497006658 wait complete. 2023-11-28T20:34:10.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:34:10.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.239+0000 7ff597cd7640 1 Processor -- start 2023-11-28T20:34:10.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.239+0000 7ff597cd7640 1 -- start start 2023-11-28T20:34:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.240+0000 7ff597cd7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 0x7ff5900d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.240+0000 7ff597cd7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff590071230 0x7ff5900b6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.241+0000 7ff597cd7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59006b580 con 0x7ff5900d2750 2023-11-28T20:34:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.241+0000 7ff597cd7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59006b2a0 con 0x7ff590071230 2023-11-28T20:34:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.241+0000 7ff597cd7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5900d1730 con 0x7ff59015d730 2023-11-28T20:34:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.241+0000 7ff596274640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff590071230 0x7ff5900b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.241+0000 7ff595272640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff5900d2750 0x7ff590070b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51096/0 (socket says 172.21.15.45:51096) 2023-11-28T20:34:10.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.241+0000 7ff595272640 1 -- 172.21.15.45:0/937060166 learned_addr learned my addr 172.21.15.45:0/937060166 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:10.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.241+0000 7ff595a73640 1 --2- 172.21.15.45:0/937060166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 0x7ff5900d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.242+0000 7ff596274640 1 -- 172.21.15.45:0/937060166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 msgr2=0x7ff5900d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:10.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.242+0000 7ff596274640 1 --2- 172.21.15.45:0/937060166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 0x7ff5900d2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.242+0000 7ff596274640 1 -- 172.21.15.45:0/937060166 >> v1:172.21.15.45:6789/0 conn(0x7ff5900d2750 legacy=0x7ff590070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.242+0000 7ff596274640 1 -- 172.21.15.45:0/937060166 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5900d1a10 con 0x7ff590071230 2023-11-28T20:34:10.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.243+0000 7ff596274640 1 --2- 172.21.15.45:0/937060166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff590071230 0x7ff5900b6670 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5880097a0 tx=0x7ff58802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456fbc6e3fba0f9e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:10.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.243+0000 7ff594a71640 1 -- 172.21.15.45:0/937060166 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58802d040 con 0x7ff590071230 2023-11-28T20:34:10.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.243+0000 7ff594a71640 1 -- 172.21.15.45:0/937060166 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff588031810 con 0x7ff590071230 2023-11-28T20:34:10.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.243+0000 7ff594a71640 1 -- 172.21.15.45:0/937060166 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff588031ba0 con 0x7ff590071230 2023-11-28T20:34:10.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.244+0000 7ff597cd7640 1 -- 172.21.15.45:0/937060166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff590071230 msgr2=0x7ff5900b6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.244+0000 7ff597cd7640 1 --2- 172.21.15.45:0/937060166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff590071230 0x7ff5900b6670 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5880097a0 tx=0x7ff58802b8b0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.244+0000 7ff597cd7640 1 -- 172.21.15.45:0/937060166 shutdown_connections 2023-11-28T20:34:10.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.244+0000 7ff597cd7640 1 --2- 172.21.15.45:0/937060166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff590071230 0x7ff5900b6670 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.244+0000 7ff597cd7640 1 --2- 172.21.15.45:0/937060166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 0x7ff5900d2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.244+0000 7ff597cd7640 1 -- 172.21.15.45:0/937060166 >> 172.21.15.45:0/937060166 conn(0x7ff5900cfc70 msgr2=0x7ff5900d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:10.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.244+0000 7ff597cd7640 1 -- 172.21.15.45:0/937060166 shutdown_connections 2023-11-28T20:34:10.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.244+0000 7ff597cd7640 1 -- 172.21.15.45:0/937060166 wait complete. 2023-11-28T20:34:10.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.246+0000 7ff597cd7640 1 Processor -- start 2023-11-28T20:34:10.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.246+0000 7ff597cd7640 1 -- start start 2023-11-28T20:34:10.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.247+0000 7ff597cd7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590071230 0x7ff5900c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.247+0000 7ff597cd7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5900d2750 0x7ff590156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.247+0000 7ff597cd7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 0x7ff5901573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.247+0000 7ff597cd7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5900cba70 con 0x7ff5900d2750 2023-11-28T20:34:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.247+0000 7ff597cd7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59015a180 con 0x7ff590071230 2023-11-28T20:34:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.247+0000 7ff597cd7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff590159ea0 con 0x7ff59015d730 2023-11-28T20:34:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.247+0000 7ff595272640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5900d2750 0x7ff590156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.247+0000 7ff595272640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5900d2750 0x7ff590156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56558/0 (socket says 172.21.15.45:56558) 2023-11-28T20:34:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.247+0000 7ff595272640 1 -- 172.21.15.45:0/2438147020 learned_addr learned my addr 172.21.15.45:0/2438147020 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff595a73640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590071230 0x7ff5900c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff596274640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 0x7ff5901573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff595272640 1 -- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 msgr2=0x7ff5901573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff595272640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 0x7ff5901573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff595272640 1 -- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590071230 msgr2=0x7ff5900c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff595272640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590071230 0x7ff5900c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff595272640 1 -- 172.21.15.45:0/2438147020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff588009400 con 0x7ff5900d2750 2023-11-28T20:34:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff595272640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5900d2750 0x7ff590156e70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff58400c500 tx=0x7ff58400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff5827fc640 1 -- 172.21.15.45:0/2438147020 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584005ce0 con 0x7ff5900d2750 2023-11-28T20:34:10.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff5827fc640 1 -- 172.21.15.45:0/2438147020 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff584005e40 con 0x7ff5900d2750 2023-11-28T20:34:10.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.248+0000 7ff5827fc640 1 -- 172.21.15.45:0/2438147020 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5840072f0 con 0x7ff5900d2750 2023-11-28T20:34:10.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.249+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59015dc90 con 0x7ff5900d2750 2023-11-28T20:34:10.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.249+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5900cb690 con 0x7ff5900d2750 2023-11-28T20:34:10.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.250+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5900cf830 con 0x7ff5900d2750 2023-11-28T20:34:10.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.251+0000 7ff5827fc640 1 -- 172.21.15.45:0/2438147020 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff584007510 con 0x7ff5900d2750 2023-11-28T20:34:10.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.252+0000 7ff5827fc640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff57406ae40 0x7ff57406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.252+0000 7ff5827fc640 1 -- 172.21.15.45:0/2438147020 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff584088d50 con 0x7ff5900d2750 2023-11-28T20:34:10.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.252+0000 7ff595a73640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff57406ae40 0x7ff57406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.253+0000 7ff595a73640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff57406ae40 0x7ff57406d300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0073f0 tx=0x7ff58c00b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:10.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.256+0000 7ff5827fc640 1 -- 172.21.15.45:0/2438147020 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff584053bb0 con 0x7ff5900d2750 2023-11-28T20:34:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.464+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff590073440 con 0x7ff5900d2750 2023-11-28T20:34:10.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.465+0000 7ff5827fc640 1 -- 172.21.15.45:0/2438147020 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff584017020 con 0x7ff5900d2750 2023-11-28T20:34:10.466 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.467+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff57406ae40 msgr2=0x7ff57406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.467+0000 7ff597cd7640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff57406ae40 0x7ff57406d300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0073f0 tx=0x7ff58c00b040 comp rx=0 tx=0).stop 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.467+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5900d2750 msgr2=0x7ff590156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.467+0000 7ff597cd7640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5900d2750 0x7ff590156e70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff58400c500 tx=0x7ff58400c9c0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.467+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 shutdown_connections 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.467+0000 7ff597cd7640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff57406ae40 0x7ff57406d300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.467+0000 7ff597cd7640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff59015d730 0x7ff5901573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.467+0000 7ff597cd7640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5900d2750 0x7ff590156e70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.467+0000 7ff597cd7640 1 --2- 172.21.15.45:0/2438147020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff590071230 0x7ff5900c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.468+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 >> 172.21.15.45:0/2438147020 conn(0x7ff5900cfc70 msgr2=0x7ff5900d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.468+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 shutdown_connections 2023-11-28T20:34:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.468+0000 7ff597cd7640 1 -- 172.21.15.45:0/2438147020 wait complete. 2023-11-28T20:34:10.486 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:34:10.487 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:34:10.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:10.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.753+0000 7f5e85f63640 1 Processor -- start 2023-11-28T20:34:10.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.753+0000 7f5e85f63640 1 -- start start 2023-11-28T20:34:10.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.755+0000 7f5e85f63640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5e80155780 0x7f5e800abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.755+0000 7f5e85f63640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e800ae540 0x7f5e800ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.755+0000 7f5e85f63640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800ce630 0x7f5e8015ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.755+0000 7f5e85f63640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e800ceac0 con 0x7f5e800ae540 2023-11-28T20:34:10.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.755+0000 7f5e85f63640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8015c3c0 con 0x7f5e80155780 2023-11-28T20:34:10.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.755+0000 7f5e85f63640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80155b60 con 0x7f5e800ce630 2023-11-28T20:34:10.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.755+0000 7f5e7ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e800ae540 0x7f5e800ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.755+0000 7f5e7ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e800ae540 0x7f5e800ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56562/0 (socket says 172.21.15.45:56562) 2023-11-28T20:34:10.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.755+0000 7f5e7ffff640 1 -- 172.21.15.45:0/2651685692 learned_addr learned my addr 172.21.15.45:0/2651685692 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:10.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.756+0000 7f5e7ffff640 1 -- 172.21.15.45:0/2651685692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800ce630 msgr2=0x7f5e8015ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:10.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.756+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/2651685692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800ce630 0x7f5e8015ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.756+0000 7f5e84f61640 1 --2- 172.21.15.45:0/2651685692 >> v2:172.21.15.190:3300/0 conn(0x7f5e80155780 0x7f5e800abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.756+0000 7f5e7ffff640 1 -- 172.21.15.45:0/2651685692 >> v2:172.21.15.190:3300/0 conn(0x7f5e80155780 msgr2=0x7f5e800abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:10.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.756+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/2651685692 >> v2:172.21.15.190:3300/0 conn(0x7f5e80155780 0x7f5e800abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.756+0000 7f5e7ffff640 1 -- 172.21.15.45:0/2651685692 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8015ba90 con 0x7f5e800ae540 2023-11-28T20:34:10.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.757+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/2651685692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ae540 0x7f5e800ce0f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5e8006e6d0 tx=0x7f5e6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fffe1c550810eb8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:10.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.757+0000 7f5e7f7fe640 1 -- 172.21.15.45:0/2651685692 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c031870 con 0x7f5e800ae540 2023-11-28T20:34:10.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.757+0000 7f5e7f7fe640 1 -- 172.21.15.45:0/2651685692 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c0319d0 con 0x7f5e800ae540 2023-11-28T20:34:10.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.757+0000 7f5e7f7fe640 1 -- 172.21.15.45:0/2651685692 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c031d60 con 0x7f5e800ae540 2023-11-28T20:34:10.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.758+0000 7f5e85f63640 1 -- 172.21.15.45:0/2651685692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ae540 msgr2=0x7f5e800ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.758+0000 7f5e85f63640 1 --2- 172.21.15.45:0/2651685692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ae540 0x7f5e800ce0f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5e8006e6d0 tx=0x7f5e6c02b910 comp rx=0 tx=0).stop 2023-11-28T20:34:10.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.758+0000 7f5e85f63640 1 -- 172.21.15.45:0/2651685692 shutdown_connections 2023-11-28T20:34:10.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.758+0000 7f5e85f63640 1 --2- 172.21.15.45:0/2651685692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800ce630 0x7f5e8015ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.758+0000 7f5e85f63640 1 --2- 172.21.15.45:0/2651685692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ae540 0x7f5e800ce0f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.758+0000 7f5e85f63640 1 --2- 172.21.15.45:0/2651685692 >> v2:172.21.15.190:3300/0 conn(0x7f5e80155780 0x7f5e800abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.758+0000 7f5e85f63640 1 -- 172.21.15.45:0/2651685692 >> 172.21.15.45:0/2651685692 conn(0x7f5e8006ab30 msgr2=0x7f5e800b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:10.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.758+0000 7f5e85f63640 1 -- 172.21.15.45:0/2651685692 shutdown_connections 2023-11-28T20:34:10.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.758+0000 7f5e85f63640 1 -- 172.21.15.45:0/2651685692 wait complete. 2023-11-28T20:34:10.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.760+0000 7f5e85f63640 1 Processor -- start 2023-11-28T20:34:10.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.760+0000 7f5e85f63640 1 -- start start 2023-11-28T20:34:10.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.761+0000 7f5e85f63640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e800ae540 0x7f5e8013c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.761+0000 7f5e85f63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ce630 0x7f5e801e4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.761+0000 7f5e85f63640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e80155780 0x7f5e801e5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.761+0000 7f5e85f63640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801e39d0 con 0x7f5e800ce630 2023-11-28T20:34:10.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.761+0000 7f5e85f63640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801e4830 con 0x7f5e800ae540 2023-11-28T20:34:10.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.761+0000 7f5e85f63640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8013ccf0 con 0x7f5e80155780 2023-11-28T20:34:10.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.761+0000 7f5e7ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ce630 0x7f5e801e4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.761+0000 7f5e7ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ce630 0x7f5e801e4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56566/0 (socket says 172.21.15.45:56566) 2023-11-28T20:34:10.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.762+0000 7f5e7ffff640 1 -- 172.21.15.45:0/1328813630 learned_addr learned my addr 172.21.15.45:0/1328813630 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:10.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.762+0000 7f5e84f61640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e800ae540 0x7f5e8013c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.762+0000 7f5e85762640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e80155780 0x7f5e801e5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.762+0000 7f5e7ffff640 1 -- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e80155780 msgr2=0x7f5e801e5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.762+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e80155780 0x7f5e801e5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.762+0000 7f5e7ffff640 1 -- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e800ae540 msgr2=0x7f5e8013c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.762+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e800ae540 0x7f5e8013c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.762+0000 7f5e7ffff640 1 -- 172.21.15.45:0/1328813630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c0094d0 con 0x7f5e800ce630 2023-11-28T20:34:10.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.762+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ce630 0x7f5e801e4cc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c02b950 tx=0x7f5e6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:10.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.763+0000 7f5e7d7fa640 1 -- 172.21.15.45:0/1328813630 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c031860 con 0x7f5e800ce630 2023-11-28T20:34:10.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.763+0000 7f5e7d7fa640 1 -- 172.21.15.45:0/1328813630 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c0443f0 con 0x7f5e800ce630 2023-11-28T20:34:10.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.763+0000 7f5e7d7fa640 1 -- 172.21.15.45:0/1328813630 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c0447a0 con 0x7f5e800ce630 2023-11-28T20:34:10.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.763+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e801e3c20 con 0x7f5e800ce630 2023-11-28T20:34:10.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.763+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e801e41e0 con 0x7f5e800ce630 2023-11-28T20:34:10.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.764+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e801e49a0 con 0x7f5e800ce630 2023-11-28T20:34:10.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.765+0000 7f5e7d7fa640 1 -- 172.21.15.45:0/1328813630 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5e6c04d450 con 0x7f5e800ce630 2023-11-28T20:34:10.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.766+0000 7f5e7d7fa640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5e6406ae40 0x7f5e6406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:10.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.766+0000 7f5e7d7fa640 1 -- 172.21.15.45:0/1328813630 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5e6c02d040 con 0x7f5e800ce630 2023-11-28T20:34:10.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.767+0000 7f5e84f61640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5e6406ae40 0x7f5e6406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:10.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.768+0000 7f5e84f61640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5e6406ae40 0x7f5e6406d300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400a970 tx=0x7f5e74008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:10.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.771+0000 7f5e7d7fa640 1 -- 172.21.15.45:0/1328813630 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e6c076db0 con 0x7f5e800ce630 2023-11-28T20:34:10.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.965+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e8004e920 con 0x7f5e800ce630 2023-11-28T20:34:10.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.968+0000 7f5e7d7fa640 1 -- 172.21.15.45:0/1328813630 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7f5e6c035070 con 0x7f5e800ce630 2023-11-28T20:34:10.970 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:10.970 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5e6406ae40 msgr2=0x7f5e6406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5e6406ae40 0x7f5e6406d300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400a970 tx=0x7f5e74008040 comp rx=0 tx=0).stop 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ce630 msgr2=0x7f5e801e4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ce630 0x7f5e801e4cc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c02b950 tx=0x7f5e6c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 shutdown_connections 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5e6406ae40 0x7f5e6406d300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e80155780 0x7f5e801e5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e800ce630 0x7f5e801e4cc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 --2- 172.21.15.45:0/1328813630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e800ae540 0x7f5e8013c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.971+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 >> 172.21.15.45:0/1328813630 conn(0x7f5e8006ab30 msgr2=0x7f5e8006d380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:10.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.972+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 shutdown_connections 2023-11-28T20:34:10.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:10.972+0000 7f5e85f63640 1 -- 172.21.15.45:0/1328813630 wait complete. 2023-11-28T20:34:10.993 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:34:10.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:34:11.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.218+0000 7f0b4242e640 1 Processor -- start 2023-11-28T20:34:11.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.219+0000 7f0b4242e640 1 -- start start 2023-11-28T20:34:11.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.220+0000 7f0b4242e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0b3c159fc0 0x7f0b3c0b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.220+0000 7f0b4242e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c09ceb0 con 0x7f0b3c071ac0 2023-11-28T20:34:11.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.221+0000 7f0b4242e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c0b2cb0 con 0x7f0b3c0cdba0 2023-11-28T20:34:11.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.221+0000 7f0b4242e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3c071450 con 0x7f0b3c159fc0 2023-11-28T20:34:11.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.221+0000 7f0b4142c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0b3c159fc0 0x7f0b3c0b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.221+0000 7f0b40c2b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0b3c071ac0 0x7f0b3c15dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51102/0 (socket says 172.21.15.45:51102) 2023-11-28T20:34:11.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.221+0000 7f0b40c2b640 1 -- 172.21.15.45:0/4261333768 learned_addr learned my addr 172.21.15.45:0/4261333768 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:11.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.221+0000 7f0b4142c640 1 -- 172.21.15.45:0/4261333768 >> v1:172.21.15.190:6789/0 conn(0x7f0b3c0cdba0 legacy=0x7f0b3c15e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:11.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.221+0000 7f0b4142c640 1 -- 172.21.15.45:0/4261333768 >> v1:172.21.15.45:6789/0 conn(0x7f0b3c071ac0 legacy=0x7f0b3c15dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.221+0000 7f0b4142c640 1 -- 172.21.15.45:0/4261333768 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c15e9d0 con 0x7f0b3c159fc0 2023-11-28T20:34:11.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.221+0000 7f0b33fff640 1 -- 172.21.15.45:0/4261333768 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106588166 0 0) 0x7f0b3c09ceb0 con 0x7f0b3c071ac0 2023-11-28T20:34:11.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.223+0000 7f0b4142c640 1 --2- 172.21.15.45:0/4261333768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c159fc0 0x7f0b3c0b1d20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0b380097a0 tx=0x7f0b3802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966b958c88a2860 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:11.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.223+0000 7f0b33fff640 1 -- 172.21.15.45:0/4261333768 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3802e070 con 0x7f0b3c159fc0 2023-11-28T20:34:11.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.223+0000 7f0b33fff640 1 -- 172.21.15.45:0/4261333768 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b380315a0 con 0x7f0b3c159fc0 2023-11-28T20:34:11.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.223+0000 7f0b33fff640 1 -- 172.21.15.45:0/4261333768 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38031930 con 0x7f0b3c159fc0 2023-11-28T20:34:11.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.223+0000 7f0b4242e640 1 -- 172.21.15.45:0/4261333768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c159fc0 msgr2=0x7f0b3c0b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.223+0000 7f0b4242e640 1 --2- 172.21.15.45:0/4261333768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c159fc0 0x7f0b3c0b1d20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0b380097a0 tx=0x7f0b3802b840 comp rx=0 tx=0).stop 2023-11-28T20:34:11.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.224+0000 7f0b4242e640 1 -- 172.21.15.45:0/4261333768 shutdown_connections 2023-11-28T20:34:11.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.224+0000 7f0b4242e640 1 --2- 172.21.15.45:0/4261333768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c159fc0 0x7f0b3c0b1d20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.224+0000 7f0b4242e640 1 -- 172.21.15.45:0/4261333768 >> 172.21.15.45:0/4261333768 conn(0x7f0b3c0b1050 msgr2=0x7f0b3c0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:11.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.224+0000 7f0b4242e640 1 -- 172.21.15.45:0/4261333768 shutdown_connections 2023-11-28T20:34:11.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.224+0000 7f0b4242e640 1 -- 172.21.15.45:0/4261333768 wait complete. 2023-11-28T20:34:11.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.225+0000 7f0b4242e640 1 Processor -- start 2023-11-28T20:34:11.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.226+0000 7f0b4242e640 1 -- start start 2023-11-28T20:34:11.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.226+0000 7f0b4242e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c071ac0 0x7f0b3c140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.226+0000 7f0b4242e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b3c0cdba0 0x7f0b3c144bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4242e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b3c159fc0 0x7f0b3c13f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4242e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c140e90 con 0x7f0b3c071ac0 2023-11-28T20:34:11.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4242e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c167df0 con 0x7f0b3c159fc0 2023-11-28T20:34:11.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4242e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c1453e0 con 0x7f0b3c0cdba0 2023-11-28T20:34:11.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4142c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c071ac0 0x7f0b3c140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4142c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c071ac0 0x7f0b3c140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56594/0 (socket says 172.21.15.45:56594) 2023-11-28T20:34:11.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4142c640 1 -- 172.21.15.45:0/1038091462 learned_addr learned my addr 172.21.15.45:0/1038091462 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b41c2d640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b3c159fc0 0x7f0b3c13f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4142c640 1 -- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b3c0cdba0 msgr2=0x7f0b3c144bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b40c2b640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b3c0cdba0 0x7f0b3c144bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4142c640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b3c0cdba0 0x7f0b3c144bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4142c640 1 -- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b3c159fc0 msgr2=0x7f0b3c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4142c640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b3c159fc0 0x7f0b3c13f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b4142c640 1 -- 172.21.15.45:0/1038091462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b38009400 con 0x7f0b3c071ac0 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b41c2d640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b3c159fc0 0x7f0b3c13f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.227+0000 7f0b40c2b640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b3c0cdba0 0x7f0b3c144bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.228+0000 7f0b4142c640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c071ac0 0x7f0b3c140950 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0b38034040 tx=0x7f0b380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:11.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.228+0000 7f0b31ffb640 1 -- 172.21.15.45:0/1038091462 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38032900 con 0x7f0b3c071ac0 2023-11-28T20:34:11.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.228+0000 7f0b31ffb640 1 -- 172.21.15.45:0/1038091462 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b38032a60 con 0x7f0b3c071ac0 2023-11-28T20:34:11.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.228+0000 7f0b31ffb640 1 -- 172.21.15.45:0/1038091462 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38032e10 con 0x7f0b3c071ac0 2023-11-28T20:34:11.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.228+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c1456c0 con 0x7f0b3c071ac0 2023-11-28T20:34:11.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.228+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c145c80 con 0x7f0b3c071ac0 2023-11-28T20:34:11.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.229+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c04e920 con 0x7f0b3c071ac0 2023-11-28T20:34:11.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.230+0000 7f0b31ffb640 1 -- 172.21.15.45:0/1038091462 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0b3803b3f0 con 0x7f0b3c071ac0 2023-11-28T20:34:11.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.231+0000 7f0b31ffb640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0b2006ae40 0x7f0b2006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.231+0000 7f0b31ffb640 1 -- 172.21.15.45:0/1038091462 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0b380ab9e0 con 0x7f0b3c071ac0 2023-11-28T20:34:11.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.232+0000 7f0b40c2b640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0b2006ae40 0x7f0b2006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.233+0000 7f0b40c2b640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0b2006ae40 0x7f0b2006d300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c169f20 tx=0x7f0b2c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:11.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.236+0000 7f0b31ffb640 1 -- 172.21.15.45:0/1038091462 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b38076790 con 0x7f0b3c071ac0 2023-11-28T20:34:11.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.421+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b3c06bf60 con 0x7f0b3c071ac0 2023-11-28T20:34:11.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.422+0000 7f0b31ffb640 1 -- 172.21.15.45:0/1038091462 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b38079db0 con 0x7f0b3c071ac0 2023-11-28T20:34:11.423 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:34:11.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.424+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0b2006ae40 msgr2=0x7f0b2006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.424+0000 7f0b4242e640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0b2006ae40 0x7f0b2006d300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c169f20 tx=0x7f0b2c009040 comp rx=0 tx=0).stop 2023-11-28T20:34:11.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.424+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c071ac0 msgr2=0x7f0b3c140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.424+0000 7f0b4242e640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c071ac0 0x7f0b3c140950 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0b38034040 tx=0x7f0b380046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.425+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 shutdown_connections 2023-11-28T20:34:11.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.425+0000 7f0b4242e640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0b2006ae40 0x7f0b2006d300 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.425+0000 7f0b4242e640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b3c159fc0 0x7f0b3c13f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.425+0000 7f0b4242e640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b3c0cdba0 0x7f0b3c144bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.425+0000 7f0b4242e640 1 --2- 172.21.15.45:0/1038091462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b3c071ac0 0x7f0b3c140950 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.425+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 >> 172.21.15.45:0/1038091462 conn(0x7f0b3c0b1050 msgr2=0x7f0b3c0cd270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:11.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.425+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 shutdown_connections 2023-11-28T20:34:11.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.425+0000 7f0b4242e640 1 -- 172.21.15.45:0/1038091462 wait complete. 2023-11-28T20:34:11.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:34:11.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.670+0000 7f6ffa716640 1 Processor -- start 2023-11-28T20:34:11.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.670+0000 7f6ffa716640 1 -- start start 2023-11-28T20:34:11.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.671+0000 7f6ffa716640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6ff40adfa0 0x7f6ff40abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.671+0000 7f6ffa716640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff40ae480 0x7f6ff40717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.671+0000 7f6ffa716640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40708d0 con 0x7f6ff415d4b0 2023-11-28T20:34:11.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.671+0000 7f6ffa716640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff40b3bb0 con 0x7f6ff40adfa0 2023-11-28T20:34:11.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.671+0000 7f6ffa716640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40b3e90 con 0x7f6ff40ae480 2023-11-28T20:34:11.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6ff9714640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6ff415d4b0 0x7f6ff4070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51104/0 (socket says 172.21.15.45:51104) 2023-11-28T20:34:11.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6ff9714640 1 -- 172.21.15.45:0/1858926014 learned_addr learned my addr 172.21.15.45:0/1858926014 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825612634 0 0) 0x7f6ff40708d0 con 0x7f6ff415d4b0 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/1858926014 >> v2:172.21.15.190:3300/0 conn(0x7f6ff40adfa0 0x7f6ff40abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe0003520 con 0x7f6ff415d4b0 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6ff9f15640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff40ae480 0x7f6ff40717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3397462385 0 0) 0x7f6fe0003520 con 0x7f6ff415d4b0 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff40ae480 msgr2=0x7f6ff40717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff40ae480 0x7f6ff40717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 >> v2:172.21.15.190:3300/0 conn(0x7f6ff40adfa0 msgr2=0x7f6ff40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 --2- 172.21.15.45:0/1858926014 >> v2:172.21.15.190:3300/0 conn(0x7f6ff40adfa0 0x7f6ff40abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff4070280 con 0x7f6ff415d4b0 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6ff0001670 con 0x7f6ff415d4b0 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 >> v1:172.21.15.45:6789/0 conn(0x7f6ff415d4b0 legacy=0x7f6ff4070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6fe0003970 0x7f6fe0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.672+0000 7f6febfff640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6fe0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.673+0000 7f6febfff640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fe0005180 0x7f6fe000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.673+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40b3bb0 con 0x7f6fe00047c0 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.673+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40b3e90 con 0x7f6fe0005180 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.673+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe000c0a0 con 0x7f6fe0003970 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.673+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/1858926014 >> v2:172.21.15.190:3300/0 conn(0x7f6ff40adfa0 0x7f6ff40abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:11.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.673+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6fe0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.673+0000 7f6ff9f15640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fe0005180 0x7f6fe000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.673+0000 7f6ff9714640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6fe0003970 0x7f6fe0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.674+0000 7f6ff8f13640 1 -- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6fe0003970 msgr2=0x7f6fe0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:11.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.674+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6fe0003970 0x7f6fe0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.674+0000 7f6ff8f13640 1 -- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fe0005180 msgr2=0x7f6fe000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:11.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.674+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fe0005180 0x7f6fe000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.674+0000 7f6ff8f13640 1 -- 172.21.15.45:0/1858926014 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0003520 con 0x7f6fe00047c0 2023-11-28T20:34:11.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.675+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6fe0004c40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40cc400 tx=0x7f6fe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbc1f91dc8c4a23 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:11.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.675+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4004230 con 0x7f6fe00047c0 2023-11-28T20:34:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.676+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4004390 con 0x7f6fe00047c0 2023-11-28T20:34:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.676+0000 7f6febfff640 1 -- 172.21.15.45:0/1858926014 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4003850 con 0x7f6fe00047c0 2023-11-28T20:34:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.676+0000 7f6ffa716640 1 -- 172.21.15.45:0/1858926014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 msgr2=0x7f6fe0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.676+0000 7f6ffa716640 1 --2- 172.21.15.45:0/1858926014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6fe0004c40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40cc400 tx=0x7f6fe00057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.676+0000 7f6ff9f15640 1 -- 172.21.15.45:0/1858926014 reap_dead start 2023-11-28T20:34:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.676+0000 7f6ffa716640 1 -- 172.21.15.45:0/1858926014 shutdown_connections 2023-11-28T20:34:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.676+0000 7f6ffa716640 1 -- 172.21.15.45:0/1858926014 >> 172.21.15.45:0/1858926014 conn(0x7f6ff40b1050 msgr2=0x7f6ff40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.676+0000 7f6ffa716640 1 -- 172.21.15.45:0/1858926014 shutdown_connections 2023-11-28T20:34:11.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.676+0000 7f6ffa716640 1 -- 172.21.15.45:0/1858926014 wait complete. 2023-11-28T20:34:11.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.678+0000 7f6ffa716640 1 Processor -- start 2023-11-28T20:34:11.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.678+0000 7f6ffa716640 1 -- start start 2023-11-28T20:34:11.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.679+0000 7f6ffa716640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6ff41f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.679+0000 7f6ffa716640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff41605b0 0x7f6ff4158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.679+0000 7f6ffa716640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ff4099a40 0x7f6ff414dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.679+0000 7f6ffa716640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff409cd10 con 0x7f6fe00047c0 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.679+0000 7f6ffa716640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff4158520 con 0x7f6ff4099a40 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.679+0000 7f6ffa716640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff409c750 con 0x7f6ff41605b0 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9714640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6ff41f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9714640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6ff41f64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56612/0 (socket says 172.21.15.45:56612) 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9714640 1 -- 172.21.15.45:0/358037789 learned_addr learned my addr 172.21.15.45:0/358037789 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9f15640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ff4099a40 0x7f6ff414dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff41605b0 0x7f6ff4158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9714640 1 -- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff41605b0 msgr2=0x7f6ff4158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9714640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff41605b0 0x7f6ff4158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9714640 1 -- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ff4099a40 msgr2=0x7f6ff414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9714640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ff4099a40 0x7f6ff414dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9714640 1 -- 172.21.15.45:0/358037789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff409ca30 con 0x7f6fe00047c0 2023-11-28T20:34:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff41605b0 0x7f6ff4158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:11.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.680+0000 7f6ff9714640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6ff41f64d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0004f70 tx=0x7f6ff0007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:11.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.681+0000 7f6fe9ffb640 1 -- 172.21.15.45:0/358037789 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff000f6d0 con 0x7f6fe00047c0 2023-11-28T20:34:11.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.681+0000 7f6fe9ffb640 1 -- 172.21.15.45:0/358037789 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff000f870 con 0x7f6fe00047c0 2023-11-28T20:34:11.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.681+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff41587c0 con 0x7f6fe00047c0 2023-11-28T20:34:11.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.681+0000 7f6fe9ffb640 1 -- 172.21.15.45:0/358037789 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff000fc20 con 0x7f6fe00047c0 2023-11-28T20:34:11.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.681+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff41f6bf0 con 0x7f6fe00047c0 2023-11-28T20:34:11.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.682+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff41589b0 con 0x7f6fe00047c0 2023-11-28T20:34:11.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.683+0000 7f6fe9ffb640 1 -- 172.21.15.45:0/358037789 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6ff000fd80 con 0x7f6fe00047c0 2023-11-28T20:34:11.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.684+0000 7f6fe9ffb640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6fd806ae40 0x7f6fd806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:11.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.684+0000 7f6fe9ffb640 1 -- 172.21.15.45:0/358037789 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6ff0089850 con 0x7f6fe00047c0 2023-11-28T20:34:11.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.685+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6fd806ae40 0x7f6fd806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:11.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.686+0000 7f6ff8f13640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6fd806ae40 0x7f6fd806d300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4035000 tx=0x7f6fe403a040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:11.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.688+0000 7f6fe9ffb640 1 -- 172.21.15.45:0/358037789 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ff0054570 con 0x7f6fe00047c0 2023-11-28T20:34:11.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.874+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6ff41f6bf0 con 0x7f6fe00047c0 2023-11-28T20:34:11.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.876+0000 7f6fe9ffb640 1 -- 172.21.15.45:0/358037789 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ff41f6bf0 con 0x7f6fe00047c0 2023-11-28T20:34:11.877 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:11.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6fd806ae40 msgr2=0x7f6fd806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6fd806ae40 0x7f6fd806d300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4035000 tx=0x7f6fe403a040 comp rx=0 tx=0).stop 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 msgr2=0x7f6ff41f64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6ff41f64d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0004f70 tx=0x7f6ff0007430 comp rx=0 tx=0).stop 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 shutdown_connections 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6fd806ae40 0x7f6fd806d300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ff4099a40 0x7f6ff414dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ff41605b0 0x7f6ff4158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 --2- 172.21.15.45:0/358037789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00047c0 0x7f6ff41f64d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 >> 172.21.15.45:0/358037789 conn(0x7f6ff40b1050 msgr2=0x7f6ff40ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 shutdown_connections 2023-11-28T20:34:11.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:11.878+0000 7f6ffa716640 1 -- 172.21.15.45:0/358037789 wait complete. 2023-11-28T20:34:11.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:34:12.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.126+0000 7fdba82ad640 1 Processor -- start 2023-11-28T20:34:12.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.126+0000 7fdba82ad640 1 -- start start 2023-11-28T20:34:12.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.128+0000 7fdba82ad640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdba006dc50 0x7fdba00cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.128+0000 7fdba82ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdba006ac10 0x7fdba00c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.128+0000 7fdba82ad640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba006e030 con 0x7fdba00ce600 2023-11-28T20:34:12.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.128+0000 7fdba82ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba00ccf60 con 0x7fdba006ac10 2023-11-28T20:34:12.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.128+0000 7fdba82ad640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba00cd930 con 0x7fdba006dc50 2023-11-28T20:34:12.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.129+0000 7fdba684a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdba006ac10 0x7fdba00c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:12.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.129+0000 7fdba684a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdba006ac10 0x7fdba00c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56618/0 (socket says 172.21.15.45:56618) 2023-11-28T20:34:12.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.129+0000 7fdba684a640 1 -- 172.21.15.45:0/606743464 learned_addr learned my addr 172.21.15.45:0/606743464 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:12.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.129+0000 7fdba6049640 1 --2- 172.21.15.45:0/606743464 >> v2:172.21.15.190:3300/0 conn(0x7fdba006dc50 0x7fdba00cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:12.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.129+0000 7fdba684a640 1 -- 172.21.15.45:0/606743464 >> v2:172.21.15.190:3300/0 conn(0x7fdba006dc50 msgr2=0x7fdba00cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:12.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.129+0000 7fdba684a640 1 --2- 172.21.15.45:0/606743464 >> v2:172.21.15.190:3300/0 conn(0x7fdba006dc50 0x7fdba00cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.129+0000 7fdba684a640 1 -- 172.21.15.45:0/606743464 >> v1:172.21.15.190:6789/0 conn(0x7fdba00ce600 legacy=0x7fdba006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:12.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.129+0000 7fdba684a640 1 -- 172.21.15.45:0/606743464 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba00cc930 con 0x7fdba006ac10 2023-11-28T20:34:12.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.129+0000 7fdba6049640 1 --2- 172.21.15.45:0/606743464 >> v2:172.21.15.190:3300/0 conn(0x7fdba006dc50 0x7fdba00cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:12.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.130+0000 7fdba684a640 1 --2- 172.21.15.45:0/606743464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006ac10 0x7fdba00c93b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdba006b230 tx=0x7fdb9802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9260d8ee5f032222 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:12.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.131+0000 7fdba5047640 1 -- 172.21.15.45:0/606743464 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9802ca80 con 0x7fdba006ac10 2023-11-28T20:34:12.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.131+0000 7fdba5047640 1 -- 172.21.15.45:0/606743464 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9802cbe0 con 0x7fdba006ac10 2023-11-28T20:34:12.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.131+0000 7fdba5047640 1 -- 172.21.15.45:0/606743464 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb980343f0 con 0x7fdba006ac10 2023-11-28T20:34:12.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.131+0000 7fdba82ad640 1 -- 172.21.15.45:0/606743464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006ac10 msgr2=0x7fdba00c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.131+0000 7fdba82ad640 1 --2- 172.21.15.45:0/606743464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006ac10 0x7fdba00c93b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdba006b230 tx=0x7fdb9802bee0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.132+0000 7fdba82ad640 1 -- 172.21.15.45:0/606743464 shutdown_connections 2023-11-28T20:34:12.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.132+0000 7fdba82ad640 1 --2- 172.21.15.45:0/606743464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006ac10 0x7fdba00c93b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.132+0000 7fdba82ad640 1 --2- 172.21.15.45:0/606743464 >> v2:172.21.15.190:3300/0 conn(0x7fdba006dc50 0x7fdba00cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.132+0000 7fdba82ad640 1 -- 172.21.15.45:0/606743464 >> 172.21.15.45:0/606743464 conn(0x7fdba00ced60 msgr2=0x7fdba00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:12.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.132+0000 7fdba82ad640 1 -- 172.21.15.45:0/606743464 shutdown_connections 2023-11-28T20:34:12.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.132+0000 7fdba82ad640 1 -- 172.21.15.45:0/606743464 wait complete. 2023-11-28T20:34:12.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.133+0000 7fdba82ad640 1 Processor -- start 2023-11-28T20:34:12.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.134+0000 7fdba82ad640 1 -- start start 2023-11-28T20:34:12.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.134+0000 7fdba82ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdba006ac10 0x7fdba00c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.134+0000 7fdba82ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006dc50 0x7fdba01f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.134+0000 7fdba82ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba00ce600 0x7fdba01f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba82ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00c71b0 con 0x7fdba006dc50 2023-11-28T20:34:12.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba82ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00c5e30 con 0x7fdba006ac10 2023-11-28T20:34:12.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba82ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00c6110 con 0x7fdba00ce600 2023-11-28T20:34:12.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba5848640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006dc50 0x7fdba01f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:12.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba5848640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006dc50 0x7fdba01f5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56622/0 (socket says 172.21.15.45:56622) 2023-11-28T20:34:12.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba5848640 1 -- 172.21.15.45:0/145246088 learned_addr learned my addr 172.21.15.45:0/145246088 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:12.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba5848640 1 -- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba00ce600 msgr2=0x7fdba01f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba5848640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba00ce600 0x7fdba01f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba5848640 1 -- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdba006ac10 msgr2=0x7fdba00c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba5848640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdba006ac10 0x7fdba00c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba5848640 1 -- 172.21.15.45:0/145246088 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb98009b30 con 0x7fdba006dc50 2023-11-28T20:34:12.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.135+0000 7fdba5848640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006dc50 0x7fdba01f5f90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdb9000c430 tx=0x7fdb9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:12.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.136+0000 7fdb96ffd640 1 -- 172.21.15.45:0/145246088 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb90011790 con 0x7fdba006dc50 2023-11-28T20:34:12.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.136+0000 7fdb96ffd640 1 -- 172.21.15.45:0/145246088 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb900118f0 con 0x7fdba006dc50 2023-11-28T20:34:12.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.136+0000 7fdb96ffd640 1 -- 172.21.15.45:0/145246088 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb90011ce0 con 0x7fdba006dc50 2023-11-28T20:34:12.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.136+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba00c63f0 con 0x7fdba006dc50 2023-11-28T20:34:12.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.136+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba00c69b0 con 0x7fdba006dc50 2023-11-28T20:34:12.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.137+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba00733d0 con 0x7fdba006dc50 2023-11-28T20:34:12.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.138+0000 7fdb96ffd640 1 -- 172.21.15.45:0/145246088 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdb9001c460 con 0x7fdba006dc50 2023-11-28T20:34:12.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.139+0000 7fdb96ffd640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdb8406ae40 0x7fdb8406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.139+0000 7fdb96ffd640 1 -- 172.21.15.45:0/145246088 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdb900898a0 con 0x7fdba006dc50 2023-11-28T20:34:12.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.140+0000 7fdba6049640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdb8406ae40 0x7fdb8406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:12.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.141+0000 7fdba6049640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdb8406ae40 0x7fdb8406d300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c006fd0 tx=0x7fdb9c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.144+0000 7fdb96ffd640 1 -- 172.21.15.45:0/145246088 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb90054660 con 0x7fdba006dc50 2023-11-28T20:34:12.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.341+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdba00caf40 con 0x7fdba006dc50 2023-11-28T20:34:12.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.342+0000 7fdb96ffd640 1 -- 172.21.15.45:0/145246088 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdb90057c80 con 0x7fdba006dc50 2023-11-28T20:34:12.343 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:34:12.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdb8406ae40 msgr2=0x7fdb8406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdb8406ae40 0x7fdb8406d300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c006fd0 tx=0x7fdb9c008040 comp rx=0 tx=0).stop 2023-11-28T20:34:12.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006dc50 msgr2=0x7fdba01f5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006dc50 0x7fdba01f5f90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdb9000c430 tx=0x7fdb9000c8f0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 shutdown_connections 2023-11-28T20:34:12.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdb8406ae40 0x7fdb8406d300 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba00ce600 0x7fdba01f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdba006dc50 0x7fdba01f5f90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 --2- 172.21.15.45:0/145246088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdba006ac10 0x7fdba00c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 >> 172.21.15.45:0/145246088 conn(0x7fdba00ced60 msgr2=0x7fdba006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:12.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 shutdown_connections 2023-11-28T20:34:12.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.345+0000 7fdba82ad640 1 -- 172.21.15.45:0/145246088 wait complete. 2023-11-28T20:34:12.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:34:12.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.591+0000 7fdf7a848640 1 Processor -- start 2023-11-28T20:34:12.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.591+0000 7fdf7a848640 1 -- start start 2023-11-28T20:34:12.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.592+0000 7fdf7a848640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdf74157c10 0x7fdf7406da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.593+0000 7fdf7a848640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7406ca60 con 0x7fdf7415e4e0 2023-11-28T20:34:12.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.593+0000 7fdf7a848640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7415eb40 con 0x7fdf7406a500 2023-11-28T20:34:12.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.593+0000 7fdf7a848640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7415e930 con 0x7fdf74157c10 2023-11-28T20:34:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.593+0000 7fdf78de5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdf7415e4e0 0x7fdf740c9e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51120/0 (socket says 172.21.15.45:51120) 2023-11-28T20:34:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.593+0000 7fdf78de5640 1 -- 172.21.15.45:0/2059260795 learned_addr learned my addr 172.21.15.45:0/2059260795 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.593+0000 7fdf73fff640 1 --2- 172.21.15.45:0/2059260795 >> v2:172.21.15.190:3300/0 conn(0x7fdf74157c10 0x7fdf7406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.593+0000 7fdf72ffd640 1 -- 172.21.15.45:0/2059260795 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041200404 0 0) 0x7fdf7406ca60 con 0x7fdf7415e4e0 2023-11-28T20:34:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.594+0000 7fdf72ffd640 1 -- 172.21.15.45:0/2059260795 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c003520 con 0x7fdf7415e4e0 2023-11-28T20:34:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.594+0000 7fdf73fff640 1 -- 172.21.15.45:0/2059260795 >> v1:172.21.15.190:6789/0 conn(0x7fdf7406a500 legacy=0x7fdf7415ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.594+0000 7fdf72ffd640 1 -- 172.21.15.45:0/2059260795 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659212241 0 0) 0x7fdf5c003520 con 0x7fdf7415e4e0 2023-11-28T20:34:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.594+0000 7fdf73fff640 1 -- 172.21.15.45:0/2059260795 >> v1:172.21.15.45:6789/0 conn(0x7fdf7415e4e0 legacy=0x7fdf740c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.594+0000 7fdf73fff640 1 -- 172.21.15.45:0/2059260795 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf74157ff0 con 0x7fdf74157c10 2023-11-28T20:34:12.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.595+0000 7fdf73fff640 1 --2- 172.21.15.45:0/2059260795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf74157c10 0x7fdf7406da50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf640097a0 tx=0x7fdf6402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=444d7720e25b5d8d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:12.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.596+0000 7fdf72ffd640 1 -- 172.21.15.45:0/2059260795 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6402f050 con 0x7fdf74157c10 2023-11-28T20:34:12.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.596+0000 7fdf72ffd640 1 -- 172.21.15.45:0/2059260795 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf64031500 con 0x7fdf74157c10 2023-11-28T20:34:12.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.596+0000 7fdf72ffd640 1 -- 172.21.15.45:0/2059260795 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c0019f0 con 0x7fdf74157c10 2023-11-28T20:34:12.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.596+0000 7fdf7a848640 1 -- 172.21.15.45:0/2059260795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf74157c10 msgr2=0x7fdf7406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.596+0000 7fdf7a848640 1 --2- 172.21.15.45:0/2059260795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf74157c10 0x7fdf7406da50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf640097a0 tx=0x7fdf6402b820 comp rx=0 tx=0).stop 2023-11-28T20:34:12.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.596+0000 7fdf7a848640 1 -- 172.21.15.45:0/2059260795 shutdown_connections 2023-11-28T20:34:12.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.596+0000 7fdf7a848640 1 --2- 172.21.15.45:0/2059260795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf74157c10 0x7fdf7406da50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.596+0000 7fdf7a848640 1 -- 172.21.15.45:0/2059260795 >> 172.21.15.45:0/2059260795 conn(0x7fdf740ced60 msgr2=0x7fdf740cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:12.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.597+0000 7fdf7a848640 1 -- 172.21.15.45:0/2059260795 shutdown_connections 2023-11-28T20:34:12.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.597+0000 7fdf7a848640 1 -- 172.21.15.45:0/2059260795 wait complete. 2023-11-28T20:34:12.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.598+0000 7fdf7a848640 1 Processor -- start 2023-11-28T20:34:12.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.599+0000 7fdf7a848640 1 -- start start 2023-11-28T20:34:12.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.599+0000 7fdf7a848640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7406a500 0x7fdf74156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.599+0000 7fdf7a848640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf74157c10 0x7fdf740adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf7a848640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7415e4e0 0x7fdf740c2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf7a848640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740c88c0 con 0x7fdf74157c10 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf7a848640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74157460 con 0x7fdf7406a500 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf7a848640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740c8300 con 0x7fdf7415e4e0 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf74157c10 0x7fdf740adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf74157c10 0x7fdf740adaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56630/0 (socket says 172.21.15.45:56630) 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf737fe640 1 -- 172.21.15.45:0/333923914 learned_addr learned my addr 172.21.15.45:0/333923914 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf73fff640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7406a500 0x7fdf74156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf78de5640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7415e4e0 0x7fdf740c2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:12.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf737fe640 1 -- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7415e4e0 msgr2=0x7fdf740c2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf737fe640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7415e4e0 0x7fdf740c2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf737fe640 1 -- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7406a500 msgr2=0x7fdf74156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf737fe640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7406a500 0x7fdf74156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf737fe640 1 -- 172.21.15.45:0/333923914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf64009400 con 0x7fdf74157c10 2023-11-28T20:34:12.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.600+0000 7fdf73fff640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7406a500 0x7fdf74156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:12.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.601+0000 7fdf737fe640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf74157c10 0x7fdf740adaa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800c450 tx=0x7fdf6800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:12.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.601+0000 7fdf70ff9640 1 -- 172.21.15.45:0/333923914 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf680052d0 con 0x7fdf74157c10 2023-11-28T20:34:12.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.601+0000 7fdf70ff9640 1 -- 172.21.15.45:0/333923914 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf68005430 con 0x7fdf74157c10 2023-11-28T20:34:12.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.601+0000 7fdf70ff9640 1 -- 172.21.15.45:0/333923914 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68012870 con 0x7fdf74157c10 2023-11-28T20:34:12.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.601+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf740c85e0 con 0x7fdf74157c10 2023-11-28T20:34:12.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.601+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf740c3040 con 0x7fdf74157c10 2023-11-28T20:34:12.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.602+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf740ae2e0 con 0x7fdf74157c10 2023-11-28T20:34:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.604+0000 7fdf70ff9640 1 -- 172.21.15.45:0/333923914 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdf680129d0 con 0x7fdf74157c10 2023-11-28T20:34:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.605+0000 7fdf70ff9640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf6006ae40 0x7fdf6006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:12.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.605+0000 7fdf70ff9640 1 -- 172.21.15.45:0/333923914 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdf680894e0 con 0x7fdf74157c10 2023-11-28T20:34:12.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.605+0000 7fdf73fff640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf6006ae40 0x7fdf6006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:12.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.606+0000 7fdf73fff640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf6006ae40 0x7fdf6006d300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdf64035000 tx=0x7fdf64008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:12.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.609+0000 7fdf70ff9640 1 -- 172.21.15.45:0/333923914 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf68054200 con 0x7fdf74157c10 2023-11-28T20:34:12.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.806+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdf740c3a10 con 0x7fdf74157c10 2023-11-28T20:34:12.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.807+0000 7fdf70ff9640 1 -- 172.21.15.45:0/333923914 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdf740c3a10 con 0x7fdf74157c10 2023-11-28T20:34:12.809 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:12.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.810+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf6006ae40 msgr2=0x7fdf6006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.810+0000 7fdf7a848640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf6006ae40 0x7fdf6006d300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdf64035000 tx=0x7fdf64008030 comp rx=0 tx=0).stop 2023-11-28T20:34:12.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.810+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf74157c10 msgr2=0x7fdf740adaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:12.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.810+0000 7fdf7a848640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf74157c10 0x7fdf740adaa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800c450 tx=0x7fdf6800c910 comp rx=0 tx=0).stop 2023-11-28T20:34:12.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.811+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 shutdown_connections 2023-11-28T20:34:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.811+0000 7fdf7a848640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf6006ae40 0x7fdf6006d300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.811+0000 7fdf7a848640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7415e4e0 0x7fdf740c2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.811+0000 7fdf7a848640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf74157c10 0x7fdf740adaa0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.811+0000 7fdf7a848640 1 --2- 172.21.15.45:0/333923914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7406a500 0x7fdf74156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.811+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 >> 172.21.15.45:0/333923914 conn(0x7fdf740ced60 msgr2=0x7fdf7415a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.811+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 shutdown_connections 2023-11-28T20:34:12.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:12.811+0000 7fdf7a848640 1 -- 172.21.15.45:0/333923914 wait complete. 2023-11-28T20:34:12.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:34:13.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.055+0000 7f1efc9c9640 1 Processor -- start 2023-11-28T20:34:13.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.055+0000 7f1efc9c9640 1 -- start start 2023-11-28T20:34:13.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.057+0000 7f1efc9c9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1ef8160e70 0x7f1ef80cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:13.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.057+0000 7f1efc9c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef80ce7d0 con 0x7f1ef815d310 2023-11-28T20:34:13.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.057+0000 7f1efc9c9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef80d14d0 con 0x7f1ef80ce3a0 2023-11-28T20:34:13.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.057+0000 7f1efc9c9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ef81613c0 con 0x7f1ef8160e70 2023-11-28T20:34:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.057+0000 7f1ef6ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1ef8160e70 0x7f1ef80cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.057+0000 7f1ef77fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1ef815d310 0x7f1ef80b8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51128/0 (socket says 172.21.15.45:51128) 2023-11-28T20:34:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.057+0000 7f1ef77fe640 1 -- 172.21.15.45:0/3287739976 learned_addr learned my addr 172.21.15.45:0/3287739976 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.058+0000 7f1ef6ffd640 1 -- 172.21.15.45:0/3287739976 >> v1:172.21.15.190:6789/0 conn(0x7f1ef80ce3a0 legacy=0x7f1ef80d19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.058+0000 7f1ef6ffd640 1 -- 172.21.15.45:0/3287739976 >> v1:172.21.15.45:6789/0 conn(0x7f1ef815d310 legacy=0x7f1ef80b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:13.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.058+0000 7f1ef6ffd640 1 -- 172.21.15.45:0/3287739976 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef81616a0 con 0x7f1ef8160e70 2023-11-28T20:34:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.059+0000 7f1ef6ffd640 1 --2- 172.21.15.45:0/3287739976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef8160e70 0x7f1ef80cde60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0097a0 tx=0x7f1eec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142285457c2d6ce5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.059+0000 7f1ef67fc640 1 -- 172.21.15.45:0/3287739976 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec02e070 con 0x7f1ef8160e70 2023-11-28T20:34:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.059+0000 7f1ef67fc640 1 -- 172.21.15.45:0/3287739976 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eec0315a0 con 0x7f1ef8160e70 2023-11-28T20:34:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.059+0000 7f1ef67fc640 1 -- 172.21.15.45:0/3287739976 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec031930 con 0x7f1ef8160e70 2023-11-28T20:34:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.060+0000 7f1efc9c9640 1 -- 172.21.15.45:0/3287739976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef8160e70 msgr2=0x7f1ef80cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.060+0000 7f1efc9c9640 1 --2- 172.21.15.45:0/3287739976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef8160e70 0x7f1ef80cde60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0097a0 tx=0x7f1eec02b840 comp rx=0 tx=0).stop 2023-11-28T20:34:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.060+0000 7f1efc9c9640 1 -- 172.21.15.45:0/3287739976 shutdown_connections 2023-11-28T20:34:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.060+0000 7f1efc9c9640 1 --2- 172.21.15.45:0/3287739976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef8160e70 0x7f1ef80cde60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.060+0000 7f1efc9c9640 1 -- 172.21.15.45:0/3287739976 >> 172.21.15.45:0/3287739976 conn(0x7f1ef806a500 msgr2=0x7f1ef806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.060+0000 7f1efc9c9640 1 -- 172.21.15.45:0/3287739976 shutdown_connections 2023-11-28T20:34:13.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.060+0000 7f1efc9c9640 1 -- 172.21.15.45:0/3287739976 wait complete. 2023-11-28T20:34:13.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.062+0000 7f1efc9c9640 1 Processor -- start 2023-11-28T20:34:13.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.062+0000 7f1efc9c9640 1 -- start start 2023-11-28T20:34:13.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1efc9c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ef80ce3a0 0x7f1ef8151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:13.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1efc9c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef815d310 0x7f1ef80b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:13.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1efc9c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ef8160e70 0x7f1ef80b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:13.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1efc9c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef81e6d10 con 0x7f1ef815d310 2023-11-28T20:34:13.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1efc9c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef81e6ff0 con 0x7f1ef8160e70 2023-11-28T20:34:13.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1efc9c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef81e72d0 con 0x7f1ef80ce3a0 2023-11-28T20:34:13.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1ef6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef815d310 0x7f1ef80b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:13.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1ef6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef815d310 0x7f1ef80b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56648/0 (socket says 172.21.15.45:56648) 2023-11-28T20:34:13.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1ef6ffd640 1 -- 172.21.15.45:0/602361230 learned_addr learned my addr 172.21.15.45:0/602361230 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:13.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.063+0000 7f1ef77fe640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ef80ce3a0 0x7f1ef8151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:13.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1ef7fff640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ef8160e70 0x7f1ef80b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:13.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1ef6ffd640 1 -- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ef80ce3a0 msgr2=0x7f1ef8151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:13.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1ef6ffd640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ef80ce3a0 0x7f1ef8151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1ef6ffd640 1 -- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ef8160e70 msgr2=0x7f1ef80b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1ef6ffd640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ef8160e70 0x7f1ef80b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1ef6ffd640 1 -- 172.21.15.45:0/602361230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec009400 con 0x7f1ef815d310 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1ef7fff640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ef8160e70 0x7f1ef80b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1ef6ffd640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef815d310 0x7f1ef80b3b40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1eec034040 tx=0x7f1eec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1edffff640 1 -- 172.21.15.45:0/602361230 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec032900 con 0x7f1ef815d310 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1edffff640 1 -- 172.21.15.45:0/602361230 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eec032a80 con 0x7f1ef815d310 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.064+0000 7f1edffff640 1 -- 172.21.15.45:0/602361230 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec032e30 con 0x7f1ef815d310 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.065+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef81e75b0 con 0x7f1ef815d310 2023-11-28T20:34:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.065+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef80cbd10 con 0x7f1ef815d310 2023-11-28T20:34:13.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.066+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef81e7ad0 con 0x7f1ef815d310 2023-11-28T20:34:13.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.067+0000 7f1edffff640 1 -- 172.21.15.45:0/602361230 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1eec03b3f0 con 0x7f1ef815d310 2023-11-28T20:34:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.067+0000 7f1edffff640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1ee406ae40 0x7f1ee406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.068+0000 7f1edffff640 1 -- 172.21.15.45:0/602361230 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1eec0ab910 con 0x7f1ef815d310 2023-11-28T20:34:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.068+0000 7f1ef77fe640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1ee406ae40 0x7f1ee406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:13.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.069+0000 7f1ef77fe640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1ee406ae40 0x7f1ee406d300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8005c00 tx=0x7f1ee8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:13.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.072+0000 7f1edffff640 1 -- 172.21.15.45:0/602361230 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ef81e7ad0 con 0x7f1ef815d310 2023-11-28T20:34:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.282+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1ef81e7ad0 con 0x7f1ef815d310 2023-11-28T20:34:13.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.283+0000 7f1edffff640 1 -- 172.21.15.45:0/602361230 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1ef81e7ad0 con 0x7f1ef815d310 2023-11-28T20:34:13.284 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2023-11-28T20:34:13.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1ee406ae40 msgr2=0x7f1ee406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:13.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1ee406ae40 0x7f1ee406d300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8005c00 tx=0x7f1ee8009040 comp rx=0 tx=0).stop 2023-11-28T20:34:13.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef815d310 msgr2=0x7f1ef80b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:13.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef815d310 0x7f1ef80b3b40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1eec034040 tx=0x7f1eec0046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:13.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 shutdown_connections 2023-11-28T20:34:13.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1ee406ae40 0x7f1ee406d300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:13.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ef8160e70 0x7f1ef80b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:13.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ef815d310 0x7f1ef80b3b40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:13.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 --2- 172.21.15.45:0/602361230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ef80ce3a0 0x7f1ef8151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:13.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 >> 172.21.15.45:0/602361230 conn(0x7f1ef806a500 msgr2=0x7f1ef80c6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:13.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 shutdown_connections 2023-11-28T20:34:13.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:13.286+0000 7f1efc9c9640 1 -- 172.21.15.45:0/602361230 wait complete. 2023-11-28T20:34:13.307 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi045.front.sepia.ceph.com... 2023-11-28T20:34:13.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:13.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-11-28T20:34:13.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi045.stderr:ceph-fuse[94005]: fuse finished with error 0 and tester_r 0 2023-11-28T20:34:13.401 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:34:14.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:34:14.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:34:19.503 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:34:19.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:19.503 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-11-28T20:34:19.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:19.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.814+0000 7f93a5747640 1 Processor -- start 2023-11-28T20:34:19.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.814+0000 7f93a5747640 1 -- start start 2023-11-28T20:34:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.816+0000 7f93a5747640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f93a00b24a0 0x7f93a00ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.816+0000 7f93a5747640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93a00ae590 0x7f93a00ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.816+0000 7f93a5747640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a00abf10 con 0x7f93a014b090 2023-11-28T20:34:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.816+0000 7f93a5747640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a006db90 con 0x7f93a00ae590 2023-11-28T20:34:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.816+0000 7f93a5747640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a006d2f0 con 0x7f93a00b24a0 2023-11-28T20:34:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.816+0000 7f93a4f46640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93a00ae590 0x7f93a00ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.816+0000 7f93a4f46640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93a00ae590 0x7f93a00ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58300/0 (socket says 172.21.15.45:58300) 2023-11-28T20:34:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.817+0000 7f93a4f46640 1 -- 172.21.15.45:0/993155488 learned_addr learned my addr 172.21.15.45:0/993155488 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.817+0000 7f939f7fe640 1 --2- 172.21.15.45:0/993155488 >> v2:172.21.15.190:3300/0 conn(0x7f93a00b24a0 0x7f93a00ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:19.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.817+0000 7f93a4f46640 1 -- 172.21.15.45:0/993155488 >> v2:172.21.15.190:3300/0 conn(0x7f93a00b24a0 msgr2=0x7f93a00ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:19.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.817+0000 7f93a4f46640 1 --2- 172.21.15.45:0/993155488 >> v2:172.21.15.190:3300/0 conn(0x7f93a00b24a0 0x7f93a00ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:19.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.817+0000 7f93a4f46640 1 -- 172.21.15.45:0/993155488 >> v1:172.21.15.190:6789/0 conn(0x7f93a014b090 legacy=0x7f93a00b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:19.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.817+0000 7f93a4f46640 1 -- 172.21.15.45:0/993155488 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a006d5d0 con 0x7f93a00ae590 2023-11-28T20:34:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.819+0000 7f93a4f46640 1 --2- 172.21.15.45:0/993155488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 0x7f93a00ac8e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93a00ced90 tx=0x7f939802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba4a2da9cef3d71 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.819+0000 7f939effd640 1 -- 172.21.15.45:0/993155488 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939802cb20 con 0x7f93a00ae590 2023-11-28T20:34:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.819+0000 7f939effd640 1 -- 172.21.15.45:0/993155488 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939802cc80 con 0x7f93a00ae590 2023-11-28T20:34:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.819+0000 7f939effd640 1 -- 172.21.15.45:0/993155488 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93980343f0 con 0x7f93a00ae590 2023-11-28T20:34:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.820+0000 7f93a5747640 1 -- 172.21.15.45:0/993155488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 msgr2=0x7f93a00ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.820+0000 7f93a5747640 1 --2- 172.21.15.45:0/993155488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 0x7f93a00ac8e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93a00ced90 tx=0x7f939802bf80 comp rx=0 tx=0).stop 2023-11-28T20:34:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.820+0000 7f93a5747640 1 -- 172.21.15.45:0/993155488 shutdown_connections 2023-11-28T20:34:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.820+0000 7f93a5747640 1 --2- 172.21.15.45:0/993155488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 0x7f93a00ac8e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:19.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.820+0000 7f93a5747640 1 --2- 172.21.15.45:0/993155488 >> v2:172.21.15.190:3300/0 conn(0x7f93a00b24a0 0x7f93a00ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:19.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.820+0000 7f93a5747640 1 -- 172.21.15.45:0/993155488 >> 172.21.15.45:0/993155488 conn(0x7f93a006ab30 msgr2=0x7f93a00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:19.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.820+0000 7f93a5747640 1 -- 172.21.15.45:0/993155488 shutdown_connections 2023-11-28T20:34:19.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.820+0000 7f93a5747640 1 -- 172.21.15.45:0/993155488 wait complete. 2023-11-28T20:34:19.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.822+0000 7f93a5747640 1 Processor -- start 2023-11-28T20:34:19.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.822+0000 7f93a5747640 1 -- start start 2023-11-28T20:34:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.823+0000 7f93a5747640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 0x7f93a0145810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.823+0000 7f93a5747640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93a00b24a0 0x7f93a0160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.823+0000 7f93a5747640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93a014b090 0x7f93a01eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.823+0000 7f93a5747640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0146bc0 con 0x7f93a00ae590 2023-11-28T20:34:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.823+0000 7f93a5747640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0146040 con 0x7f93a014b090 2023-11-28T20:34:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.823+0000 7f93a5747640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0146320 con 0x7f93a00b24a0 2023-11-28T20:34:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 0x7f93a0145810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 0x7f93a0145810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58310/0 (socket says 172.21.15.45:58310) 2023-11-28T20:34:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939ffff640 1 -- 172.21.15.45:0/3697921421 learned_addr learned my addr 172.21.15.45:0/3697921421 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f93a4f46640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93a014b090 0x7f93a01eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939f7fe640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93a00b24a0 0x7f93a0160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939ffff640 1 -- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93a00b24a0 msgr2=0x7f93a0160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939ffff640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93a00b24a0 0x7f93a0160440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939ffff640 1 -- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93a014b090 msgr2=0x7f93a01eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939ffff640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93a014b090 0x7f93a01eb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939ffff640 1 -- 172.21.15.45:0/3697921421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398009b30 con 0x7f93a00ae590 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f93a4f46640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93a014b090 0x7f93a01eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.824+0000 7f939ffff640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 0x7f93a0145810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f939000c400 tx=0x7f939000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.825+0000 7f939cff9640 1 -- 172.21.15.45:0/3697921421 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93900052d0 con 0x7f93a00ae590 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.825+0000 7f939cff9640 1 -- 172.21.15.45:0/3697921421 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9390005430 con 0x7f93a00ae590 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.825+0000 7f939cff9640 1 -- 172.21.15.45:0/3697921421 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390003830 con 0x7f93a00ae590 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.825+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a0146600 con 0x7f93a00ae590 2023-11-28T20:34:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.825+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a0145d50 con 0x7f93a00ae590 2023-11-28T20:34:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.826+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a00ae3b0 con 0x7f93a00ae590 2023-11-28T20:34:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.827+0000 7f939cff9640 1 -- 172.21.15.45:0/3697921421 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9390003990 con 0x7f93a00ae590 2023-11-28T20:34:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.828+0000 7f939cff9640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f938c06ae90 0x7f938c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.828+0000 7f939cff9640 1 -- 172.21.15.45:0/3697921421 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9390088f80 con 0x7f93a00ae590 2023-11-28T20:34:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.829+0000 7f939f7fe640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f938c06ae90 0x7f938c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.830+0000 7f939f7fe640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f938c06ae90 0x7f938c06d350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f93a0142d20 tx=0x7f9394008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:19.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:19.833+0000 7f939cff9640 1 -- 172.21.15.45:0/3697921421 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9390053c90 con 0x7f93a00ae590 2023-11-28T20:34:20.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.028+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93a009d0f0 con 0x7f93a00ae590 2023-11-28T20:34:20.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.030+0000 7f939cff9640 1 -- 172.21.15.45:0/3697921421 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7f9390017020 con 0x7f93a00ae590 2023-11-28T20:34:20.032 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:20.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.032+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f938c06ae90 msgr2=0x7f938c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.032+0000 7f93a5747640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f938c06ae90 0x7f938c06d350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f93a0142d20 tx=0x7f9394008040 comp rx=0 tx=0).stop 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.032+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 msgr2=0x7f93a0145810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.032+0000 7f93a5747640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 0x7f93a0145810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f939000c400 tx=0x7f939000c8c0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.032+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 shutdown_connections 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.032+0000 7f93a5747640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f938c06ae90 0x7f938c06d350 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.032+0000 7f93a5747640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93a014b090 0x7f93a01eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.033+0000 7f93a5747640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93a00b24a0 0x7f93a0160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.033+0000 7f93a5747640 1 --2- 172.21.15.45:0/3697921421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93a00ae590 0x7f93a0145810 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.033+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 >> 172.21.15.45:0/3697921421 conn(0x7f93a006ab30 msgr2=0x7f93a00aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.033+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 shutdown_connections 2023-11-28T20:34:20.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.033+0000 7f93a5747640 1 -- 172.21.15.45:0/3697921421 wait complete. 2023-11-28T20:34:20.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.286+0000 7fdfe19f4640 1 Processor -- start 2023-11-28T20:34:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.286+0000 7fdfe19f4640 1 -- start start 2023-11-28T20:34:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.288+0000 7fdfe19f4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdfdc155780 0x7fdfdc0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.288+0000 7fdfe19f4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfdc15c3c0 con 0x7fdfdc0ae540 2023-11-28T20:34:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.288+0000 7fdfe19f4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfdc155b60 con 0x7fdfdc0ce800 2023-11-28T20:34:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.288+0000 7fdfe19f4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc15ba90 con 0x7fdfdc155780 2023-11-28T20:34:20.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.288+0000 7fdfdbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdfdc0ae540 0x7fdfdc0ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41462/0 (socket says 172.21.15.45:41462) 2023-11-28T20:34:20.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.288+0000 7fdfdbfff640 1 -- 172.21.15.45:0/2405692848 learned_addr learned my addr 172.21.15.45:0/2405692848 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:20.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.288+0000 7fdfe09f2640 1 --2- 172.21.15.45:0/2405692848 >> v2:172.21.15.190:3300/0 conn(0x7fdfdc155780 0x7fdfdc0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586505495 0 0) 0x7fdfdc15c3c0 con 0x7fdfdc0ae540 2023-11-28T20:34:20.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfcc003520 con 0x7fdfdc0ae540 2023-11-28T20:34:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4233906524 0 0) 0x7fdfcc003520 con 0x7fdfdc0ae540 2023-11-28T20:34:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 >> v2:172.21.15.190:3300/0 conn(0x7fdfdc155780 msgr2=0x7fdfdc0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 --2- 172.21.15.45:0/2405692848 >> v2:172.21.15.190:3300/0 conn(0x7fdfdc155780 0x7fdfdc0abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 >> v1:172.21.15.190:6789/0 conn(0x7fdfdc0ce800 legacy=0x7fdfdc0b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc15bd70 con 0x7fdfdc0ae540 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fdfc8001670 con 0x7fdfdc0ae540 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 >> v1:172.21.15.45:6789/0 conn(0x7fdfdc0ae540 legacy=0x7fdfdc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfcc0039c0 0x7fdfcc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfcc0047c0 0x7fdfcc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfcc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc15ba90 con 0x7fdfcc005180 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc00c080 con 0x7fdfcc0039c0 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.289+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc00c240 con 0x7fdfcc0047c0 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.290+0000 7fdfe11f3640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfcc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.290+0000 7fdfe09f2640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfcc0039c0 0x7fdfcc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.290+0000 7fdfdbfff640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfcc0047c0 0x7fdfcc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.290+0000 7fdfe11f3640 1 -- 172.21.15.45:0/2405692848 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfcc0047c0 msgr2=0x7fdfcc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:20.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.290+0000 7fdfe11f3640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfcc0047c0 0x7fdfcc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.290+0000 7fdfe11f3640 1 -- 172.21.15.45:0/2405692848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfcc0039c0 msgr2=0x7fdfcc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:20.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.290+0000 7fdfe11f3640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfcc0039c0 0x7fdfcc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.290+0000 7fdfe11f3640 1 -- 172.21.15.45:0/2405692848 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc003520 con 0x7fdfcc005180 2023-11-28T20:34:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.291+0000 7fdfe11f3640 1 -- 172.21.15.45:0/2405692848 reap_dead start 2023-11-28T20:34:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.291+0000 7fdfe11f3640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfcc00b980 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4001e50 tx=0x7fdfcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d06c146b908216 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.291+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4003b70 con 0x7fdfcc005180 2023-11-28T20:34:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.291+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd4003d10 con 0x7fdfcc005180 2023-11-28T20:34:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.291+0000 7fdfdb7fe640 1 -- 172.21.15.45:0/2405692848 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4005310 con 0x7fdfcc005180 2023-11-28T20:34:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.292+0000 7fdfe19f4640 1 -- 172.21.15.45:0/2405692848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 msgr2=0x7fdfcc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.292+0000 7fdfe19f4640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfcc00b980 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4001e50 tx=0x7fdfcc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.292+0000 7fdfe19f4640 1 -- 172.21.15.45:0/2405692848 shutdown_connections 2023-11-28T20:34:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.292+0000 7fdfe19f4640 1 --2- 172.21.15.45:0/2405692848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfcc00b980 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.292+0000 7fdfe19f4640 1 -- 172.21.15.45:0/2405692848 >> 172.21.15.45:0/2405692848 conn(0x7fdfdc06ab30 msgr2=0x7fdfdc0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:20.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.292+0000 7fdfe19f4640 1 -- 172.21.15.45:0/2405692848 shutdown_connections 2023-11-28T20:34:20.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.292+0000 7fdfe19f4640 1 -- 172.21.15.45:0/2405692848 wait complete. 2023-11-28T20:34:20.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.294+0000 7fdfe19f4640 1 Processor -- start 2023-11-28T20:34:20.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.294+0000 7fdfe19f4640 1 -- start start 2023-11-28T20:34:20.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.295+0000 7fdfe19f4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfdc14acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.295+0000 7fdfe19f4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfdc070e00 0x7fdfdc09bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.295+0000 7fdfe19f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfdc1f6b50 0x7fdfdc15a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.295+0000 7fdfe19f4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc14b1f0 con 0x7fdfcc005180 2023-11-28T20:34:20.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.295+0000 7fdfe19f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc0718c0 con 0x7fdfdc1f6b50 2023-11-28T20:34:20.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.295+0000 7fdfe19f4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc071300 con 0x7fdfdc070e00 2023-11-28T20:34:20.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.295+0000 7fdfe09f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfdc14acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfe09f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfdc14acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58334/0 (socket says 172.21.15.45:58334) 2023-11-28T20:34:20.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfe09f2640 1 -- 172.21.15.45:0/437374887 learned_addr learned my addr 172.21.15.45:0/437374887 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:20.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfdbfff640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfdc070e00 0x7fdfdc09bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfe11f3640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfdc1f6b50 0x7fdfdc15a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfe09f2640 1 -- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfdc070e00 msgr2=0x7fdfdc09bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfe09f2640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfdc070e00 0x7fdfdc09bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfe09f2640 1 -- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfdc1f6b50 msgr2=0x7fdfdc15a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfe09f2640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfdc1f6b50 0x7fdfdc15a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfe09f2640 1 -- 172.21.15.45:0/437374887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc0715e0 con 0x7fdfcc005180 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfdbfff640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfdc070e00 0x7fdfdc09bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.296+0000 7fdfe09f2640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfdc14acb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0011540 tx=0x7fdfd0011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.297+0000 7fdfd97fa640 1 -- 172.21.15.45:0/437374887 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0015930 con 0x7fdfcc005180 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.297+0000 7fdfd97fa640 1 -- 172.21.15.45:0/437374887 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd0015a90 con 0x7fdfcc005180 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.297+0000 7fdfd97fa640 1 -- 172.21.15.45:0/437374887 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0015e60 con 0x7fdfcc005180 2023-11-28T20:34:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.297+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc1f6350 con 0x7fdfcc005180 2023-11-28T20:34:20.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.297+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc09c4d0 con 0x7fdfcc005180 2023-11-28T20:34:20.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.298+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc09c850 con 0x7fdfcc005180 2023-11-28T20:34:20.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.299+0000 7fdfd97fa640 1 -- 172.21.15.45:0/437374887 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdfd000f460 con 0x7fdfcc005180 2023-11-28T20:34:20.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.300+0000 7fdfd97fa640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdfc006ae40 0x7fdfc006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.300+0000 7fdfd97fa640 1 -- 172.21.15.45:0/437374887 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdfd008a1a0 con 0x7fdfcc005180 2023-11-28T20:34:20.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.301+0000 7fdfdbfff640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdfc006ae40 0x7fdfc006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.302+0000 7fdfdbfff640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdfc006ae40 0x7fdfc006d300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc09c820 tx=0x7fdfc8000f70 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:20.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.305+0000 7fdfd97fa640 1 -- 172.21.15.45:0/437374887 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdfd0054ec0 con 0x7fdfcc005180 2023-11-28T20:34:20.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.502+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfdc04e920 con 0x7fdfcc005180 2023-11-28T20:34:20.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.505+0000 7fdfd97fa640 1 -- 172.21.15.45:0/437374887 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7fdfd00584e0 con 0x7fdfcc005180 2023-11-28T20:34:20.506 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:20.507 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:20.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.507+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdfc006ae40 msgr2=0x7fdfc006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.507+0000 7fdfe19f4640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdfc006ae40 0x7fdfc006d300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc09c820 tx=0x7fdfc8000f70 comp rx=0 tx=0).stop 2023-11-28T20:34:20.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.507+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 msgr2=0x7fdfdc14acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.507+0000 7fdfe19f4640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfdc14acb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0011540 tx=0x7fdfd0011a00 comp rx=0 tx=0).stop 2023-11-28T20:34:20.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.508+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 shutdown_connections 2023-11-28T20:34:20.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.508+0000 7fdfe19f4640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdfc006ae40 0x7fdfc006d300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.508+0000 7fdfe19f4640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfdc1f6b50 0x7fdfdc15a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.508+0000 7fdfe19f4640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdfdc070e00 0x7fdfdc09bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.508+0000 7fdfe19f4640 1 --2- 172.21.15.45:0/437374887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfcc005180 0x7fdfdc14acb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.508+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 >> 172.21.15.45:0/437374887 conn(0x7fdfdc06ab30 msgr2=0x7fdfdc06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:20.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.508+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 shutdown_connections 2023-11-28T20:34:20.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.508+0000 7fdfe19f4640 1 -- 172.21.15.45:0/437374887 wait complete. 2023-11-28T20:34:20.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T20:34:20.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.757+0000 7f59166a3640 1 Processor -- start 2023-11-28T20:34:20.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.757+0000 7f59166a3640 1 -- start start 2023-11-28T20:34:20.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.759+0000 7f59166a3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 0x7f5910099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.759+0000 7f59166a3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f591015c250 0x7f59100cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.759+0000 7f59166a3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59100cf230 con 0x7f591015b6c0 2023-11-28T20:34:20.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.759+0000 7f59166a3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59100c7c70 con 0x7f591015c250 2023-11-28T20:34:20.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.759+0000 7f59166a3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59100cf0a0 con 0x7f59100c7890 2023-11-28T20:34:20.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.759+0000 7f5914ea0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f591015b6c0 0x7f591015bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41476/0 (socket says 172.21.15.45:41476) 2023-11-28T20:34:20.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.759+0000 7f5914ea0640 1 -- 172.21.15.45:0/3195862354 learned_addr learned my addr 172.21.15.45:0/3195862354 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:20.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.759+0000 7f59156a1640 1 --2- 172.21.15.45:0/3195862354 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 0x7f5910099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.759+0000 7f5915ea2640 1 --2- 172.21.15.45:0/3195862354 >> v2:172.21.15.190:3300/0 conn(0x7f591015c250 0x7f59100cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.760+0000 7f590ffff640 1 -- 172.21.15.45:0/3195862354 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293440200 0 0) 0x7f59100cf230 con 0x7f591015b6c0 2023-11-28T20:34:20.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.760+0000 7f590ffff640 1 -- 172.21.15.45:0/3195862354 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5900003520 con 0x7f591015b6c0 2023-11-28T20:34:20.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.760+0000 7f5915ea2640 1 -- 172.21.15.45:0/3195862354 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 msgr2=0x7f5910099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:20.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.760+0000 7f5915ea2640 1 --2- 172.21.15.45:0/3195862354 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 0x7f5910099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.760+0000 7f5915ea2640 1 -- 172.21.15.45:0/3195862354 >> v1:172.21.15.45:6789/0 conn(0x7f591015b6c0 legacy=0x7f591015bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.760+0000 7f5915ea2640 1 -- 172.21.15.45:0/3195862354 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59100d1d80 con 0x7f591015c250 2023-11-28T20:34:20.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.760+0000 7f59156a1640 1 --2- 172.21.15.45:0/3195862354 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 0x7f5910099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:20.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.762+0000 7f5915ea2640 1 --2- 172.21.15.45:0/3195862354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591015c250 0x7f59100cf6c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59080097a0 tx=0x7f590802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=269ddc817178acd8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.762+0000 7f590ffff640 1 -- 172.21.15.45:0/3195862354 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908031780 con 0x7f591015c250 2023-11-28T20:34:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.762+0000 7f590ffff640 1 -- 172.21.15.45:0/3195862354 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59080318e0 con 0x7f591015c250 2023-11-28T20:34:20.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.762+0000 7f590ffff640 1 -- 172.21.15.45:0/3195862354 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908031c70 con 0x7f591015c250 2023-11-28T20:34:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.763+0000 7f59166a3640 1 -- 172.21.15.45:0/3195862354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591015c250 msgr2=0x7f59100cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.763+0000 7f59166a3640 1 --2- 172.21.15.45:0/3195862354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591015c250 0x7f59100cf6c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59080097a0 tx=0x7f590802b820 comp rx=0 tx=0).stop 2023-11-28T20:34:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.763+0000 7f59166a3640 1 -- 172.21.15.45:0/3195862354 shutdown_connections 2023-11-28T20:34:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.763+0000 7f59166a3640 1 --2- 172.21.15.45:0/3195862354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591015c250 0x7f59100cf6c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.763+0000 7f59166a3640 1 --2- 172.21.15.45:0/3195862354 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 0x7f5910099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.763+0000 7f59166a3640 1 -- 172.21.15.45:0/3195862354 >> 172.21.15.45:0/3195862354 conn(0x7f59100d0f10 msgr2=0x7f59100d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:20.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.763+0000 7f59166a3640 1 -- 172.21.15.45:0/3195862354 shutdown_connections 2023-11-28T20:34:20.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.763+0000 7f59166a3640 1 -- 172.21.15.45:0/3195862354 wait complete. 2023-11-28T20:34:20.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.765+0000 7f59166a3640 1 Processor -- start 2023-11-28T20:34:20.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.765+0000 7f59166a3640 1 -- start start 2023-11-28T20:34:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f59166a3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 0x7f5910145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f59166a3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591015b6c0 0x7f5910156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f59166a3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f591015c250 0x7f59101551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f59166a3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5910158060 con 0x7f591015c250 2023-11-28T20:34:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f59166a3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5910157450 con 0x7f591015b6c0 2023-11-28T20:34:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f59166a3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5910157730 con 0x7f59100c7890 2023-11-28T20:34:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f5915ea2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f591015c250 0x7f59101551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f5915ea2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f591015c250 0x7f59101551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58342/0 (socket says 172.21.15.45:58342) 2023-11-28T20:34:20.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f5915ea2640 1 -- 172.21.15.45:0/2424711159 learned_addr learned my addr 172.21.15.45:0/2424711159 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:20.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f59156a1640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 0x7f5910145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.767+0000 7f5914ea0640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591015b6c0 0x7f5910156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f5915ea2640 1 -- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 msgr2=0x7f5910145840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f5915ea2640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 0x7f5910145840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f5915ea2640 1 -- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591015b6c0 msgr2=0x7f5910156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:20.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f5915ea2640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591015b6c0 0x7f5910156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:20.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f5915ea2640 1 -- 172.21.15.45:0/2424711159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5904009a70 con 0x7f591015c250 2023-11-28T20:34:20.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f5915ea2640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f591015c250 0x7f59101551e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f59080097a0 tx=0x7f59080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:20.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f590dffb640 1 -- 172.21.15.45:0/2424711159 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908032820 con 0x7f591015c250 2023-11-28T20:34:20.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f590dffb640 1 -- 172.21.15.45:0/2424711159 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5908032980 con 0x7f591015c250 2023-11-28T20:34:20.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f590dffb640 1 -- 172.21.15.45:0/2424711159 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908032d30 con 0x7f591015c250 2023-11-28T20:34:20.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.768+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5908009400 con 0x7f591015c250 2023-11-28T20:34:20.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.769+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5910157cf0 con 0x7f591015c250 2023-11-28T20:34:20.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.770+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591009a2e0 con 0x7f591015c250 2023-11-28T20:34:20.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.771+0000 7f590dffb640 1 -- 172.21.15.45:0/2424711159 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f590804d3f0 con 0x7f591015c250 2023-11-28T20:34:20.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.771+0000 7f590dffb640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f58f406ae40 0x7f58f406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:20.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.772+0000 7f590dffb640 1 -- 172.21.15.45:0/2424711159 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f59080ab460 con 0x7f591015c250 2023-11-28T20:34:20.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.772+0000 7f59156a1640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f58f406ae40 0x7f58f406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:20.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.773+0000 7f59156a1640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f58f406ae40 0x7f58f406d300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5910099ef0 tx=0x7f590400b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:20.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:20.776+0000 7f590dffb640 1 -- 172.21.15.45:0/2424711159 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f591009a2e0 con 0x7f591015c250 2023-11-28T20:34:21.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:21.009+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f591009a2e0 con 0x7f591015c250 2023-11-28T20:34:21.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:21.995+0000 7f590dffb640 1 -- 172.21.15.45:0/2424711159 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f591009a2e0 con 0x7f591015c250 2023-11-28T20:34:22.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:21.999+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f58f406ae40 msgr2=0x7f58f406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:22.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f58f406ae40 0x7f58f406d300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5910099ef0 tx=0x7f590400b040 comp rx=0 tx=0).stop 2023-11-28T20:34:22.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f591015c250 msgr2=0x7f59101551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f591015c250 0x7f59101551e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f59080097a0 tx=0x7f59080046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 shutdown_connections 2023-11-28T20:34:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f58f406ae40 0x7f58f406d300 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f591015c250 0x7f59101551e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591015b6c0 0x7f5910156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 --2- 172.21.15.45:0/2424711159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59100c7890 0x7f5910145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 >> 172.21.15.45:0/2424711159 conn(0x7f59100d0f10 msgr2=0x7f59100b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 shutdown_connections 2023-11-28T20:34:22.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.000+0000 7f59166a3640 1 -- 172.21.15.45:0/2424711159 wait complete. 2023-11-28T20:34:22.027 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2023-11-28T20:34:22.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T20:34:22.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.260+0000 7f8668aad640 1 Processor -- start 2023-11-28T20:34:22.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.260+0000 7f8668aad640 1 -- start start 2023-11-28T20:34:22.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.261+0000 7f8668aad640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f86640d0040 0x7f86640d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:22.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.261+0000 7f8668aad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86640b1d40 0x7f86640d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:22.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.261+0000 7f8668aad640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86640b90a0 con 0x7f86640b2290 2023-11-28T20:34:22.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.262+0000 7f8668aad640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86640d1920 con 0x7f86640d0040 2023-11-28T20:34:22.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.262+0000 7f8668aad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86640d2dc0 con 0x7f86640b1d40 2023-11-28T20:34:22.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.262+0000 7f8662ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f86640b2290 0x7f86640d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41478/0 (socket says 172.21.15.45:41478) 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.262+0000 7f8662ffd640 1 -- 172.21.15.45:0/2698002840 learned_addr learned my addr 172.21.15.45:0/2698002840 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.262+0000 7f8663fff640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86640b1d40 0x7f86640d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.262+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919186539 0 0) 0x7f86640b90a0 con 0x7f86640b2290 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.262+0000 7f86637fe640 1 --2- 172.21.15.45:0/2698002840 >> v2:172.21.15.190:3300/0 conn(0x7f86640d0040 0x7f86640d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.262+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864c003520 con 0x7f86640b2290 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 100038452 0 0) 0x7f864c003520 con 0x7f86640b2290 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86640b1d40 msgr2=0x7f86640d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86640b1d40 0x7f86640d3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 >> v2:172.21.15.190:3300/0 conn(0x7f86640d0040 msgr2=0x7f86640d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 --2- 172.21.15.45:0/2698002840 >> v2:172.21.15.190:3300/0 conn(0x7f86640d0040 0x7f86640d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86640d0960 con 0x7f86640b2290 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f8658001670 con 0x7f86640b2290 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86637fe640 1 --2- 172.21.15.45:0/2698002840 >> v2:172.21.15.190:3300/0 conn(0x7f86640d0040 0x7f86640d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 >> v1:172.21.15.45:6789/0 conn(0x7f86640b2290 legacy=0x7f86640d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f864c003970 0x7f864c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f864c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86640b1d40 0x7f864c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86640d1920 con 0x7f864c0047c0 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86640d2dc0 con 0x7f864c003970 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c00ba60 con 0x7f86640b1d40 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f8662ffd640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f864c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.263+0000 7f86637fe640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f864c003970 0x7f864c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:22.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.264+0000 7f8663fff640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86640b1d40 0x7f864c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:22.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.264+0000 7f8662ffd640 1 -- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86640b1d40 msgr2=0x7f864c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:22.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.264+0000 7f8662ffd640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86640b1d40 0x7f864c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.264+0000 7f8662ffd640 1 -- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f864c003970 msgr2=0x7f864c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:22.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.264+0000 7f8662ffd640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f864c003970 0x7f864c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.264+0000 7f8662ffd640 1 -- 172.21.15.45:0/2698002840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c003520 con 0x7f864c0047c0 2023-11-28T20:34:22.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.265+0000 7f8662ffd640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f864c004c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f865800c240 tx=0x7f864c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17de4a8326632ea6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:22.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.265+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865802cb00 con 0x7f864c0047c0 2023-11-28T20:34:22.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.265+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865802cc60 con 0x7f864c0047c0 2023-11-28T20:34:22.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.265+0000 7f86627fc640 1 -- 172.21.15.45:0/2698002840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86580058d0 con 0x7f864c0047c0 2023-11-28T20:34:22.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.266+0000 7f8668aad640 1 -- 172.21.15.45:0/2698002840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 msgr2=0x7f864c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:22.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.266+0000 7f8668aad640 1 --2- 172.21.15.45:0/2698002840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f864c004c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f865800c240 tx=0x7f864c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.266+0000 7f8663fff640 1 -- 172.21.15.45:0/2698002840 reap_dead start 2023-11-28T20:34:22.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.266+0000 7f8668aad640 1 -- 172.21.15.45:0/2698002840 shutdown_connections 2023-11-28T20:34:22.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.266+0000 7f8668aad640 1 -- 172.21.15.45:0/2698002840 >> 172.21.15.45:0/2698002840 conn(0x7f8664070f40 msgr2=0x7f8664071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:22.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.266+0000 7f8668aad640 1 -- 172.21.15.45:0/2698002840 shutdown_connections 2023-11-28T20:34:22.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.266+0000 7f8668aad640 1 -- 172.21.15.45:0/2698002840 wait complete. 2023-11-28T20:34:22.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.268+0000 7f8668aad640 1 Processor -- start 2023-11-28T20:34:22.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.268+0000 7f8668aad640 1 -- start start 2023-11-28T20:34:22.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f8668aad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f86640ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:22.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f8668aad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f866414c900 0x7f8664159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:22.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f8668aad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86640b7c20 0x7f86640b80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:22.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f8668aad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86640cb640 con 0x7f864c0047c0 2023-11-28T20:34:22.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f8668aad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86640b8610 con 0x7f86640b7c20 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f8668aad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866414cd80 con 0x7f866414c900 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f86637fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f86640ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f86637fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f86640ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58364/0 (socket says 172.21.15.45:58364) 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f86637fe640 1 -- 172.21.15.45:0/3485022735 learned_addr learned my addr 172.21.15.45:0/3485022735 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f8662ffd640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f866414c900 0x7f8664159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.269+0000 7f8663fff640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86640b7c20 0x7f86640b80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f86637fe640 1 -- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f866414c900 msgr2=0x7f8664159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f86637fe640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f866414c900 0x7f8664159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f86637fe640 1 -- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86640b7c20 msgr2=0x7f86640b80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f86637fe640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86640b7c20 0x7f86640b80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:22.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f86637fe640 1 -- 172.21.15.45:0/3485022735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86640ca760 con 0x7f864c0047c0 2023-11-28T20:34:22.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f86637fe640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f86640ca180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f86540046b0 tx=0x7f8654007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:22.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f864bfff640 1 -- 172.21.15.45:0/3485022735 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8654014610 con 0x7f864c0047c0 2023-11-28T20:34:22.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f864bfff640 1 -- 172.21.15.45:0/3485022735 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865400f050 con 0x7f864c0047c0 2023-11-28T20:34:22.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f864bfff640 1 -- 172.21.15.45:0/3485022735 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86540149b0 con 0x7f864c0047c0 2023-11-28T20:34:22.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.270+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86640caa40 con 0x7f864c0047c0 2023-11-28T20:34:22.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.271+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86640cb000 con 0x7f864c0047c0 2023-11-28T20:34:22.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.272+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86640cb190 con 0x7f864c0047c0 2023-11-28T20:34:22.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.273+0000 7f864bfff640 1 -- 172.21.15.45:0/3485022735 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8654014b10 con 0x7f864c0047c0 2023-11-28T20:34:22.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.274+0000 7f864bfff640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f865006ae40 0x7f865006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:22.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.274+0000 7f864bfff640 1 -- 172.21.15.45:0/3485022735 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f865408a2b0 con 0x7f864c0047c0 2023-11-28T20:34:22.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.275+0000 7f8662ffd640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f865006ae40 0x7f865006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:22.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.276+0000 7f8662ffd640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f865006ae40 0x7f865006d300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8658034000 tx=0x7f8658007000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:22.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.279+0000 7f864bfff640 1 -- 172.21.15.45:0/3485022735 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8654055060 con 0x7f864c0047c0 2023-11-28T20:34:22.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:22.515+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8664071b30 con 0x7f864c0047c0 2023-11-28T20:34:23.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.006+0000 7f864bfff640 1 -- 172.21.15.45:0/3485022735 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8654058680 con 0x7f864c0047c0 2023-11-28T20:34:23.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f865006ae40 msgr2=0x7f865006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f865006ae40 0x7f865006d300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8658034000 tx=0x7f8658007000 comp rx=0 tx=0).stop 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 msgr2=0x7f86640ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f86640ca180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f86540046b0 tx=0x7f8654007700 comp rx=0 tx=0).stop 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 shutdown_connections 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f865006ae40 0x7f865006d300 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86640b7c20 0x7f86640b80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f866414c900 0x7f8664159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 --2- 172.21.15.45:0/3485022735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0047c0 0x7f86640ca180 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.010+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 >> 172.21.15.45:0/3485022735 conn(0x7f8664070f40 msgr2=0x7f86640cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.011+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 shutdown_connections 2023-11-28T20:34:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.011+0000 7f8668aad640 1 -- 172.21.15.45:0/3485022735 wait complete. 2023-11-28T20:34:23.034 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T20:34:23.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:34:23.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.257+0000 7f2fcceef640 1 Processor -- start 2023-11-28T20:34:23.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.257+0000 7f2fcceef640 1 -- start start 2023-11-28T20:34:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.258+0000 7f2fcceef640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2fc80d02d0 0x7f2fc80d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.258+0000 7f2fcceef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d1ab0 0x7f2fc80b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.258+0000 7f2fcceef640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc80d2230 con 0x7f2fc815d550 2023-11-28T20:34:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.258+0000 7f2fcceef640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc80d2510 con 0x7f2fc80d02d0 2023-11-28T20:34:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.258+0000 7f2fcceef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80d27f0 con 0x7f2fc80d1ab0 2023-11-28T20:34:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.259+0000 7f2fc6ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2fc80d02d0 0x7f2fc80d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.259+0000 7f2fc6ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2fc80d02d0 0x7f2fc80d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58376/0 (socket says 172.21.15.45:58376) 2023-11-28T20:34:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.259+0000 7f2fc6ffd640 1 -- 172.21.15.45:0/4010756252 learned_addr learned my addr 172.21.15.45:0/4010756252 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.259+0000 7f2fc7fff640 1 --2- 172.21.15.45:0/4010756252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d1ab0 0x7f2fc80b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.259+0000 7f2fc6ffd640 1 -- 172.21.15.45:0/4010756252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d1ab0 msgr2=0x7f2fc80b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.259+0000 7f2fc6ffd640 1 --2- 172.21.15.45:0/4010756252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d1ab0 0x7f2fc80b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.259+0000 7f2fc6ffd640 1 -- 172.21.15.45:0/4010756252 >> v1:172.21.15.190:6789/0 conn(0x7f2fc815d550 legacy=0x7f2fc80cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.260+0000 7f2fc6ffd640 1 -- 172.21.15.45:0/4010756252 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80b6a30 con 0x7f2fc80d02d0 2023-11-28T20:34:23.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.261+0000 7f2fc6ffd640 1 --2- 172.21.15.45:0/4010756252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc80d02d0 0x7f2fc80d1570 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009870 tx=0x7f2fbc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f56ff7393f7754 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.261+0000 7f2fc67fc640 1 -- 172.21.15.45:0/4010756252 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc030840 con 0x7f2fc80d02d0 2023-11-28T20:34:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.261+0000 7f2fc67fc640 1 -- 172.21.15.45:0/4010756252 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc0309a0 con 0x7f2fc80d02d0 2023-11-28T20:34:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.261+0000 7f2fc67fc640 1 -- 172.21.15.45:0/4010756252 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc030d30 con 0x7f2fc80d02d0 2023-11-28T20:34:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.261+0000 7f2fcceef640 1 -- 172.21.15.45:0/4010756252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc80d02d0 msgr2=0x7f2fc80d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.261+0000 7f2fcceef640 1 --2- 172.21.15.45:0/4010756252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc80d02d0 0x7f2fc80d1570 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009870 tx=0x7f2fbc02b980 comp rx=0 tx=0).stop 2023-11-28T20:34:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.262+0000 7f2fcceef640 1 -- 172.21.15.45:0/4010756252 shutdown_connections 2023-11-28T20:34:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.262+0000 7f2fcceef640 1 --2- 172.21.15.45:0/4010756252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d1ab0 0x7f2fc80b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.262+0000 7f2fcceef640 1 --2- 172.21.15.45:0/4010756252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc80d02d0 0x7f2fc80d1570 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.262+0000 7f2fcceef640 1 -- 172.21.15.45:0/4010756252 >> 172.21.15.45:0/4010756252 conn(0x7f2fc806ac70 msgr2=0x7f2fc80b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:23.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.262+0000 7f2fcceef640 1 -- 172.21.15.45:0/4010756252 shutdown_connections 2023-11-28T20:34:23.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.262+0000 7f2fcceef640 1 -- 172.21.15.45:0/4010756252 wait complete. 2023-11-28T20:34:23.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.263+0000 7f2fcceef640 1 Processor -- start 2023-11-28T20:34:23.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.264+0000 7f2fcceef640 1 -- start start 2023-11-28T20:34:23.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fcceef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d02d0 0x7f2fc814b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fcceef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fc80d1ab0 0x7f2fc80cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fcceef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc815d550 0x7f2fc8158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fcceef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80ca2a0 con 0x7f2fc815d550 2023-11-28T20:34:23.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fcceef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc814bb30 con 0x7f2fc80d1ab0 2023-11-28T20:34:23.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fcceef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80ca4b0 con 0x7f2fc80d02d0 2023-11-28T20:34:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc815d550 0x7f2fc8158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc815d550 0x7f2fc8158870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58390/0 (socket says 172.21.15.45:58390) 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc7fff640 1 -- 172.21.15.45:0/3231043660 learned_addr learned my addr 172.21.15.45:0/3231043660 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc6ffd640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fc80d1ab0 0x7f2fc80cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc77fe640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d02d0 0x7f2fc814b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc7fff640 1 -- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d02d0 msgr2=0x7f2fc814b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc7fff640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d02d0 0x7f2fc814b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc7fff640 1 -- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fc80d1ab0 msgr2=0x7f2fc80cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc7fff640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fc80d1ab0 0x7f2fc80cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.265+0000 7f2fc7fff640 1 -- 172.21.15.45:0/3231043660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc0094d0 con 0x7f2fc815d550 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.266+0000 7f2fc7fff640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc815d550 0x7f2fc8158870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000d800 tx=0x7f2fc000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.266+0000 7f2faffff640 1 -- 172.21.15.45:0/3231043660 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc000b6c0 con 0x7f2fc815d550 2023-11-28T20:34:23.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.266+0000 7f2faffff640 1 -- 172.21.15.45:0/3231043660 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc000b840 con 0x7f2fc815d550 2023-11-28T20:34:23.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.266+0000 7f2faffff640 1 -- 172.21.15.45:0/3231043660 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc000bbf0 con 0x7f2fc815d550 2023-11-28T20:34:23.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.266+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc80ca790 con 0x7f2fc815d550 2023-11-28T20:34:23.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.266+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc80cad50 con 0x7f2fc815d550 2023-11-28T20:34:23.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.267+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc804e920 con 0x7f2fc815d550 2023-11-28T20:34:23.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.268+0000 7f2faffff640 1 -- 172.21.15.45:0/3231043660 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2fc000bd50 con 0x7f2fc815d550 2023-11-28T20:34:23.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.269+0000 7f2faffff640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2fb406ae70 0x7f2fb406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.269+0000 7f2faffff640 1 -- 172.21.15.45:0/3231043660 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2fc0089eb0 con 0x7f2fc815d550 2023-11-28T20:34:23.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.270+0000 7f2fc77fe640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2fb406ae70 0x7f2fb406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.271+0000 7f2fc77fe640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2fb406ae70 0x7f2fb406d330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80012e0 tx=0x7f2fb8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:23.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.273+0000 7f2faffff640 1 -- 172.21.15.45:0/3231043660 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fc0054bd0 con 0x7f2fc815d550 2023-11-28T20:34:23.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.504+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fc815e6a0 con 0x7f2fc815d550 2023-11-28T20:34:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.505+0000 7f2faffff640 1 -- 172.21.15.45:0/3231043660 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4915 (secure 0 0 0) 0x7f2fc00581f0 con 0x7f2fc815d550 2023-11-28T20:34:23.506 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:23.506 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:34:22.516874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T20:34:23.507 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2023-11-28T20:34:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2fb406ae70 msgr2=0x7f2fb406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2fb406ae70 0x7f2fb406d330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80012e0 tx=0x7f2fb8009040 comp rx=0 tx=0).stop 2023-11-28T20:34:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc815d550 msgr2=0x7f2fc8158870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc815d550 0x7f2fc8158870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000d800 tx=0x7f2fc000dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 shutdown_connections 2023-11-28T20:34:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2fb406ae70 0x7f2fb406d330 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fc815d550 0x7f2fc8158870 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2fc80d1ab0 0x7f2fc80cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 --2- 172.21.15.45:0/3231043660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2fc80d02d0 0x7f2fc814b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 >> 172.21.15.45:0/3231043660 conn(0x7f2fc806ac70 msgr2=0x7f2fc80ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 shutdown_connections 2023-11-28T20:34:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.508+0000 7f2fcceef640 1 -- 172.21.15.45:0/3231043660 wait complete. 2023-11-28T20:34:23.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:23.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.753+0000 7f0233fff640 1 Processor -- start 2023-11-28T20:34:23.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.754+0000 7f0233fff640 1 -- start start 2023-11-28T20:34:23.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.755+0000 7f0233fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f023415d550 0x7f02340cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.755+0000 7f0233fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f02340d0100 0x7f02340d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.755+0000 7f0233fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f02340d1ab0 0x7f0234070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.755+0000 7f0233fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02340d0560 con 0x7f02340d0100 2023-11-28T20:34:23.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.755+0000 7f0233fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02340d2230 con 0x7f02340d1ab0 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.755+0000 7f0233fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02340d2510 con 0x7f023415d550 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f02327fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f02340d0100 0x7f02340d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f02327fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f02340d0100 0x7f02340d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58392/0 (socket says 172.21.15.45:58392) 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f02327fc640 1 -- 172.21.15.45:0/2436113948 learned_addr learned my addr 172.21.15.45:0/2436113948 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f0232ffd640 1 --2- 172.21.15.45:0/2436113948 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f023415d550 0x7f02340cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f02327fc640 1 -- 172.21.15.45:0/2436113948 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f023415d550 msgr2=0x7f02340cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f02327fc640 1 --2- 172.21.15.45:0/2436113948 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f023415d550 0x7f02340cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f02337fe640 1 --2- 172.21.15.45:0/2436113948 >> v2:172.21.15.190:3300/0 conn(0x7f02340d1ab0 0x7f0234070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f02327fc640 1 -- 172.21.15.45:0/2436113948 >> v2:172.21.15.190:3300/0 conn(0x7f02340d1ab0 msgr2=0x7f0234070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:23.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f02327fc640 1 --2- 172.21.15.45:0/2436113948 >> v2:172.21.15.190:3300/0 conn(0x7f02340d1ab0 0x7f0234070b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.756+0000 7f02327fc640 1 -- 172.21.15.45:0/2436113948 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02340d27f0 con 0x7f02340d0100 2023-11-28T20:34:23.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.757+0000 7f02327fc640 1 --2- 172.21.15.45:0/2436113948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 0x7f02340d1570 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f02340c9ec0 tx=0x7f021c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4cdd2ed769ccfa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:23.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.758+0000 7f0231ffb640 1 -- 172.21.15.45:0/2436113948 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c031870 con 0x7f02340d0100 2023-11-28T20:34:23.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.758+0000 7f0231ffb640 1 -- 172.21.15.45:0/2436113948 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021c0319d0 con 0x7f02340d0100 2023-11-28T20:34:23.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.758+0000 7f0231ffb640 1 -- 172.21.15.45:0/2436113948 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c031d60 con 0x7f02340d0100 2023-11-28T20:34:23.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.758+0000 7f0233fff640 1 -- 172.21.15.45:0/2436113948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 msgr2=0x7f02340d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.758+0000 7f0233fff640 1 --2- 172.21.15.45:0/2436113948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 0x7f02340d1570 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f02340c9ec0 tx=0x7f021c02b910 comp rx=0 tx=0).stop 2023-11-28T20:34:23.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.759+0000 7f0233fff640 1 -- 172.21.15.45:0/2436113948 shutdown_connections 2023-11-28T20:34:23.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.759+0000 7f0233fff640 1 --2- 172.21.15.45:0/2436113948 >> v2:172.21.15.190:3300/0 conn(0x7f02340d1ab0 0x7f0234070b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.759+0000 7f0233fff640 1 --2- 172.21.15.45:0/2436113948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 0x7f02340d1570 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.759+0000 7f0233fff640 1 --2- 172.21.15.45:0/2436113948 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f023415d550 0x7f02340cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.759+0000 7f0233fff640 1 -- 172.21.15.45:0/2436113948 >> 172.21.15.45:0/2436113948 conn(0x7f023406ac70 msgr2=0x7f02340b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:23.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.759+0000 7f0233fff640 1 -- 172.21.15.45:0/2436113948 shutdown_connections 2023-11-28T20:34:23.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.759+0000 7f0233fff640 1 -- 172.21.15.45:0/2436113948 wait complete. 2023-11-28T20:34:23.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.761+0000 7f0233fff640 1 Processor -- start 2023-11-28T20:34:23.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.761+0000 7f0233fff640 1 -- start start 2023-11-28T20:34:23.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.762+0000 7f0233fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 0x7f02341f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.762+0000 7f0233fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02340d1ab0 0x7f023414c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.762+0000 7f0233fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f023415d550 0x7f02340ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.762+0000 7f0233fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02340cd570 con 0x7f02340d0100 2023-11-28T20:34:23.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.762+0000 7f0233fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02341f2cf0 con 0x7f023415d550 2023-11-28T20:34:23.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.762+0000 7f0233fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023414c610 con 0x7f02340d1ab0 2023-11-28T20:34:23.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f0232ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 0x7f02341f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f0232ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 0x7f02341f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58398/0 (socket says 172.21.15.45:58398) 2023-11-28T20:34:23.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f0232ffd640 1 -- 172.21.15.45:0/2776947981 learned_addr learned my addr 172.21.15.45:0/2776947981 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:23.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f02337fe640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f023415d550 0x7f02340ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f02327fc640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02340d1ab0 0x7f023414c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f0232ffd640 1 -- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02340d1ab0 msgr2=0x7f023414c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f0232ffd640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02340d1ab0 0x7f023414c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f0232ffd640 1 -- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f023415d550 msgr2=0x7f02340ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f0232ffd640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f023415d550 0x7f02340ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f0232ffd640 1 -- 172.21.15.45:0/2776947981 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c0094d0 con 0x7f02340d0100 2023-11-28T20:34:23.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.763+0000 7f02337fe640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f023415d550 0x7f02340ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:23.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.764+0000 7f0232ffd640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 0x7f02341f27b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0224007e40 tx=0x7f022400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:23.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.764+0000 7f022bfff640 1 -- 172.21.15.45:0/2776947981 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0224014620 con 0x7f02340d0100 2023-11-28T20:34:23.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.764+0000 7f022bfff640 1 -- 172.21.15.45:0/2776947981 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022400ee90 con 0x7f02340d0100 2023-11-28T20:34:23.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.764+0000 7f022bfff640 1 -- 172.21.15.45:0/2776947981 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02240149c0 con 0x7f02340d0100 2023-11-28T20:34:23.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.764+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0234149b00 con 0x7f02340d0100 2023-11-28T20:34:23.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.764+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023414a0c0 con 0x7f02340d0100 2023-11-28T20:34:23.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.765+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023404e920 con 0x7f02340d0100 2023-11-28T20:34:23.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.766+0000 7f022bfff640 1 -- 172.21.15.45:0/2776947981 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0224014b20 con 0x7f02340d0100 2023-11-28T20:34:23.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.768+0000 7f022bfff640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f021406ae40 0x7f021406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:23.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.768+0000 7f022bfff640 1 -- 172.21.15.45:0/2776947981 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f022408a1b0 con 0x7f02340d0100 2023-11-28T20:34:23.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.769+0000 7f02327fc640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f021406ae40 0x7f021406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:23.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.770+0000 7f02327fc640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f021406ae40 0x7f021406d300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f021c036000 tx=0x7f021c03c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:23.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.772+0000 7f022bfff640 1 -- 172.21.15.45:0/2776947981 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0224055000 con 0x7f02340d0100 2023-11-28T20:34:23.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.970+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02340d34d0 con 0x7f02340d0100 2023-11-28T20:34:23.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.972+0000 7f022bfff640 1 -- 172.21.15.45:0/2776947981 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7f0224058620 con 0x7f02340d0100 2023-11-28T20:34:23.973 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:23.973 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:23.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.974+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f021406ae40 msgr2=0x7f021406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f021406ae40 0x7f021406d300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f021c036000 tx=0x7f021c03c040 comp rx=0 tx=0).stop 2023-11-28T20:34:23.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 msgr2=0x7f02341f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:23.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 0x7f02341f27b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0224007e40 tx=0x7f022400e4b0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 shutdown_connections 2023-11-28T20:34:23.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f021406ae40 0x7f021406d300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f023415d550 0x7f02340ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02340d1ab0 0x7f023414c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 --2- 172.21.15.45:0/2776947981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02340d0100 0x7f02341f27b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:23.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 >> 172.21.15.45:0/2776947981 conn(0x7f023406ac70 msgr2=0x7f02340b8300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:23.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 shutdown_connections 2023-11-28T20:34:23.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:23.975+0000 7f0233fff640 1 -- 172.21.15.45:0/2776947981 wait complete. 2023-11-28T20:34:23.997 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T20:34:23.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:34:24.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.214+0000 7fc7c9901640 1 Processor -- start 2023-11-28T20:34:24.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.215+0000 7fc7c9901640 1 -- start start 2023-11-28T20:34:24.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.216+0000 7fc7c9901640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc7c40b35b0 0x7fc7c415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.216+0000 7fc7c9901640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7c415b850 0x7fc7c40b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.216+0000 7fc7c9901640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40b1fa0 0x7fc7c40b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.216+0000 7fc7c9901640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c415ee40 con 0x7fc7c415b850 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.216+0000 7fc7c9901640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c415bda0 con 0x7fc7c40b35b0 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.216+0000 7fc7c9901640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c415c080 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7bbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7c415b850 0x7fc7c40b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7bbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7c415b850 0x7fc7c40b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58692/0 (socket says 172.21.15.45:58692) 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7bbfff640 1 -- 172.21.15.45:0/2639410837 learned_addr learned my addr 172.21.15.45:0/2639410837 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7c2ffd640 1 --2- 172.21.15.45:0/2639410837 >> v2:172.21.15.190:3300/0 conn(0x7fc7c40b35b0 0x7fc7c415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7c37fe640 1 --2- 172.21.15.45:0/2639410837 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40b1fa0 0x7fc7c40b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7bbfff640 1 -- 172.21.15.45:0/2639410837 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40b1fa0 msgr2=0x7fc7c40b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7bbfff640 1 --2- 172.21.15.45:0/2639410837 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40b1fa0 0x7fc7c40b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7bbfff640 1 -- 172.21.15.45:0/2639410837 >> v2:172.21.15.190:3300/0 conn(0x7fc7c40b35b0 msgr2=0x7fc7c415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7bbfff640 1 --2- 172.21.15.45:0/2639410837 >> v2:172.21.15.190:3300/0 conn(0x7fc7c40b35b0 0x7fc7c415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.217+0000 7fc7bbfff640 1 -- 172.21.15.45:0/2639410837 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c415c360 con 0x7fc7c415b850 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.218+0000 7fc7bbfff640 1 --2- 172.21.15.45:0/2639410837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c415b850 0x7fc7c40b0c30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc7c406e7f0 tx=0x7fc7ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe1361535c7678a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c27fc640 1 -- 172.21.15.45:0/2639410837 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac031870 con 0x7fc7c415b850 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c27fc640 1 -- 172.21.15.45:0/2639410837 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac0319d0 con 0x7fc7c415b850 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c27fc640 1 -- 172.21.15.45:0/2639410837 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac031d60 con 0x7fc7c415b850 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c9901640 1 -- 172.21.15.45:0/2639410837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c415b850 msgr2=0x7fc7c40b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c9901640 1 --2- 172.21.15.45:0/2639410837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c415b850 0x7fc7c40b0c30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc7c406e7f0 tx=0x7fc7ac02b910 comp rx=0 tx=0).stop 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c9901640 1 -- 172.21.15.45:0/2639410837 shutdown_connections 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c9901640 1 --2- 172.21.15.45:0/2639410837 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40b1fa0 0x7fc7c40b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c9901640 1 --2- 172.21.15.45:0/2639410837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c415b850 0x7fc7c40b0c30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c9901640 1 --2- 172.21.15.45:0/2639410837 >> v2:172.21.15.190:3300/0 conn(0x7fc7c40b35b0 0x7fc7c415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c9901640 1 -- 172.21.15.45:0/2639410837 >> 172.21.15.45:0/2639410837 conn(0x7fc7c406a500 msgr2=0x7fc7c406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:24.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c9901640 1 -- 172.21.15.45:0/2639410837 shutdown_connections 2023-11-28T20:34:24.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.219+0000 7fc7c9901640 1 -- 172.21.15.45:0/2639410837 wait complete. 2023-11-28T20:34:24.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.221+0000 7fc7c9901640 1 Processor -- start 2023-11-28T20:34:24.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.221+0000 7fc7c9901640 1 -- start start 2023-11-28T20:34:24.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.222+0000 7fc7c9901640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c40b1fa0 0x7fc7c40cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.222+0000 7fc7c9901640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40b35b0 0x7fc7c40ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.222+0000 7fc7c9901640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c415b850 0x7fc7c40adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.222+0000 7fc7c9901640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c4142cd0 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.222+0000 7fc7c9901640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40cbd70 con 0x7fc7c415b850 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c9901640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c4136ea0 con 0x7fc7c40b35b0 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c40b1fa0 0x7fc7c40cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c40b1fa0 0x7fc7c40cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58704/0 (socket says 172.21.15.45:58704) 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c2ffd640 1 -- 172.21.15.45:0/1255824355 learned_addr learned my addr 172.21.15.45:0/1255824355 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c37fe640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c415b850 0x7fc7c40adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c2ffd640 1 -- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40b35b0 msgr2=0x7fc7c40ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c2ffd640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40b35b0 0x7fc7c40ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c2ffd640 1 -- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c415b850 msgr2=0x7fc7c40adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:24.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c2ffd640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c415b850 0x7fc7c40adf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c2ffd640 1 -- 172.21.15.45:0/1255824355 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0094d0 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.223+0000 7fc7c2ffd640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c40b1fa0 0x7fc7c40cb830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400d800 tx=0x7fc7b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:24.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.224+0000 7fc7c0ff9640 1 -- 172.21.15.45:0/1255824355 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b400b6c0 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.224+0000 7fc7c0ff9640 1 -- 172.21.15.45:0/1255824355 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b400b820 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.224+0000 7fc7c0ff9640 1 -- 172.21.15.45:0/1255824355 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b400bbb0 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.224+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c4137180 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.224+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c4138ef0 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.225+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c41376a0 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.226+0000 7fc7c0ff9640 1 -- 172.21.15.45:0/1255824355 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc7b400bd10 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.227+0000 7fc7c0ff9640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7a406ae40 0x7fc7a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.227+0000 7fc7c0ff9640 1 -- 172.21.15.45:0/1255824355 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc7b4089ee0 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.228+0000 7fc7bbfff640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7a406ae40 0x7fc7a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.229+0000 7fc7bbfff640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7a406ae40 0x7fc7a406d300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac036000 tx=0x7fc7ac030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:24.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.232+0000 7fc7c0ff9640 1 -- 172.21.15.45:0/1255824355 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7c41376a0 con 0x7fc7c40b1fa0 2023-11-28T20:34:24.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.423+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7c415cf50 con 0x7fc7a406ae40 2023-11-28T20:34:24.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.456+0000 7fc7c0ff9640 1 -- 172.21.15.45:0/1255824355 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345164 (secure 0 0 0) 0x7fc7ac03a420 con 0x7fc7a406ae40 2023-11-28T20:34:24.461 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:24.465 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:34:24.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.464+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7a406ae40 msgr2=0x7fc7a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:24.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.464+0000 7fc7c9901640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7a406ae40 0x7fc7a406d300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac036000 tx=0x7fc7ac030000 comp rx=0 tx=0).stop 2023-11-28T20:34:24.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c40b1fa0 msgr2=0x7fc7c40cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:24.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c40b1fa0 0x7fc7c40cb830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400d800 tx=0x7fc7b400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 shutdown_connections 2023-11-28T20:34:24.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7a406ae40 0x7fc7a406d300 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c415b850 0x7fc7c40adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40b35b0 0x7fc7c40ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 --2- 172.21.15.45:0/1255824355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7c40b1fa0 0x7fc7c40cb830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 >> 172.21.15.45:0/1255824355 conn(0x7fc7c406a500 msgr2=0x7fc7c4144850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:24.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 shutdown_connections 2023-11-28T20:34:24.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.465+0000 7fc7c9901640 1 -- 172.21.15.45:0/1255824355 wait complete. 2023-11-28T20:34:24.490 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-11-28T20:34:23.133726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236124,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738596,"statfs":{"total":773094113280,"available":772852322304,"internally_reserved":0,"allocated":3383296,"data_stored":2025718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005078"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119305+0000","last_change":"2023-11-28T20:33:53.064970+0000","last_active":"2023-11-28T20:33:58.119305+0000","last_peered":"2023-11-28T20:33:58.119305+0000","last_clean":"2023-11-28T20:33:58.119305+0000","last_became_active":"2023-11-28T20:33:53.064659+0000","last_became_peered":"2023-11-28T20:33:53.064659+0000","last_unstale":"2023-11-28T20:33:58.119305+0000","last_undegraded":"2023-11-28T20:33:58.119305+0000","last_fullsized":"2023-11-28T20:33:58.119305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.109629+0000","last_change":"2023-11-28T20:33:53.948203+0000","last_active":"2023-11-28T20:33:58.109629+0000","last_peered":"2023-11-28T20:33:58.109629+0000","last_clean":"2023-11-28T20:33:58.109629+0000","last_became_active":"2023-11-28T20:33:53.947906+0000","last_became_peered":"2023-11-28T20:33:53.947906+0000","last_unstale":"2023-11-28T20:33:58.109629+0000","last_undegraded":"2023-11-28T20:33:58.109629+0000","last_fullsized":"2023-11-28T20:33:58.109629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918353+0000","last_change":"2023-11-28T20:33:58.158488+0000","last_active":"2023-11-28T20:33:58.918353+0000","last_peered":"2023-11-28T20:33:58.918353+0000","last_clean":"2023-11-28T20:33:58.918353+0000","last_became_active":"2023-11-28T20:33:58.158046+0000","last_became_peered":"2023-11-28T20:33:58.158046+0000","last_unstale":"2023-11-28T20:33:58.918353+0000","last_undegraded":"2023-11-28T20:33:58.918353+0000","last_fullsized":"2023-11-28T20:33:58.918353+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:20:48.980178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.111851+0000","last_change":"2023-11-28T20:33:53.050283+0000","last_active":"2023-11-28T20:33:58.111851+0000","last_peered":"2023-11-28T20:33:58.111851+0000","last_clean":"2023-11-28T20:33:58.111851+0000","last_became_active":"2023-11-28T20:33:53.049963+0000","last_became_peered":"2023-11-28T20:33:53.049963+0000","last_unstale":"2023-11-28T20:33:58.111851+0000","last_undegraded":"2023-11-28T20:33:58.111851+0000","last_fullsized":"2023-11-28T20:33:58.111851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116825+0000","last_change":"2023-11-28T20:33:53.943001+0000","last_active":"2023-11-28T20:33:58.116825+0000","last_peered":"2023-11-28T20:33:58.116825+0000","last_clean":"2023-11-28T20:33:58.116825+0000","last_became_active":"2023-11-28T20:33:53.940632+0000","last_became_peered":"2023-11-28T20:33:53.940632+0000","last_unstale":"2023-11-28T20:33:58.116825+0000","last_undegraded":"2023-11-28T20:33:58.116825+0000","last_fullsized":"2023-11-28T20:33:58.116825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916311+0000","last_change":"2023-11-28T20:33:58.127527+0000","last_active":"2023-11-28T20:33:58.916311+0000","last_peered":"2023-11-28T20:33:58.916311+0000","last_clean":"2023-11-28T20:33:58.916311+0000","last_became_active":"2023-11-28T20:33:58.127058+0000","last_became_peered":"2023-11-28T20:33:58.127058+0000","last_unstale":"2023-11-28T20:33:58.916311+0000","last_undegraded":"2023-11-28T20:33:58.916311+0000","last_fullsized":"2023-11-28T20:33:58.916311+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:12.932976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151962+0000","last_change":"2023-11-28T20:33:53.073019+0000","last_active":"2023-11-28T20:33:58.151962+0000","last_peered":"2023-11-28T20:33:58.151962+0000","last_clean":"2023-11-28T20:33:58.151962+0000","last_became_active":"2023-11-28T20:33:53.072728+0000","last_became_peered":"2023-11-28T20:33:53.072728+0000","last_unstale":"2023-11-28T20:33:58.151962+0000","last_undegraded":"2023-11-28T20:33:58.151962+0000","last_fullsized":"2023-11-28T20:33:58.151962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122157+0000","last_change":"2023-11-28T20:33:53.951054+0000","last_active":"2023-11-28T20:33:58.122157+0000","last_peered":"2023-11-28T20:33:58.122157+0000","last_clean":"2023-11-28T20:33:58.122157+0000","last_became_active":"2023-11-28T20:33:53.950730+0000","last_became_peered":"2023-11-28T20:33:53.950730+0000","last_unstale":"2023-11-28T20:33:58.122157+0000","last_undegraded":"2023-11-28T20:33:58.122157+0000","last_fullsized":"2023-11-28T20:33:58.122157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:19.573371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916721+0000","last_change":"2023-11-28T20:33:58.131034+0000","last_active":"2023-11-28T20:33:58.916721+0000","last_peered":"2023-11-28T20:33:58.916721+0000","last_clean":"2023-11-28T20:33:58.916721+0000","last_became_active":"2023-11-28T20:33:58.129161+0000","last_became_peered":"2023-11-28T20:33:58.129161+0000","last_unstale":"2023-11-28T20:33:58.916721+0000","last_undegraded":"2023-11-28T20:33:58.916721+0000","last_fullsized":"2023-11-28T20:33:58.916721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:19.781603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151744+0000","last_change":"2023-11-28T20:33:53.068043+0000","last_active":"2023-11-28T20:33:58.151744+0000","last_peered":"2023-11-28T20:33:58.151744+0000","last_clean":"2023-11-28T20:33:58.151744+0000","last_became_active":"2023-11-28T20:33:53.067650+0000","last_became_peered":"2023-11-28T20:33:53.067650+0000","last_unstale":"2023-11-28T20:33:58.151744+0000","last_undegraded":"2023-11-28T20:33:58.151744+0000","last_fullsized":"2023-11-28T20:33:58.151744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123332+0000","last_change":"2023-11-28T20:33:53.954109+0000","last_active":"2023-11-28T20:33:58.123332+0000","last_peered":"2023-11-28T20:33:58.123332+0000","last_clean":"2023-11-28T20:33:58.123332+0000","last_became_active":"2023-11-28T20:33:53.953749+0000","last_became_peered":"2023-11-28T20:33:53.953749+0000","last_unstale":"2023-11-28T20:33:58.123332+0000","last_undegraded":"2023-11-28T20:33:58.123332+0000","last_fullsized":"2023-11-28T20:33:58.123332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:21.394618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916961+0000","last_change":"2023-11-28T20:33:58.132306+0000","last_active":"2023-11-28T20:33:58.916961+0000","last_peered":"2023-11-28T20:33:58.916961+0000","last_clean":"2023-11-28T20:33:58.916961+0000","last_became_active":"2023-11-28T20:33:58.131782+0000","last_became_peered":"2023-11-28T20:33:58.131782+0000","last_unstale":"2023-11-28T20:33:58.916961+0000","last_undegraded":"2023-11-28T20:33:58.916961+0000","last_fullsized":"2023-11-28T20:33:58.916961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:30.608189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121351+0000","last_change":"2023-11-28T20:33:53.954276+0000","last_active":"2023-11-28T20:33:58.121351+0000","last_peered":"2023-11-28T20:33:58.121351+0000","last_clean":"2023-11-28T20:33:58.121351+0000","last_became_active":"2023-11-28T20:33:53.953993+0000","last_became_peered":"2023-11-28T20:33:53.953993+0000","last_unstale":"2023-11-28T20:33:58.121351+0000","last_undegraded":"2023-11-28T20:33:58.121351+0000","last_fullsized":"2023-11-28T20:33:58.121351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151814+0000","last_change":"2023-11-28T20:33:53.075863+0000","last_active":"2023-11-28T20:33:58.151814+0000","last_peered":"2023-11-28T20:33:58.151814+0000","last_clean":"2023-11-28T20:33:58.151814+0000","last_became_active":"2023-11-28T20:33:53.075523+0000","last_became_peered":"2023-11-28T20:33:53.075523+0000","last_unstale":"2023-11-28T20:33:58.151814+0000","last_undegraded":"2023-11-28T20:33:58.151814+0000","last_fullsized":"2023-11-28T20:33:58.151814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917964+0000","last_change":"2023-11-28T20:33:58.130991+0000","last_active":"2023-11-28T20:33:58.917964+0000","last_peered":"2023-11-28T20:33:58.917964+0000","last_clean":"2023-11-28T20:33:58.917964+0000","last_became_active":"2023-11-28T20:33:58.130377+0000","last_became_peered":"2023-11-28T20:33:58.130377+0000","last_unstale":"2023-11-28T20:33:58.917964+0000","last_undegraded":"2023-11-28T20:33:58.917964+0000","last_fullsized":"2023-11-28T20:33:58.917964+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103760+0000","last_change":"2023-11-28T20:33:53.943400+0000","last_active":"2023-11-28T20:33:58.103760+0000","last_peered":"2023-11-28T20:33:58.103760+0000","last_clean":"2023-11-28T20:33:58.103760+0000","last_became_active":"2023-11-28T20:33:53.937673+0000","last_became_peered":"2023-11-28T20:33:53.937673+0000","last_unstale":"2023-11-28T20:33:58.103760+0000","last_undegraded":"2023-11-28T20:33:58.103760+0000","last_fullsized":"2023-11-28T20:33:58.103760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125979+0000","last_change":"2023-11-28T20:33:53.062456+0000","last_active":"2023-11-28T20:33:58.125979+0000","last_peered":"2023-11-28T20:33:58.125979+0000","last_clean":"2023-11-28T20:33:58.125979+0000","last_became_active":"2023-11-28T20:33:53.062163+0000","last_became_peered":"2023-11-28T20:33:53.062163+0000","last_unstale":"2023-11-28T20:33:58.125979+0000","last_undegraded":"2023-11-28T20:33:58.125979+0000","last_fullsized":"2023-11-28T20:33:58.125979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917431+0000","last_change":"2023-11-28T20:33:58.137561+0000","last_active":"2023-11-28T20:33:58.917431+0000","last_peered":"2023-11-28T20:33:58.917431+0000","last_clean":"2023-11-28T20:33:58.917431+0000","last_became_active":"2023-11-28T20:33:58.137065+0000","last_became_peered":"2023-11-28T20:33:58.137065+0000","last_unstale":"2023-11-28T20:33:58.917431+0000","last_undegraded":"2023-11-28T20:33:58.917431+0000","last_fullsized":"2023-11-28T20:33:58.917431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103501+0000","last_change":"2023-11-28T20:33:53.940322+0000","last_active":"2023-11-28T20:33:58.103501+0000","last_peered":"2023-11-28T20:33:58.103501+0000","last_clean":"2023-11-28T20:33:58.103501+0000","last_became_active":"2023-11-28T20:33:53.939986+0000","last_became_peered":"2023-11-28T20:33:53.939986+0000","last_unstale":"2023-11-28T20:33:58.103501+0000","last_undegraded":"2023-11-28T20:33:58.103501+0000","last_fullsized":"2023-11-28T20:33:58.103501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123391+0000","last_change":"2023-11-28T20:33:53.062082+0000","last_active":"2023-11-28T20:33:58.123391+0000","last_peered":"2023-11-28T20:33:58.123391+0000","last_clean":"2023-11-28T20:33:58.123391+0000","last_became_active":"2023-11-28T20:33:53.060583+0000","last_became_peered":"2023-11-28T20:33:53.060583+0000","last_unstale":"2023-11-28T20:33:58.123391+0000","last_undegraded":"2023-11-28T20:33:58.123391+0000","last_fullsized":"2023-11-28T20:33:58.123391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916613+0000","last_change":"2023-11-28T20:33:58.158289+0000","last_active":"2023-11-28T20:33:58.916613+0000","last_peered":"2023-11-28T20:33:58.916613+0000","last_clean":"2023-11-28T20:33:58.916613+0000","last_became_active":"2023-11-28T20:33:58.157856+0000","last_became_peered":"2023-11-28T20:33:58.157856+0000","last_unstale":"2023-11-28T20:33:58.916613+0000","last_undegraded":"2023-11-28T20:33:58.916613+0000","last_fullsized":"2023-11-28T20:33:58.916613+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:17.807344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.112802+0000","last_change":"2023-11-28T20:33:53.942280+0000","last_active":"2023-11-28T20:33:58.112802+0000","last_peered":"2023-11-28T20:33:58.112802+0000","last_clean":"2023-11-28T20:33:58.112802+0000","last_became_active":"2023-11-28T20:33:53.939778+0000","last_became_peered":"2023-11-28T20:33:53.939778+0000","last_unstale":"2023-11-28T20:33:58.112802+0000","last_undegraded":"2023-11-28T20:33:58.112802+0000","last_fullsized":"2023-11-28T20:33:58.112802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:33.050269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151419+0000","last_change":"2023-11-28T20:33:53.068971+0000","last_active":"2023-11-28T20:33:58.151419+0000","last_peered":"2023-11-28T20:33:58.151419+0000","last_clean":"2023-11-28T20:33:58.151419+0000","last_became_active":"2023-11-28T20:33:53.068069+0000","last_became_peered":"2023-11-28T20:33:53.068069+0000","last_unstale":"2023-11-28T20:33:58.151419+0000","last_undegraded":"2023-11-28T20:33:58.151419+0000","last_fullsized":"2023-11-28T20:33:58.151419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:41.371596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917429+0000","last_change":"2023-11-28T20:33:58.137597+0000","last_active":"2023-11-28T20:33:58.917429+0000","last_peered":"2023-11-28T20:33:58.917429+0000","last_clean":"2023-11-28T20:33:58.917429+0000","last_became_active":"2023-11-28T20:33:58.137078+0000","last_became_peered":"2023-11-28T20:33:58.137078+0000","last_unstale":"2023-11-28T20:33:58.917429+0000","last_undegraded":"2023-11-28T20:33:58.917429+0000","last_fullsized":"2023-11-28T20:33:58.917429+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:23.642548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120275+0000","last_change":"2023-11-28T20:33:53.945506+0000","last_active":"2023-11-28T20:33:58.120275+0000","last_peered":"2023-11-28T20:33:58.120275+0000","last_clean":"2023-11-28T20:33:58.120275+0000","last_became_active":"2023-11-28T20:33:53.945153+0000","last_became_peered":"2023-11-28T20:33:53.945153+0000","last_unstale":"2023-11-28T20:33:58.120275+0000","last_undegraded":"2023-11-28T20:33:58.120275+0000","last_fullsized":"2023-11-28T20:33:58.120275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126982+0000","last_change":"2023-11-28T20:33:53.064943+0000","last_active":"2023-11-28T20:33:58.126982+0000","last_peered":"2023-11-28T20:33:58.126982+0000","last_clean":"2023-11-28T20:33:58.126982+0000","last_became_active":"2023-11-28T20:33:53.064644+0000","last_became_peered":"2023-11-28T20:33:53.064644+0000","last_unstale":"2023-11-28T20:33:58.126982+0000","last_undegraded":"2023-11-28T20:33:58.126982+0000","last_fullsized":"2023-11-28T20:33:58.126982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916509+0000","last_change":"2023-11-28T20:33:58.158266+0000","last_active":"2023-11-28T20:33:58.916509+0000","last_peered":"2023-11-28T20:33:58.916509+0000","last_clean":"2023-11-28T20:33:58.916509+0000","last_became_active":"2023-11-28T20:33:58.157836+0000","last_became_peered":"2023-11-28T20:33:58.157836+0000","last_unstale":"2023-11-28T20:33:58.916509+0000","last_undegraded":"2023-11-28T20:33:58.916509+0000","last_fullsized":"2023-11-28T20:33:58.916509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:44.771180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.114503+0000","last_change":"2023-11-28T20:33:53.941572+0000","last_active":"2023-11-28T20:33:58.114503+0000","last_peered":"2023-11-28T20:33:58.114503+0000","last_clean":"2023-11-28T20:33:58.114503+0000","last_became_active":"2023-11-28T20:33:53.941245+0000","last_became_peered":"2023-11-28T20:33:53.941245+0000","last_unstale":"2023-11-28T20:33:58.114503+0000","last_undegraded":"2023-11-28T20:33:58.114503+0000","last_fullsized":"2023-11-28T20:33:58.114503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123698+0000","last_change":"2023-11-28T20:33:53.058678+0000","last_active":"2023-11-28T20:33:58.123698+0000","last_peered":"2023-11-28T20:33:58.123698+0000","last_clean":"2023-11-28T20:33:58.123698+0000","last_became_active":"2023-11-28T20:33:53.058375+0000","last_became_peered":"2023-11-28T20:33:53.058375+0000","last_unstale":"2023-11-28T20:33:58.123698+0000","last_undegraded":"2023-11-28T20:33:58.123698+0000","last_fullsized":"2023-11-28T20:33:58.123698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:12.793548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918259+0000","last_change":"2023-11-28T20:33:58.154128+0000","last_active":"2023-11-28T20:33:58.918259+0000","last_peered":"2023-11-28T20:33:58.918259+0000","last_clean":"2023-11-28T20:33:58.918259+0000","last_became_active":"2023-11-28T20:33:58.153697+0000","last_became_peered":"2023-11-28T20:33:58.153697+0000","last_unstale":"2023-11-28T20:33:58.918259+0000","last_undegraded":"2023-11-28T20:33:58.918259+0000","last_fullsized":"2023-11-28T20:33:58.918259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:47.677895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123582+0000","last_change":"2023-11-28T20:33:53.941553+0000","last_active":"2023-11-28T20:33:58.123582+0000","last_peered":"2023-11-28T20:33:58.123582+0000","last_clean":"2023-11-28T20:33:58.123582+0000","last_became_active":"2023-11-28T20:33:53.941248+0000","last_became_peered":"2023-11-28T20:33:53.941248+0000","last_unstale":"2023-11-28T20:33:58.123582+0000","last_undegraded":"2023-11-28T20:33:58.123582+0000","last_fullsized":"2023-11-28T20:33:58.123582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152051+0000","last_change":"2023-11-28T20:33:53.069890+0000","last_active":"2023-11-28T20:33:58.152051+0000","last_peered":"2023-11-28T20:33:58.152051+0000","last_clean":"2023-11-28T20:33:58.152051+0000","last_became_active":"2023-11-28T20:33:53.069572+0000","last_became_peered":"2023-11-28T20:33:53.069572+0000","last_unstale":"2023-11-28T20:33:58.152051+0000","last_undegraded":"2023-11-28T20:33:58.152051+0000","last_fullsized":"2023-11-28T20:33:58.152051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:57.471000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589761+0000","last_change":"2023-11-28T20:33:58.157661+0000","last_active":"2023-11-28T20:33:59.589761+0000","last_peered":"2023-11-28T20:33:59.589761+0000","last_clean":"2023-11-28T20:33:59.589761+0000","last_became_active":"2023-11-28T20:33:58.157072+0000","last_became_peered":"2023-11-28T20:33:58.157072+0000","last_unstale":"2023-11-28T20:33:59.589761+0000","last_undegraded":"2023-11-28T20:33:59.589761+0000","last_fullsized":"2023-11-28T20:33:59.589761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:11.495353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123465+0000","last_change":"2023-11-28T20:33:53.947235+0000","last_active":"2023-11-28T20:33:58.123465+0000","last_peered":"2023-11-28T20:33:58.123465+0000","last_clean":"2023-11-28T20:33:58.123465+0000","last_became_active":"2023-11-28T20:33:53.946977+0000","last_became_peered":"2023-11-28T20:33:53.946977+0000","last_unstale":"2023-11-28T20:33:58.123465+0000","last_undegraded":"2023-11-28T20:33:58.123465+0000","last_fullsized":"2023-11-28T20:33:58.123465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:08.318223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123720+0000","last_change":"2023-11-28T20:33:53.060664+0000","last_active":"2023-11-28T20:33:58.123720+0000","last_peered":"2023-11-28T20:33:58.123720+0000","last_clean":"2023-11-28T20:33:58.123720+0000","last_became_active":"2023-11-28T20:33:53.058196+0000","last_became_peered":"2023-11-28T20:33:53.058196+0000","last_unstale":"2023-11-28T20:33:58.123720+0000","last_undegraded":"2023-11-28T20:33:58.123720+0000","last_fullsized":"2023-11-28T20:33:58.123720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.915328+0000","last_change":"2023-11-28T20:33:58.133412+0000","last_active":"2023-11-28T20:33:58.915328+0000","last_peered":"2023-11-28T20:33:58.915328+0000","last_clean":"2023-11-28T20:33:58.915328+0000","last_became_active":"2023-11-28T20:33:58.132991+0000","last_became_peered":"2023-11-28T20:33:58.132991+0000","last_unstale":"2023-11-28T20:33:58.915328+0000","last_undegraded":"2023-11-28T20:33:58.915328+0000","last_fullsized":"2023-11-28T20:33:58.915328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.107225+0000","last_change":"2023-11-28T20:33:53.942153+0000","last_active":"2023-11-28T20:33:58.107225+0000","last_peered":"2023-11-28T20:33:58.107225+0000","last_clean":"2023-11-28T20:33:58.107225+0000","last_became_active":"2023-11-28T20:33:53.941788+0000","last_became_peered":"2023-11-28T20:33:53.941788+0000","last_unstale":"2023-11-28T20:33:58.107225+0000","last_undegraded":"2023-11-28T20:33:58.107225+0000","last_fullsized":"2023-11-28T20:33:58.107225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:47.311066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126159+0000","last_change":"2023-11-28T20:33:53.058321+0000","last_active":"2023-11-28T20:33:58.126159+0000","last_peered":"2023-11-28T20:33:58.126159+0000","last_clean":"2023-11-28T20:33:58.126159+0000","last_became_active":"2023-11-28T20:33:53.058170+0000","last_became_peered":"2023-11-28T20:33:53.058170+0000","last_unstale":"2023-11-28T20:33:58.126159+0000","last_undegraded":"2023-11-28T20:33:58.126159+0000","last_fullsized":"2023-11-28T20:33:58.126159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:12.793548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917495+0000","last_change":"2023-11-28T20:33:58.138024+0000","last_active":"2023-11-28T20:33:58.917495+0000","last_peered":"2023-11-28T20:33:58.917495+0000","last_clean":"2023-11-28T20:33:58.917495+0000","last_became_active":"2023-11-28T20:33:58.137601+0000","last_became_peered":"2023-11-28T20:33:58.137601+0000","last_unstale":"2023-11-28T20:33:58.917495+0000","last_undegraded":"2023-11-28T20:33:58.917495+0000","last_fullsized":"2023-11-28T20:33:58.917495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:45.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122244+0000","last_change":"2023-11-28T20:33:53.942125+0000","last_active":"2023-11-28T20:33:58.122244+0000","last_peered":"2023-11-28T20:33:58.122244+0000","last_clean":"2023-11-28T20:33:58.122244+0000","last_became_active":"2023-11-28T20:33:53.941741+0000","last_became_peered":"2023-11-28T20:33:53.941741+0000","last_unstale":"2023-11-28T20:33:58.122244+0000","last_undegraded":"2023-11-28T20:33:58.122244+0000","last_fullsized":"2023-11-28T20:33:58.122244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099678+0000","last_change":"2023-11-28T20:33:53.050707+0000","last_active":"2023-11-28T20:33:58.099678+0000","last_peered":"2023-11-28T20:33:58.099678+0000","last_clean":"2023-11-28T20:33:58.099678+0000","last_became_active":"2023-11-28T20:33:53.050403+0000","last_became_peered":"2023-11-28T20:33:53.050403+0000","last_unstale":"2023-11-28T20:33:58.099678+0000","last_undegraded":"2023-11-28T20:33:58.099678+0000","last_fullsized":"2023-11-28T20:33:58.099678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:32.036639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916922+0000","last_change":"2023-11-28T20:33:58.139720+0000","last_active":"2023-11-28T20:33:58.916922+0000","last_peered":"2023-11-28T20:33:58.916922+0000","last_clean":"2023-11-28T20:33:58.916922+0000","last_became_active":"2023-11-28T20:33:58.138372+0000","last_became_peered":"2023-11-28T20:33:58.138372+0000","last_unstale":"2023-11-28T20:33:58.916922+0000","last_undegraded":"2023-11-28T20:33:58.916922+0000","last_fullsized":"2023-11-28T20:33:58.916922+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:11.495353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117362+0000","last_change":"2023-11-28T20:33:53.934500+0000","last_active":"2023-11-28T20:33:58.117362+0000","last_peered":"2023-11-28T20:33:58.117362+0000","last_clean":"2023-11-28T20:33:58.117362+0000","last_became_active":"2023-11-28T20:33:53.933165+0000","last_became_peered":"2023-11-28T20:33:53.933165+0000","last_unstale":"2023-11-28T20:33:58.117362+0000","last_undegraded":"2023-11-28T20:33:58.117362+0000","last_fullsized":"2023-11-28T20:33:58.117362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:48.148924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151406+0000","last_change":"2023-11-28T20:33:53.069660+0000","last_active":"2023-11-28T20:33:58.151406+0000","last_peered":"2023-11-28T20:33:58.151406+0000","last_clean":"2023-11-28T20:33:58.151406+0000","last_became_active":"2023-11-28T20:33:53.069198+0000","last_became_peered":"2023-11-28T20:33:53.069198+0000","last_unstale":"2023-11-28T20:33:58.151406+0000","last_undegraded":"2023-11-28T20:33:58.151406+0000","last_fullsized":"2023-11-28T20:33:58.151406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:46.297436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590181+0000","last_change":"2023-11-28T20:33:58.162077+0000","last_active":"2023-11-28T20:33:59.590181+0000","last_peered":"2023-11-28T20:33:59.590181+0000","last_clean":"2023-11-28T20:33:59.590181+0000","last_became_active":"2023-11-28T20:33:58.159109+0000","last_became_peered":"2023-11-28T20:33:58.159109+0000","last_unstale":"2023-11-28T20:33:59.590181+0000","last_undegraded":"2023-11-28T20:33:59.590181+0000","last_fullsized":"2023-11-28T20:33:59.590181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:47.677895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103295+0000","last_change":"2023-11-28T20:33:53.952209+0000","last_active":"2023-11-28T20:33:58.103295+0000","last_peered":"2023-11-28T20:33:58.103295+0000","last_clean":"2023-11-28T20:33:58.103295+0000","last_became_active":"2023-11-28T20:33:53.949676+0000","last_became_peered":"2023-11-28T20:33:53.949676+0000","last_unstale":"2023-11-28T20:33:58.103295+0000","last_undegraded":"2023-11-28T20:33:58.103295+0000","last_fullsized":"2023-11-28T20:33:58.103295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121849+0000","last_change":"2023-11-28T20:33:53.053226+0000","last_active":"2023-11-28T20:33:58.121849+0000","last_peered":"2023-11-28T20:33:58.121849+0000","last_clean":"2023-11-28T20:33:58.121849+0000","last_became_active":"2023-11-28T20:33:53.052910+0000","last_became_peered":"2023-11-28T20:33:53.052910+0000","last_unstale":"2023-11-28T20:33:58.121849+0000","last_undegraded":"2023-11-28T20:33:58.121849+0000","last_fullsized":"2023-11-28T20:33:58.121849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:41.755062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":169,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151553+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:33:58.151553+0000","last_peered":"2023-11-28T20:33:58.151553+0000","last_clean":"2023-11-28T20:33:58.151553+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:33:58.151553+0000","last_undegraded":"2023-11-28T20:33:58.151553+0000","last_fullsized":"2023-11-28T20:33:58.151553+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589763+0000","last_change":"2023-11-28T20:33:58.157671+0000","last_active":"2023-11-28T20:33:59.589763+0000","last_peered":"2023-11-28T20:33:59.589763+0000","last_clean":"2023-11-28T20:33:59.589763+0000","last_became_active":"2023-11-28T20:33:58.157254+0000","last_became_peered":"2023-11-28T20:33:58.157254+0000","last_unstale":"2023-11-28T20:33:59.589763+0000","last_undegraded":"2023-11-28T20:33:59.589763+0000","last_fullsized":"2023-11-28T20:33:59.589763+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151876+0000","last_change":"2023-11-28T20:33:53.960256+0000","last_active":"2023-11-28T20:33:58.151876+0000","last_peered":"2023-11-28T20:33:58.151876+0000","last_clean":"2023-11-28T20:33:58.151876+0000","last_became_active":"2023-11-28T20:33:53.959635+0000","last_became_peered":"2023-11-28T20:33:53.959635+0000","last_unstale":"2023-11-28T20:33:58.151876+0000","last_undegraded":"2023-11-28T20:33:58.151876+0000","last_fullsized":"2023-11-28T20:33:58.151876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:08.318223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099297+0000","last_change":"2023-11-28T20:33:53.054253+0000","last_active":"2023-11-28T20:33:58.099297+0000","last_peered":"2023-11-28T20:33:58.099297+0000","last_clean":"2023-11-28T20:33:58.099297+0000","last_became_active":"2023-11-28T20:33:53.053912+0000","last_became_peered":"2023-11-28T20:33:53.053912+0000","last_unstale":"2023-11-28T20:33:58.099297+0000","last_undegraded":"2023-11-28T20:33:58.099297+0000","last_fullsized":"2023-11-28T20:33:58.099297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:53.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589287+0000","last_change":"2023-11-28T20:33:58.154204+0000","last_active":"2023-11-28T20:33:59.589287+0000","last_peered":"2023-11-28T20:33:59.589287+0000","last_clean":"2023-11-28T20:33:59.589287+0000","last_became_active":"2023-11-28T20:33:58.153585+0000","last_became_peered":"2023-11-28T20:33:58.153585+0000","last_unstale":"2023-11-28T20:33:59.589287+0000","last_undegraded":"2023-11-28T20:33:59.589287+0000","last_fullsized":"2023-11-28T20:33:59.589287+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:23.642548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122263+0000","last_change":"2023-11-28T20:33:53.950370+0000","last_active":"2023-11-28T20:33:58.122263+0000","last_peered":"2023-11-28T20:33:58.122263+0000","last_clean":"2023-11-28T20:33:58.122263+0000","last_became_active":"2023-11-28T20:33:53.946975+0000","last_became_peered":"2023-11-28T20:33:53.946975+0000","last_unstale":"2023-11-28T20:33:58.122263+0000","last_undegraded":"2023-11-28T20:33:58.122263+0000","last_fullsized":"2023-11-28T20:33:58.122263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:41.830889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123611+0000","last_change":"2023-11-28T20:33:53.062319+0000","last_active":"2023-11-28T20:33:58.123611+0000","last_peered":"2023-11-28T20:33:58.123611+0000","last_clean":"2023-11-28T20:33:58.123611+0000","last_became_active":"2023-11-28T20:33:53.062034+0000","last_became_peered":"2023-11-28T20:33:53.062034+0000","last_unstale":"2023-11-28T20:33:58.123611+0000","last_undegraded":"2023-11-28T20:33:58.123611+0000","last_fullsized":"2023-11-28T20:33:58.123611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590144+0000","last_change":"2023-11-28T20:33:58.129150+0000","last_active":"2023-11-28T20:33:59.590144+0000","last_peered":"2023-11-28T20:33:59.590144+0000","last_clean":"2023-11-28T20:33:59.590144+0000","last_became_active":"2023-11-28T20:33:58.128658+0000","last_became_peered":"2023-11-28T20:33:58.128658+0000","last_unstale":"2023-11-28T20:33:59.590144+0000","last_undegraded":"2023-11-28T20:33:59.590144+0000","last_fullsized":"2023-11-28T20:33:59.590144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:24.263445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152745+0000","last_change":"2023-11-28T20:33:53.964226+0000","last_active":"2023-11-28T20:33:58.152745+0000","last_peered":"2023-11-28T20:33:58.152745+0000","last_clean":"2023-11-28T20:33:58.152745+0000","last_became_active":"2023-11-28T20:33:53.963614+0000","last_became_peered":"2023-11-28T20:33:53.963614+0000","last_unstale":"2023-11-28T20:33:58.152745+0000","last_undegraded":"2023-11-28T20:33:58.152745+0000","last_fullsized":"2023-11-28T20:33:58.152745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:20.194268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099544+0000","last_change":"2023-11-28T20:33:53.049547+0000","last_active":"2023-11-28T20:33:58.099544+0000","last_peered":"2023-11-28T20:33:58.099544+0000","last_clean":"2023-11-28T20:33:58.099544+0000","last_became_active":"2023-11-28T20:33:53.049252+0000","last_became_peered":"2023-11-28T20:33:53.049252+0000","last_unstale":"2023-11-28T20:33:58.099544+0000","last_undegraded":"2023-11-28T20:33:58.099544+0000","last_fullsized":"2023-11-28T20:33:58.099544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:41.371596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.252105+0000","last_change":"2023-11-28T20:33:58.158216+0000","last_active":"2023-11-28T20:33:59.252105+0000","last_peered":"2023-11-28T20:33:59.252105+0000","last_clean":"2023-11-28T20:33:59.252105+0000","last_became_active":"2023-11-28T20:33:58.157621+0000","last_became_peered":"2023-11-28T20:33:58.157621+0000","last_unstale":"2023-11-28T20:33:59.252105+0000","last_undegraded":"2023-11-28T20:33:59.252105+0000","last_fullsized":"2023-11-28T20:33:59.252105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:43:13.638963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151918+0000","last_change":"2023-11-28T20:33:53.959472+0000","last_active":"2023-11-28T20:33:58.151918+0000","last_peered":"2023-11-28T20:33:58.151918+0000","last_clean":"2023-11-28T20:33:58.151918+0000","last_became_active":"2023-11-28T20:33:53.959110+0000","last_became_peered":"2023-11-28T20:33:53.959110+0000","last_unstale":"2023-11-28T20:33:58.151918+0000","last_undegraded":"2023-11-28T20:33:58.151918+0000","last_fullsized":"2023-11-28T20:33:58.151918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:41.830889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.104034+0000","last_change":"2023-11-28T20:33:53.047976+0000","last_active":"2023-11-28T20:33:58.104034+0000","last_peered":"2023-11-28T20:33:58.104034+0000","last_clean":"2023-11-28T20:33:58.104034+0000","last_became_active":"2023-11-28T20:33:53.047700+0000","last_became_peered":"2023-11-28T20:33:53.047700+0000","last_unstale":"2023-11-28T20:33:58.104034+0000","last_undegraded":"2023-11-28T20:33:58.104034+0000","last_fullsized":"2023-11-28T20:33:58.104034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:53.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916144+0000","last_change":"2023-11-28T20:33:58.128125+0000","last_active":"2023-11-28T20:33:58.916144+0000","last_peered":"2023-11-28T20:33:58.916144+0000","last_clean":"2023-11-28T20:33:58.916144+0000","last_became_active":"2023-11-28T20:33:58.127738+0000","last_became_peered":"2023-11-28T20:33:58.127738+0000","last_unstale":"2023-11-28T20:33:58.916144+0000","last_undegraded":"2023-11-28T20:33:58.916144+0000","last_fullsized":"2023-11-28T20:33:58.916144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:29.532639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151544+0000","last_change":"2023-11-28T20:33:53.960235+0000","last_active":"2023-11-28T20:33:58.151544+0000","last_peered":"2023-11-28T20:33:58.151544+0000","last_clean":"2023-11-28T20:33:58.151544+0000","last_became_active":"2023-11-28T20:33:53.959892+0000","last_became_peered":"2023-11-28T20:33:53.959892+0000","last_unstale":"2023-11-28T20:33:58.151544+0000","last_undegraded":"2023-11-28T20:33:58.151544+0000","last_fullsized":"2023-11-28T20:33:58.151544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099168+0000","last_change":"2023-11-28T20:33:53.056980+0000","last_active":"2023-11-28T20:33:58.099168+0000","last_peered":"2023-11-28T20:33:58.099168+0000","last_clean":"2023-11-28T20:33:58.099168+0000","last_became_active":"2023-11-28T20:33:53.056663+0000","last_became_peered":"2023-11-28T20:33:53.056663+0000","last_unstale":"2023-11-28T20:33:58.099168+0000","last_undegraded":"2023-11-28T20:33:58.099168+0000","last_fullsized":"2023-11-28T20:33:58.099168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:11:01.063288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590096+0000","last_change":"2023-11-28T20:33:58.137476+0000","last_active":"2023-11-28T20:33:59.590096+0000","last_peered":"2023-11-28T20:33:59.590096+0000","last_clean":"2023-11-28T20:33:59.590096+0000","last_became_active":"2023-11-28T20:33:58.136106+0000","last_became_peered":"2023-11-28T20:33:58.136106+0000","last_unstale":"2023-11-28T20:33:59.590096+0000","last_undegraded":"2023-11-28T20:33:59.590096+0000","last_fullsized":"2023-11-28T20:33:59.590096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:39.912968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119486+0000","last_change":"2023-11-28T20:33:53.940374+0000","last_active":"2023-11-28T20:33:58.119486+0000","last_peered":"2023-11-28T20:33:58.119486+0000","last_clean":"2023-11-28T20:33:58.119486+0000","last_became_active":"2023-11-28T20:33:53.939873+0000","last_became_peered":"2023-11-28T20:33:53.939873+0000","last_unstale":"2023-11-28T20:33:58.119486+0000","last_undegraded":"2023-11-28T20:33:58.119486+0000","last_fullsized":"2023-11-28T20:33:58.119486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120431+0000","last_change":"2023-11-28T20:33:53.055672+0000","last_active":"2023-11-28T20:33:58.120431+0000","last_peered":"2023-11-28T20:33:58.120431+0000","last_clean":"2023-11-28T20:33:58.120431+0000","last_became_active":"2023-11-28T20:33:53.055320+0000","last_became_peered":"2023-11-28T20:33:53.055320+0000","last_unstale":"2023-11-28T20:33:58.120431+0000","last_undegraded":"2023-11-28T20:33:58.120431+0000","last_fullsized":"2023-11-28T20:33:58.120431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.920861+0000","last_change":"2023-11-28T20:33:58.162629+0000","last_active":"2023-11-28T20:33:58.920861+0000","last_peered":"2023-11-28T20:33:58.920861+0000","last_clean":"2023-11-28T20:33:58.920861+0000","last_became_active":"2023-11-28T20:33:58.162003+0000","last_became_peered":"2023-11-28T20:33:58.162003+0000","last_unstale":"2023-11-28T20:33:58.920861+0000","last_undegraded":"2023-11-28T20:33:58.920861+0000","last_fullsized":"2023-11-28T20:33:58.920861+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:31.738218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151734+0000","last_change":"2023-11-28T20:33:53.961916+0000","last_active":"2023-11-28T20:33:58.151734+0000","last_peered":"2023-11-28T20:33:58.151734+0000","last_clean":"2023-11-28T20:33:58.151734+0000","last_became_active":"2023-11-28T20:33:53.961577+0000","last_became_peered":"2023-11-28T20:33:53.961577+0000","last_unstale":"2023-11-28T20:33:58.151734+0000","last_undegraded":"2023-11-28T20:33:58.151734+0000","last_fullsized":"2023-11-28T20:33:58.151734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126069+0000","last_change":"2023-11-28T20:33:53.056677+0000","last_active":"2023-11-28T20:33:58.126069+0000","last_peered":"2023-11-28T20:33:58.126069+0000","last_clean":"2023-11-28T20:33:58.126069+0000","last_became_active":"2023-11-28T20:33:53.055737+0000","last_became_peered":"2023-11-28T20:33:53.055737+0000","last_unstale":"2023-11-28T20:33:58.126069+0000","last_undegraded":"2023-11-28T20:33:58.126069+0000","last_fullsized":"2023-11-28T20:33:58.126069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.919758+0000","last_change":"2023-11-28T20:33:58.155270+0000","last_active":"2023-11-28T20:33:58.919758+0000","last_peered":"2023-11-28T20:33:58.919758+0000","last_clean":"2023-11-28T20:33:58.919758+0000","last_became_active":"2023-11-28T20:33:58.153764+0000","last_became_peered":"2023-11-28T20:33:58.153764+0000","last_unstale":"2023-11-28T20:33:58.919758+0000","last_undegraded":"2023-11-28T20:33:58.919758+0000","last_fullsized":"2023-11-28T20:33:58.919758+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:29.020740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117654+0000","last_change":"2023-11-28T20:33:53.943637+0000","last_active":"2023-11-28T20:33:58.117654+0000","last_peered":"2023-11-28T20:33:58.117654+0000","last_clean":"2023-11-28T20:33:58.117654+0000","last_became_active":"2023-11-28T20:33:53.941418+0000","last_became_peered":"2023-11-28T20:33:53.941418+0000","last_unstale":"2023-11-28T20:33:58.117654+0000","last_undegraded":"2023-11-28T20:33:58.117654+0000","last_fullsized":"2023-11-28T20:33:58.117654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118661+0000","last_change":"2023-11-28T20:33:53.052522+0000","last_active":"2023-11-28T20:33:58.118661+0000","last_peered":"2023-11-28T20:33:58.118661+0000","last_clean":"2023-11-28T20:33:58.118661+0000","last_became_active":"2023-11-28T20:33:53.052202+0000","last_became_peered":"2023-11-28T20:33:53.052202+0000","last_unstale":"2023-11-28T20:33:58.118661+0000","last_undegraded":"2023-11-28T20:33:58.118661+0000","last_fullsized":"2023-11-28T20:33:58.118661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:46.454394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.253371+0000","last_change":"2023-11-28T20:33:58.157854+0000","last_active":"2023-11-28T20:33:59.253371+0000","last_peered":"2023-11-28T20:33:59.253371+0000","last_clean":"2023-11-28T20:33:59.253371+0000","last_became_active":"2023-11-28T20:33:58.157424+0000","last_became_peered":"2023-11-28T20:33:58.157424+0000","last_unstale":"2023-11-28T20:33:59.253371+0000","last_undegraded":"2023-11-28T20:33:59.253371+0000","last_fullsized":"2023-11-28T20:33:59.253371+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:41.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151781+0000","last_change":"2023-11-28T20:33:53.959159+0000","last_active":"2023-11-28T20:33:58.151781+0000","last_peered":"2023-11-28T20:33:58.151781+0000","last_clean":"2023-11-28T20:33:58.151781+0000","last_became_active":"2023-11-28T20:33:53.958752+0000","last_became_peered":"2023-11-28T20:33:53.958752+0000","last_unstale":"2023-11-28T20:33:58.151781+0000","last_undegraded":"2023-11-28T20:33:58.151781+0000","last_fullsized":"2023-11-28T20:33:58.151781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:21.394618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:34:13.400104+0000","last_change":"2023-11-28T20:33:53.051304+0000","last_active":"2023-11-28T20:34:13.400104+0000","last_peered":"2023-11-28T20:34:13.400104+0000","last_clean":"2023-11-28T20:34:13.400104+0000","last_became_active":"2023-11-28T20:33:53.050957+0000","last_became_peered":"2023-11-28T20:33:53.050957+0000","last_unstale":"2023-11-28T20:34:13.400104+0000","last_undegraded":"2023-11-28T20:34:13.400104+0000","last_fullsized":"2023-11-28T20:34:13.400104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:41.755062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.915782+0000","last_change":"2023-11-28T20:33:58.133335+0000","last_active":"2023-11-28T20:33:58.915782+0000","last_peered":"2023-11-28T20:33:58.915782+0000","last_clean":"2023-11-28T20:33:58.915782+0000","last_became_active":"2023-11-28T20:33:58.132968+0000","last_became_peered":"2023-11-28T20:33:58.132968+0000","last_unstale":"2023-11-28T20:33:58.915782+0000","last_undegraded":"2023-11-28T20:33:58.915782+0000","last_fullsized":"2023-11-28T20:33:58.915782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105225+0000","last_change":"2023-11-28T20:33:53.944947+0000","last_active":"2023-11-28T20:33:58.105225+0000","last_peered":"2023-11-28T20:33:58.105225+0000","last_clean":"2023-11-28T20:33:58.105225+0000","last_became_active":"2023-11-28T20:33:53.944492+0000","last_became_peered":"2023-11-28T20:33:53.944492+0000","last_unstale":"2023-11-28T20:33:58.105225+0000","last_undegraded":"2023-11-28T20:33:58.105225+0000","last_fullsized":"2023-11-28T20:33:58.105225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:42.768692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123500+0000","last_change":"2023-11-28T20:33:53.063285+0000","last_active":"2023-11-28T20:33:58.123500+0000","last_peered":"2023-11-28T20:33:58.123500+0000","last_clean":"2023-11-28T20:33:58.123500+0000","last_became_active":"2023-11-28T20:33:53.063017+0000","last_became_peered":"2023-11-28T20:33:53.063017+0000","last_unstale":"2023-11-28T20:33:58.123500+0000","last_undegraded":"2023-11-28T20:33:58.123500+0000","last_fullsized":"2023-11-28T20:33:58.123500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917261+0000","last_change":"2023-11-28T20:33:58.125259+0000","last_active":"2023-11-28T20:33:58.917261+0000","last_peered":"2023-11-28T20:33:58.917261+0000","last_clean":"2023-11-28T20:33:58.917261+0000","last_became_active":"2023-11-28T20:33:58.124810+0000","last_became_peered":"2023-11-28T20:33:58.124810+0000","last_unstale":"2023-11-28T20:33:58.917261+0000","last_undegraded":"2023-11-28T20:33:58.917261+0000","last_fullsized":"2023-11-28T20:33:58.917261+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:22.122661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116919+0000","last_change":"2023-11-28T20:33:53.940927+0000","last_active":"2023-11-28T20:33:58.116919+0000","last_peered":"2023-11-28T20:33:58.116919+0000","last_clean":"2023-11-28T20:33:58.116919+0000","last_became_active":"2023-11-28T20:33:53.940527+0000","last_became_peered":"2023-11-28T20:33:53.940527+0000","last_unstale":"2023-11-28T20:33:58.116919+0000","last_undegraded":"2023-11-28T20:33:58.116919+0000","last_fullsized":"2023-11-28T20:33:58.116919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123801+0000","last_change":"2023-11-28T20:33:53.055804+0000","last_active":"2023-11-28T20:33:58.123801+0000","last_peered":"2023-11-28T20:33:58.123801+0000","last_clean":"2023-11-28T20:33:58.123801+0000","last_became_active":"2023-11-28T20:33:53.053184+0000","last_became_peered":"2023-11-28T20:33:53.053184+0000","last_unstale":"2023-11-28T20:33:58.123801+0000","last_undegraded":"2023-11-28T20:33:58.123801+0000","last_fullsized":"2023-11-28T20:33:58.123801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:57.471000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918702+0000","last_change":"2023-11-28T20:33:58.133079+0000","last_active":"2023-11-28T20:33:58.918702+0000","last_peered":"2023-11-28T20:33:58.918702+0000","last_clean":"2023-11-28T20:33:58.918702+0000","last_became_active":"2023-11-28T20:33:58.126051+0000","last_became_peered":"2023-11-28T20:33:58.126051+0000","last_unstale":"2023-11-28T20:33:58.918702+0000","last_undegraded":"2023-11-28T20:33:58.918702+0000","last_fullsized":"2023-11-28T20:33:58.918702+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:41.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123681+0000","last_change":"2023-11-28T20:33:53.958488+0000","last_active":"2023-11-28T20:33:58.123681+0000","last_peered":"2023-11-28T20:33:58.123681+0000","last_clean":"2023-11-28T20:33:58.123681+0000","last_became_active":"2023-11-28T20:33:53.958041+0000","last_became_peered":"2023-11-28T20:33:53.958041+0000","last_unstale":"2023-11-28T20:33:58.123681+0000","last_undegraded":"2023-11-28T20:33:58.123681+0000","last_fullsized":"2023-11-28T20:33:58.123681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:07.426176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126850+0000","last_change":"2023-11-28T20:33:53.054003+0000","last_active":"2023-11-28T20:33:58.126850+0000","last_peered":"2023-11-28T20:33:58.126850+0000","last_clean":"2023-11-28T20:33:58.126850+0000","last_became_active":"2023-11-28T20:33:53.053773+0000","last_became_peered":"2023-11-28T20:33:53.053773+0000","last_unstale":"2023-11-28T20:33:58.126850+0000","last_undegraded":"2023-11-28T20:33:58.126850+0000","last_fullsized":"2023-11-28T20:33:58.126850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918600+0000","last_change":"2023-11-28T20:33:58.157661+0000","last_active":"2023-11-28T20:33:58.918600+0000","last_peered":"2023-11-28T20:33:58.918600+0000","last_clean":"2023-11-28T20:33:58.918600+0000","last_became_active":"2023-11-28T20:33:58.157274+0000","last_became_peered":"2023-11-28T20:33:58.157274+0000","last_unstale":"2023-11-28T20:33:58.918600+0000","last_undegraded":"2023-11-28T20:33:58.918600+0000","last_fullsized":"2023-11-28T20:33:58.918600+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:43:13.638963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589987+0000","last_change":"2023-11-28T20:33:58.159599+0000","last_active":"2023-11-28T20:33:59.589987+0000","last_peered":"2023-11-28T20:33:59.589987+0000","last_clean":"2023-11-28T20:33:59.589987+0000","last_became_active":"2023-11-28T20:33:58.158483+0000","last_became_peered":"2023-11-28T20:33:58.158483+0000","last_unstale":"2023-11-28T20:33:59.589987+0000","last_undegraded":"2023-11-28T20:33:59.589987+0000","last_fullsized":"2023-11-28T20:33:59.589987+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:45.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122123+0000","last_change":"2023-11-28T20:33:53.950947+0000","last_active":"2023-11-28T20:33:58.122123+0000","last_peered":"2023-11-28T20:33:58.122123+0000","last_clean":"2023-11-28T20:33:58.122123+0000","last_became_active":"2023-11-28T20:33:53.948976+0000","last_became_peered":"2023-11-28T20:33:53.948976+0000","last_unstale":"2023-11-28T20:33:58.122123+0000","last_undegraded":"2023-11-28T20:33:58.122123+0000","last_fullsized":"2023-11-28T20:33:58.122123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120814+0000","last_change":"2023-11-28T20:33:53.059457+0000","last_active":"2023-11-28T20:33:58.120814+0000","last_peered":"2023-11-28T20:33:58.120814+0000","last_clean":"2023-11-28T20:33:58.120814+0000","last_became_active":"2023-11-28T20:33:53.059116+0000","last_became_peered":"2023-11-28T20:33:53.059116+0000","last_unstale":"2023-11-28T20:33:58.120814+0000","last_undegraded":"2023-11-28T20:33:58.120814+0000","last_fullsized":"2023-11-28T20:33:58.120814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:32.036639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590492+0000","last_change":"2023-11-28T20:33:58.141507+0000","last_active":"2023-11-28T20:33:59.590492+0000","last_peered":"2023-11-28T20:33:59.590492+0000","last_clean":"2023-11-28T20:33:59.590492+0000","last_became_active":"2023-11-28T20:33:58.141072+0000","last_became_peered":"2023-11-28T20:33:58.141072+0000","last_unstale":"2023-11-28T20:33:59.590492+0000","last_undegraded":"2023-11-28T20:33:59.590492+0000","last_fullsized":"2023-11-28T20:33:59.590492+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103356+0000","last_change":"2023-11-28T20:33:53.937448+0000","last_active":"2023-11-28T20:33:58.103356+0000","last_peered":"2023-11-28T20:33:58.103356+0000","last_clean":"2023-11-28T20:33:58.103356+0000","last_became_active":"2023-11-28T20:33:53.937097+0000","last_became_peered":"2023-11-28T20:33:53.937097+0000","last_unstale":"2023-11-28T20:33:58.103356+0000","last_undegraded":"2023-11-28T20:33:58.103356+0000","last_fullsized":"2023-11-28T20:33:58.103356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105061+0000","last_change":"2023-11-28T20:33:53.047089+0000","last_active":"2023-11-28T20:33:58.105061+0000","last_peered":"2023-11-28T20:33:58.105061+0000","last_clean":"2023-11-28T20:33:58.105061+0000","last_became_active":"2023-11-28T20:33:53.046299+0000","last_became_peered":"2023-11-28T20:33:53.046299+0000","last_unstale":"2023-11-28T20:33:58.105061+0000","last_undegraded":"2023-11-28T20:33:58.105061+0000","last_fullsized":"2023-11-28T20:33:58.105061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121518+0000","last_change":"2023-11-28T20:33:53.057990+0000","last_active":"2023-11-28T20:33:58.121518+0000","last_peered":"2023-11-28T20:33:58.121518+0000","last_clean":"2023-11-28T20:33:58.121518+0000","last_became_active":"2023-11-28T20:33:53.057437+0000","last_became_peered":"2023-11-28T20:33:53.057437+0000","last_unstale":"2023-11-28T20:33:58.121518+0000","last_undegraded":"2023-11-28T20:33:58.121518+0000","last_fullsized":"2023-11-28T20:33:58.121518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118455+0000","last_change":"2023-11-28T20:33:53.944979+0000","last_active":"2023-11-28T20:33:58.118455+0000","last_peered":"2023-11-28T20:33:58.118455+0000","last_clean":"2023-11-28T20:33:58.118455+0000","last_became_active":"2023-11-28T20:33:53.944587+0000","last_became_peered":"2023-11-28T20:33:53.944587+0000","last_unstale":"2023-11-28T20:33:58.118455+0000","last_undegraded":"2023-11-28T20:33:58.118455+0000","last_fullsized":"2023-11-28T20:33:58.118455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.226534+0000","last_change":"2023-11-28T20:33:58.128961+0000","last_active":"2023-11-28T20:33:59.226534+0000","last_peered":"2023-11-28T20:33:59.226534+0000","last_clean":"2023-11-28T20:33:59.226534+0000","last_became_active":"2023-11-28T20:33:58.128548+0000","last_became_peered":"2023-11-28T20:33:58.128548+0000","last_unstale":"2023-11-28T20:33:59.226534+0000","last_undegraded":"2023-11-28T20:33:59.226534+0000","last_fullsized":"2023-11-28T20:33:59.226534+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:39.520026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103995+0000","last_change":"2023-11-28T20:33:53.061966+0000","last_active":"2023-11-28T20:33:58.103995+0000","last_peered":"2023-11-28T20:33:58.103995+0000","last_clean":"2023-11-28T20:33:58.103995+0000","last_became_active":"2023-11-28T20:33:53.061538+0000","last_became_peered":"2023-11-28T20:33:53.061538+0000","last_unstale":"2023-11-28T20:33:58.103995+0000","last_undegraded":"2023-11-28T20:33:58.103995+0000","last_fullsized":"2023-11-28T20:33:58.103995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.113040+0000","last_change":"2023-11-28T20:33:53.943755+0000","last_active":"2023-11-28T20:33:58.113040+0000","last_peered":"2023-11-28T20:33:58.113040+0000","last_clean":"2023-11-28T20:33:58.113040+0000","last_became_active":"2023-11-28T20:33:53.943464+0000","last_became_peered":"2023-11-28T20:33:53.943464+0000","last_unstale":"2023-11-28T20:33:58.113040+0000","last_undegraded":"2023-11-28T20:33:58.113040+0000","last_fullsized":"2023-11-28T20:33:58.113040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:42.385226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.919987+0000","last_change":"2023-11-28T20:33:58.142246+0000","last_active":"2023-11-28T20:33:58.919987+0000","last_peered":"2023-11-28T20:33:58.919987+0000","last_clean":"2023-11-28T20:33:58.919987+0000","last_became_active":"2023-11-28T20:33:58.134395+0000","last_became_peered":"2023-11-28T20:33:58.134395+0000","last_unstale":"2023-11-28T20:33:58.919987+0000","last_undegraded":"2023-11-28T20:33:58.919987+0000","last_fullsized":"2023-11-28T20:33:58.919987+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:24:08.097260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119379+0000","last_change":"2023-11-28T20:33:53.948082+0000","last_active":"2023-11-28T20:33:58.119379+0000","last_peered":"2023-11-28T20:33:58.119379+0000","last_clean":"2023-11-28T20:33:58.119379+0000","last_became_active":"2023-11-28T20:33:53.947760+0000","last_became_peered":"2023-11-28T20:33:53.947760+0000","last_unstale":"2023-11-28T20:33:58.119379+0000","last_undegraded":"2023-11-28T20:33:58.119379+0000","last_fullsized":"2023-11-28T20:33:58.119379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:35.450849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099225+0000","last_change":"2023-11-28T20:33:53.947745+0000","last_active":"2023-11-28T20:33:58.099225+0000","last_peered":"2023-11-28T20:33:58.099225+0000","last_clean":"2023-11-28T20:33:58.099225+0000","last_became_active":"2023-11-28T20:33:53.947452+0000","last_became_peered":"2023-11-28T20:33:53.947452+0000","last_unstale":"2023-11-28T20:33:58.099225+0000","last_undegraded":"2023-11-28T20:33:58.099225+0000","last_fullsized":"2023-11-28T20:33:58.099225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:42.768692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125984+0000","last_change":"2023-11-28T20:33:53.947868+0000","last_active":"2023-11-28T20:33:58.125984+0000","last_peered":"2023-11-28T20:33:58.125984+0000","last_clean":"2023-11-28T20:33:58.125984+0000","last_became_active":"2023-11-28T20:33:53.947304+0000","last_became_peered":"2023-11-28T20:33:53.947304+0000","last_unstale":"2023-11-28T20:33:58.125984+0000","last_undegraded":"2023-11-28T20:33:58.125984+0000","last_fullsized":"2023-11-28T20:33:58.125984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118368+0000","last_change":"2023-11-28T20:33:53.947122+0000","last_active":"2023-11-28T20:33:58.118368+0000","last_peered":"2023-11-28T20:33:58.118368+0000","last_clean":"2023-11-28T20:33:58.118368+0000","last_became_active":"2023-11-28T20:33:53.946556+0000","last_became_peered":"2023-11-28T20:33:53.946556+0000","last_unstale":"2023-11-28T20:33:58.118368+0000","last_undegraded":"2023-11-28T20:33:58.118368+0000","last_fullsized":"2023-11-28T20:33:58.118368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.109664+0000","last_change":"2023-11-28T20:33:53.946180+0000","last_active":"2023-11-28T20:33:58.109664+0000","last_peered":"2023-11-28T20:33:58.109664+0000","last_clean":"2023-11-28T20:33:58.109664+0000","last_became_active":"2023-11-28T20:33:53.945758+0000","last_became_peered":"2023-11-28T20:33:53.945758+0000","last_unstale":"2023-11-28T20:33:58.109664+0000","last_undegraded":"2023-11-28T20:33:58.109664+0000","last_fullsized":"2023-11-28T20:33:58.109664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123704+0000","last_change":"2023-11-28T20:33:53.950532+0000","last_active":"2023-11-28T20:33:58.123704+0000","last_peered":"2023-11-28T20:33:58.123704+0000","last_clean":"2023-11-28T20:33:58.123704+0000","last_became_active":"2023-11-28T20:33:53.950206+0000","last_became_peered":"2023-11-28T20:33:53.950206+0000","last_unstale":"2023-11-28T20:33:58.123704+0000","last_undegraded":"2023-11-28T20:33:58.123704+0000","last_fullsized":"2023-11-28T20:33:58.123704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126244+0000","last_change":"2023-11-28T20:33:53.958413+0000","last_active":"2023-11-28T20:33:58.126244+0000","last_peered":"2023-11-28T20:33:58.126244+0000","last_clean":"2023-11-28T20:33:58.126244+0000","last_became_active":"2023-11-28T20:33:53.958179+0000","last_became_peered":"2023-11-28T20:33:53.958179+0000","last_unstale":"2023-11-28T20:33:58.126244+0000","last_undegraded":"2023-11-28T20:33:58.126244+0000","last_fullsized":"2023-11-28T20:33:58.126244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.113900+0000","last_change":"2023-11-28T20:33:53.952207+0000","last_active":"2023-11-28T20:33:58.113900+0000","last_peered":"2023-11-28T20:33:58.113900+0000","last_clean":"2023-11-28T20:33:58.113900+0000","last_became_active":"2023-11-28T20:33:53.950011+0000","last_became_peered":"2023-11-28T20:33:53.950011+0000","last_unstale":"2023-11-28T20:33:58.113900+0000","last_undegraded":"2023-11-28T20:33:58.113900+0000","last_fullsized":"2023-11-28T20:33:58.113900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120660+0000","last_change":"2023-11-28T20:33:53.952356+0000","last_active":"2023-11-28T20:33:58.120660+0000","last_peered":"2023-11-28T20:33:58.120660+0000","last_clean":"2023-11-28T20:33:58.120660+0000","last_became_active":"2023-11-28T20:33:53.947787+0000","last_became_peered":"2023-11-28T20:33:53.947787+0000","last_unstale":"2023-11-28T20:33:58.120660+0000","last_undegraded":"2023-11-28T20:33:58.120660+0000","last_fullsized":"2023-11-28T20:33:58.120660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105059+0000","last_change":"2023-11-28T20:33:53.949973+0000","last_active":"2023-11-28T20:33:58.105059+0000","last_peered":"2023-11-28T20:33:58.105059+0000","last_clean":"2023-11-28T20:33:58.105059+0000","last_became_active":"2023-11-28T20:33:53.946758+0000","last_became_peered":"2023-11-28T20:33:53.946758+0000","last_unstale":"2023-11-28T20:33:58.105059+0000","last_undegraded":"2023-11-28T20:33:58.105059+0000","last_fullsized":"2023-11-28T20:33:58.105059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125895+0000","last_change":"2023-11-28T20:33:53.950994+0000","last_active":"2023-11-28T20:33:58.125895+0000","last_peered":"2023-11-28T20:33:58.125895+0000","last_clean":"2023-11-28T20:33:58.125895+0000","last_became_active":"2023-11-28T20:33:53.946524+0000","last_became_peered":"2023-11-28T20:33:53.946524+0000","last_unstale":"2023-11-28T20:33:58.125895+0000","last_undegraded":"2023-11-28T20:33:58.125895+0000","last_fullsized":"2023-11-28T20:33:58.125895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118275+0000","last_change":"2023-11-28T20:33:53.939684+0000","last_active":"2023-11-28T20:33:58.118275+0000","last_peered":"2023-11-28T20:33:58.118275+0000","last_clean":"2023-11-28T20:33:58.118275+0000","last_became_active":"2023-11-28T20:33:53.939225+0000","last_became_peered":"2023-11-28T20:33:53.939225+0000","last_unstale":"2023-11-28T20:33:58.118275+0000","last_undegraded":"2023-11-28T20:33:58.118275+0000","last_fullsized":"2023-11-28T20:33:58.118275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125812+0000","last_change":"2023-11-28T20:33:53.951199+0000","last_active":"2023-11-28T20:33:58.125812+0000","last_peered":"2023-11-28T20:33:58.125812+0000","last_clean":"2023-11-28T20:33:58.125812+0000","last_became_active":"2023-11-28T20:33:53.949262+0000","last_became_peered":"2023-11-28T20:33:53.949262+0000","last_unstale":"2023-11-28T20:33:58.125812+0000","last_undegraded":"2023-11-28T20:33:58.125812+0000","last_fullsized":"2023-11-28T20:33:58.125812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152008+0000","last_change":"2023-11-28T20:33:53.958492+0000","last_active":"2023-11-28T20:33:58.152008+0000","last_peered":"2023-11-28T20:33:58.152008+0000","last_clean":"2023-11-28T20:33:58.152008+0000","last_became_active":"2023-11-28T20:33:53.958141+0000","last_became_peered":"2023-11-28T20:33:53.958141+0000","last_unstale":"2023-11-28T20:33:58.152008+0000","last_undegraded":"2023-11-28T20:33:58.152008+0000","last_fullsized":"2023-11-28T20:33:58.152008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:13.738167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099160+0000","last_change":"2023-11-28T20:33:53.948508+0000","last_active":"2023-11-28T20:33:58.099160+0000","last_peered":"2023-11-28T20:33:58.099160+0000","last_clean":"2023-11-28T20:33:58.099160+0000","last_became_active":"2023-11-28T20:33:53.947303+0000","last_became_peered":"2023-11-28T20:33:53.947303+0000","last_unstale":"2023-11-28T20:33:58.099160+0000","last_undegraded":"2023-11-28T20:33:58.099160+0000","last_fullsized":"2023-11-28T20:33:58.099160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:23.240825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.127814+0000","last_change":"2023-11-28T20:33:53.939826+0000","last_active":"2023-11-28T20:33:58.127814+0000","last_peered":"2023-11-28T20:33:58.127814+0000","last_clean":"2023-11-28T20:33:58.127814+0000","last_became_active":"2023-11-28T20:33:53.939349+0000","last_became_peered":"2023-11-28T20:33:53.939349+0000","last_unstale":"2023-11-28T20:33:58.127814+0000","last_undegraded":"2023-11-28T20:33:58.127814+0000","last_fullsized":"2023-11-28T20:33:58.127814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:13.738167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120426+0000","last_change":"2023-11-28T20:33:53.939569+0000","last_active":"2023-11-28T20:33:58.120426+0000","last_peered":"2023-11-28T20:33:58.120426+0000","last_clean":"2023-11-28T20:33:58.120426+0000","last_became_active":"2023-11-28T20:33:53.936362+0000","last_became_peered":"2023-11-28T20:33:53.936362+0000","last_unstale":"2023-11-28T20:33:58.120426+0000","last_undegraded":"2023-11-28T20:33:58.120426+0000","last_fullsized":"2023-11-28T20:33:58.120426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123031+0000","last_change":"2023-11-28T20:33:53.950757+0000","last_active":"2023-11-28T20:33:58.123031+0000","last_peered":"2023-11-28T20:33:58.123031+0000","last_clean":"2023-11-28T20:33:58.123031+0000","last_became_active":"2023-11-28T20:33:53.948014+0000","last_became_peered":"2023-11-28T20:33:53.948014+0000","last_unstale":"2023-11-28T20:33:58.123031+0000","last_undegraded":"2023-11-28T20:33:58.123031+0000","last_fullsized":"2023-11-28T20:33:58.123031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117566+0000","last_change":"2023-11-28T20:33:53.938422+0000","last_active":"2023-11-28T20:33:58.117566+0000","last_peered":"2023-11-28T20:33:58.117566+0000","last_clean":"2023-11-28T20:33:58.117566+0000","last_became_active":"2023-11-28T20:33:53.936194+0000","last_became_peered":"2023-11-28T20:33:53.936194+0000","last_unstale":"2023-11-28T20:33:58.117566+0000","last_undegraded":"2023-11-28T20:33:58.117566+0000","last_fullsized":"2023-11-28T20:33:58.117566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:35.450849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103553+0000","last_change":"2023-11-28T20:33:53.946481+0000","last_active":"2023-11-28T20:33:58.103553+0000","last_peered":"2023-11-28T20:33:58.103553+0000","last_clean":"2023-11-28T20:33:58.103553+0000","last_became_active":"2023-11-28T20:33:53.946123+0000","last_became_peered":"2023-11-28T20:33:53.946123+0000","last_unstale":"2023-11-28T20:33:58.103553+0000","last_undegraded":"2023-11-28T20:33:58.103553+0000","last_fullsized":"2023-11-28T20:33:58.103553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105089+0000","last_change":"2023-11-28T20:33:53.960696+0000","last_active":"2023-11-28T20:33:58.105089+0000","last_peered":"2023-11-28T20:33:58.105089+0000","last_clean":"2023-11-28T20:33:58.105089+0000","last_became_active":"2023-11-28T20:33:53.953779+0000","last_became_peered":"2023-11-28T20:33:53.953779+0000","last_unstale":"2023-11-28T20:33:58.105089+0000","last_undegraded":"2023-11-28T20:33:58.105089+0000","last_fullsized":"2023-11-28T20:33:58.105089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:47.468024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126504+0000","last_change":"2023-11-28T20:33:53.938502+0000","last_active":"2023-11-28T20:33:58.126504+0000","last_peered":"2023-11-28T20:33:58.126504+0000","last_clean":"2023-11-28T20:33:58.126504+0000","last_became_active":"2023-11-28T20:33:53.938133+0000","last_became_peered":"2023-11-28T20:33:53.938133+0000","last_unstale":"2023-11-28T20:33:58.126504+0000","last_undegraded":"2023-11-28T20:33:58.126504+0000","last_fullsized":"2023-11-28T20:33:58.126504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126197+0000","last_change":"2023-11-28T20:33:53.961946+0000","last_active":"2023-11-28T20:33:58.126197+0000","last_peered":"2023-11-28T20:33:58.126197+0000","last_clean":"2023-11-28T20:33:58.126197+0000","last_became_active":"2023-11-28T20:33:53.961681+0000","last_became_peered":"2023-11-28T20:33:53.961681+0000","last_unstale":"2023-11-28T20:33:58.126197+0000","last_undegraded":"2023-11-28T20:33:58.126197+0000","last_fullsized":"2023-11-28T20:33:58.126197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152700+0000","last_change":"2023-11-28T20:33:53.963253+0000","last_active":"2023-11-28T20:33:58.152700+0000","last_peered":"2023-11-28T20:33:58.152700+0000","last_clean":"2023-11-28T20:33:58.152700+0000","last_became_active":"2023-11-28T20:33:53.961179+0000","last_became_peered":"2023-11-28T20:33:53.961179+0000","last_unstale":"2023-11-28T20:33:58.152700+0000","last_undegraded":"2023-11-28T20:33:58.152700+0000","last_fullsized":"2023-11-28T20:33:58.152700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:35.843791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103213+0000","last_change":"2023-11-28T20:33:53.952078+0000","last_active":"2023-11-28T20:33:58.103213+0000","last_peered":"2023-11-28T20:33:58.103213+0000","last_clean":"2023-11-28T20:33:58.103213+0000","last_became_active":"2023-11-28T20:33:53.949626+0000","last_became_peered":"2023-11-28T20:33:53.949626+0000","last_unstale":"2023-11-28T20:33:58.103213+0000","last_undegraded":"2023-11-28T20:33:58.103213+0000","last_fullsized":"2023-11-28T20:33:58.103213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151515+0000","last_change":"2023-11-28T20:33:53.955688+0000","last_active":"2023-11-28T20:33:58.151515+0000","last_peered":"2023-11-28T20:33:58.151515+0000","last_clean":"2023-11-28T20:33:58.151515+0000","last_became_active":"2023-11-28T20:33:53.955384+0000","last_became_peered":"2023-11-28T20:33:53.955384+0000","last_unstale":"2023-11-28T20:33:58.151515+0000","last_undegraded":"2023-11-28T20:33:58.151515+0000","last_fullsized":"2023-11-28T20:33:58.151515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125802+0000","last_change":"2023-11-28T20:33:53.958084+0000","last_active":"2023-11-28T20:33:58.125802+0000","last_peered":"2023-11-28T20:33:58.125802+0000","last_clean":"2023-11-28T20:33:58.125802+0000","last_became_active":"2023-11-28T20:33:53.953483+0000","last_became_peered":"2023-11-28T20:33:53.953483+0000","last_unstale":"2023-11-28T20:33:58.125802+0000","last_undegraded":"2023-11-28T20:33:58.125802+0000","last_fullsized":"2023-11-28T20:33:58.125802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151629+0000","last_change":"2023-11-28T20:33:53.955124+0000","last_active":"2023-11-28T20:33:58.151629+0000","last_peered":"2023-11-28T20:33:58.151629+0000","last_clean":"2023-11-28T20:33:58.151629+0000","last_became_active":"2023-11-28T20:33:53.954422+0000","last_became_peered":"2023-11-28T20:33:53.954422+0000","last_unstale":"2023-11-28T20:33:58.151629+0000","last_undegraded":"2023-11-28T20:33:58.151629+0000","last_fullsized":"2023-11-28T20:33:58.151629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:19.573371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121991+0000","last_change":"2023-11-28T20:33:53.957855+0000","last_active":"2023-11-28T20:33:58.121991+0000","last_peered":"2023-11-28T20:33:58.121991+0000","last_clean":"2023-11-28T20:33:58.121991+0000","last_became_active":"2023-11-28T20:33:53.955645+0000","last_became_peered":"2023-11-28T20:33:53.955645+0000","last_unstale":"2023-11-28T20:33:58.121991+0000","last_undegraded":"2023-11-28T20:33:58.121991+0000","last_fullsized":"2023-11-28T20:33:58.121991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116851+0000","last_change":"2023-11-28T20:33:53.937690+0000","last_active":"2023-11-28T20:33:58.116851+0000","last_peered":"2023-11-28T20:33:58.116851+0000","last_clean":"2023-11-28T20:33:58.116851+0000","last_became_active":"2023-11-28T20:33:53.937374+0000","last_became_peered":"2023-11-28T20:33:53.937374+0000","last_unstale":"2023-11-28T20:33:58.116851+0000","last_undegraded":"2023-11-28T20:33:58.116851+0000","last_fullsized":"2023-11-28T20:33:58.116851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123601+0000","last_change":"2023-11-28T20:33:53.950129+0000","last_active":"2023-11-28T20:33:58.123601+0000","last_peered":"2023-11-28T20:33:58.123601+0000","last_clean":"2023-11-28T20:33:58.123601+0000","last_became_active":"2023-11-28T20:33:53.947280+0000","last_became_peered":"2023-11-28T20:33:53.947280+0000","last_unstale":"2023-11-28T20:33:58.123601+0000","last_undegraded":"2023-11-28T20:33:58.123601+0000","last_fullsized":"2023-11-28T20:33:58.123601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152690+0000","last_change":"2023-11-28T20:33:53.964432+0000","last_active":"2023-11-28T20:33:58.152690+0000","last_peered":"2023-11-28T20:33:58.152690+0000","last_clean":"2023-11-28T20:33:58.152690+0000","last_became_active":"2023-11-28T20:33:53.963967+0000","last_became_peered":"2023-11-28T20:33:53.963967+0000","last_unstale":"2023-11-28T20:33:58.152690+0000","last_undegraded":"2023-11-28T20:33:58.152690+0000","last_fullsized":"2023-11-28T20:33:58.152690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:07.426176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":695413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6220000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":104463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":274432,"data_stored":104623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":695469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6120000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.093}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:34:24.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:34:24.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.767+0000 7fa95b393640 1 Processor -- start 2023-11-28T20:34:24.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.767+0000 7fa95b393640 1 -- start start 2023-11-28T20:34:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.769+0000 7fa95b393640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa9540b2830 0x7fa9540b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.769+0000 7fa95b393640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 0x7fa95406d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.769+0000 7fa95b393640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9540708d0 con 0x7fa954159650 2023-11-28T20:34:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.769+0000 7fa95b393640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9540b31b0 con 0x7fa9540b2830 2023-11-28T20:34:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.769+0000 7fa95b393640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95406c390 con 0x7fa95406cf30 2023-11-28T20:34:24.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.769+0000 7fa959b90640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa9540b2830 0x7fa9540b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.769+0000 7fa959b90640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa9540b2830 0x7fa9540b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58706/0 (socket says 172.21.15.45:58706) 2023-11-28T20:34:24.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.769+0000 7fa959b90640 1 -- 172.21.15.45:0/2899536757 learned_addr learned my addr 172.21.15.45:0/2899536757 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.770+0000 7fa95ab92640 1 --2- 172.21.15.45:0/2899536757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 0x7fa95406d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.770+0000 7fa959b90640 1 -- 172.21.15.45:0/2899536757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 msgr2=0x7fa95406d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.770+0000 7fa959b90640 1 --2- 172.21.15.45:0/2899536757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 0x7fa95406d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.770+0000 7fa959b90640 1 -- 172.21.15.45:0/2899536757 >> v1:172.21.15.190:6789/0 conn(0x7fa954159650 legacy=0x7fa9540abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.770+0000 7fa959b90640 1 -- 172.21.15.45:0/2899536757 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95406d8e0 con 0x7fa9540b2830 2023-11-28T20:34:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.771+0000 7fa959b90640 1 --2- 172.21.15.45:0/2899536757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9540b2830 0x7fa9540b2c70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa944009870 tx=0x7fa94402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc78087f670f8876 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:24.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.771+0000 7fa95938f640 1 -- 172.21.15.45:0/2899536757 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944031640 con 0x7fa9540b2830 2023-11-28T20:34:24.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.771+0000 7fa95938f640 1 -- 172.21.15.45:0/2899536757 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9440317a0 con 0x7fa9540b2830 2023-11-28T20:34:24.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.771+0000 7fa95938f640 1 -- 172.21.15.45:0/2899536757 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944031b30 con 0x7fa9540b2830 2023-11-28T20:34:24.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.772+0000 7fa95b393640 1 -- 172.21.15.45:0/2899536757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9540b2830 msgr2=0x7fa9540b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.772+0000 7fa95b393640 1 --2- 172.21.15.45:0/2899536757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9540b2830 0x7fa9540b2c70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa944009870 tx=0x7fa94402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.772+0000 7fa95b393640 1 -- 172.21.15.45:0/2899536757 shutdown_connections 2023-11-28T20:34:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.772+0000 7fa95b393640 1 --2- 172.21.15.45:0/2899536757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 0x7fa95406d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.772+0000 7fa95b393640 1 --2- 172.21.15.45:0/2899536757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9540b2830 0x7fa9540b2c70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.772+0000 7fa95b393640 1 -- 172.21.15.45:0/2899536757 >> 172.21.15.45:0/2899536757 conn(0x7fa95406ab30 msgr2=0x7fa9540b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.772+0000 7fa95b393640 1 -- 172.21.15.45:0/2899536757 shutdown_connections 2023-11-28T20:34:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.772+0000 7fa95b393640 1 -- 172.21.15.45:0/2899536757 wait complete. 2023-11-28T20:34:24.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.774+0000 7fa95b393640 1 Processor -- start 2023-11-28T20:34:24.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.774+0000 7fa95b393640 1 -- start start 2023-11-28T20:34:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.775+0000 7fa95b393640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 0x7fa95415c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.775+0000 7fa95b393640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9540b2830 0x7fa9541479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.775+0000 7fa95b393640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa954159650 0x7fa9540afe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.775+0000 7fa95b393640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95415ac50 con 0x7fa954159650 2023-11-28T20:34:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.775+0000 7fa95b393640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95415af30 con 0x7fa9540b2830 2023-11-28T20:34:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.775+0000 7fa95b393640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa954166db0 con 0x7fa95406cf30 2023-11-28T20:34:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95ab92640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa954159650 0x7fa9540afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95ab92640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa954159650 0x7fa9540afe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58718/0 (socket says 172.21.15.45:58718) 2023-11-28T20:34:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95ab92640 1 -- 172.21.15.45:0/858223827 learned_addr learned my addr 172.21.15.45:0/858223827 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa959b90640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9540b2830 0x7fa9541479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95a391640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 0x7fa95415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95ab92640 1 -- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 msgr2=0x7fa95415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95ab92640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 0x7fa95415c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95ab92640 1 -- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9540b2830 msgr2=0x7fa9541479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95ab92640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9540b2830 0x7fa9541479d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95ab92640 1 -- 172.21.15.45:0/858223827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9440094d0 con 0x7fa954159650 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa959b90640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9540b2830 0x7fa9541479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95a391640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 0x7fa95415c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.776+0000 7fa95ab92640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa954159650 0x7fa9540afe60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00cc30 tx=0x7fa94c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.777+0000 7fa94affd640 1 -- 172.21.15.45:0/858223827 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c00ce40 con 0x7fa954159650 2023-11-28T20:34:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.777+0000 7fa94affd640 1 -- 172.21.15.45:0/858223827 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c003990 con 0x7fa954159650 2023-11-28T20:34:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.777+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa954167090 con 0x7fa954159650 2023-11-28T20:34:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.777+0000 7fa94affd640 1 -- 172.21.15.45:0/858223827 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c003d20 con 0x7fa954159650 2023-11-28T20:34:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.777+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95415b210 con 0x7fa954159650 2023-11-28T20:34:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.778+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9541672d0 con 0x7fa954159650 2023-11-28T20:34:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.779+0000 7fa94affd640 1 -- 172.21.15.45:0/858223827 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa94c003e80 con 0x7fa954159650 2023-11-28T20:34:24.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.781+0000 7fa94affd640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa93806ae40 0x7fa93806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:24.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.781+0000 7fa94affd640 1 -- 172.21.15.45:0/858223827 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa94c089ce0 con 0x7fa954159650 2023-11-28T20:34:24.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.782+0000 7fa95a391640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa93806ae40 0x7fa93806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:24.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.783+0000 7fa95a391640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa93806ae40 0x7fa93806d300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa9541666b0 tx=0x7fa950009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:24.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.785+0000 7fa94affd640 1 -- 172.21.15.45:0/858223827 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9541672d0 con 0x7fa954159650 2023-11-28T20:34:24.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:24.975+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa95415b0a0 con 0x7fa93806ae40 2023-11-28T20:34:25.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.001+0000 7fa94affd640 1 -- 172.21.15.45:0/858223827 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345164 (secure 0 0 0) 0x7fa95415b0a0 con 0x7fa93806ae40 2023-11-28T20:34:25.005 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:34:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa93806ae40 msgr2=0x7fa93806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa93806ae40 0x7fa93806d300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa9541666b0 tx=0x7fa950009040 comp rx=0 tx=0).stop 2023-11-28T20:34:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa954159650 msgr2=0x7fa9540afe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa954159650 0x7fa9540afe60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00cc30 tx=0x7fa94c00a5c0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 shutdown_connections 2023-11-28T20:34:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa93806ae40 0x7fa93806d300 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa954159650 0x7fa9540afe60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9540b2830 0x7fa9541479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 --2- 172.21.15.45:0/858223827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa95406cf30 0x7fa95415c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 >> 172.21.15.45:0/858223827 conn(0x7fa95406ab30 msgr2=0x7fa9540aca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:25.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 shutdown_connections 2023-11-28T20:34:25.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.010+0000 7fa95b393640 1 -- 172.21.15.45:0/858223827 wait complete. 2023-11-28T20:34:25.035 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-11-28T20:34:23.133726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236124,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738596,"statfs":{"total":773094113280,"available":772852322304,"internally_reserved":0,"allocated":3383296,"data_stored":2025718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005078"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119305+0000","last_change":"2023-11-28T20:33:53.064970+0000","last_active":"2023-11-28T20:33:58.119305+0000","last_peered":"2023-11-28T20:33:58.119305+0000","last_clean":"2023-11-28T20:33:58.119305+0000","last_became_active":"2023-11-28T20:33:53.064659+0000","last_became_peered":"2023-11-28T20:33:53.064659+0000","last_unstale":"2023-11-28T20:33:58.119305+0000","last_undegraded":"2023-11-28T20:33:58.119305+0000","last_fullsized":"2023-11-28T20:33:58.119305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.109629+0000","last_change":"2023-11-28T20:33:53.948203+0000","last_active":"2023-11-28T20:33:58.109629+0000","last_peered":"2023-11-28T20:33:58.109629+0000","last_clean":"2023-11-28T20:33:58.109629+0000","last_became_active":"2023-11-28T20:33:53.947906+0000","last_became_peered":"2023-11-28T20:33:53.947906+0000","last_unstale":"2023-11-28T20:33:58.109629+0000","last_undegraded":"2023-11-28T20:33:58.109629+0000","last_fullsized":"2023-11-28T20:33:58.109629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918353+0000","last_change":"2023-11-28T20:33:58.158488+0000","last_active":"2023-11-28T20:33:58.918353+0000","last_peered":"2023-11-28T20:33:58.918353+0000","last_clean":"2023-11-28T20:33:58.918353+0000","last_became_active":"2023-11-28T20:33:58.158046+0000","last_became_peered":"2023-11-28T20:33:58.158046+0000","last_unstale":"2023-11-28T20:33:58.918353+0000","last_undegraded":"2023-11-28T20:33:58.918353+0000","last_fullsized":"2023-11-28T20:33:58.918353+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:20:48.980178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.111851+0000","last_change":"2023-11-28T20:33:53.050283+0000","last_active":"2023-11-28T20:33:58.111851+0000","last_peered":"2023-11-28T20:33:58.111851+0000","last_clean":"2023-11-28T20:33:58.111851+0000","last_became_active":"2023-11-28T20:33:53.049963+0000","last_became_peered":"2023-11-28T20:33:53.049963+0000","last_unstale":"2023-11-28T20:33:58.111851+0000","last_undegraded":"2023-11-28T20:33:58.111851+0000","last_fullsized":"2023-11-28T20:33:58.111851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116825+0000","last_change":"2023-11-28T20:33:53.943001+0000","last_active":"2023-11-28T20:33:58.116825+0000","last_peered":"2023-11-28T20:33:58.116825+0000","last_clean":"2023-11-28T20:33:58.116825+0000","last_became_active":"2023-11-28T20:33:53.940632+0000","last_became_peered":"2023-11-28T20:33:53.940632+0000","last_unstale":"2023-11-28T20:33:58.116825+0000","last_undegraded":"2023-11-28T20:33:58.116825+0000","last_fullsized":"2023-11-28T20:33:58.116825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916311+0000","last_change":"2023-11-28T20:33:58.127527+0000","last_active":"2023-11-28T20:33:58.916311+0000","last_peered":"2023-11-28T20:33:58.916311+0000","last_clean":"2023-11-28T20:33:58.916311+0000","last_became_active":"2023-11-28T20:33:58.127058+0000","last_became_peered":"2023-11-28T20:33:58.127058+0000","last_unstale":"2023-11-28T20:33:58.916311+0000","last_undegraded":"2023-11-28T20:33:58.916311+0000","last_fullsized":"2023-11-28T20:33:58.916311+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:12.932976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151962+0000","last_change":"2023-11-28T20:33:53.073019+0000","last_active":"2023-11-28T20:33:58.151962+0000","last_peered":"2023-11-28T20:33:58.151962+0000","last_clean":"2023-11-28T20:33:58.151962+0000","last_became_active":"2023-11-28T20:33:53.072728+0000","last_became_peered":"2023-11-28T20:33:53.072728+0000","last_unstale":"2023-11-28T20:33:58.151962+0000","last_undegraded":"2023-11-28T20:33:58.151962+0000","last_fullsized":"2023-11-28T20:33:58.151962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122157+0000","last_change":"2023-11-28T20:33:53.951054+0000","last_active":"2023-11-28T20:33:58.122157+0000","last_peered":"2023-11-28T20:33:58.122157+0000","last_clean":"2023-11-28T20:33:58.122157+0000","last_became_active":"2023-11-28T20:33:53.950730+0000","last_became_peered":"2023-11-28T20:33:53.950730+0000","last_unstale":"2023-11-28T20:33:58.122157+0000","last_undegraded":"2023-11-28T20:33:58.122157+0000","last_fullsized":"2023-11-28T20:33:58.122157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:19.573371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916721+0000","last_change":"2023-11-28T20:33:58.131034+0000","last_active":"2023-11-28T20:33:58.916721+0000","last_peered":"2023-11-28T20:33:58.916721+0000","last_clean":"2023-11-28T20:33:58.916721+0000","last_became_active":"2023-11-28T20:33:58.129161+0000","last_became_peered":"2023-11-28T20:33:58.129161+0000","last_unstale":"2023-11-28T20:33:58.916721+0000","last_undegraded":"2023-11-28T20:33:58.916721+0000","last_fullsized":"2023-11-28T20:33:58.916721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:19.781603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151744+0000","last_change":"2023-11-28T20:33:53.068043+0000","last_active":"2023-11-28T20:33:58.151744+0000","last_peered":"2023-11-28T20:33:58.151744+0000","last_clean":"2023-11-28T20:33:58.151744+0000","last_became_active":"2023-11-28T20:33:53.067650+0000","last_became_peered":"2023-11-28T20:33:53.067650+0000","last_unstale":"2023-11-28T20:33:58.151744+0000","last_undegraded":"2023-11-28T20:33:58.151744+0000","last_fullsized":"2023-11-28T20:33:58.151744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123332+0000","last_change":"2023-11-28T20:33:53.954109+0000","last_active":"2023-11-28T20:33:58.123332+0000","last_peered":"2023-11-28T20:33:58.123332+0000","last_clean":"2023-11-28T20:33:58.123332+0000","last_became_active":"2023-11-28T20:33:53.953749+0000","last_became_peered":"2023-11-28T20:33:53.953749+0000","last_unstale":"2023-11-28T20:33:58.123332+0000","last_undegraded":"2023-11-28T20:33:58.123332+0000","last_fullsized":"2023-11-28T20:33:58.123332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:21.394618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916961+0000","last_change":"2023-11-28T20:33:58.132306+0000","last_active":"2023-11-28T20:33:58.916961+0000","last_peered":"2023-11-28T20:33:58.916961+0000","last_clean":"2023-11-28T20:33:58.916961+0000","last_became_active":"2023-11-28T20:33:58.131782+0000","last_became_peered":"2023-11-28T20:33:58.131782+0000","last_unstale":"2023-11-28T20:33:58.916961+0000","last_undegraded":"2023-11-28T20:33:58.916961+0000","last_fullsized":"2023-11-28T20:33:58.916961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:30.608189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121351+0000","last_change":"2023-11-28T20:33:53.954276+0000","last_active":"2023-11-28T20:33:58.121351+0000","last_peered":"2023-11-28T20:33:58.121351+0000","last_clean":"2023-11-28T20:33:58.121351+0000","last_became_active":"2023-11-28T20:33:53.953993+0000","last_became_peered":"2023-11-28T20:33:53.953993+0000","last_unstale":"2023-11-28T20:33:58.121351+0000","last_undegraded":"2023-11-28T20:33:58.121351+0000","last_fullsized":"2023-11-28T20:33:58.121351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151814+0000","last_change":"2023-11-28T20:33:53.075863+0000","last_active":"2023-11-28T20:33:58.151814+0000","last_peered":"2023-11-28T20:33:58.151814+0000","last_clean":"2023-11-28T20:33:58.151814+0000","last_became_active":"2023-11-28T20:33:53.075523+0000","last_became_peered":"2023-11-28T20:33:53.075523+0000","last_unstale":"2023-11-28T20:33:58.151814+0000","last_undegraded":"2023-11-28T20:33:58.151814+0000","last_fullsized":"2023-11-28T20:33:58.151814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917964+0000","last_change":"2023-11-28T20:33:58.130991+0000","last_active":"2023-11-28T20:33:58.917964+0000","last_peered":"2023-11-28T20:33:58.917964+0000","last_clean":"2023-11-28T20:33:58.917964+0000","last_became_active":"2023-11-28T20:33:58.130377+0000","last_became_peered":"2023-11-28T20:33:58.130377+0000","last_unstale":"2023-11-28T20:33:58.917964+0000","last_undegraded":"2023-11-28T20:33:58.917964+0000","last_fullsized":"2023-11-28T20:33:58.917964+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103760+0000","last_change":"2023-11-28T20:33:53.943400+0000","last_active":"2023-11-28T20:33:58.103760+0000","last_peered":"2023-11-28T20:33:58.103760+0000","last_clean":"2023-11-28T20:33:58.103760+0000","last_became_active":"2023-11-28T20:33:53.937673+0000","last_became_peered":"2023-11-28T20:33:53.937673+0000","last_unstale":"2023-11-28T20:33:58.103760+0000","last_undegraded":"2023-11-28T20:33:58.103760+0000","last_fullsized":"2023-11-28T20:33:58.103760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125979+0000","last_change":"2023-11-28T20:33:53.062456+0000","last_active":"2023-11-28T20:33:58.125979+0000","last_peered":"2023-11-28T20:33:58.125979+0000","last_clean":"2023-11-28T20:33:58.125979+0000","last_became_active":"2023-11-28T20:33:53.062163+0000","last_became_peered":"2023-11-28T20:33:53.062163+0000","last_unstale":"2023-11-28T20:33:58.125979+0000","last_undegraded":"2023-11-28T20:33:58.125979+0000","last_fullsized":"2023-11-28T20:33:58.125979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917431+0000","last_change":"2023-11-28T20:33:58.137561+0000","last_active":"2023-11-28T20:33:58.917431+0000","last_peered":"2023-11-28T20:33:58.917431+0000","last_clean":"2023-11-28T20:33:58.917431+0000","last_became_active":"2023-11-28T20:33:58.137065+0000","last_became_peered":"2023-11-28T20:33:58.137065+0000","last_unstale":"2023-11-28T20:33:58.917431+0000","last_undegraded":"2023-11-28T20:33:58.917431+0000","last_fullsized":"2023-11-28T20:33:58.917431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103501+0000","last_change":"2023-11-28T20:33:53.940322+0000","last_active":"2023-11-28T20:33:58.103501+0000","last_peered":"2023-11-28T20:33:58.103501+0000","last_clean":"2023-11-28T20:33:58.103501+0000","last_became_active":"2023-11-28T20:33:53.939986+0000","last_became_peered":"2023-11-28T20:33:53.939986+0000","last_unstale":"2023-11-28T20:33:58.103501+0000","last_undegraded":"2023-11-28T20:33:58.103501+0000","last_fullsized":"2023-11-28T20:33:58.103501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123391+0000","last_change":"2023-11-28T20:33:53.062082+0000","last_active":"2023-11-28T20:33:58.123391+0000","last_peered":"2023-11-28T20:33:58.123391+0000","last_clean":"2023-11-28T20:33:58.123391+0000","last_became_active":"2023-11-28T20:33:53.060583+0000","last_became_peered":"2023-11-28T20:33:53.060583+0000","last_unstale":"2023-11-28T20:33:58.123391+0000","last_undegraded":"2023-11-28T20:33:58.123391+0000","last_fullsized":"2023-11-28T20:33:58.123391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916613+0000","last_change":"2023-11-28T20:33:58.158289+0000","last_active":"2023-11-28T20:33:58.916613+0000","last_peered":"2023-11-28T20:33:58.916613+0000","last_clean":"2023-11-28T20:33:58.916613+0000","last_became_active":"2023-11-28T20:33:58.157856+0000","last_became_peered":"2023-11-28T20:33:58.157856+0000","last_unstale":"2023-11-28T20:33:58.916613+0000","last_undegraded":"2023-11-28T20:33:58.916613+0000","last_fullsized":"2023-11-28T20:33:58.916613+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:17.807344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.112802+0000","last_change":"2023-11-28T20:33:53.942280+0000","last_active":"2023-11-28T20:33:58.112802+0000","last_peered":"2023-11-28T20:33:58.112802+0000","last_clean":"2023-11-28T20:33:58.112802+0000","last_became_active":"2023-11-28T20:33:53.939778+0000","last_became_peered":"2023-11-28T20:33:53.939778+0000","last_unstale":"2023-11-28T20:33:58.112802+0000","last_undegraded":"2023-11-28T20:33:58.112802+0000","last_fullsized":"2023-11-28T20:33:58.112802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:33.050269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151419+0000","last_change":"2023-11-28T20:33:53.068971+0000","last_active":"2023-11-28T20:33:58.151419+0000","last_peered":"2023-11-28T20:33:58.151419+0000","last_clean":"2023-11-28T20:33:58.151419+0000","last_became_active":"2023-11-28T20:33:53.068069+0000","last_became_peered":"2023-11-28T20:33:53.068069+0000","last_unstale":"2023-11-28T20:33:58.151419+0000","last_undegraded":"2023-11-28T20:33:58.151419+0000","last_fullsized":"2023-11-28T20:33:58.151419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:41.371596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917429+0000","last_change":"2023-11-28T20:33:58.137597+0000","last_active":"2023-11-28T20:33:58.917429+0000","last_peered":"2023-11-28T20:33:58.917429+0000","last_clean":"2023-11-28T20:33:58.917429+0000","last_became_active":"2023-11-28T20:33:58.137078+0000","last_became_peered":"2023-11-28T20:33:58.137078+0000","last_unstale":"2023-11-28T20:33:58.917429+0000","last_undegraded":"2023-11-28T20:33:58.917429+0000","last_fullsized":"2023-11-28T20:33:58.917429+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:23.642548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120275+0000","last_change":"2023-11-28T20:33:53.945506+0000","last_active":"2023-11-28T20:33:58.120275+0000","last_peered":"2023-11-28T20:33:58.120275+0000","last_clean":"2023-11-28T20:33:58.120275+0000","last_became_active":"2023-11-28T20:33:53.945153+0000","last_became_peered":"2023-11-28T20:33:53.945153+0000","last_unstale":"2023-11-28T20:33:58.120275+0000","last_undegraded":"2023-11-28T20:33:58.120275+0000","last_fullsized":"2023-11-28T20:33:58.120275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126982+0000","last_change":"2023-11-28T20:33:53.064943+0000","last_active":"2023-11-28T20:33:58.126982+0000","last_peered":"2023-11-28T20:33:58.126982+0000","last_clean":"2023-11-28T20:33:58.126982+0000","last_became_active":"2023-11-28T20:33:53.064644+0000","last_became_peered":"2023-11-28T20:33:53.064644+0000","last_unstale":"2023-11-28T20:33:58.126982+0000","last_undegraded":"2023-11-28T20:33:58.126982+0000","last_fullsized":"2023-11-28T20:33:58.126982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916509+0000","last_change":"2023-11-28T20:33:58.158266+0000","last_active":"2023-11-28T20:33:58.916509+0000","last_peered":"2023-11-28T20:33:58.916509+0000","last_clean":"2023-11-28T20:33:58.916509+0000","last_became_active":"2023-11-28T20:33:58.157836+0000","last_became_peered":"2023-11-28T20:33:58.157836+0000","last_unstale":"2023-11-28T20:33:58.916509+0000","last_undegraded":"2023-11-28T20:33:58.916509+0000","last_fullsized":"2023-11-28T20:33:58.916509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:44.771180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.114503+0000","last_change":"2023-11-28T20:33:53.941572+0000","last_active":"2023-11-28T20:33:58.114503+0000","last_peered":"2023-11-28T20:33:58.114503+0000","last_clean":"2023-11-28T20:33:58.114503+0000","last_became_active":"2023-11-28T20:33:53.941245+0000","last_became_peered":"2023-11-28T20:33:53.941245+0000","last_unstale":"2023-11-28T20:33:58.114503+0000","last_undegraded":"2023-11-28T20:33:58.114503+0000","last_fullsized":"2023-11-28T20:33:58.114503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123698+0000","last_change":"2023-11-28T20:33:53.058678+0000","last_active":"2023-11-28T20:33:58.123698+0000","last_peered":"2023-11-28T20:33:58.123698+0000","last_clean":"2023-11-28T20:33:58.123698+0000","last_became_active":"2023-11-28T20:33:53.058375+0000","last_became_peered":"2023-11-28T20:33:53.058375+0000","last_unstale":"2023-11-28T20:33:58.123698+0000","last_undegraded":"2023-11-28T20:33:58.123698+0000","last_fullsized":"2023-11-28T20:33:58.123698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:12.793548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918259+0000","last_change":"2023-11-28T20:33:58.154128+0000","last_active":"2023-11-28T20:33:58.918259+0000","last_peered":"2023-11-28T20:33:58.918259+0000","last_clean":"2023-11-28T20:33:58.918259+0000","last_became_active":"2023-11-28T20:33:58.153697+0000","last_became_peered":"2023-11-28T20:33:58.153697+0000","last_unstale":"2023-11-28T20:33:58.918259+0000","last_undegraded":"2023-11-28T20:33:58.918259+0000","last_fullsized":"2023-11-28T20:33:58.918259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:47.677895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123582+0000","last_change":"2023-11-28T20:33:53.941553+0000","last_active":"2023-11-28T20:33:58.123582+0000","last_peered":"2023-11-28T20:33:58.123582+0000","last_clean":"2023-11-28T20:33:58.123582+0000","last_became_active":"2023-11-28T20:33:53.941248+0000","last_became_peered":"2023-11-28T20:33:53.941248+0000","last_unstale":"2023-11-28T20:33:58.123582+0000","last_undegraded":"2023-11-28T20:33:58.123582+0000","last_fullsized":"2023-11-28T20:33:58.123582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152051+0000","last_change":"2023-11-28T20:33:53.069890+0000","last_active":"2023-11-28T20:33:58.152051+0000","last_peered":"2023-11-28T20:33:58.152051+0000","last_clean":"2023-11-28T20:33:58.152051+0000","last_became_active":"2023-11-28T20:33:53.069572+0000","last_became_peered":"2023-11-28T20:33:53.069572+0000","last_unstale":"2023-11-28T20:33:58.152051+0000","last_undegraded":"2023-11-28T20:33:58.152051+0000","last_fullsized":"2023-11-28T20:33:58.152051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:57.471000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589761+0000","last_change":"2023-11-28T20:33:58.157661+0000","last_active":"2023-11-28T20:33:59.589761+0000","last_peered":"2023-11-28T20:33:59.589761+0000","last_clean":"2023-11-28T20:33:59.589761+0000","last_became_active":"2023-11-28T20:33:58.157072+0000","last_became_peered":"2023-11-28T20:33:58.157072+0000","last_unstale":"2023-11-28T20:33:59.589761+0000","last_undegraded":"2023-11-28T20:33:59.589761+0000","last_fullsized":"2023-11-28T20:33:59.589761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:11.495353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123465+0000","last_change":"2023-11-28T20:33:53.947235+0000","last_active":"2023-11-28T20:33:58.123465+0000","last_peered":"2023-11-28T20:33:58.123465+0000","last_clean":"2023-11-28T20:33:58.123465+0000","last_became_active":"2023-11-28T20:33:53.946977+0000","last_became_peered":"2023-11-28T20:33:53.946977+0000","last_unstale":"2023-11-28T20:33:58.123465+0000","last_undegraded":"2023-11-28T20:33:58.123465+0000","last_fullsized":"2023-11-28T20:33:58.123465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:08.318223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123720+0000","last_change":"2023-11-28T20:33:53.060664+0000","last_active":"2023-11-28T20:33:58.123720+0000","last_peered":"2023-11-28T20:33:58.123720+0000","last_clean":"2023-11-28T20:33:58.123720+0000","last_became_active":"2023-11-28T20:33:53.058196+0000","last_became_peered":"2023-11-28T20:33:53.058196+0000","last_unstale":"2023-11-28T20:33:58.123720+0000","last_undegraded":"2023-11-28T20:33:58.123720+0000","last_fullsized":"2023-11-28T20:33:58.123720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.915328+0000","last_change":"2023-11-28T20:33:58.133412+0000","last_active":"2023-11-28T20:33:58.915328+0000","last_peered":"2023-11-28T20:33:58.915328+0000","last_clean":"2023-11-28T20:33:58.915328+0000","last_became_active":"2023-11-28T20:33:58.132991+0000","last_became_peered":"2023-11-28T20:33:58.132991+0000","last_unstale":"2023-11-28T20:33:58.915328+0000","last_undegraded":"2023-11-28T20:33:58.915328+0000","last_fullsized":"2023-11-28T20:33:58.915328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.107225+0000","last_change":"2023-11-28T20:33:53.942153+0000","last_active":"2023-11-28T20:33:58.107225+0000","last_peered":"2023-11-28T20:33:58.107225+0000","last_clean":"2023-11-28T20:33:58.107225+0000","last_became_active":"2023-11-28T20:33:53.941788+0000","last_became_peered":"2023-11-28T20:33:53.941788+0000","last_unstale":"2023-11-28T20:33:58.107225+0000","last_undegraded":"2023-11-28T20:33:58.107225+0000","last_fullsized":"2023-11-28T20:33:58.107225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:47.311066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126159+0000","last_change":"2023-11-28T20:33:53.058321+0000","last_active":"2023-11-28T20:33:58.126159+0000","last_peered":"2023-11-28T20:33:58.126159+0000","last_clean":"2023-11-28T20:33:58.126159+0000","last_became_active":"2023-11-28T20:33:53.058170+0000","last_became_peered":"2023-11-28T20:33:53.058170+0000","last_unstale":"2023-11-28T20:33:58.126159+0000","last_undegraded":"2023-11-28T20:33:58.126159+0000","last_fullsized":"2023-11-28T20:33:58.126159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:12.793548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917495+0000","last_change":"2023-11-28T20:33:58.138024+0000","last_active":"2023-11-28T20:33:58.917495+0000","last_peered":"2023-11-28T20:33:58.917495+0000","last_clean":"2023-11-28T20:33:58.917495+0000","last_became_active":"2023-11-28T20:33:58.137601+0000","last_became_peered":"2023-11-28T20:33:58.137601+0000","last_unstale":"2023-11-28T20:33:58.917495+0000","last_undegraded":"2023-11-28T20:33:58.917495+0000","last_fullsized":"2023-11-28T20:33:58.917495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:45.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122244+0000","last_change":"2023-11-28T20:33:53.942125+0000","last_active":"2023-11-28T20:33:58.122244+0000","last_peered":"2023-11-28T20:33:58.122244+0000","last_clean":"2023-11-28T20:33:58.122244+0000","last_became_active":"2023-11-28T20:33:53.941741+0000","last_became_peered":"2023-11-28T20:33:53.941741+0000","last_unstale":"2023-11-28T20:33:58.122244+0000","last_undegraded":"2023-11-28T20:33:58.122244+0000","last_fullsized":"2023-11-28T20:33:58.122244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099678+0000","last_change":"2023-11-28T20:33:53.050707+0000","last_active":"2023-11-28T20:33:58.099678+0000","last_peered":"2023-11-28T20:33:58.099678+0000","last_clean":"2023-11-28T20:33:58.099678+0000","last_became_active":"2023-11-28T20:33:53.050403+0000","last_became_peered":"2023-11-28T20:33:53.050403+0000","last_unstale":"2023-11-28T20:33:58.099678+0000","last_undegraded":"2023-11-28T20:33:58.099678+0000","last_fullsized":"2023-11-28T20:33:58.099678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:32.036639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916922+0000","last_change":"2023-11-28T20:33:58.139720+0000","last_active":"2023-11-28T20:33:58.916922+0000","last_peered":"2023-11-28T20:33:58.916922+0000","last_clean":"2023-11-28T20:33:58.916922+0000","last_became_active":"2023-11-28T20:33:58.138372+0000","last_became_peered":"2023-11-28T20:33:58.138372+0000","last_unstale":"2023-11-28T20:33:58.916922+0000","last_undegraded":"2023-11-28T20:33:58.916922+0000","last_fullsized":"2023-11-28T20:33:58.916922+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:11.495353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117362+0000","last_change":"2023-11-28T20:33:53.934500+0000","last_active":"2023-11-28T20:33:58.117362+0000","last_peered":"2023-11-28T20:33:58.117362+0000","last_clean":"2023-11-28T20:33:58.117362+0000","last_became_active":"2023-11-28T20:33:53.933165+0000","last_became_peered":"2023-11-28T20:33:53.933165+0000","last_unstale":"2023-11-28T20:33:58.117362+0000","last_undegraded":"2023-11-28T20:33:58.117362+0000","last_fullsized":"2023-11-28T20:33:58.117362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:48.148924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151406+0000","last_change":"2023-11-28T20:33:53.069660+0000","last_active":"2023-11-28T20:33:58.151406+0000","last_peered":"2023-11-28T20:33:58.151406+0000","last_clean":"2023-11-28T20:33:58.151406+0000","last_became_active":"2023-11-28T20:33:53.069198+0000","last_became_peered":"2023-11-28T20:33:53.069198+0000","last_unstale":"2023-11-28T20:33:58.151406+0000","last_undegraded":"2023-11-28T20:33:58.151406+0000","last_fullsized":"2023-11-28T20:33:58.151406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:46.297436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590181+0000","last_change":"2023-11-28T20:33:58.162077+0000","last_active":"2023-11-28T20:33:59.590181+0000","last_peered":"2023-11-28T20:33:59.590181+0000","last_clean":"2023-11-28T20:33:59.590181+0000","last_became_active":"2023-11-28T20:33:58.159109+0000","last_became_peered":"2023-11-28T20:33:58.159109+0000","last_unstale":"2023-11-28T20:33:59.590181+0000","last_undegraded":"2023-11-28T20:33:59.590181+0000","last_fullsized":"2023-11-28T20:33:59.590181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:47.677895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103295+0000","last_change":"2023-11-28T20:33:53.952209+0000","last_active":"2023-11-28T20:33:58.103295+0000","last_peered":"2023-11-28T20:33:58.103295+0000","last_clean":"2023-11-28T20:33:58.103295+0000","last_became_active":"2023-11-28T20:33:53.949676+0000","last_became_peered":"2023-11-28T20:33:53.949676+0000","last_unstale":"2023-11-28T20:33:58.103295+0000","last_undegraded":"2023-11-28T20:33:58.103295+0000","last_fullsized":"2023-11-28T20:33:58.103295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121849+0000","last_change":"2023-11-28T20:33:53.053226+0000","last_active":"2023-11-28T20:33:58.121849+0000","last_peered":"2023-11-28T20:33:58.121849+0000","last_clean":"2023-11-28T20:33:58.121849+0000","last_became_active":"2023-11-28T20:33:53.052910+0000","last_became_peered":"2023-11-28T20:33:53.052910+0000","last_unstale":"2023-11-28T20:33:58.121849+0000","last_undegraded":"2023-11-28T20:33:58.121849+0000","last_fullsized":"2023-11-28T20:33:58.121849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:41.755062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":169,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151553+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:33:58.151553+0000","last_peered":"2023-11-28T20:33:58.151553+0000","last_clean":"2023-11-28T20:33:58.151553+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:33:58.151553+0000","last_undegraded":"2023-11-28T20:33:58.151553+0000","last_fullsized":"2023-11-28T20:33:58.151553+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589763+0000","last_change":"2023-11-28T20:33:58.157671+0000","last_active":"2023-11-28T20:33:59.589763+0000","last_peered":"2023-11-28T20:33:59.589763+0000","last_clean":"2023-11-28T20:33:59.589763+0000","last_became_active":"2023-11-28T20:33:58.157254+0000","last_became_peered":"2023-11-28T20:33:58.157254+0000","last_unstale":"2023-11-28T20:33:59.589763+0000","last_undegraded":"2023-11-28T20:33:59.589763+0000","last_fullsized":"2023-11-28T20:33:59.589763+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151876+0000","last_change":"2023-11-28T20:33:53.960256+0000","last_active":"2023-11-28T20:33:58.151876+0000","last_peered":"2023-11-28T20:33:58.151876+0000","last_clean":"2023-11-28T20:33:58.151876+0000","last_became_active":"2023-11-28T20:33:53.959635+0000","last_became_peered":"2023-11-28T20:33:53.959635+0000","last_unstale":"2023-11-28T20:33:58.151876+0000","last_undegraded":"2023-11-28T20:33:58.151876+0000","last_fullsized":"2023-11-28T20:33:58.151876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:08.318223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099297+0000","last_change":"2023-11-28T20:33:53.054253+0000","last_active":"2023-11-28T20:33:58.099297+0000","last_peered":"2023-11-28T20:33:58.099297+0000","last_clean":"2023-11-28T20:33:58.099297+0000","last_became_active":"2023-11-28T20:33:53.053912+0000","last_became_peered":"2023-11-28T20:33:53.053912+0000","last_unstale":"2023-11-28T20:33:58.099297+0000","last_undegraded":"2023-11-28T20:33:58.099297+0000","last_fullsized":"2023-11-28T20:33:58.099297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:53.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589287+0000","last_change":"2023-11-28T20:33:58.154204+0000","last_active":"2023-11-28T20:33:59.589287+0000","last_peered":"2023-11-28T20:33:59.589287+0000","last_clean":"2023-11-28T20:33:59.589287+0000","last_became_active":"2023-11-28T20:33:58.153585+0000","last_became_peered":"2023-11-28T20:33:58.153585+0000","last_unstale":"2023-11-28T20:33:59.589287+0000","last_undegraded":"2023-11-28T20:33:59.589287+0000","last_fullsized":"2023-11-28T20:33:59.589287+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:23.642548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122263+0000","last_change":"2023-11-28T20:33:53.950370+0000","last_active":"2023-11-28T20:33:58.122263+0000","last_peered":"2023-11-28T20:33:58.122263+0000","last_clean":"2023-11-28T20:33:58.122263+0000","last_became_active":"2023-11-28T20:33:53.946975+0000","last_became_peered":"2023-11-28T20:33:53.946975+0000","last_unstale":"2023-11-28T20:33:58.122263+0000","last_undegraded":"2023-11-28T20:33:58.122263+0000","last_fullsized":"2023-11-28T20:33:58.122263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:41.830889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123611+0000","last_change":"2023-11-28T20:33:53.062319+0000","last_active":"2023-11-28T20:33:58.123611+0000","last_peered":"2023-11-28T20:33:58.123611+0000","last_clean":"2023-11-28T20:33:58.123611+0000","last_became_active":"2023-11-28T20:33:53.062034+0000","last_became_peered":"2023-11-28T20:33:53.062034+0000","last_unstale":"2023-11-28T20:33:58.123611+0000","last_undegraded":"2023-11-28T20:33:58.123611+0000","last_fullsized":"2023-11-28T20:33:58.123611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590144+0000","last_change":"2023-11-28T20:33:58.129150+0000","last_active":"2023-11-28T20:33:59.590144+0000","last_peered":"2023-11-28T20:33:59.590144+0000","last_clean":"2023-11-28T20:33:59.590144+0000","last_became_active":"2023-11-28T20:33:58.128658+0000","last_became_peered":"2023-11-28T20:33:58.128658+0000","last_unstale":"2023-11-28T20:33:59.590144+0000","last_undegraded":"2023-11-28T20:33:59.590144+0000","last_fullsized":"2023-11-28T20:33:59.590144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:24.263445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152745+0000","last_change":"2023-11-28T20:33:53.964226+0000","last_active":"2023-11-28T20:33:58.152745+0000","last_peered":"2023-11-28T20:33:58.152745+0000","last_clean":"2023-11-28T20:33:58.152745+0000","last_became_active":"2023-11-28T20:33:53.963614+0000","last_became_peered":"2023-11-28T20:33:53.963614+0000","last_unstale":"2023-11-28T20:33:58.152745+0000","last_undegraded":"2023-11-28T20:33:58.152745+0000","last_fullsized":"2023-11-28T20:33:58.152745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:20.194268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099544+0000","last_change":"2023-11-28T20:33:53.049547+0000","last_active":"2023-11-28T20:33:58.099544+0000","last_peered":"2023-11-28T20:33:58.099544+0000","last_clean":"2023-11-28T20:33:58.099544+0000","last_became_active":"2023-11-28T20:33:53.049252+0000","last_became_peered":"2023-11-28T20:33:53.049252+0000","last_unstale":"2023-11-28T20:33:58.099544+0000","last_undegraded":"2023-11-28T20:33:58.099544+0000","last_fullsized":"2023-11-28T20:33:58.099544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:41.371596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.252105+0000","last_change":"2023-11-28T20:33:58.158216+0000","last_active":"2023-11-28T20:33:59.252105+0000","last_peered":"2023-11-28T20:33:59.252105+0000","last_clean":"2023-11-28T20:33:59.252105+0000","last_became_active":"2023-11-28T20:33:58.157621+0000","last_became_peered":"2023-11-28T20:33:58.157621+0000","last_unstale":"2023-11-28T20:33:59.252105+0000","last_undegraded":"2023-11-28T20:33:59.252105+0000","last_fullsized":"2023-11-28T20:33:59.252105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:43:13.638963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151918+0000","last_change":"2023-11-28T20:33:53.959472+0000","last_active":"2023-11-28T20:33:58.151918+0000","last_peered":"2023-11-28T20:33:58.151918+0000","last_clean":"2023-11-28T20:33:58.151918+0000","last_became_active":"2023-11-28T20:33:53.959110+0000","last_became_peered":"2023-11-28T20:33:53.959110+0000","last_unstale":"2023-11-28T20:33:58.151918+0000","last_undegraded":"2023-11-28T20:33:58.151918+0000","last_fullsized":"2023-11-28T20:33:58.151918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:41.830889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.104034+0000","last_change":"2023-11-28T20:33:53.047976+0000","last_active":"2023-11-28T20:33:58.104034+0000","last_peered":"2023-11-28T20:33:58.104034+0000","last_clean":"2023-11-28T20:33:58.104034+0000","last_became_active":"2023-11-28T20:33:53.047700+0000","last_became_peered":"2023-11-28T20:33:53.047700+0000","last_unstale":"2023-11-28T20:33:58.104034+0000","last_undegraded":"2023-11-28T20:33:58.104034+0000","last_fullsized":"2023-11-28T20:33:58.104034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:53.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916144+0000","last_change":"2023-11-28T20:33:58.128125+0000","last_active":"2023-11-28T20:33:58.916144+0000","last_peered":"2023-11-28T20:33:58.916144+0000","last_clean":"2023-11-28T20:33:58.916144+0000","last_became_active":"2023-11-28T20:33:58.127738+0000","last_became_peered":"2023-11-28T20:33:58.127738+0000","last_unstale":"2023-11-28T20:33:58.916144+0000","last_undegraded":"2023-11-28T20:33:58.916144+0000","last_fullsized":"2023-11-28T20:33:58.916144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:29.532639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151544+0000","last_change":"2023-11-28T20:33:53.960235+0000","last_active":"2023-11-28T20:33:58.151544+0000","last_peered":"2023-11-28T20:33:58.151544+0000","last_clean":"2023-11-28T20:33:58.151544+0000","last_became_active":"2023-11-28T20:33:53.959892+0000","last_became_peered":"2023-11-28T20:33:53.959892+0000","last_unstale":"2023-11-28T20:33:58.151544+0000","last_undegraded":"2023-11-28T20:33:58.151544+0000","last_fullsized":"2023-11-28T20:33:58.151544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099168+0000","last_change":"2023-11-28T20:33:53.056980+0000","last_active":"2023-11-28T20:33:58.099168+0000","last_peered":"2023-11-28T20:33:58.099168+0000","last_clean":"2023-11-28T20:33:58.099168+0000","last_became_active":"2023-11-28T20:33:53.056663+0000","last_became_peered":"2023-11-28T20:33:53.056663+0000","last_unstale":"2023-11-28T20:33:58.099168+0000","last_undegraded":"2023-11-28T20:33:58.099168+0000","last_fullsized":"2023-11-28T20:33:58.099168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:11:01.063288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590096+0000","last_change":"2023-11-28T20:33:58.137476+0000","last_active":"2023-11-28T20:33:59.590096+0000","last_peered":"2023-11-28T20:33:59.590096+0000","last_clean":"2023-11-28T20:33:59.590096+0000","last_became_active":"2023-11-28T20:33:58.136106+0000","last_became_peered":"2023-11-28T20:33:58.136106+0000","last_unstale":"2023-11-28T20:33:59.590096+0000","last_undegraded":"2023-11-28T20:33:59.590096+0000","last_fullsized":"2023-11-28T20:33:59.590096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:39.912968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119486+0000","last_change":"2023-11-28T20:33:53.940374+0000","last_active":"2023-11-28T20:33:58.119486+0000","last_peered":"2023-11-28T20:33:58.119486+0000","last_clean":"2023-11-28T20:33:58.119486+0000","last_became_active":"2023-11-28T20:33:53.939873+0000","last_became_peered":"2023-11-28T20:33:53.939873+0000","last_unstale":"2023-11-28T20:33:58.119486+0000","last_undegraded":"2023-11-28T20:33:58.119486+0000","last_fullsized":"2023-11-28T20:33:58.119486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120431+0000","last_change":"2023-11-28T20:33:53.055672+0000","last_active":"2023-11-28T20:33:58.120431+0000","last_peered":"2023-11-28T20:33:58.120431+0000","last_clean":"2023-11-28T20:33:58.120431+0000","last_became_active":"2023-11-28T20:33:53.055320+0000","last_became_peered":"2023-11-28T20:33:53.055320+0000","last_unstale":"2023-11-28T20:33:58.120431+0000","last_undegraded":"2023-11-28T20:33:58.120431+0000","last_fullsized":"2023-11-28T20:33:58.120431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.920861+0000","last_change":"2023-11-28T20:33:58.162629+0000","last_active":"2023-11-28T20:33:58.920861+0000","last_peered":"2023-11-28T20:33:58.920861+0000","last_clean":"2023-11-28T20:33:58.920861+0000","last_became_active":"2023-11-28T20:33:58.162003+0000","last_became_peered":"2023-11-28T20:33:58.162003+0000","last_unstale":"2023-11-28T20:33:58.920861+0000","last_undegraded":"2023-11-28T20:33:58.920861+0000","last_fullsized":"2023-11-28T20:33:58.920861+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:31.738218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151734+0000","last_change":"2023-11-28T20:33:53.961916+0000","last_active":"2023-11-28T20:33:58.151734+0000","last_peered":"2023-11-28T20:33:58.151734+0000","last_clean":"2023-11-28T20:33:58.151734+0000","last_became_active":"2023-11-28T20:33:53.961577+0000","last_became_peered":"2023-11-28T20:33:53.961577+0000","last_unstale":"2023-11-28T20:33:58.151734+0000","last_undegraded":"2023-11-28T20:33:58.151734+0000","last_fullsized":"2023-11-28T20:33:58.151734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126069+0000","last_change":"2023-11-28T20:33:53.056677+0000","last_active":"2023-11-28T20:33:58.126069+0000","last_peered":"2023-11-28T20:33:58.126069+0000","last_clean":"2023-11-28T20:33:58.126069+0000","last_became_active":"2023-11-28T20:33:53.055737+0000","last_became_peered":"2023-11-28T20:33:53.055737+0000","last_unstale":"2023-11-28T20:33:58.126069+0000","last_undegraded":"2023-11-28T20:33:58.126069+0000","last_fullsized":"2023-11-28T20:33:58.126069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.919758+0000","last_change":"2023-11-28T20:33:58.155270+0000","last_active":"2023-11-28T20:33:58.919758+0000","last_peered":"2023-11-28T20:33:58.919758+0000","last_clean":"2023-11-28T20:33:58.919758+0000","last_became_active":"2023-11-28T20:33:58.153764+0000","last_became_peered":"2023-11-28T20:33:58.153764+0000","last_unstale":"2023-11-28T20:33:58.919758+0000","last_undegraded":"2023-11-28T20:33:58.919758+0000","last_fullsized":"2023-11-28T20:33:58.919758+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:29.020740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117654+0000","last_change":"2023-11-28T20:33:53.943637+0000","last_active":"2023-11-28T20:33:58.117654+0000","last_peered":"2023-11-28T20:33:58.117654+0000","last_clean":"2023-11-28T20:33:58.117654+0000","last_became_active":"2023-11-28T20:33:53.941418+0000","last_became_peered":"2023-11-28T20:33:53.941418+0000","last_unstale":"2023-11-28T20:33:58.117654+0000","last_undegraded":"2023-11-28T20:33:58.117654+0000","last_fullsized":"2023-11-28T20:33:58.117654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118661+0000","last_change":"2023-11-28T20:33:53.052522+0000","last_active":"2023-11-28T20:33:58.118661+0000","last_peered":"2023-11-28T20:33:58.118661+0000","last_clean":"2023-11-28T20:33:58.118661+0000","last_became_active":"2023-11-28T20:33:53.052202+0000","last_became_peered":"2023-11-28T20:33:53.052202+0000","last_unstale":"2023-11-28T20:33:58.118661+0000","last_undegraded":"2023-11-28T20:33:58.118661+0000","last_fullsized":"2023-11-28T20:33:58.118661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:46.454394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.253371+0000","last_change":"2023-11-28T20:33:58.157854+0000","last_active":"2023-11-28T20:33:59.253371+0000","last_peered":"2023-11-28T20:33:59.253371+0000","last_clean":"2023-11-28T20:33:59.253371+0000","last_became_active":"2023-11-28T20:33:58.157424+0000","last_became_peered":"2023-11-28T20:33:58.157424+0000","last_unstale":"2023-11-28T20:33:59.253371+0000","last_undegraded":"2023-11-28T20:33:59.253371+0000","last_fullsized":"2023-11-28T20:33:59.253371+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:41.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151781+0000","last_change":"2023-11-28T20:33:53.959159+0000","last_active":"2023-11-28T20:33:58.151781+0000","last_peered":"2023-11-28T20:33:58.151781+0000","last_clean":"2023-11-28T20:33:58.151781+0000","last_became_active":"2023-11-28T20:33:53.958752+0000","last_became_peered":"2023-11-28T20:33:53.958752+0000","last_unstale":"2023-11-28T20:33:58.151781+0000","last_undegraded":"2023-11-28T20:33:58.151781+0000","last_fullsized":"2023-11-28T20:33:58.151781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:21.394618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:34:13.400104+0000","last_change":"2023-11-28T20:33:53.051304+0000","last_active":"2023-11-28T20:34:13.400104+0000","last_peered":"2023-11-28T20:34:13.400104+0000","last_clean":"2023-11-28T20:34:13.400104+0000","last_became_active":"2023-11-28T20:33:53.050957+0000","last_became_peered":"2023-11-28T20:33:53.050957+0000","last_unstale":"2023-11-28T20:34:13.400104+0000","last_undegraded":"2023-11-28T20:34:13.400104+0000","last_fullsized":"2023-11-28T20:34:13.400104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:41.755062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.915782+0000","last_change":"2023-11-28T20:33:58.133335+0000","last_active":"2023-11-28T20:33:58.915782+0000","last_peered":"2023-11-28T20:33:58.915782+0000","last_clean":"2023-11-28T20:33:58.915782+0000","last_became_active":"2023-11-28T20:33:58.132968+0000","last_became_peered":"2023-11-28T20:33:58.132968+0000","last_unstale":"2023-11-28T20:33:58.915782+0000","last_undegraded":"2023-11-28T20:33:58.915782+0000","last_fullsized":"2023-11-28T20:33:58.915782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105225+0000","last_change":"2023-11-28T20:33:53.944947+0000","last_active":"2023-11-28T20:33:58.105225+0000","last_peered":"2023-11-28T20:33:58.105225+0000","last_clean":"2023-11-28T20:33:58.105225+0000","last_became_active":"2023-11-28T20:33:53.944492+0000","last_became_peered":"2023-11-28T20:33:53.944492+0000","last_unstale":"2023-11-28T20:33:58.105225+0000","last_undegraded":"2023-11-28T20:33:58.105225+0000","last_fullsized":"2023-11-28T20:33:58.105225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:42.768692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123500+0000","last_change":"2023-11-28T20:33:53.063285+0000","last_active":"2023-11-28T20:33:58.123500+0000","last_peered":"2023-11-28T20:33:58.123500+0000","last_clean":"2023-11-28T20:33:58.123500+0000","last_became_active":"2023-11-28T20:33:53.063017+0000","last_became_peered":"2023-11-28T20:33:53.063017+0000","last_unstale":"2023-11-28T20:33:58.123500+0000","last_undegraded":"2023-11-28T20:33:58.123500+0000","last_fullsized":"2023-11-28T20:33:58.123500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917261+0000","last_change":"2023-11-28T20:33:58.125259+0000","last_active":"2023-11-28T20:33:58.917261+0000","last_peered":"2023-11-28T20:33:58.917261+0000","last_clean":"2023-11-28T20:33:58.917261+0000","last_became_active":"2023-11-28T20:33:58.124810+0000","last_became_peered":"2023-11-28T20:33:58.124810+0000","last_unstale":"2023-11-28T20:33:58.917261+0000","last_undegraded":"2023-11-28T20:33:58.917261+0000","last_fullsized":"2023-11-28T20:33:58.917261+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:22.122661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116919+0000","last_change":"2023-11-28T20:33:53.940927+0000","last_active":"2023-11-28T20:33:58.116919+0000","last_peered":"2023-11-28T20:33:58.116919+0000","last_clean":"2023-11-28T20:33:58.116919+0000","last_became_active":"2023-11-28T20:33:53.940527+0000","last_became_peered":"2023-11-28T20:33:53.940527+0000","last_unstale":"2023-11-28T20:33:58.116919+0000","last_undegraded":"2023-11-28T20:33:58.116919+0000","last_fullsized":"2023-11-28T20:33:58.116919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123801+0000","last_change":"2023-11-28T20:33:53.055804+0000","last_active":"2023-11-28T20:33:58.123801+0000","last_peered":"2023-11-28T20:33:58.123801+0000","last_clean":"2023-11-28T20:33:58.123801+0000","last_became_active":"2023-11-28T20:33:53.053184+0000","last_became_peered":"2023-11-28T20:33:53.053184+0000","last_unstale":"2023-11-28T20:33:58.123801+0000","last_undegraded":"2023-11-28T20:33:58.123801+0000","last_fullsized":"2023-11-28T20:33:58.123801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:57.471000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918702+0000","last_change":"2023-11-28T20:33:58.133079+0000","last_active":"2023-11-28T20:33:58.918702+0000","last_peered":"2023-11-28T20:33:58.918702+0000","last_clean":"2023-11-28T20:33:58.918702+0000","last_became_active":"2023-11-28T20:33:58.126051+0000","last_became_peered":"2023-11-28T20:33:58.126051+0000","last_unstale":"2023-11-28T20:33:58.918702+0000","last_undegraded":"2023-11-28T20:33:58.918702+0000","last_fullsized":"2023-11-28T20:33:58.918702+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:41.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123681+0000","last_change":"2023-11-28T20:33:53.958488+0000","last_active":"2023-11-28T20:33:58.123681+0000","last_peered":"2023-11-28T20:33:58.123681+0000","last_clean":"2023-11-28T20:33:58.123681+0000","last_became_active":"2023-11-28T20:33:53.958041+0000","last_became_peered":"2023-11-28T20:33:53.958041+0000","last_unstale":"2023-11-28T20:33:58.123681+0000","last_undegraded":"2023-11-28T20:33:58.123681+0000","last_fullsized":"2023-11-28T20:33:58.123681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:07.426176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126850+0000","last_change":"2023-11-28T20:33:53.054003+0000","last_active":"2023-11-28T20:33:58.126850+0000","last_peered":"2023-11-28T20:33:58.126850+0000","last_clean":"2023-11-28T20:33:58.126850+0000","last_became_active":"2023-11-28T20:33:53.053773+0000","last_became_peered":"2023-11-28T20:33:53.053773+0000","last_unstale":"2023-11-28T20:33:58.126850+0000","last_undegraded":"2023-11-28T20:33:58.126850+0000","last_fullsized":"2023-11-28T20:33:58.126850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918600+0000","last_change":"2023-11-28T20:33:58.157661+0000","last_active":"2023-11-28T20:33:58.918600+0000","last_peered":"2023-11-28T20:33:58.918600+0000","last_clean":"2023-11-28T20:33:58.918600+0000","last_became_active":"2023-11-28T20:33:58.157274+0000","last_became_peered":"2023-11-28T20:33:58.157274+0000","last_unstale":"2023-11-28T20:33:58.918600+0000","last_undegraded":"2023-11-28T20:33:58.918600+0000","last_fullsized":"2023-11-28T20:33:58.918600+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:43:13.638963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589987+0000","last_change":"2023-11-28T20:33:58.159599+0000","last_active":"2023-11-28T20:33:59.589987+0000","last_peered":"2023-11-28T20:33:59.589987+0000","last_clean":"2023-11-28T20:33:59.589987+0000","last_became_active":"2023-11-28T20:33:58.158483+0000","last_became_peered":"2023-11-28T20:33:58.158483+0000","last_unstale":"2023-11-28T20:33:59.589987+0000","last_undegraded":"2023-11-28T20:33:59.589987+0000","last_fullsized":"2023-11-28T20:33:59.589987+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:45.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122123+0000","last_change":"2023-11-28T20:33:53.950947+0000","last_active":"2023-11-28T20:33:58.122123+0000","last_peered":"2023-11-28T20:33:58.122123+0000","last_clean":"2023-11-28T20:33:58.122123+0000","last_became_active":"2023-11-28T20:33:53.948976+0000","last_became_peered":"2023-11-28T20:33:53.948976+0000","last_unstale":"2023-11-28T20:33:58.122123+0000","last_undegraded":"2023-11-28T20:33:58.122123+0000","last_fullsized":"2023-11-28T20:33:58.122123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120814+0000","last_change":"2023-11-28T20:33:53.059457+0000","last_active":"2023-11-28T20:33:58.120814+0000","last_peered":"2023-11-28T20:33:58.120814+0000","last_clean":"2023-11-28T20:33:58.120814+0000","last_became_active":"2023-11-28T20:33:53.059116+0000","last_became_peered":"2023-11-28T20:33:53.059116+0000","last_unstale":"2023-11-28T20:33:58.120814+0000","last_undegraded":"2023-11-28T20:33:58.120814+0000","last_fullsized":"2023-11-28T20:33:58.120814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:32.036639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590492+0000","last_change":"2023-11-28T20:33:58.141507+0000","last_active":"2023-11-28T20:33:59.590492+0000","last_peered":"2023-11-28T20:33:59.590492+0000","last_clean":"2023-11-28T20:33:59.590492+0000","last_became_active":"2023-11-28T20:33:58.141072+0000","last_became_peered":"2023-11-28T20:33:58.141072+0000","last_unstale":"2023-11-28T20:33:59.590492+0000","last_undegraded":"2023-11-28T20:33:59.590492+0000","last_fullsized":"2023-11-28T20:33:59.590492+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103356+0000","last_change":"2023-11-28T20:33:53.937448+0000","last_active":"2023-11-28T20:33:58.103356+0000","last_peered":"2023-11-28T20:33:58.103356+0000","last_clean":"2023-11-28T20:33:58.103356+0000","last_became_active":"2023-11-28T20:33:53.937097+0000","last_became_peered":"2023-11-28T20:33:53.937097+0000","last_unstale":"2023-11-28T20:33:58.103356+0000","last_undegraded":"2023-11-28T20:33:58.103356+0000","last_fullsized":"2023-11-28T20:33:58.103356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105061+0000","last_change":"2023-11-28T20:33:53.047089+0000","last_active":"2023-11-28T20:33:58.105061+0000","last_peered":"2023-11-28T20:33:58.105061+0000","last_clean":"2023-11-28T20:33:58.105061+0000","last_became_active":"2023-11-28T20:33:53.046299+0000","last_became_peered":"2023-11-28T20:33:53.046299+0000","last_unstale":"2023-11-28T20:33:58.105061+0000","last_undegraded":"2023-11-28T20:33:58.105061+0000","last_fullsized":"2023-11-28T20:33:58.105061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121518+0000","last_change":"2023-11-28T20:33:53.057990+0000","last_active":"2023-11-28T20:33:58.121518+0000","last_peered":"2023-11-28T20:33:58.121518+0000","last_clean":"2023-11-28T20:33:58.121518+0000","last_became_active":"2023-11-28T20:33:53.057437+0000","last_became_peered":"2023-11-28T20:33:53.057437+0000","last_unstale":"2023-11-28T20:33:58.121518+0000","last_undegraded":"2023-11-28T20:33:58.121518+0000","last_fullsized":"2023-11-28T20:33:58.121518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118455+0000","last_change":"2023-11-28T20:33:53.944979+0000","last_active":"2023-11-28T20:33:58.118455+0000","last_peered":"2023-11-28T20:33:58.118455+0000","last_clean":"2023-11-28T20:33:58.118455+0000","last_became_active":"2023-11-28T20:33:53.944587+0000","last_became_peered":"2023-11-28T20:33:53.944587+0000","last_unstale":"2023-11-28T20:33:58.118455+0000","last_undegraded":"2023-11-28T20:33:58.118455+0000","last_fullsized":"2023-11-28T20:33:58.118455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.226534+0000","last_change":"2023-11-28T20:33:58.128961+0000","last_active":"2023-11-28T20:33:59.226534+0000","last_peered":"2023-11-28T20:33:59.226534+0000","last_clean":"2023-11-28T20:33:59.226534+0000","last_became_active":"2023-11-28T20:33:58.128548+0000","last_became_peered":"2023-11-28T20:33:58.128548+0000","last_unstale":"2023-11-28T20:33:59.226534+0000","last_undegraded":"2023-11-28T20:33:59.226534+0000","last_fullsized":"2023-11-28T20:33:59.226534+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:39.520026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103995+0000","last_change":"2023-11-28T20:33:53.061966+0000","last_active":"2023-11-28T20:33:58.103995+0000","last_peered":"2023-11-28T20:33:58.103995+0000","last_clean":"2023-11-28T20:33:58.103995+0000","last_became_active":"2023-11-28T20:33:53.061538+0000","last_became_peered":"2023-11-28T20:33:53.061538+0000","last_unstale":"2023-11-28T20:33:58.103995+0000","last_undegraded":"2023-11-28T20:33:58.103995+0000","last_fullsized":"2023-11-28T20:33:58.103995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.113040+0000","last_change":"2023-11-28T20:33:53.943755+0000","last_active":"2023-11-28T20:33:58.113040+0000","last_peered":"2023-11-28T20:33:58.113040+0000","last_clean":"2023-11-28T20:33:58.113040+0000","last_became_active":"2023-11-28T20:33:53.943464+0000","last_became_peered":"2023-11-28T20:33:53.943464+0000","last_unstale":"2023-11-28T20:33:58.113040+0000","last_undegraded":"2023-11-28T20:33:58.113040+0000","last_fullsized":"2023-11-28T20:33:58.113040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:42.385226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.919987+0000","last_change":"2023-11-28T20:33:58.142246+0000","last_active":"2023-11-28T20:33:58.919987+0000","last_peered":"2023-11-28T20:33:58.919987+0000","last_clean":"2023-11-28T20:33:58.919987+0000","last_became_active":"2023-11-28T20:33:58.134395+0000","last_became_peered":"2023-11-28T20:33:58.134395+0000","last_unstale":"2023-11-28T20:33:58.919987+0000","last_undegraded":"2023-11-28T20:33:58.919987+0000","last_fullsized":"2023-11-28T20:33:58.919987+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:24:08.097260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119379+0000","last_change":"2023-11-28T20:33:53.948082+0000","last_active":"2023-11-28T20:33:58.119379+0000","last_peered":"2023-11-28T20:33:58.119379+0000","last_clean":"2023-11-28T20:33:58.119379+0000","last_became_active":"2023-11-28T20:33:53.947760+0000","last_became_peered":"2023-11-28T20:33:53.947760+0000","last_unstale":"2023-11-28T20:33:58.119379+0000","last_undegraded":"2023-11-28T20:33:58.119379+0000","last_fullsized":"2023-11-28T20:33:58.119379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:35.450849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099225+0000","last_change":"2023-11-28T20:33:53.947745+0000","last_active":"2023-11-28T20:33:58.099225+0000","last_peered":"2023-11-28T20:33:58.099225+0000","last_clean":"2023-11-28T20:33:58.099225+0000","last_became_active":"2023-11-28T20:33:53.947452+0000","last_became_peered":"2023-11-28T20:33:53.947452+0000","last_unstale":"2023-11-28T20:33:58.099225+0000","last_undegraded":"2023-11-28T20:33:58.099225+0000","last_fullsized":"2023-11-28T20:33:58.099225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:42.768692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125984+0000","last_change":"2023-11-28T20:33:53.947868+0000","last_active":"2023-11-28T20:33:58.125984+0000","last_peered":"2023-11-28T20:33:58.125984+0000","last_clean":"2023-11-28T20:33:58.125984+0000","last_became_active":"2023-11-28T20:33:53.947304+0000","last_became_peered":"2023-11-28T20:33:53.947304+0000","last_unstale":"2023-11-28T20:33:58.125984+0000","last_undegraded":"2023-11-28T20:33:58.125984+0000","last_fullsized":"2023-11-28T20:33:58.125984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118368+0000","last_change":"2023-11-28T20:33:53.947122+0000","last_active":"2023-11-28T20:33:58.118368+0000","last_peered":"2023-11-28T20:33:58.118368+0000","last_clean":"2023-11-28T20:33:58.118368+0000","last_became_active":"2023-11-28T20:33:53.946556+0000","last_became_peered":"2023-11-28T20:33:53.946556+0000","last_unstale":"2023-11-28T20:33:58.118368+0000","last_undegraded":"2023-11-28T20:33:58.118368+0000","last_fullsized":"2023-11-28T20:33:58.118368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.109664+0000","last_change":"2023-11-28T20:33:53.946180+0000","last_active":"2023-11-28T20:33:58.109664+0000","last_peered":"2023-11-28T20:33:58.109664+0000","last_clean":"2023-11-28T20:33:58.109664+0000","last_became_active":"2023-11-28T20:33:53.945758+0000","last_became_peered":"2023-11-28T20:33:53.945758+0000","last_unstale":"2023-11-28T20:33:58.109664+0000","last_undegraded":"2023-11-28T20:33:58.109664+0000","last_fullsized":"2023-11-28T20:33:58.109664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123704+0000","last_change":"2023-11-28T20:33:53.950532+0000","last_active":"2023-11-28T20:33:58.123704+0000","last_peered":"2023-11-28T20:33:58.123704+0000","last_clean":"2023-11-28T20:33:58.123704+0000","last_became_active":"2023-11-28T20:33:53.950206+0000","last_became_peered":"2023-11-28T20:33:53.950206+0000","last_unstale":"2023-11-28T20:33:58.123704+0000","last_undegraded":"2023-11-28T20:33:58.123704+0000","last_fullsized":"2023-11-28T20:33:58.123704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126244+0000","last_change":"2023-11-28T20:33:53.958413+0000","last_active":"2023-11-28T20:33:58.126244+0000","last_peered":"2023-11-28T20:33:58.126244+0000","last_clean":"2023-11-28T20:33:58.126244+0000","last_became_active":"2023-11-28T20:33:53.958179+0000","last_became_peered":"2023-11-28T20:33:53.958179+0000","last_unstale":"2023-11-28T20:33:58.126244+0000","last_undegraded":"2023-11-28T20:33:58.126244+0000","last_fullsized":"2023-11-28T20:33:58.126244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.113900+0000","last_change":"2023-11-28T20:33:53.952207+0000","last_active":"2023-11-28T20:33:58.113900+0000","last_peered":"2023-11-28T20:33:58.113900+0000","last_clean":"2023-11-28T20:33:58.113900+0000","last_became_active":"2023-11-28T20:33:53.950011+0000","last_became_peered":"2023-11-28T20:33:53.950011+0000","last_unstale":"2023-11-28T20:33:58.113900+0000","last_undegraded":"2023-11-28T20:33:58.113900+0000","last_fullsized":"2023-11-28T20:33:58.113900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120660+0000","last_change":"2023-11-28T20:33:53.952356+0000","last_active":"2023-11-28T20:33:58.120660+0000","last_peered":"2023-11-28T20:33:58.120660+0000","last_clean":"2023-11-28T20:33:58.120660+0000","last_became_active":"2023-11-28T20:33:53.947787+0000","last_became_peered":"2023-11-28T20:33:53.947787+0000","last_unstale":"2023-11-28T20:33:58.120660+0000","last_undegraded":"2023-11-28T20:33:58.120660+0000","last_fullsized":"2023-11-28T20:33:58.120660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105059+0000","last_change":"2023-11-28T20:33:53.949973+0000","last_active":"2023-11-28T20:33:58.105059+0000","last_peered":"2023-11-28T20:33:58.105059+0000","last_clean":"2023-11-28T20:33:58.105059+0000","last_became_active":"2023-11-28T20:33:53.946758+0000","last_became_peered":"2023-11-28T20:33:53.946758+0000","last_unstale":"2023-11-28T20:33:58.105059+0000","last_undegraded":"2023-11-28T20:33:58.105059+0000","last_fullsized":"2023-11-28T20:33:58.105059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125895+0000","last_change":"2023-11-28T20:33:53.950994+0000","last_active":"2023-11-28T20:33:58.125895+0000","last_peered":"2023-11-28T20:33:58.125895+0000","last_clean":"2023-11-28T20:33:58.125895+0000","last_became_active":"2023-11-28T20:33:53.946524+0000","last_became_peered":"2023-11-28T20:33:53.946524+0000","last_unstale":"2023-11-28T20:33:58.125895+0000","last_undegraded":"2023-11-28T20:33:58.125895+0000","last_fullsized":"2023-11-28T20:33:58.125895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118275+0000","last_change":"2023-11-28T20:33:53.939684+0000","last_active":"2023-11-28T20:33:58.118275+0000","last_peered":"2023-11-28T20:33:58.118275+0000","last_clean":"2023-11-28T20:33:58.118275+0000","last_became_active":"2023-11-28T20:33:53.939225+0000","last_became_peered":"2023-11-28T20:33:53.939225+0000","last_unstale":"2023-11-28T20:33:58.118275+0000","last_undegraded":"2023-11-28T20:33:58.118275+0000","last_fullsized":"2023-11-28T20:33:58.118275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125812+0000","last_change":"2023-11-28T20:33:53.951199+0000","last_active":"2023-11-28T20:33:58.125812+0000","last_peered":"2023-11-28T20:33:58.125812+0000","last_clean":"2023-11-28T20:33:58.125812+0000","last_became_active":"2023-11-28T20:33:53.949262+0000","last_became_peered":"2023-11-28T20:33:53.949262+0000","last_unstale":"2023-11-28T20:33:58.125812+0000","last_undegraded":"2023-11-28T20:33:58.125812+0000","last_fullsized":"2023-11-28T20:33:58.125812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152008+0000","last_change":"2023-11-28T20:33:53.958492+0000","last_active":"2023-11-28T20:33:58.152008+0000","last_peered":"2023-11-28T20:33:58.152008+0000","last_clean":"2023-11-28T20:33:58.152008+0000","last_became_active":"2023-11-28T20:33:53.958141+0000","last_became_peered":"2023-11-28T20:33:53.958141+0000","last_unstale":"2023-11-28T20:33:58.152008+0000","last_undegraded":"2023-11-28T20:33:58.152008+0000","last_fullsized":"2023-11-28T20:33:58.152008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:13.738167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099160+0000","last_change":"2023-11-28T20:33:53.948508+0000","last_active":"2023-11-28T20:33:58.099160+0000","last_peered":"2023-11-28T20:33:58.099160+0000","last_clean":"2023-11-28T20:33:58.099160+0000","last_became_active":"2023-11-28T20:33:53.947303+0000","last_became_peered":"2023-11-28T20:33:53.947303+0000","last_unstale":"2023-11-28T20:33:58.099160+0000","last_undegraded":"2023-11-28T20:33:58.099160+0000","last_fullsized":"2023-11-28T20:33:58.099160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:23.240825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.127814+0000","last_change":"2023-11-28T20:33:53.939826+0000","last_active":"2023-11-28T20:33:58.127814+0000","last_peered":"2023-11-28T20:33:58.127814+0000","last_clean":"2023-11-28T20:33:58.127814+0000","last_became_active":"2023-11-28T20:33:53.939349+0000","last_became_peered":"2023-11-28T20:33:53.939349+0000","last_unstale":"2023-11-28T20:33:58.127814+0000","last_undegraded":"2023-11-28T20:33:58.127814+0000","last_fullsized":"2023-11-28T20:33:58.127814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:13.738167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120426+0000","last_change":"2023-11-28T20:33:53.939569+0000","last_active":"2023-11-28T20:33:58.120426+0000","last_peered":"2023-11-28T20:33:58.120426+0000","last_clean":"2023-11-28T20:33:58.120426+0000","last_became_active":"2023-11-28T20:33:53.936362+0000","last_became_peered":"2023-11-28T20:33:53.936362+0000","last_unstale":"2023-11-28T20:33:58.120426+0000","last_undegraded":"2023-11-28T20:33:58.120426+0000","last_fullsized":"2023-11-28T20:33:58.120426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123031+0000","last_change":"2023-11-28T20:33:53.950757+0000","last_active":"2023-11-28T20:33:58.123031+0000","last_peered":"2023-11-28T20:33:58.123031+0000","last_clean":"2023-11-28T20:33:58.123031+0000","last_became_active":"2023-11-28T20:33:53.948014+0000","last_became_peered":"2023-11-28T20:33:53.948014+0000","last_unstale":"2023-11-28T20:33:58.123031+0000","last_undegraded":"2023-11-28T20:33:58.123031+0000","last_fullsized":"2023-11-28T20:33:58.123031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117566+0000","last_change":"2023-11-28T20:33:53.938422+0000","last_active":"2023-11-28T20:33:58.117566+0000","last_peered":"2023-11-28T20:33:58.117566+0000","last_clean":"2023-11-28T20:33:58.117566+0000","last_became_active":"2023-11-28T20:33:53.936194+0000","last_became_peered":"2023-11-28T20:33:53.936194+0000","last_unstale":"2023-11-28T20:33:58.117566+0000","last_undegraded":"2023-11-28T20:33:58.117566+0000","last_fullsized":"2023-11-28T20:33:58.117566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:35.450849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103553+0000","last_change":"2023-11-28T20:33:53.946481+0000","last_active":"2023-11-28T20:33:58.103553+0000","last_peered":"2023-11-28T20:33:58.103553+0000","last_clean":"2023-11-28T20:33:58.103553+0000","last_became_active":"2023-11-28T20:33:53.946123+0000","last_became_peered":"2023-11-28T20:33:53.946123+0000","last_unstale":"2023-11-28T20:33:58.103553+0000","last_undegraded":"2023-11-28T20:33:58.103553+0000","last_fullsized":"2023-11-28T20:33:58.103553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105089+0000","last_change":"2023-11-28T20:33:53.960696+0000","last_active":"2023-11-28T20:33:58.105089+0000","last_peered":"2023-11-28T20:33:58.105089+0000","last_clean":"2023-11-28T20:33:58.105089+0000","last_became_active":"2023-11-28T20:33:53.953779+0000","last_became_peered":"2023-11-28T20:33:53.953779+0000","last_unstale":"2023-11-28T20:33:58.105089+0000","last_undegraded":"2023-11-28T20:33:58.105089+0000","last_fullsized":"2023-11-28T20:33:58.105089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:47.468024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126504+0000","last_change":"2023-11-28T20:33:53.938502+0000","last_active":"2023-11-28T20:33:58.126504+0000","last_peered":"2023-11-28T20:33:58.126504+0000","last_clean":"2023-11-28T20:33:58.126504+0000","last_became_active":"2023-11-28T20:33:53.938133+0000","last_became_peered":"2023-11-28T20:33:53.938133+0000","last_unstale":"2023-11-28T20:33:58.126504+0000","last_undegraded":"2023-11-28T20:33:58.126504+0000","last_fullsized":"2023-11-28T20:33:58.126504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126197+0000","last_change":"2023-11-28T20:33:53.961946+0000","last_active":"2023-11-28T20:33:58.126197+0000","last_peered":"2023-11-28T20:33:58.126197+0000","last_clean":"2023-11-28T20:33:58.126197+0000","last_became_active":"2023-11-28T20:33:53.961681+0000","last_became_peered":"2023-11-28T20:33:53.961681+0000","last_unstale":"2023-11-28T20:33:58.126197+0000","last_undegraded":"2023-11-28T20:33:58.126197+0000","last_fullsized":"2023-11-28T20:33:58.126197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152700+0000","last_change":"2023-11-28T20:33:53.963253+0000","last_active":"2023-11-28T20:33:58.152700+0000","last_peered":"2023-11-28T20:33:58.152700+0000","last_clean":"2023-11-28T20:33:58.152700+0000","last_became_active":"2023-11-28T20:33:53.961179+0000","last_became_peered":"2023-11-28T20:33:53.961179+0000","last_unstale":"2023-11-28T20:33:58.152700+0000","last_undegraded":"2023-11-28T20:33:58.152700+0000","last_fullsized":"2023-11-28T20:33:58.152700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:35.843791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103213+0000","last_change":"2023-11-28T20:33:53.952078+0000","last_active":"2023-11-28T20:33:58.103213+0000","last_peered":"2023-11-28T20:33:58.103213+0000","last_clean":"2023-11-28T20:33:58.103213+0000","last_became_active":"2023-11-28T20:33:53.949626+0000","last_became_peered":"2023-11-28T20:33:53.949626+0000","last_unstale":"2023-11-28T20:33:58.103213+0000","last_undegraded":"2023-11-28T20:33:58.103213+0000","last_fullsized":"2023-11-28T20:33:58.103213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151515+0000","last_change":"2023-11-28T20:33:53.955688+0000","last_active":"2023-11-28T20:33:58.151515+0000","last_peered":"2023-11-28T20:33:58.151515+0000","last_clean":"2023-11-28T20:33:58.151515+0000","last_became_active":"2023-11-28T20:33:53.955384+0000","last_became_peered":"2023-11-28T20:33:53.955384+0000","last_unstale":"2023-11-28T20:33:58.151515+0000","last_undegraded":"2023-11-28T20:33:58.151515+0000","last_fullsized":"2023-11-28T20:33:58.151515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125802+0000","last_change":"2023-11-28T20:33:53.958084+0000","last_active":"2023-11-28T20:33:58.125802+0000","last_peered":"2023-11-28T20:33:58.125802+0000","last_clean":"2023-11-28T20:33:58.125802+0000","last_became_active":"2023-11-28T20:33:53.953483+0000","last_became_peered":"2023-11-28T20:33:53.953483+0000","last_unstale":"2023-11-28T20:33:58.125802+0000","last_undegraded":"2023-11-28T20:33:58.125802+0000","last_fullsized":"2023-11-28T20:33:58.125802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151629+0000","last_change":"2023-11-28T20:33:53.955124+0000","last_active":"2023-11-28T20:33:58.151629+0000","last_peered":"2023-11-28T20:33:58.151629+0000","last_clean":"2023-11-28T20:33:58.151629+0000","last_became_active":"2023-11-28T20:33:53.954422+0000","last_became_peered":"2023-11-28T20:33:53.954422+0000","last_unstale":"2023-11-28T20:33:58.151629+0000","last_undegraded":"2023-11-28T20:33:58.151629+0000","last_fullsized":"2023-11-28T20:33:58.151629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:19.573371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121991+0000","last_change":"2023-11-28T20:33:53.957855+0000","last_active":"2023-11-28T20:33:58.121991+0000","last_peered":"2023-11-28T20:33:58.121991+0000","last_clean":"2023-11-28T20:33:58.121991+0000","last_became_active":"2023-11-28T20:33:53.955645+0000","last_became_peered":"2023-11-28T20:33:53.955645+0000","last_unstale":"2023-11-28T20:33:58.121991+0000","last_undegraded":"2023-11-28T20:33:58.121991+0000","last_fullsized":"2023-11-28T20:33:58.121991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116851+0000","last_change":"2023-11-28T20:33:53.937690+0000","last_active":"2023-11-28T20:33:58.116851+0000","last_peered":"2023-11-28T20:33:58.116851+0000","last_clean":"2023-11-28T20:33:58.116851+0000","last_became_active":"2023-11-28T20:33:53.937374+0000","last_became_peered":"2023-11-28T20:33:53.937374+0000","last_unstale":"2023-11-28T20:33:58.116851+0000","last_undegraded":"2023-11-28T20:33:58.116851+0000","last_fullsized":"2023-11-28T20:33:58.116851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123601+0000","last_change":"2023-11-28T20:33:53.950129+0000","last_active":"2023-11-28T20:33:58.123601+0000","last_peered":"2023-11-28T20:33:58.123601+0000","last_clean":"2023-11-28T20:33:58.123601+0000","last_became_active":"2023-11-28T20:33:53.947280+0000","last_became_peered":"2023-11-28T20:33:53.947280+0000","last_unstale":"2023-11-28T20:33:58.123601+0000","last_undegraded":"2023-11-28T20:33:58.123601+0000","last_fullsized":"2023-11-28T20:33:58.123601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152690+0000","last_change":"2023-11-28T20:33:53.964432+0000","last_active":"2023-11-28T20:33:58.152690+0000","last_peered":"2023-11-28T20:33:58.152690+0000","last_clean":"2023-11-28T20:33:58.152690+0000","last_became_active":"2023-11-28T20:33:53.963967+0000","last_became_peered":"2023-11-28T20:33:53.963967+0000","last_unstale":"2023-11-28T20:33:58.152690+0000","last_undegraded":"2023-11-28T20:33:58.152690+0000","last_fullsized":"2023-11-28T20:33:58.152690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:07.426176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":695413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6220000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":104463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":274432,"data_stored":104623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":695469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6120000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.093}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:34:25.040 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T20:34:25.040 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-28T20:34:25.041 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-28T20:34:25.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-28T20:34:25.041 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-28T20:34:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.304+0000 7f03f4c35640 1 Processor -- start 2023-11-28T20:34:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.304+0000 7f03f4c35640 1 -- start start 2023-11-28T20:34:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03f4c35640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f03f015d550 0x7f03f00cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03f4c35640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f03f00d1c80 0x7f03f00b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03f4c35640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f00d2230 con 0x7f03f00d0100 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03f4c35640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f00d0590 con 0x7f03f00d1c80 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03f4c35640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f00d2510 con 0x7f03f015d550 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03effff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f03f00d1c80 0x7f03f00b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03e6ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f03f00d0100 0x7f03f00d1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51800/0 (socket says 172.21.15.45:51800) 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03effff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f03f00d1c80 0x7f03f00b6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58724/0 (socket says 172.21.15.45:58724) 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03e6ffd640 1 -- 172.21.15.45:0/3252491841 learned_addr learned my addr 172.21.15.45:0/3252491841 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.306+0000 7f03ef7fe640 1 --2- 172.21.15.45:0/3252491841 >> v2:172.21.15.190:3300/0 conn(0x7f03f015d550 0x7f03f00cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.307+0000 7f03effff640 1 -- 172.21.15.45:0/3252491841 >> v2:172.21.15.190:3300/0 conn(0x7f03f015d550 msgr2=0x7f03f00cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.307+0000 7f03effff640 1 --2- 172.21.15.45:0/3252491841 >> v2:172.21.15.190:3300/0 conn(0x7f03f015d550 0x7f03f00cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.307+0000 7f03effff640 1 -- 172.21.15.45:0/3252491841 >> v1:172.21.15.45:6789/0 conn(0x7f03f00d0100 legacy=0x7f03f00d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.307+0000 7f03effff640 1 -- 172.21.15.45:0/3252491841 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f00d27f0 con 0x7f03f00d1c80 2023-11-28T20:34:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.307+0000 7f03eeffd640 1 -- 172.21.15.45:0/3252491841 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147736317 0 0) 0x7f03f00d2230 con 0x7f03f00d0100 2023-11-28T20:34:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.308+0000 7f03effff640 1 --2- 172.21.15.45:0/3252491841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f00d1c80 0x7f03f00b6f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f03f00b5dc0 tx=0x7f03e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73188034cee265 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.308+0000 7f03eeffd640 1 -- 172.21.15.45:0/3252491841 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8031830 con 0x7f03f00d1c80 2023-11-28T20:34:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.308+0000 7f03eeffd640 1 -- 172.21.15.45:0/3252491841 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e8031990 con 0x7f03f00d1c80 2023-11-28T20:34:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.308+0000 7f03eeffd640 1 -- 172.21.15.45:0/3252491841 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8031d20 con 0x7f03f00d1c80 2023-11-28T20:34:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.309+0000 7f03f4c35640 1 -- 172.21.15.45:0/3252491841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f00d1c80 msgr2=0x7f03f00b6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.309+0000 7f03f4c35640 1 --2- 172.21.15.45:0/3252491841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f00d1c80 0x7f03f00b6f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f03f00b5dc0 tx=0x7f03e802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.309+0000 7f03f4c35640 1 -- 172.21.15.45:0/3252491841 shutdown_connections 2023-11-28T20:34:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.309+0000 7f03f4c35640 1 --2- 172.21.15.45:0/3252491841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f00d1c80 0x7f03f00b6f50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.309+0000 7f03f4c35640 1 --2- 172.21.15.45:0/3252491841 >> v2:172.21.15.190:3300/0 conn(0x7f03f015d550 0x7f03f00cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.309+0000 7f03f4c35640 1 -- 172.21.15.45:0/3252491841 >> 172.21.15.45:0/3252491841 conn(0x7f03f006ac70 msgr2=0x7f03f00b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.309+0000 7f03f4c35640 1 -- 172.21.15.45:0/3252491841 shutdown_connections 2023-11-28T20:34:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.309+0000 7f03f4c35640 1 -- 172.21.15.45:0/3252491841 wait complete. 2023-11-28T20:34:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.311+0000 7f03f4c35640 1 Processor -- start 2023-11-28T20:34:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.311+0000 7f03f4c35640 1 -- start start 2023-11-28T20:34:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.312+0000 7f03f4c35640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03f00d0100 0x7f03f015b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.312+0000 7f03f4c35640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03f00d1c80 0x7f03f00aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.312+0000 7f03f4c35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f015d550 0x7f03f00abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.312+0000 7f03f4c35640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f00ccd60 con 0x7f03f015d550 2023-11-28T20:34:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.312+0000 7f03f4c35640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f015b5b0 con 0x7f03f00d0100 2023-11-28T20:34:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.312+0000 7f03f4c35640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f00cc1e0 con 0x7f03f00d1c80 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.312+0000 7f03effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f015d550 0x7f03f00abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.312+0000 7f03effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f015d550 0x7f03f00abfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58734/0 (socket says 172.21.15.45:58734) 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.312+0000 7f03effff640 1 -- 172.21.15.45:0/405242658 learned_addr learned my addr 172.21.15.45:0/405242658 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.313+0000 7f03ef7fe640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03f00d0100 0x7f03f015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.313+0000 7f03e6ffd640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03f00d1c80 0x7f03f00aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.313+0000 7f03effff640 1 -- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03f00d1c80 msgr2=0x7f03f00aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.313+0000 7f03effff640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03f00d1c80 0x7f03f00aba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.313+0000 7f03effff640 1 -- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03f00d0100 msgr2=0x7f03f015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.313+0000 7f03effff640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03f00d0100 0x7f03f015b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.313+0000 7f03effff640 1 -- 172.21.15.45:0/405242658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e8009400 con 0x7f03f015d550 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.313+0000 7f03e6ffd640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03f00d1c80 0x7f03f00aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.313+0000 7f03effff640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f015d550 0x7f03f00abfa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009860 tx=0x7f03e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.314+0000 7f03ecff9640 1 -- 172.21.15.45:0/405242658 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e80317d0 con 0x7f03f015d550 2023-11-28T20:34:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.314+0000 7f03ecff9640 1 -- 172.21.15.45:0/405242658 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e803ba20 con 0x7f03f015d550 2023-11-28T20:34:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.314+0000 7f03ecff9640 1 -- 172.21.15.45:0/405242658 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e803bdd0 con 0x7f03f015d550 2023-11-28T20:34:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.314+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f00cc4c0 con 0x7f03f015d550 2023-11-28T20:34:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.314+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f00cca80 con 0x7f03f015d550 2023-11-28T20:34:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.315+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f004e920 con 0x7f03f015d550 2023-11-28T20:34:25.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.316+0000 7f03ecff9640 1 -- 172.21.15.45:0/405242658 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f03e80433f0 con 0x7f03f015d550 2023-11-28T20:34:25.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.317+0000 7f03ecff9640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03d006ae40 0x7f03d006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.317+0000 7f03ecff9640 1 -- 172.21.15.45:0/405242658 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f03e80ab7b0 con 0x7f03f015d550 2023-11-28T20:34:25.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.318+0000 7f03ef7fe640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03d006ae40 0x7f03d006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.319+0000 7f03ef7fe640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03d006ae40 0x7f03d006d300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f03e0006fd0 tx=0x7f03e0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.322+0000 7f03ecff9640 1 -- 172.21.15.45:0/405242658 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03e8076680 con 0x7f03f015d550 2023-11-28T20:34:25.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.562+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f03f015f140 con 0x7f03f015d550 2023-11-28T20:34:25.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.572+0000 7f03ecff9640 1 -- 172.21.15.45:0/405242658 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171039 (secure 0 0 0) 0x7f03e8079ca0 con 0x7f03f015d550 2023-11-28T20:34:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03d006ae40 msgr2=0x7f03d006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03d006ae40 0x7f03d006d300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f03e0006fd0 tx=0x7f03e0008040 comp rx=0 tx=0).stop 2023-11-28T20:34:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f015d550 msgr2=0x7f03f00abfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f015d550 0x7f03f00abfa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009860 tx=0x7f03e80046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 shutdown_connections 2023-11-28T20:34:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03d006ae40 0x7f03d006d300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f015d550 0x7f03f00abfa0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03f00d1c80 0x7f03f00aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 --2- 172.21.15.45:0/405242658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03f00d0100 0x7f03f015b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 >> 172.21.15.45:0/405242658 conn(0x7f03f006ac70 msgr2=0x7f03f015c820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 shutdown_connections 2023-11-28T20:34:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.576+0000 7f03f4c35640 1 -- 172.21.15.45:0/405242658 wait complete. 2023-11-28T20:34:25.599 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":5,"active_gid":4109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1081538426},{"type":"v1","addr":"172.21.15.190:6833","nonce":1081538426}]},"active_addr":"172.21.15.190:6833/1081538426","active_change":"2023-11-28T20:33:39.104307+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":1866894774}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":1813745990}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3262798906}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3717946281}]}]} 2023-11-28T20:34:25.605 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-28T20:34:25.605 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-28T20:34:25.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:25.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.834+0000 7f1523425640 1 Processor -- start 2023-11-28T20:34:25.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.834+0000 7f1523425640 1 -- start start 2023-11-28T20:34:25.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.835+0000 7f1523425640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f151c155780 0x7f151c0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.835+0000 7f1523425640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c0ae540 0x7f151c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.836+0000 7f1523425640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c15c3c0 con 0x7f151c0ce630 2023-11-28T20:34:25.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.836+0000 7f1523425640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151c0ceac0 con 0x7f151c155780 2023-11-28T20:34:25.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.836+0000 7f1523425640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c155b60 con 0x7f151c0ae540 2023-11-28T20:34:25.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.836+0000 7f1522c24640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f151c0ce630 0x7f151c15ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51812/0 (socket says 172.21.15.45:51812) 2023-11-28T20:34:25.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.836+0000 7f1522423640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f151c155780 0x7f151c0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.836+0000 7f1522c24640 1 -- 172.21.15.45:0/2974753501 learned_addr learned my addr 172.21.15.45:0/2974753501 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.836+0000 7f1521c22640 1 --2- 172.21.15.45:0/2974753501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c0ae540 0x7f151c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.836+0000 7f1521421640 1 -- 172.21.15.45:0/2974753501 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953691399 0 0) 0x7f151c15c3c0 con 0x7f151c0ce630 2023-11-28T20:34:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.837+0000 7f1521421640 1 -- 172.21.15.45:0/2974753501 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1508003520 con 0x7f151c0ce630 2023-11-28T20:34:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.837+0000 7f1522423640 1 -- 172.21.15.45:0/2974753501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c0ae540 msgr2=0x7f151c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.837+0000 7f1522423640 1 --2- 172.21.15.45:0/2974753501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c0ae540 0x7f151c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.837+0000 7f1522423640 1 -- 172.21.15.45:0/2974753501 >> v1:172.21.15.45:6789/0 conn(0x7f151c0ce630 legacy=0x7f151c15ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.837+0000 7f1522423640 1 -- 172.21.15.45:0/2974753501 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c15ba90 con 0x7f151c155780 2023-11-28T20:34:25.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.838+0000 7f1522423640 1 --2- 172.21.15.45:0/2974753501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c155780 0x7f151c0abd70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f15180097d0 tx=0x7f151802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627a53a01028b982 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:25.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.839+0000 7f1521421640 1 -- 172.21.15.45:0/2974753501 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518031800 con 0x7f151c155780 2023-11-28T20:34:25.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.839+0000 7f1521421640 1 -- 172.21.15.45:0/2974753501 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1518031960 con 0x7f151c155780 2023-11-28T20:34:25.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.839+0000 7f1521421640 1 -- 172.21.15.45:0/2974753501 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518031cd0 con 0x7f151c155780 2023-11-28T20:34:25.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.839+0000 7f1523425640 1 -- 172.21.15.45:0/2974753501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c155780 msgr2=0x7f151c0abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.839+0000 7f1523425640 1 --2- 172.21.15.45:0/2974753501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c155780 0x7f151c0abd70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f15180097d0 tx=0x7f151802b8a0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.839+0000 7f1523425640 1 -- 172.21.15.45:0/2974753501 shutdown_connections 2023-11-28T20:34:25.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.839+0000 7f1523425640 1 --2- 172.21.15.45:0/2974753501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c0ae540 0x7f151c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.839+0000 7f1523425640 1 --2- 172.21.15.45:0/2974753501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c155780 0x7f151c0abd70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.839+0000 7f1523425640 1 -- 172.21.15.45:0/2974753501 >> 172.21.15.45:0/2974753501 conn(0x7f151c06ab30 msgr2=0x7f151c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:25.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.840+0000 7f1523425640 1 -- 172.21.15.45:0/2974753501 shutdown_connections 2023-11-28T20:34:25.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.840+0000 7f1523425640 1 -- 172.21.15.45:0/2974753501 wait complete. 2023-11-28T20:34:25.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.841+0000 7f1523425640 1 Processor -- start 2023-11-28T20:34:25.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.841+0000 7f1523425640 1 -- start start 2023-11-28T20:34:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.842+0000 7f1523425640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f151c0ae540 0x7f151c14fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.842+0000 7f1523425640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c0ce630 0x7f151c0afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.842+0000 7f1523425640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c155780 0x7f151c0b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.842+0000 7f1523425640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c150bd0 con 0x7f151c0ce630 2023-11-28T20:34:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.842+0000 7f1523425640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c1f2c50 con 0x7f151c0ae540 2023-11-28T20:34:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.842+0000 7f1523425640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c1f2f10 con 0x7f151c155780 2023-11-28T20:34:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1521c22640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c0ce630 0x7f151c0afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1521c22640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c0ce630 0x7f151c0afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58752/0 (socket says 172.21.15.45:58752) 2023-11-28T20:34:25.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1521c22640 1 -- 172.21.15.45:0/611325979 learned_addr learned my addr 172.21.15.45:0/611325979 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:25.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1522c24640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c155780 0x7f151c0b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1522423640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f151c0ae540 0x7f151c14fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1521c22640 1 -- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c155780 msgr2=0x7f151c0b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1521c22640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c155780 0x7f151c0b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1521c22640 1 -- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f151c0ae540 msgr2=0x7f151c14fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1521c22640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f151c0ae540 0x7f151c14fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1521c22640 1 -- 172.21.15.45:0/611325979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1518009400 con 0x7f151c0ce630 2023-11-28T20:34:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.843+0000 7f1522423640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f151c0ae540 0x7f151c14fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.844+0000 7f1521c22640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c0ce630 0x7f151c0afb80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f150c00e870 tx=0x7f150c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.844+0000 7f1512ffd640 1 -- 172.21.15.45:0/611325979 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c0094f0 con 0x7f151c0ce630 2023-11-28T20:34:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.844+0000 7f1512ffd640 1 -- 172.21.15.45:0/611325979 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c00c940 con 0x7f151c0ce630 2023-11-28T20:34:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.844+0000 7f1512ffd640 1 -- 172.21.15.45:0/611325979 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c00ccf0 con 0x7f151c0ce630 2023-11-28T20:34:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.844+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c1f31f0 con 0x7f151c0ce630 2023-11-28T20:34:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.844+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c1f37b0 con 0x7f151c0ce630 2023-11-28T20:34:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.845+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c0cdca0 con 0x7f151c0ce630 2023-11-28T20:34:25.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.847+0000 7f1512ffd640 1 -- 172.21.15.45:0/611325979 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f150c013470 con 0x7f151c0ce630 2023-11-28T20:34:25.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.848+0000 7f1512ffd640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f150006ae40 0x7f150006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:25.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.848+0000 7f1512ffd640 1 -- 172.21.15.45:0/611325979 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f150c089a30 con 0x7f151c0ce630 2023-11-28T20:34:25.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.849+0000 7f1522423640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f150006ae40 0x7f150006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:25.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.850+0000 7f1522423640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f150006ae40 0x7f150006d300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1518036000 tx=0x7f1518008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:25.853+0000 7f1512ffd640 1 -- 172.21.15.45:0/611325979 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f150c0547e0 con 0x7f151c0ce630 2023-11-28T20:34:26.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.054+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f151c04e920 con 0x7f151c0ce630 2023-11-28T20:34:26.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.055+0000 7f1512ffd640 1 -- 172.21.15.45:0/611325979 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7f150c017090 con 0x7f151c0ce630 2023-11-28T20:34:26.057 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:26.057 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:26.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f150006ae40 msgr2=0x7f150006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f150006ae40 0x7f150006d300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1518036000 tx=0x7f1518008030 comp rx=0 tx=0).stop 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c0ce630 msgr2=0x7f151c0afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c0ce630 0x7f151c0afb80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f150c00e870 tx=0x7f150c00ed30 comp rx=0 tx=0).stop 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 shutdown_connections 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f150006ae40 0x7f150006d300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f151c155780 0x7f151c0b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f151c0ce630 0x7f151c0afb80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 --2- 172.21.15.45:0/611325979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f151c0ae540 0x7f151c14fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 >> 172.21.15.45:0/611325979 conn(0x7f151c06ab30 msgr2=0x7f151c0ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 shutdown_connections 2023-11-28T20:34:26.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.058+0000 7f1523425640 1 -- 172.21.15.45:0/611325979 wait complete. 2023-11-28T20:34:26.081 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-28T20:34:26.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:26.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.302+0000 7fc8e0aef640 1 Processor -- start 2023-11-28T20:34:26.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.302+0000 7fc8e0aef640 1 -- start start 2023-11-28T20:34:26.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.304+0000 7fc8e0aef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc06bc20 0x7fc8dc06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.304+0000 7fc8e0aef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc0aee20 con 0x7fc8dc159ec0 2023-11-28T20:34:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.304+0000 7fc8e0aef640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc15a2a0 con 0x7fc8dc0ae9b0 2023-11-28T20:34:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.304+0000 7fc8e0aef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc06e2e0 con 0x7fc8dc06bc20 2023-11-28T20:34:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.304+0000 7fc8db7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc8dc159ec0 0x7fc8dc06b510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51818/0 (socket says 172.21.15.45:51818) 2023-11-28T20:34:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.304+0000 7fc8db7fe640 1 -- 172.21.15.45:0/3795443922 learned_addr learned my addr 172.21.15.45:0/3795443922 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:26.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.304+0000 7fc8daffd640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc06bc20 0x7fc8dc06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.304+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415456478 0 0) 0x7fc8cc001110 con 0x7fc8dc159ec0 2023-11-28T20:34:26.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c8003520 con 0x7fc8dc159ec0 2023-11-28T20:34:26.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3490586264 0 0) 0x7fc8c8003520 con 0x7fc8dc159ec0 2023-11-28T20:34:26.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc06bc20 msgr2=0x7fc8dc06dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc06bc20 0x7fc8dc06dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 >> v1:172.21.15.190:6789/0 conn(0x7fc8dc0ae9b0 legacy=0x7fc8dc0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc06e5c0 con 0x7fc8dc159ec0 2023-11-28T20:34:26.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc8cc001800 con 0x7fc8dc159ec0 2023-11-28T20:34:26.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 >> v1:172.21.15.45:6789/0 conn(0x7fc8dc159ec0 legacy=0x7fc8dc06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8c80039a0 0x7fc8c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc06bc20 0x7fc8c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c8004d00 0x7fc8c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc06e2e0 con 0x7fc8c80039a0 2023-11-28T20:34:26.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c800ba60 con 0x7fc8c8004d00 2023-11-28T20:34:26.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c800de30 con 0x7fc8dc06bc20 2023-11-28T20:34:26.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.305+0000 7fc8db7fe640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8c80039a0 0x7fc8c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.306+0000 7fc8db7fe640 1 -- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc06bc20 msgr2=0x7fc8c80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.306+0000 7fc8db7fe640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc06bc20 0x7fc8c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.306+0000 7fc8dbfff640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c8004d00 0x7fc8c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.306+0000 7fc8db7fe640 1 -- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c8004d00 msgr2=0x7fc8c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.306+0000 7fc8db7fe640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c8004d00 0x7fc8c8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.306+0000 7fc8db7fe640 1 -- 172.21.15.45:0/3795443922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c8003520 con 0x7fc8c80039a0 2023-11-28T20:34:26.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.306+0000 7fc8dbfff640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c8004d00 0x7fc8c8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:26.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.307+0000 7fc8db7fe640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8c80039a0 0x7fc8c8003dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc001eb0 tx=0x7fc8c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68852752599e0728 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.307+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc005ad0 con 0x7fc8c80039a0 2023-11-28T20:34:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.307+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8cc005c50 con 0x7fc8c80039a0 2023-11-28T20:34:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.307+0000 7fc8da7fc640 1 -- 172.21.15.45:0/3795443922 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc02c7c0 con 0x7fc8c80039a0 2023-11-28T20:34:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.308+0000 7fc8e0aef640 1 -- 172.21.15.45:0/3795443922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8c80039a0 msgr2=0x7fc8c8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.308+0000 7fc8e0aef640 1 --2- 172.21.15.45:0/3795443922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8c80039a0 0x7fc8c8003dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc001eb0 tx=0x7fc8c80057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.308+0000 7fc8dbfff640 1 -- 172.21.15.45:0/3795443922 reap_dead start 2023-11-28T20:34:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.308+0000 7fc8e0aef640 1 -- 172.21.15.45:0/3795443922 shutdown_connections 2023-11-28T20:34:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.308+0000 7fc8e0aef640 1 -- 172.21.15.45:0/3795443922 >> 172.21.15.45:0/3795443922 conn(0x7fc8dc0712f0 msgr2=0x7fc8dc0716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.308+0000 7fc8e0aef640 1 -- 172.21.15.45:0/3795443922 shutdown_connections 2023-11-28T20:34:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.308+0000 7fc8e0aef640 1 -- 172.21.15.45:0/3795443922 wait complete. 2023-11-28T20:34:26.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.310+0000 7fc8e0aef640 1 Processor -- start 2023-11-28T20:34:26.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.310+0000 7fc8e0aef640 1 -- start start 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.311+0000 7fc8e0aef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c80039a0 0x7fc8dc155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.311+0000 7fc8e0aef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8dc0cda00 0x7fc8dc15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.311+0000 7fc8e0aef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc1e2590 0x7fc8dc1e2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.311+0000 7fc8e0aef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1e2f50 con 0x7fc8dc0cda00 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.311+0000 7fc8e0aef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1f2f30 con 0x7fc8c80039a0 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.311+0000 7fc8e0aef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc0cdf70 con 0x7fc8dc1e2590 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.311+0000 7fc8daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8dc0cda00 0x7fc8dc15c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.311+0000 7fc8daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8dc0cda00 0x7fc8dc15c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58762/0 (socket says 172.21.15.45:58762) 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.311+0000 7fc8daffd640 1 -- 172.21.15.45:0/1001383235 learned_addr learned my addr 172.21.15.45:0/1001383235 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8db7fe640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c80039a0 0x7fc8dc155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8daffd640 1 -- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc1e2590 msgr2=0x7fc8dc1e2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8daffd640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc1e2590 0x7fc8dc1e2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8daffd640 1 -- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c80039a0 msgr2=0x7fc8dc155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8daffd640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c80039a0 0x7fc8dc155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8daffd640 1 -- 172.21.15.45:0/1001383235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc0ce250 con 0x7fc8dc0cda00 2023-11-28T20:34:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8daffd640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8dc0cda00 0x7fc8dc15c7e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400e7c0 tx=0x7fc8c400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8d3fff640 1 -- 172.21.15.45:0/1001383235 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400eed0 con 0x7fc8dc0cda00 2023-11-28T20:34:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8d3fff640 1 -- 172.21.15.45:0/1001383235 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c400c940 con 0x7fc8dc0cda00 2023-11-28T20:34:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8d3fff640 1 -- 172.21.15.45:0/1001383235 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400ccd0 con 0x7fc8dc0cda00 2023-11-28T20:34:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.312+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc0ce530 con 0x7fc8dc0cda00 2023-11-28T20:34:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.313+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc1556d0 con 0x7fc8dc0cda00 2023-11-28T20:34:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.314+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc0cea50 con 0x7fc8dc0cda00 2023-11-28T20:34:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.315+0000 7fc8d3fff640 1 -- 172.21.15.45:0/1001383235 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc8c4013470 con 0x7fc8dc0cda00 2023-11-28T20:34:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.316+0000 7fc8d3fff640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc8bc06ae70 0x7fc8bc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.316+0000 7fc8d3fff640 1 -- 172.21.15.45:0/1001383235 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc8c4089860 con 0x7fc8dc0cda00 2023-11-28T20:34:26.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.316+0000 7fc8db7fe640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc8bc06ae70 0x7fc8bc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.318+0000 7fc8db7fe640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc8bc06ae70 0x7fc8bc06d330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc034000 tx=0x7fc8cc009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.320+0000 7fc8d3fff640 1 -- 172.21.15.45:0/1001383235 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8c40546b0 con 0x7fc8dc0cda00 2023-11-28T20:34:26.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.517+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8dc09e100 con 0x7fc8dc0cda00 2023-11-28T20:34:26.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.520+0000 7fc8d3fff640 1 -- 172.21.15.45:0/1001383235 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7fc8c4057cd0 con 0x7fc8dc0cda00 2023-11-28T20:34:26.523 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:26.524 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:26.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.523+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc8bc06ae70 msgr2=0x7fc8bc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.523+0000 7fc8e0aef640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc8bc06ae70 0x7fc8bc06d330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc034000 tx=0x7fc8cc009030 comp rx=0 tx=0).stop 2023-11-28T20:34:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8dc0cda00 msgr2=0x7fc8dc15c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8dc0cda00 0x7fc8dc15c7e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400e7c0 tx=0x7fc8c400ec80 comp rx=0 tx=0).stop 2023-11-28T20:34:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 shutdown_connections 2023-11-28T20:34:26.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc8bc06ae70 0x7fc8bc06d330 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8dc1e2590 0x7fc8dc1e2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8dc0cda00 0x7fc8dc15c7e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 --2- 172.21.15.45:0/1001383235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8c80039a0 0x7fc8dc155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 >> 172.21.15.45:0/1001383235 conn(0x7fc8dc0712f0 msgr2=0x7fc8dc1591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 shutdown_connections 2023-11-28T20:34:26.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.524+0000 7fc8e0aef640 1 -- 172.21.15.45:0/1001383235 wait complete. 2023-11-28T20:34:26.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-28T20:34:26.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-28T20:34:26.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-28T20:34:26.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-28T20:34:26.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-28T20:34:26.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-28T20:34:26.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-28T20:34:26.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-28T20:34:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.852+0000 7ffa80543640 1 Processor -- start 2023-11-28T20:34:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.853+0000 7ffa80543640 1 -- start start 2023-11-28T20:34:26.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.855+0000 7ffa80543640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa700e3120 0x7ffa700e7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.855+0000 7ffa80543640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ffa7003c060 0x7ffa70058010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.856+0000 7ffa80543640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa70053c70 0x7ffa700540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.856+0000 7ffa80543640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa70058550 con 0x7ffa70053c70 2023-11-28T20:34:26.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.856+0000 7ffa80543640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa70054620 con 0x7ffa7003c060 2023-11-28T20:34:26.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.857+0000 7ffa80543640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa700587d0 con 0x7ffa700e3120 2023-11-28T20:34:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.857+0000 7ffa7dade640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ffa7003c060 0x7ffa70058010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.857+0000 7ffa7e2df640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa700e3120 0x7ffa700e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.858+0000 7ffa7e2df640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa700e3120 0x7ffa700e7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:50810/0 (socket says 172.21.15.45:50810) 2023-11-28T20:34:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.858+0000 7ffa7e2df640 1 -- 172.21.15.45:0/1118296559 learned_addr learned my addr 172.21.15.45:0/1118296559 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.859+0000 7ffa7eae0640 1 --2- 172.21.15.45:0/1118296559 >> v2:172.21.15.45:3300/0 conn(0x7ffa70053c70 0x7ffa700540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.859+0000 7ffa7dade640 1 -- 172.21.15.45:0/1118296559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa700e3120 msgr2=0x7ffa700e7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.859+0000 7ffa7dade640 1 --2- 172.21.15.45:0/1118296559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa700e3120 0x7ffa700e7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.859+0000 7ffa7dade640 1 -- 172.21.15.45:0/1118296559 >> v2:172.21.15.45:3300/0 conn(0x7ffa70053c70 msgr2=0x7ffa700540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.859+0000 7ffa7dade640 1 --2- 172.21.15.45:0/1118296559 >> v2:172.21.15.45:3300/0 conn(0x7ffa70053c70 0x7ffa700540e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.859+0000 7ffa7dade640 1 -- 172.21.15.45:0/1118296559 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa70058ab0 con 0x7ffa7003c060 2023-11-28T20:34:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.859+0000 7ffa7e2df640 1 --2- 172.21.15.45:0/1118296559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa700e3120 0x7ffa700e7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.860+0000 7fcd8b59e640 1 Processor -- start 2023-11-28T20:34:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.861+0000 7fcd8b59e640 1 -- start start 2023-11-28T20:34:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.862+0000 7ffa7dade640 1 --2- 172.21.15.45:0/1118296559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa7003c060 0x7ffa70058010 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa70008530 tx=0x7ffa7402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12528a7d88ba9539 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.862+0000 7ffa7d2dd640 1 -- 172.21.15.45:0/1118296559 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa740317c0 con 0x7ffa7003c060 2023-11-28T20:34:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.862+0000 7ffa7d2dd640 1 -- 172.21.15.45:0/1118296559 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa74031920 con 0x7ffa7003c060 2023-11-28T20:34:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.863+0000 7fcd8b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd8404fa60 0x7fcd8404fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.863+0000 7fcd8b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcd840df3f0 0x7fcd84038290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.863+0000 7fcd8b59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcd840387d0 0x7fcd84038c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.863+0000 7fcd8b59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd84007740 con 0x7fcd840387d0 2023-11-28T20:34:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.863+0000 7fcd8b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd84006870 con 0x7fcd840df3f0 2023-11-28T20:34:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.863+0000 7fcd8b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd84050380 con 0x7fcd8404fa60 2023-11-28T20:34:26.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.862+0000 7ffa7d2dd640 1 -- 172.21.15.45:0/1118296559 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74031cf0 con 0x7ffa7003c060 2023-11-28T20:34:26.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7ffa80543640 1 -- 172.21.15.45:0/1118296559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa7003c060 msgr2=0x7ffa70058010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7ffa80543640 1 --2- 172.21.15.45:0/1118296559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa7003c060 0x7ffa70058010 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa70008530 tx=0x7ffa7402b860 comp rx=0 tx=0).stop 2023-11-28T20:34:26.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7ffa80543640 1 -- 172.21.15.45:0/1118296559 shutdown_connections 2023-11-28T20:34:26.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7ffa80543640 1 --2- 172.21.15.45:0/1118296559 >> v2:172.21.15.45:3300/0 conn(0x7ffa70053c70 0x7ffa700540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7ffa80543640 1 --2- 172.21.15.45:0/1118296559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa7003c060 0x7ffa70058010 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7ffa80543640 1 --2- 172.21.15.45:0/1118296559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa700e3120 0x7ffa700e7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7ffa80543640 1 -- 172.21.15.45:0/1118296559 >> 172.21.15.45:0/1118296559 conn(0x7ffa70058ea0 msgr2=0x7ffa70059280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7fcd89d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcd840df3f0 0x7fcd84038290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7fcd89d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcd840df3f0 0x7fcd84038290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49430/0 (socket says 172.21.15.45:49430) 2023-11-28T20:34:26.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7fcd89d9b640 1 -- 172.21.15.45:0/2780819984 learned_addr learned my addr 172.21.15.45:0/2780819984 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.864+0000 7fcd8a59c640 1 --2- 172.21.15.45:0/2780819984 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd8404fa60 0x7fcd8404fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.865+0000 7ffa80543640 1 -- 172.21.15.45:0/1118296559 shutdown_connections 2023-11-28T20:34:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.865+0000 7fcd8ad9d640 1 --2- 172.21.15.45:0/2780819984 >> v2:172.21.15.45:3300/0 conn(0x7fcd840387d0 0x7fcd84038c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.865+0000 7fcd89d9b640 1 -- 172.21.15.45:0/2780819984 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd8404fa60 msgr2=0x7fcd8404fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.865+0000 7fcd89d9b640 1 --2- 172.21.15.45:0/2780819984 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd8404fa60 0x7fcd8404fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.865+0000 7fcd89d9b640 1 -- 172.21.15.45:0/2780819984 >> v2:172.21.15.45:3300/0 conn(0x7fcd840387d0 msgr2=0x7fcd84038c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.865+0000 7fcd89d9b640 1 --2- 172.21.15.45:0/2780819984 >> v2:172.21.15.45:3300/0 conn(0x7fcd840387d0 0x7fcd84038c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.865+0000 7fcd89d9b640 1 -- 172.21.15.45:0/2780819984 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd84006ac0 con 0x7fcd840df3f0 2023-11-28T20:34:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.866+0000 7fcd8ad9d640 1 --2- 172.21.15.45:0/2780819984 >> v2:172.21.15.45:3300/0 conn(0x7fcd840387d0 0x7fcd84038c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.865+0000 7ffa80543640 1 -- 172.21.15.45:0/1118296559 wait complete. 2023-11-28T20:34:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.868+0000 7ffa80543640 1 Processor -- start 2023-11-28T20:34:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.868+0000 7fcd89d9b640 1 --2- 172.21.15.45:0/2780819984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd840df3f0 0x7fcd84038290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcd84052580 tx=0x7fcd8002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=654f7856738ebcab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.868+0000 7ffa80543640 1 -- start start 2023-11-28T20:34:26.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.869+0000 7fcd8959a640 1 -- 172.21.15.45:0/2780819984 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd80031870 con 0x7fcd840df3f0 2023-11-28T20:34:26.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.869+0000 7fcd8959a640 1 -- 172.21.15.45:0/2780819984 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd800319d0 con 0x7fcd840df3f0 2023-11-28T20:34:26.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.869+0000 7fcd8959a640 1 -- 172.21.15.45:0/2780819984 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd80031d60 con 0x7fcd840df3f0 2023-11-28T20:34:26.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.869+0000 7ffa80543640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa7003c060 0x7ffa700e4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.870+0000 7ffa80543640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa70053c70 0x7ffa700e5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.870+0000 7ffa80543640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa700e3120 0x7ffa70189670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.870+0000 7fcd8b59e640 1 -- 172.21.15.45:0/2780819984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd840df3f0 msgr2=0x7fcd84038290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7fcd8b59e640 1 --2- 172.21.15.45:0/2780819984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd840df3f0 0x7fcd84038290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcd84052580 tx=0x7fcd8002b910 comp rx=0 tx=0).stop 2023-11-28T20:34:26.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7fcd8b59e640 1 -- 172.21.15.45:0/2780819984 shutdown_connections 2023-11-28T20:34:26.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7fcd8b59e640 1 --2- 172.21.15.45:0/2780819984 >> v2:172.21.15.45:3300/0 conn(0x7fcd840387d0 0x7fcd84038c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7fcd8b59e640 1 --2- 172.21.15.45:0/2780819984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd840df3f0 0x7fcd84038290 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7fcd8b59e640 1 --2- 172.21.15.45:0/2780819984 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd8404fa60 0x7fcd8404fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7fcd8b59e640 1 -- 172.21.15.45:0/2780819984 >> 172.21.15.45:0/2780819984 conn(0x7fcd84009f80 msgr2=0x7fcd8400a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7fcd8b59e640 1 -- 172.21.15.45:0/2780819984 shutdown_connections 2023-11-28T20:34:26.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7ffa80543640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa70189bb0 con 0x7ffa7003c060 2023-11-28T20:34:26.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7fcd8b59e640 1 -- 172.21.15.45:0/2780819984 wait complete. 2023-11-28T20:34:26.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7ffa80543640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa700e64c0 con 0x7ffa700e3120 2023-11-28T20:34:26.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.871+0000 7ffa80543640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa700e3890 con 0x7ffa70053c70 2023-11-28T20:34:26.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.872+0000 7ffa7e2df640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa7003c060 0x7ffa700e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.874+0000 7fcd8b59e640 1 Processor -- start 2023-11-28T20:34:26.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.872+0000 7ffa7e2df640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa7003c060 0x7ffa700e4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58784/0 (socket says 172.21.15.45:58784) 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.872+0000 7ffa7e2df640 1 -- 172.21.15.45:0/2184391730 learned_addr learned my addr 172.21.15.45:0/2184391730 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.872+0000 7ffa7e2df640 1 -- 172.21.15.45:0/2184391730 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa70053c70 msgr2=0x7ffa700e5e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.872+0000 7ffa7e2df640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa70053c70 0x7ffa700e5e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.872+0000 7ffa7e2df640 1 -- 172.21.15.45:0/2184391730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa700e3120 msgr2=0x7ffa70189670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.872+0000 7ffa7e2df640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa700e3120 0x7ffa70189670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.872+0000 7ffa7e2df640 1 -- 172.21.15.45:0/2184391730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa680094d0 con 0x7ffa7003c060 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.873+0000 7ffa7e2df640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa7003c060 0x7ffa700e4650 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800e9b0 tx=0x7ffa6800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.873+0000 7ffa6effd640 1 -- 172.21.15.45:0/2184391730 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800c9a0 con 0x7ffa7003c060 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.873+0000 7ffa6effd640 1 -- 172.21.15.45:0/2184391730 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6800cb00 con 0x7ffa7003c060 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.873+0000 7ffa6effd640 1 -- 172.21.15.45:0/2184391730 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800ce70 con 0x7ffa7003c060 2023-11-28T20:34:26.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.874+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa74009400 con 0x7ffa7003c060 2023-11-28T20:34:26.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.874+0000 7fcd8b59e640 1 -- start start 2023-11-28T20:34:26.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.875+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa700e3e50 con 0x7ffa7003c060 2023-11-28T20:34:26.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.876+0000 7fcd8b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd840387d0 0x7fcd84185470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.877+0000 7fcd8b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd8404fa60 0x7fcd840dd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.877+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffa700e3500 con 0x7ffa7003c060 2023-11-28T20:34:26.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.877+0000 7fcd8b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd840df3f0 0x7fcd840ddc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.877+0000 7fcd8a59c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd840387d0 0x7fcd84185470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.877+0000 7fcd8a59c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd840387d0 0x7fcd84185470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:50836/0 (socket says 172.21.15.45:50836) 2023-11-28T20:34:26.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.877+0000 7fcd89d9b640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd8404fa60 0x7fcd840dd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.877+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd840d7070 con 0x7fcd8404fa60 2023-11-28T20:34:26.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.878+0000 7fcd8ad9d640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd840df3f0 0x7fcd840ddc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.878+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd840d7290 con 0x7fcd840df3f0 2023-11-28T20:34:26.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.878+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd840e0d40 con 0x7fcd840387d0 2023-11-28T20:34:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.877+0000 7fcd8a59c640 1 -- 172.21.15.45:0/1380031652 learned_addr learned my addr 172.21.15.45:0/1380031652 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.879+0000 7fcff1b0f640 1 Processor -- start 2023-11-28T20:34:26.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7ffa6effd640 1 -- 172.21.15.45:0/2184391730 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffa68013470 con 0x7ffa7003c060 2023-11-28T20:34:26.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7fcd8a59c640 1 -- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd840df3f0 msgr2=0x7fcd840ddc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7fcd8a59c640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd840df3f0 0x7fcd840ddc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7fcd8a59c640 1 -- 172.21.15.45:0/1380031652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd8404fa60 msgr2=0x7fcd840dd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.877+0000 7f474441e640 1 Processor -- start 2023-11-28T20:34:26.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.878+0000 7f474441e640 1 -- start start 2023-11-28T20:34:26.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.879+0000 7fcff1b0f640 1 -- start start 2023-11-28T20:34:26.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7f474441e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f473c161b30 0x7f473c06ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7f474441e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f473c0cd600 0x7f473c0cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7f474441e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473c0b2ca0 con 0x7f473c0cf800 2023-11-28T20:34:26.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7f474441e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473c0b2ea0 con 0x7f473c0cd600 2023-11-28T20:34:26.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7f474441e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c06e1d0 con 0x7f473c161b30 2023-11-28T20:34:26.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.880+0000 7fcd8a59c640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd8404fa60 0x7fcd840dd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7fcd8a59c640 1 -- 172.21.15.45:0/1380031652 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd800094d0 con 0x7fcd840387d0 2023-11-28T20:34:26.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7f47429bb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f473c0cd600 0x7f473c0cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7f47429bb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f473c0cd600 0x7f473c0cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58792/0 (socket says 172.21.15.45:58792) 2023-11-28T20:34:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7f47429bb640 1 -- 172.21.15.45:0/3834021496 learned_addr learned my addr 172.21.15.45:0/3834021496 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7f47421ba640 1 --2- 172.21.15.45:0/3834021496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f473c161b30 0x7f473c06ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7fcff1b0f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcfec15d730 0x7fcfec0d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7fcff1b0f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7fcff1b0f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfec06b580 con 0x7fcfec071060 2023-11-28T20:34:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7fcff1b0f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfec0714f0 con 0x7fcfec15d730 2023-11-28T20:34:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7fcff1b0f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec06b2a0 con 0x7fcfec0d2750 2023-11-28T20:34:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.882+0000 7fcd8ad9d640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd840df3f0 0x7fcd840ddc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7f08b6e2d640 1 Processor -- start 2023-11-28T20:34:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.881+0000 7f08b6e2d640 1 -- start start 2023-11-28T20:34:26.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.882+0000 7fcd89d9b640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd8404fa60 0x7fcd840dd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.882+0000 7f47421ba640 1 -- 172.21.15.45:0/3834021496 >> v2:172.21.15.45:3300/0 conn(0x7f473c0cd600 msgr2=0x7f473c0cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.882+0000 7f47421ba640 1 --2- 172.21.15.45:0/3834021496 >> v2:172.21.15.45:3300/0 conn(0x7f473c0cd600 0x7f473c0cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.882+0000 7f47421ba640 1 -- 172.21.15.45:0/3834021496 >> v1:172.21.15.45:6789/0 conn(0x7f473c0cf800 legacy=0x7f473c0ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.882+0000 7f47421ba640 1 -- 172.21.15.45:0/3834021496 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c0cba30 con 0x7f473c161b30 2023-11-28T20:34:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.882+0000 7fcfeb7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcfec15d730 0x7fcfec0d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.882+0000 7fcfeb7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcfec15d730 0x7fcfec0d2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49442/0 (socket says 172.21.15.45:49442) 2023-11-28T20:34:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.882+0000 7fcfeb7fe640 1 -- 172.21.15.45:0/2415767794 learned_addr learned my addr 172.21.15.45:0/2415767794 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.883+0000 7fcfeaffd640 1 --2- 172.21.15.45:0/2415767794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.883+0000 7ffa6effd640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa5c06af10 0x7ffa5c06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7ffa6effd640 1 -- 172.21.15.45:0/2184391730 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ffa68089ec0 con 0x7ffa7003c060 2023-11-28T20:34:26.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.883+0000 7fcd8a59c640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd840387d0 0x7fcd84185470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00d860 tx=0x7fcd7c00dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.883+0000 7f08b6e2d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08b00ce610 0x7f08b0159550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f08b6e2d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f08b00cf110 0x7f08b00cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f08b6e2d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b006cf90 con 0x7f08b006e370 2023-11-28T20:34:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f08b6e2d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b00cfb00 con 0x7f08b00cf110 2023-11-28T20:34:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f08b6e2d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00b3bc0 con 0x7f08b00ce610 2023-11-28T20:34:26.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7ffa6effd640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ffa5c074550 0x7ffa5c076990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7ffa6effd640 1 -- 172.21.15.45:0/2184391730 --> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa5c077060 con 0x7ffa5c074550 2023-11-28T20:34:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7ffa6effd640 1 -- 172.21.15.45:0/2184391730 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa6808c4e0 con 0x7ffa7003c060 2023-11-28T20:34:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7ffa7dade640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa5c06af10 0x7ffa5c06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7fcfeaffd640 1 -- 172.21.15.45:0/2415767794 >> v2:172.21.15.190:3300/0 conn(0x7fcfec15d730 msgr2=0x7fcfec0d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7fcfeaffd640 1 --2- 172.21.15.45:0/2415767794 >> v2:172.21.15.190:3300/0 conn(0x7fcfec15d730 0x7fcfec0d2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7fcfeaffd640 1 -- 172.21.15.45:0/2415767794 >> v1:172.21.15.190:6789/0 conn(0x7fcfec071060 legacy=0x7fcfec0b6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7fcfeaffd640 1 -- 172.21.15.45:0/2415767794 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec0d1730 con 0x7fcfec0d2750 2023-11-28T20:34:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f08b53ca640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f08b00cf110 0x7f08b00cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f572cbb6640 1 Processor -- start 2023-11-28T20:34:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f08b53ca640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f08b00cf110 0x7f08b00cf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58794/0 (socket says 172.21.15.45:58794) 2023-11-28T20:34:26.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f08b53ca640 1 -- 172.21.15.45:0/2163102285 learned_addr learned my addr 172.21.15.45:0/2163102285 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f47421ba640 1 --2- 172.21.15.45:0/3834021496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f473c161b30 0x7f473c06ce10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f47380097a0 tx=0x7f473802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8278e39d93942cb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.884+0000 7f572cbb6640 1 -- start start 2023-11-28T20:34:26.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7fcd7affd640 1 -- 172.21.15.45:0/1380031652 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c00c940 con 0x7fcd840387d0 2023-11-28T20:34:26.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7fcd7affd640 1 -- 172.21.15.45:0/1380031652 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c00caa0 con 0x7fcd840387d0 2023-11-28T20:34:26.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7fcd7affd640 1 -- 172.21.15.45:0/1380031652 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c00ce30 con 0x7fcd840387d0 2023-11-28T20:34:26.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd840d7550 con 0x7fcd840387d0 2023-11-28T20:34:26.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7f08b53ca640 1 -- 172.21.15.45:0/2163102285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08b00ce610 msgr2=0x7f08b0159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7f08b53ca640 1 --2- 172.21.15.45:0/2163102285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08b00ce610 0x7f08b0159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7f08b53ca640 1 -- 172.21.15.45:0/2163102285 >> v1:172.21.15.190:6789/0 conn(0x7f08b006e370 legacy=0x7f08b00cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7f47411b8640 1 -- 172.21.15.45:0/3834021496 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738031780 con 0x7f473c161b30 2023-11-28T20:34:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7f08b53ca640 1 -- 172.21.15.45:0/2163102285 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b00ceae0 con 0x7f08b00cf110 2023-11-28T20:34:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7f47411b8640 1 -- 172.21.15.45:0/3834021496 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47380318e0 con 0x7f473c161b30 2023-11-28T20:34:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7f47411b8640 1 -- 172.21.15.45:0/3834021496 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738031c70 con 0x7f473c161b30 2023-11-28T20:34:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.885+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd840d7b10 con 0x7fcd840387d0 2023-11-28T20:34:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.886+0000 7ffa7eae0640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ffa5c074550 0x7ffa5c076990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.886+0000 7ffa7dade640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa5c06af10 0x7ffa5c06d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa74036000 tx=0x7ffa74008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.886+0000 7f474441e640 1 -- 172.21.15.45:0/3834021496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f473c161b30 msgr2=0x7f473c06ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.886+0000 7f474441e640 1 --2- 172.21.15.45:0/3834021496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f473c161b30 0x7f473c06ce10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f47380097a0 tx=0x7f473802b820 comp rx=0 tx=0).stop 2023-11-28T20:34:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f08b53ca640 1 --2- 172.21.15.45:0/2163102285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08b00cf110 0x7f08b00cf5c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f08a8009e50 tx=0x7f08a802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982fb360e696e4b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f474441e640 1 -- 172.21.15.45:0/3834021496 shutdown_connections 2023-11-28T20:34:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f474441e640 1 --2- 172.21.15.45:0/3834021496 >> v2:172.21.15.45:3300/0 conn(0x7f473c0cd600 0x7f473c0cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f474441e640 1 --2- 172.21.15.45:0/3834021496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f473c161b30 0x7f473c06ce10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f474441e640 1 -- 172.21.15.45:0/3834021496 >> 172.21.15.45:0/3834021496 conn(0x7f473c06a500 msgr2=0x7f473c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcd84007360 con 0x7fcd840387d0 2023-11-28T20:34:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f474441e640 1 -- 172.21.15.45:0/3834021496 shutdown_connections 2023-11-28T20:34:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f08a77fe640 1 -- 172.21.15.45:0/2163102285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a802cb20 con 0x7f08b00cf110 2023-11-28T20:34:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f08a77fe640 1 -- 172.21.15.45:0/2163102285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a802cc80 con 0x7f08b00cf110 2023-11-28T20:34:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f08a77fe640 1 -- 172.21.15.45:0/2163102285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a80343f0 con 0x7f08b00cf110 2023-11-28T20:34:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f5c36fb7640 1 Processor -- start 2023-11-28T20:34:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f572cbb6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f572815d730 0x7f57280d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f572cbb6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 0x7f5728070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f572cbb6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572806b580 con 0x7f5728071060 2023-11-28T20:34:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f572cbb6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57280714f0 con 0x7f572815d730 2023-11-28T20:34:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f572cbb6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572806b2a0 con 0x7f57280d2750 2023-11-28T20:34:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.888+0000 7f1f87351640 1 Processor -- start 2023-11-28T20:34:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.888+0000 7f1f87351640 1 -- start start 2023-11-28T20:34:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.888+0000 7f5726d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5728071060 0x7f57280b6670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51836/0 (socket says 172.21.15.45:51836) 2023-11-28T20:34:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.888+0000 7f5726d9d640 1 -- 172.21.15.45:0/3168377299 learned_addr learned my addr 172.21.15.45:0/3168377299 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:26.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.888+0000 7f08b6e2d640 1 -- 172.21.15.45:0/2163102285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08b00cf110 msgr2=0x7f08b00cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.888+0000 7f08b6e2d640 1 --2- 172.21.15.45:0/2163102285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08b00cf110 0x7f08b00cf5c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f08a8009e50 tx=0x7f08a802bf80 comp rx=0 tx=0).stop 2023-11-28T20:34:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.888+0000 7fcfeaffd640 1 --2- 172.21.15.45:0/2415767794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec070b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40097a0 tx=0x7fcfd402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ffc6d27cad7b3b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.888+0000 7f5725d9b640 1 --2- 172.21.15.45:0/3168377299 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 0x7f5728070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f08b6e2d640 1 -- 172.21.15.45:0/2163102285 shutdown_connections 2023-11-28T20:34:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f08b6e2d640 1 --2- 172.21.15.45:0/2163102285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08b00cf110 0x7f08b00cf5c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f08b6e2d640 1 --2- 172.21.15.45:0/2163102285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08b00ce610 0x7f08b0159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f08b6e2d640 1 -- 172.21.15.45:0/2163102285 >> 172.21.15.45:0/2163102285 conn(0x7f08b006a6e0 msgr2=0x7f08b006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7fcfea7fc640 1 -- 172.21.15.45:0/2415767794 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd402d040 con 0x7fcfec0d2750 2023-11-28T20:34:26.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f08b6e2d640 1 -- 172.21.15.45:0/2163102285 shutdown_connections 2023-11-28T20:34:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7ffa7eae0640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ffa5c074550 0x7ffa5c076990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7fcfea7fc640 1 -- 172.21.15.45:0/2415767794 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd4031810 con 0x7fcfec0d2750 2023-11-28T20:34:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f572659c640 1 --2- 172.21.15.45:0/3168377299 >> v2:172.21.15.45:3300/0 conn(0x7f572815d730 0x7f57280d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.888+0000 7f5c36fb7640 1 -- start start 2023-11-28T20:34:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f5725d9b640 1 -- 172.21.15.45:0/3168377299 >> v2:172.21.15.45:3300/0 conn(0x7f572815d730 msgr2=0x7f57280d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f5725d9b640 1 --2- 172.21.15.45:0/3168377299 >> v2:172.21.15.45:3300/0 conn(0x7f572815d730 0x7f57280d2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f5725d9b640 1 -- 172.21.15.45:0/3168377299 >> v1:172.21.15.45:6789/0 conn(0x7f5728071060 legacy=0x7f57280b6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f5725d9b640 1 -- 172.21.15.45:0/3168377299 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57280d1730 con 0x7f57280d2750 2023-11-28T20:34:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7f5c36fb7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 0x7f5c300d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7fcfea7fc640 1 -- 172.21.15.45:0/2415767794 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4031ba0 con 0x7fcfec0d2750 2023-11-28T20:34:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7f5c36fb7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5c300d2750 0x7f5c30070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f572559a640 1 -- 172.21.15.45:0/3168377299 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705664196 0 0) 0x7f572806b580 con 0x7f5728071060 2023-11-28T20:34:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7f5c36fb7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5c30071060 0x7f5c300b6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.887+0000 7f474441e640 1 -- 172.21.15.45:0/3834021496 wait complete. 2023-11-28T20:34:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7fcff1b0f640 1 -- 172.21.15.45:0/2415767794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 msgr2=0x7fcfec070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7fcff1b0f640 1 --2- 172.21.15.45:0/2415767794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec070b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40097a0 tx=0x7fcfd402b8b0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7f5c36fb7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c30071510 con 0x7f5c300d2750 2023-11-28T20:34:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7f5c36fb7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3006b580 con 0x7f5c30071060 2023-11-28T20:34:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f5c36fb7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3006b2a0 con 0x7f5c3015d730 2023-11-28T20:34:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.889+0000 7f08b6e2d640 1 -- 172.21.15.45:0/2163102285 wait complete. 2023-11-28T20:34:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7f474441e640 1 Processor -- start 2023-11-28T20:34:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7fcff1b0f640 1 -- 172.21.15.45:0/2415767794 shutdown_connections 2023-11-28T20:34:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7fcff1b0f640 1 --2- 172.21.15.45:0/2415767794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec070b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7fcff1b0f640 1 --2- 172.21.15.45:0/2415767794 >> v2:172.21.15.190:3300/0 conn(0x7fcfec15d730 0x7fcfec0d2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7fcff1b0f640 1 -- 172.21.15.45:0/2415767794 >> 172.21.15.45:0/2415767794 conn(0x7fcfec0cfc70 msgr2=0x7fcfec0d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7f1f87351640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1f800cc3a0 0x7f1f8006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.890+0000 7f1f87351640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cf020 0x7f1f800cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f1f87351640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800b2fd0 con 0x7f1f80165e50 2023-11-28T20:34:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f1f87351640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f800cc780 con 0x7f1f800cc3a0 2023-11-28T20:34:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f1f87351640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800cf9f0 con 0x7f1f800cf020 2023-11-28T20:34:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f5c34d53640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 0x7f5c300d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7fcff1b0f640 1 -- 172.21.15.45:0/2415767794 shutdown_connections 2023-11-28T20:34:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f5c35554640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5c30071060 0x7f5c300b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f5c34d53640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 0x7f5c300d2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:50870/0 (socket says 172.21.15.45:50870) 2023-11-28T20:34:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f5c35554640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5c30071060 0x7f5c300b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49456/0 (socket says 172.21.15.45:49456) 2023-11-28T20:34:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f5c35554640 1 -- 172.21.15.45:0/32059955 learned_addr learned my addr 172.21.15.45:0/32059955 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f1f850ed640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1f800cc3a0 0x7f1f8006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f1f850ed640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1f800cc3a0 0x7f1f8006d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49472/0 (socket says 172.21.15.45:49472) 2023-11-28T20:34:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f1f850ed640 1 -- 172.21.15.45:0/2382558949 learned_addr learned my addr 172.21.15.45:0/2382558949 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f5725d9b640 1 --2- 172.21.15.45:0/3168377299 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 0x7f5728070b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f571c0097a0 tx=0x7f571c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c77fb47f044a86d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f1f858ee640 1 --2- 172.21.15.45:0/2382558949 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cf020 0x7f1f800cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f5c2ffff640 1 --2- 172.21.15.45:0/32059955 >> v2:172.21.15.45:3300/0 conn(0x7f5c300d2750 0x7f5c30070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7fcd7affd640 1 -- 172.21.15.45:0/1380031652 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcd7c01f440 con 0x7fcd840387d0 2023-11-28T20:34:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7fcff1b0f640 1 -- 172.21.15.45:0/2415767794 wait complete. 2023-11-28T20:34:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f08b6e2d640 1 Processor -- start 2023-11-28T20:34:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f572559a640 1 -- 172.21.15.45:0/3168377299 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c02d040 con 0x7f57280d2750 2023-11-28T20:34:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f572559a640 1 -- 172.21.15.45:0/3168377299 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c031810 con 0x7f57280d2750 2023-11-28T20:34:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f572559a640 1 -- 172.21.15.45:0/3168377299 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c031ba0 con 0x7f57280d2750 2023-11-28T20:34:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7ffa6effd640 1 -- 172.21.15.45:0/2184391730 <== osd.0 v2:172.21.15.45:6816/2634303336 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ffa5c077060 con 0x7ffa5c074550 2023-11-28T20:34:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f1f858ee640 1 -- 172.21.15.45:0/2382558949 >> v2:172.21.15.190:3300/0 conn(0x7f1f800cc3a0 msgr2=0x7f1f8006d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f1f858ee640 1 --2- 172.21.15.45:0/2382558949 >> v2:172.21.15.190:3300/0 conn(0x7f1f800cc3a0 0x7f1f8006d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f1f858ee640 1 -- 172.21.15.45:0/2382558949 >> v1:172.21.15.190:6789/0 conn(0x7f1f80165e50 legacy=0x7f1f800cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f1f858ee640 1 -- 172.21.15.45:0/2382558949 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8006e170 con 0x7f1f800cf020 2023-11-28T20:34:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f5c2ffff640 1 -- 172.21.15.45:0/32059955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 msgr2=0x7f5c300d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f5c2ffff640 1 --2- 172.21.15.45:0/32059955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 0x7f5c300d2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.891+0000 7f474441e640 1 -- start start 2023-11-28T20:34:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f5c2ffff640 1 -- 172.21.15.45:0/32059955 >> v2:172.21.15.190:3300/0 conn(0x7f5c30071060 msgr2=0x7f5c300b6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f5c2ffff640 1 --2- 172.21.15.45:0/32059955 >> v2:172.21.15.190:3300/0 conn(0x7f5c30071060 0x7f5c300b6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f5c2ffff640 1 -- 172.21.15.45:0/32059955 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c300d1730 con 0x7f5c300d2750 2023-11-28T20:34:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7fcd7affd640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd6c06ae70 0x7fcd6c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7fcd7affd640 1 -- 172.21.15.45:0/1380031652 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fcd7c089550 con 0x7fcd840387d0 2023-11-28T20:34:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7fcd7affd640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7fcd6c0744b0 0x7fcd6c0768f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7fcd7affd640 1 -- 172.21.15.45:0/1380031652 --> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd6c076fc0 con 0x7fcd6c0744b0 2023-11-28T20:34:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7fcd7affd640 1 -- 172.21.15.45:0/1380031652 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd7c039ea0 con 0x7fcd840387d0 2023-11-28T20:34:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7fcd89d9b640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd6c06ae70 0x7fcd6c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7fcd8ad9d640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7fcd6c0744b0 0x7fcd6c0768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.892+0000 7f08b6e2d640 1 -- start start 2023-11-28T20:34:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f08b6e2d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08b006e370 0x7f08b0135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f08b6e2d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08b00ce610 0x7f08b009c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f08b6e2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08b00cf110 0x7f08b01376e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f08b6e2d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b009c8e0 con 0x7f08b00cf110 2023-11-28T20:34:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f08b6e2d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b0137250 con 0x7f08b00ce610 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f08b6e2d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b0136640 con 0x7f08b006e370 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f08b53ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08b00cf110 0x7f08b01376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f08b4bc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08b006e370 0x7f08b0135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f08a7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08b00ce610 0x7f08b009c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f08a7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08b00ce610 0x7f08b009c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49486/0 (socket says 172.21.15.45:49486) 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f08a7fff640 1 -- 172.21.15.45:0/1570715977 learned_addr learned my addr 172.21.15.45:0/1570715977 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f474441e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f473c0cd600 0x7f473c1646a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f474441e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f473c0cf800 0x7f473c15be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f474441e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f473c161b30 0x7f473c1633f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f474441e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0aeea0 con 0x7f473c161b30 2023-11-28T20:34:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f474441e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c160900 con 0x7f473c0cd600 2023-11-28T20:34:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f474441e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0ae220 con 0x7f473c0cf800 2023-11-28T20:34:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f47421ba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f473c0cd600 0x7f473c1646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f47421ba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f473c0cd600 0x7f473c1646a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49496/0 (socket says 172.21.15.45:49496) 2023-11-28T20:34:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f47421ba640 1 -- 172.21.15.45:0/2596927900 learned_addr learned my addr 172.21.15.45:0/2596927900 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f1f858ee640 1 --2- 172.21.15.45:0/2382558949 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cf020 0x7f1f800cf4b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009d90 tx=0x7f1f7802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=630eeb6ec4cb474b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f1f77fff640 1 -- 172.21.15.45:0/2382558949 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7802cc50 con 0x7f1f800cf020 2023-11-28T20:34:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f1f77fff640 1 -- 172.21.15.45:0/2382558949 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7802cdb0 con 0x7f1f800cf020 2023-11-28T20:34:26.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f572cbb6640 1 -- 172.21.15.45:0/3168377299 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 msgr2=0x7f5728070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f572cbb6640 1 --2- 172.21.15.45:0/3168377299 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 0x7f5728070b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f571c0097a0 tx=0x7f571c02b8b0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.893+0000 7f572cbb6640 1 -- 172.21.15.45:0/3168377299 shutdown_connections 2023-11-28T20:34:26.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f572cbb6640 1 --2- 172.21.15.45:0/3168377299 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 0x7f5728070b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f572cbb6640 1 --2- 172.21.15.45:0/3168377299 >> v2:172.21.15.45:3300/0 conn(0x7f572815d730 0x7f57280d2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f572cbb6640 1 -- 172.21.15.45:0/3168377299 >> 172.21.15.45:0/3168377299 conn(0x7f57280cfc70 msgr2=0x7f57280d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f572cbb6640 1 -- 172.21.15.45:0/3168377299 shutdown_connections 2023-11-28T20:34:26.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.894+0000 7f572cbb6640 1 -- 172.21.15.45:0/3168377299 wait complete. 2023-11-28T20:34:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f08b53ca640 1 -- 172.21.15.45:0/1570715977 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08b006e370 msgr2=0x7f08b0135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f08b53ca640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08b006e370 0x7f08b0135f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f08b53ca640 1 -- 172.21.15.45:0/1570715977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08b00ce610 msgr2=0x7f08b009c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f08b53ca640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08b00ce610 0x7f08b009c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f1f77fff640 1 -- 172.21.15.45:0/2382558949 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78034660 con 0x7f1f800cf020 2023-11-28T20:34:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f08b53ca640 1 -- 172.21.15.45:0/1570715977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a8009b30 con 0x7f08b00cf110 2023-11-28T20:34:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7f5c2ffff640 1 --2- 172.21.15.45:0/32059955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c300d2750 0x7f5c30070b20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c300b27c0 tx=0x7f5c2002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bb66da6dcf4bab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.895+0000 7fcff1b0f640 1 Processor -- start 2023-11-28T20:34:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.896+0000 7f08b53ca640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08b00cf110 0x7f08b01376e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f08a8032000 tx=0x7f08a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.896+0000 7f47421ba640 1 -- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f473c0cf800 msgr2=0x7f473c15be20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T20:34:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.896+0000 7f47421ba640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f473c0cf800 0x7f473c15be20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.896+0000 7f47421ba640 1 -- 172.21.15.45:0/2596927900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f473c161b30 msgr2=0x7f473c1633f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.896+0000 7f47421ba640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f473c161b30 0x7f473c1633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.896+0000 7f47421ba640 1 -- 172.21.15.45:0/2596927900 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4738009400 con 0x7f473c0cd600 2023-11-28T20:34:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.896+0000 7f1f87351640 1 -- 172.21.15.45:0/2382558949 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cf020 msgr2=0x7f1f800cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.896+0000 7f1f87351640 1 --2- 172.21.15.45:0/2382558949 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cf020 0x7f1f800cf4b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009d90 tx=0x7f1f7802bea0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.896+0000 7fcd8ad9d640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7fcd6c0744b0 0x7fcd6c0768f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7fcd89d9b640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd6c06ae70 0x7fcd6c06d330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcd80036000 tx=0x7fcd8003c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f47421ba640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f473c0cd600 0x7f473c1646a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4738034040 tx=0x7f47380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7fcff1b0f640 1 -- start start 2023-11-28T20:34:26.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f08a57fa640 1 -- 172.21.15.45:0/1570715977 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a8004570 con 0x7f08b00cf110 2023-11-28T20:34:26.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f5c2f7fe640 1 -- 172.21.15.45:0/32059955 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2002e070 con 0x7f5c300d2750 2023-11-28T20:34:26.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f5c2f7fe640 1 -- 172.21.15.45:0/32059955 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c200315a0 con 0x7f5c300d2750 2023-11-28T20:34:26.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f5c2f7fe640 1 -- 172.21.15.45:0/32059955 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20031930 con 0x7f5c300d2750 2023-11-28T20:34:26.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b0136920 con 0x7f08b00cf110 2023-11-28T20:34:26.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f572cbb6640 1 Processor -- start 2023-11-28T20:34:26.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f4732ffd640 1 -- 172.21.15.45:0/2596927900 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738032820 con 0x7f473c0cd600 2023-11-28T20:34:26.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b0136ee0 con 0x7f08b00cf110 2023-11-28T20:34:26.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f1f87351640 1 -- 172.21.15.45:0/2382558949 shutdown_connections 2023-11-28T20:34:26.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f1f87351640 1 --2- 172.21.15.45:0/2382558949 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cf020 0x7f1f800cf4b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f1f87351640 1 --2- 172.21.15.45:0/2382558949 >> v2:172.21.15.190:3300/0 conn(0x7f1f800cc3a0 0x7f1f8006d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f1f87351640 1 -- 172.21.15.45:0/2382558949 >> 172.21.15.45:0/2382558949 conn(0x7f1f8006a6e0 msgr2=0x7f1f8006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f4732ffd640 1 -- 172.21.15.45:0/2596927900 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47380329a0 con 0x7f473c0cd600 2023-11-28T20:34:26.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f08a57fa640 1 -- 172.21.15.45:0/1570715977 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a80046d0 con 0x7f08b00cf110 2023-11-28T20:34:26.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7fcd7affd640 1 -- 172.21.15.45:0/1380031652 <== osd.1 v2:172.21.15.45:6800/901684499 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fcd6c076fc0 con 0x7fcd6c0744b0 2023-11-28T20:34:26.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f08a57fa640 1 -- 172.21.15.45:0/1570715977 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a803e640 con 0x7f08b00cf110 2023-11-28T20:34:26.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.898+0000 7f1f87351640 1 -- 172.21.15.45:0/2382558949 shutdown_connections 2023-11-28T20:34:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.898+0000 7f572cbb6640 1 -- start start 2023-11-28T20:34:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.897+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c0ae500 con 0x7f473c0cd600 2023-11-28T20:34:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.898+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c0aeac0 con 0x7f473c0cd600 2023-11-28T20:34:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.898+0000 7f4732ffd640 1 -- 172.21.15.45:0/2596927900 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738032d50 con 0x7f473c0cd600 2023-11-28T20:34:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.898+0000 7f1f87351640 1 -- 172.21.15.45:0/2382558949 wait complete. 2023-11-28T20:34:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.899+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4724000fc0 con 0x7f473c0cd600 2023-11-28T20:34:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.899+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0890000fc0 con 0x7f08b00cf110 2023-11-28T20:34:26.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f5c36fb7640 1 -- 172.21.15.45:0/32059955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c300d2750 msgr2=0x7f5c30070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f5c36fb7640 1 --2- 172.21.15.45:0/32059955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c300d2750 0x7f5c30070b20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c300b27c0 tx=0x7f5c2002b840 comp rx=0 tx=0).stop 2023-11-28T20:34:26.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f5c36fb7640 1 -- 172.21.15.45:0/32059955 shutdown_connections 2023-11-28T20:34:26.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f5c36fb7640 1 --2- 172.21.15.45:0/32059955 >> v2:172.21.15.190:3300/0 conn(0x7f5c30071060 0x7f5c300b6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f5c36fb7640 1 --2- 172.21.15.45:0/32059955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c300d2750 0x7f5c30070b20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f5c36fb7640 1 --2- 172.21.15.45:0/32059955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 0x7f5c300d2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f5c36fb7640 1 -- 172.21.15.45:0/32059955 >> 172.21.15.45:0/32059955 conn(0x7f5c300cfc70 msgr2=0x7f5c300d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f1f87351640 1 Processor -- start 2023-11-28T20:34:26.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f5c36fb7640 1 -- 172.21.15.45:0/32059955 shutdown_connections 2023-11-28T20:34:26.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7fcff1b0f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfec071060 0x7fcfec06c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7f1f87351640 1 -- start start 2023-11-28T20:34:26.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7fcff1b0f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec203e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7fcff1b0f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfec15d730 0x7fcfec1ff7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7fcff1b0f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec0cba30 con 0x7fcfec071060 2023-11-28T20:34:26.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7f4732ffd640 1 -- 172.21.15.45:0/2596927900 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f473804e440 con 0x7f473c0cd600 2023-11-28T20:34:26.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.900+0000 7f572cbb6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5728071060 0x7f572814f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7f572cbb6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 0x7f5728150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7f572cbb6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f572815d730 0x7f57280ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7f572cbb6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57280aeb50 con 0x7f5728071060 2023-11-28T20:34:26.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7f572cbb6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572814fc20 con 0x7f572815d730 2023-11-28T20:34:26.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7f572cbb6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57280adf40 con 0x7f57280d2750 2023-11-28T20:34:26.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7f572659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5728071060 0x7f572814f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7f572659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5728071060 0x7f572814f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58840/0 (socket says 172.21.15.45:58840) 2023-11-28T20:34:26.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7f572659c640 1 -- 172.21.15.45:0/3391967962 learned_addr learned my addr 172.21.15.45:0/3391967962 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7f5726d9d640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f572815d730 0x7f57280ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.903+0000 7f572659c640 1 -- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 msgr2=0x7f5728150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7f5725d9b640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 0x7f5728150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.903+0000 7f572659c640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 0x7f5728150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.903+0000 7f572659c640 1 -- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f572815d730 msgr2=0x7f57280ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.903+0000 7f572659c640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f572815d730 0x7f57280ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.903+0000 7f572659c640 1 -- 172.21.15.45:0/3391967962 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c009400 con 0x7f5728071060 2023-11-28T20:34:26.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.903+0000 7f5725d9b640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57280d2750 0x7f5728150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:26.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.901+0000 7f5c36fb7640 1 -- 172.21.15.45:0/32059955 wait complete. 2023-11-28T20:34:26.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.903+0000 7f5c36fb7640 1 Processor -- start 2023-11-28T20:34:26.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.903+0000 7f572659c640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5728071060 0x7f572814f6e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5720009d10 tx=0x7f572000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f1f87351640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cc3a0 0x7f1f800af5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f5c36fb7640 1 -- start start 2023-11-28T20:34:26.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f08a57fa640 1 -- 172.21.15.45:0/1570715977 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f08a803e7a0 con 0x7f08b00cf110 2023-11-28T20:34:26.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec203e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec203e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:50934/0 (socket says 172.21.15.45:50934) 2023-11-28T20:34:26.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7fcfeaffd640 1 -- 172.21.15.45:0/4248218079 learned_addr learned my addr 172.21.15.45:0/4248218079 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.902+0000 7fcfeb7fe640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfec071060 0x7fcfec06c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.903+0000 7fcfebfff640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfec15d730 0x7fcfec1ff7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec2043d0 con 0x7fcfec15d730 2023-11-28T20:34:26.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec1ffd00 con 0x7fcfec0d2750 2023-11-28T20:34:26.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f1f87351640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f800cf020 0x7f1f8015dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f1f87351640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f80165e50 0x7f1f800ad580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f5716ffd640 1 -- 172.21.15.45:0/3391967962 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5720009b80 con 0x7f5728071060 2023-11-28T20:34:26.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f1f850ed640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cc3a0 0x7f1f800af5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f5716ffd640 1 -- 172.21.15.45:0/3391967962 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572000f040 con 0x7f5728071060 2023-11-28T20:34:26.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572815dc90 con 0x7f5728071060 2023-11-28T20:34:26.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f1f87351640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800afb30 con 0x7f1f800cf020 2023-11-28T20:34:26.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f1f87351640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800cabd0 con 0x7f1f80165e50 2023-11-28T20:34:26.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f1f87351640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800abd70 con 0x7f1f800cc3a0 2023-11-28T20:34:26.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f1f850ed640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cc3a0 0x7f1f800af5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:50950/0 (socket says 172.21.15.45:50950) 2023-11-28T20:34:26.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.904+0000 7f1f850ed640 1 -- 172.21.15.45:0/3858156445 learned_addr learned my addr 172.21.15.45:0/3858156445 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57280ae500 con 0x7f5728071060 2023-11-28T20:34:26.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7f5716ffd640 1 -- 172.21.15.45:0/3391967962 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5720014680 con 0x7f5728071060 2023-11-28T20:34:26.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7f1f848ec640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f800cf020 0x7f1f8015dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7f1f858ee640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f80165e50 0x7f1f800ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7f4732ffd640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f472006af10 0x7f472006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f4732ffd640 1 -- 172.21.15.45:0/2596927900 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f47380abdd0 con 0x7f473c0cd600 2023-11-28T20:34:26.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f4732ffd640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f4720074550 0x7f4720076990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f4732ffd640 1 -- 172.21.15.45:0/2596927900 --> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4720077060 con 0x7f4720074550 2023-11-28T20:34:26.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7fcfeaffd640 1 -- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfec15d730 msgr2=0x7fcfec1ff7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7fcfeaffd640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfec15d730 0x7fcfec1ff7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7fcfeaffd640 1 -- 172.21.15.45:0/4248218079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfec071060 msgr2=0x7fcfec06c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7fcfeaffd640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfec071060 0x7fcfec06c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.905+0000 7fcfeaffd640 1 -- 172.21.15.45:0/4248218079 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd4009400 con 0x7fcfec0d2750 2023-11-28T20:34:26.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7fcfeaffd640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec203e90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40098c0 tx=0x7fcfd40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f1f850ed640 1 -- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f80165e50 msgr2=0x7f1f800ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f1f850ed640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f80165e50 0x7f1f800ad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f1f850ed640 1 -- 172.21.15.45:0/3858156445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f800cf020 msgr2=0x7f1f8015dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f1f850ed640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f800cf020 0x7f1f8015dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f1f850ed640 1 -- 172.21.15.45:0/3858156445 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78009a70 con 0x7f1f800cc3a0 2023-11-28T20:34:26.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f570c000fc0 con 0x7f5728071060 2023-11-28T20:34:26.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f5c36fb7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c30071060 0x7f5c300ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f5c36fb7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c300d2750 0x7f5c3015de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f5c34d53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c30071060 0x7f5c300ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f4732ffd640 1 -- 172.21.15.45:0/2596927900 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47380ac180 con 0x7f473c0cd600 2023-11-28T20:34:26.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f5c34d53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c30071060 0x7f5c300ca2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58870/0 (socket says 172.21.15.45:58870) 2023-11-28T20:34:26.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f5c34d53640 1 -- 172.21.15.45:0/2252407040 learned_addr learned my addr 172.21.15.45:0/2252407040 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:26.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.906+0000 7f47429bb640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f4720074550 0x7f4720076990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f5c36fb7640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 0x7f5c30159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f08a57fa640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f089c06ae40 0x7f089c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c30200d90 con 0x7f5c30071060 2023-11-28T20:34:26.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f1f850ed640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cc3a0 0x7f1f800af5f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009fd0 tx=0x7f1f7000ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7fcfe3fff640 1 -- 172.21.15.45:0/4248218079 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd403ad60 con 0x7fcfec0d2750 2023-11-28T20:34:26.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7fcfe3fff640 1 -- 172.21.15.45:0/4248218079 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd40443f0 con 0x7fcfec0d2750 2023-11-28T20:34:26.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f08a57fa640 1 -- 172.21.15.45:0/1570715977 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f08a80ac8a0 con 0x7f08b00cf110 2023-11-28T20:34:26.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7fcfe3fff640 1 -- 172.21.15.45:0/4248218079 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd40447a0 con 0x7fcfec0d2750 2023-11-28T20:34:26.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7f08a57fa640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7f089c074400 0x7f089c076840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7f47419b9640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f472006af10 0x7f472006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7f47429bb640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f4720074550 0x7f4720076990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec15dc90 con 0x7fcfec0d2750 2023-11-28T20:34:26.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec06d040 con 0x7fcfec0d2750 2023-11-28T20:34:26.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7f08a57fa640 1 -- 172.21.15.45:0/1570715977 --> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f089c076f10 con 0x7f089c074400 2023-11-28T20:34:26.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7f08a57fa640 1 -- 172.21.15.45:0/1570715977 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08a80accf0 con 0x7f08b00cf110 2023-11-28T20:34:26.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7f08b4bc9640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f089c06ae40 0x7f089c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f08a7fff640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7f089c074400 0x7f089c076840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f4732ffd640 1 -- 172.21.15.45:0/2596927900 <== osd.3 v2:172.21.15.45:6808/1024370250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4720077060 con 0x7f4720074550 2023-11-28T20:34:26.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.910+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcfec15db10 con 0x7fcfec0d2750 2023-11-28T20:34:26.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f1f75ffb640 1 -- 172.21.15.45:0/3858156445 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7000c940 con 0x7f1f800cc3a0 2023-11-28T20:34:26.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f1f75ffb640 1 -- 172.21.15.45:0/3858156445 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7000caa0 con 0x7f1f800cc3a0 2023-11-28T20:34:26.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f1f75ffb640 1 -- 172.21.15.45:0/3858156445 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7000ce10 con 0x7f1f800cc3a0 2023-11-28T20:34:26.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f800afd80 con 0x7f1f800cc3a0 2023-11-28T20:34:26.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f800b0340 con 0x7f1f800cc3a0 2023-11-28T20:34:26.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3015eea0 con 0x7f5c300d2750 2023-11-28T20:34:26.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.907+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3015e8e0 con 0x7f5c3015d730 2023-11-28T20:34:26.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7f5c35554640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 0x7f5c30159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.908+0000 7f5c2ffff640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c300d2750 0x7f5c3015de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f5c35554640 1 -- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c300d2750 msgr2=0x7f5c3015de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f5c35554640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c300d2750 0x7f5c3015de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.909+0000 7f5c35554640 1 -- 172.21.15.45:0/2252407040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c30071060 msgr2=0x7f5c300ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.910+0000 7f5c35554640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c30071060 0x7f5c300ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.910+0000 7f5c35554640 1 -- 172.21.15.45:0/2252407040 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c20009400 con 0x7f5c3015d730 2023-11-28T20:34:26.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.910+0000 7f08a7fff640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7f089c074400 0x7f089c076840 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.910+0000 7f08b4bc9640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f089c06ae40 0x7f089c06d300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f08b00cc870 tx=0x7f08ac009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.910+0000 7f5c34d53640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c30071060 0x7f5c300ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:26.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.911+0000 7f5c35554640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 0x7f5c30159240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009d90 tx=0x7f5c2800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.911+0000 7f47419b9640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f472006af10 0x7f472006d3d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f472c000f60 tx=0x7f472c0079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.911+0000 7f5c2d7fa640 1 -- 172.21.15.45:0/2252407040 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2800cd20 con 0x7f5c3015d730 2023-11-28T20:34:26.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.911+0000 7f5716ffd640 1 -- 172.21.15.45:0/3391967962 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f57200147e0 con 0x7f5728071060 2023-11-28T20:34:26.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.912+0000 7f5c2d7fa640 1 -- 172.21.15.45:0/2252407040 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2800f040 con 0x7f5c3015d730 2023-11-28T20:34:26.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.912+0000 7f08a57fa640 1 -- 172.21.15.45:0/1570715977 <== osd.2 v2:172.21.15.45:6824/539709697 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f089c076f10 con 0x7f089c074400 2023-11-28T20:34:26.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.912+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3015dc90 con 0x7f5c3015d730 2023-11-28T20:34:26.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.912+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3015f860 con 0x7f5c3015d730 2023-11-28T20:34:26.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.912+0000 7f5c2d7fa640 1 -- 172.21.15.45:0/2252407040 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28003bb0 con 0x7f5c3015d730 2023-11-28T20:34:26.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.912+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1f8006dbb0 con 0x7f1f800cc3a0 2023-11-28T20:34:26.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.913+0000 7f5716ffd640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f570806af10 0x7f570806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.913+0000 7f5716ffd640 1 -- 172.21.15.45:0/3391967962 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f572008ab40 con 0x7f5728071060 2023-11-28T20:34:26.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.913+0000 7f5716ffd640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f5708074550 0x7f5708076990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.913+0000 7f5716ffd640 1 -- 172.21.15.45:0/3391967962 --> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5708077060 con 0x7f5708074550 2023-11-28T20:34:26.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.913+0000 7f5716ffd640 1 -- 172.21.15.45:0/3391967962 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f572008aef0 con 0x7f5728071060 2023-11-28T20:34:26.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.913+0000 7fcfe3fff640 1 -- 172.21.15.45:0/4248218079 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcfd404d450 con 0x7fcfec0d2750 2023-11-28T20:34:26.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.914+0000 7f5726d9d640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f5708074550 0x7f5708076990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.913+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5c10000fc0 con 0x7f5c3015d730 2023-11-28T20:34:26.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.915+0000 7f5725d9b640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f570806af10 0x7f570806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.915+0000 7fcfe3fff640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcfcc06af40 0x7fcfcc06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.915+0000 7f5726d9d640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f5708074550 0x7f5708076990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.915+0000 7fcfe3fff640 1 -- 172.21.15.45:0/4248218079 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fcfd40ac330 con 0x7fcfec0d2750 2023-11-28T20:34:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.916+0000 7fcfe3fff640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fcfcc074580 0x7fcfcc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.916+0000 7fcfe3fff640 1 -- 172.21.15.45:0/4248218079 --> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfcc077090 con 0x7fcfcc074580 2023-11-28T20:34:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.916+0000 7fcfe3fff640 1 -- 172.21.15.45:0/4248218079 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfd40ac6e0 con 0x7fcfec0d2750 2023-11-28T20:34:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.916+0000 7f5c2d7fa640 1 -- 172.21.15.45:0/2252407040 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5c28003d10 con 0x7f5c3015d730 2023-11-28T20:34:26.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.916+0000 7f1f75ffb640 1 -- 172.21.15.45:0/3858156445 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1f70013470 con 0x7f1f800cc3a0 2023-11-28T20:34:26.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.916+0000 7f5725d9b640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f570806af10 0x7f570806d3d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f571c036000 tx=0x7f571c03c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.917+0000 7f1f75ffb640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1f6406ae40 0x7f1f6406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f1f75ffb640 1 -- 172.21.15.45:0/3858156445 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1f70089a00 con 0x7f1f800cc3a0 2023-11-28T20:34:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7fcfeb7fe640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcfcc06af40 0x7fcfcc06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f1f75ffb640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f1f64074480 0x7f1f640768c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f1f75ffb640 1 -- 172.21.15.45:0/3858156445 --> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f64076f90 con 0x7f1f64074480 2023-11-28T20:34:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f1f75ffb640 1 -- 172.21.15.45:0/3858156445 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f7008e2a0 con 0x7f1f800cc3a0 2023-11-28T20:34:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f5716ffd640 1 -- 172.21.15.45:0/3391967962 <== osd.6 v2:172.21.15.190:6820/2089844775 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5708077060 con 0x7f5708074550 2023-11-28T20:34:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f1f848ec640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1f6406ae40 0x7f1f6406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7fcfebfff640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fcfcc074580 0x7fcfcc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f5c2d7fa640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5c1c06af10 0x7f5c1c06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f5c2d7fa640 1 -- 172.21.15.45:0/2252407040 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5c2808abc0 con 0x7f5c3015d730 2023-11-28T20:34:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f5c2d7fa640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f5c1c074550 0x7f5c1c076990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f5c2d7fa640 1 -- 172.21.15.45:0/2252407040 --> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c1c077060 con 0x7f5c1c074550 2023-11-28T20:34:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.918+0000 7f5c2d7fa640 1 -- 172.21.15.45:0/2252407040 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c2808af70 con 0x7f5c3015d730 2023-11-28T20:34:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.919+0000 7fcfeb7fe640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcfcc06af40 0x7fcfcc06d400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc011340 tx=0x7fcfdc00b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.919+0000 7f5c34d53640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5c1c06af10 0x7f5c1c06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.920+0000 7f5c2ffff640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f5c1c074550 0x7f5c1c076990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.920+0000 7f1f848ec640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1f6406ae40 0x7f1f6406d300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1f8006c840 tx=0x7f1f7c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.920+0000 7fcfebfff640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fcfcc074580 0x7fcfcc0769c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.920+0000 7f1f858ee640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f1f64074480 0x7f1f640768c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.921+0000 7f5c34d53640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5c1c06af10 0x7f5c1c06d3d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5c300d0640 tx=0x7f5c24008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.921+0000 7f1f858ee640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f1f64074480 0x7f1f640768c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.922+0000 7f5c2ffff640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f5c1c074550 0x7f5c1c076990 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.923+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 --> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcd84007080 con 0x7fcd6c0744b0 2023-11-28T20:34:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.923+0000 7fcfe3fff640 1 -- 172.21.15.45:0/4248218079 <== osd.4 v2:172.21.15.190:6800/1794504124 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fcfcc077090 con 0x7fcfcc074580 2023-11-28T20:34:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.924+0000 7fcd7affd640 1 -- 172.21.15.45:0/1380031652 <== osd.1 v2:172.21.15.45:6800/901684499 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcd6c076fc0 con 0x7fcd6c0744b0 2023-11-28T20:34:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.924+0000 7f1f75ffb640 1 -- 172.21.15.45:0/3858156445 <== osd.5 v2:172.21.15.190:6808/3522971559 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1f64076f90 con 0x7f1f64074480 2023-11-28T20:34:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.924+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7fcd6c0744b0 msgr2=0x7fcd6c0768f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.924+0000 7fcd8b59e640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7fcd6c0744b0 0x7fcd6c0768f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.924+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd6c06ae70 msgr2=0x7fcd6c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.924+0000 7fcd8b59e640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd6c06ae70 0x7fcd6c06d330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcd80036000 tx=0x7fcd8003c040 comp rx=0 tx=0).stop 2023-11-28T20:34:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.924+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd840387d0 msgr2=0x7fcd84185470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.924+0000 7fcd8b59e640 1 --2- 172.21.15.45:0/1380031652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd840387d0 0x7fcd84185470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00d860 tx=0x7fcd7c00dd20 comp rx=0 tx=0).stop 2023-11-28T20:34:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.925+0000 7fcd8ad9d640 1 -- 172.21.15.45:0/1380031652 reap_dead start 2023-11-28T20:34:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.925+0000 7f5c2d7fa640 1 -- 172.21.15.45:0/2252407040 <== osd.7 v2:172.21.15.190:6816/3686540929 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5c1c077060 con 0x7f5c1c074550 2023-11-28T20:34:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.926+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 shutdown_connections 2023-11-28T20:34:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.926+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 >> 172.21.15.45:0/1380031652 conn(0x7fcd84009f80 msgr2=0x7fcd84038e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.926+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 shutdown_connections 2023-11-28T20:34:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.926+0000 7fcd8b59e640 1 -- 172.21.15.45:0/1380031652 wait complete. 2023-11-28T20:34:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.929+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 --> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffa70006600 con 0x7ffa5c074550 2023-11-28T20:34:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.930+0000 7ffa6effd640 1 -- 172.21.15.45:0/2184391730 <== osd.0 v2:172.21.15.45:6816/2634303336 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ffa7804f730 con 0x7ffa5c074550 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.931+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ffa5c074550 msgr2=0x7ffa5c076990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.931+0000 7ffa80543640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ffa5c074550 0x7ffa5c076990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.932+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa5c06af10 msgr2=0x7ffa5c06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.932+0000 7ffa80543640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffa5c06af10 0x7ffa5c06d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa74036000 tx=0x7ffa74008030 comp rx=0 tx=0).stop 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.932+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa7003c060 msgr2=0x7ffa700e4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.932+0000 7ffa80543640 1 --2- 172.21.15.45:0/2184391730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa7003c060 0x7ffa700e4650 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800e9b0 tx=0x7ffa6800ee70 comp rx=0 tx=0).stop 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.932+0000 7ffa7eae0640 1 -- 172.21.15.45:0/2184391730 reap_dead start 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.932+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 shutdown_connections 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.932+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 >> 172.21.15.45:0/2184391730 conn(0x7ffa70058ea0 msgr2=0x7ffa7003dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.932+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 shutdown_connections 2023-11-28T20:34:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.933+0000 7ffa80543640 1 -- 172.21.15.45:0/2184391730 wait complete. 2023-11-28T20:34:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.936+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 --> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4724002cb0 con 0x7f4720074550 2023-11-28T20:34:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.937+0000 7f4732ffd640 1 -- 172.21.15.45:0/2596927900 <== osd.3 v2:172.21.15.45:6808/1024370250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4724002cb0 con 0x7f4720074550 2023-11-28T20:34:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.937+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f4720074550 msgr2=0x7f4720076990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.937+0000 7f474441e640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f4720074550 0x7f4720076990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.938+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f472006af10 msgr2=0x7f472006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.938+0000 7f474441e640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f472006af10 0x7f472006d3d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f472c000f60 tx=0x7f472c0079b0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.938+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f473c0cd600 msgr2=0x7f473c1646a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.938+0000 7f474441e640 1 --2- 172.21.15.45:0/2596927900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f473c0cd600 0x7f473c1646a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4738034040 tx=0x7f47380046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.938+0000 7f47429bb640 1 -- 172.21.15.45:0/2596927900 reap_dead start 2023-11-28T20:34:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.938+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 shutdown_connections 2023-11-28T20:34:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.938+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 >> 172.21.15.45:0/2596927900 conn(0x7f473c06a500 msgr2=0x7f473c161090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.940+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 shutdown_connections 2023-11-28T20:34:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.940+0000 7f474441e640 1 -- 172.21.15.45:0/2596927900 wait complete. 2023-11-28T20:34:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.942+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 --> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0890002cb0 con 0x7f089c074400 2023-11-28T20:34:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.943+0000 7f08a57fa640 1 -- 172.21.15.45:0/1570715977 <== osd.2 v2:172.21.15.45:6824/539709697 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0890002cb0 con 0x7f089c074400 2023-11-28T20:34:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.944+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7f089c074400 msgr2=0x7f089c076840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.944+0000 7f08b6e2d640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7f089c074400 0x7f089c076840 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.945+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f089c06ae40 msgr2=0x7f089c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.945+0000 7f08b6e2d640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f089c06ae40 0x7f089c06d300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f08b00cc870 tx=0x7f08ac009040 comp rx=0 tx=0).stop 2023-11-28T20:34:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.945+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08b00cf110 msgr2=0x7f08b01376e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.945+0000 7f08b6e2d640 1 --2- 172.21.15.45:0/1570715977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08b00cf110 0x7f08b01376e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f08a8032000 tx=0x7f08a8004a40 comp rx=0 tx=0).stop 2023-11-28T20:34:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.946+0000 7f08b53ca640 1 -- 172.21.15.45:0/1570715977 reap_dead start 2023-11-28T20:34:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.946+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 shutdown_connections 2023-11-28T20:34:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.946+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 >> 172.21.15.45:0/1570715977 conn(0x7f08b006a6e0 msgr2=0x7f08b00cb580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.946+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 shutdown_connections 2023-11-28T20:34:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.946+0000 7f08b6e2d640 1 -- 172.21.15.45:0/1570715977 wait complete. 2023-11-28T20:34:26.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.952+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 --> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1f8006d8d0 con 0x7f1f64074480 2023-11-28T20:34:26.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.954+0000 7f1f75ffb640 1 -- 172.21.15.45:0/3858156445 <== osd.5 v2:172.21.15.190:6808/3522971559 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1f7802ca50 con 0x7f1f64074480 2023-11-28T20:34:26.975 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:26.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T20:34:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.953+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 --> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcfec0d1cf0 con 0x7fcfcc074580 2023-11-28T20:34:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.954+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f1f64074480 msgr2=0x7f1f640768c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.954+0000 7f1f87351640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f1f64074480 0x7f1f640768c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 --> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f570c002cb0 con 0x7f5708074550 2023-11-28T20:34:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.954+0000 7fcfe3fff640 1 -- 172.21.15.45:0/4248218079 <== osd.4 v2:172.21.15.190:6800/1794504124 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcfcc077090 con 0x7fcfcc074580 2023-11-28T20:34:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fcfcc074580 msgr2=0x7fcfcc0769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7fcff1b0f640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fcfcc074580 0x7fcfcc0769c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1f6406ae40 msgr2=0x7f1f6406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7f1f87351640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1f6406ae40 0x7f1f6406d300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1f8006c840 tx=0x7f1f7c009040 comp rx=0 tx=0).stop 2023-11-28T20:34:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cc3a0 msgr2=0x7f1f800af5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7f1f87351640 1 --2- 172.21.15.45:0/3858156445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f800cc3a0 0x7f1f800af5f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009fd0 tx=0x7f1f7000ed40 comp rx=0 tx=0).stop 2023-11-28T20:34:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7f1f858ee640 1 -- 172.21.15.45:0/3858156445 reap_dead start 2023-11-28T20:34:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcfcc06af40 msgr2=0x7fcfcc06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7fcff1b0f640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcfcc06af40 0x7fcfcc06d400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc011340 tx=0x7fcfdc00b040 comp rx=0 tx=0).stop 2023-11-28T20:34:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 msgr2=0x7fcfec203e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7fcff1b0f640 1 --2- 172.21.15.45:0/4248218079 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfec0d2750 0x7fcfec203e90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40098c0 tx=0x7fcfd40046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 shutdown_connections 2023-11-28T20:34:26.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 >> 172.21.15.45:0/3858156445 conn(0x7f1f8006a6e0 msgr2=0x7f1f80071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.955+0000 7fcfebfff640 1 -- 172.21.15.45:0/4248218079 reap_dead start 2023-11-28T20:34:26.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.956+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 shutdown_connections 2023-11-28T20:34:26.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.956+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 shutdown_connections 2023-11-28T20:34:26.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.956+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 >> 172.21.15.45:0/4248218079 conn(0x7fcfec0cfc70 msgr2=0x7fcfec1603f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.956+0000 7f1f87351640 1 -- 172.21.15.45:0/3858156445 wait complete. 2023-11-28T20:34:26.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.956+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 shutdown_connections 2023-11-28T20:34:26.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.956+0000 7fcff1b0f640 1 -- 172.21.15.45:0/4248218079 wait complete. 2023-11-28T20:34:26.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.956+0000 7f5716ffd640 1 -- 172.21.15.45:0/3391967962 <== osd.6 v2:172.21.15.190:6820/2089844775 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f570c002cb0 con 0x7f5708074550 2023-11-28T20:34:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.957+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f5708074550 msgr2=0x7f5708076990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.957+0000 7f572cbb6640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f5708074550 0x7f5708076990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.958+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f570806af10 msgr2=0x7f570806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.958+0000 7f572cbb6640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f570806af10 0x7f570806d3d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f571c036000 tx=0x7f571c03c040 comp rx=0 tx=0).stop 2023-11-28T20:34:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.958+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5728071060 msgr2=0x7f572814f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.958+0000 7f572cbb6640 1 --2- 172.21.15.45:0/3391967962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5728071060 0x7f572814f6e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5720009d10 tx=0x7f572000c490 comp rx=0 tx=0).stop 2023-11-28T20:34:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.958+0000 7f5726d9d640 1 -- 172.21.15.45:0/3391967962 reap_dead start 2023-11-28T20:34:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.958+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 shutdown_connections 2023-11-28T20:34:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.958+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 >> 172.21.15.45:0/3391967962 conn(0x7f57280cfc70 msgr2=0x7f572806a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.958+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 shutdown_connections 2023-11-28T20:34:26.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.958+0000 7f572cbb6640 1 -- 172.21.15.45:0/3391967962 wait complete. 2023-11-28T20:34:26.983 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:26.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T20:34:26.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.961+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 --> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5c10002cb0 con 0x7f5c1c074550 2023-11-28T20:34:26.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.962+0000 7f5c2d7fa640 1 -- 172.21.15.45:0/2252407040 <== osd.7 v2:172.21.15.190:6816/3686540929 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5c10002cb0 con 0x7f5c1c074550 2023-11-28T20:34:26.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.963+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f5c1c074550 msgr2=0x7f5c1c076990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.963+0000 7f5c36fb7640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f5c1c074550 0x7f5c1c076990 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.963+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5c1c06af10 msgr2=0x7f5c1c06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.963+0000 7f5c36fb7640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5c1c06af10 0x7f5c1c06d3d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5c300d0640 tx=0x7f5c24008040 comp rx=0 tx=0).stop 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.963+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 msgr2=0x7f5c30159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.963+0000 7f5c36fb7640 1 --2- 172.21.15.45:0/2252407040 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c3015d730 0x7f5c30159240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009d90 tx=0x7f5c2800c490 comp rx=0 tx=0).stop 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.963+0000 7f5c35554640 1 -- 172.21.15.45:0/2252407040 reap_dead start 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.963+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 shutdown_connections 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.963+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 >> 172.21.15.45:0/2252407040 conn(0x7f5c300cfc70 msgr2=0x7f5c300b8140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.964+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 shutdown_connections 2023-11-28T20:34:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:26.964+0000 7f5c36fb7640 1 -- 172.21.15.45:0/2252407040 wait complete. 2023-11-28T20:34:26.987 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:26.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T20:34:26.988 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:26.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T20:34:26.989 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:26.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-28T20:34:26.991 INFO:teuthology.orchestra.run.smithi045.stdout:55834574858 2023-11-28T20:34:26.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-28T20:34:26.991 INFO:teuthology.orchestra.run.smithi045.stdout:55834574858 2023-11-28T20:34:26.992 INFO:teuthology.orchestra.run.smithi045.stdout:55834574858 2023-11-28T20:34:26.992 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-28T20:34:26.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-28T20:34:27.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.329+0000 7f3da0c2b640 1 Processor -- start 2023-11-28T20:34:27.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.330+0000 7f3da0c2b640 1 -- start start 2023-11-28T20:34:27.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.332+0000 7f3da0c2b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c0b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.332+0000 7f3da0c2b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3d9c070880 0x7f3d9c165120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.333+0000 7f3da0c2b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0acd90 con 0x7f3d9c0b3970 2023-11-28T20:34:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.333+0000 7f3da0c2b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c0ac780 con 0x7f3d9c070880 2023-11-28T20:34:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.333+0000 7f3da0c2b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0b4010 con 0x7f3d9c15bcf0 2023-11-28T20:34:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.333+0000 7f3d9a59c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c0b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.333+0000 7f3d9a59c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c0b3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:50968/0 (socket says 172.21.15.45:50968) 2023-11-28T20:34:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.333+0000 7f3d9a59c640 1 -- 172.21.15.45:0/41410633 learned_addr learned my addr 172.21.15.45:0/41410633 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.334+0000 7f3d9ad9d640 1 --2- 172.21.15.45:0/41410633 >> v2:172.21.15.190:3300/0 conn(0x7f3d9c070880 0x7f3d9c165120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.334+0000 7f3d9a59c640 1 -- 172.21.15.45:0/41410633 >> v2:172.21.15.190:3300/0 conn(0x7f3d9c070880 msgr2=0x7f3d9c165120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.334+0000 7f3d9a59c640 1 --2- 172.21.15.45:0/41410633 >> v2:172.21.15.190:3300/0 conn(0x7f3d9c070880 0x7f3d9c165120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.334+0000 7f3d9a59c640 1 -- 172.21.15.45:0/41410633 >> v1:172.21.15.45:6789/0 conn(0x7f3d9c0b3970 legacy=0x7f3d9c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.334+0000 7f3d9a59c640 1 -- 172.21.15.45:0/41410633 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c165cf0 con 0x7f3d9c15bcf0 2023-11-28T20:34:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.336+0000 7f3d9a59c640 1 --2- 172.21.15.45:0/41410633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c0b3430 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d84009870 tx=0x7f3d8402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad7f78e961bb2f27 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.337+0000 7f3d99d9b640 1 -- 172.21.15.45:0/41410633 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d84031640 con 0x7f3d9c15bcf0 2023-11-28T20:34:27.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.337+0000 7f3d99d9b640 1 -- 172.21.15.45:0/41410633 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d840317a0 con 0x7f3d9c15bcf0 2023-11-28T20:34:27.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.337+0000 7f3d99d9b640 1 -- 172.21.15.45:0/41410633 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d84031b30 con 0x7f3d9c15bcf0 2023-11-28T20:34:27.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7f3da0c2b640 1 -- 172.21.15.45:0/41410633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 msgr2=0x7f3d9c0b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7f3da0c2b640 1 --2- 172.21.15.45:0/41410633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c0b3430 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d84009870 tx=0x7f3d8402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7ff86eb0c640 1 Processor -- start 2023-11-28T20:34:27.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7f3da0c2b640 1 -- 172.21.15.45:0/41410633 shutdown_connections 2023-11-28T20:34:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7f3da0c2b640 1 --2- 172.21.15.45:0/41410633 >> v2:172.21.15.190:3300/0 conn(0x7f3d9c070880 0x7f3d9c165120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7f3da0c2b640 1 --2- 172.21.15.45:0/41410633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c0b3430 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7f3da0c2b640 1 -- 172.21.15.45:0/41410633 >> 172.21.15.45:0/41410633 conn(0x7f3d9c0b08b0 msgr2=0x7f3d9c0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7ff86eb0c640 1 -- start start 2023-11-28T20:34:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7f3da0c2b640 1 -- 172.21.15.45:0/41410633 shutdown_connections 2023-11-28T20:34:27.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.338+0000 7f3da0c2b640 1 -- 172.21.15.45:0/41410633 wait complete. 2023-11-28T20:34:27.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.341+0000 7ff86eb0c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff86815d210 0x7ff8680b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.341+0000 7f3da0c2b640 1 Processor -- start 2023-11-28T20:34:27.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.341+0000 7f3da0c2b640 1 -- start start 2023-11-28T20:34:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7ff86eb0c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8680d11b0 con 0x7ff8680cddd0 2023-11-28T20:34:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7ff86eb0c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8680d1470 con 0x7ff8680b2620 2023-11-28T20:34:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7ff86eb0c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8680b90c0 con 0x7ff86815d210 2023-11-28T20:34:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7f3da0c2b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9c0b3970 0x7f3d9c148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7f3da0c2b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7f3da0c2b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d9c14e710 0x7f3d9c099f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7f3da0c2b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c146350 con 0x7f3d9c0b3970 2023-11-28T20:34:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7f3da0c2b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c1490e0 con 0x7f3d9c14e710 2023-11-28T20:34:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7f3da0c2b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c09bae0 con 0x7f3d9c15bcf0 2023-11-28T20:34:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.343+0000 7f3d9a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9c0b3970 0x7f3d9c148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.343+0000 7f3d9a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9c0b3970 0x7f3d9c148ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58888/0 (socket says 172.21.15.45:58888) 2023-11-28T20:34:27.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7ff86e30b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff8680cddd0 0x7ff8680ce1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51846/0 (socket says 172.21.15.45:51846) 2023-11-28T20:34:27.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7ff86e30b640 1 -- 172.21.15.45:0/2378767850 learned_addr learned my addr 172.21.15.45:0/2378767850 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:27.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.342+0000 7ff86db0a640 1 --2- 172.21.15.45:0/2378767850 >> v2:172.21.15.45:3300/0 conn(0x7ff86815d210 0x7ff8680b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.343+0000 7ff86db0a640 1 -- 172.21.15.45:0/2378767850 >> v1:172.21.15.190:6789/0 conn(0x7ff8680b2620 legacy=0x7ff8680b89b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T20:34:27.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.343+0000 7ff86db0a640 1 -- 172.21.15.45:0/2378767850 >> v1:172.21.15.45:6789/0 conn(0x7ff8680cddd0 legacy=0x7ff8680ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:27.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.343+0000 7ff86db0a640 1 -- 172.21.15.45:0/2378767850 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8680d1750 con 0x7ff86815d210 2023-11-28T20:34:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.343+0000 7f3d9a59c640 1 -- 172.21.15.45:0/4058351912 learned_addr learned my addr 172.21.15.45:0/4058351912 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.343+0000 7f3d9ad9d640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d9c14e710 0x7f3d9c099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.343+0000 7f3d91d9b640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.344+0000 7f3d9a59c640 1 -- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 msgr2=0x7f3d9c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.344+0000 7f3d9a59c640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.344+0000 7f3d9a59c640 1 -- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d9c14e710 msgr2=0x7f3d9c099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.344+0000 7f3d9a59c640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d9c14e710 0x7f3d9c099f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.344+0000 7f3d9a59c640 1 -- 172.21.15.45:0/4058351912 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d840094d0 con 0x7f3d9c0b3970 2023-11-28T20:34:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.345+0000 7ff86db0a640 1 --2- 172.21.15.45:0/2378767850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86815d210 0x7ff8680b61c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0097a0 tx=0x7ff85c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6031d7a2cfc22763 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.346+0000 7f3d9a59c640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9c0b3970 0x7f3d9c148ba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3d840090b0 tx=0x7f3d840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.346+0000 7ff86cb08640 1 -- 172.21.15.45:0/2378767850 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c031830 con 0x7ff86815d210 2023-11-28T20:34:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.346+0000 7f3d937fe640 1 -- 172.21.15.45:0/4058351912 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8403ab10 con 0x7f3d9c0b3970 2023-11-28T20:34:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.346+0000 7ff86cb08640 1 -- 172.21.15.45:0/2378767850 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85c031990 con 0x7ff86815d210 2023-11-28T20:34:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.346+0000 7ff86cb08640 1 -- 172.21.15.45:0/2378767850 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c031d20 con 0x7ff86815d210 2023-11-28T20:34:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.347+0000 7f3d937fe640 1 -- 172.21.15.45:0/4058351912 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8403ac90 con 0x7f3d9c0b3970 2023-11-28T20:34:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.347+0000 7f3d937fe640 1 -- 172.21.15.45:0/4058351912 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d840443f0 con 0x7f3d9c0b3970 2023-11-28T20:34:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.347+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2378767850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86815d210 msgr2=0x7ff8680b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.347+0000 7ff86eb0c640 1 --2- 172.21.15.45:0/2378767850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86815d210 0x7ff8680b61c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0097a0 tx=0x7ff85c02b8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.348+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2378767850 shutdown_connections 2023-11-28T20:34:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.348+0000 7ff86eb0c640 1 --2- 172.21.15.45:0/2378767850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86815d210 0x7ff8680b61c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.348+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2378767850 >> 172.21.15.45:0/2378767850 conn(0x7ff868070f40 msgr2=0x7ff868071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.348+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2378767850 shutdown_connections 2023-11-28T20:34:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.348+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2378767850 wait complete. 2023-11-28T20:34:27.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.348+0000 7f0600f47640 1 Processor -- start 2023-11-28T20:34:27.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.347+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c09bd80 con 0x7f3d9c0b3970 2023-11-28T20:34:27.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.347+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c09c340 con 0x7f3d9c0b3970 2023-11-28T20:34:27.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.348+0000 7f0600f47640 1 -- start start 2023-11-28T20:34:27.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.350+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c0733d0 con 0x7f3d9c0b3970 2023-11-28T20:34:27.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.351+0000 7ff86eb0c640 1 Processor -- start 2023-11-28T20:34:27.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.351+0000 7ff86eb0c640 1 -- start start 2023-11-28T20:34:27.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.351+0000 7f0600f47640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f05fc0cf030 0x7f05fc0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.351+0000 7f0600f47640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0cf9d0 con 0x7f05fc1590e0 2023-11-28T20:34:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.351+0000 7f0600f47640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0716d0 con 0x7f05fc0ac050 2023-11-28T20:34:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.351+0000 7f0600f47640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc070880 con 0x7f05fc0cf030 2023-11-28T20:34:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.352+0000 7fb903b97640 1 Processor -- start 2023-11-28T20:34:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.352+0000 7f05faffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f05fc0cf030 0x7f05fc0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.352+0000 7f05faffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f05fc0cf030 0x7f05fc0cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49558/0 (socket says 172.21.15.45:49558) 2023-11-28T20:34:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.352+0000 7f05faffd640 1 -- 172.21.15.45:0/1141330262 learned_addr learned my addr 172.21.15.45:0/1141330262 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.352+0000 7ff86eb0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8680b2620 0x7ff8681558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7ff86eb0c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8680cddd0 0x7ff8680c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7ff86eb0c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff86815d210 0x7ff86815dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7ff86eb0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8681569a0 con 0x7ff8680b2620 2023-11-28T20:34:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7ff86eb0c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86815ffc0 con 0x7ff8680cddd0 2023-11-28T20:34:27.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7ff86eb0c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8681602a0 con 0x7ff86815d210 2023-11-28T20:34:27.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7fb903b97640 1 -- start start 2023-11-28T20:34:27.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7f05faffd640 1 -- 172.21.15.45:0/1141330262 >> v1:172.21.15.190:6789/0 conn(0x7f05fc0ac050 legacy=0x7f05fc0ac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7f05faffd640 1 -- 172.21.15.45:0/1141330262 >> v1:172.21.15.45:6789/0 conn(0x7f05fc1590e0 legacy=0x7f05fc070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:27.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7f3d937fe640 1 -- 172.21.15.45:0/4058351912 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3d8402f050 con 0x7f3d9c0b3970 2023-11-28T20:34:27.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.354+0000 7f05faffd640 1 -- 172.21.15.45:0/1141330262 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc070b50 con 0x7f05fc0cf030 2023-11-28T20:34:27.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7ff86db0a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8680b2620 0x7ff8681558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7ff86db0a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8680b2620 0x7ff8681558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58896/0 (socket says 172.21.15.45:58896) 2023-11-28T20:34:27.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7ff86db0a640 1 -- 172.21.15.45:0/2561528245 learned_addr learned my addr 172.21.15.45:0/2561528245 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.353+0000 7ff86e30b640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff86815d210 0x7ff86815dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.354+0000 7ff86db0a640 1 -- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff86815d210 msgr2=0x7ff86815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.354+0000 7ff86db0a640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff86815d210 0x7ff86815dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.354+0000 7ff86db0a640 1 -- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8680cddd0 msgr2=0x7ff8680c7590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T20:34:27.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.354+0000 7ff86db0a640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8680cddd0 0x7ff8680c7590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.354+0000 7ff86db0a640 1 -- 172.21.15.45:0/2561528245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c009400 con 0x7ff8680b2620 2023-11-28T20:34:27.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.355+0000 7ff86db0a640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8680b2620 0x7ff8681558e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0097a0 tx=0x7ff85c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.355+0000 7f3d937fe640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3d7c06ae40 0x7f3d7c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.355+0000 7f3d937fe640 1 -- 172.21.15.45:0/4058351912 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3d8407a0e0 con 0x7f3d9c0b3970 2023-11-28T20:34:27.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.355+0000 7fb903b97640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8fc0b4c40 0x7fb8fc0b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.355+0000 7fb903b97640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb8fc0b0c30 0x7fb8fc0b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.355+0000 7fb903b97640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8fc0b1580 con 0x7fb8fc161960 2023-11-28T20:34:27.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.355+0000 7fb903b97640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc071e40 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.355+0000 7fb903b97640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc0711c0 con 0x7fb8fc0b0c30 2023-11-28T20:34:27.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.356+0000 7f3d91d9b640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3d7c06ae40 0x7f3d7c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.356+0000 7fb901132640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8fc0b4c40 0x7fb8fc0b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.356+0000 7ff367fff640 1 Processor -- start 2023-11-28T20:34:27.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.356+0000 7fb901933640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb8fc161960 0x7fb8fc070330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:38400/0 (socket says 172.21.15.45:38400) 2023-11-28T20:34:27.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.356+0000 7fb901933640 1 -- 172.21.15.45:0/28947091 learned_addr learned my addr 172.21.15.45:0/28947091 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:27.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.356+0000 7ff367fff640 1 -- start start 2023-11-28T20:34:27.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.356+0000 7fb902134640 1 --2- 172.21.15.45:0/28947091 >> v2:172.21.15.190:3300/0 conn(0x7fb8fc0b0c30 0x7fb8fc0b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.356+0000 7ff85a7fc640 1 -- 172.21.15.45:0/2561528245 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c03aaa0 con 0x7ff8680b2620 2023-11-28T20:34:27.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.357+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff868160580 con 0x7ff8680b2620 2023-11-28T20:34:27.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.357+0000 7fb900931640 1 -- 172.21.15.45:0/28947091 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672652599 0 0) 0x7fb8fc0b1580 con 0x7fb8fc161960 2023-11-28T20:34:27.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.357+0000 7fb900931640 1 -- 172.21.15.45:0/28947091 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e8003520 con 0x7fb8fc161960 2023-11-28T20:34:27.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.357+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff868155e20 con 0x7ff8680b2620 2023-11-28T20:34:27.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.357+0000 7fb902134640 1 -- 172.21.15.45:0/28947091 >> v2:172.21.15.45:3300/0 conn(0x7fb8fc0b4c40 msgr2=0x7fb8fc0b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:27.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.357+0000 7fb902134640 1 --2- 172.21.15.45:0/28947091 >> v2:172.21.15.45:3300/0 conn(0x7fb8fc0b4c40 0x7fb8fc0b2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.357+0000 7fb902134640 1 -- 172.21.15.45:0/28947091 >> v1:172.21.15.190:6789/0 conn(0x7fb8fc161960 legacy=0x7fb8fc070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.357+0000 7fb902134640 1 -- 172.21.15.45:0/28947091 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc0714a0 con 0x7fb8fc0b0c30 2023-11-28T20:34:27.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.357+0000 7f05faffd640 1 --2- 172.21.15.45:0/1141330262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 0x7f05fc0cf490 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f05f000c970 tx=0x7f05f002dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfe5680f9c93bac7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.358+0000 7f3d91d9b640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3d7c06ae40 0x7f3d7c06d300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c09b800 tx=0x7f3d8c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.358+0000 7ff367fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff36003c070 0x7ff3600e7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.358+0000 7ff367fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff360007b40 con 0x7ff360053c80 2023-11-28T20:34:27.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.358+0000 7ff367fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3600587e0 con 0x7ff3600e3130 2023-11-28T20:34:27.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.358+0000 7ff367fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff360058ac0 con 0x7ff36003c070 2023-11-28T20:34:27.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.359+0000 7f05fa7fc640 1 -- 172.21.15.45:0/1141330262 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f002e9f0 con 0x7f05fc0cf030 2023-11-28T20:34:27.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.359+0000 7f05fa7fc640 1 -- 172.21.15.45:0/1141330262 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f0032070 con 0x7f05fc0cf030 2023-11-28T20:34:27.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.359+0000 7f05fa7fc640 1 -- 172.21.15.45:0/1141330262 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f002ed70 con 0x7f05fc0cf030 2023-11-28T20:34:27.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.359+0000 7ff3677fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff360053c80 0x7ff3600540f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51878/0 (socket says 172.21.15.45:51878) 2023-11-28T20:34:27.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.359+0000 7ff3677fe640 1 -- 172.21.15.45:0/196878659 learned_addr learned my addr 172.21.15.45:0/196878659 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:27.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.359+0000 7ff3667fc640 1 --2- 172.21.15.45:0/196878659 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff36003c070 0x7ff3600e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.359+0000 7ff3667fc640 1 -- 172.21.15.45:0/196878659 >> v1:172.21.15.190:6789/0 conn(0x7ff3600e3130 legacy=0x7ff360058020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T20:34:27.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.359+0000 7ff3667fc640 1 -- 172.21.15.45:0/196878659 >> v1:172.21.15.45:6789/0 conn(0x7ff360053c80 legacy=0x7ff3600540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.359+0000 7ff3667fc640 1 -- 172.21.15.45:0/196878659 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff360006330 con 0x7ff36003c070 2023-11-28T20:34:27.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.360+0000 7f0600f47640 1 -- 172.21.15.45:0/1141330262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 msgr2=0x7f05fc0cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.360+0000 7f0600f47640 1 --2- 172.21.15.45:0/1141330262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 0x7f05fc0cf490 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f05f000c970 tx=0x7f05f002dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.360+0000 7fdab6943640 1 Processor -- start 2023-11-28T20:34:27.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.360+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff868160790 con 0x7ff8680b2620 2023-11-28T20:34:27.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.360+0000 7fb902134640 1 --2- 172.21.15.45:0/28947091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc0b0c30 0x7fb8fc0b1040 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0727c0 tx=0x7fb8f402be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f99c83333e651990 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.360+0000 7f0600f47640 1 -- 172.21.15.45:0/1141330262 shutdown_connections 2023-11-28T20:34:27.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.360+0000 7f0600f47640 1 --2- 172.21.15.45:0/1141330262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 0x7f05fc0cf490 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.360+0000 7f0600f47640 1 -- 172.21.15.45:0/1141330262 >> 172.21.15.45:0/1141330262 conn(0x7f05fc06a500 msgr2=0x7f05fc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.360+0000 7fdab6943640 1 -- start start 2023-11-28T20:34:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.361+0000 7f0600f47640 1 -- 172.21.15.45:0/1141330262 shutdown_connections 2023-11-28T20:34:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.361+0000 7fb900931640 1 -- 172.21.15.45:0/28947091 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f402cbe0 con 0x7fb8fc0b0c30 2023-11-28T20:34:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.361+0000 7fb900931640 1 -- 172.21.15.45:0/28947091 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f402cd40 con 0x7fb8fc0b0c30 2023-11-28T20:34:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.361+0000 7fb900931640 1 -- 172.21.15.45:0/28947091 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4034660 con 0x7fb8fc0b0c30 2023-11-28T20:34:27.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.361+0000 7ff3667fc640 1 --2- 172.21.15.45:0/196878659 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff36003c070 0x7ff3600e7750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff36804f5f0 tx=0x7ff3680a5d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=282c19352da614d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.361+0000 7f0600f47640 1 -- 172.21.15.45:0/1141330262 wait complete. 2023-11-28T20:34:27.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.361+0000 7ff365ffb640 1 -- 172.21.15.45:0/196878659 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3680a6bc0 con 0x7ff36003c070 2023-11-28T20:34:27.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7fb903b97640 1 -- 172.21.15.45:0/28947091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc0b0c30 msgr2=0x7fb8fc0b1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7fb903b97640 1 --2- 172.21.15.45:0/28947091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc0b0c30 0x7fb8fc0b1040 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0727c0 tx=0x7fb8f402be30 comp rx=0 tx=0).stop 2023-11-28T20:34:27.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7ff365ffb640 1 -- 172.21.15.45:0/196878659 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3680a6d20 con 0x7ff36003c070 2023-11-28T20:34:27.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7fb903b97640 1 -- 172.21.15.45:0/28947091 shutdown_connections 2023-11-28T20:34:27.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7fb903b97640 1 --2- 172.21.15.45:0/28947091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc0b0c30 0x7fb8fc0b1040 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7fb903b97640 1 --2- 172.21.15.45:0/28947091 >> v2:172.21.15.45:3300/0 conn(0x7fb8fc0b4c40 0x7fb8fc0b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7fb903b97640 1 -- 172.21.15.45:0/28947091 >> 172.21.15.45:0/28947091 conn(0x7fb8fc06a500 msgr2=0x7fb8fc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7ff365ffb640 1 -- 172.21.15.45:0/196878659 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3680ae660 con 0x7ff36003c070 2023-11-28T20:34:27.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7fdab6943640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab00b6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.362+0000 7fdab6943640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdab006a500 0x7fdab0070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7ff367fff640 1 -- 172.21.15.45:0/196878659 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff36003c070 msgr2=0x7ff3600e7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7ff367fff640 1 --2- 172.21.15.45:0/196878659 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff36003c070 0x7ff3600e7750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff36804f5f0 tx=0x7ff3680a5d50 comp rx=0 tx=0).stop 2023-11-28T20:34:27.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7fdab6943640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdab00b2430 0x7fdab0071060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7fdab6943640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab006aa70 con 0x7fdab006a500 2023-11-28T20:34:27.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7fdab6943640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab006ad50 con 0x7fdab00b2430 2023-11-28T20:34:27.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7fdab6943640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab006b030 con 0x7fdab0160e70 2023-11-28T20:34:27.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7f0600f47640 1 Processor -- start 2023-11-28T20:34:27.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7ff367fff640 1 -- 172.21.15.45:0/196878659 shutdown_connections 2023-11-28T20:34:27.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7ff367fff640 1 --2- 172.21.15.45:0/196878659 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff36003c070 0x7ff3600e7750 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7ff367fff640 1 -- 172.21.15.45:0/196878659 >> 172.21.15.45:0/196878659 conn(0x7ff360058eb0 msgr2=0x7ff360059290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7ff367fff640 1 -- 172.21.15.45:0/196878659 shutdown_connections 2023-11-28T20:34:27.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7fdaaffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab00b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7fdaaffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab00b6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:51006/0 (socket says 172.21.15.45:51006) 2023-11-28T20:34:27.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7fdaaffff640 1 -- 172.21.15.45:0/67177203 learned_addr learned my addr 172.21.15.45:0/67177203 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.364+0000 7ff367fff640 1 -- 172.21.15.45:0/196878659 wait complete. 2023-11-28T20:34:27.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.364+0000 7fdaaf7fe640 1 --2- 172.21.15.45:0/67177203 >> v2:172.21.15.45:3300/0 conn(0x7fdab006a500 0x7fdab0070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.364+0000 7fdab4ee0640 1 --2- 172.21.15.45:0/67177203 >> v2:172.21.15.190:3300/0 conn(0x7fdab00b2430 0x7fdab0071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.363+0000 7f0600f47640 1 -- start start 2023-11-28T20:34:27.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.364+0000 7f0600f47640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05fc0ac050 0x7f05fc156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.364+0000 7f0600f47640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 0x7f05fc0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.364+0000 7f0600f47640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05fc1590e0 0x7f05fc0cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.364+0000 7f0600f47640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc0ae2a0 con 0x7f05fc1590e0 2023-11-28T20:34:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7fb903b97640 1 -- 172.21.15.45:0/28947091 shutdown_connections 2023-11-28T20:34:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7fdaaf7fe640 1 -- 172.21.15.45:0/67177203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 msgr2=0x7fdab00b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7fdaaf7fe640 1 --2- 172.21.15.45:0/67177203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab00b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7fdaaf7fe640 1 -- 172.21.15.45:0/67177203 >> v2:172.21.15.190:3300/0 conn(0x7fdab00b2430 msgr2=0x7fdab0071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:27.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7fdaaf7fe640 1 --2- 172.21.15.45:0/67177203 >> v2:172.21.15.190:3300/0 conn(0x7fdab00b2430 0x7fdab0071060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7f05fb7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05fc0ac050 0x7f05fc156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7f05fb7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05fc0ac050 0x7f05fc156e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:51020/0 (socket says 172.21.15.45:51020) 2023-11-28T20:34:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7f05fb7fe640 1 -- 172.21.15.45:0/3899443233 learned_addr learned my addr 172.21.15.45:0/3899443233 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7f0600f47640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc0ce120 con 0x7f05fc0cf030 2023-11-28T20:34:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7f05faffd640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 0x7f05fc0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.366+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc15c580 con 0x7f05fc0ac050 2023-11-28T20:34:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7fb903b97640 1 -- 172.21.15.45:0/28947091 wait complete. 2023-11-28T20:34:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.365+0000 7fdaaf7fe640 1 -- 172.21.15.45:0/67177203 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab006b310 con 0x7fdab006a500 2023-11-28T20:34:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.366+0000 7fdab4ee0640 1 --2- 172.21.15.45:0/67177203 >> v2:172.21.15.190:3300/0 conn(0x7fdab00b2430 0x7fdab0071060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.366+0000 7fdaaffff640 1 --2- 172.21.15.45:0/67177203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab00b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.366+0000 7f05fb7fe640 1 -- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 msgr2=0x7f05fc0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.366+0000 7f05fb7fe640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 0x7f05fc0cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.366+0000 7f05fb7fe640 1 -- 172.21.15.45:0/3899443233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05fc1590e0 msgr2=0x7f05fc0cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.366+0000 7f05fb7fe640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05fc1590e0 0x7f05fc0cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.367+0000 7f05fb7fe640 1 -- 172.21.15.45:0/3899443233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f000c650 con 0x7f05fc0ac050 2023-11-28T20:34:27.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.367+0000 7ff367fff640 1 Processor -- start 2023-11-28T20:34:27.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.367+0000 7f54c12ad640 1 Processor -- start 2023-11-28T20:34:27.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.367+0000 7fdaaf7fe640 1 --2- 172.21.15.45:0/67177203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab006a500 0x7fdab0070b20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdab00d30a0 tx=0x7fda9802bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe7dc771ed8ac7d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.367+0000 7f05fb7fe640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05fc0ac050 0x7f05fc156e40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00b410 tx=0x7f05ec00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.367+0000 7f05faffd640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 0x7f05fc0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:27.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.367+0000 7ff367fff640 1 -- start start 2023-11-28T20:34:27.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.367+0000 7f54c12ad640 1 -- start start 2023-11-28T20:34:27.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7f05e3fff640 1 -- 172.21.15.45:0/3899443233 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec011830 con 0x7f05fc0ac050 2023-11-28T20:34:27.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7f05e3fff640 1 -- 172.21.15.45:0/3899443233 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec011990 con 0x7f05fc0ac050 2023-11-28T20:34:27.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7f05e3fff640 1 -- 172.21.15.45:0/3899443233 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec011d20 con 0x7f05fc0ac050 2023-11-28T20:34:27.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7f3d937fe640 1 -- 172.21.15.45:0/4058351912 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d84076bb0 con 0x7f3d9c0b3970 2023-11-28T20:34:27.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7ff85a7fc640 1 -- 172.21.15.45:0/2561528245 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85c03ac00 con 0x7ff8680b2620 2023-11-28T20:34:27.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7ff85a7fc640 1 -- 172.21.15.45:0/2561528245 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c03aaa0 con 0x7ff8680b2620 2023-11-28T20:34:27.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7fb903b97640 1 Processor -- start 2023-11-28T20:34:27.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc0cd7f0 con 0x7f05fc0ac050 2023-11-28T20:34:27.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc0cddb0 con 0x7f05fc0ac050 2023-11-28T20:34:27.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.368+0000 7ff367fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36003c070 0x7ff360050cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.369+0000 7ff367fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff360053c80 0x7ff3600d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.369+0000 7ff367fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3600e3130 0x7ff36003d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.369+0000 7ff367fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff360050890 con 0x7ff360053c80 2023-11-28T20:34:27.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.369+0000 7ff367fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36004ff80 con 0x7ff36003c070 2023-11-28T20:34:27.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.369+0000 7ff367fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600d16f0 con 0x7ff3600e3130 2023-11-28T20:34:27.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7ff366ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36003c070 0x7ff360050cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7ff366ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36003c070 0x7ff360050cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49612/0 (socket says 172.21.15.45:49612) 2023-11-28T20:34:27.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7ff366ffd640 1 -- 172.21.15.45:0/3841836367 learned_addr learned my addr 172.21.15.45:0/3841836367 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7f54c12ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc161960 0x7f54bc0b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7f54c12ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f54bc0b0c30 0x7f54bc0b1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7f54c12ad640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc0b15a0 con 0x7f54bc0b4c40 2023-11-28T20:34:27.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7f54c12ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54bc071e40 con 0x7f54bc0b0c30 2023-11-28T20:34:27.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7f54c12ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0711c0 con 0x7f54bc161960 2023-11-28T20:34:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.369+0000 7fb903b97640 1 -- start start 2023-11-28T20:34:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7ff3667fc640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff360053c80 0x7ff3600d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7f9cfa5d7640 1 Processor -- start 2023-11-28T20:34:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7f54bb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f54bc0b0c30 0x7f54bc0b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7f54baffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc161960 0x7f54bc0b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7f54baffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc161960 0x7f54bc0b2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:51040/0 (socket says 172.21.15.45:51040) 2023-11-28T20:34:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7f54baffd640 1 -- 172.21.15.45:0/3294768149 learned_addr learned my addr 172.21.15.45:0/3294768149 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7ff366ffd640 1 -- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3600e3130 msgr2=0x7ff36003d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7f9cfa5d7640 1 -- start start 2023-11-28T20:34:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7ff3677fe640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3600e3130 0x7ff36003d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7ff366ffd640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3600e3130 0x7ff36003d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7ff366ffd640 1 -- 172.21.15.45:0/3841836367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff360053c80 msgr2=0x7ff3600d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7ff366ffd640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff360053c80 0x7ff3600d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7ff366ffd640 1 -- 172.21.15.45:0/3841836367 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3600d2560 con 0x7ff36003c070 2023-11-28T20:34:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7fb903b97640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8fc0b0c30 0x7fb8fc0aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7fb903b97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8fc0b4c40 0x7fb8fc0a8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7ff3667fc640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff360053c80 0x7ff3600d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7ff3677fe640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3600e3130 0x7ff36003d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.371+0000 7f54baffd640 1 -- 172.21.15.45:0/3294768149 >> v2:172.21.15.45:3300/0 conn(0x7f54bc0b0c30 msgr2=0x7f54bc0b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7f54baffd640 1 --2- 172.21.15.45:0/3294768149 >> v2:172.21.15.45:3300/0 conn(0x7f54bc0b0c30 0x7f54bc0b1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7fb903b97640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc161960 0x7fb8fc0a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7fb901933640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8fc0b0c30 0x7fb8fc0aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7fb901132640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8fc0b4c40 0x7fb8fc0a8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7fb901132640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8fc0b4c40 0x7fb8fc0a8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58936/0 (socket says 172.21.15.45:58936) 2023-11-28T20:34:27.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7fb901132640 1 -- 172.21.15.45:0/2670690844 learned_addr learned my addr 172.21.15.45:0/2670690844 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7f54baffd640 1 -- 172.21.15.45:0/3294768149 >> v1:172.21.15.190:6789/0 conn(0x7f54bc0b4c40 legacy=0x7f54bc070330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T20:34:27.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7f54baffd640 1 -- 172.21.15.45:0/3294768149 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc0714a0 con 0x7f54bc161960 2023-11-28T20:34:27.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7fb902134640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc161960 0x7fb8fc0a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7f54bb7fe640 1 --2- 172.21.15.45:0/3294768149 >> v2:172.21.15.45:3300/0 conn(0x7f54bc0b0c30 0x7f54bc0b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0ab4c0 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.370+0000 7ff85a7fc640 1 -- 172.21.15.45:0/2561528245 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff85c04d450 con 0x7ff8680b2620 2023-11-28T20:34:27.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7ff85a7fc640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff84c06aea0 0x7ff84c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7ff85a7fc640 1 -- 172.21.15.45:0/2561528245 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff85c0aba20 con 0x7ff8680b2620 2023-11-28T20:34:27.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc15e5b0 con 0x7fb8fc161960 2023-11-28T20:34:27.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7ff366ffd640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36003c070 0x7ff360050cb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff35800b3e0 tx=0x7ff35800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.372+0000 7ff34ffff640 1 -- 172.21.15.45:0/3841836367 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff358011830 con 0x7ff36003c070 2023-11-28T20:34:27.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7ff34ffff640 1 -- 172.21.15.45:0/3841836367 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3580119b0 con 0x7ff36003c070 2023-11-28T20:34:27.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7ff34ffff640 1 -- 172.21.15.45:0/3841836367 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff358011d60 con 0x7ff36003c070 2023-11-28T20:34:27.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3600501d0 con 0x7ff36003c070 2023-11-28T20:34:27.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3600515b0 con 0x7ff36003c070 2023-11-28T20:34:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7f9cfa5d7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9cf40cf9f0 0x7f9cf40cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7f9cfa5d7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9cf40d0eb0 0x7f9cf40d1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7f9cfa5d7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf40b6f50 con 0x7f9cf41617f0 2023-11-28T20:34:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7f9cfa5d7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf40b83a0 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7f9cfa5d7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf40b71f0 con 0x7f9cf40d0eb0 2023-11-28T20:34:27.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7ff86d309640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff84c06aea0 0x7ff84c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7f9cf37fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9cf40cf9f0 0x7f9cf40cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.373+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc1649d0 con 0x7fb8fc0b0c30 2023-11-28T20:34:27.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.374+0000 7f9cf37fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9cf40cf9f0 0x7f9cf40cfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58952/0 (socket says 172.21.15.45:58952) 2023-11-28T20:34:27.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.374+0000 7f9cf37fe640 1 -- 172.21.15.45:0/3910827765 learned_addr learned my addr 172.21.15.45:0/3910827765 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.375+0000 7fb901132640 1 -- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8fc0b0c30 msgr2=0x7fb8fc0aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.375+0000 7fb901132640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8fc0b0c30 0x7fb8fc0aaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.375+0000 7fb901132640 1 -- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc161960 msgr2=0x7fb8fc0a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.375+0000 7fb901132640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc161960 0x7fb8fc0a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.375+0000 7fb901132640 1 -- 172.21.15.45:0/2670690844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f4009a70 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.375+0000 7fb901132640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8fc0b4c40 0x7fb8fc0a8f30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00c7f0 tx=0x7fb8ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.375+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3600509d0 con 0x7ff36003c070 2023-11-28T20:34:27.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.374+0000 7f9cf37fe640 1 -- 172.21.15.45:0/3910827765 >> v2:172.21.15.190:3300/0 conn(0x7f9cf40d0eb0 msgr2=0x7f9cf40d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:27.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.374+0000 7f9cf37fe640 1 --2- 172.21.15.45:0/3910827765 >> v2:172.21.15.190:3300/0 conn(0x7f9cf40d0eb0 0x7f9cf40d1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.374+0000 7f9cf37fe640 1 -- 172.21.15.45:0/3910827765 >> v1:172.21.15.190:6789/0 conn(0x7f9cf41617f0 legacy=0x7f9cf40b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.375+0000 7ff86d309640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff84c06aea0 0x7ff84c06d360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff8680c7520 tx=0x7ff864006bd0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.374+0000 7f9cf37fe640 1 -- 172.21.15.45:0/3910827765 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf40cf590 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.374+0000 7f54baffd640 1 --2- 172.21.15.45:0/3294768149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc161960 0x7f54bc0b2cd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009e50 tx=0x7f54ac02be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be62e7e92546567a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.376+0000 7f54ba7fc640 1 -- 172.21.15.45:0/3294768149 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac02c9f0 con 0x7f54bc161960 2023-11-28T20:34:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.376+0000 7f54ba7fc640 1 -- 172.21.15.45:0/3294768149 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac02cb50 con 0x7f54bc161960 2023-11-28T20:34:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.376+0000 7f54ba7fc640 1 -- 172.21.15.45:0/3294768149 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0343f0 con 0x7f54bc161960 2023-11-28T20:34:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.374+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc09d0f0 con 0x7f05fc0ac050 2023-11-28T20:34:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.376+0000 7f9cf37fe640 1 --2- 172.21.15.45:0/3910827765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 0x7f9cf40cfe50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9cf41663f0 tx=0x7f9cec02bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d977a4e215f8bee2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.376+0000 7fb901933640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8fc0b0c30 0x7fb8fc0aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:27.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.376+0000 7fb902134640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc161960 0x7fb8fc0a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7f54c12ad640 1 -- 172.21.15.45:0/3294768149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc161960 msgr2=0x7f54bc0b2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7f54c12ad640 1 --2- 172.21.15.45:0/3294768149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc161960 0x7f54bc0b2cd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009e50 tx=0x7f54ac02be50 comp rx=0 tx=0).stop 2023-11-28T20:34:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.376+0000 7f05e3fff640 1 -- 172.21.15.45:0/3899443233 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f05ec01a400 con 0x7f05fc0ac050 2023-11-28T20:34:27.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7f54c12ad640 1 -- 172.21.15.45:0/3294768149 shutdown_connections 2023-11-28T20:34:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7f54c12ad640 1 --2- 172.21.15.45:0/3294768149 >> v2:172.21.15.45:3300/0 conn(0x7f54bc0b0c30 0x7f54bc0b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7f54c12ad640 1 --2- 172.21.15.45:0/3294768149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc161960 0x7f54bc0b2cd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7f54c12ad640 1 -- 172.21.15.45:0/3294768149 >> 172.21.15.45:0/3294768149 conn(0x7f54bc06a500 msgr2=0x7f54bc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7fdaaeffd640 1 -- 172.21.15.45:0/67177203 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9802d680 con 0x7fdab006a500 2023-11-28T20:34:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7f54c12ad640 1 -- 172.21.15.45:0/3294768149 shutdown_connections 2023-11-28T20:34:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7fdaaeffd640 1 -- 172.21.15.45:0/67177203 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9802d7e0 con 0x7fdab006a500 2023-11-28T20:34:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7f9cf2ffd640 1 -- 172.21.15.45:0/3910827765 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec02caa0 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7f9cf2ffd640 1 -- 172.21.15.45:0/3910827765 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cec02cc00 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7f05e3fff640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f05e806af10 0x7f05e806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7fb8f27fc640 1 -- 172.21.15.45:0/2670690844 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec004220 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7fb8f27fc640 1 -- 172.21.15.45:0/2670690844 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec004380 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7fb8f27fc640 1 -- 172.21.15.45:0/2670690844 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec007840 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc0a84a0 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7ff85a7fc640 1 -- 172.21.15.45:0/2561528245 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff85c076850 con 0x7ff8680b2620 2023-11-28T20:34:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7fdab6943640 1 -- 172.21.15.45:0/67177203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab006a500 msgr2=0x7fdab0070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7fdab6943640 1 --2- 172.21.15.45:0/67177203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab006a500 0x7fdab0070b20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdab00d30a0 tx=0x7fda9802bad0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7ff34ffff640 1 -- 172.21.15.45:0/3841836367 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff35801a400 con 0x7ff36003c070 2023-11-28T20:34:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7fdaaeffd640 1 -- 172.21.15.45:0/67177203 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9802d680 con 0x7fdab006a500 2023-11-28T20:34:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.378+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8fc0a8a60 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.379+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/3910827765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 msgr2=0x7f9cf40cfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.379+0000 7f9cfa5d7640 1 --2- 172.21.15.45:0/3910827765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 0x7f9cf40cfe50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9cf41663f0 tx=0x7f9cec02bf00 comp rx=0 tx=0).stop 2023-11-28T20:34:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.377+0000 7f54c12ad640 1 -- 172.21.15.45:0/3294768149 wait complete. 2023-11-28T20:34:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.379+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/3910827765 shutdown_connections 2023-11-28T20:34:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.379+0000 7f9cfa5d7640 1 --2- 172.21.15.45:0/3910827765 >> v2:172.21.15.190:3300/0 conn(0x7f9cf40d0eb0 0x7f9cf40d1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.379+0000 7f9cfa5d7640 1 --2- 172.21.15.45:0/3910827765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 0x7f9cf40cfe50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.379+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/3910827765 >> 172.21.15.45:0/3910827765 conn(0x7f9cf40b4e10 msgr2=0x7f9cf40b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.379+0000 7f05e3fff640 1 -- 172.21.15.45:0/3899443233 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f05ec089990 con 0x7f05fc0ac050 2023-11-28T20:34:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7fdab6943640 1 -- 172.21.15.45:0/67177203 shutdown_connections 2023-11-28T20:34:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7ff34ffff640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff35406af10 0x7ff35406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7fdab6943640 1 --2- 172.21.15.45:0/67177203 >> v2:172.21.15.190:3300/0 conn(0x7fdab00b2430 0x7fdab0071060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7fdab6943640 1 --2- 172.21.15.45:0/67177203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab006a500 0x7fdab0070b20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7fdab6943640 1 --2- 172.21.15.45:0/67177203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab00b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7fdab6943640 1 -- 172.21.15.45:0/67177203 >> 172.21.15.45:0/67177203 conn(0x7fdab00cfc60 msgr2=0x7fdab00d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7ff34ffff640 1 -- 172.21.15.45:0/3841836367 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff358089d30 con 0x7ff36003c070 2023-11-28T20:34:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7fdab6943640 1 -- 172.21.15.45:0/67177203 shutdown_connections 2023-11-28T20:34:27.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7fdab6943640 1 -- 172.21.15.45:0/67177203 wait complete. 2023-11-28T20:34:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/3910827765 shutdown_connections 2023-11-28T20:34:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.381+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/3910827765 wait complete. 2023-11-28T20:34:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.381+0000 7f54c12ad640 1 Processor -- start 2023-11-28T20:34:27.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7f05faffd640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f05e806af10 0x7f05e806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.380+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8fc0cecf0 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.381+0000 7ff3667fc640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff35406af10 0x7ff35406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.381+0000 7f54c12ad640 1 -- start start 2023-11-28T20:34:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.382+0000 7f05faffd640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f05e806af10 0x7f05e806d3d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05f0039000 tx=0x7f05f0034000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.382+0000 7ff3667fc640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff35406af10 0x7ff35406d3d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff36004ff30 tx=0x7ff36806e030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.382+0000 7f54c12ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54bc0b0c30 0x7f54bc15e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.382+0000 7f54c12ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc0b4c40 0x7f54bc1563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.382+0000 7f54c12ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54bc161960 0x7f54bc156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f54c12ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc15ea10 con 0x7f54bc161960 2023-11-28T20:34:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f54c12ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc163350 con 0x7f54bc0b0c30 2023-11-28T20:34:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f54c12ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc162a20 con 0x7f54bc0b4c40 2023-11-28T20:34:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f9cfa5d7640 1 Processor -- start 2023-11-28T20:34:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f54b27fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc0b4c40 0x7f54bc1563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f54baffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54bc0b0c30 0x7f54bc15e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f54b27fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc0b4c40 0x7f54bc1563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:51054/0 (socket says 172.21.15.45:51054) 2023-11-28T20:34:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f54b27fc640 1 -- 172.21.15.45:0/964216754 learned_addr learned my addr 172.21.15.45:0/964216754 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f54bb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54bc161960 0x7f54bc156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.383+0000 7f9cfa5d7640 1 -- start start 2023-11-28T20:34:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.384+0000 7fdab6943640 1 Processor -- start 2023-11-28T20:34:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.384+0000 7fb8f27fc640 1 -- 172.21.15.45:0/2670690844 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb8ec0079a0 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.385+0000 7fb8f27fc640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb8e006af10 0x7fb8e006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.386+0000 7fb8f27fc640 1 -- 172.21.15.45:0/2670690844 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb8ec089710 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.384+0000 7fdab6943640 1 -- start start 2023-11-28T20:34:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.385+0000 7f9cfa5d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 0x7f9cf41f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.385+0000 7f9cfa5d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cf40d0eb0 0x7f9cf40cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.385+0000 7f9cfa5d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cf41617f0 0x7f9cf415c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.385+0000 7f9cfa5d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40c8010 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.385+0000 7f9cfa5d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40cd3f0 con 0x7f9cf40d0eb0 2023-11-28T20:34:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.385+0000 7f9cfa5d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf415c8c0 con 0x7f9cf41617f0 2023-11-28T20:34:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.386+0000 7f9cf37fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cf40d0eb0 0x7f9cf40cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.386+0000 7f9cf37fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cf40d0eb0 0x7f9cf40cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49652/0 (socket says 172.21.15.45:49652) 2023-11-28T20:34:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.386+0000 7f9cf37fe640 1 -- 172.21.15.45:0/2909502811 learned_addr learned my addr 172.21.15.45:0/2909502811 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.386+0000 7f9cf3fff640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 0x7f9cf41f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.386+0000 7f9cf8b74640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cf41617f0 0x7f9cf415c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.386+0000 7fdab6943640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdab006a500 0x7fdab01550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.384+0000 7f54b27fc640 1 -- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54bc0b0c30 msgr2=0x7f54bc15e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.384+0000 7f54b27fc640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54bc0b0c30 0x7f54bc15e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.384+0000 7f54b27fc640 1 -- 172.21.15.45:0/964216754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54bc161960 msgr2=0x7f54bc156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.384+0000 7f54b27fc640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54bc161960 0x7f54bc156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.384+0000 7f54b27fc640 1 -- 172.21.15.45:0/964216754 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b4009400 con 0x7f54bc0b4c40 2023-11-28T20:34:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.385+0000 7f54b27fc640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc0b4c40 0x7f54bc1563c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f54a400b410 tx=0x7f54a400b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.386+0000 7fb901933640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb8e006af10 0x7fb8e006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.386+0000 7fdab6943640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab00b2430 0x7fdab006dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7f9cf3fff640 1 -- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cf41617f0 msgr2=0x7f9cf415c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7f9cf3fff640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cf41617f0 0x7f9cf415c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7f9cf3fff640 1 -- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cf40d0eb0 msgr2=0x7f9cf40cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7f9cf3fff640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cf40d0eb0 0x7f9cf40cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7f9cf3fff640 1 -- 172.21.15.45:0/2909502811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec009b30 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7f9cf37fe640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cf40d0eb0 0x7f9cf40cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7fdab6943640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab0070960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7fdaaffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdab006a500 0x7fdab01550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7fdaaffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdab006a500 0x7fdab01550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49662/0 (socket says 172.21.15.45:49662) 2023-11-28T20:34:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7f9cf3fff640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 0x7f9cf41f7500 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c390 tx=0x7f9ce800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7fdaaffff640 1 -- 172.21.15.45:0/3303907119 learned_addr learned my addr 172.21.15.45:0/3303907119 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.387+0000 7fdaaf7fe640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab00b2430 0x7fdab006dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab0155600 con 0x7fdab00b2430 2023-11-28T20:34:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab006e170 con 0x7fdab006a500 2023-11-28T20:34:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab01527c0 con 0x7fdab0160e70 2023-11-28T20:34:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fdab4ee0640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab0070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fb901933640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb8e006af10 0x7fb8e006d3d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8000f60 tx=0x7fb8f8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fdaaf7fe640 1 -- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 msgr2=0x7fdab0070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fdaaf7fe640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab0070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fdaaf7fe640 1 -- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdab006a500 msgr2=0x7fdab01550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fdaaf7fe640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdab006a500 0x7fdab01550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.388+0000 7fdaaf7fe640 1 -- 172.21.15.45:0/3303907119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa0009b30 con 0x7fdab00b2430 2023-11-28T20:34:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.389+0000 7fdaaf7fe640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab00b2430 0x7fdab006dc30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fda98033000 tx=0x7fda980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.390+0000 7fdaacff9640 1 -- 172.21.15.45:0/3303907119 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98003810 con 0x7fdab00b2430 2023-11-28T20:34:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.390+0000 7f9cf0ff9640 1 -- 172.21.15.45:0/2909502811 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce80052d0 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.390+0000 7f9cf0ff9640 1 -- 172.21.15.45:0/2909502811 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce8005430 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.390+0000 7f9cf0ff9640 1 -- 172.21.15.45:0/2909502811 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8003830 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.391+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf41f8490 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.390+0000 7fdaacff9640 1 -- 172.21.15.45:0/3303907119 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda980039b0 con 0x7fdab00b2430 2023-11-28T20:34:27.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.390+0000 7fdaacff9640 1 -- 172.21.15.45:0/3303907119 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98003d60 con 0x7fdab00b2430 2023-11-28T20:34:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.391+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda98009400 con 0x7fdab00b2430 2023-11-28T20:34:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.391+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab0152d80 con 0x7fdab00b2430 2023-11-28T20:34:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.391+0000 7fb8f27fc640 1 -- 172.21.15.45:0/2670690844 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8ec0544b0 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.391+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf41f7ed0 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.392+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf40c8150 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.392+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab00733d0 con 0x7fdab00b2430 2023-11-28T20:34:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.395+0000 7f54b3fff640 1 -- 172.21.15.45:0/964216754 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4011830 con 0x7f54bc0b4c40 2023-11-28T20:34:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.395+0000 7f54b3fff640 1 -- 172.21.15.45:0/964216754 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a40119b0 con 0x7f54bc0b4c40 2023-11-28T20:34:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.395+0000 7f54b3fff640 1 -- 172.21.15.45:0/964216754 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4011d60 con 0x7f54bc0b4c40 2023-11-28T20:34:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.395+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ac009b30 con 0x7f54bc0b4c40 2023-11-28T20:34:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.394+0000 7f9cf0ff9640 1 -- 172.21.15.45:0/2909502811 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9ce8003990 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.395+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc162fe0 con 0x7f54bc0b4c40 2023-11-28T20:34:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.396+0000 7ff34ffff640 1 -- 172.21.15.45:0/3841836367 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff358054b60 con 0x7ff36003c070 2023-11-28T20:34:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.395+0000 7fdaacff9640 1 -- 172.21.15.45:0/3303907119 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fda9802e460 con 0x7fdab00b2430 2023-11-28T20:34:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.396+0000 7f9cf0ff9640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f9ce006ae90 0x7f9ce006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.396+0000 7f9cf0ff9640 1 -- 172.21.15.45:0/2909502811 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9ce8088d40 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.397+0000 7f9cf37fe640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f9ce006ae90 0x7f9ce006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.397+0000 7f05e3fff640 1 -- 172.21.15.45:0/3899443233 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05ec0547c0 con 0x7f05fc0ac050 2023-11-28T20:34:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.397+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a0005250 con 0x7f54bc0b4c40 2023-11-28T20:34:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.398+0000 7fdaacff9640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fda9006af10 0x7fda9006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.399+0000 7fdaacff9640 1 -- 172.21.15.45:0/3303907119 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fda980ab460 con 0x7fdab00b2430 2023-11-28T20:34:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.399+0000 7f9cf37fe640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f9ce006ae90 0x7f9ce006d350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9cec000f80 tx=0x7f9cec030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.399+0000 7f54b3fff640 1 -- 172.21.15.45:0/964216754 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f54a401a400 con 0x7f54bc0b4c40 2023-11-28T20:34:27.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.399+0000 7fdaaffff640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fda9006af10 0x7fda9006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.400+0000 7f54b3fff640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f549c06ae90 0x7f549c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:27.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.400+0000 7fdaaffff640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fda9006af10 0x7fda9006d3d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009340 tx=0x7fdaa000b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.400+0000 7f54b3fff640 1 -- 172.21.15.45:0/964216754 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f54a4089790 con 0x7f54bc0b4c40 2023-11-28T20:34:27.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.401+0000 7f54baffd640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f549c06ae90 0x7f549c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:27.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.402+0000 7f54baffd640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f549c06ae90 0x7f549c06d350 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f54ac035000 tx=0x7f54ac030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:27.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.405+0000 7fdaacff9640 1 -- 172.21.15.45:0/3303907119 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda98076290 con 0x7fdab00b2430 2023-11-28T20:34:27.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.414+0000 7f9cf0ff9640 1 -- 172.21.15.45:0/2909502811 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ce8053c20 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.414+0000 7f54b3fff640 1 -- 172.21.15.45:0/964216754 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54a40546c0 con 0x7f54bc0b4c40 2023-11-28T20:34:27.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.804+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff868160790 con 0x7ff8680b2620 2023-11-28T20:34:27.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.804+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3d9c0718d0 con 0x7f3d9c0b3970 2023-11-28T20:34:27.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.805+0000 7ff85a7fc640 1 -- 172.21.15.45:0/2561528245 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff85c079e70 con 0x7ff8680b2620 2023-11-28T20:34:27.806 INFO:teuthology.orchestra.run.smithi045.stdout:55834574857 2023-11-28T20:34:27.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.806+0000 7f3d937fe640 1 -- 172.21.15.45:0/4058351912 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3d84076bb0 con 0x7f3d9c0b3970 2023-11-28T20:34:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.808+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff84c06aea0 msgr2=0x7ff84c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.808+0000 7ff86eb0c640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff84c06aea0 0x7ff84c06d360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff8680c7520 tx=0x7ff864006bd0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.808+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8680b2620 msgr2=0x7ff8681558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.808+0000 7ff86eb0c640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8680b2620 0x7ff8681558e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0097a0 tx=0x7ff85c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.809+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 shutdown_connections 2023-11-28T20:34:27.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.809+0000 7ff86eb0c640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff84c06aea0 0x7ff84c06d360 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.809+0000 7ff86eb0c640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff86815d210 0x7ff86815dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.809+0000 7ff86eb0c640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8680cddd0 0x7ff8680c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.809+0000 7ff86eb0c640 1 --2- 172.21.15.45:0/2561528245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8680b2620 0x7ff8681558e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.809+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 >> 172.21.15.45:0/2561528245 conn(0x7ff868070f40 msgr2=0x7ff868071320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.811 INFO:teuthology.orchestra.run.smithi045.stdout:55834574857 2023-11-28T20:34:27.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.810+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 shutdown_connections 2023-11-28T20:34:27.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.810+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3d7c06ae40 msgr2=0x7f3d7c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.810+0000 7ff86eb0c640 1 -- 172.21.15.45:0/2561528245 wait complete. 2023-11-28T20:34:27.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.810+0000 7f3da0c2b640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3d7c06ae40 0x7f3d7c06d300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c09b800 tx=0x7f3d8c009040 comp rx=0 tx=0).stop 2023-11-28T20:34:27.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.810+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9c0b3970 msgr2=0x7f3d9c148ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.810+0000 7f3da0c2b640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9c0b3970 0x7f3d9c148ba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3d840090b0 tx=0x7f3d840046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.811+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 shutdown_connections 2023-11-28T20:34:27.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.811+0000 7f3da0c2b640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3d7c06ae40 0x7f3d7c06d300 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.811+0000 7f3da0c2b640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d9c14e710 0x7f3d9c099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.811+0000 7f3da0c2b640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9c15bcf0 0x7f3d9c099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.811+0000 7f3da0c2b640 1 --2- 172.21.15.45:0/4058351912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9c0b3970 0x7f3d9c148ba0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.811+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 >> 172.21.15.45:0/4058351912 conn(0x7f3d9c0b08b0 msgr2=0x7f3d9c070170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.812+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 shutdown_connections 2023-11-28T20:34:27.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.812+0000 7f3da0c2b640 1 -- 172.21.15.45:0/4058351912 wait complete. 2023-11-28T20:34:27.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.816+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff3600509d0 con 0x7ff36003c070 2023-11-28T20:34:27.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.818+0000 7ff34ffff640 1 -- 172.21.15.45:0/3841836367 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff3600509d0 con 0x7ff36003c070 2023-11-28T20:34:27.820 INFO:teuthology.orchestra.run.smithi045.stdout:55834574857 2023-11-28T20:34:27.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.819+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb8fc0733d0 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.820+0000 7fb8f27fc640 1 -- 172.21.15.45:0/2670690844 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8ec01e020 con 0x7fb8fc0b4c40 2023-11-28T20:34:27.824 INFO:teuthology.orchestra.run.smithi045.stdout:55834574857 2023-11-28T20:34:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.822+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff35406af10 msgr2=0x7ff35406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.822+0000 7ff367fff640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff35406af10 0x7ff35406d3d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff36004ff30 tx=0x7ff36806e030 comp rx=0 tx=0).stop 2023-11-28T20:34:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36003c070 msgr2=0x7ff360050cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36003c070 0x7ff360050cb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff35800b3e0 tx=0x7ff35800b8a0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 shutdown_connections 2023-11-28T20:34:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff35406af10 0x7ff35406d3d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3600e3130 0x7ff36003d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff360053c80 0x7ff3600d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 --2- 172.21.15.45:0/3841836367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36003c070 0x7ff360050cb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 >> 172.21.15.45:0/3841836367 conn(0x7ff360058eb0 msgr2=0x7ff3600daa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 shutdown_connections 2023-11-28T20:34:27.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.823+0000 7ff367fff640 1 -- 172.21.15.45:0/3841836367 wait complete. 2023-11-28T20:34:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.824+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb8e006af10 msgr2=0x7fb8e006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.824+0000 7fb903b97640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb8e006af10 0x7fb8e006d3d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8000f60 tx=0x7fb8f8009040 comp rx=0 tx=0).stop 2023-11-28T20:34:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.824+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8fc0b4c40 msgr2=0x7fb8fc0a8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.824+0000 7fb903b97640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8fc0b4c40 0x7fb8fc0a8f30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00c7f0 tx=0x7fb8ec00ccb0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.825+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 shutdown_connections 2023-11-28T20:34:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.825+0000 7fb903b97640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb8e006af10 0x7fb8e006d3d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.825+0000 7fb903b97640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8fc161960 0x7fb8fc0a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.825+0000 7fb903b97640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8fc0b4c40 0x7fb8fc0a8f30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.825+0000 7fb903b97640 1 --2- 172.21.15.45:0/2670690844 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8fc0b0c30 0x7fb8fc0aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.825+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 >> 172.21.15.45:0/2670690844 conn(0x7fb8fc06a500 msgr2=0x7fb8fc0af240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.825+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f05fc04e920 con 0x7f05fc0ac050 2023-11-28T20:34:27.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.825+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 shutdown_connections 2023-11-28T20:34:27.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.825+0000 7fb903b97640 1 -- 172.21.15.45:0/2670690844 wait complete. 2023-11-28T20:34:27.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.826+0000 7f05e3fff640 1 -- 172.21.15.45:0/3899443233 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f05ec057de0 con 0x7f05fc0ac050 2023-11-28T20:34:27.829 INFO:teuthology.orchestra.run.smithi045.stdout:55834574857 2023-11-28T20:34:27.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.830+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f05e806af10 msgr2=0x7f05e806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.831+0000 7f0600f47640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f05e806af10 0x7f05e806d3d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05f0039000 tx=0x7f05f0034000 comp rx=0 tx=0).stop 2023-11-28T20:34:27.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.831+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05fc0ac050 msgr2=0x7f05fc156e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.831+0000 7f0600f47640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05fc0ac050 0x7f05fc156e40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00b410 tx=0x7f05ec00b8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.831+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 shutdown_connections 2023-11-28T20:34:27.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.831+0000 7f0600f47640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f05e806af10 0x7f05e806d3d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.831+0000 7f0600f47640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05fc1590e0 0x7f05fc0cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.831+0000 7f0600f47640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05fc0cf030 0x7f05fc0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.832+0000 7f0600f47640 1 --2- 172.21.15.45:0/3899443233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05fc0ac050 0x7f05fc156e40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.832+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 >> 172.21.15.45:0/3899443233 conn(0x7f05fc06a500 msgr2=0x7f05fc157e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.832+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 shutdown_connections 2023-11-28T20:34:27.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.832+0000 7f0600f47640 1 -- 172.21.15.45:0/3899443233 wait complete. 2023-11-28T20:34:27.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.835+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdab0155600 con 0x7fdab00b2430 2023-11-28T20:34:27.838 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.0 2023-11-28T20:34:27.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.837+0000 7fdaacff9640 1 -- 172.21.15.45:0/3303907119 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fda980798b0 con 0x7fdab00b2430 2023-11-28T20:34:27.839 INFO:teuthology.orchestra.run.smithi045.stdout:55834574856 2023-11-28T20:34:27.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fda9006af10 msgr2=0x7fda9006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fda9006af10 0x7fda9006d3d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009340 tx=0x7fdaa000b040 comp rx=0 tx=0).stop 2023-11-28T20:34:27.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab00b2430 msgr2=0x7fdab006dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab00b2430 0x7fdab006dc30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fda98033000 tx=0x7fda980046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 shutdown_connections 2023-11-28T20:34:27.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fda9006af10 0x7fda9006d3d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdab0160e70 0x7fdab0070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab00b2430 0x7fdab006dc30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 --2- 172.21.15.45:0/3303907119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdab006a500 0x7fdab01550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.841+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 >> 172.21.15.45:0/3303907119 conn(0x7fdab00cfc60 msgr2=0x7fdab00c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.844 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.1 2023-11-28T20:34:27.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.842+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 shutdown_connections 2023-11-28T20:34:27.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.842+0000 7fdab6943640 1 -- 172.21.15.45:0/3303907119 wait complete. 2023-11-28T20:34:27.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.842+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9cf40733d0 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.843+0000 7f9cf0ff9640 1 -- 172.21.15.45:0/2909502811 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9ce8017090 con 0x7f9cf40cf9f0 2023-11-28T20:34:27.845 INFO:teuthology.orchestra.run.smithi045.stdout:55834574856 2023-11-28T20:34:27.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.843+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f54a00057d0 con 0x7f54bc0b4c40 2023-11-28T20:34:27.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.845+0000 7f54b3fff640 1 -- 172.21.15.45:0/964216754 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f54a4015090 con 0x7f54bc0b4c40 2023-11-28T20:34:27.847 INFO:teuthology.orchestra.run.smithi045.stdout:55834574856 2023-11-28T20:34:27.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f9ce006ae90 msgr2=0x7f9ce006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f9ce006ae90 0x7f9ce006d350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9cec000f80 tx=0x7f9cec030000 comp rx=0 tx=0).stop 2023-11-28T20:34:27.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 msgr2=0x7f9cf41f7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 0x7f9cf41f7500 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c390 tx=0x7f9ce800c850 comp rx=0 tx=0).stop 2023-11-28T20:34:27.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 shutdown_connections 2023-11-28T20:34:27.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f9ce006ae90 0x7f9ce006d350 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cf41617f0 0x7f9cf415c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cf40d0eb0 0x7f9cf40cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 --2- 172.21.15.45:0/2909502811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cf40cf9f0 0x7f9cf41f7500 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 >> 172.21.15.45:0/2909502811 conn(0x7f9cf40b4e10 msgr2=0x7f9cf40b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 shutdown_connections 2023-11-28T20:34:27.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.846+0000 7f9cfa5d7640 1 -- 172.21.15.45:0/2909502811 wait complete. 2023-11-28T20:34:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f549c06ae90 msgr2=0x7f549c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f549c06ae90 0x7f549c06d350 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f54ac035000 tx=0x7f54ac030000 comp rx=0 tx=0).stop 2023-11-28T20:34:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc0b4c40 msgr2=0x7f54bc1563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc0b4c40 0x7f54bc1563c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f54a400b410 tx=0x7f54a400b810 comp rx=0 tx=0).stop 2023-11-28T20:34:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 shutdown_connections 2023-11-28T20:34:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f549c06ae90 0x7f549c06d350 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54bc161960 0x7f54bc156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f54bc0b4c40 0x7f54bc1563c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 --2- 172.21.15.45:0/964216754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54bc0b0c30 0x7f54bc15e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:27.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.848+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 >> 172.21.15.45:0/964216754 conn(0x7f54bc06a500 msgr2=0x7f54bc160990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:27.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.849+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 shutdown_connections 2023-11-28T20:34:27.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:27.849+0000 7f54c12ad640 1 -- 172.21.15.45:0/964216754 wait complete. 2023-11-28T20:34:27.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.4 2023-11-28T20:34:27.854 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.2 2023-11-28T20:34:27.860 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.3 2023-11-28T20:34:27.867 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.5 2023-11-28T20:34:27.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.7 2023-11-28T20:34:27.872 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.6 2023-11-28T20:34:28.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T20:34:28.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T20:34:28.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-28T20:34:28.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T20:34:28.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T20:34:28.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-28T20:34:28.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-28T20:34:28.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-28T20:34:29.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.166+0000 7fbb03a9b640 1 Processor -- start 2023-11-28T20:34:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.167+0000 7fbb03a9b640 1 -- start start 2023-11-28T20:34:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.169+0000 7fbb03a9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbaf40e3120 0x7fbaf40e7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.170+0000 7fbb03a9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbaf4053c70 0x7fbaf40540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.170+0000 7fbb03a9b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf40587d0 con 0x7fbaf403c060 2023-11-28T20:34:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.170+0000 7fbb03a9b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf4007b30 con 0x7fbaf4053c70 2023-11-28T20:34:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.170+0000 7fbb03a9b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf4058ab0 con 0x7fbaf40e3120 2023-11-28T20:34:29.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.171+0000 7fbb01837640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbaf40e3120 0x7fbaf40e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.171+0000 7fbb02038640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbaf4053c70 0x7fbaf40540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.172+0000 7fbb01837640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbaf40e3120 0x7fbaf40e7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49666/0 (socket says 172.21.15.45:49666) 2023-11-28T20:34:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.172+0000 7fbb01036640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbaf403c060 0x7fbaf4058010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51880/0 (socket says 172.21.15.45:51880) 2023-11-28T20:34:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.172+0000 7fbb02038640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbaf4053c70 0x7fbaf40540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58976/0 (socket says 172.21.15.45:58976) 2023-11-28T20:34:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.172+0000 7fbb02038640 1 -- 172.21.15.45:0/1494455779 learned_addr learned my addr 172.21.15.45:0/1494455779 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.174+0000 7fbb02038640 1 -- 172.21.15.45:0/1494455779 >> v2:172.21.15.190:3300/0 conn(0x7fbaf40e3120 msgr2=0x7fbaf40e7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.174+0000 7fbb02038640 1 --2- 172.21.15.45:0/1494455779 >> v2:172.21.15.190:3300/0 conn(0x7fbaf40e3120 0x7fbaf40e7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.174+0000 7fbb02038640 1 -- 172.21.15.45:0/1494455779 >> v1:172.21.15.45:6789/0 conn(0x7fbaf403c060 legacy=0x7fbaf4058010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.174+0000 7fbb02038640 1 -- 172.21.15.45:0/1494455779 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf4006320 con 0x7fbaf4053c70 2023-11-28T20:34:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.176+0000 7fbb02038640 1 --2- 172.21.15.45:0/1494455779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 0x7fbaf40540c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbafc074320 tx=0x7fbafc0a5f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22ed9cb7b01e1e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.177+0000 7fbb00835640 1 -- 172.21.15.45:0/1494455779 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0a6d70 con 0x7fbaf4053c70 2023-11-28T20:34:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.177+0000 7fbb00835640 1 -- 172.21.15.45:0/1494455779 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc0ae410 con 0x7fbaf4053c70 2023-11-28T20:34:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.177+0000 7fbb00835640 1 -- 172.21.15.45:0/1494455779 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0ae7c0 con 0x7fbaf4053c70 2023-11-28T20:34:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.178+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1494455779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 msgr2=0x7fbaf40540c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.178+0000 7fbb03a9b640 1 --2- 172.21.15.45:0/1494455779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 0x7fbaf40540c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbafc074320 tx=0x7fbafc0a5f00 comp rx=0 tx=0).stop 2023-11-28T20:34:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.179+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1494455779 shutdown_connections 2023-11-28T20:34:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.179+0000 7fbb03a9b640 1 --2- 172.21.15.45:0/1494455779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 0x7fbaf40540c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.179+0000 7fbb03a9b640 1 --2- 172.21.15.45:0/1494455779 >> v2:172.21.15.190:3300/0 conn(0x7fbaf40e3120 0x7fbaf40e7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.179+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1494455779 >> 172.21.15.45:0/1494455779 conn(0x7fbaf4058ea0 msgr2=0x7fbaf4059280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.180+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1494455779 shutdown_connections 2023-11-28T20:34:29.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.180+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1494455779 wait complete. 2023-11-28T20:34:29.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.184+0000 7fbb03a9b640 1 Processor -- start 2023-11-28T20:34:29.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.184+0000 7fbb03a9b640 1 -- start start 2023-11-28T20:34:29.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.189+0000 7fbb03a9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf403c060 0x7fbaf4052960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.189+0000 7fbb03a9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 0x7fbaf4051d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.189+0000 7fbb03a9b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbaf40e3120 0x7fbaf40d71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.189+0000 7fbb03a9b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf4052470 con 0x7fbaf4053c70 2023-11-28T20:34:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.189+0000 7fbb03a9b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf4052280 con 0x7fbaf403c060 2023-11-28T20:34:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.189+0000 7fbb03a9b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf40d7710 con 0x7fbaf40e3120 2023-11-28T20:34:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.189+0000 7fbb01837640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf403c060 0x7fbaf4052960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.189+0000 7fbb01837640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf403c060 0x7fbaf4052960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49678/0 (socket says 172.21.15.45:49678) 2023-11-28T20:34:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.189+0000 7fbb01837640 1 -- 172.21.15.45:0/1807045543 learned_addr learned my addr 172.21.15.45:0/1807045543 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.190+0000 7fbb02038640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbaf40e3120 0x7fbaf40d71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.190+0000 7fbb01036640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 0x7fbaf4051d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.190+0000 7f3f294ad640 1 Processor -- start 2023-11-28T20:34:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.190+0000 7f3f294ad640 1 -- start start 2023-11-28T20:34:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.191+0000 7fbb01036640 1 -- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbaf40e3120 msgr2=0x7fbaf40d71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.191+0000 7fbb01036640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbaf40e3120 0x7fbaf40d71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.191+0000 7fbb01036640 1 -- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf403c060 msgr2=0x7fbaf4052960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.191+0000 7fbb01036640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf403c060 0x7fbaf4052960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.191+0000 7fbb01036640 1 -- 172.21.15.45:0/1807045543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0731b0 con 0x7fbaf4053c70 2023-11-28T20:34:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.192+0000 7fbb01036640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 0x7fbaf4051d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800c460 tx=0x7fbaf800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.192+0000 7f3f294ad640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3f24159270 0x7f3f2406c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.193+0000 7f3f294ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f240b3b70 0x7f3f2415d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.193+0000 7f3f294ad640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f24159650 con 0x7f3f240b13d0 2023-11-28T20:34:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.193+0000 7f3f294ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f24070210 con 0x7f3f240b3b70 2023-11-28T20:34:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.193+0000 7f3f294ad640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f240704f0 con 0x7f3f24159270 2023-11-28T20:34:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.192+0000 7fbaf27fc640 1 -- 172.21.15.45:0/1807045543 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80074e0 con 0x7fbaf4053c70 2023-11-28T20:34:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.192+0000 7fbaf27fc640 1 -- 172.21.15.45:0/1807045543 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf8007640 con 0x7fbaf4053c70 2023-11-28T20:34:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.193+0000 7f3f22ffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3f24159270 0x7f3f2406c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.194+0000 7f3f22ffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3f24159270 0x7f3f2406c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49694/0 (socket says 172.21.15.45:49694) 2023-11-28T20:34:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.194+0000 7f3f22ffd640 1 -- 172.21.15.45:0/3213635901 learned_addr learned my addr 172.21.15.45:0/3213635901 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.192+0000 7fbaf27fc640 1 -- 172.21.15.45:0/1807045543 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8004a00 con 0x7fbaf4053c70 2023-11-28T20:34:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.192+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf404cb70 con 0x7fbaf4053c70 2023-11-28T20:34:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.193+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf404d130 con 0x7fbaf4053c70 2023-11-28T20:34:29.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.195+0000 7f3f237fe640 1 --2- 172.21.15.45:0/3213635901 >> v2:172.21.15.45:3300/0 conn(0x7f3f240b3b70 0x7f3f2415d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.196+0000 7f3f237fe640 1 -- 172.21.15.45:0/3213635901 >> v2:172.21.15.190:3300/0 conn(0x7f3f24159270 msgr2=0x7f3f2406c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:29.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.197+0000 7fbaf27fc640 1 -- 172.21.15.45:0/1807045543 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbaf8004b60 con 0x7fbaf4053c70 2023-11-28T20:34:29.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.197+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf4005350 con 0x7fbaf4053c70 2023-11-28T20:34:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.196+0000 7f3f237fe640 1 --2- 172.21.15.45:0/3213635901 >> v2:172.21.15.190:3300/0 conn(0x7f3f24159270 0x7f3f2406c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.196+0000 7f3f237fe640 1 -- 172.21.15.45:0/3213635901 >> v1:172.21.15.190:6789/0 conn(0x7f3f240b13d0 legacy=0x7f3f2415cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.196+0000 7f3f237fe640 1 -- 172.21.15.45:0/3213635901 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f240707d0 con 0x7f3f240b3b70 2023-11-28T20:34:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.198+0000 7fbaf27fc640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbae006aec0 0x7fbae006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.199+0000 7fbaf27fc640 1 -- 172.21.15.45:0/1807045543 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbaf8089240 con 0x7fbaf4053c70 2023-11-28T20:34:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.199+0000 7f3f237fe640 1 --2- 172.21.15.45:0/3213635901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f240b3b70 0x7f3f2415d470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3f24072270 tx=0x7f3f1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73cf3e2086b455ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.200+0000 7fbb01837640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbae006aec0 0x7fbae006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.201+0000 7fbb01837640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbae006aec0 0x7fbae006d380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009b60 tx=0x7fbaec009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.205+0000 7f3bb9a64640 1 Processor -- start 2023-11-28T20:34:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.205+0000 7f3bb9a64640 1 -- start start 2023-11-28T20:34:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.207+0000 7f3bb9a64640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3bac0e3120 0x7f3bac0e7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.207+0000 7f3bb9a64640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3bac053c70 0x7f3bac0540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.207+0000 7f3bb9a64640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac0587d0 con 0x7f3bac03c060 2023-11-28T20:34:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.207+0000 7f3bb9a64640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bac007b30 con 0x7f3bac053c70 2023-11-28T20:34:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.207+0000 7f3bb9a64640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bac058ab0 con 0x7f3bac0e3120 2023-11-28T20:34:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.208+0000 7f3f21ffb640 1 -- 172.21.15.45:0/3213635901 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18031830 con 0x7f3f240b3b70 2023-11-28T20:34:29.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.208+0000 7f3f21ffb640 1 -- 172.21.15.45:0/3213635901 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f18031990 con 0x7f3f240b3b70 2023-11-28T20:34:29.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.208+0000 7f3f21ffb640 1 -- 172.21.15.45:0/3213635901 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18031d20 con 0x7f3f240b3b70 2023-11-28T20:34:29.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.207+0000 7f3bb37fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3bac0e3120 0x7f3bac0e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.208+0000 7f3bb37fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3bac0e3120 0x7f3bac0e7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49700/0 (socket says 172.21.15.45:49700) 2023-11-28T20:34:29.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.208+0000 7f3bb37fe640 1 -- 172.21.15.45:0/4128733392 learned_addr learned my addr 172.21.15.45:0/4128733392 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.208+0000 7fbaf27fc640 1 -- 172.21.15.45:0/1807045543 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbaf8053fe0 con 0x7fbaf4053c70 2023-11-28T20:34:29.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.208+0000 7f3bb37fe640 1 -- 172.21.15.45:0/4128733392 >> v2:172.21.15.45:3300/0 conn(0x7f3bac053c70 msgr2=0x7f3bac0540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:29.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.209+0000 7f3bb27fc640 1 -- 172.21.15.45:0/4128733392 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94392184 0 0) 0x7f3bac0587d0 con 0x7f3bac03c060 2023-11-28T20:34:29.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.209+0000 7fe1f5ca6640 1 Processor -- start 2023-11-28T20:34:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.209+0000 7f3bb3fff640 1 --2- 172.21.15.45:0/4128733392 >> v2:172.21.15.45:3300/0 conn(0x7f3bac053c70 0x7f3bac0540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.209+0000 7f3bb37fe640 1 --2- 172.21.15.45:0/4128733392 >> v2:172.21.15.45:3300/0 conn(0x7f3bac053c70 0x7f3bac0540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.209+0000 7f3bb37fe640 1 -- 172.21.15.45:0/4128733392 >> v1:172.21.15.45:6789/0 conn(0x7f3bac03c060 legacy=0x7f3bac058010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.210+0000 7f3bb37fe640 1 -- 172.21.15.45:0/4128733392 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac006320 con 0x7f3bac0e3120 2023-11-28T20:34:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.211+0000 7f3bb3fff640 1 --2- 172.21.15.45:0/4128733392 >> v2:172.21.15.45:3300/0 conn(0x7f3bac053c70 0x7f3bac0540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.209+0000 7fe1f5ca6640 1 -- start start 2023-11-28T20:34:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.211+0000 7fe1f5ca6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe1f00cf9f0 0x7fe1f00cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.211+0000 7fe1f5ca6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe1f00d0eb0 0x7fe1f00d1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.211+0000 7fe1f5ca6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f00b6f50 con 0x7fe1f01617f0 2023-11-28T20:34:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.212+0000 7f3f294ad640 1 -- 172.21.15.45:0/3213635901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f240b3b70 msgr2=0x7f3f2415d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.212+0000 7f3f294ad640 1 --2- 172.21.15.45:0/3213635901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f240b3b70 0x7f3f2415d470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3f24072270 tx=0x7f3f1802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.212+0000 7f3bb37fe640 1 --2- 172.21.15.45:0/4128733392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac0e3120 0x7f3bac0e7740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3bac0388d0 tx=0x7f3ba802b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58ee15bb89f29c84 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.211+0000 7fe1f5ca6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f00b83a0 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.211+0000 7fe1f5ca6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f00b71f0 con 0x7fe1f00d0eb0 2023-11-28T20:34:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.212+0000 7f3f294ad640 1 -- 172.21.15.45:0/3213635901 shutdown_connections 2023-11-28T20:34:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.212+0000 7f3f294ad640 1 --2- 172.21.15.45:0/3213635901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f240b3b70 0x7f3f2415d470 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.212+0000 7f3f294ad640 1 --2- 172.21.15.45:0/3213635901 >> v2:172.21.15.190:3300/0 conn(0x7f3f24159270 0x7f3f2406c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.212+0000 7f3f294ad640 1 -- 172.21.15.45:0/3213635901 >> 172.21.15.45:0/3213635901 conn(0x7f3f240b08b0 msgr2=0x7f3f240b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.212+0000 7f3f294ad640 1 -- 172.21.15.45:0/3213635901 shutdown_connections 2023-11-28T20:34:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.212+0000 7f3f294ad640 1 -- 172.21.15.45:0/3213635901 wait complete. 2023-11-28T20:34:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.213+0000 7fe1ef7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe1f01617f0 0x7fe1f00b7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51902/0 (socket says 172.21.15.45:51902) 2023-11-28T20:34:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.213+0000 7fe1effff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe1f00d0eb0 0x7fe1f00d1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.213+0000 7fe1eeffd640 1 --2- 172.21.15.45:0/442295301 >> v2:172.21.15.45:3300/0 conn(0x7fe1f00cf9f0 0x7fe1f00cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.213+0000 7f3bb27fc640 1 -- 172.21.15.45:0/4128733392 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8030890 con 0x7f3bac0e3120 2023-11-28T20:34:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.213+0000 7fe1ef7fe640 1 -- 172.21.15.45:0/442295301 learned_addr learned my addr 172.21.15.45:0/442295301 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.214+0000 7fe1eeffd640 1 -- 172.21.15.45:0/442295301 >> v2:172.21.15.190:3300/0 conn(0x7fe1f00d0eb0 msgr2=0x7fe1f00d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.214+0000 7f3bb27fc640 1 -- 172.21.15.45:0/4128733392 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba80309f0 con 0x7f3bac0e3120 2023-11-28T20:34:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.214+0000 7f3bb27fc640 1 -- 172.21.15.45:0/4128733392 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8030d60 con 0x7f3bac0e3120 2023-11-28T20:34:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.214+0000 7fe1eeffd640 1 --2- 172.21.15.45:0/442295301 >> v2:172.21.15.190:3300/0 conn(0x7fe1f00d0eb0 0x7fe1f00d1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.214+0000 7fe1eeffd640 1 -- 172.21.15.45:0/442295301 >> v1:172.21.15.45:6789/0 conn(0x7fe1f01617f0 legacy=0x7fe1f00b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.214+0000 7faf26961640 1 Processor -- start 2023-11-28T20:34:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7f3bb9a64640 1 -- 172.21.15.45:0/4128733392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac0e3120 msgr2=0x7f3bac0e7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7f3bb9a64640 1 --2- 172.21.15.45:0/4128733392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac0e3120 0x7f3bac0e7740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3bac0388d0 tx=0x7f3ba802b9d0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7faf26961640 1 -- start start 2023-11-28T20:34:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7f3f294ad640 1 Processor -- start 2023-11-28T20:34:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7fe1eeffd640 1 -- 172.21.15.45:0/442295301 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f00cf590 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7f3bb9a64640 1 -- 172.21.15.45:0/4128733392 shutdown_connections 2023-11-28T20:34:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7f3bb9a64640 1 --2- 172.21.15.45:0/4128733392 >> v2:172.21.15.45:3300/0 conn(0x7f3bac053c70 0x7f3bac0540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7f3bb9a64640 1 --2- 172.21.15.45:0/4128733392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac0e3120 0x7f3bac0e7740 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7f3bb9a64640 1 -- 172.21.15.45:0/4128733392 >> 172.21.15.45:0/4128733392 conn(0x7f3bac058ea0 msgr2=0x7f3bac059280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7f61a3378640 1 Processor -- start 2023-11-28T20:34:29.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.215+0000 7f3f294ad640 1 -- start start 2023-11-28T20:34:29.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.216+0000 7fe1effff640 1 --2- 172.21.15.45:0/442295301 >> v2:172.21.15.190:3300/0 conn(0x7fe1f00d0eb0 0x7fe1f00d1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:29.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.216+0000 7f3bb9a64640 1 -- 172.21.15.45:0/4128733392 shutdown_connections 2023-11-28T20:34:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.216+0000 7f3bb9a64640 1 -- 172.21.15.45:0/4128733392 wait complete. 2023-11-28T20:34:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.216+0000 7f61a3378640 1 -- start start 2023-11-28T20:34:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7faf26961640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf2015b850 0x7faf200b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7faf26961640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b1fa0 0x7faf200b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7f3f294ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f240b13d0 0x7f3f24157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7f3f294ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f240b3b70 0x7f3f241494b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7f3f294ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f24159270 0x7f3f2416afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7f3f294ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24157b30 con 0x7f3f240b3b70 2023-11-28T20:34:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7fe1eeffd640 1 --2- 172.21.15.45:0/442295301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 0x7fe1f00cfe50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe1f01663f0 tx=0x7fe1e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c18cbcbcf92ac5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7f3f294ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24157520 con 0x7f3f240b13d0 2023-11-28T20:34:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7f3f294ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2409a6c0 con 0x7f3f24159270 2023-11-28T20:34:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7faf26961640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2015bda0 con 0x7faf200b35b0 2023-11-28T20:34:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7faf26961640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf2015ee40 con 0x7faf2015b850 2023-11-28T20:34:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.217+0000 7faf26961640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2015c080 con 0x7faf200b1fa0 2023-11-28T20:34:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7fe1ee7fc640 1 -- 172.21.15.45:0/442295301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e402e070 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7faf1f7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf2015b850 0x7faf200b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7fe1ee7fc640 1 -- 172.21.15.45:0/442295301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e40315a0 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7fe1ee7fc640 1 -- 172.21.15.45:0/442295301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4031930 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7f3f22ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f240b13d0 0x7f3f24157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7f3f22ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f240b13d0 0x7f3f24157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49738/0 (socket says 172.21.15.45:49738) 2023-11-28T20:34:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7f3f22ffd640 1 -- 172.21.15.45:0/2798054689 learned_addr learned my addr 172.21.15.45:0/2798054689 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7f3f227fc640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f240b3b70 0x7f3f241494b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7f3f237fe640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f24159270 0x7f3f2416afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.219+0000 7f3bb9a64640 1 Processor -- start 2023-11-28T20:34:29.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.219+0000 7f3bb9a64640 1 -- start start 2023-11-28T20:34:29.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7faf24efe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b1fa0 0x7faf200b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7faf1f7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf2015b850 0x7faf200b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49724/0 (socket says 172.21.15.45:49724) 2023-11-28T20:34:29.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.218+0000 7faf1f7fe640 1 -- 172.21.15.45:0/919128185 learned_addr learned my addr 172.21.15.45:0/919128185 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.219+0000 7f3f22ffd640 1 -- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f24159270 msgr2=0x7f3f2416afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.219+0000 7f3f22ffd640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f24159270 0x7f3f2416afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.219+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/442295301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 msgr2=0x7fe1f00cfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.219+0000 7fe1f5ca6640 1 --2- 172.21.15.45:0/442295301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 0x7fe1f00cfe50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe1f01663f0 tx=0x7fe1e402b840 comp rx=0 tx=0).stop 2023-11-28T20:34:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.219+0000 7f3f22ffd640 1 -- 172.21.15.45:0/2798054689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f240b3b70 msgr2=0x7f3f241494b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7f3f22ffd640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f240b3b70 0x7f3f241494b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7f3f22ffd640 1 -- 172.21.15.45:0/2798054689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f18009400 con 0x7f3f240b13d0 2023-11-28T20:34:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.221+0000 7f3f22ffd640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f240b13d0 0x7f3f24157fc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f14009fd0 tx=0x7f3f1400ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7f3bb9a64640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac03c060 0x7f3bac0d6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7f3bb9a64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bac053c70 0x7f3bac050b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7f3bb9a64640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bac0e3120 0x7f3bac03db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7f3bb9a64640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac0520f0 con 0x7f3bac053c70 2023-11-28T20:34:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7f3bb9a64640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac051b30 con 0x7f3bac03c060 2023-11-28T20:34:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7f3bb9a64640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac051e10 con 0x7f3bac0e3120 2023-11-28T20:34:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/442295301 shutdown_connections 2023-11-28T20:34:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7fe1f5ca6640 1 --2- 172.21.15.45:0/442295301 >> v2:172.21.15.190:3300/0 conn(0x7fe1f00d0eb0 0x7fe1f00d1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7fe1f5ca6640 1 --2- 172.21.15.45:0/442295301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 0x7fe1f00cfe50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/442295301 >> 172.21.15.45:0/442295301 conn(0x7fe1f00b4e10 msgr2=0x7fe1f00b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/442295301 shutdown_connections 2023-11-28T20:34:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7faf1f7fe640 1 -- 172.21.15.45:0/919128185 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b1fa0 msgr2=0x7faf200b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7faf1f7fe640 1 --2- 172.21.15.45:0/919128185 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b1fa0 0x7faf200b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7faf1f7fe640 1 -- 172.21.15.45:0/919128185 >> v1:172.21.15.190:6789/0 conn(0x7faf200b35b0 legacy=0x7faf2015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7faf1f7fe640 1 -- 172.21.15.45:0/919128185 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2015c360 con 0x7faf2015b850 2023-11-28T20:34:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.221+0000 7f61a3378640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f619c0b13a0 0x7f619c0ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.221+0000 7f61a3378640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c071a10 0x7f619c071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.220+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/442295301 wait complete. 2023-11-28T20:34:29.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.221+0000 7f3bb3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bac0e3120 0x7f3bac03db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.221+0000 7f3bb37fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac03c060 0x7f3bac0d6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.221+0000 7f3bb37fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac03c060 0x7f3bac0d6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49754/0 (socket says 172.21.15.45:49754) 2023-11-28T20:34:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.221+0000 7f3bb37fe640 1 -- 172.21.15.45:0/1257292820 learned_addr learned my addr 172.21.15.45:0/1257292820 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.221+0000 7f3bb3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bac0e3120 0x7f3bac03db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:51092/0 (socket says 172.21.15.45:51092) 2023-11-28T20:34:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.221+0000 7f3bb2ffd640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bac053c70 0x7f3bac050b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f61a3378640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c0723e0 con 0x7f619c0b0fc0 2023-11-28T20:34:29.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f3f0b7fe640 1 -- 172.21.15.45:0/2798054689 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1400c940 con 0x7f3f240b13d0 2023-11-28T20:34:29.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f61a1114640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f619c0b0fc0 0x7f619c0cb940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:38460/0 (socket says 172.21.15.45:38460) 2023-11-28T20:34:29.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f61a1114640 1 -- 172.21.15.45:0/3603722648 learned_addr learned my addr 172.21.15.45:0/3603722648 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:29.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f3f0b7fe640 1 -- 172.21.15.45:0/2798054689 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1400caa0 con 0x7f3f240b13d0 2023-11-28T20:34:29.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f3f0b7fe640 1 -- 172.21.15.45:0/2798054689 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1400ce30 con 0x7f3f240b13d0 2023-11-28T20:34:29.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f61a0913640 1 --2- 172.21.15.45:0/3603722648 >> v2:172.21.15.190:3300/0 conn(0x7f619c0b13a0 0x7f619c0ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f241577c0 con 0x7f3f240b13d0 2023-11-28T20:34:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7faf1f7fe640 1 --2- 172.21.15.45:0/919128185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 0x7faf200b0c30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faf1400f9b0 tx=0x7faf1402fd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdef6a80eab8a2e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f3bb37fe640 1 -- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bac0e3120 msgr2=0x7f3bac03db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f3bb37fe640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bac0e3120 0x7f3bac03db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f3bb37fe640 1 -- 172.21.15.45:0/1257292820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bac053c70 msgr2=0x7f3bac050b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f61a3378640 1 -- 172.21.15.45:0/3603722648 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c15a940 con 0x7f619c0b13a0 2023-11-28T20:34:29.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f61a3378640 1 -- 172.21.15.45:0/3603722648 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c159dc0 con 0x7f619c071a10 2023-11-28T20:34:29.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.223+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2409a9a0 con 0x7f3f240b13d0 2023-11-28T20:34:29.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f3bb37fe640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bac053c70 0x7f3bac050b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.222+0000 7f3bb37fe640 1 -- 172.21.15.45:0/1257292820 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba80094d0 con 0x7f3bac03c060 2023-11-28T20:34:29.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.223+0000 7f3bb2ffd640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bac053c70 0x7f3bac050b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:29.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.223+0000 7fe1f5ca6640 1 Processor -- start 2023-11-28T20:34:29.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.223+0000 7f61a1915640 1 --2- 172.21.15.45:0/3603722648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c071a10 0x7f619c071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.223+0000 7f3bb37fe640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac03c060 0x7f3bac0d6630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009960 tx=0x7f3ba80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.223+0000 7f6193fff640 1 -- 172.21.15.45:0/3603722648 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049553294 0 0) 0x7f619c0723e0 con 0x7f619c0b0fc0 2023-11-28T20:34:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f6193fff640 1 -- 172.21.15.45:0/3603722648 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188003520 con 0x7f619c0b0fc0 2023-11-28T20:34:29.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.223+0000 7fe1f5ca6640 1 -- start start 2023-11-28T20:34:29.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f3b9ffff640 1 -- 172.21.15.45:0/1257292820 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8003a20 con 0x7f3bac03c060 2023-11-28T20:34:29.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f3b9ffff640 1 -- 172.21.15.45:0/1257292820 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba8003b80 con 0x7f3bac03c060 2023-11-28T20:34:29.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f3b9ffff640 1 -- 172.21.15.45:0/1257292820 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8039450 con 0x7f3bac03c060 2023-11-28T20:34:29.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7faf1effd640 1 -- 172.21.15.45:0/919128185 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14030b40 con 0x7faf2015b850 2023-11-28T20:34:29.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bac0500a0 con 0x7f3bac03c060 2023-11-28T20:34:29.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f61a0913640 1 -- 172.21.15.45:0/3603722648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c071a10 msgr2=0x7f619c071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:29.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f61a0913640 1 --2- 172.21.15.45:0/3603722648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c071a10 0x7f619c071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f61a0913640 1 -- 172.21.15.45:0/3603722648 >> v1:172.21.15.190:6789/0 conn(0x7f619c0b0fc0 legacy=0x7f619c0cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f61a0913640 1 -- 172.21.15.45:0/3603722648 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c15a0a0 con 0x7f619c0b13a0 2023-11-28T20:34:29.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7faf1effd640 1 -- 172.21.15.45:0/919128185 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf14030ca0 con 0x7faf2015b850 2023-11-28T20:34:29.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.225+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f24158500 con 0x7f3f240b13d0 2023-11-28T20:34:29.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7faf1effd640 1 -- 172.21.15.45:0/919128185 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14038660 con 0x7faf2015b850 2023-11-28T20:34:29.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.225+0000 7faf26961640 1 -- 172.21.15.45:0/919128185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 msgr2=0x7faf200b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.225+0000 7faf26961640 1 --2- 172.21.15.45:0/919128185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 0x7faf200b0c30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faf1400f9b0 tx=0x7faf1402fd90 comp rx=0 tx=0).stop 2023-11-28T20:34:29.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.224+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bac050660 con 0x7f3bac03c060 2023-11-28T20:34:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7fe1f5ca6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 0x7fe1f0153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7fe1f5ca6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1f00d0eb0 0x7fe1f006c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7fe1f5ca6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f01617f0 0x7fe1f006cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7fe1f5ca6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0151e40 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7fe1f5ca6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0152090 con 0x7fe1f01617f0 2023-11-28T20:34:29.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7fe1f5ca6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0152370 con 0x7fe1f00d0eb0 2023-11-28T20:34:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7faf26961640 1 -- 172.21.15.45:0/919128185 shutdown_connections 2023-11-28T20:34:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7faf26961640 1 --2- 172.21.15.45:0/919128185 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b1fa0 0x7faf200b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7faf26961640 1 --2- 172.21.15.45:0/919128185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 0x7faf200b0c30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7faf26961640 1 -- 172.21.15.45:0/919128185 >> 172.21.15.45:0/919128185 conn(0x7faf2006a500 msgr2=0x7faf2006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7fe1eeffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1f00d0eb0 0x7fe1f006c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7fe1ef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 0x7fe1f0153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.227+0000 7fe1eeffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1f00d0eb0 0x7fe1f006c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:51118/0 (socket says 172.21.15.45:51118) 2023-11-28T20:34:29.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.227+0000 7fe1eeffd640 1 -- 172.21.15.45:0/3949178821 learned_addr learned my addr 172.21.15.45:0/3949178821 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.227+0000 7f61a0913640 1 --2- 172.21.15.45:0/3603722648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0b13a0 0x7f619c0ce680 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f618c0097a0 tx=0x7f618c02da40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea5e97d7270eabb0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7f6193fff640 1 -- 172.21.15.45:0/3603722648 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c031860 con 0x7f619c0b13a0 2023-11-28T20:34:29.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.227+0000 7f3f0b7fe640 1 -- 172.21.15.45:0/2798054689 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3f14013470 con 0x7f3f240b13d0 2023-11-28T20:34:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bac005350 con 0x7f3bac03c060 2023-11-28T20:34:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7fe1ef7fe640 1 -- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1f00d0eb0 msgr2=0x7fe1f006c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7fe1ef7fe640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1f00d0eb0 0x7fe1f006c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7fe1ef7fe640 1 -- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f01617f0 msgr2=0x7fe1f006cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7fe1ef7fe640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f01617f0 0x7fe1f006cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7fe1ef7fe640 1 -- 172.21.15.45:0/3949178821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e0009400 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7f6193fff640 1 -- 172.21.15.45:0/3603722648 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c0319c0 con 0x7f619c0b13a0 2023-11-28T20:34:29.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7f1a3f768640 1 Processor -- start 2023-11-28T20:34:29.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7f1a3f768640 1 -- start start 2023-11-28T20:34:29.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.228+0000 7fe1ef7fe640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 0x7fe1f0153330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800caa0 tx=0x7fe1e800cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f61a3378640 1 -- 172.21.15.45:0/3603722648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0b13a0 msgr2=0x7f619c0ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f61a3378640 1 --2- 172.21.15.45:0/3603722648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0b13a0 0x7f619c0ce680 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f618c0097a0 tx=0x7f618c02da40 comp rx=0 tx=0).stop 2023-11-28T20:34:29.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f17ee04f640 1 Processor -- start 2023-11-28T20:34:29.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7faf26961640 1 -- 172.21.15.45:0/919128185 shutdown_connections 2023-11-28T20:34:29.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.226+0000 7faf26961640 1 -- 172.21.15.45:0/919128185 wait complete. 2023-11-28T20:34:29.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7faf26961640 1 Processor -- start 2023-11-28T20:34:29.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f3f0b7fe640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f1006ae90 0x7f3f1006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7faf26961640 1 -- start start 2023-11-28T20:34:29.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f61a3378640 1 -- 172.21.15.45:0/3603722648 shutdown_connections 2023-11-28T20:34:29.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f61a3378640 1 --2- 172.21.15.45:0/3603722648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c071a10 0x7f619c071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f61a3378640 1 --2- 172.21.15.45:0/3603722648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0b13a0 0x7f619c0ce680 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f61a3378640 1 -- 172.21.15.45:0/3603722648 >> 172.21.15.45:0/3603722648 conn(0x7f619c06a500 msgr2=0x7f619c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f3f0b7fe640 1 -- 172.21.15.45:0/2798054689 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3f14089920 con 0x7f3f240b13d0 2023-11-28T20:34:29.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7f3b9ffff640 1 -- 172.21.15.45:0/1257292820 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3ba80395b0 con 0x7f3bac03c060 2023-11-28T20:34:29.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7f3f227fc640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f1006ae90 0x7f3f1006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7f17ee04f640 1 -- start start 2023-11-28T20:34:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.229+0000 7fe1d7fff640 1 -- 172.21.15.45:0/3949178821 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e800ab40 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7faf26961640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf200b1fa0 0x7faf200adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7fe1d7fff640 1 -- 172.21.15.45:0/3949178821 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e800aca0 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e4009400 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7fe1d7fff640 1 -- 172.21.15.45:0/3949178821 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8012600 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f0152930 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7faf26961640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b35b0 0x7faf201450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7faf26961640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 0x7faf201e4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7faf26961640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf200ae060 con 0x7faf200b1fa0 2023-11-28T20:34:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7faf26961640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf201e6110 con 0x7faf2015b850 2023-11-28T20:34:29.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.230+0000 7faf26961640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf201455e0 con 0x7faf200b35b0 2023-11-28T20:34:29.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7faf24efe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 0x7faf201e4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7faf24efe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 0x7faf201e4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:49786/0 (socket says 172.21.15.45:49786) 2023-11-28T20:34:29.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7faf24efe640 1 -- 172.21.15.45:0/1738190197 learned_addr learned my addr 172.21.15.45:0/1738190197 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7f17ee04f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17e8160e70 0x7f17e80b6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7f17ee04f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f17e806a500 0x7f17e8070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7f17ee04f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e806aa70 con 0x7f17e80b2430 2023-11-28T20:34:29.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7f17ee04f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e806ad50 con 0x7f17e8160e70 2023-11-28T20:34:29.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7f17ee04f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e806b030 con 0x7f17e806a500 2023-11-28T20:34:29.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7f61a3378640 1 -- 172.21.15.45:0/3603722648 shutdown_connections 2023-11-28T20:34:29.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.231+0000 7faf1f7fe640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b35b0 0x7faf201450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.232+0000 7f3f227fc640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f1006ae90 0x7f3f1006d350 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f240ce950 tx=0x7f3f1c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.232+0000 7f3b9ffff640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3b9806ae40 0x7f3b9806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.232+0000 7f3b9ffff640 1 -- 172.21.15.45:0/1257292820 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3ba80abfa0 con 0x7f3bac03c060 2023-11-28T20:34:29.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.232+0000 7faf1ffff640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf200b1fa0 0x7faf200adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.232+0000 7faf24efe640 1 -- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b35b0 msgr2=0x7faf201450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7faf24efe640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b35b0 0x7faf201450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.232+0000 7f17e7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f17e80b2430 0x7f17e80d1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51906/0 (socket says 172.21.15.45:51906) 2023-11-28T20:34:29.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7f17e7fff640 1 -- 172.21.15.45:0/101682408 learned_addr learned my addr 172.21.15.45:0/101682408 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:29.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.232+0000 7f17e77fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17e8160e70 0x7f17e80b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7f1a3d504640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1a380b0fc0 0x7f1a380cb940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51922/0 (socket says 172.21.15.45:51922) 2023-11-28T20:34:29.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7f1a3d504640 1 -- 172.21.15.45:0/1788505415 learned_addr learned my addr 172.21.15.45:0/1788505415 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:29.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f0152b50 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7faf24efe640 1 -- 172.21.15.45:0/1738190197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf200b1fa0 msgr2=0x7faf200adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7f17e6ffd640 1 --2- 172.21.15.45:0/101682408 >> v2:172.21.15.190:3300/0 conn(0x7f17e806a500 0x7f17e8070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1788505415 >> v2:172.21.15.45:3300/0 conn(0x7f1a380b13a0 0x7f1a380ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7faf24efe640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf200b1fa0 0x7faf200adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1788505415 >> v2:172.21.15.190:3300/0 conn(0x7f1a38071a10 0x7f1a38071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7faf24efe640 1 -- 172.21.15.45:0/1738190197 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1400f610 con 0x7faf2015b850 2023-11-28T20:34:29.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7faf1ffff640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf200b1fa0 0x7faf200adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:29.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7f17e77fe640 1 -- 172.21.15.45:0/101682408 >> v2:172.21.15.190:3300/0 conn(0x7f17e806a500 msgr2=0x7f17e8070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:29.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7f17e77fe640 1 --2- 172.21.15.45:0/101682408 >> v2:172.21.15.190:3300/0 conn(0x7f17e806a500 0x7f17e8070b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7f17e77fe640 1 -- 172.21.15.45:0/101682408 >> v1:172.21.15.45:6789/0 conn(0x7f17e80b2430 legacy=0x7f17e80d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7f1a3f768640 1 -- 172.21.15.45:0/1788505415 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3815a940 con 0x7f1a380b0fc0 2023-11-28T20:34:29.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7f1a3f768640 1 -- 172.21.15.45:0/1788505415 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a380723a0 con 0x7f1a380b13a0 2023-11-28T20:34:29.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.233+0000 7f1a3f768640 1 -- 172.21.15.45:0/1788505415 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a38159dc0 con 0x7f1a38071a10 2023-11-28T20:34:29.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7f1a3cd03640 1 --2- 172.21.15.45:0/1788505415 >> v2:172.21.15.45:3300/0 conn(0x7f1a380b13a0 0x7f1a380ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7faf24efe640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 0x7faf201e4990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faf1800efc0 tx=0x7faf1800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.232+0000 7f61a3378640 1 -- 172.21.15.45:0/3603722648 wait complete. 2023-11-28T20:34:29.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7f61a3378640 1 Processor -- start 2023-11-28T20:34:29.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7f17e67fc640 1 -- 172.21.15.45:0/101682408 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015218548 0 0) 0x7f17e806aa70 con 0x7f17e80b2430 2023-11-28T20:34:29.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7f17e77fe640 1 -- 172.21.15.45:0/101682408 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e806b310 con 0x7f17e8160e70 2023-11-28T20:34:29.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.234+0000 7f17e6ffd640 1 --2- 172.21.15.45:0/101682408 >> v2:172.21.15.190:3300/0 conn(0x7f17e806a500 0x7f17e8070b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:34:29.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7f1a3dd05640 1 --2- 172.21.15.45:0/1788505415 >> v2:172.21.15.190:3300/0 conn(0x7f1a38071a10 0x7f1a38071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7f61a3378640 1 -- start start 2023-11-28T20:34:29.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7f17e77fe640 1 --2- 172.21.15.45:0/101682408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e8160e70 0x7f17e80b6940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f17e80d30a0 tx=0x7f17e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4218f65b614911b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7f3bb2ffd640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3b9806ae40 0x7f3b9806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7fe1d7fff640 1 -- 172.21.15.45:0/3949178821 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe1e8012760 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.236+0000 7f17e67fc640 1 -- 172.21.15.45:0/101682408 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e002cc90 con 0x7f17e8160e70 2023-11-28T20:34:29.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.236+0000 7f17e67fc640 1 -- 172.21.15.45:0/101682408 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e002cdf0 con 0x7f17e8160e70 2023-11-28T20:34:29.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.236+0000 7f17e67fc640 1 -- 172.21.15.45:0/101682408 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0034660 con 0x7f17e8160e70 2023-11-28T20:34:29.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7f1a3cd03640 1 -- 172.21.15.45:0/1788505415 >> v2:172.21.15.190:3300/0 conn(0x7f1a38071a10 msgr2=0x7f1a38071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:29.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7f1a3cd03640 1 --2- 172.21.15.45:0/1788505415 >> v2:172.21.15.190:3300/0 conn(0x7f1a38071a10 0x7f1a38071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7f1a3cd03640 1 -- 172.21.15.45:0/1788505415 >> v1:172.21.15.45:6789/0 conn(0x7f1a380b0fc0 legacy=0x7f1a380cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7f1a3cd03640 1 -- 172.21.15.45:0/1788505415 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3815a0a0 con 0x7f1a380b13a0 2023-11-28T20:34:29.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.235+0000 7f1a2ffff640 1 -- 172.21.15.45:0/1788505415 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047117421 0 0) 0x7f1a3815a940 con 0x7f1a380b0fc0 2023-11-28T20:34:29.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.236+0000 7f61a3378640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c071a10 0x7f619c1f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f17ee04f640 1 -- 172.21.15.45:0/101682408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e8160e70 msgr2=0x7f17e80b6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f17ee04f640 1 --2- 172.21.15.45:0/101682408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e8160e70 0x7f17e80b6940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f17e80d30a0 tx=0x7f17e002bee0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.236+0000 7f1a3cd03640 1 --2- 172.21.15.45:0/1788505415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 0x7f1a380ce680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1a380cd3a0 tx=0x7f1a2802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1510192356d721 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f61a3378640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c0b0fc0 0x7f619c1f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f17ee04f640 1 -- 172.21.15.45:0/101682408 shutdown_connections 2023-11-28T20:34:29.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f61a3378640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f619c0b13a0 0x7f619c0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f3bb2ffd640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3b9806ae40 0x7f3b9806d300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3bac03e910 tx=0x7f3ba0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f17ee04f640 1 --2- 172.21.15.45:0/101682408 >> v2:172.21.15.190:3300/0 conn(0x7f17e806a500 0x7f17e8070b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f17ee04f640 1 --2- 172.21.15.45:0/101682408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e8160e70 0x7f17e80b6940 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f17ee04f640 1 -- 172.21.15.45:0/101682408 >> 172.21.15.45:0/101682408 conn(0x7f17e80cfc60 msgr2=0x7f17e80d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f17ee04f640 1 -- 172.21.15.45:0/101682408 shutdown_connections 2023-11-28T20:34:29.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f17ee04f640 1 -- 172.21.15.45:0/101682408 wait complete. 2023-11-28T20:34:29.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f61a1114640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c071a10 0x7f619c1f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7fe1d7fff640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1dc06ae40 0x7fe1dc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f61a3378640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c1f5240 con 0x7f619c0b13a0 2023-11-28T20:34:29.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f61a3378640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c0caef0 con 0x7f619c071a10 2023-11-28T20:34:29.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.237+0000 7f61a3378640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c0cac10 con 0x7f619c0b0fc0 2023-11-28T20:34:29.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7f61a1915640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f619c0b13a0 0x7f619c0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7f61a0913640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c0b0fc0 0x7f619c1f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7f61a1915640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f619c0b13a0 0x7f619c0c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59038/0 (socket says 172.21.15.45:59038) 2023-11-28T20:34:29.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7f61a1915640 1 -- 172.21.15.45:0/1071026845 learned_addr learned my addr 172.21.15.45:0/1071026845 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7fe1d7fff640 1 -- 172.21.15.45:0/3949178821 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe1e8089780 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7fe1eeffd640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1dc06ae40 0x7fe1dc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7f61a1915640 1 -- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c0b0fc0 msgr2=0x7f619c1f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7f61a1915640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c0b0fc0 0x7f619c1f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7f61a1915640 1 -- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c071a10 msgr2=0x7f619c1f5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7f61a1915640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c071a10 0x7f619c1f5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.238+0000 7f61a1915640 1 -- 172.21.15.45:0/1071026845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c009400 con 0x7f619c0b13a0 2023-11-28T20:34:29.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.239+0000 7f61a1114640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c071a10 0x7f619c1f5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:29.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.239+0000 7f61a1915640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f619c0b13a0 0x7f619c0c4c10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f619400efc0 tx=0x7f619400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.240+0000 7fe1eeffd640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1dc06ae40 0x7fe1dc06d300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00c9d40 tx=0x7fe1e403d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.240+0000 7f17ee04f640 1 Processor -- start 2023-11-28T20:34:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.240+0000 7f17ee04f640 1 -- start start 2023-11-28T20:34:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.241+0000 7f17ee04f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e806a500 0x7f17e80ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.241+0000 7f17ee04f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80b2430 0x7f17e80ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.241+0000 7f17ee04f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17e8160e70 0x7f17e80c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.241+0000 7f17ee04f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80ca8d0 con 0x7f17e80b2430 2023-11-28T20:34:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.241+0000 7f17ee04f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80cd0c0 con 0x7f17e806a500 2023-11-28T20:34:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.241+0000 7f17ee04f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e814f820 con 0x7f17e8160e70 2023-11-28T20:34:29.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.241+0000 7f17e6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80b2430 0x7f17e80ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.241+0000 7f17e6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80b2430 0x7f17e80ccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59042/0 (socket says 172.21.15.45:59042) 2023-11-28T20:34:29.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.241+0000 7f17e6ffd640 1 -- 172.21.15.45:0/1890602121 learned_addr learned my addr 172.21.15.45:0/1890602121 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f17e77fe640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e806a500 0x7f17e80ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f1a2ffff640 1 -- 172.21.15.45:0/1788505415 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2802e070 con 0x7f1a380b13a0 2023-11-28T20:34:29.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f1a2ffff640 1 -- 172.21.15.45:0/1788505415 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a280315a0 con 0x7f1a380b13a0 2023-11-28T20:34:29.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f1a2ffff640 1 -- 172.21.15.45:0/1788505415 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28031930 con 0x7f1a380b13a0 2023-11-28T20:34:29.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f17e7fff640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17e8160e70 0x7f17e80c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f6191ffb640 1 -- 172.21.15.45:0/1071026845 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194014480 con 0x7f619c0b13a0 2023-11-28T20:34:29.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f6191ffb640 1 -- 172.21.15.45:0/1071026845 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619400f040 con 0x7f619c0b13a0 2023-11-28T20:34:29.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f6191ffb640 1 -- 172.21.15.45:0/1071026845 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194014820 con 0x7f619c0b13a0 2023-11-28T20:34:29.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7fe1d7fff640 1 -- 172.21.15.45:0/3949178821 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1e8054560 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f17e6ffd640 1 -- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17e8160e70 msgr2=0x7f17e80c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f17e6ffd640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17e8160e70 0x7f17e80c8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f17e6ffd640 1 -- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e806a500 msgr2=0x7f17e80ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f17e6ffd640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e806a500 0x7f17e80ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c1f4b80 con 0x7f619c0b13a0 2023-11-28T20:34:29.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.242+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c0c5150 con 0x7f619c0b13a0 2023-11-28T20:34:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f17e6ffd640 1 -- 172.21.15.45:0/1890602121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d8009400 con 0x7f17e80b2430 2023-11-28T20:34:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f1a3f768640 1 -- 172.21.15.45:0/1788505415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 msgr2=0x7f1a380ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1788505415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 0x7f1a380ce680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1a380cd3a0 tx=0x7f1a2802b840 comp rx=0 tx=0).stop 2023-11-28T20:34:29.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f1a3f768640 1 -- 172.21.15.45:0/1788505415 shutdown_connections 2023-11-28T20:34:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1788505415 >> v2:172.21.15.190:3300/0 conn(0x7f1a38071a10 0x7f1a38071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1788505415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 0x7f1a380ce680 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f1a3f768640 1 -- 172.21.15.45:0/1788505415 >> 172.21.15.45:0/1788505415 conn(0x7f1a3806a500 msgr2=0x7f1a3806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f1a3f768640 1 -- 172.21.15.45:0/1788505415 shutdown_connections 2023-11-28T20:34:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c1f5380 con 0x7f619c0b13a0 2023-11-28T20:34:29.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.243+0000 7f1a3f768640 1 -- 172.21.15.45:0/1788505415 wait complete. 2023-11-28T20:34:29.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.244+0000 7f17e6ffd640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80b2430 0x7f17e80ccb80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ebe0 tx=0x7f17d800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.244+0000 7f17cffff640 1 -- 172.21.15.45:0/1890602121 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d8009760 con 0x7f17e80b2430 2023-11-28T20:34:29.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.245+0000 7f17cffff640 1 -- 172.21.15.45:0/1890602121 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d800cc30 con 0x7f17e80b2430 2023-11-28T20:34:29.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.245+0000 7f17cffff640 1 -- 172.21.15.45:0/1890602121 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d800eee0 con 0x7f17e80b2430 2023-11-28T20:34:29.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.245+0000 7f1a3f768640 1 Processor -- start 2023-11-28T20:34:29.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.245+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e0009a70 con 0x7f17e80b2430 2023-11-28T20:34:29.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.245+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e814dce0 con 0x7f17e80b2430 2023-11-28T20:34:29.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.246+0000 7f6191ffb640 1 -- 172.21.15.45:0/1071026845 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6194014980 con 0x7f619c0b13a0 2023-11-28T20:34:29.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.246+0000 7f1a3f768640 1 -- start start 2023-11-28T20:34:29.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3f768640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a38071a10 0x7f1a380c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3f768640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a380b0fc0 0x7f1a380b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3f768640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 0x7f1a380c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3f768640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a380c7ff0 con 0x7f1a380b13a0 2023-11-28T20:34:29.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3f768640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a380c73e0 con 0x7f1a380b0fc0 2023-11-28T20:34:29.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3f768640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a380c76c0 con 0x7f1a38071a10 2023-11-28T20:34:29.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e80733d0 con 0x7f17e80b2430 2023-11-28T20:34:29.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3dd05640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 0x7f1a380c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f6191ffb640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f618006ae70 0x7f618006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3dd05640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 0x7f1a380c4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59046/0 (socket says 172.21.15.45:59046) 2023-11-28T20:34:29.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3dd05640 1 -- 172.21.15.45:0/1798358587 learned_addr learned my addr 172.21.15.45:0/1798358587 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f6191ffb640 1 -- 172.21.15.45:0/1071026845 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f619408a3e0 con 0x7f619c0b13a0 2023-11-28T20:34:29.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.247+0000 7f1a3cd03640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a380b0fc0 0x7f1a380b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.248+0000 7f1a3d504640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a38071a10 0x7f1a380c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.248+0000 7f1a3dd05640 1 -- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a38071a10 msgr2=0x7f1a380c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.248+0000 7f1a3dd05640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a38071a10 0x7f1a380c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.248+0000 7f1a3dd05640 1 -- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a380b0fc0 msgr2=0x7f1a380b39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.248+0000 7f1a3dd05640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a380b0fc0 0x7f1a380b39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.248+0000 7f1a3dd05640 1 -- 172.21.15.45:0/1798358587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a28009400 con 0x7f1a380b13a0 2023-11-28T20:34:29.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.248+0000 7f3f0b7fe640 1 -- 172.21.15.45:0/2798054689 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f14054870 con 0x7f3f240b13d0 2023-11-28T20:34:29.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.248+0000 7f1a3dd05640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 0x7f1a380c4f60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000cc30 tx=0x7f1a3000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.248+0000 7f61a1114640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f618006ae70 0x7f618006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.249+0000 7f61a1114640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f618006ae70 0x7f618006d330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f61980011e0 tx=0x7f6198009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.251+0000 7f1a2dffb640 1 -- 172.21.15.45:0/1798358587 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3000ce40 con 0x7f1a380b13a0 2023-11-28T20:34:29.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.252+0000 7f1a2dffb640 1 -- 172.21.15.45:0/1798358587 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a30003990 con 0x7f1a380b13a0 2023-11-28T20:34:29.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.252+0000 7f1a2dffb640 1 -- 172.21.15.45:0/1798358587 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30003d20 con 0x7f1a380b13a0 2023-11-28T20:34:29.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.252+0000 7f6191ffb640 1 -- 172.21.15.45:0/1071026845 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f619c1f5380 con 0x7f619c0b13a0 2023-11-28T20:34:29.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.252+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a380c79a0 con 0x7f1a380b13a0 2023-11-28T20:34:29.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.252+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a380c7040 con 0x7f1a380b13a0 2023-11-28T20:34:29.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.253+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a380c81a0 con 0x7f1a380b13a0 2023-11-28T20:34:29.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.255+0000 7f17cffff640 1 -- 172.21.15.45:0/1890602121 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f17d8013470 con 0x7f17e80b2430 2023-11-28T20:34:29.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.256+0000 7f17cffff640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f17d406af10 0x7f17d406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.256+0000 7f17cffff640 1 -- 172.21.15.45:0/1890602121 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f17d808a340 con 0x7f17e80b2430 2023-11-28T20:34:29.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.257+0000 7f17e77fe640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f17d406af10 0x7f17d406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.258+0000 7f17e77fe640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f17d406af10 0x7f17d406d3d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17e0036000 tx=0x7f17e0030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.259+0000 7f17cffff640 1 -- 172.21.15.45:0/1890602121 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17d8055190 con 0x7f17e80b2430 2023-11-28T20:34:29.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.263+0000 7faf1cff9640 1 -- 172.21.15.45:0/1738190197 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf18009ca0 con 0x7faf2015b850 2023-11-28T20:34:29.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.263+0000 7faf1cff9640 1 -- 172.21.15.45:0/1738190197 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1800f040 con 0x7faf2015b850 2023-11-28T20:34:29.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.263+0000 7faf1cff9640 1 -- 172.21.15.45:0/1738190197 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf180146c0 con 0x7faf2015b850 2023-11-28T20:34:29.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.263+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf201e5500 con 0x7faf2015b850 2023-11-28T20:34:29.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.263+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf201e5ac0 con 0x7faf2015b850 2023-11-28T20:34:29.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.262+0000 7f1a2dffb640 1 -- 172.21.15.45:0/1798358587 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1a30003e80 con 0x7f1a380b13a0 2023-11-28T20:34:29.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.263+0000 7f1a2dffb640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a1c06ae90 0x7f1a1c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.263+0000 7f1a2dffb640 1 -- 172.21.15.45:0/1798358587 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1a30089fe0 con 0x7f1a380b13a0 2023-11-28T20:34:29.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.264+0000 7f3b9ffff640 1 -- 172.21.15.45:0/1257292820 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ba8076d50 con 0x7f3bac03c060 2023-11-28T20:34:29.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.264+0000 7f1a3d504640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a1c06ae90 0x7f1a1c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.265+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf00005250 con 0x7faf2015b850 2023-11-28T20:34:29.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.266+0000 7f1a3d504640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a1c06ae90 0x7f1a1c06d350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1a34000f60 tx=0x7f1a34007b90 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.268+0000 7f1a2dffb640 1 -- 172.21.15.45:0/1798358587 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a30054d80 con 0x7f1a380b13a0 2023-11-28T20:34:29.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.269+0000 7faf1cff9640 1 -- 172.21.15.45:0/1738190197 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faf18014820 con 0x7faf2015b850 2023-11-28T20:34:29.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.270+0000 7faf1cff9640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faf0c06af40 0x7faf0c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.271+0000 7faf1cff9640 1 -- 172.21.15.45:0/1738190197 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faf1808ae30 con 0x7faf2015b850 2023-11-28T20:34:29.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.271+0000 7faf1ffff640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faf0c06af40 0x7faf0c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.273+0000 7faf1ffff640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faf0c06af40 0x7faf0c06d400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faf10007fd0 tx=0x7faf10009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.281+0000 7faf1cff9640 1 -- 172.21.15.45:0/1738190197 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf18055bd0 con 0x7faf2015b850 2023-11-28T20:34:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.622+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbaf40584f0 con 0x7fbaf4053c70 2023-11-28T20:34:29.625 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.623+0000 7fbaf27fc640 1 -- 172.21.15.45:0/1807045543 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbaf8057600 con 0x7fbaf4053c70 2023-11-28T20:34:29.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.626+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbae006aec0 msgr2=0x7fbae006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.626+0000 7fbb03a9b640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbae006aec0 0x7fbae006d380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009b60 tx=0x7fbaec009190 comp rx=0 tx=0).stop 2023-11-28T20:34:29.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.627+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 msgr2=0x7fbaf4051d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.627+0000 7fbb03a9b640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 0x7fbaf4051d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800c460 tx=0x7fbaf800c920 comp rx=0 tx=0).stop 2023-11-28T20:34:29.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.627+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 shutdown_connections 2023-11-28T20:34:29.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.627+0000 7fbb03a9b640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbae006aec0 0x7fbae006d380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.627+0000 7fbb03a9b640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbaf40e3120 0x7fbaf40d71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.627+0000 7fbb03a9b640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf4053c70 0x7fbaf4051d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.627+0000 7fbb03a9b640 1 --2- 172.21.15.45:0/1807045543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf403c060 0x7fbaf4052960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.627+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 >> 172.21.15.45:0/1807045543 conn(0x7fbaf4058ea0 msgr2=0x7fbaf4006ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.628+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 shutdown_connections 2023-11-28T20:34:29.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.628+0000 7fbb03a9b640 1 -- 172.21.15.45:0/1807045543 wait complete. 2023-11-28T20:34:29.657 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.0 2023-11-28T20:34:29.658 DEBUG:teuthology.parallel:result is None 2023-11-28T20:34:29.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.666+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe1f00b5ff0 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.667+0000 7fe1d7fff640 1 -- 172.21.15.45:0/3949178821 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe1e8057b80 con 0x7fe1f00cf9f0 2023-11-28T20:34:29.669 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:29.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.668+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3f24157950 con 0x7f3f240b13d0 2023-11-28T20:34:29.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.669+0000 7f3f0b7fe640 1 -- 172.21.15.45:0/2798054689 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f14057e90 con 0x7f3f240b13d0 2023-11-28T20:34:29.672 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.670+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1dc06ae40 msgr2=0x7fe1dc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1dc06ae40 0x7fe1dc06d300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00c9d40 tx=0x7fe1e403d040 comp rx=0 tx=0).stop 2023-11-28T20:34:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3bac0545d0 con 0x7f3bac03c060 2023-11-28T20:34:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 msgr2=0x7fe1f0153330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 0x7fe1f0153330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800caa0 tx=0x7fe1e800cea0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 shutdown_connections 2023-11-28T20:34:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1dc06ae40 0x7fe1dc06d300 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f01617f0 0x7fe1f006cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1f00d0eb0 0x7fe1f006c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 --2- 172.21.15.45:0/3949178821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1f00cf9f0 0x7fe1f0153330 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 >> 172.21.15.45:0/3949178821 conn(0x7fe1f00b4e10 msgr2=0x7fe1f00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 shutdown_connections 2023-11-28T20:34:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.671+0000 7fe1f5ca6640 1 -- 172.21.15.45:0/3949178821 wait complete. 2023-11-28T20:34:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.672+0000 7f3b9ffff640 1 -- 172.21.15.45:0/1257292820 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ba807a370 con 0x7f3bac03c060 2023-11-28T20:34:29.675 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f1006ae90 msgr2=0x7f3f1006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f1006ae90 0x7f3f1006d350 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f240ce950 tx=0x7f3f1c009040 comp rx=0 tx=0).stop 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f240b13d0 msgr2=0x7f3f24157fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f240b13d0 0x7f3f24157fc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f14009fd0 tx=0x7f3f1400ed40 comp rx=0 tx=0).stop 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 shutdown_connections 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f1006ae90 0x7f3f1006d350 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f24159270 0x7f3f2416afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f240b3b70 0x7f3f241494b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 --2- 172.21.15.45:0/2798054689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f240b13d0 0x7f3f24157fc0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f14009fd0 tx=0x7f3f1400ed40 comp rx=0 tx=0).stop 2023-11-28T20:34:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.673+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 >> 172.21.15.45:0/2798054689 conn(0x7f3f240b08b0 msgr2=0x7f3f2414b790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.674+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 shutdown_connections 2023-11-28T20:34:29.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.674+0000 7f3f294ad640 1 -- 172.21.15.45:0/2798054689 wait complete. 2023-11-28T20:34:29.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3b9806ae40 msgr2=0x7f3b9806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3b9806ae40 0x7f3b9806d300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3bac03e910 tx=0x7f3ba0009040 comp rx=0 tx=0).stop 2023-11-28T20:34:29.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac03c060 msgr2=0x7f3bac0d6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac03c060 0x7f3bac0d6630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009960 tx=0x7f3ba80046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 shutdown_connections 2023-11-28T20:34:29.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3b9806ae40 0x7f3b9806d300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bac0e3120 0x7f3bac03db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bac053c70 0x7f3bac050b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 --2- 172.21.15.45:0/1257292820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bac03c060 0x7f3bac0d6630 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.675+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 >> 172.21.15.45:0/1257292820 conn(0x7f3bac058ea0 msgr2=0x7f3bac006eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.676+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f619c1f5380 con 0x7f619c0b13a0 2023-11-28T20:34:29.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.676+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 shutdown_connections 2023-11-28T20:34:29.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.676+0000 7f3bb9a64640 1 -- 172.21.15.45:0/1257292820 wait complete. 2023-11-28T20:34:29.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.677+0000 7f6191ffb640 1 -- 172.21.15.45:0/1071026845 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f619c1f5380 con 0x7f619c0b13a0 2023-11-28T20:34:29.679 INFO:teuthology.orchestra.run.smithi045.stdout:55834574858 2023-11-28T20:34:29.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.680+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f618006ae70 msgr2=0x7f618006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.680+0000 7f61a3378640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f618006ae70 0x7f618006d330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f61980011e0 tx=0x7f6198009040 comp rx=0 tx=0).stop 2023-11-28T20:34:29.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.680+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f619c0b13a0 msgr2=0x7f619c0c4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.680+0000 7f61a3378640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f619c0b13a0 0x7f619c0c4c10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f619400efc0 tx=0x7f619400c490 comp rx=0 tx=0).stop 2023-11-28T20:34:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.681+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 shutdown_connections 2023-11-28T20:34:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.681+0000 7f61a3378640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f618006ae70 0x7f618006d330 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.681+0000 7f61a3378640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f619c0b13a0 0x7f619c0c4c10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.681+0000 7f61a3378640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f619c0b0fc0 0x7f619c1f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.681+0000 7f61a3378640 1 --2- 172.21.15.45:0/1071026845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c071a10 0x7f619c1f5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.681+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 >> 172.21.15.45:0/1071026845 conn(0x7f619c06a500 msgr2=0x7f619c06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.681+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 shutdown_connections 2023-11-28T20:34:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.682+0000 7f61a3378640 1 -- 172.21.15.45:0/1071026845 wait complete. 2023-11-28T20:34:29.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.683+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f17e80b2e70 con 0x7f17e80b2430 2023-11-28T20:34:29.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.684+0000 7f17cffff640 1 -- 172.21.15.45:0/1890602121 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f17d80587b0 con 0x7f17e80b2430 2023-11-28T20:34:29.687 INFO:teuthology.orchestra.run.smithi045.stdout:55834574858 2023-11-28T20:34:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f1a380c81a0 con 0x7f1a380b13a0 2023-11-28T20:34:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f17d406af10 msgr2=0x7f17d406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f17ee04f640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f17d406af10 0x7f17d406d3d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17e0036000 tx=0x7f17e0030000 comp rx=0 tx=0).stop 2023-11-28T20:34:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80b2430 msgr2=0x7f17e80ccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f17ee04f640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80b2430 0x7f17e80ccb80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ebe0 tx=0x7f17d800c520 comp rx=0 tx=0).stop 2023-11-28T20:34:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 shutdown_connections 2023-11-28T20:34:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f17ee04f640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f17d406af10 0x7f17d406d3d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f17ee04f640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17e8160e70 0x7f17e80c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f17ee04f640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e80b2430 0x7f17e80ccb80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.688+0000 7f17ee04f640 1 --2- 172.21.15.45:0/1890602121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e806a500 0x7f17e80ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.689+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 >> 172.21.15.45:0/1890602121 conn(0x7f17e80cfc60 msgr2=0x7f17e80b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.689+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 shutdown_connections 2023-11-28T20:34:29.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.689+0000 7f17ee04f640 1 -- 172.21.15.45:0/1890602121 wait complete. 2023-11-28T20:34:29.692 INFO:teuthology.orchestra.run.smithi045.stdout:55834574858 2023-11-28T20:34:29.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.689+0000 7f1a2dffb640 1 -- 172.21.15.45:0/1798358587 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1a380c81a0 con 0x7f1a380b13a0 2023-11-28T20:34:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.691+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a1c06ae90 msgr2=0x7f1a1c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.691+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a1c06ae90 0x7f1a1c06d350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1a34000f60 tx=0x7f1a34007b90 comp rx=0 tx=0).stop 2023-11-28T20:34:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.691+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 msgr2=0x7f1a380c4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.692+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 0x7f1a380c4f60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000cc30 tx=0x7f1a3000a5c0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.692+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 shutdown_connections 2023-11-28T20:34:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.692+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a1c06ae90 0x7f1a1c06d350 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.692+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a380b13a0 0x7f1a380c4f60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.692+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a380b0fc0 0x7f1a380b39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.692+0000 7f1a3f768640 1 --2- 172.21.15.45:0/1798358587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a38071a10 0x7f1a380c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.692+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 >> 172.21.15.45:0/1798358587 conn(0x7f1a3806a500 msgr2=0x7f1a3806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.692+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 shutdown_connections 2023-11-28T20:34:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.692+0000 7f1a3f768640 1 -- 172.21.15.45:0/1798358587 wait complete. 2023-11-28T20:34:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.699+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7faf000057d0 con 0x7faf2015b850 2023-11-28T20:34:29.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.700+0000 7faf1cff9640 1 -- 172.21.15.45:0/1738190197 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faf18018090 con 0x7faf2015b850 2023-11-28T20:34:29.702 INFO:teuthology.orchestra.run.smithi045.stdout:55834574859 2023-11-28T20:34:29.704 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2023-11-28T20:34:29.704 DEBUG:teuthology.parallel:result is None 2023-11-28T20:34:29.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faf0c06af40 msgr2=0x7faf0c06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faf0c06af40 0x7faf0c06d400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faf10007fd0 tx=0x7faf10009040 comp rx=0 tx=0).stop 2023-11-28T20:34:29.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 msgr2=0x7faf201e4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 0x7faf201e4990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faf1800efc0 tx=0x7faf1800c490 comp rx=0 tx=0).stop 2023-11-28T20:34:29.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 shutdown_connections 2023-11-28T20:34:29.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faf0c06af40 0x7faf0c06d400 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf2015b850 0x7faf201e4990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf200b35b0 0x7faf201450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 --2- 172.21.15.45:0/1738190197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf200b1fa0 0x7faf200adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.703+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 >> 172.21.15.45:0/1738190197 conn(0x7faf2006a500 msgr2=0x7faf2015d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.1 2023-11-28T20:34:29.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.704+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 shutdown_connections 2023-11-28T20:34:29.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.704+0000 7faf26961640 1 -- 172.21.15.45:0/1738190197 wait complete. 2023-11-28T20:34:29.710 DEBUG:teuthology.parallel:result is None 2023-11-28T20:34:29.711 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2023-11-28T20:34:29.711 DEBUG:teuthology.parallel:result is None 2023-11-28T20:34:29.712 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.5 2023-11-28T20:34:29.712 DEBUG:teuthology.parallel:result is None 2023-11-28T20:34:29.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2023-11-28T20:34:29.717 DEBUG:teuthology.parallel:result is None 2023-11-28T20:34:29.719 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.6 2023-11-28T20:34:29.720 DEBUG:teuthology.parallel:result is None 2023-11-28T20:34:29.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.3 2023-11-28T20:34:29.722 DEBUG:teuthology.parallel:result is None 2023-11-28T20:34:29.723 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T20:34:29.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:34:29.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.979+0000 7f42c98ca640 1 Processor -- start 2023-11-28T20:34:29.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.979+0000 7f42c98ca640 1 -- start start 2023-11-28T20:34:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.981+0000 7f42c98ca640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 0x7f42c40abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.981+0000 7f42c98ca640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f42c40ae540 0x7f42c40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.981+0000 7f42c98ca640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c415c3c0 con 0x7f42c40ce630 2023-11-28T20:34:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.981+0000 7f42c98ca640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c40ceac0 con 0x7f42c40ae540 2023-11-28T20:34:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.981+0000 7f42c98ca640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c4155b60 con 0x7f42c4155780 2023-11-28T20:34:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.981+0000 7f42c3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f42c40ae540 0x7f42c40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.981+0000 7f42c90c9640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f42c40ce630 0x7f42c415ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51932/0 (socket says 172.21.15.45:51932) 2023-11-28T20:34:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.981+0000 7f42c90c9640 1 -- 172.21.15.45:0/79889516 learned_addr learned my addr 172.21.15.45:0/79889516 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.981+0000 7f42c88c8640 1 --2- 172.21.15.45:0/79889516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 0x7f42c40abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.982+0000 7f42c3fff640 1 -- 172.21.15.45:0/79889516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 msgr2=0x7f42c40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.982+0000 7f42c3fff640 1 --2- 172.21.15.45:0/79889516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 0x7f42c40abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.982+0000 7f42c3fff640 1 -- 172.21.15.45:0/79889516 >> v1:172.21.15.45:6789/0 conn(0x7f42c40ce630 legacy=0x7f42c415ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.982+0000 7f42c3fff640 1 -- 172.21.15.45:0/79889516 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c415ba90 con 0x7f42c40ae540 2023-11-28T20:34:29.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.983+0000 7f42c3fff640 1 --2- 172.21.15.45:0/79889516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ae540 0x7f42c40ce0f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f42b00097d0 tx=0x7f42b002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8237dd783ab0493 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.983+0000 7f42c37fe640 1 -- 172.21.15.45:0/79889516 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b002d040 con 0x7f42c40ae540 2023-11-28T20:34:29.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.983+0000 7f42c37fe640 1 -- 172.21.15.45:0/79889516 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b0031810 con 0x7f42c40ae540 2023-11-28T20:34:29.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.983+0000 7f42c37fe640 1 -- 172.21.15.45:0/79889516 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0031ba0 con 0x7f42c40ae540 2023-11-28T20:34:29.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.984+0000 7f42c98ca640 1 -- 172.21.15.45:0/79889516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ae540 msgr2=0x7f42c40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.984+0000 7f42c98ca640 1 --2- 172.21.15.45:0/79889516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ae540 0x7f42c40ce0f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f42b00097d0 tx=0x7f42b002b8b0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.984+0000 7f42c98ca640 1 -- 172.21.15.45:0/79889516 shutdown_connections 2023-11-28T20:34:29.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.984+0000 7f42c98ca640 1 --2- 172.21.15.45:0/79889516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ae540 0x7f42c40ce0f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.984+0000 7f42c98ca640 1 --2- 172.21.15.45:0/79889516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 0x7f42c40abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.984+0000 7f42c98ca640 1 -- 172.21.15.45:0/79889516 >> 172.21.15.45:0/79889516 conn(0x7f42c406ab30 msgr2=0x7f42c40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:29.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.984+0000 7f42c98ca640 1 -- 172.21.15.45:0/79889516 shutdown_connections 2023-11-28T20:34:29.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.984+0000 7f42c98ca640 1 -- 172.21.15.45:0/79889516 wait complete. 2023-11-28T20:34:29.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.985+0000 7f42c98ca640 1 Processor -- start 2023-11-28T20:34:29.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.986+0000 7f42c98ca640 1 -- start start 2023-11-28T20:34:29.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.986+0000 7f42c98ca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42c40ae540 0x7f42c41f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.986+0000 7f42c98ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ce630 0x7f42c4140630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c98ca640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 0x7f42c41464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c98ca640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c41469f0 con 0x7f42c40ce630 2023-11-28T20:34:29.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c98ca640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c41f70c0 con 0x7f42c40ae540 2023-11-28T20:34:29.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c98ca640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c41f6820 con 0x7f42c4155780 2023-11-28T20:34:29.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ce630 0x7f42c4140630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ce630 0x7f42c4140630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59076/0 (socket says 172.21.15.45:59076) 2023-11-28T20:34:29.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c3fff640 1 -- 172.21.15.45:0/3600766077 learned_addr learned my addr 172.21.15.45:0/3600766077 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:29.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c88c8640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42c40ae540 0x7f42c41f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c90c9640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 0x7f42c41464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c3fff640 1 -- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 msgr2=0x7f42c41464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.987+0000 7f42c3fff640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 0x7f42c41464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c3fff640 1 -- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42c40ae540 msgr2=0x7f42c41f61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c3fff640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42c40ae540 0x7f42c41f61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c3fff640 1 -- 172.21.15.45:0/3600766077 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b0009400 con 0x7f42c40ce630 2023-11-28T20:34:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c90c9640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 0x7f42c41464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c3fff640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ce630 0x7f42c4140630 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f42b0000c00 tx=0x7f42b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c17fa640 1 -- 172.21.15.45:0/3600766077 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b003ad60 con 0x7f42c40ce630 2023-11-28T20:34:29.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c17fa640 1 -- 172.21.15.45:0/3600766077 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b00443f0 con 0x7f42c40ce630 2023-11-28T20:34:29.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c17fa640 1 -- 172.21.15.45:0/3600766077 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b00447a0 con 0x7f42c40ce630 2023-11-28T20:34:29.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c41f6b00 con 0x7f42c40ce630 2023-11-28T20:34:29.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.988+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c409ab30 con 0x7f42c40ce630 2023-11-28T20:34:29.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.989+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c404e920 con 0x7f42c40ce630 2023-11-28T20:34:29.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.990+0000 7f42c17fa640 1 -- 172.21.15.45:0/3600766077 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f42b004d450 con 0x7f42c40ce630 2023-11-28T20:34:29.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.991+0000 7f42c17fa640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f42a806ae70 0x7f42a806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:29.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.991+0000 7f42c17fa640 1 -- 172.21.15.45:0/3600766077 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f42b00abab0 con 0x7f42c40ce630 2023-11-28T20:34:29.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.992+0000 7f42c88c8640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f42a806ae70 0x7f42a806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:29.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.993+0000 7f42c88c8640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f42a806ae70 0x7f42a806d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f42b8006fd0 tx=0x7f42b8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:29.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:29.995+0000 7f42c17fa640 1 -- 172.21.15.45:0/3600766077 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42b0076860 con 0x7f42c40ce630 2023-11-28T20:34:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.189+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f42c415b740 con 0x7f42a806ae70 2023-11-28T20:34:30.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.215+0000 7f42c17fa640 1 -- 172.21.15.45:0/3600766077 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345032 (secure 0 0 0) 0x7f42b8010990 con 0x7f42a806ae70 2023-11-28T20:34:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:30.224 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:34:30.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f42a806ae70 msgr2=0x7f42a806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:30.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f42a806ae70 0x7f42a806d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f42b8006fd0 tx=0x7f42b8008040 comp rx=0 tx=0).stop 2023-11-28T20:34:30.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ce630 msgr2=0x7f42c4140630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:30.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ce630 0x7f42c4140630 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f42b0000c00 tx=0x7f42b00046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 shutdown_connections 2023-11-28T20:34:30.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f42a806ae70 0x7f42a806d330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42c4155780 0x7f42c41464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42c40ce630 0x7f42c4140630 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 --2- 172.21.15.45:0/3600766077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42c40ae540 0x7f42c41f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 >> 172.21.15.45:0/3600766077 conn(0x7f42c406ab30 msgr2=0x7f42c40b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:30.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 shutdown_connections 2023-11-28T20:34:30.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.221+0000 7f42c98ca640 1 -- 172.21.15.45:0/3600766077 wait complete. 2023-11-28T20:34:30.242 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2023-11-28T20:34:29.136780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236124,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738596,"statfs":{"total":773094113280,"available":772852322304,"internally_reserved":0,"allocated":3383296,"data_stored":2025718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005628"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119305+0000","last_change":"2023-11-28T20:33:53.064970+0000","last_active":"2023-11-28T20:33:58.119305+0000","last_peered":"2023-11-28T20:33:58.119305+0000","last_clean":"2023-11-28T20:33:58.119305+0000","last_became_active":"2023-11-28T20:33:53.064659+0000","last_became_peered":"2023-11-28T20:33:53.064659+0000","last_unstale":"2023-11-28T20:33:58.119305+0000","last_undegraded":"2023-11-28T20:33:58.119305+0000","last_fullsized":"2023-11-28T20:33:58.119305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.109629+0000","last_change":"2023-11-28T20:33:53.948203+0000","last_active":"2023-11-28T20:33:58.109629+0000","last_peered":"2023-11-28T20:33:58.109629+0000","last_clean":"2023-11-28T20:33:58.109629+0000","last_became_active":"2023-11-28T20:33:53.947906+0000","last_became_peered":"2023-11-28T20:33:53.947906+0000","last_unstale":"2023-11-28T20:33:58.109629+0000","last_undegraded":"2023-11-28T20:33:58.109629+0000","last_fullsized":"2023-11-28T20:33:58.109629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918353+0000","last_change":"2023-11-28T20:33:58.158488+0000","last_active":"2023-11-28T20:33:58.918353+0000","last_peered":"2023-11-28T20:33:58.918353+0000","last_clean":"2023-11-28T20:33:58.918353+0000","last_became_active":"2023-11-28T20:33:58.158046+0000","last_became_peered":"2023-11-28T20:33:58.158046+0000","last_unstale":"2023-11-28T20:33:58.918353+0000","last_undegraded":"2023-11-28T20:33:58.918353+0000","last_fullsized":"2023-11-28T20:33:58.918353+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:20:48.980178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.111851+0000","last_change":"2023-11-28T20:33:53.050283+0000","last_active":"2023-11-28T20:33:58.111851+0000","last_peered":"2023-11-28T20:33:58.111851+0000","last_clean":"2023-11-28T20:33:58.111851+0000","last_became_active":"2023-11-28T20:33:53.049963+0000","last_became_peered":"2023-11-28T20:33:53.049963+0000","last_unstale":"2023-11-28T20:33:58.111851+0000","last_undegraded":"2023-11-28T20:33:58.111851+0000","last_fullsized":"2023-11-28T20:33:58.111851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116825+0000","last_change":"2023-11-28T20:33:53.943001+0000","last_active":"2023-11-28T20:33:58.116825+0000","last_peered":"2023-11-28T20:33:58.116825+0000","last_clean":"2023-11-28T20:33:58.116825+0000","last_became_active":"2023-11-28T20:33:53.940632+0000","last_became_peered":"2023-11-28T20:33:53.940632+0000","last_unstale":"2023-11-28T20:33:58.116825+0000","last_undegraded":"2023-11-28T20:33:58.116825+0000","last_fullsized":"2023-11-28T20:33:58.116825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916311+0000","last_change":"2023-11-28T20:33:58.127527+0000","last_active":"2023-11-28T20:33:58.916311+0000","last_peered":"2023-11-28T20:33:58.916311+0000","last_clean":"2023-11-28T20:33:58.916311+0000","last_became_active":"2023-11-28T20:33:58.127058+0000","last_became_peered":"2023-11-28T20:33:58.127058+0000","last_unstale":"2023-11-28T20:33:58.916311+0000","last_undegraded":"2023-11-28T20:33:58.916311+0000","last_fullsized":"2023-11-28T20:33:58.916311+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:12.932976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151962+0000","last_change":"2023-11-28T20:33:53.073019+0000","last_active":"2023-11-28T20:33:58.151962+0000","last_peered":"2023-11-28T20:33:58.151962+0000","last_clean":"2023-11-28T20:33:58.151962+0000","last_became_active":"2023-11-28T20:33:53.072728+0000","last_became_peered":"2023-11-28T20:33:53.072728+0000","last_unstale":"2023-11-28T20:33:58.151962+0000","last_undegraded":"2023-11-28T20:33:58.151962+0000","last_fullsized":"2023-11-28T20:33:58.151962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122157+0000","last_change":"2023-11-28T20:33:53.951054+0000","last_active":"2023-11-28T20:33:58.122157+0000","last_peered":"2023-11-28T20:33:58.122157+0000","last_clean":"2023-11-28T20:33:58.122157+0000","last_became_active":"2023-11-28T20:33:53.950730+0000","last_became_peered":"2023-11-28T20:33:53.950730+0000","last_unstale":"2023-11-28T20:33:58.122157+0000","last_undegraded":"2023-11-28T20:33:58.122157+0000","last_fullsized":"2023-11-28T20:33:58.122157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:19.573371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916721+0000","last_change":"2023-11-28T20:33:58.131034+0000","last_active":"2023-11-28T20:33:58.916721+0000","last_peered":"2023-11-28T20:33:58.916721+0000","last_clean":"2023-11-28T20:33:58.916721+0000","last_became_active":"2023-11-28T20:33:58.129161+0000","last_became_peered":"2023-11-28T20:33:58.129161+0000","last_unstale":"2023-11-28T20:33:58.916721+0000","last_undegraded":"2023-11-28T20:33:58.916721+0000","last_fullsized":"2023-11-28T20:33:58.916721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:19.781603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151744+0000","last_change":"2023-11-28T20:33:53.068043+0000","last_active":"2023-11-28T20:33:58.151744+0000","last_peered":"2023-11-28T20:33:58.151744+0000","last_clean":"2023-11-28T20:33:58.151744+0000","last_became_active":"2023-11-28T20:33:53.067650+0000","last_became_peered":"2023-11-28T20:33:53.067650+0000","last_unstale":"2023-11-28T20:33:58.151744+0000","last_undegraded":"2023-11-28T20:33:58.151744+0000","last_fullsized":"2023-11-28T20:33:58.151744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123332+0000","last_change":"2023-11-28T20:33:53.954109+0000","last_active":"2023-11-28T20:33:58.123332+0000","last_peered":"2023-11-28T20:33:58.123332+0000","last_clean":"2023-11-28T20:33:58.123332+0000","last_became_active":"2023-11-28T20:33:53.953749+0000","last_became_peered":"2023-11-28T20:33:53.953749+0000","last_unstale":"2023-11-28T20:33:58.123332+0000","last_undegraded":"2023-11-28T20:33:58.123332+0000","last_fullsized":"2023-11-28T20:33:58.123332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:21.394618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916961+0000","last_change":"2023-11-28T20:33:58.132306+0000","last_active":"2023-11-28T20:33:58.916961+0000","last_peered":"2023-11-28T20:33:58.916961+0000","last_clean":"2023-11-28T20:33:58.916961+0000","last_became_active":"2023-11-28T20:33:58.131782+0000","last_became_peered":"2023-11-28T20:33:58.131782+0000","last_unstale":"2023-11-28T20:33:58.916961+0000","last_undegraded":"2023-11-28T20:33:58.916961+0000","last_fullsized":"2023-11-28T20:33:58.916961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:30.608189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121351+0000","last_change":"2023-11-28T20:33:53.954276+0000","last_active":"2023-11-28T20:33:58.121351+0000","last_peered":"2023-11-28T20:33:58.121351+0000","last_clean":"2023-11-28T20:33:58.121351+0000","last_became_active":"2023-11-28T20:33:53.953993+0000","last_became_peered":"2023-11-28T20:33:53.953993+0000","last_unstale":"2023-11-28T20:33:58.121351+0000","last_undegraded":"2023-11-28T20:33:58.121351+0000","last_fullsized":"2023-11-28T20:33:58.121351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151814+0000","last_change":"2023-11-28T20:33:53.075863+0000","last_active":"2023-11-28T20:33:58.151814+0000","last_peered":"2023-11-28T20:33:58.151814+0000","last_clean":"2023-11-28T20:33:58.151814+0000","last_became_active":"2023-11-28T20:33:53.075523+0000","last_became_peered":"2023-11-28T20:33:53.075523+0000","last_unstale":"2023-11-28T20:33:58.151814+0000","last_undegraded":"2023-11-28T20:33:58.151814+0000","last_fullsized":"2023-11-28T20:33:58.151814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917964+0000","last_change":"2023-11-28T20:33:58.130991+0000","last_active":"2023-11-28T20:33:58.917964+0000","last_peered":"2023-11-28T20:33:58.917964+0000","last_clean":"2023-11-28T20:33:58.917964+0000","last_became_active":"2023-11-28T20:33:58.130377+0000","last_became_peered":"2023-11-28T20:33:58.130377+0000","last_unstale":"2023-11-28T20:33:58.917964+0000","last_undegraded":"2023-11-28T20:33:58.917964+0000","last_fullsized":"2023-11-28T20:33:58.917964+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103760+0000","last_change":"2023-11-28T20:33:53.943400+0000","last_active":"2023-11-28T20:33:58.103760+0000","last_peered":"2023-11-28T20:33:58.103760+0000","last_clean":"2023-11-28T20:33:58.103760+0000","last_became_active":"2023-11-28T20:33:53.937673+0000","last_became_peered":"2023-11-28T20:33:53.937673+0000","last_unstale":"2023-11-28T20:33:58.103760+0000","last_undegraded":"2023-11-28T20:33:58.103760+0000","last_fullsized":"2023-11-28T20:33:58.103760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125979+0000","last_change":"2023-11-28T20:33:53.062456+0000","last_active":"2023-11-28T20:33:58.125979+0000","last_peered":"2023-11-28T20:33:58.125979+0000","last_clean":"2023-11-28T20:33:58.125979+0000","last_became_active":"2023-11-28T20:33:53.062163+0000","last_became_peered":"2023-11-28T20:33:53.062163+0000","last_unstale":"2023-11-28T20:33:58.125979+0000","last_undegraded":"2023-11-28T20:33:58.125979+0000","last_fullsized":"2023-11-28T20:33:58.125979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917431+0000","last_change":"2023-11-28T20:33:58.137561+0000","last_active":"2023-11-28T20:33:58.917431+0000","last_peered":"2023-11-28T20:33:58.917431+0000","last_clean":"2023-11-28T20:33:58.917431+0000","last_became_active":"2023-11-28T20:33:58.137065+0000","last_became_peered":"2023-11-28T20:33:58.137065+0000","last_unstale":"2023-11-28T20:33:58.917431+0000","last_undegraded":"2023-11-28T20:33:58.917431+0000","last_fullsized":"2023-11-28T20:33:58.917431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103501+0000","last_change":"2023-11-28T20:33:53.940322+0000","last_active":"2023-11-28T20:33:58.103501+0000","last_peered":"2023-11-28T20:33:58.103501+0000","last_clean":"2023-11-28T20:33:58.103501+0000","last_became_active":"2023-11-28T20:33:53.939986+0000","last_became_peered":"2023-11-28T20:33:53.939986+0000","last_unstale":"2023-11-28T20:33:58.103501+0000","last_undegraded":"2023-11-28T20:33:58.103501+0000","last_fullsized":"2023-11-28T20:33:58.103501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123391+0000","last_change":"2023-11-28T20:33:53.062082+0000","last_active":"2023-11-28T20:33:58.123391+0000","last_peered":"2023-11-28T20:33:58.123391+0000","last_clean":"2023-11-28T20:33:58.123391+0000","last_became_active":"2023-11-28T20:33:53.060583+0000","last_became_peered":"2023-11-28T20:33:53.060583+0000","last_unstale":"2023-11-28T20:33:58.123391+0000","last_undegraded":"2023-11-28T20:33:58.123391+0000","last_fullsized":"2023-11-28T20:33:58.123391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916613+0000","last_change":"2023-11-28T20:33:58.158289+0000","last_active":"2023-11-28T20:33:58.916613+0000","last_peered":"2023-11-28T20:33:58.916613+0000","last_clean":"2023-11-28T20:33:58.916613+0000","last_became_active":"2023-11-28T20:33:58.157856+0000","last_became_peered":"2023-11-28T20:33:58.157856+0000","last_unstale":"2023-11-28T20:33:58.916613+0000","last_undegraded":"2023-11-28T20:33:58.916613+0000","last_fullsized":"2023-11-28T20:33:58.916613+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:17.807344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.112802+0000","last_change":"2023-11-28T20:33:53.942280+0000","last_active":"2023-11-28T20:33:58.112802+0000","last_peered":"2023-11-28T20:33:58.112802+0000","last_clean":"2023-11-28T20:33:58.112802+0000","last_became_active":"2023-11-28T20:33:53.939778+0000","last_became_peered":"2023-11-28T20:33:53.939778+0000","last_unstale":"2023-11-28T20:33:58.112802+0000","last_undegraded":"2023-11-28T20:33:58.112802+0000","last_fullsized":"2023-11-28T20:33:58.112802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:33.050269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151419+0000","last_change":"2023-11-28T20:33:53.068971+0000","last_active":"2023-11-28T20:33:58.151419+0000","last_peered":"2023-11-28T20:33:58.151419+0000","last_clean":"2023-11-28T20:33:58.151419+0000","last_became_active":"2023-11-28T20:33:53.068069+0000","last_became_peered":"2023-11-28T20:33:53.068069+0000","last_unstale":"2023-11-28T20:33:58.151419+0000","last_undegraded":"2023-11-28T20:33:58.151419+0000","last_fullsized":"2023-11-28T20:33:58.151419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:41.371596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917429+0000","last_change":"2023-11-28T20:33:58.137597+0000","last_active":"2023-11-28T20:33:58.917429+0000","last_peered":"2023-11-28T20:33:58.917429+0000","last_clean":"2023-11-28T20:33:58.917429+0000","last_became_active":"2023-11-28T20:33:58.137078+0000","last_became_peered":"2023-11-28T20:33:58.137078+0000","last_unstale":"2023-11-28T20:33:58.917429+0000","last_undegraded":"2023-11-28T20:33:58.917429+0000","last_fullsized":"2023-11-28T20:33:58.917429+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:23.642548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120275+0000","last_change":"2023-11-28T20:33:53.945506+0000","last_active":"2023-11-28T20:33:58.120275+0000","last_peered":"2023-11-28T20:33:58.120275+0000","last_clean":"2023-11-28T20:33:58.120275+0000","last_became_active":"2023-11-28T20:33:53.945153+0000","last_became_peered":"2023-11-28T20:33:53.945153+0000","last_unstale":"2023-11-28T20:33:58.120275+0000","last_undegraded":"2023-11-28T20:33:58.120275+0000","last_fullsized":"2023-11-28T20:33:58.120275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126982+0000","last_change":"2023-11-28T20:33:53.064943+0000","last_active":"2023-11-28T20:33:58.126982+0000","last_peered":"2023-11-28T20:33:58.126982+0000","last_clean":"2023-11-28T20:33:58.126982+0000","last_became_active":"2023-11-28T20:33:53.064644+0000","last_became_peered":"2023-11-28T20:33:53.064644+0000","last_unstale":"2023-11-28T20:33:58.126982+0000","last_undegraded":"2023-11-28T20:33:58.126982+0000","last_fullsized":"2023-11-28T20:33:58.126982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916509+0000","last_change":"2023-11-28T20:33:58.158266+0000","last_active":"2023-11-28T20:33:58.916509+0000","last_peered":"2023-11-28T20:33:58.916509+0000","last_clean":"2023-11-28T20:33:58.916509+0000","last_became_active":"2023-11-28T20:33:58.157836+0000","last_became_peered":"2023-11-28T20:33:58.157836+0000","last_unstale":"2023-11-28T20:33:58.916509+0000","last_undegraded":"2023-11-28T20:33:58.916509+0000","last_fullsized":"2023-11-28T20:33:58.916509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:44.771180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.114503+0000","last_change":"2023-11-28T20:33:53.941572+0000","last_active":"2023-11-28T20:33:58.114503+0000","last_peered":"2023-11-28T20:33:58.114503+0000","last_clean":"2023-11-28T20:33:58.114503+0000","last_became_active":"2023-11-28T20:33:53.941245+0000","last_became_peered":"2023-11-28T20:33:53.941245+0000","last_unstale":"2023-11-28T20:33:58.114503+0000","last_undegraded":"2023-11-28T20:33:58.114503+0000","last_fullsized":"2023-11-28T20:33:58.114503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123698+0000","last_change":"2023-11-28T20:33:53.058678+0000","last_active":"2023-11-28T20:33:58.123698+0000","last_peered":"2023-11-28T20:33:58.123698+0000","last_clean":"2023-11-28T20:33:58.123698+0000","last_became_active":"2023-11-28T20:33:53.058375+0000","last_became_peered":"2023-11-28T20:33:53.058375+0000","last_unstale":"2023-11-28T20:33:58.123698+0000","last_undegraded":"2023-11-28T20:33:58.123698+0000","last_fullsized":"2023-11-28T20:33:58.123698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:12.793548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918259+0000","last_change":"2023-11-28T20:33:58.154128+0000","last_active":"2023-11-28T20:33:58.918259+0000","last_peered":"2023-11-28T20:33:58.918259+0000","last_clean":"2023-11-28T20:33:58.918259+0000","last_became_active":"2023-11-28T20:33:58.153697+0000","last_became_peered":"2023-11-28T20:33:58.153697+0000","last_unstale":"2023-11-28T20:33:58.918259+0000","last_undegraded":"2023-11-28T20:33:58.918259+0000","last_fullsized":"2023-11-28T20:33:58.918259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:47.677895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123582+0000","last_change":"2023-11-28T20:33:53.941553+0000","last_active":"2023-11-28T20:33:58.123582+0000","last_peered":"2023-11-28T20:33:58.123582+0000","last_clean":"2023-11-28T20:33:58.123582+0000","last_became_active":"2023-11-28T20:33:53.941248+0000","last_became_peered":"2023-11-28T20:33:53.941248+0000","last_unstale":"2023-11-28T20:33:58.123582+0000","last_undegraded":"2023-11-28T20:33:58.123582+0000","last_fullsized":"2023-11-28T20:33:58.123582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152051+0000","last_change":"2023-11-28T20:33:53.069890+0000","last_active":"2023-11-28T20:33:58.152051+0000","last_peered":"2023-11-28T20:33:58.152051+0000","last_clean":"2023-11-28T20:33:58.152051+0000","last_became_active":"2023-11-28T20:33:53.069572+0000","last_became_peered":"2023-11-28T20:33:53.069572+0000","last_unstale":"2023-11-28T20:33:58.152051+0000","last_undegraded":"2023-11-28T20:33:58.152051+0000","last_fullsized":"2023-11-28T20:33:58.152051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:57.471000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589761+0000","last_change":"2023-11-28T20:33:58.157661+0000","last_active":"2023-11-28T20:33:59.589761+0000","last_peered":"2023-11-28T20:33:59.589761+0000","last_clean":"2023-11-28T20:33:59.589761+0000","last_became_active":"2023-11-28T20:33:58.157072+0000","last_became_peered":"2023-11-28T20:33:58.157072+0000","last_unstale":"2023-11-28T20:33:59.589761+0000","last_undegraded":"2023-11-28T20:33:59.589761+0000","last_fullsized":"2023-11-28T20:33:59.589761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:11.495353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123465+0000","last_change":"2023-11-28T20:33:53.947235+0000","last_active":"2023-11-28T20:33:58.123465+0000","last_peered":"2023-11-28T20:33:58.123465+0000","last_clean":"2023-11-28T20:33:58.123465+0000","last_became_active":"2023-11-28T20:33:53.946977+0000","last_became_peered":"2023-11-28T20:33:53.946977+0000","last_unstale":"2023-11-28T20:33:58.123465+0000","last_undegraded":"2023-11-28T20:33:58.123465+0000","last_fullsized":"2023-11-28T20:33:58.123465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:08.318223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123720+0000","last_change":"2023-11-28T20:33:53.060664+0000","last_active":"2023-11-28T20:33:58.123720+0000","last_peered":"2023-11-28T20:33:58.123720+0000","last_clean":"2023-11-28T20:33:58.123720+0000","last_became_active":"2023-11-28T20:33:53.058196+0000","last_became_peered":"2023-11-28T20:33:53.058196+0000","last_unstale":"2023-11-28T20:33:58.123720+0000","last_undegraded":"2023-11-28T20:33:58.123720+0000","last_fullsized":"2023-11-28T20:33:58.123720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.915328+0000","last_change":"2023-11-28T20:33:58.133412+0000","last_active":"2023-11-28T20:33:58.915328+0000","last_peered":"2023-11-28T20:33:58.915328+0000","last_clean":"2023-11-28T20:33:58.915328+0000","last_became_active":"2023-11-28T20:33:58.132991+0000","last_became_peered":"2023-11-28T20:33:58.132991+0000","last_unstale":"2023-11-28T20:33:58.915328+0000","last_undegraded":"2023-11-28T20:33:58.915328+0000","last_fullsized":"2023-11-28T20:33:58.915328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.107225+0000","last_change":"2023-11-28T20:33:53.942153+0000","last_active":"2023-11-28T20:33:58.107225+0000","last_peered":"2023-11-28T20:33:58.107225+0000","last_clean":"2023-11-28T20:33:58.107225+0000","last_became_active":"2023-11-28T20:33:53.941788+0000","last_became_peered":"2023-11-28T20:33:53.941788+0000","last_unstale":"2023-11-28T20:33:58.107225+0000","last_undegraded":"2023-11-28T20:33:58.107225+0000","last_fullsized":"2023-11-28T20:33:58.107225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:47.311066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126159+0000","last_change":"2023-11-28T20:33:53.058321+0000","last_active":"2023-11-28T20:33:58.126159+0000","last_peered":"2023-11-28T20:33:58.126159+0000","last_clean":"2023-11-28T20:33:58.126159+0000","last_became_active":"2023-11-28T20:33:53.058170+0000","last_became_peered":"2023-11-28T20:33:53.058170+0000","last_unstale":"2023-11-28T20:33:58.126159+0000","last_undegraded":"2023-11-28T20:33:58.126159+0000","last_fullsized":"2023-11-28T20:33:58.126159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:12.793548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917495+0000","last_change":"2023-11-28T20:33:58.138024+0000","last_active":"2023-11-28T20:33:58.917495+0000","last_peered":"2023-11-28T20:33:58.917495+0000","last_clean":"2023-11-28T20:33:58.917495+0000","last_became_active":"2023-11-28T20:33:58.137601+0000","last_became_peered":"2023-11-28T20:33:58.137601+0000","last_unstale":"2023-11-28T20:33:58.917495+0000","last_undegraded":"2023-11-28T20:33:58.917495+0000","last_fullsized":"2023-11-28T20:33:58.917495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:45.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122244+0000","last_change":"2023-11-28T20:33:53.942125+0000","last_active":"2023-11-28T20:33:58.122244+0000","last_peered":"2023-11-28T20:33:58.122244+0000","last_clean":"2023-11-28T20:33:58.122244+0000","last_became_active":"2023-11-28T20:33:53.941741+0000","last_became_peered":"2023-11-28T20:33:53.941741+0000","last_unstale":"2023-11-28T20:33:58.122244+0000","last_undegraded":"2023-11-28T20:33:58.122244+0000","last_fullsized":"2023-11-28T20:33:58.122244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099678+0000","last_change":"2023-11-28T20:33:53.050707+0000","last_active":"2023-11-28T20:33:58.099678+0000","last_peered":"2023-11-28T20:33:58.099678+0000","last_clean":"2023-11-28T20:33:58.099678+0000","last_became_active":"2023-11-28T20:33:53.050403+0000","last_became_peered":"2023-11-28T20:33:53.050403+0000","last_unstale":"2023-11-28T20:33:58.099678+0000","last_undegraded":"2023-11-28T20:33:58.099678+0000","last_fullsized":"2023-11-28T20:33:58.099678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:32.036639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916922+0000","last_change":"2023-11-28T20:33:58.139720+0000","last_active":"2023-11-28T20:33:58.916922+0000","last_peered":"2023-11-28T20:33:58.916922+0000","last_clean":"2023-11-28T20:33:58.916922+0000","last_became_active":"2023-11-28T20:33:58.138372+0000","last_became_peered":"2023-11-28T20:33:58.138372+0000","last_unstale":"2023-11-28T20:33:58.916922+0000","last_undegraded":"2023-11-28T20:33:58.916922+0000","last_fullsized":"2023-11-28T20:33:58.916922+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:11.495353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117362+0000","last_change":"2023-11-28T20:33:53.934500+0000","last_active":"2023-11-28T20:33:58.117362+0000","last_peered":"2023-11-28T20:33:58.117362+0000","last_clean":"2023-11-28T20:33:58.117362+0000","last_became_active":"2023-11-28T20:33:53.933165+0000","last_became_peered":"2023-11-28T20:33:53.933165+0000","last_unstale":"2023-11-28T20:33:58.117362+0000","last_undegraded":"2023-11-28T20:33:58.117362+0000","last_fullsized":"2023-11-28T20:33:58.117362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:48.148924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151406+0000","last_change":"2023-11-28T20:33:53.069660+0000","last_active":"2023-11-28T20:33:58.151406+0000","last_peered":"2023-11-28T20:33:58.151406+0000","last_clean":"2023-11-28T20:33:58.151406+0000","last_became_active":"2023-11-28T20:33:53.069198+0000","last_became_peered":"2023-11-28T20:33:53.069198+0000","last_unstale":"2023-11-28T20:33:58.151406+0000","last_undegraded":"2023-11-28T20:33:58.151406+0000","last_fullsized":"2023-11-28T20:33:58.151406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:46.297436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590181+0000","last_change":"2023-11-28T20:33:58.162077+0000","last_active":"2023-11-28T20:33:59.590181+0000","last_peered":"2023-11-28T20:33:59.590181+0000","last_clean":"2023-11-28T20:33:59.590181+0000","last_became_active":"2023-11-28T20:33:58.159109+0000","last_became_peered":"2023-11-28T20:33:58.159109+0000","last_unstale":"2023-11-28T20:33:59.590181+0000","last_undegraded":"2023-11-28T20:33:59.590181+0000","last_fullsized":"2023-11-28T20:33:59.590181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:47.677895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103295+0000","last_change":"2023-11-28T20:33:53.952209+0000","last_active":"2023-11-28T20:33:58.103295+0000","last_peered":"2023-11-28T20:33:58.103295+0000","last_clean":"2023-11-28T20:33:58.103295+0000","last_became_active":"2023-11-28T20:33:53.949676+0000","last_became_peered":"2023-11-28T20:33:53.949676+0000","last_unstale":"2023-11-28T20:33:58.103295+0000","last_undegraded":"2023-11-28T20:33:58.103295+0000","last_fullsized":"2023-11-28T20:33:58.103295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121849+0000","last_change":"2023-11-28T20:33:53.053226+0000","last_active":"2023-11-28T20:33:58.121849+0000","last_peered":"2023-11-28T20:33:58.121849+0000","last_clean":"2023-11-28T20:33:58.121849+0000","last_became_active":"2023-11-28T20:33:53.052910+0000","last_became_peered":"2023-11-28T20:33:53.052910+0000","last_unstale":"2023-11-28T20:33:58.121849+0000","last_undegraded":"2023-11-28T20:33:58.121849+0000","last_fullsized":"2023-11-28T20:33:58.121849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:41.755062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":169,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151553+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:33:58.151553+0000","last_peered":"2023-11-28T20:33:58.151553+0000","last_clean":"2023-11-28T20:33:58.151553+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:33:58.151553+0000","last_undegraded":"2023-11-28T20:33:58.151553+0000","last_fullsized":"2023-11-28T20:33:58.151553+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589763+0000","last_change":"2023-11-28T20:33:58.157671+0000","last_active":"2023-11-28T20:33:59.589763+0000","last_peered":"2023-11-28T20:33:59.589763+0000","last_clean":"2023-11-28T20:33:59.589763+0000","last_became_active":"2023-11-28T20:33:58.157254+0000","last_became_peered":"2023-11-28T20:33:58.157254+0000","last_unstale":"2023-11-28T20:33:59.589763+0000","last_undegraded":"2023-11-28T20:33:59.589763+0000","last_fullsized":"2023-11-28T20:33:59.589763+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151876+0000","last_change":"2023-11-28T20:33:53.960256+0000","last_active":"2023-11-28T20:33:58.151876+0000","last_peered":"2023-11-28T20:33:58.151876+0000","last_clean":"2023-11-28T20:33:58.151876+0000","last_became_active":"2023-11-28T20:33:53.959635+0000","last_became_peered":"2023-11-28T20:33:53.959635+0000","last_unstale":"2023-11-28T20:33:58.151876+0000","last_undegraded":"2023-11-28T20:33:58.151876+0000","last_fullsized":"2023-11-28T20:33:58.151876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:08.318223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099297+0000","last_change":"2023-11-28T20:33:53.054253+0000","last_active":"2023-11-28T20:33:58.099297+0000","last_peered":"2023-11-28T20:33:58.099297+0000","last_clean":"2023-11-28T20:33:58.099297+0000","last_became_active":"2023-11-28T20:33:53.053912+0000","last_became_peered":"2023-11-28T20:33:53.053912+0000","last_unstale":"2023-11-28T20:33:58.099297+0000","last_undegraded":"2023-11-28T20:33:58.099297+0000","last_fullsized":"2023-11-28T20:33:58.099297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:53.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589287+0000","last_change":"2023-11-28T20:33:58.154204+0000","last_active":"2023-11-28T20:33:59.589287+0000","last_peered":"2023-11-28T20:33:59.589287+0000","last_clean":"2023-11-28T20:33:59.589287+0000","last_became_active":"2023-11-28T20:33:58.153585+0000","last_became_peered":"2023-11-28T20:33:58.153585+0000","last_unstale":"2023-11-28T20:33:59.589287+0000","last_undegraded":"2023-11-28T20:33:59.589287+0000","last_fullsized":"2023-11-28T20:33:59.589287+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:23.642548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122263+0000","last_change":"2023-11-28T20:33:53.950370+0000","last_active":"2023-11-28T20:33:58.122263+0000","last_peered":"2023-11-28T20:33:58.122263+0000","last_clean":"2023-11-28T20:33:58.122263+0000","last_became_active":"2023-11-28T20:33:53.946975+0000","last_became_peered":"2023-11-28T20:33:53.946975+0000","last_unstale":"2023-11-28T20:33:58.122263+0000","last_undegraded":"2023-11-28T20:33:58.122263+0000","last_fullsized":"2023-11-28T20:33:58.122263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:41.830889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123611+0000","last_change":"2023-11-28T20:33:53.062319+0000","last_active":"2023-11-28T20:33:58.123611+0000","last_peered":"2023-11-28T20:33:58.123611+0000","last_clean":"2023-11-28T20:33:58.123611+0000","last_became_active":"2023-11-28T20:33:53.062034+0000","last_became_peered":"2023-11-28T20:33:53.062034+0000","last_unstale":"2023-11-28T20:33:58.123611+0000","last_undegraded":"2023-11-28T20:33:58.123611+0000","last_fullsized":"2023-11-28T20:33:58.123611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590144+0000","last_change":"2023-11-28T20:33:58.129150+0000","last_active":"2023-11-28T20:33:59.590144+0000","last_peered":"2023-11-28T20:33:59.590144+0000","last_clean":"2023-11-28T20:33:59.590144+0000","last_became_active":"2023-11-28T20:33:58.128658+0000","last_became_peered":"2023-11-28T20:33:58.128658+0000","last_unstale":"2023-11-28T20:33:59.590144+0000","last_undegraded":"2023-11-28T20:33:59.590144+0000","last_fullsized":"2023-11-28T20:33:59.590144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:24.263445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152745+0000","last_change":"2023-11-28T20:33:53.964226+0000","last_active":"2023-11-28T20:33:58.152745+0000","last_peered":"2023-11-28T20:33:58.152745+0000","last_clean":"2023-11-28T20:33:58.152745+0000","last_became_active":"2023-11-28T20:33:53.963614+0000","last_became_peered":"2023-11-28T20:33:53.963614+0000","last_unstale":"2023-11-28T20:33:58.152745+0000","last_undegraded":"2023-11-28T20:33:58.152745+0000","last_fullsized":"2023-11-28T20:33:58.152745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:20.194268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099544+0000","last_change":"2023-11-28T20:33:53.049547+0000","last_active":"2023-11-28T20:33:58.099544+0000","last_peered":"2023-11-28T20:33:58.099544+0000","last_clean":"2023-11-28T20:33:58.099544+0000","last_became_active":"2023-11-28T20:33:53.049252+0000","last_became_peered":"2023-11-28T20:33:53.049252+0000","last_unstale":"2023-11-28T20:33:58.099544+0000","last_undegraded":"2023-11-28T20:33:58.099544+0000","last_fullsized":"2023-11-28T20:33:58.099544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:41.371596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.252105+0000","last_change":"2023-11-28T20:33:58.158216+0000","last_active":"2023-11-28T20:33:59.252105+0000","last_peered":"2023-11-28T20:33:59.252105+0000","last_clean":"2023-11-28T20:33:59.252105+0000","last_became_active":"2023-11-28T20:33:58.157621+0000","last_became_peered":"2023-11-28T20:33:58.157621+0000","last_unstale":"2023-11-28T20:33:59.252105+0000","last_undegraded":"2023-11-28T20:33:59.252105+0000","last_fullsized":"2023-11-28T20:33:59.252105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:43:13.638963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151918+0000","last_change":"2023-11-28T20:33:53.959472+0000","last_active":"2023-11-28T20:33:58.151918+0000","last_peered":"2023-11-28T20:33:58.151918+0000","last_clean":"2023-11-28T20:33:58.151918+0000","last_became_active":"2023-11-28T20:33:53.959110+0000","last_became_peered":"2023-11-28T20:33:53.959110+0000","last_unstale":"2023-11-28T20:33:58.151918+0000","last_undegraded":"2023-11-28T20:33:58.151918+0000","last_fullsized":"2023-11-28T20:33:58.151918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:41.830889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.104034+0000","last_change":"2023-11-28T20:33:53.047976+0000","last_active":"2023-11-28T20:33:58.104034+0000","last_peered":"2023-11-28T20:33:58.104034+0000","last_clean":"2023-11-28T20:33:58.104034+0000","last_became_active":"2023-11-28T20:33:53.047700+0000","last_became_peered":"2023-11-28T20:33:53.047700+0000","last_unstale":"2023-11-28T20:33:58.104034+0000","last_undegraded":"2023-11-28T20:33:58.104034+0000","last_fullsized":"2023-11-28T20:33:58.104034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:53.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916144+0000","last_change":"2023-11-28T20:33:58.128125+0000","last_active":"2023-11-28T20:33:58.916144+0000","last_peered":"2023-11-28T20:33:58.916144+0000","last_clean":"2023-11-28T20:33:58.916144+0000","last_became_active":"2023-11-28T20:33:58.127738+0000","last_became_peered":"2023-11-28T20:33:58.127738+0000","last_unstale":"2023-11-28T20:33:58.916144+0000","last_undegraded":"2023-11-28T20:33:58.916144+0000","last_fullsized":"2023-11-28T20:33:58.916144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:29.532639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151544+0000","last_change":"2023-11-28T20:33:53.960235+0000","last_active":"2023-11-28T20:33:58.151544+0000","last_peered":"2023-11-28T20:33:58.151544+0000","last_clean":"2023-11-28T20:33:58.151544+0000","last_became_active":"2023-11-28T20:33:53.959892+0000","last_became_peered":"2023-11-28T20:33:53.959892+0000","last_unstale":"2023-11-28T20:33:58.151544+0000","last_undegraded":"2023-11-28T20:33:58.151544+0000","last_fullsized":"2023-11-28T20:33:58.151544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099168+0000","last_change":"2023-11-28T20:33:53.056980+0000","last_active":"2023-11-28T20:33:58.099168+0000","last_peered":"2023-11-28T20:33:58.099168+0000","last_clean":"2023-11-28T20:33:58.099168+0000","last_became_active":"2023-11-28T20:33:53.056663+0000","last_became_peered":"2023-11-28T20:33:53.056663+0000","last_unstale":"2023-11-28T20:33:58.099168+0000","last_undegraded":"2023-11-28T20:33:58.099168+0000","last_fullsized":"2023-11-28T20:33:58.099168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:11:01.063288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590096+0000","last_change":"2023-11-28T20:33:58.137476+0000","last_active":"2023-11-28T20:33:59.590096+0000","last_peered":"2023-11-28T20:33:59.590096+0000","last_clean":"2023-11-28T20:33:59.590096+0000","last_became_active":"2023-11-28T20:33:58.136106+0000","last_became_peered":"2023-11-28T20:33:58.136106+0000","last_unstale":"2023-11-28T20:33:59.590096+0000","last_undegraded":"2023-11-28T20:33:59.590096+0000","last_fullsized":"2023-11-28T20:33:59.590096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:39.912968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119486+0000","last_change":"2023-11-28T20:33:53.940374+0000","last_active":"2023-11-28T20:33:58.119486+0000","last_peered":"2023-11-28T20:33:58.119486+0000","last_clean":"2023-11-28T20:33:58.119486+0000","last_became_active":"2023-11-28T20:33:53.939873+0000","last_became_peered":"2023-11-28T20:33:53.939873+0000","last_unstale":"2023-11-28T20:33:58.119486+0000","last_undegraded":"2023-11-28T20:33:58.119486+0000","last_fullsized":"2023-11-28T20:33:58.119486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120431+0000","last_change":"2023-11-28T20:33:53.055672+0000","last_active":"2023-11-28T20:33:58.120431+0000","last_peered":"2023-11-28T20:33:58.120431+0000","last_clean":"2023-11-28T20:33:58.120431+0000","last_became_active":"2023-11-28T20:33:53.055320+0000","last_became_peered":"2023-11-28T20:33:53.055320+0000","last_unstale":"2023-11-28T20:33:58.120431+0000","last_undegraded":"2023-11-28T20:33:58.120431+0000","last_fullsized":"2023-11-28T20:33:58.120431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.920861+0000","last_change":"2023-11-28T20:33:58.162629+0000","last_active":"2023-11-28T20:33:58.920861+0000","last_peered":"2023-11-28T20:33:58.920861+0000","last_clean":"2023-11-28T20:33:58.920861+0000","last_became_active":"2023-11-28T20:33:58.162003+0000","last_became_peered":"2023-11-28T20:33:58.162003+0000","last_unstale":"2023-11-28T20:33:58.920861+0000","last_undegraded":"2023-11-28T20:33:58.920861+0000","last_fullsized":"2023-11-28T20:33:58.920861+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:31.738218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151734+0000","last_change":"2023-11-28T20:33:53.961916+0000","last_active":"2023-11-28T20:33:58.151734+0000","last_peered":"2023-11-28T20:33:58.151734+0000","last_clean":"2023-11-28T20:33:58.151734+0000","last_became_active":"2023-11-28T20:33:53.961577+0000","last_became_peered":"2023-11-28T20:33:53.961577+0000","last_unstale":"2023-11-28T20:33:58.151734+0000","last_undegraded":"2023-11-28T20:33:58.151734+0000","last_fullsized":"2023-11-28T20:33:58.151734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126069+0000","last_change":"2023-11-28T20:33:53.056677+0000","last_active":"2023-11-28T20:33:58.126069+0000","last_peered":"2023-11-28T20:33:58.126069+0000","last_clean":"2023-11-28T20:33:58.126069+0000","last_became_active":"2023-11-28T20:33:53.055737+0000","last_became_peered":"2023-11-28T20:33:53.055737+0000","last_unstale":"2023-11-28T20:33:58.126069+0000","last_undegraded":"2023-11-28T20:33:58.126069+0000","last_fullsized":"2023-11-28T20:33:58.126069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.919758+0000","last_change":"2023-11-28T20:33:58.155270+0000","last_active":"2023-11-28T20:33:58.919758+0000","last_peered":"2023-11-28T20:33:58.919758+0000","last_clean":"2023-11-28T20:33:58.919758+0000","last_became_active":"2023-11-28T20:33:58.153764+0000","last_became_peered":"2023-11-28T20:33:58.153764+0000","last_unstale":"2023-11-28T20:33:58.919758+0000","last_undegraded":"2023-11-28T20:33:58.919758+0000","last_fullsized":"2023-11-28T20:33:58.919758+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:29.020740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117654+0000","last_change":"2023-11-28T20:33:53.943637+0000","last_active":"2023-11-28T20:33:58.117654+0000","last_peered":"2023-11-28T20:33:58.117654+0000","last_clean":"2023-11-28T20:33:58.117654+0000","last_became_active":"2023-11-28T20:33:53.941418+0000","last_became_peered":"2023-11-28T20:33:53.941418+0000","last_unstale":"2023-11-28T20:33:58.117654+0000","last_undegraded":"2023-11-28T20:33:58.117654+0000","last_fullsized":"2023-11-28T20:33:58.117654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118661+0000","last_change":"2023-11-28T20:33:53.052522+0000","last_active":"2023-11-28T20:33:58.118661+0000","last_peered":"2023-11-28T20:33:58.118661+0000","last_clean":"2023-11-28T20:33:58.118661+0000","last_became_active":"2023-11-28T20:33:53.052202+0000","last_became_peered":"2023-11-28T20:33:53.052202+0000","last_unstale":"2023-11-28T20:33:58.118661+0000","last_undegraded":"2023-11-28T20:33:58.118661+0000","last_fullsized":"2023-11-28T20:33:58.118661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:46.454394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.253371+0000","last_change":"2023-11-28T20:33:58.157854+0000","last_active":"2023-11-28T20:33:59.253371+0000","last_peered":"2023-11-28T20:33:59.253371+0000","last_clean":"2023-11-28T20:33:59.253371+0000","last_became_active":"2023-11-28T20:33:58.157424+0000","last_became_peered":"2023-11-28T20:33:58.157424+0000","last_unstale":"2023-11-28T20:33:59.253371+0000","last_undegraded":"2023-11-28T20:33:59.253371+0000","last_fullsized":"2023-11-28T20:33:59.253371+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:41.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151781+0000","last_change":"2023-11-28T20:33:53.959159+0000","last_active":"2023-11-28T20:33:58.151781+0000","last_peered":"2023-11-28T20:33:58.151781+0000","last_clean":"2023-11-28T20:33:58.151781+0000","last_became_active":"2023-11-28T20:33:53.958752+0000","last_became_peered":"2023-11-28T20:33:53.958752+0000","last_unstale":"2023-11-28T20:33:58.151781+0000","last_undegraded":"2023-11-28T20:33:58.151781+0000","last_fullsized":"2023-11-28T20:33:58.151781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:21.394618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:34:13.400104+0000","last_change":"2023-11-28T20:33:53.051304+0000","last_active":"2023-11-28T20:34:13.400104+0000","last_peered":"2023-11-28T20:34:13.400104+0000","last_clean":"2023-11-28T20:34:13.400104+0000","last_became_active":"2023-11-28T20:33:53.050957+0000","last_became_peered":"2023-11-28T20:33:53.050957+0000","last_unstale":"2023-11-28T20:34:13.400104+0000","last_undegraded":"2023-11-28T20:34:13.400104+0000","last_fullsized":"2023-11-28T20:34:13.400104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:41.755062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.915782+0000","last_change":"2023-11-28T20:33:58.133335+0000","last_active":"2023-11-28T20:33:58.915782+0000","last_peered":"2023-11-28T20:33:58.915782+0000","last_clean":"2023-11-28T20:33:58.915782+0000","last_became_active":"2023-11-28T20:33:58.132968+0000","last_became_peered":"2023-11-28T20:33:58.132968+0000","last_unstale":"2023-11-28T20:33:58.915782+0000","last_undegraded":"2023-11-28T20:33:58.915782+0000","last_fullsized":"2023-11-28T20:33:58.915782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105225+0000","last_change":"2023-11-28T20:33:53.944947+0000","last_active":"2023-11-28T20:33:58.105225+0000","last_peered":"2023-11-28T20:33:58.105225+0000","last_clean":"2023-11-28T20:33:58.105225+0000","last_became_active":"2023-11-28T20:33:53.944492+0000","last_became_peered":"2023-11-28T20:33:53.944492+0000","last_unstale":"2023-11-28T20:33:58.105225+0000","last_undegraded":"2023-11-28T20:33:58.105225+0000","last_fullsized":"2023-11-28T20:33:58.105225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:42.768692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123500+0000","last_change":"2023-11-28T20:33:53.063285+0000","last_active":"2023-11-28T20:33:58.123500+0000","last_peered":"2023-11-28T20:33:58.123500+0000","last_clean":"2023-11-28T20:33:58.123500+0000","last_became_active":"2023-11-28T20:33:53.063017+0000","last_became_peered":"2023-11-28T20:33:53.063017+0000","last_unstale":"2023-11-28T20:33:58.123500+0000","last_undegraded":"2023-11-28T20:33:58.123500+0000","last_fullsized":"2023-11-28T20:33:58.123500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917261+0000","last_change":"2023-11-28T20:33:58.125259+0000","last_active":"2023-11-28T20:33:58.917261+0000","last_peered":"2023-11-28T20:33:58.917261+0000","last_clean":"2023-11-28T20:33:58.917261+0000","last_became_active":"2023-11-28T20:33:58.124810+0000","last_became_peered":"2023-11-28T20:33:58.124810+0000","last_unstale":"2023-11-28T20:33:58.917261+0000","last_undegraded":"2023-11-28T20:33:58.917261+0000","last_fullsized":"2023-11-28T20:33:58.917261+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:22.122661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116919+0000","last_change":"2023-11-28T20:33:53.940927+0000","last_active":"2023-11-28T20:33:58.116919+0000","last_peered":"2023-11-28T20:33:58.116919+0000","last_clean":"2023-11-28T20:33:58.116919+0000","last_became_active":"2023-11-28T20:33:53.940527+0000","last_became_peered":"2023-11-28T20:33:53.940527+0000","last_unstale":"2023-11-28T20:33:58.116919+0000","last_undegraded":"2023-11-28T20:33:58.116919+0000","last_fullsized":"2023-11-28T20:33:58.116919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123801+0000","last_change":"2023-11-28T20:33:53.055804+0000","last_active":"2023-11-28T20:33:58.123801+0000","last_peered":"2023-11-28T20:33:58.123801+0000","last_clean":"2023-11-28T20:33:58.123801+0000","last_became_active":"2023-11-28T20:33:53.053184+0000","last_became_peered":"2023-11-28T20:33:53.053184+0000","last_unstale":"2023-11-28T20:33:58.123801+0000","last_undegraded":"2023-11-28T20:33:58.123801+0000","last_fullsized":"2023-11-28T20:33:58.123801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:57.471000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918702+0000","last_change":"2023-11-28T20:33:58.133079+0000","last_active":"2023-11-28T20:33:58.918702+0000","last_peered":"2023-11-28T20:33:58.918702+0000","last_clean":"2023-11-28T20:33:58.918702+0000","last_became_active":"2023-11-28T20:33:58.126051+0000","last_became_peered":"2023-11-28T20:33:58.126051+0000","last_unstale":"2023-11-28T20:33:58.918702+0000","last_undegraded":"2023-11-28T20:33:58.918702+0000","last_fullsized":"2023-11-28T20:33:58.918702+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:41.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123681+0000","last_change":"2023-11-28T20:33:53.958488+0000","last_active":"2023-11-28T20:33:58.123681+0000","last_peered":"2023-11-28T20:33:58.123681+0000","last_clean":"2023-11-28T20:33:58.123681+0000","last_became_active":"2023-11-28T20:33:53.958041+0000","last_became_peered":"2023-11-28T20:33:53.958041+0000","last_unstale":"2023-11-28T20:33:58.123681+0000","last_undegraded":"2023-11-28T20:33:58.123681+0000","last_fullsized":"2023-11-28T20:33:58.123681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:07.426176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126850+0000","last_change":"2023-11-28T20:33:53.054003+0000","last_active":"2023-11-28T20:33:58.126850+0000","last_peered":"2023-11-28T20:33:58.126850+0000","last_clean":"2023-11-28T20:33:58.126850+0000","last_became_active":"2023-11-28T20:33:53.053773+0000","last_became_peered":"2023-11-28T20:33:53.053773+0000","last_unstale":"2023-11-28T20:33:58.126850+0000","last_undegraded":"2023-11-28T20:33:58.126850+0000","last_fullsized":"2023-11-28T20:33:58.126850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918600+0000","last_change":"2023-11-28T20:33:58.157661+0000","last_active":"2023-11-28T20:33:58.918600+0000","last_peered":"2023-11-28T20:33:58.918600+0000","last_clean":"2023-11-28T20:33:58.918600+0000","last_became_active":"2023-11-28T20:33:58.157274+0000","last_became_peered":"2023-11-28T20:33:58.157274+0000","last_unstale":"2023-11-28T20:33:58.918600+0000","last_undegraded":"2023-11-28T20:33:58.918600+0000","last_fullsized":"2023-11-28T20:33:58.918600+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:43:13.638963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589987+0000","last_change":"2023-11-28T20:33:58.159599+0000","last_active":"2023-11-28T20:33:59.589987+0000","last_peered":"2023-11-28T20:33:59.589987+0000","last_clean":"2023-11-28T20:33:59.589987+0000","last_became_active":"2023-11-28T20:33:58.158483+0000","last_became_peered":"2023-11-28T20:33:58.158483+0000","last_unstale":"2023-11-28T20:33:59.589987+0000","last_undegraded":"2023-11-28T20:33:59.589987+0000","last_fullsized":"2023-11-28T20:33:59.589987+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:45.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122123+0000","last_change":"2023-11-28T20:33:53.950947+0000","last_active":"2023-11-28T20:33:58.122123+0000","last_peered":"2023-11-28T20:33:58.122123+0000","last_clean":"2023-11-28T20:33:58.122123+0000","last_became_active":"2023-11-28T20:33:53.948976+0000","last_became_peered":"2023-11-28T20:33:53.948976+0000","last_unstale":"2023-11-28T20:33:58.122123+0000","last_undegraded":"2023-11-28T20:33:58.122123+0000","last_fullsized":"2023-11-28T20:33:58.122123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120814+0000","last_change":"2023-11-28T20:33:53.059457+0000","last_active":"2023-11-28T20:33:58.120814+0000","last_peered":"2023-11-28T20:33:58.120814+0000","last_clean":"2023-11-28T20:33:58.120814+0000","last_became_active":"2023-11-28T20:33:53.059116+0000","last_became_peered":"2023-11-28T20:33:53.059116+0000","last_unstale":"2023-11-28T20:33:58.120814+0000","last_undegraded":"2023-11-28T20:33:58.120814+0000","last_fullsized":"2023-11-28T20:33:58.120814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:32.036639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590492+0000","last_change":"2023-11-28T20:33:58.141507+0000","last_active":"2023-11-28T20:33:59.590492+0000","last_peered":"2023-11-28T20:33:59.590492+0000","last_clean":"2023-11-28T20:33:59.590492+0000","last_became_active":"2023-11-28T20:33:58.141072+0000","last_became_peered":"2023-11-28T20:33:58.141072+0000","last_unstale":"2023-11-28T20:33:59.590492+0000","last_undegraded":"2023-11-28T20:33:59.590492+0000","last_fullsized":"2023-11-28T20:33:59.590492+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103356+0000","last_change":"2023-11-28T20:33:53.937448+0000","last_active":"2023-11-28T20:33:58.103356+0000","last_peered":"2023-11-28T20:33:58.103356+0000","last_clean":"2023-11-28T20:33:58.103356+0000","last_became_active":"2023-11-28T20:33:53.937097+0000","last_became_peered":"2023-11-28T20:33:53.937097+0000","last_unstale":"2023-11-28T20:33:58.103356+0000","last_undegraded":"2023-11-28T20:33:58.103356+0000","last_fullsized":"2023-11-28T20:33:58.103356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105061+0000","last_change":"2023-11-28T20:33:53.047089+0000","last_active":"2023-11-28T20:33:58.105061+0000","last_peered":"2023-11-28T20:33:58.105061+0000","last_clean":"2023-11-28T20:33:58.105061+0000","last_became_active":"2023-11-28T20:33:53.046299+0000","last_became_peered":"2023-11-28T20:33:53.046299+0000","last_unstale":"2023-11-28T20:33:58.105061+0000","last_undegraded":"2023-11-28T20:33:58.105061+0000","last_fullsized":"2023-11-28T20:33:58.105061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121518+0000","last_change":"2023-11-28T20:33:53.057990+0000","last_active":"2023-11-28T20:33:58.121518+0000","last_peered":"2023-11-28T20:33:58.121518+0000","last_clean":"2023-11-28T20:33:58.121518+0000","last_became_active":"2023-11-28T20:33:53.057437+0000","last_became_peered":"2023-11-28T20:33:53.057437+0000","last_unstale":"2023-11-28T20:33:58.121518+0000","last_undegraded":"2023-11-28T20:33:58.121518+0000","last_fullsized":"2023-11-28T20:33:58.121518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118455+0000","last_change":"2023-11-28T20:33:53.944979+0000","last_active":"2023-11-28T20:33:58.118455+0000","last_peered":"2023-11-28T20:33:58.118455+0000","last_clean":"2023-11-28T20:33:58.118455+0000","last_became_active":"2023-11-28T20:33:53.944587+0000","last_became_peered":"2023-11-28T20:33:53.944587+0000","last_unstale":"2023-11-28T20:33:58.118455+0000","last_undegraded":"2023-11-28T20:33:58.118455+0000","last_fullsized":"2023-11-28T20:33:58.118455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.226534+0000","last_change":"2023-11-28T20:33:58.128961+0000","last_active":"2023-11-28T20:33:59.226534+0000","last_peered":"2023-11-28T20:33:59.226534+0000","last_clean":"2023-11-28T20:33:59.226534+0000","last_became_active":"2023-11-28T20:33:58.128548+0000","last_became_peered":"2023-11-28T20:33:58.128548+0000","last_unstale":"2023-11-28T20:33:59.226534+0000","last_undegraded":"2023-11-28T20:33:59.226534+0000","last_fullsized":"2023-11-28T20:33:59.226534+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:39.520026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103995+0000","last_change":"2023-11-28T20:33:53.061966+0000","last_active":"2023-11-28T20:33:58.103995+0000","last_peered":"2023-11-28T20:33:58.103995+0000","last_clean":"2023-11-28T20:33:58.103995+0000","last_became_active":"2023-11-28T20:33:53.061538+0000","last_became_peered":"2023-11-28T20:33:53.061538+0000","last_unstale":"2023-11-28T20:33:58.103995+0000","last_undegraded":"2023-11-28T20:33:58.103995+0000","last_fullsized":"2023-11-28T20:33:58.103995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.113040+0000","last_change":"2023-11-28T20:33:53.943755+0000","last_active":"2023-11-28T20:33:58.113040+0000","last_peered":"2023-11-28T20:33:58.113040+0000","last_clean":"2023-11-28T20:33:58.113040+0000","last_became_active":"2023-11-28T20:33:53.943464+0000","last_became_peered":"2023-11-28T20:33:53.943464+0000","last_unstale":"2023-11-28T20:33:58.113040+0000","last_undegraded":"2023-11-28T20:33:58.113040+0000","last_fullsized":"2023-11-28T20:33:58.113040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:42.385226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.919987+0000","last_change":"2023-11-28T20:33:58.142246+0000","last_active":"2023-11-28T20:33:58.919987+0000","last_peered":"2023-11-28T20:33:58.919987+0000","last_clean":"2023-11-28T20:33:58.919987+0000","last_became_active":"2023-11-28T20:33:58.134395+0000","last_became_peered":"2023-11-28T20:33:58.134395+0000","last_unstale":"2023-11-28T20:33:58.919987+0000","last_undegraded":"2023-11-28T20:33:58.919987+0000","last_fullsized":"2023-11-28T20:33:58.919987+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:24:08.097260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119379+0000","last_change":"2023-11-28T20:33:53.948082+0000","last_active":"2023-11-28T20:33:58.119379+0000","last_peered":"2023-11-28T20:33:58.119379+0000","last_clean":"2023-11-28T20:33:58.119379+0000","last_became_active":"2023-11-28T20:33:53.947760+0000","last_became_peered":"2023-11-28T20:33:53.947760+0000","last_unstale":"2023-11-28T20:33:58.119379+0000","last_undegraded":"2023-11-28T20:33:58.119379+0000","last_fullsized":"2023-11-28T20:33:58.119379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:35.450849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099225+0000","last_change":"2023-11-28T20:33:53.947745+0000","last_active":"2023-11-28T20:33:58.099225+0000","last_peered":"2023-11-28T20:33:58.099225+0000","last_clean":"2023-11-28T20:33:58.099225+0000","last_became_active":"2023-11-28T20:33:53.947452+0000","last_became_peered":"2023-11-28T20:33:53.947452+0000","last_unstale":"2023-11-28T20:33:58.099225+0000","last_undegraded":"2023-11-28T20:33:58.099225+0000","last_fullsized":"2023-11-28T20:33:58.099225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:42.768692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125984+0000","last_change":"2023-11-28T20:33:53.947868+0000","last_active":"2023-11-28T20:33:58.125984+0000","last_peered":"2023-11-28T20:33:58.125984+0000","last_clean":"2023-11-28T20:33:58.125984+0000","last_became_active":"2023-11-28T20:33:53.947304+0000","last_became_peered":"2023-11-28T20:33:53.947304+0000","last_unstale":"2023-11-28T20:33:58.125984+0000","last_undegraded":"2023-11-28T20:33:58.125984+0000","last_fullsized":"2023-11-28T20:33:58.125984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118368+0000","last_change":"2023-11-28T20:33:53.947122+0000","last_active":"2023-11-28T20:33:58.118368+0000","last_peered":"2023-11-28T20:33:58.118368+0000","last_clean":"2023-11-28T20:33:58.118368+0000","last_became_active":"2023-11-28T20:33:53.946556+0000","last_became_peered":"2023-11-28T20:33:53.946556+0000","last_unstale":"2023-11-28T20:33:58.118368+0000","last_undegraded":"2023-11-28T20:33:58.118368+0000","last_fullsized":"2023-11-28T20:33:58.118368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.109664+0000","last_change":"2023-11-28T20:33:53.946180+0000","last_active":"2023-11-28T20:33:58.109664+0000","last_peered":"2023-11-28T20:33:58.109664+0000","last_clean":"2023-11-28T20:33:58.109664+0000","last_became_active":"2023-11-28T20:33:53.945758+0000","last_became_peered":"2023-11-28T20:33:53.945758+0000","last_unstale":"2023-11-28T20:33:58.109664+0000","last_undegraded":"2023-11-28T20:33:58.109664+0000","last_fullsized":"2023-11-28T20:33:58.109664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123704+0000","last_change":"2023-11-28T20:33:53.950532+0000","last_active":"2023-11-28T20:33:58.123704+0000","last_peered":"2023-11-28T20:33:58.123704+0000","last_clean":"2023-11-28T20:33:58.123704+0000","last_became_active":"2023-11-28T20:33:53.950206+0000","last_became_peered":"2023-11-28T20:33:53.950206+0000","last_unstale":"2023-11-28T20:33:58.123704+0000","last_undegraded":"2023-11-28T20:33:58.123704+0000","last_fullsized":"2023-11-28T20:33:58.123704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126244+0000","last_change":"2023-11-28T20:33:53.958413+0000","last_active":"2023-11-28T20:33:58.126244+0000","last_peered":"2023-11-28T20:33:58.126244+0000","last_clean":"2023-11-28T20:33:58.126244+0000","last_became_active":"2023-11-28T20:33:53.958179+0000","last_became_peered":"2023-11-28T20:33:53.958179+0000","last_unstale":"2023-11-28T20:33:58.126244+0000","last_undegraded":"2023-11-28T20:33:58.126244+0000","last_fullsized":"2023-11-28T20:33:58.126244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.113900+0000","last_change":"2023-11-28T20:33:53.952207+0000","last_active":"2023-11-28T20:33:58.113900+0000","last_peered":"2023-11-28T20:33:58.113900+0000","last_clean":"2023-11-28T20:33:58.113900+0000","last_became_active":"2023-11-28T20:33:53.950011+0000","last_became_peered":"2023-11-28T20:33:53.950011+0000","last_unstale":"2023-11-28T20:33:58.113900+0000","last_undegraded":"2023-11-28T20:33:58.113900+0000","last_fullsized":"2023-11-28T20:33:58.113900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120660+0000","last_change":"2023-11-28T20:33:53.952356+0000","last_active":"2023-11-28T20:33:58.120660+0000","last_peered":"2023-11-28T20:33:58.120660+0000","last_clean":"2023-11-28T20:33:58.120660+0000","last_became_active":"2023-11-28T20:33:53.947787+0000","last_became_peered":"2023-11-28T20:33:53.947787+0000","last_unstale":"2023-11-28T20:33:58.120660+0000","last_undegraded":"2023-11-28T20:33:58.120660+0000","last_fullsized":"2023-11-28T20:33:58.120660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105059+0000","last_change":"2023-11-28T20:33:53.949973+0000","last_active":"2023-11-28T20:33:58.105059+0000","last_peered":"2023-11-28T20:33:58.105059+0000","last_clean":"2023-11-28T20:33:58.105059+0000","last_became_active":"2023-11-28T20:33:53.946758+0000","last_became_peered":"2023-11-28T20:33:53.946758+0000","last_unstale":"2023-11-28T20:33:58.105059+0000","last_undegraded":"2023-11-28T20:33:58.105059+0000","last_fullsized":"2023-11-28T20:33:58.105059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125895+0000","last_change":"2023-11-28T20:33:53.950994+0000","last_active":"2023-11-28T20:33:58.125895+0000","last_peered":"2023-11-28T20:33:58.125895+0000","last_clean":"2023-11-28T20:33:58.125895+0000","last_became_active":"2023-11-28T20:33:53.946524+0000","last_became_peered":"2023-11-28T20:33:53.946524+0000","last_unstale":"2023-11-28T20:33:58.125895+0000","last_undegraded":"2023-11-28T20:33:58.125895+0000","last_fullsized":"2023-11-28T20:33:58.125895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118275+0000","last_change":"2023-11-28T20:33:53.939684+0000","last_active":"2023-11-28T20:33:58.118275+0000","last_peered":"2023-11-28T20:33:58.118275+0000","last_clean":"2023-11-28T20:33:58.118275+0000","last_became_active":"2023-11-28T20:33:53.939225+0000","last_became_peered":"2023-11-28T20:33:53.939225+0000","last_unstale":"2023-11-28T20:33:58.118275+0000","last_undegraded":"2023-11-28T20:33:58.118275+0000","last_fullsized":"2023-11-28T20:33:58.118275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125812+0000","last_change":"2023-11-28T20:33:53.951199+0000","last_active":"2023-11-28T20:33:58.125812+0000","last_peered":"2023-11-28T20:33:58.125812+0000","last_clean":"2023-11-28T20:33:58.125812+0000","last_became_active":"2023-11-28T20:33:53.949262+0000","last_became_peered":"2023-11-28T20:33:53.949262+0000","last_unstale":"2023-11-28T20:33:58.125812+0000","last_undegraded":"2023-11-28T20:33:58.125812+0000","last_fullsized":"2023-11-28T20:33:58.125812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152008+0000","last_change":"2023-11-28T20:33:53.958492+0000","last_active":"2023-11-28T20:33:58.152008+0000","last_peered":"2023-11-28T20:33:58.152008+0000","last_clean":"2023-11-28T20:33:58.152008+0000","last_became_active":"2023-11-28T20:33:53.958141+0000","last_became_peered":"2023-11-28T20:33:53.958141+0000","last_unstale":"2023-11-28T20:33:58.152008+0000","last_undegraded":"2023-11-28T20:33:58.152008+0000","last_fullsized":"2023-11-28T20:33:58.152008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:13.738167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099160+0000","last_change":"2023-11-28T20:33:53.948508+0000","last_active":"2023-11-28T20:33:58.099160+0000","last_peered":"2023-11-28T20:33:58.099160+0000","last_clean":"2023-11-28T20:33:58.099160+0000","last_became_active":"2023-11-28T20:33:53.947303+0000","last_became_peered":"2023-11-28T20:33:53.947303+0000","last_unstale":"2023-11-28T20:33:58.099160+0000","last_undegraded":"2023-11-28T20:33:58.099160+0000","last_fullsized":"2023-11-28T20:33:58.099160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:23.240825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.127814+0000","last_change":"2023-11-28T20:33:53.939826+0000","last_active":"2023-11-28T20:33:58.127814+0000","last_peered":"2023-11-28T20:33:58.127814+0000","last_clean":"2023-11-28T20:33:58.127814+0000","last_became_active":"2023-11-28T20:33:53.939349+0000","last_became_peered":"2023-11-28T20:33:53.939349+0000","last_unstale":"2023-11-28T20:33:58.127814+0000","last_undegraded":"2023-11-28T20:33:58.127814+0000","last_fullsized":"2023-11-28T20:33:58.127814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:13.738167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120426+0000","last_change":"2023-11-28T20:33:53.939569+0000","last_active":"2023-11-28T20:33:58.120426+0000","last_peered":"2023-11-28T20:33:58.120426+0000","last_clean":"2023-11-28T20:33:58.120426+0000","last_became_active":"2023-11-28T20:33:53.936362+0000","last_became_peered":"2023-11-28T20:33:53.936362+0000","last_unstale":"2023-11-28T20:33:58.120426+0000","last_undegraded":"2023-11-28T20:33:58.120426+0000","last_fullsized":"2023-11-28T20:33:58.120426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123031+0000","last_change":"2023-11-28T20:33:53.950757+0000","last_active":"2023-11-28T20:33:58.123031+0000","last_peered":"2023-11-28T20:33:58.123031+0000","last_clean":"2023-11-28T20:33:58.123031+0000","last_became_active":"2023-11-28T20:33:53.948014+0000","last_became_peered":"2023-11-28T20:33:53.948014+0000","last_unstale":"2023-11-28T20:33:58.123031+0000","last_undegraded":"2023-11-28T20:33:58.123031+0000","last_fullsized":"2023-11-28T20:33:58.123031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117566+0000","last_change":"2023-11-28T20:33:53.938422+0000","last_active":"2023-11-28T20:33:58.117566+0000","last_peered":"2023-11-28T20:33:58.117566+0000","last_clean":"2023-11-28T20:33:58.117566+0000","last_became_active":"2023-11-28T20:33:53.936194+0000","last_became_peered":"2023-11-28T20:33:53.936194+0000","last_unstale":"2023-11-28T20:33:58.117566+0000","last_undegraded":"2023-11-28T20:33:58.117566+0000","last_fullsized":"2023-11-28T20:33:58.117566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:35.450849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103553+0000","last_change":"2023-11-28T20:33:53.946481+0000","last_active":"2023-11-28T20:33:58.103553+0000","last_peered":"2023-11-28T20:33:58.103553+0000","last_clean":"2023-11-28T20:33:58.103553+0000","last_became_active":"2023-11-28T20:33:53.946123+0000","last_became_peered":"2023-11-28T20:33:53.946123+0000","last_unstale":"2023-11-28T20:33:58.103553+0000","last_undegraded":"2023-11-28T20:33:58.103553+0000","last_fullsized":"2023-11-28T20:33:58.103553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105089+0000","last_change":"2023-11-28T20:33:53.960696+0000","last_active":"2023-11-28T20:33:58.105089+0000","last_peered":"2023-11-28T20:33:58.105089+0000","last_clean":"2023-11-28T20:33:58.105089+0000","last_became_active":"2023-11-28T20:33:53.953779+0000","last_became_peered":"2023-11-28T20:33:53.953779+0000","last_unstale":"2023-11-28T20:33:58.105089+0000","last_undegraded":"2023-11-28T20:33:58.105089+0000","last_fullsized":"2023-11-28T20:33:58.105089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:47.468024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126504+0000","last_change":"2023-11-28T20:33:53.938502+0000","last_active":"2023-11-28T20:33:58.126504+0000","last_peered":"2023-11-28T20:33:58.126504+0000","last_clean":"2023-11-28T20:33:58.126504+0000","last_became_active":"2023-11-28T20:33:53.938133+0000","last_became_peered":"2023-11-28T20:33:53.938133+0000","last_unstale":"2023-11-28T20:33:58.126504+0000","last_undegraded":"2023-11-28T20:33:58.126504+0000","last_fullsized":"2023-11-28T20:33:58.126504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126197+0000","last_change":"2023-11-28T20:33:53.961946+0000","last_active":"2023-11-28T20:33:58.126197+0000","last_peered":"2023-11-28T20:33:58.126197+0000","last_clean":"2023-11-28T20:33:58.126197+0000","last_became_active":"2023-11-28T20:33:53.961681+0000","last_became_peered":"2023-11-28T20:33:53.961681+0000","last_unstale":"2023-11-28T20:33:58.126197+0000","last_undegraded":"2023-11-28T20:33:58.126197+0000","last_fullsized":"2023-11-28T20:33:58.126197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152700+0000","last_change":"2023-11-28T20:33:53.963253+0000","last_active":"2023-11-28T20:33:58.152700+0000","last_peered":"2023-11-28T20:33:58.152700+0000","last_clean":"2023-11-28T20:33:58.152700+0000","last_became_active":"2023-11-28T20:33:53.961179+0000","last_became_peered":"2023-11-28T20:33:53.961179+0000","last_unstale":"2023-11-28T20:33:58.152700+0000","last_undegraded":"2023-11-28T20:33:58.152700+0000","last_fullsized":"2023-11-28T20:33:58.152700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:35.843791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103213+0000","last_change":"2023-11-28T20:33:53.952078+0000","last_active":"2023-11-28T20:33:58.103213+0000","last_peered":"2023-11-28T20:33:58.103213+0000","last_clean":"2023-11-28T20:33:58.103213+0000","last_became_active":"2023-11-28T20:33:53.949626+0000","last_became_peered":"2023-11-28T20:33:53.949626+0000","last_unstale":"2023-11-28T20:33:58.103213+0000","last_undegraded":"2023-11-28T20:33:58.103213+0000","last_fullsized":"2023-11-28T20:33:58.103213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151515+0000","last_change":"2023-11-28T20:33:53.955688+0000","last_active":"2023-11-28T20:33:58.151515+0000","last_peered":"2023-11-28T20:33:58.151515+0000","last_clean":"2023-11-28T20:33:58.151515+0000","last_became_active":"2023-11-28T20:33:53.955384+0000","last_became_peered":"2023-11-28T20:33:53.955384+0000","last_unstale":"2023-11-28T20:33:58.151515+0000","last_undegraded":"2023-11-28T20:33:58.151515+0000","last_fullsized":"2023-11-28T20:33:58.151515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125802+0000","last_change":"2023-11-28T20:33:53.958084+0000","last_active":"2023-11-28T20:33:58.125802+0000","last_peered":"2023-11-28T20:33:58.125802+0000","last_clean":"2023-11-28T20:33:58.125802+0000","last_became_active":"2023-11-28T20:33:53.953483+0000","last_became_peered":"2023-11-28T20:33:53.953483+0000","last_unstale":"2023-11-28T20:33:58.125802+0000","last_undegraded":"2023-11-28T20:33:58.125802+0000","last_fullsized":"2023-11-28T20:33:58.125802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151629+0000","last_change":"2023-11-28T20:33:53.955124+0000","last_active":"2023-11-28T20:33:58.151629+0000","last_peered":"2023-11-28T20:33:58.151629+0000","last_clean":"2023-11-28T20:33:58.151629+0000","last_became_active":"2023-11-28T20:33:53.954422+0000","last_became_peered":"2023-11-28T20:33:53.954422+0000","last_unstale":"2023-11-28T20:33:58.151629+0000","last_undegraded":"2023-11-28T20:33:58.151629+0000","last_fullsized":"2023-11-28T20:33:58.151629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:19.573371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121991+0000","last_change":"2023-11-28T20:33:53.957855+0000","last_active":"2023-11-28T20:33:58.121991+0000","last_peered":"2023-11-28T20:33:58.121991+0000","last_clean":"2023-11-28T20:33:58.121991+0000","last_became_active":"2023-11-28T20:33:53.955645+0000","last_became_peered":"2023-11-28T20:33:53.955645+0000","last_unstale":"2023-11-28T20:33:58.121991+0000","last_undegraded":"2023-11-28T20:33:58.121991+0000","last_fullsized":"2023-11-28T20:33:58.121991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116851+0000","last_change":"2023-11-28T20:33:53.937690+0000","last_active":"2023-11-28T20:33:58.116851+0000","last_peered":"2023-11-28T20:33:58.116851+0000","last_clean":"2023-11-28T20:33:58.116851+0000","last_became_active":"2023-11-28T20:33:53.937374+0000","last_became_peered":"2023-11-28T20:33:53.937374+0000","last_unstale":"2023-11-28T20:33:58.116851+0000","last_undegraded":"2023-11-28T20:33:58.116851+0000","last_fullsized":"2023-11-28T20:33:58.116851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123601+0000","last_change":"2023-11-28T20:33:53.950129+0000","last_active":"2023-11-28T20:33:58.123601+0000","last_peered":"2023-11-28T20:33:58.123601+0000","last_clean":"2023-11-28T20:33:58.123601+0000","last_became_active":"2023-11-28T20:33:53.947280+0000","last_became_peered":"2023-11-28T20:33:53.947280+0000","last_unstale":"2023-11-28T20:33:58.123601+0000","last_undegraded":"2023-11-28T20:33:58.123601+0000","last_fullsized":"2023-11-28T20:33:58.123601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152690+0000","last_change":"2023-11-28T20:33:53.964432+0000","last_active":"2023-11-28T20:33:58.152690+0000","last_peered":"2023-11-28T20:33:58.152690+0000","last_clean":"2023-11-28T20:33:58.152690+0000","last_became_active":"2023-11-28T20:33:53.963967+0000","last_became_peered":"2023-11-28T20:33:53.963967+0000","last_unstale":"2023-11-28T20:33:58.152690+0000","last_undegraded":"2023-11-28T20:33:58.152690+0000","last_fullsized":"2023-11-28T20:33:58.152690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:07.426176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":695413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7989999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.851}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3580000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5980000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5569999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.577}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4930000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.637}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8169999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7569999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9179999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":104463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.855}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":274432,"data_stored":104623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.788}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":695469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:34:30.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:34:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.506+0000 7efc5d064640 1 Processor -- start 2023-11-28T20:34:30.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.506+0000 7efc5d064640 1 -- start start 2023-11-28T20:34:30.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.507+0000 7efc5d064640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580721e0 0x7efc5806d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:30.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.507+0000 7efc5d064640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efc5806d6b0 0x7efc580ac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:30.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.507+0000 7efc5d064640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5815e680 con 0x7efc580b3870 2023-11-28T20:34:30.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.507+0000 7efc5d064640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5815a460 con 0x7efc5806d6b0 2023-11-28T20:34:30.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.507+0000 7efc5d064640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5806db40 con 0x7efc580721e0 2023-11-28T20:34:30.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.508+0000 7efc5c863640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efc5806d6b0 0x7efc580ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:30.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.508+0000 7efc57fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efc580b3870 0x7efc5815db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51938/0 (socket says 172.21.15.45:51938) 2023-11-28T20:34:30.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.508+0000 7efc57fff640 1 -- 172.21.15.45:0/781547285 learned_addr learned my addr 172.21.15.45:0/781547285 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:30.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.508+0000 7efc5c863640 1 -- 172.21.15.45:0/781547285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580721e0 msgr2=0x7efc5806d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:30.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.508+0000 7efc5c863640 1 --2- 172.21.15.45:0/781547285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580721e0 0x7efc5806d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.508+0000 7efc5c863640 1 -- 172.21.15.45:0/781547285 >> v1:172.21.15.45:6789/0 conn(0x7efc580b3870 legacy=0x7efc5815db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:30.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.508+0000 7efc5c863640 1 -- 172.21.15.45:0/781547285 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5815e960 con 0x7efc5806d6b0 2023-11-28T20:34:30.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.509+0000 7efc5c863640 1 --2- 172.21.15.45:0/781547285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 0x7efc580ac840 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efc50009d90 tx=0x7efc5002e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44097a8c84749e80 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:30.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.509+0000 7efc56ffd640 1 -- 172.21.15.45:0/781547285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5002ee60 con 0x7efc5806d6b0 2023-11-28T20:34:30.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc56ffd640 1 -- 172.21.15.45:0/781547285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc50036440 con 0x7efc5806d6b0 2023-11-28T20:34:30.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc56ffd640 1 -- 172.21.15.45:0/781547285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc500367d0 con 0x7efc5806d6b0 2023-11-28T20:34:30.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc5d064640 1 -- 172.21.15.45:0/781547285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 msgr2=0x7efc580ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:30.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc5d064640 1 --2- 172.21.15.45:0/781547285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 0x7efc580ac840 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efc50009d90 tx=0x7efc5002e0b0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc5d064640 1 -- 172.21.15.45:0/781547285 shutdown_connections 2023-11-28T20:34:30.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc5d064640 1 --2- 172.21.15.45:0/781547285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 0x7efc580ac840 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc5d064640 1 --2- 172.21.15.45:0/781547285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580721e0 0x7efc5806d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc5d064640 1 -- 172.21.15.45:0/781547285 >> 172.21.15.45:0/781547285 conn(0x7efc5806ab30 msgr2=0x7efc580b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:30.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc5d064640 1 -- 172.21.15.45:0/781547285 shutdown_connections 2023-11-28T20:34:30.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.510+0000 7efc5d064640 1 -- 172.21.15.45:0/781547285 wait complete. 2023-11-28T20:34:30.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.512+0000 7efc5d064640 1 Processor -- start 2023-11-28T20:34:30.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.512+0000 7efc5d064640 1 -- start start 2023-11-28T20:34:30.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.513+0000 7efc5d064640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 0x7efc5809c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:30.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.513+0000 7efc5d064640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580721e0 0x7efc5815be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:30.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.513+0000 7efc5d064640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc580b3870 0x7efc5815c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:30.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.513+0000 7efc5d064640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5809b770 con 0x7efc5806d6b0 2023-11-28T20:34:30.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.513+0000 7efc5d064640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5809aaf0 con 0x7efc580b3870 2023-11-28T20:34:30.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.513+0000 7efc5d064640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5809add0 con 0x7efc580721e0 2023-11-28T20:34:30.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.513+0000 7efc57fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 0x7efc5809c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:30.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.513+0000 7efc57fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 0x7efc5809c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59094/0 (socket says 172.21.15.45:59094) 2023-11-28T20:34:30.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.513+0000 7efc57fff640 1 -- 172.21.15.45:0/3454783450 learned_addr learned my addr 172.21.15.45:0/3454783450 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc57fff640 1 -- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580721e0 msgr2=0x7efc5815be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc57fff640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580721e0 0x7efc5815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc57fff640 1 -- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc580b3870 msgr2=0x7efc5815c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc57fff640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc580b3870 0x7efc5815c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc57fff640 1 -- 172.21.15.45:0/3454783450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50009a70 con 0x7efc5806d6b0 2023-11-28T20:34:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc57fff640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 0x7efc5809c430 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efc4800c480 tx=0x7efc4800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc54ff9640 1 -- 172.21.15.45:0/3454783450 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48005ce0 con 0x7efc5806d6b0 2023-11-28T20:34:30.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc54ff9640 1 -- 172.21.15.45:0/3454783450 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc48005e40 con 0x7efc5806d6b0 2023-11-28T20:34:30.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc54ff9640 1 -- 172.21.15.45:0/3454783450 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc480072f0 con 0x7efc5806d6b0 2023-11-28T20:34:30.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5809b0b0 con 0x7efc5806d6b0 2023-11-28T20:34:30.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.514+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5809c970 con 0x7efc5806d6b0 2023-11-28T20:34:30.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.516+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5809b8b0 con 0x7efc5806d6b0 2023-11-28T20:34:30.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.516+0000 7efc54ff9640 1 -- 172.21.15.45:0/3454783450 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efc48007510 con 0x7efc5806d6b0 2023-11-28T20:34:30.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.518+0000 7efc54ff9640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efc4406ae40 0x7efc4406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:30.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.518+0000 7efc54ff9640 1 -- 172.21.15.45:0/3454783450 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efc48088e30 con 0x7efc5806d6b0 2023-11-28T20:34:30.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.518+0000 7efc577fe640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efc4406ae40 0x7efc4406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:30.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.519+0000 7efc577fe640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efc4406ae40 0x7efc4406d300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc581f66d0 tx=0x7efc4c006c90 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:30.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.522+0000 7efc54ff9640 1 -- 172.21.15.45:0/3454783450 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc5809b8b0 con 0x7efc5806d6b0 2023-11-28T20:34:30.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.703+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc58071cc0 con 0x7efc4406ae40 2023-11-28T20:34:30.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.729+0000 7efc54ff9640 1 -- 172.21.15.45:0/3454783450 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345032 (secure 0 0 0) 0x7efc4c013400 con 0x7efc4406ae40 2023-11-28T20:34:30.736 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:30.738 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:34:30.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efc4406ae40 msgr2=0x7efc4406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:30.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efc4406ae40 0x7efc4406d300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc581f66d0 tx=0x7efc4c006c90 comp rx=0 tx=0).stop 2023-11-28T20:34:30.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 msgr2=0x7efc5809c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:30.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 0x7efc5809c430 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efc4800c480 tx=0x7efc4800c940 comp rx=0 tx=0).stop 2023-11-28T20:34:30.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 shutdown_connections 2023-11-28T20:34:30.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efc4406ae40 0x7efc4406d300 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc580b3870 0x7efc5815c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580721e0 0x7efc5815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 --2- 172.21.15.45:0/3454783450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc5806d6b0 0x7efc5809c430 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:30.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.735+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 >> 172.21.15.45:0/3454783450 conn(0x7efc5806ab30 msgr2=0x7efc5806cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:30.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.736+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 shutdown_connections 2023-11-28T20:34:30.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:30.736+0000 7efc5d064640 1 -- 172.21.15.45:0/3454783450 wait complete. 2023-11-28T20:34:30.758 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2023-11-28T20:34:29.136780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236124,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738596,"statfs":{"total":773094113280,"available":772852322304,"internally_reserved":0,"allocated":3383296,"data_stored":2025718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005628"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119305+0000","last_change":"2023-11-28T20:33:53.064970+0000","last_active":"2023-11-28T20:33:58.119305+0000","last_peered":"2023-11-28T20:33:58.119305+0000","last_clean":"2023-11-28T20:33:58.119305+0000","last_became_active":"2023-11-28T20:33:53.064659+0000","last_became_peered":"2023-11-28T20:33:53.064659+0000","last_unstale":"2023-11-28T20:33:58.119305+0000","last_undegraded":"2023-11-28T20:33:58.119305+0000","last_fullsized":"2023-11-28T20:33:58.119305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.109629+0000","last_change":"2023-11-28T20:33:53.948203+0000","last_active":"2023-11-28T20:33:58.109629+0000","last_peered":"2023-11-28T20:33:58.109629+0000","last_clean":"2023-11-28T20:33:58.109629+0000","last_became_active":"2023-11-28T20:33:53.947906+0000","last_became_peered":"2023-11-28T20:33:53.947906+0000","last_unstale":"2023-11-28T20:33:58.109629+0000","last_undegraded":"2023-11-28T20:33:58.109629+0000","last_fullsized":"2023-11-28T20:33:58.109629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918353+0000","last_change":"2023-11-28T20:33:58.158488+0000","last_active":"2023-11-28T20:33:58.918353+0000","last_peered":"2023-11-28T20:33:58.918353+0000","last_clean":"2023-11-28T20:33:58.918353+0000","last_became_active":"2023-11-28T20:33:58.158046+0000","last_became_peered":"2023-11-28T20:33:58.158046+0000","last_unstale":"2023-11-28T20:33:58.918353+0000","last_undegraded":"2023-11-28T20:33:58.918353+0000","last_fullsized":"2023-11-28T20:33:58.918353+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:20:48.980178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.111851+0000","last_change":"2023-11-28T20:33:53.050283+0000","last_active":"2023-11-28T20:33:58.111851+0000","last_peered":"2023-11-28T20:33:58.111851+0000","last_clean":"2023-11-28T20:33:58.111851+0000","last_became_active":"2023-11-28T20:33:53.049963+0000","last_became_peered":"2023-11-28T20:33:53.049963+0000","last_unstale":"2023-11-28T20:33:58.111851+0000","last_undegraded":"2023-11-28T20:33:58.111851+0000","last_fullsized":"2023-11-28T20:33:58.111851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116825+0000","last_change":"2023-11-28T20:33:53.943001+0000","last_active":"2023-11-28T20:33:58.116825+0000","last_peered":"2023-11-28T20:33:58.116825+0000","last_clean":"2023-11-28T20:33:58.116825+0000","last_became_active":"2023-11-28T20:33:53.940632+0000","last_became_peered":"2023-11-28T20:33:53.940632+0000","last_unstale":"2023-11-28T20:33:58.116825+0000","last_undegraded":"2023-11-28T20:33:58.116825+0000","last_fullsized":"2023-11-28T20:33:58.116825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916311+0000","last_change":"2023-11-28T20:33:58.127527+0000","last_active":"2023-11-28T20:33:58.916311+0000","last_peered":"2023-11-28T20:33:58.916311+0000","last_clean":"2023-11-28T20:33:58.916311+0000","last_became_active":"2023-11-28T20:33:58.127058+0000","last_became_peered":"2023-11-28T20:33:58.127058+0000","last_unstale":"2023-11-28T20:33:58.916311+0000","last_undegraded":"2023-11-28T20:33:58.916311+0000","last_fullsized":"2023-11-28T20:33:58.916311+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:12.932976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151962+0000","last_change":"2023-11-28T20:33:53.073019+0000","last_active":"2023-11-28T20:33:58.151962+0000","last_peered":"2023-11-28T20:33:58.151962+0000","last_clean":"2023-11-28T20:33:58.151962+0000","last_became_active":"2023-11-28T20:33:53.072728+0000","last_became_peered":"2023-11-28T20:33:53.072728+0000","last_unstale":"2023-11-28T20:33:58.151962+0000","last_undegraded":"2023-11-28T20:33:58.151962+0000","last_fullsized":"2023-11-28T20:33:58.151962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122157+0000","last_change":"2023-11-28T20:33:53.951054+0000","last_active":"2023-11-28T20:33:58.122157+0000","last_peered":"2023-11-28T20:33:58.122157+0000","last_clean":"2023-11-28T20:33:58.122157+0000","last_became_active":"2023-11-28T20:33:53.950730+0000","last_became_peered":"2023-11-28T20:33:53.950730+0000","last_unstale":"2023-11-28T20:33:58.122157+0000","last_undegraded":"2023-11-28T20:33:58.122157+0000","last_fullsized":"2023-11-28T20:33:58.122157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:19.573371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916721+0000","last_change":"2023-11-28T20:33:58.131034+0000","last_active":"2023-11-28T20:33:58.916721+0000","last_peered":"2023-11-28T20:33:58.916721+0000","last_clean":"2023-11-28T20:33:58.916721+0000","last_became_active":"2023-11-28T20:33:58.129161+0000","last_became_peered":"2023-11-28T20:33:58.129161+0000","last_unstale":"2023-11-28T20:33:58.916721+0000","last_undegraded":"2023-11-28T20:33:58.916721+0000","last_fullsized":"2023-11-28T20:33:58.916721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:19.781603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151744+0000","last_change":"2023-11-28T20:33:53.068043+0000","last_active":"2023-11-28T20:33:58.151744+0000","last_peered":"2023-11-28T20:33:58.151744+0000","last_clean":"2023-11-28T20:33:58.151744+0000","last_became_active":"2023-11-28T20:33:53.067650+0000","last_became_peered":"2023-11-28T20:33:53.067650+0000","last_unstale":"2023-11-28T20:33:58.151744+0000","last_undegraded":"2023-11-28T20:33:58.151744+0000","last_fullsized":"2023-11-28T20:33:58.151744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123332+0000","last_change":"2023-11-28T20:33:53.954109+0000","last_active":"2023-11-28T20:33:58.123332+0000","last_peered":"2023-11-28T20:33:58.123332+0000","last_clean":"2023-11-28T20:33:58.123332+0000","last_became_active":"2023-11-28T20:33:53.953749+0000","last_became_peered":"2023-11-28T20:33:53.953749+0000","last_unstale":"2023-11-28T20:33:58.123332+0000","last_undegraded":"2023-11-28T20:33:58.123332+0000","last_fullsized":"2023-11-28T20:33:58.123332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:21.394618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916961+0000","last_change":"2023-11-28T20:33:58.132306+0000","last_active":"2023-11-28T20:33:58.916961+0000","last_peered":"2023-11-28T20:33:58.916961+0000","last_clean":"2023-11-28T20:33:58.916961+0000","last_became_active":"2023-11-28T20:33:58.131782+0000","last_became_peered":"2023-11-28T20:33:58.131782+0000","last_unstale":"2023-11-28T20:33:58.916961+0000","last_undegraded":"2023-11-28T20:33:58.916961+0000","last_fullsized":"2023-11-28T20:33:58.916961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:30.608189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121351+0000","last_change":"2023-11-28T20:33:53.954276+0000","last_active":"2023-11-28T20:33:58.121351+0000","last_peered":"2023-11-28T20:33:58.121351+0000","last_clean":"2023-11-28T20:33:58.121351+0000","last_became_active":"2023-11-28T20:33:53.953993+0000","last_became_peered":"2023-11-28T20:33:53.953993+0000","last_unstale":"2023-11-28T20:33:58.121351+0000","last_undegraded":"2023-11-28T20:33:58.121351+0000","last_fullsized":"2023-11-28T20:33:58.121351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151814+0000","last_change":"2023-11-28T20:33:53.075863+0000","last_active":"2023-11-28T20:33:58.151814+0000","last_peered":"2023-11-28T20:33:58.151814+0000","last_clean":"2023-11-28T20:33:58.151814+0000","last_became_active":"2023-11-28T20:33:53.075523+0000","last_became_peered":"2023-11-28T20:33:53.075523+0000","last_unstale":"2023-11-28T20:33:58.151814+0000","last_undegraded":"2023-11-28T20:33:58.151814+0000","last_fullsized":"2023-11-28T20:33:58.151814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917964+0000","last_change":"2023-11-28T20:33:58.130991+0000","last_active":"2023-11-28T20:33:58.917964+0000","last_peered":"2023-11-28T20:33:58.917964+0000","last_clean":"2023-11-28T20:33:58.917964+0000","last_became_active":"2023-11-28T20:33:58.130377+0000","last_became_peered":"2023-11-28T20:33:58.130377+0000","last_unstale":"2023-11-28T20:33:58.917964+0000","last_undegraded":"2023-11-28T20:33:58.917964+0000","last_fullsized":"2023-11-28T20:33:58.917964+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103760+0000","last_change":"2023-11-28T20:33:53.943400+0000","last_active":"2023-11-28T20:33:58.103760+0000","last_peered":"2023-11-28T20:33:58.103760+0000","last_clean":"2023-11-28T20:33:58.103760+0000","last_became_active":"2023-11-28T20:33:53.937673+0000","last_became_peered":"2023-11-28T20:33:53.937673+0000","last_unstale":"2023-11-28T20:33:58.103760+0000","last_undegraded":"2023-11-28T20:33:58.103760+0000","last_fullsized":"2023-11-28T20:33:58.103760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125979+0000","last_change":"2023-11-28T20:33:53.062456+0000","last_active":"2023-11-28T20:33:58.125979+0000","last_peered":"2023-11-28T20:33:58.125979+0000","last_clean":"2023-11-28T20:33:58.125979+0000","last_became_active":"2023-11-28T20:33:53.062163+0000","last_became_peered":"2023-11-28T20:33:53.062163+0000","last_unstale":"2023-11-28T20:33:58.125979+0000","last_undegraded":"2023-11-28T20:33:58.125979+0000","last_fullsized":"2023-11-28T20:33:58.125979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917431+0000","last_change":"2023-11-28T20:33:58.137561+0000","last_active":"2023-11-28T20:33:58.917431+0000","last_peered":"2023-11-28T20:33:58.917431+0000","last_clean":"2023-11-28T20:33:58.917431+0000","last_became_active":"2023-11-28T20:33:58.137065+0000","last_became_peered":"2023-11-28T20:33:58.137065+0000","last_unstale":"2023-11-28T20:33:58.917431+0000","last_undegraded":"2023-11-28T20:33:58.917431+0000","last_fullsized":"2023-11-28T20:33:58.917431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103501+0000","last_change":"2023-11-28T20:33:53.940322+0000","last_active":"2023-11-28T20:33:58.103501+0000","last_peered":"2023-11-28T20:33:58.103501+0000","last_clean":"2023-11-28T20:33:58.103501+0000","last_became_active":"2023-11-28T20:33:53.939986+0000","last_became_peered":"2023-11-28T20:33:53.939986+0000","last_unstale":"2023-11-28T20:33:58.103501+0000","last_undegraded":"2023-11-28T20:33:58.103501+0000","last_fullsized":"2023-11-28T20:33:58.103501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123391+0000","last_change":"2023-11-28T20:33:53.062082+0000","last_active":"2023-11-28T20:33:58.123391+0000","last_peered":"2023-11-28T20:33:58.123391+0000","last_clean":"2023-11-28T20:33:58.123391+0000","last_became_active":"2023-11-28T20:33:53.060583+0000","last_became_peered":"2023-11-28T20:33:53.060583+0000","last_unstale":"2023-11-28T20:33:58.123391+0000","last_undegraded":"2023-11-28T20:33:58.123391+0000","last_fullsized":"2023-11-28T20:33:58.123391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916613+0000","last_change":"2023-11-28T20:33:58.158289+0000","last_active":"2023-11-28T20:33:58.916613+0000","last_peered":"2023-11-28T20:33:58.916613+0000","last_clean":"2023-11-28T20:33:58.916613+0000","last_became_active":"2023-11-28T20:33:58.157856+0000","last_became_peered":"2023-11-28T20:33:58.157856+0000","last_unstale":"2023-11-28T20:33:58.916613+0000","last_undegraded":"2023-11-28T20:33:58.916613+0000","last_fullsized":"2023-11-28T20:33:58.916613+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:17.807344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.112802+0000","last_change":"2023-11-28T20:33:53.942280+0000","last_active":"2023-11-28T20:33:58.112802+0000","last_peered":"2023-11-28T20:33:58.112802+0000","last_clean":"2023-11-28T20:33:58.112802+0000","last_became_active":"2023-11-28T20:33:53.939778+0000","last_became_peered":"2023-11-28T20:33:53.939778+0000","last_unstale":"2023-11-28T20:33:58.112802+0000","last_undegraded":"2023-11-28T20:33:58.112802+0000","last_fullsized":"2023-11-28T20:33:58.112802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:33.050269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151419+0000","last_change":"2023-11-28T20:33:53.068971+0000","last_active":"2023-11-28T20:33:58.151419+0000","last_peered":"2023-11-28T20:33:58.151419+0000","last_clean":"2023-11-28T20:33:58.151419+0000","last_became_active":"2023-11-28T20:33:53.068069+0000","last_became_peered":"2023-11-28T20:33:53.068069+0000","last_unstale":"2023-11-28T20:33:58.151419+0000","last_undegraded":"2023-11-28T20:33:58.151419+0000","last_fullsized":"2023-11-28T20:33:58.151419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:41.371596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917429+0000","last_change":"2023-11-28T20:33:58.137597+0000","last_active":"2023-11-28T20:33:58.917429+0000","last_peered":"2023-11-28T20:33:58.917429+0000","last_clean":"2023-11-28T20:33:58.917429+0000","last_became_active":"2023-11-28T20:33:58.137078+0000","last_became_peered":"2023-11-28T20:33:58.137078+0000","last_unstale":"2023-11-28T20:33:58.917429+0000","last_undegraded":"2023-11-28T20:33:58.917429+0000","last_fullsized":"2023-11-28T20:33:58.917429+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:23.642548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120275+0000","last_change":"2023-11-28T20:33:53.945506+0000","last_active":"2023-11-28T20:33:58.120275+0000","last_peered":"2023-11-28T20:33:58.120275+0000","last_clean":"2023-11-28T20:33:58.120275+0000","last_became_active":"2023-11-28T20:33:53.945153+0000","last_became_peered":"2023-11-28T20:33:53.945153+0000","last_unstale":"2023-11-28T20:33:58.120275+0000","last_undegraded":"2023-11-28T20:33:58.120275+0000","last_fullsized":"2023-11-28T20:33:58.120275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126982+0000","last_change":"2023-11-28T20:33:53.064943+0000","last_active":"2023-11-28T20:33:58.126982+0000","last_peered":"2023-11-28T20:33:58.126982+0000","last_clean":"2023-11-28T20:33:58.126982+0000","last_became_active":"2023-11-28T20:33:53.064644+0000","last_became_peered":"2023-11-28T20:33:53.064644+0000","last_unstale":"2023-11-28T20:33:58.126982+0000","last_undegraded":"2023-11-28T20:33:58.126982+0000","last_fullsized":"2023-11-28T20:33:58.126982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:02.779844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916509+0000","last_change":"2023-11-28T20:33:58.158266+0000","last_active":"2023-11-28T20:33:58.916509+0000","last_peered":"2023-11-28T20:33:58.916509+0000","last_clean":"2023-11-28T20:33:58.916509+0000","last_became_active":"2023-11-28T20:33:58.157836+0000","last_became_peered":"2023-11-28T20:33:58.157836+0000","last_unstale":"2023-11-28T20:33:58.916509+0000","last_undegraded":"2023-11-28T20:33:58.916509+0000","last_fullsized":"2023-11-28T20:33:58.916509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:44.771180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.114503+0000","last_change":"2023-11-28T20:33:53.941572+0000","last_active":"2023-11-28T20:33:58.114503+0000","last_peered":"2023-11-28T20:33:58.114503+0000","last_clean":"2023-11-28T20:33:58.114503+0000","last_became_active":"2023-11-28T20:33:53.941245+0000","last_became_peered":"2023-11-28T20:33:53.941245+0000","last_unstale":"2023-11-28T20:33:58.114503+0000","last_undegraded":"2023-11-28T20:33:58.114503+0000","last_fullsized":"2023-11-28T20:33:58.114503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123698+0000","last_change":"2023-11-28T20:33:53.058678+0000","last_active":"2023-11-28T20:33:58.123698+0000","last_peered":"2023-11-28T20:33:58.123698+0000","last_clean":"2023-11-28T20:33:58.123698+0000","last_became_active":"2023-11-28T20:33:53.058375+0000","last_became_peered":"2023-11-28T20:33:53.058375+0000","last_unstale":"2023-11-28T20:33:58.123698+0000","last_undegraded":"2023-11-28T20:33:58.123698+0000","last_fullsized":"2023-11-28T20:33:58.123698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:12.793548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918259+0000","last_change":"2023-11-28T20:33:58.154128+0000","last_active":"2023-11-28T20:33:58.918259+0000","last_peered":"2023-11-28T20:33:58.918259+0000","last_clean":"2023-11-28T20:33:58.918259+0000","last_became_active":"2023-11-28T20:33:58.153697+0000","last_became_peered":"2023-11-28T20:33:58.153697+0000","last_unstale":"2023-11-28T20:33:58.918259+0000","last_undegraded":"2023-11-28T20:33:58.918259+0000","last_fullsized":"2023-11-28T20:33:58.918259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:47.677895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123582+0000","last_change":"2023-11-28T20:33:53.941553+0000","last_active":"2023-11-28T20:33:58.123582+0000","last_peered":"2023-11-28T20:33:58.123582+0000","last_clean":"2023-11-28T20:33:58.123582+0000","last_became_active":"2023-11-28T20:33:53.941248+0000","last_became_peered":"2023-11-28T20:33:53.941248+0000","last_unstale":"2023-11-28T20:33:58.123582+0000","last_undegraded":"2023-11-28T20:33:58.123582+0000","last_fullsized":"2023-11-28T20:33:58.123582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152051+0000","last_change":"2023-11-28T20:33:53.069890+0000","last_active":"2023-11-28T20:33:58.152051+0000","last_peered":"2023-11-28T20:33:58.152051+0000","last_clean":"2023-11-28T20:33:58.152051+0000","last_became_active":"2023-11-28T20:33:53.069572+0000","last_became_peered":"2023-11-28T20:33:53.069572+0000","last_unstale":"2023-11-28T20:33:58.152051+0000","last_undegraded":"2023-11-28T20:33:58.152051+0000","last_fullsized":"2023-11-28T20:33:58.152051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:57.471000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589761+0000","last_change":"2023-11-28T20:33:58.157661+0000","last_active":"2023-11-28T20:33:59.589761+0000","last_peered":"2023-11-28T20:33:59.589761+0000","last_clean":"2023-11-28T20:33:59.589761+0000","last_became_active":"2023-11-28T20:33:58.157072+0000","last_became_peered":"2023-11-28T20:33:58.157072+0000","last_unstale":"2023-11-28T20:33:59.589761+0000","last_undegraded":"2023-11-28T20:33:59.589761+0000","last_fullsized":"2023-11-28T20:33:59.589761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:11.495353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123465+0000","last_change":"2023-11-28T20:33:53.947235+0000","last_active":"2023-11-28T20:33:58.123465+0000","last_peered":"2023-11-28T20:33:58.123465+0000","last_clean":"2023-11-28T20:33:58.123465+0000","last_became_active":"2023-11-28T20:33:53.946977+0000","last_became_peered":"2023-11-28T20:33:53.946977+0000","last_unstale":"2023-11-28T20:33:58.123465+0000","last_undegraded":"2023-11-28T20:33:58.123465+0000","last_fullsized":"2023-11-28T20:33:58.123465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:08.318223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123720+0000","last_change":"2023-11-28T20:33:53.060664+0000","last_active":"2023-11-28T20:33:58.123720+0000","last_peered":"2023-11-28T20:33:58.123720+0000","last_clean":"2023-11-28T20:33:58.123720+0000","last_became_active":"2023-11-28T20:33:53.058196+0000","last_became_peered":"2023-11-28T20:33:53.058196+0000","last_unstale":"2023-11-28T20:33:58.123720+0000","last_undegraded":"2023-11-28T20:33:58.123720+0000","last_fullsized":"2023-11-28T20:33:58.123720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.915328+0000","last_change":"2023-11-28T20:33:58.133412+0000","last_active":"2023-11-28T20:33:58.915328+0000","last_peered":"2023-11-28T20:33:58.915328+0000","last_clean":"2023-11-28T20:33:58.915328+0000","last_became_active":"2023-11-28T20:33:58.132991+0000","last_became_peered":"2023-11-28T20:33:58.132991+0000","last_unstale":"2023-11-28T20:33:58.915328+0000","last_undegraded":"2023-11-28T20:33:58.915328+0000","last_fullsized":"2023-11-28T20:33:58.915328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.107225+0000","last_change":"2023-11-28T20:33:53.942153+0000","last_active":"2023-11-28T20:33:58.107225+0000","last_peered":"2023-11-28T20:33:58.107225+0000","last_clean":"2023-11-28T20:33:58.107225+0000","last_became_active":"2023-11-28T20:33:53.941788+0000","last_became_peered":"2023-11-28T20:33:53.941788+0000","last_unstale":"2023-11-28T20:33:58.107225+0000","last_undegraded":"2023-11-28T20:33:58.107225+0000","last_fullsized":"2023-11-28T20:33:58.107225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:47.311066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126159+0000","last_change":"2023-11-28T20:33:53.058321+0000","last_active":"2023-11-28T20:33:58.126159+0000","last_peered":"2023-11-28T20:33:58.126159+0000","last_clean":"2023-11-28T20:33:58.126159+0000","last_became_active":"2023-11-28T20:33:53.058170+0000","last_became_peered":"2023-11-28T20:33:53.058170+0000","last_unstale":"2023-11-28T20:33:58.126159+0000","last_undegraded":"2023-11-28T20:33:58.126159+0000","last_fullsized":"2023-11-28T20:33:58.126159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:12.793548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917495+0000","last_change":"2023-11-28T20:33:58.138024+0000","last_active":"2023-11-28T20:33:58.917495+0000","last_peered":"2023-11-28T20:33:58.917495+0000","last_clean":"2023-11-28T20:33:58.917495+0000","last_became_active":"2023-11-28T20:33:58.137601+0000","last_became_peered":"2023-11-28T20:33:58.137601+0000","last_unstale":"2023-11-28T20:33:58.917495+0000","last_undegraded":"2023-11-28T20:33:58.917495+0000","last_fullsized":"2023-11-28T20:33:58.917495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:45.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122244+0000","last_change":"2023-11-28T20:33:53.942125+0000","last_active":"2023-11-28T20:33:58.122244+0000","last_peered":"2023-11-28T20:33:58.122244+0000","last_clean":"2023-11-28T20:33:58.122244+0000","last_became_active":"2023-11-28T20:33:53.941741+0000","last_became_peered":"2023-11-28T20:33:53.941741+0000","last_unstale":"2023-11-28T20:33:58.122244+0000","last_undegraded":"2023-11-28T20:33:58.122244+0000","last_fullsized":"2023-11-28T20:33:58.122244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099678+0000","last_change":"2023-11-28T20:33:53.050707+0000","last_active":"2023-11-28T20:33:58.099678+0000","last_peered":"2023-11-28T20:33:58.099678+0000","last_clean":"2023-11-28T20:33:58.099678+0000","last_became_active":"2023-11-28T20:33:53.050403+0000","last_became_peered":"2023-11-28T20:33:53.050403+0000","last_unstale":"2023-11-28T20:33:58.099678+0000","last_undegraded":"2023-11-28T20:33:58.099678+0000","last_fullsized":"2023-11-28T20:33:58.099678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:32.036639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916922+0000","last_change":"2023-11-28T20:33:58.139720+0000","last_active":"2023-11-28T20:33:58.916922+0000","last_peered":"2023-11-28T20:33:58.916922+0000","last_clean":"2023-11-28T20:33:58.916922+0000","last_became_active":"2023-11-28T20:33:58.138372+0000","last_became_peered":"2023-11-28T20:33:58.138372+0000","last_unstale":"2023-11-28T20:33:58.916922+0000","last_undegraded":"2023-11-28T20:33:58.916922+0000","last_fullsized":"2023-11-28T20:33:58.916922+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:11.495353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117362+0000","last_change":"2023-11-28T20:33:53.934500+0000","last_active":"2023-11-28T20:33:58.117362+0000","last_peered":"2023-11-28T20:33:58.117362+0000","last_clean":"2023-11-28T20:33:58.117362+0000","last_became_active":"2023-11-28T20:33:53.933165+0000","last_became_peered":"2023-11-28T20:33:53.933165+0000","last_unstale":"2023-11-28T20:33:58.117362+0000","last_undegraded":"2023-11-28T20:33:58.117362+0000","last_fullsized":"2023-11-28T20:33:58.117362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:48.148924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151406+0000","last_change":"2023-11-28T20:33:53.069660+0000","last_active":"2023-11-28T20:33:58.151406+0000","last_peered":"2023-11-28T20:33:58.151406+0000","last_clean":"2023-11-28T20:33:58.151406+0000","last_became_active":"2023-11-28T20:33:53.069198+0000","last_became_peered":"2023-11-28T20:33:53.069198+0000","last_unstale":"2023-11-28T20:33:58.151406+0000","last_undegraded":"2023-11-28T20:33:58.151406+0000","last_fullsized":"2023-11-28T20:33:58.151406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:46.297436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590181+0000","last_change":"2023-11-28T20:33:58.162077+0000","last_active":"2023-11-28T20:33:59.590181+0000","last_peered":"2023-11-28T20:33:59.590181+0000","last_clean":"2023-11-28T20:33:59.590181+0000","last_became_active":"2023-11-28T20:33:58.159109+0000","last_became_peered":"2023-11-28T20:33:58.159109+0000","last_unstale":"2023-11-28T20:33:59.590181+0000","last_undegraded":"2023-11-28T20:33:59.590181+0000","last_fullsized":"2023-11-28T20:33:59.590181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:47.677895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103295+0000","last_change":"2023-11-28T20:33:53.952209+0000","last_active":"2023-11-28T20:33:58.103295+0000","last_peered":"2023-11-28T20:33:58.103295+0000","last_clean":"2023-11-28T20:33:58.103295+0000","last_became_active":"2023-11-28T20:33:53.949676+0000","last_became_peered":"2023-11-28T20:33:53.949676+0000","last_unstale":"2023-11-28T20:33:58.103295+0000","last_undegraded":"2023-11-28T20:33:58.103295+0000","last_fullsized":"2023-11-28T20:33:58.103295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121849+0000","last_change":"2023-11-28T20:33:53.053226+0000","last_active":"2023-11-28T20:33:58.121849+0000","last_peered":"2023-11-28T20:33:58.121849+0000","last_clean":"2023-11-28T20:33:58.121849+0000","last_became_active":"2023-11-28T20:33:53.052910+0000","last_became_peered":"2023-11-28T20:33:53.052910+0000","last_unstale":"2023-11-28T20:33:58.121849+0000","last_undegraded":"2023-11-28T20:33:58.121849+0000","last_fullsized":"2023-11-28T20:33:58.121849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:41.755062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":169,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151553+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:33:58.151553+0000","last_peered":"2023-11-28T20:33:58.151553+0000","last_clean":"2023-11-28T20:33:58.151553+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:33:58.151553+0000","last_undegraded":"2023-11-28T20:33:58.151553+0000","last_fullsized":"2023-11-28T20:33:58.151553+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589763+0000","last_change":"2023-11-28T20:33:58.157671+0000","last_active":"2023-11-28T20:33:59.589763+0000","last_peered":"2023-11-28T20:33:59.589763+0000","last_clean":"2023-11-28T20:33:59.589763+0000","last_became_active":"2023-11-28T20:33:58.157254+0000","last_became_peered":"2023-11-28T20:33:58.157254+0000","last_unstale":"2023-11-28T20:33:59.589763+0000","last_undegraded":"2023-11-28T20:33:59.589763+0000","last_fullsized":"2023-11-28T20:33:59.589763+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:25.463795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151876+0000","last_change":"2023-11-28T20:33:53.960256+0000","last_active":"2023-11-28T20:33:58.151876+0000","last_peered":"2023-11-28T20:33:58.151876+0000","last_clean":"2023-11-28T20:33:58.151876+0000","last_became_active":"2023-11-28T20:33:53.959635+0000","last_became_peered":"2023-11-28T20:33:53.959635+0000","last_unstale":"2023-11-28T20:33:58.151876+0000","last_undegraded":"2023-11-28T20:33:58.151876+0000","last_fullsized":"2023-11-28T20:33:58.151876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:08.318223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099297+0000","last_change":"2023-11-28T20:33:53.054253+0000","last_active":"2023-11-28T20:33:58.099297+0000","last_peered":"2023-11-28T20:33:58.099297+0000","last_clean":"2023-11-28T20:33:58.099297+0000","last_became_active":"2023-11-28T20:33:53.053912+0000","last_became_peered":"2023-11-28T20:33:53.053912+0000","last_unstale":"2023-11-28T20:33:58.099297+0000","last_undegraded":"2023-11-28T20:33:58.099297+0000","last_fullsized":"2023-11-28T20:33:58.099297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:53.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589287+0000","last_change":"2023-11-28T20:33:58.154204+0000","last_active":"2023-11-28T20:33:59.589287+0000","last_peered":"2023-11-28T20:33:59.589287+0000","last_clean":"2023-11-28T20:33:59.589287+0000","last_became_active":"2023-11-28T20:33:58.153585+0000","last_became_peered":"2023-11-28T20:33:58.153585+0000","last_unstale":"2023-11-28T20:33:59.589287+0000","last_undegraded":"2023-11-28T20:33:59.589287+0000","last_fullsized":"2023-11-28T20:33:59.589287+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:23.642548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122263+0000","last_change":"2023-11-28T20:33:53.950370+0000","last_active":"2023-11-28T20:33:58.122263+0000","last_peered":"2023-11-28T20:33:58.122263+0000","last_clean":"2023-11-28T20:33:58.122263+0000","last_became_active":"2023-11-28T20:33:53.946975+0000","last_became_peered":"2023-11-28T20:33:53.946975+0000","last_unstale":"2023-11-28T20:33:58.122263+0000","last_undegraded":"2023-11-28T20:33:58.122263+0000","last_fullsized":"2023-11-28T20:33:58.122263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:41.830889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123611+0000","last_change":"2023-11-28T20:33:53.062319+0000","last_active":"2023-11-28T20:33:58.123611+0000","last_peered":"2023-11-28T20:33:58.123611+0000","last_clean":"2023-11-28T20:33:58.123611+0000","last_became_active":"2023-11-28T20:33:53.062034+0000","last_became_peered":"2023-11-28T20:33:53.062034+0000","last_unstale":"2023-11-28T20:33:58.123611+0000","last_undegraded":"2023-11-28T20:33:58.123611+0000","last_fullsized":"2023-11-28T20:33:58.123611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590144+0000","last_change":"2023-11-28T20:33:58.129150+0000","last_active":"2023-11-28T20:33:59.590144+0000","last_peered":"2023-11-28T20:33:59.590144+0000","last_clean":"2023-11-28T20:33:59.590144+0000","last_became_active":"2023-11-28T20:33:58.128658+0000","last_became_peered":"2023-11-28T20:33:58.128658+0000","last_unstale":"2023-11-28T20:33:59.590144+0000","last_undegraded":"2023-11-28T20:33:59.590144+0000","last_fullsized":"2023-11-28T20:33:59.590144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:24.263445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152745+0000","last_change":"2023-11-28T20:33:53.964226+0000","last_active":"2023-11-28T20:33:58.152745+0000","last_peered":"2023-11-28T20:33:58.152745+0000","last_clean":"2023-11-28T20:33:58.152745+0000","last_became_active":"2023-11-28T20:33:53.963614+0000","last_became_peered":"2023-11-28T20:33:53.963614+0000","last_unstale":"2023-11-28T20:33:58.152745+0000","last_undegraded":"2023-11-28T20:33:58.152745+0000","last_fullsized":"2023-11-28T20:33:58.152745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:20.194268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099544+0000","last_change":"2023-11-28T20:33:53.049547+0000","last_active":"2023-11-28T20:33:58.099544+0000","last_peered":"2023-11-28T20:33:58.099544+0000","last_clean":"2023-11-28T20:33:58.099544+0000","last_became_active":"2023-11-28T20:33:53.049252+0000","last_became_peered":"2023-11-28T20:33:53.049252+0000","last_unstale":"2023-11-28T20:33:58.099544+0000","last_undegraded":"2023-11-28T20:33:58.099544+0000","last_fullsized":"2023-11-28T20:33:58.099544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:41.371596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.252105+0000","last_change":"2023-11-28T20:33:58.158216+0000","last_active":"2023-11-28T20:33:59.252105+0000","last_peered":"2023-11-28T20:33:59.252105+0000","last_clean":"2023-11-28T20:33:59.252105+0000","last_became_active":"2023-11-28T20:33:58.157621+0000","last_became_peered":"2023-11-28T20:33:58.157621+0000","last_unstale":"2023-11-28T20:33:59.252105+0000","last_undegraded":"2023-11-28T20:33:59.252105+0000","last_fullsized":"2023-11-28T20:33:59.252105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:43:13.638963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151918+0000","last_change":"2023-11-28T20:33:53.959472+0000","last_active":"2023-11-28T20:33:58.151918+0000","last_peered":"2023-11-28T20:33:58.151918+0000","last_clean":"2023-11-28T20:33:58.151918+0000","last_became_active":"2023-11-28T20:33:53.959110+0000","last_became_peered":"2023-11-28T20:33:53.959110+0000","last_unstale":"2023-11-28T20:33:58.151918+0000","last_undegraded":"2023-11-28T20:33:58.151918+0000","last_fullsized":"2023-11-28T20:33:58.151918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:41.830889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.104034+0000","last_change":"2023-11-28T20:33:53.047976+0000","last_active":"2023-11-28T20:33:58.104034+0000","last_peered":"2023-11-28T20:33:58.104034+0000","last_clean":"2023-11-28T20:33:58.104034+0000","last_became_active":"2023-11-28T20:33:53.047700+0000","last_became_peered":"2023-11-28T20:33:53.047700+0000","last_unstale":"2023-11-28T20:33:58.104034+0000","last_undegraded":"2023-11-28T20:33:58.104034+0000","last_fullsized":"2023-11-28T20:33:58.104034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:53.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.916144+0000","last_change":"2023-11-28T20:33:58.128125+0000","last_active":"2023-11-28T20:33:58.916144+0000","last_peered":"2023-11-28T20:33:58.916144+0000","last_clean":"2023-11-28T20:33:58.916144+0000","last_became_active":"2023-11-28T20:33:58.127738+0000","last_became_peered":"2023-11-28T20:33:58.127738+0000","last_unstale":"2023-11-28T20:33:58.916144+0000","last_undegraded":"2023-11-28T20:33:58.916144+0000","last_fullsized":"2023-11-28T20:33:58.916144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:29.532639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151544+0000","last_change":"2023-11-28T20:33:53.960235+0000","last_active":"2023-11-28T20:33:58.151544+0000","last_peered":"2023-11-28T20:33:58.151544+0000","last_clean":"2023-11-28T20:33:58.151544+0000","last_became_active":"2023-11-28T20:33:53.959892+0000","last_became_peered":"2023-11-28T20:33:53.959892+0000","last_unstale":"2023-11-28T20:33:58.151544+0000","last_undegraded":"2023-11-28T20:33:58.151544+0000","last_fullsized":"2023-11-28T20:33:58.151544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099168+0000","last_change":"2023-11-28T20:33:53.056980+0000","last_active":"2023-11-28T20:33:58.099168+0000","last_peered":"2023-11-28T20:33:58.099168+0000","last_clean":"2023-11-28T20:33:58.099168+0000","last_became_active":"2023-11-28T20:33:53.056663+0000","last_became_peered":"2023-11-28T20:33:53.056663+0000","last_unstale":"2023-11-28T20:33:58.099168+0000","last_undegraded":"2023-11-28T20:33:58.099168+0000","last_fullsized":"2023-11-28T20:33:58.099168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:11:01.063288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590096+0000","last_change":"2023-11-28T20:33:58.137476+0000","last_active":"2023-11-28T20:33:59.590096+0000","last_peered":"2023-11-28T20:33:59.590096+0000","last_clean":"2023-11-28T20:33:59.590096+0000","last_became_active":"2023-11-28T20:33:58.136106+0000","last_became_peered":"2023-11-28T20:33:58.136106+0000","last_unstale":"2023-11-28T20:33:59.590096+0000","last_undegraded":"2023-11-28T20:33:59.590096+0000","last_fullsized":"2023-11-28T20:33:59.590096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:39.912968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119486+0000","last_change":"2023-11-28T20:33:53.940374+0000","last_active":"2023-11-28T20:33:58.119486+0000","last_peered":"2023-11-28T20:33:58.119486+0000","last_clean":"2023-11-28T20:33:58.119486+0000","last_became_active":"2023-11-28T20:33:53.939873+0000","last_became_peered":"2023-11-28T20:33:53.939873+0000","last_unstale":"2023-11-28T20:33:58.119486+0000","last_undegraded":"2023-11-28T20:33:58.119486+0000","last_fullsized":"2023-11-28T20:33:58.119486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120431+0000","last_change":"2023-11-28T20:33:53.055672+0000","last_active":"2023-11-28T20:33:58.120431+0000","last_peered":"2023-11-28T20:33:58.120431+0000","last_clean":"2023-11-28T20:33:58.120431+0000","last_became_active":"2023-11-28T20:33:53.055320+0000","last_became_peered":"2023-11-28T20:33:53.055320+0000","last_unstale":"2023-11-28T20:33:58.120431+0000","last_undegraded":"2023-11-28T20:33:58.120431+0000","last_fullsized":"2023-11-28T20:33:58.120431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.920861+0000","last_change":"2023-11-28T20:33:58.162629+0000","last_active":"2023-11-28T20:33:58.920861+0000","last_peered":"2023-11-28T20:33:58.920861+0000","last_clean":"2023-11-28T20:33:58.920861+0000","last_became_active":"2023-11-28T20:33:58.162003+0000","last_became_peered":"2023-11-28T20:33:58.162003+0000","last_unstale":"2023-11-28T20:33:58.920861+0000","last_undegraded":"2023-11-28T20:33:58.920861+0000","last_fullsized":"2023-11-28T20:33:58.920861+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:31.738218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151734+0000","last_change":"2023-11-28T20:33:53.961916+0000","last_active":"2023-11-28T20:33:58.151734+0000","last_peered":"2023-11-28T20:33:58.151734+0000","last_clean":"2023-11-28T20:33:58.151734+0000","last_became_active":"2023-11-28T20:33:53.961577+0000","last_became_peered":"2023-11-28T20:33:53.961577+0000","last_unstale":"2023-11-28T20:33:58.151734+0000","last_undegraded":"2023-11-28T20:33:58.151734+0000","last_fullsized":"2023-11-28T20:33:58.151734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126069+0000","last_change":"2023-11-28T20:33:53.056677+0000","last_active":"2023-11-28T20:33:58.126069+0000","last_peered":"2023-11-28T20:33:58.126069+0000","last_clean":"2023-11-28T20:33:58.126069+0000","last_became_active":"2023-11-28T20:33:53.055737+0000","last_became_peered":"2023-11-28T20:33:53.055737+0000","last_unstale":"2023-11-28T20:33:58.126069+0000","last_undegraded":"2023-11-28T20:33:58.126069+0000","last_fullsized":"2023-11-28T20:33:58.126069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:34.437219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.919758+0000","last_change":"2023-11-28T20:33:58.155270+0000","last_active":"2023-11-28T20:33:58.919758+0000","last_peered":"2023-11-28T20:33:58.919758+0000","last_clean":"2023-11-28T20:33:58.919758+0000","last_became_active":"2023-11-28T20:33:58.153764+0000","last_became_peered":"2023-11-28T20:33:58.153764+0000","last_unstale":"2023-11-28T20:33:58.919758+0000","last_undegraded":"2023-11-28T20:33:58.919758+0000","last_fullsized":"2023-11-28T20:33:58.919758+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:29.020740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117654+0000","last_change":"2023-11-28T20:33:53.943637+0000","last_active":"2023-11-28T20:33:58.117654+0000","last_peered":"2023-11-28T20:33:58.117654+0000","last_clean":"2023-11-28T20:33:58.117654+0000","last_became_active":"2023-11-28T20:33:53.941418+0000","last_became_peered":"2023-11-28T20:33:53.941418+0000","last_unstale":"2023-11-28T20:33:58.117654+0000","last_undegraded":"2023-11-28T20:33:58.117654+0000","last_fullsized":"2023-11-28T20:33:58.117654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118661+0000","last_change":"2023-11-28T20:33:53.052522+0000","last_active":"2023-11-28T20:33:58.118661+0000","last_peered":"2023-11-28T20:33:58.118661+0000","last_clean":"2023-11-28T20:33:58.118661+0000","last_became_active":"2023-11-28T20:33:53.052202+0000","last_became_peered":"2023-11-28T20:33:53.052202+0000","last_unstale":"2023-11-28T20:33:58.118661+0000","last_undegraded":"2023-11-28T20:33:58.118661+0000","last_fullsized":"2023-11-28T20:33:58.118661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:46.454394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.253371+0000","last_change":"2023-11-28T20:33:58.157854+0000","last_active":"2023-11-28T20:33:59.253371+0000","last_peered":"2023-11-28T20:33:59.253371+0000","last_clean":"2023-11-28T20:33:59.253371+0000","last_became_active":"2023-11-28T20:33:58.157424+0000","last_became_peered":"2023-11-28T20:33:58.157424+0000","last_unstale":"2023-11-28T20:33:59.253371+0000","last_undegraded":"2023-11-28T20:33:59.253371+0000","last_fullsized":"2023-11-28T20:33:59.253371+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:41.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151781+0000","last_change":"2023-11-28T20:33:53.959159+0000","last_active":"2023-11-28T20:33:58.151781+0000","last_peered":"2023-11-28T20:33:58.151781+0000","last_clean":"2023-11-28T20:33:58.151781+0000","last_became_active":"2023-11-28T20:33:53.958752+0000","last_became_peered":"2023-11-28T20:33:53.958752+0000","last_unstale":"2023-11-28T20:33:58.151781+0000","last_undegraded":"2023-11-28T20:33:58.151781+0000","last_fullsized":"2023-11-28T20:33:58.151781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:21.394618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:34:13.400104+0000","last_change":"2023-11-28T20:33:53.051304+0000","last_active":"2023-11-28T20:34:13.400104+0000","last_peered":"2023-11-28T20:34:13.400104+0000","last_clean":"2023-11-28T20:34:13.400104+0000","last_became_active":"2023-11-28T20:33:53.050957+0000","last_became_peered":"2023-11-28T20:33:53.050957+0000","last_unstale":"2023-11-28T20:34:13.400104+0000","last_undegraded":"2023-11-28T20:34:13.400104+0000","last_fullsized":"2023-11-28T20:34:13.400104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:41.755062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.915782+0000","last_change":"2023-11-28T20:33:58.133335+0000","last_active":"2023-11-28T20:33:58.915782+0000","last_peered":"2023-11-28T20:33:58.915782+0000","last_clean":"2023-11-28T20:33:58.915782+0000","last_became_active":"2023-11-28T20:33:58.132968+0000","last_became_peered":"2023-11-28T20:33:58.132968+0000","last_unstale":"2023-11-28T20:33:58.915782+0000","last_undegraded":"2023-11-28T20:33:58.915782+0000","last_fullsized":"2023-11-28T20:33:58.915782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105225+0000","last_change":"2023-11-28T20:33:53.944947+0000","last_active":"2023-11-28T20:33:58.105225+0000","last_peered":"2023-11-28T20:33:58.105225+0000","last_clean":"2023-11-28T20:33:58.105225+0000","last_became_active":"2023-11-28T20:33:53.944492+0000","last_became_peered":"2023-11-28T20:33:53.944492+0000","last_unstale":"2023-11-28T20:33:58.105225+0000","last_undegraded":"2023-11-28T20:33:58.105225+0000","last_fullsized":"2023-11-28T20:33:58.105225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:42.768692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123500+0000","last_change":"2023-11-28T20:33:53.063285+0000","last_active":"2023-11-28T20:33:58.123500+0000","last_peered":"2023-11-28T20:33:58.123500+0000","last_clean":"2023-11-28T20:33:58.123500+0000","last_became_active":"2023-11-28T20:33:53.063017+0000","last_became_peered":"2023-11-28T20:33:53.063017+0000","last_unstale":"2023-11-28T20:33:58.123500+0000","last_undegraded":"2023-11-28T20:33:58.123500+0000","last_fullsized":"2023-11-28T20:33:58.123500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.917261+0000","last_change":"2023-11-28T20:33:58.125259+0000","last_active":"2023-11-28T20:33:58.917261+0000","last_peered":"2023-11-28T20:33:58.917261+0000","last_clean":"2023-11-28T20:33:58.917261+0000","last_became_active":"2023-11-28T20:33:58.124810+0000","last_became_peered":"2023-11-28T20:33:58.124810+0000","last_unstale":"2023-11-28T20:33:58.917261+0000","last_undegraded":"2023-11-28T20:33:58.917261+0000","last_fullsized":"2023-11-28T20:33:58.917261+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:22.122661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116919+0000","last_change":"2023-11-28T20:33:53.940927+0000","last_active":"2023-11-28T20:33:58.116919+0000","last_peered":"2023-11-28T20:33:58.116919+0000","last_clean":"2023-11-28T20:33:58.116919+0000","last_became_active":"2023-11-28T20:33:53.940527+0000","last_became_peered":"2023-11-28T20:33:53.940527+0000","last_unstale":"2023-11-28T20:33:58.116919+0000","last_undegraded":"2023-11-28T20:33:58.116919+0000","last_fullsized":"2023-11-28T20:33:58.116919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123801+0000","last_change":"2023-11-28T20:33:53.055804+0000","last_active":"2023-11-28T20:33:58.123801+0000","last_peered":"2023-11-28T20:33:58.123801+0000","last_clean":"2023-11-28T20:33:58.123801+0000","last_became_active":"2023-11-28T20:33:53.053184+0000","last_became_peered":"2023-11-28T20:33:53.053184+0000","last_unstale":"2023-11-28T20:33:58.123801+0000","last_undegraded":"2023-11-28T20:33:58.123801+0000","last_fullsized":"2023-11-28T20:33:58.123801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:57.471000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918702+0000","last_change":"2023-11-28T20:33:58.133079+0000","last_active":"2023-11-28T20:33:58.918702+0000","last_peered":"2023-11-28T20:33:58.918702+0000","last_clean":"2023-11-28T20:33:58.918702+0000","last_became_active":"2023-11-28T20:33:58.126051+0000","last_became_peered":"2023-11-28T20:33:58.126051+0000","last_unstale":"2023-11-28T20:33:58.918702+0000","last_undegraded":"2023-11-28T20:33:58.918702+0000","last_fullsized":"2023-11-28T20:33:58.918702+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:41.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123681+0000","last_change":"2023-11-28T20:33:53.958488+0000","last_active":"2023-11-28T20:33:58.123681+0000","last_peered":"2023-11-28T20:33:58.123681+0000","last_clean":"2023-11-28T20:33:58.123681+0000","last_became_active":"2023-11-28T20:33:53.958041+0000","last_became_peered":"2023-11-28T20:33:53.958041+0000","last_unstale":"2023-11-28T20:33:58.123681+0000","last_undegraded":"2023-11-28T20:33:58.123681+0000","last_fullsized":"2023-11-28T20:33:58.123681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:07.426176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126850+0000","last_change":"2023-11-28T20:33:53.054003+0000","last_active":"2023-11-28T20:33:58.126850+0000","last_peered":"2023-11-28T20:33:58.126850+0000","last_clean":"2023-11-28T20:33:58.126850+0000","last_became_active":"2023-11-28T20:33:53.053773+0000","last_became_peered":"2023-11-28T20:33:53.053773+0000","last_unstale":"2023-11-28T20:33:58.126850+0000","last_undegraded":"2023-11-28T20:33:58.126850+0000","last_fullsized":"2023-11-28T20:33:58.126850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:22.227195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.918600+0000","last_change":"2023-11-28T20:33:58.157661+0000","last_active":"2023-11-28T20:33:58.918600+0000","last_peered":"2023-11-28T20:33:58.918600+0000","last_clean":"2023-11-28T20:33:58.918600+0000","last_became_active":"2023-11-28T20:33:58.157274+0000","last_became_peered":"2023-11-28T20:33:58.157274+0000","last_unstale":"2023-11-28T20:33:58.918600+0000","last_undegraded":"2023-11-28T20:33:58.918600+0000","last_fullsized":"2023-11-28T20:33:58.918600+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:43:13.638963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.589987+0000","last_change":"2023-11-28T20:33:58.159599+0000","last_active":"2023-11-28T20:33:59.589987+0000","last_peered":"2023-11-28T20:33:59.589987+0000","last_clean":"2023-11-28T20:33:59.589987+0000","last_became_active":"2023-11-28T20:33:58.158483+0000","last_became_peered":"2023-11-28T20:33:58.158483+0000","last_unstale":"2023-11-28T20:33:59.589987+0000","last_undegraded":"2023-11-28T20:33:59.589987+0000","last_fullsized":"2023-11-28T20:33:59.589987+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:45.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.122123+0000","last_change":"2023-11-28T20:33:53.950947+0000","last_active":"2023-11-28T20:33:58.122123+0000","last_peered":"2023-11-28T20:33:58.122123+0000","last_clean":"2023-11-28T20:33:58.122123+0000","last_became_active":"2023-11-28T20:33:53.948976+0000","last_became_peered":"2023-11-28T20:33:53.948976+0000","last_unstale":"2023-11-28T20:33:58.122123+0000","last_undegraded":"2023-11-28T20:33:58.122123+0000","last_fullsized":"2023-11-28T20:33:58.122123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120814+0000","last_change":"2023-11-28T20:33:53.059457+0000","last_active":"2023-11-28T20:33:58.120814+0000","last_peered":"2023-11-28T20:33:58.120814+0000","last_clean":"2023-11-28T20:33:58.120814+0000","last_became_active":"2023-11-28T20:33:53.059116+0000","last_became_peered":"2023-11-28T20:33:53.059116+0000","last_unstale":"2023-11-28T20:33:58.120814+0000","last_undegraded":"2023-11-28T20:33:58.120814+0000","last_fullsized":"2023-11-28T20:33:58.120814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:32.036639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.590492+0000","last_change":"2023-11-28T20:33:58.141507+0000","last_active":"2023-11-28T20:33:59.590492+0000","last_peered":"2023-11-28T20:33:59.590492+0000","last_clean":"2023-11-28T20:33:59.590492+0000","last_became_active":"2023-11-28T20:33:58.141072+0000","last_became_peered":"2023-11-28T20:33:58.141072+0000","last_unstale":"2023-11-28T20:33:59.590492+0000","last_undegraded":"2023-11-28T20:33:59.590492+0000","last_fullsized":"2023-11-28T20:33:59.590492+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:12.387400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103356+0000","last_change":"2023-11-28T20:33:53.937448+0000","last_active":"2023-11-28T20:33:58.103356+0000","last_peered":"2023-11-28T20:33:58.103356+0000","last_clean":"2023-11-28T20:33:58.103356+0000","last_became_active":"2023-11-28T20:33:53.937097+0000","last_became_peered":"2023-11-28T20:33:53.937097+0000","last_unstale":"2023-11-28T20:33:58.103356+0000","last_undegraded":"2023-11-28T20:33:58.103356+0000","last_fullsized":"2023-11-28T20:33:58.103356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:33.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105061+0000","last_change":"2023-11-28T20:33:53.047089+0000","last_active":"2023-11-28T20:33:58.105061+0000","last_peered":"2023-11-28T20:33:58.105061+0000","last_clean":"2023-11-28T20:33:58.105061+0000","last_became_active":"2023-11-28T20:33:53.046299+0000","last_became_peered":"2023-11-28T20:33:53.046299+0000","last_unstale":"2023-11-28T20:33:58.105061+0000","last_undegraded":"2023-11-28T20:33:58.105061+0000","last_fullsized":"2023-11-28T20:33:58.105061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:00:47.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121518+0000","last_change":"2023-11-28T20:33:53.057990+0000","last_active":"2023-11-28T20:33:58.121518+0000","last_peered":"2023-11-28T20:33:58.121518+0000","last_clean":"2023-11-28T20:33:58.121518+0000","last_became_active":"2023-11-28T20:33:53.057437+0000","last_became_peered":"2023-11-28T20:33:53.057437+0000","last_unstale":"2023-11-28T20:33:58.121518+0000","last_undegraded":"2023-11-28T20:33:58.121518+0000","last_fullsized":"2023-11-28T20:33:58.121518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118455+0000","last_change":"2023-11-28T20:33:53.944979+0000","last_active":"2023-11-28T20:33:58.118455+0000","last_peered":"2023-11-28T20:33:58.118455+0000","last_clean":"2023-11-28T20:33:58.118455+0000","last_became_active":"2023-11-28T20:33:53.944587+0000","last_became_peered":"2023-11-28T20:33:53.944587+0000","last_unstale":"2023-11-28T20:33:58.118455+0000","last_undegraded":"2023-11-28T20:33:58.118455+0000","last_fullsized":"2023-11-28T20:33:58.118455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:59.226534+0000","last_change":"2023-11-28T20:33:58.128961+0000","last_active":"2023-11-28T20:33:59.226534+0000","last_peered":"2023-11-28T20:33:59.226534+0000","last_clean":"2023-11-28T20:33:59.226534+0000","last_became_active":"2023-11-28T20:33:58.128548+0000","last_became_peered":"2023-11-28T20:33:58.128548+0000","last_unstale":"2023-11-28T20:33:59.226534+0000","last_undegraded":"2023-11-28T20:33:59.226534+0000","last_fullsized":"2023-11-28T20:33:59.226534+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:39.520026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103995+0000","last_change":"2023-11-28T20:33:53.061966+0000","last_active":"2023-11-28T20:33:58.103995+0000","last_peered":"2023-11-28T20:33:58.103995+0000","last_clean":"2023-11-28T20:33:58.103995+0000","last_became_active":"2023-11-28T20:33:53.061538+0000","last_became_peered":"2023-11-28T20:33:53.061538+0000","last_unstale":"2023-11-28T20:33:58.103995+0000","last_undegraded":"2023-11-28T20:33:58.103995+0000","last_fullsized":"2023-11-28T20:33:58.103995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:51.996077+0000","last_clean_scrub_stamp":"2023-11-28T20:33:51.996077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:19.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.113040+0000","last_change":"2023-11-28T20:33:53.943755+0000","last_active":"2023-11-28T20:33:58.113040+0000","last_peered":"2023-11-28T20:33:58.113040+0000","last_clean":"2023-11-28T20:33:58.113040+0000","last_became_active":"2023-11-28T20:33:53.943464+0000","last_became_peered":"2023-11-28T20:33:53.943464+0000","last_unstale":"2023-11-28T20:33:58.113040+0000","last_undegraded":"2023-11-28T20:33:58.113040+0000","last_fullsized":"2023-11-28T20:33:58.113040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:42.385226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.919987+0000","last_change":"2023-11-28T20:33:58.142246+0000","last_active":"2023-11-28T20:33:58.919987+0000","last_peered":"2023-11-28T20:33:58.919987+0000","last_clean":"2023-11-28T20:33:58.919987+0000","last_became_active":"2023-11-28T20:33:58.134395+0000","last_became_peered":"2023-11-28T20:33:58.134395+0000","last_unstale":"2023-11-28T20:33:58.919987+0000","last_undegraded":"2023-11-28T20:33:58.919987+0000","last_fullsized":"2023-11-28T20:33:58.919987+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:57.078884+0000","last_clean_scrub_stamp":"2023-11-28T20:33:57.078884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:24:08.097260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.119379+0000","last_change":"2023-11-28T20:33:53.948082+0000","last_active":"2023-11-28T20:33:58.119379+0000","last_peered":"2023-11-28T20:33:58.119379+0000","last_clean":"2023-11-28T20:33:58.119379+0000","last_became_active":"2023-11-28T20:33:53.947760+0000","last_became_peered":"2023-11-28T20:33:53.947760+0000","last_unstale":"2023-11-28T20:33:58.119379+0000","last_undegraded":"2023-11-28T20:33:58.119379+0000","last_fullsized":"2023-11-28T20:33:58.119379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:35.450849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099225+0000","last_change":"2023-11-28T20:33:53.947745+0000","last_active":"2023-11-28T20:33:58.099225+0000","last_peered":"2023-11-28T20:33:58.099225+0000","last_clean":"2023-11-28T20:33:58.099225+0000","last_became_active":"2023-11-28T20:33:53.947452+0000","last_became_peered":"2023-11-28T20:33:53.947452+0000","last_unstale":"2023-11-28T20:33:58.099225+0000","last_undegraded":"2023-11-28T20:33:58.099225+0000","last_fullsized":"2023-11-28T20:33:58.099225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:42.768692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125984+0000","last_change":"2023-11-28T20:33:53.947868+0000","last_active":"2023-11-28T20:33:58.125984+0000","last_peered":"2023-11-28T20:33:58.125984+0000","last_clean":"2023-11-28T20:33:58.125984+0000","last_became_active":"2023-11-28T20:33:53.947304+0000","last_became_peered":"2023-11-28T20:33:53.947304+0000","last_unstale":"2023-11-28T20:33:58.125984+0000","last_undegraded":"2023-11-28T20:33:58.125984+0000","last_fullsized":"2023-11-28T20:33:58.125984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:02:18.053484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118368+0000","last_change":"2023-11-28T20:33:53.947122+0000","last_active":"2023-11-28T20:33:58.118368+0000","last_peered":"2023-11-28T20:33:58.118368+0000","last_clean":"2023-11-28T20:33:58.118368+0000","last_became_active":"2023-11-28T20:33:53.946556+0000","last_became_peered":"2023-11-28T20:33:53.946556+0000","last_unstale":"2023-11-28T20:33:58.118368+0000","last_undegraded":"2023-11-28T20:33:58.118368+0000","last_fullsized":"2023-11-28T20:33:58.118368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.109664+0000","last_change":"2023-11-28T20:33:53.946180+0000","last_active":"2023-11-28T20:33:58.109664+0000","last_peered":"2023-11-28T20:33:58.109664+0000","last_clean":"2023-11-28T20:33:58.109664+0000","last_became_active":"2023-11-28T20:33:53.945758+0000","last_became_peered":"2023-11-28T20:33:53.945758+0000","last_unstale":"2023-11-28T20:33:58.109664+0000","last_undegraded":"2023-11-28T20:33:58.109664+0000","last_fullsized":"2023-11-28T20:33:58.109664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123704+0000","last_change":"2023-11-28T20:33:53.950532+0000","last_active":"2023-11-28T20:33:58.123704+0000","last_peered":"2023-11-28T20:33:58.123704+0000","last_clean":"2023-11-28T20:33:58.123704+0000","last_became_active":"2023-11-28T20:33:53.950206+0000","last_became_peered":"2023-11-28T20:33:53.950206+0000","last_unstale":"2023-11-28T20:33:58.123704+0000","last_undegraded":"2023-11-28T20:33:58.123704+0000","last_fullsized":"2023-11-28T20:33:58.123704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126244+0000","last_change":"2023-11-28T20:33:53.958413+0000","last_active":"2023-11-28T20:33:58.126244+0000","last_peered":"2023-11-28T20:33:58.126244+0000","last_clean":"2023-11-28T20:33:58.126244+0000","last_became_active":"2023-11-28T20:33:53.958179+0000","last_became_peered":"2023-11-28T20:33:53.958179+0000","last_unstale":"2023-11-28T20:33:58.126244+0000","last_undegraded":"2023-11-28T20:33:58.126244+0000","last_fullsized":"2023-11-28T20:33:58.126244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.113900+0000","last_change":"2023-11-28T20:33:53.952207+0000","last_active":"2023-11-28T20:33:58.113900+0000","last_peered":"2023-11-28T20:33:58.113900+0000","last_clean":"2023-11-28T20:33:58.113900+0000","last_became_active":"2023-11-28T20:33:53.950011+0000","last_became_peered":"2023-11-28T20:33:53.950011+0000","last_unstale":"2023-11-28T20:33:58.113900+0000","last_undegraded":"2023-11-28T20:33:58.113900+0000","last_fullsized":"2023-11-28T20:33:58.113900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120660+0000","last_change":"2023-11-28T20:33:53.952356+0000","last_active":"2023-11-28T20:33:58.120660+0000","last_peered":"2023-11-28T20:33:58.120660+0000","last_clean":"2023-11-28T20:33:58.120660+0000","last_became_active":"2023-11-28T20:33:53.947787+0000","last_became_peered":"2023-11-28T20:33:53.947787+0000","last_unstale":"2023-11-28T20:33:58.120660+0000","last_undegraded":"2023-11-28T20:33:58.120660+0000","last_fullsized":"2023-11-28T20:33:58.120660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105059+0000","last_change":"2023-11-28T20:33:53.949973+0000","last_active":"2023-11-28T20:33:58.105059+0000","last_peered":"2023-11-28T20:33:58.105059+0000","last_clean":"2023-11-28T20:33:58.105059+0000","last_became_active":"2023-11-28T20:33:53.946758+0000","last_became_peered":"2023-11-28T20:33:53.946758+0000","last_unstale":"2023-11-28T20:33:58.105059+0000","last_undegraded":"2023-11-28T20:33:58.105059+0000","last_fullsized":"2023-11-28T20:33:58.105059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125895+0000","last_change":"2023-11-28T20:33:53.950994+0000","last_active":"2023-11-28T20:33:58.125895+0000","last_peered":"2023-11-28T20:33:58.125895+0000","last_clean":"2023-11-28T20:33:58.125895+0000","last_became_active":"2023-11-28T20:33:53.946524+0000","last_became_peered":"2023-11-28T20:33:53.946524+0000","last_unstale":"2023-11-28T20:33:58.125895+0000","last_undegraded":"2023-11-28T20:33:58.125895+0000","last_fullsized":"2023-11-28T20:33:58.125895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.118275+0000","last_change":"2023-11-28T20:33:53.939684+0000","last_active":"2023-11-28T20:33:58.118275+0000","last_peered":"2023-11-28T20:33:58.118275+0000","last_clean":"2023-11-28T20:33:58.118275+0000","last_became_active":"2023-11-28T20:33:53.939225+0000","last_became_peered":"2023-11-28T20:33:53.939225+0000","last_unstale":"2023-11-28T20:33:58.118275+0000","last_undegraded":"2023-11-28T20:33:58.118275+0000","last_fullsized":"2023-11-28T20:33:58.118275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125812+0000","last_change":"2023-11-28T20:33:53.951199+0000","last_active":"2023-11-28T20:33:58.125812+0000","last_peered":"2023-11-28T20:33:58.125812+0000","last_clean":"2023-11-28T20:33:58.125812+0000","last_became_active":"2023-11-28T20:33:53.949262+0000","last_became_peered":"2023-11-28T20:33:53.949262+0000","last_unstale":"2023-11-28T20:33:58.125812+0000","last_undegraded":"2023-11-28T20:33:58.125812+0000","last_fullsized":"2023-11-28T20:33:58.125812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:23:38.261296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152008+0000","last_change":"2023-11-28T20:33:53.958492+0000","last_active":"2023-11-28T20:33:58.152008+0000","last_peered":"2023-11-28T20:33:58.152008+0000","last_clean":"2023-11-28T20:33:58.152008+0000","last_became_active":"2023-11-28T20:33:53.958141+0000","last_became_peered":"2023-11-28T20:33:53.958141+0000","last_unstale":"2023-11-28T20:33:58.152008+0000","last_undegraded":"2023-11-28T20:33:58.152008+0000","last_fullsized":"2023-11-28T20:33:58.152008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:13.738167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.099160+0000","last_change":"2023-11-28T20:33:53.948508+0000","last_active":"2023-11-28T20:33:58.099160+0000","last_peered":"2023-11-28T20:33:58.099160+0000","last_clean":"2023-11-28T20:33:58.099160+0000","last_became_active":"2023-11-28T20:33:53.947303+0000","last_became_peered":"2023-11-28T20:33:53.947303+0000","last_unstale":"2023-11-28T20:33:58.099160+0000","last_undegraded":"2023-11-28T20:33:58.099160+0000","last_fullsized":"2023-11-28T20:33:58.099160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:23.240825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.127814+0000","last_change":"2023-11-28T20:33:53.939826+0000","last_active":"2023-11-28T20:33:58.127814+0000","last_peered":"2023-11-28T20:33:58.127814+0000","last_clean":"2023-11-28T20:33:58.127814+0000","last_became_active":"2023-11-28T20:33:53.939349+0000","last_became_peered":"2023-11-28T20:33:53.939349+0000","last_unstale":"2023-11-28T20:33:58.127814+0000","last_undegraded":"2023-11-28T20:33:58.127814+0000","last_fullsized":"2023-11-28T20:33:58.127814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:33:13.738167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.120426+0000","last_change":"2023-11-28T20:33:53.939569+0000","last_active":"2023-11-28T20:33:58.120426+0000","last_peered":"2023-11-28T20:33:58.120426+0000","last_clean":"2023-11-28T20:33:58.120426+0000","last_became_active":"2023-11-28T20:33:53.936362+0000","last_became_peered":"2023-11-28T20:33:53.936362+0000","last_unstale":"2023-11-28T20:33:58.120426+0000","last_undegraded":"2023-11-28T20:33:58.120426+0000","last_fullsized":"2023-11-28T20:33:58.120426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123031+0000","last_change":"2023-11-28T20:33:53.950757+0000","last_active":"2023-11-28T20:33:58.123031+0000","last_peered":"2023-11-28T20:33:58.123031+0000","last_clean":"2023-11-28T20:33:58.123031+0000","last_became_active":"2023-11-28T20:33:53.948014+0000","last_became_peered":"2023-11-28T20:33:53.948014+0000","last_unstale":"2023-11-28T20:33:58.123031+0000","last_undegraded":"2023-11-28T20:33:58.123031+0000","last_fullsized":"2023-11-28T20:33:58.123031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:25.463462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.117566+0000","last_change":"2023-11-28T20:33:53.938422+0000","last_active":"2023-11-28T20:33:58.117566+0000","last_peered":"2023-11-28T20:33:58.117566+0000","last_clean":"2023-11-28T20:33:58.117566+0000","last_became_active":"2023-11-28T20:33:53.936194+0000","last_became_peered":"2023-11-28T20:33:53.936194+0000","last_unstale":"2023-11-28T20:33:58.117566+0000","last_undegraded":"2023-11-28T20:33:58.117566+0000","last_fullsized":"2023-11-28T20:33:58.117566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:35.450849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103553+0000","last_change":"2023-11-28T20:33:53.946481+0000","last_active":"2023-11-28T20:33:58.103553+0000","last_peered":"2023-11-28T20:33:58.103553+0000","last_clean":"2023-11-28T20:33:58.103553+0000","last_became_active":"2023-11-28T20:33:53.946123+0000","last_became_peered":"2023-11-28T20:33:53.946123+0000","last_unstale":"2023-11-28T20:33:58.103553+0000","last_undegraded":"2023-11-28T20:33:58.103553+0000","last_fullsized":"2023-11-28T20:33:58.103553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:40.702003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.105089+0000","last_change":"2023-11-28T20:33:53.960696+0000","last_active":"2023-11-28T20:33:58.105089+0000","last_peered":"2023-11-28T20:33:58.105089+0000","last_clean":"2023-11-28T20:33:58.105089+0000","last_became_active":"2023-11-28T20:33:53.953779+0000","last_became_peered":"2023-11-28T20:33:53.953779+0000","last_unstale":"2023-11-28T20:33:58.105089+0000","last_undegraded":"2023-11-28T20:33:58.105089+0000","last_fullsized":"2023-11-28T20:33:58.105089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:47.468024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126504+0000","last_change":"2023-11-28T20:33:53.938502+0000","last_active":"2023-11-28T20:33:58.126504+0000","last_peered":"2023-11-28T20:33:58.126504+0000","last_clean":"2023-11-28T20:33:58.126504+0000","last_became_active":"2023-11-28T20:33:53.938133+0000","last_became_peered":"2023-11-28T20:33:53.938133+0000","last_unstale":"2023-11-28T20:33:58.126504+0000","last_undegraded":"2023-11-28T20:33:58.126504+0000","last_fullsized":"2023-11-28T20:33:58.126504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:33.749310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.126197+0000","last_change":"2023-11-28T20:33:53.961946+0000","last_active":"2023-11-28T20:33:58.126197+0000","last_peered":"2023-11-28T20:33:58.126197+0000","last_clean":"2023-11-28T20:33:58.126197+0000","last_became_active":"2023-11-28T20:33:53.961681+0000","last_became_peered":"2023-11-28T20:33:53.961681+0000","last_unstale":"2023-11-28T20:33:58.126197+0000","last_undegraded":"2023-11-28T20:33:58.126197+0000","last_fullsized":"2023-11-28T20:33:58.126197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152700+0000","last_change":"2023-11-28T20:33:53.963253+0000","last_active":"2023-11-28T20:33:58.152700+0000","last_peered":"2023-11-28T20:33:58.152700+0000","last_clean":"2023-11-28T20:33:58.152700+0000","last_became_active":"2023-11-28T20:33:53.961179+0000","last_became_peered":"2023-11-28T20:33:53.961179+0000","last_unstale":"2023-11-28T20:33:58.152700+0000","last_undegraded":"2023-11-28T20:33:58.152700+0000","last_fullsized":"2023-11-28T20:33:58.152700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:43:35.843791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.103213+0000","last_change":"2023-11-28T20:33:53.952078+0000","last_active":"2023-11-28T20:33:58.103213+0000","last_peered":"2023-11-28T20:33:58.103213+0000","last_clean":"2023-11-28T20:33:58.103213+0000","last_became_active":"2023-11-28T20:33:53.949626+0000","last_became_peered":"2023-11-28T20:33:53.949626+0000","last_unstale":"2023-11-28T20:33:58.103213+0000","last_undegraded":"2023-11-28T20:33:58.103213+0000","last_fullsized":"2023-11-28T20:33:58.103213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151515+0000","last_change":"2023-11-28T20:33:53.955688+0000","last_active":"2023-11-28T20:33:58.151515+0000","last_peered":"2023-11-28T20:33:58.151515+0000","last_clean":"2023-11-28T20:33:58.151515+0000","last_became_active":"2023-11-28T20:33:53.955384+0000","last_became_peered":"2023-11-28T20:33:53.955384+0000","last_unstale":"2023-11-28T20:33:58.151515+0000","last_undegraded":"2023-11-28T20:33:58.151515+0000","last_fullsized":"2023-11-28T20:33:58.151515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:15.712426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.125802+0000","last_change":"2023-11-28T20:33:53.958084+0000","last_active":"2023-11-28T20:33:58.125802+0000","last_peered":"2023-11-28T20:33:58.125802+0000","last_clean":"2023-11-28T20:33:58.125802+0000","last_became_active":"2023-11-28T20:33:53.953483+0000","last_became_peered":"2023-11-28T20:33:53.953483+0000","last_unstale":"2023-11-28T20:33:58.125802+0000","last_undegraded":"2023-11-28T20:33:58.125802+0000","last_fullsized":"2023-11-28T20:33:58.125802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:08.863799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.151629+0000","last_change":"2023-11-28T20:33:53.955124+0000","last_active":"2023-11-28T20:33:58.151629+0000","last_peered":"2023-11-28T20:33:58.151629+0000","last_clean":"2023-11-28T20:33:58.151629+0000","last_became_active":"2023-11-28T20:33:53.954422+0000","last_became_peered":"2023-11-28T20:33:53.954422+0000","last_unstale":"2023-11-28T20:33:58.151629+0000","last_undegraded":"2023-11-28T20:33:58.151629+0000","last_fullsized":"2023-11-28T20:33:58.151629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:19.573371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.121991+0000","last_change":"2023-11-28T20:33:53.957855+0000","last_active":"2023-11-28T20:33:58.121991+0000","last_peered":"2023-11-28T20:33:58.121991+0000","last_clean":"2023-11-28T20:33:58.121991+0000","last_became_active":"2023-11-28T20:33:53.955645+0000","last_became_peered":"2023-11-28T20:33:53.955645+0000","last_unstale":"2023-11-28T20:33:58.121991+0000","last_undegraded":"2023-11-28T20:33:58.121991+0000","last_fullsized":"2023-11-28T20:33:58.121991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:26.539012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.116851+0000","last_change":"2023-11-28T20:33:53.937690+0000","last_active":"2023-11-28T20:33:58.116851+0000","last_peered":"2023-11-28T20:33:58.116851+0000","last_clean":"2023-11-28T20:33:58.116851+0000","last_became_active":"2023-11-28T20:33:53.937374+0000","last_became_peered":"2023-11-28T20:33:53.937374+0000","last_unstale":"2023-11-28T20:33:58.116851+0000","last_undegraded":"2023-11-28T20:33:58.116851+0000","last_fullsized":"2023-11-28T20:33:58.116851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:13.807178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.123601+0000","last_change":"2023-11-28T20:33:53.950129+0000","last_active":"2023-11-28T20:33:58.123601+0000","last_peered":"2023-11-28T20:33:58.123601+0000","last_clean":"2023-11-28T20:33:58.123601+0000","last_became_active":"2023-11-28T20:33:53.947280+0000","last_became_peered":"2023-11-28T20:33:53.947280+0000","last_unstale":"2023-11-28T20:33:58.123601+0000","last_undegraded":"2023-11-28T20:33:58.123601+0000","last_fullsized":"2023-11-28T20:33:58.123601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:58.484630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-11-28T20:33:58.152690+0000","last_change":"2023-11-28T20:33:53.964432+0000","last_active":"2023-11-28T20:33:58.152690+0000","last_peered":"2023-11-28T20:33:58.152690+0000","last_clean":"2023-11-28T20:33:58.152690+0000","last_became_active":"2023-11-28T20:33:53.963967+0000","last_became_peered":"2023-11-28T20:33:53.963967+0000","last_unstale":"2023-11-28T20:33:58.152690+0000","last_undegraded":"2023-11-28T20:33:58.152690+0000","last_fullsized":"2023-11-28T20:33:58.152690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:53.009707+0000","last_clean_scrub_stamp":"2023-11-28T20:33:53.009707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:07.426176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":695413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7989999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.851}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3580000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5980000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5569999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.577}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4930000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.637}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8169999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7569999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9179999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":104463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.855}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":274432,"data_stored":104623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.788}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":695469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:34:30.766 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T20:34:30.766 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-28T20:34:30.766 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-28T20:34:30.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T20:34:31.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.019+0000 7fdf29200640 1 Processor -- start 2023-11-28T20:34:31.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.020+0000 7fdf29200640 1 -- start start 2023-11-28T20:34:31.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.021+0000 7fdf29200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf240b35b0 0x7fdf2415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.021+0000 7fdf29200640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2415bda0 con 0x7fdf240b1340 2023-11-28T20:34:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.021+0000 7fdf29200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2415c080 con 0x7fdf2415b850 2023-11-28T20:34:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.021+0000 7fdf29200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2415ee40 con 0x7fdf240b35b0 2023-11-28T20:34:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.022+0000 7fdf2359e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdf240b1340 0x7fdf2415f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51950/0 (socket says 172.21.15.45:51950) 2023-11-28T20:34:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.022+0000 7fdf2359e640 1 -- 172.21.15.45:0/1441003283 learned_addr learned my addr 172.21.15.45:0/1441003283 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.022+0000 7fdf22d9d640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf240b35b0 0x7fdf2415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.022+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675912169 0 0) 0x7fdf2415bda0 con 0x7fdf240b1340 2023-11-28T20:34:31.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.022+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf0c003520 con 0x7fdf240b1340 2023-11-28T20:34:31.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735789773 0 0) 0x7fdf0c003520 con 0x7fdf240b1340 2023-11-28T20:34:31.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf240b35b0 msgr2=0x7fdf2415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:31.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf240b35b0 0x7fdf2415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 >> v1:172.21.15.190:6789/0 conn(0x7fdf2415b850 legacy=0x7fdf240b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2415c360 con 0x7fdf240b1340 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fdf1c0019f0 con 0x7fdf240b1340 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 >> v1:172.21.15.45:6789/0 conn(0x7fdf240b1340 legacy=0x7fdf2415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf0c0039a0 0x7fdf0c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c0047c0 0x7fdf0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf240b35b0 0x7fdf0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2415ee40 con 0x7fdf0c0047c0 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c00ba30 con 0x7fdf0c0039a0 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c00bbf0 con 0x7fdf240b35b0 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf2259c640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c0047c0 0x7fdf0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf2359e640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf240b35b0 0x7fdf0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.023+0000 7fdf22d9d640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf0c0039a0 0x7fdf0c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.024+0000 7fdf2259c640 1 -- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf240b35b0 msgr2=0x7fdf0c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.024+0000 7fdf2259c640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf240b35b0 0x7fdf0c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.024+0000 7fdf2259c640 1 -- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf0c0039a0 msgr2=0x7fdf0c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.024+0000 7fdf2259c640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf0c0039a0 0x7fdf0c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.024+0000 7fdf2259c640 1 -- 172.21.15.45:0/1441003283 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c003520 con 0x7fdf0c0047c0 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.025+0000 7fdf2259c640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c0047c0 0x7fdf0c004c40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdf2415bcd0 tx=0x7fdf0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542a3cdbf15b8886 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.025+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf180052d0 con 0x7fdf0c0047c0 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.025+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1802e050 con 0x7fdf0c0047c0 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.025+0000 7fdf21d9b640 1 -- 172.21.15.45:0/1441003283 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf180035f0 con 0x7fdf0c0047c0 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.026+0000 7fdf29200640 1 -- 172.21.15.45:0/1441003283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c0047c0 msgr2=0x7fdf0c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.026+0000 7fdf29200640 1 --2- 172.21.15.45:0/1441003283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf0c0047c0 0x7fdf0c004c40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdf2415bcd0 tx=0x7fdf0c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.026+0000 7fdf2359e640 1 -- 172.21.15.45:0/1441003283 reap_dead start 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.026+0000 7fdf29200640 1 -- 172.21.15.45:0/1441003283 shutdown_connections 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.026+0000 7fdf29200640 1 -- 172.21.15.45:0/1441003283 >> 172.21.15.45:0/1441003283 conn(0x7fdf2406a500 msgr2=0x7fdf2406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.026+0000 7fdf29200640 1 -- 172.21.15.45:0/1441003283 shutdown_connections 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.026+0000 7fdf29200640 1 -- 172.21.15.45:0/1441003283 wait complete. 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.028+0000 7fdf29200640 1 Processor -- start 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.028+0000 7fdf29200640 1 -- start start 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf29200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf0c0047c0 0x7fdf24154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf29200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf24156c30 0x7fdf24141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf29200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf24158220 0x7fdf241586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf29200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24157d90 con 0x7fdf24158220 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf29200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24157180 con 0x7fdf0c0047c0 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf29200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24157460 con 0x7fdf24156c30 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf2359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf24158220 0x7fdf241586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf2359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf24158220 0x7fdf241586a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59120/0 (socket says 172.21.15.45:59120) 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf2359e640 1 -- 172.21.15.45:0/814459864 learned_addr learned my addr 172.21.15.45:0/814459864 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf22d9d640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf0c0047c0 0x7fdf24154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.029+0000 7fdf2259c640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf24156c30 0x7fdf24141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf2359e640 1 -- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf24156c30 msgr2=0x7fdf24141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf2359e640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf24156c30 0x7fdf24141580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf2359e640 1 -- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf0c0047c0 msgr2=0x7fdf24154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf2359e640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf0c0047c0 0x7fdf24154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf2359e640 1 -- 172.21.15.45:0/814459864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf24157740 con 0x7fdf24158220 2023-11-28T20:34:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf2359e640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf24158220 0x7fdf241586a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdf2406e440 tx=0x7fdf1c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf0b7fe640 1 -- 172.21.15.45:0/814459864 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c00ec10 con 0x7fdf24158220 2023-11-28T20:34:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf0b7fe640 1 -- 172.21.15.45:0/814459864 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c00ed70 con 0x7fdf24158220 2023-11-28T20:34:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf0b7fe640 1 -- 172.21.15.45:0/814459864 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c00eee0 con 0x7fdf24158220 2023-11-28T20:34:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf24157a20 con 0x7fdf24158220 2023-11-28T20:34:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.030+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf24141ac0 con 0x7fdf24158220 2023-11-28T20:34:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.031+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf240733d0 con 0x7fdf24158220 2023-11-28T20:34:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.032+0000 7fdf0b7fe640 1 -- 172.21.15.45:0/814459864 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdf1c013660 con 0x7fdf24158220 2023-11-28T20:34:31.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.033+0000 7fdf0b7fe640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf1006ae40 0x7fdf1006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.033+0000 7fdf0b7fe640 1 -- 172.21.15.45:0/814459864 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdf1c089ea0 con 0x7fdf24158220 2023-11-28T20:34:31.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.034+0000 7fdf22d9d640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf1006ae40 0x7fdf1006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.035+0000 7fdf22d9d640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf1006ae40 0x7fdf1006d300 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009a10 tx=0x7fdf14008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:31.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.038+0000 7fdf0b7fe640 1 -- 172.21.15.45:0/814459864 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf1c054bf0 con 0x7fdf24158220 2023-11-28T20:34:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.274+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdf240cd3b0 con 0x7fdf24158220 2023-11-28T20:34:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.275+0000 7fdf0b7fe640 1 -- 172.21.15.45:0/814459864 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fdf1c058210 con 0x7fdf24158220 2023-11-28T20:34:31.278 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:31.278 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-28T20:34:31.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf1006ae40 msgr2=0x7fdf1006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf1006ae40 0x7fdf1006d300 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009a10 tx=0x7fdf14008040 comp rx=0 tx=0).stop 2023-11-28T20:34:31.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf24158220 msgr2=0x7fdf241586a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf24158220 0x7fdf241586a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdf2406e440 tx=0x7fdf1c00e530 comp rx=0 tx=0).stop 2023-11-28T20:34:31.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 shutdown_connections 2023-11-28T20:34:31.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fdf1006ae40 0x7fdf1006d300 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf24158220 0x7fdf241586a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf24156c30 0x7fdf24141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 --2- 172.21.15.45:0/814459864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf0c0047c0 0x7fdf24154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.278+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 >> 172.21.15.45:0/814459864 conn(0x7fdf2406a500 msgr2=0x7fdf24143b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:31.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.279+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 shutdown_connections 2023-11-28T20:34:31.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.279+0000 7fdf29200640 1 -- 172.21.15.45:0/814459864 wait complete. 2023-11-28T20:34:31.299 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-28T20:34:31.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.508+0000 7f992b59e640 1 Processor -- start 2023-11-28T20:34:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.508+0000 7f992b59e640 1 -- start start 2023-11-28T20:34:31.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.510+0000 7f992b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f992c14b090 0x7f992c0b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.510+0000 7f992b59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992c0abf10 con 0x7f992c0ad3b0 2023-11-28T20:34:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.510+0000 7f992b59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992c06db90 con 0x7f992c0ae590 2023-11-28T20:34:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.510+0000 7f992b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992c166070 con 0x7f992c14b090 2023-11-28T20:34:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.510+0000 7f992ad9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f992c0ad3b0 0x7f992c165960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51952/0 (socket says 172.21.15.45:51952) 2023-11-28T20:34:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.510+0000 7f992ad9d640 1 -- 172.21.15.45:0/1060271467 learned_addr learned my addr 172.21.15.45:0/1060271467 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.510+0000 7f992a59c640 1 --2- 172.21.15.45:0/1060271467 >> v2:172.21.15.190:3300/0 conn(0x7f992c14b090 0x7f992c0b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146351984 0 0) 0x7f992c0abf10 con 0x7f992c0ad3b0 2023-11-28T20:34:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9918003520 con 0x7f992c0ad3b0 2023-11-28T20:34:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3181874090 0 0) 0x7f9918003520 con 0x7f992c0ad3b0 2023-11-28T20:34:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 >> v2:172.21.15.190:3300/0 conn(0x7f992c14b090 msgr2=0x7f992c0b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 --2- 172.21.15.45:0/1060271467 >> v2:172.21.15.190:3300/0 conn(0x7f992c14b090 0x7f992c0b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 >> v1:172.21.15.190:6789/0 conn(0x7f992c0ae590 legacy=0x7f992c165250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c06d2f0 con 0x7f992c0ad3b0 2023-11-28T20:34:31.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f9924001670 con 0x7f992c0ad3b0 2023-11-28T20:34:31.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 >> v1:172.21.15.45:6789/0 conn(0x7f992c0ad3b0 legacy=0x7f992c165960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9918003970 0x7f9918003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99180047c0 0x7f9918004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9918005180 0x7f991800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c166070 con 0x7f9918003970 2023-11-28T20:34:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991800c020 con 0x7f9918005180 2023-11-28T20:34:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.511+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991800c1e0 con 0x7f99180047c0 2023-11-28T20:34:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.512+0000 7f992a59c640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9918003970 0x7f9918003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.512+0000 7f992ad9d640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9918005180 0x7f991800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.512+0000 7f9929d9b640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99180047c0 0x7f9918004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.512+0000 7f992a59c640 1 -- 172.21.15.45:0/1060271467 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99180047c0 msgr2=0x7f9918004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.512+0000 7f992a59c640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99180047c0 0x7f9918004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.512+0000 7f992a59c640 1 -- 172.21.15.45:0/1060271467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9918005180 msgr2=0x7f991800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.512+0000 7f992a59c640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9918005180 0x7f991800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.512+0000 7f992a59c640 1 -- 172.21.15.45:0/1060271467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918003520 con 0x7f9918003970 2023-11-28T20:34:31.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.512+0000 7f992ad9d640 1 -- 172.21.15.45:0/1060271467 reap_dead start 2023-11-28T20:34:31.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.513+0000 7f992a59c640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9918003970 0x7f9918003d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f991c009fc0 tx=0x7f99180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814751295d27c100 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:31.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.513+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c02cc20 con 0x7f9918003970 2023-11-28T20:34:31.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.513+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991c00f040 con 0x7f9918003970 2023-11-28T20:34:31.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.513+0000 7f992959a640 1 -- 172.21.15.45:0/1060271467 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c034440 con 0x7f9918003970 2023-11-28T20:34:31.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.514+0000 7f992b59e640 1 -- 172.21.15.45:0/1060271467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9918003970 msgr2=0x7f9918003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.514+0000 7f992b59e640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9918003970 0x7f9918003d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f991c009fc0 tx=0x7f99180057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.514+0000 7f992b59e640 1 -- 172.21.15.45:0/1060271467 shutdown_connections 2023-11-28T20:34:31.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.514+0000 7f992b59e640 1 --2- 172.21.15.45:0/1060271467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9918003970 0x7f9918003d90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.514+0000 7f992b59e640 1 -- 172.21.15.45:0/1060271467 >> 172.21.15.45:0/1060271467 conn(0x7f992c06ab30 msgr2=0x7f992c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:31.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.514+0000 7f992b59e640 1 -- 172.21.15.45:0/1060271467 shutdown_connections 2023-11-28T20:34:31.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.514+0000 7f992b59e640 1 -- 172.21.15.45:0/1060271467 wait complete. 2023-11-28T20:34:31.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.516+0000 7f992b59e640 1 Processor -- start 2023-11-28T20:34:31.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.516+0000 7f992b59e640 1 -- start start 2023-11-28T20:34:31.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.517+0000 7f992b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9918003970 0x7f992c1611c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.517+0000 7f992b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f992c148570 0x7f992c1ed2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.517+0000 7f992b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992c138940 0x7f992c138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.517+0000 7f992b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c161c80 con 0x7f992c138940 2023-11-28T20:34:31.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.517+0000 7f992b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c165900 con 0x7f992c148570 2023-11-28T20:34:31.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.517+0000 7f992b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992c165be0 con 0x7f9918003970 2023-11-28T20:34:31.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992c138940 0x7f992c138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992c138940 0x7f992c138dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59138/0 (socket says 172.21.15.45:59138) 2023-11-28T20:34:31.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992ad9d640 1 -- 172.21.15.45:0/1891250886 learned_addr learned my addr 172.21.15.45:0/1891250886 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:31.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f9929d9b640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f992c148570 0x7f992c1ed2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992a59c640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9918003970 0x7f992c1611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992ad9d640 1 -- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9918003970 msgr2=0x7f992c1611c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992ad9d640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9918003970 0x7f992c1611c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992ad9d640 1 -- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f992c148570 msgr2=0x7f992c1ed2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992ad9d640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f992c148570 0x7f992c1ed2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992ad9d640 1 -- 172.21.15.45:0/1891250886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c161ed0 con 0x7f992c138940 2023-11-28T20:34:31.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.518+0000 7f992a59c640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9918003970 0x7f992c1611c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:31.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.519+0000 7f992ad9d640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992c138940 0x7f992c138dc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9924001050 tx=0x7f992400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:31.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.519+0000 7f9922ffd640 1 -- 172.21.15.45:0/1891250886 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924010510 con 0x7f992c138940 2023-11-28T20:34:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.519+0000 7f9922ffd640 1 -- 172.21.15.45:0/1891250886 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9924010670 con 0x7f992c138940 2023-11-28T20:34:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.519+0000 7f9922ffd640 1 -- 172.21.15.45:0/1891250886 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924010a20 con 0x7f992c138940 2023-11-28T20:34:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.519+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992c1621b0 con 0x7f992c138940 2023-11-28T20:34:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.519+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992c1626d0 con 0x7f992c138940 2023-11-28T20:34:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.520+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992c1623a0 con 0x7f992c138940 2023-11-28T20:34:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.521+0000 7f9922ffd640 1 -- 172.21.15.45:0/1891250886 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9924010b80 con 0x7f992c138940 2023-11-28T20:34:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.522+0000 7f9922ffd640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f990c06ae70 0x7f990c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.522+0000 7f9922ffd640 1 -- 172.21.15.45:0/1891250886 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9924089050 con 0x7f992c138940 2023-11-28T20:34:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.523+0000 7f992a59c640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f990c06ae70 0x7f990c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.524+0000 7f992a59c640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f990c06ae70 0x7f990c06d330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f991c036000 tx=0x7f991c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:31.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.526+0000 7f9922ffd640 1 -- 172.21.15.45:0/1891250886 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99240bbe90 con 0x7f992c138940 2023-11-28T20:34:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.720+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f992c09d0f0 con 0x7f992c138940 2023-11-28T20:34:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.723+0000 7f9922ffd640 1 -- 172.21.15.45:0/1891250886 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7f9924019070 con 0x7f992c138940 2023-11-28T20:34:31.726 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:31.726 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.725+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f990c06ae70 msgr2=0x7f990c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.725+0000 7f992b59e640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f990c06ae70 0x7f990c06d330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f991c036000 tx=0x7f991c02e000 comp rx=0 tx=0).stop 2023-11-28T20:34:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.725+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992c138940 msgr2=0x7f992c138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.725+0000 7f992b59e640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992c138940 0x7f992c138dc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9924001050 tx=0x7f992400e960 comp rx=0 tx=0).stop 2023-11-28T20:34:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.726+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 shutdown_connections 2023-11-28T20:34:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.726+0000 7f992b59e640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f990c06ae70 0x7f990c06d330 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.726+0000 7f992b59e640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992c138940 0x7f992c138dc0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.726+0000 7f992b59e640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f992c148570 0x7f992c1ed2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.726+0000 7f992b59e640 1 --2- 172.21.15.45:0/1891250886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9918003970 0x7f992c1611c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.726+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 >> 172.21.15.45:0/1891250886 conn(0x7f992c06ab30 msgr2=0x7f992c0703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.726+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 shutdown_connections 2023-11-28T20:34:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.726+0000 7f992b59e640 1 -- 172.21.15.45:0/1891250886 wait complete. 2023-11-28T20:34:31.747 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:34:31.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:34:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.968+0000 7f8e3bf53640 1 Processor -- start 2023-11-28T20:34:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.968+0000 7f8e3bf53640 1 -- start start 2023-11-28T20:34:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.969+0000 7f8e3bf53640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340afb60 0x7f8e340b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.969+0000 7f8e3bf53640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8e340b01a0 0x7f8e340b0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.969+0000 7f8e3bf53640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e340c9950 con 0x7f8e340cc900 2023-11-28T20:34:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.969+0000 7f8e3bf53640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3415daa0 con 0x7f8e340b01a0 2023-11-28T20:34:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.969+0000 7f8e3bf53640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e340b0b50 con 0x7f8e340afb60 2023-11-28T20:34:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.970+0000 7f8e39cef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340afb60 0x7f8e340b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.970+0000 7f8e3a4f0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8e340b01a0 0x7f8e340b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.970+0000 7f8e39cef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340afb60 0x7f8e340b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:51256/0 (socket says 172.21.15.45:51256) 2023-11-28T20:34:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.970+0000 7f8e39cef640 1 -- 172.21.15.45:0/1850293897 learned_addr learned my addr 172.21.15.45:0/1850293897 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.970+0000 7f8e39cef640 1 -- 172.21.15.45:0/1850293897 >> v2:172.21.15.190:3300/0 conn(0x7f8e340b01a0 msgr2=0x7f8e340b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.970+0000 7f8e39cef640 1 --2- 172.21.15.45:0/1850293897 >> v2:172.21.15.190:3300/0 conn(0x7f8e340b01a0 0x7f8e340b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.970+0000 7f8e39cef640 1 -- 172.21.15.45:0/1850293897 >> v1:172.21.15.190:6789/0 conn(0x7f8e340cc900 legacy=0x7f8e340ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.970+0000 7f8e39cef640 1 -- 172.21.15.45:0/1850293897 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e340c9c30 con 0x7f8e340afb60 2023-11-28T20:34:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.972+0000 7f8e39cef640 1 --2- 172.21.15.45:0/1850293897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340afb60 0x7f8e340b20f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009e50 tx=0x7f8e3002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64875d5c542609f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.972+0000 7f8e38ced640 1 -- 172.21.15.45:0/1850293897 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3002cb00 con 0x7f8e340afb60 2023-11-28T20:34:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.972+0000 7f8e38ced640 1 -- 172.21.15.45:0/1850293897 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3002cc60 con 0x7f8e340afb60 2023-11-28T20:34:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.972+0000 7f8e38ced640 1 -- 172.21.15.45:0/1850293897 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e300343f0 con 0x7f8e340afb60 2023-11-28T20:34:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.973+0000 7f8e3bf53640 1 -- 172.21.15.45:0/1850293897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340afb60 msgr2=0x7f8e340b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.973+0000 7f8e3bf53640 1 --2- 172.21.15.45:0/1850293897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340afb60 0x7f8e340b20f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009e50 tx=0x7f8e3002bf60 comp rx=0 tx=0).stop 2023-11-28T20:34:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.973+0000 7f8e3bf53640 1 -- 172.21.15.45:0/1850293897 shutdown_connections 2023-11-28T20:34:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.973+0000 7f8e3bf53640 1 --2- 172.21.15.45:0/1850293897 >> v2:172.21.15.190:3300/0 conn(0x7f8e340b01a0 0x7f8e340b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.973+0000 7f8e3bf53640 1 --2- 172.21.15.45:0/1850293897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340afb60 0x7f8e340b20f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.973+0000 7f8e3bf53640 1 -- 172.21.15.45:0/1850293897 >> 172.21.15.45:0/1850293897 conn(0x7f8e34071a40 msgr2=0x7f8e34071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.973+0000 7f8e3bf53640 1 -- 172.21.15.45:0/1850293897 shutdown_connections 2023-11-28T20:34:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.973+0000 7f8e3bf53640 1 -- 172.21.15.45:0/1850293897 wait complete. 2023-11-28T20:34:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.975+0000 7f8e3bf53640 1 Processor -- start 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.975+0000 7f8e3bf53640 1 -- start start 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.976+0000 7f8e3bf53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e340afb60 0x7f8e340c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.976+0000 7f8e3bf53640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340b01a0 0x7f8e341f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.976+0000 7f8e3bf53640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e340cc900 0x7f8e341f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.976+0000 7f8e3bf53640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e34160ab0 con 0x7f8e340afb60 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.976+0000 7f8e3bf53640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e341f6d50 con 0x7f8e340cc900 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.976+0000 7f8e3bf53640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e34151990 con 0x7f8e340b01a0 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.976+0000 7f8e39cef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e340afb60 0x7f8e340c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.976+0000 7f8e39cef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e340afb60 0x7f8e340c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59146/0 (socket says 172.21.15.45:59146) 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.976+0000 7f8e39cef640 1 -- 172.21.15.45:0/3622757627 learned_addr learned my addr 172.21.15.45:0/3622757627 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.977+0000 7f8e3a4f0640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e340cc900 0x7f8e341f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.977+0000 7f8e394ee640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340b01a0 0x7f8e341f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.977+0000 7f8e39cef640 1 -- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340b01a0 msgr2=0x7f8e341f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.977+0000 7f8e39cef640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340b01a0 0x7f8e341f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.977+0000 7f8e39cef640 1 -- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e340cc900 msgr2=0x7f8e341f6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.977+0000 7f8e39cef640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e340cc900 0x7f8e341f6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.977+0000 7f8e39cef640 1 -- 172.21.15.45:0/3622757627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e30009b30 con 0x7f8e340afb60 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.977+0000 7f8e39cef640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e340afb60 0x7f8e340c8d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8e30036040 tx=0x7f8e30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.978+0000 7f8e2a7fc640 1 -- 172.21.15.45:0/3622757627 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30009510 con 0x7f8e340afb60 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.978+0000 7f8e2a7fc640 1 -- 172.21.15.45:0/3622757627 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e30040470 con 0x7f8e340afb60 2023-11-28T20:34:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.978+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e340b22c0 con 0x7f8e340afb60 2023-11-28T20:34:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.978+0000 7f8e2a7fc640 1 -- 172.21.15.45:0/3622757627 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30040800 con 0x7f8e340afb60 2023-11-28T20:34:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.978+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e34151c10 con 0x7f8e340afb60 2023-11-28T20:34:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.979+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e340733d0 con 0x7f8e340afb60 2023-11-28T20:34:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.980+0000 7f8e2a7fc640 1 -- 172.21.15.45:0/3622757627 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8e30040960 con 0x7f8e340afb60 2023-11-28T20:34:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.981+0000 7f8e2a7fc640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8e1806ae40 0x7f8e1806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.981+0000 7f8e2a7fc640 1 -- 172.21.15.45:0/3622757627 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8e300ac980 con 0x7f8e340afb60 2023-11-28T20:34:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.982+0000 7f8e394ee640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8e1806ae40 0x7f8e1806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.983+0000 7f8e394ee640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8e1806ae40 0x7f8e1806d300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8e340c9760 tx=0x7f8e24009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:31.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:31.986+0000 7f8e2a7fc640 1 -- 172.21.15.45:0/3622757627 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e30077870 con 0x7f8e340afb60 2023-11-28T20:34:32.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.180+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e340b13c0 con 0x7f8e340afb60 2023-11-28T20:34:32.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.181+0000 7f8e2a7fc640 1 -- 172.21.15.45:0/3622757627 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e3007ae90 con 0x7f8e340afb60 2023-11-28T20:34:32.183 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:34:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8e1806ae40 msgr2=0x7f8e1806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8e1806ae40 0x7f8e1806d300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8e340c9760 tx=0x7f8e24009040 comp rx=0 tx=0).stop 2023-11-28T20:34:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e340afb60 msgr2=0x7f8e340c8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e340afb60 0x7f8e340c8d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8e30036040 tx=0x7f8e30004a40 comp rx=0 tx=0).stop 2023-11-28T20:34:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 shutdown_connections 2023-11-28T20:34:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8e1806ae40 0x7f8e1806d300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e340cc900 0x7f8e341f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e340b01a0 0x7f8e341f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 --2- 172.21.15.45:0/3622757627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e340afb60 0x7f8e340c8d40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.183+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 >> 172.21.15.45:0/3622757627 conn(0x7f8e34071a40 msgr2=0x7f8e3406dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:32.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.184+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 shutdown_connections 2023-11-28T20:34:32.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.184+0000 7f8e3bf53640 1 -- 172.21.15.45:0/3622757627 wait complete. 2023-11-28T20:34:32.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:34:32.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.427+0000 7efdcf375640 1 Processor -- start 2023-11-28T20:34:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.427+0000 7efdcf375640 1 -- start start 2023-11-28T20:34:32.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.429+0000 7efdcf375640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efdc80b3040 0x7efdc806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.429+0000 7efdcf375640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efdc815b270 0x7efdc815f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.429+0000 7efdcf375640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efdc80b3420 0x7efdc80b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.429+0000 7efdcf375640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc80ceaa0 con 0x7efdc80b3420 2023-11-28T20:34:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.429+0000 7efdcf375640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc815faf0 con 0x7efdc80b3040 2023-11-28T20:34:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.429+0000 7efdcf375640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc806d410 con 0x7efdc815b270 2023-11-28T20:34:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdceb74640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efdc80b3420 0x7efdc80b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdceb74640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efdc80b3420 0x7efdc80b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59158/0 (socket says 172.21.15.45:59158) 2023-11-28T20:34:32.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdceb74640 1 -- 172.21.15.45:0/345661793 learned_addr learned my addr 172.21.15.45:0/345661793 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:32.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdce373640 1 --2- 172.21.15.45:0/345661793 >> v2:172.21.15.190:3300/0 conn(0x7efdc80b3040 0x7efdc806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdcdb72640 1 --2- 172.21.15.45:0/345661793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efdc815b270 0x7efdc815f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdceb74640 1 -- 172.21.15.45:0/345661793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efdc815b270 msgr2=0x7efdc815f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:32.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdceb74640 1 --2- 172.21.15.45:0/345661793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efdc815b270 0x7efdc815f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdceb74640 1 -- 172.21.15.45:0/345661793 >> v2:172.21.15.190:3300/0 conn(0x7efdc80b3040 msgr2=0x7efdc806cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:32.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdceb74640 1 --2- 172.21.15.45:0/345661793 >> v2:172.21.15.190:3300/0 conn(0x7efdc80b3040 0x7efdc806cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.430+0000 7efdceb74640 1 -- 172.21.15.45:0/345661793 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc806d6f0 con 0x7efdc80b3420 2023-11-28T20:34:32.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.431+0000 7efdceb74640 1 --2- 172.21.15.45:0/345661793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3420 0x7efdc80b1d20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efdc80b4410 tx=0x7efdc002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbeded85935508b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:32.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.432+0000 7efdcd371640 1 -- 172.21.15.45:0/345661793 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc002c950 con 0x7efdc80b3420 2023-11-28T20:34:32.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.432+0000 7efdcd371640 1 -- 172.21.15.45:0/345661793 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc002cab0 con 0x7efdc80b3420 2023-11-28T20:34:32.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.432+0000 7efdcd371640 1 -- 172.21.15.45:0/345661793 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc002ce40 con 0x7efdc80b3420 2023-11-28T20:34:32.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.432+0000 7efdcf375640 1 -- 172.21.15.45:0/345661793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3420 msgr2=0x7efdc80b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.432+0000 7efdcf375640 1 --2- 172.21.15.45:0/345661793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3420 0x7efdc80b1d20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efdc80b4410 tx=0x7efdc002bdb0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.433+0000 7efdcf375640 1 -- 172.21.15.45:0/345661793 shutdown_connections 2023-11-28T20:34:32.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.433+0000 7efdcf375640 1 --2- 172.21.15.45:0/345661793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3420 0x7efdc80b1d20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.433+0000 7efdcf375640 1 --2- 172.21.15.45:0/345661793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efdc815b270 0x7efdc815f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.433+0000 7efdcf375640 1 --2- 172.21.15.45:0/345661793 >> v2:172.21.15.190:3300/0 conn(0x7efdc80b3040 0x7efdc806cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.433+0000 7efdcf375640 1 -- 172.21.15.45:0/345661793 >> 172.21.15.45:0/345661793 conn(0x7efdc80b1050 msgr2=0x7efdc80cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.433+0000 7efdcf375640 1 -- 172.21.15.45:0/345661793 shutdown_connections 2023-11-28T20:34:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.433+0000 7efdcf375640 1 -- 172.21.15.45:0/345661793 wait complete. 2023-11-28T20:34:32.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.435+0000 7efdcf375640 1 Processor -- start 2023-11-28T20:34:32.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.435+0000 7efdcf375640 1 -- start start 2023-11-28T20:34:32.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdcf375640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3040 0x7efdc815b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdcf375640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efdc80b3420 0x7efdc81f0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdcf375640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc815b270 0x7efdc813e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdcf375640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc813d310 con 0x7efdc80b3040 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdcf375640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc813d560 con 0x7efdc815b270 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdcf375640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc813d840 con 0x7efdc80b3420 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdce373640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3040 0x7efdc815b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdce373640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3040 0x7efdc815b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59160/0 (socket says 172.21.15.45:59160) 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdce373640 1 -- 172.21.15.45:0/2584414791 learned_addr learned my addr 172.21.15.45:0/2584414791 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdce373640 1 -- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efdc80b3420 msgr2=0x7efdc81f0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdce373640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efdc80b3420 0x7efdc81f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdce373640 1 -- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc815b270 msgr2=0x7efdc813e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.436+0000 7efdceb74640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc815b270 0x7efdc813e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.437+0000 7efdce373640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc815b270 0x7efdc813e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.437+0000 7efdce373640 1 -- 172.21.15.45:0/2584414791 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc0009b30 con 0x7efdc80b3040 2023-11-28T20:34:32.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.437+0000 7efdce373640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3040 0x7efdc815b6a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efdc400d800 tx=0x7efdc400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:32.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.437+0000 7efdbeffd640 1 -- 172.21.15.45:0/2584414791 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc40096a0 con 0x7efdc80b3040 2023-11-28T20:34:32.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.437+0000 7efdbeffd640 1 -- 172.21.15.45:0/2584414791 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc4009800 con 0x7efdc80b3040 2023-11-28T20:34:32.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.437+0000 7efdbeffd640 1 -- 172.21.15.45:0/2584414791 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc4009990 con 0x7efdc80b3040 2023-11-28T20:34:32.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.437+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc813db20 con 0x7efdc80b3040 2023-11-28T20:34:32.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.437+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc813e0e0 con 0x7efdc80b3040 2023-11-28T20:34:32.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.438+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc813e270 con 0x7efdc80b3040 2023-11-28T20:34:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.439+0000 7efdbeffd640 1 -- 172.21.15.45:0/2584414791 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efdc4009af0 con 0x7efdc80b3040 2023-11-28T20:34:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.440+0000 7efdbeffd640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efdac06ae40 0x7efdac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.440+0000 7efdbeffd640 1 -- 172.21.15.45:0/2584414791 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efdc4089d80 con 0x7efdc80b3040 2023-11-28T20:34:32.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.441+0000 7efdcdb72640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efdac06ae40 0x7efdac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.442+0000 7efdcdb72640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efdac06ae40 0x7efdac06d300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efdb8006fd0 tx=0x7efdb8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:32.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.445+0000 7efdbeffd640 1 -- 172.21.15.45:0/2584414791 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efdc4054bf0 con 0x7efdc80b3040 2023-11-28T20:34:32.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.639+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efdc809d0f0 con 0x7efdc80b3040 2023-11-28T20:34:32.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.640+0000 7efdbeffd640 1 -- 172.21.15.45:0/2584414791 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7efdc4009da0 con 0x7efdc80b3040 2023-11-28T20:34:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:32.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efdac06ae40 msgr2=0x7efdac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efdac06ae40 0x7efdac06d300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efdb8006fd0 tx=0x7efdb8008040 comp rx=0 tx=0).stop 2023-11-28T20:34:32.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3040 msgr2=0x7efdc815b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3040 0x7efdc815b6a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efdc400d800 tx=0x7efdc400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 shutdown_connections 2023-11-28T20:34:32.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efdac06ae40 0x7efdac06d300 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc815b270 0x7efdc813e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efdc80b3420 0x7efdc81f0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 --2- 172.21.15.45:0/2584414791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc80b3040 0x7efdc815b6a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 >> 172.21.15.45:0/2584414791 conn(0x7efdc80b1050 msgr2=0x7efdc80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:32.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 shutdown_connections 2023-11-28T20:34:32.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.643+0000 7efdcf375640 1 -- 172.21.15.45:0/2584414791 wait complete. 2023-11-28T20:34:32.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:34:32.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.889+0000 7f7187fff640 1 Processor -- start 2023-11-28T20:34:32.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.889+0000 7f7187fff640 1 -- start start 2023-11-28T20:34:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.890+0000 7f7187fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f71880d0040 0x7f71880d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.891+0000 7f7187fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71880b90a0 con 0x7f71880b1d40 2023-11-28T20:34:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.891+0000 7f7187fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71880d1920 con 0x7f71880b2290 2023-11-28T20:34:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.891+0000 7f7187fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71880d3e60 con 0x7f71880d0040 2023-11-28T20:34:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.891+0000 7f7186ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f71880d0040 0x7f71880d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.891+0000 7f7186ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f71880d0040 0x7f71880d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59170/0 (socket says 172.21.15.45:59170) 2023-11-28T20:34:32.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.891+0000 7f7186ffd640 1 -- 172.21.15.45:0/1525227750 learned_addr learned my addr 172.21.15.45:0/1525227750 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.891+0000 7f7186ffd640 1 -- 172.21.15.45:0/1525227750 >> v1:172.21.15.190:6789/0 conn(0x7f71880b2290 legacy=0x7f71880d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.892+0000 7f7186ffd640 1 -- 172.21.15.45:0/1525227750 >> v1:172.21.15.45:6789/0 conn(0x7f71880b1d40 legacy=0x7f71880d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.892+0000 7f7186ffd640 1 -- 172.21.15.45:0/1525227750 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71880d2dc0 con 0x7f71880d0040 2023-11-28T20:34:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.893+0000 7f7186ffd640 1 --2- 172.21.15.45:0/1525227750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 0x7f71880d0420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7178009e50 tx=0x7f717802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746d7dfdaf20e38 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:32.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.893+0000 7f7185ffb640 1 -- 172.21.15.45:0/1525227750 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717802cbb0 con 0x7f71880d0040 2023-11-28T20:34:32.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.893+0000 7f7185ffb640 1 -- 172.21.15.45:0/1525227750 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717802cd10 con 0x7f71880d0040 2023-11-28T20:34:32.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.893+0000 7f7185ffb640 1 -- 172.21.15.45:0/1525227750 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7178034610 con 0x7f71880d0040 2023-11-28T20:34:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.894+0000 7f7187fff640 1 -- 172.21.15.45:0/1525227750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 msgr2=0x7f71880d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.894+0000 7f7187fff640 1 --2- 172.21.15.45:0/1525227750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 0x7f71880d0420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7178009e50 tx=0x7f717802c010 comp rx=0 tx=0).stop 2023-11-28T20:34:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.894+0000 7f7187fff640 1 -- 172.21.15.45:0/1525227750 shutdown_connections 2023-11-28T20:34:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.894+0000 7f7187fff640 1 --2- 172.21.15.45:0/1525227750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 0x7f71880d0420 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.894+0000 7f7187fff640 1 -- 172.21.15.45:0/1525227750 >> 172.21.15.45:0/1525227750 conn(0x7f7188070f40 msgr2=0x7f7188071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.894+0000 7f7187fff640 1 -- 172.21.15.45:0/1525227750 shutdown_connections 2023-11-28T20:34:32.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.894+0000 7f7187fff640 1 -- 172.21.15.45:0/1525227750 wait complete. 2023-11-28T20:34:32.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.896+0000 7f7187fff640 1 Processor -- start 2023-11-28T20:34:32.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.896+0000 7f7187fff640 1 -- start start 2023-11-28T20:34:32.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f7187fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71880b1d40 0x7f71880c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f7187fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71880b2290 0x7f71880c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f7187fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 0x7f71880cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f7187fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71880ca180 con 0x7f71880d0040 2023-11-28T20:34:32.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f7187fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71880c7960 con 0x7f71880b2290 2023-11-28T20:34:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f7187fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71880c7c00 con 0x7f71880b1d40 2023-11-28T20:34:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f71877fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 0x7f71880cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f71877fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 0x7f71880cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59178/0 (socket says 172.21.15.45:59178) 2023-11-28T20:34:32.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f71877fe640 1 -- 172.21.15.45:0/1559812315 learned_addr learned my addr 172.21.15.45:0/1559812315 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:32.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f7186ffd640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71880b1d40 0x7f71880c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.898+0000 7f71867fc640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71880b2290 0x7f71880c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f71877fe640 1 -- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71880b1d40 msgr2=0x7f71880c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f71877fe640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71880b1d40 0x7f71880c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f71877fe640 1 -- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71880b2290 msgr2=0x7f71880c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f71877fe640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71880b2290 0x7f71880c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f71877fe640 1 -- 172.21.15.45:0/1559812315 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7178009b30 con 0x7f71880d0040 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f71877fe640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 0x7f71880cc680 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f718000c390 tx=0x7f718000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f717ffff640 1 -- 172.21.15.45:0/1559812315 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71800052d0 con 0x7f71880d0040 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f717ffff640 1 -- 172.21.15.45:0/1559812315 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7180005430 con 0x7f71880d0040 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71880c7ee0 con 0x7f71880d0040 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f717ffff640 1 -- 172.21.15.45:0/1559812315 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180003810 con 0x7f71880d0040 2023-11-28T20:34:32.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.899+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71880ca390 con 0x7f71880d0040 2023-11-28T20:34:32.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.901+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7188071b30 con 0x7f71880d0040 2023-11-28T20:34:32.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.902+0000 7f717ffff640 1 -- 172.21.15.45:0/1559812315 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7180003970 con 0x7f71880d0040 2023-11-28T20:34:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.903+0000 7f717ffff640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f716806ae40 0x7f716806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.903+0000 7f717ffff640 1 -- 172.21.15.45:0/1559812315 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7180088e10 con 0x7f71880d0040 2023-11-28T20:34:32.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.903+0000 7f7186ffd640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f716806ae40 0x7f716806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:32.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.904+0000 7f7186ffd640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f716806ae40 0x7f716806d300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7178035000 tx=0x7f7178030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:32.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:32.907+0000 7f717ffff640 1 -- 172.21.15.45:0/1559812315 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7180053bd0 con 0x7f71880d0040 2023-11-28T20:34:33.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.105+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f718804e920 con 0x7f71880d0040 2023-11-28T20:34:33.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.105+0000 7f717ffff640 1 -- 172.21.15.45:0/1559812315 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f71800170a0 con 0x7f71880d0040 2023-11-28T20:34:33.108 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:34:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.108+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f716806ae40 msgr2=0x7f716806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.108+0000 7f7187fff640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f716806ae40 0x7f716806d300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7178035000 tx=0x7f7178030000 comp rx=0 tx=0).stop 2023-11-28T20:34:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.108+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 msgr2=0x7f71880cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.108+0000 7f7187fff640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 0x7f71880cc680 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f718000c390 tx=0x7f718000c850 comp rx=0 tx=0).stop 2023-11-28T20:34:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.109+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 shutdown_connections 2023-11-28T20:34:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.109+0000 7f7187fff640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f716806ae40 0x7f716806d300 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.109+0000 7f7187fff640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71880d0040 0x7f71880cc680 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.109+0000 7f7187fff640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71880b2290 0x7f71880c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.109+0000 7f7187fff640 1 --2- 172.21.15.45:0/1559812315 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71880b1d40 0x7f71880c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.109+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 >> 172.21.15.45:0/1559812315 conn(0x7f7188070f40 msgr2=0x7f71880b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:33.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.109+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 shutdown_connections 2023-11-28T20:34:33.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.109+0000 7f7187fff640 1 -- 172.21.15.45:0/1559812315 wait complete. 2023-11-28T20:34:33.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:34:33.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.353+0000 7fb75759e640 1 Processor -- start 2023-11-28T20:34:33.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.353+0000 7fb75759e640 1 -- start start 2023-11-28T20:34:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.355+0000 7fb75759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 0x7fb75806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.355+0000 7fb75759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7580b3420 0x7fb7580ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.355+0000 7fb75759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75806d410 con 0x7fb7580b3040 2023-11-28T20:34:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.355+0000 7fb75759e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7580ceaa0 con 0x7fb7580b3420 2023-11-28T20:34:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.355+0000 7fb75759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75806d6f0 con 0x7fb75815b270 2023-11-28T20:34:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb756d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7580b3420 0x7fb7580ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb75659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb7580b3040 0x7fb75815f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51970/0 (socket says 172.21.15.45:51970) 2023-11-28T20:34:33.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb75659c640 1 -- 172.21.15.45:0/3834713818 learned_addr learned my addr 172.21.15.45:0/3834713818 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb755d9b640 1 --2- 172.21.15.45:0/3834713818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 0x7fb75806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb756d9d640 1 -- 172.21.15.45:0/3834713818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 msgr2=0x7fb75806cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb756d9d640 1 --2- 172.21.15.45:0/3834713818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 0x7fb75806cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb756d9d640 1 -- 172.21.15.45:0/3834713818 >> v1:172.21.15.45:6789/0 conn(0x7fb7580b3040 legacy=0x7fb75815f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb756d9d640 1 -- 172.21.15.45:0/3834713818 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75806d9d0 con 0x7fb7580b3420 2023-11-28T20:34:33.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb75559a640 1 -- 172.21.15.45:0/3834713818 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404754102 0 0) 0x7fb75806d410 con 0x7fb7580b3040 2023-11-28T20:34:33.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.356+0000 7fb755d9b640 1 --2- 172.21.15.45:0/3834713818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 0x7fb75806cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:34:33.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.357+0000 7fb756d9d640 1 --2- 172.21.15.45:0/3834713818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 0x7fb7580ccc90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7500097a0 tx=0x7fb75002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6705795f94e7fcf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.358+0000 7fb75559a640 1 -- 172.21.15.45:0/3834713818 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7500316f0 con 0x7fb7580b3420 2023-11-28T20:34:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.358+0000 7fb75559a640 1 -- 172.21.15.45:0/3834713818 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb750031850 con 0x7fb7580b3420 2023-11-28T20:34:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.358+0000 7fb75559a640 1 -- 172.21.15.45:0/3834713818 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750031be0 con 0x7fb7580b3420 2023-11-28T20:34:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.358+0000 7fb75759e640 1 -- 172.21.15.45:0/3834713818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 msgr2=0x7fb7580ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.358+0000 7fb75759e640 1 --2- 172.21.15.45:0/3834713818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 0x7fb7580ccc90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7500097a0 tx=0x7fb75002b790 comp rx=0 tx=0).stop 2023-11-28T20:34:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.358+0000 7fb75759e640 1 -- 172.21.15.45:0/3834713818 shutdown_connections 2023-11-28T20:34:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.358+0000 7fb75759e640 1 --2- 172.21.15.45:0/3834713818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 0x7fb7580ccc90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.358+0000 7fb75759e640 1 --2- 172.21.15.45:0/3834713818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 0x7fb75806cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.358+0000 7fb75759e640 1 -- 172.21.15.45:0/3834713818 >> 172.21.15.45:0/3834713818 conn(0x7fb7580b1050 msgr2=0x7fb7580cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:33.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.359+0000 7fb75759e640 1 -- 172.21.15.45:0/3834713818 shutdown_connections 2023-11-28T20:34:33.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.359+0000 7fb75759e640 1 -- 172.21.15.45:0/3834713818 wait complete. 2023-11-28T20:34:33.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.360+0000 7fb75759e640 1 Processor -- start 2023-11-28T20:34:33.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.361+0000 7fb75759e640 1 -- start start 2023-11-28T20:34:33.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.361+0000 7fb75759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7580b3040 0x7fb75815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb75759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 0x7fb758150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb75759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 0x7fb7581598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb75759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75815b740 con 0x7fb7580b3420 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb75759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75815ba20 con 0x7fb7580b3040 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb75759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75815bd00 con 0x7fb75815b270 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb755d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 0x7fb758150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb755d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 0x7fb758150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59196/0 (socket says 172.21.15.45:59196) 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb755d9b640 1 -- 172.21.15.45:0/1425880751 learned_addr learned my addr 172.21.15.45:0/1425880751 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb75659c640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7580b3040 0x7fb75815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb756d9d640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 0x7fb7581598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb755d9b640 1 -- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 msgr2=0x7fb7581598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb755d9b640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 0x7fb7581598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb755d9b640 1 -- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7580b3040 msgr2=0x7fb75815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb755d9b640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7580b3040 0x7fb75815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.362+0000 7fb755d9b640 1 -- 172.21.15.45:0/1425880751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb750009400 con 0x7fb7580b3420 2023-11-28T20:34:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.363+0000 7fb755d9b640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 0x7fb758150950 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d800 tx=0x7fb74000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.363+0000 7fb74effd640 1 -- 172.21.15.45:0/1425880751 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740004030 con 0x7fb7580b3420 2023-11-28T20:34:33.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.363+0000 7fb74effd640 1 -- 172.21.15.45:0/1425880751 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb740004190 con 0x7fb7580b3420 2023-11-28T20:34:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.363+0000 7fb74effd640 1 -- 172.21.15.45:0/1425880751 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740004520 con 0x7fb7580b3420 2023-11-28T20:34:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.363+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75815bfe0 con 0x7fb7580b3420 2023-11-28T20:34:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.363+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb758150e90 con 0x7fb7580b3420 2023-11-28T20:34:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.364+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75809d0f0 con 0x7fb7580b3420 2023-11-28T20:34:33.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.365+0000 7fb74effd640 1 -- 172.21.15.45:0/1425880751 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb7400096a0 con 0x7fb7580b3420 2023-11-28T20:34:33.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.367+0000 7fb74effd640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb73806ae40 0x7fb73806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.367+0000 7fb74effd640 1 -- 172.21.15.45:0/1425880751 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb740089b80 con 0x7fb7580b3420 2023-11-28T20:34:33.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.367+0000 7fb75659c640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb73806ae40 0x7fb73806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.368+0000 7fb75659c640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb73806ae40 0x7fb73806d300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb7480012e0 tx=0x7fb748007ed0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:33.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.371+0000 7fb74effd640 1 -- 172.21.15.45:0/1425880751 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7400548a0 con 0x7fb7580b3420 2023-11-28T20:34:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.568+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb75815b880 con 0x7fb7580b3420 2023-11-28T20:34:33.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.569+0000 7fb74effd640 1 -- 172.21.15.45:0/1425880751 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb75815b880 con 0x7fb7580b3420 2023-11-28T20:34:33.571 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:33.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb73806ae40 msgr2=0x7fb73806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb73806ae40 0x7fb73806d300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb7480012e0 tx=0x7fb748007ed0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 msgr2=0x7fb758150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 0x7fb758150950 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d800 tx=0x7fb74000dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 shutdown_connections 2023-11-28T20:34:33.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb73806ae40 0x7fb73806d300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb75815b270 0x7fb7581598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7580b3420 0x7fb758150950 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 --2- 172.21.15.45:0/1425880751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7580b3040 0x7fb75815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 >> 172.21.15.45:0/1425880751 conn(0x7fb7580b1050 msgr2=0x7fb758157270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 shutdown_connections 2023-11-28T20:34:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.572+0000 7fb75759e640 1 -- 172.21.15.45:0/1425880751 wait complete. 2023-11-28T20:34:33.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:34:33.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.817+0000 7f326cd33640 1 Processor -- start 2023-11-28T20:34:33.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.818+0000 7f326cd33640 1 -- start start 2023-11-28T20:34:33.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.819+0000 7f326cd33640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f32680b0c30 0x7f32680b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.820+0000 7f326cd33640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32680ce710 con 0x7f326815d320 2023-11-28T20:34:33.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.820+0000 7f326cd33640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32680ae2c0 con 0x7f3268071190 2023-11-28T20:34:33.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.820+0000 7f326cd33640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32680ae5a0 con 0x7f32680b0c30 2023-11-28T20:34:33.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.820+0000 7f3265d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f32680b0c30 0x7f32680b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.820+0000 7f3265d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f32680b0c30 0x7f32680b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59208/0 (socket says 172.21.15.45:59208) 2023-11-28T20:34:33.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.820+0000 7f3265d9b640 1 -- 172.21.15.45:0/2940892024 learned_addr learned my addr 172.21.15.45:0/2940892024 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:33.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.820+0000 7f3265d9b640 1 -- 172.21.15.45:0/2940892024 >> v1:172.21.15.190:6789/0 conn(0x7f3268071190 legacy=0x7f3268071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.820+0000 7f3265d9b640 1 -- 172.21.15.45:0/2940892024 >> v1:172.21.15.45:6789/0 conn(0x7f326815d320 legacy=0x7f32680ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.821+0000 7f3265d9b640 1 -- 172.21.15.45:0/2940892024 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32681611f0 con 0x7f32680b0c30 2023-11-28T20:34:33.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.822+0000 7f3265d9b640 1 --2- 172.21.15.45:0/2940892024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32680b0c30 0x7f32680b10b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3250009870 tx=0x7f325002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95322950cba6c8ff server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:33.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.822+0000 7f326559a640 1 -- 172.21.15.45:0/2940892024 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325002f840 con 0x7f32680b0c30 2023-11-28T20:34:33.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.822+0000 7f326559a640 1 -- 172.21.15.45:0/2940892024 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325002f9a0 con 0x7f32680b0c30 2023-11-28T20:34:33.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.822+0000 7f326559a640 1 -- 172.21.15.45:0/2940892024 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325002fd30 con 0x7f32680b0c30 2023-11-28T20:34:33.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.823+0000 7f326cd33640 1 -- 172.21.15.45:0/2940892024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32680b0c30 msgr2=0x7f32680b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.823+0000 7f326cd33640 1 --2- 172.21.15.45:0/2940892024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32680b0c30 0x7f32680b10b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3250009870 tx=0x7f325002ba20 comp rx=0 tx=0).stop 2023-11-28T20:34:33.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.823+0000 7f326cd33640 1 -- 172.21.15.45:0/2940892024 shutdown_connections 2023-11-28T20:34:33.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.823+0000 7f326cd33640 1 --2- 172.21.15.45:0/2940892024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32680b0c30 0x7f32680b10b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.823+0000 7f326cd33640 1 -- 172.21.15.45:0/2940892024 >> 172.21.15.45:0/2940892024 conn(0x7f326806a500 msgr2=0x7f326806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:33.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.823+0000 7f326cd33640 1 -- 172.21.15.45:0/2940892024 shutdown_connections 2023-11-28T20:34:33.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.823+0000 7f326cd33640 1 -- 172.21.15.45:0/2940892024 wait complete. 2023-11-28T20:34:33.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.825+0000 7f326cd33640 1 Processor -- start 2023-11-28T20:34:33.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.825+0000 7f326cd33640 1 -- start start 2023-11-28T20:34:33.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.826+0000 7f326cd33640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3268071190 0x7f326813d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.826+0000 7f326cd33640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680b0c30 0x7f3268158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.826+0000 7f326cd33640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d320 0x7f326815d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.826+0000 7f326cd33640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32680b2c10 con 0x7f326815d320 2023-11-28T20:34:33.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.826+0000 7f326cd33640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32680b1f90 con 0x7f32680b0c30 2023-11-28T20:34:33.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.826+0000 7f326cd33640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32680b2270 con 0x7f3268071190 2023-11-28T20:34:33.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.826+0000 7f3266d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d320 0x7f326815d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.826+0000 7f3266d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d320 0x7f326815d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59214/0 (socket says 172.21.15.45:59214) 2023-11-28T20:34:33.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.826+0000 7f3266d9d640 1 -- 172.21.15.45:0/348867956 learned_addr learned my addr 172.21.15.45:0/348867956 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:33.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f3265d9b640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680b0c30 0x7f3268158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f326659c640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3268071190 0x7f326813d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f3266d9d640 1 -- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3268071190 msgr2=0x7f326813d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f3266d9d640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3268071190 0x7f326813d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f3266d9d640 1 -- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680b0c30 msgr2=0x7f3268158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f3266d9d640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680b0c30 0x7f3268158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f3266d9d640 1 -- 172.21.15.45:0/348867956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32500094d0 con 0x7f326815d320 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f3265d9b640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680b0c30 0x7f3268158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f326659c640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3268071190 0x7f326813d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f3266d9d640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d320 0x7f326815d970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3260001eb0 tx=0x7f326000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f325effd640 1 -- 172.21.15.45:0/348867956 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32600052d0 con 0x7f326815d320 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f325effd640 1 -- 172.21.15.45:0/348867956 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3260005430 con 0x7f326815d320 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f325effd640 1 -- 172.21.15.45:0/348867956 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260003810 con 0x7f326815d320 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.827+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32680ae090 con 0x7f326815d320 2023-11-28T20:34:33.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.828+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32680b2830 con 0x7f326815d320 2023-11-28T20:34:33.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.829+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32680733d0 con 0x7f326815d320 2023-11-28T20:34:33.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.830+0000 7f325effd640 1 -- 172.21.15.45:0/348867956 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3260003970 con 0x7f326815d320 2023-11-28T20:34:33.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.831+0000 7f325effd640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f324806ae70 0x7f324806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:33.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.831+0000 7f325effd640 1 -- 172.21.15.45:0/348867956 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3260088f20 con 0x7f326815d320 2023-11-28T20:34:33.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.831+0000 7f326659c640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f324806ae70 0x7f324806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:33.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.832+0000 7f326659c640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f324806ae70 0x7f324806d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f32580012e0 tx=0x7f3258009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:33.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:33.835+0000 7f325effd640 1 -- 172.21.15.45:0/348867956 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3260053c40 con 0x7f326815d320 2023-11-28T20:34:34.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.077+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32680cd6b0 con 0x7f326815d320 2023-11-28T20:34:34.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.080+0000 7f325effd640 1 -- 172.21.15.45:0/348867956 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4915 (secure 0 0 0) 0x7f3260017090 con 0x7f326815d320 2023-11-28T20:34:34.082 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:34.082 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:34:22.516874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T20:34:34.083 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2023-11-28T20:34:34.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f324806ae70 msgr2=0x7f324806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f324806ae70 0x7f324806d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f32580012e0 tx=0x7f3258009040 comp rx=0 tx=0).stop 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d320 msgr2=0x7f326815d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d320 0x7f326815d970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3260001eb0 tx=0x7f326000c890 comp rx=0 tx=0).stop 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 shutdown_connections 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f324806ae70 0x7f324806d330 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f326815d320 0x7f326815d970 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32680b0c30 0x7f3268158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 --2- 172.21.15.45:0/348867956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3268071190 0x7f326813d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 >> 172.21.15.45:0/348867956 conn(0x7f326806a500 msgr2=0x7f32680cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:34.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 shutdown_connections 2023-11-28T20:34:34.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.083+0000 7f326cd33640 1 -- 172.21.15.45:0/348867956 wait complete. 2023-11-28T20:34:34.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:34.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.313+0000 7fb575fcb640 1 Processor -- start 2023-11-28T20:34:34.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.314+0000 7fb575fcb640 1 -- start start 2023-11-28T20:34:34.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.315+0000 7fb575fcb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 0x7fb5700abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.315+0000 7fb575fcb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5700ae540 0x7fb5700ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb575fcb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57015c3c0 con 0x7fb5700ce630 2023-11-28T20:34:34.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb575fcb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5700ceac0 con 0x7fb5700ae540 2023-11-28T20:34:34.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb575fcb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb570155b60 con 0x7fb570155780 2023-11-28T20:34:34.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb56ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5700ae540 0x7fb5700ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb56ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5700ae540 0x7fb5700ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51922/0 (socket says 172.21.15.45:51922) 2023-11-28T20:34:34.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb56ffff640 1 -- 172.21.15.45:0/1464466237 learned_addr learned my addr 172.21.15.45:0/1464466237 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:34.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb574fc9640 1 --2- 172.21.15.45:0/1464466237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 0x7fb5700abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb56ffff640 1 -- 172.21.15.45:0/1464466237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 msgr2=0x7fb5700abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:34.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb56ffff640 1 --2- 172.21.15.45:0/1464466237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 0x7fb5700abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb56ffff640 1 -- 172.21.15.45:0/1464466237 >> v1:172.21.15.45:6789/0 conn(0x7fb5700ce630 legacy=0x7fb57015ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb56ffff640 1 -- 172.21.15.45:0/1464466237 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57015ba90 con 0x7fb5700ae540 2023-11-28T20:34:34.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.316+0000 7fb56f7fe640 1 -- 172.21.15.45:0/1464466237 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138603970 0 0) 0x7fb57015c3c0 con 0x7fb5700ce630 2023-11-28T20:34:34.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.318+0000 7fb56ffff640 1 --2- 172.21.15.45:0/1464466237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 0x7fb5700ce0f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0098a0 tx=0x7fb55c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b258e5290c91c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:34.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.318+0000 7fb56f7fe640 1 -- 172.21.15.45:0/1464466237 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c030840 con 0x7fb5700ae540 2023-11-28T20:34:34.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.318+0000 7fb56f7fe640 1 -- 172.21.15.45:0/1464466237 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c0309a0 con 0x7fb5700ae540 2023-11-28T20:34:34.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.318+0000 7fb56f7fe640 1 -- 172.21.15.45:0/1464466237 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c030d30 con 0x7fb5700ae540 2023-11-28T20:34:34.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.318+0000 7fb575fcb640 1 -- 172.21.15.45:0/1464466237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 msgr2=0x7fb5700ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.318+0000 7fb575fcb640 1 --2- 172.21.15.45:0/1464466237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 0x7fb5700ce0f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0098a0 tx=0x7fb55c02b980 comp rx=0 tx=0).stop 2023-11-28T20:34:34.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.319+0000 7fb575fcb640 1 -- 172.21.15.45:0/1464466237 shutdown_connections 2023-11-28T20:34:34.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.319+0000 7fb575fcb640 1 --2- 172.21.15.45:0/1464466237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 0x7fb5700ce0f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.319+0000 7fb575fcb640 1 --2- 172.21.15.45:0/1464466237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 0x7fb5700abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.319+0000 7fb575fcb640 1 -- 172.21.15.45:0/1464466237 >> 172.21.15.45:0/1464466237 conn(0x7fb57006ab30 msgr2=0x7fb5700b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:34.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.319+0000 7fb575fcb640 1 -- 172.21.15.45:0/1464466237 shutdown_connections 2023-11-28T20:34:34.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.319+0000 7fb575fcb640 1 -- 172.21.15.45:0/1464466237 wait complete. 2023-11-28T20:34:34.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.321+0000 7fb575fcb640 1 Processor -- start 2023-11-28T20:34:34.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.321+0000 7fb575fcb640 1 -- start start 2023-11-28T20:34:34.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb575fcb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 0x7fb570140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb575fcb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5700ce630 0x7fb570153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb575fcb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 0x7fb570153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb575fcb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb570072040 con 0x7fb5700ae540 2023-11-28T20:34:34.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb575fcb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57009ac30 con 0x7fb5700ce630 2023-11-28T20:34:34.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb575fcb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb570071d60 con 0x7fb570155780 2023-11-28T20:34:34.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb574fc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 0x7fb570140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb574fc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 0x7fb570140780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51938/0 (socket says 172.21.15.45:51938) 2023-11-28T20:34:34.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb574fc9640 1 -- 172.21.15.45:0/62843327 learned_addr learned my addr 172.21.15.45:0/62843327 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:34.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.322+0000 7fb5757ca640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 0x7fb570153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb56ffff640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5700ce630 0x7fb570153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb574fc9640 1 -- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 msgr2=0x7fb570153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb574fc9640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 0x7fb570153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb574fc9640 1 -- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5700ce630 msgr2=0x7fb570153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb574fc9640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5700ce630 0x7fb570153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb574fc9640 1 -- 172.21.15.45:0/62843327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c0094d0 con 0x7fb5700ae540 2023-11-28T20:34:34.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb5757ca640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 0x7fb570153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:34.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb574fc9640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 0x7fb570140780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb56400d800 tx=0x7fb56400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:34.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb56d7fa640 1 -- 172.21.15.45:0/62843327 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56400b6c0 con 0x7fb5700ae540 2023-11-28T20:34:34.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb56d7fa640 1 -- 172.21.15.45:0/62843327 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56400b820 con 0x7fb5700ae540 2023-11-28T20:34:34.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb56d7fa640 1 -- 172.21.15.45:0/62843327 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56400bbb0 con 0x7fb5700ae540 2023-11-28T20:34:34.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57009aed0 con 0x7fb5700ae540 2023-11-28T20:34:34.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.323+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb570140cc0 con 0x7fb5700ae540 2023-11-28T20:34:34.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.324+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57009aa90 con 0x7fb5700ae540 2023-11-28T20:34:34.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.325+0000 7fb56d7fa640 1 -- 172.21.15.45:0/62843327 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb56400bd10 con 0x7fb5700ae540 2023-11-28T20:34:34.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.326+0000 7fb56d7fa640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb55406ae40 0x7fb55406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.326+0000 7fb56d7fa640 1 -- 172.21.15.45:0/62843327 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb564089f20 con 0x7fb5700ae540 2023-11-28T20:34:34.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.327+0000 7fb56ffff640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb55406ae40 0x7fb55406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.328+0000 7fb56ffff640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb55406ae40 0x7fb55406d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb55c036000 tx=0x7fb55c03b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:34.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.331+0000 7fb56d7fa640 1 -- 172.21.15.45:0/62843327 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb57009aa90 con 0x7fb5700ae540 2023-11-28T20:34:34.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.525+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb57009aa90 con 0x7fb5700ae540 2023-11-28T20:34:34.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.528+0000 7fb56d7fa640 1 -- 172.21.15.45:0/62843327 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7fb57009aa90 con 0x7fb5700ae540 2023-11-28T20:34:34.531 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:34.531 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:34.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.530+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb55406ae40 msgr2=0x7fb55406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.530+0000 7fb575fcb640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb55406ae40 0x7fb55406d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb55c036000 tx=0x7fb55c03b040 comp rx=0 tx=0).stop 2023-11-28T20:34:34.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.530+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 msgr2=0x7fb570140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.530+0000 7fb575fcb640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 0x7fb570140780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb56400d800 tx=0x7fb56400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.531+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 shutdown_connections 2023-11-28T20:34:34.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.531+0000 7fb575fcb640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb55406ae40 0x7fb55406d300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.531+0000 7fb575fcb640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb570155780 0x7fb570153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.531+0000 7fb575fcb640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5700ce630 0x7fb570153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.531+0000 7fb575fcb640 1 --2- 172.21.15.45:0/62843327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5700ae540 0x7fb570140780 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.531+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 >> 172.21.15.45:0/62843327 conn(0x7fb57006ab30 msgr2=0x7fb57015ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:34.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.531+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 shutdown_connections 2023-11-28T20:34:34.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.531+0000 7fb575fcb640 1 -- 172.21.15.45:0/62843327 wait complete. 2023-11-28T20:34:34.554 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:34:34.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:34:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.781+0000 7f67f5954640 1 Processor -- start 2023-11-28T20:34:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.782+0000 7f67f5954640 1 -- start start 2023-11-28T20:34:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.783+0000 7f67f5954640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f67f00d0040 0x7f67f00d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.783+0000 7f67f5954640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f67f00b2290 0x7f67f00d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.783+0000 7f67f5954640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f00b90a0 con 0x7f67f00b1d40 2023-11-28T20:34:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.783+0000 7f67f5954640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f00d1920 con 0x7f67f00b2290 2023-11-28T20:34:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.783+0000 7f67f5954640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f00d2dc0 con 0x7f67f00d0040 2023-11-28T20:34:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.784+0000 7f67effff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f67f00b2290 0x7f67f00d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.784+0000 7f67effff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f67f00b2290 0x7f67f00d3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51952/0 (socket says 172.21.15.45:51952) 2023-11-28T20:34:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.784+0000 7f67effff640 1 -- 172.21.15.45:0/2676000398 learned_addr learned my addr 172.21.15.45:0/2676000398 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.784+0000 7f67f4952640 1 --2- 172.21.15.45:0/2676000398 >> v2:172.21.15.190:3300/0 conn(0x7f67f00d0040 0x7f67f00d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.784+0000 7f67effff640 1 -- 172.21.15.45:0/2676000398 >> v2:172.21.15.190:3300/0 conn(0x7f67f00d0040 msgr2=0x7f67f00d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:34.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.784+0000 7f67effff640 1 --2- 172.21.15.45:0/2676000398 >> v2:172.21.15.190:3300/0 conn(0x7f67f00d0040 0x7f67f00d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.784+0000 7f67effff640 1 -- 172.21.15.45:0/2676000398 >> v1:172.21.15.190:6789/0 conn(0x7f67f00b1d40 legacy=0x7f67f00d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:34.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.784+0000 7f67effff640 1 -- 172.21.15.45:0/2676000398 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f00d0960 con 0x7f67f00b2290 2023-11-28T20:34:34.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.785+0000 7f67effff640 1 --2- 172.21.15.45:0/2676000398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 0x7f67f00d3040 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f67f00b53c0 tx=0x7f67dc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f300132d47e35db server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67ef7fe640 1 -- 172.21.15.45:0/2676000398 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc030860 con 0x7f67f00b2290 2023-11-28T20:34:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67ef7fe640 1 -- 172.21.15.45:0/2676000398 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc0309c0 con 0x7f67f00b2290 2023-11-28T20:34:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67ef7fe640 1 -- 172.21.15.45:0/2676000398 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc030d50 con 0x7f67f00b2290 2023-11-28T20:34:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67f5954640 1 -- 172.21.15.45:0/2676000398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 msgr2=0x7f67f00d3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67f5954640 1 --2- 172.21.15.45:0/2676000398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 0x7f67f00d3040 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f67f00b53c0 tx=0x7f67dc02b9a0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67f5954640 1 -- 172.21.15.45:0/2676000398 shutdown_connections 2023-11-28T20:34:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67f5954640 1 --2- 172.21.15.45:0/2676000398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 0x7f67f00d3040 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67f5954640 1 --2- 172.21.15.45:0/2676000398 >> v2:172.21.15.190:3300/0 conn(0x7f67f00d0040 0x7f67f00d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67f5954640 1 -- 172.21.15.45:0/2676000398 >> 172.21.15.45:0/2676000398 conn(0x7f67f0070f40 msgr2=0x7f67f0071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67f5954640 1 -- 172.21.15.45:0/2676000398 shutdown_connections 2023-11-28T20:34:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.786+0000 7f67f5954640 1 -- 172.21.15.45:0/2676000398 wait complete. 2023-11-28T20:34:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.788+0000 7f67f5954640 1 Processor -- start 2023-11-28T20:34:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.788+0000 7f67f5954640 1 -- start start 2023-11-28T20:34:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.789+0000 7f67f5954640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67f00b1d40 0x7f67f01f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.789+0000 7f67f5954640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 0x7f67f01f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.789+0000 7f67f5954640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67f00d0040 0x7f67f00cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.789+0000 7f67f5954640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f0149990 con 0x7f67f00b2290 2023-11-28T20:34:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.789+0000 7f67f5954640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00c8860 con 0x7f67f00d0040 2023-11-28T20:34:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.789+0000 7f67f5954640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00c7c50 con 0x7f67f00b1d40 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 0x7f67f01f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 0x7f67f01f2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51966/0 (socket says 172.21.15.45:51966) 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67effff640 1 -- 172.21.15.45:0/743786641 learned_addr learned my addr 172.21.15.45:0/743786641 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67f4952640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67f00b1d40 0x7f67f01f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67f5153640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67f00d0040 0x7f67f00cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67effff640 1 -- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67f00b1d40 msgr2=0x7f67f01f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67effff640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67f00b1d40 0x7f67f01f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67effff640 1 -- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67f00d0040 msgr2=0x7f67f00cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67effff640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67f00d0040 0x7f67f00cd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67effff640 1 -- 172.21.15.45:0/743786641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc0094d0 con 0x7f67f00b2290 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67f5153640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67f00d0040 0x7f67f00cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.790+0000 7f67effff640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 0x7f67f01f2df0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f67dc000c00 tx=0x7f67dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:34.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.791+0000 7f67ed7fa640 1 -- 172.21.15.45:0/743786641 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc039c90 con 0x7f67f00b2290 2023-11-28T20:34:34.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.791+0000 7f67ed7fa640 1 -- 172.21.15.45:0/743786641 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc039df0 con 0x7f67f00b2290 2023-11-28T20:34:34.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.791+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f00c7f30 con 0x7f67f00b2290 2023-11-28T20:34:34.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.791+0000 7f67ed7fa640 1 -- 172.21.15.45:0/743786641 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc043620 con 0x7f67f00b2290 2023-11-28T20:34:34.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.791+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f00c84f0 con 0x7f67f00b2290 2023-11-28T20:34:34.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.792+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f00c89d0 con 0x7f67f00b2290 2023-11-28T20:34:34.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.793+0000 7f67ed7fa640 1 -- 172.21.15.45:0/743786641 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f67dc043780 con 0x7f67f00b2290 2023-11-28T20:34:34.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.794+0000 7f67ed7fa640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67d406ae70 0x7f67d406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:34.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.794+0000 7f67ed7fa640 1 -- 172.21.15.45:0/743786641 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f67dc03e070 con 0x7f67f00b2290 2023-11-28T20:34:34.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.795+0000 7f67f4952640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67d406ae70 0x7f67d406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:34.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.796+0000 7f67f4952640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67d406ae70 0x7f67d406d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f67e4006fd0 tx=0x7f67e4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:34.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.798+0000 7f67ed7fa640 1 -- 172.21.15.45:0/743786641 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67dc077840 con 0x7f67f00b2290 2023-11-28T20:34:34.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.990+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f67f0071b30 con 0x7f67f00b2290 2023-11-28T20:34:34.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.991+0000 7f67ed7fa640 1 -- 172.21.15.45:0/743786641 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f67dc07ae60 con 0x7f67f00b2290 2023-11-28T20:34:34.993 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:34:34.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.993+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67d406ae70 msgr2=0x7f67d406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.993+0000 7f67f5954640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67d406ae70 0x7f67d406d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f67e4006fd0 tx=0x7f67e4008040 comp rx=0 tx=0).stop 2023-11-28T20:34:34.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.993+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 msgr2=0x7f67f01f2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:34.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.993+0000 7f67f5954640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 0x7f67f01f2df0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f67dc000c00 tx=0x7f67dc004730 comp rx=0 tx=0).stop 2023-11-28T20:34:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.993+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 shutdown_connections 2023-11-28T20:34:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.993+0000 7f67f5954640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67d406ae70 0x7f67d406d330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.994+0000 7f67f5954640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67f00d0040 0x7f67f00cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.994+0000 7f67f5954640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f00b2290 0x7f67f01f2df0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.994+0000 7f67f5954640 1 --2- 172.21.15.45:0/743786641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67f00b1d40 0x7f67f01f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.994+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 >> 172.21.15.45:0/743786641 conn(0x7f67f0070f40 msgr2=0x7f67f00b6210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.994+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 shutdown_connections 2023-11-28T20:34:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:34.994+0000 7f67f5954640 1 -- 172.21.15.45:0/743786641 wait complete. 2023-11-28T20:34:35.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:34:35.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.251+0000 7f3adf59e640 1 Processor -- start 2023-11-28T20:34:35.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.251+0000 7f3adf59e640 1 -- start start 2023-11-28T20:34:35.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.253+0000 7f3adf59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3ae00b39d0 0x7f3ae00717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.253+0000 7f3adf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b3db0 0x7f3ae006bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.253+0000 7f3adf59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3ae00cb630 0x7f3ae00cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.253+0000 7f3adf59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae0071d20 con 0x7f3ae00b39d0 2023-11-28T20:34:35.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.253+0000 7f3adf59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae00cc000 con 0x7f3ae00cb630 2023-11-28T20:34:35.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.253+0000 7f3adf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae0161bc0 con 0x7f3ae00b3db0 2023-11-28T20:34:35.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.253+0000 7f3ade59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3ae00b39d0 0x7f3ae00717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.253+0000 7f3ade59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3ae00b39d0 0x7f3ae00717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51970/0 (socket says 172.21.15.45:51970) 2023-11-28T20:34:35.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.253+0000 7f3ade59c640 1 -- 172.21.15.45:0/1820616442 learned_addr learned my addr 172.21.15.45:0/1820616442 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:35.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.254+0000 7f3aded9d640 1 --2- 172.21.15.45:0/1820616442 >> v2:172.21.15.190:3300/0 conn(0x7f3ae00cb630 0x7f3ae00cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.254+0000 7f3ade59c640 1 -- 172.21.15.45:0/1820616442 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b3db0 msgr2=0x7f3ae006bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:35.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.254+0000 7f3ad7fff640 1 --2- 172.21.15.45:0/1820616442 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b3db0 0x7f3ae006bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.254+0000 7f3ade59c640 1 --2- 172.21.15.45:0/1820616442 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b3db0 0x7f3ae006bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.254+0000 7f3ade59c640 1 -- 172.21.15.45:0/1820616442 >> v2:172.21.15.190:3300/0 conn(0x7f3ae00cb630 msgr2=0x7f3ae00cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:35.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.254+0000 7f3ade59c640 1 --2- 172.21.15.45:0/1820616442 >> v2:172.21.15.190:3300/0 conn(0x7f3ae00cb630 0x7f3ae00cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.254+0000 7f3ade59c640 1 -- 172.21.15.45:0/1820616442 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00724d0 con 0x7f3ae00b39d0 2023-11-28T20:34:35.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.254+0000 7f3ad7fff640 1 --2- 172.21.15.45:0/1820616442 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b3db0 0x7f3ae006bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:35.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.254+0000 7f3aded9d640 1 --2- 172.21.15.45:0/1820616442 >> v2:172.21.15.190:3300/0 conn(0x7f3ae00cb630 0x7f3ae00cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:35.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.255+0000 7f3ade59c640 1 --2- 172.21.15.45:0/1820616442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00b39d0 0x7f3ae00717e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ae00cca30 tx=0x7f3ad002b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643774cbd8cd945d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:35.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.255+0000 7f3addd9b640 1 -- 172.21.15.45:0/1820616442 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad00317f0 con 0x7f3ae00b39d0 2023-11-28T20:34:35.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.255+0000 7f3addd9b640 1 -- 172.21.15.45:0/1820616442 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad0031950 con 0x7f3ae00b39d0 2023-11-28T20:34:35.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.255+0000 7f3addd9b640 1 -- 172.21.15.45:0/1820616442 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0031ce0 con 0x7f3ae00b39d0 2023-11-28T20:34:35.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.256+0000 7f3adf59e640 1 -- 172.21.15.45:0/1820616442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00b39d0 msgr2=0x7f3ae00717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.256+0000 7f3adf59e640 1 --2- 172.21.15.45:0/1820616442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00b39d0 0x7f3ae00717e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ae00cca30 tx=0x7f3ad002b890 comp rx=0 tx=0).stop 2023-11-28T20:34:35.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.256+0000 7f3adf59e640 1 -- 172.21.15.45:0/1820616442 shutdown_connections 2023-11-28T20:34:35.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.256+0000 7f3adf59e640 1 --2- 172.21.15.45:0/1820616442 >> v2:172.21.15.190:3300/0 conn(0x7f3ae00cb630 0x7f3ae00cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.256+0000 7f3adf59e640 1 --2- 172.21.15.45:0/1820616442 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b3db0 0x7f3ae006bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.256+0000 7f3adf59e640 1 --2- 172.21.15.45:0/1820616442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00b39d0 0x7f3ae00717e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.256+0000 7f3adf59e640 1 -- 172.21.15.45:0/1820616442 >> 172.21.15.45:0/1820616442 conn(0x7f3ae00b1050 msgr2=0x7f3ae00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:35.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.256+0000 7f3adf59e640 1 -- 172.21.15.45:0/1820616442 shutdown_connections 2023-11-28T20:34:35.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.256+0000 7f3adf59e640 1 -- 172.21.15.45:0/1820616442 wait complete. 2023-11-28T20:34:35.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.258+0000 7f3adf59e640 1 Processor -- start 2023-11-28T20:34:35.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.258+0000 7f3adf59e640 1 -- start start 2023-11-28T20:34:35.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.259+0000 7f3adf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b39d0 0x7f3ae01f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.259+0000 7f3adf59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ae00b3db0 0x7f3ae009b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.259+0000 7f3adf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00cb630 0x7f3ae01fa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.259+0000 7f3adf59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae01f8230 con 0x7f3ae00cb630 2023-11-28T20:34:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.259+0000 7f3adf59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae009c0a0 con 0x7f3ae00b3db0 2023-11-28T20:34:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.259+0000 7f3adf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae009c380 con 0x7f3ae00b39d0 2023-11-28T20:34:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3aded9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00cb630 0x7f3ae01fa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3aded9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00cb630 0x7f3ae01fa7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51980/0 (socket says 172.21.15.45:51980) 2023-11-28T20:34:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3aded9d640 1 -- 172.21.15.45:0/2065337612 learned_addr learned my addr 172.21.15.45:0/2065337612 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3ade59c640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b39d0 0x7f3ae01f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3ad7fff640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ae00b3db0 0x7f3ae009b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3aded9d640 1 -- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b39d0 msgr2=0x7f3ae01f7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3aded9d640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b39d0 0x7f3ae01f7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3aded9d640 1 -- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ae00b3db0 msgr2=0x7f3ae009b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3aded9d640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ae00b3db0 0x7f3ae009b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3aded9d640 1 -- 172.21.15.45:0/2065337612 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad00094d0 con 0x7f3ae00cb630 2023-11-28T20:34:35.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3ade59c640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b39d0 0x7f3ae01f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:35.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3ad7fff640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ae00b3db0 0x7f3ae009b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:35.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.260+0000 7f3aded9d640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00cb630 0x7f3ae01fa7d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800cd50 tx=0x7f3ad800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:35.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.261+0000 7f3ad6ffd640 1 -- 172.21.15.45:0/2065337612 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad800f680 con 0x7f3ae00cb630 2023-11-28T20:34:35.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.261+0000 7f3ad6ffd640 1 -- 172.21.15.45:0/2065337612 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad800f800 con 0x7f3ae00cb630 2023-11-28T20:34:35.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.261+0000 7f3ad6ffd640 1 -- 172.21.15.45:0/2065337612 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad800fbb0 con 0x7f3ae00cb630 2023-11-28T20:34:35.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.261+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae009c660 con 0x7f3ae00cb630 2023-11-28T20:34:35.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.261+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae00abd70 con 0x7f3ae00cb630 2023-11-28T20:34:35.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.262+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae009d0f0 con 0x7f3ae00cb630 2023-11-28T20:34:35.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.263+0000 7f3ad6ffd640 1 -- 172.21.15.45:0/2065337612 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3ad800b040 con 0x7f3ae00cb630 2023-11-28T20:34:35.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.264+0000 7f3ad6ffd640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3ac006ae40 0x7f3ac006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.264+0000 7f3ad6ffd640 1 -- 172.21.15.45:0/2065337612 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3ad808a340 con 0x7f3ae00cb630 2023-11-28T20:34:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.265+0000 7f3ade59c640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3ac006ae40 0x7f3ac006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.266+0000 7f3ade59c640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3ac006ae40 0x7f3ac006d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0036000 tx=0x7f3ad003d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.268+0000 7f3ad6ffd640 1 -- 172.21.15.45:0/2065337612 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ad8055060 con 0x7f3ae00cb630 2023-11-28T20:34:35.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.462+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3ae0165030 con 0x7f3ae00cb630 2023-11-28T20:34:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.463+0000 7f3ad6ffd640 1 -- 172.21.15.45:0/2065337612 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ad8058680 con 0x7f3ae00cb630 2023-11-28T20:34:35.466 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3ac006ae40 msgr2=0x7f3ac006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3ac006ae40 0x7f3ac006d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0036000 tx=0x7f3ad003d040 comp rx=0 tx=0).stop 2023-11-28T20:34:35.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00cb630 msgr2=0x7f3ae01fa7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00cb630 0x7f3ae01fa7d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800cd50 tx=0x7f3ad800a650 comp rx=0 tx=0).stop 2023-11-28T20:34:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 shutdown_connections 2023-11-28T20:34:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3ac006ae40 0x7f3ac006d300 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae00cb630 0x7f3ae01fa7d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ae00b3db0 0x7f3ae009b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 --2- 172.21.15.45:0/2065337612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ae00b39d0 0x7f3ae01f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 >> 172.21.15.45:0/2065337612 conn(0x7f3ae00b1050 msgr2=0x7f3ae00afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 shutdown_connections 2023-11-28T20:34:35.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.466+0000 7f3adf59e640 1 -- 172.21.15.45:0/2065337612 wait complete. 2023-11-28T20:34:35.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:34:35.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.714+0000 7f5ad0a22640 1 Processor -- start 2023-11-28T20:34:35.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.714+0000 7f5ad0a22640 1 -- start start 2023-11-28T20:34:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.716+0000 7f5ad0a22640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc0cdca0 0x7f5acc06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.716+0000 7f5ad0a22640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc06d410 con 0x7f5acc15b270 2023-11-28T20:34:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.716+0000 7f5ad0a22640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc06d6f0 con 0x7f5acc0b3040 2023-11-28T20:34:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.716+0000 7f5ad0a22640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0ceaa0 con 0x7f5acc0cdca0 2023-11-28T20:34:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.716+0000 7f5aca59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5acc15b270 0x7f5acc0b3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33454/0 (socket says 172.21.15.45:33454) 2023-11-28T20:34:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.716+0000 7f5aca59c640 1 -- 172.21.15.45:0/3547720513 learned_addr learned my addr 172.21.15.45:0/3547720513 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.717+0000 7f5acb59e640 1 --2- 172.21.15.45:0/3547720513 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc0cdca0 0x7f5acc06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.717+0000 7f5ac9d9b640 1 -- 172.21.15.45:0/3547720513 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170941451 0 0) 0x7f5acc06d410 con 0x7f5acc15b270 2023-11-28T20:34:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.717+0000 7f5ac9d9b640 1 -- 172.21.15.45:0/3547720513 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab4003520 con 0x7f5acc15b270 2023-11-28T20:34:35.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.717+0000 7f5acb59e640 1 -- 172.21.15.45:0/3547720513 >> v1:172.21.15.190:6789/0 conn(0x7f5acc0b3040 legacy=0x7f5acc15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.717+0000 7f5acb59e640 1 -- 172.21.15.45:0/3547720513 >> v1:172.21.15.45:6789/0 conn(0x7f5acc15b270 legacy=0x7f5acc0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.717+0000 7f5acb59e640 1 -- 172.21.15.45:0/3547720513 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc06d9d0 con 0x7f5acc0cdca0 2023-11-28T20:34:35.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.719+0000 7f5acb59e640 1 --2- 172.21.15.45:0/3547720513 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc0cdca0 0x7f5acc06cda0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009dc0 tx=0x7f5ac402bdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77863d283e1b2a0e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:35.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.719+0000 7f5ac9d9b640 1 -- 172.21.15.45:0/3547720513 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac402cba0 con 0x7f5acc0cdca0 2023-11-28T20:34:35.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.719+0000 7f5ac9d9b640 1 -- 172.21.15.45:0/3547720513 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac402cd00 con 0x7f5acc0cdca0 2023-11-28T20:34:35.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.719+0000 7f5ac9d9b640 1 -- 172.21.15.45:0/3547720513 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4034660 con 0x7f5acc0cdca0 2023-11-28T20:34:35.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.720+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3547720513 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc0cdca0 msgr2=0x7f5acc06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.720+0000 7f5ad0a22640 1 --2- 172.21.15.45:0/3547720513 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc0cdca0 0x7f5acc06cda0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009dc0 tx=0x7f5ac402bdf0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.720+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3547720513 shutdown_connections 2023-11-28T20:34:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.720+0000 7f5ad0a22640 1 --2- 172.21.15.45:0/3547720513 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc0cdca0 0x7f5acc06cda0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.720+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3547720513 >> 172.21.15.45:0/3547720513 conn(0x7f5acc0b1050 msgr2=0x7f5acc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.720+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3547720513 shutdown_connections 2023-11-28T20:34:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.720+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3547720513 wait complete. 2023-11-28T20:34:35.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.723+0000 7f5ad0a22640 1 Processor -- start 2023-11-28T20:34:35.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.723+0000 7f5ad0a22640 1 -- start start 2023-11-28T20:34:35.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.724+0000 7f5ad0a22640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5acc0b3040 0x7f5acc14a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.724+0000 7f5ad0a22640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc0cdca0 0x7f5acc158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.724+0000 7f5ad0a22640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc15b270 0x7f5acc0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.724+0000 7f5ad0a22640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc15a930 con 0x7f5acc0b3040 2023-11-28T20:34:35.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.724+0000 7f5ad0a22640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc1647a0 con 0x7f5acc0cdca0 2023-11-28T20:34:35.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.724+0000 7f5ad0a22640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc1644c0 con 0x7f5acc15b270 2023-11-28T20:34:35.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.724+0000 7f5acad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5acc0b3040 0x7f5acc14a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5acad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5acc0b3040 0x7f5acc14a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51988/0 (socket says 172.21.15.45:51988) 2023-11-28T20:34:35.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5acad9d640 1 -- 172.21.15.45:0/3929863339 learned_addr learned my addr 172.21.15.45:0/3929863339 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:35.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5acb59e640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc15b270 0x7f5acc0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5aca59c640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc0cdca0 0x7f5acc158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5acad9d640 1 -- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc15b270 msgr2=0x7f5acc0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5acad9d640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc15b270 0x7f5acc0abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5acad9d640 1 -- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc0cdca0 msgr2=0x7f5acc158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5acad9d640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc0cdca0 0x7f5acc158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5acad9d640 1 -- 172.21.15.45:0/3929863339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac4009a70 con 0x7f5acc0b3040 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5aca59c640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc0cdca0 0x7f5acc158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.725+0000 7f5acad9d640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5acc0b3040 0x7f5acc14a050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00c400 tx=0x7f5abc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.726+0000 7f5ab37fe640 1 -- 172.21.15.45:0/3929863339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc0052d0 con 0x7f5acc0b3040 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.726+0000 7f5ab37fe640 1 -- 172.21.15.45:0/3929863339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc005450 con 0x7f5acc0b3040 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.726+0000 7f5ab37fe640 1 -- 172.21.15.45:0/3929863339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc007260 con 0x7f5acc0b3040 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.726+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5acc15a370 con 0x7f5acc0b3040 2023-11-28T20:34:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.726+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5acc14a590 con 0x7f5acc0b3040 2023-11-28T20:34:35.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.727+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5acc0b3db0 con 0x7f5acc0b3040 2023-11-28T20:34:35.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.728+0000 7f5ab37fe640 1 -- 172.21.15.45:0/3929863339 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5abc0073c0 con 0x7f5acc0b3040 2023-11-28T20:34:35.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.729+0000 7f5ab37fe640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5ab806ae40 0x7f5ab806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:35.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.729+0000 7f5ab37fe640 1 -- 172.21.15.45:0/3929863339 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5abc088da0 con 0x7f5acc0b3040 2023-11-28T20:34:35.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.730+0000 7f5aca59c640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5ab806ae40 0x7f5ab806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:35.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.731+0000 7f5aca59c640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5ab806ae40 0x7f5ab806d300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5acc1f6810 tx=0x7f5ac0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:35.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.733+0000 7f5ab37fe640 1 -- 172.21.15.45:0/3929863339 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5abc053ac0 con 0x7f5acc0b3040 2023-11-28T20:34:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.927+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5acc09d0f0 con 0x7f5acc0b3040 2023-11-28T20:34:35.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.928+0000 7f5ab37fe640 1 -- 172.21.15.45:0/3929863339 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5abc0570e0 con 0x7f5acc0b3040 2023-11-28T20:34:35.930 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:34:35.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5ab806ae40 msgr2=0x7f5ab806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5ab806ae40 0x7f5ab806d300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5acc1f6810 tx=0x7f5ac0009040 comp rx=0 tx=0).stop 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5acc0b3040 msgr2=0x7f5acc14a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5acc0b3040 0x7f5acc14a050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00c400 tx=0x7f5abc00c8c0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 shutdown_connections 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5ab806ae40 0x7f5ab806d300 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5acc15b270 0x7f5acc0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5acc0cdca0 0x7f5acc158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 --2- 172.21.15.45:0/3929863339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5acc0b3040 0x7f5acc14a050 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 >> 172.21.15.45:0/3929863339 conn(0x7f5acc0b1050 msgr2=0x7f5acc0b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 shutdown_connections 2023-11-28T20:34:35.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:35.931+0000 7f5ad0a22640 1 -- 172.21.15.45:0/3929863339 wait complete. 2023-11-28T20:34:35.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:34:36.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.182+0000 7fece8b47640 1 Processor -- start 2023-11-28T20:34:36.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.182+0000 7fece8b47640 1 -- start start 2023-11-28T20:34:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.184+0000 7fece8b47640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece415d4b0 0x7fece4070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.184+0000 7fece8b47640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fece40adfa0 0x7fece40717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.184+0000 7fece8b47640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece40708d0 con 0x7fece40712a0 2023-11-28T20:34:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.184+0000 7fece8b47640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece40b3bb0 con 0x7fece40adfa0 2023-11-28T20:34:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.184+0000 7fece8b47640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece40b3e90 con 0x7fece415d4b0 2023-11-28T20:34:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.184+0000 7fece3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fece40adfa0 0x7fece40717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.184+0000 7fecdbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fece40712a0 0x7fece40abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33462/0 (socket says 172.21.15.45:33462) 2023-11-28T20:34:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.184+0000 7fecdbfff640 1 -- 172.21.15.45:0/911683054 learned_addr learned my addr 172.21.15.45:0/911683054 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.185+0000 7fece37fe640 1 --2- 172.21.15.45:0/911683054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece415d4b0 0x7fece4070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.185+0000 7fece3fff640 1 -- 172.21.15.45:0/911683054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece415d4b0 msgr2=0x7fece4070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.185+0000 7fece3fff640 1 --2- 172.21.15.45:0/911683054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece415d4b0 0x7fece4070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.185+0000 7fece3fff640 1 -- 172.21.15.45:0/911683054 >> v1:172.21.15.45:6789/0 conn(0x7fece40712a0 legacy=0x7fece40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.185+0000 7fece2ffd640 1 -- 172.21.15.45:0/911683054 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944655731 0 0) 0x7fece40708d0 con 0x7fece40712a0 2023-11-28T20:34:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.185+0000 7fece3fff640 1 -- 172.21.15.45:0/911683054 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece4070280 con 0x7fece40adfa0 2023-11-28T20:34:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.187+0000 7fece3fff640 1 --2- 172.21.15.45:0/911683054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40adfa0 0x7fece40717e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009d90 tx=0x7fecdc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32f5009e1a5ba6a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.187+0000 7fece2ffd640 1 -- 172.21.15.45:0/911683054 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc02cbc0 con 0x7fece40adfa0 2023-11-28T20:34:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.187+0000 7fece2ffd640 1 -- 172.21.15.45:0/911683054 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc02cd20 con 0x7fece40adfa0 2023-11-28T20:34:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.187+0000 7fece2ffd640 1 -- 172.21.15.45:0/911683054 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc034660 con 0x7fece40adfa0 2023-11-28T20:34:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.187+0000 7fece8b47640 1 -- 172.21.15.45:0/911683054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40adfa0 msgr2=0x7fece40717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.188+0000 7fece8b47640 1 --2- 172.21.15.45:0/911683054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40adfa0 0x7fece40717e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009d90 tx=0x7fecdc02be10 comp rx=0 tx=0).stop 2023-11-28T20:34:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.188+0000 7fece8b47640 1 -- 172.21.15.45:0/911683054 shutdown_connections 2023-11-28T20:34:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.188+0000 7fece8b47640 1 --2- 172.21.15.45:0/911683054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40adfa0 0x7fece40717e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.188+0000 7fece8b47640 1 --2- 172.21.15.45:0/911683054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece415d4b0 0x7fece4070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.188+0000 7fece8b47640 1 -- 172.21.15.45:0/911683054 >> 172.21.15.45:0/911683054 conn(0x7fece40b1050 msgr2=0x7fece40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.188+0000 7fece8b47640 1 -- 172.21.15.45:0/911683054 shutdown_connections 2023-11-28T20:34:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.188+0000 7fece8b47640 1 -- 172.21.15.45:0/911683054 wait complete. 2023-11-28T20:34:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.190+0000 7fece8b47640 1 Processor -- start 2023-11-28T20:34:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.190+0000 7fece8b47640 1 -- start start 2023-11-28T20:34:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece8b47640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40712a0 0x7fece414c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece8b47640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece40adfa0 0x7fece406cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece8b47640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fece415d4b0 0x7fece406d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece8b47640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece414ce50 con 0x7fece40712a0 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece8b47640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece4099a40 con 0x7fece415d4b0 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece8b47640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece409a8a0 con 0x7fece40adfa0 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40712a0 0x7fece414c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40712a0 0x7fece414c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52016/0 (socket says 172.21.15.45:52016) 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece37fe640 1 -- 172.21.15.45:0/2676218243 learned_addr learned my addr 172.21.15.45:0/2676218243 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.191+0000 7fece3fff640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fece415d4b0 0x7fece406d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece37fe640 1 -- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece40adfa0 msgr2=0x7fece406cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fecdbfff640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece40adfa0 0x7fece406cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece37fe640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece40adfa0 0x7fece406cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece37fe640 1 -- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fece415d4b0 msgr2=0x7fece406d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece37fe640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fece415d4b0 0x7fece406d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece37fe640 1 -- 172.21.15.45:0/2676218243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc009a70 con 0x7fece40712a0 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece3fff640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fece415d4b0 0x7fece406d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece37fe640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40712a0 0x7fece414c910 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fecd400e8a0 tx=0x7fecd400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece0ff9640 1 -- 172.21.15.45:0/2676218243 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd40094f0 con 0x7fece40712a0 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece0ff9640 1 -- 172.21.15.45:0/2676218243 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd400c940 con 0x7fece40712a0 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece409ad30 con 0x7fece40712a0 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece0ff9640 1 -- 172.21.15.45:0/2676218243 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd400ccf0 con 0x7fece40712a0 2023-11-28T20:34:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.192+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece4099f70 con 0x7fece40712a0 2023-11-28T20:34:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.193+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece409d0f0 con 0x7fece40712a0 2023-11-28T20:34:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.194+0000 7fece0ff9640 1 -- 172.21.15.45:0/2676218243 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fecd4013470 con 0x7fece40712a0 2023-11-28T20:34:36.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.196+0000 7fece0ff9640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fecc406ae40 0x7fecc406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.196+0000 7fece0ff9640 1 -- 172.21.15.45:0/2676218243 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fecd4089aa0 con 0x7fece40712a0 2023-11-28T20:34:36.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.197+0000 7fecdbfff640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fecc406ae40 0x7fecc406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.198+0000 7fecdbfff640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fecc406ae40 0x7fecc406d300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fece40b0980 tx=0x7feccc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.201+0000 7fece0ff9640 1 -- 172.21.15.45:0/2676218243 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fecd40547c0 con 0x7fece40712a0 2023-11-28T20:34:36.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.394+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fece404e920 con 0x7fece40712a0 2023-11-28T20:34:36.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.395+0000 7fece0ff9640 1 -- 172.21.15.45:0/2676218243 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fecd40210a0 con 0x7fece40712a0 2023-11-28T20:34:36.397 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fecc406ae40 msgr2=0x7fecc406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fecc406ae40 0x7fecc406d300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fece40b0980 tx=0x7feccc009040 comp rx=0 tx=0).stop 2023-11-28T20:34:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40712a0 msgr2=0x7fece414c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40712a0 0x7fece414c910 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fecd400e8a0 tx=0x7fecd400ed60 comp rx=0 tx=0).stop 2023-11-28T20:34:36.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 shutdown_connections 2023-11-28T20:34:36.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fecc406ae40 0x7fecc406d300 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fece415d4b0 0x7fece406d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fece40adfa0 0x7fece406cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 --2- 172.21.15.45:0/2676218243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece40712a0 0x7fece414c910 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 >> 172.21.15.45:0/2676218243 conn(0x7fece40b1050 msgr2=0x7fece415c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 shutdown_connections 2023-11-28T20:34:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.398+0000 7fece8b47640 1 -- 172.21.15.45:0/2676218243 wait complete. 2023-11-28T20:34:36.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:34:36.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.644+0000 7fc675389640 1 Processor -- start 2023-11-28T20:34:36.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.644+0000 7fc675389640 1 -- start start 2023-11-28T20:34:36.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.645+0000 7fc675389640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc670160e70 0x7fc6700b6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.645+0000 7fc675389640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc67006a500 0x7fc670070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc675389640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67006aa70 con 0x7fc6700b2430 2023-11-28T20:34:36.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc675389640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc67006ad50 con 0x7fc67006a500 2023-11-28T20:34:36.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc675389640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc67006b030 con 0x7fc670160e70 2023-11-28T20:34:36.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc66e7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc67006a500 0x7fc670070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc66f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc6700b2430 0x7fc6700d1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33472/0 (socket says 172.21.15.45:33472) 2023-11-28T20:34:36.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc66e7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc67006a500 0x7fc670070b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52028/0 (socket says 172.21.15.45:52028) 2023-11-28T20:34:36.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc66f7fe640 1 -- 172.21.15.45:0/468526941 learned_addr learned my addr 172.21.15.45:0/468526941 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:36.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc66effd640 1 --2- 172.21.15.45:0/468526941 >> v2:172.21.15.190:3300/0 conn(0x7fc670160e70 0x7fc6700b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc66e7fc640 1 -- 172.21.15.45:0/468526941 >> v2:172.21.15.190:3300/0 conn(0x7fc670160e70 msgr2=0x7fc6700b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:36.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc66e7fc640 1 --2- 172.21.15.45:0/468526941 >> v2:172.21.15.190:3300/0 conn(0x7fc670160e70 0x7fc6700b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc66e7fc640 1 -- 172.21.15.45:0/468526941 >> v1:172.21.15.45:6789/0 conn(0x7fc6700b2430 legacy=0x7fc6700d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.646+0000 7fc66e7fc640 1 -- 172.21.15.45:0/468526941 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67006b310 con 0x7fc67006a500 2023-11-28T20:34:36.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.648+0000 7fc66e7fc640 1 --2- 172.21.15.45:0/468526941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc67006a500 0x7fc670070b20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6700d0d30 tx=0x7fc66402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8937f11c4a8f5e63 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:36.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.648+0000 7fc66dffb640 1 -- 172.21.15.45:0/468526941 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664031830 con 0x7fc67006a500 2023-11-28T20:34:36.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.648+0000 7fc66dffb640 1 -- 172.21.15.45:0/468526941 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc664031990 con 0x7fc67006a500 2023-11-28T20:34:36.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.648+0000 7fc66dffb640 1 -- 172.21.15.45:0/468526941 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664031d20 con 0x7fc67006a500 2023-11-28T20:34:36.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.648+0000 7fc675389640 1 -- 172.21.15.45:0/468526941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc67006a500 msgr2=0x7fc670070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.648+0000 7fc675389640 1 --2- 172.21.15.45:0/468526941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc67006a500 0x7fc670070b20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6700d0d30 tx=0x7fc66402b8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.648+0000 7fc675389640 1 -- 172.21.15.45:0/468526941 shutdown_connections 2023-11-28T20:34:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.649+0000 7fc675389640 1 --2- 172.21.15.45:0/468526941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc67006a500 0x7fc670070b20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.649+0000 7fc675389640 1 --2- 172.21.15.45:0/468526941 >> v2:172.21.15.190:3300/0 conn(0x7fc670160e70 0x7fc6700b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.649+0000 7fc675389640 1 -- 172.21.15.45:0/468526941 >> 172.21.15.45:0/468526941 conn(0x7fc6700cfc60 msgr2=0x7fc6700d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.649+0000 7fc675389640 1 -- 172.21.15.45:0/468526941 shutdown_connections 2023-11-28T20:34:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.649+0000 7fc675389640 1 -- 172.21.15.45:0/468526941 wait complete. 2023-11-28T20:34:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.650+0000 7fc675389640 1 Processor -- start 2023-11-28T20:34:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.651+0000 7fc675389640 1 -- start start 2023-11-28T20:34:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc675389640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6700b2430 0x7fc6700b7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc675389640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc670160e70 0x7fc6700b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc675389640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6701f6cb0 0x7fc6701f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc675389640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6701f4460 con 0x7fc6701f6cb0 2023-11-28T20:34:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc675389640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6701f79e0 con 0x7fc670160e70 2023-11-28T20:34:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc675389640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6701f3fd0 con 0x7fc6700b2430 2023-11-28T20:34:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc66f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6701f6cb0 0x7fc6701f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc66f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6701f6cb0 0x7fc6701f7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52040/0 (socket says 172.21.15.45:52040) 2023-11-28T20:34:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc66f7fe640 1 -- 172.21.15.45:0/3506591292 learned_addr learned my addr 172.21.15.45:0/3506591292 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc66e7fc640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc670160e70 0x7fc6700b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc66effd640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6700b2430 0x7fc6700b7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc66f7fe640 1 -- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6700b2430 msgr2=0x7fc6700b7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.652+0000 7fc66f7fe640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6700b2430 0x7fc6700b7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.653+0000 7fc66f7fe640 1 -- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc670160e70 msgr2=0x7fc6700b7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.653+0000 7fc66f7fe640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc670160e70 0x7fc6700b7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.653+0000 7fc66f7fe640 1 -- 172.21.15.45:0/3506591292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc664009400 con 0x7fc6701f6cb0 2023-11-28T20:34:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.653+0000 7fc66f7fe640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6701f6cb0 0x7fc6701f7140 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc66800c450 tx=0x7fc66800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.653+0000 7fc6577fe640 1 -- 172.21.15.45:0/3506591292 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668005ce0 con 0x7fc6701f6cb0 2023-11-28T20:34:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.653+0000 7fc6577fe640 1 -- 172.21.15.45:0/3506591292 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc668005e40 con 0x7fc6701f6cb0 2023-11-28T20:34:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.653+0000 7fc6577fe640 1 -- 172.21.15.45:0/3506591292 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668007270 con 0x7fc6701f6cb0 2023-11-28T20:34:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.653+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc67014d3a0 con 0x7fc6701f6cb0 2023-11-28T20:34:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.653+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6701f3980 con 0x7fc6701f6cb0 2023-11-28T20:34:36.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.655+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6700733d0 con 0x7fc6701f6cb0 2023-11-28T20:34:36.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.656+0000 7fc6577fe640 1 -- 172.21.15.45:0/3506591292 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc6680073d0 con 0x7fc6701f6cb0 2023-11-28T20:34:36.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.656+0000 7fc6577fe640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc65c06ae70 0x7fc65c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.657+0000 7fc6577fe640 1 -- 172.21.15.45:0/3506591292 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc668088dd0 con 0x7fc6701f6cb0 2023-11-28T20:34:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.657+0000 7fc66effd640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc65c06ae70 0x7fc65c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.658+0000 7fc66effd640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc65c06ae70 0x7fc65c06d330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc6600099c0 tx=0x7fc660003990 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.661+0000 7fc6577fe640 1 -- 172.21.15.45:0/3506591292 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc668053ba0 con 0x7fc6701f6cb0 2023-11-28T20:34:36.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.891+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6701f4110 con 0x7fc6701f6cb0 2023-11-28T20:34:36.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.892+0000 7fc6577fe640 1 -- 172.21.15.45:0/3506591292 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4915 (secure 0 0 0) 0x7fc668017020 con 0x7fc6701f6cb0 2023-11-28T20:34:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:36.894 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:34:22.516874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T20:34:36.894 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2023-11-28T20:34:36.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.894+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc65c06ae70 msgr2=0x7fc65c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.894+0000 7fc675389640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc65c06ae70 0x7fc65c06d330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc6600099c0 tx=0x7fc660003990 comp rx=0 tx=0).stop 2023-11-28T20:34:36.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.894+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6701f6cb0 msgr2=0x7fc6701f7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:36.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.894+0000 7fc675389640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6701f6cb0 0x7fc6701f7140 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc66800c450 tx=0x7fc66800c910 comp rx=0 tx=0).stop 2023-11-28T20:34:36.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.895+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 shutdown_connections 2023-11-28T20:34:36.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.895+0000 7fc675389640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc65c06ae70 0x7fc65c06d330 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.895+0000 7fc675389640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6701f6cb0 0x7fc6701f7140 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.895+0000 7fc675389640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc670160e70 0x7fc6700b7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.895+0000 7fc675389640 1 --2- 172.21.15.45:0/3506591292 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6700b2430 0x7fc6700b7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:36.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.895+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 >> 172.21.15.45:0/3506591292 conn(0x7fc6700cfc60 msgr2=0x7fc6700b6890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:36.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.895+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 shutdown_connections 2023-11-28T20:34:36.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:36.895+0000 7fc675389640 1 -- 172.21.15.45:0/3506591292 wait complete. 2023-11-28T20:34:36.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:37.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.142+0000 7f7a41e87640 1 Processor -- start 2023-11-28T20:34:37.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.143+0000 7f7a41e87640 1 -- start start 2023-11-28T20:34:37.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.144+0000 7f7a41e87640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7a3c0b35b0 0x7f7a3c15ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.144+0000 7f7a41e87640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7a3c15b850 0x7f7a3c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.144+0000 7f7a41e87640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.144+0000 7f7a41e87640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3c15ee40 con 0x7f7a3c15b850 2023-11-28T20:34:37.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.144+0000 7f7a41e87640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3c15bda0 con 0x7f7a3c0b35b0 2023-11-28T20:34:37.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.144+0000 7f7a41e87640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c15c080 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a33fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7a3c15b850 0x7f7a3c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a33fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7a3c15b850 0x7f7a3c0b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52056/0 (socket says 172.21.15.45:52056) 2023-11-28T20:34:37.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a33fff640 1 -- 172.21.15.45:0/3861809695 learned_addr learned my addr 172.21.15.45:0/3861809695 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:37.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a3b7fe640 1 --2- 172.21.15.45:0/3861809695 >> v2:172.21.15.190:3300/0 conn(0x7f7a3c0b35b0 0x7f7a3c15ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a3bfff640 1 --2- 172.21.15.45:0/3861809695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a33fff640 1 -- 172.21.15.45:0/3861809695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b1fa0 msgr2=0x7f7a3c0b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:37.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a33fff640 1 --2- 172.21.15.45:0/3861809695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a33fff640 1 -- 172.21.15.45:0/3861809695 >> v2:172.21.15.190:3300/0 conn(0x7f7a3c0b35b0 msgr2=0x7f7a3c15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:37.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a33fff640 1 --2- 172.21.15.45:0/3861809695 >> v2:172.21.15.190:3300/0 conn(0x7f7a3c0b35b0 0x7f7a3c15ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.145+0000 7f7a33fff640 1 -- 172.21.15.45:0/3861809695 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c15c360 con 0x7f7a3c15b850 2023-11-28T20:34:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.147+0000 7f7a33fff640 1 --2- 172.21.15.45:0/3861809695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c15b850 0x7f7a3c0b0c30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c06e7f0 tx=0x7f7a2402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82091fd6422bc6e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.147+0000 7f7a3affd640 1 -- 172.21.15.45:0/3861809695 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24031870 con 0x7f7a3c15b850 2023-11-28T20:34:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.147+0000 7f7a3affd640 1 -- 172.21.15.45:0/3861809695 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a240319d0 con 0x7f7a3c15b850 2023-11-28T20:34:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.147+0000 7f7a3affd640 1 -- 172.21.15.45:0/3861809695 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24031d60 con 0x7f7a3c15b850 2023-11-28T20:34:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.148+0000 7f7a41e87640 1 -- 172.21.15.45:0/3861809695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c15b850 msgr2=0x7f7a3c0b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.148+0000 7f7a41e87640 1 --2- 172.21.15.45:0/3861809695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c15b850 0x7f7a3c0b0c30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c06e7f0 tx=0x7f7a2402b910 comp rx=0 tx=0).stop 2023-11-28T20:34:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.148+0000 7f7a41e87640 1 -- 172.21.15.45:0/3861809695 shutdown_connections 2023-11-28T20:34:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.148+0000 7f7a41e87640 1 --2- 172.21.15.45:0/3861809695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.148+0000 7f7a41e87640 1 --2- 172.21.15.45:0/3861809695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c15b850 0x7f7a3c0b0c30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.148+0000 7f7a41e87640 1 --2- 172.21.15.45:0/3861809695 >> v2:172.21.15.190:3300/0 conn(0x7f7a3c0b35b0 0x7f7a3c15ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.148+0000 7f7a41e87640 1 -- 172.21.15.45:0/3861809695 >> 172.21.15.45:0/3861809695 conn(0x7f7a3c06a500 msgr2=0x7f7a3c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.148+0000 7f7a41e87640 1 -- 172.21.15.45:0/3861809695 shutdown_connections 2023-11-28T20:34:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.148+0000 7f7a41e87640 1 -- 172.21.15.45:0/3861809695 wait complete. 2023-11-28T20:34:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.150+0000 7f7a41e87640 1 Processor -- start 2023-11-28T20:34:37.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.150+0000 7f7a41e87640 1 -- start start 2023-11-28T20:34:37.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.151+0000 7f7a41e87640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.151+0000 7f7a41e87640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b35b0 0x7f7a3c155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.151+0000 7f7a41e87640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c15b850 0x7f7a3c0c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.151+0000 7f7a41e87640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c0ac590 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.151+0000 7f7a41e87640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c1558c0 con 0x7f7a3c15b850 2023-11-28T20:34:37.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.151+0000 7f7a41e87640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c135770 con 0x7f7a3c0b35b0 2023-11-28T20:34:37.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.151+0000 7f7a3b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.151+0000 7f7a3b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52072/0 (socket says 172.21.15.45:52072) 2023-11-28T20:34:37.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.151+0000 7f7a3b7fe640 1 -- 172.21.15.45:0/1721284210 learned_addr learned my addr 172.21.15.45:0/1721284210 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:37.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a3bfff640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c15b850 0x7f7a3c0c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a33fff640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b35b0 0x7f7a3c155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a3b7fe640 1 -- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b35b0 msgr2=0x7f7a3c155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a3b7fe640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b35b0 0x7f7a3c155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a3b7fe640 1 -- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c15b850 msgr2=0x7f7a3c0c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a3b7fe640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c15b850 0x7f7a3c0c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a3b7fe640 1 -- 172.21.15.45:0/1721284210 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a240094d0 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a33fff640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b35b0 0x7f7a3c155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a3b7fe640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0ac050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00d800 tx=0x7f7a2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a38ff9640 1 -- 172.21.15.45:0/1721284210 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c00b6c0 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a38ff9640 1 -- 172.21.15.45:0/1721284210 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c00b820 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a38ff9640 1 -- 172.21.15.45:0/1721284210 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c00bbb0 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.152+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3c135a50 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.153+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c136010 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.154+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3c0733d0 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.155+0000 7f7a38ff9640 1 -- 172.21.15.45:0/1721284210 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7a2c00bd10 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.156+0000 7f7a38ff9640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7a1c06ae40 0x7f7a1c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.156+0000 7f7a38ff9640 1 -- 172.21.15.45:0/1721284210 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7a2c089e50 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.156+0000 7f7a33fff640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7a1c06ae40 0x7f7a1c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.157+0000 7f7a33fff640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7a1c06ae40 0x7f7a1c06d300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7a24036000 tx=0x7f7a2403c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:37.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.160+0000 7f7a38ff9640 1 -- 172.21.15.45:0/1721284210 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a2c054c00 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.353+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a3c0cd3b0 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.355+0000 7f7a38ff9640 1 -- 172.21.15.45:0/1721284210 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18334 (secure 0 0 0) 0x7f7a2c058220 con 0x7f7a3c0b1fa0 2023-11-28T20:34:37.360 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:37.360 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:33:58.584018+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:37.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7a1c06ae40 msgr2=0x7f7a1c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7a1c06ae40 0x7f7a1c06d300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7a24036000 tx=0x7f7a2403c040 comp rx=0 tx=0).stop 2023-11-28T20:34:37.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c0b1fa0 msgr2=0x7f7a3c0ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0ac050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00d800 tx=0x7f7a2c00dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 shutdown_connections 2023-11-28T20:34:37.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7a1c06ae40 0x7f7a1c06d300 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a3c15b850 0x7f7a3c0c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a3c0b35b0 0x7f7a3c155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 --2- 172.21.15.45:0/1721284210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a3c0b1fa0 0x7f7a3c0ac050 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 >> 172.21.15.45:0/1721284210 conn(0x7f7a3c06a500 msgr2=0x7f7a3c0afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:37.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 shutdown_connections 2023-11-28T20:34:37.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.359+0000 7f7a41e87640 1 -- 172.21.15.45:0/1721284210 wait complete. 2023-11-28T20:34:37.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:34:37.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.599+0000 7f03bf59e640 1 Processor -- start 2023-11-28T20:34:37.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.599+0000 7f03bf59e640 1 -- start start 2023-11-28T20:34:37.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.601+0000 7f03bf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c0155780 0x7f03c00abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.601+0000 7f03bf59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f03c00ce800 0x7f03c00b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.601+0000 7f03bf59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c015c3c0 con 0x7f03c00ae540 2023-11-28T20:34:37.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.601+0000 7f03bf59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c0155b60 con 0x7f03c00ce800 2023-11-28T20:34:37.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.601+0000 7f03bf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c015ba90 con 0x7f03c0155780 2023-11-28T20:34:37.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.601+0000 7f03bed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f03c00ce800 0x7f03c00b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.601+0000 7f03b5d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f03c00ae540 0x7f03c00ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33482/0 (socket says 172.21.15.45:33482) 2023-11-28T20:34:37.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.601+0000 7f03b5d9b640 1 -- 172.21.15.45:0/737370784 learned_addr learned my addr 172.21.15.45:0/737370784 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:37.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.602+0000 7f03be59c640 1 --2- 172.21.15.45:0/737370784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c0155780 0x7f03c00abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.602+0000 7f03bed9d640 1 -- 172.21.15.45:0/737370784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c0155780 msgr2=0x7f03c00abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:37.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.602+0000 7f03bed9d640 1 --2- 172.21.15.45:0/737370784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c0155780 0x7f03c00abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.602+0000 7f03bed9d640 1 -- 172.21.15.45:0/737370784 >> v1:172.21.15.45:6789/0 conn(0x7f03c00ae540 legacy=0x7f03c00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.602+0000 7f03bed9d640 1 -- 172.21.15.45:0/737370784 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c015bd70 con 0x7f03c00ce800 2023-11-28T20:34:37.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.602+0000 7f03bdd9b640 1 -- 172.21.15.45:0/737370784 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410139879 0 0) 0x7f03c015c3c0 con 0x7f03c00ae540 2023-11-28T20:34:37.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.603+0000 7f03bed9d640 1 --2- 172.21.15.45:0/737370784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ce800 0x7f03c00b20a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b676cf3d8a7bee3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:37.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.603+0000 7f03bdd9b640 1 -- 172.21.15.45:0/737370784 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8031830 con 0x7f03c00ce800 2023-11-28T20:34:37.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.604+0000 7f03bdd9b640 1 -- 172.21.15.45:0/737370784 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b8031990 con 0x7f03c00ce800 2023-11-28T20:34:37.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.604+0000 7f03bdd9b640 1 -- 172.21.15.45:0/737370784 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8031d20 con 0x7f03c00ce800 2023-11-28T20:34:37.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.604+0000 7f03bf59e640 1 -- 172.21.15.45:0/737370784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ce800 msgr2=0x7f03c00b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.604+0000 7f03bf59e640 1 --2- 172.21.15.45:0/737370784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ce800 0x7f03c00b20a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.604+0000 7f03bf59e640 1 -- 172.21.15.45:0/737370784 shutdown_connections 2023-11-28T20:34:37.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.604+0000 7f03bf59e640 1 --2- 172.21.15.45:0/737370784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ce800 0x7f03c00b20a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.604+0000 7f03bf59e640 1 --2- 172.21.15.45:0/737370784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c0155780 0x7f03c00abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.604+0000 7f03bf59e640 1 -- 172.21.15.45:0/737370784 >> 172.21.15.45:0/737370784 conn(0x7f03c006ab30 msgr2=0x7f03c00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.605+0000 7f03bf59e640 1 -- 172.21.15.45:0/737370784 shutdown_connections 2023-11-28T20:34:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.605+0000 7f03bf59e640 1 -- 172.21.15.45:0/737370784 wait complete. 2023-11-28T20:34:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.606+0000 7f03bf59e640 1 Processor -- start 2023-11-28T20:34:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.606+0000 7f03bf59e640 1 -- start start 2023-11-28T20:34:37.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.607+0000 7f03bf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ae540 0x7f03c01f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.607+0000 7f03bf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c00ce800 0x7f03c009a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.607+0000 7f03bf59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03c0155780 0x7f03c015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.607+0000 7f03bf59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c009bca0 con 0x7f03c00ae540 2023-11-28T20:34:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.607+0000 7f03bf59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01f61e0 con 0x7f03c0155780 2023-11-28T20:34:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.607+0000 7f03bf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c009b020 con 0x7f03c00ce800 2023-11-28T20:34:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03be59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ae540 0x7f03c01f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03be59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ae540 0x7f03c01f5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52096/0 (socket says 172.21.15.45:52096) 2023-11-28T20:34:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03be59c640 1 -- 172.21.15.45:0/666393407 learned_addr learned my addr 172.21.15.45:0/666393407 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03b5d9b640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c00ce800 0x7f03c009a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03be59c640 1 -- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c00ce800 msgr2=0x7f03c009a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03be59c640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c00ce800 0x7f03c009a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03be59c640 1 -- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03c0155780 msgr2=0x7f03c015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03bed9d640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03c0155780 0x7f03c015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03be59c640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03c0155780 0x7f03c015d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03be59c640 1 -- 172.21.15.45:0/666393407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b8009400 con 0x7f03c00ae540 2023-11-28T20:34:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.608+0000 7f03bed9d640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03c0155780 0x7f03c015d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.609+0000 7f03be59c640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ae540 0x7f03c01f5ca0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f03b000d5f0 tx=0x7f03b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.609+0000 7f03b77fe640 1 -- 172.21.15.45:0/666393407 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b00125f0 con 0x7f03c00ae540 2023-11-28T20:34:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.609+0000 7f03b77fe640 1 -- 172.21.15.45:0/666393407 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b0012750 con 0x7f03c00ae540 2023-11-28T20:34:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.609+0000 7f03b77fe640 1 -- 172.21.15.45:0/666393407 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0012b00 con 0x7f03c00ae540 2023-11-28T20:34:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.609+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c009b300 con 0x7f03c00ae540 2023-11-28T20:34:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.609+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c009b8c0 con 0x7f03c00ae540 2023-11-28T20:34:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.610+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c004e920 con 0x7f03c00ae540 2023-11-28T20:34:37.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.611+0000 7f03b77fe640 1 -- 172.21.15.45:0/666393407 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f03b0012c60 con 0x7f03c00ae540 2023-11-28T20:34:37.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.612+0000 7f03b77fe640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03a006ae40 0x7f03a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:37.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.612+0000 7f03b77fe640 1 -- 172.21.15.45:0/666393407 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f03b0089d40 con 0x7f03c00ae540 2023-11-28T20:34:37.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.613+0000 7f03b5d9b640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03a006ae40 0x7f03a006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:37.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.614+0000 7f03b5d9b640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03a006ae40 0x7f03a006d300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f03c009cb40 tx=0x7f03a8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:37.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.616+0000 7f03b77fe640 1 -- 172.21.15.45:0/666393407 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03b0054b90 con 0x7f03c00ae540 2023-11-28T20:34:37.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.849+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03c015d490 con 0x7f03c00ae540 2023-11-28T20:34:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.850+0000 7f03b77fe640 1 -- 172.21.15.45:0/666393407 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4915 (secure 0 0 0) 0x7f03b00581b0 con 0x7f03c00ae540 2023-11-28T20:34:37.853 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:37.853 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:34:22.516874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T20:34:37.854 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2023-11-28T20:34:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03a006ae40 msgr2=0x7f03a006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03a006ae40 0x7f03a006d300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f03c009cb40 tx=0x7f03a8009040 comp rx=0 tx=0).stop 2023-11-28T20:34:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ae540 msgr2=0x7f03c01f5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ae540 0x7f03c01f5ca0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f03b000d5f0 tx=0x7f03b000dab0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 shutdown_connections 2023-11-28T20:34:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f03a006ae40 0x7f03a006d300 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03c0155780 0x7f03c015d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03c00ce800 0x7f03c009a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 --2- 172.21.15.45:0/666393407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ae540 0x7f03c01f5ca0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:37.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 >> 172.21.15.45:0/666393407 conn(0x7f03c006ab30 msgr2=0x7f03c015c870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:37.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.853+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 shutdown_connections 2023-11-28T20:34:37.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:37.854+0000 7f03bf59e640 1 -- 172.21.15.45:0/666393407 wait complete. 2023-11-28T20:34:38.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:33:53.520990+0000', 'modified': '2023-11-28T20:34:22.516874+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/2066232376', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6834', 'nonce': 2066232376}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 2066232376}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T20:34:38.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:34:38.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T20:34:38.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.509+0000 7f95259ff640 1 Processor -- start 2023-11-28T20:34:38.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.509+0000 7f95259ff640 1 -- start start 2023-11-28T20:34:38.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.511+0000 7f95259ff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f95200cf610 0x7f95200ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.511+0000 7f95259ff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95200b2fb0 0x7f952015eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.511+0000 7f95259ff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95200abd70 con 0x7f952015b070 2023-11-28T20:34:38.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.511+0000 7f95259ff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952015b450 con 0x7f95200cf610 2023-11-28T20:34:38.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.511+0000 7f95259ff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95200b3ed0 con 0x7f95200b2fb0 2023-11-28T20:34:38.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.511+0000 7f95249fd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f952015b070 0x7f95200cef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33492/0 (socket says 172.21.15.45:33492) 2023-11-28T20:34:38.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.511+0000 7f95249fd640 1 -- 172.21.15.45:0/3219111511 learned_addr learned my addr 172.21.15.45:0/3219111511 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.511+0000 7f951ffff640 1 --2- 172.21.15.45:0/3219111511 >> v2:172.21.15.190:3300/0 conn(0x7f95200cf610 0x7f95200ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f95251fe640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95200b2fb0 0x7f952015eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505346006 0 0) 0x7f95200abd70 con 0x7f952015b070 2023-11-28T20:34:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9510003520 con 0x7f952015b070 2023-11-28T20:34:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3882266587 0 0) 0x7f9510003520 con 0x7f952015b070 2023-11-28T20:34:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95200b2fb0 msgr2=0x7f952015eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95200b2fb0 0x7f952015eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 >> v2:172.21.15.190:3300/0 conn(0x7f95200cf610 msgr2=0x7f95200ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 --2- 172.21.15.45:0/3219111511 >> v2:172.21.15.190:3300/0 conn(0x7f95200cf610 0x7f95200ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95200abfc0 con 0x7f952015b070 2023-11-28T20:34:38.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f9514001670 con 0x7f952015b070 2023-11-28T20:34:38.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 >> v1:172.21.15.45:6789/0 conn(0x7f952015b070 legacy=0x7f95200cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9510003970 0x7f9510003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95200b2fb0 0x7f95100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9510004d00 0x7f9510005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95200b3ed0 con 0x7f9510004d00 2023-11-28T20:34:38.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951000b980 con 0x7f9510003970 2023-11-28T20:34:38.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.512+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951000baf0 con 0x7f95200b2fb0 2023-11-28T20:34:38.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.513+0000 7f95251fe640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9510004d00 0x7f9510005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.513+0000 7f95249fd640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9510003970 0x7f9510003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.513+0000 7f951ffff640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95200b2fb0 0x7f95100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.513+0000 7f95251fe640 1 -- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95200b2fb0 msgr2=0x7f95100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.513+0000 7f95251fe640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95200b2fb0 0x7f95100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.513+0000 7f95251fe640 1 -- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9510003970 msgr2=0x7f9510003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.513+0000 7f95251fe640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9510003970 0x7f9510003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.513+0000 7f95251fe640 1 -- 172.21.15.45:0/3219111511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9510003520 con 0x7f9510004d00 2023-11-28T20:34:38.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.514+0000 7f95251fe640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9510004d00 0x7f9510005180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9518009fc0 tx=0x7f95100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca64a4ec01a0201 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:38.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.515+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951802cc20 con 0x7f9510004d00 2023-11-28T20:34:38.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.515+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951800f040 con 0x7f9510004d00 2023-11-28T20:34:38.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.515+0000 7f951f7fe640 1 -- 172.21.15.45:0/3219111511 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518034440 con 0x7f9510004d00 2023-11-28T20:34:38.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.515+0000 7f95259ff640 1 -- 172.21.15.45:0/3219111511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9510004d00 msgr2=0x7f9510005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.515+0000 7f95259ff640 1 --2- 172.21.15.45:0/3219111511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9510004d00 0x7f9510005180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9518009fc0 tx=0x7f95100057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.515+0000 7f95251fe640 1 -- 172.21.15.45:0/3219111511 reap_dead start 2023-11-28T20:34:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.515+0000 7f95259ff640 1 -- 172.21.15.45:0/3219111511 shutdown_connections 2023-11-28T20:34:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.515+0000 7f95259ff640 1 -- 172.21.15.45:0/3219111511 >> 172.21.15.45:0/3219111511 conn(0x7f952006a710 msgr2=0x7f952006ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.516+0000 7f95259ff640 1 -- 172.21.15.45:0/3219111511 shutdown_connections 2023-11-28T20:34:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.516+0000 7f95259ff640 1 -- 172.21.15.45:0/3219111511 wait complete. 2023-11-28T20:34:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.517+0000 7f95259ff640 1 Processor -- start 2023-11-28T20:34:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.518+0000 7f95259ff640 1 -- start start 2023-11-28T20:34:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.518+0000 7f95259ff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9510004d00 0x7f95201485b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.518+0000 7f95259ff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95201591e0 0x7f95201452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95259ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9520071400 0x7f9520071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95259ff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952015a420 con 0x7f9520071400 2023-11-28T20:34:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95259ff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9520071dc0 con 0x7f95201591e0 2023-11-28T20:34:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95259ff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9520157630 con 0x7f9510004d00 2023-11-28T20:34:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95251fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9520071400 0x7f9520071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95251fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9520071400 0x7f9520071880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52114/0 (socket says 172.21.15.45:52114) 2023-11-28T20:34:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95251fe640 1 -- 172.21.15.45:0/2923176423 learned_addr learned my addr 172.21.15.45:0/2923176423 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95249fd640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9510004d00 0x7f95201485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f951ffff640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95201591e0 0x7f95201452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95251fe640 1 -- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9510004d00 msgr2=0x7f95201485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95251fe640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9510004d00 0x7f95201485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95251fe640 1 -- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95201591e0 msgr2=0x7f95201452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95251fe640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95201591e0 0x7f95201452f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.519+0000 7f95251fe640 1 -- 172.21.15.45:0/2923176423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952015a840 con 0x7f9520071400 2023-11-28T20:34:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.520+0000 7f95251fe640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9520071400 0x7f9520071880 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9518002230 tx=0x7f9518009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.520+0000 7f951d7fa640 1 -- 172.21.15.45:0/2923176423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518034ad0 con 0x7f9520071400 2023-11-28T20:34:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.520+0000 7f951d7fa640 1 -- 172.21.15.45:0/2923176423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9518034c30 con 0x7f9520071400 2023-11-28T20:34:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.520+0000 7f95259ff640 1 -- 172.21.15.45:0/2923176423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9520148af0 con 0x7f9520071400 2023-11-28T20:34:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.520+0000 7f951d7fa640 1 -- 172.21.15.45:0/2923176423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518003990 con 0x7f9520071400 2023-11-28T20:34:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.520+0000 7f95259ff640 1 -- 172.21.15.45:0/2923176423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95201597a0 con 0x7f9520071400 2023-11-28T20:34:38.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.523+0000 7f951d7fa640 1 -- 172.21.15.45:0/2923176423 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f951803e440 con 0x7f9520071400 2023-11-28T20:34:38.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.524+0000 7f951d7fa640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f950406ae70 0x7f950406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.525+0000 7f951d7fa640 1 -- 172.21.15.45:0/2923176423 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f95180ab6f0 con 0x7f9520071400 2023-11-28T20:34:38.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.525+0000 7f95249fd640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f950406ae70 0x7f950406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.525+0000 7f95297f5740 1 Processor -- start 2023-11-28T20:34:38.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.525+0000 7f95297f5740 1 -- start start 2023-11-28T20:34:38.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.526+0000 7f95249fd640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f950406ae70 0x7f950406d330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9514001050 tx=0x7f9514009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.526+0000 7f95297f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b0004a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.526+0000 7f95297f5740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 0x559b00049270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.526+0000 7f95297f5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 0x559b0008d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.526+0000 7f95297f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afffa8880 con 0x559b0004a070 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.526+0000 7f95297f5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afffb7ae0 con 0x559b000497b0 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.526+0000 7f95297f5740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x559afffb51d0 con 0x559b00044d70 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95249fd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b0004a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95249fd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b0004a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52122/0 (socket says 172.21.15.45:52122) 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95249fd640 1 -- 172.21.15.45:0/2483523735 learned_addr learned my addr 172.21.15.45:0/2483523735 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:38.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95251fe640 1 --2- 172.21.15.45:0/2483523735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 0x559b0008d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f951ffff640 1 --2- 172.21.15.45:0/2483523735 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 0x559b00049270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95249fd640 1 -- 172.21.15.45:0/2483523735 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 msgr2=0x559b00049270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95249fd640 1 --2- 172.21.15.45:0/2483523735 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 0x559b00049270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95249fd640 1 -- 172.21.15.45:0/2483523735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 msgr2=0x559b0008d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95249fd640 1 --2- 172.21.15.45:0/2483523735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 0x559b0008d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95249fd640 1 -- 172.21.15.45:0/2483523735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559afff78860 con 0x559b0004a070 2023-11-28T20:34:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.527+0000 7f95249fd640 1 --2- 172.21.15.45:0/2483523735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b0004a450 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9514010bf0 tx=0x7f951400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4feff8a4e79ba74a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f950a7fc640 1 -- 172.21.15.45:0/2483523735 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9514005250 con 0x559b0004a070 2023-11-28T20:34:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f950a7fc640 1 -- 172.21.15.45:0/2483523735 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95140053b0 con 0x559b0004a070 2023-11-28T20:34:38.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f950a7fc640 1 -- 172.21.15.45:0/2483523735 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9514005720 con 0x559b0004a070 2023-11-28T20:34:38.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f95297f5740 1 -- 172.21.15.45:0/2483523735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 msgr2=0x559b0004a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f95297f5740 1 --2- 172.21.15.45:0/2483523735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b0004a450 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9514010bf0 tx=0x7f951400e000 comp rx=0 tx=0).stop 2023-11-28T20:34:38.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f95297f5740 1 -- 172.21.15.45:0/2483523735 shutdown_connections 2023-11-28T20:34:38.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f95297f5740 1 --2- 172.21.15.45:0/2483523735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 0x559b0008d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f95297f5740 1 --2- 172.21.15.45:0/2483523735 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 0x559b00049270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f95297f5740 1 --2- 172.21.15.45:0/2483523735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b0004a450 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f95297f5740 1 -- 172.21.15.45:0/2483523735 >> 172.21.15.45:0/2483523735 conn(0x559b0000bc70 msgr2=0x559b0008fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f95297f5740 1 -- 172.21.15.45:0/2483523735 shutdown_connections 2023-11-28T20:34:38.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.528+0000 7f95297f5740 1 -- 172.21.15.45:0/2483523735 wait complete. 2023-11-28T20:34:38.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.529+0000 7f95297f5740 1 -- start start 2023-11-28T20:34:38.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.529+0000 7f95297f5740 1 Processor -- start 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.530+0000 7f95297f5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b00044d70 0x559b00095940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.530+0000 7f95297f5740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b000497b0 0x559b00092890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.530+0000 7f95297f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00092dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.530+0000 7f95297f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afffb0ac0 con 0x559b0004a070 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.530+0000 7f95297f5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afff4d2f0 con 0x559b00044d70 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.530+0000 7f95297f5740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x559afff6ac70 con 0x559b000497b0 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.530+0000 7f95297f5740 10 client.0 _ll_register_callbacks cb 0x559afff6f3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95251fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00092dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95251fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00092dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52134/0 (socket says 172.21.15.45:52134) 2023-11-28T20:34:38.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95251fe640 1 -- 172.21.15.45:0/3595854144 learned_addr learned my addr 172.21.15.45:0/3595854144 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95249fd640 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b00044d70 0x559b00095940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95251fe640 1 -- 172.21.15.45:0/3595854144 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b000497b0 msgr2=0x559b00092890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95251fe640 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b000497b0 0x559b00092890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95251fe640 1 -- 172.21.15.45:0/3595854144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b00044d70 msgr2=0x559b00095940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95251fe640 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b00044d70 0x559b00095940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95251fe640 1 -- 172.21.15.45:0/3595854144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951400f040 con 0x559b0004a070 2023-11-28T20:34:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95249fd640 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b00044d70 0x559b00095940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f95251fe640 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00092dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f951802c5c0 tx=0x7f951807a990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b7305102b1f04 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:38.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f94ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:34:38.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.531+0000 7f94ff7fe640 1 -- 172.21.15.45:0/3595854144 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518082460 con 0x559b0004a070 2023-11-28T20:34:38.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f94ff7fe640 1 -- 172.21.15.45:0/3595854144 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9518076870 con 0x559b0004a070 2023-11-28T20:34:38.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f95297f5740 1 -- 172.21.15.45:0/3595854144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559afffa9aa0 con 0x559b0004a070 2023-11-28T20:34:38.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f94ff7fe640 1 -- 172.21.15.45:0/3595854144 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9518076a30 con 0x559b0004a070 2023-11-28T20:34:38.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f95297f5740 10 client.4839 fetch_fsmap learned FSMap version 9 2023-11-28T20:34:38.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f95297f5740 1 -- 172.21.15.45:0/3595854144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x559afffaf140 con 0x559b0004a070 2023-11-28T20:34:38.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f94ff7fe640 1 -- 172.21.15.45:0/3595854144 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559afffa9aa0 con 0x559b0004a070 2023-11-28T20:34:38.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f94ff7fe640 1 -- 172.21.15.45:0/3595854144 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f951807a3c0 con 0x559b0004a070 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f95297f5740 10 client.4839 fetch_fsmap finished waiting for FSMap version 9 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f95297f5740 10 client.4839 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f95297f5740 20 client.4839 populate_metadata read hostname 'smithi045' 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f95297f5740 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x559b00094940 0x559b00097230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f95297f5740 4 client.4839 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.532+0000 7f95297f5740 1 -- 172.21.15.45:0/3595854144 --> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559afffa97c0 con 0x559b00094940 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.533+0000 7f95249fd640 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x559b00094940 0x559b00097230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.533+0000 7f95249fd640 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x559b00094940 0x559b00097230 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81292e5a68a1c40b server_cookie=1082ba19ab052a75 in_seq=0 out_seq=0 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.533+0000 7f94ff7fe640 10 client.4839 ms_handle_connect on v2:172.21.15.45:6834/2066232376 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f94ff7fe640 1 -- 172.21.15.45:0/3595854144 <== mds.0 v2:172.21.15.45:6834/2066232376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559afffa97c0 con 0x559b00094940 2023-11-28T20:34:38.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f94ff7fe640 10 client.4839 handle_command_reply: tid=0 2023-11-28T20:34:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 client.4839 shutdown 2023-11-28T20:34:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 -- 172.21.15.45:0/3595854144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 msgr2=0x559b00092dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00092dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f951802c5c0 tx=0x7f951807a990 comp rx=0 tx=0).stop 2023-11-28T20:34:38.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 -- 172.21.15.45:0/3595854144 shutdown_connections 2023-11-28T20:34:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x559b00094940 0x559b00097230 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00092dd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b000497b0 0x559b00092890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 --2- 172.21.15.45:0/3595854144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b00044d70 0x559b00095940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f94ff7fe640 0 client.4839 ms_handle_reset on v2:172.21.15.45:6834/2066232376 2023-11-28T20:34:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 -- 172.21.15.45:0/3595854144 >> 172.21.15.45:0/3595854144 conn(0x559b0000bc70 msgr2=0x559b0008fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 -- 172.21.15.45:0/3595854144 shutdown_connections 2023-11-28T20:34:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.534+0000 7f95297f5740 1 -- 172.21.15.45:0/3595854144 wait complete. 2023-11-28T20:34:38.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.535+0000 7f95297f5740 20 client.4839 trim_cache size 0 max 16384 2023-11-28T20:34:38.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.552+0000 7f95297f5740 1 Processor -- start 2023-11-28T20:34:38.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.552+0000 7f95297f5740 1 -- start start 2023-11-28T20:34:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.552+0000 7f95297f5740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 0x559b000873e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95297f5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 0x559b00093660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95297f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00093ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95297f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559affd13f20 con 0x559b0004a070 2023-11-28T20:34:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95297f5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b00090580 con 0x559b000497b0 2023-11-28T20:34:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95297f5740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b0004c160 con 0x559b00044d70 2023-11-28T20:34:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95251fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00093ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95251fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00093ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52138/0 (socket says 172.21.15.45:52138) 2023-11-28T20:34:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95251fe640 1 -- 172.21.15.45:0/3670877567 learned_addr learned my addr 172.21.15.45:0/3670877567 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f951ffff640 1 --2- 172.21.15.45:0/3670877567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 0x559b00093660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95249fd640 1 --2- 172.21.15.45:0/3670877567 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 0x559b000873e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95251fe640 1 -- 172.21.15.45:0/3670877567 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 msgr2=0x559b000873e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95251fe640 1 --2- 172.21.15.45:0/3670877567 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 0x559b000873e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95251fe640 1 -- 172.21.15.45:0/3670877567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 msgr2=0x559b00093660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95251fe640 1 --2- 172.21.15.45:0/3670877567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 0x559b00093660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.553+0000 7f95251fe640 1 -- 172.21.15.45:0/3670877567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559afff4cb80 con 0x559b0004a070 2023-11-28T20:34:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.554+0000 7f95251fe640 1 --2- 172.21.15.45:0/3670877567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00093ba0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95180784b0 tx=0x7f9518040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c34d3d74fae3a08 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.554+0000 7f950affd640 1 -- 172.21.15.45:0/3670877567 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518082460 con 0x559b0004a070 2023-11-28T20:34:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.554+0000 7f950affd640 1 -- 172.21.15.45:0/3670877567 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9518079650 con 0x559b0004a070 2023-11-28T20:34:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.554+0000 7f950affd640 1 -- 172.21.15.45:0/3670877567 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518009090 con 0x559b0004a070 2023-11-28T20:34:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.554+0000 7f95297f5740 1 -- 172.21.15.45:0/3670877567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 msgr2=0x559b00093ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.554+0000 7f95297f5740 1 --2- 172.21.15.45:0/3670877567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00093ba0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95180784b0 tx=0x7f9518040000 comp rx=0 tx=0).stop 2023-11-28T20:34:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.555+0000 7f95297f5740 1 -- 172.21.15.45:0/3670877567 shutdown_connections 2023-11-28T20:34:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.555+0000 7f95297f5740 1 --2- 172.21.15.45:0/3670877567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b0004a070 0x559b00093ba0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.555+0000 7f95297f5740 1 --2- 172.21.15.45:0/3670877567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b000497b0 0x559b00093660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.555+0000 7f95297f5740 1 --2- 172.21.15.45:0/3670877567 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b00044d70 0x559b000873e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.555+0000 7f95297f5740 1 -- 172.21.15.45:0/3670877567 >> 172.21.15.45:0/3670877567 conn(0x559b0000bc70 msgr2=0x559b0008f250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.555+0000 7f95297f5740 1 -- 172.21.15.45:0/3670877567 shutdown_connections 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.555+0000 7f95297f5740 1 -- 172.21.15.45:0/3670877567 wait complete. 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.555+0000 7f95297f5740 1 -- start start 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.556+0000 7f95297f5740 1 Processor -- start 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95297f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b00044d70 0x559b000f8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95297f5740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b000497b0 0x559b00097590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95297f5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b0004a070 0x559b00097ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95297f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b000b1fc0 con 0x559b00044d70 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95297f5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0008abd0 con 0x559b0004a070 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95297f5740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b00093c00 con 0x559b000497b0 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95297f5740 10 client.0 _ll_register_callbacks cb 0x7f9518079d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:34:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95249fd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b00044d70 0x559b000f8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95249fd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b00044d70 0x559b000f8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52150/0 (socket says 172.21.15.45:52150) 2023-11-28T20:34:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95249fd640 1 -- 172.21.15.45:0/516134872 learned_addr learned my addr 172.21.15.45:0/516134872 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95249fd640 1 -- 172.21.15.45:0/516134872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b000497b0 msgr2=0x559b00097590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95249fd640 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b000497b0 0x559b00097590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95251fe640 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b0004a070 0x559b00097ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95249fd640 1 -- 172.21.15.45:0/516134872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b0004a070 msgr2=0x559b00097ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95249fd640 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b0004a070 0x559b00097ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.557+0000 7f95249fd640 1 -- 172.21.15.45:0/516134872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951400f040 con 0x559b00044d70 2023-11-28T20:34:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.558+0000 7f95249fd640 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b00044d70 0x559b000f8500 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9514010bc0 tx=0x7f95140019a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2620c4e1ca38f817 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.558+0000 7f94ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.558+0000 7f94ff7fe640 1 -- 172.21.15.45:0/516134872 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9514001670 con 0x559b00044d70 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.558+0000 7f94ff7fe640 1 -- 172.21.15.45:0/516134872 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9514031e40 con 0x559b00044d70 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.558+0000 7f95297f5740 1 -- 172.21.15.45:0/516134872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95140053b0 con 0x559b00044d70 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.558+0000 7f94ff7fe640 1 -- 172.21.15.45:0/516134872 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9514005820 con 0x559b00044d70 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.558+0000 7f95297f5740 10 client.4845 fetch_fsmap learned FSMap version 9 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.558+0000 7f95297f5740 1 -- 172.21.15.45:0/516134872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x559afff72170 con 0x559b00044d70 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.559+0000 7f94ff7fe640 1 -- 172.21.15.45:0/516134872 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95140053b0 con 0x559b00044d70 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.559+0000 7f94ff7fe640 1 -- 172.21.15.45:0/516134872 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f95140310d0 con 0x559b00044d70 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.559+0000 7f95297f5740 10 client.4845 fetch_fsmap finished waiting for FSMap version 9 2023-11-28T20:34:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.559+0000 7f95297f5740 10 client.4845 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:34:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.559+0000 7f95297f5740 20 client.4845 populate_metadata read hostname 'smithi045' 2023-11-28T20:34:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.559+0000 7f95297f5740 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x559b000443d0 0x559b00104010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.559+0000 7f95297f5740 4 client.4845 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:34:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.559+0000 7f95297f5740 1 -- 172.21.15.45:0/516134872 --> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559affcb16c0 con 0x559b000443d0 2023-11-28T20:34:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.559+0000 7f951ffff640 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x559b000443d0 0x559b00104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f951ffff640 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x559b000443d0 0x559b00104010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6d86d6ffa92aab5 server_cookie=a8e8f4ca089cdad7 in_seq=0 out_seq=0 2023-11-28T20:34:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f94ff7fe640 10 client.4845 ms_handle_connect on v2:172.21.15.45:6834/2066232376 2023-11-28T20:34:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f94ff7fe640 1 -- 172.21.15.45:0/516134872 <== mds.0 v2:172.21.15.45:6834/2066232376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x559affcb16c0 con 0x559b000443d0 2023-11-28T20:34:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f94ff7fe640 10 client.4845 handle_command_reply: tid=0 2023-11-28T20:34:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 client.4845 shutdown 2023-11-28T20:34:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 -- 172.21.15.45:0/516134872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b00044d70 msgr2=0x559b000f8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b00044d70 0x559b000f8500 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9514010bc0 tx=0x7f95140019a0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 -- 172.21.15.45:0/516134872 shutdown_connections 2023-11-28T20:34:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.45:6834/2066232376,v1:172.21.15.45:6835/2066232376] conn(0x559b000443d0 0x559b00104010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559b0004a070 0x559b00097ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559b000497b0 0x559b00097590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 --2- 172.21.15.45:0/516134872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b00044d70 0x559b000f8500 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f94ff7fe640 0 client.4845 ms_handle_reset on v2:172.21.15.45:6834/2066232376 2023-11-28T20:34:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 -- 172.21.15.45:0/516134872 >> 172.21.15.45:0/516134872 conn(0x559b0000bc70 msgr2=0x559b00045150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 -- 172.21.15.45:0/516134872 shutdown_connections 2023-11-28T20:34:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.560+0000 7f95297f5740 1 -- 172.21.15.45:0/516134872 wait complete. 2023-11-28T20:34:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95297f5740 20 client.4845 trim_cache size 0 max 16384 2023-11-28T20:34:38.570 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:34:38.570 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:34:38.570 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:34:38.570 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 4257, 2023-11-28T20:34:38.570 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:34:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:34:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:34:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 44.507366670000003, 2023-11-28T20:34:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 9, 2023-11-28T20:34:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 25, 2023-11-28T20:34:38.571 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 25, 2023-11-28T20:34:38.572 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 52.228431602000001 2023-11-28T20:34:38.572 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:34:38.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 -- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f950406ae70 msgr2=0x7f950406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f950406ae70 0x7f950406d330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9514001050 tx=0x7f9514009040 comp rx=0 tx=0).stop 2023-11-28T20:34:38.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 -- 172.21.15.45:0/2923176423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9520071400 msgr2=0x7f9520071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:38.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9520071400 0x7f9520071880 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9518002230 tx=0x7f9518009750 comp rx=0 tx=0).stop 2023-11-28T20:34:38.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 -- 172.21.15.45:0/2923176423 shutdown_connections 2023-11-28T20:34:38.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f950406ae70 0x7f950406d330 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9520071400 0x7f9520071880 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95201591e0 0x7f95201452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 --2- 172.21.15.45:0/2923176423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9510004d00 0x7f95201485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 -- 172.21.15.45:0/2923176423 >> 172.21.15.45:0/2923176423 conn(0x7f952006a710 msgr2=0x7f952009ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.561+0000 7f95259ff640 1 -- 172.21.15.45:0/2923176423 shutdown_connections 2023-11-28T20:34:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:38.562+0000 7f95259ff640 1 -- 172.21.15.45:0/2923176423 wait complete. 2023-11-28T20:34:38.581 INFO:teuthology.run_tasks:Running task kclient... 2023-11-28T20:34:38.600 INFO:tasks.kclient:Mounting kernel clients... 2023-11-28T20:34:38.601 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-11-28T20:34:38.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:38.601 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:34:38.659 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-11-28T20:34:38.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:38.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T20:34:38.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:38.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2023-11-28T20:34:38.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:38.895 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:34:38.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:38.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2023-11-28T20:34:39.080 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2023-11-28T20:34:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T20:34:39.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:39.083 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2023-11-28T20:34:39.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:39.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link delete ceph-brx 2023-11-28T20:34:39.244 INFO:teuthology.orchestra.run.smithi190.stderr:Cannot find device "ceph-brx" 2023-11-28T20:34:39.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T20:34:39.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:39.248 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2023-11-28T20:34:39.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:39.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link delete ceph-brx 2023-11-28T20:34:39.391 INFO:teuthology.orchestra.run.smithi190.stderr:Cannot find device "ceph-brx" 2023-11-28T20:34:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T20:34:39.395 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-11-28T20:34:39.395 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T20:34:39.395 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:34:39.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:34:39.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.625+0000 7f352a5a5640 1 Processor -- start 2023-11-28T20:34:39.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.625+0000 7f352a5a5640 1 -- start start 2023-11-28T20:34:39.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.626+0000 7f352a5a5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3524155780 0x7f35240abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:39.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.626+0000 7f352a5a5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35240ce800 0x7f35240b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:39.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.626+0000 7f352a5a5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352415c3c0 con 0x7f35240ae540 2023-11-28T20:34:39.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.626+0000 7f352a5a5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3524155b60 con 0x7f3524155780 2023-11-28T20:34:39.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.626+0000 7f352a5a5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352415ba90 con 0x7f35240ce800 2023-11-28T20:34:39.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.626+0000 7f3528da2640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f35240ae540 0x7f35240ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33498/0 (socket says 172.21.15.45:33498) 2023-11-28T20:34:39.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.626+0000 7f3528da2640 1 -- 172.21.15.45:0/771390972 learned_addr learned my addr 172.21.15.45:0/771390972 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:39.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f35295a3640 1 --2- 172.21.15.45:0/771390972 >> v2:172.21.15.190:3300/0 conn(0x7f3524155780 0x7f35240abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:39.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038136870 0 0) 0x7f352415c3c0 con 0x7f35240ae540 2023-11-28T20:34:39.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3510003520 con 0x7f35240ae540 2023-11-28T20:34:39.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f3529da4640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35240ce800 0x7f35240b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:39.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1758493767 0 0) 0x7f3510003520 con 0x7f35240ae540 2023-11-28T20:34:39.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35240ce800 msgr2=0x7f35240b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:39.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35240ce800 0x7f35240b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 >> v2:172.21.15.190:3300/0 conn(0x7f3524155780 msgr2=0x7f35240abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 --2- 172.21.15.45:0/771390972 >> v2:172.21.15.190:3300/0 conn(0x7f3524155780 0x7f35240abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352415bd70 con 0x7f35240ae540 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3514001670 con 0x7f35240ae540 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 >> v1:172.21.15.45:6789/0 conn(0x7f35240ae540 legacy=0x7f35240ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.627+0000 7f351bfff640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f3510003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f351bfff640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35240ce800 0x7f35100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f351bfff640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3510004d00 0x7f3510005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3524155b60 con 0x7f3510003970 2023-11-28T20:34:39.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352415ba90 con 0x7f3510004d00 2023-11-28T20:34:39.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351000b980 con 0x7f35240ce800 2023-11-28T20:34:39.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f35295a3640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f3510003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:39.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f3529da4640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3510004d00 0x7f3510005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:39.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f35295a3640 1 -- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35240ce800 msgr2=0x7f35100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:39.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f35295a3640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35240ce800 0x7f35100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f35295a3640 1 -- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3510004d00 msgr2=0x7f3510005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:39.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f35295a3640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3510004d00 0x7f3510005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f35295a3640 1 -- 172.21.15.45:0/771390972 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510003520 con 0x7f3510003970 2023-11-28T20:34:39.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.628+0000 7f3529da4640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3510004d00 0x7f3510005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:39.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.629+0000 7f35295a3640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f3510003d90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3520009ac0 tx=0x7f35100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8c9e96ffd29da server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:39.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352002c6d0 con 0x7f3510003970 2023-11-28T20:34:39.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352002c830 con 0x7f3510003970 2023-11-28T20:34:39.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f351bfff640 1 -- 172.21.15.45:0/771390972 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352002cbc0 con 0x7f3510003970 2023-11-28T20:34:39.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f352a5a5640 1 -- 172.21.15.45:0/771390972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 msgr2=0x7f3510003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:39.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f352a5a5640 1 --2- 172.21.15.45:0/771390972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f3510003d90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3520009ac0 tx=0x7f35100057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f3529da4640 1 -- 172.21.15.45:0/771390972 reap_dead start 2023-11-28T20:34:39.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f352a5a5640 1 -- 172.21.15.45:0/771390972 shutdown_connections 2023-11-28T20:34:39.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f352a5a5640 1 -- 172.21.15.45:0/771390972 >> 172.21.15.45:0/771390972 conn(0x7f352406ab30 msgr2=0x7f35240b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:39.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f352a5a5640 1 -- 172.21.15.45:0/771390972 shutdown_connections 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.630+0000 7f352a5a5640 1 -- 172.21.15.45:0/771390972 wait complete. 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.632+0000 7f352a5a5640 1 Processor -- start 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.632+0000 7f352a5a5640 1 -- start start 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.633+0000 7f352a5a5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f35241f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.633+0000 7f352a5a5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3524152140 0x7f3524153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.633+0000 7f352a5a5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3524143f00 0x7f352409c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.633+0000 7f352a5a5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352414a8f0 con 0x7f3510003970 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.633+0000 7f352a5a5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241f71b0 con 0x7f3524143f00 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.633+0000 7f352a5a5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3524153c50 con 0x7f3524152140 2023-11-28T20:34:39.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.633+0000 7f35295a3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f35241f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:39.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f35295a3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f35241f6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52176/0 (socket says 172.21.15.45:52176) 2023-11-28T20:34:39.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f35295a3640 1 -- 172.21.15.45:0/1326026920 learned_addr learned my addr 172.21.15.45:0/1326026920 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:39.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f3529da4640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3524143f00 0x7f352409c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:39.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f35295a3640 1 -- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3524152140 msgr2=0x7f3524153710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:39.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f35295a3640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3524152140 0x7f3524153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f35295a3640 1 -- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3524143f00 msgr2=0x7f352409c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:39.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f35295a3640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3524143f00 0x7f352409c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f35295a3640 1 -- 172.21.15.45:0/1326026920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3524152670 con 0x7f3510003970 2023-11-28T20:34:39.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f3529da4640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3524143f00 0x7f352409c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:39.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.634+0000 7f35295a3640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f35241f6c70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f352002c2d0 tx=0x7f3520004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:39.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.635+0000 7f3519ffb640 1 -- 172.21.15.45:0/1326026920 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520003810 con 0x7f3510003970 2023-11-28T20:34:39.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.635+0000 7f3519ffb640 1 -- 172.21.15.45:0/1326026920 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3520003970 con 0x7f3510003970 2023-11-28T20:34:39.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.635+0000 7f3519ffb640 1 -- 172.21.15.45:0/1326026920 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520003ce0 con 0x7f3510003970 2023-11-28T20:34:39.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.635+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3524152950 con 0x7f3510003970 2023-11-28T20:34:39.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.635+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3524152f10 con 0x7f3510003970 2023-11-28T20:34:39.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.636+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3524099e50 con 0x7f3510003970 2023-11-28T20:34:39.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.637+0000 7f3519ffb640 1 -- 172.21.15.45:0/1326026920 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3520040450 con 0x7f3510003970 2023-11-28T20:34:39.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.638+0000 7f3519ffb640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f350806ae40 0x7f350806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:39.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.638+0000 7f3519ffb640 1 -- 172.21.15.45:0/1326026920 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f35200ab790 con 0x7f3510003970 2023-11-28T20:34:39.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.639+0000 7f3528da2640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f350806ae40 0x7f350806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:39.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.640+0000 7f3528da2640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f350806ae40 0x7f350806d300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3514000f60 tx=0x7f3514001540 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:39.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.642+0000 7f3519ffb640 1 -- 172.21.15.45:0/1326026920 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3520076660 con 0x7f3510003970 2023-11-28T20:34:39.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.874+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f352404e920 con 0x7f3510003970 2023-11-28T20:34:39.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.874+0000 7f3519ffb640 1 -- 172.21.15.45:0/1326026920 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f35200044e0 con 0x7f3510003970 2023-11-28T20:34:39.877 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T20:34:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f350806ae40 msgr2=0x7f350806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:39.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f350806ae40 0x7f350806d300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3514000f60 tx=0x7f3514001540 comp rx=0 tx=0).stop 2023-11-28T20:34:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 msgr2=0x7f35241f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f35241f6c70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f352002c2d0 tx=0x7f3520004730 comp rx=0 tx=0).stop 2023-11-28T20:34:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 shutdown_connections 2023-11-28T20:34:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f350806ae40 0x7f350806d300 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3524143f00 0x7f352409c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3524152140 0x7f3524153710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 --2- 172.21.15.45:0/1326026920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3510003970 0x7f35241f6c70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:39.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 >> 172.21.15.45:0/1326026920 conn(0x7f352406ab30 msgr2=0x7f35240b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:39.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 shutdown_connections 2023-11-28T20:34:39.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:39.877+0000 7f352a5a5640 1 -- 172.21.15.45:0/1326026920 wait complete. 2023-11-28T20:34:39.899 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T20:34:39.899 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:34:39.900 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:34:39.900 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T20:34:39.900 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:34:39.900 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T20:34:39.900 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T20:34:39.900 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:34:39.900 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:34:39.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:39.900 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:34:39.952 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:34:39.952 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:34:39.953 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:34:39.954 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:39.954 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:34:39.954 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:34:39.954 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42263sec preferred_lft 42263sec 2023-11-28T20:34:39.954 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:34:39.954 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:39.954 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:39.954 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:39.954 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:34:39.955 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T20:34:39.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:39.956 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:39.956 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:39.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add name ceph-brx type bridge 2023-11-28T20:34:39.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr flush dev ceph-brx 2023-11-28T20:34:39.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set ceph-brx up 2023-11-28T20:34:39.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T20:34:39.956 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:40.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:40.193 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:34:40.268 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:34:40.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:40.273 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:34:40.322 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:34:40.323 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:34:40.323 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:34:40.323 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:34:40.323 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:34:40.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:40.324 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:40.324 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:40.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:34:40.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:34:40.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:34:40.324 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:40.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:40.520 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:34:40.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:40.574 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2023-11-28T20:34:40.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:40.630 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:40.631 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:40.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:34:40.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-28T20:34:40.631 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:40.783 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-28T20:34:40.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:40.783 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:40.783 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:40.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-28T20:34:40.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T20:34:40.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-28T20:34:40.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-28T20:34:40.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-28T20:34:40.785 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:41.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:41.050 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:41.050 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:41.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:34:41.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T20:34:41.050 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:41.224 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:41.224 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T20:34:41.274 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:34:41.274 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:41.275 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:34:41.368 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:34:41.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:41.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:34:41.509 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:34:41.509 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:34:41.510 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:34:42.004 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:34:42.004 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:34:42.004 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:34:42.004 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:34:42.004 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:34:42.036 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:34:42.036 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:34:42.036 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:34:42.036 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:34:42.036 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:34:42.036 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:34:42.036 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:34:42.037 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:34:42.039 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T20:34:42.039 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:34:42.040 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:34:42.040 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:34:42.040 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:34:42.046 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:34:42.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:42.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:34:42.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:42.150 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:42.214 INFO:teuthology.orchestra.run.smithi045.stdout:client4559 2023-11-28T20:34:42.215 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:34:42.215 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:34:42.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:42.271 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:42.333 INFO:teuthology.orchestra.run.smithi045.stdout:client4559 2023-11-28T20:34:42.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:42.334 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:42.399 INFO:teuthology.orchestra.run.smithi045.stdout:client4559 2023-11-28T20:34:42.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:42.399 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ef8eac8d-57a5-4d94-ab39-848e2a7bfeef.client4559/status') 2023-11-28T20:34:42.497 INFO:teuthology.orchestra.run.smithi045.stdout:instance: client.4559 (3)172.21.15.45:0/3686624720 2023-11-28T20:34:42.497 INFO:teuthology.orchestra.run.smithi045.stdout:blocklisted: false 2023-11-28T20:34:42.497 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records in 2023-11-28T20:34:42.497 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records out 2023-11-28T20:34:42.497 INFO:teuthology.orchestra.run.smithi045.stderr:70 bytes copied, 0.000102309 s, 684 kB/s 2023-11-28T20:34:42.501 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2023-11-28T20:34:42.501 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T20:34:42.501 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-28T20:34:42.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:34:42.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.726+0000 7fbd4d41a640 1 Processor -- start 2023-11-28T20:34:42.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.727+0000 7fbd4d41a640 1 -- start start 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.728+0000 7fbd4d41a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbd48155780 0x7fbd480abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.728+0000 7fbd4d41a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbd480ce800 0x7fbd480b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.728+0000 7fbd4d41a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4815c3c0 con 0x7fbd480ae540 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.728+0000 7fbd4d41a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd48155b60 con 0x7fbd480ce800 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.728+0000 7fbd4d41a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4815ba90 con 0x7fbd48155780 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.728+0000 7fbd4cc19640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbd480ce800 0x7fbd480b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.728+0000 7fbd4cc19640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbd480ce800 0x7fbd480b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52190/0 (socket says 172.21.15.45:52190) 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.728+0000 7fbd4cc19640 1 -- 172.21.15.45:0/2991268379 learned_addr learned my addr 172.21.15.45:0/2991268379 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.729+0000 7fbd47fff640 1 --2- 172.21.15.45:0/2991268379 >> v2:172.21.15.190:3300/0 conn(0x7fbd48155780 0x7fbd480abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.729+0000 7fbd4cc19640 1 -- 172.21.15.45:0/2991268379 >> v2:172.21.15.190:3300/0 conn(0x7fbd48155780 msgr2=0x7fbd480abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.729+0000 7fbd4cc19640 1 --2- 172.21.15.45:0/2991268379 >> v2:172.21.15.190:3300/0 conn(0x7fbd48155780 0x7fbd480abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.729+0000 7fbd4cc19640 1 -- 172.21.15.45:0/2991268379 >> v1:172.21.15.45:6789/0 conn(0x7fbd480ae540 legacy=0x7fbd480ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.729+0000 7fbd4cc19640 1 -- 172.21.15.45:0/2991268379 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4815bd70 con 0x7fbd480ce800 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.729+0000 7fbd477fe640 1 -- 172.21.15.45:0/2991268379 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796285284 0 0) 0x7fbd4815c3c0 con 0x7fbd480ae540 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.730+0000 7fbd4cc19640 1 --2- 172.21.15.45:0/2991268379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 0x7fbd480b20a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbd4806e6d0 tx=0x7fbd4002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c9147f7b9d8461 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.730+0000 7fbd477fe640 1 -- 172.21.15.45:0/2991268379 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40030860 con 0x7fbd480ce800 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.730+0000 7fbd477fe640 1 -- 172.21.15.45:0/2991268379 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd400309c0 con 0x7fbd480ce800 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.730+0000 7fbd477fe640 1 -- 172.21.15.45:0/2991268379 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40030d50 con 0x7fbd480ce800 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.731+0000 7fbd4d41a640 1 -- 172.21.15.45:0/2991268379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 msgr2=0x7fbd480b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.731+0000 7fbd4d41a640 1 --2- 172.21.15.45:0/2991268379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 0x7fbd480b20a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbd4806e6d0 tx=0x7fbd4002b9a0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.731+0000 7fbd4d41a640 1 -- 172.21.15.45:0/2991268379 shutdown_connections 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.731+0000 7fbd4d41a640 1 --2- 172.21.15.45:0/2991268379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 0x7fbd480b20a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.731+0000 7fbd4d41a640 1 --2- 172.21.15.45:0/2991268379 >> v2:172.21.15.190:3300/0 conn(0x7fbd48155780 0x7fbd480abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.731+0000 7fbd4d41a640 1 -- 172.21.15.45:0/2991268379 >> 172.21.15.45:0/2991268379 conn(0x7fbd4806ab30 msgr2=0x7fbd480b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:42.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.731+0000 7fbd4d41a640 1 -- 172.21.15.45:0/2991268379 shutdown_connections 2023-11-28T20:34:42.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.731+0000 7fbd4d41a640 1 -- 172.21.15.45:0/2991268379 wait complete. 2023-11-28T20:34:42.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.733+0000 7fbd4d41a640 1 Processor -- start 2023-11-28T20:34:42.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.733+0000 7fbd4d41a640 1 -- start start 2023-11-28T20:34:42.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.734+0000 7fbd4d41a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd480ae540 0x7fbd4814b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.734+0000 7fbd4d41a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 0x7fbd481500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd4d41a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd48155780 0x7fbd4809b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd4d41a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4814b6b0 con 0x7fbd480ce800 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd4d41a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4809c960 con 0x7fbd48155780 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd4d41a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4809c3a0 con 0x7fbd480ae540 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd3f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 0x7fbd481500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd3f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 0x7fbd481500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52192/0 (socket says 172.21.15.45:52192) 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd3f7fe640 1 -- 172.21.15.45:0/1615933850 learned_addr learned my addr 172.21.15.45:0/1615933850 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd47fff640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd480ae540 0x7fbd4814b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd3f7fe640 1 -- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd480ae540 msgr2=0x7fbd4814b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd3f7fe640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd480ae540 0x7fbd4814b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd3f7fe640 1 -- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd48155780 msgr2=0x7fbd4809b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd3f7fe640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd48155780 0x7fbd4809b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd3f7fe640 1 -- 172.21.15.45:0/1615933850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd400094d0 con 0x7fbd480ce800 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.735+0000 7fbd3f7fe640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 0x7fbd481500c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000c4b0 tx=0x7fbd3000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.736+0000 7fbd457fa640 1 -- 172.21.15.45:0/1615933850 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd30005480 con 0x7fbd480ce800 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.736+0000 7fbd457fa640 1 -- 172.21.15.45:0/1615933850 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd300035f0 con 0x7fbd480ce800 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.736+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4809c680 con 0x7fbd480ce800 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.736+0000 7fbd457fa640 1 -- 172.21.15.45:0/1615933850 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd300039e0 con 0x7fbd480ce800 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.736+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd48099e50 con 0x7fbd480ce800 2023-11-28T20:34:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.737+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4809a070 con 0x7fbd480ce800 2023-11-28T20:34:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.738+0000 7fbd457fa640 1 -- 172.21.15.45:0/1615933850 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbd30003b40 con 0x7fbd480ce800 2023-11-28T20:34:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.739+0000 7fbd457fa640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbd2806ae40 0x7fbd2806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.740+0000 7fbd457fa640 1 -- 172.21.15.45:0/1615933850 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbd30088fd0 con 0x7fbd480ce800 2023-11-28T20:34:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.740+0000 7fbd47fff640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbd2806ae40 0x7fbd2806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:42.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.741+0000 7fbd47fff640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbd2806ae40 0x7fbd2806d300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009e90 tx=0x7fbd3800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:42.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.744+0000 7fbd457fa640 1 -- 172.21.15.45:0/1615933850 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd30053cb0 con 0x7fbd480ce800 2023-11-28T20:34:42.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.977+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbd480ad580 con 0x7fbd480ce800 2023-11-28T20:34:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.977+0000 7fbd457fa640 1 -- 172.21.15.45:0/1615933850 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fbd30017070 con 0x7fbd480ce800 2023-11-28T20:34:42.980 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T20:34:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbd2806ae40 msgr2=0x7fbd2806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbd2806ae40 0x7fbd2806d300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009e90 tx=0x7fbd3800b040 comp rx=0 tx=0).stop 2023-11-28T20:34:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 msgr2=0x7fbd481500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 0x7fbd481500c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000c4b0 tx=0x7fbd3000c970 comp rx=0 tx=0).stop 2023-11-28T20:34:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 shutdown_connections 2023-11-28T20:34:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbd2806ae40 0x7fbd2806d300 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd48155780 0x7fbd4809b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd480ce800 0x7fbd481500c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 --2- 172.21.15.45:0/1615933850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd480ae540 0x7fbd4814b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.980+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 >> 172.21.15.45:0/1615933850 conn(0x7fbd4806ab30 msgr2=0x7fbd48154950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.981+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 shutdown_connections 2023-11-28T20:34:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:42.981+0000 7fbd4d41a640 1 -- 172.21.15.45:0/1615933850 wait complete. 2023-11-28T20:34:43.003 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T20:34:43.003 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:34:43.003 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:34:43.003 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-11-28T20:34:43.003 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-28T20:34:43.003 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T20:34:43.004 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T20:34:43.004 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:34:43.004 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-11-28T20:34:43.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.004 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:34:43.059 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:34:43.060 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:34:43.060 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:34:43.060 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.060 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:34:43.060 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.060 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:43.061 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:43.061 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:34:43.061 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:43.061 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:43.061 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:34:43.061 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:34:43.061 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:43.062 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:34:43.062 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:34:43.062 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42259sec preferred_lft 42259sec 2023-11-28T20:34:43.062 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:34:43.062 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.063 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:43.063 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:43.063 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:34:43.063 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:34:43.063 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:43.063 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:34:43.063 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.064 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T20:34:43.064 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.064 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:34:43.064 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:34:43.064 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link tentative 2023-11-28T20:34:43.064 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.065 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:34:43.165 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:34:43.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.169 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:34:43.226 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:34:43.226 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:34:43.226 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:34:43.226 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:34:43.226 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:34:43.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.227 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:43.227 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:43.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:34:43.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:34:43.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:34:43.227 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:43.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.419 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:34:43.476 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T20:34:43.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.477 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2023-11-28T20:34:43.533 INFO:teuthology.orchestra.run.smithi045.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-11-28T20:34:43.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.535 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:43.536 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:43.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T20:34:43.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-11-28T20:34:43.536 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:43.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-28T20:34:43.790 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether a6:ab:96:40:75:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.791 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::a4ab:96ff:fe40:75c6/64 scope link 2023-11-28T20:34:43.792 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-28T20:34:43.890 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:34:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:34:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:34:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:34:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.892 INFO:teuthology.orchestra.run.smithi045.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:34:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether a6:ab:96:40:75:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T20:34:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T20:34:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::a4ab:96ff:fe40:75c6/64 scope link 2023-11-28T20:34:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:43.894 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-11-28T20:34:43.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:43.895 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:43.895 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:43.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-11-28T20:34:43.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-28T20:34:43.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-11-28T20:34:43.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-11-28T20:34:43.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-11-28T20:34:43.896 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:44.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:44.193 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:34:44.193 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:34:44.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.1 up 2023-11-28T20:34:44.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.1 master ceph-brx 2023-11-28T20:34:44.194 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:34:44.368 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:44.369 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-11-28T20:34:44.420 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-11-28T20:34:44.421 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:44.421 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-11-28T20:34:44.474 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:34:44.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:44.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-28T20:34:44.628 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-28T20:34:44.628 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-11-28T20:34:44.628 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:34:44.650 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-28T20:34:44.650 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-11-28T20:34:44.650 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:34:44.650 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:34:44.650 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:34:44.677 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-28T20:34:44.677 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-11-28T20:34:44.677 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:34:44.677 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:34:44.677 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:34:44.678 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:34:44.678 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:34:44.678 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:34:44.679 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2023-11-28T20:34:44.679 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:34:44.679 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:34:44.679 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:34:44.679 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:34:44.684 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:34:44.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:44.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-11-28T20:34:44.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:34:44.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:34:44.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:44.780 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:44.883 INFO:teuthology.orchestra.run.smithi045.stdout:client4863 2023-11-28T20:34:44.884 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:34:44.884 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:34:44.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:44.989 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:45.050 INFO:teuthology.orchestra.run.smithi045.stdout:client4863 2023-11-28T20:34:45.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:45.051 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:45.113 INFO:teuthology.orchestra.run.smithi045.stdout:client4863 2023-11-28T20:34:45.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:45.114 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ef8eac8d-57a5-4d94-ab39-848e2a7bfeef.client4863/status') 2023-11-28T20:34:45.214 INFO:teuthology.orchestra.run.smithi045.stdout:instance: client.4863 (3)192.168.0.2:0/3733782471 2023-11-28T20:34:45.214 INFO:teuthology.orchestra.run.smithi045.stdout:blocklisted: false 2023-11-28T20:34:45.214 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records in 2023-11-28T20:34:45.214 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records out 2023-11-28T20:34:45.214 INFO:teuthology.orchestra.run.smithi045.stderr:69 bytes copied, 0.000101531 s, 680 kB/s 2023-11-28T20:34:45.218 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-11-28T20:34:45.218 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T20:34:45.218 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-28T20:34:45.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:34:45.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.444+0000 7f3c1c8ce640 1 Processor -- start 2023-11-28T20:34:45.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.445+0000 7f3c1c8ce640 1 -- start start 2023-11-28T20:34:45.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.446+0000 7f3c1c8ce640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3c180ce800 0x7f3c18165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:45.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.446+0000 7f3c1c8ce640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1815c3c0 con 0x7f3c18155780 2023-11-28T20:34:45.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.446+0000 7f3c1c8ce640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c18155b60 con 0x7f3c180ae540 2023-11-28T20:34:45.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.447+0000 7f3c1c8ce640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1815ba90 con 0x7f3c180ce800 2023-11-28T20:34:45.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.447+0000 7f3c16d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3c18155780 0x7f3c180abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38564/0 (socket says 172.21.15.45:38564) 2023-11-28T20:34:45.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.447+0000 7f3c16d9d640 1 -- 172.21.15.45:0/2181921243 learned_addr learned my addr 172.21.15.45:0/2181921243 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:45.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.447+0000 7f3c1759e640 1 --2- 172.21.15.45:0/2181921243 >> v2:172.21.15.45:3300/0 conn(0x7f3c180ce800 0x7f3c18165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:45.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.447+0000 7f3c1759e640 1 -- 172.21.15.45:0/2181921243 >> v1:172.21.15.190:6789/0 conn(0x7f3c180ae540 legacy=0x7f3c180ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:45.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.447+0000 7f3c1759e640 1 -- 172.21.15.45:0/2181921243 >> v1:172.21.15.45:6789/0 conn(0x7f3c18155780 legacy=0x7f3c180abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:45.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.447+0000 7f3c1759e640 1 -- 172.21.15.45:0/2181921243 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1815bd70 con 0x7f3c180ce800 2023-11-28T20:34:45.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.447+0000 7f3c1659c640 1 -- 172.21.15.45:0/2181921243 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345136168 0 0) 0x7f3c1815c3c0 con 0x7f3c18155780 2023-11-28T20:34:45.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.449+0000 7f3c1759e640 1 --2- 172.21.15.45:0/2181921243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 0x7f3c18165e90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009d90 tx=0x7f3c1002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eabfc0ebfb4a594 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:45.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.449+0000 7f3c1659c640 1 -- 172.21.15.45:0/2181921243 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1002cd00 con 0x7f3c180ce800 2023-11-28T20:34:45.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.449+0000 7f3c1659c640 1 -- 172.21.15.45:0/2181921243 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1002ce60 con 0x7f3c180ce800 2023-11-28T20:34:45.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.449+0000 7f3c1659c640 1 -- 172.21.15.45:0/2181921243 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c10034660 con 0x7f3c180ce800 2023-11-28T20:34:45.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.449+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/2181921243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 msgr2=0x7f3c18165e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:45.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.449+0000 7f3c1c8ce640 1 --2- 172.21.15.45:0/2181921243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 0x7f3c18165e90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009d90 tx=0x7f3c1002bf50 comp rx=0 tx=0).stop 2023-11-28T20:34:45.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.450+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/2181921243 shutdown_connections 2023-11-28T20:34:45.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.450+0000 7f3c1c8ce640 1 --2- 172.21.15.45:0/2181921243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 0x7f3c18165e90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:45.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.450+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/2181921243 >> 172.21.15.45:0/2181921243 conn(0x7f3c1806ab30 msgr2=0x7f3c180b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:45.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.450+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/2181921243 shutdown_connections 2023-11-28T20:34:45.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.450+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/2181921243 wait complete. 2023-11-28T20:34:45.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.452+0000 7f3c1c8ce640 1 Processor -- start 2023-11-28T20:34:45.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.452+0000 7f3c1c8ce640 1 -- start start 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.453+0000 7f3c1c8ce640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c180ae540 0x7f3c18148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.453+0000 7f3c1c8ce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 0x7f3c1809ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.453+0000 7f3c1c8ce640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c18155780 0x7f3c180ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.453+0000 7f3c1c8ce640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1809c240 con 0x7f3c180ce800 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.453+0000 7f3c1c8ce640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1809b5c0 con 0x7f3c18155780 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.453+0000 7f3c1c8ce640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1809b8a0 con 0x7f3c180ae540 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.453+0000 7f3c0ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 0x7f3c1809ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.453+0000 7f3c0ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 0x7f3c1809ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49086/0 (socket says 172.21.15.45:49086) 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.453+0000 7f3c0ffff640 1 -- 172.21.15.45:0/1190535776 learned_addr learned my addr 172.21.15.45:0/1190535776 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c16d9d640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c180ae540 0x7f3c18148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:45.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c1759e640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c18155780 0x7f3c180ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:45.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c0ffff640 1 -- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c180ae540 msgr2=0x7f3c18148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:45.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c0ffff640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c180ae540 0x7f3c18148870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:45.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c0ffff640 1 -- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c18155780 msgr2=0x7f3c180ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:45.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c0ffff640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c18155780 0x7f3c180ad9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:45.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c0ffff640 1 -- 172.21.15.45:0/1190535776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c10009a70 con 0x7f3c180ce800 2023-11-28T20:34:45.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c1759e640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c18155780 0x7f3c180ad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:45.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c0ffff640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 0x7f3c1809ab70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000c450 tx=0x7f3c0000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:45.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c0f7fe640 1 -- 172.21.15.45:0/1190535776 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c000052d0 con 0x7f3c180ce800 2023-11-28T20:34:45.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.454+0000 7f3c0f7fe640 1 -- 172.21.15.45:0/1190535776 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c00005430 con 0x7f3c180ce800 2023-11-28T20:34:45.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.455+0000 7f3c0f7fe640 1 -- 172.21.15.45:0/1190535776 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c00012870 con 0x7f3c180ce800 2023-11-28T20:34:45.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.455+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1809bb80 con 0x7f3c180ce800 2023-11-28T20:34:45.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.455+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c18148db0 con 0x7f3c180ce800 2023-11-28T20:34:45.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.456+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1809c380 con 0x7f3c180ce800 2023-11-28T20:34:45.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.457+0000 7f3c0f7fe640 1 -- 172.21.15.45:0/1190535776 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3c000129d0 con 0x7f3c180ce800 2023-11-28T20:34:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.458+0000 7f3c0f7fe640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3bf806ae40 0x7f3bf806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.458+0000 7f3c0f7fe640 1 -- 172.21.15.45:0/1190535776 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3c000895b0 con 0x7f3c180ce800 2023-11-28T20:34:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.458+0000 7f3c16d9d640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3bf806ae40 0x7f3bf806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.459+0000 7f3c16d9d640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3bf806ae40 0x7f3bf806d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3c08000f60 tx=0x7f3c08007d20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.462+0000 7f3c0f7fe640 1 -- 172.21.15.45:0/1190535776 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c1809c380 con 0x7f3c180ce800 2023-11-28T20:34:45.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.700+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3c1809c380 con 0x7f3c180ce800 2023-11-28T20:34:45.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.701+0000 7f3c0f7fe640 1 -- 172.21.15.45:0/1190535776 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f3c1809c380 con 0x7f3c180ce800 2023-11-28T20:34:45.704 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T20:34:45.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3bf806ae40 msgr2=0x7f3bf806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:45.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3bf806ae40 0x7f3bf806d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3c08000f60 tx=0x7f3c08007d20 comp rx=0 tx=0).stop 2023-11-28T20:34:45.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 msgr2=0x7f3c1809ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:45.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 0x7f3c1809ab70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000c450 tx=0x7f3c0000c910 comp rx=0 tx=0).stop 2023-11-28T20:34:45.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 shutdown_connections 2023-11-28T20:34:45.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3bf806ae40 0x7f3bf806d300 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:45.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c18155780 0x7f3c180ad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:45.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c180ce800 0x7f3c1809ab70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 --2- 172.21.15.45:0/1190535776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c180ae540 0x7f3c18148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.704+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 >> 172.21.15.45:0/1190535776 conn(0x7f3c1806ab30 msgr2=0x7f3c1806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.705+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 shutdown_connections 2023-11-28T20:34:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:45.705+0000 7f3c1c8ce640 1 -- 172.21.15.45:0/1190535776 wait complete. 2023-11-28T20:34:45.725 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T20:34:45.725 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:34:45.725 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi190.front.sepia.ceph.com 2023-11-28T20:34:45.726 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-28T20:34:45.726 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-28T20:34:45.726 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T20:34:45.726 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T20:34:45.726 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:34:45.726 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-28T20:34:45.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:45.726 DEBUG:teuthology.orchestra.run.smithi190:> ip addr 2023-11-28T20:34:45.788 INFO:teuthology.orchestra.run.smithi190.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:34:45.788 INFO:teuthology.orchestra.run.smithi190.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:34:45.788 INFO:teuthology.orchestra.run.smithi190.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:34:45.788 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:45.788 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 ::1/128 scope host 2023-11-28T20:34:45.788 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:d9:36:d0 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout: altname enp7s0f0 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:d9:36:d1 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout: altname enp7s0f1 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:88:77:62 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:88:77:63 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:45.789 INFO:teuthology.orchestra.run.smithi190.stdout: inet 172.21.15.190/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T20:34:45.790 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft 42237sec preferred_lft 42237sec 2023-11-28T20:34:45.790 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 fe80::ec4:7aff:fe88:7763/64 scope link 2023-11-28T20:34:45.790 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:45.791 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T20:34:45.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:45.791 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec bash -c ' 2023-11-28T20:34:45.791 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2023-11-28T20:34:45.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link add name ceph-brx type bridge 2023-11-28T20:34:45.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip addr flush dev ceph-brx 2023-11-28T20:34:45.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set ceph-brx up 2023-11-28T20:34:45.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T20:34:45.792 DEBUG:teuthology.orchestra.run.smithi190:> ') 2023-11-28T20:34:46.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:46.238 DEBUG:teuthology.orchestra.run.smithi190:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:34:46.316 INFO:teuthology.orchestra.run.smithi190.stdout:1 2023-11-28T20:34:46.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:46.320 DEBUG:teuthology.orchestra.run.smithi190:> route 2023-11-28T20:34:46.371 INFO:teuthology.orchestra.run.smithi190.stdout:Kernel IP routing table 2023-11-28T20:34:46.372 INFO:teuthology.orchestra.run.smithi190.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:34:46.372 INFO:teuthology.orchestra.run.smithi190.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T20:34:46.372 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T20:34:46.372 INFO:teuthology.orchestra.run.smithi190.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:34:46.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:46.372 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec bash -c ' 2023-11-28T20:34:46.372 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2023-11-28T20:34:46.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T20:34:46.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T20:34:46.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T20:34:46.373 DEBUG:teuthology.orchestra.run.smithi190:> ') 2023-11-28T20:34:47.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:47.074 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2023-11-28T20:34:47.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:47.131 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list-id 2023-11-28T20:34:47.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:47.186 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec bash -c ' 2023-11-28T20:34:47.186 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2023-11-28T20:34:47.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T20:34:47.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-11-28T20:34:47.186 DEBUG:teuthology.orchestra.run.smithi190:> ') 2023-11-28T20:34:47.329 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-11-28T20:34:47.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:47.329 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec bash -c ' 2023-11-28T20:34:47.329 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2023-11-28T20:34:47.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-11-28T20:34:47.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T20:34:47.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-11-28T20:34:47.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-11-28T20:34:47.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-11-28T20:34:47.330 DEBUG:teuthology.orchestra.run.smithi190:> ') 2023-11-28T20:34:47.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:47.604 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec bash -c ' 2023-11-28T20:34:47.604 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2023-11-28T20:34:47.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set brx.0 up 2023-11-28T20:34:47.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T20:34:47.605 DEBUG:teuthology.orchestra.run.smithi190:> ') 2023-11-28T20:34:47.767 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:47.768 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-28T20:34:47.812 INFO:teuthology.orchestra.run.smithi190.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-28T20:34:47.813 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:47.813 DEBUG:teuthology.orchestra.run.smithi190:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-28T20:34:47.910 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:34:47.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:47.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-28T20:34:48.057 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-28T20:34:48.057 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-28T20:34:48.058 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2023-11-28T20:34:48.500 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-28T20:34:48.500 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-28T20:34:48.501 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2023-11-28T20:34:48.501 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:34:48.501 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:34:48.532 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-28T20:34:48.532 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-28T20:34:48.532 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2023-11-28T20:34:48.532 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:34:48.532 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:34:48.532 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:34:48.532 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:34:48.532 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:34:48.534 INFO:teuthology.orchestra.run.smithi190.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-11-28T20:34:48.534 INFO:teuthology.orchestra.run.smithi190.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:34:48.535 INFO:teuthology.orchestra.run.smithi190.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:34:48.535 INFO:teuthology.orchestra.run.smithi190.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:34:48.535 INFO:teuthology.orchestra.run.smithi190.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:34:48.539 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:34:48.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:48.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-28T20:34:48.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:48.624 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:48.680 INFO:teuthology.orchestra.run.smithi190.stdout:client4678 2023-11-28T20:34:48.680 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:34:48.680 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:34:48.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:48.732 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:48.790 INFO:teuthology.orchestra.run.smithi190.stdout:client4678 2023-11-28T20:34:48.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:48.790 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:48.847 INFO:teuthology.orchestra.run.smithi190.stdout:client4678 2023-11-28T20:34:48.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:48.847 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ef8eac8d-57a5-4d94-ab39-848e2a7bfeef.client4678/status') 2023-11-28T20:34:48.941 INFO:teuthology.orchestra.run.smithi190.stdout:instance: client.4678 (3)192.168.0.1:0/3550180562 2023-11-28T20:34:48.941 INFO:teuthology.orchestra.run.smithi190.stdout:blocklisted: false 2023-11-28T20:34:48.941 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2023-11-28T20:34:48.942 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2023-11-28T20:34:48.942 INFO:teuthology.orchestra.run.smithi190.stderr:69 bytes copied, 0.000108715 s, 635 kB/s 2023-11-28T20:34:48.945 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2023-11-28T20:34:48.945 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T20:34:48.945 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-28T20:34:48.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:34:49.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.181+0000 7f75c4c4b640 1 Processor -- start 2023-11-28T20:34:49.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.182+0000 7f75c4c4b640 1 -- start start 2023-11-28T20:34:49.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.183+0000 7f75c4c4b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c00d02d0 0x7f75c00d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:49.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.183+0000 7f75c4c4b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c00d2230 con 0x7f75c015d550 2023-11-28T20:34:49.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.183+0000 7f75c4c4b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c00d2510 con 0x7f75c00d1ab0 2023-11-28T20:34:49.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.183+0000 7f75c4c4b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c00d27f0 con 0x7f75c00d02d0 2023-11-28T20:34:49.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.184+0000 7f75bf7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f75c015d550 0x7f75c00cfbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38574/0 (socket says 172.21.15.45:38574) 2023-11-28T20:34:49.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.184+0000 7f75bf7fe640 1 -- 172.21.15.45:0/2042450539 learned_addr learned my addr 172.21.15.45:0/2042450539 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:49.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.184+0000 7f75beffd640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c00d02d0 0x7f75c00d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:49.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.184+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320673749 0 0) 0x7f75c00d2230 con 0x7f75c015d550 2023-11-28T20:34:49.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.184+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ac003520 con 0x7f75c015d550 2023-11-28T20:34:49.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2638401741 0 0) 0x7f75ac003520 con 0x7f75c015d550 2023-11-28T20:34:49.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c00d02d0 msgr2=0x7f75c00d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:49.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c00d02d0 0x7f75c00d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 >> v1:172.21.15.190:6789/0 conn(0x7f75c00d1ab0 legacy=0x7f75c00b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:49.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c00b6a30 con 0x7f75c015d550 2023-11-28T20:34:49.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f75b00019f0 con 0x7f75c015d550 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 >> v1:172.21.15.45:6789/0 conn(0x7f75c015d550 legacy=0x7f75c00cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c00d02d0 0x7f75ac0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75ac003f70 0x7f75ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75ac004d00 0x7f75ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c00d27f0 con 0x7f75ac004d00 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac00bac0 con 0x7f75ac003f70 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac00bc80 con 0x7f75c00d02d0 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75bffff640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75ac004d00 0x7f75ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75bf7fe640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c00d02d0 0x7f75ac0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.185+0000 7f75beffd640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75ac003f70 0x7f75ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.186+0000 7f75bffff640 1 -- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c00d02d0 msgr2=0x7f75ac0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.186+0000 7f75bffff640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c00d02d0 0x7f75ac0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.186+0000 7f75bffff640 1 -- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75ac003f70 msgr2=0x7f75ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:49.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.186+0000 7f75bffff640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75ac003f70 0x7f75ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.186+0000 7f75bffff640 1 -- 172.21.15.45:0/2042450539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac003520 con 0x7f75ac004d00 2023-11-28T20:34:49.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.186+0000 7f75beffd640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75ac003f70 0x7f75ac0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:49.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.187+0000 7f75bffff640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75ac004d00 0x7f75ac005180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f75b8001750 tx=0x7f75ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6474c1d948f3c303 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:49.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.187+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b80052d0 con 0x7f75ac004d00 2023-11-28T20:34:49.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.187+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b802e050 con 0x7f75ac004d00 2023-11-28T20:34:49.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.187+0000 7f75be7fc640 1 -- 172.21.15.45:0/2042450539 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8003830 con 0x7f75ac004d00 2023-11-28T20:34:49.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.188+0000 7f75c4c4b640 1 -- 172.21.15.45:0/2042450539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75ac004d00 msgr2=0x7f75ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:49.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.188+0000 7f75c4c4b640 1 --2- 172.21.15.45:0/2042450539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75ac004d00 0x7f75ac005180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f75b8001750 tx=0x7f75ac0057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.188+0000 7f75bffff640 1 -- 172.21.15.45:0/2042450539 reap_dead start 2023-11-28T20:34:49.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.188+0000 7f75c4c4b640 1 -- 172.21.15.45:0/2042450539 shutdown_connections 2023-11-28T20:34:49.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.188+0000 7f75c4c4b640 1 -- 172.21.15.45:0/2042450539 >> 172.21.15.45:0/2042450539 conn(0x7f75c006ac70 msgr2=0x7f75c00b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:49.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.188+0000 7f75c4c4b640 1 -- 172.21.15.45:0/2042450539 shutdown_connections 2023-11-28T20:34:49.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.188+0000 7f75c4c4b640 1 -- 172.21.15.45:0/2042450539 wait complete. 2023-11-28T20:34:49.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.190+0000 7f75c4c4b640 1 Processor -- start 2023-11-28T20:34:49.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.190+0000 7f75c4c4b640 1 -- start start 2023-11-28T20:34:49.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.191+0000 7f75c4c4b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75c00cb820 0x7f75c014b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:49.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.191+0000 7f75c4c4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c014b6a0 0x7f75c00cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:49.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.191+0000 7f75c4c4b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c014bb00 0x7f75c00cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:49.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.191+0000 7f75c4c4b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c00ccc80 con 0x7f75c014b6a0 2023-11-28T20:34:49.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.191+0000 7f75c4c4b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c01e6f80 con 0x7f75c00cb820 2023-11-28T20:34:49.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.191+0000 7f75c4c4b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c01e7260 con 0x7f75c014bb00 2023-11-28T20:34:49.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.191+0000 7f75beffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c014b6a0 0x7f75c00cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:49.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.191+0000 7f75beffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c014b6a0 0x7f75c00cbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49096/0 (socket says 172.21.15.45:49096) 2023-11-28T20:34:49.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.191+0000 7f75beffd640 1 -- 172.21.15.45:0/69709593 learned_addr learned my addr 172.21.15.45:0/69709593 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:49.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75bffff640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c014bb00 0x7f75c00cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:49.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75bf7fe640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75c00cb820 0x7f75c014b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:49.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75beffd640 1 -- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c014bb00 msgr2=0x7f75c00cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:49.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75beffd640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c014bb00 0x7f75c00cc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75beffd640 1 -- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75c00cb820 msgr2=0x7f75c014b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:49.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75beffd640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75c00cb820 0x7f75c014b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75beffd640 1 -- 172.21.15.45:0/69709593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c01e7540 con 0x7f75c014b6a0 2023-11-28T20:34:49.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75beffd640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c014b6a0 0x7f75c00cbc00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f75a8010890 tx=0x7f75a8010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:49.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75b7fff640 1 -- 172.21.15.45:0/69709593 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a800eb00 con 0x7f75c014b6a0 2023-11-28T20:34:49.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75b7fff640 1 -- 172.21.15.45:0/69709593 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a800ec60 con 0x7f75c014b6a0 2023-11-28T20:34:49.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75b7fff640 1 -- 172.21.15.45:0/69709593 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a801c400 con 0x7f75c014b6a0 2023-11-28T20:34:49.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c01e7820 con 0x7f75c014b6a0 2023-11-28T20:34:49.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.192+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c00cc680 con 0x7f75c014b6a0 2023-11-28T20:34:49.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.194+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c00c6b50 con 0x7f75c014b6a0 2023-11-28T20:34:49.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.194+0000 7f75b7fff640 1 -- 172.21.15.45:0/69709593 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f75a801c560 con 0x7f75c014b6a0 2023-11-28T20:34:49.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.195+0000 7f75b7fff640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f75a006ae40 0x7f75a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:49.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.196+0000 7f75b7fff640 1 -- 172.21.15.45:0/69709593 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f75a8089ee0 con 0x7f75c014b6a0 2023-11-28T20:34:49.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.196+0000 7f75bf7fe640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f75a006ae40 0x7f75a006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:49.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.197+0000 7f75bf7fe640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f75a006ae40 0x7f75a006d300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f75b0001b10 tx=0x7f75b0001250 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:49.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.200+0000 7f75b7fff640 1 -- 172.21.15.45:0/69709593 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75a8054c70 con 0x7f75c014b6a0 2023-11-28T20:34:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.436+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f75c01e6c30 con 0x7f75c014b6a0 2023-11-28T20:34:49.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.437+0000 7f75b7fff640 1 -- 172.21.15.45:0/69709593 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f75a8021070 con 0x7f75c014b6a0 2023-11-28T20:34:49.440 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T20:34:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f75a006ae40 msgr2=0x7f75a006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f75a006ae40 0x7f75a006d300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f75b0001b10 tx=0x7f75b0001250 comp rx=0 tx=0).stop 2023-11-28T20:34:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c014b6a0 msgr2=0x7f75c00cbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:49.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c014b6a0 0x7f75c00cbc00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f75a8010890 tx=0x7f75a8010d50 comp rx=0 tx=0).stop 2023-11-28T20:34:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 shutdown_connections 2023-11-28T20:34:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f75a006ae40 0x7f75a006d300 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75c014bb00 0x7f75c00cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c014b6a0 0x7f75c00cbc00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 --2- 172.21.15.45:0/69709593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75c00cb820 0x7f75c014b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 >> 172.21.15.45:0/69709593 conn(0x7f75c006ac70 msgr2=0x7f75c00c8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 shutdown_connections 2023-11-28T20:34:49.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:49.440+0000 7f75c4c4b640 1 -- 172.21.15.45:0/69709593 wait complete. 2023-11-28T20:34:49.463 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T20:34:49.463 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:34:49.463 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi190.front.sepia.ceph.com 2023-11-28T20:34:49.463 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-11-28T20:34:49.463 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-28T20:34:49.463 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T20:34:49.463 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T20:34:49.463 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:34:49.464 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-11-28T20:34:49.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:49.464 DEBUG:teuthology.orchestra.run.smithi190:> ip addr 2023-11-28T20:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 ::1/128 scope host 2023-11-28T20:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:d9:36:d0 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: altname enp7s0f0 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:d9:36:d1 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: altname enp7s0f1 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:88:77:62 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:88:77:63 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: inet 172.21.15.190/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T20:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft 42233sec preferred_lft 42233sec 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 fe80::ec4:7aff:fe88:7763/64 scope link 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 86:93:5f:03:2c:8f brd ff:ff:ff:ff:ff:ff 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 fe80::cb2:e4ff:fe74:4ba4/64 scope link 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 86:93:5f:03:2c:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 fe80::8493:5fff:fe03:2c8f/64 scope link 2023-11-28T20:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:49.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:49.529 DEBUG:teuthology.orchestra.run.smithi190:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:34:49.621 INFO:teuthology.orchestra.run.smithi190.stdout:1 2023-11-28T20:34:49.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:49.625 DEBUG:teuthology.orchestra.run.smithi190:> route 2023-11-28T20:34:49.675 INFO:teuthology.orchestra.run.smithi190.stdout:Kernel IP routing table 2023-11-28T20:34:49.676 INFO:teuthology.orchestra.run.smithi190.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:34:49.676 INFO:teuthology.orchestra.run.smithi190.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T20:34:49.676 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T20:34:49.676 INFO:teuthology.orchestra.run.smithi190.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:34:49.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:49.676 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec bash -c ' 2023-11-28T20:34:49.676 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2023-11-28T20:34:49.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T20:34:49.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T20:34:49.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T20:34:49.677 DEBUG:teuthology.orchestra.run.smithi190:> ') 2023-11-28T20:34:49.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:49.904 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2023-11-28T20:34:49.955 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-28T20:34:49.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:49.956 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list-id 2023-11-28T20:34:50.004 INFO:teuthology.orchestra.run.smithi190.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-11-28T20:34:50.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:50.005 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec bash -c ' 2023-11-28T20:34:50.005 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2023-11-28T20:34:50.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T20:34:50.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-11-28T20:34:50.006 DEBUG:teuthology.orchestra.run.smithi190:> ') 2023-11-28T20:34:50.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:50.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 ::1/128 scope host 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 52:b5:d2:ba:5b:da brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T20:34:50.291 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:50.292 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 fe80::50b5:d2ff:feba:5bda/64 scope link 2023-11-28T20:34:50.292 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:50.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:50.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-28T20:34:50.388 INFO:teuthology.orchestra.run.smithi190.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:34:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:34:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:34:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 ::1/128 scope host 2023-11-28T20:34:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:50.389 INFO:teuthology.orchestra.run.smithi190.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:34:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 52:b5:d2:ba:5b:da brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T20:34:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T20:34:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 fe80::50b5:d2ff:feba:5bda/64 scope link 2023-11-28T20:34:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:34:50.393 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-11-28T20:34:50.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:50.393 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec bash -c ' 2023-11-28T20:34:50.394 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2023-11-28T20:34:50.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-11-28T20:34:50.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-28T20:34:50.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-11-28T20:34:50.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-11-28T20:34:50.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-11-28T20:34:50.394 DEBUG:teuthology.orchestra.run.smithi190:> ') 2023-11-28T20:34:50.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:50.660 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec bash -c ' 2023-11-28T20:34:50.660 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2023-11-28T20:34:50.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set brx.1 up 2023-11-28T20:34:50.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set dev brx.1 master ceph-brx 2023-11-28T20:34:50.661 DEBUG:teuthology.orchestra.run.smithi190:> ') 2023-11-28T20:34:50.824 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:50.824 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-11-28T20:34:50.875 INFO:teuthology.orchestra.run.smithi190.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-11-28T20:34:50.875 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:34:50.875 DEBUG:teuthology.orchestra.run.smithi190:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-11-28T20:34:50.927 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:34:50.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:50.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-28T20:34:51.066 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-28T20:34:51.066 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-11-28T20:34:51.066 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2023-11-28T20:34:51.086 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-28T20:34:51.087 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-11-28T20:34:51.087 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2023-11-28T20:34:51.087 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:34:51.087 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:34:51.112 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-28T20:34:51.113 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-11-28T20:34:51.113 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2023-11-28T20:34:51.113 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:34:51.113 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:34:51.113 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:34:51.113 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:34:51.113 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:34:51.114 INFO:teuthology.orchestra.run.smithi190.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2023-11-28T20:34:51.114 INFO:teuthology.orchestra.run.smithi190.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:34:51.114 INFO:teuthology.orchestra.run.smithi190.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:34:51.114 INFO:teuthology.orchestra.run.smithi190.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:34:51.115 INFO:teuthology.orchestra.run.smithi190.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:34:51.118 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:34:51.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:51.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-11-28T20:34:51.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:51.198 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:51.249 INFO:teuthology.orchestra.run.smithi190.stdout:client4568 2023-11-28T20:34:51.249 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:34:51.249 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:34:51.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:51.292 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:51.393 INFO:teuthology.orchestra.run.smithi190.stdout:client4568 2023-11-28T20:34:51.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:51.393 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:34:51.443 INFO:teuthology.orchestra.run.smithi190.stdout:client4568 2023-11-28T20:34:51.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:34:51.444 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ef8eac8d-57a5-4d94-ab39-848e2a7bfeef.client4568/status') 2023-11-28T20:34:51.527 INFO:teuthology.orchestra.run.smithi190.stdout:instance: client.4568 (3)192.168.0.2:0/3700927300 2023-11-28T20:34:51.527 INFO:teuthology.orchestra.run.smithi190.stdout:blocklisted: false 2023-11-28T20:34:51.528 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2023-11-28T20:34:51.528 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2023-11-28T20:34:51.528 INFO:teuthology.orchestra.run.smithi190.stderr:69 bytes copied, 6.0345e-05 s, 1.1 MB/s 2023-11-28T20:34:51.530 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-11-28T20:34:51.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:51.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.780+0000 7fb3dbfff640 1 Processor -- start 2023-11-28T20:34:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.780+0000 7fb3dbfff640 1 -- start start 2023-11-28T20:34:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.783+0000 7fb3dbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb3dc0b24a0 0x7fb3dc0703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.783+0000 7fb3dbfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 0x7fb3dc070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.783+0000 7fb3dbfff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc0b4310 con 0x7fb3dc072200 2023-11-28T20:34:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.783+0000 7fb3dbfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3dc0725e0 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.783+0000 7fb3dbfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc06d8d0 con 0x7fb3dc146610 2023-11-28T20:34:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.783+0000 7fb3d3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb3dc0b24a0 0x7fb3dc0703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.783+0000 7fb3d3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb3dc0b24a0 0x7fb3dc0703d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49098/0 (socket says 172.21.15.45:49098) 2023-11-28T20:34:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.783+0000 7fb3d3fff640 1 -- 172.21.15.45:0/1693017196 learned_addr learned my addr 172.21.15.45:0/1693017196 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.783+0000 7fb3db7fe640 1 --2- 172.21.15.45:0/1693017196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 0x7fb3dc070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.784+0000 7fb3d3fff640 1 -- 172.21.15.45:0/1693017196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 msgr2=0x7fb3dc070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.784+0000 7fb3d3fff640 1 --2- 172.21.15.45:0/1693017196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 0x7fb3dc070910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.784+0000 7fb3d3fff640 1 -- 172.21.15.45:0/1693017196 >> v1:172.21.15.190:6789/0 conn(0x7fb3dc072200 legacy=0x7fb3dc0b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.784+0000 7fb3d3fff640 1 -- 172.21.15.45:0/1693017196 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc06ca20 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.785+0000 7fb3d3fff640 1 --2- 172.21.15.45:0/1693017196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 0x7fb3dc0703d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009870 tx=0x7fb3c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ab56faff0bf57e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.785+0000 7fb3da7fc640 1 -- 172.21.15.45:0/1693017196 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4031640 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.785+0000 7fb3da7fc640 1 -- 172.21.15.45:0/1693017196 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c40317a0 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.785+0000 7fb3da7fc640 1 -- 172.21.15.45:0/1693017196 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4031b30 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.786+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1693017196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 msgr2=0x7fb3dc0703d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.786+0000 7fb3dbfff640 1 --2- 172.21.15.45:0/1693017196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 0x7fb3dc0703d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009870 tx=0x7fb3c402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:34:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.786+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1693017196 shutdown_connections 2023-11-28T20:34:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.786+0000 7fb3dbfff640 1 --2- 172.21.15.45:0/1693017196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 0x7fb3dc070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.786+0000 7fb3dbfff640 1 --2- 172.21.15.45:0/1693017196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 0x7fb3dc0703d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.786+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1693017196 >> 172.21.15.45:0/1693017196 conn(0x7fb3dc06ab30 msgr2=0x7fb3dc0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.786+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1693017196 shutdown_connections 2023-11-28T20:34:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.786+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1693017196 wait complete. 2023-11-28T20:34:51.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.788+0000 7fb3dbfff640 1 Processor -- start 2023-11-28T20:34:51.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.788+0000 7fb3dbfff640 1 -- start start 2023-11-28T20:34:51.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.789+0000 7fb3dbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3dc072200 0x7fb3dc15c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.789+0000 7fb3dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 0x7fb3dc09c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.789+0000 7fb3dbfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 0x7fb3dc09ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.789+0000 7fb3dbfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc09c380 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.789+0000 7fb3dbfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc09b700 con 0x7fb3dc072200 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.789+0000 7fb3dbfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc09b9e0 con 0x7fb3dc146610 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.789+0000 7fb3d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 0x7fb3dc09c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.789+0000 7fb3d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 0x7fb3dc09c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49114/0 (socket says 172.21.15.45:49114) 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.789+0000 7fb3d3fff640 1 -- 172.21.15.45:0/1140811794 learned_addr learned my addr 172.21.15.45:0/1140811794 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.790+0000 7fb3db7fe640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 0x7fb3dc09ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.790+0000 7fb3daffd640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3dc072200 0x7fb3dc15c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.790+0000 7fb3d3fff640 1 -- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 msgr2=0x7fb3dc09ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.790+0000 7fb3d3fff640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 0x7fb3dc09ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.790+0000 7fb3d3fff640 1 -- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3dc072200 msgr2=0x7fb3dc15c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.790+0000 7fb3d3fff640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3dc072200 0x7fb3dc15c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.790+0000 7fb3d3fff640 1 -- 172.21.15.45:0/1140811794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c40094d0 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.790+0000 7fb3d3fff640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 0x7fb3dc09c7a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009870 tx=0x7fb3c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.791+0000 7fb3d8ff9640 1 -- 172.21.15.45:0/1140811794 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c403ac60 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.791+0000 7fb3d8ff9640 1 -- 172.21.15.45:0/1140811794 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c403adc0 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.791+0000 7fb3d8ff9640 1 -- 172.21.15.45:0/1140811794 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4044630 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.791+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc09bcc0 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.791+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc15cae0 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.792+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc09c4c0 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.793+0000 7fb3d8ff9640 1 -- 172.21.15.45:0/1140811794 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb3c402f050 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.794+0000 7fb3d8ff9640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb3bc06ae40 0x7fb3bc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.794+0000 7fb3d8ff9640 1 -- 172.21.15.45:0/1140811794 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb3c40ab640 con 0x7fb3dc0b24a0 2023-11-28T20:34:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.795+0000 7fb3daffd640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb3bc06ae40 0x7fb3bc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.796+0000 7fb3daffd640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb3bc06ae40 0x7fb3bc06d300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc006fd0 tx=0x7fb3cc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.799+0000 7fb3d8ff9640 1 -- 172.21.15.45:0/1140811794 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3dc09c4c0 con 0x7fb3dc0b24a0 2023-11-28T20:34:52.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:51.999+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3dc0b4310 con 0x7fb3dc0b24a0 2023-11-28T20:34:52.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.002+0000 7fb3d8ff9640 1 -- 172.21.15.45:0/1140811794 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7fb3dc0b4310 con 0x7fb3dc0b24a0 2023-11-28T20:34:52.004 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:52.005 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:52.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb3bc06ae40 msgr2=0x7fb3bc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb3bc06ae40 0x7fb3bc06d300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc006fd0 tx=0x7fb3cc008040 comp rx=0 tx=0).stop 2023-11-28T20:34:52.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 msgr2=0x7fb3dc09c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 0x7fb3dc09c7a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009870 tx=0x7fb3c40046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 shutdown_connections 2023-11-28T20:34:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb3bc06ae40 0x7fb3bc06d300 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3dc146610 0x7fb3dc09ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3dc0b24a0 0x7fb3dc09c7a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 --2- 172.21.15.45:0/1140811794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3dc072200 0x7fb3dc15c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 >> 172.21.15.45:0/1140811794 conn(0x7fb3dc06ab30 msgr2=0x7fb3dc0abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 shutdown_connections 2023-11-28T20:34:52.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.005+0000 7fb3dbfff640 1 -- 172.21.15.45:0/1140811794 wait complete. 2023-11-28T20:34:52.028 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:34:52.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:34:52.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.256+0000 7fe2dbfff640 1 Processor -- start 2023-11-28T20:34:52.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.256+0000 7fe2dbfff640 1 -- start start 2023-11-28T20:34:52.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.258+0000 7fe2dbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2dc0b3040 0x7fe2dc06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.258+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc15b270 0x7fe2dc15f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.258+0000 7fe2dbfff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc15faf0 con 0x7fe2dc0b3420 2023-11-28T20:34:52.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.258+0000 7fe2dbfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc0ceaa0 con 0x7fe2dc0b3040 2023-11-28T20:34:52.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.258+0000 7fe2dbfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc06d410 con 0x7fe2dc15b270 2023-11-28T20:34:52.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.259+0000 7fe2daffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2dc0b3040 0x7fe2dc06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:52.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.259+0000 7fe2daffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2dc0b3040 0x7fe2dc06cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49124/0 (socket says 172.21.15.45:49124) 2023-11-28T20:34:52.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.259+0000 7fe2daffd640 1 -- 172.21.15.45:0/1599608017 learned_addr learned my addr 172.21.15.45:0/1599608017 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:52.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.259+0000 7fe2da7fc640 1 --2- 172.21.15.45:0/1599608017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc15b270 0x7fe2dc15f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:52.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.259+0000 7fe2daffd640 1 -- 172.21.15.45:0/1599608017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc15b270 msgr2=0x7fe2dc15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:52.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.259+0000 7fe2daffd640 1 --2- 172.21.15.45:0/1599608017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc15b270 0x7fe2dc15f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.259+0000 7fe2daffd640 1 -- 172.21.15.45:0/1599608017 >> v1:172.21.15.190:6789/0 conn(0x7fe2dc0b3420 legacy=0x7fe2dc0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.259+0000 7fe2daffd640 1 -- 172.21.15.45:0/1599608017 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc06d6f0 con 0x7fe2dc0b3040 2023-11-28T20:34:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.260+0000 7fe2daffd640 1 --2- 172.21.15.45:0/1599608017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3040 0x7fe2dc06cda0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009870 tx=0x7fe2cc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf845666b47e27f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.261+0000 7fe2d9ffb640 1 -- 172.21.15.45:0/1599608017 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0317c0 con 0x7fe2dc0b3040 2023-11-28T20:34:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.261+0000 7fe2d9ffb640 1 -- 172.21.15.45:0/1599608017 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2cc031920 con 0x7fe2dc0b3040 2023-11-28T20:34:52.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.261+0000 7fe2d9ffb640 1 -- 172.21.15.45:0/1599608017 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc031cb0 con 0x7fe2dc0b3040 2023-11-28T20:34:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.261+0000 7fe2dbfff640 1 -- 172.21.15.45:0/1599608017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3040 msgr2=0x7fe2dc06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.261+0000 7fe2dbfff640 1 --2- 172.21.15.45:0/1599608017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3040 0x7fe2dc06cda0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009870 tx=0x7fe2cc02b860 comp rx=0 tx=0).stop 2023-11-28T20:34:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.262+0000 7fe2dbfff640 1 -- 172.21.15.45:0/1599608017 shutdown_connections 2023-11-28T20:34:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.262+0000 7fe2dbfff640 1 --2- 172.21.15.45:0/1599608017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc15b270 0x7fe2dc15f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.262+0000 7fe2dbfff640 1 --2- 172.21.15.45:0/1599608017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3040 0x7fe2dc06cda0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.262+0000 7fe2dbfff640 1 -- 172.21.15.45:0/1599608017 >> 172.21.15.45:0/1599608017 conn(0x7fe2dc0b1050 msgr2=0x7fe2dc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.262+0000 7fe2dbfff640 1 -- 172.21.15.45:0/1599608017 shutdown_connections 2023-11-28T20:34:52.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.262+0000 7fe2dbfff640 1 -- 172.21.15.45:0/1599608017 wait complete. 2023-11-28T20:34:52.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.264+0000 7fe2dbfff640 1 Processor -- start 2023-11-28T20:34:52.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.264+0000 7fe2dbfff640 1 -- start start 2023-11-28T20:34:52.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc0b3040 0x7fe2dc14c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3420 0x7fe2dc14d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2dc15b270 0x7fe2dc151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2dbfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc14eb20 con 0x7fe2dc0b3420 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2dbfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc14cd70 con 0x7fe2dc15b270 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2dbfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0af800 con 0x7fe2dc0b3040 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2da7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3420 0x7fe2dc14d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2da7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3420 0x7fe2dc14d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49128/0 (socket says 172.21.15.45:49128) 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2da7fc640 1 -- 172.21.15.45:0/3301565532 learned_addr learned my addr 172.21.15.45:0/3301565532 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2db7fe640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2dc15b270 0x7fe2dc151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2daffd640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc0b3040 0x7fe2dc14c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:52.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2da7fc640 1 -- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc0b3040 msgr2=0x7fe2dc14c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2da7fc640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc0b3040 0x7fe2dc14c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2da7fc640 1 -- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2dc15b270 msgr2=0x7fe2dc151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2da7fc640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2dc15b270 0x7fe2dc151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.265+0000 7fe2da7fc640 1 -- 172.21.15.45:0/3301565532 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc0094d0 con 0x7fe2dc0b3420 2023-11-28T20:34:52.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.266+0000 7fe2da7fc640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3420 0x7fe2dc14d5f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000d800 tx=0x7fe2d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.266+0000 7fe2c37fe640 1 -- 172.21.15.45:0/3301565532 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d000b6c0 con 0x7fe2dc0b3420 2023-11-28T20:34:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.266+0000 7fe2c37fe640 1 -- 172.21.15.45:0/3301565532 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d000b820 con 0x7fe2dc0b3420 2023-11-28T20:34:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.266+0000 7fe2c37fe640 1 -- 172.21.15.45:0/3301565532 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d000bbd0 con 0x7fe2dc0b3420 2023-11-28T20:34:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.266+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc14d160 con 0x7fe2dc0b3420 2023-11-28T20:34:52.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.266+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc14e280 con 0x7fe2dc0b3420 2023-11-28T20:34:52.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.267+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc14cf30 con 0x7fe2dc0b3420 2023-11-28T20:34:52.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.269+0000 7fe2c37fe640 1 -- 172.21.15.45:0/3301565532 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe2d000bd30 con 0x7fe2dc0b3420 2023-11-28T20:34:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.270+0000 7fe2c37fe640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe2c806ae70 0x7fe2c806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.270+0000 7fe2c37fe640 1 -- 172.21.15.45:0/3301565532 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe2d0089ff0 con 0x7fe2dc0b3420 2023-11-28T20:34:52.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.270+0000 7fe2daffd640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe2c806ae70 0x7fe2c806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:52.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.271+0000 7fe2daffd640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe2c806ae70 0x7fe2c806d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc036000 tx=0x7fe2cc03d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:52.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.274+0000 7fe2c37fe640 1 -- 172.21.15.45:0/3301565532 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2dc14cf30 con 0x7fe2dc0b3420 2023-11-28T20:34:52.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.467+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2dc14cf30 con 0x7fe2dc0b3420 2023-11-28T20:34:52.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.468+0000 7fe2c37fe640 1 -- 172.21.15.45:0/3301565532 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe2dc14cf30 con 0x7fe2dc0b3420 2023-11-28T20:34:52.471 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe2c806ae70 msgr2=0x7fe2c806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe2c806ae70 0x7fe2c806d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc036000 tx=0x7fe2cc03d040 comp rx=0 tx=0).stop 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3420 msgr2=0x7fe2dc14d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3420 0x7fe2dc14d5f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000d800 tx=0x7fe2d000dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 shutdown_connections 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe2c806ae70 0x7fe2c806d330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2dc15b270 0x7fe2dc151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2dc0b3420 0x7fe2dc14d5f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 --2- 172.21.15.45:0/3301565532 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2dc0b3040 0x7fe2dc14c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 >> 172.21.15.45:0/3301565532 conn(0x7fe2dc0b1050 msgr2=0x7fe2dc0717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 shutdown_connections 2023-11-28T20:34:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.471+0000 7fe2dbfff640 1 -- 172.21.15.45:0/3301565532 wait complete. 2023-11-28T20:34:52.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:34:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.713+0000 7f34c819e640 1 Processor -- start 2023-11-28T20:34:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.714+0000 7f34c819e640 1 -- start start 2023-11-28T20:34:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.715+0000 7f34c819e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34c015e840 0x7f34c006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.715+0000 7f34c819e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c015ecd0 con 0x7f34c006ac10 2023-11-28T20:34:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.715+0000 7f34c819e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c015f290 con 0x7f34c015f720 2023-11-28T20:34:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.715+0000 7f34c819e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c015fb60 con 0x7f34c015e840 2023-11-28T20:34:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.715+0000 7f34c5f3a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34c015e840 0x7f34c006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.716+0000 7f34c673b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f34c006ac10 0x7f34c00cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38584/0 (socket says 172.21.15.45:38584) 2023-11-28T20:34:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.716+0000 7f34c673b640 1 -- 172.21.15.45:0/385409809 learned_addr learned my addr 172.21.15.45:0/385409809 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.716+0000 7f34c5f3a640 1 -- 172.21.15.45:0/385409809 >> v1:172.21.15.190:6789/0 conn(0x7f34c015f720 legacy=0x7f34c006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:52.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.716+0000 7f34c5f3a640 1 -- 172.21.15.45:0/385409809 >> v1:172.21.15.45:6789/0 conn(0x7f34c006ac10 legacy=0x7f34c00cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.716+0000 7f34c5f3a640 1 -- 172.21.15.45:0/385409809 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c0072020 con 0x7f34c015e840 2023-11-28T20:34:52.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.717+0000 7f34c5f3a640 1 --2- 172.21.15.45:0/385409809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 0x7f34c006c350 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0097a0 tx=0x7f34bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465e2c8e0806d9d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:52.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.717+0000 7f34c4f38640 1 -- 172.21.15.45:0/385409809 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc031830 con 0x7f34c015e840 2023-11-28T20:34:52.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.717+0000 7f34c4f38640 1 -- 172.21.15.45:0/385409809 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc031990 con 0x7f34c015e840 2023-11-28T20:34:52.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.717+0000 7f34c4f38640 1 -- 172.21.15.45:0/385409809 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc031d20 con 0x7f34c015e840 2023-11-28T20:34:52.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.718+0000 7f34c819e640 1 -- 172.21.15.45:0/385409809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 msgr2=0x7f34c006c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.718+0000 7f34c819e640 1 --2- 172.21.15.45:0/385409809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 0x7f34c006c350 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0097a0 tx=0x7f34bc02b8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.718+0000 7f34c819e640 1 -- 172.21.15.45:0/385409809 shutdown_connections 2023-11-28T20:34:52.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.718+0000 7f34c819e640 1 --2- 172.21.15.45:0/385409809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 0x7f34c006c350 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.718+0000 7f34c819e640 1 -- 172.21.15.45:0/385409809 >> 172.21.15.45:0/385409809 conn(0x7f34c00ced60 msgr2=0x7f34c00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:52.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.718+0000 7f34c819e640 1 -- 172.21.15.45:0/385409809 shutdown_connections 2023-11-28T20:34:52.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.718+0000 7f34c819e640 1 -- 172.21.15.45:0/385409809 wait complete. 2023-11-28T20:34:52.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.720+0000 7f34c819e640 1 Processor -- start 2023-11-28T20:34:52.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.721+0000 7f34c819e640 1 -- start start 2023-11-28T20:34:52.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.722+0000 7f34c819e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34c006ac10 0x7f34c015a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.722+0000 7f34c819e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 0x7f34c00b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.722+0000 7f34c819e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c015f720 0x7f34c00b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.722+0000 7f34c819e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c015a650 con 0x7f34c015e840 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.722+0000 7f34c819e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c00b4410 con 0x7f34c015f720 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.722+0000 7f34c819e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c00cacb0 con 0x7f34c006ac10 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.722+0000 7f34c5739640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 0x7f34c00b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.722+0000 7f34c5739640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 0x7f34c00b2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49146/0 (socket says 172.21.15.45:49146) 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.722+0000 7f34c5739640 1 -- 172.21.15.45:0/2471680440 learned_addr learned my addr 172.21.15.45:0/2471680440 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34c5739640 1 -- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34c006ac10 msgr2=0x7f34c015a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34c5739640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34c006ac10 0x7f34c015a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34c5739640 1 -- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c015f720 msgr2=0x7f34c00b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34c5739640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c015f720 0x7f34c00b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34c5739640 1 -- 172.21.15.45:0/2471680440 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc009400 con 0x7f34c015e840 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34c5739640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 0x7f34c00b2f90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f34b000b420 tx=0x7f34b000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34b67fc640 1 -- 172.21.15.45:0/2471680440 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b0011830 con 0x7f34c015e840 2023-11-28T20:34:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34b67fc640 1 -- 172.21.15.45:0/2471680440 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b0011990 con 0x7f34c015e840 2023-11-28T20:34:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34b67fc640 1 -- 172.21.15.45:0/2471680440 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b0011d40 con 0x7f34c015e840 2023-11-28T20:34:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c00c9dc0 con 0x7f34c015e840 2023-11-28T20:34:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.723+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c00ca380 con 0x7f34c015e840 2023-11-28T20:34:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.724+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c00733d0 con 0x7f34c015e840 2023-11-28T20:34:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.725+0000 7f34b67fc640 1 -- 172.21.15.45:0/2471680440 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f34b001b460 con 0x7f34c015e840 2023-11-28T20:34:52.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.726+0000 7f34b67fc640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f34a406ae90 0x7f34a406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:52.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.726+0000 7f34b67fc640 1 -- 172.21.15.45:0/2471680440 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f34b0089520 con 0x7f34c015e840 2023-11-28T20:34:52.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.727+0000 7f34c5f3a640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f34a406ae90 0x7f34a406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:52.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.728+0000 7f34c5f3a640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f34a406ae90 0x7f34a406d350 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f34bc036000 tx=0x7f34bc030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.730+0000 7f34b67fc640 1 -- 172.21.15.45:0/2471680440 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34b00541f0 con 0x7f34c015e840 2023-11-28T20:34:52.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.923+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34c00cae20 con 0x7f34c015e840 2023-11-28T20:34:52.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.924+0000 7f34b67fc640 1 -- 172.21.15.45:0/2471680440 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f34b0057810 con 0x7f34c015e840 2023-11-28T20:34:52.926 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f34a406ae90 msgr2=0x7f34a406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f34a406ae90 0x7f34a406d350 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f34bc036000 tx=0x7f34bc030000 comp rx=0 tx=0).stop 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 msgr2=0x7f34c00b2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 0x7f34c00b2f90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f34b000b420 tx=0x7f34b000b8e0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 shutdown_connections 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f34a406ae90 0x7f34a406d350 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c015f720 0x7f34c00b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34c015e840 0x7f34c00b2f90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 --2- 172.21.15.45:0/2471680440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34c006ac10 0x7f34c015a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.926+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 >> 172.21.15.45:0/2471680440 conn(0x7f34c00ced60 msgr2=0x7f34c00afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.927+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 shutdown_connections 2023-11-28T20:34:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:52.927+0000 7f34c819e640 1 -- 172.21.15.45:0/2471680440 wait complete. 2023-11-28T20:34:52.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:34:53.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.177+0000 7f99cb59e640 1 Processor -- start 2023-11-28T20:34:53.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.177+0000 7f99cb59e640 1 -- start start 2023-11-28T20:34:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.179+0000 7f99cb59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f99cc0b39d0 0x7f99cc0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.179+0000 7f99cb59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99cc0724d0 con 0x7f99cc0b3db0 2023-11-28T20:34:53.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.179+0000 7f99cb59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99cc06c670 con 0x7f99cc0cb630 2023-11-28T20:34:53.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.179+0000 7f99cb59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99cc071d20 con 0x7f99cc0b39d0 2023-11-28T20:34:53.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.179+0000 7f99c9d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f99cc0b3db0 0x7f99cc06bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38592/0 (socket says 172.21.15.45:38592) 2023-11-28T20:34:53.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99c9d9b640 1 -- 172.21.15.45:0/2758969462 learned_addr learned my addr 172.21.15.45:0/2758969462 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:53.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99ca59c640 1 --2- 172.21.15.45:0/2758969462 >> v2:172.21.15.190:3300/0 conn(0x7f99cc0b39d0 0x7f99cc0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134672536 0 0) 0x7f99cc0724d0 con 0x7f99cc0b3db0 2023-11-28T20:34:53.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99b4003520 con 0x7f99cc0b3db0 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956425397 0 0) 0x7f99b4003520 con 0x7f99cc0b3db0 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 >> v2:172.21.15.190:3300/0 conn(0x7f99cc0b39d0 msgr2=0x7f99cc0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99c959a640 1 --2- 172.21.15.45:0/2758969462 >> v2:172.21.15.190:3300/0 conn(0x7f99cc0b39d0 0x7f99cc0717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 >> v1:172.21.15.190:6789/0 conn(0x7f99cc0cb630 legacy=0x7f99cc070b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc0b4240 con 0x7f99cc0b3db0 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.180+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f99c0001670 con 0x7f99cc0b3db0 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 >> v1:172.21.15.45:6789/0 conn(0x7f99cc0b3db0 legacy=0x7f99cc06bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99c959a640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99c959a640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b40047c0 0x7f99b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99c959a640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99b4005180 0x7f99b400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b400c0f0 con 0x7f99b4003970 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b400c260 con 0x7f99b40047c0 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b400c420 con 0x7f99b4005180 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99ca59c640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99c9d9b640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b40047c0 0x7f99b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99ca59c640 1 -- 172.21.15.45:0/2758969462 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99b4005180 msgr2=0x7f99b400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99ca59c640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99b4005180 0x7f99b400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99ca59c640 1 -- 172.21.15.45:0/2758969462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b40047c0 msgr2=0x7f99b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99ca59c640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b40047c0 0x7f99b4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99ca59c640 1 -- 172.21.15.45:0/2758969462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b4003520 con 0x7f99b4003970 2023-11-28T20:34:53.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99cad9d640 1 -- 172.21.15.45:0/2758969462 reap_dead start 2023-11-28T20:34:53.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.181+0000 7f99c9d9b640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b40047c0 0x7f99b4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:53.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.182+0000 7f99ca59c640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99b4003d90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009ac0 tx=0x7f99b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1c5a52426ccf6f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:53.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc02c6b0 con 0x7f99b4003970 2023-11-28T20:34:53.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99bc02c810 con 0x7f99b4003970 2023-11-28T20:34:53.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99c959a640 1 -- 172.21.15.45:0/2758969462 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc02cba0 con 0x7f99b4003970 2023-11-28T20:34:53.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99cb59e640 1 -- 172.21.15.45:0/2758969462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 msgr2=0x7f99b4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99cb59e640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99b4003d90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009ac0 tx=0x7f99b40057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99cb59e640 1 -- 172.21.15.45:0/2758969462 shutdown_connections 2023-11-28T20:34:53.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99cb59e640 1 --2- 172.21.15.45:0/2758969462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99b4003d90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99cb59e640 1 -- 172.21.15.45:0/2758969462 >> 172.21.15.45:0/2758969462 conn(0x7f99cc0b1050 msgr2=0x7f99cc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:53.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99cb59e640 1 -- 172.21.15.45:0/2758969462 shutdown_connections 2023-11-28T20:34:53.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.183+0000 7f99cb59e640 1 -- 172.21.15.45:0/2758969462 wait complete. 2023-11-28T20:34:53.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.185+0000 7f99cb59e640 1 Processor -- start 2023-11-28T20:34:53.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.185+0000 7f99cb59e640 1 -- start start 2023-11-28T20:34:53.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.186+0000 7f99cb59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99cc154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.186+0000 7f99cb59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99cc162bf0 0x7f99cc0ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.186+0000 7f99cb59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99cc0ada80 0x7f99cc09b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.186+0000 7f99cb59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc0adf00 con 0x7f99b4003970 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.186+0000 7f99cb59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc09c050 con 0x7f99cc0ada80 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.186+0000 7f99cb59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc161e70 con 0x7f99cc162bf0 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.186+0000 7f99ca59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99cc154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.186+0000 7f99ca59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99cc154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49168/0 (socket says 172.21.15.45:49168) 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.186+0000 7f99ca59c640 1 -- 172.21.15.45:0/4137530235 learned_addr learned my addr 172.21.15.45:0/4137530235 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.187+0000 7f99ca59c640 1 -- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99cc162bf0 msgr2=0x7f99cc0ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.187+0000 7f99ca59c640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99cc162bf0 0x7f99cc0ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.187+0000 7f99ca59c640 1 -- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99cc0ada80 msgr2=0x7f99cc09b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.187+0000 7f99ca59c640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99cc0ada80 0x7f99cc09b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.187+0000 7f99ca59c640 1 -- 172.21.15.45:0/4137530235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc162110 con 0x7f99b4003970 2023-11-28T20:34:53.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.187+0000 7f99ca59c640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99cc154f00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f99bc02c250 tx=0x7f99bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:53.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.187+0000 7f99baffd640 1 -- 172.21.15.45:0/4137530235 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc003810 con 0x7f99b4003970 2023-11-28T20:34:53.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.187+0000 7f99baffd640 1 -- 172.21.15.45:0/4137530235 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99bc003970 con 0x7f99b4003970 2023-11-28T20:34:53.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.187+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99cc1623f0 con 0x7f99b4003970 2023-11-28T20:34:53.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.188+0000 7f99baffd640 1 -- 172.21.15.45:0/4137530235 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc003d00 con 0x7f99b4003970 2023-11-28T20:34:53.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.188+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99cc155440 con 0x7f99b4003970 2023-11-28T20:34:53.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.189+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99cc0adea0 con 0x7f99b4003970 2023-11-28T20:34:53.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.190+0000 7f99baffd640 1 -- 172.21.15.45:0/4137530235 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f99bc034440 con 0x7f99b4003970 2023-11-28T20:34:53.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.191+0000 7f99baffd640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f99ac06ae90 0x7f99ac06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.191+0000 7f99baffd640 1 -- 172.21.15.45:0/4137530235 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f99bc0aba40 con 0x7f99b4003970 2023-11-28T20:34:53.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.191+0000 7f99c9d9b640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f99ac06ae90 0x7f99ac06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.192+0000 7f99c9d9b640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f99ac06ae90 0x7f99ac06d350 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f99c00017a0 tx=0x7f99c0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:53.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.195+0000 7f99baffd640 1 -- 172.21.15.45:0/4137530235 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99cc0adea0 con 0x7f99b4003970 2023-11-28T20:34:53.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.391+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f99cc0adea0 con 0x7f99b4003970 2023-11-28T20:34:53.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.391+0000 7f99baffd640 1 -- 172.21.15.45:0/4137530235 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f99bc076950 con 0x7f99b4003970 2023-11-28T20:34:53.394 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:34:53.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f99ac06ae90 msgr2=0x7f99ac06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f99ac06ae90 0x7f99ac06d350 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f99c00017a0 tx=0x7f99c0009040 comp rx=0 tx=0).stop 2023-11-28T20:34:53.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 msgr2=0x7f99cc154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99cc154f00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f99bc02c250 tx=0x7f99bc004730 comp rx=0 tx=0).stop 2023-11-28T20:34:53.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 shutdown_connections 2023-11-28T20:34:53.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f99ac06ae90 0x7f99ac06d350 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99cc0ada80 0x7f99cc09b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99cc162bf0 0x7f99cc0ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 --2- 172.21.15.45:0/4137530235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99b4003970 0x7f99cc154f00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 >> 172.21.15.45:0/4137530235 conn(0x7f99cc0b1050 msgr2=0x7f99cc0ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:53.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.394+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 shutdown_connections 2023-11-28T20:34:53.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.395+0000 7f99cb59e640 1 -- 172.21.15.45:0/4137530235 wait complete. 2023-11-28T20:34:53.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:34:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.641+0000 7f3f99316640 1 Processor -- start 2023-11-28T20:34:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.642+0000 7f3f99316640 1 -- start start 2023-11-28T20:34:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.643+0000 7f3f99316640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f9415e840 0x7f3f9406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.643+0000 7f3f99316640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9415ecd0 con 0x7f3f9415f720 2023-11-28T20:34:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.643+0000 7f3f99316640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9415f290 con 0x7f3f9406ac10 2023-11-28T20:34:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.643+0000 7f3f99316640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9415fb60 con 0x7f3f9415e840 2023-11-28T20:34:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.643+0000 7f3f927fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3f9415f720 0x7f3f9406a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38608/0 (socket says 172.21.15.45:38608) 2023-11-28T20:34:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.643+0000 7f3f927fc640 1 -- 172.21.15.45:0/2386476517 learned_addr learned my addr 172.21.15.45:0/2386476517 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f92ffd640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f9415e840 0x7f3f9406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321585928 0 0) 0x7f3f9415ecd0 con 0x7f3f9415f720 2023-11-28T20:34:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f80003520 con 0x7f3f9415f720 2023-11-28T20:34:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 733626072 0 0) 0x7f3f80003520 con 0x7f3f9415f720 2023-11-28T20:34:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f9415e840 msgr2=0x7f3f9406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f9415e840 0x7f3f9406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 >> v1:172.21.15.190:6789/0 conn(0x7f3f9406ac10 legacy=0x7f3f940cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f94072020 con 0x7f3f9415f720 2023-11-28T20:34:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3f7c001670 con 0x7f3f9415f720 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 >> v1:172.21.15.45:6789/0 conn(0x7f3f9415f720 legacy=0x7f3f9406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f800039a0 0x7f3f80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f9415e840 0x7f3f800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f80004d00 0x7f3f80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9415fb60 con 0x7f3f80004d00 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8000ba30 con 0x7f3f800039a0 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.644+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8000bbf0 con 0x7f3f9415e840 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.645+0000 7f3f937fe640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f80004d00 0x7f3f80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.645+0000 7f3f92ffd640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f800039a0 0x7f3f80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.645+0000 7f3f927fc640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f9415e840 0x7f3f800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.645+0000 7f3f937fe640 1 -- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f9415e840 msgr2=0x7f3f800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.645+0000 7f3f937fe640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f9415e840 0x7f3f800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.645+0000 7f3f937fe640 1 -- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f800039a0 msgr2=0x7f3f80003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.645+0000 7f3f937fe640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f800039a0 0x7f3f80003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.645+0000 7f3f937fe640 1 -- 172.21.15.45:0/2386476517 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f80003520 con 0x7f3f80004d00 2023-11-28T20:34:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.646+0000 7f3f937fe640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f80004d00 0x7f3f80005180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c001720 tx=0x7f3f800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32ed80b041b7d4a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.646+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c0052d0 con 0x7f3f80004d00 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.647+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8c02e050 con 0x7f3f80004d00 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.647+0000 7f3f91ffb640 1 -- 172.21.15.45:0/2386476517 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c0035f0 con 0x7f3f80004d00 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.647+0000 7f3f99316640 1 -- 172.21.15.45:0/2386476517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f80004d00 msgr2=0x7f3f80005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.647+0000 7f3f99316640 1 --2- 172.21.15.45:0/2386476517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f80004d00 0x7f3f80005180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c001720 tx=0x7f3f800057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.647+0000 7f3f937fe640 1 -- 172.21.15.45:0/2386476517 reap_dead start 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.647+0000 7f3f99316640 1 -- 172.21.15.45:0/2386476517 shutdown_connections 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.647+0000 7f3f99316640 1 -- 172.21.15.45:0/2386476517 >> 172.21.15.45:0/2386476517 conn(0x7f3f940ced60 msgr2=0x7f3f940cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.647+0000 7f3f99316640 1 -- 172.21.15.45:0/2386476517 shutdown_connections 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.647+0000 7f3f99316640 1 -- 172.21.15.45:0/2386476517 wait complete. 2023-11-28T20:34:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.649+0000 7f3f99316640 1 Processor -- start 2023-11-28T20:34:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.649+0000 7f3f99316640 1 -- start start 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.650+0000 7f3f99316640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f80004d00 0x7f3f94158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.650+0000 7f3f99316640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f941552f0 0x7f3f940c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.650+0000 7f3f99316640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f940c5220 0x7f3f94159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.650+0000 7f3f99316640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f940c56a0 con 0x7f3f941552f0 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.650+0000 7f3f99316640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f94158cb0 con 0x7f3f940c5220 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.650+0000 7f3f99316640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f941f8db0 con 0x7f3f80004d00 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.650+0000 7f3f927fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f941552f0 0x7f3f940c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.650+0000 7f3f927fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f941552f0 0x7f3f940c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49178/0 (socket says 172.21.15.45:49178) 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.650+0000 7f3f927fc640 1 -- 172.21.15.45:0/4257627863 learned_addr learned my addr 172.21.15.45:0/4257627863 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f937fe640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f940c5220 0x7f3f94159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f927fc640 1 -- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f80004d00 msgr2=0x7f3f94158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f92ffd640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f80004d00 0x7f3f94158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f927fc640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f80004d00 0x7f3f94158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f927fc640 1 -- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f940c5220 msgr2=0x7f3f94159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f927fc640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f940c5220 0x7f3f94159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f927fc640 1 -- 172.21.15.45:0/4257627863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f941f81a0 con 0x7f3f941552f0 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f937fe640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f940c5220 0x7f3f94159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f927fc640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f941552f0 0x7f3f940c4ce0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00e820 tx=0x7f3f7c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.651+0000 7f3f8b7fe640 1 -- 172.21.15.45:0/4257627863 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c00c8d0 con 0x7f3f941552f0 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.652+0000 7f3f8b7fe640 1 -- 172.21.15.45:0/4257627863 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c00ca30 con 0x7f3f941552f0 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.652+0000 7f3f8b7fe640 1 -- 172.21.15.45:0/4257627863 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c00cdc0 con 0x7f3f941552f0 2023-11-28T20:34:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.652+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f941f8480 con 0x7f3f941552f0 2023-11-28T20:34:53.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.652+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f941f8a40 con 0x7f3f941552f0 2023-11-28T20:34:53.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.653+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f940733d0 con 0x7f3f941552f0 2023-11-28T20:34:53.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.654+0000 7f3f8b7fe640 1 -- 172.21.15.45:0/4257627863 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3f7c01d460 con 0x7f3f941552f0 2023-11-28T20:34:53.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.655+0000 7f3f8b7fe640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f7406ae40 0x7f3f7406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:53.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.655+0000 7f3f8b7fe640 1 -- 172.21.15.45:0/4257627863 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3f7c089a10 con 0x7f3f941552f0 2023-11-28T20:34:53.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.655+0000 7f3f92ffd640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f7406ae40 0x7f3f7406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:53.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.656+0000 7f3f92ffd640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f7406ae40 0x7f3f7406d300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3f84007380 tx=0x7f3f8400b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:53.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.659+0000 7f3f8b7fe640 1 -- 172.21.15.45:0/4257627863 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f7c0546f0 con 0x7f3f941552f0 2023-11-28T20:34:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.856+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3f940ce2b0 con 0x7f3f941552f0 2023-11-28T20:34:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.857+0000 7f3f8b7fe640 1 -- 172.21.15.45:0/4257627863 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3f7c057d10 con 0x7f3f941552f0 2023-11-28T20:34:53.860 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:53.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f7406ae40 msgr2=0x7f3f7406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f7406ae40 0x7f3f7406d300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3f84007380 tx=0x7f3f8400b040 comp rx=0 tx=0).stop 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f941552f0 msgr2=0x7f3f940c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f941552f0 0x7f3f940c4ce0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00e820 tx=0x7f3f7c00ece0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 shutdown_connections 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3f7406ae40 0x7f3f7406d300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f940c5220 0x7f3f94159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f941552f0 0x7f3f940c4ce0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 --2- 172.21.15.45:0/4257627863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f80004d00 0x7f3f94158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 >> 172.21.15.45:0/4257627863 conn(0x7f3f940ced60 msgr2=0x7f3f940abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 shutdown_connections 2023-11-28T20:34:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:53.860+0000 7f3f99316640 1 -- 172.21.15.45:0/4257627863 wait complete. 2023-11-28T20:34:53.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:54.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.099+0000 7f92c4b16640 1 Processor -- start 2023-11-28T20:34:54.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.099+0000 7f92c4b16640 1 -- start start 2023-11-28T20:34:54.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.100+0000 7f92c4b16640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f92c015d550 0x7f92c00cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.100+0000 7f92c4b16640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f92c00d0100 0x7f92c00d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.100+0000 7f92c4b16640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c00d2230 con 0x7f92c00d1ab0 2023-11-28T20:34:54.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92c4b16640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c00d0590 con 0x7f92c015d550 2023-11-28T20:34:54.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92c4b16640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c00d2510 con 0x7f92c00d0100 2023-11-28T20:34:54.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92bf7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f92c015d550 0x7f92c00cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92bf7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f92c015d550 0x7f92c00cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36748/0 (socket says 172.21.15.45:36748) 2023-11-28T20:34:54.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92bf7fe640 1 -- 172.21.15.45:0/1957217024 learned_addr learned my addr 172.21.15.45:0/1957217024 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:54.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92beffd640 1 --2- 172.21.15.45:0/1957217024 >> v2:172.21.15.190:3300/0 conn(0x7f92c00d0100 0x7f92c00d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92bf7fe640 1 -- 172.21.15.45:0/1957217024 >> v2:172.21.15.190:3300/0 conn(0x7f92c00d0100 msgr2=0x7f92c00d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:54.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92bf7fe640 1 --2- 172.21.15.45:0/1957217024 >> v2:172.21.15.190:3300/0 conn(0x7f92c00d0100 0x7f92c00d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92bf7fe640 1 -- 172.21.15.45:0/1957217024 >> v1:172.21.15.190:6789/0 conn(0x7f92c00d1ab0 legacy=0x7f92c0070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:54.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92bf7fe640 1 -- 172.21.15.45:0/1957217024 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00d27f0 con 0x7f92c015d550 2023-11-28T20:34:54.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.101+0000 7f92beffd640 1 --2- 172.21.15.45:0/1957217024 >> v2:172.21.15.190:3300/0 conn(0x7f92c00d0100 0x7f92c00d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:54.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.102+0000 7f92bf7fe640 1 --2- 172.21.15.45:0/1957217024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c015d550 0x7f92c00cfbc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c9ef0 tx=0x7f92b002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b468ce35c7e7813 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:54.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.103+0000 7f92be7fc640 1 -- 172.21.15.45:0/1957217024 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b002ca80 con 0x7f92c015d550 2023-11-28T20:34:54.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.103+0000 7f92be7fc640 1 -- 172.21.15.45:0/1957217024 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b002cbe0 con 0x7f92c015d550 2023-11-28T20:34:54.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.103+0000 7f92be7fc640 1 -- 172.21.15.45:0/1957217024 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b00343f0 con 0x7f92c015d550 2023-11-28T20:34:54.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.103+0000 7f92c4b16640 1 -- 172.21.15.45:0/1957217024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c015d550 msgr2=0x7f92c00cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.103+0000 7f92c4b16640 1 --2- 172.21.15.45:0/1957217024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c015d550 0x7f92c00cfbc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c9ef0 tx=0x7f92b002bee0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.104+0000 7f92c4b16640 1 -- 172.21.15.45:0/1957217024 shutdown_connections 2023-11-28T20:34:54.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.104+0000 7f92c4b16640 1 --2- 172.21.15.45:0/1957217024 >> v2:172.21.15.190:3300/0 conn(0x7f92c00d0100 0x7f92c00d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.104+0000 7f92c4b16640 1 --2- 172.21.15.45:0/1957217024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c015d550 0x7f92c00cfbc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.104+0000 7f92c4b16640 1 -- 172.21.15.45:0/1957217024 >> 172.21.15.45:0/1957217024 conn(0x7f92c006ac70 msgr2=0x7f92c00b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:54.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.104+0000 7f92c4b16640 1 -- 172.21.15.45:0/1957217024 shutdown_connections 2023-11-28T20:34:54.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.104+0000 7f92c4b16640 1 -- 172.21.15.45:0/1957217024 wait complete. 2023-11-28T20:34:54.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.105+0000 7f92c4b16640 1 Processor -- start 2023-11-28T20:34:54.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.106+0000 7f92c4b16640 1 -- start start 2023-11-28T20:34:54.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92c4b16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c00d0100 0x7f92c00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92c4b16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92c00d1ab0 0x7f92c00cd620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92c4b16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92c015d550 0x7f92c00abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92c4b16640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c01f3340 con 0x7f92c00d0100 2023-11-28T20:34:54.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92c4b16640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00cc2e0 con 0x7f92c015d550 2023-11-28T20:34:54.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92c4b16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00cc5c0 con 0x7f92c00d1ab0 2023-11-28T20:34:54.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92bf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c00d0100 0x7f92c00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92bf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c00d0100 0x7f92c00cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36756/0 (socket says 172.21.15.45:36756) 2023-11-28T20:34:54.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92bf7fe640 1 -- 172.21.15.45:0/1484006228 learned_addr learned my addr 172.21.15.45:0/1484006228 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:54.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92beffd640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92c00d1ab0 0x7f92c00cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92bf7fe640 1 -- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92c00d1ab0 msgr2=0x7f92c00cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92bf7fe640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92c00d1ab0 0x7f92c00cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92bf7fe640 1 -- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92c015d550 msgr2=0x7f92c00abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92bf7fe640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92c015d550 0x7f92c00abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.107+0000 7f92bf7fe640 1 -- 172.21.15.45:0/1484006228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b0009b30 con 0x7f92c00d0100 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.108+0000 7f92bf7fe640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c00d0100 0x7f92c00cbd00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f92b0000f80 tx=0x7f92b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.108+0000 7f92b7fff640 1 -- 172.21.15.45:0/1484006228 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b003e450 con 0x7f92c00d0100 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.108+0000 7f92b7fff640 1 -- 172.21.15.45:0/1484006228 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b003e5f0 con 0x7f92c00d0100 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.108+0000 7f92b7fff640 1 -- 172.21.15.45:0/1484006228 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b003e9a0 con 0x7f92c00d0100 2023-11-28T20:34:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.108+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c00cc8a0 con 0x7f92c00d0100 2023-11-28T20:34:54.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.108+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c00cdb60 con 0x7f92c00d0100 2023-11-28T20:34:54.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.109+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c004e920 con 0x7f92c00d0100 2023-11-28T20:34:54.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.110+0000 7f92b7fff640 1 -- 172.21.15.45:0/1484006228 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f92b002cd50 con 0x7f92c00d0100 2023-11-28T20:34:54.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.111+0000 7f92b7fff640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92a006ae40 0x7f92a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.111+0000 7f92b7fff640 1 -- 172.21.15.45:0/1484006228 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f92b00ac710 con 0x7f92c00d0100 2023-11-28T20:34:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.112+0000 7f92beffd640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92a006ae40 0x7f92a006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.113+0000 7f92beffd640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92a006ae40 0x7f92a006d300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c61a0 tx=0x7f92a8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.117+0000 7f92b7fff640 1 -- 172.21.15.45:0/1484006228 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92b0077480 con 0x7f92c00d0100 2023-11-28T20:34:54.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.318+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92c015f870 con 0x7f92c00d0100 2023-11-28T20:34:54.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.321+0000 7f92b7fff640 1 -- 172.21.15.45:0/1484006228 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7f92b007aaa0 con 0x7f92c00d0100 2023-11-28T20:34:54.324 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:54.324 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.324+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92a006ae40 msgr2=0x7f92a006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.324+0000 7f92c4b16640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92a006ae40 0x7f92a006d300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c61a0 tx=0x7f92a8008040 comp rx=0 tx=0).stop 2023-11-28T20:34:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.324+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c00d0100 msgr2=0x7f92c00cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.324+0000 7f92c4b16640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c00d0100 0x7f92c00cbd00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f92b0000f80 tx=0x7f92b0004a40 comp rx=0 tx=0).stop 2023-11-28T20:34:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.325+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 shutdown_connections 2023-11-28T20:34:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.325+0000 7f92c4b16640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92a006ae40 0x7f92a006d300 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.325+0000 7f92c4b16640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92c015d550 0x7f92c00abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.325+0000 7f92c4b16640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92c00d1ab0 0x7f92c00cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.325+0000 7f92c4b16640 1 --2- 172.21.15.45:0/1484006228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92c00d0100 0x7f92c00cbd00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.325+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 >> 172.21.15.45:0/1484006228 conn(0x7f92c006ac70 msgr2=0x7f92c015c790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:54.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.325+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 shutdown_connections 2023-11-28T20:34:54.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.325+0000 7f92c4b16640 1 -- 172.21.15.45:0/1484006228 wait complete. 2023-11-28T20:34:54.347 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:34:54.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:34:54.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.578+0000 7f2dabfff640 1 Processor -- start 2023-11-28T20:34:54.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.579+0000 7f2dabfff640 1 -- start start 2023-11-28T20:34:54.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.580+0000 7f2dabfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac15b270 0x7f2dac06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.580+0000 7f2dabfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2dac0b3420 0x7f2dac0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.580+0000 7f2dabfff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dac06d410 con 0x7f2dac0b3040 2023-11-28T20:34:54.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.580+0000 7f2dabfff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dac0ceaa0 con 0x7f2dac0b3420 2023-11-28T20:34:54.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.580+0000 7f2dabfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac06d6f0 con 0x7f2dac15b270 2023-11-28T20:34:54.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.580+0000 7f2daaffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2dac0b3040 0x7f2dac15f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60832/0 (socket says 172.21.15.45:60832) 2023-11-28T20:34:54.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.580+0000 7f2daaffd640 1 -- 172.21.15.45:0/3468583994 learned_addr learned my addr 172.21.15.45:0/3468583994 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2dab7fe640 1 --2- 172.21.15.45:0/3468583994 >> v2:172.21.15.190:3300/0 conn(0x7f2dac0b3420 0x7f2dac0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2daa7fc640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac15b270 0x7f2dac06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665951524 0 0) 0x7f2dac06d410 con 0x7f2dac0b3040 2023-11-28T20:34:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d94003520 con 0x7f2dac0b3040 2023-11-28T20:34:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2521571639 0 0) 0x7f2d94003520 con 0x7f2dac0b3040 2023-11-28T20:34:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac15b270 msgr2=0x7f2dac06cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac15b270 0x7f2dac06cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 >> v2:172.21.15.190:3300/0 conn(0x7f2dac0b3420 msgr2=0x7f2dac0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 --2- 172.21.15.45:0/3468583994 >> v2:172.21.15.190:3300/0 conn(0x7f2dac0b3420 0x7f2dac0ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac06d9d0 con 0x7f2dac0b3040 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f2d9c0019f0 con 0x7f2dac0b3040 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 >> v1:172.21.15.45:6789/0 conn(0x7f2dac0b3040 legacy=0x7f2dac15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94003970 0x7f2d94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac15b270 0x7f2d940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d94004d00 0x7f2d94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac06d6f0 con 0x7f2d94004d00 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9400b980 con 0x7f2d94003970 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.581+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9400bb40 con 0x7f2dac15b270 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.582+0000 7f2dab7fe640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d94004d00 0x7f2d94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.582+0000 7f2daa7fc640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac15b270 0x7f2d940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.582+0000 7f2daaffd640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94003970 0x7f2d94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.582+0000 7f2dab7fe640 1 -- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac15b270 msgr2=0x7f2d940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:54.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.582+0000 7f2dab7fe640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac15b270 0x7f2d940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.582+0000 7f2dab7fe640 1 -- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94003970 msgr2=0x7f2d94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:54.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.582+0000 7f2dab7fe640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94003970 0x7f2d94003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.582+0000 7f2dab7fe640 1 -- 172.21.15.45:0/3468583994 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d94003520 con 0x7f2d94004d00 2023-11-28T20:34:54.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.582+0000 7f2daaffd640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94003970 0x7f2d94003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:54.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.583+0000 7f2dab7fe640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d94004d00 0x7f2d94005180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009ac0 tx=0x7f2d940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955b96078a41996a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:54.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.584+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da402c710 con 0x7f2d94004d00 2023-11-28T20:34:54.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.584+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da402c870 con 0x7f2d94004d00 2023-11-28T20:34:54.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.584+0000 7f2da9ffb640 1 -- 172.21.15.45:0/3468583994 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da402cc00 con 0x7f2d94004d00 2023-11-28T20:34:54.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.584+0000 7f2dabfff640 1 -- 172.21.15.45:0/3468583994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d94004d00 msgr2=0x7f2d94005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.584+0000 7f2dabfff640 1 --2- 172.21.15.45:0/3468583994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d94004d00 0x7f2d94005180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009ac0 tx=0x7f2d940057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.584+0000 7f2dab7fe640 1 -- 172.21.15.45:0/3468583994 reap_dead start 2023-11-28T20:34:54.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.584+0000 7f2dabfff640 1 -- 172.21.15.45:0/3468583994 shutdown_connections 2023-11-28T20:34:54.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.584+0000 7f2dabfff640 1 -- 172.21.15.45:0/3468583994 >> 172.21.15.45:0/3468583994 conn(0x7f2dac0b1050 msgr2=0x7f2dac0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:54.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.584+0000 7f2dabfff640 1 -- 172.21.15.45:0/3468583994 shutdown_connections 2023-11-28T20:34:54.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.585+0000 7f2dabfff640 1 -- 172.21.15.45:0/3468583994 wait complete. 2023-11-28T20:34:54.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.587+0000 7f2dabfff640 1 Processor -- start 2023-11-28T20:34:54.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.587+0000 7f2dabfff640 1 -- start start 2023-11-28T20:34:54.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dabfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94004d00 0x7f2dac0717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dabfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac14a7d0 0x7f2dac15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dabfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dac159720 0x7f2dac159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dabfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac15a0e0 con 0x7f2dac159720 2023-11-28T20:34:54.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dabfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac070720 con 0x7f2d94004d00 2023-11-28T20:34:54.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dabfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac070a00 con 0x7f2dac14a7d0 2023-11-28T20:34:54.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dab7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dac159720 0x7f2dac159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dab7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dac159720 0x7f2dac159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36788/0 (socket says 172.21.15.45:36788) 2023-11-28T20:34:54.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dab7fe640 1 -- 172.21.15.45:0/2765848716 learned_addr learned my addr 172.21.15.45:0/2765848716 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2daaffd640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94004d00 0x7f2dac0717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2daa7fc640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac14a7d0 0x7f2dac15d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dab7fe640 1 -- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac14a7d0 msgr2=0x7f2dac15d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dab7fe640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac14a7d0 0x7f2dac15d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dab7fe640 1 -- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94004d00 msgr2=0x7f2dac0717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.588+0000 7f2dab7fe640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94004d00 0x7f2dac0717c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.589+0000 7f2dab7fe640 1 -- 172.21.15.45:0/2765848716 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac070ce0 con 0x7f2dac159720 2023-11-28T20:34:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.589+0000 7f2dab7fe640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dac159720 0x7f2dac159ba0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009ac0 tx=0x7f2da4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.589+0000 7f2d937fe640 1 -- 172.21.15.45:0/2765848716 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da402c710 con 0x7f2dac159720 2023-11-28T20:34:54.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.589+0000 7f2d937fe640 1 -- 172.21.15.45:0/2765848716 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da4003830 con 0x7f2dac159720 2023-11-28T20:34:54.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.589+0000 7f2d937fe640 1 -- 172.21.15.45:0/2765848716 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4003be0 con 0x7f2dac159720 2023-11-28T20:34:54.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.589+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac070fc0 con 0x7f2dac159720 2023-11-28T20:34:54.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.589+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac15da00 con 0x7f2dac159720 2023-11-28T20:34:54.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.591+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac0714e0 con 0x7f2dac159720 2023-11-28T20:34:54.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.592+0000 7f2d937fe640 1 -- 172.21.15.45:0/2765848716 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2da4003d40 con 0x7f2dac159720 2023-11-28T20:34:54.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.592+0000 7f2d937fe640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2d9806ae70 0x7f2d9806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:54.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.593+0000 7f2d937fe640 1 -- 172.21.15.45:0/2765848716 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2da40aaa60 con 0x7f2dac159720 2023-11-28T20:34:54.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.593+0000 7f2daaffd640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2d9806ae70 0x7f2d9806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:54.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.594+0000 7f2daaffd640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2d9806ae70 0x7f2d9806d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c002230 tx=0x7f2d9c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:54.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.597+0000 7f2d937fe640 1 -- 172.21.15.45:0/2765848716 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dac0714e0 con 0x7f2dac159720 2023-11-28T20:34:54.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.788+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2dac0714e0 con 0x7f2dac159720 2023-11-28T20:34:54.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.789+0000 7f2d937fe640 1 -- 172.21.15.45:0/2765848716 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2dac0714e0 con 0x7f2dac159720 2023-11-28T20:34:54.791 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:34:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.791+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2d9806ae70 msgr2=0x7f2d9806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.791+0000 7f2dabfff640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2d9806ae70 0x7f2d9806d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c002230 tx=0x7f2d9c009040 comp rx=0 tx=0).stop 2023-11-28T20:34:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.791+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dac159720 msgr2=0x7f2dac159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.791+0000 7f2dabfff640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dac159720 0x7f2dac159ba0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009ac0 tx=0x7f2da4004730 comp rx=0 tx=0).stop 2023-11-28T20:34:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.792+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 shutdown_connections 2023-11-28T20:34:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.792+0000 7f2dabfff640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2d9806ae70 0x7f2d9806d330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.792+0000 7f2dabfff640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dac159720 0x7f2dac159ba0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.792+0000 7f2dabfff640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dac14a7d0 0x7f2dac15d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.792+0000 7f2dabfff640 1 --2- 172.21.15.45:0/2765848716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d94004d00 0x7f2dac0717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:54.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.792+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 >> 172.21.15.45:0/2765848716 conn(0x7f2dac0b1050 msgr2=0x7f2dac0abfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:54.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.792+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 shutdown_connections 2023-11-28T20:34:54.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:54.792+0000 7f2dabfff640 1 -- 172.21.15.45:0/2765848716 wait complete. 2023-11-28T20:34:54.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:34:55.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.041+0000 7f8dd7052640 1 Processor -- start 2023-11-28T20:34:55.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.042+0000 7f8dd7052640 1 -- start start 2023-11-28T20:34:55.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.043+0000 7f8dd7052640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dd00cdca0 0x7f8dd006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.043+0000 7f8dd7052640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd006d410 con 0x7f8dd00b3040 2023-11-28T20:34:55.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.043+0000 7f8dd7052640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd006d6f0 con 0x7f8dd015b270 2023-11-28T20:34:55.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.043+0000 7f8dd7052640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00ceaa0 con 0x7f8dd00cdca0 2023-11-28T20:34:55.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.044+0000 7f8dd6050640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8dd00b3040 0x7f8dd015f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60834/0 (socket says 172.21.15.45:60834) 2023-11-28T20:34:55.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.044+0000 7f8dd6050640 1 -- 172.21.15.45:0/2795639459 learned_addr learned my addr 172.21.15.45:0/2795639459 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.044+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896263092 0 0) 0x7f8dd006d410 con 0x7f8dd00b3040 2023-11-28T20:34:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.044+0000 7f8dd6851640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dd00cdca0 0x7f8dd006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.044+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dbc003520 con 0x7f8dd00b3040 2023-11-28T20:34:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1330200380 0 0) 0x7f8dbc003520 con 0x7f8dd00b3040 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dd00cdca0 msgr2=0x7f8dd006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dd00cdca0 0x7f8dd006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 >> v1:172.21.15.190:6789/0 conn(0x7f8dd015b270 legacy=0x7f8dd00b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd006d9d0 con 0x7f8dd00b3040 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f8dcc0019f0 con 0x7f8dd00b3040 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 >> v1:172.21.15.45:6789/0 conn(0x7f8dd00b3040 legacy=0x7f8dd015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dd00cdca0 0x7f8dbc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dbc003f70 0x7f8dbc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc004d00 0x7f8dbc005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc00bbe0 con 0x7f8dbc003f70 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc00bd50 con 0x7f8dbc004d00 2023-11-28T20:34:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc00bf10 con 0x7f8dd00cdca0 2023-11-28T20:34:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd584f640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dbc003f70 0x7f8dbc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.045+0000 7f8dd6050640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dd00cdca0 0x7f8dbc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.046+0000 7f8dd6851640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc004d00 0x7f8dbc005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.046+0000 7f8dd584f640 1 -- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dd00cdca0 msgr2=0x7f8dbc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.046+0000 7f8dd584f640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dd00cdca0 0x7f8dbc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.046+0000 7f8dd584f640 1 -- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc004d00 msgr2=0x7f8dbc005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.046+0000 7f8dd584f640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc004d00 0x7f8dbc005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.046+0000 7f8dd584f640 1 -- 172.21.15.45:0/2795639459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc003520 con 0x7f8dbc003f70 2023-11-28T20:34:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.047+0000 7f8dd584f640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dbc003f70 0x7f8dbc0047c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0001720 tx=0x7f8dbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c4d23ed67cc64f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.047+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc00052d0 con 0x7f8dbc003f70 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.047+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc002e050 con 0x7f8dbc003f70 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.047+0000 7f8dd504e640 1 -- 172.21.15.45:0/2795639459 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0003830 con 0x7f8dbc003f70 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.047+0000 7f8dd7052640 1 -- 172.21.15.45:0/2795639459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dbc003f70 msgr2=0x7f8dbc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.048+0000 7f8dd7052640 1 --2- 172.21.15.45:0/2795639459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dbc003f70 0x7f8dbc0047c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0001720 tx=0x7f8dbc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.048+0000 7f8dd6851640 1 -- 172.21.15.45:0/2795639459 reap_dead start 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.048+0000 7f8dd7052640 1 -- 172.21.15.45:0/2795639459 shutdown_connections 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.048+0000 7f8dd7052640 1 -- 172.21.15.45:0/2795639459 >> 172.21.15.45:0/2795639459 conn(0x7f8dd00b1050 msgr2=0x7f8dd00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.048+0000 7f8dd7052640 1 -- 172.21.15.45:0/2795639459 shutdown_connections 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.048+0000 7f8dd7052640 1 -- 172.21.15.45:0/2795639459 wait complete. 2023-11-28T20:34:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.050+0000 7f8dd7052640 1 Processor -- start 2023-11-28T20:34:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.050+0000 7f8dd7052640 1 -- start start 2023-11-28T20:34:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd7052640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc003f70 0x7f8dd014ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd7052640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd015a3c0 0x7f8dd00aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd7052640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dd014dbd0 0x7f8dd014a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd7052640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd01527c0 con 0x7f8dd015a3c0 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd7052640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0152050 con 0x7f8dd014dbd0 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd7052640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0152330 con 0x7f8dbc003f70 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd584f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd015a3c0 0x7f8dd00aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd584f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd015a3c0 0x7f8dd00aeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36812/0 (socket says 172.21.15.45:36812) 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd584f640 1 -- 172.21.15.45:0/3918649892 learned_addr learned my addr 172.21.15.45:0/3918649892 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd6050640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc003f70 0x7f8dd014ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd6851640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dd014dbd0 0x7f8dd014a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd584f640 1 -- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc003f70 msgr2=0x7f8dd014ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.051+0000 7f8dd584f640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc003f70 0x7f8dd014ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.052+0000 7f8dd584f640 1 -- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dd014dbd0 msgr2=0x7f8dd014a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.052+0000 7f8dd584f640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dd014dbd0 0x7f8dd014a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.052+0000 7f8dd584f640 1 -- 172.21.15.45:0/3918649892 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0158a00 con 0x7f8dd015a3c0 2023-11-28T20:34:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.052+0000 7f8dd584f640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd015a3c0 0x7f8dd00aeed0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0001720 tx=0x7f8dc0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.052+0000 7f8dc6ffd640 1 -- 172.21.15.45:0/3918649892 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0007a30 con 0x7f8dd015a3c0 2023-11-28T20:34:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.052+0000 7f8dc6ffd640 1 -- 172.21.15.45:0/3918649892 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc003c640 con 0x7f8dd015a3c0 2023-11-28T20:34:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.052+0000 7f8dc6ffd640 1 -- 172.21.15.45:0/3918649892 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc003c9f0 con 0x7f8dd015a3c0 2023-11-28T20:34:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.052+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd0158ce0 con 0x7f8dd015a3c0 2023-11-28T20:34:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.052+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd00af410 con 0x7f8dd015a3c0 2023-11-28T20:34:55.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.054+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd01524a0 con 0x7f8dd015a3c0 2023-11-28T20:34:55.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.055+0000 7f8dc6ffd640 1 -- 172.21.15.45:0/3918649892 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8dc003cb50 con 0x7f8dd015a3c0 2023-11-28T20:34:55.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.056+0000 7f8dc6ffd640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8db406ae70 0x7f8db406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.056+0000 7f8dc6ffd640 1 -- 172.21.15.45:0/3918649892 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8dc00ab810 con 0x7f8dd015a3c0 2023-11-28T20:34:55.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.056+0000 7f8dd6050640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8db406ae70 0x7f8db406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.057+0000 7f8dd6050640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8db406ae70 0x7f8db406d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc002230 tx=0x7f8dcc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.060+0000 7f8dc6ffd640 1 -- 172.21.15.45:0/3918649892 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8dc00764f0 con 0x7f8dd015a3c0 2023-11-28T20:34:55.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.257+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8dd009d0f0 con 0x7f8dd015a3c0 2023-11-28T20:34:55.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.258+0000 7f8dc6ffd640 1 -- 172.21.15.45:0/3918649892 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8dc004c020 con 0x7f8dd015a3c0 2023-11-28T20:34:55.261 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8db406ae70 msgr2=0x7f8db406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8db406ae70 0x7f8db406d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc002230 tx=0x7f8dcc009040 comp rx=0 tx=0).stop 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd015a3c0 msgr2=0x7f8dd00aeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd015a3c0 0x7f8dd00aeed0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0001720 tx=0x7f8dc0001120 comp rx=0 tx=0).stop 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 shutdown_connections 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8db406ae70 0x7f8db406d330 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dd014dbd0 0x7f8dd014a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dd015a3c0 0x7f8dd00aeed0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 --2- 172.21.15.45:0/3918649892 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc003f70 0x7f8dd014ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 >> 172.21.15.45:0/3918649892 conn(0x7f8dd00b1050 msgr2=0x7f8dd00b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.261+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 shutdown_connections 2023-11-28T20:34:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.262+0000 7f8dd7052640 1 -- 172.21.15.45:0/3918649892 wait complete. 2023-11-28T20:34:55.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:34:55.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.504+0000 7f3309367640 1 Processor -- start 2023-11-28T20:34:55.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.505+0000 7f3309367640 1 -- start start 2023-11-28T20:34:55.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.506+0000 7f3309367640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040d0040 0x7f33040d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.506+0000 7f3309367640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f33040b2290 0x7f33040d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.506+0000 7f3309367640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33040b90a0 con 0x7f33040b1d40 2023-11-28T20:34:55.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.506+0000 7f3309367640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33040d1920 con 0x7f33040b2290 2023-11-28T20:34:55.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.506+0000 7f3309367640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33040d2dc0 con 0x7f33040d0040 2023-11-28T20:34:55.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.507+0000 7f3303fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040d0040 0x7f33040d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.507+0000 7f33037fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f33040b2290 0x7f33040d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.507+0000 7f3303fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040d0040 0x7f33040d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:46302/0 (socket says 172.21.15.45:46302) 2023-11-28T20:34:55.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.507+0000 7f3303fff640 1 -- 172.21.15.45:0/2137123777 learned_addr learned my addr 172.21.15.45:0/2137123777 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.508+0000 7f3303fff640 1 -- 172.21.15.45:0/2137123777 >> v2:172.21.15.190:3300/0 conn(0x7f33040b2290 msgr2=0x7f33040d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.508+0000 7f3303fff640 1 --2- 172.21.15.45:0/2137123777 >> v2:172.21.15.190:3300/0 conn(0x7f33040b2290 0x7f33040d3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.508+0000 7f3303fff640 1 -- 172.21.15.45:0/2137123777 >> v1:172.21.15.190:6789/0 conn(0x7f33040b1d40 legacy=0x7f33040d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.508+0000 7f3303fff640 1 -- 172.21.15.45:0/2137123777 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33040d0960 con 0x7f33040d0040 2023-11-28T20:34:55.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.510+0000 7f3303fff640 1 --2- 172.21.15.45:0/2137123777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040d0040 0x7f33040d0420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009e50 tx=0x7f32f402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f7a550112de7401 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:55.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.510+0000 7f3302ffd640 1 -- 172.21.15.45:0/2137123777 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f402cb90 con 0x7f33040d0040 2023-11-28T20:34:55.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.510+0000 7f3302ffd640 1 -- 172.21.15.45:0/2137123777 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f402ccf0 con 0x7f33040d0040 2023-11-28T20:34:55.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.510+0000 7f3302ffd640 1 -- 172.21.15.45:0/2137123777 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4034610 con 0x7f33040d0040 2023-11-28T20:34:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.511+0000 7f3309367640 1 -- 172.21.15.45:0/2137123777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040d0040 msgr2=0x7f33040d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.511+0000 7f3309367640 1 --2- 172.21.15.45:0/2137123777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040d0040 0x7f33040d0420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009e50 tx=0x7f32f402bff0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.511+0000 7f3309367640 1 -- 172.21.15.45:0/2137123777 shutdown_connections 2023-11-28T20:34:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.511+0000 7f3309367640 1 --2- 172.21.15.45:0/2137123777 >> v2:172.21.15.190:3300/0 conn(0x7f33040b2290 0x7f33040d3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.511+0000 7f3309367640 1 --2- 172.21.15.45:0/2137123777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040d0040 0x7f33040d0420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.511+0000 7f3309367640 1 -- 172.21.15.45:0/2137123777 >> 172.21.15.45:0/2137123777 conn(0x7f3304070f40 msgr2=0x7f3304071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.511+0000 7f3309367640 1 -- 172.21.15.45:0/2137123777 shutdown_connections 2023-11-28T20:34:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.511+0000 7f3309367640 1 -- 172.21.15.45:0/2137123777 wait complete. 2023-11-28T20:34:55.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.513+0000 7f3309367640 1 Processor -- start 2023-11-28T20:34:55.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.513+0000 7f3309367640 1 -- start start 2023-11-28T20:34:55.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.514+0000 7f3309367640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040b1d40 0x7f3304147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.514+0000 7f3309367640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33040b2290 0x7f33040cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.514+0000 7f3309367640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33040d0040 0x7f33041e4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.514+0000 7f3309367640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33040ccbd0 con 0x7f33040b2290 2023-11-28T20:34:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.514+0000 7f3309367640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3304148a60 con 0x7f33040d0040 2023-11-28T20:34:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.514+0000 7f3309367640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33040cc610 con 0x7f33040b1d40 2023-11-28T20:34:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.514+0000 7f33037fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33040b2290 0x7f33040cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.514+0000 7f33037fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33040b2290 0x7f33040cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36818/0 (socket says 172.21.15.45:36818) 2023-11-28T20:34:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.514+0000 7f33037fe640 1 -- 172.21.15.45:0/1760316359 learned_addr learned my addr 172.21.15.45:0/1760316359 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.515+0000 7f3308b66640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33040d0040 0x7f33041e4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.515+0000 7f3303fff640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040b1d40 0x7f3304147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.515+0000 7f33037fe640 1 -- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040b1d40 msgr2=0x7f3304147bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.515+0000 7f33037fe640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040b1d40 0x7f3304147bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.515+0000 7f33037fe640 1 -- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33040d0040 msgr2=0x7f33041e4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.515+0000 7f33037fe640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33040d0040 0x7f33041e4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.515+0000 7f33037fe640 1 -- 172.21.15.45:0/1760316359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f4009b30 con 0x7f33040b2290 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.515+0000 7f33037fe640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33040b2290 0x7f33040cbbe0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00e9b0 tx=0x7f32ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.515+0000 7f3300ff9640 1 -- 172.21.15.45:0/1760316359 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec009670 con 0x7f33040b2290 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.516+0000 7f3300ff9640 1 -- 172.21.15.45:0/1760316359 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ec00c9a0 con 0x7f33040b2290 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.516+0000 7f3300ff9640 1 -- 172.21.15.45:0/1760316359 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec00cd50 con 0x7f33040b2290 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.516+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33040cc8f0 con 0x7f33040b2290 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.516+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330415b810 con 0x7f33040b2290 2023-11-28T20:34:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.517+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3304148550 con 0x7f33040b2290 2023-11-28T20:34:55.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.518+0000 7f3300ff9640 1 -- 172.21.15.45:0/1760316359 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f32ec013470 con 0x7f33040b2290 2023-11-28T20:34:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.519+0000 7f3300ff9640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32e406ae40 0x7f32e406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.519+0000 7f3300ff9640 1 -- 172.21.15.45:0/1760316359 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f32ec089c00 con 0x7f33040b2290 2023-11-28T20:34:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.520+0000 7f3303fff640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32e406ae40 0x7f32e406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.521+0000 7f3303fff640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32e406ae40 0x7f32e406d300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f32f4035000 tx=0x7f32f4030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:55.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.523+0000 7f3300ff9640 1 -- 172.21.15.45:0/1760316359 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3304148550 con 0x7f33040b2290 2023-11-28T20:34:55.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.722+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3304148550 con 0x7f33040b2290 2023-11-28T20:34:55.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.723+0000 7f3300ff9640 1 -- 172.21.15.45:0/1760316359 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3304148550 con 0x7f33040b2290 2023-11-28T20:34:55.725 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32e406ae40 msgr2=0x7f32e406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32e406ae40 0x7f32e406d300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f32f4035000 tx=0x7f32f4030000 comp rx=0 tx=0).stop 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33040b2290 msgr2=0x7f33040cbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33040b2290 0x7f33040cbbe0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00e9b0 tx=0x7f32ec00ee70 comp rx=0 tx=0).stop 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 shutdown_connections 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32e406ae40 0x7f32e406d300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33040d0040 0x7f33041e4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33040b2290 0x7f33040cbbe0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 --2- 172.21.15.45:0/1760316359 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33040b1d40 0x7f3304147bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.725+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 >> 172.21.15.45:0/1760316359 conn(0x7f3304070f40 msgr2=0x7f33040b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:55.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.726+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 shutdown_connections 2023-11-28T20:34:55.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.726+0000 7f3309367640 1 -- 172.21.15.45:0/1760316359 wait complete. 2023-11-28T20:34:55.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:34:55.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.980+0000 7f56d1362640 1 Processor -- start 2023-11-28T20:34:55.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.980+0000 7f56d1362640 1 -- start start 2023-11-28T20:34:55.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.982+0000 7f56d1362640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f56cc0b39d0 0x7f56cc0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.982+0000 7f56d1362640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56cc0cb630 0x7f56cc0cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.982+0000 7f56d1362640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc0724d0 con 0x7f56cc0b3db0 2023-11-28T20:34:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.982+0000 7f56d1362640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc071d20 con 0x7f56cc0cb630 2023-11-28T20:34:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.982+0000 7f56d1362640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc0cc000 con 0x7f56cc0b39d0 2023-11-28T20:34:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.982+0000 7f56d0b61640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56cc0cb630 0x7f56cc0cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.982+0000 7f56d0b61640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56cc0cb630 0x7f56cc0cbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36824/0 (socket says 172.21.15.45:36824) 2023-11-28T20:34:55.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.982+0000 7f56d0b61640 1 -- 172.21.15.45:0/958730103 learned_addr learned my addr 172.21.15.45:0/958730103 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:55.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.982+0000 7f56cbfff640 1 --2- 172.21.15.45:0/958730103 >> v2:172.21.15.190:3300/0 conn(0x7f56cc0b39d0 0x7f56cc0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.983+0000 7f56d0b61640 1 -- 172.21.15.45:0/958730103 >> v2:172.21.15.190:3300/0 conn(0x7f56cc0b39d0 msgr2=0x7f56cc0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.983+0000 7f56d0b61640 1 --2- 172.21.15.45:0/958730103 >> v2:172.21.15.190:3300/0 conn(0x7f56cc0b39d0 0x7f56cc0717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.983+0000 7f56d0b61640 1 -- 172.21.15.45:0/958730103 >> v1:172.21.15.190:6789/0 conn(0x7f56cc0b3db0 legacy=0x7f56cc06bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.983+0000 7f56d0b61640 1 -- 172.21.15.45:0/958730103 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc06c670 con 0x7f56cc0cb630 2023-11-28T20:34:55.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.984+0000 7f56d0b61640 1 --2- 172.21.15.45:0/958730103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0cb630 0x7f56cc0cbac0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0cc400 tx=0x7f56c402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acfc5b4723bd703 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:55.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.984+0000 7f56caffd640 1 -- 172.21.15.45:0/958730103 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c402ca80 con 0x7f56cc0cb630 2023-11-28T20:34:55.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.984+0000 7f56caffd640 1 -- 172.21.15.45:0/958730103 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c402cbe0 con 0x7f56cc0cb630 2023-11-28T20:34:55.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.984+0000 7f56caffd640 1 -- 172.21.15.45:0/958730103 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c40343f0 con 0x7f56cc0cb630 2023-11-28T20:34:55.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.985+0000 7f56d1362640 1 -- 172.21.15.45:0/958730103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0cb630 msgr2=0x7f56cc0cbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.985+0000 7f56d1362640 1 --2- 172.21.15.45:0/958730103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0cb630 0x7f56cc0cbac0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0cc400 tx=0x7f56c402bee0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.985+0000 7f56d1362640 1 -- 172.21.15.45:0/958730103 shutdown_connections 2023-11-28T20:34:55.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.985+0000 7f56d1362640 1 --2- 172.21.15.45:0/958730103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0cb630 0x7f56cc0cbac0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.985+0000 7f56d1362640 1 --2- 172.21.15.45:0/958730103 >> v2:172.21.15.190:3300/0 conn(0x7f56cc0b39d0 0x7f56cc0717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.985+0000 7f56d1362640 1 -- 172.21.15.45:0/958730103 >> 172.21.15.45:0/958730103 conn(0x7f56cc0b1050 msgr2=0x7f56cc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:55.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.985+0000 7f56d1362640 1 -- 172.21.15.45:0/958730103 shutdown_connections 2023-11-28T20:34:55.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.985+0000 7f56d1362640 1 -- 172.21.15.45:0/958730103 wait complete. 2023-11-28T20:34:55.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.987+0000 7f56d1362640 1 Processor -- start 2023-11-28T20:34:55.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.987+0000 7f56d1362640 1 -- start start 2023-11-28T20:34:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.988+0000 7f56d1362640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0b39d0 0x7f56cc09c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.988+0000 7f56d1362640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56cc0b3db0 0x7f56cc06cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.988+0000 7f56d1362640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56cc0cb630 0x7f56cc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.988+0000 7f56d1362640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc14ca70 con 0x7f56cc0b39d0 2023-11-28T20:34:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.988+0000 7f56d1362640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc163e00 con 0x7f56cc0b3db0 2023-11-28T20:34:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.988+0000 7f56d1362640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc1640e0 con 0x7f56cc0cb630 2023-11-28T20:34:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0b39d0 0x7f56cc09c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0b39d0 0x7f56cc09c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36840/0 (socket says 172.21.15.45:36840) 2023-11-28T20:34:55.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cbfff640 1 -- 172.21.15.45:0/88718046 learned_addr learned my addr 172.21.15.45:0/88718046 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:55.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cb7fe640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56cc0b3db0 0x7f56cc06cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56d0b61640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56cc0cb630 0x7f56cc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cbfff640 1 -- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56cc0cb630 msgr2=0x7f56cc06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cbfff640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56cc0cb630 0x7f56cc06d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cbfff640 1 -- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56cc0b3db0 msgr2=0x7f56cc06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cbfff640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56cc0b3db0 0x7f56cc06cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cbfff640 1 -- 172.21.15.45:0/88718046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4009b30 con 0x7f56cc0b39d0 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.989+0000 7f56cb7fe640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56cc0b3db0 0x7f56cc06cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.990+0000 7f56cbfff640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0b39d0 0x7f56cc09c410 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00d800 tx=0x7f56bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.990+0000 7f56c8ff9640 1 -- 172.21.15.45:0/88718046 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00b6c0 con 0x7f56cc0b39d0 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.990+0000 7f56c8ff9640 1 -- 172.21.15.45:0/88718046 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc00b820 con 0x7f56cc0b39d0 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.990+0000 7f56c8ff9640 1 -- 172.21.15.45:0/88718046 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00bbb0 con 0x7f56cc0b39d0 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.990+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc1643c0 con 0x7f56cc0b39d0 2023-11-28T20:34:55.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.990+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc1648e0 con 0x7f56cc0b39d0 2023-11-28T20:34:55.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.992+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc165030 con 0x7f56cc0b39d0 2023-11-28T20:34:55.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.993+0000 7f56c8ff9640 1 -- 172.21.15.45:0/88718046 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f56bc00bd10 con 0x7f56cc0b39d0 2023-11-28T20:34:55.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.994+0000 7f56c8ff9640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56b806ae90 0x7f56b806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:55.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.994+0000 7f56c8ff9640 1 -- 172.21.15.45:0/88718046 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f56bc089f00 con 0x7f56cc0b39d0 2023-11-28T20:34:55.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.995+0000 7f56cb7fe640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56b806ae90 0x7f56b806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:55.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.996+0000 7f56cb7fe640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56b806ae90 0x7f56b806d350 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f56cc06dad0 tx=0x7f56c0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:56.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:55.999+0000 7f56c8ff9640 1 -- 172.21.15.45:0/88718046 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56bc054c60 con 0x7f56cc0b39d0 2023-11-28T20:34:56.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.194+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f56cc04e920 con 0x7f56cc0b39d0 2023-11-28T20:34:56.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.195+0000 7f56c8ff9640 1 -- 172.21.15.45:0/88718046 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f56bc058280 con 0x7f56cc0b39d0 2023-11-28T20:34:56.198 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56b806ae90 msgr2=0x7f56b806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56b806ae90 0x7f56b806d350 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f56cc06dad0 tx=0x7f56c0009040 comp rx=0 tx=0).stop 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0b39d0 msgr2=0x7f56cc09c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0b39d0 0x7f56cc09c410 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00d800 tx=0x7f56bc00dcc0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 shutdown_connections 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56b806ae90 0x7f56b806d350 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56cc0cb630 0x7f56cc06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56cc0b3db0 0x7f56cc06cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 --2- 172.21.15.45:0/88718046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0b39d0 0x7f56cc09c410 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 >> 172.21.15.45:0/88718046 conn(0x7f56cc0b1050 msgr2=0x7f56cc0aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:56.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.198+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 shutdown_connections 2023-11-28T20:34:56.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.199+0000 7f56d1362640 1 -- 172.21.15.45:0/88718046 wait complete. 2023-11-28T20:34:56.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:56.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.445+0000 7f5a4d059640 1 Processor -- start 2023-11-28T20:34:56.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.445+0000 7f5a4d059640 1 -- start start 2023-11-28T20:34:56.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.447+0000 7f5a4d059640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a480ae540 0x7f5a480ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.447+0000 7f5a4d059640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4815c3c0 con 0x7f5a48155780 2023-11-28T20:34:56.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.447+0000 7f5a4d059640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a48155b60 con 0x7f5a480ce630 2023-11-28T20:34:56.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.447+0000 7f5a4d059640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a480ceac0 con 0x7f5a480ae540 2023-11-28T20:34:56.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.447+0000 7f5a47fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5a48155780 0x7f5a480abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60844/0 (socket says 172.21.15.45:60844) 2023-11-28T20:34:56.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.447+0000 7f5a477fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a480ae540 0x7f5a480ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:56.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.447+0000 7f5a47fff640 1 -- 172.21.15.45:0/266769793 learned_addr learned my addr 172.21.15.45:0/266769793 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:56.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.448+0000 7f5a477fe640 1 -- 172.21.15.45:0/266769793 >> v1:172.21.15.190:6789/0 conn(0x7f5a480ce630 legacy=0x7f5a480b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:56.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.448+0000 7f5a477fe640 1 -- 172.21.15.45:0/266769793 >> v1:172.21.15.45:6789/0 conn(0x7f5a48155780 legacy=0x7f5a480abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.448+0000 7f5a477fe640 1 -- 172.21.15.45:0/266769793 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4815ba90 con 0x7f5a480ae540 2023-11-28T20:34:56.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.449+0000 7f5a477fe640 1 --2- 172.21.15.45:0/266769793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ae540 0x7f5a480ce0f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0097a0 tx=0x7f5a3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18ef7857c00a701 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:56.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.449+0000 7f5a46ffd640 1 -- 172.21.15.45:0/266769793 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c031830 con 0x7f5a480ae540 2023-11-28T20:34:56.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.450+0000 7f5a46ffd640 1 -- 172.21.15.45:0/266769793 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c031990 con 0x7f5a480ae540 2023-11-28T20:34:56.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.450+0000 7f5a46ffd640 1 -- 172.21.15.45:0/266769793 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c031d20 con 0x7f5a480ae540 2023-11-28T20:34:56.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.450+0000 7f5a4d059640 1 -- 172.21.15.45:0/266769793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ae540 msgr2=0x7f5a480ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.450+0000 7f5a4d059640 1 --2- 172.21.15.45:0/266769793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ae540 0x7f5a480ce0f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0097a0 tx=0x7f5a3c02b8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.450+0000 7f5a4d059640 1 -- 172.21.15.45:0/266769793 shutdown_connections 2023-11-28T20:34:56.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.450+0000 7f5a4d059640 1 --2- 172.21.15.45:0/266769793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ae540 0x7f5a480ce0f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.450+0000 7f5a4d059640 1 -- 172.21.15.45:0/266769793 >> 172.21.15.45:0/266769793 conn(0x7f5a4806ab30 msgr2=0x7f5a480b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:56.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.450+0000 7f5a4d059640 1 -- 172.21.15.45:0/266769793 shutdown_connections 2023-11-28T20:34:56.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.450+0000 7f5a4d059640 1 -- 172.21.15.45:0/266769793 wait complete. 2023-11-28T20:34:56.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.452+0000 7f5a4d059640 1 Processor -- start 2023-11-28T20:34:56.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.452+0000 7f5a4d059640 1 -- start start 2023-11-28T20:34:56.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.453+0000 7f5a4d059640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a480ae540 0x7f5a481f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.453+0000 7f5a4d059640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ce630 0x7f5a4814deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.453+0000 7f5a4d059640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a48155780 0x7f5a481435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.453+0000 7f5a4d059640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4809ac40 con 0x7f5a480ce630 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.453+0000 7f5a4d059640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a481f1620 con 0x7f5a48155780 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.453+0000 7f5a4d059640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4809ae20 con 0x7f5a480ae540 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.454+0000 7f5a477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ce630 0x7f5a4814deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.454+0000 7f5a477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ce630 0x7f5a4814deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36858/0 (socket says 172.21.15.45:36858) 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.454+0000 7f5a477fe640 1 -- 172.21.15.45:0/1317783393 learned_addr learned my addr 172.21.15.45:0/1317783393 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.454+0000 7f5a477fe640 1 -- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a480ae540 msgr2=0x7f5a481f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.454+0000 7f5a477fe640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a480ae540 0x7f5a481f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.454+0000 7f5a477fe640 1 -- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a48155780 msgr2=0x7f5a481435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.454+0000 7f5a477fe640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a48155780 0x7f5a481435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.454+0000 7f5a477fe640 1 -- 172.21.15.45:0/1317783393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c009400 con 0x7f5a480ce630 2023-11-28T20:34:56.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.455+0000 7f5a477fe640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ce630 0x7f5a4814deb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0098c0 tx=0x7f5a3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:56.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.455+0000 7f5a44ff9640 1 -- 172.21.15.45:0/1317783393 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c03ba70 con 0x7f5a480ce630 2023-11-28T20:34:56.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.455+0000 7f5a44ff9640 1 -- 172.21.15.45:0/1317783393 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c03bbf0 con 0x7f5a480ce630 2023-11-28T20:34:56.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.455+0000 7f5a44ff9640 1 -- 172.21.15.45:0/1317783393 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0453f0 con 0x7f5a480ce630 2023-11-28T20:34:56.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.455+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4809b100 con 0x7f5a480ce630 2023-11-28T20:34:56.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.455+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4809b6c0 con 0x7f5a480ce630 2023-11-28T20:34:56.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.456+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4809b8b0 con 0x7f5a480ce630 2023-11-28T20:34:56.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.457+0000 7f5a44ff9640 1 -- 172.21.15.45:0/1317783393 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5a3c04e450 con 0x7f5a480ce630 2023-11-28T20:34:56.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.458+0000 7f5a44ff9640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5a3406ae40 0x7f5a3406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.458+0000 7f5a44ff9640 1 -- 172.21.15.45:0/1317783393 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5a3c0abd10 con 0x7f5a480ce630 2023-11-28T20:34:56.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.459+0000 7f5a47fff640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5a3406ae40 0x7f5a3406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:56.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.460+0000 7f5a47fff640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5a3406ae40 0x7f5a3406d300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5a38001550 tx=0x7f5a38007a40 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:56.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.462+0000 7f5a44ff9640 1 -- 172.21.15.45:0/1317783393 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a3c0b1020 con 0x7f5a480ce630 2023-11-28T20:34:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.659+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a4804e920 con 0x7f5a480ce630 2023-11-28T20:34:56.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.662+0000 7f5a44ff9640 1 -- 172.21.15.45:0/1317783393 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7f5a3c035020 con 0x7f5a480ce630 2023-11-28T20:34:56.665 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:56.665 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:56.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5a3406ae40 msgr2=0x7f5a3406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5a3406ae40 0x7f5a3406d300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5a38001550 tx=0x7f5a38007a40 comp rx=0 tx=0).stop 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ce630 msgr2=0x7f5a4814deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ce630 0x7f5a4814deb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0098c0 tx=0x7f5a3c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 shutdown_connections 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5a3406ae40 0x7f5a3406d300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a48155780 0x7f5a481435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a480ce630 0x7f5a4814deb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 --2- 172.21.15.45:0/1317783393 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a480ae540 0x7f5a481f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 >> 172.21.15.45:0/1317783393 conn(0x7f5a4806ab30 msgr2=0x7f5a48152340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 shutdown_connections 2023-11-28T20:34:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.665+0000 7f5a4d059640 1 -- 172.21.15.45:0/1317783393 wait complete. 2023-11-28T20:34:56.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:34:56.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:34:56.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.914+0000 7f0117fff640 1 Processor -- start 2023-11-28T20:34:56.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.914+0000 7f0117fff640 1 -- start start 2023-11-28T20:34:56.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.917+0000 7f0117fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0118159fc0 0x7f01180b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.917+0000 7f0117fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118071ac0 0x7f01180b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.917+0000 7f0117fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01180b27a0 con 0x7f0118072010 2023-11-28T20:34:56.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.917+0000 7f0117fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0118071450 con 0x7f0118159fc0 2023-11-28T20:34:56.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.917+0000 7f0117fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011809ceb0 con 0x7f0118071ac0 2023-11-28T20:34:56.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.917+0000 7f0116ffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0118159fc0 0x7f01180b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:56.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.917+0000 7f0116ffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0118159fc0 0x7f01180b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:60306/0 (socket says 172.21.15.45:60306) 2023-11-28T20:34:56.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.917+0000 7f0116ffd640 1 -- 172.21.15.45:0/2538822310 learned_addr learned my addr 172.21.15.45:0/2538822310 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:56.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.918+0000 7f010ffff640 1 --2- 172.21.15.45:0/2538822310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118071ac0 0x7f01180b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:56.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.918+0000 7f0116ffd640 1 -- 172.21.15.45:0/2538822310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118071ac0 msgr2=0x7f01180b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:56.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.918+0000 7f0116ffd640 1 --2- 172.21.15.45:0/2538822310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118071ac0 0x7f01180b2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.918+0000 7f0116ffd640 1 -- 172.21.15.45:0/2538822310 >> v1:172.21.15.190:6789/0 conn(0x7f0118072010 legacy=0x7f011815dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.918+0000 7f0116ffd640 1 -- 172.21.15.45:0/2538822310 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01180b2cb0 con 0x7f0118159fc0 2023-11-28T20:34:56.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.920+0000 7f0116ffd640 1 --2- 172.21.15.45:0/2538822310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0118159fc0 0x7f01180b1d20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0108009870 tx=0x7f010802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca29854f2f3b55b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:56.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.920+0000 7f01167fc640 1 -- 172.21.15.45:0/2538822310 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108031640 con 0x7f0118159fc0 2023-11-28T20:34:56.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.920+0000 7f01167fc640 1 -- 172.21.15.45:0/2538822310 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01080317a0 con 0x7f0118159fc0 2023-11-28T20:34:56.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.920+0000 7f01167fc640 1 -- 172.21.15.45:0/2538822310 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108031b30 con 0x7f0118159fc0 2023-11-28T20:34:56.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.921+0000 7f0117fff640 1 -- 172.21.15.45:0/2538822310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0118159fc0 msgr2=0x7f01180b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.921+0000 7f0117fff640 1 --2- 172.21.15.45:0/2538822310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0118159fc0 0x7f01180b1d20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0108009870 tx=0x7f010802b8f0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.921+0000 7f0117fff640 1 -- 172.21.15.45:0/2538822310 shutdown_connections 2023-11-28T20:34:56.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.921+0000 7f0117fff640 1 --2- 172.21.15.45:0/2538822310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118071ac0 0x7f01180b2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.921+0000 7f0117fff640 1 --2- 172.21.15.45:0/2538822310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0118159fc0 0x7f01180b1d20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.921+0000 7f0117fff640 1 -- 172.21.15.45:0/2538822310 >> 172.21.15.45:0/2538822310 conn(0x7f01180b1050 msgr2=0x7f01180cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:56.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.921+0000 7f0117fff640 1 -- 172.21.15.45:0/2538822310 shutdown_connections 2023-11-28T20:34:56.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.921+0000 7f0117fff640 1 -- 172.21.15.45:0/2538822310 wait complete. 2023-11-28T20:34:56.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.923+0000 7f0117fff640 1 Processor -- start 2023-11-28T20:34:56.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.923+0000 7f0117fff640 1 -- start start 2023-11-28T20:34:56.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.924+0000 7f0117fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0118071ac0 0x7f0118099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.924+0000 7f0117fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118072010 0x7f011814f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.924+0000 7f0117fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0118159fc0 0x7f0118150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.924+0000 7f0117fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011815c2d0 con 0x7f0118159fc0 2023-11-28T20:34:56.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.924+0000 7f0117fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011814f6f0 con 0x7f0118071ac0 2023-11-28T20:34:56.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.924+0000 7f0117fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0118151390 con 0x7f0118072010 2023-11-28T20:34:56.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f01177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0118159fc0 0x7f0118150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:56.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f01177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0118159fc0 0x7f0118150a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36860/0 (socket says 172.21.15.45:36860) 2023-11-28T20:34:56.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f01177fe640 1 -- 172.21.15.45:0/2216799889 learned_addr learned my addr 172.21.15.45:0/2216799889 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:56.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f010ffff640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118072010 0x7f011814f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:56.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f0116ffd640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0118071ac0 0x7f0118099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:56.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f01177fe640 1 -- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118072010 msgr2=0x7f011814f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f01177fe640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118072010 0x7f011814f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f01177fe640 1 -- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0118071ac0 msgr2=0x7f0118099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:56.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f01177fe640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0118071ac0 0x7f0118099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:56.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.925+0000 7f01177fe640 1 -- 172.21.15.45:0/2216799889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080094d0 con 0x7f0118159fc0 2023-11-28T20:34:56.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.926+0000 7f01177fe640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0118159fc0 0x7f0118150a90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f011000c7d0 tx=0x7f011000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:56.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.926+0000 7f0114ff9640 1 -- 172.21.15.45:0/2216799889 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110003b70 con 0x7f0118159fc0 2023-11-28T20:34:56.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.926+0000 7f0114ff9640 1 -- 172.21.15.45:0/2216799889 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0110003cf0 con 0x7f0118159fc0 2023-11-28T20:34:56.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.926+0000 7f0114ff9640 1 -- 172.21.15.45:0/2216799889 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011000aa90 con 0x7f0118159fc0 2023-11-28T20:34:56.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.926+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0118151670 con 0x7f0118159fc0 2023-11-28T20:34:56.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.926+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011815bba0 con 0x7f0118159fc0 2023-11-28T20:34:56.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.928+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011815bd30 con 0x7f0118159fc0 2023-11-28T20:34:56.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.928+0000 7f0114ff9640 1 -- 172.21.15.45:0/2216799889 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f011000abf0 con 0x7f0118159fc0 2023-11-28T20:34:56.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.929+0000 7f0114ff9640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f00f806ae90 0x7f00f806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.930+0000 7f0114ff9640 1 -- 172.21.15.45:0/2216799889 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f01100897c0 con 0x7f0118159fc0 2023-11-28T20:34:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.930+0000 7f0116ffd640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f00f806ae90 0x7f00f806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:56.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.931+0000 7f0116ffd640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f00f806ae90 0x7f00f806d350 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0108036000 tx=0x7f010803c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:56.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:56.934+0000 7f0114ff9640 1 -- 172.21.15.45:0/2216799889 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0110054520 con 0x7f0118159fc0 2023-11-28T20:34:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.134+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f011804e920 con 0x7f0118159fc0 2023-11-28T20:34:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.135+0000 7f0114ff9640 1 -- 172.21.15.45:0/2216799889 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0110003e60 con 0x7f0118159fc0 2023-11-28T20:34:57.137 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:34:57.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f00f806ae90 msgr2=0x7f00f806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f00f806ae90 0x7f00f806d350 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0108036000 tx=0x7f010803c040 comp rx=0 tx=0).stop 2023-11-28T20:34:57.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0118159fc0 msgr2=0x7f0118150a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0118159fc0 0x7f0118150a90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f011000c7d0 tx=0x7f011000cc90 comp rx=0 tx=0).stop 2023-11-28T20:34:57.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 shutdown_connections 2023-11-28T20:34:57.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f00f806ae90 0x7f00f806d350 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0118159fc0 0x7f0118150a90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0118072010 0x7f011814f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 --2- 172.21.15.45:0/2216799889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0118071ac0 0x7f0118099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 >> 172.21.15.45:0/2216799889 conn(0x7f01180b1050 msgr2=0x7f011806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:57.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 shutdown_connections 2023-11-28T20:34:57.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.138+0000 7f0117fff640 1 -- 172.21.15.45:0/2216799889 wait complete. 2023-11-28T20:34:57.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:34:57.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.389+0000 7fe1e2581640 1 Processor -- start 2023-11-28T20:34:57.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.389+0000 7fe1e2581640 1 -- start start 2023-11-28T20:34:57.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.391+0000 7fe1e2581640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe1dc15d4b0 0x7fe1dc070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.391+0000 7fe1e2581640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe1dc0712a0 0x7fe1dc0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.391+0000 7fe1e2581640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc0708d0 con 0x7fe1dc0adfa0 2023-11-28T20:34:57.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.391+0000 7fe1e2581640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b3bb0 con 0x7fe1dc0712a0 2023-11-28T20:34:57.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.391+0000 7fe1e2581640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b3e90 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.391+0000 7fe1e0d7e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe1dc0712a0 0x7fe1dc0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.391+0000 7fe1e0d7e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe1dc0712a0 0x7fe1dc0abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36866/0 (socket says 172.21.15.45:36866) 2023-11-28T20:34:57.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.391+0000 7fe1e0d7e640 1 -- 172.21.15.45:0/3793492784 learned_addr learned my addr 172.21.15.45:0/3793492784 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:57.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.392+0000 7fe1e157f640 1 --2- 172.21.15.45:0/3793492784 >> v2:172.21.15.190:3300/0 conn(0x7fe1dc15d4b0 0x7fe1dc070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.392+0000 7fe1e0d7e640 1 -- 172.21.15.45:0/3793492784 >> v2:172.21.15.190:3300/0 conn(0x7fe1dc15d4b0 msgr2=0x7fe1dc070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:57.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.392+0000 7fe1e0d7e640 1 --2- 172.21.15.45:0/3793492784 >> v2:172.21.15.190:3300/0 conn(0x7fe1dc15d4b0 0x7fe1dc070d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.392+0000 7fe1e0d7e640 1 -- 172.21.15.45:0/3793492784 >> v1:172.21.15.190:6789/0 conn(0x7fe1dc0adfa0 legacy=0x7fe1dc0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:57.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.392+0000 7fe1e0d7e640 1 -- 172.21.15.45:0/3793492784 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc070280 con 0x7fe1dc0712a0 2023-11-28T20:34:57.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.393+0000 7fe1e0d7e640 1 --2- 172.21.15.45:0/3793492784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc0712a0 0x7fe1dc0abd70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0ce4a0 tx=0x7fe1cc02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd89d2e6f10106f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:57.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.393+0000 7fe1d3fff640 1 -- 172.21.15.45:0/3793492784 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc031800 con 0x7fe1dc0712a0 2023-11-28T20:34:57.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.393+0000 7fe1d3fff640 1 -- 172.21.15.45:0/3793492784 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc031960 con 0x7fe1dc0712a0 2023-11-28T20:34:57.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.393+0000 7fe1d3fff640 1 -- 172.21.15.45:0/3793492784 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc031cf0 con 0x7fe1dc0712a0 2023-11-28T20:34:57.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.394+0000 7fe1e2581640 1 -- 172.21.15.45:0/3793492784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc0712a0 msgr2=0x7fe1dc0abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.394+0000 7fe1e2581640 1 --2- 172.21.15.45:0/3793492784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc0712a0 0x7fe1dc0abd70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0ce4a0 tx=0x7fe1cc02b8a0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.394+0000 7fe1e2581640 1 -- 172.21.15.45:0/3793492784 shutdown_connections 2023-11-28T20:34:57.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.394+0000 7fe1e2581640 1 --2- 172.21.15.45:0/3793492784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc0712a0 0x7fe1dc0abd70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.394+0000 7fe1e2581640 1 --2- 172.21.15.45:0/3793492784 >> v2:172.21.15.190:3300/0 conn(0x7fe1dc15d4b0 0x7fe1dc070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.394+0000 7fe1e2581640 1 -- 172.21.15.45:0/3793492784 >> 172.21.15.45:0/3793492784 conn(0x7fe1dc0b1050 msgr2=0x7fe1dc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:57.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.394+0000 7fe1e2581640 1 -- 172.21.15.45:0/3793492784 shutdown_connections 2023-11-28T20:34:57.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.394+0000 7fe1e2581640 1 -- 172.21.15.45:0/3793492784 wait complete. 2023-11-28T20:34:57.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.396+0000 7fe1e2581640 1 Processor -- start 2023-11-28T20:34:57.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.396+0000 7fe1e2581640 1 -- start start 2023-11-28T20:34:57.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.397+0000 7fe1e2581640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1dc0712a0 0x7fe1dc159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.397+0000 7fe1e2581640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1dc0adfa0 0x7fe1dc09a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.397+0000 7fe1e2581640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc15d4b0 0x7fe1dc15fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.397+0000 7fe1e2581640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc09aea0 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.397+0000 7fe1e2581640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc09b180 con 0x7fe1dc0712a0 2023-11-28T20:34:57.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.397+0000 7fe1e2581640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc15daa0 con 0x7fe1dc0adfa0 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e1d80640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc15d4b0 0x7fe1dc15fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e1d80640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc15d4b0 0x7fe1dc15fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36872/0 (socket says 172.21.15.45:36872) 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e1d80640 1 -- 172.21.15.45:0/2071835872 learned_addr learned my addr 172.21.15.45:0/2071835872 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e157f640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1dc0712a0 0x7fe1dc159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e0d7e640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1dc0adfa0 0x7fe1dc09a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e1d80640 1 -- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1dc0adfa0 msgr2=0x7fe1dc09a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e1d80640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1dc0adfa0 0x7fe1dc09a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e1d80640 1 -- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1dc0712a0 msgr2=0x7fe1dc159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e1d80640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1dc0712a0 0x7fe1dc159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.398+0000 7fe1e1d80640 1 -- 172.21.15.45:0/2071835872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc0094d0 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.399+0000 7fe1e1d80640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc15d4b0 0x7fe1dc15fb40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400b9e0 tx=0x7fe1d400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:57.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.399+0000 7fe1d1ffb640 1 -- 172.21.15.45:0/2071835872 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d400cb20 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.399+0000 7fe1d1ffb640 1 -- 172.21.15.45:0/2071835872 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d400cc80 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.399+0000 7fe1d1ffb640 1 -- 172.21.15.45:0/2071835872 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4014680 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.399+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc15dd80 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.399+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc159b60 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.400+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc09d0f0 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.401+0000 7fe1d1ffb640 1 -- 172.21.15.45:0/2071835872 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe1d401d3b0 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.402+0000 7fe1d1ffb640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c006ae40 0x7fe1c006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.402+0000 7fe1d1ffb640 1 -- 172.21.15.45:0/2071835872 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe1d4089930 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.403+0000 7fe1e157f640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c006ae40 0x7fe1c006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.404+0000 7fe1e157f640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c006ae40 0x7fe1c006d300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe1d800a970 tx=0x7fe1d8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:57.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.406+0000 7fe1d1ffb640 1 -- 172.21.15.45:0/2071835872 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1d4054610 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.599+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe1dc04e920 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.600+0000 7fe1d1ffb640 1 -- 172.21.15.45:0/2071835872 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe1d4019020 con 0x7fe1dc15d4b0 2023-11-28T20:34:57.602 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:57.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c006ae40 msgr2=0x7fe1c006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c006ae40 0x7fe1c006d300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe1d800a970 tx=0x7fe1d8008040 comp rx=0 tx=0).stop 2023-11-28T20:34:57.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc15d4b0 msgr2=0x7fe1dc15fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc15d4b0 0x7fe1dc15fb40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400b9e0 tx=0x7fe1d400bea0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 shutdown_connections 2023-11-28T20:34:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe1c006ae40 0x7fe1c006d300 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1dc15d4b0 0x7fe1dc15fb40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1dc0adfa0 0x7fe1dc09a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 --2- 172.21.15.45:0/2071835872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1dc0712a0 0x7fe1dc159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 >> 172.21.15.45:0/2071835872 conn(0x7fe1dc0b1050 msgr2=0x7fe1dc15a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 shutdown_connections 2023-11-28T20:34:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.603+0000 7fe1e2581640 1 -- 172.21.15.45:0/2071835872 wait complete. 2023-11-28T20:34:57.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:34:57.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.844+0000 7f5b37f05640 1 Processor -- start 2023-11-28T20:34:57.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.844+0000 7f5b37f05640 1 -- start start 2023-11-28T20:34:57.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.846+0000 7f5b37f05640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5b300ce600 0x7f5b3006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.846+0000 7f5b37f05640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b3006aa40 0x7f5b300c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.846+0000 7f5b37f05640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3006e030 con 0x7f5b3006dc50 2023-11-28T20:34:57.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.846+0000 7f5b37f05640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b3006aed0 con 0x7f5b3006aa40 2023-11-28T20:34:57.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.846+0000 7f5b37f05640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b300ccf60 con 0x7f5b300ce600 2023-11-28T20:34:57.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.846+0000 7f5b364a2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b3006aa40 0x7f5b300c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.846+0000 7f5b35ca1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5b3006dc50 0x7f5b300cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60854/0 (socket says 172.21.15.45:60854) 2023-11-28T20:34:57.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.846+0000 7f5b35ca1640 1 -- 172.21.15.45:0/2772572052 learned_addr learned my addr 172.21.15.45:0/2772572052 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:57.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.847+0000 7f5b354a0640 1 --2- 172.21.15.45:0/2772572052 >> v2:172.21.15.190:3300/0 conn(0x7f5b300ce600 0x7f5b3006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.847+0000 7f5b34c9f640 1 -- 172.21.15.45:0/2772572052 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435662207 0 0) 0x7f5b2c001490 con 0x7f5b3006dc50 2023-11-28T20:34:57.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.847+0000 7f5b34c9f640 1 -- 172.21.15.45:0/2772572052 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c003520 con 0x7f5b3006dc50 2023-11-28T20:34:57.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.847+0000 7f5b364a2640 1 -- 172.21.15.45:0/2772572052 >> v2:172.21.15.190:3300/0 conn(0x7f5b300ce600 msgr2=0x7f5b3006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:57.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.847+0000 7f5b364a2640 1 --2- 172.21.15.45:0/2772572052 >> v2:172.21.15.190:3300/0 conn(0x7f5b300ce600 0x7f5b3006a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.847+0000 7f5b364a2640 1 -- 172.21.15.45:0/2772572052 >> v1:172.21.15.45:6789/0 conn(0x7f5b3006dc50 legacy=0x7f5b300cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.847+0000 7f5b364a2640 1 -- 172.21.15.45:0/2772572052 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300cc930 con 0x7f5b3006aa40 2023-11-28T20:34:57.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.848+0000 7f5b364a2640 1 --2- 172.21.15.45:0/2772572052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006aa40 0x7f5b300c93b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b3009d0c0 tx=0x7f5b2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f967041dcee65ca2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:57.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.849+0000 7f5b34c9f640 1 -- 172.21.15.45:0/2772572052 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28031830 con 0x7f5b3006aa40 2023-11-28T20:34:57.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.849+0000 7f5b34c9f640 1 -- 172.21.15.45:0/2772572052 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b28031990 con 0x7f5b3006aa40 2023-11-28T20:34:57.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.849+0000 7f5b34c9f640 1 -- 172.21.15.45:0/2772572052 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28031d20 con 0x7f5b3006aa40 2023-11-28T20:34:57.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.849+0000 7f5b37f05640 1 -- 172.21.15.45:0/2772572052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006aa40 msgr2=0x7f5b300c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.849+0000 7f5b37f05640 1 --2- 172.21.15.45:0/2772572052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006aa40 0x7f5b300c93b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b3009d0c0 tx=0x7f5b2802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.850+0000 7f5b37f05640 1 -- 172.21.15.45:0/2772572052 shutdown_connections 2023-11-28T20:34:57.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.850+0000 7f5b37f05640 1 --2- 172.21.15.45:0/2772572052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006aa40 0x7f5b300c93b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.850+0000 7f5b37f05640 1 --2- 172.21.15.45:0/2772572052 >> v2:172.21.15.190:3300/0 conn(0x7f5b300ce600 0x7f5b3006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.850+0000 7f5b37f05640 1 -- 172.21.15.45:0/2772572052 >> 172.21.15.45:0/2772572052 conn(0x7f5b300ced60 msgr2=0x7f5b300cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:57.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.850+0000 7f5b37f05640 1 -- 172.21.15.45:0/2772572052 shutdown_connections 2023-11-28T20:34:57.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.850+0000 7f5b37f05640 1 -- 172.21.15.45:0/2772572052 wait complete. 2023-11-28T20:34:57.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.852+0000 7f5b37f05640 1 Processor -- start 2023-11-28T20:34:57.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.852+0000 7f5b37f05640 1 -- start start 2023-11-28T20:34:57.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b37f05640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b3006aa40 0x7f5b3014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b37f05640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006dc50 0x7f5b300c2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b37f05640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b300ce600 0x7f5b3015e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b37f05640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300c3da0 con 0x7f5b3006dc50 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b37f05640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3014e730 con 0x7f5b300ce600 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b37f05640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300c3190 con 0x7f5b3006aa40 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b354a0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006dc50 0x7f5b300c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b354a0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006dc50 0x7f5b300c2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36890/0 (socket says 172.21.15.45:36890) 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b354a0640 1 -- 172.21.15.45:0/2467874845 learned_addr learned my addr 172.21.15.45:0/2467874845 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.853+0000 7f5b364a2640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b300ce600 0x7f5b3015e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b35ca1640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b3006aa40 0x7f5b3014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b354a0640 1 -- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b3006aa40 msgr2=0x7f5b3014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b354a0640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b3006aa40 0x7f5b3014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b354a0640 1 -- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b300ce600 msgr2=0x7f5b3015e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:57.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b354a0640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b300ce600 0x7f5b3015e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:57.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b354a0640 1 -- 172.21.15.45:0/2467874845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b28009400 con 0x7f5b3006dc50 2023-11-28T20:34:57.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b354a0640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006dc50 0x7f5b300c2890 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000d5f0 tx=0x7f5b2000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:57.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b267fc640 1 -- 172.21.15.45:0/2467874845 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b200125f0 con 0x7f5b3006dc50 2023-11-28T20:34:57.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b267fc640 1 -- 172.21.15.45:0/2467874845 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b20012750 con 0x7f5b3006dc50 2023-11-28T20:34:57.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b300c3470 con 0x7f5b3006dc50 2023-11-28T20:34:57.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b267fc640 1 -- 172.21.15.45:0/2467874845 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b20012b00 con 0x7f5b3006dc50 2023-11-28T20:34:57.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.854+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b300c3a30 con 0x7f5b3006dc50 2023-11-28T20:34:57.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.856+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b300af740 con 0x7f5b3006dc50 2023-11-28T20:34:57.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.857+0000 7f5b267fc640 1 -- 172.21.15.45:0/2467874845 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5b20012c60 con 0x7f5b3006dc50 2023-11-28T20:34:57.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.858+0000 7f5b267fc640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5b1406ae40 0x7f5b1406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:57.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.859+0000 7f5b267fc640 1 -- 172.21.15.45:0/2467874845 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5b20089f60 con 0x7f5b3006dc50 2023-11-28T20:34:57.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.859+0000 7f5b35ca1640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5b1406ae40 0x7f5b1406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:57.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.860+0000 7f5b35ca1640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5b1406ae40 0x7f5b1406d300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0015f0 tx=0x7f5b2c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:57.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:57.863+0000 7f5b267fc640 1 -- 172.21.15.45:0/2467874845 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b20054bb0 con 0x7f5b3006dc50 2023-11-28T20:34:58.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.064+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b300733d0 con 0x7f5b3006dc50 2023-11-28T20:34:58.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.065+0000 7f5b267fc640 1 -- 172.21.15.45:0/2467874845 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b200581d0 con 0x7f5b3006dc50 2023-11-28T20:34:58.067 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:34:58.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5b1406ae40 msgr2=0x7f5b1406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5b1406ae40 0x7f5b1406d300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0015f0 tx=0x7f5b2c009040 comp rx=0 tx=0).stop 2023-11-28T20:34:58.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006dc50 msgr2=0x7f5b300c2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006dc50 0x7f5b300c2890 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000d5f0 tx=0x7f5b2000dab0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 shutdown_connections 2023-11-28T20:34:58.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5b1406ae40 0x7f5b1406d300 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b300ce600 0x7f5b3015e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3006dc50 0x7f5b300c2890 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 --2- 172.21.15.45:0/2467874845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b3006aa40 0x7f5b3014e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 >> 172.21.15.45:0/2467874845 conn(0x7f5b300ced60 msgr2=0x7f5b300ae710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:58.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 shutdown_connections 2023-11-28T20:34:58.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.068+0000 7f5b37f05640 1 -- 172.21.15.45:0/2467874845 wait complete. 2023-11-28T20:34:58.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:34:58.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.316+0000 7f667d343640 1 Processor -- start 2023-11-28T20:34:58.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.316+0000 7f667d343640 1 -- start start 2023-11-28T20:34:58.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.317+0000 7f667d343640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f667815d7a0 0x7f66780cda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.317+0000 7f667d343640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f66780d02d0 0x7f66780b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.317+0000 7f667d343640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780d2e70 con 0x7f667815d3c0 2023-11-28T20:34:58.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.317+0000 7f667d343640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66780b6920 con 0x7f66780d02d0 2023-11-28T20:34:58.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.317+0000 7f667d343640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780ce110 con 0x7f667815d7a0 2023-11-28T20:34:58.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.317+0000 7f66777fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f66780d02d0 0x7f66780b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:58.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.318+0000 7f66777fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f66780d02d0 0x7f66780b63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36902/0 (socket says 172.21.15.45:36902) 2023-11-28T20:34:58.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.318+0000 7f66777fe640 1 -- 172.21.15.45:0/293641161 learned_addr learned my addr 172.21.15.45:0/293641161 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:58.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.318+0000 7f66767fc640 1 --2- 172.21.15.45:0/293641161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f667815d7a0 0x7f66780cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:58.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.318+0000 7f66777fe640 1 -- 172.21.15.45:0/293641161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f667815d7a0 msgr2=0x7f66780cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:58.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.318+0000 7f66777fe640 1 --2- 172.21.15.45:0/293641161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f667815d7a0 0x7f66780cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.318+0000 7f66777fe640 1 -- 172.21.15.45:0/293641161 >> v1:172.21.15.190:6789/0 conn(0x7f667815d3c0 legacy=0x7f66780cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:58.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.318+0000 7f66777fe640 1 -- 172.21.15.45:0/293641161 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780ce3f0 con 0x7f66780d02d0 2023-11-28T20:34:58.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.318+0000 7f66767fc640 1 --2- 172.21.15.45:0/293641161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f667815d7a0 0x7f66780cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:58.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.319+0000 7f66777fe640 1 --2- 172.21.15.45:0/293641161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66780d02d0 0x7f66780b63e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6670009870 tx=0x7f667002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b363ab9fdbc821b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:58.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.319+0000 7f6675ffb640 1 -- 172.21.15.45:0/293641161 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66700317c0 con 0x7f66780d02d0 2023-11-28T20:34:58.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.319+0000 7f6675ffb640 1 -- 172.21.15.45:0/293641161 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6670031920 con 0x7f66780d02d0 2023-11-28T20:34:58.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.319+0000 7f6675ffb640 1 -- 172.21.15.45:0/293641161 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670031cb0 con 0x7f66780d02d0 2023-11-28T20:34:58.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.320+0000 7f667d343640 1 -- 172.21.15.45:0/293641161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66780d02d0 msgr2=0x7f66780b63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.320+0000 7f667d343640 1 --2- 172.21.15.45:0/293641161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66780d02d0 0x7f66780b63e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6670009870 tx=0x7f667002b860 comp rx=0 tx=0).stop 2023-11-28T20:34:58.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.320+0000 7f667d343640 1 -- 172.21.15.45:0/293641161 shutdown_connections 2023-11-28T20:34:58.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.320+0000 7f667d343640 1 --2- 172.21.15.45:0/293641161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66780d02d0 0x7f66780b63e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.320+0000 7f667d343640 1 --2- 172.21.15.45:0/293641161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f667815d7a0 0x7f66780cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.320+0000 7f667d343640 1 -- 172.21.15.45:0/293641161 >> 172.21.15.45:0/293641161 conn(0x7f66780b4ca0 msgr2=0x7f66780b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:58.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.320+0000 7f667d343640 1 -- 172.21.15.45:0/293641161 shutdown_connections 2023-11-28T20:34:58.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.320+0000 7f667d343640 1 -- 172.21.15.45:0/293641161 wait complete. 2023-11-28T20:34:58.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.322+0000 7f667d343640 1 Processor -- start 2023-11-28T20:34:58.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.322+0000 7f667d343640 1 -- start start 2023-11-28T20:34:58.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.323+0000 7f667d343640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66780d02d0 0x7f66781f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.323+0000 7f667d343640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667815d3c0 0x7f66780ac150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.323+0000 7f667d343640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66780cbbe0 0x7f66780cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.323+0000 7f667d343640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780aa540 con 0x7f667815d3c0 2023-11-28T20:34:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.323+0000 7f667d343640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780cc660 con 0x7f66780cbbe0 2023-11-28T20:34:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.323+0000 7f667d343640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780a9c10 con 0x7f66780d02d0 2023-11-28T20:34:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.323+0000 7f66767fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667815d3c0 0x7f66780ac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.323+0000 7f66767fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667815d3c0 0x7f66780ac150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36910/0 (socket says 172.21.15.45:36910) 2023-11-28T20:34:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.323+0000 7f66767fc640 1 -- 172.21.15.45:0/1670509002 learned_addr learned my addr 172.21.15.45:0/1670509002 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:58.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.324+0000 7f66767fc640 1 -- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66780d02d0 msgr2=0x7f66781f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.324+0000 7f66767fc640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66780d02d0 0x7f66781f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.324+0000 7f66767fc640 1 -- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66780cbbe0 msgr2=0x7f66780cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.324+0000 7f66767fc640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66780cbbe0 0x7f66780cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.324+0000 7f66767fc640 1 -- 172.21.15.45:0/1670509002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66700094d0 con 0x7f667815d3c0 2023-11-28T20:34:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.324+0000 7f66767fc640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667815d3c0 0x7f66780ac150 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f666000c7f0 tx=0x7f666000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.324+0000 7f666f7fe640 1 -- 172.21.15.45:0/1670509002 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660007820 con 0x7f667815d3c0 2023-11-28T20:34:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.325+0000 7f666f7fe640 1 -- 172.21.15.45:0/1670509002 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6660007980 con 0x7f667815d3c0 2023-11-28T20:34:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.325+0000 7f666f7fe640 1 -- 172.21.15.45:0/1670509002 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660007cf0 con 0x7f667815d3c0 2023-11-28T20:34:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.325+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66780a9ef0 con 0x7f667815d3c0 2023-11-28T20:34:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.325+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66780caed0 con 0x7f667815d3c0 2023-11-28T20:34:58.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.326+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6678161070 con 0x7f667815d3c0 2023-11-28T20:34:58.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.327+0000 7f666f7fe640 1 -- 172.21.15.45:0/1670509002 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f66600113f0 con 0x7f667815d3c0 2023-11-28T20:34:58.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.328+0000 7f666f7fe640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f665806ae40 0x7f665806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.328+0000 7f666f7fe640 1 -- 172.21.15.45:0/1670509002 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6660089d80 con 0x7f667815d3c0 2023-11-28T20:34:58.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.329+0000 7f6676ffd640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f665806ae40 0x7f665806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.330+0000 7f6676ffd640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f665806ae40 0x7f665806d300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f66680043c0 tx=0x7f6668007d30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.332+0000 7f666f7fe640 1 -- 172.21.15.45:0/1670509002 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6678161070 con 0x7f667815d3c0 2023-11-28T20:34:58.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.528+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6678161070 con 0x7f667815d3c0 2023-11-28T20:34:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.529+0000 7f666f7fe640 1 -- 172.21.15.45:0/1670509002 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6678161070 con 0x7f667815d3c0 2023-11-28T20:34:58.531 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:58.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f665806ae40 msgr2=0x7f665806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f665806ae40 0x7f665806d300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f66680043c0 tx=0x7f6668007d30 comp rx=0 tx=0).stop 2023-11-28T20:34:58.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667815d3c0 msgr2=0x7f66780ac150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667815d3c0 0x7f66780ac150 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f666000c7f0 tx=0x7f666000ccb0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 shutdown_connections 2023-11-28T20:34:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f665806ae40 0x7f665806d300 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66780cbbe0 0x7f66780cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667815d3c0 0x7f66780ac150 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 --2- 172.21.15.45:0/1670509002 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66780d02d0 0x7f66781f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 >> 172.21.15.45:0/1670509002 conn(0x7f66780b4ca0 msgr2=0x7f66780b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 shutdown_connections 2023-11-28T20:34:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.532+0000 7f667d343640 1 -- 172.21.15.45:0/1670509002 wait complete. 2023-11-28T20:34:58.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:34:58.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.778+0000 7f8696b16640 1 Processor -- start 2023-11-28T20:34:58.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.778+0000 7f8696b16640 1 -- start start 2023-11-28T20:34:58.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.780+0000 7f8696b16640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f86900ae540 0x7f86900ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.780+0000 7f8696b16640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869015c3c0 con 0x7f8690155780 2023-11-28T20:34:58.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.780+0000 7f8696b16640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8690155b60 con 0x7f86900ce630 2023-11-28T20:34:58.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.780+0000 7f8696b16640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86900ceac0 con 0x7f86900ae540 2023-11-28T20:34:58.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.781+0000 7f8695313640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f86900ae540 0x7f86900ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:58.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.781+0000 7f8695b14640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8690155780 0x7f86900abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60856/0 (socket says 172.21.15.45:60856) 2023-11-28T20:34:58.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.781+0000 7f8695b14640 1 -- 172.21.15.45:0/2105307681 learned_addr learned my addr 172.21.15.45:0/2105307681 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:58.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.781+0000 7f8694b12640 1 -- 172.21.15.45:0/2105307681 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138346797 0 0) 0x7f869015c3c0 con 0x7f8690155780 2023-11-28T20:34:58.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.781+0000 7f8695313640 1 -- 172.21.15.45:0/2105307681 >> v1:172.21.15.190:6789/0 conn(0x7f86900ce630 legacy=0x7f86900b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:58.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.781+0000 7f8695313640 1 -- 172.21.15.45:0/2105307681 >> v1:172.21.15.45:6789/0 conn(0x7f8690155780 legacy=0x7f86900abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.782+0000 7f8695313640 1 -- 172.21.15.45:0/2105307681 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869015ba90 con 0x7f86900ae540 2023-11-28T20:34:58.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.783+0000 7f8695313640 1 --2- 172.21.15.45:0/2105307681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ae540 0x7f86900ce0f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f86800097a0 tx=0x7f868002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622e8c9233650528 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:58.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.783+0000 7f8694b12640 1 -- 172.21.15.45:0/2105307681 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8680030610 con 0x7f86900ae540 2023-11-28T20:34:58.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.783+0000 7f8694b12640 1 -- 172.21.15.45:0/2105307681 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8680030770 con 0x7f86900ae540 2023-11-28T20:34:58.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.783+0000 7f8694b12640 1 -- 172.21.15.45:0/2105307681 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8680030b00 con 0x7f86900ae540 2023-11-28T20:34:58.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.784+0000 7f8696b16640 1 -- 172.21.15.45:0/2105307681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ae540 msgr2=0x7f86900ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.784+0000 7f8696b16640 1 --2- 172.21.15.45:0/2105307681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ae540 0x7f86900ce0f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f86800097a0 tx=0x7f868002b960 comp rx=0 tx=0).stop 2023-11-28T20:34:58.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.784+0000 7f8696b16640 1 -- 172.21.15.45:0/2105307681 shutdown_connections 2023-11-28T20:34:58.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.784+0000 7f8696b16640 1 --2- 172.21.15.45:0/2105307681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ae540 0x7f86900ce0f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.784+0000 7f8696b16640 1 -- 172.21.15.45:0/2105307681 >> 172.21.15.45:0/2105307681 conn(0x7f869006ab30 msgr2=0x7f86900b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:58.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.784+0000 7f8696b16640 1 -- 172.21.15.45:0/2105307681 shutdown_connections 2023-11-28T20:34:58.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.784+0000 7f8696b16640 1 -- 172.21.15.45:0/2105307681 wait complete. 2023-11-28T20:34:58.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.786+0000 7f8696b16640 1 Processor -- start 2023-11-28T20:34:58.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.786+0000 7f8696b16640 1 -- start start 2023-11-28T20:34:58.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8696b16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86900ae540 0x7f869013fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8696b16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ce630 0x7f869006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8696b16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8690155780 0x7f869006c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8696b16640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86901e5640 con 0x7f86900ce630 2023-11-28T20:34:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8696b16640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8690143700 con 0x7f86900ae540 2023-11-28T20:34:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8696b16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869009a9a0 con 0x7f8690155780 2023-11-28T20:34:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8695313640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ce630 0x7f869006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8695313640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ce630 0x7f869006c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36940/0 (socket says 172.21.15.45:36940) 2023-11-28T20:34:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8695313640 1 -- 172.21.15.45:0/237238084 learned_addr learned my addr 172.21.15.45:0/237238084 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8695b14640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86900ae540 0x7f869013fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.787+0000 7f8696315640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8690155780 0x7f869006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.788+0000 7f8695313640 1 -- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8690155780 msgr2=0x7f869006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.788+0000 7f8695313640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8690155780 0x7f869006c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.788+0000 7f8695313640 1 -- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86900ae540 msgr2=0x7f869013fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:58.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.788+0000 7f8695313640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86900ae540 0x7f869013fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.788+0000 7f8695313640 1 -- 172.21.15.45:0/237238084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680009400 con 0x7f86900ce630 2023-11-28T20:34:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.788+0000 7f8695313640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ce630 0x7f869006c2c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f868002bf70 tx=0x7f8680004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.788+0000 7f86867fc640 1 -- 172.21.15.45:0/237238084 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868003e040 con 0x7f86900ce630 2023-11-28T20:34:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.789+0000 7f86867fc640 1 -- 172.21.15.45:0/237238084 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8680039cb0 con 0x7f86900ce630 2023-11-28T20:34:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.789+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8690099be0 con 0x7f86900ce630 2023-11-28T20:34:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.789+0000 7f86867fc640 1 -- 172.21.15.45:0/237238084 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8680044630 con 0x7f86900ce630 2023-11-28T20:34:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.789+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869009a1a0 con 0x7f86900ce630 2023-11-28T20:34:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.790+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869004e920 con 0x7f86900ce630 2023-11-28T20:34:58.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.791+0000 7f86867fc640 1 -- 172.21.15.45:0/237238084 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f868004d3b0 con 0x7f86900ce630 2023-11-28T20:34:58.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.792+0000 7f86867fc640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f867406ae40 0x7f867406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:58.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.792+0000 7f86867fc640 1 -- 172.21.15.45:0/237238084 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f86800abff0 con 0x7f86900ce630 2023-11-28T20:34:58.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.793+0000 7f8695b14640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f867406ae40 0x7f867406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:58.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.794+0000 7f8695b14640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f867406ae40 0x7f867406d300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f868c000f60 tx=0x7f868c007d00 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:58.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.796+0000 7f86867fc640 1 -- 172.21.15.45:0/237238084 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8680076ca0 con 0x7f86900ce630 2023-11-28T20:34:58.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.993+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f869015cbb0 con 0x7f86900ce630 2023-11-28T20:34:58.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.995+0000 7f86867fc640 1 -- 172.21.15.45:0/237238084 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7f868007a2c0 con 0x7f86900ce630 2023-11-28T20:34:58.998 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:34:58.998 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f867406ae40 msgr2=0x7f867406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f867406ae40 0x7f867406d300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f868c000f60 tx=0x7f868c007d00 comp rx=0 tx=0).stop 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ce630 msgr2=0x7f869006c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ce630 0x7f869006c2c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f868002bf70 tx=0x7f8680004730 comp rx=0 tx=0).stop 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 shutdown_connections 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f867406ae40 0x7f867406d300 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8690155780 0x7f869006c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86900ce630 0x7f869006c2c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 --2- 172.21.15.45:0/237238084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86900ae540 0x7f869013fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 >> 172.21.15.45:0/237238084 conn(0x7f869006ab30 msgr2=0x7f86900b3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:59.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 shutdown_connections 2023-11-28T20:34:59.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:58.998+0000 7f8696b16640 1 -- 172.21.15.45:0/237238084 wait complete. 2023-11-28T20:34:59.021 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:34:59.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:34:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.249+0000 7effc3fff640 1 Processor -- start 2023-11-28T20:34:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.249+0000 7effc3fff640 1 -- start start 2023-11-28T20:34:59.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.251+0000 7effc3fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7effc415b270 0x7effc406cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.251+0000 7effc3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc40b3420 0x7effc40ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.251+0000 7effc3fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc406d410 con 0x7effc40b3040 2023-11-28T20:34:59.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.251+0000 7effc3fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc40ceaa0 con 0x7effc415b270 2023-11-28T20:34:59.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.251+0000 7effc3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc406d6f0 con 0x7effc40b3420 2023-11-28T20:34:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.252+0000 7effc27fc640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7effc415b270 0x7effc406cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.252+0000 7effc37fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc40b3420 0x7effc40ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.252+0000 7effc27fc640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7effc415b270 0x7effc406cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:60400/0 (socket says 172.21.15.45:60400) 2023-11-28T20:34:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.252+0000 7effc27fc640 1 -- 172.21.15.45:0/1415303565 learned_addr learned my addr 172.21.15.45:0/1415303565 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:59.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.252+0000 7effc27fc640 1 -- 172.21.15.45:0/1415303565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc40b3420 msgr2=0x7effc40ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:59.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.252+0000 7effc27fc640 1 --2- 172.21.15.45:0/1415303565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc40b3420 0x7effc40ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.253+0000 7effc27fc640 1 -- 172.21.15.45:0/1415303565 >> v1:172.21.15.190:6789/0 conn(0x7effc40b3040 legacy=0x7effc415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.253+0000 7effc27fc640 1 -- 172.21.15.45:0/1415303565 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc406d9d0 con 0x7effc415b270 2023-11-28T20:34:59.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.253+0000 7effc1ffb640 1 -- 172.21.15.45:0/1415303565 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520794388 0 0) 0x7effc406d410 con 0x7effc40b3040 2023-11-28T20:34:59.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.254+0000 7effc27fc640 1 --2- 172.21.15.45:0/1415303565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc415b270 0x7effc406cda0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7effac009870 tx=0x7effac02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72037ea8bed723f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:59.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.254+0000 7effc1ffb640 1 -- 172.21.15.45:0/1415303565 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac0317c0 con 0x7effc415b270 2023-11-28T20:34:59.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.255+0000 7effc1ffb640 1 -- 172.21.15.45:0/1415303565 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac031920 con 0x7effc415b270 2023-11-28T20:34:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.255+0000 7effc1ffb640 1 -- 172.21.15.45:0/1415303565 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac031cb0 con 0x7effc415b270 2023-11-28T20:34:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.255+0000 7effc3fff640 1 -- 172.21.15.45:0/1415303565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc415b270 msgr2=0x7effc406cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.255+0000 7effc3fff640 1 --2- 172.21.15.45:0/1415303565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc415b270 0x7effc406cda0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7effac009870 tx=0x7effac02b860 comp rx=0 tx=0).stop 2023-11-28T20:34:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.255+0000 7effc3fff640 1 -- 172.21.15.45:0/1415303565 shutdown_connections 2023-11-28T20:34:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.255+0000 7effc3fff640 1 --2- 172.21.15.45:0/1415303565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc40b3420 0x7effc40ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.255+0000 7effc3fff640 1 --2- 172.21.15.45:0/1415303565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc415b270 0x7effc406cda0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.255+0000 7effc3fff640 1 -- 172.21.15.45:0/1415303565 >> 172.21.15.45:0/1415303565 conn(0x7effc40b1050 msgr2=0x7effc40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.255+0000 7effc3fff640 1 -- 172.21.15.45:0/1415303565 shutdown_connections 2023-11-28T20:34:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.256+0000 7effc3fff640 1 -- 172.21.15.45:0/1415303565 wait complete. 2023-11-28T20:34:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.257+0000 7effc3fff640 1 Processor -- start 2023-11-28T20:34:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.257+0000 7effc3fff640 1 -- start start 2023-11-28T20:34:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.258+0000 7effc3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc40b3040 0x7effc409acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.258+0000 7effc3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc40b3420 0x7effc414d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.258+0000 7effc3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc415b270 0x7effc415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.258+0000 7effc3fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc409b3d0 con 0x7effc40b3040 2023-11-28T20:34:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.258+0000 7effc3fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc409b6b0 con 0x7effc40b3420 2023-11-28T20:34:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.258+0000 7effc3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc409b990 con 0x7effc415b270 2023-11-28T20:34:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc40b3040 0x7effc409acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc40b3040 0x7effc409acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36950/0 (socket says 172.21.15.45:36950) 2023-11-28T20:34:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc2ffd640 1 -- 172.21.15.45:0/2036522723 learned_addr learned my addr 172.21.15.45:0/2036522723 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc37fe640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc415b270 0x7effc415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc27fc640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc40b3420 0x7effc414d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc2ffd640 1 -- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc415b270 msgr2=0x7effc415c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc2ffd640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc415b270 0x7effc415c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc2ffd640 1 -- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc40b3420 msgr2=0x7effc414d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc2ffd640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc40b3420 0x7effc414d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc2ffd640 1 -- 172.21.15.45:0/2036522723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac0094d0 con 0x7effc40b3040 2023-11-28T20:34:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.259+0000 7effc27fc640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc40b3420 0x7effc414d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:34:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.260+0000 7effc2ffd640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc40b3040 0x7effc409acf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7effb400c410 tx=0x7effb400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.260+0000 7effbbfff640 1 -- 172.21.15.45:0/2036522723 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb40035f0 con 0x7effc40b3040 2023-11-28T20:34:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.260+0000 7effbbfff640 1 -- 172.21.15.45:0/2036522723 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb4003770 con 0x7effc40b3040 2023-11-28T20:34:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.260+0000 7effbbfff640 1 -- 172.21.15.45:0/2036522723 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4003b20 con 0x7effc40b3040 2023-11-28T20:34:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.260+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc409bc70 con 0x7effc40b3040 2023-11-28T20:34:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.260+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc414dcd0 con 0x7effc40b3040 2023-11-28T20:34:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.261+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc409c190 con 0x7effc40b3040 2023-11-28T20:34:59.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.262+0000 7effbbfff640 1 -- 172.21.15.45:0/2036522723 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7effb4003c80 con 0x7effc40b3040 2023-11-28T20:34:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.263+0000 7effbbfff640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.263+0000 7effbbfff640 1 -- 172.21.15.45:0/2036522723 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7effb4089090 con 0x7effc40b3040 2023-11-28T20:34:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.264+0000 7effc27fc640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.265+0000 7effc27fc640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7effac036000 tx=0x7effac03d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.268+0000 7effbbfff640 1 -- 172.21.15.45:0/2036522723 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effc409c190 con 0x7effc40b3040 2023-11-28T20:34:59.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.464+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7effc409cea0 con 0x7effc40b3040 2023-11-28T20:34:59.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.465+0000 7effbbfff640 1 -- 172.21.15.45:0/2036522723 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7effb40170a0 con 0x7effc40b3040 2023-11-28T20:34:59.468 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:34:59.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.468+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 msgr2=0x7effa406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.468+0000 7effc3fff640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7effac036000 tx=0x7effac03d040 comp rx=0 tx=0).stop 2023-11-28T20:34:59.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.468+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc40b3040 msgr2=0x7effc409acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.468+0000 7effc3fff640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc40b3040 0x7effc409acf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7effb400c410 tx=0x7effb400c8d0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.468+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 shutdown_connections 2023-11-28T20:34:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.468+0000 7effc3fff640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.468+0000 7effc3fff640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc415b270 0x7effc415c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.469+0000 7effc3fff640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc40b3420 0x7effc414d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.469+0000 7effc3fff640 1 --2- 172.21.15.45:0/2036522723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc40b3040 0x7effc409acf0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.469+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 >> 172.21.15.45:0/2036522723 conn(0x7effc40b1050 msgr2=0x7effc40ae990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.469+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 shutdown_connections 2023-11-28T20:34:59.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.469+0000 7effc3fff640 1 -- 172.21.15.45:0/2036522723 wait complete. 2023-11-28T20:34:59.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:34:59.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.713+0000 7ffb1bfff640 1 Processor -- start 2023-11-28T20:34:59.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.714+0000 7ffb1bfff640 1 -- start start 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.715+0000 7ffb1bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ffb1c0b3040 0x7ffb1c06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.715+0000 7ffb1bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffb1c15b270 0x7ffb1c15f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.715+0000 7ffb1bfff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c15faf0 con 0x7ffb1c0b3420 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.715+0000 7ffb1bfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c0ceaa0 con 0x7ffb1c15b270 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.715+0000 7ffb1bfff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c06d410 con 0x7ffb1c0b3040 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb1b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ffb1c0b3420 0x7ffb1c0b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60872/0 (socket says 172.21.15.45:60872) 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb1b7fe640 1 -- 172.21.15.45:0/1755681270 learned_addr learned my addr 172.21.15.45:0/1755681270 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb1a7fc640 1 --2- 172.21.15.45:0/1755681270 >> v2:172.21.15.45:3300/0 conn(0x7ffb1c15b270 0x7ffb1c15f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb1affd640 1 --2- 172.21.15.45:0/1755681270 >> v2:172.21.15.190:3300/0 conn(0x7ffb1c0b3040 0x7ffb1c06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb19ffb640 1 -- 172.21.15.45:0/1755681270 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963844826 0 0) 0x7ffb1c15faf0 con 0x7ffb1c0b3420 2023-11-28T20:34:59.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb19ffb640 1 -- 172.21.15.45:0/1755681270 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb08003520 con 0x7ffb1c0b3420 2023-11-28T20:34:59.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb1a7fc640 1 -- 172.21.15.45:0/1755681270 >> v2:172.21.15.190:3300/0 conn(0x7ffb1c0b3040 msgr2=0x7ffb1c06cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:34:59.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb1a7fc640 1 --2- 172.21.15.45:0/1755681270 >> v2:172.21.15.190:3300/0 conn(0x7ffb1c0b3040 0x7ffb1c06cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb1a7fc640 1 -- 172.21.15.45:0/1755681270 >> v1:172.21.15.45:6789/0 conn(0x7ffb1c0b3420 legacy=0x7ffb1c0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.716+0000 7ffb1a7fc640 1 -- 172.21.15.45:0/1755681270 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c06d6f0 con 0x7ffb1c15b270 2023-11-28T20:34:59.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.717+0000 7ffb1a7fc640 1 --2- 172.21.15.45:0/1755681270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c15b270 0x7ffb1c15f5b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c0b4410 tx=0x7ffb0402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcb6f5d36cb69be server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:59.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb19ffb640 1 -- 172.21.15.45:0/1755681270 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04031710 con 0x7ffb1c15b270 2023-11-28T20:34:59.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb19ffb640 1 -- 172.21.15.45:0/1755681270 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0402f050 con 0x7ffb1c15b270 2023-11-28T20:34:59.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb19ffb640 1 -- 172.21.15.45:0/1755681270 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04031a90 con 0x7ffb1c15b270 2023-11-28T20:34:59.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb1bfff640 1 -- 172.21.15.45:0/1755681270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c15b270 msgr2=0x7ffb1c15f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb1bfff640 1 --2- 172.21.15.45:0/1755681270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c15b270 0x7ffb1c15f5b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c0b4410 tx=0x7ffb0402b7b0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb1bfff640 1 -- 172.21.15.45:0/1755681270 shutdown_connections 2023-11-28T20:34:59.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb1bfff640 1 --2- 172.21.15.45:0/1755681270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c15b270 0x7ffb1c15f5b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb1bfff640 1 --2- 172.21.15.45:0/1755681270 >> v2:172.21.15.190:3300/0 conn(0x7ffb1c0b3040 0x7ffb1c06cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb1bfff640 1 -- 172.21.15.45:0/1755681270 >> 172.21.15.45:0/1755681270 conn(0x7ffb1c0b1050 msgr2=0x7ffb1c0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:59.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb1bfff640 1 -- 172.21.15.45:0/1755681270 shutdown_connections 2023-11-28T20:34:59.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.718+0000 7ffb1bfff640 1 -- 172.21.15.45:0/1755681270 wait complete. 2023-11-28T20:34:59.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.720+0000 7ffb1bfff640 1 Processor -- start 2023-11-28T20:34:59.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.720+0000 7ffb1bfff640 1 -- start start 2023-11-28T20:34:59.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.721+0000 7ffb1bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb1c0b3040 0x7ffb1c13fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.721+0000 7ffb1bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c0b3420 0x7ffb1c09bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.721+0000 7ffb1bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c15b270 0x7ffb1c099de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.721+0000 7ffb1bfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c13f670 con 0x7ffb1c0b3420 2023-11-28T20:34:59.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.721+0000 7ffb1bfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c09c4a0 con 0x7ffb1c15b270 2023-11-28T20:34:59.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.721+0000 7ffb1bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c09c780 con 0x7ffb1c0b3040 2023-11-28T20:34:59.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c0b3420 0x7ffb1c09bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c0b3420 0x7ffb1c09bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36970/0 (socket says 172.21.15.45:36970) 2023-11-28T20:34:59.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1a7fc640 1 -- 172.21.15.45:0/336088049 learned_addr learned my addr 172.21.15.45:0/336088049 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:34:59.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1affd640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb1c0b3040 0x7ffb1c13fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1b7fe640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c15b270 0x7ffb1c099de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1a7fc640 1 -- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb1c0b3040 msgr2=0x7ffb1c13fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1a7fc640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb1c0b3040 0x7ffb1c13fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1a7fc640 1 -- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c15b270 msgr2=0x7ffb1c099de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1a7fc640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c15b270 0x7ffb1c099de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1a7fc640 1 -- 172.21.15.45:0/336088049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04009400 con 0x7ffb1c0b3420 2023-11-28T20:34:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.722+0000 7ffb1a7fc640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c0b3420 0x7ffb1c09bae0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffb04000c00 tx=0x7ffb040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.723+0000 7ffb137fe640 1 -- 172.21.15.45:0/336088049 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0403c810 con 0x7ffb1c0b3420 2023-11-28T20:34:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.723+0000 7ffb137fe640 1 -- 172.21.15.45:0/336088049 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0403c970 con 0x7ffb1c0b3420 2023-11-28T20:34:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.723+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c13f910 con 0x7ffb1c0b3420 2023-11-28T20:34:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.723+0000 7ffb137fe640 1 -- 172.21.15.45:0/336088049 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0403cd20 con 0x7ffb1c0b3420 2023-11-28T20:34:59.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.723+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c140350 con 0x7ffb1c0b3420 2023-11-28T20:34:59.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.724+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c0cc390 con 0x7ffb1c0b3420 2023-11-28T20:34:59.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.725+0000 7ffb137fe640 1 -- 172.21.15.45:0/336088049 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffb0403a040 con 0x7ffb1c0b3420 2023-11-28T20:34:59.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.726+0000 7ffb137fe640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffafc06ae40 0x7ffafc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:34:59.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.726+0000 7ffb137fe640 1 -- 172.21.15.45:0/336088049 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ffb040aabd0 con 0x7ffb1c0b3420 2023-11-28T20:34:59.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.727+0000 7ffb1affd640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffafc06ae40 0x7ffafc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:34:59.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.728+0000 7ffb1affd640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffafc06ae40 0x7ffafc06d300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009970 tx=0x7ffb0c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:34:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.730+0000 7ffb137fe640 1 -- 172.21.15.45:0/336088049 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb04075820 con 0x7ffb1c0b3420 2023-11-28T20:34:59.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.921+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffb1c13faa0 con 0x7ffb1c0b3420 2023-11-28T20:34:59.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.921+0000 7ffb137fe640 1 -- 172.21.15.45:0/336088049 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffb04078e40 con 0x7ffb1c0b3420 2023-11-28T20:34:59.924 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:34:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffafc06ae40 msgr2=0x7ffafc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffafc06ae40 0x7ffafc06d300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009970 tx=0x7ffb0c008040 comp rx=0 tx=0).stop 2023-11-28T20:34:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c0b3420 msgr2=0x7ffb1c09bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:34:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c0b3420 0x7ffb1c09bae0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffb04000c00 tx=0x7ffb040046c0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 shutdown_connections 2023-11-28T20:34:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ffafc06ae40 0x7ffafc06d300 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c15b270 0x7ffb1c099de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c0b3420 0x7ffb1c09bae0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 --2- 172.21.15.45:0/336088049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb1c0b3040 0x7ffb1c13fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:34:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 >> 172.21.15.45:0/336088049 conn(0x7ffb1c0b1050 msgr2=0x7ffb1c0ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:34:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 shutdown_connections 2023-11-28T20:34:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:34:59.924+0000 7ffb1bfff640 1 -- 172.21.15.45:0/336088049 wait complete. 2023-11-28T20:34:59.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:35:00.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.167+0000 7fcdaf59e640 1 Processor -- start 2023-11-28T20:35:00.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.168+0000 7fcdaf59e640 1 -- start start 2023-11-28T20:35:00.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.169+0000 7fcdaf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdb00d0040 0x7fcdb00d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.169+0000 7fcdaf59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcdb00b2290 0x7fcdb00d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.169+0000 7fcdaf59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb00b90a0 con 0x7fcdb00b1d40 2023-11-28T20:35:00.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.169+0000 7fcdaf59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb00d1920 con 0x7fcdb00b2290 2023-11-28T20:35:00.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.169+0000 7fcdaf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb00d2dc0 con 0x7fcdb00d0040 2023-11-28T20:35:00.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.169+0000 7fcdadd9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcdb00b2290 0x7fcdb00d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.169+0000 7fcdaed9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcdb00b1d40 0x7fcdb00d3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60886/0 (socket says 172.21.15.45:60886) 2023-11-28T20:35:00.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.169+0000 7fcdaed9d640 1 -- 172.21.15.45:0/1971984285 learned_addr learned my addr 172.21.15.45:0/1971984285 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.170+0000 7fcdae59c640 1 --2- 172.21.15.45:0/1971984285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdb00d0040 0x7fcdb00d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.170+0000 7fcdadd9b640 1 -- 172.21.15.45:0/1971984285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdb00d0040 msgr2=0x7fcdb00d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.170+0000 7fcdadd9b640 1 --2- 172.21.15.45:0/1971984285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdb00d0040 0x7fcdb00d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.170+0000 7fcdadd9b640 1 -- 172.21.15.45:0/1971984285 >> v1:172.21.15.45:6789/0 conn(0x7fcdb00b1d40 legacy=0x7fcdb00d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.170+0000 7fcdadd9b640 1 -- 172.21.15.45:0/1971984285 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb00d0960 con 0x7fcdb00b2290 2023-11-28T20:35:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.170+0000 7fcdad59a640 1 -- 172.21.15.45:0/1971984285 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104193894 0 0) 0x7fcdb00b90a0 con 0x7fcdb00b1d40 2023-11-28T20:35:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.171+0000 7fcdadd9b640 1 --2- 172.21.15.45:0/1971984285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 0x7fcdb00d3040 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcd980097a0 tx=0x7fcd9802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32853a2bc7480bd7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.171+0000 7fcdad59a640 1 -- 172.21.15.45:0/1971984285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd98030610 con 0x7fcdb00b2290 2023-11-28T20:35:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdad59a640 1 -- 172.21.15.45:0/1971984285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd98030770 con 0x7fcdb00b2290 2023-11-28T20:35:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdad59a640 1 -- 172.21.15.45:0/1971984285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd98030b00 con 0x7fcdb00b2290 2023-11-28T20:35:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdaf59e640 1 -- 172.21.15.45:0/1971984285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 msgr2=0x7fcdb00d3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdaf59e640 1 --2- 172.21.15.45:0/1971984285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 0x7fcdb00d3040 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcd980097a0 tx=0x7fcd9802b960 comp rx=0 tx=0).stop 2023-11-28T20:35:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdaf59e640 1 -- 172.21.15.45:0/1971984285 shutdown_connections 2023-11-28T20:35:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdaf59e640 1 --2- 172.21.15.45:0/1971984285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 0x7fcdb00d3040 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdaf59e640 1 --2- 172.21.15.45:0/1971984285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdb00d0040 0x7fcdb00d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdaf59e640 1 -- 172.21.15.45:0/1971984285 >> 172.21.15.45:0/1971984285 conn(0x7fcdb0070f40 msgr2=0x7fcdb0071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdaf59e640 1 -- 172.21.15.45:0/1971984285 shutdown_connections 2023-11-28T20:35:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.172+0000 7fcdaf59e640 1 -- 172.21.15.45:0/1971984285 wait complete. 2023-11-28T20:35:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.174+0000 7fcdaf59e640 1 Processor -- start 2023-11-28T20:35:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.174+0000 7fcdaf59e640 1 -- start start 2023-11-28T20:35:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.175+0000 7fcdaf59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb00b1d40 0x7fcdb00ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.175+0000 7fcdaf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 0x7fcdb00b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.175+0000 7fcdaf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdb00d0040 0x7fcdb015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.175+0000 7fcdaf59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb00caa50 con 0x7fcdb00b2290 2023-11-28T20:35:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.175+0000 7fcdaf59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb0159f90 con 0x7fcdb00b1d40 2023-11-28T20:35:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.175+0000 7fcdaf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb00c6850 con 0x7fcdb00d0040 2023-11-28T20:35:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.175+0000 7fcdadd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 0x7fcdb00b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.175+0000 7fcdadd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 0x7fcdb00b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36984/0 (socket says 172.21.15.45:36984) 2023-11-28T20:35:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcdadd9b640 1 -- 172.21.15.45:0/3119204368 learned_addr learned my addr 172.21.15.45:0/3119204368 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcdadd9b640 1 -- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdb00d0040 msgr2=0x7fcdb015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcdadd9b640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdb00d0040 0x7fcdb015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcdadd9b640 1 -- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb00b1d40 msgr2=0x7fcdb00ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcdadd9b640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb00b1d40 0x7fcdb00ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcdadd9b640 1 -- 172.21.15.45:0/3119204368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd98009400 con 0x7fcdb00b2290 2023-11-28T20:35:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcdadd9b640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 0x7fcdb00b72e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcd9802bf70 tx=0x7fcd980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcda6ffd640 1 -- 172.21.15.45:0/3119204368 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9803e070 con 0x7fcdb00b2290 2023-11-28T20:35:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcda6ffd640 1 -- 172.21.15.45:0/3119204368 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd98039990 con 0x7fcdb00b2290 2023-11-28T20:35:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.176+0000 7fcda6ffd640 1 -- 172.21.15.45:0/3119204368 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd98039d20 con 0x7fcdb00b2290 2023-11-28T20:35:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.177+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb00c6b30 con 0x7fcdb00b2290 2023-11-28T20:35:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.177+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb00c70f0 con 0x7fcdb00b2290 2023-11-28T20:35:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.178+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb0071b30 con 0x7fcdb00b2290 2023-11-28T20:35:00.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.179+0000 7fcda6ffd640 1 -- 172.21.15.45:0/3119204368 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcd980423f0 con 0x7fcdb00b2290 2023-11-28T20:35:00.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.180+0000 7fcda6ffd640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd9006ae40 0x7fcd9006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.180+0000 7fcda6ffd640 1 -- 172.21.15.45:0/3119204368 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fcd980ab4d0 con 0x7fcdb00b2290 2023-11-28T20:35:00.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.180+0000 7fcdae59c640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd9006ae40 0x7fcd9006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.181+0000 7fcdae59c640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd9006ae40 0x7fcd9006d300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcda00098f0 tx=0x7fcda0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:00.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.184+0000 7fcda6ffd640 1 -- 172.21.15.45:0/3119204368 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd98076240 con 0x7fcdb00b2290 2023-11-28T20:35:00.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.380+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcdb004e920 con 0x7fcdb00b2290 2023-11-28T20:35:00.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.382+0000 7fcda6ffd640 1 -- 172.21.15.45:0/3119204368 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcd98079860 con 0x7fcdb00b2290 2023-11-28T20:35:00.384 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:35:00.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd9006ae40 msgr2=0x7fcd9006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd9006ae40 0x7fcd9006d300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcda00098f0 tx=0x7fcda0008040 comp rx=0 tx=0).stop 2023-11-28T20:35:00.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 msgr2=0x7fcdb00b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 0x7fcdb00b72e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcd9802bf70 tx=0x7fcd980046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 shutdown_connections 2023-11-28T20:35:00.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fcd9006ae40 0x7fcd9006d300 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdb00d0040 0x7fcdb015ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdb00b2290 0x7fcdb00b72e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 --2- 172.21.15.45:0/3119204368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb00b1d40 0x7fcdb00ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.384+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 >> 172.21.15.45:0/3119204368 conn(0x7fcdb0070f40 msgr2=0x7fcdb00b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:00.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.385+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 shutdown_connections 2023-11-28T20:35:00.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.385+0000 7fcdaf59e640 1 -- 172.21.15.45:0/3119204368 wait complete. 2023-11-28T20:35:00.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:35:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.636+0000 7fc006f73640 1 Processor -- start 2023-11-28T20:35:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.636+0000 7fc006f73640 1 -- start start 2023-11-28T20:35:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.638+0000 7fc006f73640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc0000b3040 0x7fc00006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.638+0000 7fc006f73640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc00015b270 0x7fc00015f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.638+0000 7fc006f73640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00015faf0 con 0x7fc0000b3420 2023-11-28T20:35:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.638+0000 7fc006f73640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0000ceaa0 con 0x7fc0000b3040 2023-11-28T20:35:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.638+0000 7fc006f73640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00006d410 con 0x7fc00015b270 2023-11-28T20:35:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.639+0000 7fc005770640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc00015b270 0x7fc00015f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.639+0000 7fc005f71640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc0000b3040 0x7fc00006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.639+0000 7fc006772640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fc0000b3420 0x7fc0000b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:56406/0 (socket says 172.21.15.45:56406) 2023-11-28T20:35:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.639+0000 7fc006772640 1 -- 172.21.15.45:0/1008955874 learned_addr learned my addr 172.21.15.45:0/1008955874 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.640+0000 7fc004f6f640 1 -- 172.21.15.45:0/1008955874 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499285027 0 0) 0x7fc00015faf0 con 0x7fc0000b3420 2023-11-28T20:35:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.640+0000 7fc005770640 1 -- 172.21.15.45:0/1008955874 >> v2:172.21.15.190:3300/0 conn(0x7fc0000b3040 msgr2=0x7fc00006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.640+0000 7fc005770640 1 --2- 172.21.15.45:0/1008955874 >> v2:172.21.15.190:3300/0 conn(0x7fc0000b3040 0x7fc00006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.640+0000 7fc005770640 1 -- 172.21.15.45:0/1008955874 >> v1:172.21.15.190:6789/0 conn(0x7fc0000b3420 legacy=0x7fc0000b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.640+0000 7fc005770640 1 -- 172.21.15.45:0/1008955874 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00006d6f0 con 0x7fc00015b270 2023-11-28T20:35:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.640+0000 7fc005f71640 1 --2- 172.21.15.45:0/1008955874 >> v2:172.21.15.190:3300/0 conn(0x7fc0000b3040 0x7fc00006cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:35:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.642+0000 7fc005770640 1 --2- 172.21.15.45:0/1008955874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc00015b270 0x7fc00015f5b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbff00097a0 tx=0x7fbff002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2d2de6d18c9ddb8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.642+0000 7fc004f6f640 1 -- 172.21.15.45:0/1008955874 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff00316f0 con 0x7fc00015b270 2023-11-28T20:35:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.642+0000 7fc004f6f640 1 -- 172.21.15.45:0/1008955874 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff0031850 con 0x7fc00015b270 2023-11-28T20:35:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.642+0000 7fc004f6f640 1 -- 172.21.15.45:0/1008955874 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0031be0 con 0x7fc00015b270 2023-11-28T20:35:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.643+0000 7fc006f73640 1 -- 172.21.15.45:0/1008955874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc00015b270 msgr2=0x7fc00015f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.643+0000 7fc006f73640 1 --2- 172.21.15.45:0/1008955874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc00015b270 0x7fc00015f5b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbff00097a0 tx=0x7fbff002b790 comp rx=0 tx=0).stop 2023-11-28T20:35:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.643+0000 7fc006f73640 1 -- 172.21.15.45:0/1008955874 shutdown_connections 2023-11-28T20:35:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.643+0000 7fc006f73640 1 --2- 172.21.15.45:0/1008955874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc00015b270 0x7fc00015f5b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.643+0000 7fc006f73640 1 --2- 172.21.15.45:0/1008955874 >> v2:172.21.15.190:3300/0 conn(0x7fc0000b3040 0x7fc00006cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.643+0000 7fc006f73640 1 -- 172.21.15.45:0/1008955874 >> 172.21.15.45:0/1008955874 conn(0x7fc0000b1050 msgr2=0x7fc0000cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.643+0000 7fc006f73640 1 -- 172.21.15.45:0/1008955874 shutdown_connections 2023-11-28T20:35:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.644+0000 7fc006f73640 1 -- 172.21.15.45:0/1008955874 wait complete. 2023-11-28T20:35:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.646+0000 7fc006f73640 1 Processor -- start 2023-11-28T20:35:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.646+0000 7fc006f73640 1 -- start start 2023-11-28T20:35:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.647+0000 7fc006f73640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0000b3040 0x7fc00015d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.647+0000 7fc006f73640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0000b3420 0x7fc0001552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc006f73640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc00015b270 0x7fc000155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc006f73640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00015dc70 con 0x7fc00015b270 2023-11-28T20:35:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc006f73640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00015d990 con 0x7fc0000b3420 2023-11-28T20:35:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc006f73640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc000099be0 con 0x7fc0000b3040 2023-11-28T20:35:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc006772640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc00015b270 0x7fc000155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc006772640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc00015b270 0x7fc000155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36996/0 (socket says 172.21.15.45:36996) 2023-11-28T20:35:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc006772640 1 -- 172.21.15.45:0/277011341 learned_addr learned my addr 172.21.15.45:0/277011341 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc005770640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0000b3420 0x7fc0001552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc005f71640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0000b3040 0x7fc00015d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.648+0000 7fc006772640 1 -- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0000b3040 msgr2=0x7fc00015d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fc006772640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0000b3040 0x7fc00015d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fc006772640 1 -- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0000b3420 msgr2=0x7fc0001552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fc006772640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0000b3420 0x7fc0001552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fc006772640 1 -- 172.21.15.45:0/277011341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc009400 con 0x7fc00015b270 2023-11-28T20:35:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fc005f71640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0000b3040 0x7fc00015d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fc005770640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0000b3420 0x7fc0001552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:00.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fc006772640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc00015b270 0x7fc000155820 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c7a0 tx=0x7fbff800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fbff67fc640 1 -- 172.21.15.45:0/277011341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff800a640 con 0x7fc00015b270 2023-11-28T20:35:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fbff67fc640 1 -- 172.21.15.45:0/277011341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff800a7a0 con 0x7fc00015b270 2023-11-28T20:35:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fbff67fc640 1 -- 172.21.15.45:0/277011341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff800abd0 con 0x7fc00015b270 2023-11-28T20:35:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff0009400 con 0x7fc00015b270 2023-11-28T20:35:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.649+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00009a1a0 con 0x7fc00015b270 2023-11-28T20:35:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.651+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00009a390 con 0x7fc00015b270 2023-11-28T20:35:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.651+0000 7fbff67fc640 1 -- 172.21.15.45:0/277011341 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbff800ad30 con 0x7fc00015b270 2023-11-28T20:35:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.652+0000 7fbff67fc640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfe406ae40 0x7fbfe406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.653+0000 7fbff67fc640 1 -- 172.21.15.45:0/277011341 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbff8089000 con 0x7fc00015b270 2023-11-28T20:35:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.653+0000 7fc005f71640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfe406ae40 0x7fbfe406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.654+0000 7fc005f71640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfe406ae40 0x7fbfe406d300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009c80 tx=0x7fbffc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.657+0000 7fbff67fc640 1 -- 172.21.15.45:0/277011341 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbff8053c50 con 0x7fc00015b270 2023-11-28T20:35:00.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.852+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc000158620 con 0x7fc00015b270 2023-11-28T20:35:00.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.853+0000 7fbff67fc640 1 -- 172.21.15.45:0/277011341 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbff8018070 con 0x7fc00015b270 2023-11-28T20:35:00.855 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:00.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfe406ae40 msgr2=0x7fbfe406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfe406ae40 0x7fbfe406d300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009c80 tx=0x7fbffc008040 comp rx=0 tx=0).stop 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc00015b270 msgr2=0x7fc000155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc00015b270 0x7fc000155820 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c7a0 tx=0x7fbff800cc60 comp rx=0 tx=0).stop 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 shutdown_connections 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfe406ae40 0x7fbfe406d300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc00015b270 0x7fc000155820 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0000b3420 0x7fc0001552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 --2- 172.21.15.45:0/277011341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0000b3040 0x7fc00015d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 >> 172.21.15.45:0/277011341 conn(0x7fc0000b1050 msgr2=0x7fc0000cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 shutdown_connections 2023-11-28T20:35:00.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:00.856+0000 7fc006f73640 1 -- 172.21.15.45:0/277011341 wait complete. 2023-11-28T20:35:00.896 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-11-28T20:35:00.896 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T20:35:00.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:01.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.129+0000 7f8d6f59e640 1 Processor -- start 2023-11-28T20:35:01.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.129+0000 7f8d6f59e640 1 -- start start 2023-11-28T20:35:01.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.130+0000 7f8d6f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8d700d02d0 0x7f8d700d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.130+0000 7f8d6f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700d2230 con 0x7f8d700d1ab0 2023-11-28T20:35:01.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.130+0000 7f8d6f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700d2510 con 0x7f8d7015d550 2023-11-28T20:35:01.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.131+0000 7f8d6f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d700d27f0 con 0x7f8d700d02d0 2023-11-28T20:35:01.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.131+0000 7f8d6ed9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8d700d1ab0 0x7f8d700b6f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60894/0 (socket says 172.21.15.45:60894) 2023-11-28T20:35:01.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.131+0000 7f8d6ed9d640 1 -- 172.21.15.45:0/4004620811 learned_addr learned my addr 172.21.15.45:0/4004620811 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:01.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.131+0000 7f8d6dd9b640 1 --2- 172.21.15.45:0/4004620811 >> v2:172.21.15.190:3300/0 conn(0x7f8d700d02d0 0x7f8d700d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.131+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959225848 0 0) 0x7f8d700d2230 con 0x7f8d700d1ab0 2023-11-28T20:35:01.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.131+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c003520 con 0x7f8d700d1ab0 2023-11-28T20:35:01.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2361710153 0 0) 0x7f8d5c003520 con 0x7f8d700d1ab0 2023-11-28T20:35:01.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 >> v2:172.21.15.190:3300/0 conn(0x7f8d700d02d0 msgr2=0x7f8d700d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:01.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 --2- 172.21.15.45:0/4004620811 >> v2:172.21.15.190:3300/0 conn(0x7f8d700d02d0 0x7f8d700d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 >> v1:172.21.15.190:6789/0 conn(0x7f8d7015d550 legacy=0x7f8d700cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d700b6a30 con 0x7f8d700d1ab0 2023-11-28T20:35:01.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f8d68001670 con 0x7f8d700d1ab0 2023-11-28T20:35:01.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 >> v1:172.21.15.45:6789/0 conn(0x7f8d700d1ab0 legacy=0x7f8d700b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d5c003970 0x7f8d5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d5c0047c0 0x7f8d5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d5c005180 0x7f8d5c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700d27f0 con 0x7f8d5c003970 2023-11-28T20:35:01.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c00c020 con 0x7f8d5c005180 2023-11-28T20:35:01.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c00c1e0 con 0x7f8d5c0047c0 2023-11-28T20:35:01.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.132+0000 7f8d6e59c640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d5c003970 0x7f8d5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.133+0000 7f8d6ed9d640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d5c005180 0x7f8d5c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.133+0000 7f8d6e59c640 1 -- 172.21.15.45:0/4004620811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d5c0047c0 msgr2=0x7f8d5c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:01.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.133+0000 7f8d6e59c640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d5c0047c0 0x7f8d5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.133+0000 7f8d6e59c640 1 -- 172.21.15.45:0/4004620811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d5c005180 msgr2=0x7f8d5c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:01.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.133+0000 7f8d6e59c640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d5c005180 0x7f8d5c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.133+0000 7f8d6e59c640 1 -- 172.21.15.45:0/4004620811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c003520 con 0x7f8d5c003970 2023-11-28T20:35:01.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.133+0000 7f8d6ed9d640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d5c005180 0x7f8d5c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:01.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.133+0000 7f8d6ed9d640 1 -- 172.21.15.45:0/4004620811 reap_dead start 2023-11-28T20:35:01.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.134+0000 7f8d6e59c640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d5c003970 0x7f8d5c003d90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8d58001750 tx=0x7f8d5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5c7824b837217f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:01.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.134+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d580052d0 con 0x7f8d5c003970 2023-11-28T20:35:01.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.134+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5802e050 con 0x7f8d5c003970 2023-11-28T20:35:01.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.134+0000 7f8d6d59a640 1 -- 172.21.15.45:0/4004620811 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58003830 con 0x7f8d5c003970 2023-11-28T20:35:01.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.135+0000 7f8d6f59e640 1 -- 172.21.15.45:0/4004620811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d5c003970 msgr2=0x7f8d5c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.135+0000 7f8d6f59e640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d5c003970 0x7f8d5c003d90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8d58001750 tx=0x7f8d5c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.135+0000 7f8d6f59e640 1 -- 172.21.15.45:0/4004620811 shutdown_connections 2023-11-28T20:35:01.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.135+0000 7f8d6f59e640 1 --2- 172.21.15.45:0/4004620811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d5c003970 0x7f8d5c003d90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.135+0000 7f8d6f59e640 1 -- 172.21.15.45:0/4004620811 >> 172.21.15.45:0/4004620811 conn(0x7f8d7006ac70 msgr2=0x7f8d700b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:01.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.135+0000 7f8d6f59e640 1 -- 172.21.15.45:0/4004620811 shutdown_connections 2023-11-28T20:35:01.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.135+0000 7f8d6f59e640 1 -- 172.21.15.45:0/4004620811 wait complete. 2023-11-28T20:35:01.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.137+0000 7f8d6f59e640 1 Processor -- start 2023-11-28T20:35:01.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.137+0000 7f8d6f59e640 1 -- start start 2023-11-28T20:35:01.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.138+0000 7f8d6f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d5c003970 0x7f8d70160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.138+0000 7f8d6f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d701528f0 0x7f8d70160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.138+0000 7f8d6f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d700c6d90 0x7f8d700c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.138+0000 7f8d6f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700c7a80 con 0x7f8d700c6d90 2023-11-28T20:35:01.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.138+0000 7f8d6f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700c7d60 con 0x7f8d701528f0 2023-11-28T20:35:01.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.138+0000 7f8d6f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700c8040 con 0x7f8d5c003970 2023-11-28T20:35:01.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.138+0000 7f8d6ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d700c6d90 0x7f8d700c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.138+0000 7f8d6ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d700c6d90 0x7f8d700c7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37012/0 (socket says 172.21.15.45:37012) 2023-11-28T20:35:01.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.138+0000 7f8d6ed9d640 1 -- 172.21.15.45:0/1783507568 learned_addr learned my addr 172.21.15.45:0/1783507568 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:01.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6dd9b640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d701528f0 0x7f8d70160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6e59c640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d5c003970 0x7f8d70160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6ed9d640 1 -- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d5c003970 msgr2=0x7f8d70160130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6ed9d640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d5c003970 0x7f8d70160130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6ed9d640 1 -- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d701528f0 msgr2=0x7f8d70160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6ed9d640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d701528f0 0x7f8d70160670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6ed9d640 1 -- 172.21.15.45:0/1783507568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70154af0 con 0x7f8d700c6d90 2023-11-28T20:35:01.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6dd9b640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d701528f0 0x7f8d70160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:01.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6ed9d640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d700c6d90 0x7f8d700c7210 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000f60 tx=0x7f8d6800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:01.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d66ffd640 1 -- 172.21.15.45:0/1783507568 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d680074e0 con 0x7f8d700c6d90 2023-11-28T20:35:01.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d66ffd640 1 -- 172.21.15.45:0/1783507568 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d68007640 con 0x7f8d700c6d90 2023-11-28T20:35:01.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d700c7750 con 0x7f8d700c6d90 2023-11-28T20:35:01.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d66ffd640 1 -- 172.21.15.45:0/1783507568 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68004a00 con 0x7f8d700c6d90 2023-11-28T20:35:01.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.139+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d700cb2f0 con 0x7f8d700c6d90 2023-11-28T20:35:01.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.141+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7004e920 con 0x7f8d700c6d90 2023-11-28T20:35:01.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.142+0000 7f8d66ffd640 1 -- 172.21.15.45:0/1783507568 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8d68004b60 con 0x7f8d700c6d90 2023-11-28T20:35:01.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.143+0000 7f8d66ffd640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8d5006ae40 0x7f8d5006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.143+0000 7f8d66ffd640 1 -- 172.21.15.45:0/1783507568 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8d680897d0 con 0x7f8d700c6d90 2023-11-28T20:35:01.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.143+0000 7f8d6e59c640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8d5006ae40 0x7f8d5006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.144+0000 7f8d6e59c640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8d5006ae40 0x7f8d5006d300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8d58033000 tx=0x7f8d58009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:01.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.147+0000 7f8d66ffd640 1 -- 172.21.15.45:0/1783507568 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d68057eb0 con 0x7f8d700c6d90 2023-11-28T20:35:01.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.346+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d700ced00 con 0x7f8d700c6d90 2023-11-28T20:35:01.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.349+0000 7f8d66ffd640 1 -- 172.21.15.45:0/1783507568 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7f8d68057a40 con 0x7f8d700c6d90 2023-11-28T20:35:01.352 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:01.352 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:01.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8d5006ae40 msgr2=0x7f8d5006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8d5006ae40 0x7f8d5006d300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8d58033000 tx=0x7f8d58009030 comp rx=0 tx=0).stop 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d700c6d90 msgr2=0x7f8d700c7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d700c6d90 0x7f8d700c7210 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000f60 tx=0x7f8d6800d9d0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 shutdown_connections 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8d5006ae40 0x7f8d5006d300 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d700c6d90 0x7f8d700c7210 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d701528f0 0x7f8d70160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 --2- 172.21.15.45:0/1783507568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d5c003970 0x7f8d70160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 >> 172.21.15.45:0/1783507568 conn(0x7f8d7006ac70 msgr2=0x7f8d7015dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 shutdown_connections 2023-11-28T20:35:01.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.352+0000 7f8d6f59e640 1 -- 172.21.15.45:0/1783507568 wait complete. 2023-11-28T20:35:01.375 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:35:01.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:35:01.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.597+0000 7f3deb402640 1 Processor -- start 2023-11-28T20:35:01.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.598+0000 7f3deb402640 1 -- start start 2023-11-28T20:35:01.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.599+0000 7f3deb402640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3de40cbd40 0x7f3de40b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.600+0000 7f3deb402640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40724d0 con 0x7f3de40b39d0 2023-11-28T20:35:01.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.600+0000 7f3deb402640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de406c670 con 0x7f3de40717e0 2023-11-28T20:35:01.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.600+0000 7f3deb402640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de40b42f0 con 0x7f3de40cbd40 2023-11-28T20:35:01.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.600+0000 7f3dea400640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3de40b39d0 0x7f3de406bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60906/0 (socket says 172.21.15.45:60906) 2023-11-28T20:35:01.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.600+0000 7f3dea400640 1 -- 172.21.15.45:0/2836609438 learned_addr learned my addr 172.21.15.45:0/2836609438 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:01.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.600+0000 7f3deac01640 1 --2- 172.21.15.45:0/2836609438 >> v2:172.21.15.190:3300/0 conn(0x7f3de40cbd40 0x7f3de40b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964248974 0 0) 0x7f3de40724d0 con 0x7f3de40b39d0 2023-11-28T20:35:01.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd0003520 con 0x7f3de40b39d0 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298529944 0 0) 0x7f3dd0003520 con 0x7f3de40b39d0 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 >> v2:172.21.15.190:3300/0 conn(0x7f3de40cbd40 msgr2=0x7f3de40b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 --2- 172.21.15.45:0/2836609438 >> v2:172.21.15.190:3300/0 conn(0x7f3de40cbd40 0x7f3de40b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 >> v1:172.21.15.190:6789/0 conn(0x7f3de40717e0 legacy=0x7f3de40cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de4071c70 con 0x7f3de40b39d0 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3de0001670 con 0x7f3de40b39d0 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 >> v1:172.21.15.45:6789/0 conn(0x7f3de40b39d0 legacy=0x7f3de406bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00039c0 0x7f3dd0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3dd00047c0 0x7f3dd0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dd00051d0 0x7f3dd000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd000c0f0 con 0x7f3dd00039c0 2023-11-28T20:35:01.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd000c260 con 0x7f3dd00047c0 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.601+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd000c420 con 0x7f3dd00051d0 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.602+0000 7f3dea400640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00039c0 0x7f3dd0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.602+0000 7f3de9bff640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3dd00047c0 0x7f3dd0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.602+0000 7f3deac01640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dd00051d0 0x7f3dd000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.602+0000 7f3dea400640 1 -- 172.21.15.45:0/2836609438 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dd00051d0 msgr2=0x7f3dd000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.602+0000 7f3dea400640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dd00051d0 0x7f3dd000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.602+0000 7f3dea400640 1 -- 172.21.15.45:0/2836609438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3dd00047c0 msgr2=0x7f3dd0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.602+0000 7f3dea400640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3dd00047c0 0x7f3dd0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.602+0000 7f3dea400640 1 -- 172.21.15.45:0/2836609438 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd0003520 con 0x7f3dd00039c0 2023-11-28T20:35:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.602+0000 7f3deac01640 1 -- 172.21.15.45:0/2836609438 reap_dead start 2023-11-28T20:35:01.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.603+0000 7f3dea400640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00039c0 0x7f3dd0003de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3de0007fb0 tx=0x7f3dd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af97e68c7ce54b6b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:01.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de0005ad0 con 0x7f3dd00039c0 2023-11-28T20:35:01.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de0005c30 con 0x7f3dd00039c0 2023-11-28T20:35:01.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3de93fe640 1 -- 172.21.15.45:0/2836609438 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de0007070 con 0x7f3dd00039c0 2023-11-28T20:35:01.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3deb402640 1 -- 172.21.15.45:0/2836609438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00039c0 msgr2=0x7f3dd0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3deb402640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00039c0 0x7f3dd0003de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3de0007fb0 tx=0x7f3dd00057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3deb402640 1 -- 172.21.15.45:0/2836609438 shutdown_connections 2023-11-28T20:35:01.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3deb402640 1 --2- 172.21.15.45:0/2836609438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3dd00039c0 0x7f3dd0003de0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3deb402640 1 -- 172.21.15.45:0/2836609438 >> 172.21.15.45:0/2836609438 conn(0x7f3de40b1050 msgr2=0x7f3de40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:01.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3deb402640 1 -- 172.21.15.45:0/2836609438 shutdown_connections 2023-11-28T20:35:01.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.604+0000 7f3deb402640 1 -- 172.21.15.45:0/2836609438 wait complete. 2023-11-28T20:35:01.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.606+0000 7f3deb402640 1 Processor -- start 2023-11-28T20:35:01.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.606+0000 7f3deb402640 1 -- start start 2023-11-28T20:35:01.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.607+0000 7f3deb402640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dd00039c0 0x7f3de409b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.607+0000 7f3deb402640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3de409a450 0x7f3de415e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deb402640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de41f6f90 0x7f3de41f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deb402640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de4161d40 con 0x7f3de41f6f90 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deb402640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de41f7950 con 0x7f3de409a450 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deb402640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de409be80 con 0x7f3dd00039c0 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deac01640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de41f6f90 0x7f3de41f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deac01640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de41f6f90 0x7f3de41f7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37024/0 (socket says 172.21.15.45:37024) 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deac01640 1 -- 172.21.15.45:0/2784469298 learned_addr learned my addr 172.21.15.45:0/2784469298 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3dea400640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dd00039c0 0x7f3de409b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3de9bff640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3de409a450 0x7f3de415e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deac01640 1 -- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dd00039c0 msgr2=0x7f3de409b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deac01640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dd00039c0 0x7f3de409b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deac01640 1 -- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3de409a450 msgr2=0x7f3de415e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deac01640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3de409a450 0x7f3de415e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.608+0000 7f3deac01640 1 -- 172.21.15.45:0/2784469298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de409aa10 con 0x7f3de41f6f90 2023-11-28T20:35:01.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.609+0000 7f3deac01640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de41f6f90 0x7f3de41f7410 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3de406ffc0 tx=0x7f3ddc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:01.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.609+0000 7f3ddaffd640 1 -- 172.21.15.45:0/2784469298 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc00e070 con 0x7f3de41f6f90 2023-11-28T20:35:01.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.609+0000 7f3ddaffd640 1 -- 172.21.15.45:0/2784469298 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ddc011de0 con 0x7f3de41f6f90 2023-11-28T20:35:01.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.609+0000 7f3ddaffd640 1 -- 172.21.15.45:0/2784469298 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc003990 con 0x7f3de41f6f90 2023-11-28T20:35:01.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.609+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de409acf0 con 0x7f3de41f6f90 2023-11-28T20:35:01.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.609+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de409b2b0 con 0x7f3de41f6f90 2023-11-28T20:35:01.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.610+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de409b470 con 0x7f3de41f6f90 2023-11-28T20:35:01.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.611+0000 7f3ddaffd640 1 -- 172.21.15.45:0/2784469298 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3ddc003af0 con 0x7f3de41f6f90 2023-11-28T20:35:01.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.612+0000 7f3ddaffd640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3dc806ae70 0x7f3dc806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:01.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.612+0000 7f3ddaffd640 1 -- 172.21.15.45:0/2784469298 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3ddc08a190 con 0x7f3de41f6f90 2023-11-28T20:35:01.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.613+0000 7f3dea400640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3dc806ae70 0x7f3dc806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:01.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.614+0000 7f3dea400640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3dc806ae70 0x7f3dc806d330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3de0034000 tx=0x7f3de0009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:01.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.617+0000 7f3ddaffd640 1 -- 172.21.15.45:0/2784469298 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ddc054de0 con 0x7f3de41f6f90 2023-11-28T20:35:01.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.810+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3de4161e80 con 0x7f3de41f6f90 2023-11-28T20:35:01.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.811+0000 7f3ddaffd640 1 -- 172.21.15.45:0/2784469298 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ddc058400 con 0x7f3de41f6f90 2023-11-28T20:35:01.813 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:35:01.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.813+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3dc806ae70 msgr2=0x7f3dc806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.813+0000 7f3deb402640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3dc806ae70 0x7f3dc806d330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3de0034000 tx=0x7f3de0009030 comp rx=0 tx=0).stop 2023-11-28T20:35:01.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.813+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de41f6f90 msgr2=0x7f3de41f7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:01.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.813+0000 7f3deb402640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de41f6f90 0x7f3de41f7410 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3de406ffc0 tx=0x7f3ddc0115a0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.814+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 shutdown_connections 2023-11-28T20:35:01.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.814+0000 7f3deb402640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f3dc806ae70 0x7f3dc806d330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.814+0000 7f3deb402640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de41f6f90 0x7f3de41f7410 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.814+0000 7f3deb402640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3de409a450 0x7f3de415e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.814+0000 7f3deb402640 1 --2- 172.21.15.45:0/2784469298 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dd00039c0 0x7f3de409b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:01.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.814+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 >> 172.21.15.45:0/2784469298 conn(0x7f3de40b1050 msgr2=0x7f3de406c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:01.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.814+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 shutdown_connections 2023-11-28T20:35:01.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:01.814+0000 7f3deb402640 1 -- 172.21.15.45:0/2784469298 wait complete. 2023-11-28T20:35:01.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:35:02.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.067+0000 7f7093d78640 1 Processor -- start 2023-11-28T20:35:02.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.067+0000 7f7093d78640 1 -- start start 2023-11-28T20:35:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.069+0000 7f7093d78640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f708c099a40 0x7f708c06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.069+0000 7f7093d78640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c0cbd30 0x7f708c0cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.069+0000 7f7093d78640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708c0d1fc0 con 0x7f708c0aff90 2023-11-28T20:35:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.069+0000 7f7093d78640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f708c06a910 con 0x7f708c099a40 2023-11-28T20:35:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.069+0000 7f7093d78640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708c0cb8a0 con 0x7f708c0cbd30 2023-11-28T20:35:02.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.069+0000 7f7091b14640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f708c099a40 0x7f708c06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.069+0000 7f7091313640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f708c0aff90 0x7f708c099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60920/0 (socket says 172.21.15.45:60920) 2023-11-28T20:35:02.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.069+0000 7f7091313640 1 -- 172.21.15.45:0/138320798 learned_addr learned my addr 172.21.15.45:0/138320798 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:02.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.070+0000 7f7092315640 1 --2- 172.21.15.45:0/138320798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c0cbd30 0x7f708c0cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.070+0000 7f7090b12640 1 -- 172.21.15.45:0/138320798 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445944795 0 0) 0x7f708c0d1fc0 con 0x7f708c0aff90 2023-11-28T20:35:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.070+0000 7f7090b12640 1 -- 172.21.15.45:0/138320798 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7078003520 con 0x7f708c0aff90 2023-11-28T20:35:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.070+0000 7f7091b14640 1 -- 172.21.15.45:0/138320798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c0cbd30 msgr2=0x7f708c0cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.070+0000 7f7091b14640 1 --2- 172.21.15.45:0/138320798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c0cbd30 0x7f708c0cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.070+0000 7f7091b14640 1 -- 172.21.15.45:0/138320798 >> v1:172.21.15.45:6789/0 conn(0x7f708c0aff90 legacy=0x7f708c099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.070+0000 7f7091b14640 1 -- 172.21.15.45:0/138320798 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c06a630 con 0x7f708c099a40 2023-11-28T20:35:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.070+0000 7f7092315640 1 --2- 172.21.15.45:0/138320798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c0cbd30 0x7f708c0cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.071+0000 7f7091b14640 1 --2- 172.21.15.45:0/138320798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c099a40 0x7f708c06b4d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f70880097a0 tx=0x7f708802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb2d5bccf13aa5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.071+0000 7f7090b12640 1 -- 172.21.15.45:0/138320798 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7088031780 con 0x7f708c099a40 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7090b12640 1 -- 172.21.15.45:0/138320798 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70880318e0 con 0x7f708c099a40 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7090b12640 1 -- 172.21.15.45:0/138320798 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7088031c70 con 0x7f708c099a40 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7093d78640 1 -- 172.21.15.45:0/138320798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c099a40 msgr2=0x7f708c06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7093d78640 1 --2- 172.21.15.45:0/138320798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c099a40 0x7f708c06b4d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f70880097a0 tx=0x7f708802b820 comp rx=0 tx=0).stop 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7093d78640 1 -- 172.21.15.45:0/138320798 shutdown_connections 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7093d78640 1 --2- 172.21.15.45:0/138320798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c0cbd30 0x7f708c0cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7093d78640 1 --2- 172.21.15.45:0/138320798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c099a40 0x7f708c06b4d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7093d78640 1 -- 172.21.15.45:0/138320798 >> 172.21.15.45:0/138320798 conn(0x7f708c0d1020 msgr2=0x7f708c0d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7093d78640 1 -- 172.21.15.45:0/138320798 shutdown_connections 2023-11-28T20:35:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.072+0000 7f7093d78640 1 -- 172.21.15.45:0/138320798 wait complete. 2023-11-28T20:35:02.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.074+0000 7f7093d78640 1 Processor -- start 2023-11-28T20:35:02.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.075+0000 7f7093d78640 1 -- start start 2023-11-28T20:35:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.075+0000 7f7093d78640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c099a40 0x7f708c1f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7093d78640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f708c0aff90 0x7f708c0cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7093d78640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c0cbd30 0x7f708c14e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7093d78640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1f2860 con 0x7f708c0cbd30 2023-11-28T20:35:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7093d78640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c14bd10 con 0x7f708c0aff90 2023-11-28T20:35:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7093d78640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708c0b5c30 con 0x7f708c099a40 2023-11-28T20:35:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7092315640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c0cbd30 0x7f708c14e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7092315640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c0cbd30 0x7f708c14e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37038/0 (socket says 172.21.15.45:37038) 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7092315640 1 -- 172.21.15.45:0/823471476 learned_addr learned my addr 172.21.15.45:0/823471476 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7091b14640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c099a40 0x7f708c1f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7091313640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f708c0aff90 0x7f708c0cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7092315640 1 -- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c099a40 msgr2=0x7f708c1f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7092315640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c099a40 0x7f708c1f2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7092315640 1 -- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f708c0aff90 msgr2=0x7f708c0cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7092315640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f708c0aff90 0x7f708c0cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7092315640 1 -- 172.21.15.45:0/823471476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7088009400 con 0x7f708c0cbd30 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.076+0000 7f7091313640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f708c0aff90 0x7f708c0cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.077+0000 7f7092315640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c0cbd30 0x7f708c14e060 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f708400cd50 tx=0x7f708400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.077+0000 7f70827fc640 1 -- 172.21.15.45:0/823471476 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400f680 con 0x7f708c0cbd30 2023-11-28T20:35:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.077+0000 7f70827fc640 1 -- 172.21.15.45:0/823471476 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708400f800 con 0x7f708c0cbd30 2023-11-28T20:35:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.077+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708c0b5f10 con 0x7f708c0cbd30 2023-11-28T20:35:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.077+0000 7f70827fc640 1 -- 172.21.15.45:0/823471476 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400fbb0 con 0x7f708c0cbd30 2023-11-28T20:35:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.077+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708c0b64d0 con 0x7f708c0cbd30 2023-11-28T20:35:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.078+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708c0733d0 con 0x7f708c0cbd30 2023-11-28T20:35:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.079+0000 7f70827fc640 1 -- 172.21.15.45:0/823471476 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f708400b040 con 0x7f708c0cbd30 2023-11-28T20:35:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.080+0000 7f70827fc640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f707006ae40 0x7f707006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.080+0000 7f70827fc640 1 -- 172.21.15.45:0/823471476 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f708408a450 con 0x7f708c0cbd30 2023-11-28T20:35:02.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.081+0000 7f7091b14640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f707006ae40 0x7f707006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.082+0000 7f7091b14640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f707006ae40 0x7f707006d300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7088036000 tx=0x7f708803d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:02.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.085+0000 7f70827fc640 1 -- 172.21.15.45:0/823471476 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70840550a0 con 0x7f708c0cbd30 2023-11-28T20:35:02.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.286+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f708c0d0550 con 0x7f708c0cbd30 2023-11-28T20:35:02.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.288+0000 7f70827fc640 1 -- 172.21.15.45:0/823471476 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f70840586c0 con 0x7f708c0cbd30 2023-11-28T20:35:02.290 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f707006ae40 msgr2=0x7f707006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f707006ae40 0x7f707006d300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7088036000 tx=0x7f708803d040 comp rx=0 tx=0).stop 2023-11-28T20:35:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c0cbd30 msgr2=0x7f708c14e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c0cbd30 0x7f708c14e060 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f708400cd50 tx=0x7f708400a650 comp rx=0 tx=0).stop 2023-11-28T20:35:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 shutdown_connections 2023-11-28T20:35:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f707006ae40 0x7f707006d300 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c0cbd30 0x7f708c14e060 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f708c0aff90 0x7f708c0cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 --2- 172.21.15.45:0/823471476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f708c099a40 0x7f708c1f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.290+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 >> 172.21.15.45:0/823471476 conn(0x7f708c0d1020 msgr2=0x7f708c0b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:02.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.291+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 shutdown_connections 2023-11-28T20:35:02.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.291+0000 7f7093d78640 1 -- 172.21.15.45:0/823471476 wait complete. 2023-11-28T20:35:02.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:35:02.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.532+0000 7f7455554640 1 Processor -- start 2023-11-28T20:35:02.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.532+0000 7f7455554640 1 -- start start 2023-11-28T20:35:02.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.533+0000 7f7455554640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f74500d0040 0x7f74500d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.533+0000 7f7455554640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f74500b2290 0x7f74500d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.534+0000 7f7455554640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74500b90a0 con 0x7f74500b1d40 2023-11-28T20:35:02.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.534+0000 7f7455554640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74500d1920 con 0x7f74500d0040 2023-11-28T20:35:02.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.534+0000 7f7455554640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74500d2dc0 con 0x7f74500b2290 2023-11-28T20:35:02.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.534+0000 7f7454d53640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f74500b1d40 0x7f74500d3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60934/0 (socket says 172.21.15.45:60934) 2023-11-28T20:35:02.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.534+0000 7f7454d53640 1 -- 172.21.15.45:0/538278846 learned_addr learned my addr 172.21.15.45:0/538278846 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:02.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.534+0000 7f744ffff640 1 --2- 172.21.15.45:0/538278846 >> v2:172.21.15.45:3300/0 conn(0x7f74500d0040 0x7f74500d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.534+0000 7f744f7fe640 1 --2- 172.21.15.45:0/538278846 >> v2:172.21.15.190:3300/0 conn(0x7f74500b2290 0x7f74500d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.534+0000 7f744effd640 1 -- 172.21.15.45:0/538278846 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661948539 0 0) 0x7f74500b90a0 con 0x7f74500b1d40 2023-11-28T20:35:02.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.535+0000 7f744effd640 1 -- 172.21.15.45:0/538278846 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f743c003520 con 0x7f74500b1d40 2023-11-28T20:35:02.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.535+0000 7f744ffff640 1 -- 172.21.15.45:0/538278846 >> v2:172.21.15.190:3300/0 conn(0x7f74500b2290 msgr2=0x7f74500d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:02.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.535+0000 7f744ffff640 1 --2- 172.21.15.45:0/538278846 >> v2:172.21.15.190:3300/0 conn(0x7f74500b2290 0x7f74500d3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.535+0000 7f744ffff640 1 -- 172.21.15.45:0/538278846 >> v1:172.21.15.45:6789/0 conn(0x7f74500b1d40 legacy=0x7f74500d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.535+0000 7f744ffff640 1 -- 172.21.15.45:0/538278846 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74500d0960 con 0x7f74500d0040 2023-11-28T20:35:02.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.536+0000 7f744ffff640 1 --2- 172.21.15.45:0/538278846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500d0040 0x7f74500d0420 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f74500b8800 tx=0x7f744002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd533cc4f22ed6e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:02.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.536+0000 7f744effd640 1 -- 172.21.15.45:0/538278846 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440030610 con 0x7f74500d0040 2023-11-28T20:35:02.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.536+0000 7f744effd640 1 -- 172.21.15.45:0/538278846 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7440030770 con 0x7f74500d0040 2023-11-28T20:35:02.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.536+0000 7f744effd640 1 -- 172.21.15.45:0/538278846 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440030b00 con 0x7f74500d0040 2023-11-28T20:35:02.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.537+0000 7f7455554640 1 -- 172.21.15.45:0/538278846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500d0040 msgr2=0x7f74500d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.537+0000 7f7455554640 1 --2- 172.21.15.45:0/538278846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500d0040 0x7f74500d0420 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f74500b8800 tx=0x7f744002b960 comp rx=0 tx=0).stop 2023-11-28T20:35:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.537+0000 7f7455554640 1 -- 172.21.15.45:0/538278846 shutdown_connections 2023-11-28T20:35:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.537+0000 7f7455554640 1 --2- 172.21.15.45:0/538278846 >> v2:172.21.15.190:3300/0 conn(0x7f74500b2290 0x7f74500d3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.537+0000 7f7455554640 1 --2- 172.21.15.45:0/538278846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500d0040 0x7f74500d0420 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.537+0000 7f7455554640 1 -- 172.21.15.45:0/538278846 >> 172.21.15.45:0/538278846 conn(0x7f7450070f40 msgr2=0x7f7450071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.537+0000 7f7455554640 1 -- 172.21.15.45:0/538278846 shutdown_connections 2023-11-28T20:35:02.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.537+0000 7f7455554640 1 -- 172.21.15.45:0/538278846 wait complete. 2023-11-28T20:35:02.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.539+0000 7f7455554640 1 Processor -- start 2023-11-28T20:35:02.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.539+0000 7f7455554640 1 -- start start 2023-11-28T20:35:02.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.540+0000 7f7455554640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74500b1d40 0x7f74500ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.540+0000 7f7455554640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500b2290 0x7f74501564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.540+0000 7f7455554640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74500d0040 0x7f7450159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.540+0000 7f7455554640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74500cd4b0 con 0x7f74500b2290 2023-11-28T20:35:02.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.540+0000 7f7455554640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74500cbbf0 con 0x7f74500d0040 2023-11-28T20:35:02.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.540+0000 7f7455554640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74500cbed0 con 0x7f74500b1d40 2023-11-28T20:35:02.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.540+0000 7f744f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500b2290 0x7f74501564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.540+0000 7f744f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500b2290 0x7f74501564f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37056/0 (socket says 172.21.15.45:37056) 2023-11-28T20:35:02.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.540+0000 7f744f7fe640 1 -- 172.21.15.45:0/181621428 learned_addr learned my addr 172.21.15.45:0/181621428 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:02.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f744f7fe640 1 -- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74500b1d40 msgr2=0x7f74500ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f744f7fe640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74500b1d40 0x7f74500ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f7454d53640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74500d0040 0x7f7450159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f744f7fe640 1 -- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74500d0040 msgr2=0x7f7450159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f744f7fe640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74500d0040 0x7f7450159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f744f7fe640 1 -- 172.21.15.45:0/181621428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7440009400 con 0x7f74500b2290 2023-11-28T20:35:02.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f744f7fe640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500b2290 0x7f74501564f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7438007ef0 tx=0x7f743800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:02.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f744cff9640 1 -- 172.21.15.45:0/181621428 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438014620 con 0x7f74500b2290 2023-11-28T20:35:02.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f744cff9640 1 -- 172.21.15.45:0/181621428 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7438014780 con 0x7f74500b2290 2023-11-28T20:35:02.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.541+0000 7f744cff9640 1 -- 172.21.15.45:0/181621428 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438014b30 con 0x7f74500b2290 2023-11-28T20:35:02.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.542+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74500cc1b0 con 0x7f74500b2290 2023-11-28T20:35:02.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.542+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74500cc770 con 0x7f74500b2290 2023-11-28T20:35:02.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.543+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7450071b30 con 0x7f74500b2290 2023-11-28T20:35:02.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.544+0000 7f744cff9640 1 -- 172.21.15.45:0/181621428 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7438014c90 con 0x7f74500b2290 2023-11-28T20:35:02.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.545+0000 7f744cff9640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f743006ae40 0x7f743006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:02.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.545+0000 7f744cff9640 1 -- 172.21.15.45:0/181621428 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f743808a3b0 con 0x7f74500b2290 2023-11-28T20:35:02.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.545+0000 7f744ffff640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f743006ae40 0x7f743006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:02.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.546+0000 7f744ffff640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f743006ae40 0x7f743006d300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7440036000 tx=0x7f744003b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:02.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.549+0000 7f744cff9640 1 -- 172.21.15.45:0/181621428 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7438055090 con 0x7f74500b2290 2023-11-28T20:35:02.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.742+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f745004e920 con 0x7f74500b2290 2023-11-28T20:35:02.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.743+0000 7f744cff9640 1 -- 172.21.15.45:0/181621428 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f74380586b0 con 0x7f74500b2290 2023-11-28T20:35:02.746 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:35:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.745+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f743006ae40 msgr2=0x7f743006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.745+0000 7f7455554640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f743006ae40 0x7f743006d300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7440036000 tx=0x7f744003b040 comp rx=0 tx=0).stop 2023-11-28T20:35:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.745+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500b2290 msgr2=0x7f74501564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.745+0000 7f7455554640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500b2290 0x7f74501564f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7438007ef0 tx=0x7f743800e4b0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.746+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 shutdown_connections 2023-11-28T20:35:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.746+0000 7f7455554640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f743006ae40 0x7f743006d300 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.746+0000 7f7455554640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74500d0040 0x7f7450159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.746+0000 7f7455554640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74500b2290 0x7f74501564f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.746+0000 7f7455554640 1 --2- 172.21.15.45:0/181621428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74500b1d40 0x7f74500ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.746+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 >> 172.21.15.45:0/181621428 conn(0x7f7450070f40 msgr2=0x7f74500b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.746+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 shutdown_connections 2023-11-28T20:35:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:02.746+0000 7f7455554640 1 -- 172.21.15.45:0/181621428 wait complete. 2023-11-28T20:35:02.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:35:03.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.000+0000 7f73a490f640 1 Processor -- start 2023-11-28T20:35:03.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.000+0000 7f73a490f640 1 -- start start 2023-11-28T20:35:03.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.002+0000 7f73a490f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a0159780 0x7f73a015d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.002+0000 7f73a490f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f73a00ae030 0x7f73a015d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.002+0000 7f73a490f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73a015de10 0x7f73a00701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.002+0000 7f73a490f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a00ce330 con 0x7f73a015de10 2023-11-28T20:35:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.002+0000 7f73a490f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a015e260 con 0x7f73a00ae030 2023-11-28T20:35:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.002+0000 7f73a490f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a00cd6b0 con 0x7f73a0159780 2023-11-28T20:35:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739ed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73a015de10 0x7f73a00701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739ed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73a015de10 0x7f73a00701c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37072/0 (socket says 172.21.15.45:37072) 2023-11-28T20:35:03.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739ed9d640 1 -- 172.21.15.45:0/2521755283 learned_addr learned my addr 172.21.15.45:0/2521755283 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:03.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739dd9b640 1 --2- 172.21.15.45:0/2521755283 >> v2:172.21.15.190:3300/0 conn(0x7f73a00ae030 0x7f73a015d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739e59c640 1 --2- 172.21.15.45:0/2521755283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a0159780 0x7f73a015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739ed9d640 1 -- 172.21.15.45:0/2521755283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a0159780 msgr2=0x7f73a015d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:03.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739ed9d640 1 --2- 172.21.15.45:0/2521755283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a0159780 0x7f73a015d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739ed9d640 1 -- 172.21.15.45:0/2521755283 >> v2:172.21.15.190:3300/0 conn(0x7f73a00ae030 msgr2=0x7f73a015d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:03.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739ed9d640 1 --2- 172.21.15.45:0/2521755283 >> v2:172.21.15.190:3300/0 conn(0x7f73a00ae030 0x7f73a015d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739ed9d640 1 -- 172.21.15.45:0/2521755283 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a00cd990 con 0x7f73a015de10 2023-11-28T20:35:03.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.003+0000 7f739dd9b640 1 --2- 172.21.15.45:0/2521755283 >> v2:172.21.15.190:3300/0 conn(0x7f73a00ae030 0x7f73a015d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:03.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.004+0000 7f739ed9d640 1 --2- 172.21.15.45:0/2521755283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 0x7f73a00701c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f73a00cfd50 tx=0x7f739802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1892d3f23d1a838 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:03.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.005+0000 7f739d59a640 1 -- 172.21.15.45:0/2521755283 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739802ec80 con 0x7f73a015de10 2023-11-28T20:35:03.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.005+0000 7f739d59a640 1 -- 172.21.15.45:0/2521755283 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739802ede0 con 0x7f73a015de10 2023-11-28T20:35:03.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.005+0000 7f739d59a640 1 -- 172.21.15.45:0/2521755283 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398036610 con 0x7f73a015de10 2023-11-28T20:35:03.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.005+0000 7f73a490f640 1 -- 172.21.15.45:0/2521755283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 msgr2=0x7f73a00701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.005+0000 7f73a490f640 1 --2- 172.21.15.45:0/2521755283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 0x7f73a00701c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f73a00cfd50 tx=0x7f739802e0e0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.006+0000 7f73a490f640 1 -- 172.21.15.45:0/2521755283 shutdown_connections 2023-11-28T20:35:03.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.006+0000 7f73a490f640 1 --2- 172.21.15.45:0/2521755283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 0x7f73a00701c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.006+0000 7f73a490f640 1 --2- 172.21.15.45:0/2521755283 >> v2:172.21.15.190:3300/0 conn(0x7f73a00ae030 0x7f73a015d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.006+0000 7f73a490f640 1 --2- 172.21.15.45:0/2521755283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a0159780 0x7f73a015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.006+0000 7f73a490f640 1 -- 172.21.15.45:0/2521755283 >> 172.21.15.45:0/2521755283 conn(0x7f73a00ced60 msgr2=0x7f73a00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:03.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.006+0000 7f73a490f640 1 -- 172.21.15.45:0/2521755283 shutdown_connections 2023-11-28T20:35:03.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.006+0000 7f73a490f640 1 -- 172.21.15.45:0/2521755283 wait complete. 2023-11-28T20:35:03.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.008+0000 7f73a490f640 1 Processor -- start 2023-11-28T20:35:03.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.008+0000 7f73a490f640 1 -- start start 2023-11-28T20:35:03.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f73a490f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a00ae030 0x7f73a01f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f73a490f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73a0159780 0x7f73a00c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f73a490f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 0x7f73a00af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f73a490f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a00b1aa0 con 0x7f73a015de10 2023-11-28T20:35:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f73a490f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a01f30c0 con 0x7f73a0159780 2023-11-28T20:35:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f73a490f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a00c8700 con 0x7f73a00ae030 2023-11-28T20:35:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 0x7f73a00af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 0x7f73a00af960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37086/0 (socket says 172.21.15.45:37086) 2023-11-28T20:35:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739ed9d640 1 -- 172.21.15.45:0/1233966150 learned_addr learned my addr 172.21.15.45:0/1233966150 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739e59c640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a00ae030 0x7f73a01f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739ed9d640 1 -- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a00ae030 msgr2=0x7f73a01f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739dd9b640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73a0159780 0x7f73a00c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739ed9d640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a00ae030 0x7f73a01f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739ed9d640 1 -- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73a0159780 msgr2=0x7f73a00c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739ed9d640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73a0159780 0x7f73a00c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739ed9d640 1 -- 172.21.15.45:0/1233966150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7398009b30 con 0x7f73a015de10 2023-11-28T20:35:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.009+0000 7f739dd9b640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73a0159780 0x7f73a00c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.010+0000 7f739ed9d640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 0x7f73a00af960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7398038040 tx=0x7f7398004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.010+0000 7f738effd640 1 -- 172.21.15.45:0/1233966150 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739802ec80 con 0x7f73a015de10 2023-11-28T20:35:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.010+0000 7f738effd640 1 -- 172.21.15.45:0/1233966150 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7398009510 con 0x7f73a015de10 2023-11-28T20:35:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.010+0000 7f738effd640 1 -- 172.21.15.45:0/1233966150 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739803e610 con 0x7f73a015de10 2023-11-28T20:35:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.010+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a00c4f30 con 0x7f73a015de10 2023-11-28T20:35:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.010+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a00c7b80 con 0x7f73a015de10 2023-11-28T20:35:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.012+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a00733d0 con 0x7f73a015de10 2023-11-28T20:35:03.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.012+0000 7f738effd640 1 -- 172.21.15.45:0/1233966150 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f739803e770 con 0x7f73a015de10 2023-11-28T20:35:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.013+0000 7f738effd640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f738006ae40 0x7f738006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.014+0000 7f738effd640 1 -- 172.21.15.45:0/1233966150 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f73980ab950 con 0x7f73a015de10 2023-11-28T20:35:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.014+0000 7f739e59c640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f738006ae40 0x7f738006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.015+0000 7f739e59c640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f738006ae40 0x7f738006d300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7390009a10 tx=0x7f7390008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:03.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.018+0000 7f738effd640 1 -- 172.21.15.45:0/1233966150 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73980766d0 con 0x7f73a015de10 2023-11-28T20:35:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.216+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f73a00c4f30 con 0x7f73a015de10 2023-11-28T20:35:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.217+0000 7f738effd640 1 -- 172.21.15.45:0/1233966150 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7398042020 con 0x7f73a015de10 2023-11-28T20:35:03.219 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.219+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f738006ae40 msgr2=0x7f738006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f738006ae40 0x7f738006d300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7390009a10 tx=0x7f7390008040 comp rx=0 tx=0).stop 2023-11-28T20:35:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 msgr2=0x7f73a00af960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 0x7f73a00af960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7398038040 tx=0x7f7398004a40 comp rx=0 tx=0).stop 2023-11-28T20:35:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 shutdown_connections 2023-11-28T20:35:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f738006ae40 0x7f738006d300 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73a015de10 0x7f73a00af960 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73a0159780 0x7f73a00c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 --2- 172.21.15.45:0/1233966150 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73a00ae030 0x7f73a01f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 >> 172.21.15.45:0/1233966150 conn(0x7f73a00ced60 msgr2=0x7f73a01582d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:03.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 shutdown_connections 2023-11-28T20:35:03.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.220+0000 7f73a490f640 1 -- 172.21.15.45:0/1233966150 wait complete. 2023-11-28T20:35:03.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-11-28T20:35:03.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.463+0000 7fd595d33640 1 Processor -- start 2023-11-28T20:35:03.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.463+0000 7fd595d33640 1 -- start start 2023-11-28T20:35:03.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.465+0000 7fd595d33640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900b0c30 0x7fd5900b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.465+0000 7fd595d33640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59015dcc0 con 0x7fd5901438b0 2023-11-28T20:35:03.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.465+0000 7fd595d33640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59015e840 con 0x7fd5900b3d60 2023-11-28T20:35:03.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.465+0000 7fd595d33640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5900b15f0 con 0x7fd5900b0c30 2023-11-28T20:35:03.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.465+0000 7fd58f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd5901438b0 0x7fd5900b1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60936/0 (socket says 172.21.15.45:60936) 2023-11-28T20:35:03.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.465+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2883042122 learned_addr learned my addr 172.21.15.45:0/2883042122 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:03.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58effd640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900b0c30 0x7fd5900b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840588862 0 0) 0x7fd59015dcc0 con 0x7fd5901438b0 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd578005730 con 0x7fd5901438b0 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1805873109 0 0) 0x7fd578005730 con 0x7fd5901438b0 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900b0c30 msgr2=0x7fd5900b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900b0c30 0x7fd5900b10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 >> v1:172.21.15.190:6789/0 conn(0x7fd5900b3d60 legacy=0x7fd5900b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5900b26b0 con 0x7fd5901438b0 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd580001670 con 0x7fd5901438b0 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 >> v1:172.21.15.45:6789/0 conn(0x7fd5901438b0 legacy=0x7fd5900b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd578005bb0 0x7fd578005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5780069d0 0x7fd578006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900b0c30 0x7fd578007460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5900b15f0 con 0x7fd578005bb0 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57800dd50 con 0x7fd5780069d0 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.466+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57800df10 con 0x7fd5900b0c30 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.467+0000 7fd58f7fe640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd578005bb0 0x7fd578005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.467+0000 7fd58effd640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5780069d0 0x7fd578006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.467+0000 7fd58ffff640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900b0c30 0x7fd578007460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.467+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900b0c30 msgr2=0x7fd578007460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.467+0000 7fd58f7fe640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900b0c30 0x7fd578007460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.467+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5780069d0 msgr2=0x7fd578006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:03.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.467+0000 7fd58f7fe640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5780069d0 0x7fd578006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.467+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2883042122 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd578005730 con 0x7fd578005bb0 2023-11-28T20:35:03.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.467+0000 7fd58effd640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5780069d0 0x7fd578006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.469+0000 7fd58f7fe640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd578005bb0 0x7fd578005fd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd58000c190 tx=0x7fd578007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abda1ff74b6b5fdd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.469+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd580005ad0 con 0x7fd578005bb0 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.469+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd580005c30 con 0x7fd578005bb0 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.469+0000 7fd58e7fc640 1 -- 172.21.15.45:0/2883042122 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58002c8a0 con 0x7fd578005bb0 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.469+0000 7fd595d33640 1 -- 172.21.15.45:0/2883042122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd578005bb0 msgr2=0x7fd578005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.469+0000 7fd595d33640 1 --2- 172.21.15.45:0/2883042122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd578005bb0 0x7fd578005fd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd58000c190 tx=0x7fd578007a00 comp rx=0 tx=0).stop 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.469+0000 7fd58ffff640 1 -- 172.21.15.45:0/2883042122 reap_dead start 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.469+0000 7fd595d33640 1 -- 172.21.15.45:0/2883042122 shutdown_connections 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.470+0000 7fd595d33640 1 -- 172.21.15.45:0/2883042122 >> 172.21.15.45:0/2883042122 conn(0x7fd59006a500 msgr2=0x7fd59006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:03.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.470+0000 7fd595d33640 1 -- 172.21.15.45:0/2883042122 shutdown_connections 2023-11-28T20:35:03.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.470+0000 7fd595d33640 1 -- 172.21.15.45:0/2883042122 wait complete. 2023-11-28T20:35:03.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.471+0000 7fd595d33640 1 Processor -- start 2023-11-28T20:35:03.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.472+0000 7fd595d33640 1 -- start start 2023-11-28T20:35:03.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.472+0000 7fd595d33640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd578005bb0 0x7fd5901e5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.472+0000 7fd595d33640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5901582b0 0x7fd590157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd595d33640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd590158730 0x7fd590157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd595d33640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd590158010 con 0x7fd590158730 2023-11-28T20:35:03.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd595d33640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd590158c90 con 0x7fd578005bb0 2023-11-28T20:35:03.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd595d33640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd590158f70 con 0x7fd5901582b0 2023-11-28T20:35:03.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd590158730 0x7fd590157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd590158730 0x7fd590157ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37104/0 (socket says 172.21.15.45:37104) 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58ffff640 1 -- 172.21.15.45:0/2638927406 learned_addr learned my addr 172.21.15.45:0/2638927406 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58f7fe640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd578005bb0 0x7fd5901e5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58effd640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5901582b0 0x7fd590157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58ffff640 1 -- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5901582b0 msgr2=0x7fd590157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58ffff640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5901582b0 0x7fd590157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58ffff640 1 -- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd578005bb0 msgr2=0x7fd5901e5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58ffff640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd578005bb0 0x7fd5901e5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.473+0000 7fd58ffff640 1 -- 172.21.15.45:0/2638927406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd590159250 con 0x7fd590158730 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.474+0000 7fd58ffff640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd590158730 0x7fd590157ad0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd588007fd0 tx=0x7fd58800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.474+0000 7fd577fff640 1 -- 172.21.15.45:0/2638927406 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588003970 con 0x7fd590158730 2023-11-28T20:35:03.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.474+0000 7fd577fff640 1 -- 172.21.15.45:0/2638927406 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd588003af0 con 0x7fd590158730 2023-11-28T20:35:03.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.474+0000 7fd577fff640 1 -- 172.21.15.45:0/2638927406 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588003ea0 con 0x7fd590158730 2023-11-28T20:35:03.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.474+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd590159530 con 0x7fd590158730 2023-11-28T20:35:03.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.474+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5901e5590 con 0x7fd590158730 2023-11-28T20:35:03.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.475+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5901e5720 con 0x7fd590158730 2023-11-28T20:35:03.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.476+0000 7fd577fff640 1 -- 172.21.15.45:0/2638927406 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd5880050d0 con 0x7fd590158730 2023-11-28T20:35:03.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.478+0000 7fd577fff640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd57c06ae40 0x7fd57c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:03.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.478+0000 7fd577fff640 1 -- 172.21.15.45:0/2638927406 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd588089910 con 0x7fd590158730 2023-11-28T20:35:03.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.479+0000 7fd58f7fe640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd57c06ae40 0x7fd57c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:03.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.480+0000 7fd58f7fe640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd57c06ae40 0x7fd57c06d300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd580034000 tx=0x7fd580009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:03.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.482+0000 7fd577fff640 1 -- 172.21.15.45:0/2638927406 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd588054560 con 0x7fd590158730 2023-11-28T20:35:03.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:03.748+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7fd5900733d0 con 0x7fd590158730 2023-11-28T20:35:04.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.032+0000 7fd577fff640 1 -- 172.21.15.45:0/2638927406 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v81) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd588057b80 con 0x7fd590158730 2023-11-28T20:35:04.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd57c06ae40 msgr2=0x7fd57c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:04.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd57c06ae40 0x7fd57c06d300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd580034000 tx=0x7fd580009030 comp rx=0 tx=0).stop 2023-11-28T20:35:04.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd590158730 msgr2=0x7fd590157ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:04.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd590158730 0x7fd590157ad0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd588007fd0 tx=0x7fd58800eb90 comp rx=0 tx=0).stop 2023-11-28T20:35:04.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 shutdown_connections 2023-11-28T20:35:04.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd57c06ae40 0x7fd57c06d300 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd590158730 0x7fd590157ad0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5901582b0 0x7fd590157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 --2- 172.21.15.45:0/2638927406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd578005bb0 0x7fd5901e5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 >> 172.21.15.45:0/2638927406 conn(0x7fd59006a500 msgr2=0x7fd5900b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:04.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 shutdown_connections 2023-11-28T20:35:04.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.036+0000 7fd595d33640 1 -- 172.21.15.45:0/2638927406 wait complete. 2023-11-28T20:35:04.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T20:35:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.291+0000 7f32d4a39640 1 Processor -- start 2023-11-28T20:35:04.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.291+0000 7f32d4a39640 1 -- start start 2023-11-28T20:35:04.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.292+0000 7f32d4a39640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f32d015d150 0x7f32d00b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.292+0000 7f32d4a39640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 0x7f32d00d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.293+0000 7f32d4a39640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d00b9030 con 0x7f32d00d1390 2023-11-28T20:35:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.293+0000 7f32d4a39640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d00d0980 con 0x7f32d015d150 2023-11-28T20:35:04.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.293+0000 7f32d4a39640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00b7880 con 0x7f32d00d0000 2023-11-28T20:35:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.293+0000 7f32cf7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f32d015d150 0x7f32d00b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.293+0000 7f32cf7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f32d015d150 0x7f32d00b8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47736/0 (socket says 172.21.15.45:47736) 2023-11-28T20:35:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.293+0000 7f32cf7fe640 1 -- 172.21.15.45:0/1425655146 learned_addr learned my addr 172.21.15.45:0/1425655146 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.293+0000 7f32ceffd640 1 --2- 172.21.15.45:0/1425655146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 0x7f32d00d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.293+0000 7f32cf7fe640 1 -- 172.21.15.45:0/1425655146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 msgr2=0x7f32d00d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.294+0000 7f32cf7fe640 1 --2- 172.21.15.45:0/1425655146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 0x7f32d00d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.294+0000 7f32cf7fe640 1 -- 172.21.15.45:0/1425655146 >> v1:172.21.15.190:6789/0 conn(0x7f32d00d1390 legacy=0x7f32d00d17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:04.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.294+0000 7f32cf7fe640 1 -- 172.21.15.45:0/1425655146 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d00b7b60 con 0x7f32d015d150 2023-11-28T20:35:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.295+0000 7f32cf7fe640 1 --2- 172.21.15.45:0/1425655146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 0x7f32d00b8af0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009e50 tx=0x7f32c002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3061c9a006a6b0d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.295+0000 7f32ce7fc640 1 -- 172.21.15.45:0/1425655146 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c002cb00 con 0x7f32d015d150 2023-11-28T20:35:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.295+0000 7f32ce7fc640 1 -- 172.21.15.45:0/1425655146 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c002cc60 con 0x7f32d015d150 2023-11-28T20:35:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.295+0000 7f32ce7fc640 1 -- 172.21.15.45:0/1425655146 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c00343f0 con 0x7f32d015d150 2023-11-28T20:35:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.296+0000 7f32d4a39640 1 -- 172.21.15.45:0/1425655146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 msgr2=0x7f32d00b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.296+0000 7f32d4a39640 1 --2- 172.21.15.45:0/1425655146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 0x7f32d00b8af0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009e50 tx=0x7f32c002bf60 comp rx=0 tx=0).stop 2023-11-28T20:35:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.296+0000 7f32d4a39640 1 -- 172.21.15.45:0/1425655146 shutdown_connections 2023-11-28T20:35:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.296+0000 7f32d4a39640 1 --2- 172.21.15.45:0/1425655146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 0x7f32d00d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.296+0000 7f32d4a39640 1 --2- 172.21.15.45:0/1425655146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 0x7f32d00b8af0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.296+0000 7f32d4a39640 1 -- 172.21.15.45:0/1425655146 >> 172.21.15.45:0/1425655146 conn(0x7f32d0070f40 msgr2=0x7f32d0071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.296+0000 7f32d4a39640 1 -- 172.21.15.45:0/1425655146 shutdown_connections 2023-11-28T20:35:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.296+0000 7f32d4a39640 1 -- 172.21.15.45:0/1425655146 wait complete. 2023-11-28T20:35:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.299+0000 7f32d4a39640 1 Processor -- start 2023-11-28T20:35:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.299+0000 7f32d4a39640 1 -- start start 2023-11-28T20:35:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32d4a39640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 0x7f32d00ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32d4a39640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d00d1390 0x7f32d014e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32d4a39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 0x7f32d00cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32d4a39640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d014ecb0 con 0x7f32d015d150 2023-11-28T20:35:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32d4a39640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00c94f0 con 0x7f32d00d1390 2023-11-28T20:35:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32d4a39640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00c89b0 con 0x7f32d00d0000 2023-11-28T20:35:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 0x7f32d00cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 0x7f32d00cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47738/0 (socket says 172.21.15.45:47738) 2023-11-28T20:35:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cffff640 1 -- 172.21.15.45:0/904362256 learned_addr learned my addr 172.21.15.45:0/904362256 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:04.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cf7fe640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 0x7f32d00ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:04.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32ceffd640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d00d1390 0x7f32d014e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:04.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cffff640 1 -- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 msgr2=0x7f32d00ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:04.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cffff640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 0x7f32d00ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cffff640 1 -- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d00d1390 msgr2=0x7f32d014e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:04.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cffff640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d00d1390 0x7f32d014e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cffff640 1 -- 172.21.15.45:0/904362256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c0009b30 con 0x7f32d015d150 2023-11-28T20:35:04.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32cffff640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 0x7f32d00cba90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f32c800c410 tx=0x7f32c800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:04.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.300+0000 7f32c7fff640 1 -- 172.21.15.45:0/904362256 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c8011790 con 0x7f32d015d150 2023-11-28T20:35:04.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.301+0000 7f32c7fff640 1 -- 172.21.15.45:0/904362256 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c80118f0 con 0x7f32d015d150 2023-11-28T20:35:04.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.301+0000 7f32c7fff640 1 -- 172.21.15.45:0/904362256 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c8011ce0 con 0x7f32d015d150 2023-11-28T20:35:04.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.301+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d00c8c50 con 0x7f32d015d150 2023-11-28T20:35:04.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.301+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d00c9210 con 0x7f32d015d150 2023-11-28T20:35:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.302+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d015d880 con 0x7f32d015d150 2023-11-28T20:35:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.303+0000 7f32c7fff640 1 -- 172.21.15.45:0/904362256 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f32c801c460 con 0x7f32d015d150 2023-11-28T20:35:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.305+0000 7f32c7fff640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32b006ae40 0x7f32b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.305+0000 7f32c7fff640 1 -- 172.21.15.45:0/904362256 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f32c80895f0 con 0x7f32d015d150 2023-11-28T20:35:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.306+0000 7f32cf7fe640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32b006ae40 0x7f32b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.307+0000 7f32cf7fe640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32b006ae40 0x7f32b006d300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32c0035000 tx=0x7f32c0030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.309+0000 7f32c7fff640 1 -- 172.21.15.45:0/904362256 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32d015d880 con 0x7f32d015d150 2023-11-28T20:35:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.505+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f32d015d880 con 0x7f32d015d150 2023-11-28T20:35:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.512+0000 7f32c7fff640 1 -- 172.21.15.45:0/904362256 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f32d015d880 con 0x7f32d015d150 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32b006ae40 msgr2=0x7f32b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32b006ae40 0x7f32b006d300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32c0035000 tx=0x7f32c0030000 comp rx=0 tx=0).stop 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 msgr2=0x7f32d00cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 0x7f32d00cba90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f32c800c410 tx=0x7f32c800c8d0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 shutdown_connections 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f32b006ae40 0x7f32b006d300 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32d015d150 0x7f32d00cba90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d00d1390 0x7f32d014e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 --2- 172.21.15.45:0/904362256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d00d0000 0x7f32d00ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 >> 172.21.15.45:0/904362256 conn(0x7f32d0070f40 msgr2=0x7f32d00d33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 shutdown_connections 2023-11-28T20:35:04.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:04.515+0000 7f32d4a39640 1 -- 172.21.15.45:0/904362256 wait complete. 2023-11-28T20:35:04.537 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:35:04.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:35:04.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:35:04.793 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:35:04.793 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:35:04.793 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:04.858 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:35:04.858 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:04.916 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:35:04.917 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:04.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:05.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.203+0000 7f85f9316640 1 Processor -- start 2023-11-28T20:35:05.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.203+0000 7f85f9316640 1 -- start start 2023-11-28T20:35:05.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f9316640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f40d02d0 0x7f85f40d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f9316640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f85f40d1ab0 0x7f85f40b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f9316640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f40d2230 con 0x7f85f415d550 2023-11-28T20:35:05.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f9316640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f40d2510 con 0x7f85f40d1ab0 2023-11-28T20:35:05.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f9316640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f40d27f0 con 0x7f85f40d02d0 2023-11-28T20:35:05.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f3fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f85f415d550 0x7f85f40cfbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57336/0 (socket says 172.21.15.45:57336) 2023-11-28T20:35:05.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f3fff640 1 -- 172.21.15.45:0/612018431 learned_addr learned my addr 172.21.15.45:0/612018431 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727269876 0 0) 0x7f85f40d2230 con 0x7f85f415d550 2023-11-28T20:35:05.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e0003520 con 0x7f85f415d550 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85ebfff640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f40d02d0 0x7f85f40d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.205+0000 7f85f8b15640 1 --2- 172.21.15.45:0/612018431 >> v2:172.21.15.190:3300/0 conn(0x7f85f40d1ab0 0x7f85f40b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 516762041 0 0) 0x7f85e0003520 con 0x7f85f415d550 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f40d02d0 msgr2=0x7f85f40d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f40d02d0 0x7f85f40d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 >> v2:172.21.15.190:3300/0 conn(0x7f85f40d1ab0 msgr2=0x7f85f40b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 --2- 172.21.15.45:0/612018431 >> v2:172.21.15.190:3300/0 conn(0x7f85f40d1ab0 0x7f85f40b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f40b6a30 con 0x7f85f415d550 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f85e4001670 con 0x7f85f415d550 2023-11-28T20:35:05.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 >> v1:172.21.15.45:6789/0 conn(0x7f85f415d550 legacy=0x7f85f40cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85e0003970 0x7f85e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f40d02d0 0x7f85e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f40d2510 con 0x7f85e0004d00 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f40d27f0 con 0x7f85e0003970 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e000bae0 con 0x7f85f40d02d0 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f8b15640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.206+0000 7f85f3fff640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85e0003970 0x7f85e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.207+0000 7f85ebfff640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f40d02d0 0x7f85e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.207+0000 7f85f8b15640 1 -- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f40d02d0 msgr2=0x7f85e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:05.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.207+0000 7f85f8b15640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f40d02d0 0x7f85e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.207+0000 7f85f8b15640 1 -- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85e0003970 msgr2=0x7f85e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:05.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.207+0000 7f85f8b15640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85e0003970 0x7f85e0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.207+0000 7f85f8b15640 1 -- 172.21.15.45:0/612018431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e0003520 con 0x7f85e0004d00 2023-11-28T20:35:05.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.207+0000 7f85f3fff640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85e0003970 0x7f85e0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:05.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.208+0000 7f85f8b15640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85e0005180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f85ec007360 tx=0x7f85e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafd4f060caba651 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:05.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.208+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec02ccf0 con 0x7f85e0004d00 2023-11-28T20:35:05.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.208+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ec02ce50 con 0x7f85e0004d00 2023-11-28T20:35:05.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.208+0000 7f85f37fe640 1 -- 172.21.15.45:0/612018431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec034660 con 0x7f85e0004d00 2023-11-28T20:35:05.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.209+0000 7f85f9316640 1 -- 172.21.15.45:0/612018431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 msgr2=0x7f85e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.209+0000 7f85f9316640 1 --2- 172.21.15.45:0/612018431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85e0005180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f85ec007360 tx=0x7f85e00057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.209+0000 7f85f8b15640 1 -- 172.21.15.45:0/612018431 reap_dead start 2023-11-28T20:35:05.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.209+0000 7f85f9316640 1 -- 172.21.15.45:0/612018431 shutdown_connections 2023-11-28T20:35:05.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.209+0000 7f85f9316640 1 -- 172.21.15.45:0/612018431 >> 172.21.15.45:0/612018431 conn(0x7f85f406ac70 msgr2=0x7f85f40b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:05.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.209+0000 7f85f9316640 1 -- 172.21.15.45:0/612018431 shutdown_connections 2023-11-28T20:35:05.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.209+0000 7f85f9316640 1 -- 172.21.15.45:0/612018431 wait complete. 2023-11-28T20:35:05.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.211+0000 7f85f9316640 1 Processor -- start 2023-11-28T20:35:05.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.211+0000 7f85f9316640 1 -- start start 2023-11-28T20:35:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.212+0000 7f85f9316640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85f4149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.212+0000 7f85f9316640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f415e300 0x7f85f40b76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.212+0000 7f85f9316640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f40b7c00 0x7f85f41f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.212+0000 7f85f9316640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4149b80 con 0x7f85e0004d00 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.212+0000 7f85f9316640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f40b8080 con 0x7f85f40b7c00 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.212+0000 7f85f9316640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4155a30 con 0x7f85f415e300 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.212+0000 7f85f3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85f4149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.212+0000 7f85f3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85f4149640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47748/0 (socket says 172.21.15.45:47748) 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.212+0000 7f85f3fff640 1 -- 172.21.15.45:0/357726123 learned_addr learned my addr 172.21.15.45:0/357726123 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.213+0000 7f85f3fff640 1 -- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f415e300 msgr2=0x7f85f40b76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.213+0000 7f85f3fff640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f415e300 0x7f85f40b76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.213+0000 7f85f3fff640 1 -- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f40b7c00 msgr2=0x7f85f41f6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.213+0000 7f85f8b15640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f40b7c00 0x7f85f41f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.213+0000 7f85f3fff640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f40b7c00 0x7f85f41f6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.213+0000 7f85f3fff640 1 -- 172.21.15.45:0/357726123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f4155d10 con 0x7f85e0004d00 2023-11-28T20:35:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.213+0000 7f85f3fff640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85f4149640 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f85e400e7f0 tx=0x7f85e400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:05.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.214+0000 7f85f17fa640 1 -- 172.21.15.45:0/357726123 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e400c870 con 0x7f85e0004d00 2023-11-28T20:35:05.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.214+0000 7f85f17fa640 1 -- 172.21.15.45:0/357726123 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e400c9d0 con 0x7f85e0004d00 2023-11-28T20:35:05.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.214+0000 7f85f17fa640 1 -- 172.21.15.45:0/357726123 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e400cd40 con 0x7f85e0004d00 2023-11-28T20:35:05.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.214+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f4155ff0 con 0x7f85e0004d00 2023-11-28T20:35:05.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.214+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f41565b0 con 0x7f85e0004d00 2023-11-28T20:35:05.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.215+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f404e920 con 0x7f85e0004d00 2023-11-28T20:35:05.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.216+0000 7f85f17fa640 1 -- 172.21.15.45:0/357726123 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f85e401d460 con 0x7f85e0004d00 2023-11-28T20:35:05.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.217+0000 7f85f17fa640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f85d406ae40 0x7f85d406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.217+0000 7f85f17fa640 1 -- 172.21.15.45:0/357726123 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f85e40899e0 con 0x7f85e0004d00 2023-11-28T20:35:05.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.218+0000 7f85ebfff640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f85d406ae40 0x7f85d406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.219+0000 7f85ebfff640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f85d406ae40 0x7f85d406d300 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f85f40cf090 tx=0x7f85dc00b2e0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:05.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.222+0000 7f85f17fa640 1 -- 172.21.15.45:0/357726123 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85e40546c0 con 0x7f85e0004d00 2023-11-28T20:35:05.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.416+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85f40cede0 con 0x7f85e0004d00 2023-11-28T20:35:05.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.419+0000 7f85f17fa640 1 -- 172.21.15.45:0/357726123 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7f85e4001520 con 0x7f85e0004d00 2023-11-28T20:35:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:05.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f85d406ae40 msgr2=0x7f85d406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f85d406ae40 0x7f85d406d300 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f85f40cf090 tx=0x7f85dc00b2e0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 msgr2=0x7f85f4149640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85f4149640 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f85e400e7f0 tx=0x7f85e400ecb0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 shutdown_connections 2023-11-28T20:35:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f85d406ae40 0x7f85d406d300 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f40b7c00 0x7f85f41f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f85f415e300 0x7f85f40b76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 --2- 172.21.15.45:0/357726123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85e0004d00 0x7f85f4149640 secure :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f85e400e7f0 tx=0x7f85e400ecb0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.422+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 >> 172.21.15.45:0/357726123 conn(0x7f85f406ac70 msgr2=0x7f85f415eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:05.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.423+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 shutdown_connections 2023-11-28T20:35:05.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.423+0000 7f85f9316640 1 -- 172.21.15.45:0/357726123 wait complete. 2023-11-28T20:35:05.443 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:35:05.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:35:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.672+0000 7fc6272ef640 1 Processor -- start 2023-11-28T20:35:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.672+0000 7fc6272ef640 1 -- start start 2023-11-28T20:35:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.674+0000 7fc6272ef640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6200cdca0 0x7fc62006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.674+0000 7fc6272ef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62006d410 con 0x7fc6200b3040 2023-11-28T20:35:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.674+0000 7fc6272ef640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62006d6f0 con 0x7fc62015b270 2023-11-28T20:35:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.674+0000 7fc6272ef640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6200ceaa0 con 0x7fc6200cdca0 2023-11-28T20:35:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.674+0000 7fc6262ed640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc6200b3040 0x7fc62015f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57342/0 (socket says 172.21.15.45:57342) 2023-11-28T20:35:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.674+0000 7fc6262ed640 1 -- 172.21.15.45:0/1868788221 learned_addr learned my addr 172.21.15.45:0/1868788221 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.674+0000 7fc626aee640 1 --2- 172.21.15.45:0/1868788221 >> v2:172.21.15.45:3300/0 conn(0x7fc6200cdca0 0x7fc62006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.675+0000 7fc6252eb640 1 -- 172.21.15.45:0/1868788221 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757861644 0 0) 0x7fc62006d410 con 0x7fc6200b3040 2023-11-28T20:35:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.675+0000 7fc6252eb640 1 -- 172.21.15.45:0/1868788221 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc60c003520 con 0x7fc6200b3040 2023-11-28T20:35:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.675+0000 7fc626aee640 1 -- 172.21.15.45:0/1868788221 >> v1:172.21.15.190:6789/0 conn(0x7fc62015b270 legacy=0x7fc6200b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.675+0000 7fc626aee640 1 -- 172.21.15.45:0/1868788221 >> v1:172.21.15.45:6789/0 conn(0x7fc6200b3040 legacy=0x7fc62015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.675+0000 7fc626aee640 1 -- 172.21.15.45:0/1868788221 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62006d9d0 con 0x7fc6200cdca0 2023-11-28T20:35:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.676+0000 7fc626aee640 1 --2- 172.21.15.45:0/1868788221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 0x7fc62006cda0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc618009d90 tx=0x7fc61802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8c3f946516c0bb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:05.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.676+0000 7fc6252eb640 1 -- 172.21.15.45:0/1868788221 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61802cbe0 con 0x7fc6200cdca0 2023-11-28T20:35:05.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.676+0000 7fc6252eb640 1 -- 172.21.15.45:0/1868788221 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61802cd40 con 0x7fc6200cdca0 2023-11-28T20:35:05.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.677+0000 7fc6252eb640 1 -- 172.21.15.45:0/1868788221 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618034660 con 0x7fc6200cdca0 2023-11-28T20:35:05.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.677+0000 7fc6272ef640 1 -- 172.21.15.45:0/1868788221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 msgr2=0x7fc62006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.677+0000 7fc6272ef640 1 --2- 172.21.15.45:0/1868788221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 0x7fc62006cda0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc618009d90 tx=0x7fc61802be30 comp rx=0 tx=0).stop 2023-11-28T20:35:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.677+0000 7fc6272ef640 1 -- 172.21.15.45:0/1868788221 shutdown_connections 2023-11-28T20:35:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.677+0000 7fc6272ef640 1 --2- 172.21.15.45:0/1868788221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 0x7fc62006cda0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.677+0000 7fc6272ef640 1 -- 172.21.15.45:0/1868788221 >> 172.21.15.45:0/1868788221 conn(0x7fc6200b1050 msgr2=0x7fc6200cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.677+0000 7fc6272ef640 1 -- 172.21.15.45:0/1868788221 shutdown_connections 2023-11-28T20:35:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.677+0000 7fc6272ef640 1 -- 172.21.15.45:0/1868788221 wait complete. 2023-11-28T20:35:05.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.679+0000 7fc6272ef640 1 Processor -- start 2023-11-28T20:35:05.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.679+0000 7fc6272ef640 1 -- start start 2023-11-28T20:35:05.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.680+0000 7fc6272ef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6200b3040 0x7fc62013cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.680+0000 7fc6272ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 0x7fc620157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.680+0000 7fc6272ef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62015b270 0x7fc620157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.680+0000 7fc6272ef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62014cca0 con 0x7fc6200cdca0 2023-11-28T20:35:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.680+0000 7fc6272ef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62013dcd0 con 0x7fc62015b270 2023-11-28T20:35:05.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.680+0000 7fc6272ef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc62013d710 con 0x7fc6200b3040 2023-11-28T20:35:05.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.680+0000 7fc625aec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 0x7fc620157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc625aec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 0x7fc620157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47766/0 (socket says 172.21.15.45:47766) 2023-11-28T20:35:05.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc625aec640 1 -- 172.21.15.45:0/712486372 learned_addr learned my addr 172.21.15.45:0/712486372 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:05.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc626aee640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62015b270 0x7fc620157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc6262ed640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6200b3040 0x7fc62013cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc625aec640 1 -- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6200b3040 msgr2=0x7fc62013cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc625aec640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6200b3040 0x7fc62013cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc625aec640 1 -- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62015b270 msgr2=0x7fc620157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc625aec640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62015b270 0x7fc620157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc625aec640 1 -- 172.21.15.45:0/712486372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc618009a70 con 0x7fc6200cdca0 2023-11-28T20:35:05.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc6262ed640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6200b3040 0x7fc62013cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:05.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc626aee640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62015b270 0x7fc620157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:05.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.681+0000 7fc625aec640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 0x7fc620157620 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c3c0 tx=0x7fc61000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:05.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.682+0000 7fc616ffd640 1 -- 172.21.15.45:0/712486372 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6100052d0 con 0x7fc6200cdca0 2023-11-28T20:35:05.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.682+0000 7fc616ffd640 1 -- 172.21.15.45:0/712486372 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc610005450 con 0x7fc6200cdca0 2023-11-28T20:35:05.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.682+0000 7fc616ffd640 1 -- 172.21.15.45:0/712486372 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc610003830 con 0x7fc6200cdca0 2023-11-28T20:35:05.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.682+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62013d9f0 con 0x7fc6200cdca0 2023-11-28T20:35:05.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.682+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc62013b120 con 0x7fc6200cdca0 2023-11-28T20:35:05.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.683+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc62013aca0 con 0x7fc6200cdca0 2023-11-28T20:35:05.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.684+0000 7fc616ffd640 1 -- 172.21.15.45:0/712486372 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc610003990 con 0x7fc6200cdca0 2023-11-28T20:35:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.685+0000 7fc616ffd640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc60406ae40 0x7fc60406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.685+0000 7fc616ffd640 1 -- 172.21.15.45:0/712486372 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc610088ee0 con 0x7fc6200cdca0 2023-11-28T20:35:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.686+0000 7fc6262ed640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc60406ae40 0x7fc60406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.687+0000 7fc6262ed640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc60406ae40 0x7fc60406d300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc61c007fd0 tx=0x7fc61c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.690+0000 7fc616ffd640 1 -- 172.21.15.45:0/712486372 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc62013aca0 con 0x7fc6200cdca0 2023-11-28T20:35:05.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.882+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc62013aca0 con 0x7fc6200cdca0 2023-11-28T20:35:05.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.883+0000 7fc616ffd640 1 -- 172.21.15.45:0/712486372 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc62013aca0 con 0x7fc6200cdca0 2023-11-28T20:35:05.885 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:35:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc60406ae40 msgr2=0x7fc60406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc60406ae40 0x7fc60406d300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc61c007fd0 tx=0x7fc61c009040 comp rx=0 tx=0).stop 2023-11-28T20:35:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 msgr2=0x7fc620157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 0x7fc620157620 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c3c0 tx=0x7fc61000c880 comp rx=0 tx=0).stop 2023-11-28T20:35:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 shutdown_connections 2023-11-28T20:35:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc60406ae40 0x7fc60406d300 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62015b270 0x7fc620157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6200cdca0 0x7fc620157620 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 --2- 172.21.15.45:0/712486372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6200b3040 0x7fc62013cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 >> 172.21.15.45:0/712486372 conn(0x7fc6200b1050 msgr2=0x7fc6200aefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 shutdown_connections 2023-11-28T20:35:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:05.886+0000 7fc6272ef640 1 -- 172.21.15.45:0/712486372 wait complete. 2023-11-28T20:35:05.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:35:06.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.135+0000 7febd1335640 1 Processor -- start 2023-11-28T20:35:06.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.136+0000 7febd1335640 1 -- start start 2023-11-28T20:35:06.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.137+0000 7febd1335640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0d0040 0x7febcc0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.137+0000 7febd1335640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7febcc0b1d40 0x7febcc0d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.137+0000 7febd1335640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febcc0b90a0 con 0x7febcc0b2290 2023-11-28T20:35:06.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.137+0000 7febd1335640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febcc0d1920 con 0x7febcc0b1d40 2023-11-28T20:35:06.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.137+0000 7febd1335640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febcc0d2dc0 con 0x7febcc0d0040 2023-11-28T20:35:06.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.137+0000 7febc3fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7febcc0b2290 0x7febcc0d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57356/0 (socket says 172.21.15.45:57356) 2023-11-28T20:35:06.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.137+0000 7febc3fff640 1 -- 172.21.15.45:0/243834358 learned_addr learned my addr 172.21.15.45:0/243834358 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:06.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.137+0000 7febd0b34640 1 --2- 172.21.15.45:0/243834358 >> v2:172.21.15.45:3300/0 conn(0x7febcc0b1d40 0x7febcc0d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.138+0000 7febcbfff640 1 --2- 172.21.15.45:0/243834358 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0d0040 0x7febcc0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.138+0000 7febd0b34640 1 -- 172.21.15.45:0/243834358 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0d0040 msgr2=0x7febcc0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:06.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.138+0000 7febd0b34640 1 --2- 172.21.15.45:0/243834358 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0d0040 0x7febcc0d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.138+0000 7febd0b34640 1 -- 172.21.15.45:0/243834358 >> v1:172.21.15.45:6789/0 conn(0x7febcc0b2290 legacy=0x7febcc0d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.138+0000 7febcb7fe640 1 -- 172.21.15.45:0/243834358 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506936170 0 0) 0x7febcc0b90a0 con 0x7febcc0b2290 2023-11-28T20:35:06.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.138+0000 7febd0b34640 1 -- 172.21.15.45:0/243834358 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febcc0d0960 con 0x7febcc0b1d40 2023-11-28T20:35:06.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.138+0000 7febcbfff640 1 --2- 172.21.15.45:0/243834358 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0d0040 0x7febcc0d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:06.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.139+0000 7febd0b34640 1 --2- 172.21.15.45:0/243834358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b1d40 0x7febcc0d3750 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7febc4009d90 tx=0x7febc402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5364d1bab2fc7d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:06.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.139+0000 7febcb7fe640 1 -- 172.21.15.45:0/243834358 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc402cce0 con 0x7febcc0b1d40 2023-11-28T20:35:06.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.139+0000 7febcb7fe640 1 -- 172.21.15.45:0/243834358 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc402ce40 con 0x7febcc0b1d40 2023-11-28T20:35:06.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.139+0000 7febcb7fe640 1 -- 172.21.15.45:0/243834358 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc4034660 con 0x7febcc0b1d40 2023-11-28T20:35:06.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.140+0000 7febd1335640 1 -- 172.21.15.45:0/243834358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b1d40 msgr2=0x7febcc0d3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.140+0000 7febd1335640 1 --2- 172.21.15.45:0/243834358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b1d40 0x7febcc0d3750 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7febc4009d90 tx=0x7febc402bf30 comp rx=0 tx=0).stop 2023-11-28T20:35:06.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.140+0000 7febd1335640 1 -- 172.21.15.45:0/243834358 shutdown_connections 2023-11-28T20:35:06.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.140+0000 7febd1335640 1 --2- 172.21.15.45:0/243834358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b1d40 0x7febcc0d3750 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.140+0000 7febd1335640 1 --2- 172.21.15.45:0/243834358 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0d0040 0x7febcc0d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.140+0000 7febd1335640 1 -- 172.21.15.45:0/243834358 >> 172.21.15.45:0/243834358 conn(0x7febcc070f40 msgr2=0x7febcc071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:06.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.140+0000 7febd1335640 1 -- 172.21.15.45:0/243834358 shutdown_connections 2023-11-28T20:35:06.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.140+0000 7febd1335640 1 -- 172.21.15.45:0/243834358 wait complete. 2023-11-28T20:35:06.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.142+0000 7febd1335640 1 Processor -- start 2023-11-28T20:35:06.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.142+0000 7febd1335640 1 -- start start 2023-11-28T20:35:06.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febd1335640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0b1d40 0x7febcc155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febd1335640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b2290 0x7febcc0ac270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febd1335640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febcc0d0040 0x7febcc0ac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febd1335640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febcc155870 con 0x7febcc0b2290 2023-11-28T20:35:06.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febd1335640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febcc0cc860 con 0x7febcc0d0040 2023-11-28T20:35:06.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febd1335640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febcc0ccb40 con 0x7febcc0b1d40 2023-11-28T20:35:06.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febc3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b2290 0x7febcc0ac270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febc3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b2290 0x7febcc0ac270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47794/0 (socket says 172.21.15.45:47794) 2023-11-28T20:35:06.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febc3fff640 1 -- 172.21.15.45:0/129567498 learned_addr learned my addr 172.21.15.45:0/129567498 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:06.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febcbfff640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0b1d40 0x7febcc155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febc3fff640 1 -- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0b1d40 msgr2=0x7febcc155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febc3fff640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0b1d40 0x7febcc155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febc3fff640 1 -- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febcc0d0040 msgr2=0x7febcc0ac7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febc3fff640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febcc0d0040 0x7febcc0ac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.143+0000 7febc3fff640 1 -- 172.21.15.45:0/129567498 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc4009a70 con 0x7febcc0b2290 2023-11-28T20:35:06.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.144+0000 7febc3fff640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b2290 0x7febcc0ac270 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7febb400c430 tx=0x7febb400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:06.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.144+0000 7febc97fa640 1 -- 172.21.15.45:0/129567498 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb40074e0 con 0x7febcc0b2290 2023-11-28T20:35:06.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.144+0000 7febc97fa640 1 -- 172.21.15.45:0/129567498 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb4007660 con 0x7febcc0b2290 2023-11-28T20:35:06.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.144+0000 7febc97fa640 1 -- 172.21.15.45:0/129567498 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4004a20 con 0x7febcc0b2290 2023-11-28T20:35:06.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.144+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febcc0cce20 con 0x7febcc0b2290 2023-11-28T20:35:06.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.144+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febcc0a9d10 con 0x7febcc0b2290 2023-11-28T20:35:06.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.145+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febcc0a9f30 con 0x7febcc0b2290 2023-11-28T20:35:06.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.146+0000 7febc97fa640 1 -- 172.21.15.45:0/129567498 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7febb4004b80 con 0x7febcc0b2290 2023-11-28T20:35:06.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.147+0000 7febc97fa640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7febac06ae40 0x7febac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.147+0000 7febc97fa640 1 -- 172.21.15.45:0/129567498 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7febb4089810 con 0x7febcc0b2290 2023-11-28T20:35:06.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.148+0000 7febcbfff640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7febac06ae40 0x7febac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.149+0000 7febcbfff640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7febac06ae40 0x7febac06d300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7febbc00a990 tx=0x7febbc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:06.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.151+0000 7febc97fa640 1 -- 172.21.15.45:0/129567498 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7febb40544f0 con 0x7febcc0b2290 2023-11-28T20:35:06.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.338+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7febcc0b6a60 con 0x7febcc0b2290 2023-11-28T20:35:06.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.339+0000 7febc97fa640 1 -- 172.21.15.45:0/129567498 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7febb4057b10 con 0x7febcc0b2290 2023-11-28T20:35:06.340 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:06.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7febac06ae40 msgr2=0x7febac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7febac06ae40 0x7febac06d300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7febbc00a990 tx=0x7febbc008040 comp rx=0 tx=0).stop 2023-11-28T20:35:06.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b2290 msgr2=0x7febcc0ac270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b2290 0x7febcc0ac270 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7febb400c430 tx=0x7febb400c8f0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 shutdown_connections 2023-11-28T20:35:06.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7febac06ae40 0x7febac06d300 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febcc0d0040 0x7febcc0ac7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febcc0b2290 0x7febcc0ac270 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 --2- 172.21.15.45:0/129567498 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febcc0b1d40 0x7febcc155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.341+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 >> 172.21.15.45:0/129567498 conn(0x7febcc070f40 msgr2=0x7febcc0b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:06.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.342+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 shutdown_connections 2023-11-28T20:35:06.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.342+0000 7febd1335640 1 -- 172.21.15.45:0/129567498 wait complete. 2023-11-28T20:35:06.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:35:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.591+0000 7f53f74a6640 1 Processor -- start 2023-11-28T20:35:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.591+0000 7f53f74a6640 1 -- start start 2023-11-28T20:35:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.592+0000 7f53f74a6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f53f006dc50 0x7f53f00cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.592+0000 7f53f74a6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f00ce600 0x7f53f006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.592+0000 7f53f74a6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f006e030 con 0x7f53f006aa40 2023-11-28T20:35:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.592+0000 7f53f74a6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f006aed0 con 0x7f53f006dc50 2023-11-28T20:35:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.592+0000 7f53f74a6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f00ccf60 con 0x7f53f00ce600 2023-11-28T20:35:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.593+0000 7f53f5a43640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f53f006aa40 0x7f53f00c93b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57366/0 (socket says 172.21.15.45:57366) 2023-11-28T20:35:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.593+0000 7f53f5a43640 1 -- 172.21.15.45:0/3987330956 learned_addr learned my addr 172.21.15.45:0/3987330956 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.593+0000 7f53f5242640 1 --2- 172.21.15.45:0/3987330956 >> v2:172.21.15.190:3300/0 conn(0x7f53f006dc50 0x7f53f00cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.593+0000 7f53f4a41640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f00ce600 0x7f53f006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.593+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131434647 0 0) 0x7f53e8001110 con 0x7f53f006aa40 2023-11-28T20:35:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.593+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53dc003520 con 0x7f53f006aa40 2023-11-28T20:35:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566769686 0 0) 0x7f53dc003520 con 0x7f53f006aa40 2023-11-28T20:35:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f00ce600 msgr2=0x7f53f006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f00ce600 0x7f53f006a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 >> v2:172.21.15.190:3300/0 conn(0x7f53f006dc50 msgr2=0x7f53f00cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 --2- 172.21.15.45:0/3987330956 >> v2:172.21.15.190:3300/0 conn(0x7f53f006dc50 0x7f53f00cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00cd930 con 0x7f53f006aa40 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f53e8001800 con 0x7f53f006aa40 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 >> v1:172.21.15.45:6789/0 conn(0x7f53f006aa40 legacy=0x7f53f00c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f00ce600 0x7f53dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53dc004d00 0x7f53dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f006aed0 con 0x7f53dc003970 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f00ccf60 con 0x7f53dc004d00 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc00b980 con 0x7f53f00ce600 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.594+0000 7f53f5242640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.595+0000 7f53f5242640 1 -- 172.21.15.45:0/3987330956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f00ce600 msgr2=0x7f53dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.595+0000 7f53f5242640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f00ce600 0x7f53dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.595+0000 7f53f5242640 1 -- 172.21.15.45:0/3987330956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53dc004d00 msgr2=0x7f53dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.595+0000 7f53f5242640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53dc004d00 0x7f53dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.595+0000 7f53f5242640 1 -- 172.21.15.45:0/3987330956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc003520 con 0x7f53dc003970 2023-11-28T20:35:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.596+0000 7f53f5242640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53dc003d90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009fc0 tx=0x7f53dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29888c30e170bef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.596+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec02cc30 con 0x7f53dc003970 2023-11-28T20:35:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.596+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ec00f040 con 0x7f53dc003970 2023-11-28T20:35:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.596+0000 7f53e7fff640 1 -- 172.21.15.45:0/3987330956 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec034440 con 0x7f53dc003970 2023-11-28T20:35:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.597+0000 7f53f74a6640 1 -- 172.21.15.45:0/3987330956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 msgr2=0x7f53dc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.597+0000 7f53f74a6640 1 --2- 172.21.15.45:0/3987330956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53dc003d90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009fc0 tx=0x7f53dc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.597+0000 7f53f5a43640 1 -- 172.21.15.45:0/3987330956 reap_dead start 2023-11-28T20:35:06.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.597+0000 7f53f74a6640 1 -- 172.21.15.45:0/3987330956 shutdown_connections 2023-11-28T20:35:06.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.597+0000 7f53f74a6640 1 -- 172.21.15.45:0/3987330956 >> 172.21.15.45:0/3987330956 conn(0x7f53f00ced60 msgr2=0x7f53f00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:06.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.597+0000 7f53f74a6640 1 -- 172.21.15.45:0/3987330956 shutdown_connections 2023-11-28T20:35:06.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.597+0000 7f53f74a6640 1 -- 172.21.15.45:0/3987330956 wait complete. 2023-11-28T20:35:06.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.599+0000 7f53f74a6640 1 Processor -- start 2023-11-28T20:35:06.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.599+0000 7f53f74a6640 1 -- start start 2023-11-28T20:35:06.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f74a6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53f00c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f74a6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f009c6b0 0x7f53f009cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f74a6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f0099c00 0x7f53f009a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f74a6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f009a5c0 con 0x7f53dc003970 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f74a6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f00c7c30 con 0x7f53f009c6b0 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f74a6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f00c7020 con 0x7f53f0099c00 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f5242640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53f00c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f5242640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53f00c80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47806/0 (socket says 172.21.15.45:47806) 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f5242640 1 -- 172.21.15.45:0/1324835722 learned_addr learned my addr 172.21.15.45:0/1324835722 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f4a41640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f009c6b0 0x7f53f009cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f5242640 1 -- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f0099c00 msgr2=0x7f53f009a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f5242640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f0099c00 0x7f53f009a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f5242640 1 -- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f009c6b0 msgr2=0x7f53f009cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f5242640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f009c6b0 0x7f53f009cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.600+0000 7f53f5242640 1 -- 172.21.15.45:0/1324835722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00c7300 con 0x7f53dc003970 2023-11-28T20:35:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.601+0000 7f53f5242640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53f00c80c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f53ec037040 tx=0x7f53ec009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.601+0000 7f53e5ffb640 1 -- 172.21.15.45:0/1324835722 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec034dd0 con 0x7f53dc003970 2023-11-28T20:35:06.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.601+0000 7f53e5ffb640 1 -- 172.21.15.45:0/1324835722 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ec040450 con 0x7f53dc003970 2023-11-28T20:35:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.601+0000 7f53e5ffb640 1 -- 172.21.15.45:0/1324835722 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec0407c0 con 0x7f53dc003970 2023-11-28T20:35:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.601+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f00c75e0 con 0x7f53dc003970 2023-11-28T20:35:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.601+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f01547e0 con 0x7f53dc003970 2023-11-28T20:35:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.602+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f00733d0 con 0x7f53dc003970 2023-11-28T20:35:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.603+0000 7f53e5ffb640 1 -- 172.21.15.45:0/1324835722 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f53ec040920 con 0x7f53dc003970 2023-11-28T20:35:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.604+0000 7f53e5ffb640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f53d406ae40 0x7f53d406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:06.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.605+0000 7f53e5ffb640 1 -- 172.21.15.45:0/1324835722 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f53ec0ac2a0 con 0x7f53dc003970 2023-11-28T20:35:06.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.605+0000 7f53f4a41640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f53d406ae40 0x7f53d406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:06.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.606+0000 7f53f4a41640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f53d406ae40 0x7f53d406d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f53f0161c00 tx=0x7f53e0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:06.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.609+0000 7f53e5ffb640 1 -- 172.21.15.45:0/1324835722 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53ec076ef0 con 0x7f53dc003970 2023-11-28T20:35:06.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.801+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f53f00c9120 con 0x7f53dc003970 2023-11-28T20:35:06.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.802+0000 7f53e5ffb640 1 -- 172.21.15.45:0/1324835722 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f53ec07a510 con 0x7f53dc003970 2023-11-28T20:35:06.803 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:35:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.804+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f53d406ae40 msgr2=0x7f53d406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.804+0000 7f53f74a6640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f53d406ae40 0x7f53d406d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f53f0161c00 tx=0x7f53e0008040 comp rx=0 tx=0).stop 2023-11-28T20:35:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.804+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 msgr2=0x7f53f00c80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.804+0000 7f53f74a6640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53f00c80c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f53ec037040 tx=0x7f53ec009750 comp rx=0 tx=0).stop 2023-11-28T20:35:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.805+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 shutdown_connections 2023-11-28T20:35:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.805+0000 7f53f74a6640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f53d406ae40 0x7f53d406d300 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.805+0000 7f53f74a6640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53f0099c00 0x7f53f009a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.805+0000 7f53f74a6640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f009c6b0 0x7f53f009cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.805+0000 7f53f74a6640 1 --2- 172.21.15.45:0/1324835722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53dc003970 0x7f53f00c80c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.805+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 >> 172.21.15.45:0/1324835722 conn(0x7f53f00ced60 msgr2=0x7f53f015fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.805+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 shutdown_connections 2023-11-28T20:35:06.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:06.805+0000 7f53f74a6640 1 -- 172.21.15.45:0/1324835722 wait complete. 2023-11-28T20:35:06.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:35:07.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.053+0000 7fd6ebead640 1 Processor -- start 2023-11-28T20:35:07.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.053+0000 7fd6ebead640 1 -- start start 2023-11-28T20:35:07.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.054+0000 7fd6ebead640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd6e415e840 0x7fd6e406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.054+0000 7fd6ebead640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd6e415f720 0x7fd6e40717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.054+0000 7fd6ebead640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e415ecd0 con 0x7fd6e406a500 2023-11-28T20:35:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.054+0000 7fd6ebead640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e4071ce0 con 0x7fd6e415e840 2023-11-28T20:35:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.054+0000 7fd6ebead640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e415f290 con 0x7fd6e415f720 2023-11-28T20:35:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6e9c49640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd6e415e840 0x7fd6e406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6e9c49640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd6e415e840 0x7fd6e406c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47810/0 (socket says 172.21.15.45:47810) 2023-11-28T20:35:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6e9c49640 1 -- 172.21.15.45:0/3300341137 learned_addr learned my addr 172.21.15.45:0/3300341137 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:07.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6ea44a640 1 --1- 172.21.15.45:0/3300341137 >> v1:172.21.15.45:6789/0 conn(0x7fd6e406a500 0x7fd6e40cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57378/0 (socket says 172.21.15.45:57378) 2023-11-28T20:35:07.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6e9448640 1 --2- 172.21.15.45:0/3300341137 >> v2:172.21.15.190:3300/0 conn(0x7fd6e415f720 0x7fd6e40717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6e9c49640 1 -- 172.21.15.45:0/3300341137 >> v2:172.21.15.190:3300/0 conn(0x7fd6e415f720 msgr2=0x7fd6e40717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:07.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6e9c49640 1 --2- 172.21.15.45:0/3300341137 >> v2:172.21.15.190:3300/0 conn(0x7fd6e415f720 0x7fd6e40717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6e9c49640 1 -- 172.21.15.45:0/3300341137 >> v1:172.21.15.45:6789/0 conn(0x7fd6e406a500 legacy=0x7fd6e40cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6e9c49640 1 -- 172.21.15.45:0/3300341137 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e415fb60 con 0x7fd6e415e840 2023-11-28T20:35:07.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.055+0000 7fd6e9448640 1 --2- 172.21.15.45:0/3300341137 >> v2:172.21.15.190:3300/0 conn(0x7fd6e415f720 0x7fd6e40717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:07.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.056+0000 7fd6e9c49640 1 --2- 172.21.15.45:0/3300341137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 0x7fd6e406c350 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd6e406e6d0 tx=0x7fd6e002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975adf37c08094fc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.056+0000 7fd6e8c47640 1 -- 172.21.15.45:0/3300341137 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0030870 con 0x7fd6e415e840 2023-11-28T20:35:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6e8c47640 1 -- 172.21.15.45:0/3300341137 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e00309d0 con 0x7fd6e415e840 2023-11-28T20:35:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6e8c47640 1 -- 172.21.15.45:0/3300341137 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0030d60 con 0x7fd6e415e840 2023-11-28T20:35:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6ebead640 1 -- 172.21.15.45:0/3300341137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 msgr2=0x7fd6e406c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6ebead640 1 --2- 172.21.15.45:0/3300341137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 0x7fd6e406c350 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd6e406e6d0 tx=0x7fd6e002b9b0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6ebead640 1 -- 172.21.15.45:0/3300341137 shutdown_connections 2023-11-28T20:35:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6ebead640 1 --2- 172.21.15.45:0/3300341137 >> v2:172.21.15.190:3300/0 conn(0x7fd6e415f720 0x7fd6e40717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6ebead640 1 --2- 172.21.15.45:0/3300341137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 0x7fd6e406c350 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6ebead640 1 -- 172.21.15.45:0/3300341137 >> 172.21.15.45:0/3300341137 conn(0x7fd6e40ced60 msgr2=0x7fd6e40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:07.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6ebead640 1 -- 172.21.15.45:0/3300341137 shutdown_connections 2023-11-28T20:35:07.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.057+0000 7fd6ebead640 1 -- 172.21.15.45:0/3300341137 wait complete. 2023-11-28T20:35:07.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.059+0000 7fd6ebead640 1 Processor -- start 2023-11-28T20:35:07.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.059+0000 7fd6ebead640 1 -- start start 2023-11-28T20:35:07.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.060+0000 7fd6ebead640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6e406a500 0x7fd6e4157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.060+0000 7fd6ebead640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 0x7fd6e41f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.060+0000 7fd6ebead640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6e415f720 0x7fd6e41f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.060+0000 7fd6ebead640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40b3b30 con 0x7fd6e415e840 2023-11-28T20:35:07.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.060+0000 7fd6ebead640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40b3e10 con 0x7fd6e406a500 2023-11-28T20:35:07.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.060+0000 7fd6ebead640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40b40f0 con 0x7fd6e415f720 2023-11-28T20:35:07.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.061+0000 7fd6e9448640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 0x7fd6e41f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.061+0000 7fd6e9448640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 0x7fd6e41f3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47818/0 (socket says 172.21.15.45:47818) 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.061+0000 7fd6e9448640 1 -- 172.21.15.45:0/1905183171 learned_addr learned my addr 172.21.15.45:0/1905183171 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.061+0000 7fd6e9448640 1 -- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6e415f720 msgr2=0x7fd6e41f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.061+0000 7fd6e9448640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6e415f720 0x7fd6e41f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.061+0000 7fd6e9448640 1 -- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6e406a500 msgr2=0x7fd6e4157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.061+0000 7fd6e9448640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6e406a500 0x7fd6e4157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.061+0000 7fd6e9448640 1 -- 172.21.15.45:0/1905183171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e00094d0 con 0x7fd6e415e840 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.061+0000 7fd6e9448640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 0x7fd6e41f3390 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c7f0 tx=0x7fd6d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.062+0000 7fd6da7fc640 1 -- 172.21.15.45:0/1905183171 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4007820 con 0x7fd6e415e840 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.062+0000 7fd6da7fc640 1 -- 172.21.15.45:0/1905183171 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4007980 con 0x7fd6e415e840 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.062+0000 7fd6da7fc640 1 -- 172.21.15.45:0/1905183171 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4007cf0 con 0x7fd6e415e840 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.062+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e40b43d0 con 0x7fd6e415e840 2023-11-28T20:35:07.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.062+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e41f3e10 con 0x7fd6e415e840 2023-11-28T20:35:07.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.063+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e40c4150 con 0x7fd6e415e840 2023-11-28T20:35:07.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.064+0000 7fd6da7fc640 1 -- 172.21.15.45:0/1905183171 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd6d40113f0 con 0x7fd6e415e840 2023-11-28T20:35:07.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.065+0000 7fd6da7fc640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd6c806ae40 0x7fd6c806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.065+0000 7fd6da7fc640 1 -- 172.21.15.45:0/1905183171 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd6d4089d10 con 0x7fd6e415e840 2023-11-28T20:35:07.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.066+0000 7fd6e9c49640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd6c806ae40 0x7fd6c806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.067+0000 7fd6e9c49640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd6c806ae40 0x7fd6c806d300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0036000 tx=0x7fd6e003b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:07.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.070+0000 7fd6da7fc640 1 -- 172.21.15.45:0/1905183171 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6d4054a00 con 0x7fd6e415e840 2023-11-28T20:35:07.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.269+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd6e40733d0 con 0x7fd6e415e840 2023-11-28T20:35:07.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.270+0000 7fd6da7fc640 1 -- 172.21.15.45:0/1905183171 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd6d401f0a0 con 0x7fd6e415e840 2023-11-28T20:35:07.271 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:07.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.272+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd6c806ae40 msgr2=0x7fd6c806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.272+0000 7fd6ebead640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd6c806ae40 0x7fd6c806d300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0036000 tx=0x7fd6e003b040 comp rx=0 tx=0).stop 2023-11-28T20:35:07.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.272+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 msgr2=0x7fd6e41f3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.272+0000 7fd6ebead640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 0x7fd6e41f3390 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c7f0 tx=0x7fd6d400ccb0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.273+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 shutdown_connections 2023-11-28T20:35:07.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.273+0000 7fd6ebead640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd6c806ae40 0x7fd6c806d300 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.273+0000 7fd6ebead640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6e415f720 0x7fd6e41f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.273+0000 7fd6ebead640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e415e840 0x7fd6e41f3390 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.273+0000 7fd6ebead640 1 --2- 172.21.15.45:0/1905183171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6e406a500 0x7fd6e4157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.273+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 >> 172.21.15.45:0/1905183171 conn(0x7fd6e40ced60 msgr2=0x7fd6e406da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:07.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.273+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 shutdown_connections 2023-11-28T20:35:07.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.273+0000 7fd6ebead640 1 -- 172.21.15.45:0/1905183171 wait complete. 2023-11-28T20:35:07.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:35:07.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.525+0000 7fa40a51a640 1 Processor -- start 2023-11-28T20:35:07.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.526+0000 7fa40a51a640 1 -- start start 2023-11-28T20:35:07.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.528+0000 7fa40a51a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa404161540 0x7fa4040b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.528+0000 7fa40a51a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40406c6e0 con 0x7fa40406c1d0 2023-11-28T20:35:07.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.528+0000 7fa40a51a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40406c9c0 con 0x7fa4040cf610 2023-11-28T20:35:07.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.528+0000 7fa40a51a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa40406cca0 con 0x7fa404161540 2023-11-28T20:35:07.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.528+0000 7fa408d17640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa40406c1d0 0x7fa4040cef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57388/0 (socket says 172.21.15.45:57388) 2023-11-28T20:35:07.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.528+0000 7fa408d17640 1 -- 172.21.15.45:0/3919214074 learned_addr learned my addr 172.21.15.45:0/3919214074 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:07.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa409518640 1 --2- 172.21.15.45:0/3919214074 >> v2:172.21.15.190:3300/0 conn(0x7fa404161540 0x7fa4040b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552816544 0 0) 0x7fa40406c6e0 con 0x7fa40406c1d0 2023-11-28T20:35:07.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f0003520 con 0x7fa40406c1d0 2023-11-28T20:35:07.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 38498990 0 0) 0x7fa3f0003520 con 0x7fa40406c1d0 2023-11-28T20:35:07.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 >> v2:172.21.15.190:3300/0 conn(0x7fa404161540 msgr2=0x7fa4040b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:07.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 --2- 172.21.15.45:0/3919214074 >> v2:172.21.15.190:3300/0 conn(0x7fa404161540 0x7fa4040b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 >> v1:172.21.15.190:6789/0 conn(0x7fa4040cf610 legacy=0x7fa4040704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40406cf80 con 0x7fa40406c1d0 2023-11-28T20:35:07.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fa3f4001670 con 0x7fa40406c1d0 2023-11-28T20:35:07.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 >> v1:172.21.15.45:6789/0 conn(0x7fa40406c1d0 legacy=0x7fa4040cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00039c0 0x7fa3f0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3f00047c0 0x7fa3f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3f00051d0 0x7fa3f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40406cca0 con 0x7fa3f00047c0 2023-11-28T20:35:07.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f000c120 con 0x7fa3f00039c0 2023-11-28T20:35:07.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.529+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f000c2e0 con 0x7fa3f00051d0 2023-11-28T20:35:07.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa408d17640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3f00047c0 0x7fa3f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa409518640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00039c0 0x7fa3f0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa409d19640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3f00051d0 0x7fa3f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa408d17640 1 -- 172.21.15.45:0/3919214074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3f00051d0 msgr2=0x7fa3f000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:07.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa408d17640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3f00051d0 0x7fa3f000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa408d17640 1 -- 172.21.15.45:0/3919214074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00039c0 msgr2=0x7fa3f0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:07.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa408d17640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00039c0 0x7fa3f0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa408d17640 1 -- 172.21.15.45:0/3919214074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0003520 con 0x7fa3f00047c0 2023-11-28T20:35:07.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa409518640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00039c0 0x7fa3f0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:07.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.530+0000 7fa409d19640 1 -- 172.21.15.45:0/3919214074 reap_dead start 2023-11-28T20:35:07.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.531+0000 7fa408d17640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3f00047c0 0x7fa3f0004c40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4001eb0 tx=0x7fa3f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2a0ad49bb43c4b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.532+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4005ad0 con 0x7fa3f00047c0 2023-11-28T20:35:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.532+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4005c30 con 0x7fa3f00047c0 2023-11-28T20:35:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.532+0000 7fa3fbfff640 1 -- 172.21.15.45:0/3919214074 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f402c790 con 0x7fa3f00047c0 2023-11-28T20:35:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.532+0000 7fa40a51a640 1 -- 172.21.15.45:0/3919214074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3f00047c0 msgr2=0x7fa3f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.532+0000 7fa40a51a640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3f00047c0 0x7fa3f0004c40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4001eb0 tx=0x7fa3f00057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.532+0000 7fa40a51a640 1 -- 172.21.15.45:0/3919214074 shutdown_connections 2023-11-28T20:35:07.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.532+0000 7fa40a51a640 1 --2- 172.21.15.45:0/3919214074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3f00047c0 0x7fa3f0004c40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.532+0000 7fa40a51a640 1 -- 172.21.15.45:0/3919214074 >> 172.21.15.45:0/3919214074 conn(0x7fa40406a710 msgr2=0x7fa40406ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:07.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.533+0000 7fa40a51a640 1 -- 172.21.15.45:0/3919214074 shutdown_connections 2023-11-28T20:35:07.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.533+0000 7fa40a51a640 1 -- 172.21.15.45:0/3919214074 wait complete. 2023-11-28T20:35:07.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.534+0000 7fa40a51a640 1 Processor -- start 2023-11-28T20:35:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.535+0000 7fa40a51a640 1 -- start start 2023-11-28T20:35:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.535+0000 7fa40a51a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00047c0 0x7fa404158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.535+0000 7fa40a51a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4040cde90 0x7fa4040aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.535+0000 7fa40a51a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4040ce310 0x7fa4040af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.535+0000 7fa40a51a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40415d280 con 0x7fa4040cde90 2023-11-28T20:35:07.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.535+0000 7fa40a51a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40415d460 con 0x7fa3f00047c0 2023-11-28T20:35:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa40a51a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40415d740 con 0x7fa4040ce310 2023-11-28T20:35:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa408d17640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4040cde90 0x7fa4040aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa408d17640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4040cde90 0x7fa4040aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47836/0 (socket says 172.21.15.45:47836) 2023-11-28T20:35:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa408d17640 1 -- 172.21.15.45:0/4014211376 learned_addr learned my addr 172.21.15.45:0/4014211376 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:07.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa409518640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00047c0 0x7fa404158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa409d19640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4040ce310 0x7fa4040af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa408d17640 1 -- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4040ce310 msgr2=0x7fa4040af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa408d17640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4040ce310 0x7fa4040af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa408d17640 1 -- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00047c0 msgr2=0x7fa404158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa408d17640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00047c0 0x7fa404158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa408d17640 1 -- 172.21.15.45:0/4014211376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40415da20 con 0x7fa4040cde90 2023-11-28T20:35:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa409d19640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4040ce310 0x7fa4040af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.536+0000 7fa409518640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00047c0 0x7fa404158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.537+0000 7fa408d17640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4040cde90 0x7fa4040aef70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4000c00 tx=0x7fa3f4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.537+0000 7fa3f9ffb640 1 -- 172.21.15.45:0/4014211376 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f403c040 con 0x7fa4040cde90 2023-11-28T20:35:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.537+0000 7fa3f9ffb640 1 -- 172.21.15.45:0/4014211376 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f402cdd0 con 0x7fa4040cde90 2023-11-28T20:35:07.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.537+0000 7fa3f9ffb640 1 -- 172.21.15.45:0/4014211376 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4007290 con 0x7fa4040cde90 2023-11-28T20:35:07.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.537+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40415dd00 con 0x7fa4040cde90 2023-11-28T20:35:07.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.537+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40415e500 con 0x7fa4040cde90 2023-11-28T20:35:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.538+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40409d0f0 con 0x7fa4040cde90 2023-11-28T20:35:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.539+0000 7fa3f9ffb640 1 -- 172.21.15.45:0/4014211376 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa3f4032070 con 0x7fa4040cde90 2023-11-28T20:35:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.540+0000 7fa3f9ffb640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa3e806ae70 0x7fa3e806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.541+0000 7fa3f9ffb640 1 -- 172.21.15.45:0/4014211376 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa3f40aad20 con 0x7fa4040cde90 2023-11-28T20:35:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.541+0000 7fa409518640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa3e806ae70 0x7fa3e806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.542+0000 7fa409518640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa3e806ae70 0x7fa3e806d330 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa400009af0 tx=0x7fa400008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:07.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.545+0000 7fa3f9ffb640 1 -- 172.21.15.45:0/4014211376 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3f4075a10 con 0x7fa4040cde90 2023-11-28T20:35:07.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.754+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa40415d880 con 0x7fa4040cde90 2023-11-28T20:35:07.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.755+0000 7fa3f9ffb640 1 -- 172.21.15.45:0/4014211376 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa40415d880 con 0x7fa4040cde90 2023-11-28T20:35:07.758 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2023-11-28T20:35:07.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa3e806ae70 msgr2=0x7fa3e806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa3e806ae70 0x7fa3e806d330 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa400009af0 tx=0x7fa400008040 comp rx=0 tx=0).stop 2023-11-28T20:35:07.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4040cde90 msgr2=0x7fa4040aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:07.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4040cde90 0x7fa4040aef70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4000c00 tx=0x7fa3f4001930 comp rx=0 tx=0).stop 2023-11-28T20:35:07.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 shutdown_connections 2023-11-28T20:35:07.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fa3e806ae70 0x7fa3e806d330 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4040ce310 0x7fa4040af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4040cde90 0x7fa4040aef70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 --2- 172.21.15.45:0/4014211376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3f00047c0 0x7fa404158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:07.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 >> 172.21.15.45:0/4014211376 conn(0x7fa40406a710 msgr2=0x7fa4040ad090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:07.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 shutdown_connections 2023-11-28T20:35:07.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:07.758+0000 7fa40a51a640 1 -- 172.21.15.45:0/4014211376 wait complete. 2023-11-28T20:35:07.780 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T20:35:07.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:07.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-11-28T20:35:07.902 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:35:07.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:07.902 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T20:35:07.958 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:35:07.958 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:08.022 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:35:08.022 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:08.078 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:35:08.078 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:08.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.361+0000 7f6d26432640 1 Processor -- start 2023-11-28T20:35:08.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.361+0000 7f6d26432640 1 -- start start 2023-11-28T20:35:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.363+0000 7f6d26432640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6d2014b090 0x7f6d200b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.363+0000 7f6d26432640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d200ae590 0x7f6d200b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.363+0000 7f6d26432640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2006c060 0x7f6d20165250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.363+0000 7f6d26432640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d200abf10 con 0x7f6d200ae590 2023-11-28T20:35:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.363+0000 7f6d26432640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2006db90 con 0x7f6d2014b090 2023-11-28T20:35:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.363+0000 7f6d26432640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2006d2f0 con 0x7f6d2006c060 2023-11-28T20:35:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d24c2f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d200ae590 0x7f6d200b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d24c2f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d200ae590 0x7f6d200b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47842/0 (socket says 172.21.15.45:47842) 2023-11-28T20:35:08.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d24c2f640 1 -- 172.21.15.45:0/1161352436 learned_addr learned my addr 172.21.15.45:0/1161352436 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:08.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d25430640 1 --2- 172.21.15.45:0/1161352436 >> v2:172.21.15.190:3300/0 conn(0x7f6d2014b090 0x7f6d200b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d25c31640 1 --2- 172.21.15.45:0/1161352436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2006c060 0x7f6d20165250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d24c2f640 1 -- 172.21.15.45:0/1161352436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2006c060 msgr2=0x7f6d20165250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:08.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d24c2f640 1 --2- 172.21.15.45:0/1161352436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2006c060 0x7f6d20165250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d24c2f640 1 -- 172.21.15.45:0/1161352436 >> v2:172.21.15.190:3300/0 conn(0x7f6d2014b090 msgr2=0x7f6d200b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:08.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d24c2f640 1 --2- 172.21.15.45:0/1161352436 >> v2:172.21.15.190:3300/0 conn(0x7f6d2014b090 0x7f6d200b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d24c2f640 1 -- 172.21.15.45:0/1161352436 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2006d5d0 con 0x7f6d200ae590 2023-11-28T20:35:08.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.364+0000 7f6d25430640 1 --2- 172.21.15.45:0/1161352436 >> v2:172.21.15.190:3300/0 conn(0x7f6d2014b090 0x7f6d200b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:08.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.365+0000 7f6d24c2f640 1 --2- 172.21.15.45:0/1161352436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 0x7f6d200b22d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6d2006e3c0 tx=0x7f6d1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fbbfd041ed7e06 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:08.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.366+0000 7f6d13fff640 1 -- 172.21.15.45:0/1161352436 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14031870 con 0x7f6d200ae590 2023-11-28T20:35:08.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.366+0000 7f6d13fff640 1 -- 172.21.15.45:0/1161352436 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d140319d0 con 0x7f6d200ae590 2023-11-28T20:35:08.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.366+0000 7f6d13fff640 1 -- 172.21.15.45:0/1161352436 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14031d60 con 0x7f6d200ae590 2023-11-28T20:35:08.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.367+0000 7f6d26432640 1 -- 172.21.15.45:0/1161352436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 msgr2=0x7f6d200b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:08.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.367+0000 7f6d26432640 1 --2- 172.21.15.45:0/1161352436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 0x7f6d200b22d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6d2006e3c0 tx=0x7f6d1402b910 comp rx=0 tx=0).stop 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.367+0000 7f6d26432640 1 -- 172.21.15.45:0/1161352436 shutdown_connections 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.367+0000 7f6d26432640 1 --2- 172.21.15.45:0/1161352436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2006c060 0x7f6d20165250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.367+0000 7f6d26432640 1 --2- 172.21.15.45:0/1161352436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 0x7f6d200b22d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.367+0000 7f6d26432640 1 --2- 172.21.15.45:0/1161352436 >> v2:172.21.15.190:3300/0 conn(0x7f6d2014b090 0x7f6d200b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.367+0000 7f6d26432640 1 -- 172.21.15.45:0/1161352436 >> 172.21.15.45:0/1161352436 conn(0x7f6d2006ab30 msgr2=0x7f6d200b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.367+0000 7f6d26432640 1 -- 172.21.15.45:0/1161352436 shutdown_connections 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.367+0000 7f6d26432640 1 -- 172.21.15.45:0/1161352436 wait complete. 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.369+0000 7f6d26432640 1 Processor -- start 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.369+0000 7f6d26432640 1 -- start start 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d26432640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d2006c060 0x7f6d201431e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d26432640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 0x7f6d201f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d26432640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2014b090 0x7f6d20144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d26432640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d20099ba0 con 0x7f6d200ae590 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d26432640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d20145ae0 con 0x7f6d2006c060 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d26432640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d20145dc0 con 0x7f6d2014b090 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d24c2f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 0x7f6d201f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d24c2f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 0x7f6d201f73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47848/0 (socket says 172.21.15.45:47848) 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d24c2f640 1 -- 172.21.15.45:0/1140038383 learned_addr learned my addr 172.21.15.45:0/1140038383 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d25430640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d2006c060 0x7f6d201431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.371+0000 7f6d25c31640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2014b090 0x7f6d20144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d24c2f640 1 -- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2014b090 msgr2=0x7f6d20144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d24c2f640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2014b090 0x7f6d20144c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d24c2f640 1 -- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d2006c060 msgr2=0x7f6d201431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:08.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d24c2f640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d2006c060 0x7f6d201431e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d24c2f640 1 -- 172.21.15.45:0/1140038383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d140094d0 con 0x7f6d200ae590 2023-11-28T20:35:08.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d25c31640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2014b090 0x7f6d20144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:08.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d24c2f640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 0x7f6d201f73f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6d14000c00 tx=0x7f6d140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:08.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1140038383 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1403aca0 con 0x7f6d200ae590 2023-11-28T20:35:08.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1140038383 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1403ae00 con 0x7f6d200ae590 2023-11-28T20:35:08.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1140038383 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14044630 con 0x7f6d200ae590 2023-11-28T20:35:08.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.372+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d201460a0 con 0x7f6d200ae590 2023-11-28T20:35:08.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.373+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d20143720 con 0x7f6d200ae590 2023-11-28T20:35:08.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.374+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d200b2d00 con 0x7f6d200ae590 2023-11-28T20:35:08.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.375+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1140038383 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6d1404d450 con 0x7f6d200ae590 2023-11-28T20:35:08.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.376+0000 7f6d11ffb640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6d0406ae40 0x7f6d0406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.376+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1140038383 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6d1402d040 con 0x7f6d200ae590 2023-11-28T20:35:08.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.376+0000 7f6d25430640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6d0406ae40 0x7f6d0406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.377+0000 7f6d25430640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6d0406ae40 0x7f6d0406d300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c006fd0 tx=0x7f6d1c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:08.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.381+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1140038383 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d14076d10 con 0x7f6d200ae590 2023-11-28T20:35:08.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.575+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d2006cd40 con 0x7f6d200ae590 2023-11-28T20:35:08.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.577+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1140038383 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7f6d14035090 con 0x7f6d200ae590 2023-11-28T20:35:08.579 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:08.579 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6d0406ae40 msgr2=0x7f6d0406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6d0406ae40 0x7f6d0406d300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c006fd0 tx=0x7f6d1c008040 comp rx=0 tx=0).stop 2023-11-28T20:35:08.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 msgr2=0x7f6d201f73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 0x7f6d201f73f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6d14000c00 tx=0x7f6d140046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 shutdown_connections 2023-11-28T20:35:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6d0406ae40 0x7f6d0406d300 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d2014b090 0x7f6d20144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d200ae590 0x7f6d201f73f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 --2- 172.21.15.45:0/1140038383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d2006c060 0x7f6d201431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 >> 172.21.15.45:0/1140038383 conn(0x7f6d2006ab30 msgr2=0x7f6d200af4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:08.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 shutdown_connections 2023-11-28T20:35:08.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.580+0000 7f6d26432640 1 -- 172.21.15.45:0/1140038383 wait complete. 2023-11-28T20:35:08.602 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:35:08.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:35:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.836+0000 7fc81703d640 1 Processor -- start 2023-11-28T20:35:08.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.836+0000 7fc81703d640 1 -- start start 2023-11-28T20:35:08.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.838+0000 7fc81703d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc81006dc50 0x7fc8100cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.838+0000 7fc81703d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8100ce600 0x7fc81006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.838+0000 7fc81703d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81006e030 con 0x7fc81006aa40 2023-11-28T20:35:08.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.838+0000 7fc81703d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81006aed0 con 0x7fc8100ce600 2023-11-28T20:35:08.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.838+0000 7fc81703d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8100ccf60 con 0x7fc81006dc50 2023-11-28T20:35:08.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.838+0000 7fc80ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8100ce600 0x7fc81006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.838+0000 7fc8155da640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc81006aa40 0x7fc8100c93b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57398/0 (socket says 172.21.15.45:57398) 2023-11-28T20:35:08.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.838+0000 7fc8155da640 1 -- 172.21.15.45:0/1091851483 learned_addr learned my addr 172.21.15.45:0/1091851483 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:08.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.839+0000 7fc814dd9640 1 --2- 172.21.15.45:0/1091851483 >> v2:172.21.15.190:3300/0 conn(0x7fc81006dc50 0x7fc8100cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.839+0000 7fc80ffff640 1 -- 172.21.15.45:0/1091851483 >> v2:172.21.15.190:3300/0 conn(0x7fc81006dc50 msgr2=0x7fc8100cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.839+0000 7fc80ffff640 1 --2- 172.21.15.45:0/1091851483 >> v2:172.21.15.190:3300/0 conn(0x7fc81006dc50 0x7fc8100cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.839+0000 7fc80ffff640 1 -- 172.21.15.45:0/1091851483 >> v1:172.21.15.45:6789/0 conn(0x7fc81006aa40 legacy=0x7fc8100c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.839+0000 7fc80ffff640 1 -- 172.21.15.45:0/1091851483 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8100cd930 con 0x7fc8100ce600 2023-11-28T20:35:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.840+0000 7fc80ffff640 1 --2- 172.21.15.45:0/1091851483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8100ce600 0x7fc81006a500 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc81006b230 tx=0x7fc80402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4ab9945690ff0c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc80f7fe640 1 -- 172.21.15.45:0/1091851483 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804031830 con 0x7fc8100ce600 2023-11-28T20:35:08.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc80f7fe640 1 -- 172.21.15.45:0/1091851483 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc804031990 con 0x7fc8100ce600 2023-11-28T20:35:08.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc80f7fe640 1 -- 172.21.15.45:0/1091851483 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804031d20 con 0x7fc8100ce600 2023-11-28T20:35:08.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc81703d640 1 -- 172.21.15.45:0/1091851483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8100ce600 msgr2=0x7fc81006a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:08.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc81703d640 1 --2- 172.21.15.45:0/1091851483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8100ce600 0x7fc81006a500 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc81006b230 tx=0x7fc80402b8d0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc81703d640 1 -- 172.21.15.45:0/1091851483 shutdown_connections 2023-11-28T20:35:08.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc81703d640 1 --2- 172.21.15.45:0/1091851483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8100ce600 0x7fc81006a500 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc81703d640 1 --2- 172.21.15.45:0/1091851483 >> v2:172.21.15.190:3300/0 conn(0x7fc81006dc50 0x7fc8100cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc81703d640 1 -- 172.21.15.45:0/1091851483 >> 172.21.15.45:0/1091851483 conn(0x7fc8100ced60 msgr2=0x7fc8100cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:08.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc81703d640 1 -- 172.21.15.45:0/1091851483 shutdown_connections 2023-11-28T20:35:08.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.841+0000 7fc81703d640 1 -- 172.21.15.45:0/1091851483 wait complete. 2023-11-28T20:35:08.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.843+0000 7fc81703d640 1 Processor -- start 2023-11-28T20:35:08.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.843+0000 7fc81703d640 1 -- start start 2023-11-28T20:35:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.844+0000 7fc81703d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc81006aa40 0x7fc8100b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.844+0000 7fc81703d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81006dc50 0x7fc8100c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.844+0000 7fc81703d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8100ce600 0x7fc8101522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.844+0000 7fc81703d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100b43f0 con 0x7fc81006dc50 2023-11-28T20:35:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.844+0000 7fc81703d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100c2b30 con 0x7fc81006aa40 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.844+0000 7fc81703d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100b37e0 con 0x7fc8100ce600 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc80ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81006dc50 0x7fc8100c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc80ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81006dc50 0x7fc8100c4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47864/0 (socket says 172.21.15.45:47864) 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc80ffff640 1 -- 172.21.15.45:0/622085869 learned_addr learned my addr 172.21.15.45:0/622085869 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc814dd9640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc81006aa40 0x7fc8100b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc8155da640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8100ce600 0x7fc8101522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc80ffff640 1 -- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8100ce600 msgr2=0x7fc8101522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc80ffff640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8100ce600 0x7fc8101522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc80ffff640 1 -- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc81006aa40 msgr2=0x7fc8100b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc80ffff640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc81006aa40 0x7fc8100b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.845+0000 7fc80ffff640 1 -- 172.21.15.45:0/622085869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804009400 con 0x7fc81006dc50 2023-11-28T20:35:08.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.846+0000 7fc80ffff640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81006dc50 0x7fc8100c4000 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc804009890 tx=0x7fc8040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:08.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.846+0000 7fc80d7fa640 1 -- 172.21.15.45:0/622085869 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8040317d0 con 0x7fc81006dc50 2023-11-28T20:35:08.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.846+0000 7fc80d7fa640 1 -- 172.21.15.45:0/622085869 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80403ba70 con 0x7fc81006dc50 2023-11-28T20:35:08.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.846+0000 7fc80d7fa640 1 -- 172.21.15.45:0/622085869 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80403be20 con 0x7fc81006dc50 2023-11-28T20:35:08.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.846+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8100b3ac0 con 0x7fc81006dc50 2023-11-28T20:35:08.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.846+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8100b4080 con 0x7fc81006dc50 2023-11-28T20:35:08.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.847+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8100733d0 con 0x7fc81006dc50 2023-11-28T20:35:08.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.848+0000 7fc80d7fa640 1 -- 172.21.15.45:0/622085869 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc8040453f0 con 0x7fc81006dc50 2023-11-28T20:35:08.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.849+0000 7fc80d7fa640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7fc06ae40 0x7fc7fc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:08.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.849+0000 7fc80d7fa640 1 -- 172.21.15.45:0/622085869 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc8040abce0 con 0x7fc81006dc50 2023-11-28T20:35:08.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.850+0000 7fc814dd9640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7fc06ae40 0x7fc7fc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:08.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.851+0000 7fc814dd9640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7fc06ae40 0x7fc7fc06d300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc800009a60 tx=0x7fc800003990 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:08.854+0000 7fc80d7fa640 1 -- 172.21.15.45:0/622085869 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc804076ae0 con 0x7fc81006dc50 2023-11-28T20:35:09.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.049+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc81006a500 con 0x7fc81006dc50 2023-11-28T20:35:09.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.050+0000 7fc80d7fa640 1 -- 172.21.15.45:0/622085869 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc80407a100 con 0x7fc81006dc50 2023-11-28T20:35:09.052 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:35:09.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7fc06ae40 msgr2=0x7fc7fc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7fc06ae40 0x7fc7fc06d300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc800009a60 tx=0x7fc800003990 comp rx=0 tx=0).stop 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81006dc50 msgr2=0x7fc8100c4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81006dc50 0x7fc8100c4000 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc804009890 tx=0x7fc8040046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 shutdown_connections 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc7fc06ae40 0x7fc7fc06d300 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8100ce600 0x7fc8101522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81006dc50 0x7fc8100c4000 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 --2- 172.21.15.45:0/622085869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc81006aa40 0x7fc8100b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 >> 172.21.15.45:0/622085869 conn(0x7fc8100ced60 msgr2=0x7fc8100b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.054+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 shutdown_connections 2023-11-28T20:35:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.055+0000 7fc81703d640 1 -- 172.21.15.45:0/622085869 wait complete. 2023-11-28T20:35:09.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:35:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.308+0000 7f2be5650640 1 Processor -- start 2023-11-28T20:35:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.308+0000 7f2be5650640 1 -- start start 2023-11-28T20:35:09.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.310+0000 7f2be5650640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2be015b270 0x7f2be006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.310+0000 7f2be5650640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3420 0x7f2be00ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.310+0000 7f2be5650640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be006d410 con 0x7f2be00b3040 2023-11-28T20:35:09.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.310+0000 7f2be5650640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be00ceaa0 con 0x7f2be015b270 2023-11-28T20:35:09.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.310+0000 7f2be5650640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be006d6f0 con 0x7f2be00b3420 2023-11-28T20:35:09.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.310+0000 7f2bdf7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2be015b270 0x7f2be006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.310+0000 7f2bdf7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2be015b270 0x7f2be006cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47872/0 (socket says 172.21.15.45:47872) 2023-11-28T20:35:09.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.310+0000 7f2bdf7fe640 1 -- 172.21.15.45:0/257823807 learned_addr learned my addr 172.21.15.45:0/257823807 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:09.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.311+0000 7f2be4e4f640 1 --2- 172.21.15.45:0/257823807 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3420 0x7f2be00ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.311+0000 7f2bdf7fe640 1 -- 172.21.15.45:0/257823807 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3420 msgr2=0x7f2be00ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:09.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.311+0000 7f2bdf7fe640 1 --2- 172.21.15.45:0/257823807 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3420 0x7f2be00ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.311+0000 7f2bdf7fe640 1 -- 172.21.15.45:0/257823807 >> v1:172.21.15.190:6789/0 conn(0x7f2be00b3040 legacy=0x7f2be015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:09.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.311+0000 7f2bdf7fe640 1 -- 172.21.15.45:0/257823807 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be006d9d0 con 0x7f2be015b270 2023-11-28T20:35:09.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.312+0000 7f2bdf7fe640 1 --2- 172.21.15.45:0/257823807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 0x7f2be006cda0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009870 tx=0x7f2bc802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30cd30849b248928 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:09.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.312+0000 7f2bdeffd640 1 -- 172.21.15.45:0/257823807 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8031740 con 0x7f2be015b270 2023-11-28T20:35:09.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.312+0000 7f2bdeffd640 1 -- 172.21.15.45:0/257823807 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc80318a0 con 0x7f2be015b270 2023-11-28T20:35:09.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.312+0000 7f2bdeffd640 1 -- 172.21.15.45:0/257823807 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8031c30 con 0x7f2be015b270 2023-11-28T20:35:09.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.313+0000 7f2be5650640 1 -- 172.21.15.45:0/257823807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 msgr2=0x7f2be006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.313+0000 7f2be5650640 1 --2- 172.21.15.45:0/257823807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 0x7f2be006cda0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009870 tx=0x7f2bc802b7e0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.313+0000 7f2be5650640 1 -- 172.21.15.45:0/257823807 shutdown_connections 2023-11-28T20:35:09.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.313+0000 7f2be5650640 1 --2- 172.21.15.45:0/257823807 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3420 0x7f2be00ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.313+0000 7f2be5650640 1 --2- 172.21.15.45:0/257823807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 0x7f2be006cda0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.313+0000 7f2be5650640 1 -- 172.21.15.45:0/257823807 >> 172.21.15.45:0/257823807 conn(0x7f2be00b1050 msgr2=0x7f2be00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:09.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.313+0000 7f2be5650640 1 -- 172.21.15.45:0/257823807 shutdown_connections 2023-11-28T20:35:09.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.313+0000 7f2be5650640 1 -- 172.21.15.45:0/257823807 wait complete. 2023-11-28T20:35:09.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.315+0000 7f2be5650640 1 Processor -- start 2023-11-28T20:35:09.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.315+0000 7f2be5650640 1 -- start start 2023-11-28T20:35:09.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.316+0000 7f2be5650640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3040 0x7f2be01436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.316+0000 7f2be5650640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2be00b3420 0x7f2be0153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.316+0000 7f2be5650640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 0x7f2be0153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.316+0000 7f2be5650640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be0155f50 con 0x7f2be015b270 2023-11-28T20:35:09.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.316+0000 7f2be5650640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be01563e0 con 0x7f2be00b3420 2023-11-28T20:35:09.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.316+0000 7f2be5650640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be0155380 con 0x7f2be00b3040 2023-11-28T20:35:09.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2be4e4f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 0x7f2be0153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2be4e4f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 0x7f2be0153d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47888/0 (socket says 172.21.15.45:47888) 2023-11-28T20:35:09.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2be4e4f640 1 -- 172.21.15.45:0/529514545 learned_addr learned my addr 172.21.15.45:0/529514545 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2bdf7fe640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2be00b3420 0x7f2be0153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2bdffff640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3040 0x7f2be01436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2be4e4f640 1 -- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3040 msgr2=0x7f2be01436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2be4e4f640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3040 0x7f2be01436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2be4e4f640 1 -- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2be00b3420 msgr2=0x7f2be0153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2be4e4f640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2be00b3420 0x7f2be0153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.317+0000 7f2be4e4f640 1 -- 172.21.15.45:0/529514545 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc80094d0 con 0x7f2be015b270 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.318+0000 7f2bdf7fe640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2be00b3420 0x7f2be0153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.318+0000 7f2be4e4f640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 0x7f2be0153d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800d800 tx=0x7f2bd800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.318+0000 7f2bdcff9640 1 -- 172.21.15.45:0/529514545 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd800b6c0 con 0x7f2be015b270 2023-11-28T20:35:09.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.318+0000 7f2bdcff9640 1 -- 172.21.15.45:0/529514545 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd800b840 con 0x7f2be015b270 2023-11-28T20:35:09.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.318+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be0155620 con 0x7f2be015b270 2023-11-28T20:35:09.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.318+0000 7f2bdcff9640 1 -- 172.21.15.45:0/529514545 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd800bbf0 con 0x7f2be015b270 2023-11-28T20:35:09.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.318+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be0155be0 con 0x7f2be015b270 2023-11-28T20:35:09.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.319+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be01560c0 con 0x7f2be015b270 2023-11-28T20:35:09.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.320+0000 7f2bdcff9640 1 -- 172.21.15.45:0/529514545 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2bd800bd50 con 0x7f2be015b270 2023-11-28T20:35:09.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.321+0000 7f2bdcff9640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2bc006aec0 0x7f2bc006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.321+0000 7f2bdcff9640 1 -- 172.21.15.45:0/529514545 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2bd8089f20 con 0x7f2be015b270 2023-11-28T20:35:09.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.322+0000 7f2bdffff640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2bc006aec0 0x7f2bc006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.323+0000 7f2bdffff640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2bc006aec0 0x7f2bc006d380 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0005520 tx=0x7f2bd0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.326+0000 7f2bdcff9640 1 -- 172.21.15.45:0/529514545 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bd8054bf0 con 0x7f2be015b270 2023-11-28T20:35:09.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.520+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2be009d0f0 con 0x7f2be015b270 2023-11-28T20:35:09.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.521+0000 7f2bdcff9640 1 -- 172.21.15.45:0/529514545 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2bd8058210 con 0x7f2be015b270 2023-11-28T20:35:09.522 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:09.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.523+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2bc006aec0 msgr2=0x7f2bc006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.523+0000 7f2be5650640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2bc006aec0 0x7f2bc006d380 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0005520 tx=0x7f2bd0009040 comp rx=0 tx=0).stop 2023-11-28T20:35:09.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.523+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 msgr2=0x7f2be0153d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.523+0000 7f2be5650640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 0x7f2be0153d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800d800 tx=0x7f2bd800dcc0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.524+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 shutdown_connections 2023-11-28T20:35:09.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.524+0000 7f2be5650640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2bc006aec0 0x7f2bc006d380 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.524+0000 7f2be5650640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2be015b270 0x7f2be0153d90 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.524+0000 7f2be5650640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2be00b3420 0x7f2be0153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.524+0000 7f2be5650640 1 --2- 172.21.15.45:0/529514545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2be00b3040 0x7f2be01436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.524+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 >> 172.21.15.45:0/529514545 conn(0x7f2be00b1050 msgr2=0x7f2be00abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:09.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.524+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 shutdown_connections 2023-11-28T20:35:09.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.524+0000 7f2be5650640 1 -- 172.21.15.45:0/529514545 wait complete. 2023-11-28T20:35:09.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:35:09.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.772+0000 7fd6b9f5a640 1 Processor -- start 2023-11-28T20:35:09.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.773+0000 7fd6b9f5a640 1 -- start start 2023-11-28T20:35:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.774+0000 7fd6b9f5a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6b406c350 0x7fd6b415f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.774+0000 7fd6b9f5a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b415ecd0 con 0x7fd6b406ac10 2023-11-28T20:35:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.775+0000 7fd6b9f5a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b415f290 con 0x7fd6b415e840 2023-11-28T20:35:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.775+0000 7fd6b9f5a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4072020 con 0x7fd6b406c350 2023-11-28T20:35:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.775+0000 7fd6abfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd6b406ac10 0x7fd6b40cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57406/0 (socket says 172.21.15.45:57406) 2023-11-28T20:35:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.775+0000 7fd6abfff640 1 -- 172.21.15.45:0/2668145693 learned_addr learned my addr 172.21.15.45:0/2668145693 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.775+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362037506 0 0) 0x7fd6b415ecd0 con 0x7fd6b406ac10 2023-11-28T20:35:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.775+0000 7fd6b3fff640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6b406c350 0x7fd6b415f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.775+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a0003520 con 0x7fd6b406ac10 2023-11-28T20:35:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095153465 0 0) 0x7fd6a0003520 con 0x7fd6b406ac10 2023-11-28T20:35:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6b406c350 msgr2=0x7fd6b415f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6b406c350 0x7fd6b415f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 >> v1:172.21.15.190:6789/0 conn(0x7fd6b415e840 legacy=0x7fd6b406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b4072300 con 0x7fd6b406ac10 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd69c001670 con 0x7fd6b406ac10 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 >> v1:172.21.15.45:6789/0 conn(0x7fd6b406ac10 legacy=0x7fd6b40cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a00039a0 0x7fd6a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6b406c350 0x7fd6a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a0004d00 0x7fd6a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4072020 con 0x7fd6a0004d00 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a000bb90 con 0x7fd6a00039a0 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a000bd50 con 0x7fd6b406c350 2023-11-28T20:35:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b3fff640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a0004d00 0x7fd6a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.776+0000 7fd6b37fe640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a00039a0 0x7fd6a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.777+0000 7fd6abfff640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6b406c350 0x7fd6a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.777+0000 7fd6b3fff640 1 -- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6b406c350 msgr2=0x7fd6a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.777+0000 7fd6b3fff640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6b406c350 0x7fd6a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.777+0000 7fd6b3fff640 1 -- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a00039a0 msgr2=0x7fd6a0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.777+0000 7fd6b3fff640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a00039a0 0x7fd6a0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.777+0000 7fd6b3fff640 1 -- 172.21.15.45:0/2668145693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a0003520 con 0x7fd6a0004d00 2023-11-28T20:35:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.778+0000 7fd6b3fff640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a0004d00 0x7fd6a0005180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac007360 tx=0x7fd6a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfc92b30a14b336 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.778+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac02ccd0 con 0x7fd6a0004d00 2023-11-28T20:35:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.778+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac02ce30 con 0x7fd6a0004d00 2023-11-28T20:35:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.778+0000 7fd6b2ffd640 1 -- 172.21.15.45:0/2668145693 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac034660 con 0x7fd6a0004d00 2023-11-28T20:35:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.779+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/2668145693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a0004d00 msgr2=0x7fd6a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.779+0000 7fd6b9f5a640 1 --2- 172.21.15.45:0/2668145693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a0004d00 0x7fd6a0005180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac007360 tx=0x7fd6a00057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.779+0000 7fd6b3fff640 1 -- 172.21.15.45:0/2668145693 reap_dead start 2023-11-28T20:35:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.779+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/2668145693 shutdown_connections 2023-11-28T20:35:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.779+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/2668145693 >> 172.21.15.45:0/2668145693 conn(0x7fd6b40ced60 msgr2=0x7fd6b40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.779+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/2668145693 shutdown_connections 2023-11-28T20:35:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.779+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/2668145693 wait complete. 2023-11-28T20:35:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.781+0000 7fd6b9f5a640 1 Processor -- start 2023-11-28T20:35:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.781+0000 7fd6b9f5a640 1 -- start start 2023-11-28T20:35:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.782+0000 7fd6b9f5a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a0004d00 0x7fd6b4149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.782+0000 7fd6b9f5a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6b40c2890 0x7fd6b40c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.782+0000 7fd6b9f5a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6b40c2d10 0x7fd6b40c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.782+0000 7fd6b9f5a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40c3270 con 0x7fd6b40c2d10 2023-11-28T20:35:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.782+0000 7fd6b9f5a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40c3550 con 0x7fd6b40c2890 2023-11-28T20:35:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.782+0000 7fd6b9f5a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40c3830 con 0x7fd6a0004d00 2023-11-28T20:35:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.782+0000 7fd6b3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6b40c2d10 0x7fd6b40c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.782+0000 7fd6b3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6b40c2d10 0x7fd6b40c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47898/0 (socket says 172.21.15.45:47898) 2023-11-28T20:35:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.782+0000 7fd6b3fff640 1 -- 172.21.15.45:0/3245587748 learned_addr learned my addr 172.21.15.45:0/3245587748 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6abfff640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6b40c2890 0x7fd6b40c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b37fe640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a0004d00 0x7fd6b4149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b3fff640 1 -- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a0004d00 msgr2=0x7fd6b4149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b3fff640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a0004d00 0x7fd6b4149070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b3fff640 1 -- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6b40c2890 msgr2=0x7fd6b40c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b3fff640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6b40c2890 0x7fd6b40c8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b3fff640 1 -- 172.21.15.45:0/3245587748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b40c3b10 con 0x7fd6b40c2d10 2023-11-28T20:35:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b3fff640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6b40c2d10 0x7fd6b40c8540 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac037040 tx=0x7fd6ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b0ff9640 1 -- 172.21.15.45:0/3245587748 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac044070 con 0x7fd6b40c2d10 2023-11-28T20:35:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b0ff9640 1 -- 172.21.15.45:0/3245587748 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac03e400 con 0x7fd6b40c2d10 2023-11-28T20:35:09.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b0ff9640 1 -- 172.21.15.45:0/3245587748 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac03e770 con 0x7fd6b40c2d10 2023-11-28T20:35:09.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.783+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b40c3df0 con 0x7fd6b40c2d10 2023-11-28T20:35:09.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.784+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b4159c70 con 0x7fd6b40c2d10 2023-11-28T20:35:09.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.785+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b40733d0 con 0x7fd6b40c2d10 2023-11-28T20:35:09.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.786+0000 7fd6b0ff9640 1 -- 172.21.15.45:0/3245587748 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd6ac03e8d0 con 0x7fd6b40c2d10 2023-11-28T20:35:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.787+0000 7fd6b0ff9640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd69406ae40 0x7fd69406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.787+0000 7fd6b0ff9640 1 -- 172.21.15.45:0/3245587748 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd6ac0ab7e0 con 0x7fd6b40c2d10 2023-11-28T20:35:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.787+0000 7fd6b37fe640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd69406ae40 0x7fd69406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.788+0000 7fd6b37fe640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd69406ae40 0x7fd69406d300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4007fd0 tx=0x7fd6a4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.791+0000 7fd6b0ff9640 1 -- 172.21.15.45:0/3245587748 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6ac0766a0 con 0x7fd6b40c2d10 2023-11-28T20:35:09.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.984+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd6b40ce2b0 con 0x7fd6b40c2d10 2023-11-28T20:35:09.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.985+0000 7fd6b0ff9640 1 -- 172.21.15.45:0/3245587748 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd6ac079cc0 con 0x7fd6b40c2d10 2023-11-28T20:35:09.986 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:35:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd69406ae40 msgr2=0x7fd69406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd69406ae40 0x7fd69406d300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4007fd0 tx=0x7fd6a4009040 comp rx=0 tx=0).stop 2023-11-28T20:35:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6b40c2d10 msgr2=0x7fd6b40c8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6b40c2d10 0x7fd6b40c8540 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac037040 tx=0x7fd6ac004ab0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 shutdown_connections 2023-11-28T20:35:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd69406ae40 0x7fd69406d300 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6b40c2d10 0x7fd6b40c8540 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6b40c2890 0x7fd6b40c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 --2- 172.21.15.45:0/3245587748 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a0004d00 0x7fd6b4149070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:09.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.987+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 >> 172.21.15.45:0/3245587748 conn(0x7fd6b40ced60 msgr2=0x7fd6b40ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:09.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.988+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 shutdown_connections 2023-11-28T20:35:09.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:09.988+0000 7fd6b9f5a640 1 -- 172.21.15.45:0/3245587748 wait complete. 2023-11-28T20:35:10.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:35:10.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.251+0000 7f385f2bb640 1 Processor -- start 2023-11-28T20:35:10.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.251+0000 7f385f2bb640 1 -- start start 2023-11-28T20:35:10.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.253+0000 7f385f2bb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f385806ac10 0x7f385806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.253+0000 7f385f2bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815f720 0x7f38580717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.253+0000 7f385f2bb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385815ecd0 con 0x7f385815e840 2023-11-28T20:35:10.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.253+0000 7f385f2bb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3858071ce0 con 0x7f385806ac10 2023-11-28T20:35:10.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.253+0000 7f385f2bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385815fbd0 con 0x7f385815f720 2023-11-28T20:35:10.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.253+0000 7f385c856640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f385806ac10 0x7f385806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.253+0000 7f385c856640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f385806ac10 0x7f385806c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47906/0 (socket says 172.21.15.45:47906) 2023-11-28T20:35:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.254+0000 7f385c856640 1 -- 172.21.15.45:0/856244976 learned_addr learned my addr 172.21.15.45:0/856244976 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.254+0000 7f385d858640 1 --2- 172.21.15.45:0/856244976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815f720 0x7f38580717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.254+0000 7f385c856640 1 -- 172.21.15.45:0/856244976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815f720 msgr2=0x7f38580717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.254+0000 7f385c856640 1 --2- 172.21.15.45:0/856244976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815f720 0x7f38580717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.254+0000 7f385c856640 1 -- 172.21.15.45:0/856244976 >> v1:172.21.15.190:6789/0 conn(0x7f385815e840 legacy=0x7f385806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.254+0000 7f385c856640 1 -- 172.21.15.45:0/856244976 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385815f290 con 0x7f385806ac10 2023-11-28T20:35:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.255+0000 7f385c856640 1 --2- 172.21.15.45:0/856244976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 0x7f385806c350 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3848009870 tx=0x7f384802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d567c23fc86851e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.256+0000 7f384ffff640 1 -- 172.21.15.45:0/856244976 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848030840 con 0x7f385806ac10 2023-11-28T20:35:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.256+0000 7f384ffff640 1 -- 172.21.15.45:0/856244976 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38480309a0 con 0x7f385806ac10 2023-11-28T20:35:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.256+0000 7f384ffff640 1 -- 172.21.15.45:0/856244976 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848030d30 con 0x7f385806ac10 2023-11-28T20:35:10.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.256+0000 7f385f2bb640 1 -- 172.21.15.45:0/856244976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 msgr2=0x7f385806c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.256+0000 7f385f2bb640 1 --2- 172.21.15.45:0/856244976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 0x7f385806c350 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3848009870 tx=0x7f384802b980 comp rx=0 tx=0).stop 2023-11-28T20:35:10.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.257+0000 7f385f2bb640 1 -- 172.21.15.45:0/856244976 shutdown_connections 2023-11-28T20:35:10.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.257+0000 7f385f2bb640 1 --2- 172.21.15.45:0/856244976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815f720 0x7f38580717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.257+0000 7f385f2bb640 1 --2- 172.21.15.45:0/856244976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 0x7f385806c350 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.257+0000 7f385f2bb640 1 -- 172.21.15.45:0/856244976 >> 172.21.15.45:0/856244976 conn(0x7f38580ced60 msgr2=0x7f38580cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:10.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.257+0000 7f385f2bb640 1 -- 172.21.15.45:0/856244976 shutdown_connections 2023-11-28T20:35:10.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.257+0000 7f385f2bb640 1 -- 172.21.15.45:0/856244976 wait complete. 2023-11-28T20:35:10.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.259+0000 7f385f2bb640 1 Processor -- start 2023-11-28T20:35:10.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.259+0000 7f385f2bb640 1 -- start start 2023-11-28T20:35:10.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385f2bb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 0x7f385815d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385f2bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815e840 0x7f38580c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385f2bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f385815f720 0x7f38581e5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385f2bb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38580c9940 con 0x7f385806ac10 2023-11-28T20:35:10.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385f2bb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38580c9c20 con 0x7f385815f720 2023-11-28T20:35:10.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385f2bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38580c9f00 con 0x7f385815e840 2023-11-28T20:35:10.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385d057640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 0x7f385815d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385d057640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 0x7f385815d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47922/0 (socket says 172.21.15.45:47922) 2023-11-28T20:35:10.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385d057640 1 -- 172.21.15.45:0/981717535 learned_addr learned my addr 172.21.15.45:0/981717535 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:10.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385c856640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815e840 0x7f38580c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385d858640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f385815f720 0x7f38581e5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385d057640 1 -- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815e840 msgr2=0x7f38580c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385d057640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815e840 0x7f38580c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385d057640 1 -- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f385815f720 msgr2=0x7f38581e5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385d057640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f385815f720 0x7f38581e5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.261+0000 7f385d057640 1 -- 172.21.15.45:0/981717535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38480094d0 con 0x7f385806ac10 2023-11-28T20:35:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.262+0000 7f385d858640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f385815f720 0x7f38581e5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.262+0000 7f385d057640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 0x7f385815d350 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f385400c450 tx=0x7f385400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.262+0000 7f384dffb640 1 -- 172.21.15.45:0/981717535 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38540052d0 con 0x7f385806ac10 2023-11-28T20:35:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.262+0000 7f384dffb640 1 -- 172.21.15.45:0/981717535 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3854005430 con 0x7f385806ac10 2023-11-28T20:35:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.262+0000 7f384dffb640 1 -- 172.21.15.45:0/981717535 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854012870 con 0x7f385806ac10 2023-11-28T20:35:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.262+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38580ca1e0 con 0x7f385806ac10 2023-11-28T20:35:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.262+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385815d890 con 0x7f385806ac10 2023-11-28T20:35:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.264+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38580733d0 con 0x7f385806ac10 2023-11-28T20:35:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.265+0000 7f384dffb640 1 -- 172.21.15.45:0/981717535 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f38540129d0 con 0x7f385806ac10 2023-11-28T20:35:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.265+0000 7f384dffb640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f383c06ae40 0x7f383c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.266+0000 7f384dffb640 1 -- 172.21.15.45:0/981717535 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f38540895b0 con 0x7f385806ac10 2023-11-28T20:35:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.266+0000 7f385c856640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f383c06ae40 0x7f383c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.267+0000 7f385c856640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f383c06ae40 0x7f383c06d300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3848036000 tx=0x7f3848008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:10.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.270+0000 7f384dffb640 1 -- 172.21.15.45:0/981717535 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3854054200 con 0x7f385806ac10 2023-11-28T20:35:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.464+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f38580ce2b0 con 0x7f385806ac10 2023-11-28T20:35:10.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.465+0000 7f384dffb640 1 -- 172.21.15.45:0/981717535 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3854057820 con 0x7f385806ac10 2023-11-28T20:35:10.466 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f383c06ae40 msgr2=0x7f383c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f383c06ae40 0x7f383c06d300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3848036000 tx=0x7f3848008030 comp rx=0 tx=0).stop 2023-11-28T20:35:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 msgr2=0x7f385815d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 0x7f385815d350 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f385400c450 tx=0x7f385400c910 comp rx=0 tx=0).stop 2023-11-28T20:35:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 shutdown_connections 2023-11-28T20:35:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f383c06ae40 0x7f383c06d300 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f385815f720 0x7f38581e5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f385815e840 0x7f38580c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 --2- 172.21.15.45:0/981717535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f385806ac10 0x7f385815d350 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 >> 172.21.15.45:0/981717535 conn(0x7f38580ced60 msgr2=0x7f38580701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 shutdown_connections 2023-11-28T20:35:10.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.468+0000 7f385f2bb640 1 -- 172.21.15.45:0/981717535 wait complete. 2023-11-28T20:35:10.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:35:10.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.721+0000 7f41fe2d1640 1 Processor -- start 2023-11-28T20:35:10.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.721+0000 7f41fe2d1640 1 -- start start 2023-11-28T20:35:10.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.722+0000 7f41fe2d1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41f815d120 0x7f41f80d2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.722+0000 7f41fe2d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 0x7f41f80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.722+0000 7f41fe2d1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f80d3040 con 0x7f41f80d2a40 2023-11-28T20:35:10.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.722+0000 7f41fe2d1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f80d3320 con 0x7f41f815d120 2023-11-28T20:35:10.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.722+0000 7f41fe2d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f80d3600 con 0x7f41f80d0690 2023-11-28T20:35:10.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.723+0000 7f41fd2cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41f815d120 0x7f41f80d2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.723+0000 7f41fd2cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41f815d120 0x7f41f80d2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47934/0 (socket says 172.21.15.45:47934) 2023-11-28T20:35:10.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.723+0000 7f41fd2cf640 1 -- 172.21.15.45:0/4126275597 learned_addr learned my addr 172.21.15.45:0/4126275597 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:10.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.723+0000 7f41fdad0640 1 --2- 172.21.15.45:0/4126275597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 0x7f41f80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.723+0000 7f41fd2cf640 1 -- 172.21.15.45:0/4126275597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 msgr2=0x7f41f80b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:10.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.723+0000 7f41fd2cf640 1 --2- 172.21.15.45:0/4126275597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 0x7f41f80b6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.723+0000 7f41fd2cf640 1 -- 172.21.15.45:0/4126275597 >> v1:172.21.15.190:6789/0 conn(0x7f41f80d2a40 legacy=0x7f41f80cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.724+0000 7f41fd2cf640 1 -- 172.21.15.45:0/4126275597 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f80d38e0 con 0x7f41f815d120 2023-11-28T20:35:10.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.725+0000 7f41fd2cf640 1 --2- 172.21.15.45:0/4126275597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 0x7f41f80d2500 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009e50 tx=0x7f41f402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac8629ac86542b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:10.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.725+0000 7f41effff640 1 -- 172.21.15.45:0/4126275597 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f402cb90 con 0x7f41f815d120 2023-11-28T20:35:10.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.725+0000 7f41effff640 1 -- 172.21.15.45:0/4126275597 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f402ccf0 con 0x7f41f815d120 2023-11-28T20:35:10.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.725+0000 7f41effff640 1 -- 172.21.15.45:0/4126275597 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4034610 con 0x7f41f815d120 2023-11-28T20:35:10.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.725+0000 7f41fe2d1640 1 -- 172.21.15.45:0/4126275597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 msgr2=0x7f41f80d2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.725+0000 7f41fe2d1640 1 --2- 172.21.15.45:0/4126275597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 0x7f41f80d2500 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009e50 tx=0x7f41f402bff0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.726+0000 7f41fe2d1640 1 -- 172.21.15.45:0/4126275597 shutdown_connections 2023-11-28T20:35:10.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.726+0000 7f41fe2d1640 1 --2- 172.21.15.45:0/4126275597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 0x7f41f80b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.726+0000 7f41fe2d1640 1 --2- 172.21.15.45:0/4126275597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 0x7f41f80d2500 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.726+0000 7f41fe2d1640 1 -- 172.21.15.45:0/4126275597 >> 172.21.15.45:0/4126275597 conn(0x7f41f8070f40 msgr2=0x7f41f8071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:10.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.726+0000 7f41fe2d1640 1 -- 172.21.15.45:0/4126275597 shutdown_connections 2023-11-28T20:35:10.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.726+0000 7f41fe2d1640 1 -- 172.21.15.45:0/4126275597 wait complete. 2023-11-28T20:35:10.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.727+0000 7f41fe2d1640 1 Processor -- start 2023-11-28T20:35:10.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.728+0000 7f41fe2d1640 1 -- start start 2023-11-28T20:35:10.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.728+0000 7f41fe2d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 0x7f41f8157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fe2d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41f80d2a40 0x7f41f8159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fe2d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 0x7f41f80ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fe2d1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f815a760 con 0x7f41f815d120 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fe2d1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f815abf0 con 0x7f41f80d2a40 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fe2d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f8159b50 con 0x7f41f80d0690 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fdad0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 0x7f41f80ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fdad0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 0x7f41f80ca100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47944/0 (socket says 172.21.15.45:47944) 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fdad0640 1 -- 172.21.15.45:0/757635215 learned_addr learned my addr 172.21.15.45:0/757635215 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fd2cf640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 0x7f41f8157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fcace640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41f80d2a40 0x7f41f8159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fdad0640 1 -- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 msgr2=0x7f41f8157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fdad0640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 0x7f41f8157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fdad0640 1 -- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41f80d2a40 msgr2=0x7f41f8159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fdad0640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41f80d2a40 0x7f41f8159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fdad0640 1 -- 172.21.15.45:0/757635215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f4009b30 con 0x7f41f815d120 2023-11-28T20:35:10.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.729+0000 7f41fcace640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41f80d2a40 0x7f41f8159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:10.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.730+0000 7f41fdad0640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 0x7f41f80ca100 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f41f000d800 tx=0x7f41f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:10.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.730+0000 7f41edffb640 1 -- 172.21.15.45:0/757635215 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f000b6c0 con 0x7f41f815d120 2023-11-28T20:35:10.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.730+0000 7f41edffb640 1 -- 172.21.15.45:0/757635215 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f000b840 con 0x7f41f815d120 2023-11-28T20:35:10.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.730+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f8159e30 con 0x7f41f815d120 2023-11-28T20:35:10.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.730+0000 7f41edffb640 1 -- 172.21.15.45:0/757635215 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f000bbf0 con 0x7f41f815d120 2023-11-28T20:35:10.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.730+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f815a3f0 con 0x7f41f815d120 2023-11-28T20:35:10.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.731+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f815a8d0 con 0x7f41f815d120 2023-11-28T20:35:10.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.732+0000 7f41edffb640 1 -- 172.21.15.45:0/757635215 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f41f000bd50 con 0x7f41f815d120 2023-11-28T20:35:10.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.733+0000 7f41edffb640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f41dc06ae40 0x7f41dc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:10.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.733+0000 7f41edffb640 1 -- 172.21.15.45:0/757635215 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f41f0089ed0 con 0x7f41f815d120 2023-11-28T20:35:10.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.734+0000 7f41fd2cf640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f41dc06ae40 0x7f41dc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:10.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.735+0000 7f41fd2cf640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f41dc06ae40 0x7f41dc06d300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f41f4035000 tx=0x7f41f4030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:10.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.738+0000 7f41edffb640 1 -- 172.21.15.45:0/757635215 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41f0054bb0 con 0x7f41f815d120 2023-11-28T20:35:10.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.947+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f41f8071b30 con 0x7f41f815d120 2023-11-28T20:35:10.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.948+0000 7f41edffb640 1 -- 172.21.15.45:0/757635215 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f41f00581d0 con 0x7f41f815d120 2023-11-28T20:35:10.949 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2023-11-28T20:35:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.950+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f41dc06ae40 msgr2=0x7f41dc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f41dc06ae40 0x7f41dc06d300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f41f4035000 tx=0x7f41f4030000 comp rx=0 tx=0).stop 2023-11-28T20:35:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 msgr2=0x7f41f80ca100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 0x7f41f80ca100 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f41f000d800 tx=0x7f41f000dcc0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 shutdown_connections 2023-11-28T20:35:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f41dc06ae40 0x7f41dc06d300 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41f815d120 0x7f41f80ca100 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41f80d2a40 0x7f41f8159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 --2- 172.21.15.45:0/757635215 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41f80d0690 0x7f41f8157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 >> 172.21.15.45:0/757635215 conn(0x7f41f8070f40 msgr2=0x7f41f80d10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 shutdown_connections 2023-11-28T20:35:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:10.951+0000 7f41fe2d1640 1 -- 172.21.15.45:0/757635215 wait complete. 2023-11-28T20:35:10.973 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-28T20:35:10.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:10.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-11-28T20:35:11.085 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T20:35:11.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:11.085 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-28T20:35:11.151 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:35:11.151 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:11.216 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:35:11.216 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:11.279 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:35:11.279 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:11.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:11.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.570+0000 7f6098cff640 1 Processor -- start 2023-11-28T20:35:11.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.570+0000 7f6098cff640 1 -- start start 2023-11-28T20:35:11.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.571+0000 7f6098cff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfed0 0x7f60940cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:11.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.571+0000 7f6098cff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f60940ce010 0x7f609415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:11.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.571+0000 7f6098cff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60940d1250 con 0x7f60940cfaf0 2023-11-28T20:35:11.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.571+0000 7f6098cff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60940b6c70 con 0x7f60940ce010 2023-11-28T20:35:11.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.571+0000 7f6098cff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60940ce430 con 0x7f60940cfed0 2023-11-28T20:35:11.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.572+0000 7f6093fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f60940ce010 0x7f609415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:11.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.572+0000 7f6093fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f60940ce010 0x7f609415cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:46070/0 (socket says 172.21.15.45:46070) 2023-11-28T20:35:11.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.572+0000 7f6092ffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfed0 0x7f60940cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:11.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.572+0000 7f6093fff640 1 -- 172.21.15.45:0/3152165857 learned_addr learned my addr 172.21.15.45:0/3152165857 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:11.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.573+0000 7f60927fc640 1 -- 172.21.15.45:0/3152165857 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913711745 0 0) 0x7f60940d1250 con 0x7f60940cfaf0 2023-11-28T20:35:11.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.573+0000 7f60927fc640 1 -- 172.21.15.45:0/3152165857 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6080003520 con 0x7f60940cfaf0 2023-11-28T20:35:11.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.573+0000 7f6092ffd640 1 -- 172.21.15.45:0/3152165857 >> v2:172.21.15.190:3300/0 conn(0x7f60940ce010 msgr2=0x7f609415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:11.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.573+0000 7f6092ffd640 1 --2- 172.21.15.45:0/3152165857 >> v2:172.21.15.190:3300/0 conn(0x7f60940ce010 0x7f609415cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.573+0000 7f6092ffd640 1 -- 172.21.15.45:0/3152165857 >> v1:172.21.15.190:6789/0 conn(0x7f60940cfaf0 legacy=0x7f60940b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:11.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.573+0000 7f6092ffd640 1 -- 172.21.15.45:0/3152165857 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60940ce6b0 con 0x7f60940cfed0 2023-11-28T20:35:11.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.575+0000 7f6092ffd640 1 --2- 172.21.15.45:0/3152165857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfed0 0x7f60940cdad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f607c0097a0 tx=0x7f607c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15be6a99b997f679 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:11.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.575+0000 7f60927fc640 1 -- 172.21.15.45:0/3152165857 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c031830 con 0x7f60940cfed0 2023-11-28T20:35:11.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.575+0000 7f60927fc640 1 -- 172.21.15.45:0/3152165857 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607c031990 con 0x7f60940cfed0 2023-11-28T20:35:11.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.575+0000 7f60927fc640 1 -- 172.21.15.45:0/3152165857 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c031d20 con 0x7f60940cfed0 2023-11-28T20:35:11.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.576+0000 7f6098cff640 1 -- 172.21.15.45:0/3152165857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfed0 msgr2=0x7f60940cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:11.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.576+0000 7f6098cff640 1 --2- 172.21.15.45:0/3152165857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfed0 0x7f60940cdad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f607c0097a0 tx=0x7f607c02b8d0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.576+0000 7f6098cff640 1 -- 172.21.15.45:0/3152165857 shutdown_connections 2023-11-28T20:35:11.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.576+0000 7f6098cff640 1 --2- 172.21.15.45:0/3152165857 >> v2:172.21.15.190:3300/0 conn(0x7f60940ce010 0x7f609415cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.576+0000 7f6098cff640 1 --2- 172.21.15.45:0/3152165857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfed0 0x7f60940cdad0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.576+0000 7f6098cff640 1 -- 172.21.15.45:0/3152165857 >> 172.21.15.45:0/3152165857 conn(0x7f60940b4c40 msgr2=0x7f60940b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:11.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.576+0000 7f6098cff640 1 -- 172.21.15.45:0/3152165857 shutdown_connections 2023-11-28T20:35:11.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.577+0000 7f6098cff640 1 -- 172.21.15.45:0/3152165857 wait complete. 2023-11-28T20:35:11.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.579+0000 7f6098cff640 1 Processor -- start 2023-11-28T20:35:11.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.579+0000 7f6098cff640 1 -- start start 2023-11-28T20:35:11.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.580+0000 7f6098cff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60940ce010 0x7f6094160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.580+0000 7f6098cff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfaf0 0x7f6094157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.580+0000 7f6098cff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60940cfed0 0x7f609415ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.580+0000 7f6098cff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6094160670 con 0x7f60940ce010 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.580+0000 7f6098cff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60940c6c60 con 0x7f60940cfed0 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.580+0000 7f6098cff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60940c7500 con 0x7f60940cfaf0 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f60937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60940ce010 0x7f6094160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f60937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60940ce010 0x7f6094160130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47956/0 (socket says 172.21.15.45:47956) 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f60937fe640 1 -- 172.21.15.45:0/3032662512 learned_addr learned my addr 172.21.15.45:0/3032662512 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f6092ffd640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfaf0 0x7f6094157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:11.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f6093fff640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60940cfed0 0x7f609415ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:11.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f60937fe640 1 -- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfaf0 msgr2=0x7f6094157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:11.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f60937fe640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfaf0 0x7f6094157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f60937fe640 1 -- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60940cfed0 msgr2=0x7f609415ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:11.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f60937fe640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60940cfed0 0x7f609415ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.581+0000 7f60937fe640 1 -- 172.21.15.45:0/3032662512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c009400 con 0x7f60940ce010 2023-11-28T20:35:11.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.582+0000 7f60937fe640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60940ce010 0x7f6094160130 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f608400c830 tx=0x7f608400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:11.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.582+0000 7f608bfff640 1 -- 172.21.15.45:0/3032662512 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084003b70 con 0x7f60940ce010 2023-11-28T20:35:11.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.582+0000 7f608bfff640 1 -- 172.21.15.45:0/3032662512 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6084003cd0 con 0x7f60940ce010 2023-11-28T20:35:11.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.582+0000 7f608bfff640 1 -- 172.21.15.45:0/3032662512 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608400aa90 con 0x7f60940ce010 2023-11-28T20:35:11.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.582+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6094158470 con 0x7f60940ce010 2023-11-28T20:35:11.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.582+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60940c6eb0 con 0x7f60940ce010 2023-11-28T20:35:11.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.583+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609404e920 con 0x7f60940ce010 2023-11-28T20:35:11.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.584+0000 7f608bfff640 1 -- 172.21.15.45:0/3032662512 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f608400abf0 con 0x7f60940ce010 2023-11-28T20:35:11.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.585+0000 7f608bfff640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f607406ae40 0x7f607406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:11.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.585+0000 7f608bfff640 1 -- 172.21.15.45:0/3032662512 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6084089910 con 0x7f60940ce010 2023-11-28T20:35:11.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.586+0000 7f6092ffd640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f607406ae40 0x7f607406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:11.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.587+0000 7f6092ffd640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f607406ae40 0x7f607406d300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f607c036000 tx=0x7f607c03c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:11.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.590+0000 7f608bfff640 1 -- 172.21.15.45:0/3032662512 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60840546b0 con 0x7f60940ce010 2023-11-28T20:35:11.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.790+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60940b5270 con 0x7f60940ce010 2023-11-28T20:35:11.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.793+0000 7f608bfff640 1 -- 172.21.15.45:0/3032662512 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7f6084057cd0 con 0x7f60940ce010 2023-11-28T20:35:11.794 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:11.795 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:11.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f607406ae40 msgr2=0x7f607406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f607406ae40 0x7f607406d300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f607c036000 tx=0x7f607c03c040 comp rx=0 tx=0).stop 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60940ce010 msgr2=0x7f6094160130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60940ce010 0x7f6094160130 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f608400c830 tx=0x7f608400ccf0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 shutdown_connections 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f607406ae40 0x7f607406d300 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60940cfed0 0x7f609415ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60940cfaf0 0x7f6094157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 --2- 172.21.15.45:0/3032662512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60940ce010 0x7f6094160130 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 >> 172.21.15.45:0/3032662512 conn(0x7f60940b4c40 msgr2=0x7f60940b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 shutdown_connections 2023-11-28T20:35:11.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:11.796+0000 7f6098cff640 1 -- 172.21.15.45:0/3032662512 wait complete. 2023-11-28T20:35:11.816 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:35:11.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:35:12.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.047+0000 7fd415593640 1 Processor -- start 2023-11-28T20:35:12.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.048+0000 7fd415593640 1 -- start start 2023-11-28T20:35:12.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.049+0000 7fd415593640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4100d0040 0x7fd4100d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.049+0000 7fd415593640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd4100b2290 0x7fd4100d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.049+0000 7fd415593640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 0x7fd4100d3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.049+0000 7fd415593640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4100b90a0 con 0x7fd4100d0040 2023-11-28T20:35:12.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.049+0000 7fd415593640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4100d1920 con 0x7fd4100b2290 2023-11-28T20:35:12.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.049+0000 7fd415593640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100d0960 con 0x7fd4100b1d40 2023-11-28T20:35:12.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd40ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4100d0040 0x7fd4100d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd40ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4100d0040 0x7fd4100d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47968/0 (socket says 172.21.15.45:47968) 2023-11-28T20:35:12.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd40ffff640 1 -- 172.21.15.45:0/2207314199 learned_addr learned my addr 172.21.15.45:0/2207314199 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:12.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd407fff640 1 --2- 172.21.15.45:0/2207314199 >> v2:172.21.15.190:3300/0 conn(0x7fd4100b2290 0x7fd4100d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd40ffff640 1 -- 172.21.15.45:0/2207314199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 msgr2=0x7fd4100d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:12.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd414d92640 1 --2- 172.21.15.45:0/2207314199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 0x7fd4100d3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd40ffff640 1 --2- 172.21.15.45:0/2207314199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 0x7fd4100d3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd40ffff640 1 -- 172.21.15.45:0/2207314199 >> v2:172.21.15.190:3300/0 conn(0x7fd4100b2290 msgr2=0x7fd4100d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:12.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd40ffff640 1 --2- 172.21.15.45:0/2207314199 >> v2:172.21.15.190:3300/0 conn(0x7fd4100b2290 0x7fd4100d3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd40ffff640 1 -- 172.21.15.45:0/2207314199 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4100b8ae0 con 0x7fd4100d0040 2023-11-28T20:35:12.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd407fff640 1 --2- 172.21.15.45:0/2207314199 >> v2:172.21.15.190:3300/0 conn(0x7fd4100b2290 0x7fd4100d3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:12.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.050+0000 7fd414d92640 1 --2- 172.21.15.45:0/2207314199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 0x7fd4100d3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:12.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.051+0000 7fd40ffff640 1 --2- 172.21.15.45:0/2207314199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 0x7fd4100d0420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd4100b8800 tx=0x7fd40002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578616a26a5912d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:12.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.052+0000 7fd40f7fe640 1 -- 172.21.15.45:0/2207314199 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400030850 con 0x7fd4100d0040 2023-11-28T20:35:12.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.052+0000 7fd40f7fe640 1 -- 172.21.15.45:0/2207314199 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4000309b0 con 0x7fd4100d0040 2023-11-28T20:35:12.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.052+0000 7fd40f7fe640 1 -- 172.21.15.45:0/2207314199 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400030d40 con 0x7fd4100d0040 2023-11-28T20:35:12.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.052+0000 7fd415593640 1 -- 172.21.15.45:0/2207314199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 msgr2=0x7fd4100d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.052+0000 7fd415593640 1 --2- 172.21.15.45:0/2207314199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 0x7fd4100d0420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd4100b8800 tx=0x7fd40002b990 comp rx=0 tx=0).stop 2023-11-28T20:35:12.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.053+0000 7fd415593640 1 -- 172.21.15.45:0/2207314199 shutdown_connections 2023-11-28T20:35:12.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.053+0000 7fd415593640 1 --2- 172.21.15.45:0/2207314199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 0x7fd4100d3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.053+0000 7fd415593640 1 --2- 172.21.15.45:0/2207314199 >> v2:172.21.15.190:3300/0 conn(0x7fd4100b2290 0x7fd4100d3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.053+0000 7fd415593640 1 --2- 172.21.15.45:0/2207314199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 0x7fd4100d0420 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.053+0000 7fd415593640 1 -- 172.21.15.45:0/2207314199 >> 172.21.15.45:0/2207314199 conn(0x7fd410070f40 msgr2=0x7fd410071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:12.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.053+0000 7fd415593640 1 -- 172.21.15.45:0/2207314199 shutdown_connections 2023-11-28T20:35:12.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.053+0000 7fd415593640 1 -- 172.21.15.45:0/2207314199 wait complete. 2023-11-28T20:35:12.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.054+0000 7fd415593640 1 Processor -- start 2023-11-28T20:35:12.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.055+0000 7fd415593640 1 -- start start 2023-11-28T20:35:12.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.055+0000 7fd415593640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 0x7fd4100cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd415593640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4100b2290 0x7fd4101f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd415593640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 0x7fd4100c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd415593640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100cb380 con 0x7fd4100d0040 2023-11-28T20:35:12.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd415593640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100cbce0 con 0x7fd4100b2290 2023-11-28T20:35:12.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd415593640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101f2ee0 con 0x7fd4100b1d40 2023-11-28T20:35:12.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd414d92640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 0x7fd4100c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd414d92640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 0x7fd4100c7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47978/0 (socket says 172.21.15.45:47978) 2023-11-28T20:35:12.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd414d92640 1 -- 172.21.15.45:0/2072350655 learned_addr learned my addr 172.21.15.45:0/2072350655 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:12.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd414d92640 1 -- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 msgr2=0x7fd4100cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd414d92640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 0x7fd4100cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd414d92640 1 -- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4100b2290 msgr2=0x7fd4101f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd414d92640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4100b2290 0x7fd4101f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.056+0000 7fd414d92640 1 -- 172.21.15.45:0/2072350655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4000094d0 con 0x7fd4100d0040 2023-11-28T20:35:12.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.057+0000 7fd414d92640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 0x7fd4100c7a70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd40800cd80 tx=0x7fd40800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:12.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.057+0000 7fd40d7fa640 1 -- 172.21.15.45:0/2072350655 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40800f680 con 0x7fd4100d0040 2023-11-28T20:35:12.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.057+0000 7fd40d7fa640 1 -- 172.21.15.45:0/2072350655 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40800f800 con 0x7fd4100d0040 2023-11-28T20:35:12.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.057+0000 7fd40d7fa640 1 -- 172.21.15.45:0/2072350655 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40800fbb0 con 0x7fd4100d0040 2023-11-28T20:35:12.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.057+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4100ca700 con 0x7fd4100d0040 2023-11-28T20:35:12.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.057+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4100cacc0 con 0x7fd4100d0040 2023-11-28T20:35:12.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.058+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd410071b30 con 0x7fd4100d0040 2023-11-28T20:35:12.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.059+0000 7fd40d7fa640 1 -- 172.21.15.45:0/2072350655 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd40800b040 con 0x7fd4100d0040 2023-11-28T20:35:12.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.060+0000 7fd40d7fa640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd3f006ae40 0x7fd3f006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.060+0000 7fd40d7fa640 1 -- 172.21.15.45:0/2072350655 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd40808a400 con 0x7fd4100d0040 2023-11-28T20:35:12.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.061+0000 7fd40ffff640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd3f006ae40 0x7fd3f006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.062+0000 7fd40ffff640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd3f006ae40 0x7fd3f006d300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd400036000 tx=0x7fd40003b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:12.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.064+0000 7fd40d7fa640 1 -- 172.21.15.45:0/2072350655 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd408055050 con 0x7fd4100d0040 2023-11-28T20:35:12.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.262+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd410159b20 con 0x7fd4100d0040 2023-11-28T20:35:12.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.264+0000 7fd40d7fa640 1 -- 172.21.15.45:0/2072350655 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd408058670 con 0x7fd4100d0040 2023-11-28T20:35:12.265 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:35:12.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd3f006ae40 msgr2=0x7fd3f006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd3f006ae40 0x7fd3f006d300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd400036000 tx=0x7fd40003b040 comp rx=0 tx=0).stop 2023-11-28T20:35:12.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 msgr2=0x7fd4100c7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 0x7fd4100c7a70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd40800cd80 tx=0x7fd40800a650 comp rx=0 tx=0).stop 2023-11-28T20:35:12.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 shutdown_connections 2023-11-28T20:35:12.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd3f006ae40 0x7fd3f006d300 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4100d0040 0x7fd4100c7a70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4100b2290 0x7fd4101f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 --2- 172.21.15.45:0/2072350655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4100b1d40 0x7fd4100cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.266+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 >> 172.21.15.45:0/2072350655 conn(0x7fd410070f40 msgr2=0x7fd4100d3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:12.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.267+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 shutdown_connections 2023-11-28T20:35:12.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.267+0000 7fd415593640 1 -- 172.21.15.45:0/2072350655 wait complete. 2023-11-28T20:35:12.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:35:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.517+0000 7facdda48640 1 Processor -- start 2023-11-28T20:35:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.517+0000 7facdda48640 1 -- start start 2023-11-28T20:35:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.519+0000 7facdda48640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd806ac10 0x7facd806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.519+0000 7facdda48640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd815ecd0 con 0x7facd815e840 2023-11-28T20:35:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.519+0000 7facdda48640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd815f290 con 0x7facd815f720 2023-11-28T20:35:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.519+0000 7facdda48640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd815fbd0 con 0x7facd806ac10 2023-11-28T20:35:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.519+0000 7facd6ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7facd815e840 0x7facd806a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57418/0 (socket says 172.21.15.45:57418) 2023-11-28T20:35:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.519+0000 7facd6ffd640 1 -- 172.21.15.45:0/3484342851 learned_addr learned my addr 172.21.15.45:0/3484342851 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.519+0000 7facd67fc640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd806ac10 0x7facd806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.519+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543442261 0 0) 0x7facd815ecd0 con 0x7facd815e840 2023-11-28T20:35:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.519+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc4003520 con 0x7facd815e840 2023-11-28T20:35:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 625779018 0 0) 0x7facc4003520 con 0x7facd815e840 2023-11-28T20:35:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd806ac10 msgr2=0x7facd806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd806ac10 0x7facd806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 >> v1:172.21.15.190:6789/0 conn(0x7facd815f720 legacy=0x7facd80cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd8072020 con 0x7facd815e840 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7facc8001670 con 0x7facd815e840 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 >> v1:172.21.15.45:6789/0 conn(0x7facd815e840 legacy=0x7facd806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd806ac10 0x7facc40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc4003f70 0x7facc40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facc4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd815fbd0 con 0x7facc4004d00 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc400bac0 con 0x7facc4003f70 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc400bc80 con 0x7facd806ac10 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.520+0000 7facd77fe640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facc4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.521+0000 7facd67fc640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc4003f70 0x7facc40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.521+0000 7facd6ffd640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd806ac10 0x7facc40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.521+0000 7facd77fe640 1 -- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd806ac10 msgr2=0x7facc40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:12.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.521+0000 7facd77fe640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd806ac10 0x7facc40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.521+0000 7facd77fe640 1 -- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc4003f70 msgr2=0x7facc40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:12.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.521+0000 7facd77fe640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc4003f70 0x7facc40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.521+0000 7facd77fe640 1 -- 172.21.15.45:0/3484342851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc4003520 con 0x7facc4004d00 2023-11-28T20:35:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.522+0000 7facd77fe640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facc4005180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7facd0001750 tx=0x7facc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e86faf8c5730c7a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.522+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd00052d0 con 0x7facc4004d00 2023-11-28T20:35:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.522+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd002e050 con 0x7facc4004d00 2023-11-28T20:35:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.522+0000 7facd5ffb640 1 -- 172.21.15.45:0/3484342851 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0003830 con 0x7facc4004d00 2023-11-28T20:35:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.523+0000 7facdda48640 1 -- 172.21.15.45:0/3484342851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 msgr2=0x7facc4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.523+0000 7facdda48640 1 --2- 172.21.15.45:0/3484342851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facc4005180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7facd0001750 tx=0x7facc40057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.523+0000 7facd77fe640 1 -- 172.21.15.45:0/3484342851 reap_dead start 2023-11-28T20:35:12.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.523+0000 7facdda48640 1 -- 172.21.15.45:0/3484342851 shutdown_connections 2023-11-28T20:35:12.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.523+0000 7facdda48640 1 -- 172.21.15.45:0/3484342851 >> 172.21.15.45:0/3484342851 conn(0x7facd80ced60 msgr2=0x7facd80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:12.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.523+0000 7facdda48640 1 -- 172.21.15.45:0/3484342851 shutdown_connections 2023-11-28T20:35:12.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.523+0000 7facdda48640 1 -- 172.21.15.45:0/3484342851 wait complete. 2023-11-28T20:35:12.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.525+0000 7facdda48640 1 Processor -- start 2023-11-28T20:35:12.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.525+0000 7facdda48640 1 -- start start 2023-11-28T20:35:12.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.526+0000 7facdda48640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facd80c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.526+0000 7facdda48640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd8099a40 0x7facd8099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.526+0000 7facdda48640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facd809a3e0 0x7facd80b2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.526+0000 7facdda48640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd80b4090 con 0x7facc4004d00 2023-11-28T20:35:12.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.526+0000 7facdda48640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd80c8840 con 0x7facd809a3e0 2023-11-28T20:35:12.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.526+0000 7facdda48640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd80c7260 con 0x7facd8099a40 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.526+0000 7facd6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facd80c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.526+0000 7facd6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facd80c8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48000/0 (socket says 172.21.15.45:48000) 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.526+0000 7facd6ffd640 1 -- 172.21.15.45:0/2241649886 learned_addr learned my addr 172.21.15.45:0/2241649886 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.527+0000 7facd67fc640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd8099a40 0x7facd8099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.527+0000 7facd6ffd640 1 -- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd8099a40 msgr2=0x7facd8099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.527+0000 7facd6ffd640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd8099a40 0x7facd8099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.527+0000 7facd6ffd640 1 -- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facd809a3e0 msgr2=0x7facd80b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.527+0000 7facd77fe640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facd809a3e0 0x7facd80b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.527+0000 7facd6ffd640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facd809a3e0 0x7facd80b2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.527+0000 7facd6ffd640 1 -- 172.21.15.45:0/2241649886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd80c7540 con 0x7facc4004d00 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.527+0000 7facd6ffd640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facd80c8300 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7facc800e730 tx=0x7facc800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:12.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.527+0000 7faccffff640 1 -- 172.21.15.45:0/2241649886 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc800ee40 con 0x7facc4004d00 2023-11-28T20:35:12.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.528+0000 7faccffff640 1 -- 172.21.15.45:0/2241649886 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc800c8d0 con 0x7facc4004d00 2023-11-28T20:35:12.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.528+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd80c7820 con 0x7facc4004d00 2023-11-28T20:35:12.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.528+0000 7faccffff640 1 -- 172.21.15.45:0/2241649886 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc800cc80 con 0x7facc4004d00 2023-11-28T20:35:12.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.528+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd80c6e80 con 0x7facc4004d00 2023-11-28T20:35:12.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.529+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd80c8020 con 0x7facc4004d00 2023-11-28T20:35:12.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.530+0000 7faccffff640 1 -- 172.21.15.45:0/2241649886 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7facc801d460 con 0x7facc4004d00 2023-11-28T20:35:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.531+0000 7faccffff640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7facb806ae40 0x7facb806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.531+0000 7faccffff640 1 -- 172.21.15.45:0/2241649886 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7facc8089b10 con 0x7facc4004d00 2023-11-28T20:35:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.531+0000 7facd67fc640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7facb806ae40 0x7facb806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.532+0000 7facd67fc640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7facb806ae40 0x7facb806d300 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7facd80cd850 tx=0x7facc000b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:12.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.535+0000 7faccffff640 1 -- 172.21.15.45:0/2241649886 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facd80c8020 con 0x7facc4004d00 2023-11-28T20:35:12.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.730+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7facd80c8020 con 0x7facc4004d00 2023-11-28T20:35:12.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.731+0000 7faccffff640 1 -- 172.21.15.45:0/2241649886 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7facd80c8020 con 0x7facc4004d00 2023-11-28T20:35:12.732 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:12.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7facb806ae40 msgr2=0x7facb806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7facb806ae40 0x7facb806d300 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7facd80cd850 tx=0x7facc000b040 comp rx=0 tx=0).stop 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 msgr2=0x7facd80c8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facd80c8300 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7facc800e730 tx=0x7facc800ebf0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 shutdown_connections 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7facb806ae40 0x7facb806d300 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facd809a3e0 0x7facd80b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facd8099a40 0x7facd8099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 --2- 172.21.15.45:0/2241649886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc4004d00 0x7facd80c8300 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 >> 172.21.15.45:0/2241649886 conn(0x7facd80ced60 msgr2=0x7facd80ad5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 shutdown_connections 2023-11-28T20:35:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.735+0000 7facdda48640 1 -- 172.21.15.45:0/2241649886 wait complete. 2023-11-28T20:35:12.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:35:12.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.987+0000 7f6f69891640 1 Processor -- start 2023-11-28T20:35:12.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.987+0000 7f6f69891640 1 -- start start 2023-11-28T20:35:12.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.989+0000 7f6f69891640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f6415e800 0x7f6f640b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.989+0000 7f6f69891640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6f640cf6f0 0x7f6f640cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.989+0000 7f6f69891640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f640c77b0 con 0x7f6f640c6450 2023-11-28T20:35:12.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.989+0000 7f6f69891640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f640c6ba0 con 0x7f6f640cf6f0 2023-11-28T20:35:12.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.989+0000 7f6f69891640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640c6e80 con 0x7f6f6415e800 2023-11-28T20:35:12.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.989+0000 7f6f63fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6f640c6450 0x7f6f640cefe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57432/0 (socket says 172.21.15.45:57432) 2023-11-28T20:35:12.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.989+0000 7f6f63fff640 1 -- 172.21.15.45:0/3277885453 learned_addr learned my addr 172.21.15.45:0/3277885453 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:12.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.989+0000 7f6f6888f640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f6415e800 0x7f6f640b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.989+0000 7f6f69090640 1 --2- 172.21.15.45:0/3277885453 >> v2:172.21.15.190:3300/0 conn(0x7f6f640cf6f0 0x7f6f640cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332630558 0 0) 0x7f6f640c77b0 con 0x7f6f640c6450 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f4c003520 con 0x7f6f640c6450 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342355380 0 0) 0x7f6f4c003520 con 0x7f6f640c6450 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f6415e800 msgr2=0x7f6f640b23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f6415e800 0x7f6f640b23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 >> v2:172.21.15.190:3300/0 conn(0x7f6f640cf6f0 msgr2=0x7f6f640cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 --2- 172.21.15.45:0/3277885453 >> v2:172.21.15.190:3300/0 conn(0x7f6f640cf6f0 0x7f6f640cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f640c7160 con 0x7f6f640c6450 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6f58001670 con 0x7f6f640c6450 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 >> v1:172.21.15.45:6789/0 conn(0x7f6f640c6450 legacy=0x7f6f640cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f4c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f6415e800 0x7f6f4c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f4c004d00 0x7f6f4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640c6ba0 con 0x7f6f4c003970 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640c6e80 con 0x7f6f4c004d00 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.990+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c00b980 con 0x7f6f6415e800 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.991+0000 7f6f6888f640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f4c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.991+0000 7f6f63fff640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f6415e800 0x7f6f4c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.991+0000 7f6f69090640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f4c004d00 0x7f6f4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.991+0000 7f6f6888f640 1 -- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f6415e800 msgr2=0x7f6f4c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.991+0000 7f6f6888f640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f6415e800 0x7f6f4c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.991+0000 7f6f6888f640 1 -- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f4c004d00 msgr2=0x7f6f4c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:12.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.991+0000 7f6f6888f640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f4c004d00 0x7f6f4c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.991+0000 7f6f6888f640 1 -- 172.21.15.45:0/3277885453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c003520 con 0x7f6f4c003970 2023-11-28T20:35:12.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.991+0000 7f6f69090640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f4c004d00 0x7f6f4c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:12.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.992+0000 7f6f6888f640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f4c003d90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009fc0 tx=0x7f6f4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caebfa396945e6c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:12.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.993+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5402cc20 con 0x7f6f4c003970 2023-11-28T20:35:12.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.993+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5400f040 con 0x7f6f4c003970 2023-11-28T20:35:12.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.993+0000 7f6f637fe640 1 -- 172.21.15.45:0/3277885453 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54034440 con 0x7f6f4c003970 2023-11-28T20:35:12.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.993+0000 7f6f69891640 1 -- 172.21.15.45:0/3277885453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 msgr2=0x7f6f4c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:12.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.993+0000 7f6f69891640 1 --2- 172.21.15.45:0/3277885453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f4c003d90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009fc0 tx=0x7f6f4c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:12.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.993+0000 7f6f69090640 1 -- 172.21.15.45:0/3277885453 reap_dead start 2023-11-28T20:35:12.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.993+0000 7f6f69891640 1 -- 172.21.15.45:0/3277885453 shutdown_connections 2023-11-28T20:35:12.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.993+0000 7f6f69891640 1 -- 172.21.15.45:0/3277885453 >> 172.21.15.45:0/3277885453 conn(0x7f6f640d0ee0 msgr2=0x7f6f640d12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:12.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.994+0000 7f6f69891640 1 -- 172.21.15.45:0/3277885453 shutdown_connections 2023-11-28T20:35:12.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.994+0000 7f6f69891640 1 -- 172.21.15.45:0/3277885453 wait complete. 2023-11-28T20:35:12.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.996+0000 7f6f69891640 1 Processor -- start 2023-11-28T20:35:12.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.996+0000 7f6f69891640 1 -- start start 2023-11-28T20:35:12.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.997+0000 7f6f69891640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f6415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.997+0000 7f6f69891640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f641e4cc0 0x7f6f641e5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.997+0000 7f6f69891640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f640c9d50 0x7f6f640ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:12.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.997+0000 7f6f69891640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f641e5680 con 0x7f6f4c003970 2023-11-28T20:35:12.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.997+0000 7f6f69891640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6415b5e0 con 0x7f6f640c9d50 2023-11-28T20:35:13.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.997+0000 7f6f69891640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6415b8c0 con 0x7f6f641e4cc0 2023-11-28T20:35:13.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.997+0000 7f6f6888f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f6415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.997+0000 7f6f6888f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f6415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48032/0 (socket says 172.21.15.45:48032) 2023-11-28T20:35:13.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.997+0000 7f6f6888f640 1 -- 172.21.15.45:0/607334960 learned_addr learned my addr 172.21.15.45:0/607334960 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:13.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f69090640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f640c9d50 0x7f6f640ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f6888f640 1 -- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f641e4cc0 msgr2=0x7f6f641e5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f6888f640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f641e4cc0 0x7f6f641e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f6888f640 1 -- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f640c9d50 msgr2=0x7f6f640ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f6888f640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f640c9d50 0x7f6f640ca200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f6888f640 1 -- 172.21.15.45:0/607334960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6415bba0 con 0x7f6f4c003970 2023-11-28T20:35:13.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f6888f640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f6415a2e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6f5400e230 tx=0x7f6f54009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f617fa640 1 -- 172.21.15.45:0/607334960 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54034bd0 con 0x7f6f4c003970 2023-11-28T20:35:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f617fa640 1 -- 172.21.15.45:0/607334960 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f54034d30 con 0x7f6f4c003970 2023-11-28T20:35:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f617fa640 1 -- 172.21.15.45:0/607334960 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54034ea0 con 0x7f6f4c003970 2023-11-28T20:35:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.998+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6415be80 con 0x7f6f4c003970 2023-11-28T20:35:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:12.999+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f640ca740 con 0x7f6f4c003970 2023-11-28T20:35:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.000+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6409f030 con 0x7f6f4c003970 2023-11-28T20:35:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.001+0000 7f6f617fa640 1 -- 172.21.15.45:0/607334960 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6f54040450 con 0x7f6f4c003970 2023-11-28T20:35:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.002+0000 7f6f617fa640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6f5006ae70 0x7f6f5006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.002+0000 7f6f617fa640 1 -- 172.21.15.45:0/607334960 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6f540ac110 con 0x7f6f4c003970 2023-11-28T20:35:13.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.002+0000 7f6f63fff640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6f5006ae70 0x7f6f5006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.003+0000 7f6f63fff640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6f5006ae70 0x7f6f5006d330 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6f58000f60 tx=0x7f6f58000ed0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:13.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.006+0000 7f6f617fa640 1 -- 172.21.15.45:0/607334960 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f54076e00 con 0x7f6f4c003970 2023-11-28T20:35:13.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.209+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6f6404e920 con 0x7f6f4c003970 2023-11-28T20:35:13.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.210+0000 7f6f617fa640 1 -- 172.21.15.45:0/607334960 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6f5407a420 con 0x7f6f4c003970 2023-11-28T20:35:13.212 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:35:13.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6f5006ae70 msgr2=0x7f6f5006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6f5006ae70 0x7f6f5006d330 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6f58000f60 tx=0x7f6f58000ed0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 msgr2=0x7f6f6415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f6415a2e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6f5400e230 tx=0x7f6f54009750 comp rx=0 tx=0).stop 2023-11-28T20:35:13.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 shutdown_connections 2023-11-28T20:35:13.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6f5006ae70 0x7f6f5006d330 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f640c9d50 0x7f6f640ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f641e4cc0 0x7f6f641e5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 --2- 172.21.15.45:0/607334960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f4c003970 0x7f6f6415a2e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 >> 172.21.15.45:0/607334960 conn(0x7f6f640d0ee0 msgr2=0x7f6f640cea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:13.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 shutdown_connections 2023-11-28T20:35:13.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.213+0000 7f6f69891640 1 -- 172.21.15.45:0/607334960 wait complete. 2023-11-28T20:35:13.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:35:13.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.461+0000 7f5839954640 1 Processor -- start 2023-11-28T20:35:13.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.462+0000 7f5839954640 1 -- start start 2023-11-28T20:35:13.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.463+0000 7f5839954640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58341593a0 0x7f583406ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.463+0000 7f5839954640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f58340ceb70 0x7f583415db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.463+0000 7f5839954640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5834159780 0x7f583406a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.463+0000 7f5839954640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583406d000 con 0x7f5834159780 2023-11-28T20:35:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.463+0000 7f5839954640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583415e080 con 0x7f58340ceb70 2023-11-28T20:35:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.463+0000 7f5839954640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340cf2f0 con 0x7f58341593a0 2023-11-28T20:35:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.464+0000 7f5839153640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5834159780 0x7f583406a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.464+0000 7f5839153640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5834159780 0x7f583406a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48036/0 (socket says 172.21.15.45:48036) 2023-11-28T20:35:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.464+0000 7f5839153640 1 -- 172.21.15.45:0/498841057 learned_addr learned my addr 172.21.15.45:0/498841057 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.464+0000 7f5833fff640 1 --2- 172.21.15.45:0/498841057 >> v2:172.21.15.190:3300/0 conn(0x7f58340ceb70 0x7f583415db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.464+0000 7f5838952640 1 --2- 172.21.15.45:0/498841057 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58341593a0 0x7f583406ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.464+0000 7f5839153640 1 -- 172.21.15.45:0/498841057 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58341593a0 msgr2=0x7f583406ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.465+0000 7f5839153640 1 --2- 172.21.15.45:0/498841057 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58341593a0 0x7f583406ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.465+0000 7f5839153640 1 -- 172.21.15.45:0/498841057 >> v2:172.21.15.190:3300/0 conn(0x7f58340ceb70 msgr2=0x7f583415db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:13.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.465+0000 7f5839153640 1 --2- 172.21.15.45:0/498841057 >> v2:172.21.15.190:3300/0 conn(0x7f58340ceb70 0x7f583415db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.465+0000 7f5839153640 1 -- 172.21.15.45:0/498841057 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340cf010 con 0x7f5834159780 2023-11-28T20:35:13.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.466+0000 7f5839153640 1 --2- 172.21.15.45:0/498841057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5834159780 0x7f583406a500 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f58340702e0 tx=0x7f582c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f9c8d64dfc33cf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:13.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.466+0000 7f58337fe640 1 -- 172.21.15.45:0/498841057 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c0317e0 con 0x7f5834159780 2023-11-28T20:35:13.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.466+0000 7f58337fe640 1 -- 172.21.15.45:0/498841057 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c031940 con 0x7f5834159780 2023-11-28T20:35:13.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.466+0000 7f58337fe640 1 -- 172.21.15.45:0/498841057 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c031cd0 con 0x7f5834159780 2023-11-28T20:35:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.467+0000 7f5839954640 1 -- 172.21.15.45:0/498841057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5834159780 msgr2=0x7f583406a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.467+0000 7f5839954640 1 --2- 172.21.15.45:0/498841057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5834159780 0x7f583406a500 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f58340702e0 tx=0x7f582c02b880 comp rx=0 tx=0).stop 2023-11-28T20:35:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.467+0000 7f5839954640 1 -- 172.21.15.45:0/498841057 shutdown_connections 2023-11-28T20:35:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.467+0000 7f5839954640 1 --2- 172.21.15.45:0/498841057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5834159780 0x7f583406a500 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.467+0000 7f5839954640 1 --2- 172.21.15.45:0/498841057 >> v2:172.21.15.190:3300/0 conn(0x7f58340ceb70 0x7f583415db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.467+0000 7f5839954640 1 --2- 172.21.15.45:0/498841057 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58341593a0 0x7f583406ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.467+0000 7f5839954640 1 -- 172.21.15.45:0/498841057 >> 172.21.15.45:0/498841057 conn(0x7f58340b1170 msgr2=0x7f58340b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.467+0000 7f5839954640 1 -- 172.21.15.45:0/498841057 shutdown_connections 2023-11-28T20:35:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.467+0000 7f5839954640 1 -- 172.21.15.45:0/498841057 wait complete. 2023-11-28T20:35:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.469+0000 7f5839954640 1 Processor -- start 2023-11-28T20:35:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.469+0000 7f5839954640 1 -- start start 2023-11-28T20:35:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.470+0000 7f5839954640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58340ceb70 0x7f583415a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.470+0000 7f5839954640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58341593a0 0x7f58341529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.470+0000 7f5839954640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5834159780 0x7f5834152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.470+0000 7f5839954640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583414a1a0 con 0x7f58341593a0 2023-11-28T20:35:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.470+0000 7f5839954640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583414e050 con 0x7f58340ceb70 2023-11-28T20:35:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.470+0000 7f5839954640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583414e2f0 con 0x7f5834159780 2023-11-28T20:35:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5833fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58341593a0 0x7f58341529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5833fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58341593a0 0x7f58341529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48050/0 (socket says 172.21.15.45:48050) 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5833fff640 1 -- 172.21.15.45:0/2238843438 learned_addr learned my addr 172.21.15.45:0/2238843438 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5838952640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58340ceb70 0x7f583415a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5839153640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5834159780 0x7f5834152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5833fff640 1 -- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5834159780 msgr2=0x7f5834152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5833fff640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5834159780 0x7f5834152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5833fff640 1 -- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58340ceb70 msgr2=0x7f583415a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5833fff640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58340ceb70 0x7f583415a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5833fff640 1 -- 172.21.15.45:0/2238843438 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c0094d0 con 0x7f58341593a0 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.471+0000 7f5839153640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5834159780 0x7f5834152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.472+0000 7f5833fff640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58341593a0 0x7f58341529e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f582800e8c0 tx=0x7f582800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.472+0000 7f58317fa640 1 -- 172.21.15.45:0/2238843438 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582800c940 con 0x7f58341593a0 2023-11-28T20:35:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.472+0000 7f58317fa640 1 -- 172.21.15.45:0/2238843438 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582800caa0 con 0x7f58341593a0 2023-11-28T20:35:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.472+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583415b360 con 0x7f58341593a0 2023-11-28T20:35:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.472+0000 7f58317fa640 1 -- 172.21.15.45:0/2238843438 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582800ce30 con 0x7f58341593a0 2023-11-28T20:35:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.472+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583415b920 con 0x7f58341593a0 2023-11-28T20:35:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.473+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58340722d0 con 0x7f58341593a0 2023-11-28T20:35:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.474+0000 7f58317fa640 1 -- 172.21.15.45:0/2238843438 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5828013470 con 0x7f58341593a0 2023-11-28T20:35:13.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.475+0000 7f58317fa640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f582006ae40 0x7f582006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.475+0000 7f58317fa640 1 -- 172.21.15.45:0/2238843438 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5828089c20 con 0x7f58341593a0 2023-11-28T20:35:13.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.476+0000 7f5838952640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f582006ae40 0x7f582006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.477+0000 7f5838952640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f582006ae40 0x7f582006d300 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f58240074a0 tx=0x7f582400b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:13.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.479+0000 7f58317fa640 1 -- 172.21.15.45:0/2238843438 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5828054870 con 0x7f58341593a0 2023-11-28T20:35:13.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.672+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f583406aa10 con 0x7f58341593a0 2023-11-28T20:35:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.673+0000 7f58317fa640 1 -- 172.21.15.45:0/2238843438 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5828021020 con 0x7f58341593a0 2023-11-28T20:35:13.675 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.676+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f582006ae40 msgr2=0x7f582006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.676+0000 7f5839954640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f582006ae40 0x7f582006d300 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f58240074a0 tx=0x7f582400b040 comp rx=0 tx=0).stop 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.676+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58341593a0 msgr2=0x7f58341529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.676+0000 7f5839954640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58341593a0 0x7f58341529e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f582800e8c0 tx=0x7f582800ed80 comp rx=0 tx=0).stop 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.676+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 shutdown_connections 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.676+0000 7f5839954640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f582006ae40 0x7f582006d300 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.676+0000 7f5839954640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5834159780 0x7f5834152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.676+0000 7f5839954640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58341593a0 0x7f58341529e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.677+0000 7f5839954640 1 --2- 172.21.15.45:0/2238843438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58340ceb70 0x7f583415a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.677+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 >> 172.21.15.45:0/2238843438 conn(0x7f58340b1170 msgr2=0x7f58340b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.677+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 shutdown_connections 2023-11-28T20:35:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.677+0000 7f5839954640 1 -- 172.21.15.45:0/2238843438 wait complete. 2023-11-28T20:35:13.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:35:13.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.915+0000 7f36c5f6e640 1 Processor -- start 2023-11-28T20:35:13.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.915+0000 7f36c5f6e640 1 -- start start 2023-11-28T20:35:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.917+0000 7f36c5f6e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f36c015b070 0x7f36c00ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.917+0000 7f36c5f6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c00cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.917+0000 7f36c5f6e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c00abd70 con 0x7f36c00cf440 2023-11-28T20:35:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.917+0000 7f36c5f6e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c015fb30 con 0x7f36c015b070 2023-11-28T20:35:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.917+0000 7f36c5f6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c015b450 con 0x7f36c00b2fb0 2023-11-28T20:35:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.917+0000 7f36b7fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c00cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.917+0000 7f36c4f6c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f36c015b070 0x7f36c00ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.918+0000 7f36b7fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c00cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:39664/0 (socket says 172.21.15.45:39664) 2023-11-28T20:35:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.918+0000 7f36b7fff640 1 -- 172.21.15.45:0/782840647 learned_addr learned my addr 172.21.15.45:0/782840647 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.918+0000 7f36c4f6c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f36c015b070 0x7f36c00ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:53304/0 (socket says 172.21.15.45:53304) 2023-11-28T20:35:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.918+0000 7f36b7fff640 1 -- 172.21.15.45:0/782840647 >> v2:172.21.15.190:3300/0 conn(0x7f36c015b070 msgr2=0x7f36c00ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.918+0000 7f36b7fff640 1 --2- 172.21.15.45:0/782840647 >> v2:172.21.15.190:3300/0 conn(0x7f36c015b070 0x7f36c00ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.918+0000 7f36b7fff640 1 -- 172.21.15.45:0/782840647 >> v1:172.21.15.190:6789/0 conn(0x7f36c00cf440 legacy=0x7f36c00b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.918+0000 7f36b7fff640 1 -- 172.21.15.45:0/782840647 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c00b3ed0 con 0x7f36c00b2fb0 2023-11-28T20:35:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.920+0000 7f36b7fff640 1 --2- 172.21.15.45:0/782840647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c00cef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009870 tx=0x7f36b002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a920da12b335b28 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.920+0000 7f36b77fe640 1 -- 172.21.15.45:0/782840647 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b0031640 con 0x7f36c00b2fb0 2023-11-28T20:35:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.920+0000 7f36b77fe640 1 -- 172.21.15.45:0/782840647 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b00317a0 con 0x7f36c00b2fb0 2023-11-28T20:35:13.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.920+0000 7f36b77fe640 1 -- 172.21.15.45:0/782840647 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b0031b30 con 0x7f36c00b2fb0 2023-11-28T20:35:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.921+0000 7f36c5f6e640 1 -- 172.21.15.45:0/782840647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 msgr2=0x7f36c00cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.921+0000 7f36c5f6e640 1 --2- 172.21.15.45:0/782840647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c00cef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009870 tx=0x7f36b002b8f0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.921+0000 7f36c5f6e640 1 -- 172.21.15.45:0/782840647 shutdown_connections 2023-11-28T20:35:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.921+0000 7f36c5f6e640 1 --2- 172.21.15.45:0/782840647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c00cef00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.921+0000 7f36c5f6e640 1 --2- 172.21.15.45:0/782840647 >> v2:172.21.15.190:3300/0 conn(0x7f36c015b070 0x7f36c00ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.921+0000 7f36c5f6e640 1 -- 172.21.15.45:0/782840647 >> 172.21.15.45:0/782840647 conn(0x7f36c006a710 msgr2=0x7f36c006ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.921+0000 7f36c5f6e640 1 -- 172.21.15.45:0/782840647 shutdown_connections 2023-11-28T20:35:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.921+0000 7f36c5f6e640 1 -- 172.21.15.45:0/782840647 wait complete. 2023-11-28T20:35:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.923+0000 7f36c5f6e640 1 Processor -- start 2023-11-28T20:35:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.923+0000 7f36c5f6e640 1 -- start start 2023-11-28T20:35:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.924+0000 7f36c5f6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c015e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.924+0000 7f36c5f6e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c00cf440 0x7f36c01e5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.924+0000 7f36c5f6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c015b070 0x7f36c014f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.924+0000 7f36c5f6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c00703a0 con 0x7f36c015b070 2023-11-28T20:35:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.924+0000 7f36c5f6e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c01e6500 con 0x7f36c00cf440 2023-11-28T20:35:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.924+0000 7f36c5f6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c01e6fd0 con 0x7f36c00b2fb0 2023-11-28T20:35:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c576d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c015b070 0x7f36c014f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c576d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c015b070 0x7f36c014f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56064/0 (socket says 172.21.15.45:56064) 2023-11-28T20:35:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c576d640 1 -- 172.21.15.45:0/1177246139 learned_addr learned my addr 172.21.15.45:0/1177246139 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36b7fff640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c00cf440 0x7f36c01e5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c4f6c640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c015e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c576d640 1 -- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 msgr2=0x7f36c015e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c576d640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c015e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c576d640 1 -- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c00cf440 msgr2=0x7f36c01e5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c576d640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c00cf440 0x7f36c01e5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c576d640 1 -- 172.21.15.45:0/1177246139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b00094d0 con 0x7f36c015b070 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36c4f6c640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c015e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.925+0000 7f36b7fff640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c00cf440 0x7f36c01e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.926+0000 7f36c576d640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c015b070 0x7f36c014f6e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f36b800c400 tx=0x7f36b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.926+0000 7f36b57fa640 1 -- 172.21.15.45:0/1177246139 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b80052d0 con 0x7f36c015b070 2023-11-28T20:35:13.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.926+0000 7f36b57fa640 1 -- 172.21.15.45:0/1177246139 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b8005430 con 0x7f36c015b070 2023-11-28T20:35:13.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.926+0000 7f36b57fa640 1 -- 172.21.15.45:0/1177246139 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8003810 con 0x7f36c015b070 2023-11-28T20:35:13.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.926+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c01e6730 con 0x7f36c015b070 2023-11-28T20:35:13.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.926+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c01e6cf0 con 0x7f36c015b070 2023-11-28T20:35:13.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.927+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c00704e0 con 0x7f36c015b070 2023-11-28T20:35:13.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.928+0000 7f36b57fa640 1 -- 172.21.15.45:0/1177246139 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f36b8003970 con 0x7f36c015b070 2023-11-28T20:35:13.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.929+0000 7f36b57fa640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f36ac06ae40 0x7f36ac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:13.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.929+0000 7f36b57fa640 1 -- 172.21.15.45:0/1177246139 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f36b8088f60 con 0x7f36c015b070 2023-11-28T20:35:13.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.930+0000 7f36c4f6c640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f36ac06ae40 0x7f36ac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:13.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.931+0000 7f36c4f6c640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f36ac06ae40 0x7f36ac06d300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009fd0 tx=0x7f36bc00b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:13.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:13.933+0000 7f36b57fa640 1 -- 172.21.15.45:0/1177246139 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36c00704e0 con 0x7f36c015b070 2023-11-28T20:35:14.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.148+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f36c00704e0 con 0x7f36c015b070 2023-11-28T20:35:14.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.148+0000 7f36b57fa640 1 -- 172.21.15.45:0/1177246139 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f36c00704e0 con 0x7f36c015b070 2023-11-28T20:35:14.150 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2023-11-28T20:35:14.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f36ac06ae40 msgr2=0x7f36ac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:14.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f36ac06ae40 0x7f36ac06d300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009fd0 tx=0x7f36bc00b040 comp rx=0 tx=0).stop 2023-11-28T20:35:14.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c015b070 msgr2=0x7f36c014f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:14.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c015b070 0x7f36c014f6e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f36b800c400 tx=0x7f36b800c8c0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 shutdown_connections 2023-11-28T20:35:14.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f36ac06ae40 0x7f36ac06d300 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36c015b070 0x7f36c014f6e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36c00cf440 0x7f36c01e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 --2- 172.21.15.45:0/1177246139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36c00b2fb0 0x7f36c015e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.151+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 >> 172.21.15.45:0/1177246139 conn(0x7f36c006a710 msgr2=0x7f36c0155980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:14.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.152+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 shutdown_connections 2023-11-28T20:35:14.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.152+0000 7f36c5f6e640 1 -- 172.21.15.45:0/1177246139 wait complete. 2023-11-28T20:35:14.170 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-11-28T20:35:14.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:14.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-11-28T20:35:14.260 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:35:14.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:14.260 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-28T20:35:14.311 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:35:14.311 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:14.371 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:35:14.372 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:14.427 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:35:14.427 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:14.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:14.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.684+0000 7efea6139640 1 Processor -- start 2023-11-28T20:35:14.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.684+0000 7efea6139640 1 -- start start 2023-11-28T20:35:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.686+0000 7efea6139640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efea00b35b0 0x7efea015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:14.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.686+0000 7efea6139640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efea015b850 0x7efea00b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.686+0000 7efea6139640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea015bda0 con 0x7efea00b1fa0 2023-11-28T20:35:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.686+0000 7efea6139640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea015ee40 con 0x7efea00b35b0 2023-11-28T20:35:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.686+0000 7efea6139640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea015c080 con 0x7efea015b850 2023-11-28T20:35:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.686+0000 7efe9ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efea00b1fa0 0x7efea015f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58794/0 (socket says 172.21.15.45:58794) 2023-11-28T20:35:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.686+0000 7efe9ffff640 1 -- 172.21.15.45:0/2433273929 learned_addr learned my addr 172.21.15.45:0/2433273929 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:14.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe97fff640 1 --2- 172.21.15.45:0/2433273929 >> v2:172.21.15.190:3300/0 conn(0x7efea015b850 0x7efea00b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:14.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9f7fe640 1 --2- 172.21.15.45:0/2433273929 >> v2:172.21.15.45:3300/0 conn(0x7efea00b35b0 0x7efea015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:14.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375440795 0 0) 0x7efea015bda0 con 0x7efea00b1fa0 2023-11-28T20:35:14.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe8c003520 con 0x7efea00b1fa0 2023-11-28T20:35:14.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2321650820 0 0) 0x7efe8c003520 con 0x7efea00b1fa0 2023-11-28T20:35:14.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 >> v2:172.21.15.190:3300/0 conn(0x7efea015b850 msgr2=0x7efea00b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:14.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 --2- 172.21.15.45:0/2433273929 >> v2:172.21.15.190:3300/0 conn(0x7efea015b850 0x7efea00b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 >> v2:172.21.15.45:3300/0 conn(0x7efea00b35b0 msgr2=0x7efea015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 --2- 172.21.15.45:0/2433273929 >> v2:172.21.15.45:3300/0 conn(0x7efea00b35b0 0x7efea015ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea015c360 con 0x7efea00b1fa0 2023-11-28T20:35:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7efe980019f0 con 0x7efea00b1fa0 2023-11-28T20:35:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9f7fe640 1 --2- 172.21.15.45:0/2433273929 >> v2:172.21.15.45:3300/0 conn(0x7efea00b35b0 0x7efea015ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:35:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.687+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 >> v1:172.21.15.45:6789/0 conn(0x7efea00b1fa0 legacy=0x7efea015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9effd640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe8c0039c0 0x7efe8c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9effd640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0047c0 0x7efe8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9effd640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe8c005180 0x7efe8c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea015ee40 con 0x7efe8c0039c0 2023-11-28T20:35:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea015c080 con 0x7efe8c0047c0 2023-11-28T20:35:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8c00bf70 con 0x7efe8c005180 2023-11-28T20:35:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9f7fe640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe8c0039c0 0x7efe8c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe97fff640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0047c0 0x7efe8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9f7fe640 1 -- 172.21.15.45:0/2433273929 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe8c005180 msgr2=0x7efe8c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9f7fe640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe8c005180 0x7efe8c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9f7fe640 1 -- 172.21.15.45:0/2433273929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0047c0 msgr2=0x7efe8c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9f7fe640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0047c0 0x7efe8c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9f7fe640 1 -- 172.21.15.45:0/2433273929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c003520 con 0x7efe8c0039c0 2023-11-28T20:35:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe97fff640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0047c0 0x7efe8c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.688+0000 7efe9ffff640 1 -- 172.21.15.45:0/2433273929 reap_dead start 2023-11-28T20:35:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.690+0000 7efe9f7fe640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe8c0039c0 0x7efe8c003de0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efe90009c90 tx=0x7efe8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a2e0d90426c1d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.690+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9002c8f0 con 0x7efe8c0039c0 2023-11-28T20:35:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.690+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9002ca50 con 0x7efe8c0039c0 2023-11-28T20:35:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.690+0000 7efe9effd640 1 -- 172.21.15.45:0/2433273929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9002ce40 con 0x7efe8c0039c0 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.691+0000 7efea6139640 1 -- 172.21.15.45:0/2433273929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe8c0039c0 msgr2=0x7efe8c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.691+0000 7efea6139640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe8c0039c0 0x7efe8c003de0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efe90009c90 tx=0x7efe8c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.691+0000 7efea6139640 1 -- 172.21.15.45:0/2433273929 shutdown_connections 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.691+0000 7efea6139640 1 --2- 172.21.15.45:0/2433273929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe8c0039c0 0x7efe8c003de0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.691+0000 7efea6139640 1 -- 172.21.15.45:0/2433273929 >> 172.21.15.45:0/2433273929 conn(0x7efea006a500 msgr2=0x7efea006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.691+0000 7efea6139640 1 -- 172.21.15.45:0/2433273929 shutdown_connections 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.691+0000 7efea6139640 1 -- 172.21.15.45:0/2433273929 wait complete. 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.693+0000 7efea6139640 1 Processor -- start 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.693+0000 7efea6139640 1 -- start start 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.694+0000 7efea6139640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0039c0 0x7efea0143890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.694+0000 7efea6139640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea00abf10 0x7efea00a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:14.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.694+0000 7efea6139640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea00ac390 0x7efea00aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.694+0000 7efea6139640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea0143dd0 con 0x7efea00ac390 2023-11-28T20:35:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.694+0000 7efea6139640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea01e6140 con 0x7efe8c0039c0 2023-11-28T20:35:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.694+0000 7efea6139640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea01e6420 con 0x7efea00abf10 2023-11-28T20:35:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.694+0000 7efe9ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea00ac390 0x7efea00aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.694+0000 7efe9ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea00ac390 0x7efea00aa420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56094/0 (socket says 172.21.15.45:56094) 2023-11-28T20:35:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.694+0000 7efe9ffff640 1 -- 172.21.15.45:0/624948099 learned_addr learned my addr 172.21.15.45:0/624948099 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe9f7fe640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0039c0 0x7efea0143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe97fff640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea00abf10 0x7efea00a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe9ffff640 1 -- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea00abf10 msgr2=0x7efea00a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe9ffff640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea00abf10 0x7efea00a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe9ffff640 1 -- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0039c0 msgr2=0x7efea0143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe9ffff640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0039c0 0x7efea0143890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe9ffff640 1 -- 172.21.15.45:0/624948099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea01e6700 con 0x7efea00ac390 2023-11-28T20:35:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe9ffff640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea00ac390 0x7efea00aa420 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efea00b2920 tx=0x7efe9800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe9cff9640 1 -- 172.21.15.45:0/624948099 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98004d60 con 0x7efea00ac390 2023-11-28T20:35:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.695+0000 7efe9cff9640 1 -- 172.21.15.45:0/624948099 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9800a610 con 0x7efea00ac390 2023-11-28T20:35:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.696+0000 7efe9cff9640 1 -- 172.21.15.45:0/624948099 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9800a9c0 con 0x7efea00ac390 2023-11-28T20:35:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.696+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea01e69e0 con 0x7efea00ac390 2023-11-28T20:35:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.696+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea00a90e0 con 0x7efea00ac390 2023-11-28T20:35:14.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.697+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea00733d0 con 0x7efea00ac390 2023-11-28T20:35:14.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.698+0000 7efe9cff9640 1 -- 172.21.15.45:0/624948099 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efe9800ab20 con 0x7efea00ac390 2023-11-28T20:35:14.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.699+0000 7efe9cff9640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efe8006ae70 0x7efe8006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:14.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.699+0000 7efe9cff9640 1 -- 172.21.15.45:0/624948099 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7efe980891f0 con 0x7efea00ac390 2023-11-28T20:35:14.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.700+0000 7efe9f7fe640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efe8006ae70 0x7efe8006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:14.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.701+0000 7efe9f7fe640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efe8006ae70 0x7efe8006d330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efe90036000 tx=0x7efe90030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:14.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.704+0000 7efe9cff9640 1 -- 172.21.15.45:0/624948099 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe98053e40 con 0x7efea00ac390 2023-11-28T20:35:14.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:35:14.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:35:14.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.891+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efea00cd3b0 con 0x7efea00ac390 2023-11-28T20:35:14.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.893+0000 7efe9cff9640 1 -- 172.21.15.45:0/624948099 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7efe98057460 con 0x7efea00ac390 2023-11-28T20:35:14.895 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:14.896 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efe8006ae70 msgr2=0x7efe8006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efe8006ae70 0x7efe8006d330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efe90036000 tx=0x7efe90030000 comp rx=0 tx=0).stop 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea00ac390 msgr2=0x7efea00aa420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea00ac390 0x7efea00aa420 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efea00b2920 tx=0x7efe9800cb80 comp rx=0 tx=0).stop 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 shutdown_connections 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7efe8006ae70 0x7efe8006d330 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efea00ac390 0x7efea00aa420 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea00abf10 0x7efea00a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 --2- 172.21.15.45:0/624948099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe8c0039c0 0x7efea0143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.897+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 >> 172.21.15.45:0/624948099 conn(0x7efea006a500 msgr2=0x7efea00ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.898+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 shutdown_connections 2023-11-28T20:35:14.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:14.898+0000 7efea6139640 1 -- 172.21.15.45:0/624948099 wait complete. 2023-11-28T20:35:14.925 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:35:14.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:35:15.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.157+0000 7f67e1059640 1 Processor -- start 2023-11-28T20:35:15.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.157+0000 7f67e1059640 1 -- start start 2023-11-28T20:35:15.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.158+0000 7f67e1059640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0b2430 0x7f67dc0ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.158+0000 7f67e1059640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc09ceb0 con 0x7f67dc159fc0 2023-11-28T20:35:15.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.159+0000 7f67e1059640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc0b2cb0 con 0x7f67dc071ac0 2023-11-28T20:35:15.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.159+0000 7f67e1059640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc071450 con 0x7f67dc0b2430 2023-11-28T20:35:15.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.159+0000 7f67dbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f67dc159fc0 0x7f67dc0b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58804/0 (socket says 172.21.15.45:58804) 2023-11-28T20:35:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.159+0000 7f67dbfff640 1 -- 172.21.15.45:0/1537360458 learned_addr learned my addr 172.21.15.45:0/1537360458 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.159+0000 7f67d3fff640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0b2430 0x7f67dc0ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.159+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046261110 0 0) 0x7f67dc09ceb0 con 0x7f67dc159fc0 2023-11-28T20:35:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.159+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c8005730 con 0x7f67dc159fc0 2023-11-28T20:35:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.159+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216290453 0 0) 0x7f67c8005730 con 0x7f67dc159fc0 2023-11-28T20:35:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0b2430 msgr2=0x7f67dc0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0b2430 0x7f67dc0ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 >> v1:172.21.15.190:6789/0 conn(0x7f67dc071ac0 legacy=0x7f67dc0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc0b29d0 con 0x7f67dc159fc0 2023-11-28T20:35:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f67cc001670 con 0x7f67dc159fc0 2023-11-28T20:35:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 >> v1:172.21.15.45:6789/0 conn(0x7f67dc159fc0 legacy=0x7f67dc0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67c8005bb0 0x7f67c8005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67c80069d0 0x7f67c8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0b2430 0x7f67c800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc071450 con 0x7f67c8005bb0 2023-11-28T20:35:15.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80076c0 con 0x7f67c80069d0 2023-11-28T20:35:15.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c8007880 con 0x7f67dc0b2430 2023-11-28T20:35:15.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67dbfff640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67c8005bb0 0x7f67c8005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67dbfff640 1 -- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0b2430 msgr2=0x7f67c800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:15.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67dbfff640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0b2430 0x7f67c800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67d3fff640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67c80069d0 0x7f67c8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67dbfff640 1 -- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67c80069d0 msgr2=0x7f67c8006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67dbfff640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67c80069d0 0x7f67c8006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.160+0000 7f67dbfff640 1 -- 172.21.15.45:0/1537360458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c8005730 con 0x7f67c8005bb0 2023-11-28T20:35:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.161+0000 7f67dbfff640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67c8005bb0 0x7f67c8005fd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f67cc00c0e0 tx=0x7f67c8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9007858b7b855c1b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.162+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc005ad0 con 0x7f67c8005bb0 2023-11-28T20:35:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.162+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67cc005c50 con 0x7f67c8005bb0 2023-11-28T20:35:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.162+0000 7f67db7fe640 1 -- 172.21.15.45:0/1537360458 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc02c810 con 0x7f67c8005bb0 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.162+0000 7f67e1059640 1 -- 172.21.15.45:0/1537360458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67c8005bb0 msgr2=0x7f67c8005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.162+0000 7f67e1059640 1 --2- 172.21.15.45:0/1537360458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67c8005bb0 0x7f67c8005fd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f67cc00c0e0 tx=0x7f67c8007a00 comp rx=0 tx=0).stop 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.162+0000 7f67e0858640 1 -- 172.21.15.45:0/1537360458 reap_dead start 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.162+0000 7f67e1059640 1 -- 172.21.15.45:0/1537360458 shutdown_connections 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.162+0000 7f67e1059640 1 -- 172.21.15.45:0/1537360458 >> 172.21.15.45:0/1537360458 conn(0x7f67dc0b1050 msgr2=0x7f67dc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.163+0000 7f67e1059640 1 -- 172.21.15.45:0/1537360458 shutdown_connections 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.163+0000 7f67e1059640 1 -- 172.21.15.45:0/1537360458 wait complete. 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.164+0000 7f67e1059640 1 Processor -- start 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.165+0000 7f67e1059640 1 -- start start 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.165+0000 7f67e1059640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67c8005bb0 0x7f67dc1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.165+0000 7f67e1059640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0cad70 0x7f67dc0c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e1059640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67dc0af010 0x7f67dc0af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e1059640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc0c97f0 con 0x7f67dc0af010 2023-11-28T20:35:15.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e1059640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc09ab70 con 0x7f67c8005bb0 2023-11-28T20:35:15.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e1059640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc0af9d0 con 0x7f67dc0cad70 2023-11-28T20:35:15.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e0858640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67dc0af010 0x7f67dc0af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e0858640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67dc0af010 0x7f67dc0af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56118/0 (socket says 172.21.15.45:56118) 2023-11-28T20:35:15.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e0858640 1 -- 172.21.15.45:0/1901243092 learned_addr learned my addr 172.21.15.45:0/1901243092 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:15.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67dbfff640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67c8005bb0 0x7f67dc1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e0858640 1 -- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0cad70 msgr2=0x7f67dc0c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e0858640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0cad70 0x7f67dc0c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e0858640 1 -- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67c8005bb0 msgr2=0x7f67dc1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e0858640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67c8005bb0 0x7f67dc1535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.166+0000 7f67e0858640 1 -- 172.21.15.45:0/1901243092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc09b0f0 con 0x7f67dc0af010 2023-11-28T20:35:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.167+0000 7f67e0858640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67dc0af010 0x7f67dc0af490 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0cb660 tx=0x7f67d400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.167+0000 7f67d97fa640 1 -- 172.21.15.45:0/1901243092 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d4005d40 con 0x7f67dc0af010 2023-11-28T20:35:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.167+0000 7f67d97fa640 1 -- 172.21.15.45:0/1901243092 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d400ced0 con 0x7f67dc0af010 2023-11-28T20:35:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.167+0000 7f67d97fa640 1 -- 172.21.15.45:0/1901243092 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d400a850 con 0x7f67dc0af010 2023-11-28T20:35:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.167+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc0c9230 con 0x7f67dc0af010 2023-11-28T20:35:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.167+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67dc09ad80 con 0x7f67dc0af010 2023-11-28T20:35:15.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.168+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67dc0b3190 con 0x7f67dc0af010 2023-11-28T20:35:15.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.169+0000 7f67d97fa640 1 -- 172.21.15.45:0/1901243092 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f67d400a9b0 con 0x7f67dc0af010 2023-11-28T20:35:15.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.170+0000 7f67d97fa640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67bc06ae40 0x7f67bc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.170+0000 7f67d97fa640 1 -- 172.21.15.45:0/1901243092 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f67d4088960 con 0x7f67dc0af010 2023-11-28T20:35:15.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.171+0000 7f67dbfff640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67bc06ae40 0x7f67bc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.172+0000 7f67dbfff640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67bc06ae40 0x7f67bc06d300 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f67cc034000 tx=0x7f67cc009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:15.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.174+0000 7f67d97fa640 1 -- 172.21.15.45:0/1901243092 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67d40535b0 con 0x7f67dc0af010 2023-11-28T20:35:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.376+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f67dc04e920 con 0x7f67dc0af010 2023-11-28T20:35:15.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.377+0000 7f67d97fa640 1 -- 172.21.15.45:0/1901243092 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f67d4056bd0 con 0x7f67dc0af010 2023-11-28T20:35:15.378 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:35:15.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67bc06ae40 msgr2=0x7f67bc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67bc06ae40 0x7f67bc06d300 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f67cc034000 tx=0x7f67cc009030 comp rx=0 tx=0).stop 2023-11-28T20:35:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67dc0af010 msgr2=0x7f67dc0af490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67dc0af010 0x7f67dc0af490 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0cb660 tx=0x7f67d400cb90 comp rx=0 tx=0).stop 2023-11-28T20:35:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 shutdown_connections 2023-11-28T20:35:15.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f67bc06ae40 0x7f67bc06d300 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67dc0af010 0x7f67dc0af490 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67dc0cad70 0x7f67dc0c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 --2- 172.21.15.45:0/1901243092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67c8005bb0 0x7f67dc1535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.379+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 >> 172.21.15.45:0/1901243092 conn(0x7f67dc0b1050 msgr2=0x7f67dc0cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:15.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.380+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 shutdown_connections 2023-11-28T20:35:15.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.380+0000 7f67e1059640 1 -- 172.21.15.45:0/1901243092 wait complete. 2023-11-28T20:35:15.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:35:15.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.607+0000 7f571081e640 1 Processor -- start 2023-11-28T20:35:15.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.607+0000 7f571081e640 1 -- start start 2023-11-28T20:35:15.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.608+0000 7f571081e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f570c160aa0 0x7f570c0b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.608+0000 7f571081e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570c0ce7f0 0x7f570c0d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f571081e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c160e80 con 0x7f570c0d24f0 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f571081e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570c0d1220 con 0x7f570c160aa0 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f571081e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570c0d2e70 con 0x7f570c0ce7f0 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f570a59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f570c160aa0 0x7f570c0b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f570a59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f570c160aa0 0x7f570c0b6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56134/0 (socket says 172.21.15.45:56134) 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f570a59c640 1 -- 172.21.15.45:0/2939668031 learned_addr learned my addr 172.21.15.45:0/2939668031 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f570a59c640 1 -- 172.21.15.45:0/2939668031 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570c0ce7f0 msgr2=0x7f570c0d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f570a59c640 1 --2- 172.21.15.45:0/2939668031 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570c0ce7f0 0x7f570c0d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f570a59c640 1 -- 172.21.15.45:0/2939668031 >> v1:172.21.15.190:6789/0 conn(0x7f570c0d24f0 legacy=0x7f570c0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:15.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.609+0000 7f570a59c640 1 -- 172.21.15.45:0/2939668031 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c0b6960 con 0x7f570c160aa0 2023-11-28T20:35:15.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.610+0000 7f570a59c640 1 --2- 172.21.15.45:0/2939668031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c160aa0 0x7f570c0b6420 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009870 tx=0x7f56fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afadecacc89045 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:15.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.611+0000 7f5709d9b640 1 -- 172.21.15.45:0/2939668031 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc030840 con 0x7f570c160aa0 2023-11-28T20:35:15.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.611+0000 7f5709d9b640 1 -- 172.21.15.45:0/2939668031 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc0309a0 con 0x7f570c160aa0 2023-11-28T20:35:15.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.611+0000 7f5709d9b640 1 -- 172.21.15.45:0/2939668031 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc030d30 con 0x7f570c160aa0 2023-11-28T20:35:15.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.611+0000 7f571081e640 1 -- 172.21.15.45:0/2939668031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c160aa0 msgr2=0x7f570c0b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.611+0000 7f571081e640 1 --2- 172.21.15.45:0/2939668031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c160aa0 0x7f570c0b6420 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009870 tx=0x7f56fc02b980 comp rx=0 tx=0).stop 2023-11-28T20:35:15.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.611+0000 7f571081e640 1 -- 172.21.15.45:0/2939668031 shutdown_connections 2023-11-28T20:35:15.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.611+0000 7f571081e640 1 --2- 172.21.15.45:0/2939668031 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570c0ce7f0 0x7f570c0d2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.611+0000 7f571081e640 1 --2- 172.21.15.45:0/2939668031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c160aa0 0x7f570c0b6420 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.611+0000 7f571081e640 1 -- 172.21.15.45:0/2939668031 >> 172.21.15.45:0/2939668031 conn(0x7f570c070b20 msgr2=0x7f570c070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:15.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.612+0000 7f571081e640 1 -- 172.21.15.45:0/2939668031 shutdown_connections 2023-11-28T20:35:15.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.612+0000 7f571081e640 1 -- 172.21.15.45:0/2939668031 wait complete. 2023-11-28T20:35:15.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.613+0000 7f571081e640 1 Processor -- start 2023-11-28T20:35:15.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.613+0000 7f571081e640 1 -- start start 2023-11-28T20:35:15.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.614+0000 7f571081e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f570c0ce7f0 0x7f570c0c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.614+0000 7f571081e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c0d24f0 0x7f570c1f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.614+0000 7f571081e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570c160aa0 0x7f570c0c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.614+0000 7f571081e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c15f850 con 0x7f570c0d24f0 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.614+0000 7f571081e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c15ec30 con 0x7f570c0ce7f0 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.614+0000 7f571081e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570c14fa30 con 0x7f570c160aa0 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f5703fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c0d24f0 0x7f570c1f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f5703fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c0d24f0 0x7f570c1f2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56142/0 (socket says 172.21.15.45:56142) 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f5703fff640 1 -- 172.21.15.45:0/2955182516 learned_addr learned my addr 172.21.15.45:0/2955182516 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f570ad9d640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570c160aa0 0x7f570c0c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f570a59c640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f570c0ce7f0 0x7f570c0c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f5703fff640 1 -- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570c160aa0 msgr2=0x7f570c0c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f5703fff640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570c160aa0 0x7f570c0c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f5703fff640 1 -- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f570c0ce7f0 msgr2=0x7f570c0c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f5703fff640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f570c0ce7f0 0x7f570c0c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.615+0000 7f5703fff640 1 -- 172.21.15.45:0/2955182516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc0094d0 con 0x7f570c0d24f0 2023-11-28T20:35:15.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.616+0000 7f5703fff640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c0d24f0 0x7f570c1f2630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f56f400c410 tx=0x7f56f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:15.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.616+0000 7f5702ffd640 1 -- 172.21.15.45:0/2955182516 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f40035f0 con 0x7f570c0d24f0 2023-11-28T20:35:15.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.616+0000 7f5702ffd640 1 -- 172.21.15.45:0/2955182516 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f4003750 con 0x7f570c0d24f0 2023-11-28T20:35:15.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.616+0000 7f5702ffd640 1 -- 172.21.15.45:0/2955182516 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4003b40 con 0x7f570c0d24f0 2023-11-28T20:35:15.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.616+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570c15eeb0 con 0x7f570c0d24f0 2023-11-28T20:35:15.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.616+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c15f470 con 0x7f570c0d24f0 2023-11-28T20:35:15.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.617+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c15f600 con 0x7f570c0d24f0 2023-11-28T20:35:15.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.618+0000 7f5702ffd640 1 -- 172.21.15.45:0/2955182516 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f56f4003ca0 con 0x7f570c0d24f0 2023-11-28T20:35:15.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.619+0000 7f5702ffd640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56ec06ae40 0x7f56ec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:15.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.619+0000 7f5702ffd640 1 -- 172.21.15.45:0/2955182516 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f56f4088c80 con 0x7f570c0d24f0 2023-11-28T20:35:15.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.620+0000 7f570a59c640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56ec06ae40 0x7f56ec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:15.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.621+0000 7f570a59c640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56ec06ae40 0x7f56ec06d300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f56fc036000 tx=0x7f56fc03b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:15.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.623+0000 7f5702ffd640 1 -- 172.21.15.45:0/2955182516 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56f40538d0 con 0x7f570c0d24f0 2023-11-28T20:35:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.809+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f570c0733d0 con 0x7f570c0d24f0 2023-11-28T20:35:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.810+0000 7f5702ffd640 1 -- 172.21.15.45:0/2955182516 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f56f40880a0 con 0x7f570c0d24f0 2023-11-28T20:35:15.811 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:15.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56ec06ae40 msgr2=0x7f56ec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56ec06ae40 0x7f56ec06d300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f56fc036000 tx=0x7f56fc03b040 comp rx=0 tx=0).stop 2023-11-28T20:35:15.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c0d24f0 msgr2=0x7f570c1f2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:15.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c0d24f0 0x7f570c1f2630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f56f400c410 tx=0x7f56f400c8d0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 shutdown_connections 2023-11-28T20:35:15.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f56ec06ae40 0x7f56ec06d300 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570c160aa0 0x7f570c0c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f570c0d24f0 0x7f570c1f2630 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 --2- 172.21.15.45:0/2955182516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f570c0ce7f0 0x7f570c0c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 >> 172.21.15.45:0/2955182516 conn(0x7f570c070b20 msgr2=0x7f570c0cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 shutdown_connections 2023-11-28T20:35:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:15.813+0000 7f571081e640 1 -- 172.21.15.45:0/2955182516 wait complete. 2023-11-28T20:35:15.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:35:16.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.053+0000 7f2a4041e640 1 Processor -- start 2023-11-28T20:35:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.053+0000 7f2a4041e640 1 -- start start 2023-11-28T20:35:16.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.055+0000 7f2a4041e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a38071620 0x7f2a380ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.055+0000 7f2a4041e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38070a60 0x7f2a38070f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.055+0000 7f2a4041e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a380b30b0 con 0x7f2a381607e0 2023-11-28T20:35:16.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.055+0000 7f2a4041e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a38071450 con 0x7f2a38071620 2023-11-28T20:35:16.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.055+0000 7f2a4041e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a380b3300 con 0x7f2a38070a60 2023-11-28T20:35:16.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.055+0000 7f2a3d9b9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a38071620 0x7f2a380ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.056+0000 7f2a3e9bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38070a60 0x7f2a38070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.056+0000 7f2a3e9bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38070a60 0x7f2a38070f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:39742/0 (socket says 172.21.15.45:39742) 2023-11-28T20:35:16.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.056+0000 7f2a3e9bb640 1 -- 172.21.15.45:0/1566250323 learned_addr learned my addr 172.21.15.45:0/1566250323 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:16.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.056+0000 7f2a3d9b9640 1 -- 172.21.15.45:0/1566250323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38070a60 msgr2=0x7f2a38070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:16.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.056+0000 7f2a3d9b9640 1 --2- 172.21.15.45:0/1566250323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38070a60 0x7f2a38070f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.056+0000 7f2a3d9b9640 1 -- 172.21.15.45:0/1566250323 >> v1:172.21.15.190:6789/0 conn(0x7f2a381607e0 legacy=0x7f2a3806cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.056+0000 7f2a3d9b9640 1 -- 172.21.15.45:0/1566250323 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a380b35e0 con 0x7f2a38071620 2023-11-28T20:35:16.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.058+0000 7f2a3d9b9640 1 --2- 172.21.15.45:0/1566250323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38071620 0x7f2a380ad2c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2a280097a0 tx=0x7f2a2802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1efa5eed132abeb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:16.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.058+0000 7f2a3d1b8640 1 -- 172.21.15.45:0/1566250323 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2802d040 con 0x7f2a38071620 2023-11-28T20:35:16.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.058+0000 7f2a3d1b8640 1 -- 172.21.15.45:0/1566250323 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a28031810 con 0x7f2a38071620 2023-11-28T20:35:16.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.058+0000 7f2a3d1b8640 1 -- 172.21.15.45:0/1566250323 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28031ba0 con 0x7f2a38071620 2023-11-28T20:35:16.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.059+0000 7f2a4041e640 1 -- 172.21.15.45:0/1566250323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38071620 msgr2=0x7f2a380ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.059+0000 7f2a4041e640 1 --2- 172.21.15.45:0/1566250323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38071620 0x7f2a380ad2c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2a280097a0 tx=0x7f2a2802b8b0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.059+0000 7f2a4041e640 1 -- 172.21.15.45:0/1566250323 shutdown_connections 2023-11-28T20:35:16.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.059+0000 7f2a4041e640 1 --2- 172.21.15.45:0/1566250323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38070a60 0x7f2a38070f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.059+0000 7f2a4041e640 1 --2- 172.21.15.45:0/1566250323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38071620 0x7f2a380ad2c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.059+0000 7f2a4041e640 1 -- 172.21.15.45:0/1566250323 >> 172.21.15.45:0/1566250323 conn(0x7f2a380b13a0 msgr2=0x7f2a380b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:16.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.059+0000 7f2a4041e640 1 -- 172.21.15.45:0/1566250323 shutdown_connections 2023-11-28T20:35:16.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.059+0000 7f2a4041e640 1 -- 172.21.15.45:0/1566250323 wait complete. 2023-11-28T20:35:16.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.061+0000 7f2a4041e640 1 Processor -- start 2023-11-28T20:35:16.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.061+0000 7f2a4041e640 1 -- start start 2023-11-28T20:35:16.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.062+0000 7f2a4041e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38070a60 0x7f2a3814ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.062+0000 7f2a4041e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38071620 0x7f2a38157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.062+0000 7f2a4041e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a381607e0 0x7f2a38158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.062+0000 7f2a4041e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a381f2bc0 con 0x7f2a38070a60 2023-11-28T20:35:16.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.062+0000 7f2a4041e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a381557a0 con 0x7f2a381607e0 2023-11-28T20:35:16.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.062+0000 7f2a4041e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a381551e0 con 0x7f2a38071620 2023-11-28T20:35:16.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.062+0000 7f2a3e1ba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38070a60 0x7f2a3814ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.062+0000 7f2a3e1ba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38070a60 0x7f2a3814ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56154/0 (socket says 172.21.15.45:56154) 2023-11-28T20:35:16.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.062+0000 7f2a3e1ba640 1 -- 172.21.15.45:0/794601923 learned_addr learned my addr 172.21.15.45:0/794601923 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:16.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a3d9b9640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38071620 0x7f2a38157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a3e9bb640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a381607e0 0x7f2a38158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a3e1ba640 1 -- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38071620 msgr2=0x7f2a38157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a3e1ba640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38071620 0x7f2a38157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a3e1ba640 1 -- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a381607e0 msgr2=0x7f2a38158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a3e1ba640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a381607e0 0x7f2a38158190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a3e1ba640 1 -- 172.21.15.45:0/794601923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28009400 con 0x7f2a38070a60 2023-11-28T20:35:16.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a3e1ba640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38070a60 0x7f2a3814ca70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400c400 tx=0x7f2a3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:16.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a2effd640 1 -- 172.21.15.45:0/794601923 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a340052d0 con 0x7f2a38070a60 2023-11-28T20:35:16.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a2effd640 1 -- 172.21.15.45:0/794601923 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a34005430 con 0x7f2a38070a60 2023-11-28T20:35:16.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a2effd640 1 -- 172.21.15.45:0/794601923 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34003830 con 0x7f2a38070a60 2023-11-28T20:35:16.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a381554c0 con 0x7f2a38070a60 2023-11-28T20:35:16.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.063+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3815e520 con 0x7f2a38070a60 2023-11-28T20:35:16.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.064+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3815e0a0 con 0x7f2a38070a60 2023-11-28T20:35:16.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.066+0000 7f2a2effd640 1 -- 172.21.15.45:0/794601923 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2a34003990 con 0x7f2a38070a60 2023-11-28T20:35:16.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.066+0000 7f2a2effd640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a1c06ae40 0x7f2a1c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.067+0000 7f2a2effd640 1 -- 172.21.15.45:0/794601923 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2a34088ff0 con 0x7f2a38070a60 2023-11-28T20:35:16.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.067+0000 7f2a3d9b9640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a1c06ae40 0x7f2a1c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.068+0000 7f2a3d9b9640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a1c06ae40 0x7f2a1c06d300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2a28036000 tx=0x7f2a2803c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:16.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.071+0000 7f2a2effd640 1 -- 172.21.15.45:0/794601923 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a3815e0a0 con 0x7f2a38070a60 2023-11-28T20:35:16.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.256+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2a3815e0a0 con 0x7f2a38070a60 2023-11-28T20:35:16.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.257+0000 7f2a2effd640 1 -- 172.21.15.45:0/794601923 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2a3815e0a0 con 0x7f2a38070a60 2023-11-28T20:35:16.258 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a1c06ae40 msgr2=0x7f2a1c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a1c06ae40 0x7f2a1c06d300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2a28036000 tx=0x7f2a2803c040 comp rx=0 tx=0).stop 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38070a60 msgr2=0x7f2a3814ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38070a60 0x7f2a3814ca70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400c400 tx=0x7f2a3400c8c0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 shutdown_connections 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2a1c06ae40 0x7f2a1c06d300 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a381607e0 0x7f2a38158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a38071620 0x7f2a38157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 --2- 172.21.15.45:0/794601923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a38070a60 0x7f2a3814ca70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 >> 172.21.15.45:0/794601923 conn(0x7f2a380b13a0 msgr2=0x7f2a38156b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:16.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.259+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 shutdown_connections 2023-11-28T20:35:16.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.260+0000 7f2a4041e640 1 -- 172.21.15.45:0/794601923 wait complete. 2023-11-28T20:35:16.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:35:16.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.504+0000 7fe65556e640 1 Processor -- start 2023-11-28T20:35:16.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.504+0000 7fe65556e640 1 -- start start 2023-11-28T20:35:16.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.506+0000 7fe65556e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500d0040 0x7fe6500d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.506+0000 7fe65556e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe6500b1d40 0x7fe6500d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.506+0000 7fe65556e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6500b90a0 con 0x7fe6500b2290 2023-11-28T20:35:16.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.506+0000 7fe65556e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6500d1920 con 0x7fe6500b1d40 2023-11-28T20:35:16.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.506+0000 7fe65556e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6500d2dc0 con 0x7fe6500d0040 2023-11-28T20:35:16.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.506+0000 7fe64f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe6500b2290 0x7fe6500d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58814/0 (socket says 172.21.15.45:58814) 2023-11-28T20:35:16.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.506+0000 7fe64f7fe640 1 -- 172.21.15.45:0/726168615 learned_addr learned my addr 172.21.15.45:0/726168615 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:16.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64ffff640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500d0040 0x7fe6500d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe654d6d640 1 --2- 172.21.15.45:0/726168615 >> v2:172.21.15.190:3300/0 conn(0x7fe6500b1d40 0x7fe6500d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526314925 0 0) 0x7fe6500b90a0 con 0x7fe6500b2290 2023-11-28T20:35:16.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe638003520 con 0x7fe6500b2290 2023-11-28T20:35:16.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1585382139 0 0) 0x7fe638003520 con 0x7fe6500b2290 2023-11-28T20:35:16.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500d0040 msgr2=0x7fe6500d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:16.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500d0040 0x7fe6500d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 >> v2:172.21.15.190:3300/0 conn(0x7fe6500b1d40 msgr2=0x7fe6500d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 --2- 172.21.15.45:0/726168615 >> v2:172.21.15.190:3300/0 conn(0x7fe6500b1d40 0x7fe6500d3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6500d0960 con 0x7fe6500b2290 2023-11-28T20:35:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fe644001670 con 0x7fe6500b2290 2023-11-28T20:35:16.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.507+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 >> v1:172.21.15.45:6789/0 conn(0x7fe6500b2290 legacy=0x7fe6500d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64effd640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe638003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64effd640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6380047c0 0x7fe638004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64effd640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500d0040 0x7fe638005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6500d1920 con 0x7fe638003970 2023-11-28T20:35:16.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6500d2dc0 con 0x7fe6380047c0 2023-11-28T20:35:16.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63800b9b0 con 0x7fe6500d0040 2023-11-28T20:35:16.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64ffff640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe638003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe654d6d640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500d0040 0x7fe638005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64f7fe640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6380047c0 0x7fe638004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64ffff640 1 -- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500d0040 msgr2=0x7fe638005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64ffff640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500d0040 0x7fe638005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64ffff640 1 -- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6380047c0 msgr2=0x7fe638004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64ffff640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6380047c0 0x7fe638004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64ffff640 1 -- 172.21.15.45:0/726168615 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe638003520 con 0x7fe638003970 2023-11-28T20:35:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.508+0000 7fe64f7fe640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6380047c0 0x7fe638004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.509+0000 7fe64ffff640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe638003d90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe640009ac0 tx=0x7fe6380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f44161340fb7265 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:16.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.510+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64002c6b0 con 0x7fe638003970 2023-11-28T20:35:16.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.510+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64002c810 con 0x7fe638003970 2023-11-28T20:35:16.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.510+0000 7fe64effd640 1 -- 172.21.15.45:0/726168615 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64002cba0 con 0x7fe638003970 2023-11-28T20:35:16.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.510+0000 7fe65556e640 1 -- 172.21.15.45:0/726168615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 msgr2=0x7fe638003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.510+0000 7fe65556e640 1 --2- 172.21.15.45:0/726168615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe638003d90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe640009ac0 tx=0x7fe6380057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.510+0000 7fe654d6d640 1 -- 172.21.15.45:0/726168615 reap_dead start 2023-11-28T20:35:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.510+0000 7fe65556e640 1 -- 172.21.15.45:0/726168615 shutdown_connections 2023-11-28T20:35:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.510+0000 7fe65556e640 1 -- 172.21.15.45:0/726168615 >> 172.21.15.45:0/726168615 conn(0x7fe650070f40 msgr2=0x7fe650071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.511+0000 7fe65556e640 1 -- 172.21.15.45:0/726168615 shutdown_connections 2023-11-28T20:35:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.511+0000 7fe65556e640 1 -- 172.21.15.45:0/726168615 wait complete. 2023-11-28T20:35:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.512+0000 7fe65556e640 1 Processor -- start 2023-11-28T20:35:16.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.512+0000 7fe65556e640 1 -- start start 2023-11-28T20:35:16.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.513+0000 7fe65556e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe6500c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.513+0000 7fe65556e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500c7c70 0x7fe650156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.513+0000 7fe65556e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6500b70a0 0x7fe6500b7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.513+0000 7fe65556e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6500c80f0 con 0x7fe638003970 2023-11-28T20:35:16.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.513+0000 7fe65556e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6500b7a60 con 0x7fe6500b70a0 2023-11-28T20:35:16.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.513+0000 7fe65556e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe650157470 con 0x7fe6500c7c70 2023-11-28T20:35:16.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.514+0000 7fe64ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe6500c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.514+0000 7fe64ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe6500c9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56174/0 (socket says 172.21.15.45:56174) 2023-11-28T20:35:16.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.514+0000 7fe64ffff640 1 -- 172.21.15.45:0/1248329528 learned_addr learned my addr 172.21.15.45:0/1248329528 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:16.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.514+0000 7fe64ffff640 1 -- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500c7c70 msgr2=0x7fe650156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.514+0000 7fe64ffff640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500c7c70 0x7fe650156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.514+0000 7fe64ffff640 1 -- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6500b70a0 msgr2=0x7fe6500b7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.514+0000 7fe64ffff640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6500b70a0 0x7fe6500b7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.514+0000 7fe64ffff640 1 -- 172.21.15.45:0/1248329528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6500c82e0 con 0x7fe638003970 2023-11-28T20:35:16.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.514+0000 7fe64ffff640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe6500c9380 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe64002c250 tx=0x7fe640004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:16.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.515+0000 7fe64cff9640 1 -- 172.21.15.45:0/1248329528 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640003810 con 0x7fe638003970 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.515+0000 7fe64cff9640 1 -- 172.21.15.45:0/1248329528 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe640003970 con 0x7fe638003970 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.515+0000 7fe64cff9640 1 -- 172.21.15.45:0/1248329528 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640003ce0 con 0x7fe638003970 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.515+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6500c85c0 con 0x7fe638003970 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.515+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6500c8b80 con 0x7fe638003970 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.516+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe650071b30 con 0x7fe638003970 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.517+0000 7fe64cff9640 1 -- 172.21.15.45:0/1248329528 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe640047450 con 0x7fe638003970 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.518+0000 7fe64cff9640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe63c06ae40 0x7fe63c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.518+0000 7fe64cff9640 1 -- 172.21.15.45:0/1248329528 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe6400abe30 con 0x7fe638003970 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.519+0000 7fe64f7fe640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe63c06ae40 0x7fe63c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.520+0000 7fe64f7fe640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe63c06ae40 0x7fe63c06d300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe644000c00 tx=0x7fe644009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:16.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.522+0000 7fe64cff9640 1 -- 172.21.15.45:0/1248329528 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe640076aa0 con 0x7fe638003970 2023-11-28T20:35:16.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.716+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe65004e920 con 0x7fe638003970 2023-11-28T20:35:16.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.718+0000 7fe64cff9640 1 -- 172.21.15.45:0/1248329528 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe64004c020 con 0x7fe638003970 2023-11-28T20:35:16.719 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:16.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe63c06ae40 msgr2=0x7fe63c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe63c06ae40 0x7fe63c06d300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe644000c00 tx=0x7fe644009040 comp rx=0 tx=0).stop 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 msgr2=0x7fe6500c9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe6500c9380 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe64002c250 tx=0x7fe640004730 comp rx=0 tx=0).stop 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 shutdown_connections 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe63c06ae40 0x7fe63c06d300 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6500b70a0 0x7fe6500b7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6500c7c70 0x7fe650156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 --2- 172.21.15.45:0/1248329528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe638003970 0x7fe6500c9380 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 >> 172.21.15.45:0/1248329528 conn(0x7fe650070f40 msgr2=0x7fe6500d3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.720+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 shutdown_connections 2023-11-28T20:35:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.721+0000 7fe65556e640 1 -- 172.21.15.45:0/1248329528 wait complete. 2023-11-28T20:35:16.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:35:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.973+0000 7f7f1030f640 1 Processor -- start 2023-11-28T20:35:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.973+0000 7f7f1030f640 1 -- start start 2023-11-28T20:35:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.974+0000 7f7f1030f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f080cf020 0x7f7f080cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.974+0000 7f7f1030f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f080b3110 con 0x7f7f080ce390 2023-11-28T20:35:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.974+0000 7f7f1030f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f08070990 con 0x7f7f0815d590 2023-11-28T20:35:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.974+0000 7f7f1030f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080cf9f0 con 0x7f7f080cf020 2023-11-28T20:35:16.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.975+0000 7f7f0d8aa640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7f080ce390 0x7f7f08070280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58818/0 (socket says 172.21.15.45:58818) 2023-11-28T20:35:16.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.975+0000 7f7f0d8aa640 1 -- 172.21.15.45:0/1244136111 learned_addr learned my addr 172.21.15.45:0/1244136111 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:16.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.975+0000 7f7f0e8ac640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f080cf020 0x7f7f080cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.975+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341231034 0 0) 0x7f7f080b3110 con 0x7f7f080ce390 2023-11-28T20:35:16.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.975+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef4003520 con 0x7f7f080ce390 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081406217 0 0) 0x7f7f08070990 con 0x7f7f0815d590 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f080b3110 con 0x7f7f0815d590 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659237926 0 0) 0x7f7ef4003520 con 0x7f7f080ce390 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f080cf020 msgr2=0x7f7f080cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f080cf020 0x7f7f080cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 >> v1:172.21.15.190:6789/0 conn(0x7f7f0815d590 legacy=0x7f7f0806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0806d9e0 con 0x7f7f080ce390 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f7ef8001670 con 0x7f7f080ce390 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 >> v1:172.21.15.45:6789/0 conn(0x7f7f080ce390 legacy=0x7f7f08070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f080cf020 0x7f7ef4003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef40040e0 0x7f7ef40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef4004d00 0x7f7ef4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080cf9f0 con 0x7f7ef40040e0 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef400bb40 con 0x7f7ef4004d00 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef400bd00 con 0x7f7f080cf020 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.976+0000 7f7f0d8aa640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef40040e0 0x7f7ef40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.977+0000 7f7f0e0ab640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f080cf020 0x7f7ef4003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.977+0000 7f7f0e8ac640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef4004d00 0x7f7ef4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.977+0000 7f7f0d8aa640 1 -- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f080cf020 msgr2=0x7f7ef4003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.977+0000 7f7f0d8aa640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f080cf020 0x7f7ef4003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.977+0000 7f7f0d8aa640 1 -- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef4004d00 msgr2=0x7f7ef4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.977+0000 7f7f0d8aa640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef4004d00 0x7f7ef4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.977+0000 7f7f0d8aa640 1 -- 172.21.15.45:0/1244136111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4003520 con 0x7f7ef40040e0 2023-11-28T20:35:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.977+0000 7f7f0e8ac640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef4004d00 0x7f7ef4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:16.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.978+0000 7f7f0d8aa640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef40040e0 0x7f7ef40047c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8007f90 tx=0x7f7ef40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8d3c07328f03cf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:16.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.978+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8005ad0 con 0x7f7ef40040e0 2023-11-28T20:35:16.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.978+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef8005c30 con 0x7f7ef40040e0 2023-11-28T20:35:16.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.978+0000 7f7f0d0a9640 1 -- 172.21.15.45:0/1244136111 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8007290 con 0x7f7ef40040e0 2023-11-28T20:35:16.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.979+0000 7f7f1030f640 1 -- 172.21.15.45:0/1244136111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef40040e0 msgr2=0x7f7ef40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.979+0000 7f7f1030f640 1 --2- 172.21.15.45:0/1244136111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef40040e0 0x7f7ef40047c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8007f90 tx=0x7f7ef40057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.979+0000 7f7f0e8ac640 1 -- 172.21.15.45:0/1244136111 reap_dead start 2023-11-28T20:35:16.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.979+0000 7f7f1030f640 1 -- 172.21.15.45:0/1244136111 shutdown_connections 2023-11-28T20:35:16.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.979+0000 7f7f1030f640 1 -- 172.21.15.45:0/1244136111 >> 172.21.15.45:0/1244136111 conn(0x7f7f0806a6e0 msgr2=0x7f7f0806aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:16.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.979+0000 7f7f1030f640 1 -- 172.21.15.45:0/1244136111 shutdown_connections 2023-11-28T20:35:16.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.979+0000 7f7f1030f640 1 -- 172.21.15.45:0/1244136111 wait complete. 2023-11-28T20:35:16.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.981+0000 7f7f1030f640 1 Processor -- start 2023-11-28T20:35:16.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.981+0000 7f7f1030f640 1 -- start start 2023-11-28T20:35:16.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.982+0000 7f7f1030f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ef40040e0 0x7f7f081e6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.982+0000 7f7f1030f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f0815d970 0x7f7f0815efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.982+0000 7f7f1030f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f081e7ad0 0x7f7f0814b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.982+0000 7f7f1030f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f081e72b0 con 0x7f7f081e7ad0 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.982+0000 7f7f1030f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0815df30 con 0x7f7f0815d970 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.982+0000 7f7f1030f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0815e210 con 0x7f7ef40040e0 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.982+0000 7f7f0e8ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f081e7ad0 0x7f7f0814b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.982+0000 7f7f0e8ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f081e7ad0 0x7f7f0814b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56192/0 (socket says 172.21.15.45:56192) 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.982+0000 7f7f0e8ac640 1 -- 172.21.15.45:0/1015756956 learned_addr learned my addr 172.21.15.45:0/1015756956 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0d8aa640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f0815d970 0x7f7f0815efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0e8ac640 1 -- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ef40040e0 msgr2=0x7f7f081e6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0e0ab640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ef40040e0 0x7f7f081e6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0e8ac640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ef40040e0 0x7f7f081e6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0e8ac640 1 -- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f0815d970 msgr2=0x7f7f0815efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0e8ac640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f0815d970 0x7f7f0815efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0e8ac640 1 -- 172.21.15.45:0/1015756956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0815e4f0 con 0x7f7f081e7ad0 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0d8aa640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f0815d970 0x7f7f0815efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0e0ab640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ef40040e0 0x7f7f081e6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.983+0000 7f7f0e8ac640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f081e7ad0 0x7f7f0814b490 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7f080cfeb0 tx=0x7f7f0000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.984+0000 7f7efeffd640 1 -- 172.21.15.45:0/1015756956 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00017400 con 0x7f7f081e7ad0 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.984+0000 7f7efeffd640 1 -- 172.21.15.45:0/1015756956 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f00011040 con 0x7f7f081e7ad0 2023-11-28T20:35:16.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.984+0000 7f7efeffd640 1 -- 172.21.15.45:0/1015756956 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f000177c0 con 0x7f7f081e7ad0 2023-11-28T20:35:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.984+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0815e7d0 con 0x7f7f081e7ad0 2023-11-28T20:35:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.984+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0815f510 con 0x7f7f081e7ad0 2023-11-28T20:35:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.985+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f080b1e40 con 0x7f7f081e7ad0 2023-11-28T20:35:16.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.986+0000 7f7efeffd640 1 -- 172.21.15.45:0/1015756956 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7f000203b0 con 0x7f7f081e7ad0 2023-11-28T20:35:16.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.987+0000 7f7efeffd640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7eec06ae40 0x7f7eec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:16.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.987+0000 7f7efeffd640 1 -- 172.21.15.45:0/1015756956 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7f00089b10 con 0x7f7f081e7ad0 2023-11-28T20:35:16.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.988+0000 7f7f0e0ab640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7eec06ae40 0x7f7eec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:16.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.989+0000 7f7f0e0ab640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7eec06ae40 0x7f7eec06d300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7f04007ae0 tx=0x7f7f04009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:16.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:16.992+0000 7f7efeffd640 1 -- 172.21.15.45:0/1015756956 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f00054800 con 0x7f7f081e7ad0 2023-11-28T20:35:17.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.203+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7f0815ecf0 con 0x7f7f081e7ad0 2023-11-28T20:35:17.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.204+0000 7f7efeffd640 1 -- 172.21.15.45:0/1015756956 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7f0815ecf0 con 0x7f7f081e7ad0 2023-11-28T20:35:17.205 INFO:teuthology.orchestra.run.smithi045.stderr:listed 0 entries 2023-11-28T20:35:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.206+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7eec06ae40 msgr2=0x7f7eec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.206+0000 7f7f1030f640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7eec06ae40 0x7f7eec06d300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7f04007ae0 tx=0x7f7f04009040 comp rx=0 tx=0).stop 2023-11-28T20:35:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.206+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f081e7ad0 msgr2=0x7f7f0814b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.206+0000 7f7f1030f640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f081e7ad0 0x7f7f0814b490 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7f080cfeb0 tx=0x7f7f0000f4a0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.207+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 shutdown_connections 2023-11-28T20:35:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.207+0000 7f7f1030f640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f7eec06ae40 0x7f7eec06d300 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.207+0000 7f7f1030f640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f081e7ad0 0x7f7f0814b490 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.207+0000 7f7f1030f640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f0815d970 0x7f7f0815efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.207+0000 7f7f1030f640 1 --2- 172.21.15.45:0/1015756956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ef40040e0 0x7f7f081e6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.207+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 >> 172.21.15.45:0/1015756956 conn(0x7f7f0806a6e0 msgr2=0x7f7f080cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:17.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.207+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 shutdown_connections 2023-11-28T20:35:17.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.207+0000 7f7f1030f640 1 -- 172.21.15.45:0/1015756956 wait complete. 2023-11-28T20:35:17.228 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-11-28T20:35:17.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:17.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-11-28T20:35:17.338 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T20:35:17.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:17.338 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-28T20:35:17.388 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T20:35:17.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:35:17.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.625+0000 7fb814c09640 1 Processor -- start 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.625+0000 7fb814c09640 1 -- start start 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.626+0000 7fb814c09640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8100ae540 0x7fb8100ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.627+0000 7fb814c09640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81015c3c0 con 0x7fb8100ce630 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.627+0000 7fb814c09640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb810155b60 con 0x7fb810155780 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.627+0000 7fb814c09640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100ceac0 con 0x7fb8100ae540 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.627+0000 7fb80ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb8100ce630 0x7fb8100b20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58824/0 (socket says 172.21.15.45:58824) 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.627+0000 7fb80ffff640 1 -- 172.21.15.45:0/2847211797 learned_addr learned my addr 172.21.15.45:0/2847211797 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.627+0000 7fb807fff640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8100ae540 0x7fb8100ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.627+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366134254 0 0) 0x7fb81015c3c0 con 0x7fb8100ce630 2023-11-28T20:35:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc003520 con 0x7fb8100ce630 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2265587819 0 0) 0x7fb7fc003520 con 0x7fb8100ce630 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8100ae540 msgr2=0x7fb8100ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8100ae540 0x7fb8100ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 >> v1:172.21.15.190:6789/0 conn(0x7fb810155780 legacy=0x7fb8100abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81015ba90 con 0x7fb8100ce630 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb8080019f0 con 0x7fb8100ce630 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 >> v1:172.21.15.45:6789/0 conn(0x7fb8100ce630 legacy=0x7fb8100b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb7fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7fc0047c0 0x7fb7fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8100ae540 0x7fb7fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100ceac0 con 0x7fb7fc0039a0 2023-11-28T20:35:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc00ba60 con 0x7fb7fc0047c0 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.628+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc00bc20 con 0x7fb8100ae540 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb80f7fe640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb7fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb807fff640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7fc0047c0 0x7fb7fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb80ffff640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8100ae540 0x7fb7fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb80f7fe640 1 -- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8100ae540 msgr2=0x7fb7fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb80f7fe640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8100ae540 0x7fb7fc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb80f7fe640 1 -- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7fc0047c0 msgr2=0x7fb7fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb80f7fe640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7fc0047c0 0x7fb7fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb80f7fe640 1 -- 172.21.15.45:0/2847211797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc003520 con 0x7fb7fc0039a0 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb807fff640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7fc0047c0 0x7fb7fc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.629+0000 7fb80ffff640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8100ae540 0x7fb7fc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:17.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.631+0000 7fb80f7fe640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb7fc003dc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb800001720 tx=0x7fb7fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67954b19569732b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.631+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8000052d0 con 0x7fb7fc0039a0 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.631+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80002e050 con 0x7fb7fc0039a0 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.631+0000 7fb80effd640 1 -- 172.21.15.45:0/2847211797 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8000035f0 con 0x7fb7fc0039a0 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.632+0000 7fb814c09640 1 -- 172.21.15.45:0/2847211797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 msgr2=0x7fb7fc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.632+0000 7fb814c09640 1 --2- 172.21.15.45:0/2847211797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb7fc003dc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb800001720 tx=0x7fb7fc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.632+0000 7fb80ffff640 1 -- 172.21.15.45:0/2847211797 reap_dead start 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.632+0000 7fb814c09640 1 -- 172.21.15.45:0/2847211797 shutdown_connections 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.632+0000 7fb814c09640 1 -- 172.21.15.45:0/2847211797 >> 172.21.15.45:0/2847211797 conn(0x7fb81006ab30 msgr2=0x7fb8100b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.632+0000 7fb814c09640 1 -- 172.21.15.45:0/2847211797 shutdown_connections 2023-11-28T20:35:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.632+0000 7fb814c09640 1 -- 172.21.15.45:0/2847211797 wait complete. 2023-11-28T20:35:17.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.634+0000 7fb814c09640 1 Processor -- start 2023-11-28T20:35:17.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.634+0000 7fb814c09640 1 -- start start 2023-11-28T20:35:17.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.635+0000 7fb814c09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb810099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:17.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.635+0000 7fb814c09640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb810143920 0x7fb810071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:17.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.635+0000 7fb814c09640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8100afa50 0x7fb8100afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:17.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.635+0000 7fb814c09640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb810072290 con 0x7fb7fc0039a0 2023-11-28T20:35:17.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.635+0000 7fb814c09640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100b0410 con 0x7fb8100afa50 2023-11-28T20:35:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.635+0000 7fb814c09640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81014bf50 con 0x7fb810143920 2023-11-28T20:35:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.635+0000 7fb80f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb810099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.635+0000 7fb80f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb810099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56214/0 (socket says 172.21.15.45:56214) 2023-11-28T20:35:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.635+0000 7fb80f7fe640 1 -- 172.21.15.45:0/4215228176 learned_addr learned my addr 172.21.15.45:0/4215228176 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.636+0000 7fb80ffff640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8100afa50 0x7fb8100afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.636+0000 7fb807fff640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb810143920 0x7fb810071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.636+0000 7fb80f7fe640 1 -- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb810143920 msgr2=0x7fb810071d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.636+0000 7fb80f7fe640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb810143920 0x7fb810071d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.636+0000 7fb80f7fe640 1 -- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8100afa50 msgr2=0x7fb8100afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:17.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.636+0000 7fb80f7fe640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8100afa50 0x7fb8100afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.636+0000 7fb80f7fe640 1 -- 172.21.15.45:0/4215228176 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb810143d80 con 0x7fb7fc0039a0 2023-11-28T20:35:17.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.636+0000 7fb80f7fe640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb810099a40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb800008fd0 tx=0x7fb800001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:17.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.636+0000 7fb80cff9640 1 -- 172.21.15.45:0/4215228176 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8000076c0 con 0x7fb7fc0039a0 2023-11-28T20:35:17.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.637+0000 7fb80cff9640 1 -- 172.21.15.45:0/4215228176 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8000069e0 con 0x7fb7fc0039a0 2023-11-28T20:35:17.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.637+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81009a100 con 0x7fb7fc0039a0 2023-11-28T20:35:17.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.637+0000 7fb80cff9640 1 -- 172.21.15.45:0/4215228176 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800006d50 con 0x7fb7fc0039a0 2023-11-28T20:35:17.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.637+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81009a6c0 con 0x7fb7fc0039a0 2023-11-28T20:35:17.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.638+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81014c210 con 0x7fb7fc0039a0 2023-11-28T20:35:17.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.639+0000 7fb80cff9640 1 -- 172.21.15.45:0/4215228176 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb8000035f0 con 0x7fb7fc0039a0 2023-11-28T20:35:17.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.640+0000 7fb80cff9640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb7f006ae70 0x7fb7f006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:17.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.640+0000 7fb80cff9640 1 -- 172.21.15.45:0/4215228176 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb8000ab790 con 0x7fb7fc0039a0 2023-11-28T20:35:17.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.640+0000 7fb807fff640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb7f006ae70 0x7fb7f006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:17.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.642+0000 7fb807fff640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb7f006ae70 0x7fb7f006d330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb81015a1e0 tx=0x7fb7f8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:17.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.644+0000 7fb80cff9640 1 -- 172.21.15.45:0/4215228176 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb800076500 con 0x7fb7fc0039a0 2023-11-28T20:35:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.876+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb81004e920 con 0x7fb7fc0039a0 2023-11-28T20:35:17.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.877+0000 7fb80cff9640 1 -- 172.21.15.45:0/4215228176 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4915 (secure 0 0 0) 0x7fb800079b20 con 0x7fb7fc0039a0 2023-11-28T20:35:17.879 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:34:22.516874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T20:35:17.879 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2023-11-28T20:35:17.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.879+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb7f006ae70 msgr2=0x7fb7f006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:17.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.879+0000 7fb814c09640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb7f006ae70 0x7fb7f006d330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb81015a1e0 tx=0x7fb7f8008040 comp rx=0 tx=0).stop 2023-11-28T20:35:17.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.879+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 msgr2=0x7fb810099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:17.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.879+0000 7fb814c09640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb810099a40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb800008fd0 tx=0x7fb800001120 comp rx=0 tx=0).stop 2023-11-28T20:35:17.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.880+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 shutdown_connections 2023-11-28T20:35:17.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.880+0000 7fb814c09640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb7f006ae70 0x7fb7f006d330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.880+0000 7fb814c09640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8100afa50 0x7fb8100afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.880+0000 7fb814c09640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb810143920 0x7fb810071d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.880+0000 7fb814c09640 1 --2- 172.21.15.45:0/4215228176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc0039a0 0x7fb810099a40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:17.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.880+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 >> 172.21.15.45:0/4215228176 conn(0x7fb81006ab30 msgr2=0x7fb81015d730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:17.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.880+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 shutdown_connections 2023-11-28T20:35:17.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:17.880+0000 7fb814c09640 1 -- 172.21.15.45:0/4215228176 wait complete. 2023-11-28T20:35:17.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:18.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.117+0000 7f60bfbd0640 1 Processor -- start 2023-11-28T20:35:18.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.118+0000 7f60bfbd0640 1 -- start start 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.119+0000 7f60bfbd0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f60b80b10b0 0x7f60b80718a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.119+0000 7f60bfbd0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b80ce710 con 0x7f60b815d320 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.119+0000 7f60bfbd0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b80ae2c0 con 0x7f60b80b0c30 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.119+0000 7f60bfbd0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b80ae5a0 con 0x7f60b80b10b0 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.119+0000 7f60be16d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f60b80b10b0 0x7f60b80718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.119+0000 7f60be16d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f60b80b10b0 0x7f60b80718a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56228/0 (socket says 172.21.15.45:56228) 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.119+0000 7f60be16d640 1 -- 172.21.15.45:0/2104972804 learned_addr learned my addr 172.21.15.45:0/2104972804 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.119+0000 7f60bd96c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f60b815d320 0x7f60b80ad800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58832/0 (socket says 172.21.15.45:58832) 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.120+0000 7f60be16d640 1 -- 172.21.15.45:0/2104972804 >> v1:172.21.15.190:6789/0 conn(0x7f60b80b0c30 legacy=0x7f60b8071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.120+0000 7f60be16d640 1 -- 172.21.15.45:0/2104972804 >> v1:172.21.15.45:6789/0 conn(0x7f60b815d320 legacy=0x7f60b80ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.120+0000 7f60be16d640 1 -- 172.21.15.45:0/2104972804 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b81611f0 con 0x7f60b80b10b0 2023-11-28T20:35:18.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.121+0000 7f60be16d640 1 --2- 172.21.15.45:0/2104972804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 0x7f60b80718a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009e50 tx=0x7f60b002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44186d7daf182b39 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:18.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.121+0000 7f60bc96a640 1 -- 172.21.15.45:0/2104972804 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b002cb10 con 0x7f60b80b10b0 2023-11-28T20:35:18.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.121+0000 7f60bc96a640 1 -- 172.21.15.45:0/2104972804 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b002cc70 con 0x7f60b80b10b0 2023-11-28T20:35:18.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.121+0000 7f60bc96a640 1 -- 172.21.15.45:0/2104972804 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b00343f0 con 0x7f60b80b10b0 2023-11-28T20:35:18.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.122+0000 7f60bfbd0640 1 -- 172.21.15.45:0/2104972804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 msgr2=0x7f60b80718a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.122+0000 7f60bfbd0640 1 --2- 172.21.15.45:0/2104972804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 0x7f60b80718a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009e50 tx=0x7f60b002bf70 comp rx=0 tx=0).stop 2023-11-28T20:35:18.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.122+0000 7f60bfbd0640 1 -- 172.21.15.45:0/2104972804 shutdown_connections 2023-11-28T20:35:18.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.122+0000 7f60bfbd0640 1 --2- 172.21.15.45:0/2104972804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 0x7f60b80718a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.122+0000 7f60bfbd0640 1 -- 172.21.15.45:0/2104972804 >> 172.21.15.45:0/2104972804 conn(0x7f60b806a500 msgr2=0x7f60b806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.122+0000 7f60bfbd0640 1 -- 172.21.15.45:0/2104972804 shutdown_connections 2023-11-28T20:35:18.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.122+0000 7f60bfbd0640 1 -- 172.21.15.45:0/2104972804 wait complete. 2023-11-28T20:35:18.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.124+0000 7f60bfbd0640 1 Processor -- start 2023-11-28T20:35:18.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.124+0000 7f60bfbd0640 1 -- start start 2023-11-28T20:35:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.126+0000 7f60bfbd0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b80b0c30 0x7f60b815e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.126+0000 7f60bfbd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 0x7f60b814f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.126+0000 7f60bfbd0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b815d320 0x7f60b815eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.126+0000 7f60bfbd0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b8160790 con 0x7f60b80b10b0 2023-11-28T20:35:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.126+0000 7f60bfbd0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b8160300 con 0x7f60b815d320 2023-11-28T20:35:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.126+0000 7f60bfbd0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b815f6f0 con 0x7f60b80b0c30 2023-11-28T20:35:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.126+0000 7f60bd16b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 0x7f60b814f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.126+0000 7f60bd16b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 0x7f60b814f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56234/0 (socket says 172.21.15.45:56234) 2023-11-28T20:35:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.126+0000 7f60bd16b640 1 -- 172.21.15.45:0/898129476 learned_addr learned my addr 172.21.15.45:0/898129476 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60bd96c640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b80b0c30 0x7f60b815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60bd16b640 1 -- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b80b0c30 msgr2=0x7f60b815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60be16d640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b815d320 0x7f60b815eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60bd16b640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b80b0c30 0x7f60b815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60bd16b640 1 -- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b815d320 msgr2=0x7f60b815eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60bd16b640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b815d320 0x7f60b815eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60bd16b640 1 -- 172.21.15.45:0/898129476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b0009b30 con 0x7f60b80b10b0 2023-11-28T20:35:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60bd96c640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b80b0c30 0x7f60b815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60be16d640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b815d320 0x7f60b815eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:18.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60bd16b640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 0x7f60b814f950 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00b3e0 tx=0x7f60ac00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:18.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60b67fc640 1 -- 172.21.15.45:0/898129476 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac011830 con 0x7f60b80b10b0 2023-11-28T20:35:18.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.127+0000 7f60b67fc640 1 -- 172.21.15.45:0/898129476 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac0119b0 con 0x7f60b80b10b0 2023-11-28T20:35:18.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.128+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b815f9d0 con 0x7f60b80b10b0 2023-11-28T20:35:18.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.128+0000 7f60b67fc640 1 -- 172.21.15.45:0/898129476 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac011d60 con 0x7f60b80b10b0 2023-11-28T20:35:18.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.128+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b815ff90 con 0x7f60b80b10b0 2023-11-28T20:35:18.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.129+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b80733d0 con 0x7f60b80b10b0 2023-11-28T20:35:18.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.130+0000 7f60b67fc640 1 -- 172.21.15.45:0/898129476 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f60ac01a400 con 0x7f60b80b10b0 2023-11-28T20:35:18.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.131+0000 7f60b67fc640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f60a406ae40 0x7f60a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.131+0000 7f60b67fc640 1 -- 172.21.15.45:0/898129476 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f60ac089bf0 con 0x7f60b80b10b0 2023-11-28T20:35:18.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.131+0000 7f60bd96c640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f60a406ae40 0x7f60a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.133+0000 7f60bd96c640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f60a406ae40 0x7f60a406d300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f60a8000f60 tx=0x7f60a8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:18.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.136+0000 7f60b67fc640 1 -- 172.21.15.45:0/898129476 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60ac054840 con 0x7f60b80b10b0 2023-11-28T20:35:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.334+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60b815d9b0 con 0x7f60b80b10b0 2023-11-28T20:35:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.336+0000 7f60b67fc640 1 -- 172.21.15.45:0/898129476 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7f60ac015020 con 0x7f60b80b10b0 2023-11-28T20:35:18.337 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:18.338 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.338+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f60a406ae40 msgr2=0x7f60a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f60a406ae40 0x7f60a406d300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f60a8000f60 tx=0x7f60a8009040 comp rx=0 tx=0).stop 2023-11-28T20:35:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 msgr2=0x7f60b814f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 0x7f60b814f950 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00b3e0 tx=0x7f60ac00b8a0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 shutdown_connections 2023-11-28T20:35:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f60a406ae40 0x7f60a406d300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b815d320 0x7f60b815eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b80b10b0 0x7f60b814f950 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 --2- 172.21.15.45:0/898129476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b80b0c30 0x7f60b815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 >> 172.21.15.45:0/898129476 conn(0x7f60b806a500 msgr2=0x7f60b8155eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 shutdown_connections 2023-11-28T20:35:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.339+0000 7f60bfbd0640 1 -- 172.21.15.45:0/898129476 wait complete. 2023-11-28T20:35:18.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:33:51.417060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:33:52.566772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T20:33:56.137628+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T20:35:18.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:35:18.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.585+0000 7feb53fff640 1 Processor -- start 2023-11-28T20:35:18.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.585+0000 7feb53fff640 1 -- start start 2023-11-28T20:35:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.586+0000 7feb53fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb540b3040 0x7feb5406cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb53fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb5415b270 0x7feb5415f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb53fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb540b3420 0x7feb540b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb53fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb540ceaa0 con 0x7feb540b3420 2023-11-28T20:35:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb53fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5415faf0 con 0x7feb540b3040 2023-11-28T20:35:18.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb53fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5406d410 con 0x7feb5415b270 2023-11-28T20:35:18.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb537fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb540b3420 0x7feb540b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb537fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb540b3420 0x7feb540b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56250/0 (socket says 172.21.15.45:56250) 2023-11-28T20:35:18.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb537fe640 1 -- 172.21.15.45:0/310214663 learned_addr learned my addr 172.21.15.45:0/310214663 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:18.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb527fc640 1 --2- 172.21.15.45:0/310214663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb5415b270 0x7feb5415f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.587+0000 7feb52ffd640 1 --2- 172.21.15.45:0/310214663 >> v2:172.21.15.190:3300/0 conn(0x7feb540b3040 0x7feb5406cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.588+0000 7feb537fe640 1 -- 172.21.15.45:0/310214663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb5415b270 msgr2=0x7feb5415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:18.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.588+0000 7feb537fe640 1 --2- 172.21.15.45:0/310214663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb5415b270 0x7feb5415f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.588+0000 7feb537fe640 1 -- 172.21.15.45:0/310214663 >> v2:172.21.15.190:3300/0 conn(0x7feb540b3040 msgr2=0x7feb5406cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:18.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.588+0000 7feb537fe640 1 --2- 172.21.15.45:0/310214663 >> v2:172.21.15.190:3300/0 conn(0x7feb540b3040 0x7feb5406cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.588+0000 7feb537fe640 1 -- 172.21.15.45:0/310214663 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5406d6f0 con 0x7feb540b3420 2023-11-28T20:35:18.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.588+0000 7feb527fc640 1 --2- 172.21.15.45:0/310214663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb5415b270 0x7feb5415f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:18.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.589+0000 7feb537fe640 1 --2- 172.21.15.45:0/310214663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3420 0x7feb540b1d20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feb540b4410 tx=0x7feb4c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2940ffdd2fb8ba7f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:18.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.589+0000 7feb51ffb640 1 -- 172.21.15.45:0/310214663 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c02c950 con 0x7feb540b3420 2023-11-28T20:35:18.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.589+0000 7feb51ffb640 1 -- 172.21.15.45:0/310214663 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c02cab0 con 0x7feb540b3420 2023-11-28T20:35:18.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.589+0000 7feb51ffb640 1 -- 172.21.15.45:0/310214663 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c02ce40 con 0x7feb540b3420 2023-11-28T20:35:18.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.590+0000 7feb53fff640 1 -- 172.21.15.45:0/310214663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3420 msgr2=0x7feb540b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.590+0000 7feb53fff640 1 --2- 172.21.15.45:0/310214663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3420 0x7feb540b1d20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feb540b4410 tx=0x7feb4c02bdb0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.590+0000 7feb53fff640 1 -- 172.21.15.45:0/310214663 shutdown_connections 2023-11-28T20:35:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.590+0000 7feb53fff640 1 --2- 172.21.15.45:0/310214663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3420 0x7feb540b1d20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.590+0000 7feb53fff640 1 --2- 172.21.15.45:0/310214663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb5415b270 0x7feb5415f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.590+0000 7feb53fff640 1 --2- 172.21.15.45:0/310214663 >> v2:172.21.15.190:3300/0 conn(0x7feb540b3040 0x7feb5406cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.590+0000 7feb53fff640 1 -- 172.21.15.45:0/310214663 >> 172.21.15.45:0/310214663 conn(0x7feb540b1050 msgr2=0x7feb540cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.590+0000 7feb53fff640 1 -- 172.21.15.45:0/310214663 shutdown_connections 2023-11-28T20:35:18.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.590+0000 7feb53fff640 1 -- 172.21.15.45:0/310214663 wait complete. 2023-11-28T20:35:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.592+0000 7feb53fff640 1 Processor -- start 2023-11-28T20:35:18.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.592+0000 7feb53fff640 1 -- start start 2023-11-28T20:35:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.593+0000 7feb53fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3040 0x7feb54158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.593+0000 7feb53fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb540b3420 0x7feb5414f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.593+0000 7feb53fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5415b270 0x7feb5415a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.593+0000 7feb53fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb540acc60 con 0x7feb540b3040 2023-11-28T20:35:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.593+0000 7feb53fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb54158da0 con 0x7feb5415b270 2023-11-28T20:35:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.593+0000 7feb53fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb540ac0e0 con 0x7feb540b3420 2023-11-28T20:35:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.593+0000 7feb52ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3040 0x7feb54158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.593+0000 7feb52ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3040 0x7feb54158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56260/0 (socket says 172.21.15.45:56260) 2023-11-28T20:35:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.593+0000 7feb52ffd640 1 -- 172.21.15.45:0/3977368897 learned_addr learned my addr 172.21.15.45:0/3977368897 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb527fc640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb540b3420 0x7feb5414f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb52ffd640 1 -- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb540b3420 msgr2=0x7feb5414f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb537fe640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5415b270 0x7feb5415a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb52ffd640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb540b3420 0x7feb5414f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb52ffd640 1 -- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5415b270 msgr2=0x7feb5415a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb52ffd640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5415b270 0x7feb5415a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb52ffd640 1 -- 172.21.15.45:0/3977368897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c009b30 con 0x7feb540b3040 2023-11-28T20:35:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb527fc640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb540b3420 0x7feb5414f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb52ffd640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3040 0x7feb54158860 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7feb4400d830 tx=0x7feb4400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb3b7fe640 1 -- 172.21.15.45:0/3977368897 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4400be10 con 0x7feb540b3040 2023-11-28T20:35:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb3b7fe640 1 -- 172.21.15.45:0/3977368897 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb440098b0 con 0x7feb540b3040 2023-11-28T20:35:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb3b7fe640 1 -- 172.21.15.45:0/3977368897 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44009a40 con 0x7feb540b3040 2023-11-28T20:35:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.594+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb540ac3c0 con 0x7feb540b3040 2023-11-28T20:35:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.595+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb540ac980 con 0x7feb540b3040 2023-11-28T20:35:18.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.596+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5409d0f0 con 0x7feb540b3040 2023-11-28T20:35:18.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.597+0000 7feb3b7fe640 1 -- 172.21.15.45:0/3977368897 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7feb44009ba0 con 0x7feb540b3040 2023-11-28T20:35:18.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.598+0000 7feb3b7fe640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7feb4006ae40 0x7feb4006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:18.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.598+0000 7feb3b7fe640 1 -- 172.21.15.45:0/3977368897 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7feb44089d30 con 0x7feb540b3040 2023-11-28T20:35:18.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.598+0000 7feb527fc640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7feb4006ae40 0x7feb4006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:18.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.599+0000 7feb527fc640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7feb4006ae40 0x7feb4006d300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb48006fd0 tx=0x7feb48008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:18.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.602+0000 7feb3b7fe640 1 -- 172.21.15.45:0/3977368897 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb44054ab0 con 0x7feb540b3040 2023-11-28T20:35:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.796+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb5404e920 con 0x7feb540b3040 2023-11-28T20:35:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.797+0000 7feb3b7fe640 1 -- 172.21.15.45:0/3977368897 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb440580d0 con 0x7feb540b3040 2023-11-28T20:35:18.798 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:35:18.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7feb4006ae40 msgr2=0x7feb4006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7feb4006ae40 0x7feb4006d300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb48006fd0 tx=0x7feb48008040 comp rx=0 tx=0).stop 2023-11-28T20:35:18.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3040 msgr2=0x7feb54158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:18.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3040 0x7feb54158860 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7feb4400d830 tx=0x7feb4400dcf0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 shutdown_connections 2023-11-28T20:35:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7feb4006ae40 0x7feb4006d300 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb5415b270 0x7feb5415a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb540b3420 0x7feb5414f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 --2- 172.21.15.45:0/3977368897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb540b3040 0x7feb54158860 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 >> 172.21.15.45:0/3977368897 conn(0x7feb540b1050 msgr2=0x7feb54071280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 shutdown_connections 2023-11-28T20:35:18.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:18.799+0000 7feb53fff640 1 -- 172.21.15.45:0/3977368897 wait complete. 2023-11-28T20:35:18.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:35:19.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.052+0000 7fefa2ffb640 1 Processor -- start 2023-11-28T20:35:19.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.052+0000 7fefa2ffb640 1 -- start start 2023-11-28T20:35:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.053+0000 7fefa2ffb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef9c15b5e0 0x7fef9c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.053+0000 7fefa2ffb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fef9c0cfd20 0x7fef9c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.053+0000 7fefa2ffb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c15b9c0 con 0x7fef9c09a450 2023-11-28T20:35:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.053+0000 7fefa2ffb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c0cbe90 con 0x7fef9c15b5e0 2023-11-28T20:35:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.053+0000 7fefa2ffb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c0cd120 con 0x7fef9c0cfd20 2023-11-28T20:35:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.054+0000 7fefa0d97640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef9c15b5e0 0x7fef9c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.054+0000 7fefa0d97640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef9c15b5e0 0x7fef9c0ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56274/0 (socket says 172.21.15.45:56274) 2023-11-28T20:35:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.054+0000 7fefa0d97640 1 -- 172.21.15.45:0/2560572683 learned_addr learned my addr 172.21.15.45:0/2560572683 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.054+0000 7fefa0d97640 1 -- 172.21.15.45:0/2560572683 >> v2:172.21.15.190:3300/0 conn(0x7fef9c0cfd20 msgr2=0x7fef9c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:19.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.054+0000 7fefa0d97640 1 --2- 172.21.15.45:0/2560572683 >> v2:172.21.15.190:3300/0 conn(0x7fef9c0cfd20 0x7fef9c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.054+0000 7fefa0d97640 1 -- 172.21.15.45:0/2560572683 >> v1:172.21.15.190:6789/0 conn(0x7fef9c09a450 legacy=0x7fef9c0cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:19.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.054+0000 7fefa0d97640 1 -- 172.21.15.45:0/2560572683 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c0cc100 con 0x7fef9c15b5e0 2023-11-28T20:35:19.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.056+0000 7fefa0d97640 1 --2- 172.21.15.45:0/2560572683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c15b5e0 0x7fef9c0ccbe0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fef9c09f0c0 tx=0x7fef8c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25e514772de2b62 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:19.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.056+0000 7fef9b7fe640 1 -- 172.21.15.45:0/2560572683 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c02ca80 con 0x7fef9c15b5e0 2023-11-28T20:35:19.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.056+0000 7fef9b7fe640 1 -- 172.21.15.45:0/2560572683 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c02cbe0 con 0x7fef9c15b5e0 2023-11-28T20:35:19.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.056+0000 7fef9b7fe640 1 -- 172.21.15.45:0/2560572683 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0343f0 con 0x7fef9c15b5e0 2023-11-28T20:35:19.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.057+0000 7fefa2ffb640 1 -- 172.21.15.45:0/2560572683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c15b5e0 msgr2=0x7fef9c0ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.057+0000 7fefa2ffb640 1 --2- 172.21.15.45:0/2560572683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c15b5e0 0x7fef9c0ccbe0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fef9c09f0c0 tx=0x7fef8c02bee0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.057+0000 7fefa2ffb640 1 -- 172.21.15.45:0/2560572683 shutdown_connections 2023-11-28T20:35:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.057+0000 7fefa2ffb640 1 --2- 172.21.15.45:0/2560572683 >> v2:172.21.15.190:3300/0 conn(0x7fef9c0cfd20 0x7fef9c0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.057+0000 7fefa2ffb640 1 --2- 172.21.15.45:0/2560572683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c15b5e0 0x7fef9c0ccbe0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.057+0000 7fefa2ffb640 1 -- 172.21.15.45:0/2560572683 >> 172.21.15.45:0/2560572683 conn(0x7fef9c0b2b20 msgr2=0x7fef9c0b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.057+0000 7fefa2ffb640 1 -- 172.21.15.45:0/2560572683 shutdown_connections 2023-11-28T20:35:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.057+0000 7fefa2ffb640 1 -- 172.21.15.45:0/2560572683 wait complete. 2023-11-28T20:35:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.059+0000 7fefa2ffb640 1 Processor -- start 2023-11-28T20:35:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.059+0000 7fefa2ffb640 1 -- start start 2023-11-28T20:35:19.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fefa2ffb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9c09a450 0x7fef9c0c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fefa2ffb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c0cfd20 0x7fef9c15e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fefa2ffb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef9c15b5e0 0x7fef9c15c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fefa2ffb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c8fe0 con 0x7fef9c0cfd20 2023-11-28T20:35:19.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fefa2ffb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c158ba0 con 0x7fef9c15b5e0 2023-11-28T20:35:19.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fefa2ffb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c158e80 con 0x7fef9c09a450 2023-11-28T20:35:19.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fef9bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c0cfd20 0x7fef9c15e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fef9bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c0cfd20 0x7fef9c15e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56284/0 (socket says 172.21.15.45:56284) 2023-11-28T20:35:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fef9bfff640 1 -- 172.21.15.45:0/3978811383 learned_addr learned my addr 172.21.15.45:0/3978811383 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fefa0d97640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9c09a450 0x7fef9c0c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fefa1598640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef9c15b5e0 0x7fef9c15c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fef9bfff640 1 -- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9c09a450 msgr2=0x7fef9c0c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fef9bfff640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9c09a450 0x7fef9c0c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fef9bfff640 1 -- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef9c15b5e0 msgr2=0x7fef9c15c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.060+0000 7fef9bfff640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef9c15b5e0 0x7fef9c15c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.061+0000 7fef9bfff640 1 -- 172.21.15.45:0/3978811383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c009b30 con 0x7fef9c0cfd20 2023-11-28T20:35:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.061+0000 7fef9bfff640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c0cfd20 0x7fef9c15e310 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fef9000b3e0 tx=0x7fef9000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:19.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.061+0000 7fef997fa640 1 -- 172.21.15.45:0/3978811383 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90011830 con 0x7fef9c0cfd20 2023-11-28T20:35:19.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.061+0000 7fef997fa640 1 -- 172.21.15.45:0/3978811383 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef900119b0 con 0x7fef9c0cfd20 2023-11-28T20:35:19.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.061+0000 7fef997fa640 1 -- 172.21.15.45:0/3978811383 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90011d60 con 0x7fef9c0cfd20 2023-11-28T20:35:19.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.061+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c159160 con 0x7fef9c0cfd20 2023-11-28T20:35:19.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.062+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c0c92b0 con 0x7fef9c0cfd20 2023-11-28T20:35:19.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.063+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c06b150 con 0x7fef9c0cfd20 2023-11-28T20:35:19.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.063+0000 7fef997fa640 1 -- 172.21.15.45:0/3978811383 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fef9001a400 con 0x7fef9c0cfd20 2023-11-28T20:35:19.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.064+0000 7fef997fa640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fef8806ae40 0x7fef8806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.065+0000 7fef997fa640 1 -- 172.21.15.45:0/3978811383 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fef90089b40 con 0x7fef9c0cfd20 2023-11-28T20:35:19.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.065+0000 7fefa0d97640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fef8806ae40 0x7fef8806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.066+0000 7fefa0d97640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fef8806ae40 0x7fef8806d300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fef8c035000 tx=0x7fef8c030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.069+0000 7fef997fa640 1 -- 172.21.15.45:0/3978811383 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef9c06b150 con 0x7fef9c0cfd20 2023-11-28T20:35:19.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.272+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fef9c06b150 con 0x7fef9c0cfd20 2023-11-28T20:35:19.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.273+0000 7fef997fa640 1 -- 172.21.15.45:0/3978811383 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fef9c06b150 con 0x7fef9c0cfd20 2023-11-28T20:35:19.274 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.277+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fef8806ae40 msgr2=0x7fef8806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.277+0000 7fefa2ffb640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fef8806ae40 0x7fef8806d300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fef8c035000 tx=0x7fef8c030000 comp rx=0 tx=0).stop 2023-11-28T20:35:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.277+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c0cfd20 msgr2=0x7fef9c15e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.277+0000 7fefa2ffb640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c0cfd20 0x7fef9c15e310 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fef9000b3e0 tx=0x7fef9000b8a0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.278+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 shutdown_connections 2023-11-28T20:35:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.278+0000 7fefa2ffb640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fef8806ae40 0x7fef8806d300 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.278+0000 7fefa2ffb640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef9c15b5e0 0x7fef9c15c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.278+0000 7fefa2ffb640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef9c0cfd20 0x7fef9c15e310 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.278+0000 7fefa2ffb640 1 --2- 172.21.15.45:0/3978811383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9c09a450 0x7fef9c0c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.278+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 >> 172.21.15.45:0/3978811383 conn(0x7fef9c0b2b20 msgr2=0x7fef9c0b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.278+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 shutdown_connections 2023-11-28T20:35:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.278+0000 7fefa2ffb640 1 -- 172.21.15.45:0/3978811383 wait complete. 2023-11-28T20:35:19.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:35:19.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.523+0000 7f699b59e640 1 Processor -- start 2023-11-28T20:35:19.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.523+0000 7f699b59e640 1 -- start start 2023-11-28T20:35:19.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.525+0000 7f699b59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f699c0b3040 0x7f699c06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.525+0000 7f699b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f699c0b3420 0x7f699c0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.525+0000 7f699b59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699c06d410 con 0x7f699c15b270 2023-11-28T20:35:19.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.525+0000 7f699b59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699c0ceaa0 con 0x7f699c0b3040 2023-11-28T20:35:19.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.525+0000 7f699b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699c06d6f0 con 0x7f699c0b3420 2023-11-28T20:35:19.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.525+0000 7f699a59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f699c0b3040 0x7f699c06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.525+0000 7f699a59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f699c0b3040 0x7f699c06cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56286/0 (socket says 172.21.15.45:56286) 2023-11-28T20:35:19.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.525+0000 7f699a59c640 1 -- 172.21.15.45:0/3947300483 learned_addr learned my addr 172.21.15.45:0/3947300483 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:19.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.525+0000 7f6993fff640 1 --1- 172.21.15.45:0/3947300483 >> v1:172.21.15.45:6789/0 conn(0x7f699c15b270 0x7f699c15f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58836/0 (socket says 172.21.15.45:58836) 2023-11-28T20:35:19.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.526+0000 7f699ad9d640 1 --2- 172.21.15.45:0/3947300483 >> v2:172.21.15.190:3300/0 conn(0x7f699c0b3420 0x7f699c0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.526+0000 7f699a59c640 1 -- 172.21.15.45:0/3947300483 >> v2:172.21.15.190:3300/0 conn(0x7f699c0b3420 msgr2=0x7f699c0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:19.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.526+0000 7f699a59c640 1 --2- 172.21.15.45:0/3947300483 >> v2:172.21.15.190:3300/0 conn(0x7f699c0b3420 0x7f699c0ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.526+0000 7f699a59c640 1 -- 172.21.15.45:0/3947300483 >> v1:172.21.15.45:6789/0 conn(0x7f699c15b270 legacy=0x7f699c15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.526+0000 7f699a59c640 1 -- 172.21.15.45:0/3947300483 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c06d9d0 con 0x7f699c0b3040 2023-11-28T20:35:19.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.526+0000 7f699ad9d640 1 --2- 172.21.15.45:0/3947300483 >> v2:172.21.15.190:3300/0 conn(0x7f699c0b3420 0x7f699c0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:19.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.527+0000 7f699a59c640 1 --2- 172.21.15.45:0/3947300483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c0b3040 0x7f699c06cda0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f699c0adfd0 tx=0x7f698c02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c92185264d0f17 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:19.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.527+0000 7f6999d9b640 1 -- 172.21.15.45:0/3947300483 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c0317f0 con 0x7f699c0b3040 2023-11-28T20:35:19.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.527+0000 7f6999d9b640 1 -- 172.21.15.45:0/3947300483 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c031950 con 0x7f699c0b3040 2023-11-28T20:35:19.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.527+0000 7f6999d9b640 1 -- 172.21.15.45:0/3947300483 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c031ce0 con 0x7f699c0b3040 2023-11-28T20:35:19.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.528+0000 7f699b59e640 1 -- 172.21.15.45:0/3947300483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c0b3040 msgr2=0x7f699c06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.528+0000 7f699b59e640 1 --2- 172.21.15.45:0/3947300483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c0b3040 0x7f699c06cda0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f699c0adfd0 tx=0x7f698c02b890 comp rx=0 tx=0).stop 2023-11-28T20:35:19.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.528+0000 7f699b59e640 1 -- 172.21.15.45:0/3947300483 shutdown_connections 2023-11-28T20:35:19.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.528+0000 7f699b59e640 1 --2- 172.21.15.45:0/3947300483 >> v2:172.21.15.190:3300/0 conn(0x7f699c0b3420 0x7f699c0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.528+0000 7f699b59e640 1 --2- 172.21.15.45:0/3947300483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c0b3040 0x7f699c06cda0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.528+0000 7f699b59e640 1 -- 172.21.15.45:0/3947300483 >> 172.21.15.45:0/3947300483 conn(0x7f699c0b1050 msgr2=0x7f699c0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:19.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.528+0000 7f699b59e640 1 -- 172.21.15.45:0/3947300483 shutdown_connections 2023-11-28T20:35:19.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.528+0000 7f699b59e640 1 -- 172.21.15.45:0/3947300483 wait complete. 2023-11-28T20:35:19.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.530+0000 7f699b59e640 1 Processor -- start 2023-11-28T20:35:19.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.530+0000 7f699b59e640 1 -- start start 2023-11-28T20:35:19.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.531+0000 7f699b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c0b3040 0x7f699c09bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.531+0000 7f699b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f699c0b3420 0x7f699c151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.531+0000 7f699b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c15b270 0x7f699c152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.531+0000 7f699b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c09c1f0 con 0x7f699c15b270 2023-11-28T20:35:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.531+0000 7f699b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c09c4d0 con 0x7f699c0b3040 2023-11-28T20:35:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.531+0000 7f699b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c09c7b0 con 0x7f699c0b3420 2023-11-28T20:35:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.531+0000 7f699ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c15b270 0x7f699c152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.531+0000 7f699ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c15b270 0x7f699c152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56300/0 (socket says 172.21.15.45:56300) 2023-11-28T20:35:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.531+0000 7f699ad9d640 1 -- 172.21.15.45:0/3657576173 learned_addr learned my addr 172.21.15.45:0/3657576173 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:19.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f699a59c640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c0b3040 0x7f699c09bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f6993fff640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f699c0b3420 0x7f699c151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f699ad9d640 1 -- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f699c0b3420 msgr2=0x7f699c151510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f699ad9d640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f699c0b3420 0x7f699c151510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f699ad9d640 1 -- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c0b3040 msgr2=0x7f699c09bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f699ad9d640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c0b3040 0x7f699c09bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f699ad9d640 1 -- 172.21.15.45:0/3657576173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c0094d0 con 0x7f699c15b270 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f699a59c640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c0b3040 0x7f699c09bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f699ad9d640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c15b270 0x7f699c152ec0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f699400cef0 tx=0x7f699400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.532+0000 7f6992ffd640 1 -- 172.21.15.45:0/3657576173 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699400d840 con 0x7f699c15b270 2023-11-28T20:35:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.533+0000 7f6992ffd640 1 -- 172.21.15.45:0/3657576173 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699400d9c0 con 0x7f699c15b270 2023-11-28T20:35:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.533+0000 7f6992ffd640 1 -- 172.21.15.45:0/3657576173 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699400dd70 con 0x7f699c15b270 2023-11-28T20:35:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.533+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c09ca90 con 0x7f699c15b270 2023-11-28T20:35:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.533+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c153400 con 0x7f699c15b270 2023-11-28T20:35:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.534+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c09d0f0 con 0x7f699c15b270 2023-11-28T20:35:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.535+0000 7f6992ffd640 1 -- 172.21.15.45:0/3657576173 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f699401e3b0 con 0x7f699c15b270 2023-11-28T20:35:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.536+0000 7f6992ffd640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f697c06ae70 0x7f697c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.536+0000 7f6992ffd640 1 -- 172.21.15.45:0/3657576173 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f699408a3e0 con 0x7f699c15b270 2023-11-28T20:35:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.536+0000 7f699a59c640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f697c06ae70 0x7f697c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.537+0000 7f699a59c640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f697c06ae70 0x7f697c06d330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f698c036000 tx=0x7f698c03d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:19.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.540+0000 7f6992ffd640 1 -- 172.21.15.45:0/3657576173 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6994055220 con 0x7f699c15b270 2023-11-28T20:35:19.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.737+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f699c04e920 con 0x7f699c15b270 2023-11-28T20:35:19.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.738+0000 7f6992ffd640 1 -- 172.21.15.45:0/3657576173 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6994058840 con 0x7f699c15b270 2023-11-28T20:35:19.740 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:35:19.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f697c06ae70 msgr2=0x7f697c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f697c06ae70 0x7f697c06d330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f698c036000 tx=0x7f698c03d040 comp rx=0 tx=0).stop 2023-11-28T20:35:19.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c15b270 msgr2=0x7f699c152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c15b270 0x7f699c152ec0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f699400cef0 tx=0x7f699400a710 comp rx=0 tx=0).stop 2023-11-28T20:35:19.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 shutdown_connections 2023-11-28T20:35:19.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f697c06ae70 0x7f697c06d330 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f699c15b270 0x7f699c152ec0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f699c0b3420 0x7f699c151510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 --2- 172.21.15.45:0/3657576173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f699c0b3040 0x7f699c09bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.741+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 >> 172.21.15.45:0/3657576173 conn(0x7f699c0b1050 msgr2=0x7f699c0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:19.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.742+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 shutdown_connections 2023-11-28T20:35:19.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.742+0000 7f699b59e640 1 -- 172.21.15.45:0/3657576173 wait complete. 2023-11-28T20:35:19.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T20:35:19.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.984+0000 7ff87b59e640 1 Processor -- start 2023-11-28T20:35:19.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.984+0000 7ff87b59e640 1 -- start start 2023-11-28T20:35:19.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.986+0000 7ff87b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff87c0b3040 0x7ff87c06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.986+0000 7ff87b59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff87c15b270 0x7ff87c15f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.986+0000 7ff87b59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c15faf0 con 0x7ff87c0b3420 2023-11-28T20:35:19.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.986+0000 7ff87b59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c0ceaa0 con 0x7ff87c15b270 2023-11-28T20:35:19.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.986+0000 7ff87b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c06d410 con 0x7ff87c0b3040 2023-11-28T20:35:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.986+0000 7ff87ad9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff87c0b3420 0x7ff87c0b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58848/0 (socket says 172.21.15.45:58848) 2023-11-28T20:35:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.986+0000 7ff879d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff87c15b270 0x7ff87c15f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.986+0000 7ff87ad9d640 1 -- 172.21.15.45:0/2888873255 learned_addr learned my addr 172.21.15.45:0/2888873255 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.987+0000 7ff87a59c640 1 --2- 172.21.15.45:0/2888873255 >> v2:172.21.15.190:3300/0 conn(0x7ff87c0b3040 0x7ff87c06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.987+0000 7ff87959a640 1 -- 172.21.15.45:0/2888873255 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772315386 0 0) 0x7ff87c15faf0 con 0x7ff87c0b3420 2023-11-28T20:35:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.987+0000 7ff879d9b640 1 -- 172.21.15.45:0/2888873255 >> v2:172.21.15.190:3300/0 conn(0x7ff87c0b3040 msgr2=0x7ff87c06cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.987+0000 7ff879d9b640 1 --2- 172.21.15.45:0/2888873255 >> v2:172.21.15.190:3300/0 conn(0x7ff87c0b3040 0x7ff87c06cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.987+0000 7ff879d9b640 1 -- 172.21.15.45:0/2888873255 >> v1:172.21.15.45:6789/0 conn(0x7ff87c0b3420 legacy=0x7ff87c0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.987+0000 7ff879d9b640 1 -- 172.21.15.45:0/2888873255 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c06d6f0 con 0x7ff87c15b270 2023-11-28T20:35:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.988+0000 7ff879d9b640 1 --2- 172.21.15.45:0/2888873255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c15b270 0x7ff87c15f5b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0b4410 tx=0x7ff86402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a7ad291a16d166 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.989+0000 7ff87959a640 1 -- 172.21.15.45:0/2888873255 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864031710 con 0x7ff87c15b270 2023-11-28T20:35:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.989+0000 7ff87959a640 1 -- 172.21.15.45:0/2888873255 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86402f050 con 0x7ff87c15b270 2023-11-28T20:35:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.989+0000 7ff87959a640 1 -- 172.21.15.45:0/2888873255 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864031a90 con 0x7ff87c15b270 2023-11-28T20:35:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.989+0000 7ff87b59e640 1 -- 172.21.15.45:0/2888873255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c15b270 msgr2=0x7ff87c15f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.989+0000 7ff87b59e640 1 --2- 172.21.15.45:0/2888873255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c15b270 0x7ff87c15f5b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0b4410 tx=0x7ff86402b7b0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.989+0000 7ff87b59e640 1 -- 172.21.15.45:0/2888873255 shutdown_connections 2023-11-28T20:35:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.989+0000 7ff87b59e640 1 --2- 172.21.15.45:0/2888873255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c15b270 0x7ff87c15f5b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.989+0000 7ff87b59e640 1 --2- 172.21.15.45:0/2888873255 >> v2:172.21.15.190:3300/0 conn(0x7ff87c0b3040 0x7ff87c06cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.989+0000 7ff87b59e640 1 -- 172.21.15.45:0/2888873255 >> 172.21.15.45:0/2888873255 conn(0x7ff87c0b1050 msgr2=0x7ff87c0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:19.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.990+0000 7ff87b59e640 1 -- 172.21.15.45:0/2888873255 shutdown_connections 2023-11-28T20:35:19.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.990+0000 7ff87b59e640 1 -- 172.21.15.45:0/2888873255 wait complete. 2023-11-28T20:35:19.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.991+0000 7ff87b59e640 1 Processor -- start 2023-11-28T20:35:19.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.992+0000 7ff87b59e640 1 -- start start 2023-11-28T20:35:19.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.992+0000 7ff87b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c0b3040 0x7ff87c1455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.992+0000 7ff87b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff87c0b3420 0x7ff87c1514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.992+0000 7ff87b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c15b270 0x7ff87c15deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c145b20 con 0x7ff87c0b3040 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c15cbc0 con 0x7ff87c15b270 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c15da20 con 0x7ff87c0b3420 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c0b3040 0x7ff87c1455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c0b3040 0x7ff87c1455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56330/0 (socket says 172.21.15.45:56330) 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87a59c640 1 -- 172.21.15.45:0/768956811 learned_addr learned my addr 172.21.15.45:0/768956811 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87a59c640 1 -- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff87c0b3420 msgr2=0x7ff87c1514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87a59c640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff87c0b3420 0x7ff87c1514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87a59c640 1 -- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c15b270 msgr2=0x7ff87c15deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87a59c640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c15b270 0x7ff87c15deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87a59c640 1 -- 172.21.15.45:0/768956811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff864009400 con 0x7ff87c0b3040 2023-11-28T20:35:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.993+0000 7ff87a59c640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c0b3040 0x7ff87c1455e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00efc0 tx=0x7ff86c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.994+0000 7ff872ffd640 1 -- 172.21.15.45:0/768956811 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c009b80 con 0x7ff87c0b3040 2023-11-28T20:35:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.994+0000 7ff872ffd640 1 -- 172.21.15.45:0/768956811 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c00f040 con 0x7ff87c0b3040 2023-11-28T20:35:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.994+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c151a00 con 0x7ff87c0b3040 2023-11-28T20:35:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.994+0000 7ff872ffd640 1 -- 172.21.15.45:0/768956811 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c0146a0 con 0x7ff87c0b3040 2023-11-28T20:35:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.994+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c15d0f0 con 0x7ff87c0b3040 2023-11-28T20:35:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.995+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c09d0f0 con 0x7ff87c0b3040 2023-11-28T20:35:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.996+0000 7ff872ffd640 1 -- 172.21.15.45:0/768956811 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff86c014800 con 0x7ff87c0b3040 2023-11-28T20:35:19.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.997+0000 7ff872ffd640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff85c06ae70 0x7ff85c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:19.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.997+0000 7ff872ffd640 1 -- 172.21.15.45:0/768956811 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff86c08a500 con 0x7ff87c0b3040 2023-11-28T20:35:19.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.997+0000 7ff879d9b640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff85c06ae70 0x7ff85c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:19.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:19.999+0000 7ff879d9b640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff85c06ae70 0x7ff85c06d330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff864035000 tx=0x7ff864030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.001+0000 7ff872ffd640 1 -- 172.21.15.45:0/768956811 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff86c055280 con 0x7ff87c0b3040 2023-11-28T20:35:20.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.195+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff87c04e920 con 0x7ff87c0b3040 2023-11-28T20:35:20.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.196+0000 7ff872ffd640 1 -- 172.21.15.45:0/768956811 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff86c0588a0 con 0x7ff87c0b3040 2023-11-28T20:35:20.197 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:35:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.198+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff85c06ae70 msgr2=0x7ff85c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.198+0000 7ff87b59e640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff85c06ae70 0x7ff85c06d330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff864035000 tx=0x7ff864030000 comp rx=0 tx=0).stop 2023-11-28T20:35:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.198+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c0b3040 msgr2=0x7ff87c1455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.198+0000 7ff87b59e640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c0b3040 0x7ff87c1455e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00efc0 tx=0x7ff86c00c490 comp rx=0 tx=0).stop 2023-11-28T20:35:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.199+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 shutdown_connections 2023-11-28T20:35:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.199+0000 7ff87b59e640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff85c06ae70 0x7ff85c06d330 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.199+0000 7ff87b59e640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c15b270 0x7ff87c15deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.199+0000 7ff87b59e640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff87c0b3420 0x7ff87c1514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.199+0000 7ff87b59e640 1 --2- 172.21.15.45:0/768956811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff87c0b3040 0x7ff87c1455e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.199+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 >> 172.21.15.45:0/768956811 conn(0x7ff87c0b1050 msgr2=0x7ff87c0cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:20.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.199+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 shutdown_connections 2023-11-28T20:35:20.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.199+0000 7ff87b59e640 1 -- 172.21.15.45:0/768956811 wait complete. 2023-11-28T20:35:20.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:35:20.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.444+0000 7fbabaddb640 1 Processor -- start 2023-11-28T20:35:20.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.445+0000 7fbabaddb640 1 -- start start 2023-11-28T20:35:20.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.446+0000 7fbabaddb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbab415d390 0x7fbab40cd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.446+0000 7fbabaddb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40d0050 0x7fbab40d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.446+0000 7fbabaddb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab40d1300 con 0x7fbab40cded0 2023-11-28T20:35:20.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.446+0000 7fbabaddb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab4161230 con 0x7fbab415d390 2023-11-28T20:35:20.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.446+0000 7fbabaddb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40ce500 con 0x7fbab40d0050 2023-11-28T20:35:20.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.447+0000 7fbaba5da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40d0050 0x7fbab40d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.447+0000 7fbab9dd9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbab415d390 0x7fbab40cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.447+0000 7fbaba5da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40d0050 0x7fbab40d1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:39902/0 (socket says 172.21.15.45:39902) 2023-11-28T20:35:20.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.447+0000 7fbaba5da640 1 -- 172.21.15.45:0/1055341000 learned_addr learned my addr 172.21.15.45:0/1055341000 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:20.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.448+0000 7fbaba5da640 1 -- 172.21.15.45:0/1055341000 >> v2:172.21.15.190:3300/0 conn(0x7fbab415d390 msgr2=0x7fbab40cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:20.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.448+0000 7fbaba5da640 1 --2- 172.21.15.45:0/1055341000 >> v2:172.21.15.190:3300/0 conn(0x7fbab415d390 0x7fbab40cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.448+0000 7fbaba5da640 1 -- 172.21.15.45:0/1055341000 >> v1:172.21.15.190:6789/0 conn(0x7fbab40cded0 legacy=0x7fbab40cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.448+0000 7fbaba5da640 1 -- 172.21.15.45:0/1055341000 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab40ce7e0 con 0x7fbab40d0050 2023-11-28T20:35:20.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.448+0000 7fbab9dd9640 1 --2- 172.21.15.45:0/1055341000 >> v2:172.21.15.190:3300/0 conn(0x7fbab415d390 0x7fbab40cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:35:20.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.449+0000 7fbaba5da640 1 --2- 172.21.15.45:0/1055341000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40d0050 0x7fbab40d1dc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009e50 tx=0x7fbaac02bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=326841e66bdcb722 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:20.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.450+0000 7fbab8dd7640 1 -- 172.21.15.45:0/1055341000 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac02ca80 con 0x7fbab40d0050 2023-11-28T20:35:20.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.450+0000 7fbab8dd7640 1 -- 172.21.15.45:0/1055341000 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac02cbe0 con 0x7fbab40d0050 2023-11-28T20:35:20.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.450+0000 7fbab8dd7640 1 -- 172.21.15.45:0/1055341000 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0343f0 con 0x7fbab40d0050 2023-11-28T20:35:20.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.451+0000 7fbabaddb640 1 -- 172.21.15.45:0/1055341000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40d0050 msgr2=0x7fbab40d1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.451+0000 7fbabaddb640 1 --2- 172.21.15.45:0/1055341000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40d0050 0x7fbab40d1dc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009e50 tx=0x7fbaac02bee0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.451+0000 7fbabaddb640 1 -- 172.21.15.45:0/1055341000 shutdown_connections 2023-11-28T20:35:20.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.451+0000 7fbabaddb640 1 --2- 172.21.15.45:0/1055341000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40d0050 0x7fbab40d1dc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.451+0000 7fbabaddb640 1 --2- 172.21.15.45:0/1055341000 >> v2:172.21.15.190:3300/0 conn(0x7fbab415d390 0x7fbab40cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.451+0000 7fbabaddb640 1 -- 172.21.15.45:0/1055341000 >> 172.21.15.45:0/1055341000 conn(0x7fbab406a500 msgr2=0x7fbab406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:20.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.451+0000 7fbabaddb640 1 -- 172.21.15.45:0/1055341000 shutdown_connections 2023-11-28T20:35:20.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.451+0000 7fbabaddb640 1 -- 172.21.15.45:0/1055341000 wait complete. 2023-11-28T20:35:20.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.453+0000 7fbabaddb640 1 Processor -- start 2023-11-28T20:35:20.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.453+0000 7fbabaddb640 1 -- start start 2023-11-28T20:35:20.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.454+0000 7fbabaddb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40cded0 0x7fbab415e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.454+0000 7fbabaddb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab40d0050 0x7fbab415a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.454+0000 7fbabaddb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab415d390 0x7fbab415af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.454+0000 7fbabaddb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4160a60 con 0x7fbab415d390 2023-11-28T20:35:20.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.454+0000 7fbabaddb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab41f7260 con 0x7fbab40d0050 2023-11-28T20:35:20.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.454+0000 7fbabaddb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab41f64a0 con 0x7fbab40cded0 2023-11-28T20:35:20.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.454+0000 7fbaba5da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab415d390 0x7fbab415af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbaba5da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab415d390 0x7fbab415af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56342/0 (socket says 172.21.15.45:56342) 2023-11-28T20:35:20.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbaba5da640 1 -- 172.21.15.45:0/69281274 learned_addr learned my addr 172.21.15.45:0/69281274 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:20.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbab95d8640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab40d0050 0x7fbab415a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbab9dd9640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40cded0 0x7fbab415e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbaba5da640 1 -- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40cded0 msgr2=0x7fbab415e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbaba5da640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40cded0 0x7fbab415e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbaba5da640 1 -- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab40d0050 msgr2=0x7fbab415a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbaba5da640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab40d0050 0x7fbab415a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbaba5da640 1 -- 172.21.15.45:0/69281274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab0009400 con 0x7fbab415d390 2023-11-28T20:35:20.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.455+0000 7fbaba5da640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab415d390 0x7fbab415af30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbaac036040 tx=0x7fbaac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:20.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.456+0000 7fbaaa7fc640 1 -- 172.21.15.45:0/69281274 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac007ea0 con 0x7fbab415d390 2023-11-28T20:35:20.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.456+0000 7fbaaa7fc640 1 -- 172.21.15.45:0/69281274 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac03e450 con 0x7fbab415d390 2023-11-28T20:35:20.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.456+0000 7fbaaa7fc640 1 -- 172.21.15.45:0/69281274 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac03e800 con 0x7fbab415d390 2023-11-28T20:35:20.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.456+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaac009b30 con 0x7fbab415d390 2023-11-28T20:35:20.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.456+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab41f6a60 con 0x7fbab415d390 2023-11-28T20:35:20.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.457+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab4071b30 con 0x7fbab415d390 2023-11-28T20:35:20.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.458+0000 7fbaaa7fc640 1 -- 172.21.15.45:0/69281274 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbaac02cd50 con 0x7fbab415d390 2023-11-28T20:35:20.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.459+0000 7fbaaa7fc640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fba9806ae70 0x7fba9806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.460+0000 7fbaaa7fc640 1 -- 172.21.15.45:0/69281274 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbaac0ac310 con 0x7fbab415d390 2023-11-28T20:35:20.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.460+0000 7fbab9dd9640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fba9806ae70 0x7fba9806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.461+0000 7fbab9dd9640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fba9806ae70 0x7fba9806d330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009c60 tx=0x7fbab0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:20.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.464+0000 7fbaaa7fc640 1 -- 172.21.15.45:0/69281274 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbaac076ff0 con 0x7fbab415d390 2023-11-28T20:35:20.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.698+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbab404e920 con 0x7fbab415d390 2023-11-28T20:35:20.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.699+0000 7fbaaa7fc640 1 -- 172.21.15.45:0/69281274 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4915 (secure 0 0 0) 0x7fbaac07a610 con 0x7fbab415d390 2023-11-28T20:35:20.700 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:20.701 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:34:22.516874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T20:35:20.701 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2023-11-28T20:35:20.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.701+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fba9806ae70 msgr2=0x7fba9806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.701+0000 7fbabaddb640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fba9806ae70 0x7fba9806d330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009c60 tx=0x7fbab0008040 comp rx=0 tx=0).stop 2023-11-28T20:35:20.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab415d390 msgr2=0x7fbab415af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab415d390 0x7fbab415af30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbaac036040 tx=0x7fbaac004a40 comp rx=0 tx=0).stop 2023-11-28T20:35:20.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 shutdown_connections 2023-11-28T20:35:20.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fba9806ae70 0x7fba9806d330 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab415d390 0x7fbab415af30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab40d0050 0x7fbab415a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 --2- 172.21.15.45:0/69281274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab40cded0 0x7fbab415e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 >> 172.21.15.45:0/69281274 conn(0x7fbab406a500 msgr2=0x7fbab41536f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:20.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 shutdown_connections 2023-11-28T20:35:20.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.702+0000 7fbabaddb640 1 -- 172.21.15.45:0/69281274 wait complete. 2023-11-28T20:35:20.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:20.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.944+0000 7effc6eff640 1 Processor -- start 2023-11-28T20:35:20.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.945+0000 7effc6eff640 1 -- start start 2023-11-28T20:35:20.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.946+0000 7effc6eff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ae540 0x7effc00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.946+0000 7effc6eff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7effc00ce630 0x7effc00b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.946+0000 7effc6eff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc015c3c0 con 0x7effc0155780 2023-11-28T20:35:20.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.946+0000 7effc6eff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc00ceac0 con 0x7effc00ce630 2023-11-28T20:35:20.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.946+0000 7effc6eff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc0155b60 con 0x7effc00ae540 2023-11-28T20:35:20.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.947+0000 7effc66fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7effc00ce630 0x7effc00b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.947+0000 7effc66fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7effc00ce630 0x7effc00b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56344/0 (socket says 172.21.15.45:56344) 2023-11-28T20:35:20.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.947+0000 7effc66fe640 1 -- 172.21.15.45:0/4084449020 learned_addr learned my addr 172.21.15.45:0/4084449020 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:20.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.947+0000 7effc56fc640 1 --2- 172.21.15.45:0/4084449020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ae540 0x7effc00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.947+0000 7effc66fe640 1 -- 172.21.15.45:0/4084449020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ae540 msgr2=0x7effc00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:20.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.947+0000 7effc66fe640 1 --2- 172.21.15.45:0/4084449020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ae540 0x7effc00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.947+0000 7effc66fe640 1 -- 172.21.15.45:0/4084449020 >> v1:172.21.15.190:6789/0 conn(0x7effc0155780 legacy=0x7effc00abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:20.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.947+0000 7effc66fe640 1 -- 172.21.15.45:0/4084449020 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc015ba90 con 0x7effc00ce630 2023-11-28T20:35:20.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.947+0000 7effc56fc640 1 --2- 172.21.15.45:0/4084449020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ae540 0x7effc00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.948+0000 7effc66fe640 1 --2- 172.21.15.45:0/4084449020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ce630 0x7effc00b20a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7effb8009e80 tx=0x7effb802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53cf2f42bf144eef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.949+0000 7effc4efb640 1 -- 172.21.15.45:0/4084449020 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb802ca50 con 0x7effc00ce630 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.949+0000 7effc4efb640 1 -- 172.21.15.45:0/4084449020 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb802cbb0 con 0x7effc00ce630 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.949+0000 7effc4efb640 1 -- 172.21.15.45:0/4084449020 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb80343f0 con 0x7effc00ce630 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.949+0000 7effc6eff640 1 -- 172.21.15.45:0/4084449020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ce630 msgr2=0x7effc00b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.949+0000 7effc6eff640 1 --2- 172.21.15.45:0/4084449020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ce630 0x7effc00b20a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7effb8009e80 tx=0x7effb802beb0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.949+0000 7effc6eff640 1 -- 172.21.15.45:0/4084449020 shutdown_connections 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.949+0000 7effc6eff640 1 --2- 172.21.15.45:0/4084449020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ce630 0x7effc00b20a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.949+0000 7effc6eff640 1 --2- 172.21.15.45:0/4084449020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ae540 0x7effc00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.949+0000 7effc6eff640 1 -- 172.21.15.45:0/4084449020 >> 172.21.15.45:0/4084449020 conn(0x7effc006ab30 msgr2=0x7effc00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.950+0000 7effc6eff640 1 -- 172.21.15.45:0/4084449020 shutdown_connections 2023-11-28T20:35:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.950+0000 7effc6eff640 1 -- 172.21.15.45:0/4084449020 wait complete. 2023-11-28T20:35:20.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.951+0000 7effc6eff640 1 Processor -- start 2023-11-28T20:35:20.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.951+0000 7effc6eff640 1 -- start start 2023-11-28T20:35:20.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.952+0000 7effc6eff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ae540 0x7effc0071740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.952+0000 7effc6eff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ce630 0x7effc0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.952+0000 7effc6eff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc0155780 0x7effc01499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.952+0000 7effc6eff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc014a430 con 0x7effc00ae540 2023-11-28T20:35:20.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.952+0000 7effc6eff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00706c0 con 0x7effc0155780 2023-11-28T20:35:20.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.952+0000 7effc6eff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00709a0 con 0x7effc00ce630 2023-11-28T20:35:20.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc5efd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ae540 0x7effc0071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc5efd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ae540 0x7effc0071740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56360/0 (socket says 172.21.15.45:56360) 2023-11-28T20:35:20.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc5efd640 1 -- 172.21.15.45:0/3653825610 learned_addr learned my addr 172.21.15.45:0/3653825610 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:20.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc66fe640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc0155780 0x7effc01499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc5efd640 1 -- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ce630 msgr2=0x7effc0099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc56fc640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ce630 0x7effc0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc5efd640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ce630 0x7effc0099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc5efd640 1 -- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc0155780 msgr2=0x7effc01499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:20.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc5efd640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc0155780 0x7effc01499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:20.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc5efd640 1 -- 172.21.15.45:0/3653825610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb8009b30 con 0x7effc00ae540 2023-11-28T20:35:20.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc56fc640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ce630 0x7effc0099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:20.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.953+0000 7effc66fe640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc0155780 0x7effc01499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:20.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.954+0000 7effc5efd640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ae540 0x7effc0071740 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7effbc00b3f0 tx=0x7effbc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:20.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.954+0000 7effb67fc640 1 -- 172.21.15.45:0/3653825610 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc011830 con 0x7effc00ae540 2023-11-28T20:35:20.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.954+0000 7effb67fc640 1 -- 172.21.15.45:0/3653825610 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effbc011990 con 0x7effc00ae540 2023-11-28T20:35:20.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.954+0000 7effb67fc640 1 -- 172.21.15.45:0/3653825610 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc011d40 con 0x7effc00ae540 2023-11-28T20:35:20.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.954+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc0070c80 con 0x7effc00ae540 2023-11-28T20:35:20.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.954+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc0071240 con 0x7effc00ae540 2023-11-28T20:35:20.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.955+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc004e920 con 0x7effc00ae540 2023-11-28T20:35:20.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.956+0000 7effb67fc640 1 -- 172.21.15.45:0/3653825610 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7effbc01a400 con 0x7effc00ae540 2023-11-28T20:35:20.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.957+0000 7effb67fc640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:20.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.957+0000 7effb67fc640 1 -- 172.21.15.45:0/3653825610 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7effbc089880 con 0x7effc00ae540 2023-11-28T20:35:20.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.958+0000 7effc56fc640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:20.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.959+0000 7effc56fc640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7effb0006fd0 tx=0x7effb0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:20.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:20.962+0000 7effb67fc640 1 -- 172.21.15.45:0/3653825610 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effbc054570 con 0x7effc00ae540 2023-11-28T20:35:21.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.160+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effc0149ef0 con 0x7effc00ae540 2023-11-28T20:35:21.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.163+0000 7effb67fc640 1 -- 172.21.15.45:0/3653825610 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7effbc057b90 con 0x7effc00ae540 2023-11-28T20:35:21.165 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:21.166 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:21.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 msgr2=0x7effa406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7effb0006fd0 tx=0x7effb0008040 comp rx=0 tx=0).stop 2023-11-28T20:35:21.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ae540 msgr2=0x7effc0071740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ae540 0x7effc0071740 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7effbc00b3f0 tx=0x7effbc00b8b0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 shutdown_connections 2023-11-28T20:35:21.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7effa406ae40 0x7effa406d300 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effc0155780 0x7effc01499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effc00ce630 0x7effc0099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 --2- 172.21.15.45:0/3653825610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effc00ae540 0x7effc0071740 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 >> 172.21.15.45:0/3653825610 conn(0x7effc006ab30 msgr2=0x7effc00ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:21.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 shutdown_connections 2023-11-28T20:35:21.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.166+0000 7effc6eff640 1 -- 172.21.15.45:0/3653825610 wait complete. 2023-11-28T20:35:21.188 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T20:35:21.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:35:21.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.411+0000 7f1d6759e640 1 Processor -- start 2023-11-28T20:35:21.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.411+0000 7f1d6759e640 1 -- start start 2023-11-28T20:35:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.413+0000 7f1d6759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d68155780 0x7f1d680abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.413+0000 7f1d6759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1d680ce800 0x7f1d680b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.413+0000 7f1d6759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6815c3c0 con 0x7f1d680ae540 2023-11-28T20:35:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.413+0000 7f1d6759e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d68155b60 con 0x7f1d680ce800 2023-11-28T20:35:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.413+0000 7f1d6759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6815ba90 con 0x7f1d68155780 2023-11-28T20:35:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d5ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1d680ae540 0x7f1d680ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58862/0 (socket says 172.21.15.45:58862) 2023-11-28T20:35:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d5ffff640 1 -- 172.21.15.45:0/713755109 learned_addr learned my addr 172.21.15.45:0/713755109 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d66d9d640 1 --2- 172.21.15.45:0/713755109 >> v2:172.21.15.45:3300/0 conn(0x7f1d680ce800 0x7f1d680b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d6659c640 1 --2- 172.21.15.45:0/713755109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d68155780 0x7f1d680abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d65d9b640 1 -- 172.21.15.45:0/713755109 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871774593 0 0) 0x7f1d6815c3c0 con 0x7f1d680ae540 2023-11-28T20:35:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d65d9b640 1 -- 172.21.15.45:0/713755109 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d54003520 con 0x7f1d680ae540 2023-11-28T20:35:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d66d9d640 1 -- 172.21.15.45:0/713755109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d68155780 msgr2=0x7f1d680abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d66d9d640 1 --2- 172.21.15.45:0/713755109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d68155780 0x7f1d680abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d66d9d640 1 -- 172.21.15.45:0/713755109 >> v1:172.21.15.45:6789/0 conn(0x7f1d680ae540 legacy=0x7f1d680ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.414+0000 7f1d66d9d640 1 -- 172.21.15.45:0/713755109 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6815bd70 con 0x7f1d680ce800 2023-11-28T20:35:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.416+0000 7f1d66d9d640 1 --2- 172.21.15.45:0/713755109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 0x7f1d680b20a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009d90 tx=0x7f1d6002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a83d9ed22a9123 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.416+0000 7f1d65d9b640 1 -- 172.21.15.45:0/713755109 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6002cc50 con 0x7f1d680ce800 2023-11-28T20:35:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.416+0000 7f1d65d9b640 1 -- 172.21.15.45:0/713755109 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6002cdb0 con 0x7f1d680ce800 2023-11-28T20:35:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.416+0000 7f1d65d9b640 1 -- 172.21.15.45:0/713755109 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d60034660 con 0x7f1d680ce800 2023-11-28T20:35:21.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.417+0000 7f1d6759e640 1 -- 172.21.15.45:0/713755109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 msgr2=0x7f1d680b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.417+0000 7f1d6759e640 1 --2- 172.21.15.45:0/713755109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 0x7f1d680b20a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009d90 tx=0x7f1d6002bea0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.417+0000 7f1d6759e640 1 -- 172.21.15.45:0/713755109 shutdown_connections 2023-11-28T20:35:21.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.417+0000 7f1d6759e640 1 --2- 172.21.15.45:0/713755109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 0x7f1d680b20a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.417+0000 7f1d6759e640 1 --2- 172.21.15.45:0/713755109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d68155780 0x7f1d680abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.417+0000 7f1d6759e640 1 -- 172.21.15.45:0/713755109 >> 172.21.15.45:0/713755109 conn(0x7f1d6806ab30 msgr2=0x7f1d680b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:21.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.417+0000 7f1d6759e640 1 -- 172.21.15.45:0/713755109 shutdown_connections 2023-11-28T20:35:21.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.417+0000 7f1d6759e640 1 -- 172.21.15.45:0/713755109 wait complete. 2023-11-28T20:35:21.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.419+0000 7f1d6759e640 1 Processor -- start 2023-11-28T20:35:21.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.419+0000 7f1d6759e640 1 -- start start 2023-11-28T20:35:21.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.420+0000 7f1d6759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d680ae540 0x7f1d68148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.420+0000 7f1d6759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 0x7f1d6809c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.420+0000 7f1d6759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d68155780 0x7f1d68071be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.420+0000 7f1d6759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68070bd0 con 0x7f1d680ce800 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.420+0000 7f1d6759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68071750 con 0x7f1d68155780 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.420+0000 7f1d6759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68148db0 con 0x7f1d680ae540 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.420+0000 7f1d5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 0x7f1d6809c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.420+0000 7f1d5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 0x7f1d6809c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56376/0 (socket says 172.21.15.45:56376) 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.420+0000 7f1d5ffff640 1 -- 172.21.15.45:0/1702176115 learned_addr learned my addr 172.21.15.45:0/1702176115 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d6659c640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d680ae540 0x7f1d68148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d66d9d640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d68155780 0x7f1d68071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d5ffff640 1 -- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d680ae540 msgr2=0x7f1d68148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d5ffff640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d680ae540 0x7f1d68148870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d5ffff640 1 -- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d68155780 msgr2=0x7f1d68071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d5ffff640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d68155780 0x7f1d68071be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d5ffff640 1 -- 172.21.15.45:0/1702176115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d60009a70 con 0x7f1d680ce800 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d66d9d640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d68155780 0x7f1d68071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d5ffff640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 0x7f1d6809c210 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000c530 tx=0x7f1d5000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d5effd640 1 -- 172.21.15.45:0/1702176115 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d50005ce0 con 0x7f1d680ce800 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d5effd640 1 -- 172.21.15.45:0/1702176115 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d50005e40 con 0x7f1d680ce800 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d5effd640 1 -- 172.21.15.45:0/1702176115 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d500072b0 con 0x7f1d680ce800 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.421+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d68070e20 con 0x7f1d680ce800 2023-11-28T20:35:21.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.422+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d680713e0 con 0x7f1d680ce800 2023-11-28T20:35:21.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.423+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d680718c0 con 0x7f1d680ce800 2023-11-28T20:35:21.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.424+0000 7f1d5effd640 1 -- 172.21.15.45:0/1702176115 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1d50007410 con 0x7f1d680ce800 2023-11-28T20:35:21.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.425+0000 7f1d5effd640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1d4806ae70 0x7f1d4806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.425+0000 7f1d5effd640 1 -- 172.21.15.45:0/1702176115 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1d50088fa0 con 0x7f1d680ce800 2023-11-28T20:35:21.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.426+0000 7f1d6659c640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1d4806ae70 0x7f1d4806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.427+0000 7f1d6659c640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1d4806ae70 0x7f1d4806d330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1d58006fd0 tx=0x7f1d58008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:21.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.430+0000 7f1d5effd640 1 -- 172.21.15.45:0/1702176115 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d50053d40 con 0x7f1d680ce800 2023-11-28T20:35:21.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.667+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d6804e920 con 0x7f1d680ce800 2023-11-28T20:35:21.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.668+0000 7f1d5effd640 1 -- 172.21.15.45:0/1702176115 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4915 (secure 0 0 0) 0x7f1d50017020 con 0x7f1d680ce800 2023-11-28T20:35:21.669 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:21.670 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:34:22.516874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T20:35:21.670 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1d4806ae70 msgr2=0x7f1d4806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1d4806ae70 0x7f1d4806d330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1d58006fd0 tx=0x7f1d58008040 comp rx=0 tx=0).stop 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 msgr2=0x7f1d6809c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 0x7f1d6809c210 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000c530 tx=0x7f1d5000c9f0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 shutdown_connections 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1d4806ae70 0x7f1d4806d330 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d68155780 0x7f1d68071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d680ce800 0x7f1d6809c210 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 --2- 172.21.15.45:0/1702176115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1d680ae540 0x7f1d68148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 >> 172.21.15.45:0/1702176115 conn(0x7f1d6806ab30 msgr2=0x7f1d6806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:21.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.671+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 shutdown_connections 2023-11-28T20:35:21.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.672+0000 7f1d6759e640 1 -- 172.21.15.45:0/1702176115 wait complete. 2023-11-28T20:35:21.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:35:21.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.912+0000 7f846692c640 1 Processor -- start 2023-11-28T20:35:21.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.913+0000 7f846692c640 1 -- start start 2023-11-28T20:35:21.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.914+0000 7f846692c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8460072200 0x7f84600b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.914+0000 7f846692c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f84600713e0 0x7f84600b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.914+0000 7f846692c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84600b4310 con 0x7f8460146610 2023-11-28T20:35:21.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.914+0000 7f846692c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84600725e0 con 0x7f8460072200 2023-11-28T20:35:21.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.914+0000 7f846692c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f846006d8d0 con 0x7f84600713e0 2023-11-28T20:35:21.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.915+0000 7f846592a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8460072200 0x7f84600b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.915+0000 7f846592a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8460072200 0x7f84600b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56380/0 (socket says 172.21.15.45:56380) 2023-11-28T20:35:21.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.915+0000 7f846592a640 1 -- 172.21.15.45:0/396552596 learned_addr learned my addr 172.21.15.45:0/396552596 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:21.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.915+0000 7f846612b640 1 --2- 172.21.15.45:0/396552596 >> v2:172.21.15.190:3300/0 conn(0x7f84600713e0 0x7f84600b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.915+0000 7f846592a640 1 -- 172.21.15.45:0/396552596 >> v2:172.21.15.190:3300/0 conn(0x7f84600713e0 msgr2=0x7f84600b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:21.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.915+0000 7f846592a640 1 --2- 172.21.15.45:0/396552596 >> v2:172.21.15.190:3300/0 conn(0x7f84600713e0 0x7f84600b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.915+0000 7f846592a640 1 -- 172.21.15.45:0/396552596 >> v1:172.21.15.190:6789/0 conn(0x7f8460146610 legacy=0x7f8460160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:21.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.915+0000 7f846592a640 1 -- 172.21.15.45:0/396552596 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846006ca20 con 0x7f8460072200 2023-11-28T20:35:21.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.915+0000 7f846612b640 1 --2- 172.21.15.45:0/396552596 >> v2:172.21.15.190:3300/0 conn(0x7f84600713e0 0x7f84600b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.917+0000 7f846592a640 1 --2- 172.21.15.45:0/396552596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 0x7f84600b1d90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f846006b390 tx=0x7f845c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1118ae7a92a66d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.917+0000 7f8464928640 1 -- 172.21.15.45:0/396552596 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c031870 con 0x7f8460072200 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.917+0000 7f8464928640 1 -- 172.21.15.45:0/396552596 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845c0319d0 con 0x7f8460072200 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.917+0000 7f8464928640 1 -- 172.21.15.45:0/396552596 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c031d60 con 0x7f8460072200 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.917+0000 7f846692c640 1 -- 172.21.15.45:0/396552596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 msgr2=0x7f84600b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.917+0000 7f846692c640 1 --2- 172.21.15.45:0/396552596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 0x7f84600b1d90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f846006b390 tx=0x7f845c02b910 comp rx=0 tx=0).stop 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.918+0000 7f846692c640 1 -- 172.21.15.45:0/396552596 shutdown_connections 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.918+0000 7f846692c640 1 --2- 172.21.15.45:0/396552596 >> v2:172.21.15.190:3300/0 conn(0x7f84600713e0 0x7f84600b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.918+0000 7f846692c640 1 --2- 172.21.15.45:0/396552596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 0x7f84600b1d90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.918+0000 7f846692c640 1 -- 172.21.15.45:0/396552596 >> 172.21.15.45:0/396552596 conn(0x7f846006ab30 msgr2=0x7f84600b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:21.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.918+0000 7f846692c640 1 -- 172.21.15.45:0/396552596 shutdown_connections 2023-11-28T20:35:21.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.918+0000 7f846692c640 1 -- 172.21.15.45:0/396552596 wait complete. 2023-11-28T20:35:21.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.920+0000 7f846692c640 1 Processor -- start 2023-11-28T20:35:21.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.920+0000 7f846692c640 1 -- start start 2023-11-28T20:35:21.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f846692c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84600713e0 0x7f846012e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f846692c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 0x7f846012ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f846692c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8460146610 0x7f846013f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f846692c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846012d850 con 0x7f8460072200 2023-11-28T20:35:21.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f846692c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846012db30 con 0x7f84600713e0 2023-11-28T20:35:21.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f846692c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846012de10 con 0x7f8460146610 2023-11-28T20:35:21.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f8465129640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 0x7f846012ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f8465129640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 0x7f846012ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56396/0 (socket says 172.21.15.45:56396) 2023-11-28T20:35:21.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f8465129640 1 -- 172.21.15.45:0/622405700 learned_addr learned my addr 172.21.15.45:0/622405700 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:21.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.921+0000 7f846612b640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8460146610 0x7f846013f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f846592a640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84600713e0 0x7f846012e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f8465129640 1 -- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8460146610 msgr2=0x7f846013f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f8465129640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8460146610 0x7f846013f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f8465129640 1 -- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84600713e0 msgr2=0x7f846012e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:21.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f8465129640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84600713e0 0x7f846012e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:21.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f8465129640 1 -- 172.21.15.45:0/622405700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c0094d0 con 0x7f8460072200 2023-11-28T20:35:21.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f8465129640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 0x7f846012ed30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f845000b3f0 tx=0x7f845000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:21.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f84567fc640 1 -- 172.21.15.45:0/622405700 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8450011830 con 0x7f8460072200 2023-11-28T20:35:21.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f84567fc640 1 -- 172.21.15.45:0/622405700 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8450011990 con 0x7f8460072200 2023-11-28T20:35:21.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f84567fc640 1 -- 172.21.15.45:0/622405700 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8450011d40 con 0x7f8460072200 2023-11-28T20:35:21.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846012e0f0 con 0x7f8460072200 2023-11-28T20:35:21.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.922+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846013f6b0 con 0x7f8460072200 2023-11-28T20:35:21.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.924+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846009d0f0 con 0x7f8460072200 2023-11-28T20:35:21.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.924+0000 7f84567fc640 1 -- 172.21.15.45:0/622405700 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f845001a400 con 0x7f8460072200 2023-11-28T20:35:21.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.925+0000 7f84567fc640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f844406ae40 0x7f844406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:21.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.926+0000 7f84567fc640 1 -- 172.21.15.45:0/622405700 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f84500898b0 con 0x7f8460072200 2023-11-28T20:35:21.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.926+0000 7f846592a640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f844406ae40 0x7f844406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:21.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.927+0000 7f846592a640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f844406ae40 0x7f844406d300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f845c036000 tx=0x7f845c03c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:21.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:21.932+0000 7f84567fc640 1 -- 172.21.15.45:0/622405700 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8450054500 con 0x7f8460072200 2023-11-28T20:35:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.171+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f846004e920 con 0x7f8460072200 2023-11-28T20:35:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.172+0000 7f84567fc640 1 -- 172.21.15.45:0/622405700 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4915 (secure 0 0 0) 0x7f8450057b20 con 0x7f8460072200 2023-11-28T20:35:22.173 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:22.174 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/3055251901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3055251901},{"type":"v1","addr":"172.21.15.190:6836","nonce":3055251901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4070299107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4070299107},{"type":"v1","addr":"172.21.15.190:6837","nonce":4070299107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/2959520831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2959520831},{"type":"v1","addr":"172.21.15.45:6833","nonce":2959520831}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:33:53.520990+0000","modified":"2023-11-28T20:34:22.516874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/2066232376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2066232376},{"type":"v1","addr":"172.21.15.45:6835","nonce":2066232376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T20:35:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2023-11-28T20:35:22.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f844406ae40 msgr2=0x7f844406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f844406ae40 0x7f844406d300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f845c036000 tx=0x7f845c03c040 comp rx=0 tx=0).stop 2023-11-28T20:35:22.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 msgr2=0x7f846012ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 0x7f846012ed30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f845000b3f0 tx=0x7f845000b8b0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 shutdown_connections 2023-11-28T20:35:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f844406ae40 0x7f844406d300 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8460146610 0x7f846013f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8460072200 0x7f846012ed30 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 --2- 172.21.15.45:0/622405700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84600713e0 0x7f846012e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 >> 172.21.15.45:0/622405700 conn(0x7f846006ab30 msgr2=0x7f84600b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 shutdown_connections 2023-11-28T20:35:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.175+0000 7f846692c640 1 -- 172.21.15.45:0/622405700 wait complete. 2023-11-28T20:35:22.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:22.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.422+0000 7f6a1d3e2640 1 Processor -- start 2023-11-28T20:35:22.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.422+0000 7f6a1d3e2640 1 -- start start 2023-11-28T20:35:22.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.423+0000 7f6a1d3e2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a1815d550 0x7f6a180cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.424+0000 7f6a1d3e2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6a180d1c80 0x7f6a180b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.424+0000 7f6a1d3e2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a180d0560 con 0x7f6a180d0100 2023-11-28T20:35:22.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.424+0000 7f6a1d3e2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a180d2230 con 0x7f6a180d1c80 2023-11-28T20:35:22.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.424+0000 7f6a1d3e2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180d2510 con 0x7f6a1815d550 2023-11-28T20:35:22.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.424+0000 7f6a177fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6a180d0100 0x7f6a180d1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58868/0 (socket says 172.21.15.45:58868) 2023-11-28T20:35:22.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.424+0000 7f6a177fe640 1 -- 172.21.15.45:0/2559464815 learned_addr learned my addr 172.21.15.45:0/2559464815 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:22.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.424+0000 7f6a17fff640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a1815d550 0x7f6a180cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.424+0000 7f6a1cbe1640 1 --2- 172.21.15.45:0/2559464815 >> v2:172.21.15.190:3300/0 conn(0x7f6a180d1c80 0x7f6a180b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715320935 0 0) 0x7f6a180d0560 con 0x7f6a180d0100 2023-11-28T20:35:22.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a04003520 con 0x7f6a180d0100 2023-11-28T20:35:22.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3751274207 0 0) 0x7f6a04003520 con 0x7f6a180d0100 2023-11-28T20:35:22.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a1815d550 msgr2=0x7f6a180cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a1815d550 0x7f6a180cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 >> v2:172.21.15.190:3300/0 conn(0x7f6a180d1c80 msgr2=0x7f6a180b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 --2- 172.21.15.45:0/2559464815 >> v2:172.21.15.190:3300/0 conn(0x7f6a180d1c80 0x7f6a180b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a180d27f0 con 0x7f6a180d0100 2023-11-28T20:35:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6a000019f0 con 0x7f6a180d0100 2023-11-28T20:35:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 >> v1:172.21.15.45:6789/0 conn(0x7f6a180d0100 legacy=0x7f6a180d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a04003970 0x7f6a04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040047c0 0x7f6a04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a1815d550 0x7f6a04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180d2230 con 0x7f6a040047c0 2023-11-28T20:35:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180d2510 con 0x7f6a04003970 2023-11-28T20:35:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.425+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0400b980 con 0x7f6a1815d550 2023-11-28T20:35:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a177fe640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040047c0 0x7f6a04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a17fff640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a04003970 0x7f6a04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a1cbe1640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a1815d550 0x7f6a04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a177fe640 1 -- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a1815d550 msgr2=0x7f6a04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:22.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a177fe640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a1815d550 0x7f6a04005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a177fe640 1 -- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a04003970 msgr2=0x7f6a04003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:22.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a177fe640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a04003970 0x7f6a04003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a177fe640 1 -- 172.21.15.45:0/2559464815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a04003520 con 0x7f6a040047c0 2023-11-28T20:35:22.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a1cbe1640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a1815d550 0x7f6a04005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:22.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.426+0000 7f6a17fff640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a04003970 0x7f6a04003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:22.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.427+0000 7f6a177fe640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040047c0 0x7f6a04004c40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000a790 tx=0x7f6a040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1403c352360a0388 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00034430 con 0x7f6a040047c0 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0002f070 con 0x7f6a040047c0 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a16ffd640 1 -- 172.21.15.45:0/2559464815 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a000347f0 con 0x7f6a040047c0 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/2559464815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040047c0 msgr2=0x7f6a04004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a1d3e2640 1 --2- 172.21.15.45:0/2559464815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040047c0 0x7f6a04004c40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000a790 tx=0x7f6a040057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a1cbe1640 1 -- 172.21.15.45:0/2559464815 reap_dead start 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/2559464815 shutdown_connections 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/2559464815 >> 172.21.15.45:0/2559464815 conn(0x7f6a1806ac70 msgr2=0x7f6a180b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/2559464815 shutdown_connections 2023-11-28T20:35:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.428+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/2559464815 wait complete. 2023-11-28T20:35:22.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.430+0000 7f6a1d3e2640 1 Processor -- start 2023-11-28T20:35:22.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.430+0000 7f6a1d3e2640 1 -- start start 2023-11-28T20:35:22.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.431+0000 7f6a1d3e2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a040047c0 0x7f6a181f66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.431+0000 7f6a1d3e2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a180cc200 0x7f6a180ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.431+0000 7f6a1d3e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a180ca7e0 0x7f6a181f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.431+0000 7f6a1d3e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180cac90 con 0x7f6a180ca7e0 2023-11-28T20:35:22.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.431+0000 7f6a1d3e2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a181fa3e0 con 0x7f6a040047c0 2023-11-28T20:35:22.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.431+0000 7f6a1d3e2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a181faf60 con 0x7f6a180cc200 2023-11-28T20:35:22.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a1cbe1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a180ca7e0 0x7f6a181f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a1cbe1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a180ca7e0 0x7f6a181f70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56426/0 (socket says 172.21.15.45:56426) 2023-11-28T20:35:22.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a1cbe1640 1 -- 172.21.15.45:0/1971626540 learned_addr learned my addr 172.21.15.45:0/1971626540 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:22.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a17fff640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a040047c0 0x7f6a181f66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a177fe640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a180cc200 0x7f6a180ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a1cbe1640 1 -- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a180cc200 msgr2=0x7f6a180ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a1cbe1640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a180cc200 0x7f6a180ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a1cbe1640 1 -- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a040047c0 msgr2=0x7f6a181f66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a1cbe1640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a040047c0 0x7f6a181f66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a1cbe1640 1 -- 172.21.15.45:0/1971626540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a181fa630 con 0x7f6a180ca7e0 2023-11-28T20:35:22.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.432+0000 7f6a177fe640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a180cc200 0x7f6a180ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:22.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.433+0000 7f6a1cbe1640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a180ca7e0 0x7f6a181f70b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6a181610f0 tx=0x7f6a10010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:22.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.433+0000 7f6a14ff9640 1 -- 172.21.15.45:0/1971626540 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1000e870 con 0x7f6a180ca7e0 2023-11-28T20:35:22.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.433+0000 7f6a14ff9640 1 -- 172.21.15.45:0/1971626540 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1000e9d0 con 0x7f6a180ca7e0 2023-11-28T20:35:22.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.433+0000 7f6a14ff9640 1 -- 172.21.15.45:0/1971626540 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1000ed60 con 0x7f6a180ca7e0 2023-11-28T20:35:22.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.433+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a181fa910 con 0x7f6a180ca7e0 2023-11-28T20:35:22.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.433+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a181f6c30 con 0x7f6a180ca7e0 2023-11-28T20:35:22.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.434+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a181fb0a0 con 0x7f6a180ca7e0 2023-11-28T20:35:22.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.435+0000 7f6a14ff9640 1 -- 172.21.15.45:0/1971626540 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6a100143f0 con 0x7f6a180ca7e0 2023-11-28T20:35:22.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.437+0000 7f6a14ff9640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f69f806ae40 0x7f69f806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.437+0000 7f6a14ff9640 1 -- 172.21.15.45:0/1971626540 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6a10089bc0 con 0x7f6a180ca7e0 2023-11-28T20:35:22.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.437+0000 7f6a17fff640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f69f806ae40 0x7f69f806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.438+0000 7f6a17fff640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f69f806ae40 0x7f69f806d300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a080099c0 tx=0x7f6a08008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:22.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.441+0000 7f6a14ff9640 1 -- 172.21.15.45:0/1971626540 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a100548a0 con 0x7f6a180ca7e0 2023-11-28T20:35:22.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.636+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a1804e920 con 0x7f6a180ca7e0 2023-11-28T20:35:22.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.638+0000 7f6a14ff9640 1 -- 172.21.15.45:0/1971626540 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18721 (secure 0 0 0) 0x7f6a10057ec0 con 0x7f6a180ca7e0 2023-11-28T20:35:22.640 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:22.640 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:34:41.117178+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:33:51.417060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T20:33:52.566772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T20:33:56.137628+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f69f806ae40 msgr2=0x7f69f806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f69f806ae40 0x7f69f806d300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a080099c0 tx=0x7f6a08008040 comp rx=0 tx=0).stop 2023-11-28T20:35:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a180ca7e0 msgr2=0x7f6a181f70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a180ca7e0 0x7f6a181f70b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6a181610f0 tx=0x7f6a10010c20 comp rx=0 tx=0).stop 2023-11-28T20:35:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 shutdown_connections 2023-11-28T20:35:22.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f69f806ae40 0x7f69f806d300 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a180ca7e0 0x7f6a181f70b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a180cc200 0x7f6a180ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 --2- 172.21.15.45:0/1971626540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a040047c0 0x7f6a181f66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 >> 172.21.15.45:0/1971626540 conn(0x7f6a1806ac70 msgr2=0x7f6a180cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 shutdown_connections 2023-11-28T20:35:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.641+0000 7f6a1d3e2640 1 -- 172.21.15.45:0/1971626540 wait complete. 2023-11-28T20:35:22.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T20:35:22.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.889+0000 7fd988056640 1 Processor -- start 2023-11-28T20:35:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.889+0000 7fd988056640 1 -- start start 2023-11-28T20:35:22.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.891+0000 7fd988056640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd9800ce390 0x7fd980070280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.891+0000 7fd988056640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9800b3110 con 0x7fd98015d590 2023-11-28T20:35:22.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.891+0000 7fd988056640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98006d9e0 con 0x7fd9800cf020 2023-11-28T20:35:22.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.891+0000 7fd988056640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd98006dcc0 con 0x7fd9800ce390 2023-11-28T20:35:22.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.891+0000 7fd9855f1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd9800ce390 0x7fd980070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.891+0000 7fd9855f1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd9800ce390 0x7fd980070280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56434/0 (socket says 172.21.15.45:56434) 2023-11-28T20:35:22.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.891+0000 7fd9855f1640 1 -- 172.21.15.45:0/864210347 learned_addr learned my addr 172.21.15.45:0/864210347 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:22.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.891+0000 7fd985df2640 1 --1- 172.21.15.45:0/864210347 >> v1:172.21.15.45:6789/0 conn(0x7fd98015d590 0x7fd98006d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58872/0 (socket says 172.21.15.45:58872) 2023-11-28T20:35:22.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.892+0000 7fd9855f1640 1 -- 172.21.15.45:0/864210347 >> v1:172.21.15.190:6789/0 conn(0x7fd9800cf020 legacy=0x7fd9800b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:22.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.892+0000 7fd9855f1640 1 -- 172.21.15.45:0/864210347 >> v1:172.21.15.45:6789/0 conn(0x7fd98015d590 legacy=0x7fd98006d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.892+0000 7fd984df0640 1 -- 172.21.15.45:0/864210347 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094204252 0 0) 0x7fd9800b3110 con 0x7fd98015d590 2023-11-28T20:35:22.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.892+0000 7fd9855f1640 1 -- 172.21.15.45:0/864210347 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98006dfa0 con 0x7fd9800ce390 2023-11-28T20:35:22.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.893+0000 7fd9855f1640 1 --2- 172.21.15.45:0/864210347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9800ce390 0x7fd980070280 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd970009870 tx=0x7fd97002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8311020d96f9d263 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.893+0000 7fd984df0640 1 -- 172.21.15.45:0/864210347 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970031870 con 0x7fd9800ce390 2023-11-28T20:35:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.893+0000 7fd984df0640 1 -- 172.21.15.45:0/864210347 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9700319d0 con 0x7fd9800ce390 2023-11-28T20:35:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.893+0000 7fd984df0640 1 -- 172.21.15.45:0/864210347 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970031d60 con 0x7fd9800ce390 2023-11-28T20:35:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.894+0000 7fd988056640 1 -- 172.21.15.45:0/864210347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9800ce390 msgr2=0x7fd980070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.894+0000 7fd988056640 1 --2- 172.21.15.45:0/864210347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9800ce390 0x7fd980070280 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd970009870 tx=0x7fd97002b910 comp rx=0 tx=0).stop 2023-11-28T20:35:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.894+0000 7fd988056640 1 -- 172.21.15.45:0/864210347 shutdown_connections 2023-11-28T20:35:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.894+0000 7fd988056640 1 --2- 172.21.15.45:0/864210347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9800ce390 0x7fd980070280 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.894+0000 7fd988056640 1 -- 172.21.15.45:0/864210347 >> 172.21.15.45:0/864210347 conn(0x7fd98006a6e0 msgr2=0x7fd98006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.894+0000 7fd988056640 1 -- 172.21.15.45:0/864210347 shutdown_connections 2023-11-28T20:35:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.894+0000 7fd988056640 1 -- 172.21.15.45:0/864210347 wait complete. 2023-11-28T20:35:22.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.896+0000 7fd988056640 1 Processor -- start 2023-11-28T20:35:22.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.896+0000 7fd988056640 1 -- start start 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.897+0000 7fd988056640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9800ce390 0x7fd9801e7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.897+0000 7fd988056640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9800cf020 0x7fd980152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.897+0000 7fd988056640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98015d590 0x7fd9800acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.897+0000 7fd988056640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801582e0 con 0x7fd98015d590 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.897+0000 7fd988056640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801e7dc0 con 0x7fd9800ce390 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.897+0000 7fd988056640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801e7ff0 con 0x7fd9800cf020 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.897+0000 7fd9865f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98015d590 0x7fd9800acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.897+0000 7fd9865f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98015d590 0x7fd9800acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56444/0 (socket says 172.21.15.45:56444) 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.897+0000 7fd9865f3640 1 -- 172.21.15.45:0/4056099622 learned_addr learned my addr 172.21.15.45:0/4056099622 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd9865f3640 1 -- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9800cf020 msgr2=0x7fd980152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd9865f3640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9800cf020 0x7fd980152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd9865f3640 1 -- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9800ce390 msgr2=0x7fd9801e7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:22.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd9865f3640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9800ce390 0x7fd9801e7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:22.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd9865f3640 1 -- 172.21.15.45:0/4056099622 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9700094d0 con 0x7fd98015d590 2023-11-28T20:35:22.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd9865f3640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98015d590 0x7fd9800acdc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd97800b970 tx=0x7fd97800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:22.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd9767fc640 1 -- 172.21.15.45:0/4056099622 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97800cb40 con 0x7fd98015d590 2023-11-28T20:35:22.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd9767fc640 1 -- 172.21.15.45:0/4056099622 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97800cca0 con 0x7fd98015d590 2023-11-28T20:35:22.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd9767fc640 1 -- 172.21.15.45:0/4056099622 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978014680 con 0x7fd98015d590 2023-11-28T20:35:22.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9801e82d0 con 0x7fd98015d590 2023-11-28T20:35:22.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.898+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9801e8890 con 0x7fd98015d590 2023-11-28T20:35:22.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.900+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd980073440 con 0x7fd98015d590 2023-11-28T20:35:22.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.901+0000 7fd9767fc640 1 -- 172.21.15.45:0/4056099622 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd97801d3b0 con 0x7fd98015d590 2023-11-28T20:35:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.902+0000 7fd9767fc640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd96406ae40 0x7fd96406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.902+0000 7fd9767fc640 1 -- 172.21.15.45:0/4056099622 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd978089810 con 0x7fd98015d590 2023-11-28T20:35:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.903+0000 7fd985df2640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd96406ae40 0x7fd96406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:22.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.904+0000 7fd985df2640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd96406ae40 0x7fd96406d300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd97c007fd0 tx=0x7fd97c007920 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:22.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:22.907+0000 7fd9767fc640 1 -- 172.21.15.45:0/4056099622 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9780544f0 con 0x7fd98015d590 2023-11-28T20:35:23.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.144+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd9800ae050 con 0x7fd98015d590 2023-11-28T20:35:23.146 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T20:35:23.145+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T20:35:23.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.148+0000 7fd9767fc640 1 -- 172.21.15.45:0/4056099622 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd978057b10 con 0x7fd98015d590 2023-11-28T20:35:23.150 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T20:35:23.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.152+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd96406ae40 msgr2=0x7fd96406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:23.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.152+0000 7fd988056640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd96406ae40 0x7fd96406d300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd97c007fd0 tx=0x7fd97c007920 comp rx=0 tx=0).stop 2023-11-28T20:35:23.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.152+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98015d590 msgr2=0x7fd9800acdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:23.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.152+0000 7fd988056640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98015d590 0x7fd9800acdc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd97800b970 tx=0x7fd97800be30 comp rx=0 tx=0).stop 2023-11-28T20:35:23.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.153+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 shutdown_connections 2023-11-28T20:35:23.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.153+0000 7fd988056640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd96406ae40 0x7fd96406d300 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.153+0000 7fd988056640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd98015d590 0x7fd9800acdc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.153+0000 7fd988056640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9800cf020 0x7fd980152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.153+0000 7fd988056640 1 --2- 172.21.15.45:0/4056099622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9800ce390 0x7fd9801e7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.153+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 >> 172.21.15.45:0/4056099622 conn(0x7fd98006a6e0 msgr2=0x7fd980070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:23.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.153+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 shutdown_connections 2023-11-28T20:35:23.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.154+0000 7fd988056640 1 -- 172.21.15.45:0/4056099622 wait complete. 2023-11-28T20:35:23.170 INFO:tasks.ceph.mds.c.smithi045.stderr: -5863> 2023-11-28T20:33:57.347+0000 7f9540fa8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:35:23.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T20:35:23.233 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:35:23.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.388+0000 7f37678e0640 1 Processor -- start 2023-11-28T20:35:23.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.389+0000 7f37678e0640 1 -- start start 2023-11-28T20:35:23.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.390+0000 7f37678e0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f37600ce600 0x7f376006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:23.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.390+0000 7f37678e0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376006e030 con 0x7f376006dc50 2023-11-28T20:35:23.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.390+0000 7f37678e0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37600ccf60 con 0x7f376006aa40 2023-11-28T20:35:23.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.390+0000 7f37678e0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376006aed0 con 0x7f37600ce600 2023-11-28T20:35:23.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.391+0000 7f376567c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f376006dc50 0x7f37600cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58884/0 (socket says 172.21.15.45:58884) 2023-11-28T20:35:23.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.391+0000 7f376567c640 1 -- 172.21.15.45:0/1316516835 learned_addr learned my addr 172.21.15.45:0/1316516835 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.391+0000 7f3764e7b640 1 --2- 172.21.15.45:0/1316516835 >> v2:172.21.15.190:3300/0 conn(0x7f37600ce600 0x7f376006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.391+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102940516 0 0) 0x7f375c001490 con 0x7f376006dc50 2023-11-28T20:35:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.391+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f374c003520 con 0x7f376006dc50 2023-11-28T20:35:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 852502707 0 0) 0x7f374c003520 con 0x7f376006dc50 2023-11-28T20:35:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 >> v2:172.21.15.190:3300/0 conn(0x7f37600ce600 msgr2=0x7f376006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 --2- 172.21.15.45:0/1316516835 >> v2:172.21.15.190:3300/0 conn(0x7f37600ce600 0x7f376006a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 >> v1:172.21.15.190:6789/0 conn(0x7f376006aa40 legacy=0x7f37600c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37600cc930 con 0x7f376006dc50 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f375c001b80 con 0x7f376006dc50 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 >> v1:172.21.15.45:6789/0 conn(0x7f376006dc50 legacy=0x7f37600cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f374c0039c0 0x7f374c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f374c0047c0 0x7f374c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c005180 0x7f374c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376006aed0 con 0x7f374c0047c0 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c00c080 con 0x7f374c005180 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374c00c240 con 0x7f374c0039c0 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3764e7b640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f374c0047c0 0x7f374c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3765e7d640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c005180 0x7f374c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:23.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f376567c640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f374c0039c0 0x7f374c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:23.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.392+0000 7f3764e7b640 1 -- 172.21.15.45:0/1316516835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f374c0039c0 msgr2=0x7f374c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:23.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.393+0000 7f3764e7b640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f374c0039c0 0x7f374c003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.393+0000 7f3764e7b640 1 -- 172.21.15.45:0/1316516835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c005180 msgr2=0x7f374c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:23.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.393+0000 7f3764e7b640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c005180 0x7f374c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.393+0000 7f3764e7b640 1 -- 172.21.15.45:0/1316516835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c003520 con 0x7f374c0047c0 2023-11-28T20:35:23.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.393+0000 7f3765e7d640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c005180 0x7f374c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:23.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.393+0000 7f3765e7d640 1 -- 172.21.15.45:0/1316516835 reap_dead start 2023-11-28T20:35:23.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.394+0000 7f3764e7b640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f374c0047c0 0x7f374c004c40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3750009ac0 tx=0x7f374c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a1fc2a2edd5f20 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.394+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375002c6b0 con 0x7f374c0047c0 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.394+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375002c810 con 0x7f374c0047c0 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.394+0000 7f3757fff640 1 -- 172.21.15.45:0/1316516835 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375002cba0 con 0x7f374c0047c0 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.394+0000 7f37678e0640 1 -- 172.21.15.45:0/1316516835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f374c0047c0 msgr2=0x7f374c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.394+0000 7f37678e0640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f374c0047c0 0x7f374c004c40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3750009ac0 tx=0x7f374c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.395+0000 7f37678e0640 1 -- 172.21.15.45:0/1316516835 shutdown_connections 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.395+0000 7f37678e0640 1 --2- 172.21.15.45:0/1316516835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f374c0047c0 0x7f374c004c40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.395+0000 7f37678e0640 1 -- 172.21.15.45:0/1316516835 >> 172.21.15.45:0/1316516835 conn(0x7f37600ced60 msgr2=0x7f37600cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.395+0000 7f37678e0640 1 -- 172.21.15.45:0/1316516835 shutdown_connections 2023-11-28T20:35:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.395+0000 7f37678e0640 1 -- 172.21.15.45:0/1316516835 wait complete. 2023-11-28T20:35:23.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.397+0000 7f37678e0640 1 Processor -- start 2023-11-28T20:35:23.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.397+0000 7f37678e0640 1 -- start start 2023-11-28T20:35:23.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f37678e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c0047c0 0x7f376015c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:23.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f37678e0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37600b4110 0x7f37600b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f37678e0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3760160da0 0x7f3760161220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f37678e0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3760161760 con 0x7f37600b4110 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f37678e0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376009b5e0 con 0x7f374c0047c0 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f37678e0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376009a9d0 con 0x7f3760160da0 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f3764e7b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37600b4110 0x7f37600b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f3764e7b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37600b4110 0x7f37600b2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56496/0 (socket says 172.21.15.45:56496) 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f3764e7b640 1 -- 172.21.15.45:0/4190343106 learned_addr learned my addr 172.21.15.45:0/4190343106 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f376567c640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c0047c0 0x7f376015c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f3764e7b640 1 -- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3760160da0 msgr2=0x7f3760161220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f3764e7b640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3760160da0 0x7f3760161220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f3764e7b640 1 -- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c0047c0 msgr2=0x7f376015c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f3764e7b640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c0047c0 0x7f376015c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f3764e7b640 1 -- 172.21.15.45:0/4190343106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376009acb0 con 0x7f37600b4110 2023-11-28T20:35:23.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.398+0000 7f376567c640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c0047c0 0x7f376015c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:23.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.399+0000 7f3764e7b640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37600b4110 0x7f37600b2350 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3750037040 tx=0x7f3750004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:23.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.399+0000 7f3755ffb640 1 -- 172.21.15.45:0/4190343106 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750003810 con 0x7f37600b4110 2023-11-28T20:35:23.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.399+0000 7f3755ffb640 1 -- 172.21.15.45:0/4190343106 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3750003970 con 0x7f37600b4110 2023-11-28T20:35:23.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.399+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376009af90 con 0x7f37600b4110 2023-11-28T20:35:23.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.399+0000 7f3755ffb640 1 -- 172.21.15.45:0/4190343106 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750003d00 con 0x7f37600b4110 2023-11-28T20:35:23.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.399+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37600b2890 con 0x7f37600b4110 2023-11-28T20:35:23.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.400+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37600733d0 con 0x7f37600b4110 2023-11-28T20:35:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.401+0000 7f3755ffb640 1 -- 172.21.15.45:0/4190343106 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3750040450 con 0x7f37600b4110 2023-11-28T20:35:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.402+0000 7f3755ffb640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f374406ae70 0x7f374406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.403+0000 7f3755ffb640 1 -- 172.21.15.45:0/4190343106 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f37500ab9d0 con 0x7f37600b4110 2023-11-28T20:35:23.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.403+0000 7f376567c640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f374406ae70 0x7f374406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:23.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.405+0000 7f376567c640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f374406ae70 0x7f374406d330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f375c001cb0 tx=0x7f375c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.407+0000 7f3755ffb640 1 -- 172.21.15.45:0/4190343106 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3750076660 con 0x7f37600b4110 2023-11-28T20:35:23.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:23.642+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f37600cd920 con 0x7f37600b4110 2023-11-28T20:35:24.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.163+0000 7f3755ffb640 1 -- 172.21.15.45:0/4190343106 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3750003e60 con 0x7f37600b4110 2023-11-28T20:35:24.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f374406ae70 msgr2=0x7f374406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f374406ae70 0x7f374406d330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f375c001cb0 tx=0x7f375c009040 comp rx=0 tx=0).stop 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37600b4110 msgr2=0x7f37600b2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37600b4110 0x7f37600b2350 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3750037040 tx=0x7f3750004730 comp rx=0 tx=0).stop 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 shutdown_connections 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f374406ae70 0x7f374406d330 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3760160da0 0x7f3760161220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37600b4110 0x7f37600b2350 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 --2- 172.21.15.45:0/4190343106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c0047c0 0x7f376015c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 >> 172.21.15.45:0/4190343106 conn(0x7f37600ced60 msgr2=0x7f37600af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.167+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 shutdown_connections 2023-11-28T20:35:24.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.168+0000 7f37678e0640 1 -- 172.21.15.45:0/4190343106 wait complete. 2023-11-28T20:35:24.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T20:35:24.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.420+0000 7f871eead640 1 Processor -- start 2023-11-28T20:35:24.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.420+0000 7f871eead640 1 -- start start 2023-11-28T20:35:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871eead640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f871815d310 0x7f87180b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871eead640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8718160e70 0x7f87180cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871eead640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87180ce3a0 0x7f87180b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871eead640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87180ce810 con 0x7f871815d310 2023-11-28T20:35:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871eead640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87180d14d0 con 0x7f8718160e70 2023-11-28T20:35:24.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871eead640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87181613c0 con 0x7f87180ce3a0 2023-11-28T20:35:24.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871deab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f871815d310 0x7f87180b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:24.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871deab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f871815d310 0x7f87180b8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57478/0 (socket says 172.21.15.45:57478) 2023-11-28T20:35:24.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871deab640 1 -- 172.21.15.45:0/3079125280 learned_addr learned my addr 172.21.15.45:0/3079125280 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:24.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871d6aa640 1 --2- 172.21.15.45:0/3079125280 >> v2:172.21.15.190:3300/0 conn(0x7f8718160e70 0x7f87180cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:24.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.422+0000 7f871e6ac640 1 --2- 172.21.15.45:0/3079125280 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87180ce3a0 0x7f87180b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:24.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.423+0000 7f871deab640 1 -- 172.21.15.45:0/3079125280 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87180ce3a0 msgr2=0x7f87180b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:24.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.423+0000 7f871deab640 1 --2- 172.21.15.45:0/3079125280 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87180ce3a0 0x7f87180b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.423+0000 7f871deab640 1 -- 172.21.15.45:0/3079125280 >> v2:172.21.15.190:3300/0 conn(0x7f8718160e70 msgr2=0x7f87180cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:24.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.423+0000 7f871deab640 1 --2- 172.21.15.45:0/3079125280 >> v2:172.21.15.190:3300/0 conn(0x7f8718160e70 0x7f87180cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.423+0000 7f871deab640 1 -- 172.21.15.45:0/3079125280 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87181616a0 con 0x7f871815d310 2023-11-28T20:35:24.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.423+0000 7f871d6aa640 1 --2- 172.21.15.45:0/3079125280 >> v2:172.21.15.190:3300/0 conn(0x7f8718160e70 0x7f87180cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:24.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.424+0000 7f871deab640 1 --2- 172.21.15.45:0/3079125280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f871815d310 0x7f87180b8a20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f87180d4130 tx=0x7f871402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac42ff67141246a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:24.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.424+0000 7f871cea9640 1 -- 172.21.15.45:0/3079125280 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87140317d0 con 0x7f871815d310 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.424+0000 7f871cea9640 1 -- 172.21.15.45:0/3079125280 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8714031930 con 0x7f871815d310 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.424+0000 7f871cea9640 1 -- 172.21.15.45:0/3079125280 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8714031cc0 con 0x7f871815d310 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.425+0000 7f871eead640 1 -- 172.21.15.45:0/3079125280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f871815d310 msgr2=0x7f87180b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.425+0000 7f871eead640 1 --2- 172.21.15.45:0/3079125280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f871815d310 0x7f87180b8a20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f87180d4130 tx=0x7f871402b870 comp rx=0 tx=0).stop 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.425+0000 7f871eead640 1 -- 172.21.15.45:0/3079125280 shutdown_connections 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.425+0000 7f871eead640 1 --2- 172.21.15.45:0/3079125280 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87180ce3a0 0x7f87180b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.425+0000 7f871eead640 1 --2- 172.21.15.45:0/3079125280 >> v2:172.21.15.190:3300/0 conn(0x7f8718160e70 0x7f87180cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.425+0000 7f871eead640 1 --2- 172.21.15.45:0/3079125280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f871815d310 0x7f87180b8a20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.425+0000 7f871eead640 1 -- 172.21.15.45:0/3079125280 >> 172.21.15.45:0/3079125280 conn(0x7f871806a500 msgr2=0x7f871806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.425+0000 7f871eead640 1 -- 172.21.15.45:0/3079125280 shutdown_connections 2023-11-28T20:35:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.425+0000 7f871eead640 1 -- 172.21.15.45:0/3079125280 wait complete. 2023-11-28T20:35:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.427+0000 7f871eead640 1 Processor -- start 2023-11-28T20:35:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.427+0000 7f871eead640 1 -- start start 2023-11-28T20:35:24.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.428+0000 7f871eead640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87180ce3a0 0x7f8718160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:24.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.428+0000 7f871eead640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f871815d310 0x7f871815a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:24.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.428+0000 7f871eead640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8718160e70 0x7f871815a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:24.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.428+0000 7f871eead640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87180b2e20 con 0x7f87180ce3a0 2023-11-28T20:35:24.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.428+0000 7f871eead640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871815ae20 con 0x7f8718160e70 2023-11-28T20:35:24.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.428+0000 7f871eead640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87180b2ad0 con 0x7f871815d310 2023-11-28T20:35:24.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.428+0000 7f871deab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87180ce3a0 0x7f8718160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:24.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.428+0000 7f871deab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87180ce3a0 0x7f8718160110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57488/0 (socket says 172.21.15.45:57488) 2023-11-28T20:35:24.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.428+0000 7f871deab640 1 -- 172.21.15.45:0/1175309705 learned_addr learned my addr 172.21.15.45:0/1175309705 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:24.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.429+0000 7f871e6ac640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8718160e70 0x7f871815a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:24.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.429+0000 7f871deab640 1 -- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f871815d310 msgr2=0x7f871815a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:24.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.429+0000 7f871deab640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f871815d310 0x7f871815a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.429+0000 7f871deab640 1 -- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8718160e70 msgr2=0x7f871815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:24.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.429+0000 7f871deab640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8718160e70 0x7f871815a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:24.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.429+0000 7f871deab640 1 -- 172.21.15.45:0/1175309705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87140094d0 con 0x7f87180ce3a0 2023-11-28T20:35:24.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.429+0000 7f871deab640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87180ce3a0 0x7f8718160110 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8714000c00 tx=0x7f87140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:24.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.430+0000 7f870e7fc640 1 -- 172.21.15.45:0/1175309705 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87140328f0 con 0x7f87180ce3a0 2023-11-28T20:35:24.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.430+0000 7f870e7fc640 1 -- 172.21.15.45:0/1175309705 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8714032a70 con 0x7f87180ce3a0 2023-11-28T20:35:24.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.430+0000 7f870e7fc640 1 -- 172.21.15.45:0/1175309705 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8714032e20 con 0x7f87180ce3a0 2023-11-28T20:35:24.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.430+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871815ea20 con 0x7f87180ce3a0 2023-11-28T20:35:24.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.430+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871815efe0 con 0x7f87180ce3a0 2023-11-28T20:35:24.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.431+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8718071b30 con 0x7f87180ce3a0 2023-11-28T20:35:24.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.432+0000 7f870e7fc640 1 -- 172.21.15.45:0/1175309705 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8714030070 con 0x7f87180ce3a0 2023-11-28T20:35:24.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.433+0000 7f870e7fc640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f86fc06ae70 0x7f86fc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:24.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.433+0000 7f870e7fc640 1 -- 172.21.15.45:0/1175309705 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f87140abbb0 con 0x7f87180ce3a0 2023-11-28T20:35:24.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.433+0000 7f871d6aa640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f86fc06ae70 0x7f86fc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:24.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.434+0000 7f871d6aa640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f86fc06ae70 0x7f86fc06d330 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8708006fd0 tx=0x7f8708008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:24.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.438+0000 7f870e7fc640 1 -- 172.21.15.45:0/1175309705 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87140767b0 con 0x7f87180ce3a0 2023-11-28T20:35:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:24.634+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f871804e920 con 0x7f87180ce3a0 2023-11-28T20:35:25.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.183+0000 7f870e7fc640 1 -- 172.21.15.45:0/1175309705 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8714079dd0 con 0x7f87180ce3a0 2023-11-28T20:35:25.184 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T20:35:25.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f86fc06ae70 msgr2=0x7f86fc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:25.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f86fc06ae70 0x7f86fc06d330 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8708006fd0 tx=0x7f8708008040 comp rx=0 tx=0).stop 2023-11-28T20:35:25.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87180ce3a0 msgr2=0x7f8718160110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:25.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87180ce3a0 0x7f8718160110 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8714000c00 tx=0x7f87140046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 shutdown_connections 2023-11-28T20:35:25.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f86fc06ae70 0x7f86fc06d330 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8708006fd0 tx=0x7f8708008040 comp rx=0 tx=0).stop 2023-11-28T20:35:25.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8718160e70 0x7f871815a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f871815d310 0x7f871815a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 --2- 172.21.15.45:0/1175309705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87180ce3a0 0x7f8718160110 secure :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8714000c00 tx=0x7f87140046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.187+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 >> 172.21.15.45:0/1175309705 conn(0x7f871806a500 msgr2=0x7f87180c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:25.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.189+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 shutdown_connections 2023-11-28T20:35:25.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.189+0000 7f871eead640 1 -- 172.21.15.45:0/1175309705 wait complete. 2023-11-28T20:35:25.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T20:35:25.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.404+0000 7f312512c640 1 Processor -- start 2023-11-28T20:35:25.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.404+0000 7f312512c640 1 -- start start 2023-11-28T20:35:25.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.405+0000 7f312512c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31200cde00 0x7f31200ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:25.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.405+0000 7f312512c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312015d790 con 0x7f31200b6380 2023-11-28T20:35:25.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.406+0000 7f312512c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31200ce7c0 con 0x7f312015d3b0 2023-11-28T20:35:25.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.406+0000 7f312512c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31200d2bf0 con 0x7f31200cde00 2023-11-28T20:35:25.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.406+0000 7f312492b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f31200b6380 0x7f31200b6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39236/0 (socket says 172.21.15.45:39236) 2023-11-28T20:35:25.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.406+0000 7f312492b640 1 -- 172.21.15.45:0/2323681974 learned_addr learned my addr 172.21.15.45:0/2323681974 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:25.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.406+0000 7f311f7fe640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31200cde00 0x7f31200ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:25.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.406+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181990119 0 0) 0x7f312015d790 con 0x7f31200b6380 2023-11-28T20:35:25.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.406+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3108003520 con 0x7f31200b6380 2023-11-28T20:35:25.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3937960717 0 0) 0x7f3108003520 con 0x7f31200b6380 2023-11-28T20:35:25.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31200cde00 msgr2=0x7f31200ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:25.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31200cde00 0x7f31200ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 >> v1:172.21.15.190:6789/0 conn(0x7f312015d3b0 legacy=0x7f31200d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:25.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31200d2ed0 con 0x7f31200b6380 2023-11-28T20:35:25.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f31180019f0 con 0x7f31200b6380 2023-11-28T20:35:25.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 >> v1:172.21.15.45:6789/0 conn(0x7f31200b6380 legacy=0x7f31200b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:25.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31200cde00 0x7f31080039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:25.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:25.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3108004d00 0x7f3108005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:25.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31200d2bf0 con 0x7f3108003f70 2023-11-28T20:35:25.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310800bac0 con 0x7f3108004d00 2023-11-28T20:35:25.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310800bc80 con 0x7f31200cde00 2023-11-28T20:35:25.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.407+0000 7f311f7fe640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:25.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.408+0000 7f312492b640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3108004d00 0x7f3108005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:25.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.408+0000 7f311ffff640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31200cde00 0x7f31080039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:25.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.408+0000 7f311f7fe640 1 -- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31200cde00 msgr2=0x7f31080039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:25.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.408+0000 7f311f7fe640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31200cde00 0x7f31080039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.408+0000 7f311f7fe640 1 -- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3108004d00 msgr2=0x7f3108005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:25.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.408+0000 7f311f7fe640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3108004d00 0x7f3108005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.408+0000 7f311f7fe640 1 -- 172.21.15.45:0/2323681974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3108003520 con 0x7f3108003f70 2023-11-28T20:35:25.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.408+0000 7f312492b640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3108004d00 0x7f3108005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:25.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.408+0000 7f311ffff640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31200cde00 0x7f31080039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:25.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.409+0000 7f311f7fe640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31080047c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3114009ac0 tx=0x7f31080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c972ca97f900140 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:25.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.409+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311402c600 con 0x7f3108003f70 2023-11-28T20:35:25.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.409+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311402c760 con 0x7f3108003f70 2023-11-28T20:35:25.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.409+0000 7f311effd640 1 -- 172.21.15.45:0/2323681974 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311402caf0 con 0x7f3108003f70 2023-11-28T20:35:25.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.410+0000 7f312512c640 1 -- 172.21.15.45:0/2323681974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 msgr2=0x7f31080047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:25.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.410+0000 7f312512c640 1 --2- 172.21.15.45:0/2323681974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31080047c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3114009ac0 tx=0x7f31080057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.410+0000 7f312492b640 1 -- 172.21.15.45:0/2323681974 reap_dead start 2023-11-28T20:35:25.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.410+0000 7f312512c640 1 -- 172.21.15.45:0/2323681974 shutdown_connections 2023-11-28T20:35:25.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.410+0000 7f312512c640 1 -- 172.21.15.45:0/2323681974 >> 172.21.15.45:0/2323681974 conn(0x7f3120070f40 msgr2=0x7f3120071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:25.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.410+0000 7f312512c640 1 -- 172.21.15.45:0/2323681974 shutdown_connections 2023-11-28T20:35:25.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.410+0000 7f312512c640 1 -- 172.21.15.45:0/2323681974 wait complete. 2023-11-28T20:35:25.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.412+0000 7f312512c640 1 Processor -- start 2023-11-28T20:35:25.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.412+0000 7f312512c640 1 -- start start 2023-11-28T20:35:25.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.413+0000 7f312512c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31200c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:25.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.413+0000 7f312512c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31201495c0 0x7f31200c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:25.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.413+0000 7f312512c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31200c7e60 0x7f31200cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:25.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.413+0000 7f312512c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31200c8300 con 0x7f3108003f70 2023-11-28T20:35:25.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.413+0000 7f312512c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31200c85e0 con 0x7f31200c7e60 2023-11-28T20:35:25.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.413+0000 7f312512c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31200c88c0 con 0x7f31201495c0 2023-11-28T20:35:25.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.413+0000 7f311ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31200c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:25.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f311ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31200c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57492/0 (socket says 172.21.15.45:57492) 2023-11-28T20:35:25.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f311ffff640 1 -- 172.21.15.45:0/2286181698 learned_addr learned my addr 172.21.15.45:0/2286181698 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:25.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f311ffff640 1 -- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31201495c0 msgr2=0x7f31200c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:25.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f311ffff640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31201495c0 0x7f31200c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f312492b640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31200c7e60 0x7f31200cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f311ffff640 1 -- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31200c7e60 msgr2=0x7f31200cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f311ffff640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31200c7e60 0x7f31200cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f311ffff640 1 -- 172.21.15.45:0/2286181698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31200c8ba0 con 0x7f3108003f70 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f311ffff640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31200c93a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f311000d7a0 tx=0x7f311000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.414+0000 7f311cff9640 1 -- 172.21.15.45:0/2286181698 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110017070 con 0x7f3108003f70 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.415+0000 7f311cff9640 1 -- 172.21.15.45:0/2286181698 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110005070 con 0x7f3108003f70 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.415+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31200c8e80 con 0x7f3108003f70 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.415+0000 7f311cff9640 1 -- 172.21.15.45:0/2286181698 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110005420 con 0x7f3108003f70 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.415+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3120155b00 con 0x7f3108003f70 2023-11-28T20:35:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.416+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31200c8720 con 0x7f3108003f70 2023-11-28T20:35:25.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.417+0000 7f311cff9640 1 -- 172.21.15.45:0/2286181698 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3110007030 con 0x7f3108003f70 2023-11-28T20:35:25.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.419+0000 7f311cff9640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f310c06ae40 0x7f310c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:25.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.420+0000 7f311cff9640 1 -- 172.21.15.45:0/2286181698 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f31100891e0 con 0x7f3108003f70 2023-11-28T20:35:25.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.420+0000 7f311f7fe640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f310c06ae40 0x7f310c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:25.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.421+0000 7f311f7fe640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f310c06ae40 0x7f310c06d300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3114036000 tx=0x7f311402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.425+0000 7f311cff9640 1 -- 172.21.15.45:0/2286181698 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31200c8720 con 0x7f3108003f70 2023-11-28T20:35:25.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:25.617+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f31200c8720 con 0x7f3108003f70 2023-11-28T20:35:26.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.195+0000 7f311cff9640 1 -- 172.21.15.45:0/2286181698 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f31200c8720 con 0x7f3108003f70 2023-11-28T20:35:26.197 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T20:35:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f310c06ae40 msgr2=0x7f310c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f310c06ae40 0x7f310c06d300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3114036000 tx=0x7f311402e000 comp rx=0 tx=0).stop 2023-11-28T20:35:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 msgr2=0x7f31200c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31200c93a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f311000d7a0 tx=0x7f311000dc60 comp rx=0 tx=0).stop 2023-11-28T20:35:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 shutdown_connections 2023-11-28T20:35:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f310c06ae40 0x7f310c06d300 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31200c7e60 0x7f31200cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31201495c0 0x7f31200c7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 --2- 172.21.15.45:0/2286181698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3108003f70 0x7f31200c93a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.199+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 >> 172.21.15.45:0/2286181698 conn(0x7f3120070f40 msgr2=0x7f31200b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.200+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 shutdown_connections 2023-11-28T20:35:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.200+0000 7f312512c640 1 -- 172.21.15.45:0/2286181698 wait complete. 2023-11-28T20:35:26.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-11-28T20:35:26.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.435+0000 7f92d64d7640 1 Processor -- start 2023-11-28T20:35:26.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.435+0000 7f92d64d7640 1 -- start start 2023-11-28T20:35:26.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.437+0000 7f92d64d7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f92d00cbe10 0x7f92d0071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:26.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.437+0000 7f92d64d7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f92d0071a40 0x7f92d006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:26.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.437+0000 7f92d64d7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d0070e20 con 0x7f92d01448e0 2023-11-28T20:35:26.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.437+0000 7f92d64d7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d0144cc0 con 0x7f92d00cbe10 2023-11-28T20:35:26.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.437+0000 7f92d64d7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d0070280 con 0x7f92d0071a40 2023-11-28T20:35:26.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.437+0000 7f92d4cd4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f92d00cbe10 0x7f92d0071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:26.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.438+0000 7f92d4cd4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f92d00cbe10 0x7f92d0071500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57506/0 (socket says 172.21.15.45:57506) 2023-11-28T20:35:26.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.438+0000 7f92d4cd4640 1 -- 172.21.15.45:0/102574276 learned_addr learned my addr 172.21.15.45:0/102574276 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:26.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.438+0000 7f92d5cd6640 1 --2- 172.21.15.45:0/102574276 >> v2:172.21.15.190:3300/0 conn(0x7f92d0071a40 0x7f92d006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:26.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.438+0000 7f92d4cd4640 1 -- 172.21.15.45:0/102574276 >> v2:172.21.15.190:3300/0 conn(0x7f92d0071a40 msgr2=0x7f92d006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:26.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.438+0000 7f92d4cd4640 1 --2- 172.21.15.45:0/102574276 >> v2:172.21.15.190:3300/0 conn(0x7f92d0071a40 0x7f92d006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:26.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.438+0000 7f92d4cd4640 1 -- 172.21.15.45:0/102574276 >> v1:172.21.15.190:6789/0 conn(0x7f92d01448e0 legacy=0x7f92d00cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:26.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.438+0000 7f92d4cd4640 1 -- 172.21.15.45:0/102574276 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00704f0 con 0x7f92d00cbe10 2023-11-28T20:35:26.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.439+0000 7f92d4cd4640 1 --2- 172.21.15.45:0/102574276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d00cbe10 0x7f92d0071500 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f92d006b610 tx=0x7f92c002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0370271f62a816e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:26.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.440+0000 7f92c7fff640 1 -- 172.21.15.45:0/102574276 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c00317e0 con 0x7f92d00cbe10 2023-11-28T20:35:26.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.440+0000 7f92c7fff640 1 -- 172.21.15.45:0/102574276 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c0031940 con 0x7f92d00cbe10 2023-11-28T20:35:26.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.440+0000 7f92c7fff640 1 -- 172.21.15.45:0/102574276 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c0031cd0 con 0x7f92d00cbe10 2023-11-28T20:35:26.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.440+0000 7f92d64d7640 1 -- 172.21.15.45:0/102574276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d00cbe10 msgr2=0x7f92d0071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:26.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.440+0000 7f92d64d7640 1 --2- 172.21.15.45:0/102574276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d00cbe10 0x7f92d0071500 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f92d006b610 tx=0x7f92c002b880 comp rx=0 tx=0).stop 2023-11-28T20:35:26.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.440+0000 7f92d64d7640 1 -- 172.21.15.45:0/102574276 shutdown_connections 2023-11-28T20:35:26.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.440+0000 7f92d64d7640 1 --2- 172.21.15.45:0/102574276 >> v2:172.21.15.190:3300/0 conn(0x7f92d0071a40 0x7f92d006cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:26.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.440+0000 7f92d64d7640 1 --2- 172.21.15.45:0/102574276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d00cbe10 0x7f92d0071500 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:26.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.440+0000 7f92d64d7640 1 -- 172.21.15.45:0/102574276 >> 172.21.15.45:0/102574276 conn(0x7f92d00cec70 msgr2=0x7f92d00cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:26.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.441+0000 7f92d64d7640 1 -- 172.21.15.45:0/102574276 shutdown_connections 2023-11-28T20:35:26.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.441+0000 7f92d64d7640 1 -- 172.21.15.45:0/102574276 wait complete. 2023-11-28T20:35:26.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.442+0000 7f92d64d7640 1 Processor -- start 2023-11-28T20:35:26.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.443+0000 7f92d64d7640 1 -- start start 2023-11-28T20:35:26.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d64d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92d0071a40 0x7f92d00ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d64d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92d00cbe10 0x7f92d00c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d64d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d01448e0 0x7f92d0159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d64d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d00ca820 con 0x7f92d01448e0 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d64d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d00ca540 con 0x7f92d00cbe10 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d64d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d0099ae0 con 0x7f92d0071a40 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d5cd6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d01448e0 0x7f92d0159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d5cd6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d01448e0 0x7f92d0159f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57518/0 (socket says 172.21.15.45:57518) 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d5cd6640 1 -- 172.21.15.45:0/2861028760 learned_addr learned my addr 172.21.15.45:0/2861028760 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d4cd4640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92d00cbe10 0x7f92d00c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d5cd6640 1 -- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92d0071a40 msgr2=0x7f92d00ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d54d5640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92d0071a40 0x7f92d00ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d5cd6640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92d0071a40 0x7f92d00ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d5cd6640 1 -- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92d00cbe10 msgr2=0x7f92d00c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.444+0000 7f92d5cd6640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92d00cbe10 0x7f92d00c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.445+0000 7f92d5cd6640 1 -- 172.21.15.45:0/2861028760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00094d0 con 0x7f92d01448e0 2023-11-28T20:35:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.445+0000 7f92d54d5640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92d0071a40 0x7f92d00ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:26.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.445+0000 7f92d5cd6640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d01448e0 0x7f92d0159f50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f92c8007e60 tx=0x7f92c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:26.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.445+0000 7f92c5ffb640 1 -- 172.21.15.45:0/2861028760 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8014420 con 0x7f92d01448e0 2023-11-28T20:35:26.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.445+0000 7f92c5ffb640 1 -- 172.21.15.45:0/2861028760 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c80145a0 con 0x7f92d01448e0 2023-11-28T20:35:26.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.445+0000 7f92c5ffb640 1 -- 172.21.15.45:0/2861028760 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8014950 con 0x7f92d01448e0 2023-11-28T20:35:26.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.445+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d0099dc0 con 0x7f92d01448e0 2023-11-28T20:35:26.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.445+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d00ca230 con 0x7f92d01448e0 2023-11-28T20:35:26.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.446+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d009a2e0 con 0x7f92d01448e0 2023-11-28T20:35:26.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.447+0000 7f92c5ffb640 1 -- 172.21.15.45:0/2861028760 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f92c8014ab0 con 0x7f92d01448e0 2023-11-28T20:35:26.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.448+0000 7f92c5ffb640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92b406ae70 0x7f92b406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:26.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.448+0000 7f92c5ffb640 1 -- 172.21.15.45:0/2861028760 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f92c8089fe0 con 0x7f92d01448e0 2023-11-28T20:35:26.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.449+0000 7f92d54d5640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92b406ae70 0x7f92b406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:26.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.450+0000 7f92d54d5640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92b406ae70 0x7f92b406d330 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f92cc003eb0 tx=0x7f92cc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:26.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.453+0000 7f92c5ffb640 1 -- 172.21.15.45:0/2861028760 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92d009a2e0 con 0x7f92d01448e0 2023-11-28T20:35:26.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:26.646+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f92d009a2e0 con 0x7f92d01448e0 2023-11-28T20:35:27.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.207+0000 7f92c5ffb640 1 -- 172.21.15.45:0/2861028760 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7f92d009a2e0 con 0x7f92d01448e0 2023-11-28T20:35:27.209 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data_ec' removed 2023-11-28T20:35:27.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.212+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92b406ae70 msgr2=0x7f92b406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:27.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92b406ae70 0x7f92b406d330 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f92cc003eb0 tx=0x7f92cc009040 comp rx=0 tx=0).stop 2023-11-28T20:35:27.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d01448e0 msgr2=0x7f92d0159f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:27.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d01448e0 0x7f92d0159f50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f92c8007e60 tx=0x7f92c800e4b0 comp rx=0 tx=0).stop 2023-11-28T20:35:27.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 shutdown_connections 2023-11-28T20:35:27.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f92b406ae70 0x7f92b406d330 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:27.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d01448e0 0x7f92d0159f50 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:27.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92d00cbe10 0x7f92d00c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:27.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 --2- 172.21.15.45:0/2861028760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92d0071a40 0x7f92d00ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:27.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 >> 172.21.15.45:0/2861028760 conn(0x7f92d00cec70 msgr2=0x7f92d00cb700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:27.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 shutdown_connections 2023-11-28T20:35:27.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.213+0000 7f92d64d7640 1 -- 172.21.15.45:0/2861028760 wait complete. 2023-11-28T20:35:27.239 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T20:35:27.239 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T20:35:27.239 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T20:35:27.239 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:35:27.239 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T20:35:27.240 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T20:35:27.240 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T20:35:27.240 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:35:27.240 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T20:35:27.240 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T20:35:27.240 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T20:35:27.240 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:35:27.241 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T20:35:27.241 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T20:35:27.241 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T20:35:27.241 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:35:27.260 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T20:35:27.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T20:35:27.264 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T20:35:27.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T20:35:27.265 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T20:35:27.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T20:35:27.270 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T20:35:27.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T20:35:27.309 INFO:tasks.ceph.mds.c:Started 2023-11-28T20:35:27.310 DEBUG:teuthology.parallel:result is None 2023-11-28T20:35:27.314 INFO:tasks.ceph.mds.a:Started 2023-11-28T20:35:27.314 DEBUG:teuthology.parallel:result is None 2023-11-28T20:35:27.318 INFO:tasks.ceph.mds.d:Started 2023-11-28T20:35:27.318 DEBUG:teuthology.parallel:result is None 2023-11-28T20:35:27.326 INFO:tasks.ceph.mds.b:Started 2023-11-28T20:35:27.326 DEBUG:teuthology.parallel:result is None 2023-11-28T20:35:27.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T20:35:27.476 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:35:27.476 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:35:27.479 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:35:27.480 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T20:35:27.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.523+0000 7f8dc759e640 1 Processor -- start 2023-11-28T20:35:27.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.523+0000 7f8dc759e640 1 -- start start 2023-11-28T20:35:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.524+0000 7f8dc759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc80cfcd0 0x7f8dc80b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.524+0000 7f8dc759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc80b68d0 con 0x7f8dc815d620 2023-11-28T20:35:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.524+0000 7f8dc759e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc8161dc0 con 0x7f8dc80d00b0 2023-11-28T20:35:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.524+0000 7f8dc759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80d1ce0 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.525+0000 7f8dc6d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8dc815d620 0x7f8dc8161070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39252/0 (socket says 172.21.15.45:39252) 2023-11-28T20:35:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.525+0000 7f8dc6d9d640 1 -- 172.21.15.45:0/2820121047 learned_addr learned my addr 172.21.15.45:0/2820121047 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.525+0000 7f8dc659c640 1 --2- 172.21.15.45:0/2820121047 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc80cfcd0 0x7f8dc80b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:27.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.525+0000 7f8dc559a640 1 -- 172.21.15.45:0/2820121047 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114067903 0 0) 0x7f8dc80b68d0 con 0x7f8dc815d620 2023-11-28T20:35:27.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.525+0000 7f8dc559a640 1 -- 172.21.15.45:0/2820121047 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db0003520 con 0x7f8dc815d620 2023-11-28T20:35:27.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.525+0000 7f8dc659c640 1 -- 172.21.15.45:0/2820121047 >> v1:172.21.15.190:6789/0 conn(0x7f8dc80d00b0 legacy=0x7f8dc815cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:27.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.525+0000 7f8dc659c640 1 -- 172.21.15.45:0/2820121047 >> v1:172.21.15.45:6789/0 conn(0x7f8dc815d620 legacy=0x7f8dc8161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.525+0000 7f8dc559a640 1 -- 172.21.15.45:0/2820121047 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367528695 0 0) 0x7f8dc8161dc0 con 0x7f8dc80d00b0 2023-11-28T20:35:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.525+0000 7f8dc659c640 1 -- 172.21.15.45:0/2820121047 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80d1340 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.527+0000 7f8dc659c640 1 --2- 172.21.15.45:0/2820121047 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc80cfcd0 0x7f8dc80b6390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8db80097d0 tx=0x7f8db802b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d58edeb6d7477172 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.527+0000 7f8dc559a640 1 -- 172.21.15.45:0/2820121047 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8030890 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.527+0000 7f8dc559a640 1 -- 172.21.15.45:0/2820121047 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db80309f0 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.527+0000 7f8dc559a640 1 -- 172.21.15.45:0/2820121047 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8030d80 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.528+0000 7f8dc759e640 1 -- 172.21.15.45:0/2820121047 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc80cfcd0 msgr2=0x7f8dc80b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:27.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.528+0000 7f8dc759e640 1 --2- 172.21.15.45:0/2820121047 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc80cfcd0 0x7f8dc80b6390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8db80097d0 tx=0x7f8db802b9d0 comp rx=0 tx=0).stop 2023-11-28T20:35:27.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.528+0000 7f8dc759e640 1 -- 172.21.15.45:0/2820121047 shutdown_connections 2023-11-28T20:35:27.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.528+0000 7f8dc759e640 1 --2- 172.21.15.45:0/2820121047 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc80cfcd0 0x7f8dc80b6390 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:27.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.528+0000 7f8dc759e640 1 -- 172.21.15.45:0/2820121047 >> 172.21.15.45:0/2820121047 conn(0x7f8dc806a500 msgr2=0x7f8dc806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:27.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.528+0000 7f8dc759e640 1 -- 172.21.15.45:0/2820121047 shutdown_connections 2023-11-28T20:35:27.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.528+0000 7f8dc759e640 1 -- 172.21.15.45:0/2820121047 wait complete. 2023-11-28T20:35:27.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.530+0000 7f8dc759e640 1 Processor -- start 2023-11-28T20:35:27.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.530+0000 7f8dc759e640 1 -- start start 2023-11-28T20:35:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.531+0000 7f8dc759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dc80cfcd0 0x7f8dc80cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.531+0000 7f8dc759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dc80d00b0 0x7f8dc814f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.531+0000 7f8dc759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc815d620 0x7f8dc81f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.531+0000 7f8dc759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80ca420 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.531+0000 7f8dc759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80ce390 con 0x7f8dc80d00b0 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.531+0000 7f8dc759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc814fac0 con 0x7f8dc815d620 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.531+0000 7f8dc659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dc80cfcd0 0x7f8dc80cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.531+0000 7f8dc659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dc80cfcd0 0x7f8dc80cde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57554/0 (socket says 172.21.15.45:57554) 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.531+0000 7f8dc659c640 1 -- 172.21.15.45:0/2749134656 learned_addr learned my addr 172.21.15.45:0/2749134656 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.532+0000 7f8dc6d9d640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc815d620 0x7f8dc81f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.532+0000 7f8dc5d9b640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dc80d00b0 0x7f8dc814f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.532+0000 7f8dc659c640 1 -- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc815d620 msgr2=0x7f8dc81f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.532+0000 7f8dc659c640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc815d620 0x7f8dc81f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.532+0000 7f8dc659c640 1 -- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dc80d00b0 msgr2=0x7f8dc814f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.532+0000 7f8dc659c640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dc80d00b0 0x7f8dc814f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.532+0000 7f8dc659c640 1 -- 172.21.15.45:0/2749134656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db8009400 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.532+0000 7f8dc659c640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dc80cfcd0 0x7f8dc80cde50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8db80097d0 tx=0x7f8db8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.533+0000 7f8db6ffd640 1 -- 172.21.15.45:0/2749134656 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db803e040 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.533+0000 7f8db6ffd640 1 -- 172.21.15.45:0/2749134656 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db8039c10 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.533+0000 7f8db6ffd640 1 -- 172.21.15.45:0/2749134656 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8004030 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.533+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc81f3110 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.533+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc81f36d0 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.534+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc80ca560 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.535+0000 7f8db6ffd640 1 -- 172.21.15.45:0/2749134656 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8db8004190 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.536+0000 7f8db6ffd640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8da806ae40 0x7f8da806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:27.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.536+0000 7f8db6ffd640 1 -- 172.21.15.45:0/2749134656 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f8db80ab930 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.537+0000 7f8dc5d9b640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8da806ae40 0x7f8da806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.538+0000 7f8dc5d9b640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8da806ae40 0x7f8da806d300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8dc815c5d0 tx=0x7f8dbc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.540+0000 7f8db6ffd640 1 -- 172.21.15.45:0/2749134656 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8dc80ca560 con 0x7f8dc80cfcd0 2023-11-28T20:35:27.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:27.747+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8dc80ca560 con 0x7f8dc80cfcd0 2023-11-28T20:35:28.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.232+0000 7f8db6ffd640 1 -- 172.21.15.45:0/2749134656 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8dc80ca560 con 0x7f8dc80cfcd0 2023-11-28T20:35:28.234 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T20:35:28.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8da806ae40 msgr2=0x7f8da806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:28.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8da806ae40 0x7f8da806d300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8dc815c5d0 tx=0x7f8dbc009040 comp rx=0 tx=0).stop 2023-11-28T20:35:28.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dc80cfcd0 msgr2=0x7f8dc80cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:28.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dc80cfcd0 0x7f8dc80cde50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8db80097d0 tx=0x7f8db8004730 comp rx=0 tx=0).stop 2023-11-28T20:35:28.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 shutdown_connections 2023-11-28T20:35:28.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f8da806ae40 0x7f8da806d300 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dc815d620 0x7f8dc81f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dc80d00b0 0x7f8dc814f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 --2- 172.21.15.45:0/2749134656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8dc80cfcd0 0x7f8dc80cde50 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 >> 172.21.15.45:0/2749134656 conn(0x7f8dc806a500 msgr2=0x7f8dc80683f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:28.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.236+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 shutdown_connections 2023-11-28T20:35:28.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.237+0000 7f8dc759e640 1 -- 172.21.15.45:0/2749134656 wait complete. 2023-11-28T20:35:28.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T20:35:28.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.469+0000 7f78317de640 1 Processor -- start 2023-11-28T20:35:28.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.469+0000 7f78317de640 1 -- start start 2023-11-28T20:35:28.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.472+0000 7f78317de640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f782c0b35b0 0x7f782c15ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.472+0000 7f78317de640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c15bda0 con 0x7f782c0b1340 2023-11-28T20:35:28.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.472+0000 7f78317de640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c15c080 con 0x7f782c15b850 2023-11-28T20:35:28.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.472+0000 7f78317de640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782c15ee40 con 0x7f782c0b35b0 2023-11-28T20:35:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.472+0000 7f782b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f782c0b1340 0x7f782c15f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39260/0 (socket says 172.21.15.45:39260) 2023-11-28T20:35:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.472+0000 7f782b7fe640 1 -- 172.21.15.45:0/1507024025 learned_addr learned my addr 172.21.15.45:0/1507024025 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.472+0000 7f782affd640 1 --2- 172.21.15.45:0/1507024025 >> v2:172.21.15.190:3300/0 conn(0x7f782c0b35b0 0x7f782c15ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638219298 0 0) 0x7f782c15bda0 con 0x7f782c0b1340 2023-11-28T20:35:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7818003520 con 0x7f782c0b1340 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1983875189 0 0) 0x7f7818003520 con 0x7f782c0b1340 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 >> v2:172.21.15.190:3300/0 conn(0x7f782c0b35b0 msgr2=0x7f782c15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 --2- 172.21.15.45:0/1507024025 >> v2:172.21.15.190:3300/0 conn(0x7f782c0b35b0 0x7f782c15ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 >> v1:172.21.15.190:6789/0 conn(0x7f782c15b850 legacy=0x7f782c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c15c360 con 0x7f782c0b1340 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f78240019f0 con 0x7f782c0b1340 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 >> v1:172.21.15.45:6789/0 conn(0x7f782c0b1340 legacy=0x7f782c15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.473+0000 7f782a7fc640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180039c0 0x7f7818003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.474+0000 7f782a7fc640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78180047c0 0x7f7818004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.474+0000 7f782a7fc640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7818005180 0x7f781800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.474+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c15ee40 con 0x7f78180047c0 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.474+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781800c080 con 0x7f7818005180 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.474+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781800c240 con 0x7f78180039c0 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.474+0000 7f7823fff640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78180047c0 0x7f7818004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.474+0000 7f782affd640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180039c0 0x7f7818003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.474+0000 7f782b7fe640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7818005180 0x7f781800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.475+0000 7f7823fff640 1 -- 172.21.15.45:0/1507024025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180039c0 msgr2=0x7f7818003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.475+0000 7f7823fff640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180039c0 0x7f7818003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.475+0000 7f7823fff640 1 -- 172.21.15.45:0/1507024025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7818005180 msgr2=0x7f781800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.475+0000 7f7823fff640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7818005180 0x7f781800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.475+0000 7f7823fff640 1 -- 172.21.15.45:0/1507024025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818003520 con 0x7f78180047c0 2023-11-28T20:35:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.475+0000 7f782affd640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180039c0 0x7f7818003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:28.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.475+0000 7f782b7fe640 1 -- 172.21.15.45:0/1507024025 reap_dead start 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.476+0000 7f7823fff640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78180047c0 0x7f7818004c40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7814001720 tx=0x7f78180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1394e9c0dca9aa87 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.476+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78140052d0 con 0x7f78180047c0 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.476+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781402e050 con 0x7f78180047c0 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.476+0000 7f782a7fc640 1 -- 172.21.15.45:0/1507024025 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814003830 con 0x7f78180047c0 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.477+0000 7f78317de640 1 -- 172.21.15.45:0/1507024025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78180047c0 msgr2=0x7f7818004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.477+0000 7f78317de640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78180047c0 0x7f7818004c40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7814001720 tx=0x7f78180057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.477+0000 7f78317de640 1 -- 172.21.15.45:0/1507024025 shutdown_connections 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.477+0000 7f78317de640 1 --2- 172.21.15.45:0/1507024025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78180047c0 0x7f7818004c40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.477+0000 7f78317de640 1 -- 172.21.15.45:0/1507024025 >> 172.21.15.45:0/1507024025 conn(0x7f782c06a500 msgr2=0x7f782c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.477+0000 7f78317de640 1 -- 172.21.15.45:0/1507024025 shutdown_connections 2023-11-28T20:35:28.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.477+0000 7f78317de640 1 -- 172.21.15.45:0/1507024025 wait complete. 2023-11-28T20:35:28.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.479+0000 7f78317de640 1 Processor -- start 2023-11-28T20:35:28.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.479+0000 7f78317de640 1 -- start start 2023-11-28T20:35:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.480+0000 7f78317de640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180047c0 0x7f782c0a9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.480+0000 7f78317de640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f782c142590 0x7f782c139250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.480+0000 7f78317de640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f782c142a10 0x7f782c139790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.480+0000 7f78317de640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0aa170 con 0x7f782c142590 2023-11-28T20:35:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.480+0000 7f78317de640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c1f4a50 con 0x7f782c142a10 2023-11-28T20:35:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.480+0000 7f78317de640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c1f4d30 con 0x7f78180047c0 2023-11-28T20:35:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.480+0000 7f7823fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f782c142590 0x7f782c139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f7823fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f782c142590 0x7f782c139250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57580/0 (socket says 172.21.15.45:57580) 2023-11-28T20:35:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f7823fff640 1 -- 172.21.15.45:0/960963226 learned_addr learned my addr 172.21.15.45:0/960963226 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:28.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f782b7fe640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f782c142a10 0x7f782c139790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:28.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f782affd640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180047c0 0x7f782c0a9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f7823fff640 1 -- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180047c0 msgr2=0x7f782c0a9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f7823fff640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180047c0 0x7f782c0a9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f7823fff640 1 -- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f782c142a10 msgr2=0x7f782c139790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f7823fff640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f782c142a10 0x7f782c139790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f7823fff640 1 -- 172.21.15.45:0/960963226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c1f5010 con 0x7f782c142590 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.481+0000 7f7823fff640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f782c142590 0x7f782c139250 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7814000c00 tx=0x7f7814001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.482+0000 7f78237fe640 1 -- 172.21.15.45:0/960963226 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78140076c0 con 0x7f782c142590 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.482+0000 7f78237fe640 1 -- 172.21.15.45:0/960963226 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7814007a30 con 0x7f782c142590 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.482+0000 7f78237fe640 1 -- 172.21.15.45:0/960963226 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814006be0 con 0x7f782c142590 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.482+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c1f52f0 con 0x7f782c142590 2023-11-28T20:35:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.482+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c1f3e00 con 0x7f782c142590 2023-11-28T20:35:28.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.483+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c1f4e70 con 0x7f782c142590 2023-11-28T20:35:28.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.484+0000 7f78237fe640 1 -- 172.21.15.45:0/960963226 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7814006d40 con 0x7f782c142590 2023-11-28T20:35:28.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.485+0000 7f78237fe640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f780c06ae40 0x7f780c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.485+0000 7f78237fe640 1 -- 172.21.15.45:0/960963226 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f78140ab190 con 0x7f782c142590 2023-11-28T20:35:28.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.486+0000 7f782affd640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f780c06ae40 0x7f780c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:28.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.487+0000 7f782affd640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f780c06ae40 0x7f780c06d300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f782c0692a0 tx=0x7f781c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:28.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.489+0000 7f78237fe640 1 -- 172.21.15.45:0/960963226 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f782c1f4e70 con 0x7f782c142590 2023-11-28T20:35:28.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.753+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f782c1f4e70 con 0x7f782c142590 2023-11-28T20:35:28.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.754+0000 7f78237fe640 1 -- 172.21.15.45:0/960963226 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f782c1f4e70 con 0x7f782c142590 2023-11-28T20:35:28.756 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:28.756 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:35:28.756 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T20:35:28.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.756 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T20:35:28.756 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T20:35:28.756 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.756 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:35:28.757 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:35:28.758 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T20:35:28.760 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.761 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:35:28.762 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T20:35:28.764 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:35:28.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:35:28.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.769 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-mds", 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCdTmZlbzDQNhAA81vk6v4Iif7vDrpgahEP7g==", 2023-11-28T20:35:28.770 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-mds" 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-mgr", 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCdTmZlszvQNhAA9vorPLc/IeiZ7bw26+GzLA==", 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-mgr" 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.771 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-osd", 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCdTmZld0TQNhAAxjfd6X0XnzLxDC54LgoIyg==", 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-osd" 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-rbd", 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCdTmZlHE3QNhAAXC2TC+krQlSLqQMb6FtRCA==", 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-rbd" 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCdTmZlEFbQNhAAzU1n85W3iDvB/WCclUQ86Q==", 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.bootstrap-rgw", 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCdTmZlSGLQNhAAjqy8HFPa7J/6mnXmtMLwzQ==", 2023-11-28T20:35:28.773 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile bootstrap-rgw" 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T20:35:28.775 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f780c06ae40 msgr2=0x7f780c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f780c06ae40 0x7f780c06d300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f782c0692a0 tx=0x7f781c008040 comp rx=0 tx=0).stop 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f782c142590 msgr2=0x7f782c139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f782c142590 0x7f782c139250 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7814000c00 tx=0x7f7814001120 comp rx=0 tx=0).stop 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 shutdown_connections 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f780c06ae40 0x7f780c06d300 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f782c142a10 0x7f782c139790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f782c142590 0x7f782c139250 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 --2- 172.21.15.45:0/960963226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78180047c0 0x7f782c0a9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:28.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 >> 172.21.15.45:0/960963226 conn(0x7f782c06a500 msgr2=0x7f782c070330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:28.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 shutdown_connections 2023-11-28T20:35:28.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.757+0000 7f78317de640 1 -- 172.21.15.45:0/960963226 wait complete. 2023-11-28T20:35:28.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-11-28T20:35:28.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.994+0000 7fe27e4da640 1 Processor -- start 2023-11-28T20:35:28.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.995+0000 7fe27e4da640 1 -- start start 2023-11-28T20:35:28.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.996+0000 7fe27e4da640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe27815b850 0x7fe2780b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.996+0000 7fe27e4da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2780b1fa0 0x7fe2780b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:28.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.996+0000 7fe27e4da640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27815bda0 con 0x7fe2780b35b0 2023-11-28T20:35:28.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.996+0000 7fe27e4da640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe27815ee40 con 0x7fe27815b850 2023-11-28T20:35:28.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.996+0000 7fe27e4da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27815c080 con 0x7fe2780b1fa0 2023-11-28T20:35:28.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.997+0000 7fe277fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe2780b35b0 0x7fe27815ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39274/0 (socket says 172.21.15.45:39274) 2023-11-28T20:35:28.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.997+0000 7fe277fff640 1 -- 172.21.15.45:0/3667480326 learned_addr learned my addr 172.21.15.45:0/3667480326 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:29.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.997+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599528838 0 0) 0x7fe27815bda0 con 0x7fe2780b35b0 2023-11-28T20:35:29.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.997+0000 7fe27ca77640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2780b1fa0 0x7fe2780b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.997+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe264003520 con 0x7fe2780b35b0 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.997+0000 7fe2777fe640 1 --2- 172.21.15.45:0/3667480326 >> v2:172.21.15.190:3300/0 conn(0x7fe27815b850 0x7fe2780b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.997+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534798114 0 0) 0x7fe264003520 con 0x7fe2780b35b0 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2780b1fa0 msgr2=0x7fe2780b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2780b1fa0 0x7fe2780b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 >> v2:172.21.15.190:3300/0 conn(0x7fe27815b850 msgr2=0x7fe2780b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 --2- 172.21.15.45:0/3667480326 >> v2:172.21.15.190:3300/0 conn(0x7fe27815b850 0x7fe2780b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27815c360 con 0x7fe2780b35b0 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fe2680019f0 con 0x7fe2780b35b0 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 >> v1:172.21.15.45:6789/0 conn(0x7fe2780b35b0 legacy=0x7fe27815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2780b1fa0 0x7fe264003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe264003f40 0x7fe2640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe264004d00 0x7fe264005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27815ee40 con 0x7fe264003f40 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27815c080 con 0x7fe264004d00 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26400ba30 con 0x7fe2780b1fa0 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe2777fe640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe264003f40 0x7fe2640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe277fff640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2780b1fa0 0x7fe264003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.998+0000 7fe27ca77640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe264004d00 0x7fe264005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.999+0000 7fe2777fe640 1 -- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2780b1fa0 msgr2=0x7fe264003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.999+0000 7fe2777fe640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2780b1fa0 0x7fe264003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.999+0000 7fe2777fe640 1 -- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe264004d00 msgr2=0x7fe264005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:29.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.999+0000 7fe2777fe640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe264004d00 0x7fe264005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.999+0000 7fe2777fe640 1 -- 172.21.15.45:0/3667480326 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe264003520 con 0x7fe264003f40 2023-11-28T20:35:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:28.999+0000 7fe277fff640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2780b1fa0 0x7fe264003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.000+0000 7fe2777fe640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe264003f40 0x7fe2640047c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe26000ba10 tx=0x7fe2640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2ee00780340e16 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.000+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260032680 con 0x7fe264003f40 2023-11-28T20:35:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.000+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2600327e0 con 0x7fe264003f40 2023-11-28T20:35:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.000+0000 7fe276ffd640 1 -- 172.21.15.45:0/3667480326 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260032b70 con 0x7fe264003f40 2023-11-28T20:35:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.001+0000 7fe27e4da640 1 -- 172.21.15.45:0/3667480326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe264003f40 msgr2=0x7fe2640047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.001+0000 7fe27e4da640 1 --2- 172.21.15.45:0/3667480326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe264003f40 0x7fe2640047c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe26000ba10 tx=0x7fe2640057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.001+0000 7fe27ca77640 1 -- 172.21.15.45:0/3667480326 reap_dead start 2023-11-28T20:35:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.001+0000 7fe27e4da640 1 -- 172.21.15.45:0/3667480326 shutdown_connections 2023-11-28T20:35:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.001+0000 7fe27e4da640 1 -- 172.21.15.45:0/3667480326 >> 172.21.15.45:0/3667480326 conn(0x7fe27806a500 msgr2=0x7fe27806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.001+0000 7fe27e4da640 1 -- 172.21.15.45:0/3667480326 shutdown_connections 2023-11-28T20:35:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.001+0000 7fe27e4da640 1 -- 172.21.15.45:0/3667480326 wait complete. 2023-11-28T20:35:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.003+0000 7fe27e4da640 1 Processor -- start 2023-11-28T20:35:29.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.003+0000 7fe27e4da640 1 -- start start 2023-11-28T20:35:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.004+0000 7fe27e4da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe264003f40 0x7fe278140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.004+0000 7fe27e4da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe278160770 0x7fe2781615a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.004+0000 7fe27e4da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe278161f30 0x7fe2781623b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.004+0000 7fe27e4da640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2781631b0 con 0x7fe278161f30 2023-11-28T20:35:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.004+0000 7fe27e4da640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe278160350 con 0x7fe278160770 2023-11-28T20:35:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.004+0000 7fe27e4da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2781411f0 con 0x7fe264003f40 2023-11-28T20:35:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.004+0000 7fe27ca77640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe278161f30 0x7fe2781623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.004+0000 7fe27ca77640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe278161f30 0x7fe2781623b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57608/0 (socket says 172.21.15.45:57608) 2023-11-28T20:35:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.004+0000 7fe27ca77640 1 -- 172.21.15.45:0/3180353055 learned_addr learned my addr 172.21.15.45:0/3180353055 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe2777fe640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe278160770 0x7fe2781615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe277fff640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe264003f40 0x7fe278140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe27ca77640 1 -- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe264003f40 msgr2=0x7fe278140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe27ca77640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe264003f40 0x7fe278140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe27ca77640 1 -- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe278160770 msgr2=0x7fe2781615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe27ca77640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe278160770 0x7fe2781615a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe27ca77640 1 -- 172.21.15.45:0/3180353055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe278162ed0 con 0x7fe278161f30 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe277fff640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe264003f40 0x7fe278140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe27ca77640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe278161f30 0x7fe2781623b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe2780b2920 tx=0x7fe270011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe274ff9640 1 -- 172.21.15.45:0/3180353055 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe270004500 con 0x7fe278161f30 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe274ff9640 1 -- 172.21.15.45:0/3180353055 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe270003610 con 0x7fe278161f30 2023-11-28T20:35:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe274ff9640 1 -- 172.21.15.45:0/3180353055 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2700039a0 con 0x7fe278161f30 2023-11-28T20:35:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.005+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27815fc90 con 0x7fe278161f30 2023-11-28T20:35:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.006+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe278161ae0 con 0x7fe278161f30 2023-11-28T20:35:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.007+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe278160490 con 0x7fe278161f30 2023-11-28T20:35:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.008+0000 7fe274ff9640 1 -- 172.21.15.45:0/3180353055 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe270003b00 con 0x7fe278161f30 2023-11-28T20:35:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.009+0000 7fe274ff9640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe25806ae70 0x7fe25806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.009+0000 7fe274ff9640 1 -- 172.21.15.45:0/3180353055 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe2700896e0 con 0x7fe278161f30 2023-11-28T20:35:29.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.010+0000 7fe277fff640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe25806ae70 0x7fe25806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.011+0000 7fe277fff640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe25806ae70 0x7fe25806d330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe268005c30 tx=0x7fe2680012f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:29.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.013+0000 7fe274ff9640 1 -- 172.21.15.45:0/3180353055 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe278160490 con 0x7fe278161f30 2023-11-28T20:35:29.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.285+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fe278160490 con 0x7fe278161f30 2023-11-28T20:35:29.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.290+0000 7fe274ff9640 1 -- 172.21.15.45:0/3180353055 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe278160490 con 0x7fe278161f30 2023-11-28T20:35:29.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe25806ae70 msgr2=0x7fe25806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe25806ae70 0x7fe25806d330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe268005c30 tx=0x7fe2680012f0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe278161f30 msgr2=0x7fe2781623b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe278161f30 0x7fe2781623b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe2780b2920 tx=0x7fe270011920 comp rx=0 tx=0).stop 2023-11-28T20:35:29.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 shutdown_connections 2023-11-28T20:35:29.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fe25806ae70 0x7fe25806d330 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe278161f30 0x7fe2781623b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe278160770 0x7fe2781615a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 --2- 172.21.15.45:0/3180353055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe264003f40 0x7fe278140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 >> 172.21.15.45:0/3180353055 conn(0x7fe27806a500 msgr2=0x7fe278070230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:29.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 shutdown_connections 2023-11-28T20:35:29.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.293+0000 7fe27e4da640 1 -- 172.21.15.45:0/3180353055 wait complete. 2023-11-28T20:35:29.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-11-28T20:35:29.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.538+0000 7f954d664640 1 Processor -- start 2023-11-28T20:35:29.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.539+0000 7f954d664640 1 -- start start 2023-11-28T20:35:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.540+0000 7f954d664640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 0x7f9548160da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.540+0000 7f954d664640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f95480b89c0 0x7f95480d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.540+0000 7f954d664640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95480b61d0 con 0x7f954815d290 2023-11-28T20:35:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.540+0000 7f954d664640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95481612e0 con 0x7f95480b89c0 2023-11-28T20:35:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.540+0000 7f954d664640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954815d670 con 0x7f95480d0130 2023-11-28T20:35:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.541+0000 7f954ce63640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f95480b89c0 0x7f95480d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.541+0000 7f954ce63640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f95480b89c0 0x7f95480d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57614/0 (socket says 172.21.15.45:57614) 2023-11-28T20:35:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.541+0000 7f954ce63640 1 -- 172.21.15.45:0/2459871733 learned_addr learned my addr 172.21.15.45:0/2459871733 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.541+0000 7f954ce63640 1 -- 172.21.15.45:0/2459871733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 msgr2=0x7f9548160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.541+0000 7f95477fe640 1 --2- 172.21.15.45:0/2459871733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 0x7f9548160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.541+0000 7f954ce63640 1 --2- 172.21.15.45:0/2459871733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 0x7f9548160da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.541+0000 7f954ce63640 1 -- 172.21.15.45:0/2459871733 >> v1:172.21.15.190:6789/0 conn(0x7f954815d290 legacy=0x7f95480cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:29.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.541+0000 7f954ce63640 1 -- 172.21.15.45:0/2459871733 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95481614f0 con 0x7f95480b89c0 2023-11-28T20:35:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.542+0000 7f954ce63640 1 --2- 172.21.15.45:0/2459871733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95480b89c0 0x7f95480d1b10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9540009e50 tx=0x7f954002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319f8edffe3ca2f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.543+0000 7f9546ffd640 1 -- 172.21.15.45:0/2459871733 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954002cb00 con 0x7f95480b89c0 2023-11-28T20:35:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.543+0000 7f9546ffd640 1 -- 172.21.15.45:0/2459871733 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954002cc60 con 0x7f95480b89c0 2023-11-28T20:35:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.543+0000 7f9546ffd640 1 -- 172.21.15.45:0/2459871733 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95400343f0 con 0x7f95480b89c0 2023-11-28T20:35:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.543+0000 7f954d664640 1 -- 172.21.15.45:0/2459871733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95480b89c0 msgr2=0x7f95480d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.543+0000 7f954d664640 1 --2- 172.21.15.45:0/2459871733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95480b89c0 0x7f95480d1b10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9540009e50 tx=0x7f954002bf60 comp rx=0 tx=0).stop 2023-11-28T20:35:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.544+0000 7f954d664640 1 -- 172.21.15.45:0/2459871733 shutdown_connections 2023-11-28T20:35:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.544+0000 7f954d664640 1 --2- 172.21.15.45:0/2459871733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95480b89c0 0x7f95480d1b10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.544+0000 7f954d664640 1 --2- 172.21.15.45:0/2459871733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 0x7f9548160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.544+0000 7f954d664640 1 -- 172.21.15.45:0/2459871733 >> 172.21.15.45:0/2459871733 conn(0x7f95480b4d00 msgr2=0x7f95480b5110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.544+0000 7f954d664640 1 -- 172.21.15.45:0/2459871733 shutdown_connections 2023-11-28T20:35:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.544+0000 7f954d664640 1 -- 172.21.15.45:0/2459871733 wait complete. 2023-11-28T20:35:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.546+0000 7f954d664640 1 Processor -- start 2023-11-28T20:35:29.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.546+0000 7f954d664640 1 -- start start 2023-11-28T20:35:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954d664640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95480b89c0 0x7f95480cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954d664640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 0x7f95480cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954d664640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f954815d290 0x7f9548155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954d664640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95480c9950 con 0x7f954815d290 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954d664640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95480b53c0 con 0x7f95480b89c0 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954d664640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95480b50e0 con 0x7f95480d0130 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954ce63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f954815d290 0x7f9548155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954ce63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f954815d290 0x7f9548155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57630/0 (socket says 172.21.15.45:57630) 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954ce63640 1 -- 172.21.15.45:0/2264348290 learned_addr learned my addr 172.21.15.45:0/2264348290 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f95477fe640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 0x7f95480cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f9547fff640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95480b89c0 0x7f95480cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954ce63640 1 -- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 msgr2=0x7f95480cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954ce63640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 0x7f95480cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954ce63640 1 -- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95480b89c0 msgr2=0x7f95480cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954ce63640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95480b89c0 0x7f95480cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f954ce63640 1 -- 172.21.15.45:0/2264348290 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9540009b30 con 0x7f954815d290 2023-11-28T20:35:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.547+0000 7f95477fe640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 0x7f95480cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.548+0000 7f954ce63640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f954815d290 0x7f9548155950 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9540009e50 tx=0x7f9540004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.548+0000 7f9544ff9640 1 -- 172.21.15.45:0/2264348290 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540009510 con 0x7f954815d290 2023-11-28T20:35:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.548+0000 7f9544ff9640 1 -- 172.21.15.45:0/2264348290 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954003e450 con 0x7f954815d290 2023-11-28T20:35:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.548+0000 7f9544ff9640 1 -- 172.21.15.45:0/2264348290 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954003e800 con 0x7f954815d290 2023-11-28T20:35:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.548+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9548154830 con 0x7f954815d290 2023-11-28T20:35:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.548+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95480cc470 con 0x7f954815d290 2023-11-28T20:35:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.549+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9548071c50 con 0x7f954815d290 2023-11-28T20:35:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.550+0000 7f9544ff9640 1 -- 172.21.15.45:0/2264348290 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f954003e960 con 0x7f954815d290 2023-11-28T20:35:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.552+0000 7f9544ff9640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f953406ae40 0x7f953406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.552+0000 7f9544ff9640 1 -- 172.21.15.45:0/2264348290 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f95400ac360 con 0x7f954815d290 2023-11-28T20:35:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.552+0000 7f9547fff640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f953406ae40 0x7f953406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:29.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.553+0000 7f9547fff640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f953406ae40 0x7f953406d300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f953800a750 tx=0x7f9538008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:29.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.557+0000 7f9544ff9640 1 -- 172.21.15.45:0/2264348290 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9540077630 con 0x7f954815d290 2023-11-28T20:35:29.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.821+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f954804e920 con 0x7f954815d290 2023-11-28T20:35:29.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.825+0000 7f9544ff9640 1 -- 172.21.15.45:0/2264348290 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f954007ac50 con 0x7f954815d290 2023-11-28T20:35:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f953406ae40 msgr2=0x7f953406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f953406ae40 0x7f953406d300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f953800a750 tx=0x7f9538008040 comp rx=0 tx=0).stop 2023-11-28T20:35:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f954815d290 msgr2=0x7f9548155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f954815d290 0x7f9548155950 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9540009e50 tx=0x7f9540004a40 comp rx=0 tx=0).stop 2023-11-28T20:35:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 shutdown_connections 2023-11-28T20:35:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f953406ae40 0x7f953406d300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f954815d290 0x7f9548155950 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95480d0130 0x7f95480cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 --2- 172.21.15.45:0/2264348290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95480b89c0 0x7f95480cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:29.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 >> 172.21.15.45:0/2264348290 conn(0x7f95480b4d00 msgr2=0x7f95481523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:29.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.828+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 shutdown_connections 2023-11-28T20:35:29.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:29.829+0000 7f954d664640 1 -- 172.21.15.45:0/2264348290 wait complete. 2023-11-28T20:35:29.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-11-28T20:35:30.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.080+0000 7fc30b402640 1 Processor -- start 2023-11-28T20:35:30.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.080+0000 7fc30b402640 1 -- start start 2023-11-28T20:35:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30b402640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc304164be0 0x7fc3040b2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30b402640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc304071ec0 0x7fc30406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30b402640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc30406c480 0x7fc3040713c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30b402640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc304070340 con 0x7fc30406c480 2023-11-28T20:35:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30b402640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc304070600 con 0x7fc304071ec0 2023-11-28T20:35:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30b402640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3040708e0 con 0x7fc304164be0 2023-11-28T20:35:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30ac01640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc30406c480 0x7fc3040713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30ac01640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc30406c480 0x7fc3040713c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57632/0 (socket says 172.21.15.45:57632) 2023-11-28T20:35:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30ac01640 1 -- 172.21.15.45:0/1782722203 learned_addr learned my addr 172.21.15.45:0/1782722203 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.082+0000 7fc30a400640 1 --2- 172.21.15.45:0/1782722203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc304164be0 0x7fc3040b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.083+0000 7fc309bff640 1 --2- 172.21.15.45:0/1782722203 >> v2:172.21.15.190:3300/0 conn(0x7fc304071ec0 0x7fc30406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.083+0000 7fc30ac01640 1 -- 172.21.15.45:0/1782722203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc304164be0 msgr2=0x7fc3040b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.083+0000 7fc30ac01640 1 --2- 172.21.15.45:0/1782722203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc304164be0 0x7fc3040b2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.083+0000 7fc30ac01640 1 -- 172.21.15.45:0/1782722203 >> v2:172.21.15.190:3300/0 conn(0x7fc304071ec0 msgr2=0x7fc30406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.083+0000 7fc30ac01640 1 --2- 172.21.15.45:0/1782722203 >> v2:172.21.15.190:3300/0 conn(0x7fc304071ec0 0x7fc30406bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.083+0000 7fc30ac01640 1 -- 172.21.15.45:0/1782722203 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc304070bc0 con 0x7fc30406c480 2023-11-28T20:35:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.083+0000 7fc309bff640 1 --2- 172.21.15.45:0/1782722203 >> v2:172.21.15.190:3300/0 conn(0x7fc304071ec0 0x7fc30406bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.084+0000 7fc30ac01640 1 --2- 172.21.15.45:0/1782722203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30406c480 0x7fc3040713c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc3040cc7d0 tx=0x7fc2fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e2e5221cda5aa4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.084+0000 7fc3093fe640 1 -- 172.21.15.45:0/1782722203 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc02ca70 con 0x7fc30406c480 2023-11-28T20:35:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.084+0000 7fc3093fe640 1 -- 172.21.15.45:0/1782722203 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2fc02cbd0 con 0x7fc30406c480 2023-11-28T20:35:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.084+0000 7fc3093fe640 1 -- 172.21.15.45:0/1782722203 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0343f0 con 0x7fc30406c480 2023-11-28T20:35:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.085+0000 7fc30b402640 1 -- 172.21.15.45:0/1782722203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30406c480 msgr2=0x7fc3040713c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.085+0000 7fc30b402640 1 --2- 172.21.15.45:0/1782722203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30406c480 0x7fc3040713c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc3040cc7d0 tx=0x7fc2fc02bed0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.085+0000 7fc30b402640 1 -- 172.21.15.45:0/1782722203 shutdown_connections 2023-11-28T20:35:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.085+0000 7fc30b402640 1 --2- 172.21.15.45:0/1782722203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc30406c480 0x7fc3040713c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.085+0000 7fc30b402640 1 --2- 172.21.15.45:0/1782722203 >> v2:172.21.15.190:3300/0 conn(0x7fc304071ec0 0x7fc30406bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.085+0000 7fc30b402640 1 --2- 172.21.15.45:0/1782722203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc304164be0 0x7fc3040b2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.085+0000 7fc30b402640 1 -- 172.21.15.45:0/1782722203 >> 172.21.15.45:0/1782722203 conn(0x7fc3040b0e70 msgr2=0x7fc3040b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:30.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.085+0000 7fc30b402640 1 -- 172.21.15.45:0/1782722203 shutdown_connections 2023-11-28T20:35:30.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.085+0000 7fc30b402640 1 -- 172.21.15.45:0/1782722203 wait complete. 2023-11-28T20:35:30.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.087+0000 7fc30b402640 1 Processor -- start 2023-11-28T20:35:30.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.087+0000 7fc30b402640 1 -- start start 2023-11-28T20:35:30.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.088+0000 7fc30b402640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc30406c480 0x7fc304159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.088+0000 7fc30b402640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc304071ec0 0x7fc30409a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.088+0000 7fc30b402640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc304164be0 0x7fc3041f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.088+0000 7fc30b402640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3040af910 con 0x7fc304164be0 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.088+0000 7fc30b402640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc304159660 con 0x7fc304071ec0 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.088+0000 7fc30b402640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3040afda0 con 0x7fc30406c480 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc30ac01640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc304164be0 0x7fc3041f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc30ac01640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc304164be0 0x7fc3041f6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57642/0 (socket says 172.21.15.45:57642) 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc30ac01640 1 -- 172.21.15.45:0/580670750 learned_addr learned my addr 172.21.15.45:0/580670750 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc309bff640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc304071ec0 0x7fc30409a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc30a400640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc30406c480 0x7fc304159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc30ac01640 1 -- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc30406c480 msgr2=0x7fc304159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc30ac01640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc30406c480 0x7fc304159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc30ac01640 1 -- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc304071ec0 msgr2=0x7fc30409a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc30ac01640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc304071ec0 0x7fc30409a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.089+0000 7fc30ac01640 1 -- 172.21.15.45:0/580670750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc009b30 con 0x7fc304164be0 2023-11-28T20:35:30.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.090+0000 7fc30ac01640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc304164be0 0x7fc3041f6ec0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc036040 tx=0x7fc2fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.090+0000 7fc2faffd640 1 -- 172.21.15.45:0/580670750 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc009510 con 0x7fc304164be0 2023-11-28T20:35:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.090+0000 7fc2faffd640 1 -- 172.21.15.45:0/580670750 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2fc03e450 con 0x7fc304164be0 2023-11-28T20:35:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.090+0000 7fc2faffd640 1 -- 172.21.15.45:0/580670750 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc03e800 con 0x7fc304164be0 2023-11-28T20:35:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.090+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc30409ac80 con 0x7fc304164be0 2023-11-28T20:35:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.090+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3040af2c0 con 0x7fc304164be0 2023-11-28T20:35:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.091+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3040afa80 con 0x7fc304164be0 2023-11-28T20:35:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.092+0000 7fc2faffd640 1 -- 172.21.15.45:0/580670750 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc2fc03e960 con 0x7fc304164be0 2023-11-28T20:35:30.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.093+0000 7fc2faffd640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc2e806ae40 0x7fc2e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.093+0000 7fc2faffd640 1 -- 172.21.15.45:0/580670750 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fc2fc0ac300 con 0x7fc304164be0 2023-11-28T20:35:30.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.094+0000 7fc30a400640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc2e806ae40 0x7fc2e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.095+0000 7fc30a400640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc2e806ae40 0x7fc2e806d300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc300006fd0 tx=0x7fc300008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:30.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.098+0000 7fc2faffd640 1 -- 172.21.15.45:0/580670750 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2fc0775d0 con 0x7fc304164be0 2023-11-28T20:35:30.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.367+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fc30409d0f0 con 0x7fc304164be0 2023-11-28T20:35:30.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.370+0000 7fc2faffd640 1 -- 172.21.15.45:0/580670750 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc2fc07abf0 con 0x7fc304164be0 2023-11-28T20:35:30.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc2e806ae40 msgr2=0x7fc2e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc2e806ae40 0x7fc2e806d300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc300006fd0 tx=0x7fc300008040 comp rx=0 tx=0).stop 2023-11-28T20:35:30.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc304164be0 msgr2=0x7fc3041f6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc304164be0 0x7fc3041f6ec0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc036040 tx=0x7fc2fc004a40 comp rx=0 tx=0).stop 2023-11-28T20:35:30.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 shutdown_connections 2023-11-28T20:35:30.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc2e806ae40 0x7fc2e806d300 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc304164be0 0x7fc3041f6ec0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc304071ec0 0x7fc30409a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 --2- 172.21.15.45:0/580670750 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc30406c480 0x7fc304159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.373+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 >> 172.21.15.45:0/580670750 conn(0x7fc3040b0e70 msgr2=0x7fc304163a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:30.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.374+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 shutdown_connections 2023-11-28T20:35:30.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.374+0000 7fc30b402640 1 -- 172.21.15.45:0/580670750 wait complete. 2023-11-28T20:35:30.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-11-28T20:35:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.603+0000 7f563b0ef640 1 Processor -- start 2023-11-28T20:35:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.603+0000 7f563b0ef640 1 -- start start 2023-11-28T20:35:30.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.604+0000 7f563b0ef640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f563415d140 0x7f56340d1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.604+0000 7f563b0ef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 0x7f56340cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.604+0000 7f563b0ef640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f56340d1680 0x7f56340cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.604+0000 7f563b0ef640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56340d2810 con 0x7f563415d140 2023-11-28T20:35:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.604+0000 7f563b0ef640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56340d1c00 con 0x7f56340d1680 2023-11-28T20:35:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.604+0000 7f563b0ef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56340d1ee0 con 0x7f563415d520 2023-11-28T20:35:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f563a0ed640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f563415d140 0x7f56340d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f563a0ed640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f563415d140 0x7f56340d1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57644/0 (socket says 172.21.15.45:57644) 2023-11-28T20:35:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f563a0ed640 1 -- 172.21.15.45:0/204158310 learned_addr learned my addr 172.21.15.45:0/204158310 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f563a8ee640 1 --2- 172.21.15.45:0/204158310 >> v2:172.21.15.190:3300/0 conn(0x7f56340d1680 0x7f56340cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f56398ec640 1 --2- 172.21.15.45:0/204158310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 0x7f56340cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f563a0ed640 1 -- 172.21.15.45:0/204158310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 msgr2=0x7f56340cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f563a0ed640 1 --2- 172.21.15.45:0/204158310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 0x7f56340cfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f563a0ed640 1 -- 172.21.15.45:0/204158310 >> v2:172.21.15.190:3300/0 conn(0x7f56340d1680 msgr2=0x7f56340cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f563a0ed640 1 --2- 172.21.15.45:0/204158310 >> v2:172.21.15.190:3300/0 conn(0x7f56340d1680 0x7f56340cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f563a0ed640 1 -- 172.21.15.45:0/204158310 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56340d21c0 con 0x7f563415d140 2023-11-28T20:35:30.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.605+0000 7f56398ec640 1 --2- 172.21.15.45:0/204158310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 0x7f56340cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:30.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.606+0000 7f563a0ed640 1 --2- 172.21.15.45:0/204158310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563415d140 0x7f56340d1140 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f56340d0920 tx=0x7f563002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266fdf2d5b8b34bb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f56390eb640 1 -- 172.21.15.45:0/204158310 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630031870 con 0x7f563415d140 2023-11-28T20:35:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f56390eb640 1 -- 172.21.15.45:0/204158310 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56300319d0 con 0x7f563415d140 2023-11-28T20:35:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f56390eb640 1 -- 172.21.15.45:0/204158310 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630031d60 con 0x7f563415d140 2023-11-28T20:35:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f563b0ef640 1 -- 172.21.15.45:0/204158310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563415d140 msgr2=0x7f56340d1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204158310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563415d140 0x7f56340d1140 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f56340d0920 tx=0x7f563002b910 comp rx=0 tx=0).stop 2023-11-28T20:35:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f563b0ef640 1 -- 172.21.15.45:0/204158310 shutdown_connections 2023-11-28T20:35:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204158310 >> v2:172.21.15.190:3300/0 conn(0x7f56340d1680 0x7f56340cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204158310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 0x7f56340cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204158310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563415d140 0x7f56340d1140 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f563b0ef640 1 -- 172.21.15.45:0/204158310 >> 172.21.15.45:0/204158310 conn(0x7f563406a500 msgr2=0x7f563406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.607+0000 7f563b0ef640 1 -- 172.21.15.45:0/204158310 shutdown_connections 2023-11-28T20:35:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.608+0000 7f563b0ef640 1 -- 172.21.15.45:0/204158310 wait complete. 2023-11-28T20:35:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.609+0000 7f563b0ef640 1 Processor -- start 2023-11-28T20:35:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.609+0000 7f563b0ef640 1 -- start start 2023-11-28T20:35:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.610+0000 7f563b0ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56340d1680 0x7f56340cc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.610+0000 7f563b0ef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f563415d140 0x7f56340ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.610+0000 7f563b0ef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 0x7f56340aa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563b0ef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563415e840 con 0x7f56340d1680 2023-11-28T20:35:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563b0ef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56340a99f0 con 0x7f563415d140 2023-11-28T20:35:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563b0ef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56340a9cd0 con 0x7f563415d520 2023-11-28T20:35:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563a0ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56340d1680 0x7f56340cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563a0ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56340d1680 0x7f56340cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57658/0 (socket says 172.21.15.45:57658) 2023-11-28T20:35:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563a0ed640 1 -- 172.21.15.45:0/204296201 learned_addr learned my addr 172.21.15.45:0/204296201 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f56398ec640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f563415d140 0x7f56340ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563a0ed640 1 -- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 msgr2=0x7f56340aa7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563a8ee640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 0x7f56340aa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563a0ed640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 0x7f56340aa7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563a0ed640 1 -- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f563415d140 msgr2=0x7f56340ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563a0ed640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f563415d140 0x7f56340ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f563a0ed640 1 -- 172.21.15.45:0/204296201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56300094d0 con 0x7f56340d1680 2023-11-28T20:35:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.611+0000 7f56398ec640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f563415d140 0x7f56340ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.612+0000 7f563a0ed640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56340d1680 0x7f56340cc590 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f563002b950 tx=0x7f56300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.612+0000 7f562affd640 1 -- 172.21.15.45:0/204296201 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563003ad30 con 0x7f56340d1680 2023-11-28T20:35:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.612+0000 7f562affd640 1 -- 172.21.15.45:0/204296201 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56300443f0 con 0x7f56340d1680 2023-11-28T20:35:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.612+0000 7f562affd640 1 -- 172.21.15.45:0/204296201 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56300447a0 con 0x7f56340d1680 2023-11-28T20:35:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.612+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56340a9fb0 con 0x7f56340d1680 2023-11-28T20:35:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.612+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56340aacf0 con 0x7f56340d1680 2023-11-28T20:35:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.613+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5634071b30 con 0x7f56340d1680 2023-11-28T20:35:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.614+0000 7f562affd640 1 -- 172.21.15.45:0/204296201 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f563004d450 con 0x7f56340d1680 2023-11-28T20:35:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.615+0000 7f562affd640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f561806ae40 0x7f561806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.615+0000 7f562affd640 1 -- 172.21.15.45:0/204296201 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f563002d040 con 0x7f56340d1680 2023-11-28T20:35:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.616+0000 7f56398ec640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f561806ae40 0x7f561806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.617+0000 7f56398ec640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f561806ae40 0x7f561806d300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:30.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.619+0000 7f562affd640 1 -- 172.21.15.45:0/204296201 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5630076cd0 con 0x7f56340d1680 2023-11-28T20:35:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.881+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f563404e920 con 0x7f56340d1680 2023-11-28T20:35:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.886+0000 7f562affd640 1 -- 172.21.15.45:0/204296201 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5630035070 con 0x7f56340d1680 2023-11-28T20:35:30.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f561806ae40 msgr2=0x7f561806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f561806ae40 0x7f561806d300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624008040 comp rx=0 tx=0).stop 2023-11-28T20:35:30.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56340d1680 msgr2=0x7f56340cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:30.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56340d1680 0x7f56340cc590 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f563002b950 tx=0x7f56300046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 shutdown_connections 2023-11-28T20:35:30.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f561806ae40 0x7f561806d300 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f563415d520 0x7f56340aa7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f563415d140 0x7f56340ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 --2- 172.21.15.45:0/204296201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56340d1680 0x7f56340cc590 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:30.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 >> 172.21.15.45:0/204296201 conn(0x7f563406a500 msgr2=0x7f56340cfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:30.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 shutdown_connections 2023-11-28T20:35:30.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:30.889+0000 7f563b0ef640 1 -- 172.21.15.45:0/204296201 wait complete. 2023-11-28T20:35:30.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-11-28T20:35:31.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.118+0000 7f303a21a640 1 Processor -- start 2023-11-28T20:35:31.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.118+0000 7f303a21a640 1 -- start start 2023-11-28T20:35:31.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.119+0000 7f303a21a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f30340cfcd0 0x7f30340b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.119+0000 7f303a21a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340d00b0 0x7f303415cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.119+0000 7f303a21a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30340b68d0 con 0x7f303415d450 2023-11-28T20:35:31.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.119+0000 7f303a21a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30340d1ce0 con 0x7f30340cfcd0 2023-11-28T20:35:31.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.119+0000 7f303a21a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303415d8e0 con 0x7f30340d00b0 2023-11-28T20:35:31.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.120+0000 7f3039218640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f30340cfcd0 0x7f30340b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.120+0000 7f3039218640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f30340cfcd0 0x7f30340b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57672/0 (socket says 172.21.15.45:57672) 2023-11-28T20:35:31.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.120+0000 7f3039218640 1 -- 172.21.15.45:0/3751059672 learned_addr learned my addr 172.21.15.45:0/3751059672 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:31.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.120+0000 7f3038a17640 1 --2- 172.21.15.45:0/3751059672 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340d00b0 0x7f303415cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.120+0000 7f3039218640 1 -- 172.21.15.45:0/3751059672 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340d00b0 msgr2=0x7f303415cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:31.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.120+0000 7f3039218640 1 --2- 172.21.15.45:0/3751059672 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340d00b0 0x7f303415cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.120+0000 7f3039218640 1 -- 172.21.15.45:0/3751059672 >> v1:172.21.15.190:6789/0 conn(0x7f303415d450 legacy=0x7f3034161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:31.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.120+0000 7f3039218640 1 -- 172.21.15.45:0/3751059672 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3034161dc0 con 0x7f30340cfcd0 2023-11-28T20:35:31.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.121+0000 7f3039218640 1 --2- 172.21.15.45:0/3751059672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30340cfcd0 0x7f30340b6390 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3024009e50 tx=0x7f302402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbfd50d197af48d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:31.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.122+0000 7f3033fff640 1 -- 172.21.15.45:0/3751059672 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302402cb00 con 0x7f30340cfcd0 2023-11-28T20:35:31.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.122+0000 7f3033fff640 1 -- 172.21.15.45:0/3751059672 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302402cc60 con 0x7f30340cfcd0 2023-11-28T20:35:31.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.122+0000 7f3033fff640 1 -- 172.21.15.45:0/3751059672 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30240343f0 con 0x7f30340cfcd0 2023-11-28T20:35:31.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.122+0000 7f303a21a640 1 -- 172.21.15.45:0/3751059672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30340cfcd0 msgr2=0x7f30340b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.122+0000 7f303a21a640 1 --2- 172.21.15.45:0/3751059672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30340cfcd0 0x7f30340b6390 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3024009e50 tx=0x7f302402bf60 comp rx=0 tx=0).stop 2023-11-28T20:35:31.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.123+0000 7f303a21a640 1 -- 172.21.15.45:0/3751059672 shutdown_connections 2023-11-28T20:35:31.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.123+0000 7f303a21a640 1 --2- 172.21.15.45:0/3751059672 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340d00b0 0x7f303415cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.123+0000 7f303a21a640 1 --2- 172.21.15.45:0/3751059672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30340cfcd0 0x7f30340b6390 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.123+0000 7f303a21a640 1 -- 172.21.15.45:0/3751059672 >> 172.21.15.45:0/3751059672 conn(0x7f303406a500 msgr2=0x7f303406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.123+0000 7f303a21a640 1 -- 172.21.15.45:0/3751059672 shutdown_connections 2023-11-28T20:35:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.123+0000 7f303a21a640 1 -- 172.21.15.45:0/3751059672 wait complete. 2023-11-28T20:35:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.125+0000 7f303a21a640 1 Processor -- start 2023-11-28T20:35:31.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.125+0000 7f303a21a640 1 -- start start 2023-11-28T20:35:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f303a21a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340cfcd0 0x7f30340c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f303a21a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30340d00b0 0x7f3034143c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f303a21a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f303415d450 0x7f30341441b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f303a21a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30340c7620 con 0x7f303415d450 2023-11-28T20:35:31.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f303a21a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30340c72d0 con 0x7f30340d00b0 2023-11-28T20:35:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f303a21a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30341601b0 con 0x7f30340cfcd0 2023-11-28T20:35:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f3039a19640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f303415d450 0x7f30341441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f3039a19640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f303415d450 0x7f30341441b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57686/0 (socket says 172.21.15.45:57686) 2023-11-28T20:35:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f3039a19640 1 -- 172.21.15.45:0/4224297460 learned_addr learned my addr 172.21.15.45:0/4224297460 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f3038a17640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30340d00b0 0x7f3034143c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.126+0000 7f3039218640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340cfcd0 0x7f30340c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3039a19640 1 -- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340cfcd0 msgr2=0x7f30340c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3039a19640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340cfcd0 0x7f30340c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3039a19640 1 -- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30340d00b0 msgr2=0x7f3034143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3039a19640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30340d00b0 0x7f3034143c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3039a19640 1 -- 172.21.15.45:0/4224297460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3024009b30 con 0x7f303415d450 2023-11-28T20:35:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3038a17640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30340d00b0 0x7f3034143c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3039218640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340cfcd0 0x7f30340c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3039a19640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f303415d450 0x7f30341441b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f302c00b440 tx=0x7f302c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3031ffb640 1 -- 172.21.15.45:0/4224297460 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302c011830 con 0x7f303415d450 2023-11-28T20:35:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3031ffb640 1 -- 172.21.15.45:0/4224297460 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302c011990 con 0x7f303415d450 2023-11-28T20:35:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3034160490 con 0x7f303415d450 2023-11-28T20:35:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.127+0000 7f3031ffb640 1 -- 172.21.15.45:0/4224297460 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302c011d40 con 0x7f303415d450 2023-11-28T20:35:31.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.128+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30340c9a50 con 0x7f303415d450 2023-11-28T20:35:31.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.129+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30341609b0 con 0x7f303415d450 2023-11-28T20:35:31.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.129+0000 7f3031ffb640 1 -- 172.21.15.45:0/4224297460 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f302c01a400 con 0x7f303415d450 2023-11-28T20:35:31.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.130+0000 7f3031ffb640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f302006ae40 0x7f302006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.130+0000 7f3031ffb640 1 -- 172.21.15.45:0/4224297460 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f302c0891e0 con 0x7f303415d450 2023-11-28T20:35:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.131+0000 7f3039218640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f302006ae40 0x7f302006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.132+0000 7f3039218640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f302006ae40 0x7f302006d300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3024035000 tx=0x7f3024030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.134+0000 7f3031ffb640 1 -- 172.21.15.45:0/4224297460 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f302c054410 con 0x7f303415d450 2023-11-28T20:35:31.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.401+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f3034071b30 con 0x7f303415d450 2023-11-28T20:35:31.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.405+0000 7f3031ffb640 1 -- 172.21.15.45:0/4224297460 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f302c057a30 con 0x7f303415d450 2023-11-28T20:35:31.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f302006ae40 msgr2=0x7f302006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f302006ae40 0x7f302006d300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3024035000 tx=0x7f3024030000 comp rx=0 tx=0).stop 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f303415d450 msgr2=0x7f30341441b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f303415d450 0x7f30341441b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f302c00b440 tx=0x7f302c00b900 comp rx=0 tx=0).stop 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 shutdown_connections 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f302006ae40 0x7f302006d300 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f303415d450 0x7f30341441b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30340d00b0 0x7f3034143c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 --2- 172.21.15.45:0/4224297460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30340cfcd0 0x7f30340c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 >> 172.21.15.45:0/4224297460 conn(0x7f303406a500 msgr2=0x7f30340b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:31.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 shutdown_connections 2023-11-28T20:35:31.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.408+0000 7f303a21a640 1 -- 172.21.15.45:0/4224297460 wait complete. 2023-11-28T20:35:31.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-11-28T20:35:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.653+0000 7ff3cbfff640 1 Processor -- start 2023-11-28T20:35:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.653+0000 7ff3cbfff640 1 -- start start 2023-11-28T20:35:31.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.656+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc141600 0x7ff3cc0cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.656+0000 7ff3cbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3cc0cc0d0 0x7ff3cc15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.656+0000 7ff3cbfff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc15c670 con 0x7ff3cc15be80 2023-11-28T20:35:31.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.656+0000 7ff3cbfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc0cc550 con 0x7ff3cc0cc0d0 2023-11-28T20:35:31.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.656+0000 7ff3cbfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0cda40 con 0x7ff3cc141600 2023-11-28T20:35:31.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.656+0000 7ff3c3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3cc0cc0d0 0x7ff3cc15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.656+0000 7ff3c3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3cc0cc0d0 0x7ff3cc15b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57692/0 (socket says 172.21.15.45:57692) 2023-11-28T20:35:31.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.656+0000 7ff3c3fff640 1 -- 172.21.15.45:0/3080049895 learned_addr learned my addr 172.21.15.45:0/3080049895 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:31.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.656+0000 7ff3caffd640 1 --2- 172.21.15.45:0/3080049895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc141600 0x7ff3cc0cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.657+0000 7ff3c3fff640 1 -- 172.21.15.45:0/3080049895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc141600 msgr2=0x7ff3cc0cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:31.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.657+0000 7ff3c3fff640 1 --2- 172.21.15.45:0/3080049895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc141600 0x7ff3cc0cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.657+0000 7ff3c3fff640 1 -- 172.21.15.45:0/3080049895 >> v1:172.21.15.190:6789/0 conn(0x7ff3cc15be80 legacy=0x7ff3cc15ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.657+0000 7ff3c3fff640 1 -- 172.21.15.45:0/3080049895 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0ccc80 con 0x7ff3cc0cc0d0 2023-11-28T20:35:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.658+0000 7ff3c3fff640 1 --2- 172.21.15.45:0/3080049895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc0cc0d0 0x7ff3cc15b940 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009870 tx=0x7ff3b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da310bd02b5fadf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.658+0000 7ff3ca7fc640 1 -- 172.21.15.45:0/3080049895 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4031640 con 0x7ff3cc0cc0d0 2023-11-28T20:35:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.658+0000 7ff3ca7fc640 1 -- 172.21.15.45:0/3080049895 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b40317a0 con 0x7ff3cc0cc0d0 2023-11-28T20:35:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.658+0000 7ff3ca7fc640 1 -- 172.21.15.45:0/3080049895 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4031b30 con 0x7ff3cc0cc0d0 2023-11-28T20:35:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.659+0000 7ff3cbfff640 1 -- 172.21.15.45:0/3080049895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc0cc0d0 msgr2=0x7ff3cc15b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.659+0000 7ff3cbfff640 1 --2- 172.21.15.45:0/3080049895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc0cc0d0 0x7ff3cc15b940 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009870 tx=0x7ff3b402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.659+0000 7ff3cbfff640 1 -- 172.21.15.45:0/3080049895 shutdown_connections 2023-11-28T20:35:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.659+0000 7ff3cbfff640 1 --2- 172.21.15.45:0/3080049895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc0cc0d0 0x7ff3cc15b940 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.659+0000 7ff3cbfff640 1 --2- 172.21.15.45:0/3080049895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc141600 0x7ff3cc0cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.659+0000 7ff3cbfff640 1 -- 172.21.15.45:0/3080049895 >> 172.21.15.45:0/3080049895 conn(0x7ff3cc0b0e70 msgr2=0x7ff3cc0b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.659+0000 7ff3cbfff640 1 -- 172.21.15.45:0/3080049895 shutdown_connections 2023-11-28T20:35:31.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.659+0000 7ff3cbfff640 1 -- 172.21.15.45:0/3080049895 wait complete. 2023-11-28T20:35:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.661+0000 7ff3cbfff640 1 Processor -- start 2023-11-28T20:35:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.661+0000 7ff3cbfff640 1 -- start start 2023-11-28T20:35:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.662+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3cc0cc0d0 0x7ff3cc13c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.662+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc141600 0x7ff3cc0af380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.662+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc15be80 0x7ff3cc0af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.662+0000 7ff3cbfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc099a40 con 0x7ff3cc141600 2023-11-28T20:35:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.662+0000 7ff3cbfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0b3ad0 con 0x7ff3cc0cc0d0 2023-11-28T20:35:31.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.662+0000 7ff3cbfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0b3db0 con 0x7ff3cc15be80 2023-11-28T20:35:31.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3c3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc141600 0x7ff3cc0af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3c3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc141600 0x7ff3cc0af380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57700/0 (socket says 172.21.15.45:57700) 2023-11-28T20:35:31.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3c3fff640 1 -- 172.21.15.45:0/2097221191 learned_addr learned my addr 172.21.15.45:0/2097221191 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:31.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3caffd640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3cc0cc0d0 0x7ff3cc13c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3cb7fe640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc15be80 0x7ff3cc0af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3c3fff640 1 -- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc15be80 msgr2=0x7ff3cc0af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3c3fff640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc15be80 0x7ff3cc0af8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3c3fff640 1 -- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3cc0cc0d0 msgr2=0x7ff3cc13c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3c3fff640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3cc0cc0d0 0x7ff3cc13c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3c3fff640 1 -- 172.21.15.45:0/2097221191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b40094d0 con 0x7ff3cc141600 2023-11-28T20:35:31.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.663+0000 7ff3cb7fe640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc15be80 0x7ff3cc0af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:31.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.664+0000 7ff3c3fff640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc141600 0x7ff3cc0af380 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009870 tx=0x7ff3b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:31.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.664+0000 7ff3c8ff9640 1 -- 172.21.15.45:0/2097221191 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b403ac60 con 0x7ff3cc141600 2023-11-28T20:35:31.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.664+0000 7ff3c8ff9640 1 -- 172.21.15.45:0/2097221191 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b403adc0 con 0x7ff3cc141600 2023-11-28T20:35:31.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.664+0000 7ff3c8ff9640 1 -- 172.21.15.45:0/2097221191 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4044630 con 0x7ff3cc141600 2023-11-28T20:35:31.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.664+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc0b4090 con 0x7ff3cc141600 2023-11-28T20:35:31.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.664+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc13ce60 con 0x7ff3cc141600 2023-11-28T20:35:31.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.665+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc099f20 con 0x7ff3cc141600 2023-11-28T20:35:31.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.666+0000 7ff3c8ff9640 1 -- 172.21.15.45:0/2097221191 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff3b402f050 con 0x7ff3cc141600 2023-11-28T20:35:31.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.667+0000 7ff3c8ff9640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff3ac06ae70 0x7ff3ac06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:31.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.668+0000 7ff3c8ff9640 1 -- 172.21.15.45:0/2097221191 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff3b40ab370 con 0x7ff3cc141600 2023-11-28T20:35:31.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.668+0000 7ff3caffd640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff3ac06ae70 0x7ff3ac06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:31.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.669+0000 7ff3caffd640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff3ac06ae70 0x7ff3ac06d330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006fd0 tx=0x7ff3bc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:31.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.672+0000 7ff3c8ff9640 1 -- 172.21.15.45:0/2097221191 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3cc099f20 con 0x7ff3cc141600 2023-11-28T20:35:31.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.936+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7ff3cc099f20 con 0x7ff3cc141600 2023-11-28T20:35:31.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.942+0000 7ff3c8ff9640 1 -- 172.21.15.45:0/2097221191 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff3cc099f20 con 0x7ff3cc141600 2023-11-28T20:35:31.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff3ac06ae70 msgr2=0x7ff3ac06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff3ac06ae70 0x7ff3ac06d330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006fd0 tx=0x7ff3bc008040 comp rx=0 tx=0).stop 2023-11-28T20:35:31.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc141600 msgr2=0x7ff3cc0af380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:31.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc141600 0x7ff3cc0af380 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009870 tx=0x7ff3b40046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 shutdown_connections 2023-11-28T20:35:31.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7ff3ac06ae70 0x7ff3ac06d330 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3cc15be80 0x7ff3cc0af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3cc141600 0x7ff3cc0af380 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 --2- 172.21.15.45:0/2097221191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3cc0cc0d0 0x7ff3cc13c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:31.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 >> 172.21.15.45:0/2097221191 conn(0x7ff3cc0b0e70 msgr2=0x7ff3cc15a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:31.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 shutdown_connections 2023-11-28T20:35:31.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:31.944+0000 7ff3cbfff640 1 -- 172.21.15.45:0/2097221191 wait complete. 2023-11-28T20:35:31.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:32.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.180+0000 7f0bf3fff640 1 Processor -- start 2023-11-28T20:35:32.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.180+0000 7f0bf3fff640 1 -- start start 2023-11-28T20:35:32.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.182+0000 7f0bf3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf406d880 0x7f0bf4164c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.182+0000 7f0bf3fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf415e680 con 0x7f0bf40721e0 2023-11-28T20:35:32.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.182+0000 7f0bf3fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf415e960 con 0x7f0bf40b3870 2023-11-28T20:35:32.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.182+0000 7f0bf3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf415a460 con 0x7f0bf406d880 2023-11-28T20:35:32.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.182+0000 7f0bea7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0bf40721e0 0x7f0bf406d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39280/0 (socket says 172.21.15.45:39280) 2023-11-28T20:35:32.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.182+0000 7f0bea7fc640 1 -- 172.21.15.45:0/3026922118 learned_addr learned my addr 172.21.15.45:0/3026922118 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:32.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121362920 0 0) 0x7f0bf415e680 con 0x7f0bf40721e0 2023-11-28T20:35:32.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be0003520 con 0x7f0bf40721e0 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf37fe640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf406d880 0x7f0bf4164c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183003364 0 0) 0x7f0be0003520 con 0x7f0bf40721e0 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf406d880 msgr2=0x7f0bf4164c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf406d880 0x7f0bf4164c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 >> v1:172.21.15.190:6789/0 conn(0x7f0bf40b3870 legacy=0x7f0bf415db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf4071700 con 0x7f0bf40721e0 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f0bdc001670 con 0x7f0bf40721e0 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 >> v1:172.21.15.45:6789/0 conn(0x7f0bf40721e0 legacy=0x7f0bf406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.183+0000 7f0bf27fc640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be00039a0 0x7f0be0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf27fc640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf406d880 0x7f0be00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf27fc640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0be0004d00 0x7f0be0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be000dd70 con 0x7f0be00039a0 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be000dee0 con 0x7f0be0004d00 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be000e0a0 con 0x7f0bf406d880 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf2ffd640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be00039a0 0x7f0be0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf2ffd640 1 -- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf406d880 msgr2=0x7f0be00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf2ffd640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf406d880 0x7f0be00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf2ffd640 1 -- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0be0004d00 msgr2=0x7f0be0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf37fe640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0be0004d00 0x7f0be0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf2ffd640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0be0004d00 0x7f0be0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.184+0000 7f0bf2ffd640 1 -- 172.21.15.45:0/3026922118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be0003520 con 0x7f0be00039a0 2023-11-28T20:35:32.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.185+0000 7f0bf2ffd640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be00039a0 0x7f0be0003dc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0be4001e50 tx=0x7f0be00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9dfd257a3584f3b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:32.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.186+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4003b70 con 0x7f0be00039a0 2023-11-28T20:35:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.186+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be4003cd0 con 0x7f0be00039a0 2023-11-28T20:35:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.186+0000 7f0bf27fc640 1 -- 172.21.15.45:0/3026922118 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4005310 con 0x7f0be00039a0 2023-11-28T20:35:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.186+0000 7f0bf3fff640 1 -- 172.21.15.45:0/3026922118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be00039a0 msgr2=0x7f0be0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.186+0000 7f0bf3fff640 1 --2- 172.21.15.45:0/3026922118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be00039a0 0x7f0be0003dc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0be4001e50 tx=0x7f0be00057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.186+0000 7f0bf37fe640 1 -- 172.21.15.45:0/3026922118 reap_dead start 2023-11-28T20:35:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.186+0000 7f0bf3fff640 1 -- 172.21.15.45:0/3026922118 shutdown_connections 2023-11-28T20:35:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.186+0000 7f0bf3fff640 1 -- 172.21.15.45:0/3026922118 >> 172.21.15.45:0/3026922118 conn(0x7f0bf406ab30 msgr2=0x7f0bf40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.186+0000 7f0bf3fff640 1 -- 172.21.15.45:0/3026922118 shutdown_connections 2023-11-28T20:35:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.187+0000 7f0bf3fff640 1 -- 172.21.15.45:0/3026922118 wait complete. 2023-11-28T20:35:32.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.188+0000 7f0bf3fff640 1 Processor -- start 2023-11-28T20:35:32.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.188+0000 7f0bf3fff640 1 -- start start 2023-11-28T20:35:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.189+0000 7f0bf3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0be00039a0 0x7f0bf4151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.189+0000 7f0bf3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf40cc480 0x7f0bf40cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.190+0000 7f0bf3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf40b22c0 0x7f0bf40b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.190+0000 7f0bf3fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf41520d0 con 0x7f0bf40b22c0 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.190+0000 7f0bf3fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40b2c80 con 0x7f0be00039a0 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.190+0000 7f0bf3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40ccfd0 con 0x7f0bf40cc480 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.190+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf40b22c0 0x7f0bf40b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.190+0000 7f0bf2ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0be00039a0 0x7f0bf4151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.190+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf40b22c0 0x7f0bf40b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57718/0 (socket says 172.21.15.45:57718) 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.190+0000 7f0bf37fe640 1 -- 172.21.15.45:0/1496861510 learned_addr learned my addr 172.21.15.45:0/1496861510 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.190+0000 7f0bea7fc640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf40cc480 0x7f0bf40cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf37fe640 1 -- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf40cc480 msgr2=0x7f0bf40cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf37fe640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf40cc480 0x7f0bf40cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf37fe640 1 -- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0be00039a0 msgr2=0x7f0bf4151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf37fe640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0be00039a0 0x7f0bf4151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf37fe640 1 -- 172.21.15.45:0/1496861510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf40cd2b0 con 0x7f0bf40b22c0 2023-11-28T20:35:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf37fe640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf40b22c0 0x7f0bf40b2740 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40ce860 tx=0x7f0bec00e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf0ff9640 1 -- 172.21.15.45:0/1496861510 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec00c070 con 0x7f0bf40b22c0 2023-11-28T20:35:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf0ff9640 1 -- 172.21.15.45:0/1496861510 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec0136f0 con 0x7f0bf40b22c0 2023-11-28T20:35:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf0ff9640 1 -- 172.21.15.45:0/1496861510 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec013aa0 con 0x7f0bf40b22c0 2023-11-28T20:35:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf40cd590 con 0x7f0bf40b22c0 2023-11-28T20:35:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.191+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf40cdb50 con 0x7f0bf40b22c0 2023-11-28T20:35:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.193+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf4071f00 con 0x7f0bf40b22c0 2023-11-28T20:35:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.194+0000 7f0bf0ff9640 1 -- 172.21.15.45:0/1496861510 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0bec013c00 con 0x7f0bf40b22c0 2023-11-28T20:35:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.194+0000 7f0bf0ff9640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0bd406ae40 0x7f0bd406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.195+0000 7f0bf0ff9640 1 -- 172.21.15.45:0/1496861510 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f0bec088b10 con 0x7f0bf40b22c0 2023-11-28T20:35:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.195+0000 7f0bf2ffd640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0bd406ae40 0x7f0bd406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.196+0000 7f0bf2ffd640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0bd406ae40 0x7f0bd406d300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0be4034000 tx=0x7f0be403b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.199+0000 7f0bf0ff9640 1 -- 172.21.15.45:0/1496861510 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bec053df0 con 0x7f0bf40b22c0 2023-11-28T20:35:32.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.397+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bf409d0f0 con 0x7f0bf40b22c0 2023-11-28T20:35:32.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.399+0000 7f0bf0ff9640 1 -- 172.21.15.45:0/1496861510 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+13050 (secure 0 0 0) 0x7f0bec085310 con 0x7f0bf40b22c0 2023-11-28T20:35:32.400 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:32.401 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":35,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:35:28.244050+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0bd406ae40 msgr2=0x7f0bd406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0bd406ae40 0x7f0bd406d300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0be4034000 tx=0x7f0be403b040 comp rx=0 tx=0).stop 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf40b22c0 msgr2=0x7f0bf40b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf40b22c0 0x7f0bf40b2740 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40ce860 tx=0x7f0bec00e8b0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 shutdown_connections 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0bd406ae40 0x7f0bd406d300 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf40b22c0 0x7f0bf40b2740 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bf40cc480 0x7f0bf40cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 --2- 172.21.15.45:0/1496861510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0be00039a0 0x7f0bf4151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 >> 172.21.15.45:0/1496861510 conn(0x7f0bf406ab30 msgr2=0x7f0bf40ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 shutdown_connections 2023-11-28T20:35:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.402+0000 7f0bf3fff640 1 -- 172.21.15.45:0/1496861510 wait complete. 2023-11-28T20:35:32.424 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:35:32.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:35:32.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.650+0000 7f07cb052640 1 Processor -- start 2023-11-28T20:35:32.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.650+0000 7f07cb052640 1 -- start start 2023-11-28T20:35:32.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.652+0000 7f07cb052640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f07c415d4b0 0x7f07c4070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.652+0000 7f07cb052640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07c40adfa0 0x7f07c40717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.652+0000 7f07cb052640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c40708d0 con 0x7f07c40712a0 2023-11-28T20:35:32.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.652+0000 7f07cb052640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c40b3bb0 con 0x7f07c40adfa0 2023-11-28T20:35:32.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.652+0000 7f07cb052640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c40b3e90 con 0x7f07c415d4b0 2023-11-28T20:35:32.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.653+0000 7f07ca851640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07c40adfa0 0x7f07c40717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.653+0000 7f07c984f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07c40712a0 0x7f07c40abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39292/0 (socket says 172.21.15.45:39292) 2023-11-28T20:35:32.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.653+0000 7f07c984f640 1 -- 172.21.15.45:0/4074278895 learned_addr learned my addr 172.21.15.45:0/4074278895 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:32.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.653+0000 7f07ca050640 1 --2- 172.21.15.45:0/4074278895 >> v2:172.21.15.190:3300/0 conn(0x7f07c415d4b0 0x7f07c4070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.653+0000 7f07ca851640 1 -- 172.21.15.45:0/4074278895 >> v2:172.21.15.190:3300/0 conn(0x7f07c415d4b0 msgr2=0x7f07c4070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:32.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.653+0000 7f07ca851640 1 --2- 172.21.15.45:0/4074278895 >> v2:172.21.15.190:3300/0 conn(0x7f07c415d4b0 0x7f07c4070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.653+0000 7f07ca851640 1 -- 172.21.15.45:0/4074278895 >> v1:172.21.15.45:6789/0 conn(0x7f07c40712a0 legacy=0x7f07c40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.653+0000 7f07ca851640 1 -- 172.21.15.45:0/4074278895 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4070280 con 0x7f07c40adfa0 2023-11-28T20:35:32.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.653+0000 7f07c904e640 1 -- 172.21.15.45:0/4074278895 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743779089 0 0) 0x7f07c40708d0 con 0x7f07c40712a0 2023-11-28T20:35:32.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.655+0000 7f07ca851640 1 --2- 172.21.15.45:0/4074278895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c40adfa0 0x7f07c40717e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f07c40ce4a0 tx=0x7f07bc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad81dfa39d1f23a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:32.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.655+0000 7f07c904e640 1 -- 172.21.15.45:0/4074278895 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc02cbf0 con 0x7f07c40adfa0 2023-11-28T20:35:32.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.655+0000 7f07c904e640 1 -- 172.21.15.45:0/4074278895 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07bc02cd50 con 0x7f07c40adfa0 2023-11-28T20:35:32.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.655+0000 7f07c904e640 1 -- 172.21.15.45:0/4074278895 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc034660 con 0x7f07c40adfa0 2023-11-28T20:35:32.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.655+0000 7f07cb052640 1 -- 172.21.15.45:0/4074278895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c40adfa0 msgr2=0x7f07c40717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.655+0000 7f07cb052640 1 --2- 172.21.15.45:0/4074278895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c40adfa0 0x7f07c40717e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f07c40ce4a0 tx=0x7f07bc02be40 comp rx=0 tx=0).stop 2023-11-28T20:35:32.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.656+0000 7f07cb052640 1 -- 172.21.15.45:0/4074278895 shutdown_connections 2023-11-28T20:35:32.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.656+0000 7f07cb052640 1 --2- 172.21.15.45:0/4074278895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c40adfa0 0x7f07c40717e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.656+0000 7f07cb052640 1 --2- 172.21.15.45:0/4074278895 >> v2:172.21.15.190:3300/0 conn(0x7f07c415d4b0 0x7f07c4070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.656+0000 7f07cb052640 1 -- 172.21.15.45:0/4074278895 >> 172.21.15.45:0/4074278895 conn(0x7f07c40b1050 msgr2=0x7f07c40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:32.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.656+0000 7f07cb052640 1 -- 172.21.15.45:0/4074278895 shutdown_connections 2023-11-28T20:35:32.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.656+0000 7f07cb052640 1 -- 172.21.15.45:0/4074278895 wait complete. 2023-11-28T20:35:32.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.657+0000 7f07cb052640 1 Processor -- start 2023-11-28T20:35:32.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.658+0000 7f07cb052640 1 -- start start 2023-11-28T20:35:32.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07cb052640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07c40712a0 0x7f07c4147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07cb052640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07c40adfa0 0x7f07c415d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07cb052640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c415d4b0 0x7f07c4143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07cb052640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c4160140 con 0x7f07c415d4b0 2023-11-28T20:35:32.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07cb052640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c415f570 con 0x7f07c40adfa0 2023-11-28T20:35:32.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07cb052640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c415de60 con 0x7f07c40712a0 2023-11-28T20:35:32.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07ca851640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c415d4b0 0x7f07c4143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07ca851640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c415d4b0 0x7f07c4143e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57734/0 (socket says 172.21.15.45:57734) 2023-11-28T20:35:32.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07ca851640 1 -- 172.21.15.45:0/266409043 learned_addr learned my addr 172.21.15.45:0/266409043 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:32.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07ca050640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07c40712a0 0x7f07c4147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.659+0000 7f07c984f640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07c40adfa0 0x7f07c415d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07ca851640 1 -- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07c40712a0 msgr2=0x7f07c4147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07ca851640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07c40712a0 0x7f07c4147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07ca851640 1 -- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07c40adfa0 msgr2=0x7f07c415d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07ca851640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07c40adfa0 0x7f07c415d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07ca851640 1 -- 172.21.15.45:0/266409043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc009a70 con 0x7f07c415d4b0 2023-11-28T20:35:32.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07c984f640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07c40adfa0 0x7f07c415d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:32.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07ca851640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c415d4b0 0x7f07c4143e10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009e80 tx=0x7f07bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:32.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07baffd640 1 -- 172.21.15.45:0/266409043 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc03e610 con 0x7f07c415d4b0 2023-11-28T20:35:32.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07baffd640 1 -- 172.21.15.45:0/266409043 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07bc03e770 con 0x7f07c415d4b0 2023-11-28T20:35:32.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07baffd640 1 -- 172.21.15.45:0/266409043 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc03eb00 con 0x7f07c415d4b0 2023-11-28T20:35:32.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c415f7a0 con 0x7f07c415d4b0 2023-11-28T20:35:32.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.660+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c415fd60 con 0x7f07c415d4b0 2023-11-28T20:35:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.662+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c4160280 con 0x7f07c415d4b0 2023-11-28T20:35:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.663+0000 7f07baffd640 1 -- 172.21.15.45:0/266409043 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f07bc03b050 con 0x7f07c415d4b0 2023-11-28T20:35:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.663+0000 7f07baffd640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f07a806ae40 0x7f07a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.664+0000 7f07baffd640 1 -- 172.21.15.45:0/266409043 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f07bc0abcf0 con 0x7f07c415d4b0 2023-11-28T20:35:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.664+0000 7f07ca050640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f07a806ae40 0x7f07a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:32.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.665+0000 7f07ca050640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f07a806ae40 0x7f07a806d300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f07c0006fd0 tx=0x7f07c0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:32.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.668+0000 7f07baffd640 1 -- 172.21.15.45:0/266409043 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07c4160280 con 0x7f07c415d4b0 2023-11-28T20:35:32.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.860+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f07c4160280 con 0x7f07c415d4b0 2023-11-28T20:35:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.860+0000 7f07baffd640 1 -- 172.21.15.45:0/266409043 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f07bc039020 con 0x7f07c415d4b0 2023-11-28T20:35:32.863 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:35:32.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f07a806ae40 msgr2=0x7f07a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f07a806ae40 0x7f07a806d300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f07c0006fd0 tx=0x7f07c0008040 comp rx=0 tx=0).stop 2023-11-28T20:35:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c415d4b0 msgr2=0x7f07c4143e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c415d4b0 0x7f07c4143e10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009e80 tx=0x7f07bc004a40 comp rx=0 tx=0).stop 2023-11-28T20:35:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 shutdown_connections 2023-11-28T20:35:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f07a806ae40 0x7f07a806d300 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c415d4b0 0x7f07c4143e10 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07c40adfa0 0x7f07c415d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 --2- 172.21.15.45:0/266409043 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07c40712a0 0x7f07c4147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:32.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 >> 172.21.15.45:0/266409043 conn(0x7f07c40b1050 msgr2=0x7f07c40ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:32.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 shutdown_connections 2023-11-28T20:35:32.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:32.863+0000 7f07cb052640 1 -- 172.21.15.45:0/266409043 wait complete. 2023-11-28T20:35:32.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T20:35:33.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.099+0000 7f2c416bb640 1 Processor -- start 2023-11-28T20:35:33.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.099+0000 7f2c416bb640 1 -- start start 2023-11-28T20:35:33.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.100+0000 7f2c416bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 0x7f2c3c0b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.100+0000 7f2c416bb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c0ce710 con 0x7f2c3c071190 2023-11-28T20:35:33.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.100+0000 7f2c416bb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c0ae2c0 con 0x7f2c3c15d320 2023-11-28T20:35:33.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.100+0000 7f2c416bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0ae5a0 con 0x7f2c3c0b0c30 2023-11-28T20:35:33.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.101+0000 7f2c3b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2c3c071190 0x7f2c3c071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39306/0 (socket says 172.21.15.45:39306) 2023-11-28T20:35:33.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.101+0000 7f2c3b7fe640 1 -- 172.21.15.45:0/1189780619 learned_addr learned my addr 172.21.15.45:0/1189780619 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:33.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.101+0000 7f2c3a7fc640 1 --2- 172.21.15.45:0/1189780619 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 0x7f2c3c0b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.102+0000 7f2c39ffb640 1 -- 172.21.15.45:0/1189780619 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711023588 0 0) 0x7f2c3c0ce710 con 0x7f2c3c071190 2023-11-28T20:35:33.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.102+0000 7f2c39ffb640 1 -- 172.21.15.45:0/1189780619 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c28003520 con 0x7f2c3c071190 2023-11-28T20:35:33.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.102+0000 7f2c39ffb640 1 -- 172.21.15.45:0/1189780619 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192646783 0 0) 0x7f2c3c0ae2c0 con 0x7f2c3c15d320 2023-11-28T20:35:33.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.102+0000 7f2c3a7fc640 1 -- 172.21.15.45:0/1189780619 >> v1:172.21.15.190:6789/0 conn(0x7f2c3c15d320 legacy=0x7f2c3c0ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.102+0000 7f2c3a7fc640 1 -- 172.21.15.45:0/1189780619 >> v1:172.21.15.45:6789/0 conn(0x7f2c3c071190 legacy=0x7f2c3c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.102+0000 7f2c3a7fc640 1 -- 172.21.15.45:0/1189780619 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c1611f0 con 0x7f2c3c0b0c30 2023-11-28T20:35:33.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.103+0000 7f2c3a7fc640 1 --2- 172.21.15.45:0/1189780619 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 0x7f2c3c0b10b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c24009dc0 tx=0x7f2c2402bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b603a21c790439a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:33.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.104+0000 7f2c39ffb640 1 -- 172.21.15.45:0/1189780619 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c340015f0 con 0x7f2c3c0b0c30 2023-11-28T20:35:33.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.104+0000 7f2c39ffb640 1 -- 172.21.15.45:0/1189780619 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2402c840 con 0x7f2c3c0b0c30 2023-11-28T20:35:33.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.104+0000 7f2c39ffb640 1 -- 172.21.15.45:0/1189780619 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2402cbd0 con 0x7f2c3c0b0c30 2023-11-28T20:35:33.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.104+0000 7f2c416bb640 1 -- 172.21.15.45:0/1189780619 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 msgr2=0x7f2c3c0b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.104+0000 7f2c416bb640 1 --2- 172.21.15.45:0/1189780619 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 0x7f2c3c0b10b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c24009dc0 tx=0x7f2c2402bd10 comp rx=0 tx=0).stop 2023-11-28T20:35:33.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.105+0000 7f2c416bb640 1 -- 172.21.15.45:0/1189780619 shutdown_connections 2023-11-28T20:35:33.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.105+0000 7f2c416bb640 1 --2- 172.21.15.45:0/1189780619 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 0x7f2c3c0b10b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.105+0000 7f2c416bb640 1 -- 172.21.15.45:0/1189780619 >> 172.21.15.45:0/1189780619 conn(0x7f2c3c06a500 msgr2=0x7f2c3c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:33.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.105+0000 7f2c416bb640 1 -- 172.21.15.45:0/1189780619 shutdown_connections 2023-11-28T20:35:33.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.105+0000 7f2c416bb640 1 -- 172.21.15.45:0/1189780619 wait complete. 2023-11-28T20:35:33.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.106+0000 7f2c416bb640 1 Processor -- start 2023-11-28T20:35:33.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.107+0000 7f2c416bb640 1 -- start start 2023-11-28T20:35:33.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.107+0000 7f2c416bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c3c071190 0x7f2c3c1e8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.107+0000 7f2c416bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 0x7f2c3c0b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.107+0000 7f2c416bb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c15d320 0x7f2c3c0b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c416bb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c1f45c0 con 0x7f2c3c15d320 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c416bb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c1f42e0 con 0x7f2c3c071190 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c416bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c1f48c0 con 0x7f2c3c0b0c30 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c3b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c15d320 0x7f2c3c0b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c3b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c15d320 0x7f2c3c0b38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57742/0 (socket says 172.21.15.45:57742) 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c3b7fe640 1 -- 172.21.15.45:0/3113186492 learned_addr learned my addr 172.21.15.45:0/3113186492 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c3affd640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c3c071190 0x7f2c3c1e8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c3b7fe640 1 -- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 msgr2=0x7f2c3c0b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c3b7fe640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 0x7f2c3c0b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c3b7fe640 1 -- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c3c071190 msgr2=0x7f2c3c1e8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c3b7fe640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c3c071190 0x7f2c3c1e8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.108+0000 7f2c3b7fe640 1 -- 172.21.15.45:0/3113186492 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c24009a70 con 0x7f2c3c15d320 2023-11-28T20:35:33.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.109+0000 7f2c3b7fe640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c15d320 0x7f2c3c0b38c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c3400c780 tx=0x7f2c3400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:33.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.109+0000 7f2c337fe640 1 -- 172.21.15.45:0/3113186492 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c340074e0 con 0x7f2c3c15d320 2023-11-28T20:35:33.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.109+0000 7f2c337fe640 1 -- 172.21.15.45:0/3113186492 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c34007660 con 0x7f2c3c15d320 2023-11-28T20:35:33.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.109+0000 7f2c337fe640 1 -- 172.21.15.45:0/3113186492 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34004a00 con 0x7f2c3c15d320 2023-11-28T20:35:33.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.109+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c0ae090 con 0x7f2c3c15d320 2023-11-28T20:35:33.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.109+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c1f4e80 con 0x7f2c3c15d320 2023-11-28T20:35:33.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.110+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c0b0a10 con 0x7f2c3c15d320 2023-11-28T20:35:33.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.111+0000 7f2c337fe640 1 -- 172.21.15.45:0/3113186492 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2c34004b60 con 0x7f2c3c15d320 2023-11-28T20:35:33.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.112+0000 7f2c337fe640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2c1c06ae40 0x7f2c1c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.112+0000 7f2c337fe640 1 -- 172.21.15.45:0/3113186492 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f2c340887a0 con 0x7f2c3c15d320 2023-11-28T20:35:33.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.113+0000 7f2c3affd640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2c1c06ae40 0x7f2c1c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.114+0000 7f2c3affd640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2c1c06ae40 0x7f2c1c06d300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c007fd0 tx=0x7f2c2c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:33.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.118+0000 7f2c337fe640 1 -- 172.21.15.45:0/3113186492 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c34053a50 con 0x7f2c3c15d320 2023-11-28T20:35:33.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.323+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2c3c1f5070 con 0x7f2c3c15d320 2023-11-28T20:35:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.324+0000 7f2c337fe640 1 -- 172.21.15.45:0/3113186492 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2c34018090 con 0x7f2c3c15d320 2023-11-28T20:35:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:33.326 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:35:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T20:35:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T20:35:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:33.327 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:35:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.326+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2c1c06ae40 msgr2=0x7f2c1c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.326+0000 7f2c416bb640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2c1c06ae40 0x7f2c1c06d300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c007fd0 tx=0x7f2c2c009040 comp rx=0 tx=0).stop 2023-11-28T20:35:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.326+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c15d320 msgr2=0x7f2c3c0b38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.326+0000 7f2c416bb640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c15d320 0x7f2c3c0b38c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c3400c780 tx=0x7f2c3400cc40 comp rx=0 tx=0).stop 2023-11-28T20:35:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.327+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 shutdown_connections 2023-11-28T20:35:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.327+0000 7f2c416bb640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f2c1c06ae40 0x7f2c1c06d300 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.327+0000 7f2c416bb640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c3c15d320 0x7f2c3c0b38c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.327+0000 7f2c416bb640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c3c0b0c30 0x7f2c3c0b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.327+0000 7f2c416bb640 1 --2- 172.21.15.45:0/3113186492 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c3c071190 0x7f2c3c1e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.327+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 >> 172.21.15.45:0/3113186492 conn(0x7f2c3c06a500 msgr2=0x7f2c3c0b0300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.327+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 shutdown_connections 2023-11-28T20:35:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.327+0000 7f2c416bb640 1 -- 172.21.15.45:0/3113186492 wait complete. 2023-11-28T20:35:33.348 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T20:35:33.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T20:35:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.575+0000 7f25119d5640 1 Processor -- start 2023-11-28T20:35:33.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.575+0000 7f25119d5640 1 -- start start 2023-11-28T20:35:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.577+0000 7f25119d5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c06ac10 0x7f250c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.577+0000 7f25119d5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f250c15f720 0x7f250c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.577+0000 7f25119d5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c15ecd0 con 0x7f250c15e840 2023-11-28T20:35:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.577+0000 7f25119d5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250c071ce0 con 0x7f250c15f720 2023-11-28T20:35:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.577+0000 7f25119d5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250c15fbd0 con 0x7f250c06ac10 2023-11-28T20:35:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.578+0000 7f250b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f250c15f720 0x7f250c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.578+0000 7f250affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f250c15e840 0x7f250c06a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39316/0 (socket says 172.21.15.45:39316) 2023-11-28T20:35:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.578+0000 7f250affd640 1 -- 172.21.15.45:0/1171076215 learned_addr learned my addr 172.21.15.45:0/1171076215 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.578+0000 7f250a7fc640 1 --2- 172.21.15.45:0/1171076215 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c06ac10 0x7f250c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.578+0000 7f250b7fe640 1 -- 172.21.15.45:0/1171076215 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c06ac10 msgr2=0x7f250c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.578+0000 7f250b7fe640 1 --2- 172.21.15.45:0/1171076215 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c06ac10 0x7f250c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.578+0000 7f250b7fe640 1 -- 172.21.15.45:0/1171076215 >> v1:172.21.15.45:6789/0 conn(0x7f250c15e840 legacy=0x7f250c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.578+0000 7f2509ffb640 1 -- 172.21.15.45:0/1171076215 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051921656 0 0) 0x7f250c15ecd0 con 0x7f250c15e840 2023-11-28T20:35:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.578+0000 7f250b7fe640 1 -- 172.21.15.45:0/1171076215 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c15f290 con 0x7f250c15f720 2023-11-28T20:35:33.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.579+0000 7f250b7fe640 1 --2- 172.21.15.45:0/1171076215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15f720 0x7f250c0717a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2504009dc0 tx=0x7f250402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1c8114364e3fb2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.580+0000 7f2509ffb640 1 -- 172.21.15.45:0/1171076215 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250402cc40 con 0x7f250c15f720 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.580+0000 7f2509ffb640 1 -- 172.21.15.45:0/1171076215 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250402cda0 con 0x7f250c15f720 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.580+0000 7f2509ffb640 1 -- 172.21.15.45:0/1171076215 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504034660 con 0x7f250c15f720 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.580+0000 7f25119d5640 1 -- 172.21.15.45:0/1171076215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15f720 msgr2=0x7f250c0717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.580+0000 7f25119d5640 1 --2- 172.21.15.45:0/1171076215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15f720 0x7f250c0717a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2504009dc0 tx=0x7f250402be90 comp rx=0 tx=0).stop 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.580+0000 7f25119d5640 1 -- 172.21.15.45:0/1171076215 shutdown_connections 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.580+0000 7f25119d5640 1 --2- 172.21.15.45:0/1171076215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15f720 0x7f250c0717a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.580+0000 7f25119d5640 1 --2- 172.21.15.45:0/1171076215 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c06ac10 0x7f250c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.580+0000 7f25119d5640 1 -- 172.21.15.45:0/1171076215 >> 172.21.15.45:0/1171076215 conn(0x7f250c0ced60 msgr2=0x7f250c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.581+0000 7f25119d5640 1 -- 172.21.15.45:0/1171076215 shutdown_connections 2023-11-28T20:35:33.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.581+0000 7f25119d5640 1 -- 172.21.15.45:0/1171076215 wait complete. 2023-11-28T20:35:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.582+0000 7f25119d5640 1 Processor -- start 2023-11-28T20:35:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.583+0000 7f25119d5640 1 -- start start 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f25119d5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f250c06ac10 0x7f250c0aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f25119d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15e840 0x7f250c151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f25119d5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c15f720 0x7f250c152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f25119d5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c152b60 con 0x7f250c15e840 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f25119d5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c0b2750 con 0x7f250c06ac10 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f25119d5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250c0b1b40 con 0x7f250c15f720 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f250a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15e840 0x7f250c151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f250a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15e840 0x7f250c151de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57762/0 (socket says 172.21.15.45:57762) 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f250a7fc640 1 -- 172.21.15.45:0/3019164875 learned_addr learned my addr 172.21.15.45:0/3019164875 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f250a7fc640 1 -- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c15f720 msgr2=0x7f250c152320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f250b7fe640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c15f720 0x7f250c152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.584+0000 7f250affd640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f250c06ac10 0x7f250c0aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f250a7fc640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c15f720 0x7f250c152320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f250a7fc640 1 -- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f250c06ac10 msgr2=0x7f250c0aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f250a7fc640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f250c06ac10 0x7f250c0aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f250a7fc640 1 -- 172.21.15.45:0/3019164875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504009a70 con 0x7f250c15e840 2023-11-28T20:35:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f250b7fe640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c15f720 0x7f250c152320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f250affd640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f250c06ac10 0x7f250c0aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f250a7fc640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15e840 0x7f250c151de0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f250000d800 tx=0x7f250000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:33.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f24f3fff640 1 -- 172.21.15.45:0/3019164875 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000b6c0 con 0x7f250c15e840 2023-11-28T20:35:33.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f24f3fff640 1 -- 172.21.15.45:0/3019164875 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250000b820 con 0x7f250c15e840 2023-11-28T20:35:33.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f24f3fff640 1 -- 172.21.15.45:0/3019164875 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000bbb0 con 0x7f250c15e840 2023-11-28T20:35:33.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250c0b1e20 con 0x7f250c15e840 2023-11-28T20:35:33.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.585+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250c0b23e0 con 0x7f250c15e840 2023-11-28T20:35:33.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.586+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c0b4ca0 con 0x7f250c15e840 2023-11-28T20:35:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.587+0000 7f24f3fff640 1 -- 172.21.15.45:0/3019164875 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f250000bd10 con 0x7f250c15e840 2023-11-28T20:35:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.588+0000 7f24f3fff640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f24f806ae40 0x7f24f806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.588+0000 7f24f3fff640 1 -- 172.21.15.45:0/3019164875 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f25000899b0 con 0x7f250c15e840 2023-11-28T20:35:33.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.589+0000 7f250affd640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f24f806ae40 0x7f24f806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:33.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.590+0000 7f250affd640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f24f806ae40 0x7f24f806d300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24fc000f60 tx=0x7f24fc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:33.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.593+0000 7f24f3fff640 1 -- 172.21.15.45:0/3019164875 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2500054c60 con 0x7f250c15e840 2023-11-28T20:35:33.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:33.791+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f250c0b28c0 con 0x7f250c15e840 2023-11-28T20:35:34.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.258+0000 7f24f3fff640 1 -- 172.21.15.45:0/3019164875 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2500058280 con 0x7f250c15e840 2023-11-28T20:35:34.261 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T20:35:34.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.262+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f24f806ae40 msgr2=0x7f24f806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:34.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.262+0000 7f25119d5640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f24f806ae40 0x7f24f806d300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24fc000f60 tx=0x7f24fc009040 comp rx=0 tx=0).stop 2023-11-28T20:35:34.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.262+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15e840 msgr2=0x7f250c151de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:34.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.262+0000 7f25119d5640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15e840 0x7f250c151de0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f250000d800 tx=0x7f250000dcc0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.263+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 shutdown_connections 2023-11-28T20:35:34.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.263+0000 7f25119d5640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f24f806ae40 0x7f24f806d300 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.263+0000 7f25119d5640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f250c15f720 0x7f250c152320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.263+0000 7f25119d5640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f250c15e840 0x7f250c151de0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.263+0000 7f25119d5640 1 --2- 172.21.15.45:0/3019164875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f250c06ac10 0x7f250c0aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.263+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 >> 172.21.15.45:0/3019164875 conn(0x7f250c0ced60 msgr2=0x7f250c070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.263+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 shutdown_connections 2023-11-28T20:35:34.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.265+0000 7f25119d5640 1 -- 172.21.15.45:0/3019164875 wait complete. 2023-11-28T20:35:34.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T20:35:34.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.507+0000 7f1a6f59e640 1 Processor -- start 2023-11-28T20:35:34.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.507+0000 7f1a6f59e640 1 -- start start 2023-11-28T20:35:34.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.509+0000 7f1a6f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1a7015d010 0x7f1a7006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:34.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.509+0000 7f1a6f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7006e3b0 con 0x7f1a700661f0 2023-11-28T20:35:34.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.509+0000 7f1a6f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a700b2e40 con 0x7f1a7006df20 2023-11-28T20:35:34.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.509+0000 7f1a6f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a700b5200 con 0x7f1a7015d010 2023-11-28T20:35:34.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.509+0000 7f1a6ed9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1a700661f0 0x7f1a7006a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40728/0 (socket says 172.21.15.45:40728) 2023-11-28T20:35:34.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.509+0000 7f1a6ed9d640 1 -- 172.21.15.45:0/1783220313 learned_addr learned my addr 172.21.15.45:0/1783220313 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:34.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.510+0000 7f1a6e59c640 1 --2- 172.21.15.45:0/1783220313 >> v2:172.21.15.190:3300/0 conn(0x7f1a7015d010 0x7f1a7006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.510+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961321968 0 0) 0x7f1a7006e3b0 con 0x7f1a700661f0 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.510+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a5c003520 con 0x7f1a700661f0 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.510+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 859686158 0 0) 0x7f1a5c003520 con 0x7f1a700661f0 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.510+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 >> v2:172.21.15.190:3300/0 conn(0x7f1a7015d010 msgr2=0x7f1a7006ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.510+0000 7f1a6d59a640 1 --2- 172.21.15.45:0/1783220313 >> v2:172.21.15.190:3300/0 conn(0x7f1a7015d010 0x7f1a7006ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.510+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 >> v1:172.21.15.190:6789/0 conn(0x7f1a7006df20 legacy=0x7f1a700b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.510+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a700b3120 con 0x7f1a700661f0 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.510+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f1a68001670 con 0x7f1a700661f0 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 >> v1:172.21.15.45:6789/0 conn(0x7f1a700661f0 legacy=0x7f1a7006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6d59a640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a5c003970 0x7f1a5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:34.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6d59a640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5c0047c0 0x7f1a5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6d59a640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a5c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a700b5200 con 0x7f1a5c005180 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c00c120 con 0x7f1a5c0047c0 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c00c2e0 con 0x7f1a5c003970 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6ed9d640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a5c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6e59c640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a5c003970 0x7f1a5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.511+0000 7f1a6dd9b640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5c0047c0 0x7f1a5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.512+0000 7f1a6ed9d640 1 -- 172.21.15.45:0/1783220313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a5c003970 msgr2=0x7f1a5c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.512+0000 7f1a6ed9d640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a5c003970 0x7f1a5c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.512+0000 7f1a6ed9d640 1 -- 172.21.15.45:0/1783220313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5c0047c0 msgr2=0x7f1a5c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.512+0000 7f1a6ed9d640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5c0047c0 0x7f1a5c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.512+0000 7f1a6ed9d640 1 -- 172.21.15.45:0/1783220313 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c003520 con 0x7f1a5c005180 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.513+0000 7f1a6ed9d640 1 -- 172.21.15.45:0/1783220313 reap_dead start 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.513+0000 7f1a6ed9d640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a5c00b980 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800c240 tx=0x7f1a5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5557fd1cb4f627 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.513+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a68005ad0 con 0x7f1a5c005180 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.513+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a68005c70 con 0x7f1a5c005180 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.513+0000 7f1a6d59a640 1 -- 172.21.15.45:0/1783220313 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6802c8b0 con 0x7f1a5c005180 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.514+0000 7f1a6f59e640 1 -- 172.21.15.45:0/1783220313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 msgr2=0x7f1a5c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.514+0000 7f1a6f59e640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a5c00b980 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800c240 tx=0x7f1a5c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.514+0000 7f1a6f59e640 1 -- 172.21.15.45:0/1783220313 shutdown_connections 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.514+0000 7f1a6f59e640 1 --2- 172.21.15.45:0/1783220313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a5c00b980 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.514+0000 7f1a6f59e640 1 -- 172.21.15.45:0/1783220313 >> 172.21.15.45:0/1783220313 conn(0x7f1a700b2500 msgr2=0x7f1a700cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.514+0000 7f1a6f59e640 1 -- 172.21.15.45:0/1783220313 shutdown_connections 2023-11-28T20:35:34.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.514+0000 7f1a6f59e640 1 -- 172.21.15.45:0/1783220313 wait complete. 2023-11-28T20:35:34.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.516+0000 7f1a6f59e640 1 Processor -- start 2023-11-28T20:35:34.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.516+0000 7f1a6f59e640 1 -- start start 2023-11-28T20:35:34.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.517+0000 7f1a6f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a701513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:34.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.517+0000 7f1a6f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a70152440 0x7f1a70159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:34.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.517+0000 7f1a6f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a701528c0 0x7f1a7015a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:34.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.517+0000 7f1a6f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a70154130 con 0x7f1a5c005180 2023-11-28T20:35:34.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.517+0000 7f1a6f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a701533e0 con 0x7f1a701528c0 2023-11-28T20:35:34.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.517+0000 7f1a6f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a70152e20 con 0x7f1a70152440 2023-11-28T20:35:34.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.517+0000 7f1a6e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a701513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:34.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a701513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59350/0 (socket says 172.21.15.45:59350) 2023-11-28T20:35:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6e59c640 1 -- 172.21.15.45:0/3469176232 learned_addr learned my addr 172.21.15.45:0/3469176232 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6dd9b640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a70152440 0x7f1a70159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6ed9d640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a701528c0 0x7f1a7015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6e59c640 1 -- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a70152440 msgr2=0x7f1a70159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6e59c640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a70152440 0x7f1a70159be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6e59c640 1 -- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a701528c0 msgr2=0x7f1a7015a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6e59c640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a701528c0 0x7f1a7015a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6e59c640 1 -- 172.21.15.45:0/3469176232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a70153100 con 0x7f1a5c005180 2023-11-28T20:35:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6ed9d640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a701528c0 0x7f1a7015a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.518+0000 7f1a6e59c640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a701513f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1a60007f00 tx=0x7f1a600115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.519+0000 7f1a66ffd640 1 -- 172.21.15.45:0/3469176232 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000e030 con 0x7f1a5c005180 2023-11-28T20:35:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.519+0000 7f1a66ffd640 1 -- 172.21.15.45:0/3469176232 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a60016470 con 0x7f1a5c005180 2023-11-28T20:35:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.519+0000 7f1a66ffd640 1 -- 172.21.15.45:0/3469176232 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a60016820 con 0x7f1a5c005180 2023-11-28T20:35:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.519+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7015d880 con 0x7f1a5c005180 2023-11-28T20:35:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.519+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a70151930 con 0x7f1a5c005180 2023-11-28T20:35:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.520+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a700733d0 con 0x7f1a5c005180 2023-11-28T20:35:34.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.522+0000 7f1a66ffd640 1 -- 172.21.15.45:0/3469176232 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1a60016980 con 0x7f1a5c005180 2023-11-28T20:35:34.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.523+0000 7f1a66ffd640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a5006ae40 0x7f1a5006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:34.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.523+0000 7f1a66ffd640 1 -- 172.21.15.45:0/3469176232 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f1a6000f070 con 0x7f1a5c005180 2023-11-28T20:35:34.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.524+0000 7f1a6dd9b640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a5006ae40 0x7f1a5006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:34.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.525+0000 7f1a6dd9b640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a5006ae40 0x7f1a5006d300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1a58005490 tx=0x7f1a58009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:34.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.529+0000 7f1a66ffd640 1 -- 172.21.15.45:0/3469176232 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a60055be0 con 0x7f1a5c005180 2023-11-28T20:35:34.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:34.726+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1a7006b290 con 0x7f1a5c005180 2023-11-28T20:35:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.264+0000 7f1a66ffd640 1 -- 172.21.15.45:0/3469176232 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1a60059200 con 0x7f1a5c005180 2023-11-28T20:35:35.267 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T20:35:35.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a5006ae40 msgr2=0x7f1a5006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:35.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a5006ae40 0x7f1a5006d300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1a58005490 tx=0x7f1a58009040 comp rx=0 tx=0).stop 2023-11-28T20:35:35.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 msgr2=0x7f1a701513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:35.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a701513f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1a60007f00 tx=0x7f1a600115a0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 shutdown_connections 2023-11-28T20:35:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f1a5006ae40 0x7f1a5006d300 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a701528c0 0x7f1a7015a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a70152440 0x7f1a70159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 --2- 172.21.15.45:0/3469176232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a5c005180 0x7f1a701513f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 >> 172.21.15.45:0/3469176232 conn(0x7f1a700b2500 msgr2=0x7f1a7006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 shutdown_connections 2023-11-28T20:35:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.268+0000 7f1a6f59e640 1 -- 172.21.15.45:0/3469176232 wait complete. 2023-11-28T20:35:35.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T20:35:35.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.495+0000 7fb0032ad640 1 Processor -- start 2023-11-28T20:35:35.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.495+0000 7fb0032ad640 1 -- start start 2023-11-28T20:35:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.497+0000 7fb0032ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc0b3040 0x7faffc06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.497+0000 7fb0032ad640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faffc0b3420 0x7faffc0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:35.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.497+0000 7fb0032ad640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc06d410 con 0x7faffc15b270 2023-11-28T20:35:35.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.497+0000 7fb0032ad640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc0ceaa0 con 0x7faffc0b3420 2023-11-28T20:35:35.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.497+0000 7fb0032ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc06d6f0 con 0x7faffc0b3040 2023-11-28T20:35:35.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.497+0000 7fb001aaa640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faffc15b270 0x7faffc15f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40744/0 (socket says 172.21.15.45:40744) 2023-11-28T20:35:35.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.497+0000 7fb001aaa640 1 -- 172.21.15.45:0/2170148120 learned_addr learned my addr 172.21.15.45:0/2170148120 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb002aac640 1 --2- 172.21.15.45:0/2170148120 >> v2:172.21.15.190:3300/0 conn(0x7faffc0b3420 0x7faffc0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286401998 0 0) 0x7faffc06d410 con 0x7faffc15b270 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe8003520 con 0x7faffc15b270 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0022ab640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc0b3040 0x7faffc06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2601953485 0 0) 0x7fafe8003520 con 0x7faffc15b270 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc0b3040 msgr2=0x7faffc06cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc0b3040 0x7faffc06cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 >> v2:172.21.15.190:3300/0 conn(0x7faffc0b3420 msgr2=0x7faffc0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 --2- 172.21.15.45:0/2170148120 >> v2:172.21.15.190:3300/0 conn(0x7faffc0b3420 0x7faffc0ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc06d9d0 con 0x7faffc15b270 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fafec001670 con 0x7faffc15b270 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 >> v1:172.21.15.45:6789/0 conn(0x7faffc15b270 legacy=0x7faffc15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.498+0000 7fb0012a9640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8003970 0x7fafe8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb0012a9640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc0b3040 0x7fafe80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb0012a9640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe8004d00 0x7fafe8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc06d6f0 con 0x7fafe8004d00 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe800b9d0 con 0x7fafe8003970 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe800bb90 con 0x7faffc0b3040 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb002aac640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe8004d00 0x7fafe8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb001aaa640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc0b3040 0x7fafe80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb0022ab640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8003970 0x7fafe8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:35.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb002aac640 1 -- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc0b3040 msgr2=0x7fafe80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:35.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb002aac640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc0b3040 0x7fafe80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb002aac640 1 -- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8003970 msgr2=0x7fafe8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:35.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb002aac640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8003970 0x7fafe8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb002aac640 1 -- 172.21.15.45:0/2170148120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe8003520 con 0x7fafe8004d00 2023-11-28T20:35:35.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.499+0000 7fb0022ab640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8003970 0x7fafe8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:35.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.500+0000 7fb002aac640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe8004d00 0x7fafe8005180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faff4009fc0 tx=0x7fafe80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddaa2cfe285dd2d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.501+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff402cc60 con 0x7fafe8004d00 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.501+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff400f040 con 0x7fafe8004d00 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.501+0000 7fb0012a9640 1 -- 172.21.15.45:0/2170148120 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4034440 con 0x7fafe8004d00 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.501+0000 7fb0032ad640 1 -- 172.21.15.45:0/2170148120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe8004d00 msgr2=0x7fafe8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.501+0000 7fb0032ad640 1 --2- 172.21.15.45:0/2170148120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafe8004d00 0x7fafe8005180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faff4009fc0 tx=0x7fafe80057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.501+0000 7fb002aac640 1 -- 172.21.15.45:0/2170148120 reap_dead start 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.501+0000 7fb0032ad640 1 -- 172.21.15.45:0/2170148120 shutdown_connections 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.501+0000 7fb0032ad640 1 -- 172.21.15.45:0/2170148120 >> 172.21.15.45:0/2170148120 conn(0x7faffc0b1050 msgr2=0x7faffc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.502+0000 7fb0032ad640 1 -- 172.21.15.45:0/2170148120 shutdown_connections 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.502+0000 7fb0032ad640 1 -- 172.21.15.45:0/2170148120 wait complete. 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.503+0000 7fb0032ad640 1 Processor -- start 2023-11-28T20:35:35.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.504+0000 7fb0032ad640 1 -- start start 2023-11-28T20:35:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.504+0000 7fb0032ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8004d00 0x7faffc099f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.504+0000 7fb0032ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc143630 0x7faffc14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.504+0000 7fb0032ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc14c180 0x7faffc14c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.504+0000 7fb0032ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc14cb40 con 0x7faffc14c180 2023-11-28T20:35:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb0032ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc14d430 con 0x7fafe8004d00 2023-11-28T20:35:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb0032ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc15b6a0 con 0x7faffc143630 2023-11-28T20:35:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb002aac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc14c180 0x7faffc14c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb002aac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc14c180 0x7faffc14c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59360/0 (socket says 172.21.15.45:59360) 2023-11-28T20:35:35.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb002aac640 1 -- 172.21.15.45:0/1911075125 learned_addr learned my addr 172.21.15.45:0/1911075125 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb0022ab640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8004d00 0x7faffc099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb001aaa640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc143630 0x7faffc14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb002aac640 1 -- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc143630 msgr2=0x7faffc14cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb002aac640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc143630 0x7faffc14cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb002aac640 1 -- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8004d00 msgr2=0x7faffc099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb002aac640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8004d00 0x7faffc099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.505+0000 7fb002aac640 1 -- 172.21.15.45:0/1911075125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc15b980 con 0x7faffc14c180 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.506+0000 7fb002aac640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc14c180 0x7faffc14c600 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7faff4037040 tx=0x7faff4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.506+0000 7faff2ffd640 1 -- 172.21.15.45:0/1911075125 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4034d50 con 0x7faffc14c180 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.506+0000 7faff2ffd640 1 -- 172.21.15.45:0/1911075125 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff4003990 con 0x7faffc14c180 2023-11-28T20:35:35.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.506+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc15bc60 con 0x7faffc14c180 2023-11-28T20:35:35.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.506+0000 7faff2ffd640 1 -- 172.21.15.45:0/1911075125 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4003d40 con 0x7faffc14c180 2023-11-28T20:35:35.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.506+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc15c220 con 0x7faffc14c180 2023-11-28T20:35:35.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.507+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc15c410 con 0x7faffc14c180 2023-11-28T20:35:35.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.509+0000 7faff2ffd640 1 -- 172.21.15.45:0/1911075125 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faff4004030 con 0x7faffc14c180 2023-11-28T20:35:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.509+0000 7faff2ffd640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fafe006ae90 0x7fafe006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.510+0000 7faff2ffd640 1 -- 172.21.15.45:0/1911075125 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5929+0+0 (secure 0 0 0) 0x7faff40ab5f0 con 0x7faffc14c180 2023-11-28T20:35:35.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.510+0000 7fb0022ab640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fafe006ae90 0x7fafe006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:35.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.512+0000 7fb0022ab640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fafe006ae90 0x7fafe006d350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faff8009aa0 tx=0x7faff8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.516+0000 7faff2ffd640 1 -- 172.21.15.45:0/1911075125 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faff4076770 con 0x7faffc14c180 2023-11-28T20:35:35.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:35.748+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faffc072200 con 0x7faffc14c180 2023-11-28T20:35:36.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.317+0000 7faff2ffd640 1 -- 172.21.15.45:0/1911075125 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7faff4079d90 con 0x7faffc14c180 2023-11-28T20:35:36.320 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 5 and data pool 6 2023-11-28T20:35:36.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.324+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fafe006ae90 msgr2=0x7fafe006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:36.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.324+0000 7fb0032ad640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fafe006ae90 0x7fafe006d350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faff8009aa0 tx=0x7faff8008040 comp rx=0 tx=0).stop 2023-11-28T20:35:36.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.324+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc14c180 msgr2=0x7faffc14c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:36.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.324+0000 7fb0032ad640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc14c180 0x7faffc14c600 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7faff4037040 tx=0x7faff4009750 comp rx=0 tx=0).stop 2023-11-28T20:35:36.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.326+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 shutdown_connections 2023-11-28T20:35:36.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.326+0000 7fb0032ad640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fafe006ae90 0x7fafe006d350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.326+0000 7fb0032ad640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faffc14c180 0x7faffc14c600 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.326+0000 7fb0032ad640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faffc143630 0x7faffc14cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.326+0000 7fb0032ad640 1 --2- 172.21.15.45:0/1911075125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafe8004d00 0x7faffc099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.326+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 >> 172.21.15.45:0/1911075125 conn(0x7faffc0b1050 msgr2=0x7faffc14dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:36.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.327+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 shutdown_connections 2023-11-28T20:35:36.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.327+0000 7fb0032ad640 1 -- 172.21.15.45:0/1911075125 wait complete. 2023-11-28T20:35:36.357 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:36.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.543+0000 7fc63bb48640 1 Processor -- start 2023-11-28T20:35:36.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.544+0000 7fc63bb48640 1 -- start start 2023-11-28T20:35:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.545+0000 7fc63bb48640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6340d26d0 0x7fc6340b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.545+0000 7fc63bb48640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340b5710 0x7fc6340b6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.545+0000 7fc63bb48640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6340d2240 con 0x7fc63415d250 2023-11-28T20:35:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.545+0000 7fc63bb48640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6340d2fc0 con 0x7fc6340d26d0 2023-11-28T20:35:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.545+0000 7fc63bb48640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6340b5ba0 con 0x7fc6340b5710 2023-11-28T20:35:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.546+0000 7fc6390e3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6340d26d0 0x7fc6340b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.546+0000 7fc6390e3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6340d26d0 0x7fc6340b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59364/0 (socket says 172.21.15.45:59364) 2023-11-28T20:35:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.546+0000 7fc6390e3640 1 -- 172.21.15.45:0/3121364123 learned_addr learned my addr 172.21.15.45:0/3121364123 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:36.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.546+0000 7fc63a0e5640 1 --2- 172.21.15.45:0/3121364123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340b5710 0x7fc6340b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:36.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.546+0000 7fc6390e3640 1 -- 172.21.15.45:0/3121364123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340b5710 msgr2=0x7fc6340b6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:36.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.546+0000 7fc6390e3640 1 --2- 172.21.15.45:0/3121364123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340b5710 0x7fc6340b6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.546+0000 7fc6390e3640 1 -- 172.21.15.45:0/3121364123 >> v1:172.21.15.190:6789/0 conn(0x7fc63415d250 legacy=0x7fc6340b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:36.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.546+0000 7fc6390e3640 1 -- 172.21.15.45:0/3121364123 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6340d1630 con 0x7fc6340d26d0 2023-11-28T20:35:36.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.547+0000 7fc6390e3640 1 --2- 172.21.15.45:0/3121364123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340d26d0 0x7fc6340b51d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc624009870 tx=0x7fc62402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8feff72366d6f9f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:36.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.548+0000 7fc6388e2640 1 -- 172.21.15.45:0/3121364123 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624030840 con 0x7fc6340d26d0 2023-11-28T20:35:36.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.548+0000 7fc6388e2640 1 -- 172.21.15.45:0/3121364123 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6240309a0 con 0x7fc6340d26d0 2023-11-28T20:35:36.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.548+0000 7fc6388e2640 1 -- 172.21.15.45:0/3121364123 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624030d30 con 0x7fc6340d26d0 2023-11-28T20:35:36.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.548+0000 7fc63bb48640 1 -- 172.21.15.45:0/3121364123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340d26d0 msgr2=0x7fc6340b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.548+0000 7fc63bb48640 1 --2- 172.21.15.45:0/3121364123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340d26d0 0x7fc6340b51d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc624009870 tx=0x7fc62402b980 comp rx=0 tx=0).stop 2023-11-28T20:35:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.549+0000 7fc63bb48640 1 -- 172.21.15.45:0/3121364123 shutdown_connections 2023-11-28T20:35:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.549+0000 7fc63bb48640 1 --2- 172.21.15.45:0/3121364123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340b5710 0x7fc6340b6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.549+0000 7fc63bb48640 1 --2- 172.21.15.45:0/3121364123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340d26d0 0x7fc6340b51d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.549+0000 7fc63bb48640 1 -- 172.21.15.45:0/3121364123 >> 172.21.15.45:0/3121364123 conn(0x7fc634070f40 msgr2=0x7fc634071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.549+0000 7fc63bb48640 1 -- 172.21.15.45:0/3121364123 shutdown_connections 2023-11-28T20:35:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.549+0000 7fc63bb48640 1 -- 172.21.15.45:0/3121364123 wait complete. 2023-11-28T20:35:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.550+0000 7fc63bb48640 1 Processor -- start 2023-11-28T20:35:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.550+0000 7fc63bb48640 1 -- start start 2023-11-28T20:35:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.551+0000 7fc63bb48640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340b5710 0x7fc63414b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.551+0000 7fc63bb48640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340d26d0 0x7fc63415da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.551+0000 7fc63bb48640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63415d250 0x7fc6340cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.551+0000 7fc63bb48640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc634152ef0 con 0x7fc6340b5710 2023-11-28T20:35:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.551+0000 7fc63bb48640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63414bbc0 con 0x7fc63415d250 2023-11-28T20:35:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.551+0000 7fc63bb48640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6340cb300 con 0x7fc6340d26d0 2023-11-28T20:35:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc6398e4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340b5710 0x7fc63414b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc6398e4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340b5710 0x7fc63414b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59370/0 (socket says 172.21.15.45:59370) 2023-11-28T20:35:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc6398e4640 1 -- 172.21.15.45:0/3815347898 learned_addr learned my addr 172.21.15.45:0/3815347898 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc63a0e5640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63415d250 0x7fc6340cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc6390e3640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340d26d0 0x7fc63415da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc6398e4640 1 -- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340d26d0 msgr2=0x7fc63415da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc6398e4640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340d26d0 0x7fc63415da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc6398e4640 1 -- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63415d250 msgr2=0x7fc6340cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc6398e4640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63415d250 0x7fc6340cbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.552+0000 7fc6398e4640 1 -- 172.21.15.45:0/3815347898 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6240094d0 con 0x7fc6340b5710 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.553+0000 7fc6398e4640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340b5710 0x7fc63414b680 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c400 tx=0x7fc63000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.553+0000 7fc62a7fc640 1 -- 172.21.15.45:0/3815347898 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6300052d0 con 0x7fc6340b5710 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.553+0000 7fc62a7fc640 1 -- 172.21.15.45:0/3815347898 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc630005430 con 0x7fc6340b5710 2023-11-28T20:35:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.553+0000 7fc62a7fc640 1 -- 172.21.15.45:0/3815347898 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630003830 con 0x7fc6340b5710 2023-11-28T20:35:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.553+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc634152930 con 0x7fc6340b5710 2023-11-28T20:35:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.553+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6340cb5a0 con 0x7fc6340b5710 2023-11-28T20:35:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.554+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6340733d0 con 0x7fc6340b5710 2023-11-28T20:35:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.555+0000 7fc62a7fc640 1 -- 172.21.15.45:0/3815347898 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc630003990 con 0x7fc6340b5710 2023-11-28T20:35:36.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.556+0000 7fc62a7fc640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc61806ae40 0x7fc61806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:36.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.556+0000 7fc62a7fc640 1 -- 172.21.15.45:0/3815347898 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc630088e00 con 0x7fc6340b5710 2023-11-28T20:35:36.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.557+0000 7fc6390e3640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc61806ae40 0x7fc61806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:36.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.558+0000 7fc6390e3640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc61806ae40 0x7fc61806d300 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc624036000 tx=0x7fc624008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.561+0000 7fc62a7fc640 1 -- 172.21.15.45:0/3815347898 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc630053c40 con 0x7fc6340b5710 2023-11-28T20:35:36.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.755+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6340c9b30 con 0x7fc6340b5710 2023-11-28T20:35:36.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.757+0000 7fc62a7fc640 1 -- 172.21.15.45:0/3815347898 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16775 (secure 0 0 0) 0x7fc630017020 con 0x7fc6340b5710 2023-11-28T20:35:36.760 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:36.760 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":38,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:35:35.750415+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:35:33.849797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-28T20:35:34.783808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc61806ae40 msgr2=0x7fc61806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc61806ae40 0x7fc61806d300 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc624036000 tx=0x7fc624008030 comp rx=0 tx=0).stop 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340b5710 msgr2=0x7fc63414b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340b5710 0x7fc63414b680 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c400 tx=0x7fc63000c8c0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 shutdown_connections 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc61806ae40 0x7fc61806d300 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc63415d250 0x7fc6340cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6340d26d0 0x7fc63415da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 --2- 172.21.15.45:0/3815347898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6340b5710 0x7fc63414b680 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:36.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 >> 172.21.15.45:0/3815347898 conn(0x7fc634070f40 msgr2=0x7fc634071320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:36.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 shutdown_connections 2023-11-28T20:35:36.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:36.760+0000 7fc63bb48640 1 -- 172.21.15.45:0/3815347898 wait complete. 2023-11-28T20:35:36.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:37.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.000+0000 7fc86ae60640 1 Processor -- start 2023-11-28T20:35:37.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.000+0000 7fc86ae60640 1 -- start start 2023-11-28T20:35:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.001+0000 7fc86ae60640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc86415b170 0x7fc8640cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.001+0000 7fc86ae60640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc86415ef90 0x7fc86415f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.001+0000 7fc86ae60640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8640cb960 con 0x7fc8640d0da0 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.001+0000 7fc86ae60640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640cc4c0 con 0x7fc86415b170 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.001+0000 7fc86ae60640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86415fd00 con 0x7fc86415ef90 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.001+0000 7fc86965d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc8640d0da0 0x7fc8640d1220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40760/0 (socket says 172.21.15.45:40760) 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.001+0000 7fc86965d640 1 -- 172.21.15.45:0/326110417 learned_addr learned my addr 172.21.15.45:0/326110417 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc869e5e640 1 --2- 172.21.15.45:0/326110417 >> v2:172.21.15.190:3300/0 conn(0x7fc86415b170 0x7fc8640cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc86a65f640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc86415ef90 0x7fc86415f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96351033 0 0) 0x7fc8640cb960 con 0x7fc8640d0da0 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc84c003520 con 0x7fc8640d0da0 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223850625 0 0) 0x7fc84c003520 con 0x7fc8640d0da0 2023-11-28T20:35:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc86415ef90 msgr2=0x7fc86415f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc86415ef90 0x7fc86415f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 >> v2:172.21.15.190:3300/0 conn(0x7fc86415b170 msgr2=0x7fc8640cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 --2- 172.21.15.45:0/326110417 >> v2:172.21.15.190:3300/0 conn(0x7fc86415b170 0x7fc8640cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640cbb90 con 0x7fc8640d0da0 2023-11-28T20:35:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc858001670 con 0x7fc8640d0da0 2023-11-28T20:35:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 >> v1:172.21.15.45:6789/0 conn(0x7fc8640d0da0 legacy=0x7fc8640d1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc86415ef90 0x7fc84c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc84c003ef0 0x7fc84c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc84c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640cc4c0 con 0x7fc84c004d00 2023-11-28T20:35:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86415fd00 con 0x7fc84c003ef0 2023-11-28T20:35:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.002+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c00b980 con 0x7fc86415ef90 2023-11-28T20:35:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.003+0000 7fc86a65f640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc84c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.003+0000 7fc86a65f640 1 -- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc86415ef90 msgr2=0x7fc84c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.003+0000 7fc869e5e640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc86415ef90 0x7fc84c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.003+0000 7fc86965d640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc84c003ef0 0x7fc84c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.003+0000 7fc86a65f640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc86415ef90 0x7fc84c003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.003+0000 7fc86a65f640 1 -- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc84c003ef0 msgr2=0x7fc84c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.003+0000 7fc86a65f640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc84c003ef0 0x7fc84c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.003+0000 7fc86a65f640 1 -- 172.21.15.45:0/326110417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c003520 con 0x7fc84c004d00 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.004+0000 7fc86a65f640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc84c005180 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009ac0 tx=0x7fc84c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f223c61e5059e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.005+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c02c6b0 con 0x7fc84c004d00 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.005+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c02c810 con 0x7fc84c004d00 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.005+0000 7fc868e5c640 1 -- 172.21.15.45:0/326110417 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c02cba0 con 0x7fc84c004d00 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.005+0000 7fc86ae60640 1 -- 172.21.15.45:0/326110417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 msgr2=0x7fc84c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.005+0000 7fc86ae60640 1 --2- 172.21.15.45:0/326110417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc84c005180 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009ac0 tx=0x7fc84c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.005+0000 7fc86a65f640 1 -- 172.21.15.45:0/326110417 reap_dead start 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.005+0000 7fc86ae60640 1 -- 172.21.15.45:0/326110417 shutdown_connections 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.005+0000 7fc86ae60640 1 -- 172.21.15.45:0/326110417 >> 172.21.15.45:0/326110417 conn(0x7fc864099a40 msgr2=0x7fc864099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:37.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.006+0000 7fc86ae60640 1 -- 172.21.15.45:0/326110417 shutdown_connections 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.006+0000 7fc86ae60640 1 -- 172.21.15.45:0/326110417 wait complete. 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.007+0000 7fc86ae60640 1 Processor -- start 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.008+0000 7fc86ae60640 1 -- start start 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86ae60640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc86415b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86ae60640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86415e470 0x7fc8641ee190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86ae60640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8641ee6d0 0x7fc8641450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86ae60640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640add20 con 0x7fc84c004d00 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86ae60640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86415bdf0 con 0x7fc86415e470 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86ae60640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640adf30 con 0x7fc8641ee6d0 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc869e5e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc86415b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc869e5e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc86415b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59386/0 (socket says 172.21.15.45:59386) 2023-11-28T20:35:37.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc869e5e640 1 -- 172.21.15.45:0/423550346 learned_addr learned my addr 172.21.15.45:0/423550346 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86a65f640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8641ee6d0 0x7fc8641450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86965d640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86415e470 0x7fc8641ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc869e5e640 1 -- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8641ee6d0 msgr2=0x7fc8641450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc869e5e640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8641ee6d0 0x7fc8641450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc869e5e640 1 -- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86415e470 msgr2=0x7fc8641ee190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc869e5e640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86415e470 0x7fc8641ee190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc869e5e640 1 -- 172.21.15.45:0/423550346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640ae210 con 0x7fc84c004d00 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86a65f640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8641ee6d0 0x7fc8641450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.009+0000 7fc86965d640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86415e470 0x7fc8641ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:37.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.010+0000 7fc869e5e640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc86415b8b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc85400f3d0 tx=0x7fc85400f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:37.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.010+0000 7fc8627fc640 1 -- 172.21.15.45:0/423550346 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854016620 con 0x7fc84c004d00 2023-11-28T20:35:37.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.010+0000 7fc8627fc640 1 -- 172.21.15.45:0/423550346 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8540167a0 con 0x7fc84c004d00 2023-11-28T20:35:37.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.010+0000 7fc8627fc640 1 -- 172.21.15.45:0/423550346 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854016b50 con 0x7fc84c004d00 2023-11-28T20:35:37.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.010+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8640ae4f0 con 0x7fc84c004d00 2023-11-28T20:35:37.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.010+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8640aeab0 con 0x7fc84c004d00 2023-11-28T20:35:37.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.011+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8640aec70 con 0x7fc84c004d00 2023-11-28T20:35:37.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.012+0000 7fc8627fc640 1 -- 172.21.15.45:0/423550346 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc854016cb0 con 0x7fc84c004d00 2023-11-28T20:35:37.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.013+0000 7fc8627fc640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc85006ae70 0x7fc85006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.013+0000 7fc8627fc640 1 -- 172.21.15.45:0/423550346 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc85408a790 con 0x7fc84c004d00 2023-11-28T20:35:37.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.014+0000 7fc86965d640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc85006ae70 0x7fc85006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.015+0000 7fc86965d640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc85006ae70 0x7fc85006d330 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc8580018a0 tx=0x7fc858009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:37.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.018+0000 7fc8627fc640 1 -- 172.21.15.45:0/423550346 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc854055660 con 0x7fc84c004d00 2023-11-28T20:35:37.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.218+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc86409f0a0 con 0x7fc84c004d00 2023-11-28T20:35:37.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.220+0000 7fc8627fc640 1 -- 172.21.15.45:0/423550346 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16775 (secure 0 0 0) 0x7fc854058c80 con 0x7fc84c004d00 2023-11-28T20:35:37.222 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:37.223 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":38,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:35:35.750415+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:35:33.849797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-28T20:35:34.783808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:37.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.222+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc85006ae70 msgr2=0x7fc85006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.222+0000 7fc86ae60640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc85006ae70 0x7fc85006d330 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc8580018a0 tx=0x7fc858009040 comp rx=0 tx=0).stop 2023-11-28T20:35:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.222+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 msgr2=0x7fc86415b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.222+0000 7fc86ae60640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc86415b8b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc85400f3d0 tx=0x7fc85400f890 comp rx=0 tx=0).stop 2023-11-28T20:35:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.223+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 shutdown_connections 2023-11-28T20:35:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.223+0000 7fc86ae60640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc85006ae70 0x7fc85006d330 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.223+0000 7fc86ae60640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8641ee6d0 0x7fc8641450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.223+0000 7fc86ae60640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86415e470 0x7fc8641ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.223+0000 7fc86ae60640 1 --2- 172.21.15.45:0/423550346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc84c004d00 0x7fc86415b8b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.223+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 >> 172.21.15.45:0/423550346 conn(0x7fc864099a40 msgr2=0x7fc86406e690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:37.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.223+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 shutdown_connections 2023-11-28T20:35:37.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.223+0000 7fc86ae60640 1 -- 172.21.15.45:0/423550346 wait complete. 2023-11-28T20:35:37.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T20:35:37.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.473+0000 7f3559cee640 1 Processor -- start 2023-11-28T20:35:37.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.473+0000 7f3559cee640 1 -- start start 2023-11-28T20:35:37.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.474+0000 7f3559cee640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f355406e540 0x7f355406a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.474+0000 7f3559cee640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355406eb90 con 0x7f355415b610 2023-11-28T20:35:37.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.474+0000 7f3559cee640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35540cf9c0 con 0x7f35540c35f0 2023-11-28T20:35:37.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.474+0000 7f3559cee640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355415b9f0 con 0x7f355406e540 2023-11-28T20:35:37.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.475+0000 7f35537fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f355415b610 0x7f355406de30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40770/0 (socket says 172.21.15.45:40770) 2023-11-28T20:35:37.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.475+0000 7f35537fe640 1 -- 172.21.15.45:0/3578257503 learned_addr learned my addr 172.21.15.45:0/3578257503 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:37.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.475+0000 7f3552ffd640 1 --2- 172.21.15.45:0/3578257503 >> v2:172.21.15.190:3300/0 conn(0x7f355406e540 0x7f355406a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.475+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082549155 0 0) 0x7f355406eb90 con 0x7f355415b610 2023-11-28T20:35:37.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353c003520 con 0x7f355415b610 2023-11-28T20:35:37.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823777521 0 0) 0x7f353c003520 con 0x7f355415b610 2023-11-28T20:35:37.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 >> v2:172.21.15.190:3300/0 conn(0x7f355406e540 msgr2=0x7f355406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:37.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 --2- 172.21.15.45:0/3578257503 >> v2:172.21.15.190:3300/0 conn(0x7f355406e540 0x7f355406a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 >> v1:172.21.15.190:6789/0 conn(0x7f35540c35f0 legacy=0x7f355406aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35540cedb0 con 0x7f355415b610 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3544001670 con 0x7f355415b610 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 >> v1:172.21.15.45:6789/0 conn(0x7f355415b610 legacy=0x7f355406de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f353c003970 0x7f353c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f353c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f353c005180 0x7f353c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355415b9f0 con 0x7f353c0047c0 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c00c020 con 0x7f353c003970 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c00c1e0 con 0x7f353c005180 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.476+0000 7f3552ffd640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f353c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.477+0000 7f35537fe640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f353c003970 0x7f353c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.477+0000 7f3552ffd640 1 -- 172.21.15.45:0/3578257503 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f353c005180 msgr2=0x7f353c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:37.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.477+0000 7f3552ffd640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f353c005180 0x7f353c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.477+0000 7f3552ffd640 1 -- 172.21.15.45:0/3578257503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f353c003970 msgr2=0x7f353c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:37.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.477+0000 7f3552ffd640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f353c003970 0x7f353c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.477+0000 7f3552ffd640 1 -- 172.21.15.45:0/3578257503 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c003520 con 0x7f353c0047c0 2023-11-28T20:35:37.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.477+0000 7f3553fff640 1 -- 172.21.15.45:0/3578257503 reap_dead start 2023-11-28T20:35:37.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.478+0000 7f3552ffd640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f353c004c40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f355406c780 tx=0x7f353c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e66dba00c39375 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:37.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.478+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354802cb90 con 0x7f353c0047c0 2023-11-28T20:35:37.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.478+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354800f040 con 0x7f353c0047c0 2023-11-28T20:35:37.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.478+0000 7f35527fc640 1 -- 172.21.15.45:0/3578257503 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548034440 con 0x7f353c0047c0 2023-11-28T20:35:37.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.479+0000 7f3559cee640 1 -- 172.21.15.45:0/3578257503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 msgr2=0x7f353c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.479+0000 7f3559cee640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f353c004c40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f355406c780 tx=0x7f353c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.479+0000 7f3559cee640 1 -- 172.21.15.45:0/3578257503 shutdown_connections 2023-11-28T20:35:37.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.479+0000 7f3559cee640 1 --2- 172.21.15.45:0/3578257503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f353c004c40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.479+0000 7f3559cee640 1 -- 172.21.15.45:0/3578257503 >> 172.21.15.45:0/3578257503 conn(0x7f35540cc9c0 msgr2=0x7f35540ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:37.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.479+0000 7f3559cee640 1 -- 172.21.15.45:0/3578257503 shutdown_connections 2023-11-28T20:35:37.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.479+0000 7f3559cee640 1 -- 172.21.15.45:0/3578257503 wait complete. 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.481+0000 7f3559cee640 1 Processor -- start 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.481+0000 7f3559cee640 1 -- start start 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.482+0000 7f3559cee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f355414fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.482+0000 7f3559cee640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554156ee0 0x7f35540c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.482+0000 7f3559cee640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f355415e770 0x7f35540c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.482+0000 7f3559cee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35540c62a0 con 0x7f353c0047c0 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.482+0000 7f3559cee640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35540c6580 con 0x7f3554156ee0 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.482+0000 7f3559cee640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35540c6860 con 0x7f355415e770 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f35537fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f355414fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f35537fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f355414fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59410/0 (socket says 172.21.15.45:59410) 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f35537fe640 1 -- 172.21.15.45:0/211314580 learned_addr learned my addr 172.21.15.45:0/211314580 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f3552ffd640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554156ee0 0x7f35540c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f3553fff640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f355415e770 0x7f35540c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f35537fe640 1 -- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f355415e770 msgr2=0x7f35540c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f35537fe640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f355415e770 0x7f35540c55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f35537fe640 1 -- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554156ee0 msgr2=0x7f35540c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f35537fe640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554156ee0 0x7f35540c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f35537fe640 1 -- 172.21.15.45:0/211314580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355415bf60 con 0x7f353c0047c0 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f3553fff640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f355415e770 0x7f35540c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.483+0000 7f3552ffd640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554156ee0 0x7f35540c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.484+0000 7f35537fe640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f355414fbc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f354400e730 tx=0x7f354400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.484+0000 7f353bfff640 1 -- 172.21.15.45:0/211314580 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354400ee40 con 0x7f353c0047c0 2023-11-28T20:35:37.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.484+0000 7f353bfff640 1 -- 172.21.15.45:0/211314580 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354400c850 con 0x7f353c0047c0 2023-11-28T20:35:37.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.484+0000 7f353bfff640 1 -- 172.21.15.45:0/211314580 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354400cc00 con 0x7f353c0047c0 2023-11-28T20:35:37.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.484+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355415c240 con 0x7f353c0047c0 2023-11-28T20:35:37.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.484+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35540c3fb0 con 0x7f353c0047c0 2023-11-28T20:35:37.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.486+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35540c41a0 con 0x7f353c0047c0 2023-11-28T20:35:37.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.487+0000 7f353bfff640 1 -- 172.21.15.45:0/211314580 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f354401d460 con 0x7f353c0047c0 2023-11-28T20:35:37.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.488+0000 7f353bfff640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f354006ae70 0x7f354006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:37.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.488+0000 7f353bfff640 1 -- 172.21.15.45:0/211314580 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3544089730 con 0x7f353c0047c0 2023-11-28T20:35:37.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.489+0000 7f3552ffd640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f354006ae70 0x7f354006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:37.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.490+0000 7f3552ffd640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f354006ae70 0x7f354006d330 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3548036000 tx=0x7f354802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.493+0000 7f353bfff640 1 -- 172.21.15.45:0/211314580 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3544054610 con 0x7f353c0047c0 2023-11-28T20:35:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:37.730+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f35540c63e0 con 0x7f353c0047c0 2023-11-28T20:35:38.234 INFO:tasks.ceph.mds.c.smithi045.stderr:2023-11-28T20:35:38.232+0000 7f4a6b4d0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:35:38.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.329+0000 7f353bfff640 1 -- 172.21.15.45:0/211314580 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f35540c63e0 con 0x7f353c0047c0 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.332+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f354006ae70 msgr2=0x7f354006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.332+0000 7f3559cee640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f354006ae70 0x7f354006d330 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3548036000 tx=0x7f354802e000 comp rx=0 tx=0).stop 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.332+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 msgr2=0x7f355414fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.332+0000 7f3559cee640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f355414fbc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f354400e730 tx=0x7f354400ebf0 comp rx=0 tx=0).stop 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.333+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 shutdown_connections 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.333+0000 7f3559cee640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f354006ae70 0x7f354006d330 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.333+0000 7f3559cee640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f355415e770 0x7f35540c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.333+0000 7f3559cee640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3554156ee0 0x7f35540c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.333+0000 7f3559cee640 1 --2- 172.21.15.45:0/211314580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f353c0047c0 0x7f355414fbc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.333+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 >> 172.21.15.45:0/211314580 conn(0x7f35540cc9c0 msgr2=0x7f3554158690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:38.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.333+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 shutdown_connections 2023-11-28T20:35:38.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.333+0000 7f3559cee640 1 -- 172.21.15.45:0/211314580 wait complete. 2023-11-28T20:35:38.359 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T20:35:38.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T20:35:38.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.575+0000 7fd377f05640 1 Processor -- start 2023-11-28T20:35:38.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.575+0000 7fd377f05640 1 -- start start 2023-11-28T20:35:38.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.577+0000 7fd377f05640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37006ac10 0x7fd37006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:38.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.577+0000 7fd377f05640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37015ecd0 con 0x7fd37015f720 2023-11-28T20:35:38.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.577+0000 7fd377f05640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37015f290 con 0x7fd37015e840 2023-11-28T20:35:38.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.577+0000 7fd377f05640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37015fbd0 con 0x7fd37006ac10 2023-11-28T20:35:38.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.577+0000 7fd3764a2640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd37015f720 0x7fd3700cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40778/0 (socket says 172.21.15.45:40778) 2023-11-28T20:35:38.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.577+0000 7fd3764a2640 1 -- 172.21.15.45:0/128041340 learned_addr learned my addr 172.21.15.45:0/128041340 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:38.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.577+0000 7fd3754a0640 1 --2- 172.21.15.45:0/128041340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37006ac10 0x7fd37006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:38.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.578+0000 7fd374c9f640 1 -- 172.21.15.45:0/128041340 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403198528 0 0) 0x7fd37015ecd0 con 0x7fd37015f720 2023-11-28T20:35:38.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.578+0000 7fd374c9f640 1 -- 172.21.15.45:0/128041340 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd35c003520 con 0x7fd37015f720 2023-11-28T20:35:38.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.578+0000 7fd3754a0640 1 -- 172.21.15.45:0/128041340 >> v1:172.21.15.190:6789/0 conn(0x7fd37015e840 legacy=0x7fd37006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:38.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.578+0000 7fd3754a0640 1 -- 172.21.15.45:0/128041340 >> v1:172.21.15.45:6789/0 conn(0x7fd37015f720 legacy=0x7fd3700cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:38.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.578+0000 7fd3754a0640 1 -- 172.21.15.45:0/128041340 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd370072020 con 0x7fd37006ac10 2023-11-28T20:35:38.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.579+0000 7fd3754a0640 1 --2- 172.21.15.45:0/128041340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37006ac10 0x7fd37006c350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd360009800 tx=0x7fd36002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7423d039e61dacd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:38.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.580+0000 7fd374c9f640 1 -- 172.21.15.45:0/128041340 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360031890 con 0x7fd37006ac10 2023-11-28T20:35:38.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.580+0000 7fd374c9f640 1 -- 172.21.15.45:0/128041340 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3600319f0 con 0x7fd37006ac10 2023-11-28T20:35:38.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.580+0000 7fd374c9f640 1 -- 172.21.15.45:0/128041340 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360031d80 con 0x7fd37006ac10 2023-11-28T20:35:38.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.580+0000 7fd377f05640 1 -- 172.21.15.45:0/128041340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37006ac10 msgr2=0x7fd37006c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:38.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.580+0000 7fd377f05640 1 --2- 172.21.15.45:0/128041340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37006ac10 0x7fd37006c350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd360009800 tx=0x7fd36002b930 comp rx=0 tx=0).stop 2023-11-28T20:35:38.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.581+0000 7fd377f05640 1 -- 172.21.15.45:0/128041340 shutdown_connections 2023-11-28T20:35:38.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.581+0000 7fd377f05640 1 --2- 172.21.15.45:0/128041340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37006ac10 0x7fd37006c350 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:38.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.581+0000 7fd377f05640 1 -- 172.21.15.45:0/128041340 >> 172.21.15.45:0/128041340 conn(0x7fd3700ced60 msgr2=0x7fd3700cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:38.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.581+0000 7fd377f05640 1 -- 172.21.15.45:0/128041340 shutdown_connections 2023-11-28T20:35:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.581+0000 7fd377f05640 1 -- 172.21.15.45:0/128041340 wait complete. 2023-11-28T20:35:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.582+0000 7fd377f05640 1 Processor -- start 2023-11-28T20:35:38.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.583+0000 7fd377f05640 1 -- start start 2023-11-28T20:35:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.583+0000 7fd377f05640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd37006ac10 0x7fd370156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.583+0000 7fd377f05640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd37015e840 0x7fd3700b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd377f05640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37015f720 0x7fd3700c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd377f05640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd370156680 con 0x7fd37006ac10 2023-11-28T20:35:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd377f05640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3700c74c0 con 0x7fd37015e840 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd377f05640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37009ad10 con 0x7fd37015f720 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd375ca1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd37006ac10 0x7fd370156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd375ca1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd37006ac10 0x7fd370156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59416/0 (socket says 172.21.15.45:59416) 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd375ca1640 1 -- 172.21.15.45:0/3025655370 learned_addr learned my addr 172.21.15.45:0/3025655370 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd3754a0640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd37015e840 0x7fd3700b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd3764a2640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37015f720 0x7fd3700c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd375ca1640 1 -- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37015f720 msgr2=0x7fd3700c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd375ca1640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37015f720 0x7fd3700c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd375ca1640 1 -- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd37015e840 msgr2=0x7fd3700b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd375ca1640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd37015e840 0x7fd3700b19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.584+0000 7fd375ca1640 1 -- 172.21.15.45:0/3025655370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360009400 con 0x7fd37006ac10 2023-11-28T20:35:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.585+0000 7fd375ca1640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd37006ac10 0x7fd370156140 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00c400 tx=0x7fd36c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.585+0000 7fd3667fc640 1 -- 172.21.15.45:0/3025655370 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c0052d0 con 0x7fd37006ac10 2023-11-28T20:35:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.585+0000 7fd3667fc640 1 -- 172.21.15.45:0/3025655370 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c005450 con 0x7fd37006ac10 2023-11-28T20:35:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.585+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37009aff0 con 0x7fd37006ac10 2023-11-28T20:35:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.585+0000 7fd3667fc640 1 -- 172.21.15.45:0/3025655370 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c007260 con 0x7fd37006ac10 2023-11-28T20:35:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.585+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37009b5b0 con 0x7fd37006ac10 2023-11-28T20:35:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.586+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3700733d0 con 0x7fd37006ac10 2023-11-28T20:35:38.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.588+0000 7fd3667fc640 1 -- 172.21.15.45:0/3025655370 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd36c0073c0 con 0x7fd37006ac10 2023-11-28T20:35:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.589+0000 7fd3667fc640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd35406ae40 0x7fd35406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.589+0000 7fd3667fc640 1 -- 172.21.15.45:0/3025655370 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd36c088b60 con 0x7fd37006ac10 2023-11-28T20:35:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.590+0000 7fd3754a0640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd35406ae40 0x7fd35406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:38.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.591+0000 7fd3754a0640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd35406ae40 0x7fd35406d300 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd360036000 tx=0x7fd36003c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:38.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.593+0000 7fd3667fc640 1 -- 172.21.15.45:0/3025655370 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd36c053ac0 con 0x7fd37006ac10 2023-11-28T20:35:38.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:38.827+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd3700ce2b0 con 0x7fd37006ac10 2023-11-28T20:35:39.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.332+0000 7fd3667fc640 1 -- 172.21.15.45:0/3025655370 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd36c0570e0 con 0x7fd37006ac10 2023-11-28T20:35:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.335+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd35406ae40 msgr2=0x7fd35406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.335+0000 7fd377f05640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd35406ae40 0x7fd35406d300 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd360036000 tx=0x7fd36003c040 comp rx=0 tx=0).stop 2023-11-28T20:35:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.335+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd37006ac10 msgr2=0x7fd370156140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.335+0000 7fd377f05640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd37006ac10 0x7fd370156140 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00c400 tx=0x7fd36c00c8c0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.336+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 shutdown_connections 2023-11-28T20:35:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.336+0000 7fd377f05640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fd35406ae40 0x7fd35406d300 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.336+0000 7fd377f05640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd37015f720 0x7fd3700c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.336+0000 7fd377f05640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd37015e840 0x7fd3700b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.336+0000 7fd377f05640 1 --2- 172.21.15.45:0/3025655370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd37006ac10 0x7fd370156140 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.336+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 >> 172.21.15.45:0/3025655370 conn(0x7fd3700ced60 msgr2=0x7fd3700cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.336+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 shutdown_connections 2023-11-28T20:35:39.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.336+0000 7fd377f05640 1 -- 172.21.15.45:0/3025655370 wait complete. 2023-11-28T20:35:39.359 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T20:35:39.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:35:39.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.582+0000 7f7589727640 1 Processor -- start 2023-11-28T20:35:39.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.583+0000 7f7589727640 1 -- start start 2023-11-28T20:35:39.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.584+0000 7f7589727640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7584155780 0x7f75840abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:39.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.585+0000 7f7589727640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f75840ce800 0x7f75840b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:39.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.585+0000 7f7589727640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758415c3c0 con 0x7f75840ae540 2023-11-28T20:35:39.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.585+0000 7f7589727640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7584155b60 con 0x7f75840ce800 2023-11-28T20:35:39.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.585+0000 7f7589727640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758415ba90 con 0x7f7584155780 2023-11-28T20:35:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.585+0000 7f7588f26640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f75840ce800 0x7f75840b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.585+0000 7f7588f26640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f75840ce800 0x7f75840b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59428/0 (socket says 172.21.15.45:59428) 2023-11-28T20:35:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.585+0000 7f7588f26640 1 -- 172.21.15.45:0/3756831694 learned_addr learned my addr 172.21.15.45:0/3756831694 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.585+0000 7f7583fff640 1 --2- 172.21.15.45:0/3756831694 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7584155780 0x7f75840abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.586+0000 7f7588f26640 1 -- 172.21.15.45:0/3756831694 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7584155780 msgr2=0x7f75840abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.586+0000 7f7588f26640 1 --2- 172.21.15.45:0/3756831694 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7584155780 0x7f75840abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.586+0000 7f7588f26640 1 -- 172.21.15.45:0/3756831694 >> v1:172.21.15.190:6789/0 conn(0x7f75840ae540 legacy=0x7f75840ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.586+0000 7f7588f26640 1 -- 172.21.15.45:0/3756831694 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758415bd70 con 0x7f75840ce800 2023-11-28T20:35:39.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.586+0000 7f7583fff640 1 --2- 172.21.15.45:0/3756831694 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7584155780 0x7f75840abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:39.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.587+0000 7f7588f26640 1 --2- 172.21.15.45:0/3756831694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ce800 0x7f75840b20a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f757c009e50 tx=0x7f757c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9098e5ae89bd0fce server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:39.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.587+0000 7f75837fe640 1 -- 172.21.15.45:0/3756831694 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c02cb00 con 0x7f75840ce800 2023-11-28T20:35:39.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.587+0000 7f75837fe640 1 -- 172.21.15.45:0/3756831694 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c02cc60 con 0x7f75840ce800 2023-11-28T20:35:39.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.587+0000 7f75837fe640 1 -- 172.21.15.45:0/3756831694 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c0343f0 con 0x7f75840ce800 2023-11-28T20:35:39.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.588+0000 7f7589727640 1 -- 172.21.15.45:0/3756831694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ce800 msgr2=0x7f75840b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:39.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.588+0000 7f7589727640 1 --2- 172.21.15.45:0/3756831694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ce800 0x7f75840b20a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f757c009e50 tx=0x7f757c02bf60 comp rx=0 tx=0).stop 2023-11-28T20:35:39.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.588+0000 7f7589727640 1 -- 172.21.15.45:0/3756831694 shutdown_connections 2023-11-28T20:35:39.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.588+0000 7f7589727640 1 --2- 172.21.15.45:0/3756831694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ce800 0x7f75840b20a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.588+0000 7f7589727640 1 --2- 172.21.15.45:0/3756831694 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7584155780 0x7f75840abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.588+0000 7f7589727640 1 -- 172.21.15.45:0/3756831694 >> 172.21.15.45:0/3756831694 conn(0x7f758406ab30 msgr2=0x7f75840b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:39.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.588+0000 7f7589727640 1 -- 172.21.15.45:0/3756831694 shutdown_connections 2023-11-28T20:35:39.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.588+0000 7f7589727640 1 -- 172.21.15.45:0/3756831694 wait complete. 2023-11-28T20:35:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.590+0000 7f7589727640 1 Processor -- start 2023-11-28T20:35:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.590+0000 7f7589727640 1 -- start start 2023-11-28T20:35:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f7589727640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ae540 0x7f75841514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f7589727640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75840ce800 0x7f7584153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:39.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f7589727640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7584155780 0x7f7584154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:39.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f7589727640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75841519f0 con 0x7f75840ae540 2023-11-28T20:35:39.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f7589727640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7584099a40 con 0x7f7584155780 2023-11-28T20:35:39.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f7589727640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7584099d20 con 0x7f75840ce800 2023-11-28T20:35:39.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f7583fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ae540 0x7f75841514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:39.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f7583fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ae540 0x7f75841514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59436/0 (socket says 172.21.15.45:59436) 2023-11-28T20:35:39.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f7583fff640 1 -- 172.21.15.45:0/2145138526 learned_addr learned my addr 172.21.15.45:0/2145138526 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:39.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f7588f26640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7584155780 0x7f7584154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:39.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.591+0000 7f757b7fe640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75840ce800 0x7f7584153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f7583fff640 1 -- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75840ce800 msgr2=0x7f7584153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f7583fff640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75840ce800 0x7f7584153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f7583fff640 1 -- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7584155780 msgr2=0x7f7584154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f7583fff640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7584155780 0x7f7584154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f7583fff640 1 -- 172.21.15.45:0/2145138526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c009b30 con 0x7f75840ae540 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f757b7fe640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75840ce800 0x7f7584153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f7583fff640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ae540 0x7f75841514b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f757400c7f0 tx=0x7f757400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f75817fa640 1 -- 172.21.15.45:0/2145138526 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7574007820 con 0x7f75840ae540 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f75817fa640 1 -- 172.21.15.45:0/2145138526 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7574007980 con 0x7f75840ae540 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f75817fa640 1 -- 172.21.15.45:0/2145138526 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7574007cf0 con 0x7f75840ae540 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.592+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758409a000 con 0x7f75840ae540 2023-11-28T20:35:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.593+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758409a5c0 con 0x7f75840ae540 2023-11-28T20:35:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.594+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75840b3290 con 0x7f75840ae540 2023-11-28T20:35:39.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.595+0000 7f75817fa640 1 -- 172.21.15.45:0/2145138526 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f75740113f0 con 0x7f75840ae540 2023-11-28T20:35:39.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.596+0000 7f75817fa640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f756406ae40 0x7f756406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:39.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.596+0000 7f75817fa640 1 -- 172.21.15.45:0/2145138526 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7574089af0 con 0x7f75840ae540 2023-11-28T20:35:39.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.596+0000 7f757b7fe640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f756406ae40 0x7f756406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.598+0000 7f757b7fe640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f756406ae40 0x7f756406d300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f758409aea0 tx=0x7f756c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:39.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.601+0000 7f75817fa640 1 -- 172.21.15.45:0/2145138526 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75740549d0 con 0x7f75840ae540 2023-11-28T20:35:39.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.835+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75840b1030 con 0x7f75840ae540 2023-11-28T20:35:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.836+0000 7f75817fa640 1 -- 172.21.15.45:0/2145138526 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f757401f020 con 0x7f75840ae540 2023-11-28T20:35:39.839 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:39.839 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3050590116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3050590116},{"type":"v1","addr":"172.21.15.45:6835","nonce":3050590116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2465910573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2465910573},{"type":"v1","addr":"172.21.15.190:6836","nonce":2465910573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4090116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4090116745},{"type":"v1","addr":"172.21.15.190:6837","nonce":4090116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:35:35.751211+0000","modified":"2023-11-28T20:35:38.829781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4945":{"gid":4945,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6834/1984946086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1984946086},{"type":"v1","addr":"172.21.15.45:6834","nonce":1984946086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T20:35:39.839 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 18 2023-11-28T20:35:39.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f756406ae40 msgr2=0x7f756406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:39.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f756406ae40 0x7f756406d300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f758409aea0 tx=0x7f756c009040 comp rx=0 tx=0).stop 2023-11-28T20:35:39.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ae540 msgr2=0x7f75841514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:39.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ae540 0x7f75841514b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f757400c7f0 tx=0x7f757400ccb0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 shutdown_connections 2023-11-28T20:35:39.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f756406ae40 0x7f756406d300 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7584155780 0x7f7584154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75840ce800 0x7f7584153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 --2- 172.21.15.45:0/2145138526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75840ae540 0x7f75841514b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 >> 172.21.15.45:0/2145138526 conn(0x7f758406ab30 msgr2=0x7f758415dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:39.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.839+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 shutdown_connections 2023-11-28T20:35:39.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:39.840+0000 7f7589727640 1 -- 172.21.15.45:0/2145138526 wait complete. 2023-11-28T20:35:39.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:35:40.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.087+0000 7fbfe88ce640 1 Processor -- start 2023-11-28T20:35:40.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.087+0000 7fbfe88ce640 1 -- start start 2023-11-28T20:35:40.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.089+0000 7fbfe88ce640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe4155780 0x7fbfe40abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.089+0000 7fbfe88ce640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbfe40ce800 0x7fbfe40b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.089+0000 7fbfe88ce640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe415c3c0 con 0x7fbfe40ae540 2023-11-28T20:35:40.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.089+0000 7fbfe88ce640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe4155b60 con 0x7fbfe40ce800 2023-11-28T20:35:40.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.089+0000 7fbfe88ce640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe415ba90 con 0x7fbfe4155780 2023-11-28T20:35:40.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.089+0000 7fbfdbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbfe40ae540 0x7fbfe40ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40794/0 (socket says 172.21.15.45:40794) 2023-11-28T20:35:40.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.090+0000 7fbfdbfff640 1 -- 172.21.15.45:0/2137954476 learned_addr learned my addr 172.21.15.45:0/2137954476 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:40.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.090+0000 7fbfe359e640 1 --2- 172.21.15.45:0/2137954476 >> v2:172.21.15.45:3300/0 conn(0x7fbfe40ce800 0x7fbfe40b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.090+0000 7fbfe2d9d640 1 --2- 172.21.15.45:0/2137954476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe4155780 0x7fbfe40abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.090+0000 7fbfe359e640 1 -- 172.21.15.45:0/2137954476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe4155780 msgr2=0x7fbfe40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.090+0000 7fbfe359e640 1 --2- 172.21.15.45:0/2137954476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe4155780 0x7fbfe40abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.090+0000 7fbfe359e640 1 -- 172.21.15.45:0/2137954476 >> v1:172.21.15.45:6789/0 conn(0x7fbfe40ae540 legacy=0x7fbfe40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.090+0000 7fbfe359e640 1 -- 172.21.15.45:0/2137954476 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe415bd70 con 0x7fbfe40ce800 2023-11-28T20:35:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.091+0000 7fbfe359e640 1 --2- 172.21.15.45:0/2137954476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe40ce800 0x7fbfe40b20a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009d90 tx=0x7fbfdc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e667ad9ea7d409ca server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe259c640 1 -- 172.21.15.45:0/2137954476 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc02cc60 con 0x7fbfe40ce800 2023-11-28T20:35:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe259c640 1 -- 172.21.15.45:0/2137954476 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc02cdc0 con 0x7fbfe40ce800 2023-11-28T20:35:40.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe259c640 1 -- 172.21.15.45:0/2137954476 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc034660 con 0x7fbfe40ce800 2023-11-28T20:35:40.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2137954476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe40ce800 msgr2=0x7fbfe40b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe88ce640 1 --2- 172.21.15.45:0/2137954476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe40ce800 0x7fbfe40b20a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009d90 tx=0x7fbfdc02beb0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2137954476 shutdown_connections 2023-11-28T20:35:40.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe88ce640 1 --2- 172.21.15.45:0/2137954476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe40ce800 0x7fbfe40b20a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe88ce640 1 --2- 172.21.15.45:0/2137954476 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe4155780 0x7fbfe40abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2137954476 >> 172.21.15.45:0/2137954476 conn(0x7fbfe406ab30 msgr2=0x7fbfe40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:40.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2137954476 shutdown_connections 2023-11-28T20:35:40.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.092+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2137954476 wait complete. 2023-11-28T20:35:40.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.094+0000 7fbfe88ce640 1 Processor -- start 2023-11-28T20:35:40.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.094+0000 7fbfe88ce640 1 -- start start 2023-11-28T20:35:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.095+0000 7fbfe88ce640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe40ae540 0x7fbfe4153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.095+0000 7fbfe88ce640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe40ce800 0x7fbfe414dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.095+0000 7fbfe88ce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe4155780 0x7fbfe409be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.095+0000 7fbfe88ce640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe409b280 con 0x7fbfe4155780 2023-11-28T20:35:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.095+0000 7fbfe88ce640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe409afa0 con 0x7fbfe40ae540 2023-11-28T20:35:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.095+0000 7fbfe88ce640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41531b0 con 0x7fbfe40ce800 2023-11-28T20:35:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe4155780 0x7fbfe409be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe4155780 0x7fbfe409be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59458/0 (socket says 172.21.15.45:59458) 2023-11-28T20:35:40.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe359e640 1 -- 172.21.15.45:0/2357548193 learned_addr learned my addr 172.21.15.45:0/2357548193 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:40.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfdbfff640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe40ce800 0x7fbfe414dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe359e640 1 -- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe40ce800 msgr2=0x7fbfe414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe359e640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe40ce800 0x7fbfe414dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe2d9d640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe40ae540 0x7fbfe4153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe359e640 1 -- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe40ae540 msgr2=0x7fbfe4153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe359e640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe40ae540 0x7fbfe4153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe359e640 1 -- 172.21.15.45:0/2357548193 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc009a70 con 0x7fbfe4155780 2023-11-28T20:35:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.096+0000 7fbfe359e640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe4155780 0x7fbfe409be00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009d90 tx=0x7fbfdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.097+0000 7fbfdb7fe640 1 -- 172.21.15.45:0/2357548193 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc040650 con 0x7fbfe4155780 2023-11-28T20:35:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.097+0000 7fbfdb7fe640 1 -- 172.21.15.45:0/2357548193 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc0407d0 con 0x7fbfe4155780 2023-11-28T20:35:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.097+0000 7fbfdb7fe640 1 -- 172.21.15.45:0/2357548193 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc040b80 con 0x7fbfe4155780 2023-11-28T20:35:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.097+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe4153490 con 0x7fbfe4155780 2023-11-28T20:35:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.097+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe414e1b0 con 0x7fbfe4155780 2023-11-28T20:35:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.098+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe4153010 con 0x7fbfe4155780 2023-11-28T20:35:40.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.099+0000 7fbfdb7fe640 1 -- 172.21.15.45:0/2357548193 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbfdc040ce0 con 0x7fbfe4155780 2023-11-28T20:35:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.100+0000 7fbfdb7fe640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfc406ae70 0x7fbfc406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.101+0000 7fbfdb7fe640 1 -- 172.21.15.45:0/2357548193 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbfdc0ac610 con 0x7fbfe4155780 2023-11-28T20:35:40.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.101+0000 7fbfe2d9d640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfc406ae70 0x7fbfc406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.102+0000 7fbfe2d9d640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfc406ae70 0x7fbfc406d330 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4006fd0 tx=0x7fbfd4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:40.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.105+0000 7fbfdb7fe640 1 -- 172.21.15.45:0/2357548193 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbfe4153010 con 0x7fbfe4155780 2023-11-28T20:35:40.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.299+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfe41539b0 con 0x7fbfe4155780 2023-11-28T20:35:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.302+0000 7fbfdb7fe640 1 -- 172.21.15.45:0/2357548193 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16762 (secure 0 0 0) 0x7fbfe41539b0 con 0x7fbfe4155780 2023-11-28T20:35:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:40.305 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":39,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:35:37.285974+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:35:33.849797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-28T20:35:34.783808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:35:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfc406ae70 msgr2=0x7fbfc406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfc406ae70 0x7fbfc406d330 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4006fd0 tx=0x7fbfd4008040 comp rx=0 tx=0).stop 2023-11-28T20:35:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe4155780 msgr2=0x7fbfe409be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe4155780 0x7fbfe409be00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009d90 tx=0x7fbfdc004a40 comp rx=0 tx=0).stop 2023-11-28T20:35:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 shutdown_connections 2023-11-28T20:35:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fbfc406ae70 0x7fbfc406d330 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe4155780 0x7fbfe409be00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfe40ce800 0x7fbfe414dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 --2- 172.21.15.45:0/2357548193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe40ae540 0x7fbfe4153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 >> 172.21.15.45:0/2357548193 conn(0x7fbfe406ab30 msgr2=0x7fbfe415ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 shutdown_connections 2023-11-28T20:35:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.305+0000 7fbfe88ce640 1 -- 172.21.15.45:0/2357548193 wait complete. 2023-11-28T20:35:40.328 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T20:35:40.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:35:40.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.546+0000 7f95db4cf640 1 Processor -- start 2023-11-28T20:35:40.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.546+0000 7f95db4cf640 1 -- start start 2023-11-28T20:35:40.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.548+0000 7f95db4cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95d415b850 0x7f95d40b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.548+0000 7f95db4cf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d415bda0 con 0x7f95d40b35b0 2023-11-28T20:35:40.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.548+0000 7f95db4cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d415c080 con 0x7f95d40b1fa0 2023-11-28T20:35:40.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.548+0000 7f95db4cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d415ee40 con 0x7f95d415b850 2023-11-28T20:35:40.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.548+0000 7f95d926b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f95d40b35b0 0x7f95d415ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40798/0 (socket says 172.21.15.45:40798) 2023-11-28T20:35:40.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.548+0000 7f95d926b640 1 -- 172.21.15.45:0/3185354435 learned_addr learned my addr 172.21.15.45:0/3185354435 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95d8a6a640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95d415b850 0x7f95d40b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248230764 0 0) 0x7f95d415bda0 con 0x7f95d40b35b0 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c0003520 con 0x7f95d40b35b0 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048251650 0 0) 0x7f95c0003520 con 0x7f95d40b35b0 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95d415b850 msgr2=0x7f95d40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95d415b850 0x7f95d40b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 >> v1:172.21.15.190:6789/0 conn(0x7f95d40b1fa0 legacy=0x7f95d415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d415c360 con 0x7f95d40b35b0 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f95d0001670 con 0x7f95d40b35b0 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 >> v1:172.21.15.45:6789/0 conn(0x7f95d40b35b0 legacy=0x7f95d415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95c00039a0 0x7f95c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c00047c0 0x7f95c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95d415b850 0x7f95c00051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d415ee40 con 0x7f95c00039a0 2023-11-28T20:35:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c000ba90 con 0x7f95c00047c0 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.549+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c000bc50 con 0x7f95d415b850 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d926b640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95c00039a0 0x7f95c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d8a6a640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c00047c0 0x7f95c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d926b640 1 -- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95d415b850 msgr2=0x7f95c00051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d9a6c640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95d415b850 0x7f95c00051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d926b640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95d415b850 0x7f95c00051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d926b640 1 -- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c00047c0 msgr2=0x7f95c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d926b640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c00047c0 0x7f95c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d926b640 1 -- 172.21.15.45:0/3185354435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c0003520 con 0x7f95c00039a0 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d8a6a640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c00047c0 0x7f95c0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:40.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.550+0000 7f95d9a6c640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95d415b850 0x7f95c00051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:40.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.551+0000 7f95d926b640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95c00039a0 0x7f95c0003dc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f95d000c0e0 tx=0x7f95c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4aecdcbedda4b9e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:40.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.551+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0005ad0 con 0x7f95c00039a0 2023-11-28T20:35:40.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.551+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d0005c50 con 0x7f95c00039a0 2023-11-28T20:35:40.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.551+0000 7f95cbfff640 1 -- 172.21.15.45:0/3185354435 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d002c810 con 0x7f95c00039a0 2023-11-28T20:35:40.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.552+0000 7f95db4cf640 1 -- 172.21.15.45:0/3185354435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95c00039a0 msgr2=0x7f95c0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.552+0000 7f95db4cf640 1 --2- 172.21.15.45:0/3185354435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95c00039a0 0x7f95c0003dc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f95d000c0e0 tx=0x7f95c00057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.552+0000 7f95d9a6c640 1 -- 172.21.15.45:0/3185354435 reap_dead start 2023-11-28T20:35:40.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.552+0000 7f95db4cf640 1 -- 172.21.15.45:0/3185354435 shutdown_connections 2023-11-28T20:35:40.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.552+0000 7f95db4cf640 1 -- 172.21.15.45:0/3185354435 >> 172.21.15.45:0/3185354435 conn(0x7f95d406a500 msgr2=0x7f95d406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:40.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.552+0000 7f95db4cf640 1 -- 172.21.15.45:0/3185354435 shutdown_connections 2023-11-28T20:35:40.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.552+0000 7f95db4cf640 1 -- 172.21.15.45:0/3185354435 wait complete. 2023-11-28T20:35:40.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.554+0000 7f95db4cf640 1 Processor -- start 2023-11-28T20:35:40.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.554+0000 7f95db4cf640 1 -- start start 2023-11-28T20:35:40.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.555+0000 7f95db4cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95c00039a0 0x7f95d4142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.555+0000 7f95db4cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95d40ad030 0x7f95d40ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.555+0000 7f95db4cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95d40cdae0 0x7f95d40cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.555+0000 7f95db4cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d40ce4a0 con 0x7f95d40cdae0 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.555+0000 7f95db4cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d41f2cd0 con 0x7f95d40ad030 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.555+0000 7f95db4cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d40706a0 con 0x7f95c00039a0 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.555+0000 7f95d9a6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95d40cdae0 0x7f95d40cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.555+0000 7f95d9a6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95d40cdae0 0x7f95d40cdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59484/0 (socket says 172.21.15.45:59484) 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.555+0000 7f95d9a6c640 1 -- 172.21.15.45:0/1684014785 learned_addr learned my addr 172.21.15.45:0/1684014785 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d926b640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95c00039a0 0x7f95d4142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d8a6a640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95d40ad030 0x7f95d40ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d9a6c640 1 -- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95c00039a0 msgr2=0x7f95d4142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d9a6c640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95c00039a0 0x7f95d4142ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d9a6c640 1 -- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95d40ad030 msgr2=0x7f95d40ad4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d9a6c640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95d40ad030 0x7f95d40ad4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d9a6c640 1 -- 172.21.15.45:0/1684014785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d4070980 con 0x7f95d40cdae0 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d8a6a640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95d40ad030 0x7f95d40ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d926b640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95c00039a0 0x7f95d4142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95d9a6c640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95d40cdae0 0x7f95d40cdf60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f95d40b2920 tx=0x7f95cc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95c9ffb640 1 -- 172.21.15.45:0/1684014785 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc003b70 con 0x7f95d40cdae0 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95c9ffb640 1 -- 172.21.15.45:0/1684014785 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc003cf0 con 0x7f95d40cdae0 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95c9ffb640 1 -- 172.21.15.45:0/1684014785 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc0052b0 con 0x7f95d40cdae0 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.556+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d4070c60 con 0x7f95d40cdae0 2023-11-28T20:35:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.557+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d4070330 con 0x7f95d40cdae0 2023-11-28T20:35:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.558+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d41f2e40 con 0x7f95d40cdae0 2023-11-28T20:35:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.559+0000 7f95c9ffb640 1 -- 172.21.15.45:0/1684014785 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f95cc005410 con 0x7f95d40cdae0 2023-11-28T20:35:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.559+0000 7f95c9ffb640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f95b806ae40 0x7f95b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.560+0000 7f95c9ffb640 1 -- 172.21.15.45:0/1684014785 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f95cc089090 con 0x7f95d40cdae0 2023-11-28T20:35:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.560+0000 7f95d926b640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f95b806ae40 0x7f95b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:40.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.561+0000 7f95d926b640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f95b806ae40 0x7f95b806d300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f95d0034000 tx=0x7f95d0009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:40.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.564+0000 7f95c9ffb640 1 -- 172.21.15.45:0/1684014785 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95cc053ff0 con 0x7f95d40cdae0 2023-11-28T20:35:40.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.747+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95d40b3990 con 0x7f95b806ae40 2023-11-28T20:35:40.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.774+0000 7f95c9ffb640 1 -- 172.21.15.45:0/1684014785 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281834 (secure 0 0 0) 0x7f95d0001530 con 0x7f95b806ae40 2023-11-28T20:35:40.778 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:35:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f95b806ae40 msgr2=0x7f95b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f95b806ae40 0x7f95b806d300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f95d0034000 tx=0x7f95d0009030 comp rx=0 tx=0).stop 2023-11-28T20:35:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95d40cdae0 msgr2=0x7f95d40cdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95d40cdae0 0x7f95d40cdf60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f95d40b2920 tx=0x7f95cc00dc00 comp rx=0 tx=0).stop 2023-11-28T20:35:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 shutdown_connections 2023-11-28T20:35:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f95b806ae40 0x7f95b806d300 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95d40cdae0 0x7f95d40cdf60 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95d40ad030 0x7f95d40ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 --2- 172.21.15.45:0/1684014785 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95c00039a0 0x7f95d4142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.780+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 >> 172.21.15.45:0/1684014785 conn(0x7f95d406a500 msgr2=0x7f95d40b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.781+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 shutdown_connections 2023-11-28T20:35:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:40.781+0000 7f95db4cf640 1 -- 172.21.15.45:0/1684014785 wait complete. 2023-11-28T20:35:40.802 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2023-11-28T20:35:39.165031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237292,"kb_used_data":4488,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737428,"statfs":{"total":773094113280,"available":772851126272,"internally_reserved":0,"allocated":4595712,"data_stored":2616256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.902590"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.273744+0000","last_change":"2023-11-28T20:35:35.273744+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.273744+0000","last_undegraded":"2023-11-28T20:35:35.273744+0000","last_fullsized":"2023-11-28T20:35:35.273744+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274640+0000","last_change":"2023-11-28T20:35:35.274640+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274640+0000","last_undegraded":"2023-11-28T20:35:35.274640+0000","last_fullsized":"2023-11-28T20:35:35.274640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287799+0000","last_change":"2023-11-28T20:35:35.287799+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287799+0000","last_undegraded":"2023-11-28T20:35:35.287799+0000","last_fullsized":"2023-11-28T20:35:35.287799+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.283269+0000","last_change":"2023-11-28T20:35:35.283269+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.283269+0000","last_undegraded":"2023-11-28T20:35:35.283269+0000","last_fullsized":"2023-11-28T20:35:35.283269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284083+0000","last_change":"2023-11-28T20:35:35.284083+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284083+0000","last_undegraded":"2023-11-28T20:35:35.284083+0000","last_fullsized":"2023-11-28T20:35:35.284083+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289049+0000","last_change":"2023-11-28T20:35:35.289049+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289049+0000","last_undegraded":"2023-11-28T20:35:35.289049+0000","last_fullsized":"2023-11-28T20:35:35.289049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.300220+0000","last_change":"2023-11-28T20:35:35.300220+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.300220+0000","last_undegraded":"2023-11-28T20:35:35.300220+0000","last_fullsized":"2023-11-28T20:35:35.300220+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275156+0000","last_change":"2023-11-28T20:35:35.275156+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275156+0000","last_undegraded":"2023-11-28T20:35:35.275156+0000","last_fullsized":"2023-11-28T20:35:35.275156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288046+0000","last_change":"2023-11-28T20:35:35.288046+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288046+0000","last_undegraded":"2023-11-28T20:35:35.288046+0000","last_fullsized":"2023-11-28T20:35:35.288046+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.302551+0000","last_change":"2023-11-28T20:35:35.302551+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.302551+0000","last_undegraded":"2023-11-28T20:35:35.302551+0000","last_fullsized":"2023-11-28T20:35:35.302551+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.292849+0000","last_change":"2023-11-28T20:35:35.292849+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.292849+0000","last_undegraded":"2023-11-28T20:35:35.292849+0000","last_fullsized":"2023-11-28T20:35:35.292849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286256+0000","last_change":"2023-11-28T20:35:35.286256+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286256+0000","last_undegraded":"2023-11-28T20:35:35.286256+0000","last_fullsized":"2023-11-28T20:35:35.286256+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296324+0000","last_change":"2023-11-28T20:35:35.296324+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296324+0000","last_undegraded":"2023-11-28T20:35:35.296324+0000","last_fullsized":"2023-11-28T20:35:35.296324+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280741+0000","last_change":"2023-11-28T20:35:35.280741+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280741+0000","last_undegraded":"2023-11-28T20:35:35.280741+0000","last_fullsized":"2023-11-28T20:35:35.280741+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.298641+0000","last_change":"2023-11-28T20:35:35.298641+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.298641+0000","last_undegraded":"2023-11-28T20:35:35.298641+0000","last_fullsized":"2023-11-28T20:35:35.298641+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288030+0000","last_change":"2023-11-28T20:35:35.288030+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288030+0000","last_undegraded":"2023-11-28T20:35:35.288030+0000","last_fullsized":"2023-11-28T20:35:35.288030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.294765+0000","last_change":"2023-11-28T20:35:35.294765+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.294765+0000","last_undegraded":"2023-11-28T20:35:35.294765+0000","last_fullsized":"2023-11-28T20:35:35.294765+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286541+0000","last_change":"2023-11-28T20:35:35.286541+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286541+0000","last_undegraded":"2023-11-28T20:35:35.286541+0000","last_fullsized":"2023-11-28T20:35:35.286541+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.281506+0000","last_change":"2023-11-28T20:35:35.281506+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.281506+0000","last_undegraded":"2023-11-28T20:35:35.281506+0000","last_fullsized":"2023-11-28T20:35:35.281506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284804+0000","last_change":"2023-11-28T20:35:35.284804+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284804+0000","last_undegraded":"2023-11-28T20:35:35.284804+0000","last_fullsized":"2023-11-28T20:35:35.284804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.295351+0000","last_change":"2023-11-28T20:35:35.295351+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.295351+0000","last_undegraded":"2023-11-28T20:35:35.295351+0000","last_fullsized":"2023-11-28T20:35:35.295351+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.278315+0000","last_change":"2023-11-28T20:35:35.278315+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.278315+0000","last_undegraded":"2023-11-28T20:35:35.278315+0000","last_fullsized":"2023-11-28T20:35:35.278315+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296171+0000","last_change":"2023-11-28T20:35:35.296171+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296171+0000","last_undegraded":"2023-11-28T20:35:35.296171+0000","last_fullsized":"2023-11-28T20:35:35.296171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.282313+0000","last_change":"2023-11-28T20:35:35.282313+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.282313+0000","last_undegraded":"2023-11-28T20:35:35.282313+0000","last_fullsized":"2023-11-28T20:35:35.282313+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.290562+0000","last_change":"2023-11-28T20:35:35.290562+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.290562+0000","last_undegraded":"2023-11-28T20:35:35.290562+0000","last_fullsized":"2023-11-28T20:35:35.290562+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288231+0000","last_change":"2023-11-28T20:35:35.288231+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288231+0000","last_undegraded":"2023-11-28T20:35:35.288231+0000","last_fullsized":"2023-11-28T20:35:35.288231+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286660+0000","last_change":"2023-11-28T20:35:35.286660+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286660+0000","last_undegraded":"2023-11-28T20:35:35.286660+0000","last_fullsized":"2023-11-28T20:35:35.286660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.300373+0000","last_change":"2023-11-28T20:35:35.300373+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.300373+0000","last_undegraded":"2023-11-28T20:35:35.300373+0000","last_fullsized":"2023-11-28T20:35:35.300373+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289057+0000","last_change":"2023-11-28T20:35:35.289057+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289057+0000","last_undegraded":"2023-11-28T20:35:35.289057+0000","last_fullsized":"2023-11-28T20:35:35.289057+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288500+0000","last_change":"2023-11-28T20:35:35.288500+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288500+0000","last_undegraded":"2023-11-28T20:35:35.288500+0000","last_fullsized":"2023-11-28T20:35:35.288500+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286602+0000","last_change":"2023-11-28T20:35:35.286602+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286602+0000","last_undegraded":"2023-11-28T20:35:35.286602+0000","last_fullsized":"2023-11-28T20:35:35.286602+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284839+0000","last_change":"2023-11-28T20:35:35.284839+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284839+0000","last_undegraded":"2023-11-28T20:35:35.284839+0000","last_fullsized":"2023-11-28T20:35:35.284839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.294772+0000","last_change":"2023-11-28T20:35:35.294772+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.294772+0000","last_undegraded":"2023-11-28T20:35:35.294772+0000","last_fullsized":"2023-11-28T20:35:35.294772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310344+0000","last_change":"2023-11-28T20:35:35.304535+0000","last_active":"2023-11-28T20:35:35.310344+0000","last_peered":"2023-11-28T20:35:35.310344+0000","last_clean":"2023-11-28T20:35:35.310344+0000","last_became_active":"2023-11-28T20:35:35.304193+0000","last_became_peered":"2023-11-28T20:35:35.304193+0000","last_unstale":"2023-11-28T20:35:35.310344+0000","last_undegraded":"2023-11-28T20:35:35.310344+0000","last_fullsized":"2023-11-28T20:35:35.310344+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:14.497404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289588+0000","last_change":"2023-11-28T20:35:35.289588+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289588+0000","last_undegraded":"2023-11-28T20:35:35.289588+0000","last_fullsized":"2023-11-28T20:35:35.289588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302107+0000","last_change":"2023-11-28T20:35:35.300035+0000","last_active":"2023-11-28T20:35:35.302107+0000","last_peered":"2023-11-28T20:35:35.302107+0000","last_clean":"2023-11-28T20:35:35.302107+0000","last_became_active":"2023-11-28T20:35:35.299808+0000","last_became_peered":"2023-11-28T20:35:35.299808+0000","last_unstale":"2023-11-28T20:35:35.302107+0000","last_undegraded":"2023-11-28T20:35:35.302107+0000","last_fullsized":"2023-11-28T20:35:35.302107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276409+0000","last_change":"2023-11-28T20:35:35.276409+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276409+0000","last_undegraded":"2023-11-28T20:35:35.276409+0000","last_fullsized":"2023-11-28T20:35:35.276409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.304864+0000","last_change":"2023-11-28T20:35:35.303683+0000","last_active":"2023-11-28T20:35:35.304864+0000","last_peered":"2023-11-28T20:35:35.304864+0000","last_clean":"2023-11-28T20:35:35.304864+0000","last_became_active":"2023-11-28T20:35:35.303422+0000","last_became_peered":"2023-11-28T20:35:35.303422+0000","last_unstale":"2023-11-28T20:35:35.304864+0000","last_undegraded":"2023-11-28T20:35:35.304864+0000","last_fullsized":"2023-11-28T20:35:35.304864+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274604+0000","last_change":"2023-11-28T20:35:35.274604+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274604+0000","last_undegraded":"2023-11-28T20:35:35.274604+0000","last_fullsized":"2023-11-28T20:35:35.274604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.298920+0000","last_change":"2023-11-28T20:35:35.290259+0000","last_active":"2023-11-28T20:35:35.298920+0000","last_peered":"2023-11-28T20:35:35.298920+0000","last_clean":"2023-11-28T20:35:35.298920+0000","last_became_active":"2023-11-28T20:35:35.286178+0000","last_became_peered":"2023-11-28T20:35:35.286178+0000","last_unstale":"2023-11-28T20:35:35.298920+0000","last_undegraded":"2023-11-28T20:35:35.298920+0000","last_fullsized":"2023-11-28T20:35:35.298920+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.285970+0000","last_change":"2023-11-28T20:35:35.285970+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.285970+0000","last_undegraded":"2023-11-28T20:35:35.285970+0000","last_fullsized":"2023-11-28T20:35:35.285970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310834+0000","last_change":"2023-11-28T20:35:35.306240+0000","last_active":"2023-11-28T20:35:35.310834+0000","last_peered":"2023-11-28T20:35:35.310834+0000","last_clean":"2023-11-28T20:35:35.310834+0000","last_became_active":"2023-11-28T20:35:35.305889+0000","last_became_peered":"2023-11-28T20:35:35.305889+0000","last_unstale":"2023-11-28T20:35:35.310834+0000","last_undegraded":"2023-11-28T20:35:35.310834+0000","last_fullsized":"2023-11-28T20:35:35.310834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:29.262104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.298012+0000","last_change":"2023-11-28T20:35:35.298012+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.298012+0000","last_undegraded":"2023-11-28T20:35:35.298012+0000","last_fullsized":"2023-11-28T20:35:35.298012+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.313113+0000","last_change":"2023-11-28T20:35:35.311106+0000","last_active":"2023-11-28T20:35:35.313113+0000","last_peered":"2023-11-28T20:35:35.313113+0000","last_clean":"2023-11-28T20:35:35.313113+0000","last_became_active":"2023-11-28T20:35:35.310832+0000","last_became_peered":"2023-11-28T20:35:35.310832+0000","last_unstale":"2023-11-28T20:35:35.313113+0000","last_undegraded":"2023-11-28T20:35:35.313113+0000","last_fullsized":"2023-11-28T20:35:35.313113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:01:49.996784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276568+0000","last_change":"2023-11-28T20:35:35.276568+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276568+0000","last_undegraded":"2023-11-28T20:35:35.276568+0000","last_fullsized":"2023-11-28T20:35:35.276568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312883+0000","last_change":"2023-11-28T20:35:35.311801+0000","last_active":"2023-11-28T20:35:35.312883+0000","last_peered":"2023-11-28T20:35:35.312883+0000","last_clean":"2023-11-28T20:35:35.312883+0000","last_became_active":"2023-11-28T20:35:35.311381+0000","last_became_peered":"2023-11-28T20:35:35.311381+0000","last_unstale":"2023-11-28T20:35:35.312883+0000","last_undegraded":"2023-11-28T20:35:35.312883+0000","last_fullsized":"2023-11-28T20:35:35.312883+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276964+0000","last_change":"2023-11-28T20:35:35.276964+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276964+0000","last_undegraded":"2023-11-28T20:35:35.276964+0000","last_fullsized":"2023-11-28T20:35:35.276964+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310284+0000","last_change":"2023-11-28T20:35:35.305104+0000","last_active":"2023-11-28T20:35:35.310284+0000","last_peered":"2023-11-28T20:35:35.310284+0000","last_clean":"2023-11-28T20:35:35.310284+0000","last_became_active":"2023-11-28T20:35:35.304907+0000","last_became_peered":"2023-11-28T20:35:35.304907+0000","last_unstale":"2023-11-28T20:35:35.310284+0000","last_undegraded":"2023-11-28T20:35:35.310284+0000","last_fullsized":"2023-11-28T20:35:35.310284+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:44.520691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280744+0000","last_change":"2023-11-28T20:35:35.280744+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280744+0000","last_undegraded":"2023-11-28T20:35:35.280744+0000","last_fullsized":"2023-11-28T20:35:35.280744+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.296213+0000","last_change":"2023-11-28T20:35:35.295019+0000","last_active":"2023-11-28T20:35:35.296213+0000","last_peered":"2023-11-28T20:35:35.296213+0000","last_clean":"2023-11-28T20:35:35.296213+0000","last_became_active":"2023-11-28T20:35:35.294753+0000","last_became_peered":"2023-11-28T20:35:35.294753+0000","last_unstale":"2023-11-28T20:35:35.296213+0000","last_undegraded":"2023-11-28T20:35:35.296213+0000","last_fullsized":"2023-11-28T20:35:35.296213+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:24.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.293244+0000","last_change":"2023-11-28T20:35:35.293244+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.293244+0000","last_undegraded":"2023-11-28T20:35:35.293244+0000","last_fullsized":"2023-11-28T20:35:35.293244+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312174+0000","last_change":"2023-11-28T20:35:35.308107+0000","last_active":"2023-11-28T20:35:35.312174+0000","last_peered":"2023-11-28T20:35:35.312174+0000","last_clean":"2023-11-28T20:35:35.312174+0000","last_became_active":"2023-11-28T20:35:35.307828+0000","last_became_peered":"2023-11-28T20:35:35.307828+0000","last_unstale":"2023-11-28T20:35:35.312174+0000","last_undegraded":"2023-11-28T20:35:35.312174+0000","last_fullsized":"2023-11-28T20:35:35.312174+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:24.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287024+0000","last_change":"2023-11-28T20:35:35.287024+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287024+0000","last_undegraded":"2023-11-28T20:35:35.287024+0000","last_fullsized":"2023-11-28T20:35:35.287024+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311202+0000","last_change":"2023-11-28T20:35:35.307353+0000","last_active":"2023-11-28T20:35:35.311202+0000","last_peered":"2023-11-28T20:35:35.311202+0000","last_clean":"2023-11-28T20:35:35.311202+0000","last_became_active":"2023-11-28T20:35:35.307054+0000","last_became_peered":"2023-11-28T20:35:35.307054+0000","last_unstale":"2023-11-28T20:35:35.311202+0000","last_undegraded":"2023-11-28T20:35:35.311202+0000","last_fullsized":"2023-11-28T20:35:35.311202+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:01:49.996784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275888+0000","last_change":"2023-11-28T20:35:35.275888+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275888+0000","last_undegraded":"2023-11-28T20:35:35.275888+0000","last_fullsized":"2023-11-28T20:35:35.275888+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.301882+0000","last_change":"2023-11-28T20:35:35.298444+0000","last_active":"2023-11-28T20:35:35.301882+0000","last_peered":"2023-11-28T20:35:35.301882+0000","last_clean":"2023-11-28T20:35:35.301882+0000","last_became_active":"2023-11-28T20:35:35.297824+0000","last_became_peered":"2023-11-28T20:35:35.297824+0000","last_unstale":"2023-11-28T20:35:35.301882+0000","last_undegraded":"2023-11-28T20:35:35.301882+0000","last_fullsized":"2023-11-28T20:35:35.301882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:29.262104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.285405+0000","last_change":"2023-11-28T20:35:35.285405+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.285405+0000","last_undegraded":"2023-11-28T20:35:35.285405+0000","last_fullsized":"2023-11-28T20:35:35.285405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.292992+0000","last_change":"2023-11-28T20:35:35.290127+0000","last_active":"2023-11-28T20:35:35.292992+0000","last_peered":"2023-11-28T20:35:35.292992+0000","last_clean":"2023-11-28T20:35:35.292992+0000","last_became_active":"2023-11-28T20:35:35.289788+0000","last_became_peered":"2023-11-28T20:35:35.289788+0000","last_unstale":"2023-11-28T20:35:35.292992+0000","last_undegraded":"2023-11-28T20:35:35.292992+0000","last_fullsized":"2023-11-28T20:35:35.292992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.273794+0000","last_change":"2023-11-28T20:35:35.273794+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.273794+0000","last_undegraded":"2023-11-28T20:35:35.273794+0000","last_fullsized":"2023-11-28T20:35:35.273794+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311743+0000","last_change":"2023-11-28T20:35:35.310764+0000","last_active":"2023-11-28T20:35:35.311743+0000","last_peered":"2023-11-28T20:35:35.311743+0000","last_clean":"2023-11-28T20:35:35.311743+0000","last_became_active":"2023-11-28T20:35:35.310478+0000","last_became_peered":"2023-11-28T20:35:35.310478+0000","last_unstale":"2023-11-28T20:35:35.311743+0000","last_undegraded":"2023-11-28T20:35:35.311743+0000","last_fullsized":"2023-11-28T20:35:35.311743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.295908+0000","last_change":"2023-11-28T20:35:35.295908+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.295908+0000","last_undegraded":"2023-11-28T20:35:35.295908+0000","last_fullsized":"2023-11-28T20:35:35.295908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310256+0000","last_change":"2023-11-28T20:35:35.304319+0000","last_active":"2023-11-28T20:35:35.310256+0000","last_peered":"2023-11-28T20:35:35.310256+0000","last_clean":"2023-11-28T20:35:35.310256+0000","last_became_active":"2023-11-28T20:35:35.302594+0000","last_became_peered":"2023-11-28T20:35:35.302594+0000","last_unstale":"2023-11-28T20:35:35.310256+0000","last_undegraded":"2023-11-28T20:35:35.310256+0000","last_fullsized":"2023-11-28T20:35:35.310256+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:01.686608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276470+0000","last_change":"2023-11-28T20:35:35.276470+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276470+0000","last_undegraded":"2023-11-28T20:35:35.276470+0000","last_fullsized":"2023-11-28T20:35:35.276470+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312059+0000","last_change":"2023-11-28T20:35:35.310387+0000","last_active":"2023-11-28T20:35:35.312059+0000","last_peered":"2023-11-28T20:35:35.312059+0000","last_clean":"2023-11-28T20:35:35.312059+0000","last_became_active":"2023-11-28T20:35:35.310118+0000","last_became_peered":"2023-11-28T20:35:35.310118+0000","last_unstale":"2023-11-28T20:35:35.312059+0000","last_undegraded":"2023-11-28T20:35:35.312059+0000","last_fullsized":"2023-11-28T20:35:35.312059+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275327+0000","last_change":"2023-11-28T20:35:35.275327+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275327+0000","last_undegraded":"2023-11-28T20:35:35.275327+0000","last_fullsized":"2023-11-28T20:35:35.275327+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311828+0000","last_change":"2023-11-28T20:35:35.308420+0000","last_active":"2023-11-28T20:35:35.311828+0000","last_peered":"2023-11-28T20:35:35.311828+0000","last_clean":"2023-11-28T20:35:35.311828+0000","last_became_active":"2023-11-28T20:35:35.305473+0000","last_became_peered":"2023-11-28T20:35:35.305473+0000","last_unstale":"2023-11-28T20:35:35.311828+0000","last_undegraded":"2023-11-28T20:35:35.311828+0000","last_fullsized":"2023-11-28T20:35:35.311828+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.292286+0000","last_change":"2023-11-28T20:35:35.292286+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.292286+0000","last_undegraded":"2023-11-28T20:35:35.292286+0000","last_fullsized":"2023-11-28T20:35:35.292286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.313168+0000","last_change":"2023-11-28T20:35:35.311195+0000","last_active":"2023-11-28T20:35:35.313168+0000","last_peered":"2023-11-28T20:35:35.313168+0000","last_clean":"2023-11-28T20:35:35.313168+0000","last_became_active":"2023-11-28T20:35:35.310875+0000","last_became_peered":"2023-11-28T20:35:35.310875+0000","last_unstale":"2023-11-28T20:35:35.313168+0000","last_undegraded":"2023-11-28T20:35:35.313168+0000","last_fullsized":"2023-11-28T20:35:35.313168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.277762+0000","last_change":"2023-11-28T20:35:35.277762+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.277762+0000","last_undegraded":"2023-11-28T20:35:35.277762+0000","last_fullsized":"2023-11-28T20:35:35.277762+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302313+0000","last_change":"2023-11-28T20:35:35.298942+0000","last_active":"2023-11-28T20:35:35.302313+0000","last_peered":"2023-11-28T20:35:35.302313+0000","last_clean":"2023-11-28T20:35:35.302313+0000","last_became_active":"2023-11-28T20:35:35.298685+0000","last_became_peered":"2023-11-28T20:35:35.298685+0000","last_unstale":"2023-11-28T20:35:35.302313+0000","last_undegraded":"2023-11-28T20:35:35.302313+0000","last_fullsized":"2023-11-28T20:35:35.302313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:45.323950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280904+0000","last_change":"2023-11-28T20:35:35.280904+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280904+0000","last_undegraded":"2023-11-28T20:35:35.280904+0000","last_fullsized":"2023-11-28T20:35:35.280904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.304860+0000","last_change":"2023-11-28T20:35:35.303747+0000","last_active":"2023-11-28T20:35:35.304860+0000","last_peered":"2023-11-28T20:35:35.304860+0000","last_clean":"2023-11-28T20:35:35.304860+0000","last_became_active":"2023-11-28T20:35:35.303441+0000","last_became_peered":"2023-11-28T20:35:35.303441+0000","last_unstale":"2023-11-28T20:35:35.304860+0000","last_undegraded":"2023-11-28T20:35:35.304860+0000","last_fullsized":"2023-11-28T20:35:35.304860+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:44.520691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284186+0000","last_change":"2023-11-28T20:35:35.284186+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284186+0000","last_undegraded":"2023-11-28T20:35:35.284186+0000","last_fullsized":"2023-11-28T20:35:35.284186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.292924+0000","last_change":"2023-11-28T20:35:35.290919+0000","last_active":"2023-11-28T20:35:35.292924+0000","last_peered":"2023-11-28T20:35:35.292924+0000","last_clean":"2023-11-28T20:35:35.292924+0000","last_became_active":"2023-11-28T20:35:35.290407+0000","last_became_peered":"2023-11-28T20:35:35.290407+0000","last_unstale":"2023-11-28T20:35:35.292924+0000","last_undegraded":"2023-11-28T20:35:35.292924+0000","last_fullsized":"2023-11-28T20:35:35.292924+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:51.691786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286534+0000","last_change":"2023-11-28T20:35:35.286534+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286534+0000","last_undegraded":"2023-11-28T20:35:35.286534+0000","last_fullsized":"2023-11-28T20:35:35.286534+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302345+0000","last_change":"2023-11-28T20:35:35.299354+0000","last_active":"2023-11-28T20:35:35.302345+0000","last_peered":"2023-11-28T20:35:35.302345+0000","last_clean":"2023-11-28T20:35:35.302345+0000","last_became_active":"2023-11-28T20:35:35.299056+0000","last_became_peered":"2023-11-28T20:35:35.299056+0000","last_unstale":"2023-11-28T20:35:35.302345+0000","last_undegraded":"2023-11-28T20:35:35.302345+0000","last_fullsized":"2023-11-28T20:35:35.302345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:18.168444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.297513+0000","last_change":"2023-11-28T20:35:35.297513+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.297513+0000","last_undegraded":"2023-11-28T20:35:35.297513+0000","last_fullsized":"2023-11-28T20:35:35.297513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310069+0000","last_change":"2023-11-28T20:35:35.306856+0000","last_active":"2023-11-28T20:35:35.310069+0000","last_peered":"2023-11-28T20:35:35.310069+0000","last_clean":"2023-11-28T20:35:35.310069+0000","last_became_active":"2023-11-28T20:35:35.306562+0000","last_became_peered":"2023-11-28T20:35:35.306562+0000","last_unstale":"2023-11-28T20:35:35.310069+0000","last_undegraded":"2023-11-28T20:35:35.310069+0000","last_fullsized":"2023-11-28T20:35:35.310069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275988+0000","last_change":"2023-11-28T20:35:35.275988+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275988+0000","last_undegraded":"2023-11-28T20:35:35.275988+0000","last_fullsized":"2023-11-28T20:35:35.275988+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.293453+0000","last_change":"2023-11-28T20:35:35.291161+0000","last_active":"2023-11-28T20:35:35.293453+0000","last_peered":"2023-11-28T20:35:35.293453+0000","last_clean":"2023-11-28T20:35:35.293453+0000","last_became_active":"2023-11-28T20:35:35.290905+0000","last_became_peered":"2023-11-28T20:35:35.290905+0000","last_unstale":"2023-11-28T20:35:35.293453+0000","last_undegraded":"2023-11-28T20:35:35.293453+0000","last_fullsized":"2023-11-28T20:35:35.293453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:57.631654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287224+0000","last_change":"2023-11-28T20:35:35.287224+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287224+0000","last_undegraded":"2023-11-28T20:35:35.287224+0000","last_fullsized":"2023-11-28T20:35:35.287224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311849+0000","last_change":"2023-11-28T20:35:35.308961+0000","last_active":"2023-11-28T20:35:35.311849+0000","last_peered":"2023-11-28T20:35:35.311849+0000","last_clean":"2023-11-28T20:35:35.311849+0000","last_became_active":"2023-11-28T20:35:35.308687+0000","last_became_peered":"2023-11-28T20:35:35.308687+0000","last_unstale":"2023-11-28T20:35:35.311849+0000","last_undegraded":"2023-11-28T20:35:35.311849+0000","last_fullsized":"2023-11-28T20:35:35.311849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:18.243749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.291381+0000","last_change":"2023-11-28T20:35:35.291381+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.291381+0000","last_undegraded":"2023-11-28T20:35:35.291381+0000","last_fullsized":"2023-11-28T20:35:35.291381+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312886+0000","last_change":"2023-11-28T20:35:35.311276+0000","last_active":"2023-11-28T20:35:35.312886+0000","last_peered":"2023-11-28T20:35:35.312886+0000","last_clean":"2023-11-28T20:35:35.312886+0000","last_became_active":"2023-11-28T20:35:35.310955+0000","last_became_peered":"2023-11-28T20:35:35.310955+0000","last_unstale":"2023-11-28T20:35:35.312886+0000","last_undegraded":"2023-11-28T20:35:35.312886+0000","last_fullsized":"2023-11-28T20:35:35.312886+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:07:14.394525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274111+0000","last_change":"2023-11-28T20:35:35.274111+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274111+0000","last_undegraded":"2023-11-28T20:35:35.274111+0000","last_fullsized":"2023-11-28T20:35:35.274111+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.297236+0000","last_change":"2023-11-28T20:35:35.295046+0000","last_active":"2023-11-28T20:35:35.297236+0000","last_peered":"2023-11-28T20:35:35.297236+0000","last_clean":"2023-11-28T20:35:35.297236+0000","last_became_active":"2023-11-28T20:35:35.294261+0000","last_became_peered":"2023-11-28T20:35:35.294261+0000","last_unstale":"2023-11-28T20:35:35.297236+0000","last_undegraded":"2023-11-28T20:35:35.297236+0000","last_fullsized":"2023-11-28T20:35:35.297236+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284924+0000","last_change":"2023-11-28T20:35:35.284924+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284924+0000","last_undegraded":"2023-11-28T20:35:35.284924+0000","last_fullsized":"2023-11-28T20:35:35.284924+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.305226+0000","last_change":"2023-11-28T20:35:35.304486+0000","last_active":"2023-11-28T20:35:35.305226+0000","last_peered":"2023-11-28T20:35:35.305226+0000","last_clean":"2023-11-28T20:35:35.305226+0000","last_became_active":"2023-11-28T20:35:35.304168+0000","last_became_peered":"2023-11-28T20:35:35.304168+0000","last_unstale":"2023-11-28T20:35:35.305226+0000","last_undegraded":"2023-11-28T20:35:35.305226+0000","last_fullsized":"2023-11-28T20:35:35.305226+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289424+0000","last_change":"2023-11-28T20:35:35.289424+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289424+0000","last_undegraded":"2023-11-28T20:35:35.289424+0000","last_fullsized":"2023-11-28T20:35:35.289424+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.305224+0000","last_change":"2023-11-28T20:35:35.300968+0000","last_active":"2023-11-28T20:35:35.305224+0000","last_peered":"2023-11-28T20:35:35.305224+0000","last_clean":"2023-11-28T20:35:35.305224+0000","last_became_active":"2023-11-28T20:35:35.300389+0000","last_became_peered":"2023-11-28T20:35:35.300389+0000","last_unstale":"2023-11-28T20:35:35.305224+0000","last_undegraded":"2023-11-28T20:35:35.305224+0000","last_fullsized":"2023-11-28T20:35:35.305224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:18:34.499129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.279637+0000","last_change":"2023-11-28T20:35:35.279637+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.279637+0000","last_undegraded":"2023-11-28T20:35:35.279637+0000","last_fullsized":"2023-11-28T20:35:35.279637+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.298860+0000","last_change":"2023-11-28T20:35:35.289145+0000","last_active":"2023-11-28T20:35:35.298860+0000","last_peered":"2023-11-28T20:35:35.298860+0000","last_clean":"2023-11-28T20:35:35.298860+0000","last_became_active":"2023-11-28T20:35:35.288419+0000","last_became_peered":"2023-11-28T20:35:35.288419+0000","last_unstale":"2023-11-28T20:35:35.298860+0000","last_undegraded":"2023-11-28T20:35:35.298860+0000","last_fullsized":"2023-11-28T20:35:35.298860+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:14.497404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296014+0000","last_change":"2023-11-28T20:35:35.296014+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296014+0000","last_undegraded":"2023-11-28T20:35:35.296014+0000","last_fullsized":"2023-11-28T20:35:35.296014+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.295837+0000","last_change":"2023-11-28T20:35:35.293571+0000","last_active":"2023-11-28T20:35:35.295837+0000","last_peered":"2023-11-28T20:35:35.295837+0000","last_clean":"2023-11-28T20:35:35.295837+0000","last_became_active":"2023-11-28T20:35:35.293294+0000","last_became_peered":"2023-11-28T20:35:35.293294+0000","last_unstale":"2023-11-28T20:35:35.295837+0000","last_undegraded":"2023-11-28T20:35:35.295837+0000","last_fullsized":"2023-11-28T20:35:35.295837+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:10.536815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.293482+0000","last_change":"2023-11-28T20:35:35.293482+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.293482+0000","last_undegraded":"2023-11-28T20:35:35.293482+0000","last_fullsized":"2023-11-28T20:35:35.293482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.308144+0000","last_change":"2023-11-28T20:35:35.305966+0000","last_active":"2023-11-28T20:35:35.308144+0000","last_peered":"2023-11-28T20:35:35.308144+0000","last_clean":"2023-11-28T20:35:35.308144+0000","last_became_active":"2023-11-28T20:35:35.305715+0000","last_became_peered":"2023-11-28T20:35:35.305715+0000","last_unstale":"2023-11-28T20:35:35.308144+0000","last_undegraded":"2023-11-28T20:35:35.308144+0000","last_fullsized":"2023-11-28T20:35:35.308144+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:18.243749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-11-28T20:35:34.277946+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:35:34.277946+0000","last_peered":"2023-11-28T20:35:34.277946+0000","last_clean":"2023-11-28T20:35:34.277946+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:35:34.277946+0000","last_undegraded":"2023-11-28T20:35:34.277946+0000","last_fullsized":"2023-11-28T20:35:34.277946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":769808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.753},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.2370000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.048},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.3839999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.355}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.151},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.5259999999999998,"5min":2.5259999999999998,"15min":2.5259999999999998},"last":1.1120000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.4390000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"last":1.4690000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.619,"15min":1.619},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.516},{"interface":"front","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.7909999999999999,"5min":2.7909999999999999,"15min":2.7909999999999999},"last":1.28}]}]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":5.0430000000000001,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.242}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.736},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":1.54}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":1.637}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":4.7649999999999997,"5min":4.7649999999999997,"15min":4.7649999999999997},"last":1.1870000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":3.1989999999999998,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":3.6429999999999998,"5min":3.6429999999999998,"15min":3.6429999999999998},"last":1.8120000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.917,"5min":1.917,"15min":1.917},"min":{"1min":1.018,"5min":1.018,"15min":1.018},"max":{"1min":4.9790000000000001,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.5700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.216},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.268}]},{"osd":1,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.877},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.071}]},{"osd":2,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":1.1259999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":1.2410000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":1.4019999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.9319999999999999,"5min":2.9319999999999999,"15min":2.9319999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":1.3049999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":0.997}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.83099999999999996}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.167},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.98399999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.419},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":0.88100000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.3140000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.268},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.2270000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.627},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.3520000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":2.54,"15min":2.54},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":20.568999999999999,"5min":20.568999999999999,"15min":20.568999999999999},"last":1.3},{"interface":"front","average":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":20.677,"5min":20.677,"15min":20.677},"last":1.5029999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.80500000000000005}]},{"osd":1,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.7090000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":0.624},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.4950000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.71999999999999997}]},{"osd":2,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":5.6509999999999998,"5min":5.6509999999999998,"15min":5.6509999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.2470000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.399},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":0.872}]},{"osd":5,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.514,"15min":1.514},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.7610000000000001,"5min":2.7610000000000001,"15min":2.7610000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.492}]},{"osd":6,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.98,"5min":2.98,"15min":2.98},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.585,"15min":1.585},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.5289999999999999,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.1339999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.556},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.1859999999999999,"5min":3.1859999999999999,"15min":3.1859999999999999},"last":0.76200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.625},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.57299999999999995}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.79400000000000004}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.83699999999999997}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.3160000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":1.113}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.931,"5min":2.931,"15min":2.931},"last":1.452},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.202}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.9649999999999999,"5min":2.9649999999999999,"15min":2.9649999999999999},"last":1.395}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.89600000000000002}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.637}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.276},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.165}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":3.5259999999999998,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":1.339}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":3.637,"5min":3.637,"15min":3.637},"last":1.484},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.534}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":1.5920000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.395},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.4660000000000002,"5min":4.4660000000000002,"15min":4.4660000000000002},"last":1.5109999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":769808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.964,"5min":2.964,"15min":2.964},"last":0.61099999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.504},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.52500000000000002}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.68300000000000005}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.89700000000000002}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.536,"5min":2.536,"15min":2.536},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":0.96099999999999997}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":2.8650000000000002,"5min":2.8650000000000002,"15min":2.8650000000000002},"last":1.085},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":1.123,"5min":1.123,"15min":1.123},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:35:40.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:35:41.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.069+0000 7fc078ada640 1 Processor -- start 2023-11-28T20:35:41.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.069+0000 7fc078ada640 1 -- start start 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.071+0000 7fc078ada640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc0740ae540 0x7fc0740ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.071+0000 7fc078ada640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc0740ce630 0x7fc0740b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.071+0000 7fc078ada640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07415c3c0 con 0x7fc074155780 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.071+0000 7fc078ada640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0740ceac0 con 0x7fc0740ae540 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.071+0000 7fc078ada640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc074155b60 con 0x7fc0740ce630 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.071+0000 7fc072ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc0740ae540 0x7fc0740ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.071+0000 7fc072ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc0740ae540 0x7fc0740ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59496/0 (socket says 172.21.15.45:59496) 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.071+0000 7fc072ffd640 1 -- 172.21.15.45:0/3585487692 learned_addr learned my addr 172.21.15.45:0/3585487692 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.072+0000 7fc073fff640 1 --2- 172.21.15.45:0/3585487692 >> v2:172.21.15.190:3300/0 conn(0x7fc0740ce630 0x7fc0740b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.072+0000 7fc072ffd640 1 -- 172.21.15.45:0/3585487692 >> v2:172.21.15.190:3300/0 conn(0x7fc0740ce630 msgr2=0x7fc0740b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:41.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.072+0000 7fc072ffd640 1 --2- 172.21.15.45:0/3585487692 >> v2:172.21.15.190:3300/0 conn(0x7fc0740ce630 0x7fc0740b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.072+0000 7fc072ffd640 1 -- 172.21.15.45:0/3585487692 >> v1:172.21.15.190:6789/0 conn(0x7fc074155780 legacy=0x7fc0740abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:41.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.072+0000 7fc072ffd640 1 -- 172.21.15.45:0/3585487692 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07415ba90 con 0x7fc0740ae540 2023-11-28T20:35:41.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.073+0000 7fc072ffd640 1 --2- 172.21.15.45:0/3585487692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ae540 0x7fc0740ce0f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc0740b1690 tx=0x7fc06802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca4ef16ae074747 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:41.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.073+0000 7fc0727fc640 1 -- 172.21.15.45:0/3585487692 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068031870 con 0x7fc0740ae540 2023-11-28T20:35:41.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.073+0000 7fc0727fc640 1 -- 172.21.15.45:0/3585487692 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0680319d0 con 0x7fc0740ae540 2023-11-28T20:35:41.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.073+0000 7fc0727fc640 1 -- 172.21.15.45:0/3585487692 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068031d60 con 0x7fc0740ae540 2023-11-28T20:35:41.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.074+0000 7fc078ada640 1 -- 172.21.15.45:0/3585487692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ae540 msgr2=0x7fc0740ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.074+0000 7fc078ada640 1 --2- 172.21.15.45:0/3585487692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ae540 0x7fc0740ce0f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc0740b1690 tx=0x7fc06802b910 comp rx=0 tx=0).stop 2023-11-28T20:35:41.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.074+0000 7fc078ada640 1 -- 172.21.15.45:0/3585487692 shutdown_connections 2023-11-28T20:35:41.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.074+0000 7fc078ada640 1 --2- 172.21.15.45:0/3585487692 >> v2:172.21.15.190:3300/0 conn(0x7fc0740ce630 0x7fc0740b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.074+0000 7fc078ada640 1 --2- 172.21.15.45:0/3585487692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ae540 0x7fc0740ce0f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.074+0000 7fc078ada640 1 -- 172.21.15.45:0/3585487692 >> 172.21.15.45:0/3585487692 conn(0x7fc07406ab30 msgr2=0x7fc0740b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:41.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.074+0000 7fc078ada640 1 -- 172.21.15.45:0/3585487692 shutdown_connections 2023-11-28T20:35:41.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.074+0000 7fc078ada640 1 -- 172.21.15.45:0/3585487692 wait complete. 2023-11-28T20:35:41.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.076+0000 7fc078ada640 1 Processor -- start 2023-11-28T20:35:41.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.076+0000 7fc078ada640 1 -- start start 2023-11-28T20:35:41.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.077+0000 7fc078ada640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0740ae540 0x7fc074144be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.077+0000 7fc078ada640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ce630 0x7fc0740cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.077+0000 7fc078ada640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc074155780 0x7fc074145d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.077+0000 7fc078ada640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc074146250 con 0x7fc0740ce630 2023-11-28T20:35:41.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.077+0000 7fc078ada640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740ccc10 con 0x7fc0740ae540 2023-11-28T20:35:41.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.077+0000 7fc078ada640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740ccef0 con 0x7fc074155780 2023-11-28T20:35:41.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.077+0000 7fc072ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ce630 0x7fc0740cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.077+0000 7fc072ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ce630 0x7fc0740cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59512/0 (socket says 172.21.15.45:59512) 2023-11-28T20:35:41.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.077+0000 7fc072ffd640 1 -- 172.21.15.45:0/443142125 learned_addr learned my addr 172.21.15.45:0/443142125 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:41.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.078+0000 7fc0737fe640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0740ae540 0x7fc074144be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.078+0000 7fc072ffd640 1 -- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc074155780 msgr2=0x7fc074145d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.078+0000 7fc073fff640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc074155780 0x7fc074145d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.078+0000 7fc072ffd640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc074155780 0x7fc074145d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.078+0000 7fc072ffd640 1 -- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0740ae540 msgr2=0x7fc074144be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.078+0000 7fc072ffd640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0740ae540 0x7fc074144be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.078+0000 7fc072ffd640 1 -- 172.21.15.45:0/443142125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0680094d0 con 0x7fc0740ce630 2023-11-28T20:35:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.078+0000 7fc0737fe640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0740ae540 0x7fc074144be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.078+0000 7fc072ffd640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ce630 0x7fc0740cc510 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc068000c00 tx=0x7fc0680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.079+0000 7fc05bfff640 1 -- 172.21.15.45:0/443142125 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06803aca0 con 0x7fc0740ce630 2023-11-28T20:35:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.079+0000 7fc05bfff640 1 -- 172.21.15.45:0/443142125 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06803ae00 con 0x7fc0740ce630 2023-11-28T20:35:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.079+0000 7fc05bfff640 1 -- 172.21.15.45:0/443142125 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068044630 con 0x7fc0740ce630 2023-11-28T20:35:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.079+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0740cd1d0 con 0x7fc0740ce630 2023-11-28T20:35:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.079+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0740cd790 con 0x7fc0740ce630 2023-11-28T20:35:41.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.080+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07404e920 con 0x7fc0740ce630 2023-11-28T20:35:41.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.081+0000 7fc05bfff640 1 -- 172.21.15.45:0/443142125 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc068044790 con 0x7fc0740ce630 2023-11-28T20:35:41.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.082+0000 7fc05bfff640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc06006ae40 0x7fc06006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.082+0000 7fc05bfff640 1 -- 172.21.15.45:0/443142125 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc06802d040 con 0x7fc0740ce630 2023-11-28T20:35:41.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.083+0000 7fc0737fe640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc06006ae40 0x7fc06006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.084+0000 7fc0737fe640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc06006ae40 0x7fc06006d300 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc06400a810 tx=0x7fc064008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:41.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.087+0000 7fc05bfff640 1 -- 172.21.15.45:0/443142125 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc068076f20 con 0x7fc0740ce630 2023-11-28T20:35:41.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.281+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc07415b740 con 0x7fc06006ae40 2023-11-28T20:35:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.303+0000 7fc05bfff640 1 -- 172.21.15.45:0/443142125 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281834 (secure 0 0 0) 0x7fc06400ad90 con 0x7fc06006ae40 2023-11-28T20:35:41.309 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:41.311 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:35:41.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc06006ae40 msgr2=0x7fc06006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc06006ae40 0x7fc06006d300 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc06400a810 tx=0x7fc064008040 comp rx=0 tx=0).stop 2023-11-28T20:35:41.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ce630 msgr2=0x7fc0740cc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ce630 0x7fc0740cc510 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc068000c00 tx=0x7fc0680046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 shutdown_connections 2023-11-28T20:35:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fc06006ae40 0x7fc06006d300 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc074155780 0x7fc074145d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0740ce630 0x7fc0740cc510 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 --2- 172.21.15.45:0/443142125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0740ae540 0x7fc074144be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 >> 172.21.15.45:0/443142125 conn(0x7fc07406ab30 msgr2=0x7fc0740b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 shutdown_connections 2023-11-28T20:35:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.310+0000 7fc078ada640 1 -- 172.21.15.45:0/443142125 wait complete. 2023-11-28T20:35:41.334 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2023-11-28T20:35:41.165641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237292,"kb_used_data":4488,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737428,"statfs":{"total":773094113280,"available":772851126272,"internally_reserved":0,"allocated":4595712,"data_stored":2616256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001885"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.273744+0000","last_change":"2023-11-28T20:35:35.273744+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.273744+0000","last_undegraded":"2023-11-28T20:35:35.273744+0000","last_fullsized":"2023-11-28T20:35:35.273744+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274640+0000","last_change":"2023-11-28T20:35:35.274640+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274640+0000","last_undegraded":"2023-11-28T20:35:35.274640+0000","last_fullsized":"2023-11-28T20:35:35.274640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287799+0000","last_change":"2023-11-28T20:35:35.287799+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287799+0000","last_undegraded":"2023-11-28T20:35:35.287799+0000","last_fullsized":"2023-11-28T20:35:35.287799+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.283269+0000","last_change":"2023-11-28T20:35:35.283269+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.283269+0000","last_undegraded":"2023-11-28T20:35:35.283269+0000","last_fullsized":"2023-11-28T20:35:35.283269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284083+0000","last_change":"2023-11-28T20:35:35.284083+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284083+0000","last_undegraded":"2023-11-28T20:35:35.284083+0000","last_fullsized":"2023-11-28T20:35:35.284083+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289049+0000","last_change":"2023-11-28T20:35:35.289049+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289049+0000","last_undegraded":"2023-11-28T20:35:35.289049+0000","last_fullsized":"2023-11-28T20:35:35.289049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.300220+0000","last_change":"2023-11-28T20:35:35.300220+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.300220+0000","last_undegraded":"2023-11-28T20:35:35.300220+0000","last_fullsized":"2023-11-28T20:35:35.300220+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275156+0000","last_change":"2023-11-28T20:35:35.275156+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275156+0000","last_undegraded":"2023-11-28T20:35:35.275156+0000","last_fullsized":"2023-11-28T20:35:35.275156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288046+0000","last_change":"2023-11-28T20:35:35.288046+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288046+0000","last_undegraded":"2023-11-28T20:35:35.288046+0000","last_fullsized":"2023-11-28T20:35:35.288046+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.302551+0000","last_change":"2023-11-28T20:35:35.302551+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.302551+0000","last_undegraded":"2023-11-28T20:35:35.302551+0000","last_fullsized":"2023-11-28T20:35:35.302551+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.292849+0000","last_change":"2023-11-28T20:35:35.292849+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.292849+0000","last_undegraded":"2023-11-28T20:35:35.292849+0000","last_fullsized":"2023-11-28T20:35:35.292849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286256+0000","last_change":"2023-11-28T20:35:35.286256+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286256+0000","last_undegraded":"2023-11-28T20:35:35.286256+0000","last_fullsized":"2023-11-28T20:35:35.286256+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296324+0000","last_change":"2023-11-28T20:35:35.296324+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296324+0000","last_undegraded":"2023-11-28T20:35:35.296324+0000","last_fullsized":"2023-11-28T20:35:35.296324+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280741+0000","last_change":"2023-11-28T20:35:35.280741+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280741+0000","last_undegraded":"2023-11-28T20:35:35.280741+0000","last_fullsized":"2023-11-28T20:35:35.280741+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.298641+0000","last_change":"2023-11-28T20:35:35.298641+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.298641+0000","last_undegraded":"2023-11-28T20:35:35.298641+0000","last_fullsized":"2023-11-28T20:35:35.298641+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288030+0000","last_change":"2023-11-28T20:35:35.288030+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288030+0000","last_undegraded":"2023-11-28T20:35:35.288030+0000","last_fullsized":"2023-11-28T20:35:35.288030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.294765+0000","last_change":"2023-11-28T20:35:35.294765+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.294765+0000","last_undegraded":"2023-11-28T20:35:35.294765+0000","last_fullsized":"2023-11-28T20:35:35.294765+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286541+0000","last_change":"2023-11-28T20:35:35.286541+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286541+0000","last_undegraded":"2023-11-28T20:35:35.286541+0000","last_fullsized":"2023-11-28T20:35:35.286541+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.281506+0000","last_change":"2023-11-28T20:35:35.281506+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.281506+0000","last_undegraded":"2023-11-28T20:35:35.281506+0000","last_fullsized":"2023-11-28T20:35:35.281506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284804+0000","last_change":"2023-11-28T20:35:35.284804+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284804+0000","last_undegraded":"2023-11-28T20:35:35.284804+0000","last_fullsized":"2023-11-28T20:35:35.284804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.295351+0000","last_change":"2023-11-28T20:35:35.295351+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.295351+0000","last_undegraded":"2023-11-28T20:35:35.295351+0000","last_fullsized":"2023-11-28T20:35:35.295351+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.278315+0000","last_change":"2023-11-28T20:35:35.278315+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.278315+0000","last_undegraded":"2023-11-28T20:35:35.278315+0000","last_fullsized":"2023-11-28T20:35:35.278315+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296171+0000","last_change":"2023-11-28T20:35:35.296171+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296171+0000","last_undegraded":"2023-11-28T20:35:35.296171+0000","last_fullsized":"2023-11-28T20:35:35.296171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.282313+0000","last_change":"2023-11-28T20:35:35.282313+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.282313+0000","last_undegraded":"2023-11-28T20:35:35.282313+0000","last_fullsized":"2023-11-28T20:35:35.282313+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.290562+0000","last_change":"2023-11-28T20:35:35.290562+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.290562+0000","last_undegraded":"2023-11-28T20:35:35.290562+0000","last_fullsized":"2023-11-28T20:35:35.290562+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288231+0000","last_change":"2023-11-28T20:35:35.288231+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288231+0000","last_undegraded":"2023-11-28T20:35:35.288231+0000","last_fullsized":"2023-11-28T20:35:35.288231+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286660+0000","last_change":"2023-11-28T20:35:35.286660+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286660+0000","last_undegraded":"2023-11-28T20:35:35.286660+0000","last_fullsized":"2023-11-28T20:35:35.286660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.300373+0000","last_change":"2023-11-28T20:35:35.300373+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.300373+0000","last_undegraded":"2023-11-28T20:35:35.300373+0000","last_fullsized":"2023-11-28T20:35:35.300373+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289057+0000","last_change":"2023-11-28T20:35:35.289057+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289057+0000","last_undegraded":"2023-11-28T20:35:35.289057+0000","last_fullsized":"2023-11-28T20:35:35.289057+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288500+0000","last_change":"2023-11-28T20:35:35.288500+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288500+0000","last_undegraded":"2023-11-28T20:35:35.288500+0000","last_fullsized":"2023-11-28T20:35:35.288500+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286602+0000","last_change":"2023-11-28T20:35:35.286602+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286602+0000","last_undegraded":"2023-11-28T20:35:35.286602+0000","last_fullsized":"2023-11-28T20:35:35.286602+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284839+0000","last_change":"2023-11-28T20:35:35.284839+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284839+0000","last_undegraded":"2023-11-28T20:35:35.284839+0000","last_fullsized":"2023-11-28T20:35:35.284839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.294772+0000","last_change":"2023-11-28T20:35:35.294772+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.294772+0000","last_undegraded":"2023-11-28T20:35:35.294772+0000","last_fullsized":"2023-11-28T20:35:35.294772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310344+0000","last_change":"2023-11-28T20:35:35.304535+0000","last_active":"2023-11-28T20:35:35.310344+0000","last_peered":"2023-11-28T20:35:35.310344+0000","last_clean":"2023-11-28T20:35:35.310344+0000","last_became_active":"2023-11-28T20:35:35.304193+0000","last_became_peered":"2023-11-28T20:35:35.304193+0000","last_unstale":"2023-11-28T20:35:35.310344+0000","last_undegraded":"2023-11-28T20:35:35.310344+0000","last_fullsized":"2023-11-28T20:35:35.310344+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:14.497404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289588+0000","last_change":"2023-11-28T20:35:35.289588+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289588+0000","last_undegraded":"2023-11-28T20:35:35.289588+0000","last_fullsized":"2023-11-28T20:35:35.289588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302107+0000","last_change":"2023-11-28T20:35:35.300035+0000","last_active":"2023-11-28T20:35:35.302107+0000","last_peered":"2023-11-28T20:35:35.302107+0000","last_clean":"2023-11-28T20:35:35.302107+0000","last_became_active":"2023-11-28T20:35:35.299808+0000","last_became_peered":"2023-11-28T20:35:35.299808+0000","last_unstale":"2023-11-28T20:35:35.302107+0000","last_undegraded":"2023-11-28T20:35:35.302107+0000","last_fullsized":"2023-11-28T20:35:35.302107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276409+0000","last_change":"2023-11-28T20:35:35.276409+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276409+0000","last_undegraded":"2023-11-28T20:35:35.276409+0000","last_fullsized":"2023-11-28T20:35:35.276409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.304864+0000","last_change":"2023-11-28T20:35:35.303683+0000","last_active":"2023-11-28T20:35:35.304864+0000","last_peered":"2023-11-28T20:35:35.304864+0000","last_clean":"2023-11-28T20:35:35.304864+0000","last_became_active":"2023-11-28T20:35:35.303422+0000","last_became_peered":"2023-11-28T20:35:35.303422+0000","last_unstale":"2023-11-28T20:35:35.304864+0000","last_undegraded":"2023-11-28T20:35:35.304864+0000","last_fullsized":"2023-11-28T20:35:35.304864+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274604+0000","last_change":"2023-11-28T20:35:35.274604+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274604+0000","last_undegraded":"2023-11-28T20:35:35.274604+0000","last_fullsized":"2023-11-28T20:35:35.274604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.298920+0000","last_change":"2023-11-28T20:35:35.290259+0000","last_active":"2023-11-28T20:35:35.298920+0000","last_peered":"2023-11-28T20:35:35.298920+0000","last_clean":"2023-11-28T20:35:35.298920+0000","last_became_active":"2023-11-28T20:35:35.286178+0000","last_became_peered":"2023-11-28T20:35:35.286178+0000","last_unstale":"2023-11-28T20:35:35.298920+0000","last_undegraded":"2023-11-28T20:35:35.298920+0000","last_fullsized":"2023-11-28T20:35:35.298920+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.285970+0000","last_change":"2023-11-28T20:35:35.285970+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.285970+0000","last_undegraded":"2023-11-28T20:35:35.285970+0000","last_fullsized":"2023-11-28T20:35:35.285970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310834+0000","last_change":"2023-11-28T20:35:35.306240+0000","last_active":"2023-11-28T20:35:35.310834+0000","last_peered":"2023-11-28T20:35:35.310834+0000","last_clean":"2023-11-28T20:35:35.310834+0000","last_became_active":"2023-11-28T20:35:35.305889+0000","last_became_peered":"2023-11-28T20:35:35.305889+0000","last_unstale":"2023-11-28T20:35:35.310834+0000","last_undegraded":"2023-11-28T20:35:35.310834+0000","last_fullsized":"2023-11-28T20:35:35.310834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:29.262104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.298012+0000","last_change":"2023-11-28T20:35:35.298012+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.298012+0000","last_undegraded":"2023-11-28T20:35:35.298012+0000","last_fullsized":"2023-11-28T20:35:35.298012+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.313113+0000","last_change":"2023-11-28T20:35:35.311106+0000","last_active":"2023-11-28T20:35:35.313113+0000","last_peered":"2023-11-28T20:35:35.313113+0000","last_clean":"2023-11-28T20:35:35.313113+0000","last_became_active":"2023-11-28T20:35:35.310832+0000","last_became_peered":"2023-11-28T20:35:35.310832+0000","last_unstale":"2023-11-28T20:35:35.313113+0000","last_undegraded":"2023-11-28T20:35:35.313113+0000","last_fullsized":"2023-11-28T20:35:35.313113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:01:49.996784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276568+0000","last_change":"2023-11-28T20:35:35.276568+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276568+0000","last_undegraded":"2023-11-28T20:35:35.276568+0000","last_fullsized":"2023-11-28T20:35:35.276568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312883+0000","last_change":"2023-11-28T20:35:35.311801+0000","last_active":"2023-11-28T20:35:35.312883+0000","last_peered":"2023-11-28T20:35:35.312883+0000","last_clean":"2023-11-28T20:35:35.312883+0000","last_became_active":"2023-11-28T20:35:35.311381+0000","last_became_peered":"2023-11-28T20:35:35.311381+0000","last_unstale":"2023-11-28T20:35:35.312883+0000","last_undegraded":"2023-11-28T20:35:35.312883+0000","last_fullsized":"2023-11-28T20:35:35.312883+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276964+0000","last_change":"2023-11-28T20:35:35.276964+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276964+0000","last_undegraded":"2023-11-28T20:35:35.276964+0000","last_fullsized":"2023-11-28T20:35:35.276964+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310284+0000","last_change":"2023-11-28T20:35:35.305104+0000","last_active":"2023-11-28T20:35:35.310284+0000","last_peered":"2023-11-28T20:35:35.310284+0000","last_clean":"2023-11-28T20:35:35.310284+0000","last_became_active":"2023-11-28T20:35:35.304907+0000","last_became_peered":"2023-11-28T20:35:35.304907+0000","last_unstale":"2023-11-28T20:35:35.310284+0000","last_undegraded":"2023-11-28T20:35:35.310284+0000","last_fullsized":"2023-11-28T20:35:35.310284+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:44.520691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280744+0000","last_change":"2023-11-28T20:35:35.280744+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280744+0000","last_undegraded":"2023-11-28T20:35:35.280744+0000","last_fullsized":"2023-11-28T20:35:35.280744+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.296213+0000","last_change":"2023-11-28T20:35:35.295019+0000","last_active":"2023-11-28T20:35:35.296213+0000","last_peered":"2023-11-28T20:35:35.296213+0000","last_clean":"2023-11-28T20:35:35.296213+0000","last_became_active":"2023-11-28T20:35:35.294753+0000","last_became_peered":"2023-11-28T20:35:35.294753+0000","last_unstale":"2023-11-28T20:35:35.296213+0000","last_undegraded":"2023-11-28T20:35:35.296213+0000","last_fullsized":"2023-11-28T20:35:35.296213+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:24.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.293244+0000","last_change":"2023-11-28T20:35:35.293244+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.293244+0000","last_undegraded":"2023-11-28T20:35:35.293244+0000","last_fullsized":"2023-11-28T20:35:35.293244+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312174+0000","last_change":"2023-11-28T20:35:35.308107+0000","last_active":"2023-11-28T20:35:35.312174+0000","last_peered":"2023-11-28T20:35:35.312174+0000","last_clean":"2023-11-28T20:35:35.312174+0000","last_became_active":"2023-11-28T20:35:35.307828+0000","last_became_peered":"2023-11-28T20:35:35.307828+0000","last_unstale":"2023-11-28T20:35:35.312174+0000","last_undegraded":"2023-11-28T20:35:35.312174+0000","last_fullsized":"2023-11-28T20:35:35.312174+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:24.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287024+0000","last_change":"2023-11-28T20:35:35.287024+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287024+0000","last_undegraded":"2023-11-28T20:35:35.287024+0000","last_fullsized":"2023-11-28T20:35:35.287024+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311202+0000","last_change":"2023-11-28T20:35:35.307353+0000","last_active":"2023-11-28T20:35:35.311202+0000","last_peered":"2023-11-28T20:35:35.311202+0000","last_clean":"2023-11-28T20:35:35.311202+0000","last_became_active":"2023-11-28T20:35:35.307054+0000","last_became_peered":"2023-11-28T20:35:35.307054+0000","last_unstale":"2023-11-28T20:35:35.311202+0000","last_undegraded":"2023-11-28T20:35:35.311202+0000","last_fullsized":"2023-11-28T20:35:35.311202+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:01:49.996784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275888+0000","last_change":"2023-11-28T20:35:35.275888+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275888+0000","last_undegraded":"2023-11-28T20:35:35.275888+0000","last_fullsized":"2023-11-28T20:35:35.275888+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.301882+0000","last_change":"2023-11-28T20:35:35.298444+0000","last_active":"2023-11-28T20:35:35.301882+0000","last_peered":"2023-11-28T20:35:35.301882+0000","last_clean":"2023-11-28T20:35:35.301882+0000","last_became_active":"2023-11-28T20:35:35.297824+0000","last_became_peered":"2023-11-28T20:35:35.297824+0000","last_unstale":"2023-11-28T20:35:35.301882+0000","last_undegraded":"2023-11-28T20:35:35.301882+0000","last_fullsized":"2023-11-28T20:35:35.301882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:29.262104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.285405+0000","last_change":"2023-11-28T20:35:35.285405+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.285405+0000","last_undegraded":"2023-11-28T20:35:35.285405+0000","last_fullsized":"2023-11-28T20:35:35.285405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.292992+0000","last_change":"2023-11-28T20:35:35.290127+0000","last_active":"2023-11-28T20:35:35.292992+0000","last_peered":"2023-11-28T20:35:35.292992+0000","last_clean":"2023-11-28T20:35:35.292992+0000","last_became_active":"2023-11-28T20:35:35.289788+0000","last_became_peered":"2023-11-28T20:35:35.289788+0000","last_unstale":"2023-11-28T20:35:35.292992+0000","last_undegraded":"2023-11-28T20:35:35.292992+0000","last_fullsized":"2023-11-28T20:35:35.292992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.273794+0000","last_change":"2023-11-28T20:35:35.273794+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.273794+0000","last_undegraded":"2023-11-28T20:35:35.273794+0000","last_fullsized":"2023-11-28T20:35:35.273794+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311743+0000","last_change":"2023-11-28T20:35:35.310764+0000","last_active":"2023-11-28T20:35:35.311743+0000","last_peered":"2023-11-28T20:35:35.311743+0000","last_clean":"2023-11-28T20:35:35.311743+0000","last_became_active":"2023-11-28T20:35:35.310478+0000","last_became_peered":"2023-11-28T20:35:35.310478+0000","last_unstale":"2023-11-28T20:35:35.311743+0000","last_undegraded":"2023-11-28T20:35:35.311743+0000","last_fullsized":"2023-11-28T20:35:35.311743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.295908+0000","last_change":"2023-11-28T20:35:35.295908+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.295908+0000","last_undegraded":"2023-11-28T20:35:35.295908+0000","last_fullsized":"2023-11-28T20:35:35.295908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310256+0000","last_change":"2023-11-28T20:35:35.304319+0000","last_active":"2023-11-28T20:35:35.310256+0000","last_peered":"2023-11-28T20:35:35.310256+0000","last_clean":"2023-11-28T20:35:35.310256+0000","last_became_active":"2023-11-28T20:35:35.302594+0000","last_became_peered":"2023-11-28T20:35:35.302594+0000","last_unstale":"2023-11-28T20:35:35.310256+0000","last_undegraded":"2023-11-28T20:35:35.310256+0000","last_fullsized":"2023-11-28T20:35:35.310256+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:01.686608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276470+0000","last_change":"2023-11-28T20:35:35.276470+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276470+0000","last_undegraded":"2023-11-28T20:35:35.276470+0000","last_fullsized":"2023-11-28T20:35:35.276470+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312059+0000","last_change":"2023-11-28T20:35:35.310387+0000","last_active":"2023-11-28T20:35:35.312059+0000","last_peered":"2023-11-28T20:35:35.312059+0000","last_clean":"2023-11-28T20:35:35.312059+0000","last_became_active":"2023-11-28T20:35:35.310118+0000","last_became_peered":"2023-11-28T20:35:35.310118+0000","last_unstale":"2023-11-28T20:35:35.312059+0000","last_undegraded":"2023-11-28T20:35:35.312059+0000","last_fullsized":"2023-11-28T20:35:35.312059+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275327+0000","last_change":"2023-11-28T20:35:35.275327+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275327+0000","last_undegraded":"2023-11-28T20:35:35.275327+0000","last_fullsized":"2023-11-28T20:35:35.275327+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311828+0000","last_change":"2023-11-28T20:35:35.308420+0000","last_active":"2023-11-28T20:35:35.311828+0000","last_peered":"2023-11-28T20:35:35.311828+0000","last_clean":"2023-11-28T20:35:35.311828+0000","last_became_active":"2023-11-28T20:35:35.305473+0000","last_became_peered":"2023-11-28T20:35:35.305473+0000","last_unstale":"2023-11-28T20:35:35.311828+0000","last_undegraded":"2023-11-28T20:35:35.311828+0000","last_fullsized":"2023-11-28T20:35:35.311828+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.292286+0000","last_change":"2023-11-28T20:35:35.292286+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.292286+0000","last_undegraded":"2023-11-28T20:35:35.292286+0000","last_fullsized":"2023-11-28T20:35:35.292286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.313168+0000","last_change":"2023-11-28T20:35:35.311195+0000","last_active":"2023-11-28T20:35:35.313168+0000","last_peered":"2023-11-28T20:35:35.313168+0000","last_clean":"2023-11-28T20:35:35.313168+0000","last_became_active":"2023-11-28T20:35:35.310875+0000","last_became_peered":"2023-11-28T20:35:35.310875+0000","last_unstale":"2023-11-28T20:35:35.313168+0000","last_undegraded":"2023-11-28T20:35:35.313168+0000","last_fullsized":"2023-11-28T20:35:35.313168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.277762+0000","last_change":"2023-11-28T20:35:35.277762+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.277762+0000","last_undegraded":"2023-11-28T20:35:35.277762+0000","last_fullsized":"2023-11-28T20:35:35.277762+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302313+0000","last_change":"2023-11-28T20:35:35.298942+0000","last_active":"2023-11-28T20:35:35.302313+0000","last_peered":"2023-11-28T20:35:35.302313+0000","last_clean":"2023-11-28T20:35:35.302313+0000","last_became_active":"2023-11-28T20:35:35.298685+0000","last_became_peered":"2023-11-28T20:35:35.298685+0000","last_unstale":"2023-11-28T20:35:35.302313+0000","last_undegraded":"2023-11-28T20:35:35.302313+0000","last_fullsized":"2023-11-28T20:35:35.302313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:45.323950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280904+0000","last_change":"2023-11-28T20:35:35.280904+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280904+0000","last_undegraded":"2023-11-28T20:35:35.280904+0000","last_fullsized":"2023-11-28T20:35:35.280904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.304860+0000","last_change":"2023-11-28T20:35:35.303747+0000","last_active":"2023-11-28T20:35:35.304860+0000","last_peered":"2023-11-28T20:35:35.304860+0000","last_clean":"2023-11-28T20:35:35.304860+0000","last_became_active":"2023-11-28T20:35:35.303441+0000","last_became_peered":"2023-11-28T20:35:35.303441+0000","last_unstale":"2023-11-28T20:35:35.304860+0000","last_undegraded":"2023-11-28T20:35:35.304860+0000","last_fullsized":"2023-11-28T20:35:35.304860+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:44.520691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284186+0000","last_change":"2023-11-28T20:35:35.284186+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284186+0000","last_undegraded":"2023-11-28T20:35:35.284186+0000","last_fullsized":"2023-11-28T20:35:35.284186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.292924+0000","last_change":"2023-11-28T20:35:35.290919+0000","last_active":"2023-11-28T20:35:35.292924+0000","last_peered":"2023-11-28T20:35:35.292924+0000","last_clean":"2023-11-28T20:35:35.292924+0000","last_became_active":"2023-11-28T20:35:35.290407+0000","last_became_peered":"2023-11-28T20:35:35.290407+0000","last_unstale":"2023-11-28T20:35:35.292924+0000","last_undegraded":"2023-11-28T20:35:35.292924+0000","last_fullsized":"2023-11-28T20:35:35.292924+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:51.691786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286534+0000","last_change":"2023-11-28T20:35:35.286534+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286534+0000","last_undegraded":"2023-11-28T20:35:35.286534+0000","last_fullsized":"2023-11-28T20:35:35.286534+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302345+0000","last_change":"2023-11-28T20:35:35.299354+0000","last_active":"2023-11-28T20:35:35.302345+0000","last_peered":"2023-11-28T20:35:35.302345+0000","last_clean":"2023-11-28T20:35:35.302345+0000","last_became_active":"2023-11-28T20:35:35.299056+0000","last_became_peered":"2023-11-28T20:35:35.299056+0000","last_unstale":"2023-11-28T20:35:35.302345+0000","last_undegraded":"2023-11-28T20:35:35.302345+0000","last_fullsized":"2023-11-28T20:35:35.302345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:18.168444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.297513+0000","last_change":"2023-11-28T20:35:35.297513+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.297513+0000","last_undegraded":"2023-11-28T20:35:35.297513+0000","last_fullsized":"2023-11-28T20:35:35.297513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310069+0000","last_change":"2023-11-28T20:35:35.306856+0000","last_active":"2023-11-28T20:35:35.310069+0000","last_peered":"2023-11-28T20:35:35.310069+0000","last_clean":"2023-11-28T20:35:35.310069+0000","last_became_active":"2023-11-28T20:35:35.306562+0000","last_became_peered":"2023-11-28T20:35:35.306562+0000","last_unstale":"2023-11-28T20:35:35.310069+0000","last_undegraded":"2023-11-28T20:35:35.310069+0000","last_fullsized":"2023-11-28T20:35:35.310069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275988+0000","last_change":"2023-11-28T20:35:35.275988+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275988+0000","last_undegraded":"2023-11-28T20:35:35.275988+0000","last_fullsized":"2023-11-28T20:35:35.275988+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.293453+0000","last_change":"2023-11-28T20:35:35.291161+0000","last_active":"2023-11-28T20:35:35.293453+0000","last_peered":"2023-11-28T20:35:35.293453+0000","last_clean":"2023-11-28T20:35:35.293453+0000","last_became_active":"2023-11-28T20:35:35.290905+0000","last_became_peered":"2023-11-28T20:35:35.290905+0000","last_unstale":"2023-11-28T20:35:35.293453+0000","last_undegraded":"2023-11-28T20:35:35.293453+0000","last_fullsized":"2023-11-28T20:35:35.293453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:57.631654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287224+0000","last_change":"2023-11-28T20:35:35.287224+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287224+0000","last_undegraded":"2023-11-28T20:35:35.287224+0000","last_fullsized":"2023-11-28T20:35:35.287224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311849+0000","last_change":"2023-11-28T20:35:35.308961+0000","last_active":"2023-11-28T20:35:35.311849+0000","last_peered":"2023-11-28T20:35:35.311849+0000","last_clean":"2023-11-28T20:35:35.311849+0000","last_became_active":"2023-11-28T20:35:35.308687+0000","last_became_peered":"2023-11-28T20:35:35.308687+0000","last_unstale":"2023-11-28T20:35:35.311849+0000","last_undegraded":"2023-11-28T20:35:35.311849+0000","last_fullsized":"2023-11-28T20:35:35.311849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:18.243749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.291381+0000","last_change":"2023-11-28T20:35:35.291381+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.291381+0000","last_undegraded":"2023-11-28T20:35:35.291381+0000","last_fullsized":"2023-11-28T20:35:35.291381+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312886+0000","last_change":"2023-11-28T20:35:35.311276+0000","last_active":"2023-11-28T20:35:35.312886+0000","last_peered":"2023-11-28T20:35:35.312886+0000","last_clean":"2023-11-28T20:35:35.312886+0000","last_became_active":"2023-11-28T20:35:35.310955+0000","last_became_peered":"2023-11-28T20:35:35.310955+0000","last_unstale":"2023-11-28T20:35:35.312886+0000","last_undegraded":"2023-11-28T20:35:35.312886+0000","last_fullsized":"2023-11-28T20:35:35.312886+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:07:14.394525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274111+0000","last_change":"2023-11-28T20:35:35.274111+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274111+0000","last_undegraded":"2023-11-28T20:35:35.274111+0000","last_fullsized":"2023-11-28T20:35:35.274111+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.297236+0000","last_change":"2023-11-28T20:35:35.295046+0000","last_active":"2023-11-28T20:35:35.297236+0000","last_peered":"2023-11-28T20:35:35.297236+0000","last_clean":"2023-11-28T20:35:35.297236+0000","last_became_active":"2023-11-28T20:35:35.294261+0000","last_became_peered":"2023-11-28T20:35:35.294261+0000","last_unstale":"2023-11-28T20:35:35.297236+0000","last_undegraded":"2023-11-28T20:35:35.297236+0000","last_fullsized":"2023-11-28T20:35:35.297236+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284924+0000","last_change":"2023-11-28T20:35:35.284924+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284924+0000","last_undegraded":"2023-11-28T20:35:35.284924+0000","last_fullsized":"2023-11-28T20:35:35.284924+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.305226+0000","last_change":"2023-11-28T20:35:35.304486+0000","last_active":"2023-11-28T20:35:35.305226+0000","last_peered":"2023-11-28T20:35:35.305226+0000","last_clean":"2023-11-28T20:35:35.305226+0000","last_became_active":"2023-11-28T20:35:35.304168+0000","last_became_peered":"2023-11-28T20:35:35.304168+0000","last_unstale":"2023-11-28T20:35:35.305226+0000","last_undegraded":"2023-11-28T20:35:35.305226+0000","last_fullsized":"2023-11-28T20:35:35.305226+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289424+0000","last_change":"2023-11-28T20:35:35.289424+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289424+0000","last_undegraded":"2023-11-28T20:35:35.289424+0000","last_fullsized":"2023-11-28T20:35:35.289424+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.305224+0000","last_change":"2023-11-28T20:35:35.300968+0000","last_active":"2023-11-28T20:35:35.305224+0000","last_peered":"2023-11-28T20:35:35.305224+0000","last_clean":"2023-11-28T20:35:35.305224+0000","last_became_active":"2023-11-28T20:35:35.300389+0000","last_became_peered":"2023-11-28T20:35:35.300389+0000","last_unstale":"2023-11-28T20:35:35.305224+0000","last_undegraded":"2023-11-28T20:35:35.305224+0000","last_fullsized":"2023-11-28T20:35:35.305224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:18:34.499129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.279637+0000","last_change":"2023-11-28T20:35:35.279637+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.279637+0000","last_undegraded":"2023-11-28T20:35:35.279637+0000","last_fullsized":"2023-11-28T20:35:35.279637+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.298860+0000","last_change":"2023-11-28T20:35:35.289145+0000","last_active":"2023-11-28T20:35:35.298860+0000","last_peered":"2023-11-28T20:35:35.298860+0000","last_clean":"2023-11-28T20:35:35.298860+0000","last_became_active":"2023-11-28T20:35:35.288419+0000","last_became_peered":"2023-11-28T20:35:35.288419+0000","last_unstale":"2023-11-28T20:35:35.298860+0000","last_undegraded":"2023-11-28T20:35:35.298860+0000","last_fullsized":"2023-11-28T20:35:35.298860+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:14.497404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296014+0000","last_change":"2023-11-28T20:35:35.296014+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296014+0000","last_undegraded":"2023-11-28T20:35:35.296014+0000","last_fullsized":"2023-11-28T20:35:35.296014+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.295837+0000","last_change":"2023-11-28T20:35:35.293571+0000","last_active":"2023-11-28T20:35:35.295837+0000","last_peered":"2023-11-28T20:35:35.295837+0000","last_clean":"2023-11-28T20:35:35.295837+0000","last_became_active":"2023-11-28T20:35:35.293294+0000","last_became_peered":"2023-11-28T20:35:35.293294+0000","last_unstale":"2023-11-28T20:35:35.295837+0000","last_undegraded":"2023-11-28T20:35:35.295837+0000","last_fullsized":"2023-11-28T20:35:35.295837+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:10.536815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.293482+0000","last_change":"2023-11-28T20:35:35.293482+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.293482+0000","last_undegraded":"2023-11-28T20:35:35.293482+0000","last_fullsized":"2023-11-28T20:35:35.293482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.308144+0000","last_change":"2023-11-28T20:35:35.305966+0000","last_active":"2023-11-28T20:35:35.308144+0000","last_peered":"2023-11-28T20:35:35.308144+0000","last_clean":"2023-11-28T20:35:35.308144+0000","last_became_active":"2023-11-28T20:35:35.305715+0000","last_became_peered":"2023-11-28T20:35:35.305715+0000","last_unstale":"2023-11-28T20:35:35.308144+0000","last_undegraded":"2023-11-28T20:35:35.308144+0000","last_fullsized":"2023-11-28T20:35:35.308144+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:18.243749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-11-28T20:35:34.277946+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:35:34.277946+0000","last_peered":"2023-11-28T20:35:34.277946+0000","last_clean":"2023-11-28T20:35:34.277946+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:35:34.277946+0000","last_undegraded":"2023-11-28T20:35:34.277946+0000","last_fullsized":"2023-11-28T20:35:34.277946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":769808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.753},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.2370000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.048},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.3839999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.355}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.151},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.5259999999999998,"5min":2.5259999999999998,"15min":2.5259999999999998},"last":1.1120000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.4390000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"last":1.4690000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.619,"15min":1.619},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.516},{"interface":"front","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.7909999999999999,"5min":2.7909999999999999,"15min":2.7909999999999999},"last":1.28}]}]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":5.0430000000000001,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.242}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.736},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":1.54}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":1.637}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":4.7649999999999997,"5min":4.7649999999999997,"15min":4.7649999999999997},"last":1.1870000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":3.1989999999999998,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":3.6429999999999998,"5min":3.6429999999999998,"15min":3.6429999999999998},"last":1.8120000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.917,"5min":1.917,"15min":1.917},"min":{"1min":1.018,"5min":1.018,"15min":1.018},"max":{"1min":4.9790000000000001,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.5700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.216},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.268}]},{"osd":1,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.877},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.071}]},{"osd":2,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":1.1259999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":1.2410000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":1.4019999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.9319999999999999,"5min":2.9319999999999999,"15min":2.9319999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":1.3049999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":0.997}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.83099999999999996}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.167},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.98399999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.419},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":0.88100000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.3140000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.268},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.2270000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.627},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.3520000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":2.54,"15min":2.54},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":20.568999999999999,"5min":20.568999999999999,"15min":20.568999999999999},"last":1.3},{"interface":"front","average":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":20.677,"5min":20.677,"15min":20.677},"last":1.5029999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.80500000000000005}]},{"osd":1,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.7090000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":0.624},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.4950000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.71999999999999997}]},{"osd":2,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":5.6509999999999998,"5min":5.6509999999999998,"15min":5.6509999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.2470000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.399},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":0.872}]},{"osd":5,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.514,"15min":1.514},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.7610000000000001,"5min":2.7610000000000001,"15min":2.7610000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.492}]},{"osd":6,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.98,"5min":2.98,"15min":2.98},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.585,"15min":1.585},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.5289999999999999,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.1339999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.556},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.1859999999999999,"5min":3.1859999999999999,"15min":3.1859999999999999},"last":0.76200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.625},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.57299999999999995}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.79400000000000004}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.83699999999999997}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.3160000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":1.113}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.931,"5min":2.931,"15min":2.931},"last":1.452},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.202}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.9649999999999999,"5min":2.9649999999999999,"15min":2.9649999999999999},"last":1.395}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.89600000000000002}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.637}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.276},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.165}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":3.5259999999999998,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":1.339}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":3.637,"5min":3.637,"15min":3.637},"last":1.484},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.534}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":1.5920000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.395},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.4660000000000002,"5min":4.4660000000000002,"15min":4.4660000000000002},"last":1.5109999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":769808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.964,"5min":2.964,"15min":2.964},"last":0.61099999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.504},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.52500000000000002}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.68300000000000005}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.89700000000000002}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.536,"5min":2.536,"15min":2.536},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":0.96099999999999997}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":2.8650000000000002,"5min":2.8650000000000002,"15min":2.8650000000000002},"last":1.085},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":1.123,"5min":1.123,"15min":1.123},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:35:41.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:35:41.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.605+0000 7f189a541640 1 Processor -- start 2023-11-28T20:35:41.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.606+0000 7f189a541640 1 -- start start 2023-11-28T20:35:41.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.607+0000 7f189a541640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f18940b35b0 0x7f189415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.608+0000 7f189a541640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f18940b1340 0x7f189415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.608+0000 7f189a541640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f189415bda0 con 0x7f189415b850 2023-11-28T20:35:41.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.608+0000 7f189a541640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f189415ee40 con 0x7f18940b1340 2023-11-28T20:35:41.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.608+0000 7f189a541640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f189415c080 con 0x7f18940b35b0 2023-11-28T20:35:41.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.608+0000 7f1898ade640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f18940b1340 0x7f189415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.608+0000 7f1898ade640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f18940b1340 0x7f189415f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59516/0 (socket says 172.21.15.45:59516) 2023-11-28T20:35:41.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.608+0000 7f1898ade640 1 -- 172.21.15.45:0/3246100819 learned_addr learned my addr 172.21.15.45:0/3246100819 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:41.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.608+0000 7f1893fff640 1 --2- 172.21.15.45:0/3246100819 >> v2:172.21.15.190:3300/0 conn(0x7f18940b35b0 0x7f189415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.609+0000 7f1898ade640 1 -- 172.21.15.45:0/3246100819 >> v2:172.21.15.190:3300/0 conn(0x7f18940b35b0 msgr2=0x7f189415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:41.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.609+0000 7f1898ade640 1 --2- 172.21.15.45:0/3246100819 >> v2:172.21.15.190:3300/0 conn(0x7f18940b35b0 0x7f189415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.609+0000 7f1898ade640 1 -- 172.21.15.45:0/3246100819 >> v1:172.21.15.190:6789/0 conn(0x7f189415b850 legacy=0x7f18940b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:41.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.609+0000 7f1898ade640 1 -- 172.21.15.45:0/3246100819 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189415c360 con 0x7f18940b1340 2023-11-28T20:35:41.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.610+0000 7f1898ade640 1 --2- 172.21.15.45:0/3246100819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18940b1340 0x7f189415f290 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f189406e7f0 tx=0x7f188c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1a1864f00a8fe6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:41.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.610+0000 7f1892ffd640 1 -- 172.21.15.45:0/3246100819 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c031870 con 0x7f18940b1340 2023-11-28T20:35:41.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.610+0000 7f1892ffd640 1 -- 172.21.15.45:0/3246100819 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c0319d0 con 0x7f18940b1340 2023-11-28T20:35:41.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.610+0000 7f1892ffd640 1 -- 172.21.15.45:0/3246100819 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c031d60 con 0x7f18940b1340 2023-11-28T20:35:41.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.611+0000 7f189a541640 1 -- 172.21.15.45:0/3246100819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18940b1340 msgr2=0x7f189415f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.611+0000 7f189a541640 1 --2- 172.21.15.45:0/3246100819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18940b1340 0x7f189415f290 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f189406e7f0 tx=0x7f188c02b910 comp rx=0 tx=0).stop 2023-11-28T20:35:41.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.611+0000 7f189a541640 1 -- 172.21.15.45:0/3246100819 shutdown_connections 2023-11-28T20:35:41.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.611+0000 7f189a541640 1 --2- 172.21.15.45:0/3246100819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18940b1340 0x7f189415f290 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.611+0000 7f189a541640 1 --2- 172.21.15.45:0/3246100819 >> v2:172.21.15.190:3300/0 conn(0x7f18940b35b0 0x7f189415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.611+0000 7f189a541640 1 -- 172.21.15.45:0/3246100819 >> 172.21.15.45:0/3246100819 conn(0x7f189406a500 msgr2=0x7f189406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:41.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.611+0000 7f189a541640 1 -- 172.21.15.45:0/3246100819 shutdown_connections 2023-11-28T20:35:41.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.611+0000 7f189a541640 1 -- 172.21.15.45:0/3246100819 wait complete. 2023-11-28T20:35:41.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.613+0000 7f189a541640 1 Processor -- start 2023-11-28T20:35:41.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.613+0000 7f189a541640 1 -- start start 2023-11-28T20:35:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.614+0000 7f189a541640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18940b1340 0x7f1894155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.614+0000 7f189a541640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18940b35b0 0x7f18940adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.614+0000 7f189a541640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f189415b850 0x7f1894141ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.614+0000 7f189a541640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18940ac900 con 0x7f189415b850 2023-11-28T20:35:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.614+0000 7f189a541640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18940a84a0 con 0x7f18940b1340 2023-11-28T20:35:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.614+0000 7f189a541640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18940a9500 con 0x7f18940b35b0 2023-11-28T20:35:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1898ade640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f189415b850 0x7f1894141ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1898ade640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f189415b850 0x7f1894141ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59522/0 (socket says 172.21.15.45:59522) 2023-11-28T20:35:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1898ade640 1 -- 172.21.15.45:0/1373491311 learned_addr learned my addr 172.21.15.45:0/1373491311 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1893fff640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18940b1340 0x7f1894155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f18937fe640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18940b35b0 0x7f18940adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1898ade640 1 -- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18940b35b0 msgr2=0x7f18940adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1898ade640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18940b35b0 0x7f18940adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1898ade640 1 -- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18940b1340 msgr2=0x7f1894155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1898ade640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18940b1340 0x7f1894155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1898ade640 1 -- 172.21.15.45:0/1373491311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188c0094d0 con 0x7f189415b850 2023-11-28T20:35:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.615+0000 7f1893fff640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18940b1340 0x7f1894155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.616+0000 7f1898ade640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f189415b850 0x7f1894141ff0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f188c0090b0 tx=0x7f188c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.616+0000 7f1890ff9640 1 -- 172.21.15.45:0/1373491311 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c03ad20 con 0x7f189415b850 2023-11-28T20:35:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.616+0000 7f1890ff9640 1 -- 172.21.15.45:0/1373491311 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c0443f0 con 0x7f189415b850 2023-11-28T20:35:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.616+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18940a8740 con 0x7f189415b850 2023-11-28T20:35:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.616+0000 7f1890ff9640 1 -- 172.21.15.45:0/1373491311 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c03aea0 con 0x7f189415b850 2023-11-28T20:35:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.616+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18940a8d00 con 0x7f189415b850 2023-11-28T20:35:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.617+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18940733d0 con 0x7f189415b850 2023-11-28T20:35:41.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.618+0000 7f1890ff9640 1 -- 172.21.15.45:0/1373491311 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f188c04d450 con 0x7f189415b850 2023-11-28T20:35:41.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.619+0000 7f1890ff9640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f187406ae40 0x7f187406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:41.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.619+0000 7f1890ff9640 1 -- 172.21.15.45:0/1373491311 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f188c02d040 con 0x7f189415b850 2023-11-28T20:35:41.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.620+0000 7f1893fff640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f187406ae40 0x7f187406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:41.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.621+0000 7f1893fff640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f187406ae40 0x7f187406d300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1884009b20 tx=0x7f18840039d0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:41.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.624+0000 7f1890ff9640 1 -- 172.21.15.45:0/1373491311 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f188c076cf0 con 0x7f189415b850 2023-11-28T20:35:41.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.815+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f189415b6e0 con 0x7f187406ae40 2023-11-28T20:35:41.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.838+0000 7f1890ff9640 1 -- 172.21.15.45:0/1373491311 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281834 (secure 0 0 0) 0x7f189415b6e0 con 0x7f187406ae40 2023-11-28T20:35:41.843 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:41.846 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:35:41.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f187406ae40 msgr2=0x7f187406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f187406ae40 0x7f187406d300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1884009b20 tx=0x7f18840039d0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f189415b850 msgr2=0x7f1894141ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:41.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f189415b850 0x7f1894141ff0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f188c0090b0 tx=0x7f188c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 shutdown_connections 2023-11-28T20:35:41.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f187406ae40 0x7f187406d300 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f189415b850 0x7f1894141ff0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18940b35b0 0x7f18940adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 --2- 172.21.15.45:0/1373491311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18940b1340 0x7f1894155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:41.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 >> 172.21.15.45:0/1373491311 conn(0x7f189406a500 msgr2=0x7f1894071620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:41.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 shutdown_connections 2023-11-28T20:35:41.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:41.845+0000 7f189a541640 1 -- 172.21.15.45:0/1373491311 wait complete. 2023-11-28T20:35:41.867 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2023-11-28T20:35:41.165641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237292,"kb_used_data":4488,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737428,"statfs":{"total":773094113280,"available":772851126272,"internally_reserved":0,"allocated":4595712,"data_stored":2616256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001885"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.273744+0000","last_change":"2023-11-28T20:35:35.273744+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.273744+0000","last_undegraded":"2023-11-28T20:35:35.273744+0000","last_fullsized":"2023-11-28T20:35:35.273744+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274640+0000","last_change":"2023-11-28T20:35:35.274640+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274640+0000","last_undegraded":"2023-11-28T20:35:35.274640+0000","last_fullsized":"2023-11-28T20:35:35.274640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287799+0000","last_change":"2023-11-28T20:35:35.287799+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287799+0000","last_undegraded":"2023-11-28T20:35:35.287799+0000","last_fullsized":"2023-11-28T20:35:35.287799+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.283269+0000","last_change":"2023-11-28T20:35:35.283269+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.283269+0000","last_undegraded":"2023-11-28T20:35:35.283269+0000","last_fullsized":"2023-11-28T20:35:35.283269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284083+0000","last_change":"2023-11-28T20:35:35.284083+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284083+0000","last_undegraded":"2023-11-28T20:35:35.284083+0000","last_fullsized":"2023-11-28T20:35:35.284083+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289049+0000","last_change":"2023-11-28T20:35:35.289049+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289049+0000","last_undegraded":"2023-11-28T20:35:35.289049+0000","last_fullsized":"2023-11-28T20:35:35.289049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.300220+0000","last_change":"2023-11-28T20:35:35.300220+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.300220+0000","last_undegraded":"2023-11-28T20:35:35.300220+0000","last_fullsized":"2023-11-28T20:35:35.300220+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275156+0000","last_change":"2023-11-28T20:35:35.275156+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275156+0000","last_undegraded":"2023-11-28T20:35:35.275156+0000","last_fullsized":"2023-11-28T20:35:35.275156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288046+0000","last_change":"2023-11-28T20:35:35.288046+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288046+0000","last_undegraded":"2023-11-28T20:35:35.288046+0000","last_fullsized":"2023-11-28T20:35:35.288046+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.302551+0000","last_change":"2023-11-28T20:35:35.302551+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.302551+0000","last_undegraded":"2023-11-28T20:35:35.302551+0000","last_fullsized":"2023-11-28T20:35:35.302551+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.292849+0000","last_change":"2023-11-28T20:35:35.292849+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.292849+0000","last_undegraded":"2023-11-28T20:35:35.292849+0000","last_fullsized":"2023-11-28T20:35:35.292849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286256+0000","last_change":"2023-11-28T20:35:35.286256+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286256+0000","last_undegraded":"2023-11-28T20:35:35.286256+0000","last_fullsized":"2023-11-28T20:35:35.286256+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296324+0000","last_change":"2023-11-28T20:35:35.296324+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296324+0000","last_undegraded":"2023-11-28T20:35:35.296324+0000","last_fullsized":"2023-11-28T20:35:35.296324+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280741+0000","last_change":"2023-11-28T20:35:35.280741+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280741+0000","last_undegraded":"2023-11-28T20:35:35.280741+0000","last_fullsized":"2023-11-28T20:35:35.280741+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.298641+0000","last_change":"2023-11-28T20:35:35.298641+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.298641+0000","last_undegraded":"2023-11-28T20:35:35.298641+0000","last_fullsized":"2023-11-28T20:35:35.298641+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288030+0000","last_change":"2023-11-28T20:35:35.288030+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288030+0000","last_undegraded":"2023-11-28T20:35:35.288030+0000","last_fullsized":"2023-11-28T20:35:35.288030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.294765+0000","last_change":"2023-11-28T20:35:35.294765+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.294765+0000","last_undegraded":"2023-11-28T20:35:35.294765+0000","last_fullsized":"2023-11-28T20:35:35.294765+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286541+0000","last_change":"2023-11-28T20:35:35.286541+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286541+0000","last_undegraded":"2023-11-28T20:35:35.286541+0000","last_fullsized":"2023-11-28T20:35:35.286541+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.281506+0000","last_change":"2023-11-28T20:35:35.281506+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.281506+0000","last_undegraded":"2023-11-28T20:35:35.281506+0000","last_fullsized":"2023-11-28T20:35:35.281506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284804+0000","last_change":"2023-11-28T20:35:35.284804+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284804+0000","last_undegraded":"2023-11-28T20:35:35.284804+0000","last_fullsized":"2023-11-28T20:35:35.284804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.295351+0000","last_change":"2023-11-28T20:35:35.295351+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.295351+0000","last_undegraded":"2023-11-28T20:35:35.295351+0000","last_fullsized":"2023-11-28T20:35:35.295351+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.278315+0000","last_change":"2023-11-28T20:35:35.278315+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.278315+0000","last_undegraded":"2023-11-28T20:35:35.278315+0000","last_fullsized":"2023-11-28T20:35:35.278315+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296171+0000","last_change":"2023-11-28T20:35:35.296171+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296171+0000","last_undegraded":"2023-11-28T20:35:35.296171+0000","last_fullsized":"2023-11-28T20:35:35.296171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.282313+0000","last_change":"2023-11-28T20:35:35.282313+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.282313+0000","last_undegraded":"2023-11-28T20:35:35.282313+0000","last_fullsized":"2023-11-28T20:35:35.282313+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.290562+0000","last_change":"2023-11-28T20:35:35.290562+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.290562+0000","last_undegraded":"2023-11-28T20:35:35.290562+0000","last_fullsized":"2023-11-28T20:35:35.290562+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288231+0000","last_change":"2023-11-28T20:35:35.288231+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288231+0000","last_undegraded":"2023-11-28T20:35:35.288231+0000","last_fullsized":"2023-11-28T20:35:35.288231+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286660+0000","last_change":"2023-11-28T20:35:35.286660+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286660+0000","last_undegraded":"2023-11-28T20:35:35.286660+0000","last_fullsized":"2023-11-28T20:35:35.286660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.300373+0000","last_change":"2023-11-28T20:35:35.300373+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.300373+0000","last_undegraded":"2023-11-28T20:35:35.300373+0000","last_fullsized":"2023-11-28T20:35:35.300373+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289057+0000","last_change":"2023-11-28T20:35:35.289057+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289057+0000","last_undegraded":"2023-11-28T20:35:35.289057+0000","last_fullsized":"2023-11-28T20:35:35.289057+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288500+0000","last_change":"2023-11-28T20:35:35.288500+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288500+0000","last_undegraded":"2023-11-28T20:35:35.288500+0000","last_fullsized":"2023-11-28T20:35:35.288500+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286602+0000","last_change":"2023-11-28T20:35:35.286602+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286602+0000","last_undegraded":"2023-11-28T20:35:35.286602+0000","last_fullsized":"2023-11-28T20:35:35.286602+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284839+0000","last_change":"2023-11-28T20:35:35.284839+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284839+0000","last_undegraded":"2023-11-28T20:35:35.284839+0000","last_fullsized":"2023-11-28T20:35:35.284839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.294772+0000","last_change":"2023-11-28T20:35:35.294772+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.294772+0000","last_undegraded":"2023-11-28T20:35:35.294772+0000","last_fullsized":"2023-11-28T20:35:35.294772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310344+0000","last_change":"2023-11-28T20:35:35.304535+0000","last_active":"2023-11-28T20:35:35.310344+0000","last_peered":"2023-11-28T20:35:35.310344+0000","last_clean":"2023-11-28T20:35:35.310344+0000","last_became_active":"2023-11-28T20:35:35.304193+0000","last_became_peered":"2023-11-28T20:35:35.304193+0000","last_unstale":"2023-11-28T20:35:35.310344+0000","last_undegraded":"2023-11-28T20:35:35.310344+0000","last_fullsized":"2023-11-28T20:35:35.310344+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:14.497404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289588+0000","last_change":"2023-11-28T20:35:35.289588+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289588+0000","last_undegraded":"2023-11-28T20:35:35.289588+0000","last_fullsized":"2023-11-28T20:35:35.289588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302107+0000","last_change":"2023-11-28T20:35:35.300035+0000","last_active":"2023-11-28T20:35:35.302107+0000","last_peered":"2023-11-28T20:35:35.302107+0000","last_clean":"2023-11-28T20:35:35.302107+0000","last_became_active":"2023-11-28T20:35:35.299808+0000","last_became_peered":"2023-11-28T20:35:35.299808+0000","last_unstale":"2023-11-28T20:35:35.302107+0000","last_undegraded":"2023-11-28T20:35:35.302107+0000","last_fullsized":"2023-11-28T20:35:35.302107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276409+0000","last_change":"2023-11-28T20:35:35.276409+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276409+0000","last_undegraded":"2023-11-28T20:35:35.276409+0000","last_fullsized":"2023-11-28T20:35:35.276409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.304864+0000","last_change":"2023-11-28T20:35:35.303683+0000","last_active":"2023-11-28T20:35:35.304864+0000","last_peered":"2023-11-28T20:35:35.304864+0000","last_clean":"2023-11-28T20:35:35.304864+0000","last_became_active":"2023-11-28T20:35:35.303422+0000","last_became_peered":"2023-11-28T20:35:35.303422+0000","last_unstale":"2023-11-28T20:35:35.304864+0000","last_undegraded":"2023-11-28T20:35:35.304864+0000","last_fullsized":"2023-11-28T20:35:35.304864+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274604+0000","last_change":"2023-11-28T20:35:35.274604+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274604+0000","last_undegraded":"2023-11-28T20:35:35.274604+0000","last_fullsized":"2023-11-28T20:35:35.274604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.298920+0000","last_change":"2023-11-28T20:35:35.290259+0000","last_active":"2023-11-28T20:35:35.298920+0000","last_peered":"2023-11-28T20:35:35.298920+0000","last_clean":"2023-11-28T20:35:35.298920+0000","last_became_active":"2023-11-28T20:35:35.286178+0000","last_became_peered":"2023-11-28T20:35:35.286178+0000","last_unstale":"2023-11-28T20:35:35.298920+0000","last_undegraded":"2023-11-28T20:35:35.298920+0000","last_fullsized":"2023-11-28T20:35:35.298920+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.285970+0000","last_change":"2023-11-28T20:35:35.285970+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.285970+0000","last_undegraded":"2023-11-28T20:35:35.285970+0000","last_fullsized":"2023-11-28T20:35:35.285970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310834+0000","last_change":"2023-11-28T20:35:35.306240+0000","last_active":"2023-11-28T20:35:35.310834+0000","last_peered":"2023-11-28T20:35:35.310834+0000","last_clean":"2023-11-28T20:35:35.310834+0000","last_became_active":"2023-11-28T20:35:35.305889+0000","last_became_peered":"2023-11-28T20:35:35.305889+0000","last_unstale":"2023-11-28T20:35:35.310834+0000","last_undegraded":"2023-11-28T20:35:35.310834+0000","last_fullsized":"2023-11-28T20:35:35.310834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:29.262104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.298012+0000","last_change":"2023-11-28T20:35:35.298012+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.298012+0000","last_undegraded":"2023-11-28T20:35:35.298012+0000","last_fullsized":"2023-11-28T20:35:35.298012+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.313113+0000","last_change":"2023-11-28T20:35:35.311106+0000","last_active":"2023-11-28T20:35:35.313113+0000","last_peered":"2023-11-28T20:35:35.313113+0000","last_clean":"2023-11-28T20:35:35.313113+0000","last_became_active":"2023-11-28T20:35:35.310832+0000","last_became_peered":"2023-11-28T20:35:35.310832+0000","last_unstale":"2023-11-28T20:35:35.313113+0000","last_undegraded":"2023-11-28T20:35:35.313113+0000","last_fullsized":"2023-11-28T20:35:35.313113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:01:49.996784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276568+0000","last_change":"2023-11-28T20:35:35.276568+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276568+0000","last_undegraded":"2023-11-28T20:35:35.276568+0000","last_fullsized":"2023-11-28T20:35:35.276568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312883+0000","last_change":"2023-11-28T20:35:35.311801+0000","last_active":"2023-11-28T20:35:35.312883+0000","last_peered":"2023-11-28T20:35:35.312883+0000","last_clean":"2023-11-28T20:35:35.312883+0000","last_became_active":"2023-11-28T20:35:35.311381+0000","last_became_peered":"2023-11-28T20:35:35.311381+0000","last_unstale":"2023-11-28T20:35:35.312883+0000","last_undegraded":"2023-11-28T20:35:35.312883+0000","last_fullsized":"2023-11-28T20:35:35.312883+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276964+0000","last_change":"2023-11-28T20:35:35.276964+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276964+0000","last_undegraded":"2023-11-28T20:35:35.276964+0000","last_fullsized":"2023-11-28T20:35:35.276964+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310284+0000","last_change":"2023-11-28T20:35:35.305104+0000","last_active":"2023-11-28T20:35:35.310284+0000","last_peered":"2023-11-28T20:35:35.310284+0000","last_clean":"2023-11-28T20:35:35.310284+0000","last_became_active":"2023-11-28T20:35:35.304907+0000","last_became_peered":"2023-11-28T20:35:35.304907+0000","last_unstale":"2023-11-28T20:35:35.310284+0000","last_undegraded":"2023-11-28T20:35:35.310284+0000","last_fullsized":"2023-11-28T20:35:35.310284+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:44.520691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280744+0000","last_change":"2023-11-28T20:35:35.280744+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280744+0000","last_undegraded":"2023-11-28T20:35:35.280744+0000","last_fullsized":"2023-11-28T20:35:35.280744+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.296213+0000","last_change":"2023-11-28T20:35:35.295019+0000","last_active":"2023-11-28T20:35:35.296213+0000","last_peered":"2023-11-28T20:35:35.296213+0000","last_clean":"2023-11-28T20:35:35.296213+0000","last_became_active":"2023-11-28T20:35:35.294753+0000","last_became_peered":"2023-11-28T20:35:35.294753+0000","last_unstale":"2023-11-28T20:35:35.296213+0000","last_undegraded":"2023-11-28T20:35:35.296213+0000","last_fullsized":"2023-11-28T20:35:35.296213+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:24.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.293244+0000","last_change":"2023-11-28T20:35:35.293244+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.293244+0000","last_undegraded":"2023-11-28T20:35:35.293244+0000","last_fullsized":"2023-11-28T20:35:35.293244+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312174+0000","last_change":"2023-11-28T20:35:35.308107+0000","last_active":"2023-11-28T20:35:35.312174+0000","last_peered":"2023-11-28T20:35:35.312174+0000","last_clean":"2023-11-28T20:35:35.312174+0000","last_became_active":"2023-11-28T20:35:35.307828+0000","last_became_peered":"2023-11-28T20:35:35.307828+0000","last_unstale":"2023-11-28T20:35:35.312174+0000","last_undegraded":"2023-11-28T20:35:35.312174+0000","last_fullsized":"2023-11-28T20:35:35.312174+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:24.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287024+0000","last_change":"2023-11-28T20:35:35.287024+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287024+0000","last_undegraded":"2023-11-28T20:35:35.287024+0000","last_fullsized":"2023-11-28T20:35:35.287024+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311202+0000","last_change":"2023-11-28T20:35:35.307353+0000","last_active":"2023-11-28T20:35:35.311202+0000","last_peered":"2023-11-28T20:35:35.311202+0000","last_clean":"2023-11-28T20:35:35.311202+0000","last_became_active":"2023-11-28T20:35:35.307054+0000","last_became_peered":"2023-11-28T20:35:35.307054+0000","last_unstale":"2023-11-28T20:35:35.311202+0000","last_undegraded":"2023-11-28T20:35:35.311202+0000","last_fullsized":"2023-11-28T20:35:35.311202+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:01:49.996784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275888+0000","last_change":"2023-11-28T20:35:35.275888+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275888+0000","last_undegraded":"2023-11-28T20:35:35.275888+0000","last_fullsized":"2023-11-28T20:35:35.275888+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.301882+0000","last_change":"2023-11-28T20:35:35.298444+0000","last_active":"2023-11-28T20:35:35.301882+0000","last_peered":"2023-11-28T20:35:35.301882+0000","last_clean":"2023-11-28T20:35:35.301882+0000","last_became_active":"2023-11-28T20:35:35.297824+0000","last_became_peered":"2023-11-28T20:35:35.297824+0000","last_unstale":"2023-11-28T20:35:35.301882+0000","last_undegraded":"2023-11-28T20:35:35.301882+0000","last_fullsized":"2023-11-28T20:35:35.301882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:29.262104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.285405+0000","last_change":"2023-11-28T20:35:35.285405+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.285405+0000","last_undegraded":"2023-11-28T20:35:35.285405+0000","last_fullsized":"2023-11-28T20:35:35.285405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.292992+0000","last_change":"2023-11-28T20:35:35.290127+0000","last_active":"2023-11-28T20:35:35.292992+0000","last_peered":"2023-11-28T20:35:35.292992+0000","last_clean":"2023-11-28T20:35:35.292992+0000","last_became_active":"2023-11-28T20:35:35.289788+0000","last_became_peered":"2023-11-28T20:35:35.289788+0000","last_unstale":"2023-11-28T20:35:35.292992+0000","last_undegraded":"2023-11-28T20:35:35.292992+0000","last_fullsized":"2023-11-28T20:35:35.292992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.273794+0000","last_change":"2023-11-28T20:35:35.273794+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.273794+0000","last_undegraded":"2023-11-28T20:35:35.273794+0000","last_fullsized":"2023-11-28T20:35:35.273794+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311743+0000","last_change":"2023-11-28T20:35:35.310764+0000","last_active":"2023-11-28T20:35:35.311743+0000","last_peered":"2023-11-28T20:35:35.311743+0000","last_clean":"2023-11-28T20:35:35.311743+0000","last_became_active":"2023-11-28T20:35:35.310478+0000","last_became_peered":"2023-11-28T20:35:35.310478+0000","last_unstale":"2023-11-28T20:35:35.311743+0000","last_undegraded":"2023-11-28T20:35:35.311743+0000","last_fullsized":"2023-11-28T20:35:35.311743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.295908+0000","last_change":"2023-11-28T20:35:35.295908+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.295908+0000","last_undegraded":"2023-11-28T20:35:35.295908+0000","last_fullsized":"2023-11-28T20:35:35.295908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310256+0000","last_change":"2023-11-28T20:35:35.304319+0000","last_active":"2023-11-28T20:35:35.310256+0000","last_peered":"2023-11-28T20:35:35.310256+0000","last_clean":"2023-11-28T20:35:35.310256+0000","last_became_active":"2023-11-28T20:35:35.302594+0000","last_became_peered":"2023-11-28T20:35:35.302594+0000","last_unstale":"2023-11-28T20:35:35.310256+0000","last_undegraded":"2023-11-28T20:35:35.310256+0000","last_fullsized":"2023-11-28T20:35:35.310256+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:01.686608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276470+0000","last_change":"2023-11-28T20:35:35.276470+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276470+0000","last_undegraded":"2023-11-28T20:35:35.276470+0000","last_fullsized":"2023-11-28T20:35:35.276470+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312059+0000","last_change":"2023-11-28T20:35:35.310387+0000","last_active":"2023-11-28T20:35:35.312059+0000","last_peered":"2023-11-28T20:35:35.312059+0000","last_clean":"2023-11-28T20:35:35.312059+0000","last_became_active":"2023-11-28T20:35:35.310118+0000","last_became_peered":"2023-11-28T20:35:35.310118+0000","last_unstale":"2023-11-28T20:35:35.312059+0000","last_undegraded":"2023-11-28T20:35:35.312059+0000","last_fullsized":"2023-11-28T20:35:35.312059+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275327+0000","last_change":"2023-11-28T20:35:35.275327+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275327+0000","last_undegraded":"2023-11-28T20:35:35.275327+0000","last_fullsized":"2023-11-28T20:35:35.275327+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311828+0000","last_change":"2023-11-28T20:35:35.308420+0000","last_active":"2023-11-28T20:35:35.311828+0000","last_peered":"2023-11-28T20:35:35.311828+0000","last_clean":"2023-11-28T20:35:35.311828+0000","last_became_active":"2023-11-28T20:35:35.305473+0000","last_became_peered":"2023-11-28T20:35:35.305473+0000","last_unstale":"2023-11-28T20:35:35.311828+0000","last_undegraded":"2023-11-28T20:35:35.311828+0000","last_fullsized":"2023-11-28T20:35:35.311828+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.292286+0000","last_change":"2023-11-28T20:35:35.292286+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.292286+0000","last_undegraded":"2023-11-28T20:35:35.292286+0000","last_fullsized":"2023-11-28T20:35:35.292286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.313168+0000","last_change":"2023-11-28T20:35:35.311195+0000","last_active":"2023-11-28T20:35:35.313168+0000","last_peered":"2023-11-28T20:35:35.313168+0000","last_clean":"2023-11-28T20:35:35.313168+0000","last_became_active":"2023-11-28T20:35:35.310875+0000","last_became_peered":"2023-11-28T20:35:35.310875+0000","last_unstale":"2023-11-28T20:35:35.313168+0000","last_undegraded":"2023-11-28T20:35:35.313168+0000","last_fullsized":"2023-11-28T20:35:35.313168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.277762+0000","last_change":"2023-11-28T20:35:35.277762+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.277762+0000","last_undegraded":"2023-11-28T20:35:35.277762+0000","last_fullsized":"2023-11-28T20:35:35.277762+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302313+0000","last_change":"2023-11-28T20:35:35.298942+0000","last_active":"2023-11-28T20:35:35.302313+0000","last_peered":"2023-11-28T20:35:35.302313+0000","last_clean":"2023-11-28T20:35:35.302313+0000","last_became_active":"2023-11-28T20:35:35.298685+0000","last_became_peered":"2023-11-28T20:35:35.298685+0000","last_unstale":"2023-11-28T20:35:35.302313+0000","last_undegraded":"2023-11-28T20:35:35.302313+0000","last_fullsized":"2023-11-28T20:35:35.302313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:45.323950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280904+0000","last_change":"2023-11-28T20:35:35.280904+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280904+0000","last_undegraded":"2023-11-28T20:35:35.280904+0000","last_fullsized":"2023-11-28T20:35:35.280904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.304860+0000","last_change":"2023-11-28T20:35:35.303747+0000","last_active":"2023-11-28T20:35:35.304860+0000","last_peered":"2023-11-28T20:35:35.304860+0000","last_clean":"2023-11-28T20:35:35.304860+0000","last_became_active":"2023-11-28T20:35:35.303441+0000","last_became_peered":"2023-11-28T20:35:35.303441+0000","last_unstale":"2023-11-28T20:35:35.304860+0000","last_undegraded":"2023-11-28T20:35:35.304860+0000","last_fullsized":"2023-11-28T20:35:35.304860+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:44.520691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284186+0000","last_change":"2023-11-28T20:35:35.284186+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284186+0000","last_undegraded":"2023-11-28T20:35:35.284186+0000","last_fullsized":"2023-11-28T20:35:35.284186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.292924+0000","last_change":"2023-11-28T20:35:35.290919+0000","last_active":"2023-11-28T20:35:35.292924+0000","last_peered":"2023-11-28T20:35:35.292924+0000","last_clean":"2023-11-28T20:35:35.292924+0000","last_became_active":"2023-11-28T20:35:35.290407+0000","last_became_peered":"2023-11-28T20:35:35.290407+0000","last_unstale":"2023-11-28T20:35:35.292924+0000","last_undegraded":"2023-11-28T20:35:35.292924+0000","last_fullsized":"2023-11-28T20:35:35.292924+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:51.691786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286534+0000","last_change":"2023-11-28T20:35:35.286534+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286534+0000","last_undegraded":"2023-11-28T20:35:35.286534+0000","last_fullsized":"2023-11-28T20:35:35.286534+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302345+0000","last_change":"2023-11-28T20:35:35.299354+0000","last_active":"2023-11-28T20:35:35.302345+0000","last_peered":"2023-11-28T20:35:35.302345+0000","last_clean":"2023-11-28T20:35:35.302345+0000","last_became_active":"2023-11-28T20:35:35.299056+0000","last_became_peered":"2023-11-28T20:35:35.299056+0000","last_unstale":"2023-11-28T20:35:35.302345+0000","last_undegraded":"2023-11-28T20:35:35.302345+0000","last_fullsized":"2023-11-28T20:35:35.302345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:18.168444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.297513+0000","last_change":"2023-11-28T20:35:35.297513+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.297513+0000","last_undegraded":"2023-11-28T20:35:35.297513+0000","last_fullsized":"2023-11-28T20:35:35.297513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310069+0000","last_change":"2023-11-28T20:35:35.306856+0000","last_active":"2023-11-28T20:35:35.310069+0000","last_peered":"2023-11-28T20:35:35.310069+0000","last_clean":"2023-11-28T20:35:35.310069+0000","last_became_active":"2023-11-28T20:35:35.306562+0000","last_became_peered":"2023-11-28T20:35:35.306562+0000","last_unstale":"2023-11-28T20:35:35.310069+0000","last_undegraded":"2023-11-28T20:35:35.310069+0000","last_fullsized":"2023-11-28T20:35:35.310069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275988+0000","last_change":"2023-11-28T20:35:35.275988+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275988+0000","last_undegraded":"2023-11-28T20:35:35.275988+0000","last_fullsized":"2023-11-28T20:35:35.275988+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.293453+0000","last_change":"2023-11-28T20:35:35.291161+0000","last_active":"2023-11-28T20:35:35.293453+0000","last_peered":"2023-11-28T20:35:35.293453+0000","last_clean":"2023-11-28T20:35:35.293453+0000","last_became_active":"2023-11-28T20:35:35.290905+0000","last_became_peered":"2023-11-28T20:35:35.290905+0000","last_unstale":"2023-11-28T20:35:35.293453+0000","last_undegraded":"2023-11-28T20:35:35.293453+0000","last_fullsized":"2023-11-28T20:35:35.293453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:57.631654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287224+0000","last_change":"2023-11-28T20:35:35.287224+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287224+0000","last_undegraded":"2023-11-28T20:35:35.287224+0000","last_fullsized":"2023-11-28T20:35:35.287224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311849+0000","last_change":"2023-11-28T20:35:35.308961+0000","last_active":"2023-11-28T20:35:35.311849+0000","last_peered":"2023-11-28T20:35:35.311849+0000","last_clean":"2023-11-28T20:35:35.311849+0000","last_became_active":"2023-11-28T20:35:35.308687+0000","last_became_peered":"2023-11-28T20:35:35.308687+0000","last_unstale":"2023-11-28T20:35:35.311849+0000","last_undegraded":"2023-11-28T20:35:35.311849+0000","last_fullsized":"2023-11-28T20:35:35.311849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:18.243749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.291381+0000","last_change":"2023-11-28T20:35:35.291381+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.291381+0000","last_undegraded":"2023-11-28T20:35:35.291381+0000","last_fullsized":"2023-11-28T20:35:35.291381+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312886+0000","last_change":"2023-11-28T20:35:35.311276+0000","last_active":"2023-11-28T20:35:35.312886+0000","last_peered":"2023-11-28T20:35:35.312886+0000","last_clean":"2023-11-28T20:35:35.312886+0000","last_became_active":"2023-11-28T20:35:35.310955+0000","last_became_peered":"2023-11-28T20:35:35.310955+0000","last_unstale":"2023-11-28T20:35:35.312886+0000","last_undegraded":"2023-11-28T20:35:35.312886+0000","last_fullsized":"2023-11-28T20:35:35.312886+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:07:14.394525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274111+0000","last_change":"2023-11-28T20:35:35.274111+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274111+0000","last_undegraded":"2023-11-28T20:35:35.274111+0000","last_fullsized":"2023-11-28T20:35:35.274111+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.297236+0000","last_change":"2023-11-28T20:35:35.295046+0000","last_active":"2023-11-28T20:35:35.297236+0000","last_peered":"2023-11-28T20:35:35.297236+0000","last_clean":"2023-11-28T20:35:35.297236+0000","last_became_active":"2023-11-28T20:35:35.294261+0000","last_became_peered":"2023-11-28T20:35:35.294261+0000","last_unstale":"2023-11-28T20:35:35.297236+0000","last_undegraded":"2023-11-28T20:35:35.297236+0000","last_fullsized":"2023-11-28T20:35:35.297236+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284924+0000","last_change":"2023-11-28T20:35:35.284924+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284924+0000","last_undegraded":"2023-11-28T20:35:35.284924+0000","last_fullsized":"2023-11-28T20:35:35.284924+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.305226+0000","last_change":"2023-11-28T20:35:35.304486+0000","last_active":"2023-11-28T20:35:35.305226+0000","last_peered":"2023-11-28T20:35:35.305226+0000","last_clean":"2023-11-28T20:35:35.305226+0000","last_became_active":"2023-11-28T20:35:35.304168+0000","last_became_peered":"2023-11-28T20:35:35.304168+0000","last_unstale":"2023-11-28T20:35:35.305226+0000","last_undegraded":"2023-11-28T20:35:35.305226+0000","last_fullsized":"2023-11-28T20:35:35.305226+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289424+0000","last_change":"2023-11-28T20:35:35.289424+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289424+0000","last_undegraded":"2023-11-28T20:35:35.289424+0000","last_fullsized":"2023-11-28T20:35:35.289424+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.305224+0000","last_change":"2023-11-28T20:35:35.300968+0000","last_active":"2023-11-28T20:35:35.305224+0000","last_peered":"2023-11-28T20:35:35.305224+0000","last_clean":"2023-11-28T20:35:35.305224+0000","last_became_active":"2023-11-28T20:35:35.300389+0000","last_became_peered":"2023-11-28T20:35:35.300389+0000","last_unstale":"2023-11-28T20:35:35.305224+0000","last_undegraded":"2023-11-28T20:35:35.305224+0000","last_fullsized":"2023-11-28T20:35:35.305224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:18:34.499129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.279637+0000","last_change":"2023-11-28T20:35:35.279637+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.279637+0000","last_undegraded":"2023-11-28T20:35:35.279637+0000","last_fullsized":"2023-11-28T20:35:35.279637+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.298860+0000","last_change":"2023-11-28T20:35:35.289145+0000","last_active":"2023-11-28T20:35:35.298860+0000","last_peered":"2023-11-28T20:35:35.298860+0000","last_clean":"2023-11-28T20:35:35.298860+0000","last_became_active":"2023-11-28T20:35:35.288419+0000","last_became_peered":"2023-11-28T20:35:35.288419+0000","last_unstale":"2023-11-28T20:35:35.298860+0000","last_undegraded":"2023-11-28T20:35:35.298860+0000","last_fullsized":"2023-11-28T20:35:35.298860+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:14.497404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296014+0000","last_change":"2023-11-28T20:35:35.296014+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296014+0000","last_undegraded":"2023-11-28T20:35:35.296014+0000","last_fullsized":"2023-11-28T20:35:35.296014+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.295837+0000","last_change":"2023-11-28T20:35:35.293571+0000","last_active":"2023-11-28T20:35:35.295837+0000","last_peered":"2023-11-28T20:35:35.295837+0000","last_clean":"2023-11-28T20:35:35.295837+0000","last_became_active":"2023-11-28T20:35:35.293294+0000","last_became_peered":"2023-11-28T20:35:35.293294+0000","last_unstale":"2023-11-28T20:35:35.295837+0000","last_undegraded":"2023-11-28T20:35:35.295837+0000","last_fullsized":"2023-11-28T20:35:35.295837+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:10.536815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.293482+0000","last_change":"2023-11-28T20:35:35.293482+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.293482+0000","last_undegraded":"2023-11-28T20:35:35.293482+0000","last_fullsized":"2023-11-28T20:35:35.293482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.308144+0000","last_change":"2023-11-28T20:35:35.305966+0000","last_active":"2023-11-28T20:35:35.308144+0000","last_peered":"2023-11-28T20:35:35.308144+0000","last_clean":"2023-11-28T20:35:35.308144+0000","last_became_active":"2023-11-28T20:35:35.305715+0000","last_became_peered":"2023-11-28T20:35:35.305715+0000","last_unstale":"2023-11-28T20:35:35.308144+0000","last_undegraded":"2023-11-28T20:35:35.308144+0000","last_fullsized":"2023-11-28T20:35:35.308144+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:18.243749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-11-28T20:35:34.277946+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:35:34.277946+0000","last_peered":"2023-11-28T20:35:34.277946+0000","last_clean":"2023-11-28T20:35:34.277946+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:35:34.277946+0000","last_undegraded":"2023-11-28T20:35:34.277946+0000","last_fullsized":"2023-11-28T20:35:34.277946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":769808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.753},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.2370000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.048},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.3839999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.355}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.151},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.5259999999999998,"5min":2.5259999999999998,"15min":2.5259999999999998},"last":1.1120000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.4390000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"last":1.4690000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.619,"15min":1.619},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.516},{"interface":"front","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.7909999999999999,"5min":2.7909999999999999,"15min":2.7909999999999999},"last":1.28}]}]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":5.0430000000000001,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.242}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.736},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":1.54}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":1.637}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":4.7649999999999997,"5min":4.7649999999999997,"15min":4.7649999999999997},"last":1.1870000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":3.1989999999999998,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":3.6429999999999998,"5min":3.6429999999999998,"15min":3.6429999999999998},"last":1.8120000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.917,"5min":1.917,"15min":1.917},"min":{"1min":1.018,"5min":1.018,"15min":1.018},"max":{"1min":4.9790000000000001,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.5700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.216},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.268}]},{"osd":1,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.877},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.071}]},{"osd":2,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":1.1259999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":1.2410000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":1.4019999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.9319999999999999,"5min":2.9319999999999999,"15min":2.9319999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":1.3049999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":0.997}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.83099999999999996}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.167},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.98399999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.419},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":0.88100000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.3140000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.268},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.2270000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.627},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.3520000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":2.54,"15min":2.54},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":20.568999999999999,"5min":20.568999999999999,"15min":20.568999999999999},"last":1.3},{"interface":"front","average":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":20.677,"5min":20.677,"15min":20.677},"last":1.5029999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.80500000000000005}]},{"osd":1,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.7090000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":0.624},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.4950000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.71999999999999997}]},{"osd":2,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":5.6509999999999998,"5min":5.6509999999999998,"15min":5.6509999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.2470000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.399},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":0.872}]},{"osd":5,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.514,"15min":1.514},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.7610000000000001,"5min":2.7610000000000001,"15min":2.7610000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.492}]},{"osd":6,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.98,"5min":2.98,"15min":2.98},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.585,"15min":1.585},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.5289999999999999,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.1339999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.556},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.1859999999999999,"5min":3.1859999999999999,"15min":3.1859999999999999},"last":0.76200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.625},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.57299999999999995}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.79400000000000004}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.83699999999999997}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.3160000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":1.113}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.931,"5min":2.931,"15min":2.931},"last":1.452},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.202}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.9649999999999999,"5min":2.9649999999999999,"15min":2.9649999999999999},"last":1.395}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.89600000000000002}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.637}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.276},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.165}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":3.5259999999999998,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":1.339}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":3.637,"5min":3.637,"15min":3.637},"last":1.484},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.534}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":1.5920000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.395},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.4660000000000002,"5min":4.4660000000000002,"15min":4.4660000000000002},"last":1.5109999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":769808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.964,"5min":2.964,"15min":2.964},"last":0.61099999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.504},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.52500000000000002}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.68300000000000005}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.89700000000000002}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.536,"5min":2.536,"15min":2.536},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":0.96099999999999997}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":2.8650000000000002,"5min":2.8650000000000002,"15min":2.8650000000000002},"last":1.085},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":1.123,"5min":1.123,"15min":1.123},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:35:41.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2023-11-28T20:35:41.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.273744+0000', 'last_change': '2023-11-28T20:35:35.273744+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.273744+0000', 'last_undegraded': '2023-11-28T20:35:35.273744+0000', 'last_fullsized': '2023-11-28T20:35:35.273744+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2023-11-28T20:35:41.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.274640+0000', 'last_change': '2023-11-28T20:35:35.274640+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.274640+0000', 'last_undegraded': '2023-11-28T20:35:35.274640+0000', 'last_fullsized': '2023-11-28T20:35:35.274640+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2023-11-28T20:35:41.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.287799+0000', 'last_change': '2023-11-28T20:35:35.287799+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.287799+0000', 'last_undegraded': '2023-11-28T20:35:35.287799+0000', 'last_fullsized': '2023-11-28T20:35:35.287799+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T20:35:41.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2023-11-28T20:35:41.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.283269+0000', 'last_change': '2023-11-28T20:35:35.283269+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.283269+0000', 'last_undegraded': '2023-11-28T20:35:35.283269+0000', 'last_fullsized': '2023-11-28T20:35:35.283269+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-28T20:35:41.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2023-11-28T20:35:41.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.284083+0000', 'last_change': '2023-11-28T20:35:35.284083+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.284083+0000', 'last_undegraded': '2023-11-28T20:35:35.284083+0000', 'last_fullsized': '2023-11-28T20:35:35.284083+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-28T20:35:41.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2023-11-28T20:35:41.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.289049+0000', 'last_change': '2023-11-28T20:35:35.289049+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.289049+0000', 'last_undegraded': '2023-11-28T20:35:35.289049+0000', 'last_fullsized': '2023-11-28T20:35:35.289049+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-28T20:35:41.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2023-11-28T20:35:41.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.300220+0000', 'last_change': '2023-11-28T20:35:35.300220+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.300220+0000', 'last_undegraded': '2023-11-28T20:35:35.300220+0000', 'last_fullsized': '2023-11-28T20:35:35.300220+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2023-11-28T20:35:41.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.275156+0000', 'last_change': '2023-11-28T20:35:35.275156+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.275156+0000', 'last_undegraded': '2023-11-28T20:35:35.275156+0000', 'last_fullsized': '2023-11-28T20:35:35.275156+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2023-11-28T20:35:41.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.288046+0000', 'last_change': '2023-11-28T20:35:35.288046+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.288046+0000', 'last_undegraded': '2023-11-28T20:35:35.288046+0000', 'last_fullsized': '2023-11-28T20:35:35.288046+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.302551+0000', 'last_change': '2023-11-28T20:35:35.302551+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.302551+0000', 'last_undegraded': '2023-11-28T20:35:35.302551+0000', 'last_fullsized': '2023-11-28T20:35:35.302551+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.292849+0000', 'last_change': '2023-11-28T20:35:35.292849+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.292849+0000', 'last_undegraded': '2023-11-28T20:35:35.292849+0000', 'last_fullsized': '2023-11-28T20:35:35.292849+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.286256+0000', 'last_change': '2023-11-28T20:35:35.286256+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.286256+0000', 'last_undegraded': '2023-11-28T20:35:35.286256+0000', 'last_fullsized': '2023-11-28T20:35:35.286256+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.296324+0000', 'last_change': '2023-11-28T20:35:35.296324+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.296324+0000', 'last_undegraded': '2023-11-28T20:35:35.296324+0000', 'last_fullsized': '2023-11-28T20:35:35.296324+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2023-11-28T20:35:41.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.280741+0000', 'last_change': '2023-11-28T20:35:35.280741+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.280741+0000', 'last_undegraded': '2023-11-28T20:35:35.280741+0000', 'last_fullsized': '2023-11-28T20:35:35.280741+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T20:35:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2023-11-28T20:35:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.298641+0000', 'last_change': '2023-11-28T20:35:35.298641+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.298641+0000', 'last_undegraded': '2023-11-28T20:35:35.298641+0000', 'last_fullsized': '2023-11-28T20:35:35.298641+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2023-11-28T20:35:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.288030+0000', 'last_change': '2023-11-28T20:35:35.288030+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.288030+0000', 'last_undegraded': '2023-11-28T20:35:35.288030+0000', 'last_fullsized': '2023-11-28T20:35:35.288030+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2023-11-28T20:35:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.294765+0000', 'last_change': '2023-11-28T20:35:35.294765+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.294765+0000', 'last_undegraded': '2023-11-28T20:35:35.294765+0000', 'last_fullsized': '2023-11-28T20:35:35.294765+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2023-11-28T20:35:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.286541+0000', 'last_change': '2023-11-28T20:35:35.286541+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.286541+0000', 'last_undegraded': '2023-11-28T20:35:35.286541+0000', 'last_fullsized': '2023-11-28T20:35:35.286541+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-28T20:35:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2023-11-28T20:35:41.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.281506+0000', 'last_change': '2023-11-28T20:35:35.281506+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.281506+0000', 'last_undegraded': '2023-11-28T20:35:35.281506+0000', 'last_fullsized': '2023-11-28T20:35:35.281506+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-28T20:35:41.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2023-11-28T20:35:41.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.284804+0000', 'last_change': '2023-11-28T20:35:35.284804+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.284804+0000', 'last_undegraded': '2023-11-28T20:35:35.284804+0000', 'last_fullsized': '2023-11-28T20:35:35.284804+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-28T20:35:41.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2023-11-28T20:35:41.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.295351+0000', 'last_change': '2023-11-28T20:35:35.295351+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.295351+0000', 'last_undegraded': '2023-11-28T20:35:35.295351+0000', 'last_fullsized': '2023-11-28T20:35:35.295351+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2023-11-28T20:35:41.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.278315+0000', 'last_change': '2023-11-28T20:35:35.278315+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.278315+0000', 'last_undegraded': '2023-11-28T20:35:35.278315+0000', 'last_fullsized': '2023-11-28T20:35:35.278315+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T20:35:41.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2023-11-28T20:35:41.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.296171+0000', 'last_change': '2023-11-28T20:35:35.296171+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.296171+0000', 'last_undegraded': '2023-11-28T20:35:35.296171+0000', 'last_fullsized': '2023-11-28T20:35:35.296171+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2023-11-28T20:35:41.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.282313+0000', 'last_change': '2023-11-28T20:35:35.282313+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.282313+0000', 'last_undegraded': '2023-11-28T20:35:35.282313+0000', 'last_fullsized': '2023-11-28T20:35:35.282313+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-28T20:35:41.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2023-11-28T20:35:41.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.290562+0000', 'last_change': '2023-11-28T20:35:35.290562+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.290562+0000', 'last_undegraded': '2023-11-28T20:35:35.290562+0000', 'last_fullsized': '2023-11-28T20:35:35.290562+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-28T20:35:41.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2023-11-28T20:35:41.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.288231+0000', 'last_change': '2023-11-28T20:35:35.288231+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.288231+0000', 'last_undegraded': '2023-11-28T20:35:35.288231+0000', 'last_fullsized': '2023-11-28T20:35:35.288231+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-28T20:35:41.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2023-11-28T20:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.286660+0000', 'last_change': '2023-11-28T20:35:35.286660+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.286660+0000', 'last_undegraded': '2023-11-28T20:35:35.286660+0000', 'last_fullsized': '2023-11-28T20:35:35.286660+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-28T20:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2023-11-28T20:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.300373+0000', 'last_change': '2023-11-28T20:35:35.300373+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.300373+0000', 'last_undegraded': '2023-11-28T20:35:35.300373+0000', 'last_fullsized': '2023-11-28T20:35:35.300373+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2023-11-28T20:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.289057+0000', 'last_change': '2023-11-28T20:35:35.289057+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.289057+0000', 'last_undegraded': '2023-11-28T20:35:35.289057+0000', 'last_fullsized': '2023-11-28T20:35:35.289057+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-28T20:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2023-11-28T20:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.288500+0000', 'last_change': '2023-11-28T20:35:35.288500+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.288500+0000', 'last_undegraded': '2023-11-28T20:35:35.288500+0000', 'last_fullsized': '2023-11-28T20:35:35.288500+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-28T20:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2023-11-28T20:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.286602+0000', 'last_change': '2023-11-28T20:35:35.286602+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.286602+0000', 'last_undegraded': '2023-11-28T20:35:35.286602+0000', 'last_fullsized': '2023-11-28T20:35:35.286602+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2023-11-28T20:35:41.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.284839+0000', 'last_change': '2023-11-28T20:35:35.284839+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.284839+0000', 'last_undegraded': '2023-11-28T20:35:35.284839+0000', 'last_fullsized': '2023-11-28T20:35:35.284839+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-28T20:35:41.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2023-11-28T20:35:41.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.294772+0000', 'last_change': '2023-11-28T20:35:35.294772+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.294772+0000', 'last_undegraded': '2023-11-28T20:35:35.294772+0000', 'last_fullsized': '2023-11-28T20:35:35.294772+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2023-11-28T20:35:41.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.289588+0000', 'last_change': '2023-11-28T20:35:35.289588+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.289588+0000', 'last_undegraded': '2023-11-28T20:35:35.289588+0000', 'last_fullsized': '2023-11-28T20:35:35.289588+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-28T20:35:41.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2023-11-28T20:35:41.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.276409+0000', 'last_change': '2023-11-28T20:35:35.276409+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.276409+0000', 'last_undegraded': '2023-11-28T20:35:35.276409+0000', 'last_fullsized': '2023-11-28T20:35:35.276409+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T20:35:41.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2023-11-28T20:35:41.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.274604+0000', 'last_change': '2023-11-28T20:35:35.274604+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.274604+0000', 'last_undegraded': '2023-11-28T20:35:35.274604+0000', 'last_fullsized': '2023-11-28T20:35:35.274604+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2023-11-28T20:35:41.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.285970+0000', 'last_change': '2023-11-28T20:35:35.285970+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.285970+0000', 'last_undegraded': '2023-11-28T20:35:35.285970+0000', 'last_fullsized': '2023-11-28T20:35:35.285970+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T20:35:41.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2023-11-28T20:35:41.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.298012+0000', 'last_change': '2023-11-28T20:35:35.298012+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.298012+0000', 'last_undegraded': '2023-11-28T20:35:35.298012+0000', 'last_fullsized': '2023-11-28T20:35:35.298012+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2023-11-28T20:35:41.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.276568+0000', 'last_change': '2023-11-28T20:35:35.276568+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.276568+0000', 'last_undegraded': '2023-11-28T20:35:35.276568+0000', 'last_fullsized': '2023-11-28T20:35:35.276568+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T20:35:41.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2023-11-28T20:35:41.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.276964+0000', 'last_change': '2023-11-28T20:35:35.276964+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.276964+0000', 'last_undegraded': '2023-11-28T20:35:35.276964+0000', 'last_fullsized': '2023-11-28T20:35:35.276964+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.280744+0000', 'last_change': '2023-11-28T20:35:35.280744+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.280744+0000', 'last_undegraded': '2023-11-28T20:35:35.280744+0000', 'last_fullsized': '2023-11-28T20:35:35.280744+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.293244+0000', 'last_change': '2023-11-28T20:35:35.293244+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.293244+0000', 'last_undegraded': '2023-11-28T20:35:35.293244+0000', 'last_fullsized': '2023-11-28T20:35:35.293244+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.287024+0000', 'last_change': '2023-11-28T20:35:35.287024+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.287024+0000', 'last_undegraded': '2023-11-28T20:35:35.287024+0000', 'last_fullsized': '2023-11-28T20:35:35.287024+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.275888+0000', 'last_change': '2023-11-28T20:35:35.275888+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.275888+0000', 'last_undegraded': '2023-11-28T20:35:35.275888+0000', 'last_fullsized': '2023-11-28T20:35:35.275888+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2023-11-28T20:35:41.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.285405+0000', 'last_change': '2023-11-28T20:35:35.285405+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.285405+0000', 'last_undegraded': '2023-11-28T20:35:35.285405+0000', 'last_fullsized': '2023-11-28T20:35:35.285405+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T20:35:41.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2023-11-28T20:35:41.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.273794+0000', 'last_change': '2023-11-28T20:35:35.273794+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.273794+0000', 'last_undegraded': '2023-11-28T20:35:35.273794+0000', 'last_fullsized': '2023-11-28T20:35:35.273794+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2023-11-28T20:35:41.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.295908+0000', 'last_change': '2023-11-28T20:35:35.295908+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.295908+0000', 'last_undegraded': '2023-11-28T20:35:35.295908+0000', 'last_fullsized': '2023-11-28T20:35:35.295908+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2023-11-28T20:35:41.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.276470+0000', 'last_change': '2023-11-28T20:35:35.276470+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.276470+0000', 'last_undegraded': '2023-11-28T20:35:35.276470+0000', 'last_fullsized': '2023-11-28T20:35:35.276470+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2023-11-28T20:35:41.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.275327+0000', 'last_change': '2023-11-28T20:35:35.275327+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.275327+0000', 'last_undegraded': '2023-11-28T20:35:35.275327+0000', 'last_fullsized': '2023-11-28T20:35:35.275327+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2023-11-28T20:35:41.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.292286+0000', 'last_change': '2023-11-28T20:35:35.292286+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.292286+0000', 'last_undegraded': '2023-11-28T20:35:35.292286+0000', 'last_fullsized': '2023-11-28T20:35:35.292286+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2023-11-28T20:35:41.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.277762+0000', 'last_change': '2023-11-28T20:35:35.277762+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.277762+0000', 'last_undegraded': '2023-11-28T20:35:35.277762+0000', 'last_fullsized': '2023-11-28T20:35:35.277762+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T20:35:41.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2023-11-28T20:35:41.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.280904+0000', 'last_change': '2023-11-28T20:35:35.280904+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.280904+0000', 'last_undegraded': '2023-11-28T20:35:35.280904+0000', 'last_fullsized': '2023-11-28T20:35:35.280904+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-28T20:35:41.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2023-11-28T20:35:41.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.284186+0000', 'last_change': '2023-11-28T20:35:35.284186+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.284186+0000', 'last_undegraded': '2023-11-28T20:35:35.284186+0000', 'last_fullsized': '2023-11-28T20:35:35.284186+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2023-11-28T20:35:41.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.286534+0000', 'last_change': '2023-11-28T20:35:35.286534+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.286534+0000', 'last_undegraded': '2023-11-28T20:35:35.286534+0000', 'last_fullsized': '2023-11-28T20:35:35.286534+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.297513+0000', 'last_change': '2023-11-28T20:35:35.297513+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.297513+0000', 'last_undegraded': '2023-11-28T20:35:35.297513+0000', 'last_fullsized': '2023-11-28T20:35:35.297513+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.275988+0000', 'last_change': '2023-11-28T20:35:35.275988+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.275988+0000', 'last_undegraded': '2023-11-28T20:35:35.275988+0000', 'last_fullsized': '2023-11-28T20:35:35.275988+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.287224+0000', 'last_change': '2023-11-28T20:35:35.287224+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.287224+0000', 'last_undegraded': '2023-11-28T20:35:35.287224+0000', 'last_fullsized': '2023-11-28T20:35:35.287224+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.291381+0000', 'last_change': '2023-11-28T20:35:35.291381+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.291381+0000', 'last_undegraded': '2023-11-28T20:35:35.291381+0000', 'last_fullsized': '2023-11-28T20:35:35.291381+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2023-11-28T20:35:41.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.274111+0000', 'last_change': '2023-11-28T20:35:35.274111+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.274111+0000', 'last_undegraded': '2023-11-28T20:35:35.274111+0000', 'last_fullsized': '2023-11-28T20:35:35.274111+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T20:35:41.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2023-11-28T20:35:41.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.284924+0000', 'last_change': '2023-11-28T20:35:35.284924+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.284924+0000', 'last_undegraded': '2023-11-28T20:35:35.284924+0000', 'last_fullsized': '2023-11-28T20:35:35.284924+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T20:35:41.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2023-11-28T20:35:41.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.289424+0000', 'last_change': '2023-11-28T20:35:35.289424+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.289424+0000', 'last_undegraded': '2023-11-28T20:35:35.289424+0000', 'last_fullsized': '2023-11-28T20:35:35.289424+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-28T20:35:41.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2023-11-28T20:35:41.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.279637+0000', 'last_change': '2023-11-28T20:35:35.279637+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.279637+0000', 'last_undegraded': '2023-11-28T20:35:35.279637+0000', 'last_fullsized': '2023-11-28T20:35:35.279637+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T20:35:41.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2023-11-28T20:35:41.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.296014+0000', 'last_change': '2023-11-28T20:35:35.296014+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.296014+0000', 'last_undegraded': '2023-11-28T20:35:35.296014+0000', 'last_fullsized': '2023-11-28T20:35:35.296014+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T20:35:41.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2023-11-28T20:35:41.886 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-11-28T20:35:35.293482+0000', 'last_change': '2023-11-28T20:35:35.293482+0000', 'last_active': '2023-11-28T20:35:35.259596+0000', 'last_peered': '2023-11-28T20:35:35.259596+0000', 'last_clean': '2023-11-28T20:35:35.259596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T20:35:35.293482+0000', 'last_undegraded': '2023-11-28T20:35:35.293482+0000', 'last_fullsized': '2023-11-28T20:35:35.293482+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'last_clean_scrub_stamp': '2023-11-28T20:35:35.259596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T20:35:41.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-11-28T20:35:42.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.140+0000 7f980a232640 1 Processor -- start 2023-11-28T20:35:42.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.141+0000 7f980a232640 1 -- start start 2023-11-28T20:35:42.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.142+0000 7f980a232640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9804072200 0x7f98040b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.142+0000 7f980a232640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9804146610 0x7f98040b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.142+0000 7f980a232640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98040b4310 con 0x7f9804146b60 2023-11-28T20:35:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.142+0000 7f980a232640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98040725e0 con 0x7f9804072200 2023-11-28T20:35:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.142+0000 7f980a232640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980406d8d0 con 0x7f9804146610 2023-11-28T20:35:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.143+0000 7f9809230640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9804072200 0x7f98040b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.143+0000 7f9809230640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9804072200 0x7f98040b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59534/0 (socket says 172.21.15.45:59534) 2023-11-28T20:35:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.143+0000 7f9809230640 1 -- 172.21.15.45:0/710144827 learned_addr learned my addr 172.21.15.45:0/710144827 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.143+0000 7f9808a2f640 1 --2- 172.21.15.45:0/710144827 >> v2:172.21.15.190:3300/0 conn(0x7f9804146610 0x7f98040b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.143+0000 7f9809230640 1 -- 172.21.15.45:0/710144827 >> v2:172.21.15.190:3300/0 conn(0x7f9804146610 msgr2=0x7f98040b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.143+0000 7f9809230640 1 --2- 172.21.15.45:0/710144827 >> v2:172.21.15.190:3300/0 conn(0x7f9804146610 0x7f98040b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.143+0000 7f9809230640 1 -- 172.21.15.45:0/710144827 >> v1:172.21.15.190:6789/0 conn(0x7f9804146b60 legacy=0x7f9804160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.143+0000 7f9809230640 1 -- 172.21.15.45:0/710144827 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980406ca20 con 0x7f9804072200 2023-11-28T20:35:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.143+0000 7f9808a2f640 1 --2- 172.21.15.45:0/710144827 >> v2:172.21.15.190:3300/0 conn(0x7f9804146610 0x7f98040b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.144+0000 7f9809230640 1 --2- 172.21.15.45:0/710144827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 0x7f98040b1d90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f980409d0c0 tx=0x7f980002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca438a0707330fc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.145+0000 7f97fbfff640 1 -- 172.21.15.45:0/710144827 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9800031870 con 0x7f9804072200 2023-11-28T20:35:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.145+0000 7f97fbfff640 1 -- 172.21.15.45:0/710144827 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98000319d0 con 0x7f9804072200 2023-11-28T20:35:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.145+0000 7f97fbfff640 1 -- 172.21.15.45:0/710144827 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9800031d60 con 0x7f9804072200 2023-11-28T20:35:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.145+0000 7f980a232640 1 -- 172.21.15.45:0/710144827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 msgr2=0x7f98040b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.145+0000 7f980a232640 1 --2- 172.21.15.45:0/710144827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 0x7f98040b1d90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f980409d0c0 tx=0x7f980002b910 comp rx=0 tx=0).stop 2023-11-28T20:35:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.146+0000 7f980a232640 1 -- 172.21.15.45:0/710144827 shutdown_connections 2023-11-28T20:35:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.146+0000 7f980a232640 1 --2- 172.21.15.45:0/710144827 >> v2:172.21.15.190:3300/0 conn(0x7f9804146610 0x7f98040b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.146+0000 7f980a232640 1 --2- 172.21.15.45:0/710144827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 0x7f98040b1d90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.146+0000 7f980a232640 1 -- 172.21.15.45:0/710144827 >> 172.21.15.45:0/710144827 conn(0x7f980406ab30 msgr2=0x7f98040b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.146+0000 7f980a232640 1 -- 172.21.15.45:0/710144827 shutdown_connections 2023-11-28T20:35:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.146+0000 7f980a232640 1 -- 172.21.15.45:0/710144827 wait complete. 2023-11-28T20:35:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.147+0000 7f980a232640 1 Processor -- start 2023-11-28T20:35:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.148+0000 7f980a232640 1 -- start start 2023-11-28T20:35:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f980a232640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 0x7f980415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f980a232640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9804146610 0x7f9804140870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f980a232640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9804146b60 0x7f98040b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f980a232640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9804141ed0 con 0x7f9804072200 2023-11-28T20:35:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f980a232640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980415d040 con 0x7f9804146b60 2023-11-28T20:35:42.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f980a232640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98041412c0 con 0x7f9804146610 2023-11-28T20:35:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f9809230640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 0x7f980415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f9809230640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 0x7f980415cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59548/0 (socket says 172.21.15.45:59548) 2023-11-28T20:35:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f9809230640 1 -- 172.21.15.45:0/2790140660 learned_addr learned my addr 172.21.15.45:0/2790140660 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f9808a2f640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9804146610 0x7f9804140870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.149+0000 7f9809a31640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9804146b60 0x7f98040b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f9809230640 1 -- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9804146610 msgr2=0x7f9804140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f9809230640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9804146610 0x7f9804140870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f9809230640 1 -- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9804146b60 msgr2=0x7f98040b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f9809230640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9804146b60 0x7f98040b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f9809230640 1 -- 172.21.15.45:0/2790140660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98000094d0 con 0x7f9804072200 2023-11-28T20:35:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f9809230640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 0x7f980415cb00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f9800000c00 tx=0x7f98000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f97f9ffb640 1 -- 172.21.15.45:0/2790140660 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9800031840 con 0x7f9804072200 2023-11-28T20:35:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f97f9ffb640 1 -- 172.21.15.45:0/2790140660 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98000443f0 con 0x7f9804072200 2023-11-28T20:35:42.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98041415a0 con 0x7f9804072200 2023-11-28T20:35:42.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f97f9ffb640 1 -- 172.21.15.45:0/2790140660 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98000447a0 con 0x7f9804072200 2023-11-28T20:35:42.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.150+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9804141b60 con 0x7f9804072200 2023-11-28T20:35:42.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.152+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980409d0f0 con 0x7f9804072200 2023-11-28T20:35:42.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.152+0000 7f97f9ffb640 1 -- 172.21.15.45:0/2790140660 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f980004d450 con 0x7f9804072200 2023-11-28T20:35:42.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.153+0000 7f97f9ffb640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97e806ae40 0x7f97e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.153+0000 7f97f9ffb640 1 -- 172.21.15.45:0/2790140660 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f980002d040 con 0x7f9804072200 2023-11-28T20:35:42.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.154+0000 7f9808a2f640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97e806ae40 0x7f97e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.155+0000 7f9808a2f640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97e806ae40 0x7f97e806d300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f98041f65e0 tx=0x7f97f4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:42.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.158+0000 7f97f9ffb640 1 -- 172.21.15.45:0/2790140660 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9800076d00 con 0x7f9804072200 2023-11-28T20:35:42.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.426+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f98040b42e0 con 0x7f9804072200 2023-11-28T20:35:42.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.427+0000 7f97f9ffb640 1 -- 172.21.15.45:0/2790140660 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1033 (secure 0 0 0) 0x7f980007a320 con 0x7f9804072200 2023-11-28T20:35:42.429 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:42.429 INFO:teuthology.orchestra.run.smithi045.stdout:{"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":128,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1701203625,"num_in_osds":8,"osd_in_since":1701203617,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":242987008,"bytes_avail":772851126272,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":18,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":4945}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-11-28T20:34:59.149597+0000","services":{}},"progress_events":{}} 2023-11-28T20:35:42.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97e806ae40 msgr2=0x7f97e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97e806ae40 0x7f97e806d300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f98041f65e0 tx=0x7f97f4008040 comp rx=0 tx=0).stop 2023-11-28T20:35:42.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 msgr2=0x7f980415cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 0x7f980415cb00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f9800000c00 tx=0x7f98000046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 shutdown_connections 2023-11-28T20:35:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f97e806ae40 0x7f97e806d300 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9804146b60 0x7f98040b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9804146610 0x7f9804140870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 --2- 172.21.15.45:0/2790140660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9804072200 0x7f980415cb00 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 >> 172.21.15.45:0/2790140660 conn(0x7f980406ab30 msgr2=0x7f9804099b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 shutdown_connections 2023-11-28T20:35:42.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.430+0000 7f980a232640 1 -- 172.21.15.45:0/2790140660 wait complete. 2023-11-28T20:35:42.452 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-11-28T20:35:42.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:35:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.667+0000 7f4d8a8b4640 1 Processor -- start 2023-11-28T20:35:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.668+0000 7f4d8a8b4640 1 -- start start 2023-11-28T20:35:42.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.669+0000 7f4d8a8b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d8415d550 0x7f4d840cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.669+0000 7f4d8a8b4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4d840d1c80 0x7f4d840b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.669+0000 7f4d8a8b4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d840d0560 con 0x7f4d840d0100 2023-11-28T20:35:42.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.669+0000 7f4d8a8b4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d840d2230 con 0x7f4d840d1c80 2023-11-28T20:35:42.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.669+0000 7f4d8a8b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840d2510 con 0x7f4d8415d550 2023-11-28T20:35:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.669+0000 7f4d8a0b3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4d840d1c80 0x7f4d840b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.669+0000 7f4d8a0b3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4d840d1c80 0x7f4d840b6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59552/0 (socket says 172.21.15.45:59552) 2023-11-28T20:35:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.669+0000 7f4d8a0b3640 1 -- 172.21.15.45:0/3967775391 learned_addr learned my addr 172.21.15.45:0/3967775391 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:42.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.670+0000 7f4d898b2640 1 --2- 172.21.15.45:0/3967775391 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d8415d550 0x7f4d840cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.670+0000 7f4d8a0b3640 1 -- 172.21.15.45:0/3967775391 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d8415d550 msgr2=0x7f4d840cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.670+0000 7f4d8a0b3640 1 --2- 172.21.15.45:0/3967775391 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d8415d550 0x7f4d840cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.670+0000 7f4d8a0b3640 1 -- 172.21.15.45:0/3967775391 >> v1:172.21.15.190:6789/0 conn(0x7f4d840d0100 legacy=0x7f4d840d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.670+0000 7f4d8a0b3640 1 -- 172.21.15.45:0/3967775391 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d840d27f0 con 0x7f4d840d1c80 2023-11-28T20:35:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.671+0000 7f4d8a0b3640 1 --2- 172.21.15.45:0/3967775391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 0x7f4d840b6f50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009e50 tx=0x7f4d7c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a66846b777f3ac server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d888b0640 1 -- 172.21.15.45:0/3967775391 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c02cb00 con 0x7f4d840d1c80 2023-11-28T20:35:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d888b0640 1 -- 172.21.15.45:0/3967775391 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c02cc60 con 0x7f4d840d1c80 2023-11-28T20:35:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d888b0640 1 -- 172.21.15.45:0/3967775391 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0343f0 con 0x7f4d840d1c80 2023-11-28T20:35:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/3967775391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 msgr2=0x7f4d840b6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d8a8b4640 1 --2- 172.21.15.45:0/3967775391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 0x7f4d840b6f50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009e50 tx=0x7f4d7c02bf60 comp rx=0 tx=0).stop 2023-11-28T20:35:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/3967775391 shutdown_connections 2023-11-28T20:35:42.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d8a8b4640 1 --2- 172.21.15.45:0/3967775391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 0x7f4d840b6f50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d8a8b4640 1 --2- 172.21.15.45:0/3967775391 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d8415d550 0x7f4d840cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/3967775391 >> 172.21.15.45:0/3967775391 conn(0x7f4d8406ac70 msgr2=0x7f4d840b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/3967775391 shutdown_connections 2023-11-28T20:35:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.672+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/3967775391 wait complete. 2023-11-28T20:35:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.674+0000 7f4d8a8b4640 1 Processor -- start 2023-11-28T20:35:42.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.674+0000 7f4d8a8b4640 1 -- start start 2023-11-28T20:35:42.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.675+0000 7f4d8a8b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d840d0100 0x7f4d8415b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.675+0000 7f4d8a8b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 0x7f4d840ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.675+0000 7f4d8a8b4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8415d550 0x7f4d840abc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.675+0000 7f4d8a8b4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840aaa80 con 0x7f4d840d1c80 2023-11-28T20:35:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.675+0000 7f4d8a8b4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840aad60 con 0x7f4d8415d550 2023-11-28T20:35:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.675+0000 7f4d8a8b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840ab040 con 0x7f4d840d0100 2023-11-28T20:35:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d890b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 0x7f4d840ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d890b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 0x7f4d840ab710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59560/0 (socket says 172.21.15.45:59560) 2023-11-28T20:35:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d890b1640 1 -- 172.21.15.45:0/734123115 learned_addr learned my addr 172.21.15.45:0/734123115 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d898b2640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d840d0100 0x7f4d8415b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d8a0b3640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8415d550 0x7f4d840abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d890b1640 1 -- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d840d0100 msgr2=0x7f4d8415b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d890b1640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d840d0100 0x7f4d8415b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d890b1640 1 -- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8415d550 msgr2=0x7f4d840abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d890b1640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8415d550 0x7f4d840abc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d890b1640 1 -- 172.21.15.45:0/734123115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c009b30 con 0x7f4d840d1c80 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.676+0000 7f4d890b1640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 0x7f4d840ab710 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400c400 tx=0x7f4d7400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.677+0000 7f4d7a7fc640 1 -- 172.21.15.45:0/734123115 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d740052d0 con 0x7f4d840d1c80 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.677+0000 7f4d7a7fc640 1 -- 172.21.15.45:0/734123115 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d74005450 con 0x7f4d840d1c80 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.677+0000 7f4d7a7fc640 1 -- 172.21.15.45:0/734123115 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d74007260 con 0x7f4d840d1c80 2023-11-28T20:35:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.677+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d840ab320 con 0x7f4d840d1c80 2023-11-28T20:35:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.677+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8415ba70 con 0x7f4d840d1c80 2023-11-28T20:35:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.678+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8404e920 con 0x7f4d840d1c80 2023-11-28T20:35:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.679+0000 7f4d7a7fc640 1 -- 172.21.15.45:0/734123115 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4d740073c0 con 0x7f4d840d1c80 2023-11-28T20:35:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.680+0000 7f4d7a7fc640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f4d6806ae70 0x7f4d6806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.680+0000 7f4d7a7fc640 1 -- 172.21.15.45:0/734123115 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4d74088b70 con 0x7f4d840d1c80 2023-11-28T20:35:42.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.681+0000 7f4d898b2640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f4d6806ae70 0x7f4d6806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:42.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.682+0000 7f4d898b2640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f4d6806ae70 0x7f4d6806d330 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4d80006fd0 tx=0x7f4d80008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:42.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.685+0000 7f4d7a7fc640 1 -- 172.21.15.45:0/734123115 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d74053ad0 con 0x7f4d840d1c80 2023-11-28T20:35:42.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.880+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d8415d930 con 0x7f4d6806ae70 2023-11-28T20:35:42.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.902+0000 7f4d7a7fc640 1 -- 172.21.15.45:0/734123115 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281834 (secure 0 0 0) 0x7f4d8000e4b0 con 0x7f4d6806ae70 2023-11-28T20:35:42.907 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:42.910 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:35:42.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f4d6806ae70 msgr2=0x7f4d6806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f4d6806ae70 0x7f4d6806d330 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4d80006fd0 tx=0x7f4d80008040 comp rx=0 tx=0).stop 2023-11-28T20:35:42.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 msgr2=0x7f4d840ab710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:42.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 0x7f4d840ab710 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400c400 tx=0x7f4d7400c8c0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 shutdown_connections 2023-11-28T20:35:42.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f4d6806ae70 0x7f4d6806d330 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d8415d550 0x7f4d840abc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d840d1c80 0x7f4d840ab710 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 --2- 172.21.15.45:0/734123115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d840d0100 0x7f4d8415b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:42.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 >> 172.21.15.45:0/734123115 conn(0x7f4d8406ac70 msgr2=0x7f4d840b8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:42.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 shutdown_connections 2023-11-28T20:35:42.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:42.909+0000 7f4d8a8b4640 1 -- 172.21.15.45:0/734123115 wait complete. 2023-11-28T20:35:42.931 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2023-11-28T20:35:41.165641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237292,"kb_used_data":4488,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737428,"statfs":{"total":773094113280,"available":772851126272,"internally_reserved":0,"allocated":4595712,"data_stored":2616256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001885"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.273744+0000","last_change":"2023-11-28T20:35:35.273744+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.273744+0000","last_undegraded":"2023-11-28T20:35:35.273744+0000","last_fullsized":"2023-11-28T20:35:35.273744+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274640+0000","last_change":"2023-11-28T20:35:35.274640+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274640+0000","last_undegraded":"2023-11-28T20:35:35.274640+0000","last_fullsized":"2023-11-28T20:35:35.274640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287799+0000","last_change":"2023-11-28T20:35:35.287799+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287799+0000","last_undegraded":"2023-11-28T20:35:35.287799+0000","last_fullsized":"2023-11-28T20:35:35.287799+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.283269+0000","last_change":"2023-11-28T20:35:35.283269+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.283269+0000","last_undegraded":"2023-11-28T20:35:35.283269+0000","last_fullsized":"2023-11-28T20:35:35.283269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284083+0000","last_change":"2023-11-28T20:35:35.284083+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284083+0000","last_undegraded":"2023-11-28T20:35:35.284083+0000","last_fullsized":"2023-11-28T20:35:35.284083+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289049+0000","last_change":"2023-11-28T20:35:35.289049+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289049+0000","last_undegraded":"2023-11-28T20:35:35.289049+0000","last_fullsized":"2023-11-28T20:35:35.289049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.300220+0000","last_change":"2023-11-28T20:35:35.300220+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.300220+0000","last_undegraded":"2023-11-28T20:35:35.300220+0000","last_fullsized":"2023-11-28T20:35:35.300220+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275156+0000","last_change":"2023-11-28T20:35:35.275156+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275156+0000","last_undegraded":"2023-11-28T20:35:35.275156+0000","last_fullsized":"2023-11-28T20:35:35.275156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288046+0000","last_change":"2023-11-28T20:35:35.288046+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288046+0000","last_undegraded":"2023-11-28T20:35:35.288046+0000","last_fullsized":"2023-11-28T20:35:35.288046+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.302551+0000","last_change":"2023-11-28T20:35:35.302551+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.302551+0000","last_undegraded":"2023-11-28T20:35:35.302551+0000","last_fullsized":"2023-11-28T20:35:35.302551+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.292849+0000","last_change":"2023-11-28T20:35:35.292849+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.292849+0000","last_undegraded":"2023-11-28T20:35:35.292849+0000","last_fullsized":"2023-11-28T20:35:35.292849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286256+0000","last_change":"2023-11-28T20:35:35.286256+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286256+0000","last_undegraded":"2023-11-28T20:35:35.286256+0000","last_fullsized":"2023-11-28T20:35:35.286256+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296324+0000","last_change":"2023-11-28T20:35:35.296324+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296324+0000","last_undegraded":"2023-11-28T20:35:35.296324+0000","last_fullsized":"2023-11-28T20:35:35.296324+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280741+0000","last_change":"2023-11-28T20:35:35.280741+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280741+0000","last_undegraded":"2023-11-28T20:35:35.280741+0000","last_fullsized":"2023-11-28T20:35:35.280741+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.298641+0000","last_change":"2023-11-28T20:35:35.298641+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.298641+0000","last_undegraded":"2023-11-28T20:35:35.298641+0000","last_fullsized":"2023-11-28T20:35:35.298641+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288030+0000","last_change":"2023-11-28T20:35:35.288030+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288030+0000","last_undegraded":"2023-11-28T20:35:35.288030+0000","last_fullsized":"2023-11-28T20:35:35.288030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.294765+0000","last_change":"2023-11-28T20:35:35.294765+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.294765+0000","last_undegraded":"2023-11-28T20:35:35.294765+0000","last_fullsized":"2023-11-28T20:35:35.294765+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286541+0000","last_change":"2023-11-28T20:35:35.286541+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286541+0000","last_undegraded":"2023-11-28T20:35:35.286541+0000","last_fullsized":"2023-11-28T20:35:35.286541+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.281506+0000","last_change":"2023-11-28T20:35:35.281506+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.281506+0000","last_undegraded":"2023-11-28T20:35:35.281506+0000","last_fullsized":"2023-11-28T20:35:35.281506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284804+0000","last_change":"2023-11-28T20:35:35.284804+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284804+0000","last_undegraded":"2023-11-28T20:35:35.284804+0000","last_fullsized":"2023-11-28T20:35:35.284804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.295351+0000","last_change":"2023-11-28T20:35:35.295351+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.295351+0000","last_undegraded":"2023-11-28T20:35:35.295351+0000","last_fullsized":"2023-11-28T20:35:35.295351+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.278315+0000","last_change":"2023-11-28T20:35:35.278315+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.278315+0000","last_undegraded":"2023-11-28T20:35:35.278315+0000","last_fullsized":"2023-11-28T20:35:35.278315+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296171+0000","last_change":"2023-11-28T20:35:35.296171+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296171+0000","last_undegraded":"2023-11-28T20:35:35.296171+0000","last_fullsized":"2023-11-28T20:35:35.296171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.282313+0000","last_change":"2023-11-28T20:35:35.282313+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.282313+0000","last_undegraded":"2023-11-28T20:35:35.282313+0000","last_fullsized":"2023-11-28T20:35:35.282313+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.290562+0000","last_change":"2023-11-28T20:35:35.290562+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.290562+0000","last_undegraded":"2023-11-28T20:35:35.290562+0000","last_fullsized":"2023-11-28T20:35:35.290562+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288231+0000","last_change":"2023-11-28T20:35:35.288231+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288231+0000","last_undegraded":"2023-11-28T20:35:35.288231+0000","last_fullsized":"2023-11-28T20:35:35.288231+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286660+0000","last_change":"2023-11-28T20:35:35.286660+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286660+0000","last_undegraded":"2023-11-28T20:35:35.286660+0000","last_fullsized":"2023-11-28T20:35:35.286660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.300373+0000","last_change":"2023-11-28T20:35:35.300373+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.300373+0000","last_undegraded":"2023-11-28T20:35:35.300373+0000","last_fullsized":"2023-11-28T20:35:35.300373+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289057+0000","last_change":"2023-11-28T20:35:35.289057+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289057+0000","last_undegraded":"2023-11-28T20:35:35.289057+0000","last_fullsized":"2023-11-28T20:35:35.289057+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.288500+0000","last_change":"2023-11-28T20:35:35.288500+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.288500+0000","last_undegraded":"2023-11-28T20:35:35.288500+0000","last_fullsized":"2023-11-28T20:35:35.288500+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286602+0000","last_change":"2023-11-28T20:35:35.286602+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286602+0000","last_undegraded":"2023-11-28T20:35:35.286602+0000","last_fullsized":"2023-11-28T20:35:35.286602+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284839+0000","last_change":"2023-11-28T20:35:35.284839+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284839+0000","last_undegraded":"2023-11-28T20:35:35.284839+0000","last_fullsized":"2023-11-28T20:35:35.284839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.294772+0000","last_change":"2023-11-28T20:35:35.294772+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.294772+0000","last_undegraded":"2023-11-28T20:35:35.294772+0000","last_fullsized":"2023-11-28T20:35:35.294772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310344+0000","last_change":"2023-11-28T20:35:35.304535+0000","last_active":"2023-11-28T20:35:35.310344+0000","last_peered":"2023-11-28T20:35:35.310344+0000","last_clean":"2023-11-28T20:35:35.310344+0000","last_became_active":"2023-11-28T20:35:35.304193+0000","last_became_peered":"2023-11-28T20:35:35.304193+0000","last_unstale":"2023-11-28T20:35:35.310344+0000","last_undegraded":"2023-11-28T20:35:35.310344+0000","last_fullsized":"2023-11-28T20:35:35.310344+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:14.497404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289588+0000","last_change":"2023-11-28T20:35:35.289588+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289588+0000","last_undegraded":"2023-11-28T20:35:35.289588+0000","last_fullsized":"2023-11-28T20:35:35.289588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302107+0000","last_change":"2023-11-28T20:35:35.300035+0000","last_active":"2023-11-28T20:35:35.302107+0000","last_peered":"2023-11-28T20:35:35.302107+0000","last_clean":"2023-11-28T20:35:35.302107+0000","last_became_active":"2023-11-28T20:35:35.299808+0000","last_became_peered":"2023-11-28T20:35:35.299808+0000","last_unstale":"2023-11-28T20:35:35.302107+0000","last_undegraded":"2023-11-28T20:35:35.302107+0000","last_fullsized":"2023-11-28T20:35:35.302107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276409+0000","last_change":"2023-11-28T20:35:35.276409+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276409+0000","last_undegraded":"2023-11-28T20:35:35.276409+0000","last_fullsized":"2023-11-28T20:35:35.276409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.304864+0000","last_change":"2023-11-28T20:35:35.303683+0000","last_active":"2023-11-28T20:35:35.304864+0000","last_peered":"2023-11-28T20:35:35.304864+0000","last_clean":"2023-11-28T20:35:35.304864+0000","last_became_active":"2023-11-28T20:35:35.303422+0000","last_became_peered":"2023-11-28T20:35:35.303422+0000","last_unstale":"2023-11-28T20:35:35.304864+0000","last_undegraded":"2023-11-28T20:35:35.304864+0000","last_fullsized":"2023-11-28T20:35:35.304864+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274604+0000","last_change":"2023-11-28T20:35:35.274604+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274604+0000","last_undegraded":"2023-11-28T20:35:35.274604+0000","last_fullsized":"2023-11-28T20:35:35.274604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.298920+0000","last_change":"2023-11-28T20:35:35.290259+0000","last_active":"2023-11-28T20:35:35.298920+0000","last_peered":"2023-11-28T20:35:35.298920+0000","last_clean":"2023-11-28T20:35:35.298920+0000","last_became_active":"2023-11-28T20:35:35.286178+0000","last_became_peered":"2023-11-28T20:35:35.286178+0000","last_unstale":"2023-11-28T20:35:35.298920+0000","last_undegraded":"2023-11-28T20:35:35.298920+0000","last_fullsized":"2023-11-28T20:35:35.298920+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.285970+0000","last_change":"2023-11-28T20:35:35.285970+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.285970+0000","last_undegraded":"2023-11-28T20:35:35.285970+0000","last_fullsized":"2023-11-28T20:35:35.285970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310834+0000","last_change":"2023-11-28T20:35:35.306240+0000","last_active":"2023-11-28T20:35:35.310834+0000","last_peered":"2023-11-28T20:35:35.310834+0000","last_clean":"2023-11-28T20:35:35.310834+0000","last_became_active":"2023-11-28T20:35:35.305889+0000","last_became_peered":"2023-11-28T20:35:35.305889+0000","last_unstale":"2023-11-28T20:35:35.310834+0000","last_undegraded":"2023-11-28T20:35:35.310834+0000","last_fullsized":"2023-11-28T20:35:35.310834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:29.262104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.298012+0000","last_change":"2023-11-28T20:35:35.298012+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.298012+0000","last_undegraded":"2023-11-28T20:35:35.298012+0000","last_fullsized":"2023-11-28T20:35:35.298012+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.313113+0000","last_change":"2023-11-28T20:35:35.311106+0000","last_active":"2023-11-28T20:35:35.313113+0000","last_peered":"2023-11-28T20:35:35.313113+0000","last_clean":"2023-11-28T20:35:35.313113+0000","last_became_active":"2023-11-28T20:35:35.310832+0000","last_became_peered":"2023-11-28T20:35:35.310832+0000","last_unstale":"2023-11-28T20:35:35.313113+0000","last_undegraded":"2023-11-28T20:35:35.313113+0000","last_fullsized":"2023-11-28T20:35:35.313113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:01:49.996784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276568+0000","last_change":"2023-11-28T20:35:35.276568+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276568+0000","last_undegraded":"2023-11-28T20:35:35.276568+0000","last_fullsized":"2023-11-28T20:35:35.276568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312883+0000","last_change":"2023-11-28T20:35:35.311801+0000","last_active":"2023-11-28T20:35:35.312883+0000","last_peered":"2023-11-28T20:35:35.312883+0000","last_clean":"2023-11-28T20:35:35.312883+0000","last_became_active":"2023-11-28T20:35:35.311381+0000","last_became_peered":"2023-11-28T20:35:35.311381+0000","last_unstale":"2023-11-28T20:35:35.312883+0000","last_undegraded":"2023-11-28T20:35:35.312883+0000","last_fullsized":"2023-11-28T20:35:35.312883+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276964+0000","last_change":"2023-11-28T20:35:35.276964+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276964+0000","last_undegraded":"2023-11-28T20:35:35.276964+0000","last_fullsized":"2023-11-28T20:35:35.276964+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310284+0000","last_change":"2023-11-28T20:35:35.305104+0000","last_active":"2023-11-28T20:35:35.310284+0000","last_peered":"2023-11-28T20:35:35.310284+0000","last_clean":"2023-11-28T20:35:35.310284+0000","last_became_active":"2023-11-28T20:35:35.304907+0000","last_became_peered":"2023-11-28T20:35:35.304907+0000","last_unstale":"2023-11-28T20:35:35.310284+0000","last_undegraded":"2023-11-28T20:35:35.310284+0000","last_fullsized":"2023-11-28T20:35:35.310284+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:44.520691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280744+0000","last_change":"2023-11-28T20:35:35.280744+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280744+0000","last_undegraded":"2023-11-28T20:35:35.280744+0000","last_fullsized":"2023-11-28T20:35:35.280744+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.296213+0000","last_change":"2023-11-28T20:35:35.295019+0000","last_active":"2023-11-28T20:35:35.296213+0000","last_peered":"2023-11-28T20:35:35.296213+0000","last_clean":"2023-11-28T20:35:35.296213+0000","last_became_active":"2023-11-28T20:35:35.294753+0000","last_became_peered":"2023-11-28T20:35:35.294753+0000","last_unstale":"2023-11-28T20:35:35.296213+0000","last_undegraded":"2023-11-28T20:35:35.296213+0000","last_fullsized":"2023-11-28T20:35:35.296213+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:24.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.293244+0000","last_change":"2023-11-28T20:35:35.293244+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.293244+0000","last_undegraded":"2023-11-28T20:35:35.293244+0000","last_fullsized":"2023-11-28T20:35:35.293244+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312174+0000","last_change":"2023-11-28T20:35:35.308107+0000","last_active":"2023-11-28T20:35:35.312174+0000","last_peered":"2023-11-28T20:35:35.312174+0000","last_clean":"2023-11-28T20:35:35.312174+0000","last_became_active":"2023-11-28T20:35:35.307828+0000","last_became_peered":"2023-11-28T20:35:35.307828+0000","last_unstale":"2023-11-28T20:35:35.312174+0000","last_undegraded":"2023-11-28T20:35:35.312174+0000","last_fullsized":"2023-11-28T20:35:35.312174+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:24.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287024+0000","last_change":"2023-11-28T20:35:35.287024+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287024+0000","last_undegraded":"2023-11-28T20:35:35.287024+0000","last_fullsized":"2023-11-28T20:35:35.287024+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311202+0000","last_change":"2023-11-28T20:35:35.307353+0000","last_active":"2023-11-28T20:35:35.311202+0000","last_peered":"2023-11-28T20:35:35.311202+0000","last_clean":"2023-11-28T20:35:35.311202+0000","last_became_active":"2023-11-28T20:35:35.307054+0000","last_became_peered":"2023-11-28T20:35:35.307054+0000","last_unstale":"2023-11-28T20:35:35.311202+0000","last_undegraded":"2023-11-28T20:35:35.311202+0000","last_fullsized":"2023-11-28T20:35:35.311202+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:01:49.996784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275888+0000","last_change":"2023-11-28T20:35:35.275888+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275888+0000","last_undegraded":"2023-11-28T20:35:35.275888+0000","last_fullsized":"2023-11-28T20:35:35.275888+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.301882+0000","last_change":"2023-11-28T20:35:35.298444+0000","last_active":"2023-11-28T20:35:35.301882+0000","last_peered":"2023-11-28T20:35:35.301882+0000","last_clean":"2023-11-28T20:35:35.301882+0000","last_became_active":"2023-11-28T20:35:35.297824+0000","last_became_peered":"2023-11-28T20:35:35.297824+0000","last_unstale":"2023-11-28T20:35:35.301882+0000","last_undegraded":"2023-11-28T20:35:35.301882+0000","last_fullsized":"2023-11-28T20:35:35.301882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:29.262104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.285405+0000","last_change":"2023-11-28T20:35:35.285405+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.285405+0000","last_undegraded":"2023-11-28T20:35:35.285405+0000","last_fullsized":"2023-11-28T20:35:35.285405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.292992+0000","last_change":"2023-11-28T20:35:35.290127+0000","last_active":"2023-11-28T20:35:35.292992+0000","last_peered":"2023-11-28T20:35:35.292992+0000","last_clean":"2023-11-28T20:35:35.292992+0000","last_became_active":"2023-11-28T20:35:35.289788+0000","last_became_peered":"2023-11-28T20:35:35.289788+0000","last_unstale":"2023-11-28T20:35:35.292992+0000","last_undegraded":"2023-11-28T20:35:35.292992+0000","last_fullsized":"2023-11-28T20:35:35.292992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.273794+0000","last_change":"2023-11-28T20:35:35.273794+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.273794+0000","last_undegraded":"2023-11-28T20:35:35.273794+0000","last_fullsized":"2023-11-28T20:35:35.273794+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311743+0000","last_change":"2023-11-28T20:35:35.310764+0000","last_active":"2023-11-28T20:35:35.311743+0000","last_peered":"2023-11-28T20:35:35.311743+0000","last_clean":"2023-11-28T20:35:35.311743+0000","last_became_active":"2023-11-28T20:35:35.310478+0000","last_became_peered":"2023-11-28T20:35:35.310478+0000","last_unstale":"2023-11-28T20:35:35.311743+0000","last_undegraded":"2023-11-28T20:35:35.311743+0000","last_fullsized":"2023-11-28T20:35:35.311743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:05.096415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.295908+0000","last_change":"2023-11-28T20:35:35.295908+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.295908+0000","last_undegraded":"2023-11-28T20:35:35.295908+0000","last_fullsized":"2023-11-28T20:35:35.295908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310256+0000","last_change":"2023-11-28T20:35:35.304319+0000","last_active":"2023-11-28T20:35:35.310256+0000","last_peered":"2023-11-28T20:35:35.310256+0000","last_clean":"2023-11-28T20:35:35.310256+0000","last_became_active":"2023-11-28T20:35:35.302594+0000","last_became_peered":"2023-11-28T20:35:35.302594+0000","last_unstale":"2023-11-28T20:35:35.310256+0000","last_undegraded":"2023-11-28T20:35:35.310256+0000","last_fullsized":"2023-11-28T20:35:35.310256+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:01.686608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.276470+0000","last_change":"2023-11-28T20:35:35.276470+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.276470+0000","last_undegraded":"2023-11-28T20:35:35.276470+0000","last_fullsized":"2023-11-28T20:35:35.276470+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312059+0000","last_change":"2023-11-28T20:35:35.310387+0000","last_active":"2023-11-28T20:35:35.312059+0000","last_peered":"2023-11-28T20:35:35.312059+0000","last_clean":"2023-11-28T20:35:35.312059+0000","last_became_active":"2023-11-28T20:35:35.310118+0000","last_became_peered":"2023-11-28T20:35:35.310118+0000","last_unstale":"2023-11-28T20:35:35.312059+0000","last_undegraded":"2023-11-28T20:35:35.312059+0000","last_fullsized":"2023-11-28T20:35:35.312059+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275327+0000","last_change":"2023-11-28T20:35:35.275327+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275327+0000","last_undegraded":"2023-11-28T20:35:35.275327+0000","last_fullsized":"2023-11-28T20:35:35.275327+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311828+0000","last_change":"2023-11-28T20:35:35.308420+0000","last_active":"2023-11-28T20:35:35.311828+0000","last_peered":"2023-11-28T20:35:35.311828+0000","last_clean":"2023-11-28T20:35:35.311828+0000","last_became_active":"2023-11-28T20:35:35.305473+0000","last_became_peered":"2023-11-28T20:35:35.305473+0000","last_unstale":"2023-11-28T20:35:35.311828+0000","last_undegraded":"2023-11-28T20:35:35.311828+0000","last_fullsized":"2023-11-28T20:35:35.311828+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.292286+0000","last_change":"2023-11-28T20:35:35.292286+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.292286+0000","last_undegraded":"2023-11-28T20:35:35.292286+0000","last_fullsized":"2023-11-28T20:35:35.292286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.313168+0000","last_change":"2023-11-28T20:35:35.311195+0000","last_active":"2023-11-28T20:35:35.313168+0000","last_peered":"2023-11-28T20:35:35.313168+0000","last_clean":"2023-11-28T20:35:35.313168+0000","last_became_active":"2023-11-28T20:35:35.310875+0000","last_became_peered":"2023-11-28T20:35:35.310875+0000","last_unstale":"2023-11-28T20:35:35.313168+0000","last_undegraded":"2023-11-28T20:35:35.313168+0000","last_fullsized":"2023-11-28T20:35:35.313168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.277762+0000","last_change":"2023-11-28T20:35:35.277762+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.277762+0000","last_undegraded":"2023-11-28T20:35:35.277762+0000","last_fullsized":"2023-11-28T20:35:35.277762+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302313+0000","last_change":"2023-11-28T20:35:35.298942+0000","last_active":"2023-11-28T20:35:35.302313+0000","last_peered":"2023-11-28T20:35:35.302313+0000","last_clean":"2023-11-28T20:35:35.302313+0000","last_became_active":"2023-11-28T20:35:35.298685+0000","last_became_peered":"2023-11-28T20:35:35.298685+0000","last_unstale":"2023-11-28T20:35:35.302313+0000","last_undegraded":"2023-11-28T20:35:35.302313+0000","last_fullsized":"2023-11-28T20:35:35.302313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:45.323950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.280904+0000","last_change":"2023-11-28T20:35:35.280904+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.280904+0000","last_undegraded":"2023-11-28T20:35:35.280904+0000","last_fullsized":"2023-11-28T20:35:35.280904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.304860+0000","last_change":"2023-11-28T20:35:35.303747+0000","last_active":"2023-11-28T20:35:35.304860+0000","last_peered":"2023-11-28T20:35:35.304860+0000","last_clean":"2023-11-28T20:35:35.304860+0000","last_became_active":"2023-11-28T20:35:35.303441+0000","last_became_peered":"2023-11-28T20:35:35.303441+0000","last_unstale":"2023-11-28T20:35:35.304860+0000","last_undegraded":"2023-11-28T20:35:35.304860+0000","last_fullsized":"2023-11-28T20:35:35.304860+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:44.520691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284186+0000","last_change":"2023-11-28T20:35:35.284186+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284186+0000","last_undegraded":"2023-11-28T20:35:35.284186+0000","last_fullsized":"2023-11-28T20:35:35.284186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.292924+0000","last_change":"2023-11-28T20:35:35.290919+0000","last_active":"2023-11-28T20:35:35.292924+0000","last_peered":"2023-11-28T20:35:35.292924+0000","last_clean":"2023-11-28T20:35:35.292924+0000","last_became_active":"2023-11-28T20:35:35.290407+0000","last_became_peered":"2023-11-28T20:35:35.290407+0000","last_unstale":"2023-11-28T20:35:35.292924+0000","last_undegraded":"2023-11-28T20:35:35.292924+0000","last_fullsized":"2023-11-28T20:35:35.292924+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:51.691786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.286534+0000","last_change":"2023-11-28T20:35:35.286534+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.286534+0000","last_undegraded":"2023-11-28T20:35:35.286534+0000","last_fullsized":"2023-11-28T20:35:35.286534+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.302345+0000","last_change":"2023-11-28T20:35:35.299354+0000","last_active":"2023-11-28T20:35:35.302345+0000","last_peered":"2023-11-28T20:35:35.302345+0000","last_clean":"2023-11-28T20:35:35.302345+0000","last_became_active":"2023-11-28T20:35:35.299056+0000","last_became_peered":"2023-11-28T20:35:35.299056+0000","last_unstale":"2023-11-28T20:35:35.302345+0000","last_undegraded":"2023-11-28T20:35:35.302345+0000","last_fullsized":"2023-11-28T20:35:35.302345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:18.168444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.297513+0000","last_change":"2023-11-28T20:35:35.297513+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.297513+0000","last_undegraded":"2023-11-28T20:35:35.297513+0000","last_fullsized":"2023-11-28T20:35:35.297513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.310069+0000","last_change":"2023-11-28T20:35:35.306856+0000","last_active":"2023-11-28T20:35:35.310069+0000","last_peered":"2023-11-28T20:35:35.310069+0000","last_clean":"2023-11-28T20:35:35.310069+0000","last_became_active":"2023-11-28T20:35:35.306562+0000","last_became_peered":"2023-11-28T20:35:35.306562+0000","last_unstale":"2023-11-28T20:35:35.310069+0000","last_undegraded":"2023-11-28T20:35:35.310069+0000","last_fullsized":"2023-11-28T20:35:35.310069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.275988+0000","last_change":"2023-11-28T20:35:35.275988+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.275988+0000","last_undegraded":"2023-11-28T20:35:35.275988+0000","last_fullsized":"2023-11-28T20:35:35.275988+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.293453+0000","last_change":"2023-11-28T20:35:35.291161+0000","last_active":"2023-11-28T20:35:35.293453+0000","last_peered":"2023-11-28T20:35:35.293453+0000","last_clean":"2023-11-28T20:35:35.293453+0000","last_became_active":"2023-11-28T20:35:35.290905+0000","last_became_peered":"2023-11-28T20:35:35.290905+0000","last_unstale":"2023-11-28T20:35:35.293453+0000","last_undegraded":"2023-11-28T20:35:35.293453+0000","last_fullsized":"2023-11-28T20:35:35.293453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:57.631654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.287224+0000","last_change":"2023-11-28T20:35:35.287224+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.287224+0000","last_undegraded":"2023-11-28T20:35:35.287224+0000","last_fullsized":"2023-11-28T20:35:35.287224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.311849+0000","last_change":"2023-11-28T20:35:35.308961+0000","last_active":"2023-11-28T20:35:35.311849+0000","last_peered":"2023-11-28T20:35:35.311849+0000","last_clean":"2023-11-28T20:35:35.311849+0000","last_became_active":"2023-11-28T20:35:35.308687+0000","last_became_peered":"2023-11-28T20:35:35.308687+0000","last_unstale":"2023-11-28T20:35:35.311849+0000","last_undegraded":"2023-11-28T20:35:35.311849+0000","last_fullsized":"2023-11-28T20:35:35.311849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:18.243749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.291381+0000","last_change":"2023-11-28T20:35:35.291381+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.291381+0000","last_undegraded":"2023-11-28T20:35:35.291381+0000","last_fullsized":"2023-11-28T20:35:35.291381+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.312886+0000","last_change":"2023-11-28T20:35:35.311276+0000","last_active":"2023-11-28T20:35:35.312886+0000","last_peered":"2023-11-28T20:35:35.312886+0000","last_clean":"2023-11-28T20:35:35.312886+0000","last_became_active":"2023-11-28T20:35:35.310955+0000","last_became_peered":"2023-11-28T20:35:35.310955+0000","last_unstale":"2023-11-28T20:35:35.312886+0000","last_undegraded":"2023-11-28T20:35:35.312886+0000","last_fullsized":"2023-11-28T20:35:35.312886+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:07:14.394525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.274111+0000","last_change":"2023-11-28T20:35:35.274111+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.274111+0000","last_undegraded":"2023-11-28T20:35:35.274111+0000","last_fullsized":"2023-11-28T20:35:35.274111+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.297236+0000","last_change":"2023-11-28T20:35:35.295046+0000","last_active":"2023-11-28T20:35:35.297236+0000","last_peered":"2023-11-28T20:35:35.297236+0000","last_clean":"2023-11-28T20:35:35.297236+0000","last_became_active":"2023-11-28T20:35:35.294261+0000","last_became_peered":"2023-11-28T20:35:35.294261+0000","last_unstale":"2023-11-28T20:35:35.297236+0000","last_undegraded":"2023-11-28T20:35:35.297236+0000","last_fullsized":"2023-11-28T20:35:35.297236+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:01.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.284924+0000","last_change":"2023-11-28T20:35:35.284924+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.284924+0000","last_undegraded":"2023-11-28T20:35:35.284924+0000","last_fullsized":"2023-11-28T20:35:35.284924+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.305226+0000","last_change":"2023-11-28T20:35:35.304486+0000","last_active":"2023-11-28T20:35:35.305226+0000","last_peered":"2023-11-28T20:35:35.305226+0000","last_clean":"2023-11-28T20:35:35.305226+0000","last_became_active":"2023-11-28T20:35:35.304168+0000","last_became_peered":"2023-11-28T20:35:35.304168+0000","last_unstale":"2023-11-28T20:35:35.305226+0000","last_undegraded":"2023-11-28T20:35:35.305226+0000","last_fullsized":"2023-11-28T20:35:35.305226+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:36.422005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.289424+0000","last_change":"2023-11-28T20:35:35.289424+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.289424+0000","last_undegraded":"2023-11-28T20:35:35.289424+0000","last_fullsized":"2023-11-28T20:35:35.289424+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.305224+0000","last_change":"2023-11-28T20:35:35.300968+0000","last_active":"2023-11-28T20:35:35.305224+0000","last_peered":"2023-11-28T20:35:35.305224+0000","last_clean":"2023-11-28T20:35:35.305224+0000","last_became_active":"2023-11-28T20:35:35.300389+0000","last_became_peered":"2023-11-28T20:35:35.300389+0000","last_unstale":"2023-11-28T20:35:35.305224+0000","last_undegraded":"2023-11-28T20:35:35.305224+0000","last_fullsized":"2023-11-28T20:35:35.305224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:18:34.499129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.279637+0000","last_change":"2023-11-28T20:35:35.279637+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.279637+0000","last_undegraded":"2023-11-28T20:35:35.279637+0000","last_fullsized":"2023-11-28T20:35:35.279637+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.298860+0000","last_change":"2023-11-28T20:35:35.289145+0000","last_active":"2023-11-28T20:35:35.298860+0000","last_peered":"2023-11-28T20:35:35.298860+0000","last_clean":"2023-11-28T20:35:35.298860+0000","last_became_active":"2023-11-28T20:35:35.288419+0000","last_became_peered":"2023-11-28T20:35:35.288419+0000","last_unstale":"2023-11-28T20:35:35.298860+0000","last_undegraded":"2023-11-28T20:35:35.298860+0000","last_fullsized":"2023-11-28T20:35:35.298860+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:14.497404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.296014+0000","last_change":"2023-11-28T20:35:35.296014+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.296014+0000","last_undegraded":"2023-11-28T20:35:35.296014+0000","last_fullsized":"2023-11-28T20:35:35.296014+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.295837+0000","last_change":"2023-11-28T20:35:35.293571+0000","last_active":"2023-11-28T20:35:35.295837+0000","last_peered":"2023-11-28T20:35:35.295837+0000","last_clean":"2023-11-28T20:35:35.295837+0000","last_became_active":"2023-11-28T20:35:35.293294+0000","last_became_peered":"2023-11-28T20:35:35.293294+0000","last_unstale":"2023-11-28T20:35:35.295837+0000","last_undegraded":"2023-11-28T20:35:35.295837+0000","last_fullsized":"2023-11-28T20:35:35.295837+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:10.536815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-11-28T20:35:35.293482+0000","last_change":"2023-11-28T20:35:35.293482+0000","last_active":"2023-11-28T20:35:35.259596+0000","last_peered":"2023-11-28T20:35:35.259596+0000","last_clean":"2023-11-28T20:35:35.259596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T20:35:35.293482+0000","last_undegraded":"2023-11-28T20:35:35.293482+0000","last_fullsized":"2023-11-28T20:35:35.293482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T20:35:35.308144+0000","last_change":"2023-11-28T20:35:35.305966+0000","last_active":"2023-11-28T20:35:35.308144+0000","last_peered":"2023-11-28T20:35:35.308144+0000","last_clean":"2023-11-28T20:35:35.308144+0000","last_became_active":"2023-11-28T20:35:35.305715+0000","last_became_peered":"2023-11-28T20:35:35.305715+0000","last_unstale":"2023-11-28T20:35:35.308144+0000","last_undegraded":"2023-11-28T20:35:35.308144+0000","last_fullsized":"2023-11-28T20:35:35.308144+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:18.243749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-11-28T20:35:34.277946+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:35:34.277946+0000","last_peered":"2023-11-28T20:35:34.277946+0000","last_clean":"2023-11-28T20:35:34.277946+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:35:34.277946+0000","last_undegraded":"2023-11-28T20:35:34.277946+0000","last_fullsized":"2023-11-28T20:35:34.277946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":769808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.753},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.2370000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.048},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.3839999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.355}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.151},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.5259999999999998,"5min":2.5259999999999998,"15min":2.5259999999999998},"last":1.1120000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.4390000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"last":1.4690000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.619,"15min":1.619},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.516},{"interface":"front","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.7909999999999999,"5min":2.7909999999999999,"15min":2.7909999999999999},"last":1.28}]}]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":5.0430000000000001,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.242}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.736},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":1.54}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":1.637}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":4.7649999999999997,"5min":4.7649999999999997,"15min":4.7649999999999997},"last":1.1870000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":3.1989999999999998,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":3.6429999999999998,"5min":3.6429999999999998,"15min":3.6429999999999998},"last":1.8120000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.917,"5min":1.917,"15min":1.917},"min":{"1min":1.018,"5min":1.018,"15min":1.018},"max":{"1min":4.9790000000000001,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.5700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.216},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.268}]},{"osd":1,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.877},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.071}]},{"osd":2,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":1.1259999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":1.2410000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":1.4019999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.9319999999999999,"5min":2.9319999999999999,"15min":2.9319999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":1.3049999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":0.997}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.83099999999999996}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.167},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.98399999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.419},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":0.88100000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.3140000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.268},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.2270000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.627},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.3520000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":2.54,"15min":2.54},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":20.568999999999999,"5min":20.568999999999999,"15min":20.568999999999999},"last":1.3},{"interface":"front","average":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":20.677,"5min":20.677,"15min":20.677},"last":1.5029999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.80500000000000005}]},{"osd":1,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.7090000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":0.624},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.4950000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.71999999999999997}]},{"osd":2,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":5.6509999999999998,"5min":5.6509999999999998,"15min":5.6509999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.2470000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.399},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":0.872}]},{"osd":5,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.514,"15min":1.514},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.7610000000000001,"5min":2.7610000000000001,"15min":2.7610000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.492}]},{"osd":6,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.98,"5min":2.98,"15min":2.98},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.585,"15min":1.585},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.5289999999999999,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.1339999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.556},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.1859999999999999,"5min":3.1859999999999999,"15min":3.1859999999999999},"last":0.76200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.625},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.57299999999999995}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.79400000000000004}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.83699999999999997}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.3160000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":1.113}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.931,"5min":2.931,"15min":2.931},"last":1.452},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.202}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.9649999999999999,"5min":2.9649999999999999,"15min":2.9649999999999999},"last":1.395}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":425984,"data_stored":179440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.89600000000000002}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.637}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.276},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.165}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":3.5259999999999998,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":1.339}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":3.637,"5min":3.637,"15min":3.637},"last":1.484},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.534}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":1.5920000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.395},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.4660000000000002,"5min":4.4660000000000002,"15min":4.4660000000000002},"last":1.5109999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":769808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.964,"5min":2.964,"15min":2.964},"last":0.61099999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.504},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.52500000000000002}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.68300000000000005}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.89700000000000002}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.536,"5min":2.536,"15min":2.536},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":0.96099999999999997}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":2.8650000000000002,"5min":2.8650000000000002,"15min":2.8650000000000002},"last":1.085},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":1.123,"5min":1.123,"15min":1.123},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:35:45.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:35:45.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:35:45.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:35:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.169+0000 7f454e1f1640 1 Processor -- start 2023-11-28T20:35:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.169+0000 7f454e1f1640 1 -- start start 2023-11-28T20:35:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.171+0000 7f454e1f1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f45480b10e0 0x7f45480b4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.171+0000 7f454e1f1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454815dcc0 con 0x7f45481438b0 2023-11-28T20:35:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.171+0000 7f454e1f1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454815e840 con 0x7f45480b0c30 2023-11-28T20:35:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.171+0000 7f454e1f1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45480b26b0 con 0x7f45480b10e0 2023-11-28T20:35:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.171+0000 7f45477fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f45481438b0 0x7f45480b1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51580/0 (socket says 172.21.15.45:51580) 2023-11-28T20:35:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.171+0000 7f45477fe640 1 -- 172.21.15.45:0/3703025916 learned_addr learned my addr 172.21.15.45:0/3703025916 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.171+0000 7f4547fff640 1 --2- 172.21.15.45:0/3703025916 >> v2:172.21.15.45:3300/0 conn(0x7f45480b10e0 0x7f45480b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.172+0000 7f4546ffd640 1 -- 172.21.15.45:0/3703025916 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255116226 0 0) 0x7f454815dcc0 con 0x7f45481438b0 2023-11-28T20:35:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.172+0000 7f4546ffd640 1 -- 172.21.15.45:0/3703025916 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534005730 con 0x7f45481438b0 2023-11-28T20:35:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.172+0000 7f4547fff640 1 -- 172.21.15.45:0/3703025916 >> v1:172.21.15.190:6789/0 conn(0x7f45480b0c30 legacy=0x7f45480b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.172+0000 7f4547fff640 1 -- 172.21.15.45:0/3703025916 >> v1:172.21.15.45:6789/0 conn(0x7f45481438b0 legacy=0x7f45480b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.172+0000 7f4547fff640 1 -- 172.21.15.45:0/3703025916 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480b1530 con 0x7f45480b10e0 2023-11-28T20:35:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.173+0000 7f4547fff640 1 --2- 172.21.15.45:0/3703025916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b10e0 0x7f45480b4470 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4540009d90 tx=0x7f454002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9030d132a92cb63 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:46.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.173+0000 7f4546ffd640 1 -- 172.21.15.45:0/3703025916 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454002cb40 con 0x7f45480b10e0 2023-11-28T20:35:46.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.173+0000 7f4546ffd640 1 -- 172.21.15.45:0/3703025916 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454002cca0 con 0x7f45480b10e0 2023-11-28T20:35:46.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.173+0000 7f4546ffd640 1 -- 172.21.15.45:0/3703025916 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540034640 con 0x7f45480b10e0 2023-11-28T20:35:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.174+0000 7f454e1f1640 1 -- 172.21.15.45:0/3703025916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b10e0 msgr2=0x7f45480b4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.174+0000 7f454e1f1640 1 --2- 172.21.15.45:0/3703025916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b10e0 0x7f45480b4470 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4540009d90 tx=0x7f454002bd90 comp rx=0 tx=0).stop 2023-11-28T20:35:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.174+0000 7f454e1f1640 1 -- 172.21.15.45:0/3703025916 shutdown_connections 2023-11-28T20:35:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.174+0000 7f454e1f1640 1 --2- 172.21.15.45:0/3703025916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b10e0 0x7f45480b4470 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.174+0000 7f454e1f1640 1 -- 172.21.15.45:0/3703025916 >> 172.21.15.45:0/3703025916 conn(0x7f454806a500 msgr2=0x7f454806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.174+0000 7f454e1f1640 1 -- 172.21.15.45:0/3703025916 shutdown_connections 2023-11-28T20:35:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.174+0000 7f454e1f1640 1 -- 172.21.15.45:0/3703025916 wait complete. 2023-11-28T20:35:46.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.176+0000 7f454e1f1640 1 Processor -- start 2023-11-28T20:35:46.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.176+0000 7f454e1f1640 1 -- start start 2023-11-28T20:35:46.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.177+0000 7f454e1f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b0c30 0x7f454813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.177+0000 7f454e1f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45480b10e0 0x7f45480adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.177+0000 7f454e1f1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f45481438b0 0x7f4548159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.177+0000 7f454e1f1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45481f7a90 con 0x7f45480b0c30 2023-11-28T20:35:46.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.177+0000 7f454e1f1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45481f6bc0 con 0x7f45480b10e0 2023-11-28T20:35:46.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.177+0000 7f454e1f1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454813f2b0 con 0x7f45481438b0 2023-11-28T20:35:46.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.177+0000 7f45477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b0c30 0x7f454813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:46.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.177+0000 7f45477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b0c30 0x7f454813ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40856/0 (socket says 172.21.15.45:40856) 2023-11-28T20:35:46.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.177+0000 7f45477fe640 1 -- 172.21.15.45:0/2242143660 learned_addr learned my addr 172.21.15.45:0/2242143660 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:46.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.178+0000 7f45477fe640 1 -- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f45481438b0 msgr2=0x7f4548159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.178+0000 7f45477fe640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f45481438b0 0x7f4548159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.178+0000 7f45477fe640 1 -- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45480b10e0 msgr2=0x7f45480adbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.178+0000 7f453ffff640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45480b10e0 0x7f45480adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:46.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.178+0000 7f45477fe640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45480b10e0 0x7f45480adbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.178+0000 7f45477fe640 1 -- 172.21.15.45:0/2242143660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540009a70 con 0x7f45480b0c30 2023-11-28T20:35:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.178+0000 7f45477fe640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b0c30 0x7f454813ed70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f453800c390 tx=0x7f453800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.178+0000 7f4544ff9640 1 -- 172.21.15.45:0/2242143660 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538005ce0 con 0x7f45480b0c30 2023-11-28T20:35:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.178+0000 7f4544ff9640 1 -- 172.21.15.45:0/2242143660 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4538005e40 con 0x7f45480b0c30 2023-11-28T20:35:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.179+0000 7f4544ff9640 1 -- 172.21.15.45:0/2242143660 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45380072f0 con 0x7f45480b0c30 2023-11-28T20:35:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.179+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45481f6e10 con 0x7f45480b0c30 2023-11-28T20:35:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.179+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45481f73d0 con 0x7f45480b0c30 2023-11-28T20:35:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.180+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45481f36c0 con 0x7f45480b0c30 2023-11-28T20:35:46.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.181+0000 7f4544ff9640 1 -- 172.21.15.45:0/2242143660 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4538007450 con 0x7f45480b0c30 2023-11-28T20:35:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.182+0000 7f4544ff9640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f452806ae40 0x7f452806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.182+0000 7f4544ff9640 1 -- 172.21.15.45:0/2242143660 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4538088d80 con 0x7f45480b0c30 2023-11-28T20:35:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.182+0000 7f453ffff640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f452806ae40 0x7f452806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.183+0000 7f453ffff640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f452806ae40 0x7f452806d300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f45480ae620 tx=0x7f4530008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.186+0000 7f4544ff9640 1 -- 172.21.15.45:0/2242143660 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4538053ce0 con 0x7f45480b0c30 2023-11-28T20:35:46.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.378+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f454815e4d0 con 0x7f452806ae40 2023-11-28T20:35:46.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.402+0000 7f4544ff9640 1 -- 172.21.15.45:0/2242143660 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287204 (secure 0 0 0) 0x7f453000c4f0 con 0x7f452806ae40 2023-11-28T20:35:46.408 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:46.409 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:35:46.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.407+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f452806ae40 msgr2=0x7f452806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.407+0000 7f454e1f1640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f452806ae40 0x7f452806d300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f45480ae620 tx=0x7f4530008040 comp rx=0 tx=0).stop 2023-11-28T20:35:46.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.407+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b0c30 msgr2=0x7f454813ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.407+0000 7f454e1f1640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b0c30 0x7f454813ed70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f453800c390 tx=0x7f453800c850 comp rx=0 tx=0).stop 2023-11-28T20:35:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.408+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 shutdown_connections 2023-11-28T20:35:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.408+0000 7f454e1f1640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f452806ae40 0x7f452806d300 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.408+0000 7f454e1f1640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f45481438b0 0x7f4548159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.408+0000 7f454e1f1640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45480b10e0 0x7f45480adbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.408+0000 7f454e1f1640 1 --2- 172.21.15.45:0/2242143660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45480b0c30 0x7f454813ed70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.408+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 >> 172.21.15.45:0/2242143660 conn(0x7f454806a500 msgr2=0x7f454815a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.408+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 shutdown_connections 2023-11-28T20:35:46.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.408+0000 7f454e1f1640 1 -- 172.21.15.45:0/2242143660 wait complete. 2023-11-28T20:35:46.427 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2023-11-28T20:35:45.167271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237596,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737124,"statfs":{"total":773094113280,"available":772850814976,"internally_reserved":0,"allocated":4857856,"data_stored":2730796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.877791"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.295020+0000","last_change":"2023-11-28T20:35:36.314366+0000","last_active":"2023-11-28T20:35:37.295020+0000","last_peered":"2023-11-28T20:35:37.295020+0000","last_clean":"2023-11-28T20:35:37.295020+0000","last_became_active":"2023-11-28T20:35:36.313899+0000","last_became_peered":"2023-11-28T20:35:36.313899+0000","last_unstale":"2023-11-28T20:35:37.295020+0000","last_undegraded":"2023-11-28T20:35:37.295020+0000","last_fullsized":"2023-11-28T20:35:37.295020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:29:00.040689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294790+0000","last_change":"2023-11-28T20:35:36.318879+0000","last_active":"2023-11-28T20:35:37.294790+0000","last_peered":"2023-11-28T20:35:37.294790+0000","last_clean":"2023-11-28T20:35:37.294790+0000","last_became_active":"2023-11-28T20:35:36.318488+0000","last_became_peered":"2023-11-28T20:35:36.318488+0000","last_unstale":"2023-11-28T20:35:37.294790+0000","last_undegraded":"2023-11-28T20:35:37.294790+0000","last_fullsized":"2023-11-28T20:35:37.294790+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:05.518189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853866+0000","last_change":"2023-11-28T20:35:36.325984+0000","last_active":"2023-11-28T20:35:37.853866+0000","last_peered":"2023-11-28T20:35:37.853866+0000","last_clean":"2023-11-28T20:35:37.853866+0000","last_became_active":"2023-11-28T20:35:36.322475+0000","last_became_peered":"2023-11-28T20:35:36.322475+0000","last_unstale":"2023-11-28T20:35:37.853866+0000","last_undegraded":"2023-11-28T20:35:37.853866+0000","last_fullsized":"2023-11-28T20:35:37.853866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:38.870814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.486671+0000","last_change":"2023-11-28T20:35:36.312625+0000","last_active":"2023-11-28T20:35:37.486671+0000","last_peered":"2023-11-28T20:35:37.486671+0000","last_clean":"2023-11-28T20:35:37.486671+0000","last_became_active":"2023-11-28T20:35:36.312319+0000","last_became_peered":"2023-11-28T20:35:36.312319+0000","last_unstale":"2023-11-28T20:35:37.486671+0000","last_undegraded":"2023-11-28T20:35:37.486671+0000","last_fullsized":"2023-11-28T20:35:37.486671+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:01:51.001215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.486309+0000","last_change":"2023-11-28T20:35:36.314021+0000","last_active":"2023-11-28T20:35:37.486309+0000","last_peered":"2023-11-28T20:35:37.486309+0000","last_clean":"2023-11-28T20:35:37.486309+0000","last_became_active":"2023-11-28T20:35:36.313701+0000","last_became_peered":"2023-11-28T20:35:36.313701+0000","last_unstale":"2023-11-28T20:35:37.486309+0000","last_undegraded":"2023-11-28T20:35:37.486309+0000","last_fullsized":"2023-11-28T20:35:37.486309+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:19.248180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.292893+0000","last_change":"2023-11-28T20:35:36.320368+0000","last_active":"2023-11-28T20:35:37.292893+0000","last_peered":"2023-11-28T20:35:37.292893+0000","last_clean":"2023-11-28T20:35:37.292893+0000","last_became_active":"2023-11-28T20:35:36.320062+0000","last_became_peered":"2023-11-28T20:35:36.320062+0000","last_unstale":"2023-11-28T20:35:37.292893+0000","last_undegraded":"2023-11-28T20:35:37.292893+0000","last_fullsized":"2023-11-28T20:35:37.292893+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:41.294831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.491832+0000","last_change":"2023-11-28T20:35:36.337417+0000","last_active":"2023-11-28T20:35:37.491832+0000","last_peered":"2023-11-28T20:35:37.491832+0000","last_clean":"2023-11-28T20:35:37.491832+0000","last_became_active":"2023-11-28T20:35:36.337078+0000","last_became_peered":"2023-11-28T20:35:36.337078+0000","last_unstale":"2023-11-28T20:35:37.491832+0000","last_undegraded":"2023-11-28T20:35:37.491832+0000","last_fullsized":"2023-11-28T20:35:37.491832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:04:05.274243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294164+0000","last_change":"2023-11-28T20:35:36.327392+0000","last_active":"2023-11-28T20:35:37.294164+0000","last_peered":"2023-11-28T20:35:37.294164+0000","last_clean":"2023-11-28T20:35:37.294164+0000","last_became_active":"2023-11-28T20:35:36.327056+0000","last_became_peered":"2023-11-28T20:35:36.327056+0000","last_unstale":"2023-11-28T20:35:37.294164+0000","last_undegraded":"2023-11-28T20:35:37.294164+0000","last_fullsized":"2023-11-28T20:35:37.294164+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:58:57.536138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853735+0000","last_change":"2023-11-28T20:35:36.327633+0000","last_active":"2023-11-28T20:35:37.853735+0000","last_peered":"2023-11-28T20:35:37.853735+0000","last_clean":"2023-11-28T20:35:37.853735+0000","last_became_active":"2023-11-28T20:35:36.322700+0000","last_became_peered":"2023-11-28T20:35:36.322700+0000","last_unstale":"2023-11-28T20:35:37.853735+0000","last_undegraded":"2023-11-28T20:35:37.853735+0000","last_fullsized":"2023-11-28T20:35:37.853735+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:21.423595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.492712+0000","last_change":"2023-11-28T20:35:36.326819+0000","last_active":"2023-11-28T20:35:37.492712+0000","last_peered":"2023-11-28T20:35:37.492712+0000","last_clean":"2023-11-28T20:35:37.492712+0000","last_became_active":"2023-11-28T20:35:36.326135+0000","last_became_peered":"2023-11-28T20:35:36.326135+0000","last_unstale":"2023-11-28T20:35:37.492712+0000","last_undegraded":"2023-11-28T20:35:37.492712+0000","last_fullsized":"2023-11-28T20:35:37.492712+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:29:00.040689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.492655+0000","last_change":"2023-11-28T20:35:36.333912+0000","last_active":"2023-11-28T20:35:37.492655+0000","last_peered":"2023-11-28T20:35:37.492655+0000","last_clean":"2023-11-28T20:35:37.492655+0000","last_became_active":"2023-11-28T20:35:36.333216+0000","last_became_peered":"2023-11-28T20:35:36.333216+0000","last_unstale":"2023-11-28T20:35:37.492655+0000","last_undegraded":"2023-11-28T20:35:37.492655+0000","last_fullsized":"2023-11-28T20:35:37.492655+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:32.946714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853288+0000","last_change":"2023-11-28T20:35:36.317460+0000","last_active":"2023-11-28T20:35:37.853288+0000","last_peered":"2023-11-28T20:35:37.853288+0000","last_clean":"2023-11-28T20:35:37.853288+0000","last_became_active":"2023-11-28T20:35:36.316754+0000","last_became_peered":"2023-11-28T20:35:36.316754+0000","last_unstale":"2023-11-28T20:35:37.853288+0000","last_undegraded":"2023-11-28T20:35:37.853288+0000","last_fullsized":"2023-11-28T20:35:37.853288+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:58.731453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.492035+0000","last_change":"2023-11-28T20:35:36.324532+0000","last_active":"2023-11-28T20:35:37.492035+0000","last_peered":"2023-11-28T20:35:37.492035+0000","last_clean":"2023-11-28T20:35:37.492035+0000","last_became_active":"2023-11-28T20:35:36.324225+0000","last_became_peered":"2023-11-28T20:35:36.324225+0000","last_unstale":"2023-11-28T20:35:37.492035+0000","last_undegraded":"2023-11-28T20:35:37.492035+0000","last_fullsized":"2023-11-28T20:35:37.492035+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:48.957197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294874+0000","last_change":"2023-11-28T20:35:36.332038+0000","last_active":"2023-11-28T20:35:37.294874+0000","last_peered":"2023-11-28T20:35:37.294874+0000","last_clean":"2023-11-28T20:35:37.294874+0000","last_became_active":"2023-11-28T20:35:36.326407+0000","last_became_peered":"2023-11-28T20:35:36.326407+0000","last_unstale":"2023-11-28T20:35:37.294874+0000","last_undegraded":"2023-11-28T20:35:37.294874+0000","last_fullsized":"2023-11-28T20:35:37.294874+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:53:50.031652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.492039+0000","last_change":"2023-11-28T20:35:36.328320+0000","last_active":"2023-11-28T20:35:37.492039+0000","last_peered":"2023-11-28T20:35:37.492039+0000","last_clean":"2023-11-28T20:35:37.492039+0000","last_became_active":"2023-11-28T20:35:36.328031+0000","last_became_peered":"2023-11-28T20:35:36.328031+0000","last_unstale":"2023-11-28T20:35:37.492039+0000","last_undegraded":"2023-11-28T20:35:37.492039+0000","last_fullsized":"2023-11-28T20:35:37.492039+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:10:13.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853163+0000","last_change":"2023-11-28T20:35:36.334119+0000","last_active":"2023-11-28T20:35:37.853163+0000","last_peered":"2023-11-28T20:35:37.853163+0000","last_clean":"2023-11-28T20:35:37.853163+0000","last_became_active":"2023-11-28T20:35:36.333852+0000","last_became_peered":"2023-11-28T20:35:36.333852+0000","last_unstale":"2023-11-28T20:35:37.853163+0000","last_undegraded":"2023-11-28T20:35:37.853163+0000","last_fullsized":"2023-11-28T20:35:37.853163+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:38.870814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.854402+0000","last_change":"2023-11-28T20:35:36.331790+0000","last_active":"2023-11-28T20:35:37.854402+0000","last_peered":"2023-11-28T20:35:37.854402+0000","last_clean":"2023-11-28T20:35:37.854402+0000","last_became_active":"2023-11-28T20:35:36.331427+0000","last_became_peered":"2023-11-28T20:35:36.331427+0000","last_unstale":"2023-11-28T20:35:37.854402+0000","last_undegraded":"2023-11-28T20:35:37.854402+0000","last_fullsized":"2023-11-28T20:35:37.854402+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:05.518189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.856012+0000","last_change":"2023-11-28T20:35:36.330932+0000","last_active":"2023-11-28T20:35:37.856012+0000","last_peered":"2023-11-28T20:35:37.856012+0000","last_clean":"2023-11-28T20:35:37.856012+0000","last_became_active":"2023-11-28T20:35:36.329493+0000","last_became_peered":"2023-11-28T20:35:36.329493+0000","last_unstale":"2023-11-28T20:35:37.856012+0000","last_undegraded":"2023-11-28T20:35:37.856012+0000","last_fullsized":"2023-11-28T20:35:37.856012+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:15:56.311963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.486944+0000","last_change":"2023-11-28T20:35:36.316816+0000","last_active":"2023-11-28T20:35:37.486944+0000","last_peered":"2023-11-28T20:35:37.486944+0000","last_clean":"2023-11-28T20:35:37.486944+0000","last_became_active":"2023-11-28T20:35:36.316513+0000","last_became_peered":"2023-11-28T20:35:36.316513+0000","last_unstale":"2023-11-28T20:35:37.486944+0000","last_undegraded":"2023-11-28T20:35:37.486944+0000","last_fullsized":"2023-11-28T20:35:37.486944+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:09.368273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.855635+0000","last_change":"2023-11-28T20:35:36.317161+0000","last_active":"2023-11-28T20:35:37.855635+0000","last_peered":"2023-11-28T20:35:37.855635+0000","last_clean":"2023-11-28T20:35:37.855635+0000","last_became_active":"2023-11-28T20:35:36.315412+0000","last_became_peered":"2023-11-28T20:35:36.315412+0000","last_unstale":"2023-11-28T20:35:37.855635+0000","last_undegraded":"2023-11-28T20:35:37.855635+0000","last_fullsized":"2023-11-28T20:35:37.855635+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:27.447365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.492601+0000","last_change":"2023-11-28T20:35:36.330852+0000","last_active":"2023-11-28T20:35:37.492601+0000","last_peered":"2023-11-28T20:35:37.492601+0000","last_clean":"2023-11-28T20:35:37.492601+0000","last_became_active":"2023-11-28T20:35:36.330296+0000","last_became_peered":"2023-11-28T20:35:36.330296+0000","last_unstale":"2023-11-28T20:35:37.492601+0000","last_undegraded":"2023-11-28T20:35:37.492601+0000","last_fullsized":"2023-11-28T20:35:37.492601+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:56:23.270789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.295843+0000","last_change":"2023-11-28T20:35:36.320356+0000","last_active":"2023-11-28T20:35:37.295843+0000","last_peered":"2023-11-28T20:35:37.295843+0000","last_clean":"2023-11-28T20:35:37.295843+0000","last_became_active":"2023-11-28T20:35:36.319062+0000","last_became_peered":"2023-11-28T20:35:36.319062+0000","last_unstale":"2023-11-28T20:35:37.295843+0000","last_undegraded":"2023-11-28T20:35:37.295843+0000","last_fullsized":"2023-11-28T20:35:37.295843+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:43:18.090861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853285+0000","last_change":"2023-11-28T20:35:36.336718+0000","last_active":"2023-11-28T20:35:37.853285+0000","last_peered":"2023-11-28T20:35:37.853285+0000","last_clean":"2023-11-28T20:35:37.853285+0000","last_became_active":"2023-11-28T20:35:36.335938+0000","last_became_peered":"2023-11-28T20:35:36.335938+0000","last_unstale":"2023-11-28T20:35:37.853285+0000","last_undegraded":"2023-11-28T20:35:37.853285+0000","last_fullsized":"2023-11-28T20:35:37.853285+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:19.446747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.486249+0000","last_change":"2023-11-28T20:35:36.327070+0000","last_active":"2023-11-28T20:35:37.486249+0000","last_peered":"2023-11-28T20:35:37.486249+0000","last_clean":"2023-11-28T20:35:37.486249+0000","last_became_active":"2023-11-28T20:35:36.326797+0000","last_became_peered":"2023-11-28T20:35:36.326797+0000","last_unstale":"2023-11-28T20:35:37.486249+0000","last_undegraded":"2023-11-28T20:35:37.486249+0000","last_fullsized":"2023-11-28T20:35:37.486249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:37.426437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.856287+0000","last_change":"2023-11-28T20:35:36.326941+0000","last_active":"2023-11-28T20:35:37.856287+0000","last_peered":"2023-11-28T20:35:37.856287+0000","last_clean":"2023-11-28T20:35:37.856287+0000","last_became_active":"2023-11-28T20:35:36.326612+0000","last_became_peered":"2023-11-28T20:35:36.326612+0000","last_unstale":"2023-11-28T20:35:37.856287+0000","last_undegraded":"2023-11-28T20:35:37.856287+0000","last_fullsized":"2023-11-28T20:35:37.856287+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:48:46.978339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.293603+0000","last_change":"2023-11-28T20:35:36.330270+0000","last_active":"2023-11-28T20:35:37.293603+0000","last_peered":"2023-11-28T20:35:37.293603+0000","last_clean":"2023-11-28T20:35:37.293603+0000","last_became_active":"2023-11-28T20:35:36.329972+0000","last_became_peered":"2023-11-28T20:35:36.329972+0000","last_unstale":"2023-11-28T20:35:37.293603+0000","last_undegraded":"2023-11-28T20:35:37.293603+0000","last_fullsized":"2023-11-28T20:35:37.293603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:58.731453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.856027+0000","last_change":"2023-11-28T20:35:36.314026+0000","last_active":"2023-11-28T20:35:37.856027+0000","last_peered":"2023-11-28T20:35:37.856027+0000","last_clean":"2023-11-28T20:35:37.856027+0000","last_became_active":"2023-11-28T20:35:36.313510+0000","last_became_peered":"2023-11-28T20:35:36.313510+0000","last_unstale":"2023-11-28T20:35:37.856027+0000","last_undegraded":"2023-11-28T20:35:37.856027+0000","last_fullsized":"2023-11-28T20:35:37.856027+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:53:50.031652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.492799+0000","last_change":"2023-11-28T20:35:36.334171+0000","last_active":"2023-11-28T20:35:37.492799+0000","last_peered":"2023-11-28T20:35:37.492799+0000","last_clean":"2023-11-28T20:35:37.492799+0000","last_became_active":"2023-11-28T20:35:36.333887+0000","last_became_peered":"2023-11-28T20:35:36.333887+0000","last_unstale":"2023-11-28T20:35:37.492799+0000","last_undegraded":"2023-11-28T20:35:37.492799+0000","last_fullsized":"2023-11-28T20:35:37.492799+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:41:16.935279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.855723+0000","last_change":"2023-11-28T20:35:36.314860+0000","last_active":"2023-11-28T20:35:37.855723+0000","last_peered":"2023-11-28T20:35:37.855723+0000","last_clean":"2023-11-28T20:35:37.855723+0000","last_became_active":"2023-11-28T20:35:36.314265+0000","last_became_peered":"2023-11-28T20:35:36.314265+0000","last_unstale":"2023-11-28T20:35:37.855723+0000","last_undegraded":"2023-11-28T20:35:37.855723+0000","last_fullsized":"2023-11-28T20:35:37.855723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:19:08.853417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.855545+0000","last_change":"2023-11-28T20:35:36.315975+0000","last_active":"2023-11-28T20:35:37.855545+0000","last_peered":"2023-11-28T20:35:37.855545+0000","last_clean":"2023-11-28T20:35:37.855545+0000","last_became_active":"2023-11-28T20:35:36.315688+0000","last_became_peered":"2023-11-28T20:35:36.315688+0000","last_unstale":"2023-11-28T20:35:37.855545+0000","last_undegraded":"2023-11-28T20:35:37.855545+0000","last_fullsized":"2023-11-28T20:35:37.855545+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:31:52.068678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853663+0000","last_change":"2023-11-28T20:35:36.336378+0000","last_active":"2023-11-28T20:35:37.853663+0000","last_peered":"2023-11-28T20:35:37.853663+0000","last_clean":"2023-11-28T20:35:37.853663+0000","last_became_active":"2023-11-28T20:35:36.336001+0000","last_became_peered":"2023-11-28T20:35:36.336001+0000","last_unstale":"2023-11-28T20:35:37.853663+0000","last_undegraded":"2023-11-28T20:35:37.853663+0000","last_fullsized":"2023-11-28T20:35:37.853663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:45.573798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.292494+0000","last_change":"2023-11-28T20:35:36.318103+0000","last_active":"2023-11-28T20:35:37.292494+0000","last_peered":"2023-11-28T20:35:37.292494+0000","last_clean":"2023-11-28T20:35:37.292494+0000","last_became_active":"2023-11-28T20:35:36.317795+0000","last_became_peered":"2023-11-28T20:35:36.317795+0000","last_unstale":"2023-11-28T20:35:37.292494+0000","last_undegraded":"2023-11-28T20:35:37.292494+0000","last_fullsized":"2023-11-28T20:35:37.292494+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:53:50.031652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.854192+0000","last_change":"2023-11-28T20:35:36.335362+0000","last_active":"2023-11-28T20:35:37.854192+0000","last_peered":"2023-11-28T20:35:37.854192+0000","last_clean":"2023-11-28T20:35:37.854192+0000","last_became_active":"2023-11-28T20:35:36.334625+0000","last_became_peered":"2023-11-28T20:35:36.334625+0000","last_unstale":"2023-11-28T20:35:37.854192+0000","last_undegraded":"2023-11-28T20:35:37.854192+0000","last_fullsized":"2023-11-28T20:35:37.854192+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:48.957197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.347566+0000","last_change":"2023-11-28T20:35:35.304535+0000","last_active":"2023-11-28T20:35:36.347566+0000","last_peered":"2023-11-28T20:35:36.347566+0000","last_clean":"2023-11-28T20:35:36.347566+0000","last_became_active":"2023-11-28T20:35:35.304193+0000","last_became_peered":"2023-11-28T20:35:35.304193+0000","last_unstale":"2023-11-28T20:35:36.347566+0000","last_undegraded":"2023-11-28T20:35:36.347566+0000","last_fullsized":"2023-11-28T20:35:36.347566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.293529+0000","last_change":"2023-11-28T20:35:36.330623+0000","last_active":"2023-11-28T20:35:37.293529+0000","last_peered":"2023-11-28T20:35:37.293529+0000","last_clean":"2023-11-28T20:35:37.293529+0000","last_became_active":"2023-11-28T20:35:36.330308+0000","last_became_peered":"2023-11-28T20:35:36.330308+0000","last_unstale":"2023-11-28T20:35:37.293529+0000","last_undegraded":"2023-11-28T20:35:37.293529+0000","last_fullsized":"2023-11-28T20:35:37.293529+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:08:10.354829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.346694+0000","last_change":"2023-11-28T20:35:35.300035+0000","last_active":"2023-11-28T20:35:36.346694+0000","last_peered":"2023-11-28T20:35:36.346694+0000","last_clean":"2023-11-28T20:35:36.346694+0000","last_became_active":"2023-11-28T20:35:35.299808+0000","last_became_peered":"2023-11-28T20:35:35.299808+0000","last_unstale":"2023-11-28T20:35:36.346694+0000","last_undegraded":"2023-11-28T20:35:36.346694+0000","last_fullsized":"2023-11-28T20:35:36.346694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:10.536815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.296094+0000","last_change":"2023-11-28T20:35:36.320062+0000","last_active":"2023-11-28T20:35:37.296094+0000","last_peered":"2023-11-28T20:35:37.296094+0000","last_clean":"2023-11-28T20:35:37.296094+0000","last_became_active":"2023-11-28T20:35:36.319456+0000","last_became_peered":"2023-11-28T20:35:36.319456+0000","last_unstale":"2023-11-28T20:35:37.296094+0000","last_undegraded":"2023-11-28T20:35:37.296094+0000","last_fullsized":"2023-11-28T20:35:37.296094+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:08:10.354829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.349995+0000","last_change":"2023-11-28T20:35:35.303683+0000","last_active":"2023-11-28T20:35:36.349995+0000","last_peered":"2023-11-28T20:35:36.349995+0000","last_clean":"2023-11-28T20:35:36.349995+0000","last_became_active":"2023-11-28T20:35:35.303422+0000","last_became_peered":"2023-11-28T20:35:35.303422+0000","last_unstale":"2023-11-28T20:35:36.349995+0000","last_undegraded":"2023-11-28T20:35:36.349995+0000","last_fullsized":"2023-11-28T20:35:36.349995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:15:55.307531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294976+0000","last_change":"2023-11-28T20:35:36.320878+0000","last_active":"2023-11-28T20:35:37.294976+0000","last_peered":"2023-11-28T20:35:37.294976+0000","last_clean":"2023-11-28T20:35:37.294976+0000","last_became_active":"2023-11-28T20:35:36.319348+0000","last_became_peered":"2023-11-28T20:35:36.319348+0000","last_unstale":"2023-11-28T20:35:37.294976+0000","last_undegraded":"2023-11-28T20:35:37.294976+0000","last_fullsized":"2023-11-28T20:35:37.294976+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:56:23.270789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.346897+0000","last_change":"2023-11-28T20:35:35.290259+0000","last_active":"2023-11-28T20:35:36.346897+0000","last_peered":"2023-11-28T20:35:36.346897+0000","last_clean":"2023-11-28T20:35:36.346897+0000","last_became_active":"2023-11-28T20:35:35.286178+0000","last_became_peered":"2023-11-28T20:35:35.286178+0000","last_unstale":"2023-11-28T20:35:36.346897+0000","last_undegraded":"2023-11-28T20:35:36.346897+0000","last_fullsized":"2023-11-28T20:35:36.346897+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:26.442933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.852816+0000","last_change":"2023-11-28T20:35:36.317496+0000","last_active":"2023-11-28T20:35:37.852816+0000","last_peered":"2023-11-28T20:35:37.852816+0000","last_clean":"2023-11-28T20:35:37.852816+0000","last_became_active":"2023-11-28T20:35:36.317224+0000","last_became_peered":"2023-11-28T20:35:36.317224+0000","last_unstale":"2023-11-28T20:35:37.852816+0000","last_undegraded":"2023-11-28T20:35:37.852816+0000","last_fullsized":"2023-11-28T20:35:37.852816+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:41.294831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.344233+0000","last_change":"2023-11-28T20:35:35.306240+0000","last_active":"2023-11-28T20:35:36.344233+0000","last_peered":"2023-11-28T20:35:36.344233+0000","last_clean":"2023-11-28T20:35:36.344233+0000","last_became_active":"2023-11-28T20:35:35.305889+0000","last_became_peered":"2023-11-28T20:35:35.305889+0000","last_unstale":"2023-11-28T20:35:36.344233+0000","last_undegraded":"2023-11-28T20:35:36.344233+0000","last_fullsized":"2023-11-28T20:35:36.344233+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:20.419164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.492219+0000","last_change":"2023-11-28T20:35:36.325191+0000","last_active":"2023-11-28T20:35:37.492219+0000","last_peered":"2023-11-28T20:35:37.492219+0000","last_clean":"2023-11-28T20:35:37.492219+0000","last_became_active":"2023-11-28T20:35:36.324914+0000","last_became_peered":"2023-11-28T20:35:36.324914+0000","last_unstale":"2023-11-28T20:35:37.492219+0000","last_undegraded":"2023-11-28T20:35:37.492219+0000","last_fullsized":"2023-11-28T20:35:37.492219+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:05.518189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.347167+0000","last_change":"2023-11-28T20:35:35.311106+0000","last_active":"2023-11-28T20:35:36.347167+0000","last_peered":"2023-11-28T20:35:36.347167+0000","last_clean":"2023-11-28T20:35:36.347167+0000","last_became_active":"2023-11-28T20:35:35.310832+0000","last_became_peered":"2023-11-28T20:35:35.310832+0000","last_unstale":"2023-11-28T20:35:36.347167+0000","last_undegraded":"2023-11-28T20:35:36.347167+0000","last_fullsized":"2023-11-28T20:35:36.347167+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:20.419164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.296337+0000","last_change":"2023-11-28T20:35:36.323488+0000","last_active":"2023-11-28T20:35:37.296337+0000","last_peered":"2023-11-28T20:35:37.296337+0000","last_clean":"2023-11-28T20:35:37.296337+0000","last_became_active":"2023-11-28T20:35:36.323153+0000","last_became_peered":"2023-11-28T20:35:36.323153+0000","last_unstale":"2023-11-28T20:35:37.296337+0000","last_undegraded":"2023-11-28T20:35:37.296337+0000","last_fullsized":"2023-11-28T20:35:37.296337+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:58.731453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.306540+0000","last_change":"2023-11-28T20:35:35.311801+0000","last_active":"2023-11-28T20:35:36.306540+0000","last_peered":"2023-11-28T20:35:36.306540+0000","last_clean":"2023-11-28T20:35:36.306540+0000","last_became_active":"2023-11-28T20:35:35.311381+0000","last_became_peered":"2023-11-28T20:35:35.311381+0000","last_unstale":"2023-11-28T20:35:36.306540+0000","last_undegraded":"2023-11-28T20:35:36.306540+0000","last_fullsized":"2023-11-28T20:35:36.306540+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:31:51.064247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294522+0000","last_change":"2023-11-28T20:35:36.320976+0000","last_active":"2023-11-28T20:35:37.294522+0000","last_peered":"2023-11-28T20:35:37.294522+0000","last_clean":"2023-11-28T20:35:37.294522+0000","last_became_active":"2023-11-28T20:35:36.320501+0000","last_became_peered":"2023-11-28T20:35:36.320501+0000","last_unstale":"2023-11-28T20:35:37.294522+0000","last_undegraded":"2023-11-28T20:35:37.294522+0000","last_fullsized":"2023-11-28T20:35:37.294522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:04:05.274243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.308727+0000","last_change":"2023-11-28T20:35:35.305104+0000","last_active":"2023-11-28T20:35:36.308727+0000","last_peered":"2023-11-28T20:35:36.308727+0000","last_clean":"2023-11-28T20:35:36.308727+0000","last_became_active":"2023-11-28T20:35:35.304907+0000","last_became_peered":"2023-11-28T20:35:35.304907+0000","last_unstale":"2023-11-28T20:35:36.308727+0000","last_undegraded":"2023-11-28T20:35:36.308727+0000","last_fullsized":"2023-11-28T20:35:36.308727+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:31:51.064247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.487139+0000","last_change":"2023-11-28T20:35:36.325825+0000","last_active":"2023-11-28T20:35:37.487139+0000","last_peered":"2023-11-28T20:35:37.487139+0000","last_clean":"2023-11-28T20:35:37.487139+0000","last_became_active":"2023-11-28T20:35:36.323673+0000","last_became_peered":"2023-11-28T20:35:36.323673+0000","last_unstale":"2023-11-28T20:35:37.487139+0000","last_undegraded":"2023-11-28T20:35:37.487139+0000","last_fullsized":"2023-11-28T20:35:37.487139+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:25.660549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.297069+0000","last_change":"2023-11-28T20:35:35.295019+0000","last_active":"2023-11-28T20:35:36.297069+0000","last_peered":"2023-11-28T20:35:36.297069+0000","last_clean":"2023-11-28T20:35:36.297069+0000","last_became_active":"2023-11-28T20:35:35.294753+0000","last_became_peered":"2023-11-28T20:35:35.294753+0000","last_unstale":"2023-11-28T20:35:36.297069+0000","last_undegraded":"2023-11-28T20:35:36.297069+0000","last_fullsized":"2023-11-28T20:35:36.297069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:00.999710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.854550+0000","last_change":"2023-11-28T20:35:36.331968+0000","last_active":"2023-11-28T20:35:37.854550+0000","last_peered":"2023-11-28T20:35:37.854550+0000","last_clean":"2023-11-28T20:35:37.854550+0000","last_became_active":"2023-11-28T20:35:36.331310+0000","last_became_peered":"2023-11-28T20:35:36.331310+0000","last_unstale":"2023-11-28T20:35:37.854550+0000","last_undegraded":"2023-11-28T20:35:37.854550+0000","last_fullsized":"2023-11-28T20:35:37.854550+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:29.199438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.318738+0000","last_change":"2023-11-28T20:35:35.308107+0000","last_active":"2023-11-28T20:35:36.318738+0000","last_peered":"2023-11-28T20:35:36.318738+0000","last_clean":"2023-11-28T20:35:36.318738+0000","last_became_active":"2023-11-28T20:35:35.307828+0000","last_became_peered":"2023-11-28T20:35:35.307828+0000","last_unstale":"2023-11-28T20:35:36.318738+0000","last_undegraded":"2023-11-28T20:35:36.318738+0000","last_fullsized":"2023-11-28T20:35:36.318738+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:58:56.531707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853704+0000","last_change":"2023-11-28T20:35:36.328344+0000","last_active":"2023-11-28T20:35:37.853704+0000","last_peered":"2023-11-28T20:35:37.853704+0000","last_clean":"2023-11-28T20:35:37.853704+0000","last_became_active":"2023-11-28T20:35:36.327733+0000","last_became_peered":"2023-11-28T20:35:36.327733+0000","last_unstale":"2023-11-28T20:35:37.853704+0000","last_undegraded":"2023-11-28T20:35:37.853704+0000","last_fullsized":"2023-11-28T20:35:37.853704+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:02.004142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.345111+0000","last_change":"2023-11-28T20:35:35.307353+0000","last_active":"2023-11-28T20:35:36.345111+0000","last_peered":"2023-11-28T20:35:36.345111+0000","last_clean":"2023-11-28T20:35:36.345111+0000","last_became_active":"2023-11-28T20:35:35.307054+0000","last_became_peered":"2023-11-28T20:35:35.307054+0000","last_unstale":"2023-11-28T20:35:36.345111+0000","last_undegraded":"2023-11-28T20:35:36.345111+0000","last_fullsized":"2023-11-28T20:35:36.345111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:44.569366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.296758+0000","last_change":"2023-11-28T20:35:36.320521+0000","last_active":"2023-11-28T20:35:37.296758+0000","last_peered":"2023-11-28T20:35:37.296758+0000","last_clean":"2023-11-28T20:35:37.296758+0000","last_became_active":"2023-11-28T20:35:36.320236+0000","last_became_peered":"2023-11-28T20:35:36.320236+0000","last_unstale":"2023-11-28T20:35:37.296758+0000","last_undegraded":"2023-11-28T20:35:37.296758+0000","last_fullsized":"2023-11-28T20:35:37.296758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:02.004142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.351689+0000","last_change":"2023-11-28T20:35:35.298444+0000","last_active":"2023-11-28T20:35:36.351689+0000","last_peered":"2023-11-28T20:35:36.351689+0000","last_clean":"2023-11-28T20:35:36.351689+0000","last_became_active":"2023-11-28T20:35:35.297824+0000","last_became_peered":"2023-11-28T20:35:35.297824+0000","last_unstale":"2023-11-28T20:35:36.351689+0000","last_undegraded":"2023-11-28T20:35:36.351689+0000","last_fullsized":"2023-11-28T20:35:36.351689+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:37.866382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.852728+0000","last_change":"2023-11-28T20:35:36.325218+0000","last_active":"2023-11-28T20:35:37.852728+0000","last_peered":"2023-11-28T20:35:37.852728+0000","last_clean":"2023-11-28T20:35:37.852728+0000","last_became_active":"2023-11-28T20:35:36.324902+0000","last_became_peered":"2023-11-28T20:35:36.324902+0000","last_unstale":"2023-11-28T20:35:37.852728+0000","last_undegraded":"2023-11-28T20:35:37.852728+0000","last_fullsized":"2023-11-28T20:35:37.852728+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:48:46.978339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.343641+0000","last_change":"2023-11-28T20:35:35.290127+0000","last_active":"2023-11-28T20:35:36.343641+0000","last_peered":"2023-11-28T20:35:36.343641+0000","last_clean":"2023-11-28T20:35:36.343641+0000","last_became_active":"2023-11-28T20:35:35.289788+0000","last_became_peered":"2023-11-28T20:35:35.289788+0000","last_unstale":"2023-11-28T20:35:36.343641+0000","last_undegraded":"2023-11-28T20:35:36.343641+0000","last_fullsized":"2023-11-28T20:35:36.343641+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:18.442315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294373+0000","last_change":"2023-11-28T20:35:36.317042+0000","last_active":"2023-11-28T20:35:37.294373+0000","last_peered":"2023-11-28T20:35:37.294373+0000","last_clean":"2023-11-28T20:35:37.294373+0000","last_became_active":"2023-11-28T20:35:36.316772+0000","last_became_peered":"2023-11-28T20:35:36.316772+0000","last_unstale":"2023-11-28T20:35:37.294373+0000","last_undegraded":"2023-11-28T20:35:37.294373+0000","last_fullsized":"2023-11-28T20:35:37.294373+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:45.573798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.306538+0000","last_change":"2023-11-28T20:35:35.310764+0000","last_active":"2023-11-28T20:35:36.306538+0000","last_peered":"2023-11-28T20:35:36.306538+0000","last_clean":"2023-11-28T20:35:36.306538+0000","last_became_active":"2023-11-28T20:35:35.310478+0000","last_became_peered":"2023-11-28T20:35:35.310478+0000","last_unstale":"2023-11-28T20:35:36.306538+0000","last_undegraded":"2023-11-28T20:35:36.306538+0000","last_fullsized":"2023-11-28T20:35:36.306538+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:10.536815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853403+0000","last_change":"2023-11-28T20:35:36.337291+0000","last_active":"2023-11-28T20:35:37.853403+0000","last_peered":"2023-11-28T20:35:37.853403+0000","last_clean":"2023-11-28T20:35:37.853403+0000","last_became_active":"2023-11-28T20:35:36.337045+0000","last_became_peered":"2023-11-28T20:35:36.337045+0000","last_unstale":"2023-11-28T20:35:37.853403+0000","last_undegraded":"2023-11-28T20:35:37.853403+0000","last_fullsized":"2023-11-28T20:35:37.853403+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:58:57.536138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.296317+0000","last_change":"2023-11-28T20:35:35.304319+0000","last_active":"2023-11-28T20:35:36.296317+0000","last_peered":"2023-11-28T20:35:36.296317+0000","last_clean":"2023-11-28T20:35:36.296317+0000","last_became_active":"2023-11-28T20:35:35.302594+0000","last_became_peered":"2023-11-28T20:35:35.302594+0000","last_unstale":"2023-11-28T20:35:36.296317+0000","last_undegraded":"2023-11-28T20:35:36.296317+0000","last_fullsized":"2023-11-28T20:35:36.296317+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:45:24.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294533+0000","last_change":"2023-11-28T20:35:36.322500+0000","last_active":"2023-11-28T20:35:37.294533+0000","last_peered":"2023-11-28T20:35:37.294533+0000","last_clean":"2023-11-28T20:35:37.294533+0000","last_became_active":"2023-11-28T20:35:36.322187+0000","last_became_peered":"2023-11-28T20:35:36.322187+0000","last_unstale":"2023-11-28T20:35:37.294533+0000","last_undegraded":"2023-11-28T20:35:37.294533+0000","last_fullsized":"2023-11-28T20:35:37.294533+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:48.957197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.350166+0000","last_change":"2023-11-28T20:35:35.310387+0000","last_active":"2023-11-28T20:35:36.350166+0000","last_peered":"2023-11-28T20:35:36.350166+0000","last_clean":"2023-11-28T20:35:36.350166+0000","last_became_active":"2023-11-28T20:35:35.310118+0000","last_became_peered":"2023-11-28T20:35:35.310118+0000","last_unstale":"2023-11-28T20:35:36.350166+0000","last_undegraded":"2023-11-28T20:35:36.350166+0000","last_fullsized":"2023-11-28T20:35:36.350166+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:26.442933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294714+0000","last_change":"2023-11-28T20:35:36.316023+0000","last_active":"2023-11-28T20:35:37.294714+0000","last_peered":"2023-11-28T20:35:37.294714+0000","last_clean":"2023-11-28T20:35:37.294714+0000","last_became_active":"2023-11-28T20:35:36.315041+0000","last_became_peered":"2023-11-28T20:35:36.315041+0000","last_unstale":"2023-11-28T20:35:37.294714+0000","last_undegraded":"2023-11-28T20:35:37.294714+0000","last_fullsized":"2023-11-28T20:35:37.294714+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:10:13.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.347340+0000","last_change":"2023-11-28T20:35:35.308420+0000","last_active":"2023-11-28T20:35:36.347340+0000","last_peered":"2023-11-28T20:35:36.347340+0000","last_clean":"2023-11-28T20:35:36.347340+0000","last_became_active":"2023-11-28T20:35:35.305473+0000","last_became_peered":"2023-11-28T20:35:35.305473+0000","last_unstale":"2023-11-28T20:35:36.347340+0000","last_undegraded":"2023-11-28T20:35:36.347340+0000","last_fullsized":"2023-11-28T20:35:36.347340+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:37.866382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.492793+0000","last_change":"2023-11-28T20:35:36.331832+0000","last_active":"2023-11-28T20:35:37.492793+0000","last_peered":"2023-11-28T20:35:37.492793+0000","last_clean":"2023-11-28T20:35:37.492793+0000","last_became_active":"2023-11-28T20:35:36.331421+0000","last_became_peered":"2023-11-28T20:35:36.331421+0000","last_unstale":"2023-11-28T20:35:37.492793+0000","last_undegraded":"2023-11-28T20:35:37.492793+0000","last_fullsized":"2023-11-28T20:35:37.492793+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:29:25.437611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.347157+0000","last_change":"2023-11-28T20:35:35.311195+0000","last_active":"2023-11-28T20:35:36.347157+0000","last_peered":"2023-11-28T20:35:36.347157+0000","last_clean":"2023-11-28T20:35:36.347157+0000","last_became_active":"2023-11-28T20:35:35.310875+0000","last_became_peered":"2023-11-28T20:35:35.310875+0000","last_unstale":"2023-11-28T20:35:36.347157+0000","last_undegraded":"2023-11-28T20:35:36.347157+0000","last_fullsized":"2023-11-28T20:35:36.347157+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:15:55.307531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.295739+0000","last_change":"2023-11-28T20:35:36.315733+0000","last_active":"2023-11-28T20:35:37.295739+0000","last_peered":"2023-11-28T20:35:37.295739+0000","last_clean":"2023-11-28T20:35:37.295739+0000","last_became_active":"2023-11-28T20:35:36.315405+0000","last_became_peered":"2023-11-28T20:35:36.315405+0000","last_unstale":"2023-11-28T20:35:37.295739+0000","last_undegraded":"2023-11-28T20:35:37.295739+0000","last_fullsized":"2023-11-28T20:35:37.295739+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:41.294831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.349691+0000","last_change":"2023-11-28T20:35:35.298942+0000","last_active":"2023-11-28T20:35:36.349691+0000","last_peered":"2023-11-28T20:35:36.349691+0000","last_clean":"2023-11-28T20:35:36.349691+0000","last_became_active":"2023-11-28T20:35:35.298685+0000","last_became_peered":"2023-11-28T20:35:35.298685+0000","last_unstale":"2023-11-28T20:35:36.349691+0000","last_undegraded":"2023-11-28T20:35:36.349691+0000","last_fullsized":"2023-11-28T20:35:36.349691+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:25.676223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.487059+0000","last_change":"2023-11-28T20:35:36.316087+0000","last_active":"2023-11-28T20:35:37.487059+0000","last_peered":"2023-11-28T20:35:37.487059+0000","last_clean":"2023-11-28T20:35:37.487059+0000","last_became_active":"2023-11-28T20:35:36.315779+0000","last_became_peered":"2023-11-28T20:35:36.315779+0000","last_unstale":"2023-11-28T20:35:37.487059+0000","last_undegraded":"2023-11-28T20:35:37.487059+0000","last_fullsized":"2023-11-28T20:35:37.487059+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:30.266536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.305162+0000","last_change":"2023-11-28T20:35:35.303747+0000","last_active":"2023-11-28T20:35:36.305162+0000","last_peered":"2023-11-28T20:35:36.305162+0000","last_clean":"2023-11-28T20:35:36.305162+0000","last_became_active":"2023-11-28T20:35:35.303441+0000","last_became_peered":"2023-11-28T20:35:35.303441+0000","last_unstale":"2023-11-28T20:35:36.305162+0000","last_undegraded":"2023-11-28T20:35:36.305162+0000","last_fullsized":"2023-11-28T20:35:36.305162+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:08.363841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853713+0000","last_change":"2023-11-28T20:35:36.336695+0000","last_active":"2023-11-28T20:35:37.853713+0000","last_peered":"2023-11-28T20:35:37.853713+0000","last_clean":"2023-11-28T20:35:37.853713+0000","last_became_active":"2023-11-28T20:35:36.336422+0000","last_became_peered":"2023-11-28T20:35:36.336422+0000","last_unstale":"2023-11-28T20:35:37.853713+0000","last_undegraded":"2023-11-28T20:35:37.853713+0000","last_fullsized":"2023-11-28T20:35:37.853713+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:04:05.274243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.343669+0000","last_change":"2023-11-28T20:35:35.290919+0000","last_active":"2023-11-28T20:35:36.343669+0000","last_peered":"2023-11-28T20:35:36.343669+0000","last_clean":"2023-11-28T20:35:36.343669+0000","last_became_active":"2023-11-28T20:35:35.290407+0000","last_became_peered":"2023-11-28T20:35:35.290407+0000","last_unstale":"2023-11-28T20:35:36.343669+0000","last_undegraded":"2023-11-28T20:35:36.343669+0000","last_fullsized":"2023-11-28T20:35:36.343669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:10.367707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.852948+0000","last_change":"2023-11-28T20:35:36.324195+0000","last_active":"2023-11-28T20:35:37.852948+0000","last_peered":"2023-11-28T20:35:37.852948+0000","last_clean":"2023-11-28T20:35:37.852948+0000","last_became_active":"2023-11-28T20:35:36.323925+0000","last_became_peered":"2023-11-28T20:35:36.323925+0000","last_unstale":"2023-11-28T20:35:37.852948+0000","last_undegraded":"2023-11-28T20:35:37.852948+0000","last_fullsized":"2023-11-28T20:35:37.852948+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:43:18.090861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.292033+0000","last_change":"2023-11-28T20:35:35.299354+0000","last_active":"2023-11-28T20:35:36.292033+0000","last_peered":"2023-11-28T20:35:36.292033+0000","last_clean":"2023-11-28T20:35:36.292033+0000","last_became_active":"2023-11-28T20:35:35.299056+0000","last_became_peered":"2023-11-28T20:35:35.299056+0000","last_unstale":"2023-11-28T20:35:36.292033+0000","last_undegraded":"2023-11-28T20:35:36.292033+0000","last_fullsized":"2023-11-28T20:35:36.292033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:23:51.697749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.492256+0000","last_change":"2023-11-28T20:35:36.326774+0000","last_active":"2023-11-28T20:35:37.492256+0000","last_peered":"2023-11-28T20:35:37.492256+0000","last_clean":"2023-11-28T20:35:37.492256+0000","last_became_active":"2023-11-28T20:35:36.326465+0000","last_became_peered":"2023-11-28T20:35:36.326465+0000","last_unstale":"2023-11-28T20:35:37.492256+0000","last_undegraded":"2023-11-28T20:35:37.492256+0000","last_fullsized":"2023-11-28T20:35:37.492256+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:29.199438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.314256+0000","last_change":"2023-11-28T20:35:35.306856+0000","last_active":"2023-11-28T20:35:36.314256+0000","last_peered":"2023-11-28T20:35:36.314256+0000","last_clean":"2023-11-28T20:35:36.314256+0000","last_became_active":"2023-11-28T20:35:35.306562+0000","last_became_peered":"2023-11-28T20:35:35.306562+0000","last_unstale":"2023-11-28T20:35:36.314256+0000","last_undegraded":"2023-11-28T20:35:36.314256+0000","last_fullsized":"2023-11-28T20:35:36.314256+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:08:09.350397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294977+0000","last_change":"2023-11-28T20:35:36.320789+0000","last_active":"2023-11-28T20:35:37.294977+0000","last_peered":"2023-11-28T20:35:37.294977+0000","last_clean":"2023-11-28T20:35:37.294977+0000","last_became_active":"2023-11-28T20:35:36.320079+0000","last_became_peered":"2023-11-28T20:35:36.320079+0000","last_unstale":"2023-11-28T20:35:37.294977+0000","last_undegraded":"2023-11-28T20:35:37.294977+0000","last_fullsized":"2023-11-28T20:35:37.294977+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:32.946714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.292110+0000","last_change":"2023-11-28T20:35:35.291161+0000","last_active":"2023-11-28T20:35:36.292110+0000","last_peered":"2023-11-28T20:35:36.292110+0000","last_clean":"2023-11-28T20:35:36.292110+0000","last_became_active":"2023-11-28T20:35:35.290905+0000","last_became_peered":"2023-11-28T20:35:35.290905+0000","last_unstale":"2023-11-28T20:35:36.292110+0000","last_undegraded":"2023-11-28T20:35:36.292110+0000","last_fullsized":"2023-11-28T20:35:36.292110+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:40.958168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.293249+0000","last_change":"2023-11-28T20:35:36.323706+0000","last_active":"2023-11-28T20:35:37.293249+0000","last_peered":"2023-11-28T20:35:37.293249+0000","last_clean":"2023-11-28T20:35:37.293249+0000","last_became_active":"2023-11-28T20:35:36.323417+0000","last_became_peered":"2023-11-28T20:35:36.323417+0000","last_unstale":"2023-11-28T20:35:37.293249+0000","last_undegraded":"2023-11-28T20:35:37.293249+0000","last_fullsized":"2023-11-28T20:35:37.293249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:43:18.090861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.345128+0000","last_change":"2023-11-28T20:35:35.308961+0000","last_active":"2023-11-28T20:35:36.345128+0000","last_peered":"2023-11-28T20:35:36.345128+0000","last_clean":"2023-11-28T20:35:36.345128+0000","last_became_active":"2023-11-28T20:35:35.308687+0000","last_became_peered":"2023-11-28T20:35:35.308687+0000","last_unstale":"2023-11-28T20:35:36.345128+0000","last_undegraded":"2023-11-28T20:35:36.345128+0000","last_fullsized":"2023-11-28T20:35:36.345128+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:18.442315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.854718+0000","last_change":"2023-11-28T20:35:36.332053+0000","last_active":"2023-11-28T20:35:37.854718+0000","last_peered":"2023-11-28T20:35:37.854718+0000","last_clean":"2023-11-28T20:35:37.854718+0000","last_became_active":"2023-11-28T20:35:36.331709+0000","last_became_peered":"2023-11-28T20:35:36.331709+0000","last_unstale":"2023-11-28T20:35:37.854718+0000","last_undegraded":"2023-11-28T20:35:37.854718+0000","last_fullsized":"2023-11-28T20:35:37.854718+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:56:23.270789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.487039+0000","last_change":"2023-11-28T20:35:35.311276+0000","last_active":"2023-11-28T20:35:37.487039+0000","last_peered":"2023-11-28T20:35:37.487039+0000","last_clean":"2023-11-28T20:35:37.487039+0000","last_became_active":"2023-11-28T20:35:35.310955+0000","last_became_peered":"2023-11-28T20:35:35.310955+0000","last_unstale":"2023-11-28T20:35:37.487039+0000","last_undegraded":"2023-11-28T20:35:37.487039+0000","last_fullsized":"2023-11-28T20:35:37.487039+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:14.497404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.294884+0000","last_change":"2023-11-28T20:35:36.319070+0000","last_active":"2023-11-28T20:35:37.294884+0000","last_peered":"2023-11-28T20:35:37.294884+0000","last_clean":"2023-11-28T20:35:37.294884+0000","last_became_active":"2023-11-28T20:35:36.318054+0000","last_became_peered":"2023-11-28T20:35:36.318054+0000","last_unstale":"2023-11-28T20:35:37.294884+0000","last_undegraded":"2023-11-28T20:35:37.294884+0000","last_fullsized":"2023-11-28T20:35:37.294884+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:29.199438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.309048+0000","last_change":"2023-11-28T20:35:35.295046+0000","last_active":"2023-11-28T20:35:36.309048+0000","last_peered":"2023-11-28T20:35:36.309048+0000","last_clean":"2023-11-28T20:35:36.309048+0000","last_became_active":"2023-11-28T20:35:35.294261+0000","last_became_peered":"2023-11-28T20:35:35.294261+0000","last_unstale":"2023-11-28T20:35:36.309048+0000","last_undegraded":"2023-11-28T20:35:36.309048+0000","last_fullsized":"2023-11-28T20:35:36.309048+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:19:07.848985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.853226+0000","last_change":"2023-11-28T20:35:36.330348+0000","last_active":"2023-11-28T20:35:37.853226+0000","last_peered":"2023-11-28T20:35:37.853226+0000","last_clean":"2023-11-28T20:35:37.853226+0000","last_became_active":"2023-11-28T20:35:36.330058+0000","last_became_peered":"2023-11-28T20:35:36.330058+0000","last_unstale":"2023-11-28T20:35:37.853226+0000","last_undegraded":"2023-11-28T20:35:37.853226+0000","last_fullsized":"2023-11-28T20:35:37.853226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:08:10.354829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.309417+0000","last_change":"2023-11-28T20:35:35.304486+0000","last_active":"2023-11-28T20:35:36.309417+0000","last_peered":"2023-11-28T20:35:36.309417+0000","last_clean":"2023-11-28T20:35:36.309417+0000","last_became_active":"2023-11-28T20:35:35.304168+0000","last_became_peered":"2023-11-28T20:35:35.304168+0000","last_unstale":"2023-11-28T20:35:36.309417+0000","last_undegraded":"2023-11-28T20:35:36.309417+0000","last_fullsized":"2023-11-28T20:35:36.309417+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:53:49.027220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.292574+0000","last_change":"2023-11-28T20:35:36.325510+0000","last_active":"2023-11-28T20:35:37.292574+0000","last_peered":"2023-11-28T20:35:37.292574+0000","last_clean":"2023-11-28T20:35:37.292574+0000","last_became_active":"2023-11-28T20:35:36.325121+0000","last_became_peered":"2023-11-28T20:35:36.325121+0000","last_unstale":"2023-11-28T20:35:37.292574+0000","last_undegraded":"2023-11-28T20:35:37.292574+0000","last_fullsized":"2023-11-28T20:35:37.292574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:48:46.978339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.296299+0000","last_change":"2023-11-28T20:35:35.300968+0000","last_active":"2023-11-28T20:35:36.296299+0000","last_peered":"2023-11-28T20:35:36.296299+0000","last_clean":"2023-11-28T20:35:36.296299+0000","last_became_active":"2023-11-28T20:35:35.300389+0000","last_became_peered":"2023-11-28T20:35:35.300389+0000","last_unstale":"2023-11-28T20:35:36.296299+0000","last_undegraded":"2023-11-28T20:35:36.296299+0000","last_fullsized":"2023-11-28T20:35:36.296299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:27:29.262104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.295264+0000","last_change":"2023-11-28T20:35:36.318186+0000","last_active":"2023-11-28T20:35:37.295264+0000","last_peered":"2023-11-28T20:35:37.295264+0000","last_clean":"2023-11-28T20:35:37.295264+0000","last_became_active":"2023-11-28T20:35:36.317914+0000","last_became_peered":"2023-11-28T20:35:36.317914+0000","last_unstale":"2023-11-28T20:35:37.295264+0000","last_undegraded":"2023-11-28T20:35:37.295264+0000","last_fullsized":"2023-11-28T20:35:37.295264+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:48:46.978339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.310249+0000","last_change":"2023-11-28T20:35:35.289145+0000","last_active":"2023-11-28T20:35:36.310249+0000","last_peered":"2023-11-28T20:35:36.310249+0000","last_clean":"2023-11-28T20:35:36.310249+0000","last_became_active":"2023-11-28T20:35:35.288419+0000","last_became_peered":"2023-11-28T20:35:35.288419+0000","last_unstale":"2023-11-28T20:35:36.310249+0000","last_undegraded":"2023-11-28T20:35:36.310249+0000","last_fullsized":"2023-11-28T20:35:36.310249+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:19:07.848985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.854266+0000","last_change":"2023-11-28T20:35:36.323996+0000","last_active":"2023-11-28T20:35:37.854266+0000","last_peered":"2023-11-28T20:35:37.854266+0000","last_clean":"2023-11-28T20:35:37.854266+0000","last_became_active":"2023-11-28T20:35:36.323717+0000","last_became_peered":"2023-11-28T20:35:36.323717+0000","last_unstale":"2023-11-28T20:35:37.854266+0000","last_undegraded":"2023-11-28T20:35:37.854266+0000","last_fullsized":"2023-11-28T20:35:37.854266+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:10:13.005262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.295111+0000","last_change":"2023-11-28T20:35:35.293571+0000","last_active":"2023-11-28T20:35:36.295111+0000","last_peered":"2023-11-28T20:35:36.295111+0000","last_clean":"2023-11-28T20:35:36.295111+0000","last_became_active":"2023-11-28T20:35:35.293294+0000","last_became_peered":"2023-11-28T20:35:35.293294+0000","last_unstale":"2023-11-28T20:35:36.295111+0000","last_undegraded":"2023-11-28T20:35:36.295111+0000","last_fullsized":"2023-11-28T20:35:36.295111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:40.290399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-11-28T20:35:37.493205+0000","last_change":"2023-11-28T20:35:36.336912+0000","last_active":"2023-11-28T20:35:37.493205+0000","last_peered":"2023-11-28T20:35:37.493205+0000","last_clean":"2023-11-28T20:35:37.493205+0000","last_became_active":"2023-11-28T20:35:36.336644+0000","last_became_peered":"2023-11-28T20:35:36.336644+0000","last_unstale":"2023-11-28T20:35:37.493205+0000","last_undegraded":"2023-11-28T20:35:37.493205+0000","last_fullsized":"2023-11-28T20:35:37.493205+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:35.259596+0000","last_clean_scrub_stamp":"2023-11-28T20:35:35.259596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:55:15.782288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.312780+0000","last_change":"2023-11-28T20:35:35.305966+0000","last_active":"2023-11-28T20:35:36.312780+0000","last_peered":"2023-11-28T20:35:36.312780+0000","last_clean":"2023-11-28T20:35:36.312780+0000","last_became_active":"2023-11-28T20:35:35.305715+0000","last_became_peered":"2023-11-28T20:35:35.305715+0000","last_unstale":"2023-11-28T20:35:36.312780+0000","last_undegraded":"2023-11-28T20:35:36.312780+0000","last_fullsized":"2023-11-28T20:35:36.312780+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:35:34.255164+0000","last_clean_scrub_stamp":"2023-11-28T20:35:34.255164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:15:55.307531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":180,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-11-28T20:35:36.312533+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:35:36.312533+0000","last_peered":"2023-11-28T20:35:36.312533+0000","last_clean":"2023-11-28T20:35:36.312533+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:35:36.312533+0000","last_undegraded":"2023-11-28T20:35:36.312533+0000","last_fullsized":"2023-11-28T20:35:36.312533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1060864,"data_stored":785205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.323}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.92900000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.26},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.997}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.5259999999999998,"5min":2.5259999999999998,"15min":2.5259999999999998},"last":1.373}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.675},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.514}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.472},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"last":1.8260000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.619,"15min":1.619},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.7909999999999999,"5min":2.7909999999999999,"15min":2.7909999999999999},"last":1.7789999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":193737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":5.0430000000000001,"5min":5.0430000000000001,"15min":5.0430000000000001},"last":1.0249999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":1.486}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":1.2390000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":4.7649999999999997,"5min":4.7649999999999997,"15min":4.7649999999999997},"last":1.2769999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":3.1989999999999998,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":1.194}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":3.6429999999999998,"5min":3.6429999999999998,"15min":3.6429999999999998},"last":1.337}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.589},{"interface":"front","average":{"1min":1.917,"5min":1.917,"15min":1.917},"min":{"1min":1.018,"5min":1.018,"15min":1.018},"max":{"1min":4.9790000000000001,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.4239999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":193267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.302},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.3959999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.5}]},{"osd":2,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":0.96399999999999997}]},{"osd":3,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":1.639}]},{"osd":6,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.9319999999999999,"5min":2.9319999999999999,"15min":2.9319999999999999},"last":1.075},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":1.448}]},{"osd":7,"last update":"Tue Nov 28 20:34:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.6639999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":193759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.96299999999999997}]},{"osd":1,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":1.248}]},{"osd":2,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":1.0389999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.294}]},{"osd":5,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.6180000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.546}]},{"osd":7,"last update":"Tue Nov 28 20:34:47 2023","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":2.54,"15min":2.54},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":20.568999999999999,"5min":20.568999999999999,"15min":20.568999999999999},"last":1.72},{"interface":"front","average":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":20.677,"5min":20.677,"15min":20.677},"last":1.5980000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31644,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340196,"statfs":{"total":96636764160,"available":96604360704,"internally_reserved":0,"allocated":466944,"data_stored":194801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.90000000000000002}]},{"osd":1,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.7090000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.4950000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":5.6509999999999998,"5min":5.6509999999999998,"15min":5.6509999999999998},"last":1.107},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.1599999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.212}]},{"osd":5,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.514,"15min":1.514},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.7610000000000001,"5min":2.7610000000000001,"15min":2.7610000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.93200000000000005}]},{"osd":6,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.98,"5min":2.98,"15min":2.98},"last":1.276},{"interface":"front","average":{"1min":1.585,"5min":1.585,"15min":1.585},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.5289999999999999,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.2350000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:34:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.1859999999999999,"5min":3.1859999999999999,"15min":3.1859999999999999},"last":1.3700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31632,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":454656,"data_stored":193225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.173}]},{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.85199999999999998}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.879},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.097}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.577}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.137},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":1.8169999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.931,"5min":2.931,"15min":2.931},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.9259999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.9649999999999999,"5min":2.9649999999999999,"15min":2.9649999999999999},"last":1.8799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":454656,"data_stored":193245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.90800000000000003}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.66400000000000003}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":3.5259999999999998,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":1.0760000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":3.637,"5min":3.637,"15min":3.637},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.103}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":1.5029999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.248},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.4660000000000002,"5min":4.4660000000000002,"15min":4.4660000000000002},"last":0.98199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339632,"statfs":{"total":96636764160,"available":96603783168,"internally_reserved":0,"allocated":1048576,"data_stored":783557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.1859999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.964,"5min":2.964,"15min":2.964},"last":0.88500000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.99299999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":1.01},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.96399999999999997}]},{"osd":5,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.165}]},{"osd":6,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.536,"5min":2.536,"15min":2.536},"last":1.256},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":1.089}]},{"osd":7,"last update":"Tue Nov 28 20:34:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":2.8650000000000002,"5min":2.8650000000000002,"15min":2.8650000000000002},"last":1.286},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":1.123,"5min":1.123,"15min":1.123},"max":{"1min":2.653,"5min":2.653,"15min":2.653},"last":1.2350000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:35:46.434 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T20:35:46.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T20:35:46.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.704+0000 7fb5e5fd4640 1 Processor -- start 2023-11-28T20:35:46.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.704+0000 7fb5e5fd4640 1 -- start start 2023-11-28T20:35:46.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.706+0000 7fb5e5fd4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5e015d210 0x7fb5e00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.706+0000 7fb5e5fd4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00d11b0 con 0x7fb5e00b2620 2023-11-28T20:35:46.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.706+0000 7fb5e5fd4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00d1470 con 0x7fb5e00cddd0 2023-11-28T20:35:46.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.706+0000 7fb5e5fd4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e00b90c0 con 0x7fb5e015d210 2023-11-28T20:35:46.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.706+0000 7fb5e4fd2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5e015d210 0x7fb5e00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:46.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.706+0000 7fb5dffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb5e00b2620 0x7fb5e00b89b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51584/0 (socket says 172.21.15.45:51584) 2023-11-28T20:35:46.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.706+0000 7fb5dffff640 1 -- 172.21.15.45:0/2108343902 learned_addr learned my addr 172.21.15.45:0/2108343902 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:46.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.707+0000 7fb5e4fd2640 1 -- 172.21.15.45:0/2108343902 >> v1:172.21.15.190:6789/0 conn(0x7fb5e00cddd0 legacy=0x7fb5e00ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:46.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.707+0000 7fb5e4fd2640 1 -- 172.21.15.45:0/2108343902 >> v1:172.21.15.45:6789/0 conn(0x7fb5e00b2620 legacy=0x7fb5e00b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.707+0000 7fb5e4fd2640 1 -- 172.21.15.45:0/2108343902 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00d1750 con 0x7fb5e015d210 2023-11-28T20:35:46.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.707+0000 7fb5df7fe640 1 -- 172.21.15.45:0/2108343902 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110477226 0 0) 0x7fb5e00d11b0 con 0x7fb5e00b2620 2023-11-28T20:35:46.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.708+0000 7fb5e4fd2640 1 --2- 172.21.15.45:0/2108343902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 0x7fb5e00b61c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009d90 tx=0x7fb5d002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638f4c721b8c1d52 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:46.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.708+0000 7fb5df7fe640 1 -- 172.21.15.45:0/2108343902 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d002cd00 con 0x7fb5e015d210 2023-11-28T20:35:46.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.708+0000 7fb5df7fe640 1 -- 172.21.15.45:0/2108343902 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d002ce60 con 0x7fb5e015d210 2023-11-28T20:35:46.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.708+0000 7fb5df7fe640 1 -- 172.21.15.45:0/2108343902 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0034660 con 0x7fb5e015d210 2023-11-28T20:35:46.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.709+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2108343902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 msgr2=0x7fb5e00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.709+0000 7fb5e5fd4640 1 --2- 172.21.15.45:0/2108343902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 0x7fb5e00b61c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009d90 tx=0x7fb5d002bf50 comp rx=0 tx=0).stop 2023-11-28T20:35:46.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.709+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2108343902 shutdown_connections 2023-11-28T20:35:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.709+0000 7fb5e5fd4640 1 --2- 172.21.15.45:0/2108343902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 0x7fb5e00b61c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.709+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2108343902 >> 172.21.15.45:0/2108343902 conn(0x7fb5e0070f40 msgr2=0x7fb5e0071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.709+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2108343902 shutdown_connections 2023-11-28T20:35:46.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.709+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2108343902 wait complete. 2023-11-28T20:35:46.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.711+0000 7fb5e5fd4640 1 Processor -- start 2023-11-28T20:35:46.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.711+0000 7fb5e5fd4640 1 -- start start 2023-11-28T20:35:46.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e5fd4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e00b2620 0x7fb5e015f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e5fd4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5e00cddd0 0x7fb5e0151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e5fd4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 0x7fb5e00c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e5fd4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e015fb30 con 0x7fb5e015d210 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e5fd4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e015fe10 con 0x7fb5e00b2620 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e5fd4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01600f0 con 0x7fb5e00cddd0 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e57d3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 0x7fb5e00c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e57d3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 0x7fb5e00c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40866/0 (socket says 172.21.15.45:40866) 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e57d3640 1 -- 172.21.15.45:0/2269858045 learned_addr learned my addr 172.21.15.45:0/2269858045 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e4fd2640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e00b2620 0x7fb5e015f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e57d3640 1 -- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5e00cddd0 msgr2=0x7fb5e0151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e57d3640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5e00cddd0 0x7fb5e0151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e57d3640 1 -- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e00b2620 msgr2=0x7fb5e015f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.712+0000 7fb5e57d3640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e00b2620 0x7fb5e015f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:46.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.713+0000 7fb5e57d3640 1 -- 172.21.15.45:0/2269858045 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d0009a70 con 0x7fb5e015d210 2023-11-28T20:35:46.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.713+0000 7fb5e4fd2640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e00b2620 0x7fb5e015f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:46.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.713+0000 7fb5e57d3640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 0x7fb5e00c8a60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c410 tx=0x7fb5d800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:46.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.713+0000 7fb5dd7fa640 1 -- 172.21.15.45:0/2269858045 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8011790 con 0x7fb5e015d210 2023-11-28T20:35:46.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.713+0000 7fb5dd7fa640 1 -- 172.21.15.45:0/2269858045 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d80118f0 con 0x7fb5e015d210 2023-11-28T20:35:46.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.713+0000 7fb5dd7fa640 1 -- 172.21.15.45:0/2269858045 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8011ce0 con 0x7fb5e015d210 2023-11-28T20:35:46.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.713+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e01603d0 con 0x7fb5e015d210 2023-11-28T20:35:46.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.713+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e0151b50 con 0x7fb5e015d210 2023-11-28T20:35:46.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.714+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e0071b30 con 0x7fb5e015d210 2023-11-28T20:35:46.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.716+0000 7fb5dd7fa640 1 -- 172.21.15.45:0/2269858045 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb5d801c460 con 0x7fb5e015d210 2023-11-28T20:35:46.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.716+0000 7fb5dd7fa640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb5cc06ae40 0x7fb5cc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:46.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.717+0000 7fb5dd7fa640 1 -- 172.21.15.45:0/2269858045 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb5d80894c0 con 0x7fb5e015d210 2023-11-28T20:35:46.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.717+0000 7fb5e4fd2640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb5cc06ae40 0x7fb5cc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:46.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.718+0000 7fb5e4fd2640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb5cc06ae40 0x7fb5cc06d300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0036000 tx=0x7fb5d0030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:46.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.721+0000 7fb5dd7fa640 1 -- 172.21.15.45:0/2269858045 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5d8054390 con 0x7fb5e015d210 2023-11-28T20:35:46.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:46.996+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb5e004e920 con 0x7fb5e015d210 2023-11-28T20:35:47.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.000+0000 7fb5dd7fa640 1 -- 172.21.15.45:0/2269858045 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb5d80579b0 con 0x7fb5e015d210 2023-11-28T20:35:47.003 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T20:35:47.003 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:35:47.003 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T20:35:47.003 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T20:35:47.004 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T20:35:47.004 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T20:35:47.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb5cc06ae40 msgr2=0x7fb5cc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:47.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb5cc06ae40 0x7fb5cc06d300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0036000 tx=0x7fb5d0030000 comp rx=0 tx=0).stop 2023-11-28T20:35:47.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 msgr2=0x7fb5e00c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:47.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 0x7fb5e00c8a60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c410 tx=0x7fb5d800c8d0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 shutdown_connections 2023-11-28T20:35:47.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb5cc06ae40 0x7fb5cc06d300 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e015d210 0x7fb5e00c8a60 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5e00cddd0 0x7fb5e0151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 --2- 172.21.15.45:0/2269858045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e00b2620 0x7fb5e015f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 >> 172.21.15.45:0/2269858045 conn(0x7fb5e0070f40 msgr2=0x7fb5e00d2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:47.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 shutdown_connections 2023-11-28T20:35:47.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.004+0000 7fb5e5fd4640 1 -- 172.21.15.45:0/2269858045 wait complete. 2023-11-28T20:35:47.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:35:47.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.245+0000 7fadf7fff640 1 Processor -- start 2023-11-28T20:35:47.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.245+0000 7fadf7fff640 1 -- start start 2023-11-28T20:35:47.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.247+0000 7fadf7fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fadf80b3870 0x7fadf815db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:47.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.247+0000 7fadf7fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fadf80721e0 0x7fadf815e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:47.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.247+0000 7fadf7fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf815e680 con 0x7fadf806d170 2023-11-28T20:35:47.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.247+0000 7fadf7fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf815a460 con 0x7fadf80b3870 2023-11-28T20:35:47.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.247+0000 7fadf7fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf815e960 con 0x7fadf80721e0 2023-11-28T20:35:47.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.247+0000 7fadf6ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fadf80b3870 0x7fadf815db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:47.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.247+0000 7fadf6ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fadf80b3870 0x7fadf815db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40872/0 (socket says 172.21.15.45:40872) 2023-11-28T20:35:47.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.247+0000 7fadf6ffd640 1 -- 172.21.15.45:0/144394528 learned_addr learned my addr 172.21.15.45:0/144394528 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:47.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.247+0000 7fadeffff640 1 --2- 172.21.15.45:0/144394528 >> v2:172.21.15.190:3300/0 conn(0x7fadf80721e0 0x7fadf815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:47.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.248+0000 7fadf6ffd640 1 -- 172.21.15.45:0/144394528 >> v2:172.21.15.190:3300/0 conn(0x7fadf80721e0 msgr2=0x7fadf815e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:47.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.248+0000 7fadf6ffd640 1 --2- 172.21.15.45:0/144394528 >> v2:172.21.15.190:3300/0 conn(0x7fadf80721e0 0x7fadf815e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.248+0000 7fadf6ffd640 1 -- 172.21.15.45:0/144394528 >> v1:172.21.15.190:6789/0 conn(0x7fadf806d170 legacy=0x7fadf80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:47.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.248+0000 7fadf6ffd640 1 -- 172.21.15.45:0/144394528 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf8071700 con 0x7fadf80b3870 2023-11-28T20:35:47.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.249+0000 7fadf6ffd640 1 --2- 172.21.15.45:0/144394528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 0x7fadf815db10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fadf806e2d0 tx=0x7fade802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40072817ab96e591 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:47.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.249+0000 7fadf67fc640 1 -- 172.21.15.45:0/144394528 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8031870 con 0x7fadf80b3870 2023-11-28T20:35:47.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.249+0000 7fadf67fc640 1 -- 172.21.15.45:0/144394528 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade80319d0 con 0x7fadf80b3870 2023-11-28T20:35:47.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.249+0000 7fadf67fc640 1 -- 172.21.15.45:0/144394528 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8031d60 con 0x7fadf80b3870 2023-11-28T20:35:47.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.250+0000 7fadf7fff640 1 -- 172.21.15.45:0/144394528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 msgr2=0x7fadf815db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:47.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.250+0000 7fadf7fff640 1 --2- 172.21.15.45:0/144394528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 0x7fadf815db10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fadf806e2d0 tx=0x7fade802b910 comp rx=0 tx=0).stop 2023-11-28T20:35:47.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.250+0000 7fadf7fff640 1 -- 172.21.15.45:0/144394528 shutdown_connections 2023-11-28T20:35:47.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.250+0000 7fadf7fff640 1 --2- 172.21.15.45:0/144394528 >> v2:172.21.15.190:3300/0 conn(0x7fadf80721e0 0x7fadf815e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.250+0000 7fadf7fff640 1 --2- 172.21.15.45:0/144394528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 0x7fadf815db10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.250+0000 7fadf7fff640 1 -- 172.21.15.45:0/144394528 >> 172.21.15.45:0/144394528 conn(0x7fadf806ab30 msgr2=0x7fadf80b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:47.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.250+0000 7fadf7fff640 1 -- 172.21.15.45:0/144394528 shutdown_connections 2023-11-28T20:35:47.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.250+0000 7fadf7fff640 1 -- 172.21.15.45:0/144394528 wait complete. 2023-11-28T20:35:47.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.252+0000 7fadf7fff640 1 Processor -- start 2023-11-28T20:35:47.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.252+0000 7fadf7fff640 1 -- start start 2023-11-28T20:35:47.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.253+0000 7fadf7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadf806d170 0x7fadf809c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:47.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.253+0000 7fadf7fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadf80721e0 0x7fadf809b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:47.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.253+0000 7fadf7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 0x7fadf81508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:47.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.253+0000 7fadf7fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf809a9e0 con 0x7fadf80b3870 2023-11-28T20:35:47.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.253+0000 7fadf7fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf809b520 con 0x7fadf806d170 2023-11-28T20:35:47.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.253+0000 7fadf7fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf809c930 con 0x7fadf80721e0 2023-11-28T20:35:47.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.253+0000 7fadf77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 0x7fadf81508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:47.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.253+0000 7fadf77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 0x7fadf81508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40880/0 (socket says 172.21.15.45:40880) 2023-11-28T20:35:47.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.253+0000 7fadf77fe640 1 -- 172.21.15.45:0/2307714143 learned_addr learned my addr 172.21.15.45:0/2307714143 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:47.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadf6ffd640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadf806d170 0x7fadf809c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:47.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadeffff640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadf80721e0 0x7fadf809b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:47.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadf77fe640 1 -- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadf80721e0 msgr2=0x7fadf809b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:47.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadf77fe640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadf80721e0 0x7fadf809b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadf77fe640 1 -- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadf806d170 msgr2=0x7fadf809c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadf77fe640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadf806d170 0x7fadf809c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadf77fe640 1 -- 172.21.15.45:0/2307714143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade80094d0 con 0x7fadf80b3870 2023-11-28T20:35:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadf77fe640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 0x7fadf81508c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fadf000c7b0 tx=0x7fadf000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadef7fe640 1 -- 172.21.15.45:0/2307714143 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0004b60 con 0x7fadf80b3870 2023-11-28T20:35:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadef7fe640 1 -- 172.21.15.45:0/2307714143 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf0004cc0 con 0x7fadf80b3870 2023-11-28T20:35:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadef7fe640 1 -- 172.21.15.45:0/2307714143 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf000aa90 con 0x7fadf80b3870 2023-11-28T20:35:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.254+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf809abf0 con 0x7fadf80b3870 2023-11-28T20:35:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.255+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf809b1b0 con 0x7fadf80b3870 2023-11-28T20:35:47.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.256+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf809b6d0 con 0x7fadf80b3870 2023-11-28T20:35:47.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.257+0000 7fadef7fe640 1 -- 172.21.15.45:0/2307714143 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fadf000abf0 con 0x7fadf80b3870 2023-11-28T20:35:47.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.258+0000 7fadef7fe640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fadd806ae40 0x7fadd806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:47.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.258+0000 7fadef7fe640 1 -- 172.21.15.45:0/2307714143 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fadf0089890 con 0x7fadf80b3870 2023-11-28T20:35:47.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.259+0000 7fadf6ffd640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fadd806ae40 0x7fadd806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:47.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.260+0000 7fadf6ffd640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fadd806ae40 0x7fadd806d300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fade8036000 tx=0x7fade803c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.263+0000 7fadef7fe640 1 -- 172.21.15.45:0/2307714143 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fadf809b6d0 con 0x7fadf80b3870 2023-11-28T20:35:47.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.493+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadf809b6d0 con 0x7fadf80b3870 2023-11-28T20:35:47.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.494+0000 7fadef7fe640 1 -- 172.21.15.45:0/2307714143 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fadf809b6d0 con 0x7fadf80b3870 2023-11-28T20:35:47.498 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:35:47.498 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3050590116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3050590116},{"type":"v1","addr":"172.21.15.45:6835","nonce":3050590116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2465910573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2465910573},{"type":"v1","addr":"172.21.15.190:6836","nonce":2465910573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4090116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4090116745},{"type":"v1","addr":"172.21.15.190:6837","nonce":4090116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:35:35.751211+0000","modified":"2023-11-28T20:35:38.829781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4945":{"gid":4945,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6834/1984946086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1984946086},{"type":"v1","addr":"172.21.15.45:6834","nonce":1984946086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T20:35:47.499 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 18 2023-11-28T20:35:47.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.497+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fadd806ae40 msgr2=0x7fadd806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:47.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.497+0000 7fadf7fff640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fadd806ae40 0x7fadd806d300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fade8036000 tx=0x7fade803c040 comp rx=0 tx=0).stop 2023-11-28T20:35:47.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.497+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 msgr2=0x7fadf81508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:47.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.497+0000 7fadf7fff640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 0x7fadf81508c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fadf000c7b0 tx=0x7fadf000cc70 comp rx=0 tx=0).stop 2023-11-28T20:35:47.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.498+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 shutdown_connections 2023-11-28T20:35:47.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.498+0000 7fadf7fff640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fadd806ae40 0x7fadd806d300 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.498+0000 7fadf7fff640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadf80b3870 0x7fadf81508c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.498+0000 7fadf7fff640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadf80721e0 0x7fadf809b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.498+0000 7fadf7fff640 1 --2- 172.21.15.45:0/2307714143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadf806d170 0x7fadf809c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:47.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.498+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 >> 172.21.15.45:0/2307714143 conn(0x7fadf806ab30 msgr2=0x7fadf80ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:47.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.498+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 shutdown_connections 2023-11-28T20:35:47.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:47.498+0000 7fadf7fff640 1 -- 172.21.15.45:0/2307714143 wait complete. 2023-11-28T20:35:47.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:35:35.751211+0000', 'modified': '2023-11-28T20:35:38.829781+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4945}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4945': {'gid': 4945, 'name': 'c', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.45:6834/1984946086', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 1984946086}, {'type': 'v1', 'addr': '172.21.15.45:6834', 'nonce': 1984946086}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T20:35:47.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:35:47.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T20:35:48.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.156+0000 7faa2e26d640 1 Processor -- start 2023-11-28T20:35:48.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.156+0000 7faa2e26d640 1 -- start start 2023-11-28T20:35:48.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.158+0000 7faa2e26d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa280cba30 0x7faa280cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.158+0000 7faa2e26d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2806e510 con 0x7faa2806b080 2023-11-28T20:35:48.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.158+0000 7faa2e26d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2806e7f0 con 0x7faa2815cef0 2023-11-28T20:35:48.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.158+0000 7faa2e26d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2806a5a0 con 0x7faa280cba30 2023-11-28T20:35:48.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.158+0000 7faa2da6c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faa2806b080 0x7faa28160d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51592/0 (socket says 172.21.15.45:51592) 2023-11-28T20:35:48.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.158+0000 7faa2da6c640 1 -- 172.21.15.45:0/1752115523 learned_addr learned my addr 172.21.15.45:0/1752115523 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:48.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa280cba30 0x7faa280cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408786176 0 0) 0x7faa2806e510 con 0x7faa2806b080 2023-11-28T20:35:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa14003520 con 0x7faa2806b080 2023-11-28T20:35:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4245594277 0 0) 0x7faa14003520 con 0x7faa2806b080 2023-11-28T20:35:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa280cba30 msgr2=0x7faa280cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa1ffff640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa280cba30 0x7faa280cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 >> v1:172.21.15.190:6789/0 conn(0x7faa2815cef0 legacy=0x7faa2806dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2806a880 con 0x7faa2806b080 2023-11-28T20:35:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7faa200019f0 con 0x7faa2806b080 2023-11-28T20:35:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.159+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa280cba30 0x7faa280cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 >> v1:172.21.15.45:6789/0 conn(0x7faa2806b080 legacy=0x7faa28160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa1ffff640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa14003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa1ffff640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa280cba30 0x7faa140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa1ffff640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa14004d00 0x7faa14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2806a5a0 con 0x7faa140039a0 2023-11-28T20:35:48.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1400bac0 con 0x7faa14004d00 2023-11-28T20:35:48.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1400bc80 con 0x7faa280cba30 2023-11-28T20:35:48.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa14003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa280cba30 0x7faa140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa2d26b640 1 -- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa280cba30 msgr2=0x7faa140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa280cba30 0x7faa140047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa2d26b640 1 -- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa14004d00 msgr2=0x7faa14005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa14004d00 0x7faa14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.160+0000 7faa2d26b640 1 -- 172.21.15.45:0/1752115523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14003520 con 0x7faa140039a0 2023-11-28T20:35:48.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.161+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa14003dc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7faa24001720 tx=0x7faa140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7158c2792f302d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.162+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa240052d0 con 0x7faa140039a0 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.162+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2402e050 con 0x7faa140039a0 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.162+0000 7faa1ffff640 1 -- 172.21.15.45:0/1752115523 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa240035f0 con 0x7faa140039a0 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.162+0000 7faa2e26d640 1 -- 172.21.15.45:0/1752115523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 msgr2=0x7faa14003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.162+0000 7faa2e26d640 1 --2- 172.21.15.45:0/1752115523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa14003dc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7faa24001720 tx=0x7faa140057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.162+0000 7faa2da6c640 1 -- 172.21.15.45:0/1752115523 reap_dead start 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.162+0000 7faa2e26d640 1 -- 172.21.15.45:0/1752115523 shutdown_connections 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.162+0000 7faa2e26d640 1 -- 172.21.15.45:0/1752115523 >> 172.21.15.45:0/1752115523 conn(0x7faa280b1170 msgr2=0x7faa280b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.163+0000 7faa2e26d640 1 -- 172.21.15.45:0/1752115523 shutdown_connections 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.163+0000 7faa2e26d640 1 -- 172.21.15.45:0/1752115523 wait complete. 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.164+0000 7faa2e26d640 1 Processor -- start 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.165+0000 7faa2e26d640 1 -- start start 2023-11-28T20:35:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2e26d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa2814dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2e26d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa2815add0 0x7faa2815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2e26d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa281480e0 0x7faa28148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2e26d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28149a90 con 0x7faa140039a0 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2e26d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2815b790 con 0x7faa2815add0 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2e26d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa28147b30 con 0x7faa281480e0 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2d26b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa2814dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2d26b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa2814dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40890/0 (socket says 172.21.15.45:40890) 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2d26b640 1 -- 172.21.15.45:0/2078067548 learned_addr learned my addr 172.21.15.45:0/2078067548 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2d26b640 1 -- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa281480e0 msgr2=0x7faa28148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2d26b640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa281480e0 0x7faa28148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2d26b640 1 -- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa2815add0 msgr2=0x7faa2815b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2d26b640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa2815add0 0x7faa2815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.166+0000 7faa2d26b640 1 -- 172.21.15.45:0/2078067548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa28148e10 con 0x7faa140039a0 2023-11-28T20:35:48.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.167+0000 7faa2d26b640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa2814dd00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7faa24008fd0 tx=0x7faa24001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.167+0000 7faa1dffb640 1 -- 172.21.15.45:0/2078067548 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24007a30 con 0x7faa140039a0 2023-11-28T20:35:48.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.167+0000 7faa1dffb640 1 -- 172.21.15.45:0/2078067548 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa240069e0 con 0x7faa140039a0 2023-11-28T20:35:48.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.167+0000 7faa1dffb640 1 -- 172.21.15.45:0/2078067548 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24006d90 con 0x7faa140039a0 2023-11-28T20:35:48.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.167+0000 7faa2e26d640 1 -- 172.21.15.45:0/2078067548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa281490f0 con 0x7faa140039a0 2023-11-28T20:35:48.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.167+0000 7faa2e26d640 1 -- 172.21.15.45:0/2078067548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa281496b0 con 0x7faa140039a0 2023-11-28T20:35:48.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.170+0000 7faa1dffb640 1 -- 172.21.15.45:0/2078067548 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faa240035f0 con 0x7faa140039a0 2023-11-28T20:35:48.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.170+0000 7faa1dffb640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa0c06ae40 0x7faa0c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.171+0000 7faa1dffb640 1 -- 172.21.15.45:0/2078067548 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7faa240ab670 con 0x7faa140039a0 2023-11-28T20:35:48.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.171+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa0c06ae40 0x7faa0c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.171+0000 7faa3205d740 1 Processor -- start 2023-11-28T20:35:48.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.172+0000 7faa3205d740 1 -- start start 2023-11-28T20:35:48.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.172+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa0c06ae40 0x7faa0c06d300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7faa18006d40 tx=0x7faa18006c30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa3205d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823c070 0x555d6823c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa3205d740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d6823b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa3205d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823b7b0 0x555d6827f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa3205d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6819a880 con 0x555d6823c070 2023-11-28T20:35:48.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa3205d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d681a9ae0 con 0x555d6823b7b0 2023-11-28T20:35:48.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa3205d740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d681a71d0 con 0x555d68236d70 2023-11-28T20:35:48.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa2d26b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823c070 0x555d6823c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa2d26b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823c070 0x555d6823c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40896/0 (socket says 172.21.15.45:40896) 2023-11-28T20:35:48.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa2d26b640 1 -- 172.21.15.45:0/1299321374 learned_addr learned my addr 172.21.15.45:0/1299321374 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:48.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa2d26b640 1 -- 172.21.15.45:0/1299321374 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 msgr2=0x555d6823b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1299321374 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d6823b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa2d26b640 1 -- 172.21.15.45:0/1299321374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823b7b0 msgr2=0x555d6827f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1299321374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823b7b0 0x555d6827f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa2d26b640 1 -- 172.21.15.45:0/1299321374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d6816a860 con 0x555d6823c070 2023-11-28T20:35:48.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.173+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1299321374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823c070 0x555d6823c450 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7faa24079ff0 tx=0x7faa2407a020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62a7787e373ed5c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa0affd640 1 -- 172.21.15.45:0/1299321374 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2407e500 con 0x555d6823c070 2023-11-28T20:35:48.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa0affd640 1 -- 172.21.15.45:0/1299321374 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa240826b0 con 0x555d6823c070 2023-11-28T20:35:48.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa0affd640 1 -- 172.21.15.45:0/1299321374 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24001370 con 0x555d6823c070 2023-11-28T20:35:48.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa3205d740 1 -- 172.21.15.45:0/1299321374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823c070 msgr2=0x555d6823c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa3205d740 1 --2- 172.21.15.45:0/1299321374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823c070 0x555d6823c450 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7faa24079ff0 tx=0x7faa2407a020 comp rx=0 tx=0).stop 2023-11-28T20:35:48.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa3205d740 1 -- 172.21.15.45:0/1299321374 shutdown_connections 2023-11-28T20:35:48.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa3205d740 1 --2- 172.21.15.45:0/1299321374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823b7b0 0x555d6827f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa3205d740 1 --2- 172.21.15.45:0/1299321374 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d6823b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa3205d740 1 --2- 172.21.15.45:0/1299321374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823c070 0x555d6823c450 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.174+0000 7faa3205d740 1 -- 172.21.15.45:0/1299321374 >> 172.21.15.45:0/1299321374 conn(0x555d681fdc70 msgr2=0x555d68281b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:48.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.175+0000 7faa3205d740 1 -- 172.21.15.45:0/1299321374 shutdown_connections 2023-11-28T20:35:48.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.175+0000 7faa3205d740 1 -- 172.21.15.45:0/1299321374 wait complete. 2023-11-28T20:35:48.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.175+0000 7faa3205d740 1 -- start start 2023-11-28T20:35:48.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.176+0000 7faa3205d740 1 Processor -- start 2023-11-28T20:35:48.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa3205d740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d68283c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa3205d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6825baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa3205d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6825bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa3205d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d681a2ac0 con 0x555d6823b7b0 2023-11-28T20:35:48.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa3205d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6813f2f0 con 0x555d6823c070 2023-11-28T20:35:48.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa3205d740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d6815cc70 con 0x555d68236d70 2023-11-28T20:35:48.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa3205d740 10 client.0 _ll_register_callbacks cb 0x555d681613b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:35:48.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2ca6a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6825baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2ca6a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6825baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40906/0 (socket says 172.21.15.45:40906) 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2ca6a640 1 -- 172.21.15.45:0/2463004864 learned_addr learned my addr 172.21.15.45:0/2463004864 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2da6c640 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6825bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2d26b640 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d68283c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2ca6a640 1 -- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 msgr2=0x555d68283c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d68283c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2ca6a640 1 -- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 msgr2=0x555d6825bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6825bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2ca6a640 1 -- 172.21.15.45:0/2463004864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa281496b0 con 0x555d6823b7b0 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2d26b640 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d68283c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.177+0000 7faa2da6c640 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6825bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.178+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6825baa0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faa1800ddd0 tx=0x7faa18014830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb6a85d82f7e914 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.178+0000 7faa03fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.178+0000 7faa03fff640 1 -- 172.21.15.45:0/2463004864 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18006e30 con 0x555d6823b7b0 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.178+0000 7faa3205d740 1 -- 172.21.15.45:0/2463004864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d6819baa0 con 0x555d6823b7b0 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.178+0000 7faa03fff640 1 -- 172.21.15.45:0/2463004864 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7faa1801f3f0 con 0x555d6823b7b0 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.178+0000 7faa03fff640 1 -- 172.21.15.45:0/2463004864 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa18005990 con 0x555d6823b7b0 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.178+0000 7faa3205d740 10 client.5541 fetch_fsmap learned FSMap version 18 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.178+0000 7faa3205d740 1 -- 172.21.15.45:0/2463004864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x555d681a1140 con 0x555d6823b7b0 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.179+0000 7faa03fff640 1 -- 172.21.15.45:0/2463004864 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555d6819baa0 con 0x555d6823b7b0 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.179+0000 7faa03fff640 1 -- 172.21.15.45:0/2463004864 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7faa18005af0 con 0x555d6823b7b0 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.179+0000 7faa3205d740 10 client.5541 fetch_fsmap finished waiting for FSMap version 18 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.179+0000 7faa3205d740 10 client.5541 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:35:48.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.179+0000 7faa3205d740 20 client.5541 populate_metadata read hostname 'smithi045' 2023-11-28T20:35:48.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.179+0000 7faa3205d740 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] conn(0x555d6825aba0 0x555d6829eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.179+0000 7faa3205d740 4 client.5541 mds_command: new command op to 4945 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:35:48.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.179+0000 7faa3205d740 1 -- 172.21.15.45:0/2463004864 --> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d6819b7c0 con 0x555d6825aba0 2023-11-28T20:35:48.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.179+0000 7faa2d26b640 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] conn(0x555d6825aba0 0x555d6829eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.180+0000 7faa2d26b640 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] conn(0x555d6825aba0 0x555d6829eb90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b10aa605a00b90e1 server_cookie=c98b7eb2a42b5652 in_seq=0 out_seq=0 2023-11-28T20:35:48.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.180+0000 7faa03fff640 10 client.5541 ms_handle_connect on v2:172.21.15.45:6832/1984946086 2023-11-28T20:35:48.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa03fff640 1 -- 172.21.15.45:0/2463004864 <== mds.0 v2:172.21.15.45:6832/1984946086 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555d6819b7c0 con 0x555d6825aba0 2023-11-28T20:35:48.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa03fff640 10 client.5541 handle_command_reply: tid=0 2023-11-28T20:35:48.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 client.5541 shutdown 2023-11-28T20:35:48.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 -- 172.21.15.45:0/2463004864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 msgr2=0x555d6825baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6825baa0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faa1800ddd0 tx=0x7faa18014830 comp rx=0 tx=0).stop 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 -- 172.21.15.45:0/2463004864 shutdown_connections 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] conn(0x555d6825aba0 0x555d6829eb90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6825bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6825baa0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 --2- 172.21.15.45:0/2463004864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d68283c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa03fff640 0 client.5541 ms_handle_reset on v2:172.21.15.45:6832/1984946086 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 -- 172.21.15.45:0/2463004864 >> 172.21.15.45:0/2463004864 conn(0x555d681fdc70 msgr2=0x555d68235dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 -- 172.21.15.45:0/2463004864 shutdown_connections 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 1 -- 172.21.15.45:0/2463004864 wait complete. 2023-11-28T20:35:48.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.181+0000 7faa3205d740 20 client.5541 trim_cache size 0 max 16384 2023-11-28T20:35:48.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.198+0000 7faa3205d740 1 Processor -- start 2023-11-28T20:35:48.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.199+0000 7faa3205d740 1 -- start start 2023-11-28T20:35:48.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.199+0000 7faa3205d740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d6826a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.199+0000 7faa3205d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6827fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.199+0000 7faa3205d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6825ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.199+0000 7faa3205d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d67f05f20 con 0x555d6823b7b0 2023-11-28T20:35:48.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.199+0000 7faa3205d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d682858f0 con 0x555d6823c070 2023-11-28T20:35:48.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.199+0000 7faa3205d740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d68264170 con 0x555d68236d70 2023-11-28T20:35:48.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.200+0000 7faa2ca6a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6827fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.200+0000 7faa2ca6a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6827fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40916/0 (socket says 172.21.15.45:40916) 2023-11-28T20:35:48.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.200+0000 7faa2ca6a640 1 -- 172.21.15.45:0/637850035 learned_addr learned my addr 172.21.15.45:0/637850035 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:48.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.200+0000 7faa2ca6a640 1 -- 172.21.15.45:0/637850035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 msgr2=0x555d6826a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.200+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/637850035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d6826a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.200+0000 7faa2ca6a640 1 -- 172.21.15.45:0/637850035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 msgr2=0x555d6825ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.200+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/637850035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6825ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.200+0000 7faa2ca6a640 1 -- 172.21.15.45:0/637850035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d6813eb80 con 0x555d6823b7b0 2023-11-28T20:35:48.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.200+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/637850035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6827fe50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7faa1800de00 tx=0x7faa18018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e1825c0ce78c66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa0b7fe640 1 -- 172.21.15.45:0/637850035 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18005b90 con 0x555d6823b7b0 2023-11-28T20:35:48.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa0b7fe640 1 -- 172.21.15.45:0/637850035 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18006000 con 0x555d6823b7b0 2023-11-28T20:35:48.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa0b7fe640 1 -- 172.21.15.45:0/637850035 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa180063b0 con 0x555d6823b7b0 2023-11-28T20:35:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa3205d740 1 -- 172.21.15.45:0/637850035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 msgr2=0x555d6827fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa3205d740 1 --2- 172.21.15.45:0/637850035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6827fe50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7faa1800de00 tx=0x7faa18018000 comp rx=0 tx=0).stop 2023-11-28T20:35:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa3205d740 1 -- 172.21.15.45:0/637850035 shutdown_connections 2023-11-28T20:35:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa3205d740 1 --2- 172.21.15.45:0/637850035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6825ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa3205d740 1 --2- 172.21.15.45:0/637850035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6823b7b0 0x555d6827fe50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa3205d740 1 --2- 172.21.15.45:0/637850035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d68236d70 0x555d6826a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa3205d740 1 -- 172.21.15.45:0/637850035 >> 172.21.15.45:0/637850035 conn(0x555d681fdc70 msgr2=0x555d68258680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa3205d740 1 -- 172.21.15.45:0/637850035 shutdown_connections 2023-11-28T20:35:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.201+0000 7faa3205d740 1 -- 172.21.15.45:0/637850035 wait complete. 2023-11-28T20:35:48.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.202+0000 7faa3205d740 1 -- start start 2023-11-28T20:35:48.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.202+0000 7faa3205d740 1 Processor -- start 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa3205d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d68236d70 0x555d68263870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa3205d740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d6823b7b0 0x555d6829ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa3205d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6829a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa3205d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d68263db0 con 0x555d68236d70 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa3205d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6825c350 con 0x555d6823c070 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa3205d740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d6826a060 con 0x555d6823b7b0 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa3205d740 10 client.0 _ll_register_callbacks cb 0x7faa1800d4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa2d26b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d68236d70 0x555d68263870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa2d26b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d68236d70 0x555d68263870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40932/0 (socket says 172.21.15.45:40932) 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa2d26b640 1 -- 172.21.15.45:0/1352514649 learned_addr learned my addr 172.21.15.45:0/1352514649 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d6823b7b0 0x555d6829ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa2d26b640 1 -- 172.21.15.45:0/1352514649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d6823b7b0 msgr2=0x555d6829ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d6823b7b0 0x555d6829ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa2d26b640 1 -- 172.21.15.45:0/1352514649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 msgr2=0x555d6829a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6829a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.204+0000 7faa2d26b640 1 -- 172.21.15.45:0/1352514649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d681a1140 con 0x555d68236d70 2023-11-28T20:35:48.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.205+0000 7faa2d26b640 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d68236d70 0x555d68263870 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7faa2407aa10 tx=0x7faa240ab070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b29dd13667a23f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.205+0000 7faa03fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:35:48.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.205+0000 7faa03fff640 1 -- 172.21.15.45:0/1352514649 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa240a9040 con 0x555d68236d70 2023-11-28T20:35:48.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.205+0000 7faa03fff640 1 -- 172.21.15.45:0/1352514649 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7faa240a8040 con 0x555d68236d70 2023-11-28T20:35:48.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.205+0000 7faa3205d740 1 -- 172.21.15.45:0/1352514649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d68284130 con 0x555d68236d70 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.205+0000 7faa03fff640 1 -- 172.21.15.45:0/1352514649 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa240766f0 con 0x555d68236d70 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.205+0000 7faa3205d740 10 client.5547 fetch_fsmap learned FSMap version 18 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.205+0000 7faa3205d740 1 -- 172.21.15.45:0/1352514649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x555d68164170 con 0x555d68236d70 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.205+0000 7faa03fff640 1 -- 172.21.15.45:0/1352514649 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2409b020 con 0x555d68236d70 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.206+0000 7faa03fff640 1 -- 172.21.15.45:0/1352514649 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7faa2408a030 con 0x555d68236d70 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.206+0000 7faa3205d740 10 client.5547 fetch_fsmap finished waiting for FSMap version 18 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.206+0000 7faa3205d740 10 client.5547 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.206+0000 7faa3205d740 20 client.5547 populate_metadata read hostname 'smithi045' 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.206+0000 7faa3205d740 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] conn(0x555d682eb060 0x555d682eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.206+0000 7faa3205d740 4 client.5547 mds_command: new command op to 4945 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:35:48.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.206+0000 7faa3205d740 1 -- 172.21.15.45:0/1352514649 --> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7faa240826b0 con 0x555d682eb060 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.206+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] conn(0x555d682eb060 0x555d682eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa2ca6a640 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] conn(0x555d682eb060 0x555d682eb440 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f5101df81604ae1 server_cookie=4dd8082c0ca2372a in_seq=0 out_seq=0 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa03fff640 10 client.5547 ms_handle_connect on v2:172.21.15.45:6832/1984946086 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa03fff640 1 -- 172.21.15.45:0/1352514649 <== mds.0 v2:172.21.15.45:6832/1984946086 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7faa240826b0 con 0x555d682eb060 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa03fff640 10 client.5547 handle_command_reply: tid=0 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 client.5547 shutdown 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 -- 172.21.15.45:0/1352514649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d68236d70 msgr2=0x555d68263870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d68236d70 0x555d68263870 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7faa2407aa10 tx=0x7faa240ab070 comp rx=0 tx=0).stop 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 -- 172.21.15.45:0/1352514649 shutdown_connections 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.45:6832/1984946086,v1:172.21.15.45:6834/1984946086] conn(0x555d682eb060 0x555d682eb440 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d6823c070 0x555d6829a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x555d6823b7b0 0x555d6829ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 --2- 172.21.15.45:0/1352514649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d68236d70 0x555d68263870 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa03fff640 0 client.5547 ms_handle_reset on v2:172.21.15.45:6832/1984946086 2023-11-28T20:35:48.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 -- 172.21.15.45:0/1352514649 >> 172.21.15.45:0/1352514649 conn(0x555d681fdc70 msgr2=0x555d68282c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:48.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 -- 172.21.15.45:0/1352514649 shutdown_connections 2023-11-28T20:35:48.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.207+0000 7faa3205d740 1 -- 172.21.15.45:0/1352514649 wait complete. 2023-11-28T20:35:48.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa3205d740 20 client.5547 trim_cache size 0 max 16384 2023-11-28T20:35:48.213 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:35:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:35:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:35:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 4945, 2023-11-28T20:35:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:35:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:35:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:35:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 11.887112045, 2023-11-28T20:35:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 18, 2023-11-28T20:35:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 39, 2023-11-28T20:35:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 39, 2023-11-28T20:35:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 20.728742306000001 2023-11-28T20:35:48.214 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:35:48.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 -- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa0c06ae40 msgr2=0x7faa0c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa0c06ae40 0x7faa0c06d300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7faa18006d40 tx=0x7faa18006c30 comp rx=0 tx=0).stop 2023-11-28T20:35:48.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 -- 172.21.15.45:0/2078067548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 msgr2=0x7faa2814dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa2814dd00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7faa24008fd0 tx=0x7faa24001120 comp rx=0 tx=0).stop 2023-11-28T20:35:48.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 -- 172.21.15.45:0/2078067548 shutdown_connections 2023-11-28T20:35:48.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa0c06ae40 0x7faa0c06d300 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa281480e0 0x7faa28148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa2815add0 0x7faa2815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 --2- 172.21.15.45:0/2078067548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa140039a0 0x7faa2814dd00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 -- 172.21.15.45:0/2078067548 >> 172.21.15.45:0/2078067548 conn(0x7faa280b1170 msgr2=0x7faa2815ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:48.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 -- 172.21.15.45:0/2078067548 shutdown_connections 2023-11-28T20:35:48.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.208+0000 7faa2e26d640 1 -- 172.21.15.45:0/2078067548 wait complete. 2023-11-28T20:35:48.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:35:48.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.445+0000 7f0fdcb88640 1 Processor -- start 2023-11-28T20:35:48.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.445+0000 7f0fdcb88640 1 -- start start 2023-11-28T20:35:48.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.447+0000 7f0fdcb88640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0fd8155780 0x7f0fd80abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.447+0000 7f0fdcb88640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fd80ce800 0x7f0fd80b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.447+0000 7f0fdcb88640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd815c3c0 con 0x7f0fd80ae540 2023-11-28T20:35:48.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.447+0000 7f0fdcb88640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd8155b60 con 0x7f0fd8155780 2023-11-28T20:35:48.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.447+0000 7f0fdcb88640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd815ba90 con 0x7f0fd80ce800 2023-11-28T20:35:48.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.447+0000 7f0fceffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0fd80ae540 0x7f0fd80ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51606/0 (socket says 172.21.15.45:51606) 2023-11-28T20:35:48.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.447+0000 7f0fceffd640 1 -- 172.21.15.45:0/297217932 learned_addr learned my addr 172.21.15.45:0/297217932 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd77fe640 1 --2- 172.21.15.45:0/297217932 >> v2:172.21.15.190:3300/0 conn(0x7f0fd8155780 0x7f0fd80abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd7fff640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fd80ce800 0x7f0fd80b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930844600 0 0) 0x7f0fd815c3c0 con 0x7f0fd80ae540 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc4003520 con 0x7f0fd80ae540 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095576922 0 0) 0x7f0fc4003520 con 0x7f0fd80ae540 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fd80ce800 msgr2=0x7f0fd80b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fd80ce800 0x7f0fd80b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 >> v2:172.21.15.190:3300/0 conn(0x7f0fd8155780 msgr2=0x7f0fd80abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 --2- 172.21.15.45:0/297217932 >> v2:172.21.15.190:3300/0 conn(0x7f0fd8155780 0x7f0fd80abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd815bd70 con 0x7f0fd80ae540 2023-11-28T20:35:48.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f0fc0001670 con 0x7f0fd80ae540 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.448+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 >> v1:172.21.15.45:6789/0 conn(0x7f0fd80ae540 legacy=0x7f0fd80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fd6ffd640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fc4003970 0x7f0fc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fd6ffd640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40047c0 0x7f0fc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fd6ffd640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fd80ce800 0x7f0fc4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd8155b60 con 0x7f0fc40047c0 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd815ba90 con 0x7f0fc4003970 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc400b980 con 0x7f0fd80ce800 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fceffd640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40047c0 0x7f0fc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fd77fe640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fc4003970 0x7f0fc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fceffd640 1 -- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fd80ce800 msgr2=0x7f0fc4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fceffd640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fd80ce800 0x7f0fc4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fceffd640 1 -- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fc4003970 msgr2=0x7f0fc4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:48.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fceffd640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fc4003970 0x7f0fc4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.449+0000 7f0fceffd640 1 -- 172.21.15.45:0/297217932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4003520 con 0x7f0fc40047c0 2023-11-28T20:35:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.450+0000 7f0fceffd640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40047c0 0x7f0fc4004c40 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c270 tx=0x7f0fc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c563acaf104970d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc002cae0 con 0x7f0fc40047c0 2023-11-28T20:35:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc002cc40 con 0x7f0fc40047c0 2023-11-28T20:35:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fd6ffd640 1 -- 172.21.15.45:0/297217932 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00058d0 con 0x7f0fc40047c0 2023-11-28T20:35:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fdcb88640 1 -- 172.21.15.45:0/297217932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40047c0 msgr2=0x7f0fc4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fdcb88640 1 --2- 172.21.15.45:0/297217932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40047c0 0x7f0fc4004c40 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c270 tx=0x7f0fc40057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fd7fff640 1 -- 172.21.15.45:0/297217932 reap_dead start 2023-11-28T20:35:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fdcb88640 1 -- 172.21.15.45:0/297217932 shutdown_connections 2023-11-28T20:35:48.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fdcb88640 1 -- 172.21.15.45:0/297217932 >> 172.21.15.45:0/297217932 conn(0x7f0fd806ab30 msgr2=0x7f0fd80b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:48.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fdcb88640 1 -- 172.21.15.45:0/297217932 shutdown_connections 2023-11-28T20:35:48.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.451+0000 7f0fdcb88640 1 -- 172.21.15.45:0/297217932 wait complete. 2023-11-28T20:35:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.453+0000 7f0fdcb88640 1 Processor -- start 2023-11-28T20:35:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.453+0000 7f0fdcb88640 1 -- start start 2023-11-28T20:35:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.454+0000 7f0fdcb88640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fc40047c0 0x7f0fd8148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.454+0000 7f0fdcb88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fd80c3d00 0x7f0fd8149350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.454+0000 7f0fdcb88640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fd80c4680 0x7f0fd80c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.454+0000 7f0fdcb88640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80c5040 con 0x7f0fd80c3d00 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.454+0000 7f0fdcb88640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd8148e90 con 0x7f0fd80c4680 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.454+0000 7f0fdcb88640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd814c230 con 0x7f0fc40047c0 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fd80c3d00 0x7f0fd8149350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fd80c3d00 0x7f0fd8149350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40952/0 (socket says 172.21.15.45:40952) 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fceffd640 1 -- 172.21.15.45:0/8988320 learned_addr learned my addr 172.21.15.45:0/8988320 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fd7fff640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fd80c4680 0x7f0fd80c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fd77fe640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fc40047c0 0x7f0fd8148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fceffd640 1 -- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fc40047c0 msgr2=0x7f0fd8148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fceffd640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fc40047c0 0x7f0fd8148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fceffd640 1 -- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fd80c4680 msgr2=0x7f0fd80c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fceffd640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fd80c4680 0x7f0fd80c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fceffd640 1 -- 172.21.15.45:0/8988320 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd8070730 con 0x7f0fd80c3d00 2023-11-28T20:35:48.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.455+0000 7f0fceffd640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fd80c3d00 0x7f0fd8149350 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c390 tx=0x7f0fc0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.456+0000 7f0fd4ff9640 1 -- 172.21.15.45:0/8988320 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0039070 con 0x7f0fd80c3d00 2023-11-28T20:35:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.456+0000 7f0fd4ff9640 1 -- 172.21.15.45:0/8988320 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc0001630 con 0x7f0fd80c3d00 2023-11-28T20:35:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.456+0000 7f0fd4ff9640 1 -- 172.21.15.45:0/8988320 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc003c040 con 0x7f0fd80c3d00 2023-11-28T20:35:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.456+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd8070a10 con 0x7f0fd80c3d00 2023-11-28T20:35:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.456+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd8070fd0 con 0x7f0fd80c3d00 2023-11-28T20:35:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.457+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd804e920 con 0x7f0fd80c3d00 2023-11-28T20:35:48.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.458+0000 7f0fd4ff9640 1 -- 172.21.15.45:0/8988320 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0fc0032070 con 0x7f0fd80c3d00 2023-11-28T20:35:48.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.459+0000 7f0fd4ff9640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0fb806ae70 0x7f0fb806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:48.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.459+0000 7f0fd4ff9640 1 -- 172.21.15.45:0/8988320 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0fc00aadb0 con 0x7f0fd80c3d00 2023-11-28T20:35:48.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.460+0000 7f0fd77fe640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0fb806ae70 0x7f0fb806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:48.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.461+0000 7f0fd77fe640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0fb806ae70 0x7f0fb806d330 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009fc0 tx=0x7f0fc800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.463+0000 7f0fd4ff9640 1 -- 172.21.15.45:0/8988320 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0fc0075bf0 con 0x7f0fd80c3d00 2023-11-28T20:35:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.694+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0fd80b0c80 con 0x7f0fd80c3d00 2023-11-28T20:35:48.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.695+0000 7f0fd4ff9640 1 -- 172.21.15.45:0/8988320 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0fc0046070 con 0x7f0fd80c3d00 2023-11-28T20:35:48.698 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T20:35:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.697+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0fb806ae70 msgr2=0x7f0fb806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.697+0000 7f0fdcb88640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0fb806ae70 0x7f0fb806d330 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009fc0 tx=0x7f0fc800b040 comp rx=0 tx=0).stop 2023-11-28T20:35:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.697+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fd80c3d00 msgr2=0x7f0fd8149350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.697+0000 7f0fdcb88640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fd80c3d00 0x7f0fd8149350 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c390 tx=0x7f0fc0001a00 comp rx=0 tx=0).stop 2023-11-28T20:35:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.698+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 shutdown_connections 2023-11-28T20:35:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.698+0000 7f0fdcb88640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f0fb806ae70 0x7f0fb806d330 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.698+0000 7f0fdcb88640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fd80c4680 0x7f0fd80c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.698+0000 7f0fdcb88640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fd80c3d00 0x7f0fd8149350 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.698+0000 7f0fdcb88640 1 --2- 172.21.15.45:0/8988320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fc40047c0 0x7f0fd8148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.698+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 >> 172.21.15.45:0/8988320 conn(0x7f0fd806ab30 msgr2=0x7f0fd806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.698+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 shutdown_connections 2023-11-28T20:35:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:48.698+0000 7f0fdcb88640 1 -- 172.21.15.45:0/8988320 wait complete. 2023-11-28T20:35:48.719 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T20:35:48.719 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:35:48.720 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:35:48.720 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T20:35:48.720 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:35:48.720 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T20:35:48.720 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T20:35:48.720 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:35:48.720 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:35:48.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:48.721 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:35:48.779 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:35:48.782 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:35:48.782 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:35:48.782 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:35:48.782 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:35:48.782 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:35:48.782 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:35:48.782 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42194sec preferred_lft 42194sec 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:35:48.783 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:35:48.784 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:35:48.784 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:35:48.784 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T20:35:48.784 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:35:48.784 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:35:48.784 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T20:35:48.784 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:35:48.784 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:35:48.785 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:35:48.785 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T20:35:48.785 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:35:48.785 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:35:48.785 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T20:35:48.785 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T20:35:48.785 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:35:48.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:48.787 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:35:48.884 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:35:48.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:48.888 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:35:48.946 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:35:48.946 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:35:48.946 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:35:48.946 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:35:48.946 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:35:48.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:48.947 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:35:48.947 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:35:48.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:35:48.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:35:48.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:35:48.947 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:35:49.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:49.144 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:35:49.199 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T20:35:49.199 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T20:35:49.200 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:35:49.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:49.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:35:49.302 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:35:49.302 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T20:35:49.358 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:35:49.359 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:35:49.359 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:35:49.413 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:35:49.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:49.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:35:49.563 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:35:49.563 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:35:49.563 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:35:49.587 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:35:49.588 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:35:49.588 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:35:49.588 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:35:49.588 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:35:49.614 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:35:49.614 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:35:49.614 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:35:49.615 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:35:49.615 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:35:49.615 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:35:49.615 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:35:49.615 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:35:49.616 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T20:35:49.616 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:35:49.617 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:35:49.617 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:35:49.617 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:35:49.624 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:35:49.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:49.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:35:49.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:49.724 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:35:49.793 INFO:teuthology.orchestra.run.smithi045.stdout:client5556 2023-11-28T20:35:49.793 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:35:49.794 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:35:49.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T20:35:50.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.080+0000 7fb36d69b640 1 Processor -- start 2023-11-28T20:35:50.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.081+0000 7fb36d69b640 1 -- start start 2023-11-28T20:35:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.082+0000 7fb36d69b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb36815b820 0x7fb36806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.082+0000 7fb36d69b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36815f4d0 0x7fb3680cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.082+0000 7fb36d69b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36815fa80 con 0x7fb3680cf020 2023-11-28T20:35:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.082+0000 7fb36d69b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36815fd60 con 0x7fb36815b820 2023-11-28T20:35:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.082+0000 7fb36d69b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680cf9f0 con 0x7fb36815f4d0 2023-11-28T20:35:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.083+0000 7fb3667fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb3680cf020 0x7fb36815edc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51614/0 (socket says 172.21.15.45:51614) 2023-11-28T20:35:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.083+0000 7fb3667fc640 1 -- 172.21.15.45:0/1861605526 learned_addr learned my addr 172.21.15.45:0/1861605526 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.083+0000 7fb3677fe640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36815f4d0 0x7fb3680cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.083+0000 7fb366ffd640 1 --2- 172.21.15.45:0/1861605526 >> v2:172.21.15.190:3300/0 conn(0x7fb36815b820 0x7fb36806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.083+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357747928 0 0) 0x7fb36815fa80 con 0x7fb3680cf020 2023-11-28T20:35:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.083+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb354003520 con 0x7fb3680cf020 2023-11-28T20:35:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 405811213 0 0) 0x7fb354003520 con 0x7fb3680cf020 2023-11-28T20:35:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36815f4d0 msgr2=0x7fb3680cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36815f4d0 0x7fb3680cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 >> v2:172.21.15.190:3300/0 conn(0x7fb36815b820 msgr2=0x7fb36806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 --2- 172.21.15.45:0/1861605526 >> v2:172.21.15.190:3300/0 conn(0x7fb36815b820 0x7fb36806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36806cf40 con 0x7fb3680cf020 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb350001690 con 0x7fb3680cf020 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 >> v1:172.21.15.45:6789/0 conn(0x7fb3680cf020 legacy=0x7fb36815edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36815f4d0 0x7fb3540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb354003f20 0x7fb3540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb354004d00 0x7fb354005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36815fd60 con 0x7fb354003f20 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680cf9f0 con 0x7fb354004d00 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35400ba00 con 0x7fb36815f4d0 2023-11-28T20:35:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb3667fc640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb354003f20 0x7fb3540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.084+0000 7fb366ffd640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36815f4d0 0x7fb3540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.085+0000 7fb3677fe640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb354004d00 0x7fb354005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.085+0000 7fb3667fc640 1 -- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36815f4d0 msgr2=0x7fb3540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.085+0000 7fb3667fc640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36815f4d0 0x7fb3540039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.085+0000 7fb3667fc640 1 -- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb354004d00 msgr2=0x7fb354005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.085+0000 7fb3667fc640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb354004d00 0x7fb354005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.085+0000 7fb3667fc640 1 -- 172.21.15.45:0/1861605526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb354003520 con 0x7fb354003f20 2023-11-28T20:35:50.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.086+0000 7fb3667fc640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb354003f20 0x7fb3540047c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb3680cb750 tx=0x7fb3540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c333f26c0214aff0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:50.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.086+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb350005ad0 con 0x7fb354003f20 2023-11-28T20:35:50.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.086+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb350005c30 con 0x7fb354003f20 2023-11-28T20:35:50.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.086+0000 7fb365ffb640 1 -- 172.21.15.45:0/1861605526 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35002c890 con 0x7fb354003f20 2023-11-28T20:35:50.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.087+0000 7fb36d69b640 1 -- 172.21.15.45:0/1861605526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb354003f20 msgr2=0x7fb3540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.087+0000 7fb36d69b640 1 --2- 172.21.15.45:0/1861605526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb354003f20 0x7fb3540047c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb3680cb750 tx=0x7fb3540057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.087+0000 7fb3677fe640 1 -- 172.21.15.45:0/1861605526 reap_dead start 2023-11-28T20:35:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.087+0000 7fb36d69b640 1 -- 172.21.15.45:0/1861605526 shutdown_connections 2023-11-28T20:35:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.087+0000 7fb36d69b640 1 -- 172.21.15.45:0/1861605526 >> 172.21.15.45:0/1861605526 conn(0x7fb36806a6e0 msgr2=0x7fb36806aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.087+0000 7fb36d69b640 1 -- 172.21.15.45:0/1861605526 shutdown_connections 2023-11-28T20:35:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.087+0000 7fb36d69b640 1 -- 172.21.15.45:0/1861605526 wait complete. 2023-11-28T20:35:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.089+0000 7fb36d69b640 1 Processor -- start 2023-11-28T20:35:50.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.089+0000 7fb36d69b640 1 -- start start 2023-11-28T20:35:50.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.090+0000 7fb36d69b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb354003f20 0x7fb3680ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.090+0000 7fb36d69b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3680acae0 0x7fb3681456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.090+0000 7fb36d69b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36809ab30 0x7fb36809afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.090+0000 7fb36d69b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680adc60 con 0x7fb3680acae0 2023-11-28T20:35:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.090+0000 7fb36d69b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680ae5c0 con 0x7fb36809ab30 2023-11-28T20:35:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.090+0000 7fb36d69b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36809b4f0 con 0x7fb354003f20 2023-11-28T20:35:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.090+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3680acae0 0x7fb3681456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.090+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3680acae0 0x7fb3681456a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40978/0 (socket says 172.21.15.45:40978) 2023-11-28T20:35:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.090+0000 7fb3667fc640 1 -- 172.21.15.45:0/1540242025 learned_addr learned my addr 172.21.15.45:0/1540242025 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb366ffd640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb354003f20 0x7fb3680ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb3677fe640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36809ab30 0x7fb36809afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb3667fc640 1 -- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb354003f20 msgr2=0x7fb3680ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb3667fc640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb354003f20 0x7fb3680ae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb3667fc640 1 -- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36809ab30 msgr2=0x7fb36809afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb3667fc640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36809ab30 0x7fb36809afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb3667fc640 1 -- 172.21.15.45:0/1540242025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3681534e0 con 0x7fb3680acae0 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb3677fe640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36809ab30 0x7fb36809afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb3667fc640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3680acae0 0x7fb3681456a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb350000f60 tx=0x7fb350001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.091+0000 7fb35f7fe640 1 -- 172.21.15.45:0/1540242025 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35003c040 con 0x7fb3680acae0 2023-11-28T20:35:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.092+0000 7fb35f7fe640 1 -- 172.21.15.45:0/1540242025 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35002dd30 con 0x7fb3680acae0 2023-11-28T20:35:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.092+0000 7fb35f7fe640 1 -- 172.21.15.45:0/1540242025 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35002dea0 con 0x7fb3680acae0 2023-11-28T20:35:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.092+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3680acfe0 con 0x7fb3680acae0 2023-11-28T20:35:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.092+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3680ad5a0 con 0x7fb3680acae0 2023-11-28T20:35:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.093+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3680adda0 con 0x7fb3680acae0 2023-11-28T20:35:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.094+0000 7fb35f7fe640 1 -- 172.21.15.45:0/1540242025 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb350032070 con 0x7fb3680acae0 2023-11-28T20:35:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.095+0000 7fb35f7fe640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb34806ae70 0x7fb34806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.095+0000 7fb35f7fe640 1 -- 172.21.15.45:0/1540242025 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb3500aae40 con 0x7fb3680acae0 2023-11-28T20:35:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.096+0000 7fb366ffd640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb34806ae70 0x7fb34806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.097+0000 7fb366ffd640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb34806ae70 0x7fb34806d330 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb358007520 tx=0x7fb35800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:50.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.100+0000 7fb35f7fe640 1 -- 172.21.15.45:0/1540242025 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3680adda0 con 0x7fb3680acae0 2023-11-28T20:35:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.317+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb36806d500 con 0x7fb34806ae70 2023-11-28T20:35:50.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.322+0000 7fb35f7fe640 1 -- 172.21.15.45:0/1540242025 <== mgr.4109 v2:172.21.15.190:6832/1081538426 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb358003990 con 0x7fb34806ae70 2023-11-28T20:35:50.324 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:35:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:35:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T20:35:50.324 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:35:50.324 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:35:50.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb34806ae70 msgr2=0x7fb34806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:50.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb34806ae70 0x7fb34806d330 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb358007520 tx=0x7fb35800b040 comp rx=0 tx=0).stop 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3680acae0 msgr2=0x7fb3681456a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3680acae0 0x7fb3681456a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb350000f60 tx=0x7fb350001a20 comp rx=0 tx=0).stop 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 shutdown_connections 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7fb34806ae70 0x7fb34806d330 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36809ab30 0x7fb36809afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3680acae0 0x7fb3681456a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 --2- 172.21.15.45:0/1540242025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb354003f20 0x7fb3680ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 >> 172.21.15.45:0/1540242025 conn(0x7fb36806a6e0 msgr2=0x7fb3680cc050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 shutdown_connections 2023-11-28T20:35:50.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.324+0000 7fb36d69b640 1 -- 172.21.15.45:0/1540242025 wait complete. 2023-11-28T20:35:50.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T20:35:50.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.581+0000 7f024dbe3640 1 Processor -- start 2023-11-28T20:35:50.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.581+0000 7f024dbe3640 1 -- start start 2023-11-28T20:35:50.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.583+0000 7f024dbe3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0248157c10 0x7f024806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.583+0000 7f024dbe3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f024806a500 0x7f024806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.583+0000 7f024dbe3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024806ca60 con 0x7f024815ddd0 2023-11-28T20:35:50.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.583+0000 7f024dbe3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024815e930 con 0x7f024806a500 2023-11-28T20:35:50.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.583+0000 7f024dbe3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024815eb40 con 0x7f0248157c10 2023-11-28T20:35:50.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.583+0000 7f0246ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f024806a500 0x7f024806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.583+0000 7f0246ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f024806a500 0x7f024806a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40990/0 (socket says 172.21.15.45:40990) 2023-11-28T20:35:50.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.583+0000 7f0246ffd640 1 -- 172.21.15.45:0/2457025026 learned_addr learned my addr 172.21.15.45:0/2457025026 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:50.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.584+0000 7f02477fe640 1 --2- 172.21.15.45:0/2457025026 >> v2:172.21.15.190:3300/0 conn(0x7f0248157c10 0x7f024806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.584+0000 7f0246ffd640 1 -- 172.21.15.45:0/2457025026 >> v2:172.21.15.190:3300/0 conn(0x7f0248157c10 msgr2=0x7f024806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.584+0000 7f0246ffd640 1 --2- 172.21.15.45:0/2457025026 >> v2:172.21.15.190:3300/0 conn(0x7f0248157c10 0x7f024806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.584+0000 7f0246ffd640 1 -- 172.21.15.45:0/2457025026 >> v1:172.21.15.190:6789/0 conn(0x7f024815ddd0 legacy=0x7f024815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.584+0000 7f0246ffd640 1 -- 172.21.15.45:0/2457025026 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024806aea0 con 0x7f024806a500 2023-11-28T20:35:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.585+0000 7f0246ffd640 1 --2- 172.21.15.45:0/2457025026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f024806a500 0x7f024806a960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f02480c9540 tx=0x7f023002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a9292800f6025d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.585+0000 7f02467fc640 1 -- 172.21.15.45:0/2457025026 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230031870 con 0x7f024806a500 2023-11-28T20:35:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f02467fc640 1 -- 172.21.15.45:0/2457025026 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02300319d0 con 0x7f024806a500 2023-11-28T20:35:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f02467fc640 1 -- 172.21.15.45:0/2457025026 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230031d60 con 0x7f024806a500 2023-11-28T20:35:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f024dbe3640 1 -- 172.21.15.45:0/2457025026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f024806a500 msgr2=0x7f024806a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f024dbe3640 1 --2- 172.21.15.45:0/2457025026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f024806a500 0x7f024806a960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f02480c9540 tx=0x7f023002b910 comp rx=0 tx=0).stop 2023-11-28T20:35:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f024dbe3640 1 -- 172.21.15.45:0/2457025026 shutdown_connections 2023-11-28T20:35:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f024dbe3640 1 --2- 172.21.15.45:0/2457025026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f024806a500 0x7f024806a960 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f024dbe3640 1 --2- 172.21.15.45:0/2457025026 >> v2:172.21.15.190:3300/0 conn(0x7f0248157c10 0x7f024806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f024dbe3640 1 -- 172.21.15.45:0/2457025026 >> 172.21.15.45:0/2457025026 conn(0x7f02480ced60 msgr2=0x7f02480cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f024dbe3640 1 -- 172.21.15.45:0/2457025026 shutdown_connections 2023-11-28T20:35:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.586+0000 7f024dbe3640 1 -- 172.21.15.45:0/2457025026 wait complete. 2023-11-28T20:35:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.588+0000 7f024dbe3640 1 Processor -- start 2023-11-28T20:35:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.588+0000 7f024dbe3640 1 -- start start 2023-11-28T20:35:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.589+0000 7f024dbe3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f024806a500 0x7f02480c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.589+0000 7f024dbe3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0248157c10 0x7f024815d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.589+0000 7f024dbe3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f024815ddd0 0x7f0248156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.589+0000 7f024dbe3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02480c3730 con 0x7f0248157c10 2023-11-28T20:35:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.589+0000 7f024dbe3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0248156e90 con 0x7f024815ddd0 2023-11-28T20:35:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.589+0000 7f024dbe3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02480c2e00 con 0x7f024806a500 2023-11-28T20:35:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.590+0000 7f0246ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0248157c10 0x7f024815d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.590+0000 7f0246ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0248157c10 0x7f024815d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40998/0 (socket says 172.21.15.45:40998) 2023-11-28T20:35:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.590+0000 7f0246ffd640 1 -- 172.21.15.45:0/2411092934 learned_addr learned my addr 172.21.15.45:0/2411092934 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.590+0000 7f0246ffd640 1 -- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f024806a500 msgr2=0x7f02480c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.590+0000 7f0246ffd640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f024806a500 0x7f02480c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.590+0000 7f0246ffd640 1 -- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f024815ddd0 msgr2=0x7f0248156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.590+0000 7f0246ffd640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f024815ddd0 0x7f0248156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.590+0000 7f0246ffd640 1 -- 172.21.15.45:0/2411092934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02300094d0 con 0x7f0248157c10 2023-11-28T20:35:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.590+0000 7f0246ffd640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0248157c10 0x7f024815d130 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0230000c00 tx=0x7f02300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.591+0000 7f023ffff640 1 -- 172.21.15.45:0/2411092934 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023003abe0 con 0x7f0248157c10 2023-11-28T20:35:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.591+0000 7f023ffff640 1 -- 172.21.15.45:0/2411092934 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023003ad40 con 0x7f0248157c10 2023-11-28T20:35:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.591+0000 7f023ffff640 1 -- 172.21.15.45:0/2411092934 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230044610 con 0x7f0248157c10 2023-11-28T20:35:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.591+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02480c2b50 con 0x7f0248157c10 2023-11-28T20:35:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.591+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02480c3350 con 0x7f0248157c10 2023-11-28T20:35:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.592+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02480c39a0 con 0x7f0248157c10 2023-11-28T20:35:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.593+0000 7f023ffff640 1 -- 172.21.15.45:0/2411092934 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f023004d450 con 0x7f0248157c10 2023-11-28T20:35:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.594+0000 7f023ffff640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f022806ae70 0x7f022806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.595+0000 7f023ffff640 1 -- 172.21.15.45:0/2411092934 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f023002d040 con 0x7f0248157c10 2023-11-28T20:35:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.595+0000 7f02477fe640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f022806ae70 0x7f022806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.596+0000 7f02477fe640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f022806ae70 0x7f022806d330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f02380070b0 tx=0x7f0238007040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.599+0000 7f023ffff640 1 -- 172.21.15.45:0/2411092934 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0230076bf0 con 0x7f0248157c10 2023-11-28T20:35:50.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:50.846+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f02480733d0 con 0x7f0248157c10 2023-11-28T20:35:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.016+0000 7f023ffff640 1 -- 172.21.15.45:0/2411092934 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f02300350a0 con 0x7f0248157c10 2023-11-28T20:35:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f022806ae70 msgr2=0x7f022806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f022806ae70 0x7f022806d330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f02380070b0 tx=0x7f0238007040 comp rx=0 tx=0).stop 2023-11-28T20:35:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0248157c10 msgr2=0x7f024815d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0248157c10 0x7f024815d130 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0230000c00 tx=0x7f02300046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 shutdown_connections 2023-11-28T20:35:51.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f022806ae70 0x7f022806d330 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f024815ddd0 0x7f0248156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0248157c10 0x7f024815d130 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 --2- 172.21.15.45:0/2411092934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f024806a500 0x7f02480c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 >> 172.21.15.45:0/2411092934 conn(0x7f02480ced60 msgr2=0x7f02480caad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:51.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 shutdown_connections 2023-11-28T20:35:51.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.020+0000 7f024dbe3640 1 -- 172.21.15.45:0/2411092934 wait complete. 2023-11-28T20:35:51.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T20:35:51.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.256+0000 7f5afeec9640 1 Processor -- start 2023-11-28T20:35:51.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.256+0000 7f5afeec9640 1 -- start start 2023-11-28T20:35:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.258+0000 7f5afeec9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5af80d24f0 0x7f5af80d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.258+0000 7f5afeec9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af80ce0e0 0x7f5af80ce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.258+0000 7f5afeec9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af8160e80 con 0x7f5af8160aa0 2023-11-28T20:35:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.258+0000 7f5afeec9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af80d1220 con 0x7f5af80d24f0 2023-11-28T20:35:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.258+0000 7f5afeec9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80d2e70 con 0x7f5af80ce0e0 2023-11-28T20:35:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.258+0000 7f5af7fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5af80d24f0 0x7f5af80d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.258+0000 7f5af7fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5af80d24f0 0x7f5af80d2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41030/0 (socket says 172.21.15.45:41030) 2023-11-28T20:35:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.258+0000 7f5af7fff640 1 -- 172.21.15.45:0/4101571809 learned_addr learned my addr 172.21.15.45:0/4101571809 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.258+0000 7f5afcc65640 1 --1- 172.21.15.45:0/4101571809 >> v1:172.21.15.45:6789/0 conn(0x7f5af8160aa0 0x7f5af80b6420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51628/0 (socket says 172.21.15.45:51628) 2023-11-28T20:35:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.259+0000 7f5afd466640 1 --2- 172.21.15.45:0/4101571809 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af80ce0e0 0x7f5af80ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.259+0000 7f5af7fff640 1 -- 172.21.15.45:0/4101571809 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af80ce0e0 msgr2=0x7f5af80ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.259+0000 7f5af7fff640 1 --2- 172.21.15.45:0/4101571809 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af80ce0e0 0x7f5af80ce570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.259+0000 7f5af7fff640 1 -- 172.21.15.45:0/4101571809 >> v1:172.21.15.45:6789/0 conn(0x7f5af8160aa0 legacy=0x7f5af80b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.259+0000 7f5af7fff640 1 -- 172.21.15.45:0/4101571809 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af80ceab0 con 0x7f5af80d24f0 2023-11-28T20:35:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.260+0000 7f5af7fff640 1 --2- 172.21.15.45:0/4101571809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 0x7f5af80d2930 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009870 tx=0x7f5ae402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11dff200c3ce8b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.260+0000 7f5af77fe640 1 -- 172.21.15.45:0/4101571809 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae402f830 con 0x7f5af80d24f0 2023-11-28T20:35:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.260+0000 7f5af77fe640 1 -- 172.21.15.45:0/4101571809 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae402f990 con 0x7f5af80d24f0 2023-11-28T20:35:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.260+0000 7f5af77fe640 1 -- 172.21.15.45:0/4101571809 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae402fd20 con 0x7f5af80d24f0 2023-11-28T20:35:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.261+0000 7f5afeec9640 1 -- 172.21.15.45:0/4101571809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 msgr2=0x7f5af80d2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.261+0000 7f5afeec9640 1 --2- 172.21.15.45:0/4101571809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 0x7f5af80d2930 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009870 tx=0x7f5ae402ba10 comp rx=0 tx=0).stop 2023-11-28T20:35:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.261+0000 7f5afeec9640 1 -- 172.21.15.45:0/4101571809 shutdown_connections 2023-11-28T20:35:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.261+0000 7f5afeec9640 1 --2- 172.21.15.45:0/4101571809 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af80ce0e0 0x7f5af80ce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.261+0000 7f5afeec9640 1 --2- 172.21.15.45:0/4101571809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 0x7f5af80d2930 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.261+0000 7f5afeec9640 1 -- 172.21.15.45:0/4101571809 >> 172.21.15.45:0/4101571809 conn(0x7f5af8070b20 msgr2=0x7f5af8070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.261+0000 7f5afeec9640 1 -- 172.21.15.45:0/4101571809 shutdown_connections 2023-11-28T20:35:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.261+0000 7f5afeec9640 1 -- 172.21.15.45:0/4101571809 wait complete. 2023-11-28T20:35:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.263+0000 7f5afeec9640 1 Processor -- start 2023-11-28T20:35:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.263+0000 7f5afeec9640 1 -- start start 2023-11-28T20:35:51.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.264+0000 7f5afeec9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af80ce0e0 0x7f5af81f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.264+0000 7f5afeec9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 0x7f5af80b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.264+0000 7f5afeec9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af8160aa0 0x7f5af80b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.264+0000 7f5afeec9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80c7c30 con 0x7f5af80d24f0 2023-11-28T20:35:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.264+0000 7f5afeec9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80c7f10 con 0x7f5af8160aa0 2023-11-28T20:35:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.264+0000 7f5afeec9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80c81f0 con 0x7f5af80ce0e0 2023-11-28T20:35:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5af7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 0x7f5af80b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5af7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 0x7f5af80b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41036/0 (socket says 172.21.15.45:41036) 2023-11-28T20:35:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5af7fff640 1 -- 172.21.15.45:0/1528747124 learned_addr learned my addr 172.21.15.45:0/1528747124 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:51.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5af7fff640 1 -- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af80ce0e0 msgr2=0x7f5af81f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5af7fff640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af80ce0e0 0x7f5af81f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5afd466640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af8160aa0 0x7f5af80b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5af7fff640 1 -- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af8160aa0 msgr2=0x7f5af80b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5af7fff640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af8160aa0 0x7f5af80b5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5af7fff640 1 -- 172.21.15.45:0/1528747124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae40094d0 con 0x7f5af80d24f0 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.265+0000 7f5af7fff640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 0x7f5af80b4b00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40090b0 tx=0x7f5ae40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.266+0000 7f5af57fa640 1 -- 172.21.15.45:0/1528747124 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4046070 con 0x7f5af80d24f0 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.266+0000 7f5af57fa640 1 -- 172.21.15.45:0/1528747124 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae4037b00 con 0x7f5af80d24f0 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.266+0000 7f5af57fa640 1 -- 172.21.15.45:0/1528747124 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae403b040 con 0x7f5af80d24f0 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.266+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af80c84d0 con 0x7f5af80d24f0 2023-11-28T20:35:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.266+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af80c6e90 con 0x7f5af80d24f0 2023-11-28T20:35:51.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.267+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af80733d0 con 0x7f5af80d24f0 2023-11-28T20:35:51.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.268+0000 7f5af57fa640 1 -- 172.21.15.45:0/1528747124 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f5ae4042480 con 0x7f5af80d24f0 2023-11-28T20:35:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.269+0000 7f5af57fa640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5adc06aee0 0x7f5adc06d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.269+0000 7f5af57fa640 1 -- 172.21.15.45:0/1528747124 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5ae40ac280 con 0x7f5af80d24f0 2023-11-28T20:35:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.269+0000 7f5afcc65640 1 -- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5adc06aee0 msgr2=0x7f5adc06d3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1081538426 2023-11-28T20:35:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.269+0000 7f5afcc65640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5adc06aee0 0x7f5adc06d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:35:51.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.273+0000 7f5af57fa640 1 -- 172.21.15.45:0/1528747124 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ae40771d0 con 0x7f5af80d24f0 2023-11-28T20:35:51.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.470+0000 7f5afcc65640 1 -- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5adc06aee0 msgr2=0x7f5adc06d3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1081538426 2023-11-28T20:35:51.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.470+0000 7f5afcc65640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5adc06aee0 0x7f5adc06d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:35:51.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.475+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f5af80d1000 con 0x7f5af80d24f0 2023-11-28T20:35:51.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.481+0000 7f5af57fa640 1 -- 172.21.15.45:0/1528747124 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5ae407a7f0 con 0x7f5af80d24f0 2023-11-28T20:35:51.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5adc06aee0 msgr2=0x7f5adc06d3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:35:51.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5adc06aee0 0x7f5adc06d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 msgr2=0x7f5af80b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 0x7f5af80b4b00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40090b0 tx=0x7f5ae40046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 shutdown_connections 2023-11-28T20:35:51.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f5adc06aee0 0x7f5adc06d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af8160aa0 0x7f5af80b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af80d24f0 0x7f5af80b4b00 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 --2- 172.21.15.45:0/1528747124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af80ce0e0 0x7f5af81f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 >> 172.21.15.45:0/1528747124 conn(0x7f5af8070b20 msgr2=0x7f5af80d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:51.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 shutdown_connections 2023-11-28T20:35:51.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.484+0000 7f5afeec9640 1 -- 172.21.15.45:0/1528747124 wait complete. 2023-11-28T20:35:51.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T20:35:51.561 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:51.560+0000 7f55d5a2d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:35:51.568 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:51.566+0000 7f8af102b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:35:51.614 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:51.613+0000 7f55d5a2d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:35:51.626 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:51.624+0000 7f8af102b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:35:51.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.708+0000 7f6e0326d640 1 Processor -- start 2023-11-28T20:35:51.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.709+0000 7f6e0326d640 1 -- start start 2023-11-28T20:35:51.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.710+0000 7f6e0326d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6dfc15d580 0x7f6dfc0cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.710+0000 7f6e0326d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 0x7f6dfc0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.710+0000 7f6e0326d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc0d0490 con 0x7f6dfc0b2500 2023-11-28T20:35:51.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.710+0000 7f6e0326d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dfc0d1360 con 0x7f6dfc15d580 2023-11-28T20:35:51.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.710+0000 7f6e0326d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc0d1640 con 0x7f6dfc0d1e40 2023-11-28T20:35:51.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.710+0000 7f6e0226b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6dfc15d580 0x7f6dfc0cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:51.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.711+0000 7f6e0226b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6dfc15d580 0x7f6dfc0cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41048/0 (socket says 172.21.15.45:41048) 2023-11-28T20:35:51.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.711+0000 7f6e0226b640 1 -- 172.21.15.45:0/671341010 learned_addr learned my addr 172.21.15.45:0/671341010 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:51.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.711+0000 7f6e01a6a640 1 --2- 172.21.15.45:0/671341010 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 0x7f6dfc0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:51.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.711+0000 7f6e0226b640 1 -- 172.21.15.45:0/671341010 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 msgr2=0x7f6dfc0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:51.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.711+0000 7f6e0226b640 1 --2- 172.21.15.45:0/671341010 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 0x7f6dfc0ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.711+0000 7f6e0226b640 1 -- 172.21.15.45:0/671341010 >> v1:172.21.15.190:6789/0 conn(0x7f6dfc0b2500 legacy=0x7f6dfc0b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:51.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.711+0000 7f6e0226b640 1 -- 172.21.15.45:0/671341010 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc0d1920 con 0x7f6dfc15d580 2023-11-28T20:35:51.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.712+0000 7f6e0226b640 1 --2- 172.21.15.45:0/671341010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc15d580 0x7f6dfc0cfb00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009870 tx=0x7f6df802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8da00d2fd8f37 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:51.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.712+0000 7f6e01269640 1 -- 172.21.15.45:0/671341010 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df80317c0 con 0x7f6dfc15d580 2023-11-28T20:35:51.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e01269640 1 -- 172.21.15.45:0/671341010 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8031920 con 0x7f6dfc15d580 2023-11-28T20:35:51.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e01269640 1 -- 172.21.15.45:0/671341010 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8031cb0 con 0x7f6dfc15d580 2023-11-28T20:35:51.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e0326d640 1 -- 172.21.15.45:0/671341010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc15d580 msgr2=0x7f6dfc0cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e0326d640 1 --2- 172.21.15.45:0/671341010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc15d580 0x7f6dfc0cfb00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009870 tx=0x7f6df802b860 comp rx=0 tx=0).stop 2023-11-28T20:35:51.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e0326d640 1 -- 172.21.15.45:0/671341010 shutdown_connections 2023-11-28T20:35:51.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e0326d640 1 --2- 172.21.15.45:0/671341010 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 0x7f6dfc0ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e0326d640 1 --2- 172.21.15.45:0/671341010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc15d580 0x7f6dfc0cfb00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e0326d640 1 -- 172.21.15.45:0/671341010 >> 172.21.15.45:0/671341010 conn(0x7f6dfc06a500 msgr2=0x7f6dfc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:51.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e0326d640 1 -- 172.21.15.45:0/671341010 shutdown_connections 2023-11-28T20:35:51.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.713+0000 7f6e0326d640 1 -- 172.21.15.45:0/671341010 wait complete. 2023-11-28T20:35:51.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.715+0000 7f6e0326d640 1 Processor -- start 2023-11-28T20:35:51.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.715+0000 7f6e0326d640 1 -- start start 2023-11-28T20:35:51.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.716+0000 7f6e0326d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc0b2500 0x7f6dfc1f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.716+0000 7f6e0326d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 0x7f6dfc14c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.716+0000 7f6e0326d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6dfc15d580 0x7f6dfc1602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.716+0000 7f6e0326d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc160b70 con 0x7f6dfc0b2500 2023-11-28T20:35:51.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.716+0000 7f6e0326d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc0b46e0 con 0x7f6dfc15d580 2023-11-28T20:35:51.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.716+0000 7f6e0326d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc0b3db0 con 0x7f6dfc0d1e40 2023-11-28T20:35:51.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.716+0000 7f6e0226b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc0b2500 0x7f6dfc1f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:51.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.716+0000 7f6e0226b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc0b2500 0x7f6dfc1f33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41064/0 (socket says 172.21.15.45:41064) 2023-11-28T20:35:51.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.716+0000 7f6e0226b640 1 -- 172.21.15.45:0/3548885055 learned_addr learned my addr 172.21.15.45:0/3548885055 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:51.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6e01a6a640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 0x7f6dfc14c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:51.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6e02a6c640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6dfc15d580 0x7f6dfc1602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:51.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6e0226b640 1 -- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 msgr2=0x7f6dfc14c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6e0226b640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 0x7f6dfc14c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6e0226b640 1 -- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6dfc15d580 msgr2=0x7f6dfc1602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6e0226b640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6dfc15d580 0x7f6dfc1602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6e0226b640 1 -- 172.21.15.45:0/3548885055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80094d0 con 0x7f6dfc0b2500 2023-11-28T20:35:51.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6e02a6c640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6dfc15d580 0x7f6dfc1602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:51.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6e0226b640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc0b2500 0x7f6dfc1f33a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6df802b8a0 tx=0x7f6df80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:51.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.717+0000 7f6df2ffd640 1 -- 172.21.15.45:0/3548885055 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8032860 con 0x7f6dfc0b2500 2023-11-28T20:35:51.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.718+0000 7f6df2ffd640 1 -- 172.21.15.45:0/3548885055 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df80329c0 con 0x7f6dfc0b2500 2023-11-28T20:35:51.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.718+0000 7f6df2ffd640 1 -- 172.21.15.45:0/3548885055 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8032d70 con 0x7f6dfc0b2500 2023-11-28T20:35:51.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.718+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc0b4090 con 0x7f6dfc0b2500 2023-11-28T20:35:51.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.718+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc14ca50 con 0x7f6dfc0b2500 2023-11-28T20:35:51.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.719+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc0b3ef0 con 0x7f6dfc0b2500 2023-11-28T20:35:51.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.720+0000 7f6df2ffd640 1 -- 172.21.15.45:0/3548885055 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f6df804e450 con 0x7f6dfc0b2500 2023-11-28T20:35:51.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.721+0000 7f6df2ffd640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6de006ae90 0x7f6de006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:51.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.721+0000 7f6df2ffd640 1 -- 172.21.15.45:0/3548885055 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6df80abac0 con 0x7f6dfc0b2500 2023-11-28T20:35:51.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.721+0000 7f6e01a6a640 1 -- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6de006ae90 msgr2=0x7f6de006d350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1081538426 2023-11-28T20:35:51.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.721+0000 7f6e01a6a640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6de006ae90 0x7f6de006d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:35:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.726+0000 7f6df2ffd640 1 -- 172.21.15.45:0/3548885055 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6dfc0b3ef0 con 0x7f6dfc0b2500 2023-11-28T20:35:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.918+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f6dfc0b3ef0 con 0x7f6dfc0b2500 2023-11-28T20:35:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.922+0000 7f6e01a6a640 1 -- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6de006ae90 msgr2=0x7f6de006d350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1081538426 2023-11-28T20:35:51.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.922+0000 7f6e01a6a640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6de006ae90 0x7f6de006d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:35:51.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.923+0000 7f6df2ffd640 1 -- 172.21.15.45:0/3548885055 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6dfc0b3ef0 con 0x7f6dfc0b2500 2023-11-28T20:35:51.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6de006ae90 msgr2=0x7f6de006d350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6de006ae90 0x7f6de006d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc0b2500 msgr2=0x7f6dfc1f33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc0b2500 0x7f6dfc1f33a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6df802b8a0 tx=0x7f6df80046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 shutdown_connections 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7f6de006ae90 0x7f6de006d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6dfc15d580 0x7f6dfc1602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dfc0d1e40 0x7f6dfc14c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 --2- 172.21.15.45:0/3548885055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dfc0b2500 0x7f6dfc1f33a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 >> 172.21.15.45:0/3548885055 conn(0x7f6dfc06a500 msgr2=0x7f6dfc0ca5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:51.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 shutdown_connections 2023-11-28T20:35:51.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:51.928+0000 7f6e0326d640 1 -- 172.21.15.45:0/3548885055 wait complete. 2023-11-28T20:35:51.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:51.949 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-11-28T20:35:52.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:52.013 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-11-28T20:35:52.065 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.063+0000 7f55d5a2d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:35:52.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2023-11-28T20:35:52.116 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.114+0000 7f8af102b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:35:52.117 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.115+0000 7f55d5a2d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:35:52.167 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.166+0000 7f55d5a2d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:35:52.172 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.170+0000 7f8af102b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:35:52.226 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.224+0000 7f8af102b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:35:52.227 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.226+0000 7f55d5a2d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:35:52.277 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.275+0000 7f55d5a2d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:35:52.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.277+0000 7faa7af35640 1 Processor -- start 2023-11-28T20:35:52.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.277+0000 7faa7af35640 1 -- start start 2023-11-28T20:35:52.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.279+0000 7faa7af35640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa740d0040 0x7faa740d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:52.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.279+0000 7faa7af35640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faa740b1d40 0x7faa740d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:52.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.279+0000 7faa7af35640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740b90a0 con 0x7faa740b2290 2023-11-28T20:35:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.279+0000 7faa7af35640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa740d1920 con 0x7faa740d0040 2023-11-28T20:35:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.279+0000 7faa7af35640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa740d2dc0 con 0x7faa740b1d40 2023-11-28T20:35:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.279+0000 7faa79f33640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa740d0040 0x7faa740d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.279+0000 7faa79732640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faa740b2290 0x7faa740d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51630/0 (socket says 172.21.15.45:51630) 2023-11-28T20:35:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.279+0000 7faa79f33640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa740d0040 0x7faa740d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41074/0 (socket says 172.21.15.45:41074) 2023-11-28T20:35:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.279+0000 7faa79732640 1 -- 172.21.15.45:0/1023889207 learned_addr learned my addr 172.21.15.45:0/1023889207 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:52.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.280+0000 7faa7a734640 1 --2- 172.21.15.45:0/1023889207 >> v2:172.21.15.190:3300/0 conn(0x7faa740b1d40 0x7faa740d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.280+0000 7faa79f33640 1 -- 172.21.15.45:0/1023889207 >> v2:172.21.15.190:3300/0 conn(0x7faa740b1d40 msgr2=0x7faa740d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.280+0000 7faa79f33640 1 --2- 172.21.15.45:0/1023889207 >> v2:172.21.15.190:3300/0 conn(0x7faa740b1d40 0x7faa740d3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.280+0000 7faa79f33640 1 -- 172.21.15.45:0/1023889207 >> v1:172.21.15.45:6789/0 conn(0x7faa740b2290 legacy=0x7faa740d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.280+0000 7faa79f33640 1 -- 172.21.15.45:0/1023889207 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa740d0960 con 0x7faa740d0040 2023-11-28T20:35:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.280+0000 7faa78f31640 1 -- 172.21.15.45:0/1023889207 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522669676 0 0) 0x7faa740b90a0 con 0x7faa740b2290 2023-11-28T20:35:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.281+0000 7faa79f33640 1 --2- 172.21.15.45:0/1023889207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 0x7faa740d0420 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7faa7406bf90 tx=0x7faa7002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d39b6307ae9db26 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:52.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.281+0000 7faa78f31640 1 -- 172.21.15.45:0/1023889207 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70030610 con 0x7faa740d0040 2023-11-28T20:35:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.281+0000 7faa78f31640 1 -- 172.21.15.45:0/1023889207 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa70030770 con 0x7faa740d0040 2023-11-28T20:35:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.281+0000 7faa78f31640 1 -- 172.21.15.45:0/1023889207 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70030b00 con 0x7faa740d0040 2023-11-28T20:35:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.282+0000 7faa7af35640 1 -- 172.21.15.45:0/1023889207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 msgr2=0x7faa740d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:52.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.282+0000 7faa7af35640 1 --2- 172.21.15.45:0/1023889207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 0x7faa740d0420 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7faa7406bf90 tx=0x7faa7002b960 comp rx=0 tx=0).stop 2023-11-28T20:35:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.282+0000 7faa7af35640 1 -- 172.21.15.45:0/1023889207 shutdown_connections 2023-11-28T20:35:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.282+0000 7faa7af35640 1 --2- 172.21.15.45:0/1023889207 >> v2:172.21.15.190:3300/0 conn(0x7faa740b1d40 0x7faa740d3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.282+0000 7faa7af35640 1 --2- 172.21.15.45:0/1023889207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 0x7faa740d0420 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.282+0000 7faa7af35640 1 -- 172.21.15.45:0/1023889207 >> 172.21.15.45:0/1023889207 conn(0x7faa74070f40 msgr2=0x7faa74071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.282+0000 7faa7af35640 1 -- 172.21.15.45:0/1023889207 shutdown_connections 2023-11-28T20:35:52.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.282+0000 7faa7af35640 1 -- 172.21.15.45:0/1023889207 wait complete. 2023-11-28T20:35:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.284+0000 7faa7af35640 1 Processor -- start 2023-11-28T20:35:52.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.284+0000 7faa7af35640 1 -- start start 2023-11-28T20:35:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa7af35640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa740b1d40 0x7faa741f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa7af35640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740b2290 0x7faa74159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa7af35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 0x7faa74146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa7af35640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa741586e0 con 0x7faa740d0040 2023-11-28T20:35:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa7af35640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa740cc5a0 con 0x7faa740b1d40 2023-11-28T20:35:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa7af35640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa740cb990 con 0x7faa740b2290 2023-11-28T20:35:52.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa7a734640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 0x7faa74146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa7a734640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 0x7faa74146ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41082/0 (socket says 172.21.15.45:41082) 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa7a734640 1 -- 172.21.15.45:0/1317739220 learned_addr learned my addr 172.21.15.45:0/1317739220 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.285+0000 7faa79732640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740b2290 0x7faa74159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa79f33640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa740b1d40 0x7faa741f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa7a734640 1 -- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740b2290 msgr2=0x7faa74159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa7a734640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740b2290 0x7faa74159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa7a734640 1 -- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa740b1d40 msgr2=0x7faa741f29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa7a734640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa740b1d40 0x7faa741f29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa7a734640 1 -- 172.21.15.45:0/1317739220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa70009400 con 0x7faa740d0040 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa79732640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740b2290 0x7faa74159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:35:52.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa7a734640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 0x7faa74146ca0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7faa6c007e60 tx=0x7faa6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:52.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c014420 con 0x7faa740d0040 2023-11-28T20:35:52.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c0145a0 con 0x7faa740d0040 2023-11-28T20:35:52.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa740cbc70 con 0x7faa740d0040 2023-11-28T20:35:52.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.286+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c014950 con 0x7faa740d0040 2023-11-28T20:35:52.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.287+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa740cc230 con 0x7faa740d0040 2023-11-28T20:35:52.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.288+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa740cc710 con 0x7faa740d0040 2023-11-28T20:35:52.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.289+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 6) v1 ==== 89248+0+0 (secure 0 0 0) 0x7faa6c014ab0 con 0x7faa740d0040 2023-11-28T20:35:52.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.290+0000 7faa6a7fc640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 0x7faa5806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:52.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.290+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7faa6c08a0c0 con 0x7faa740d0040 2023-11-28T20:35:52.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.290+0000 7faa79f33640 1 -- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 msgr2=0x7faa5806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1081538426 2023-11-28T20:35:52.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.290+0000 7faa79f33640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 0x7faa5806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:35:52.292 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.290+0000 7f8af102b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:35:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.294+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa6c055030 con 0x7faa740d0040 2023-11-28T20:35:52.350 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.348+0000 7f8af102b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:35:52.394 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.393+0000 7f55d5a2d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:35:52.456 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.455+0000 7f55d5a2d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:35:52.472 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.470+0000 7f8af102b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:35:52.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.491+0000 7faa79f33640 1 -- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 msgr2=0x7faa5806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1081538426 2023-11-28T20:35:52.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.491+0000 7faa79f33640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 0x7faa5806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:35:52.506 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.505+0000 7f55d5a2d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:35:52.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.515+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 --> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faa740d15b0 con 0x7faa5806aec0 2023-11-28T20:35:52.541 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.537+0000 7f8af102b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:35:52.564 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.563+0000 7f55d5a2d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:35:52.591 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.589+0000 7f8af102b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:35:52.614 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.613+0000 7f55d5a2d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:35:52.650 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.648+0000 7f8af102b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:35:52.702 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:52.700+0000 7f8af102b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:35:52.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.892+0000 7faa79f33640 1 -- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 msgr2=0x7faa5806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1081538426 2023-11-28T20:35:52.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:52.892+0000 7faa79f33640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 0x7faa5806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T20:35:52.996 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:52.994+0000 7f55d5a2d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:35:53.103 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:53.101+0000 7f8af102b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:35:53.155 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:53.153+0000 7f55d5a2d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:35:53.264 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:53.262+0000 7f55d5a2d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:35:53.271 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:53.269+0000 7f8af102b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:35:53.314 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:53.312+0000 7f55d5a2d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:35:53.385 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:53.382+0000 7f8af102b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:35:53.421 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:53.420+0000 7f55d5a2d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:35:53.437 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:53.435+0000 7f8af102b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:35:53.528 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:53.527+0000 7f55d5a2d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:35:53.549 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:53.546+0000 7f8af102b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:35:53.661 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:53.659+0000 7f8af102b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:35:53.685 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:53.684+0000 7f55d5a2d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:35:53.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:53.693+0000 7faa79f33640 1 -- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 msgr2=0x7faa5806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1081538426 2023-11-28T20:35:53.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:53.693+0000 7faa79f33640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 0x7faa5806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T20:35:53.734 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:53.732+0000 7f55d5a2d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:35:53.826 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:53.823+0000 7f8af102b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:35:53.854 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:53.852+0000 7f55d5a2d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:35:53.877 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:53.875+0000 7f8af102b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:35:54.004 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:54.001+0000 7f8af102b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:35:54.642 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:54.640+0000 7f55d5a2d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:35:54.866 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:54.863+0000 7f8af102b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:35:54.938 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:54.935+0000 7f55d5a2d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:35:54.993 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:54.991+0000 7f55d5a2d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:35:55.106 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.105+0000 7f55d5a2d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:35:55.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:55.126+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 7) v1 ==== 88959+0+0 (secure 0 0 0) 0x7faa6c055400 con 0x7faa740d0040 2023-11-28T20:35:55.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:55.127+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 msgr2=0x7faa5806d380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:35:55.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:55.127+0000 7faa6a7fc640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/1081538426,v1:172.21.15.190:6833/1081538426] conn(0x7faa5806aec0 0x7faa5806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:55.184 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:55.182+0000 7f8af102b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:35:55.248 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:55.246+0000 7f8af102b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:35:55.328 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f555451e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.328 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f555451e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.328 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f555451e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.328 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f555451e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.328 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f555451e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.328 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f5551d19640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.329 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f5551d19640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.329 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f5551d19640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.329 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f5551d19640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.329 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:35:55.326+0000 7f5551d19640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:35:55.372 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:35:55.369+0000 7f8af102b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:35:56.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.139+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 8) v1 ==== 89086+0+0 (secure 0 0 0) 0x7faa6c056aa0 con 0x7faa740d0040 2023-11-28T20:35:56.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.139+0000 7faa6a7fc640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa58035ae0 0x7faa58075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:56.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.140+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faa58035ec0 con 0x7faa58035ae0 2023-11-28T20:35:56.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.142+0000 7faa79f33640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa58035ae0 0x7faa58075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:56.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.153+0000 7faa79f33640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa58035ae0 0x7faa58075290 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faa70036000 tx=0x7faa70008030 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:56.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.338+0000 7faa6a7fc640 1 -- 172.21.15.45:0/1317739220 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7faa58035ec0 con 0x7faa58035ae0 2023-11-28T20:35:56.341 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir0 2023-11-28T20:35:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.342+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa58035ae0 msgr2=0x7faa58075290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.342+0000 7faa7af35640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa58035ae0 0x7faa58075290 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faa70036000 tx=0x7faa70008030 comp rx=0 tx=0).stop 2023-11-28T20:35:56.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.342+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 msgr2=0x7faa74146ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.342+0000 7faa7af35640 1 --2- 172.21.15.45:0/1317739220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740d0040 0x7faa74146ca0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7faa6c007e60 tx=0x7faa6c00e4b0 comp rx=0 tx=0).stop 2023-11-28T20:35:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.342+0000 7faa7a734640 1 -- 172.21.15.45:0/1317739220 reap_dead start 2023-11-28T20:35:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.342+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 shutdown_connections 2023-11-28T20:35:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.342+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 >> 172.21.15.45:0/1317739220 conn(0x7faa74070f40 msgr2=0x7faa740b6210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.342+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 shutdown_connections 2023-11-28T20:35:56.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.342+0000 7faa7af35640 1 -- 172.21.15.45:0/1317739220 wait complete. 2023-11-28T20:35:56.366 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-11-28T20:35:56.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:56.366 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-11-28T20:35:56.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2023-11-28T20:35:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.647+0000 7faa7acb4640 1 Processor -- start 2023-11-28T20:35:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.648+0000 7faa7acb4640 1 -- start start 2023-11-28T20:35:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.650+0000 7faa7acb4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa740adfa0 0x7faa740abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.650+0000 7faa7acb4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740708d0 con 0x7faa740ae480 2023-11-28T20:35:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.650+0000 7faa7acb4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740b3bb0 con 0x7faa7415d4b0 2023-11-28T20:35:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.650+0000 7faa7acb4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa740b3e90 con 0x7faa740adfa0 2023-11-28T20:35:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.650+0000 7faa794b1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa740adfa0 0x7faa740abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.650+0000 7faa794b1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa740adfa0 0x7faa740abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35780/0 (socket says 172.21.15.45:35780) 2023-11-28T20:35:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.650+0000 7faa794b1640 1 -- 172.21.15.45:0/3886620293 learned_addr learned my addr 172.21.15.45:0/3886620293 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.651+0000 7faa794b1640 1 -- 172.21.15.45:0/3886620293 >> v1:172.21.15.190:6789/0 conn(0x7faa7415d4b0 legacy=0x7faa74070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.651+0000 7faa794b1640 1 -- 172.21.15.45:0/3886620293 >> v1:172.21.15.45:6789/0 conn(0x7faa740ae480 legacy=0x7faa740717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:56.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.651+0000 7faa794b1640 1 -- 172.21.15.45:0/3886620293 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa74070280 con 0x7faa740adfa0 2023-11-28T20:35:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.652+0000 7faa794b1640 1 --2- 172.21.15.45:0/3886620293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740adfa0 0x7faa740abd70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7faa64009870 tx=0x7faa6402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7130392467c923 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.652+0000 7faa78cb0640 1 -- 172.21.15.45:0/3886620293 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64031870 con 0x7faa740adfa0 2023-11-28T20:35:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.652+0000 7faa78cb0640 1 -- 172.21.15.45:0/3886620293 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa640319d0 con 0x7faa740adfa0 2023-11-28T20:35:56.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.652+0000 7faa78cb0640 1 -- 172.21.15.45:0/3886620293 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64031d60 con 0x7faa740adfa0 2023-11-28T20:35:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.653+0000 7faa7acb4640 1 -- 172.21.15.45:0/3886620293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740adfa0 msgr2=0x7faa740abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.653+0000 7faa7acb4640 1 --2- 172.21.15.45:0/3886620293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740adfa0 0x7faa740abd70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7faa64009870 tx=0x7faa6402b910 comp rx=0 tx=0).stop 2023-11-28T20:35:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.653+0000 7faa7acb4640 1 -- 172.21.15.45:0/3886620293 shutdown_connections 2023-11-28T20:35:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.653+0000 7faa7acb4640 1 --2- 172.21.15.45:0/3886620293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740adfa0 0x7faa740abd70 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.653+0000 7faa7acb4640 1 -- 172.21.15.45:0/3886620293 >> 172.21.15.45:0/3886620293 conn(0x7faa740b1050 msgr2=0x7faa740cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.653+0000 7faa7acb4640 1 -- 172.21.15.45:0/3886620293 shutdown_connections 2023-11-28T20:35:56.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.653+0000 7faa7acb4640 1 -- 172.21.15.45:0/3886620293 wait complete. 2023-11-28T20:35:56.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.655+0000 7faa7acb4640 1 Processor -- start 2023-11-28T20:35:56.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.656+0000 7faa7acb4640 1 -- start start 2023-11-28T20:35:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.657+0000 7faa7acb4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740adfa0 0x7faa741601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.657+0000 7faa7acb4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740ae480 0x7faa74158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.657+0000 7faa7acb4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa7415d4b0 0x7faa74159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.657+0000 7faa7acb4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7409cab0 con 0x7faa740ae480 2023-11-28T20:35:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.657+0000 7faa7acb4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7409c1f0 con 0x7faa7415d4b0 2023-11-28T20:35:56.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.657+0000 7faa7acb4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa74160710 con 0x7faa740adfa0 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.657+0000 7faa794b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740ae480 0x7faa74158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.657+0000 7faa794b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740ae480 0x7faa74158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35792/0 (socket says 172.21.15.45:35792) 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.657+0000 7faa794b1640 1 -- 172.21.15.45:0/3644659579 learned_addr learned my addr 172.21.15.45:0/3644659579 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa7a4b3640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa7415d4b0 0x7faa74159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa79cb2640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740adfa0 0x7faa741601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa794b1640 1 -- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740adfa0 msgr2=0x7faa741601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa794b1640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740adfa0 0x7faa741601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa794b1640 1 -- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa7415d4b0 msgr2=0x7faa74159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa794b1640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa7415d4b0 0x7faa74159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa794b1640 1 -- 172.21.15.45:0/3644659579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa640094d0 con 0x7faa740ae480 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa7a4b3640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa7415d4b0 0x7faa74159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa794b1640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740ae480 0x7faa74158e10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7faa64000c00 tx=0x7faa640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:56.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.658+0000 7faa6a7fc640 1 -- 172.21.15.45:0/3644659579 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64031820 con 0x7faa740ae480 2023-11-28T20:35:56.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.659+0000 7faa6a7fc640 1 -- 172.21.15.45:0/3644659579 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa640443f0 con 0x7faa740ae480 2023-11-28T20:35:56.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.659+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7409c460 con 0x7faa740ae480 2023-11-28T20:35:56.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.659+0000 7faa6a7fc640 1 -- 172.21.15.45:0/3644659579 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa640447a0 con 0x7faa740ae480 2023-11-28T20:35:56.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.659+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa74150b70 con 0x7faa740ae480 2023-11-28T20:35:56.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.660+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7409d0f0 con 0x7faa740ae480 2023-11-28T20:35:56.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.661+0000 7faa6a7fc640 1 -- 172.21.15.45:0/3644659579 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 89086+0+0 (secure 0 0 0) 0x7faa6404d450 con 0x7faa740ae480 2023-11-28T20:35:56.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.662+0000 7faa6a7fc640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa5806ace0 0x7faa5806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:56.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.662+0000 7faa6a7fc640 1 -- 172.21.15.45:0/3644659579 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7faa6402d040 con 0x7faa740ae480 2023-11-28T20:35:56.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.663+0000 7faa79cb2640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa5806ace0 0x7faa5806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:56.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.664+0000 7faa79cb2640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa5806ace0 0x7faa5806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa7000a750 tx=0x7faa70008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:56.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.666+0000 7faa6a7fc640 1 -- 172.21.15.45:0/3644659579 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa64076e40 con 0x7faa740ae480 2023-11-28T20:35:56.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:56.878+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faa74071470 con 0x7faa5806ace0 2023-11-28T20:35:57.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.025+0000 7faa6a7fc640 1 -- 172.21.15.45:0/3644659579 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7faa7000a380 con 0x7faa5806ace0 2023-11-28T20:35:57.028 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir1 2023-11-28T20:35:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa5806ace0 msgr2=0x7faa5806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa5806ace0 0x7faa5806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa7000a750 tx=0x7faa70008040 comp rx=0 tx=0).stop 2023-11-28T20:35:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740ae480 msgr2=0x7faa74158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:57.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740ae480 0x7faa74158e10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7faa64000c00 tx=0x7faa640046c0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 shutdown_connections 2023-11-28T20:35:57.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7faa5806ace0 0x7faa5806d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa7415d4b0 0x7faa74159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa740ae480 0x7faa74158e10 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 --2- 172.21.15.45:0/3644659579 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa740adfa0 0x7faa741601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.028+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 >> 172.21.15.45:0/3644659579 conn(0x7faa740b1050 msgr2=0x7faa740cedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:57.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.029+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 shutdown_connections 2023-11-28T20:35:57.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.029+0000 7faa7acb4640 1 -- 172.21.15.45:0/3644659579 wait complete. 2023-11-28T20:35:57.051 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-11-28T20:35:57.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:57.052 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-11-28T20:35:57.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2023-11-28T20:35:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.343+0000 7eff1d01e640 1 Processor -- start 2023-11-28T20:35:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.343+0000 7eff1d01e640 1 -- start start 2023-11-28T20:35:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.345+0000 7eff1d01e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff180c1d30 0x7eff180ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:57.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.345+0000 7eff1d01e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7eff181653a0 0x7eff180c2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.345+0000 7eff1d01e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1815f7e0 con 0x7eff180712c0 2023-11-28T20:35:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.345+0000 7eff1d01e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff180c2650 con 0x7eff181653a0 2023-11-28T20:35:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.345+0000 7eff1d01e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff181658a0 con 0x7eff180c1d30 2023-11-28T20:35:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.345+0000 7eff0e59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7eff181653a0 0x7eff180c2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.345+0000 7eff0e59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7eff181653a0 0x7eff180c2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35794/0 (socket says 172.21.15.45:35794) 2023-11-28T20:35:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.345+0000 7eff0e59c640 1 -- 172.21.15.45:0/2483737111 learned_addr learned my addr 172.21.15.45:0/2483737111 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:57.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.345+0000 7eff1759e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7eff180712c0 0x7eff18162290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48472/0 (socket says 172.21.15.45:48472) 2023-11-28T20:35:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.346+0000 7eff16d9d640 1 --2- 172.21.15.45:0/2483737111 >> v2:172.21.15.190:3300/0 conn(0x7eff180c1d30 0x7eff180ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.346+0000 7eff0e59c640 1 -- 172.21.15.45:0/2483737111 >> v2:172.21.15.190:3300/0 conn(0x7eff180c1d30 msgr2=0x7eff180ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.346+0000 7eff0e59c640 1 --2- 172.21.15.45:0/2483737111 >> v2:172.21.15.190:3300/0 conn(0x7eff180c1d30 0x7eff180ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.346+0000 7eff0e59c640 1 -- 172.21.15.45:0/2483737111 >> v1:172.21.15.45:6789/0 conn(0x7eff180712c0 legacy=0x7eff18162290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:57.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.346+0000 7eff0e59c640 1 -- 172.21.15.45:0/2483737111 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff18165b80 con 0x7eff181653a0 2023-11-28T20:35:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.347+0000 7eff0e59c640 1 --2- 172.21.15.45:0/2483737111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 0x7eff180c2110 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7eff00009870 tx=0x7eff0002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cab00184a90388 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.347+0000 7eff1659c640 1 -- 172.21.15.45:0/2483737111 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00030860 con 0x7eff181653a0 2023-11-28T20:35:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.347+0000 7eff1659c640 1 -- 172.21.15.45:0/2483737111 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff000309c0 con 0x7eff181653a0 2023-11-28T20:35:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.347+0000 7eff1659c640 1 -- 172.21.15.45:0/2483737111 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00030d50 con 0x7eff181653a0 2023-11-28T20:35:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.348+0000 7eff1d01e640 1 -- 172.21.15.45:0/2483737111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 msgr2=0x7eff180c2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.348+0000 7eff1d01e640 1 --2- 172.21.15.45:0/2483737111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 0x7eff180c2110 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7eff00009870 tx=0x7eff0002b9a0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.348+0000 7eff1d01e640 1 -- 172.21.15.45:0/2483737111 shutdown_connections 2023-11-28T20:35:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.348+0000 7eff1d01e640 1 --2- 172.21.15.45:0/2483737111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 0x7eff180c2110 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.348+0000 7eff1d01e640 1 --2- 172.21.15.45:0/2483737111 >> v2:172.21.15.190:3300/0 conn(0x7eff180c1d30 0x7eff180ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.348+0000 7eff1d01e640 1 -- 172.21.15.45:0/2483737111 >> 172.21.15.45:0/2483737111 conn(0x7eff180ced60 msgr2=0x7eff180cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.348+0000 7eff1d01e640 1 -- 172.21.15.45:0/2483737111 shutdown_connections 2023-11-28T20:35:57.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.348+0000 7eff1d01e640 1 -- 172.21.15.45:0/2483737111 wait complete. 2023-11-28T20:35:57.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.350+0000 7eff1d01e640 1 Processor -- start 2023-11-28T20:35:57.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.350+0000 7eff1d01e640 1 -- start start 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.351+0000 7eff1d01e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff180712c0 0x7eff18154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.351+0000 7eff1d01e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff180c1d30 0x7eff180c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.351+0000 7eff1d01e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 0x7eff180c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.351+0000 7eff1d01e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180c6c90 con 0x7eff181653a0 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.351+0000 7eff1d01e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180c81d0 con 0x7eff180712c0 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.351+0000 7eff1d01e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff18154c50 con 0x7eff180c1d30 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.351+0000 7eff1759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 0x7eff180c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.351+0000 7eff1759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 0x7eff180c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35796/0 (socket says 172.21.15.45:35796) 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.352+0000 7eff1759e640 1 -- 172.21.15.45:0/1536908421 learned_addr learned my addr 172.21.15.45:0/1536908421 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:57.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.352+0000 7eff16d9d640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff180712c0 0x7eff18154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.352+0000 7eff1759e640 1 -- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff180c1d30 msgr2=0x7eff180c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.352+0000 7eff1759e640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff180c1d30 0x7eff180c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.352+0000 7eff1759e640 1 -- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff180712c0 msgr2=0x7eff18154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.352+0000 7eff1759e640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff180712c0 0x7eff18154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.352+0000 7eff1759e640 1 -- 172.21.15.45:0/1536908421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff000094d0 con 0x7eff181653a0 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.352+0000 7eff16d9d640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff180712c0 0x7eff18154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.352+0000 7eff1759e640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 0x7eff180c83f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7eff1000c3c0 tx=0x7eff10001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.353+0000 7eff0ffff640 1 -- 172.21.15.45:0/1536908421 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff100074e0 con 0x7eff181653a0 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.353+0000 7eff0ffff640 1 -- 172.21.15.45:0/1536908421 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff10007640 con 0x7eff181653a0 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.353+0000 7eff0ffff640 1 -- 172.21.15.45:0/1536908421 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10004a00 con 0x7eff181653a0 2023-11-28T20:35:57.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.353+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff180c5e30 con 0x7eff181653a0 2023-11-28T20:35:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.353+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff180c63f0 con 0x7eff181653a0 2023-11-28T20:35:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.354+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff181faa90 con 0x7eff181653a0 2023-11-28T20:35:57.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.355+0000 7eff0ffff640 1 -- 172.21.15.45:0/1536908421 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 89351+0+0 (secure 0 0 0) 0x7eff10004b60 con 0x7eff181653a0 2023-11-28T20:35:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.356+0000 7eff0ffff640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7efef806afb0 0x7efef806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:57.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.356+0000 7eff0ffff640 1 -- 172.21.15.45:0/1536908421 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7eff10089390 con 0x7eff181653a0 2023-11-28T20:35:57.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.357+0000 7eff16d9d640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7efef806afb0 0x7efef806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.358+0000 7eff16d9d640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7efef806afb0 0x7efef806d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7eff08007330 tx=0x7eff0800b040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:57.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.361+0000 7eff0ffff640 1 -- 172.21.15.45:0/1536908421 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff181faa90 con 0x7eff181653a0 2023-11-28T20:35:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.580+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7eff18161d50 con 0x7efef806afb0 2023-11-28T20:35:57.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.731+0000 7eff0ffff640 1 -- 172.21.15.45:0/1536908421 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7eff08009c00 con 0x7efef806afb0 2023-11-28T20:35:57.733 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir2 2023-11-28T20:35:57.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.734+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7efef806afb0 msgr2=0x7efef806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:57.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.734+0000 7eff1d01e640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7efef806afb0 0x7efef806d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7eff08007330 tx=0x7eff0800b040 comp rx=0 tx=0).stop 2023-11-28T20:35:57.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.734+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 msgr2=0x7eff180c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:57.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.734+0000 7eff1d01e640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 0x7eff180c83f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7eff1000c3c0 tx=0x7eff10001e70 comp rx=0 tx=0).stop 2023-11-28T20:35:57.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.735+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 shutdown_connections 2023-11-28T20:35:57.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.735+0000 7eff1d01e640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7efef806afb0 0x7efef806d470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.735+0000 7eff1d01e640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff181653a0 0x7eff180c83f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.735+0000 7eff1d01e640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff180c1d30 0x7eff180c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.735+0000 7eff1d01e640 1 --2- 172.21.15.45:0/1536908421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff180712c0 0x7eff18154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:57.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.735+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 >> 172.21.15.45:0/1536908421 conn(0x7eff180ced60 msgr2=0x7eff18162f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:57.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.735+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 shutdown_connections 2023-11-28T20:35:57.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:57.735+0000 7eff1d01e640 1 -- 172.21.15.45:0/1536908421 wait complete. 2023-11-28T20:35:57.758 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-11-28T20:35:57.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:57.758 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-11-28T20:35:57.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2023-11-28T20:35:58.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.043+0000 7f41ed79e640 1 Processor -- start 2023-11-28T20:35:58.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.044+0000 7f41ed79e640 1 -- start start 2023-11-28T20:35:58.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.045+0000 7f41ed79e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f41e815b270 0x7f41e806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.046+0000 7f41ed79e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e806d410 con 0x7f41e80b3040 2023-11-28T20:35:58.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.046+0000 7f41ed79e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e806d6f0 con 0x7f41e80b3420 2023-11-28T20:35:58.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.046+0000 7f41ed79e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e80ceaa0 con 0x7f41e815b270 2023-11-28T20:35:58.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.046+0000 7f41e7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f41e80b3040 0x7f41e815f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48484/0 (socket says 172.21.15.45:48484) 2023-11-28T20:35:58.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.046+0000 7f41e7fff640 1 -- 172.21.15.45:0/2935486821 learned_addr learned my addr 172.21.15.45:0/2935486821 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:58.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.046+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478256908 0 0) 0x7f41e806d410 con 0x7f41e80b3040 2023-11-28T20:35:58.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.046+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d4003520 con 0x7f41e80b3040 2023-11-28T20:35:58.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.046+0000 7f41dffff640 1 --2- 172.21.15.45:0/2935486821 >> v2:172.21.15.190:3300/0 conn(0x7f41e815b270 0x7f41e806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2612279907 0 0) 0x7f41d4003520 con 0x7f41e80b3040 2023-11-28T20:35:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 >> v2:172.21.15.190:3300/0 conn(0x7f41e815b270 msgr2=0x7f41e806cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 --2- 172.21.15.45:0/2935486821 >> v2:172.21.15.190:3300/0 conn(0x7f41e815b270 0x7f41e806cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 >> v1:172.21.15.190:6789/0 conn(0x7f41e80b3420 legacy=0x7f41e80ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e806d9d0 con 0x7f41e80b3040 2023-11-28T20:35:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f41d8001670 con 0x7f41e80b3040 2023-11-28T20:35:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 >> v1:172.21.15.45:6789/0 conn(0x7f41e80b3040 legacy=0x7f41e815f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d4003970 0x7f41d4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d40047c0 0x7f41d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41d40052a0 0x7f41d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d400c1c0 con 0x7f41d4003970 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d400c330 con 0x7f41d40052a0 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d400c4f0 con 0x7f41d40047c0 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.047+0000 7f41e7fff640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d4003970 0x7f41d4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.048+0000 7f41ecf9d640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41d40052a0 0x7f41d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.048+0000 7f41dffff640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d40047c0 0x7f41d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.048+0000 7f41e7fff640 1 -- 172.21.15.45:0/2935486821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d40047c0 msgr2=0x7f41d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:58.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.048+0000 7f41e7fff640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d40047c0 0x7f41d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.048+0000 7f41e7fff640 1 -- 172.21.15.45:0/2935486821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41d40052a0 msgr2=0x7f41d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.048+0000 7f41e7fff640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41d40052a0 0x7f41d400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.048+0000 7f41e7fff640 1 -- 172.21.15.45:0/2935486821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d4003520 con 0x7f41d4003970 2023-11-28T20:35:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.048+0000 7f41ecf9d640 1 -- 172.21.15.45:0/2935486821 reap_dead start 2023-11-28T20:35:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.048+0000 7f41dffff640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d40047c0 0x7f41d4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:58.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.049+0000 7f41e7fff640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d4003970 0x7f41d4003d70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f41d800c210 tx=0x7f41d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fae264385eca047 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:58.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.049+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8005ad0 con 0x7f41d4003970 2023-11-28T20:35:58.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.049+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d8005c30 con 0x7f41d4003970 2023-11-28T20:35:58.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.049+0000 7f41e77fe640 1 -- 172.21.15.45:0/2935486821 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d802c890 con 0x7f41d4003970 2023-11-28T20:35:58.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.050+0000 7f41ed79e640 1 -- 172.21.15.45:0/2935486821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d4003970 msgr2=0x7f41d4003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.050+0000 7f41ed79e640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d4003970 0x7f41d4003d70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f41d800c210 tx=0x7f41d40057f0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.050+0000 7f41ed79e640 1 -- 172.21.15.45:0/2935486821 shutdown_connections 2023-11-28T20:35:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.050+0000 7f41ed79e640 1 --2- 172.21.15.45:0/2935486821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d4003970 0x7f41d4003d70 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.050+0000 7f41ed79e640 1 -- 172.21.15.45:0/2935486821 >> 172.21.15.45:0/2935486821 conn(0x7f41e80b1050 msgr2=0x7f41e80cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.050+0000 7f41ed79e640 1 -- 172.21.15.45:0/2935486821 shutdown_connections 2023-11-28T20:35:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.050+0000 7f41ed79e640 1 -- 172.21.15.45:0/2935486821 wait complete. 2023-11-28T20:35:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.052+0000 7f41ed79e640 1 Processor -- start 2023-11-28T20:35:58.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.052+0000 7f41ed79e640 1 -- start start 2023-11-28T20:35:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.053+0000 7f41ed79e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d4003970 0x7f41e814ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.053+0000 7f41ed79e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41e8150f70 0x7f41e81f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.053+0000 7f41ed79e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80ce490 0x7f41e80ce940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.053+0000 7f41ed79e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80cee80 con 0x7f41e80ce490 2023-11-28T20:35:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.053+0000 7f41ed79e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e81f3ea0 con 0x7f41e8150f70 2023-11-28T20:35:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.053+0000 7f41ed79e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e8156290 con 0x7f41d4003970 2023-11-28T20:35:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41ecf9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80ce490 0x7f41e80ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41ecf9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80ce490 0x7f41e80ce940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35804/0 (socket says 172.21.15.45:35804) 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41ecf9d640 1 -- 172.21.15.45:0/2391684679 learned_addr learned my addr 172.21.15.45:0/2391684679 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41dffff640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41e8150f70 0x7f41e81f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41e7fff640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d4003970 0x7f41e814ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41ecf9d640 1 -- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d4003970 msgr2=0x7f41e814ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41ecf9d640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d4003970 0x7f41e814ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41ecf9d640 1 -- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41e8150f70 msgr2=0x7f41e81f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41ecf9d640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41e8150f70 0x7f41e81f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.054+0000 7f41ecf9d640 1 -- 172.21.15.45:0/2391684679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e81514a0 con 0x7f41e80ce490 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.055+0000 7f41ecf9d640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80ce490 0x7f41e80ce940 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f41e80cc840 tx=0x7f41e000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.055+0000 7f41e57fa640 1 -- 172.21.15.45:0/2391684679 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0006d60 con 0x7f41e80ce490 2023-11-28T20:35:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.055+0000 7f41e57fa640 1 -- 172.21.15.45:0/2391684679 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e0014480 con 0x7f41e80ce490 2023-11-28T20:35:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.055+0000 7f41e57fa640 1 -- 172.21.15.45:0/2391684679 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0014830 con 0x7f41e80ce490 2023-11-28T20:35:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.055+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e8151780 con 0x7f41e80ce490 2023-11-28T20:35:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.055+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e8151d40 con 0x7f41e80ce490 2023-11-28T20:35:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.056+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e809d0f0 con 0x7f41e80ce490 2023-11-28T20:35:58.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.057+0000 7f41e57fa640 1 -- 172.21.15.45:0/2391684679 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 89351+0+0 (secure 0 0 0) 0x7f41e0014990 con 0x7f41e80ce490 2023-11-28T20:35:58.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.058+0000 7f41e57fa640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f41c806afb0 0x7f41c806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.058+0000 7f41e57fa640 1 -- 172.21.15.45:0/2391684679 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f41e0089bb0 con 0x7f41e80ce490 2023-11-28T20:35:58.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.059+0000 7f41e7fff640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f41c806afb0 0x7f41c806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.060+0000 7f41e7fff640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f41c806afb0 0x7f41c806d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41d8034000 tx=0x7f41d8009030 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.062+0000 7f41e57fa640 1 -- 172.21.15.45:0/2391684679 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41e0054870 con 0x7f41e80ce490 2023-11-28T20:35:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.286+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f41e806df90 con 0x7f41c806afb0 2023-11-28T20:35:58.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.434+0000 7f41e57fa640 1 -- 172.21.15.45:0/2391684679 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f41d803d040 con 0x7f41c806afb0 2023-11-28T20:35:58.436 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir3 2023-11-28T20:35:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.436+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f41c806afb0 msgr2=0x7f41c806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f41c806afb0 0x7f41c806d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41d8034000 tx=0x7f41d8009030 comp rx=0 tx=0).stop 2023-11-28T20:35:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80ce490 msgr2=0x7f41e80ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80ce490 0x7f41e80ce940 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f41e80cc840 tx=0x7f41e000e9e0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 shutdown_connections 2023-11-28T20:35:58.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f41c806afb0 0x7f41c806d470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e80ce490 0x7f41e80ce940 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41e8150f70 0x7f41e81f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 --2- 172.21.15.45:0/2391684679 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41d4003970 0x7f41e814ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 >> 172.21.15.45:0/2391684679 conn(0x7f41e80b1050 msgr2=0x7f41e80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:58.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 shutdown_connections 2023-11-28T20:35:58.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.437+0000 7f41ed79e640 1 -- 172.21.15.45:0/2391684679 wait complete. 2023-11-28T20:35:58.458 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-11-28T20:35:58.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:58.459 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-11-28T20:35:58.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2023-11-28T20:35:58.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.740+0000 7fa9e1e44640 1 Processor -- start 2023-11-28T20:35:58.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.740+0000 7fa9e1e44640 1 -- start start 2023-11-28T20:35:58.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.742+0000 7fa9e1e44640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa9dc06ac10 0x7fa9dc06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.742+0000 7fa9e1e44640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc15ecd0 con 0x7fa9dc15e840 2023-11-28T20:35:58.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.742+0000 7fa9e1e44640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc15f290 con 0x7fa9dc15f720 2023-11-28T20:35:58.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.742+0000 7fa9e1e44640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc15fbd0 con 0x7fa9dc06ac10 2023-11-28T20:35:58.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.743+0000 7fa9d3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa9dc06ac10 0x7fa9dc06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.743+0000 7fa9db7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa9dc15e840 0x7fa9dc06a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48498/0 (socket says 172.21.15.45:48498) 2023-11-28T20:35:58.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.743+0000 7fa9db7fe640 1 -- 172.21.15.45:0/514616284 learned_addr learned my addr 172.21.15.45:0/514616284 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:35:58.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.743+0000 7fa9d3fff640 1 -- 172.21.15.45:0/514616284 >> v1:172.21.15.190:6789/0 conn(0x7fa9dc15f720 legacy=0x7fa9dc0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:58.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.743+0000 7fa9d3fff640 1 -- 172.21.15.45:0/514616284 >> v1:172.21.15.45:6789/0 conn(0x7fa9dc15e840 legacy=0x7fa9dc06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.743+0000 7fa9d3fff640 1 -- 172.21.15.45:0/514616284 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc072020 con 0x7fa9dc06ac10 2023-11-28T20:35:58.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.743+0000 7fa9daffd640 1 -- 172.21.15.45:0/514616284 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284075322 0 0) 0x7fa9dc15ecd0 con 0x7fa9dc15e840 2023-11-28T20:35:58.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.745+0000 7fa9d3fff640 1 --2- 172.21.15.45:0/514616284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc06ac10 0x7fa9dc06c350 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40097a0 tx=0x7fa9c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a1a21b36b0bc8b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:58.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.745+0000 7fa9daffd640 1 -- 172.21.15.45:0/514616284 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4031830 con 0x7fa9dc06ac10 2023-11-28T20:35:58.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.745+0000 7fa9daffd640 1 -- 172.21.15.45:0/514616284 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c4031990 con 0x7fa9dc06ac10 2023-11-28T20:35:58.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.745+0000 7fa9daffd640 1 -- 172.21.15.45:0/514616284 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4031d20 con 0x7fa9dc06ac10 2023-11-28T20:35:58.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.745+0000 7fa9e1e44640 1 -- 172.21.15.45:0/514616284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc06ac10 msgr2=0x7fa9dc06c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.745+0000 7fa9e1e44640 1 --2- 172.21.15.45:0/514616284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc06ac10 0x7fa9dc06c350 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40097a0 tx=0x7fa9c402b8d0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.746+0000 7fa9e1e44640 1 -- 172.21.15.45:0/514616284 shutdown_connections 2023-11-28T20:35:58.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.746+0000 7fa9e1e44640 1 --2- 172.21.15.45:0/514616284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc06ac10 0x7fa9dc06c350 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.746+0000 7fa9e1e44640 1 -- 172.21.15.45:0/514616284 >> 172.21.15.45:0/514616284 conn(0x7fa9dc0ced60 msgr2=0x7fa9dc0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:58.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.746+0000 7fa9e1e44640 1 -- 172.21.15.45:0/514616284 shutdown_connections 2023-11-28T20:35:58.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.746+0000 7fa9e1e44640 1 -- 172.21.15.45:0/514616284 wait complete. 2023-11-28T20:35:58.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.748+0000 7fa9e1e44640 1 Processor -- start 2023-11-28T20:35:58.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.748+0000 7fa9e1e44640 1 -- start start 2023-11-28T20:35:58.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9e1e44640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9dc06ac10 0x7fa9dc15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9e1e44640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9dc15e840 0x7fa9dc0b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9e1e44640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc15f720 0x7fa9dc0b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9e1e44640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc15dbe0 con 0x7fa9dc15f720 2023-11-28T20:35:58.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9e1e44640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc15cfd0 con 0x7fa9dc06ac10 2023-11-28T20:35:58.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9e1e44640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc15d2b0 con 0x7fa9dc15e840 2023-11-28T20:35:58.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc15f720 0x7fa9dc0b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc15f720 0x7fa9dc0b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35818/0 (socket says 172.21.15.45:35818) 2023-11-28T20:35:58.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9dbfff640 1 -- 172.21.15.45:0/60177252 learned_addr learned my addr 172.21.15.45:0/60177252 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:58.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9db7fe640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9dc06ac10 0x7fa9dc15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9d3fff640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9dc15e840 0x7fa9dc0b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9dbfff640 1 -- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9dc15e840 msgr2=0x7fa9dc0b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9dbfff640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9dc15e840 0x7fa9dc0b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9dbfff640 1 -- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9dc06ac10 msgr2=0x7fa9dc15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9dbfff640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9dc06ac10 0x7fa9dc15c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9dbfff640 1 -- 172.21.15.45:0/60177252 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c4009400 con 0x7fa9dc15f720 2023-11-28T20:35:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.749+0000 7fa9d3fff640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9dc15e840 0x7fa9dc0b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:35:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.750+0000 7fa9dbfff640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc15f720 0x7fa9dc0b40b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400b9e0 tx=0x7fa9d400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:58.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.750+0000 7fa9d8ff9640 1 -- 172.21.15.45:0/60177252 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4003b70 con 0x7fa9dc15f720 2023-11-28T20:35:58.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.750+0000 7fa9d8ff9640 1 -- 172.21.15.45:0/60177252 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d4003cd0 con 0x7fa9dc15f720 2023-11-28T20:35:58.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.750+0000 7fa9d8ff9640 1 -- 172.21.15.45:0/60177252 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d400cae0 con 0x7fa9dc15f720 2023-11-28T20:35:58.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.750+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc15d590 con 0x7fa9dc15f720 2023-11-28T20:35:58.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.750+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc15cc10 con 0x7fa9dc15f720 2023-11-28T20:35:58.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.752+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc15cda0 con 0x7fa9dc15f720 2023-11-28T20:35:58.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.753+0000 7fa9d8ff9640 1 -- 172.21.15.45:0/60177252 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 89351+0+0 (secure 0 0 0) 0x7fa9d400cc40 con 0x7fa9dc15f720 2023-11-28T20:35:58.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.754+0000 7fa9d8ff9640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa9bc06afb0 0x7fa9bc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:58.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.754+0000 7fa9d8ff9640 1 -- 172.21.15.45:0/60177252 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fa9d4089820 con 0x7fa9dc15f720 2023-11-28T20:35:58.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.754+0000 7fa9db7fe640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa9bc06afb0 0x7fa9bc06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:58.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.755+0000 7fa9db7fe640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa9bc06afb0 0x7fa9bc06d470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc000f60 tx=0x7fa9cc007b50 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:58.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.758+0000 7fa9d8ff9640 1 -- 172.21.15.45:0/60177252 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9d4054450 con 0x7fa9dc15f720 2023-11-28T20:35:58.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:58.973+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9dc0ce120 con 0x7fa9bc06afb0 2023-11-28T20:35:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.134+0000 7fa9d8ff9640 1 -- 172.21.15.45:0/60177252 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa9dc0ce120 con 0x7fa9bc06afb0 2023-11-28T20:35:59.137 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir4 2023-11-28T20:35:59.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.137+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa9bc06afb0 msgr2=0x7fa9bc06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:59.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.137+0000 7fa9e1e44640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa9bc06afb0 0x7fa9bc06d470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc000f60 tx=0x7fa9cc007b50 comp rx=0 tx=0).stop 2023-11-28T20:35:59.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.137+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc15f720 msgr2=0x7fa9dc0b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:59.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.137+0000 7fa9e1e44640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc15f720 0x7fa9dc0b40b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400b9e0 tx=0x7fa9d400bea0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.138+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 shutdown_connections 2023-11-28T20:35:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.138+0000 7fa9e1e44640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa9bc06afb0 0x7fa9bc06d470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.138+0000 7fa9e1e44640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9dc15f720 0x7fa9dc0b40b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.138+0000 7fa9e1e44640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9dc15e840 0x7fa9dc0b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.138+0000 7fa9e1e44640 1 --2- 172.21.15.45:0/60177252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9dc06ac10 0x7fa9dc15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.138+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 >> 172.21.15.45:0/60177252 conn(0x7fa9dc0ced60 msgr2=0x7fa9dc0af030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.138+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 shutdown_connections 2023-11-28T20:35:59.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.138+0000 7fa9e1e44640 1 -- 172.21.15.45:0/60177252 wait complete. 2023-11-28T20:35:59.162 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-11-28T20:35:59.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:59.162 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-11-28T20:35:59.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2023-11-28T20:35:59.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.454+0000 7f14314c2640 1 Processor -- start 2023-11-28T20:35:59.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.454+0000 7f14314c2640 1 -- start start 2023-11-28T20:35:59.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.456+0000 7f14314c2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f142c06a500 0x7f142c06a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:59.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.456+0000 7f14314c2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142c06ca60 con 0x7f142c15ddd0 2023-11-28T20:35:59.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.456+0000 7f14314c2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142c06aea0 con 0x7f142c157c10 2023-11-28T20:35:59.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.456+0000 7f14314c2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142c15e930 con 0x7f142c06a500 2023-11-28T20:35:59.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.456+0000 7f1423fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f142c06a500 0x7f142c06a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:59.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.456+0000 7f1423fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f142c06a500 0x7f142c06a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35830/0 (socket says 172.21.15.45:35830) 2023-11-28T20:35:59.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.456+0000 7f1423fff640 1 -- 172.21.15.45:0/1190495578 learned_addr learned my addr 172.21.15.45:0/1190495578 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:59.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.456+0000 7f142b7fe640 1 --1- 172.21.15.45:0/1190495578 >> v1:172.21.15.45:6789/0 conn(0x7f142c15ddd0 0x7f142c15e220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48512/0 (socket says 172.21.15.45:48512) 2023-11-28T20:35:59.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.457+0000 7f1423fff640 1 -- 172.21.15.45:0/1190495578 >> v1:172.21.15.190:6789/0 conn(0x7f142c157c10 legacy=0x7f142c06da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:35:59.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.457+0000 7f1423fff640 1 -- 172.21.15.45:0/1190495578 >> v1:172.21.15.45:6789/0 conn(0x7f142c15ddd0 legacy=0x7f142c15e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:59.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.457+0000 7f1423fff640 1 -- 172.21.15.45:0/1190495578 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142c15eb40 con 0x7f142c06a500 2023-11-28T20:35:59.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.457+0000 7f142a7fc640 1 -- 172.21.15.45:0/1190495578 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336879200 0 0) 0x7f142c06ca60 con 0x7f142c15ddd0 2023-11-28T20:35:59.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.458+0000 7f1423fff640 1 --2- 172.21.15.45:0/1190495578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c06a500 0x7f142c06a960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1414009870 tx=0x7f141402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa956d44e64463e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:59.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.458+0000 7f142a7fc640 1 -- 172.21.15.45:0/1190495578 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414030860 con 0x7f142c06a500 2023-11-28T20:35:59.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.458+0000 7f142a7fc640 1 -- 172.21.15.45:0/1190495578 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14140309c0 con 0x7f142c06a500 2023-11-28T20:35:59.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.458+0000 7f142a7fc640 1 -- 172.21.15.45:0/1190495578 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414030d50 con 0x7f142c06a500 2023-11-28T20:35:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.459+0000 7f14314c2640 1 -- 172.21.15.45:0/1190495578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c06a500 msgr2=0x7f142c06a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.459+0000 7f14314c2640 1 --2- 172.21.15.45:0/1190495578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c06a500 0x7f142c06a960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1414009870 tx=0x7f141402b9a0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.459+0000 7f14314c2640 1 -- 172.21.15.45:0/1190495578 shutdown_connections 2023-11-28T20:35:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.459+0000 7f14314c2640 1 --2- 172.21.15.45:0/1190495578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c06a500 0x7f142c06a960 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.459+0000 7f14314c2640 1 -- 172.21.15.45:0/1190495578 >> 172.21.15.45:0/1190495578 conn(0x7f142c0ced60 msgr2=0x7f142c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.459+0000 7f14314c2640 1 -- 172.21.15.45:0/1190495578 shutdown_connections 2023-11-28T20:35:59.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.459+0000 7f14314c2640 1 -- 172.21.15.45:0/1190495578 wait complete. 2023-11-28T20:35:59.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.461+0000 7f14314c2640 1 Processor -- start 2023-11-28T20:35:59.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.461+0000 7f14314c2640 1 -- start start 2023-11-28T20:35:59.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.462+0000 7f14314c2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f142c06a500 0x7f142c0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:59.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.462+0000 7f14314c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f142c157c10 0x7f142c0c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:59.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.462+0000 7f14314c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c15ddd0 0x7f142c15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.462+0000 7f14314c2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0c7220 con 0x7f142c15ddd0 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.462+0000 7f14314c2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0c7500 con 0x7f142c06a500 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.462+0000 7f14314c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0c77e0 con 0x7f142c157c10 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.462+0000 7f142b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c15ddd0 0x7f142c15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.462+0000 7f142b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c15ddd0 0x7f142c15c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35834/0 (socket says 172.21.15.45:35834) 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.462+0000 7f142b7fe640 1 -- 172.21.15.45:0/2245967678 learned_addr learned my addr 172.21.15.45:0/2245967678 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.463+0000 7f142b7fe640 1 -- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f142c157c10 msgr2=0x7f142c0c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.463+0000 7f142b7fe640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f142c157c10 0x7f142c0c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.463+0000 7f142b7fe640 1 -- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f142c06a500 msgr2=0x7f142c0c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:59.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.463+0000 7f142affd640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f142c06a500 0x7f142c0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:59.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.463+0000 7f142b7fe640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f142c06a500 0x7f142c0c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.463+0000 7f142b7fe640 1 -- 172.21.15.45:0/2245967678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14140094d0 con 0x7f142c15ddd0 2023-11-28T20:35:59.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.463+0000 7f142b7fe640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c15ddd0 0x7f142c15c7a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f142400c780 tx=0x7f142400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:59.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.463+0000 7f14237fe640 1 -- 172.21.15.45:0/2245967678 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400ce30 con 0x7f142c15ddd0 2023-11-28T20:35:59.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.464+0000 7f14237fe640 1 -- 172.21.15.45:0/2245967678 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142400a640 con 0x7f142c15ddd0 2023-11-28T20:35:59.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.464+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c0c7ac0 con 0x7f142c15ddd0 2023-11-28T20:35:59.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.464+0000 7f14237fe640 1 -- 172.21.15.45:0/2245967678 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400aa70 con 0x7f142c15ddd0 2023-11-28T20:35:59.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.464+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142c0c82c0 con 0x7f142c15ddd0 2023-11-28T20:35:59.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.465+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142c0c7fe0 con 0x7f142c15ddd0 2023-11-28T20:35:59.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.466+0000 7f14237fe640 1 -- 172.21.15.45:0/2245967678 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 89351+0+0 (secure 0 0 0) 0x7f142400abd0 con 0x7f142c15ddd0 2023-11-28T20:35:59.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.467+0000 7f14237fe640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f140c06afb0 0x7f140c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:35:59.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.467+0000 7f14237fe640 1 -- 172.21.15.45:0/2245967678 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f1424088f70 con 0x7f142c15ddd0 2023-11-28T20:35:59.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.468+0000 7f142affd640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f140c06afb0 0x7f140c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:35:59.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.469+0000 7f142affd640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f140c06afb0 0x7f140c06d470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f141c003d90 tx=0x7f141c009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:35:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.472+0000 7f14237fe640 1 -- 172.21.15.45:0/2245967678 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14240bbe90 con 0x7f142c15ddd0 2023-11-28T20:35:59.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.689+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f142c15e7c0 con 0x7f140c06afb0 2023-11-28T20:35:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.846+0000 7f14237fe640 1 -- 172.21.15.45:0/2245967678 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f142c15e7c0 con 0x7f140c06afb0 2023-11-28T20:35:59.848 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir5 2023-11-28T20:35:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.849+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f140c06afb0 msgr2=0x7f140c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f140c06afb0 0x7f140c06d470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f141c003d90 tx=0x7f141c009040 comp rx=0 tx=0).stop 2023-11-28T20:35:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c15ddd0 msgr2=0x7f142c15c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:35:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c15ddd0 0x7f142c15c7a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f142400c780 tx=0x7f142400cc40 comp rx=0 tx=0).stop 2023-11-28T20:35:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 shutdown_connections 2023-11-28T20:35:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f140c06afb0 0x7f140c06d470 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142c15ddd0 0x7f142c15c7a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f142c157c10 0x7f142c0c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 --2- 172.21.15.45:0/2245967678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f142c06a500 0x7f142c0c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:35:59.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 >> 172.21.15.45:0/2245967678 conn(0x7f142c0ced60 msgr2=0x7f142c06da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:35:59.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 shutdown_connections 2023-11-28T20:35:59.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:35:59.850+0000 7f14314c2640 1 -- 172.21.15.45:0/2245967678 wait complete. 2023-11-28T20:35:59.874 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-11-28T20:35:59.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:35:59.874 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-11-28T20:35:59.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2023-11-28T20:36:00.048 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:36:00.047+0000 7f5543cfd640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:36:00.048 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:36:00.047+0000 7f5543cfd640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:36:00.048 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:36:00.047+0000 7f5543cfd640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:36:00.048 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:36:00.047+0000 7f5543cfd640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:36:00.048 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:36:00.047+0000 7f5543cfd640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:36:00.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.171+0000 7f39a1da5640 1 Processor -- start 2023-11-28T20:36:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.171+0000 7f39a1da5640 1 -- start start 2023-11-28T20:36:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.173+0000 7f39a1da5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f399c0b39d0 0x7f399c0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.173+0000 7f39a1da5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 0x7f399c0cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.173+0000 7f39a1da5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c0724d0 con 0x7f399c0b3db0 2023-11-28T20:36:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.173+0000 7f39a1da5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f399c071d20 con 0x7f399c0b39d0 2023-11-28T20:36:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.173+0000 7f39a1da5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0cc000 con 0x7f399c0cb630 2023-11-28T20:36:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.173+0000 7f39a0da3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f399c0b39d0 0x7f399c0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:00.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.173+0000 7f399bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f399c0b3db0 0x7f399c06bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48516/0 (socket says 172.21.15.45:48516) 2023-11-28T20:36:00.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.173+0000 7f399bfff640 1 -- 172.21.15.45:0/2486657944 learned_addr learned my addr 172.21.15.45:0/2486657944 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:36:00.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.174+0000 7f39a15a4640 1 --2- 172.21.15.45:0/2486657944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 0x7f399c0cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.174+0000 7f399b7fe640 1 -- 172.21.15.45:0/2486657944 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758366346 0 0) 0x7f399c0724d0 con 0x7f399c0b3db0 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.174+0000 7f399b7fe640 1 -- 172.21.15.45:0/2486657944 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f398c003520 con 0x7f399c0b3db0 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.174+0000 7f39a0da3640 1 -- 172.21.15.45:0/2486657944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 msgr2=0x7f399c0cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.174+0000 7f39a0da3640 1 --2- 172.21.15.45:0/2486657944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 0x7f399c0cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.174+0000 7f39a0da3640 1 -- 172.21.15.45:0/2486657944 >> v1:172.21.15.45:6789/0 conn(0x7f399c0b3db0 legacy=0x7f399c06bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.174+0000 7f39a0da3640 1 -- 172.21.15.45:0/2486657944 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c06c670 con 0x7f399c0b39d0 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.175+0000 7f39a0da3640 1 --2- 172.21.15.45:0/2486657944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b39d0 0x7f399c0717e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f39900097a0 tx=0x7f399002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6073709673fcf889 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f399b7fe640 1 -- 172.21.15.45:0/2486657944 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399002e070 con 0x7f399c0b39d0 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f399b7fe640 1 -- 172.21.15.45:0/2486657944 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39900315a0 con 0x7f399c0b39d0 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f399b7fe640 1 -- 172.21.15.45:0/2486657944 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990031910 con 0x7f399c0b39d0 2023-11-28T20:36:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f39a1da5640 1 -- 172.21.15.45:0/2486657944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b39d0 msgr2=0x7f399c0717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f39a1da5640 1 --2- 172.21.15.45:0/2486657944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b39d0 0x7f399c0717e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f39900097a0 tx=0x7f399002b840 comp rx=0 tx=0).stop 2023-11-28T20:36:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f39a1da5640 1 -- 172.21.15.45:0/2486657944 shutdown_connections 2023-11-28T20:36:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f39a1da5640 1 --2- 172.21.15.45:0/2486657944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 0x7f399c0cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f39a1da5640 1 --2- 172.21.15.45:0/2486657944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b39d0 0x7f399c0717e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f39a1da5640 1 -- 172.21.15.45:0/2486657944 >> 172.21.15.45:0/2486657944 conn(0x7f399c0b1050 msgr2=0x7f399c0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.176+0000 7f39a1da5640 1 -- 172.21.15.45:0/2486657944 shutdown_connections 2023-11-28T20:36:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.177+0000 7f39a1da5640 1 -- 172.21.15.45:0/2486657944 wait complete. 2023-11-28T20:36:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.178+0000 7f39a1da5640 1 Processor -- start 2023-11-28T20:36:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.179+0000 7f39a1da5640 1 -- start start 2023-11-28T20:36:00.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f39a1da5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0b39d0 0x7f399c15aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f39a1da5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b3db0 0x7f399c1646d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f39a1da5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 0x7f399c15c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f39a1da5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c164c10 con 0x7f399c0b3db0 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f39a1da5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c15c850 con 0x7f399c0b39d0 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f39a1da5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399c15cb10 con 0x7f399c0cb630 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f399bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b3db0 0x7f399c1646d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f399bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b3db0 0x7f399c1646d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35860/0 (socket says 172.21.15.45:35860) 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f399bfff640 1 -- 172.21.15.45:0/878118241 learned_addr learned my addr 172.21.15.45:0/878118241 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f39a0da3640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0b39d0 0x7f399c15aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.180+0000 7f39a15a4640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 0x7f399c15c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f399bfff640 1 -- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 msgr2=0x7f399c15c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f399bfff640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 0x7f399c15c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f399bfff640 1 -- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0b39d0 msgr2=0x7f399c15aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f399bfff640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0b39d0 0x7f399c15aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f399bfff640 1 -- 172.21.15.45:0/878118241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3990009400 con 0x7f399c0b3db0 2023-11-28T20:36:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f399bfff640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b3db0 0x7f399c1646d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f398800c500 tx=0x7f398800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f39997fa640 1 -- 172.21.15.45:0/878118241 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988005ce0 con 0x7f399c0b3db0 2023-11-28T20:36:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f39997fa640 1 -- 172.21.15.45:0/878118241 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3988005e40 con 0x7f399c0b3db0 2023-11-28T20:36:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399c1f7c40 con 0x7f399c0b3db0 2023-11-28T20:36:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.181+0000 7f39997fa640 1 -- 172.21.15.45:0/878118241 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988007330 con 0x7f399c0b3db0 2023-11-28T20:36:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.182+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399c1f8200 con 0x7f399c0b3db0 2023-11-28T20:36:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.183+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399c1f83f0 con 0x7f399c0b3db0 2023-11-28T20:36:00.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.184+0000 7f39997fa640 1 -- 172.21.15.45:0/878118241 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f3988007550 con 0x7f399c0b3db0 2023-11-28T20:36:00.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.185+0000 7f39997fa640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f398006b040 0x7f398006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:00.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.186+0000 7f39997fa640 1 -- 172.21.15.45:0/878118241 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f3988089280 con 0x7f399c0b3db0 2023-11-28T20:36:00.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.186+0000 7f39a0da3640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f398006b040 0x7f398006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.187+0000 7f39a0da3640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f398006b040 0x7f398006d500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3990036000 tx=0x7f3990008030 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:00.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.190+0000 7f39997fa640 1 -- 172.21.15.45:0/878118241 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3988053f20 con 0x7f399c0b3db0 2023-11-28T20:36:00.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.436+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f399c0cecb0 con 0x7f398006b040 2023-11-28T20:36:00.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.854+0000 7f39997fa640 1 -- 172.21.15.45:0/878118241 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f399c0cecb0 con 0x7f398006b040 2023-11-28T20:36:00.857 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir6 2023-11-28T20:36:00.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f398006b040 msgr2=0x7f398006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:00.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f398006b040 0x7f398006d500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3990036000 tx=0x7f3990008030 comp rx=0 tx=0).stop 2023-11-28T20:36:00.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b3db0 msgr2=0x7f399c1646d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:00.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b3db0 0x7f399c1646d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f398800c500 tx=0x7f398800c9c0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 shutdown_connections 2023-11-28T20:36:00.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f398006b040 0x7f398006d500 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c0cb630 0x7f399c15c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f399c0b3db0 0x7f399c1646d0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 --2- 172.21.15.45:0/878118241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0b39d0 0x7f399c15aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:00.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 >> 172.21.15.45:0/878118241 conn(0x7f399c0b1050 msgr2=0x7f399c06bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:00.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 shutdown_connections 2023-11-28T20:36:00.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:00.858+0000 7f39a1da5640 1 -- 172.21.15.45:0/878118241 wait complete. 2023-11-28T20:36:00.886 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-11-28T20:36:00.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:36:00.887 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-11-28T20:36:00.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2023-11-28T20:36:01.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.165+0000 7f0c4759e640 1 Processor -- start 2023-11-28T20:36:01.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.165+0000 7f0c4759e640 1 -- start start 2023-11-28T20:36:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.167+0000 7f0c4759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c48099a40 0x7f0c480d1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.167+0000 7f0c4759e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0c4806ea30 0x7f0c48099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.167+0000 7f0c4759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0c48160e20 0x7f0c481612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.167+0000 7f0c4759e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4806ee90 con 0x7f0c48160e20 2023-11-28T20:36:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.167+0000 7f0c4759e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4815d760 con 0x7f0c4806ea30 2023-11-28T20:36:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.167+0000 7f0c4759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4806fcb0 con 0x7f0c48099a40 2023-11-28T20:36:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.167+0000 7f0c46d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0c48160e20 0x7f0c481612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.168+0000 7f0c46d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0c48160e20 0x7f0c481612b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35868/0 (socket says 172.21.15.45:35868) 2023-11-28T20:36:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.168+0000 7f0c46d9d640 1 -- 172.21.15.45:0/1826286411 learned_addr learned my addr 172.21.15.45:0/1826286411 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:01.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.168+0000 7f0c4659c640 1 --2- 172.21.15.45:0/1826286411 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c48099a40 0x7f0c480d1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.168+0000 7f0c45d9b640 1 --2- 172.21.15.45:0/1826286411 >> v2:172.21.15.190:3300/0 conn(0x7f0c4806ea30 0x7f0c48099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.168+0000 7f0c46d9d640 1 -- 172.21.15.45:0/1826286411 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c48099a40 msgr2=0x7f0c480d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.168+0000 7f0c46d9d640 1 --2- 172.21.15.45:0/1826286411 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c48099a40 0x7f0c480d1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.168+0000 7f0c46d9d640 1 -- 172.21.15.45:0/1826286411 >> v2:172.21.15.190:3300/0 conn(0x7f0c4806ea30 msgr2=0x7f0c48099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.168+0000 7f0c46d9d640 1 --2- 172.21.15.45:0/1826286411 >> v2:172.21.15.190:3300/0 conn(0x7f0c4806ea30 0x7f0c48099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.168+0000 7f0c46d9d640 1 -- 172.21.15.45:0/1826286411 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4809a360 con 0x7f0c48160e20 2023-11-28T20:36:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.169+0000 7f0c46d9d640 1 --2- 172.21.15.45:0/1826286411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 0x7f0c481612b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009e50 tx=0x7f0c4002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9a799cf0e43607 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4559a640 1 -- 172.21.15.45:0/1826286411 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4002ca80 con 0x7f0c48160e20 2023-11-28T20:36:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4559a640 1 -- 172.21.15.45:0/1826286411 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4002cbe0 con 0x7f0c48160e20 2023-11-28T20:36:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4559a640 1 -- 172.21.15.45:0/1826286411 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c400343f0 con 0x7f0c48160e20 2023-11-28T20:36:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4759e640 1 -- 172.21.15.45:0/1826286411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 msgr2=0x7f0c481612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4759e640 1 --2- 172.21.15.45:0/1826286411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 0x7f0c481612b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009e50 tx=0x7f0c4002bee0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4759e640 1 -- 172.21.15.45:0/1826286411 shutdown_connections 2023-11-28T20:36:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4759e640 1 --2- 172.21.15.45:0/1826286411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 0x7f0c481612b0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4759e640 1 --2- 172.21.15.45:0/1826286411 >> v2:172.21.15.190:3300/0 conn(0x7f0c4806ea30 0x7f0c48099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4759e640 1 --2- 172.21.15.45:0/1826286411 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c48099a40 0x7f0c480d1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4759e640 1 -- 172.21.15.45:0/1826286411 >> 172.21.15.45:0/1826286411 conn(0x7f0c480d0f10 msgr2=0x7f0c480d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:01.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4759e640 1 -- 172.21.15.45:0/1826286411 shutdown_connections 2023-11-28T20:36:01.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.170+0000 7f0c4759e640 1 -- 172.21.15.45:0/1826286411 wait complete. 2023-11-28T20:36:01.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.172+0000 7f0c4759e640 1 Processor -- start 2023-11-28T20:36:01.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.172+0000 7f0c4759e640 1 -- start start 2023-11-28T20:36:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.173+0000 7f0c4759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c4806ea30 0x7f0c480b6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.173+0000 7f0c4759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c48099a40 0x7f0c480b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.173+0000 7f0c4759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 0x7f0c4815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.173+0000 7f0c4759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4815f7c0 con 0x7f0c48160e20 2023-11-28T20:36:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.173+0000 7f0c4759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4815ec60 con 0x7f0c48099a40 2023-11-28T20:36:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.173+0000 7f0c4759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c480b6940 con 0x7f0c4806ea30 2023-11-28T20:36:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c46d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 0x7f0c4815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c46d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 0x7f0c4815fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35882/0 (socket says 172.21.15.45:35882) 2023-11-28T20:36:01.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c46d9d640 1 -- 172.21.15.45:0/3402007261 learned_addr learned my addr 172.21.15.45:0/3402007261 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:01.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c4659c640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c4806ea30 0x7f0c480b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c45d9b640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c48099a40 0x7f0c480b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c46d9d640 1 -- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c4806ea30 msgr2=0x7f0c480b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:01.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c46d9d640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c4806ea30 0x7f0c480b6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c46d9d640 1 -- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c48099a40 msgr2=0x7f0c480b1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:01.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c46d9d640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c48099a40 0x7f0c480b1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c46d9d640 1 -- 172.21.15.45:0/3402007261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c40009b30 con 0x7f0c48160e20 2023-11-28T20:36:01.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c4659c640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c4806ea30 0x7f0c480b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:01.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.174+0000 7f0c46d9d640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 0x7f0c4815fbe0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0c40036040 tx=0x7f0c40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:01.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.175+0000 7f0c36ffd640 1 -- 172.21.15.45:0/3402007261 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c40040670 con 0x7f0c48160e20 2023-11-28T20:36:01.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.175+0000 7f0c36ffd640 1 -- 172.21.15.45:0/3402007261 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c400407d0 con 0x7f0c48160e20 2023-11-28T20:36:01.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.175+0000 7f0c36ffd640 1 -- 172.21.15.45:0/3402007261 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c40040b80 con 0x7f0c48160e20 2023-11-28T20:36:01.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.175+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c480b6be0 con 0x7f0c48160e20 2023-11-28T20:36:01.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.175+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4815f3e0 con 0x7f0c48160e20 2023-11-28T20:36:01.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.176+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4815f900 con 0x7f0c48160e20 2023-11-28T20:36:01.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.177+0000 7f0c36ffd640 1 -- 172.21.15.45:0/3402007261 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f0c4002cd50 con 0x7f0c48160e20 2023-11-28T20:36:01.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.178+0000 7f0c36ffd640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f0c2806b040 0x7f0c2806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.178+0000 7f0c36ffd640 1 -- 172.21.15.45:0/3402007261 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f0c400acc40 con 0x7f0c48160e20 2023-11-28T20:36:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.179+0000 7f0c4659c640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f0c2806b040 0x7f0c2806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.180+0000 7f0c4659c640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f0c2806b040 0x7f0c2806d500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c38006fd0 tx=0x7f0c38008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:01.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.183+0000 7f0c36ffd640 1 -- 172.21.15.45:0/3402007261 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c40077940 con 0x7f0c48160e20 2023-11-28T20:36:01.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.402+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c4806f660 con 0x7f0c2806b040 2023-11-28T20:36:01.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.566+0000 7f0c36ffd640 1 -- 172.21.15.45:0/3402007261 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0c380095d0 con 0x7f0c2806b040 2023-11-28T20:36:01.568 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir7 2023-11-28T20:36:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.568+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f0c2806b040 msgr2=0x7f0c2806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.568+0000 7f0c4759e640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f0c2806b040 0x7f0c2806d500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c38006fd0 tx=0x7f0c38008040 comp rx=0 tx=0).stop 2023-11-28T20:36:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.568+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 msgr2=0x7f0c4815fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.568+0000 7f0c4759e640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 0x7f0c4815fbe0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0c40036040 tx=0x7f0c40004a40 comp rx=0 tx=0).stop 2023-11-28T20:36:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.569+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 shutdown_connections 2023-11-28T20:36:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.569+0000 7f0c4759e640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f0c2806b040 0x7f0c2806d500 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.569+0000 7f0c4759e640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c48160e20 0x7f0c4815fbe0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.569+0000 7f0c4759e640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c48099a40 0x7f0c480b1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.569+0000 7f0c4759e640 1 --2- 172.21.15.45:0/3402007261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c4806ea30 0x7f0c480b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.569+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 >> 172.21.15.45:0/3402007261 conn(0x7f0c480d0f10 msgr2=0x7f0c480ce1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:01.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.569+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 shutdown_connections 2023-11-28T20:36:01.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.569+0000 7f0c4759e640 1 -- 172.21.15.45:0/3402007261 wait complete. 2023-11-28T20:36:01.591 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-11-28T20:36:01.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:36:01.591 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-11-28T20:36:01.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2023-11-28T20:36:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.871+0000 7fa20a3c9640 1 Processor -- start 2023-11-28T20:36:01.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.871+0000 7fa20a3c9640 1 -- start start 2023-11-28T20:36:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.873+0000 7fa20a3c9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa2040b3040 0x7fa20406cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.873+0000 7fa20a3c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa20415b270 0x7fa20415f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.873+0000 7fa20a3c9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa2040b3420 0x7fa2040b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.873+0000 7fa20a3c9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2040ceaa0 con 0x7fa2040b3420 2023-11-28T20:36:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.873+0000 7fa20a3c9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa20415faf0 con 0x7fa2040b3040 2023-11-28T20:36:01.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.873+0000 7fa20a3c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20406d410 con 0x7fa20415b270 2023-11-28T20:36:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.873+0000 7fa209bc8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa2040b3420 0x7fa2040b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.873+0000 7fa209bc8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa2040b3420 0x7fa2040b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35892/0 (socket says 172.21.15.45:35892) 2023-11-28T20:36:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.873+0000 7fa209bc8640 1 -- 172.21.15.45:0/2615266578 learned_addr learned my addr 172.21.15.45:0/2615266578 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.874+0000 7fa2093c7640 1 --2- 172.21.15.45:0/2615266578 >> v2:172.21.15.190:3300/0 conn(0x7fa2040b3040 0x7fa20406cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.874+0000 7fa208bc6640 1 --2- 172.21.15.45:0/2615266578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa20415b270 0x7fa20415f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.874+0000 7fa209bc8640 1 -- 172.21.15.45:0/2615266578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa20415b270 msgr2=0x7fa20415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:01.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.874+0000 7fa209bc8640 1 --2- 172.21.15.45:0/2615266578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa20415b270 0x7fa20415f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.874+0000 7fa209bc8640 1 -- 172.21.15.45:0/2615266578 >> v2:172.21.15.190:3300/0 conn(0x7fa2040b3040 msgr2=0x7fa20406cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:01.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.874+0000 7fa209bc8640 1 --2- 172.21.15.45:0/2615266578 >> v2:172.21.15.190:3300/0 conn(0x7fa2040b3040 0x7fa20406cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.874+0000 7fa209bc8640 1 -- 172.21.15.45:0/2615266578 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20406d6f0 con 0x7fa2040b3420 2023-11-28T20:36:01.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.874+0000 7fa208bc6640 1 --2- 172.21.15.45:0/2615266578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa20415b270 0x7fa20415f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:01.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.875+0000 7fa209bc8640 1 --2- 172.21.15.45:0/2615266578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2040b3420 0x7fa2040b1d20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa2040b4410 tx=0x7fa1fc02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45dd87538e4a26e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:01.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.875+0000 7fa1fbfff640 1 -- 172.21.15.45:0/2615266578 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc02c960 con 0x7fa2040b3420 2023-11-28T20:36:01.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.876+0000 7fa1fbfff640 1 -- 172.21.15.45:0/2615266578 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc02cac0 con 0x7fa2040b3420 2023-11-28T20:36:01.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.876+0000 7fa1fbfff640 1 -- 172.21.15.45:0/2615266578 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc02ce50 con 0x7fa2040b3420 2023-11-28T20:36:01.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.876+0000 7fa20a3c9640 1 -- 172.21.15.45:0/2615266578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2040b3420 msgr2=0x7fa2040b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:01.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.876+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/2615266578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2040b3420 0x7fa2040b1d20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa2040b4410 tx=0x7fa1fc02bdc0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.876+0000 7fa20a3c9640 1 -- 172.21.15.45:0/2615266578 shutdown_connections 2023-11-28T20:36:01.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.876+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/2615266578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2040b3420 0x7fa2040b1d20 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.876+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/2615266578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa20415b270 0x7fa20415f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.876+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/2615266578 >> v2:172.21.15.190:3300/0 conn(0x7fa2040b3040 0x7fa20406cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.876+0000 7fa20a3c9640 1 -- 172.21.15.45:0/2615266578 >> 172.21.15.45:0/2615266578 conn(0x7fa2040b1050 msgr2=0x7fa2040cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:01.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.877+0000 7fa20a3c9640 1 -- 172.21.15.45:0/2615266578 shutdown_connections 2023-11-28T20:36:01.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.877+0000 7fa20a3c9640 1 -- 172.21.15.45:0/2615266578 wait complete. 2023-11-28T20:36:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.878+0000 7fa20a3c9640 1 Processor -- start 2023-11-28T20:36:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.878+0000 7fa20a3c9640 1 -- start start 2023-11-28T20:36:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.879+0000 7fa20a3c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2040b3040 0x7fa20415e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.879+0000 7fa20a3c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2040b3420 0x7fa20415a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.879+0000 7fa20a3c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa20415b270 0x7fa2041435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.879+0000 7fa20a3c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20415dd10 con 0x7fa20415b270 2023-11-28T20:36:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.879+0000 7fa20a3c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20415d090 con 0x7fa2040b3420 2023-11-28T20:36:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.879+0000 7fa20a3c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20415d370 con 0x7fa2040b3040 2023-11-28T20:36:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.880+0000 7fa209bc8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa20415b270 0x7fa2041435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.880+0000 7fa209bc8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa20415b270 0x7fa2041435c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35896/0 (socket says 172.21.15.45:35896) 2023-11-28T20:36:01.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.880+0000 7fa209bc8640 1 -- 172.21.15.45:0/3783780431 learned_addr learned my addr 172.21.15.45:0/3783780431 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.880+0000 7fa209bc8640 1 -- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2040b3040 msgr2=0x7fa20415e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.880+0000 7fa209bc8640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2040b3040 0x7fa20415e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.880+0000 7fa209bc8640 1 -- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2040b3420 msgr2=0x7fa20415a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.880+0000 7fa209bc8640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2040b3420 0x7fa20415a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.880+0000 7fa209bc8640 1 -- 172.21.15.45:0/3783780431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc009b30 con 0x7fa20415b270 2023-11-28T20:36:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.881+0000 7fa209bc8640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa20415b270 0x7fa2041435c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc036040 tx=0x7fa1fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:01.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.881+0000 7fa1f9ffb640 1 -- 172.21.15.45:0/3783780431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc009510 con 0x7fa20415b270 2023-11-28T20:36:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.881+0000 7fa1f9ffb640 1 -- 172.21.15.45:0/3783780431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc03e450 con 0x7fa20415b270 2023-11-28T20:36:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.881+0000 7fa1f9ffb640 1 -- 172.21.15.45:0/3783780431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc03e800 con 0x7fa20415b270 2023-11-28T20:36:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.881+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20415d650 con 0x7fa20415b270 2023-11-28T20:36:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.881+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20415a980 con 0x7fa20415b270 2023-11-28T20:36:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.882+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20409d0f0 con 0x7fa20415b270 2023-11-28T20:36:01.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.883+0000 7fa1f9ffb640 1 -- 172.21.15.45:0/3783780431 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 10) v1 ==== 89407+0+0 (secure 0 0 0) 0x7fa1fc03e960 con 0x7fa20415b270 2023-11-28T20:36:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.884+0000 7fa1f9ffb640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa1e806b070 0x7fa1e806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.884+0000 7fa1f9ffb640 1 -- 172.21.15.45:0/3783780431 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fa1fc0ac960 con 0x7fa20415b270 2023-11-28T20:36:01.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.885+0000 7fa2093c7640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa1e806b070 0x7fa1e806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.886+0000 7fa2093c7640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa1e806b070 0x7fa1e806d530 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa20000b7f0 tx=0x7fa200008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:01.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.888+0000 7fa1f9ffb640 1 -- 172.21.15.45:0/3783780431 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1fc077540 con 0x7fa20415b270 2023-11-28T20:36:01.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:01.981+0000 7fa1f9ffb640 1 -- 172.21.15.45:0/3783780431 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7fa1fc077910 con 0x7fa20415b270 2023-11-28T20:36:02.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.108+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa20406df90 con 0x7fa1e806b070 2023-11-28T20:36:02.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.265+0000 7fa1f9ffb640 1 -- 172.21.15.45:0/3783780431 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa20000bdd0 con 0x7fa1e806b070 2023-11-28T20:36:02.268 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir8 2023-11-28T20:36:02.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.268+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa1e806b070 msgr2=0x7fa1e806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:02.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.268+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa1e806b070 0x7fa1e806d530 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa20000b7f0 tx=0x7fa200008040 comp rx=0 tx=0).stop 2023-11-28T20:36:02.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa20415b270 msgr2=0x7fa2041435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:02.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa20415b270 0x7fa2041435c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc036040 tx=0x7fa1fc004a40 comp rx=0 tx=0).stop 2023-11-28T20:36:02.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 shutdown_connections 2023-11-28T20:36:02.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa1e806b070 0x7fa1e806d530 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa20415b270 0x7fa2041435c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2040b3420 0x7fa20415a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 --2- 172.21.15.45:0/3783780431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2040b3040 0x7fa20415e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 >> 172.21.15.45:0/3783780431 conn(0x7fa2040b1050 msgr2=0x7fa20415f190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:02.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 shutdown_connections 2023-11-28T20:36:02.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.269+0000 7fa20a3c9640 1 -- 172.21.15.45:0/3783780431 wait complete. 2023-11-28T20:36:02.293 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-11-28T20:36:02.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:36:02.294 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-11-28T20:36:02.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2023-11-28T20:36:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.580+0000 7fc96f009640 1 Processor -- start 2023-11-28T20:36:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.580+0000 7fc96f009640 1 -- start start 2023-11-28T20:36:02.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.581+0000 7fc96f009640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9680d1e40 0x7fc9680ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:02.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.581+0000 7fc96f009640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680d0210 con 0x7fc96815d580 2023-11-28T20:36:02.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.581+0000 7fc96f009640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680d0490 con 0x7fc9680b2500 2023-11-28T20:36:02.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.581+0000 7fc96f009640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680d1360 con 0x7fc9680d1e40 2023-11-28T20:36:02.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.582+0000 7fc96e007640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc96815d580 0x7fc9680cfb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48518/0 (socket says 172.21.15.45:48518) 2023-11-28T20:36:02.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.582+0000 7fc96e007640 1 -- 172.21.15.45:0/664908481 learned_addr learned my addr 172.21.15.45:0/664908481 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:36:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.582+0000 7fc96d806640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9680d1e40 0x7fc9680ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:02.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.582+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295082532 0 0) 0x7fc9680d0210 con 0x7fc96815d580 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.582+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc954003520 con 0x7fc96815d580 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2848648062 0 0) 0x7fc954003520 con 0x7fc96815d580 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9680d1e40 msgr2=0x7fc9680ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9680d1e40 0x7fc9680ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 >> v1:172.21.15.190:6789/0 conn(0x7fc9680b2500 legacy=0x7fc968160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680d1640 con 0x7fc96815d580 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc9640019f0 con 0x7fc96815d580 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 >> v1:172.21.15.45:6789/0 conn(0x7fc96815d580 legacy=0x7fc9680cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9540039a0 0x7fc954003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9680d1e40 0x7fc9540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc954004d00 0x7fc954005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:02.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680d1360 con 0x7fc9540039a0 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95400bb40 con 0x7fc954004d00 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95400bd00 con 0x7fc9680d1e40 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.583+0000 7fc96e007640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9540039a0 0x7fc954003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.584+0000 7fc96d806640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9680d1e40 0x7fc9540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.584+0000 7fc96e808640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc954004d00 0x7fc954005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.584+0000 7fc96e007640 1 -- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9680d1e40 msgr2=0x7fc9540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.584+0000 7fc96e007640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9680d1e40 0x7fc9540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.584+0000 7fc96e007640 1 -- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc954004d00 msgr2=0x7fc954005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.584+0000 7fc96e007640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc954004d00 0x7fc954005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.584+0000 7fc96e007640 1 -- 172.21.15.45:0/664908481 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954003520 con 0x7fc9540039a0 2023-11-28T20:36:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.585+0000 7fc96e007640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9540039a0 0x7fc954003dc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc96400a760 tx=0x7fc9540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7880f63ed34b2f10 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.585+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964034630 con 0x7fc9540039a0 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.585+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96402f070 con 0x7fc9540039a0 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.585+0000 7fc96d005640 1 -- 172.21.15.45:0/664908481 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9640349f0 con 0x7fc9540039a0 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.586+0000 7fc96f009640 1 -- 172.21.15.45:0/664908481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9540039a0 msgr2=0x7fc954003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.586+0000 7fc96f009640 1 --2- 172.21.15.45:0/664908481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9540039a0 0x7fc954003dc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc96400a760 tx=0x7fc9540057f0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.586+0000 7fc96e808640 1 -- 172.21.15.45:0/664908481 reap_dead start 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.586+0000 7fc96f009640 1 -- 172.21.15.45:0/664908481 shutdown_connections 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.586+0000 7fc96f009640 1 -- 172.21.15.45:0/664908481 >> 172.21.15.45:0/664908481 conn(0x7fc96806a500 msgr2=0x7fc96806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.586+0000 7fc96f009640 1 -- 172.21.15.45:0/664908481 shutdown_connections 2023-11-28T20:36:02.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.586+0000 7fc96f009640 1 -- 172.21.15.45:0/664908481 wait complete. 2023-11-28T20:36:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.588+0000 7fc96f009640 1 Processor -- start 2023-11-28T20:36:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.588+0000 7fc96f009640 1 -- start start 2023-11-28T20:36:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.589+0000 7fc96f009640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9540039a0 0x7fc9681e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.589+0000 7fc96f009640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9681f6d00 0x7fc9680c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.589+0000 7fc96f009640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc96815fb30 0x7fc96815ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.589+0000 7fc96f009640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9681604f0 con 0x7fc96815fb30 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.589+0000 7fc96f009640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9681f7280 con 0x7fc9681f6d00 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.589+0000 7fc96f009640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9681f7560 con 0x7fc9540039a0 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.589+0000 7fc96e808640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc96815fb30 0x7fc96815ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.589+0000 7fc96e808640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc96815fb30 0x7fc96815ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35918/0 (socket says 172.21.15.45:35918) 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.589+0000 7fc96e808640 1 -- 172.21.15.45:0/2610840259 learned_addr learned my addr 172.21.15.45:0/2610840259 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc96d806640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9681f6d00 0x7fc9680c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc96e808640 1 -- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9540039a0 msgr2=0x7fc9681e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc96e808640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9540039a0 0x7fc9681e6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc96e808640 1 -- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9681f6d00 msgr2=0x7fc9680c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc96e808640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9681f6d00 0x7fc9680c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc96e808640 1 -- 172.21.15.45:0/2610840259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9681f7840 con 0x7fc96815fb30 2023-11-28T20:36:02.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc96e808640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc96815fb30 0x7fc96815ffb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc960000c00 tx=0x7fc96000e670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:02.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc95effd640 1 -- 172.21.15.45:0/2610840259 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96000ee70 con 0x7fc96815fb30 2023-11-28T20:36:02.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc95effd640 1 -- 172.21.15.45:0/2610840259 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9600052e0 con 0x7fc96815fb30 2023-11-28T20:36:02.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.590+0000 7fc95effd640 1 -- 172.21.15.45:0/2610840259 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc960003830 con 0x7fc96815fb30 2023-11-28T20:36:02.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.591+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9681f7b20 con 0x7fc96815fb30 2023-11-28T20:36:02.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.591+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9681e7150 con 0x7fc96815fb30 2023-11-28T20:36:02.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.592+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9680c6ee0 con 0x7fc96815fb30 2023-11-28T20:36:02.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.593+0000 7fc95effd640 1 -- 172.21.15.45:0/2610840259 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7fc960003990 con 0x7fc96815fb30 2023-11-28T20:36:02.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.594+0000 7fc95effd640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc94c06b100 0x7fc94c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:02.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.594+0000 7fc95effd640 1 -- 172.21.15.45:0/2610840259 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fc9600892a0 con 0x7fc96815fb30 2023-11-28T20:36:02.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.595+0000 7fc96e007640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc94c06b100 0x7fc94c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:02.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.596+0000 7fc96e007640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc94c06b100 0x7fc94c06d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc964034000 tx=0x7fc964009030 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:02.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.598+0000 7fc95effd640 1 -- 172.21.15.45:0/2610840259 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9680c6ee0 con 0x7fc96815fb30 2023-11-28T20:36:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.818+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9681f76d0 con 0x7fc94c06b100 2023-11-28T20:36:02.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.976+0000 7fc95effd640 1 -- 172.21.15.45:0/2610840259 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc9681f76d0 con 0x7fc94c06b100 2023-11-28T20:36:02.979 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/dir9 2023-11-28T20:36:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc94c06b100 msgr2=0x7fc94c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc94c06b100 0x7fc94c06d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc964034000 tx=0x7fc964009030 comp rx=0 tx=0).stop 2023-11-28T20:36:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc96815fb30 msgr2=0x7fc96815ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc96815fb30 0x7fc96815ffb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc960000c00 tx=0x7fc96000e670 comp rx=0 tx=0).stop 2023-11-28T20:36:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 shutdown_connections 2023-11-28T20:36:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc94c06b100 0x7fc94c06d5c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc96815fb30 0x7fc96815ffb0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9681f6d00 0x7fc9680c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 --2- 172.21.15.45:0/2610840259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9540039a0 0x7fc9681e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 >> 172.21.15.45:0/2610840259 conn(0x7fc96806a500 msgr2=0x7fc9680cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 shutdown_connections 2023-11-28T20:36:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:02.981+0000 7fc96f009640 1 -- 172.21.15.45:0/2610840259 wait complete. 2023-11-28T20:36:03.004 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-11-28T20:36:03.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2023-11-28T20:36:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.232+0000 7fc5e359e640 1 Processor -- start 2023-11-28T20:36:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.232+0000 7fc5e359e640 1 -- start start 2023-11-28T20:36:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.233+0000 7fc5e359e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc5e415cf10 0x7fc5e40d1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.233+0000 7fc5e359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e40ce000 0x7fc5e40d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.233+0000 7fc5e359e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e40d02b0 con 0x7fc5e415d2f0 2023-11-28T20:36:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.233+0000 7fc5e359e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e40d1720 con 0x7fc5e415cf10 2023-11-28T20:36:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.233+0000 7fc5e359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40d1a00 con 0x7fc5e40ce000 2023-11-28T20:36:03.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.234+0000 7fc5e1d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc5e415d2f0 0x7fc5e40cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48530/0 (socket says 172.21.15.45:48530) 2023-11-28T20:36:03.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.234+0000 7fc5e1d9b640 1 -- 172.21.15.45:0/843976535 learned_addr learned my addr 172.21.15.45:0/843976535 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:36:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.234+0000 7fc5e2d9d640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e40ce000 0x7fc5e40d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.234+0000 7fc5e259c640 1 --2- 172.21.15.45:0/843976535 >> v2:172.21.15.190:3300/0 conn(0x7fc5e415cf10 0x7fc5e40d1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.234+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196736896 0 0) 0x7fc5e40d02b0 con 0x7fc5e415d2f0 2023-11-28T20:36:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.234+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d0003520 con 0x7fc5e415d2f0 2023-11-28T20:36:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1351937880 0 0) 0x7fc5d0003520 con 0x7fc5e415d2f0 2023-11-28T20:36:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e40ce000 msgr2=0x7fc5e40d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e40ce000 0x7fc5e40d2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 >> v2:172.21.15.190:3300/0 conn(0x7fc5e415cf10 msgr2=0x7fc5e40d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 --2- 172.21.15.45:0/843976535 >> v2:172.21.15.190:3300/0 conn(0x7fc5e415cf10 0x7fc5e40d1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e40d1ce0 con 0x7fc5e415d2f0 2023-11-28T20:36:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc5cc005090 con 0x7fc5e415d2f0 2023-11-28T20:36:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 >> v1:172.21.15.45:6789/0 conn(0x7fc5e415d2f0 legacy=0x7fc5e40cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d0004360 0x7fc5d0004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e40ce000 0x7fc5d0004f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40d1720 con 0x7fc5d0003970 2023-11-28T20:36:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40d1a00 con 0x7fc5d0004360 2023-11-28T20:36:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d000b980 con 0x7fc5e40ce000 2023-11-28T20:36:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.235+0000 7fc5e259c640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.236+0000 7fc5e1d9b640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d0004360 0x7fc5d0004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.236+0000 7fc5e2d9d640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e40ce000 0x7fc5d0004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.236+0000 7fc5e259c640 1 -- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e40ce000 msgr2=0x7fc5d0004f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.236+0000 7fc5e259c640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e40ce000 0x7fc5d0004f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.236+0000 7fc5e259c640 1 -- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d0004360 msgr2=0x7fc5d0004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.236+0000 7fc5e259c640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d0004360 0x7fc5d0004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.236+0000 7fc5e259c640 1 -- 172.21.15.45:0/843976535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e40d1ce0 con 0x7fc5d0003970 2023-11-28T20:36:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.236+0000 7fc5e1d9b640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d0004360 0x7fc5d0004840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.236+0000 7fc5e2d9d640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e40ce000 0x7fc5d0004f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.237+0000 7fc5e259c640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5d0003d90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4009fc0 tx=0x7fc5d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf17826ac48b4522 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.237+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d402cc30 con 0x7fc5d0003970 2023-11-28T20:36:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.237+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d400f040 con 0x7fc5d0003970 2023-11-28T20:36:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.237+0000 7fc5e159a640 1 -- 172.21.15.45:0/843976535 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4034440 con 0x7fc5d0003970 2023-11-28T20:36:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.238+0000 7fc5e359e640 1 -- 172.21.15.45:0/843976535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 msgr2=0x7fc5d0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.238+0000 7fc5e359e640 1 --2- 172.21.15.45:0/843976535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5d0003d90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4009fc0 tx=0x7fc5d00057f0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.238+0000 7fc5e2d9d640 1 -- 172.21.15.45:0/843976535 reap_dead start 2023-11-28T20:36:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.238+0000 7fc5e359e640 1 -- 172.21.15.45:0/843976535 shutdown_connections 2023-11-28T20:36:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.238+0000 7fc5e359e640 1 -- 172.21.15.45:0/843976535 >> 172.21.15.45:0/843976535 conn(0x7fc5e4070f40 msgr2=0x7fc5e4071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.238+0000 7fc5e359e640 1 -- 172.21.15.45:0/843976535 shutdown_connections 2023-11-28T20:36:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.238+0000 7fc5e359e640 1 -- 172.21.15.45:0/843976535 wait complete. 2023-11-28T20:36:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.240+0000 7fc5e359e640 1 Processor -- start 2023-11-28T20:36:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.240+0000 7fc5e359e640 1 -- start start 2023-11-28T20:36:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.241+0000 7fc5e359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5e40cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.241+0000 7fc5e359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4154160 0x7fc5e4150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.241+0000 7fc5e359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4159210 0x7fc5e41596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.241+0000 7fc5e359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e4159c00 con 0x7fc5d0003970 2023-11-28T20:36:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.241+0000 7fc5e359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40ccce0 con 0x7fc5e4159210 2023-11-28T20:36:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.241+0000 7fc5e359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40cbc40 con 0x7fc5e4154160 2023-11-28T20:36:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5e40cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5e40cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35932/0 (socket says 172.21.15.45:35932) 2023-11-28T20:36:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e259c640 1 -- 172.21.15.45:0/2882060915 learned_addr learned my addr 172.21.15.45:0/2882060915 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e1d9b640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4154160 0x7fc5e4150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e2d9d640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4159210 0x7fc5e41596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e259c640 1 -- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4154160 msgr2=0x7fc5e4150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e259c640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4154160 0x7fc5e4150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e259c640 1 -- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4159210 msgr2=0x7fc5e41596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e259c640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4159210 0x7fc5e41596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e259c640 1 -- 172.21.15.45:0/2882060915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e40cbf20 con 0x7fc5d0003970 2023-11-28T20:36:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e1d9b640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4154160 0x7fc5e4150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.242+0000 7fc5e259c640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5e40cad00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400e230 tx=0x7fc5d4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.243+0000 7fc5daffd640 1 -- 172.21.15.45:0/2882060915 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4034bd0 con 0x7fc5d0003970 2023-11-28T20:36:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.243+0000 7fc5daffd640 1 -- 172.21.15.45:0/2882060915 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d4034d30 con 0x7fc5d0003970 2023-11-28T20:36:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.243+0000 7fc5daffd640 1 -- 172.21.15.45:0/2882060915 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4034ea0 con 0x7fc5d0003970 2023-11-28T20:36:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.243+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e40cc200 con 0x7fc5d0003970 2023-11-28T20:36:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.243+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e40cc7c0 con 0x7fc5d0003970 2023-11-28T20:36:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.244+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e40cc950 con 0x7fc5d0003970 2023-11-28T20:36:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.245+0000 7fc5daffd640 1 -- 172.21.15.45:0/2882060915 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7fc5d4040450 con 0x7fc5d0003970 2023-11-28T20:36:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.246+0000 7fc5daffd640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc5c406b0d0 0x7fc5c406d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.246+0000 7fc5daffd640 1 -- 172.21.15.45:0/2882060915 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fc5d40ac560 con 0x7fc5d0003970 2023-11-28T20:36:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.247+0000 7fc5e1d9b640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc5c406b0d0 0x7fc5c406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.248+0000 7fc5e1d9b640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc5c406b0d0 0x7fc5c406d590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc5e415c420 tx=0x7fc5cc009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.251+0000 7fc5daffd640 1 -- 172.21.15.45:0/2882060915 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5d4077060 con 0x7fc5d0003970 2023-11-28T20:36:03.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.473+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5e4160c60 con 0x7fc5c406b0d0 2023-11-28T20:36:03.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.603+0000 7fc5daffd640 1 -- 172.21.15.45:0/2882060915 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc5cc00d040 con 0x7fc5c406b0d0 2023-11-28T20:36:03.605 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-11-28T20:36:03.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.606+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc5c406b0d0 msgr2=0x7fc5c406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:03.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.606+0000 7fc5e359e640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc5c406b0d0 0x7fc5c406d590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc5e415c420 tx=0x7fc5cc009040 comp rx=0 tx=0).stop 2023-11-28T20:36:03.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 msgr2=0x7fc5e40cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:03.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5e40cad00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400e230 tx=0x7fc5d4009750 comp rx=0 tx=0).stop 2023-11-28T20:36:03.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 shutdown_connections 2023-11-28T20:36:03.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fc5c406b0d0 0x7fc5c406d590 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4159210 0x7fc5e41596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4154160 0x7fc5e4150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 --2- 172.21.15.45:0/2882060915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0003970 0x7fc5e40cad00 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 >> 172.21.15.45:0/2882060915 conn(0x7fc5e4070f40 msgr2=0x7fc5e4161070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:03.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 shutdown_connections 2023-11-28T20:36:03.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.607+0000 7fc5e359e640 1 -- 172.21.15.45:0/2882060915 wait complete. 2023-11-28T20:36:03.631 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-11-28T20:36:03.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2023-11-28T20:36:03.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.866+0000 7f15ebac9640 1 Processor -- start 2023-11-28T20:36:03.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.866+0000 7f15ebac9640 1 -- start start 2023-11-28T20:36:03.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15ebac9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15e415e840 0x7f15e406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15ebac9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 0x7f15e40717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15ebac9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f15e406a500 0x7f15e406a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15ebac9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e4071ce0 con 0x7f15e415e840 2023-11-28T20:36:03.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15ebac9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e406aed0 con 0x7f15e406a500 2023-11-28T20:36:03.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15ebac9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e415ecd0 con 0x7f15e415f720 2023-11-28T20:36:03.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15e9865640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15e415e840 0x7f15e406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15e9865640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15e415e840 0x7f15e406c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35944/0 (socket says 172.21.15.45:35944) 2023-11-28T20:36:03.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15e9865640 1 -- 172.21.15.45:0/2281249877 learned_addr learned my addr 172.21.15.45:0/2281249877 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:03.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.868+0000 7f15e9064640 1 --2- 172.21.15.45:0/2281249877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 0x7f15e40717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.869+0000 7f15ea066640 1 --2- 172.21.15.45:0/2281249877 >> v2:172.21.15.190:3300/0 conn(0x7f15e406a500 0x7f15e406a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.869+0000 7f15e9865640 1 -- 172.21.15.45:0/2281249877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 msgr2=0x7f15e40717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:03.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.869+0000 7f15e9865640 1 --2- 172.21.15.45:0/2281249877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 0x7f15e40717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.869+0000 7f15e9865640 1 -- 172.21.15.45:0/2281249877 >> v2:172.21.15.190:3300/0 conn(0x7f15e406a500 msgr2=0x7f15e406a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:03.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.869+0000 7f15e9865640 1 --2- 172.21.15.45:0/2281249877 >> v2:172.21.15.190:3300/0 conn(0x7f15e406a500 0x7f15e406a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.869+0000 7f15e9865640 1 -- 172.21.15.45:0/2281249877 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e415f290 con 0x7f15e415e840 2023-11-28T20:36:03.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.870+0000 7f15e9865640 1 --2- 172.21.15.45:0/2281249877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 0x7f15e406c350 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009870 tx=0x7f15e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa2a79380c7ac7c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.870+0000 7f15e8863640 1 -- 172.21.15.45:0/2281249877 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e0031640 con 0x7f15e415e840 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.870+0000 7f15e8863640 1 -- 172.21.15.45:0/2281249877 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e00317a0 con 0x7f15e415e840 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.870+0000 7f15e8863640 1 -- 172.21.15.45:0/2281249877 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e0031b30 con 0x7f15e415e840 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.871+0000 7f15ebac9640 1 -- 172.21.15.45:0/2281249877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 msgr2=0x7f15e406c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.871+0000 7f15ebac9640 1 --2- 172.21.15.45:0/2281249877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 0x7f15e406c350 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009870 tx=0x7f15e002b8f0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.871+0000 7f15ebac9640 1 -- 172.21.15.45:0/2281249877 shutdown_connections 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.871+0000 7f15ebac9640 1 --2- 172.21.15.45:0/2281249877 >> v2:172.21.15.190:3300/0 conn(0x7f15e406a500 0x7f15e406a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.871+0000 7f15ebac9640 1 --2- 172.21.15.45:0/2281249877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 0x7f15e40717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.871+0000 7f15ebac9640 1 --2- 172.21.15.45:0/2281249877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 0x7f15e406c350 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.871+0000 7f15ebac9640 1 -- 172.21.15.45:0/2281249877 >> 172.21.15.45:0/2281249877 conn(0x7f15e40ced60 msgr2=0x7f15e40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:03.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.871+0000 7f15ebac9640 1 -- 172.21.15.45:0/2281249877 shutdown_connections 2023-11-28T20:36:03.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.871+0000 7f15ebac9640 1 -- 172.21.15.45:0/2281249877 wait complete. 2023-11-28T20:36:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.874+0000 7f15ebac9640 1 Processor -- start 2023-11-28T20:36:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.874+0000 7f15ebac9640 1 -- start start 2023-11-28T20:36:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.875+0000 7f15ebac9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e406a500 0x7f15e41552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.875+0000 7f15ebac9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 0x7f15e40c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.875+0000 7f15ebac9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 0x7f15e406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.875+0000 7f15ebac9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4155810 con 0x7f15e415e840 2023-11-28T20:36:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.875+0000 7f15ebac9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e40c9cb0 con 0x7f15e406a500 2023-11-28T20:36:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.875+0000 7f15ebac9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e40c9f20 con 0x7f15e415f720 2023-11-28T20:36:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.875+0000 7f15e9064640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 0x7f15e40c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.875+0000 7f15e9064640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 0x7f15e40c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35952/0 (socket says 172.21.15.45:35952) 2023-11-28T20:36:03.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15e9064640 1 -- 172.21.15.45:0/3879975275 learned_addr learned my addr 172.21.15.45:0/3879975275 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15e9865640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e406a500 0x7f15e41552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15ea066640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 0x7f15e406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15e9064640 1 -- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 msgr2=0x7f15e406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15e9064640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 0x7f15e406da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15e9064640 1 -- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e406a500 msgr2=0x7f15e41552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15e9064640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e406a500 0x7f15e41552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15e9064640 1 -- 172.21.15.45:0/3879975275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e00094d0 con 0x7f15e415e840 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15e9865640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e406a500 0x7f15e41552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.876+0000 7f15e9064640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 0x7f15e40c9460 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f15d400d5f0 tx=0x7f15d400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.877+0000 7f15da7fc640 1 -- 172.21.15.45:0/3879975275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d40125f0 con 0x7f15e415e840 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.877+0000 7f15da7fc640 1 -- 172.21.15.45:0/3879975275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d4012750 con 0x7f15e415e840 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.877+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e40ca200 con 0x7f15e415e840 2023-11-28T20:36:03.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.877+0000 7f15da7fc640 1 -- 172.21.15.45:0/3879975275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4012b00 con 0x7f15e415e840 2023-11-28T20:36:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.877+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e40ca7c0 con 0x7f15e415e840 2023-11-28T20:36:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.878+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e40733d0 con 0x7f15e415e840 2023-11-28T20:36:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.879+0000 7f15da7fc640 1 -- 172.21.15.45:0/3879975275 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f15d4012c60 con 0x7f15e415e840 2023-11-28T20:36:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.880+0000 7f15da7fc640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f15c806b0d0 0x7f15c806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.881+0000 7f15da7fc640 1 -- 172.21.15.45:0/3879975275 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f15d408a1d0 con 0x7f15e415e840 2023-11-28T20:36:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.881+0000 7f15e9865640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f15c806b0d0 0x7f15c806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.882+0000 7f15e9865640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f15c806b0d0 0x7f15c806d590 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f15e0036000 tx=0x7f15e003c040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:03.885+0000 7f15da7fc640 1 -- 172.21.15.45:0/3879975275 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15d4054ce0 con 0x7f15e415e840 2023-11-28T20:36:04.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.103+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f15e4072300 con 0x7f15c806b0d0 2023-11-28T20:36:04.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.237+0000 7f15da7fc640 1 -- 172.21.15.45:0/3879975275 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f15e0031990 con 0x7f15c806b0d0 2023-11-28T20:36:04.239 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-11-28T20:36:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f15c806b0d0 msgr2=0x7f15c806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f15c806b0d0 0x7f15c806d590 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f15e0036000 tx=0x7f15e003c040 comp rx=0 tx=0).stop 2023-11-28T20:36:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 msgr2=0x7f15e40c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 0x7f15e40c9460 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f15d400d5f0 tx=0x7f15d400dab0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 shutdown_connections 2023-11-28T20:36:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f15c806b0d0 0x7f15c806d590 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e415f720 0x7f15e406da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15e415e840 0x7f15e40c9460 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 --2- 172.21.15.45:0/3879975275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e406a500 0x7f15e41552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 >> 172.21.15.45:0/3879975275 conn(0x7f15e40ced60 msgr2=0x7f15e40ad080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 shutdown_connections 2023-11-28T20:36:04.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.241+0000 7f15ebac9640 1 -- 172.21.15.45:0/3879975275 wait complete. 2023-11-28T20:36:04.267 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-11-28T20:36:04.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2023-11-28T20:36:04.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.497+0000 7f4af881a640 1 Processor -- start 2023-11-28T20:36:04.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.497+0000 7f4af881a640 1 -- start start 2023-11-28T20:36:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.499+0000 7f4af881a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41437b0 0x7f4af415abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.499+0000 7f4af881a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4af415b110 0x7f4af40cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.499+0000 7f4af881a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af4071560 con 0x7f4af40ce1c0 2023-11-28T20:36:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.499+0000 7f4af881a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af40ce630 con 0x7f4af415b110 2023-11-28T20:36:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.499+0000 7f4af881a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af4071800 con 0x7f4af41437b0 2023-11-28T20:36:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.500+0000 7f4af3fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4af40ce1c0 0x7f4af415dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46284/0 (socket says 172.21.15.45:46284) 2023-11-28T20:36:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.500+0000 7f4af3fff640 1 -- 172.21.15.45:0/1482958653 learned_addr learned my addr 172.21.15.45:0/1482958653 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:36:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.500+0000 7f4af37fe640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41437b0 0x7f4af415abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.500+0000 7f4aeaffd640 1 --2- 172.21.15.45:0/1482958653 >> v2:172.21.15.190:3300/0 conn(0x7f4af415b110 0x7f4af40cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.500+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997109927 0 0) 0x7f4af4071560 con 0x7f4af40ce1c0 2023-11-28T20:36:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.500+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae0003520 con 0x7f4af40ce1c0 2023-11-28T20:36:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4222308468 0 0) 0x7f4ae0003520 con 0x7f4af40ce1c0 2023-11-28T20:36:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41437b0 msgr2=0x7f4af415abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41437b0 0x7f4af415abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 >> v2:172.21.15.190:3300/0 conn(0x7f4af415b110 msgr2=0x7f4af40cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 --2- 172.21.15.45:0/1482958653 >> v2:172.21.15.190:3300/0 conn(0x7f4af415b110 0x7f4af40cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af4071ae0 con 0x7f4af40ce1c0 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4aec001670 con 0x7f4af40ce1c0 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 >> v1:172.21.15.45:6789/0 conn(0x7f4af40ce1c0 legacy=0x7f4af415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae0003970 0x7f4ae0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41437b0 0x7f4ae00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0004d00 0x7f4ae0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af40ce630 con 0x7f4ae0003970 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af4071800 con 0x7f4ae0004d00 2023-11-28T20:36:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae000b980 con 0x7f4af41437b0 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.501+0000 7f4af37fe640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae0003970 0x7f4ae0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.502+0000 7f4aeaffd640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41437b0 0x7f4ae00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.502+0000 7f4af3fff640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0004d00 0x7f4ae0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.502+0000 7f4af37fe640 1 -- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41437b0 msgr2=0x7f4ae00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.502+0000 7f4af37fe640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41437b0 0x7f4ae00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.502+0000 7f4af37fe640 1 -- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0004d00 msgr2=0x7f4ae0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.502+0000 7f4af37fe640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0004d00 0x7f4ae0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.502+0000 7f4af37fe640 1 -- 172.21.15.45:0/1482958653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae0003520 con 0x7f4ae0003970 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.502+0000 7f4aeaffd640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41437b0 0x7f4ae00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:04.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.502+0000 7f4af3fff640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0004d00 0x7f4ae0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.503+0000 7f4af37fe640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae0003970 0x7f4ae0003d90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009ac0 tx=0x7f4ae00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6554f72445088c97 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.503+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae402c6b0 con 0x7f4ae0003970 2023-11-28T20:36:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.503+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae402c810 con 0x7f4ae0003970 2023-11-28T20:36:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.503+0000 7f4af2ffd640 1 -- 172.21.15.45:0/1482958653 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae402cba0 con 0x7f4ae0003970 2023-11-28T20:36:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.504+0000 7f4af881a640 1 -- 172.21.15.45:0/1482958653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae0003970 msgr2=0x7f4ae0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.504+0000 7f4af881a640 1 --2- 172.21.15.45:0/1482958653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ae0003970 0x7f4ae0003d90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009ac0 tx=0x7f4ae00057f0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.504+0000 7f4af3fff640 1 -- 172.21.15.45:0/1482958653 reap_dead start 2023-11-28T20:36:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.504+0000 7f4af881a640 1 -- 172.21.15.45:0/1482958653 shutdown_connections 2023-11-28T20:36:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.504+0000 7f4af881a640 1 -- 172.21.15.45:0/1482958653 >> 172.21.15.45:0/1482958653 conn(0x7f4af40b1050 msgr2=0x7f4af406a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.504+0000 7f4af881a640 1 -- 172.21.15.45:0/1482958653 shutdown_connections 2023-11-28T20:36:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.504+0000 7f4af881a640 1 -- 172.21.15.45:0/1482958653 wait complete. 2023-11-28T20:36:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.506+0000 7f4af881a640 1 Processor -- start 2023-11-28T20:36:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.506+0000 7f4af881a640 1 -- start start 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.507+0000 7f4af881a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0003970 0x7f4af4157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.507+0000 7f4af881a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415bfa0 0x7f4af41f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.507+0000 7f4af881a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41f3640 0x7f4af41f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.507+0000 7f4af881a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af41f4000 con 0x7f4af415bfa0 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.507+0000 7f4af881a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af415d300 con 0x7f4ae0003970 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.507+0000 7f4af881a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af415c780 con 0x7f4af41f3640 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4aeaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415bfa0 0x7f4af41f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4aeaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415bfa0 0x7f4af41f3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44732/0 (socket says 172.21.15.45:44732) 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4aeaffd640 1 -- 172.21.15.45:0/3284753 learned_addr learned my addr 172.21.15.45:0/3284753 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4aeaffd640 1 -- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41f3640 msgr2=0x7f4af41f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4aeaffd640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41f3640 0x7f4af41f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4af37fe640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0003970 0x7f4af4157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4aeaffd640 1 -- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0003970 msgr2=0x7f4af4157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4aeaffd640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0003970 0x7f4af4157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4aeaffd640 1 -- 172.21.15.45:0/3284753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af415ca60 con 0x7f4af415bfa0 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4af37fe640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0003970 0x7f4af4157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.508+0000 7f4aeaffd640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415bfa0 0x7f4af41f3100 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00f3d0 tx=0x7f4adc00f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.509+0000 7f4af0ff9640 1 -- 172.21.15.45:0/3284753 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc01a070 con 0x7f4af415bfa0 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.509+0000 7f4af0ff9640 1 -- 172.21.15.45:0/3284753 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4adc003990 con 0x7f4af415bfa0 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.509+0000 7f4af0ff9640 1 -- 172.21.15.45:0/3284753 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc003d00 con 0x7f4af415bfa0 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.509+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af415cd40 con 0x7f4af415bfa0 2023-11-28T20:36:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.509+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af415c420 con 0x7f4af415bfa0 2023-11-28T20:36:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.510+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af41f42a0 con 0x7f4af415bfa0 2023-11-28T20:36:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.511+0000 7f4af0ff9640 1 -- 172.21.15.45:0/3284753 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f4adc004020 con 0x7f4af415bfa0 2023-11-28T20:36:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.512+0000 7f4af0ff9640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4ad406b0d0 0x7f4ad406d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.512+0000 7f4af0ff9640 1 -- 172.21.15.45:0/3284753 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f4adc08aa30 con 0x7f4af415bfa0 2023-11-28T20:36:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.513+0000 7f4af37fe640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4ad406b0d0 0x7f4ad406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.514+0000 7f4af37fe640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4ad406b0d0 0x7f4ad406d590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4036000 tx=0x7f4ae4030000 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:04.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.516+0000 7f4af0ff9640 1 -- 172.21.15.45:0/3284753 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4adc055390 con 0x7f4af415bfa0 2023-11-28T20:36:04.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.733+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4af40720a0 con 0x7f4ad406b0d0 2023-11-28T20:36:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.860+0000 7f4af0ff9640 1 -- 172.21.15.45:0/3284753 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f4ae40041c0 con 0x7f4ad406b0d0 2023-11-28T20:36:04.862 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-11-28T20:36:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.863+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4ad406b0d0 msgr2=0x7f4ad406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.863+0000 7f4af881a640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4ad406b0d0 0x7f4ad406d590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4036000 tx=0x7f4ae4030000 comp rx=0 tx=0).stop 2023-11-28T20:36:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.863+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415bfa0 msgr2=0x7f4af41f3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:04.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.864+0000 7f4af881a640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415bfa0 0x7f4af41f3100 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00f3d0 tx=0x7f4adc00f890 comp rx=0 tx=0).stop 2023-11-28T20:36:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.864+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 shutdown_connections 2023-11-28T20:36:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.864+0000 7f4af881a640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4ad406b0d0 0x7f4ad406d590 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.864+0000 7f4af881a640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4af41f3640 0x7f4af41f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.864+0000 7f4af881a640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4af415bfa0 0x7f4af41f3100 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.864+0000 7f4af881a640 1 --2- 172.21.15.45:0/3284753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0003970 0x7f4af4157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.864+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 >> 172.21.15.45:0/3284753 conn(0x7f4af40b1050 msgr2=0x7f4af40af250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.864+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 shutdown_connections 2023-11-28T20:36:04.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:04.864+0000 7f4af881a640 1 -- 172.21.15.45:0/3284753 wait complete. 2023-11-28T20:36:04.889 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-11-28T20:36:04.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2023-11-28T20:36:05.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.118+0000 7ff3e2316640 1 Processor -- start 2023-11-28T20:36:05.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.118+0000 7ff3e2316640 1 -- start start 2023-11-28T20:36:05.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.119+0000 7ff3e2316640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3dc157c10 0x7ff3dc06da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.120+0000 7ff3e2316640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3dc06ca60 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.120+0000 7ff3e2316640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3dc15eb40 con 0x7ff3dc06a500 2023-11-28T20:36:05.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.120+0000 7ff3e2316640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc15e930 con 0x7ff3dc157c10 2023-11-28T20:36:05.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.120+0000 7ff3dbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3dc157c10 0x7ff3dc06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.120+0000 7ff3e08b3640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff3dc15e4e0 0x7ff3dc0c9e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46286/0 (socket says 172.21.15.45:46286) 2023-11-28T20:36:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.120+0000 7ff3dbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3dc157c10 0x7ff3dc06da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44738/0 (socket says 172.21.15.45:44738) 2023-11-28T20:36:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.120+0000 7ff3e08b3640 1 -- 172.21.15.45:0/2403182141 learned_addr learned my addr 172.21.15.45:0/2403182141 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:36:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.120+0000 7ff3dbfff640 1 -- 172.21.15.45:0/2403182141 >> v1:172.21.15.190:6789/0 conn(0x7ff3dc06a500 legacy=0x7ff3dc15ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.121+0000 7ff3dbfff640 1 -- 172.21.15.45:0/2403182141 >> v1:172.21.15.45:6789/0 conn(0x7ff3dc15e4e0 legacy=0x7ff3dc0c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.121+0000 7ff3daffd640 1 -- 172.21.15.45:0/2403182141 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181209798 0 0) 0x7ff3dc06ca60 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.121+0000 7ff3dbfff640 1 -- 172.21.15.45:0/2403182141 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc157ff0 con 0x7ff3dc157c10 2023-11-28T20:36:05.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.122+0000 7ff3dbfff640 1 --2- 172.21.15.45:0/2403182141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc157c10 0x7ff3dc06da50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0097a0 tx=0x7ff3cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8347d3a7b43dc23 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:05.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.122+0000 7ff3daffd640 1 -- 172.21.15.45:0/2403182141 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc030610 con 0x7ff3dc157c10 2023-11-28T20:36:05.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.122+0000 7ff3daffd640 1 -- 172.21.15.45:0/2403182141 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc030770 con 0x7ff3dc157c10 2023-11-28T20:36:05.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.122+0000 7ff3daffd640 1 -- 172.21.15.45:0/2403182141 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc030b00 con 0x7ff3dc157c10 2023-11-28T20:36:05.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.123+0000 7ff3e2316640 1 -- 172.21.15.45:0/2403182141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc157c10 msgr2=0x7ff3dc06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.123+0000 7ff3e2316640 1 --2- 172.21.15.45:0/2403182141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc157c10 0x7ff3dc06da50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0097a0 tx=0x7ff3cc02b960 comp rx=0 tx=0).stop 2023-11-28T20:36:05.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.123+0000 7ff3e2316640 1 -- 172.21.15.45:0/2403182141 shutdown_connections 2023-11-28T20:36:05.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.123+0000 7ff3e2316640 1 --2- 172.21.15.45:0/2403182141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc157c10 0x7ff3dc06da50 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.123+0000 7ff3e2316640 1 -- 172.21.15.45:0/2403182141 >> 172.21.15.45:0/2403182141 conn(0x7ff3dc0ced60 msgr2=0x7ff3dc0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:05.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.123+0000 7ff3e2316640 1 -- 172.21.15.45:0/2403182141 shutdown_connections 2023-11-28T20:36:05.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.123+0000 7ff3e2316640 1 -- 172.21.15.45:0/2403182141 wait complete. 2023-11-28T20:36:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.125+0000 7ff3e2316640 1 Processor -- start 2023-11-28T20:36:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.125+0000 7ff3e2316640 1 -- start start 2023-11-28T20:36:05.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.126+0000 7ff3e2316640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3dc06a500 0x7ff3dc1f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.126+0000 7ff3e2316640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3dc157c10 0x7ff3dc153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.126+0000 7ff3e2316640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc15e4e0 0x7ff3dc0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.126+0000 7ff3e2316640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc1f5270 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.126+0000 7ff3e2316640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc15a070 con 0x7ff3dc06a500 2023-11-28T20:36:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.126+0000 7ff3e2316640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc153b80 con 0x7ff3dc157c10 2023-11-28T20:36:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.126+0000 7ff3e08b3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc15e4e0 0x7ff3dc0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.126+0000 7ff3e08b3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc15e4e0 0x7ff3dc0706d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44754/0 (socket says 172.21.15.45:44754) 2023-11-28T20:36:05.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.126+0000 7ff3e08b3640 1 -- 172.21.15.45:0/3504685220 learned_addr learned my addr 172.21.15.45:0/3504685220 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3dbfff640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3dc06a500 0x7ff3dc1f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3db7fe640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3dc157c10 0x7ff3dc153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3e08b3640 1 -- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3dc157c10 msgr2=0x7ff3dc153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3e08b3640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3dc157c10 0x7ff3dc153640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3e08b3640 1 -- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3dc06a500 msgr2=0x7ff3dc1f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3e08b3640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3dc06a500 0x7ff3dc1f4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3e08b3640 1 -- 172.21.15.45:0/3504685220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc009400 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3dbfff640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3dc06a500 0x7ff3dc1f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3e08b3640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc15e4e0 0x7ff3dc0706d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c780 tx=0x7ff3d400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3d8ff9640 1 -- 172.21.15.45:0/3504685220 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d400ce30 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3d8ff9640 1 -- 172.21.15.45:0/3504685220 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d400a5e0 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3d8ff9640 1 -- 172.21.15.45:0/3504685220 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d400a990 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.127+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc159ab0 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.128+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc0abf10 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.129+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc0abd70 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.130+0000 7ff3d8ff9640 1 -- 172.21.15.45:0/3504685220 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7ff3d400aaf0 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.131+0000 7ff3d8ff9640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff3bc06b0d0 0x7ff3bc06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.131+0000 7ff3d8ff9640 1 -- 172.21.15.45:0/3504685220 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7ff3d4089140 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.132+0000 7ff3dbfff640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff3bc06b0d0 0x7ff3bc06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.133+0000 7ff3dbfff640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff3bc06b0d0 0x7ff3bc06d590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc036000 tx=0x7ff3cc03b040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:05.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.136+0000 7ff3d8ff9640 1 -- 172.21.15.45:0/3504685220 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3dc0abd70 con 0x7ff3dc15e4e0 2023-11-28T20:36:05.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.353+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3dc06d250 con 0x7ff3bc06b0d0 2023-11-28T20:36:05.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.487+0000 7ff3d8ff9640 1 -- 172.21.15.45:0/3504685220 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff3cc0309e0 con 0x7ff3bc06b0d0 2023-11-28T20:36:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-11-28T20:36:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff3bc06b0d0 msgr2=0x7ff3bc06d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff3bc06b0d0 0x7ff3bc06d590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc036000 tx=0x7ff3cc03b040 comp rx=0 tx=0).stop 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc15e4e0 msgr2=0x7ff3dc0706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc15e4e0 0x7ff3dc0706d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c780 tx=0x7ff3d400cc40 comp rx=0 tx=0).stop 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 shutdown_connections 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff3bc06b0d0 0x7ff3bc06d590 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3dc15e4e0 0x7ff3dc0706d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3dc157c10 0x7ff3dc153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 --2- 172.21.15.45:0/3504685220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3dc06a500 0x7ff3dc1f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 >> 172.21.15.45:0/3504685220 conn(0x7ff3dc0ced60 msgr2=0x7ff3dc15ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 shutdown_connections 2023-11-28T20:36:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.491+0000 7ff3e2316640 1 -- 172.21.15.45:0/3504685220 wait complete. 2023-11-28T20:36:05.513 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-11-28T20:36:05.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2023-11-28T20:36:05.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.744+0000 7f52887b6640 1 Processor -- start 2023-11-28T20:36:05.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.744+0000 7f52887b6640 1 -- start start 2023-11-28T20:36:05.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f52887b6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f52801653a0 0x7f52800ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f52887b6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52800712c0 0x7f5280162290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f52887b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528015f7e0 con 0x7f52800c1d30 2023-11-28T20:36:05.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f52887b6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52801658a0 con 0x7f52801653a0 2023-11-28T20:36:05.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f52887b6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5280165b80 con 0x7f52800712c0 2023-11-28T20:36:05.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f5285d51640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f52801653a0 0x7f52800ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f5285d51640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f52801653a0 0x7f52800ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44758/0 (socket says 172.21.15.45:44758) 2023-11-28T20:36:05.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f5285d51640 1 -- 172.21.15.45:0/582221089 learned_addr learned my addr 172.21.15.45:0/582221089 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:05.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f5286552640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f52800c1d30 0x7f52800c2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46292/0 (socket says 172.21.15.45:46292) 2023-11-28T20:36:05.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.746+0000 7f5286d53640 1 --2- 172.21.15.45:0/582221089 >> v2:172.21.15.190:3300/0 conn(0x7f52800712c0 0x7f5280162290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.747+0000 7f5285d51640 1 -- 172.21.15.45:0/582221089 >> v2:172.21.15.190:3300/0 conn(0x7f52800712c0 msgr2=0x7f5280162290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:05.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.747+0000 7f5285d51640 1 --2- 172.21.15.45:0/582221089 >> v2:172.21.15.190:3300/0 conn(0x7f52800712c0 0x7f5280162290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.747+0000 7f5285d51640 1 -- 172.21.15.45:0/582221089 >> v1:172.21.15.45:6789/0 conn(0x7f52800c1d30 legacy=0x7f52800c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.747+0000 7f5285d51640 1 -- 172.21.15.45:0/582221089 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5280165e60 con 0x7f52801653a0 2023-11-28T20:36:05.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.747+0000 7f5285550640 1 -- 172.21.15.45:0/582221089 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499154991 0 0) 0x7f528015f7e0 con 0x7f52800c1d30 2023-11-28T20:36:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.748+0000 7f5285d51640 1 --2- 172.21.15.45:0/582221089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52801653a0 0x7f52800ce480 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5274009870 tx=0x7f527402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715f0fb6ea86ec6f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.748+0000 7f5285550640 1 -- 172.21.15.45:0/582221089 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5274030860 con 0x7f52801653a0 2023-11-28T20:36:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.748+0000 7f5285550640 1 -- 172.21.15.45:0/582221089 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52740309c0 con 0x7f52801653a0 2023-11-28T20:36:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.748+0000 7f5285550640 1 -- 172.21.15.45:0/582221089 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5274030d50 con 0x7f52801653a0 2023-11-28T20:36:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.749+0000 7f52887b6640 1 -- 172.21.15.45:0/582221089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52801653a0 msgr2=0x7f52800ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.749+0000 7f52887b6640 1 --2- 172.21.15.45:0/582221089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52801653a0 0x7f52800ce480 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5274009870 tx=0x7f527402b9a0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.749+0000 7f52887b6640 1 -- 172.21.15.45:0/582221089 shutdown_connections 2023-11-28T20:36:05.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.749+0000 7f52887b6640 1 --2- 172.21.15.45:0/582221089 >> v2:172.21.15.190:3300/0 conn(0x7f52800712c0 0x7f5280162290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.749+0000 7f52887b6640 1 --2- 172.21.15.45:0/582221089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52801653a0 0x7f52800ce480 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.749+0000 7f52887b6640 1 -- 172.21.15.45:0/582221089 >> 172.21.15.45:0/582221089 conn(0x7f52800ced60 msgr2=0x7f52800cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:05.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.749+0000 7f52887b6640 1 -- 172.21.15.45:0/582221089 shutdown_connections 2023-11-28T20:36:05.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.749+0000 7f52887b6640 1 -- 172.21.15.45:0/582221089 wait complete. 2023-11-28T20:36:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.751+0000 7f52887b6640 1 Processor -- start 2023-11-28T20:36:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.751+0000 7f52887b6640 1 -- start start 2023-11-28T20:36:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.752+0000 7f52887b6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52800712c0 0x7f52801f85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.752+0000 7f52887b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52800c1d30 0x7f528015b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.752+0000 7f52887b6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52801653a0 0x7f528009c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.752+0000 7f52887b6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52800b2ee0 con 0x7f52800c1d30 2023-11-28T20:36:05.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.752+0000 7f52887b6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52800b3160 con 0x7f52800712c0 2023-11-28T20:36:05.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.752+0000 7f52887b6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52800b3440 con 0x7f52801653a0 2023-11-28T20:36:05.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.752+0000 7f5285d51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52800c1d30 0x7f528015b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.752+0000 7f5285d51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52800c1d30 0x7f528015b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44774/0 (socket says 172.21.15.45:44774) 2023-11-28T20:36:05.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5285d51640 1 -- 172.21.15.45:0/1483682250 learned_addr learned my addr 172.21.15.45:0/1483682250 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:05.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5286552640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52800712c0 0x7f52801f85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5286d53640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52801653a0 0x7f528009c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5285d51640 1 -- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52801653a0 msgr2=0x7f528009c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5285d51640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52801653a0 0x7f528009c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5285d51640 1 -- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52800712c0 msgr2=0x7f52801f85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:05.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5285d51640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52800712c0 0x7f52801f85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:05.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5285d51640 1 -- 172.21.15.45:0/1483682250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52740094d0 con 0x7f52800c1d30 2023-11-28T20:36:05.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5286552640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52800712c0 0x7f52801f85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:05.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5286d53640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52801653a0 0x7f528009c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:05.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.753+0000 7f5285d51640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52800c1d30 0x7f528015b250 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f52740090b0 tx=0x7f52740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:05.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.754+0000 7f527effd640 1 -- 172.21.15.45:0/1483682250 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527403e070 con 0x7f52800c1d30 2023-11-28T20:36:05.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.754+0000 7f527effd640 1 -- 172.21.15.45:0/1483682250 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52740398d0 con 0x7f52800c1d30 2023-11-28T20:36:05.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.754+0000 7f527effd640 1 -- 172.21.15.45:0/1483682250 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5274039c80 con 0x7f52800c1d30 2023-11-28T20:36:05.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.754+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52800b3720 con 0x7f52800c1d30 2023-11-28T20:36:05.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.754+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52801f8b00 con 0x7f52800c1d30 2023-11-28T20:36:05.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.755+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52800b3c40 con 0x7f52800c1d30 2023-11-28T20:36:05.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.756+0000 7f527effd640 1 -- 172.21.15.45:0/1483682250 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f5274042070 con 0x7f52800c1d30 2023-11-28T20:36:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.757+0000 7f527effd640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f526c06b0d0 0x7f526c06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.758+0000 7f527effd640 1 -- 172.21.15.45:0/1483682250 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f52740abca0 con 0x7f52800c1d30 2023-11-28T20:36:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.758+0000 7f5286552640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f526c06b0d0 0x7f526c06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.759+0000 7f5286552640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f526c06b0d0 0x7f526c06d590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52700012e0 tx=0x7f5270009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:05.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.763+0000 7f527effd640 1 -- 172.21.15.45:0/1483682250 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52800b3c40 con 0x7f52800c1d30 2023-11-28T20:36:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:05.982+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f52800cf710 con 0x7f526c06b0d0 2023-11-28T20:36:06.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.114+0000 7f527effd640 1 -- 172.21.15.45:0/1483682250 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f52800cf710 con 0x7f526c06b0d0 2023-11-28T20:36:06.117 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-11-28T20:36:06.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.117+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f526c06b0d0 msgr2=0x7f526c06d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.117+0000 7f52887b6640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f526c06b0d0 0x7f526c06d590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52700012e0 tx=0x7f5270009040 comp rx=0 tx=0).stop 2023-11-28T20:36:06.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.117+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52800c1d30 msgr2=0x7f528015b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.117+0000 7f52887b6640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52800c1d30 0x7f528015b250 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f52740090b0 tx=0x7f52740046c0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.118+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 shutdown_connections 2023-11-28T20:36:06.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.118+0000 7f52887b6640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f526c06b0d0 0x7f526c06d590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.118+0000 7f52887b6640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52801653a0 0x7f528009c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.118+0000 7f52887b6640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52800c1d30 0x7f528015b250 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.118+0000 7f52887b6640 1 --2- 172.21.15.45:0/1483682250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52800712c0 0x7f52801f85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.118+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 >> 172.21.15.45:0/1483682250 conn(0x7f52800ced60 msgr2=0x7f52800ca880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:06.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.118+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 shutdown_connections 2023-11-28T20:36:06.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.118+0000 7f52887b6640 1 -- 172.21.15.45:0/1483682250 wait complete. 2023-11-28T20:36:06.141 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-11-28T20:36:06.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2023-11-28T20:36:06.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.368+0000 7f316722e640 1 Processor -- start 2023-11-28T20:36:06.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.369+0000 7f316722e640 1 -- start start 2023-11-28T20:36:06.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.370+0000 7f316722e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f31600d0710 0x7f31600b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.370+0000 7f316722e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31600d23a0 0x7f31600b6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.370+0000 7f316722e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600b9030 con 0x7f316015d150 2023-11-28T20:36:06.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.370+0000 7f316722e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31600b75c0 con 0x7f31600d23a0 2023-11-28T20:36:06.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.370+0000 7f316722e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31600b7880 con 0x7f31600d0710 2023-11-28T20:36:06.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.370+0000 7f3166a2d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31600d23a0 0x7f31600b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.371+0000 7f3166a2d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31600d23a0 0x7f31600b6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44780/0 (socket says 172.21.15.45:44780) 2023-11-28T20:36:06.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.371+0000 7f3166a2d640 1 -- 172.21.15.45:0/1380733347 learned_addr learned my addr 172.21.15.45:0/1380733347 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:06.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.371+0000 7f3166a2d640 1 -- 172.21.15.45:0/1380733347 >> v2:172.21.15.190:3300/0 conn(0x7f31600d0710 msgr2=0x7f31600b8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:06.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.371+0000 7f3165a2b640 1 --2- 172.21.15.45:0/1380733347 >> v2:172.21.15.190:3300/0 conn(0x7f31600d0710 0x7f31600b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.371+0000 7f3166a2d640 1 --2- 172.21.15.45:0/1380733347 >> v2:172.21.15.190:3300/0 conn(0x7f31600d0710 0x7f31600b8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.371+0000 7f3166a2d640 1 -- 172.21.15.45:0/1380733347 >> v1:172.21.15.45:6789/0 conn(0x7f316015d150 legacy=0x7f31600d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.371+0000 7f3166a2d640 1 -- 172.21.15.45:0/1380733347 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31600b7b60 con 0x7f31600d23a0 2023-11-28T20:36:06.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.371+0000 7f316522a640 1 -- 172.21.15.45:0/1380733347 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120950334 0 0) 0x7f315c001110 con 0x7f316015d150 2023-11-28T20:36:06.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.371+0000 7f3165a2b640 1 --2- 172.21.15.45:0/1380733347 >> v2:172.21.15.190:3300/0 conn(0x7f31600d0710 0x7f31600b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:06.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.372+0000 7f3166a2d640 1 --2- 172.21.15.45:0/1380733347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31600d23a0 0x7f31600b6070 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f31600d4190 tx=0x7f315802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836fbbc32b6361bb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316522a640 1 -- 172.21.15.45:0/1380733347 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315802cb10 con 0x7f31600d23a0 2023-11-28T20:36:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316522a640 1 -- 172.21.15.45:0/1380733347 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315802cc70 con 0x7f31600d23a0 2023-11-28T20:36:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316522a640 1 -- 172.21.15.45:0/1380733347 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31580343f0 con 0x7f31600d23a0 2023-11-28T20:36:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316722e640 1 -- 172.21.15.45:0/1380733347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31600d23a0 msgr2=0x7f31600b6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316722e640 1 --2- 172.21.15.45:0/1380733347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31600d23a0 0x7f31600b6070 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f31600d4190 tx=0x7f315802bf70 comp rx=0 tx=0).stop 2023-11-28T20:36:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316722e640 1 -- 172.21.15.45:0/1380733347 shutdown_connections 2023-11-28T20:36:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316722e640 1 --2- 172.21.15.45:0/1380733347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31600d23a0 0x7f31600b6070 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316722e640 1 --2- 172.21.15.45:0/1380733347 >> v2:172.21.15.190:3300/0 conn(0x7f31600d0710 0x7f31600b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316722e640 1 -- 172.21.15.45:0/1380733347 >> 172.21.15.45:0/1380733347 conn(0x7f3160070f40 msgr2=0x7f3160071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:06.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316722e640 1 -- 172.21.15.45:0/1380733347 shutdown_connections 2023-11-28T20:36:06.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.373+0000 7f316722e640 1 -- 172.21.15.45:0/1380733347 wait complete. 2023-11-28T20:36:06.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.375+0000 7f316722e640 1 Processor -- start 2023-11-28T20:36:06.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.375+0000 7f316722e640 1 -- start start 2023-11-28T20:36:06.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.376+0000 7f316722e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31600d0710 0x7f316014e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.376+0000 7f316722e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31600d23a0 0x7f3160159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.376+0000 7f316722e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f316015d150 0x7f31600cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.376+0000 7f316722e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3160159b60 con 0x7f316015d150 2023-11-28T20:36:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.376+0000 7f316722e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3160159e40 con 0x7f31600d0710 2023-11-28T20:36:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.376+0000 7f316722e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316015a120 con 0x7f31600d23a0 2023-11-28T20:36:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f3166a2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f316015d150 0x7f31600cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f3166a2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f316015d150 0x7f31600cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44782/0 (socket says 172.21.15.45:44782) 2023-11-28T20:36:06.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f3166a2d640 1 -- 172.21.15.45:0/507073207 learned_addr learned my addr 172.21.15.45:0/507073207 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:06.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f316622c640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31600d0710 0x7f316014e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f3165a2b640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31600d23a0 0x7f3160159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f3166a2d640 1 -- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31600d23a0 msgr2=0x7f3160159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f3166a2d640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31600d23a0 0x7f3160159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f3166a2d640 1 -- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31600d0710 msgr2=0x7f316014e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f3166a2d640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31600d0710 0x7f316014e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.377+0000 7f3166a2d640 1 -- 172.21.15.45:0/507073207 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158009b30 con 0x7f316015d150 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.378+0000 7f3166a2d640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f316015d150 0x7f31600cc6e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3158036040 tx=0x7f3158004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.378+0000 7f3156ffd640 1 -- 172.21.15.45:0/507073207 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315803e450 con 0x7f316015d150 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.378+0000 7f3156ffd640 1 -- 172.21.15.45:0/507073207 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315803e5d0 con 0x7f316015d150 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.378+0000 7f3156ffd640 1 -- 172.21.15.45:0/507073207 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315803e980 con 0x7f316015d150 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.378+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316015a400 con 0x7f316015d150 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.378+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316014ee20 con 0x7f316015d150 2023-11-28T20:36:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.380+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316015a8c0 con 0x7f316015d150 2023-11-28T20:36:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.380+0000 7f3156ffd640 1 -- 172.21.15.45:0/507073207 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f315803eae0 con 0x7f316015d150 2023-11-28T20:36:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.381+0000 7f3156ffd640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f314406b0d0 0x7f314406d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.382+0000 7f3156ffd640 1 -- 172.21.15.45:0/507073207 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f31580acc00 con 0x7f316015d150 2023-11-28T20:36:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.382+0000 7f316622c640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f314406b0d0 0x7f314406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.383+0000 7f316622c640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f314406b0d0 0x7f314406d590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f315c007fd0 tx=0x7f315c009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.386+0000 7f3156ffd640 1 -- 172.21.15.45:0/507073207 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3158077720 con 0x7f316015d150 2023-11-28T20:36:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.602+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f31600b8400 con 0x7f314406b0d0 2023-11-28T20:36:06.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.724+0000 7f3156ffd640 1 -- 172.21.15.45:0/507073207 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f315c0012a0 con 0x7f314406b0d0 2023-11-28T20:36:06.727 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-11-28T20:36:06.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.727+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f314406b0d0 msgr2=0x7f314406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.727+0000 7f316722e640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f314406b0d0 0x7f314406d590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f315c007fd0 tx=0x7f315c009040 comp rx=0 tx=0).stop 2023-11-28T20:36:06.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.727+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f316015d150 msgr2=0x7f31600cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.727+0000 7f316722e640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f316015d150 0x7f31600cc6e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3158036040 tx=0x7f3158004a40 comp rx=0 tx=0).stop 2023-11-28T20:36:06.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.728+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 shutdown_connections 2023-11-28T20:36:06.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.728+0000 7f316722e640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f314406b0d0 0x7f314406d590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.728+0000 7f316722e640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f316015d150 0x7f31600cc6e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.728+0000 7f316722e640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31600d23a0 0x7f3160159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.728+0000 7f316722e640 1 --2- 172.21.15.45:0/507073207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31600d0710 0x7f316014e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.728+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 >> 172.21.15.45:0/507073207 conn(0x7f3160070f40 msgr2=0x7f31600b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:06.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.728+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 shutdown_connections 2023-11-28T20:36:06.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.728+0000 7f316722e640 1 -- 172.21.15.45:0/507073207 wait complete. 2023-11-28T20:36:06.751 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-11-28T20:36:06.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2023-11-28T20:36:06.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.968+0000 7f808261a640 1 Processor -- start 2023-11-28T20:36:06.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.968+0000 7f808261a640 1 -- start start 2023-11-28T20:36:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.969+0000 7f808261a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f807c0af2c0 0x7f807c06b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.969+0000 7f808261a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f807c154420 0x7f807c06ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.969+0000 7f808261a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807c0ad920 con 0x7f807c154040 2023-11-28T20:36:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.969+0000 7f808261a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c160e30 con 0x7f807c0af2c0 2023-11-28T20:36:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.969+0000 7f808261a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c161110 con 0x7f807c154420 2023-11-28T20:36:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.970+0000 7f8081618640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f807c154040 0x7f807c0acf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46320/0 (socket says 172.21.15.45:46320) 2023-11-28T20:36:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.970+0000 7f8081618640 1 -- 172.21.15.45:0/3485170168 learned_addr learned my addr 172.21.15.45:0/3485170168 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:36:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.970+0000 7f8081e19640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f807c154420 0x7f807c06ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.970+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419609929 0 0) 0x7f807c0ad920 con 0x7f807c154040 2023-11-28T20:36:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.970+0000 7f8080e17640 1 --2- 172.21.15.45:0/3485170168 >> v2:172.21.15.190:3300/0 conn(0x7f807c0af2c0 0x7f807c06b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.970+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8064003520 con 0x7f807c154040 2023-11-28T20:36:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1231980685 0 0) 0x7f8064003520 con 0x7f807c154040 2023-11-28T20:36:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f807c154420 msgr2=0x7f807c06ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f807c154420 0x7f807c06ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 >> v2:172.21.15.190:3300/0 conn(0x7f807c0af2c0 msgr2=0x7f807c06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 --2- 172.21.15.45:0/3485170168 >> v2:172.21.15.190:3300/0 conn(0x7f807c0af2c0 0x7f807c06b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c1613f0 con 0x7f807c154040 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f806c0019f0 con 0x7f807c154040 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 >> v1:172.21.15.45:6789/0 conn(0x7f807c154040 legacy=0x7f807c0acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f807c154420 0x7f8064003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8064003ef0 0x7f80640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064004d00 0x7f8064005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c160e30 con 0x7f8064003ef0 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c161110 con 0x7f8064004d00 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806400ba80 con 0x7f807c154420 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f8080e17640 1 --2- 172.21.15.45:0/3485170168 >> v2:172.21.15.190:3300/0 conn(0x7f807c0af2c0 0x7f807c06b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f8080e17640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8064003ef0 0x7f80640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f8081618640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f807c154420 0x7f8064003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.971+0000 7f8081e19640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064004d00 0x7f8064005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.972+0000 7f8080e17640 1 -- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f807c154420 msgr2=0x7f8064003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:06.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.972+0000 7f8080e17640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f807c154420 0x7f8064003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.972+0000 7f8080e17640 1 -- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064004d00 msgr2=0x7f8064005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:06.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.972+0000 7f8080e17640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064004d00 0x7f8064005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.972+0000 7f8080e17640 1 -- 172.21.15.45:0/3485170168 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8064003520 con 0x7f8064003ef0 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.973+0000 7f8080e17640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8064003ef0 0x7f80640047c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f80700098d0 tx=0x7f80640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826be23a67f56416 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.973+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8070030680 con 0x7f8064003ef0 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.973+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80700307e0 con 0x7f8064003ef0 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.973+0000 7f807bfff640 1 -- 172.21.15.45:0/3485170168 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8070030b50 con 0x7f8064003ef0 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.974+0000 7f808261a640 1 -- 172.21.15.45:0/3485170168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8064003ef0 msgr2=0x7f80640047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.974+0000 7f808261a640 1 --2- 172.21.15.45:0/3485170168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8064003ef0 0x7f80640047c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f80700098d0 tx=0x7f80640057f0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.974+0000 7f8081e19640 1 -- 172.21.15.45:0/3485170168 reap_dead start 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.974+0000 7f808261a640 1 -- 172.21.15.45:0/3485170168 shutdown_connections 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.974+0000 7f808261a640 1 -- 172.21.15.45:0/3485170168 >> 172.21.15.45:0/3485170168 conn(0x7f807c071220 msgr2=0x7f807c071630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:06.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.974+0000 7f808261a640 1 -- 172.21.15.45:0/3485170168 shutdown_connections 2023-11-28T20:36:06.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.974+0000 7f808261a640 1 -- 172.21.15.45:0/3485170168 wait complete. 2023-11-28T20:36:06.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.976+0000 7f808261a640 1 Processor -- start 2023-11-28T20:36:06.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.976+0000 7f808261a640 1 -- start start 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.977+0000 7f808261a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8064003ef0 0x7f807c154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.977+0000 7f808261a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f807c155430 0x7f807c151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.977+0000 7f808261a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f807c155890 0x7f807c151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.977+0000 7f808261a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c06e010 con 0x7f807c155890 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.977+0000 7f808261a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c06e2a0 con 0x7f807c155430 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.977+0000 7f808261a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c06e580 con 0x7f8064003ef0 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.977+0000 7f8081e19640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f807c155890 0x7f807c151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.977+0000 7f8081e19640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f807c155890 0x7f807c151d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44794/0 (socket says 172.21.15.45:44794) 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.977+0000 7f8081e19640 1 -- 172.21.15.45:0/2925040779 learned_addr learned my addr 172.21.15.45:0/2925040779 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8081618640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8064003ef0 0x7f807c154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8080e17640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f807c155430 0x7f807c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8081e19640 1 -- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8064003ef0 msgr2=0x7f807c154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8081e19640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8064003ef0 0x7f807c154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8081e19640 1 -- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f807c155430 msgr2=0x7f807c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8081e19640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f807c155430 0x7f807c151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8081e19640 1 -- 172.21.15.45:0/2925040779 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c06e860 con 0x7f807c155890 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8081e19640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f807c155890 0x7f807c151d60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f807c0b2ea0 tx=0x7f8074011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8079ffb640 1 -- 172.21.15.45:0/2925040779 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074004500 con 0x7f807c155890 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8079ffb640 1 -- 172.21.15.45:0/2925040779 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8074003610 con 0x7f807c155890 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f8079ffb640 1 -- 172.21.15.45:0/2925040779 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80740039c0 con 0x7f807c155890 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.978+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c06eb40 con 0x7f807c155890 2023-11-28T20:36:06.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.979+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c1e9480 con 0x7f807c155890 2023-11-28T20:36:06.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.980+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c06f060 con 0x7f807c155890 2023-11-28T20:36:06.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.981+0000 7f8079ffb640 1 -- 172.21.15.45:0/2925040779 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f8074003b20 con 0x7f807c155890 2023-11-28T20:36:06.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.982+0000 7f8079ffb640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f806806b100 0x7f806806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:06.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.982+0000 7f8079ffb640 1 -- 172.21.15.45:0/2925040779 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f8074089a30 con 0x7f807c155890 2023-11-28T20:36:06.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.982+0000 7f8081618640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f806806b100 0x7f806806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:06.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.983+0000 7f8081618640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f806806b100 0x7f806806d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f806c004c40 tx=0x7f806c006040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:06.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:06.987+0000 7f8079ffb640 1 -- 172.21.15.45:0/2925040779 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f807c06f060 con 0x7f807c155890 2023-11-28T20:36:07.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.204+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f807c0adc00 con 0x7f806806b100 2023-11-28T20:36:07.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.339+0000 7f8079ffb640 1 -- 172.21.15.45:0/2925040779 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f806c004d30 con 0x7f806806b100 2023-11-28T20:36:07.341 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-11-28T20:36:07.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f806806b100 msgr2=0x7f806806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:07.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f806806b100 0x7f806806d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f806c004c40 tx=0x7f806c006040 comp rx=0 tx=0).stop 2023-11-28T20:36:07.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f807c155890 msgr2=0x7f807c151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:07.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f807c155890 0x7f807c151d60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f807c0b2ea0 tx=0x7f8074011920 comp rx=0 tx=0).stop 2023-11-28T20:36:07.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 shutdown_connections 2023-11-28T20:36:07.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f806806b100 0x7f806806d5c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f807c155890 0x7f807c151d60 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f807c155430 0x7f807c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 --2- 172.21.15.45:0/2925040779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8064003ef0 0x7f807c154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 >> 172.21.15.45:0/2925040779 conn(0x7f807c071220 msgr2=0x7f807c0b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:07.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 shutdown_connections 2023-11-28T20:36:07.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.343+0000 7f808261a640 1 -- 172.21.15.45:0/2925040779 wait complete. 2023-11-28T20:36:07.368 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-11-28T20:36:07.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2023-11-28T20:36:07.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.600+0000 7fd89c2c2640 1 Processor -- start 2023-11-28T20:36:07.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.600+0000 7fd89c2c2640 1 -- start start 2023-11-28T20:36:07.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89c2c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89415e840 0x7fd89406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:07.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89c2c2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd89415f720 0x7fd8940717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:07.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89c2c2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd89406a500 0x7fd89406a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:07.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89c2c2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd894071ce0 con 0x7fd89415f720 2023-11-28T20:36:07.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89c2c2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89406aed0 con 0x7fd89406a500 2023-11-28T20:36:07.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89c2c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89415ecd0 con 0x7fd89415e840 2023-11-28T20:36:07.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89985d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd89415f720 0x7fd8940717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:07.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89985d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd89415f720 0x7fd8940717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44798/0 (socket says 172.21.15.45:44798) 2023-11-28T20:36:07.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89985d640 1 -- 172.21.15.45:0/3021382045 learned_addr learned my addr 172.21.15.45:0/3021382045 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:07.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.601+0000 7fd89a05e640 1 --2- 172.21.15.45:0/3021382045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89415e840 0x7fd89406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:07.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.602+0000 7fd89985d640 1 -- 172.21.15.45:0/3021382045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89415e840 msgr2=0x7fd89406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:07.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.602+0000 7fd89985d640 1 --2- 172.21.15.45:0/3021382045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89415e840 0x7fd89406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.602+0000 7fd89985d640 1 -- 172.21.15.45:0/3021382045 >> v2:172.21.15.190:3300/0 conn(0x7fd89406a500 msgr2=0x7fd89406a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:07.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.602+0000 7fd89a85f640 1 --2- 172.21.15.45:0/3021382045 >> v2:172.21.15.190:3300/0 conn(0x7fd89406a500 0x7fd89406a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:07.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.602+0000 7fd89985d640 1 --2- 172.21.15.45:0/3021382045 >> v2:172.21.15.190:3300/0 conn(0x7fd89406a500 0x7fd89406a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.602+0000 7fd89985d640 1 -- 172.21.15.45:0/3021382045 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89415f290 con 0x7fd89415f720 2023-11-28T20:36:07.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.602+0000 7fd89a85f640 1 --2- 172.21.15.45:0/3021382045 >> v2:172.21.15.190:3300/0 conn(0x7fd89406a500 0x7fd89406a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:07.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.603+0000 7fd89985d640 1 --2- 172.21.15.45:0/3021382045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415f720 0x7fd8940717a0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd884009870 tx=0x7fd88402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6c26278ecbeb8f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:07.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.603+0000 7fd89905c640 1 -- 172.21.15.45:0/3021382045 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884031640 con 0x7fd89415f720 2023-11-28T20:36:07.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.603+0000 7fd89905c640 1 -- 172.21.15.45:0/3021382045 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8840317a0 con 0x7fd89415f720 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.603+0000 7fd89905c640 1 -- 172.21.15.45:0/3021382045 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884031b30 con 0x7fd89415f720 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.604+0000 7fd89c2c2640 1 -- 172.21.15.45:0/3021382045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415f720 msgr2=0x7fd8940717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.604+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/3021382045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415f720 0x7fd8940717a0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd884009870 tx=0x7fd88402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.604+0000 7fd89c2c2640 1 -- 172.21.15.45:0/3021382045 shutdown_connections 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.604+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/3021382045 >> v2:172.21.15.190:3300/0 conn(0x7fd89406a500 0x7fd89406a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.604+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/3021382045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415f720 0x7fd8940717a0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.604+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/3021382045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89415e840 0x7fd89406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.604+0000 7fd89c2c2640 1 -- 172.21.15.45:0/3021382045 >> 172.21.15.45:0/3021382045 conn(0x7fd8940ced60 msgr2=0x7fd8940cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.604+0000 7fd89c2c2640 1 -- 172.21.15.45:0/3021382045 shutdown_connections 2023-11-28T20:36:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.604+0000 7fd89c2c2640 1 -- 172.21.15.45:0/3021382045 wait complete. 2023-11-28T20:36:07.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.606+0000 7fd89c2c2640 1 Processor -- start 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.606+0000 7fd89c2c2640 1 -- start start 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89c2c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89406a500 0x7fd8941505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89c2c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415e840 0x7fd8940c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89c2c2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89415f720 0x7fd8940c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89c2c2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940c2f20 con 0x7fd89415e840 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89c2c2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd894150ae0 con 0x7fd89415f720 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89c2c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940c3170 con 0x7fd89406a500 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89985d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415e840 0x7fd8940c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89985d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415e840 0x7fd8940c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44812/0 (socket says 172.21.15.45:44812) 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89985d640 1 -- 172.21.15.45:0/2737729916 learned_addr learned my addr 172.21.15.45:0/2737729916 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.607+0000 7fd89a85f640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89415f720 0x7fd8940c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89a05e640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89406a500 0x7fd8941505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89985d640 1 -- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89406a500 msgr2=0x7fd8941505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89985d640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89406a500 0x7fd8941505a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89985d640 1 -- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89415f720 msgr2=0x7fd8940c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89985d640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89415f720 0x7fd8940c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89985d640 1 -- 172.21.15.45:0/2737729916 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8840094d0 con 0x7fd89415e840 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89a85f640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89415f720 0x7fd8940c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89985d640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415e840 0x7fd8940c7780 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd884000c00 tx=0x7fd8840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd88affd640 1 -- 172.21.15.45:0/2737729916 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88403ac60 con 0x7fd89415e840 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd88affd640 1 -- 172.21.15.45:0/2737729916 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88403adc0 con 0x7fd89415e840 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8940c3450 con 0x7fd89415e840 2023-11-28T20:36:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd88affd640 1 -- 172.21.15.45:0/2737729916 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884044630 con 0x7fd89415e840 2023-11-28T20:36:07.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.608+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8940c3a10 con 0x7fd89415e840 2023-11-28T20:36:07.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.609+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8940733d0 con 0x7fd89415e840 2023-11-28T20:36:07.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.611+0000 7fd88affd640 1 -- 172.21.15.45:0/2737729916 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7fd88402f050 con 0x7fd89415e840 2023-11-28T20:36:07.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.612+0000 7fd88affd640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd87806b100 0x7fd87806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:07.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.612+0000 7fd88affd640 1 -- 172.21.15.45:0/2737729916 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fd8840abd10 con 0x7fd89415e840 2023-11-28T20:36:07.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.614+0000 7fd89a05e640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd87806b100 0x7fd87806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:07.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.615+0000 7fd89a05e640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd87806b100 0x7fd87806d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd890006fd0 tx=0x7fd890008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:07.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.617+0000 7fd88affd640 1 -- 172.21.15.45:0/2737729916 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8840b1020 con 0x7fd89415e840 2023-11-28T20:36:07.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.831+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd89406c630 con 0x7fd87806b100 2023-11-28T20:36:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.959+0000 7fd88affd640 1 -- 172.21.15.45:0/2737729916 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd89406c630 con 0x7fd87806b100 2023-11-28T20:36:07.961 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-11-28T20:36:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd87806b100 msgr2=0x7fd87806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd87806b100 0x7fd87806d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd890006fd0 tx=0x7fd890008040 comp rx=0 tx=0).stop 2023-11-28T20:36:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415e840 msgr2=0x7fd8940c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415e840 0x7fd8940c7780 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd884000c00 tx=0x7fd8840046c0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 shutdown_connections 2023-11-28T20:36:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd87806b100 0x7fd87806d5c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89415f720 0x7fd8940c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd89415e840 0x7fd8940c7780 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 --2- 172.21.15.45:0/2737729916 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89406a500 0x7fd8941505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 >> 172.21.15.45:0/2737729916 conn(0x7fd8940ced60 msgr2=0x7fd8940ad4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.961+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 shutdown_connections 2023-11-28T20:36:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:07.962+0000 7fd89c2c2640 1 -- 172.21.15.45:0/2737729916 wait complete. 2023-11-28T20:36:07.983 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-11-28T20:36:07.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2023-11-28T20:36:08.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.205+0000 7fcd60e59640 1 Processor -- start 2023-11-28T20:36:08.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.205+0000 7fcd60e59640 1 -- start start 2023-11-28T20:36:08.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.206+0000 7fcd60e59640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcd5c1437b0 0x7fcd5c15abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.206+0000 7fcd60e59640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c15b110 0x7fcd5c0cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.206+0000 7fcd60e59640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5c071560 con 0x7fcd5c0ce1c0 2023-11-28T20:36:08.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.206+0000 7fcd60e59640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd5c0ce630 con 0x7fcd5c1437b0 2023-11-28T20:36:08.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.206+0000 7fcd60e59640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c071800 con 0x7fcd5c15b110 2023-11-28T20:36:08.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.206+0000 7fcd5affd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c15b110 0x7fcd5c0cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.208+0000 7fcd5affd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c15b110 0x7fcd5c0cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:42230/0 (socket says 172.21.15.45:42230) 2023-11-28T20:36:08.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.208+0000 7fcd5affd640 1 -- 172.21.15.45:0/3294271814 learned_addr learned my addr 172.21.15.45:0/3294271814 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:08.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.208+0000 7fcd5bfff640 1 --1- 172.21.15.45:0/3294271814 >> v1:172.21.15.190:6789/0 conn(0x7fcd5c0ce1c0 0x7fcd5c15dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:58744/0 (socket says 172.21.15.45:58744) 2023-11-28T20:36:08.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.208+0000 7fcd5b7fe640 1 --2- 172.21.15.45:0/3294271814 >> v2:172.21.15.190:3300/0 conn(0x7fcd5c1437b0 0x7fcd5c15abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.208+0000 7fcd5affd640 1 -- 172.21.15.45:0/3294271814 >> v2:172.21.15.190:3300/0 conn(0x7fcd5c1437b0 msgr2=0x7fcd5c15abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:08.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.208+0000 7fcd5affd640 1 --2- 172.21.15.45:0/3294271814 >> v2:172.21.15.190:3300/0 conn(0x7fcd5c1437b0 0x7fcd5c15abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.208+0000 7fcd5affd640 1 -- 172.21.15.45:0/3294271814 >> v1:172.21.15.190:6789/0 conn(0x7fcd5c0ce1c0 legacy=0x7fcd5c15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:08.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.208+0000 7fcd5affd640 1 -- 172.21.15.45:0/3294271814 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c071ae0 con 0x7fcd5c15b110 2023-11-28T20:36:08.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.208+0000 7fcd5b7fe640 1 --2- 172.21.15.45:0/3294271814 >> v2:172.21.15.190:3300/0 conn(0x7fcd5c1437b0 0x7fcd5c15abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:36:08.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd5affd640 1 --2- 172.21.15.45:0/3294271814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c15b110 0x7fcd5c0cdc80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009870 tx=0x7fcd5002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1216acbed5257ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:08.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd5a7fc640 1 -- 172.21.15.45:0/3294271814 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd50030840 con 0x7fcd5c15b110 2023-11-28T20:36:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd5a7fc640 1 -- 172.21.15.45:0/3294271814 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd500309a0 con 0x7fcd5c15b110 2023-11-28T20:36:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd5a7fc640 1 -- 172.21.15.45:0/3294271814 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd50030d30 con 0x7fcd5c15b110 2023-11-28T20:36:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd60e59640 1 -- 172.21.15.45:0/3294271814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c15b110 msgr2=0x7fcd5c0cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd60e59640 1 --2- 172.21.15.45:0/3294271814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c15b110 0x7fcd5c0cdc80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009870 tx=0x7fcd5002b980 comp rx=0 tx=0).stop 2023-11-28T20:36:08.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd60e59640 1 -- 172.21.15.45:0/3294271814 shutdown_connections 2023-11-28T20:36:08.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd60e59640 1 --2- 172.21.15.45:0/3294271814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c15b110 0x7fcd5c0cdc80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd60e59640 1 --2- 172.21.15.45:0/3294271814 >> v2:172.21.15.190:3300/0 conn(0x7fcd5c1437b0 0x7fcd5c15abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd60e59640 1 -- 172.21.15.45:0/3294271814 >> 172.21.15.45:0/3294271814 conn(0x7fcd5c0b1050 msgr2=0x7fcd5c06a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:08.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd60e59640 1 -- 172.21.15.45:0/3294271814 shutdown_connections 2023-11-28T20:36:08.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.210+0000 7fcd60e59640 1 -- 172.21.15.45:0/3294271814 wait complete. 2023-11-28T20:36:08.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd60e59640 1 Processor -- start 2023-11-28T20:36:08.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd60e59640 1 -- start start 2023-11-28T20:36:08.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd60e59640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c0ce1c0 0x7fcd5c153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd60e59640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd5c1437b0 0x7fcd5c153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd60e59640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd5c15b110 0x7fcd5c138f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd60e59640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c153fb0 con 0x7fcd5c1437b0 2023-11-28T20:36:08.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd60e59640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c1387e0 con 0x7fcd5c15b110 2023-11-28T20:36:08.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd60e59640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c141760 con 0x7fcd5c0ce1c0 2023-11-28T20:36:08.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd5affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd5c1437b0 0x7fcd5c153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.213+0000 7fcd5affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd5c1437b0 0x7fcd5c153a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44814/0 (socket says 172.21.15.45:44814) 2023-11-28T20:36:08.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.214+0000 7fcd5affd640 1 -- 172.21.15.45:0/2322303293 learned_addr learned my addr 172.21.15.45:0/2322303293 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:08.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.214+0000 7fcd5b7fe640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c0ce1c0 0x7fcd5c153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.214+0000 7fcd5bfff640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd5c15b110 0x7fcd5c138f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.214+0000 7fcd5affd640 1 -- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c0ce1c0 msgr2=0x7fcd5c153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:08.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.214+0000 7fcd5affd640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c0ce1c0 0x7fcd5c153530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.214+0000 7fcd5affd640 1 -- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd5c15b110 msgr2=0x7fcd5c138f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:08.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.214+0000 7fcd5affd640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd5c15b110 0x7fcd5c138f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.214+0000 7fcd5affd640 1 -- 172.21.15.45:0/2322303293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4c009400 con 0x7fcd5c1437b0 2023-11-28T20:36:08.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.214+0000 7fcd5affd640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd5c1437b0 0x7fcd5c153a70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fcd5002b9c0 tx=0x7fcd50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:08.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.215+0000 7fcd43fff640 1 -- 172.21.15.45:0/2322303293 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd50039a90 con 0x7fcd5c1437b0 2023-11-28T20:36:08.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.215+0000 7fcd43fff640 1 -- 172.21.15.45:0/2322303293 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd50039bf0 con 0x7fcd5c1437b0 2023-11-28T20:36:08.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.215+0000 7fcd43fff640 1 -- 172.21.15.45:0/2322303293 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd50043410 con 0x7fcd5c1437b0 2023-11-28T20:36:08.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.215+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd500094d0 con 0x7fcd5c1437b0 2023-11-28T20:36:08.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.215+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5c141ce0 con 0x7fcd5c1437b0 2023-11-28T20:36:08.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.216+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5c04e920 con 0x7fcd5c1437b0 2023-11-28T20:36:08.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.217+0000 7fcd43fff640 1 -- 172.21.15.45:0/2322303293 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7fcd5002f390 con 0x7fcd5c1437b0 2023-11-28T20:36:08.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.218+0000 7fcd43fff640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fcd4806b0d0 0x7fcd4806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.218+0000 7fcd43fff640 1 -- 172.21.15.45:0/2322303293 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fcd500abf90 con 0x7fcd5c1437b0 2023-11-28T20:36:08.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.219+0000 7fcd5b7fe640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fcd4806b0d0 0x7fcd4806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.220+0000 7fcd5b7fe640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fcd4806b0d0 0x7fcd4806d590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c009c60 tx=0x7fcd4c008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:08.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.222+0000 7fcd43fff640 1 -- 172.21.15.45:0/2322303293 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd50076a10 con 0x7fcd5c1437b0 2023-11-28T20:36:08.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.449+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd5c0720a0 con 0x7fcd4806b0d0 2023-11-28T20:36:08.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.577+0000 7fcd43fff640 1 -- 172.21.15.45:0/2322303293 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcd4c011e90 con 0x7fcd4806b0d0 2023-11-28T20:36:08.578 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fcd4806b0d0 msgr2=0x7fcd4806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fcd4806b0d0 0x7fcd4806d590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c009c60 tx=0x7fcd4c008040 comp rx=0 tx=0).stop 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd5c1437b0 msgr2=0x7fcd5c153a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd5c1437b0 0x7fcd5c153a70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fcd5002b9c0 tx=0x7fcd50004730 comp rx=0 tx=0).stop 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 shutdown_connections 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fcd4806b0d0 0x7fcd4806d590 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd5c15b110 0x7fcd5c138f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd5c1437b0 0x7fcd5c153a70 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 --2- 172.21.15.45:0/2322303293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd5c0ce1c0 0x7fcd5c153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 >> 172.21.15.45:0/2322303293 conn(0x7fcd5c0b1050 msgr2=0x7fcd5c06cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:08.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.580+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 shutdown_connections 2023-11-28T20:36:08.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.581+0000 7fcd60e59640 1 -- 172.21.15.45:0/2322303293 wait complete. 2023-11-28T20:36:08.602 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-11-28T20:36:08.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2023-11-28T20:36:08.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.841+0000 7f8baa3a9640 1 Processor -- start 2023-11-28T20:36:08.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.841+0000 7f8baa3a9640 1 -- start start 2023-11-28T20:36:08.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.843+0000 7f8baa3a9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40cda70 0x7f8ba40cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.843+0000 7f8baa3a9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ba40d0070 0x7f8ba40d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.843+0000 7f8baa3a9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba40cdf00 con 0x7f8ba415d420 2023-11-28T20:36:08.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.843+0000 7f8baa3a9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba40d0500 con 0x7f8ba40d0070 2023-11-28T20:36:08.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.843+0000 7f8baa3a9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba40ce1e0 con 0x7f8ba40cda70 2023-11-28T20:36:08.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.843+0000 7f8ba8946640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ba40d0070 0x7f8ba40d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.843+0000 7f8ba8946640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ba40d0070 0x7f8ba40d1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44828/0 (socket says 172.21.15.45:44828) 2023-11-28T20:36:08.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.843+0000 7f8ba8946640 1 -- 172.21.15.45:0/3276957665 learned_addr learned my addr 172.21.15.45:0/3276957665 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:08.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.843+0000 7f8ba37fe640 1 --2- 172.21.15.45:0/3276957665 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40cda70 0x7f8ba40cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.844+0000 7f8ba8946640 1 -- 172.21.15.45:0/3276957665 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40cda70 msgr2=0x7f8ba40cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:08.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.844+0000 7f8ba8946640 1 --2- 172.21.15.45:0/3276957665 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40cda70 0x7f8ba40cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.844+0000 7f8ba8946640 1 -- 172.21.15.45:0/3276957665 >> v1:172.21.15.190:6789/0 conn(0x7f8ba415d420 legacy=0x7f8ba4161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:36:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.844+0000 7f8ba8946640 1 -- 172.21.15.45:0/3276957665 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40ce4c0 con 0x7f8ba40d0070 2023-11-28T20:36:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.845+0000 7f8ba8946640 1 --2- 172.21.15.45:0/3276957665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba40d0070 0x7f8ba40d1aa0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c009870 tx=0x7f8b9c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40c743feb192060 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.845+0000 7f8ba2ffd640 1 -- 172.21.15.45:0/3276957665 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c031640 con 0x7f8ba40d0070 2023-11-28T20:36:08.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.845+0000 7f8ba2ffd640 1 -- 172.21.15.45:0/3276957665 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c0317a0 con 0x7f8ba40d0070 2023-11-28T20:36:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.845+0000 7f8ba2ffd640 1 -- 172.21.15.45:0/3276957665 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c031b30 con 0x7f8ba40d0070 2023-11-28T20:36:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.846+0000 7f8baa3a9640 1 -- 172.21.15.45:0/3276957665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba40d0070 msgr2=0x7f8ba40d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.846+0000 7f8baa3a9640 1 --2- 172.21.15.45:0/3276957665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba40d0070 0x7f8ba40d1aa0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c009870 tx=0x7f8b9c02b8f0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.846+0000 7f8baa3a9640 1 -- 172.21.15.45:0/3276957665 shutdown_connections 2023-11-28T20:36:08.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.846+0000 7f8baa3a9640 1 --2- 172.21.15.45:0/3276957665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba40d0070 0x7f8ba40d1aa0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.846+0000 7f8baa3a9640 1 --2- 172.21.15.45:0/3276957665 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40cda70 0x7f8ba40cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.846+0000 7f8baa3a9640 1 -- 172.21.15.45:0/3276957665 >> 172.21.15.45:0/3276957665 conn(0x7f8ba4070b20 msgr2=0x7f8ba4070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:08.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.846+0000 7f8baa3a9640 1 -- 172.21.15.45:0/3276957665 shutdown_connections 2023-11-28T20:36:08.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.846+0000 7f8baa3a9640 1 -- 172.21.15.45:0/3276957665 wait complete. 2023-11-28T20:36:08.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.848+0000 7f8baa3a9640 1 Processor -- start 2023-11-28T20:36:08.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.848+0000 7f8baa3a9640 1 -- start start 2023-11-28T20:36:08.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.849+0000 7f8baa3a9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40d0070 0x7f8ba40cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.849+0000 7f8baa3a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba415d420 0x7f8ba40cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.849+0000 7f8baa3a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba4149050 0x7f8ba415f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.849+0000 7f8baa3a9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba40cd250 con 0x7f8ba4149050 2023-11-28T20:36:08.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.849+0000 7f8baa3a9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba41e9150 con 0x7f8ba415d420 2023-11-28T20:36:08.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.849+0000 7f8baa3a9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba41e9390 con 0x7f8ba40d0070 2023-11-28T20:36:08.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.849+0000 7f8ba8946640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba4149050 0x7f8ba415f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.850+0000 7f8ba8946640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba4149050 0x7f8ba415f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44838/0 (socket says 172.21.15.45:44838) 2023-11-28T20:36:08.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.850+0000 7f8ba8946640 1 -- 172.21.15.45:0/2807135080 learned_addr learned my addr 172.21.15.45:0/2807135080 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:36:08.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.850+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40d0070 0x7f8ba40cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.850+0000 7f8ba8946640 1 -- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40d0070 msgr2=0x7f8ba40cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:08.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.850+0000 7f8ba8946640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40d0070 0x7f8ba40cbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.850+0000 7f8ba8946640 1 -- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba415d420 msgr2=0x7f8ba40cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:08.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.850+0000 7f8ba8946640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba415d420 0x7f8ba40cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:08.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.850+0000 7f8ba8946640 1 -- 172.21.15.45:0/2807135080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c0094d0 con 0x7f8ba4149050 2023-11-28T20:36:08.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.850+0000 7f8ba8946640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba4149050 0x7f8ba415f6c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c02b930 tx=0x7f8b9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:08.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.851+0000 7f8ba0ff9640 1 -- 172.21.15.45:0/2807135080 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c03ac60 con 0x7f8ba4149050 2023-11-28T20:36:08.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.851+0000 7f8ba0ff9640 1 -- 172.21.15.45:0/2807135080 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c03adc0 con 0x7f8ba4149050 2023-11-28T20:36:08.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.851+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba41e9670 con 0x7f8ba4149050 2023-11-28T20:36:08.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.851+0000 7f8ba0ff9640 1 -- 172.21.15.45:0/2807135080 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c044630 con 0x7f8ba4149050 2023-11-28T20:36:08.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.851+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba40cc8b0 con 0x7f8ba4149050 2023-11-28T20:36:08.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.852+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba40733d0 con 0x7f8ba4149050 2023-11-28T20:36:08.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.853+0000 7f8ba0ff9640 1 -- 172.21.15.45:0/2807135080 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f8b9c02f050 con 0x7f8ba4149050 2023-11-28T20:36:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.854+0000 7f8ba0ff9640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8b9006b0d0 0x7f8b9006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:36:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.854+0000 7f8ba0ff9640 1 -- 172.21.15.45:0/2807135080 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f8b9c0abdb0 con 0x7f8ba4149050 2023-11-28T20:36:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.854+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8b9006b0d0 0x7f8b9006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:36:08.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.856+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8b9006b0d0 0x7f8b9006d590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009f20 tx=0x7f8b940094d0 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:36:08.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:08.858+0000 7f8ba0ff9640 1 -- 172.21.15.45:0/2807135080 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b9c0b0070 con 0x7f8ba4149050 2023-11-28T20:36:09.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.077+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ba40b2860 con 0x7f8b9006b0d0 2023-11-28T20:36:09.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.203+0000 7f8ba0ff9640 1 -- 172.21.15.45:0/2807135080 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f8b94009680 con 0x7f8b9006b0d0 2023-11-28T20:36:09.204 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-11-28T20:36:09.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.206+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8b9006b0d0 msgr2=0x7f8b9006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:09.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.206+0000 7f8baa3a9640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8b9006b0d0 0x7f8b9006d590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009f20 tx=0x7f8b940094d0 comp rx=0 tx=0).stop 2023-11-28T20:36:09.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.206+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba4149050 msgr2=0x7f8ba415f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:36:09.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.206+0000 7f8baa3a9640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba4149050 0x7f8ba415f6c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c02b930 tx=0x7f8b9c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:36:09.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.207+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 shutdown_connections 2023-11-28T20:36:09.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.207+0000 7f8baa3a9640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8b9006b0d0 0x7f8b9006d590 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:09.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.207+0000 7f8baa3a9640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ba4149050 0x7f8ba415f6c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:09.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.207+0000 7f8baa3a9640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba415d420 0x7f8ba40cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:09.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.207+0000 7f8baa3a9640 1 --2- 172.21.15.45:0/2807135080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ba40d0070 0x7f8ba40cbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:36:09.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.207+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 >> 172.21.15.45:0/2807135080 conn(0x7f8ba4070b20 msgr2=0x7f8ba4070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:36:09.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.207+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 shutdown_connections 2023-11-28T20:36:09.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:36:09.207+0000 7f8baa3a9640 1 -- 172.21.15.45:0/2807135080 wait complete. 2023-11-28T20:36:09.230 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-11-28T20:36:15.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:36:15.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:36:45.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:36:45.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:37:15.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:37:15.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:37:45.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:37:45.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:38:15.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:38:15.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:38:45.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:38:45.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:39:16.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:39:16.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:39:46.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:39:46.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:40:01.317 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:40:01.316+0000 7f5537c65640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:40:01.317 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:40:01.316+0000 7f5537c65640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:40:01.317 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:40:01.316+0000 7f5537c65640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:40:01.317 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:40:01.316+0000 7f5537c65640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:40:01.317 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:40:01.316+0000 7f5537c65640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:40:16.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:40:16.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:40:46.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:40:46.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:41:16.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:41:16.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:41:46.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:41:46.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:42:17.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:42:17.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:42:47.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:42:47.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:43:17.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:43:17.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:43:47.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:43:47.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:44:17.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:44:17.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:44:47.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:44:47.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:45:17.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:45:17.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:45:48.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:45:48.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:46:18.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:46:18.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:46:48.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:46:48.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:47:18.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:47:18.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:47:21.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2023-11-28T20:47:21.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.471+0000 7f913dfb6640 1 Processor -- start 2023-11-28T20:47:21.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.471+0000 7f913dfb6640 1 -- start start 2023-11-28T20:47:21.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.473+0000 7f913dfb6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9138159780 0x7f913815d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:21.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.473+0000 7f913dfb6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91380cd6b0 con 0x7f91380c9dc0 2023-11-28T20:47:21.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.473+0000 7f913dfb6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91380cd990 con 0x7f91380ae030 2023-11-28T20:47:21.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.473+0000 7f913dfb6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91380ce330 con 0x7f9138159780 2023-11-28T20:47:21.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.473+0000 7f9137fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f91380c9dc0 0x7f91380ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39948/0 (socket says 172.21.15.45:39948) 2023-11-28T20:47:21.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.473+0000 7f9137fff640 1 -- 172.21.15.45:0/1057281112 learned_addr learned my addr 172.21.15.45:0/1057281112 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:21.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.473+0000 7f91377fe640 1 --2- 172.21.15.45:0/1057281112 >> v2:172.21.15.45:3300/0 conn(0x7f9138159780 0x7f913815d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:21.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.474+0000 7f91367fc640 1 -- 172.21.15.45:0/1057281112 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697339526 0 0) 0x7f91380cd6b0 con 0x7f91380c9dc0 2023-11-28T20:47:21.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.474+0000 7f91367fc640 1 -- 172.21.15.45:0/1057281112 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9124003520 con 0x7f91380c9dc0 2023-11-28T20:47:21.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.474+0000 7f91377fe640 1 -- 172.21.15.45:0/1057281112 >> v1:172.21.15.190:6789/0 conn(0x7f91380ae030 legacy=0x7f913815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:21.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.474+0000 7f91377fe640 1 -- 172.21.15.45:0/1057281112 >> v1:172.21.15.45:6789/0 conn(0x7f91380c9dc0 legacy=0x7f91380ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:21.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.474+0000 7f91377fe640 1 -- 172.21.15.45:0/1057281112 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91380cdc70 con 0x7f9138159780 2023-11-28T20:47:21.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.475+0000 7f91377fe640 1 --2- 172.21.15.45:0/1057281112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 0x7f913815d390 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9128009d90 tx=0x7f912802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273fcb987da08f24 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:21.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.475+0000 7f91367fc640 1 -- 172.21.15.45:0/1057281112 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912802cd00 con 0x7f9138159780 2023-11-28T20:47:21.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.475+0000 7f91367fc640 1 -- 172.21.15.45:0/1057281112 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912802ce60 con 0x7f9138159780 2023-11-28T20:47:21.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.475+0000 7f91367fc640 1 -- 172.21.15.45:0/1057281112 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128034660 con 0x7f9138159780 2023-11-28T20:47:21.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.476+0000 7f913dfb6640 1 -- 172.21.15.45:0/1057281112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 msgr2=0x7f913815d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:21.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.476+0000 7f913dfb6640 1 --2- 172.21.15.45:0/1057281112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 0x7f913815d390 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9128009d90 tx=0x7f912802bf50 comp rx=0 tx=0).stop 2023-11-28T20:47:21.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.476+0000 7f913dfb6640 1 -- 172.21.15.45:0/1057281112 shutdown_connections 2023-11-28T20:47:21.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.476+0000 7f913dfb6640 1 --2- 172.21.15.45:0/1057281112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 0x7f913815d390 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:21.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.476+0000 7f913dfb6640 1 -- 172.21.15.45:0/1057281112 >> 172.21.15.45:0/1057281112 conn(0x7f91380ced60 msgr2=0x7f91380cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:21.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.476+0000 7f913dfb6640 1 -- 172.21.15.45:0/1057281112 shutdown_connections 2023-11-28T20:47:21.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.476+0000 7f913dfb6640 1 -- 172.21.15.45:0/1057281112 wait complete. 2023-11-28T20:47:21.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.478+0000 7f913dfb6640 1 Processor -- start 2023-11-28T20:47:21.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.478+0000 7f913dfb6640 1 -- start start 2023-11-28T20:47:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.479+0000 7f913dfb6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91380ae030 0x7f9138149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.479+0000 7f913dfb6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91380c9dc0 0x7f91380cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.479+0000 7f913dfb6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 0x7f91380cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.479+0000 7f913dfb6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9138099de0 con 0x7f9138159780 2023-11-28T20:47:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.479+0000 7f913dfb6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91380cc6c0 con 0x7f91380ae030 2023-11-28T20:47:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.479+0000 7f913dfb6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9138147880 con 0x7f91380c9dc0 2023-11-28T20:47:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.479+0000 7f9137fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 0x7f91380cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.479+0000 7f9137fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 0x7f91380cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49062/0 (socket says 172.21.15.45:49062) 2023-11-28T20:47:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.479+0000 7f9137fff640 1 -- 172.21.15.45:0/3661477876 learned_addr learned my addr 172.21.15.45:0/3661477876 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:21.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f91377fe640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91380ae030 0x7f9138149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:21.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f9136ffd640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91380c9dc0 0x7f91380cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:21.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f9137fff640 1 -- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91380c9dc0 msgr2=0x7f91380cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:21.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f9137fff640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91380c9dc0 0x7f91380cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:21.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f9137fff640 1 -- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91380ae030 msgr2=0x7f9138149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:21.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f9137fff640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91380ae030 0x7f9138149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:21.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f9137fff640 1 -- 172.21.15.45:0/3661477876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128009a70 con 0x7f9138159780 2023-11-28T20:47:21.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f9136ffd640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91380c9dc0 0x7f91380cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:21.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f9137fff640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 0x7f91380cbfb0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f913000c3c0 tx=0x7f913000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:21.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f912ffff640 1 -- 172.21.15.45:0/3661477876 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130005ce0 con 0x7f9138159780 2023-11-28T20:47:21.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f912ffff640 1 -- 172.21.15.45:0/3661477876 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9130005e40 con 0x7f9138159780 2023-11-28T20:47:21.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.480+0000 7f912ffff640 1 -- 172.21.15.45:0/3661477876 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130007310 con 0x7f9138159780 2023-11-28T20:47:21.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.481+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9138146ac0 con 0x7f9138159780 2023-11-28T20:47:21.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.481+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9138147080 con 0x7f9138159780 2023-11-28T20:47:21.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.482+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91380733d0 con 0x7f9138159780 2023-11-28T20:47:21.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.483+0000 7f912ffff640 1 -- 172.21.15.45:0/3661477876 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f9130007470 con 0x7f9138159780 2023-11-28T20:47:21.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.484+0000 7f912ffff640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f911806b040 0x7f911806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:21.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.484+0000 7f912ffff640 1 -- 172.21.15.45:0/3661477876 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f9130089270 con 0x7f9138159780 2023-11-28T20:47:21.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.484+0000 7f91377fe640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f911806b040 0x7f911806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:21.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.486+0000 7f91377fe640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f911806b040 0x7f911806d500 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9128036000 tx=0x7f9128030000 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:21.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.489+0000 7f912ffff640 1 -- 172.21.15.45:0/3661477876 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9130053da0 con 0x7f9138159780 2023-11-28T20:47:21.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.712+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9138159b60 con 0x7f911806b040 2023-11-28T20:47:21.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.894+0000 7f912ffff640 1 -- 172.21.15.45:0/3661477876 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9128009450 con 0x7f911806b040 2023-11-28T20:47:21.896 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-11-28T20:47:21.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.897+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f911806b040 msgr2=0x7f911806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:21.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.897+0000 7f913dfb6640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f911806b040 0x7f911806d500 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9128036000 tx=0x7f9128030000 comp rx=0 tx=0).stop 2023-11-28T20:47:21.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 msgr2=0x7f91380cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:21.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 0x7f91380cbfb0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f913000c3c0 tx=0x7f913000c880 comp rx=0 tx=0).stop 2023-11-28T20:47:21.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 shutdown_connections 2023-11-28T20:47:21.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f911806b040 0x7f911806d500 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:21.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9138159780 0x7f91380cbfb0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:21.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91380c9dc0 0x7f91380cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:21.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 --2- 172.21.15.45:0/3661477876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91380ae030 0x7f9138149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:21.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 >> 172.21.15.45:0/3661477876 conn(0x7f91380ced60 msgr2=0x7f913806d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:21.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 shutdown_connections 2023-11-28T20:47:21.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:21.898+0000 7f913dfb6640 1 -- 172.21.15.45:0/3661477876 wait complete. 2023-11-28T20:47:21.923 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-11-28T20:47:21.924 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2023-11-28T20:47:22.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.158+0000 7f72f5aff640 1 Processor -- start 2023-11-28T20:47:22.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.158+0000 7f72f5aff640 1 -- start start 2023-11-28T20:47:22.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.160+0000 7f72f5aff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72f00b3040 0x7f72f006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.160+0000 7f72f5aff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f015b270 0x7f72f015f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.160+0000 7f72f5aff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f72f00b3420 0x7f72f00b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.160+0000 7f72f5aff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f00ceaa0 con 0x7f72f00b3040 2023-11-28T20:47:22.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.160+0000 7f72f5aff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f015faf0 con 0x7f72f00b3420 2023-11-28T20:47:22.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.160+0000 7f72f5aff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f006d410 con 0x7f72f015b270 2023-11-28T20:47:22.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.160+0000 7f72f4afd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72f00b3040 0x7f72f006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72f4afd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f72f00b3040 0x7f72f006cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49070/0 (socket says 172.21.15.45:49070) 2023-11-28T20:47:22.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72f4afd640 1 -- 172.21.15.45:0/3453303980 learned_addr learned my addr 172.21.15.45:0/3453303980 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:22.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72f52fe640 1 --2- 172.21.15.45:0/3453303980 >> v2:172.21.15.190:3300/0 conn(0x7f72f00b3420 0x7f72f00b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72e7fff640 1 --2- 172.21.15.45:0/3453303980 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f015b270 0x7f72f015f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72f4afd640 1 -- 172.21.15.45:0/3453303980 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f015b270 msgr2=0x7f72f015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:22.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72f4afd640 1 --2- 172.21.15.45:0/3453303980 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f015b270 0x7f72f015f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72f4afd640 1 -- 172.21.15.45:0/3453303980 >> v2:172.21.15.190:3300/0 conn(0x7f72f00b3420 msgr2=0x7f72f00b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:22.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72f4afd640 1 --2- 172.21.15.45:0/3453303980 >> v2:172.21.15.190:3300/0 conn(0x7f72f00b3420 0x7f72f00b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72f4afd640 1 -- 172.21.15.45:0/3453303980 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f006d6f0 con 0x7f72f00b3040 2023-11-28T20:47:22.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.161+0000 7f72f52fe640 1 --2- 172.21.15.45:0/3453303980 >> v2:172.21.15.190:3300/0 conn(0x7f72f00b3420 0x7f72f00b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:47:22.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.162+0000 7f72f4afd640 1 --2- 172.21.15.45:0/3453303980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 0x7f72f006cda0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f72f00b1a10 tx=0x7f72ec02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec272407cac40509 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:22.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.163+0000 7f72e77fe640 1 -- 172.21.15.45:0/3453303980 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec031750 con 0x7f72f00b3040 2023-11-28T20:47:22.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.163+0000 7f72e77fe640 1 -- 172.21.15.45:0/3453303980 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec0318b0 con 0x7f72f00b3040 2023-11-28T20:47:22.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.163+0000 7f72e77fe640 1 -- 172.21.15.45:0/3453303980 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec031c40 con 0x7f72f00b3040 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.163+0000 7f72f5aff640 1 -- 172.21.15.45:0/3453303980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 msgr2=0x7f72f006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.163+0000 7f72f5aff640 1 --2- 172.21.15.45:0/3453303980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 0x7f72f006cda0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f72f00b1a10 tx=0x7f72ec02b7f0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.164+0000 7f72f5aff640 1 -- 172.21.15.45:0/3453303980 shutdown_connections 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.164+0000 7f72f5aff640 1 --2- 172.21.15.45:0/3453303980 >> v2:172.21.15.190:3300/0 conn(0x7f72f00b3420 0x7f72f00b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.164+0000 7f72f5aff640 1 --2- 172.21.15.45:0/3453303980 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f015b270 0x7f72f015f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.164+0000 7f72f5aff640 1 --2- 172.21.15.45:0/3453303980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 0x7f72f006cda0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.164+0000 7f72f5aff640 1 -- 172.21.15.45:0/3453303980 >> 172.21.15.45:0/3453303980 conn(0x7f72f00b1050 msgr2=0x7f72f00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.164+0000 7f72f5aff640 1 -- 172.21.15.45:0/3453303980 shutdown_connections 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.164+0000 7f72f5aff640 1 -- 172.21.15.45:0/3453303980 wait complete. 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.165+0000 7f72f5aff640 1 Processor -- start 2023-11-28T20:47:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.166+0000 7f72f5aff640 1 -- start start 2023-11-28T20:47:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f5aff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 0x7f72f00cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f5aff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f00b3420 0x7f72f014c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f5aff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f015b270 0x7f72f015d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f5aff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f0070bf0 con 0x7f72f00b3040 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f5aff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f00cd9e0 con 0x7f72f015b270 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f5aff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f014e280 con 0x7f72f00b3420 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f4afd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 0x7f72f00cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f4afd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 0x7f72f00cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49086/0 (socket says 172.21.15.45:49086) 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f4afd640 1 -- 172.21.15.45:0/2959289835 learned_addr learned my addr 172.21.15.45:0/2959289835 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72e7fff640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f00b3420 0x7f72f014c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f52fe640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f015b270 0x7f72f015d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f4afd640 1 -- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f00b3420 msgr2=0x7f72f014c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f4afd640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f00b3420 0x7f72f014c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f4afd640 1 -- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f015b270 msgr2=0x7f72f015d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f4afd640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f015b270 0x7f72f015d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f4afd640 1 -- 172.21.15.45:0/2959289835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0094d0 con 0x7f72f00b3040 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.167+0000 7f72f52fe640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f015b270 0x7f72f015d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.168+0000 7f72f4afd640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 0x7f72f00cd4a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f72ec000c00 tx=0x7f72ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.168+0000 7f72e57fa640 1 -- 172.21.15.45:0/2959289835 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec033c80 con 0x7f72f00b3040 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.168+0000 7f72e57fa640 1 -- 172.21.15.45:0/2959289835 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec033e00 con 0x7f72f00b3040 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.168+0000 7f72e57fa640 1 -- 172.21.15.45:0/2959289835 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec03c640 con 0x7f72f00b3040 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.168+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f014fa50 con 0x7f72f00b3040 2023-11-28T20:47:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.168+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f0070e40 con 0x7f72f00b3040 2023-11-28T20:47:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.170+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f009d0f0 con 0x7f72f00b3040 2023-11-28T20:47:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.171+0000 7f72e57fa640 1 -- 172.21.15.45:0/2959289835 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f72ec03a040 con 0x7f72f00b3040 2023-11-28T20:47:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.171+0000 7f72e57fa640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f72dc06b070 0x7f72dc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.172+0000 7f72e57fa640 1 -- 172.21.15.45:0/2959289835 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f72ec0ab760 con 0x7f72f00b3040 2023-11-28T20:47:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.172+0000 7f72e7fff640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f72dc06b070 0x7f72dc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.173+0000 7f72e7fff640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f72dc06b070 0x7f72dc06d530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f72f015ac80 tx=0x7f72e0008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.176+0000 7f72e57fa640 1 -- 172.21.15.45:0/2959289835 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72ec076320 con 0x7f72f00b3040 2023-11-28T20:47:22.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.396+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f72f006df90 con 0x7f72dc06b070 2023-11-28T20:47:22.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.565+0000 7f72e57fa640 1 -- 172.21.15.45:0/2959289835 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f72e0009520 con 0x7f72dc06b070 2023-11-28T20:47:22.568 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-11-28T20:47:22.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f72dc06b070 msgr2=0x7f72dc06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f72dc06b070 0x7f72dc06d530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f72f015ac80 tx=0x7f72e0008040 comp rx=0 tx=0).stop 2023-11-28T20:47:22.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 msgr2=0x7f72f00cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 0x7f72f00cd4a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f72ec000c00 tx=0x7f72ec0046c0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 shutdown_connections 2023-11-28T20:47:22.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f72dc06b070 0x7f72dc06d530 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f015b270 0x7f72f015d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f72f00b3420 0x7f72f014c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 --2- 172.21.15.45:0/2959289835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72f00b3040 0x7f72f00cd4a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 >> 172.21.15.45:0/2959289835 conn(0x7f72f00b1050 msgr2=0x7f72f01569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:22.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 shutdown_connections 2023-11-28T20:47:22.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.569+0000 7f72f5aff640 1 -- 172.21.15.45:0/2959289835 wait complete. 2023-11-28T20:47:22.593 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-11-28T20:47:22.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2023-11-28T20:47:22.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.816+0000 7f8ae597c640 1 Processor -- start 2023-11-28T20:47:22.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.816+0000 7f8ae597c640 1 -- start start 2023-11-28T20:47:22.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.817+0000 7f8ae597c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae006dc50 0x7f8ae00cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.818+0000 7f8ae597c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae006e030 con 0x7f8ae00ce600 2023-11-28T20:47:22.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.818+0000 7f8ae597c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae00ccf60 con 0x7f8ae006ac10 2023-11-28T20:47:22.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.818+0000 7f8ae597c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00cd930 con 0x7f8ae006dc50 2023-11-28T20:47:22.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.818+0000 7f8ad7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8ae00ce600 0x7f8ae006a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39958/0 (socket says 172.21.15.45:39958) 2023-11-28T20:47:22.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.818+0000 7f8ad7fff640 1 -- 172.21.15.45:0/2694531979 learned_addr learned my addr 172.21.15.45:0/2694531979 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:22.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.818+0000 7f8adeffd640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae006dc50 0x7f8ae00cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.818+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003179234 0 0) 0x7f8ac8001110 con 0x7f8ae00ce600 2023-11-28T20:47:22.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8acc003520 con 0x7f8ae00ce600 2023-11-28T20:47:22.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2210805140 0 0) 0x7f8acc003520 con 0x7f8ae00ce600 2023-11-28T20:47:22.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae006dc50 msgr2=0x7f8ae00cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:22.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae006dc50 0x7f8ae00cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 >> v1:172.21.15.190:6789/0 conn(0x7f8ae006ac10 legacy=0x7f8ae00c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae00cc930 con 0x7f8ae00ce600 2023-11-28T20:47:22.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f8ac8001800 con 0x7f8ae00ce600 2023-11-28T20:47:22.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 >> v1:172.21.15.45:6789/0 conn(0x7f8ae00ce600 legacy=0x7f8ae006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0039a0 0x7f8acc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8acc0047c0 0x7f8acc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae006dc50 0x7f8acc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00cd930 con 0x7f8acc0047c0 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc00ba60 con 0x7f8acc0039a0 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc00bc20 con 0x7f8ae006dc50 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.819+0000 7f8ad7fff640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8acc0047c0 0x7f8acc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.820+0000 7f8adeffd640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0039a0 0x7f8acc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.820+0000 7f8ad7fff640 1 -- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae006dc50 msgr2=0x7f8acc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.820+0000 7f8ad7fff640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae006dc50 0x7f8acc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.820+0000 7f8ad7fff640 1 -- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0039a0 msgr2=0x7f8acc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:22.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.820+0000 7f8ad7fff640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0039a0 0x7f8acc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.820+0000 7f8ad7fff640 1 -- 172.21.15.45:0/2694531979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc003520 con 0x7f8acc0047c0 2023-11-28T20:47:22.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.820+0000 7f8adeffd640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0039a0 0x7f8acc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.821+0000 7f8ad7fff640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8acc0047c0 0x7f8acc004c40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8001eb0 tx=0x7f8acc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32624c555d7a8832 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.821+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8005ad0 con 0x7f8acc0047c0 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.821+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac8005c50 con 0x7f8acc0047c0 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.821+0000 7f8ade7fc640 1 -- 172.21.15.45:0/2694531979 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac802c790 con 0x7f8acc0047c0 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.822+0000 7f8ae597c640 1 -- 172.21.15.45:0/2694531979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8acc0047c0 msgr2=0x7f8acc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.822+0000 7f8ae597c640 1 --2- 172.21.15.45:0/2694531979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8acc0047c0 0x7f8acc004c40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8001eb0 tx=0x7f8acc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.822+0000 7f8adf7fe640 1 -- 172.21.15.45:0/2694531979 reap_dead start 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.822+0000 7f8ae597c640 1 -- 172.21.15.45:0/2694531979 shutdown_connections 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.822+0000 7f8ae597c640 1 -- 172.21.15.45:0/2694531979 >> 172.21.15.45:0/2694531979 conn(0x7f8ae00ced60 msgr2=0x7f8ae00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.822+0000 7f8ae597c640 1 -- 172.21.15.45:0/2694531979 shutdown_connections 2023-11-28T20:47:22.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.822+0000 7f8ae597c640 1 -- 172.21.15.45:0/2694531979 wait complete. 2023-11-28T20:47:22.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.824+0000 7f8ae597c640 1 Processor -- start 2023-11-28T20:47:22.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.824+0000 7f8ae597c640 1 -- start start 2023-11-28T20:47:22.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.825+0000 7f8ae597c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0047c0 0x7f8ae00c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.825+0000 7f8ae597c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae015b7c0 0x7f8ae015bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.825+0000 7f8ae597c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae009b5e0 0x7f8ae009ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.825+0000 7f8ae597c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae009a580 con 0x7f8ae015b7c0 2023-11-28T20:47:22.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.825+0000 7f8ae597c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae009a820 con 0x7f8acc0047c0 2023-11-28T20:47:22.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.825+0000 7f8ae597c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae009ab00 con 0x7f8ae009b5e0 2023-11-28T20:47:22.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.825+0000 7f8ad7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae015b7c0 0x7f8ae015bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.825+0000 7f8ad7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae015b7c0 0x7f8ae015bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49098/0 (socket says 172.21.15.45:49098) 2023-11-28T20:47:22.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.825+0000 7f8ad7fff640 1 -- 172.21.15.45:0/2655071635 learned_addr learned my addr 172.21.15.45:0/2655071635 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:22.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.826+0000 7f8adeffd640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0047c0 0x7f8ae00c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.826+0000 7f8adf7fe640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae009b5e0 0x7f8ae009ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.826+0000 7f8ad7fff640 1 -- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae009b5e0 msgr2=0x7f8ae009ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.826+0000 7f8ad7fff640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae009b5e0 0x7f8ae009ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.826+0000 7f8ad7fff640 1 -- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0047c0 msgr2=0x7f8ae00c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:22.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.826+0000 7f8ad7fff640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0047c0 0x7f8ae00c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:22.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.826+0000 7f8ad7fff640 1 -- 172.21.15.45:0/2655071635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae009ade0 con 0x7f8ae015b7c0 2023-11-28T20:47:22.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.826+0000 7f8ad7fff640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae015b7c0 0x7f8ae015bc00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8ac802c330 tx=0x7f8ac8001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:22.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.827+0000 7f8adcff9640 1 -- 172.21.15.45:0/2655071635 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8039070 con 0x7f8ae015b7c0 2023-11-28T20:47:22.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.827+0000 7f8adcff9640 1 -- 172.21.15.45:0/2655071635 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac8000f10 con 0x7f8ae015b7c0 2023-11-28T20:47:22.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.827+0000 7f8adcff9640 1 -- 172.21.15.45:0/2655071635 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac803c040 con 0x7f8ae015b7c0 2023-11-28T20:47:22.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.827+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae009b0c0 con 0x7f8ae015b7c0 2023-11-28T20:47:22.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.827+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae00c8bd0 con 0x7f8ae015b7c0 2023-11-28T20:47:22.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.828+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae00733d0 con 0x7f8ae015b7c0 2023-11-28T20:47:22.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.829+0000 7f8adcff9640 1 -- 172.21.15.45:0/2655071635 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f8ac8032070 con 0x7f8ae015b7c0 2023-11-28T20:47:22.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.830+0000 7f8adcff9640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8ac006b040 0x7f8ac006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:22.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.830+0000 7f8adcff9640 1 -- 172.21.15.45:0/2655071635 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f8ac80aaea0 con 0x7f8ae015b7c0 2023-11-28T20:47:22.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.831+0000 7f8adeffd640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8ac006b040 0x7f8ac006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:22.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.832+0000 7f8adeffd640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8ac006b040 0x7f8ac006d500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009ac0 tx=0x7f8ad0008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:22.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:22.834+0000 7f8adcff9640 1 -- 172.21.15.45:0/2655071635 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ac80759d0 con 0x7f8ae015b7c0 2023-11-28T20:47:23.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.049+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ae0070880 con 0x7f8ac006b040 2023-11-28T20:47:23.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.232+0000 7f8adcff9640 1 -- 172.21.15.45:0/2655071635 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f8ad0009710 con 0x7f8ac006b040 2023-11-28T20:47:23.235 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-11-28T20:47:23.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.236+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8ac006b040 msgr2=0x7f8ac006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:23.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.236+0000 7f8ae597c640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8ac006b040 0x7f8ac006d500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009ac0 tx=0x7f8ad0008040 comp rx=0 tx=0).stop 2023-11-28T20:47:23.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.236+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae015b7c0 msgr2=0x7f8ae015bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:23.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.236+0000 7f8ae597c640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae015b7c0 0x7f8ae015bc00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8ac802c330 tx=0x7f8ac8001ac0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.237+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 shutdown_connections 2023-11-28T20:47:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.237+0000 7f8ae597c640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f8ac006b040 0x7f8ac006d500 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.237+0000 7f8ae597c640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ae009b5e0 0x7f8ae009ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.237+0000 7f8ae597c640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae015b7c0 0x7f8ae015bc00 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.237+0000 7f8ae597c640 1 --2- 172.21.15.45:0/2655071635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc0047c0 0x7f8ae00c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.237+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 >> 172.21.15.45:0/2655071635 conn(0x7f8ae00ced60 msgr2=0x7f8ae006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.237+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 shutdown_connections 2023-11-28T20:47:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.237+0000 7f8ae597c640 1 -- 172.21.15.45:0/2655071635 wait complete. 2023-11-28T20:47:23.262 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-11-28T20:47:23.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2023-11-28T20:47:23.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.497+0000 7f22effff640 1 Processor -- start 2023-11-28T20:47:23.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.497+0000 7f22effff640 1 -- start start 2023-11-28T20:47:23.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.499+0000 7f22effff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b39d0 0x7f22f00717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:23.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.499+0000 7f22effff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f22f00cb630 0x7f22f00cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:23.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.499+0000 7f22effff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f00724d0 con 0x7f22f00b3db0 2023-11-28T20:47:23.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.499+0000 7f22effff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f0071d20 con 0x7f22f00cb630 2023-11-28T20:47:23.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.499+0000 7f22effff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f00cc000 con 0x7f22f00b39d0 2023-11-28T20:47:23.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.500+0000 7f22eeffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b39d0 0x7f22f00717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.500+0000 7f22ef7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f22f00cb630 0x7f22f00cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.500+0000 7f22eeffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b39d0 0x7f22f00717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:53176/0 (socket says 172.21.15.45:53176) 2023-11-28T20:47:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.500+0000 7f22eeffd640 1 -- 172.21.15.45:0/3357231772 learned_addr learned my addr 172.21.15.45:0/3357231772 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.500+0000 7f22ef7fe640 1 -- 172.21.15.45:0/3357231772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b39d0 msgr2=0x7f22f00717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.500+0000 7f22ef7fe640 1 --2- 172.21.15.45:0/3357231772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b39d0 0x7f22f00717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.500+0000 7f22ef7fe640 1 -- 172.21.15.45:0/3357231772 >> v1:172.21.15.190:6789/0 conn(0x7f22f00b3db0 legacy=0x7f22f006bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.500+0000 7f22ef7fe640 1 -- 172.21.15.45:0/3357231772 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f006c670 con 0x7f22f00cb630 2023-11-28T20:47:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.500+0000 7f22eeffd640 1 --2- 172.21.15.45:0/3357231772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b39d0 0x7f22f00717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:47:23.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.502+0000 7f22ef7fe640 1 --2- 172.21.15.45:0/3357231772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 0x7f22f00cbac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f22e8009d90 tx=0x7f22e802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d91dad0af2241560 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:23.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.503+0000 7f22ee7fc640 1 -- 172.21.15.45:0/3357231772 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e802cbc0 con 0x7f22f00cb630 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.503+0000 7f22ee7fc640 1 -- 172.21.15.45:0/3357231772 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e802cd20 con 0x7f22f00cb630 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.503+0000 7f22ee7fc640 1 -- 172.21.15.45:0/3357231772 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8034660 con 0x7f22f00cb630 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.503+0000 7f22effff640 1 -- 172.21.15.45:0/3357231772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 msgr2=0x7f22f00cbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.503+0000 7f22effff640 1 --2- 172.21.15.45:0/3357231772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 0x7f22f00cbac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f22e8009d90 tx=0x7f22e802be10 comp rx=0 tx=0).stop 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.503+0000 7f22effff640 1 -- 172.21.15.45:0/3357231772 shutdown_connections 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.503+0000 7f22effff640 1 --2- 172.21.15.45:0/3357231772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 0x7f22f00cbac0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.503+0000 7f22effff640 1 --2- 172.21.15.45:0/3357231772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b39d0 0x7f22f00717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.503+0000 7f22effff640 1 -- 172.21.15.45:0/3357231772 >> 172.21.15.45:0/3357231772 conn(0x7f22f00b1050 msgr2=0x7f22f00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.504+0000 7f22effff640 1 -- 172.21.15.45:0/3357231772 shutdown_connections 2023-11-28T20:47:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.504+0000 7f22effff640 1 -- 172.21.15.45:0/3357231772 wait complete. 2023-11-28T20:47:23.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.505+0000 7f22effff640 1 Processor -- start 2023-11-28T20:47:23.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.506+0000 7f22effff640 1 -- start start 2023-11-28T20:47:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22f00b39d0 0x7f22f00cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22effff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b3db0 0x7f22f015bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22effff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 0x7f22f015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22effff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f015ce50 con 0x7f22f00b39d0 2023-11-28T20:47:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22effff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f015c890 con 0x7f22f00cb630 2023-11-28T20:47:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22effff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f015cb70 con 0x7f22f00b3db0 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22f00b39d0 0x7f22f00cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22f00b39d0 0x7f22f00cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49112/0 (socket says 172.21.15.45:49112) 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22eeffd640 1 -- 172.21.15.45:0/3073870518 learned_addr learned my addr 172.21.15.45:0/3073870518 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22ef7fe640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 0x7f22f015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22e7fff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b3db0 0x7f22f015bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22eeffd640 1 -- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b3db0 msgr2=0x7f22f015bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22eeffd640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b3db0 0x7f22f015bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22eeffd640 1 -- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 msgr2=0x7f22f015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22eeffd640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 0x7f22f015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22eeffd640 1 -- 172.21.15.45:0/3073870518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e00094d0 con 0x7f22f00b39d0 2023-11-28T20:47:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22ef7fe640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 0x7f22f015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.507+0000 7f22e7fff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b3db0 0x7f22f015bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.508+0000 7f22eeffd640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22f00b39d0 0x7f22f00cce00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f22e000e9d0 tx=0x7f22e000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.508+0000 7f22e77fe640 1 -- 172.21.15.45:0/3073870518 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e000ca60 con 0x7f22f00b39d0 2023-11-28T20:47:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.508+0000 7f22e77fe640 1 -- 172.21.15.45:0/3073870518 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e000cbc0 con 0x7f22f00b39d0 2023-11-28T20:47:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.508+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e8009a70 con 0x7f22f00b39d0 2023-11-28T20:47:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.508+0000 7f22e77fe640 1 -- 172.21.15.45:0/3073870518 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e0013470 con 0x7f22f00b39d0 2023-11-28T20:47:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.508+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f0162af0 con 0x7f22f00b39d0 2023-11-28T20:47:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.509+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f0161b70 con 0x7f22f00b39d0 2023-11-28T20:47:23.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.510+0000 7f22e77fe640 1 -- 172.21.15.45:0/3073870518 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f22e00135d0 con 0x7f22f00b39d0 2023-11-28T20:47:23.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.511+0000 7f22e77fe640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f22d006b040 0x7f22d006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:23.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.511+0000 7f22e77fe640 1 -- 172.21.15.45:0/3073870518 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f22e008a2b0 con 0x7f22f00b39d0 2023-11-28T20:47:23.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.512+0000 7f22e7fff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f22d006b040 0x7f22d006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:23.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.513+0000 7f22e7fff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f22d006b040 0x7f22d006d500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f22f0164500 tx=0x7f22d8009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:23.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.515+0000 7f22e77fe640 1 -- 172.21.15.45:0/3073870518 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22e0054e70 con 0x7f22f00b39d0 2023-11-28T20:47:23.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.734+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f22f0071f10 con 0x7f22d006b040 2023-11-28T20:47:23.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.918+0000 7f22e77fe640 1 -- 172.21.15.45:0/3073870518 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f22d8001d30 con 0x7f22d006b040 2023-11-28T20:47:23.920 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-11-28T20:47:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.921+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f22d006b040 msgr2=0x7f22d006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.921+0000 7f22effff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f22d006b040 0x7f22d006d500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f22f0164500 tx=0x7f22d8009040 comp rx=0 tx=0).stop 2023-11-28T20:47:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.921+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22f00b39d0 msgr2=0x7f22f00cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.921+0000 7f22effff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22f00b39d0 0x7f22f00cce00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f22e000e9d0 tx=0x7f22e000ee90 comp rx=0 tx=0).stop 2023-11-28T20:47:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.921+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 shutdown_connections 2023-11-28T20:47:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.921+0000 7f22effff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f22d006b040 0x7f22d006d500 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.921+0000 7f22effff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22f00cb630 0x7f22f015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.921+0000 7f22effff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22f00b3db0 0x7f22f015bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.922+0000 7f22effff640 1 --2- 172.21.15.45:0/3073870518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22f00b39d0 0x7f22f00cce00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:23.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.922+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 >> 172.21.15.45:0/3073870518 conn(0x7f22f00b1050 msgr2=0x7f22f006cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:23.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.922+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 shutdown_connections 2023-11-28T20:47:23.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:23.922+0000 7f22effff640 1 -- 172.21.15.45:0/3073870518 wait complete. 2023-11-28T20:47:23.947 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-11-28T20:47:23.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2023-11-28T20:47:24.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.169+0000 7fa144701640 1 Processor -- start 2023-11-28T20:47:24.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.169+0000 7fa144701640 1 -- start start 2023-11-28T20:47:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.170+0000 7fa144701640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15e840 0x7fa13c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.170+0000 7fa144701640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa13c15f720 0x7fa13c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.170+0000 7fa144701640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c15ecd0 con 0x7fa13c06a500 2023-11-28T20:47:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.170+0000 7fa144701640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa13c071ce0 con 0x7fa13c15f720 2023-11-28T20:47:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.170+0000 7fa144701640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c15f290 con 0x7fa13c15e840 2023-11-28T20:47:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.171+0000 7fa141c9c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa13c15f720 0x7fa13c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.171+0000 7fa141c9c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa13c15f720 0x7fa13c0717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57190/0 (socket says 172.21.15.45:57190) 2023-11-28T20:47:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.171+0000 7fa141c9c640 1 -- 172.21.15.45:0/1207342288 learned_addr learned my addr 172.21.15.45:0/1207342288 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.171+0000 7fa14249d640 1 --2- 172.21.15.45:0/1207342288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15e840 0x7fa13c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.171+0000 7fa141c9c640 1 -- 172.21.15.45:0/1207342288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15e840 msgr2=0x7fa13c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.171+0000 7fa141c9c640 1 --2- 172.21.15.45:0/1207342288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15e840 0x7fa13c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.171+0000 7fa141c9c640 1 -- 172.21.15.45:0/1207342288 >> v1:172.21.15.45:6789/0 conn(0x7fa13c06a500 legacy=0x7fa13c0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.171+0000 7fa141c9c640 1 -- 172.21.15.45:0/1207342288 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c15fb60 con 0x7fa13c15f720 2023-11-28T20:47:24.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.171+0000 7fa14249d640 1 --2- 172.21.15.45:0/1207342288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15e840 0x7fa13c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:24.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.172+0000 7fa141c9c640 1 --2- 172.21.15.45:0/1207342288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15f720 0x7fa13c0717a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0098a0 tx=0x7fa12c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618ac619b57dfc2b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.173+0000 7fa14149b640 1 -- 172.21.15.45:0/1207342288 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c030840 con 0x7fa13c15f720 2023-11-28T20:47:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.173+0000 7fa14149b640 1 -- 172.21.15.45:0/1207342288 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12c0309a0 con 0x7fa13c15f720 2023-11-28T20:47:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.173+0000 7fa14149b640 1 -- 172.21.15.45:0/1207342288 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c030d30 con 0x7fa13c15f720 2023-11-28T20:47:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.173+0000 7fa144701640 1 -- 172.21.15.45:0/1207342288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15f720 msgr2=0x7fa13c0717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.173+0000 7fa144701640 1 --2- 172.21.15.45:0/1207342288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15f720 0x7fa13c0717a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0098a0 tx=0x7fa12c02b980 comp rx=0 tx=0).stop 2023-11-28T20:47:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.173+0000 7fa144701640 1 -- 172.21.15.45:0/1207342288 shutdown_connections 2023-11-28T20:47:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.174+0000 7fa144701640 1 --2- 172.21.15.45:0/1207342288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15f720 0x7fa13c0717a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.174+0000 7fa144701640 1 --2- 172.21.15.45:0/1207342288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15e840 0x7fa13c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.174+0000 7fa144701640 1 -- 172.21.15.45:0/1207342288 >> 172.21.15.45:0/1207342288 conn(0x7fa13c0ced60 msgr2=0x7fa13c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:24.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.174+0000 7fa144701640 1 -- 172.21.15.45:0/1207342288 shutdown_connections 2023-11-28T20:47:24.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.174+0000 7fa144701640 1 -- 172.21.15.45:0/1207342288 wait complete. 2023-11-28T20:47:24.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.175+0000 7fa144701640 1 Processor -- start 2023-11-28T20:47:24.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.176+0000 7fa144701640 1 -- start start 2023-11-28T20:47:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.176+0000 7fa144701640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa13c06a500 0x7fa13c156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.176+0000 7fa144701640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15e840 0x7fa13c0afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa144701640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15f720 0x7fa13c0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa144701640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c0b3f50 con 0x7fa13c15e840 2023-11-28T20:47:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa144701640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c0b3340 con 0x7fa13c06a500 2023-11-28T20:47:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa144701640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c0b3620 con 0x7fa13c15f720 2023-11-28T20:47:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa141c9c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15e840 0x7fa13c0afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa141c9c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15e840 0x7fa13c0afd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57192/0 (socket says 172.21.15.45:57192) 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa141c9c640 1 -- 172.21.15.45:0/2651479484 learned_addr learned my addr 172.21.15.45:0/2651479484 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa14249d640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa13c06a500 0x7fa13c156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa142c9e640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15f720 0x7fa13c0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa141c9c640 1 -- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15f720 msgr2=0x7fa13c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa141c9c640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15f720 0x7fa13c0b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa141c9c640 1 -- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa13c06a500 msgr2=0x7fa13c156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa141c9c640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa13c06a500 0x7fa13c156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa141c9c640 1 -- 172.21.15.45:0/2651479484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12c0094d0 con 0x7fa13c15e840 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.177+0000 7fa142c9e640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15f720 0x7fa13c0b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.178+0000 7fa141c9c640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15e840 0x7fa13c0afd90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0097c0 tx=0x7fa12c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:24.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.178+0000 7fa132ffd640 1 -- 172.21.15.45:0/2651479484 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c039c20 con 0x7fa13c15e840 2023-11-28T20:47:24.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.178+0000 7fa132ffd640 1 -- 172.21.15.45:0/2651479484 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12c039d80 con 0x7fa13c15e840 2023-11-28T20:47:24.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.178+0000 7fa132ffd640 1 -- 172.21.15.45:0/2651479484 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c043650 con 0x7fa13c15e840 2023-11-28T20:47:24.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.178+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13c0b3900 con 0x7fa13c15e840 2023-11-28T20:47:24.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.178+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c156840 con 0x7fa13c15e840 2023-11-28T20:47:24.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.179+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13c0b40c0 con 0x7fa13c15e840 2023-11-28T20:47:24.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.180+0000 7fa132ffd640 1 -- 172.21.15.45:0/2651479484 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7fa12c02f390 con 0x7fa13c15e840 2023-11-28T20:47:24.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.181+0000 7fa132ffd640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa12006b040 0x7fa12006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.181+0000 7fa132ffd640 1 -- 172.21.15.45:0/2651479484 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fa12c0abff0 con 0x7fa13c15e840 2023-11-28T20:47:24.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.182+0000 7fa14249d640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa12006b040 0x7fa12006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.183+0000 7fa14249d640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa12006b040 0x7fa12006d500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa13c071160 tx=0x7fa138008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:24.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.186+0000 7fa132ffd640 1 -- 172.21.15.45:0/2651479484 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa12c076b30 con 0x7fa13c15e840 2023-11-28T20:47:24.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.410+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa13c15ef20 con 0x7fa12006b040 2023-11-28T20:47:24.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.583+0000 7fa132ffd640 1 -- 172.21.15.45:0/2651479484 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa13800a1f0 con 0x7fa12006b040 2023-11-28T20:47:24.586 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa12006b040 msgr2=0x7fa12006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa12006b040 0x7fa12006d500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa13c071160 tx=0x7fa138008040 comp rx=0 tx=0).stop 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15e840 msgr2=0x7fa13c0afd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15e840 0x7fa13c0afd90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0097c0 tx=0x7fa12c004730 comp rx=0 tx=0).stop 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 shutdown_connections 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa12006b040 0x7fa12006d500 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa13c15f720 0x7fa13c0b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa13c15e840 0x7fa13c0afd90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 --2- 172.21.15.45:0/2651479484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa13c06a500 0x7fa13c156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.587+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 >> 172.21.15.45:0/2651479484 conn(0x7fa13c0ced60 msgr2=0x7fa13c0ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:24.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.588+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 shutdown_connections 2023-11-28T20:47:24.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.588+0000 7fa144701640 1 -- 172.21.15.45:0/2651479484 wait complete. 2023-11-28T20:47:24.611 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-11-28T20:47:24.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2023-11-28T20:47:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.837+0000 7fe8e3fff640 1 Processor -- start 2023-11-28T20:47:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.837+0000 7fe8e3fff640 1 -- start start 2023-11-28T20:47:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.839+0000 7fe8e3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e415d4b0 0x7fe8e4070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.839+0000 7fe8e3fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e40708d0 con 0x7fe8e40adfa0 2023-11-28T20:47:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.839+0000 7fe8e3fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e40b3bb0 con 0x7fe8e40712a0 2023-11-28T20:47:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.839+0000 7fe8e3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40b3e90 con 0x7fe8e415d4b0 2023-11-28T20:47:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.840+0000 7fe8e37fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe8e40adfa0 0x7fe8e40717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55340/0 (socket says 172.21.15.45:55340) 2023-11-28T20:47:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.840+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2510722721 learned_addr learned my addr 172.21.15.45:0/2510722721 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:24.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.840+0000 7fe8e2ffd640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e415d4b0 0x7fe8e4070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.840+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725219320 0 0) 0x7fe8e40708d0 con 0x7fe8e40adfa0 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.840+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d0003520 con 0x7fe8e40adfa0 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3619702321 0 0) 0x7fe8d0003520 con 0x7fe8e40adfa0 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e415d4b0 msgr2=0x7fe8e4070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e415d4b0 0x7fe8e4070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 >> v1:172.21.15.190:6789/0 conn(0x7fe8e40712a0 legacy=0x7fe8e40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e4070280 con 0x7fe8e40adfa0 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fe8dc0019f0 con 0x7fe8e40adfa0 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 >> v1:172.21.15.45:6789/0 conn(0x7fe8e40adfa0 legacy=0x7fe8e40717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8d00039a0 0x7fe8d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d00047c0 0x7fe8d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e415d4b0 0x7fe8d0005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40b3e90 con 0x7fe8d00047c0 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d000bb10 con 0x7fe8d00039a0 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d000bcd0 con 0x7fe8e415d4b0 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.841+0000 7fe8dbfff640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d00047c0 0x7fe8d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.842+0000 7fe8e2ffd640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8d00039a0 0x7fe8d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.842+0000 7fe8e37fe640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e415d4b0 0x7fe8d0005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.842+0000 7fe8dbfff640 1 -- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e415d4b0 msgr2=0x7fe8d0005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.842+0000 7fe8dbfff640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e415d4b0 0x7fe8d0005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.842+0000 7fe8dbfff640 1 -- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8d00039a0 msgr2=0x7fe8d0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.842+0000 7fe8dbfff640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8d00039a0 0x7fe8d0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.842+0000 7fe8dbfff640 1 -- 172.21.15.45:0/2510722721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0003520 con 0x7fe8d00047c0 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.843+0000 7fe8dbfff640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d00047c0 0x7fe8d0004c40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc001750 tx=0x7fe8d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b6f46d6bf6b917a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.843+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc0052d0 con 0x7fe8d00047c0 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.843+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8cc02e050 con 0x7fe8d00047c0 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.843+0000 7fe8e27fc640 1 -- 172.21.15.45:0/2510722721 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc003830 con 0x7fe8d00047c0 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.844+0000 7fe8e3fff640 1 -- 172.21.15.45:0/2510722721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d00047c0 msgr2=0x7fe8d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.844+0000 7fe8e3fff640 1 --2- 172.21.15.45:0/2510722721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d00047c0 0x7fe8d0004c40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc001750 tx=0x7fe8d00057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.844+0000 7fe8e37fe640 1 -- 172.21.15.45:0/2510722721 reap_dead start 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.844+0000 7fe8e3fff640 1 -- 172.21.15.45:0/2510722721 shutdown_connections 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.844+0000 7fe8e3fff640 1 -- 172.21.15.45:0/2510722721 >> 172.21.15.45:0/2510722721 conn(0x7fe8e40b1050 msgr2=0x7fe8e40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.844+0000 7fe8e3fff640 1 -- 172.21.15.45:0/2510722721 shutdown_connections 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.844+0000 7fe8e3fff640 1 -- 172.21.15.45:0/2510722721 wait complete. 2023-11-28T20:47:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.846+0000 7fe8e3fff640 1 Processor -- start 2023-11-28T20:47:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.846+0000 7fe8e3fff640 1 -- start start 2023-11-28T20:47:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.847+0000 7fe8e3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8d00047c0 0x7fe8e4143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.847+0000 7fe8e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e40aed90 0x7fe8e4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.847+0000 7fe8e3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40b02f0 0x7fe8e406d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.847+0000 7fe8e3fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e4143cc0 con 0x7fe8e40aed90 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.847+0000 7fe8e3fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40afe60 con 0x7fe8e40b02f0 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.847+0000 7fe8e3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e4099f80 con 0x7fe8d00047c0 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.847+0000 7fe8dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e40aed90 0x7fe8e4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e40aed90 0x7fe8e4099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57214/0 (socket says 172.21.15.45:57214) 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8dbfff640 1 -- 172.21.15.45:0/21396647 learned_addr learned my addr 172.21.15.45:0/21396647 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8e2ffd640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8d00047c0 0x7fe8e4143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8e37fe640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40b02f0 0x7fe8e406d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8dbfff640 1 -- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8d00047c0 msgr2=0x7fe8e4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8dbfff640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8d00047c0 0x7fe8e4143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8dbfff640 1 -- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40b02f0 msgr2=0x7fe8e406d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8dbfff640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40b02f0 0x7fe8e406d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8dbfff640 1 -- 172.21.15.45:0/21396647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e4153e20 con 0x7fe8e40aed90 2023-11-28T20:47:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.848+0000 7fe8dbfff640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e40aed90 0x7fe8e4099a40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc001870 tx=0x7fe8cc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.849+0000 7fe8db7fe640 1 -- 172.21.15.45:0/21396647 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc007640 con 0x7fe8e40aed90 2023-11-28T20:47:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.849+0000 7fe8db7fe640 1 -- 172.21.15.45:0/21396647 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8cc0069e0 con 0x7fe8e40aed90 2023-11-28T20:47:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.849+0000 7fe8db7fe640 1 -- 172.21.15.45:0/21396647 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc006d90 con 0x7fe8e40aed90 2023-11-28T20:47:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.849+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e40af250 con 0x7fe8e40aed90 2023-11-28T20:47:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.849+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e40af810 con 0x7fe8e40aed90 2023-11-28T20:47:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.850+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e409d0f0 con 0x7fe8e40aed90 2023-11-28T20:47:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.851+0000 7fe8db7fe640 1 -- 172.21.15.45:0/21396647 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7fe8cc0035f0 con 0x7fe8e40aed90 2023-11-28T20:47:24.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.852+0000 7fe8db7fe640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fe8c406b040 0x7fe8c406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:24.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.853+0000 7fe8db7fe640 1 -- 172.21.15.45:0/21396647 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fe8cc0aba00 con 0x7fe8e40aed90 2023-11-28T20:47:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.853+0000 7fe8e2ffd640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fe8c406b040 0x7fe8c406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.854+0000 7fe8e2ffd640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fe8c406b040 0x7fe8c406d500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4009a10 tx=0x7fe8d4008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:24.857+0000 7fe8db7fe640 1 -- 172.21.15.45:0/21396647 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8cc076420 con 0x7fe8e40aed90 2023-11-28T20:47:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.074+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8e40b4170 con 0x7fe8c406b040 2023-11-28T20:47:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.257+0000 7fe8db7fe640 1 -- 172.21.15.45:0/21396647 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe8d4013960 con 0x7fe8c406b040 2023-11-28T20:47:25.259 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-11-28T20:47:25.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fe8c406b040 msgr2=0x7fe8c406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fe8c406b040 0x7fe8c406d500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4009a10 tx=0x7fe8d4008040 comp rx=0 tx=0).stop 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e40aed90 msgr2=0x7fe8e4099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e40aed90 0x7fe8e4099a40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc001870 tx=0x7fe8cc001120 comp rx=0 tx=0).stop 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 shutdown_connections 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fe8c406b040 0x7fe8c406d500 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40b02f0 0x7fe8e406d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e40aed90 0x7fe8e4099a40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 --2- 172.21.15.45:0/21396647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8d00047c0 0x7fe8e4143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 >> 172.21.15.45:0/21396647 conn(0x7fe8e40b1050 msgr2=0x7fe8e40cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.260+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 shutdown_connections 2023-11-28T20:47:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.261+0000 7fe8e3fff640 1 -- 172.21.15.45:0/21396647 wait complete. 2023-11-28T20:47:25.284 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-11-28T20:47:25.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2023-11-28T20:47:25.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.507+0000 7f5036c64640 1 Processor -- start 2023-11-28T20:47:25.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.507+0000 7f5036c64640 1 -- start start 2023-11-28T20:47:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.508+0000 7f5036c64640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 0x7f50300cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.508+0000 7f5036c64640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f50300ce600 0x7f503006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.508+0000 7f5036c64640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503006e030 con 0x7f503006aa40 2023-11-28T20:47:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.508+0000 7f5036c64640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f503006aed0 con 0x7f50300ce600 2023-11-28T20:47:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.508+0000 7f5036c64640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50300ccf60 con 0x7f503006dc50 2023-11-28T20:47:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.509+0000 7f5034a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 0x7f50300cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.509+0000 7f502ffff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f50300ce600 0x7f503006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.509+0000 7f502ffff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f50300ce600 0x7f503006a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:55710/0 (socket says 172.21.15.45:55710) 2023-11-28T20:47:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.509+0000 7f502ffff640 1 -- 172.21.15.45:0/3555804379 learned_addr learned my addr 172.21.15.45:0/3555804379 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.510+0000 7f502ffff640 1 -- 172.21.15.45:0/3555804379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 msgr2=0x7f50300cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.510+0000 7f502ffff640 1 --2- 172.21.15.45:0/3555804379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 0x7f50300cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.510+0000 7f502ffff640 1 -- 172.21.15.45:0/3555804379 >> v1:172.21.15.190:6789/0 conn(0x7f503006aa40 legacy=0x7f50300c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.510+0000 7f502ffff640 1 -- 172.21.15.45:0/3555804379 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50300cd930 con 0x7f50300ce600 2023-11-28T20:47:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.512+0000 7f502ffff640 1 --2- 172.21.15.45:0/3555804379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50300ce600 0x7f503006a500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f501c009870 tx=0x7f501c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a953ee26e546eab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:25.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.512+0000 7f502f7fe640 1 -- 172.21.15.45:0/3555804379 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c030840 con 0x7f50300ce600 2023-11-28T20:47:25.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.512+0000 7f502f7fe640 1 -- 172.21.15.45:0/3555804379 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501c0309a0 con 0x7f50300ce600 2023-11-28T20:47:25.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.512+0000 7f502f7fe640 1 -- 172.21.15.45:0/3555804379 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c030d30 con 0x7f50300ce600 2023-11-28T20:47:25.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.513+0000 7f5036c64640 1 -- 172.21.15.45:0/3555804379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50300ce600 msgr2=0x7f503006a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:25.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.513+0000 7f5036c64640 1 --2- 172.21.15.45:0/3555804379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50300ce600 0x7f503006a500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f501c009870 tx=0x7f501c02b980 comp rx=0 tx=0).stop 2023-11-28T20:47:25.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.513+0000 7f5036c64640 1 -- 172.21.15.45:0/3555804379 shutdown_connections 2023-11-28T20:47:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.513+0000 7f5036c64640 1 --2- 172.21.15.45:0/3555804379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50300ce600 0x7f503006a500 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.513+0000 7f5036c64640 1 --2- 172.21.15.45:0/3555804379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 0x7f50300cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.513+0000 7f5036c64640 1 -- 172.21.15.45:0/3555804379 >> 172.21.15.45:0/3555804379 conn(0x7f50300ced60 msgr2=0x7f50300cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.513+0000 7f5036c64640 1 -- 172.21.15.45:0/3555804379 shutdown_connections 2023-11-28T20:47:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.513+0000 7f5036c64640 1 -- 172.21.15.45:0/3555804379 wait complete. 2023-11-28T20:47:25.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.515+0000 7f5036c64640 1 Processor -- start 2023-11-28T20:47:25.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.515+0000 7f5036c64640 1 -- start start 2023-11-28T20:47:25.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.516+0000 7f5036c64640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503006aa40 0x7f50300b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:25.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.516+0000 7f5036c64640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 0x7f503014c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:25.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.516+0000 7f5036c64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50300ce600 0x7f503014c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.516+0000 7f5036c64640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50300c5b90 con 0x7f50300ce600 2023-11-28T20:47:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.516+0000 7f5036c64640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50300b3f70 con 0x7f503006aa40 2023-11-28T20:47:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.516+0000 7f5036c64640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50300c6210 con 0x7f503006dc50 2023-11-28T20:47:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.516+0000 7f5035201640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50300ce600 0x7f503014c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5035201640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50300ce600 0x7f503014c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57218/0 (socket says 172.21.15.45:57218) 2023-11-28T20:47:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5035201640 1 -- 172.21.15.45:0/836533305 learned_addr learned my addr 172.21.15.45:0/836533305 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5034a00640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503006aa40 0x7f50300b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f502ffff640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 0x7f503014c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5035201640 1 -- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 msgr2=0x7f503014c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5035201640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 0x7f503014c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5035201640 1 -- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503006aa40 msgr2=0x7f50300b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5035201640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503006aa40 0x7f50300b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5035201640 1 -- 172.21.15.45:0/836533305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c0094d0 con 0x7f50300ce600 2023-11-28T20:47:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5034a00640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503006aa40 0x7f50300b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f502ffff640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 0x7f503014c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.517+0000 7f5035201640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50300ce600 0x7f503014c680 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f502800c730 tx=0x7f502800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.518+0000 7f502d7fa640 1 -- 172.21.15.45:0/836533305 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028003b70 con 0x7f50300ce600 2023-11-28T20:47:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.518+0000 7f502d7fa640 1 -- 172.21.15.45:0/836533305 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5028003cf0 con 0x7f50300ce600 2023-11-28T20:47:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.518+0000 7f502d7fa640 1 -- 172.21.15.45:0/836533305 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028003ea0 con 0x7f50300ce600 2023-11-28T20:47:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.518+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50300c64f0 con 0x7f50300ce600 2023-11-28T20:47:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.518+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50300b3510 con 0x7f50300ce600 2023-11-28T20:47:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.519+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50300733d0 con 0x7f50300ce600 2023-11-28T20:47:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.520+0000 7f502d7fa640 1 -- 172.21.15.45:0/836533305 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f5028005520 con 0x7f50300ce600 2023-11-28T20:47:25.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.521+0000 7f502d7fa640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f501406b040 0x7f501406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:25.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.521+0000 7f502d7fa640 1 -- 172.21.15.45:0/836533305 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f5028089a80 con 0x7f50300ce600 2023-11-28T20:47:25.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.522+0000 7f5034a00640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f501406b040 0x7f501406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.522+0000 7f5034a00640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f501406b040 0x7f501406d500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5024009ac0 tx=0x7f5024008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.525+0000 7f502d7fa640 1 -- 172.21.15.45:0/836533305 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50280545b0 con 0x7f50300ce600 2023-11-28T20:47:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.747+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f503006a500 con 0x7f501406b040 2023-11-28T20:47:25.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.920+0000 7f502d7fa640 1 -- 172.21.15.45:0/836533305 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f503006a500 con 0x7f501406b040 2023-11-28T20:47:25.923 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-11-28T20:47:25.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f501406b040 msgr2=0x7f501406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f501406b040 0x7f501406d500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5024009ac0 tx=0x7f5024008040 comp rx=0 tx=0).stop 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50300ce600 msgr2=0x7f503014c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50300ce600 0x7f503014c680 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f502800c730 tx=0x7f502800cbf0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 shutdown_connections 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f501406b040 0x7f501406d500 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50300ce600 0x7f503014c680 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f503006dc50 0x7f503014c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 --2- 172.21.15.45:0/836533305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503006aa40 0x7f50300b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.924+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 >> 172.21.15.45:0/836533305 conn(0x7f50300ced60 msgr2=0x7f50300cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.925+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 shutdown_connections 2023-11-28T20:47:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:25.925+0000 7f5036c64640 1 -- 172.21.15.45:0/836533305 wait complete. 2023-11-28T20:47:25.951 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-11-28T20:47:25.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2023-11-28T20:47:26.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.185+0000 7fee7d6e1640 1 Processor -- start 2023-11-28T20:47:26.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.185+0000 7fee7d6e1640 1 -- start start 2023-11-28T20:47:26.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.186+0000 7fee7d6e1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fee780d0040 0x7fee780d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.186+0000 7fee7d6e1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fee780b1d40 0x7fee780d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee7d6e1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee780b90a0 con 0x7fee780b2290 2023-11-28T20:47:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee7d6e1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee780d1920 con 0x7fee780b1d40 2023-11-28T20:47:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee7d6e1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee780d2dc0 con 0x7fee780d0040 2023-11-28T20:47:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee777fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fee780b2290 0x7fee780d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55348/0 (socket says 172.21.15.45:55348) 2023-11-28T20:47:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee777fe640 1 -- 172.21.15.45:0/1254354303 learned_addr learned my addr 172.21.15.45:0/1254354303 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee7cee0640 1 --2- 172.21.15.45:0/1254354303 >> v2:172.21.15.45:3300/0 conn(0x7fee780b1d40 0x7fee780d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee77fff640 1 --2- 172.21.15.45:0/1254354303 >> v2:172.21.15.190:3300/0 conn(0x7fee780d0040 0x7fee780d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee7cee0640 1 -- 172.21.15.45:0/1254354303 >> v2:172.21.15.190:3300/0 conn(0x7fee780d0040 msgr2=0x7fee780d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:26.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee7cee0640 1 --2- 172.21.15.45:0/1254354303 >> v2:172.21.15.190:3300/0 conn(0x7fee780d0040 0x7fee780d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee7cee0640 1 -- 172.21.15.45:0/1254354303 >> v1:172.21.15.45:6789/0 conn(0x7fee780b2290 legacy=0x7fee780d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.187+0000 7fee7cee0640 1 -- 172.21.15.45:0/1254354303 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee780d0960 con 0x7fee780b1d40 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.189+0000 7fee7cee0640 1 --2- 172.21.15.45:0/1254354303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b1d40 0x7fee780d3750 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fee780b53c0 tx=0x7fee7002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd15107f0f8e386 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.189+0000 7fee76ffd640 1 -- 172.21.15.45:0/1254354303 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee70030610 con 0x7fee780b1d40 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.189+0000 7fee76ffd640 1 -- 172.21.15.45:0/1254354303 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee70030770 con 0x7fee780b1d40 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.189+0000 7fee76ffd640 1 -- 172.21.15.45:0/1254354303 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee70030b00 con 0x7fee780b1d40 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.190+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1254354303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b1d40 msgr2=0x7fee780d3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.190+0000 7fee7d6e1640 1 --2- 172.21.15.45:0/1254354303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b1d40 0x7fee780d3750 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fee780b53c0 tx=0x7fee7002b960 comp rx=0 tx=0).stop 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.190+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1254354303 shutdown_connections 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.190+0000 7fee7d6e1640 1 --2- 172.21.15.45:0/1254354303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b1d40 0x7fee780d3750 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.190+0000 7fee7d6e1640 1 --2- 172.21.15.45:0/1254354303 >> v2:172.21.15.190:3300/0 conn(0x7fee780d0040 0x7fee780d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.190+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1254354303 >> 172.21.15.45:0/1254354303 conn(0x7fee78070f40 msgr2=0x7fee78071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.190+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1254354303 shutdown_connections 2023-11-28T20:47:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.190+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1254354303 wait complete. 2023-11-28T20:47:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.192+0000 7fee7d6e1640 1 Processor -- start 2023-11-28T20:47:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.192+0000 7fee7d6e1640 1 -- start start 2023-11-28T20:47:26.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee7d6e1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee780b1d40 0x7fee780ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee7d6e1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b2290 0x7fee7815ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee7d6e1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee780d0040 0x7fee781e4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee7d6e1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee780cd380 con 0x7fee780b2290 2023-11-28T20:47:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee7d6e1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee780cab90 con 0x7fee780d0040 2023-11-28T20:47:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee7d6e1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee780cae70 con 0x7fee780b1d40 2023-11-28T20:47:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b2290 0x7fee7815ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b2290 0x7fee7815ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57228/0 (socket says 172.21.15.45:57228) 2023-11-28T20:47:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee777fe640 1 -- 172.21.15.45:0/1570689741 learned_addr learned my addr 172.21.15.45:0/1570689741 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee7cee0640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee780d0040 0x7fee781e4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.194+0000 7fee77fff640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee780b1d40 0x7fee780ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee777fe640 1 -- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee780b1d40 msgr2=0x7fee780ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee777fe640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee780b1d40 0x7fee780ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee777fe640 1 -- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee780d0040 msgr2=0x7fee781e4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee777fe640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee780d0040 0x7fee781e4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee777fe640 1 -- 172.21.15.45:0/1570689741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee70009400 con 0x7fee780b2290 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee777fe640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b2290 0x7fee7815ab10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00cba0 tx=0x7fee6c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee74ff9640 1 -- 172.21.15.45:0/1570689741 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c00ad40 con 0x7fee780b2290 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee74ff9640 1 -- 172.21.15.45:0/1570689741 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6c012400 con 0x7fee780b2290 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee780cb150 con 0x7fee780b2290 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.195+0000 7fee74ff9640 1 -- 172.21.15.45:0/1570689741 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c00aea0 con 0x7fee780b2290 2023-11-28T20:47:26.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.196+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7815b050 con 0x7fee780b2290 2023-11-28T20:47:26.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.197+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee780cb670 con 0x7fee780b2290 2023-11-28T20:47:26.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.198+0000 7fee74ff9640 1 -- 172.21.15.45:0/1570689741 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7fee6c007070 con 0x7fee780b2290 2023-11-28T20:47:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.199+0000 7fee74ff9640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee6406b040 0x7fee6406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.199+0000 7fee74ff9640 1 -- 172.21.15.45:0/1570689741 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fee6c089b50 con 0x7fee780b2290 2023-11-28T20:47:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.200+0000 7fee77fff640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee6406b040 0x7fee6406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.201+0000 7fee77fff640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee6406b040 0x7fee6406d500 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee68006fd0 tx=0x7fee68008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:26.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.203+0000 7fee74ff9640 1 -- 172.21.15.45:0/1570689741 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee780cb670 con 0x7fee780b2290 2023-11-28T20:47:26.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.423+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fee780cad00 con 0x7fee6406b040 2023-11-28T20:47:26.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.599+0000 7fee74ff9640 1 -- 172.21.15.45:0/1570689741 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fee780cad00 con 0x7fee6406b040 2023-11-28T20:47:26.601 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-11-28T20:47:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee6406b040 msgr2=0x7fee6406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee6406b040 0x7fee6406d500 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee68006fd0 tx=0x7fee68008040 comp rx=0 tx=0).stop 2023-11-28T20:47:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b2290 msgr2=0x7fee7815ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b2290 0x7fee7815ab10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00cba0 tx=0x7fee6c00a430 comp rx=0 tx=0).stop 2023-11-28T20:47:26.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 shutdown_connections 2023-11-28T20:47:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee6406b040 0x7fee6406d500 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee780d0040 0x7fee781e4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee780b2290 0x7fee7815ab10 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 --2- 172.21.15.45:0/1570689741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee780b1d40 0x7fee780ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 >> 172.21.15.45:0/1570689741 conn(0x7fee78070f40 msgr2=0x7fee780b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 shutdown_connections 2023-11-28T20:47:26.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.603+0000 7fee7d6e1640 1 -- 172.21.15.45:0/1570689741 wait complete. 2023-11-28T20:47:26.629 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-11-28T20:47:26.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2023-11-28T20:47:26.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.855+0000 7fd6af6c9640 1 Processor -- start 2023-11-28T20:47:26.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.855+0000 7fd6af6c9640 1 -- start start 2023-11-28T20:47:26.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.856+0000 7fd6af6c9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd6a81653a0 0x7fd6a80ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.856+0000 7fd6af6c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80712c0 0x7fd6a8162290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.856+0000 7fd6af6c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a815f7e0 con 0x7fd6a80c1d30 2023-11-28T20:47:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.856+0000 7fd6af6c9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a81658a0 con 0x7fd6a81653a0 2023-11-28T20:47:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.856+0000 7fd6af6c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8165b80 con 0x7fd6a80712c0 2023-11-28T20:47:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.856+0000 7fd6ad465640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd6a80c1d30 0x7fd6a80c2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55350/0 (socket says 172.21.15.45:55350) 2023-11-28T20:47:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.856+0000 7fd6ad465640 1 -- 172.21.15.45:0/710305632 learned_addr learned my addr 172.21.15.45:0/710305632 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.857+0000 7fd6acc64640 1 --2- 172.21.15.45:0/710305632 >> v2:172.21.15.45:3300/0 conn(0x7fd6a81653a0 0x7fd6a80ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.857+0000 7fd6adc66640 1 --2- 172.21.15.45:0/710305632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80712c0 0x7fd6a8162290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.857+0000 7fd69ffff640 1 -- 172.21.15.45:0/710305632 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126026145 0 0) 0x7fd6a815f7e0 con 0x7fd6a80c1d30 2023-11-28T20:47:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.857+0000 7fd69ffff640 1 -- 172.21.15.45:0/710305632 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd694003520 con 0x7fd6a80c1d30 2023-11-28T20:47:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.857+0000 7fd6acc64640 1 -- 172.21.15.45:0/710305632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80712c0 msgr2=0x7fd6a8162290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.857+0000 7fd6acc64640 1 --2- 172.21.15.45:0/710305632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80712c0 0x7fd6a8162290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.857+0000 7fd6acc64640 1 -- 172.21.15.45:0/710305632 >> v1:172.21.15.45:6789/0 conn(0x7fd6a80c1d30 legacy=0x7fd6a80c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.857+0000 7fd69ffff640 1 -- 172.21.15.45:0/710305632 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3964552316 0 0) 0x7fd694003520 con 0x7fd6a80c1d30 2023-11-28T20:47:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.857+0000 7fd6acc64640 1 -- 172.21.15.45:0/710305632 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a8165e60 con 0x7fd6a81653a0 2023-11-28T20:47:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd6acc64640 1 --2- 172.21.15.45:0/710305632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 0x7fd6a80ce480 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd6980097d0 tx=0x7fd69802b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed454ecb0d958fa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd69ffff640 1 -- 172.21.15.45:0/710305632 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4001670 con 0x7fd6a81653a0 2023-11-28T20:47:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd69ffff640 1 -- 172.21.15.45:0/710305632 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69802e050 con 0x7fd6a81653a0 2023-11-28T20:47:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd69ffff640 1 -- 172.21.15.45:0/710305632 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698030730 con 0x7fd6a81653a0 2023-11-28T20:47:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd6af6c9640 1 -- 172.21.15.45:0/710305632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 msgr2=0x7fd6a80ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd6af6c9640 1 --2- 172.21.15.45:0/710305632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 0x7fd6a80ce480 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd6980097d0 tx=0x7fd69802b770 comp rx=0 tx=0).stop 2023-11-28T20:47:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd6af6c9640 1 -- 172.21.15.45:0/710305632 shutdown_connections 2023-11-28T20:47:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd6af6c9640 1 --2- 172.21.15.45:0/710305632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80712c0 0x7fd6a8162290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd6af6c9640 1 --2- 172.21.15.45:0/710305632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 0x7fd6a80ce480 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.859+0000 7fd6af6c9640 1 -- 172.21.15.45:0/710305632 >> 172.21.15.45:0/710305632 conn(0x7fd6a80ced60 msgr2=0x7fd6a80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.860+0000 7fd6af6c9640 1 -- 172.21.15.45:0/710305632 shutdown_connections 2023-11-28T20:47:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.860+0000 7fd6af6c9640 1 -- 172.21.15.45:0/710305632 wait complete. 2023-11-28T20:47:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.861+0000 7fd6af6c9640 1 Processor -- start 2023-11-28T20:47:26.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.862+0000 7fd6af6c9640 1 -- start start 2023-11-28T20:47:26.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.862+0000 7fd6af6c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a80712c0 0x7fd6a8154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.862+0000 7fd6af6c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80c1d30 0x7fd6a80c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6af6c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 0x7fd6a80ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6af6c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80af9a0 con 0x7fd6a81653a0 2023-11-28T20:47:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6af6c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80c3e70 con 0x7fd6a80712c0 2023-11-28T20:47:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6af6c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a815e850 con 0x7fd6a80c1d30 2023-11-28T20:47:26.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6adc66640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 0x7fd6a80ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6adc66640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 0x7fd6a80ace40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57250/0 (socket says 172.21.15.45:57250) 2023-11-28T20:47:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6adc66640 1 -- 172.21.15.45:0/838967682 learned_addr learned my addr 172.21.15.45:0/838967682 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6acc64640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80c1d30 0x7fd6a80c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6ad465640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a80712c0 0x7fd6a8154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6adc66640 1 -- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80c1d30 msgr2=0x7fd6a80c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6adc66640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80c1d30 0x7fd6a80c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6adc66640 1 -- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a80712c0 msgr2=0x7fd6a8154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6adc66640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a80712c0 0x7fd6a8154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.863+0000 7fd6adc66640 1 -- 172.21.15.45:0/838967682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd698009400 con 0x7fd6a81653a0 2023-11-28T20:47:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.864+0000 7fd6adc66640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 0x7fd6a80ace40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0007d90 tx=0x7fd6a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.864+0000 7fd69dffb640 1 -- 172.21.15.45:0/838967682 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0014420 con 0x7fd6a81653a0 2023-11-28T20:47:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.864+0000 7fd69dffb640 1 -- 172.21.15.45:0/838967682 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a00145a0 con 0x7fd6a81653a0 2023-11-28T20:47:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.864+0000 7fd69dffb640 1 -- 172.21.15.45:0/838967682 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0014950 con 0x7fd6a81653a0 2023-11-28T20:47:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.864+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a80af3e0 con 0x7fd6a81653a0 2023-11-28T20:47:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.864+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a80ac920 con 0x7fd6a81653a0 2023-11-28T20:47:26.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.865+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a80cde90 con 0x7fd6a81653a0 2023-11-28T20:47:26.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.866+0000 7fd69dffb640 1 -- 172.21.15.45:0/838967682 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7fd6a0014ab0 con 0x7fd6a81653a0 2023-11-28T20:47:26.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.867+0000 7fd69dffb640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd68c06b040 0x7fd68c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:26.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.868+0000 7fd69dffb640 1 -- 172.21.15.45:0/838967682 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd6a008a570 con 0x7fd6a81653a0 2023-11-28T20:47:26.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.868+0000 7fd6ad465640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd68c06b040 0x7fd68c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:26.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.869+0000 7fd6ad465640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd68c06b040 0x7fd68c06d500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80b92d0 tx=0x7fd6a4009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:26.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:26.872+0000 7fd69dffb640 1 -- 172.21.15.45:0/838967682 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6a80cde90 con 0x7fd6a81653a0 2023-11-28T20:47:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.090+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6a8161d50 con 0x7fd68c06b040 2023-11-28T20:47:27.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.274+0000 7fd69dffb640 1 -- 172.21.15.45:0/838967682 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd6a4001490 con 0x7fd68c06b040 2023-11-28T20:47:27.277 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd68c06b040 msgr2=0x7fd68c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd68c06b040 0x7fd68c06d500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80b92d0 tx=0x7fd6a4009040 comp rx=0 tx=0).stop 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 msgr2=0x7fd6a80ace40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 0x7fd6a80ace40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0007d90 tx=0x7fd6a000e4b0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 shutdown_connections 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd68c06b040 0x7fd68c06d500 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6a81653a0 0x7fd6a80ace40 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6a80c1d30 0x7fd6a80c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 --2- 172.21.15.45:0/838967682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a80712c0 0x7fd6a8154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.278+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 >> 172.21.15.45:0/838967682 conn(0x7fd6a80ced60 msgr2=0x7fd6a80cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:27.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.279+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 shutdown_connections 2023-11-28T20:47:27.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.279+0000 7fd6af6c9640 1 -- 172.21.15.45:0/838967682 wait complete. 2023-11-28T20:47:27.302 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-11-28T20:47:27.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2023-11-28T20:47:27.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.538+0000 7f89032d1640 1 Processor -- start 2023-11-28T20:47:27.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.538+0000 7f89032d1640 1 -- start start 2023-11-28T20:47:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.540+0000 7f89032d1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f88fc159720 0x7f88fc0b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.540+0000 7f89032d1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f88fc06dfa0 0x7f88fc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.540+0000 7f89032d1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc0cc120 con 0x7f88fc071b90 2023-11-28T20:47:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.540+0000 7f89032d1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88fc06e430 con 0x7f88fc159720 2023-11-28T20:47:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.540+0000 7f89032d1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88fc0b2420 con 0x7f88fc06dfa0 2023-11-28T20:47:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.540+0000 7f89022cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f88fc159720 0x7f88fc0b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.540+0000 7f89022cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f88fc159720 0x7f88fc0b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57266/0 (socket says 172.21.15.45:57266) 2023-11-28T20:47:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.540+0000 7f89022cf640 1 -- 172.21.15.45:0/3025087020 learned_addr learned my addr 172.21.15.45:0/3025087020 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.540+0000 7f8901ace640 1 --2- 172.21.15.45:0/3025087020 >> v2:172.21.15.190:3300/0 conn(0x7f88fc06dfa0 0x7f88fc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.541+0000 7f89022cf640 1 -- 172.21.15.45:0/3025087020 >> v2:172.21.15.190:3300/0 conn(0x7f88fc06dfa0 msgr2=0x7f88fc0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.541+0000 7f89022cf640 1 --2- 172.21.15.45:0/3025087020 >> v2:172.21.15.190:3300/0 conn(0x7f88fc06dfa0 0x7f88fc0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.541+0000 7f89022cf640 1 -- 172.21.15.45:0/3025087020 >> v1:172.21.15.190:6789/0 conn(0x7f88fc071b90 legacy=0x7f88fc071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.541+0000 7f89022cf640 1 -- 172.21.15.45:0/3025087020 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc0cb880 con 0x7f88fc159720 2023-11-28T20:47:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.542+0000 7f89022cf640 1 --2- 172.21.15.45:0/3025087020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 0x7f88fc0b1d20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f88fc06dcd0 tx=0x7f88f802cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527d2dece6b0e8b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.542+0000 7f89012cd640 1 -- 172.21.15.45:0/3025087020 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f802dba0 con 0x7f88fc159720 2023-11-28T20:47:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.542+0000 7f89012cd640 1 -- 172.21.15.45:0/3025087020 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f802dd00 con 0x7f88fc159720 2023-11-28T20:47:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.542+0000 7f89012cd640 1 -- 172.21.15.45:0/3025087020 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8035660 con 0x7f88fc159720 2023-11-28T20:47:27.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.543+0000 7f89032d1640 1 -- 172.21.15.45:0/3025087020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 msgr2=0x7f88fc0b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:27.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.543+0000 7f89032d1640 1 --2- 172.21.15.45:0/3025087020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 0x7f88fc0b1d20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f88fc06dcd0 tx=0x7f88f802cdf0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.543+0000 7f89032d1640 1 -- 172.21.15.45:0/3025087020 shutdown_connections 2023-11-28T20:47:27.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.543+0000 7f89032d1640 1 --2- 172.21.15.45:0/3025087020 >> v2:172.21.15.190:3300/0 conn(0x7f88fc06dfa0 0x7f88fc0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.543+0000 7f89032d1640 1 --2- 172.21.15.45:0/3025087020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 0x7f88fc0b1d20 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.543+0000 7f89032d1640 1 -- 172.21.15.45:0/3025087020 >> 172.21.15.45:0/3025087020 conn(0x7f88fc0b1050 msgr2=0x7f88fc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:27.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.543+0000 7f89032d1640 1 -- 172.21.15.45:0/3025087020 shutdown_connections 2023-11-28T20:47:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.543+0000 7f89032d1640 1 -- 172.21.15.45:0/3025087020 wait complete. 2023-11-28T20:47:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.546+0000 7f89032d1640 1 Processor -- start 2023-11-28T20:47:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.546+0000 7f89032d1640 1 -- start start 2023-11-28T20:47:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.547+0000 7f89032d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88fc06dfa0 0x7f88fc15b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.547+0000 7f89032d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88fc071b90 0x7f88fc14db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.547+0000 7f89032d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 0x7f88fc06bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.547+0000 7f89032d1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc15c7c0 con 0x7f88fc159720 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.547+0000 7f89032d1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc15b960 con 0x7f88fc06dfa0 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.547+0000 7f89032d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc15cc30 con 0x7f88fc071b90 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8902ad0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 0x7f88fc06bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8902ad0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 0x7f88fc06bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57274/0 (socket says 172.21.15.45:57274) 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8902ad0640 1 -- 172.21.15.45:0/270315950 learned_addr learned my addr 172.21.15.45:0/270315950 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f89022cf640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88fc06dfa0 0x7f88fc15b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8901ace640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88fc071b90 0x7f88fc14db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8902ad0640 1 -- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88fc071b90 msgr2=0x7f88fc14db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8902ad0640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88fc071b90 0x7f88fc14db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8902ad0640 1 -- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88fc06dfa0 msgr2=0x7f88fc15b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8902ad0640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88fc06dfa0 0x7f88fc15b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8902ad0640 1 -- 172.21.15.45:0/270315950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f800a670 con 0x7f88fc159720 2023-11-28T20:47:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.548+0000 7f8901ace640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88fc071b90 0x7f88fc14db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:47:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.549+0000 7f8902ad0640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 0x7f88fc06bf60 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f88f400b970 tx=0x7f88f400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.549+0000 7f88f2ffd640 1 -- 172.21.15.45:0/270315950 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f400cae0 con 0x7f88fc159720 2023-11-28T20:47:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.549+0000 7f88f2ffd640 1 -- 172.21.15.45:0/270315950 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f400cc40 con 0x7f88fc159720 2023-11-28T20:47:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.549+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc15bb40 con 0x7f88fc159720 2023-11-28T20:47:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.549+0000 7f88f2ffd640 1 -- 172.21.15.45:0/270315950 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4014680 con 0x7f88fc159720 2023-11-28T20:47:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.549+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc15c100 con 0x7f88fc159720 2023-11-28T20:47:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.551+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88fc0cc710 con 0x7f88fc159720 2023-11-28T20:47:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.552+0000 7f88f2ffd640 1 -- 172.21.15.45:0/270315950 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f88f401d3b0 con 0x7f88fc159720 2023-11-28T20:47:27.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.553+0000 7f88f2ffd640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f88e006b070 0x7f88e006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:27.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.553+0000 7f88f2ffd640 1 -- 172.21.15.45:0/270315950 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f88f4089880 con 0x7f88fc159720 2023-11-28T20:47:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.554+0000 7f89022cf640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f88e006b070 0x7f88e006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:27.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.555+0000 7f89022cf640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f88e006b070 0x7f88e006d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f88f8037000 tx=0x7f88f8008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:27.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.558+0000 7f88f2ffd640 1 -- 172.21.15.45:0/270315950 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88f4054590 con 0x7f88fc159720 2023-11-28T20:47:27.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.781+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f88fc070720 con 0x7f88e006b070 2023-11-28T20:47:27.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.949+0000 7f88f2ffd640 1 -- 172.21.15.45:0/270315950 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f88f80450c0 con 0x7f88e006b070 2023-11-28T20:47:27.952 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-11-28T20:47:27.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f88e006b070 msgr2=0x7f88e006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:27.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f88e006b070 0x7f88e006d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f88f8037000 tx=0x7f88f8008040 comp rx=0 tx=0).stop 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 msgr2=0x7f88fc06bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 0x7f88fc06bf60 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f88f400b970 tx=0x7f88f400be30 comp rx=0 tx=0).stop 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 shutdown_connections 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f88e006b070 0x7f88e006d530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88fc159720 0x7f88fc06bf60 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88fc071b90 0x7f88fc14db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 --2- 172.21.15.45:0/270315950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88fc06dfa0 0x7f88fc15b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 >> 172.21.15.45:0/270315950 conn(0x7f88fc0b1050 msgr2=0x7f88fc158960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.953+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 shutdown_connections 2023-11-28T20:47:27.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:27.954+0000 7f89032d1640 1 -- 172.21.15.45:0/270315950 wait complete. 2023-11-28T20:47:27.977 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-11-28T20:47:27.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:27.977 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-11-28T20:47:28.042 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:28.042 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:28.042 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:28.042 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:28.043 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:28.043 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:28.043 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:28.043 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:28.043 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:28.043 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:28.043 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:28.043 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:28.044 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_36_00_UTC', 'scheduled-2023-11-28-20_37_00_UTC', 'scheduled-2023-11-28-20_38_00_UTC', 'scheduled-2023-11-28-20_39_00_UTC', 'scheduled-2023-11-28-20_40_00_UTC', 'scheduled-2023-11-28-20_41_00_UTC', 'scheduled-2023-11-28-20_42_00_UTC', 'scheduled-2023-11-28-20_43_00_UTC', 'scheduled-2023-11-28-20_44_00_UTC', 'scheduled-2023-11-28-20_45_00_UTC', 'scheduled-2023-11-28-20_46_01_UTC', 'scheduled-2023-11-28-20_47_00_UTC'] 2023-11-28T20:47:28.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2023-11-28T20:47:28.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.277+0000 7f356b59e640 1 Processor -- start 2023-11-28T20:47:28.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.278+0000 7f356b59e640 1 -- start start 2023-11-28T20:47:28.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.279+0000 7f356b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f356c15b080 0x7f356c0ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.279+0000 7f356b59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0ac270 con 0x7f356c0abd70 2023-11-28T20:47:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.280+0000 7f356b59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c15b830 con 0x7f356c071fd0 2023-11-28T20:47:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.280+0000 7f356b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c15bb10 con 0x7f356c15b080 2023-11-28T20:47:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.280+0000 7f3563fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f356c0abd70 0x7f356c0718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55352/0 (socket says 172.21.15.45:55352) 2023-11-28T20:47:28.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.280+0000 7f3563fff640 1 -- 172.21.15.45:0/3261667371 learned_addr learned my addr 172.21.15.45:0/3261667371 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.280+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863259403 0 0) 0x7f356c0ac270 con 0x7f356c0abd70 2023-11-28T20:47:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.280+0000 7f356a59c640 1 --2- 172.21.15.45:0/3261667371 >> v2:172.21.15.190:3300/0 conn(0x7f356c15b080 0x7f356c0ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.280+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3558003520 con 0x7f356c0abd70 2023-11-28T20:47:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153075098 0 0) 0x7f3558003520 con 0x7f356c0abd70 2023-11-28T20:47:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 >> v2:172.21.15.190:3300/0 conn(0x7f356c15b080 msgr2=0x7f356c0ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 --2- 172.21.15.45:0/3261667371 >> v2:172.21.15.190:3300/0 conn(0x7f356c15b080 0x7f356c0ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 >> v1:172.21.15.190:6789/0 conn(0x7f356c071fd0 legacy=0x7f356c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:28.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c15bdf0 con 0x7f356c0abd70 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3554001670 con 0x7f356c0abd70 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 >> v1:172.21.15.45:6789/0 conn(0x7f356c0abd70 legacy=0x7f356c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3558003970 0x7f3558003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35580047c0 0x7f3558004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3558005180 0x7f355800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c15bb10 con 0x7f3558003970 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355800c0d0 con 0x7f35580047c0 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355800c290 con 0x7f3558005180 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f356a59c640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3558003970 0x7f3558003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.281+0000 7f3563fff640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35580047c0 0x7f3558004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.282+0000 7f356a59c640 1 -- 172.21.15.45:0/3261667371 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3558005180 msgr2=0x7f355800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:28.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.282+0000 7f356a59c640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3558005180 0x7f355800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.282+0000 7f356a59c640 1 -- 172.21.15.45:0/3261667371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35580047c0 msgr2=0x7f3558004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:28.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.282+0000 7f356a59c640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35580047c0 0x7f3558004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.282+0000 7f356ad9d640 1 -- 172.21.15.45:0/3261667371 reap_dead start 2023-11-28T20:47:28.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.282+0000 7f356a59c640 1 -- 172.21.15.45:0/3261667371 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3558003520 con 0x7f3558003970 2023-11-28T20:47:28.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.282+0000 7f3563fff640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35580047c0 0x7f3558004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:28.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.283+0000 7f356a59c640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3558003970 0x7f3558003d90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f355c00ba30 tx=0x7f35580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27ebb8cf29e220a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:28.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.283+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c032680 con 0x7f3558003970 2023-11-28T20:47:28.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.283+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c0327e0 con 0x7f3558003970 2023-11-28T20:47:28.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.283+0000 7f3569d9b640 1 -- 172.21.15.45:0/3261667371 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c032b70 con 0x7f3558003970 2023-11-28T20:47:28.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.284+0000 7f356b59e640 1 -- 172.21.15.45:0/3261667371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3558003970 msgr2=0x7f3558003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:28.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.284+0000 7f356b59e640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3558003970 0x7f3558003d90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f355c00ba30 tx=0x7f35580057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.284+0000 7f356b59e640 1 -- 172.21.15.45:0/3261667371 shutdown_connections 2023-11-28T20:47:28.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.284+0000 7f356b59e640 1 --2- 172.21.15.45:0/3261667371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3558003970 0x7f3558003d90 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.284+0000 7f356b59e640 1 -- 172.21.15.45:0/3261667371 >> 172.21.15.45:0/3261667371 conn(0x7f356c0cecd0 msgr2=0x7f356c0cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:28.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.284+0000 7f356b59e640 1 -- 172.21.15.45:0/3261667371 shutdown_connections 2023-11-28T20:47:28.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.284+0000 7f356b59e640 1 -- 172.21.15.45:0/3261667371 wait complete. 2023-11-28T20:47:28.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.286+0000 7f356b59e640 1 Processor -- start 2023-11-28T20:47:28.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.286+0000 7f356b59e640 1 -- start start 2023-11-28T20:47:28.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.287+0000 7f356b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3558003970 0x7f356c157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.287+0000 7f356b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f356c153720 0x7f356c145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.287+0000 7f356b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f356c151b50 0x7f356c09bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.287+0000 7f356b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c09b680 con 0x7f356c153720 2023-11-28T20:47:28.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.287+0000 7f356b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c153f90 con 0x7f356c151b50 2023-11-28T20:47:28.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.287+0000 7f356b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c145c90 con 0x7f3558003970 2023-11-28T20:47:28.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.287+0000 7f3563fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f356c153720 0x7f356c145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.287+0000 7f3563fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f356c153720 0x7f356c145750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57300/0 (socket says 172.21.15.45:57300) 2023-11-28T20:47:28.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.287+0000 7f3563fff640 1 -- 172.21.15.45:0/1005547884 learned_addr learned my addr 172.21.15.45:0/1005547884 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:28.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.288+0000 7f356a59c640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3558003970 0x7f356c157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.288+0000 7f3563fff640 1 -- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3558003970 msgr2=0x7f356c157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:28.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.288+0000 7f3563fff640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3558003970 0x7f356c157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.288+0000 7f3563fff640 1 -- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f356c151b50 msgr2=0x7f356c09bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:28.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.288+0000 7f3563fff640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f356c151b50 0x7f356c09bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.288+0000 7f3563fff640 1 -- 172.21.15.45:0/1005547884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c153cb0 con 0x7f356c153720 2023-11-28T20:47:28.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.288+0000 7f3563fff640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f356c153720 0x7f356c145750 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f355400d7a0 tx=0x7f355400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:28.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.288+0000 7f3562ffd640 1 -- 172.21.15.45:0/1005547884 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35540074e0 con 0x7f356c153720 2023-11-28T20:47:28.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.289+0000 7f3562ffd640 1 -- 172.21.15.45:0/1005547884 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3554007640 con 0x7f356c153720 2023-11-28T20:47:28.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.289+0000 7f3562ffd640 1 -- 172.21.15.45:0/1005547884 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554004a20 con 0x7f356c153720 2023-11-28T20:47:28.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.289+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c09b030 con 0x7f356c153720 2023-11-28T20:47:28.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.289+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c151fd0 con 0x7f356c153720 2023-11-28T20:47:28.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.290+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c09b7f0 con 0x7f356c153720 2023-11-28T20:47:28.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.291+0000 7f3562ffd640 1 -- 172.21.15.45:0/1005547884 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f3554004b80 con 0x7f356c153720 2023-11-28T20:47:28.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.292+0000 7f3562ffd640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f354c06b070 0x7f354c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.292+0000 7f3562ffd640 1 -- 172.21.15.45:0/1005547884 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f35540896a0 con 0x7f356c153720 2023-11-28T20:47:28.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.293+0000 7f356a59c640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f354c06b070 0x7f354c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.294+0000 7f356a59c640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f354c06b070 0x7f354c06d530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f355c00ba00 tx=0x7f355c03d040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:28.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.296+0000 7f3562ffd640 1 -- 172.21.15.45:0/1005547884 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35540540c0 con 0x7f356c153720 2023-11-28T20:47:28.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.520+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f356c15c310 con 0x7f354c06b070 2023-11-28T20:47:28.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.539+0000 7f3562ffd640 1 -- 172.21.15.45:0/1005547884 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f355c0329d0 con 0x7f354c06b070 2023-11-28T20:47:28.541 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:28.542 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:56", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:28.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f354c06b070 msgr2=0x7f354c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:28.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f354c06b070 0x7f354c06d530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f355c00ba00 tx=0x7f355c03d040 comp rx=0 tx=0).stop 2023-11-28T20:47:28.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f356c153720 msgr2=0x7f356c145750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f356c153720 0x7f356c145750 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f355400d7a0 tx=0x7f355400dc60 comp rx=0 tx=0).stop 2023-11-28T20:47:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 shutdown_connections 2023-11-28T20:47:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f354c06b070 0x7f354c06d530 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f356c151b50 0x7f356c09bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f356c153720 0x7f356c145750 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 --2- 172.21.15.45:0/1005547884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3558003970 0x7f356c157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 >> 172.21.15.45:0/1005547884 conn(0x7f356c0cecd0 msgr2=0x7f356c0af740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 shutdown_connections 2023-11-28T20:47:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.542+0000 7f356b59e640 1 -- 172.21.15.45:0/1005547884 wait complete. 2023-11-28T20:47:28.564 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:56", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:28.564 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:35:56', 'first': '2023-11-28T20:36:00', 'last': '2023-11-28T20:47:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:28.564 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-11-28T20:47:28.565 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-11-28T20:47:28.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:28.565 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-11-28T20:47:28.626 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:28.626 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:28.626 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:28.626 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:28.627 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:28.627 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:28.627 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:28.627 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:28.627 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:28.627 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_01_UTC 2023-11-28T20:47:28.627 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:28.627 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:28.627 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_36_00_UTC', 'scheduled-2023-11-28-20_37_00_UTC', 'scheduled-2023-11-28-20_38_00_UTC', 'scheduled-2023-11-28-20_39_00_UTC', 'scheduled-2023-11-28-20_40_00_UTC', 'scheduled-2023-11-28-20_41_00_UTC', 'scheduled-2023-11-28-20_42_00_UTC', 'scheduled-2023-11-28-20_43_00_UTC', 'scheduled-2023-11-28-20_44_01_UTC', 'scheduled-2023-11-28-20_45_01_UTC', 'scheduled-2023-11-28-20_46_01_UTC', 'scheduled-2023-11-28-20_47_00_UTC'] 2023-11-28T20:47:28.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2023-11-28T20:47:28.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.849+0000 7f9bd498f640 1 Processor -- start 2023-11-28T20:47:28.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.849+0000 7f9bd498f640 1 -- start start 2023-11-28T20:47:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bd498f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9bd006dbc0 0x7f9bd00ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bd498f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9bd00ac7f0 0x7f9bd006dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bd498f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9bd00b1d00 0x7f9bd00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bd498f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd00acc50 con 0x7f9bd00b1d00 2023-11-28T20:47:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bd498f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd00addf0 con 0x7f9bd006dbc0 2023-11-28T20:47:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bd498f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd0070c20 con 0x7f9bd00ac7f0 2023-11-28T20:47:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bcffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9bd00b1d00 0x7f9bd00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bcffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9bd00b1d00 0x7f9bd00710b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57312/0 (socket says 172.21.15.45:57312) 2023-11-28T20:47:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bcffff640 1 -- 172.21.15.45:0/183273697 learned_addr learned my addr 172.21.15.45:0/183273697 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.851+0000 7f9bcf7fe640 1 --2- 172.21.15.45:0/183273697 >> v2:172.21.15.190:3300/0 conn(0x7f9bd006dbc0 0x7f9bd00ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.852+0000 7f9bceffd640 1 --2- 172.21.15.45:0/183273697 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9bd00ac7f0 0x7f9bd006dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.852+0000 7f9bcffff640 1 -- 172.21.15.45:0/183273697 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9bd00ac7f0 msgr2=0x7f9bd006dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.852+0000 7f9bcffff640 1 --2- 172.21.15.45:0/183273697 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9bd00ac7f0 0x7f9bd006dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.852+0000 7f9bcffff640 1 -- 172.21.15.45:0/183273697 >> v2:172.21.15.190:3300/0 conn(0x7f9bd006dbc0 msgr2=0x7f9bd00ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.852+0000 7f9bcffff640 1 --2- 172.21.15.45:0/183273697 >> v2:172.21.15.190:3300/0 conn(0x7f9bd006dbc0 0x7f9bd00ad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.852+0000 7f9bcffff640 1 -- 172.21.15.45:0/183273697 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd006e4e0 con 0x7f9bd00b1d00 2023-11-28T20:47:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.853+0000 7f9bcffff640 1 --2- 172.21.15.45:0/183273697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd00b1d00 0x7f9bd00710b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0073220 tx=0x7f9bc802e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf83ee671f5f8c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.853+0000 7f9bce7fc640 1 -- 172.21.15.45:0/183273697 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc802ea00 con 0x7f9bd00b1d00 2023-11-28T20:47:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bce7fc640 1 -- 172.21.15.45:0/183273697 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc802eb60 con 0x7f9bd00b1d00 2023-11-28T20:47:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bce7fc640 1 -- 172.21.15.45:0/183273697 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8009600 con 0x7f9bd00b1d00 2023-11-28T20:47:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bd498f640 1 -- 172.21.15.45:0/183273697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd00b1d00 msgr2=0x7f9bd00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bd498f640 1 --2- 172.21.15.45:0/183273697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd00b1d00 0x7f9bd00710b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0073220 tx=0x7f9bc802e060 comp rx=0 tx=0).stop 2023-11-28T20:47:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bd498f640 1 -- 172.21.15.45:0/183273697 shutdown_connections 2023-11-28T20:47:28.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bd498f640 1 --2- 172.21.15.45:0/183273697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd00b1d00 0x7f9bd00710b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bd498f640 1 --2- 172.21.15.45:0/183273697 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9bd00ac7f0 0x7f9bd006dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bd498f640 1 --2- 172.21.15.45:0/183273697 >> v2:172.21.15.190:3300/0 conn(0x7f9bd006dbc0 0x7f9bd00ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bd498f640 1 -- 172.21.15.45:0/183273697 >> 172.21.15.45:0/183273697 conn(0x7f9bd00cecd0 msgr2=0x7f9bd00cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:28.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bd498f640 1 -- 172.21.15.45:0/183273697 shutdown_connections 2023-11-28T20:47:28.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.854+0000 7f9bd498f640 1 -- 172.21.15.45:0/183273697 wait complete. 2023-11-28T20:47:28.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.856+0000 7f9bd498f640 1 Processor -- start 2023-11-28T20:47:28.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.856+0000 7f9bd498f640 1 -- start start 2023-11-28T20:47:28.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.857+0000 7f9bd498f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd006dbc0 0x7f9bd0159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.857+0000 7f9bd498f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd00ac7f0 0x7f9bd006a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.857+0000 7f9bd498f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9bd00b1d00 0x7f9bd006aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.857+0000 7f9bd498f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd0141420 con 0x7f9bd006dbc0 2023-11-28T20:47:28.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.857+0000 7f9bd498f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd015c180 con 0x7f9bd00ac7f0 2023-11-28T20:47:28.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.857+0000 7f9bd498f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd0159ab0 con 0x7f9bd00b1d00 2023-11-28T20:47:28.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.857+0000 7f9bcf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd006dbc0 0x7f9bd0159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bcf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd006dbc0 0x7f9bd0159570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57318/0 (socket says 172.21.15.45:57318) 2023-11-28T20:47:28.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bcf7fe640 1 -- 172.21.15.45:0/249010550 learned_addr learned my addr 172.21.15.45:0/249010550 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bcf7fe640 1 -- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9bd00b1d00 msgr2=0x7f9bd006aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bcf7fe640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9bd00b1d00 0x7f9bd006aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bcf7fe640 1 -- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd00ac7f0 msgr2=0x7f9bd006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bceffd640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd00ac7f0 0x7f9bd006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bcf7fe640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd00ac7f0 0x7f9bd006a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bcf7fe640 1 -- 172.21.15.45:0/249010550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc800bd40 con 0x7f9bd006dbc0 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bceffd640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd00ac7f0 0x7f9bd006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.858+0000 7f9bcf7fe640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd006dbc0 0x7f9bd0159570 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000e870 tx=0x7f9bc000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.859+0000 7f9bb7fff640 1 -- 172.21.15.45:0/249010550 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000ee60 con 0x7f9bd006dbc0 2023-11-28T20:47:28.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.859+0000 7f9bb7fff640 1 -- 172.21.15.45:0/249010550 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc000c940 con 0x7f9bd006dbc0 2023-11-28T20:47:28.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.859+0000 7f9bb7fff640 1 -- 172.21.15.45:0/249010550 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000ccf0 con 0x7f9bd006dbc0 2023-11-28T20:47:28.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.859+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd015c390 con 0x7f9bd006dbc0 2023-11-28T20:47:28.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.859+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd015c950 con 0x7f9bd006dbc0 2023-11-28T20:47:28.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.860+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd009d0f0 con 0x7f9bd006dbc0 2023-11-28T20:47:28.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.861+0000 7f9bb7fff640 1 -- 172.21.15.45:0/249010550 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f9bc0013470 con 0x7f9bd006dbc0 2023-11-28T20:47:28.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.862+0000 7f9bb7fff640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9bbc06b040 0x7f9bbc06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:28.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.862+0000 7f9bb7fff640 1 -- 172.21.15.45:0/249010550 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f9bc0089d20 con 0x7f9bd006dbc0 2023-11-28T20:47:28.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.863+0000 7f9bceffd640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9bbc06b040 0x7f9bbc06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:28.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.864+0000 7f9bceffd640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9bbc06b040 0x7f9bbc06d500 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9bd01402c0 tx=0x7f9bc4008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:28.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:28.866+0000 7f9bb7fff640 1 -- 172.21.15.45:0/249010550 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9bc0054850 con 0x7f9bd006dbc0 2023-11-28T20:47:29.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.080+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bd00708b0 con 0x7f9bbc06b040 2023-11-28T20:47:29.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.100+0000 7f9bb7fff640 1 -- 172.21.15.45:0/249010550 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f9bc40095f0 con 0x7f9bbc06b040 2023-11-28T20:47:29.102 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:29.102 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:56", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:29.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9bbc06b040 msgr2=0x7f9bbc06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9bbc06b040 0x7f9bbc06d500 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9bd01402c0 tx=0x7f9bc4008040 comp rx=0 tx=0).stop 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd006dbc0 msgr2=0x7f9bd0159570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd006dbc0 0x7f9bd0159570 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000e870 tx=0x7f9bc000ed30 comp rx=0 tx=0).stop 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 shutdown_connections 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9bbc06b040 0x7f9bbc06d500 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9bd00b1d00 0x7f9bd006aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd00ac7f0 0x7f9bd006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 --2- 172.21.15.45:0/249010550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bd006dbc0 0x7f9bd0159570 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.102+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 >> 172.21.15.45:0/249010550 conn(0x7f9bd00cecd0 msgr2=0x7f9bd013ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.103+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 shutdown_connections 2023-11-28T20:47:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.103+0000 7f9bd498f640 1 -- 172.21.15.45:0/249010550 wait complete. 2023-11-28T20:47:29.124 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:56", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:29.124 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:35:56', 'first': '2023-11-28T20:36:00', 'last': '2023-11-28T20:47:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:29.124 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-11-28T20:47:29.124 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-11-28T20:47:29.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:29.124 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-11-28T20:47:29.183 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_01_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_01_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_01_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:29.184 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_01_UTC 2023-11-28T20:47:29.185 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_36_00_UTC', 'scheduled-2023-11-28-20_37_01_UTC', 'scheduled-2023-11-28-20_38_01_UTC', 'scheduled-2023-11-28-20_39_01_UTC', 'scheduled-2023-11-28-20_40_01_UTC', 'scheduled-2023-11-28-20_41_00_UTC', 'scheduled-2023-11-28-20_42_00_UTC', 'scheduled-2023-11-28-20_43_00_UTC', 'scheduled-2023-11-28-20_44_01_UTC', 'scheduled-2023-11-28-20_45_00_UTC', 'scheduled-2023-11-28-20_46_00_UTC', 'scheduled-2023-11-28-20_47_01_UTC'] 2023-11-28T20:47:29.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2023-11-28T20:47:29.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.413+0000 7f01af4cf640 1 Processor -- start 2023-11-28T20:47:29.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.413+0000 7f01af4cf640 1 -- start start 2023-11-28T20:47:29.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.415+0000 7f01af4cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f01a815e840 0x7f01a806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.415+0000 7f01af4cf640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f01a815f720 0x7f01a80717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.415+0000 7f01af4cf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a815ecd0 con 0x7f01a806a500 2023-11-28T20:47:29.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.415+0000 7f01af4cf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a8071ce0 con 0x7f01a815e840 2023-11-28T20:47:29.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.415+0000 7f01af4cf640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a815f290 con 0x7f01a815f720 2023-11-28T20:47:29.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.415+0000 7f01ada6c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f01a806a500 0x7f01a80cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55362/0 (socket says 172.21.15.45:55362) 2023-11-28T20:47:29.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.415+0000 7f01ada6c640 1 -- 172.21.15.45:0/4272089847 learned_addr learned my addr 172.21.15.45:0/4272089847 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:29.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.415+0000 7f01ad26b640 1 --2- 172.21.15.45:0/4272089847 >> v2:172.21.15.45:3300/0 conn(0x7f01a815e840 0x7f01a806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:29.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.415+0000 7f01aca6a640 1 --2- 172.21.15.45:0/4272089847 >> v2:172.21.15.190:3300/0 conn(0x7f01a815f720 0x7f01a80717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:29.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.416+0000 7f019ffff640 1 -- 172.21.15.45:0/4272089847 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488870050 0 0) 0x7f01a815ecd0 con 0x7f01a806a500 2023-11-28T20:47:29.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.416+0000 7f019ffff640 1 -- 172.21.15.45:0/4272089847 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0194003520 con 0x7f01a806a500 2023-11-28T20:47:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.416+0000 7f01ad26b640 1 -- 172.21.15.45:0/4272089847 >> v2:172.21.15.190:3300/0 conn(0x7f01a815f720 msgr2=0x7f01a80717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.416+0000 7f01ad26b640 1 --2- 172.21.15.45:0/4272089847 >> v2:172.21.15.190:3300/0 conn(0x7f01a815f720 0x7f01a80717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.416+0000 7f01ad26b640 1 -- 172.21.15.45:0/4272089847 >> v1:172.21.15.45:6789/0 conn(0x7f01a806a500 legacy=0x7f01a80cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.416+0000 7f01ad26b640 1 -- 172.21.15.45:0/4272089847 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a815fb60 con 0x7f01a815e840 2023-11-28T20:47:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.417+0000 7f01ad26b640 1 --2- 172.21.15.45:0/4272089847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815e840 0x7f01a806c350 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f01a806e6d0 tx=0x7f01a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cef3e15c9efa0e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.417+0000 7f019ffff640 1 -- 172.21.15.45:0/4272089847 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a4031830 con 0x7f01a815e840 2023-11-28T20:47:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.417+0000 7f019ffff640 1 -- 172.21.15.45:0/4272089847 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a4031990 con 0x7f01a815e840 2023-11-28T20:47:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.417+0000 7f019ffff640 1 -- 172.21.15.45:0/4272089847 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a4031d20 con 0x7f01a815e840 2023-11-28T20:47:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.418+0000 7f01af4cf640 1 -- 172.21.15.45:0/4272089847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815e840 msgr2=0x7f01a806c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.418+0000 7f01af4cf640 1 --2- 172.21.15.45:0/4272089847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815e840 0x7f01a806c350 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f01a806e6d0 tx=0x7f01a402b8d0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.418+0000 7f01af4cf640 1 -- 172.21.15.45:0/4272089847 shutdown_connections 2023-11-28T20:47:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.418+0000 7f01af4cf640 1 --2- 172.21.15.45:0/4272089847 >> v2:172.21.15.190:3300/0 conn(0x7f01a815f720 0x7f01a80717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.418+0000 7f01af4cf640 1 --2- 172.21.15.45:0/4272089847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815e840 0x7f01a806c350 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.418+0000 7f01af4cf640 1 -- 172.21.15.45:0/4272089847 >> 172.21.15.45:0/4272089847 conn(0x7f01a80ced60 msgr2=0x7f01a80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.418+0000 7f01af4cf640 1 -- 172.21.15.45:0/4272089847 shutdown_connections 2023-11-28T20:47:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.418+0000 7f01af4cf640 1 -- 172.21.15.45:0/4272089847 wait complete. 2023-11-28T20:47:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.420+0000 7f01af4cf640 1 Processor -- start 2023-11-28T20:47:29.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.420+0000 7f01af4cf640 1 -- start start 2023-11-28T20:47:29.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01af4cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01a806a500 0x7f01a80c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01af4cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01a815e840 0x7f01a80c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01af4cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815f720 0x7f01a8158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01af4cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80c4dd0 con 0x7f01a815f720 2023-11-28T20:47:29.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01af4cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80c8050 con 0x7f01a815e840 2023-11-28T20:47:29.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01af4cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80c4530 con 0x7f01a806a500 2023-11-28T20:47:29.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01ada6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815f720 0x7f01a8158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:29.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01ada6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815f720 0x7f01a8158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57330/0 (socket says 172.21.15.45:57330) 2023-11-28T20:47:29.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01ada6c640 1 -- 172.21.15.45:0/4197225388 learned_addr learned my addr 172.21.15.45:0/4197225388 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:29.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01aca6a640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01a815e840 0x7f01a80c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:29.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.421+0000 7f01ada6c640 1 -- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01a806a500 msgr2=0x7f01a80c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f01ada6c640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01a806a500 0x7f01a80c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f01ada6c640 1 -- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01a815e840 msgr2=0x7f01a80c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f01ada6c640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01a815e840 0x7f01a80c3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f01ada6c640 1 -- 172.21.15.45:0/4197225388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a4009400 con 0x7f01a815f720 2023-11-28T20:47:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f01ada6c640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815f720 0x7f01a8158320 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f01a000caa0 tx=0x7f01a000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f019dffb640 1 -- 172.21.15.45:0/4197225388 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a000ac00 con 0x7f01a815f720 2023-11-28T20:47:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f019dffb640 1 -- 172.21.15.45:0/4197225388 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a000ad60 con 0x7f01a815f720 2023-11-28T20:47:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f019dffb640 1 -- 172.21.15.45:0/4197225388 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0012620 con 0x7f01a815f720 2023-11-28T20:47:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a80c4810 con 0x7f01a815f720 2023-11-28T20:47:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.422+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a8158860 con 0x7f01a815f720 2023-11-28T20:47:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.423+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a80733d0 con 0x7f01a815f720 2023-11-28T20:47:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.424+0000 7f019dffb640 1 -- 172.21.15.45:0/4197225388 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f01a0007070 con 0x7f01a815f720 2023-11-28T20:47:29.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.425+0000 7f019dffb640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f018c06b040 0x7f018c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.425+0000 7f019dffb640 1 -- 172.21.15.45:0/4197225388 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f01a0089c50 con 0x7f01a815f720 2023-11-28T20:47:29.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.426+0000 7f01ad26b640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f018c06b040 0x7f018c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:29.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.427+0000 7f01ad26b640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f018c06b040 0x7f018c06d500 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f01a4036000 tx=0x7f01a4030000 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:29.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.430+0000 7f019dffb640 1 -- 172.21.15.45:0/4197225388 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01a0054780 con 0x7f01a815f720 2023-11-28T20:47:29.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.645+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01a80c7ed0 con 0x7f018c06b040 2023-11-28T20:47:29.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.666+0000 7f019dffb640 1 -- 172.21.15.45:0/4197225388 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f01a4031630 con 0x7f018c06b040 2023-11-28T20:47:29.668 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:29.669 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:57", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:29.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.668+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f018c06b040 msgr2=0x7f018c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.668+0000 7f01af4cf640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f018c06b040 0x7f018c06d500 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f01a4036000 tx=0x7f01a4030000 comp rx=0 tx=0).stop 2023-11-28T20:47:29.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815f720 msgr2=0x7f01a8158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815f720 0x7f01a8158320 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f01a000caa0 tx=0x7f01a000cf60 comp rx=0 tx=0).stop 2023-11-28T20:47:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 shutdown_connections 2023-11-28T20:47:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f018c06b040 0x7f018c06d500 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01a815f720 0x7f01a8158320 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01a815e840 0x7f01a80c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 --2- 172.21.15.45:0/4197225388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01a806a500 0x7f01a80c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 >> 172.21.15.45:0/4197225388 conn(0x7f01a80ced60 msgr2=0x7f01a80abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 shutdown_connections 2023-11-28T20:47:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.669+0000 7f01af4cf640 1 -- 172.21.15.45:0/4197225388 wait complete. 2023-11-28T20:47:29.691 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:57", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:29.692 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:35:57', 'first': '2023-11-28T20:36:00', 'last': '2023-11-28T20:47:01', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:29.692 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-11-28T20:47:29.692 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-11-28T20:47:29.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:29.693 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-11-28T20:47:29.755 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:29.755 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:29.755 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:29.756 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:29.756 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:29.756 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:29.756 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:29.756 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:29.756 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:29.757 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_01_UTC 2023-11-28T20:47:29.757 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:29.757 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:29.757 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_36_00_UTC', 'scheduled-2023-11-28-20_37_00_UTC', 'scheduled-2023-11-28-20_38_00_UTC', 'scheduled-2023-11-28-20_39_00_UTC', 'scheduled-2023-11-28-20_40_00_UTC', 'scheduled-2023-11-28-20_41_00_UTC', 'scheduled-2023-11-28-20_42_00_UTC', 'scheduled-2023-11-28-20_43_00_UTC', 'scheduled-2023-11-28-20_44_00_UTC', 'scheduled-2023-11-28-20_45_01_UTC', 'scheduled-2023-11-28-20_46_01_UTC', 'scheduled-2023-11-28-20_47_00_UTC'] 2023-11-28T20:47:29.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2023-11-28T20:47:29.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.986+0000 7f7a22264640 1 Processor -- start 2023-11-28T20:47:29.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.987+0000 7f7a22264640 1 -- start start 2023-11-28T20:47:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.988+0000 7f7a22264640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a1c15d7b0 0x7f7a1c06b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.988+0000 7f7a22264640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1c06b820 con 0x7f7a1c15db90 2023-11-28T20:47:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.988+0000 7f7a22264640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1c0cf780 con 0x7f7a1c09a150 2023-11-28T20:47:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.988+0000 7f7a22264640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0cfa60 con 0x7f7a1c15d7b0 2023-11-28T20:47:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.989+0000 7f7a1b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7a1c15db90 0x7f7a1c099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55370/0 (socket says 172.21.15.45:55370) 2023-11-28T20:47:29.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.989+0000 7f7a1b7fe640 1 -- 172.21.15.45:0/489631212 learned_addr learned my addr 172.21.15.45:0/489631212 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:29.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.989+0000 7f7a1bfff640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a1c15d7b0 0x7f7a1c06b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:29.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.989+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985054506 0 0) 0x7f7a1c06b820 con 0x7f7a1c15db90 2023-11-28T20:47:29.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.989+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a08003520 con 0x7f7a1c15db90 2023-11-28T20:47:29.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043936017 0 0) 0x7f7a08003520 con 0x7f7a1c15db90 2023-11-28T20:47:29.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a1c15d7b0 msgr2=0x7f7a1c06b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:29.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a1c15d7b0 0x7f7a1c06b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 >> v1:172.21.15.190:6789/0 conn(0x7f7a1c09a150 legacy=0x7f7a1c0d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c0cfd40 con 0x7f7a1c15db90 2023-11-28T20:47:29.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f7a0c001670 con 0x7f7a1c15db90 2023-11-28T20:47:29.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 >> v1:172.21.15.45:6789/0 conn(0x7f7a1c15db90 legacy=0x7f7a1c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a080039a0 0x7f7a08003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a1c15d7b0 0x7f7a080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a08004d00 0x7f7a08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:29.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0cfa60 con 0x7f7a080039a0 2023-11-28T20:47:29.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0800ba60 con 0x7f7a08004d00 2023-11-28T20:47:29.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0800bc20 con 0x7f7a1c15d7b0 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.990+0000 7f7a1bfff640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a080039a0 0x7f7a08003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.991+0000 7f7a1b7fe640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a1c15d7b0 0x7f7a080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.991+0000 7f7a1bfff640 1 -- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a1c15d7b0 msgr2=0x7f7a080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.991+0000 7f7a1bfff640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a1c15d7b0 0x7f7a080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.991+0000 7f7a1bfff640 1 -- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a08004d00 msgr2=0x7f7a08005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.991+0000 7f7a20801640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a08004d00 0x7f7a08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.991+0000 7f7a1bfff640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a08004d00 0x7f7a08005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.991+0000 7f7a1bfff640 1 -- 172.21.15.45:0/489631212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a08003520 con 0x7f7a080039a0 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.991+0000 7f7a1b7fe640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a1c15d7b0 0x7f7a080047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.992+0000 7f7a1bfff640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a080039a0 0x7f7a08003dc0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009a90 tx=0x7f7a080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91a47ecb46a80b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:29.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.992+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0402c680 con 0x7f7a080039a0 2023-11-28T20:47:29.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.992+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0402c7e0 con 0x7f7a080039a0 2023-11-28T20:47:29.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.992+0000 7f7a1affd640 1 -- 172.21.15.45:0/489631212 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0402cb70 con 0x7f7a080039a0 2023-11-28T20:47:29.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.993+0000 7f7a22264640 1 -- 172.21.15.45:0/489631212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a080039a0 msgr2=0x7f7a08003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:29.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.993+0000 7f7a22264640 1 --2- 172.21.15.45:0/489631212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a080039a0 0x7f7a08003dc0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009a90 tx=0x7f7a080057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:29.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.993+0000 7f7a20801640 1 -- 172.21.15.45:0/489631212 reap_dead start 2023-11-28T20:47:29.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.993+0000 7f7a22264640 1 -- 172.21.15.45:0/489631212 shutdown_connections 2023-11-28T20:47:29.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.993+0000 7f7a22264640 1 -- 172.21.15.45:0/489631212 >> 172.21.15.45:0/489631212 conn(0x7f7a1c0b2b20 msgr2=0x7f7a1c0b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:29.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.993+0000 7f7a22264640 1 -- 172.21.15.45:0/489631212 shutdown_connections 2023-11-28T20:47:29.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.993+0000 7f7a22264640 1 -- 172.21.15.45:0/489631212 wait complete. 2023-11-28T20:47:30.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.995+0000 7f7a22264640 1 Processor -- start 2023-11-28T20:47:30.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.995+0000 7f7a22264640 1 -- start start 2023-11-28T20:47:30.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.996+0000 7f7a22264640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a080039a0 0x7f7a1c0c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.996+0000 7f7a22264640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a1c0c5a80 0x7f7a1c0b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.996+0000 7f7a22264640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1c0c8100 0x7f7a1c0c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.996+0000 7f7a22264640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0cb070 con 0x7f7a1c0c8100 2023-11-28T20:47:30.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.996+0000 7f7a22264640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0c4cc0 con 0x7f7a1c0c5a80 2023-11-28T20:47:30.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.996+0000 7f7a22264640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0c4fa0 con 0x7f7a080039a0 2023-11-28T20:47:30.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a20801640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1c0c8100 0x7f7a1c0c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:30.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a20801640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1c0c8100 0x7f7a1c0c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57338/0 (socket says 172.21.15.45:57338) 2023-11-28T20:47:30.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a20801640 1 -- 172.21.15.45:0/2667256452 learned_addr learned my addr 172.21.15.45:0/2667256452 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:30.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a1bfff640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a080039a0 0x7f7a1c0c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:30.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a20801640 1 -- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a080039a0 msgr2=0x7f7a1c0c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a20801640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a080039a0 0x7f7a1c0c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a20801640 1 -- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a1c0c5a80 msgr2=0x7f7a1c0b60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a20801640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a1c0c5a80 0x7f7a1c0b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a20801640 1 -- 172.21.15.45:0/2667256452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c0c5280 con 0x7f7a1c0c8100 2023-11-28T20:47:30.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.997+0000 7f7a20801640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1c0c8100 0x7f7a1c0c8580 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0c4b10 tx=0x7f7a1400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:30.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.998+0000 7f7a18ff9640 1 -- 172.21.15.45:0/2667256452 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14003b70 con 0x7f7a1c0c8100 2023-11-28T20:47:30.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.998+0000 7f7a18ff9640 1 -- 172.21.15.45:0/2667256452 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a14003cd0 con 0x7f7a1c0c8100 2023-11-28T20:47:30.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.998+0000 7f7a18ff9640 1 -- 172.21.15.45:0/2667256452 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a140052b0 con 0x7f7a1c0c8100 2023-11-28T20:47:30.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.998+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1c0c5560 con 0x7f7a1c0c8100 2023-11-28T20:47:30.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.998+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c0b67e0 con 0x7f7a1c0c8100 2023-11-28T20:47:30.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:29.999+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1c0b2920 con 0x7f7a1c0c8100 2023-11-28T20:47:30.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.000+0000 7f7a18ff9640 1 -- 172.21.15.45:0/2667256452 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f7a14005410 con 0x7f7a1c0c8100 2023-11-28T20:47:30.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.001+0000 7f7a18ff9640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f79fc06b040 0x7f79fc06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.001+0000 7f7a18ff9640 1 -- 172.21.15.45:0/2667256452 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f7a140894c0 con 0x7f7a1c0c8100 2023-11-28T20:47:30.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.002+0000 7f7a1bfff640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f79fc06b040 0x7f79fc06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:30.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.003+0000 7f7a1bfff640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f79fc06b040 0x7f79fc06d500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7a04036000 tx=0x7f7a04030000 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:30.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.005+0000 7f7a18ff9640 1 -- 172.21.15.45:0/2667256452 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a14053ee0 con 0x7f7a1c0c8100 2023-11-28T20:47:30.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.224+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7a1c0b67e0 con 0x7f79fc06b040 2023-11-28T20:47:30.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.243+0000 7f7a18ff9640 1 -- 172.21.15.45:0/2667256452 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f7a1c0b67e0 con 0x7f79fc06b040 2023-11-28T20:47:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:30.246 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:58", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.245+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f79fc06b040 msgr2=0x7f79fc06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.245+0000 7f7a22264640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f79fc06b040 0x7f79fc06d500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7a04036000 tx=0x7f7a04030000 comp rx=0 tx=0).stop 2023-11-28T20:47:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.245+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1c0c8100 msgr2=0x7f7a1c0c8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.245+0000 7f7a22264640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1c0c8100 0x7f7a1c0c8580 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0c4b10 tx=0x7f7a1400dc00 comp rx=0 tx=0).stop 2023-11-28T20:47:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.246+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 shutdown_connections 2023-11-28T20:47:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.246+0000 7f7a22264640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f79fc06b040 0x7f79fc06d500 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.246+0000 7f7a22264640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1c0c8100 0x7f7a1c0c8580 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.246+0000 7f7a22264640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a1c0c5a80 0x7f7a1c0b60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.246+0000 7f7a22264640 1 --2- 172.21.15.45:0/2667256452 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a080039a0 0x7f7a1c0c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.246+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 >> 172.21.15.45:0/2667256452 conn(0x7f7a1c0b2b20 msgr2=0x7f7a1c099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:30.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.246+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 shutdown_connections 2023-11-28T20:47:30.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.246+0000 7f7a22264640 1 -- 172.21.15.45:0/2667256452 wait complete. 2023-11-28T20:47:30.267 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:58", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:30.268 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:35:58', 'first': '2023-11-28T20:36:00', 'last': '2023-11-28T20:47:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:30.268 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-11-28T20:47:30.268 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-11-28T20:47:30.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:30.268 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-11-28T20:47:30.326 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:30.326 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:30.326 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:30.326 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:30.326 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:30.327 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_01_UTC 2023-11-28T20:47:30.327 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:30.327 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:30.327 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:30.327 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:30.327 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:30.327 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:30.328 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_36_00_UTC', 'scheduled-2023-11-28-20_37_00_UTC', 'scheduled-2023-11-28-20_38_00_UTC', 'scheduled-2023-11-28-20_39_00_UTC', 'scheduled-2023-11-28-20_40_00_UTC', 'scheduled-2023-11-28-20_41_01_UTC', 'scheduled-2023-11-28-20_42_00_UTC', 'scheduled-2023-11-28-20_43_00_UTC', 'scheduled-2023-11-28-20_44_00_UTC', 'scheduled-2023-11-28-20_45_00_UTC', 'scheduled-2023-11-28-20_46_00_UTC', 'scheduled-2023-11-28-20_47_00_UTC'] 2023-11-28T20:47:30.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2023-11-28T20:47:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.560+0000 7f4950748640 1 Processor -- start 2023-11-28T20:47:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.560+0000 7f4950748640 1 -- start start 2023-11-28T20:47:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.561+0000 7f4950748640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 0x7f494806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.561+0000 7f4950748640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f494815f720 0x7f49480717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.561+0000 7f4950748640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494815ecd0 con 0x7f494815e840 2023-11-28T20:47:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.561+0000 7f4950748640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4948071ce0 con 0x7f494815f720 2023-11-28T20:47:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.561+0000 7f4950748640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494815fbd0 con 0x7f494806ac10 2023-11-28T20:47:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.561+0000 7f494ece5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f494815f720 0x7f49480717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.561+0000 7f494ece5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f494815f720 0x7f49480717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57352/0 (socket says 172.21.15.45:57352) 2023-11-28T20:47:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.561+0000 7f494ece5640 1 -- 172.21.15.45:0/4175590780 learned_addr learned my addr 172.21.15.45:0/4175590780 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.562+0000 7f494dce3640 1 --2- 172.21.15.45:0/4175590780 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 0x7f494806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.562+0000 7f494ece5640 1 -- 172.21.15.45:0/4175590780 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 msgr2=0x7f494806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.562+0000 7f494ece5640 1 --2- 172.21.15.45:0/4175590780 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 0x7f494806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.562+0000 7f494ece5640 1 -- 172.21.15.45:0/4175590780 >> v1:172.21.15.190:6789/0 conn(0x7f494815e840 legacy=0x7f494806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.562+0000 7f494ece5640 1 -- 172.21.15.45:0/4175590780 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494815f290 con 0x7f494815f720 2023-11-28T20:47:30.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.563+0000 7f494ece5640 1 --2- 172.21.15.45:0/4175590780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 0x7f49480717a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4940009e80 tx=0x7f494002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e476bf3ece54d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.563+0000 7f494d4e2640 1 -- 172.21.15.45:0/4175590780 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494002caf0 con 0x7f494815f720 2023-11-28T20:47:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.563+0000 7f494d4e2640 1 -- 172.21.15.45:0/4175590780 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494002cc50 con 0x7f494815f720 2023-11-28T20:47:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.563+0000 7f494d4e2640 1 -- 172.21.15.45:0/4175590780 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49400343f0 con 0x7f494815f720 2023-11-28T20:47:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.564+0000 7f4950748640 1 -- 172.21.15.45:0/4175590780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 msgr2=0x7f49480717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.564+0000 7f4950748640 1 --2- 172.21.15.45:0/4175590780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 0x7f49480717a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4940009e80 tx=0x7f494002bf50 comp rx=0 tx=0).stop 2023-11-28T20:47:30.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.564+0000 7f4950748640 1 -- 172.21.15.45:0/4175590780 shutdown_connections 2023-11-28T20:47:30.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.564+0000 7f4950748640 1 --2- 172.21.15.45:0/4175590780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 0x7f49480717a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.564+0000 7f4950748640 1 --2- 172.21.15.45:0/4175590780 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 0x7f494806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.564+0000 7f4950748640 1 -- 172.21.15.45:0/4175590780 >> 172.21.15.45:0/4175590780 conn(0x7f49480ced60 msgr2=0x7f49480cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:30.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.564+0000 7f4950748640 1 -- 172.21.15.45:0/4175590780 shutdown_connections 2023-11-28T20:47:30.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.564+0000 7f4950748640 1 -- 172.21.15.45:0/4175590780 wait complete. 2023-11-28T20:47:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.566+0000 7f4950748640 1 Processor -- start 2023-11-28T20:47:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.566+0000 7f4950748640 1 -- start start 2023-11-28T20:47:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.567+0000 7f4950748640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 0x7f49481e4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.567+0000 7f4950748640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f494815e840 0x7f4948148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.567+0000 7f4950748640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 0x7f4948149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.567+0000 7f4950748640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4948147840 con 0x7f494815f720 2023-11-28T20:47:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.567+0000 7f4950748640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49480c8c70 con 0x7f494815e840 2023-11-28T20:47:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.567+0000 7f4950748640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49481e5370 con 0x7f494806ac10 2023-11-28T20:47:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.567+0000 7f494ece5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 0x7f4948149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.567+0000 7f494ece5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 0x7f4948149440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57366/0 (socket says 172.21.15.45:57366) 2023-11-28T20:47:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.567+0000 7f494ece5640 1 -- 172.21.15.45:0/2979145765 learned_addr learned my addr 172.21.15.45:0/2979145765 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.568+0000 7f494dce3640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f494815e840 0x7f4948148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.568+0000 7f494e4e4640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 0x7f49481e4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.568+0000 7f494ece5640 1 -- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 msgr2=0x7f49481e4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.568+0000 7f494ece5640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 0x7f49481e4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.568+0000 7f494ece5640 1 -- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f494815e840 msgr2=0x7f4948148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.568+0000 7f494ece5640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f494815e840 0x7f4948148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.568+0000 7f494ece5640 1 -- 172.21.15.45:0/2979145765 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4940009b30 con 0x7f494815f720 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.568+0000 7f494dce3640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f494815e840 0x7f4948148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.568+0000 7f494ece5640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 0x7f4948149440 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4940036040 tx=0x7f4940004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.569+0000 7f493effd640 1 -- 172.21.15.45:0/2979145765 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940040670 con 0x7f494815f720 2023-11-28T20:47:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.569+0000 7f493effd640 1 -- 172.21.15.45:0/2979145765 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49400407f0 con 0x7f494815f720 2023-11-28T20:47:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.569+0000 7f493effd640 1 -- 172.21.15.45:0/2979145765 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940040ba0 con 0x7f494815f720 2023-11-28T20:47:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.569+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4948146f10 con 0x7f494815f720 2023-11-28T20:47:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.569+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49481474d0 con 0x7f494815f720 2023-11-28T20:47:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.570+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49480ce9a0 con 0x7f494815f720 2023-11-28T20:47:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.571+0000 7f493effd640 1 -- 172.21.15.45:0/2979145765 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f4940040d00 con 0x7f494815f720 2023-11-28T20:47:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.572+0000 7f493effd640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f492c06b040 0x7f492c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.572+0000 7f493effd640 1 -- 172.21.15.45:0/2979145765 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f49400acbe0 con 0x7f494815f720 2023-11-28T20:47:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.573+0000 7f494e4e4640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f492c06b040 0x7f492c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.574+0000 7f494e4e4640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f492c06b040 0x7f492c06d500 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4944000f60 tx=0x7f4944009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:30.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.576+0000 7f493effd640 1 -- 172.21.15.45:0/2979145765 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49400778c0 con 0x7f494815f720 2023-11-28T20:47:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.790+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f494815ef20 con 0x7f492c06b040 2023-11-28T20:47:30.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.810+0000 7f493effd640 1 -- 172.21.15.45:0/2979145765 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f4944001360 con 0x7f492c06b040 2023-11-28T20:47:30.811 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:30.812 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:58", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f492c06b040 msgr2=0x7f492c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f492c06b040 0x7f492c06d500 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4944000f60 tx=0x7f4944009040 comp rx=0 tx=0).stop 2023-11-28T20:47:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 msgr2=0x7f4948149440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 0x7f4948149440 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4940036040 tx=0x7f4940004a40 comp rx=0 tx=0).stop 2023-11-28T20:47:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 shutdown_connections 2023-11-28T20:47:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f492c06b040 0x7f492c06d500 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f494815f720 0x7f4948149440 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f494815e840 0x7f4948148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 --2- 172.21.15.45:0/2979145765 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f494806ac10 0x7f49481e4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 >> 172.21.15.45:0/2979145765 conn(0x7f49480ced60 msgr2=0x7f49480cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 shutdown_connections 2023-11-28T20:47:30.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:30.812+0000 7f4950748640 1 -- 172.21.15.45:0/2979145765 wait complete. 2023-11-28T20:47:30.834 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:58", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:30.834 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:35:58', 'first': '2023-11-28T20:36:00', 'last': '2023-11-28T20:47:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:30.834 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-11-28T20:47:30.834 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-11-28T20:47:30.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:30.835 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-11-28T20:47:30.898 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:30.898 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_01_UTC 2023-11-28T20:47:30.898 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:30.898 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:30.898 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:30.898 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:30.899 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:30.899 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:30.899 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:30.900 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:30.900 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:30.900 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_01_UTC 2023-11-28T20:47:30.901 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_36_00_UTC', 'scheduled-2023-11-28-20_37_01_UTC', 'scheduled-2023-11-28-20_38_00_UTC', 'scheduled-2023-11-28-20_39_00_UTC', 'scheduled-2023-11-28-20_40_00_UTC', 'scheduled-2023-11-28-20_41_00_UTC', 'scheduled-2023-11-28-20_42_01_UTC', 'scheduled-2023-11-28-20_43_01_UTC', 'scheduled-2023-11-28-20_44_00_UTC', 'scheduled-2023-11-28-20_45_00_UTC', 'scheduled-2023-11-28-20_46_01_UTC', 'scheduled-2023-11-28-20_47_01_UTC'] 2023-11-28T20:47:30.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2023-11-28T20:47:31.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.133+0000 7f17fa21e640 1 Processor -- start 2023-11-28T20:47:31.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.133+0000 7f17fa21e640 1 -- start start 2023-11-28T20:47:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.134+0000 7f17fa21e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40b1500 0x7f17f40b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.134+0000 7f17fa21e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17f4162a10 0x7f17f40cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.134+0000 7f17fa21e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f40b18e0 con 0x7f17f40c4080 2023-11-28T20:47:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17fa21e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f4162ed0 con 0x7f17f4162a10 2023-11-28T20:47:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17fa21e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f406a5a0 con 0x7f17f40b1500 2023-11-28T20:47:31.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17f2ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17f4162a10 0x7f17f40cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17f2ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17f4162a10 0x7f17f40cec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57378/0 (socket says 172.21.15.45:57378) 2023-11-28T20:47:31.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17f2ffd640 1 -- 172.21.15.45:0/2659683857 learned_addr learned my addr 172.21.15.45:0/2659683857 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:31.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17f3fff640 1 --1- 172.21.15.45:0/2659683857 >> v1:172.21.15.45:6789/0 conn(0x7f17f40c4080 0x7f17f40c44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55378/0 (socket says 172.21.15.45:55378) 2023-11-28T20:47:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17f37fe640 1 --2- 172.21.15.45:0/2659683857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40b1500 0x7f17f40b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17f2ffd640 1 -- 172.21.15.45:0/2659683857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40b1500 msgr2=0x7f17f40b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17f2ffd640 1 --2- 172.21.15.45:0/2659683857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40b1500 0x7f17f40b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17f2ffd640 1 -- 172.21.15.45:0/2659683857 >> v1:172.21.15.45:6789/0 conn(0x7f17f40c4080 legacy=0x7f17f40c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.135+0000 7f17f2ffd640 1 -- 172.21.15.45:0/2659683857 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f406a880 con 0x7f17f4162a10 2023-11-28T20:47:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.137+0000 7f17f2ffd640 1 --2- 172.21.15.45:0/2659683857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 0x7f17f40cec80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009870 tx=0x7f17e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a30369f3e1fc6e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.137+0000 7f17f27fc640 1 -- 172.21.15.45:0/2659683857 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8030840 con 0x7f17f4162a10 2023-11-28T20:47:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.137+0000 7f17f27fc640 1 -- 172.21.15.45:0/2659683857 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e80309a0 con 0x7f17f4162a10 2023-11-28T20:47:31.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.137+0000 7f17f27fc640 1 -- 172.21.15.45:0/2659683857 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8030d30 con 0x7f17f4162a10 2023-11-28T20:47:31.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.137+0000 7f17fa21e640 1 -- 172.21.15.45:0/2659683857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 msgr2=0x7f17f40cec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:31.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.137+0000 7f17fa21e640 1 --2- 172.21.15.45:0/2659683857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 0x7f17f40cec80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009870 tx=0x7f17e802b980 comp rx=0 tx=0).stop 2023-11-28T20:47:31.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.138+0000 7f17fa21e640 1 -- 172.21.15.45:0/2659683857 shutdown_connections 2023-11-28T20:47:31.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.138+0000 7f17fa21e640 1 --2- 172.21.15.45:0/2659683857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 0x7f17f40cec80 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.138+0000 7f17fa21e640 1 --2- 172.21.15.45:0/2659683857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40b1500 0x7f17f40b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.138+0000 7f17fa21e640 1 -- 172.21.15.45:0/2659683857 >> 172.21.15.45:0/2659683857 conn(0x7f17f40b2ba0 msgr2=0x7f17f40b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:31.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.138+0000 7f17fa21e640 1 -- 172.21.15.45:0/2659683857 shutdown_connections 2023-11-28T20:47:31.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.138+0000 7f17fa21e640 1 -- 172.21.15.45:0/2659683857 wait complete. 2023-11-28T20:47:31.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.139+0000 7f17fa21e640 1 Processor -- start 2023-11-28T20:47:31.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.140+0000 7f17fa21e640 1 -- start start 2023-11-28T20:47:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17fa21e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f40b1500 0x7f17f40cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17fa21e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40c4080 0x7f17f40c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17fa21e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 0x7f17f40c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17fa21e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f40c6330 con 0x7f17f4162a10 2023-11-28T20:47:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17fa21e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f40c6610 con 0x7f17f40b1500 2023-11-28T20:47:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17fa21e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f40c68f0 con 0x7f17f40c4080 2023-11-28T20:47:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 0x7f17f40c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 0x7f17f40c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57382/0 (socket says 172.21.15.45:57382) 2023-11-28T20:47:31.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f3fff640 1 -- 172.21.15.45:0/845582724 learned_addr learned my addr 172.21.15.45:0/845582724 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:31.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f37fe640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f40b1500 0x7f17f40cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f2ffd640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40c4080 0x7f17f40c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f3fff640 1 -- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40c4080 msgr2=0x7f17f40c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:31.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f3fff640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40c4080 0x7f17f40c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f3fff640 1 -- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f40b1500 msgr2=0x7f17f40cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:31.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f3fff640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f40b1500 0x7f17f40cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f3fff640 1 -- 172.21.15.45:0/845582724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80094d0 con 0x7f17f4162a10 2023-11-28T20:47:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f2ffd640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40c4080 0x7f17f40c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.141+0000 7f17f37fe640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f40b1500 0x7f17f40cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.142+0000 7f17f3fff640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 0x7f17f40c86c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00c7a0 tx=0x7f17ec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.142+0000 7f17dbfff640 1 -- 172.21.15.45:0/845582724 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec00a640 con 0x7f17f4162a10 2023-11-28T20:47:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.142+0000 7f17dbfff640 1 -- 172.21.15.45:0/845582724 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17ec00a7c0 con 0x7f17f4162a10 2023-11-28T20:47:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.142+0000 7f17dbfff640 1 -- 172.21.15.45:0/845582724 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec00ab70 con 0x7f17f4162a10 2023-11-28T20:47:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.142+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f40c6bd0 con 0x7f17f4162a10 2023-11-28T20:47:31.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.142+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f40c5600 con 0x7f17f4162a10 2023-11-28T20:47:31.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.143+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f40733d0 con 0x7f17f4162a10 2023-11-28T20:47:31.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.144+0000 7f17dbfff640 1 -- 172.21.15.45:0/845582724 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f17ec00acd0 con 0x7f17f4162a10 2023-11-28T20:47:31.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.145+0000 7f17dbfff640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f17e006b040 0x7f17e006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.145+0000 7f17dbfff640 1 -- 172.21.15.45:0/845582724 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f17ec0894c0 con 0x7f17f4162a10 2023-11-28T20:47:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.146+0000 7f17f37fe640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f17e006b040 0x7f17e006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.147+0000 7f17f37fe640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f17e006b040 0x7f17e006d500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f17e4006fd0 tx=0x7f17e4008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:31.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.149+0000 7f17dbfff640 1 -- 172.21.15.45:0/845582724 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17ec053ff0 con 0x7f17f4162a10 2023-11-28T20:47:31.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.373+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17f40c6780 con 0x7f17e006b040 2023-11-28T20:47:31.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.393+0000 7f17dbfff640 1 -- 172.21.15.45:0/845582724 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f17f40c6780 con 0x7f17e006b040 2023-11-28T20:47:31.397 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:31.398 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:59", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f17e006b040 msgr2=0x7f17e006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f17e006b040 0x7f17e006d500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f17e4006fd0 tx=0x7f17e4008040 comp rx=0 tx=0).stop 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 msgr2=0x7f17f40c86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 0x7f17f40c86c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00c7a0 tx=0x7f17ec00cc60 comp rx=0 tx=0).stop 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 shutdown_connections 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f17e006b040 0x7f17e006d500 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17f4162a10 0x7f17f40c86c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f40c4080 0x7f17f40c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 --2- 172.21.15.45:0/845582724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f40b1500 0x7f17f40cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 >> 172.21.15.45:0/845582724 conn(0x7f17f40b2ba0 msgr2=0x7f17f40c7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:31.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 shutdown_connections 2023-11-28T20:47:31.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.396+0000 7f17fa21e640 1 -- 172.21.15.45:0/845582724 wait complete. 2023-11-28T20:47:31.420 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:35:59", "first": "2023-11-28T20:36:00", "last": "2023-11-28T20:47:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-11-28T20:47:31.420 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:35:59', 'first': '2023-11-28T20:36:00', 'last': '2023-11-28T20:47:01', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:31.420 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-11-28T20:47:31.420 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-11-28T20:47:31.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:31.421 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-11-28T20:47:31.477 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:31.477 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_01_UTC 2023-11-28T20:47:31.478 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_01_UTC 2023-11-28T20:47:31.478 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:31.478 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_01_UTC 2023-11-28T20:47:31.478 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:31.478 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:31.478 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:31.478 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:31.478 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:31.478 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:31.479 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_37_00_UTC', 'scheduled-2023-11-28-20_38_01_UTC', 'scheduled-2023-11-28-20_39_01_UTC', 'scheduled-2023-11-28-20_40_01_UTC', 'scheduled-2023-11-28-20_41_01_UTC', 'scheduled-2023-11-28-20_42_01_UTC', 'scheduled-2023-11-28-20_43_01_UTC', 'scheduled-2023-11-28-20_44_01_UTC', 'scheduled-2023-11-28-20_45_00_UTC', 'scheduled-2023-11-28-20_46_00_UTC', 'scheduled-2023-11-28-20_47_00_UTC'] 2023-11-28T20:47:31.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2023-11-28T20:47:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.709+0000 7f8282472640 1 Processor -- start 2023-11-28T20:47:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.709+0000 7f8282472640 1 -- start start 2023-11-28T20:47:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.711+0000 7f8282472640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f827c06bc20 0x7f827c0ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.711+0000 7f8282472640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f827c161900 0x7f827c0b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.711+0000 7f8282472640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c0af3e0 con 0x7f827c15d060 2023-11-28T20:47:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.711+0000 7f8282472640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827c160e20 con 0x7f827c161900 2023-11-28T20:47:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.711+0000 7f8282472640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827c161100 con 0x7f827c06bc20 2023-11-28T20:47:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.711+0000 7f8281c71640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f827c161900 0x7f827c0b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.711+0000 7f8281c71640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f827c161900 0x7f827c0b30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57386/0 (socket says 172.21.15.45:57386) 2023-11-28T20:47:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.711+0000 7f8281c71640 1 -- 172.21.15.45:0/1779218948 learned_addr learned my addr 172.21.15.45:0/1779218948 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.712+0000 7f8280c6f640 1 --2- 172.21.15.45:0/1779218948 >> v2:172.21.15.190:3300/0 conn(0x7f827c06bc20 0x7f827c0ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.712+0000 7f8281c71640 1 -- 172.21.15.45:0/1779218948 >> v2:172.21.15.190:3300/0 conn(0x7f827c06bc20 msgr2=0x7f827c0ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.712+0000 7f8281c71640 1 --2- 172.21.15.45:0/1779218948 >> v2:172.21.15.190:3300/0 conn(0x7f827c06bc20 0x7f827c0ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.712+0000 7f8281c71640 1 -- 172.21.15.45:0/1779218948 >> v1:172.21.15.190:6789/0 conn(0x7f827c15d060 legacy=0x7f827c06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.712+0000 7f8281c71640 1 -- 172.21.15.45:0/1779218948 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c1613e0 con 0x7f827c161900 2023-11-28T20:47:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.713+0000 7f8281c71640 1 --2- 172.21.15.45:0/1779218948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 0x7f827c0b30d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f827c0740f0 tx=0x7f827402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478fd25148338e6a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.713+0000 7f827bfff640 1 -- 172.21.15.45:0/1779218948 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274031870 con 0x7f827c161900 2023-11-28T20:47:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.713+0000 7f827bfff640 1 -- 172.21.15.45:0/1779218948 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82740319d0 con 0x7f827c161900 2023-11-28T20:47:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.713+0000 7f827bfff640 1 -- 172.21.15.45:0/1779218948 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274031d60 con 0x7f827c161900 2023-11-28T20:47:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.714+0000 7f8282472640 1 -- 172.21.15.45:0/1779218948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 msgr2=0x7f827c0b30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.714+0000 7f8282472640 1 --2- 172.21.15.45:0/1779218948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 0x7f827c0b30d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f827c0740f0 tx=0x7f827402b910 comp rx=0 tx=0).stop 2023-11-28T20:47:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.714+0000 7f8282472640 1 -- 172.21.15.45:0/1779218948 shutdown_connections 2023-11-28T20:47:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.714+0000 7f8282472640 1 --2- 172.21.15.45:0/1779218948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 0x7f827c0b30d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.714+0000 7f8282472640 1 --2- 172.21.15.45:0/1779218948 >> v2:172.21.15.190:3300/0 conn(0x7f827c06bc20 0x7f827c0ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.714+0000 7f8282472640 1 -- 172.21.15.45:0/1779218948 >> 172.21.15.45:0/1779218948 conn(0x7f827c071290 msgr2=0x7f827c0716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.714+0000 7f8282472640 1 -- 172.21.15.45:0/1779218948 shutdown_connections 2023-11-28T20:47:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.714+0000 7f8282472640 1 -- 172.21.15.45:0/1779218948 wait complete. 2023-11-28T20:47:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.716+0000 7f8282472640 1 Processor -- start 2023-11-28T20:47:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.716+0000 7f8282472640 1 -- start start 2023-11-28T20:47:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.717+0000 7f8282472640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f827c06bc20 0x7f827c158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.717+0000 7f8282472640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f827c15d060 0x7f827c14f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.717+0000 7f8282472640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 0x7f827c144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.717+0000 7f8282472640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c153c80 con 0x7f827c161900 2023-11-28T20:47:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.717+0000 7f8282472640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c150940 con 0x7f827c06bc20 2023-11-28T20:47:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.717+0000 7f8282472640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c150c20 con 0x7f827c15d060 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281c71640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 0x7f827c144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281c71640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 0x7f827c144150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57388/0 (socket says 172.21.15.45:57388) 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281c71640 1 -- 172.21.15.45:0/4134799674 learned_addr learned my addr 172.21.15.45:0/4134799674 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281470640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f827c06bc20 0x7f827c158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8280c6f640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f827c15d060 0x7f827c14f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281c71640 1 -- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f827c15d060 msgr2=0x7f827c14f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281c71640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f827c15d060 0x7f827c14f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281c71640 1 -- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f827c06bc20 msgr2=0x7f827c158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281c71640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f827c06bc20 0x7f827c158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281c71640 1 -- 172.21.15.45:0/4134799674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82740094d0 con 0x7f827c161900 2023-11-28T20:47:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.718+0000 7f8281c71640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 0x7f827c144150 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f82740090b0 tx=0x7f82740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.719+0000 7f8279ffb640 1 -- 172.21.15.45:0/4134799674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827403ac60 con 0x7f827c161900 2023-11-28T20:47:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.719+0000 7f8279ffb640 1 -- 172.21.15.45:0/4134799674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827403adc0 con 0x7f827c161900 2023-11-28T20:47:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.719+0000 7f8279ffb640 1 -- 172.21.15.45:0/4134799674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274044630 con 0x7f827c161900 2023-11-28T20:47:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.719+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c150f00 con 0x7f827c161900 2023-11-28T20:47:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.719+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c1514c0 con 0x7f827c161900 2023-11-28T20:47:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.720+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827c073840 con 0x7f827c161900 2023-11-28T20:47:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.721+0000 7f8279ffb640 1 -- 172.21.15.45:0/4134799674 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f827404d450 con 0x7f827c161900 2023-11-28T20:47:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.722+0000 7f8279ffb640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f826006b040 0x7f826006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.722+0000 7f8279ffb640 1 -- 172.21.15.45:0/4134799674 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f827402d040 con 0x7f827c161900 2023-11-28T20:47:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.723+0000 7f8281470640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f826006b040 0x7f826006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.724+0000 7f8281470640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f826006b040 0x7f826006d500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8270005520 tx=0x7f8270009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.727+0000 7f8279ffb640 1 -- 172.21.15.45:0/4134799674 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8274076d50 con 0x7f827c161900 2023-11-28T20:47:31.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.941+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f827c15a4c0 con 0x7f826006b040 2023-11-28T20:47:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.961+0000 7f8279ffb640 1 -- 172.21.15.45:0/4134799674 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f827c15a4c0 con 0x7f826006b040 2023-11-28T20:47:32.011 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:32.011 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:36:00", "first": "2023-11-28T20:37:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-11-28T20:47:32.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.964+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f826006b040 msgr2=0x7f826006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.964+0000 7f8282472640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f826006b040 0x7f826006d500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8270005520 tx=0x7f8270009040 comp rx=0 tx=0).stop 2023-11-28T20:47:32.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.964+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 msgr2=0x7f827c144150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.964+0000 7f8282472640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 0x7f827c144150 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f82740090b0 tx=0x7f82740046c0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.965+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 shutdown_connections 2023-11-28T20:47:32.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.965+0000 7f8282472640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f826006b040 0x7f826006d500 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.965+0000 7f8282472640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f827c161900 0x7f827c144150 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.965+0000 7f8282472640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f827c15d060 0x7f827c14f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.965+0000 7f8282472640 1 --2- 172.21.15.45:0/4134799674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f827c06bc20 0x7f827c158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.965+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 >> 172.21.15.45:0/4134799674 conn(0x7f827c071290 msgr2=0x7f827c071670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:32.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.965+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 shutdown_connections 2023-11-28T20:47:32.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:31.965+0000 7f8282472640 1 -- 172.21.15.45:0/4134799674 wait complete. 2023-11-28T20:47:32.015 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:36:00", "first": "2023-11-28T20:37:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-11-28T20:47:32.015 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:36:00', 'first': '2023-11-28T20:37:00', 'last': '2023-11-28T20:47:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:32.015 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-11-28T20:47:32.015 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-11-28T20:47:32.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:32.016 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-11-28T20:47:32.081 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_01_UTC 2023-11-28T20:47:32.081 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_01_UTC 2023-11-28T20:47:32.082 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_01_UTC 2023-11-28T20:47:32.082 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:32.082 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_01_UTC 2023-11-28T20:47:32.082 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:32.082 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:32.082 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:32.082 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:32.082 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:32.082 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:32.083 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_37_01_UTC', 'scheduled-2023-11-28-20_38_01_UTC', 'scheduled-2023-11-28-20_39_01_UTC', 'scheduled-2023-11-28-20_40_01_UTC', 'scheduled-2023-11-28-20_41_01_UTC', 'scheduled-2023-11-28-20_42_01_UTC', 'scheduled-2023-11-28-20_43_01_UTC', 'scheduled-2023-11-28-20_44_01_UTC', 'scheduled-2023-11-28-20_45_00_UTC', 'scheduled-2023-11-28-20_46_00_UTC', 'scheduled-2023-11-28-20_47_00_UTC'] 2023-11-28T20:47:32.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2023-11-28T20:47:32.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.309+0000 7f6555064640 1 Processor -- start 2023-11-28T20:47:32.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.309+0000 7f6555064640 1 -- start start 2023-11-28T20:47:32.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.311+0000 7f6555064640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f65500ad7e0 0x7f65500ac7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.311+0000 7f6555064640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f65500b1d00 0x7f65500710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.311+0000 7f6555064640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65500addf0 con 0x7f655006dbc0 2023-11-28T20:47:32.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.311+0000 7f6555064640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6550070c20 con 0x7f65500b1d00 2023-11-28T20:47:32.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.311+0000 7f6555064640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65500702f0 con 0x7f65500ad7e0 2023-11-28T20:47:32.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f654ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f655006dbc0 0x7f655006dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55384/0 (socket says 172.21.15.45:55384) 2023-11-28T20:47:32.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f654ffff640 1 -- 172.21.15.45:0/3006159510 learned_addr learned my addr 172.21.15.45:0/3006159510 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:32.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f6554863640 1 --2- 172.21.15.45:0/3006159510 >> v2:172.21.15.45:3300/0 conn(0x7f65500b1d00 0x7f65500710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f654f7fe640 1 --2- 172.21.15.45:0/3006159510 >> v2:172.21.15.190:3300/0 conn(0x7f65500ad7e0 0x7f65500ac7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f654effd640 1 -- 172.21.15.45:0/3006159510 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389025222 0 0) 0x7f65500addf0 con 0x7f655006dbc0 2023-11-28T20:47:32.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f654effd640 1 -- 172.21.15.45:0/3006159510 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6538003520 con 0x7f655006dbc0 2023-11-28T20:47:32.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f6554863640 1 -- 172.21.15.45:0/3006159510 >> v2:172.21.15.190:3300/0 conn(0x7f65500ad7e0 msgr2=0x7f65500ac7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:32.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f6554863640 1 --2- 172.21.15.45:0/3006159510 >> v2:172.21.15.190:3300/0 conn(0x7f65500ad7e0 0x7f65500ac7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f6554863640 1 -- 172.21.15.45:0/3006159510 >> v1:172.21.15.45:6789/0 conn(0x7f655006dbc0 legacy=0x7f655006dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.312+0000 7f6554863640 1 -- 172.21.15.45:0/3006159510 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65500705d0 con 0x7f65500b1d00 2023-11-28T20:47:32.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.314+0000 7f6554863640 1 --2- 172.21.15.45:0/3006159510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65500b1d00 0x7f65500710b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f655006bfd0 tx=0x7f654802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52bad169835429e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.314+0000 7f654effd640 1 -- 172.21.15.45:0/3006159510 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654802cbe0 con 0x7f65500b1d00 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.314+0000 7f654effd640 1 -- 172.21.15.45:0/3006159510 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654802cd40 con 0x7f65500b1d00 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.314+0000 7f654effd640 1 -- 172.21.15.45:0/3006159510 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6548034660 con 0x7f65500b1d00 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.315+0000 7f6555064640 1 -- 172.21.15.45:0/3006159510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65500b1d00 msgr2=0x7f65500710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.315+0000 7f6555064640 1 --2- 172.21.15.45:0/3006159510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65500b1d00 0x7f65500710b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f655006bfd0 tx=0x7f654802be30 comp rx=0 tx=0).stop 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.315+0000 7f6555064640 1 -- 172.21.15.45:0/3006159510 shutdown_connections 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.315+0000 7f6555064640 1 --2- 172.21.15.45:0/3006159510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65500b1d00 0x7f65500710b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.315+0000 7f6555064640 1 --2- 172.21.15.45:0/3006159510 >> v2:172.21.15.190:3300/0 conn(0x7f65500ad7e0 0x7f65500ac7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.315+0000 7f6555064640 1 -- 172.21.15.45:0/3006159510 >> 172.21.15.45:0/3006159510 conn(0x7f65500cecd0 msgr2=0x7f65500cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.315+0000 7f6555064640 1 -- 172.21.15.45:0/3006159510 shutdown_connections 2023-11-28T20:47:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.315+0000 7f6555064640 1 -- 172.21.15.45:0/3006159510 wait complete. 2023-11-28T20:47:32.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.317+0000 7f6555064640 1 Processor -- start 2023-11-28T20:47:32.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.317+0000 7f6555064640 1 -- start start 2023-11-28T20:47:32.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.318+0000 7f6555064640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f655006dbc0 0x7f655009a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.318+0000 7f6555064640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65500ad7e0 0x7f65500b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.318+0000 7f6555064640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65500b1d00 0x7f65500b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.318+0000 7f6555064640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655009a860 con 0x7f655006dbc0 2023-11-28T20:47:32.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.318+0000 7f6555064640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655009c8a0 con 0x7f65500b1d00 2023-11-28T20:47:32.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.318+0000 7f6555064640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65500cdb90 con 0x7f65500ad7e0 2023-11-28T20:47:32.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.318+0000 7f654ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f655006dbc0 0x7f655009a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.318+0000 7f654ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f655006dbc0 0x7f655009a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57402/0 (socket says 172.21.15.45:57402) 2023-11-28T20:47:32.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.318+0000 7f654ffff640 1 -- 172.21.15.45:0/3890284033 learned_addr learned my addr 172.21.15.45:0/3890284033 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:32.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654f7fe640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65500ad7e0 0x7f65500b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654ffff640 1 -- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65500ad7e0 msgr2=0x7f65500b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654ffff640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65500ad7e0 0x7f65500b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654ffff640 1 -- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65500b1d00 msgr2=0x7f65500b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654ffff640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65500b1d00 0x7f65500b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654ffff640 1 -- 172.21.15.45:0/3890284033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6548009a70 con 0x7f655006dbc0 2023-11-28T20:47:32.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654ffff640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f655006dbc0 0x7f655009a320 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f654000c500 tx=0x7f654000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:32.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654cff9640 1 -- 172.21.15.45:0/3890284033 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540005ce0 con 0x7f655006dbc0 2023-11-28T20:47:32.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654cff9640 1 -- 172.21.15.45:0/3890284033 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6540005e40 con 0x7f655006dbc0 2023-11-28T20:47:32.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f654cff9640 1 -- 172.21.15.45:0/3890284033 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540007330 con 0x7f655006dbc0 2023-11-28T20:47:32.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65500adbc0 con 0x7f655006dbc0 2023-11-28T20:47:32.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.319+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655013c030 con 0x7f655006dbc0 2023-11-28T20:47:32.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.321+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65500b4ff0 con 0x7f655006dbc0 2023-11-28T20:47:32.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.322+0000 7f654cff9640 1 -- 172.21.15.45:0/3890284033 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f6540007550 con 0x7f655006dbc0 2023-11-28T20:47:32.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.323+0000 7f654cff9640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f653c06b040 0x7f653c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.323+0000 7f654cff9640 1 -- 172.21.15.45:0/3890284033 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f6540089210 con 0x7f655006dbc0 2023-11-28T20:47:32.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.323+0000 7f654f7fe640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f653c06b040 0x7f653c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.324+0000 7f654f7fe640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f653c06b040 0x7f653c06d500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f655015d5e0 tx=0x7f6544008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.327+0000 7f654cff9640 1 -- 172.21.15.45:0/3890284033 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6540053f30 con 0x7f655006dbc0 2023-11-28T20:47:32.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.542+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f655009ce80 con 0x7f653c06b040 2023-11-28T20:47:32.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.561+0000 7f654cff9640 1 -- 172.21.15.45:0/3890284033 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f6544009520 con 0x7f653c06b040 2023-11-28T20:47:32.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:32.564 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:36:01", "first": "2023-11-28T20:37:01", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-11-28T20:47:32.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f653c06b040 msgr2=0x7f653c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f653c06b040 0x7f653c06d500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f655015d5e0 tx=0x7f6544008040 comp rx=0 tx=0).stop 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f655006dbc0 msgr2=0x7f655009a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f655006dbc0 0x7f655009a320 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f654000c500 tx=0x7f654000c9c0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 shutdown_connections 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f653c06b040 0x7f653c06d500 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65500b1d00 0x7f65500b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65500ad7e0 0x7f65500b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 --2- 172.21.15.45:0/3890284033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f655006dbc0 0x7f655009a320 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.564+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 >> 172.21.15.45:0/3890284033 conn(0x7f65500cecd0 msgr2=0x7f65500cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.565+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 shutdown_connections 2023-11-28T20:47:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.565+0000 7f6555064640 1 -- 172.21.15.45:0/3890284033 wait complete. 2023-11-28T20:47:32.586 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:36:01", "first": "2023-11-28T20:37:01", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-11-28T20:47:32.586 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:36:01', 'first': '2023-11-28T20:37:01', 'last': '2023-11-28T20:47:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:32.586 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-11-28T20:47:32.586 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-11-28T20:47:32.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:32.586 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-11-28T20:47:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:32.644 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:32.644 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:32.644 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:32.644 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:32.644 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:32.645 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_37_00_UTC', 'scheduled-2023-11-28-20_38_00_UTC', 'scheduled-2023-11-28-20_39_00_UTC', 'scheduled-2023-11-28-20_40_00_UTC', 'scheduled-2023-11-28-20_41_00_UTC', 'scheduled-2023-11-28-20_42_01_UTC', 'scheduled-2023-11-28-20_43_00_UTC', 'scheduled-2023-11-28-20_44_00_UTC', 'scheduled-2023-11-28-20_45_00_UTC', 'scheduled-2023-11-28-20_46_00_UTC', 'scheduled-2023-11-28-20_47_00_UTC'] 2023-11-28T20:47:32.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2023-11-28T20:47:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.871+0000 7f56c290f640 1 Processor -- start 2023-11-28T20:47:32.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.871+0000 7f56c290f640 1 -- start start 2023-11-28T20:47:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.873+0000 7f56c290f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 0x7f56bc0ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.873+0000 7f56c290f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f56bc1653a0 0x7f56bc0c2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.873+0000 7f56c290f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56bc0712c0 0x7f56bc162290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.873+0000 7f56c290f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56bc15f7e0 con 0x7f56bc0712c0 2023-11-28T20:47:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.873+0000 7f56c290f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56bc0c2650 con 0x7f56bc1653a0 2023-11-28T20:47:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.873+0000 7f56c290f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc1658a0 con 0x7f56bc0c1d30 2023-11-28T20:47:32.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.873+0000 7f56c0eac640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56bc0712c0 0x7f56bc162290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.873+0000 7f56c0eac640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56bc0712c0 0x7f56bc162290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57418/0 (socket says 172.21.15.45:57418) 2023-11-28T20:47:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.873+0000 7f56c0eac640 1 -- 172.21.15.45:0/3954205001 learned_addr learned my addr 172.21.15.45:0/3954205001 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.874+0000 7f56bbfff640 1 --2- 172.21.15.45:0/3954205001 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 0x7f56bc0ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.874+0000 7f56b3fff640 1 --2- 172.21.15.45:0/3954205001 >> v2:172.21.15.190:3300/0 conn(0x7f56bc1653a0 0x7f56bc0c2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.874+0000 7f56c0eac640 1 -- 172.21.15.45:0/3954205001 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 msgr2=0x7f56bc0ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.874+0000 7f56c0eac640 1 --2- 172.21.15.45:0/3954205001 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 0x7f56bc0ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.874+0000 7f56c0eac640 1 -- 172.21.15.45:0/3954205001 >> v2:172.21.15.190:3300/0 conn(0x7f56bc1653a0 msgr2=0x7f56bc0c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.874+0000 7f56c0eac640 1 --2- 172.21.15.45:0/3954205001 >> v2:172.21.15.190:3300/0 conn(0x7f56bc1653a0 0x7f56bc0c2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.874+0000 7f56c0eac640 1 -- 172.21.15.45:0/3954205001 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc165b80 con 0x7f56bc0712c0 2023-11-28T20:47:32.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.874+0000 7f56b3fff640 1 --2- 172.21.15.45:0/3954205001 >> v2:172.21.15.190:3300/0 conn(0x7f56bc1653a0 0x7f56bc0c2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:32.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.875+0000 7f56c0eac640 1 --2- 172.21.15.45:0/3954205001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 0x7f56bc162290 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009e50 tx=0x7f56b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf3b2e8bf01bf4f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56bb7fe640 1 -- 172.21.15.45:0/3954205001 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b402ca70 con 0x7f56bc0712c0 2023-11-28T20:47:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56bb7fe640 1 -- 172.21.15.45:0/3954205001 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b402cbd0 con 0x7f56bc0712c0 2023-11-28T20:47:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56bb7fe640 1 -- 172.21.15.45:0/3954205001 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b40343f0 con 0x7f56bc0712c0 2023-11-28T20:47:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56c290f640 1 -- 172.21.15.45:0/3954205001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 msgr2=0x7f56bc162290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56c290f640 1 --2- 172.21.15.45:0/3954205001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 0x7f56bc162290 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009e50 tx=0x7f56b402bed0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56c290f640 1 -- 172.21.15.45:0/3954205001 shutdown_connections 2023-11-28T20:47:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56c290f640 1 --2- 172.21.15.45:0/3954205001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 0x7f56bc162290 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56c290f640 1 --2- 172.21.15.45:0/3954205001 >> v2:172.21.15.190:3300/0 conn(0x7f56bc1653a0 0x7f56bc0c2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56c290f640 1 --2- 172.21.15.45:0/3954205001 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 0x7f56bc0ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56c290f640 1 -- 172.21.15.45:0/3954205001 >> 172.21.15.45:0/3954205001 conn(0x7f56bc0ced60 msgr2=0x7f56bc0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:32.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56c290f640 1 -- 172.21.15.45:0/3954205001 shutdown_connections 2023-11-28T20:47:32.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.876+0000 7f56c290f640 1 -- 172.21.15.45:0/3954205001 wait complete. 2023-11-28T20:47:32.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.878+0000 7f56c290f640 1 Processor -- start 2023-11-28T20:47:32.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.878+0000 7f56c290f640 1 -- start start 2023-11-28T20:47:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.879+0000 7f56c290f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 0x7f56bc0acb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.879+0000 7f56c290f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 0x7f56bc1fa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.879+0000 7f56c290f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc1653a0 0x7f56bc1faf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.879+0000 7f56c290f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc158cd0 con 0x7f56bc0712c0 2023-11-28T20:47:32.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.879+0000 7f56c290f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc158fb0 con 0x7f56bc1653a0 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.879+0000 7f56c290f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc159290 con 0x7f56bc0c1d30 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56bbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 0x7f56bc0acb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56bbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 0x7f56bc0acb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57422/0 (socket says 172.21.15.45:57422) 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56bbfff640 1 -- 172.21.15.45:0/1586030113 learned_addr learned my addr 172.21.15.45:0/1586030113 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56b3fff640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 0x7f56bc1fa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56c0eac640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc1653a0 0x7f56bc1faf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56bbfff640 1 -- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 msgr2=0x7f56bc1fa9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56bbfff640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 0x7f56bc1fa9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56bbfff640 1 -- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc1653a0 msgr2=0x7f56bc1faf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:32.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56bbfff640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc1653a0 0x7f56bc1faf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56bbfff640 1 -- 172.21.15.45:0/1586030113 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4009b30 con 0x7f56bc0712c0 2023-11-28T20:47:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56b3fff640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 0x7f56bc1fa9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.880+0000 7f56bbfff640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 0x7f56bc0acb70 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00d800 tx=0x7f56ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.881+0000 7f56b97fa640 1 -- 172.21.15.45:0/1586030113 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0096c0 con 0x7f56bc0712c0 2023-11-28T20:47:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.881+0000 7f56b97fa640 1 -- 172.21.15.45:0/1586030113 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac009840 con 0x7f56bc0712c0 2023-11-28T20:47:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.881+0000 7f56b97fa640 1 -- 172.21.15.45:0/1586030113 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac009bf0 con 0x7f56bc0712c0 2023-11-28T20:47:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.881+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56bc159570 con 0x7f56bc0712c0 2023-11-28T20:47:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.881+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56bc158930 con 0x7f56bc0712c0 2023-11-28T20:47:32.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.882+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56bc0733d0 con 0x7f56bc0712c0 2023-11-28T20:47:32.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.883+0000 7f56b97fa640 1 -- 172.21.15.45:0/1586030113 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f56ac009d50 con 0x7f56bc0712c0 2023-11-28T20:47:32.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.884+0000 7f56b97fa640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f569c06b040 0x7f569c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:32.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.884+0000 7f56b97fa640 1 -- 172.21.15.45:0/1586030113 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f56ac08a160 con 0x7f56bc0712c0 2023-11-28T20:47:32.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.885+0000 7f56b3fff640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f569c06b040 0x7f569c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:32.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.886+0000 7f56b3fff640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f569c06b040 0x7f569c06d500 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f56a4006fd0 tx=0x7f56a4008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:32.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:32.889+0000 7f56b97fa640 1 -- 172.21.15.45:0/1586030113 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56ac054c90 con 0x7f56bc0712c0 2023-11-28T20:47:33.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.102+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56bc0c21a0 con 0x7f569c06b040 2023-11-28T20:47:33.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.122+0000 7f56b97fa640 1 -- 172.21.15.45:0/1586030113 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f56bc0c21a0 con 0x7f569c06b040 2023-11-28T20:47:33.125 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:33.125 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:36:02", "first": "2023-11-28T20:37:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-11-28T20:47:33.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f569c06b040 msgr2=0x7f569c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f569c06b040 0x7f569c06d500 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f56a4006fd0 tx=0x7f56a4008040 comp rx=0 tx=0).stop 2023-11-28T20:47:33.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 msgr2=0x7f56bc0acb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 0x7f56bc0acb70 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00d800 tx=0x7f56ac00dcc0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 shutdown_connections 2023-11-28T20:47:33.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f569c06b040 0x7f569c06d500 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc1653a0 0x7f56bc1faf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0c1d30 0x7f56bc1fa9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 --2- 172.21.15.45:0/1586030113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0712c0 0x7f56bc0acb70 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 >> 172.21.15.45:0/1586030113 conn(0x7f56bc0ced60 msgr2=0x7f56bc0cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:33.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 shutdown_connections 2023-11-28T20:47:33.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.125+0000 7f56c290f640 1 -- 172.21.15.45:0/1586030113 wait complete. 2023-11-28T20:47:33.148 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:36:02", "first": "2023-11-28T20:37:00", "last": "2023-11-28T20:47:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-11-28T20:47:33.148 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:36:02', 'first': '2023-11-28T20:37:00', 'last': '2023-11-28T20:47:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:33.148 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-11-28T20:47:33.148 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-11-28T20:47:33.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:33.148 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-11-28T20:47:33.205 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:33.205 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_01_UTC 2023-11-28T20:47:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_01_UTC 2023-11-28T20:47:33.207 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_37_00_UTC', 'scheduled-2023-11-28-20_38_00_UTC', 'scheduled-2023-11-28-20_39_00_UTC', 'scheduled-2023-11-28-20_40_01_UTC', 'scheduled-2023-11-28-20_41_00_UTC', 'scheduled-2023-11-28-20_42_00_UTC', 'scheduled-2023-11-28-20_43_01_UTC', 'scheduled-2023-11-28-20_44_01_UTC', 'scheduled-2023-11-28-20_45_01_UTC', 'scheduled-2023-11-28-20_46_01_UTC', 'scheduled-2023-11-28-20_47_01_UTC'] 2023-11-28T20:47:33.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2023-11-28T20:47:33.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.434+0000 7f37325c9640 1 Processor -- start 2023-11-28T20:47:33.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.434+0000 7f37325c9640 1 -- start start 2023-11-28T20:47:33.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.435+0000 7f37325c9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f372c15e840 0x7f372c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.435+0000 7f37325c9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f372c06ac10 0x7f372c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.435+0000 7f37325c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f372c15ecd0 con 0x7f372c15f720 2023-11-28T20:47:33.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.435+0000 7f37325c9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372c071ce0 con 0x7f372c06ac10 2023-11-28T20:47:33.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.435+0000 7f37325c9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372c15f290 con 0x7f372c15e840 2023-11-28T20:47:33.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.435+0000 7f3730b66640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f372c06ac10 0x7f372c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.435+0000 7f3730b66640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f372c06ac10 0x7f372c0717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57424/0 (socket says 172.21.15.45:57424) 2023-11-28T20:47:33.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.435+0000 7f3730b66640 1 -- 172.21.15.45:0/2817053941 learned_addr learned my addr 172.21.15.45:0/2817053941 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.436+0000 7f3730b66640 1 -- 172.21.15.45:0/2817053941 >> v2:172.21.15.190:3300/0 conn(0x7f372c15e840 msgr2=0x7f372c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.436+0000 7f372bfff640 1 --2- 172.21.15.45:0/2817053941 >> v2:172.21.15.190:3300/0 conn(0x7f372c15e840 0x7f372c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.436+0000 7f3730b66640 1 --2- 172.21.15.45:0/2817053941 >> v2:172.21.15.190:3300/0 conn(0x7f372c15e840 0x7f372c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.436+0000 7f3730b66640 1 -- 172.21.15.45:0/2817053941 >> v1:172.21.15.45:6789/0 conn(0x7f372c15f720 legacy=0x7f372c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.436+0000 7f3730b66640 1 -- 172.21.15.45:0/2817053941 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c15fb60 con 0x7f372c06ac10 2023-11-28T20:47:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.436+0000 7f372b7fe640 1 -- 172.21.15.45:0/2817053941 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377929656 0 0) 0x7f372c15ecd0 con 0x7f372c15f720 2023-11-28T20:47:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.436+0000 7f372bfff640 1 --2- 172.21.15.45:0/2817053941 >> v2:172.21.15.190:3300/0 conn(0x7f372c15e840 0x7f372c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.437+0000 7f3730b66640 1 --2- 172.21.15.45:0/2817053941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c06ac10 0x7f372c0717a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f372c0cb5c0 tx=0x7f372402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff460796a05d7e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:33.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.437+0000 7f372b7fe640 1 -- 172.21.15.45:0/2817053941 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372402cb20 con 0x7f372c06ac10 2023-11-28T20:47:33.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.437+0000 7f372b7fe640 1 -- 172.21.15.45:0/2817053941 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372402cc80 con 0x7f372c06ac10 2023-11-28T20:47:33.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.437+0000 7f372b7fe640 1 -- 172.21.15.45:0/2817053941 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37240343f0 con 0x7f372c06ac10 2023-11-28T20:47:33.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.438+0000 7f37325c9640 1 -- 172.21.15.45:0/2817053941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c06ac10 msgr2=0x7f372c0717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.438+0000 7f37325c9640 1 --2- 172.21.15.45:0/2817053941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c06ac10 0x7f372c0717a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f372c0cb5c0 tx=0x7f372402bf80 comp rx=0 tx=0).stop 2023-11-28T20:47:33.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.438+0000 7f37325c9640 1 -- 172.21.15.45:0/2817053941 shutdown_connections 2023-11-28T20:47:33.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.438+0000 7f37325c9640 1 --2- 172.21.15.45:0/2817053941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c06ac10 0x7f372c0717a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.438+0000 7f37325c9640 1 --2- 172.21.15.45:0/2817053941 >> v2:172.21.15.190:3300/0 conn(0x7f372c15e840 0x7f372c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.438+0000 7f37325c9640 1 -- 172.21.15.45:0/2817053941 >> 172.21.15.45:0/2817053941 conn(0x7f372c0ced60 msgr2=0x7f372c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:33.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.438+0000 7f37325c9640 1 -- 172.21.15.45:0/2817053941 shutdown_connections 2023-11-28T20:47:33.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.438+0000 7f37325c9640 1 -- 172.21.15.45:0/2817053941 wait complete. 2023-11-28T20:47:33.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.440+0000 7f37325c9640 1 Processor -- start 2023-11-28T20:47:33.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.440+0000 7f37325c9640 1 -- start start 2023-11-28T20:47:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.441+0000 7f37325c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f372c06ac10 0x7f372c0b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.441+0000 7f37325c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c15e840 0x7f372c1489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.441+0000 7f37325c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f372c15f720 0x7f372c148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.441+0000 7f37325c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0b35d0 con 0x7f372c15e840 2023-11-28T20:47:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.441+0000 7f37325c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0c2930 con 0x7f372c15f720 2023-11-28T20:47:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.441+0000 7f37325c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0c2c10 con 0x7f372c06ac10 2023-11-28T20:47:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.441+0000 7f37237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c15e840 0x7f372c1489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.442+0000 7f37237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c15e840 0x7f372c1489e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57432/0 (socket says 172.21.15.45:57432) 2023-11-28T20:47:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.442+0000 7f37237fe640 1 -- 172.21.15.45:0/3094214430 learned_addr learned my addr 172.21.15.45:0/3094214430 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.442+0000 7f37237fe640 1 -- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f372c06ac10 msgr2=0x7f372c0b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.442+0000 7f37237fe640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f372c06ac10 0x7f372c0b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.442+0000 7f37237fe640 1 -- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f372c15f720 msgr2=0x7f372c148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.442+0000 7f3730b66640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f372c15f720 0x7f372c148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.442+0000 7f37237fe640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f372c15f720 0x7f372c148f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.442+0000 7f37237fe640 1 -- 172.21.15.45:0/3094214430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724009b30 con 0x7f372c15e840 2023-11-28T20:47:33.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.442+0000 7f37237fe640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c15e840 0x7f372c1489e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f371400c410 tx=0x7f371400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:33.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.443+0000 7f37297fa640 1 -- 172.21.15.45:0/3094214430 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37140035f0 con 0x7f372c15e840 2023-11-28T20:47:33.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.443+0000 7f37297fa640 1 -- 172.21.15.45:0/3094214430 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3714003770 con 0x7f372c15e840 2023-11-28T20:47:33.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.443+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372c0b3870 con 0x7f372c15e840 2023-11-28T20:47:33.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.443+0000 7f37297fa640 1 -- 172.21.15.45:0/3094214430 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714003b20 con 0x7f372c15e840 2023-11-28T20:47:33.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.443+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c0b42d0 con 0x7f372c15e840 2023-11-28T20:47:33.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.444+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372c0b44f0 con 0x7f372c15e840 2023-11-28T20:47:33.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.445+0000 7f37297fa640 1 -- 172.21.15.45:0/3094214430 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f3714003c80 con 0x7f372c15e840 2023-11-28T20:47:33.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.446+0000 7f37297fa640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f370c06b070 0x7f370c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.446+0000 7f37297fa640 1 -- 172.21.15.45:0/3094214430 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f37140892f0 con 0x7f372c15e840 2023-11-28T20:47:33.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.446+0000 7f372bfff640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f370c06b070 0x7f370c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.448+0000 7f372bfff640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f370c06b070 0x7f370c06d530 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f371c006fd0 tx=0x7f371c008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:33.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.450+0000 7f37297fa640 1 -- 172.21.15.45:0/3094214430 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3714053e20 con 0x7f372c15e840 2023-11-28T20:47:33.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.665+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f372c15ef20 con 0x7f370c06b070 2023-11-28T20:47:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.684+0000 7f37297fa640 1 -- 172.21.15.45:0/3094214430 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f371c00ae40 con 0x7f370c06b070 2023-11-28T20:47:33.686 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:33.686 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:36:02", "first": "2023-11-28T20:37:00", "last": "2023-11-28T20:47:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-11-28T20:47:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f370c06b070 msgr2=0x7f370c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f370c06b070 0x7f370c06d530 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f371c006fd0 tx=0x7f371c008040 comp rx=0 tx=0).stop 2023-11-28T20:47:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c15e840 msgr2=0x7f372c1489e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c15e840 0x7f372c1489e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f371400c410 tx=0x7f371400c8d0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 shutdown_connections 2023-11-28T20:47:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f370c06b070 0x7f370c06d530 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f372c15f720 0x7f372c148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c15e840 0x7f372c1489e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 --2- 172.21.15.45:0/3094214430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f372c06ac10 0x7f372c0b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.686+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 >> 172.21.15.45:0/3094214430 conn(0x7f372c0ced60 msgr2=0x7f372c070950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.687+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 shutdown_connections 2023-11-28T20:47:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.687+0000 7f37325c9640 1 -- 172.21.15.45:0/3094214430 wait complete. 2023-11-28T20:47:33.710 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:36:02", "first": "2023-11-28T20:37:00", "last": "2023-11-28T20:47:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-11-28T20:47:33.710 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:36:02', 'first': '2023-11-28T20:37:00', 'last': '2023-11-28T20:47:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-11-28T20:47:33.710 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-11-28T20:47:33.711 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-11-28T20:47:33.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2023-11-28T20:47:33.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.933+0000 7fa4d9cda640 1 Processor -- start 2023-11-28T20:47:33.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.933+0000 7fa4d9cda640 1 -- start start 2023-11-28T20:47:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.934+0000 7fa4d9cda640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa4d40cdfb0 0x7fa4d406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.934+0000 7fa4d9cda640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4d406c480 0x7fa4d40b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.934+0000 7fa4d9cda640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d40cdb20 con 0x7fa4d4155770 2023-11-28T20:47:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.934+0000 7fa4d9cda640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d406c8b0 con 0x7fa4d40cdfb0 2023-11-28T20:47:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.934+0000 7fa4d9cda640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d40ccf10 con 0x7fa4d406c480 2023-11-28T20:47:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.935+0000 7fa4cbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa4d40cdfb0 0x7fa4d406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.935+0000 7fa4cbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa4d40cdfb0 0x7fa4d406bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56258/0 (socket says 172.21.15.45:56258) 2023-11-28T20:47:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.935+0000 7fa4cbfff640 1 -- 172.21.15.45:0/3902369018 learned_addr learned my addr 172.21.15.45:0/3902369018 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.935+0000 7fa4cbfff640 1 -- 172.21.15.45:0/3902369018 >> v2:172.21.15.190:3300/0 conn(0x7fa4d406c480 msgr2=0x7fa4d40b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.935+0000 7fa4d94d9640 1 --2- 172.21.15.45:0/3902369018 >> v2:172.21.15.190:3300/0 conn(0x7fa4d406c480 0x7fa4d40b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.935+0000 7fa4cbfff640 1 --2- 172.21.15.45:0/3902369018 >> v2:172.21.15.190:3300/0 conn(0x7fa4d406c480 0x7fa4d40b20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.935+0000 7fa4cbfff640 1 -- 172.21.15.45:0/3902369018 >> v1:172.21.15.190:6789/0 conn(0x7fa4d4155770 legacy=0x7fa4d415c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.936+0000 7fa4cbfff640 1 -- 172.21.15.45:0/3902369018 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d40cd1f0 con 0x7fa4d40cdfb0 2023-11-28T20:47:33.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.936+0000 7fa4d94d9640 1 --2- 172.21.15.45:0/3902369018 >> v2:172.21.15.190:3300/0 conn(0x7fa4d406c480 0x7fa4d40b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:33.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.937+0000 7fa4cbfff640 1 --2- 172.21.15.45:0/3902369018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d40cdfb0 0x7fa4d406bf40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40b1ac0 tx=0x7fa4c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854fbf0757d0214f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:33.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.937+0000 7fa4cb7fe640 1 -- 172.21.15.45:0/3902369018 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c40317e0 con 0x7fa4d40cdfb0 2023-11-28T20:47:33.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.937+0000 7fa4cb7fe640 1 -- 172.21.15.45:0/3902369018 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c4031940 con 0x7fa4d40cdfb0 2023-11-28T20:47:33.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.937+0000 7fa4cb7fe640 1 -- 172.21.15.45:0/3902369018 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4031cd0 con 0x7fa4d40cdfb0 2023-11-28T20:47:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.938+0000 7fa4d9cda640 1 -- 172.21.15.45:0/3902369018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d40cdfb0 msgr2=0x7fa4d406bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.938+0000 7fa4d9cda640 1 --2- 172.21.15.45:0/3902369018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d40cdfb0 0x7fa4d406bf40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40b1ac0 tx=0x7fa4c402b880 comp rx=0 tx=0).stop 2023-11-28T20:47:33.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.938+0000 7fa4d9cda640 1 -- 172.21.15.45:0/3902369018 shutdown_connections 2023-11-28T20:47:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.938+0000 7fa4d9cda640 1 --2- 172.21.15.45:0/3902369018 >> v2:172.21.15.190:3300/0 conn(0x7fa4d406c480 0x7fa4d40b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.938+0000 7fa4d9cda640 1 --2- 172.21.15.45:0/3902369018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d40cdfb0 0x7fa4d406bf40 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.938+0000 7fa4d9cda640 1 -- 172.21.15.45:0/3902369018 >> 172.21.15.45:0/3902369018 conn(0x7fa4d40b1050 msgr2=0x7fa4d406a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.938+0000 7fa4d9cda640 1 -- 172.21.15.45:0/3902369018 shutdown_connections 2023-11-28T20:47:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.938+0000 7fa4d9cda640 1 -- 172.21.15.45:0/3902369018 wait complete. 2023-11-28T20:47:33.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.940+0000 7fa4d9cda640 1 Processor -- start 2023-11-28T20:47:33.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.940+0000 7fa4d9cda640 1 -- start start 2023-11-28T20:47:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.941+0000 7fa4d9cda640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4d406c480 0x7fa4d414b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.941+0000 7fa4d9cda640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d40cdfb0 0x7fa4d4070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.941+0000 7fa4d9cda640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d4155770 0x7fa4d40acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.941+0000 7fa4d9cda640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d414bb50 con 0x7fa4d4155770 2023-11-28T20:47:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.941+0000 7fa4d9cda640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d409bec0 con 0x7fa4d40cdfb0 2023-11-28T20:47:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.941+0000 7fa4d9cda640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d4071a00 con 0x7fa4d406c480 2023-11-28T20:47:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.941+0000 7fa4d94d9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d4155770 0x7fa4d40acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.941+0000 7fa4d94d9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d4155770 0x7fa4d40acfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56272/0 (socket says 172.21.15.45:56272) 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.941+0000 7fa4d94d9640 1 -- 172.21.15.45:0/2415839321 learned_addr learned my addr 172.21.15.45:0/2415839321 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4cbfff640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d40cdfb0 0x7fa4d4070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4d8cd8640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4d406c480 0x7fa4d414b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4d94d9640 1 -- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4d406c480 msgr2=0x7fa4d414b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4d94d9640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4d406c480 0x7fa4d414b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4d94d9640 1 -- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d40cdfb0 msgr2=0x7fa4d4070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4d94d9640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d40cdfb0 0x7fa4d4070bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4d94d9640 1 -- 172.21.15.45:0/2415839321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c40094d0 con 0x7fa4d4155770 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4cbfff640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d40cdfb0 0x7fa4d4070bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4d94d9640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d4155770 0x7fa4d40acfd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc00c7f0 tx=0x7fa4cc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.942+0000 7fa4c97fa640 1 -- 172.21.15.45:0/2415839321 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc007820 con 0x7fa4d4155770 2023-11-28T20:47:33.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.943+0000 7fa4c97fa640 1 -- 172.21.15.45:0/2415839321 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4cc007980 con 0x7fa4d4155770 2023-11-28T20:47:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.943+0000 7fa4c97fa640 1 -- 172.21.15.45:0/2415839321 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc007d70 con 0x7fa4d4155770 2023-11-28T20:47:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.943+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d409ae20 con 0x7fa4d4155770 2023-11-28T20:47:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.943+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d409b3e0 con 0x7fa4d4155770 2023-11-28T20:47:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.944+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d409d0f0 con 0x7fa4d4155770 2023-11-28T20:47:33.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.946+0000 7fa4c97fa640 1 -- 172.21.15.45:0/2415839321 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7fa4cc0113f0 con 0x7fa4d4155770 2023-11-28T20:47:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.946+0000 7fa4c97fa640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa4c006b040 0x7fa4c006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:33.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.946+0000 7fa4c97fa640 1 -- 172.21.15.45:0/2415839321 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fa4cc08a170 con 0x7fa4d4155770 2023-11-28T20:47:33.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.947+0000 7fa4d8cd8640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa4c006b040 0x7fa4c006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:33.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.948+0000 7fa4d8cd8640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa4c006b040 0x7fa4c006d500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0007dc0 tx=0x7fa4d00072f0 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:33.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:33.950+0000 7fa4c97fa640 1 -- 172.21.15.45:0/2415839321 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4cc054b00 con 0x7fa4d4155770 2023-11-28T20:47:34.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.164+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4d406e200 con 0x7fa4c006b040 2023-11-28T20:47:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.367+0000 7fa4c97fa640 1 -- 172.21.15.45:0/2415839321 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fa4d00079b0 con 0x7fa4c006b040 2023-11-28T20:47:34.369 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-11-28T20:47:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.370+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa4c006b040 msgr2=0x7fa4c006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa4c006b040 0x7fa4c006d500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0007dc0 tx=0x7fa4d00072f0 comp rx=0 tx=0).stop 2023-11-28T20:47:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d4155770 msgr2=0x7fa4d40acfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d4155770 0x7fa4d40acfd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc00c7f0 tx=0x7fa4cc00ccb0 comp rx=0 tx=0).stop 2023-11-28T20:47:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 shutdown_connections 2023-11-28T20:47:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa4c006b040 0x7fa4c006d500 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4d4155770 0x7fa4d40acfd0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d40cdfb0 0x7fa4d4070bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 --2- 172.21.15.45:0/2415839321 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4d406c480 0x7fa4d414b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 >> 172.21.15.45:0/2415839321 conn(0x7fa4d40b1050 msgr2=0x7fa4d41591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 shutdown_connections 2023-11-28T20:47:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:34.371+0000 7fa4d9cda640 1 -- 172.21.15.45:0/2415839321 wait complete. 2023-11-28T20:47:34.393 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-11-28T20:47:34.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:34.393 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-11-28T20:47:34.455 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:34.456 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:34.456 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:34.456 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:34.457 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:34.457 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:34.457 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:34.458 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:34.458 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:34.458 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:34.459 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:34.459 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:34.460 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:34.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:34.461 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_36_00_UTC) 2023-11-28T20:47:34.538 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:34.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:34.538 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_37_00_UTC) 2023-11-28T20:47:34.610 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:34.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:34.610 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_38_00_UTC) 2023-11-28T20:47:34.689 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:34.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:34.690 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_39_00_UTC) 2023-11-28T20:47:34.761 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:34.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:34.762 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_40_00_UTC) 2023-11-28T20:47:34.841 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:34.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:34.841 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_41_00_UTC) 2023-11-28T20:47:34.923 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:34.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:34.924 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_42_00_UTC) 2023-11-28T20:47:34.996 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:34.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:34.997 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_43_00_UTC) 2023-11-28T20:47:35.071 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:35.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:35.072 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_44_00_UTC) 2023-11-28T20:47:35.147 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:35.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:35.147 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_45_00_UTC) 2023-11-28T20:47:35.222 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:35.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:35.222 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_46_01_UTC) 2023-11-28T20:47:35.302 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:35.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:35.302 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-11-28-20_47_00_UTC) 2023-11-28T20:47:35.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:35.377 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-11-28T20:47:35.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2023-11-28T20:47:35.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.672+0000 7f9f2b1de640 1 Processor -- start 2023-11-28T20:47:35.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.672+0000 7f9f2b1de640 1 -- start start 2023-11-28T20:47:35.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.673+0000 7f9f2b1de640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f2415b3c0 0x7f9f240cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:35.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.673+0000 7f9f2b1de640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f240b36c0 0x7f9f240cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:35.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.673+0000 7f9f2b1de640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9f240cf880 0x7f9f24099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.673+0000 7f9f2b1de640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f240b08c0 con 0x7f9f240b36c0 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.673+0000 7f9f2b1de640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f240cfcd0 con 0x7f9f240cf880 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.673+0000 7f9f2b1de640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240b0570 con 0x7f9f2415b3c0 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f23fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f240b36c0 0x7f9f240cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f23fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f240b36c0 0x7f9f240cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56284/0 (socket says 172.21.15.45:56284) 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f23fff640 1 -- 172.21.15.45:0/1275562024 learned_addr learned my addr 172.21.15.45:0/1275562024 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f28f7a640 1 --2- 172.21.15.45:0/1275562024 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f2415b3c0 0x7f9f240cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f2977b640 1 --2- 172.21.15.45:0/1275562024 >> v2:172.21.15.190:3300/0 conn(0x7f9f240cf880 0x7f9f24099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f23fff640 1 -- 172.21.15.45:0/1275562024 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f2415b3c0 msgr2=0x7f9f240cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f23fff640 1 --2- 172.21.15.45:0/1275562024 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f2415b3c0 0x7f9f240cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:35.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f23fff640 1 -- 172.21.15.45:0/1275562024 >> v2:172.21.15.190:3300/0 conn(0x7f9f240cf880 msgr2=0x7f9f24099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:35.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f23fff640 1 --2- 172.21.15.45:0/1275562024 >> v2:172.21.15.190:3300/0 conn(0x7f9f240cf880 0x7f9f24099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:35.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.674+0000 7f9f23fff640 1 -- 172.21.15.45:0/1275562024 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f240cc800 con 0x7f9f240b36c0 2023-11-28T20:47:35.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.676+0000 7f9f23fff640 1 --2- 172.21.15.45:0/1275562024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f240b36c0 0x7f9f240cf340 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9f2415ef40 tx=0x7f9f0c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d461c19b6a57b23 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:35.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.676+0000 7f9f237fe640 1 -- 172.21.15.45:0/1275562024 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c02c950 con 0x7f9f240b36c0 2023-11-28T20:47:35.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.676+0000 7f9f237fe640 1 -- 172.21.15.45:0/1275562024 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c02cab0 con 0x7f9f240b36c0 2023-11-28T20:47:35.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.676+0000 7f9f237fe640 1 -- 172.21.15.45:0/1275562024 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c02ce40 con 0x7f9f240b36c0 2023-11-28T20:47:35.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.677+0000 7f9f2b1de640 1 -- 172.21.15.45:0/1275562024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f240b36c0 msgr2=0x7f9f240cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:35.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.677+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/1275562024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f240b36c0 0x7f9f240cf340 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9f2415ef40 tx=0x7f9f0c02bdb0 comp rx=0 tx=0).stop 2023-11-28T20:47:35.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.677+0000 7f9f2b1de640 1 -- 172.21.15.45:0/1275562024 shutdown_connections 2023-11-28T20:47:35.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.677+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/1275562024 >> v2:172.21.15.190:3300/0 conn(0x7f9f240cf880 0x7f9f24099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:35.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.677+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/1275562024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f240b36c0 0x7f9f240cf340 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:35.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.677+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/1275562024 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f2415b3c0 0x7f9f240cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:35.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.677+0000 7f9f2b1de640 1 -- 172.21.15.45:0/1275562024 >> 172.21.15.45:0/1275562024 conn(0x7f9f240b2ba0 msgr2=0x7f9f240b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:35.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.677+0000 7f9f2b1de640 1 -- 172.21.15.45:0/1275562024 shutdown_connections 2023-11-28T20:47:35.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.677+0000 7f9f2b1de640 1 -- 172.21.15.45:0/1275562024 wait complete. 2023-11-28T20:47:35.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.679+0000 7f9f2b1de640 1 Processor -- start 2023-11-28T20:47:35.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.679+0000 7f9f2b1de640 1 -- start start 2023-11-28T20:47:35.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.680+0000 7f9f2b1de640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f240b36c0 0x7f9f2409ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:35.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.680+0000 7f9f2b1de640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f240cf880 0x7f9f240ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:35.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.680+0000 7f9f2b1de640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f2415b3c0 0x7f9f240caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:35.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.680+0000 7f9f2b1de640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2414f090 con 0x7f9f2415b3c0 2023-11-28T20:47:35.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.680+0000 7f9f2b1de640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2414f320 con 0x7f9f240cf880 2023-11-28T20:47:35.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.680+0000 7f9f2b1de640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240b6180 con 0x7f9f240b36c0 2023-11-28T20:47:35.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.680+0000 7f9f2977b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f2415b3c0 0x7f9f240caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:35.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.680+0000 7f9f2977b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f2415b3c0 0x7f9f240caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56288/0 (socket says 172.21.15.45:56288) 2023-11-28T20:47:35.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.680+0000 7f9f2977b640 1 -- 172.21.15.45:0/3929483888 learned_addr learned my addr 172.21.15.45:0/3929483888 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:35.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f23fff640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f240cf880 0x7f9f240ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:35.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f28f7a640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f240b36c0 0x7f9f2409ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:35.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f2977b640 1 -- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f240b36c0 msgr2=0x7f9f2409ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:35.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f2977b640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f240b36c0 0x7f9f2409ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:35.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f2977b640 1 -- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f240cf880 msgr2=0x7f9f240ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:35.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f2977b640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f240cf880 0x7f9f240ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:35.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f2977b640 1 -- 172.21.15.45:0/3929483888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c009b30 con 0x7f9f2415b3c0 2023-11-28T20:47:35.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f2977b640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f2415b3c0 0x7f9f240caef0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00d5d0 tx=0x7f9f1c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:35.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f217fa640 1 -- 172.21.15.45:0/3929483888 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c014620 con 0x7f9f2415b3c0 2023-11-28T20:47:35.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f217fa640 1 -- 172.21.15.45:0/3929483888 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1c0147a0 con 0x7f9f2415b3c0 2023-11-28T20:47:35.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f240b6460 con 0x7f9f2415b3c0 2023-11-28T20:47:35.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.681+0000 7f9f217fa640 1 -- 172.21.15.45:0/3929483888 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c014b50 con 0x7f9f2415b3c0 2023-11-28T20:47:35.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.682+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2414f8a0 con 0x7f9f2415b3c0 2023-11-28T20:47:35.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.683+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f240d08f0 con 0x7f9f2415b3c0 2023-11-28T20:47:35.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.684+0000 7f9f217fa640 1 -- 172.21.15.45:0/3929483888 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f9f1c014cb0 con 0x7f9f2415b3c0 2023-11-28T20:47:35.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.685+0000 7f9f217fa640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9f0406b040 0x7f9f0406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:35.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.685+0000 7f9f217fa640 1 -- 172.21.15.45:0/3929483888 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f9f1c089f80 con 0x7f9f2415b3c0 2023-11-28T20:47:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.685+0000 7f9f28f7a640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9f0406b040 0x7f9f0406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.687+0000 7f9f28f7a640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9f0406b040 0x7f9f0406d500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9f14006fd0 tx=0x7f9f14008040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.690+0000 7f9f217fa640 1 -- 172.21.15.45:0/3929483888 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f1c0bce90 con 0x7f9f2415b3c0 2023-11-28T20:47:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:35.905+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f240ccdc0 con 0x7f9f0406b040 2023-11-28T20:47:36.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.102+0000 7f9f217fa640 1 -- 172.21.15.45:0/3929483888 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f9f14009480 con 0x7f9f0406b040 2023-11-28T20:47:36.104 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-11-28T20:47:36.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9f0406b040 msgr2=0x7f9f0406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:36.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9f0406b040 0x7f9f0406d500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9f14006fd0 tx=0x7f9f14008040 comp rx=0 tx=0).stop 2023-11-28T20:47:36.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f2415b3c0 msgr2=0x7f9f240caef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:36.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f2415b3c0 0x7f9f240caef0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00d5d0 tx=0x7f9f1c00da90 comp rx=0 tx=0).stop 2023-11-28T20:47:36.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 shutdown_connections 2023-11-28T20:47:36.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f9f0406b040 0x7f9f0406d500 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:36.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f2415b3c0 0x7f9f240caef0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:36.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f240cf880 0x7f9f240ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:36.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 --2- 172.21.15.45:0/3929483888 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f240b36c0 0x7f9f2409ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:36.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 >> 172.21.15.45:0/3929483888 conn(0x7f9f240b2ba0 msgr2=0x7f9f240b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:36.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 shutdown_connections 2023-11-28T20:47:36.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:36.106+0000 7f9f2b1de640 1 -- 172.21.15.45:0/3929483888 wait complete. 2023-11-28T20:47:36.132 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-11-28T20:47:36.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.132 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-11-28T20:47:36.195 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:36.195 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:36.195 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:36.195 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:36.195 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:36.196 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:36.196 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:36.196 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:36.196 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:36.196 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_01_UTC 2023-11-28T20:47:36.196 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:36.196 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:36.197 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:36.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.197 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_36_00_UTC) 2023-11-28T20:47:36.277 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:36.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.277 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_37_00_UTC) 2023-11-28T20:47:36.353 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:36.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.353 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_38_00_UTC) 2023-11-28T20:47:36.426 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:36.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.426 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_39_00_UTC) 2023-11-28T20:47:36.495 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:36.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.496 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_40_00_UTC) 2023-11-28T20:47:36.567 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:36.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.567 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_41_00_UTC) 2023-11-28T20:47:36.644 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:36.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.645 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_42_00_UTC) 2023-11-28T20:47:36.719 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:36.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.719 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_43_00_UTC) 2023-11-28T20:47:36.791 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:36.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.791 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_44_01_UTC) 2023-11-28T20:47:36.865 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_45_01_UTC 2023-11-28T20:47:36.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.866 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_45_01_UTC) 2023-11-28T20:47:36.939 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:36.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:36.939 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_46_01_UTC) 2023-11-28T20:47:37.015 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:37.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:37.015 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-11-28-20_47_00_UTC) 2023-11-28T20:47:37.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:37.087 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-11-28T20:47:37.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2023-11-28T20:47:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.384+0000 7f2b9632c640 1 Processor -- start 2023-11-28T20:47:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.385+0000 7f2b9632c640 1 -- start start 2023-11-28T20:47:37.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.386+0000 7f2b9632c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b9006c350 0x7f2b9015f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.387+0000 7f2b9632c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9015ecd0 con 0x7f2b9015e840 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.387+0000 7f2b9632c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9015f290 con 0x7f2b9006ac10 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.387+0000 7f2b9632c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90072020 con 0x7f2b9006c350 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.387+0000 7f2b8ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2b9015e840 0x7f2b9006a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44370/0 (socket says 172.21.15.45:44370) 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.387+0000 7f2b8ffff640 1 -- 172.21.15.45:0/1171241401 learned_addr learned my addr 172.21.15.45:0/1171241401 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.387+0000 7f2b948c9640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b9006c350 0x7f2b9015f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.387+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99783116 0 0) 0x7f2b9015ecd0 con 0x7f2b9015e840 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b7c003520 con 0x7f2b9015e840 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 453870998 0 0) 0x7f2b7c003520 con 0x7f2b9015e840 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b9006c350 msgr2=0x7f2b9015f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:37.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b9006c350 0x7f2b9015f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 >> v1:172.21.15.190:6789/0 conn(0x7f2b9006ac10 legacy=0x7f2b900cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b90072300 con 0x7f2b9015e840 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f2b800019f0 con 0x7f2b9015e840 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 >> v1:172.21.15.45:6789/0 conn(0x7f2b9015e840 legacy=0x7f2b9006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b9006c350 0x7f2b7c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b7c003ee0 0x7f2b7c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b7c004d00 0x7f2b7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90072020 con 0x7f2b7c004d00 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c00ba30 con 0x7f2b7c003ee0 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c00bbf0 con 0x7f2b9006c350 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.388+0000 7f2b948c9640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b7c004d00 0x7f2b7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.389+0000 7f2b8ffff640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b9006c350 0x7f2b7c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.389+0000 7f2b877fe640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b7c003ee0 0x7f2b7c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.389+0000 7f2b948c9640 1 -- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b9006c350 msgr2=0x7f2b7c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.389+0000 7f2b948c9640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b9006c350 0x7f2b7c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.389+0000 7f2b948c9640 1 -- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b7c003ee0 msgr2=0x7f2b7c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.389+0000 7f2b948c9640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b7c003ee0 0x7f2b7c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.389+0000 7f2b948c9640 1 -- 172.21.15.45:0/1171241401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c003520 con 0x7f2b7c004d00 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.389+0000 7f2b877fe640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b7c003ee0 0x7f2b7c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.390+0000 7f2b948c9640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b7c004d00 0x7f2b7c005180 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009ac0 tx=0x7f2b7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6a7498067238f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.390+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8802c6b0 con 0x7f2b7c004d00 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.390+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8802c810 con 0x7f2b7c004d00 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.390+0000 7f2b8f7fe640 1 -- 172.21.15.45:0/1171241401 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8802cba0 con 0x7f2b7c004d00 2023-11-28T20:47:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.391+0000 7f2b9632c640 1 -- 172.21.15.45:0/1171241401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b7c004d00 msgr2=0x7f2b7c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.391+0000 7f2b9632c640 1 --2- 172.21.15.45:0/1171241401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b7c004d00 0x7f2b7c005180 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009ac0 tx=0x7f2b7c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.391+0000 7f2b948c9640 1 -- 172.21.15.45:0/1171241401 reap_dead start 2023-11-28T20:47:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.391+0000 7f2b9632c640 1 -- 172.21.15.45:0/1171241401 shutdown_connections 2023-11-28T20:47:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.391+0000 7f2b9632c640 1 -- 172.21.15.45:0/1171241401 >> 172.21.15.45:0/1171241401 conn(0x7f2b900ced60 msgr2=0x7f2b900cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.391+0000 7f2b9632c640 1 -- 172.21.15.45:0/1171241401 shutdown_connections 2023-11-28T20:47:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.391+0000 7f2b9632c640 1 -- 172.21.15.45:0/1171241401 wait complete. 2023-11-28T20:47:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.393+0000 7f2b9632c640 1 Processor -- start 2023-11-28T20:47:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.393+0000 7f2b9632c640 1 -- start start 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.394+0000 7f2b9632c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b7c004d00 0x7f2b900c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.394+0000 7f2b9632c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b900b27a0 0x7f2b901f2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.394+0000 7f2b9632c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b900c79c0 0x7f2b900af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.394+0000 7f2b9632c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b901591b0 con 0x7f2b900b27a0 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.394+0000 7f2b9632c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90159490 con 0x7f2b900c79c0 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.394+0000 7f2b9632c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b900c98f0 con 0x7f2b7c004d00 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.394+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b900b27a0 0x7f2b901f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.394+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b900b27a0 0x7f2b901f2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56310/0 (socket says 172.21.15.45:56310) 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.394+0000 7f2b877fe640 1 -- 172.21.15.45:0/1320088041 learned_addr learned my addr 172.21.15.45:0/1320088041 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b877fe640 1 -- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b7c004d00 msgr2=0x7f2b900c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b8ffff640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b7c004d00 0x7f2b900c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b877fe640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b7c004d00 0x7f2b900c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b948c9640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b900c79c0 0x7f2b900af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b877fe640 1 -- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b900c79c0 msgr2=0x7f2b900af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b877fe640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b900c79c0 0x7f2b900af8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b877fe640 1 -- 172.21.15.45:0/1320088041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b901f29d0 con 0x7f2b900b27a0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b877fe640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b900b27a0 0x7f2b901f2490 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800d710 tx=0x7f2b7800dbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b8d7fa640 1 -- 172.21.15.45:0/1320088041 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78005480 con 0x7f2b900b27a0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b8d7fa640 1 -- 172.21.15.45:0/1320088041 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7800de20 con 0x7f2b900b27a0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b8d7fa640 1 -- 172.21.15.45:0/1320088041 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7800b8f0 con 0x7f2b900b27a0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b900b4050 con 0x7f2b900b27a0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.395+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b900b4610 con 0x7f2b900b27a0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.397+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b900733d0 con 0x7f2b900b27a0 2023-11-28T20:47:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.398+0000 7f2b8d7fa640 1 -- 172.21.15.45:0/1320088041 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f2b7800ba50 con 0x7f2b900b27a0 2023-11-28T20:47:37.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.399+0000 7f2b8d7fa640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2b7006b040 0x7f2b7006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:37.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.399+0000 7f2b8d7fa640 1 -- 172.21.15.45:0/1320088041 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f2b78089e30 con 0x7f2b900b27a0 2023-11-28T20:47:37.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.399+0000 7f2b8ffff640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2b7006b040 0x7f2b7006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:37.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.400+0000 7f2b8ffff640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2b7006b040 0x7f2b7006d500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2b80002230 tx=0x7f2b800012b0 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:37.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.403+0000 7f2b8d7fa640 1 -- 172.21.15.45:0/1320088041 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b780548f0 con 0x7f2b900b27a0 2023-11-28T20:47:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.628+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b901595d0 con 0x7f2b7006b040 2023-11-28T20:47:37.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.837+0000 7f2b8d7fa640 1 -- 172.21.15.45:0/1320088041 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f2b901595d0 con 0x7f2b7006b040 2023-11-28T20:47:37.839 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-11-28T20:47:37.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2b7006b040 msgr2=0x7f2b7006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2b7006b040 0x7f2b7006d500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2b80002230 tx=0x7f2b800012b0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b900b27a0 msgr2=0x7f2b901f2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b900b27a0 0x7f2b901f2490 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800d710 tx=0x7f2b7800dbd0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 shutdown_connections 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2b7006b040 0x7f2b7006d500 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b900c79c0 0x7f2b900af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b900b27a0 0x7f2b901f2490 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 --2- 172.21.15.45:0/1320088041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b7c004d00 0x7f2b900c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 >> 172.21.15.45:0/1320088041 conn(0x7f2b900ced60 msgr2=0x7f2b900ae2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.841+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 shutdown_connections 2023-11-28T20:47:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:37.842+0000 7f2b9632c640 1 -- 172.21.15.45:0/1320088041 wait complete. 2023-11-28T20:47:37.865 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-11-28T20:47:37.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:37.865 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-11-28T20:47:37.927 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_01_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_01_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_01_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:37.928 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_01_UTC 2023-11-28T20:47:37.929 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:37.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:37.930 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_36_00_UTC) 2023-11-28T20:47:38.001 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_37_01_UTC 2023-11-28T20:47:38.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.002 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_37_01_UTC) 2023-11-28T20:47:38.078 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_38_01_UTC 2023-11-28T20:47:38.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.078 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_38_01_UTC) 2023-11-28T20:47:38.151 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_39_01_UTC 2023-11-28T20:47:38.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.151 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_39_01_UTC) 2023-11-28T20:47:38.225 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:38.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.225 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_40_01_UTC) 2023-11-28T20:47:38.318 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:38.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.319 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_41_00_UTC) 2023-11-28T20:47:38.436 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:38.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.436 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_42_00_UTC) 2023-11-28T20:47:38.545 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:38.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.546 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_43_00_UTC) 2023-11-28T20:47:38.612 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:38.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.613 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_44_01_UTC) 2023-11-28T20:47:38.661 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:38.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.661 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_45_00_UTC) 2023-11-28T20:47:38.712 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:38.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.712 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_46_00_UTC) 2023-11-28T20:47:38.765 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_47_01_UTC 2023-11-28T20:47:38.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.765 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-11-28-20_47_01_UTC) 2023-11-28T20:47:38.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:38.821 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-11-28T20:47:38.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2023-11-28T20:47:39.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.089+0000 7f2f5a199640 1 Processor -- start 2023-11-28T20:47:39.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.089+0000 7f2f5a199640 1 -- start start 2023-11-28T20:47:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.091+0000 7f2f5a199640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2f540cdfb0 0x7f2f5406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.091+0000 7f2f5a199640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f5406c480 0x7f2f540b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.091+0000 7f2f5a199640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540cdb20 con 0x7f2f54155770 2023-11-28T20:47:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.091+0000 7f2f5a199640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5406c8f0 con 0x7f2f540cdfb0 2023-11-28T20:47:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.091+0000 7f2f5a199640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540ccf10 con 0x7f2f5406c480 2023-11-28T20:47:39.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.091+0000 7f2f58996640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2f540cdfb0 0x7f2f5406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.091+0000 7f2f58996640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2f540cdfb0 0x7f2f5406bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56316/0 (socket says 172.21.15.45:56316) 2023-11-28T20:47:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.091+0000 7f2f58996640 1 -- 172.21.15.45:0/1917498517 learned_addr learned my addr 172.21.15.45:0/1917498517 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.092+0000 7f2f58996640 1 -- 172.21.15.45:0/1917498517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f5406c480 msgr2=0x7f2f540b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:39.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.092+0000 7f2f58996640 1 --2- 172.21.15.45:0/1917498517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f5406c480 0x7f2f540b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.092+0000 7f2f58996640 1 -- 172.21.15.45:0/1917498517 >> v1:172.21.15.45:6789/0 conn(0x7f2f54155770 legacy=0x7f2f5415c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:39.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.092+0000 7f2f58996640 1 -- 172.21.15.45:0/1917498517 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540cd1f0 con 0x7f2f540cdfb0 2023-11-28T20:47:39.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.093+0000 7f2f58996640 1 --2- 172.21.15.45:0/1917498517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f540cdfb0 0x7f2f5406bf40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009870 tx=0x7f2f4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4b5347e3674de2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:39.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.093+0000 7f2f4bfff640 1 -- 172.21.15.45:0/1917498517 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44031640 con 0x7f2f540cdfb0 2023-11-28T20:47:39.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.093+0000 7f2f4bfff640 1 -- 172.21.15.45:0/1917498517 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f440317a0 con 0x7f2f540cdfb0 2023-11-28T20:47:39.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.093+0000 7f2f4bfff640 1 -- 172.21.15.45:0/1917498517 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44031b30 con 0x7f2f540cdfb0 2023-11-28T20:47:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.094+0000 7f2f5a199640 1 -- 172.21.15.45:0/1917498517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f540cdfb0 msgr2=0x7f2f5406bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.094+0000 7f2f5a199640 1 --2- 172.21.15.45:0/1917498517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f540cdfb0 0x7f2f5406bf40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009870 tx=0x7f2f4402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.094+0000 7f2f5a199640 1 -- 172.21.15.45:0/1917498517 shutdown_connections 2023-11-28T20:47:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.094+0000 7f2f5a199640 1 --2- 172.21.15.45:0/1917498517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f5406c480 0x7f2f540b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.094+0000 7f2f5a199640 1 --2- 172.21.15.45:0/1917498517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f540cdfb0 0x7f2f5406bf40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.094+0000 7f2f5a199640 1 -- 172.21.15.45:0/1917498517 >> 172.21.15.45:0/1917498517 conn(0x7f2f540b1050 msgr2=0x7f2f5406a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.094+0000 7f2f5a199640 1 -- 172.21.15.45:0/1917498517 shutdown_connections 2023-11-28T20:47:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.094+0000 7f2f5a199640 1 -- 172.21.15.45:0/1917498517 wait complete. 2023-11-28T20:47:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.096+0000 7f2f5a199640 1 Processor -- start 2023-11-28T20:47:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.096+0000 7f2f5a199640 1 -- start start 2023-11-28T20:47:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.097+0000 7f2f5a199640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f5406c480 0x7f2f54071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.097+0000 7f2f5a199640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f540cdfb0 0x7f2f5409b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.097+0000 7f2f5a199640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f54155770 0x7f2f54145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.097+0000 7f2f5a199640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540c80a0 con 0x7f2f5406c480 2023-11-28T20:47:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.097+0000 7f2f5a199640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540c8340 con 0x7f2f540cdfb0 2023-11-28T20:47:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.097+0000 7f2f5a199640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5409b9e0 con 0x7f2f54155770 2023-11-28T20:47:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.097+0000 7f2f59197640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f5406c480 0x7f2f54071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.097+0000 7f2f59197640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f5406c480 0x7f2f54071e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56322/0 (socket says 172.21.15.45:56322) 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.097+0000 7f2f59197640 1 -- 172.21.15.45:0/408516206 learned_addr learned my addr 172.21.15.45:0/408516206 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f59197640 1 -- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f54155770 msgr2=0x7f2f54145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f59197640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f54155770 0x7f2f54145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f59197640 1 -- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f540cdfb0 msgr2=0x7f2f5409b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f59197640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f540cdfb0 0x7f2f5409b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f59197640 1 -- 172.21.15.45:0/408516206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f440094d0 con 0x7f2f5406c480 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f59197640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f5406c480 0x7f2f54071e20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000c400 tx=0x7f2f5000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f49ffb640 1 -- 172.21.15.45:0/408516206 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500052d0 con 0x7f2f5406c480 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f49ffb640 1 -- 172.21.15.45:0/408516206 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f50005430 con 0x7f2f5406c480 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5409bc60 con 0x7f2f5406c480 2023-11-28T20:47:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f49ffb640 1 -- 172.21.15.45:0/408516206 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50003830 con 0x7f2f5406c480 2023-11-28T20:47:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.098+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5409c220 con 0x7f2f5406c480 2023-11-28T20:47:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.099+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5409c410 con 0x7f2f5406c480 2023-11-28T20:47:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.100+0000 7f2f49ffb640 1 -- 172.21.15.45:0/408516206 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f2f50003990 con 0x7f2f5406c480 2023-11-28T20:47:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.101+0000 7f2f49ffb640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2f3806b070 0x7f2f3806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.101+0000 7f2f49ffb640 1 -- 172.21.15.45:0/408516206 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 7044+0+0 (secure 0 0 0) 0x7f2f500894a0 con 0x7f2f5406c480 2023-11-28T20:47:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.102+0000 7f2f58996640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2f3806b070 0x7f2f3806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.103+0000 7f2f58996640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2f3806b070 0x7f2f3806d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2f44036000 tx=0x7f2f44030000 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.106+0000 7f2f49ffb640 1 -- 172.21.15.45:0/408516206 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f50053cf0 con 0x7f2f5406c480 2023-11-28T20:47:39.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.346+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f5406bf40 con 0x7f2f3806b070 2023-11-28T20:47:39.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.478+0000 7f2f49ffb640 1 -- 172.21.15.45:0/408516206 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f2f5406bf40 con 0x7f2f3806b070 2023-11-28T20:47:39.479 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-11-28T20:47:39.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2f3806b070 msgr2=0x7f2f3806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:39.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2f3806b070 0x7f2f3806d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2f44036000 tx=0x7f2f44030000 comp rx=0 tx=0).stop 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f5406c480 msgr2=0x7f2f54071e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f5406c480 0x7f2f54071e20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000c400 tx=0x7f2f5000c8c0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 shutdown_connections 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2f3806b070 0x7f2f3806d530 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f54155770 0x7f2f54145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f540cdfb0 0x7f2f5409b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 --2- 172.21.15.45:0/408516206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f5406c480 0x7f2f54071e20 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 >> 172.21.15.45:0/408516206 conn(0x7f2f540b1050 msgr2=0x7f2f540aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 shutdown_connections 2023-11-28T20:47:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:39.481+0000 7f2f5a199640 1 -- 172.21.15.45:0/408516206 wait complete. 2023-11-28T20:47:39.503 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-11-28T20:47:39.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:39.503 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:39.554 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_01_UTC 2023-11-28T20:47:39.555 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:39.555 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:39.556 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:39.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:39.556 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_36_00_UTC) 2023-11-28T20:47:39.629 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:39.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:39.630 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_37_00_UTC) 2023-11-28T20:47:39.704 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:39.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:39.704 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_38_00_UTC) 2023-11-28T20:47:39.774 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:39.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:39.774 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_39_00_UTC) 2023-11-28T20:47:39.846 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:39.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:39.847 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_40_00_UTC) 2023-11-28T20:47:39.924 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:39.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:39.924 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_41_00_UTC) 2023-11-28T20:47:40.001 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:40.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:40.001 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_42_00_UTC) 2023-11-28T20:47:40.070 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:40.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:40.070 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_43_00_UTC) 2023-11-28T20:47:40.133 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:40.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:40.134 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_44_00_UTC) 2023-11-28T20:47:40.204 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_45_01_UTC 2023-11-28T20:47:40.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:40.205 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_45_01_UTC) 2023-11-28T20:47:40.274 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:40.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:40.275 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_46_01_UTC) 2023-11-28T20:47:40.375 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:40.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:40.375 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-11-28-20_47_00_UTC) 2023-11-28T20:47:40.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:40.427 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-11-28T20:47:40.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2023-11-28T20:47:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.732+0000 7fee67178640 1 Processor -- start 2023-11-28T20:47:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.733+0000 7fee67178640 1 -- start start 2023-11-28T20:47:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.734+0000 7fee67178640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fee60099a40 0x7fee6006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.734+0000 7fee67178640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee600cbd30 0x7fee600cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.734+0000 7fee67178640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee600d1fc0 con 0x7fee600aff90 2023-11-28T20:47:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.735+0000 7fee67178640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6006a910 con 0x7fee60099a40 2023-11-28T20:47:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.735+0000 7fee67178640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee600cb8a0 con 0x7fee600cbd30 2023-11-28T20:47:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.735+0000 7fee64f14640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fee60099a40 0x7fee6006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.735+0000 7fee64f14640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fee60099a40 0x7fee6006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56338/0 (socket says 172.21.15.45:56338) 2023-11-28T20:47:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.735+0000 7fee64f14640 1 -- 172.21.15.45:0/2411435423 learned_addr learned my addr 172.21.15.45:0/2411435423 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.735+0000 7fee64f14640 1 -- 172.21.15.45:0/2411435423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee600cbd30 msgr2=0x7fee600cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.735+0000 7fee64f14640 1 --2- 172.21.15.45:0/2411435423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee600cbd30 0x7fee600cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.735+0000 7fee64f14640 1 -- 172.21.15.45:0/2411435423 >> v1:172.21.15.190:6789/0 conn(0x7fee600aff90 legacy=0x7fee60099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.735+0000 7fee64f14640 1 -- 172.21.15.45:0/2411435423 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6006a630 con 0x7fee60099a40 2023-11-28T20:47:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.737+0000 7fee64f14640 1 --2- 172.21.15.45:0/2411435423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee60099a40 0x7fee6006b4d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fee50009e50 tx=0x7fee5002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c06d7f2fa9f352 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.737+0000 7fee5f7fe640 1 -- 172.21.15.45:0/2411435423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5002c9e0 con 0x7fee60099a40 2023-11-28T20:47:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.737+0000 7fee5f7fe640 1 -- 172.21.15.45:0/2411435423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5002cb40 con 0x7fee60099a40 2023-11-28T20:47:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.737+0000 7fee5f7fe640 1 -- 172.21.15.45:0/2411435423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee500343f0 con 0x7fee60099a40 2023-11-28T20:47:40.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.738+0000 7fee67178640 1 -- 172.21.15.45:0/2411435423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee60099a40 msgr2=0x7fee6006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:40.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.738+0000 7fee67178640 1 --2- 172.21.15.45:0/2411435423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee60099a40 0x7fee6006b4d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fee50009e50 tx=0x7fee5002be40 comp rx=0 tx=0).stop 2023-11-28T20:47:40.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.738+0000 7fee67178640 1 -- 172.21.15.45:0/2411435423 shutdown_connections 2023-11-28T20:47:40.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.738+0000 7fee67178640 1 --2- 172.21.15.45:0/2411435423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee600cbd30 0x7fee600cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:40.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.738+0000 7fee67178640 1 --2- 172.21.15.45:0/2411435423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee60099a40 0x7fee6006b4d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.738+0000 7fee67178640 1 -- 172.21.15.45:0/2411435423 >> 172.21.15.45:0/2411435423 conn(0x7fee600d1020 msgr2=0x7fee600d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.738+0000 7fee67178640 1 -- 172.21.15.45:0/2411435423 shutdown_connections 2023-11-28T20:47:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.738+0000 7fee67178640 1 -- 172.21.15.45:0/2411435423 wait complete. 2023-11-28T20:47:40.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.740+0000 7fee67178640 1 Processor -- start 2023-11-28T20:47:40.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.741+0000 7fee67178640 1 -- start start 2023-11-28T20:47:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.743+0000 7fee67178640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee60099a40 0x7fee6015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.743+0000 7fee67178640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee600aff90 0x7fee600b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.743+0000 7fee67178640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee600cbd30 0x7fee6015fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.743+0000 7fee67178640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6015ad20 con 0x7fee600aff90 2023-11-28T20:47:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.743+0000 7fee67178640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee600c84d0 con 0x7fee60099a40 2023-11-28T20:47:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.743+0000 7fee67178640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee600c87b0 con 0x7fee600cbd30 2023-11-28T20:47:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.744+0000 7fee5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee600aff90 0x7fee600b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.744+0000 7fee5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee600aff90 0x7fee600b6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56350/0 (socket says 172.21.15.45:56350) 2023-11-28T20:47:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.744+0000 7fee5ffff640 1 -- 172.21.15.45:0/413128659 learned_addr learned my addr 172.21.15.45:0/413128659 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.744+0000 7fee64f14640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee60099a40 0x7fee6015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.745+0000 7fee5ffff640 1 -- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee600cbd30 msgr2=0x7fee6015fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.745+0000 7fee5ffff640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee600cbd30 0x7fee6015fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.745+0000 7fee5ffff640 1 -- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee60099a40 msgr2=0x7fee6015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.745+0000 7fee5ffff640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee60099a40 0x7fee6015a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.745+0000 7fee5ffff640 1 -- 172.21.15.45:0/413128659 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50009b30 con 0x7fee600aff90 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.745+0000 7fee5ffff640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee600aff90 0x7fee600b6550 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fee5400c430 tx=0x7fee5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.745+0000 7fee5d7fa640 1 -- 172.21.15.45:0/413128659 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee54011790 con 0x7fee600aff90 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.746+0000 7fee5d7fa640 1 -- 172.21.15.45:0/413128659 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee540118f0 con 0x7fee600aff90 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.746+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee600c8a90 con 0x7fee600aff90 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.746+0000 7fee5d7fa640 1 -- 172.21.15.45:0/413128659 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee54011ce0 con 0x7fee600aff90 2023-11-28T20:47:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.746+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee600c9050 con 0x7fee600aff90 2023-11-28T20:47:40.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.748+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee600c9270 con 0x7fee600aff90 2023-11-28T20:47:40.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.749+0000 7fee5d7fa640 1 -- 172.21.15.45:0/413128659 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7fee5401c460 con 0x7fee600aff90 2023-11-28T20:47:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.751+0000 7fee5d7fa640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee4c06b040 0x7fee4c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.751+0000 7fee5d7fa640 1 -- 172.21.15.45:0/413128659 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6996+0+0 (secure 0 0 0) 0x7fee54089d70 con 0x7fee600aff90 2023-11-28T20:47:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.752+0000 7fee64f14640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee4c06b040 0x7fee4c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:40.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.756+0000 7fee64f14640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee4c06b040 0x7fee4c06d500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee50035000 tx=0x7fee50030000 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:40.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.759+0000 7fee5d7fa640 1 -- 172.21.15.45:0/413128659 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee54054720 con 0x7fee600aff90 2023-11-28T20:47:40.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:40.992+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fee600c8920 con 0x7fee4c06b040 2023-11-28T20:47:41.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.184+0000 7fee5d7fa640 1 -- 172.21.15.45:0/413128659 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fee600c8920 con 0x7fee4c06b040 2023-11-28T20:47:41.186 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-11-28T20:47:41.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee4c06b040 msgr2=0x7fee4c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:41.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee4c06b040 0x7fee4c06d500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee50035000 tx=0x7fee50030000 comp rx=0 tx=0).stop 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee600aff90 msgr2=0x7fee600b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee600aff90 0x7fee600b6550 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fee5400c430 tx=0x7fee5400c8f0 comp rx=0 tx=0).stop 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 shutdown_connections 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fee4c06b040 0x7fee4c06d500 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee600cbd30 0x7fee6015fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee600aff90 0x7fee600b6550 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 --2- 172.21.15.45:0/413128659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee60099a40 0x7fee6015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 >> 172.21.15.45:0/413128659 conn(0x7fee600d1020 msgr2=0x7fee600cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.188+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 shutdown_connections 2023-11-28T20:47:41.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:41.189+0000 7fee67178640 1 -- 172.21.15.45:0/413128659 wait complete. 2023-11-28T20:47:41.211 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-11-28T20:47:41.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.211 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-11-28T20:47:41.272 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:41.272 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:41.272 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:41.272 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:41.272 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:41.272 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_01_UTC 2023-11-28T20:47:41.273 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:41.273 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:41.273 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:41.273 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:41.273 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:41.273 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:41.274 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:41.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.274 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_36_00_UTC) 2023-11-28T20:47:41.355 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:41.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.355 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_37_00_UTC) 2023-11-28T20:47:41.419 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:41.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.419 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_38_00_UTC) 2023-11-28T20:47:41.468 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:41.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.468 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_39_00_UTC) 2023-11-28T20:47:41.517 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:41.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.517 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_40_00_UTC) 2023-11-28T20:47:41.570 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_41_01_UTC 2023-11-28T20:47:41.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.570 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_41_01_UTC) 2023-11-28T20:47:41.632 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:41.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.632 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_42_00_UTC) 2023-11-28T20:47:41.699 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:41.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.699 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_43_00_UTC) 2023-11-28T20:47:41.777 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:41.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.778 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_44_00_UTC) 2023-11-28T20:47:41.829 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:41.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.829 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_45_00_UTC) 2023-11-28T20:47:41.881 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:41.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.881 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_46_00_UTC) 2023-11-28T20:47:41.937 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:41.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.937 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-11-28-20_47_00_UTC) 2023-11-28T20:47:41.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:41.998 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-11-28T20:47:42.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2023-11-28T20:47:42.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.281+0000 7f003187e640 1 Processor -- start 2023-11-28T20:47:42.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.281+0000 7f003187e640 1 -- start start 2023-11-28T20:47:42.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.283+0000 7f003187e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f002c15d2f0 0x7f002c0d1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:42.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.283+0000 7f003187e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c0d02b0 con 0x7f002c15cf10 2023-11-28T20:47:42.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.283+0000 7f003187e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c0d1720 con 0x7f002c0ce000 2023-11-28T20:47:42.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.283+0000 7f003187e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002c0d1a00 con 0x7f002c15d2f0 2023-11-28T20:47:42.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.283+0000 7f003087c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f002c15cf10 0x7f002c0cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44388/0 (socket says 172.21.15.45:44388) 2023-11-28T20:47:42.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.283+0000 7f003107d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f002c15d2f0 0x7f002c0d1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:42.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.283+0000 7f003087c640 1 -- 172.21.15.45:0/3720746782 learned_addr learned my addr 172.21.15.45:0/3720746782 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:42.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.284+0000 7f003107d640 1 -- 172.21.15.45:0/3720746782 >> v1:172.21.15.190:6789/0 conn(0x7f002c0ce000 legacy=0x7f002c0d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:42.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.284+0000 7f003107d640 1 -- 172.21.15.45:0/3720746782 >> v1:172.21.15.45:6789/0 conn(0x7f002c15cf10 legacy=0x7f002c0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:42.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.284+0000 7f003107d640 1 -- 172.21.15.45:0/3720746782 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c0d1ce0 con 0x7f002c15d2f0 2023-11-28T20:47:42.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.285+0000 7f003107d640 1 --2- 172.21.15.45:0/3720746782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15d2f0 0x7f002c0d1090 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0024009d90 tx=0x7f002402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa75ed279e7cf66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:42.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.285+0000 7f002b7fe640 1 -- 172.21.15.45:0/3720746782 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002402cc60 con 0x7f002c15d2f0 2023-11-28T20:47:42.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.285+0000 7f002b7fe640 1 -- 172.21.15.45:0/3720746782 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002402cdc0 con 0x7f002c15d2f0 2023-11-28T20:47:42.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.285+0000 7f002b7fe640 1 -- 172.21.15.45:0/3720746782 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024034660 con 0x7f002c15d2f0 2023-11-28T20:47:42.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.286+0000 7f003187e640 1 -- 172.21.15.45:0/3720746782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15d2f0 msgr2=0x7f002c0d1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:42.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.286+0000 7f003187e640 1 --2- 172.21.15.45:0/3720746782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15d2f0 0x7f002c0d1090 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0024009d90 tx=0x7f002402beb0 comp rx=0 tx=0).stop 2023-11-28T20:47:42.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.286+0000 7f003187e640 1 -- 172.21.15.45:0/3720746782 shutdown_connections 2023-11-28T20:47:42.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.286+0000 7f003187e640 1 --2- 172.21.15.45:0/3720746782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15d2f0 0x7f002c0d1090 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:42.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.286+0000 7f003187e640 1 -- 172.21.15.45:0/3720746782 >> 172.21.15.45:0/3720746782 conn(0x7f002c070f40 msgr2=0x7f002c071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:42.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.286+0000 7f003187e640 1 -- 172.21.15.45:0/3720746782 shutdown_connections 2023-11-28T20:47:42.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.286+0000 7f003187e640 1 -- 172.21.15.45:0/3720746782 wait complete. 2023-11-28T20:47:42.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.288+0000 7f003187e640 1 Processor -- start 2023-11-28T20:47:42.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.288+0000 7f003187e640 1 -- start start 2023-11-28T20:47:42.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.289+0000 7f003187e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f002c0ce000 0x7f002c15f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:42.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.289+0000 7f003187e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15cf10 0x7f002c0c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:42.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.289+0000 7f003187e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f002c15d2f0 0x7f002c1e8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:42.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.289+0000 7f003187e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0c7210 con 0x7f002c15cf10 2023-11-28T20:47:42.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.289+0000 7f003187e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c156c10 con 0x7f002c0ce000 2023-11-28T20:47:42.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.289+0000 7f003187e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0b30b0 con 0x7f002c15d2f0 2023-11-28T20:47:42.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f002bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15cf10 0x7f002c0c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:42.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f002bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15cf10 0x7f002c0c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56372/0 (socket says 172.21.15.45:56372) 2023-11-28T20:47:42.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f002bfff640 1 -- 172.21.15.45:0/3190325603 learned_addr learned my addr 172.21.15.45:0/3190325603 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:42.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f003087c640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f002c0ce000 0x7f002c15f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:42.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f003107d640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f002c15d2f0 0x7f002c1e8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:42.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f002bfff640 1 -- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f002c15d2f0 msgr2=0x7f002c1e8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:42.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f002bfff640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f002c15d2f0 0x7f002c1e8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:42.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f002bfff640 1 -- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f002c0ce000 msgr2=0x7f002c15f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:42.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f002bfff640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f002c0ce000 0x7f002c15f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:42.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f002bfff640 1 -- 172.21.15.45:0/3190325603 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0024009a70 con 0x7f002c15cf10 2023-11-28T20:47:42.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.290+0000 7f002bfff640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15cf10 0x7f002c0c76a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f001800b3e0 tx=0x7f001800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:42.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.291+0000 7f00297fa640 1 -- 172.21.15.45:0/3190325603 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018011830 con 0x7f002c15cf10 2023-11-28T20:47:42.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.291+0000 7f00297fa640 1 -- 172.21.15.45:0/3190325603 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00180119b0 con 0x7f002c15cf10 2023-11-28T20:47:42.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.291+0000 7f00297fa640 1 -- 172.21.15.45:0/3190325603 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018011da0 con 0x7f002c15cf10 2023-11-28T20:47:42.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.291+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002c0b3390 con 0x7f002c15cf10 2023-11-28T20:47:42.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.291+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002c0c6ea0 con 0x7f002c15cf10 2023-11-28T20:47:42.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.292+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002c0c7380 con 0x7f002c15cf10 2023-11-28T20:47:42.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.293+0000 7f00297fa640 1 -- 172.21.15.45:0/3190325603 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f001801a400 con 0x7f002c15cf10 2023-11-28T20:47:42.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.294+0000 7f00297fa640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f001006b040 0x7f001006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:42.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.294+0000 7f00297fa640 1 -- 172.21.15.45:0/3190325603 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7076+0+0 (secure 0 0 0) 0x7f001808a0a0 con 0x7f002c15cf10 2023-11-28T20:47:42.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.295+0000 7f003087c640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f001006b040 0x7f001006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:42.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.296+0000 7f003087c640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f001006b040 0x7f001006d500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0020005880 tx=0x7f0020009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:42.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.298+0000 7f00297fa640 1 -- 172.21.15.45:0/3190325603 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0018054950 con 0x7f002c15cf10 2023-11-28T20:47:42.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.537+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f002c160c60 con 0x7f001006b040 2023-11-28T20:47:42.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.752+0000 7f00297fa640 1 -- 172.21.15.45:0/3190325603 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f0020007e40 con 0x7f001006b040 2023-11-28T20:47:42.754 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-11-28T20:47:42.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.755+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f001006b040 msgr2=0x7f001006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:42.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.755+0000 7f003187e640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f001006b040 0x7f001006d500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0020005880 tx=0x7f0020009040 comp rx=0 tx=0).stop 2023-11-28T20:47:42.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.755+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15cf10 msgr2=0x7f002c0c76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:42.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.755+0000 7f003187e640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15cf10 0x7f002c0c76a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f001800b3e0 tx=0x7f001800b8a0 comp rx=0 tx=0).stop 2023-11-28T20:47:42.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.756+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 shutdown_connections 2023-11-28T20:47:42.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.756+0000 7f003187e640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f001006b040 0x7f001006d500 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:42.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.756+0000 7f003187e640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f002c15d2f0 0x7f002c1e8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:42.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.756+0000 7f003187e640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f002c15cf10 0x7f002c0c76a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:42.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.756+0000 7f003187e640 1 --2- 172.21.15.45:0/3190325603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f002c0ce000 0x7f002c15f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:42.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.756+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 >> 172.21.15.45:0/3190325603 conn(0x7f002c070f40 msgr2=0x7f002c0b6420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:42.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.756+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 shutdown_connections 2023-11-28T20:47:42.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:42.756+0000 7f003187e640 1 -- 172.21.15.45:0/3190325603 wait complete. 2023-11-28T20:47:42.778 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-11-28T20:47:42.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:42.778 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-11-28T20:47:42.837 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_01_UTC 2023-11-28T20:47:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:42.838 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:42.839 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:42.839 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:42.839 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:42.839 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_01_UTC 2023-11-28T20:47:42.840 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_36_00_UTC 2023-11-28T20:47:42.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:42.840 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_36_00_UTC) 2023-11-28T20:47:42.907 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_37_01_UTC 2023-11-28T20:47:42.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:42.907 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_37_01_UTC) 2023-11-28T20:47:42.977 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:42.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:42.978 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_38_00_UTC) 2023-11-28T20:47:43.051 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:43.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.052 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_39_00_UTC) 2023-11-28T20:47:43.121 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:43.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.122 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_40_00_UTC) 2023-11-28T20:47:43.189 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:43.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.193 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_41_00_UTC) 2023-11-28T20:47:43.261 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:43.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.262 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_42_01_UTC) 2023-11-28T20:47:43.367 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:43.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.368 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_43_01_UTC) 2023-11-28T20:47:43.465 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:43.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.466 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_44_00_UTC) 2023-11-28T20:47:43.515 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:43.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.516 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_45_00_UTC) 2023-11-28T20:47:43.564 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:43.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.565 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_46_01_UTC) 2023-11-28T20:47:43.618 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_47_01_UTC 2023-11-28T20:47:43.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.619 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-11-28-20_47_01_UTC) 2023-11-28T20:47:43.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:43.677 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-11-28T20:47:43.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2023-11-28T20:47:43.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.934+0000 7f5d41dfc640 1 Processor -- start 2023-11-28T20:47:43.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.934+0000 7f5d41dfc640 1 -- start start 2023-11-28T20:47:43.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.936+0000 7f5d41dfc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5d3c15e4e0 0x7f5d3c06a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.936+0000 7f5d41dfc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c06ca60 con 0x7f5d3c157c10 2023-11-28T20:47:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.936+0000 7f5d41dfc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c06aea0 con 0x7f5d3c06a500 2023-11-28T20:47:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.936+0000 7f5d41dfc640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d3c15e930 con 0x7f5d3c15e4e0 2023-11-28T20:47:43.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.936+0000 7f5d3b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5d3c157c10 0x7f5d3c06da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52350/0 (socket says 172.21.15.45:52350) 2023-11-28T20:47:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.936+0000 7f5d3b7fe640 1 -- 172.21.15.45:0/847993617 learned_addr learned my addr 172.21.15.45:0/847993617 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.936+0000 7f5d3bfff640 1 --2- 172.21.15.45:0/847993617 >> v2:172.21.15.45:3300/0 conn(0x7f5d3c15e4e0 0x7f5d3c06a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.937+0000 7f5d3bfff640 1 -- 172.21.15.45:0/847993617 >> v1:172.21.15.190:6789/0 conn(0x7f5d3c06a500 legacy=0x7f5d3c15ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.937+0000 7f5d3bfff640 1 -- 172.21.15.45:0/847993617 >> v1:172.21.15.45:6789/0 conn(0x7f5d3c157c10 legacy=0x7f5d3c06da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.937+0000 7f5d3bfff640 1 -- 172.21.15.45:0/847993617 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c15eb40 con 0x7f5d3c15e4e0 2023-11-28T20:47:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.937+0000 7f5d3affd640 1 -- 172.21.15.45:0/847993617 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545529940 0 0) 0x7f5d3c06ca60 con 0x7f5d3c157c10 2023-11-28T20:47:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.938+0000 7f5d3bfff640 1 --2- 172.21.15.45:0/847993617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c15e4e0 0x7f5d3c06a960 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5d340097a0 tx=0x7f5d3402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d430e2225c1f39 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.938+0000 7f5d3affd640 1 -- 172.21.15.45:0/847993617 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34030610 con 0x7f5d3c15e4e0 2023-11-28T20:47:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.938+0000 7f5d3affd640 1 -- 172.21.15.45:0/847993617 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d34030770 con 0x7f5d3c15e4e0 2023-11-28T20:47:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.938+0000 7f5d3affd640 1 -- 172.21.15.45:0/847993617 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34030b00 con 0x7f5d3c15e4e0 2023-11-28T20:47:43.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.939+0000 7f5d41dfc640 1 -- 172.21.15.45:0/847993617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c15e4e0 msgr2=0x7f5d3c06a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:43.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.939+0000 7f5d41dfc640 1 --2- 172.21.15.45:0/847993617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c15e4e0 0x7f5d3c06a960 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5d340097a0 tx=0x7f5d3402b960 comp rx=0 tx=0).stop 2023-11-28T20:47:43.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.939+0000 7f5d41dfc640 1 -- 172.21.15.45:0/847993617 shutdown_connections 2023-11-28T20:47:43.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.939+0000 7f5d41dfc640 1 --2- 172.21.15.45:0/847993617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c15e4e0 0x7f5d3c06a960 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:43.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.939+0000 7f5d41dfc640 1 -- 172.21.15.45:0/847993617 >> 172.21.15.45:0/847993617 conn(0x7f5d3c0ced60 msgr2=0x7f5d3c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:43.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.939+0000 7f5d41dfc640 1 -- 172.21.15.45:0/847993617 shutdown_connections 2023-11-28T20:47:43.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.939+0000 7f5d41dfc640 1 -- 172.21.15.45:0/847993617 wait complete. 2023-11-28T20:47:43.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.941+0000 7f5d41dfc640 1 Processor -- start 2023-11-28T20:47:43.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.941+0000 7f5d41dfc640 1 -- start start 2023-11-28T20:47:43.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.942+0000 7f5d41dfc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c06a500 0x7f5d3c14c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:43.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.942+0000 7f5d41dfc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d3c157c10 0x7f5d3c1478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:43.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.942+0000 7f5d41dfc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d3c15e4e0 0x7f5d3c1581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:43.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.942+0000 7f5d41dfc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c0add50 con 0x7f5d3c06a500 2023-11-28T20:47:43.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.942+0000 7f5d41dfc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c1523b0 con 0x7f5d3c15e4e0 2023-11-28T20:47:43.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.942+0000 7f5d41dfc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c14b830 con 0x7f5d3c157c10 2023-11-28T20:47:43.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.942+0000 7f5d3b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c06a500 0x7f5d3c14c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:43.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.942+0000 7f5d3b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c06a500 0x7f5d3c14c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56950/0 (socket says 172.21.15.45:56950) 2023-11-28T20:47:43.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.942+0000 7f5d3b7fe640 1 -- 172.21.15.45:0/794026546 learned_addr learned my addr 172.21.15.45:0/794026546 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.943+0000 7f5d32ffd640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d3c157c10 0x7f5d3c1478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.943+0000 7f5d3bfff640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d3c15e4e0 0x7f5d3c1581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.943+0000 7f5d3b7fe640 1 -- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d3c157c10 msgr2=0x7f5d3c1478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.943+0000 7f5d3b7fe640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d3c157c10 0x7f5d3c1478a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.943+0000 7f5d3b7fe640 1 -- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d3c15e4e0 msgr2=0x7f5d3c1581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.943+0000 7f5d3b7fe640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d3c15e4e0 0x7f5d3c1581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.943+0000 7f5d3b7fe640 1 -- 172.21.15.45:0/794026546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d34009400 con 0x7f5d3c06a500 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.943+0000 7f5d3bfff640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d3c15e4e0 0x7f5d3c1581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.943+0000 7f5d3b7fe640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c06a500 0x7f5d3c14c5f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00c8c0 tx=0x7f5d2c00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.944+0000 7f5d38ff9640 1 -- 172.21.15.45:0/794026546 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c00a640 con 0x7f5d3c06a500 2023-11-28T20:47:43.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.944+0000 7f5d38ff9640 1 -- 172.21.15.45:0/794026546 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2c00a7a0 con 0x7f5d3c06a500 2023-11-28T20:47:43.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.944+0000 7f5d38ff9640 1 -- 172.21.15.45:0/794026546 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c00ab90 con 0x7f5d3c06a500 2023-11-28T20:47:43.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.944+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c14bb10 con 0x7f5d3c06a500 2023-11-28T20:47:43.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.944+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c14c0d0 con 0x7f5d3c06a500 2023-11-28T20:47:43.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.945+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3c0733d0 con 0x7f5d3c06a500 2023-11-28T20:47:43.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.946+0000 7f5d38ff9640 1 -- 172.21.15.45:0/794026546 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f5d2c00acf0 con 0x7f5d3c06a500 2023-11-28T20:47:43.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.947+0000 7f5d38ff9640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f5d1c06b070 0x7f5d1c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:43.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.947+0000 7f5d38ff9640 1 -- 172.21.15.45:0/794026546 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7124+0+0 (secure 0 0 0) 0x7f5d2c0893f0 con 0x7f5d3c06a500 2023-11-28T20:47:43.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.947+0000 7f5d32ffd640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f5d1c06b070 0x7f5d1c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:43.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.949+0000 7f5d32ffd640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f5d1c06b070 0x7f5d1c06d530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0cf390 tx=0x7f5d24009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:43.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:43.951+0000 7f5d38ff9640 1 -- 172.21.15.45:0/794026546 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d2c053c80 con 0x7f5d3c06a500 2023-11-28T20:47:44.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.174+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d3c06cf70 con 0x7f5d1c06b070 2023-11-28T20:47:44.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.367+0000 7f5d38ff9640 1 -- 172.21.15.45:0/794026546 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f5d240012e0 con 0x7f5d1c06b070 2023-11-28T20:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-11-28T20:47:44.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f5d1c06b070 msgr2=0x7f5d1c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:44.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f5d1c06b070 0x7f5d1c06d530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0cf390 tx=0x7f5d24009040 comp rx=0 tx=0).stop 2023-11-28T20:47:44.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c06a500 msgr2=0x7f5d3c14c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:44.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c06a500 0x7f5d3c14c5f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00c8c0 tx=0x7f5d2c00cd80 comp rx=0 tx=0).stop 2023-11-28T20:47:44.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 shutdown_connections 2023-11-28T20:47:44.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f5d1c06b070 0x7f5d1c06d530 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:44.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d3c15e4e0 0x7f5d3c1581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:44.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d3c157c10 0x7f5d3c1478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:44.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 --2- 172.21.15.45:0/794026546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d3c06a500 0x7f5d3c14c5f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:44.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 >> 172.21.15.45:0/794026546 conn(0x7f5d3c0ced60 msgr2=0x7f5d3c159ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:44.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 shutdown_connections 2023-11-28T20:47:44.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:44.370+0000 7f5d41dfc640 1 -- 172.21.15.45:0/794026546 wait complete. 2023-11-28T20:47:44.392 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-11-28T20:47:44.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.392 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-11-28T20:47:44.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:44.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_01_UTC 2023-11-28T20:47:44.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_01_UTC 2023-11-28T20:47:44.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:44.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_01_UTC 2023-11-28T20:47:44.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:44.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:44.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:44.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:44.446 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:44.446 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:44.447 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:44.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.447 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_37_00_UTC) 2023-11-28T20:47:44.514 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_38_01_UTC 2023-11-28T20:47:44.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.514 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_38_01_UTC) 2023-11-28T20:47:44.584 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_39_01_UTC 2023-11-28T20:47:44.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.584 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_39_01_UTC) 2023-11-28T20:47:44.632 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:44.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.633 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_40_01_UTC) 2023-11-28T20:47:44.682 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_41_01_UTC 2023-11-28T20:47:44.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.683 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_41_01_UTC) 2023-11-28T20:47:44.734 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:44.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.735 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_42_01_UTC) 2023-11-28T20:47:44.791 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:44.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.792 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_43_01_UTC) 2023-11-28T20:47:44.853 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:44.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.853 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_44_01_UTC) 2023-11-28T20:47:44.922 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:44.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.922 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_45_00_UTC) 2023-11-28T20:47:44.990 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:44.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:44.990 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_46_00_UTC) 2023-11-28T20:47:45.059 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:45.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:45.060 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-11-28-20_47_00_UTC) 2023-11-28T20:47:45.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:45.132 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-11-28T20:47:45.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2023-11-28T20:47:45.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.431+0000 7f97c49d7640 1 Processor -- start 2023-11-28T20:47:45.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.431+0000 7f97c49d7640 1 -- start start 2023-11-28T20:47:45.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.432+0000 7f97c49d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 0x7f97c00ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:45.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.432+0000 7f97c49d7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97c00b2b70 0x7f97c00b3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:45.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97c49d7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c00c5f60 con 0x7f97c0146bc0 2023-11-28T20:47:45.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97c49d7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c0146fa0 con 0x7f97c00b2b70 2023-11-28T20:47:45.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97c49d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00b3540 con 0x7f97c006c0b0 2023-11-28T20:47:45.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97bed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97c00b2b70 0x7f97c00b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:45.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97bed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97c00b2b70 0x7f97c00b3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56964/0 (socket says 172.21.15.45:56964) 2023-11-28T20:47:45.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97bed9d640 1 -- 172.21.15.45:0/2027283495 learned_addr learned my addr 172.21.15.45:0/2027283495 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97bed9d640 1 -- 172.21.15.45:0/2027283495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 msgr2=0x7f97c00ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97bed9d640 1 --2- 172.21.15.45:0/2027283495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 0x7f97c00ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97bed9d640 1 -- 172.21.15.45:0/2027283495 >> v1:172.21.15.190:6789/0 conn(0x7f97c0146bc0 legacy=0x7f97c01618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:45.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.433+0000 7f97bed9d640 1 -- 172.21.15.45:0/2027283495 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c00c6240 con 0x7f97c00b2b70 2023-11-28T20:47:45.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.435+0000 7f97bed9d640 1 --2- 172.21.15.45:0/2027283495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c00b2b70 0x7f97c00b3000 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009e50 tx=0x7f97b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab2624e9fe15279 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:45.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.435+0000 7f97bdd9b640 1 -- 172.21.15.45:0/2027283495 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b802ca70 con 0x7f97c00b2b70 2023-11-28T20:47:45.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.435+0000 7f97bdd9b640 1 -- 172.21.15.45:0/2027283495 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b802cbd0 con 0x7f97c00b2b70 2023-11-28T20:47:45.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.435+0000 7f97bdd9b640 1 -- 172.21.15.45:0/2027283495 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b80343f0 con 0x7f97c00b2b70 2023-11-28T20:47:45.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.436+0000 7f97c49d7640 1 -- 172.21.15.45:0/2027283495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c00b2b70 msgr2=0x7f97c00b3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:45.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.436+0000 7f97c49d7640 1 --2- 172.21.15.45:0/2027283495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c00b2b70 0x7f97c00b3000 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009e50 tx=0x7f97b802bed0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.436+0000 7f97c49d7640 1 -- 172.21.15.45:0/2027283495 shutdown_connections 2023-11-28T20:47:45.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.436+0000 7f97c49d7640 1 --2- 172.21.15.45:0/2027283495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c00b2b70 0x7f97c00b3000 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.436+0000 7f97c49d7640 1 --2- 172.21.15.45:0/2027283495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 0x7f97c00ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.436+0000 7f97c49d7640 1 -- 172.21.15.45:0/2027283495 >> 172.21.15.45:0/2027283495 conn(0x7f97c0066820 msgr2=0x7f97c006c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:45.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.436+0000 7f97c49d7640 1 -- 172.21.15.45:0/2027283495 shutdown_connections 2023-11-28T20:47:45.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.436+0000 7f97c49d7640 1 -- 172.21.15.45:0/2027283495 wait complete. 2023-11-28T20:47:45.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.438+0000 7f97c49d7640 1 Processor -- start 2023-11-28T20:47:45.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.438+0000 7f97c49d7640 1 -- start start 2023-11-28T20:47:45.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.439+0000 7f97c49d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 0x7f97c00b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:45.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.439+0000 7f97c49d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c00b2b70 0x7f97c0135240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:45.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.439+0000 7f97c49d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c0146bc0 0x7f97c0142780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:45.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.439+0000 7f97c49d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00ca250 con 0x7f97c0146bc0 2023-11-28T20:47:45.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.439+0000 7f97c49d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00c9640 con 0x7f97c00b2b70 2023-11-28T20:47:45.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.439+0000 7f97c49d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00c9920 con 0x7f97c006c0b0 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.439+0000 7f97bed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c0146bc0 0x7f97c0142780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.439+0000 7f97bed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c0146bc0 0x7f97c0142780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56974/0 (socket says 172.21.15.45:56974) 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.439+0000 7f97bed9d640 1 -- 172.21.15.45:0/1807480126 learned_addr learned my addr 172.21.15.45:0/1807480126 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97b7fff640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c00b2b70 0x7f97c0135240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97be59c640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 0x7f97c00b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97bed9d640 1 -- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 msgr2=0x7f97c00b3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97bed9d640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 0x7f97c00b3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97bed9d640 1 -- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c00b2b70 msgr2=0x7f97c0135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97bed9d640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c00b2b70 0x7f97c0135240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97bed9d640 1 -- 172.21.15.45:0/1807480126 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b8009b30 con 0x7f97c0146bc0 2023-11-28T20:47:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97b7fff640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c00b2b70 0x7f97c0135240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:45.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97be59c640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 0x7f97c00b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:45.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.440+0000 7f97bed9d640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c0146bc0 0x7f97c0142780 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f97b8036040 tx=0x7f97b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:45.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.441+0000 7f97b6ffd640 1 -- 172.21.15.45:0/1807480126 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b803e450 con 0x7f97c0146bc0 2023-11-28T20:47:45.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.441+0000 7f97b6ffd640 1 -- 172.21.15.45:0/1807480126 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b803e5d0 con 0x7f97c0146bc0 2023-11-28T20:47:45.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.441+0000 7f97b6ffd640 1 -- 172.21.15.45:0/1807480126 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b803e980 con 0x7f97c0146bc0 2023-11-28T20:47:45.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.441+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c00c9c00 con 0x7f97c0146bc0 2023-11-28T20:47:45.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.441+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c00b4210 con 0x7f97c0146bc0 2023-11-28T20:47:45.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.442+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c00b43a0 con 0x7f97c0146bc0 2023-11-28T20:47:45.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.443+0000 7f97b6ffd640 1 -- 172.21.15.45:0/1807480126 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f97b803eae0 con 0x7f97c0146bc0 2023-11-28T20:47:45.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.445+0000 7f97b6ffd640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f97a006b040 0x7f97a006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:45.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.445+0000 7f97b6ffd640 1 -- 172.21.15.45:0/1807480126 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f97b80acd90 con 0x7f97c0146bc0 2023-11-28T20:47:45.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.445+0000 7f97be59c640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f97a006b040 0x7f97a006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:45.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.447+0000 7f97be59c640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f97a006b040 0x7f97a006d500 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007fd0 tx=0x7f97b0009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:45.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.449+0000 7f97b6ffd640 1 -- 172.21.15.45:0/1807480126 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f97b8077690 con 0x7f97c0146bc0 2023-11-28T20:47:45.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.682+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f97c00ca0c0 con 0x7f97a006b040 2023-11-28T20:47:45.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.910+0000 7f97b6ffd640 1 -- 172.21.15.45:0/1807480126 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f97c00ca0c0 con 0x7f97a006b040 2023-11-28T20:47:45.911 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-11-28T20:47:45.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.913+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f97a006b040 msgr2=0x7f97a006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:45.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f97a006b040 0x7f97a006d500 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007fd0 tx=0x7f97b0009040 comp rx=0 tx=0).stop 2023-11-28T20:47:45.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c0146bc0 msgr2=0x7f97c0142780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:45.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c0146bc0 0x7f97c0142780 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f97b8036040 tx=0x7f97b8004a40 comp rx=0 tx=0).stop 2023-11-28T20:47:45.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 shutdown_connections 2023-11-28T20:47:45.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f97a006b040 0x7f97a006d500 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97c0146bc0 0x7f97c0142780 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c00b2b70 0x7f97c0135240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 --2- 172.21.15.45:0/1807480126 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97c006c0b0 0x7f97c00b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:45.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 >> 172.21.15.45:0/1807480126 conn(0x7f97c0066820 msgr2=0x7f97c00cbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:45.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 shutdown_connections 2023-11-28T20:47:45.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:45.914+0000 7f97c49d7640 1 -- 172.21.15.45:0/1807480126 wait complete. 2023-11-28T20:47:45.934 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-11-28T20:47:45.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:45.935 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-11-28T20:47:45.992 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_01_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_01_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_01_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_01_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:45.993 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:45.994 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_37_01_UTC 2023-11-28T20:47:45.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:45.995 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_37_01_UTC) 2023-11-28T20:47:46.062 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_38_01_UTC 2023-11-28T20:47:46.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.063 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_38_01_UTC) 2023-11-28T20:47:46.134 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_39_01_UTC 2023-11-28T20:47:46.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.134 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_39_01_UTC) 2023-11-28T20:47:46.209 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:46.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.209 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_40_01_UTC) 2023-11-28T20:47:46.319 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_41_01_UTC 2023-11-28T20:47:46.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.319 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_41_01_UTC) 2023-11-28T20:47:46.416 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:46.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.417 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_42_01_UTC) 2023-11-28T20:47:46.487 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:46.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.488 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_43_01_UTC) 2023-11-28T20:47:46.539 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:46.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.539 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_44_01_UTC) 2023-11-28T20:47:46.595 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:46.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.596 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_45_00_UTC) 2023-11-28T20:47:46.654 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:46.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.654 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_46_00_UTC) 2023-11-28T20:47:46.729 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:46.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.729 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-11-28-20_47_00_UTC) 2023-11-28T20:47:46.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:46.807 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-11-28T20:47:46.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2023-11-28T20:47:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.106+0000 7f4cf972e640 1 Processor -- start 2023-11-28T20:47:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.106+0000 7f4cf972e640 1 -- start start 2023-11-28T20:47:47.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.107+0000 7f4cf972e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4cf4155770 0x7f4cf415c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:47.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.107+0000 7f4cf972e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf40cdb20 con 0x7f4cf406c650 2023-11-28T20:47:47.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.108+0000 7f4cf972e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf415c580 con 0x7f4cf40cdfb0 2023-11-28T20:47:47.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.108+0000 7f4cf972e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf40ccf10 con 0x7f4cf4155770 2023-11-28T20:47:47.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.108+0000 7f4cf8f2d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4cf406c650 0x7f4cf40b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52356/0 (socket says 172.21.15.45:52356) 2023-11-28T20:47:47.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.108+0000 7f4cf8f2d640 1 -- 172.21.15.45:0/2639148970 learned_addr learned my addr 172.21.15.45:0/2639148970 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.108+0000 7f4cf3fff640 1 --2- 172.21.15.45:0/2639148970 >> v2:172.21.15.190:3300/0 conn(0x7f4cf4155770 0x7f4cf415c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.108+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378161581 0 0) 0x7f4cf40cdb20 con 0x7f4cf406c650 2023-11-28T20:47:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.108+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce0003520 con 0x7f4cf406c650 2023-11-28T20:47:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236353956 0 0) 0x7f4ce0003520 con 0x7f4cf406c650 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 >> v2:172.21.15.190:3300/0 conn(0x7f4cf4155770 msgr2=0x7f4cf415c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 --2- 172.21.15.45:0/2639148970 >> v2:172.21.15.190:3300/0 conn(0x7f4cf4155770 0x7f4cf415c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 >> v1:172.21.15.190:6789/0 conn(0x7f4cf40cdfb0 legacy=0x7f4cf406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf40cd1f0 con 0x7f4cf406c650 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4cec0019f0 con 0x7f4cf406c650 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 >> v1:172.21.15.45:6789/0 conn(0x7f4cf406c650 legacy=0x7f4cf40b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 --2- 172.21.15.45:0/2639148970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4ce0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 --2- 172.21.15.45:0/2639148970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ce00047c0 0x7f4ce0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 --2- 172.21.15.45:0/2639148970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ce00051d0 0x7f4ce000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf40ccf10 con 0x7f4ce00039c0 2023-11-28T20:47:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce000c120 con 0x7f4ce00051d0 2023-11-28T20:47:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce000c2e0 con 0x7f4ce00047c0 2023-11-28T20:47:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.109+0000 7f4cf3fff640 1 --2- 172.21.15.45:0/2639148970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4ce0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.110+0000 7f4cf3fff640 1 -- 172.21.15.45:0/2639148970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ce00047c0 msgr2=0x7f4ce0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.110+0000 7f4cf3fff640 1 --2- 172.21.15.45:0/2639148970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ce00047c0 0x7f4ce0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.110+0000 7f4cf3fff640 1 -- 172.21.15.45:0/2639148970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ce00051d0 msgr2=0x7f4ce000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.110+0000 7f4cf3fff640 1 --2- 172.21.15.45:0/2639148970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ce00051d0 0x7f4ce000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.110+0000 7f4cf3fff640 1 -- 172.21.15.45:0/2639148970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce0003520 con 0x7f4ce00039c0 2023-11-28T20:47:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.110+0000 7f4cf8f2d640 1 -- 172.21.15.45:0/2639148970 reap_dead start 2023-11-28T20:47:47.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.111+0000 7f4cf3fff640 1 --2- 172.21.15.45:0/2639148970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4ce0003de0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009a90 tx=0x7f4ce00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78dd54a0324a5c1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:47.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.111+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce402c6b0 con 0x7f4ce00039c0 2023-11-28T20:47:47.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.111+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce402c810 con 0x7f4ce00039c0 2023-11-28T20:47:47.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.111+0000 7f4cf37fe640 1 -- 172.21.15.45:0/2639148970 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce402cba0 con 0x7f4ce00039c0 2023-11-28T20:47:47.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.112+0000 7f4cf972e640 1 -- 172.21.15.45:0/2639148970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 msgr2=0x7f4ce0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.112+0000 7f4cf972e640 1 --2- 172.21.15.45:0/2639148970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4ce0003de0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009a90 tx=0x7f4ce00057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.112+0000 7f4cf972e640 1 -- 172.21.15.45:0/2639148970 shutdown_connections 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.112+0000 7f4cf972e640 1 --2- 172.21.15.45:0/2639148970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4ce0003de0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.112+0000 7f4cf972e640 1 -- 172.21.15.45:0/2639148970 >> 172.21.15.45:0/2639148970 conn(0x7f4cf40b1050 msgr2=0x7f4cf406a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.112+0000 7f4cf972e640 1 -- 172.21.15.45:0/2639148970 shutdown_connections 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.112+0000 7f4cf972e640 1 -- 172.21.15.45:0/2639148970 wait complete. 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.114+0000 7f4cf972e640 1 Processor -- start 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.114+0000 7f4cf972e640 1 -- start start 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.115+0000 7f4cf972e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4cf40adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.115+0000 7f4cf972e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cf40c80a0 0x7f4cf41423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.115+0000 7f4cf972e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cf4099a40 0x7f4cf415a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.115+0000 7f4cf972e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf40ac1f0 con 0x7f4ce00039c0 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.115+0000 7f4cf972e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf4142900 con 0x7f4cf4099a40 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.115+0000 7f4cf972e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf409a220 con 0x7f4cf40c80a0 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.115+0000 7f4cf3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4cf40adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.115+0000 7f4cf3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4cf40adc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56990/0 (socket says 172.21.15.45:56990) 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.115+0000 7f4cf3fff640 1 -- 172.21.15.45:0/3715739879 learned_addr learned my addr 172.21.15.45:0/3715739879 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cf8f2d640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cf4099a40 0x7f4cf415a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cebfff640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cf40c80a0 0x7f4cf41423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cf3fff640 1 -- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cf40c80a0 msgr2=0x7f4cf41423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cf3fff640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cf40c80a0 0x7f4cf41423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cf3fff640 1 -- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cf4099a40 msgr2=0x7f4cf415a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cf3fff640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cf4099a40 0x7f4cf415a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cf3fff640 1 -- 172.21.15.45:0/3715739879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf409a500 con 0x7f4ce00039c0 2023-11-28T20:47:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cebfff640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cf40c80a0 0x7f4cf41423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cf8f2d640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cf4099a40 0x7f4cf415a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.116+0000 7f4cf3fff640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4cf40adc30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4ce402c250 tx=0x7f4ce4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.117+0000 7f4cf17fa640 1 -- 172.21.15.45:0/3715739879 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4003810 con 0x7f4ce00039c0 2023-11-28T20:47:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.117+0000 7f4cf17fa640 1 -- 172.21.15.45:0/3715739879 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce4003970 con 0x7f4ce00039c0 2023-11-28T20:47:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.117+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf40abe80 con 0x7f4ce00039c0 2023-11-28T20:47:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.117+0000 7f4cf17fa640 1 -- 172.21.15.45:0/3715739879 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4003ce0 con 0x7f4ce00039c0 2023-11-28T20:47:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.117+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf40c8520 con 0x7f4ce00039c0 2023-11-28T20:47:47.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.118+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf409d0f0 con 0x7f4ce00039c0 2023-11-28T20:47:47.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.119+0000 7f4cf17fa640 1 -- 172.21.15.45:0/3715739879 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f4ce4040450 con 0x7f4ce00039c0 2023-11-28T20:47:47.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.120+0000 7f4cf17fa640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4cd406b040 0x7f4cd406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:47.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.120+0000 7f4cf17fa640 1 -- 172.21.15.45:0/3715739879 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7044+0+0 (secure 0 0 0) 0x7f4ce40abcb0 con 0x7f4ce00039c0 2023-11-28T20:47:47.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.121+0000 7f4cebfff640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4cd406b040 0x7f4cd406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:47.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.122+0000 7f4cebfff640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4cd406b040 0x7f4cd406d500 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4072010 tx=0x7f4cdc009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:47.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.124+0000 7f4cf17fa640 1 -- 172.21.15.45:0/3715739879 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ce4076590 con 0x7f4ce00039c0 2023-11-28T20:47:47.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.359+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cf40cd7b0 con 0x7f4cd406b040 2023-11-28T20:47:47.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.557+0000 7f4cf17fa640 1 -- 172.21.15.45:0/3715739879 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f4cdc001330 con 0x7f4cd406b040 2023-11-28T20:47:47.559 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-11-28T20:47:47.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4cd406b040 msgr2=0x7f4cd406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:47.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4cd406b040 0x7f4cd406d500 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4072010 tx=0x7f4cdc009040 comp rx=0 tx=0).stop 2023-11-28T20:47:47.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 msgr2=0x7f4cf40adc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:47.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4cf40adc30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4ce402c250 tx=0x7f4ce4004730 comp rx=0 tx=0).stop 2023-11-28T20:47:47.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 shutdown_connections 2023-11-28T20:47:47.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f4cd406b040 0x7f4cd406d500 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cf4099a40 0x7f4cf415a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cf40c80a0 0x7f4cf41423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 --2- 172.21.15.45:0/3715739879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ce00039c0 0x7f4cf40adc30 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:47.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.561+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 >> 172.21.15.45:0/3715739879 conn(0x7f4cf40b1050 msgr2=0x7f4cf415d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:47.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.562+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 shutdown_connections 2023-11-28T20:47:47.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:47.562+0000 7f4cf972e640 1 -- 172.21.15.45:0/3715739879 wait complete. 2023-11-28T20:47:47.584 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-11-28T20:47:47.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:47.584 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-11-28T20:47:47.641 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:47.641 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:47.641 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:47.641 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:47.641 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:47.641 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:47.641 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:47.641 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:47.642 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:47.642 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:47.642 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:47.643 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:47.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:47.643 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_37_00_UTC) 2023-11-28T20:47:47.717 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:47.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:47.718 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_38_00_UTC) 2023-11-28T20:47:47.796 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:47.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:47.796 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_39_00_UTC) 2023-11-28T20:47:47.874 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_40_00_UTC 2023-11-28T20:47:47.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:47.874 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_40_00_UTC) 2023-11-28T20:47:47.947 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:47.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:47.947 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_41_00_UTC) 2023-11-28T20:47:48.024 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_42_01_UTC 2023-11-28T20:47:48.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:48.024 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_42_01_UTC) 2023-11-28T20:47:48.104 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_43_00_UTC 2023-11-28T20:47:48.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:48.105 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_43_00_UTC) 2023-11-28T20:47:48.176 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_44_00_UTC 2023-11-28T20:47:48.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:48.177 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_44_00_UTC) 2023-11-28T20:47:48.252 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_45_00_UTC 2023-11-28T20:47:48.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:48.253 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_45_00_UTC) 2023-11-28T20:47:48.351 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_46_00_UTC 2023-11-28T20:47:48.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:48.351 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_46_00_UTC) 2023-11-28T20:47:48.414 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_47_00_UTC 2023-11-28T20:47:48.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:48.414 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-11-28-20_47_00_UTC) 2023-11-28T20:47:48.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:48.466 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-11-28T20:47:48.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2023-11-28T20:47:48.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:47:48.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:47:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.768+0000 7ff55d0d3640 1 Processor -- start 2023-11-28T20:47:48.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.768+0000 7ff55d0d3640 1 -- start start 2023-11-28T20:47:48.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.770+0000 7ff55d0d3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff5580ae030 0x7ff55815daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:48.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.770+0000 7ff55d0d3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff5580c9dc0 0x7ff5580ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:48.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.770+0000 7ff55d0d3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580cd6b0 con 0x7ff558159780 2023-11-28T20:47:48.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.770+0000 7ff55d0d3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5580ce330 con 0x7ff5580c9dc0 2023-11-28T20:47:48.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.770+0000 7ff55d0d3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5580cd990 con 0x7ff5580ae030 2023-11-28T20:47:48.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.770+0000 7ff55759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff5580c9dc0 0x7ff5580ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:48.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.770+0000 7ff55759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff5580c9dc0 0x7ff5580ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57006/0 (socket says 172.21.15.45:57006) 2023-11-28T20:47:48.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.770+0000 7ff55759e640 1 -- 172.21.15.45:0/3625263575 learned_addr learned my addr 172.21.15.45:0/3625263575 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:48.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.770+0000 7ff55659c640 1 --2- 172.21.15.45:0/3625263575 >> v2:172.21.15.190:3300/0 conn(0x7ff5580ae030 0x7ff55815daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:48.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.771+0000 7ff55759e640 1 -- 172.21.15.45:0/3625263575 >> v2:172.21.15.190:3300/0 conn(0x7ff5580ae030 msgr2=0x7ff55815daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:48.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.771+0000 7ff55759e640 1 --2- 172.21.15.45:0/3625263575 >> v2:172.21.15.190:3300/0 conn(0x7ff5580ae030 0x7ff55815daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:48.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.771+0000 7ff55759e640 1 -- 172.21.15.45:0/3625263575 >> v1:172.21.15.190:6789/0 conn(0x7ff558159780 legacy=0x7ff55815d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:48.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.771+0000 7ff55759e640 1 -- 172.21.15.45:0/3625263575 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580cdc70 con 0x7ff5580c9dc0 2023-11-28T20:47:48.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.772+0000 7ff55759e640 1 --2- 172.21.15.45:0/3625263575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 0x7ff5580ca1e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff55815d2d0 tx=0x7ff55002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a01fc8d08cf8416 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:48.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.772+0000 7ff555d9b640 1 -- 172.21.15.45:0/3625263575 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550030860 con 0x7ff5580c9dc0 2023-11-28T20:47:48.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff555d9b640 1 -- 172.21.15.45:0/3625263575 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5500309c0 con 0x7ff5580c9dc0 2023-11-28T20:47:48.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff555d9b640 1 -- 172.21.15.45:0/3625263575 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550030d50 con 0x7ff5580c9dc0 2023-11-28T20:47:48.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff55d0d3640 1 -- 172.21.15.45:0/3625263575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 msgr2=0x7ff5580ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:48.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff55d0d3640 1 --2- 172.21.15.45:0/3625263575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 0x7ff5580ca1e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff55815d2d0 tx=0x7ff55002b9a0 comp rx=0 tx=0).stop 2023-11-28T20:47:48.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff55d0d3640 1 -- 172.21.15.45:0/3625263575 shutdown_connections 2023-11-28T20:47:48.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff55d0d3640 1 --2- 172.21.15.45:0/3625263575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 0x7ff5580ca1e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:48.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff55d0d3640 1 --2- 172.21.15.45:0/3625263575 >> v2:172.21.15.190:3300/0 conn(0x7ff5580ae030 0x7ff55815daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:48.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff55d0d3640 1 -- 172.21.15.45:0/3625263575 >> 172.21.15.45:0/3625263575 conn(0x7ff5580ced60 msgr2=0x7ff5580cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:48.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff55d0d3640 1 -- 172.21.15.45:0/3625263575 shutdown_connections 2023-11-28T20:47:48.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.773+0000 7ff55d0d3640 1 -- 172.21.15.45:0/3625263575 wait complete. 2023-11-28T20:47:48.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.775+0000 7ff55d0d3640 1 Processor -- start 2023-11-28T20:47:48.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.775+0000 7ff55d0d3640 1 -- start start 2023-11-28T20:47:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.776+0000 7ff55d0d3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580ae030 0x7ff5580c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.776+0000 7ff55d0d3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 0x7ff5580c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.776+0000 7ff55d0d3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff558159780 0x7ff55809bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.776+0000 7ff55d0d3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580c7fe0 con 0x7ff5580c9dc0 2023-11-28T20:47:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.776+0000 7ff55d0d3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580c7610 con 0x7ff5580ae030 2023-11-28T20:47:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.776+0000 7ff55d0d3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580c67b0 con 0x7ff558159780 2023-11-28T20:47:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 0x7ff5580c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 0x7ff5580c8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57014/0 (socket says 172.21.15.45:57014) 2023-11-28T20:47:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55659c640 1 -- 172.21.15.45:0/2830672209 learned_addr learned my addr 172.21.15.45:0/2830672209 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55759e640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff558159780 0x7ff55809bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff556d9d640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580ae030 0x7ff5580c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55659c640 1 -- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff558159780 msgr2=0x7ff55809bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55659c640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff558159780 0x7ff55809bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55659c640 1 -- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580ae030 msgr2=0x7ff5580c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55659c640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580ae030 0x7ff5580c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55659c640 1 -- 172.21.15.45:0/2830672209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5500094d0 con 0x7ff5580c9dc0 2023-11-28T20:47:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55759e640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff558159780 0x7ff55809bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.777+0000 7ff55659c640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 0x7ff5580c8610 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff54000e870 tx=0x7ff54000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.778+0000 7ff54f7fe640 1 -- 172.21.15.45:0/2830672209 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5400094d0 con 0x7ff5580c9dc0 2023-11-28T20:47:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.778+0000 7ff54f7fe640 1 -- 172.21.15.45:0/2830672209 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff540004240 con 0x7ff5580c9dc0 2023-11-28T20:47:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.778+0000 7ff54f7fe640 1 -- 172.21.15.45:0/2830672209 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54000c730 con 0x7ff5580c9dc0 2023-11-28T20:47:48.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.778+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5580c6a00 con 0x7ff5580c9dc0 2023-11-28T20:47:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.778+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5580c6fc0 con 0x7ff5580c9dc0 2023-11-28T20:47:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.779+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5580c77c0 con 0x7ff5580c9dc0 2023-11-28T20:47:48.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.780+0000 7ff54f7fe640 1 -- 172.21.15.45:0/2830672209 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7ff54000c890 con 0x7ff5580c9dc0 2023-11-28T20:47:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.781+0000 7ff54f7fe640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff53806b040 0x7ff53806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.781+0000 7ff54f7fe640 1 -- 172.21.15.45:0/2830672209 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6996+0+0 (secure 0 0 0) 0x7ff54008a1b0 con 0x7ff5580c9dc0 2023-11-28T20:47:48.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.782+0000 7ff556d9d640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff53806b040 0x7ff53806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:48.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.783+0000 7ff556d9d640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff53806b040 0x7ff53806d500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff548007fd0 tx=0x7ff548009040 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:48.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:48.786+0000 7ff54f7fe640 1 -- 172.21.15.45:0/2830672209 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5580c77c0 con 0x7ff5580c9dc0 2023-11-28T20:47:49.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.019+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 --> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff558159b60 con 0x7ff53806b040 2023-11-28T20:47:49.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.215+0000 7ff54f7fe640 1 -- 172.21.15.45:0/2830672209 <== mgr.4856 v2:172.21.15.190:6832/2725659217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff548006b50 con 0x7ff53806b040 2023-11-28T20:47:49.217 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-11-28T20:47:49.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff53806b040 msgr2=0x7ff53806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:49.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff53806b040 0x7ff53806d500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff548007fd0 tx=0x7ff548009040 comp rx=0 tx=0).stop 2023-11-28T20:47:49.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 msgr2=0x7ff5580c8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:49.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 0x7ff5580c8610 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff54000e870 tx=0x7ff54000ed30 comp rx=0 tx=0).stop 2023-11-28T20:47:49.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 shutdown_connections 2023-11-28T20:47:49.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7ff53806b040 0x7ff53806d500 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:49.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff558159780 0x7ff55809bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:49.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580c9dc0 0x7ff5580c8610 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:49.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 --2- 172.21.15.45:0/2830672209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580ae030 0x7ff5580c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:49.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 >> 172.21.15.45:0/2830672209 conn(0x7ff5580ced60 msgr2=0x7ff5580afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:49.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 shutdown_connections 2023-11-28T20:47:49.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:49.219+0000 7ff55d0d3640 1 -- 172.21.15.45:0/2830672209 wait complete. 2023-11-28T20:47:49.246 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-11-28T20:47:49.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.247 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-11-28T20:47:49.335 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:49.335 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:49.336 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:49.336 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:49.336 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:49.336 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:49.336 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:49.336 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:49.337 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_45_01_UTC 2023-11-28T20:47:49.337 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:49.337 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_47_01_UTC 2023-11-28T20:47:49.338 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_37_00_UTC 2023-11-28T20:47:49.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.339 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_37_00_UTC) 2023-11-28T20:47:49.459 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_38_00_UTC 2023-11-28T20:47:49.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.459 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_38_00_UTC) 2023-11-28T20:47:49.519 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_39_00_UTC 2023-11-28T20:47:49.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.519 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_39_00_UTC) 2023-11-28T20:47:49.579 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_40_01_UTC 2023-11-28T20:47:49.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.580 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_40_01_UTC) 2023-11-28T20:47:49.643 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_41_00_UTC 2023-11-28T20:47:49.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.643 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_41_00_UTC) 2023-11-28T20:47:49.720 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_42_00_UTC 2023-11-28T20:47:49.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.721 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_42_00_UTC) 2023-11-28T20:47:49.779 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_43_01_UTC 2023-11-28T20:47:49.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.779 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_43_01_UTC) 2023-11-28T20:47:49.837 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_44_01_UTC 2023-11-28T20:47:49.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.837 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_44_01_UTC) 2023-11-28T20:47:49.905 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_45_01_UTC 2023-11-28T20:47:49.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.905 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_45_01_UTC) 2023-11-28T20:47:49.982 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_46_01_UTC 2023-11-28T20:47:49.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:49.982 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_46_01_UTC) 2023-11-28T20:47:50.062 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_47_01_UTC 2023-11-28T20:47:50.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:50.063 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-11-28-20_47_01_UTC) 2023-11-28T20:47:50.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:50.144 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-11-28T20:47:50.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T20:47:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.475+0000 7f35d74c2640 1 Processor -- start 2023-11-28T20:47:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.475+0000 7f35d74c2640 1 -- start start 2023-11-28T20:47:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.476+0000 7f35d74c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d0157c10 0x7f35d006da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35d74c2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f35d015e4e0 0x7f35d006a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35d74c2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d006ca60 con 0x7f35d006a500 2023-11-28T20:47:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35d74c2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d015eb40 con 0x7f35d015e4e0 2023-11-28T20:47:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35d74c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0157ff0 con 0x7f35d0157c10 2023-11-28T20:47:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35d4a5d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f35d006a500 0x7f35d015ddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52364/0 (socket says 172.21.15.45:52364) 2023-11-28T20:47:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35d4a5d640 1 -- 172.21.15.45:0/4031406372 learned_addr learned my addr 172.21.15.45:0/4031406372 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35d5a5f640 1 --2- 172.21.15.45:0/4031406372 >> v2:172.21.15.190:3300/0 conn(0x7f35d015e4e0 0x7f35d006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:50.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35d525e640 1 --2- 172.21.15.45:0/4031406372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d0157c10 0x7f35d006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35c7fff640 1 -- 172.21.15.45:0/4031406372 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524160782 0 0) 0x7f35d006ca60 con 0x7f35d006a500 2023-11-28T20:47:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.477+0000 7f35c7fff640 1 -- 172.21.15.45:0/4031406372 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc003520 con 0x7f35d006a500 2023-11-28T20:47:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.478+0000 7f35d5a5f640 1 -- 172.21.15.45:0/4031406372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d0157c10 msgr2=0x7f35d006da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:50.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.478+0000 7f35d5a5f640 1 --2- 172.21.15.45:0/4031406372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d0157c10 0x7f35d006da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.478+0000 7f35d5a5f640 1 -- 172.21.15.45:0/4031406372 >> v1:172.21.15.45:6789/0 conn(0x7f35d006a500 legacy=0x7f35d015ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:50.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.478+0000 7f35d5a5f640 1 -- 172.21.15.45:0/4031406372 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d006aea0 con 0x7f35d015e4e0 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.479+0000 7f35d5a5f640 1 --2- 172.21.15.45:0/4031406372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 0x7f35d006a960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009d90 tx=0x7f35c802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96f30636fe705747 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.479+0000 7f35c7fff640 1 -- 172.21.15.45:0/4031406372 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c802cc60 con 0x7f35d015e4e0 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.479+0000 7f35c7fff640 1 -- 172.21.15.45:0/4031406372 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c802cdc0 con 0x7f35d015e4e0 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.479+0000 7f35c7fff640 1 -- 172.21.15.45:0/4031406372 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8034660 con 0x7f35d015e4e0 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.480+0000 7f35d74c2640 1 -- 172.21.15.45:0/4031406372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 msgr2=0x7f35d006a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.480+0000 7f35d74c2640 1 --2- 172.21.15.45:0/4031406372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 0x7f35d006a960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009d90 tx=0x7f35c802beb0 comp rx=0 tx=0).stop 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.480+0000 7f35d74c2640 1 -- 172.21.15.45:0/4031406372 shutdown_connections 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.480+0000 7f35d74c2640 1 --2- 172.21.15.45:0/4031406372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 0x7f35d006a960 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.480+0000 7f35d74c2640 1 --2- 172.21.15.45:0/4031406372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d0157c10 0x7f35d006da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.480+0000 7f35d74c2640 1 -- 172.21.15.45:0/4031406372 >> 172.21.15.45:0/4031406372 conn(0x7f35d00ced60 msgr2=0x7f35d00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:50.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.480+0000 7f35d74c2640 1 -- 172.21.15.45:0/4031406372 shutdown_connections 2023-11-28T20:47:50.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.480+0000 7f35d74c2640 1 -- 172.21.15.45:0/4031406372 wait complete. 2023-11-28T20:47:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.482+0000 7f35d74c2640 1 Processor -- start 2023-11-28T20:47:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.482+0000 7f35d74c2640 1 -- start start 2023-11-28T20:47:50.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d74c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d006a500 0x7f35d01533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d74c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35d0157c10 0x7f35d015a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d74c2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 0x7f35d015a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d74c2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0155270 con 0x7f35d0157c10 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d74c2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0153910 con 0x7f35d015e4e0 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d74c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00ace80 con 0x7f35d006a500 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d4a5d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35d0157c10 0x7f35d015a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d4a5d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35d0157c10 0x7f35d015a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57018/0 (socket says 172.21.15.45:57018) 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d4a5d640 1 -- 172.21.15.45:0/2845271363 learned_addr learned my addr 172.21.15.45:0/2845271363 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d525e640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d006a500 0x7f35d01533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d5a5f640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 0x7f35d015a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d4a5d640 1 -- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d006a500 msgr2=0x7f35d01533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:50.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d4a5d640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d006a500 0x7f35d01533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d4a5d640 1 -- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 msgr2=0x7f35d015a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d4a5d640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 0x7f35d015a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.483+0000 7f35d4a5d640 1 -- 172.21.15.45:0/2845271363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8009a70 con 0x7f35d0157c10 2023-11-28T20:47:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.484+0000 7f35d5a5f640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 0x7f35d015a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:47:50.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.484+0000 7f35d4a5d640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35d0157c10 0x7f35d015a440 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f35c000c500 tx=0x7f35c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.484+0000 7f35c5ffb640 1 -- 172.21.15.45:0/2845271363 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c0005ce0 con 0x7f35d0157c10 2023-11-28T20:47:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.484+0000 7f35c5ffb640 1 -- 172.21.15.45:0/2845271363 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c0005e40 con 0x7f35d0157c10 2023-11-28T20:47:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.484+0000 7f35c5ffb640 1 -- 172.21.15.45:0/2845271363 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c0007330 con 0x7f35d0157c10 2023-11-28T20:47:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.484+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d00ac270 con 0x7f35d0157c10 2023-11-28T20:47:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.484+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d00ac830 con 0x7f35d0157c10 2023-11-28T20:47:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.486+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d00733d0 con 0x7f35d0157c10 2023-11-28T20:47:50.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.487+0000 7f35c5ffb640 1 -- 172.21.15.45:0/2845271363 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 89419+0+0 (secure 0 0 0) 0x7f35c0007550 con 0x7f35d0157c10 2023-11-28T20:47:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.487+0000 7f35c5ffb640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f35b406b040 0x7f35b406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:50.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.488+0000 7f35c5ffb640 1 -- 172.21.15.45:0/2845271363 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6916+0+0 (secure 0 0 0) 0x7f35c0089460 con 0x7f35d0157c10 2023-11-28T20:47:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.488+0000 7f35d525e640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f35b406b040 0x7f35b406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:50.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.489+0000 7f35d525e640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f35b406b040 0x7f35b406d500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009b40 tx=0x7f35cc009190 comp rx=0 tx=0).ready entity=mgr.4856 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.492+0000 7f35c5ffb640 1 -- 172.21.15.45:0/2845271363 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35c0053f30 con 0x7f35d0157c10 2023-11-28T20:47:50.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:50.738+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f35d00c9c30 con 0x7f35d0157c10 2023-11-28T20:47:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.317+0000 7f35c5ffb640 1 -- 172.21.15.45:0/2845271363 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) v1 ==== 93+0+0 (secure 0 0 0) 0x7f35c0057550 con 0x7f35d0157c10 2023-11-28T20:47:51.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.324+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f35b406b040 msgr2=0x7f35b406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:51.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.324+0000 7f35d74c2640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f35b406b040 0x7f35b406d500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009b40 tx=0x7f35cc009190 comp rx=0 tx=0).stop 2023-11-28T20:47:51.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.324+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35d0157c10 msgr2=0x7f35d015a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:51.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.324+0000 7f35d74c2640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35d0157c10 0x7f35d015a440 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f35c000c500 tx=0x7f35c000c9c0 comp rx=0 tx=0).stop 2023-11-28T20:47:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.325+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 shutdown_connections 2023-11-28T20:47:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.325+0000 7f35d74c2640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f35b406b040 0x7f35b406d500 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.325+0000 7f35d74c2640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35d015e4e0 0x7f35d015a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.325+0000 7f35d74c2640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35d0157c10 0x7f35d015a440 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.325+0000 7f35d74c2640 1 --2- 172.21.15.45:0/2845271363 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35d006a500 0x7f35d01533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.325+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 >> 172.21.15.45:0/2845271363 conn(0x7f35d00ced60 msgr2=0x7f35d00cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.328+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 shutdown_connections 2023-11-28T20:47:51.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:51.329+0000 7f35d74c2640 1 -- 172.21.15.45:0/2845271363 wait complete. 2023-11-28T20:47:51.365 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:47:51.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:47:51.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:47:51.620 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:47:51.621 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:47:51.621 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:47:51.661 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:47:51.661 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:47:51.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:47:51.941 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:51.939+0000 7fb76dc11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:47:51.945 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:51.944+0000 7f0307c3a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:47:51.999 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:51.996+0000 7fb76dc11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:47:52.004 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:52.003+0000 7f0307c3a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:47:52.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.015+0000 7f140aa4f640 1 Processor -- start 2023-11-28T20:47:52.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.015+0000 7f140aa4f640 1 -- start start 2023-11-28T20:47:52.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.017+0000 7f140aa4f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1404070b20 0x7f1404070f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.017+0000 7f140aa4f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14040714c0 con 0x7f1404160c80 2023-11-28T20:47:52.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.017+0000 7f140aa4f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14040ce930 con 0x7f14040d1f20 2023-11-28T20:47:52.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.017+0000 7f140aa4f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14040d1440 con 0x7f1404070b20 2023-11-28T20:47:52.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.017+0000 7f1403fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1404160c80 0x7f14040b6640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52394/0 (socket says 172.21.15.45:52394) 2023-11-28T20:47:52.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.017+0000 7f1403fff640 1 -- 172.21.15.45:0/3485280056 learned_addr learned my addr 172.21.15.45:0/3485280056 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f13fb7fe640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1404070b20 0x7f1404070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484841641 0 0) 0x7f14040714c0 con 0x7f1404160c80 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f0003520 con 0x7f1404160c80 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667469189 0 0) 0x7f13f0003520 con 0x7f1404160c80 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1404070b20 msgr2=0x7f1404070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1404070b20 0x7f1404070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 >> v1:172.21.15.190:6789/0 conn(0x7f14040d1f20 legacy=0x7f14040cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14040d1720 con 0x7f1404160c80 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f13f40019f0 con 0x7f1404160c80 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 >> v1:172.21.15.45:6789/0 conn(0x7f1404160c80 legacy=0x7f14040b6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1404070b20 0x7f13f00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13f0003ee0 0x7f13f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13f0004d00 0x7f13f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14040d1440 con 0x7f13f0004d00 2023-11-28T20:47:52.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f000ba30 con 0x7f13f0003ee0 2023-11-28T20:47:52.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.018+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f000bbf0 con 0x7f1404070b20 2023-11-28T20:47:52.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.019+0000 7f1408fec640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13f0004d00 0x7f13f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.019+0000 7f13fb7fe640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13f0003ee0 0x7f13f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.019+0000 7f1403fff640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1404070b20 0x7f13f00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.019+0000 7f1408fec640 1 -- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1404070b20 msgr2=0x7f13f00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:52.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.019+0000 7f1408fec640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1404070b20 0x7f13f00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.019+0000 7f1408fec640 1 -- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13f0003ee0 msgr2=0x7f13f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:52.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.019+0000 7f1408fec640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13f0003ee0 0x7f13f00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.019+0000 7f1408fec640 1 -- 172.21.15.45:0/3485280056 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f0003520 con 0x7f13f0004d00 2023-11-28T20:47:52.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.020+0000 7f1408fec640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13f0004d00 0x7f13f0005180 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f13fc001860 tx=0x7f13f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cbd2d4e3d15d63 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:52.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.021+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc033410 con 0x7f13f0004d00 2023-11-28T20:47:52.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.021+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13fc033590 con 0x7f13f0004d00 2023-11-28T20:47:52.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.021+0000 7f14037fe640 1 -- 172.21.15.45:0/3485280056 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc033940 con 0x7f13f0004d00 2023-11-28T20:47:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.022+0000 7f140aa4f640 1 -- 172.21.15.45:0/3485280056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13f0004d00 msgr2=0x7f13f0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.022+0000 7f140aa4f640 1 --2- 172.21.15.45:0/3485280056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13f0004d00 0x7f13f0005180 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f13fc001860 tx=0x7f13f00057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.022+0000 7f1408fec640 1 -- 172.21.15.45:0/3485280056 reap_dead start 2023-11-28T20:47:52.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.022+0000 7f140aa4f640 1 -- 172.21.15.45:0/3485280056 shutdown_connections 2023-11-28T20:47:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.022+0000 7f140aa4f640 1 -- 172.21.15.45:0/3485280056 >> 172.21.15.45:0/3485280056 conn(0x7f14040cfcb0 msgr2=0x7f14040d00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.022+0000 7f140aa4f640 1 -- 172.21.15.45:0/3485280056 shutdown_connections 2023-11-28T20:47:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.022+0000 7f140aa4f640 1 -- 172.21.15.45:0/3485280056 wait complete. 2023-11-28T20:47:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.024+0000 7f140aa4f640 1 Processor -- start 2023-11-28T20:47:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.024+0000 7f140aa4f640 1 -- start start 2023-11-28T20:47:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f140aa4f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14041518d0 0x7f1404155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f140aa4f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1404151cb0 0x7f14041610a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f140aa4f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14041615e0 0x7f1404153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f140aa4f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1404161a70 con 0x7f14041615e0 2023-11-28T20:47:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f140aa4f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1404153690 con 0x7f1404151cb0 2023-11-28T20:47:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f140aa4f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1404152da0 con 0x7f14041518d0 2023-11-28T20:47:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f1408fec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14041615e0 0x7f1404153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f1408fec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14041615e0 0x7f1404153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57052/0 (socket says 172.21.15.45:57052) 2023-11-28T20:47:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f1408fec640 1 -- 172.21.15.45:0/3503366514 learned_addr learned my addr 172.21.15.45:0/3503366514 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f1403fff640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14041518d0 0x7f1404155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.025+0000 7f13fb7fe640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1404151cb0 0x7f14041610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f1408fec640 1 -- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14041518d0 msgr2=0x7f1404155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f1408fec640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14041518d0 0x7f1404155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f1408fec640 1 -- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1404151cb0 msgr2=0x7f14041610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f1408fec640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1404151cb0 0x7f14041610a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f1408fec640 1 -- 172.21.15.45:0/3503366514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14040cb030 con 0x7f14041615e0 2023-11-28T20:47:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f1408fec640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14041615e0 0x7f1404153b20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f13fc001950 tx=0x7f13fc001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f14017fa640 1 -- 172.21.15.45:0/3503366514 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc033410 con 0x7f14041615e0 2023-11-28T20:47:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f14017fa640 1 -- 172.21.15.45:0/3503366514 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13fc006d40 con 0x7f14041615e0 2023-11-28T20:47:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1404153040 con 0x7f14041615e0 2023-11-28T20:47:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f14017fa640 1 -- 172.21.15.45:0/3503366514 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc005270 con 0x7f14041615e0 2023-11-28T20:47:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.026+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1404155e70 con 0x7f14041615e0 2023-11-28T20:47:52.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.027+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14040733d0 con 0x7f14041615e0 2023-11-28T20:47:52.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.029+0000 7f14017fa640 1 -- 172.21.15.45:0/3503366514 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 89402+0+0 (secure 0 0 0) 0x7f13fc03a050 con 0x7f14041615e0 2023-11-28T20:47:52.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.030+0000 7f14017fa640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f13e406aff0 0x7f13e406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.030+0000 7f14017fa640 1 -- 172.21.15.45:0/3503366514 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7060+0+0 (secure 0 0 0) 0x7f13fc0aa6f0 con 0x7f14041615e0 2023-11-28T20:47:52.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.030+0000 7f1403fff640 1 -- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f13e406aff0 msgr2=0x7f13e406d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:52.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.030+0000 7f1403fff640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f13e406aff0 0x7f13e406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:47:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.035+0000 7f14017fa640 1 -- 172.21.15.45:0/3503366514 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13fc0750b0 con 0x7f14041615e0 2023-11-28T20:47:52.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.228+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14041537d0 con 0x7f14041615e0 2023-11-28T20:47:52.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.230+0000 7f14017fa640 1 -- 172.21.15.45:0/3503366514 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18290 (secure 0 0 0) 0x7f13fc0786d0 con 0x7f14041615e0 2023-11-28T20:47:52.232 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":56,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:47:51.249726+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:35:33.849797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-28T20:35:34.783808+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":232,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1866894774":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3717946281":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6833/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6832/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/1813745990":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3262798906":"2023-11-29T20:35:55.110774+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":2,"length":4},{"begin":9,"length":1},{"begin":11,"length":1},{"begin":13,"length":1},{"begin":17,"length":1},{"begin":20,"length":1},{"begin":22,"length":1},{"begin":26,"length":2},{"begin":30,"length":1},{"begin":34,"length":1},{"begin":36,"length":5},{"begin":43,"length":2},{"begin":46,"length":2},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":2},{"begin":60,"length":2},{"begin":65,"length":5},{"begin":75,"length":3},{"begin":79,"length":1},{"begin":84,"length":2},{"begin":87,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":94,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":101,"length":2},{"begin":106,"length":1},{"begin":108,"length":1},{"begin":110,"length":2},{"begin":116,"length":1},{"begin":189,"length":44}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":9,"length":1},{"begin":22,"length":1},{"begin":30,"length":1},{"begin":44,"length":1},{"begin":53,"length":1},{"begin":60,"length":1},{"begin":76,"length":1},{"begin":85,"length":1},{"begin":96,"length":1},{"begin":106,"length":1},{"begin":110,"length":1},{"begin":222,"length":11}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:47:52.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.231+0000 7f1403fff640 1 -- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f13e406aff0 msgr2=0x7f13e406d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:52.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.231+0000 7f1403fff640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f13e406aff0 0x7f13e406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:47:52.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.233+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f13e406aff0 msgr2=0x7f13e406d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:47:52.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.233+0000 7f140aa4f640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f13e406aff0 0x7f13e406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.233+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14041615e0 msgr2=0x7f1404153b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.233+0000 7f140aa4f640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14041615e0 0x7f1404153b20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f13fc001950 tx=0x7f13fc001230 comp rx=0 tx=0).stop 2023-11-28T20:47:52.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.234+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 shutdown_connections 2023-11-28T20:47:52.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.234+0000 7f140aa4f640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f13e406aff0 0x7f13e406d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.234+0000 7f140aa4f640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14041615e0 0x7f1404153b20 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.234+0000 7f140aa4f640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1404151cb0 0x7f14041610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.234+0000 7f140aa4f640 1 --2- 172.21.15.45:0/3503366514 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14041518d0 0x7f1404155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.234+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 >> 172.21.15.45:0/3503366514 conn(0x7f14040cfcb0 msgr2=0x7f14040b6640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:52.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.234+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 shutdown_connections 2023-11-28T20:47:52.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.234+0000 7f140aa4f640 1 -- 172.21.15.45:0/3503366514 wait complete. 2023-11-28T20:47:52.255 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:35:33.849797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:35:34.783808+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 232, 'snap_epoch': 56, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:47:52.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:47:52.451 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.449+0000 7fb76dc11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:47:52.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.478+0000 7fd433fff640 1 Processor -- start 2023-11-28T20:47:52.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.478+0000 7fd433fff640 1 -- start start 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.480+0000 7fd433fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd43415d580 0x7fd4340cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.480+0000 7fd433fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4340d1e40 0x7fd4340ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.480+0000 7fd433fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4340d0490 con 0x7fd4340b2500 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.480+0000 7fd433fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4340d1360 con 0x7fd4340d1e40 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.480+0000 7fd433fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340d1640 con 0x7fd43415d580 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.480+0000 7fd4327fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4340d1e40 0x7fd4340ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.480+0000 7fd4327fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4340d1e40 0x7fd4340ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57068/0 (socket says 172.21.15.45:57068) 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.480+0000 7fd4327fc640 1 -- 172.21.15.45:0/1379586471 learned_addr learned my addr 172.21.15.45:0/1379586471 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.481+0000 7fd432ffd640 1 --2- 172.21.15.45:0/1379586471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd43415d580 0x7fd4340cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.481+0000 7fd4327fc640 1 -- 172.21.15.45:0/1379586471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd43415d580 msgr2=0x7fd4340cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:52.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.481+0000 7fd4327fc640 1 --2- 172.21.15.45:0/1379586471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd43415d580 0x7fd4340cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.481+0000 7fd4327fc640 1 -- 172.21.15.45:0/1379586471 >> v1:172.21.15.190:6789/0 conn(0x7fd4340b2500 legacy=0x7fd4340b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:52.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.481+0000 7fd4327fc640 1 -- 172.21.15.45:0/1379586471 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4340d1920 con 0x7fd4340d1e40 2023-11-28T20:47:52.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.482+0000 7fd4327fc640 1 --2- 172.21.15.45:0/1379586471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 0x7fd4340ce040 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd424009850 tx=0x7fd42402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e448b52231b87e20 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:52.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.482+0000 7fd431ffb640 1 -- 172.21.15.45:0/1379586471 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42402e070 con 0x7fd4340d1e40 2023-11-28T20:47:52.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.482+0000 7fd431ffb640 1 -- 172.21.15.45:0/1379586471 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd424004030 con 0x7fd4340d1e40 2023-11-28T20:47:52.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.482+0000 7fd431ffb640 1 -- 172.21.15.45:0/1379586471 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424004420 con 0x7fd4340d1e40 2023-11-28T20:47:52.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.483+0000 7fd433fff640 1 -- 172.21.15.45:0/1379586471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 msgr2=0x7fd4340ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.483+0000 7fd433fff640 1 --2- 172.21.15.45:0/1379586471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 0x7fd4340ce040 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd424009850 tx=0x7fd42402b840 comp rx=0 tx=0).stop 2023-11-28T20:47:52.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.483+0000 7fd433fff640 1 -- 172.21.15.45:0/1379586471 shutdown_connections 2023-11-28T20:47:52.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.483+0000 7fd433fff640 1 --2- 172.21.15.45:0/1379586471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 0x7fd4340ce040 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.483+0000 7fd433fff640 1 --2- 172.21.15.45:0/1379586471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd43415d580 0x7fd4340cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.483+0000 7fd433fff640 1 -- 172.21.15.45:0/1379586471 >> 172.21.15.45:0/1379586471 conn(0x7fd43406a500 msgr2=0x7fd43406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:52.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.483+0000 7fd433fff640 1 -- 172.21.15.45:0/1379586471 shutdown_connections 2023-11-28T20:47:52.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.484+0000 7fd433fff640 1 -- 172.21.15.45:0/1379586471 wait complete. 2023-11-28T20:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.486+0000 7fd433fff640 1 Processor -- start 2023-11-28T20:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.486+0000 7fd433fff640 1 -- start start 2023-11-28T20:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.488+0000 7fd433fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4340b2500 0x7fd43415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.488+0000 7fd433fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 0x7fd4341601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.488+0000 7fd433fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd43415d580 0x7fd4341f4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.488+0000 7fd433fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43415e2f0 con 0x7fd4340d1e40 2023-11-28T20:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.488+0000 7fd433fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd434160980 con 0x7fd43415d580 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.488+0000 7fd433fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4341606e0 con 0x7fd4340b2500 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.488+0000 7fd4327fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 0x7fd4341601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.488+0000 7fd4327fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 0x7fd4341601a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57078/0 (socket says 172.21.15.45:57078) 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.488+0000 7fd4327fc640 1 -- 172.21.15.45:0/1393819760 learned_addr learned my addr 172.21.15.45:0/1393819760 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd432ffd640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4340b2500 0x7fd43415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd4337fe640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd43415d580 0x7fd4341f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd4327fc640 1 -- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4340b2500 msgr2=0x7fd43415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd4327fc640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4340b2500 0x7fd43415ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd4327fc640 1 -- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd43415d580 msgr2=0x7fd4341f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd4327fc640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd43415d580 0x7fd4341f4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd4327fc640 1 -- 172.21.15.45:0/1393819760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424009460 con 0x7fd4340d1e40 2023-11-28T20:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd4337fe640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd43415d580 0x7fd4341f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd4327fc640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 0x7fd4341601a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd424035040 tx=0x7fd4240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd42bfff640 1 -- 172.21.15.45:0/1393819760 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424003b20 con 0x7fd4340d1e40 2023-11-28T20:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd42bfff640 1 -- 172.21.15.45:0/1393819760 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd424003c80 con 0x7fd4340d1e40 2023-11-28T20:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd42bfff640 1 -- 172.21.15.45:0/1393819760 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424006040 con 0x7fd4340d1e40 2023-11-28T20:47:52.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.489+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4340b6580 con 0x7fd4340d1e40 2023-11-28T20:47:52.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.490+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4340b6b40 con 0x7fd4340d1e40 2023-11-28T20:47:52.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.491+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd434071b30 con 0x7fd4340d1e40 2023-11-28T20:47:52.493 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:52.492+0000 7f0307c3a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:47:52.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.492+0000 7fd42bfff640 1 -- 172.21.15.45:0/1393819760 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 89402+0+0 (secure 0 0 0) 0x7fd4240061a0 con 0x7fd4340d1e40 2023-11-28T20:47:52.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.493+0000 7fd42bfff640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd41406aff0 0x7fd41406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.494+0000 7fd42bfff640 1 -- 172.21.15.45:0/1393819760 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7060+0+0 (secure 0 0 0) 0x7fd4240abfa0 con 0x7fd4340d1e40 2023-11-28T20:47:52.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.494+0000 7fd432ffd640 1 -- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd41406aff0 msgr2=0x7fd41406d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:52.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.494+0000 7fd432ffd640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd41406aff0 0x7fd41406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:47:52.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.498+0000 7fd42bfff640 1 -- 172.21.15.45:0/1393819760 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd424076760 con 0x7fd4340d1e40 2023-11-28T20:47:52.504 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.501+0000 7fb76dc11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:47:52.551 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:52.549+0000 7f0307c3a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:47:52.556 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.554+0000 7fb76dc11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:47:52.604 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:52.603+0000 7f0307c3a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:47:52.617 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.615+0000 7fb76dc11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:47:52.666 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.663+0000 7fb76dc11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:47:52.668 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:52.667+0000 7f0307c3a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:47:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.692+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd43404e920 con 0x7fd4340d1e40 2023-11-28T20:47:52.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.693+0000 7fd42bfff640 1 -- 172.21.15.45:0/1393819760 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd424079d80 con 0x7fd4340d1e40 2023-11-28T20:47:52.694 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:47:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd432ffd640 1 -- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd41406aff0 msgr2=0x7fd41406d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd432ffd640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd41406aff0 0x7fd41406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd41406aff0 msgr2=0x7fd41406d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd41406aff0 0x7fd41406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 msgr2=0x7fd4341601a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 0x7fd4341601a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd424035040 tx=0x7fd4240046c0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 shutdown_connections 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fd41406aff0 0x7fd41406d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd43415d580 0x7fd4341f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4340d1e40 0x7fd4341601a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 --2- 172.21.15.45:0/1393819760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd4340b2500 0x7fd43415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.695+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 >> 172.21.15.45:0/1393819760 conn(0x7fd43406a500 msgr2=0x7fd4340cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.696+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 shutdown_connections 2023-11-28T20:47:52.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.696+0000 7fd433fff640 1 -- 172.21.15.45:0/1393819760 wait complete. 2023-11-28T20:47:52.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:47:52.723 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:52.722+0000 7f0307c3a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:47:52.775 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.773+0000 7fb76dc11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:47:52.838 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.836+0000 7fb76dc11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:47:52.842 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:52.840+0000 7f0307c3a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:47:52.887 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.885+0000 7fb76dc11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:47:52.908 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:52.907+0000 7f0307c3a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:47:52.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.923+0000 7fb5a67f4640 1 Processor -- start 2023-11-28T20:47:52.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.923+0000 7fb5a67f4640 1 -- start start 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.925+0000 7fb5a67f4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5a00d0040 0x7fb5a00d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.925+0000 7fb5a67f4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb5a00b2290 0x7fb5a00d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.925+0000 7fb5a67f4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5a00b1d40 0x7fb5a00d3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.925+0000 7fb5a67f4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a00b90a0 con 0x7fb5a00b1d40 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.925+0000 7fb5a67f4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a00d1920 con 0x7fb5a00b2290 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.925+0000 7fb5a67f4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00d0960 con 0x7fb5a00d0040 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.925+0000 7fb5a5ff3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5a00b1d40 0x7fb5a00d3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.925+0000 7fb5a5ff3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5a00b1d40 0x7fb5a00d3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57082/0 (socket says 172.21.15.45:57082) 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.925+0000 7fb5a5ff3640 1 -- 172.21.15.45:0/573910345 learned_addr learned my addr 172.21.15.45:0/573910345 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.926+0000 7fb5a4ff1640 1 --2- 172.21.15.45:0/573910345 >> v2:172.21.15.190:3300/0 conn(0x7fb5a00b2290 0x7fb5a00d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.926+0000 7fb5a57f2640 1 --2- 172.21.15.45:0/573910345 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5a00d0040 0x7fb5a00d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.926+0000 7fb5a5ff3640 1 -- 172.21.15.45:0/573910345 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5a00d0040 msgr2=0x7fb5a00d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:52.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.926+0000 7fb5a5ff3640 1 --2- 172.21.15.45:0/573910345 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5a00d0040 0x7fb5a00d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.926+0000 7fb5a5ff3640 1 -- 172.21.15.45:0/573910345 >> v2:172.21.15.190:3300/0 conn(0x7fb5a00b2290 msgr2=0x7fb5a00d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:52.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.926+0000 7fb5a5ff3640 1 --2- 172.21.15.45:0/573910345 >> v2:172.21.15.190:3300/0 conn(0x7fb5a00b2290 0x7fb5a00d3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.926+0000 7fb5a5ff3640 1 -- 172.21.15.45:0/573910345 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a00b8ae0 con 0x7fb5a00b1d40 2023-11-28T20:47:52.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.927+0000 7fb5a5ff3640 1 --2- 172.21.15.45:0/573910345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 0x7fb5a00d3580 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00d4110 tx=0x7fb59802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7136fc8bdd5a89e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.927+0000 7fb597fff640 1 -- 172.21.15.45:0/573910345 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59802cb00 con 0x7fb5a00b1d40 2023-11-28T20:47:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.927+0000 7fb597fff640 1 -- 172.21.15.45:0/573910345 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59802cc60 con 0x7fb5a00b1d40 2023-11-28T20:47:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.927+0000 7fb597fff640 1 -- 172.21.15.45:0/573910345 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5980343f0 con 0x7fb5a00b1d40 2023-11-28T20:47:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.928+0000 7fb5a67f4640 1 -- 172.21.15.45:0/573910345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 msgr2=0x7fb5a00d3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.928+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/573910345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 0x7fb5a00d3580 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00d4110 tx=0x7fb59802bf60 comp rx=0 tx=0).stop 2023-11-28T20:47:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.928+0000 7fb5a67f4640 1 -- 172.21.15.45:0/573910345 shutdown_connections 2023-11-28T20:47:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.928+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/573910345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 0x7fb5a00d3580 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.928+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/573910345 >> v2:172.21.15.190:3300/0 conn(0x7fb5a00b2290 0x7fb5a00d3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.928+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/573910345 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5a00d0040 0x7fb5a00d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.928+0000 7fb5a67f4640 1 -- 172.21.15.45:0/573910345 >> 172.21.15.45:0/573910345 conn(0x7fb5a0070f40 msgr2=0x7fb5a0071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.928+0000 7fb5a67f4640 1 -- 172.21.15.45:0/573910345 shutdown_connections 2023-11-28T20:47:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.928+0000 7fb5a67f4640 1 -- 172.21.15.45:0/573910345 wait complete. 2023-11-28T20:47:52.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.930+0000 7fb5a67f4640 1 Processor -- start 2023-11-28T20:47:52.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.930+0000 7fb5a67f4640 1 -- start start 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.931+0000 7fb5a67f4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 0x7fb5a0149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.931+0000 7fb5a67f4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5a00b2290 0x7fb5a01f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.931+0000 7fb5a67f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a00d0040 0x7fb5a00cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.931+0000 7fb5a67f4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00c91c0 con 0x7fb5a00b1d40 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.931+0000 7fb5a67f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a01497b0 con 0x7fb5a00d0040 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.931+0000 7fb5a67f4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a01f76e0 con 0x7fb5a00b2290 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.931+0000 7fb5a57f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 0x7fb5a0149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.931+0000 7fb5a57f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 0x7fb5a0149270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57086/0 (socket says 172.21.15.45:57086) 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.931+0000 7fb5a57f2640 1 -- 172.21.15.45:0/2886947695 learned_addr learned my addr 172.21.15.45:0/2886947695 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a5ff3640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a00d0040 0x7fb5a00cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:52.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a57f2640 1 -- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5a00b2290 msgr2=0x7fb5a01f71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a57f2640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5a00b2290 0x7fb5a01f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a57f2640 1 -- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a00d0040 msgr2=0x7fb5a00cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a57f2640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a00d0040 0x7fb5a00cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a57f2640 1 -- 172.21.15.45:0/2886947695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598009b30 con 0x7fb5a00b1d40 2023-11-28T20:47:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a5ff3640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a00d0040 0x7fb5a00cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a57f2640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 0x7fb5a0149270 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c7f0 tx=0x7fb59c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb595ffb640 1 -- 172.21.15.45:0/2886947695 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c004220 con 0x7fb5a00b1d40 2023-11-28T20:47:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb595ffb640 1 -- 172.21.15.45:0/2886947695 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c0043a0 con 0x7fb5a00b1d40 2023-11-28T20:47:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a0149c90 con 0x7fb5a00b1d40 2023-11-28T20:47:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb595ffb640 1 -- 172.21.15.45:0/2886947695 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c007860 con 0x7fb5a00b1d40 2023-11-28T20:47:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.932+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a00c8890 con 0x7fb5a00b1d40 2023-11-28T20:47:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.934+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a015d0c0 con 0x7fb5a00b1d40 2023-11-28T20:47:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.935+0000 7fb595ffb640 1 -- 172.21.15.45:0/2886947695 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 89402+0+0 (secure 0 0 0) 0x7fb59c007bd0 con 0x7fb5a00b1d40 2023-11-28T20:47:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.935+0000 7fb595ffb640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fb58406aff0 0x7fb58406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.936+0000 7fb595ffb640 1 -- 172.21.15.45:0/2886947695 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7060+0+0 (secure 0 0 0) 0x7fb59c089fc0 con 0x7fb5a00b1d40 2023-11-28T20:47:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.936+0000 7fb5a4ff1640 1 -- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fb58406aff0 msgr2=0x7fb58406d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.936+0000 7fb5a4ff1640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fb58406aff0 0x7fb58406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:47:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:52.939+0000 7fb595ffb640 1 -- 172.21.15.45:0/2886947695 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb59c0546f0 con 0x7fb5a00b1d40 2023-11-28T20:47:52.945 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.943+0000 7fb76dc11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:47:52.962 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:52.961+0000 7f0307c3a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:47:52.996 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:52.993+0000 7fb76dc11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:47:53.022 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:53.021+0000 7f0307c3a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:47:53.075 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:53.074+0000 7f0307c3a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:47:53.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.130+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5a0071b30 con 0x7fb5a00b1d40 2023-11-28T20:47:53.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.131+0000 7fb595ffb640 1 -- 172.21.15.45:0/2886947695 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb59c01e0a0 con 0x7fb5a00b1d40 2023-11-28T20:47:53.132 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:47:53.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fb58406aff0 msgr2=0x7fb58406d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:47:53.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fb58406aff0 0x7fb58406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 msgr2=0x7fb5a0149270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 0x7fb5a0149270 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c7f0 tx=0x7fb59c00ccb0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 shutdown_connections 2023-11-28T20:47:53.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fb58406aff0 0x7fb58406d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a00d0040 0x7fb5a00cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5a00b2290 0x7fb5a01f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 --2- 172.21.15.45:0/2886947695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5a00b1d40 0x7fb5a0149270 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 >> 172.21.15.45:0/2886947695 conn(0x7fb5a0070f40 msgr2=0x7fb5a00b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:53.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 shutdown_connections 2023-11-28T20:47:53.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.134+0000 7fb5a67f4640 1 -- 172.21.15.45:0/2886947695 wait complete. 2023-11-28T20:47:53.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:47:53.378 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:53.376+0000 7fb76dc11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:47:53.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.412+0000 7fa066217640 1 Processor -- start 2023-11-28T20:47:53.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.413+0000 7fa066217640 1 -- start start 2023-11-28T20:47:53.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.414+0000 7fa066217640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600ce600 0x7fa06006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.414+0000 7fa066217640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa06006aa40 0x7fa0600c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.414+0000 7fa066217640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06006e030 con 0x7fa06006dc50 2023-11-28T20:47:53.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.414+0000 7fa066217640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06006aed0 con 0x7fa06006aa40 2023-11-28T20:47:53.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.414+0000 7fa066217640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0600ccf60 con 0x7fa0600ce600 2023-11-28T20:47:53.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.415+0000 7fa05f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa06006dc50 0x7fa0600cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52408/0 (socket says 172.21.15.45:52408) 2023-11-28T20:47:53.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.415+0000 7fa05f7fe640 1 -- 172.21.15.45:0/3554456054 learned_addr learned my addr 172.21.15.45:0/3554456054 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:53.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.415+0000 7fa057fff640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600ce600 0x7fa06006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.415+0000 7fa05ffff640 1 --2- 172.21.15.45:0/3554456054 >> v2:172.21.15.190:3300/0 conn(0x7fa06006aa40 0x7fa0600c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.415+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555760698 0 0) 0x7fa050001110 con 0x7fa06006dc50 2023-11-28T20:47:53.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.415+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa04c003520 con 0x7fa06006dc50 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3755740448 0 0) 0x7fa04c003520 con 0x7fa06006dc50 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600ce600 msgr2=0x7fa06006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600ce600 0x7fa06006a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 >> v2:172.21.15.190:3300/0 conn(0x7fa06006aa40 msgr2=0x7fa0600c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 --2- 172.21.15.45:0/3554456054 >> v2:172.21.15.190:3300/0 conn(0x7fa06006aa40 0x7fa0600c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0600cc930 con 0x7fa06006dc50 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fa050001800 con 0x7fa06006dc50 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 >> v1:172.21.15.45:6789/0 conn(0x7fa06006dc50 legacy=0x7fa0600cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa04c003970 0x7fa04c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600ce600 0x7fa04c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa04c004d00 0x7fa04c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06006aed0 con 0x7fa04c003970 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0600ccf60 con 0x7fa04c004d00 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c00b980 con 0x7fa0600ce600 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05f7fe640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa04c003970 0x7fa04c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa057fff640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600ce600 0x7fa04c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05f7fe640 1 -- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600ce600 msgr2=0x7fa04c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05f7fe640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600ce600 0x7fa04c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05f7fe640 1 -- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa04c004d00 msgr2=0x7fa04c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05f7fe640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa04c004d00 0x7fa04c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.416+0000 7fa05f7fe640 1 -- 172.21.15.45:0/3554456054 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c003520 con 0x7fa04c003970 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.417+0000 7fa057fff640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600ce600 0x7fa04c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:53.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.418+0000 7fa05f7fe640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa04c003970 0x7fa04c003d90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa05000c210 tx=0x7fa04c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1708bc9c22b528 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:53.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.418+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05002cae0 con 0x7fa04c003970 2023-11-28T20:47:53.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.418+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05002cc60 con 0x7fa04c003970 2023-11-28T20:47:53.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.418+0000 7fa05effd640 1 -- 172.21.15.45:0/3554456054 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0500058d0 con 0x7fa04c003970 2023-11-28T20:47:53.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.419+0000 7fa066217640 1 -- 172.21.15.45:0/3554456054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa04c003970 msgr2=0x7fa04c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.419+0000 7fa066217640 1 --2- 172.21.15.45:0/3554456054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa04c003970 0x7fa04c003d90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa05000c210 tx=0x7fa04c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.419+0000 7fa05ffff640 1 -- 172.21.15.45:0/3554456054 reap_dead start 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.419+0000 7fa066217640 1 -- 172.21.15.45:0/3554456054 shutdown_connections 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.419+0000 7fa066217640 1 -- 172.21.15.45:0/3554456054 >> 172.21.15.45:0/3554456054 conn(0x7fa0600ced60 msgr2=0x7fa0600cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.419+0000 7fa066217640 1 -- 172.21.15.45:0/3554456054 shutdown_connections 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.419+0000 7fa066217640 1 -- 172.21.15.45:0/3554456054 wait complete. 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.421+0000 7fa066217640 1 Processor -- start 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.421+0000 7fa066217640 1 -- start start 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa066217640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa04c003970 0x7fa060155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa066217640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0600c2a40 0x7fa0600c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa066217640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600c9390 0x7fa0600c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa066217640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0600c9d50 con 0x7fa0600c2a40 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa066217640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa060156360 con 0x7fa04c003970 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa066217640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0601e6d10 con 0x7fa0600c9390 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa057fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0600c2a40 0x7fa0600c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa057fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0600c2a40 0x7fa0600c2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57112/0 (socket says 172.21.15.45:57112) 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa057fff640 1 -- 172.21.15.45:0/2286786204 learned_addr learned my addr 172.21.15.45:0/2286786204 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa05f7fe640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa04c003970 0x7fa060155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa05ffff640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600c9390 0x7fa0600c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa057fff640 1 -- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600c9390 msgr2=0x7fa0600c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa057fff640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600c9390 0x7fa0600c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa057fff640 1 -- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa04c003970 msgr2=0x7fa060155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa057fff640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa04c003970 0x7fa060155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.422+0000 7fa057fff640 1 -- 172.21.15.45:0/2286786204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0601e6ff0 con 0x7fa0600c2a40 2023-11-28T20:47:53.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.423+0000 7fa057fff640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0600c2a40 0x7fa0600c2ea0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa0480108b0 tx=0x7fa048010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:53.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.423+0000 7fa05cff9640 1 -- 172.21.15.45:0/2286786204 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04800eb60 con 0x7fa0600c2a40 2023-11-28T20:47:53.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.423+0000 7fa05cff9640 1 -- 172.21.15.45:0/2286786204 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04800ecc0 con 0x7fa0600c2a40 2023-11-28T20:47:53.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.423+0000 7fa05cff9640 1 -- 172.21.15.45:0/2286786204 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04801c400 con 0x7fa0600c2a40 2023-11-28T20:47:53.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.423+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0601e72d0 con 0x7fa0600c2a40 2023-11-28T20:47:53.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.423+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0601e7890 con 0x7fa0600c2a40 2023-11-28T20:47:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.424+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0600733d0 con 0x7fa0600c2a40 2023-11-28T20:47:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.425+0000 7fa05cff9640 1 -- 172.21.15.45:0/2286786204 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 89402+0+0 (secure 0 0 0) 0x7fa04801c560 con 0x7fa0600c2a40 2023-11-28T20:47:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.426+0000 7fa05cff9640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa04006b020 0x7fa04006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.426+0000 7fa05cff9640 1 -- 172.21.15.45:0/2286786204 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7060+0+0 (secure 0 0 0) 0x7fa04808a400 con 0x7fa0600c2a40 2023-11-28T20:47:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.426+0000 7fa05f7fe640 1 -- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa04006b020 msgr2=0x7fa04006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:53.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.426+0000 7fa05f7fe640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa04006b020 0x7fa04006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:47:53.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.434+0000 7fa05cff9640 1 -- 172.21.15.45:0/2286786204 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa048054e20 con 0x7fa0600c2a40 2023-11-28T20:47:53.489 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:53.488+0000 7f0307c3a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:47:53.537 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:53.535+0000 7fb76dc11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:47:53.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.620+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa060161db0 con 0x7fa0600c2a40 2023-11-28T20:47:53.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.621+0000 7fa05cff9640 1 -- 172.21.15.45:0/2286786204 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa048058440 con 0x7fa0600c2a40 2023-11-28T20:47:53.622 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:47:53.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa04006b020 msgr2=0x7fa04006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:47:53.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa04006b020 0x7fa04006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0600c2a40 msgr2=0x7fa0600c2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0600c2a40 0x7fa0600c2ea0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa0480108b0 tx=0x7fa048010d70 comp rx=0 tx=0).stop 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 shutdown_connections 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7fa04006b020 0x7fa04006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0600c9390 0x7fa0600c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0600c2a40 0x7fa0600c2ea0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 --2- 172.21.15.45:0/2286786204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa04c003970 0x7fa060155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 >> 172.21.15.45:0/2286786204 conn(0x7fa0600ced60 msgr2=0x7fa0600cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 shutdown_connections 2023-11-28T20:47:53.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.624+0000 7fa066217640 1 -- 172.21.15.45:0/2286786204 wait complete. 2023-11-28T20:47:53.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:47:53.645 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:53.641+0000 7fb76dc11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:47:53.659 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:53.658+0000 7f0307c3a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:47:53.693 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:53.691+0000 7fb76dc11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:47:53.774 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:53.773+0000 7f0307c3a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:47:53.799 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:53.797+0000 7fb76dc11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:47:53.828 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:53.826+0000 7f0307c3a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:47:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.848+0000 7f2d35e2b640 1 Processor -- start 2023-11-28T20:47:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.848+0000 7f2d35e2b640 1 -- start start 2023-11-28T20:47:53.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.849+0000 7f2d35e2b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2d3015d600 0x7f2d300cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.849+0000 7f2d35e2b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2d30099a40 0x7f2d30099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d35e2b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d300d06c0 con 0x7f2d3015d220 2023-11-28T20:47:53.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d35e2b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3009a410 con 0x7f2d30099a40 2023-11-28T20:47:53.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d35e2b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d300cf8a0 con 0x7f2d3015d600 2023-11-28T20:47:53.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d2ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2d30099a40 0x7f2d30099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d2ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2d30099a40 0x7f2d30099ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57122/0 (socket says 172.21.15.45:57122) 2023-11-28T20:47:53.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d2ffff640 1 -- 172.21.15.45:0/2789513009 learned_addr learned my addr 172.21.15.45:0/2789513009 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:53.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d27fff640 1 --2- 172.21.15.45:0/2789513009 >> v2:172.21.15.190:3300/0 conn(0x7f2d3015d600 0x7f2d300cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d2effd640 1 -- 172.21.15.45:0/2789513009 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51132298 0 0) 0x7f2d300d06c0 con 0x7f2d3015d220 2023-11-28T20:47:53.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d2ffff640 1 -- 172.21.15.45:0/2789513009 >> v2:172.21.15.190:3300/0 conn(0x7f2d3015d600 msgr2=0x7f2d300cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:53.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d2ffff640 1 --2- 172.21.15.45:0/2789513009 >> v2:172.21.15.190:3300/0 conn(0x7f2d3015d600 0x7f2d300cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d2ffff640 1 -- 172.21.15.45:0/2789513009 >> v1:172.21.15.45:6789/0 conn(0x7f2d3015d220 legacy=0x7f2d3006b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.850+0000 7f2d2ffff640 1 -- 172.21.15.45:0/2789513009 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d300cfab0 con 0x7f2d30099a40 2023-11-28T20:47:53.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.852+0000 7f2d2ffff640 1 --2- 172.21.15.45:0/2789513009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 0x7f2d30099ed0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2d30072980 tx=0x7f2d2802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db65f9f69a8afec server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:53.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d2effd640 1 -- 172.21.15.45:0/2789513009 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2802cb20 con 0x7f2d30099a40 2023-11-28T20:47:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d2effd640 1 -- 172.21.15.45:0/2789513009 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2802cc80 con 0x7f2d30099a40 2023-11-28T20:47:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d2effd640 1 -- 172.21.15.45:0/2789513009 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d280343f0 con 0x7f2d30099a40 2023-11-28T20:47:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d35e2b640 1 -- 172.21.15.45:0/2789513009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 msgr2=0x7f2d30099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d35e2b640 1 --2- 172.21.15.45:0/2789513009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 0x7f2d30099ed0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2d30072980 tx=0x7f2d2802bf80 comp rx=0 tx=0).stop 2023-11-28T20:47:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d35e2b640 1 -- 172.21.15.45:0/2789513009 shutdown_connections 2023-11-28T20:47:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d35e2b640 1 --2- 172.21.15.45:0/2789513009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 0x7f2d30099ed0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d35e2b640 1 --2- 172.21.15.45:0/2789513009 >> v2:172.21.15.190:3300/0 conn(0x7f2d3015d600 0x7f2d300cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d35e2b640 1 -- 172.21.15.45:0/2789513009 >> 172.21.15.45:0/2789513009 conn(0x7f2d300b2ae0 msgr2=0x7f2d300b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d35e2b640 1 -- 172.21.15.45:0/2789513009 shutdown_connections 2023-11-28T20:47:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.853+0000 7f2d35e2b640 1 -- 172.21.15.45:0/2789513009 wait complete. 2023-11-28T20:47:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.855+0000 7f2d35e2b640 1 Processor -- start 2023-11-28T20:47:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.855+0000 7f2d35e2b640 1 -- start start 2023-11-28T20:47:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.856+0000 7f2d35e2b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 0x7f2d30160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.856+0000 7f2d35e2b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d3015d220 0x7f2d3014f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.856+0000 7f2d35e2b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d3015d600 0x7f2d3014f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.856+0000 7f2d35e2b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3014ed10 con 0x7f2d30099a40 2023-11-28T20:47:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.856+0000 7f2d35e2b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300c5b70 con 0x7f2d3015d220 2023-11-28T20:47:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.856+0000 7f2d35e2b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300c5e50 con 0x7f2d3015d600 2023-11-28T20:47:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.856+0000 7f2d2f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 0x7f2d30160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d2f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 0x7f2d30160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57128/0 (socket says 172.21.15.45:57128) 2023-11-28T20:47:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d2f7fe640 1 -- 172.21.15.45:0/3432971217 learned_addr learned my addr 172.21.15.45:0/3432971217 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d27fff640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d3015d220 0x7f2d3014f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d2ffff640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d3015d600 0x7f2d3014f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d2f7fe640 1 -- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d3015d600 msgr2=0x7f2d3014f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d2f7fe640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d3015d600 0x7f2d3014f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d2f7fe640 1 -- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d3015d220 msgr2=0x7f2d3014f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d2f7fe640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d3015d220 0x7f2d3014f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d2f7fe640 1 -- 172.21.15.45:0/3432971217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28009b30 con 0x7f2d30099a40 2023-11-28T20:47:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.857+0000 7f2d2f7fe640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 0x7f2d30160210 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2d2000c4b0 tx=0x7f2d2000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.858+0000 7f2d2cff9640 1 -- 172.21.15.45:0/3432971217 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20005480 con 0x7f2d30099a40 2023-11-28T20:47:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.858+0000 7f2d2cff9640 1 -- 172.21.15.45:0/3432971217 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d200035f0 con 0x7f2d30099a40 2023-11-28T20:47:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.858+0000 7f2d2cff9640 1 -- 172.21.15.45:0/3432971217 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d200039e0 con 0x7f2d30099a40 2023-11-28T20:47:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.858+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d300c6130 con 0x7f2d30099a40 2023-11-28T20:47:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.858+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d300c66f0 con 0x7f2d30099a40 2023-11-28T20:47:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.859+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3006bc50 con 0x7f2d30099a40 2023-11-28T20:47:53.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.860+0000 7f2d2cff9640 1 -- 172.21.15.45:0/3432971217 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 89402+0+0 (secure 0 0 0) 0x7f2d20003b40 con 0x7f2d30099a40 2023-11-28T20:47:53.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.861+0000 7f2d2cff9640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2d1006aff0 0x7f2d1006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:53.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.861+0000 7f2d2cff9640 1 -- 172.21.15.45:0/3432971217 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7060+0+0 (secure 0 0 0) 0x7f2d20089490 con 0x7f2d30099a40 2023-11-28T20:47:53.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.861+0000 7f2d27fff640 1 -- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2d1006aff0 msgr2=0x7f2d1006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:53.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.861+0000 7f2d27fff640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2d1006aff0 0x7f2d1006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:47:53.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:53.865+0000 7f2d2cff9640 1 -- 172.21.15.45:0/3432971217 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d20053d60 con 0x7f2d30099a40 2023-11-28T20:47:53.906 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:53.904+0000 7fb76dc11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:47:53.944 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:53.943+0000 7f0307c3a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:47:54.058 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:54.057+0000 7f0307c3a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:47:54.061 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:54.059+0000 7fb76dc11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:47:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.062+0000 7f2d27fff640 1 -- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2d1006aff0 msgr2=0x7f2d1006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.062+0000 7f2d27fff640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2d1006aff0 0x7f2d1006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:47:54.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.069+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2d300c6910 con 0x7f2d30099a40 2023-11-28T20:47:54.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.070+0000 7f2d2cff9640 1 -- 172.21.15.45:0/3432971217 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v56) v1 ==== 80+0+360 (secure 0 0 0) 0x7f2d20017080 con 0x7f2d30099a40 2023-11-28T20:47:54.071 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1866894774 2023-11-29T20:35:55.110774+0000 2023-11-28T20:47:54.071 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3717946281 2023-11-29T20:35:55.110774+0000 2023-11-28T20:47:54.071 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/1081538426 2023-11-29T20:35:55.110774+0000 2023-11-28T20:47:54.071 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/1081538426 2023-11-29T20:35:55.110774+0000 2023-11-28T20:47:54.072 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1813745990 2023-11-29T20:35:55.110774+0000 2023-11-28T20:47:54.072 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3262798906 2023-11-29T20:35:55.110774+0000 2023-11-28T20:47:54.072 INFO:teuthology.orchestra.run.smithi045.stderr:listed 6 entries 2023-11-28T20:47:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.072+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2d1006aff0 msgr2=0x7f2d1006d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:47:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.072+0000 7f2d35e2b640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2d1006aff0 0x7f2d1006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.072+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 msgr2=0x7f2d30160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.072+0000 7f2d35e2b640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 0x7f2d30160210 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2d2000c4b0 tx=0x7f2d2000c970 comp rx=0 tx=0).stop 2023-11-28T20:47:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.073+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 shutdown_connections 2023-11-28T20:47:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.073+0000 7f2d35e2b640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2d1006aff0 0x7f2d1006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.073+0000 7f2d35e2b640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d3015d600 0x7f2d3014f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.073+0000 7f2d35e2b640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d3015d220 0x7f2d3014f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.073+0000 7f2d35e2b640 1 --2- 172.21.15.45:0/3432971217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d30099a40 0x7f2d30160210 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.073+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 >> 172.21.15.45:0/3432971217 conn(0x7f2d300b2ae0 msgr2=0x7f2d300b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.073+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 shutdown_connections 2023-11-28T20:47:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.073+0000 7f2d35e2b640 1 -- 172.21.15.45:0/3432971217 wait complete. 2023-11-28T20:47:54.094 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T20:47:54.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:54.094 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T20:47:54.109 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:54.107+0000 7fb76dc11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:47:54.225 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:54.224+0000 7f0307c3a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:47:54.228 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:54.225+0000 7fb76dc11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:47:54.239 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:47:54.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:47:54.240 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T20:47:54.284 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:54.283+0000 7f0307c3a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:47:54.286 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:47:54.287 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:47:54.337 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:47:54.337 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:47:54.337 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:47:54.383 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:47:54.383 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:47:54.384 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:47:54.415 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:54.413+0000 7f0307c3a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:47:54.433 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:47:54.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:47:54.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.637+0000 7f2c1759e640 1 Processor -- start 2023-11-28T20:47:54.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.638+0000 7f2c1759e640 1 -- start start 2023-11-28T20:47:54.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.639+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180b00d0 0x7f2c180b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:54.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.639+0000 7f2c1759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c180d0ea0 0x7f2c180d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:54.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.639+0000 7f2c1759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c180d1820 con 0x7f2c180cd9a0 2023-11-28T20:47:54.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c1759e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c180b1820 con 0x7f2c180d0ea0 2023-11-28T20:47:54.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c1759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180b0ef0 con 0x7f2c180b00d0 2023-11-28T20:47:54.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c15d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c180d0ea0 0x7f2c180d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:54.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c16d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2c180cd9a0 0x7f2c180cde30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36058/0 (socket says 172.21.15.45:36058) 2023-11-28T20:47:54.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c16d9d640 1 -- 172.21.15.45:0/1407439105 learned_addr learned my addr 172.21.15.45:0/1407439105 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:54.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c1659c640 1 --2- 172.21.15.45:0/1407439105 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180b00d0 0x7f2c180b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:54.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c15d9b640 1 -- 172.21.15.45:0/1407439105 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180b00d0 msgr2=0x7f2c180b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:54.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c15d9b640 1 --2- 172.21.15.45:0/1407439105 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180b00d0 0x7f2c180b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c15d9b640 1 -- 172.21.15.45:0/1407439105 >> v1:172.21.15.45:6789/0 conn(0x7f2c180cd9a0 legacy=0x7f2c180cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:54.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.640+0000 7f2c15d9b640 1 -- 172.21.15.45:0/1407439105 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c180b1160 con 0x7f2c180d0ea0 2023-11-28T20:47:54.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.641+0000 7f2c15d9b640 1 --2- 172.21.15.45:0/1407439105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180d0ea0 0x7f2c180d12e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0097a0 tx=0x7f2c0c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6558a46fa7c646ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:54.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.642+0000 7f2c1559a640 1 -- 172.21.15.45:0/1407439105 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0316f0 con 0x7f2c180d0ea0 2023-11-28T20:47:54.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.642+0000 7f2c1559a640 1 -- 172.21.15.45:0/1407439105 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c031850 con 0x7f2c180d0ea0 2023-11-28T20:47:54.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.642+0000 7f2c1559a640 1 -- 172.21.15.45:0/1407439105 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c031be0 con 0x7f2c180d0ea0 2023-11-28T20:47:54.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.642+0000 7f2c1759e640 1 -- 172.21.15.45:0/1407439105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180d0ea0 msgr2=0x7f2c180d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:54.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.642+0000 7f2c1759e640 1 --2- 172.21.15.45:0/1407439105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180d0ea0 0x7f2c180d12e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0097a0 tx=0x7f2c0c02b790 comp rx=0 tx=0).stop 2023-11-28T20:47:54.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.643+0000 7f2c1759e640 1 -- 172.21.15.45:0/1407439105 shutdown_connections 2023-11-28T20:47:54.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.643+0000 7f2c1759e640 1 --2- 172.21.15.45:0/1407439105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180d0ea0 0x7f2c180d12e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.643+0000 7f2c1759e640 1 --2- 172.21.15.45:0/1407439105 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180b00d0 0x7f2c180b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.643+0000 7f2c1759e640 1 -- 172.21.15.45:0/1407439105 >> 172.21.15.45:0/1407439105 conn(0x7f2c18099a40 msgr2=0x7f2c18099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:54.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.643+0000 7f2c1759e640 1 -- 172.21.15.45:0/1407439105 shutdown_connections 2023-11-28T20:47:54.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.643+0000 7f2c1759e640 1 -- 172.21.15.45:0/1407439105 wait complete. 2023-11-28T20:47:54.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.645+0000 7f2c1759e640 1 Processor -- start 2023-11-28T20:47:54.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.645+0000 7f2c1759e640 1 -- start start 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.646+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180b00d0 0x7f2c1815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.646+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180cd9a0 0x7f2c180b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.646+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c180d0ea0 0x7f2c180b6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.646+0000 7f2c1759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1815abb0 con 0x7f2c180b00d0 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.646+0000 7f2c1759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180b4210 con 0x7f2c180d0ea0 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.646+0000 7f2c1759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180b44f0 con 0x7f2c180cd9a0 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c1659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180b00d0 0x7f2c1815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c1659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180b00d0 0x7f2c1815c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58088/0 (socket says 172.21.15.45:58088) 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c1659c640 1 -- 172.21.15.45:0/3919028156 learned_addr learned my addr 172.21.15.45:0/3919028156 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:54.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c15d9b640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180cd9a0 0x7f2c180b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:54.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c16d9d640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c180d0ea0 0x7f2c180b6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c1659c640 1 -- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180cd9a0 msgr2=0x7f2c180b6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c1659c640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180cd9a0 0x7f2c180b6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c1659c640 1 -- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c180d0ea0 msgr2=0x7f2c180b6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c1659c640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c180d0ea0 0x7f2c180b6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.647+0000 7f2c1659c640 1 -- 172.21.15.45:0/3919028156 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c009400 con 0x7f2c180b00d0 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.648+0000 7f2c16d9d640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c180d0ea0 0x7f2c180b6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.648+0000 7f2c1659c640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180b00d0 0x7f2c1815c5d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800d5f0 tx=0x7f2c0800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.648+0000 7f2c06ffd640 1 -- 172.21.15.45:0/3919028156 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c080125f0 con 0x7f2c180b00d0 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.648+0000 7f2c06ffd640 1 -- 172.21.15.45:0/3919028156 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c08012750 con 0x7f2c180b00d0 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.648+0000 7f2c06ffd640 1 -- 172.21.15.45:0/3919028156 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08012b00 con 0x7f2c180b00d0 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.648+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c180b47d0 con 0x7f2c180b00d0 2023-11-28T20:47:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.648+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c180c8000 con 0x7f2c180b00d0 2023-11-28T20:47:54.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.649+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c18066ca0 con 0x7f2c180b00d0 2023-11-28T20:47:54.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.650+0000 7f2c06ffd640 1 -- 172.21.15.45:0/3919028156 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 89402+0+0 (secure 0 0 0) 0x7f2c08012c60 con 0x7f2c180b00d0 2023-11-28T20:47:54.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.651+0000 7f2c06ffd640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2bf806aff0 0x7f2bf806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:54.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.651+0000 7f2c06ffd640 1 -- 172.21.15.45:0/3919028156 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6916+0+0 (secure 0 0 0) 0x7f2c0808a270 con 0x7f2c180b00d0 2023-11-28T20:47:54.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.651+0000 7f2c15d9b640 1 -- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2bf806aff0 msgr2=0x7f2bf806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:54.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.651+0000 7f2c15d9b640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2bf806aff0 0x7f2bf806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:47:54.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.660+0000 7f2c06ffd640 1 -- 172.21.15.45:0/3919028156 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c08054b40 con 0x7f2c180b00d0 2023-11-28T20:47:54.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.852+0000 7f2c15d9b640 1 -- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2bf806aff0 msgr2=0x7f2bf806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:54.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.852+0000 7f2c15d9b640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2bf806aff0 0x7f2bf806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:47:54.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.892+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c1804e920 con 0x7f2c180b00d0 2023-11-28T20:47:54.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.892+0000 7f2c06ffd640 1 -- 172.21.15.45:0/3919028156 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f2c08058160 con 0x7f2c180b00d0 2023-11-28T20:47:54.894 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:54.894 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3050590116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3050590116},{"type":"v1","addr":"172.21.15.45:6835","nonce":3050590116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2465910573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2465910573},{"type":"v1","addr":"172.21.15.190:6836","nonce":2465910573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4090116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4090116745},{"type":"v1","addr":"172.21.15.190:6837","nonce":4090116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:35:35.751211+0000","modified":"2023-11-28T20:35:38.829781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4945":{"gid":4945,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6834/1984946086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1984946086},{"type":"v1","addr":"172.21.15.45:6834","nonce":1984946086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T20:47:54.894 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 18 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.895+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2bf806aff0 msgr2=0x7f2bf806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.895+0000 7f2c1759e640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2bf806aff0 0x7f2bf806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.895+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180b00d0 msgr2=0x7f2c1815c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.895+0000 7f2c1759e640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180b00d0 0x7f2c1815c5d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800d5f0 tx=0x7f2c0800dab0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.895+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 shutdown_connections 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.895+0000 7f2c1759e640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f2bf806aff0 0x7f2bf806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.895+0000 7f2c1759e640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c180d0ea0 0x7f2c180b6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.895+0000 7f2c1759e640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c180cd9a0 0x7f2c180b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.895+0000 7f2c1759e640 1 --2- 172.21.15.45:0/3919028156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c180b00d0 0x7f2c1815c5d0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.896+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 >> 172.21.15.45:0/3919028156 conn(0x7f2c18099a40 msgr2=0x7f2c1806ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.896+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 shutdown_connections 2023-11-28T20:47:54.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:54.896+0000 7f2c1759e640 1 -- 172.21.15.45:0/3919028156 wait complete. 2023-11-28T20:47:54.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:47:55.072 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.070+0000 7fb76dc11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:47:55.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.114+0000 7f6f9dc98640 1 Processor -- start 2023-11-28T20:47:55.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.114+0000 7f6f9dc98640 1 -- start start 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.116+0000 7f6f9dc98640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.116+0000 7f6f9dc98640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6f980b69c0 0x7f6f9815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.116+0000 7f6f9dc98640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f980d1250 con 0x7f6f980b6560 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.116+0000 7f6f9dc98640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f980ce6b0 con 0x7f6f980b69c0 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.116+0000 7f6f9dc98640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980ce990 con 0x7f6f980cfaf0 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.116+0000 7f6f9cc96640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.116+0000 7f6f9d497640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6f980b69c0 0x7f6f9815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.116+0000 7f6f9cc96640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:38226/0 (socket says 172.21.15.45:38226) 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.116+0000 7f6f9cc96640 1 -- 172.21.15.45:0/3068969733 learned_addr learned my addr 172.21.15.45:0/3068969733 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.117+0000 7f6f9cc96640 1 -- 172.21.15.45:0/3068969733 >> v2:172.21.15.190:3300/0 conn(0x7f6f980b69c0 msgr2=0x7f6f9815cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:55.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.117+0000 7f6f9cc96640 1 --2- 172.21.15.45:0/3068969733 >> v2:172.21.15.190:3300/0 conn(0x7f6f980b69c0 0x7f6f9815cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.117+0000 7f6f9cc96640 1 -- 172.21.15.45:0/3068969733 >> v1:172.21.15.190:6789/0 conn(0x7f6f980b6560 legacy=0x7f6f980cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.117+0000 7f6f9cc96640 1 -- 172.21.15.45:0/3068969733 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980d14a0 con 0x7f6f980cfaf0 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.118+0000 7f6f9cc96640 1 --2- 172.21.15.45:0/3068969733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980cfed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009870 tx=0x7f6f8c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc4a5fdcfc91a1ce server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.119+0000 7f6f977fe640 1 -- 172.21.15.45:0/3068969733 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c030840 con 0x7f6f980cfaf0 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.119+0000 7f6f977fe640 1 -- 172.21.15.45:0/3068969733 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c0309a0 con 0x7f6f980cfaf0 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.119+0000 7f6f977fe640 1 -- 172.21.15.45:0/3068969733 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c030d30 con 0x7f6f980cfaf0 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.119+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3068969733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 msgr2=0x7f6f980cfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.119+0000 7f6f9dc98640 1 --2- 172.21.15.45:0/3068969733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980cfed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009870 tx=0x7f6f8c02b980 comp rx=0 tx=0).stop 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.119+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3068969733 shutdown_connections 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.120+0000 7f6f9dc98640 1 --2- 172.21.15.45:0/3068969733 >> v2:172.21.15.190:3300/0 conn(0x7f6f980b69c0 0x7f6f9815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.120+0000 7f6f9dc98640 1 --2- 172.21.15.45:0/3068969733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980cfed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.120+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3068969733 >> 172.21.15.45:0/3068969733 conn(0x7f6f980b4c40 msgr2=0x7f6f980b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:55.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.120+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3068969733 shutdown_connections 2023-11-28T20:47:55.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.120+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3068969733 wait complete. 2023-11-28T20:47:55.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.121+0000 7f6f9dc98640 1 Processor -- start 2023-11-28T20:47:55.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.122+0000 7f6f9dc98640 1 -- start start 2023-11-28T20:47:55.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.122+0000 7f6f9dc98640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f980b6560 0x7f6f981f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f9dc98640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f980b69c0 0x7f6f9815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f9dc98640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f9dc98640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f981f68d0 con 0x7f6f980b69c0 2023-11-28T20:47:55.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f9dc98640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9815a8c0 con 0x7f6f980b6560 2023-11-28T20:47:55.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f9dc98640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9815aba0 con 0x7f6f980cfaf0 2023-11-28T20:47:55.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f980b69c0 0x7f6f9815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f980b69c0 0x7f6f9815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58100/0 (socket says 172.21.15.45:58100) 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f97fff640 1 -- 172.21.15.45:0/3957825754 learned_addr learned my addr 172.21.15.45:0/3957825754 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f9cc96640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f980b6560 0x7f6f981f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f9d497640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f97fff640 1 -- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 msgr2=0x7f6f980b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f97fff640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f97fff640 1 -- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f980b6560 msgr2=0x7f6f981f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f97fff640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f980b6560 0x7f6f981f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.123+0000 7f6f97fff640 1 -- 172.21.15.45:0/3957825754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c0094d0 con 0x7f6f980b69c0 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.124+0000 7f6f97fff640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f980b69c0 0x7f6f9815fe30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400c450 tx=0x7f6f8400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.124+0000 7f6f957fa640 1 -- 172.21.15.45:0/3957825754 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f840052d0 con 0x7f6f980b69c0 2023-11-28T20:47:55.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.124+0000 7f6f957fa640 1 -- 172.21.15.45:0/3957825754 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f84005430 con 0x7f6f980b69c0 2023-11-28T20:47:55.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.124+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9815ae80 con 0x7f6f980b69c0 2023-11-28T20:47:55.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.124+0000 7f6f957fa640 1 -- 172.21.15.45:0/3957825754 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f84012870 con 0x7f6f980b69c0 2023-11-28T20:47:55.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.124+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9815b440 con 0x7f6f980b69c0 2023-11-28T20:47:55.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.125+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f980c9c90 con 0x7f6f980b69c0 2023-11-28T20:47:55.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.126+0000 7f6f957fa640 1 -- 172.21.15.45:0/3957825754 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 89402+0+0 (secure 0 0 0) 0x7f6f840129d0 con 0x7f6f980b69c0 2023-11-28T20:47:55.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.127+0000 7f6f957fa640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f6f7c06aff0 0x7f6f7c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.127+0000 7f6f957fa640 1 -- 172.21.15.45:0/3957825754 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6916+0+0 (secure 0 0 0) 0x7f6f84089910 con 0x7f6f980b69c0 2023-11-28T20:47:55.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.127+0000 7f6f9cc96640 1 -- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f6f7c06aff0 msgr2=0x7f6f7c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:55.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.127+0000 7f6f9cc96640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f6f7c06aff0 0x7f6f7c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:47:55.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.136+0000 7f6f957fa640 1 -- 172.21.15.45:0/3957825754 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f84054270 con 0x7f6f980b69c0 2023-11-28T20:47:55.271 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:55.270+0000 7f0307c3a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:47:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.328+0000 7f6f9cc96640 1 -- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f6f7c06aff0 msgr2=0x7f6f7c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2725659217 2023-11-28T20:47:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.328+0000 7f6f9cc96640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f6f7c06aff0 0x7f6f7c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:47:55.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.337+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f9804e920 con 0x7f6f980b69c0 2023-11-28T20:47:55.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.339+0000 7f6f957fa640 1 -- 172.21.15.45:0/3957825754 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+18072 (secure 0 0 0) 0x7f6f84057890 con 0x7f6f980b69c0 2023-11-28T20:47:55.340 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:55.340 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":58,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:47:55.250355+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:35:33.849797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-28T20:35:34.783808+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":233,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1866894774":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3717946281":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6833/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6832/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/1813745990":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3262798906":"2023-11-29T20:35:55.110774+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1},{"begin":9,"length":1},{"begin":11,"length":1},{"begin":13,"length":1},{"begin":17,"length":1},{"begin":20,"length":1},{"begin":22,"length":1},{"begin":26,"length":2},{"begin":30,"length":1},{"begin":34,"length":1},{"begin":36,"length":5},{"begin":43,"length":2},{"begin":46,"length":2},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":2},{"begin":60,"length":2},{"begin":65,"length":5},{"begin":75,"length":3},{"begin":79,"length":1},{"begin":84,"length":2},{"begin":87,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":94,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":101,"length":2},{"begin":106,"length":1},{"begin":108,"length":1},{"begin":110,"length":2},{"begin":116,"length":2},{"begin":189,"length":45}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":117,"length":1},{"begin":233,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:47:55.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f6f7c06aff0 msgr2=0x7f6f7c06d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f6f7c06aff0 0x7f6f7c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f980b69c0 msgr2=0x7f6f9815fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f980b69c0 0x7f6f9815fe30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400c450 tx=0x7f6f8400c910 comp rx=0 tx=0).stop 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 shutdown_connections 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:6832/2725659217,v1:172.21.15.190:6833/2725659217] conn(0x7f6f7c06aff0 0x7f6f7c06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f980cfaf0 0x7f6f980b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f980b69c0 0x7f6f9815fe30 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 --2- 172.21.15.45:0/3957825754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f980b6560 0x7f6f981f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.341+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 >> 172.21.15.45:0/3957825754 conn(0x7f6f980b4c40 msgr2=0x7f6f9815e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.342+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 shutdown_connections 2023-11-28T20:47:55.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.342+0000 7f6f9dc98640 1 -- 172.21.15.45:0/3957825754 wait complete. 2023-11-28T20:47:55.363 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:35:33.849797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:35:34.783808+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 233, 'snap_epoch': 58, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:47:55.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:47:55.388 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.386+0000 7fb76dc11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:47:55.445 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.443+0000 7fb76dc11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:47:55.562 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.559+0000 7fb76dc11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:47:55.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.572+0000 7f8aa017c640 1 Processor -- start 2023-11-28T20:47:55.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.573+0000 7f8aa017c640 1 -- start start 2023-11-28T20:47:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.575+0000 7f8aa017c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 0x7f8a9815c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.575+0000 7f8aa017c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8a980cdf90 0x7f8a980ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.575+0000 7f8aa017c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9815c5d0 con 0x7f8a9806cbe0 2023-11-28T20:47:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.575+0000 7f8aa017c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9806d160 con 0x7f8a980cdf90 2023-11-28T20:47:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.575+0000 7f8aa017c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9806d440 con 0x7f8a980aec60 2023-11-28T20:47:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.576+0000 7f8a9e719640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8a980cdf90 0x7f8a980ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.576+0000 7f8a9e719640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8a980cdf90 0x7f8a980ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:54900/0 (socket says 172.21.15.45:54900) 2023-11-28T20:47:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.576+0000 7f8a9df18640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 0x7f8a9815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.576+0000 7f8a9e719640 1 -- 172.21.15.45:0/2881288305 learned_addr learned my addr 172.21.15.45:0/2881288305 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.577+0000 7f8a9e719640 1 -- 172.21.15.45:0/2881288305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 msgr2=0x7f8a9815c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.577+0000 7f8a9e719640 1 --2- 172.21.15.45:0/2881288305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 0x7f8a9815c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.577+0000 7f8a9e719640 1 -- 172.21.15.45:0/2881288305 >> v1:172.21.15.190:6789/0 conn(0x7f8a9806cbe0 legacy=0x7f8a9806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.577+0000 7f8a9e719640 1 -- 172.21.15.45:0/2881288305 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9806d720 con 0x7f8a980cdf90 2023-11-28T20:47:55.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.577+0000 7f8a9df18640 1 --2- 172.21.15.45:0/2881288305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 0x7f8a9815c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:47:55.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.578+0000 7f8a9e719640 1 --2- 172.21.15.45:0/2881288305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980cdf90 0x7f8a980ce390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009e50 tx=0x7f8a9002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c649261cbfbec1c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.587+0000 7f8a9cf16640 1 -- 172.21.15.45:0/2881288305 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9002cb00 con 0x7f8a980cdf90 2023-11-28T20:47:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.587+0000 7f8a9cf16640 1 -- 172.21.15.45:0/2881288305 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9002cc60 con 0x7f8a980cdf90 2023-11-28T20:47:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.587+0000 7f8a9cf16640 1 -- 172.21.15.45:0/2881288305 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a900343f0 con 0x7f8a980cdf90 2023-11-28T20:47:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.588+0000 7f8aa017c640 1 -- 172.21.15.45:0/2881288305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980cdf90 msgr2=0x7f8a980ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.588+0000 7f8aa017c640 1 --2- 172.21.15.45:0/2881288305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980cdf90 0x7f8a980ce390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009e50 tx=0x7f8a9002bf60 comp rx=0 tx=0).stop 2023-11-28T20:47:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.588+0000 7f8aa017c640 1 -- 172.21.15.45:0/2881288305 shutdown_connections 2023-11-28T20:47:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.588+0000 7f8aa017c640 1 --2- 172.21.15.45:0/2881288305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980cdf90 0x7f8a980ce390 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.588+0000 7f8aa017c640 1 --2- 172.21.15.45:0/2881288305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 0x7f8a9815c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.588+0000 7f8aa017c640 1 -- 172.21.15.45:0/2881288305 >> 172.21.15.45:0/2881288305 conn(0x7f8a9806a500 msgr2=0x7f8a9806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.589+0000 7f8aa017c640 1 -- 172.21.15.45:0/2881288305 shutdown_connections 2023-11-28T20:47:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.589+0000 7f8aa017c640 1 -- 172.21.15.45:0/2881288305 wait complete. 2023-11-28T20:47:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.590+0000 7f8aa017c640 1 Processor -- start 2023-11-28T20:47:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.591+0000 7f8aa017c640 1 -- start start 2023-11-28T20:47:55.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8aa017c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a9806cbe0 0x7f8a9813f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8aa017c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 0x7f8a981447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8aa017c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980cdf90 0x7f8a9813fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:55.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8aa017c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98144d00 con 0x7f8a980cdf90 2023-11-28T20:47:55.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8aa017c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98143400 con 0x7f8a9806cbe0 2023-11-28T20:47:55.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8aa017c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980b3140 con 0x7f8a980aec60 2023-11-28T20:47:55.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8a9e719640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980cdf90 0x7f8a9813fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8a9e719640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980cdf90 0x7f8a9813fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58110/0 (socket says 172.21.15.45:58110) 2023-11-28T20:47:55.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8a9e719640 1 -- 172.21.15.45:0/3275117704 learned_addr learned my addr 172.21.15.45:0/3275117704 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:55.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8a9df18640 1 --2- 172.21.15.45:0/3275117704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a9806cbe0 0x7f8a9813f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.592+0000 7f8a9d717640 1 --2- 172.21.15.45:0/3275117704 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 0x7f8a981447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:55.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8a9e719640 1 -- 172.21.15.45:0/3275117704 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 msgr2=0x7f8a981447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8a9e719640 1 --2- 172.21.15.45:0/3275117704 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 0x7f8a981447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8a9e719640 1 -- 172.21.15.45:0/3275117704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a9806cbe0 msgr2=0x7f8a9813f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8a9e719640 1 --2- 172.21.15.45:0/3275117704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a9806cbe0 0x7f8a9813f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8a9e719640 1 -- 172.21.15.45:0/3275117704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a94009400 con 0x7f8a980cdf90 2023-11-28T20:47:55.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8a9e719640 1 --2- 172.21.15.45:0/3275117704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980cdf90 0x7f8a9813fb30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009e50 tx=0x7f8a90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:55.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8a8e7fc640 1 -- 172.21.15.45:0/3275117704 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90007ea0 con 0x7f8a980cdf90 2023-11-28T20:47:55.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8a8e7fc640 1 -- 172.21.15.45:0/3275117704 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9003e450 con 0x7f8a980cdf90 2023-11-28T20:47:55.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8a8e7fc640 1 -- 172.21.15.45:0/3275117704 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9003e800 con 0x7f8a980cdf90 2023-11-28T20:47:55.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.593+0000 7f8aa017c640 1 -- 172.21.15.45:0/3275117704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a90009b30 con 0x7f8a980cdf90 2023-11-28T20:47:55.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.594+0000 7f8aa017c640 1 -- 172.21.15.45:0/3275117704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a980b3700 con 0x7f8a980cdf90 2023-11-28T20:47:55.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.595+0000 7f8aa017c640 1 -- 172.21.15.45:0/3275117704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a980733d0 con 0x7f8a980cdf90 2023-11-28T20:47:55.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.597+0000 7f8a8e7fc640 1 -- 172.21.15.45:0/3275117704 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f8a9003e960 con 0x7f8a980cdf90 2023-11-28T20:47:55.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.597+0000 7f8a8e7fc640 1 -- 172.21.15.45:0/3275117704 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7259+0+0 (secure 0 0 0) 0x7f8a900ac090 con 0x7f8a980cdf90 2023-11-28T20:47:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.603+0000 7f8a8e7fc640 1 -- 172.21.15.45:0/3275117704 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a90077030 con 0x7f8a980cdf90 2023-11-28T20:47:55.606 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:55.606+0000 7f0307c3a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:47:55.659 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6f22b9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6f22b9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6f22b9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6f22b9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6f22b9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6eeab2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6eeab2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6eeab2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6eeab2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:47:55.657+0000 7fb6eeab2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:47:55.668 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:55.668+0000 7f0307c3a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:47:55.793 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:47:55.792+0000 7f0307c3a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:47:55.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.803+0000 7f8aa017c640 1 -- 172.21.15.45:0/3275117704 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a980af830 con 0x7f8a980cdf90 2023-11-28T20:47:55.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.803+0000 7f8a8e7fc640 1 -- 172.21.15.45:0/3275117704 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a90043020 con 0x7f8a980cdf90 2023-11-28T20:47:55.808 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:47:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.806+0000 7f8aa017c640 1 -- 172.21.15.45:0/3275117704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980cdf90 msgr2=0x7f8a9813fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.806+0000 7f8aa017c640 1 --2- 172.21.15.45:0/3275117704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980cdf90 0x7f8a9813fb30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009e50 tx=0x7f8a90004a40 comp rx=0 tx=0).stop 2023-11-28T20:47:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.806+0000 7f8aa017c640 1 -- 172.21.15.45:0/3275117704 shutdown_connections 2023-11-28T20:47:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.806+0000 7f8aa017c640 1 --2- 172.21.15.45:0/3275117704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980cdf90 0x7f8a9813fb30 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.806+0000 7f8aa017c640 1 --2- 172.21.15.45:0/3275117704 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980aec60 0x7f8a981447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.806+0000 7f8aa017c640 1 --2- 172.21.15.45:0/3275117704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a9806cbe0 0x7f8a9813f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.806+0000 7f8aa017c640 1 -- 172.21.15.45:0/3275117704 >> 172.21.15.45:0/3275117704 conn(0x7f8a9806a500 msgr2=0x7f8a980cfbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:55.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.806+0000 7f8aa017c640 1 -- 172.21.15.45:0/3275117704 shutdown_connections 2023-11-28T20:47:55.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:55.806+0000 7f8aa017c640 1 -- 172.21.15.45:0/3275117704 wait complete. 2023-11-28T20:47:55.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:47:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.031+0000 7f07a826b640 1 Processor -- start 2023-11-28T20:47:56.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.031+0000 7f07a826b640 1 -- start start 2023-11-28T20:47:56.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.032+0000 7f07a826b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a006c350 0x7f07a015f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.033+0000 7f07a826b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a015ecd0 con 0x7f07a015e840 2023-11-28T20:47:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.033+0000 7f07a826b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a015f290 con 0x7f07a006ac10 2023-11-28T20:47:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.033+0000 7f07a826b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a0072020 con 0x7f07a006c350 2023-11-28T20:47:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.033+0000 7f07a6007640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07a015e840 0x7f07a006a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36062/0 (socket says 172.21.15.45:36062) 2023-11-28T20:47:56.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.033+0000 7f07a6007640 1 -- 172.21.15.45:0/1724835087 learned_addr learned my addr 172.21.15.45:0/1724835087 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.033+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881605752 0 0) 0x7f07a015ecd0 con 0x7f07a015e840 2023-11-28T20:47:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.033+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f078c003520 con 0x7f07a015e840 2023-11-28T20:47:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.033+0000 7f07a6808640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a006c350 0x7f07a015f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2082432255 0 0) 0x7f078c003520 con 0x7f07a015e840 2023-11-28T20:47:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a006c350 msgr2=0x7f07a015f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a006c350 0x7f07a015f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 >> v1:172.21.15.190:6789/0 conn(0x7f07a006ac10 legacy=0x7f07a00cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a0072300 con 0x7f07a015e840 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f079c001670 con 0x7f07a015e840 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 >> v1:172.21.15.45:6789/0 conn(0x7f07a015e840 legacy=0x7f07a006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0039a0 0x7f078c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f078c0047c0 0x7f078c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a006c350 0x7f078c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a0072020 con 0x7f078c0047c0 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c00bb40 con 0x7f078c0039a0 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078c00bd00 con 0x7f07a006c350 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a5806640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f078c0047c0 0x7f078c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.034+0000 7f07a6007640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0039a0 0x7f078c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.035+0000 7f07a6808640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a006c350 0x7f078c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.035+0000 7f07a5806640 1 -- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a006c350 msgr2=0x7f078c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.035+0000 7f07a5806640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a006c350 0x7f078c005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.035+0000 7f07a5806640 1 -- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0039a0 msgr2=0x7f078c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.035+0000 7f07a5806640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0039a0 0x7f078c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.035+0000 7f07a5806640 1 -- 172.21.15.45:0/1724835087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c003520 con 0x7f078c0047c0 2023-11-28T20:47:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.035+0000 7f07a6007640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0039a0 0x7f078c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:47:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.036+0000 7f07a5806640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f078c0047c0 0x7f078c004c40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0790001750 tx=0x7f078c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb1498a99e275c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.036+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07900052d0 con 0x7f078c0047c0 2023-11-28T20:47:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.036+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079002e050 con 0x7f078c0047c0 2023-11-28T20:47:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.036+0000 7f07a5005640 1 -- 172.21.15.45:0/1724835087 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07900035f0 con 0x7f078c0047c0 2023-11-28T20:47:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.037+0000 7f07a826b640 1 -- 172.21.15.45:0/1724835087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f078c0047c0 msgr2=0x7f078c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.037+0000 7f07a826b640 1 --2- 172.21.15.45:0/1724835087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f078c0047c0 0x7f078c004c40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0790001750 tx=0x7f078c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.037+0000 7f07a6808640 1 -- 172.21.15.45:0/1724835087 reap_dead start 2023-11-28T20:47:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.037+0000 7f07a826b640 1 -- 172.21.15.45:0/1724835087 shutdown_connections 2023-11-28T20:47:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.037+0000 7f07a826b640 1 -- 172.21.15.45:0/1724835087 >> 172.21.15.45:0/1724835087 conn(0x7f07a00ced60 msgr2=0x7f07a00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.037+0000 7f07a826b640 1 -- 172.21.15.45:0/1724835087 shutdown_connections 2023-11-28T20:47:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.037+0000 7f07a826b640 1 -- 172.21.15.45:0/1724835087 wait complete. 2023-11-28T20:47:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.039+0000 7f07a826b640 1 Processor -- start 2023-11-28T20:47:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.039+0000 7f07a826b640 1 -- start start 2023-11-28T20:47:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.040+0000 7f07a826b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0047c0 0x7f07a0156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.040+0000 7f07a826b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a0099b20 0x7f07a015a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.040+0000 7f07a826b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07a009a930 0x7f07a00c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.040+0000 7f07a826b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00c4730 con 0x7f07a009a930 2023-11-28T20:47:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.040+0000 7f07a826b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a0156940 con 0x7f078c0047c0 2023-11-28T20:47:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.040+0000 7f07a826b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a015ac50 con 0x7f07a0099b20 2023-11-28T20:47:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.040+0000 7f07a6808640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07a009a930 0x7f07a00c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.040+0000 7f07a6808640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07a009a930 0x7f07a00c9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58128/0 (socket says 172.21.15.45:58128) 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.040+0000 7f07a6808640 1 -- 172.21.15.45:0/1209458271 learned_addr learned my addr 172.21.15.45:0/1209458271 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.041+0000 7f07a5806640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a0099b20 0x7f07a015a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.041+0000 7f07a6007640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0047c0 0x7f07a0156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.041+0000 7f07a6808640 1 -- 172.21.15.45:0/1209458271 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a0099b20 msgr2=0x7f07a015a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.041+0000 7f07a6808640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a0099b20 0x7f07a015a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.041+0000 7f07a6808640 1 -- 172.21.15.45:0/1209458271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0047c0 msgr2=0x7f07a0156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.041+0000 7f07a6808640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0047c0 0x7f07a0156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.041+0000 7f07a6808640 1 -- 172.21.15.45:0/1209458271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a00c3ab0 con 0x7f07a009a930 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.041+0000 7f07a6007640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0047c0 0x7f07a0156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.041+0000 7f07a6808640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07a009a930 0x7f07a00c9b00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f07a00cfd60 tx=0x7f0798009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.042+0000 7f0796ffd640 1 -- 172.21.15.45:0/1209458271 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798007c70 con 0x7f07a009a930 2023-11-28T20:47:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.042+0000 7f0796ffd640 1 -- 172.21.15.45:0/1209458271 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798007dd0 con 0x7f07a009a930 2023-11-28T20:47:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.042+0000 7f0796ffd640 1 -- 172.21.15.45:0/1209458271 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798014640 con 0x7f07a009a930 2023-11-28T20:47:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.042+0000 7f07a826b640 1 -- 172.21.15.45:0/1209458271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a00c3d90 con 0x7f07a009a930 2023-11-28T20:47:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.042+0000 7f07a826b640 1 -- 172.21.15.45:0/1209458271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a00c4350 con 0x7f07a009a930 2023-11-28T20:47:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.043+0000 7f07a826b640 1 -- 172.21.15.45:0/1209458271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a00c4870 con 0x7f07a009a930 2023-11-28T20:47:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.044+0000 7f0796ffd640 1 -- 172.21.15.45:0/1209458271 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f079801d3b0 con 0x7f07a009a930 2023-11-28T20:47:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.045+0000 7f0796ffd640 1 -- 172.21.15.45:0/1209458271 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7259+0+0 (secure 0 0 0) 0x7f0798089420 con 0x7f07a009a930 2023-11-28T20:47:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.049+0000 7f0796ffd640 1 -- 172.21.15.45:0/1209458271 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0798054220 con 0x7f07a009a930 2023-11-28T20:47:56.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.249+0000 7f07a826b640 1 -- 172.21.15.45:0/1209458271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f07a00733d0 con 0x7f07a009a930 2023-11-28T20:47:56.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.250+0000 7f0796ffd640 1 -- 172.21.15.45:0/1209458271 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0798057840 con 0x7f07a009a930 2023-11-28T20:47:56.251 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:47:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.253+0000 7f07a826b640 1 -- 172.21.15.45:0/1209458271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07a009a930 msgr2=0x7f07a00c9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.253+0000 7f07a826b640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07a009a930 0x7f07a00c9b00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f07a00cfd60 tx=0x7f0798009cd0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.253+0000 7f07a826b640 1 -- 172.21.15.45:0/1209458271 shutdown_connections 2023-11-28T20:47:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.253+0000 7f07a826b640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07a009a930 0x7f07a00c9b00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.253+0000 7f07a826b640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a0099b20 0x7f07a015a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.253+0000 7f07a826b640 1 --2- 172.21.15.45:0/1209458271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f078c0047c0 0x7f07a0156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.253+0000 7f07a826b640 1 -- 172.21.15.45:0/1209458271 >> 172.21.15.45:0/1209458271 conn(0x7f07a00ced60 msgr2=0x7f07a00ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.253+0000 7f07a826b640 1 -- 172.21.15.45:0/1209458271 shutdown_connections 2023-11-28T20:47:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.253+0000 7f07a826b640 1 -- 172.21.15.45:0/1209458271 wait complete. 2023-11-28T20:47:56.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:47:56.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.502+0000 7f9784967640 1 Processor -- start 2023-11-28T20:47:56.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.502+0000 7f9784967640 1 -- start start 2023-11-28T20:47:56.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.503+0000 7f9784967640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f97800b3040 0x7f978006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.503+0000 7f9784967640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97800b3420 0x7f97800ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.503+0000 7f9784967640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978006d410 con 0x7f978015b270 2023-11-28T20:47:56.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.503+0000 7f9784967640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97800ceaa0 con 0x7f97800b3420 2023-11-28T20:47:56.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.503+0000 7f9784967640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978006d6f0 con 0x7f97800b3040 2023-11-28T20:47:56.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.504+0000 7f977ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97800b3420 0x7f97800ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.504+0000 7f977ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97800b3420 0x7f97800ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58138/0 (socket says 172.21.15.45:58138) 2023-11-28T20:47:56.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.504+0000 7f977ffff640 1 -- 172.21.15.45:0/3166404963 learned_addr learned my addr 172.21.15.45:0/3166404963 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:56.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.504+0000 7f977ffff640 1 -- 172.21.15.45:0/3166404963 >> v2:172.21.15.190:3300/0 conn(0x7f97800b3040 msgr2=0x7f978006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:56.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.504+0000 7f977ffff640 1 --2- 172.21.15.45:0/3166404963 >> v2:172.21.15.190:3300/0 conn(0x7f97800b3040 0x7f978006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.504+0000 7f977ffff640 1 -- 172.21.15.45:0/3166404963 >> v1:172.21.15.190:6789/0 conn(0x7f978015b270 legacy=0x7f978015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:56.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.504+0000 7f977ffff640 1 -- 172.21.15.45:0/3166404963 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978006d9d0 con 0x7f97800b3420 2023-11-28T20:47:56.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.505+0000 7f977ffff640 1 --2- 172.21.15.45:0/3166404963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97800b3420 0x7f97800ccc90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f97800b4410 tx=0x7f977802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448cf66172d46999 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:56.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.506+0000 7f977effd640 1 -- 172.21.15.45:0/3166404963 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977802c960 con 0x7f97800b3420 2023-11-28T20:47:56.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.506+0000 7f977effd640 1 -- 172.21.15.45:0/3166404963 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977802cac0 con 0x7f97800b3420 2023-11-28T20:47:56.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.506+0000 7f977effd640 1 -- 172.21.15.45:0/3166404963 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977802ce50 con 0x7f97800b3420 2023-11-28T20:47:56.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.506+0000 7f9784967640 1 -- 172.21.15.45:0/3166404963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97800b3420 msgr2=0x7f97800ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.506+0000 7f9784967640 1 --2- 172.21.15.45:0/3166404963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97800b3420 0x7f97800ccc90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f97800b4410 tx=0x7f977802bdc0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.507+0000 7f9784967640 1 -- 172.21.15.45:0/3166404963 shutdown_connections 2023-11-28T20:47:56.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.507+0000 7f9784967640 1 --2- 172.21.15.45:0/3166404963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97800b3420 0x7f97800ccc90 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.507+0000 7f9784967640 1 --2- 172.21.15.45:0/3166404963 >> v2:172.21.15.190:3300/0 conn(0x7f97800b3040 0x7f978006cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.507+0000 7f9784967640 1 -- 172.21.15.45:0/3166404963 >> 172.21.15.45:0/3166404963 conn(0x7f97800b1050 msgr2=0x7f97800cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:56.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.507+0000 7f9784967640 1 -- 172.21.15.45:0/3166404963 shutdown_connections 2023-11-28T20:47:56.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.507+0000 7f9784967640 1 -- 172.21.15.45:0/3166404963 wait complete. 2023-11-28T20:47:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.509+0000 7f9784967640 1 Processor -- start 2023-11-28T20:47:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.509+0000 7f9784967640 1 -- start start 2023-11-28T20:47:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f9784967640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97800b3040 0x7f978014e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f9784967640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97800b3420 0x7f9780071d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f9784967640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978015b270 0x7f97801595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f9784967640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9780099d90 con 0x7f978015b270 2023-11-28T20:47:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f9784967640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9780072240 con 0x7f97800b3040 2023-11-28T20:47:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f9784967640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978015cbf0 con 0x7f97800b3420 2023-11-28T20:47:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f977ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978015b270 0x7f97801595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f977ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978015b270 0x7f97801595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58150/0 (socket says 172.21.15.45:58150) 2023-11-28T20:47:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f977ffff640 1 -- 172.21.15.45:0/2052798351 learned_addr learned my addr 172.21.15.45:0/2052798351 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f977f7fe640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97800b3040 0x7f978014e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.510+0000 7f9777fff640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97800b3420 0x7f9780071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f977ffff640 1 -- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97800b3420 msgr2=0x7f9780071d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f977ffff640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97800b3420 0x7f9780071d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f977ffff640 1 -- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97800b3040 msgr2=0x7f978014e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f977ffff640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97800b3040 0x7f978014e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f977ffff640 1 -- 172.21.15.45:0/2052798351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778009b30 con 0x7f978015b270 2023-11-28T20:47:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f9777fff640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97800b3420 0x7f9780071d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f977ffff640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978015b270 0x7f97801595e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f977802c300 tx=0x7f9778004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f977cff9640 1 -- 172.21.15.45:0/2052798351 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977802ce30 con 0x7f978015b270 2023-11-28T20:47:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f977cff9640 1 -- 172.21.15.45:0/2052798351 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9778040470 con 0x7f978015b270 2023-11-28T20:47:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f977cff9640 1 -- 172.21.15.45:0/2052798351 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9778040800 con 0x7f978015b270 2023-11-28T20:47:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978015ced0 con 0x7f978015b270 2023-11-28T20:47:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.511+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978015d490 con 0x7f978015b270 2023-11-28T20:47:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.513+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978015d620 con 0x7f978015b270 2023-11-28T20:47:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.513+0000 7f977cff9640 1 -- 172.21.15.45:0/2052798351 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f9778040960 con 0x7f978015b270 2023-11-28T20:47:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.514+0000 7f977cff9640 1 -- 172.21.15.45:0/2052798351 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7259+0+0 (secure 0 0 0) 0x7f97780ac640 con 0x7f978015b270 2023-11-28T20:47:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.519+0000 7f977cff9640 1 -- 172.21.15.45:0/2052798351 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9778077440 con 0x7f978015b270 2023-11-28T20:47:56.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.590+0000 7f977cff9640 1 -- 172.21.15.45:0/2052798351 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 16) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f9778076ef0 con 0x7f978015b270 2023-11-28T20:47:56.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.591+0000 7f977cff9640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f97600710a0 0x7f9760073490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.592+0000 7f977f7fe640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f97600710a0 0x7f9760073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.594+0000 7f977f7fe640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f97600710a0 0x7f9760073490 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9770009890 tx=0x7f9770008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.720+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f978009d0f0 con 0x7f978015b270 2023-11-28T20:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.721+0000 7f977cff9640 1 -- 172.21.15.45:0/2052798351 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f977808f020 con 0x7f978015b270 2023-11-28T20:47:56.722 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:47:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f97600710a0 msgr2=0x7f9760073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f97600710a0 0x7f9760073490 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9770009890 tx=0x7f9770008040 comp rx=0 tx=0).stop 2023-11-28T20:47:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978015b270 msgr2=0x7f97801595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978015b270 0x7f97801595e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f977802c300 tx=0x7f9778004a40 comp rx=0 tx=0).stop 2023-11-28T20:47:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 shutdown_connections 2023-11-28T20:47:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f97600710a0 0x7f9760073490 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978015b270 0x7f97801595e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97800b3420 0x7f9780071d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 --2- 172.21.15.45:0/2052798351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97800b3040 0x7f978014e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 >> 172.21.15.45:0/2052798351 conn(0x7f97800b1050 msgr2=0x7f97800add80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:56.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 shutdown_connections 2023-11-28T20:47:56.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.724+0000 7f9784967640 1 -- 172.21.15.45:0/2052798351 wait complete. 2023-11-28T20:47:56.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:47:56.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.976+0000 7fc0f60bb640 1 Processor -- start 2023-11-28T20:47:56.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.976+0000 7fc0f60bb640 1 -- start start 2023-11-28T20:47:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.978+0000 7fc0f60bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f0155780 0x7fc0f00abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.978+0000 7fc0f60bb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc0f00ae540 0x7fc0f00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.978+0000 7fc0f60bb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f015c3c0 con 0x7fc0f00ce630 2023-11-28T20:47:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.978+0000 7fc0f60bb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f00ceac0 con 0x7fc0f00ae540 2023-11-28T20:47:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.978+0000 7fc0f60bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0155b60 con 0x7fc0f0155780 2023-11-28T20:47:56.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.978+0000 7fc0f58ba640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc0f00ce630 0x7fc0f015ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36074/0 (socket says 172.21.15.45:36074) 2023-11-28T20:47:56.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.978+0000 7fc0f58ba640 1 -- 172.21.15.45:0/2659253737 learned_addr learned my addr 172.21.15.45:0/2659253737 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:56.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0f48b8640 1 --2- 172.21.15.45:0/2659253737 >> v2:172.21.15.190:3300/0 conn(0x7fc0f00ae540 0x7fc0f00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217741282 0 0) 0x7fc0f015c3c0 con 0x7fc0f00ce630 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0dc003520 con 0x7fc0f00ce630 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0f50b9640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f0155780 0x7fc0f00abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3745921122 0 0) 0x7fc0dc003520 con 0x7fc0f00ce630 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f0155780 msgr2=0x7fc0f00abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f0155780 0x7fc0f00abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 >> v2:172.21.15.190:3300/0 conn(0x7fc0f00ae540 msgr2=0x7fc0f00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 --2- 172.21.15.45:0/2659253737 >> v2:172.21.15.190:3300/0 conn(0x7fc0f00ae540 0x7fc0f00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f015ba90 con 0x7fc0f00ce630 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc0e80019f0 con 0x7fc0f00ce630 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 >> v1:172.21.15.45:6789/0 conn(0x7fc0f00ce630 legacy=0x7fc0f015ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f0155780 0x7fc0dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.979+0000 7fc0e7fff640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc004d00 0x7fc0dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f00ceac0 con 0x7fc0dc003970 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0155b60 con 0x7fc0dc004d00 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc00b980 con 0x7fc0f0155780 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0f50b9640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0f48b8640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f0155780 0x7fc0dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0f58ba640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc004d00 0x7fc0dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0f50b9640 1 -- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f0155780 msgr2=0x7fc0dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0f50b9640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f0155780 0x7fc0dc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0f50b9640 1 -- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc004d00 msgr2=0x7fc0dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:56.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0f50b9640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc004d00 0x7fc0dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.980+0000 7fc0f50b9640 1 -- 172.21.15.45:0/2659253737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc003520 con 0x7fc0dc003970 2023-11-28T20:47:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.982+0000 7fc0f50b9640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0dc003d90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009ac0 tx=0x7fc0dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531fb0ecbeb3c75f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.982+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec02c6b0 con 0x7fc0dc003970 2023-11-28T20:47:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.982+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec02c810 con 0x7fc0dc003970 2023-11-28T20:47:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.982+0000 7fc0e7fff640 1 -- 172.21.15.45:0/2659253737 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec02cba0 con 0x7fc0dc003970 2023-11-28T20:47:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.983+0000 7fc0f60bb640 1 -- 172.21.15.45:0/2659253737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 msgr2=0x7fc0dc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.983+0000 7fc0f60bb640 1 --2- 172.21.15.45:0/2659253737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0dc003d90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009ac0 tx=0x7fc0dc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.983+0000 7fc0f58ba640 1 -- 172.21.15.45:0/2659253737 reap_dead start 2023-11-28T20:47:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.983+0000 7fc0f60bb640 1 -- 172.21.15.45:0/2659253737 shutdown_connections 2023-11-28T20:47:56.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.983+0000 7fc0f60bb640 1 -- 172.21.15.45:0/2659253737 >> 172.21.15.45:0/2659253737 conn(0x7fc0f006ab30 msgr2=0x7fc0f00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:56.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.983+0000 7fc0f60bb640 1 -- 172.21.15.45:0/2659253737 shutdown_connections 2023-11-28T20:47:56.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.983+0000 7fc0f60bb640 1 -- 172.21.15.45:0/2659253737 wait complete. 2023-11-28T20:47:56.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.985+0000 7fc0f60bb640 1 Processor -- start 2023-11-28T20:47:56.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.985+0000 7fc0f60bb640 1 -- start start 2023-11-28T20:47:56.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.986+0000 7fc0f60bb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0f009bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.986+0000 7fc0f60bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0f009c420 0x7fc0f00aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.986+0000 7fc0f60bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f00aef40 0x7fc0f01531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.986+0000 7fc0f60bb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0099a40 con 0x7fc0dc003970 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.986+0000 7fc0f60bb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f009a000 con 0x7fc0f009c420 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.986+0000 7fc0f60bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0137530 con 0x7fc0f00aef40 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.986+0000 7fc0f50b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0f009bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.986+0000 7fc0f50b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0f009bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58172/0 (socket says 172.21.15.45:58172) 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.986+0000 7fc0f50b9640 1 -- 172.21.15.45:0/3898710181 learned_addr learned my addr 172.21.15.45:0/3898710181 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0f48b8640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0f009c420 0x7fc0f00aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0f50b9640 1 -- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f00aef40 msgr2=0x7fc0f01531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0f58ba640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f00aef40 0x7fc0f01531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0f50b9640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f00aef40 0x7fc0f01531e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0f50b9640 1 -- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0f009c420 msgr2=0x7fc0f00aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0f50b9640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0f009c420 0x7fc0f00aea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0f50b9640 1 -- 172.21.15.45:0/3898710181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f00af570 con 0x7fc0dc003970 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0f48b8640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0f009c420 0x7fc0f00aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0f50b9640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0f009bee0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec000c00 tx=0x7fc0ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0e5ffb640 1 -- 172.21.15.45:0/3898710181 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec003810 con 0x7fc0dc003970 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0e5ffb640 1 -- 172.21.15.45:0/3898710181 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec003990 con 0x7fc0dc003970 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.987+0000 7fc0e5ffb640 1 -- 172.21.15.45:0/3898710181 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec003d40 con 0x7fc0dc003970 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.988+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f00af850 con 0x7fc0dc003970 2023-11-28T20:47:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.988+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f00b05e0 con 0x7fc0dc003970 2023-11-28T20:47:56.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.989+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f009a270 con 0x7fc0dc003970 2023-11-28T20:47:56.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.990+0000 7fc0e5ffb640 1 -- 172.21.15.45:0/3898710181 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fc0ec03b070 con 0x7fc0dc003970 2023-11-28T20:47:56.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.991+0000 7fc0e5ffb640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc0d406ac90 0x7fc0d406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:56.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.991+0000 7fc0e5ffb640 1 -- 172.21.15.45:0/3898710181 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7259+0+0 (secure 0 0 0) 0x7fc0ec0ab640 con 0x7fc0dc003970 2023-11-28T20:47:56.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.991+0000 7fc0f48b8640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc0d406ac90 0x7fc0d406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:56.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.992+0000 7fc0f48b8640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc0d406ac90 0x7fc0d406d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0154400 tx=0x7fc0e0008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:56.995+0000 7fc0e5ffb640 1 -- 172.21.15.45:0/3898710181 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc0ec076520 con 0x7fc0dc003970 2023-11-28T20:47:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.226+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0f00afa90 con 0x7fc0dc003970 2023-11-28T20:47:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.227+0000 7fc0e5ffb640 1 -- 172.21.15.45:0/3898710181 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fc0ec079b40 con 0x7fc0dc003970 2023-11-28T20:47:57.230 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:57.230 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3050590116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3050590116},{"type":"v1","addr":"172.21.15.45:6835","nonce":3050590116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2465910573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2465910573},{"type":"v1","addr":"172.21.15.190:6836","nonce":2465910573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4090116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4090116745},{"type":"v1","addr":"172.21.15.190:6837","nonce":4090116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:35:35.751211+0000","modified":"2023-11-28T20:35:38.829781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4945":{"gid":4945,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6834/1984946086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1984946086},{"type":"v1","addr":"172.21.15.45:6834","nonce":1984946086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T20:47:57.230 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 18 2023-11-28T20:47:57.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc0d406ac90 msgr2=0x7fc0d406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc0d406ac90 0x7fc0d406d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0154400 tx=0x7fc0e0008040 comp rx=0 tx=0).stop 2023-11-28T20:47:57.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 msgr2=0x7fc0f009bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0f009bee0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec000c00 tx=0x7fc0ec004730 comp rx=0 tx=0).stop 2023-11-28T20:47:57.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 shutdown_connections 2023-11-28T20:47:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc0d406ac90 0x7fc0d406d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0f00aef40 0x7fc0f01531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0f009c420 0x7fc0f00aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 --2- 172.21.15.45:0/3898710181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0dc003970 0x7fc0f009bee0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 >> 172.21.15.45:0/3898710181 conn(0x7fc0f006ab30 msgr2=0x7fc0f015a820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 shutdown_connections 2023-11-28T20:47:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.230+0000 7fc0f60bb640 1 -- 172.21.15.45:0/3898710181 wait complete. 2023-11-28T20:47:57.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:47:57.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.476+0000 7fc565d16640 1 Processor -- start 2023-11-28T20:47:57.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.476+0000 7fc565d16640 1 -- start start 2023-11-28T20:47:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.477+0000 7fc565d16640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc5600d02d0 0x7fc5600d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.477+0000 7fc565d16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5600d1ab0 0x7fc5600b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.477+0000 7fc565d16640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5600d2230 con 0x7fc56015d550 2023-11-28T20:47:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.477+0000 7fc565d16640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5600d2510 con 0x7fc5600d02d0 2023-11-28T20:47:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.477+0000 7fc565d16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600d27f0 con 0x7fc5600d1ab0 2023-11-28T20:47:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.478+0000 7fc557fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc5600d02d0 0x7fc5600d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.478+0000 7fc565515640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5600d1ab0 0x7fc5600b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.478+0000 7fc557fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc5600d02d0 0x7fc5600d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:54978/0 (socket says 172.21.15.45:54978) 2023-11-28T20:47:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.478+0000 7fc557fff640 1 -- 172.21.15.45:0/2725097702 learned_addr learned my addr 172.21.15.45:0/2725097702 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.479+0000 7fc565515640 1 -- 172.21.15.45:0/2725097702 >> v2:172.21.15.190:3300/0 conn(0x7fc5600d02d0 msgr2=0x7fc5600d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.479+0000 7fc565515640 1 --2- 172.21.15.45:0/2725097702 >> v2:172.21.15.190:3300/0 conn(0x7fc5600d02d0 0x7fc5600d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.479+0000 7fc565515640 1 -- 172.21.15.45:0/2725097702 >> v1:172.21.15.190:6789/0 conn(0x7fc56015d550 legacy=0x7fc5600cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.479+0000 7fc565515640 1 -- 172.21.15.45:0/2725097702 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5600b6a30 con 0x7fc5600d1ab0 2023-11-28T20:47:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.479+0000 7fc557fff640 1 --2- 172.21.15.45:0/2725097702 >> v2:172.21.15.190:3300/0 conn(0x7fc5600d02d0 0x7fc5600d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:47:57.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.480+0000 7fc565515640 1 --2- 172.21.15.45:0/2725097702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5600d1ab0 0x7fc5600b6f50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc5580097a0 tx=0x7fc55802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f888ab67c73de40 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.481+0000 7fc5577fe640 1 -- 172.21.15.45:0/2725097702 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558031830 con 0x7fc5600d1ab0 2023-11-28T20:47:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.481+0000 7fc5577fe640 1 -- 172.21.15.45:0/2725097702 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558031990 con 0x7fc5600d1ab0 2023-11-28T20:47:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.481+0000 7fc5577fe640 1 -- 172.21.15.45:0/2725097702 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558031d20 con 0x7fc5600d1ab0 2023-11-28T20:47:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.482+0000 7fc565d16640 1 -- 172.21.15.45:0/2725097702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5600d1ab0 msgr2=0x7fc5600b6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.482+0000 7fc565d16640 1 --2- 172.21.15.45:0/2725097702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5600d1ab0 0x7fc5600b6f50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc5580097a0 tx=0x7fc55802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.482+0000 7fc565d16640 1 -- 172.21.15.45:0/2725097702 shutdown_connections 2023-11-28T20:47:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.482+0000 7fc565d16640 1 --2- 172.21.15.45:0/2725097702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5600d1ab0 0x7fc5600b6f50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.482+0000 7fc565d16640 1 --2- 172.21.15.45:0/2725097702 >> v2:172.21.15.190:3300/0 conn(0x7fc5600d02d0 0x7fc5600d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.482+0000 7fc565d16640 1 -- 172.21.15.45:0/2725097702 >> 172.21.15.45:0/2725097702 conn(0x7fc56006ac70 msgr2=0x7fc5600b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.482+0000 7fc565d16640 1 -- 172.21.15.45:0/2725097702 shutdown_connections 2023-11-28T20:47:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.482+0000 7fc565d16640 1 -- 172.21.15.45:0/2725097702 wait complete. 2023-11-28T20:47:57.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.484+0000 7fc565d16640 1 Processor -- start 2023-11-28T20:47:57.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.484+0000 7fc565d16640 1 -- start start 2023-11-28T20:47:57.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.485+0000 7fc565d16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5600d02d0 0x7fc5601567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.485+0000 7fc565d16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5600d1ab0 0x7fc5601f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.485+0000 7fc565d16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc56015d550 0x7fc5600c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.485+0000 7fc565d16640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600c7110 con 0x7fc5600d1ab0 2023-11-28T20:47:57.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.485+0000 7fc565d16640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5601f3830 con 0x7fc5600d02d0 2023-11-28T20:47:57.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.485+0000 7fc565d16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5601f3b10 con 0x7fc56015d550 2023-11-28T20:47:57.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.485+0000 7fc557fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5600d1ab0 0x7fc5601f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.485+0000 7fc557fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5600d1ab0 0x7fc5601f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58180/0 (socket says 172.21.15.45:58180) 2023-11-28T20:47:57.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.485+0000 7fc557fff640 1 -- 172.21.15.45:0/3862410009 learned_addr learned my addr 172.21.15.45:0/3862410009 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:57.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc564d14640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5600d02d0 0x7fc5601567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc565515640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc56015d550 0x7fc5600c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc557fff640 1 -- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc56015d550 msgr2=0x7fc5600c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc557fff640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc56015d550 0x7fc5600c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc557fff640 1 -- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5600d02d0 msgr2=0x7fc5601567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc557fff640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5600d02d0 0x7fc5601567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc557fff640 1 -- 172.21.15.45:0/3862410009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5500094d0 con 0x7fc5600d1ab0 2023-11-28T20:47:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc565515640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc56015d550 0x7fc5600c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc557fff640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5600d1ab0 0x7fc5601f2f20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e9d0 tx=0x7fc55000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc5557fa640 1 -- 172.21.15.45:0/3862410009 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55000ca60 con 0x7fc5600d1ab0 2023-11-28T20:47:57.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc5557fa640 1 -- 172.21.15.45:0/3862410009 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55000cbc0 con 0x7fc5600d1ab0 2023-11-28T20:47:57.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc5557fa640 1 -- 172.21.15.45:0/3862410009 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550013470 con 0x7fc5600d1ab0 2023-11-28T20:47:57.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.486+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc558009400 con 0x7fc5600d1ab0 2023-11-28T20:47:57.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.487+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5600c6da0 con 0x7fc5600d1ab0 2023-11-28T20:47:57.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.488+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5600c72c0 con 0x7fc5600d1ab0 2023-11-28T20:47:57.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.489+0000 7fc5557fa640 1 -- 172.21.15.45:0/3862410009 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fc5500135d0 con 0x7fc5600d1ab0 2023-11-28T20:47:57.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.489+0000 7fc5557fa640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc54c06acc0 0x7fc54c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.490+0000 7fc5557fa640 1 -- 172.21.15.45:0/3862410009 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7259+0+0 (secure 0 0 0) 0x7fc550089b90 con 0x7fc5600d1ab0 2023-11-28T20:47:57.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.490+0000 7fc564d14640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc54c06acc0 0x7fc54c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.491+0000 7fc564d14640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc54c06acc0 0x7fc54c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc55c001ca0 tx=0x7fc55c009040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:57.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.494+0000 7fc5557fa640 1 -- 172.21.15.45:0/3862410009 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5600c72c0 con 0x7fc5600d1ab0 2023-11-28T20:47:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.606+0000 7fc5557fa640 1 -- 172.21.15.45:0/3862410009 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 17) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fc550054c70 con 0x7fc5600d1ab0 2023-11-28T20:47:57.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.699+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5600c7250 con 0x7fc5600d1ab0 2023-11-28T20:47:57.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.702+0000 7fc5557fa640 1 -- 172.21.15.45:0/3862410009 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+18574 (secure 0 0 0) 0x7fc55004ae60 con 0x7fc5600d1ab0 2023-11-28T20:47:57.706 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:57.707 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":59,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:47:55.565091+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:35:33.849797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-28T20:35:34.783808+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":233,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/2065659095":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/3262798906":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/664590003":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1813745990":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6832/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6833/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3717946281":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3679315161":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1881573438":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1866894774":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3153178694":"2023-11-29T20:47:55.564970+0000","172.21.15.190:6832/2725659217":"2023-11-29T20:47:55.564970+0000","172.21.15.190:6833/2725659217":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1599738782":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/3278716326":"2023-11-29T20:47:55.564970+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1},{"begin":9,"length":1},{"begin":11,"length":1},{"begin":13,"length":1},{"begin":17,"length":1},{"begin":20,"length":1},{"begin":22,"length":1},{"begin":26,"length":2},{"begin":30,"length":1},{"begin":34,"length":1},{"begin":36,"length":5},{"begin":43,"length":2},{"begin":46,"length":2},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":2},{"begin":60,"length":2},{"begin":65,"length":5},{"begin":75,"length":3},{"begin":79,"length":1},{"begin":84,"length":2},{"begin":87,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":94,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":101,"length":2},{"begin":106,"length":1},{"begin":108,"length":1},{"begin":110,"length":2},{"begin":116,"length":2},{"begin":189,"length":45}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:47:57.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc54c06acc0 msgr2=0x7fc54c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc54c06acc0 0x7fc54c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc55c001ca0 tx=0x7fc55c009040 comp rx=0 tx=0).stop 2023-11-28T20:47:57.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5600d1ab0 msgr2=0x7fc5601f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5600d1ab0 0x7fc5601f2f20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e9d0 tx=0x7fc55000ee90 comp rx=0 tx=0).stop 2023-11-28T20:47:57.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 shutdown_connections 2023-11-28T20:47:57.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc54c06acc0 0x7fc54c06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc56015d550 0x7fc5600c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5600d1ab0 0x7fc5601f2f20 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 --2- 172.21.15.45:0/3862410009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5600d02d0 0x7fc5601567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.705+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 >> 172.21.15.45:0/3862410009 conn(0x7fc56006ac70 msgr2=0x7fc56015c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:57.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.706+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 shutdown_connections 2023-11-28T20:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.706+0000 7fc565d16640 1 -- 172.21.15.45:0/3862410009 wait complete. 2023-11-28T20:47:57.739 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T20:47:57.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:47:57.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.960+0000 7f9d0e1c1640 1 Processor -- start 2023-11-28T20:47:57.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.960+0000 7f9d0e1c1640 1 -- start start 2023-11-28T20:47:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.962+0000 7f9d0e1c1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9d080b24a0 0x7f9d080703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.962+0000 7f9d0e1c1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 0x7f9d08070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.962+0000 7f9d0e1c1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d080b4310 con 0x7f9d08072200 2023-11-28T20:47:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.962+0000 7f9d0e1c1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d080725e0 con 0x7f9d080b24a0 2023-11-28T20:47:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.962+0000 7f9d0e1c1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0806d8d0 con 0x7f9d08146610 2023-11-28T20:47:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.963+0000 7f9d0c9be640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9d080b24a0 0x7f9d080703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.963+0000 7f9d0c9be640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9d080b24a0 0x7f9d080703d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58186/0 (socket says 172.21.15.45:58186) 2023-11-28T20:47:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.963+0000 7f9d0c9be640 1 -- 172.21.15.45:0/2985319770 learned_addr learned my addr 172.21.15.45:0/2985319770 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:57.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.963+0000 7f9d0d9c0640 1 --2- 172.21.15.45:0/2985319770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 0x7f9d08070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.963+0000 7f9d0c9be640 1 -- 172.21.15.45:0/2985319770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 msgr2=0x7f9d08070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:57.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.963+0000 7f9d0c9be640 1 --2- 172.21.15.45:0/2985319770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 0x7f9d08070910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.963+0000 7f9d0c9be640 1 -- 172.21.15.45:0/2985319770 >> v1:172.21.15.190:6789/0 conn(0x7f9d08072200 legacy=0x7f9d080b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.963+0000 7f9d0c9be640 1 -- 172.21.15.45:0/2985319770 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0806ca20 con 0x7f9d080b24a0 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.963+0000 7f9d0d9c0640 1 --2- 172.21.15.45:0/2985319770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 0x7f9d08070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.964+0000 7f9d0c9be640 1 --2- 172.21.15.45:0/2985319770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 0x7f9d080703d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009870 tx=0x7f9cf802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8aa4cdf06b6c54 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9cfffff640 1 -- 172.21.15.45:0/2985319770 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8031640 con 0x7f9d080b24a0 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9cfffff640 1 -- 172.21.15.45:0/2985319770 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf80317a0 con 0x7f9d080b24a0 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9cfffff640 1 -- 172.21.15.45:0/2985319770 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8031b30 con 0x7f9d080b24a0 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/2985319770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 msgr2=0x7f9d080703d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9d0e1c1640 1 --2- 172.21.15.45:0/2985319770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 0x7f9d080703d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009870 tx=0x7f9cf802b8f0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/2985319770 shutdown_connections 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9d0e1c1640 1 --2- 172.21.15.45:0/2985319770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 0x7f9d08070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9d0e1c1640 1 --2- 172.21.15.45:0/2985319770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 0x7f9d080703d0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/2985319770 >> 172.21.15.45:0/2985319770 conn(0x7f9d0806ab30 msgr2=0x7f9d080b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.965+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/2985319770 shutdown_connections 2023-11-28T20:47:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.966+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/2985319770 wait complete. 2023-11-28T20:47:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.967+0000 7f9d0e1c1640 1 Processor -- start 2023-11-28T20:47:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.968+0000 7f9d0e1c1640 1 -- start start 2023-11-28T20:47:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.968+0000 7f9d0e1c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d08072200 0x7f9d081342c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0e1c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 0x7f9d08160200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0e1c1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 0x7f9d08141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0e1c1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d08134eb0 con 0x7f9d080b24a0 2023-11-28T20:47:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0e1c1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0815fec0 con 0x7f9d08072200 2023-11-28T20:47:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0e1c1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0815e6a0 con 0x7f9d08146610 2023-11-28T20:47:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0c9be640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 0x7f9d08160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0c9be640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 0x7f9d08160200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58196/0 (socket says 172.21.15.45:58196) 2023-11-28T20:47:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0c9be640 1 -- 172.21.15.45:0/3853429200 learned_addr learned my addr 172.21.15.45:0/3853429200 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0c9be640 1 -- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 msgr2=0x7f9d08141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0c9be640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 0x7f9d08141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0c9be640 1 -- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d08072200 msgr2=0x7f9d081342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0c9be640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d08072200 0x7f9d081342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.969+0000 7f9d0c9be640 1 -- 172.21.15.45:0/3853429200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf80094d0 con 0x7f9d080b24a0 2023-11-28T20:47:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.970+0000 7f9d0c9be640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 0x7f9d08160200 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009870 tx=0x7f9cf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.970+0000 7f9cfdffb640 1 -- 172.21.15.45:0/3853429200 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf803ac60 con 0x7f9d080b24a0 2023-11-28T20:47:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.970+0000 7f9cfdffb640 1 -- 172.21.15.45:0/3853429200 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf803ade0 con 0x7f9d080b24a0 2023-11-28T20:47:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.970+0000 7f9cfdffb640 1 -- 172.21.15.45:0/3853429200 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8044630 con 0x7f9d080b24a0 2023-11-28T20:47:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.970+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0815e980 con 0x7f9d080b24a0 2023-11-28T20:47:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.970+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0815ef40 con 0x7f9d080b24a0 2023-11-28T20:47:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.971+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0813f1a0 con 0x7f9d080b24a0 2023-11-28T20:47:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.972+0000 7f9cfdffb640 1 -- 172.21.15.45:0/3853429200 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f9cf802f050 con 0x7f9d080b24a0 2023-11-28T20:47:57.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.973+0000 7f9cfdffb640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f9cec06aed0 0x7f9cec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:57.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.973+0000 7f9cfdffb640 1 -- 172.21.15.45:0/3853429200 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7259+0+0 (secure 0 0 0) 0x7f9cf80abe00 con 0x7f9d080b24a0 2023-11-28T20:47:57.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.974+0000 7f9d0d1bf640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f9cec06aed0 0x7f9cec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.975+0000 7f9d0d1bf640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f9cec06aed0 0x7f9cec06d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9d04006fd0 tx=0x7f9d04006bc0 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:57.978+0000 7f9cfdffb640 1 -- 172.21.15.45:0/3853429200 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d0813f1a0 con 0x7f9d080b24a0 2023-11-28T20:47:58.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.209+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d080b4310 con 0x7f9d080b24a0 2023-11-28T20:47:58.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.210+0000 7f9cfdffb640 1 -- 172.21.15.45:0/3853429200 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f9d080b4310 con 0x7f9d080b24a0 2023-11-28T20:47:58.211 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:58.212 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3050590116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3050590116},{"type":"v1","addr":"172.21.15.45:6835","nonce":3050590116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2465910573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2465910573},{"type":"v1","addr":"172.21.15.190:6836","nonce":2465910573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4090116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4090116745},{"type":"v1","addr":"172.21.15.190:6837","nonce":4090116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:35:35.751211+0000","modified":"2023-11-28T20:35:38.829781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4945":{"gid":4945,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6834/1984946086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1984946086},{"type":"v1","addr":"172.21.15.45:6834","nonce":1984946086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T20:47:58.212 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 18 2023-11-28T20:47:58.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.212+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f9cec06aed0 msgr2=0x7f9cec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.212+0000 7f9d0e1c1640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f9cec06aed0 0x7f9cec06d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9d04006fd0 tx=0x7f9d04006bc0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.212+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 msgr2=0x7f9d08160200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.212+0000 7f9d0e1c1640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 0x7f9d08160200 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009870 tx=0x7f9cf80046c0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.213+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 shutdown_connections 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.213+0000 7f9d0e1c1640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f9cec06aed0 0x7f9cec06d390 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.213+0000 7f9d0e1c1640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d08146610 0x7f9d08141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.213+0000 7f9d0e1c1640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080b24a0 0x7f9d08160200 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.213+0000 7f9d0e1c1640 1 --2- 172.21.15.45:0/3853429200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d08072200 0x7f9d081342c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.213+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 >> 172.21.15.45:0/3853429200 conn(0x7f9d0806ab30 msgr2=0x7f9d080abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.213+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 shutdown_connections 2023-11-28T20:47:58.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.213+0000 7f9d0e1c1640 1 -- 172.21.15.45:0/3853429200 wait complete. 2023-11-28T20:47:58.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:47:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.459+0000 7fa54a2ad640 1 Processor -- start 2023-11-28T20:47:58.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.460+0000 7fa54a2ad640 1 -- start start 2023-11-28T20:47:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.461+0000 7fa54a2ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa544155780 0x7fa5440abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.461+0000 7fa54a2ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5440ce800 0x7fa5440b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.461+0000 7fa54a2ad640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54415c3c0 con 0x7fa5440ae540 2023-11-28T20:47:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.461+0000 7fa54a2ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa544155b60 con 0x7fa544155780 2023-11-28T20:47:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.461+0000 7fa54a2ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54415ba90 con 0x7fa5440ce800 2023-11-28T20:47:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.462+0000 7fa5492ab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa544155780 0x7fa5440abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.462+0000 7fa5492ab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa544155780 0x7fa5440abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58208/0 (socket says 172.21.15.45:58208) 2023-11-28T20:47:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.462+0000 7fa5492ab640 1 -- 172.21.15.45:0/2019505180 learned_addr learned my addr 172.21.15.45:0/2019505180 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.462+0000 7fa549aac640 1 --2- 172.21.15.45:0/2019505180 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5440ce800 0x7fa5440b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.462+0000 7fa5492ab640 1 -- 172.21.15.45:0/2019505180 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5440ce800 msgr2=0x7fa5440b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.462+0000 7fa5492ab640 1 --2- 172.21.15.45:0/2019505180 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5440ce800 0x7fa5440b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.462+0000 7fa5492ab640 1 -- 172.21.15.45:0/2019505180 >> v1:172.21.15.190:6789/0 conn(0x7fa5440ae540 legacy=0x7fa5440ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.462+0000 7fa5492ab640 1 -- 172.21.15.45:0/2019505180 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54415bd70 con 0x7fa544155780 2023-11-28T20:47:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.464+0000 7fa5492ab640 1 --2- 172.21.15.45:0/2019505180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 0x7fa5440abd70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa540009870 tx=0x7fa54002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdefa56f88f1f8fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.464+0000 7fa53bfff640 1 -- 172.21.15.45:0/2019505180 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540030840 con 0x7fa544155780 2023-11-28T20:47:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.464+0000 7fa53bfff640 1 -- 172.21.15.45:0/2019505180 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5400309a0 con 0x7fa544155780 2023-11-28T20:47:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.464+0000 7fa53bfff640 1 -- 172.21.15.45:0/2019505180 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540030d30 con 0x7fa544155780 2023-11-28T20:47:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.465+0000 7fa54a2ad640 1 -- 172.21.15.45:0/2019505180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 msgr2=0x7fa5440abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.465+0000 7fa54a2ad640 1 --2- 172.21.15.45:0/2019505180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 0x7fa5440abd70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa540009870 tx=0x7fa54002b980 comp rx=0 tx=0).stop 2023-11-28T20:47:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.465+0000 7fa54a2ad640 1 -- 172.21.15.45:0/2019505180 shutdown_connections 2023-11-28T20:47:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.465+0000 7fa54a2ad640 1 --2- 172.21.15.45:0/2019505180 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5440ce800 0x7fa5440b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.465+0000 7fa54a2ad640 1 --2- 172.21.15.45:0/2019505180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 0x7fa5440abd70 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.465+0000 7fa54a2ad640 1 -- 172.21.15.45:0/2019505180 >> 172.21.15.45:0/2019505180 conn(0x7fa54406ab30 msgr2=0x7fa5440b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.465+0000 7fa54a2ad640 1 -- 172.21.15.45:0/2019505180 shutdown_connections 2023-11-28T20:47:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.465+0000 7fa54a2ad640 1 -- 172.21.15.45:0/2019505180 wait complete. 2023-11-28T20:47:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.467+0000 7fa54a2ad640 1 Processor -- start 2023-11-28T20:47:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.467+0000 7fa54a2ad640 1 -- start start 2023-11-28T20:47:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa54a2ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5440ae540 0x7fa544099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa54a2ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5440ce800 0x7fa54409a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa54a2ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 0x7fa54406d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa54a2ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54409bf90 con 0x7fa544155780 2023-11-28T20:47:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa54a2ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54414ff60 con 0x7fa5440ae540 2023-11-28T20:47:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa54a2ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54409c1c0 con 0x7fa5440ce800 2023-11-28T20:47:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa549aac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 0x7fa54406d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa549aac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 0x7fa54406d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58210/0 (socket says 172.21.15.45:58210) 2023-11-28T20:47:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa549aac640 1 -- 172.21.15.45:0/4169602798 learned_addr learned my addr 172.21.15.45:0/4169602798 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa5492ab640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5440ae540 0x7fa544099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:58.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.468+0000 7fa549aac640 1 -- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5440ce800 msgr2=0x7fa54409a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa549aac640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5440ce800 0x7fa54409a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa549aac640 1 -- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5440ae540 msgr2=0x7fa544099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa549aac640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5440ae540 0x7fa544099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa549aac640 1 -- 172.21.15.45:0/4169602798 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400094d0 con 0x7fa544155780 2023-11-28T20:47:58.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa549aac640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 0x7fa54406d4f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa53c007e40 tx=0x7fa53c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:58.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa539ffb640 1 -- 172.21.15.45:0/4169602798 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c014620 con 0x7fa544155780 2023-11-28T20:47:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa539ffb640 1 -- 172.21.15.45:0/4169602798 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53c00ee90 con 0x7fa544155780 2023-11-28T20:47:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa539ffb640 1 -- 172.21.15.45:0/4169602798 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c0149e0 con 0x7fa544155780 2023-11-28T20:47:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54409c4a0 con 0x7fa544155780 2023-11-28T20:47:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.469+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54409aac0 con 0x7fa544155780 2023-11-28T20:47:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.470+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa544070eb0 con 0x7fa544155780 2023-11-28T20:47:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.471+0000 7fa539ffb640 1 -- 172.21.15.45:0/4169602798 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fa53c014b40 con 0x7fa544155780 2023-11-28T20:47:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.472+0000 7fa539ffb640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fa52806aed0 0x7fa52806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.472+0000 7fa539ffb640 1 -- 172.21.15.45:0/4169602798 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7259+0+0 (secure 0 0 0) 0x7fa53c08a660 con 0x7fa544155780 2023-11-28T20:47:58.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.473+0000 7fa5492ab640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fa52806aed0 0x7fa52806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:58.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.474+0000 7fa5492ab640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fa52806aed0 0x7fa52806d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa540036000 tx=0x7fa54003b040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:58.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.477+0000 7fa539ffb640 1 -- 172.21.15.45:0/4169602798 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa544070eb0 con 0x7fa544155780 2023-11-28T20:47:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.710+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa544070eb0 con 0x7fa544155780 2023-11-28T20:47:58.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.712+0000 7fa539ffb640 1 -- 172.21.15.45:0/4169602798 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fa544070eb0 con 0x7fa544155780 2023-11-28T20:47:58.713 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:58.713 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3050590116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3050590116},{"type":"v1","addr":"172.21.15.45:6835","nonce":3050590116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2465910573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2465910573},{"type":"v1","addr":"172.21.15.190:6836","nonce":2465910573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4090116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4090116745},{"type":"v1","addr":"172.21.15.190:6837","nonce":4090116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:35:35.751211+0000","modified":"2023-11-28T20:35:38.829781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4945":{"gid":4945,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.45:6834/1984946086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1984946086},{"type":"v1","addr":"172.21.15.45:6834","nonce":1984946086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T20:47:58.714 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 18 2023-11-28T20:47:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fa52806aed0 msgr2=0x7fa52806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fa52806aed0 0x7fa52806d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa540036000 tx=0x7fa54003b040 comp rx=0 tx=0).stop 2023-11-28T20:47:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 msgr2=0x7fa54406d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 0x7fa54406d4f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa53c007e40 tx=0x7fa53c00e4b0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 shutdown_connections 2023-11-28T20:47:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fa52806aed0 0x7fa52806d390 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa544155780 0x7fa54406d4f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5440ce800 0x7fa54409a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 --2- 172.21.15.45:0/4169602798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5440ae540 0x7fa544099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 >> 172.21.15.45:0/4169602798 conn(0x7fa54406ab30 msgr2=0x7fa54415d080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 shutdown_connections 2023-11-28T20:47:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.715+0000 7fa54a2ad640 1 -- 172.21.15.45:0/4169602798 wait complete. 2023-11-28T20:47:58.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:47:58.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.967+0000 7fe530e90640 1 Processor -- start 2023-11-28T20:47:58.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.967+0000 7fe530e90640 1 -- start start 2023-11-28T20:47:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.969+0000 7fe530e90640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe52c0b1340 0x7fe52c15f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.969+0000 7fe530e90640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c15bda0 con 0x7fe52c0b35b0 2023-11-28T20:47:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.969+0000 7fe530e90640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c15c080 con 0x7fe52c15b850 2023-11-28T20:47:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.969+0000 7fe530e90640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52c15ee40 con 0x7fe52c0b1340 2023-11-28T20:47:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.969+0000 7fe52a59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe52c0b35b0 0x7fe52c15ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36078/0 (socket says 172.21.15.45:36078) 2023-11-28T20:47:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.969+0000 7fe52a59c640 1 -- 172.21.15.45:0/2139567609 learned_addr learned my addr 172.21.15.45:0/2139567609 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.969+0000 7fe52ad9d640 1 --2- 172.21.15.45:0/2139567609 >> v2:172.21.15.45:3300/0 conn(0x7fe52c0b1340 0x7fe52c15f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.970+0000 7fe52ad9d640 1 -- 172.21.15.45:0/2139567609 >> v1:172.21.15.190:6789/0 conn(0x7fe52c15b850 legacy=0x7fe52c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.970+0000 7fe52ad9d640 1 -- 172.21.15.45:0/2139567609 >> v1:172.21.15.45:6789/0 conn(0x7fe52c0b35b0 legacy=0x7fe52c15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.970+0000 7fe52ad9d640 1 -- 172.21.15.45:0/2139567609 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c15c360 con 0x7fe52c0b1340 2023-11-28T20:47:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.971+0000 7fe52ad9d640 1 --2- 172.21.15.45:0/2139567609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b1340 0x7fe52c15f290 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe5240097a0 tx=0x7fe52402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea823738ebe9e429 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.971+0000 7fe52959a640 1 -- 172.21.15.45:0/2139567609 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524030610 con 0x7fe52c0b1340 2023-11-28T20:47:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.971+0000 7fe52959a640 1 -- 172.21.15.45:0/2139567609 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe524030770 con 0x7fe52c0b1340 2023-11-28T20:47:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.971+0000 7fe52959a640 1 -- 172.21.15.45:0/2139567609 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524030b00 con 0x7fe52c0b1340 2023-11-28T20:47:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.972+0000 7fe530e90640 1 -- 172.21.15.45:0/2139567609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b1340 msgr2=0x7fe52c15f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.972+0000 7fe530e90640 1 --2- 172.21.15.45:0/2139567609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b1340 0x7fe52c15f290 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe5240097a0 tx=0x7fe52402b960 comp rx=0 tx=0).stop 2023-11-28T20:47:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.972+0000 7fe530e90640 1 -- 172.21.15.45:0/2139567609 shutdown_connections 2023-11-28T20:47:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.972+0000 7fe530e90640 1 --2- 172.21.15.45:0/2139567609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b1340 0x7fe52c15f290 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.972+0000 7fe530e90640 1 -- 172.21.15.45:0/2139567609 >> 172.21.15.45:0/2139567609 conn(0x7fe52c06a500 msgr2=0x7fe52c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.972+0000 7fe530e90640 1 -- 172.21.15.45:0/2139567609 shutdown_connections 2023-11-28T20:47:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.972+0000 7fe530e90640 1 -- 172.21.15.45:0/2139567609 wait complete. 2023-11-28T20:47:58.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.974+0000 7fe530e90640 1 Processor -- start 2023-11-28T20:47:58.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.974+0000 7fe530e90640 1 -- start start 2023-11-28T20:47:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.975+0000 7fe530e90640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe52c0b1340 0x7fe52c138370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.975+0000 7fe530e90640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b35b0 0x7fe52c0a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.975+0000 7fe530e90640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe52c15b850 0x7fe52c0c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.975+0000 7fe530e90640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c155330 con 0x7fe52c0b35b0 2023-11-28T20:47:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.975+0000 7fe530e90640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c0aafe0 con 0x7fe52c0b1340 2023-11-28T20:47:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.975+0000 7fe530e90640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c0a9f40 con 0x7fe52c15b850 2023-11-28T20:47:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.975+0000 7fe529d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b35b0 0x7fe52c0a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.975+0000 7fe529d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b35b0 0x7fe52c0a94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58214/0 (socket says 172.21.15.45:58214) 2023-11-28T20:47:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.975+0000 7fe529d9b640 1 -- 172.21.15.45:0/2196596477 learned_addr learned my addr 172.21.15.45:0/2196596477 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.976+0000 7fe529d9b640 1 -- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe52c15b850 msgr2=0x7fe52c0c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.976+0000 7fe529d9b640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe52c15b850 0x7fe52c0c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.976+0000 7fe529d9b640 1 -- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe52c0b1340 msgr2=0x7fe52c138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.976+0000 7fe52a59c640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe52c0b1340 0x7fe52c138370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.976+0000 7fe529d9b640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe52c0b1340 0x7fe52c138370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.976+0000 7fe529d9b640 1 -- 172.21.15.45:0/2196596477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524009400 con 0x7fe52c0b35b0 2023-11-28T20:47:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.976+0000 7fe529d9b640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b35b0 0x7fe52c0a94f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe51400c3c0 tx=0x7fe51400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.977+0000 7fe522ffd640 1 -- 172.21.15.45:0/2196596477 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5140052d0 con 0x7fe52c0b35b0 2023-11-28T20:47:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.977+0000 7fe522ffd640 1 -- 172.21.15.45:0/2196596477 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe514005450 con 0x7fe52c0b35b0 2023-11-28T20:47:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.977+0000 7fe522ffd640 1 -- 172.21.15.45:0/2196596477 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514003830 con 0x7fe52c0b35b0 2023-11-28T20:47:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.977+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c0aa220 con 0x7fe52c0b35b0 2023-11-28T20:47:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.977+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c0aa7e0 con 0x7fe52c0b35b0 2023-11-28T20:47:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.978+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c0733d0 con 0x7fe52c0b35b0 2023-11-28T20:47:58.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.979+0000 7fe522ffd640 1 -- 172.21.15.45:0/2196596477 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fe514003990 con 0x7fe52c0b35b0 2023-11-28T20:47:58.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.980+0000 7fe522ffd640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe50c06aed0 0x7fe50c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:58.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.980+0000 7fe522ffd640 1 -- 172.21.15.45:0/2196596477 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7259+0+0 (secure 0 0 0) 0x7fe514089110 con 0x7fe52c0b35b0 2023-11-28T20:47:58.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.980+0000 7fe52a59c640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe50c06aed0 0x7fe50c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:58.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.981+0000 7fe52a59c640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe50c06aed0 0x7fe50c06d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe51c005c00 tx=0x7fe51c009040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:58.984+0000 7fe522ffd640 1 -- 172.21.15.45:0/2196596477 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe514053bc0 con 0x7fe52c0b35b0 2023-11-28T20:47:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.180+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe52c0b3e50 con 0x7fe52c0b35b0 2023-11-28T20:47:59.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.182+0000 7fe522ffd640 1 -- 172.21.15.45:0/2196596477 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+18574 (secure 0 0 0) 0x7fe514017090 con 0x7fe52c0b35b0 2023-11-28T20:47:59.183 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:47:59.183 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":59,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:47:55.565091+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:35:33.849797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-28T20:35:34.783808+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":233,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/2065659095":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/3262798906":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/664590003":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1813745990":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6832/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6833/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3717946281":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3679315161":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1881573438":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1866894774":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3153178694":"2023-11-29T20:47:55.564970+0000","172.21.15.190:6832/2725659217":"2023-11-29T20:47:55.564970+0000","172.21.15.190:6833/2725659217":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1599738782":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/3278716326":"2023-11-29T20:47:55.564970+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1},{"begin":9,"length":1},{"begin":11,"length":1},{"begin":13,"length":1},{"begin":17,"length":1},{"begin":20,"length":1},{"begin":22,"length":1},{"begin":26,"length":2},{"begin":30,"length":1},{"begin":34,"length":1},{"begin":36,"length":5},{"begin":43,"length":2},{"begin":46,"length":2},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":2},{"begin":60,"length":2},{"begin":65,"length":5},{"begin":75,"length":3},{"begin":79,"length":1},{"begin":84,"length":2},{"begin":87,"length":1},{"begin":89,"length":1},{"begin":92,"length":1},{"begin":94,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":101,"length":2},{"begin":106,"length":1},{"begin":108,"length":1},{"begin":110,"length":2},{"begin":116,"length":2},{"begin":189,"length":45}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:47:59.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe50c06aed0 msgr2=0x7fe50c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe50c06aed0 0x7fe50c06d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe51c005c00 tx=0x7fe51c009040 comp rx=0 tx=0).stop 2023-11-28T20:47:59.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b35b0 msgr2=0x7fe52c0a94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b35b0 0x7fe52c0a94f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe51400c3c0 tx=0x7fe51400c880 comp rx=0 tx=0).stop 2023-11-28T20:47:59.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 shutdown_connections 2023-11-28T20:47:59.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe50c06aed0 0x7fe50c06d390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe52c15b850 0x7fe52c0c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0b35b0 0x7fe52c0a94f0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 --2- 172.21.15.45:0/2196596477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe52c0b1340 0x7fe52c138370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 >> 172.21.15.45:0/2196596477 conn(0x7fe52c06a500 msgr2=0x7fe52c0afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:59.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 shutdown_connections 2023-11-28T20:47:59.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.185+0000 7fe530e90640 1 -- 172.21.15.45:0/2196596477 wait complete. 2023-11-28T20:47:59.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T20:47:59.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.426+0000 7f48645de640 1 Processor -- start 2023-11-28T20:47:59.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.426+0000 7f48645de640 1 -- start start 2023-11-28T20:47:59.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.427+0000 7f48645de640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f485c143a60 0x7f485c06c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.427+0000 7f48645de640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485c06c820 0x7f485c0cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.428+0000 7f48645de640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0cd1f0 con 0x7f485c0cf650 2023-11-28T20:47:59.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.428+0000 7f48645de640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c0cfae0 con 0x7f485c143a60 2023-11-28T20:47:59.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.428+0000 7f48645de640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c06cda0 con 0x7f485c06c820 2023-11-28T20:47:59.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.428+0000 7f4862b7b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f485c0cf650 0x7f485c15dd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36088/0 (socket says 172.21.15.45:36088) 2023-11-28T20:47:59.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.428+0000 7f4862b7b640 1 -- 172.21.15.45:0/2441905543 learned_addr learned my addr 172.21.15.45:0/2441905543 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:47:59.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.428+0000 7f4861b79640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485c06c820 0x7f485c0cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.428+0000 7f486237a640 1 --2- 172.21.15.45:0/2441905543 >> v2:172.21.15.190:3300/0 conn(0x7f485c143a60 0x7f485c06c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.428+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429002908 0 0) 0x7f485c0cd1f0 con 0x7f485c0cf650 2023-11-28T20:47:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.428+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848003520 con 0x7f485c0cf650 2023-11-28T20:47:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 413628130 0 0) 0x7f4848003520 con 0x7f485c0cf650 2023-11-28T20:47:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485c06c820 msgr2=0x7f485c0cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485c06c820 0x7f485c0cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 >> v2:172.21.15.190:3300/0 conn(0x7f485c143a60 msgr2=0x7f485c06c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 --2- 172.21.15.45:0/2441905543 >> v2:172.21.15.190:3300/0 conn(0x7f485c143a60 0x7f485c06c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c06d080 con 0x7f485c0cf650 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4854001670 con 0x7f485c0cf650 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 >> v1:172.21.15.45:6789/0 conn(0x7f485c0cf650 legacy=0x7f485c15dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485c06c820 0x7f4848003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4848003ef0 0x7f48480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4848004d00 0x7f4848005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0cfae0 con 0x7f4848004d00 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c06cda0 con 0x7f4848003ef0 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484800b9d0 con 0x7f485c06c820 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.429+0000 7f4862b7b640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4848004d00 0x7f4848005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.430+0000 7f486237a640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485c06c820 0x7f4848003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.430+0000 7f4861b79640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4848003ef0 0x7f48480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.430+0000 7f4862b7b640 1 -- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485c06c820 msgr2=0x7f4848003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.430+0000 7f4862b7b640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485c06c820 0x7f4848003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.430+0000 7f4862b7b640 1 -- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4848003ef0 msgr2=0x7f48480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:59.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.430+0000 7f4862b7b640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4848003ef0 0x7f48480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.430+0000 7f4862b7b640 1 -- 172.21.15.45:0/2441905543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848003520 con 0x7f4848004d00 2023-11-28T20:47:59.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.431+0000 7f4862b7b640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4848004d00 0x7f4848005180 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f485400c240 tx=0x7f48480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7aafa4875822ba2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:59.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.431+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485402cb00 con 0x7f4848004d00 2023-11-28T20:47:59.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.431+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485402cc60 con 0x7f4848004d00 2023-11-28T20:47:59.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.431+0000 7f4861378640 1 -- 172.21.15.45:0/2441905543 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4854005af0 con 0x7f4848004d00 2023-11-28T20:47:59.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.432+0000 7f48645de640 1 -- 172.21.15.45:0/2441905543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4848004d00 msgr2=0x7f4848005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.432+0000 7f48645de640 1 --2- 172.21.15.45:0/2441905543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4848004d00 0x7f4848005180 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f485400c240 tx=0x7f48480057f0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.432+0000 7f4862b7b640 1 -- 172.21.15.45:0/2441905543 reap_dead start 2023-11-28T20:47:59.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.432+0000 7f48645de640 1 -- 172.21.15.45:0/2441905543 shutdown_connections 2023-11-28T20:47:59.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.432+0000 7f48645de640 1 -- 172.21.15.45:0/2441905543 >> 172.21.15.45:0/2441905543 conn(0x7f485c06a6e0 msgr2=0x7f485c06aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:59.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.432+0000 7f48645de640 1 -- 172.21.15.45:0/2441905543 shutdown_connections 2023-11-28T20:47:59.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.432+0000 7f48645de640 1 -- 172.21.15.45:0/2441905543 wait complete. 2023-11-28T20:47:59.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.434+0000 7f48645de640 1 Processor -- start 2023-11-28T20:47:59.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.434+0000 7f48645de640 1 -- start start 2023-11-28T20:47:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.435+0000 7f48645de640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4848004d00 0x7f485c159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.435+0000 7f48645de640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485c134480 0x7f485c1348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.435+0000 7f48645de640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f485c0abd70 0x7f485c0ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.435+0000 7f48645de640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c135e60 con 0x7f485c0abd70 2023-11-28T20:47:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.435+0000 7f48645de640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c1351e0 con 0x7f485c134480 2023-11-28T20:47:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.435+0000 7f48645de640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c1354c0 con 0x7f4848004d00 2023-11-28T20:47:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.435+0000 7f4862b7b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f485c0abd70 0x7f485c0ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.435+0000 7f4862b7b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f485c0abd70 0x7f485c0ac1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58230/0 (socket says 172.21.15.45:58230) 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.435+0000 7f4862b7b640 1 -- 172.21.15.45:0/546046539 learned_addr learned my addr 172.21.15.45:0/546046539 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f486237a640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4848004d00 0x7f485c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f4861b79640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485c134480 0x7f485c1348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f4862b7b640 1 -- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4848004d00 msgr2=0x7f485c159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f4862b7b640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4848004d00 0x7f485c159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f4862b7b640 1 -- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485c134480 msgr2=0x7f485c1348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f4862b7b640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485c134480 0x7f485c1348e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f4862b7b640 1 -- 172.21.15.45:0/546046539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c1357a0 con 0x7f485c0abd70 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f4861b79640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485c134480 0x7f485c1348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f4862b7b640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f485c0abd70 0x7f485c0ac1d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f485400c240 tx=0x7f4854001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:59.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.436+0000 7f4852ffd640 1 -- 172.21.15.45:0/546046539 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485403c040 con 0x7f485c0abd70 2023-11-28T20:47:59.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.437+0000 7f4852ffd640 1 -- 172.21.15.45:0/546046539 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4854001630 con 0x7f485c0abd70 2023-11-28T20:47:59.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.437+0000 7f4852ffd640 1 -- 172.21.15.45:0/546046539 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4854007290 con 0x7f485c0abd70 2023-11-28T20:47:59.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.437+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c135a80 con 0x7f485c0abd70 2023-11-28T20:47:59.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.437+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c134e20 con 0x7f485c0abd70 2023-11-28T20:47:59.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.438+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c135fa0 con 0x7f485c0abd70 2023-11-28T20:47:59.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.439+0000 7f4852ffd640 1 -- 172.21.15.45:0/546046539 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f4854032070 con 0x7f485c0abd70 2023-11-28T20:47:59.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.440+0000 7f4852ffd640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f484006af00 0x7f484006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.440+0000 7f4852ffd640 1 -- 172.21.15.45:0/546046539 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7259+0+0 (secure 0 0 0) 0x7f48540aa950 con 0x7f485c0abd70 2023-11-28T20:47:59.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.441+0000 7f486237a640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f484006af00 0x7f484006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.442+0000 7f486237a640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f484006af00 0x7f484006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4858003fd0 tx=0x7f485800b040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:59.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.445+0000 7f4852ffd640 1 -- 172.21.15.45:0/546046539 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f485c135fa0 con 0x7f485c0abd70 2023-11-28T20:47:59.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.678+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f485c135fa0 con 0x7f485c0abd70 2023-11-28T20:47:59.680 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T20:47:59.680+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T20:47:59.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.683+0000 7f4852ffd640 1 -- 172.21.15.45:0/546046539 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) v1 ==== 161+0+0 (secure 0 0 0) 0x7f485c135fa0 con 0x7f485c0abd70 2023-11-28T20:47:59.684 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T20:47:59.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.686+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f484006af00 msgr2=0x7f484006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.686+0000 7f48645de640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f484006af00 0x7f484006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4858003fd0 tx=0x7f485800b040 comp rx=0 tx=0).stop 2023-11-28T20:47:59.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.686+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f485c0abd70 msgr2=0x7f485c0ac1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.686+0000 7f48645de640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f485c0abd70 0x7f485c0ac1d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f485400c240 tx=0x7f4854001a70 comp rx=0 tx=0).stop 2023-11-28T20:47:59.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.687+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 shutdown_connections 2023-11-28T20:47:59.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.687+0000 7f48645de640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f484006af00 0x7f484006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.687+0000 7f48645de640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f485c0abd70 0x7f485c0ac1d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.687+0000 7f48645de640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485c134480 0x7f485c1348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.687+0000 7f48645de640 1 --2- 172.21.15.45:0/546046539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4848004d00 0x7f485c159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.687+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 >> 172.21.15.45:0/546046539 conn(0x7f485c06a6e0 msgr2=0x7f485c0cbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:59.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.688+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 shutdown_connections 2023-11-28T20:47:59.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.688+0000 7f48645de640 1 -- 172.21.15.45:0/546046539 wait complete. 2023-11-28T20:47:59.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T20:47:59.760 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:47:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.907+0000 7f7fc669b640 1 Processor -- start 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.908+0000 7f7fc669b640 1 -- start start 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.909+0000 7f7fc669b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7fc015e840 0x7f7fc006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.909+0000 7f7fc669b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc006ac10 0x7f7fc00717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.909+0000 7f7fc669b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc015ecd0 con 0x7f7fc015f720 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.909+0000 7f7fc669b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc0071ce0 con 0x7f7fc015e840 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.909+0000 7f7fc669b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc015f290 con 0x7f7fc006ac10 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.909+0000 7f7fbffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7fc015e840 0x7f7fc006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.910+0000 7f7fbffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7fc015e840 0x7f7fc006c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58270/0 (socket says 172.21.15.45:58270) 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.910+0000 7f7fbffff640 1 -- 172.21.15.45:0/783077654 learned_addr learned my addr 172.21.15.45:0/783077654 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.910+0000 7f7fc4c38640 1 --2- 172.21.15.45:0/783077654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc006ac10 0x7f7fc00717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.910+0000 7f7fbffff640 1 -- 172.21.15.45:0/783077654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc006ac10 msgr2=0x7f7fc00717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.910+0000 7f7fbffff640 1 --2- 172.21.15.45:0/783077654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc006ac10 0x7f7fc00717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.910+0000 7f7fbffff640 1 -- 172.21.15.45:0/783077654 >> v1:172.21.15.190:6789/0 conn(0x7f7fc015f720 legacy=0x7f7fc006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.910+0000 7f7fbffff640 1 -- 172.21.15.45:0/783077654 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc015fb60 con 0x7f7fc015e840 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.910+0000 7f7fc4c38640 1 --2- 172.21.15.45:0/783077654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc006ac10 0x7f7fc00717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.911+0000 7f7fbffff640 1 --2- 172.21.15.45:0/783077654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc015e840 0x7f7fc006c350 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009870 tx=0x7f7fb002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2e1941fa437815 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.912+0000 7f7fbf7fe640 1 -- 172.21.15.45:0/783077654 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0031640 con 0x7f7fc015e840 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.912+0000 7f7fbf7fe640 1 -- 172.21.15.45:0/783077654 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb00317a0 con 0x7f7fc015e840 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.912+0000 7f7fbf7fe640 1 -- 172.21.15.45:0/783077654 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0031b30 con 0x7f7fc015e840 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.912+0000 7f7fc669b640 1 -- 172.21.15.45:0/783077654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc015e840 msgr2=0x7f7fc006c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.912+0000 7f7fc669b640 1 --2- 172.21.15.45:0/783077654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc015e840 0x7f7fc006c350 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009870 tx=0x7f7fb002b8f0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.912+0000 7f7fc669b640 1 -- 172.21.15.45:0/783077654 shutdown_connections 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.912+0000 7f7fc669b640 1 --2- 172.21.15.45:0/783077654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc006ac10 0x7f7fc00717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.912+0000 7f7fc669b640 1 --2- 172.21.15.45:0/783077654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc015e840 0x7f7fc006c350 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.912+0000 7f7fc669b640 1 -- 172.21.15.45:0/783077654 >> 172.21.15.45:0/783077654 conn(0x7f7fc00ced60 msgr2=0x7f7fc00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:47:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.913+0000 7f7fc669b640 1 -- 172.21.15.45:0/783077654 shutdown_connections 2023-11-28T20:47:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.913+0000 7f7fc669b640 1 -- 172.21.15.45:0/783077654 wait complete. 2023-11-28T20:47:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.915+0000 7f7fc669b640 1 Processor -- start 2023-11-28T20:47:59.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.915+0000 7f7fc669b640 1 -- start start 2023-11-28T20:47:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.916+0000 7f7fc669b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc006ac10 0x7f7fc00b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.916+0000 7f7fc669b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc015e840 0x7f7fc0153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.916+0000 7f7fc669b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc015f720 0x7f7fc00b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.916+0000 7f7fc669b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00b2540 con 0x7f7fc006ac10 2023-11-28T20:47:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.916+0000 7f7fc669b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00b2ff0 con 0x7f7fc015f720 2023-11-28T20:47:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.916+0000 7f7fc669b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00b2750 con 0x7f7fc015e840 2023-11-28T20:47:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fbffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc006ac10 0x7f7fc00b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fbffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc006ac10 0x7f7fc00b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58280/0 (socket says 172.21.15.45:58280) 2023-11-28T20:47:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fbffff640 1 -- 172.21.15.45:0/3448595396 learned_addr learned my addr 172.21.15.45:0/3448595396 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:47:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fc4c38640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc015f720 0x7f7fc00b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fb7fff640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc015e840 0x7f7fc0153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fbffff640 1 -- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc015e840 msgr2=0x7f7fc0153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fbffff640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc015e840 0x7f7fc0153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fbffff640 1 -- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc015f720 msgr2=0x7f7fc00b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:47:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fbffff640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc015f720 0x7f7fc00b19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:47:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fbffff640 1 -- 172.21.15.45:0/3448595396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb00094d0 con 0x7f7fc006ac10 2023-11-28T20:47:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.917+0000 7f7fc4c38640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc015f720 0x7f7fc00b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:47:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.918+0000 7f7fbffff640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc006ac10 0x7f7fc00b4560 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000c00 tx=0x7f7fb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.918+0000 7f7fbd7fa640 1 -- 172.21.15.45:0/3448595396 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb003ac60 con 0x7f7fc006ac10 2023-11-28T20:47:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.918+0000 7f7fbd7fa640 1 -- 172.21.15.45:0/3448595396 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb003adc0 con 0x7f7fc006ac10 2023-11-28T20:47:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.918+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc00b2a30 con 0x7f7fc006ac10 2023-11-28T20:47:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.918+0000 7f7fbd7fa640 1 -- 172.21.15.45:0/3448595396 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0044630 con 0x7f7fc006ac10 2023-11-28T20:47:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.918+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc00b4aa0 con 0x7f7fc006ac10 2023-11-28T20:47:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.919+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc00733d0 con 0x7f7fc006ac10 2023-11-28T20:47:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.920+0000 7f7fbd7fa640 1 -- 172.21.15.45:0/3448595396 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f7fb002f050 con 0x7f7fc006ac10 2023-11-28T20:47:59.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.922+0000 7f7fbd7fa640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7fa006af00 0x7f7fa006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:47:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.922+0000 7f7fbd7fa640 1 -- 172.21.15.45:0/3448595396 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f7fb00abbd0 con 0x7f7fc006ac10 2023-11-28T20:47:59.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.922+0000 7f7fb7fff640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7fa006af00 0x7f7fa006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:47:59.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.923+0000 7f7fb7fff640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7fa006af00 0x7f7fa006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8006fd0 tx=0x7f7fa8008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:47:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:47:59.926+0000 7f7fbd7fa640 1 -- 172.21.15.45:0/3448595396 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fb0076740 con 0x7f7fc006ac10 2023-11-28T20:48:00.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.161+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7fc00ce2b0 con 0x7f7fc006ac10 2023-11-28T20:48:00.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.274+0000 7f7fbd7fa640 1 -- 172.21.15.45:0/3448595396 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7fb0076b10 con 0x7f7fc006ac10 2023-11-28T20:48:00.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.685+0000 7f7fbd7fa640 1 -- 172.21.15.45:0/3448595396 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7fb0090fd0 con 0x7f7fc006ac10 2023-11-28T20:48:00.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.688+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7fa006af00 msgr2=0x7f7fa006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.688+0000 7f7fc669b640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7fa006af00 0x7f7fa006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8006fd0 tx=0x7f7fa8008040 comp rx=0 tx=0).stop 2023-11-28T20:48:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc006ac10 msgr2=0x7f7fc00b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc006ac10 0x7f7fc00b4560 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000c00 tx=0x7f7fb00046c0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 shutdown_connections 2023-11-28T20:48:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7fa006af00 0x7f7fa006d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc015f720 0x7f7fc00b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fc015e840 0x7f7fc0153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 --2- 172.21.15.45:0/3448595396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fc006ac10 0x7f7fc00b4560 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 >> 172.21.15.45:0/3448595396 conn(0x7f7fc00ced60 msgr2=0x7f7fc0070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 shutdown_connections 2023-11-28T20:48:00.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.689+0000 7f7fc669b640 1 -- 172.21.15.45:0/3448595396 wait complete. 2023-11-28T20:48:00.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T20:48:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.941+0000 7f5d0f59e640 1 Processor -- start 2023-11-28T20:48:00.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.941+0000 7f5d0f59e640 1 -- start start 2023-11-28T20:48:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.943+0000 7f5d0f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1006ddb0 0x7f5d10071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.943+0000 7f5d0f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10070e20 con 0x7f5d100cbe10 2023-11-28T20:48:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.943+0000 7f5d0f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10070280 con 0x7f5d101448e0 2023-11-28T20:48:00.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.943+0000 7f5d0f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10144cc0 con 0x7f5d1006ddb0 2023-11-28T20:48:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.944+0000 7f5d07fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5d100cbe10 0x7f5d10071500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36104/0 (socket says 172.21.15.45:36104) 2023-11-28T20:48:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.944+0000 7f5d07fff640 1 -- 172.21.15.45:0/2264952015 learned_addr learned my addr 172.21.15.45:0/2264952015 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.944+0000 7f5d0ed9d640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1006ddb0 0x7f5d10071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:00.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.944+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61993734 0 0) 0x7f5d10070e20 con 0x7f5d100cbe10 2023-11-28T20:48:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.944+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cfc003520 con 0x7f5d100cbe10 2023-11-28T20:48:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294978910 0 0) 0x7f5cfc003520 con 0x7f5d100cbe10 2023-11-28T20:48:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1006ddb0 msgr2=0x7f5d10071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1006ddb0 0x7f5d10071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 >> v1:172.21.15.190:6789/0 conn(0x7f5d101448e0 legacy=0x7f5d100cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100704f0 con 0x7f5d100cbe10 2023-11-28T20:48:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5cf8001670 con 0x7f5d100cbe10 2023-11-28T20:48:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 >> v1:172.21.15.45:6789/0 conn(0x7f5d100cbe10 legacy=0x7f5d10071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5cfc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:00.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cfc0047c0 0x7f5cfc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1006ddb0 0x7f5cfc005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10144cc0 con 0x7f5cfc0039a0 2023-11-28T20:48:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc00bb10 con 0x7f5cfc0047c0 2023-11-28T20:48:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc00bcd0 con 0x7f5d1006ddb0 2023-11-28T20:48:00.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0e59c640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5cfc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d07fff640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cfc0047c0 0x7f5cfc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0ed9d640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1006ddb0 0x7f5cfc005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0e59c640 1 -- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1006ddb0 msgr2=0x7f5cfc005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.945+0000 7f5d0e59c640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1006ddb0 0x7f5cfc005230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.946+0000 7f5d0e59c640 1 -- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cfc0047c0 msgr2=0x7f5cfc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.946+0000 7f5d0e59c640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cfc0047c0 0x7f5cfc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.946+0000 7f5d0e59c640 1 -- 172.21.15.45:0/2264952015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc003520 con 0x7f5cfc0039a0 2023-11-28T20:48:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.947+0000 7f5d0e59c640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5cfc003dc0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5d00001750 tx=0x7f5cfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc4157be5af08db server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.947+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d000052d0 con 0x7f5cfc0039a0 2023-11-28T20:48:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.947+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0002e050 con 0x7f5cfc0039a0 2023-11-28T20:48:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.947+0000 7f5d0dd9b640 1 -- 172.21.15.45:0/2264952015 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00003830 con 0x7f5cfc0039a0 2023-11-28T20:48:00.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.947+0000 7f5d0f59e640 1 -- 172.21.15.45:0/2264952015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 msgr2=0x7f5cfc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:00.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.947+0000 7f5d0f59e640 1 --2- 172.21.15.45:0/2264952015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5cfc003dc0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5d00001750 tx=0x7f5cfc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.947+0000 7f5d0ed9d640 1 -- 172.21.15.45:0/2264952015 reap_dead start 2023-11-28T20:48:00.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.948+0000 7f5d0f59e640 1 -- 172.21.15.45:0/2264952015 shutdown_connections 2023-11-28T20:48:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.948+0000 7f5d0f59e640 1 -- 172.21.15.45:0/2264952015 >> 172.21.15.45:0/2264952015 conn(0x7f5d100cec70 msgr2=0x7f5d100cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.948+0000 7f5d0f59e640 1 -- 172.21.15.45:0/2264952015 shutdown_connections 2023-11-28T20:48:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.948+0000 7f5d0f59e640 1 -- 172.21.15.45:0/2264952015 wait complete. 2023-11-28T20:48:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.949+0000 7f5d0f59e640 1 Processor -- start 2023-11-28T20:48:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.950+0000 7f5d0f59e640 1 -- start start 2023-11-28T20:48:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.950+0000 7f5d0f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5d1009aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.950+0000 7f5d0f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d10135df0 0x7f5d101f0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.950+0000 7f5d0f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1009c110 0x7f5d100c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:00.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.950+0000 7f5d0f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1009bc80 con 0x7f5cfc0039a0 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1009b070 con 0x7f5d10135df0 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1009b350 con 0x7f5d1009c110 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5d1009aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5d1009aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58298/0 (socket says 172.21.15.45:58298) 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0e59c640 1 -- 172.21.15.45:0/1157824598 learned_addr learned my addr 172.21.15.45:0/1157824598 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0ed9d640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1009c110 0x7f5d100c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d07fff640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d10135df0 0x7f5d101f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0e59c640 1 -- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1009c110 msgr2=0x7f5d100c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0e59c640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1009c110 0x7f5d100c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0e59c640 1 -- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d10135df0 msgr2=0x7f5d101f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0e59c640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d10135df0 0x7f5d101f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d0e59c640 1 -- 172.21.15.45:0/1157824598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1009b630 con 0x7f5cfc0039a0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.951+0000 7f5d07fff640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d10135df0 0x7f5d101f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.952+0000 7f5d0e59c640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5d1009aa90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5d00008fd0 tx=0x7f5d00001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.952+0000 7f5d06ffd640 1 -- 172.21.15.45:0/1157824598 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d000067d0 con 0x7f5cfc0039a0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.952+0000 7f5d06ffd640 1 -- 172.21.15.45:0/1157824598 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00006950 con 0x7f5cfc0039a0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.952+0000 7f5d06ffd640 1 -- 172.21.15.45:0/1157824598 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00006d00 con 0x7f5cfc0039a0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.952+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1009b910 con 0x7f5cfc0039a0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.952+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d101f07a0 con 0x7f5cfc0039a0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.953+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1009baa0 con 0x7f5cfc0039a0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.954+0000 7f5d06ffd640 1 -- 172.21.15.45:0/1157824598 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5d000035f0 con 0x7f5cfc0039a0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.955+0000 7f5d06ffd640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5cf006af90 0x7f5cf006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.955+0000 7f5d06ffd640 1 -- 172.21.15.45:0/1157824598 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f5d000ab910 con 0x7f5cfc0039a0 2023-11-28T20:48:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.956+0000 7f5d07fff640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5cf006af90 0x7f5cf006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.957+0000 7f5d07fff640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5cf006af90 0x7f5cf006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80017a0 tx=0x7f5cf8000fa0 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:00.959+0000 7f5d06ffd640 1 -- 172.21.15.45:0/1157824598 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d000762c0 con 0x7f5cfc0039a0 2023-11-28T20:48:01.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.157+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5d1004e920 con 0x7f5cfc0039a0 2023-11-28T20:48:01.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.698+0000 7f5d06ffd640 1 -- 172.21.15.45:0/1157824598 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v62) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5d00006e60 con 0x7f5cfc0039a0 2023-11-28T20:48:01.701 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T20:48:01.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.701+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5cf006af90 msgr2=0x7f5cf006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.701+0000 7f5d0f59e640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5cf006af90 0x7f5cf006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80017a0 tx=0x7f5cf8000fa0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.701+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 msgr2=0x7f5d1009aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.701+0000 7f5d0f59e640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5d1009aa90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5d00008fd0 tx=0x7f5d00001120 comp rx=0 tx=0).stop 2023-11-28T20:48:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.702+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 shutdown_connections 2023-11-28T20:48:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.702+0000 7f5d0f59e640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5cf006af90 0x7f5cf006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.702+0000 7f5d0f59e640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d1009c110 0x7f5d100c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.702+0000 7f5d0f59e640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d10135df0 0x7f5d101f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.702+0000 7f5d0f59e640 1 --2- 172.21.15.45:0/1157824598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cfc0039a0 0x7f5d1009aa90 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.702+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 >> 172.21.15.45:0/1157824598 conn(0x7f5d100cec70 msgr2=0x7f5d100ad310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.703+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 shutdown_connections 2023-11-28T20:48:01.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.704+0000 7f5d0f59e640 1 -- 172.21.15.45:0/1157824598 wait complete. 2023-11-28T20:48:01.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T20:48:01.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.938+0000 7f5b34cf1640 1 Processor -- start 2023-11-28T20:48:01.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.938+0000 7f5b34cf1640 1 -- start start 2023-11-28T20:48:01.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.940+0000 7f5b34cf1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b30153cb0 0x7f5b3015a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:01.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.940+0000 7f5b34cf1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b300aeb40 0x7f5b300709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:01.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.940+0000 7f5b34cf1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5b30070ee0 0x7f5b3015d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:01.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.940+0000 7f5b34cf1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b30154090 con 0x7f5b300aeb40 2023-11-28T20:48:01.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.940+0000 7f5b34cf1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b30071310 con 0x7f5b30070ee0 2023-11-28T20:48:01.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.940+0000 7f5b34cf1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300717d0 con 0x7f5b30153cb0 2023-11-28T20:48:01.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.940+0000 7f5b27fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b300aeb40 0x7f5b300709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:01.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.940+0000 7f5b27fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b300aeb40 0x7f5b300709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58314/0 (socket says 172.21.15.45:58314) 2023-11-28T20:48:01.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.940+0000 7f5b27fff640 1 -- 172.21.15.45:0/3926194900 learned_addr learned my addr 172.21.15.45:0/3926194900 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:01.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.941+0000 7f5b2ffff640 1 --2- 172.21.15.45:0/3926194900 >> v2:172.21.15.190:3300/0 conn(0x7f5b30070ee0 0x7f5b3015d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:01.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.941+0000 7f5b27fff640 1 -- 172.21.15.45:0/3926194900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b30153cb0 msgr2=0x7f5b3015a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:01.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.941+0000 7f5b2f7fe640 1 --2- 172.21.15.45:0/3926194900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b30153cb0 0x7f5b3015a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:01.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.941+0000 7f5b27fff640 1 --2- 172.21.15.45:0/3926194900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b30153cb0 0x7f5b3015a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.941+0000 7f5b27fff640 1 -- 172.21.15.45:0/3926194900 >> v2:172.21.15.190:3300/0 conn(0x7f5b30070ee0 msgr2=0x7f5b3015d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:01.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.941+0000 7f5b27fff640 1 --2- 172.21.15.45:0/3926194900 >> v2:172.21.15.190:3300/0 conn(0x7f5b30070ee0 0x7f5b3015d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.941+0000 7f5b27fff640 1 -- 172.21.15.45:0/3926194900 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3006cfb0 con 0x7f5b300aeb40 2023-11-28T20:48:01.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.941+0000 7f5b2ffff640 1 --2- 172.21.15.45:0/3926194900 >> v2:172.21.15.190:3300/0 conn(0x7f5b30070ee0 0x7f5b3015d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:01.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.941+0000 7f5b2f7fe640 1 --2- 172.21.15.45:0/3926194900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b30153cb0 0x7f5b3015a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:01.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.942+0000 7f5b27fff640 1 --2- 172.21.15.45:0/3926194900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b300aeb40 0x7f5b300709a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5b300ae8c0 tx=0x7f5b1802ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775dc8e47eeac23c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:01.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b2effd640 1 -- 172.21.15.45:0/3926194900 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18030850 con 0x7f5b300aeb40 2023-11-28T20:48:01.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b2effd640 1 -- 172.21.15.45:0/3926194900 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b180309b0 con 0x7f5b300aeb40 2023-11-28T20:48:01.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b2effd640 1 -- 172.21.15.45:0/3926194900 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18030d40 con 0x7f5b300aeb40 2023-11-28T20:48:01.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b34cf1640 1 -- 172.21.15.45:0/3926194900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b300aeb40 msgr2=0x7f5b300709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:01.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/3926194900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b300aeb40 0x7f5b300709a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5b300ae8c0 tx=0x7f5b1802ca30 comp rx=0 tx=0).stop 2023-11-28T20:48:01.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b34cf1640 1 -- 172.21.15.45:0/3926194900 shutdown_connections 2023-11-28T20:48:01.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/3926194900 >> v2:172.21.15.190:3300/0 conn(0x7f5b30070ee0 0x7f5b3015d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/3926194900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b300aeb40 0x7f5b300709a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/3926194900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b30153cb0 0x7f5b3015a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b34cf1640 1 -- 172.21.15.45:0/3926194900 >> 172.21.15.45:0/3926194900 conn(0x7f5b30066610 msgr2=0x7f5b3006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:01.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.943+0000 7f5b34cf1640 1 -- 172.21.15.45:0/3926194900 shutdown_connections 2023-11-28T20:48:01.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.944+0000 7f5b34cf1640 1 -- 172.21.15.45:0/3926194900 wait complete. 2023-11-28T20:48:01.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.945+0000 7f5b34cf1640 1 Processor -- start 2023-11-28T20:48:01.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.945+0000 7f5b34cf1640 1 -- start start 2023-11-28T20:48:01.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.946+0000 7f5b34cf1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b30070ee0 0x7f5b300b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:01.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.946+0000 7f5b34cf1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b300aeb40 0x7f5b3015b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:01.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.946+0000 7f5b34cf1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b30153cb0 0x7f5b30152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:01.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.946+0000 7f5b34cf1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3015bbc0 con 0x7f5b30153cb0 2023-11-28T20:48:01.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.946+0000 7f5b34cf1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30152570 con 0x7f5b30070ee0 2023-11-28T20:48:01.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.946+0000 7f5b34cf1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3015be90 con 0x7f5b300aeb40 2023-11-28T20:48:01.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b2ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b30153cb0 0x7f5b30152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:01.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b2ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b30153cb0 0x7f5b30152030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58320/0 (socket says 172.21.15.45:58320) 2023-11-28T20:48:01.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b2ffff640 1 -- 172.21.15.45:0/2417549071 learned_addr learned my addr 172.21.15.45:0/2417549071 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b2f7fe640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b30070ee0 0x7f5b300b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b27fff640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b300aeb40 0x7f5b3015b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b2ffff640 1 -- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b300aeb40 msgr2=0x7f5b3015b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b2ffff640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b300aeb40 0x7f5b3015b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b2ffff640 1 -- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b30070ee0 msgr2=0x7f5b300b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b2ffff640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b30070ee0 0x7f5b300b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b2ffff640 1 -- 172.21.15.45:0/2417549071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b180094d0 con 0x7f5b30153cb0 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.947+0000 7f5b27fff640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b300aeb40 0x7f5b3015b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.948+0000 7f5b2ffff640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b30153cb0 0x7f5b30152030 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800cd10 tx=0x7f5b2800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.948+0000 7f5b2cff9640 1 -- 172.21.15.45:0/2417549071 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2800f680 con 0x7f5b30153cb0 2023-11-28T20:48:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.948+0000 7f5b2cff9640 1 -- 172.21.15.45:0/2417549071 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2800f800 con 0x7f5b30153cb0 2023-11-28T20:48:01.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.948+0000 7f5b2cff9640 1 -- 172.21.15.45:0/2417549071 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2800fbb0 con 0x7f5b30153cb0 2023-11-28T20:48:01.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.948+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b300af040 con 0x7f5b30153cb0 2023-11-28T20:48:01.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.948+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b300af600 con 0x7f5b30153cb0 2023-11-28T20:48:01.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.949+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b30067150 con 0x7f5b30153cb0 2023-11-28T20:48:01.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.950+0000 7f5b2cff9640 1 -- 172.21.15.45:0/2417549071 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5b2800b040 con 0x7f5b30153cb0 2023-11-28T20:48:01.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.952+0000 7f5b2cff9640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5b1006af60 0x7f5b1006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:01.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.952+0000 7f5b2cff9640 1 -- 172.21.15.45:0/2417549071 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6351+0+0 (secure 0 0 0) 0x7f5b2808a510 con 0x7f5b30153cb0 2023-11-28T20:48:01.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.953+0000 7f5b2f7fe640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5b1006af60 0x7f5b1006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:01.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.954+0000 7f5b2f7fe640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5b1006af60 0x7f5b1006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5b30157b90 tx=0x7f5b20008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:01.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:01.956+0000 7f5b2cff9640 1 -- 172.21.15.45:0/2417549071 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b28055120 con 0x7f5b30153cb0 2023-11-28T20:48:02.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.150+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5b300733d0 con 0x7f5b30153cb0 2023-11-28T20:48:02.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.710+0000 7f5b2cff9640 1 -- 172.21.15.45:0/2417549071 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v63) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5b28058740 con 0x7f5b30153cb0 2023-11-28T20:48:02.712 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T20:48:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.714+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5b1006af60 msgr2=0x7f5b1006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.714+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5b1006af60 0x7f5b1006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5b30157b90 tx=0x7f5b20008040 comp rx=0 tx=0).stop 2023-11-28T20:48:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.714+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b30153cb0 msgr2=0x7f5b30152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:02.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.714+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b30153cb0 0x7f5b30152030 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800cd10 tx=0x7f5b2800a650 comp rx=0 tx=0).stop 2023-11-28T20:48:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.715+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 shutdown_connections 2023-11-28T20:48:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.715+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f5b1006af60 0x7f5b1006d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.715+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b30153cb0 0x7f5b30152030 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.715+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b300aeb40 0x7f5b3015b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.715+0000 7f5b34cf1640 1 --2- 172.21.15.45:0/2417549071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b30070ee0 0x7f5b300b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.715+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 >> 172.21.15.45:0/2417549071 conn(0x7f5b30066610 msgr2=0x7f5b30152fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.716+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 shutdown_connections 2023-11-28T20:48:02.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.716+0000 7f5b34cf1640 1 -- 172.21.15.45:0/2417549071 wait complete. 2023-11-28T20:48:02.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T20:48:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.980+0000 7fe0c9367640 1 Processor -- start 2023-11-28T20:48:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.980+0000 7fe0c9367640 1 -- start start 2023-11-28T20:48:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.982+0000 7fe0c9367640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0c415b080 0x7fe0c40ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.982+0000 7fe0c9367640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe0c40abd70 0x7fe0c40718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.982+0000 7fe0c9367640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40ac270 con 0x7fe0c4071e00 2023-11-28T20:48:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.982+0000 7fe0c9367640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c40722b0 con 0x7fe0c40abd70 2023-11-28T20:48:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.982+0000 7fe0c9367640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c415b830 con 0x7fe0c415b080 2023-11-28T20:48:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.982+0000 7fe0c8b66640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe0c4071e00 0x7fe0c415eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36114/0 (socket says 172.21.15.45:36114) 2023-11-28T20:48:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.982+0000 7fe0c8b66640 1 -- 172.21.15.45:0/613352538 learned_addr learned my addr 172.21.15.45:0/613352538 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c37fe640 1 --2- 172.21.15.45:0/613352538 >> v2:172.21.15.190:3300/0 conn(0x7fe0c40abd70 0x7fe0c40718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766836354 0 0) 0x7fe0c40ac270 con 0x7fe0c4071e00 2023-11-28T20:48:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac003520 con 0x7fe0c4071e00 2023-11-28T20:48:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c3fff640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0c415b080 0x7fe0c40ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3801382784 0 0) 0x7fe0ac003520 con 0x7fe0c4071e00 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0c415b080 msgr2=0x7fe0c40ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0c415b080 0x7fe0c40ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 >> v2:172.21.15.190:3300/0 conn(0x7fe0c40abd70 msgr2=0x7fe0c40718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 --2- 172.21.15.45:0/613352538 >> v2:172.21.15.190:3300/0 conn(0x7fe0c40abd70 0x7fe0c40718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c415bb10 con 0x7fe0c4071e00 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fe0bc0019f0 con 0x7fe0c4071e00 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.983+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 >> v1:172.21.15.45:6789/0 conn(0x7fe0c4071e00 legacy=0x7fe0c415eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c2ffd640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ac003970 0x7fe0ac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c2ffd640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0ac0047c0 0x7fe0ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c2ffd640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0c415b080 0x7fe0ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:02.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40722b0 con 0x7fe0ac003970 2023-11-28T20:48:02.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c415b830 con 0x7fe0ac0047c0 2023-11-28T20:48:02.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac00b9d0 con 0x7fe0c415b080 2023-11-28T20:48:02.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c3fff640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ac003970 0x7fe0ac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:02.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c37fe640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0ac0047c0 0x7fe0ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:02.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c3fff640 1 -- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0c415b080 msgr2=0x7fe0ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:02.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c3fff640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0c415b080 0x7fe0ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c3fff640 1 -- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0ac0047c0 msgr2=0x7fe0ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:02.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c3fff640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0ac0047c0 0x7fe0ac004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.984+0000 7fe0c3fff640 1 -- 172.21.15.45:0/613352538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac003520 con 0x7fe0ac003970 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.985+0000 7fe0c3fff640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ac003970 0x7fe0ac003d90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400ba30 tx=0x7fe0ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffe5740662ca335 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.986+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4032680 con 0x7fe0ac003970 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.986+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b40327e0 con 0x7fe0ac003970 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.986+0000 7fe0c2ffd640 1 -- 172.21.15.45:0/613352538 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4032b70 con 0x7fe0ac003970 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.986+0000 7fe0c9367640 1 -- 172.21.15.45:0/613352538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ac003970 msgr2=0x7fe0ac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.986+0000 7fe0c9367640 1 --2- 172.21.15.45:0/613352538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ac003970 0x7fe0ac003d90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400ba30 tx=0x7fe0ac0057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.986+0000 7fe0c8b66640 1 -- 172.21.15.45:0/613352538 reap_dead start 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.986+0000 7fe0c9367640 1 -- 172.21.15.45:0/613352538 shutdown_connections 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.986+0000 7fe0c9367640 1 -- 172.21.15.45:0/613352538 >> 172.21.15.45:0/613352538 conn(0x7fe0c40cecd0 msgr2=0x7fe0c40cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.986+0000 7fe0c9367640 1 -- 172.21.15.45:0/613352538 shutdown_connections 2023-11-28T20:48:02.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.987+0000 7fe0c9367640 1 -- 172.21.15.45:0/613352538 wait complete. 2023-11-28T20:48:02.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.988+0000 7fe0c9367640 1 Processor -- start 2023-11-28T20:48:02.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.988+0000 7fe0c9367640 1 -- start start 2023-11-28T20:48:02.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.989+0000 7fe0c9367640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0ac003970 0x7fe0c40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:02.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.989+0000 7fe0c9367640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c415d2a0 0x7fe0c415d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:02.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.989+0000 7fe0c9367640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0c41452f0 0x7fe0c4145770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:02.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.989+0000 7fe0c9367640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c415dc60 con 0x7fe0c415d2a0 2023-11-28T20:48:02.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.989+0000 7fe0c9367640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40b1e00 con 0x7fe0c41452f0 2023-11-28T20:48:02.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.989+0000 7fe0c9367640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40ad8b0 con 0x7fe0ac003970 2023-11-28T20:48:02.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c415d2a0 0x7fe0c415d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:02.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c415d2a0 0x7fe0c415d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58346/0 (socket says 172.21.15.45:58346) 2023-11-28T20:48:02.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c37fe640 1 -- 172.21.15.45:0/3304071397 learned_addr learned my addr 172.21.15.45:0/3304071397 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:02.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c8b66640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0c41452f0 0x7fe0c4145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:02.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c37fe640 1 -- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0ac003970 msgr2=0x7fe0c40cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:02.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c3fff640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0ac003970 0x7fe0c40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:02.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c37fe640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0ac003970 0x7fe0c40cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c37fe640 1 -- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0c41452f0 msgr2=0x7fe0c4145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:02.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c37fe640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0c41452f0 0x7fe0c4145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:02.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c37fe640 1 -- 172.21.15.45:0/3304071397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c40cdb20 con 0x7fe0c415d2a0 2023-11-28T20:48:02.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.990+0000 7fe0c8b66640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0c41452f0 0x7fe0c4145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:02.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.991+0000 7fe0c37fe640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c415d2a0 0x7fe0c415d720 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80108e0 tx=0x7fe0b8010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:02.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.991+0000 7fe0c0ff9640 1 -- 172.21.15.45:0/3304071397 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b800eb60 con 0x7fe0c415d2a0 2023-11-28T20:48:02.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.991+0000 7fe0c0ff9640 1 -- 172.21.15.45:0/3304071397 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b800ecc0 con 0x7fe0c415d2a0 2023-11-28T20:48:02.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.991+0000 7fe0c0ff9640 1 -- 172.21.15.45:0/3304071397 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b801c400 con 0x7fe0c415d2a0 2023-11-28T20:48:02.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.991+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c4146080 con 0x7fe0c415d2a0 2023-11-28T20:48:02.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.991+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c40adde0 con 0x7fe0c415d2a0 2023-11-28T20:48:02.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.992+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c409d0f0 con 0x7fe0c415d2a0 2023-11-28T20:48:02.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.993+0000 7fe0c0ff9640 1 -- 172.21.15.45:0/3304071397 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe0b801c560 con 0x7fe0c415d2a0 2023-11-28T20:48:02.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.994+0000 7fe0c0ff9640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe0b006af60 0x7fe0b006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:02.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.994+0000 7fe0c0ff9640 1 -- 172.21.15.45:0/3304071397 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fe0b8089e40 con 0x7fe0c415d2a0 2023-11-28T20:48:02.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.995+0000 7fe0c3fff640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe0b006af60 0x7fe0b006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:02.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.996+0000 7fe0c3fff640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe0b006af60 0x7fe0b006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400ba00 tx=0x7fe0b403d040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:03.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:02.999+0000 7fe0c0ff9640 1 -- 172.21.15.45:0/3304071397 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0b8054db0 con 0x7fe0c415d2a0 2023-11-28T20:48:03.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.194+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe0c404e920 con 0x7fe0c415d2a0 2023-11-28T20:48:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.200+0000 7fe0c0ff9640 1 -- 172.21.15.45:0/3304071397 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v13) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe0b80583d0 con 0x7fe0c415d2a0 2023-11-28T20:48:03.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe0b006af60 msgr2=0x7fe0b006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe0b006af60 0x7fe0b006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400ba00 tx=0x7fe0b403d040 comp rx=0 tx=0).stop 2023-11-28T20:48:03.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c415d2a0 msgr2=0x7fe0c415d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c415d2a0 0x7fe0c415d720 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80108e0 tx=0x7fe0b8010da0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 shutdown_connections 2023-11-28T20:48:03.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe0b006af60 0x7fe0b006d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0c41452f0 0x7fe0c4145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c415d2a0 0x7fe0c415d720 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 --2- 172.21.15.45:0/3304071397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0ac003970 0x7fe0c40cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 >> 172.21.15.45:0/3304071397 conn(0x7fe0c40cecd0 msgr2=0x7fe0c41564c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:03.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 shutdown_connections 2023-11-28T20:48:03.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.203+0000 7fe0c9367640 1 -- 172.21.15.45:0/3304071397 wait complete. 2023-11-28T20:48:03.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T20:48:03.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.440+0000 7f31caeec640 1 Processor -- start 2023-11-28T20:48:03.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.440+0000 7f31caeec640 1 -- start start 2023-11-28T20:48:03.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.441+0000 7f31caeec640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c406a500 0x7f31c406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.441+0000 7f31caeec640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f31c415ddd0 0x7f31c415e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.441+0000 7f31caeec640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c406ca60 con 0x7f31c4157c10 2023-11-28T20:48:03.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.441+0000 7f31caeec640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c415eb40 con 0x7f31c415ddd0 2023-11-28T20:48:03.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.441+0000 7f31caeec640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4157ff0 con 0x7f31c406a500 2023-11-28T20:48:03.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.442+0000 7f31c8c88640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f31c4157c10 0x7f31c406da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36122/0 (socket says 172.21.15.45:36122) 2023-11-28T20:48:03.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.442+0000 7f31c8c88640 1 -- 172.21.15.45:0/1309226206 learned_addr learned my addr 172.21.15.45:0/1309226206 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.442+0000 7f31c3fff640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c406a500 0x7f31c406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.442+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590313448 0 0) 0x7f31c406ca60 con 0x7f31c4157c10 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.442+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b4003520 con 0x7f31c4157c10 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.442+0000 7f31c9489640 1 --2- 172.21.15.45:0/1309226206 >> v2:172.21.15.190:3300/0 conn(0x7f31c415ddd0 0x7f31c415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3354239256 0 0) 0x7f31b4003520 con 0x7f31c4157c10 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c406a500 msgr2=0x7f31c406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c406a500 0x7f31c406a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 >> v2:172.21.15.190:3300/0 conn(0x7f31c415ddd0 msgr2=0x7f31c415e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 --2- 172.21.15.45:0/1309226206 >> v2:172.21.15.190:3300/0 conn(0x7f31c415ddd0 0x7f31c415e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c406aea0 con 0x7f31c4157c10 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f31b8001670 con 0x7f31c4157c10 2023-11-28T20:48:03.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 >> v1:172.21.15.45:6789/0 conn(0x7f31c4157c10 legacy=0x7f31c406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c406a500 0x7f31b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31b4004d00 0x7f31b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c415eb40 con 0x7f31b4003970 2023-11-28T20:48:03.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4157ff0 con 0x7f31b4004d00 2023-11-28T20:48:03.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.443+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b400b980 con 0x7f31c406a500 2023-11-28T20:48:03.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.444+0000 7f31c8c88640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.444+0000 7f31c3fff640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c406a500 0x7f31b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.444+0000 7f31c9489640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31b4004d00 0x7f31b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.445+0000 7f31c8c88640 1 -- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c406a500 msgr2=0x7f31b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:03.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.445+0000 7f31c8c88640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c406a500 0x7f31b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.445+0000 7f31c8c88640 1 -- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31b4004d00 msgr2=0x7f31b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:03.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.445+0000 7f31c8c88640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31b4004d00 0x7f31b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.445+0000 7f31c8c88640 1 -- 172.21.15.45:0/1309226206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c406aea0 con 0x7f31b4003970 2023-11-28T20:48:03.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.445+0000 7f31c3fff640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c406a500 0x7f31b40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:03.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.447+0000 7f31c8c88640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31b4003d90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c340 tx=0x7f31b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2a3d2c3786418a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:03.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.447+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b802cbc0 con 0x7f31b4003970 2023-11-28T20:48:03.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.447+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b802cd20 con 0x7f31b4003970 2023-11-28T20:48:03.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.447+0000 7f31c37fe640 1 -- 172.21.15.45:0/1309226206 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8005af0 con 0x7f31b4003970 2023-11-28T20:48:03.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.448+0000 7f31caeec640 1 -- 172.21.15.45:0/1309226206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 msgr2=0x7f31b4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.448+0000 7f31caeec640 1 --2- 172.21.15.45:0/1309226206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31b4003d90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c340 tx=0x7f31b40057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.448+0000 7f31c9489640 1 -- 172.21.15.45:0/1309226206 reap_dead start 2023-11-28T20:48:03.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.448+0000 7f31caeec640 1 -- 172.21.15.45:0/1309226206 shutdown_connections 2023-11-28T20:48:03.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.448+0000 7f31caeec640 1 -- 172.21.15.45:0/1309226206 >> 172.21.15.45:0/1309226206 conn(0x7f31c40ced60 msgr2=0x7f31c40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:03.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.448+0000 7f31caeec640 1 -- 172.21.15.45:0/1309226206 shutdown_connections 2023-11-28T20:48:03.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.448+0000 7f31caeec640 1 -- 172.21.15.45:0/1309226206 wait complete. 2023-11-28T20:48:03.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.451+0000 7f31caeec640 1 Processor -- start 2023-11-28T20:48:03.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.451+0000 7f31caeec640 1 -- start start 2023-11-28T20:48:03.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.452+0000 7f31caeec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31c41f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.452+0000 7f31caeec640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c40b3950 0x7f31c40b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.452+0000 7f31caeec640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c40c9e50 0x7f31c40ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.452+0000 7f31caeec640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c40ca810 con 0x7f31b4003970 2023-11-28T20:48:03.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.452+0000 7f31caeec640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c41f2580 con 0x7f31c40c9e50 2023-11-28T20:48:03.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.452+0000 7f31caeec640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c41f2850 con 0x7f31c40b3950 2023-11-28T20:48:03.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.452+0000 7f31c8c88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31c41f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c8c88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31c41f2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58356/0 (socket says 172.21.15.45:58356) 2023-11-28T20:48:03.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c8c88640 1 -- 172.21.15.45:0/3553565163 learned_addr learned my addr 172.21.15.45:0/3553565163 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c9489640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c40c9e50 0x7f31c40ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c3fff640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c40b3950 0x7f31c40b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c8c88640 1 -- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c40b3950 msgr2=0x7f31c40b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c8c88640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c40b3950 0x7f31c40b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c8c88640 1 -- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c40c9e50 msgr2=0x7f31c40ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c8c88640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c40c9e50 0x7f31c40ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c8c88640 1 -- 172.21.15.45:0/3553565163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c41f2b30 con 0x7f31b4003970 2023-11-28T20:48:03.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c3fff640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c40b3950 0x7f31c40b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:03.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.453+0000 7f31c8c88640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31c41f2040 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c340 tx=0x7f31b8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:03.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.454+0000 7f31c17fa640 1 -- 172.21.15.45:0/3553565163 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8007070 con 0x7f31b4003970 2023-11-28T20:48:03.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.454+0000 7f31c17fa640 1 -- 172.21.15.45:0/3553565163 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b802ce90 con 0x7f31b4003970 2023-11-28T20:48:03.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.454+0000 7f31c17fa640 1 -- 172.21.15.45:0/3553565163 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b80073d0 con 0x7f31b4003970 2023-11-28T20:48:03.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.454+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c41f2e10 con 0x7f31b4003970 2023-11-28T20:48:03.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.454+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c40b3430 con 0x7f31b4003970 2023-11-28T20:48:03.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.456+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c415d580 con 0x7f31b4003970 2023-11-28T20:48:03.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.456+0000 7f31c17fa640 1 -- 172.21.15.45:0/3553565163 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f31b8032070 con 0x7f31b4003970 2023-11-28T20:48:03.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.457+0000 7f31c17fa640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f31a806af60 0x7f31a806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.457+0000 7f31c17fa640 1 -- 172.21.15.45:0/3553565163 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f31b80ab3a0 con 0x7f31b4003970 2023-11-28T20:48:03.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.458+0000 7f31c3fff640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f31a806af60 0x7f31a806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.459+0000 7f31c3fff640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f31a806af60 0x7f31a806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f31c40adf40 tx=0x7f31b0008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:03.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.462+0000 7f31c17fa640 1 -- 172.21.15.45:0/3553565163 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31b80761e0 con 0x7f31b4003970 2023-11-28T20:48:03.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.657+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f31c406d500 con 0x7f31b4003970 2023-11-28T20:48:03.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.665+0000 7f31c17fa640 1 -- 172.21.15.45:0/3553565163 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f31b8007e00 con 0x7f31b4003970 2023-11-28T20:48:03.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f31a806af60 msgr2=0x7f31a806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f31a806af60 0x7f31a806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f31c40adf40 tx=0x7f31b0008040 comp rx=0 tx=0).stop 2023-11-28T20:48:03.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 msgr2=0x7f31c41f2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31c41f2040 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c340 tx=0x7f31b8000fe0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 shutdown_connections 2023-11-28T20:48:03.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f31a806af60 0x7f31a806d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c40c9e50 0x7f31c40ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31c40b3950 0x7f31c40b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 --2- 172.21.15.45:0/3553565163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31b4003970 0x7f31c41f2040 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.667+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 >> 172.21.15.45:0/3553565163 conn(0x7f31c40ced60 msgr2=0x7f31c40aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:03.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.668+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 shutdown_connections 2023-11-28T20:48:03.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.668+0000 7f31caeec640 1 -- 172.21.15.45:0/3553565163 wait complete. 2023-11-28T20:48:03.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T20:48:03.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.908+0000 7f0ede797640 1 Processor -- start 2023-11-28T20:48:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.908+0000 7f0ede797640 1 -- start start 2023-11-28T20:48:03.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.910+0000 7f0ede797640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0ed815e840 0x7f0ed806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.910+0000 7f0ede797640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed815f720 0x7f0ed80717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.910+0000 7f0ede797640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed815ecd0 con 0x7f0ed806a500 2023-11-28T20:48:03.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.910+0000 7f0ede797640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed8071ce0 con 0x7f0ed815e840 2023-11-28T20:48:03.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.910+0000 7f0ede797640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed815f290 con 0x7f0ed815f720 2023-11-28T20:48:03.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.910+0000 7f0ed7fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0ed815e840 0x7f0ed806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.910+0000 7f0ed7fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0ed815e840 0x7f0ed806c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43280/0 (socket says 172.21.15.45:43280) 2023-11-28T20:48:03.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.910+0000 7f0ed7fff640 1 -- 172.21.15.45:0/662938771 learned_addr learned my addr 172.21.15.45:0/662938771 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:03.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.911+0000 7f0ecffff640 1 --2- 172.21.15.45:0/662938771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed815f720 0x7f0ed80717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.911+0000 7f0ed7fff640 1 -- 172.21.15.45:0/662938771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed815f720 msgr2=0x7f0ed80717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:03.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.911+0000 7f0ed7fff640 1 --2- 172.21.15.45:0/662938771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed815f720 0x7f0ed80717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.911+0000 7f0ed7fff640 1 -- 172.21.15.45:0/662938771 >> v1:172.21.15.190:6789/0 conn(0x7f0ed806a500 legacy=0x7f0ed80cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:03.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.911+0000 7f0ed7fff640 1 -- 172.21.15.45:0/662938771 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed815fb60 con 0x7f0ed815e840 2023-11-28T20:48:03.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.911+0000 7f0ecffff640 1 --2- 172.21.15.45:0/662938771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed815f720 0x7f0ed80717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:03.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.912+0000 7f0ed7fff640 1 --2- 172.21.15.45:0/662938771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 0x7f0ed806c350 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0ec80098a0 tx=0x7f0ec802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e75ca5b949afa16 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.912+0000 7f0ed77fe640 1 -- 172.21.15.45:0/662938771 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8031640 con 0x7f0ed815e840 2023-11-28T20:48:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.912+0000 7f0ed77fe640 1 -- 172.21.15.45:0/662938771 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec80317a0 con 0x7f0ed815e840 2023-11-28T20:48:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.912+0000 7f0ed77fe640 1 -- 172.21.15.45:0/662938771 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8031b30 con 0x7f0ed815e840 2023-11-28T20:48:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.913+0000 7f0ede797640 1 -- 172.21.15.45:0/662938771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 msgr2=0x7f0ed806c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.913+0000 7f0ede797640 1 --2- 172.21.15.45:0/662938771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 0x7f0ed806c350 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0ec80098a0 tx=0x7f0ec802b8f0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.913+0000 7f0ede797640 1 -- 172.21.15.45:0/662938771 shutdown_connections 2023-11-28T20:48:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.913+0000 7f0ede797640 1 --2- 172.21.15.45:0/662938771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed815f720 0x7f0ed80717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.913+0000 7f0ede797640 1 --2- 172.21.15.45:0/662938771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 0x7f0ed806c350 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.913+0000 7f0ede797640 1 -- 172.21.15.45:0/662938771 >> 172.21.15.45:0/662938771 conn(0x7f0ed80ced60 msgr2=0x7f0ed80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.913+0000 7f0ede797640 1 -- 172.21.15.45:0/662938771 shutdown_connections 2023-11-28T20:48:03.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.913+0000 7f0ede797640 1 -- 172.21.15.45:0/662938771 wait complete. 2023-11-28T20:48:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.915+0000 7f0ede797640 1 Processor -- start 2023-11-28T20:48:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.915+0000 7f0ede797640 1 -- start start 2023-11-28T20:48:03.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.916+0000 7f0ede797640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed806a500 0x7f0ed809bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.916+0000 7f0ede797640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 0x7f0ed80cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.916+0000 7f0ede797640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed815f720 0x7f0ed809a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.916+0000 7f0ede797640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed8154b90 con 0x7f0ed815e840 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.916+0000 7f0ede797640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed809c660 con 0x7f0ed815f720 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.916+0000 7f0ede797640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed80cb3a0 con 0x7f0ed806a500 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.916+0000 7f0ecffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 0x7f0ed80cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0ecffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 0x7f0ed80cae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43296/0 (socket says 172.21.15.45:43296) 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0ecffff640 1 -- 172.21.15.45:0/2460637678 learned_addr learned my addr 172.21.15.45:0/2460637678 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0ed7fff640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed806a500 0x7f0ed809bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0ecffff640 1 -- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed806a500 msgr2=0x7f0ed809bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0ecffff640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed806a500 0x7f0ed809bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0edcd34640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed815f720 0x7f0ed809a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0ecffff640 1 -- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed815f720 msgr2=0x7f0ed809a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0ecffff640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed815f720 0x7f0ed809a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0ecffff640 1 -- 172.21.15.45:0/2460637678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec80094d0 con 0x7f0ed815e840 2023-11-28T20:48:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0edcd34640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed815f720 0x7f0ed809a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.917+0000 7f0ecffff640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 0x7f0ed80cae60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000c820 tx=0x7f0ec000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.918+0000 7f0ed57fa640 1 -- 172.21.15.45:0/2460637678 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0007a30 con 0x7f0ed815e840 2023-11-28T20:48:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.918+0000 7f0ed57fa640 1 -- 172.21.15.45:0/2460637678 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec0007b90 con 0x7f0ed815e840 2023-11-28T20:48:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.918+0000 7f0ed57fa640 1 -- 172.21.15.45:0/2460637678 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec000ce10 con 0x7f0ed815e840 2023-11-28T20:48:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.918+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed809c380 con 0x7f0ed815e840 2023-11-28T20:48:03.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.918+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed8099e50 con 0x7f0ed815e840 2023-11-28T20:48:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.919+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed80733d0 con 0x7f0ed815e840 2023-11-28T20:48:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.920+0000 7f0ed57fa640 1 -- 172.21.15.45:0/2460637678 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f0ec00113f0 con 0x7f0ed815e840 2023-11-28T20:48:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.921+0000 7f0ed57fa640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f0eb806af60 0x7f0eb806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.921+0000 7f0ed57fa640 1 -- 172.21.15.45:0/2460637678 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f0ec0089aa0 con 0x7f0ed815e840 2023-11-28T20:48:03.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.922+0000 7f0ed7fff640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f0eb806af60 0x7f0eb806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:03.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.923+0000 7f0ed7fff640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f0eb806af60 0x7f0eb806d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8036000 tx=0x7f0ec803c040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:03.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:03.925+0000 7f0ed57fa640 1 -- 172.21.15.45:0/2460637678 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ec0054850 con 0x7f0ed815e840 2023-11-28T20:48:04.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.121+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f0ed80ce2b0 con 0x7f0ed815e840 2023-11-28T20:48:04.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.128+0000 7f0ed57fa640 1 -- 172.21.15.45:0/2460637678 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v15) v1 ==== 115+0+0 (secure 0 0 0) 0x7f0ec001f0b0 con 0x7f0ed815e840 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f0eb806af60 msgr2=0x7f0eb806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f0eb806af60 0x7f0eb806d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8036000 tx=0x7f0ec803c040 comp rx=0 tx=0).stop 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 msgr2=0x7f0ed80cae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 0x7f0ed80cae60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000c820 tx=0x7f0ec000cce0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 shutdown_connections 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f0eb806af60 0x7f0eb806d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed815f720 0x7f0ed809a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ed815e840 0x7f0ed80cae60 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 --2- 172.21.15.45:0/2460637678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed806a500 0x7f0ed809bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.131+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 >> 172.21.15.45:0/2460637678 conn(0x7f0ed80ced60 msgr2=0x7f0ed8159350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.132+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 shutdown_connections 2023-11-28T20:48:04.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.132+0000 7f0ede797640 1 -- 172.21.15.45:0/2460637678 wait complete. 2023-11-28T20:48:04.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-11-28T20:48:04.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.397+0000 7fe7af2e6640 1 Processor -- start 2023-11-28T20:48:04.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.397+0000 7fe7af2e6640 1 -- start start 2023-11-28T20:48:04.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.399+0000 7fe7af2e6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe7a806f880 0x7fe7a815e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.399+0000 7fe7af2e6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7a80af390 0x7fe7a80acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.399+0000 7fe7af2e6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a806b510 0x7fe7a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.399+0000 7fe7af2e6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a806bee0 con 0x7fe7a80af390 2023-11-28T20:48:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.399+0000 7fe7af2e6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a806fc60 con 0x7fe7a806f880 2023-11-28T20:48:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.399+0000 7fe7af2e6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a80ad640 con 0x7fe7a806b510 2023-11-28T20:48:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7adae3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7a80af390 0x7fe7a80acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7adae3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7a80af390 0x7fe7a80acd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43308/0 (socket says 172.21.15.45:43308) 2023-11-28T20:48:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7adae3640 1 -- 172.21.15.45:0/1669354232 learned_addr learned my addr 172.21.15.45:0/1669354232 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7ae2e4640 1 --2- 172.21.15.45:0/1669354232 >> v2:172.21.15.190:3300/0 conn(0x7fe7a806f880 0x7fe7a815e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7aeae5640 1 --2- 172.21.15.45:0/1669354232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a806b510 0x7fe7a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7adae3640 1 -- 172.21.15.45:0/1669354232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a806b510 msgr2=0x7fe7a806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7adae3640 1 --2- 172.21.15.45:0/1669354232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a806b510 0x7fe7a806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7adae3640 1 -- 172.21.15.45:0/1669354232 >> v2:172.21.15.190:3300/0 conn(0x7fe7a806f880 msgr2=0x7fe7a815e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7adae3640 1 --2- 172.21.15.45:0/1669354232 >> v2:172.21.15.190:3300/0 conn(0x7fe7a806f880 0x7fe7a815e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7adae3640 1 -- 172.21.15.45:0/1669354232 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a815d850 con 0x7fe7a80af390 2023-11-28T20:48:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.400+0000 7fe7ae2e4640 1 --2- 172.21.15.45:0/1669354232 >> v2:172.21.15.190:3300/0 conn(0x7fe7a806f880 0x7fe7a815e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:48:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.401+0000 7fe7adae3640 1 --2- 172.21.15.45:0/1669354232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80af390 0x7fe7a80acd80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe798009870 tx=0x7fe79802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb12d111b47078e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.402+0000 7fe7ad2e2640 1 -- 172.21.15.45:0/1669354232 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe798031640 con 0x7fe7a80af390 2023-11-28T20:48:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.402+0000 7fe7ad2e2640 1 -- 172.21.15.45:0/1669354232 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7980317a0 con 0x7fe7a80af390 2023-11-28T20:48:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.402+0000 7fe7ad2e2640 1 -- 172.21.15.45:0/1669354232 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe798031b30 con 0x7fe7a80af390 2023-11-28T20:48:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.402+0000 7fe7af2e6640 1 -- 172.21.15.45:0/1669354232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80af390 msgr2=0x7fe7a80acd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.402+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/1669354232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80af390 0x7fe7a80acd80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe798009870 tx=0x7fe79802b8f0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.402+0000 7fe7af2e6640 1 -- 172.21.15.45:0/1669354232 shutdown_connections 2023-11-28T20:48:04.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.402+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/1669354232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a806b510 0x7fe7a806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.403+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/1669354232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80af390 0x7fe7a80acd80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.403+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/1669354232 >> v2:172.21.15.190:3300/0 conn(0x7fe7a806f880 0x7fe7a815e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.403+0000 7fe7af2e6640 1 -- 172.21.15.45:0/1669354232 >> 172.21.15.45:0/1669354232 conn(0x7fe7a80712f0 msgr2=0x7fe7a80716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.403+0000 7fe7af2e6640 1 -- 172.21.15.45:0/1669354232 shutdown_connections 2023-11-28T20:48:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.403+0000 7fe7af2e6640 1 -- 172.21.15.45:0/1669354232 wait complete. 2023-11-28T20:48:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.404+0000 7fe7af2e6640 1 Processor -- start 2023-11-28T20:48:04.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.405+0000 7fe7af2e6640 1 -- start start 2023-11-28T20:48:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.405+0000 7fe7af2e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a806b510 0x7fe7a8153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7af2e6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a806f880 0x7fe7a80b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7af2e6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a80af390 0x7fe7a815f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7af2e6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a8156850 con 0x7fe7a806f880 2023-11-28T20:48:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7af2e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a8155580 con 0x7fe7a806b510 2023-11-28T20:48:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7af2e6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a81557b0 con 0x7fe7a80af390 2023-11-28T20:48:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7adae3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a806f880 0x7fe7a80b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7adae3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a806f880 0x7fe7a80b0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43318/0 (socket says 172.21.15.45:43318) 2023-11-28T20:48:04.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7adae3640 1 -- 172.21.15.45:0/131396547 learned_addr learned my addr 172.21.15.45:0/131396547 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7aeae5640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a80af390 0x7fe7a815f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7adae3640 1 -- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a80af390 msgr2=0x7fe7a815f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7adae3640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a80af390 0x7fe7a815f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7adae3640 1 -- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a806b510 msgr2=0x7fe7a8153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7adae3640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a806b510 0x7fe7a8153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.406+0000 7fe7adae3640 1 -- 172.21.15.45:0/131396547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7980094d0 con 0x7fe7a806f880 2023-11-28T20:48:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.407+0000 7fe7adae3640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a806f880 0x7fe7a80b0180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe798000c00 tx=0x7fe7980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.407+0000 7fe79effd640 1 -- 172.21.15.45:0/131396547 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79803ac60 con 0x7fe7a806f880 2023-11-28T20:48:04.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.407+0000 7fe79effd640 1 -- 172.21.15.45:0/131396547 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79803adc0 con 0x7fe7a806f880 2023-11-28T20:48:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.407+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a8155a90 con 0x7fe7a806f880 2023-11-28T20:48:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.407+0000 7fe79effd640 1 -- 172.21.15.45:0/131396547 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe798044630 con 0x7fe7a806f880 2023-11-28T20:48:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.407+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a8156050 con 0x7fe7a806f880 2023-11-28T20:48:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.408+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a80aea60 con 0x7fe7a806f880 2023-11-28T20:48:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.409+0000 7fe79effd640 1 -- 172.21.15.45:0/131396547 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe79802f050 con 0x7fe7a806f880 2023-11-28T20:48:04.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.410+0000 7fe79effd640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe78c06af60 0x7fe78c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.410+0000 7fe79effd640 1 -- 172.21.15.45:0/131396547 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fe7980ab880 con 0x7fe7a806f880 2023-11-28T20:48:04.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.411+0000 7fe7ae2e4640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe78c06af60 0x7fe78c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.412+0000 7fe7ae2e4640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe78c06af60 0x7fe78c06d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4006fd0 tx=0x7fe7a4008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:04.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.415+0000 7fe79effd640 1 -- 172.21.15.45:0/131396547 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7980765a0 con 0x7fe7a806f880 2023-11-28T20:48:04.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.683+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7fe7a809e100 con 0x7fe7a806f880 2023-11-28T20:48:04.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.715+0000 7fe79effd640 1 -- 172.21.15.45:0/131396547 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v518) v1 ==== 158+0+0 (secure 0 0 0) 0x7fe798031e60 con 0x7fe7a806f880 2023-11-28T20:48:04.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe78c06af60 msgr2=0x7fe78c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe78c06af60 0x7fe78c06d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4006fd0 tx=0x7fe7a4008040 comp rx=0 tx=0).stop 2023-11-28T20:48:04.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a806f880 msgr2=0x7fe7a80b0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a806f880 0x7fe7a80b0180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe798000c00 tx=0x7fe7980046c0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 shutdown_connections 2023-11-28T20:48:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe78c06af60 0x7fe78c06d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7a80af390 0x7fe7a815f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a806f880 0x7fe7a80b0180 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 --2- 172.21.15.45:0/131396547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a806b510 0x7fe7a8153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 >> 172.21.15.45:0/131396547 conn(0x7fe7a80712f0 msgr2=0x7fe7a80b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 shutdown_connections 2023-11-28T20:48:04.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.718+0000 7fe7af2e6640 1 -- 172.21.15.45:0/131396547 wait complete. 2023-11-28T20:48:04.744 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-11-28T20:48:04.745 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-11-28T20:48:04.745 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T20:48:04.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:48:04.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.972+0000 7f17dfdc9640 1 Processor -- start 2023-11-28T20:48:04.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.972+0000 7f17dfdc9640 1 -- start start 2023-11-28T20:48:04.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17dfdc9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17d81438b0 0x7f17d80b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17dfdc9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f17d80b1340 0x7f17d80b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17dfdc9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d815dcc0 con 0x7f17d80b24e0 2023-11-28T20:48:04.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17dfdc9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d815e840 con 0x7f17d81438b0 2023-11-28T20:48:04.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17dfdc9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d815df10 con 0x7f17d80b1340 2023-11-28T20:48:04.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17ddb65640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17d81438b0 0x7f17d80b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17ddb65640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17d81438b0 0x7f17d80b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43330/0 (socket says 172.21.15.45:43330) 2023-11-28T20:48:04.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17ddb65640 1 -- 172.21.15.45:0/3483502120 learned_addr learned my addr 172.21.15.45:0/3483502120 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:04.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17de366640 1 --2- 172.21.15.45:0/3483502120 >> v2:172.21.15.190:3300/0 conn(0x7f17d80b1340 0x7f17d80b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.974+0000 7f17ddb65640 1 -- 172.21.15.45:0/3483502120 >> v2:172.21.15.190:3300/0 conn(0x7f17d80b1340 msgr2=0x7f17d80b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:04.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.975+0000 7f17ddb65640 1 --2- 172.21.15.45:0/3483502120 >> v2:172.21.15.190:3300/0 conn(0x7f17d80b1340 0x7f17d80b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.975+0000 7f17ddb65640 1 -- 172.21.15.45:0/3483502120 >> v1:172.21.15.190:6789/0 conn(0x7f17d80b24e0 legacy=0x7f17d80b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:04.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.975+0000 7f17ddb65640 1 -- 172.21.15.45:0/3483502120 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d815e1f0 con 0x7f17d81438b0 2023-11-28T20:48:04.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.975+0000 7f17de366640 1 --2- 172.21.15.45:0/3483502120 >> v2:172.21.15.190:3300/0 conn(0x7f17d80b1340 0x7f17d80b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:04.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.976+0000 7f17ddb65640 1 --2- 172.21.15.45:0/3483502120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 0x7f17d80b1fa0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f17d815a6d0 tx=0x7f17d402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e375c5cc04464c6c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:04.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.976+0000 7f17dcb63640 1 -- 172.21.15.45:0/3483502120 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d4031750 con 0x7f17d81438b0 2023-11-28T20:48:04.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.976+0000 7f17dcb63640 1 -- 172.21.15.45:0/3483502120 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d40318b0 con 0x7f17d81438b0 2023-11-28T20:48:04.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.976+0000 7f17dcb63640 1 -- 172.21.15.45:0/3483502120 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d4031c40 con 0x7f17d81438b0 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.977+0000 7f17dfdc9640 1 -- 172.21.15.45:0/3483502120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 msgr2=0x7f17d80b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.977+0000 7f17dfdc9640 1 --2- 172.21.15.45:0/3483502120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 0x7f17d80b1fa0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f17d815a6d0 tx=0x7f17d402b7f0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.977+0000 7f17dfdc9640 1 -- 172.21.15.45:0/3483502120 shutdown_connections 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.977+0000 7f17dfdc9640 1 --2- 172.21.15.45:0/3483502120 >> v2:172.21.15.190:3300/0 conn(0x7f17d80b1340 0x7f17d80b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.977+0000 7f17dfdc9640 1 --2- 172.21.15.45:0/3483502120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 0x7f17d80b1fa0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.977+0000 7f17dfdc9640 1 -- 172.21.15.45:0/3483502120 >> 172.21.15.45:0/3483502120 conn(0x7f17d806a500 msgr2=0x7f17d806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.977+0000 7f17dfdc9640 1 -- 172.21.15.45:0/3483502120 shutdown_connections 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.977+0000 7f17dfdc9640 1 -- 172.21.15.45:0/3483502120 wait complete. 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.979+0000 7f17dfdc9640 1 Processor -- start 2023-11-28T20:48:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.979+0000 7f17dfdc9640 1 -- start start 2023-11-28T20:48:04.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.980+0000 7f17dfdc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17d80b1340 0x7f17d8140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.980+0000 7f17dfdc9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17d80b24e0 0x7f17d81572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.980+0000 7f17dfdc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 0x7f17d813e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.980+0000 7f17dfdc9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d813e970 con 0x7f17d81438b0 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.980+0000 7f17dfdc9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d815b8e0 con 0x7f17d80b24e0 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.980+0000 7f17dfdc9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d815bbc0 con 0x7f17d80b1340 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.980+0000 7f17de366640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 0x7f17d813e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.980+0000 7f17de366640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 0x7f17d813e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43332/0 (socket says 172.21.15.45:43332) 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.980+0000 7f17de366640 1 -- 172.21.15.45:0/1848480588 learned_addr learned my addr 172.21.15.45:0/1848480588 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17dd364640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17d80b24e0 0x7f17d81572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17ddb65640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17d80b1340 0x7f17d8140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17de366640 1 -- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17d80b1340 msgr2=0x7f17d8140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17de366640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17d80b1340 0x7f17d8140380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17de366640 1 -- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17d80b24e0 msgr2=0x7f17d81572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17de366640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17d80b24e0 0x7f17d81572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17de366640 1 -- 172.21.15.45:0/1848480588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d40094d0 con 0x7f17d81438b0 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17dd364640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17d80b24e0 0x7f17d81572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17de366640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 0x7f17d813e430 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f17d000cd80 tx=0x7f17d000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17ca7fc640 1 -- 172.21.15.45:0/1848480588 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0003b90 con 0x7f17d81438b0 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17ca7fc640 1 -- 172.21.15.45:0/1848480588 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d0003d10 con 0x7f17d81438b0 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17ca7fc640 1 -- 172.21.15.45:0/1848480588 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0004260 con 0x7f17d81438b0 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.981+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d815bea0 con 0x7f17d81438b0 2023-11-28T20:48:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.982+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d813eba0 con 0x7f17d81438b0 2023-11-28T20:48:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.983+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d815c3c0 con 0x7f17d81438b0 2023-11-28T20:48:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.984+0000 7f17ca7fc640 1 -- 172.21.15.45:0/1848480588 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f17d000b040 con 0x7f17d81438b0 2023-11-28T20:48:04.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.985+0000 7f17ca7fc640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f17bc06af90 0x7f17bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:04.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.985+0000 7f17ca7fc640 1 -- 172.21.15.45:0/1848480588 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f17d008a020 con 0x7f17d81438b0 2023-11-28T20:48:04.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.986+0000 7f17ddb65640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f17bc06af90 0x7f17bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:04.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.987+0000 7f17ddb65640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f17bc06af90 0x7f17bc06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17d4035000 tx=0x7f17d4008030 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:04.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:04.989+0000 7f17ca7fc640 1 -- 172.21.15.45:0/1848480588 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17d815c3c0 con 0x7f17d81438b0 2023-11-28T20:48:05.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.186+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17d815c3c0 con 0x7f17d81438b0 2023-11-28T20:48:05.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.188+0000 7f17ca7fc640 1 -- 172.21.15.45:0/1848480588 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+14135 (secure 0 0 0) 0x7f17d815c3c0 con 0x7f17d81438b0 2023-11-28T20:48:05.191 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:05.191 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":63,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:48:02.700251+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6834/1984946086":"2023-11-29T20:47:59.679919+0000","172.21.15.190:0/2065659095":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/3262798906":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/664590003":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1813745990":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6832/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:6833/1081538426":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3717946281":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3679315161":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1881573438":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1866894774":"2023-11-29T20:35:55.110774+0000","172.21.15.190:0/3153178694":"2023-11-29T20:47:55.564970+0000","172.21.15.190:6832/2725659217":"2023-11-29T20:47:55.564970+0000","172.21.15.45:6832/1984946086":"2023-11-29T20:47:59.679919+0000","172.21.15.190:6833/2725659217":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/1599738782":"2023-11-29T20:47:55.564970+0000","172.21.15.190:0/3278716326":"2023-11-29T20:47:55.564970+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:48:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f17bc06af90 msgr2=0x7f17bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f17bc06af90 0x7f17bc06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17d4035000 tx=0x7f17d4008030 comp rx=0 tx=0).stop 2023-11-28T20:48:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 msgr2=0x7f17d813e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 0x7f17d813e430 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f17d000cd80 tx=0x7f17d000a650 comp rx=0 tx=0).stop 2023-11-28T20:48:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 shutdown_connections 2023-11-28T20:48:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f17bc06af90 0x7f17bc06d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17d81438b0 0x7f17d813e430 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17d80b24e0 0x7f17d81572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 --2- 172.21.15.45:0/1848480588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17d80b1340 0x7f17d8140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.191+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 >> 172.21.15.45:0/1848480588 conn(0x7f17d806a500 msgr2=0x7f17d80cc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.192+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 shutdown_connections 2023-11-28T20:48:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.192+0000 7f17dfdc9640 1 -- 172.21.15.45:0/1848480588 wait complete. 2023-11-28T20:48:05.213 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:48:05.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:48:05.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.442+0000 7fb6711ae640 1 Processor -- start 2023-11-28T20:48:05.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.443+0000 7fb6711ae640 1 -- start start 2023-11-28T20:48:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.444+0000 7fb6711ae640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb66c0d0040 0x7fb66c0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.444+0000 7fb6711ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0b1d40 0x7fb66c0d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.444+0000 7fb6711ae640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c0b90a0 con 0x7fb66c0b2290 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.444+0000 7fb6711ae640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb66c0d1920 con 0x7fb66c0d0040 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.444+0000 7fb6711ae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c0d2dc0 con 0x7fb66c0b1d40 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.445+0000 7fb66bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb66c0d0040 0x7fb66c0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.445+0000 7fb66bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb66c0d0040 0x7fb66c0d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43334/0 (socket says 172.21.15.45:43334) 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.445+0000 7fb66bfff640 1 -- 172.21.15.45:0/678276457 learned_addr learned my addr 172.21.15.45:0/678276457 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.445+0000 7fb6709ad640 1 --2- 172.21.15.45:0/678276457 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0b1d40 0x7fb66c0d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.445+0000 7fb66bfff640 1 -- 172.21.15.45:0/678276457 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0b1d40 msgr2=0x7fb66c0d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.445+0000 7fb66bfff640 1 --2- 172.21.15.45:0/678276457 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0b1d40 0x7fb66c0d3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.445+0000 7fb66bfff640 1 -- 172.21.15.45:0/678276457 >> v1:172.21.15.190:6789/0 conn(0x7fb66c0b2290 legacy=0x7fb66c0d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.445+0000 7fb66bfff640 1 -- 172.21.15.45:0/678276457 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c0d0960 con 0x7fb66c0d0040 2023-11-28T20:48:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.447+0000 7fb66bfff640 1 --2- 172.21.15.45:0/678276457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0d0040 0x7fb66c0d0420 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb654009870 tx=0x7fb65402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f632c49d25613d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.447+0000 7fb66affd640 1 -- 172.21.15.45:0/678276457 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65402f830 con 0x7fb66c0d0040 2023-11-28T20:48:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.447+0000 7fb66affd640 1 -- 172.21.15.45:0/678276457 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65402f990 con 0x7fb66c0d0040 2023-11-28T20:48:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.447+0000 7fb66affd640 1 -- 172.21.15.45:0/678276457 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65402fd20 con 0x7fb66c0d0040 2023-11-28T20:48:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.448+0000 7fb6711ae640 1 -- 172.21.15.45:0/678276457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0d0040 msgr2=0x7fb66c0d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.448+0000 7fb6711ae640 1 --2- 172.21.15.45:0/678276457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0d0040 0x7fb66c0d0420 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb654009870 tx=0x7fb65402ba10 comp rx=0 tx=0).stop 2023-11-28T20:48:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.448+0000 7fb6711ae640 1 -- 172.21.15.45:0/678276457 shutdown_connections 2023-11-28T20:48:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.448+0000 7fb6711ae640 1 --2- 172.21.15.45:0/678276457 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0b1d40 0x7fb66c0d3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.448+0000 7fb6711ae640 1 --2- 172.21.15.45:0/678276457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0d0040 0x7fb66c0d0420 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.448+0000 7fb6711ae640 1 -- 172.21.15.45:0/678276457 >> 172.21.15.45:0/678276457 conn(0x7fb66c070f40 msgr2=0x7fb66c071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.448+0000 7fb6711ae640 1 -- 172.21.15.45:0/678276457 shutdown_connections 2023-11-28T20:48:05.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.448+0000 7fb6711ae640 1 -- 172.21.15.45:0/678276457 wait complete. 2023-11-28T20:48:05.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.450+0000 7fb6711ae640 1 Processor -- start 2023-11-28T20:48:05.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.450+0000 7fb6711ae640 1 -- start start 2023-11-28T20:48:05.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb6711ae640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb66c0b1d40 0x7fb66c156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb6711ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0b2290 0x7fb66c159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb6711ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0d0040 0x7fb66c0b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb6711ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c156ec0 con 0x7fb66c0b2290 2023-11-28T20:48:05.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb6711ae640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1f36f0 con 0x7fb66c0b1d40 2023-11-28T20:48:05.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb6711ae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1f39d0 con 0x7fb66c0d0040 2023-11-28T20:48:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0b2290 0x7fb66c159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0b2290 0x7fb66c159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43336/0 (socket says 172.21.15.45:43336) 2023-11-28T20:48:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb66b7fe640 1 -- 172.21.15.45:0/2829481412 learned_addr learned my addr 172.21.15.45:0/2829481412 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb66bfff640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb66c0b1d40 0x7fb66c156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb6709ad640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0d0040 0x7fb66c0b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.451+0000 7fb66b7fe640 1 -- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0d0040 msgr2=0x7fb66c0b7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb66b7fe640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0d0040 0x7fb66c0b7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb66b7fe640 1 -- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb66c0b1d40 msgr2=0x7fb66c156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb66b7fe640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb66c0b1d40 0x7fb66c156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb66b7fe640 1 -- 172.21.15.45:0/2829481412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6540094d0 con 0x7fb66c0b2290 2023-11-28T20:48:05.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb66b7fe640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0b2290 0x7fb66c159df0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb65c00c870 tx=0x7fb65c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:05.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb668ff9640 1 -- 172.21.15.45:0/2829481412 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c003b70 con 0x7fb66c0b2290 2023-11-28T20:48:05.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb668ff9640 1 -- 172.21.15.45:0/2829481412 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65c003cf0 con 0x7fb66c0b2290 2023-11-28T20:48:05.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb668ff9640 1 -- 172.21.15.45:0/2829481412 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c00aa90 con 0x7fb66c0b2290 2023-11-28T20:48:05.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66c1f3cb0 con 0x7fb66c0b2290 2023-11-28T20:48:05.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.452+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c158130 con 0x7fb66c0b2290 2023-11-28T20:48:05.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.454+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66c071b30 con 0x7fb66c0b2290 2023-11-28T20:48:05.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.455+0000 7fb668ff9640 1 -- 172.21.15.45:0/2829481412 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb65c00abf0 con 0x7fb66c0b2290 2023-11-28T20:48:05.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.455+0000 7fb668ff9640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb64c06af60 0x7fb64c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.456+0000 7fb668ff9640 1 -- 172.21.15.45:0/2829481412 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fb65c0897d0 con 0x7fb66c0b2290 2023-11-28T20:48:05.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.456+0000 7fb66bfff640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb64c06af60 0x7fb64c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.457+0000 7fb66bfff640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb64c06af60 0x7fb64c06d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb654036000 tx=0x7fb654008030 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:05.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.460+0000 7fb668ff9640 1 -- 172.21.15.45:0/2829481412 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb65c054600 con 0x7fb66c0b2290 2023-11-28T20:48:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.653+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb66c0b6480 con 0x7fb66c0b2290 2023-11-28T20:48:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.654+0000 7fb668ff9640 1 -- 172.21.15.45:0/2829481412 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb65c003e60 con 0x7fb66c0b2290 2023-11-28T20:48:05.656 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:48:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.656+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb64c06af60 msgr2=0x7fb64c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.656+0000 7fb6711ae640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb64c06af60 0x7fb64c06d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb654036000 tx=0x7fb654008030 comp rx=0 tx=0).stop 2023-11-28T20:48:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.656+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0b2290 msgr2=0x7fb66c159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.656+0000 7fb6711ae640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0b2290 0x7fb66c159df0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb65c00c870 tx=0x7fb65c00cd30 comp rx=0 tx=0).stop 2023-11-28T20:48:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.657+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 shutdown_connections 2023-11-28T20:48:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.657+0000 7fb6711ae640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb64c06af60 0x7fb64c06d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.657+0000 7fb6711ae640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb66c0d0040 0x7fb66c0b7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.657+0000 7fb6711ae640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb66c0b2290 0x7fb66c159df0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.657+0000 7fb6711ae640 1 --2- 172.21.15.45:0/2829481412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb66c0b1d40 0x7fb66c156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.657+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 >> 172.21.15.45:0/2829481412 conn(0x7fb66c070f40 msgr2=0x7fb66c0d3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:05.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.657+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 shutdown_connections 2023-11-28T20:48:05.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.657+0000 7fb6711ae640 1 -- 172.21.15.45:0/2829481412 wait complete. 2023-11-28T20:48:05.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-11-28T20:48:05.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.909+0000 7fae8e067640 1 Processor -- start 2023-11-28T20:48:05.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.909+0000 7fae8e067640 1 -- start start 2023-11-28T20:48:05.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.911+0000 7fae8e067640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fae8815d550 0x7fae880cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.911+0000 7fae8e067640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fae880d0100 0x7fae880d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.911+0000 7fae8e067640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae880d2230 con 0x7fae880d1ab0 2023-11-28T20:48:05.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.911+0000 7fae8e067640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae880d0590 con 0x7fae880d0100 2023-11-28T20:48:05.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.911+0000 7fae8e067640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae880d2510 con 0x7fae8815d550 2023-11-28T20:48:05.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.912+0000 7fae8c864640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fae880d0100 0x7fae880d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.912+0000 7fae8c864640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fae880d0100 0x7fae880d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43344/0 (socket says 172.21.15.45:43344) 2023-11-28T20:48:05.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.912+0000 7fae8c864640 1 -- 172.21.15.45:0/4138230957 learned_addr learned my addr 172.21.15.45:0/4138230957 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:05.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.912+0000 7fae8d065640 1 --2- 172.21.15.45:0/4138230957 >> v2:172.21.15.190:3300/0 conn(0x7fae8815d550 0x7fae880cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.912+0000 7fae8c864640 1 -- 172.21.15.45:0/4138230957 >> v2:172.21.15.190:3300/0 conn(0x7fae8815d550 msgr2=0x7fae880cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:05.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.912+0000 7fae8c864640 1 --2- 172.21.15.45:0/4138230957 >> v2:172.21.15.190:3300/0 conn(0x7fae8815d550 0x7fae880cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.912+0000 7fae8c864640 1 -- 172.21.15.45:0/4138230957 >> v1:172.21.15.190:6789/0 conn(0x7fae880d1ab0 legacy=0x7fae88070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:05.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.912+0000 7fae8c864640 1 -- 172.21.15.45:0/4138230957 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae880d27f0 con 0x7fae880d0100 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.913+0000 7fae8c864640 1 --2- 172.21.15.45:0/4138230957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d0100 0x7fae880d1570 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fae880b5dc0 tx=0x7fae7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877f53cbd2b1b1da server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.914+0000 7fae7ffff640 1 -- 172.21.15.45:0/4138230957 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78031870 con 0x7fae880d0100 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.914+0000 7fae7ffff640 1 -- 172.21.15.45:0/4138230957 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae780319d0 con 0x7fae880d0100 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.914+0000 7fae7ffff640 1 -- 172.21.15.45:0/4138230957 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78031d60 con 0x7fae880d0100 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.914+0000 7fae8e067640 1 -- 172.21.15.45:0/4138230957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d0100 msgr2=0x7fae880d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.914+0000 7fae8e067640 1 --2- 172.21.15.45:0/4138230957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d0100 0x7fae880d1570 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fae880b5dc0 tx=0x7fae7802b910 comp rx=0 tx=0).stop 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.914+0000 7fae8e067640 1 -- 172.21.15.45:0/4138230957 shutdown_connections 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.914+0000 7fae8e067640 1 --2- 172.21.15.45:0/4138230957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d0100 0x7fae880d1570 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.914+0000 7fae8e067640 1 --2- 172.21.15.45:0/4138230957 >> v2:172.21.15.190:3300/0 conn(0x7fae8815d550 0x7fae880cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.914+0000 7fae8e067640 1 -- 172.21.15.45:0/4138230957 >> 172.21.15.45:0/4138230957 conn(0x7fae8806ac70 msgr2=0x7fae880b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:05.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.915+0000 7fae8e067640 1 -- 172.21.15.45:0/4138230957 shutdown_connections 2023-11-28T20:48:05.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.915+0000 7fae8e067640 1 -- 172.21.15.45:0/4138230957 wait complete. 2023-11-28T20:48:05.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.916+0000 7fae8e067640 1 Processor -- start 2023-11-28T20:48:05.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.916+0000 7fae8e067640 1 -- start start 2023-11-28T20:48:05.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.917+0000 7fae8e067640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae880d0100 0x7fae88159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.917+0000 7fae8e067640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d1ab0 0x7fae881f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.917+0000 7fae8e067640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae8815d550 0x7fae8815a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.917+0000 7fae8e067640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae881f4050 con 0x7fae880d1ab0 2023-11-28T20:48:05.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.917+0000 7fae8e067640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae88159da0 con 0x7fae8815d550 2023-11-28T20:48:05.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.917+0000 7fae8e067640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae881f3a90 con 0x7fae880d0100 2023-11-28T20:48:05.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8c864640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d1ab0 0x7fae881f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8c864640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d1ab0 0x7fae881f2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43350/0 (socket says 172.21.15.45:43350) 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8c864640 1 -- 172.21.15.45:0/2726847139 learned_addr learned my addr 172.21.15.45:0/2726847139 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8c864640 1 -- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae880d0100 msgr2=0x7fae88159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8d065640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae880d0100 0x7fae88159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8d866640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae8815d550 0x7fae8815a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8c864640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae880d0100 0x7fae88159860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8c864640 1 -- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae8815d550 msgr2=0x7fae8815a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8c864640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae8815d550 0x7fae8815a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8c864640 1 -- 172.21.15.45:0/2726847139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae780094d0 con 0x7fae880d1ab0 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.918+0000 7fae8c864640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d1ab0 0x7fae881f2fa0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fae78000c00 tx=0x7fae780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.919+0000 7fae7dffb640 1 -- 172.21.15.45:0/2726847139 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7803abe0 con 0x7fae880d1ab0 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.919+0000 7fae7dffb640 1 -- 172.21.15.45:0/2726847139 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7803ad40 con 0x7fae880d1ab0 2023-11-28T20:48:05.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.919+0000 7fae7dffb640 1 -- 172.21.15.45:0/2726847139 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78044610 con 0x7fae880d1ab0 2023-11-28T20:48:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.919+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae881f3d70 con 0x7fae880d1ab0 2023-11-28T20:48:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.919+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae880cbc40 con 0x7fae880d1ab0 2023-11-28T20:48:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.920+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae880cb7c0 con 0x7fae880d1ab0 2023-11-28T20:48:05.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.921+0000 7fae7dffb640 1 -- 172.21.15.45:0/2726847139 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fae7804d450 con 0x7fae880d1ab0 2023-11-28T20:48:05.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.922+0000 7fae7dffb640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fae6c06af60 0x7fae6c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:05.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.922+0000 7fae7dffb640 1 -- 172.21.15.45:0/2726847139 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fae7802d040 con 0x7fae880d1ab0 2023-11-28T20:48:05.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.923+0000 7fae8d065640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fae6c06af60 0x7fae6c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:05.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.924+0000 7fae8d065640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fae6c06af60 0x7fae6c06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fae84009e90 tx=0x7fae8400b040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:05.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:05.926+0000 7fae7dffb640 1 -- 172.21.15.45:0/2726847139 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae880cb7c0 con 0x7fae880d1ab0 2023-11-28T20:48:06.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.193+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fae880cb7c0 con 0x7fae880d1ab0 2023-11-28T20:48:06.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.729+0000 7fae7dffb640 1 -- 172.21.15.45:0/2726847139 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v520) v1 ==== 157+0+0 (secure 0 0 0) 0x7fae880cb7c0 con 0x7fae880d1ab0 2023-11-28T20:48:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fae6c06af60 msgr2=0x7fae6c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fae6c06af60 0x7fae6c06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fae84009e90 tx=0x7fae8400b040 comp rx=0 tx=0).stop 2023-11-28T20:48:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d1ab0 msgr2=0x7fae881f2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:06.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d1ab0 0x7fae881f2fa0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fae78000c00 tx=0x7fae780046c0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 shutdown_connections 2023-11-28T20:48:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fae6c06af60 0x7fae6c06d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae8815d550 0x7fae8815a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae880d1ab0 0x7fae881f2fa0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 --2- 172.21.15.45:0/2726847139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae880d0100 0x7fae88159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 >> 172.21.15.45:0/2726847139 conn(0x7fae8806ac70 msgr2=0x7fae8815e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 shutdown_connections 2023-11-28T20:48:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.733+0000 7fae8e067640 1 -- 172.21.15.45:0/2726847139 wait complete. 2023-11-28T20:48:06.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T20:48:06.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.982+0000 7fb57e3a9640 1 Processor -- start 2023-11-28T20:48:06.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.982+0000 7fb57e3a9640 1 -- start start 2023-11-28T20:48:06.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.983+0000 7fb57e3a9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb57815b3c0 0x7fb578099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:06.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.983+0000 7fb57e3a9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780d1b40 0x7fb578099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:06.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.983+0000 7fb57e3a9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5780c4f10 con 0x7fb5780c53a0 2023-11-28T20:48:06.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.983+0000 7fb57e3a9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb57815b7a0 con 0x7fb57815b3c0 2023-11-28T20:48:06.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.983+0000 7fb57e3a9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780c4c10 con 0x7fb5780d1b40 2023-11-28T20:48:06.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.984+0000 7fb577fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb57815b3c0 0x7fb578099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:06.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.984+0000 7fb5777fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780d1b40 0x7fb578099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:06.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.984+0000 7fb5777fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780d1b40 0x7fb578099f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:36560/0 (socket says 172.21.15.45:36560) 2023-11-28T20:48:06.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.984+0000 7fb577fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb57815b3c0 0x7fb578099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:39614/0 (socket says 172.21.15.45:39614) 2023-11-28T20:48:06.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.984+0000 7fb5777fe640 1 -- 172.21.15.45:0/554927555 learned_addr learned my addr 172.21.15.45:0/554927555 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:06.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.985+0000 7fb5777fe640 1 -- 172.21.15.45:0/554927555 >> v2:172.21.15.190:3300/0 conn(0x7fb57815b3c0 msgr2=0x7fb578099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:06.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.985+0000 7fb5777fe640 1 --2- 172.21.15.45:0/554927555 >> v2:172.21.15.190:3300/0 conn(0x7fb57815b3c0 0x7fb578099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.985+0000 7fb5777fe640 1 -- 172.21.15.45:0/554927555 >> v1:172.21.15.190:6789/0 conn(0x7fb5780c53a0 legacy=0x7fb5780c5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:06.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.985+0000 7fb5777fe640 1 -- 172.21.15.45:0/554927555 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5780cf600 con 0x7fb5780d1b40 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.986+0000 7fb5777fe640 1 --2- 172.21.15.45:0/554927555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780d1b40 0x7fb578099f80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb560009870 tx=0x7fb56002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5777ffe030a138b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.987+0000 7fb576ffd640 1 -- 172.21.15.45:0/554927555 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560030840 con 0x7fb5780d1b40 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.987+0000 7fb576ffd640 1 -- 172.21.15.45:0/554927555 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5600309a0 con 0x7fb5780d1b40 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.987+0000 7fb576ffd640 1 -- 172.21.15.45:0/554927555 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560030d30 con 0x7fb5780d1b40 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.988+0000 7fb57e3a9640 1 -- 172.21.15.45:0/554927555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780d1b40 msgr2=0x7fb578099f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.988+0000 7fb57e3a9640 1 --2- 172.21.15.45:0/554927555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780d1b40 0x7fb578099f80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb560009870 tx=0x7fb56002b980 comp rx=0 tx=0).stop 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.988+0000 7fb57e3a9640 1 -- 172.21.15.45:0/554927555 shutdown_connections 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.988+0000 7fb57e3a9640 1 --2- 172.21.15.45:0/554927555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780d1b40 0x7fb578099f80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.988+0000 7fb57e3a9640 1 --2- 172.21.15.45:0/554927555 >> v2:172.21.15.190:3300/0 conn(0x7fb57815b3c0 0x7fb578099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.988+0000 7fb57e3a9640 1 -- 172.21.15.45:0/554927555 >> 172.21.15.45:0/554927555 conn(0x7fb5780d1020 msgr2=0x7fb5780d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:06.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.988+0000 7fb57e3a9640 1 -- 172.21.15.45:0/554927555 shutdown_connections 2023-11-28T20:48:06.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.988+0000 7fb57e3a9640 1 -- 172.21.15.45:0/554927555 wait complete. 2023-11-28T20:48:06.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.990+0000 7fb57e3a9640 1 Processor -- start 2023-11-28T20:48:06.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.990+0000 7fb57e3a9640 1 -- start start 2023-11-28T20:48:06.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.991+0000 7fb57e3a9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780c53a0 0x7fb57814c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.991+0000 7fb57e3a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5780d1b40 0x7fb5780b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.991+0000 7fb57e3a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb57815b3c0 0x7fb5780c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.991+0000 7fb57e3a9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780b5a10 con 0x7fb57815b3c0 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.991+0000 7fb57e3a9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57809b190 con 0x7fb5780d1b40 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.991+0000 7fb57e3a9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780b5c60 con 0x7fb5780c53a0 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb57c946640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb57815b3c0 0x7fb5780c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb57c946640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb57815b3c0 0x7fb5780c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43362/0 (socket says 172.21.15.45:43362) 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb57c946640 1 -- 172.21.15.45:0/408513942 learned_addr learned my addr 172.21.15.45:0/408513942 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb577fff640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780c53a0 0x7fb57814c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:06.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb5777fe640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5780d1b40 0x7fb5780b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb57c946640 1 -- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780c53a0 msgr2=0x7fb57814c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb57c946640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780c53a0 0x7fb57814c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb57c946640 1 -- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5780d1b40 msgr2=0x7fb5780b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb57c946640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5780d1b40 0x7fb5780b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb57c946640 1 -- 172.21.15.45:0/408513942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5600094d0 con 0x7fb57815b3c0 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.992+0000 7fb57c946640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb57815b3c0 0x7fb5780c96f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb57000c390 tx=0x7fb57000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.993+0000 7fb574ff9640 1 -- 172.21.15.45:0/408513942 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700052d0 con 0x7fb57815b3c0 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.993+0000 7fb574ff9640 1 -- 172.21.15.45:0/408513942 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb570005430 con 0x7fb57815b3c0 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.993+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5780b5f40 con 0x7fb57815b3c0 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.993+0000 7fb574ff9640 1 -- 172.21.15.45:0/408513942 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570003810 con 0x7fb57815b3c0 2023-11-28T20:48:06.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.993+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5780b6500 con 0x7fb57815b3c0 2023-11-28T20:48:06.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.994+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5780c7f40 con 0x7fb57815b3c0 2023-11-28T20:48:06.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.995+0000 7fb574ff9640 1 -- 172.21.15.45:0/408513942 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb570003970 con 0x7fb57815b3c0 2023-11-28T20:48:06.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.996+0000 7fb574ff9640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb55806af60 0x7fb55806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:06.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.996+0000 7fb574ff9640 1 -- 172.21.15.45:0/408513942 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fb570088eb0 con 0x7fb57815b3c0 2023-11-28T20:48:06.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.997+0000 7fb577fff640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb55806af60 0x7fb55806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:07.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:06.998+0000 7fb577fff640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb55806af60 0x7fb55806d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5680074a0 tx=0x7fb56800b040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:07.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.001+0000 7fb574ff9640 1 -- 172.21.15.45:0/408513942 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb570053c50 con 0x7fb57815b3c0 2023-11-28T20:48:07.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.196+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb5780733d0 con 0x7fb57815b3c0 2023-11-28T20:48:07.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.204+0000 7fb574ff9640 1 -- 172.21.15.45:0/408513942 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb570017020 con 0x7fb57815b3c0 2023-11-28T20:48:07.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb55806af60 msgr2=0x7fb55806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:07.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb55806af60 0x7fb55806d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5680074a0 tx=0x7fb56800b040 comp rx=0 tx=0).stop 2023-11-28T20:48:07.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb57815b3c0 msgr2=0x7fb5780c96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:07.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb57815b3c0 0x7fb5780c96f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb57000c390 tx=0x7fb57000c850 comp rx=0 tx=0).stop 2023-11-28T20:48:07.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 shutdown_connections 2023-11-28T20:48:07.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb55806af60 0x7fb55806d420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb57815b3c0 0x7fb5780c96f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5780d1b40 0x7fb5780b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 --2- 172.21.15.45:0/408513942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5780c53a0 0x7fb57814c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 >> 172.21.15.45:0/408513942 conn(0x7fb5780d1020 msgr2=0x7fb5780cdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:07.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 shutdown_connections 2023-11-28T20:48:07.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.207+0000 7fb57e3a9640 1 -- 172.21.15.45:0/408513942 wait complete. 2023-11-28T20:48:07.227 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:48:07.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:48:07.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:48:07.468 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:48:07.468 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:48:07.469 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:48:07.535 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:48:07.535 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:48:07.535 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:48:07.594 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:48:07.595 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:48:07.595 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:48:07.660 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:48:07.660 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:48:07.660 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:48:07.720 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:48:07.720 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T20:48:07.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:48:07.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.952+0000 7fad1e33c640 1 Processor -- start 2023-11-28T20:48:07.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.952+0000 7fad1e33c640 1 -- start start 2023-11-28T20:48:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.954+0000 7fad1e33c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad1815b850 0x7fad180b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.954+0000 7fad1e33c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fad180b1fa0 0x7fad180b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.954+0000 7fad1e33c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1815bda0 con 0x7fad180b35b0 2023-11-28T20:48:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.954+0000 7fad1e33c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1815ee40 con 0x7fad1815b850 2023-11-28T20:48:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.954+0000 7fad1e33c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1815c080 con 0x7fad180b1fa0 2023-11-28T20:48:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.954+0000 7fad177fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad1815b850 0x7fad180b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.954+0000 7fad17fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fad180b35b0 0x7fad1815ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39672/0 (socket says 172.21.15.45:39672) 2023-11-28T20:48:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.954+0000 7fad177fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad1815b850 0x7fad180b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43372/0 (socket says 172.21.15.45:43372) 2023-11-28T20:48:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.954+0000 7fad17fff640 1 -- 172.21.15.45:0/2229871154 learned_addr learned my addr 172.21.15.45:0/2229871154 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.955+0000 7fad1c8d9640 1 --2- 172.21.15.45:0/2229871154 >> v2:172.21.15.190:3300/0 conn(0x7fad180b1fa0 0x7fad180b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.955+0000 7fad177fe640 1 -- 172.21.15.45:0/2229871154 >> v2:172.21.15.190:3300/0 conn(0x7fad180b1fa0 msgr2=0x7fad180b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:07.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.955+0000 7fad177fe640 1 --2- 172.21.15.45:0/2229871154 >> v2:172.21.15.190:3300/0 conn(0x7fad180b1fa0 0x7fad180b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.955+0000 7fad177fe640 1 -- 172.21.15.45:0/2229871154 >> v1:172.21.15.45:6789/0 conn(0x7fad180b35b0 legacy=0x7fad1815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:07.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.955+0000 7fad177fe640 1 -- 172.21.15.45:0/2229871154 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1815c360 con 0x7fad1815b850 2023-11-28T20:48:07.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.955+0000 7fad16ffd640 1 -- 172.21.15.45:0/2229871154 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176558619 0 0) 0x7fad1815bda0 con 0x7fad180b35b0 2023-11-28T20:48:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.956+0000 7fad177fe640 1 --2- 172.21.15.45:0/2229871154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad1815b850 0x7fad180b0c30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fad1806e660 tx=0x7fad0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b632108048ed4eb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.956+0000 7fad16ffd640 1 -- 172.21.15.45:0/2229871154 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c031830 con 0x7fad1815b850 2023-11-28T20:48:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.956+0000 7fad16ffd640 1 -- 172.21.15.45:0/2229871154 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c031990 con 0x7fad1815b850 2023-11-28T20:48:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.956+0000 7fad16ffd640 1 -- 172.21.15.45:0/2229871154 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c031d20 con 0x7fad1815b850 2023-11-28T20:48:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.957+0000 7fad1e33c640 1 -- 172.21.15.45:0/2229871154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad1815b850 msgr2=0x7fad180b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.957+0000 7fad1e33c640 1 --2- 172.21.15.45:0/2229871154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad1815b850 0x7fad180b0c30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fad1806e660 tx=0x7fad0c02b8d0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.957+0000 7fad1e33c640 1 -- 172.21.15.45:0/2229871154 shutdown_connections 2023-11-28T20:48:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.957+0000 7fad1e33c640 1 --2- 172.21.15.45:0/2229871154 >> v2:172.21.15.190:3300/0 conn(0x7fad180b1fa0 0x7fad180b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.957+0000 7fad1e33c640 1 --2- 172.21.15.45:0/2229871154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad1815b850 0x7fad180b0c30 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.957+0000 7fad1e33c640 1 -- 172.21.15.45:0/2229871154 >> 172.21.15.45:0/2229871154 conn(0x7fad1806a500 msgr2=0x7fad1806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.957+0000 7fad1e33c640 1 -- 172.21.15.45:0/2229871154 shutdown_connections 2023-11-28T20:48:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.957+0000 7fad1e33c640 1 -- 172.21.15.45:0/2229871154 wait complete. 2023-11-28T20:48:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.959+0000 7fad1e33c640 1 Processor -- start 2023-11-28T20:48:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.959+0000 7fad1e33c640 1 -- start start 2023-11-28T20:48:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.960+0000 7fad1e33c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad180b1fa0 0x7fad181f70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.960+0000 7fad1e33c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad180b35b0 0x7fad18071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.960+0000 7fad1e33c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1815b850 0x7fad181f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.960+0000 7fad1e33c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1813f560 con 0x7fad180b1fa0 2023-11-28T20:48:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.960+0000 7fad1e33c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad18072200 con 0x7fad1815b850 2023-11-28T20:48:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.960+0000 7fad1e33c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1813f740 con 0x7fad180b35b0 2023-11-28T20:48:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad17fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad180b1fa0 0x7fad181f70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad17fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad180b1fa0 0x7fad181f70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43376/0 (socket says 172.21.15.45:43376) 2023-11-28T20:48:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad17fff640 1 -- 172.21.15.45:0/936584795 learned_addr learned my addr 172.21.15.45:0/936584795 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad1c8d9640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1815b850 0x7fad181f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad177fe640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad180b35b0 0x7fad18071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad17fff640 1 -- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad180b35b0 msgr2=0x7fad18071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad17fff640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad180b35b0 0x7fad18071cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad17fff640 1 -- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1815b850 msgr2=0x7fad181f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad17fff640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1815b850 0x7fad181f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.961+0000 7fad17fff640 1 -- 172.21.15.45:0/936584795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c009400 con 0x7fad180b1fa0 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.962+0000 7fad17fff640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad180b1fa0 0x7fad181f70e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fad0800c4d0 tx=0x7fad0800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.962+0000 7fad14ff9640 1 -- 172.21.15.45:0/936584795 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad080074e0 con 0x7fad180b1fa0 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.962+0000 7fad14ff9640 1 -- 172.21.15.45:0/936584795 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad08007660 con 0x7fad180b1fa0 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.962+0000 7fad14ff9640 1 -- 172.21.15.45:0/936584795 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08004a20 con 0x7fad180b1fa0 2023-11-28T20:48:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.962+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1813fa20 con 0x7fad180b1fa0 2023-11-28T20:48:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.962+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1813ffe0 con 0x7fad180b1fa0 2023-11-28T20:48:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.963+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad180b5290 con 0x7fad180b1fa0 2023-11-28T20:48:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.964+0000 7fad14ff9640 1 -- 172.21.15.45:0/936584795 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fad08004b80 con 0x7fad180b1fa0 2023-11-28T20:48:07.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.965+0000 7fad14ff9640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fad0406af60 0x7fad0406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:07.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.965+0000 7fad14ff9640 1 -- 172.21.15.45:0/936584795 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fad08088a10 con 0x7fad180b1fa0 2023-11-28T20:48:07.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.966+0000 7fad177fe640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fad0406af60 0x7fad0406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:07.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.967+0000 7fad177fe640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fad0406af60 0x7fad0406d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad0c036000 tx=0x7fad0c03c040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:07.969+0000 7fad14ff9640 1 -- 172.21.15.45:0/936584795 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad080538d0 con 0x7fad180b1fa0 2023-11-28T20:48:08.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.204+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad180733d0 con 0x7fad180b1fa0 2023-11-28T20:48:08.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.206+0000 7fad14ff9640 1 -- 172.21.15.45:0/936584795 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fad08018020 con 0x7fad180b1fa0 2023-11-28T20:48:08.208 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:08.208 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3050590116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3050590116},{"type":"v1","addr":"172.21.15.45:6835","nonce":3050590116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2465910573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2465910573},{"type":"v1","addr":"172.21.15.190:6836","nonce":2465910573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/4090116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":4090116745},{"type":"v1","addr":"172.21.15.190:6837","nonce":4090116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/3790403612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3790403612},{"type":"v1","addr":"172.21.15.45:6834","nonce":3790403612}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2023-11-28T20:48:08.208 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 20 2023-11-28T20:48:08.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fad0406af60 msgr2=0x7fad0406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:08.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fad0406af60 0x7fad0406d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad0c036000 tx=0x7fad0c03c040 comp rx=0 tx=0).stop 2023-11-28T20:48:08.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad180b1fa0 msgr2=0x7fad181f70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:08.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad180b1fa0 0x7fad181f70e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fad0800c4d0 tx=0x7fad0800c990 comp rx=0 tx=0).stop 2023-11-28T20:48:08.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 shutdown_connections 2023-11-28T20:48:08.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fad0406af60 0x7fad0406d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1815b850 0x7fad181f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad180b35b0 0x7fad18071cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 --2- 172.21.15.45:0/936584795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad180b1fa0 0x7fad181f70e0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.208+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 >> 172.21.15.45:0/936584795 conn(0x7fad1806a500 msgr2=0x7fad181433f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.209+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 shutdown_connections 2023-11-28T20:48:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.209+0000 7fad1e33c640 1 -- 172.21.15.45:0/936584795 wait complete. 2023-11-28T20:48:08.230 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T20:48:08.230 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T20:48:08.230 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T20:48:08.230 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:48:08.230 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T20:48:08.230 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T20:48:08.231 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T20:48:08.231 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:48:08.231 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T20:48:08.231 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T20:48:08.231 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T20:48:08.231 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:48:08.232 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T20:48:08.232 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T20:48:08.232 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T20:48:08.232 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:48:08.247 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T20:48:08.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T20:48:08.248 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T20:48:08.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T20:48:08.263 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T20:48:08.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T20:48:08.265 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T20:48:08.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T20:48:08.296 INFO:tasks.ceph.mds.a:Started 2023-11-28T20:48:08.296 DEBUG:teuthology.parallel:result is None 2023-11-28T20:48:08.303 INFO:tasks.ceph.mds.c:Started 2023-11-28T20:48:08.304 DEBUG:teuthology.parallel:result is None 2023-11-28T20:48:08.313 INFO:tasks.ceph.mds.d:Started 2023-11-28T20:48:08.313 DEBUG:teuthology.parallel:result is None 2023-11-28T20:48:08.321 INFO:tasks.ceph.mds.b:Started 2023-11-28T20:48:08.321 DEBUG:teuthology.parallel:result is None 2023-11-28T20:48:08.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T20:48:08.463 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:48:08.463 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:48:08.483 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T20:48:08.483 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:48:08.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.516+0000 7f9327417640 1 Processor -- start 2023-11-28T20:48:08.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.516+0000 7f9327417640 1 -- start start 2023-11-28T20:48:08.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.517+0000 7f9327417640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9320070b20 0x7f9320070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:08.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.517+0000 7f9327417640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93200b6670 0x7f93200b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:08.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.517+0000 7f9327417640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932006b580 con 0x7f932015d730 2023-11-28T20:48:08.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f9327417640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93200714f0 con 0x7f93200b6670 2023-11-28T20:48:08.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f9327417640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932006b2a0 con 0x7f9320070b20 2023-11-28T20:48:08.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f93251b3640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f932015d730 0x7f93200d2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39698/0 (socket says 172.21.15.45:39698) 2023-11-28T20:48:08.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f93251b3640 1 -- 172.21.15.45:0/3372845372 learned_addr learned my addr 172.21.15.45:0/3372845372 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:08.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f93259b4640 1 --2- 172.21.15.45:0/3372845372 >> v2:172.21.15.45:3300/0 conn(0x7f93200b6670 0x7f93200b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:08.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f93249b2640 1 --2- 172.21.15.45:0/3372845372 >> v2:172.21.15.190:3300/0 conn(0x7f9320070b20 0x7f9320070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:08.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f9317fff640 1 -- 172.21.15.45:0/3372845372 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216057657 0 0) 0x7f932006b580 con 0x7f932015d730 2023-11-28T20:48:08.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f93259b4640 1 -- 172.21.15.45:0/3372845372 >> v2:172.21.15.190:3300/0 conn(0x7f9320070b20 msgr2=0x7f9320070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:08.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f93259b4640 1 --2- 172.21.15.45:0/3372845372 >> v2:172.21.15.190:3300/0 conn(0x7f9320070b20 0x7f9320070fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f93259b4640 1 -- 172.21.15.45:0/3372845372 >> v1:172.21.15.45:6789/0 conn(0x7f932015d730 legacy=0x7f93200d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:08.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.518+0000 7f93259b4640 1 -- 172.21.15.45:0/3372845372 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93200d1730 con 0x7f93200b6670 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.519+0000 7f93259b4640 1 --2- 172.21.15.45:0/3372845372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93200b6670 0x7f93200b6a80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f932008c460 tx=0x7f931802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549aac0f9ac3085c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.520+0000 7f9317fff640 1 -- 172.21.15.45:0/3372845372 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931802cc80 con 0x7f93200b6670 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.520+0000 7f9317fff640 1 -- 172.21.15.45:0/3372845372 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931802cde0 con 0x7f93200b6670 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.520+0000 7f9317fff640 1 -- 172.21.15.45:0/3372845372 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318034660 con 0x7f93200b6670 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.520+0000 7f9327417640 1 -- 172.21.15.45:0/3372845372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93200b6670 msgr2=0x7f93200b6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.520+0000 7f9327417640 1 --2- 172.21.15.45:0/3372845372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93200b6670 0x7f93200b6a80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f932008c460 tx=0x7f931802bed0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.520+0000 7f9327417640 1 -- 172.21.15.45:0/3372845372 shutdown_connections 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.520+0000 7f9327417640 1 --2- 172.21.15.45:0/3372845372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93200b6670 0x7f93200b6a80 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.520+0000 7f9327417640 1 --2- 172.21.15.45:0/3372845372 >> v2:172.21.15.190:3300/0 conn(0x7f9320070b20 0x7f9320070fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.520+0000 7f9327417640 1 -- 172.21.15.45:0/3372845372 >> 172.21.15.45:0/3372845372 conn(0x7f93200cfc70 msgr2=0x7f93200d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:08.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.521+0000 7f9327417640 1 -- 172.21.15.45:0/3372845372 shutdown_connections 2023-11-28T20:48:08.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.521+0000 7f9327417640 1 -- 172.21.15.45:0/3372845372 wait complete. 2023-11-28T20:48:08.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.522+0000 7f9327417640 1 Processor -- start 2023-11-28T20:48:08.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.522+0000 7f9327417640 1 -- start start 2023-11-28T20:48:08.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.523+0000 7f9327417640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320070b20 0x7f93201f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.523+0000 7f9327417640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93200b6670 0x7f93200acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.523+0000 7f9327417640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f932015d730 0x7f93200ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.523+0000 7f9327417640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932015afb0 con 0x7f932015d730 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.523+0000 7f9327417640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93201f3be0 con 0x7f9320070b20 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.523+0000 7f9327417640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93201f3ec0 con 0x7f93200b6670 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93259b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f932015d730 0x7f93200ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93249b2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93200b6670 0x7f93200acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93259b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f932015d730 0x7f93200ad1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43408/0 (socket says 172.21.15.45:43408) 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93259b4640 1 -- 172.21.15.45:0/1626226531 learned_addr learned my addr 172.21.15.45:0/1626226531 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:08.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93251b3640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320070b20 0x7f93201f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:08.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93259b4640 1 -- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93200b6670 msgr2=0x7f93200acc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93259b4640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93200b6670 0x7f93200acc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93259b4640 1 -- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320070b20 msgr2=0x7f93201f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93259b4640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320070b20 0x7f93201f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93259b4640 1 -- 172.21.15.45:0/1626226531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9318009a70 con 0x7f932015d730 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.524+0000 7f93259b4640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f932015d730 0x7f93200ad1a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f931802c4e0 tx=0x7f9318004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.525+0000 7f9315ffb640 1 -- 172.21.15.45:0/1626226531 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931803e610 con 0x7f932015d730 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.525+0000 7f9315ffb640 1 -- 172.21.15.45:0/1626226531 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931803e770 con 0x7f932015d730 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.525+0000 7f9315ffb640 1 -- 172.21.15.45:0/1626226531 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931803eb20 con 0x7f932015d730 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.525+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93201f4460 con 0x7f932015d730 2023-11-28T20:48:08.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.525+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9320160530 con 0x7f932015d730 2023-11-28T20:48:08.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.526+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93200cf830 con 0x7f932015d730 2023-11-28T20:48:08.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.527+0000 7f9315ffb640 1 -- 172.21.15.45:0/1626226531 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f931803b050 con 0x7f932015d730 2023-11-28T20:48:08.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.528+0000 7f9315ffb640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f930406af90 0x7f930406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:08.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.529+0000 7f9315ffb640 1 -- 172.21.15.45:0/1626226531 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f93180abe90 con 0x7f932015d730 2023-11-28T20:48:08.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.529+0000 7f93251b3640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f930406af90 0x7f930406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:08.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.530+0000 7f93251b3640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f930406af90 0x7f930406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f931c007fd0 tx=0x7f931c007de0 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:08.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.533+0000 7f9315ffb640 1 -- 172.21.15.45:0/1626226531 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9318076f20 con 0x7f932015d730 2023-11-28T20:48:08.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:08.740+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f93201f4980 con 0x7f932015d730 2023-11-28T20:48:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.227+0000 7f9315ffb640 1 -- 172.21.15.45:0/1626226531 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v65) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9318039080 con 0x7f932015d730 2023-11-28T20:48:09.229 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T20:48:09.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f930406af90 msgr2=0x7f930406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:09.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f930406af90 0x7f930406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f931c007fd0 tx=0x7f931c007de0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f932015d730 msgr2=0x7f93200ad1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:09.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f932015d730 0x7f93200ad1a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f931802c4e0 tx=0x7f9318004a40 comp rx=0 tx=0).stop 2023-11-28T20:48:09.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 shutdown_connections 2023-11-28T20:48:09.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f930406af90 0x7f930406d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f932015d730 0x7f93200ad1a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93200b6670 0x7f93200acc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 --2- 172.21.15.45:0/1626226531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320070b20 0x7f93201f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.233+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 >> 172.21.15.45:0/1626226531 conn(0x7f93200cfc70 msgr2=0x7f93200d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:09.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.234+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 shutdown_connections 2023-11-28T20:48:09.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.234+0000 7f9327417640 1 -- 172.21.15.45:0/1626226531 wait complete. 2023-11-28T20:48:09.263 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T20:48:09.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.478+0000 7f042dc8a640 1 Processor -- start 2023-11-28T20:48:09.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.479+0000 7f042dc8a640 1 -- start start 2023-11-28T20:48:09.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.480+0000 7f042dc8a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280cd0a0 0x7f0428167e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:09.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.480+0000 7f042dc8a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04280b0130 con 0x7f042815d080 2023-11-28T20:48:09.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.480+0000 7f042dc8a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04280cbc00 con 0x7f042815d460 2023-11-28T20:48:09.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.480+0000 7f042dc8a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042806b530 con 0x7f04280cd0a0 2023-11-28T20:48:09.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.480+0000 7f042cc88640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f042815d080 0x7f042806a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39700/0 (socket says 172.21.15.45:39700) 2023-11-28T20:48:09.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.480+0000 7f042cc88640 1 -- 172.21.15.45:0/53117517 learned_addr learned my addr 172.21.15.45:0/53117517 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118533056 0 0) 0x7f04280b0130 con 0x7f042815d080 2023-11-28T20:48:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0418003520 con 0x7f042815d080 2023-11-28T20:48:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f042d489640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280cd0a0 0x7f0428167e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 772962457 0 0) 0x7f0418003520 con 0x7f042815d080 2023-11-28T20:48:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280cd0a0 msgr2=0x7f0428167e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280cd0a0 0x7f0428167e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 >> v1:172.21.15.190:6789/0 conn(0x7f042815d460 legacy=0x7f042806ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04280b1a00 con 0x7f042815d080 2023-11-28T20:48:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f041c001690 con 0x7f042815d080 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 >> v1:172.21.15.45:6789/0 conn(0x7f042815d080 legacy=0x7f042806a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280cd0a0 0x7f04180039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0418003fd0 0x7f04180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0418005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042806b530 con 0x7f0418004d00 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041800bb40 con 0x7f0418003fd0 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.481+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041800bd00 con 0x7f04280cd0a0 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.482+0000 7f042d489640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0418005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.482+0000 7f042cc88640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280cd0a0 0x7f04180039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.482+0000 7f0427fff640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0418003fd0 0x7f04180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.482+0000 7f042d489640 1 -- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280cd0a0 msgr2=0x7f04180039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.482+0000 7f042d489640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280cd0a0 0x7f04180039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.482+0000 7f042d489640 1 -- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0418003fd0 msgr2=0x7f04180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:09.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.482+0000 7f042d489640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0418003fd0 0x7f04180047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.482+0000 7f042d489640 1 -- 172.21.15.45:0/53117517 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0418003520 con 0x7f0418004d00 2023-11-28T20:48:09.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.482+0000 7f0427fff640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0418003fd0 0x7f04180047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:09.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.483+0000 7f042d489640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0418005180 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0420007360 tx=0x7f04180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bddd94c54530e189 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:09.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.483+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042002ccd0 con 0x7f0418004d00 2023-11-28T20:48:09.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.484+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042002ce30 con 0x7f0418004d00 2023-11-28T20:48:09.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.484+0000 7f04277fe640 1 -- 172.21.15.45:0/53117517 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420034660 con 0x7f0418004d00 2023-11-28T20:48:09.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.484+0000 7f042dc8a640 1 -- 172.21.15.45:0/53117517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 msgr2=0x7f0418005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:09.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.484+0000 7f042dc8a640 1 --2- 172.21.15.45:0/53117517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0418005180 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0420007360 tx=0x7f04180057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.484+0000 7f042d489640 1 -- 172.21.15.45:0/53117517 reap_dead start 2023-11-28T20:48:09.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.484+0000 7f042dc8a640 1 -- 172.21.15.45:0/53117517 shutdown_connections 2023-11-28T20:48:09.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.484+0000 7f042dc8a640 1 -- 172.21.15.45:0/53117517 >> 172.21.15.45:0/53117517 conn(0x7f0428072850 msgr2=0x7f0428099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:09.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.484+0000 7f042dc8a640 1 -- 172.21.15.45:0/53117517 shutdown_connections 2023-11-28T20:48:09.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.484+0000 7f042dc8a640 1 -- 172.21.15.45:0/53117517 wait complete. 2023-11-28T20:48:09.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.486+0000 7f042dc8a640 1 Processor -- start 2023-11-28T20:48:09.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.486+0000 7f042dc8a640 1 -- start start 2023-11-28T20:48:09.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.487+0000 7f042dc8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0428156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:09.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.487+0000 7f042dc8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0428157500 0x7f042815d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:09.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.487+0000 7f042dc8a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280c52e0 0x7f04280c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:09.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.487+0000 7f042dc8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280c8250 con 0x7f0418004d00 2023-11-28T20:48:09.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.487+0000 7f042dc8a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042815dec0 con 0x7f0428157500 2023-11-28T20:48:09.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.487+0000 7f042dc8a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04280c5b30 con 0x7f04280c52e0 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.487+0000 7f042cc88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0428156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.487+0000 7f042cc88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0428156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43424/0 (socket says 172.21.15.45:43424) 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.487+0000 7f042cc88640 1 -- 172.21.15.45:0/2079860776 learned_addr learned my addr 172.21.15.45:0/2079860776 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.488+0000 7f042cc88640 1 -- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280c52e0 msgr2=0x7f04280c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.488+0000 7f042d489640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280c52e0 0x7f04280c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.488+0000 7f0427fff640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0428157500 0x7f042815d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.488+0000 7f042cc88640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280c52e0 0x7f04280c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.488+0000 7f042cc88640 1 -- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0428157500 msgr2=0x7f042815d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.488+0000 7f042cc88640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0428157500 0x7f042815d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.488+0000 7f042cc88640 1 -- 172.21.15.45:0/2079860776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04280c5e10 con 0x7f0418004d00 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.488+0000 7f0427fff640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0428157500 0x7f042815d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.488+0000 7f042cc88640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0428156e70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f041c00e870 tx=0x7f041c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:09.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.489+0000 7f04257fa640 1 -- 172.21.15.45:0/2079860776 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c00c8d0 con 0x7f0418004d00 2023-11-28T20:48:09.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.489+0000 7f04257fa640 1 -- 172.21.15.45:0/2079860776 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041c00ca30 con 0x7f0418004d00 2023-11-28T20:48:09.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.489+0000 7f04257fa640 1 -- 172.21.15.45:0/2079860776 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c00cde0 con 0x7f0418004d00 2023-11-28T20:48:09.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.489+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04280c9a50 con 0x7f0418004d00 2023-11-28T20:48:09.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.489+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04280c5760 con 0x7f0418004d00 2023-11-28T20:48:09.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.490+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042809ee90 con 0x7f0418004d00 2023-11-28T20:48:09.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.491+0000 7f04257fa640 1 -- 172.21.15.45:0/2079860776 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f041c01d460 con 0x7f0418004d00 2023-11-28T20:48:09.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.492+0000 7f04257fa640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f040c06af60 0x7f040c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:09.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.492+0000 7f04257fa640 1 -- 172.21.15.45:0/2079860776 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f041c0896c0 con 0x7f0418004d00 2023-11-28T20:48:09.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.492+0000 7f0427fff640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f040c06af60 0x7f040c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:09.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.494+0000 7f0427fff640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f040c06af60 0x7f040c06d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f04280c60f0 tx=0x7f0414009040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:09.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.496+0000 7f04257fa640 1 -- 172.21.15.45:0/2079860776 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f041c0547d0 con 0x7f0418004d00 2023-11-28T20:48:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.758+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0428070180 con 0x7f0418004d00 2023-11-28T20:48:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.759+0000 7f04257fa640 1 -- 172.21.15.45:0/2079860776 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f041c057df0 con 0x7f0418004d00 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:48:09.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.763 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.764 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T20:48:09.769 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.770 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T20:48:09.771 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:48:09.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T20:48:09.774 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T20:48:09.775 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:48:09.776 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:48:09.776 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:48:09.776 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:48:09.776 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.776 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:09.776 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T20:48:09.776 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:48:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f040c06af60 msgr2=0x7f040c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f040c06af60 0x7f040c06d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f04280c60f0 tx=0x7f0414009040 comp rx=0 tx=0).stop 2023-11-28T20:48:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 msgr2=0x7f0428156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0428156e70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f041c00e870 tx=0x7f041c00ed30 comp rx=0 tx=0).stop 2023-11-28T20:48:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 shutdown_connections 2023-11-28T20:48:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f040c06af60 0x7f040c06d420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04280c52e0 0x7f04280c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0428157500 0x7f042815d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 --2- 172.21.15.45:0/2079860776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0418004d00 0x7f0428156e70 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 >> 172.21.15.45:0/2079860776 conn(0x7f0428072850 msgr2=0x7f042806ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.762+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 shutdown_connections 2023-11-28T20:48:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:09.763+0000 7f042dc8a640 1 -- 172.21.15.45:0/2079860776 wait complete. 2023-11-28T20:48:09.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:48:10.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.014+0000 7f3f060d1640 1 Processor -- start 2023-11-28T20:48:10.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.014+0000 7f3f060d1640 1 -- start start 2023-11-28T20:48:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.016+0000 7f3f060d1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f000ce800 0x7f3f00165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.016+0000 7f3f060d1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0015c3c0 con 0x7f3f000ae540 2023-11-28T20:48:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.016+0000 7f3f060d1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f00155b60 con 0x7f3f00155780 2023-11-28T20:48:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.016+0000 7f3f060d1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0015ba90 con 0x7f3f000ce800 2023-11-28T20:48:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.016+0000 7f3f058d0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f000ce800 0x7f3f00165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.016+0000 7f3f058d0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f000ce800 0x7f3f00165e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43438/0 (socket says 172.21.15.45:43438) 2023-11-28T20:48:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.016+0000 7f3f058d0640 1 -- 172.21.15.45:0/970314237 learned_addr learned my addr 172.21.15.45:0/970314237 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.016+0000 7f3f048ce640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3f000ae540 0x7f3f000ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39714/0 (socket says 172.21.15.45:39714) 2023-11-28T20:48:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.017+0000 7f3f058d0640 1 -- 172.21.15.45:0/970314237 >> v1:172.21.15.190:6789/0 conn(0x7f3f00155780 legacy=0x7f3f000abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.017+0000 7f3f058d0640 1 -- 172.21.15.45:0/970314237 >> v1:172.21.15.45:6789/0 conn(0x7f3f000ae540 legacy=0x7f3f000ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.017+0000 7f3f058d0640 1 -- 172.21.15.45:0/970314237 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0015bd70 con 0x7f3f000ce800 2023-11-28T20:48:10.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.017+0000 7f3ef7fff640 1 -- 172.21.15.45:0/970314237 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415182125 0 0) 0x7f3f0015c3c0 con 0x7f3f000ae540 2023-11-28T20:48:10.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.018+0000 7f3f058d0640 1 --2- 172.21.15.45:0/970314237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 0x7f3f00165e90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8009e50 tx=0x7f3ef802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d23efb33015aee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:10.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.018+0000 7f3ef7fff640 1 -- 172.21.15.45:0/970314237 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef802cb20 con 0x7f3f000ce800 2023-11-28T20:48:10.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.018+0000 7f3ef7fff640 1 -- 172.21.15.45:0/970314237 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef802cc80 con 0x7f3f000ce800 2023-11-28T20:48:10.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.018+0000 7f3ef7fff640 1 -- 172.21.15.45:0/970314237 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef80343f0 con 0x7f3f000ce800 2023-11-28T20:48:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.019+0000 7f3f060d1640 1 -- 172.21.15.45:0/970314237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 msgr2=0x7f3f00165e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.019+0000 7f3f060d1640 1 --2- 172.21.15.45:0/970314237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 0x7f3f00165e90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8009e50 tx=0x7f3ef802bf80 comp rx=0 tx=0).stop 2023-11-28T20:48:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.019+0000 7f3f060d1640 1 -- 172.21.15.45:0/970314237 shutdown_connections 2023-11-28T20:48:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.019+0000 7f3f060d1640 1 --2- 172.21.15.45:0/970314237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 0x7f3f00165e90 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.019+0000 7f3f060d1640 1 -- 172.21.15.45:0/970314237 >> 172.21.15.45:0/970314237 conn(0x7f3f0006ab30 msgr2=0x7f3f000b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:10.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.019+0000 7f3f060d1640 1 -- 172.21.15.45:0/970314237 shutdown_connections 2023-11-28T20:48:10.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.019+0000 7f3f060d1640 1 -- 172.21.15.45:0/970314237 wait complete. 2023-11-28T20:48:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.021+0000 7f3f060d1640 1 Processor -- start 2023-11-28T20:48:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.021+0000 7f3f060d1640 1 -- start start 2023-11-28T20:48:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f060d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f000ae540 0x7f3f0014e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f060d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 0x7f3f0014ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f060d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f00155780 0x7f3f0014ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f060d1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00150af0 con 0x7f3f000ce800 2023-11-28T20:48:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f060d1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0014e1f0 con 0x7f3f00155780 2023-11-28T20:48:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f060d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0014d8c0 con 0x7f3f000ae540 2023-11-28T20:48:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f048ce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 0x7f3f0014ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f048ce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 0x7f3f0014ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43450/0 (socket says 172.21.15.45:43450) 2023-11-28T20:48:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f048ce640 1 -- 172.21.15.45:0/1838124989 learned_addr learned my addr 172.21.15.45:0/1838124989 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.022+0000 7f3f048ce640 1 -- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f000ae540 msgr2=0x7f3f0014e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3f048ce640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f000ae540 0x7f3f0014e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3f048ce640 1 -- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f00155780 msgr2=0x7f3f0014ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3f048ce640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f00155780 0x7f3f0014ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3f048ce640 1 -- 172.21.15.45:0/1838124989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef8009b30 con 0x7f3f000ce800 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3f048ce640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 0x7f3f0014ca40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000c4d0 tx=0x7f3ef000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3ef5ffb640 1 -- 172.21.15.45:0/1838124989 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef00074e0 con 0x7f3f000ce800 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3ef5ffb640 1 -- 172.21.15.45:0/1838124989 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef0007660 con 0x7f3f000ce800 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3ef5ffb640 1 -- 172.21.15.45:0/1838124989 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0004a20 con 0x7f3f000ce800 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0014dba0 con 0x7f3f000ce800 2023-11-28T20:48:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.023+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f001f3910 con 0x7f3f000ce800 2023-11-28T20:48:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.024+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0004e920 con 0x7f3f000ce800 2023-11-28T20:48:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.025+0000 7f3ef5ffb640 1 -- 172.21.15.45:0/1838124989 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3ef0004b80 con 0x7f3f000ce800 2023-11-28T20:48:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.027+0000 7f3ef5ffb640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f3ee406af90 0x7f3ee406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.027+0000 7f3ef5ffb640 1 -- 172.21.15.45:0/1838124989 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3ef0088630 con 0x7f3f000ce800 2023-11-28T20:48:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.028+0000 7f3f050cf640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f3ee406af90 0x7f3ee406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.029+0000 7f3f050cf640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f3ee406af90 0x7f3ee406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3efc006fd0 tx=0x7f3efc006bc0 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.031+0000 7f3ef5ffb640 1 -- 172.21.15.45:0/1838124989 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ef0053870 con 0x7f3f000ce800 2023-11-28T20:48:10.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.230+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f0014e360 con 0x7f3f000ce800 2023-11-28T20:48:10.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.236+0000 7f3ef5ffb640 1 -- 172.21.15.45:0/1838124989 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+13050 (secure 0 0 0) 0x7f3ef0018090 con 0x7f3f000ce800 2023-11-28T20:48:10.238 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:10.238 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":67,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:48:09.242446+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:48:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f3ee406af90 msgr2=0x7f3ee406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f3ee406af90 0x7f3ee406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3efc006fd0 tx=0x7f3efc006bc0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 msgr2=0x7f3f0014ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 0x7f3f0014ca40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000c4d0 tx=0x7f3ef000c990 comp rx=0 tx=0).stop 2023-11-28T20:48:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 shutdown_connections 2023-11-28T20:48:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f3ee406af90 0x7f3ee406d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f00155780 0x7f3f0014ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000ce800 0x7f3f0014ca40 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 --2- 172.21.15.45:0/1838124989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f000ae540 0x7f3f0014e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.239+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 >> 172.21.15.45:0/1838124989 conn(0x7f3f0006ab30 msgr2=0x7f3f0006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:10.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.240+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 shutdown_connections 2023-11-28T20:48:10.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.240+0000 7f3f060d1640 1 -- 172.21.15.45:0/1838124989 wait complete. 2023-11-28T20:48:10.261 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:48:10.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:48:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.472+0000 7f50a2f16640 1 Processor -- start 2023-11-28T20:48:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.472+0000 7f50a2f16640 1 -- start start 2023-11-28T20:48:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.474+0000 7f50a2f16640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f509c0b3040 0x7f509c06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.474+0000 7f50a2f16640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f509c0b3420 0x7f509c0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.474+0000 7f50a2f16640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f509c06d410 con 0x7f509c15b270 2023-11-28T20:48:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.474+0000 7f50a2f16640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c0ceaa0 con 0x7f509c0b3420 2023-11-28T20:48:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.474+0000 7f50a2f16640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c06d6f0 con 0x7f509c0b3040 2023-11-28T20:48:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.475+0000 7f50a2715640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f509c0b3420 0x7f509c0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.475+0000 7f50a2715640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f509c0b3420 0x7f509c0ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43458/0 (socket says 172.21.15.45:43458) 2023-11-28T20:48:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.475+0000 7f50a2715640 1 -- 172.21.15.45:0/1432336235 learned_addr learned my addr 172.21.15.45:0/1432336235 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.475+0000 7f50a1f14640 1 --2- 172.21.15.45:0/1432336235 >> v2:172.21.15.190:3300/0 conn(0x7f509c0b3040 0x7f509c06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.475+0000 7f50a2715640 1 -- 172.21.15.45:0/1432336235 >> v2:172.21.15.190:3300/0 conn(0x7f509c0b3040 msgr2=0x7f509c06cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.475+0000 7f50a2715640 1 --2- 172.21.15.45:0/1432336235 >> v2:172.21.15.190:3300/0 conn(0x7f509c0b3040 0x7f509c06cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.475+0000 7f50a2715640 1 -- 172.21.15.45:0/1432336235 >> v1:172.21.15.45:6789/0 conn(0x7f509c15b270 legacy=0x7f509c15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.475+0000 7f50a2715640 1 -- 172.21.15.45:0/1432336235 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c06d9d0 con 0x7f509c0b3420 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.475+0000 7f50a0f12640 1 -- 172.21.15.45:0/1432336235 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712795922 0 0) 0x7f509c06d410 con 0x7f509c15b270 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.477+0000 7f50a2715640 1 --2- 172.21.15.45:0/1432336235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c0b3420 0x7f509c0ccc90 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f509c0b4410 tx=0x7f509402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23dba323c1f8fad7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.477+0000 7f50a0f12640 1 -- 172.21.15.45:0/1432336235 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509402c9f0 con 0x7f509c0b3420 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.477+0000 7f50a0f12640 1 -- 172.21.15.45:0/1432336235 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509402cb50 con 0x7f509c0b3420 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.477+0000 7f50a0f12640 1 -- 172.21.15.45:0/1432336235 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50940343f0 con 0x7f509c0b3420 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.477+0000 7f50a2f16640 1 -- 172.21.15.45:0/1432336235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c0b3420 msgr2=0x7f509c0ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.477+0000 7f50a2f16640 1 --2- 172.21.15.45:0/1432336235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c0b3420 0x7f509c0ccc90 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f509c0b4410 tx=0x7f509402be50 comp rx=0 tx=0).stop 2023-11-28T20:48:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.478+0000 7f50a2f16640 1 -- 172.21.15.45:0/1432336235 shutdown_connections 2023-11-28T20:48:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.478+0000 7f50a2f16640 1 --2- 172.21.15.45:0/1432336235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c0b3420 0x7f509c0ccc90 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.478+0000 7f50a2f16640 1 --2- 172.21.15.45:0/1432336235 >> v2:172.21.15.190:3300/0 conn(0x7f509c0b3040 0x7f509c06cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.478+0000 7f50a2f16640 1 -- 172.21.15.45:0/1432336235 >> 172.21.15.45:0/1432336235 conn(0x7f509c0b1050 msgr2=0x7f509c0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.478+0000 7f50a2f16640 1 -- 172.21.15.45:0/1432336235 shutdown_connections 2023-11-28T20:48:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.478+0000 7f50a2f16640 1 -- 172.21.15.45:0/1432336235 wait complete. 2023-11-28T20:48:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.479+0000 7f50a2f16640 1 Processor -- start 2023-11-28T20:48:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.480+0000 7f50a2f16640 1 -- start start 2023-11-28T20:48:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.480+0000 7f50a2f16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f509c0b3040 0x7f509c141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.480+0000 7f50a2f16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f509c0b3420 0x7f509c152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2f16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c15b270 0x7f509c152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2f16640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0ac240 con 0x7f509c15b270 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2f16640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c15e490 con 0x7f509c0b3040 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2f16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509c15b740 con 0x7f509c0b3420 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2715640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c15b270 0x7f509c152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2715640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c15b270 0x7f509c152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43464/0 (socket says 172.21.15.45:43464) 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2715640 1 -- 172.21.15.45:0/1371437250 learned_addr learned my addr 172.21.15.45:0/1371437250 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a1f14640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f509c0b3040 0x7f509c141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a1713640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f509c0b3420 0x7f509c152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2715640 1 -- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f509c0b3420 msgr2=0x7f509c152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2715640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f509c0b3420 0x7f509c152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2715640 1 -- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f509c0b3040 msgr2=0x7f509c141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2715640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f509c0b3040 0x7f509c141660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a2715640 1 -- 172.21.15.45:0/1371437250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5094009b30 con 0x7f509c15b270 2023-11-28T20:48:10.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.481+0000 7f50a1713640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f509c0b3420 0x7f509c152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:10.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.482+0000 7f50a2715640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c15b270 0x7f509c152dd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5094036040 tx=0x7f5094004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:10.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.482+0000 7f50927fc640 1 -- 172.21.15.45:0/1371437250 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094003990 con 0x7f509c15b270 2023-11-28T20:48:10.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.482+0000 7f50927fc640 1 -- 172.21.15.45:0/1371437250 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5094003af0 con 0x7f509c15b270 2023-11-28T20:48:10.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.482+0000 7f50927fc640 1 -- 172.21.15.45:0/1371437250 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094003cc0 con 0x7f509c15b270 2023-11-28T20:48:10.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.482+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c15ba20 con 0x7f509c15b270 2023-11-28T20:48:10.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.482+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c141ba0 con 0x7f509c15b270 2023-11-28T20:48:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.484+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c15bf40 con 0x7f509c15b270 2023-11-28T20:48:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.484+0000 7f50927fc640 1 -- 172.21.15.45:0/1371437250 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5094048460 con 0x7f509c15b270 2023-11-28T20:48:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.485+0000 7f50927fc640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f508006af60 0x7f508006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.485+0000 7f50927fc640 1 -- 172.21.15.45:0/1371437250 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f50940abc60 con 0x7f509c15b270 2023-11-28T20:48:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.486+0000 7f50a1f14640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f508006af60 0x7f508006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.487+0000 7f50a1f14640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f508006af60 0x7f508006d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5098006fd0 tx=0x7f5098008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.490+0000 7f50927fc640 1 -- 172.21.15.45:0/1371437250 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5094076e00 con 0x7f509c15b270 2023-11-28T20:48:10.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.683+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f509c09d0f0 con 0x7f509c15b270 2023-11-28T20:48:10.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.684+0000 7f50927fc640 1 -- 172.21.15.45:0/1371437250 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7f509407a420 con 0x7f509c15b270 2023-11-28T20:48:10.686 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:48:10.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f508006af60 msgr2=0x7f508006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f508006af60 0x7f508006d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5098006fd0 tx=0x7f5098008040 comp rx=0 tx=0).stop 2023-11-28T20:48:10.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c15b270 msgr2=0x7f509c152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c15b270 0x7f509c152dd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5094036040 tx=0x7f5094004a40 comp rx=0 tx=0).stop 2023-11-28T20:48:10.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 shutdown_connections 2023-11-28T20:48:10.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f508006af60 0x7f508006d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f509c15b270 0x7f509c152dd0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f509c0b3420 0x7f509c152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 --2- 172.21.15.45:0/1371437250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f509c0b3040 0x7f509c141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 >> 172.21.15.45:0/1371437250 conn(0x7f509c0b1050 msgr2=0x7f509c1577c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:10.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 shutdown_connections 2023-11-28T20:48:10.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.687+0000 7f50a2f16640 1 -- 172.21.15.45:0/1371437250 wait complete. 2023-11-28T20:48:10.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T20:48:10.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.929+0000 7fd76d148640 1 Processor -- start 2023-11-28T20:48:10.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.930+0000 7fd76d148640 1 -- start start 2023-11-28T20:48:10.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.931+0000 7fd76d148640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd7680b10b0 0x7fd7680718a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.931+0000 7fd76d148640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7680ce710 con 0x7fd76815d320 2023-11-28T20:48:10.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.931+0000 7fd76d148640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7680ae2c0 con 0x7fd7680b0c30 2023-11-28T20:48:10.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.931+0000 7fd76d148640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7680ae5a0 con 0x7fd7680b10b0 2023-11-28T20:48:10.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.932+0000 7fd766d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd76815d320 0x7fd7680ad800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39736/0 (socket says 172.21.15.45:39736) 2023-11-28T20:48:10.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.932+0000 7fd766d9d640 1 -- 172.21.15.45:0/3087647263 learned_addr learned my addr 172.21.15.45:0/3087647263 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:10.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.932+0000 7fd76759e640 1 --2- 172.21.15.45:0/3087647263 >> v2:172.21.15.190:3300/0 conn(0x7fd7680b10b0 0x7fd7680718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.932+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468719132 0 0) 0x7fd7680ce710 con 0x7fd76815d320 2023-11-28T20:48:10.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.932+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd754003520 con 0x7fd76815d320 2023-11-28T20:48:10.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 794339726 0 0) 0x7fd754003520 con 0x7fd76815d320 2023-11-28T20:48:10.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 >> v2:172.21.15.190:3300/0 conn(0x7fd7680b10b0 msgr2=0x7fd7680718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:10.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 --2- 172.21.15.45:0/3087647263 >> v2:172.21.15.190:3300/0 conn(0x7fd7680b10b0 0x7fd7680718a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 >> v1:172.21.15.190:6789/0 conn(0x7fd7680b0c30 legacy=0x7fd768071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7681611f0 con 0x7fd76815d320 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd758001670 con 0x7fd76815d320 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 >> v1:172.21.15.45:6789/0 conn(0x7fd76815d320 legacy=0x7fd7680ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd754003970 0x7fd754003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7540047c0 0x7fd754004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd754005180 0x7fd75400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75400bff0 con 0x7fd754003970 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75400c160 con 0x7fd754005180 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75400c320 con 0x7fd7540047c0 2023-11-28T20:48:10.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd766d9d640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd754003970 0x7fd754003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.933+0000 7fd75ffff640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7540047c0 0x7fd754004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.934+0000 7fd766d9d640 1 -- 172.21.15.45:0/3087647263 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7540047c0 msgr2=0x7fd754004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.934+0000 7fd766d9d640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7540047c0 0x7fd754004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.934+0000 7fd766d9d640 1 -- 172.21.15.45:0/3087647263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd754005180 msgr2=0x7fd75400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.934+0000 7fd76759e640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd754005180 0x7fd75400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.934+0000 7fd766d9d640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd754005180 0x7fd75400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.934+0000 7fd766d9d640 1 -- 172.21.15.45:0/3087647263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd754003520 con 0x7fd754003970 2023-11-28T20:48:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.934+0000 7fd76759e640 1 -- 172.21.15.45:0/3087647263 reap_dead start 2023-11-28T20:48:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.934+0000 7fd75ffff640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7540047c0 0x7fd754004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.935+0000 7fd766d9d640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd754003970 0x7fd754003d90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c240 tx=0x7fd7540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e795b8848eb6e37c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.935+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758005ad0 con 0x7fd754003970 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.935+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd758005c30 con 0x7fd754003970 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.935+0000 7fd76659c640 1 -- 172.21.15.45:0/3087647263 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75802ca90 con 0x7fd754003970 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.935+0000 7fd76d148640 1 -- 172.21.15.45:0/3087647263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd754003970 msgr2=0x7fd754003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.935+0000 7fd76d148640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd754003970 0x7fd754003d90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c240 tx=0x7fd7540057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.936+0000 7fd76d148640 1 -- 172.21.15.45:0/3087647263 shutdown_connections 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.936+0000 7fd76d148640 1 --2- 172.21.15.45:0/3087647263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd754003970 0x7fd754003d90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.936+0000 7fd76d148640 1 -- 172.21.15.45:0/3087647263 >> 172.21.15.45:0/3087647263 conn(0x7fd76806a500 msgr2=0x7fd76806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.936+0000 7fd76d148640 1 -- 172.21.15.45:0/3087647263 shutdown_connections 2023-11-28T20:48:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.936+0000 7fd76d148640 1 -- 172.21.15.45:0/3087647263 wait complete. 2023-11-28T20:48:10.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.938+0000 7fd76d148640 1 Processor -- start 2023-11-28T20:48:10.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.938+0000 7fd76d148640 1 -- start start 2023-11-28T20:48:10.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76d148640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd754003970 0x7fd768148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76d148640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7680b37d0 0x7fd7681532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76d148640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7680b3c50 0x7fd768153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76d148640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd768148e40 con 0x7fd7680b3c50 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76d148640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76815e7c0 con 0x7fd7680b37d0 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76d148640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7680b4220 con 0x7fd754003970 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7680b3c50 0x7fd768153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7680b3c50 0x7fd768153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43478/0 (socket says 172.21.15.45:43478) 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76759e640 1 -- 172.21.15.45:0/3638890093 learned_addr learned my addr 172.21.15.45:0/3638890093 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd75ffff640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7680b37d0 0x7fd7681532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd766d9d640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd754003970 0x7fd768148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76759e640 1 -- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd754003970 msgr2=0x7fd768148900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76759e640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd754003970 0x7fd768148900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76759e640 1 -- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7680b37d0 msgr2=0x7fd7681532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76759e640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7680b37d0 0x7fd7681532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:10.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd76759e640 1 -- 172.21.15.45:0/3638890093 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7680b4500 con 0x7fd7680b3c50 2023-11-28T20:48:10.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.939+0000 7fd75ffff640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7680b37d0 0x7fd7681532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:10.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.940+0000 7fd76759e640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7680b3c50 0x7fd768153820 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd76806e600 tx=0x7fd76000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:10.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.940+0000 7fd75f7fe640 1 -- 172.21.15.45:0/3638890093 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760016420 con 0x7fd7680b3c50 2023-11-28T20:48:10.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.940+0000 7fd75f7fe640 1 -- 172.21.15.45:0/3638890093 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7600165a0 con 0x7fd7680b3c50 2023-11-28T20:48:10.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.940+0000 7fd75f7fe640 1 -- 172.21.15.45:0/3638890093 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760016950 con 0x7fd7680b3c50 2023-11-28T20:48:10.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.940+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76815e450 con 0x7fd7680b3c50 2023-11-28T20:48:10.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.940+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd768155490 con 0x7fd7680b3c50 2023-11-28T20:48:10.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.941+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76815e970 con 0x7fd7680b3c50 2023-11-28T20:48:10.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.942+0000 7fd75f7fe640 1 -- 172.21.15.45:0/3638890093 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd760016ab0 con 0x7fd7680b3c50 2023-11-28T20:48:10.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.943+0000 7fd75f7fe640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fd74806af60 0x7fd74806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:10.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.943+0000 7fd75f7fe640 1 -- 172.21.15.45:0/3638890093 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd76008a480 con 0x7fd7680b3c50 2023-11-28T20:48:10.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.944+0000 7fd766d9d640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fd74806af60 0x7fd74806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:10.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.945+0000 7fd766d9d640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fd74806af60 0x7fd74806d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd758034000 tx=0x7fd758009030 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:10.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:10.947+0000 7fd75f7fe640 1 -- 172.21.15.45:0/3638890093 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd76815e970 con 0x7fd7680b3c50 2023-11-28T20:48:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.144+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd76815e930 con 0x7fd7680b3c50 2023-11-28T20:48:11.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.145+0000 7fd75f7fe640 1 -- 172.21.15.45:0/3638890093 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v67) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd7600553f0 con 0x7fd7680b3c50 2023-11-28T20:48:11.147 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:11.147 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:48:11.147 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:11.147 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T20:48:11.147 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T20:48:11.147 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:11.147 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:48:11.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fd74806af60 msgr2=0x7fd74806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:11.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fd74806af60 0x7fd74806d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd758034000 tx=0x7fd758009030 comp rx=0 tx=0).stop 2023-11-28T20:48:11.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7680b3c50 msgr2=0x7fd768153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:11.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7680b3c50 0x7fd768153820 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd76806e600 tx=0x7fd76000f520 comp rx=0 tx=0).stop 2023-11-28T20:48:11.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 shutdown_connections 2023-11-28T20:48:11.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fd74806af60 0x7fd74806d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7680b3c50 0x7fd768153820 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7680b37d0 0x7fd7681532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 --2- 172.21.15.45:0/3638890093 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd754003970 0x7fd768148900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 >> 172.21.15.45:0/3638890093 conn(0x7fd76806a500 msgr2=0x7fd7680cde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:11.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 shutdown_connections 2023-11-28T20:48:11.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.148+0000 7fd76d148640 1 -- 172.21.15.45:0/3638890093 wait complete. 2023-11-28T20:48:11.168 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T20:48:11.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T20:48:11.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.400+0000 7fb9bd106640 1 Processor -- start 2023-11-28T20:48:11.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.401+0000 7fb9bd106640 1 -- start start 2023-11-28T20:48:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.402+0000 7fb9bd106640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb9b815b190 0x7fb9b806de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.402+0000 7fb9bd106640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb9b8070030 0x7fb9b80b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.402+0000 7fb9bd106640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b806e3a0 con 0x7fb9b806c090 2023-11-28T20:48:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.403+0000 7fb9bd106640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b815b570 con 0x7fb9b8070030 2023-11-28T20:48:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.403+0000 7fb9bd106640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b80c9d20 con 0x7fb9b815b190 2023-11-28T20:48:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.403+0000 7fb9affff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb9b8070030 0x7fb9b80b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.403+0000 7fb9bc905640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb9b806c090 0x7fb9b80ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39744/0 (socket says 172.21.15.45:39744) 2023-11-28T20:48:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.403+0000 7fb9bc905640 1 -- 172.21.15.45:0/3949335291 learned_addr learned my addr 172.21.15.45:0/3949335291 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.403+0000 7fb9affff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb9b8070030 0x7fb9b80b1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43484/0 (socket says 172.21.15.45:43484) 2023-11-28T20:48:11.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.403+0000 7fb9b7fff640 1 --2- 172.21.15.45:0/3949335291 >> v2:172.21.15.190:3300/0 conn(0x7fb9b815b190 0x7fb9b806de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:11.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.404+0000 7fb9affff640 1 -- 172.21.15.45:0/3949335291 >> v2:172.21.15.190:3300/0 conn(0x7fb9b815b190 msgr2=0x7fb9b806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:11.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.404+0000 7fb9affff640 1 --2- 172.21.15.45:0/3949335291 >> v2:172.21.15.190:3300/0 conn(0x7fb9b815b190 0x7fb9b806de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.404+0000 7fb9affff640 1 -- 172.21.15.45:0/3949335291 >> v1:172.21.15.45:6789/0 conn(0x7fb9b806c090 legacy=0x7fb9b80ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:11.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.404+0000 7fb9affff640 1 -- 172.21.15.45:0/3949335291 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8071c50 con 0x7fb9b8070030 2023-11-28T20:48:11.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.404+0000 7fb9b77fe640 1 -- 172.21.15.45:0/3949335291 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207503894 0 0) 0x7fb9b0001490 con 0x7fb9b806c090 2023-11-28T20:48:11.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.404+0000 7fb9b7fff640 1 --2- 172.21.15.45:0/3949335291 >> v2:172.21.15.190:3300/0 conn(0x7fb9b815b190 0x7fb9b806de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:48:11.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.405+0000 7fb9affff640 1 --2- 172.21.15.45:0/3949335291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b8070030 0x7fb9b80b1a80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb9b806bee0 tx=0x7fb9a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9659a7e6d6d8238a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:11.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.406+0000 7fb9b77fe640 1 -- 172.21.15.45:0/3949335291 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0031830 con 0x7fb9b8070030 2023-11-28T20:48:11.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.406+0000 7fb9b77fe640 1 -- 172.21.15.45:0/3949335291 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a0031990 con 0x7fb9b8070030 2023-11-28T20:48:11.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.406+0000 7fb9b77fe640 1 -- 172.21.15.45:0/3949335291 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0031d20 con 0x7fb9b8070030 2023-11-28T20:48:11.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.407+0000 7fb9bd106640 1 -- 172.21.15.45:0/3949335291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b8070030 msgr2=0x7fb9b80b1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:11.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.407+0000 7fb9bd106640 1 --2- 172.21.15.45:0/3949335291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b8070030 0x7fb9b80b1a80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb9b806bee0 tx=0x7fb9a002b8d0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.407+0000 7fb9bd106640 1 -- 172.21.15.45:0/3949335291 shutdown_connections 2023-11-28T20:48:11.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.407+0000 7fb9bd106640 1 --2- 172.21.15.45:0/3949335291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b8070030 0x7fb9b80b1a80 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.407+0000 7fb9bd106640 1 --2- 172.21.15.45:0/3949335291 >> v2:172.21.15.190:3300/0 conn(0x7fb9b815b190 0x7fb9b806de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.407+0000 7fb9bd106640 1 -- 172.21.15.45:0/3949335291 >> 172.21.15.45:0/3949335291 conn(0x7fb9b80cec30 msgr2=0x7fb9b80cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:11.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.407+0000 7fb9bd106640 1 -- 172.21.15.45:0/3949335291 shutdown_connections 2023-11-28T20:48:11.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.407+0000 7fb9bd106640 1 -- 172.21.15.45:0/3949335291 wait complete. 2023-11-28T20:48:11.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.409+0000 7fb9bd106640 1 Processor -- start 2023-11-28T20:48:11.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.409+0000 7fb9bd106640 1 -- start start 2023-11-28T20:48:11.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.410+0000 7fb9bd106640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9b806c090 0x7fb9b80cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:11.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.410+0000 7fb9bd106640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9b8070030 0x7fb9b80b3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:11.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.410+0000 7fb9bd106640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b815b190 0x7fb9b80b3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:11.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.410+0000 7fb9bd106640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80c6c20 con 0x7fb9b815b190 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.410+0000 7fb9bd106640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80c66f0 con 0x7fb9b806c090 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.410+0000 7fb9bd106640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80c6940 con 0x7fb9b8070030 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9bc905640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b815b190 0x7fb9b80b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9bc905640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b815b190 0x7fb9b80b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43488/0 (socket says 172.21.15.45:43488) 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9bc905640 1 -- 172.21.15.45:0/2464732234 learned_addr learned my addr 172.21.15.45:0/2464732234 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9b7fff640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9b806c090 0x7fb9b80cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9affff640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9b8070030 0x7fb9b80b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9bc905640 1 -- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9b8070030 msgr2=0x7fb9b80b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9bc905640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9b8070030 0x7fb9b80b3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9bc905640 1 -- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9b806c090 msgr2=0x7fb9b80cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9bc905640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9b806c090 0x7fb9b80cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:11.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9bc905640 1 -- 172.21.15.45:0/2464732234 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a0009400 con 0x7fb9b815b190 2023-11-28T20:48:11.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.411+0000 7fb9affff640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9b8070030 0x7fb9b80b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:11.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.412+0000 7fb9bc905640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b815b190 0x7fb9b80b3760 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c7f0 tx=0x7fb9b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:11.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.412+0000 7fb9b57fa640 1 -- 172.21.15.45:0/2464732234 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b000a5e0 con 0x7fb9b815b190 2023-11-28T20:48:11.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.412+0000 7fb9b57fa640 1 -- 172.21.15.45:0/2464732234 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b000a760 con 0x7fb9b815b190 2023-11-28T20:48:11.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.412+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b80c33b0 con 0x7fb9b815b190 2023-11-28T20:48:11.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.412+0000 7fb9b57fa640 1 -- 172.21.15.45:0/2464732234 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b000ab50 con 0x7fb9b815b190 2023-11-28T20:48:11.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.412+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b80c3970 con 0x7fb9b815b190 2023-11-28T20:48:11.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.413+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b80c3210 con 0x7fb9b815b190 2023-11-28T20:48:11.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.414+0000 7fb9b57fa640 1 -- 172.21.15.45:0/2464732234 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb9b000acb0 con 0x7fb9b815b190 2023-11-28T20:48:11.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.415+0000 7fb9b57fa640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb99806af90 0x7fb99806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:11.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.415+0000 7fb9b57fa640 1 -- 172.21.15.45:0/2464732234 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb9b0088c10 con 0x7fb9b815b190 2023-11-28T20:48:11.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.416+0000 7fb9b7fff640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb99806af90 0x7fb99806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:11.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.417+0000 7fb9b7fff640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb99806af90 0x7fb99806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8150750 tx=0x7fb9a8008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:11.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.419+0000 7fb9b57fa640 1 -- 172.21.15.45:0/2464732234 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb9b80c3210 con 0x7fb9b815b190 2023-11-28T20:48:11.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:11.616+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb9b80c3210 con 0x7fb9b815b190 2023-11-28T20:48:12.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.247+0000 7fb9b57fa640 1 -- 172.21.15.45:0/2464732234 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v68) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb9b80c3210 con 0x7fb9b815b190 2023-11-28T20:48:12.250 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T20:48:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb99806af90 msgr2=0x7fb99806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb99806af90 0x7fb99806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8150750 tx=0x7fb9a8008040 comp rx=0 tx=0).stop 2023-11-28T20:48:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b815b190 msgr2=0x7fb9b80b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b815b190 0x7fb9b80b3760 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c7f0 tx=0x7fb9b000ccb0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 shutdown_connections 2023-11-28T20:48:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb99806af90 0x7fb99806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9b815b190 0x7fb9b80b3760 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9b8070030 0x7fb9b80b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 --2- 172.21.15.45:0/2464732234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9b806c090 0x7fb9b80cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.252+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 >> 172.21.15.45:0/2464732234 conn(0x7fb9b80cec30 msgr2=0x7fb9b806c680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:12.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.253+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 shutdown_connections 2023-11-28T20:48:12.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.253+0000 7fb9bd106640 1 -- 172.21.15.45:0/2464732234 wait complete. 2023-11-28T20:48:12.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T20:48:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.488+0000 7fc5ef856640 1 Processor -- start 2023-11-28T20:48:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.488+0000 7fc5ef856640 1 -- start start 2023-11-28T20:48:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.490+0000 7fc5ef856640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc5e806dc50 0x7fc5e80cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.490+0000 7fc5ef856640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e806ac10 0x7fc5e80c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.491+0000 7fc5ef856640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e806e030 con 0x7fc5e80ce600 2023-11-28T20:48:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.491+0000 7fc5ef856640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e80ccf60 con 0x7fc5e806dc50 2023-11-28T20:48:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.491+0000 7fc5ef856640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80cd930 con 0x7fc5e806ac10 2023-11-28T20:48:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.491+0000 7fc5ecdf1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc5e80ce600 0x7fc5e806a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39758/0 (socket says 172.21.15.45:39758) 2023-11-28T20:48:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.491+0000 7fc5ecdf1640 1 -- 172.21.15.45:0/1847504778 learned_addr learned my addr 172.21.15.45:0/1847504778 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.491+0000 7fc5eddf3640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e806ac10 0x7fc5e80c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.491+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518664516 0 0) 0x7fc5d8001110 con 0x7fc5e80ce600 2023-11-28T20:48:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.491+0000 7fc5ed5f2640 1 --2- 172.21.15.45:0/1847504778 >> v2:172.21.15.190:3300/0 conn(0x7fc5e806dc50 0x7fc5e80cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.491+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d4003520 con 0x7fc5e80ce600 2023-11-28T20:48:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2403342877 0 0) 0x7fc5d4003520 con 0x7fc5e80ce600 2023-11-28T20:48:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e806ac10 msgr2=0x7fc5e80c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:12.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e806ac10 0x7fc5e80c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 >> v2:172.21.15.190:3300/0 conn(0x7fc5e806dc50 msgr2=0x7fc5e80cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 --2- 172.21.15.45:0/1847504778 >> v2:172.21.15.190:3300/0 conn(0x7fc5e806dc50 0x7fc5e80cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e80cc930 con 0x7fc5e80ce600 2023-11-28T20:48:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc5d8001800 con 0x7fc5e80ce600 2023-11-28T20:48:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 >> v1:172.21.15.45:6789/0 conn(0x7fc5e80ce600 legacy=0x7fc5e806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d4003970 0x7fc5d4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5d40047c0 0x7fc5d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4005180 0x7fc5d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80ccf60 con 0x7fc5d4005180 2023-11-28T20:48:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80cd930 con 0x7fc5d4003970 2023-11-28T20:48:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d400bef0 con 0x7fc5d40047c0 2023-11-28T20:48:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5eddf3640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e806ac10 0x7fc5e80c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:48:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.492+0000 7fc5eddf3640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4005180 0x7fc5d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.493+0000 7fc5ecdf1640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5d40047c0 0x7fc5d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.493+0000 7fc5ed5f2640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d4003970 0x7fc5d4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.493+0000 7fc5eddf3640 1 -- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5d40047c0 msgr2=0x7fc5d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.493+0000 7fc5eddf3640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5d40047c0 0x7fc5d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.493+0000 7fc5eddf3640 1 -- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d4003970 msgr2=0x7fc5d4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.493+0000 7fc5eddf3640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d4003970 0x7fc5d4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.493+0000 7fc5eddf3640 1 -- 172.21.15.45:0/1847504778 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d4003520 con 0x7fc5d4005180 2023-11-28T20:48:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.494+0000 7fc5eddf3640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4005180 0x7fc5d400b980 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009f00 tx=0x7fc5d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a249cc42acddc661 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e002cc70 con 0x7fc5d4005180 2023-11-28T20:48:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e002cdd0 con 0x7fc5d4005180 2023-11-28T20:48:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5dffff640 1 -- 172.21.15.45:0/1847504778 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0003bb0 con 0x7fc5d4005180 2023-11-28T20:48:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5ef856640 1 -- 172.21.15.45:0/1847504778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4005180 msgr2=0x7fc5d400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5ef856640 1 --2- 172.21.15.45:0/1847504778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4005180 0x7fc5d400b980 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009f00 tx=0x7fc5d40057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5eddf3640 1 -- 172.21.15.45:0/1847504778 reap_dead start 2023-11-28T20:48:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5ef856640 1 -- 172.21.15.45:0/1847504778 shutdown_connections 2023-11-28T20:48:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5ef856640 1 -- 172.21.15.45:0/1847504778 >> 172.21.15.45:0/1847504778 conn(0x7fc5e80ced60 msgr2=0x7fc5e80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5ef856640 1 -- 172.21.15.45:0/1847504778 shutdown_connections 2023-11-28T20:48:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.495+0000 7fc5ef856640 1 -- 172.21.15.45:0/1847504778 wait complete. 2023-11-28T20:48:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.497+0000 7fc5ef856640 1 Processor -- start 2023-11-28T20:48:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.498+0000 7fc5ef856640 1 -- start start 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.498+0000 7fc5ef856640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5d4005180 0x7fc5e81f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.498+0000 7fc5ef856640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e80b4750 0x7fc5e815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5ef856640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e8151330 0x7fc5e815d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5ef856640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80b1940 con 0x7fc5e8151330 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5ef856640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e815dd00 con 0x7fc5e80b4750 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5ef856640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e815dfe0 con 0x7fc5d4005180 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5eddf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e8151330 0x7fc5e815d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5eddf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e8151330 0x7fc5e815d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43508/0 (socket says 172.21.15.45:43508) 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5eddf3640 1 -- 172.21.15.45:0/3706330856 learned_addr learned my addr 172.21.15.45:0/3706330856 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5ecdf1640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e80b4750 0x7fc5e815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5ed5f2640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5d4005180 0x7fc5e81f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:12.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5eddf3640 1 -- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5d4005180 msgr2=0x7fc5e81f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5eddf3640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5d4005180 0x7fc5e81f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5eddf3640 1 -- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e80b4750 msgr2=0x7fc5e815cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5eddf3640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e80b4750 0x7fc5e815cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.499+0000 7fc5eddf3640 1 -- 172.21.15.45:0/3706330856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e80b1b90 con 0x7fc5e8151330 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.500+0000 7fc5eddf3640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e8151330 0x7fc5e815d4c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000d0a0 tx=0x7fc5e0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.500+0000 7fc5ddffb640 1 -- 172.21.15.45:0/3706330856 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e003c070 con 0x7fc5e8151330 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.500+0000 7fc5ddffb640 1 -- 172.21.15.45:0/3706330856 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e0003e50 con 0x7fc5e8151330 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.500+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e815da00 con 0x7fc5e8151330 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.500+0000 7fc5ddffb640 1 -- 172.21.15.45:0/3706330856 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0004730 con 0x7fc5e8151330 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.500+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e80b20b0 con 0x7fc5e8151330 2023-11-28T20:48:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.501+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e80733d0 con 0x7fc5e8151330 2023-11-28T20:48:12.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.503+0000 7fc5ddffb640 1 -- 172.21.15.45:0/3706330856 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc5e0037460 con 0x7fc5e8151330 2023-11-28T20:48:12.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.504+0000 7fc5ddffb640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc5cc06af60 0x7fc5cc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:12.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.504+0000 7fc5ddffb640 1 -- 172.21.15.45:0/3706330856 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5547+0+0 (secure 0 0 0) 0x7fc5e00ac190 con 0x7fc5e8151330 2023-11-28T20:48:12.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.504+0000 7fc5ed5f2640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc5cc06af60 0x7fc5cc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:12.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.505+0000 7fc5ed5f2640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc5cc06af60 0x7fc5cc06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009a10 tx=0x7fc5e4008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.508+0000 7fc5ddffb640 1 -- 172.21.15.45:0/3706330856 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5e0077240 con 0x7fc5e8151330 2023-11-28T20:48:12.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:12.706+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc5e80caf40 con 0x7fc5e8151330 2023-11-28T20:48:13.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.255+0000 7fc5ddffb640 1 -- 172.21.15.45:0/3706330856 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v69) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc5e007a860 con 0x7fc5e8151330 2023-11-28T20:48:13.257 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T20:48:13.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc5cc06af60 msgr2=0x7fc5cc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:13.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc5cc06af60 0x7fc5cc06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009a10 tx=0x7fc5e4008040 comp rx=0 tx=0).stop 2023-11-28T20:48:13.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e8151330 msgr2=0x7fc5e815d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:13.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e8151330 0x7fc5e815d4c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000d0a0 tx=0x7fc5e0004ab0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 shutdown_connections 2023-11-28T20:48:13.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fc5cc06af60 0x7fc5cc06d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e8151330 0x7fc5e815d4c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e80b4750 0x7fc5e815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 --2- 172.21.15.45:0/3706330856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5d4005180 0x7fc5e81f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.259+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 >> 172.21.15.45:0/3706330856 conn(0x7fc5e80ced60 msgr2=0x7fc5e80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:13.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.261+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 shutdown_connections 2023-11-28T20:48:13.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.261+0000 7fc5ef856640 1 -- 172.21.15.45:0/3706330856 wait complete. 2023-11-28T20:48:13.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T20:48:13.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.490+0000 7faff2105640 1 Processor -- start 2023-11-28T20:48:13.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.490+0000 7faff2105640 1 -- start start 2023-11-28T20:48:13.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.492+0000 7faff2105640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fafec0c1d30 0x7fafec0ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:13.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.492+0000 7faff2105640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fafec0712c0 0x7fafec162290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:13.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.492+0000 7faff2105640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafec15f7e0 con 0x7fafec1653a0 2023-11-28T20:48:13.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.492+0000 7faff2105640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafec1658a0 con 0x7fafec0c1d30 2023-11-28T20:48:13.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.492+0000 7faff2105640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafec165b80 con 0x7fafec0712c0 2023-11-28T20:48:13.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.492+0000 7fafeb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fafec0c1d30 0x7fafec0ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:13.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.493+0000 7fafeb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fafec0c1d30 0x7fafec0ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43514/0 (socket says 172.21.15.45:43514) 2023-11-28T20:48:13.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.493+0000 7fafeb7fe640 1 -- 172.21.15.45:0/517989431 learned_addr learned my addr 172.21.15.45:0/517989431 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:13.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.493+0000 7fafebfff640 1 --2- 172.21.15.45:0/517989431 >> v2:172.21.15.190:3300/0 conn(0x7fafec0712c0 0x7fafec162290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:13.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.493+0000 7fafeb7fe640 1 -- 172.21.15.45:0/517989431 >> v2:172.21.15.190:3300/0 conn(0x7fafec0712c0 msgr2=0x7fafec162290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:13.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.493+0000 7fafeb7fe640 1 --2- 172.21.15.45:0/517989431 >> v2:172.21.15.190:3300/0 conn(0x7fafec0712c0 0x7fafec162290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.493+0000 7fafeb7fe640 1 -- 172.21.15.45:0/517989431 >> v1:172.21.15.45:6789/0 conn(0x7fafec1653a0 legacy=0x7fafec0c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:13.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.493+0000 7fafeb7fe640 1 -- 172.21.15.45:0/517989431 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec165e60 con 0x7fafec0c1d30 2023-11-28T20:48:13.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.494+0000 7fafeb7fe640 1 --2- 172.21.15.45:0/517989431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 0x7fafec0ce480 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009870 tx=0x7fafdc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1cfd8fea2458f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:13.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7fafeaffd640 1 -- 172.21.15.45:0/517989431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc030860 con 0x7fafec0c1d30 2023-11-28T20:48:13.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7fafeaffd640 1 -- 172.21.15.45:0/517989431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc0309c0 con 0x7fafec0c1d30 2023-11-28T20:48:13.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7fafeaffd640 1 -- 172.21.15.45:0/517989431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc030d50 con 0x7fafec0c1d30 2023-11-28T20:48:13.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7faff2105640 1 -- 172.21.15.45:0/517989431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 msgr2=0x7fafec0ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:13.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7faff2105640 1 --2- 172.21.15.45:0/517989431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 0x7fafec0ce480 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009870 tx=0x7fafdc02b9a0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7faff2105640 1 -- 172.21.15.45:0/517989431 shutdown_connections 2023-11-28T20:48:13.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7faff2105640 1 --2- 172.21.15.45:0/517989431 >> v2:172.21.15.190:3300/0 conn(0x7fafec0712c0 0x7fafec162290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7faff2105640 1 --2- 172.21.15.45:0/517989431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 0x7fafec0ce480 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7faff2105640 1 -- 172.21.15.45:0/517989431 >> 172.21.15.45:0/517989431 conn(0x7fafec0ced60 msgr2=0x7fafec0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:13.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.495+0000 7faff2105640 1 -- 172.21.15.45:0/517989431 shutdown_connections 2023-11-28T20:48:13.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.496+0000 7faff2105640 1 -- 172.21.15.45:0/517989431 wait complete. 2023-11-28T20:48:13.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.497+0000 7faff2105640 1 Processor -- start 2023-11-28T20:48:13.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.497+0000 7faff2105640 1 -- start start 2023-11-28T20:48:13.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.498+0000 7faff2105640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafec0712c0 0x7fafec158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:13.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.498+0000 7faff2105640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 0x7fafec157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:13.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.498+0000 7faff2105640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafec1653a0 0x7fafec157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:13.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.498+0000 7faff2105640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0b3470 con 0x7fafec0c1d30 2023-11-28T20:48:13.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.498+0000 7faff2105640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0b3750 con 0x7fafec1653a0 2023-11-28T20:48:13.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.498+0000 7faff2105640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0b3a30 con 0x7fafec0712c0 2023-11-28T20:48:13.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafe3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 0x7fafec157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:13.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafe3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 0x7fafec157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43516/0 (socket says 172.21.15.45:43516) 2023-11-28T20:48:13.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafe3fff640 1 -- 172.21.15.45:0/2847353775 learned_addr learned my addr 172.21.15.45:0/2847353775 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:13.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafebfff640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafec1653a0 0x7fafec157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:13.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafeb7fe640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafec0712c0 0x7fafec158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:13.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafe3fff640 1 -- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafec0712c0 msgr2=0x7fafec158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:13.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafe3fff640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafec0712c0 0x7fafec158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafe3fff640 1 -- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafec1653a0 msgr2=0x7fafec157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:13.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafe3fff640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafec1653a0 0x7fafec157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:13.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafe3fff640 1 -- 172.21.15.45:0/2847353775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0094d0 con 0x7fafec0c1d30 2023-11-28T20:48:13.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafebfff640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafec1653a0 0x7fafec157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:13.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.499+0000 7fafeb7fe640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafec0712c0 0x7fafec158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:13.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.500+0000 7fafe3fff640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 0x7fafec157110 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c3c0 tx=0x7fafd400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:13.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.500+0000 7fafe8ff9640 1 -- 172.21.15.45:0/2847353775 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40052d0 con 0x7fafec0c1d30 2023-11-28T20:48:13.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.500+0000 7fafe8ff9640 1 -- 172.21.15.45:0/2847353775 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd4005450 con 0x7fafec0c1d30 2023-11-28T20:48:13.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.500+0000 7fafe8ff9640 1 -- 172.21.15.45:0/2847353775 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4003830 con 0x7fafec0c1d30 2023-11-28T20:48:13.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.500+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec0b3d10 con 0x7fafec0c1d30 2023-11-28T20:48:13.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.500+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec158aa0 con 0x7fafec0c1d30 2023-11-28T20:48:13.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.502+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec0b4230 con 0x7fafec0c1d30 2023-11-28T20:48:13.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.503+0000 7fafe8ff9640 1 -- 172.21.15.45:0/2847353775 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fafd4003990 con 0x7fafec0c1d30 2023-11-28T20:48:13.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.504+0000 7fafe8ff9640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fafcc06af60 0x7fafcc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:13.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.504+0000 7fafe8ff9640 1 -- 172.21.15.45:0/2847353775 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fafd4088e30 con 0x7fafec0c1d30 2023-11-28T20:48:13.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.505+0000 7fafeb7fe640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fafcc06af60 0x7fafcc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:13.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.506+0000 7fafeb7fe640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fafcc06af60 0x7fafcc06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fafdc036000 tx=0x7fafdc03b040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:13.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.508+0000 7fafe8ff9640 1 -- 172.21.15.45:0/2847353775 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fafd4053cd0 con 0x7fafec0c1d30 2023-11-28T20:48:13.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:13.739+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fafec0b4230 con 0x7fafec0c1d30 2023-11-28T20:48:14.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.300+0000 7fafe8ff9640 1 -- 172.21.15.45:0/2847353775 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v23) v1 ==== 172+0+0 (secure 0 0 0) 0x7fafec0b4230 con 0x7fafec0c1d30 2023-11-28T20:48:14.303 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 7 and data pool 8 2023-11-28T20:48:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fafcc06af60 msgr2=0x7fafcc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fafcc06af60 0x7fafcc06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fafdc036000 tx=0x7fafdc03b040 comp rx=0 tx=0).stop 2023-11-28T20:48:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 msgr2=0x7fafec157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 0x7fafec157110 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c3c0 tx=0x7fafd400c880 comp rx=0 tx=0).stop 2023-11-28T20:48:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 shutdown_connections 2023-11-28T20:48:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fafcc06af60 0x7fafcc06d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafec1653a0 0x7fafec157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafec0c1d30 0x7fafec157110 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 --2- 172.21.15.45:0/2847353775 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafec0712c0 0x7fafec158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 >> 172.21.15.45:0/2847353775 conn(0x7fafec0ced60 msgr2=0x7fafec162f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:14.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 shutdown_connections 2023-11-28T20:48:14.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.304+0000 7faff2105640 1 -- 172.21.15.45:0/2847353775 wait complete. 2023-11-28T20:48:14.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:48:14.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.519+0000 7f5942bbe640 1 Processor -- start 2023-11-28T20:48:14.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.519+0000 7f5942bbe640 1 -- start start 2023-11-28T20:48:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.521+0000 7f5942bbe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f593c1438b0 0x7f593c0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.521+0000 7f5942bbe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.521+0000 7f5942bbe640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c15dcc0 con 0x7f593c0b1170 2023-11-28T20:48:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.521+0000 7f5942bbe640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593c15e840 con 0x7f593c1438b0 2023-11-28T20:48:14.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.521+0000 7f5942bbe640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0b15b0 con 0x7f593c0b24e0 2023-11-28T20:48:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.522+0000 7f593bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.522+0000 7f594095a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f593c1438b0 0x7f593c0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.522+0000 7f593bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c0b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:42978/0 (socket says 172.21.15.45:42978) 2023-11-28T20:48:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.522+0000 7f593bfff640 1 -- 172.21.15.45:0/1354574632 learned_addr learned my addr 172.21.15.45:0/1354574632 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:14.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.522+0000 7f593bfff640 1 -- 172.21.15.45:0/1354574632 >> v2:172.21.15.190:3300/0 conn(0x7f593c1438b0 msgr2=0x7f593c0b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:14.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.523+0000 7f593bfff640 1 --2- 172.21.15.45:0/1354574632 >> v2:172.21.15.190:3300/0 conn(0x7f593c1438b0 0x7f593c0b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.523+0000 7f593bfff640 1 -- 172.21.15.45:0/1354574632 >> v1:172.21.15.190:6789/0 conn(0x7f593c0b1170 legacy=0x7f593c0ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.523+0000 7f593bfff640 1 -- 172.21.15.45:0/1354574632 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c15df10 con 0x7f593c0b24e0 2023-11-28T20:48:14.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.524+0000 7f593bfff640 1 --2- 172.21.15.45:0/1354574632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c0b0c30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5928009870 tx=0x7f592802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef1e351193e7965 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:14.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f593b7fe640 1 -- 172.21.15.45:0/1354574632 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59280317c0 con 0x7f593c0b24e0 2023-11-28T20:48:14.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f593b7fe640 1 -- 172.21.15.45:0/1354574632 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5928031920 con 0x7f593c0b24e0 2023-11-28T20:48:14.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f593b7fe640 1 -- 172.21.15.45:0/1354574632 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928031cb0 con 0x7f593c0b24e0 2023-11-28T20:48:14.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f5942bbe640 1 -- 172.21.15.45:0/1354574632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 msgr2=0x7f593c0b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f5942bbe640 1 --2- 172.21.15.45:0/1354574632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c0b0c30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5928009870 tx=0x7f592802b860 comp rx=0 tx=0).stop 2023-11-28T20:48:14.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f5942bbe640 1 -- 172.21.15.45:0/1354574632 shutdown_connections 2023-11-28T20:48:14.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f5942bbe640 1 --2- 172.21.15.45:0/1354574632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c0b0c30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f5942bbe640 1 --2- 172.21.15.45:0/1354574632 >> v2:172.21.15.190:3300/0 conn(0x7f593c1438b0 0x7f593c0b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f5942bbe640 1 -- 172.21.15.45:0/1354574632 >> 172.21.15.45:0/1354574632 conn(0x7f593c06a500 msgr2=0x7f593c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f5942bbe640 1 -- 172.21.15.45:0/1354574632 shutdown_connections 2023-11-28T20:48:14.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.525+0000 7f5942bbe640 1 -- 172.21.15.45:0/1354574632 wait complete. 2023-11-28T20:48:14.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.527+0000 7f5942bbe640 1 Processor -- start 2023-11-28T20:48:14.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.527+0000 7f5942bbe640 1 -- start start 2023-11-28T20:48:14.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.528+0000 7f5942bbe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f593c0b1170 0x7f593c0ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.528+0000 7f5942bbe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c13df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.528+0000 7f5942bbe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f593c1438b0 0x7f593c13e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.528+0000 7f5942bbe640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0ad300 con 0x7f593c1438b0 2023-11-28T20:48:14.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.528+0000 7f5942bbe640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0ad5e0 con 0x7f593c0b1170 2023-11-28T20:48:14.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.528+0000 7f5942bbe640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0ad8c0 con 0x7f593c0b24e0 2023-11-28T20:48:14.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f594115b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f593c1438b0 0x7f593c13e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f594115b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f593c1438b0 0x7f593c13e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34290/0 (socket says 172.21.15.45:34290) 2023-11-28T20:48:14.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f594115b640 1 -- 172.21.15.45:0/3588668638 learned_addr learned my addr 172.21.15.45:0/3588668638 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:14.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f594095a640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f593c0b1170 0x7f593c0ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f593bfff640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c13df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f594115b640 1 -- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 msgr2=0x7f593c13df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f594115b640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c13df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f594115b640 1 -- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f593c0b1170 msgr2=0x7f593c0ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f594115b640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f593c0b1170 0x7f593c0ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.529+0000 7f594115b640 1 -- 172.21.15.45:0/3588668638 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59280094d0 con 0x7f593c1438b0 2023-11-28T20:48:14.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.530+0000 7f594115b640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f593c1438b0 0x7f593c13e470 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f593400c6b0 tx=0x7f593400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:14.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.530+0000 7f59397fa640 1 -- 172.21.15.45:0/3588668638 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934003b70 con 0x7f593c1438b0 2023-11-28T20:48:14.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.530+0000 7f59397fa640 1 -- 172.21.15.45:0/3588668638 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5934003cf0 con 0x7f593c1438b0 2023-11-28T20:48:14.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.530+0000 7f59397fa640 1 -- 172.21.15.45:0/3588668638 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934003ea0 con 0x7f593c1438b0 2023-11-28T20:48:14.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.530+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c0adba0 con 0x7f593c1438b0 2023-11-28T20:48:14.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.530+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c1401e0 con 0x7f593c1438b0 2023-11-28T20:48:14.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.531+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c140370 con 0x7f593c1438b0 2023-11-28T20:48:14.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.532+0000 7f59397fa640 1 -- 172.21.15.45:0/3588668638 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5934005310 con 0x7f593c1438b0 2023-11-28T20:48:14.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.533+0000 7f59397fa640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f592006af60 0x7f592006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.533+0000 7f59397fa640 1 -- 172.21.15.45:0/3588668638 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f59340895f0 con 0x7f593c1438b0 2023-11-28T20:48:14.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.534+0000 7f594095a640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f592006af60 0x7f592006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.535+0000 7f594095a640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f592006af60 0x7f592006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5930009a30 tx=0x7f5930008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:14.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.538+0000 7f59397fa640 1 -- 172.21.15.45:0/3588668638 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59340544e0 con 0x7f593c1438b0 2023-11-28T20:48:14.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.731+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f593c0ae0c0 con 0x7f593c1438b0 2023-11-28T20:48:14.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.733+0000 7f59397fa640 1 -- 172.21.15.45:0/3588668638 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16775 (secure 0 0 0) 0x7f593c0ae0c0 con 0x7f593c1438b0 2023-11-28T20:48:14.736 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:14.736 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":70,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:48:13.741688+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:48:11.673148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-28T20:48:12.760782+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:48:14.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f592006af60 msgr2=0x7f592006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f592006af60 0x7f592006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5930009a30 tx=0x7f5930008040 comp rx=0 tx=0).stop 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f593c1438b0 msgr2=0x7f593c13e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f593c1438b0 0x7f593c13e470 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f593400c6b0 tx=0x7f593400cb70 comp rx=0 tx=0).stop 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 shutdown_connections 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f592006af60 0x7f592006d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f593c1438b0 0x7f593c13e470 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f593c0b24e0 0x7f593c13df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 --2- 172.21.15.45:0/3588668638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f593c0b1170 0x7f593c0ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 >> 172.21.15.45:0/3588668638 conn(0x7f593c06a500 msgr2=0x7f593c141240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 shutdown_connections 2023-11-28T20:48:14.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.736+0000 7f5942bbe640 1 -- 172.21.15.45:0/3588668638 wait complete. 2023-11-28T20:48:14.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:48:14.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.968+0000 7fe428959640 1 Processor -- start 2023-11-28T20:48:14.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.968+0000 7fe428959640 1 -- start start 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.970+0000 7fe428959640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe424155780 0x7fe4240abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.970+0000 7fe428959640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42415c3c0 con 0x7fe4240ce800 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.970+0000 7fe428959640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe424155b60 con 0x7fe4240ae540 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.970+0000 7fe428959640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42415ba90 con 0x7fe424155780 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.970+0000 7fe42359e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe4240ce800 0x7fe4240b20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58196/0 (socket says 172.21.15.45:58196) 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.970+0000 7fe42359e640 1 -- 172.21.15.45:0/4018034484 learned_addr learned my addr 172.21.15.45:0/4018034484 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe422d9d640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe424155780 0x7fe4240abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114350051 0 0) 0x7fe42415c3c0 con 0x7fe4240ce800 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe40c003520 con 0x7fe4240ce800 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2325808214 0 0) 0x7fe40c003520 con 0x7fe4240ce800 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe424155780 msgr2=0x7fe4240abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:14.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe424155780 0x7fe4240abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 >> v1:172.21.15.190:6789/0 conn(0x7fe4240ae540 legacy=0x7fe4240ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42415bd70 con 0x7fe4240ce800 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fe41c0019f0 con 0x7fe4240ce800 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 >> v1:172.21.15.45:6789/0 conn(0x7fe4240ce800 legacy=0x7fe4240b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe40c0039a0 0x7fe40c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe424155780 0x7fe40c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe40c004d00 0x7fe40c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42415ba90 con 0x7fe40c004d00 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c00bb10 con 0x7fe40c0039a0 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.971+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c00bca0 con 0x7fe424155780 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe42359e640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe40c004d00 0x7fe40c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe422d9d640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe40c0039a0 0x7fe40c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe42259c640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe424155780 0x7fe40c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe42359e640 1 -- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe424155780 msgr2=0x7fe40c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe42359e640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe424155780 0x7fe40c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe42359e640 1 -- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe40c0039a0 msgr2=0x7fe40c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe42359e640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe40c0039a0 0x7fe40c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe42359e640 1 -- 172.21.15.45:0/4018034484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c003520 con 0x7fe40c004d00 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe422d9d640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe40c0039a0 0x7fe40c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.972+0000 7fe42259c640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe424155780 0x7fe40c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.973+0000 7fe42359e640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe40c004d00 0x7fe40c005180 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00a860 tx=0x7fe40c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f806f1b2bfb4474b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.973+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0072d0 con 0x7fe40c004d00 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.973+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c02f070 con 0x7fe40c004d00 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.973+0000 7fe421d9b640 1 -- 172.21.15.45:0/4018034484 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0076b0 con 0x7fe40c004d00 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.974+0000 7fe428959640 1 -- 172.21.15.45:0/4018034484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe40c004d00 msgr2=0x7fe40c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.974+0000 7fe428959640 1 --2- 172.21.15.45:0/4018034484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe40c004d00 0x7fe40c005180 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00a860 tx=0x7fe40c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.974+0000 7fe42359e640 1 -- 172.21.15.45:0/4018034484 reap_dead start 2023-11-28T20:48:14.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.974+0000 7fe428959640 1 -- 172.21.15.45:0/4018034484 shutdown_connections 2023-11-28T20:48:14.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.974+0000 7fe428959640 1 -- 172.21.15.45:0/4018034484 >> 172.21.15.45:0/4018034484 conn(0x7fe42406ab30 msgr2=0x7fe4240b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:14.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.974+0000 7fe428959640 1 -- 172.21.15.45:0/4018034484 shutdown_connections 2023-11-28T20:48:14.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.974+0000 7fe428959640 1 -- 172.21.15.45:0/4018034484 wait complete. 2023-11-28T20:48:14.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.976+0000 7fe428959640 1 Processor -- start 2023-11-28T20:48:14.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.976+0000 7fe428959640 1 -- start start 2023-11-28T20:48:14.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.977+0000 7fe428959640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe42413d590 0x7fe42414b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.977+0000 7fe428959640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe42413d970 0x7fe42414bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.977+0000 7fe428959640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4241506d0 0x7fe424150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.977+0000 7fe428959640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe424151090 con 0x7fe4241506d0 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.977+0000 7fe428959640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42414d2a0 con 0x7fe42413d970 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.977+0000 7fe428959640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42414c200 con 0x7fe42413d590 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.977+0000 7fe42359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4241506d0 0x7fe424150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.977+0000 7fe42359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4241506d0 0x7fe424150b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34302/0 (socket says 172.21.15.45:34302) 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.977+0000 7fe42359e640 1 -- 172.21.15.45:0/3879021656 learned_addr learned my addr 172.21.15.45:0/3879021656 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.978+0000 7fe422d9d640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe42413d590 0x7fe42414b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.978+0000 7fe42259c640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe42413d970 0x7fe42414bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.978+0000 7fe42359e640 1 -- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe42413d590 msgr2=0x7fe42414b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.978+0000 7fe42359e640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe42413d590 0x7fe42414b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.978+0000 7fe42359e640 1 -- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe42413d970 msgr2=0x7fe42414bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.978+0000 7fe42359e640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe42413d970 0x7fe42414bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:14.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.978+0000 7fe42359e640 1 -- 172.21.15.45:0/3879021656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42414c4e0 con 0x7fe4241506d0 2023-11-28T20:48:14.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.978+0000 7fe42359e640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4241506d0 0x7fe424150b50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00a800 tx=0x7fe41c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:14.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.978+0000 7fe40b7fe640 1 -- 172.21.15.45:0/3879021656 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c044070 con 0x7fe4241506d0 2023-11-28T20:48:14.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.979+0000 7fe40b7fe640 1 -- 172.21.15.45:0/3879021656 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c001250 con 0x7fe4241506d0 2023-11-28T20:48:14.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.979+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42414c7c0 con 0x7fe4241506d0 2023-11-28T20:48:14.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.979+0000 7fe40b7fe640 1 -- 172.21.15.45:0/3879021656 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0074e0 con 0x7fe4241506d0 2023-11-28T20:48:14.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.979+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42414cd80 con 0x7fe4241506d0 2023-11-28T20:48:14.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.980+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42414cf70 con 0x7fe4241506d0 2023-11-28T20:48:14.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.981+0000 7fe40b7fe640 1 -- 172.21.15.45:0/3879021656 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe41c031030 con 0x7fe4241506d0 2023-11-28T20:48:14.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.982+0000 7fe40b7fe640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe41006af60 0x7fe41006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:14.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.982+0000 7fe40b7fe640 1 -- 172.21.15.45:0/3879021656 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe41c0ab4c0 con 0x7fe4241506d0 2023-11-28T20:48:14.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.982+0000 7fe422d9d640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe41006af60 0x7fe41006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:14.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.984+0000 7fe422d9d640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe41006af60 0x7fe41006d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe414006fd0 tx=0x7fe414008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:14.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:14.986+0000 7fe40b7fe640 1 -- 172.21.15.45:0/3879021656 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe41c076170 con 0x7fe4241506d0 2023-11-28T20:48:15.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.181+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe42414cf70 con 0x7fe4241506d0 2023-11-28T20:48:15.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.183+0000 7fe40b7fe640 1 -- 172.21.15.45:0/3879021656 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16775 (secure 0 0 0) 0x7fe41c079790 con 0x7fe4241506d0 2023-11-28T20:48:15.185 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:15.185 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":70,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:48:13.741688+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:48:11.673148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-28T20:48:12.760782+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:48:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.185+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe41006af60 msgr2=0x7fe41006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.185+0000 7fe428959640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe41006af60 0x7fe41006d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe414006fd0 tx=0x7fe414008040 comp rx=0 tx=0).stop 2023-11-28T20:48:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.185+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4241506d0 msgr2=0x7fe424150b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.185+0000 7fe428959640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4241506d0 0x7fe424150b50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00a800 tx=0x7fe41c001d80 comp rx=0 tx=0).stop 2023-11-28T20:48:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.186+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 shutdown_connections 2023-11-28T20:48:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.186+0000 7fe428959640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe41006af60 0x7fe41006d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.186+0000 7fe428959640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4241506d0 0x7fe424150b50 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.186+0000 7fe428959640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe42413d970 0x7fe42414bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.186+0000 7fe428959640 1 --2- 172.21.15.45:0/3879021656 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe42413d590 0x7fe42414b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.186+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 >> 172.21.15.45:0/3879021656 conn(0x7fe42406ab30 msgr2=0x7fe42415afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.186+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 shutdown_connections 2023-11-28T20:48:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.186+0000 7fe428959640 1 -- 172.21.15.45:0/3879021656 wait complete. 2023-11-28T20:48:15.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T20:48:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.428+0000 7fbe34f75640 1 Processor -- start 2023-11-28T20:48:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.428+0000 7fbe34f75640 1 -- start start 2023-11-28T20:48:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.430+0000 7fbe34f75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015b270 0x7fbe3006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.430+0000 7fbe34f75640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3006d410 con 0x7fbe300b3040 2023-11-28T20:48:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.430+0000 7fbe34f75640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3006d6f0 con 0x7fbe300b3420 2023-11-28T20:48:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.430+0000 7fbe34f75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300ceaa0 con 0x7fbe3015b270 2023-11-28T20:48:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.430+0000 7fbe2f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbe300b3040 0x7fbe3015f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58210/0 (socket says 172.21.15.45:58210) 2023-11-28T20:48:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.430+0000 7fbe2f7fe640 1 -- 172.21.15.45:0/657572533 learned_addr learned my addr 172.21.15.45:0/657572533 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.430+0000 7fbe2effd640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015b270 0x7fbe3006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.430+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770845460 0 0) 0x7fbe3006d410 con 0x7fbe300b3040 2023-11-28T20:48:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1c003520 con 0x7fbe300b3040 2023-11-28T20:48:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360877772 0 0) 0x7fbe1c003520 con 0x7fbe300b3040 2023-11-28T20:48:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015b270 msgr2=0x7fbe3006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015b270 0x7fbe3006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 >> v1:172.21.15.190:6789/0 conn(0x7fbe300b3420 legacy=0x7fbe300ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3006d9d0 con 0x7fbe300b3040 2023-11-28T20:48:15.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fbe200019f0 con 0x7fbe300b3040 2023-11-28T20:48:15.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 >> v1:172.21.15.45:6789/0 conn(0x7fbe300b3040 legacy=0x7fbe3015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:15.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015b270 0x7fbe1c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:15.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe1c003f20 0x7fbe1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:15.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c004d00 0x7fbe1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:15.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c00bb90 con 0x7fbe1c003f20 2023-11-28T20:48:15.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c00bd00 con 0x7fbe1c004d00 2023-11-28T20:48:15.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c00bec0 con 0x7fbe3015b270 2023-11-28T20:48:15.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.431+0000 7fbe2effd640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe1c003f20 0x7fbe1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:15.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.432+0000 7fbe2ffff640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c004d00 0x7fbe1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:15.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.432+0000 7fbe2effd640 1 -- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015b270 msgr2=0x7fbe1c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:15.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.432+0000 7fbe2f7fe640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015b270 0x7fbe1c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:15.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.432+0000 7fbe2effd640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015b270 0x7fbe1c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.432+0000 7fbe2effd640 1 -- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c004d00 msgr2=0x7fbe1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:15.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.432+0000 7fbe2effd640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c004d00 0x7fbe1c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.432+0000 7fbe2effd640 1 -- 172.21.15.45:0/657572533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c003520 con 0x7fbe1c003f20 2023-11-28T20:48:15.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.432+0000 7fbe2ffff640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c004d00 0x7fbe1c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:15.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.432+0000 7fbe2f7fe640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015b270 0x7fbe1c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.433+0000 7fbe2effd640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe1c003f20 0x7fbe1c0047c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009a90 tx=0x7fbe1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a62b5e8a3cac55 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.433+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1802c680 con 0x7fbe1c003f20 2023-11-28T20:48:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.433+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1802c7e0 con 0x7fbe1c003f20 2023-11-28T20:48:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.434+0000 7fbe2e7fc640 1 -- 172.21.15.45:0/657572533 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1802cb70 con 0x7fbe1c003f20 2023-11-28T20:48:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.434+0000 7fbe34f75640 1 -- 172.21.15.45:0/657572533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe1c003f20 msgr2=0x7fbe1c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.434+0000 7fbe34f75640 1 --2- 172.21.15.45:0/657572533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe1c003f20 0x7fbe1c0047c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009a90 tx=0x7fbe1c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.434+0000 7fbe2ffff640 1 -- 172.21.15.45:0/657572533 reap_dead start 2023-11-28T20:48:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.434+0000 7fbe34f75640 1 -- 172.21.15.45:0/657572533 shutdown_connections 2023-11-28T20:48:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.434+0000 7fbe34f75640 1 -- 172.21.15.45:0/657572533 >> 172.21.15.45:0/657572533 conn(0x7fbe300b1050 msgr2=0x7fbe300cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.435+0000 7fbe34f75640 1 -- 172.21.15.45:0/657572533 shutdown_connections 2023-11-28T20:48:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.435+0000 7fbe34f75640 1 -- 172.21.15.45:0/657572533 wait complete. 2023-11-28T20:48:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.436+0000 7fbe34f75640 1 Processor -- start 2023-11-28T20:48:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.436+0000 7fbe34f75640 1 -- start start 2023-11-28T20:48:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.437+0000 7fbe34f75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c003f20 0x7fbe301509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.437+0000 7fbe34f75640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe301504b0 0x7fbe3014fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.437+0000 7fbe34f75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015dcf0 0x7fbe300ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.437+0000 7fbe34f75640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe301502b0 con 0x7fbe301504b0 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe34f75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3015c970 con 0x7fbe1c003f20 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe34f75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3015cc50 con 0x7fbe3015dcf0 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe301504b0 0x7fbe3014fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe301504b0 0x7fbe3014fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34328/0 (socket says 172.21.15.45:34328) 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2effd640 1 -- 172.21.15.45:0/4098394803 learned_addr learned my addr 172.21.15.45:0/4098394803 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2f7fe640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c003f20 0x7fbe301509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2ffff640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015dcf0 0x7fbe300ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2effd640 1 -- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015dcf0 msgr2=0x7fbe300ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2effd640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015dcf0 0x7fbe300ae990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2effd640 1 -- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c003f20 msgr2=0x7fbe301509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2effd640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c003f20 0x7fbe301509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2effd640 1 -- 172.21.15.45:0/4098394803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3015cf30 con 0x7fbe301504b0 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.438+0000 7fbe2f7fe640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c003f20 0x7fbe301509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.439+0000 7fbe2effd640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe301504b0 0x7fbe3014fd70 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009a90 tx=0x7fbe18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.439+0000 7fbe27fff640 1 -- 172.21.15.45:0/4098394803 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1802dea0 con 0x7fbe301504b0 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.439+0000 7fbe27fff640 1 -- 172.21.15.45:0/4098394803 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe18003810 con 0x7fbe301504b0 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.439+0000 7fbe27fff640 1 -- 172.21.15.45:0/4098394803 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe18003ba0 con 0x7fbe301504b0 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.439+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3015d210 con 0x7fbe301504b0 2023-11-28T20:48:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.439+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3015d7d0 con 0x7fbe301504b0 2023-11-28T20:48:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.440+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3015d960 con 0x7fbe301504b0 2023-11-28T20:48:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.441+0000 7fbe27fff640 1 -- 172.21.15.45:0/4098394803 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbe18003d00 con 0x7fbe301504b0 2023-11-28T20:48:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.442+0000 7fbe27fff640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fbe1006af90 0x7fbe1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.442+0000 7fbe27fff640 1 -- 172.21.15.45:0/4098394803 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbe180ab850 con 0x7fbe301504b0 2023-11-28T20:48:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.443+0000 7fbe2f7fe640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fbe1006af90 0x7fbe1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.444+0000 7fbe2f7fe640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fbe1006af90 0x7fbe1006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbe20007c40 tx=0x7fbe20001250 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.447+0000 7fbe27fff640 1 -- 172.21.15.45:0/4098394803 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe18076610 con 0x7fbe301504b0 2023-11-28T20:48:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:15.682+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbe3009d0f0 con 0x7fbe301504b0 2023-11-28T20:48:16.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.320+0000 7fbe27fff640 1 -- 172.21.15.45:0/4098394803 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbe18079c30 con 0x7fbe301504b0 2023-11-28T20:48:16.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.323+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fbe1006af90 msgr2=0x7fbe1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:16.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fbe1006af90 0x7fbe1006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbe20007c40 tx=0x7fbe20001250 comp rx=0 tx=0).stop 2023-11-28T20:48:16.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe301504b0 msgr2=0x7fbe3014fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:16.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe301504b0 0x7fbe3014fd70 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009a90 tx=0x7fbe18004730 comp rx=0 tx=0).stop 2023-11-28T20:48:16.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 shutdown_connections 2023-11-28T20:48:16.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fbe1006af90 0x7fbe1006d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe3015dcf0 0x7fbe300ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe301504b0 0x7fbe3014fd70 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 --2- 172.21.15.45:0/4098394803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe1c003f20 0x7fbe301509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 >> 172.21.15.45:0/4098394803 conn(0x7fbe300b1050 msgr2=0x7fbe300abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:16.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 shutdown_connections 2023-11-28T20:48:16.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.324+0000 7fbe34f75640 1 -- 172.21.15.45:0/4098394803 wait complete. 2023-11-28T20:48:16.348 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T20:48:16.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T20:48:16.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.565+0000 7fb5bb290640 1 Processor -- start 2023-11-28T20:48:16.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.565+0000 7fb5bb290640 1 -- start start 2023-11-28T20:48:16.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.567+0000 7fb5bb290640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb5b406a500 0x7fb5b406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:16.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.567+0000 7fb5bb290640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5b415ddd0 0x7fb5b415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:16.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.567+0000 7fb5bb290640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b406ca60 con 0x7fb5b4157c10 2023-11-28T20:48:16.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.567+0000 7fb5bb290640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b415eb40 con 0x7fb5b415ddd0 2023-11-28T20:48:16.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.567+0000 7fb5bb290640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b4157ff0 con 0x7fb5b406a500 2023-11-28T20:48:16.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.567+0000 7fb5b982d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5b415ddd0 0x7fb5b415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:16.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.567+0000 7fb5b982d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5b415ddd0 0x7fb5b415e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34340/0 (socket says 172.21.15.45:34340) 2023-11-28T20:48:16.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.567+0000 7fb5b982d640 1 -- 172.21.15.45:0/1614574820 learned_addr learned my addr 172.21.15.45:0/1614574820 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:16.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.567+0000 7fb5b882b640 1 --2- 172.21.15.45:0/1614574820 >> v2:172.21.15.190:3300/0 conn(0x7fb5b406a500 0x7fb5b406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:16.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.568+0000 7fb5b982d640 1 -- 172.21.15.45:0/1614574820 >> v2:172.21.15.190:3300/0 conn(0x7fb5b406a500 msgr2=0x7fb5b406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:16.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.568+0000 7fb5b982d640 1 --2- 172.21.15.45:0/1614574820 >> v2:172.21.15.190:3300/0 conn(0x7fb5b406a500 0x7fb5b406a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.568+0000 7fb5b982d640 1 -- 172.21.15.45:0/1614574820 >> v1:172.21.15.190:6789/0 conn(0x7fb5b4157c10 legacy=0x7fb5b406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:16.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.568+0000 7fb5b982d640 1 -- 172.21.15.45:0/1614574820 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b406aea0 con 0x7fb5b415ddd0 2023-11-28T20:48:16.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.569+0000 7fb5b982d640 1 --2- 172.21.15.45:0/1614574820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b415ddd0 0x7fb5b415e220 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40cfe40 tx=0x7fb5ac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad16beb2ac37435 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:16.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.569+0000 7fb5abfff640 1 -- 172.21.15.45:0/1614574820 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac02ca80 con 0x7fb5b415ddd0 2023-11-28T20:48:16.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5abfff640 1 -- 172.21.15.45:0/1614574820 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ac02cbe0 con 0x7fb5b415ddd0 2023-11-28T20:48:16.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5abfff640 1 -- 172.21.15.45:0/1614574820 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac0343f0 con 0x7fb5b415ddd0 2023-11-28T20:48:16.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5bb290640 1 -- 172.21.15.45:0/1614574820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b415ddd0 msgr2=0x7fb5b415e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:16.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5bb290640 1 --2- 172.21.15.45:0/1614574820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b415ddd0 0x7fb5b415e220 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40cfe40 tx=0x7fb5ac02bee0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5bb290640 1 -- 172.21.15.45:0/1614574820 shutdown_connections 2023-11-28T20:48:16.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5bb290640 1 --2- 172.21.15.45:0/1614574820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b415ddd0 0x7fb5b415e220 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5bb290640 1 --2- 172.21.15.45:0/1614574820 >> v2:172.21.15.190:3300/0 conn(0x7fb5b406a500 0x7fb5b406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5bb290640 1 -- 172.21.15.45:0/1614574820 >> 172.21.15.45:0/1614574820 conn(0x7fb5b40ced60 msgr2=0x7fb5b40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:16.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5bb290640 1 -- 172.21.15.45:0/1614574820 shutdown_connections 2023-11-28T20:48:16.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.570+0000 7fb5bb290640 1 -- 172.21.15.45:0/1614574820 wait complete. 2023-11-28T20:48:16.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.572+0000 7fb5bb290640 1 Processor -- start 2023-11-28T20:48:16.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.572+0000 7fb5bb290640 1 -- start start 2023-11-28T20:48:16.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.573+0000 7fb5bb290640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5b406a500 0x7fb5b414b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:16.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.573+0000 7fb5bb290640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b4157c10 0x7fb5b40c3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:16.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.573+0000 7fb5bb290640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5b415ddd0 0x7fb5b4159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:16.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.573+0000 7fb5bb290640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b415bb20 con 0x7fb5b4157c10 2023-11-28T20:48:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.573+0000 7fb5bb290640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b40c4e00 con 0x7fb5b415ddd0 2023-11-28T20:48:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.573+0000 7fb5bb290640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b415b7f0 con 0x7fb5b406a500 2023-11-28T20:48:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.573+0000 7fb5b882b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b4157c10 0x7fb5b40c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b882b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b4157c10 0x7fb5b40c3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34356/0 (socket says 172.21.15.45:34356) 2023-11-28T20:48:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b882b640 1 -- 172.21.15.45:0/1422458353 learned_addr learned my addr 172.21.15.45:0/1422458353 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b982d640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5b415ddd0 0x7fb5b4159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b902c640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5b406a500 0x7fb5b414b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b882b640 1 -- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5b406a500 msgr2=0x7fb5b414b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b882b640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5b406a500 0x7fb5b414b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b882b640 1 -- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5b415ddd0 msgr2=0x7fb5b4159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b882b640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5b415ddd0 0x7fb5b4159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b882b640 1 -- 172.21.15.45:0/1422458353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac009b30 con 0x7fb5b4157c10 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.574+0000 7fb5b882b640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b4157c10 0x7fb5b40c3210 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400d800 tx=0x7fb5a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.575+0000 7fb5a9ffb640 1 -- 172.21.15.45:0/1422458353 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400b6c0 con 0x7fb5b4157c10 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.575+0000 7fb5a9ffb640 1 -- 172.21.15.45:0/1422458353 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a400b820 con 0x7fb5b4157c10 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.575+0000 7fb5a9ffb640 1 -- 172.21.15.45:0/1422458353 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400bbb0 con 0x7fb5b4157c10 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.575+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b40c50a0 con 0x7fb5b4157c10 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.575+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b40c5660 con 0x7fb5b4157c10 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.576+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b415bcd0 con 0x7fb5b4157c10 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.577+0000 7fb5a9ffb640 1 -- 172.21.15.45:0/1422458353 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb5a400bd10 con 0x7fb5b4157c10 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.578+0000 7fb5a9ffb640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb59806af60 0x7fb59806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.578+0000 7fb5a9ffb640 1 -- 172.21.15.45:0/1422458353 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb5a4089e40 con 0x7fb5b4157c10 2023-11-28T20:48:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.579+0000 7fb5b902c640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb59806af60 0x7fb59806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.580+0000 7fb5b902c640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb59806af60 0x7fb59806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0003d90 tx=0x7fb5b0009040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:16.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.582+0000 7fb5a9ffb640 1 -- 172.21.15.45:0/1422458353 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5b415bcd0 con 0x7fb5b4157c10 2023-11-28T20:48:16.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:16.818+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb5b40733d0 con 0x7fb5b4157c10 2023-11-28T20:48:17.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.325+0000 7fb5a9ffb640 1 -- 172.21.15.45:0/1422458353 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb5a4054c90 con 0x7fb5b4157c10 2023-11-28T20:48:17.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb59806af60 msgr2=0x7fb59806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:17.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb59806af60 0x7fb59806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0003d90 tx=0x7fb5b0009040 comp rx=0 tx=0).stop 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b4157c10 msgr2=0x7fb5b40c3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b4157c10 0x7fb5b40c3210 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400d800 tx=0x7fb5a400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 shutdown_connections 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fb59806af60 0x7fb59806d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5b415ddd0 0x7fb5b4159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5b4157c10 0x7fb5b40c3210 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 --2- 172.21.15.45:0/1422458353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5b406a500 0x7fb5b414b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 >> 172.21.15.45:0/1422458353 conn(0x7fb5b40ced60 msgr2=0x7fb5b4070180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.329+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 shutdown_connections 2023-11-28T20:48:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.330+0000 7fb5bb290640 1 -- 172.21.15.45:0/1422458353 wait complete. 2023-11-28T20:48:17.354 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T20:48:17.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:48:17.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.568+0000 7f13aef51640 1 Processor -- start 2023-11-28T20:48:17.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.568+0000 7f13aef51640 1 -- start start 2023-11-28T20:48:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.570+0000 7f13aef51640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f13a80b0c30 0x7f13a80b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.570+0000 7f13aef51640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a815dcc0 con 0x7f13a80b3d60 2023-11-28T20:48:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.570+0000 7f13aef51640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a815e840 con 0x7f13a81438b0 2023-11-28T20:48:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.570+0000 7f13aef51640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a80b1620 con 0x7f13a80b0c30 2023-11-28T20:48:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.570+0000 7f13ad4ee640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f13a80b3d60 0x7f13a80b41b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58212/0 (socket says 172.21.15.45:58212) 2023-11-28T20:48:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.570+0000 7f13ad4ee640 1 -- 172.21.15.45:0/330538181 learned_addr learned my addr 172.21.15.45:0/330538181 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.570+0000 7f139ffff640 1 --2- 172.21.15.45:0/330538181 >> v2:172.21.15.190:3300/0 conn(0x7f13a80b0c30 0x7f13a80b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472123732 0 0) 0x7f13a815dcc0 con 0x7f13a80b3d60 2023-11-28T20:48:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1390005730 con 0x7f13a80b3d60 2023-11-28T20:48:17.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251489704 0 0) 0x7f1390005730 con 0x7f13a80b3d60 2023-11-28T20:48:17.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 >> v2:172.21.15.190:3300/0 conn(0x7f13a80b0c30 msgr2=0x7f13a80b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:17.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 --2- 172.21.15.45:0/330538181 >> v2:172.21.15.190:3300/0 conn(0x7f13a80b0c30 0x7f13a80b10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 >> v1:172.21.15.190:6789/0 conn(0x7f13a81438b0 legacy=0x7f13a80b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a80b26b0 con 0x7f13a80b3d60 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f13a00019f0 con 0x7f13a80b3d60 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 >> v1:172.21.15.45:6789/0 conn(0x7f13a80b3d60 legacy=0x7f13a80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1390005bd0 0x7f1390005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13900069d0 0x7f1390006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13900074b0 0x7f139000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80b1620 con 0x7f13900074b0 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139000e3b0 con 0x7f1390005bd0 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.571+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139000e570 con 0x7f13900069d0 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.572+0000 7f13ad4ee640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13900074b0 0x7f139000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.572+0000 7f13ad4ee640 1 -- 172.21.15.45:0/330538181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13900069d0 msgr2=0x7f1390006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.572+0000 7f13ad4ee640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13900069d0 0x7f1390006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.572+0000 7f13ad4ee640 1 -- 172.21.15.45:0/330538181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1390005bd0 msgr2=0x7f1390005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.572+0000 7f13acced640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1390005bd0 0x7f1390005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.572+0000 7f13ad4ee640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1390005bd0 0x7f1390005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.572+0000 7f13ad4ee640 1 -- 172.21.15.45:0/330538181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1390005730 con 0x7f13900074b0 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.572+0000 7f13acced640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1390005bd0 0x7f1390005ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.573+0000 7f13ad4ee640 1 -- 172.21.15.45:0/330538181 reap_dead start 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.573+0000 7f13ad4ee640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13900074b0 0x7f139000db90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f13a000a750 tx=0x7f1390007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0876e6394dc558 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.573+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0034630 con 0x7f13900074b0 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.573+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a002f070 con 0x7f13900074b0 2023-11-28T20:48:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.573+0000 7f139f7fe640 1 -- 172.21.15.45:0/330538181 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a00349d0 con 0x7f13900074b0 2023-11-28T20:48:17.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.574+0000 7f13aef51640 1 -- 172.21.15.45:0/330538181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13900074b0 msgr2=0x7f139000db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:17.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.574+0000 7f13aef51640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13900074b0 0x7f139000db90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f13a000a750 tx=0x7f1390007a00 comp rx=0 tx=0).stop 2023-11-28T20:48:17.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.574+0000 7f13aef51640 1 -- 172.21.15.45:0/330538181 shutdown_connections 2023-11-28T20:48:17.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.574+0000 7f13aef51640 1 --2- 172.21.15.45:0/330538181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13900074b0 0x7f139000db90 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.574+0000 7f13aef51640 1 -- 172.21.15.45:0/330538181 >> 172.21.15.45:0/330538181 conn(0x7f13a806a500 msgr2=0x7f13a806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.574+0000 7f13aef51640 1 -- 172.21.15.45:0/330538181 shutdown_connections 2023-11-28T20:48:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.574+0000 7f13aef51640 1 -- 172.21.15.45:0/330538181 wait complete. 2023-11-28T20:48:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.576+0000 7f13aef51640 1 Processor -- start 2023-11-28T20:48:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.576+0000 7f13aef51640 1 -- start start 2023-11-28T20:48:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.577+0000 7f13aef51640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13900074b0 0x7f13a8132ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.577+0000 7f13aef51640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13a81594f0 0x7f13a8130ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.577+0000 7f13aef51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a81311e0 0x7f13a81f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.577+0000 7f13aef51640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8132650 con 0x7f13a81311e0 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.577+0000 7f13aef51640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8131a40 con 0x7f13a81594f0 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.577+0000 7f13aef51640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8131d20 con 0x7f13900074b0 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.577+0000 7f13ad4ee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a81311e0 0x7f13a81f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.577+0000 7f13ad4ee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a81311e0 0x7f13a81f30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34372/0 (socket says 172.21.15.45:34372) 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.577+0000 7f13ad4ee640 1 -- 172.21.15.45:0/1041699416 learned_addr learned my addr 172.21.15.45:0/1041699416 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f13ad4ee640 1 -- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13900074b0 msgr2=0x7f13a8132ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f13ad4ee640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13900074b0 0x7f13a8132ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f13ad4ee640 1 -- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13a81594f0 msgr2=0x7f13a8130ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f13ad4ee640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13a81594f0 0x7f13a8130ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f13ad4ee640 1 -- 172.21.15.45:0/1041699416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a8132000 con 0x7f13a81311e0 2023-11-28T20:48:17.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f13ad4ee640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a81311e0 0x7f13a81f30b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f13a0000f80 tx=0x7f13a815e7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:17.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f139d7fa640 1 -- 172.21.15.45:0/1041699416 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0001940 con 0x7f13a81311e0 2023-11-28T20:48:17.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f139d7fa640 1 -- 172.21.15.45:0/1041699416 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a0004b60 con 0x7f13a81311e0 2023-11-28T20:48:17.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f139d7fa640 1 -- 172.21.15.45:0/1041699416 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0005030 con 0x7f13a81311e0 2023-11-28T20:48:17.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.578+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a81322e0 con 0x7f13a81311e0 2023-11-28T20:48:17.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.579+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a8131660 con 0x7f13a81311e0 2023-11-28T20:48:17.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.580+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a8132470 con 0x7f13a81311e0 2023-11-28T20:48:17.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.581+0000 7f139d7fa640 1 -- 172.21.15.45:0/1041699416 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f13a0031030 con 0x7f13a81311e0 2023-11-28T20:48:17.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.581+0000 7f139d7fa640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f139406af60 0x7f139406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:17.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.582+0000 7f139d7fa640 1 -- 172.21.15.45:0/1041699416 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f13a00ab350 con 0x7f13a81311e0 2023-11-28T20:48:17.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.582+0000 7f13acced640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f139406af60 0x7f139406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:17.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.583+0000 7f13acced640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f139406af60 0x7f139406d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f13a40016d0 tx=0x7f13a4009040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:17.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.586+0000 7f139d7fa640 1 -- 172.21.15.45:0/1041699416 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13a0076090 con 0x7f13a81311e0 2023-11-28T20:48:17.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.822+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13a8132470 con 0x7f13a81311e0 2023-11-28T20:48:17.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.823+0000 7f139d7fa640 1 -- 172.21.15.45:0/1041699416 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f13a00796b0 con 0x7f13a81311e0 2023-11-28T20:48:17.826 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:17.826 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5183,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/285399852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":285399852},{"type":"v1","addr":"172.21.15.190:6836","nonce":285399852}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2028545354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2028545354},{"type":"v1","addr":"172.21.15.190:6837","nonce":2028545354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6216,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/784320624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":784320624},{"type":"v1","addr":"172.21.15.45:6834","nonce":784320624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:48:13.742492+0000","modified":"2023-11-28T20:48:16.820308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3717491008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3717491008},{"type":"v1","addr":"172.21.15.45:6835","nonce":3717491008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T20:48:17.826 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 27 2023-11-28T20:48:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f139406af60 msgr2=0x7f139406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f139406af60 0x7f139406d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f13a40016d0 tx=0x7f13a4009040 comp rx=0 tx=0).stop 2023-11-28T20:48:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a81311e0 msgr2=0x7f13a81f30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a81311e0 0x7f13a81f30b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f13a0000f80 tx=0x7f13a815e7b0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 shutdown_connections 2023-11-28T20:48:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f139406af60 0x7f139406d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a81311e0 0x7f13a81f30b0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13a81594f0 0x7f13a8130ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 --2- 172.21.15.45:0/1041699416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13900074b0 0x7f13a8132ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 >> 172.21.15.45:0/1041699416 conn(0x7f13a806a500 msgr2=0x7f13a8141270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 shutdown_connections 2023-11-28T20:48:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:17.826+0000 7f13aef51640 1 -- 172.21.15.45:0/1041699416 wait complete. 2023-11-28T20:48:17.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:48:18.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.076+0000 7f949a8c3640 1 Processor -- start 2023-11-28T20:48:18.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.076+0000 7f949a8c3640 1 -- start start 2023-11-28T20:48:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.078+0000 7f949a8c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9494155780 0x7f94940abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.078+0000 7f949a8c3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f94940ae540 0x7f94940ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.078+0000 7f949a8c3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f94940ce630 0x7f949415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.078+0000 7f949a8c3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94940ceac0 con 0x7f94940ce630 2023-11-28T20:48:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.078+0000 7f949a8c3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f949415c3c0 con 0x7f94940ae540 2023-11-28T20:48:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.078+0000 7f949a8c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9494155b60 con 0x7f9494155780 2023-11-28T20:48:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.078+0000 7f949a0c2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f94940ce630 0x7f949415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.078+0000 7f949a0c2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f94940ce630 0x7f949415ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34386/0 (socket says 172.21.15.45:34386) 2023-11-28T20:48:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.078+0000 7f949a0c2640 1 -- 172.21.15.45:0/4242600406 learned_addr learned my addr 172.21.15.45:0/4242600406 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.079+0000 7f94998c1640 1 --2- 172.21.15.45:0/4242600406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9494155780 0x7f94940abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.079+0000 7f94990c0640 1 --2- 172.21.15.45:0/4242600406 >> v2:172.21.15.190:3300/0 conn(0x7f94940ae540 0x7f94940ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.079+0000 7f949a0c2640 1 -- 172.21.15.45:0/4242600406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9494155780 msgr2=0x7f94940abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.079+0000 7f949a0c2640 1 --2- 172.21.15.45:0/4242600406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9494155780 0x7f94940abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.079+0000 7f949a0c2640 1 -- 172.21.15.45:0/4242600406 >> v2:172.21.15.190:3300/0 conn(0x7f94940ae540 msgr2=0x7f94940ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.079+0000 7f949a0c2640 1 --2- 172.21.15.45:0/4242600406 >> v2:172.21.15.190:3300/0 conn(0x7f94940ae540 0x7f94940ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.079+0000 7f949a0c2640 1 -- 172.21.15.45:0/4242600406 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949415ba90 con 0x7f94940ce630 2023-11-28T20:48:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.079+0000 7f94990c0640 1 --2- 172.21.15.45:0/4242600406 >> v2:172.21.15.190:3300/0 conn(0x7f94940ae540 0x7f94940ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.080+0000 7f949a0c2640 1 --2- 172.21.15.45:0/4242600406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ce630 0x7f949415ed30 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f948c009e50 tx=0x7f948c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814aa92714262a70 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.080+0000 7f94988bf640 1 -- 172.21.15.45:0/4242600406 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c02ca70 con 0x7f94940ce630 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.080+0000 7f94988bf640 1 -- 172.21.15.45:0/4242600406 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948c02cbd0 con 0x7f94940ce630 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.080+0000 7f94988bf640 1 -- 172.21.15.45:0/4242600406 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c0343f0 con 0x7f94940ce630 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.081+0000 7f949a8c3640 1 -- 172.21.15.45:0/4242600406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ce630 msgr2=0x7f949415ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.081+0000 7f949a8c3640 1 --2- 172.21.15.45:0/4242600406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ce630 0x7f949415ed30 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f948c009e50 tx=0x7f948c02bed0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.081+0000 7f949a8c3640 1 -- 172.21.15.45:0/4242600406 shutdown_connections 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.081+0000 7f949a8c3640 1 --2- 172.21.15.45:0/4242600406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ce630 0x7f949415ed30 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.081+0000 7f949a8c3640 1 --2- 172.21.15.45:0/4242600406 >> v2:172.21.15.190:3300/0 conn(0x7f94940ae540 0x7f94940ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.081+0000 7f949a8c3640 1 --2- 172.21.15.45:0/4242600406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9494155780 0x7f94940abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.081+0000 7f949a8c3640 1 -- 172.21.15.45:0/4242600406 >> 172.21.15.45:0/4242600406 conn(0x7f949406ab30 msgr2=0x7f94940b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.081+0000 7f949a8c3640 1 -- 172.21.15.45:0/4242600406 shutdown_connections 2023-11-28T20:48:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.081+0000 7f949a8c3640 1 -- 172.21.15.45:0/4242600406 wait complete. 2023-11-28T20:48:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.083+0000 7f949a8c3640 1 Processor -- start 2023-11-28T20:48:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.083+0000 7f949a8c3640 1 -- start start 2023-11-28T20:48:18.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.084+0000 7f949a8c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ae540 0x7f9494148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.084+0000 7f949a8c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f94940ce630 0x7f9494070650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.084+0000 7f949a8c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9494155780 0x7f949406d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.084+0000 7f949a8c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9494150780 con 0x7f94940ae540 2023-11-28T20:48:18.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.084+0000 7f949a8c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94941509b0 con 0x7f9494155780 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.084+0000 7f949a8c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9494150c90 con 0x7f94940ce630 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.084+0000 7f94998c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ae540 0x7f9494148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.084+0000 7f94998c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ae540 0x7f9494148dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34394/0 (socket says 172.21.15.45:34394) 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.084+0000 7f94998c1640 1 -- 172.21.15.45:0/3121566813 learned_addr learned my addr 172.21.15.45:0/3121566813 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f94998c1640 1 -- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f94940ce630 msgr2=0x7f9494070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f94998c1640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f94940ce630 0x7f9494070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f94998c1640 1 -- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9494155780 msgr2=0x7f949406d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f94998c1640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9494155780 0x7f949406d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f94998c1640 1 -- 172.21.15.45:0/3121566813 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c009b30 con 0x7f94940ae540 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f94998c1640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ae540 0x7f9494148dc0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f949000d800 tx=0x7f949000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f948a7fc640 1 -- 172.21.15.45:0/3121566813 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949000b6c0 con 0x7f94940ae540 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f948a7fc640 1 -- 172.21.15.45:0/3121566813 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949000b820 con 0x7f94940ae540 2023-11-28T20:48:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f948a7fc640 1 -- 172.21.15.45:0/3121566813 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949000bbb0 con 0x7f94940ae540 2023-11-28T20:48:18.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.085+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9494150f70 con 0x7f94940ae540 2023-11-28T20:48:18.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.086+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9494151530 con 0x7f94940ae540 2023-11-28T20:48:18.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.087+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949404e920 con 0x7f94940ae540 2023-11-28T20:48:18.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.088+0000 7f948a7fc640 1 -- 172.21.15.45:0/3121566813 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f949000bd10 con 0x7f94940ae540 2023-11-28T20:48:18.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.089+0000 7f948a7fc640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f947806af60 0x7f947806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.089+0000 7f948a7fc640 1 -- 172.21.15.45:0/3121566813 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9490089e70 con 0x7f94940ae540 2023-11-28T20:48:18.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.090+0000 7f94990c0640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f947806af60 0x7f947806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.091+0000 7f94990c0640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f947806af60 0x7f947806d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9484006b10 tx=0x7f9484008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:18.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.093+0000 7f948a7fc640 1 -- 172.21.15.45:0/3121566813 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9490054cd0 con 0x7f94940ae540 2023-11-28T20:48:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.291+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94940b30b0 con 0x7f94940ae540 2023-11-28T20:48:18.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.295+0000 7f948a7fc640 1 -- 172.21.15.45:0/3121566813 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16762 (secure 0 0 0) 0x7f94900582f0 con 0x7f94940ae540 2023-11-28T20:48:18.297 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:18.297 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":71,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:48:15.250697+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:48:11.673148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-28T20:48:12.760782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:48:18.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f947806af60 msgr2=0x7f947806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f947806af60 0x7f947806d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9484006b10 tx=0x7f9484008040 comp rx=0 tx=0).stop 2023-11-28T20:48:18.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ae540 msgr2=0x7f9494148dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ae540 0x7f9494148dc0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f949000d800 tx=0x7f949000dcc0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 shutdown_connections 2023-11-28T20:48:18.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f947806af60 0x7f947806d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9494155780 0x7f949406d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f94940ce630 0x7f9494070650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 --2- 172.21.15.45:0/3121566813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94940ae540 0x7f9494148dc0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 >> 172.21.15.45:0/3121566813 conn(0x7f949406ab30 msgr2=0x7f949406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:18.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 shutdown_connections 2023-11-28T20:48:18.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.297+0000 7f949a8c3640 1 -- 172.21.15.45:0/3121566813 wait complete. 2023-11-28T20:48:18.318 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T20:48:18.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:48:18.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.545+0000 7f4837fff640 1 Processor -- start 2023-11-28T20:48:18.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.545+0000 7f4837fff640 1 -- start start 2023-11-28T20:48:18.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.547+0000 7f4837fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4838155780 0x7f48380abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.547+0000 7f4837fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f48380ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.547+0000 7f4837fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483815c3c0 con 0x7f48380ce630 2023-11-28T20:48:18.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.547+0000 7f4837fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48380ceac0 con 0x7f4838155780 2023-11-28T20:48:18.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.547+0000 7f4837fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4838155b60 con 0x7f48380ae540 2023-11-28T20:48:18.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.548+0000 7f482ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f48380ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.548+0000 7f4836ffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4838155780 0x7f48380abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.548+0000 7f482ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f48380ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:43108/0 (socket says 172.21.15.45:43108) 2023-11-28T20:48:18.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.548+0000 7f482ffff640 1 -- 172.21.15.45:0/1790769746 learned_addr learned my addr 172.21.15.45:0/1790769746 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:18.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.549+0000 7f482ffff640 1 -- 172.21.15.45:0/1790769746 >> v2:172.21.15.190:3300/0 conn(0x7f4838155780 msgr2=0x7f48380abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:18.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.549+0000 7f482ffff640 1 --2- 172.21.15.45:0/1790769746 >> v2:172.21.15.190:3300/0 conn(0x7f4838155780 0x7f48380abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.549+0000 7f482ffff640 1 -- 172.21.15.45:0/1790769746 >> v1:172.21.15.190:6789/0 conn(0x7f48380ce630 legacy=0x7f483815ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.549+0000 7f482ffff640 1 -- 172.21.15.45:0/1790769746 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483815ba90 con 0x7f48380ae540 2023-11-28T20:48:18.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.550+0000 7f482ffff640 1 --2- 172.21.15.45:0/1790769746 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f48380ce0f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f48200098a0 tx=0x7f482002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7bf1a62c7c1c7d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:18.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.550+0000 7f48367fc640 1 -- 172.21.15.45:0/1790769746 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4820030840 con 0x7f48380ae540 2023-11-28T20:48:18.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.551+0000 7f48367fc640 1 -- 172.21.15.45:0/1790769746 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48200309a0 con 0x7f48380ae540 2023-11-28T20:48:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.551+0000 7f48367fc640 1 -- 172.21.15.45:0/1790769746 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4820030d30 con 0x7f48380ae540 2023-11-28T20:48:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.551+0000 7f4837fff640 1 -- 172.21.15.45:0/1790769746 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 msgr2=0x7f48380ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.551+0000 7f4837fff640 1 --2- 172.21.15.45:0/1790769746 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f48380ce0f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f48200098a0 tx=0x7f482002b980 comp rx=0 tx=0).stop 2023-11-28T20:48:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.551+0000 7f4837fff640 1 -- 172.21.15.45:0/1790769746 shutdown_connections 2023-11-28T20:48:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.551+0000 7f4837fff640 1 --2- 172.21.15.45:0/1790769746 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f48380ce0f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.551+0000 7f4837fff640 1 --2- 172.21.15.45:0/1790769746 >> v2:172.21.15.190:3300/0 conn(0x7f4838155780 0x7f48380abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.551+0000 7f4837fff640 1 -- 172.21.15.45:0/1790769746 >> 172.21.15.45:0/1790769746 conn(0x7f483806ab30 msgr2=0x7f48380b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.551+0000 7f4837fff640 1 -- 172.21.15.45:0/1790769746 shutdown_connections 2023-11-28T20:48:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.552+0000 7f4837fff640 1 -- 172.21.15.45:0/1790769746 wait complete. 2023-11-28T20:48:18.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.553+0000 7f4837fff640 1 Processor -- start 2023-11-28T20:48:18.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.554+0000 7f4837fff640 1 -- start start 2023-11-28T20:48:18.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f4837fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f4838071e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f4837fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48380ce630 0x7f48380712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f4837fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4838155780 0x7f48381448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f4837fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4838070ea0 con 0x7f48380ce630 2023-11-28T20:48:18.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f4837fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4838144e00 con 0x7f4838155780 2023-11-28T20:48:18.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f4837fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4838070260 con 0x7f48380ae540 2023-11-28T20:48:18.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f482ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48380ce630 0x7f48380712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f482ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48380ce630 0x7f48380712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34398/0 (socket says 172.21.15.45:34398) 2023-11-28T20:48:18.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f482ffff640 1 -- 172.21.15.45:0/3907153785 learned_addr learned my addr 172.21.15.45:0/3907153785 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:18.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f4836ffd640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f4838071e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f482ffff640 1 -- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 msgr2=0x7f4838071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f482ffff640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f4838071e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f48377fe640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4838155780 0x7f48381448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f482ffff640 1 -- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4838155780 msgr2=0x7f48381448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.555+0000 7f482ffff640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4838155780 0x7f48381448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.556+0000 7f482ffff640 1 -- 172.21.15.45:0/3907153785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48200094d0 con 0x7f48380ce630 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.556+0000 7f482ffff640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48380ce630 0x7f48380712c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4820000c00 tx=0x7f4820004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.556+0000 7f4834ff9640 1 -- 172.21.15.45:0/3907153785 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4820039a90 con 0x7f48380ce630 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.556+0000 7f4834ff9640 1 -- 172.21.15.45:0/3907153785 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4820039c10 con 0x7f48380ce630 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.556+0000 7f4834ff9640 1 -- 172.21.15.45:0/3907153785 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4820043410 con 0x7f48380ce630 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.556+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4838070500 con 0x7f48380ce630 2023-11-28T20:48:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.556+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4838070ac0 con 0x7f48380ce630 2023-11-28T20:48:18.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.558+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483804e920 con 0x7f48380ce630 2023-11-28T20:48:18.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.558+0000 7f4834ff9640 1 -- 172.21.15.45:0/3907153785 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f482002f390 con 0x7f48380ce630 2023-11-28T20:48:18.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.559+0000 7f4834ff9640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f481806af60 0x7f481806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:18.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.559+0000 7f4834ff9640 1 -- 172.21.15.45:0/3907153785 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f48200abb40 con 0x7f48380ce630 2023-11-28T20:48:18.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.560+0000 7f4836ffd640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f481806af60 0x7f481806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:18.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.561+0000 7f4836ffd640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f481806af60 0x7f481806d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4828009ad0 tx=0x7f4828008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:18.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.564+0000 7f4834ff9640 1 -- 172.21.15.45:0/3907153785 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4820076910 con 0x7f48380ce630 2023-11-28T20:48:18.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.757+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 --> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f483815b740 con 0x7f481806af60 2023-11-28T20:48:18.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:48:18.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:48:18.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.788+0000 7f4834ff9640 1 -- 172.21.15.45:0/3907153785 <== mgr.5305 v2:172.21.15.190:6832/838517151 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286547 (secure 0 0 0) 0x7f48280119a0 con 0x7f481806af60 2023-11-28T20:48:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:18.796 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:48:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f481806af60 msgr2=0x7f481806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f481806af60 0x7f481806d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4828009ad0 tx=0x7f4828008040 comp rx=0 tx=0).stop 2023-11-28T20:48:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48380ce630 msgr2=0x7f48380712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48380ce630 0x7f48380712c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4820000c00 tx=0x7f4820004730 comp rx=0 tx=0).stop 2023-11-28T20:48:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 shutdown_connections 2023-11-28T20:48:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f481806af60 0x7f481806d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4838155780 0x7f48381448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48380ce630 0x7f48380712c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 --2- 172.21.15.45:0/3907153785 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48380ae540 0x7f4838071e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 >> 172.21.15.45:0/3907153785 conn(0x7f483806ab30 msgr2=0x7f48380b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 shutdown_connections 2023-11-28T20:48:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:18.793+0000 7f4837fff640 1 -- 172.21.15.45:0/3907153785 wait complete. 2023-11-28T20:48:18.815 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-11-28T20:48:17.609503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326692,"kb_used_data":7856,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754648028,"statfs":{"total":773094113280,"available":772759580672,"internally_reserved":0,"allocated":8044544,"data_stored":4559476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":326356528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.359097"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256814+0000","last_change":"2023-11-28T20:48:14.282138+0000","last_active":"2023-11-28T20:48:15.256814+0000","last_peered":"2023-11-28T20:48:15.256814+0000","last_clean":"2023-11-28T20:48:15.256814+0000","last_became_active":"2023-11-28T20:48:14.281794+0000","last_became_peered":"2023-11-28T20:48:14.281794+0000","last_unstale":"2023-11-28T20:48:15.256814+0000","last_undegraded":"2023-11-28T20:48:15.256814+0000","last_fullsized":"2023-11-28T20:48:15.256814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:50:20.379297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537473+0000","last_change":"2023-11-28T20:48:14.290895+0000","last_active":"2023-11-28T20:48:15.537473+0000","last_peered":"2023-11-28T20:48:15.537473+0000","last_clean":"2023-11-28T20:48:15.537473+0000","last_became_active":"2023-11-28T20:48:14.290535+0000","last_became_peered":"2023-11-28T20:48:14.290535+0000","last_unstale":"2023-11-28T20:48:15.537473+0000","last_undegraded":"2023-11-28T20:48:15.537473+0000","last_fullsized":"2023-11-28T20:48:15.537473+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:29:16.727745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292144+0000","last_change":"2023-11-28T20:48:14.286727+0000","last_active":"2023-11-28T20:48:14.292144+0000","last_peered":"2023-11-28T20:48:14.292144+0000","last_clean":"2023-11-28T20:48:14.292144+0000","last_became_active":"2023-11-28T20:48:14.285651+0000","last_became_peered":"2023-11-28T20:48:14.285651+0000","last_unstale":"2023-11-28T20:48:14.292144+0000","last_undegraded":"2023-11-28T20:48:14.292144+0000","last_fullsized":"2023-11-28T20:48:14.292144+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:21:13.104355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537424+0000","last_change":"2023-11-28T20:48:14.294399+0000","last_active":"2023-11-28T20:48:15.537424+0000","last_peered":"2023-11-28T20:48:15.537424+0000","last_clean":"2023-11-28T20:48:15.537424+0000","last_became_active":"2023-11-28T20:48:14.294137+0000","last_became_peered":"2023-11-28T20:48:14.294137+0000","last_unstale":"2023-11-28T20:48:15.537424+0000","last_undegraded":"2023-11-28T20:48:15.537424+0000","last_fullsized":"2023-11-28T20:48:15.537424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:05.552512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.262601+0000","last_change":"2023-11-28T20:48:14.276269+0000","last_active":"2023-11-28T20:48:15.262601+0000","last_peered":"2023-11-28T20:48:15.262601+0000","last_clean":"2023-11-28T20:48:15.262601+0000","last_became_active":"2023-11-28T20:48:14.275508+0000","last_became_peered":"2023-11-28T20:48:14.275508+0000","last_unstale":"2023-11-28T20:48:15.262601+0000","last_undegraded":"2023-11-28T20:48:15.262601+0000","last_fullsized":"2023-11-28T20:48:15.262601+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:24:16.445277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256077+0000","last_change":"2023-11-28T20:48:14.286697+0000","last_active":"2023-11-28T20:48:15.256077+0000","last_peered":"2023-11-28T20:48:15.256077+0000","last_clean":"2023-11-28T20:48:15.256077+0000","last_became_active":"2023-11-28T20:48:14.286411+0000","last_became_peered":"2023-11-28T20:48:14.286411+0000","last_unstale":"2023-11-28T20:48:15.256077+0000","last_undegraded":"2023-11-28T20:48:15.256077+0000","last_fullsized":"2023-11-28T20:48:15.256077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:22:28.619514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292132+0000","last_change":"2023-11-28T20:48:14.285906+0000","last_active":"2023-11-28T20:48:14.292132+0000","last_peered":"2023-11-28T20:48:14.292132+0000","last_clean":"2023-11-28T20:48:14.292132+0000","last_became_active":"2023-11-28T20:48:14.285382+0000","last_became_peered":"2023-11-28T20:48:14.285382+0000","last_unstale":"2023-11-28T20:48:14.292132+0000","last_undegraded":"2023-11-28T20:48:14.292132+0000","last_fullsized":"2023-11-28T20:48:14.292132+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:57:00.186060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258946+0000","last_change":"2023-11-28T20:48:14.297202+0000","last_active":"2023-11-28T20:48:15.258946+0000","last_peered":"2023-11-28T20:48:15.258946+0000","last_clean":"2023-11-28T20:48:15.258946+0000","last_became_active":"2023-11-28T20:48:14.296899+0000","last_became_peered":"2023-11-28T20:48:14.296899+0000","last_unstale":"2023-11-28T20:48:15.258946+0000","last_undegraded":"2023-11-28T20:48:15.258946+0000","last_fullsized":"2023-11-28T20:48:15.258946+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:54.614672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259609+0000","last_change":"2023-11-28T20:48:14.300760+0000","last_active":"2023-11-28T20:48:15.259609+0000","last_peered":"2023-11-28T20:48:15.259609+0000","last_clean":"2023-11-28T20:48:15.259609+0000","last_became_active":"2023-11-28T20:48:14.299658+0000","last_became_peered":"2023-11-28T20:48:14.299658+0000","last_unstale":"2023-11-28T20:48:15.259609+0000","last_undegraded":"2023-11-28T20:48:15.259609+0000","last_fullsized":"2023-11-28T20:48:15.259609+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:35.932070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256114+0000","last_change":"2023-11-28T20:48:14.271209+0000","last_active":"2023-11-28T20:48:15.256114+0000","last_peered":"2023-11-28T20:48:15.256114+0000","last_clean":"2023-11-28T20:48:15.256114+0000","last_became_active":"2023-11-28T20:48:14.270488+0000","last_became_peered":"2023-11-28T20:48:14.270488+0000","last_unstale":"2023-11-28T20:48:15.256114+0000","last_undegraded":"2023-11-28T20:48:15.256114+0000","last_fullsized":"2023-11-28T20:48:15.256114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:56.874613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.650119+0000","last_change":"2023-11-28T20:48:14.279333+0000","last_active":"2023-11-28T20:48:15.650119+0000","last_peered":"2023-11-28T20:48:15.650119+0000","last_clean":"2023-11-28T20:48:15.650119+0000","last_became_active":"2023-11-28T20:48:14.277790+0000","last_became_peered":"2023-11-28T20:48:14.277790+0000","last_unstale":"2023-11-28T20:48:15.650119+0000","last_undegraded":"2023-11-28T20:48:15.650119+0000","last_fullsized":"2023-11-28T20:48:15.650119+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:12:04.544981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259314+0000","last_change":"2023-11-28T20:48:14.291079+0000","last_active":"2023-11-28T20:48:15.259314+0000","last_peered":"2023-11-28T20:48:15.259314+0000","last_clean":"2023-11-28T20:48:15.259314+0000","last_became_active":"2023-11-28T20:48:14.290758+0000","last_became_peered":"2023-11-28T20:48:14.290758+0000","last_unstale":"2023-11-28T20:48:15.259314+0000","last_undegraded":"2023-11-28T20:48:15.259314+0000","last_fullsized":"2023-11-28T20:48:15.259314+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:59:53.415363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256073+0000","last_change":"2023-11-28T20:48:14.267972+0000","last_active":"2023-11-28T20:48:15.256073+0000","last_peered":"2023-11-28T20:48:15.256073+0000","last_clean":"2023-11-28T20:48:15.256073+0000","last_became_active":"2023-11-28T20:48:14.267199+0000","last_became_peered":"2023-11-28T20:48:14.267199+0000","last_unstale":"2023-11-28T20:48:15.256073+0000","last_undegraded":"2023-11-28T20:48:15.256073+0000","last_fullsized":"2023-11-28T20:48:15.256073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:36:24.424054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292715+0000","last_change":"2023-11-28T20:48:14.284015+0000","last_active":"2023-11-28T20:48:14.292715+0000","last_peered":"2023-11-28T20:48:14.292715+0000","last_clean":"2023-11-28T20:48:14.292715+0000","last_became_active":"2023-11-28T20:48:14.283678+0000","last_became_peered":"2023-11-28T20:48:14.283678+0000","last_unstale":"2023-11-28T20:48:14.292715+0000","last_undegraded":"2023-11-28T20:48:14.292715+0000","last_fullsized":"2023-11-28T20:48:14.292715+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:59:03.588833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.536937+0000","last_change":"2023-11-28T20:48:14.286266+0000","last_active":"2023-11-28T20:48:15.536937+0000","last_peered":"2023-11-28T20:48:15.536937+0000","last_clean":"2023-11-28T20:48:15.536937+0000","last_became_active":"2023-11-28T20:48:14.285759+0000","last_became_peered":"2023-11-28T20:48:14.285759+0000","last_unstale":"2023-11-28T20:48:15.536937+0000","last_undegraded":"2023-11-28T20:48:15.536937+0000","last_fullsized":"2023-11-28T20:48:15.536937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:19:15.212329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.255923+0000","last_change":"2023-11-28T20:48:14.276424+0000","last_active":"2023-11-28T20:48:15.255923+0000","last_peered":"2023-11-28T20:48:15.255923+0000","last_clean":"2023-11-28T20:48:15.255923+0000","last_became_active":"2023-11-28T20:48:14.275745+0000","last_became_peered":"2023-11-28T20:48:14.275745+0000","last_unstale":"2023-11-28T20:48:15.255923+0000","last_undegraded":"2023-11-28T20:48:15.255923+0000","last_fullsized":"2023-11-28T20:48:15.255923+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:15:23.890200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292240+0000","last_change":"2023-11-28T20:48:14.289296+0000","last_active":"2023-11-28T20:48:14.292240+0000","last_peered":"2023-11-28T20:48:14.292240+0000","last_clean":"2023-11-28T20:48:14.292240+0000","last_became_active":"2023-11-28T20:48:14.286635+0000","last_became_peered":"2023-11-28T20:48:14.286635+0000","last_unstale":"2023-11-28T20:48:14.292240+0000","last_undegraded":"2023-11-28T20:48:14.292240+0000","last_fullsized":"2023-11-28T20:48:14.292240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:40:48.337896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537619+0000","last_change":"2023-11-28T20:48:14.292874+0000","last_active":"2023-11-28T20:48:15.537619+0000","last_peered":"2023-11-28T20:48:15.537619+0000","last_clean":"2023-11-28T20:48:15.537619+0000","last_became_active":"2023-11-28T20:48:14.292571+0000","last_became_peered":"2023-11-28T20:48:14.292571+0000","last_unstale":"2023-11-28T20:48:15.537619+0000","last_undegraded":"2023-11-28T20:48:15.537619+0000","last_fullsized":"2023-11-28T20:48:15.537619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:18.178088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.302193+0000","last_change":"2023-11-28T20:48:14.299861+0000","last_active":"2023-11-28T20:48:14.302193+0000","last_peered":"2023-11-28T20:48:14.302193+0000","last_clean":"2023-11-28T20:48:14.302193+0000","last_became_active":"2023-11-28T20:48:14.298268+0000","last_became_peered":"2023-11-28T20:48:14.298268+0000","last_unstale":"2023-11-28T20:48:14.302193+0000","last_undegraded":"2023-11-28T20:48:14.302193+0000","last_fullsized":"2023-11-28T20:48:14.302193+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:14:49.829548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568771+0000","last_change":"2023-11-28T20:48:14.282949+0000","last_active":"2023-11-28T20:48:15.568771+0000","last_peered":"2023-11-28T20:48:15.568771+0000","last_clean":"2023-11-28T20:48:15.568771+0000","last_became_active":"2023-11-28T20:48:14.281952+0000","last_became_peered":"2023-11-28T20:48:14.281952+0000","last_unstale":"2023-11-28T20:48:15.568771+0000","last_undegraded":"2023-11-28T20:48:15.568771+0000","last_fullsized":"2023-11-28T20:48:15.568771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:17:35.465307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568846+0000","last_change":"2023-11-28T20:48:14.277401+0000","last_active":"2023-11-28T20:48:15.568846+0000","last_peered":"2023-11-28T20:48:15.568846+0000","last_clean":"2023-11-28T20:48:15.568846+0000","last_became_active":"2023-11-28T20:48:14.277001+0000","last_became_peered":"2023-11-28T20:48:14.277001+0000","last_unstale":"2023-11-28T20:48:15.568846+0000","last_undegraded":"2023-11-28T20:48:15.568846+0000","last_fullsized":"2023-11-28T20:48:15.568846+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:44:53.943332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258889+0000","last_change":"2023-11-28T20:48:14.297400+0000","last_active":"2023-11-28T20:48:15.258889+0000","last_peered":"2023-11-28T20:48:15.258889+0000","last_clean":"2023-11-28T20:48:15.258889+0000","last_became_active":"2023-11-28T20:48:14.297076+0000","last_became_peered":"2023-11-28T20:48:14.297076+0000","last_unstale":"2023-11-28T20:48:15.258889+0000","last_undegraded":"2023-11-28T20:48:15.258889+0000","last_fullsized":"2023-11-28T20:48:15.258889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:08.600530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.650505+0000","last_change":"2023-11-28T20:48:14.286000+0000","last_active":"2023-11-28T20:48:15.650505+0000","last_peered":"2023-11-28T20:48:15.650505+0000","last_clean":"2023-11-28T20:48:15.650505+0000","last_became_active":"2023-11-28T20:48:14.285692+0000","last_became_peered":"2023-11-28T20:48:14.285692+0000","last_unstale":"2023-11-28T20:48:15.650505+0000","last_undegraded":"2023-11-28T20:48:15.650505+0000","last_fullsized":"2023-11-28T20:48:15.650505+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:07:38.436914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292603+0000","last_change":"2023-11-28T20:48:14.277523+0000","last_active":"2023-11-28T20:48:14.292603+0000","last_peered":"2023-11-28T20:48:14.292603+0000","last_clean":"2023-11-28T20:48:14.292603+0000","last_became_active":"2023-11-28T20:48:14.277236+0000","last_became_peered":"2023-11-28T20:48:14.277236+0000","last_unstale":"2023-11-28T20:48:14.292603+0000","last_undegraded":"2023-11-28T20:48:14.292603+0000","last_fullsized":"2023-11-28T20:48:14.292603+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:38:11.116646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.298835+0000","last_change":"2023-11-28T20:48:14.295399+0000","last_active":"2023-11-28T20:48:14.298835+0000","last_peered":"2023-11-28T20:48:14.298835+0000","last_clean":"2023-11-28T20:48:14.298835+0000","last_became_active":"2023-11-28T20:48:14.294890+0000","last_became_peered":"2023-11-28T20:48:14.294890+0000","last_unstale":"2023-11-28T20:48:14.298835+0000","last_undegraded":"2023-11-28T20:48:14.298835+0000","last_fullsized":"2023-11-28T20:48:14.298835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:28.406912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259377+0000","last_change":"2023-11-28T20:48:14.289877+0000","last_active":"2023-11-28T20:48:15.259377+0000","last_peered":"2023-11-28T20:48:15.259377+0000","last_clean":"2023-11-28T20:48:15.259377+0000","last_became_active":"2023-11-28T20:48:14.289573+0000","last_became_peered":"2023-11-28T20:48:14.289573+0000","last_unstale":"2023-11-28T20:48:15.259377+0000","last_undegraded":"2023-11-28T20:48:15.259377+0000","last_fullsized":"2023-11-28T20:48:15.259377+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:57.242902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256526+0000","last_change":"2023-11-28T20:48:14.282362+0000","last_active":"2023-11-28T20:48:15.256526+0000","last_peered":"2023-11-28T20:48:15.256526+0000","last_clean":"2023-11-28T20:48:15.256526+0000","last_became_active":"2023-11-28T20:48:14.281383+0000","last_became_peered":"2023-11-28T20:48:14.281383+0000","last_unstale":"2023-11-28T20:48:15.256526+0000","last_undegraded":"2023-11-28T20:48:15.256526+0000","last_fullsized":"2023-11-28T20:48:15.256526+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:51.256993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568854+0000","last_change":"2023-11-28T20:48:14.283953+0000","last_active":"2023-11-28T20:48:15.568854+0000","last_peered":"2023-11-28T20:48:15.568854+0000","last_clean":"2023-11-28T20:48:15.568854+0000","last_became_active":"2023-11-28T20:48:14.283674+0000","last_became_peered":"2023-11-28T20:48:14.283674+0000","last_unstale":"2023-11-28T20:48:15.568854+0000","last_undegraded":"2023-11-28T20:48:15.568854+0000","last_fullsized":"2023-11-28T20:48:15.568854+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:30:53.807023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.301497+0000","last_change":"2023-11-28T20:48:14.298353+0000","last_active":"2023-11-28T20:48:14.301497+0000","last_peered":"2023-11-28T20:48:14.301497+0000","last_clean":"2023-11-28T20:48:14.301497+0000","last_became_active":"2023-11-28T20:48:14.297946+0000","last_became_peered":"2023-11-28T20:48:14.297946+0000","last_unstale":"2023-11-28T20:48:14.301497+0000","last_undegraded":"2023-11-28T20:48:14.301497+0000","last_fullsized":"2023-11-28T20:48:14.301497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:19:15.637568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537114+0000","last_change":"2023-11-28T20:48:14.288751+0000","last_active":"2023-11-28T20:48:15.537114+0000","last_peered":"2023-11-28T20:48:15.537114+0000","last_clean":"2023-11-28T20:48:15.537114+0000","last_became_active":"2023-11-28T20:48:14.288459+0000","last_became_peered":"2023-11-28T20:48:14.288459+0000","last_unstale":"2023-11-28T20:48:15.537114+0000","last_undegraded":"2023-11-28T20:48:15.537114+0000","last_fullsized":"2023-11-28T20:48:15.537114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:01.569483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292111+0000","last_change":"2023-11-28T20:48:14.285700+0000","last_active":"2023-11-28T20:48:14.292111+0000","last_peered":"2023-11-28T20:48:14.292111+0000","last_clean":"2023-11-28T20:48:14.292111+0000","last_became_active":"2023-11-28T20:48:14.284963+0000","last_became_peered":"2023-11-28T20:48:14.284963+0000","last_unstale":"2023-11-28T20:48:14.292111+0000","last_undegraded":"2023-11-28T20:48:14.292111+0000","last_fullsized":"2023-11-28T20:48:14.292111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:36:16.048653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.297515+0000","last_change":"2023-11-28T20:48:14.292228+0000","last_active":"2023-11-28T20:48:14.297515+0000","last_peered":"2023-11-28T20:48:14.297515+0000","last_clean":"2023-11-28T20:48:14.297515+0000","last_became_active":"2023-11-28T20:48:14.291887+0000","last_became_peered":"2023-11-28T20:48:14.291887+0000","last_unstale":"2023-11-28T20:48:14.297515+0000","last_undegraded":"2023-11-28T20:48:14.297515+0000","last_fullsized":"2023-11-28T20:48:14.297515+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:40:45.271219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.536722+0000","last_change":"2023-11-28T20:48:14.287511+0000","last_active":"2023-11-28T20:48:15.536722+0000","last_peered":"2023-11-28T20:48:15.536722+0000","last_clean":"2023-11-28T20:48:15.536722+0000","last_became_active":"2023-11-28T20:48:14.287228+0000","last_became_peered":"2023-11-28T20:48:14.287228+0000","last_unstale":"2023-11-28T20:48:15.536722+0000","last_undegraded":"2023-11-28T20:48:15.536722+0000","last_fullsized":"2023-11-28T20:48:15.536722+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:59:59.225574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"70'4","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568760+0000","last_change":"2023-11-28T20:48:13.287357+0000","last_active":"2023-11-28T20:48:15.568760+0000","last_peered":"2023-11-28T20:48:15.568760+0000","last_clean":"2023-11-28T20:48:15.568760+0000","last_became_active":"2023-11-28T20:48:13.286961+0000","last_became_peered":"2023-11-28T20:48:13.286961+0000","last_unstale":"2023-11-28T20:48:15.568760+0000","last_undegraded":"2023-11-28T20:48:15.568760+0000","last_fullsized":"2023-11-28T20:48:15.568760+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:22.499460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.287786+0000","last_change":"2023-11-28T20:48:14.281605+0000","last_active":"2023-11-28T20:48:14.287786+0000","last_peered":"2023-11-28T20:48:14.287786+0000","last_clean":"2023-11-28T20:48:14.287786+0000","last_became_active":"2023-11-28T20:48:14.281286+0000","last_became_peered":"2023-11-28T20:48:14.281286+0000","last_unstale":"2023-11-28T20:48:14.287786+0000","last_undegraded":"2023-11-28T20:48:14.287786+0000","last_fullsized":"2023-11-28T20:48:14.287786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:19.149186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"70'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256726+0000","last_change":"2023-11-28T20:48:13.286953+0000","last_active":"2023-11-28T20:48:15.256726+0000","last_peered":"2023-11-28T20:48:15.256726+0000","last_clean":"2023-11-28T20:48:15.256726+0000","last_became_active":"2023-11-28T20:48:13.286689+0000","last_became_peered":"2023-11-28T20:48:13.286689+0000","last_unstale":"2023-11-28T20:48:15.256726+0000","last_undegraded":"2023-11-28T20:48:15.256726+0000","last_fullsized":"2023-11-28T20:48:15.256726+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:36.638766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568500+0000","last_change":"2023-11-28T20:48:14.281744+0000","last_active":"2023-11-28T20:48:15.568500+0000","last_peered":"2023-11-28T20:48:15.568500+0000","last_clean":"2023-11-28T20:48:15.568500+0000","last_became_active":"2023-11-28T20:48:14.280512+0000","last_became_peered":"2023-11-28T20:48:14.280512+0000","last_unstale":"2023-11-28T20:48:15.568500+0000","last_undegraded":"2023-11-28T20:48:15.568500+0000","last_fullsized":"2023-11-28T20:48:15.568500+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:13.400093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"70'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256655+0000","last_change":"2023-11-28T20:48:13.289848+0000","last_active":"2023-11-28T20:48:15.256655+0000","last_peered":"2023-11-28T20:48:15.256655+0000","last_clean":"2023-11-28T20:48:15.256655+0000","last_became_active":"2023-11-28T20:48:13.289551+0000","last_became_peered":"2023-11-28T20:48:13.289551+0000","last_unstale":"2023-11-28T20:48:15.256655+0000","last_undegraded":"2023-11-28T20:48:15.256655+0000","last_fullsized":"2023-11-28T20:48:15.256655+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:30:52.792140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258951+0000","last_change":"2023-11-28T20:48:14.288614+0000","last_active":"2023-11-28T20:48:15.258951+0000","last_peered":"2023-11-28T20:48:15.258951+0000","last_clean":"2023-11-28T20:48:15.258951+0000","last_became_active":"2023-11-28T20:48:14.287369+0000","last_became_peered":"2023-11-28T20:48:14.287369+0000","last_unstale":"2023-11-28T20:48:15.258951+0000","last_undegraded":"2023-11-28T20:48:15.258951+0000","last_fullsized":"2023-11-28T20:48:15.258951+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:33:23.202275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"70'2","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.319021+0000","last_change":"2023-11-28T20:48:13.287229+0000","last_active":"2023-11-28T20:48:14.319021+0000","last_peered":"2023-11-28T20:48:14.319021+0000","last_clean":"2023-11-28T20:48:14.319021+0000","last_became_active":"2023-11-28T20:48:13.286968+0000","last_became_peered":"2023-11-28T20:48:13.286968+0000","last_unstale":"2023-11-28T20:48:14.319021+0000","last_undegraded":"2023-11-28T20:48:14.319021+0000","last_fullsized":"2023-11-28T20:48:14.319021+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:29:16.329450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.262381+0000","last_change":"2023-11-28T20:48:14.280379+0000","last_active":"2023-11-28T20:48:15.262381+0000","last_peered":"2023-11-28T20:48:15.262381+0000","last_clean":"2023-11-28T20:48:15.262381+0000","last_became_active":"2023-11-28T20:48:14.279313+0000","last_became_peered":"2023-11-28T20:48:14.279313+0000","last_unstale":"2023-11-28T20:48:15.262381+0000","last_undegraded":"2023-11-28T20:48:15.262381+0000","last_fullsized":"2023-11-28T20:48:15.262381+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:55:36.332492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"70'1","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.315850+0000","last_change":"2023-11-28T20:48:13.295973+0000","last_active":"2023-11-28T20:48:14.315850+0000","last_peered":"2023-11-28T20:48:14.315850+0000","last_clean":"2023-11-28T20:48:14.315850+0000","last_became_active":"2023-11-28T20:48:13.295690+0000","last_became_peered":"2023-11-28T20:48:13.295690+0000","last_unstale":"2023-11-28T20:48:14.315850+0000","last_undegraded":"2023-11-28T20:48:14.315850+0000","last_fullsized":"2023-11-28T20:48:14.315850+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:42:16.982555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537696+0000","last_change":"2023-11-28T20:48:14.290004+0000","last_active":"2023-11-28T20:48:15.537696+0000","last_peered":"2023-11-28T20:48:15.537696+0000","last_clean":"2023-11-28T20:48:15.537696+0000","last_became_active":"2023-11-28T20:48:14.289697+0000","last_became_peered":"2023-11-28T20:48:14.289697+0000","last_unstale":"2023-11-28T20:48:15.537696+0000","last_undegraded":"2023-11-28T20:48:15.537696+0000","last_fullsized":"2023-11-28T20:48:15.537696+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:01:42.653989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"70'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261599+0000","last_change":"2023-11-28T20:48:13.291137+0000","last_active":"2023-11-28T20:48:15.261599+0000","last_peered":"2023-11-28T20:48:15.261599+0000","last_clean":"2023-11-28T20:48:15.261599+0000","last_became_active":"2023-11-28T20:48:13.290749+0000","last_became_peered":"2023-11-28T20:48:13.290749+0000","last_unstale":"2023-11-28T20:48:15.261599+0000","last_undegraded":"2023-11-28T20:48:15.261599+0000","last_fullsized":"2023-11-28T20:48:15.261599+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:59.759169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259534+0000","last_change":"2023-11-28T20:48:14.287523+0000","last_active":"2023-11-28T20:48:15.259534+0000","last_peered":"2023-11-28T20:48:15.259534+0000","last_clean":"2023-11-28T20:48:15.259534+0000","last_became_active":"2023-11-28T20:48:14.287214+0000","last_became_peered":"2023-11-28T20:48:14.287214+0000","last_unstale":"2023-11-28T20:48:15.259534+0000","last_undegraded":"2023-11-28T20:48:15.259534+0000","last_fullsized":"2023-11-28T20:48:15.259534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:24:51.192998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568365+0000","last_change":"2023-11-28T20:48:13.294888+0000","last_active":"2023-11-28T20:48:15.568365+0000","last_peered":"2023-11-28T20:48:15.568365+0000","last_clean":"2023-11-28T20:48:15.568365+0000","last_became_active":"2023-11-28T20:48:13.293937+0000","last_became_peered":"2023-11-28T20:48:13.293937+0000","last_unstale":"2023-11-28T20:48:15.568365+0000","last_undegraded":"2023-11-28T20:48:15.568365+0000","last_fullsized":"2023-11-28T20:48:15.568365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:46:31.679930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537423+0000","last_change":"2023-11-28T20:48:14.293165+0000","last_active":"2023-11-28T20:48:15.537423+0000","last_peered":"2023-11-28T20:48:15.537423+0000","last_clean":"2023-11-28T20:48:15.537423+0000","last_became_active":"2023-11-28T20:48:14.292979+0000","last_became_peered":"2023-11-28T20:48:14.292979+0000","last_unstale":"2023-11-28T20:48:15.537423+0000","last_undegraded":"2023-11-28T20:48:15.537423+0000","last_fullsized":"2023-11-28T20:48:15.537423+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:41:43.088862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.293861+0000","last_change":"2023-11-28T20:48:13.284360+0000","last_active":"2023-11-28T20:48:14.293861+0000","last_peered":"2023-11-28T20:48:14.293861+0000","last_clean":"2023-11-28T20:48:14.293861+0000","last_became_active":"2023-11-28T20:48:13.283969+0000","last_became_peered":"2023-11-28T20:48:13.283969+0000","last_unstale":"2023-11-28T20:48:14.293861+0000","last_undegraded":"2023-11-28T20:48:14.293861+0000","last_fullsized":"2023-11-28T20:48:14.293861+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:25:13.938748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261391+0000","last_change":"2023-11-28T20:48:14.286992+0000","last_active":"2023-11-28T20:48:15.261391+0000","last_peered":"2023-11-28T20:48:15.261391+0000","last_clean":"2023-11-28T20:48:15.261391+0000","last_became_active":"2023-11-28T20:48:14.286691+0000","last_became_peered":"2023-11-28T20:48:14.286691+0000","last_unstale":"2023-11-28T20:48:15.261391+0000","last_undegraded":"2023-11-28T20:48:15.261391+0000","last_fullsized":"2023-11-28T20:48:15.261391+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:50:20.379297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537658+0000","last_change":"2023-11-28T20:48:13.294632+0000","last_active":"2023-11-28T20:48:15.537658+0000","last_peered":"2023-11-28T20:48:15.537658+0000","last_clean":"2023-11-28T20:48:15.537658+0000","last_became_active":"2023-11-28T20:48:13.294324+0000","last_became_peered":"2023-11-28T20:48:13.294324+0000","last_unstale":"2023-11-28T20:48:15.537658+0000","last_undegraded":"2023-11-28T20:48:15.537658+0000","last_fullsized":"2023-11-28T20:48:15.537658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:46:41.545296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.287768+0000","last_change":"2023-11-28T20:48:14.281436+0000","last_active":"2023-11-28T20:48:14.287768+0000","last_peered":"2023-11-28T20:48:14.287768+0000","last_clean":"2023-11-28T20:48:14.287768+0000","last_became_active":"2023-11-28T20:48:14.281119+0000","last_became_peered":"2023-11-28T20:48:14.281119+0000","last_unstale":"2023-11-28T20:48:14.287768+0000","last_undegraded":"2023-11-28T20:48:14.287768+0000","last_fullsized":"2023-11-28T20:48:14.287768+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:20:38.804327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261888+0000","last_change":"2023-11-28T20:48:13.286426+0000","last_active":"2023-11-28T20:48:15.261888+0000","last_peered":"2023-11-28T20:48:15.261888+0000","last_clean":"2023-11-28T20:48:15.261888+0000","last_became_active":"2023-11-28T20:48:13.286100+0000","last_became_peered":"2023-11-28T20:48:13.286100+0000","last_unstale":"2023-11-28T20:48:15.261888+0000","last_undegraded":"2023-11-28T20:48:15.261888+0000","last_fullsized":"2023-11-28T20:48:15.261888+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:31:44.021298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.302317+0000","last_change":"2023-11-28T20:48:14.300075+0000","last_active":"2023-11-28T20:48:14.302317+0000","last_peered":"2023-11-28T20:48:14.302317+0000","last_clean":"2023-11-28T20:48:14.302317+0000","last_became_active":"2023-11-28T20:48:14.299796+0000","last_became_peered":"2023-11-28T20:48:14.299796+0000","last_unstale":"2023-11-28T20:48:14.302317+0000","last_undegraded":"2023-11-28T20:48:14.302317+0000","last_fullsized":"2023-11-28T20:48:14.302317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:24:17.747646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"70'3","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537780+0000","last_change":"2023-11-28T20:48:13.296795+0000","last_active":"2023-11-28T20:48:15.537780+0000","last_peered":"2023-11-28T20:48:15.537780+0000","last_clean":"2023-11-28T20:48:15.537780+0000","last_became_active":"2023-11-28T20:48:13.296512+0000","last_became_peered":"2023-11-28T20:48:13.296512+0000","last_unstale":"2023-11-28T20:48:15.537780+0000","last_undegraded":"2023-11-28T20:48:15.537780+0000","last_fullsized":"2023-11-28T20:48:15.537780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:18.134304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.262780+0000","last_change":"2023-11-28T20:48:14.270755+0000","last_active":"2023-11-28T20:48:15.262780+0000","last_peered":"2023-11-28T20:48:15.262780+0000","last_clean":"2023-11-28T20:48:15.262780+0000","last_became_active":"2023-11-28T20:48:14.269949+0000","last_became_peered":"2023-11-28T20:48:14.269949+0000","last_unstale":"2023-11-28T20:48:15.262780+0000","last_undegraded":"2023-11-28T20:48:15.262780+0000","last_fullsized":"2023-11-28T20:48:15.262780+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:10:43.157562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"70'6","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259314+0000","last_change":"2023-11-28T20:48:13.269047+0000","last_active":"2023-11-28T20:48:15.259314+0000","last_peered":"2023-11-28T20:48:15.259314+0000","last_clean":"2023-11-28T20:48:15.259314+0000","last_became_active":"2023-11-28T20:48:13.268517+0000","last_became_peered":"2023-11-28T20:48:13.268517+0000","last_unstale":"2023-11-28T20:48:15.259314+0000","last_undegraded":"2023-11-28T20:48:15.259314+0000","last_fullsized":"2023-11-28T20:48:15.259314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:19:14.197446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.291945+0000","last_change":"2023-11-28T20:48:14.286659+0000","last_active":"2023-11-28T20:48:14.291945+0000","last_peered":"2023-11-28T20:48:14.291945+0000","last_clean":"2023-11-28T20:48:14.291945+0000","last_became_active":"2023-11-28T20:48:14.286132+0000","last_became_peered":"2023-11-28T20:48:14.286132+0000","last_unstale":"2023-11-28T20:48:14.291945+0000","last_undegraded":"2023-11-28T20:48:14.291945+0000","last_fullsized":"2023-11-28T20:48:14.291945+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:58:59.988274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"70'2","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.316470+0000","last_change":"2023-11-28T20:48:13.295480+0000","last_active":"2023-11-28T20:48:14.316470+0000","last_peered":"2023-11-28T20:48:14.316470+0000","last_clean":"2023-11-28T20:48:14.316470+0000","last_became_active":"2023-11-28T20:48:13.294923+0000","last_became_peered":"2023-11-28T20:48:13.294923+0000","last_unstale":"2023-11-28T20:48:14.316470+0000","last_undegraded":"2023-11-28T20:48:14.316470+0000","last_fullsized":"2023-11-28T20:48:14.316470+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:10.011402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.302246+0000","last_change":"2023-11-28T20:48:14.300360+0000","last_active":"2023-11-28T20:48:14.302246+0000","last_peered":"2023-11-28T20:48:14.302246+0000","last_clean":"2023-11-28T20:48:14.302246+0000","last_became_active":"2023-11-28T20:48:14.299480+0000","last_became_peered":"2023-11-28T20:48:14.299480+0000","last_unstale":"2023-11-28T20:48:14.302246+0000","last_undegraded":"2023-11-28T20:48:14.302246+0000","last_fullsized":"2023-11-28T20:48:14.302246+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:05:15.443215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568708+0000","last_change":"2023-11-28T20:48:13.291841+0000","last_active":"2023-11-28T20:48:15.568708+0000","last_peered":"2023-11-28T20:48:15.568708+0000","last_clean":"2023-11-28T20:48:15.568708+0000","last_became_active":"2023-11-28T20:48:13.291534+0000","last_became_peered":"2023-11-28T20:48:13.291534+0000","last_unstale":"2023-11-28T20:48:15.568708+0000","last_undegraded":"2023-11-28T20:48:15.568708+0000","last_fullsized":"2023-11-28T20:48:15.568708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:20:34.016942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259410+0000","last_change":"2023-11-28T20:48:14.291725+0000","last_active":"2023-11-28T20:48:15.259410+0000","last_peered":"2023-11-28T20:48:15.259410+0000","last_clean":"2023-11-28T20:48:15.259410+0000","last_became_active":"2023-11-28T20:48:14.290802+0000","last_became_peered":"2023-11-28T20:48:14.290802+0000","last_unstale":"2023-11-28T20:48:15.259410+0000","last_undegraded":"2023-11-28T20:48:15.259410+0000","last_fullsized":"2023-11-28T20:48:15.259410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.558006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.536992+0000","last_change":"2023-11-28T20:48:13.294280+0000","last_active":"2023-11-28T20:48:15.536992+0000","last_peered":"2023-11-28T20:48:15.536992+0000","last_clean":"2023-11-28T20:48:15.536992+0000","last_became_active":"2023-11-28T20:48:13.293809+0000","last_became_peered":"2023-11-28T20:48:13.293809+0000","last_unstale":"2023-11-28T20:48:15.536992+0000","last_undegraded":"2023-11-28T20:48:15.536992+0000","last_fullsized":"2023-11-28T20:48:15.536992+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:40:44.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292257+0000","last_change":"2023-11-28T20:48:14.289490+0000","last_active":"2023-11-28T20:48:14.292257+0000","last_peered":"2023-11-28T20:48:14.292257+0000","last_clean":"2023-11-28T20:48:14.292257+0000","last_became_active":"2023-11-28T20:48:14.286645+0000","last_became_peered":"2023-11-28T20:48:14.286645+0000","last_unstale":"2023-11-28T20:48:14.292257+0000","last_undegraded":"2023-11-28T20:48:14.292257+0000","last_fullsized":"2023-11-28T20:48:14.292257+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:19:50.350147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"70'8","reported_seq":20,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259107+0000","last_change":"2023-11-28T20:48:13.271079+0000","last_active":"2023-11-28T20:48:15.259107+0000","last_peered":"2023-11-28T20:48:15.259107+0000","last_clean":"2023-11-28T20:48:15.259107+0000","last_became_active":"2023-11-28T20:48:13.270367+0000","last_became_peered":"2023-11-28T20:48:13.270367+0000","last_unstale":"2023-11-28T20:48:15.259107+0000","last_undegraded":"2023-11-28T20:48:15.259107+0000","last_fullsized":"2023-11-28T20:48:15.259107+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:45:19.387368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259071+0000","last_change":"2023-11-28T20:48:14.289522+0000","last_active":"2023-11-28T20:48:15.259071+0000","last_peered":"2023-11-28T20:48:15.259071+0000","last_clean":"2023-11-28T20:48:15.259071+0000","last_became_active":"2023-11-28T20:48:14.286425+0000","last_became_peered":"2023-11-28T20:48:14.286425+0000","last_unstale":"2023-11-28T20:48:15.259071+0000","last_undegraded":"2023-11-28T20:48:15.259071+0000","last_fullsized":"2023-11-28T20:48:15.259071+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:55.537941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"70'1","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.316030+0000","last_change":"2023-11-28T20:48:13.290621+0000","last_active":"2023-11-28T20:48:14.316030+0000","last_peered":"2023-11-28T20:48:14.316030+0000","last_clean":"2023-11-28T20:48:14.316030+0000","last_became_active":"2023-11-28T20:48:13.290328+0000","last_became_peered":"2023-11-28T20:48:13.290328+0000","last_unstale":"2023-11-28T20:48:14.316030+0000","last_undegraded":"2023-11-28T20:48:14.316030+0000","last_fullsized":"2023-11-28T20:48:14.316030+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:18.134304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.255975+0000","last_change":"2023-11-28T20:48:14.295333+0000","last_active":"2023-11-28T20:48:15.255975+0000","last_peered":"2023-11-28T20:48:15.255975+0000","last_clean":"2023-11-28T20:48:15.255975+0000","last_became_active":"2023-11-28T20:48:14.295094+0000","last_became_peered":"2023-11-28T20:48:14.295094+0000","last_unstale":"2023-11-28T20:48:15.255975+0000","last_undegraded":"2023-11-28T20:48:15.255975+0000","last_fullsized":"2023-11-28T20:48:15.255975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:19:07.557767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"70'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.650257+0000","last_change":"2023-11-28T20:48:13.280005+0000","last_active":"2023-11-28T20:48:15.650257+0000","last_peered":"2023-11-28T20:48:15.650257+0000","last_clean":"2023-11-28T20:48:15.650257+0000","last_became_active":"2023-11-28T20:48:13.277650+0000","last_became_peered":"2023-11-28T20:48:13.277650+0000","last_unstale":"2023-11-28T20:48:15.650257+0000","last_undegraded":"2023-11-28T20:48:15.650257+0000","last_fullsized":"2023-11-28T20:48:15.650257+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:31:06.616167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261386+0000","last_change":"2023-11-28T20:48:14.298106+0000","last_active":"2023-11-28T20:48:15.261386+0000","last_peered":"2023-11-28T20:48:15.261386+0000","last_clean":"2023-11-28T20:48:15.261386+0000","last_became_active":"2023-11-28T20:48:14.297802+0000","last_became_peered":"2023-11-28T20:48:14.297802+0000","last_unstale":"2023-11-28T20:48:15.261386+0000","last_undegraded":"2023-11-28T20:48:15.261386+0000","last_fullsized":"2023-11-28T20:48:15.261386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:51.256993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"70'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.569061+0000","last_change":"2023-11-28T20:48:13.288101+0000","last_active":"2023-11-28T20:48:15.569061+0000","last_peered":"2023-11-28T20:48:15.569061+0000","last_clean":"2023-11-28T20:48:15.569061+0000","last_became_active":"2023-11-28T20:48:13.286895+0000","last_became_peered":"2023-11-28T20:48:13.286895+0000","last_unstale":"2023-11-28T20:48:15.569061+0000","last_undegraded":"2023-11-28T20:48:15.569061+0000","last_fullsized":"2023-11-28T20:48:15.569061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:14.595108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256525+0000","last_change":"2023-11-28T20:48:14.278974+0000","last_active":"2023-11-28T20:48:15.256525+0000","last_peered":"2023-11-28T20:48:15.256525+0000","last_clean":"2023-11-28T20:48:15.256525+0000","last_became_active":"2023-11-28T20:48:14.278539+0000","last_became_peered":"2023-11-28T20:48:14.278539+0000","last_unstale":"2023-11-28T20:48:15.256525+0000","last_undegraded":"2023-11-28T20:48:15.256525+0000","last_fullsized":"2023-11-28T20:48:15.256525+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:31:45.036181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568946+0000","last_change":"2023-11-28T20:48:13.287730+0000","last_active":"2023-11-28T20:48:15.568946+0000","last_peered":"2023-11-28T20:48:15.568946+0000","last_clean":"2023-11-28T20:48:15.568946+0000","last_became_active":"2023-11-28T20:48:13.287417+0000","last_became_peered":"2023-11-28T20:48:13.287417+0000","last_unstale":"2023-11-28T20:48:15.568946+0000","last_undegraded":"2023-11-28T20:48:15.568946+0000","last_fullsized":"2023-11-28T20:48:15.568946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:37.546954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.283479+0000","last_change":"2023-11-28T20:48:14.277949+0000","last_active":"2023-11-28T20:48:14.283479+0000","last_peered":"2023-11-28T20:48:14.283479+0000","last_clean":"2023-11-28T20:48:14.283479+0000","last_became_active":"2023-11-28T20:48:14.276281+0000","last_became_peered":"2023-11-28T20:48:14.276281+0000","last_unstale":"2023-11-28T20:48:14.283479+0000","last_undegraded":"2023-11-28T20:48:14.283479+0000","last_fullsized":"2023-11-28T20:48:14.283479+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:23:38.447492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"70'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256540+0000","last_change":"2023-11-28T20:48:13.283356+0000","last_active":"2023-11-28T20:48:15.256540+0000","last_peered":"2023-11-28T20:48:15.256540+0000","last_clean":"2023-11-28T20:48:15.256540+0000","last_became_active":"2023-11-28T20:48:13.283067+0000","last_became_peered":"2023-11-28T20:48:13.283067+0000","last_unstale":"2023-11-28T20:48:15.256540+0000","last_undegraded":"2023-11-28T20:48:15.256540+0000","last_fullsized":"2023-11-28T20:48:15.256540+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:43:44.562922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568900+0000","last_change":"2023-11-28T20:48:14.282096+0000","last_active":"2023-11-28T20:48:15.568900+0000","last_peered":"2023-11-28T20:48:15.568900+0000","last_clean":"2023-11-28T20:48:15.568900+0000","last_became_active":"2023-11-28T20:48:14.281219+0000","last_became_peered":"2023-11-28T20:48:14.281219+0000","last_unstale":"2023-11-28T20:48:15.568900+0000","last_undegraded":"2023-11-28T20:48:15.568900+0000","last_fullsized":"2023-11-28T20:48:15.568900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:15:23.890200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258855+0000","last_change":"2023-11-28T20:48:13.279096+0000","last_active":"2023-11-28T20:48:15.258855+0000","last_peered":"2023-11-28T20:48:15.258855+0000","last_clean":"2023-11-28T20:48:15.258855+0000","last_became_active":"2023-11-28T20:48:13.278805+0000","last_became_peered":"2023-11-28T20:48:13.278805+0000","last_unstale":"2023-11-28T20:48:15.258855+0000","last_undegraded":"2023-11-28T20:48:15.258855+0000","last_fullsized":"2023-11-28T20:48:15.258855+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:29:15.712862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.301351+0000","last_change":"2023-11-28T20:48:14.298350+0000","last_active":"2023-11-28T20:48:14.301351+0000","last_peered":"2023-11-28T20:48:14.301351+0000","last_clean":"2023-11-28T20:48:14.301351+0000","last_became_active":"2023-11-28T20:48:14.298024+0000","last_became_peered":"2023-11-28T20:48:14.298024+0000","last_unstale":"2023-11-28T20:48:14.301351+0000","last_undegraded":"2023-11-28T20:48:14.301351+0000","last_fullsized":"2023-11-28T20:48:14.301351+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:50.996911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292721+0000","last_change":"2023-11-28T20:48:13.292422+0000","last_active":"2023-11-28T20:48:14.292721+0000","last_peered":"2023-11-28T20:48:14.292721+0000","last_clean":"2023-11-28T20:48:14.292721+0000","last_became_active":"2023-11-28T20:48:13.291292+0000","last_became_peered":"2023-11-28T20:48:13.291292+0000","last_unstale":"2023-11-28T20:48:14.292721+0000","last_undegraded":"2023-11-28T20:48:14.292721+0000","last_fullsized":"2023-11-28T20:48:14.292721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:42.446875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256841+0000","last_change":"2023-11-28T20:48:14.290101+0000","last_active":"2023-11-28T20:48:15.256841+0000","last_peered":"2023-11-28T20:48:15.256841+0000","last_clean":"2023-11-28T20:48:15.256841+0000","last_became_active":"2023-11-28T20:48:14.289753+0000","last_became_peered":"2023-11-28T20:48:14.289753+0000","last_unstale":"2023-11-28T20:48:15.256841+0000","last_undegraded":"2023-11-28T20:48:15.256841+0000","last_fullsized":"2023-11-28T20:48:15.256841+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:55:36.332492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.650335+0000","last_change":"2023-11-28T20:48:13.294473+0000","last_active":"2023-11-28T20:48:15.650335+0000","last_peered":"2023-11-28T20:48:15.650335+0000","last_clean":"2023-11-28T20:48:15.650335+0000","last_became_active":"2023-11-28T20:48:13.293547+0000","last_became_peered":"2023-11-28T20:48:13.293547+0000","last_unstale":"2023-11-28T20:48:15.650335+0000","last_undegraded":"2023-11-28T20:48:15.650335+0000","last_fullsized":"2023-11-28T20:48:15.650335+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:28.650433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.297540+0000","last_change":"2023-11-28T20:48:14.292571+0000","last_active":"2023-11-28T20:48:14.297540+0000","last_peered":"2023-11-28T20:48:14.297540+0000","last_clean":"2023-11-28T20:48:14.297540+0000","last_became_active":"2023-11-28T20:48:14.292252+0000","last_became_peered":"2023-11-28T20:48:14.292252+0000","last_unstale":"2023-11-28T20:48:14.297540+0000","last_undegraded":"2023-11-28T20:48:14.297540+0000","last_fullsized":"2023-11-28T20:48:14.297540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:39.563576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"70'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.569093+0000","last_change":"2023-11-28T20:48:13.293223+0000","last_active":"2023-11-28T20:48:15.569093+0000","last_peered":"2023-11-28T20:48:15.569093+0000","last_clean":"2023-11-28T20:48:15.569093+0000","last_became_active":"2023-11-28T20:48:13.292528+0000","last_became_peered":"2023-11-28T20:48:13.292528+0000","last_unstale":"2023-11-28T20:48:15.569093+0000","last_undegraded":"2023-11-28T20:48:15.569093+0000","last_fullsized":"2023-11-28T20:48:15.569093+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:39:26.119216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256260+0000","last_change":"2023-11-28T20:48:14.269789+0000","last_active":"2023-11-28T20:48:15.256260+0000","last_peered":"2023-11-28T20:48:15.256260+0000","last_clean":"2023-11-28T20:48:15.256260+0000","last_became_active":"2023-11-28T20:48:14.269188+0000","last_became_peered":"2023-11-28T20:48:14.269188+0000","last_unstale":"2023-11-28T20:48:15.256260+0000","last_undegraded":"2023-11-28T20:48:15.256260+0000","last_fullsized":"2023-11-28T20:48:15.256260+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:36:00.774051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537817+0000","last_change":"2023-11-28T20:48:13.296112+0000","last_active":"2023-11-28T20:48:15.537817+0000","last_peered":"2023-11-28T20:48:15.537817+0000","last_clean":"2023-11-28T20:48:15.537817+0000","last_became_active":"2023-11-28T20:48:13.295794+0000","last_became_peered":"2023-11-28T20:48:13.295794+0000","last_unstale":"2023-11-28T20:48:15.537817+0000","last_undegraded":"2023-11-28T20:48:15.537817+0000","last_fullsized":"2023-11-28T20:48:15.537817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:38.548693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259151+0000","last_change":"2023-11-28T20:48:14.286675+0000","last_active":"2023-11-28T20:48:15.259151+0000","last_peered":"2023-11-28T20:48:15.259151+0000","last_clean":"2023-11-28T20:48:15.259151+0000","last_became_active":"2023-11-28T20:48:14.284310+0000","last_became_peered":"2023-11-28T20:48:14.284310+0000","last_unstale":"2023-11-28T20:48:15.259151+0000","last_undegraded":"2023-11-28T20:48:15.259151+0000","last_fullsized":"2023-11-28T20:48:15.259151+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:50:56.809237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.295777+0000","last_change":"2023-11-28T20:48:13.280959+0000","last_active":"2023-11-28T20:48:14.295777+0000","last_peered":"2023-11-28T20:48:14.295777+0000","last_clean":"2023-11-28T20:48:14.295777+0000","last_became_active":"2023-11-28T20:48:13.280665+0000","last_became_peered":"2023-11-28T20:48:13.280665+0000","last_unstale":"2023-11-28T20:48:14.295777+0000","last_undegraded":"2023-11-28T20:48:14.295777+0000","last_fullsized":"2023-11-28T20:48:14.295777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:22:50.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.295930+0000","last_change":"2023-11-28T20:48:14.293584+0000","last_active":"2023-11-28T20:48:14.295930+0000","last_peered":"2023-11-28T20:48:14.295930+0000","last_clean":"2023-11-28T20:48:14.295930+0000","last_became_active":"2023-11-28T20:48:14.293283+0000","last_became_peered":"2023-11-28T20:48:14.293283+0000","last_unstale":"2023-11-28T20:48:14.295930+0000","last_undegraded":"2023-11-28T20:48:14.295930+0000","last_fullsized":"2023-11-28T20:48:14.295930+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:29:57.111937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259063+0000","last_change":"2023-11-28T20:48:13.270826+0000","last_active":"2023-11-28T20:48:15.259063+0000","last_peered":"2023-11-28T20:48:15.259063+0000","last_clean":"2023-11-28T20:48:15.259063+0000","last_became_active":"2023-11-28T20:48:13.268261+0000","last_became_peered":"2023-11-28T20:48:13.268261+0000","last_unstale":"2023-11-28T20:48:15.259063+0000","last_undegraded":"2023-11-28T20:48:15.259063+0000","last_fullsized":"2023-11-28T20:48:15.259063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:58.165702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.289900+0000","last_change":"2023-11-28T20:48:14.285812+0000","last_active":"2023-11-28T20:48:14.289900+0000","last_peered":"2023-11-28T20:48:14.289900+0000","last_clean":"2023-11-28T20:48:14.289900+0000","last_became_active":"2023-11-28T20:48:14.285543+0000","last_became_peered":"2023-11-28T20:48:14.285543+0000","last_unstale":"2023-11-28T20:48:14.289900+0000","last_undegraded":"2023-11-28T20:48:14.289900+0000","last_fullsized":"2023-11-28T20:48:14.289900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:46:42.560179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258981+0000","last_change":"2023-11-28T20:48:13.286299+0000","last_active":"2023-11-28T20:48:15.258981+0000","last_peered":"2023-11-28T20:48:15.258981+0000","last_clean":"2023-11-28T20:48:15.258981+0000","last_became_active":"2023-11-28T20:48:13.285969+0000","last_became_peered":"2023-11-28T20:48:13.285969+0000","last_unstale":"2023-11-28T20:48:15.258981+0000","last_undegraded":"2023-11-28T20:48:15.258981+0000","last_fullsized":"2023-11-28T20:48:15.258981+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:17.163205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292190+0000","last_change":"2023-11-28T20:48:14.289317+0000","last_active":"2023-11-28T20:48:14.292190+0000","last_peered":"2023-11-28T20:48:14.292190+0000","last_clean":"2023-11-28T20:48:14.292190+0000","last_became_active":"2023-11-28T20:48:14.289016+0000","last_became_peered":"2023-11-28T20:48:14.289016+0000","last_unstale":"2023-11-28T20:48:14.292190+0000","last_undegraded":"2023-11-28T20:48:14.292190+0000","last_fullsized":"2023-11-28T20:48:14.292190+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:00.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261462+0000","last_change":"2023-11-28T20:48:13.283031+0000","last_active":"2023-11-28T20:48:15.261462+0000","last_peered":"2023-11-28T20:48:15.261462+0000","last_clean":"2023-11-28T20:48:15.261462+0000","last_became_active":"2023-11-28T20:48:13.282643+0000","last_became_peered":"2023-11-28T20:48:13.282643+0000","last_unstale":"2023-11-28T20:48:15.261462+0000","last_undegraded":"2023-11-28T20:48:15.261462+0000","last_fullsized":"2023-11-28T20:48:15.261462+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:22:27.604632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259652+0000","last_change":"2023-11-28T20:48:14.288176+0000","last_active":"2023-11-28T20:48:15.259652+0000","last_peered":"2023-11-28T20:48:15.259652+0000","last_clean":"2023-11-28T20:48:15.259652+0000","last_became_active":"2023-11-28T20:48:14.287913+0000","last_became_peered":"2023-11-28T20:48:14.287913+0000","last_unstale":"2023-11-28T20:48:15.259652+0000","last_undegraded":"2023-11-28T20:48:15.259652+0000","last_fullsized":"2023-11-28T20:48:15.259652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:12:49.555700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.270145+0000","last_change":"2023-11-28T20:48:13.286376+0000","last_active":"2023-11-28T20:48:14.270145+0000","last_peered":"2023-11-28T20:48:14.270145+0000","last_clean":"2023-11-28T20:48:14.270145+0000","last_became_active":"2023-11-28T20:48:13.286102+0000","last_became_peered":"2023-11-28T20:48:13.286102+0000","last_unstale":"2023-11-28T20:48:14.270145+0000","last_undegraded":"2023-11-28T20:48:14.270145+0000","last_fullsized":"2023-11-28T20:48:14.270145+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:58.165702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568402+0000","last_change":"2023-11-28T20:48:14.279143+0000","last_active":"2023-11-28T20:48:15.568402+0000","last_peered":"2023-11-28T20:48:15.568402+0000","last_clean":"2023-11-28T20:48:15.568402+0000","last_became_active":"2023-11-28T20:48:14.278881+0000","last_became_peered":"2023-11-28T20:48:14.278881+0000","last_unstale":"2023-11-28T20:48:15.568402+0000","last_undegraded":"2023-11-28T20:48:15.568402+0000","last_fullsized":"2023-11-28T20:48:15.568402+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:41:13.464014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.296264+0000","last_change":"2023-11-28T20:48:13.294843+0000","last_active":"2023-11-28T20:48:14.296264+0000","last_peered":"2023-11-28T20:48:14.296264+0000","last_clean":"2023-11-28T20:48:14.296264+0000","last_became_active":"2023-11-28T20:48:13.294124+0000","last_became_peered":"2023-11-28T20:48:13.294124+0000","last_unstale":"2023-11-28T20:48:14.296264+0000","last_undegraded":"2023-11-28T20:48:14.296264+0000","last_fullsized":"2023-11-28T20:48:14.296264+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:16:00.231441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"59'139","reported_seq":295,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.274854+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:48:14.274854+0000","last_peered":"2023-11-28T20:48:14.274854+0000","last_clean":"2023-11-28T20:48:14.274854+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:48:14.274854+0000","last_undegraded":"2023-11-28T20:48:14.274854+0000","last_fullsized":"2023-11-28T20:48:14.274854+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575023,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58972,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312868,"statfs":{"total":96636764160,"available":96576376832,"internally_reserved":0,"allocated":1454080,"data_stored":1013887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.202,"15min":1.1879999999999999},"min":{"1min":0.68200000000000005,"5min":0.625,"15min":0.51900000000000002},"max":{"1min":1.6140000000000001,"5min":1.7,"15min":2.23},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1910000000000001,"15min":1.1679999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.518,"5min":1.726,"15min":1.8660000000000001},"last":0.82399999999999995}]},{"osd":1,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.1990000000000001,"15min":1.1719999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.591,"5min":1.744,"15min":1.744},"last":1.399},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.2330000000000001,"15min":1.2250000000000001},"min":{"1min":0.80500000000000005,"5min":0.77200000000000002,"15min":0.63800000000000001},"max":{"1min":1.486,"5min":1.835,"15min":2.3599999999999999},"last":1.4359999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.268,"15min":1.3029999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.58799999999999997},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.9359999999999999},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.234,"5min":1.2769999999999999,"15min":1.288},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.704,"5min":1.8069999999999999,"15min":2.137},"last":0.96199999999999997}]},{"osd":3,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.296,"15min":1.2769999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7729999999999999,"5min":1.8460000000000001,"15min":2.1949999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.369,"5min":1.401,"15min":1.4059999999999999},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.68300000000000005},"max":{"1min":1.6459999999999999,"5min":1.758,"15min":2.5259999999999998},"last":1.5760000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.286,"15min":1.266},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7989999999999999,"5min":1.905,"15min":1.919},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.3400000000000001,"15min":1.3380000000000001},"min":{"1min":0.89100000000000001,"5min":0.73399999999999999,"15min":0.51500000000000001},"max":{"1min":1.863,"5min":1.863,"15min":2.8889999999999998},"last":1.1839999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.407,"15min":1.427},"min":{"1min":0.86499999999999999,"5min":0.79800000000000004,"15min":0.67900000000000005},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":2.738},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.486,"15min":1.476},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.71699999999999997},"max":{"1min":1.9099999999999999,"5min":1.974,"15min":3.4180000000000001},"last":1.1279999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5620000000000001,"15min":1.556},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.968,"5min":2.0529999999999999,"15min":2.6339999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4970000000000001,"15min":1.4990000000000001},"min":{"1min":0.94099999999999995,"5min":0.90800000000000003,"15min":0.58499999999999996},"max":{"1min":1.9259999999999999,"5min":2.0979999999999999,"15min":2.7909999999999999},"last":1.504}]}]},{"osd":6,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":424047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.1240000000000001,"15min":1.1200000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.4610000000000001,"5min":1.7350000000000001,"15min":2.1160000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.1659999999999999,"15min":1.1830000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.627},"max":{"1min":1.7230000000000001,"5min":1.8260000000000001,"15min":5.0430000000000001},"last":0.93600000000000005}]},{"osd":1,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.222,"15min":1.2609999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.52200000000000002},"max":{"1min":1.496,"5min":1.6890000000000001,"15min":2.5379999999999998},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.25,"5min":1.3799999999999999,"15min":1.381},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":3.0059999999999998},"last":1.2829999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.3049999999999999,"15min":1.282},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.69799999999999995},"max":{"1min":1.6879999999999999,"5min":1.71,"15min":4.2779999999999996},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.381,"15min":1.3759999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.60299999999999998},"max":{"1min":1.595,"5min":1.7669999999999999,"15min":3.1720000000000002},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.286,"15min":1.3100000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.68799999999999994},"max":{"1min":1.821,"5min":1.9159999999999999,"15min":2.5960000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.2829999999999999,"15min":1.319},"min":{"1min":0.879,"5min":0.879,"15min":0.751},"max":{"1min":1.4990000000000001,"5min":1.6779999999999999,"15min":4.7649999999999997},"last":1.599}]},{"osd":4,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2330000000000001,"15min":1.2909999999999999},"min":{"1min":0.83799999999999997,"5min":0.79600000000000004,"15min":0.72899999999999998},"max":{"1min":1.849,"5min":1.849,"15min":2.7290000000000001},"last":1.222},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.3029999999999999,"15min":1.365},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.76400000000000001},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":3.2519999999999998},"last":1.6559999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.62},"max":{"1min":1.6699999999999999,"5min":1.752,"15min":4.8799999999999999},"last":1.476},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.526,"15min":1.569},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.7190000000000001,"5min":1.9219999999999999,"15min":3.6429999999999998},"last":1.5489999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4610000000000001,"15min":1.5249999999999999},"min":{"1min":1.006,"5min":1.006,"15min":0.85499999999999998},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":3.379},"last":1.766},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.444,"15min":1.518},"min":{"1min":0.93300000000000005,"5min":0.92900000000000005,"15min":0.84599999999999997},"max":{"1min":1.77,"5min":1.8819999999999999,"15min":4.9790000000000001},"last":1.857}]}]},{"osd":5,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":839680,"data_stored":416775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.179,"15min":1.145},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.59499999999999997},"max":{"1min":1.5209999999999999,"5min":1.7450000000000001,"15min":1.768},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1890000000000001,"15min":1.1859999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.6080000000000001,"5min":1.774,"15min":1.996},"last":1.4350000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.1579999999999999,"15min":1.1439999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.48899999999999999},"max":{"1min":1.3360000000000001,"5min":1.663,"15min":1.796},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.139,"5min":1.236,"15min":1.2090000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.54,"5min":1.6910000000000001,"15min":1.956},"last":1.196}]},{"osd":2,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.286,"15min":1.2689999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.901},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.304,"15min":1.2969999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.6679999999999999,"5min":1.958,"15min":2.27},"last":1.1220000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.3149999999999999,"15min":1.319},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":2.298},"last":1.534},{"interface":"front","average":{"1min":1.175,"5min":1.256,"15min":1.272},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.54600000000000004},"max":{"1min":1.5629999999999999,"5min":1.7509999999999999,"15min":2.177},"last":1.0740000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.302,"15min":1.3080000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6180000000000001,"5min":1.776,"15min":2.4329999999999998},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.4199999999999999,"15min":1.4319999999999999},"min":{"1min":0.79700000000000004,"5min":0.77000000000000002,"15min":0.68100000000000005},"max":{"1min":1.744,"5min":1.806,"15min":2.7599999999999998},"last":1.6839999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.306,"15min":1.3300000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.78,"5min":2.093,"15min":2.9319999999999999},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.464,"15min":1.4670000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6339999999999999,"5min":1.806,"15min":2.339},"last":1.601}]},{"osd":7,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4339999999999999,"15min":1.421},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.73399999999999999},"max":{"1min":1.599,"5min":2.0019999999999998,"15min":2.706},"last":1.571},{"interface":"front","average":{"1min":1.381,"5min":1.504,"15min":1.4890000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.8600000000000001,"5min":2.1829999999999998,"15min":2.3849999999999998},"last":1.74}]}]},{"osd":4,"up_from":13,"seq":55834575024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":856064,"data_stored":423977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.151,"15min":1.1339999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.64400000000000002},"max":{"1min":1.536,"5min":1.879,"15min":1.879},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.131,"15min":1.105},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.891,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.006}]},{"osd":1,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.3240000000000001,"15min":1.3080000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.71599999999999997},"max":{"1min":1.734,"5min":1.923,"15min":2.3879999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.397,"15min":1.337},"min":{"1min":0.91700000000000004,"5min":0.86599999999999999,"15min":0.76700000000000002},"max":{"1min":2.2679999999999998,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.212}]},{"osd":2,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2250000000000001,"15min":1.2210000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.6699999999999999,"5min":1.71,"15min":1.71},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2889999999999999,"15min":1.2929999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67500000000000004},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.1020000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3120000000000001,"15min":1.2889999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.246},{"interface":"front","average":{"1min":1.218,"5min":1.355,"15min":1.375},"min":{"1min":0.876,"5min":0.876,"15min":0.67900000000000005},"max":{"1min":1.583,"5min":2.0030000000000001,"15min":2.2519999999999998},"last":1.391}]},{"osd":5,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3169999999999999,"15min":1.3},"min":{"1min":1.02,"5min":0.83599999999999997,"15min":0.61699999999999999},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.619},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3360000000000001,"15min":1.3100000000000001},"min":{"1min":1.1919999999999999,"5min":0.82399999999999995,"15min":0.69299999999999995},"max":{"1min":1.647,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.343}]},{"osd":6,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3879999999999999,"15min":1.4199999999999999},"min":{"1min":0.96999999999999997,"5min":0.90400000000000003,"15min":0.628},"max":{"1min":2.04,"5min":2.04,"15min":2.4900000000000002},"last":1.579},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.5960000000000001,"15min":1.5529999999999999},"min":{"1min":1.194,"5min":1.133,"15min":0.753},"max":{"1min":3.7000000000000002,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":1.8660000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5009999999999999,"15min":1.6000000000000001},"min":{"1min":1.046,"5min":1.046,"15min":0.67500000000000004},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":20.568999999999999},"last":1.55},{"interface":"front","average":{"1min":1.486,"5min":1.5069999999999999,"15min":1.673},"min":{"1min":1.2370000000000001,"5min":0.96499999999999997,"15min":0.72299999999999998},"max":{"1min":1.7549999999999999,"5min":2.1099999999999999,"15min":20.677},"last":1.657}]}]},{"osd":3,"up_from":13,"seq":55834575024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40352,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331488,"statfs":{"total":96636764160,"available":96595443712,"internally_reserved":0,"allocated":868352,"data_stored":425029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.91300000000000003,"15min":0.93500000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38800000000000001},"max":{"1min":4.7210000000000001,"5min":4.7210000000000001,"15min":4.7210000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.0609999999999999,"15min":1.0700000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.34399999999999997},"max":{"1min":1.5860000000000001,"5min":2.0950000000000002,"15min":3.968},"last":0.437}]},{"osd":1,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":0.99099999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.36299999999999999},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":3.9180000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.2290000000000001,"15min":1.218},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.52600000000000002},"max":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.67400000000000004}]},{"osd":2,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2569999999999999,"15min":1.2709999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.48099999999999998},"max":{"1min":3.444,"5min":3.444,"15min":5.6509999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.351,"5min":1.359,"15min":1.343},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.46400000000000002},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":4.5759999999999996},"last":0.745}]},{"osd":4,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.377,"15min":1.343},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.72299999999999998},"max":{"1min":1.806,"5min":1.9079999999999999,"15min":4.1779999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.369,"15min":1.3999999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.59899999999999998},"max":{"1min":1.53,"5min":2.1389999999999998,"15min":4.492},"last":1.103}]},{"osd":5,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.4910000000000001,"15min":1.478},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.66800000000000004},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":4.7169999999999996},"last":1.006},{"interface":"front","average":{"1min":1.339,"5min":1.4390000000000001,"15min":1.4410000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":4.2889999999999997},"last":0.82399999999999995}]},{"osd":6,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.488,"15min":1.508},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.72799999999999998},"max":{"1min":1.7769999999999999,"5min":2.3100000000000001,"15min":5.2279999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.6180000000000001,"15min":1.6120000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.70199999999999996},"max":{"1min":3.661,"5min":3.661,"15min":6.9740000000000002},"last":1.1850000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.546,"15min":1.552},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.73599999999999999},"max":{"1min":2.1469999999999998,"5min":2.1469999999999998,"15min":4.0819999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.591,"15min":1.6200000000000001},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.64900000000000002},"max":{"1min":1.7969999999999999,"5min":2.3919999999999999,"15min":4.1399999999999997},"last":1.2110000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575024,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58396,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313444,"statfs":{"total":96636764160,"available":96576966656,"internally_reserved":0,"allocated":864256,"data_stored":425007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.93799999999999994,"15min":0.90400000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.413,"5min":2.012,"15min":2.012},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.0389999999999999,"15min":1.0489999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":1.47,"5min":1.6539999999999999,"15min":1.9390000000000001},"last":1.151}]},{"osd":1,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.95399999999999996,"15min":0.95799999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.2669999999999999,"5min":1.998,"15min":3.2599999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.1539999999999999,"15min":1.1359999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.38500000000000001},"max":{"1min":1.506,"5min":1.7310000000000001,"15min":1.837},"last":1.3560000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.1619999999999999,"15min":1.143},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.442},"max":{"1min":1.7070000000000001,"5min":1.774,"15min":2.4319999999999999},"last":1.7070000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.26,"15min":1.2549999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.45600000000000002},"max":{"1min":1.3799999999999999,"5min":1.742,"15min":3.1440000000000001},"last":1.05}]},{"osd":4,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.401,"15min":1.3759999999999999},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.72699999999999998},"max":{"1min":1.7450000000000001,"5min":1.9390000000000001,"15min":2.8919999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.389,"15min":1.4079999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.80500000000000005},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.8900000000000001},"last":2.3740000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.4259999999999999,"15min":1.405},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.72499999999999998},"max":{"1min":1.6779999999999999,"5min":2.141,"15min":2.1459999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.488,"15min":1.464},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.751},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":3.4279999999999999},"last":1.3959999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.478,"15min":1.4970000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.76400000000000001},"max":{"1min":2.1709999999999998,"5min":2.306,"15min":2.931},"last":2.1709999999999998},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.532,"15min":1.5449999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.78300000000000003},"max":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":3.6269999999999998},"last":1.49}]},{"osd":7,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.552,"15min":1.522},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.84999999999999998},"max":{"1min":1.8879999999999999,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.5800000000000001,"15min":1.5820000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.78400000000000003},"max":{"1min":2.274,"5min":2.4849999999999999,"15min":2.9649999999999999},"last":2.274}]}]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36184,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335656,"statfs":{"total":96636764160,"available":96599711744,"internally_reserved":0,"allocated":860160,"data_stored":423555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.91900000000000004,"15min":0.91000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.6639999999999999,"5min":1.853,"15min":1.9259999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.121,"15min":1.069},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48299999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.8919999999999999},"last":1.3280000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.083,"15min":1.087},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.526,"5min":2.234,"15min":2.234},"last":1.405},{"interface":"front","average":{"1min":1.016,"5min":1.204,"15min":1.1930000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.496,"5min":2.895,"15min":2.895},"last":1.0780000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.286,"15min":1.3009999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.893,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.6859999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.1160000000000001,"15min":1.105},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.887,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.206}]},{"osd":4,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.345,"15min":1.3380000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.76600000000000001},"max":{"1min":1.569,"5min":2.1019999999999999,"15min":2.9020000000000001},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.3779999999999999,"15min":1.403},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8069999999999999,"5min":2.1469999999999998,"15min":3.5259999999999998},"last":1.619}]},{"osd":5,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.4339999999999999,"15min":1.4410000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.80900000000000005},"max":{"1min":1.8080000000000001,"5min":2.1840000000000002,"15min":3.637},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.502,"15min":1.496},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.85299999999999998},"max":{"1min":1.768,"5min":2.294,"15min":2.3050000000000002},"last":1.5700000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.423,"15min":1.446},"min":{"1min":0.874,"5min":0.874,"15min":0.81100000000000005},"max":{"1min":1.641,"5min":1.9379999999999999,"15min":4.5789999999999997},"last":1.738},{"interface":"front","average":{"1min":1.327,"5min":1.5389999999999999,"15min":1.536},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.76300000000000001},"max":{"1min":1.946,"5min":2.3300000000000001,"15min":3.7360000000000002},"last":1.79}]},{"osd":7,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.792,"5min":2.3359999999999999,"15min":2.4319999999999999},"last":1.464},{"interface":"front","average":{"1min":1.413,"5min":1.5569999999999999,"15min":1.5269999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.843,"5min":2.294,"15min":4.4660000000000002},"last":1.5129999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575024,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36752,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1441792,"data_stored":1007199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95099999999999996,"15min":0.92300000000000004},"min":{"1min":0.71099999999999997,"5min":0.59499999999999997,"15min":0.41499999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.999},{"interface":"front","average":{"1min":1.03,"5min":1.105,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.45700000000000002},"max":{"1min":1.3200000000000001,"5min":1.764,"15min":2.7229999999999999},"last":1.1659999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.163,"15min":1.1339999999999999},"min":{"1min":0.88300000000000001,"5min":0.80600000000000005,"15min":0.46600000000000003},"max":{"1min":1.5800000000000001,"5min":1.657,"15min":2.0579999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.216,"5min":1.1850000000000001,"15min":1.224},"min":{"1min":0.86399999999999999,"5min":0.82499999999999996,"15min":0.56299999999999994},"max":{"1min":1.5449999999999999,"5min":1.9159999999999999,"15min":2.964},"last":1.2909999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1220000000000001,"15min":1.1499999999999999},"min":{"1min":0.79000000000000004,"5min":0.64000000000000001,"15min":0.504},"max":{"1min":1.458,"5min":1.8420000000000001,"15min":2.6920000000000002},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.365,"15min":1.3380000000000001},"min":{"1min":1.0720000000000001,"5min":0.90200000000000002,"15min":0.52500000000000002},"max":{"1min":1.698,"5min":2.1080000000000001,"15min":2.1379999999999999},"last":1.1120000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.47,"15min":1.4430000000000001},"min":{"1min":1.1259999999999999,"5min":0.86099999999999999,"15min":0.80900000000000005},"max":{"1min":2.012,"5min":2.012,"15min":2.7989999999999999},"last":1.635},{"interface":"front","average":{"1min":1.506,"5min":1.4370000000000001,"15min":1.391},"min":{"1min":1.2,"5min":1.0489999999999999,"15min":0.68300000000000005},"max":{"1min":1.8660000000000001,"5min":2.0539999999999998,"15min":2.073},"last":1.254}]},{"osd":5,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.429,"15min":1.454},"min":{"1min":1.0760000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.528},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5429999999999999,"15min":1.5109999999999999},"min":{"1min":1.119,"5min":1.119,"15min":0.89700000000000002},"max":{"1min":2.0750000000000002,"5min":2.2490000000000001,"15min":2.278},"last":1.22}]},{"osd":6,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.6060000000000001,"15min":1.581},"min":{"1min":1.4159999999999999,"5min":1.242,"15min":0.78800000000000003},"max":{"1min":2.1160000000000001,"5min":2.2629999999999999,"15min":2.536},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5669999999999999,"15min":1.5620000000000001},"min":{"1min":1.147,"5min":1.131,"15min":0.81100000000000005},"max":{"1min":1.847,"5min":2.4209999999999998,"15min":2.7069999999999999},"last":1.405}]},{"osd":7,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.611,"15min":1.631},"min":{"1min":1.216,"5min":1.216,"15min":0.99299999999999999},"max":{"1min":1.996,"5min":2.3330000000000002,"15min":2.8650000000000002},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.5609999999999999,"15min":1.597},"min":{"1min":1.3460000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.0209999999999999,"5min":2.524,"15min":2.6640000000000001},"last":1.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:48:18.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:48:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.068+0000 7f7daede6640 1 Processor -- start 2023-11-28T20:48:19.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.068+0000 7f7daede6640 1 -- start start 2023-11-28T20:48:19.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.070+0000 7f7daede6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7da80721e0 0x7f7da806d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.070+0000 7f7daede6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7da806d6b0 0x7f7da80ac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.070+0000 7f7daede6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da815e680 con 0x7f7da80b3870 2023-11-28T20:48:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.070+0000 7f7daede6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da815a460 con 0x7f7da80721e0 2023-11-28T20:48:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.070+0000 7f7daede6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da806db40 con 0x7f7da806d6b0 2023-11-28T20:48:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dadde4640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7da80b3870 0x7f7da815db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58216/0 (socket says 172.21.15.45:58216) 2023-11-28T20:48:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dadde4640 1 -- 172.21.15.45:0/645005688 learned_addr learned my addr 172.21.15.45:0/645005688 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dad5e3640 1 --2- 172.21.15.45:0/645005688 >> v2:172.21.15.45:3300/0 conn(0x7f7da80721e0 0x7f7da806d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dae5e5640 1 --2- 172.21.15.45:0/645005688 >> v2:172.21.15.190:3300/0 conn(0x7f7da806d6b0 0x7f7da80ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dacde2640 1 -- 172.21.15.45:0/645005688 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544665428 0 0) 0x7f7da815e680 con 0x7f7da80b3870 2023-11-28T20:48:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dacde2640 1 -- 172.21.15.45:0/645005688 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d94005730 con 0x7f7da80b3870 2023-11-28T20:48:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dad5e3640 1 -- 172.21.15.45:0/645005688 >> v2:172.21.15.190:3300/0 conn(0x7f7da806d6b0 msgr2=0x7f7da80ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dad5e3640 1 --2- 172.21.15.45:0/645005688 >> v2:172.21.15.190:3300/0 conn(0x7f7da806d6b0 0x7f7da80ac840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dad5e3640 1 -- 172.21.15.45:0/645005688 >> v1:172.21.15.45:6789/0 conn(0x7f7da80b3870 legacy=0x7f7da815db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.071+0000 7f7dad5e3640 1 -- 172.21.15.45:0/645005688 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da815e960 con 0x7f7da80721e0 2023-11-28T20:48:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.073+0000 7f7dad5e3640 1 --2- 172.21.15.45:0/645005688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 0x7f7da806d170 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7da815da50 tx=0x7f7d9802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa38d8e84b7026db server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.073+0000 7f7dacde2640 1 -- 172.21.15.45:0/645005688 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98031830 con 0x7f7da80721e0 2023-11-28T20:48:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.073+0000 7f7dacde2640 1 -- 172.21.15.45:0/645005688 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d98031990 con 0x7f7da80721e0 2023-11-28T20:48:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.073+0000 7f7dacde2640 1 -- 172.21.15.45:0/645005688 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98031d20 con 0x7f7da80721e0 2023-11-28T20:48:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.073+0000 7f7daede6640 1 -- 172.21.15.45:0/645005688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 msgr2=0x7f7da806d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.073+0000 7f7daede6640 1 --2- 172.21.15.45:0/645005688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 0x7f7da806d170 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7da815da50 tx=0x7f7d9802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.074+0000 7f7daede6640 1 -- 172.21.15.45:0/645005688 shutdown_connections 2023-11-28T20:48:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.074+0000 7f7daede6640 1 --2- 172.21.15.45:0/645005688 >> v2:172.21.15.190:3300/0 conn(0x7f7da806d6b0 0x7f7da80ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.074+0000 7f7daede6640 1 --2- 172.21.15.45:0/645005688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 0x7f7da806d170 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.074+0000 7f7daede6640 1 -- 172.21.15.45:0/645005688 >> 172.21.15.45:0/645005688 conn(0x7f7da806ab30 msgr2=0x7f7da80b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.074+0000 7f7daede6640 1 -- 172.21.15.45:0/645005688 shutdown_connections 2023-11-28T20:48:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.074+0000 7f7daede6640 1 -- 172.21.15.45:0/645005688 wait complete. 2023-11-28T20:48:19.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.075+0000 7f7daede6640 1 Processor -- start 2023-11-28T20:48:19.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.076+0000 7f7daede6640 1 -- start start 2023-11-28T20:48:19.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7daede6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7da806d6b0 0x7f7da809abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7daede6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 0x7f7da8154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7daede6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da80b3870 0x7f7da80c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7daede6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da80cd090 con 0x7f7da80721e0 2023-11-28T20:48:19.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7daede6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da80ccad0 con 0x7f7da80b3870 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7daede6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da80ccdb0 con 0x7f7da806d6b0 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dad5e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 0x7f7da8154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dad5e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 0x7f7da8154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34410/0 (socket says 172.21.15.45:34410) 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dad5e3640 1 -- 172.21.15.45:0/354844065 learned_addr learned my addr 172.21.15.45:0/354844065 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dadde4640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7da806d6b0 0x7f7da809abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dae5e5640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da80b3870 0x7f7da80c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dad5e3640 1 -- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7da806d6b0 msgr2=0x7f7da809abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dad5e3640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7da806d6b0 0x7f7da809abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dad5e3640 1 -- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da80b3870 msgr2=0x7f7da80c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dad5e3640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da80b3870 0x7f7da80c9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dad5e3640 1 -- 172.21.15.45:0/354844065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98009400 con 0x7f7da80721e0 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.077+0000 7f7dadde4640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7da806d6b0 0x7f7da809abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:19.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.078+0000 7f7dad5e3640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 0x7f7da8154cc0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009890 tx=0x7f7d980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.078+0000 7f7d9e7fc640 1 -- 172.21.15.45:0/354844065 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9803a6f0 con 0x7f7da80721e0 2023-11-28T20:48:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.078+0000 7f7d9e7fc640 1 -- 172.21.15.45:0/354844065 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9803a870 con 0x7f7da80721e0 2023-11-28T20:48:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.078+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da80c8240 con 0x7f7da80721e0 2023-11-28T20:48:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.078+0000 7f7d9e7fc640 1 -- 172.21.15.45:0/354844065 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9803ac20 con 0x7f7da80721e0 2023-11-28T20:48:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.078+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da8155200 con 0x7f7da80721e0 2023-11-28T20:48:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.080+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da80c8760 con 0x7f7da80721e0 2023-11-28T20:48:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.080+0000 7f7d9e7fc640 1 -- 172.21.15.45:0/354844065 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7d98043410 con 0x7f7da80721e0 2023-11-28T20:48:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.081+0000 7f7d9e7fc640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7d8c06af60 0x7f7d8c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.081+0000 7f7d9e7fc640 1 -- 172.21.15.45:0/354844065 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7d980ab680 con 0x7f7da80721e0 2023-11-28T20:48:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.082+0000 7f7dadde4640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7d8c06af60 0x7f7d8c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.083+0000 7f7dadde4640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7d8c06af60 0x7f7d8c06d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7da40011e0 tx=0x7f7da4009040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:19.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.086+0000 7f7d9e7fc640 1 -- 172.21.15.45:0/354844065 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7da80c8760 con 0x7f7da80721e0 2023-11-28T20:48:19.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.280+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 --> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7da8071cc0 con 0x7f7d8c06af60 2023-11-28T20:48:19.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.304+0000 7f7d9e7fc640 1 -- 172.21.15.45:0/354844065 <== mgr.5305 v2:172.21.15.190:6832/838517151 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286547 (secure 0 0 0) 0x7f7da4001c80 con 0x7f7d8c06af60 2023-11-28T20:48:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:19.313 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7d8c06af60 msgr2=0x7f7d8c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7d8c06af60 0x7f7d8c06d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7da40011e0 tx=0x7f7da4009040 comp rx=0 tx=0).stop 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 msgr2=0x7f7da8154cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 0x7f7da8154cc0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009890 tx=0x7f7d980046c0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 shutdown_connections 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f7d8c06af60 0x7f7d8c06d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da80b3870 0x7f7da80c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7da80721e0 0x7f7da8154cc0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 --2- 172.21.15.45:0/354844065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7da806d6b0 0x7f7da809abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 >> 172.21.15.45:0/354844065 conn(0x7f7da806ab30 msgr2=0x7f7da80b08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 shutdown_connections 2023-11-28T20:48:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.310+0000 7f7daede6640 1 -- 172.21.15.45:0/354844065 wait complete. 2023-11-28T20:48:19.334 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-11-28T20:48:17.609503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326692,"kb_used_data":7856,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754648028,"statfs":{"total":773094113280,"available":772759580672,"internally_reserved":0,"allocated":8044544,"data_stored":4559476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":326356528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.359097"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256814+0000","last_change":"2023-11-28T20:48:14.282138+0000","last_active":"2023-11-28T20:48:15.256814+0000","last_peered":"2023-11-28T20:48:15.256814+0000","last_clean":"2023-11-28T20:48:15.256814+0000","last_became_active":"2023-11-28T20:48:14.281794+0000","last_became_peered":"2023-11-28T20:48:14.281794+0000","last_unstale":"2023-11-28T20:48:15.256814+0000","last_undegraded":"2023-11-28T20:48:15.256814+0000","last_fullsized":"2023-11-28T20:48:15.256814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:50:20.379297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537473+0000","last_change":"2023-11-28T20:48:14.290895+0000","last_active":"2023-11-28T20:48:15.537473+0000","last_peered":"2023-11-28T20:48:15.537473+0000","last_clean":"2023-11-28T20:48:15.537473+0000","last_became_active":"2023-11-28T20:48:14.290535+0000","last_became_peered":"2023-11-28T20:48:14.290535+0000","last_unstale":"2023-11-28T20:48:15.537473+0000","last_undegraded":"2023-11-28T20:48:15.537473+0000","last_fullsized":"2023-11-28T20:48:15.537473+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:29:16.727745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292144+0000","last_change":"2023-11-28T20:48:14.286727+0000","last_active":"2023-11-28T20:48:14.292144+0000","last_peered":"2023-11-28T20:48:14.292144+0000","last_clean":"2023-11-28T20:48:14.292144+0000","last_became_active":"2023-11-28T20:48:14.285651+0000","last_became_peered":"2023-11-28T20:48:14.285651+0000","last_unstale":"2023-11-28T20:48:14.292144+0000","last_undegraded":"2023-11-28T20:48:14.292144+0000","last_fullsized":"2023-11-28T20:48:14.292144+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:21:13.104355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537424+0000","last_change":"2023-11-28T20:48:14.294399+0000","last_active":"2023-11-28T20:48:15.537424+0000","last_peered":"2023-11-28T20:48:15.537424+0000","last_clean":"2023-11-28T20:48:15.537424+0000","last_became_active":"2023-11-28T20:48:14.294137+0000","last_became_peered":"2023-11-28T20:48:14.294137+0000","last_unstale":"2023-11-28T20:48:15.537424+0000","last_undegraded":"2023-11-28T20:48:15.537424+0000","last_fullsized":"2023-11-28T20:48:15.537424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:05.552512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.262601+0000","last_change":"2023-11-28T20:48:14.276269+0000","last_active":"2023-11-28T20:48:15.262601+0000","last_peered":"2023-11-28T20:48:15.262601+0000","last_clean":"2023-11-28T20:48:15.262601+0000","last_became_active":"2023-11-28T20:48:14.275508+0000","last_became_peered":"2023-11-28T20:48:14.275508+0000","last_unstale":"2023-11-28T20:48:15.262601+0000","last_undegraded":"2023-11-28T20:48:15.262601+0000","last_fullsized":"2023-11-28T20:48:15.262601+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:24:16.445277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256077+0000","last_change":"2023-11-28T20:48:14.286697+0000","last_active":"2023-11-28T20:48:15.256077+0000","last_peered":"2023-11-28T20:48:15.256077+0000","last_clean":"2023-11-28T20:48:15.256077+0000","last_became_active":"2023-11-28T20:48:14.286411+0000","last_became_peered":"2023-11-28T20:48:14.286411+0000","last_unstale":"2023-11-28T20:48:15.256077+0000","last_undegraded":"2023-11-28T20:48:15.256077+0000","last_fullsized":"2023-11-28T20:48:15.256077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:22:28.619514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292132+0000","last_change":"2023-11-28T20:48:14.285906+0000","last_active":"2023-11-28T20:48:14.292132+0000","last_peered":"2023-11-28T20:48:14.292132+0000","last_clean":"2023-11-28T20:48:14.292132+0000","last_became_active":"2023-11-28T20:48:14.285382+0000","last_became_peered":"2023-11-28T20:48:14.285382+0000","last_unstale":"2023-11-28T20:48:14.292132+0000","last_undegraded":"2023-11-28T20:48:14.292132+0000","last_fullsized":"2023-11-28T20:48:14.292132+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:57:00.186060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258946+0000","last_change":"2023-11-28T20:48:14.297202+0000","last_active":"2023-11-28T20:48:15.258946+0000","last_peered":"2023-11-28T20:48:15.258946+0000","last_clean":"2023-11-28T20:48:15.258946+0000","last_became_active":"2023-11-28T20:48:14.296899+0000","last_became_peered":"2023-11-28T20:48:14.296899+0000","last_unstale":"2023-11-28T20:48:15.258946+0000","last_undegraded":"2023-11-28T20:48:15.258946+0000","last_fullsized":"2023-11-28T20:48:15.258946+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:54.614672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259609+0000","last_change":"2023-11-28T20:48:14.300760+0000","last_active":"2023-11-28T20:48:15.259609+0000","last_peered":"2023-11-28T20:48:15.259609+0000","last_clean":"2023-11-28T20:48:15.259609+0000","last_became_active":"2023-11-28T20:48:14.299658+0000","last_became_peered":"2023-11-28T20:48:14.299658+0000","last_unstale":"2023-11-28T20:48:15.259609+0000","last_undegraded":"2023-11-28T20:48:15.259609+0000","last_fullsized":"2023-11-28T20:48:15.259609+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:35.932070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256114+0000","last_change":"2023-11-28T20:48:14.271209+0000","last_active":"2023-11-28T20:48:15.256114+0000","last_peered":"2023-11-28T20:48:15.256114+0000","last_clean":"2023-11-28T20:48:15.256114+0000","last_became_active":"2023-11-28T20:48:14.270488+0000","last_became_peered":"2023-11-28T20:48:14.270488+0000","last_unstale":"2023-11-28T20:48:15.256114+0000","last_undegraded":"2023-11-28T20:48:15.256114+0000","last_fullsized":"2023-11-28T20:48:15.256114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:56.874613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.650119+0000","last_change":"2023-11-28T20:48:14.279333+0000","last_active":"2023-11-28T20:48:15.650119+0000","last_peered":"2023-11-28T20:48:15.650119+0000","last_clean":"2023-11-28T20:48:15.650119+0000","last_became_active":"2023-11-28T20:48:14.277790+0000","last_became_peered":"2023-11-28T20:48:14.277790+0000","last_unstale":"2023-11-28T20:48:15.650119+0000","last_undegraded":"2023-11-28T20:48:15.650119+0000","last_fullsized":"2023-11-28T20:48:15.650119+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:12:04.544981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259314+0000","last_change":"2023-11-28T20:48:14.291079+0000","last_active":"2023-11-28T20:48:15.259314+0000","last_peered":"2023-11-28T20:48:15.259314+0000","last_clean":"2023-11-28T20:48:15.259314+0000","last_became_active":"2023-11-28T20:48:14.290758+0000","last_became_peered":"2023-11-28T20:48:14.290758+0000","last_unstale":"2023-11-28T20:48:15.259314+0000","last_undegraded":"2023-11-28T20:48:15.259314+0000","last_fullsized":"2023-11-28T20:48:15.259314+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:59:53.415363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256073+0000","last_change":"2023-11-28T20:48:14.267972+0000","last_active":"2023-11-28T20:48:15.256073+0000","last_peered":"2023-11-28T20:48:15.256073+0000","last_clean":"2023-11-28T20:48:15.256073+0000","last_became_active":"2023-11-28T20:48:14.267199+0000","last_became_peered":"2023-11-28T20:48:14.267199+0000","last_unstale":"2023-11-28T20:48:15.256073+0000","last_undegraded":"2023-11-28T20:48:15.256073+0000","last_fullsized":"2023-11-28T20:48:15.256073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:36:24.424054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292715+0000","last_change":"2023-11-28T20:48:14.284015+0000","last_active":"2023-11-28T20:48:14.292715+0000","last_peered":"2023-11-28T20:48:14.292715+0000","last_clean":"2023-11-28T20:48:14.292715+0000","last_became_active":"2023-11-28T20:48:14.283678+0000","last_became_peered":"2023-11-28T20:48:14.283678+0000","last_unstale":"2023-11-28T20:48:14.292715+0000","last_undegraded":"2023-11-28T20:48:14.292715+0000","last_fullsized":"2023-11-28T20:48:14.292715+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:59:03.588833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.536937+0000","last_change":"2023-11-28T20:48:14.286266+0000","last_active":"2023-11-28T20:48:15.536937+0000","last_peered":"2023-11-28T20:48:15.536937+0000","last_clean":"2023-11-28T20:48:15.536937+0000","last_became_active":"2023-11-28T20:48:14.285759+0000","last_became_peered":"2023-11-28T20:48:14.285759+0000","last_unstale":"2023-11-28T20:48:15.536937+0000","last_undegraded":"2023-11-28T20:48:15.536937+0000","last_fullsized":"2023-11-28T20:48:15.536937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:19:15.212329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.255923+0000","last_change":"2023-11-28T20:48:14.276424+0000","last_active":"2023-11-28T20:48:15.255923+0000","last_peered":"2023-11-28T20:48:15.255923+0000","last_clean":"2023-11-28T20:48:15.255923+0000","last_became_active":"2023-11-28T20:48:14.275745+0000","last_became_peered":"2023-11-28T20:48:14.275745+0000","last_unstale":"2023-11-28T20:48:15.255923+0000","last_undegraded":"2023-11-28T20:48:15.255923+0000","last_fullsized":"2023-11-28T20:48:15.255923+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:15:23.890200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292240+0000","last_change":"2023-11-28T20:48:14.289296+0000","last_active":"2023-11-28T20:48:14.292240+0000","last_peered":"2023-11-28T20:48:14.292240+0000","last_clean":"2023-11-28T20:48:14.292240+0000","last_became_active":"2023-11-28T20:48:14.286635+0000","last_became_peered":"2023-11-28T20:48:14.286635+0000","last_unstale":"2023-11-28T20:48:14.292240+0000","last_undegraded":"2023-11-28T20:48:14.292240+0000","last_fullsized":"2023-11-28T20:48:14.292240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:40:48.337896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537619+0000","last_change":"2023-11-28T20:48:14.292874+0000","last_active":"2023-11-28T20:48:15.537619+0000","last_peered":"2023-11-28T20:48:15.537619+0000","last_clean":"2023-11-28T20:48:15.537619+0000","last_became_active":"2023-11-28T20:48:14.292571+0000","last_became_peered":"2023-11-28T20:48:14.292571+0000","last_unstale":"2023-11-28T20:48:15.537619+0000","last_undegraded":"2023-11-28T20:48:15.537619+0000","last_fullsized":"2023-11-28T20:48:15.537619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:18.178088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.302193+0000","last_change":"2023-11-28T20:48:14.299861+0000","last_active":"2023-11-28T20:48:14.302193+0000","last_peered":"2023-11-28T20:48:14.302193+0000","last_clean":"2023-11-28T20:48:14.302193+0000","last_became_active":"2023-11-28T20:48:14.298268+0000","last_became_peered":"2023-11-28T20:48:14.298268+0000","last_unstale":"2023-11-28T20:48:14.302193+0000","last_undegraded":"2023-11-28T20:48:14.302193+0000","last_fullsized":"2023-11-28T20:48:14.302193+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:14:49.829548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568771+0000","last_change":"2023-11-28T20:48:14.282949+0000","last_active":"2023-11-28T20:48:15.568771+0000","last_peered":"2023-11-28T20:48:15.568771+0000","last_clean":"2023-11-28T20:48:15.568771+0000","last_became_active":"2023-11-28T20:48:14.281952+0000","last_became_peered":"2023-11-28T20:48:14.281952+0000","last_unstale":"2023-11-28T20:48:15.568771+0000","last_undegraded":"2023-11-28T20:48:15.568771+0000","last_fullsized":"2023-11-28T20:48:15.568771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:17:35.465307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568846+0000","last_change":"2023-11-28T20:48:14.277401+0000","last_active":"2023-11-28T20:48:15.568846+0000","last_peered":"2023-11-28T20:48:15.568846+0000","last_clean":"2023-11-28T20:48:15.568846+0000","last_became_active":"2023-11-28T20:48:14.277001+0000","last_became_peered":"2023-11-28T20:48:14.277001+0000","last_unstale":"2023-11-28T20:48:15.568846+0000","last_undegraded":"2023-11-28T20:48:15.568846+0000","last_fullsized":"2023-11-28T20:48:15.568846+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:44:53.943332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258889+0000","last_change":"2023-11-28T20:48:14.297400+0000","last_active":"2023-11-28T20:48:15.258889+0000","last_peered":"2023-11-28T20:48:15.258889+0000","last_clean":"2023-11-28T20:48:15.258889+0000","last_became_active":"2023-11-28T20:48:14.297076+0000","last_became_peered":"2023-11-28T20:48:14.297076+0000","last_unstale":"2023-11-28T20:48:15.258889+0000","last_undegraded":"2023-11-28T20:48:15.258889+0000","last_fullsized":"2023-11-28T20:48:15.258889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:08.600530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.650505+0000","last_change":"2023-11-28T20:48:14.286000+0000","last_active":"2023-11-28T20:48:15.650505+0000","last_peered":"2023-11-28T20:48:15.650505+0000","last_clean":"2023-11-28T20:48:15.650505+0000","last_became_active":"2023-11-28T20:48:14.285692+0000","last_became_peered":"2023-11-28T20:48:14.285692+0000","last_unstale":"2023-11-28T20:48:15.650505+0000","last_undegraded":"2023-11-28T20:48:15.650505+0000","last_fullsized":"2023-11-28T20:48:15.650505+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:07:38.436914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292603+0000","last_change":"2023-11-28T20:48:14.277523+0000","last_active":"2023-11-28T20:48:14.292603+0000","last_peered":"2023-11-28T20:48:14.292603+0000","last_clean":"2023-11-28T20:48:14.292603+0000","last_became_active":"2023-11-28T20:48:14.277236+0000","last_became_peered":"2023-11-28T20:48:14.277236+0000","last_unstale":"2023-11-28T20:48:14.292603+0000","last_undegraded":"2023-11-28T20:48:14.292603+0000","last_fullsized":"2023-11-28T20:48:14.292603+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:38:11.116646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.298835+0000","last_change":"2023-11-28T20:48:14.295399+0000","last_active":"2023-11-28T20:48:14.298835+0000","last_peered":"2023-11-28T20:48:14.298835+0000","last_clean":"2023-11-28T20:48:14.298835+0000","last_became_active":"2023-11-28T20:48:14.294890+0000","last_became_peered":"2023-11-28T20:48:14.294890+0000","last_unstale":"2023-11-28T20:48:14.298835+0000","last_undegraded":"2023-11-28T20:48:14.298835+0000","last_fullsized":"2023-11-28T20:48:14.298835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:28.406912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259377+0000","last_change":"2023-11-28T20:48:14.289877+0000","last_active":"2023-11-28T20:48:15.259377+0000","last_peered":"2023-11-28T20:48:15.259377+0000","last_clean":"2023-11-28T20:48:15.259377+0000","last_became_active":"2023-11-28T20:48:14.289573+0000","last_became_peered":"2023-11-28T20:48:14.289573+0000","last_unstale":"2023-11-28T20:48:15.259377+0000","last_undegraded":"2023-11-28T20:48:15.259377+0000","last_fullsized":"2023-11-28T20:48:15.259377+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:57.242902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256526+0000","last_change":"2023-11-28T20:48:14.282362+0000","last_active":"2023-11-28T20:48:15.256526+0000","last_peered":"2023-11-28T20:48:15.256526+0000","last_clean":"2023-11-28T20:48:15.256526+0000","last_became_active":"2023-11-28T20:48:14.281383+0000","last_became_peered":"2023-11-28T20:48:14.281383+0000","last_unstale":"2023-11-28T20:48:15.256526+0000","last_undegraded":"2023-11-28T20:48:15.256526+0000","last_fullsized":"2023-11-28T20:48:15.256526+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:51.256993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568854+0000","last_change":"2023-11-28T20:48:14.283953+0000","last_active":"2023-11-28T20:48:15.568854+0000","last_peered":"2023-11-28T20:48:15.568854+0000","last_clean":"2023-11-28T20:48:15.568854+0000","last_became_active":"2023-11-28T20:48:14.283674+0000","last_became_peered":"2023-11-28T20:48:14.283674+0000","last_unstale":"2023-11-28T20:48:15.568854+0000","last_undegraded":"2023-11-28T20:48:15.568854+0000","last_fullsized":"2023-11-28T20:48:15.568854+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:30:53.807023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.301497+0000","last_change":"2023-11-28T20:48:14.298353+0000","last_active":"2023-11-28T20:48:14.301497+0000","last_peered":"2023-11-28T20:48:14.301497+0000","last_clean":"2023-11-28T20:48:14.301497+0000","last_became_active":"2023-11-28T20:48:14.297946+0000","last_became_peered":"2023-11-28T20:48:14.297946+0000","last_unstale":"2023-11-28T20:48:14.301497+0000","last_undegraded":"2023-11-28T20:48:14.301497+0000","last_fullsized":"2023-11-28T20:48:14.301497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:19:15.637568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537114+0000","last_change":"2023-11-28T20:48:14.288751+0000","last_active":"2023-11-28T20:48:15.537114+0000","last_peered":"2023-11-28T20:48:15.537114+0000","last_clean":"2023-11-28T20:48:15.537114+0000","last_became_active":"2023-11-28T20:48:14.288459+0000","last_became_peered":"2023-11-28T20:48:14.288459+0000","last_unstale":"2023-11-28T20:48:15.537114+0000","last_undegraded":"2023-11-28T20:48:15.537114+0000","last_fullsized":"2023-11-28T20:48:15.537114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:01.569483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292111+0000","last_change":"2023-11-28T20:48:14.285700+0000","last_active":"2023-11-28T20:48:14.292111+0000","last_peered":"2023-11-28T20:48:14.292111+0000","last_clean":"2023-11-28T20:48:14.292111+0000","last_became_active":"2023-11-28T20:48:14.284963+0000","last_became_peered":"2023-11-28T20:48:14.284963+0000","last_unstale":"2023-11-28T20:48:14.292111+0000","last_undegraded":"2023-11-28T20:48:14.292111+0000","last_fullsized":"2023-11-28T20:48:14.292111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:36:16.048653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.297515+0000","last_change":"2023-11-28T20:48:14.292228+0000","last_active":"2023-11-28T20:48:14.297515+0000","last_peered":"2023-11-28T20:48:14.297515+0000","last_clean":"2023-11-28T20:48:14.297515+0000","last_became_active":"2023-11-28T20:48:14.291887+0000","last_became_peered":"2023-11-28T20:48:14.291887+0000","last_unstale":"2023-11-28T20:48:14.297515+0000","last_undegraded":"2023-11-28T20:48:14.297515+0000","last_fullsized":"2023-11-28T20:48:14.297515+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:40:45.271219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.536722+0000","last_change":"2023-11-28T20:48:14.287511+0000","last_active":"2023-11-28T20:48:15.536722+0000","last_peered":"2023-11-28T20:48:15.536722+0000","last_clean":"2023-11-28T20:48:15.536722+0000","last_became_active":"2023-11-28T20:48:14.287228+0000","last_became_peered":"2023-11-28T20:48:14.287228+0000","last_unstale":"2023-11-28T20:48:15.536722+0000","last_undegraded":"2023-11-28T20:48:15.536722+0000","last_fullsized":"2023-11-28T20:48:15.536722+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:59:59.225574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"70'4","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568760+0000","last_change":"2023-11-28T20:48:13.287357+0000","last_active":"2023-11-28T20:48:15.568760+0000","last_peered":"2023-11-28T20:48:15.568760+0000","last_clean":"2023-11-28T20:48:15.568760+0000","last_became_active":"2023-11-28T20:48:13.286961+0000","last_became_peered":"2023-11-28T20:48:13.286961+0000","last_unstale":"2023-11-28T20:48:15.568760+0000","last_undegraded":"2023-11-28T20:48:15.568760+0000","last_fullsized":"2023-11-28T20:48:15.568760+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:22.499460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.287786+0000","last_change":"2023-11-28T20:48:14.281605+0000","last_active":"2023-11-28T20:48:14.287786+0000","last_peered":"2023-11-28T20:48:14.287786+0000","last_clean":"2023-11-28T20:48:14.287786+0000","last_became_active":"2023-11-28T20:48:14.281286+0000","last_became_peered":"2023-11-28T20:48:14.281286+0000","last_unstale":"2023-11-28T20:48:14.287786+0000","last_undegraded":"2023-11-28T20:48:14.287786+0000","last_fullsized":"2023-11-28T20:48:14.287786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:19.149186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"70'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256726+0000","last_change":"2023-11-28T20:48:13.286953+0000","last_active":"2023-11-28T20:48:15.256726+0000","last_peered":"2023-11-28T20:48:15.256726+0000","last_clean":"2023-11-28T20:48:15.256726+0000","last_became_active":"2023-11-28T20:48:13.286689+0000","last_became_peered":"2023-11-28T20:48:13.286689+0000","last_unstale":"2023-11-28T20:48:15.256726+0000","last_undegraded":"2023-11-28T20:48:15.256726+0000","last_fullsized":"2023-11-28T20:48:15.256726+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:36.638766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568500+0000","last_change":"2023-11-28T20:48:14.281744+0000","last_active":"2023-11-28T20:48:15.568500+0000","last_peered":"2023-11-28T20:48:15.568500+0000","last_clean":"2023-11-28T20:48:15.568500+0000","last_became_active":"2023-11-28T20:48:14.280512+0000","last_became_peered":"2023-11-28T20:48:14.280512+0000","last_unstale":"2023-11-28T20:48:15.568500+0000","last_undegraded":"2023-11-28T20:48:15.568500+0000","last_fullsized":"2023-11-28T20:48:15.568500+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:13.400093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"70'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256655+0000","last_change":"2023-11-28T20:48:13.289848+0000","last_active":"2023-11-28T20:48:15.256655+0000","last_peered":"2023-11-28T20:48:15.256655+0000","last_clean":"2023-11-28T20:48:15.256655+0000","last_became_active":"2023-11-28T20:48:13.289551+0000","last_became_peered":"2023-11-28T20:48:13.289551+0000","last_unstale":"2023-11-28T20:48:15.256655+0000","last_undegraded":"2023-11-28T20:48:15.256655+0000","last_fullsized":"2023-11-28T20:48:15.256655+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:30:52.792140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258951+0000","last_change":"2023-11-28T20:48:14.288614+0000","last_active":"2023-11-28T20:48:15.258951+0000","last_peered":"2023-11-28T20:48:15.258951+0000","last_clean":"2023-11-28T20:48:15.258951+0000","last_became_active":"2023-11-28T20:48:14.287369+0000","last_became_peered":"2023-11-28T20:48:14.287369+0000","last_unstale":"2023-11-28T20:48:15.258951+0000","last_undegraded":"2023-11-28T20:48:15.258951+0000","last_fullsized":"2023-11-28T20:48:15.258951+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:33:23.202275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"70'2","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.319021+0000","last_change":"2023-11-28T20:48:13.287229+0000","last_active":"2023-11-28T20:48:14.319021+0000","last_peered":"2023-11-28T20:48:14.319021+0000","last_clean":"2023-11-28T20:48:14.319021+0000","last_became_active":"2023-11-28T20:48:13.286968+0000","last_became_peered":"2023-11-28T20:48:13.286968+0000","last_unstale":"2023-11-28T20:48:14.319021+0000","last_undegraded":"2023-11-28T20:48:14.319021+0000","last_fullsized":"2023-11-28T20:48:14.319021+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:29:16.329450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.262381+0000","last_change":"2023-11-28T20:48:14.280379+0000","last_active":"2023-11-28T20:48:15.262381+0000","last_peered":"2023-11-28T20:48:15.262381+0000","last_clean":"2023-11-28T20:48:15.262381+0000","last_became_active":"2023-11-28T20:48:14.279313+0000","last_became_peered":"2023-11-28T20:48:14.279313+0000","last_unstale":"2023-11-28T20:48:15.262381+0000","last_undegraded":"2023-11-28T20:48:15.262381+0000","last_fullsized":"2023-11-28T20:48:15.262381+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:55:36.332492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"70'1","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.315850+0000","last_change":"2023-11-28T20:48:13.295973+0000","last_active":"2023-11-28T20:48:14.315850+0000","last_peered":"2023-11-28T20:48:14.315850+0000","last_clean":"2023-11-28T20:48:14.315850+0000","last_became_active":"2023-11-28T20:48:13.295690+0000","last_became_peered":"2023-11-28T20:48:13.295690+0000","last_unstale":"2023-11-28T20:48:14.315850+0000","last_undegraded":"2023-11-28T20:48:14.315850+0000","last_fullsized":"2023-11-28T20:48:14.315850+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:42:16.982555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537696+0000","last_change":"2023-11-28T20:48:14.290004+0000","last_active":"2023-11-28T20:48:15.537696+0000","last_peered":"2023-11-28T20:48:15.537696+0000","last_clean":"2023-11-28T20:48:15.537696+0000","last_became_active":"2023-11-28T20:48:14.289697+0000","last_became_peered":"2023-11-28T20:48:14.289697+0000","last_unstale":"2023-11-28T20:48:15.537696+0000","last_undegraded":"2023-11-28T20:48:15.537696+0000","last_fullsized":"2023-11-28T20:48:15.537696+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:01:42.653989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"70'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261599+0000","last_change":"2023-11-28T20:48:13.291137+0000","last_active":"2023-11-28T20:48:15.261599+0000","last_peered":"2023-11-28T20:48:15.261599+0000","last_clean":"2023-11-28T20:48:15.261599+0000","last_became_active":"2023-11-28T20:48:13.290749+0000","last_became_peered":"2023-11-28T20:48:13.290749+0000","last_unstale":"2023-11-28T20:48:15.261599+0000","last_undegraded":"2023-11-28T20:48:15.261599+0000","last_fullsized":"2023-11-28T20:48:15.261599+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:59.759169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259534+0000","last_change":"2023-11-28T20:48:14.287523+0000","last_active":"2023-11-28T20:48:15.259534+0000","last_peered":"2023-11-28T20:48:15.259534+0000","last_clean":"2023-11-28T20:48:15.259534+0000","last_became_active":"2023-11-28T20:48:14.287214+0000","last_became_peered":"2023-11-28T20:48:14.287214+0000","last_unstale":"2023-11-28T20:48:15.259534+0000","last_undegraded":"2023-11-28T20:48:15.259534+0000","last_fullsized":"2023-11-28T20:48:15.259534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:24:51.192998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568365+0000","last_change":"2023-11-28T20:48:13.294888+0000","last_active":"2023-11-28T20:48:15.568365+0000","last_peered":"2023-11-28T20:48:15.568365+0000","last_clean":"2023-11-28T20:48:15.568365+0000","last_became_active":"2023-11-28T20:48:13.293937+0000","last_became_peered":"2023-11-28T20:48:13.293937+0000","last_unstale":"2023-11-28T20:48:15.568365+0000","last_undegraded":"2023-11-28T20:48:15.568365+0000","last_fullsized":"2023-11-28T20:48:15.568365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:46:31.679930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537423+0000","last_change":"2023-11-28T20:48:14.293165+0000","last_active":"2023-11-28T20:48:15.537423+0000","last_peered":"2023-11-28T20:48:15.537423+0000","last_clean":"2023-11-28T20:48:15.537423+0000","last_became_active":"2023-11-28T20:48:14.292979+0000","last_became_peered":"2023-11-28T20:48:14.292979+0000","last_unstale":"2023-11-28T20:48:15.537423+0000","last_undegraded":"2023-11-28T20:48:15.537423+0000","last_fullsized":"2023-11-28T20:48:15.537423+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:41:43.088862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.293861+0000","last_change":"2023-11-28T20:48:13.284360+0000","last_active":"2023-11-28T20:48:14.293861+0000","last_peered":"2023-11-28T20:48:14.293861+0000","last_clean":"2023-11-28T20:48:14.293861+0000","last_became_active":"2023-11-28T20:48:13.283969+0000","last_became_peered":"2023-11-28T20:48:13.283969+0000","last_unstale":"2023-11-28T20:48:14.293861+0000","last_undegraded":"2023-11-28T20:48:14.293861+0000","last_fullsized":"2023-11-28T20:48:14.293861+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:25:13.938748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261391+0000","last_change":"2023-11-28T20:48:14.286992+0000","last_active":"2023-11-28T20:48:15.261391+0000","last_peered":"2023-11-28T20:48:15.261391+0000","last_clean":"2023-11-28T20:48:15.261391+0000","last_became_active":"2023-11-28T20:48:14.286691+0000","last_became_peered":"2023-11-28T20:48:14.286691+0000","last_unstale":"2023-11-28T20:48:15.261391+0000","last_undegraded":"2023-11-28T20:48:15.261391+0000","last_fullsized":"2023-11-28T20:48:15.261391+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:50:20.379297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537658+0000","last_change":"2023-11-28T20:48:13.294632+0000","last_active":"2023-11-28T20:48:15.537658+0000","last_peered":"2023-11-28T20:48:15.537658+0000","last_clean":"2023-11-28T20:48:15.537658+0000","last_became_active":"2023-11-28T20:48:13.294324+0000","last_became_peered":"2023-11-28T20:48:13.294324+0000","last_unstale":"2023-11-28T20:48:15.537658+0000","last_undegraded":"2023-11-28T20:48:15.537658+0000","last_fullsized":"2023-11-28T20:48:15.537658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:46:41.545296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.287768+0000","last_change":"2023-11-28T20:48:14.281436+0000","last_active":"2023-11-28T20:48:14.287768+0000","last_peered":"2023-11-28T20:48:14.287768+0000","last_clean":"2023-11-28T20:48:14.287768+0000","last_became_active":"2023-11-28T20:48:14.281119+0000","last_became_peered":"2023-11-28T20:48:14.281119+0000","last_unstale":"2023-11-28T20:48:14.287768+0000","last_undegraded":"2023-11-28T20:48:14.287768+0000","last_fullsized":"2023-11-28T20:48:14.287768+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:20:38.804327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261888+0000","last_change":"2023-11-28T20:48:13.286426+0000","last_active":"2023-11-28T20:48:15.261888+0000","last_peered":"2023-11-28T20:48:15.261888+0000","last_clean":"2023-11-28T20:48:15.261888+0000","last_became_active":"2023-11-28T20:48:13.286100+0000","last_became_peered":"2023-11-28T20:48:13.286100+0000","last_unstale":"2023-11-28T20:48:15.261888+0000","last_undegraded":"2023-11-28T20:48:15.261888+0000","last_fullsized":"2023-11-28T20:48:15.261888+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:31:44.021298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.302317+0000","last_change":"2023-11-28T20:48:14.300075+0000","last_active":"2023-11-28T20:48:14.302317+0000","last_peered":"2023-11-28T20:48:14.302317+0000","last_clean":"2023-11-28T20:48:14.302317+0000","last_became_active":"2023-11-28T20:48:14.299796+0000","last_became_peered":"2023-11-28T20:48:14.299796+0000","last_unstale":"2023-11-28T20:48:14.302317+0000","last_undegraded":"2023-11-28T20:48:14.302317+0000","last_fullsized":"2023-11-28T20:48:14.302317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:24:17.747646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"70'3","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537780+0000","last_change":"2023-11-28T20:48:13.296795+0000","last_active":"2023-11-28T20:48:15.537780+0000","last_peered":"2023-11-28T20:48:15.537780+0000","last_clean":"2023-11-28T20:48:15.537780+0000","last_became_active":"2023-11-28T20:48:13.296512+0000","last_became_peered":"2023-11-28T20:48:13.296512+0000","last_unstale":"2023-11-28T20:48:15.537780+0000","last_undegraded":"2023-11-28T20:48:15.537780+0000","last_fullsized":"2023-11-28T20:48:15.537780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:18.134304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.262780+0000","last_change":"2023-11-28T20:48:14.270755+0000","last_active":"2023-11-28T20:48:15.262780+0000","last_peered":"2023-11-28T20:48:15.262780+0000","last_clean":"2023-11-28T20:48:15.262780+0000","last_became_active":"2023-11-28T20:48:14.269949+0000","last_became_peered":"2023-11-28T20:48:14.269949+0000","last_unstale":"2023-11-28T20:48:15.262780+0000","last_undegraded":"2023-11-28T20:48:15.262780+0000","last_fullsized":"2023-11-28T20:48:15.262780+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:10:43.157562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"70'6","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259314+0000","last_change":"2023-11-28T20:48:13.269047+0000","last_active":"2023-11-28T20:48:15.259314+0000","last_peered":"2023-11-28T20:48:15.259314+0000","last_clean":"2023-11-28T20:48:15.259314+0000","last_became_active":"2023-11-28T20:48:13.268517+0000","last_became_peered":"2023-11-28T20:48:13.268517+0000","last_unstale":"2023-11-28T20:48:15.259314+0000","last_undegraded":"2023-11-28T20:48:15.259314+0000","last_fullsized":"2023-11-28T20:48:15.259314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:19:14.197446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.291945+0000","last_change":"2023-11-28T20:48:14.286659+0000","last_active":"2023-11-28T20:48:14.291945+0000","last_peered":"2023-11-28T20:48:14.291945+0000","last_clean":"2023-11-28T20:48:14.291945+0000","last_became_active":"2023-11-28T20:48:14.286132+0000","last_became_peered":"2023-11-28T20:48:14.286132+0000","last_unstale":"2023-11-28T20:48:14.291945+0000","last_undegraded":"2023-11-28T20:48:14.291945+0000","last_fullsized":"2023-11-28T20:48:14.291945+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:58:59.988274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"70'2","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.316470+0000","last_change":"2023-11-28T20:48:13.295480+0000","last_active":"2023-11-28T20:48:14.316470+0000","last_peered":"2023-11-28T20:48:14.316470+0000","last_clean":"2023-11-28T20:48:14.316470+0000","last_became_active":"2023-11-28T20:48:13.294923+0000","last_became_peered":"2023-11-28T20:48:13.294923+0000","last_unstale":"2023-11-28T20:48:14.316470+0000","last_undegraded":"2023-11-28T20:48:14.316470+0000","last_fullsized":"2023-11-28T20:48:14.316470+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:10.011402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.302246+0000","last_change":"2023-11-28T20:48:14.300360+0000","last_active":"2023-11-28T20:48:14.302246+0000","last_peered":"2023-11-28T20:48:14.302246+0000","last_clean":"2023-11-28T20:48:14.302246+0000","last_became_active":"2023-11-28T20:48:14.299480+0000","last_became_peered":"2023-11-28T20:48:14.299480+0000","last_unstale":"2023-11-28T20:48:14.302246+0000","last_undegraded":"2023-11-28T20:48:14.302246+0000","last_fullsized":"2023-11-28T20:48:14.302246+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:05:15.443215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568708+0000","last_change":"2023-11-28T20:48:13.291841+0000","last_active":"2023-11-28T20:48:15.568708+0000","last_peered":"2023-11-28T20:48:15.568708+0000","last_clean":"2023-11-28T20:48:15.568708+0000","last_became_active":"2023-11-28T20:48:13.291534+0000","last_became_peered":"2023-11-28T20:48:13.291534+0000","last_unstale":"2023-11-28T20:48:15.568708+0000","last_undegraded":"2023-11-28T20:48:15.568708+0000","last_fullsized":"2023-11-28T20:48:15.568708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:20:34.016942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259410+0000","last_change":"2023-11-28T20:48:14.291725+0000","last_active":"2023-11-28T20:48:15.259410+0000","last_peered":"2023-11-28T20:48:15.259410+0000","last_clean":"2023-11-28T20:48:15.259410+0000","last_became_active":"2023-11-28T20:48:14.290802+0000","last_became_peered":"2023-11-28T20:48:14.290802+0000","last_unstale":"2023-11-28T20:48:15.259410+0000","last_undegraded":"2023-11-28T20:48:15.259410+0000","last_fullsized":"2023-11-28T20:48:15.259410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:01.558006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.536992+0000","last_change":"2023-11-28T20:48:13.294280+0000","last_active":"2023-11-28T20:48:15.536992+0000","last_peered":"2023-11-28T20:48:15.536992+0000","last_clean":"2023-11-28T20:48:15.536992+0000","last_became_active":"2023-11-28T20:48:13.293809+0000","last_became_peered":"2023-11-28T20:48:13.293809+0000","last_unstale":"2023-11-28T20:48:15.536992+0000","last_undegraded":"2023-11-28T20:48:15.536992+0000","last_fullsized":"2023-11-28T20:48:15.536992+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:40:44.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292257+0000","last_change":"2023-11-28T20:48:14.289490+0000","last_active":"2023-11-28T20:48:14.292257+0000","last_peered":"2023-11-28T20:48:14.292257+0000","last_clean":"2023-11-28T20:48:14.292257+0000","last_became_active":"2023-11-28T20:48:14.286645+0000","last_became_peered":"2023-11-28T20:48:14.286645+0000","last_unstale":"2023-11-28T20:48:14.292257+0000","last_undegraded":"2023-11-28T20:48:14.292257+0000","last_fullsized":"2023-11-28T20:48:14.292257+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:19:50.350147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"70'8","reported_seq":20,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259107+0000","last_change":"2023-11-28T20:48:13.271079+0000","last_active":"2023-11-28T20:48:15.259107+0000","last_peered":"2023-11-28T20:48:15.259107+0000","last_clean":"2023-11-28T20:48:15.259107+0000","last_became_active":"2023-11-28T20:48:13.270367+0000","last_became_peered":"2023-11-28T20:48:13.270367+0000","last_unstale":"2023-11-28T20:48:15.259107+0000","last_undegraded":"2023-11-28T20:48:15.259107+0000","last_fullsized":"2023-11-28T20:48:15.259107+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:45:19.387368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259071+0000","last_change":"2023-11-28T20:48:14.289522+0000","last_active":"2023-11-28T20:48:15.259071+0000","last_peered":"2023-11-28T20:48:15.259071+0000","last_clean":"2023-11-28T20:48:15.259071+0000","last_became_active":"2023-11-28T20:48:14.286425+0000","last_became_peered":"2023-11-28T20:48:14.286425+0000","last_unstale":"2023-11-28T20:48:15.259071+0000","last_undegraded":"2023-11-28T20:48:15.259071+0000","last_fullsized":"2023-11-28T20:48:15.259071+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:55.537941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"70'1","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.316030+0000","last_change":"2023-11-28T20:48:13.290621+0000","last_active":"2023-11-28T20:48:14.316030+0000","last_peered":"2023-11-28T20:48:14.316030+0000","last_clean":"2023-11-28T20:48:14.316030+0000","last_became_active":"2023-11-28T20:48:13.290328+0000","last_became_peered":"2023-11-28T20:48:13.290328+0000","last_unstale":"2023-11-28T20:48:14.316030+0000","last_undegraded":"2023-11-28T20:48:14.316030+0000","last_fullsized":"2023-11-28T20:48:14.316030+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:18.134304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.255975+0000","last_change":"2023-11-28T20:48:14.295333+0000","last_active":"2023-11-28T20:48:15.255975+0000","last_peered":"2023-11-28T20:48:15.255975+0000","last_clean":"2023-11-28T20:48:15.255975+0000","last_became_active":"2023-11-28T20:48:14.295094+0000","last_became_peered":"2023-11-28T20:48:14.295094+0000","last_unstale":"2023-11-28T20:48:15.255975+0000","last_undegraded":"2023-11-28T20:48:15.255975+0000","last_fullsized":"2023-11-28T20:48:15.255975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:19:07.557767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"70'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.650257+0000","last_change":"2023-11-28T20:48:13.280005+0000","last_active":"2023-11-28T20:48:15.650257+0000","last_peered":"2023-11-28T20:48:15.650257+0000","last_clean":"2023-11-28T20:48:15.650257+0000","last_became_active":"2023-11-28T20:48:13.277650+0000","last_became_peered":"2023-11-28T20:48:13.277650+0000","last_unstale":"2023-11-28T20:48:15.650257+0000","last_undegraded":"2023-11-28T20:48:15.650257+0000","last_fullsized":"2023-11-28T20:48:15.650257+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:31:06.616167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261386+0000","last_change":"2023-11-28T20:48:14.298106+0000","last_active":"2023-11-28T20:48:15.261386+0000","last_peered":"2023-11-28T20:48:15.261386+0000","last_clean":"2023-11-28T20:48:15.261386+0000","last_became_active":"2023-11-28T20:48:14.297802+0000","last_became_peered":"2023-11-28T20:48:14.297802+0000","last_unstale":"2023-11-28T20:48:15.261386+0000","last_undegraded":"2023-11-28T20:48:15.261386+0000","last_fullsized":"2023-11-28T20:48:15.261386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:04:51.256993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"70'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.569061+0000","last_change":"2023-11-28T20:48:13.288101+0000","last_active":"2023-11-28T20:48:15.569061+0000","last_peered":"2023-11-28T20:48:15.569061+0000","last_clean":"2023-11-28T20:48:15.569061+0000","last_became_active":"2023-11-28T20:48:13.286895+0000","last_became_peered":"2023-11-28T20:48:13.286895+0000","last_unstale":"2023-11-28T20:48:15.569061+0000","last_undegraded":"2023-11-28T20:48:15.569061+0000","last_fullsized":"2023-11-28T20:48:15.569061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:14.595108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256525+0000","last_change":"2023-11-28T20:48:14.278974+0000","last_active":"2023-11-28T20:48:15.256525+0000","last_peered":"2023-11-28T20:48:15.256525+0000","last_clean":"2023-11-28T20:48:15.256525+0000","last_became_active":"2023-11-28T20:48:14.278539+0000","last_became_peered":"2023-11-28T20:48:14.278539+0000","last_unstale":"2023-11-28T20:48:15.256525+0000","last_undegraded":"2023-11-28T20:48:15.256525+0000","last_fullsized":"2023-11-28T20:48:15.256525+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:31:45.036181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568946+0000","last_change":"2023-11-28T20:48:13.287730+0000","last_active":"2023-11-28T20:48:15.568946+0000","last_peered":"2023-11-28T20:48:15.568946+0000","last_clean":"2023-11-28T20:48:15.568946+0000","last_became_active":"2023-11-28T20:48:13.287417+0000","last_became_peered":"2023-11-28T20:48:13.287417+0000","last_unstale":"2023-11-28T20:48:15.568946+0000","last_undegraded":"2023-11-28T20:48:15.568946+0000","last_fullsized":"2023-11-28T20:48:15.568946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:20:37.546954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.283479+0000","last_change":"2023-11-28T20:48:14.277949+0000","last_active":"2023-11-28T20:48:14.283479+0000","last_peered":"2023-11-28T20:48:14.283479+0000","last_clean":"2023-11-28T20:48:14.283479+0000","last_became_active":"2023-11-28T20:48:14.276281+0000","last_became_peered":"2023-11-28T20:48:14.276281+0000","last_unstale":"2023-11-28T20:48:14.283479+0000","last_undegraded":"2023-11-28T20:48:14.283479+0000","last_fullsized":"2023-11-28T20:48:14.283479+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:23:38.447492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"70'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256540+0000","last_change":"2023-11-28T20:48:13.283356+0000","last_active":"2023-11-28T20:48:15.256540+0000","last_peered":"2023-11-28T20:48:15.256540+0000","last_clean":"2023-11-28T20:48:15.256540+0000","last_became_active":"2023-11-28T20:48:13.283067+0000","last_became_peered":"2023-11-28T20:48:13.283067+0000","last_unstale":"2023-11-28T20:48:15.256540+0000","last_undegraded":"2023-11-28T20:48:15.256540+0000","last_fullsized":"2023-11-28T20:48:15.256540+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:43:44.562922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568900+0000","last_change":"2023-11-28T20:48:14.282096+0000","last_active":"2023-11-28T20:48:15.568900+0000","last_peered":"2023-11-28T20:48:15.568900+0000","last_clean":"2023-11-28T20:48:15.568900+0000","last_became_active":"2023-11-28T20:48:14.281219+0000","last_became_peered":"2023-11-28T20:48:14.281219+0000","last_unstale":"2023-11-28T20:48:15.568900+0000","last_undegraded":"2023-11-28T20:48:15.568900+0000","last_fullsized":"2023-11-28T20:48:15.568900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:15:23.890200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258855+0000","last_change":"2023-11-28T20:48:13.279096+0000","last_active":"2023-11-28T20:48:15.258855+0000","last_peered":"2023-11-28T20:48:15.258855+0000","last_clean":"2023-11-28T20:48:15.258855+0000","last_became_active":"2023-11-28T20:48:13.278805+0000","last_became_peered":"2023-11-28T20:48:13.278805+0000","last_unstale":"2023-11-28T20:48:15.258855+0000","last_undegraded":"2023-11-28T20:48:15.258855+0000","last_fullsized":"2023-11-28T20:48:15.258855+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:29:15.712862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.301351+0000","last_change":"2023-11-28T20:48:14.298350+0000","last_active":"2023-11-28T20:48:14.301351+0000","last_peered":"2023-11-28T20:48:14.301351+0000","last_clean":"2023-11-28T20:48:14.301351+0000","last_became_active":"2023-11-28T20:48:14.298024+0000","last_became_peered":"2023-11-28T20:48:14.298024+0000","last_unstale":"2023-11-28T20:48:14.301351+0000","last_undegraded":"2023-11-28T20:48:14.301351+0000","last_fullsized":"2023-11-28T20:48:14.301351+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:50.996911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292721+0000","last_change":"2023-11-28T20:48:13.292422+0000","last_active":"2023-11-28T20:48:14.292721+0000","last_peered":"2023-11-28T20:48:14.292721+0000","last_clean":"2023-11-28T20:48:14.292721+0000","last_became_active":"2023-11-28T20:48:13.291292+0000","last_became_peered":"2023-11-28T20:48:13.291292+0000","last_unstale":"2023-11-28T20:48:14.292721+0000","last_undegraded":"2023-11-28T20:48:14.292721+0000","last_fullsized":"2023-11-28T20:48:14.292721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:42.446875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256841+0000","last_change":"2023-11-28T20:48:14.290101+0000","last_active":"2023-11-28T20:48:15.256841+0000","last_peered":"2023-11-28T20:48:15.256841+0000","last_clean":"2023-11-28T20:48:15.256841+0000","last_became_active":"2023-11-28T20:48:14.289753+0000","last_became_peered":"2023-11-28T20:48:14.289753+0000","last_unstale":"2023-11-28T20:48:15.256841+0000","last_undegraded":"2023-11-28T20:48:15.256841+0000","last_fullsized":"2023-11-28T20:48:15.256841+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:55:36.332492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.650335+0000","last_change":"2023-11-28T20:48:13.294473+0000","last_active":"2023-11-28T20:48:15.650335+0000","last_peered":"2023-11-28T20:48:15.650335+0000","last_clean":"2023-11-28T20:48:15.650335+0000","last_became_active":"2023-11-28T20:48:13.293547+0000","last_became_peered":"2023-11-28T20:48:13.293547+0000","last_unstale":"2023-11-28T20:48:15.650335+0000","last_undegraded":"2023-11-28T20:48:15.650335+0000","last_fullsized":"2023-11-28T20:48:15.650335+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:28.650433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.297540+0000","last_change":"2023-11-28T20:48:14.292571+0000","last_active":"2023-11-28T20:48:14.297540+0000","last_peered":"2023-11-28T20:48:14.297540+0000","last_clean":"2023-11-28T20:48:14.297540+0000","last_became_active":"2023-11-28T20:48:14.292252+0000","last_became_peered":"2023-11-28T20:48:14.292252+0000","last_unstale":"2023-11-28T20:48:14.297540+0000","last_undegraded":"2023-11-28T20:48:14.297540+0000","last_fullsized":"2023-11-28T20:48:14.297540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:39.563576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"70'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.569093+0000","last_change":"2023-11-28T20:48:13.293223+0000","last_active":"2023-11-28T20:48:15.569093+0000","last_peered":"2023-11-28T20:48:15.569093+0000","last_clean":"2023-11-28T20:48:15.569093+0000","last_became_active":"2023-11-28T20:48:13.292528+0000","last_became_peered":"2023-11-28T20:48:13.292528+0000","last_unstale":"2023-11-28T20:48:15.569093+0000","last_undegraded":"2023-11-28T20:48:15.569093+0000","last_fullsized":"2023-11-28T20:48:15.569093+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:39:26.119216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.256260+0000","last_change":"2023-11-28T20:48:14.269789+0000","last_active":"2023-11-28T20:48:15.256260+0000","last_peered":"2023-11-28T20:48:15.256260+0000","last_clean":"2023-11-28T20:48:15.256260+0000","last_became_active":"2023-11-28T20:48:14.269188+0000","last_became_peered":"2023-11-28T20:48:14.269188+0000","last_unstale":"2023-11-28T20:48:15.256260+0000","last_undegraded":"2023-11-28T20:48:15.256260+0000","last_fullsized":"2023-11-28T20:48:15.256260+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:36:00.774051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.537817+0000","last_change":"2023-11-28T20:48:13.296112+0000","last_active":"2023-11-28T20:48:15.537817+0000","last_peered":"2023-11-28T20:48:15.537817+0000","last_clean":"2023-11-28T20:48:15.537817+0000","last_became_active":"2023-11-28T20:48:13.295794+0000","last_became_peered":"2023-11-28T20:48:13.295794+0000","last_unstale":"2023-11-28T20:48:15.537817+0000","last_undegraded":"2023-11-28T20:48:15.537817+0000","last_fullsized":"2023-11-28T20:48:15.537817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:38.548693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259151+0000","last_change":"2023-11-28T20:48:14.286675+0000","last_active":"2023-11-28T20:48:15.259151+0000","last_peered":"2023-11-28T20:48:15.259151+0000","last_clean":"2023-11-28T20:48:15.259151+0000","last_became_active":"2023-11-28T20:48:14.284310+0000","last_became_peered":"2023-11-28T20:48:14.284310+0000","last_unstale":"2023-11-28T20:48:15.259151+0000","last_undegraded":"2023-11-28T20:48:15.259151+0000","last_fullsized":"2023-11-28T20:48:15.259151+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:50:56.809237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.295777+0000","last_change":"2023-11-28T20:48:13.280959+0000","last_active":"2023-11-28T20:48:14.295777+0000","last_peered":"2023-11-28T20:48:14.295777+0000","last_clean":"2023-11-28T20:48:14.295777+0000","last_became_active":"2023-11-28T20:48:13.280665+0000","last_became_peered":"2023-11-28T20:48:13.280665+0000","last_unstale":"2023-11-28T20:48:14.295777+0000","last_undegraded":"2023-11-28T20:48:14.295777+0000","last_fullsized":"2023-11-28T20:48:14.295777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:22:50.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.295930+0000","last_change":"2023-11-28T20:48:14.293584+0000","last_active":"2023-11-28T20:48:14.295930+0000","last_peered":"2023-11-28T20:48:14.295930+0000","last_clean":"2023-11-28T20:48:14.295930+0000","last_became_active":"2023-11-28T20:48:14.293283+0000","last_became_peered":"2023-11-28T20:48:14.293283+0000","last_unstale":"2023-11-28T20:48:14.295930+0000","last_undegraded":"2023-11-28T20:48:14.295930+0000","last_fullsized":"2023-11-28T20:48:14.295930+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:29:57.111937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259063+0000","last_change":"2023-11-28T20:48:13.270826+0000","last_active":"2023-11-28T20:48:15.259063+0000","last_peered":"2023-11-28T20:48:15.259063+0000","last_clean":"2023-11-28T20:48:15.259063+0000","last_became_active":"2023-11-28T20:48:13.268261+0000","last_became_peered":"2023-11-28T20:48:13.268261+0000","last_unstale":"2023-11-28T20:48:15.259063+0000","last_undegraded":"2023-11-28T20:48:15.259063+0000","last_fullsized":"2023-11-28T20:48:15.259063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:58.165702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.289900+0000","last_change":"2023-11-28T20:48:14.285812+0000","last_active":"2023-11-28T20:48:14.289900+0000","last_peered":"2023-11-28T20:48:14.289900+0000","last_clean":"2023-11-28T20:48:14.289900+0000","last_became_active":"2023-11-28T20:48:14.285543+0000","last_became_peered":"2023-11-28T20:48:14.285543+0000","last_unstale":"2023-11-28T20:48:14.289900+0000","last_undegraded":"2023-11-28T20:48:14.289900+0000","last_fullsized":"2023-11-28T20:48:14.289900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:46:42.560179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.258981+0000","last_change":"2023-11-28T20:48:13.286299+0000","last_active":"2023-11-28T20:48:15.258981+0000","last_peered":"2023-11-28T20:48:15.258981+0000","last_clean":"2023-11-28T20:48:15.258981+0000","last_became_active":"2023-11-28T20:48:13.285969+0000","last_became_peered":"2023-11-28T20:48:13.285969+0000","last_unstale":"2023-11-28T20:48:15.258981+0000","last_undegraded":"2023-11-28T20:48:15.258981+0000","last_fullsized":"2023-11-28T20:48:15.258981+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:34:17.163205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.292190+0000","last_change":"2023-11-28T20:48:14.289317+0000","last_active":"2023-11-28T20:48:14.292190+0000","last_peered":"2023-11-28T20:48:14.292190+0000","last_clean":"2023-11-28T20:48:14.292190+0000","last_became_active":"2023-11-28T20:48:14.289016+0000","last_became_peered":"2023-11-28T20:48:14.289016+0000","last_unstale":"2023-11-28T20:48:14.292190+0000","last_undegraded":"2023-11-28T20:48:14.292190+0000","last_fullsized":"2023-11-28T20:48:14.292190+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:00.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.261462+0000","last_change":"2023-11-28T20:48:13.283031+0000","last_active":"2023-11-28T20:48:15.261462+0000","last_peered":"2023-11-28T20:48:15.261462+0000","last_clean":"2023-11-28T20:48:15.261462+0000","last_became_active":"2023-11-28T20:48:13.282643+0000","last_became_peered":"2023-11-28T20:48:13.282643+0000","last_unstale":"2023-11-28T20:48:15.261462+0000","last_undegraded":"2023-11-28T20:48:15.261462+0000","last_fullsized":"2023-11-28T20:48:15.261462+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:22:27.604632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.259652+0000","last_change":"2023-11-28T20:48:14.288176+0000","last_active":"2023-11-28T20:48:15.259652+0000","last_peered":"2023-11-28T20:48:15.259652+0000","last_clean":"2023-11-28T20:48:15.259652+0000","last_became_active":"2023-11-28T20:48:14.287913+0000","last_became_peered":"2023-11-28T20:48:14.287913+0000","last_unstale":"2023-11-28T20:48:15.259652+0000","last_undegraded":"2023-11-28T20:48:15.259652+0000","last_fullsized":"2023-11-28T20:48:15.259652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:12:49.555700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.270145+0000","last_change":"2023-11-28T20:48:13.286376+0000","last_active":"2023-11-28T20:48:14.270145+0000","last_peered":"2023-11-28T20:48:14.270145+0000","last_clean":"2023-11-28T20:48:14.270145+0000","last_became_active":"2023-11-28T20:48:13.286102+0000","last_became_peered":"2023-11-28T20:48:13.286102+0000","last_unstale":"2023-11-28T20:48:14.270145+0000","last_undegraded":"2023-11-28T20:48:14.270145+0000","last_fullsized":"2023-11-28T20:48:14.270145+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:58.165702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T20:48:15.568402+0000","last_change":"2023-11-28T20:48:14.279143+0000","last_active":"2023-11-28T20:48:15.568402+0000","last_peered":"2023-11-28T20:48:15.568402+0000","last_clean":"2023-11-28T20:48:15.568402+0000","last_became_active":"2023-11-28T20:48:14.278881+0000","last_became_peered":"2023-11-28T20:48:14.278881+0000","last_unstale":"2023-11-28T20:48:15.568402+0000","last_undegraded":"2023-11-28T20:48:15.568402+0000","last_fullsized":"2023-11-28T20:48:15.568402+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:13.248650+0000","last_clean_scrub_stamp":"2023-11-28T20:48:13.248650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:41:13.464014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.296264+0000","last_change":"2023-11-28T20:48:13.294843+0000","last_active":"2023-11-28T20:48:14.296264+0000","last_peered":"2023-11-28T20:48:14.296264+0000","last_clean":"2023-11-28T20:48:14.296264+0000","last_became_active":"2023-11-28T20:48:13.294124+0000","last_became_peered":"2023-11-28T20:48:13.294124+0000","last_unstale":"2023-11-28T20:48:14.296264+0000","last_undegraded":"2023-11-28T20:48:14.296264+0000","last_fullsized":"2023-11-28T20:48:14.296264+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:48:12.233767+0000","last_clean_scrub_stamp":"2023-11-28T20:48:12.233767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:16:00.231441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"59'139","reported_seq":295,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-11-28T20:48:14.274854+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:48:14.274854+0000","last_peered":"2023-11-28T20:48:14.274854+0000","last_clean":"2023-11-28T20:48:14.274854+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:48:14.274854+0000","last_undegraded":"2023-11-28T20:48:14.274854+0000","last_fullsized":"2023-11-28T20:48:14.274854+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575023,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58972,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312868,"statfs":{"total":96636764160,"available":96576376832,"internally_reserved":0,"allocated":1454080,"data_stored":1013887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.202,"15min":1.1879999999999999},"min":{"1min":0.68200000000000005,"5min":0.625,"15min":0.51900000000000002},"max":{"1min":1.6140000000000001,"5min":1.7,"15min":2.23},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1910000000000001,"15min":1.1679999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.518,"5min":1.726,"15min":1.8660000000000001},"last":0.82399999999999995}]},{"osd":1,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.1990000000000001,"15min":1.1719999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.591,"5min":1.744,"15min":1.744},"last":1.399},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.2330000000000001,"15min":1.2250000000000001},"min":{"1min":0.80500000000000005,"5min":0.77200000000000002,"15min":0.63800000000000001},"max":{"1min":1.486,"5min":1.835,"15min":2.3599999999999999},"last":1.4359999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.268,"15min":1.3029999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.58799999999999997},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.9359999999999999},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.234,"5min":1.2769999999999999,"15min":1.288},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.704,"5min":1.8069999999999999,"15min":2.137},"last":0.96199999999999997}]},{"osd":3,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.296,"15min":1.2769999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7729999999999999,"5min":1.8460000000000001,"15min":2.1949999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.369,"5min":1.401,"15min":1.4059999999999999},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.68300000000000005},"max":{"1min":1.6459999999999999,"5min":1.758,"15min":2.5259999999999998},"last":1.5760000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.286,"15min":1.266},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7989999999999999,"5min":1.905,"15min":1.919},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.3400000000000001,"15min":1.3380000000000001},"min":{"1min":0.89100000000000001,"5min":0.73399999999999999,"15min":0.51500000000000001},"max":{"1min":1.863,"5min":1.863,"15min":2.8889999999999998},"last":1.1839999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.407,"15min":1.427},"min":{"1min":0.86499999999999999,"5min":0.79800000000000004,"15min":0.67900000000000005},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":2.738},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.486,"15min":1.476},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.71699999999999997},"max":{"1min":1.9099999999999999,"5min":1.974,"15min":3.4180000000000001},"last":1.1279999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5620000000000001,"15min":1.556},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.968,"5min":2.0529999999999999,"15min":2.6339999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4970000000000001,"15min":1.4990000000000001},"min":{"1min":0.94099999999999995,"5min":0.90800000000000003,"15min":0.58499999999999996},"max":{"1min":1.9259999999999999,"5min":2.0979999999999999,"15min":2.7909999999999999},"last":1.504}]}]},{"osd":6,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":424047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.1240000000000001,"15min":1.1200000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.4610000000000001,"5min":1.7350000000000001,"15min":2.1160000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.1659999999999999,"15min":1.1830000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.627},"max":{"1min":1.7230000000000001,"5min":1.8260000000000001,"15min":5.0430000000000001},"last":0.93600000000000005}]},{"osd":1,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.222,"15min":1.2609999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.52200000000000002},"max":{"1min":1.496,"5min":1.6890000000000001,"15min":2.5379999999999998},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.25,"5min":1.3799999999999999,"15min":1.381},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":3.0059999999999998},"last":1.2829999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.3049999999999999,"15min":1.282},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.69799999999999995},"max":{"1min":1.6879999999999999,"5min":1.71,"15min":4.2779999999999996},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.381,"15min":1.3759999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.60299999999999998},"max":{"1min":1.595,"5min":1.7669999999999999,"15min":3.1720000000000002},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.286,"15min":1.3100000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.68799999999999994},"max":{"1min":1.821,"5min":1.9159999999999999,"15min":2.5960000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.2829999999999999,"15min":1.319},"min":{"1min":0.879,"5min":0.879,"15min":0.751},"max":{"1min":1.4990000000000001,"5min":1.6779999999999999,"15min":4.7649999999999997},"last":1.599}]},{"osd":4,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2330000000000001,"15min":1.2909999999999999},"min":{"1min":0.83799999999999997,"5min":0.79600000000000004,"15min":0.72899999999999998},"max":{"1min":1.849,"5min":1.849,"15min":2.7290000000000001},"last":1.222},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.3029999999999999,"15min":1.365},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.76400000000000001},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":3.2519999999999998},"last":1.6559999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.62},"max":{"1min":1.6699999999999999,"5min":1.752,"15min":4.8799999999999999},"last":1.476},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.526,"15min":1.569},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.7190000000000001,"5min":1.9219999999999999,"15min":3.6429999999999998},"last":1.5489999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4610000000000001,"15min":1.5249999999999999},"min":{"1min":1.006,"5min":1.006,"15min":0.85499999999999998},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":3.379},"last":1.766},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.444,"15min":1.518},"min":{"1min":0.93300000000000005,"5min":0.92900000000000005,"15min":0.84599999999999997},"max":{"1min":1.77,"5min":1.8819999999999999,"15min":4.9790000000000001},"last":1.857}]}]},{"osd":5,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":839680,"data_stored":416775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.179,"15min":1.145},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.59499999999999997},"max":{"1min":1.5209999999999999,"5min":1.7450000000000001,"15min":1.768},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1890000000000001,"15min":1.1859999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.6080000000000001,"5min":1.774,"15min":1.996},"last":1.4350000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.1579999999999999,"15min":1.1439999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.48899999999999999},"max":{"1min":1.3360000000000001,"5min":1.663,"15min":1.796},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.139,"5min":1.236,"15min":1.2090000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.54,"5min":1.6910000000000001,"15min":1.956},"last":1.196}]},{"osd":2,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.286,"15min":1.2689999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.901},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.304,"15min":1.2969999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.6679999999999999,"5min":1.958,"15min":2.27},"last":1.1220000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.3149999999999999,"15min":1.319},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":2.298},"last":1.534},{"interface":"front","average":{"1min":1.175,"5min":1.256,"15min":1.272},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.54600000000000004},"max":{"1min":1.5629999999999999,"5min":1.7509999999999999,"15min":2.177},"last":1.0740000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.302,"15min":1.3080000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6180000000000001,"5min":1.776,"15min":2.4329999999999998},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.4199999999999999,"15min":1.4319999999999999},"min":{"1min":0.79700000000000004,"5min":0.77000000000000002,"15min":0.68100000000000005},"max":{"1min":1.744,"5min":1.806,"15min":2.7599999999999998},"last":1.6839999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.306,"15min":1.3300000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.78,"5min":2.093,"15min":2.9319999999999999},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.464,"15min":1.4670000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6339999999999999,"5min":1.806,"15min":2.339},"last":1.601}]},{"osd":7,"last update":"Tue Nov 28 20:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4339999999999999,"15min":1.421},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.73399999999999999},"max":{"1min":1.599,"5min":2.0019999999999998,"15min":2.706},"last":1.571},{"interface":"front","average":{"1min":1.381,"5min":1.504,"15min":1.4890000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.8600000000000001,"5min":2.1829999999999998,"15min":2.3849999999999998},"last":1.74}]}]},{"osd":4,"up_from":13,"seq":55834575024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":856064,"data_stored":423977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.151,"15min":1.1339999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.64400000000000002},"max":{"1min":1.536,"5min":1.879,"15min":1.879},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.131,"15min":1.105},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.891,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.006}]},{"osd":1,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.3240000000000001,"15min":1.3080000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.71599999999999997},"max":{"1min":1.734,"5min":1.923,"15min":2.3879999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.397,"15min":1.337},"min":{"1min":0.91700000000000004,"5min":0.86599999999999999,"15min":0.76700000000000002},"max":{"1min":2.2679999999999998,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.212}]},{"osd":2,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2250000000000001,"15min":1.2210000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.6699999999999999,"5min":1.71,"15min":1.71},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2889999999999999,"15min":1.2929999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67500000000000004},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.1020000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3120000000000001,"15min":1.2889999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.246},{"interface":"front","average":{"1min":1.218,"5min":1.355,"15min":1.375},"min":{"1min":0.876,"5min":0.876,"15min":0.67900000000000005},"max":{"1min":1.583,"5min":2.0030000000000001,"15min":2.2519999999999998},"last":1.391}]},{"osd":5,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3169999999999999,"15min":1.3},"min":{"1min":1.02,"5min":0.83599999999999997,"15min":0.61699999999999999},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.619},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3360000000000001,"15min":1.3100000000000001},"min":{"1min":1.1919999999999999,"5min":0.82399999999999995,"15min":0.69299999999999995},"max":{"1min":1.647,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.343}]},{"osd":6,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3879999999999999,"15min":1.4199999999999999},"min":{"1min":0.96999999999999997,"5min":0.90400000000000003,"15min":0.628},"max":{"1min":2.04,"5min":2.04,"15min":2.4900000000000002},"last":1.579},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.5960000000000001,"15min":1.5529999999999999},"min":{"1min":1.194,"5min":1.133,"15min":0.753},"max":{"1min":3.7000000000000002,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":1.8660000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:48:11 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5009999999999999,"15min":1.6000000000000001},"min":{"1min":1.046,"5min":1.046,"15min":0.67500000000000004},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":20.568999999999999},"last":1.55},{"interface":"front","average":{"1min":1.486,"5min":1.5069999999999999,"15min":1.673},"min":{"1min":1.2370000000000001,"5min":0.96499999999999997,"15min":0.72299999999999998},"max":{"1min":1.7549999999999999,"5min":2.1099999999999999,"15min":20.677},"last":1.657}]}]},{"osd":3,"up_from":13,"seq":55834575024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40352,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331488,"statfs":{"total":96636764160,"available":96595443712,"internally_reserved":0,"allocated":868352,"data_stored":425029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.91300000000000003,"15min":0.93500000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38800000000000001},"max":{"1min":4.7210000000000001,"5min":4.7210000000000001,"15min":4.7210000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.0609999999999999,"15min":1.0700000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.34399999999999997},"max":{"1min":1.5860000000000001,"5min":2.0950000000000002,"15min":3.968},"last":0.437}]},{"osd":1,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":0.99099999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.36299999999999999},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":3.9180000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.2290000000000001,"15min":1.218},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.52600000000000002},"max":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.67400000000000004}]},{"osd":2,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2569999999999999,"15min":1.2709999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.48099999999999998},"max":{"1min":3.444,"5min":3.444,"15min":5.6509999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.351,"5min":1.359,"15min":1.343},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.46400000000000002},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":4.5759999999999996},"last":0.745}]},{"osd":4,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.377,"15min":1.343},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.72299999999999998},"max":{"1min":1.806,"5min":1.9079999999999999,"15min":4.1779999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.369,"15min":1.3999999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.59899999999999998},"max":{"1min":1.53,"5min":2.1389999999999998,"15min":4.492},"last":1.103}]},{"osd":5,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.4910000000000001,"15min":1.478},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.66800000000000004},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":4.7169999999999996},"last":1.006},{"interface":"front","average":{"1min":1.339,"5min":1.4390000000000001,"15min":1.4410000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":4.2889999999999997},"last":0.82399999999999995}]},{"osd":6,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.488,"15min":1.508},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.72799999999999998},"max":{"1min":1.7769999999999999,"5min":2.3100000000000001,"15min":5.2279999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.6180000000000001,"15min":1.6120000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.70199999999999996},"max":{"1min":3.661,"5min":3.661,"15min":6.9740000000000002},"last":1.1850000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.546,"15min":1.552},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.73599999999999999},"max":{"1min":2.1469999999999998,"5min":2.1469999999999998,"15min":4.0819999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.591,"15min":1.6200000000000001},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.64900000000000002},"max":{"1min":1.7969999999999999,"5min":2.3919999999999999,"15min":4.1399999999999997},"last":1.2110000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575024,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58396,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313444,"statfs":{"total":96636764160,"available":96576966656,"internally_reserved":0,"allocated":864256,"data_stored":425007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.93799999999999994,"15min":0.90400000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.413,"5min":2.012,"15min":2.012},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.0389999999999999,"15min":1.0489999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":1.47,"5min":1.6539999999999999,"15min":1.9390000000000001},"last":1.151}]},{"osd":1,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.95399999999999996,"15min":0.95799999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.2669999999999999,"5min":1.998,"15min":3.2599999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.1539999999999999,"15min":1.1359999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.38500000000000001},"max":{"1min":1.506,"5min":1.7310000000000001,"15min":1.837},"last":1.3560000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.1619999999999999,"15min":1.143},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.442},"max":{"1min":1.7070000000000001,"5min":1.774,"15min":2.4319999999999999},"last":1.7070000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.26,"15min":1.2549999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.45600000000000002},"max":{"1min":1.3799999999999999,"5min":1.742,"15min":3.1440000000000001},"last":1.05}]},{"osd":4,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.401,"15min":1.3759999999999999},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.72699999999999998},"max":{"1min":1.7450000000000001,"5min":1.9390000000000001,"15min":2.8919999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.389,"15min":1.4079999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.80500000000000005},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.8900000000000001},"last":2.3740000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.4259999999999999,"15min":1.405},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.72499999999999998},"max":{"1min":1.6779999999999999,"5min":2.141,"15min":2.1459999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.488,"15min":1.464},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.751},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":3.4279999999999999},"last":1.3959999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.478,"15min":1.4970000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.76400000000000001},"max":{"1min":2.1709999999999998,"5min":2.306,"15min":2.931},"last":2.1709999999999998},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.532,"15min":1.5449999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.78300000000000003},"max":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":3.6269999999999998},"last":1.49}]},{"osd":7,"last update":"Tue Nov 28 20:48:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.552,"15min":1.522},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.84999999999999998},"max":{"1min":1.8879999999999999,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.5800000000000001,"15min":1.5820000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.78400000000000003},"max":{"1min":2.274,"5min":2.4849999999999999,"15min":2.9649999999999999},"last":2.274}]}]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36184,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335656,"statfs":{"total":96636764160,"available":96599711744,"internally_reserved":0,"allocated":860160,"data_stored":423555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.91900000000000004,"15min":0.91000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.6639999999999999,"5min":1.853,"15min":1.9259999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.121,"15min":1.069},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48299999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.8919999999999999},"last":1.3280000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.083,"15min":1.087},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.526,"5min":2.234,"15min":2.234},"last":1.405},{"interface":"front","average":{"1min":1.016,"5min":1.204,"15min":1.1930000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.496,"5min":2.895,"15min":2.895},"last":1.0780000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.286,"15min":1.3009999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.893,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.6859999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.1160000000000001,"15min":1.105},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.887,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.206}]},{"osd":4,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.345,"15min":1.3380000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.76600000000000001},"max":{"1min":1.569,"5min":2.1019999999999999,"15min":2.9020000000000001},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.3779999999999999,"15min":1.403},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8069999999999999,"5min":2.1469999999999998,"15min":3.5259999999999998},"last":1.619}]},{"osd":5,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.4339999999999999,"15min":1.4410000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.80900000000000005},"max":{"1min":1.8080000000000001,"5min":2.1840000000000002,"15min":3.637},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.502,"15min":1.496},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.85299999999999998},"max":{"1min":1.768,"5min":2.294,"15min":2.3050000000000002},"last":1.5700000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.423,"15min":1.446},"min":{"1min":0.874,"5min":0.874,"15min":0.81100000000000005},"max":{"1min":1.641,"5min":1.9379999999999999,"15min":4.5789999999999997},"last":1.738},{"interface":"front","average":{"1min":1.327,"5min":1.5389999999999999,"15min":1.536},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.76300000000000001},"max":{"1min":1.946,"5min":2.3300000000000001,"15min":3.7360000000000002},"last":1.79}]},{"osd":7,"last update":"Tue Nov 28 20:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.792,"5min":2.3359999999999999,"15min":2.4319999999999999},"last":1.464},{"interface":"front","average":{"1min":1.413,"5min":1.5569999999999999,"15min":1.5269999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.843,"5min":2.294,"15min":4.4660000000000002},"last":1.5129999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575024,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36752,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1441792,"data_stored":1007199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95099999999999996,"15min":0.92300000000000004},"min":{"1min":0.71099999999999997,"5min":0.59499999999999997,"15min":0.41499999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.999},{"interface":"front","average":{"1min":1.03,"5min":1.105,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.45700000000000002},"max":{"1min":1.3200000000000001,"5min":1.764,"15min":2.7229999999999999},"last":1.1659999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.163,"15min":1.1339999999999999},"min":{"1min":0.88300000000000001,"5min":0.80600000000000005,"15min":0.46600000000000003},"max":{"1min":1.5800000000000001,"5min":1.657,"15min":2.0579999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.216,"5min":1.1850000000000001,"15min":1.224},"min":{"1min":0.86399999999999999,"5min":0.82499999999999996,"15min":0.56299999999999994},"max":{"1min":1.5449999999999999,"5min":1.9159999999999999,"15min":2.964},"last":1.2909999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1220000000000001,"15min":1.1499999999999999},"min":{"1min":0.79000000000000004,"5min":0.64000000000000001,"15min":0.504},"max":{"1min":1.458,"5min":1.8420000000000001,"15min":2.6920000000000002},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.365,"15min":1.3380000000000001},"min":{"1min":1.0720000000000001,"5min":0.90200000000000002,"15min":0.52500000000000002},"max":{"1min":1.698,"5min":2.1080000000000001,"15min":2.1379999999999999},"last":1.1120000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.47,"15min":1.4430000000000001},"min":{"1min":1.1259999999999999,"5min":0.86099999999999999,"15min":0.80900000000000005},"max":{"1min":2.012,"5min":2.012,"15min":2.7989999999999999},"last":1.635},{"interface":"front","average":{"1min":1.506,"5min":1.4370000000000001,"15min":1.391},"min":{"1min":1.2,"5min":1.0489999999999999,"15min":0.68300000000000005},"max":{"1min":1.8660000000000001,"5min":2.0539999999999998,"15min":2.073},"last":1.254}]},{"osd":5,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.429,"15min":1.454},"min":{"1min":1.0760000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.528},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5429999999999999,"15min":1.5109999999999999},"min":{"1min":1.119,"5min":1.119,"15min":0.89700000000000002},"max":{"1min":2.0750000000000002,"5min":2.2490000000000001,"15min":2.278},"last":1.22}]},{"osd":6,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.6060000000000001,"15min":1.581},"min":{"1min":1.4159999999999999,"5min":1.242,"15min":0.78800000000000003},"max":{"1min":2.1160000000000001,"5min":2.2629999999999999,"15min":2.536},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5669999999999999,"15min":1.5620000000000001},"min":{"1min":1.147,"5min":1.131,"15min":0.81100000000000005},"max":{"1min":1.847,"5min":2.4209999999999998,"15min":2.7069999999999999},"last":1.405}]},{"osd":7,"last update":"Tue Nov 28 20:47:19 2023","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.611,"15min":1.631},"min":{"1min":1.216,"5min":1.216,"15min":0.99299999999999999},"max":{"1min":1.996,"5min":2.3330000000000002,"15min":2.8650000000000002},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.5609999999999999,"15min":1.597},"min":{"1min":1.3460000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.0209999999999999,"5min":2.524,"15min":2.6640000000000001},"last":1.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:48:19.340 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T20:48:19.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T20:48:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.607+0000 7fe8eba6b640 1 Processor -- start 2023-11-28T20:48:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.608+0000 7fe8eba6b640 1 -- start start 2023-11-28T20:48:19.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.609+0000 7fe8eba6b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e406b4d0 0x7fe8e40cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.609+0000 7fe8eba6b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e40d1fc0 con 0x7fe8e40aff90 2023-11-28T20:48:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.609+0000 7fe8eba6b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e406a910 con 0x7fe8e4099a40 2023-11-28T20:48:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.609+0000 7fe8eba6b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40cb8a0 con 0x7fe8e406b4d0 2023-11-28T20:48:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.610+0000 7fe8e9006640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe8e40aff90 0x7fe8e40cbd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58228/0 (socket says 172.21.15.45:58228) 2023-11-28T20:48:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.610+0000 7fe8e9006640 1 -- 172.21.15.45:0/4242162125 learned_addr learned my addr 172.21.15.45:0/4242162125 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.610+0000 7fe8ea008640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e406b4d0 0x7fe8e40cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.610+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597387683 0 0) 0x7fe8e40d1fc0 con 0x7fe8e40aff90 2023-11-28T20:48:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.610+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d0003520 con 0x7fe8e40aff90 2023-11-28T20:48:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3188083951 0 0) 0x7fe8d0003520 con 0x7fe8e40aff90 2023-11-28T20:48:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e406b4d0 msgr2=0x7fe8e40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e406b4d0 0x7fe8e40cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 >> v1:172.21.15.190:6789/0 conn(0x7fe8e4099a40 legacy=0x7fe8e4099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e406a630 con 0x7fe8e40aff90 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fe8d4001670 con 0x7fe8e40aff90 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 >> v1:172.21.15.45:6789/0 conn(0x7fe8e40aff90 legacy=0x7fe8e40cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8d00039a0 0x7fe8d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e406b4d0 0x7fe8d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d0004d00 0x7fe8d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40cb8a0 con 0x7fe8d0004d00 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d000bc30 con 0x7fe8d00039a0 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d000bdf0 con 0x7fe8e406b4d0 2023-11-28T20:48:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8ea008640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d0004d00 0x7fe8d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.611+0000 7fe8e9807640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8d00039a0 0x7fe8d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.612+0000 7fe8e9006640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e406b4d0 0x7fe8d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.612+0000 7fe8ea008640 1 -- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e406b4d0 msgr2=0x7fe8d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:19.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.612+0000 7fe8ea008640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e406b4d0 0x7fe8d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.612+0000 7fe8ea008640 1 -- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8d00039a0 msgr2=0x7fe8d0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.612+0000 7fe8ea008640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8d00039a0 0x7fe8d0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.612+0000 7fe8ea008640 1 -- 172.21.15.45:0/4242162125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0003520 con 0x7fe8d0004d00 2023-11-28T20:48:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.612+0000 7fe8e9006640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8e406b4d0 0x7fe8d00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.614+0000 7fe8ea008640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d0004d00 0x7fe8d0005180 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc007360 tx=0x7fe8d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0b99e23f1ada73 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.614+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc02ccd0 con 0x7fe8d0004d00 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.614+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc02ce30 con 0x7fe8d0004d00 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.614+0000 7fe8e8805640 1 -- 172.21.15.45:0/4242162125 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc034660 con 0x7fe8d0004d00 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.615+0000 7fe8eba6b640 1 -- 172.21.15.45:0/4242162125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d0004d00 msgr2=0x7fe8d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.615+0000 7fe8eba6b640 1 --2- 172.21.15.45:0/4242162125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8d0004d00 0x7fe8d0005180 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc007360 tx=0x7fe8d00057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.615+0000 7fe8ea008640 1 -- 172.21.15.45:0/4242162125 reap_dead start 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.615+0000 7fe8eba6b640 1 -- 172.21.15.45:0/4242162125 shutdown_connections 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.615+0000 7fe8eba6b640 1 -- 172.21.15.45:0/4242162125 >> 172.21.15.45:0/4242162125 conn(0x7fe8e40d1020 msgr2=0x7fe8e40d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.615+0000 7fe8eba6b640 1 -- 172.21.15.45:0/4242162125 shutdown_connections 2023-11-28T20:48:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.615+0000 7fe8eba6b640 1 -- 172.21.15.45:0/4242162125 wait complete. 2023-11-28T20:48:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.617+0000 7fe8eba6b640 1 Processor -- start 2023-11-28T20:48:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.617+0000 7fe8eba6b640 1 -- start start 2023-11-28T20:48:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.618+0000 7fe8eba6b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8d0004d00 0x7fe8e40c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.618+0000 7fe8eba6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e414b150 0x7fe8e4148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.618+0000 7fe8eba6b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40c60b0 0x7fe8e40c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.618+0000 7fe8eba6b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40c6a70 con 0x7fe8e414b150 2023-11-28T20:48:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.618+0000 7fe8eba6b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e414b8d0 con 0x7fe8e40c60b0 2023-11-28T20:48:19.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.618+0000 7fe8eba6b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e414bbb0 con 0x7fe8d0004d00 2023-11-28T20:48:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.618+0000 7fe8e9006640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e414b150 0x7fe8e4148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.618+0000 7fe8e9006640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e414b150 0x7fe8e4148730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34428/0 (socket says 172.21.15.45:34428) 2023-11-28T20:48:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.618+0000 7fe8e9006640 1 -- 172.21.15.45:0/519198275 learned_addr learned my addr 172.21.15.45:0/519198275 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8e9006640 1 -- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8d0004d00 msgr2=0x7fe8e40c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8e9006640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8d0004d00 0x7fe8e40c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8ea008640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40c60b0 0x7fe8e40c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8e9006640 1 -- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40c60b0 msgr2=0x7fe8e40c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8e9006640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40c60b0 0x7fe8e40c6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8e9006640 1 -- 172.21.15.45:0/519198275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e414be90 con 0x7fe8e414b150 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8e9006640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e414b150 0x7fe8e4148730 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4001200 tx=0x7fe8d400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8da7fc640 1 -- 172.21.15.45:0/519198275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d400c870 con 0x7fe8e414b150 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8da7fc640 1 -- 172.21.15.45:0/519198275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d400c9d0 con 0x7fe8e414b150 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8da7fc640 1 -- 172.21.15.45:0/519198275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d400cd40 con 0x7fe8e414b150 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.619+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e4149260 con 0x7fe8e414b150 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.620+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e4148c70 con 0x7fe8e414b150 2023-11-28T20:48:19.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.621+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e41490c0 con 0x7fe8e414b150 2023-11-28T20:48:19.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.622+0000 7fe8da7fc640 1 -- 172.21.15.45:0/519198275 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe8d401d460 con 0x7fe8e414b150 2023-11-28T20:48:19.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.622+0000 7fe8da7fc640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe8c806af60 0x7fe8c806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:19.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.623+0000 7fe8da7fc640 1 -- 172.21.15.45:0/519198275 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe8d40899e0 con 0x7fe8e414b150 2023-11-28T20:48:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.623+0000 7fe8e9807640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe8c806af60 0x7fe8c806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.624+0000 7fe8e9807640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe8c806af60 0x7fe8c806d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00016f0 tx=0x7fe8e0009040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:19.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.628+0000 7fe8da7fc640 1 -- 172.21.15.45:0/519198275 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8e41490c0 con 0x7fe8e414b150 2023-11-28T20:48:19.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.893+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe8e41490c0 con 0x7fe8e414b150 2023-11-28T20:48:19.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.899+0000 7fe8da7fc640 1 -- 172.21.15.45:0/519198275 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe8e41490c0 con 0x7fe8e414b150 2023-11-28T20:48:19.901 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T20:48:19.901 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:48:19.901 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T20:48:19.901 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T20:48:19.902 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T20:48:19.902 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T20:48:19.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe8c806af60 msgr2=0x7fe8c806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe8c806af60 0x7fe8c806d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00016f0 tx=0x7fe8e0009040 comp rx=0 tx=0).stop 2023-11-28T20:48:19.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e414b150 msgr2=0x7fe8e4148730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:19.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e414b150 0x7fe8e4148730 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4001200 tx=0x7fe8d400ed50 comp rx=0 tx=0).stop 2023-11-28T20:48:19.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 shutdown_connections 2023-11-28T20:48:19.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe8c806af60 0x7fe8c806d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e40c60b0 0x7fe8e40c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8e414b150 0x7fe8e4148730 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 --2- 172.21.15.45:0/519198275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8d0004d00 0x7fe8e40c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:19.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 >> 172.21.15.45:0/519198275 conn(0x7fe8e40d1020 msgr2=0x7fe8e406b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:19.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 shutdown_connections 2023-11-28T20:48:19.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:19.902+0000 7fe8eba6b640 1 -- 172.21.15.45:0/519198275 wait complete. 2023-11-28T20:48:19.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:48:20.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.147+0000 7fe229154640 1 Processor -- start 2023-11-28T20:48:20.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.148+0000 7fe229154640 1 -- start start 2023-11-28T20:48:20.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.149+0000 7fe229154640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe2240b3870 0x7fe22415db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:20.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.149+0000 7fe229154640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240721e0 0x7fe22415e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:20.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.149+0000 7fe229154640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe22406d170 0x7fe22406d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:20.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.149+0000 7fe229154640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22415a460 con 0x7fe22406d170 2023-11-28T20:48:20.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.149+0000 7fe229154640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22406db40 con 0x7fe2240b3870 2023-11-28T20:48:20.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.149+0000 7fe229154640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22415e680 con 0x7fe2240721e0 2023-11-28T20:48:20.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.150+0000 7fe228953640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe22406d170 0x7fe22406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:20.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.150+0000 7fe228953640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe22406d170 0x7fe22406d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34442/0 (socket says 172.21.15.45:34442) 2023-11-28T20:48:20.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.150+0000 7fe228953640 1 -- 172.21.15.45:0/2471165463 learned_addr learned my addr 172.21.15.45:0/2471165463 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:20.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.150+0000 7fe223fff640 1 --2- 172.21.15.45:0/2471165463 >> v2:172.21.15.190:3300/0 conn(0x7fe2240b3870 0x7fe22415db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:20.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.150+0000 7fe21bfff640 1 --2- 172.21.15.45:0/2471165463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240721e0 0x7fe22415e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:20.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.150+0000 7fe228953640 1 -- 172.21.15.45:0/2471165463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240721e0 msgr2=0x7fe22415e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:20.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.150+0000 7fe228953640 1 --2- 172.21.15.45:0/2471165463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240721e0 0x7fe22415e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.150+0000 7fe228953640 1 -- 172.21.15.45:0/2471165463 >> v2:172.21.15.190:3300/0 conn(0x7fe2240b3870 msgr2=0x7fe22415db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:20.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.150+0000 7fe228953640 1 --2- 172.21.15.45:0/2471165463 >> v2:172.21.15.190:3300/0 conn(0x7fe2240b3870 0x7fe22415db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.151+0000 7fe228953640 1 -- 172.21.15.45:0/2471165463 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22415e960 con 0x7fe22406d170 2023-11-28T20:48:20.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.152+0000 7fe228953640 1 --2- 172.21.15.45:0/2471165463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 0x7fe22406d600 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe22406df50 tx=0x7fe21c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63b437a00abb761 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.152+0000 7fe2237fe640 1 -- 172.21.15.45:0/2471165463 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c02ca70 con 0x7fe22406d170 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.152+0000 7fe2237fe640 1 -- 172.21.15.45:0/2471165463 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c02cbd0 con 0x7fe22406d170 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.152+0000 7fe2237fe640 1 -- 172.21.15.45:0/2471165463 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c0343f0 con 0x7fe22406d170 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.153+0000 7fe229154640 1 -- 172.21.15.45:0/2471165463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 msgr2=0x7fe22406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.153+0000 7fe229154640 1 --2- 172.21.15.45:0/2471165463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 0x7fe22406d600 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe22406df50 tx=0x7fe21c02bed0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.153+0000 7fe229154640 1 -- 172.21.15.45:0/2471165463 shutdown_connections 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.153+0000 7fe229154640 1 --2- 172.21.15.45:0/2471165463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 0x7fe22406d600 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.153+0000 7fe229154640 1 --2- 172.21.15.45:0/2471165463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240721e0 0x7fe22415e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.153+0000 7fe229154640 1 --2- 172.21.15.45:0/2471165463 >> v2:172.21.15.190:3300/0 conn(0x7fe2240b3870 0x7fe22415db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.153+0000 7fe229154640 1 -- 172.21.15.45:0/2471165463 >> 172.21.15.45:0/2471165463 conn(0x7fe22406ab30 msgr2=0x7fe2240b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.153+0000 7fe229154640 1 -- 172.21.15.45:0/2471165463 shutdown_connections 2023-11-28T20:48:20.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.153+0000 7fe229154640 1 -- 172.21.15.45:0/2471165463 wait complete. 2023-11-28T20:48:20.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.156+0000 7fe229154640 1 Processor -- start 2023-11-28T20:48:20.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.156+0000 7fe229154640 1 -- start start 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.157+0000 7fe229154640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 0x7fe2241ee120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.157+0000 7fe229154640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2240721e0 0x7fe224154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.157+0000 7fe229154640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240b3870 0x7fe2240acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.157+0000 7fe229154640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2241ee660 con 0x7fe22406d170 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.157+0000 7fe229154640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe224155600 con 0x7fe2240721e0 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.157+0000 7fe229154640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe224155320 con 0x7fe2240b3870 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.157+0000 7fe223fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 0x7fe2241ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.157+0000 7fe223fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 0x7fe2241ee120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34448/0 (socket says 172.21.15.45:34448) 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.157+0000 7fe223fff640 1 -- 172.21.15.45:0/3292680050 learned_addr learned my addr 172.21.15.45:0/3292680050 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe21bfff640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2240721e0 0x7fe224154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:20.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe223fff640 1 -- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240b3870 msgr2=0x7fe2240acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:20.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe228953640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240b3870 0x7fe2240acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:20.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe223fff640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240b3870 0x7fe2240acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe223fff640 1 -- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2240721e0 msgr2=0x7fe224154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:20.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe223fff640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2240721e0 0x7fe224154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe223fff640 1 -- 172.21.15.45:0/3292680050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c009b30 con 0x7fe22406d170 2023-11-28T20:48:20.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe21bfff640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2240721e0 0x7fe224154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:20.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe223fff640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 0x7fe2241ee120 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe21400d800 tx=0x7fe21400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:20.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.158+0000 7fe2217fa640 1 -- 172.21.15.45:0/3292680050 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21400b6c0 con 0x7fe22406d170 2023-11-28T20:48:20.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.159+0000 7fe2217fa640 1 -- 172.21.15.45:0/3292680050 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21400b820 con 0x7fe22406d170 2023-11-28T20:48:20.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.159+0000 7fe2217fa640 1 -- 172.21.15.45:0/3292680050 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21400bbb0 con 0x7fe22406d170 2023-11-28T20:48:20.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.159+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe224099b70 con 0x7fe22406d170 2023-11-28T20:48:20.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.159+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22409a130 con 0x7fe22406d170 2023-11-28T20:48:20.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.160+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22409d0f0 con 0x7fe22406d170 2023-11-28T20:48:20.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.161+0000 7fe2217fa640 1 -- 172.21.15.45:0/3292680050 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe21400bd10 con 0x7fe22406d170 2023-11-28T20:48:20.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.162+0000 7fe2217fa640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe20406af60 0x7fe20406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:20.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.162+0000 7fe2217fa640 1 -- 172.21.15.45:0/3292680050 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe214089de0 con 0x7fe22406d170 2023-11-28T20:48:20.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.163+0000 7fe21bfff640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe20406af60 0x7fe20406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:20.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.164+0000 7fe21bfff640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe20406af60 0x7fe20406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe20c006fd0 tx=0x7fe20c008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:20.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.166+0000 7fe2217fa640 1 -- 172.21.15.45:0/3292680050 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe214054cd0 con 0x7fe22406d170 2023-11-28T20:48:20.245 INFO:tasks.ceph.mds.a.smithi045.stderr:2023-11-28T20:48:20.242+0000 7f9f8a4fc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:48:20.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.401+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe22404e920 con 0x7fe22406d170 2023-11-28T20:48:20.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.402+0000 7fe2217fa640 1 -- 172.21.15.45:0/3292680050 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7fe2140582f0 con 0x7fe22406d170 2023-11-28T20:48:20.416 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:20.416 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5183,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/285399852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":285399852},{"type":"v1","addr":"172.21.15.190:6836","nonce":285399852}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2028545354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2028545354},{"type":"v1","addr":"172.21.15.190:6837","nonce":2028545354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6216,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/784320624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":784320624},{"type":"v1","addr":"172.21.15.45:6834","nonce":784320624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:48:13.742492+0000","modified":"2023-11-28T20:48:16.820308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3717491008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3717491008},{"type":"v1","addr":"172.21.15.45:6835","nonce":3717491008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 27 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe20406af60 msgr2=0x7fe20406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe20406af60 0x7fe20406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe20c006fd0 tx=0x7fe20c008040 comp rx=0 tx=0).stop 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 msgr2=0x7fe2241ee120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 0x7fe2241ee120 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe21400d800 tx=0x7fe21400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 shutdown_connections 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7fe20406af60 0x7fe20406d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe2240b3870 0x7fe2240acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2240721e0 0x7fe224154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 --2- 172.21.15.45:0/3292680050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe22406d170 0x7fe2241ee120 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:20.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 >> 172.21.15.45:0/3292680050 conn(0x7fe22406ab30 msgr2=0x7fe2240cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:20.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.405+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 shutdown_connections 2023-11-28T20:48:20.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:20.406+0000 7fe229154640 1 -- 172.21.15.45:0/3292680050 wait complete. 2023-11-28T20:48:20.830 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:48:13.742492+0000', 'modified': '2023-11-28T20:48:16.820308+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6219}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6219': {'gid': 6219, 'name': 'a', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/3717491008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6833', 'nonce': 3717491008}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 3717491008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T20:48:20.848 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:48:20.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-28T20:48:21.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.082+0000 7f55c5244640 1 Processor -- start 2023-11-28T20:48:21.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.082+0000 7f55c5244640 1 -- start start 2023-11-28T20:48:21.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.084+0000 7f55c5244640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f55c015d730 0x7f55c00d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.084+0000 7f55c5244640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f55c00d2750 0x7f55c0070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.084+0000 7f55c5244640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c006b580 con 0x7f55c0071060 2023-11-28T20:48:21.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.084+0000 7f55c5244640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c0071470 con 0x7f55c015d730 2023-11-28T20:48:21.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.084+0000 7f55c5244640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c006b2a0 con 0x7f55c00d2750 2023-11-28T20:48:21.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.084+0000 7f55bed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f55c015d730 0x7f55c00d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.084+0000 7f55bed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f55c015d730 0x7f55c00d2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34458/0 (socket says 172.21.15.45:34458) 2023-11-28T20:48:21.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.084+0000 7f55bed9d640 1 -- 172.21.15.45:0/22331434 learned_addr learned my addr 172.21.15.45:0/22331434 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:21.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.085+0000 7f55bed9d640 1 -- 172.21.15.45:0/22331434 >> v2:172.21.15.190:3300/0 conn(0x7f55c00d2750 msgr2=0x7f55c0070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.085+0000 7f55bed9d640 1 --2- 172.21.15.45:0/22331434 >> v2:172.21.15.190:3300/0 conn(0x7f55c00d2750 0x7f55c0070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.085+0000 7f55bed9d640 1 -- 172.21.15.45:0/22331434 >> v1:172.21.15.45:6789/0 conn(0x7f55c0071060 legacy=0x7f55c00b6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.085+0000 7f55bed9d640 1 -- 172.21.15.45:0/22331434 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c00d1730 con 0x7f55c015d730 2023-11-28T20:48:21.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.086+0000 7f55bed9d640 1 --2- 172.21.15.45:0/22331434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c015d730 0x7f55c00d2210 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f55c008c460 tx=0x7f55b002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c1edb8fab7bfde server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.086+0000 7f55be59c640 1 -- 172.21.15.45:0/22331434 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b002cb20 con 0x7f55c015d730 2023-11-28T20:48:21.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.086+0000 7f55be59c640 1 -- 172.21.15.45:0/22331434 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b002cc80 con 0x7f55c015d730 2023-11-28T20:48:21.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.086+0000 7f55be59c640 1 -- 172.21.15.45:0/22331434 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b00343f0 con 0x7f55c015d730 2023-11-28T20:48:21.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.087+0000 7f55c5244640 1 -- 172.21.15.45:0/22331434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c015d730 msgr2=0x7f55c00d2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.087+0000 7f55c5244640 1 --2- 172.21.15.45:0/22331434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c015d730 0x7f55c00d2210 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f55c008c460 tx=0x7f55b002bf80 comp rx=0 tx=0).stop 2023-11-28T20:48:21.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.087+0000 7f55c5244640 1 -- 172.21.15.45:0/22331434 shutdown_connections 2023-11-28T20:48:21.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.087+0000 7f55c5244640 1 --2- 172.21.15.45:0/22331434 >> v2:172.21.15.190:3300/0 conn(0x7f55c00d2750 0x7f55c0070b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.087+0000 7f55c5244640 1 --2- 172.21.15.45:0/22331434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c015d730 0x7f55c00d2210 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.087+0000 7f55c5244640 1 -- 172.21.15.45:0/22331434 >> 172.21.15.45:0/22331434 conn(0x7f55c00cfc70 msgr2=0x7f55c00d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:21.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.087+0000 7f55c5244640 1 -- 172.21.15.45:0/22331434 shutdown_connections 2023-11-28T20:48:21.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.087+0000 7f55c5244640 1 -- 172.21.15.45:0/22331434 wait complete. 2023-11-28T20:48:21.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.089+0000 7f55c5244640 1 Processor -- start 2023-11-28T20:48:21.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.089+0000 7f55c5244640 1 -- start start 2023-11-28T20:48:21.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.090+0000 7f55c5244640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c0071060 0x7f55c01f7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.090+0000 7f55c5244640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f55c00d2750 0x7f55c01481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.090+0000 7f55c5244640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f55c015d730 0x7f55c006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.090+0000 7f55c5244640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c01f3170 con 0x7f55c0071060 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.090+0000 7f55c5244640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c0147f20 con 0x7f55c015d730 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.090+0000 7f55c5244640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c01f76a0 con 0x7f55c00d2750 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.090+0000 7f55bed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c0071060 0x7f55c01f7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.090+0000 7f55bed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c0071060 0x7f55c01f7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34462/0 (socket says 172.21.15.45:34462) 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.090+0000 7f55bed9d640 1 -- 172.21.15.45:0/1250263318 learned_addr learned my addr 172.21.15.45:0/1250263318 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55bed9d640 1 -- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f55c00d2750 msgr2=0x7f55c01481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55bed9d640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f55c00d2750 0x7f55c01481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55bed9d640 1 -- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f55c015d730 msgr2=0x7f55c006a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55bed9d640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f55c015d730 0x7f55c006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55bed9d640 1 -- 172.21.15.45:0/1250263318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b0009b30 con 0x7f55c0071060 2023-11-28T20:48:21.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55bed9d640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c0071060 0x7f55c01f7160 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f55b0036040 tx=0x7f55b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55b7fff640 1 -- 172.21.15.45:0/1250263318 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b0009510 con 0x7f55c0071060 2023-11-28T20:48:21.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55b7fff640 1 -- 172.21.15.45:0/1250263318 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b003e450 con 0x7f55c0071060 2023-11-28T20:48:21.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55b7fff640 1 -- 172.21.15.45:0/1250263318 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b003e800 con 0x7f55c0071060 2023-11-28T20:48:21.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.091+0000 7f55c5244640 1 -- 172.21.15.45:0/1250263318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c015dc90 con 0x7f55c0071060 2023-11-28T20:48:21.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.092+0000 7f55c5244640 1 -- 172.21.15.45:0/1250263318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c014a9b0 con 0x7f55c0071060 2023-11-28T20:48:21.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.094+0000 7f55b7fff640 1 -- 172.21.15.45:0/1250263318 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f55b003e960 con 0x7f55c0071060 2023-11-28T20:48:21.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.095+0000 7f55b7fff640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f55a006af90 0x7f55a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.095+0000 7f55b7fff640 1 -- 172.21.15.45:0/1250263318 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f55b00ac2e0 con 0x7f55c0071060 2023-11-28T20:48:21.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.096+0000 7f55c7dd7740 1 Processor -- start 2023-11-28T20:48:21.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.096+0000 7f55c7dd7740 1 -- start start 2023-11-28T20:48:21.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.096+0000 7f55b659c640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f55a006af90 0x7f55a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c4ea0 0x5585a08c5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08bfba0 0x5585a08c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c45e0 0x5585a0908560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585a0823880 con 0x5585a08bfba0 2023-11-28T20:48:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585a0832ae0 con 0x5585a08c45e0 2023-11-28T20:48:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5585a08301d0 con 0x5585a08c4ea0 2023-11-28T20:48:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55b659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08bfba0 0x5585a08c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55b659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08bfba0 0x5585a08c40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34476/0 (socket says 172.21.15.45:34476) 2023-11-28T20:48:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55b659c640 1 -- 172.21.15.45:0/3453067626 learned_addr learned my addr 172.21.15.45:0/3453067626 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55b659c640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f55a006af90 0x7f55a006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f55c00d2670 tx=0x7f55a8006c90 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55bed9d640 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c4ea0 0x5585a08c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55bf59e640 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c45e0 0x5585a0908560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55b659c640 1 -- 172.21.15.45:0/3453067626 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c4ea0 msgr2=0x5585a08c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55b659c640 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c4ea0 0x5585a08c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55b659c640 1 -- 172.21.15.45:0/3453067626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c45e0 msgr2=0x5585a0908560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.097+0000 7f55b659c640 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c45e0 0x5585a0908560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.098+0000 7f55b659c640 1 -- 172.21.15.45:0/3453067626 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585a07f3860 con 0x5585a08bfba0 2023-11-28T20:48:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.098+0000 7f55bed9d640 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c4ea0 0x5585a08c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.098+0000 7f55b659c640 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08bfba0 0x5585a08c40a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f55a8015880 tx=0x7f55a800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56dc006dafd1c97e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.098+0000 7f55b4d99640 1 -- 172.21.15.45:0/3453067626 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a801a070 con 0x5585a08bfba0 2023-11-28T20:48:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.098+0000 7f55b4d99640 1 -- 172.21.15.45:0/3453067626 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a8015e00 con 0x5585a08bfba0 2023-11-28T20:48:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.098+0000 7f55b4d99640 1 -- 172.21.15.45:0/3453067626 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a80152e0 con 0x5585a08bfba0 2023-11-28T20:48:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.099+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3453067626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08bfba0 msgr2=0x5585a08c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.099+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08bfba0 0x5585a08c40a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f55a8015880 tx=0x7f55a800e000 comp rx=0 tx=0).stop 2023-11-28T20:48:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.099+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3453067626 shutdown_connections 2023-11-28T20:48:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.099+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c45e0 0x5585a0908560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.099+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08bfba0 0x5585a08c40a0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.099+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3453067626 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c4ea0 0x5585a08c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.099+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3453067626 >> 172.21.15.45:0/3453067626 conn(0x5585a0886c70 msgr2=0x5585a090a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.099+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3453067626 shutdown_connections 2023-11-28T20:48:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.099+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3453067626 wait complete. 2023-11-28T20:48:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.100+0000 7f55c7dd7740 1 -- start start 2023-11-28T20:48:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.100+0000 7f55c7dd7740 1 Processor -- start 2023-11-28T20:48:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 0x5585a091a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 0x5585a090ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a0910480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585a082bac0 con 0x5585a08c4ea0 2023-11-28T20:48:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585a07c82f0 con 0x5585a08bfba0 2023-11-28T20:48:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5585a07e5c70 con 0x5585a08c45e0 2023-11-28T20:48:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55c7dd7740 10 client.0 _ll_register_callbacks cb 0x5585a07ea3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:48:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55bf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a0910480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55bf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a0910480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34492/0 (socket says 172.21.15.45:34492) 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.101+0000 7f55bf59e640 1 -- 172.21.15.45:0/3851103796 learned_addr learned my addr 172.21.15.45:0/3851103796 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55b659c640 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 0x5585a090ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55bed9d640 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 0x5585a091a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55bf59e640 1 -- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 msgr2=0x5585a090ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55bf59e640 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 0x5585a090ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55bf59e640 1 -- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 msgr2=0x5585a091a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55bf59e640 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 0x5585a091a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55bf59e640 1 -- 172.21.15.45:0/3851103796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a8015500 con 0x5585a08c4ea0 2023-11-28T20:48:21.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55b659c640 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 0x5585a090ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55bed9d640 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 0x5585a091a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55bf59e640 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a0910480 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f55b8001d40 tx=0x7f55b800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84fa77a69bcd2bf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f559dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:48:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f559dffb640 1 -- 172.21.15.45:0/3851103796 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b800dea0 con 0x5585a08c4ea0 2023-11-28T20:48:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f559dffb640 1 -- 172.21.15.45:0/3851103796 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f55b80052d0 con 0x5585a08c4ea0 2023-11-28T20:48:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.102+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3851103796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5585a0824aa0 con 0x5585a08c4ea0 2023-11-28T20:48:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f559dffb640 1 -- 172.21.15.45:0/3851103796 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55b8005490 con 0x5585a08c4ea0 2023-11-28T20:48:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f55c7dd7740 10 client.6369 fetch_fsmap learned FSMap version 27 2023-11-28T20:48:21.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3851103796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x5585a082a140 con 0x5585a08c4ea0 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f559dffb640 1 -- 172.21.15.45:0/3851103796 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5585a0824aa0 con 0x5585a08c4ea0 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f559dffb640 1 -- 172.21.15.45:0/3851103796 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 27) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f55b8003610 con 0x5585a08c4ea0 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f55c7dd7740 10 client.6369 fetch_fsmap finished waiting for FSMap version 27 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f55c7dd7740 10 client.6369 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f55c7dd7740 20 client.6369 populate_metadata read hostname 'smithi045' 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] conn(0x5585a08eec70 0x5585a09123f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f55c7dd7740 4 client.6369 mds_command: new command op to 6219 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3851103796 --> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585a08247c0 con 0x5585a08eec70 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.103+0000 7f55bed9d640 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] conn(0x5585a08eec70 0x5585a09123f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.104+0000 7f55bed9d640 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] conn(0x5585a08eec70 0x5585a09123f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f779bc5554940133 server_cookie=99a1bd622cad95c in_seq=0 out_seq=0 2023-11-28T20:48:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.104+0000 7f559dffb640 10 client.6369 ms_handle_connect on v2:172.21.15.45:6833/3717491008 2023-11-28T20:48:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f559dffb640 1 -- 172.21.15.45:0/3851103796 <== mds.0 v2:172.21.15.45:6833/3717491008 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5585a08247c0 con 0x5585a08eec70 2023-11-28T20:48:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f559dffb640 10 client.6369 handle_command_reply: tid=0 2023-11-28T20:48:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f55c7dd7740 1 client.6369 shutdown 2023-11-28T20:48:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3851103796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 msgr2=0x5585a0910480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a0910480 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f55b8001d40 tx=0x7f55b800d520 comp rx=0 tx=0).stop 2023-11-28T20:48:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3851103796 shutdown_connections 2023-11-28T20:48:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] conn(0x5585a08eec70 0x5585a09123f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a0910480 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 0x5585a090ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3851103796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 0x5585a091a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f559dffb640 0 client.6369 ms_handle_reset on v2:172.21.15.45:6833/3717491008 2023-11-28T20:48:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.105+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3851103796 >> 172.21.15.45:0/3851103796 conn(0x5585a0886c70 msgr2=0x5585a08c5280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.106+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3851103796 shutdown_connections 2023-11-28T20:48:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.106+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3851103796 wait complete. 2023-11-28T20:48:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.106+0000 7f55c7dd7740 20 client.6369 trim_cache size 0 max 16384 2023-11-28T20:48:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.123+0000 7f55c7dd7740 1 Processor -- start 2023-11-28T20:48:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.123+0000 7f55c7dd7740 1 -- start start 2023-11-28T20:48:21.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.124+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08bfba0 0x5585a08bff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.124+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c45e0 0x5585a08c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.124+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c4ea0 0x5585a08e94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.124+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585a058ef20 con 0x5585a08c45e0 2023-11-28T20:48:21.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.124+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585a08eb090 con 0x5585a08c4ea0 2023-11-28T20:48:21.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.124+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5585a08c1090 con 0x5585a08bfba0 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.124+0000 7f55b659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c45e0 0x5585a08c04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.124+0000 7f55b659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c45e0 0x5585a08c04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34496/0 (socket says 172.21.15.45:34496) 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.124+0000 7f55b659c640 1 -- 172.21.15.45:0/2026874149 learned_addr learned my addr 172.21.15.45:0/2026874149 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55bf59e640 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c4ea0 0x5585a08e94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55bed9d640 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08bfba0 0x5585a08bff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55b659c640 1 -- 172.21.15.45:0/2026874149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08bfba0 msgr2=0x5585a08bff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55b659c640 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08bfba0 0x5585a08bff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55b659c640 1 -- 172.21.15.45:0/2026874149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c4ea0 msgr2=0x5585a08e94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55b659c640 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c4ea0 0x5585a08e94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55b659c640 1 -- 172.21.15.45:0/2026874149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585a07c7b80 con 0x5585a08c45e0 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55bf59e640 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c4ea0 0x5585a08e94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:21.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55b659c640 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c45e0 0x5585a08c04c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f55a8015850 tx=0x7f55a8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e47e5ac8b72ea5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55b559a640 1 -- 172.21.15.45:0/2026874149 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a800bb30 con 0x5585a08c45e0 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55b559a640 1 -- 172.21.15.45:0/2026874149 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a800bc90 con 0x5585a08c45e0 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.125+0000 7f55b559a640 1 -- 172.21.15.45:0/2026874149 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8027430 con 0x5585a08c45e0 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.126+0000 7f55c7dd7740 1 -- 172.21.15.45:0/2026874149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c45e0 msgr2=0x5585a08c04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.126+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c45e0 0x5585a08c04c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f55a8015850 tx=0x7f55a8012000 comp rx=0 tx=0).stop 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.126+0000 7f55c7dd7740 1 -- 172.21.15.45:0/2026874149 shutdown_connections 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.126+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08c4ea0 0x5585a08e94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.126+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c45e0 0x5585a08c04c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.126+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/2026874149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08bfba0 0x5585a08bff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.126+0000 7f55c7dd7740 1 -- 172.21.15.45:0/2026874149 >> 172.21.15.45:0/2026874149 conn(0x5585a0886c70 msgr2=0x5585a08c5280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.126+0000 7f55c7dd7740 1 -- 172.21.15.45:0/2026874149 shutdown_connections 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.126+0000 7f55c7dd7740 1 -- 172.21.15.45:0/2026874149 wait complete. 2023-11-28T20:48:21.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.127+0000 7f55c7dd7740 1 -- start start 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.127+0000 7f55c7dd7740 1 Processor -- start 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.128+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 0x5585a090feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.128+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 0x5585a09103f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.128+0000 7f55c7dd7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a08f2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.128+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585a090a590 con 0x5585a08c4ea0 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.128+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585a095abc0 con 0x5585a08bfba0 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.128+0000 7f55c7dd7740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5585a08ec140 con 0x5585a08c45e0 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.128+0000 7f55c7dd7740 10 client.0 _ll_register_callbacks cb 0x7f55b8007c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.128+0000 7f55bf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a08f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f55bf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a08f2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34500/0 (socket says 172.21.15.45:34500) 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f55bf59e640 1 -- 172.21.15.45:0/3968282560 learned_addr learned my addr 172.21.15.45:0/3968282560 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f55bed9d640 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 0x5585a090feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f55bf59e640 1 -- 172.21.15.45:0/3968282560 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 msgr2=0x5585a09103f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f55bf59e640 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 0x5585a09103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f55bf59e640 1 -- 172.21.15.45:0/3968282560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 msgr2=0x5585a090feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f55bf59e640 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 0x5585a090feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f55bf59e640 1 -- 172.21.15.45:0/3968282560 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5585a07f3860 con 0x5585a08c4ea0 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f55bf59e640 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a08f2560 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f55b8001650 tx=0x7f55b8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faae03db024a0193 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f559dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.129+0000 7f559dffb640 1 -- 172.21.15.45:0/3968282560 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b80054e0 con 0x5585a08c4ea0 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f559dffb640 1 -- 172.21.15.45:0/3968282560 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f55b80078f0 con 0x5585a08c4ea0 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3968282560 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f55a8015e00 con 0x5585a08c4ea0 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f559dffb640 1 -- 172.21.15.45:0/3968282560 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55b800da20 con 0x5585a08c4ea0 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55c7dd7740 10 client.6375 fetch_fsmap learned FSMap version 27 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3968282560 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x5585a07ed170 con 0x5585a08c4ea0 2023-11-28T20:48:21.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f559dffb640 1 -- 172.21.15.45:0/3968282560 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a8015e00 con 0x5585a08c4ea0 2023-11-28T20:48:21.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f559dffb640 1 -- 172.21.15.45:0/3968282560 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 27) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f55b8007040 con 0x5585a08c4ea0 2023-11-28T20:48:21.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55c7dd7740 10 client.6375 fetch_fsmap finished waiting for FSMap version 27 2023-11-28T20:48:21.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55c7dd7740 10 client.6375 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T20:48:21.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55c7dd7740 20 client.6375 populate_metadata read hostname 'smithi045' 2023-11-28T20:48:21.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] conn(0x5585a08f2aa0 0x5585a08f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55c7dd7740 4 client.6375 mds_command: new command op to 6219 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:48:21.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3968282560 --> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5585a052c6c0 con 0x5585a08f2aa0 2023-11-28T20:48:21.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.130+0000 7f55bed9d640 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] conn(0x5585a08f2aa0 0x5585a08f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.131+0000 7f55bed9d640 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] conn(0x5585a08f2aa0 0x5585a08f2e80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50c938a05225d95c server_cookie=8bec92fa019ac556 in_seq=0 out_seq=0 2023-11-28T20:48:21.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.131+0000 7f559dffb640 10 client.6375 ms_handle_connect on v2:172.21.15.45:6833/3717491008 2023-11-28T20:48:21.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.131+0000 7f559dffb640 1 -- 172.21.15.45:0/3968282560 <== mds.0 v2:172.21.15.45:6833/3717491008 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5585a052c6c0 con 0x5585a08f2aa0 2023-11-28T20:48:21.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.131+0000 7f559dffb640 10 client.6375 handle_command_reply: tid=0 2023-11-28T20:48:21.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.131+0000 7f55c7dd7740 1 client.6375 shutdown 2023-11-28T20:48:21.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.131+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3968282560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 msgr2=0x5585a08f2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.131+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a08f2560 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f55b8001650 tx=0x7f55b8001120 comp rx=0 tx=0).stop 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3968282560 shutdown_connections 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.45:6833/3717491008,v1:172.21.15.45:6835/3717491008] conn(0x5585a08f2aa0 0x5585a08f2e80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585a08c4ea0 0x5585a08f2560 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5585a08c45e0 0x5585a09103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c7dd7740 1 --2- 172.21.15.45:0/3968282560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585a08bfba0 0x5585a090feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3968282560 >> 172.21.15.45:0/3968282560 conn(0x5585a0886c70 msgr2=0x5585a08bff80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f559dffb640 0 client.6375 ms_handle_reset on v2:172.21.15.45:6833/3717491008 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3968282560 shutdown_connections 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c7dd7740 1 -- 172.21.15.45:0/3968282560 wait complete. 2023-11-28T20:48:21.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c7dd7740 20 client.6375 trim_cache size 0 max 16384 2023-11-28T20:48:21.145 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:48:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:48:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:48:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 6219, 2023-11-28T20:48:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:48:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:48:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:48:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.8302177070000001, 2023-11-28T20:48:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 27, 2023-11-28T20:48:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 71, 2023-11-28T20:48:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 71, 2023-11-28T20:48:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 12.665519829999999 2023-11-28T20:48:21.146 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c5244640 1 -- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f55a006af90 msgr2=0x7f55a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c5244640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f55a006af90 0x7f55a006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f55c00d2670 tx=0x7f55a8006c90 comp rx=0 tx=0).stop 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c5244640 1 -- 172.21.15.45:0/1250263318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c0071060 msgr2=0x7f55c01f7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.132+0000 7f55c5244640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c0071060 0x7f55c01f7160 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f55b0036040 tx=0x7f55b0004a40 comp rx=0 tx=0).stop 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.133+0000 7f55c5244640 1 -- 172.21.15.45:0/1250263318 shutdown_connections 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.133+0000 7f55c5244640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f55a006af90 0x7f55a006d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.133+0000 7f55c5244640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f55c015d730 0x7f55c006a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.133+0000 7f55c5244640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f55c00d2750 0x7f55c01481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.133+0000 7f55c5244640 1 --2- 172.21.15.45:0/1250263318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55c0071060 0x7f55c01f7160 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.133+0000 7f55c5244640 1 -- 172.21.15.45:0/1250263318 >> 172.21.15.45:0/1250263318 conn(0x7f55c00cfc70 msgr2=0x7f55c015fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.133+0000 7f55c5244640 1 -- 172.21.15.45:0/1250263318 shutdown_connections 2023-11-28T20:48:21.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.133+0000 7f55c5244640 1 -- 172.21.15.45:0/1250263318 wait complete. 2023-11-28T20:48:21.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:48:21.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.382+0000 7f1b9e3de640 1 Processor -- start 2023-11-28T20:48:21.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.382+0000 7f1b9e3de640 1 -- start start 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.384+0000 7f1b9e3de640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b980b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.384+0000 7f1b9e3de640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1b980b1340 0x7f1b980b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.384+0000 7f1b9e3de640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b9815dcc0 con 0x7f1b980b24e0 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.384+0000 7f1b9e3de640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9815e840 con 0x7f1b980b1340 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.384+0000 7f1b9e3de640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9815df10 con 0x7f1b981438b0 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.385+0000 7f1b97fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b980b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.385+0000 7f1b9c97b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1b980b1340 0x7f1b980b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.385+0000 7f1b9c97b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1b980b1340 0x7f1b980b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:55912/0 (socket says 172.21.15.45:55912) 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.385+0000 7f1b97fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b980b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:43230/0 (socket says 172.21.15.45:43230) 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.385+0000 7f1b9c97b640 1 -- 172.21.15.45:0/776221798 learned_addr learned my addr 172.21.15.45:0/776221798 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.386+0000 7f1b97fff640 1 -- 172.21.15.45:0/776221798 >> v2:172.21.15.190:3300/0 conn(0x7f1b980b1340 msgr2=0x7f1b980b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.386+0000 7f1b97fff640 1 --2- 172.21.15.45:0/776221798 >> v2:172.21.15.190:3300/0 conn(0x7f1b980b1340 0x7f1b980b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.386+0000 7f1b97fff640 1 -- 172.21.15.45:0/776221798 >> v1:172.21.15.190:6789/0 conn(0x7f1b980b24e0 legacy=0x7f1b980b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.386+0000 7f1b97fff640 1 -- 172.21.15.45:0/776221798 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9815e1f0 con 0x7f1b981438b0 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.388+0000 7f1b97fff640 1 --2- 172.21.15.45:0/776221798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b980b1fa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1b880097a0 tx=0x7f1b8802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a57e607001e3e752 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.388+0000 7f1b977fe640 1 -- 172.21.15.45:0/776221798 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8802f050 con 0x7f1b981438b0 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.388+0000 7f1b977fe640 1 -- 172.21.15.45:0/776221798 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b88031700 con 0x7f1b981438b0 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.388+0000 7f1b977fe640 1 -- 172.21.15.45:0/776221798 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88031a90 con 0x7f1b981438b0 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.389+0000 7f1b9e3de640 1 -- 172.21.15.45:0/776221798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 msgr2=0x7f1b980b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.389+0000 7f1b9e3de640 1 --2- 172.21.15.45:0/776221798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b980b1fa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1b880097a0 tx=0x7f1b8802b7a0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.389+0000 7f1b9e3de640 1 -- 172.21.15.45:0/776221798 shutdown_connections 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.389+0000 7f1b9e3de640 1 --2- 172.21.15.45:0/776221798 >> v2:172.21.15.190:3300/0 conn(0x7f1b980b1340 0x7f1b980b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.389+0000 7f1b9e3de640 1 --2- 172.21.15.45:0/776221798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b980b1fa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.389+0000 7f1b9e3de640 1 -- 172.21.15.45:0/776221798 >> 172.21.15.45:0/776221798 conn(0x7f1b9806a500 msgr2=0x7f1b9806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.389+0000 7f1b9e3de640 1 -- 172.21.15.45:0/776221798 shutdown_connections 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.389+0000 7f1b9e3de640 1 -- 172.21.15.45:0/776221798 wait complete. 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.391+0000 7f1b9e3de640 1 Processor -- start 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.391+0000 7f1b9e3de640 1 -- start start 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.392+0000 7f1b9e3de640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b980b1340 0x7f1b9813efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.392+0000 7f1b9e3de640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b980b24e0 0x7f1b980ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.392+0000 7f1b9e3de640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b98157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.392+0000 7f1b9e3de640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9813df00 con 0x7f1b980b24e0 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.392+0000 7f1b9e3de640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9813e1e0 con 0x7f1b980b1340 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.392+0000 7f1b9e3de640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9813e4c0 con 0x7f1b981438b0 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b8ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b980b24e0 0x7f1b980ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b8ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b980b24e0 0x7f1b980ca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34514/0 (socket says 172.21.15.45:34514) 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b8ffff640 1 -- 172.21.15.45:0/561325074 learned_addr learned my addr 172.21.15.45:0/561325074 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b97fff640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b980b1340 0x7f1b9813efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b9c97b640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b98157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b8ffff640 1 -- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 msgr2=0x7f1b98157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b8ffff640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b98157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b8ffff640 1 -- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b980b1340 msgr2=0x7f1b9813efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b8ffff640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b980b1340 0x7f1b9813efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b8ffff640 1 -- 172.21.15.45:0/561325074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b88009400 con 0x7f1b980b24e0 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.393+0000 7f1b97fff640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b980b1340 0x7f1b9813efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.394+0000 7f1b8ffff640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b980b24e0 0x7f1b980ca4c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000c3c0 tx=0x7f1b8000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.394+0000 7f1b957fa640 1 -- 172.21.15.45:0/561325074 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b800052d0 con 0x7f1b980b24e0 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.394+0000 7f1b957fa640 1 -- 172.21.15.45:0/561325074 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b80005450 con 0x7f1b980b24e0 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.394+0000 7f1b957fa640 1 -- 172.21.15.45:0/561325074 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80003830 con 0x7f1b980b24e0 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.394+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9813e7a0 con 0x7f1b980b24e0 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.394+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9813f4e0 con 0x7f1b980b24e0 2023-11-28T20:48:21.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.395+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9813ecc0 con 0x7f1b980b24e0 2023-11-28T20:48:21.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.396+0000 7f1b957fa640 1 -- 172.21.15.45:0/561325074 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1b80003990 con 0x7f1b980b24e0 2023-11-28T20:48:21.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.397+0000 7f1b957fa640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f1b7806af60 0x7f1b7806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:21.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.397+0000 7f1b957fa640 1 -- 172.21.15.45:0/561325074 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1b80088e30 con 0x7f1b980b24e0 2023-11-28T20:48:21.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.398+0000 7f1b97fff640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f1b7806af60 0x7f1b7806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:21.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.399+0000 7f1b97fff640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f1b7806af60 0x7f1b7806d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1b88035000 tx=0x7f1b88008030 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:21.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.401+0000 7f1b957fa640 1 -- 172.21.15.45:0/561325074 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b9813ecc0 con 0x7f1b980b24e0 2023-11-28T20:48:21.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.636+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1b9813ecc0 con 0x7f1b980b24e0 2023-11-28T20:48:21.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.637+0000 7f1b957fa640 1 -- 172.21.15.45:0/561325074 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1b9813ecc0 con 0x7f1b980b24e0 2023-11-28T20:48:21.639 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T20:48:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.639+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f1b7806af60 msgr2=0x7f1b7806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.639+0000 7f1b9e3de640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f1b7806af60 0x7f1b7806d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1b88035000 tx=0x7f1b88008030 comp rx=0 tx=0).stop 2023-11-28T20:48:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.639+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b980b24e0 msgr2=0x7f1b980ca4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.639+0000 7f1b9e3de640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b980b24e0 0x7f1b980ca4c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000c3c0 tx=0x7f1b8000c880 comp rx=0 tx=0).stop 2023-11-28T20:48:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.640+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 shutdown_connections 2023-11-28T20:48:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.640+0000 7f1b9e3de640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f1b7806af60 0x7f1b7806d420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.640+0000 7f1b9e3de640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b981438b0 0x7f1b98157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.640+0000 7f1b9e3de640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b980b24e0 0x7f1b980ca4c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.640+0000 7f1b9e3de640 1 --2- 172.21.15.45:0/561325074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b980b1340 0x7f1b9813efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.640+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 >> 172.21.15.45:0/561325074 conn(0x7f1b9806a500 msgr2=0x7f1b98159870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.640+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 shutdown_connections 2023-11-28T20:48:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:21.640+0000 7f1b9e3de640 1 -- 172.21.15.45:0/561325074 wait complete. 2023-11-28T20:48:21.663 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T20:48:21.663 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:48:21.663 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:48:21.663 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T20:48:21.663 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:48:21.663 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T20:48:21.663 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T20:48:21.663 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:48:21.663 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:48:21.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:21.664 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:48:21.720 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:48:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:48:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:48:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:48:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:48:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 41441sec preferred_lft 41441sec 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:48:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T20:48:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:48:21.723 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:48:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T20:48:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T20:48:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:48:21.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:21.723 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:48:21.820 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:48:21.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:21.825 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:48:21.880 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:48:21.881 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:48:21.881 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:48:21.881 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:48:21.881 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:48:21.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:21.881 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:48:21.882 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:48:21.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:48:21.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:48:21.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:48:21.882 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:48:22.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:22.080 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:48:22.136 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T20:48:22.136 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T20:48:22.137 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:48:22.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:22.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:48:22.240 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:48:22.240 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T20:48:22.303 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:48:22.304 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:48:22.304 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:48:22.360 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:48:22.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:22.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:48:22.509 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:48:22.509 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:48:22.509 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:48:22.532 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:48:22.533 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:48:22.533 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:48:22.533 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:48:22.533 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:48:22.561 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:48:22.561 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:48:22.561 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:48:22.561 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:48:22.561 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:48:22.561 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:48:22.561 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:48:22.561 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:48:22.563 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T20:48:22.563 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:48:22.563 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:48:22.563 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:48:22.564 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:48:22.567 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:48:22.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:22.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:48:22.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:22.657 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:48:22.721 INFO:teuthology.orchestra.run.smithi045.stdout:client5237 2023-11-28T20:48:22.721 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:48:22.721 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:48:22.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T20:48:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.001+0000 7f43b499d640 1 Processor -- start 2023-11-28T20:48:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.001+0000 7f43b499d640 1 -- start start 2023-11-28T20:48:23.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.002+0000 7f43b499d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f43b0159ed0 0x7f43b0071060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.002+0000 7f43b499d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0071e80 0x7f43b00ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.003+0000 7f43b499d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b0070bd0 con 0x7f43b00cef00 2023-11-28T20:48:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.003+0000 7f43b499d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b00adad0 con 0x7f43b0159ed0 2023-11-28T20:48:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.003+0000 7f43b499d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00702a0 con 0x7f43b0071e80 2023-11-28T20:48:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.003+0000 7f43aed9d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f43b0159ed0 0x7f43b0071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.003+0000 7f43aed9d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f43b0159ed0 0x7f43b0071060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:55918/0 (socket says 172.21.15.45:55918) 2023-11-28T20:48:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.003+0000 7f43a7fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0071e80 0x7f43b00ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.003+0000 7f43aed9d640 1 -- 172.21.15.45:0/827414794 learned_addr learned my addr 172.21.15.45:0/827414794 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.004+0000 7f43a7fff640 1 -- 172.21.15.45:0/827414794 >> v2:172.21.15.190:3300/0 conn(0x7f43b0159ed0 msgr2=0x7f43b0071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.004+0000 7f43a7fff640 1 --2- 172.21.15.45:0/827414794 >> v2:172.21.15.190:3300/0 conn(0x7f43b0159ed0 0x7f43b0071060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.004+0000 7f43a7fff640 1 -- 172.21.15.45:0/827414794 >> v1:172.21.15.190:6789/0 conn(0x7f43b00cef00 legacy=0x7f43b00b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.004+0000 7f43a7fff640 1 -- 172.21.15.45:0/827414794 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0070580 con 0x7f43b0071e80 2023-11-28T20:48:23.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.006+0000 7f43a7fff640 1 --2- 172.21.15.45:0/827414794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0071e80 0x7f43b00ad1d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f43980097a0 tx=0x7f439802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f857774e0b59376b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:23.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43ae59c640 1 -- 172.21.15.45:0/827414794 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439802e070 con 0x7f43b0071e80 2023-11-28T20:48:23.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43ae59c640 1 -- 172.21.15.45:0/827414794 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43980315a0 con 0x7f43b0071e80 2023-11-28T20:48:23.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43ae59c640 1 -- 172.21.15.45:0/827414794 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4398031930 con 0x7f43b0071e80 2023-11-28T20:48:23.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43b499d640 1 -- 172.21.15.45:0/827414794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0071e80 msgr2=0x7f43b00ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43b499d640 1 --2- 172.21.15.45:0/827414794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0071e80 0x7f43b00ad1d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f43980097a0 tx=0x7f439802b840 comp rx=0 tx=0).stop 2023-11-28T20:48:23.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43b499d640 1 -- 172.21.15.45:0/827414794 shutdown_connections 2023-11-28T20:48:23.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43b499d640 1 --2- 172.21.15.45:0/827414794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0071e80 0x7f43b00ad1d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43b499d640 1 --2- 172.21.15.45:0/827414794 >> v2:172.21.15.190:3300/0 conn(0x7f43b0159ed0 0x7f43b0071060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43b499d640 1 -- 172.21.15.45:0/827414794 >> 172.21.15.45:0/827414794 conn(0x7f43b006a710 msgr2=0x7f43b006ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:23.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.007+0000 7f43b499d640 1 -- 172.21.15.45:0/827414794 shutdown_connections 2023-11-28T20:48:23.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.008+0000 7f43b499d640 1 -- 172.21.15.45:0/827414794 wait complete. 2023-11-28T20:48:23.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.009+0000 7f43b499d640 1 Processor -- start 2023-11-28T20:48:23.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.009+0000 7f43b499d640 1 -- start start 2023-11-28T20:48:23.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.010+0000 7f43b499d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b0071e80 0x7f43b014dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.010+0000 7f43b499d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43b00cef00 0x7f43b014fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.010+0000 7f43b499d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0159ed0 0x7f43b0144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.010+0000 7f43b499d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b014bac0 con 0x7f43b00cef00 2023-11-28T20:48:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.010+0000 7f43b499d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b014ae80 con 0x7f43b0071e80 2023-11-28T20:48:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.010+0000 7f43b499d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b014b120 con 0x7f43b0159ed0 2023-11-28T20:48:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43a7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43b00cef00 0x7f43b014fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43a7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43b00cef00 0x7f43b014fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34528/0 (socket says 172.21.15.45:34528) 2023-11-28T20:48:23.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43a7fff640 1 -- 172.21.15.45:0/3217137526 learned_addr learned my addr 172.21.15.45:0/3217137526 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43aed9d640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b0071e80 0x7f43b014dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43af59e640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0159ed0 0x7f43b0144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43a7fff640 1 -- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0159ed0 msgr2=0x7f43b0144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43a7fff640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0159ed0 0x7f43b0144070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43a7fff640 1 -- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b0071e80 msgr2=0x7f43b014dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43a7fff640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b0071e80 0x7f43b014dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.011+0000 7f43a7fff640 1 -- 172.21.15.45:0/3217137526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4398009400 con 0x7f43b00cef00 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.012+0000 7f43a7fff640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43b00cef00 0x7f43b014fe80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4398034040 tx=0x7f43980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.012+0000 7f43a77fe640 1 -- 172.21.15.45:0/3217137526 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4398032840 con 0x7f43b00cef00 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.012+0000 7f43a77fe640 1 -- 172.21.15.45:0/3217137526 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4398031f00 con 0x7f43b00cef00 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.012+0000 7f43a77fe640 1 -- 172.21.15.45:0/3217137526 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4398032be0 con 0x7f43b00cef00 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.012+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b014b400 con 0x7f43b00cef00 2023-11-28T20:48:23.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.012+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b014e400 con 0x7f43b00cef00 2023-11-28T20:48:23.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.013+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b014e5c0 con 0x7f43b00cef00 2023-11-28T20:48:23.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.014+0000 7f43a77fe640 1 -- 172.21.15.45:0/3217137526 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4398032d40 con 0x7f43b00cef00 2023-11-28T20:48:23.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.015+0000 7f43a77fe640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f439006af60 0x7f439006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.015+0000 7f43a77fe640 1 -- 172.21.15.45:0/3217137526 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f43980ab940 con 0x7f43b00cef00 2023-11-28T20:48:23.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.016+0000 7f43aed9d640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f439006af60 0x7f439006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.017+0000 7f43aed9d640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f439006af60 0x7f439006d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009b60 tx=0x7f43a0008040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:23.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.020+0000 7f43a77fe640 1 -- 172.21.15.45:0/3217137526 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f43b014e5c0 con 0x7f43b00cef00 2023-11-28T20:48:23.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.243+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 --> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f43b0152cf0 con 0x7f439006af60 2023-11-28T20:48:23.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.248+0000 7f43a77fe640 1 -- 172.21.15.45:0/3217137526 <== mgr.5305 v2:172.21.15.190:6832/838517151 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f43b0152cf0 con 0x7f439006af60 2023-11-28T20:48:23.250 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:48:23.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:48:23.250 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T20:48:23.250 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:48:23.250 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:48:23.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f439006af60 msgr2=0x7f439006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f439006af60 0x7f439006d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009b60 tx=0x7f43a0008040 comp rx=0 tx=0).stop 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43b00cef00 msgr2=0x7f43b014fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43b00cef00 0x7f43b014fe80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4398034040 tx=0x7f43980046c0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 shutdown_connections 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f439006af60 0x7f439006d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43b0159ed0 0x7f43b0144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f43b00cef00 0x7f43b014fe80 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 --2- 172.21.15.45:0/3217137526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b0071e80 0x7f43b014dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 >> 172.21.15.45:0/3217137526 conn(0x7f43b006a710 msgr2=0x7f43b01525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 shutdown_connections 2023-11-28T20:48:23.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.250+0000 7f43b499d640 1 -- 172.21.15.45:0/3217137526 wait complete. 2023-11-28T20:48:23.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T20:48:23.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.496+0000 7f07660c3640 1 Processor -- start 2023-11-28T20:48:23.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.496+0000 7f07660c3640 1 -- start start 2023-11-28T20:48:23.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.498+0000 7f07660c3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f07600cf940 0x7f0760161800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.498+0000 7f07660c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07600b2930 con 0x7f07600d28e0 2023-11-28T20:48:23.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.498+0000 7f07660c3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0760161d40 con 0x7f07600cfd20 2023-11-28T20:48:23.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.498+0000 7f07660c3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07600b5950 con 0x7f07600cf940 2023-11-28T20:48:23.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.498+0000 7f07658c2640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07600d28e0 0x7f07600b5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58254/0 (socket says 172.21.15.45:58254) 2023-11-28T20:48:23.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.498+0000 7f07658c2640 1 -- 172.21.15.45:0/948497353 learned_addr learned my addr 172.21.15.45:0/948497353 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:23.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f07650c1640 1 --2- 172.21.15.45:0/948497353 >> v2:172.21.15.190:3300/0 conn(0x7f07600cf940 0x7f0760161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239071238 0 0) 0x7f07600b2930 con 0x7f07600d28e0 2023-11-28T20:48:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f074c003520 con 0x7f07600d28e0 2023-11-28T20:48:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2974904129 0 0) 0x7f074c003520 con 0x7f07600d28e0 2023-11-28T20:48:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 >> v2:172.21.15.190:3300/0 conn(0x7f07600cf940 msgr2=0x7f0760161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f0757fff640 1 --2- 172.21.15.45:0/948497353 >> v2:172.21.15.190:3300/0 conn(0x7f07600cf940 0x7f0760161800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 >> v1:172.21.15.190:6789/0 conn(0x7f07600cfd20 legacy=0x7f07600b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07600b2650 con 0x7f07600d28e0 2023-11-28T20:48:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f07580019f0 con 0x7f07600d28e0 2023-11-28T20:48:23.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.499+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 >> v1:172.21.15.45:6789/0 conn(0x7f07600d28e0 legacy=0x7f07600b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.500+0000 7f0757fff640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f074c003970 0x7f074c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.500+0000 7f0757fff640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f074c0047c0 0x7f074c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.500+0000 7f0757fff640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f074c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.500+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07600b5950 con 0x7f074c0052a0 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.500+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c00c170 con 0x7f074c003970 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.500+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c00c330 con 0x7f074c0047c0 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.500+0000 7f07650c1640 1 --2- 172.21.15.45:0/948497353 >> v2:172.21.15.190:3300/0 conn(0x7f07600cf940 0x7f0760161800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.500+0000 7f07658c2640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f074c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.500+0000 7f07648c0640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f074c0047c0 0x7f074c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.501+0000 7f07650c1640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f074c003970 0x7f074c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.501+0000 7f07658c2640 1 -- 172.21.15.45:0/948497353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f074c0047c0 msgr2=0x7f074c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.501+0000 7f07658c2640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f074c0047c0 0x7f074c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.501+0000 7f07658c2640 1 -- 172.21.15.45:0/948497353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f074c003970 msgr2=0x7f074c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:23.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.501+0000 7f07658c2640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f074c003970 0x7f074c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.501+0000 7f07658c2640 1 -- 172.21.15.45:0/948497353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c003520 con 0x7f074c0052a0 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.502+0000 7f07658c2640 1 -- 172.21.15.45:0/948497353 reap_dead start 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.502+0000 7f07658c2640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f074c00b980 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f07600d2d70 tx=0x7f074c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a87fb96e10eb42b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.502+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07580072d0 con 0x7f074c0052a0 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.502+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075802f070 con 0x7f074c0052a0 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.502+0000 7f0757fff640 1 -- 172.21.15.45:0/948497353 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758007670 con 0x7f074c0052a0 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.503+0000 7f07660c3640 1 -- 172.21.15.45:0/948497353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 msgr2=0x7f074c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.503+0000 7f07660c3640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f074c00b980 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f07600d2d70 tx=0x7f074c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.503+0000 7f07660c3640 1 -- 172.21.15.45:0/948497353 shutdown_connections 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.503+0000 7f07660c3640 1 --2- 172.21.15.45:0/948497353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f074c00b980 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.503+0000 7f07660c3640 1 -- 172.21.15.45:0/948497353 >> 172.21.15.45:0/948497353 conn(0x7f076006a500 msgr2=0x7f076006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:23.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.503+0000 7f07660c3640 1 -- 172.21.15.45:0/948497353 shutdown_connections 2023-11-28T20:48:23.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.503+0000 7f07660c3640 1 -- 172.21.15.45:0/948497353 wait complete. 2023-11-28T20:48:23.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.505+0000 7f07660c3640 1 Processor -- start 2023-11-28T20:48:23.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.505+0000 7f07660c3640 1 -- start start 2023-11-28T20:48:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07660c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f07601561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07660c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0760159e90 0x7f076015dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07660c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f076015e140 0x7f07600cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07660c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076015e5f0 con 0x7f074c0052a0 2023-11-28T20:48:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07660c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07600cb4e0 con 0x7f0760159e90 2023-11-28T20:48:23.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07660c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0760156700 con 0x7f076015e140 2023-11-28T20:48:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07650c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f07601561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07650c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f07601561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34544/0 (socket says 172.21.15.45:34544) 2023-11-28T20:48:23.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07650c1640 1 -- 172.21.15.45:0/3657256191 learned_addr learned my addr 172.21.15.45:0/3657256191 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07648c0640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0760159e90 0x7f076015dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07658c2640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f076015e140 0x7f07600cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07650c1640 1 -- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f076015e140 msgr2=0x7f07600cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07650c1640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f076015e140 0x7f07600cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07650c1640 1 -- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0760159e90 msgr2=0x7f076015dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07650c1640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0760159e90 0x7f076015dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07650c1640 1 -- 172.21.15.45:0/3657256191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07601e9220 con 0x7f074c0052a0 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.506+0000 7f07648c0640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0760159e90 0x7f076015dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.507+0000 7f07650c1640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f07601561c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f075c010920 tx=0x7f075c010de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.507+0000 7f0755ffb640 1 -- 172.21.15.45:0/3657256191 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c00ec20 con 0x7f074c0052a0 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.507+0000 7f0755ffb640 1 -- 172.21.15.45:0/3657256191 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c00ed80 con 0x7f074c0052a0 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.507+0000 7f0755ffb640 1 -- 172.21.15.45:0/3657256191 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c01c640 con 0x7f074c0052a0 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.507+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07601e9500 con 0x7f074c0052a0 2023-11-28T20:48:23.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.507+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07600cb170 con 0x7f074c0052a0 2023-11-28T20:48:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.509+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0760071b30 con 0x7f074c0052a0 2023-11-28T20:48:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.510+0000 7f0755ffb640 1 -- 172.21.15.45:0/3657256191 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f075c005990 con 0x7f074c0052a0 2023-11-28T20:48:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.510+0000 7f0755ffb640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f074406af90 0x7f074406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:23.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.511+0000 7f0755ffb640 1 -- 172.21.15.45:0/3657256191 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f075c089b60 con 0x7f074c0052a0 2023-11-28T20:48:23.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.511+0000 7f07648c0640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f074406af90 0x7f074406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:23.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.512+0000 7f07648c0640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f074406af90 0x7f074406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0750000c00 tx=0x7f0750009040 comp rx=0 tx=0).ready entity=mgr.5305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:23.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.515+0000 7f0755ffb640 1 -- 172.21.15.45:0/3657256191 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f075c0549c0 con 0x7f074c0052a0 2023-11-28T20:48:23.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:23.758+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f076004e920 con 0x7f074c0052a0 2023-11-28T20:48:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.377+0000 7f0755ffb640 1 -- 172.21.15.45:0/3657256191 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f075c057fe0 con 0x7f074c0052a0 2023-11-28T20:48:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.380+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f074406af90 msgr2=0x7f074406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.380+0000 7f07660c3640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f074406af90 0x7f074406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0750000c00 tx=0x7f0750009040 comp rx=0 tx=0).stop 2023-11-28T20:48:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.381+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 msgr2=0x7f07601561c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.381+0000 7f07660c3640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f07601561c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f075c010920 tx=0x7f075c010de0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.381+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 shutdown_connections 2023-11-28T20:48:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.381+0000 7f07660c3640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f074406af90 0x7f074406d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.381+0000 7f07660c3640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f076015e140 0x7f07600cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.381+0000 7f07660c3640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0760159e90 0x7f076015dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.381+0000 7f07660c3640 1 --2- 172.21.15.45:0/3657256191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f074c0052a0 0x7f07601561c0 secure :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f075c010920 tx=0x7f075c010de0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.381+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 >> 172.21.15.45:0/3657256191 conn(0x7f076006a500 msgr2=0x7f07600b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.382+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 shutdown_connections 2023-11-28T20:48:24.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.383+0000 7f07660c3640 1 -- 172.21.15.45:0/3657256191 wait complete. 2023-11-28T20:48:24.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T20:48:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.625+0000 7f4722761640 1 Processor -- start 2023-11-28T20:48:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.625+0000 7f4722761640 1 -- start start 2023-11-28T20:48:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.627+0000 7f4722761640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f471c0c1d30 0x7f471c0ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.627+0000 7f4722761640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f471c1653a0 0x7f471c0c2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.627+0000 7f4722761640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c15f7e0 con 0x7f471c0712c0 2023-11-28T20:48:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.627+0000 7f4722761640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c0c2650 con 0x7f471c0c1d30 2023-11-28T20:48:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.627+0000 7f4722761640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c1658a0 con 0x7f471c1653a0 2023-11-28T20:48:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.627+0000 7f471bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f471c0c1d30 0x7f471c0ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.627+0000 7f471bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f471c0c1d30 0x7f471c0ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45374/0 (socket says 172.21.15.45:45374) 2023-11-28T20:48:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.627+0000 7f471bfff640 1 -- 172.21.15.45:0/1245709443 learned_addr learned my addr 172.21.15.45:0/1245709443 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.628+0000 7f4713fff640 1 --2- 172.21.15.45:0/1245709443 >> v2:172.21.15.190:3300/0 conn(0x7f471c1653a0 0x7f471c0c2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.628+0000 7f471bfff640 1 -- 172.21.15.45:0/1245709443 >> v2:172.21.15.190:3300/0 conn(0x7f471c1653a0 msgr2=0x7f471c0c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.628+0000 7f471bfff640 1 --2- 172.21.15.45:0/1245709443 >> v2:172.21.15.190:3300/0 conn(0x7f471c1653a0 0x7f471c0c2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.628+0000 7f471bfff640 1 -- 172.21.15.45:0/1245709443 >> v1:172.21.15.190:6789/0 conn(0x7f471c0712c0 legacy=0x7f471c162290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.628+0000 7f471bfff640 1 -- 172.21.15.45:0/1245709443 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c165b80 con 0x7f471c0c1d30 2023-11-28T20:48:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.629+0000 7f471bfff640 1 --2- 172.21.15.45:0/1245709443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0c1d30 0x7f471c0ce480 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f470c009870 tx=0x7f470c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2d114dc8e55eef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:24.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.629+0000 7f471b7fe640 1 -- 172.21.15.45:0/1245709443 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c031860 con 0x7f471c0c1d30 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.629+0000 7f471b7fe640 1 -- 172.21.15.45:0/1245709443 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c0319c0 con 0x7f471c0c1d30 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.629+0000 7f471b7fe640 1 -- 172.21.15.45:0/1245709443 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c031d50 con 0x7f471c0c1d30 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.630+0000 7f4722761640 1 -- 172.21.15.45:0/1245709443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0c1d30 msgr2=0x7f471c0ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.630+0000 7f4722761640 1 --2- 172.21.15.45:0/1245709443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0c1d30 0x7f471c0ce480 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f470c009870 tx=0x7f470c02b900 comp rx=0 tx=0).stop 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.630+0000 7f4722761640 1 -- 172.21.15.45:0/1245709443 shutdown_connections 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.630+0000 7f4722761640 1 --2- 172.21.15.45:0/1245709443 >> v2:172.21.15.190:3300/0 conn(0x7f471c1653a0 0x7f471c0c2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.630+0000 7f4722761640 1 --2- 172.21.15.45:0/1245709443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0c1d30 0x7f471c0ce480 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.630+0000 7f4722761640 1 -- 172.21.15.45:0/1245709443 >> 172.21.15.45:0/1245709443 conn(0x7f471c0ced60 msgr2=0x7f471c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.630+0000 7f4722761640 1 -- 172.21.15.45:0/1245709443 shutdown_connections 2023-11-28T20:48:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.630+0000 7f4722761640 1 -- 172.21.15.45:0/1245709443 wait complete. 2023-11-28T20:48:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.632+0000 7f4722761640 1 Processor -- start 2023-11-28T20:48:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.632+0000 7f4722761640 1 -- start start 2023-11-28T20:48:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.633+0000 7f4722761640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0712c0 0x7f471c1f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.633+0000 7f4722761640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f471c0c1d30 0x7f471c15b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.633+0000 7f4722761640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f471c1653a0 0x7f471c1ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.633+0000 7f4722761640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c15b2d0 con 0x7f471c0712c0 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.633+0000 7f4722761640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c0c3510 con 0x7f471c1653a0 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.633+0000 7f4722761640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c0c36a0 con 0x7f471c0c1d30 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.633+0000 7f471bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0712c0 0x7f471c1f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.633+0000 7f471bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0712c0 0x7f471c1f70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45380/0 (socket says 172.21.15.45:45380) 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.633+0000 7f471bfff640 1 -- 172.21.15.45:0/2612502319 learned_addr learned my addr 172.21.15.45:0/2612502319 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.634+0000 7f4713fff640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f471c0c1d30 0x7f471c15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.634+0000 7f4720cfe640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f471c1653a0 0x7f471c1ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.634+0000 7f471bfff640 1 -- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f471c0c1d30 msgr2=0x7f471c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.634+0000 7f471bfff640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f471c0c1d30 0x7f471c15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.634+0000 7f471bfff640 1 -- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f471c1653a0 msgr2=0x7f471c1ebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.634+0000 7f471bfff640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f471c1653a0 0x7f471c1ebba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.634+0000 7f471bfff640 1 -- 172.21.15.45:0/2612502319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c0094d0 con 0x7f471c0712c0 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.634+0000 7f4720cfe640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f471c1653a0 0x7f471c1ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.634+0000 7f471bfff640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0712c0 0x7f471c1f70f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f470c009870 tx=0x7f470c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.635+0000 7f47197fa640 1 -- 172.21.15.45:0/2612502319 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c031810 con 0x7f471c0712c0 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.635+0000 7f47197fa640 1 -- 172.21.15.45:0/2612502319 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c0443f0 con 0x7f471c0712c0 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.635+0000 7f47197fa640 1 -- 172.21.15.45:0/2612502319 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c0447a0 con 0x7f471c0712c0 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.635+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c1ec260 con 0x7f471c0712c0 2023-11-28T20:48:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.635+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c1ec820 con 0x7f471c0712c0 2023-11-28T20:48:24.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.636+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c0733d0 con 0x7f471c0712c0 2023-11-28T20:48:24.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.637+0000 7f47197fa640 1 -- 172.21.15.45:0/2612502319 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f470c04d450 con 0x7f471c0712c0 2023-11-28T20:48:24.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.638+0000 7f47197fa640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f46fc06afe0 0x7f46fc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:24.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.638+0000 7f47197fa640 1 -- 172.21.15.45:0/2612502319 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f470c0ac180 con 0x7f471c0712c0 2023-11-28T20:48:24.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.638+0000 7f4713fff640 1 -- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f46fc06afe0 msgr2=0x7f46fc06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/838517151 2023-11-28T20:48:24.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.638+0000 7f4713fff640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f46fc06afe0 0x7f46fc06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:48:24.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.642+0000 7f47197fa640 1 -- 172.21.15.45:0/2612502319 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f470c076ec0 con 0x7f471c0712c0 2023-11-28T20:48:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.838+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f471c0b9300 con 0x7f471c0712c0 2023-11-28T20:48:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.839+0000 7f4713fff640 1 -- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f46fc06afe0 msgr2=0x7f46fc06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/838517151 2023-11-28T20:48:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.839+0000 7f4713fff640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f46fc06afe0 0x7f46fc06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:48:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.845+0000 7f47197fa640 1 -- 172.21.15.45:0/2612502319 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f470c07a4e0 con 0x7f471c0712c0 2023-11-28T20:48:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.847+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f46fc06afe0 msgr2=0x7f46fc06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:48:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.847+0000 7f4722761640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f46fc06afe0 0x7f46fc06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.847+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0712c0 msgr2=0x7f471c1f70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.847+0000 7f4722761640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0712c0 0x7f471c1f70f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f470c009870 tx=0x7f470c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.847+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 shutdown_connections 2023-11-28T20:48:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.847+0000 7f4722761640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f46fc06afe0 0x7f46fc06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.848+0000 7f4722761640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f471c1653a0 0x7f471c1ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.848+0000 7f4722761640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f471c0c1d30 0x7f471c15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.848+0000 7f4722761640 1 --2- 172.21.15.45:0/2612502319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f471c0712c0 0x7f471c1f70f0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.848+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 >> 172.21.15.45:0/2612502319 conn(0x7f471c0ced60 msgr2=0x7f471c164750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.848+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 shutdown_connections 2023-11-28T20:48:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:24.848+0000 7f4722761640 1 -- 172.21.15.45:0/2612502319 wait complete. 2023-11-28T20:48:24.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T20:48:24.922 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:24.919+0000 7fa8b3a30040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:48:24.946 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:24.944+0000 7f6d26a32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:48:24.974 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:24.971+0000 7fa8b3a30040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:48:25.003 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:25.001+0000 7f6d26a32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:48:25.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.069+0000 7ff3bc81a640 1 Processor -- start 2023-11-28T20:48:25.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.069+0000 7ff3bc81a640 1 -- start start 2023-11-28T20:48:25.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3bc81a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 0x7ff3b80cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3bc81a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3b80d1e40 0x7ff3b80ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3bc81a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff3b80b2500 0x7ff3b80ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3bc81a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b80d0490 con 0x7ff3b80d1e40 2023-11-28T20:48:25.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3bc81a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b80d1360 con 0x7ff3b80b2500 2023-11-28T20:48:25.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3bc81a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b80d1640 con 0x7ff3b815d580 2023-11-28T20:48:25.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3b6ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3b80d1e40 0x7ff3b80ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3b6ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3b80d1e40 0x7ff3b80ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45382/0 (socket says 172.21.15.45:45382) 2023-11-28T20:48:25.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3b6ffd640 1 -- 172.21.15.45:0/2213009873 learned_addr learned my addr 172.21.15.45:0/2213009873 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:25.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3b7fff640 1 --2- 172.21.15.45:0/2213009873 >> v2:172.21.15.190:3300/0 conn(0x7ff3b80b2500 0x7ff3b80ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.071+0000 7ff3b77fe640 1 --2- 172.21.15.45:0/2213009873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 0x7ff3b80cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.072+0000 7ff3b6ffd640 1 -- 172.21.15.45:0/2213009873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 msgr2=0x7ff3b80cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:25.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.072+0000 7ff3b6ffd640 1 --2- 172.21.15.45:0/2213009873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 0x7ff3b80cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.072+0000 7ff3b6ffd640 1 -- 172.21.15.45:0/2213009873 >> v2:172.21.15.190:3300/0 conn(0x7ff3b80b2500 msgr2=0x7ff3b80ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:25.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.072+0000 7ff3b6ffd640 1 --2- 172.21.15.45:0/2213009873 >> v2:172.21.15.190:3300/0 conn(0x7ff3b80b2500 0x7ff3b80ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.072+0000 7ff3b6ffd640 1 -- 172.21.15.45:0/2213009873 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b80d1920 con 0x7ff3b80d1e40 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.073+0000 7ff3b6ffd640 1 --2- 172.21.15.45:0/2213009873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80d1e40 0x7ff3b80ce040 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff3b80cdc10 tx=0x7ff3ac02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88313ff811fabcee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.073+0000 7ff3b67fc640 1 -- 172.21.15.45:0/2213009873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac004230 con 0x7ff3b80d1e40 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.073+0000 7ff3b67fc640 1 -- 172.21.15.45:0/2213009873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac004390 con 0x7ff3b80d1e40 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.073+0000 7ff3b67fc640 1 -- 172.21.15.45:0/2213009873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac003810 con 0x7ff3b80d1e40 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.074+0000 7ff3bc81a640 1 -- 172.21.15.45:0/2213009873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80d1e40 msgr2=0x7ff3b80ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.074+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/2213009873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80d1e40 0x7ff3b80ce040 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff3b80cdc10 tx=0x7ff3ac02b860 comp rx=0 tx=0).stop 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.074+0000 7ff3bc81a640 1 -- 172.21.15.45:0/2213009873 shutdown_connections 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.074+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/2213009873 >> v2:172.21.15.190:3300/0 conn(0x7ff3b80b2500 0x7ff3b80ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.074+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/2213009873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80d1e40 0x7ff3b80ce040 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.074+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/2213009873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 0x7ff3b80cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.074+0000 7ff3bc81a640 1 -- 172.21.15.45:0/2213009873 >> 172.21.15.45:0/2213009873 conn(0x7ff3b806a500 msgr2=0x7ff3b806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:25.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.074+0000 7ff3bc81a640 1 -- 172.21.15.45:0/2213009873 shutdown_connections 2023-11-28T20:48:25.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.074+0000 7ff3bc81a640 1 -- 172.21.15.45:0/2213009873 wait complete. 2023-11-28T20:48:25.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.075+0000 7ff3bc81a640 1 Processor -- start 2023-11-28T20:48:25.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.076+0000 7ff3bc81a640 1 -- start start 2023-11-28T20:48:25.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3bc81a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80b2500 0x7ff3b815d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3bc81a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b80d1e40 0x7ff3b814a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3bc81a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 0x7ff3b814aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3bc81a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b8158db0 con 0x7ff3b80b2500 2023-11-28T20:48:25.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3bc81a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b8156d10 con 0x7ff3b80d1e40 2023-11-28T20:48:25.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3bc81a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b814aac0 con 0x7ff3b815d580 2023-11-28T20:48:25.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80b2500 0x7ff3b815d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80b2500 0x7ff3b815d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45384/0 (socket says 172.21.15.45:45384) 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b77fe640 1 -- 172.21.15.45:0/3859663818 learned_addr learned my addr 172.21.15.45:0/3859663818 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b6ffd640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b80d1e40 0x7ff3b814a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b7fff640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 0x7ff3b814aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b77fe640 1 -- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 msgr2=0x7ff3b814aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b77fe640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 0x7ff3b814aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b77fe640 1 -- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b80d1e40 msgr2=0x7ff3b814a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b77fe640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b80d1e40 0x7ff3b814a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b77fe640 1 -- 172.21.15.45:0/3859663818 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac009460 con 0x7ff3b80b2500 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.077+0000 7ff3b7fff640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 0x7ff3b814aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:25.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.078+0000 7ff3b77fe640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80b2500 0x7ff3b815d960 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8007e40 tx=0x7ff3a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:25.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.078+0000 7ff39ffff640 1 -- 172.21.15.45:0/3859663818 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8014620 con 0x7ff3b80b2500 2023-11-28T20:48:25.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.078+0000 7ff39ffff640 1 -- 172.21.15.45:0/3859663818 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a800ee90 con 0x7ff3b80b2500 2023-11-28T20:48:25.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.078+0000 7ff39ffff640 1 -- 172.21.15.45:0/3859663818 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a80149c0 con 0x7ff3b80b2500 2023-11-28T20:48:25.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.078+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b815e060 con 0x7ff3b80b2500 2023-11-28T20:48:25.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.078+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b81586f0 con 0x7ff3b80b2500 2023-11-28T20:48:25.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.079+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b8159020 con 0x7ff3b80b2500 2023-11-28T20:48:25.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.080+0000 7ff39ffff640 1 -- 172.21.15.45:0/3859663818 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 89350+0+0 (secure 0 0 0) 0x7ff3a8014b20 con 0x7ff3b80b2500 2023-11-28T20:48:25.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.081+0000 7ff39ffff640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7ff3a406afb0 0x7ff3a406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.081+0000 7ff39ffff640 1 -- 172.21.15.45:0/3859663818 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff3a808a1d0 con 0x7ff3b80b2500 2023-11-28T20:48:25.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.081+0000 7ff3b6ffd640 1 -- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7ff3a406afb0 msgr2=0x7ff3a406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/838517151 2023-11-28T20:48:25.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.081+0000 7ff3b6ffd640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7ff3a406afb0 0x7ff3a406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:48:25.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.090+0000 7ff39ffff640 1 -- 172.21.15.45:0/3859663818 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3a80550c0 con 0x7ff3b80b2500 2023-11-28T20:48:25.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.282+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7ff3b8071b30 con 0x7ff3b80b2500 2023-11-28T20:48:25.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.282+0000 7ff3b6ffd640 1 -- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7ff3a406afb0 msgr2=0x7ff3a406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/838517151 2023-11-28T20:48:25.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.282+0000 7ff3b6ffd640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7ff3a406afb0 0x7ff3a406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:48:25.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.287+0000 7ff39ffff640 1 -- 172.21.15.45:0/3859663818 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff3a80586e0 con 0x7ff3b80b2500 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7ff3a406afb0 msgr2=0x7ff3a406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7ff3a406afb0 0x7ff3a406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80b2500 msgr2=0x7ff3b815d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80b2500 0x7ff3b815d960 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8007e40 tx=0x7ff3a800e4b0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 shutdown_connections 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7ff3a406afb0 0x7ff3a406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b815d580 0x7ff3b814aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b80d1e40 0x7ff3b814a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 --2- 172.21.15.45:0/3859663818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b80b2500 0x7ff3b815d960 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.289+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 >> 172.21.15.45:0/3859663818 conn(0x7ff3b806a500 msgr2=0x7ff3b80cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:25.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.290+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 shutdown_connections 2023-11-28T20:48:25.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.290+0000 7ff3bc81a640 1 -- 172.21.15.45:0/3859663818 wait complete. 2023-11-28T20:48:25.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:25.309 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-11-28T20:48:25.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-11-28T20:48:25.423 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.421+0000 7fa8b3a30040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:48:25.476 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.474+0000 7fa8b3a30040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:48:25.491 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:25.489+0000 7f6d26a32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:48:25.525 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.523+0000 7fa8b3a30040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:48:25.547 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:25.545+0000 7f6d26a32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:48:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.567+0000 7f6b30851640 1 Processor -- start 2023-11-28T20:48:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.567+0000 7f6b30851640 1 -- start start 2023-11-28T20:48:25.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.568+0000 7f6b30851640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6b2c15d580 0x7f6b2c0cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.569+0000 7f6b30851640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2c0d1e40 0x7f6b2c0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.569+0000 7f6b30851640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c0d0490 con 0x7f6b2c0b2500 2023-11-28T20:48:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.569+0000 7f6b30851640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b2c0d1360 con 0x7f6b2c15d580 2023-11-28T20:48:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.569+0000 7f6b30851640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0d1640 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.569+0000 7f6b2ad9d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6b2c15d580 0x7f6b2c0cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.569+0000 7f6b2ad9d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6b2c15d580 0x7f6b2c0cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:42766/0 (socket says 172.21.15.45:42766) 2023-11-28T20:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.569+0000 7f6b2ad9d640 1 -- 172.21.15.45:0/2781476916 learned_addr learned my addr 172.21.15.45:0/2781476916 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.569+0000 7f6b2a59c640 1 --2- 172.21.15.45:0/2781476916 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2c0d1e40 0x7f6b2c0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.570+0000 7f6b2ad9d640 1 -- 172.21.15.45:0/2781476916 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2c0d1e40 msgr2=0x7f6b2c0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.570+0000 7f6b2ad9d640 1 --2- 172.21.15.45:0/2781476916 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2c0d1e40 0x7f6b2c0ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.570+0000 7f6b2ad9d640 1 -- 172.21.15.45:0/2781476916 >> v1:172.21.15.190:6789/0 conn(0x7f6b2c0b2500 legacy=0x7f6b2c0b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.570+0000 7f6b2ad9d640 1 -- 172.21.15.45:0/2781476916 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c0d1920 con 0x7f6b2c15d580 2023-11-28T20:48:25.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.571+0000 7f6b2ad9d640 1 --2- 172.21.15.45:0/2781476916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2c15d580 0x7f6b2c0cfb00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009870 tx=0x7f6b1c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cc1f1a24eb7c8b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:25.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.572+0000 7f6b29d9b640 1 -- 172.21.15.45:0/2781476916 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c031640 con 0x7f6b2c15d580 2023-11-28T20:48:25.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.572+0000 7f6b29d9b640 1 -- 172.21.15.45:0/2781476916 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c0317a0 con 0x7f6b2c15d580 2023-11-28T20:48:25.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.572+0000 7f6b29d9b640 1 -- 172.21.15.45:0/2781476916 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c031b30 con 0x7f6b2c15d580 2023-11-28T20:48:25.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.572+0000 7f6b30851640 1 -- 172.21.15.45:0/2781476916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2c15d580 msgr2=0x7f6b2c0cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:25.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.572+0000 7f6b30851640 1 --2- 172.21.15.45:0/2781476916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2c15d580 0x7f6b2c0cfb00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009870 tx=0x7f6b1c02b8f0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.573+0000 7f6b30851640 1 -- 172.21.15.45:0/2781476916 shutdown_connections 2023-11-28T20:48:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.573+0000 7f6b30851640 1 --2- 172.21.15.45:0/2781476916 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2c0d1e40 0x7f6b2c0ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.573+0000 7f6b30851640 1 --2- 172.21.15.45:0/2781476916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2c15d580 0x7f6b2c0cfb00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.573+0000 7f6b30851640 1 -- 172.21.15.45:0/2781476916 >> 172.21.15.45:0/2781476916 conn(0x7f6b2c06a500 msgr2=0x7f6b2c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.573+0000 7f6b30851640 1 -- 172.21.15.45:0/2781476916 shutdown_connections 2023-11-28T20:48:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.573+0000 7f6b30851640 1 -- 172.21.15.45:0/2781476916 wait complete. 2023-11-28T20:48:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.575+0000 7f6b30851640 1 Processor -- start 2023-11-28T20:48:25.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.575+0000 7f6b30851640 1 -- start start 2023-11-28T20:48:25.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.576+0000 7f6b30851640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2c0b2500 0x7f6b2c0b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.576+0000 7f6b30851640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2c0d1e40 0x7f6b2c15e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.576+0000 7f6b30851640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2c15d580 0x7f6b2c15ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.576+0000 7f6b30851640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0b4ae0 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.576+0000 7f6b30851640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0b4d10 con 0x7f6b2c15d580 2023-11-28T20:48:25.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.576+0000 7f6b30851640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0b4ff0 con 0x7f6b2c0b2500 2023-11-28T20:48:25.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2c0d1e40 0x7f6b2c15e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2c0d1e40 0x7f6b2c15e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45396/0 (socket says 172.21.15.45:45396) 2023-11-28T20:48:25.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2a59c640 1 -- 172.21.15.45:0/531254801 learned_addr learned my addr 172.21.15.45:0/531254801 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:25.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2b59e640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2c15d580 0x7f6b2c15ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2ad9d640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2c0b2500 0x7f6b2c0b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:25.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2a59c640 1 -- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2c0b2500 msgr2=0x7f6b2c0b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:25.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2a59c640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2c0b2500 0x7f6b2c0b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2a59c640 1 -- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2c15d580 msgr2=0x7f6b2c15ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:25.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2a59c640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2c15d580 0x7f6b2c15ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2a59c640 1 -- 172.21.15.45:0/531254801 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c0094d0 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.577+0000 7f6b2a59c640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2c0d1e40 0x7f6b2c15e9d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000e870 tx=0x7f6b2000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.578+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b200095c0 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.578+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2000c940 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.578+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2000ccb0 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.578+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2c0b52d0 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.578+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c0b5890 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.579+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b2c0b5a20 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.580+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f6b20013470 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.581+0000 7f6b137fe640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 0x7f6b1806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.582+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6b20089e70 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.582+0000 7f6b2ad9d640 1 -- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 msgr2=0x7f6b1806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/838517151 2023-11-28T20:48:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.582+0000 7f6b2ad9d640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 0x7f6b1806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:48:25.587 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.585+0000 7fa8b3a30040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:48:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.586+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b20054d80 con 0x7f6b2c0d1e40 2023-11-28T20:48:25.604 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:25.601+0000 7f6d26a32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:48:25.635 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.633+0000 7fa8b3a30040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:48:25.668 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:25.666+0000 7f6d26a32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:48:25.720 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:25.718+0000 7f6d26a32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:48:25.745 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.743+0000 7fa8b3a30040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:48:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.783+0000 7f6b2ad9d640 1 -- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 msgr2=0x7f6b1806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/838517151 2023-11-28T20:48:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.783+0000 7f6b2ad9d640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 0x7f6b1806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:48:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:25.800+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 --> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b2c071b30 con 0x7f6b1806afb0 2023-11-28T20:48:25.806 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.804+0000 7fa8b3a30040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:48:25.838 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:25.836+0000 7f6d26a32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:48:25.855 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.853+0000 7fa8b3a30040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:48:25.904 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:25.902+0000 7f6d26a32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:48:25.913 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.911+0000 7fa8b3a30040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:48:25.956 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:25.954+0000 7f6d26a32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:48:25.962 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:25.960+0000 7fa8b3a30040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:48:26.016 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:26.013+0000 7f6d26a32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:48:26.068 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:26.066+0000 7f6d26a32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:48:26.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:26.184+0000 7f6b2ad9d640 1 -- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 msgr2=0x7f6b1806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/838517151 2023-11-28T20:48:26.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:26.184+0000 7f6b2ad9d640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 0x7f6b1806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T20:48:26.338 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:26.336+0000 7fa8b3a30040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:48:26.469 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:26.467+0000 7f6d26a32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:48:26.498 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:26.496+0000 7fa8b3a30040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:48:26.607 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:26.605+0000 7fa8b3a30040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:48:26.637 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:26.635+0000 7f6d26a32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:48:26.657 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:26.655+0000 7fa8b3a30040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:48:26.750 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:26.748+0000 7f6d26a32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:48:26.764 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:26.762+0000 7fa8b3a30040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:48:26.802 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:26.800+0000 7f6d26a32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:48:26.871 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:26.869+0000 7fa8b3a30040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:48:26.916 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:26.914+0000 7f6d26a32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:48:26.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:26.985+0000 7f6b2ad9d640 1 -- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 msgr2=0x7f6b1806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/838517151 2023-11-28T20:48:26.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:26.985+0000 7f6b2ad9d640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 0x7f6b1806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T20:48:27.027 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:27.024+0000 7fa8b3a30040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:48:27.029 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:27.027+0000 7f6d26a32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:48:27.075 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:27.073+0000 7fa8b3a30040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:48:27.194 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:27.191+0000 7f6d26a32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:48:27.194 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:27.191+0000 7fa8b3a30040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:48:27.245 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:27.243+0000 7f6d26a32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:48:27.372 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:27.369+0000 7f6d26a32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:48:28.018 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.015+0000 7fa8b3a30040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:48:28.225 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:28.223+0000 7f6d26a32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:48:28.312 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.310+0000 7fa8b3a30040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:48:28.369 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.367+0000 7fa8b3a30040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:48:28.486 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.483+0000 7fa8b3a30040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:48:28.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:28.507+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 20) v1 ==== 88959+0+0 (secure 0 0 0) 0x7f6b20055150 con 0x7f6b2c0d1e40 2023-11-28T20:48:28.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:28.507+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 msgr2=0x7f6b1806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:48:28.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:28.507+0000 7f6b137fe640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/838517151,v1:172.21.15.190:6833/838517151] conn(0x7f6b1806afb0 0x7f6b1806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:28.547 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:28.545+0000 7f6d26a32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:48:28.613 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:28.611+0000 7f6d26a32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:48:28.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.668+0000 7fa82fdd2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.668+0000 7fa82fdd2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.668+0000 7fa82fdd2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.668+0000 7fa82fdd2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.668+0000 7fa82fdd2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.669+0000 7fa8325d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.669+0000 7fa8325d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.669+0000 7fa8325d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.669+0000 7fa8325d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.672 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:48:28.669+0000 7fa8325d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:48:28.741 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:48:28.739+0000 7f6d26a32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:48:29.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.518+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 21) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f6b20056860 con 0x7f6b2c0d1e40 2023-11-28T20:48:29.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.519+0000 7f6b137fe640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6b18035850 0x7f6b18075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:29.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.519+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 --> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b2c071b30 con 0x7f6b18035850 2023-11-28T20:48:29.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.521+0000 7f6b2ad9d640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6b18035850 0x7f6b18075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:29.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.533+0000 7f6b2ad9d640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6b18035850 0x7f6b18075380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c036000 tx=0x7f6b1c03c040 comp rx=0 tx=0).ready entity=mgr.5246 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:29.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.694+0000 7f6b137fe640 1 -- 172.21.15.45:0/531254801 <== mgr.5246 v2:172.21.15.190:6832/979211100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6b2c071b30 con 0x7f6b18035850 2023-11-28T20:48:29.698 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1 2023-11-28T20:48:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.698+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6b18035850 msgr2=0x7f6b18075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.698+0000 7f6b30851640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6b18035850 0x7f6b18075380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c036000 tx=0x7f6b1c03c040 comp rx=0 tx=0).stop 2023-11-28T20:48:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.698+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2c0d1e40 msgr2=0x7f6b2c15e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.698+0000 7f6b30851640 1 --2- 172.21.15.45:0/531254801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2c0d1e40 0x7f6b2c15e9d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000e870 tx=0x7f6b2000ed30 comp rx=0 tx=0).stop 2023-11-28T20:48:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.698+0000 7f6b2b59e640 1 -- 172.21.15.45:0/531254801 reap_dead start 2023-11-28T20:48:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.698+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 shutdown_connections 2023-11-28T20:48:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.698+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 >> 172.21.15.45:0/531254801 conn(0x7f6b2c06a500 msgr2=0x7f6b2c0d39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.698+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 shutdown_connections 2023-11-28T20:48:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.698+0000 7f6b30851640 1 -- 172.21.15.45:0/531254801 wait complete. 2023-11-28T20:48:29.721 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-11-28T20:48:29.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2023-11-28T20:48:29.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.951+0000 7f22ae09e640 1 Processor -- start 2023-11-28T20:48:29.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.951+0000 7f22ae09e640 1 -- start start 2023-11-28T20:48:29.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.953+0000 7f22ae09e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 0x7f22a80b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:29.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.953+0000 7f22ae09e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22a8071ac0 0x7f22a80b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:29.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.953+0000 7f22ae09e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80b27a0 con 0x7f22a8072010 2023-11-28T20:48:29.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.953+0000 7f22ae09e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a8071450 con 0x7f22a8071ac0 2023-11-28T20:48:29.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.953+0000 7f22ae09e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a809ceb0 con 0x7f22a8159fc0 2023-11-28T20:48:29.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.953+0000 7f22ac89b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22a8071ac0 0x7f22a80b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:29.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.954+0000 7f22ac89b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22a8071ac0 0x7f22a80b2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45412/0 (socket says 172.21.15.45:45412) 2023-11-28T20:48:29.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.954+0000 7f22ac89b640 1 -- 172.21.15.45:0/111960198 learned_addr learned my addr 172.21.15.45:0/111960198 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:29.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.954+0000 7f22ad09c640 1 --2- 172.21.15.45:0/111960198 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 0x7f22a80b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:29.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.954+0000 7f22ac89b640 1 -- 172.21.15.45:0/111960198 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 msgr2=0x7f22a80b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:29.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.954+0000 7f22ac89b640 1 --2- 172.21.15.45:0/111960198 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 0x7f22a80b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:29.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.954+0000 7f22ac89b640 1 -- 172.21.15.45:0/111960198 >> v1:172.21.15.190:6789/0 conn(0x7f22a8072010 legacy=0x7f22a815dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:29.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.954+0000 7f22ac89b640 1 -- 172.21.15.45:0/111960198 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80b2cb0 con 0x7f22a8071ac0 2023-11-28T20:48:29.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.955+0000 7f22ac89b640 1 --2- 172.21.15.45:0/111960198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8071ac0 0x7f22a80b2260 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2298009870 tx=0x7f229802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467d0639da548805 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:29.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.956+0000 7f229ffff640 1 -- 172.21.15.45:0/111960198 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22980317c0 con 0x7f22a8071ac0 2023-11-28T20:48:29.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.956+0000 7f229ffff640 1 -- 172.21.15.45:0/111960198 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2298031920 con 0x7f22a8071ac0 2023-11-28T20:48:29.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.956+0000 7f229ffff640 1 -- 172.21.15.45:0/111960198 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298031cb0 con 0x7f22a8071ac0 2023-11-28T20:48:29.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.956+0000 7f22ae09e640 1 -- 172.21.15.45:0/111960198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8071ac0 msgr2=0x7f22a80b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:29.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.956+0000 7f22ae09e640 1 --2- 172.21.15.45:0/111960198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8071ac0 0x7f22a80b2260 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2298009870 tx=0x7f229802b860 comp rx=0 tx=0).stop 2023-11-28T20:48:29.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.957+0000 7f22ae09e640 1 -- 172.21.15.45:0/111960198 shutdown_connections 2023-11-28T20:48:29.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.957+0000 7f22ae09e640 1 --2- 172.21.15.45:0/111960198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8071ac0 0x7f22a80b2260 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:29.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.957+0000 7f22ae09e640 1 --2- 172.21.15.45:0/111960198 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 0x7f22a80b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:29.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.957+0000 7f22ae09e640 1 -- 172.21.15.45:0/111960198 >> 172.21.15.45:0/111960198 conn(0x7f22a80b1050 msgr2=0x7f22a80cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:29.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.957+0000 7f22ae09e640 1 -- 172.21.15.45:0/111960198 shutdown_connections 2023-11-28T20:48:29.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.957+0000 7f22ae09e640 1 -- 172.21.15.45:0/111960198 wait complete. 2023-11-28T20:48:29.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.959+0000 7f22ae09e640 1 Processor -- start 2023-11-28T20:48:29.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.959+0000 7f22ae09e640 1 -- start start 2023-11-28T20:48:29.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.960+0000 7f22ae09e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22a8071ac0 0x7f22a8151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:29.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.960+0000 7f22ae09e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8072010 0x7f22a80cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:29.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.960+0000 7f22ae09e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 0x7f22a806c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:29.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.960+0000 7f22ae09e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a80cd940 con 0x7f22a8072010 2023-11-28T20:48:29.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.960+0000 7f22ae09e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a81531f0 con 0x7f22a8071ac0 2023-11-28T20:48:29.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.960+0000 7f22ae09e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a81525e0 con 0x7f22a8159fc0 2023-11-28T20:48:29.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.960+0000 7f22ac89b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8072010 0x7f22a80cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:29.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.960+0000 7f22ac89b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8072010 0x7f22a80cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45424/0 (socket says 172.21.15.45:45424) 2023-11-28T20:48:29.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.960+0000 7f22ac89b640 1 -- 172.21.15.45:0/1990194423 learned_addr learned my addr 172.21.15.45:0/1990194423 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ad09c640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22a8071ac0 0x7f22a8151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ad89d640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 0x7f22a806c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ac89b640 1 -- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 msgr2=0x7f22a806c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ac89b640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 0x7f22a806c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ac89b640 1 -- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22a8071ac0 msgr2=0x7f22a8151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ac89b640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22a8071ac0 0x7f22a8151bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:29.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ac89b640 1 -- 172.21.15.45:0/1990194423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22980094d0 con 0x7f22a8072010 2023-11-28T20:48:29.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ad89d640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 0x7f22a806c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:29.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ac89b640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8072010 0x7f22a80cd400 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2298000c00 tx=0x7f22980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:29.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f229dffb640 1 -- 172.21.15.45:0/1990194423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298032860 con 0x7f22a8072010 2023-11-28T20:48:29.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f229dffb640 1 -- 172.21.15.45:0/1990194423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22980329e0 con 0x7f22a8072010 2023-11-28T20:48:29.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f229dffb640 1 -- 172.21.15.45:0/1990194423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298032d90 con 0x7f22a8072010 2023-11-28T20:48:29.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.961+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a81528c0 con 0x7f22a8072010 2023-11-28T20:48:29.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.962+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a8152e80 con 0x7f22a8072010 2023-11-28T20:48:29.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.963+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a8153380 con 0x7f22a8072010 2023-11-28T20:48:29.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.964+0000 7f229dffb640 1 -- 172.21.15.45:0/1990194423 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 21) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f229804e450 con 0x7f22a8072010 2023-11-28T20:48:29.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.965+0000 7f229dffb640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f228c06ace0 0x7f228c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:29.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.965+0000 7f229dffb640 1 -- 172.21.15.45:0/1990194423 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f22980ab770 con 0x7f22a8072010 2023-11-28T20:48:29.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.965+0000 7f22ad09c640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f228c06ace0 0x7f228c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:29.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.966+0000 7f22ad09c640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f228c06ace0 0x7f228c06d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f22a4006fd0 tx=0x7f22a4008040 comp rx=0 tx=0).ready entity=mgr.5246 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:29.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:29.969+0000 7f229dffb640 1 -- 172.21.15.45:0/1990194423 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22980768c0 con 0x7f22a8072010 2023-11-28T20:48:30.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.185+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 --> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7f22a8071070 con 0x7f228c06ace0 2023-11-28T20:48:30.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.324+0000 7f229dffb640 1 -- 172.21.15.45:0/1990194423 <== mgr.5246 v2:172.21.15.190:6832/979211100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f22a4009480 con 0x7f228c06ace0 2023-11-28T20:48:30.327 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1 2023-11-28T20:48:30.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f228c06ace0 msgr2=0x7f228c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:30.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f228c06ace0 0x7f228c06d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f22a4006fd0 tx=0x7f22a4008040 comp rx=0 tx=0).stop 2023-11-28T20:48:30.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8072010 msgr2=0x7f22a80cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:30.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8072010 0x7f22a80cd400 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2298000c00 tx=0x7f22980046c0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 shutdown_connections 2023-11-28T20:48:30.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f228c06ace0 0x7f228c06d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f22a8159fc0 0x7f22a806c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22a8072010 0x7f22a80cd400 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 --2- 172.21.15.45:0/1990194423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22a8071ac0 0x7f22a8151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 >> 172.21.15.45:0/1990194423 conn(0x7f22a80b1050 msgr2=0x7f22a80abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:30.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 shutdown_connections 2023-11-28T20:48:30.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.328+0000 7f22ae09e640 1 -- 172.21.15.45:0/1990194423 wait complete. 2023-11-28T20:48:30.355 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-11-28T20:48:30.355 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-11-28-20_49_00 in ~30s... 2023-11-28T20:48:30.356 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-11-28-20_50_00 in ~90s... 2023-11-28T20:48:30.356 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2023-11-28T20:48:30.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-11-28T20:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.586+0000 7fc1e6cbb640 1 Processor -- start 2023-11-28T20:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.586+0000 7fc1e6cbb640 1 -- start start 2023-11-28T20:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.588+0000 7fc1e6cbb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc1e006d620 0x7fc1e00cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.588+0000 7fc1e6cbb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e00cf500 con 0x7fc1e015d7a0 2023-11-28T20:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.588+0000 7fc1e6cbb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e00cf7e0 con 0x7fc1e0066900 2023-11-28T20:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.588+0000 7fc1e6cbb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e00cfda0 con 0x7fc1e006d620 2023-11-28T20:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.588+0000 7fc1e4a57640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc1e015d7a0 0x7fc1e00661f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40620/0 (socket says 172.21.15.45:40620) 2023-11-28T20:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.588+0000 7fc1e4a57640 1 -- 172.21.15.45:0/1051258710 learned_addr learned my addr 172.21.15.45:0/1051258710 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.588+0000 7fc1e5258640 1 --2- 172.21.15.45:0/1051258710 >> v2:172.21.15.190:3300/0 conn(0x7fc1e006d620 0x7fc1e00cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223845035 0 0) 0x7fc1e00cf500 con 0x7fc1e015d7a0 2023-11-28T20:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d0003520 con 0x7fc1e015d7a0 2023-11-28T20:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 925032225 0 0) 0x7fc1d0003520 con 0x7fc1e015d7a0 2023-11-28T20:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 >> v2:172.21.15.190:3300/0 conn(0x7fc1e006d620 msgr2=0x7fc1e00cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 --2- 172.21.15.45:0/1051258710 >> v2:172.21.15.190:3300/0 conn(0x7fc1e006d620 0x7fc1e00cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 >> v1:172.21.15.190:6789/0 conn(0x7fc1e0066900 legacy=0x7fc1e006abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e00cfac0 con 0x7fc1e015d7a0 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc1d4001670 con 0x7fc1e015d7a0 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 >> v1:172.21.15.45:6789/0 conn(0x7fc1e015d7a0 legacy=0x7fc1e00661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1d0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d00047c0 0x7fc1d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1d0005180 0x7fc1d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e00cfda0 con 0x7fc1d00039c0 2023-11-28T20:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d000c080 con 0x7fc1d00047c0 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d000c240 con 0x7fc1d0005180 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.589+0000 7fc1e4a57640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1d0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.590+0000 7fc1dffff640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d00047c0 0x7fc1d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.590+0000 7fc1e5258640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1d0005180 0x7fc1d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.590+0000 7fc1e4a57640 1 -- 172.21.15.45:0/1051258710 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1d0005180 msgr2=0x7fc1d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.590+0000 7fc1e4a57640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1d0005180 0x7fc1d000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.590+0000 7fc1e4a57640 1 -- 172.21.15.45:0/1051258710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d00047c0 msgr2=0x7fc1d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.590+0000 7fc1e4a57640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d00047c0 0x7fc1d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.590+0000 7fc1e4a57640 1 -- 172.21.15.45:0/1051258710 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d0003520 con 0x7fc1d00039c0 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.590+0000 7fc1dffff640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d00047c0 0x7fc1d0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:48:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.590+0000 7fc1e5258640 1 -- 172.21.15.45:0/1051258710 reap_dead start 2023-11-28T20:48:30.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.591+0000 7fc1e4a57640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1d0003de0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400c0e0 tx=0x7fc1d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8e387ee605f57 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:30.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.591+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4005ad0 con 0x7fc1d00039c0 2023-11-28T20:48:30.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.591+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d4005c30 con 0x7fc1d00039c0 2023-11-28T20:48:30.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.591+0000 7fc1df7fe640 1 -- 172.21.15.45:0/1051258710 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4007070 con 0x7fc1d00039c0 2023-11-28T20:48:30.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.592+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/1051258710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 msgr2=0x7fc1d0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:30.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.592+0000 7fc1e6cbb640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1d0003de0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400c0e0 tx=0x7fc1d00057f0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.592+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/1051258710 shutdown_connections 2023-11-28T20:48:30.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.592+0000 7fc1e6cbb640 1 --2- 172.21.15.45:0/1051258710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1d0003de0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.592+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/1051258710 >> 172.21.15.45:0/1051258710 conn(0x7fc1e00cc9c0 msgr2=0x7fc1e00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:30.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.592+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/1051258710 shutdown_connections 2023-11-28T20:48:30.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.592+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/1051258710 wait complete. 2023-11-28T20:48:30.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.594+0000 7fc1e6cbb640 1 Processor -- start 2023-11-28T20:48:30.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.594+0000 7fc1e6cbb640 1 -- start start 2023-11-28T20:48:30.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.595+0000 7fc1e6cbb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1e009e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:30.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.595+0000 7fc1e6cbb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1e0156750 0x7fc1e0160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:30.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.595+0000 7fc1e6cbb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1e009f990 0x7fc1e0158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:30.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.595+0000 7fc1e6cbb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e009f570 con 0x7fc1d00039c0 2023-11-28T20:48:30.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.595+0000 7fc1e6cbb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e009e8f0 con 0x7fc1e0156750 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.595+0000 7fc1e6cbb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e009ebd0 con 0x7fc1e009f990 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e4a57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1e009e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e4a57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1e009e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45448/0 (socket says 172.21.15.45:45448) 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e4a57640 1 -- 172.21.15.45:0/3750049906 learned_addr learned my addr 172.21.15.45:0/3750049906 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e5258640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1e009f990 0x7fc1e0158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1dffff640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1e0156750 0x7fc1e0160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e4a57640 1 -- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1e009f990 msgr2=0x7fc1e0158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e4a57640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1e009f990 0x7fc1e0158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e4a57640 1 -- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1e0156750 msgr2=0x7fc1e0160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e4a57640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1e0156750 0x7fc1e0160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e4a57640 1 -- 172.21.15.45:0/3750049906 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e009eeb0 con 0x7fc1d00039c0 2023-11-28T20:48:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.596+0000 7fc1e4a57640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1e009e310 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc1d402c2e0 tx=0x7fc1d4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.597+0000 7fc1dd7fa640 1 -- 172.21.15.45:0/3750049906 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d403c070 con 0x7fc1d00039c0 2023-11-28T20:48:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.597+0000 7fc1dd7fa640 1 -- 172.21.15.45:0/3750049906 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d4005a40 con 0x7fc1d00039c0 2023-11-28T20:48:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.597+0000 7fc1dd7fa640 1 -- 172.21.15.45:0/3750049906 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d402c740 con 0x7fc1d00039c0 2023-11-28T20:48:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.597+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e009f190 con 0x7fc1d00039c0 2023-11-28T20:48:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.597+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e0156bd0 con 0x7fc1d00039c0 2023-11-28T20:48:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.598+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e00733d0 con 0x7fc1d00039c0 2023-11-28T20:48:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.599+0000 7fc1dd7fa640 1 -- 172.21.15.45:0/3750049906 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 22) v1 ==== 89351+0+0 (secure 0 0 0) 0x7fc1d4032070 con 0x7fc1d00039c0 2023-11-28T20:48:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.600+0000 7fc1dd7fa640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fc1c406afb0 0x7fc1c406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:48:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.600+0000 7fc1dd7fa640 1 -- 172.21.15.45:0/3750049906 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fc1d40aa5f0 con 0x7fc1d00039c0 2023-11-28T20:48:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.601+0000 7fc1dffff640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fc1c406afb0 0x7fc1c406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:48:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.602+0000 7fc1dffff640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fc1c406afb0 0x7fc1c406d470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc007fd0 tx=0x7fc1cc009040 comp rx=0 tx=0).ready entity=mgr.5246 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:48:30.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.605+0000 7fc1dd7fa640 1 -- 172.21.15.45:0/3750049906 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1d40754c0 con 0x7fc1d00039c0 2023-11-28T20:48:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.821+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 --> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1e00ca370 con 0x7fc1c406afb0 2023-11-28T20:48:30.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.839+0000 7fc1dd7fa640 1 -- 172.21.15.45:0/3750049906 <== mgr.5246 v2:172.21.15.190:6832/979211100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7fc1e00ca370 con 0x7fc1c406afb0 2023-11-28T20:48:30.841 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:48:30.841 INFO:teuthology.orchestra.run.smithi045.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-11-28T20:48:30.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fc1c406afb0 msgr2=0x7fc1c406d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:30.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fc1c406afb0 0x7fc1c406d470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc007fd0 tx=0x7fc1cc009040 comp rx=0 tx=0).stop 2023-11-28T20:48:30.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 msgr2=0x7fc1e009e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:48:30.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1e009e310 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc1d402c2e0 tx=0x7fc1d4001930 comp rx=0 tx=0).stop 2023-11-28T20:48:30.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 shutdown_connections 2023-11-28T20:48:30.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fc1c406afb0 0x7fc1c406d470 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1e009f990 0x7fc1e0158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1e0156750 0x7fc1e0160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 --2- 172.21.15.45:0/3750049906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00039c0 0x7fc1e009e310 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:48:30.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 >> 172.21.15.45:0/3750049906 conn(0x7fc1e00cc9c0 msgr2=0x7fc1e00b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:48:30.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 shutdown_connections 2023-11-28T20:48:30.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:48:30.841+0000 7fc1e6cbb640 1 -- 172.21.15.45:0/3750049906 wait complete. 2023-11-28T20:48:30.863 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-11-28T20:48:30.863 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2023-11-28T20:48:30.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:30.864 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:30.922 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:30.922 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:30.922 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:31.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:31.923 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:31.992 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:31.992 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:31.992 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:32.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:32.997 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:33.065 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:33.065 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:33.065 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:34.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:34.066 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:34.135 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:34.135 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:34.135 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:35.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:35.136 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:35.203 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:35.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:35.204 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:36.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:36.205 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:36.274 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:36.274 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:36.275 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:37.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:37.276 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:37.345 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:37.345 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:37.345 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:38.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:38.347 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:38.410 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:38.410 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:38.410 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:39.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:39.411 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:39.477 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:39.477 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:39.477 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:40.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:40.478 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:40.545 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:40.546 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:40.546 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:41.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:41.547 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:41.614 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:41.614 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:41.614 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:42.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:42.615 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:42.683 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:42.683 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:42.683 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:43.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:43.684 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:43.752 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:43.752 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:43.752 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:44.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:44.753 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:44.820 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:44.820 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:44.820 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:45.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:45.821 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:45.888 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:45.888 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:45.888 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:46.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:46.890 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:46.957 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:46.957 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:46.957 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:47.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:47.958 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:48.025 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:48.025 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:48.025 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:48.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:48:48.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:48:49.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:49.027 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:49.087 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:49.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:49.087 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:50.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:50.088 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:50.153 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:50.154 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:50.155 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:51.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:51.156 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:51.225 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:51.225 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:51.225 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:52.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:52.226 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:52.293 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:52.293 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:52.293 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:48:53.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:48:59.436 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:48:59.507 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:48:59.510 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:48:59.510 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:00.355 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:49:00.353+0000 7fa821db6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:49:00.356 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:49:00.353+0000 7fa821db6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:49:00.356 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:49:00.353+0000 7fa821db6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:49:00.356 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:49:00.353+0000 7fa821db6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:49:00.356 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:49:00.353+0000 7fa821db6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:49:00.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:00.511 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:00.572 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:00.573 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:00.574 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:00.574 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:00.574 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-11-28-20_49_00_UTC'] 2023-11-28T20:49:00.574 DEBUG:tasks.cephfs.test_snap_schedules:exec=1701204510.3558583, now = 1701204540.5744188, timo = 30 2023-11-28T20:49:01.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:01.575 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:01.641 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:01.643 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:01.643 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:01.643 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:02.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:02.644 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:02.710 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:02.712 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:02.712 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:02.712 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:03.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:03.714 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:03.778 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:03.779 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:03.779 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:03.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:04.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:04.780 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:04.846 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:04.848 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:04.848 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:04.848 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:05.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:05.849 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:05.914 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:05.915 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:05.915 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:05.915 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:06.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:06.916 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:06.982 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:06.983 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:06.983 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:06.983 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:07.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:07.985 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:08.050 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:08.051 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:08.051 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:08.052 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:09.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:09.053 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:09.120 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:09.122 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:09.122 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:09.122 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:10.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:10.123 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:10.189 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:10.190 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:10.191 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:10.191 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:11.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:11.191 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:11.259 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:11.261 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:11.261 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:11.261 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:12.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:12.262 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:12.328 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:12.330 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:12.330 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:12.330 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:13.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:13.331 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:13.395 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:13.396 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:13.397 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:13.397 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:14.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:14.398 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:14.466 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:14.467 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:14.467 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:14.468 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:15.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:15.469 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:15.533 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:15.534 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:15.535 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:15.535 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:16.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:16.536 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:16.603 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:16.605 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:16.605 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:16.605 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:17.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:17.606 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:17.672 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:17.673 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:17.673 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:17.673 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:18.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:18.674 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:18.740 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:18.741 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:18.741 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:18.741 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:19.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:49:19.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:49:19.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:19.742 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:19.811 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:19.811 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:19.811 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:20.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:20.812 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:20.879 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:20.881 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:20.881 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:20.881 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:21.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:21.882 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:21.949 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:21.950 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:21.950 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:21.950 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:22.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:22.951 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:23.018 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:23.019 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:23.019 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:23.019 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:24.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:24.020 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:24.085 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:24.087 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:24.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:24.087 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:25.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:25.088 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:25.153 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:25.155 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:25.155 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:25.155 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:26.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:26.156 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:26.223 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:26.224 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:26.224 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:26.224 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:27.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:27.226 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:27.289 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:27.291 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:27.291 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:27.291 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:28.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:28.293 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:28.360 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:28.361 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:28.361 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:28.361 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:29.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:29.363 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:29.427 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:29.429 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:29.429 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:29.429 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:30.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:30.431 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:30.493 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:30.494 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:30.494 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:30.494 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:31.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:31.495 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:31.562 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:31.563 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:31.564 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:31.564 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:32.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:32.565 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:32.628 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:32.629 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:32.629 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:32.629 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:33.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:33.630 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:33.695 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:33.696 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:33.696 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:33.696 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:34.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:34.697 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:34.762 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:34.764 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:34.764 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:34.764 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:35.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:35.765 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:35.831 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:35.832 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:35.833 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:35.833 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:36.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:36.834 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:36.899 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:36.900 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:36.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:36.901 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:37.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:37.902 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:37.969 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:37.970 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:37.971 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:37.971 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:38.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:38.972 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:39.038 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:39.039 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:39.040 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:39.040 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:40.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:40.040 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:40.105 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:40.107 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:40.107 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:40.107 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:41.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:41.108 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:41.172 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:41.173 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:41.173 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:41.173 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:42.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:42.174 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:42.239 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:42.239 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:42.239 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:42.239 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:43.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:43.242 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:43.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:43.309 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:43.309 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:43.310 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:44.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:44.310 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:44.376 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:44.377 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:44.378 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:44.378 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:45.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:45.379 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:45.445 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:45.447 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:45.447 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:45.447 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:46.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:46.448 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:46.517 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:46.518 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:46.518 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:46.518 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:47.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:47.520 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:47.587 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:47.589 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:47.589 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:47.589 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:48.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:48.590 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:48.654 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:48.655 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:48.655 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:48.655 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:49.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:49:49.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:49:49.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:49.657 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:49.723 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:49.724 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:49.724 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:49.724 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:50.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:50.725 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:50.795 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:50.795 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:50.795 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:50.796 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:51.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:51.797 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:51.863 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:51.865 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:51.865 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:51.865 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:52.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:52.866 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:52.932 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:52.933 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:52.933 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:52.933 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:53.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:53.934 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:54.001 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:54.003 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:54.003 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:54.003 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:55.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:55.004 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:55.070 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:55.071 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:55.071 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:55.071 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:56.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:56.072 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:56.138 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:56.139 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:56.139 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:56.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:57.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:57.141 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:57.206 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:57.207 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:57.207 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:57.207 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:58.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:58.208 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:58.276 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:58.277 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:58.277 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:58.277 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:49:59.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:49:59.278 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:49:59.345 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:49:59.346 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:49:59.346 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:49:59.346 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:50:00.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:00.347 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:50:00.412 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:50:00.414 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:50:00.414 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:50:00.414 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:50:01.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:01.415 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:50:01.479 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:50:01.479 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_50_00_UTC 2023-11-28T20:50:01.480 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_50_00_UTC', 'scheduled-2023-11-28-20_49_00_UTC'} 2023-11-28T20:50:01.480 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-11-28-20_50_00_UTC'} 2023-11-28T20:50:01.480 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:50:01.481 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-11-28-20_50_00_UTC'] 2023-11-28T20:50:01.481 DEBUG:tasks.cephfs.test_snap_schedules:exec=1701204510.3558583, now = 1701204601.4811919, timo = 90 2023-11-28T20:50:02.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-11-28T20:50:02.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.723+0000 7fe798ead640 1 Processor -- start 2023-11-28T20:50:02.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.723+0000 7fe798ead640 1 -- start start 2023-11-28T20:50:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.725+0000 7fe798ead640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7940b3490 0x7fe79415e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.725+0000 7fe798ead640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79406dbe0 con 0x7fe794071f20 2023-11-28T20:50:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.725+0000 7fe798ead640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940b3870 con 0x7fe7940b2410 2023-11-28T20:50:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.725+0000 7fe798ead640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79406e110 con 0x7fe7940b3490 2023-11-28T20:50:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.725+0000 7fe7937fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7940b3490 0x7fe79415e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.726+0000 7fe7937fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7940b3490 0x7fe79415e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56750/0 (socket says 172.21.15.45:56750) 2023-11-28T20:50:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.726+0000 7fe7937fe640 1 -- 172.21.15.45:0/2835968584 learned_addr learned my addr 172.21.15.45:0/2835968584 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.726+0000 7fe7937fe640 1 -- 172.21.15.45:0/2835968584 >> v1:172.21.15.190:6789/0 conn(0x7fe7940b2410 legacy=0x7fe794160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.726+0000 7fe7937fe640 1 -- 172.21.15.45:0/2835968584 >> v1:172.21.15.45:6789/0 conn(0x7fe794071f20 legacy=0x7fe7940b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:02.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.726+0000 7fe7937fe640 1 -- 172.21.15.45:0/2835968584 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79406de30 con 0x7fe7940b3490 2023-11-28T20:50:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.727+0000 7fe7937fe640 1 --2- 172.21.15.45:0/2835968584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 0x7fe79415e250 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe784009870 tx=0x7fe78402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2402e71be7c77c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:02.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.728+0000 7fe7927fc640 1 -- 172.21.15.45:0/2835968584 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78402f850 con 0x7fe7940b3490 2023-11-28T20:50:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.728+0000 7fe7927fc640 1 -- 172.21.15.45:0/2835968584 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78402f9b0 con 0x7fe7940b3490 2023-11-28T20:50:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.728+0000 7fe7927fc640 1 -- 172.21.15.45:0/2835968584 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78402fd40 con 0x7fe7940b3490 2023-11-28T20:50:02.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.728+0000 7fe798ead640 1 -- 172.21.15.45:0/2835968584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 msgr2=0x7fe79415e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.728+0000 7fe798ead640 1 --2- 172.21.15.45:0/2835968584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 0x7fe79415e250 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe784009870 tx=0x7fe78402ba30 comp rx=0 tx=0).stop 2023-11-28T20:50:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.728+0000 7fe798ead640 1 -- 172.21.15.45:0/2835968584 shutdown_connections 2023-11-28T20:50:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.728+0000 7fe798ead640 1 --2- 172.21.15.45:0/2835968584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 0x7fe79415e250 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.728+0000 7fe798ead640 1 -- 172.21.15.45:0/2835968584 >> 172.21.15.45:0/2835968584 conn(0x7fe7940cecd0 msgr2=0x7fe7940cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.729+0000 7fe798ead640 1 -- 172.21.15.45:0/2835968584 shutdown_connections 2023-11-28T20:50:02.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.729+0000 7fe798ead640 1 -- 172.21.15.45:0/2835968584 wait complete. 2023-11-28T20:50:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.730+0000 7fe798ead640 1 Processor -- start 2023-11-28T20:50:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.730+0000 7fe798ead640 1 -- start start 2023-11-28T20:50:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.731+0000 7fe798ead640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe794071f20 0x7fe7941e8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.731+0000 7fe798ead640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7940b2410 0x7fe79409ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.731+0000 7fe798ead640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 0x7fe79415c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.731+0000 7fe798ead640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79409b3a0 con 0x7fe7940b3490 2023-11-28T20:50:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.731+0000 7fe798ead640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79409b590 con 0x7fe794071f20 2023-11-28T20:50:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.731+0000 7fe798ead640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79409b870 con 0x7fe7940b2410 2023-11-28T20:50:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.732+0000 7fe793fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 0x7fe79415c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.732+0000 7fe793fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 0x7fe79415c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56762/0 (socket says 172.21.15.45:56762) 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.732+0000 7fe793fff640 1 -- 172.21.15.45:0/2966740485 learned_addr learned my addr 172.21.15.45:0/2966740485 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.732+0000 7fe793fff640 1 -- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7940b2410 msgr2=0x7fe79409ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.732+0000 7fe793fff640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7940b2410 0x7fe79409ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.732+0000 7fe793fff640 1 -- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe794071f20 msgr2=0x7fe7941e8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.732+0000 7fe7937fe640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe794071f20 0x7fe7941e8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.732+0000 7fe793fff640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe794071f20 0x7fe7941e8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.732+0000 7fe793fff640 1 -- 172.21.15.45:0/2966740485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7840094d0 con 0x7fe7940b3490 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.733+0000 7fe793fff640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 0x7fe79415c410 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c7a0 tx=0x7fe78c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.733+0000 7fe77bfff640 1 -- 172.21.15.45:0/2966740485 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c003b70 con 0x7fe7940b3490 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.733+0000 7fe77bfff640 1 -- 172.21.15.45:0/2966740485 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78c003cf0 con 0x7fe7940b3490 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.733+0000 7fe77bfff640 1 -- 172.21.15.45:0/2966740485 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c003ea0 con 0x7fe7940b3490 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.733+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79409bb50 con 0x7fe7940b3490 2023-11-28T20:50:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.733+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79409c110 con 0x7fe7940b3490 2023-11-28T20:50:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.734+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79409d0f0 con 0x7fe7940b3490 2023-11-28T20:50:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.735+0000 7fe77bfff640 1 -- 172.21.15.45:0/2966740485 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) v1 ==== 89362+0+0 (secure 0 0 0) 0x7fe78c00aa90 con 0x7fe7940b3490 2023-11-28T20:50:02.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.736+0000 7fe77bfff640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fe78006afe0 0x7fe78006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:02.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.736+0000 7fe77bfff640 1 -- 172.21.15.45:0/2966740485 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe78c0898f0 con 0x7fe7940b3490 2023-11-28T20:50:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.737+0000 7fe7937fe640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fe78006afe0 0x7fe78006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.738+0000 7fe7937fe640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fe78006afe0 0x7fe78006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe784036000 tx=0x7fe784039040 comp rx=0 tx=0).ready entity=mgr.5246 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.741+0000 7fe77bfff640 1 -- 172.21.15.45:0/2966740485 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe78c0545c0 con 0x7fe7940b3490 2023-11-28T20:50:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:02.956+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 --> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7940b0720 con 0x7fe78006afe0 2023-11-28T20:50:03.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.126+0000 7fe77bfff640 1 -- 172.21.15.45:0/2966740485 <== mgr.5246 v2:172.21.15.190:6832/979211100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe7940b0720 con 0x7fe78006afe0 2023-11-28T20:50:03.128 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path snap_test_dir1 2023-11-28T20:50:03.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fe78006afe0 msgr2=0x7fe78006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:03.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fe78006afe0 0x7fe78006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe784036000 tx=0x7fe784039040 comp rx=0 tx=0).stop 2023-11-28T20:50:03.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 msgr2=0x7fe79415c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:03.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 0x7fe79415c410 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c7a0 tx=0x7fe78c00cc60 comp rx=0 tx=0).stop 2023-11-28T20:50:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 shutdown_connections 2023-11-28T20:50:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fe78006afe0 0x7fe78006d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7940b3490 0x7fe79415c410 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7940b2410 0x7fe79409ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 --2- 172.21.15.45:0/2966740485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe794071f20 0x7fe7941e8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 >> 172.21.15.45:0/2966740485 conn(0x7fe7940cecd0 msgr2=0x7fe7940cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.129+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 shutdown_connections 2023-11-28T20:50:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.130+0000 7fe798ead640 1 -- 172.21.15.45:0/2966740485 wait complete. 2023-11-28T20:50:03.155 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-11-28T20:50:03.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:03.155 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:50:03.212 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:50:03.212 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_50_00_UTC 2023-11-28T20:50:03.213 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-20_49_00_UTC 2023-11-28T20:50:03.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:03.213 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-20_49_00_UTC) 2023-11-28T20:50:03.277 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-20_50_00_UTC 2023-11-28T20:50:03.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:03.278 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-20_50_00_UTC) 2023-11-28T20:50:03.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:03.345 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-11-28T20:50:03.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T20:50:03.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.639+0000 7fb13bfff640 1 Processor -- start 2023-11-28T20:50:03.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.639+0000 7fb13bfff640 1 -- start start 2023-11-28T20:50:03.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.641+0000 7fb13bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb13c15b080 0x7fb13c0ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:03.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.641+0000 7fb13bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb13c0abd70 0x7fb13c0718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:03.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.641+0000 7fb13bfff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c0ac270 con 0x7fb13c071e00 2023-11-28T20:50:03.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.641+0000 7fb13bfff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c0722b0 con 0x7fb13c0abd70 2023-11-28T20:50:03.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.641+0000 7fb13bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c15b830 con 0x7fb13c15b080 2023-11-28T20:50:03.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.641+0000 7fb13b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb13c071e00 0x7fb13c15eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47472/0 (socket says 172.21.15.45:47472) 2023-11-28T20:50:03.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.641+0000 7fb13b7fe640 1 -- 172.21.15.45:0/2062386460 learned_addr learned my addr 172.21.15.45:0/2062386460 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb13affd640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb13c15b080 0x7fb13c0ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb13a7fc640 1 --2- 172.21.15.45:0/2062386460 >> v2:172.21.15.190:3300/0 conn(0x7fb13c0abd70 0x7fb13c0718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726059364 0 0) 0x7fb13c0ac270 con 0x7fb13c071e00 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb124003520 con 0x7fb13c071e00 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2164814395 0 0) 0x7fb124003520 con 0x7fb13c071e00 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb13c15b080 msgr2=0x7fb13c0ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb13c15b080 0x7fb13c0ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 >> v2:172.21.15.190:3300/0 conn(0x7fb13c0abd70 msgr2=0x7fb13c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 --2- 172.21.15.45:0/2062386460 >> v2:172.21.15.190:3300/0 conn(0x7fb13c0abd70 0x7fb13c0718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c15bb10 con 0x7fb13c071e00 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb1340019f0 con 0x7fb13c071e00 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 >> v1:172.21.15.45:6789/0 conn(0x7fb13c071e00 legacy=0x7fb13c15eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb124003970 0x7fb124003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1240047c0 0x7fb124004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb13c15b080 0x7fb124005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c0722b0 con 0x7fb1240047c0 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c15b830 con 0x7fb124003970 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.642+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12400b980 con 0x7fb13c15b080 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13a7fc640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1240047c0 0x7fb124004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13b7fe640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb13c15b080 0x7fb124005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13affd640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb124003970 0x7fb124003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:03.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13a7fc640 1 -- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb13c15b080 msgr2=0x7fb124005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:03.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13a7fc640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb13c15b080 0x7fb124005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13a7fc640 1 -- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb124003970 msgr2=0x7fb124003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:03.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13a7fc640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb124003970 0x7fb124003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13a7fc640 1 -- 172.21.15.45:0/2062386460 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124003520 con 0x7fb1240047c0 2023-11-28T20:50:03.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13affd640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb124003970 0x7fb124003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:03.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.643+0000 7fb13b7fe640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb13c15b080 0x7fb124005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:03.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.645+0000 7fb13a7fc640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1240047c0 0x7fb124004c40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb130009ac0 tx=0x7fb1240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0bbe2691b60e78 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:03.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.645+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13002c6d0 con 0x7fb1240047c0 2023-11-28T20:50:03.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.645+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13002c830 con 0x7fb1240047c0 2023-11-28T20:50:03.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.645+0000 7fb139ffb640 1 -- 172.21.15.45:0/2062386460 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13002cbc0 con 0x7fb1240047c0 2023-11-28T20:50:03.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.646+0000 7fb13bfff640 1 -- 172.21.15.45:0/2062386460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1240047c0 msgr2=0x7fb124004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:03.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.646+0000 7fb13bfff640 1 --2- 172.21.15.45:0/2062386460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1240047c0 0x7fb124004c40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb130009ac0 tx=0x7fb1240057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.646+0000 7fb13b7fe640 1 -- 172.21.15.45:0/2062386460 reap_dead start 2023-11-28T20:50:03.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.646+0000 7fb13bfff640 1 -- 172.21.15.45:0/2062386460 shutdown_connections 2023-11-28T20:50:03.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.646+0000 7fb13bfff640 1 -- 172.21.15.45:0/2062386460 >> 172.21.15.45:0/2062386460 conn(0x7fb13c0cecd0 msgr2=0x7fb13c0cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:03.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.646+0000 7fb13bfff640 1 -- 172.21.15.45:0/2062386460 shutdown_connections 2023-11-28T20:50:03.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.646+0000 7fb13bfff640 1 -- 172.21.15.45:0/2062386460 wait complete. 2023-11-28T20:50:03.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.648+0000 7fb13bfff640 1 Processor -- start 2023-11-28T20:50:03.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.648+0000 7fb13bfff640 1 -- start start 2023-11-28T20:50:03.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.649+0000 7fb13bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1240047c0 0x7fb13c14d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:03.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.649+0000 7fb13bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb13c14dca0 0x7fb13c14e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:03.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.649+0000 7fb13bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13c0aee60 0x7fb13c0af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:03.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.649+0000 7fb13bfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c0af820 con 0x7fb13c14dca0 2023-11-28T20:50:03.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.649+0000 7fb13bfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c14f930 con 0x7fb13c0aee60 2023-11-28T20:50:03.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.649+0000 7fb13bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c14edb0 con 0x7fb1240047c0 2023-11-28T20:50:03.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.649+0000 7fb13a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb13c14dca0 0x7fb13c14e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:03.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.649+0000 7fb13a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb13c14dca0 0x7fb13c14e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56778/0 (socket says 172.21.15.45:56778) 2023-11-28T20:50:03.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.649+0000 7fb13a7fc640 1 -- 172.21.15.45:0/2506844269 learned_addr learned my addr 172.21.15.45:0/2506844269 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:03.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb13a7fc640 1 -- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1240047c0 msgr2=0x7fb13c14d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb13affd640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1240047c0 0x7fb13c14d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb13b7fe640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13c0aee60 0x7fb13c0af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb13a7fc640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1240047c0 0x7fb13c14d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb13a7fc640 1 -- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13c0aee60 msgr2=0x7fb13c0af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb13a7fc640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13c0aee60 0x7fb13c0af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb13a7fc640 1 -- 172.21.15.45:0/2506844269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c14f090 con 0x7fb13c14dca0 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb13b7fe640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13c0aee60 0x7fb13c0af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb13a7fc640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb13c14dca0 0x7fb13c14e560 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb13002c2d0 tx=0x7fb130004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.650+0000 7fb1237fe640 1 -- 172.21.15.45:0/2506844269 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130003810 con 0x7fb13c14dca0 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.651+0000 7fb1237fe640 1 -- 172.21.15.45:0/2506844269 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb130003970 con 0x7fb13c14dca0 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.651+0000 7fb1237fe640 1 -- 172.21.15.45:0/2506844269 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130003ce0 con 0x7fb13c14dca0 2023-11-28T20:50:03.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.651+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c14f370 con 0x7fb13c14dca0 2023-11-28T20:50:03.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.651+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c14eaa0 con 0x7fb13c14dca0 2023-11-28T20:50:03.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.652+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c09d0f0 con 0x7fb13c14dca0 2023-11-28T20:50:03.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.653+0000 7fb1237fe640 1 -- 172.21.15.45:0/2506844269 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 25) v1 ==== 89362+0+0 (secure 0 0 0) 0x7fb130040660 con 0x7fb13c14dca0 2023-11-28T20:50:03.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.654+0000 7fb1237fe640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fb12806b000 0x7fb12806d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:03.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.654+0000 7fb1237fe640 1 -- 172.21.15.45:0/2506844269 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb1300abbd0 con 0x7fb13c14dca0 2023-11-28T20:50:03.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.655+0000 7fb13affd640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fb12806b000 0x7fb12806d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:03.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.656+0000 7fb13affd640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fb12806b000 0x7fb12806d4c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0098f0 tx=0x7fb12c008040 comp rx=0 tx=0).ready entity=mgr.5246 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:03.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.658+0000 7fb1237fe640 1 -- 172.21.15.45:0/2506844269 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb13007a380 con 0x7fb13c14dca0 2023-11-28T20:50:03.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:03.903+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb13c04e920 con 0x7fb13c14dca0 2023-11-28T20:50:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.390+0000 7fb1237fe640 1 -- 172.21.15.45:0/2506844269 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v26) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb1300044e0 con 0x7fb13c14dca0 2023-11-28T20:50:04.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.395+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fb12806b000 msgr2=0x7fb12806d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:04.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.395+0000 7fb13bfff640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fb12806b000 0x7fb12806d4c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0098f0 tx=0x7fb12c008040 comp rx=0 tx=0).stop 2023-11-28T20:50:04.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.395+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb13c14dca0 msgr2=0x7fb13c14e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:04.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.395+0000 7fb13bfff640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb13c14dca0 0x7fb13c14e560 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb13002c2d0 tx=0x7fb130004730 comp rx=0 tx=0).stop 2023-11-28T20:50:04.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.399+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 shutdown_connections 2023-11-28T20:50:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.399+0000 7fb13bfff640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fb12806b000 0x7fb12806d4c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.399+0000 7fb13bfff640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13c0aee60 0x7fb13c0af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.399+0000 7fb13bfff640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb13c14dca0 0x7fb13c14e560 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.399+0000 7fb13bfff640 1 --2- 172.21.15.45:0/2506844269 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1240047c0 0x7fb13c14d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.399+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 >> 172.21.15.45:0/2506844269 conn(0x7fb13c0cecd0 msgr2=0x7fb13c15c310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.400+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 shutdown_connections 2023-11-28T20:50:04.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.401+0000 7fb13bfff640 1 -- 172.21.15.45:0/2506844269 wait complete. 2023-11-28T20:50:04.434 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:50:04.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:50:04.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:50:04.660 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:50:04.661 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:50:04.661 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:04.717 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:50:04.717 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:04.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:04.956 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:04.954+0000 7f786b215040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:50:04.959 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:04.957+0000 7f083f61b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:50:04.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.980+0000 7f6cfef54640 1 Processor -- start 2023-11-28T20:50:04.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.981+0000 7f6cfef54640 1 -- start start 2023-11-28T20:50:04.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.982+0000 7f6cfef54640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf80d02d0 0x7f6cf80d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.982+0000 7f6cfef54640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6cf80d1ab0 0x7f6cf80b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.982+0000 7f6cfef54640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf80d2230 con 0x7f6cf815d550 2023-11-28T20:50:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.982+0000 7f6cfef54640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf80d2510 con 0x7f6cf80d1ab0 2023-11-28T20:50:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.982+0000 7f6cfef54640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf80d27f0 con 0x7f6cf80d02d0 2023-11-28T20:50:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.983+0000 7f6cfe753640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6cf80d1ab0 0x7f6cf80b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.983+0000 7f6cfe753640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6cf80d1ab0 0x7f6cf80b6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48816/0 (socket says 172.21.15.45:48816) 2023-11-28T20:50:04.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.983+0000 7f6cfe753640 1 -- 172.21.15.45:0/2964729204 learned_addr learned my addr 172.21.15.45:0/2964729204 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.983+0000 7f6cfd751640 1 --2- 172.21.15.45:0/2964729204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf80d02d0 0x7f6cf80d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.983+0000 7f6cfe753640 1 -- 172.21.15.45:0/2964729204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf80d02d0 msgr2=0x7f6cf80d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.983+0000 7f6cfe753640 1 --2- 172.21.15.45:0/2964729204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf80d02d0 0x7f6cf80d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.983+0000 7f6cfe753640 1 -- 172.21.15.45:0/2964729204 >> v1:172.21.15.190:6789/0 conn(0x7f6cf815d550 legacy=0x7f6cf80cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.983+0000 7f6cfe753640 1 -- 172.21.15.45:0/2964729204 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf80b6a30 con 0x7f6cf80d1ab0 2023-11-28T20:50:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.983+0000 7f6cfd751640 1 --2- 172.21.15.45:0/2964729204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf80d02d0 0x7f6cf80d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.984+0000 7f6cfe753640 1 --2- 172.21.15.45:0/2964729204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d1ab0 0x7f6cf80b6f50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009e50 tx=0x7f6cf002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2a9ee41fa19a84 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:04.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.984+0000 7f6cfcf50640 1 -- 172.21.15.45:0/2964729204 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf002ca70 con 0x7f6cf80d1ab0 2023-11-28T20:50:04.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.984+0000 7f6cfcf50640 1 -- 172.21.15.45:0/2964729204 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf002cbd0 con 0x7f6cf80d1ab0 2023-11-28T20:50:04.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.984+0000 7f6cfcf50640 1 -- 172.21.15.45:0/2964729204 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf00343f0 con 0x7f6cf80d1ab0 2023-11-28T20:50:04.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.985+0000 7f6cfef54640 1 -- 172.21.15.45:0/2964729204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d1ab0 msgr2=0x7f6cf80b6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:04.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.985+0000 7f6cfef54640 1 --2- 172.21.15.45:0/2964729204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d1ab0 0x7f6cf80b6f50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009e50 tx=0x7f6cf002bed0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.985+0000 7f6cfef54640 1 -- 172.21.15.45:0/2964729204 shutdown_connections 2023-11-28T20:50:04.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.985+0000 7f6cfef54640 1 --2- 172.21.15.45:0/2964729204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d1ab0 0x7f6cf80b6f50 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.985+0000 7f6cfef54640 1 --2- 172.21.15.45:0/2964729204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf80d02d0 0x7f6cf80d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.985+0000 7f6cfef54640 1 -- 172.21.15.45:0/2964729204 >> 172.21.15.45:0/2964729204 conn(0x7f6cf806ac70 msgr2=0x7f6cf80b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:04.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.985+0000 7f6cfef54640 1 -- 172.21.15.45:0/2964729204 shutdown_connections 2023-11-28T20:50:04.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.985+0000 7f6cfef54640 1 -- 172.21.15.45:0/2964729204 wait complete. 2023-11-28T20:50:04.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.987+0000 7f6cfef54640 1 Processor -- start 2023-11-28T20:50:04.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.988+0000 7f6cfef54640 1 -- start start 2023-11-28T20:50:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfef54640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d02d0 0x7f6cf80c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfef54640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cf80d1ab0 0x7f6cf8153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfef54640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf815d550 0x7f6cf80c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfef54640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf81581f0 con 0x7f6cf80d02d0 2023-11-28T20:50:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfef54640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8157600 con 0x7f6cf80d1ab0 2023-11-28T20:50:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfef54640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf80c8690 con 0x7f6cf815d550 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfdf52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d02d0 0x7f6cf80c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfdf52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d02d0 0x7f6cf80c7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48820/0 (socket says 172.21.15.45:48820) 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfdf52640 1 -- 172.21.15.45:0/3929508236 learned_addr learned my addr 172.21.15.45:0/3929508236 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfe753640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf815d550 0x7f6cf80c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfd751640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cf80d1ab0 0x7f6cf8153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfdf52640 1 -- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf815d550 msgr2=0x7f6cf80c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfdf52640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf815d550 0x7f6cf80c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfdf52640 1 -- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cf80d1ab0 msgr2=0x7f6cf8153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfdf52640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cf80d1ab0 0x7f6cf8153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfdf52640 1 -- 172.21.15.45:0/3929508236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf0009b30 con 0x7f6cf80d02d0 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.989+0000 7f6cfd751640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cf80d1ab0 0x7f6cf8153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.990+0000 7f6cfdf52640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d02d0 0x7f6cf80c7a20 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400c430 tx=0x7f6cf400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.990+0000 7f6cee7fc640 1 -- 172.21.15.45:0/3929508236 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4011790 con 0x7f6cf80d02d0 2023-11-28T20:50:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.990+0000 7f6cee7fc640 1 -- 172.21.15.45:0/3929508236 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf40118f0 con 0x7f6cf80d02d0 2023-11-28T20:50:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.990+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf8157850 con 0x7f6cf80d02d0 2023-11-28T20:50:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.990+0000 7f6cee7fc640 1 -- 172.21.15.45:0/3929508236 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4011ca0 con 0x7f6cf80d02d0 2023-11-28T20:50:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.990+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf8157e10 con 0x7f6cf80d02d0 2023-11-28T20:50:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.991+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf80ce810 con 0x7f6cf80d02d0 2023-11-28T20:50:04.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.992+0000 7f6cee7fc640 1 -- 172.21.15.45:0/3929508236 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f6cf401c460 con 0x7f6cf80d02d0 2023-11-28T20:50:04.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.993+0000 7f6cee7fc640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6cdc06afb0 0x7f6cdc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:04.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.994+0000 7f6cee7fc640 1 -- 172.21.15.45:0/3929508236 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6519+0+0 (secure 0 0 0) 0x7f6cf4089ad0 con 0x7f6cf80d02d0 2023-11-28T20:50:04.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.994+0000 7f6cfd751640 1 -- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6cdc06afb0 msgr2=0x7f6cdc06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:04.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.994+0000 7f6cfd751640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6cdc06afb0 0x7f6cdc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:05.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:04.998+0000 7f6cee7fc640 1 -- 172.21.15.45:0/3929508236 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6cf4054720 con 0x7f6cf80d02d0 2023-11-28T20:50:05.011 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.009+0000 7f786b215040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:50:05.017 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:05.015+0000 7f083f61b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:50:05.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.194+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cf804e920 con 0x7f6cf80d02d0 2023-11-28T20:50:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.195+0000 7f6cfd751640 1 -- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6cdc06afb0 msgr2=0x7f6cdc06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:05.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.195+0000 7f6cfd751640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6cdc06afb0 0x7f6cdc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:05.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.196+0000 7f6cee7fc640 1 -- 172.21.15.45:0/3929508236 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+17460 (secure 0 0 0) 0x7f6cf4057d40 con 0x7f6cf80d02d0 2023-11-28T20:50:05.198 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:05.199 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":75,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:04.376426+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:48:11.673148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-28T20:48:12.760782+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":75,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/2346462975":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/1966056956":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6832/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/633536721":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6833/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/4198236319":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/676976495":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/3224416347":"2023-11-29T20:48:28.490206+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.198+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6cdc06afb0 msgr2=0x7f6cdc06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.198+0000 7f6cfef54640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6cdc06afb0 0x7f6cdc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.198+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d02d0 msgr2=0x7f6cf80c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.199+0000 7f6cfef54640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d02d0 0x7f6cf80c7a20 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400c430 tx=0x7f6cf400c8f0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.199+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 shutdown_connections 2023-11-28T20:50:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.199+0000 7f6cfef54640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f6cdc06afb0 0x7f6cdc06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.199+0000 7f6cfef54640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cf815d550 0x7f6cf80c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.199+0000 7f6cfef54640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cf80d1ab0 0x7f6cf8153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.199+0000 7f6cfef54640 1 --2- 172.21.15.45:0/3929508236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cf80d02d0 0x7f6cf80c7a20 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.199+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 >> 172.21.15.45:0/3929508236 conn(0x7f6cf806ac70 msgr2=0x7f6cf815dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.199+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 shutdown_connections 2023-11-28T20:50:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.199+0000 7f6cfef54640 1 -- 172.21.15.45:0/3929508236 wait complete. 2023-11-28T20:50:05.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:48:11.673148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:48:12.760782+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 75, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:50:05.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:50:05.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.432+0000 7fcf39161640 1 Processor -- start 2023-11-28T20:50:05.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.433+0000 7fcf39161640 1 -- start start 2023-11-28T20:50:05.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.434+0000 7fcf39161640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcf3406ac10 0x7fcf3406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.434+0000 7fcf39161640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3415ecd0 con 0x7fcf3415e840 2023-11-28T20:50:05.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.434+0000 7fcf39161640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3415f290 con 0x7fcf3415f720 2023-11-28T20:50:05.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.434+0000 7fcf39161640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3415fbd0 con 0x7fcf3406ac10 2023-11-28T20:50:05.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.434+0000 7fcf3259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcf3406ac10 0x7fcf3406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:05.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.434+0000 7fcf32d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcf3415e840 0x7fcf3406a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53686/0 (socket says 172.21.15.45:53686) 2023-11-28T20:50:05.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.434+0000 7fcf32d9d640 1 -- 172.21.15.45:0/1489637182 learned_addr learned my addr 172.21.15.45:0/1489637182 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:05.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.435+0000 7fcf3259c640 1 -- 172.21.15.45:0/1489637182 >> v1:172.21.15.190:6789/0 conn(0x7fcf3415f720 legacy=0x7fcf340cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.435+0000 7fcf3259c640 1 -- 172.21.15.45:0/1489637182 >> v1:172.21.15.45:6789/0 conn(0x7fcf3415e840 legacy=0x7fcf3406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.435+0000 7fcf3259c640 1 -- 172.21.15.45:0/1489637182 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf34072020 con 0x7fcf3406ac10 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.435+0000 7fcf31d9b640 1 -- 172.21.15.45:0/1489637182 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426867882 0 0) 0x7fcf3415ecd0 con 0x7fcf3415e840 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.436+0000 7fcf3259c640 1 --2- 172.21.15.45:0/1489637182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3406ac10 0x7fcf3406c350 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcf280097a0 tx=0x7fcf2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad881a69cede9fd5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.436+0000 7fcf31d9b640 1 -- 172.21.15.45:0/1489637182 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28031830 con 0x7fcf3406ac10 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.436+0000 7fcf31d9b640 1 -- 172.21.15.45:0/1489637182 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf28031990 con 0x7fcf3406ac10 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.436+0000 7fcf31d9b640 1 -- 172.21.15.45:0/1489637182 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28031d20 con 0x7fcf3406ac10 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.437+0000 7fcf39161640 1 -- 172.21.15.45:0/1489637182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3406ac10 msgr2=0x7fcf3406c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.437+0000 7fcf39161640 1 --2- 172.21.15.45:0/1489637182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3406ac10 0x7fcf3406c350 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcf280097a0 tx=0x7fcf2802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.437+0000 7fcf39161640 1 -- 172.21.15.45:0/1489637182 shutdown_connections 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.437+0000 7fcf39161640 1 --2- 172.21.15.45:0/1489637182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3406ac10 0x7fcf3406c350 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.437+0000 7fcf39161640 1 -- 172.21.15.45:0/1489637182 >> 172.21.15.45:0/1489637182 conn(0x7fcf340ced60 msgr2=0x7fcf340cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:05.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.437+0000 7fcf39161640 1 -- 172.21.15.45:0/1489637182 shutdown_connections 2023-11-28T20:50:05.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.437+0000 7fcf39161640 1 -- 172.21.15.45:0/1489637182 wait complete. 2023-11-28T20:50:05.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.440+0000 7fcf39161640 1 Processor -- start 2023-11-28T20:50:05.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.440+0000 7fcf39161640 1 -- start start 2023-11-28T20:50:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf39161640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf3406ac10 0x7fcf340b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf39161640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcf3415e840 0x7fcf340ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf39161640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3415f720 0x7fcf34099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf39161640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340b2eb0 con 0x7fcf3415f720 2023-11-28T20:50:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf39161640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340c9cb0 con 0x7fcf3406ac10 2023-11-28T20:50:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf39161640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340ca550 con 0x7fcf3415e840 2023-11-28T20:50:05.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf3359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3415f720 0x7fcf34099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:05.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf3359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3415f720 0x7fcf34099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48844/0 (socket says 172.21.15.45:48844) 2023-11-28T20:50:05.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf3359e640 1 -- 172.21.15.45:0/2224668860 learned_addr learned my addr 172.21.15.45:0/2224668860 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf32d9d640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf3406ac10 0x7fcf340b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf3259c640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcf3415e840 0x7fcf340ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf3359e640 1 -- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcf3415e840 msgr2=0x7fcf340ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf3359e640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcf3415e840 0x7fcf340ac4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf3359e640 1 -- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf3406ac10 msgr2=0x7fcf340b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf3359e640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf3406ac10 0x7fcf340b4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.442+0000 7fcf3359e640 1 -- 172.21.15.45:0/2224668860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf28009400 con 0x7fcf3415f720 2023-11-28T20:50:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.443+0000 7fcf3359e640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3415f720 0x7fcf34099a40 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c00ba10 tx=0x7fcf2c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.443+0000 7fcf1b7fe640 1 -- 172.21.15.45:0/2224668860 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c00cac0 con 0x7fcf3415f720 2023-11-28T20:50:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.443+0000 7fcf1b7fe640 1 -- 172.21.15.45:0/2224668860 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c00cc20 con 0x7fcf3415f720 2023-11-28T20:50:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.443+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf340b47c0 con 0x7fcf3415f720 2023-11-28T20:50:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.443+0000 7fcf1b7fe640 1 -- 172.21.15.45:0/2224668860 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c014440 con 0x7fcf3415f720 2023-11-28T20:50:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.443+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf340ca1e0 con 0x7fcf3415f720 2023-11-28T20:50:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.444+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf340733d0 con 0x7fcf3415f720 2023-11-28T20:50:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.445+0000 7fcf1b7fe640 1 -- 172.21.15.45:0/2224668860 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 89345+0+0 (secure 0 0 0) 0x7fcf2c01d3b0 con 0x7fcf3415f720 2023-11-28T20:50:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.447+0000 7fcf1b7fe640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fcf2006af60 0x7fcf2006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.447+0000 7fcf1b7fe640 1 -- 172.21.15.45:0/2224668860 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fcf2c0899f0 con 0x7fcf3415f720 2023-11-28T20:50:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.447+0000 7fcf32d9d640 1 -- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fcf2006af60 msgr2=0x7fcf2006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.447+0000 7fcf32d9d640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fcf2006af60 0x7fcf2006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:05.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.451+0000 7fcf1b7fe640 1 -- 172.21.15.45:0/2224668860 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf2c0545d0 con 0x7fcf3415f720 2023-11-28T20:50:05.474 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.471+0000 7f786b215040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:50:05.508 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:05.506+0000 7f083f61b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:50:05.530 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.527+0000 7f786b215040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:50:05.563 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:05.561+0000 7f083f61b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:50:05.580 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.578+0000 7f786b215040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:50:05.616 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:05.614+0000 7f083f61b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:50:05.641 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.638+0000 7f786b215040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:50:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.641+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf340ca690 con 0x7fcf3415f720 2023-11-28T20:50:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.642+0000 7fcf1b7fe640 1 -- 172.21.15.45:0/2224668860 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf2c019020 con 0x7fcf3415f720 2023-11-28T20:50:05.644 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:50:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fcf2006af60 msgr2=0x7fcf2006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fcf2006af60 0x7fcf2006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3415f720 msgr2=0x7fcf34099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3415f720 0x7fcf34099a40 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c00ba10 tx=0x7fcf2c00bed0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 shutdown_connections 2023-11-28T20:50:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fcf2006af60 0x7fcf2006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf3415f720 0x7fcf34099a40 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcf3415e840 0x7fcf340ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 --2- 172.21.15.45:0/2224668860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf3406ac10 0x7fcf340b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 >> 172.21.15.45:0/2224668860 conn(0x7fcf340ced60 msgr2=0x7fcf340ad200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 shutdown_connections 2023-11-28T20:50:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.645+0000 7fcf39161640 1 -- 172.21.15.45:0/2224668860 wait complete. 2023-11-28T20:50:05.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:50:05.682 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:05.680+0000 7f083f61b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:50:05.694 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.692+0000 7f786b215040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:50:05.734 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:05.732+0000 7f083f61b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:50:05.807 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.804+0000 7f786b215040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:50:05.859 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:05.856+0000 7f083f61b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:50:05.869 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.867+0000 7f786b215040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:05.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.873+0000 7f73c1dd0640 1 Processor -- start 2023-11-28T20:50:05.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.873+0000 7f73c1dd0640 1 -- start start 2023-11-28T20:50:05.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.874+0000 7f73c1dd0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73bc15d3c0 0x7f73bc0cda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73c1dd0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73bc0d02d0 0x7f73bc0b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73c1dd0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73bc0d2e70 con 0x7f73bc15d7a0 2023-11-28T20:50:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73c1dd0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73bc0b6920 con 0x7f73bc15d3c0 2023-11-28T20:50:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73c1dd0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0ce110 con 0x7f73bc0d02d0 2023-11-28T20:50:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73bb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73bc15d3c0 0x7f73bc0cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73bb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73bc15d3c0 0x7f73bc0cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48854/0 (socket says 172.21.15.45:48854) 2023-11-28T20:50:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73bb7fe640 1 -- 172.21.15.45:0/1579846485 learned_addr learned my addr 172.21.15.45:0/1579846485 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73bb7fe640 1 -- 172.21.15.45:0/1579846485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73bc0d02d0 msgr2=0x7f73bc0b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73bb7fe640 1 --2- 172.21.15.45:0/1579846485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73bc0d02d0 0x7f73bc0b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73bb7fe640 1 -- 172.21.15.45:0/1579846485 >> v1:172.21.15.190:6789/0 conn(0x7f73bc15d7a0 legacy=0x7f73bc0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:05.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.875+0000 7f73bb7fe640 1 -- 172.21.15.45:0/1579846485 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc0ce3f0 con 0x7f73bc15d3c0 2023-11-28T20:50:05.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.876+0000 7f73bb7fe640 1 --2- 172.21.15.45:0/1579846485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 0x7f73bc0cda10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009870 tx=0x7f73ac02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e28587bbd375b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:05.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.877+0000 7f73ba7fc640 1 -- 172.21.15.45:0/1579846485 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0317c0 con 0x7f73bc15d3c0 2023-11-28T20:50:05.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.877+0000 7f73ba7fc640 1 -- 172.21.15.45:0/1579846485 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac031920 con 0x7f73bc15d3c0 2023-11-28T20:50:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.877+0000 7f73ba7fc640 1 -- 172.21.15.45:0/1579846485 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac031cb0 con 0x7f73bc15d3c0 2023-11-28T20:50:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.877+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1579846485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 msgr2=0x7f73bc0cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.877+0000 7f73c1dd0640 1 --2- 172.21.15.45:0/1579846485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 0x7f73bc0cda10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009870 tx=0x7f73ac02b860 comp rx=0 tx=0).stop 2023-11-28T20:50:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.877+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1579846485 shutdown_connections 2023-11-28T20:50:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.877+0000 7f73c1dd0640 1 --2- 172.21.15.45:0/1579846485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73bc0d02d0 0x7f73bc0b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.877+0000 7f73c1dd0640 1 --2- 172.21.15.45:0/1579846485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 0x7f73bc0cda10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.878+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1579846485 >> 172.21.15.45:0/1579846485 conn(0x7f73bc0b4ca0 msgr2=0x7f73bc0b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.878+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1579846485 shutdown_connections 2023-11-28T20:50:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.878+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1579846485 wait complete. 2023-11-28T20:50:05.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.880+0000 7f73c1dd0640 1 Processor -- start 2023-11-28T20:50:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.881+0000 7f73c1dd0640 1 -- start start 2023-11-28T20:50:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.882+0000 7f73c1dd0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73bc0d02d0 0x7f73bc159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.882+0000 7f73c1dd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 0x7f73bc0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.882+0000 7f73c1dd0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73bc0aa9f0 0x7f73bc15fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.882+0000 7f73c1dd0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc15a010 con 0x7f73bc15d3c0 2023-11-28T20:50:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.882+0000 7f73c1dd0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0cbe40 con 0x7f73bc0aa9f0 2023-11-28T20:50:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.882+0000 7f73c1dd0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0cc120 con 0x7f73bc0d02d0 2023-11-28T20:50:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.882+0000 7f73baffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 0x7f73bc0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:05.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.882+0000 7f73baffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 0x7f73bc0aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48860/0 (socket says 172.21.15.45:48860) 2023-11-28T20:50:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.882+0000 7f73baffd640 1 -- 172.21.15.45:0/1730667996 learned_addr learned my addr 172.21.15.45:0/1730667996 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73bb7fe640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73bc0d02d0 0x7f73bc159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:05.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73bbfff640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73bc0aa9f0 0x7f73bc15fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73baffd640 1 -- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73bc0d02d0 msgr2=0x7f73bc159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73baffd640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73bc0d02d0 0x7f73bc159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73baffd640 1 -- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73bc0aa9f0 msgr2=0x7f73bc15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73baffd640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73bc0aa9f0 0x7f73bc15fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73baffd640 1 -- 172.21.15.45:0/1730667996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0094d0 con 0x7f73bc15d3c0 2023-11-28T20:50:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73bbfff640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73bc0aa9f0 0x7f73bc15fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:05.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73baffd640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 0x7f73bc0aa4b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f73b000b3e0 tx=0x7f73b000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:05.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73a3fff640 1 -- 172.21.15.45:0/1730667996 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0011830 con 0x7f73bc15d3c0 2023-11-28T20:50:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.883+0000 7f73a3fff640 1 -- 172.21.15.45:0/1730667996 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b00119b0 con 0x7f73bc15d3c0 2023-11-28T20:50:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.884+0000 7f73a3fff640 1 -- 172.21.15.45:0/1730667996 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0011d60 con 0x7f73bc15d3c0 2023-11-28T20:50:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.884+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73bc0cc400 con 0x7f73bc15d3c0 2023-11-28T20:50:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.884+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73bc0cc9c0 con 0x7f73bc15d3c0 2023-11-28T20:50:05.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.885+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73bc0ccbe0 con 0x7f73bc15d3c0 2023-11-28T20:50:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.886+0000 7f73a3fff640 1 -- 172.21.15.45:0/1730667996 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f73b001a400 con 0x7f73bc15d3c0 2023-11-28T20:50:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.887+0000 7f73a3fff640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f73a806b060 0x7f73a806d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.887+0000 7f73a3fff640 1 -- 172.21.15.45:0/1730667996 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f73b0089c60 con 0x7f73bc15d3c0 2023-11-28T20:50:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.887+0000 7f73bb7fe640 1 -- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f73a806b060 msgr2=0x7f73a806d520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:05.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.887+0000 7f73bb7fe640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f73a806b060 0x7f73a806d520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:05.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:05.891+0000 7f73a3fff640 1 -- 172.21.15.45:0/1730667996 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73b00548c0 con 0x7f73bc15d3c0 2023-11-28T20:50:05.920 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.918+0000 7f786b215040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:50:05.928 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:05.926+0000 7f083f61b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:05.978 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:05.976+0000 7f786b215040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:50:05.986 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:05.984+0000 7f083f61b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:50:06.030 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:06.028+0000 7f786b215040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:50:06.046 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:06.044+0000 7f083f61b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:50:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.088+0000 7f73bb7fe640 1 -- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f73a806b060 msgr2=0x7f73a806d520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.088+0000 7f73bb7fe640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f73a806b060 0x7f73a806d520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.092+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f73bc0733d0 con 0x7f73bc15d3c0 2023-11-28T20:50:06.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.093+0000 7f73a3fff640 1 -- 172.21.15.45:0/1730667996 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7f73b00150a0 con 0x7f73bc15d3c0 2023-11-28T20:50:06.096 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:50:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f73a806b060 msgr2=0x7f73a806d520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f73a806b060 0x7f73a806d520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 msgr2=0x7f73bc0aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 0x7f73bc0aa4b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f73b000b3e0 tx=0x7f73b000b8a0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 shutdown_connections 2023-11-28T20:50:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f73a806b060 0x7f73a806d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73bc0aa9f0 0x7f73bc15fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73bc15d3c0 0x7f73bc0aa4b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 --2- 172.21.15.45:0/1730667996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73bc0d02d0 0x7f73bc159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 >> 172.21.15.45:0/1730667996 conn(0x7f73bc0b4ca0 msgr2=0x7f73bc15c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 shutdown_connections 2023-11-28T20:50:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.096+0000 7f73c1dd0640 1 -- 172.21.15.45:0/1730667996 wait complete. 2023-11-28T20:50:06.099 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:06.097+0000 7f083f61b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:50:06.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:50:06.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.377+0000 7f482bf51640 1 Processor -- start 2023-11-28T20:50:06.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.378+0000 7f482bf51640 1 -- start start 2023-11-28T20:50:06.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.379+0000 7f482bf51640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4824160aa0 0x7f48240b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.379+0000 7f482bf51640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4824160e80 con 0x7f48240d24f0 2023-11-28T20:50:06.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.379+0000 7f482bf51640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240d1220 con 0x7f48240ce7f0 2023-11-28T20:50:06.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.379+0000 7f482bf51640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48240b6960 con 0x7f4824160aa0 2023-11-28T20:50:06.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.379+0000 7f48294ec640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f48240d24f0 0x7f48240ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53702/0 (socket says 172.21.15.45:53702) 2023-11-28T20:50:06.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.379+0000 7f48294ec640 1 -- 172.21.15.45:0/4026435687 learned_addr learned my addr 172.21.15.45:0/4026435687 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:06.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.379+0000 7f4829ced640 1 --2- 172.21.15.45:0/4026435687 >> v2:172.21.15.190:3300/0 conn(0x7f4824160aa0 0x7f48240b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769927066 0 0) 0x7f4824160e80 con 0x7f48240d24f0 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4810003520 con 0x7f48240d24f0 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2692604477 0 0) 0x7f4810003520 con 0x7f48240d24f0 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 >> v2:172.21.15.190:3300/0 conn(0x7f4824160aa0 msgr2=0x7f48240b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 --2- 172.21.15.45:0/4026435687 >> v2:172.21.15.190:3300/0 conn(0x7f4824160aa0 0x7f48240b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 >> v1:172.21.15.190:6789/0 conn(0x7f48240ce7f0 legacy=0x7f48241612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48240d1470 con 0x7f48240d24f0 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4814001670 con 0x7f48240d24f0 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 >> v1:172.21.15.45:6789/0 conn(0x7f48240d24f0 legacy=0x7f48240ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4810003970 0x7f4810003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48100047c0 0x7f4810004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4810005180 0x7f481000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48240b6960 con 0x7f4810005180 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481000c030 con 0x7f48100047c0 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.380+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481000c1f0 con 0x7f4810003970 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.381+0000 7f482a4ee640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4810005180 0x7f481000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.381+0000 7f4829ced640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4810003970 0x7f4810003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.381+0000 7f48294ec640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48100047c0 0x7f4810004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.381+0000 7f482a4ee640 1 -- 172.21.15.45:0/4026435687 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4810003970 msgr2=0x7f4810003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.381+0000 7f482a4ee640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4810003970 0x7f4810003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.381+0000 7f482a4ee640 1 -- 172.21.15.45:0/4026435687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48100047c0 msgr2=0x7f4810004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.381+0000 7f482a4ee640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48100047c0 0x7f4810004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.381+0000 7f482a4ee640 1 -- 172.21.15.45:0/4026435687 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4810003520 con 0x7f4810005180 2023-11-28T20:50:06.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.382+0000 7f482a4ee640 1 -- 172.21.15.45:0/4026435687 reap_dead start 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.382+0000 7f482a4ee640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4810005180 0x7f481000b980 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f481c001e20 tx=0x7f48100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc880cca9de368da server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.382+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c003b70 con 0x7f4810005180 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.382+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c003cd0 con 0x7f4810005180 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.382+0000 7f4828ceb640 1 -- 172.21.15.45:0/4026435687 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c005310 con 0x7f4810005180 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.383+0000 7f482bf51640 1 -- 172.21.15.45:0/4026435687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4810005180 msgr2=0x7f481000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.383+0000 7f482bf51640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4810005180 0x7f481000b980 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f481c001e20 tx=0x7f48100057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.383+0000 7f482bf51640 1 -- 172.21.15.45:0/4026435687 shutdown_connections 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.383+0000 7f482bf51640 1 --2- 172.21.15.45:0/4026435687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4810005180 0x7f481000b980 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.383+0000 7f482bf51640 1 -- 172.21.15.45:0/4026435687 >> 172.21.15.45:0/4026435687 conn(0x7f4824070b20 msgr2=0x7f4824070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.383+0000 7f482bf51640 1 -- 172.21.15.45:0/4026435687 shutdown_connections 2023-11-28T20:50:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.383+0000 7f482bf51640 1 -- 172.21.15.45:0/4026435687 wait complete. 2023-11-28T20:50:06.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.386+0000 7f482bf51640 1 Processor -- start 2023-11-28T20:50:06.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.386+0000 7f482bf51640 1 -- start start 2023-11-28T20:50:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.388+0000 7f482bf51640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4810005180 0x7f48241e8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.388+0000 7f482bf51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48241e6900 0x7f48240b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.388+0000 7f482bf51640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f482414f600 0x7f48240b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.388+0000 7f482bf51640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48241e7bf0 con 0x7f48241e6900 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.388+0000 7f482bf51640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48241e6d80 con 0x7f4810005180 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.388+0000 7f482bf51640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48240b4360 con 0x7f482414f600 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.388+0000 7f48294ec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48241e6900 0x7f48240b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.388+0000 7f48294ec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48241e6900 0x7f48240b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48884/0 (socket says 172.21.15.45:48884) 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.388+0000 7f48294ec640 1 -- 172.21.15.45:0/121298220 learned_addr learned my addr 172.21.15.45:0/121298220 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f4829ced640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4810005180 0x7f48241e8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f48294ec640 1 -- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f482414f600 msgr2=0x7f48240b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f48294ec640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f482414f600 0x7f48240b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f48294ec640 1 -- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4810005180 msgr2=0x7f48241e8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f48294ec640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4810005180 0x7f48241e8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f48294ec640 1 -- 172.21.15.45:0/121298220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48241e6fe0 con 0x7f48241e6900 2023-11-28T20:50:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f4829ced640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4810005180 0x7f48241e8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f48294ec640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48241e6900 0x7f48240b3e20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f481400e7b0 tx=0x7f481400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f481a7fc640 1 -- 172.21.15.45:0/121298220 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481400eec0 con 0x7f48241e6900 2023-11-28T20:50:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f481a7fc640 1 -- 172.21.15.45:0/121298220 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481400c850 con 0x7f48241e6900 2023-11-28T20:50:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.389+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48241e72c0 con 0x7f48241e6900 2023-11-28T20:50:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.390+0000 7f481a7fc640 1 -- 172.21.15.45:0/121298220 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481400cc00 con 0x7f48241e6900 2023-11-28T20:50:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.390+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48241e7880 con 0x7f48241e6900 2023-11-28T20:50:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.391+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48241e7a40 con 0x7f48241e6900 2023-11-28T20:50:06.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.392+0000 7f481a7fc640 1 -- 172.21.15.45:0/121298220 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f481401d460 con 0x7f48241e6900 2023-11-28T20:50:06.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.393+0000 7f481a7fc640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f480806af90 0x7f480806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.393+0000 7f481a7fc640 1 -- 172.21.15.45:0/121298220 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f48140899d0 con 0x7f48241e6900 2023-11-28T20:50:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.394+0000 7f4829ced640 1 -- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f480806af90 msgr2=0x7f480806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.394+0000 7f4829ced640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f480806af90 0x7f480806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.397+0000 7f481a7fc640 1 -- 172.21.15.45:0/121298220 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4814054640 con 0x7f48241e6900 2023-11-28T20:50:06.421 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:06.419+0000 7f786b215040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:50:06.513 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:06.511+0000 7f083f61b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:50:06.580 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:06.578+0000 7f786b215040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:50:06.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.589+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48240733d0 con 0x7f48241e6900 2023-11-28T20:50:06.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.590+0000 7f481a7fc640 1 -- 172.21.15.45:0/121298220 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4814057c60 con 0x7f48241e6900 2023-11-28T20:50:06.592 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:50:06.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f480806af90 msgr2=0x7f480806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f480806af90 0x7f480806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48241e6900 msgr2=0x7f48240b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48241e6900 0x7f48240b3e20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f481400e7b0 tx=0x7f481400ec70 comp rx=0 tx=0).stop 2023-11-28T20:50:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 shutdown_connections 2023-11-28T20:50:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f480806af90 0x7f480806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f482414f600 0x7f48240b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48241e6900 0x7f48240b3e20 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 --2- 172.21.15.45:0/121298220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4810005180 0x7f48241e8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 >> 172.21.15.45:0/121298220 conn(0x7f4824070b20 msgr2=0x7f48240ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 shutdown_connections 2023-11-28T20:50:06.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.592+0000 7f482bf51640 1 -- 172.21.15.45:0/121298220 wait complete. 2023-11-28T20:50:06.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:50:06.683 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:06.681+0000 7f083f61b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:50:06.688 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:06.686+0000 7f786b215040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:50:06.738 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:06.736+0000 7f786b215040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:50:06.798 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:06.796+0000 7f083f61b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:50:06.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.812+0000 7f47a2889640 1 Processor -- start 2023-11-28T20:50:06.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.812+0000 7f47a2889640 1 -- start start 2023-11-28T20:50:06.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.814+0000 7f47a2889640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f479c15b820 0x7f479c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.814+0000 7f47a2889640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f479c15f4d0 0x7f479c0cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.814+0000 7f47a2889640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479c15fa80 con 0x7f479c0cf020 2023-11-28T20:50:06.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.814+0000 7f47a2889640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c15fd60 con 0x7f479c15f4d0 2023-11-28T20:50:06.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.814+0000 7f47a2889640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c0cf9f0 con 0x7f479c15b820 2023-11-28T20:50:06.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.814+0000 7f47a0e26640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f479c15f4d0 0x7f479c0cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.814+0000 7f47a0e26640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f479c15f4d0 0x7f479c0cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48890/0 (socket says 172.21.15.45:48890) 2023-11-28T20:50:06.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.814+0000 7f47a0e26640 1 -- 172.21.15.45:0/3243255341 learned_addr learned my addr 172.21.15.45:0/3243255341 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:06.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.814+0000 7f479bfff640 1 --2- 172.21.15.45:0/3243255341 >> v2:172.21.15.190:3300/0 conn(0x7f479c15b820 0x7f479c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.815+0000 7f47a0e26640 1 -- 172.21.15.45:0/3243255341 >> v2:172.21.15.190:3300/0 conn(0x7f479c15b820 msgr2=0x7f479c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:06.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.815+0000 7f47a0e26640 1 --2- 172.21.15.45:0/3243255341 >> v2:172.21.15.190:3300/0 conn(0x7f479c15b820 0x7f479c06da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.815+0000 7f47a0e26640 1 -- 172.21.15.45:0/3243255341 >> v1:172.21.15.190:6789/0 conn(0x7f479c0cf020 legacy=0x7f479c15edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:06.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.815+0000 7f47a0e26640 1 -- 172.21.15.45:0/3243255341 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c06cf40 con 0x7f479c15f4d0 2023-11-28T20:50:06.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.816+0000 7f47a0e26640 1 --2- 172.21.15.45:0/3243255341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c15f4d0 0x7f479c0cf4b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f479c0b40e0 tx=0x7f479402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faeb9a43980f372d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:06.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.816+0000 7f479b7fe640 1 -- 172.21.15.45:0/3243255341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479402ca80 con 0x7f479c15f4d0 2023-11-28T20:50:06.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.816+0000 7f479b7fe640 1 -- 172.21.15.45:0/3243255341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479402cbe0 con 0x7f479c15f4d0 2023-11-28T20:50:06.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.816+0000 7f479b7fe640 1 -- 172.21.15.45:0/3243255341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47940343f0 con 0x7f479c15f4d0 2023-11-28T20:50:06.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.817+0000 7f47a2889640 1 -- 172.21.15.45:0/3243255341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c15f4d0 msgr2=0x7f479c0cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.817+0000 7f47a2889640 1 --2- 172.21.15.45:0/3243255341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c15f4d0 0x7f479c0cf4b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f479c0b40e0 tx=0x7f479402bee0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.817+0000 7f47a2889640 1 -- 172.21.15.45:0/3243255341 shutdown_connections 2023-11-28T20:50:06.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.817+0000 7f47a2889640 1 --2- 172.21.15.45:0/3243255341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c15f4d0 0x7f479c0cf4b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.817+0000 7f47a2889640 1 --2- 172.21.15.45:0/3243255341 >> v2:172.21.15.190:3300/0 conn(0x7f479c15b820 0x7f479c06da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.817+0000 7f47a2889640 1 -- 172.21.15.45:0/3243255341 >> 172.21.15.45:0/3243255341 conn(0x7f479c06a6e0 msgr2=0x7f479c06aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:06.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.817+0000 7f47a2889640 1 -- 172.21.15.45:0/3243255341 shutdown_connections 2023-11-28T20:50:06.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.817+0000 7f47a2889640 1 -- 172.21.15.45:0/3243255341 wait complete. 2023-11-28T20:50:06.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.819+0000 7f47a2889640 1 Processor -- start 2023-11-28T20:50:06.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.819+0000 7f47a2889640 1 -- start start 2023-11-28T20:50:06.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f47a2889640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0cf020 0x7f479c158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f47a2889640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f479c15b820 0x7f479c156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f47a2889640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f479c15f4d0 0x7f479c147880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f47a2889640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c157b80 con 0x7f479c0cf020 2023-11-28T20:50:06.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f47a2889640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c156ca0 con 0x7f479c15f4d0 2023-11-28T20:50:06.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f47a2889640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479c156f70 con 0x7f479c15b820 2023-11-28T20:50:06.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f479bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0cf020 0x7f479c158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f479bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0cf020 0x7f479c158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48904/0 (socket says 172.21.15.45:48904) 2023-11-28T20:50:06.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f479bfff640 1 -- 172.21.15.45:0/1353454559 learned_addr learned my addr 172.21.15.45:0/1353454559 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.820+0000 7f47a0e26640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f479c15f4d0 0x7f479c147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f4793fff640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f479c15b820 0x7f479c156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f479bfff640 1 -- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f479c15b820 msgr2=0x7f479c156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f479bfff640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f479c15b820 0x7f479c156760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f479bfff640 1 -- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f479c15f4d0 msgr2=0x7f479c147880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f479bfff640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f479c15f4d0 0x7f479c147880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f479bfff640 1 -- 172.21.15.45:0/1353454559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4794009b30 con 0x7f479c0cf020 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f479bfff640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0cf020 0x7f479c158010 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f478c00d800 tx=0x7f478c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f47997fa640 1 -- 172.21.15.45:0/1353454559 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c00b6c0 con 0x7f479c0cf020 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f47997fa640 1 -- 172.21.15.45:0/1353454559 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478c00b820 con 0x7f479c0cf020 2023-11-28T20:50:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f47997fa640 1 -- 172.21.15.45:0/1353454559 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c00bbb0 con 0x7f479c0cf020 2023-11-28T20:50:06.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.821+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479c157250 con 0x7f479c0cf020 2023-11-28T20:50:06.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.822+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479c157810 con 0x7f479c0cf020 2023-11-28T20:50:06.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.823+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479c073440 con 0x7f479c0cf020 2023-11-28T20:50:06.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.823+0000 7f47997fa640 1 -- 172.21.15.45:0/1353454559 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f478c00bd10 con 0x7f479c0cf020 2023-11-28T20:50:06.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.824+0000 7f47997fa640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f477c06af60 0x7f477c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:06.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.824+0000 7f47997fa640 1 -- 172.21.15.45:0/1353454559 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f478c08a0c0 con 0x7f479c0cf020 2023-11-28T20:50:06.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.825+0000 7f4793fff640 1 -- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f477c06af60 msgr2=0x7f477c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:06.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.825+0000 7f4793fff640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f477c06af60 0x7f477c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:06.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:06.829+0000 7f47997fa640 1 -- 172.21.15.45:0/1353454559 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f478c054ca0 con 0x7f479c0cf020 2023-11-28T20:50:06.847 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:06.845+0000 7f786b215040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:06.853 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:06.851+0000 7f083f61b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:50:06.954 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:06.952+0000 7f786b215040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:50:06.969 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:06.965+0000 7f083f61b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:07.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.026+0000 7f4793fff640 1 -- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f477c06af60 msgr2=0x7f477c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:07.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.026+0000 7f4793fff640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f477c06af60 0x7f477c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:07.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.035+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f479c06c8c0 con 0x7f479c0cf020 2023-11-28T20:50:07.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.035+0000 7f47997fa640 1 -- 172.21.15.45:0/1353454559 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v76) v1 ==== 80+0+474 (secure 0 0 0) 0x7f478c0582c0 con 0x7f479c0cf020 2023-11-28T20:50:07.037 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2346462975 2023-11-29T20:48:28.490206+0000 2023-11-28T20:50:07.038 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1966056956 2023-11-29T20:48:28.490206+0000 2023-11-28T20:50:07.038 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/838517151 2023-11-29T20:48:28.490206+0000 2023-11-28T20:50:07.038 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/633536721 2023-11-29T20:48:28.490206+0000 2023-11-28T20:50:07.038 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/838517151 2023-11-29T20:48:28.490206+0000 2023-11-28T20:50:07.038 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/4198236319 2023-11-29T20:48:28.490206+0000 2023-11-28T20:50:07.038 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/676976495 2023-11-29T20:48:28.490206+0000 2023-11-28T20:50:07.038 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3224416347 2023-11-29T20:48:28.490206+0000 2023-11-28T20:50:07.038 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2023-11-28T20:50:07.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f477c06af60 msgr2=0x7f477c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:07.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f477c06af60 0x7f477c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0cf020 msgr2=0x7f479c158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:07.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0cf020 0x7f479c158010 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f478c00d800 tx=0x7f478c00dcc0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 shutdown_connections 2023-11-28T20:50:07.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f477c06af60 0x7f477c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f479c15f4d0 0x7f479c147880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f479c15b820 0x7f479c156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 --2- 172.21.15.45:0/1353454559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f479c0cf020 0x7f479c158010 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 >> 172.21.15.45:0/1353454559 conn(0x7f479c06a6e0 msgr2=0x7f479c15ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:07.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 shutdown_connections 2023-11-28T20:50:07.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.038+0000 7f47a2889640 1 -- 172.21.15.45:0/1353454559 wait complete. 2023-11-28T20:50:07.057 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T20:50:07.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:07.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T20:50:07.082 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:07.080+0000 7f083f61b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:50:07.112 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:07.110+0000 7f786b215040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:07.142 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:50:07.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:07.142 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T20:50:07.161 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:07.159+0000 7f786b215040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:07.236 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:50:07.236 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:07.250 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:07.248+0000 7f083f61b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:07.283 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:07.281+0000 7f786b215040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:50:07.290 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:50:07.290 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:50:07.290 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:07.302 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:07.300+0000 7f083f61b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:07.347 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:50:07.347 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:50:07.347 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:07.403 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:50:07.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:07.430 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:07.427+0000 7f083f61b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:50:07.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.617+0000 7fa90bfff640 1 Processor -- start 2023-11-28T20:50:07.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.617+0000 7fa90bfff640 1 -- start start 2023-11-28T20:50:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.618+0000 7fa90bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa90c15d550 0x7fa90c0cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.618+0000 7fa90bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 0x7fa90c0d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.618+0000 7fa90bfff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90c0d0560 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90bfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90c0d2230 con 0x7fa90c15d550 2023-11-28T20:50:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0d2510 con 0x7fa90c0d0100 2023-11-28T20:50:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90affd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa90c15d550 0x7fa90c0cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90affd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa90c15d550 0x7fa90c0cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48920/0 (socket says 172.21.15.45:48920) 2023-11-28T20:50:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90affd640 1 -- 172.21.15.45:0/710635005 learned_addr learned my addr 172.21.15.45:0/710635005 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90b7fe640 1 --1- 172.21.15.45:0/710635005 >> v1:172.21.15.45:6789/0 conn(0x7fa90c0d1ab0 0x7fa90c070b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53716/0 (socket says 172.21.15.45:53716) 2023-11-28T20:50:07.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90a7fc640 1 --2- 172.21.15.45:0/710635005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 0x7fa90c0d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:07.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90affd640 1 -- 172.21.15.45:0/710635005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 msgr2=0x7fa90c0d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:07.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90affd640 1 --2- 172.21.15.45:0/710635005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 0x7fa90c0d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90affd640 1 -- 172.21.15.45:0/710635005 >> v1:172.21.15.45:6789/0 conn(0x7fa90c0d1ab0 legacy=0x7fa90c070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:07.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.619+0000 7fa90affd640 1 -- 172.21.15.45:0/710635005 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c0d27f0 con 0x7fa90c15d550 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.620+0000 7fa90affd640 1 --2- 172.21.15.45:0/710635005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c15d550 0x7fa90c0cfbc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009870 tx=0x7fa8fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b1e8033728a096 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.621+0000 7fa909ffb640 1 -- 172.21.15.45:0/710635005 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc030840 con 0x7fa90c15d550 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.621+0000 7fa909ffb640 1 -- 172.21.15.45:0/710635005 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8fc0309a0 con 0x7fa90c15d550 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.621+0000 7fa909ffb640 1 -- 172.21.15.45:0/710635005 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc030d30 con 0x7fa90c15d550 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.621+0000 7fa90bfff640 1 -- 172.21.15.45:0/710635005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c15d550 msgr2=0x7fa90c0cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.621+0000 7fa90bfff640 1 --2- 172.21.15.45:0/710635005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c15d550 0x7fa90c0cfbc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009870 tx=0x7fa8fc02b980 comp rx=0 tx=0).stop 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.622+0000 7fa90bfff640 1 -- 172.21.15.45:0/710635005 shutdown_connections 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.622+0000 7fa90bfff640 1 --2- 172.21.15.45:0/710635005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 0x7fa90c0d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.622+0000 7fa90bfff640 1 --2- 172.21.15.45:0/710635005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c15d550 0x7fa90c0cfbc0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.622+0000 7fa90bfff640 1 -- 172.21.15.45:0/710635005 >> 172.21.15.45:0/710635005 conn(0x7fa90c06ac70 msgr2=0x7fa90c0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.622+0000 7fa90bfff640 1 -- 172.21.15.45:0/710635005 shutdown_connections 2023-11-28T20:50:07.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.622+0000 7fa90bfff640 1 -- 172.21.15.45:0/710635005 wait complete. 2023-11-28T20:50:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.623+0000 7fa90bfff640 1 Processor -- start 2023-11-28T20:50:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.624+0000 7fa90bfff640 1 -- start start 2023-11-28T20:50:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.624+0000 7fa90bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 0x7fa90c1f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c0d1ab0 0x7fa90c145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa90c15d550 0x7fa90c155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90bfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c155100 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90bfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0c7d50 con 0x7fa90c15d550 2023-11-28T20:50:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c146500 con 0x7fa90c0d0100 2023-11-28T20:50:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c0d1ab0 0x7fa90c145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c0d1ab0 0x7fa90c145fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48926/0 (socket says 172.21.15.45:48926) 2023-11-28T20:50:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90a7fc640 1 -- 172.21.15.45:0/2968814792 learned_addr learned my addr 172.21.15.45:0/2968814792 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90b7fe640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa90c15d550 0x7fa90c155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90affd640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 0x7fa90c1f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90a7fc640 1 -- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 msgr2=0x7fa90c1f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90a7fc640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 0x7fa90c1f63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90a7fc640 1 -- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa90c15d550 msgr2=0x7fa90c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90a7fc640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa90c15d550 0x7fa90c155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.625+0000 7fa90a7fc640 1 -- 172.21.15.45:0/2968814792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc0094d0 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.626+0000 7fa90a7fc640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c0d1ab0 0x7fa90c145fc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400d800 tx=0x7fa8f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.626+0000 7fa9037fe640 1 -- 172.21.15.45:0/2968814792 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f400b6c0 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.626+0000 7fa9037fe640 1 -- 172.21.15.45:0/2968814792 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f400b820 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.626+0000 7fa9037fe640 1 -- 172.21.15.45:0/2968814792 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f400bbb0 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.626+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90c155b40 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.626+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90c1560e0 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.627+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90c04e920 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.629+0000 7fa9037fe640 1 -- 172.21.15.45:0/2968814792 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 89345+0+0 (secure 0 0 0) 0x7fa8f400bd10 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.630+0000 7fa9037fe640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fa8ec06af90 0x7fa8ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.630+0000 7fa9037fe640 1 -- 172.21.15.45:0/2968814792 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fa8f4089ff0 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.630+0000 7fa90affd640 1 -- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fa8ec06af90 msgr2=0x7fa8ec06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.630+0000 7fa90affd640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fa8ec06af90 0x7fa8ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.635+0000 7fa9037fe640 1 -- 172.21.15.45:0/2968814792 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8f4054c70 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.831+0000 7fa90affd640 1 -- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fa8ec06af90 msgr2=0x7fa8ec06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:07.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.831+0000 7fa90affd640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fa8ec06af90 0x7fa8ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:07.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.870+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa90c15c000 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.870+0000 7fa9037fe640 1 -- 172.21.15.45:0/2968814792 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7fa8f4058290 con 0x7fa90c0d1ab0 2023-11-28T20:50:07.873 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:07.873 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5183,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/285399852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":285399852},{"type":"v1","addr":"172.21.15.190:6836","nonce":285399852}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2028545354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2028545354},{"type":"v1","addr":"172.21.15.190:6837","nonce":2028545354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6216,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/784320624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":784320624},{"type":"v1","addr":"172.21.15.45:6834","nonce":784320624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:48:13.742492+0000","modified":"2023-11-28T20:48:16.820308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3717491008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3717491008},{"type":"v1","addr":"172.21.15.45:6835","nonce":3717491008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T20:50:07.873 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 27 2023-11-28T20:50:07.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fa8ec06af90 msgr2=0x7fa8ec06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:07.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fa8ec06af90 0x7fa8ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c0d1ab0 msgr2=0x7fa90c145fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c0d1ab0 0x7fa90c145fc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400d800 tx=0x7fa8f400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 shutdown_connections 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7fa8ec06af90 0x7fa8ec06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa90c15d550 0x7fa90c155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa90c0d1ab0 0x7fa90c145fc0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 --2- 172.21.15.45:0/2968814792 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa90c0d0100 0x7fa90c1f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 >> 172.21.15.45:0/2968814792 conn(0x7fa90c06ac70 msgr2=0x7fa90c15f620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 shutdown_connections 2023-11-28T20:50:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:07.873+0000 7fa90bfff640 1 -- 172.21.15.45:0/2968814792 wait complete. 2023-11-28T20:50:07.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:08.129 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.127+0000 7f786b215040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:50:08.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.135+0000 7f1bd730f640 1 Processor -- start 2023-11-28T20:50:08.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.135+0000 7f1bd730f640 1 -- start start 2023-11-28T20:50:08.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.137+0000 7f1bd730f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1bd0160e70 0x7f1bd01434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.137+0000 7f1bd730f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd00703b0 0x7f1bd0070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.137+0000 7f1bd730f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd0161df0 con 0x7f1bd01402f0 2023-11-28T20:50:08.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.137+0000 7f1bd730f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd0143a00 con 0x7f1bd0160e70 2023-11-28T20:50:08.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.137+0000 7f1bd730f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0161300 con 0x7f1bd00703b0 2023-11-28T20:50:08.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.137+0000 7f1bd50ab640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1bd01402f0 0x7f1bd01409f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53720/0 (socket says 172.21.15.45:53720) 2023-11-28T20:50:08.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.137+0000 7f1bd50ab640 1 -- 172.21.15.45:0/1794948249 learned_addr learned my addr 172.21.15.45:0/1794948249 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:08.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bd48aa640 1 --2- 172.21.15.45:0/1794948249 >> v2:172.21.15.190:3300/0 conn(0x7f1bd0160e70 0x7f1bd01434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bd58ac640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd00703b0 0x7f1bd0070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213812848 0 0) 0x7f1bd0161df0 con 0x7f1bd01402f0 2023-11-28T20:50:08.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc003520 con 0x7f1bd01402f0 2023-11-28T20:50:08.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898248936 0 0) 0x7f1bbc003520 con 0x7f1bd01402f0 2023-11-28T20:50:08.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd00703b0 msgr2=0x7f1bd0070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:08.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd00703b0 0x7f1bd0070840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 >> v2:172.21.15.190:3300/0 conn(0x7f1bd0160e70 msgr2=0x7f1bd01434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 --2- 172.21.15.45:0/1794948249 >> v2:172.21.15.190:3300/0 conn(0x7f1bd0160e70 0x7f1bd01434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd01406d0 con 0x7f1bd01402f0 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f1bcc001670 con 0x7f1bd01402f0 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 >> v1:172.21.15.45:6789/0 conn(0x7f1bd01402f0 legacy=0x7f1bd01409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.138+0000 7f1bc7fff640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc003970 0x7f1bbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bc7fff640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bbc0047c0 0x7f1bbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bc7fff640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd00703b0 0x7f1bbc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0143a00 con 0x7f1bbc0047c0 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0161300 con 0x7f1bbc003970 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc00ba00 con 0x7f1bd00703b0 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bd48aa640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bbc0047c0 0x7f1bbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bd50ab640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc003970 0x7f1bbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bd58ac640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd00703b0 0x7f1bbc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bd48aa640 1 -- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd00703b0 msgr2=0x7f1bbc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:08.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bd48aa640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd00703b0 0x7f1bbc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bd48aa640 1 -- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc003970 msgr2=0x7f1bbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:08.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bd48aa640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc003970 0x7f1bbc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bd48aa640 1 -- 172.21.15.45:0/1794948249 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc003520 con 0x7f1bbc0047c0 2023-11-28T20:50:08.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.139+0000 7f1bd50ab640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc003970 0x7f1bbc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:08.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.140+0000 7f1bd48aa640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bbc0047c0 0x7f1bbc004c40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009ac0 tx=0x7f1bbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbcdf703aef44795 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:08.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.141+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc002c710 con 0x7f1bbc0047c0 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.141+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc002c870 con 0x7f1bbc0047c0 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.141+0000 7f1bc7fff640 1 -- 172.21.15.45:0/1794948249 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc002cc00 con 0x7f1bbc0047c0 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.141+0000 7f1bd730f640 1 -- 172.21.15.45:0/1794948249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bbc0047c0 msgr2=0x7f1bbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.141+0000 7f1bd730f640 1 --2- 172.21.15.45:0/1794948249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bbc0047c0 0x7f1bbc004c40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009ac0 tx=0x7f1bbc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.141+0000 7f1bd58ac640 1 -- 172.21.15.45:0/1794948249 reap_dead start 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.141+0000 7f1bd730f640 1 -- 172.21.15.45:0/1794948249 shutdown_connections 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.141+0000 7f1bd730f640 1 -- 172.21.15.45:0/1794948249 >> 172.21.15.45:0/1794948249 conn(0x7f1bd006e7c0 msgr2=0x7f1bd006ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.142+0000 7f1bd730f640 1 -- 172.21.15.45:0/1794948249 shutdown_connections 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.142+0000 7f1bd730f640 1 -- 172.21.15.45:0/1794948249 wait complete. 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.143+0000 7f1bd730f640 1 Processor -- start 2023-11-28T20:50:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.143+0000 7f1bd730f640 1 -- start start 2023-11-28T20:50:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.144+0000 7f1bd730f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc0047c0 0x7f1bd015eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.144+0000 7f1bd730f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bd0139bb0 0x7f1bd015bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.144+0000 7f1bd730f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd015c470 0x7f1bd00b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.144+0000 7f1bd730f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd015da90 con 0x7f1bd0139bb0 2023-11-28T20:50:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.144+0000 7f1bd730f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd015dd70 con 0x7f1bbc0047c0 2023-11-28T20:50:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.144+0000 7f1bd730f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd015e050 con 0x7f1bd015c470 2023-11-28T20:50:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd48aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bd0139bb0 0x7f1bd015bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd48aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bd0139bb0 0x7f1bd015bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48942/0 (socket says 172.21.15.45:48942) 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd48aa640 1 -- 172.21.15.45:0/3216597838 learned_addr learned my addr 172.21.15.45:0/3216597838 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd50ab640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc0047c0 0x7f1bd015eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd58ac640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd015c470 0x7f1bd00b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd48aa640 1 -- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd015c470 msgr2=0x7f1bd00b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd48aa640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd015c470 0x7f1bd00b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd48aa640 1 -- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc0047c0 msgr2=0x7f1bd015eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd48aa640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc0047c0 0x7f1bd015eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd48aa640 1 -- 172.21.15.45:0/3216597838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd015e330 con 0x7f1bd0139bb0 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.145+0000 7f1bd48aa640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bd0139bb0 0x7f1bd015bf30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0037040 tx=0x7f1bc0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.146+0000 7f1bc5ffb640 1 -- 172.21.15.45:0/3216597838 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0003810 con 0x7f1bd0139bb0 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.146+0000 7f1bc5ffb640 1 -- 172.21.15.45:0/3216597838 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc0003970 con 0x7f1bd0139bb0 2023-11-28T20:50:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.146+0000 7f1bc5ffb640 1 -- 172.21.15.45:0/3216597838 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0003d00 con 0x7f1bd0139bb0 2023-11-28T20:50:08.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.146+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd015e610 con 0x7f1bd0139bb0 2023-11-28T20:50:08.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.146+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd013a030 con 0x7f1bd0139bb0 2023-11-28T20:50:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.147+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd00733d0 con 0x7f1bd0139bb0 2023-11-28T20:50:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.149+0000 7f1bc5ffb640 1 -- 172.21.15.45:0/3216597838 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f1bc0040450 con 0x7f1bd0139bb0 2023-11-28T20:50:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.150+0000 7f1bc5ffb640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f1bb406af90 0x7f1bb406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.150+0000 7f1bc5ffb640 1 -- 172.21.15.45:0/3216597838 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f1bc00abab0 con 0x7f1bd0139bb0 2023-11-28T20:50:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.150+0000 7f1bd50ab640 1 -- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f1bb406af90 msgr2=0x7f1bb406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.150+0000 7f1bd50ab640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f1bb406af90 0x7f1bb406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:08.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.154+0000 7f1bc5ffb640 1 -- 172.21.15.45:0/3216597838 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bc0076690 con 0x7f1bd0139bb0 2023-11-28T20:50:08.292 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:08.290+0000 7f083f61b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:50:08.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.348+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bd00d1d80 con 0x7f1bd0139bb0 2023-11-28T20:50:08.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.349+0000 7f1bc5ffb640 1 -- 172.21.15.45:0/3216597838 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+17437 (secure 0 0 0) 0x7f1bc0003e60 con 0x7f1bd0139bb0 2023-11-28T20:50:08.353 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:08.353 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":76,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:05.390085+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:48:11.673148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-28T20:48:12.760782+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/2346462975":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/1966056956":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6832/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/633536721":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6833/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/4198236319":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/676976495":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/3224416347":"2023-11-29T20:48:28.490206+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:08.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.351+0000 7f1bd50ab640 1 -- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f1bb406af90 msgr2=0x7f1bb406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:08.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.351+0000 7f1bd50ab640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f1bb406af90 0x7f1bb406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:08.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f1bb406af90 msgr2=0x7f1bb406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f1bb406af90 0x7f1bb406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bd0139bb0 msgr2=0x7f1bd015bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bd0139bb0 0x7f1bd015bf30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0037040 tx=0x7f1bc0004730 comp rx=0 tx=0).stop 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 shutdown_connections 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f1bb406af90 0x7f1bb406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bd015c470 0x7f1bd00b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bd0139bb0 0x7f1bd015bf30 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 --2- 172.21.15.45:0/3216597838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bbc0047c0 0x7f1bd015eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 >> 172.21.15.45:0/3216597838 conn(0x7f1bd006e7c0 msgr2=0x7f1bd0145d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:08.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 shutdown_connections 2023-11-28T20:50:08.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.352+0000 7f1bd730f640 1 -- 172.21.15.45:0/3216597838 wait complete. 2023-11-28T20:50:08.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:48:11.673148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:48:12.760782+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 76, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:50:08.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:50:08.452 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.450+0000 7f786b215040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:50:08.510 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.508+0000 7f786b215040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:50:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.588+0000 7f20af4bb640 1 Processor -- start 2023-11-28T20:50:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.589+0000 7f20af4bb640 1 -- start start 2023-11-28T20:50:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20af4bb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f20a80d0040 0x7f20a80d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20af4bb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f20a80b2290 0x7f20a80d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20af4bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80b1d40 0x7f20a80d3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20af4bb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a80b90a0 con 0x7f20a80b2290 2023-11-28T20:50:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20af4bb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a80d1920 con 0x7f20a80d0040 2023-11-28T20:50:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20af4bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80d0960 con 0x7f20a80b1d40 2023-11-28T20:50:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20adcb8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f20a80b2290 0x7f20a80d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20adcb8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f20a80b2290 0x7f20a80d3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48950/0 (socket says 172.21.15.45:48950) 2023-11-28T20:50:08.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20adcb8640 1 -- 172.21.15.45:0/64764760 learned_addr learned my addr 172.21.15.45:0/64764760 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:08.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20ae4b9640 1 --2- 172.21.15.45:0/64764760 >> v2:172.21.15.190:3300/0 conn(0x7f20a80d0040 0x7f20a80d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.590+0000 7f20aecba640 1 --2- 172.21.15.45:0/64764760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80b1d40 0x7f20a80d3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.591+0000 7f20adcb8640 1 -- 172.21.15.45:0/64764760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80b1d40 msgr2=0x7f20a80d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.591+0000 7f20adcb8640 1 --2- 172.21.15.45:0/64764760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80b1d40 0x7f20a80d3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.591+0000 7f20adcb8640 1 -- 172.21.15.45:0/64764760 >> v2:172.21.15.190:3300/0 conn(0x7f20a80d0040 msgr2=0x7f20a80d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.591+0000 7f20adcb8640 1 --2- 172.21.15.45:0/64764760 >> v2:172.21.15.190:3300/0 conn(0x7f20a80d0040 0x7f20a80d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.591+0000 7f20adcb8640 1 -- 172.21.15.45:0/64764760 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a80b8ae0 con 0x7f20a80b2290 2023-11-28T20:50:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.592+0000 7f20adcb8640 1 --2- 172.21.15.45:0/64764760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 0x7f20a80d3040 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f20a80b53c0 tx=0x7f209802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28206aba2ffcb439 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.592+0000 7f20ad4b7640 1 -- 172.21.15.45:0/64764760 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098030860 con 0x7f20a80b2290 2023-11-28T20:50:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.592+0000 7f20ad4b7640 1 -- 172.21.15.45:0/64764760 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20980309c0 con 0x7f20a80b2290 2023-11-28T20:50:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.592+0000 7f20ad4b7640 1 -- 172.21.15.45:0/64764760 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098030d50 con 0x7f20a80b2290 2023-11-28T20:50:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.593+0000 7f20af4bb640 1 -- 172.21.15.45:0/64764760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 msgr2=0x7f20a80d3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.593+0000 7f20af4bb640 1 --2- 172.21.15.45:0/64764760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 0x7f20a80d3040 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f20a80b53c0 tx=0x7f209802b9a0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.593+0000 7f20af4bb640 1 -- 172.21.15.45:0/64764760 shutdown_connections 2023-11-28T20:50:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.593+0000 7f20af4bb640 1 --2- 172.21.15.45:0/64764760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80b1d40 0x7f20a80d3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.593+0000 7f20af4bb640 1 --2- 172.21.15.45:0/64764760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 0x7f20a80d3040 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.593+0000 7f20af4bb640 1 --2- 172.21.15.45:0/64764760 >> v2:172.21.15.190:3300/0 conn(0x7f20a80d0040 0x7f20a80d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.593+0000 7f20af4bb640 1 -- 172.21.15.45:0/64764760 >> 172.21.15.45:0/64764760 conn(0x7f20a8070f40 msgr2=0x7f20a8071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.593+0000 7f20af4bb640 1 -- 172.21.15.45:0/64764760 shutdown_connections 2023-11-28T20:50:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.593+0000 7f20af4bb640 1 -- 172.21.15.45:0/64764760 wait complete. 2023-11-28T20:50:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.595+0000 7f20af4bb640 1 Processor -- start 2023-11-28T20:50:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.595+0000 7f20af4bb640 1 -- start start 2023-11-28T20:50:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20af4bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a80b1d40 0x7f20a80c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20af4bb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 0x7f20a815b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20af4bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80d0040 0x7f20a81e4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20af4bb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80c8140 con 0x7f20a80b2290 2023-11-28T20:50:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20af4bb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80c74c0 con 0x7f20a80b1d40 2023-11-28T20:50:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20af4bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80c77a0 con 0x7f20a80d0040 2023-11-28T20:50:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20adcb8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 0x7f20a815b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20adcb8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 0x7f20a815b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48958/0 (socket says 172.21.15.45:48958) 2023-11-28T20:50:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20adcb8640 1 -- 172.21.15.45:0/3272367403 learned_addr learned my addr 172.21.15.45:0/3272367403 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20ae4b9640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a80b1d40 0x7f20a80c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20aecba640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80d0040 0x7f20a81e4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20adcb8640 1 -- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80d0040 msgr2=0x7f20a81e4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20adcb8640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80d0040 0x7f20a81e4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20adcb8640 1 -- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a80b1d40 msgr2=0x7f20a80c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.596+0000 7f20adcb8640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a80b1d40 0x7f20a80c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.597+0000 7f20adcb8640 1 -- 172.21.15.45:0/3272367403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980094d0 con 0x7f20a80b2290 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.597+0000 7f20aecba640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80d0040 0x7f20a81e4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.597+0000 7f20adcb8640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 0x7f20a815b060 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f209802b9e0 tx=0x7f2098004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.597+0000 7f209effd640 1 -- 172.21.15.45:0/3272367403 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098039d60 con 0x7f20a80b2290 2023-11-28T20:50:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.597+0000 7f209effd640 1 -- 172.21.15.45:0/3272367403 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2098043400 con 0x7f20a80b2290 2023-11-28T20:50:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.597+0000 7f20af4bb640 1 -- 172.21.15.45:0/3272367403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a80c7a80 con 0x7f20a80b2290 2023-11-28T20:50:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.597+0000 7f209effd640 1 -- 172.21.15.45:0/3272367403 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20980437b0 con 0x7f20a80b2290 2023-11-28T20:50:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.597+0000 7f20af4bb640 1 -- 172.21.15.45:0/3272367403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a81e4ec0 con 0x7f20a80b2290 2023-11-28T20:50:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.599+0000 7f20af4bb640 1 -- 172.21.15.45:0/3272367403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a8071b30 con 0x7f20a80b2290 2023-11-28T20:50:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.600+0000 7f209effd640 1 -- 172.21.15.45:0/3272367403 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f2098043910 con 0x7f20a80b2290 2023-11-28T20:50:08.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.600+0000 7f209effd640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f208c06af60 0x7f208c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:08.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.601+0000 7f209effd640 1 -- 172.21.15.45:0/3272367403 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f209803e070 con 0x7f20a80b2290 2023-11-28T20:50:08.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.601+0000 7f20ae4b9640 1 -- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f208c06af60 msgr2=0x7f208c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/979211100 2023-11-28T20:50:08.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.601+0000 7f20ae4b9640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f208c06af60 0x7f208c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:08.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.605+0000 7f209effd640 1 -- 172.21.15.45:0/3272367403 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2098077ad0 con 0x7f20a80b2290 2023-11-28T20:50:08.616 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:08.614+0000 7f083f61b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:50:08.625 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.623+0000 7f786b215040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:50:08.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.642+0000 7f209effd640 1 -- 172.21.15.45:0/3272367403 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 27) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f20980ab040 con 0x7f20a80b2290 2023-11-28T20:50:08.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.642+0000 7f209effd640 1 -- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f208c06af60 msgr2=0x7f208c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:08.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.642+0000 7f209effd640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f208c06af60 0x7f208c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.684 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:08.682+0000 7f083f61b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:50:08.715 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.713+0000 7f77ef8de640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.715 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.713+0000 7f77ef8de640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.715 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.713+0000 7f77ef8de640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.716 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.713+0000 7f77ef8de640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.716 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.713+0000 7f77ef8de640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.717 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.715+0000 7f77ed0d9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.717 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.715+0000 7f77ed0d9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.717 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.715+0000 7f77ed0d9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.717 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.715+0000 7f77ed0d9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.717 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:08.715+0000 7f77ed0d9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:08.808 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:08.806+0000 7f083f61b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:50:08.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.808+0000 7f20af4bb640 1 -- 172.21.15.45:0/3272367403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f20a81e4760 con 0x7f20a80b2290 2023-11-28T20:50:08.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.808+0000 7f209effd640 1 -- 172.21.15.45:0/3272367403 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f209804f280 con 0x7f20a80b2290 2023-11-28T20:50:08.810 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:50:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 -- 172.21.15.45:0/3272367403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 msgr2=0x7f20a815b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 0x7f20a815b060 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f209802b9e0 tx=0x7f2098004730 comp rx=0 tx=0).stop 2023-11-28T20:50:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 -- 172.21.15.45:0/3272367403 shutdown_connections 2023-11-28T20:50:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:6832/979211100,v1:172.21.15.190:6833/979211100] conn(0x7f208c06af60 0x7f208c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20a80d0040 0x7f20a81e4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20a80b2290 0x7f20a815b060 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 --2- 172.21.15.45:0/3272367403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a80b1d40 0x7f20a80c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 -- 172.21.15.45:0/3272367403 >> 172.21.15.45:0/3272367403 conn(0x7f20a8070f40 msgr2=0x7f20a80ce150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 -- 172.21.15.45:0/3272367403 shutdown_connections 2023-11-28T20:50:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:08.811+0000 7f20af4bb640 1 -- 172.21.15.45:0/3272367403 wait complete. 2023-11-28T20:50:08.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:50:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.034+0000 7fc34c98a640 1 Processor -- start 2023-11-28T20:50:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.035+0000 7fc34c98a640 1 -- start start 2023-11-28T20:50:09.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.036+0000 7fc34c98a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc34815e840 0x7fc34806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.036+0000 7fc34c98a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34806ac10 0x7fc3480717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.036+0000 7fc34c98a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc34815ecd0 con 0x7fc34815f720 2023-11-28T20:50:09.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.036+0000 7fc34c98a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc348071ce0 con 0x7fc34815e840 2023-11-28T20:50:09.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.036+0000 7fc34c98a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34815f290 con 0x7fc34806ac10 2023-11-28T20:50:09.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.036+0000 7fc33ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc34815f720 0x7fc34806a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53726/0 (socket says 172.21.15.45:53726) 2023-11-28T20:50:09.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.036+0000 7fc33ffff640 1 -- 172.21.15.45:0/1973131564 learned_addr learned my addr 172.21.15.45:0/1973131564 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:09.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc34659c640 1 --2- 172.21.15.45:0/1973131564 >> v2:172.21.15.190:3300/0 conn(0x7fc34815e840 0x7fc34806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc346d9d640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34806ac10 0x7fc3480717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052254670 0 0) 0x7fc34815ecd0 con 0x7fc34815f720 2023-11-28T20:50:09.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc334003520 con 0x7fc34815f720 2023-11-28T20:50:09.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1683208931 0 0) 0x7fc334003520 con 0x7fc34815f720 2023-11-28T20:50:09.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34806ac10 msgr2=0x7fc3480717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:09.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc345d9b640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34806ac10 0x7fc3480717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 >> v2:172.21.15.190:3300/0 conn(0x7fc34815e840 msgr2=0x7fc34806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:09.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc345d9b640 1 --2- 172.21.15.45:0/1973131564 >> v2:172.21.15.190:3300/0 conn(0x7fc34815e840 0x7fc34806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34815fb60 con 0x7fc34815f720 2023-11-28T20:50:09.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.037+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc330001670 con 0x7fc34815f720 2023-11-28T20:50:09.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 >> v1:172.21.15.45:6789/0 conn(0x7fc34815f720 legacy=0x7fc34806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc345d9b640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334003970 0x7fc334003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc345d9b640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34806ac10 0x7fc3340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc345d9b640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc334004d00 0x7fc334005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34815f290 con 0x7fc334004d00 2023-11-28T20:50:09.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33400b980 con 0x7fc334003970 2023-11-28T20:50:09.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33400bac0 con 0x7fc34806ac10 2023-11-28T20:50:09.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc346d9d640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc334004d00 0x7fc334005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc33ffff640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34806ac10 0x7fc3340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc34659c640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334003970 0x7fc334003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc346d9d640 1 -- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34806ac10 msgr2=0x7fc3340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:09.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc346d9d640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34806ac10 0x7fc3340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc346d9d640 1 -- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334003970 msgr2=0x7fc334003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:09.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc346d9d640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334003970 0x7fc334003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc346d9d640 1 -- 172.21.15.45:0/1973131564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334003520 con 0x7fc334004d00 2023-11-28T20:50:09.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.038+0000 7fc34659c640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334003970 0x7fc334003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:09.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.040+0000 7fc346d9d640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc334004d00 0x7fc334005180 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc34000e170 tx=0x7fc3340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6217995c7025ad server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:09.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.040+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34002cbf0 con 0x7fc334004d00 2023-11-28T20:50:09.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.040+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34000f040 con 0x7fc334004d00 2023-11-28T20:50:09.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.040+0000 7fc345d9b640 1 -- 172.21.15.45:0/1973131564 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc340034440 con 0x7fc334004d00 2023-11-28T20:50:09.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.040+0000 7fc34c98a640 1 -- 172.21.15.45:0/1973131564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc334004d00 msgr2=0x7fc334005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.040+0000 7fc34c98a640 1 --2- 172.21.15.45:0/1973131564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc334004d00 0x7fc334005180 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc34000e170 tx=0x7fc3340057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.040+0000 7fc346d9d640 1 -- 172.21.15.45:0/1973131564 reap_dead start 2023-11-28T20:50:09.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.040+0000 7fc34c98a640 1 -- 172.21.15.45:0/1973131564 shutdown_connections 2023-11-28T20:50:09.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.040+0000 7fc34c98a640 1 -- 172.21.15.45:0/1973131564 >> 172.21.15.45:0/1973131564 conn(0x7fc3480ced60 msgr2=0x7fc3480cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:09.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.041+0000 7fc34c98a640 1 -- 172.21.15.45:0/1973131564 shutdown_connections 2023-11-28T20:50:09.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.041+0000 7fc34c98a640 1 -- 172.21.15.45:0/1973131564 wait complete. 2023-11-28T20:50:09.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.042+0000 7fc34c98a640 1 Processor -- start 2023-11-28T20:50:09.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.043+0000 7fc34c98a640 1 -- start start 2023-11-28T20:50:09.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.043+0000 7fc34c98a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334004d00 0x7fc348159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.043+0000 7fc34c98a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3480c4ea0 0x7fc3480c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc34c98a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3480c8000 0x7fc3480c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc34c98a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3480c89d0 con 0x7fc3480c4ea0 2023-11-28T20:50:09.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc34c98a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3480c3a20 con 0x7fc334004d00 2023-11-28T20:50:09.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc34c98a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3481f31a0 con 0x7fc3480c8000 2023-11-28T20:50:09.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc33ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3480c4ea0 0x7fc3480c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc33ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3480c4ea0 0x7fc3480c52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48968/0 (socket says 172.21.15.45:48968) 2023-11-28T20:50:09.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc33ffff640 1 -- 172.21.15.45:0/3142626555 learned_addr learned my addr 172.21.15.45:0/3142626555 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:09.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc346d9d640 1 --2- 172.21.15.45:0/3142626555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3480c8000 0x7fc3480c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc33ffff640 1 -- 172.21.15.45:0/3142626555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3480c8000 msgr2=0x7fc3480c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc34659c640 1 --2- 172.21.15.45:0/3142626555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334004d00 0x7fc348159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc33ffff640 1 --2- 172.21.15.45:0/3142626555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3480c8000 0x7fc3480c8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc33ffff640 1 -- 172.21.15.45:0/3142626555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334004d00 msgr2=0x7fc348159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc33ffff640 1 --2- 172.21.15.45:0/3142626555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334004d00 0x7fc348159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.044+0000 7fc33ffff640 1 -- 172.21.15.45:0/3142626555 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3481f3480 con 0x7fc3480c4ea0 2023-11-28T20:50:09.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.045+0000 7fc33ffff640 1 --2- 172.21.15.45:0/3142626555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3480c4ea0 0x7fc3480c52e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc33000e7b0 tx=0x7fc33000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:09.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.045+0000 7fc33effd640 1 -- 172.21.15.45:0/3142626555 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33000eec0 con 0x7fc3480c4ea0 2023-11-28T20:50:09.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.045+0000 7fc33effd640 1 -- 172.21.15.45:0/3142626555 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33000c8d0 con 0x7fc3480c4ea0 2023-11-28T20:50:09.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.045+0000 7fc34c98a640 1 -- 172.21.15.45:0/3142626555 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3481f3760 con 0x7fc3480c4ea0 2023-11-28T20:50:09.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.045+0000 7fc33effd640 1 -- 172.21.15.45:0/3142626555 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33000cc80 con 0x7fc3480c4ea0 2023-11-28T20:50:09.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.045+0000 7fc34c98a640 1 -- 172.21.15.45:0/3142626555 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3481f3d20 con 0x7fc3480c4ea0 2023-11-28T20:50:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.046+0000 7fc34c98a640 1 -- 172.21.15.45:0/3142626555 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3481f4010 con 0x7fc3480c4ea0 2023-11-28T20:50:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.047+0000 7fc33effd640 1 -- 172.21.15.45:0/3142626555 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fc33001d460 con 0x7fc3480c4ea0 2023-11-28T20:50:09.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.048+0000 7fc33effd640 1 -- 172.21.15.45:0/3142626555 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6803+0+0 (secure 0 0 0) 0x7fc330089570 con 0x7fc3480c4ea0 2023-11-28T20:50:09.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.053+0000 7fc33effd640 1 -- 172.21.15.45:0/3142626555 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3481f4010 con 0x7fc3480c4ea0 2023-11-28T20:50:09.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.250+0000 7fc34c98a640 1 -- 172.21.15.45:0/3142626555 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc3481f4010 con 0x7fc3480c4ea0 2023-11-28T20:50:09.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.251+0000 7fc33effd640 1 -- 172.21.15.45:0/3142626555 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc3481f4010 con 0x7fc3480c4ea0 2023-11-28T20:50:09.253 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:50:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.254+0000 7fc34c98a640 1 -- 172.21.15.45:0/3142626555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3480c4ea0 msgr2=0x7fc3480c52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.254+0000 7fc34c98a640 1 --2- 172.21.15.45:0/3142626555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3480c4ea0 0x7fc3480c52e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc33000e7b0 tx=0x7fc33000ec70 comp rx=0 tx=0).stop 2023-11-28T20:50:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.254+0000 7fc34c98a640 1 -- 172.21.15.45:0/3142626555 shutdown_connections 2023-11-28T20:50:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.254+0000 7fc34c98a640 1 --2- 172.21.15.45:0/3142626555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3480c8000 0x7fc3480c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.254+0000 7fc34c98a640 1 --2- 172.21.15.45:0/3142626555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3480c4ea0 0x7fc3480c52e0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.254+0000 7fc34c98a640 1 --2- 172.21.15.45:0/3142626555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334004d00 0x7fc348159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.254+0000 7fc34c98a640 1 -- 172.21.15.45:0/3142626555 >> 172.21.15.45:0/3142626555 conn(0x7fc3480ced60 msgr2=0x7fc3480ac6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.254+0000 7fc34c98a640 1 -- 172.21.15.45:0/3142626555 shutdown_connections 2023-11-28T20:50:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.254+0000 7fc34c98a640 1 -- 172.21.15.45:0/3142626555 wait complete. 2023-11-28T20:50:09.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:50:09.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.503+0000 7f200ffff640 1 Processor -- start 2023-11-28T20:50:09.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.503+0000 7f200ffff640 1 -- start start 2023-11-28T20:50:09.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.504+0000 7f200ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f20100d1e40 0x7f20100ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.504+0000 7f200ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20100b2500 0x7f20100ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.504+0000 7f200ffff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20100d0210 con 0x7f201015d580 2023-11-28T20:50:09.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.504+0000 7f200ffff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20100d0490 con 0x7f20100d1e40 2023-11-28T20:50:09.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.504+0000 7f200ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20100d1360 con 0x7f20100b2500 2023-11-28T20:50:09.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.505+0000 7f200e7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f20100d1e40 0x7f20100ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.505+0000 7f200e7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f20100d1e40 0x7f20100ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48974/0 (socket says 172.21.15.45:48974) 2023-11-28T20:50:09.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.505+0000 7f200e7fc640 1 -- 172.21.15.45:0/1136442109 learned_addr learned my addr 172.21.15.45:0/1136442109 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:09.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.505+0000 7f200f7fe640 1 --2- 172.21.15.45:0/1136442109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20100b2500 0x7f20100ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.505+0000 7f200e7fc640 1 -- 172.21.15.45:0/1136442109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20100b2500 msgr2=0x7f20100ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:09.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.505+0000 7f200e7fc640 1 --2- 172.21.15.45:0/1136442109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20100b2500 0x7f20100ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.505+0000 7f200e7fc640 1 -- 172.21.15.45:0/1136442109 >> v1:172.21.15.190:6789/0 conn(0x7f201015d580 legacy=0x7f20100cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:09.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.505+0000 7f200e7fc640 1 -- 172.21.15.45:0/1136442109 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20100d1640 con 0x7f20100d1e40 2023-11-28T20:50:09.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.506+0000 7f200e7fc640 1 --2- 172.21.15.45:0/1136442109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 0x7f20100ce040 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009ea0 tx=0x7f1ff802be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f2740b3379d434 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:09.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.507+0000 7f200dffb640 1 -- 172.21.15.45:0/1136442109 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff802ca20 con 0x7f20100d1e40 2023-11-28T20:50:09.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.507+0000 7f200dffb640 1 -- 172.21.15.45:0/1136442109 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff802cb80 con 0x7f20100d1e40 2023-11-28T20:50:09.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.507+0000 7f200dffb640 1 -- 172.21.15.45:0/1136442109 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff80343f0 con 0x7f20100d1e40 2023-11-28T20:50:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.507+0000 7f200ffff640 1 -- 172.21.15.45:0/1136442109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 msgr2=0x7f20100ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.507+0000 7f200ffff640 1 --2- 172.21.15.45:0/1136442109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 0x7f20100ce040 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009ea0 tx=0x7f1ff802be80 comp rx=0 tx=0).stop 2023-11-28T20:50:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.507+0000 7f200ffff640 1 -- 172.21.15.45:0/1136442109 shutdown_connections 2023-11-28T20:50:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.507+0000 7f200ffff640 1 --2- 172.21.15.45:0/1136442109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20100b2500 0x7f20100ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.507+0000 7f200ffff640 1 --2- 172.21.15.45:0/1136442109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 0x7f20100ce040 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.507+0000 7f200ffff640 1 -- 172.21.15.45:0/1136442109 >> 172.21.15.45:0/1136442109 conn(0x7f201006a500 msgr2=0x7f201006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.508+0000 7f200ffff640 1 -- 172.21.15.45:0/1136442109 shutdown_connections 2023-11-28T20:50:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.508+0000 7f200ffff640 1 -- 172.21.15.45:0/1136442109 wait complete. 2023-11-28T20:50:09.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.509+0000 7f200ffff640 1 Processor -- start 2023-11-28T20:50:09.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.509+0000 7f200ffff640 1 -- start start 2023-11-28T20:50:09.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.510+0000 7f200ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20100b2500 0x7f20101f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.510+0000 7f200ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 0x7f20101f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.510+0000 7f200ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201015d580 0x7f20101f6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.510+0000 7f200ffff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20101f4200 con 0x7f20100d1e40 2023-11-28T20:50:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.510+0000 7f200ffff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20101579a0 con 0x7f201015d580 2023-11-28T20:50:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.510+0000 7f200ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20101f7080 con 0x7f20100b2500 2023-11-28T20:50:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.511+0000 7f200e7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 0x7f20101f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.511+0000 7f200e7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 0x7f20101f2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48988/0 (socket says 172.21.15.45:48988) 2023-11-28T20:50:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.511+0000 7f200e7fc640 1 -- 172.21.15.45:0/1369473548 learned_addr learned my addr 172.21.15.45:0/1369473548 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.511+0000 7f200e7fc640 1 -- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20100b2500 msgr2=0x7f20101f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.511+0000 7f200e7fc640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20100b2500 0x7f20101f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.511+0000 7f200e7fc640 1 -- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201015d580 msgr2=0x7f20101f6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.511+0000 7f200f7fe640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201015d580 0x7f20101f6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.511+0000 7f200e7fc640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201015d580 0x7f20101f6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.511+0000 7f200e7fc640 1 -- 172.21.15.45:0/1369473548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff8009b30 con 0x7f20100d1e40 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.512+0000 7f200e7fc640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 0x7f20101f2b10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8036040 tx=0x7f1ff8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.512+0000 7f20077fe640 1 -- 172.21.15.45:0/1369473548 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8009510 con 0x7f20100d1e40 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.512+0000 7f20077fe640 1 -- 172.21.15.45:0/1369473548 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff803e450 con 0x7f20100d1e40 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.512+0000 7f20077fe640 1 -- 172.21.15.45:0/1369473548 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff803e800 con 0x7f20100d1e40 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.512+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20101f7360 con 0x7f20100d1e40 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.512+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20101f3050 con 0x7f20100d1e40 2023-11-28T20:50:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.513+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2010157df0 con 0x7f20100d1e40 2023-11-28T20:50:09.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.514+0000 7f20077fe640 1 -- 172.21.15.45:0/1369473548 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f1ff803e960 con 0x7f20100d1e40 2023-11-28T20:50:09.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.515+0000 7f20077fe640 1 -- 172.21.15.45:0/1369473548 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f1ff80ac400 con 0x7f20100d1e40 2023-11-28T20:50:09.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.520+0000 7f20077fe640 1 -- 172.21.15.45:0/1369473548 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ff80773c0 con 0x7f20100d1e40 2023-11-28T20:50:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.650+0000 7f20077fe640 1 -- 172.21.15.45:0/1369473548 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 28) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f1ff8076e70 con 0x7f20100d1e40 2023-11-28T20:50:09.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.652+0000 7f20077fe640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f1ff0070c50 0x7f1ff0073040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.663+0000 7f200effd640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f1ff0070c50 0x7f1ff0073040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.671+0000 7f200effd640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f1ff0070c50 0x7f1ff0073040 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2000006fd0 tx=0x7f2000006bc0 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:09.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.722+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2010071b30 con 0x7f20100d1e40 2023-11-28T20:50:09.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.723+0000 7f20077fe640 1 -- 172.21.15.45:0/1369473548 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ff80a5960 con 0x7f20100d1e40 2023-11-28T20:50:09.726 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:50:09.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f1ff0070c50 msgr2=0x7f1ff0073040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f1ff0070c50 0x7f1ff0073040 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2000006fd0 tx=0x7f2000006bc0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 msgr2=0x7f20101f2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 0x7f20101f2b10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8036040 tx=0x7f1ff8004a40 comp rx=0 tx=0).stop 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 shutdown_connections 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f1ff0070c50 0x7f1ff0073040 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f201015d580 0x7f20101f6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20100d1e40 0x7f20101f2b10 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 --2- 172.21.15.45:0/1369473548 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20100b2500 0x7f20101f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 >> 172.21.15.45:0/1369473548 conn(0x7f201006a500 msgr2=0x7f201015b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 shutdown_connections 2023-11-28T20:50:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.726+0000 7f200ffff640 1 -- 172.21.15.45:0/1369473548 wait complete. 2023-11-28T20:50:09.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:09.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.966+0000 7fee32d27640 1 Processor -- start 2023-11-28T20:50:09.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.966+0000 7fee32d27640 1 -- start start 2023-11-28T20:50:09.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.968+0000 7fee32d27640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fee2c0ae590 0x7fee2c0acab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.968+0000 7fee32d27640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c0abf10 con 0x7fee2c0b24a0 2023-11-28T20:50:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.968+0000 7fee32d27640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c06db90 con 0x7fee2c14b090 2023-11-28T20:50:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.968+0000 7fee32d27640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee2c06d2f0 con 0x7fee2c0ae590 2023-11-28T20:50:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.969+0000 7fee31524640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fee2c0b24a0 0x7fee2c0ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53738/0 (socket says 172.21.15.45:53738) 2023-11-28T20:50:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.969+0000 7fee31524640 1 -- 172.21.15.45:0/398788890 learned_addr learned my addr 172.21.15.45:0/398788890 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.969+0000 7fee32526640 1 --2- 172.21.15.45:0/398788890 >> v2:172.21.15.45:3300/0 conn(0x7fee2c0ae590 0x7fee2c0acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.969+0000 7fee30d23640 1 -- 172.21.15.45:0/398788890 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30515725 0 0) 0x7fee2c0abf10 con 0x7fee2c0b24a0 2023-11-28T20:50:09.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.969+0000 7fee30d23640 1 -- 172.21.15.45:0/398788890 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee18003520 con 0x7fee2c0b24a0 2023-11-28T20:50:09.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.969+0000 7fee32526640 1 -- 172.21.15.45:0/398788890 >> v1:172.21.15.190:6789/0 conn(0x7fee2c14b090 legacy=0x7fee2c0b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.970+0000 7fee32526640 1 -- 172.21.15.45:0/398788890 >> v1:172.21.15.45:6789/0 conn(0x7fee2c0b24a0 legacy=0x7fee2c0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.970+0000 7fee32526640 1 -- 172.21.15.45:0/398788890 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c06d5d0 con 0x7fee2c0ae590 2023-11-28T20:50:09.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.971+0000 7fee32526640 1 --2- 172.21.15.45:0/398788890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c0ae590 0x7fee2c0acab0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fee24009d90 tx=0x7fee2402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb8c88e3c0cfeef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:09.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.971+0000 7fee30d23640 1 -- 172.21.15.45:0/398788890 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2402ccf0 con 0x7fee2c0ae590 2023-11-28T20:50:09.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.971+0000 7fee30d23640 1 -- 172.21.15.45:0/398788890 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2402ce50 con 0x7fee2c0ae590 2023-11-28T20:50:09.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.971+0000 7fee30d23640 1 -- 172.21.15.45:0/398788890 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24034660 con 0x7fee2c0ae590 2023-11-28T20:50:09.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.972+0000 7fee32d27640 1 -- 172.21.15.45:0/398788890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c0ae590 msgr2=0x7fee2c0acab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.972+0000 7fee32d27640 1 --2- 172.21.15.45:0/398788890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c0ae590 0x7fee2c0acab0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fee24009d90 tx=0x7fee2402bf40 comp rx=0 tx=0).stop 2023-11-28T20:50:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.972+0000 7fee32d27640 1 -- 172.21.15.45:0/398788890 shutdown_connections 2023-11-28T20:50:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.972+0000 7fee32d27640 1 --2- 172.21.15.45:0/398788890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c0ae590 0x7fee2c0acab0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.972+0000 7fee32d27640 1 -- 172.21.15.45:0/398788890 >> 172.21.15.45:0/398788890 conn(0x7fee2c06ab30 msgr2=0x7fee2c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.972+0000 7fee32d27640 1 -- 172.21.15.45:0/398788890 shutdown_connections 2023-11-28T20:50:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.972+0000 7fee32d27640 1 -- 172.21.15.45:0/398788890 wait complete. 2023-11-28T20:50:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.974+0000 7fee32d27640 1 Processor -- start 2023-11-28T20:50:09.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.974+0000 7fee32d27640 1 -- start start 2023-11-28T20:50:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.975+0000 7fee32d27640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee2c0ae590 0x7fee2c150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.975+0000 7fee32d27640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee2c0b24a0 0x7fee2c09be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.975+0000 7fee32d27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c14b090 0x7fee2c09afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.975+0000 7fee32d27640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c09c360 con 0x7fee2c14b090 2023-11-28T20:50:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.975+0000 7fee32d27640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c09b500 con 0x7fee2c0b24a0 2023-11-28T20:50:09.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.975+0000 7fee32d27640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c151320 con 0x7fee2c0ae590 2023-11-28T20:50:09.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.975+0000 7fee32526640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c14b090 0x7fee2c09afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.975+0000 7fee32526640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c14b090 0x7fee2c09afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49006/0 (socket says 172.21.15.45:49006) 2023-11-28T20:50:09.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.975+0000 7fee32526640 1 -- 172.21.15.45:0/344019254 learned_addr learned my addr 172.21.15.45:0/344019254 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:09.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee31524640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee2c0b24a0 0x7fee2c09be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee31d25640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee2c0ae590 0x7fee2c150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee32526640 1 -- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee2c0ae590 msgr2=0x7fee2c150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee32526640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee2c0ae590 0x7fee2c150cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee32526640 1 -- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee2c0b24a0 msgr2=0x7fee2c09be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:09.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee32526640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee2c0b24a0 0x7fee2c09be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:09.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee32526640 1 -- 172.21.15.45:0/344019254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee24009a70 con 0x7fee2c14b090 2023-11-28T20:50:09.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee31524640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee2c0b24a0 0x7fee2c09be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:09.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee32526640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c14b090 0x7fee2c09afc0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fee24037040 tx=0x7fee24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:09.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee227fc640 1 -- 172.21.15.45:0/344019254 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2403d600 con 0x7fee2c14b090 2023-11-28T20:50:09.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee227fc640 1 -- 172.21.15.45:0/344019254 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2403d760 con 0x7fee2c14b090 2023-11-28T20:50:09.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.976+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c151600 con 0x7fee2c14b090 2023-11-28T20:50:09.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.977+0000 7fee227fc640 1 -- 172.21.15.45:0/344019254 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2403db10 con 0x7fee2c14b090 2023-11-28T20:50:09.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.977+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c151bc0 con 0x7fee2c14b090 2023-11-28T20:50:09.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.978+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c09d0f0 con 0x7fee2c14b090 2023-11-28T20:50:09.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.979+0000 7fee227fc640 1 -- 172.21.15.45:0/344019254 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 28) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fee2403b040 con 0x7fee2c14b090 2023-11-28T20:50:09.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.980+0000 7fee227fc640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee1006ac90 0x7fee1006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:09.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.981+0000 7fee227fc640 1 -- 172.21.15.45:0/344019254 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6803+0+0 (secure 0 0 0) 0x7fee240abbf0 con 0x7fee2c14b090 2023-11-28T20:50:09.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.981+0000 7fee31d25640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee1006ac90 0x7fee1006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:09.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.982+0000 7fee31d25640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee1006ac90 0x7fee1006d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fee28000f60 tx=0x7fee28009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:09.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:09.986+0000 7fee227fc640 1 -- 172.21.15.45:0/344019254 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee240b2080 con 0x7fee2c14b090 2023-11-28T20:50:10.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.220+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee2c072510 con 0x7fee2c14b090 2023-11-28T20:50:10.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.222+0000 7fee227fc640 1 -- 172.21.15.45:0/344019254 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7fee24039020 con 0x7fee2c14b090 2023-11-28T20:50:10.224 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:10.224 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5183,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/285399852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":285399852},{"type":"v1","addr":"172.21.15.190:6836","nonce":285399852}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2028545354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2028545354},{"type":"v1","addr":"172.21.15.190:6837","nonce":2028545354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6216,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/784320624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":784320624},{"type":"v1","addr":"172.21.15.45:6834","nonce":784320624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:48:13.742492+0000","modified":"2023-11-28T20:48:16.820308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3717491008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3717491008},{"type":"v1","addr":"172.21.15.45:6835","nonce":3717491008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T20:50:10.224 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 27 2023-11-28T20:50:10.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.224+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee1006ac90 msgr2=0x7fee1006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.224+0000 7fee32d27640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee1006ac90 0x7fee1006d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fee28000f60 tx=0x7fee28009040 comp rx=0 tx=0).stop 2023-11-28T20:50:10.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.224+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c14b090 msgr2=0x7fee2c09afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.225+0000 7fee32d27640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c14b090 0x7fee2c09afc0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fee24037040 tx=0x7fee24004a40 comp rx=0 tx=0).stop 2023-11-28T20:50:10.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.225+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 shutdown_connections 2023-11-28T20:50:10.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.225+0000 7fee32d27640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee1006ac90 0x7fee1006d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.225+0000 7fee32d27640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee2c14b090 0x7fee2c09afc0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.225+0000 7fee32d27640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee2c0b24a0 0x7fee2c09be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.225+0000 7fee32d27640 1 --2- 172.21.15.45:0/344019254 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee2c0ae590 0x7fee2c150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.225+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 >> 172.21.15.45:0/344019254 conn(0x7fee2c06ab30 msgr2=0x7fee2c0af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:10.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.225+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 shutdown_connections 2023-11-28T20:50:10.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.225+0000 7fee32d27640 1 -- 172.21.15.45:0/344019254 wait complete. 2023-11-28T20:50:10.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:10.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.472+0000 7fc414264640 1 Processor -- start 2023-11-28T20:50:10.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.472+0000 7fc414264640 1 -- start start 2023-11-28T20:50:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.474+0000 7fc414264640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc40c1438b0 0x7fc40c0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.474+0000 7fc414264640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 0x7fc40c0b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.475+0000 7fc414264640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40c15dcc0 con 0x7fc40c0b24e0 2023-11-28T20:50:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.475+0000 7fc414264640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40c15e840 con 0x7fc40c1438b0 2023-11-28T20:50:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.475+0000 7fc414264640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c15df10 con 0x7fc40c0b1340 2023-11-28T20:50:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.475+0000 7fc412000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc40c1438b0 0x7fc40c0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.475+0000 7fc412000640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc40c1438b0 0x7fc40c0b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49014/0 (socket says 172.21.15.45:49014) 2023-11-28T20:50:10.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.475+0000 7fc412000640 1 -- 172.21.15.45:0/2847298538 learned_addr learned my addr 172.21.15.45:0/2847298538 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:10.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.475+0000 7fc412801640 1 --2- 172.21.15.45:0/2847298538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 0x7fc40c0b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.475+0000 7fc412000640 1 -- 172.21.15.45:0/2847298538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 msgr2=0x7fc40c0b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:10.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.475+0000 7fc412000640 1 --2- 172.21.15.45:0/2847298538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 0x7fc40c0b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.476+0000 7fc412000640 1 -- 172.21.15.45:0/2847298538 >> v1:172.21.15.190:6789/0 conn(0x7fc40c0b24e0 legacy=0x7fc40c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:10.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.476+0000 7fc412000640 1 -- 172.21.15.45:0/2847298538 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c15e1f0 con 0x7fc40c1438b0 2023-11-28T20:50:10.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.477+0000 7fc412000640 1 --2- 172.21.15.45:0/2847298538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 0x7fc40c0b1fa0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc408009870 tx=0x7fc40802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb820b98bf8a7959 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:10.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.477+0000 7fc410ffe640 1 -- 172.21.15.45:0/2847298538 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408031740 con 0x7fc40c1438b0 2023-11-28T20:50:10.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.477+0000 7fc410ffe640 1 -- 172.21.15.45:0/2847298538 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4080318a0 con 0x7fc40c1438b0 2023-11-28T20:50:10.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.477+0000 7fc410ffe640 1 -- 172.21.15.45:0/2847298538 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408031c30 con 0x7fc40c1438b0 2023-11-28T20:50:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.478+0000 7fc414264640 1 -- 172.21.15.45:0/2847298538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 msgr2=0x7fc40c0b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.478+0000 7fc414264640 1 --2- 172.21.15.45:0/2847298538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 0x7fc40c0b1fa0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc408009870 tx=0x7fc40802b7e0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.478+0000 7fc414264640 1 -- 172.21.15.45:0/2847298538 shutdown_connections 2023-11-28T20:50:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.478+0000 7fc414264640 1 --2- 172.21.15.45:0/2847298538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 0x7fc40c0b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.478+0000 7fc414264640 1 --2- 172.21.15.45:0/2847298538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 0x7fc40c0b1fa0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.478+0000 7fc414264640 1 -- 172.21.15.45:0/2847298538 >> 172.21.15.45:0/2847298538 conn(0x7fc40c06a500 msgr2=0x7fc40c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.478+0000 7fc414264640 1 -- 172.21.15.45:0/2847298538 shutdown_connections 2023-11-28T20:50:10.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.478+0000 7fc414264640 1 -- 172.21.15.45:0/2847298538 wait complete. 2023-11-28T20:50:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.480+0000 7fc414264640 1 Processor -- start 2023-11-28T20:50:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.480+0000 7fc414264640 1 -- start start 2023-11-28T20:50:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.481+0000 7fc414264640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 0x7fc40c155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.481+0000 7fc414264640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc40c0b24e0 0x7fc40c140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.481+0000 7fc414264640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 0x7fc40c0b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.481+0000 7fc414264640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c155cc0 con 0x7fc40c1438b0 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.481+0000 7fc414264640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c0af860 con 0x7fc40c0b24e0 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.481+0000 7fc414264640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c0afb40 con 0x7fc40c0b1340 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc412801640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 0x7fc40c0b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc412801640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 0x7fc40c0b0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49018/0 (socket says 172.21.15.45:49018) 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc412801640 1 -- 172.21.15.45:0/4035719894 learned_addr learned my addr 172.21.15.45:0/4035719894 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc4117ff640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc40c0b24e0 0x7fc40c140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc412000640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 0x7fc40c155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc412801640 1 -- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 msgr2=0x7fc40c155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc412801640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 0x7fc40c155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc412801640 1 -- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc40c0b24e0 msgr2=0x7fc40c140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc412801640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc40c0b24e0 0x7fc40c140880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.482+0000 7fc412801640 1 -- 172.21.15.45:0/4035719894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4080094d0 con 0x7fc40c1438b0 2023-11-28T20:50:10.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.483+0000 7fc412801640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 0x7fc40c0b0900 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fc40400cc80 tx=0x7fc40400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.483+0000 7fc4027fc640 1 -- 172.21.15.45:0/4035719894 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40400ce90 con 0x7fc40c1438b0 2023-11-28T20:50:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.483+0000 7fc4027fc640 1 -- 172.21.15.45:0/4035719894 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40400ae80 con 0x7fc40c1438b0 2023-11-28T20:50:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.483+0000 7fc4027fc640 1 -- 172.21.15.45:0/4035719894 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404003bd0 con 0x7fc40c1438b0 2023-11-28T20:50:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.483+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40c0afe20 con 0x7fc40c1438b0 2023-11-28T20:50:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.483+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40c0b03e0 con 0x7fc40c1438b0 2023-11-28T20:50:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.484+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40c0b0600 con 0x7fc40c1438b0 2023-11-28T20:50:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.485+0000 7fc4027fc640 1 -- 172.21.15.45:0/4035719894 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 28) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fc40400f3f0 con 0x7fc40c1438b0 2023-11-28T20:50:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.486+0000 7fc4027fc640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc3f006ac90 0x7fc3f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.486+0000 7fc4027fc640 1 -- 172.21.15.45:0/4035719894 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6803+0+0 (secure 0 0 0) 0x7fc404089af0 con 0x7fc40c1438b0 2023-11-28T20:50:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.487+0000 7fc412000640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc3f006ac90 0x7fc3f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.488+0000 7fc412000640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc3f006ac90 0x7fc3f006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc408035000 tx=0x7fc408008030 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.491+0000 7fc4027fc640 1 -- 172.21.15.45:0/4035719894 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4040549c0 con 0x7fc40c1438b0 2023-11-28T20:50:10.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.662+0000 7fc4027fc640 1 -- 172.21.15.45:0/4035719894 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 29) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fc404054d90 con 0x7fc40c1438b0 2023-11-28T20:50:10.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.692+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc40c0733d0 con 0x7fc40c1438b0 2023-11-28T20:50:10.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.694+0000 7fc4027fc640 1 -- 172.21.15.45:0/4035719894 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+17877 (secure 0 0 0) 0x7fc404074070 con 0x7fc40c1438b0 2023-11-28T20:50:10.697 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:10.697 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":77,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:08.628325+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:48:11.673148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-28T20:48:12.760782+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/4256179623":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/4143526042":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/124688230":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/1895312230":"2023-11-29T20:50:08.628199+0000","172.21.15.190:6833/979211100":"2023-11-29T20:50:08.628199+0000","172.21.15.190:6832/979211100":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/1831728153":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/105965464":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/2346462975":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/1966056956":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6832/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/633536721":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6833/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/4198236319":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/676976495":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/3224416347":"2023-11-29T20:48:28.490206+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:10.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc3f006ac90 msgr2=0x7fc3f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc3f006ac90 0x7fc3f006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc408035000 tx=0x7fc408008030 comp rx=0 tx=0).stop 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 msgr2=0x7fc40c0b0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 0x7fc40c0b0900 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fc40400cc80 tx=0x7fc40400a5c0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 shutdown_connections 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc3f006ac90 0x7fc3f006d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc40c1438b0 0x7fc40c0b0900 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc40c0b24e0 0x7fc40c140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 --2- 172.21.15.45:0/4035719894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc40c0b1340 0x7fc40c155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 >> 172.21.15.45:0/4035719894 conn(0x7fc40c06a500 msgr2=0x7fc40c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 shutdown_connections 2023-11-28T20:50:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.697+0000 7fc414264640 1 -- 172.21.15.45:0/4035719894 wait complete. 2023-11-28T20:50:10.719 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T20:50:10.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:10.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.946+0000 7f927f252640 1 Processor -- start 2023-11-28T20:50:10.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.946+0000 7f927f252640 1 -- start start 2023-11-28T20:50:10.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.948+0000 7f927f252640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f92780b35b0 0x7f927815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.948+0000 7f927f252640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f927815b850 0x7f92780b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.948+0000 7f927f252640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f92780b1fa0 0x7f92780b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.948+0000 7f927f252640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927815ee40 con 0x7f92780b35b0 2023-11-28T20:50:10.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.948+0000 7f927f252640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927815bda0 con 0x7f92780b1fa0 2023-11-28T20:50:10.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.948+0000 7f927f252640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927815c080 con 0x7f927815b850 2023-11-28T20:50:10.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.948+0000 7f927cfee640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f92780b35b0 0x7f927815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.948+0000 7f927cfee640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f92780b35b0 0x7f927815ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49032/0 (socket says 172.21.15.45:49032) 2023-11-28T20:50:10.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.948+0000 7f927cfee640 1 -- 172.21.15.45:0/4176025175 learned_addr learned my addr 172.21.15.45:0/4176025175 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.949+0000 7f927cfee640 1 -- 172.21.15.45:0/4176025175 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f927815b850 msgr2=0x7f92780b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.949+0000 7f927cfee640 1 --2- 172.21.15.45:0/4176025175 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f927815b850 0x7f92780b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.949+0000 7f927cfee640 1 -- 172.21.15.45:0/4176025175 >> v2:172.21.15.190:3300/0 conn(0x7f92780b1fa0 msgr2=0x7f92780b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.949+0000 7f927cfee640 1 --2- 172.21.15.45:0/4176025175 >> v2:172.21.15.190:3300/0 conn(0x7f92780b1fa0 0x7f92780b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.949+0000 7f927cfee640 1 -- 172.21.15.45:0/4176025175 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927815c360 con 0x7f92780b35b0 2023-11-28T20:50:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.950+0000 7f927cfee640 1 --2- 172.21.15.45:0/4176025175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92780b35b0 0x7f927815ca10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f926c009870 tx=0x7f926c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358369f0b733acdb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.950+0000 7f92777fe640 1 -- 172.21.15.45:0/4176025175 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c031640 con 0x7f92780b35b0 2023-11-28T20:50:10.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.950+0000 7f92777fe640 1 -- 172.21.15.45:0/4176025175 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c0317a0 con 0x7f92780b35b0 2023-11-28T20:50:10.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.950+0000 7f92777fe640 1 -- 172.21.15.45:0/4176025175 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c031b30 con 0x7f92780b35b0 2023-11-28T20:50:10.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.951+0000 7f927f252640 1 -- 172.21.15.45:0/4176025175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92780b35b0 msgr2=0x7f927815ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.951+0000 7f927f252640 1 --2- 172.21.15.45:0/4176025175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92780b35b0 0x7f927815ca10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f926c009870 tx=0x7f926c02b8f0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.951+0000 7f927f252640 1 -- 172.21.15.45:0/4176025175 shutdown_connections 2023-11-28T20:50:10.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.951+0000 7f927f252640 1 --2- 172.21.15.45:0/4176025175 >> v2:172.21.15.190:3300/0 conn(0x7f92780b1fa0 0x7f92780b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.951+0000 7f927f252640 1 --2- 172.21.15.45:0/4176025175 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f927815b850 0x7f92780b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.951+0000 7f927f252640 1 --2- 172.21.15.45:0/4176025175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92780b35b0 0x7f927815ca10 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.951+0000 7f927f252640 1 -- 172.21.15.45:0/4176025175 >> 172.21.15.45:0/4176025175 conn(0x7f927806a500 msgr2=0x7f927806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:10.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.951+0000 7f927f252640 1 -- 172.21.15.45:0/4176025175 shutdown_connections 2023-11-28T20:50:10.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.951+0000 7f927f252640 1 -- 172.21.15.45:0/4176025175 wait complete. 2023-11-28T20:50:10.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.953+0000 7f927f252640 1 Processor -- start 2023-11-28T20:50:10.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.953+0000 7f927f252640 1 -- start start 2023-11-28T20:50:10.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.954+0000 7f927f252640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92780b1fa0 0x7f92780ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.954+0000 7f927f252640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780b35b0 0x7f92780cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.954+0000 7f927f252640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f927815b850 0x7f92780acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.954+0000 7f927f252640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92780ad290 con 0x7f927815b850 2023-11-28T20:50:10.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.954+0000 7f927f252640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92781f6c40 con 0x7f92780b35b0 2023-11-28T20:50:10.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.954+0000 7f927f252640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92781f6f20 con 0x7f92780b1fa0 2023-11-28T20:50:10.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.954+0000 7f927d7ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f927815b850 0x7f92780acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.955+0000 7f927d7ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f927815b850 0x7f92780acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49034/0 (socket says 172.21.15.45:49034) 2023-11-28T20:50:10.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.955+0000 7f927d7ef640 1 -- 172.21.15.45:0/3877501718 learned_addr learned my addr 172.21.15.45:0/3877501718 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:10.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.955+0000 7f927d7ef640 1 -- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92780b1fa0 msgr2=0x7f92780ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.955+0000 7f927d7ef640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92780b1fa0 0x7f92780ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.955+0000 7f927d7ef640 1 -- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780b35b0 msgr2=0x7f92780cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:10.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.955+0000 7f9277fff640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780b35b0 0x7f92780cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.955+0000 7f927d7ef640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780b35b0 0x7f92780cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:10.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.955+0000 7f927d7ef640 1 -- 172.21.15.45:0/3877501718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0094d0 con 0x7f927815b850 2023-11-28T20:50:10.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.955+0000 7f927d7ef640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f927815b850 0x7f92780acd50 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f927000b9e0 tx=0x7f927000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:10.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.956+0000 7f92757fa640 1 -- 172.21.15.45:0/3877501718 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927000cb20 con 0x7f927815b850 2023-11-28T20:50:10.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.956+0000 7f92757fa640 1 -- 172.21.15.45:0/3877501718 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927000cc80 con 0x7f927815b850 2023-11-28T20:50:10.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.956+0000 7f92757fa640 1 -- 172.21.15.45:0/3877501718 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9270014680 con 0x7f927815b850 2023-11-28T20:50:10.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.956+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92781f7200 con 0x7f927815b850 2023-11-28T20:50:10.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.956+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92781f77c0 con 0x7f927815b850 2023-11-28T20:50:10.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.957+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92781f79e0 con 0x7f927815b850 2023-11-28T20:50:10.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.958+0000 7f92757fa640 1 -- 172.21.15.45:0/3877501718 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f927001d3b0 con 0x7f927815b850 2023-11-28T20:50:10.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.959+0000 7f92757fa640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f925c06aed0 0x7f925c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:10.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.959+0000 7f92757fa640 1 -- 172.21.15.45:0/3877501718 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f9270089b00 con 0x7f927815b850 2023-11-28T20:50:10.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.959+0000 7f927cfee640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f925c06aed0 0x7f925c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:10.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.961+0000 7f927cfee640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f925c06aed0 0x7f925c06d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f926c036000 tx=0x7f926c030000 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:10.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:10.964+0000 7f92757fa640 1 -- 172.21.15.45:0/3877501718 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f927008be60 con 0x7f927815b850 2023-11-28T20:50:11.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.200+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92780733d0 con 0x7f927815b850 2023-11-28T20:50:11.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.201+0000 7f92757fa640 1 -- 172.21.15.45:0/3877501718 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f9270019090 con 0x7f927815b850 2023-11-28T20:50:11.204 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:11.204 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5183,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/285399852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":285399852},{"type":"v1","addr":"172.21.15.190:6836","nonce":285399852}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2028545354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2028545354},{"type":"v1","addr":"172.21.15.190:6837","nonce":2028545354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6216,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/784320624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":784320624},{"type":"v1","addr":"172.21.15.45:6834","nonce":784320624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:48:13.742492+0000","modified":"2023-11-28T20:48:16.820308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3717491008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3717491008},{"type":"v1","addr":"172.21.15.45:6835","nonce":3717491008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T20:50:11.204 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 27 2023-11-28T20:50:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f925c06aed0 msgr2=0x7f925c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f925c06aed0 0x7f925c06d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f926c036000 tx=0x7f926c030000 comp rx=0 tx=0).stop 2023-11-28T20:50:11.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f927815b850 msgr2=0x7f92780acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f927815b850 0x7f92780acd50 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f927000b9e0 tx=0x7f927000bea0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 shutdown_connections 2023-11-28T20:50:11.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f925c06aed0 0x7f925c06d390 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f927815b850 0x7f92780acd50 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780b35b0 0x7f92780cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 --2- 172.21.15.45:0/3877501718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f92780b1fa0 0x7f92780ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 >> 172.21.15.45:0/3877501718 conn(0x7f927806a500 msgr2=0x7f9278144870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:11.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 shutdown_connections 2023-11-28T20:50:11.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.204+0000 7f927f252640 1 -- 172.21.15.45:0/3877501718 wait complete. 2023-11-28T20:50:11.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:11.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.446+0000 7f4d3359e640 1 Processor -- start 2023-11-28T20:50:11.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.447+0000 7f4d3359e640 1 -- start start 2023-11-28T20:50:11.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.448+0000 7f4d3359e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4d34155780 0x7f4d340abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.448+0000 7f4d3359e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3415c3c0 con 0x7f4d340ae540 2023-11-28T20:50:11.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.448+0000 7f4d3359e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d34155b60 con 0x7f4d340ce800 2023-11-28T20:50:11.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.448+0000 7f4d3359e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3415ba90 con 0x7f4d34155780 2023-11-28T20:50:11.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.449+0000 7f4d31d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4d340ae540 0x7f4d340ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53754/0 (socket says 172.21.15.45:53754) 2023-11-28T20:50:11.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.449+0000 7f4d3259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4d34155780 0x7f4d340abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.449+0000 7f4d31d9b640 1 -- 172.21.15.45:0/2408527347 learned_addr learned my addr 172.21.15.45:0/2408527347 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:11.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.449+0000 7f4d3159a640 1 -- 172.21.15.45:0/2408527347 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099020468 0 0) 0x7f4d3415c3c0 con 0x7f4d340ae540 2023-11-28T20:50:11.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.449+0000 7f4d3159a640 1 -- 172.21.15.45:0/2408527347 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d20003520 con 0x7f4d340ae540 2023-11-28T20:50:11.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.449+0000 7f4d3259c640 1 -- 172.21.15.45:0/2408527347 >> v1:172.21.15.190:6789/0 conn(0x7f4d340ce800 legacy=0x7f4d340b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.449+0000 7f4d3259c640 1 -- 172.21.15.45:0/2408527347 >> v1:172.21.15.45:6789/0 conn(0x7f4d340ae540 legacy=0x7f4d340ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.449+0000 7f4d3259c640 1 -- 172.21.15.45:0/2408527347 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3415bd70 con 0x7f4d34155780 2023-11-28T20:50:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.451+0000 7f4d3259c640 1 --2- 172.21.15.45:0/2408527347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 0x7f4d340abd70 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4d240097a0 tx=0x7f4d2402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23350b87d23c5314 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.451+0000 7f4d3159a640 1 -- 172.21.15.45:0/2408527347 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24030610 con 0x7f4d34155780 2023-11-28T20:50:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.451+0000 7f4d3159a640 1 -- 172.21.15.45:0/2408527347 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d24030770 con 0x7f4d34155780 2023-11-28T20:50:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.451+0000 7f4d3159a640 1 -- 172.21.15.45:0/2408527347 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24030b00 con 0x7f4d34155780 2023-11-28T20:50:11.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.452+0000 7f4d3359e640 1 -- 172.21.15.45:0/2408527347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 msgr2=0x7f4d340abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.452+0000 7f4d3359e640 1 --2- 172.21.15.45:0/2408527347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 0x7f4d340abd70 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4d240097a0 tx=0x7f4d2402b960 comp rx=0 tx=0).stop 2023-11-28T20:50:11.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.452+0000 7f4d3359e640 1 -- 172.21.15.45:0/2408527347 shutdown_connections 2023-11-28T20:50:11.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.452+0000 7f4d3359e640 1 --2- 172.21.15.45:0/2408527347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 0x7f4d340abd70 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.452+0000 7f4d3359e640 1 -- 172.21.15.45:0/2408527347 >> 172.21.15.45:0/2408527347 conn(0x7f4d3406ab30 msgr2=0x7f4d340b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:11.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.452+0000 7f4d3359e640 1 -- 172.21.15.45:0/2408527347 shutdown_connections 2023-11-28T20:50:11.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.452+0000 7f4d3359e640 1 -- 172.21.15.45:0/2408527347 wait complete. 2023-11-28T20:50:11.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.454+0000 7f4d3359e640 1 Processor -- start 2023-11-28T20:50:11.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.454+0000 7f4d3359e640 1 -- start start 2023-11-28T20:50:11.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.455+0000 7f4d3359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d340ae540 0x7f4d34143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.455+0000 7f4d3359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d340ce800 0x7f4d3415d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.455+0000 7f4d3359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 0x7f4d3415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.455+0000 7f4d3359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d34147610 con 0x7f4d34155780 2023-11-28T20:50:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.455+0000 7f4d3359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d34146a90 con 0x7f4d340ae540 2023-11-28T20:50:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.455+0000 7f4d3359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d34146d70 con 0x7f4d340ce800 2023-11-28T20:50:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.455+0000 7f4d32d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 0x7f4d3415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.455+0000 7f4d32d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 0x7f4d3415d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49052/0 (socket says 172.21.15.45:49052) 2023-11-28T20:50:11.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.455+0000 7f4d32d9d640 1 -- 172.21.15.45:0/69603097 learned_addr learned my addr 172.21.15.45:0/69603097 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.456+0000 7f4d3259c640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d340ae540 0x7f4d34143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.456+0000 7f4d32d9d640 1 -- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d340ce800 msgr2=0x7f4d3415d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.456+0000 7f4d31d9b640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d340ce800 0x7f4d3415d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.456+0000 7f4d32d9d640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d340ce800 0x7f4d3415d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.456+0000 7f4d32d9d640 1 -- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d340ae540 msgr2=0x7f4d34143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.456+0000 7f4d32d9d640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d340ae540 0x7f4d34143d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.456+0000 7f4d32d9d640 1 -- 172.21.15.45:0/69603097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d24009400 con 0x7f4d34155780 2023-11-28T20:50:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.456+0000 7f4d3259c640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d340ae540 0x7f4d34143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.456+0000 7f4d32d9d640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 0x7f4d3415d8a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c001eb0 tx=0x7f4d2c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.457+0000 7f4d2affd640 1 -- 172.21.15.45:0/69603097 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0052d0 con 0x7f4d34155780 2023-11-28T20:50:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.457+0000 7f4d2affd640 1 -- 172.21.15.45:0/69603097 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2c005430 con 0x7f4d34155780 2023-11-28T20:50:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.457+0000 7f4d2affd640 1 -- 172.21.15.45:0/69603097 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c003810 con 0x7f4d34155780 2023-11-28T20:50:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.457+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d34147050 con 0x7f4d34155780 2023-11-28T20:50:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.457+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d34144240 con 0x7f4d34155780 2023-11-28T20:50:11.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.458+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d34144690 con 0x7f4d34155780 2023-11-28T20:50:11.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.459+0000 7f4d2affd640 1 -- 172.21.15.45:0/69603097 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f4d2c003970 con 0x7f4d34155780 2023-11-28T20:50:11.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.460+0000 7f4d2affd640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4d1406af00 0x7f4d1406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.461+0000 7f4d2affd640 1 -- 172.21.15.45:0/69603097 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f4d2c089100 con 0x7f4d34155780 2023-11-28T20:50:11.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.461+0000 7f4d3259c640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4d1406af00 0x7f4d1406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.462+0000 7f4d3259c640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4d1406af00 0x7f4d1406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d24036000 tx=0x7f4d2403b040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:11.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.465+0000 7f4d2affd640 1 -- 172.21.15.45:0/69603097 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d34144690 con 0x7f4d34155780 2023-11-28T20:50:11.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.699+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d34144690 con 0x7f4d34155780 2023-11-28T20:50:11.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.700+0000 7f4d2affd640 1 -- 172.21.15.45:0/69603097 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f4d34144690 con 0x7f4d34155780 2023-11-28T20:50:11.703 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:11.703 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5183,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/285399852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":285399852},{"type":"v1","addr":"172.21.15.190:6836","nonce":285399852}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2028545354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2028545354},{"type":"v1","addr":"172.21.15.190:6837","nonce":2028545354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6216,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/784320624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":784320624},{"type":"v1","addr":"172.21.15.45:6834","nonce":784320624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:48:13.742492+0000","modified":"2023-11-28T20:48:16.820308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3717491008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3717491008},{"type":"v1","addr":"172.21.15.45:6835","nonce":3717491008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T20:50:11.703 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 27 2023-11-28T20:50:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4d1406af00 msgr2=0x7f4d1406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4d1406af00 0x7f4d1406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d24036000 tx=0x7f4d2403b040 comp rx=0 tx=0).stop 2023-11-28T20:50:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 msgr2=0x7f4d3415d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 0x7f4d3415d8a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c001eb0 tx=0x7f4d2c00c890 comp rx=0 tx=0).stop 2023-11-28T20:50:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 shutdown_connections 2023-11-28T20:50:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4d1406af00 0x7f4d1406d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d34155780 0x7f4d3415d8a0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d340ce800 0x7f4d3415d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 --2- 172.21.15.45:0/69603097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d340ae540 0x7f4d34143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 >> 172.21.15.45:0/69603097 conn(0x7f4d3406ab30 msgr2=0x7f4d340afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 shutdown_connections 2023-11-28T20:50:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.703+0000 7f4d3359e640 1 -- 172.21.15.45:0/69603097 wait complete. 2023-11-28T20:50:11.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:11.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.947+0000 7f6b2b0da640 1 Processor -- start 2023-11-28T20:50:11.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.947+0000 7f6b2b0da640 1 -- start start 2023-11-28T20:50:11.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.949+0000 7f6b2b0da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2415b850 0x7f6b240b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.949+0000 7f6b2b0da640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6b240b1fa0 0x7f6b240b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.949+0000 7f6b2b0da640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2415bda0 con 0x7f6b240b35b0 2023-11-28T20:50:11.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.949+0000 7f6b2b0da640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b2415ee40 con 0x7f6b240b1fa0 2023-11-28T20:50:11.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.949+0000 7f6b2b0da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2415c080 con 0x7f6b2415b850 2023-11-28T20:50:11.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.949+0000 7f6b29677640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6b240b1fa0 0x7f6b240b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.950+0000 7f6b29677640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6b240b1fa0 0x7f6b240b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49054/0 (socket says 172.21.15.45:49054) 2023-11-28T20:50:11.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.950+0000 7f6b29677640 1 -- 172.21.15.45:0/2784349194 learned_addr learned my addr 172.21.15.45:0/2784349194 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:11.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.950+0000 7f6b23fff640 1 --2- 172.21.15.45:0/2784349194 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2415b850 0x7f6b240b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.950+0000 7f6b29677640 1 -- 172.21.15.45:0/2784349194 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2415b850 msgr2=0x7f6b240b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:11.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.950+0000 7f6b29677640 1 --2- 172.21.15.45:0/2784349194 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2415b850 0x7f6b240b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.950+0000 7f6b29677640 1 -- 172.21.15.45:0/2784349194 >> v1:172.21.15.190:6789/0 conn(0x7f6b240b35b0 legacy=0x7f6b2415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:11.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.950+0000 7f6b29677640 1 -- 172.21.15.45:0/2784349194 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2415c360 con 0x7f6b240b1fa0 2023-11-28T20:50:11.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.950+0000 7f6b23fff640 1 --2- 172.21.15.45:0/2784349194 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2415b850 0x7f6b240b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:11.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.951+0000 7f6b29677640 1 --2- 172.21.15.45:0/2784349194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b1fa0 0x7f6b240b1170 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0098a0 tx=0x7f6b1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5eaff53b6beb75 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:11.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.951+0000 7f6b237fe640 1 -- 172.21.15.45:0/2784349194 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c031640 con 0x7f6b240b1fa0 2023-11-28T20:50:11.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b237fe640 1 -- 172.21.15.45:0/2784349194 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c0317a0 con 0x7f6b240b1fa0 2023-11-28T20:50:11.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b237fe640 1 -- 172.21.15.45:0/2784349194 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c031b30 con 0x7f6b240b1fa0 2023-11-28T20:50:11.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2784349194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b1fa0 msgr2=0x7f6b240b1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b2b0da640 1 --2- 172.21.15.45:0/2784349194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b1fa0 0x7f6b240b1170 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0098a0 tx=0x7f6b1c02b8f0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2784349194 shutdown_connections 2023-11-28T20:50:11.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b2b0da640 1 --2- 172.21.15.45:0/2784349194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b1fa0 0x7f6b240b1170 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b2b0da640 1 --2- 172.21.15.45:0/2784349194 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2415b850 0x7f6b240b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2784349194 >> 172.21.15.45:0/2784349194 conn(0x7f6b2406a500 msgr2=0x7f6b2406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:11.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2784349194 shutdown_connections 2023-11-28T20:50:11.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.952+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2784349194 wait complete. 2023-11-28T20:50:11.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.954+0000 7f6b2b0da640 1 Processor -- start 2023-11-28T20:50:11.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.954+0000 7f6b2b0da640 1 -- start start 2023-11-28T20:50:11.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.955+0000 7f6b2b0da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b240b1fa0 0x7f6b240ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.955+0000 7f6b2b0da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b35b0 0x7f6b240a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.955+0000 7f6b2b0da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2415b850 0x7f6b240716b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.955+0000 7f6b2b0da640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b24133be0 con 0x7f6b240b35b0 2023-11-28T20:50:11.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.955+0000 7f6b2b0da640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b240a9440 con 0x7f6b2415b850 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.955+0000 7f6b2b0da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b240a8e80 con 0x7f6b240b1fa0 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b23fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b35b0 0x7f6b240a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b23fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b35b0 0x7f6b240a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49066/0 (socket says 172.21.15.45:49066) 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b23fff640 1 -- 172.21.15.45:0/2748344972 learned_addr learned my addr 172.21.15.45:0/2748344972 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b23fff640 1 -- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b240b1fa0 msgr2=0x7f6b240ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b23fff640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b240b1fa0 0x7f6b240ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b23fff640 1 -- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2415b850 msgr2=0x7f6b240716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b29677640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2415b850 0x7f6b240716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b23fff640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2415b850 0x7f6b240716b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b23fff640 1 -- 172.21.15.45:0/2748344972 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c0094d0 con 0x7f6b240b35b0 2023-11-28T20:50:11.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.956+0000 7f6b23fff640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b35b0 0x7f6b240a84a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00c7f0 tx=0x7f6b0c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:11.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.957+0000 7f6b217fa640 1 -- 172.21.15.45:0/2748344972 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c007820 con 0x7f6b240b35b0 2023-11-28T20:50:11.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.957+0000 7f6b217fa640 1 -- 172.21.15.45:0/2748344972 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c007980 con 0x7f6b240b35b0 2023-11-28T20:50:11.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.957+0000 7f6b217fa640 1 -- 172.21.15.45:0/2748344972 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c007d30 con 0x7f6b240b35b0 2023-11-28T20:50:11.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.957+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b240a9160 con 0x7f6b240b35b0 2023-11-28T20:50:11.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.957+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b240acb10 con 0x7f6b240b35b0 2023-11-28T20:50:11.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.958+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b24155a70 con 0x7f6b240b35b0 2023-11-28T20:50:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.959+0000 7f6b217fa640 1 -- 172.21.15.45:0/2748344972 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f6b0c01a660 con 0x7f6b240b35b0 2023-11-28T20:50:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.960+0000 7f6b217fa640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f6b0406aed0 0x7f6b0406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.960+0000 7f6b217fa640 1 -- 172.21.15.45:0/2748344972 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f6b0c089b00 con 0x7f6b240b35b0 2023-11-28T20:50:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.961+0000 7f6b28e76640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f6b0406aed0 0x7f6b0406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.962+0000 7f6b28e76640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f6b0406aed0 0x7f6b0406d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b140043c0 tx=0x7f6b14007d30 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:11.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:11.964+0000 7f6b217fa640 1 -- 172.21.15.45:0/2748344972 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b0c08be60 con 0x7f6b240b35b0 2023-11-28T20:50:12.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.161+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b24133e50 con 0x7f6b240b35b0 2023-11-28T20:50:12.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.164+0000 7f6b217fa640 1 -- 172.21.15.45:0/2748344972 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+17877 (secure 0 0 0) 0x7f6b0c0545f0 con 0x7f6b240b35b0 2023-11-28T20:50:12.167 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:12.168 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":77,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:08.628325+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:48:11.673148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-28T20:48:12.760782+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.6","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/4256179623":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/4143526042":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/124688230":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/1895312230":"2023-11-29T20:50:08.628199+0000","172.21.15.190:6833/979211100":"2023-11-29T20:50:08.628199+0000","172.21.15.190:6832/979211100":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/1831728153":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/105965464":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/2346462975":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/1966056956":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6832/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/633536721":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6833/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/4198236319":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/676976495":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/3224416347":"2023-11-29T20:48:28.490206+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:12.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f6b0406aed0 msgr2=0x7f6b0406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f6b0406aed0 0x7f6b0406d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b140043c0 tx=0x7f6b14007d30 comp rx=0 tx=0).stop 2023-11-28T20:50:12.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b35b0 msgr2=0x7f6b240a84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b35b0 0x7f6b240a84a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00c7f0 tx=0x7f6b0c00ccb0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 shutdown_connections 2023-11-28T20:50:12.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f6b0406aed0 0x7f6b0406d390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b2415b850 0x7f6b240716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b240b35b0 0x7f6b240a84a0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 --2- 172.21.15.45:0/2748344972 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b240b1fa0 0x7f6b240ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 >> 172.21.15.45:0/2748344972 conn(0x7f6b2406a500 msgr2=0x7f6b240ce150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:12.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 shutdown_connections 2023-11-28T20:50:12.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.167+0000 7f6b2b0da640 1 -- 172.21.15.45:0/2748344972 wait complete. 2023-11-28T20:50:12.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T20:50:12.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.412+0000 7f14c2354640 1 Processor -- start 2023-11-28T20:50:12.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.412+0000 7f14c2354640 1 -- start start 2023-11-28T20:50:12.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.413+0000 7f14c2354640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f14bc15d120 0x7f14bc0d2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.413+0000 7f14c2354640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc0d3040 con 0x7f14bc0d2a40 2023-11-28T20:50:12.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.413+0000 7f14c2354640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc0d3320 con 0x7f14bc0d0690 2023-11-28T20:50:12.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.413+0000 7f14c2354640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14bc0d3600 con 0x7f14bc15d120 2023-11-28T20:50:12.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14c0b51640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f14bc0d2a40 0x7f14bc0cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53758/0 (socket says 172.21.15.45:53758) 2023-11-28T20:50:12.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14c0b51640 1 -- 172.21.15.45:0/1181663770 learned_addr learned my addr 172.21.15.45:0/1181663770 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:12.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14c1352640 1 --2- 172.21.15.45:0/1181663770 >> v2:172.21.15.190:3300/0 conn(0x7f14bc15d120 0x7f14bc0d2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474845158 0 0) 0x7f14bc0d3040 con 0x7f14bc0d2a40 2023-11-28T20:50:12.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a8003520 con 0x7f14bc0d2a40 2023-11-28T20:50:12.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1502133950 0 0) 0x7f14a8003520 con 0x7f14bc0d2a40 2023-11-28T20:50:12.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 >> v2:172.21.15.190:3300/0 conn(0x7f14bc15d120 msgr2=0x7f14bc0d2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:12.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14b3fff640 1 --2- 172.21.15.45:0/1181663770 >> v2:172.21.15.190:3300/0 conn(0x7f14bc15d120 0x7f14bc0d2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 >> v1:172.21.15.190:6789/0 conn(0x7f14bc0d0690 legacy=0x7f14bc0b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc0d38e0 con 0x7f14bc0d2a40 2023-11-28T20:50:12.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f14ac001670 con 0x7f14bc0d2a40 2023-11-28T20:50:12.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.414+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 >> v1:172.21.15.45:6789/0 conn(0x7f14bc0d2a40 legacy=0x7f14bc0cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14b3fff640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8003970 0x7f14a8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14b3fff640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14a80047c0 0x7f14a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14b3fff640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14a8005180 0x7f14a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc0d3600 con 0x7f14a8005180 2023-11-28T20:50:12.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a800c0b0 con 0x7f14a8003970 2023-11-28T20:50:12.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a800c270 con 0x7f14a80047c0 2023-11-28T20:50:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14c1352640 1 --2- 172.21.15.45:0/1181663770 >> v2:172.21.15.190:3300/0 conn(0x7f14bc15d120 0x7f14bc0d2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14c1b53640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14a8005180 0x7f14a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14c0b51640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14a80047c0 0x7f14a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14c1352640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8003970 0x7f14a8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14c1b53640 1 -- 172.21.15.45:0/1181663770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14a80047c0 msgr2=0x7f14a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.415+0000 7f14c1b53640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14a80047c0 0x7f14a8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.416+0000 7f14c1b53640 1 -- 172.21.15.45:0/1181663770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8003970 msgr2=0x7f14a8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.416+0000 7f14c1b53640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8003970 0x7f14a8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.416+0000 7f14c1b53640 1 -- 172.21.15.45:0/1181663770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a8003520 con 0x7f14a8005180 2023-11-28T20:50:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.416+0000 7f14c1352640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8003970 0x7f14a8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.417+0000 7f14c1b53640 1 -- 172.21.15.45:0/1181663770 reap_dead start 2023-11-28T20:50:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.417+0000 7f14c1b53640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14a8005180 0x7f14a800b980 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f14b4001e20 tx=0x7f14a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bacea925404de226 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.417+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4003b70 con 0x7f14a8005180 2023-11-28T20:50:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.417+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b4003cd0 con 0x7f14a8005180 2023-11-28T20:50:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.417+0000 7f14b3fff640 1 -- 172.21.15.45:0/1181663770 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4005310 con 0x7f14a8005180 2023-11-28T20:50:12.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.418+0000 7f14c2354640 1 -- 172.21.15.45:0/1181663770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14a8005180 msgr2=0x7f14a800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.418+0000 7f14c2354640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14a8005180 0x7f14a800b980 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f14b4001e20 tx=0x7f14a80057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.418+0000 7f14c2354640 1 -- 172.21.15.45:0/1181663770 shutdown_connections 2023-11-28T20:50:12.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.418+0000 7f14c2354640 1 --2- 172.21.15.45:0/1181663770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14a8005180 0x7f14a800b980 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.418+0000 7f14c2354640 1 -- 172.21.15.45:0/1181663770 >> 172.21.15.45:0/1181663770 conn(0x7f14bc070f40 msgr2=0x7f14bc071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:12.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.418+0000 7f14c2354640 1 -- 172.21.15.45:0/1181663770 shutdown_connections 2023-11-28T20:50:12.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.418+0000 7f14c2354640 1 -- 172.21.15.45:0/1181663770 wait complete. 2023-11-28T20:50:12.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.420+0000 7f14c2354640 1 Processor -- start 2023-11-28T20:50:12.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.420+0000 7f14c2354640 1 -- start start 2023-11-28T20:50:12.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.421+0000 7f14c2354640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8005180 0x7f14bc14f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.421+0000 7f14c2354640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14bc15e930 0x7f14bc15edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.421+0000 7f14c2354640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14bc1e6160 0x7f14bc1e65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.421+0000 7f14c2354640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc1e6b20 con 0x7f14bc1e6160 2023-11-28T20:50:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.421+0000 7f14c2354640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc14f9d0 con 0x7f14a8005180 2023-11-28T20:50:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.421+0000 7f14c2354640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc160000 con 0x7f14bc15e930 2023-11-28T20:50:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1b53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14bc1e6160 0x7f14bc1e65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1b53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14bc1e6160 0x7f14bc1e65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49084/0 (socket says 172.21.15.45:49084) 2023-11-28T20:50:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1b53640 1 -- 172.21.15.45:0/2985499784 learned_addr learned my addr 172.21.15.45:0/2985499784 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1352640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8005180 0x7f14bc14f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c0b51640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14bc15e930 0x7f14bc15edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1b53640 1 -- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14bc15e930 msgr2=0x7f14bc15edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1b53640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14bc15e930 0x7f14bc15edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1b53640 1 -- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8005180 msgr2=0x7f14bc14f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1b53640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8005180 0x7f14bc14f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1b53640 1 -- 172.21.15.45:0/2985499784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc15f3f0 con 0x7f14bc1e6160 2023-11-28T20:50:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1352640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8005180 0x7f14bc14f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:12.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.422+0000 7f14c1b53640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14bc1e6160 0x7f14bc1e65e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f14b402c040 tx=0x7f14b40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:12.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.423+0000 7f14b1ffb640 1 -- 172.21.15.45:0/2985499784 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4006d50 con 0x7f14bc1e6160 2023-11-28T20:50:12.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.423+0000 7f14b1ffb640 1 -- 172.21.15.45:0/2985499784 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b4007db0 con 0x7f14bc1e6160 2023-11-28T20:50:12.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.423+0000 7f14b1ffb640 1 -- 172.21.15.45:0/2985499784 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4007250 con 0x7f14bc1e6160 2023-11-28T20:50:12.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.423+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14bc15f6d0 con 0x7f14bc1e6160 2023-11-28T20:50:12.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.423+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc15fc90 con 0x7f14bc1e6160 2023-11-28T20:50:12.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.424+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc1601b0 con 0x7f14bc1e6160 2023-11-28T20:50:12.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.425+0000 7f14b1ffb640 1 -- 172.21.15.45:0/2985499784 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 29) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f14b40073b0 con 0x7f14bc1e6160 2023-11-28T20:50:12.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.426+0000 7f14b1ffb640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f14a006af00 0x7f14a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.426+0000 7f14b1ffb640 1 -- 172.21.15.45:0/2985499784 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f14b40abac0 con 0x7f14bc1e6160 2023-11-28T20:50:12.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.426+0000 7f14c1352640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f14a006af00 0x7f14a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.427+0000 7f14c1352640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f14a006af00 0x7f14a006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f14b800ba20 tx=0x7f14b8008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:12.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.430+0000 7f14b1ffb640 1 -- 172.21.15.45:0/2985499784 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14bc1601b0 con 0x7f14bc1e6160 2023-11-28T20:50:12.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.663+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f14bc160170 con 0x7f14bc1e6160 2023-11-28T20:50:12.667 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T20:50:12.665+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T20:50:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.668+0000 7f14b1ffb640 1 -- 172.21.15.45:0/2985499784 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7f14b4047090 con 0x7f14bc1e6160 2023-11-28T20:50:12.670 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T20:50:12.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.672+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f14a006af00 msgr2=0x7f14a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.672+0000 7f14c2354640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f14a006af00 0x7f14a006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f14b800ba20 tx=0x7f14b8008040 comp rx=0 tx=0).stop 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.672+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14bc1e6160 msgr2=0x7f14bc1e65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.672+0000 7f14c2354640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14bc1e6160 0x7f14bc1e65e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f14b402c040 tx=0x7f14b40014a0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.673+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 shutdown_connections 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.673+0000 7f14c2354640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f14a006af00 0x7f14a006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.673+0000 7f14c2354640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14bc1e6160 0x7f14bc1e65e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.673+0000 7f14c2354640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14bc15e930 0x7f14bc15edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.673+0000 7f14c2354640 1 --2- 172.21.15.45:0/2985499784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14a8005180 0x7f14bc14f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.673+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 >> 172.21.15.45:0/2985499784 conn(0x7f14bc070f40 msgr2=0x7f14bc0cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.673+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 shutdown_connections 2023-11-28T20:50:12.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.673+0000 7f14c2354640 1 -- 172.21.15.45:0/2985499784 wait complete. 2023-11-28T20:50:12.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T20:50:12.743 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:50:12.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.889+0000 7f83124c2640 1 Processor -- start 2023-11-28T20:50:12.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.889+0000 7f83124c2640 1 -- start start 2023-11-28T20:50:12.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.890+0000 7f83124c2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f830c06ac10 0x7f830c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.891+0000 7f83124c2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c15ecd0 con 0x7f830c15f720 2023-11-28T20:50:12.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.891+0000 7f83124c2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c15f290 con 0x7f830c15e840 2023-11-28T20:50:12.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.891+0000 7f83124c2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830c15fbd0 con 0x7f830c06ac10 2023-11-28T20:50:12.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.891+0000 7f830b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f830c06ac10 0x7f830c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.891+0000 7f8310a5f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f830c15f720 0x7f830c0cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53774/0 (socket says 172.21.15.45:53774) 2023-11-28T20:50:12.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.891+0000 7f8310a5f640 1 -- 172.21.15.45:0/377501302 learned_addr learned my addr 172.21.15.45:0/377501302 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:12.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.891+0000 7f830b7fe640 1 -- 172.21.15.45:0/377501302 >> v1:172.21.15.190:6789/0 conn(0x7f830c15e840 legacy=0x7f830c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:12.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.891+0000 7f830b7fe640 1 -- 172.21.15.45:0/377501302 >> v1:172.21.15.45:6789/0 conn(0x7f830c15f720 legacy=0x7f830c0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.891+0000 7f830b7fe640 1 -- 172.21.15.45:0/377501302 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c072020 con 0x7f830c06ac10 2023-11-28T20:50:12.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.893+0000 7f830b7fe640 1 --2- 172.21.15.45:0/377501302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c06ac10 0x7f830c06c350 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f83000097a0 tx=0x7f830002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7b5bd711ab02c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:12.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.893+0000 7f830affd640 1 -- 172.21.15.45:0/377501302 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8300031830 con 0x7f830c06ac10 2023-11-28T20:50:12.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.893+0000 7f830affd640 1 -- 172.21.15.45:0/377501302 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8300031990 con 0x7f830c06ac10 2023-11-28T20:50:12.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.893+0000 7f830affd640 1 -- 172.21.15.45:0/377501302 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8300031d20 con 0x7f830c06ac10 2023-11-28T20:50:12.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.893+0000 7f83124c2640 1 -- 172.21.15.45:0/377501302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c06ac10 msgr2=0x7f830c06c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.893+0000 7f83124c2640 1 --2- 172.21.15.45:0/377501302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c06ac10 0x7f830c06c350 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f83000097a0 tx=0x7f830002b8d0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.894+0000 7f83124c2640 1 -- 172.21.15.45:0/377501302 shutdown_connections 2023-11-28T20:50:12.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.894+0000 7f83124c2640 1 --2- 172.21.15.45:0/377501302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c06ac10 0x7f830c06c350 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.894+0000 7f83124c2640 1 -- 172.21.15.45:0/377501302 >> 172.21.15.45:0/377501302 conn(0x7f830c0ced60 msgr2=0x7f830c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:12.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.894+0000 7f83124c2640 1 -- 172.21.15.45:0/377501302 shutdown_connections 2023-11-28T20:50:12.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.894+0000 7f83124c2640 1 -- 172.21.15.45:0/377501302 wait complete. 2023-11-28T20:50:12.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.895+0000 7f83124c2640 1 Processor -- start 2023-11-28T20:50:12.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.896+0000 7f83124c2640 1 -- start start 2023-11-28T20:50:12.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f83124c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f830c06ac10 0x7f830c15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f83124c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c15e840 0x7f830c0c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f83124c2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c15f720 0x7f830c0c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f83124c2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0c39c0 con 0x7f830c15e840 2023-11-28T20:50:12.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f83124c2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0b4620 con 0x7f830c15f720 2023-11-28T20:50:12.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f83124c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0c3340 con 0x7f830c06ac10 2023-11-28T20:50:12.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f830b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c15e840 0x7f830c0c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f830b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c15e840 0x7f830c0c3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49102/0 (socket says 172.21.15.45:49102) 2023-11-28T20:50:12.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f830b7fe640 1 -- 172.21.15.45:0/319390803 learned_addr learned my addr 172.21.15.45:0/319390803 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f8310a5f640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c15f720 0x7f830c0c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f830b7fe640 1 -- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f830c06ac10 msgr2=0x7f830c15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f830b7fe640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f830c06ac10 0x7f830c15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f830b7fe640 1 -- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c15f720 msgr2=0x7f830c0c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f830b7fe640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c15f720 0x7f830c0c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f830b7fe640 1 -- 172.21.15.45:0/319390803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8300009400 con 0x7f830c15e840 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.897+0000 7f8310a5f640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c15f720 0x7f830c0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.898+0000 7f830b7fe640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c15e840 0x7f830c0c3de0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f830002bee0 tx=0x7f83000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.898+0000 7f8308ff9640 1 -- 172.21.15.45:0/319390803 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830003ba70 con 0x7f830c15e840 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.898+0000 7f8308ff9640 1 -- 172.21.15.45:0/319390803 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830003bbd0 con 0x7f830c15e840 2023-11-28T20:50:12.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.898+0000 7f8308ff9640 1 -- 172.21.15.45:0/319390803 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83000453f0 con 0x7f830c15e840 2023-11-28T20:50:12.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.898+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830c0c7ac0 con 0x7f830c15e840 2023-11-28T20:50:12.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.898+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830c0c7da0 con 0x7f830c15e840 2023-11-28T20:50:12.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.899+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c0c3c30 con 0x7f830c15e840 2023-11-28T20:50:12.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.900+0000 7f8308ff9640 1 -- 172.21.15.45:0/319390803 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8300045550 con 0x7f830c15e840 2023-11-28T20:50:12.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.901+0000 7f8308ff9640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f82f806af90 0x7f82f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:12.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.901+0000 7f8308ff9640 1 -- 172.21.15.45:0/319390803 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f83000ac140 con 0x7f830c15e840 2023-11-28T20:50:12.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.902+0000 7f830bfff640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f82f806af90 0x7f82f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:12.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.903+0000 7f830bfff640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f82f806af90 0x7f82f806d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f82fc003cc0 tx=0x7f82fc0079b0 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:12.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:12.906+0000 7f8308ff9640 1 -- 172.21.15.45:0/319390803 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8300076c40 con 0x7f830c15e840 2023-11-28T20:50:13.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.133+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f830c0733d0 con 0x7f830c15e840 2023-11-28T20:50:13.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.696+0000 7f8308ff9640 1 -- 172.21.15.45:0/319390803 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f830007a260 con 0x7f830c15e840 2023-11-28T20:50:13.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f82f806af90 msgr2=0x7f82f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:13.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f82f806af90 0x7f82f806d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f82fc003cc0 tx=0x7f82fc0079b0 comp rx=0 tx=0).stop 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c15e840 msgr2=0x7f830c0c3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c15e840 0x7f830c0c3de0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f830002bee0 tx=0x7f83000046c0 comp rx=0 tx=0).stop 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 shutdown_connections 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f82f806af90 0x7f82f806d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f830c15f720 0x7f830c0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f830c15e840 0x7f830c0c3de0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 --2- 172.21.15.45:0/319390803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f830c06ac10 0x7f830c15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 >> 172.21.15.45:0/319390803 conn(0x7f830c0ced60 msgr2=0x7f830c0cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.700+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 shutdown_connections 2023-11-28T20:50:13.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.701+0000 7f83124c2640 1 -- 172.21.15.45:0/319390803 wait complete. 2023-11-28T20:50:13.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T20:50:13.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.996+0000 7fcb9c50c640 1 Processor -- start 2023-11-28T20:50:13.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.996+0000 7fcb9c50c640 1 -- start start 2023-11-28T20:50:13.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.997+0000 7fcb9c50c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.997+0000 7fcb9c50c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcb9409a150 0x7fcb940cb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.997+0000 7fcb9c50c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb940cd510 con 0x7fcb9406a830 2023-11-28T20:50:14.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.997+0000 7fcb9c50c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb940d0790 con 0x7fcb9409a150 2023-11-28T20:50:14.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.997+0000 7fcb9c50c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940cf930 con 0x7fcb940cd130 2023-11-28T20:50:14.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.998+0000 7fcb9a2a8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.998+0000 7fcb99aa7640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fcb9406a830 0x7fcb94099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:36772/0 (socket says 172.21.15.45:36772) 2023-11-28T20:50:14.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.998+0000 7fcb99aa7640 1 -- 172.21.15.45:0/4269866202 learned_addr learned my addr 172.21.15.45:0/4269866202 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:14.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.998+0000 7fcb9aaa9640 1 --2- 172.21.15.45:0/4269866202 >> v2:172.21.15.190:3300/0 conn(0x7fcb9409a150 0x7fcb940cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.999+0000 7fcb9a2a8640 1 -- 172.21.15.45:0/4269866202 >> v2:172.21.15.190:3300/0 conn(0x7fcb9409a150 msgr2=0x7fcb940cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:14.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.999+0000 7fcb9a2a8640 1 --2- 172.21.15.45:0/4269866202 >> v2:172.21.15.190:3300/0 conn(0x7fcb9409a150 0x7fcb940cb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.999+0000 7fcb9a2a8640 1 -- 172.21.15.45:0/4269866202 >> v1:172.21.15.190:6789/0 conn(0x7fcb9406a830 legacy=0x7fcb94099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:14.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.999+0000 7fcb9a2a8640 1 -- 172.21.15.45:0/4269866202 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb940cfb80 con 0x7fcb940cd130 2023-11-28T20:50:14.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:13.999+0000 7fcb9aaa9640 1 --2- 172.21.15.45:0/4269866202 >> v2:172.21.15.190:3300/0 conn(0x7fcb9409a150 0x7fcb940cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:50:14.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.000+0000 7fcb9a2a8640 1 --2- 172.21.15.45:0/4269866202 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940cf3f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcb880097a0 tx=0x7fcb8802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1891d4aa39fb13a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:14.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.001+0000 7fcb992a6640 1 -- 172.21.15.45:0/4269866202 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8802d040 con 0x7fcb940cd130 2023-11-28T20:50:14.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.001+0000 7fcb992a6640 1 -- 172.21.15.45:0/4269866202 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb88031810 con 0x7fcb940cd130 2023-11-28T20:50:14.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.001+0000 7fcb992a6640 1 -- 172.21.15.45:0/4269866202 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88031ba0 con 0x7fcb940cd130 2023-11-28T20:50:14.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.001+0000 7fcb9c50c640 1 -- 172.21.15.45:0/4269866202 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 msgr2=0x7fcb940cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:14.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.001+0000 7fcb9c50c640 1 --2- 172.21.15.45:0/4269866202 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940cf3f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcb880097a0 tx=0x7fcb8802b8b0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.002+0000 7fcb9c50c640 1 -- 172.21.15.45:0/4269866202 shutdown_connections 2023-11-28T20:50:14.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.002+0000 7fcb9c50c640 1 --2- 172.21.15.45:0/4269866202 >> v2:172.21.15.190:3300/0 conn(0x7fcb9409a150 0x7fcb940cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.002+0000 7fcb9c50c640 1 --2- 172.21.15.45:0/4269866202 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940cf3f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.002+0000 7fcb9c50c640 1 -- 172.21.15.45:0/4269866202 >> 172.21.15.45:0/4269866202 conn(0x7fcb940b2b20 msgr2=0x7fcb940b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:14.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.002+0000 7fcb9c50c640 1 -- 172.21.15.45:0/4269866202 shutdown_connections 2023-11-28T20:50:14.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.002+0000 7fcb9c50c640 1 -- 172.21.15.45:0/4269866202 wait complete. 2023-11-28T20:50:14.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.003+0000 7fcb9c50c640 1 Processor -- start 2023-11-28T20:50:14.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.004+0000 7fcb9c50c640 1 -- start start 2023-11-28T20:50:14.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.004+0000 7fcb9c50c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb9406a830 0x7fcb9415ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.004+0000 7fcb9c50c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9409a150 0x7fcb9414f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.004+0000 7fcb9c50c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.004+0000 7fcb9c50c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9414fa60 con 0x7fcb9409a150 2023-11-28T20:50:14.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.004+0000 7fcb9c50c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940c8100 con 0x7fcb9406a830 2023-11-28T20:50:14.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.004+0000 7fcb9c50c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940c83e0 con 0x7fcb940cd130 2023-11-28T20:50:14.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb99aa7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9409a150 0x7fcb9414f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb99aa7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9409a150 0x7fcb9414f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34430/0 (socket says 172.21.15.45:34430) 2023-11-28T20:50:14.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb99aa7640 1 -- 172.21.15.45:0/2153186450 learned_addr learned my addr 172.21.15.45:0/2153186450 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:14.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb9a2a8640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb9406a830 0x7fcb9415ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb9aaa9640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb99aa7640 1 -- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 msgr2=0x7fcb940c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:14.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb99aa7640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb99aa7640 1 -- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb9406a830 msgr2=0x7fcb9415ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:14.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb99aa7640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb9406a830 0x7fcb9415ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb99aa7640 1 -- 172.21.15.45:0/2153186450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c009400 con 0x7fcb9409a150 2023-11-28T20:50:14.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb9a2a8640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb9406a830 0x7fcb9415ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:14.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.005+0000 7fcb9aaa9640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:14.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.006+0000 7fcb99aa7640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9409a150 0x7fcb9414f520 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000cbc0 tx=0x7fcb9000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:14.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.006+0000 7fcb86ffd640 1 -- 172.21.15.45:0/2153186450 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9000aca0 con 0x7fcb9409a150 2023-11-28T20:50:14.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.006+0000 7fcb86ffd640 1 -- 172.21.15.45:0/2153186450 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9000ae00 con 0x7fcb9409a150 2023-11-28T20:50:14.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.006+0000 7fcb86ffd640 1 -- 172.21.15.45:0/2153186450 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90012600 con 0x7fcb9409a150 2023-11-28T20:50:14.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.006+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb88009400 con 0x7fcb9409a150 2023-11-28T20:50:14.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.006+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb940c89a0 con 0x7fcb9409a150 2023-11-28T20:50:14.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.007+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9406bd30 con 0x7fcb9409a150 2023-11-28T20:50:14.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.008+0000 7fcb86ffd640 1 -- 172.21.15.45:0/2153186450 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fcb90007070 con 0x7fcb9409a150 2023-11-28T20:50:14.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.009+0000 7fcb86ffd640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcb7806af60 0x7fcb7806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.009+0000 7fcb86ffd640 1 -- 172.21.15.45:0/2153186450 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fcb90089b10 con 0x7fcb9409a150 2023-11-28T20:50:14.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.010+0000 7fcb9a2a8640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcb7806af60 0x7fcb7806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.011+0000 7fcb9a2a8640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcb7806af60 0x7fcb7806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcb88036000 tx=0x7fcb8803c040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:14.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.013+0000 7fcb86ffd640 1 -- 172.21.15.45:0/2153186450 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb90054620 con 0x7fcb9409a150 2023-11-28T20:50:14.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.208+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcb940733d0 con 0x7fcb9409a150 2023-11-28T20:50:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.301+0000 7fcb86ffd640 1 -- 172.21.15.45:0/2153186450 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v79) v1 ==== 181+0+0 (secure 0 0 0) 0x7fcb900170d0 con 0x7fcb9409a150 2023-11-28T20:50:14.304 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T20:50:14.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.305+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcb7806af60 msgr2=0x7fcb7806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:14.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.305+0000 7fcb9c50c640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcb7806af60 0x7fcb7806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcb88036000 tx=0x7fcb8803c040 comp rx=0 tx=0).stop 2023-11-28T20:50:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.305+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9409a150 msgr2=0x7fcb9414f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.305+0000 7fcb9c50c640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9409a150 0x7fcb9414f520 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000cbc0 tx=0x7fcb9000a060 comp rx=0 tx=0).stop 2023-11-28T20:50:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.306+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 shutdown_connections 2023-11-28T20:50:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.306+0000 7fcb9c50c640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcb7806af60 0x7fcb7806d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.306+0000 7fcb9c50c640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb940cd130 0x7fcb940c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.306+0000 7fcb9c50c640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9409a150 0x7fcb9414f520 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.306+0000 7fcb9c50c640 1 --2- 172.21.15.45:0/2153186450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb9406a830 0x7fcb9415ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.306+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 >> 172.21.15.45:0/2153186450 conn(0x7fcb940b2b20 msgr2=0x7fcb940b3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:14.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.308+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 shutdown_connections 2023-11-28T20:50:14.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.309+0000 7fcb9c50c640 1 -- 172.21.15.45:0/2153186450 wait complete. 2023-11-28T20:50:14.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T20:50:14.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.541+0000 7f3b3389b640 1 Processor -- start 2023-11-28T20:50:14.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.541+0000 7f3b3389b640 1 -- start start 2023-11-28T20:50:14.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.542+0000 7f3b3389b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3b2c15d3c0 0x7f3b2c0cda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.542+0000 7f3b3389b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c15d7a0 0x7f3b2c0b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.542+0000 7f3b3389b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3b2c161090 0x7f3b2c0cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.542+0000 7f3b3389b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c0b6920 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.542+0000 7f3b3389b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c0d2e70 con 0x7f3b2c161090 2023-11-28T20:50:14.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.542+0000 7f3b3389b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c0ce110 con 0x7f3b2c15d7a0 2023-11-28T20:50:14.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31637640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3b2c15d3c0 0x7f3b2c0cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31637640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3b2c15d3c0 0x7f3b2c0cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34434/0 (socket says 172.21.15.45:34434) 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31637640 1 -- 172.21.15.45:0/2978498145 learned_addr learned my addr 172.21.15.45:0/2978498145 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31e38640 1 --2- 172.21.15.45:0/2978498145 >> v2:172.21.15.190:3300/0 conn(0x7f3b2c161090 0x7f3b2c0cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b30e36640 1 --2- 172.21.15.45:0/2978498145 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c15d7a0 0x7f3b2c0b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31637640 1 -- 172.21.15.45:0/2978498145 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c15d7a0 msgr2=0x7f3b2c0b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31637640 1 --2- 172.21.15.45:0/2978498145 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c15d7a0 0x7f3b2c0b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31637640 1 -- 172.21.15.45:0/2978498145 >> v2:172.21.15.190:3300/0 conn(0x7f3b2c161090 msgr2=0x7f3b2c0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31637640 1 --2- 172.21.15.45:0/2978498145 >> v2:172.21.15.190:3300/0 conn(0x7f3b2c161090 0x7f3b2c0cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31637640 1 -- 172.21.15.45:0/2978498145 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0ce3f0 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.543+0000 7f3b31e38640 1 --2- 172.21.15.45:0/2978498145 >> v2:172.21.15.190:3300/0 conn(0x7f3b2c161090 0x7f3b2c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:14.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.545+0000 7f3b31637640 1 --2- 172.21.15.45:0/2978498145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 0x7f3b2c0cda10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c067f60 tx=0x7f3b1c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46d059ff1684d1b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:14.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.545+0000 7f3b2bfff640 1 -- 172.21.15.45:0/2978498145 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c02c9e0 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.545+0000 7f3b2bfff640 1 -- 172.21.15.45:0/2978498145 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c02cb40 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.545+0000 7f3b2bfff640 1 -- 172.21.15.45:0/2978498145 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0343f0 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.546+0000 7f3b3389b640 1 -- 172.21.15.45:0/2978498145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 msgr2=0x7f3b2c0cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:14.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.546+0000 7f3b3389b640 1 --2- 172.21.15.45:0/2978498145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 0x7f3b2c0cda10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c067f60 tx=0x7f3b1c02be40 comp rx=0 tx=0).stop 2023-11-28T20:50:14.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.546+0000 7f3b3389b640 1 -- 172.21.15.45:0/2978498145 shutdown_connections 2023-11-28T20:50:14.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.546+0000 7f3b3389b640 1 --2- 172.21.15.45:0/2978498145 >> v2:172.21.15.190:3300/0 conn(0x7f3b2c161090 0x7f3b2c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.546+0000 7f3b3389b640 1 --2- 172.21.15.45:0/2978498145 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c15d7a0 0x7f3b2c0b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.546+0000 7f3b3389b640 1 --2- 172.21.15.45:0/2978498145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 0x7f3b2c0cda10 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.546+0000 7f3b3389b640 1 -- 172.21.15.45:0/2978498145 >> 172.21.15.45:0/2978498145 conn(0x7f3b2c0b4ca0 msgr2=0x7f3b2c0b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:14.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.546+0000 7f3b3389b640 1 -- 172.21.15.45:0/2978498145 shutdown_connections 2023-11-28T20:50:14.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.546+0000 7f3b3389b640 1 -- 172.21.15.45:0/2978498145 wait complete. 2023-11-28T20:50:14.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.548+0000 7f3b3389b640 1 Processor -- start 2023-11-28T20:50:14.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.548+0000 7f3b3389b640 1 -- start start 2023-11-28T20:50:14.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.549+0000 7f3b3389b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 0x7f3b2c1602f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.549+0000 7f3b3389b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c161090 0x7f3b2c1f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.549+0000 7f3b3389b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b3e10 0x7f3b2c14b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.549+0000 7f3b3389b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c0ca280 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.549+0000 7f3b3389b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c1f6840 con 0x7f3b2c161090 2023-11-28T20:50:14.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.549+0000 7f3b3389b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c1f6ae0 con 0x7f3b2c0b3e10 2023-11-28T20:50:14.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.549+0000 7f3b31637640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 0x7f3b2c1602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.549+0000 7f3b31637640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 0x7f3b2c1602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34436/0 (socket says 172.21.15.45:34436) 2023-11-28T20:50:14.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.549+0000 7f3b31637640 1 -- 172.21.15.45:0/902691832 learned_addr learned my addr 172.21.15.45:0/902691832 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:14.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b30e36640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c161090 0x7f3b2c1f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b31e38640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b3e10 0x7f3b2c14b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b31637640 1 -- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b3e10 msgr2=0x7f3b2c14b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:14.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b31637640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b3e10 0x7f3b2c14b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b31637640 1 -- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c161090 msgr2=0x7f3b2c1f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:14.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b31637640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c161090 0x7f3b2c1f6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:14.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b31637640 1 -- 172.21.15.45:0/902691832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c009b30 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b30e36640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c161090 0x7f3b2c1f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:14.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b31637640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 0x7f3b2c1602f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c000f80 tx=0x7f3b1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:14.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b29ffb640 1 -- 172.21.15.45:0/902691832 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c009510 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b29ffb640 1 -- 172.21.15.45:0/902691832 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c03e450 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b29ffb640 1 -- 172.21.15.45:0/902691832 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c03e800 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.550+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c1f6dc0 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.551+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c0ca490 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.552+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c1f7280 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.553+0000 7f3b29ffb640 1 -- 172.21.15.45:0/902691832 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3b1c03e960 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.553+0000 7f3b29ffb640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f3b1806af90 0x7f3b1806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:14.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.554+0000 7f3b29ffb640 1 -- 172.21.15.45:0/902691832 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6451+0+0 (secure 0 0 0) 0x7f3b1c0ac860 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.554+0000 7f3b30e36640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f3b1806af90 0x7f3b1806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:14.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.555+0000 7f3b30e36640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f3b1806af90 0x7f3b1806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c15aad0 tx=0x7f3b20008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:14.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.558+0000 7f3b29ffb640 1 -- 172.21.15.45:0/902691832 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b1c077510 con 0x7f3b2c15d3c0 2023-11-28T20:50:14.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:14.755+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3b2c0733d0 con 0x7f3b2c15d3c0 2023-11-28T20:50:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.701+0000 7f3b29ffb640 1 -- 172.21.15.45:0/902691832 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v80) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3b1c07ab30 con 0x7f3b2c15d3c0 2023-11-28T20:50:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T20:50:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.705+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f3b1806af90 msgr2=0x7f3b1806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.705+0000 7f3b3389b640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f3b1806af90 0x7f3b1806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c15aad0 tx=0x7f3b20008040 comp rx=0 tx=0).stop 2023-11-28T20:50:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.705+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 msgr2=0x7f3b2c1602f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.705+0000 7f3b3389b640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 0x7f3b2c1602f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c000f80 tx=0x7f3b1c004a40 comp rx=0 tx=0).stop 2023-11-28T20:50:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.706+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 shutdown_connections 2023-11-28T20:50:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.706+0000 7f3b3389b640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f3b1806af90 0x7f3b1806d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.706+0000 7f3b3389b640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b3e10 0x7f3b2c14b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.706+0000 7f3b3389b640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c161090 0x7f3b2c1f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.706+0000 7f3b3389b640 1 --2- 172.21.15.45:0/902691832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b2c15d3c0 0x7f3b2c1602f0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.706+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 >> 172.21.15.45:0/902691832 conn(0x7f3b2c0b4ca0 msgr2=0x7f3b2c0d23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.706+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 shutdown_connections 2023-11-28T20:50:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.707+0000 7f3b3389b640 1 -- 172.21.15.45:0/902691832 wait complete. 2023-11-28T20:50:15.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T20:50:15.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.938+0000 7fdf98a10640 1 Processor -- start 2023-11-28T20:50:15.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.938+0000 7fdf98a10640 1 -- start start 2023-11-28T20:50:15.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.940+0000 7fdf98a10640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdf940b3490 0x7fdf9415e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:15.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.940+0000 7fdf98a10640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9406dbe0 con 0x7fdf940b2410 2023-11-28T20:50:15.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.940+0000 7fdf98a10640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf940b3870 con 0x7fdf94071f20 2023-11-28T20:50:15.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.940+0000 7fdf98a10640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9406e110 con 0x7fdf940b3490 2023-11-28T20:50:15.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.940+0000 7fdf93fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdf940b2410 0x7fdf94160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56744/0 (socket says 172.21.15.45:56744) 2023-11-28T20:50:15.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.940+0000 7fdf93fff640 1 -- 172.21.15.45:0/1394148985 learned_addr learned my addr 172.21.15.45:0/1394148985 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:15.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624002987 0 0) 0x7fdf9406dbe0 con 0x7fdf940b2410 2023-11-28T20:50:15.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf7c003520 con 0x7fdf940b2410 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf937fe640 1 --2- 172.21.15.45:0/1394148985 >> v2:172.21.15.190:3300/0 conn(0x7fdf940b3490 0x7fdf9415e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718781890 0 0) 0x7fdf7c003520 con 0x7fdf940b2410 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 >> v2:172.21.15.190:3300/0 conn(0x7fdf940b3490 msgr2=0x7fdf9415e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf927fc640 1 --2- 172.21.15.45:0/1394148985 >> v2:172.21.15.190:3300/0 conn(0x7fdf940b3490 0x7fdf9415e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 >> v1:172.21.15.190:6789/0 conn(0x7fdf94071f20 legacy=0x7fdf940b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9406de30 con 0x7fdf940b2410 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fdf8c0019f0 con 0x7fdf940b2410 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.941+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 >> v1:172.21.15.45:6789/0 conn(0x7fdf940b2410 legacy=0x7fdf94160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf927fc640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7c003970 0x7fdf7c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf927fc640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7c0047c0 0x7fdf7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf927fc640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf7c005180 0x7fdf7c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:15.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9406e110 con 0x7fdf7c005180 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c00c0d0 con 0x7fdf7c0047c0 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c00c290 con 0x7fdf7c003970 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf93fff640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf7c005180 0x7fdf7c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf937fe640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7c003970 0x7fdf7c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf92ffd640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7c0047c0 0x7fdf7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf93fff640 1 -- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7c003970 msgr2=0x7fdf7c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf93fff640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7c003970 0x7fdf7c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf93fff640 1 -- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7c0047c0 msgr2=0x7fdf7c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf93fff640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7c0047c0 0x7fdf7c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.942+0000 7fdf93fff640 1 -- 172.21.15.45:0/1394148985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c003520 con 0x7fdf7c005180 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.943+0000 7fdf937fe640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7c003970 0x7fdf7c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.943+0000 7fdf92ffd640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf7c0047c0 0x7fdf7c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.943+0000 7fdf93fff640 1 -- 172.21.15.45:0/1394148985 reap_dead start 2023-11-28T20:50:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.944+0000 7fdf93fff640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf7c005180 0x7fdf7c00b980 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00a830 tx=0x7fdf7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfefa1b5bb0bdd8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.944+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c0072d0 con 0x7fdf7c005180 2023-11-28T20:50:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.944+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c02f070 con 0x7fdf7c005180 2023-11-28T20:50:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.944+0000 7fdf927fc640 1 -- 172.21.15.45:0/1394148985 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c0076b0 con 0x7fdf7c005180 2023-11-28T20:50:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.945+0000 7fdf98a10640 1 -- 172.21.15.45:0/1394148985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf7c005180 msgr2=0x7fdf7c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.945+0000 7fdf98a10640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf7c005180 0x7fdf7c00b980 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00a830 tx=0x7fdf7c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.945+0000 7fdf98a10640 1 -- 172.21.15.45:0/1394148985 shutdown_connections 2023-11-28T20:50:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.945+0000 7fdf98a10640 1 --2- 172.21.15.45:0/1394148985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf7c005180 0x7fdf7c00b980 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.945+0000 7fdf98a10640 1 -- 172.21.15.45:0/1394148985 >> 172.21.15.45:0/1394148985 conn(0x7fdf940cecd0 msgr2=0x7fdf940cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.945+0000 7fdf98a10640 1 -- 172.21.15.45:0/1394148985 shutdown_connections 2023-11-28T20:50:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.945+0000 7fdf98a10640 1 -- 172.21.15.45:0/1394148985 wait complete. 2023-11-28T20:50:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.947+0000 7fdf98a10640 1 Processor -- start 2023-11-28T20:50:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.947+0000 7fdf98a10640 1 -- start start 2023-11-28T20:50:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf98a10640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7c005180 0x7fdf941f6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf98a10640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf940c8320 0x7fdf9414e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf98a10640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf940ad4b0 0x7fdf940ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf98a10640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9414b160 con 0x7fdf940c8320 2023-11-28T20:50:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf98a10640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf940ade70 con 0x7fdf940ad4b0 2023-11-28T20:50:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf98a10640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf941f6ff0 con 0x7fdf7c005180 2023-11-28T20:50:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf92ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf940c8320 0x7fdf9414e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf92ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf940c8320 0x7fdf9414e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34462/0 (socket says 172.21.15.45:34462) 2023-11-28T20:50:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf92ffd640 1 -- 172.21.15.45:0/3194581433 learned_addr learned my addr 172.21.15.45:0/3194581433 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf92ffd640 1 -- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7c005180 msgr2=0x7fdf941f6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf92ffd640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7c005180 0x7fdf941f6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf92ffd640 1 -- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf940ad4b0 msgr2=0x7fdf940ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf92ffd640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf940ad4b0 0x7fdf940ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.948+0000 7fdf92ffd640 1 -- 172.21.15.45:0/3194581433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9414b3b0 con 0x7fdf940c8320 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.949+0000 7fdf92ffd640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf940c8320 0x7fdf9414e850 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fdf88000c00 tx=0x7fdf8800e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.949+0000 7fdf7bfff640 1 -- 172.21.15.45:0/3194581433 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf880052e0 con 0x7fdf940c8320 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.949+0000 7fdf7bfff640 1 -- 172.21.15.45:0/3194581433 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf88005440 con 0x7fdf940c8320 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.949+0000 7fdf7bfff640 1 -- 172.21.15.45:0/3194581433 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf88003810 con 0x7fdf940c8320 2023-11-28T20:50:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.949+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9414b690 con 0x7fdf940c8320 2023-11-28T20:50:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.949+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9414bc50 con 0x7fdf940c8320 2023-11-28T20:50:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.950+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf941490d0 con 0x7fdf940c8320 2023-11-28T20:50:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.951+0000 7fdf7bfff640 1 -- 172.21.15.45:0/3194581433 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fdf88003970 con 0x7fdf940c8320 2023-11-28T20:50:15.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.952+0000 7fdf7bfff640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fdf8006af60 0x7fdf8006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:15.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.952+0000 7fdf7bfff640 1 -- 172.21.15.45:0/3194581433 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fdf88088f90 con 0x7fdf940c8320 2023-11-28T20:50:15.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.953+0000 7fdf937fe640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fdf8006af60 0x7fdf8006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:15.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.954+0000 7fdf937fe640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fdf8006af60 0x7fdf8006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf84006f00 tx=0x7fdf84006e90 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:15.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:15.957+0000 7fdf7bfff640 1 -- 172.21.15.45:0/3194581433 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf88053ee0 con 0x7fdf940c8320 2023-11-28T20:50:16.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.152+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fdf9409d0f0 con 0x7fdf940c8320 2023-11-28T20:50:16.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.157+0000 7fdf7bfff640 1 -- 172.21.15.45:0/3194581433 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v19) v1 ==== 120+0+0 (secure 0 0 0) 0x7fdf880190b0 con 0x7fdf940c8320 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.160+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fdf8006af60 msgr2=0x7fdf8006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.160+0000 7fdf98a10640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fdf8006af60 0x7fdf8006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf84006f00 tx=0x7fdf84006e90 comp rx=0 tx=0).stop 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.160+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf940c8320 msgr2=0x7fdf9414e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.160+0000 7fdf98a10640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf940c8320 0x7fdf9414e850 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fdf88000c00 tx=0x7fdf8800e8f0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.160+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 shutdown_connections 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.161+0000 7fdf98a10640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fdf8006af60 0x7fdf8006d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.161+0000 7fdf98a10640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf940ad4b0 0x7fdf940ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.161+0000 7fdf98a10640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf940c8320 0x7fdf9414e850 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.161+0000 7fdf98a10640 1 --2- 172.21.15.45:0/3194581433 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdf7c005180 0x7fdf941f6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.161+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 >> 172.21.15.45:0/3194581433 conn(0x7fdf940cecd0 msgr2=0x7fdf940af9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:16.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.161+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 shutdown_connections 2023-11-28T20:50:16.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.161+0000 7fdf98a10640 1 -- 172.21.15.45:0/3194581433 wait complete. 2023-11-28T20:50:16.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T20:50:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.408+0000 7f5a6f59e640 1 Processor -- start 2023-11-28T20:50:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.408+0000 7f5a6f59e640 1 -- start start 2023-11-28T20:50:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.410+0000 7f5a6f59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a70071fd0 0x7f5a700ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.410+0000 7f5a6f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5a700abd70 0x7f5a700702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.410+0000 7f5a6f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700ac270 con 0x7f5a7015b080 2023-11-28T20:50:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.410+0000 7f5a6f59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7015b830 con 0x7f5a70071fd0 2023-11-28T20:50:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.410+0000 7f5a6f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7015bb10 con 0x7f5a700abd70 2023-11-28T20:50:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.411+0000 7f5a67fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a70071fd0 0x7f5a700ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.411+0000 7f5a67fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a70071fd0 0x7f5a700ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34470/0 (socket says 172.21.15.45:34470) 2023-11-28T20:50:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.411+0000 7f5a6e59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5a7015b080 0x7f5a700718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56756/0 (socket says 172.21.15.45:56756) 2023-11-28T20:50:16.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.411+0000 7f5a67fff640 1 -- 172.21.15.45:0/14794932 learned_addr learned my addr 172.21.15.45:0/14794932 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:16.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.411+0000 7f5a67fff640 1 -- 172.21.15.45:0/14794932 >> v2:172.21.15.190:3300/0 conn(0x7f5a700abd70 msgr2=0x7f5a700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:16.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.411+0000 7f5a67fff640 1 --2- 172.21.15.45:0/14794932 >> v2:172.21.15.190:3300/0 conn(0x7f5a700abd70 0x7f5a700702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.411+0000 7f5a67fff640 1 -- 172.21.15.45:0/14794932 >> v1:172.21.15.45:6789/0 conn(0x7f5a7015b080 legacy=0x7f5a700718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.411+0000 7f5a67fff640 1 -- 172.21.15.45:0/14794932 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7015bdf0 con 0x7f5a70071fd0 2023-11-28T20:50:16.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.412+0000 7f5a67fff640 1 --2- 172.21.15.45:0/14794932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a70071fd0 0x7f5a700ae370 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5a70072a60 tx=0x7f5a5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63d5f9c95ad80dc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.413+0000 7f5a6dd9b640 1 -- 172.21.15.45:0/14794932 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58030860 con 0x7f5a70071fd0 2023-11-28T20:50:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.413+0000 7f5a6dd9b640 1 -- 172.21.15.45:0/14794932 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a580309c0 con 0x7f5a70071fd0 2023-11-28T20:50:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.413+0000 7f5a6dd9b640 1 -- 172.21.15.45:0/14794932 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58030d50 con 0x7f5a70071fd0 2023-11-28T20:50:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.413+0000 7f5a6f59e640 1 -- 172.21.15.45:0/14794932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a70071fd0 msgr2=0x7f5a700ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.413+0000 7f5a6f59e640 1 --2- 172.21.15.45:0/14794932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a70071fd0 0x7f5a700ae370 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5a70072a60 tx=0x7f5a5802b9a0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.414+0000 7f5a6f59e640 1 -- 172.21.15.45:0/14794932 shutdown_connections 2023-11-28T20:50:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.414+0000 7f5a6f59e640 1 --2- 172.21.15.45:0/14794932 >> v2:172.21.15.190:3300/0 conn(0x7f5a700abd70 0x7f5a700702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.414+0000 7f5a6f59e640 1 --2- 172.21.15.45:0/14794932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a70071fd0 0x7f5a700ae370 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.414+0000 7f5a6f59e640 1 -- 172.21.15.45:0/14794932 >> 172.21.15.45:0/14794932 conn(0x7f5a700cecd0 msgr2=0x7f5a700cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.414+0000 7f5a6f59e640 1 -- 172.21.15.45:0/14794932 shutdown_connections 2023-11-28T20:50:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.414+0000 7f5a6f59e640 1 -- 172.21.15.45:0/14794932 wait complete. 2023-11-28T20:50:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.416+0000 7f5a6f59e640 1 Processor -- start 2023-11-28T20:50:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.416+0000 7f5a6f59e640 1 -- start start 2023-11-28T20:50:16.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.417+0000 7f5a6f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a700abd70 0x7f5a7014e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.417+0000 7f5a6f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a7015b080 0x7f5a7015d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.417+0000 7f5a6f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a7014e660 0x7f5a7015dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.417+0000 7f5a6f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a700b2bb0 con 0x7f5a7014e660 2023-11-28T20:50:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.417+0000 7f5a6f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a700b1fa0 con 0x7f5a7015b080 2023-11-28T20:50:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.417+0000 7f5a6f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a700b2280 con 0x7f5a700abd70 2023-11-28T20:50:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.417+0000 7f5a6ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a7014e660 0x7f5a7015dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.417+0000 7f5a6ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a7014e660 0x7f5a7015dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34486/0 (socket says 172.21.15.45:34486) 2023-11-28T20:50:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.417+0000 7f5a6ed9d640 1 -- 172.21.15.45:0/1251310807 learned_addr learned my addr 172.21.15.45:0/1251310807 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:16.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a6e59c640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a700abd70 0x7f5a7014e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:16.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a67fff640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a7015b080 0x7f5a7015d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:16.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a6ed9d640 1 -- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a700abd70 msgr2=0x7f5a7014e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a6ed9d640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a700abd70 0x7f5a7014e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a6ed9d640 1 -- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a7015b080 msgr2=0x7f5a7015d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a6ed9d640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a7015b080 0x7f5a7015d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a6ed9d640 1 -- 172.21.15.45:0/1251310807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a580094d0 con 0x7f5a7014e660 2023-11-28T20:50:16.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a6ed9d640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a7014e660 0x7f5a7015dbd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800b9e0 tx=0x7f5a6800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:16.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a66ffd640 1 -- 172.21.15.45:0/1251310807 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6800cb20 con 0x7f5a7014e660 2023-11-28T20:50:16.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a66ffd640 1 -- 172.21.15.45:0/1251310807 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6800cca0 con 0x7f5a7014e660 2023-11-28T20:50:16.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a700b2560 con 0x7f5a7014e660 2023-11-28T20:50:16.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a66ffd640 1 -- 172.21.15.45:0/1251310807 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a68014680 con 0x7f5a7014e660 2023-11-28T20:50:16.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.418+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7009b5d0 con 0x7f5a7014e660 2023-11-28T20:50:16.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.420+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7009d0f0 con 0x7f5a7014e660 2023-11-28T20:50:16.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.421+0000 7f5a66ffd640 1 -- 172.21.15.45:0/1251310807 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5a6801d3b0 con 0x7f5a7014e660 2023-11-28T20:50:16.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.422+0000 7f5a66ffd640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f5a5006af90 0x7f5a5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.422+0000 7f5a66ffd640 1 -- 172.21.15.45:0/1251310807 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f5a680898d0 con 0x7f5a7014e660 2023-11-28T20:50:16.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.422+0000 7f5a6e59c640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f5a5006af90 0x7f5a5006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:16.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.424+0000 7f5a6e59c640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f5a5006af90 0x7f5a5006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a60001580 tx=0x7f5a60009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:16.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.427+0000 7f5a66ffd640 1 -- 172.21.15.45:0/1251310807 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a68054660 con 0x7f5a7014e660 2023-11-28T20:50:16.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.624+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5a700b2cf0 con 0x7f5a7014e660 2023-11-28T20:50:16.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.631+0000 7f5a66ffd640 1 -- 172.21.15.45:0/1251310807 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5a700b2cf0 con 0x7f5a7014e660 2023-11-28T20:50:16.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.634+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f5a5006af90 msgr2=0x7f5a5006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.634+0000 7f5a6f59e640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f5a5006af90 0x7f5a5006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a60001580 tx=0x7f5a60009040 comp rx=0 tx=0).stop 2023-11-28T20:50:16.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.634+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a7014e660 msgr2=0x7f5a7015dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.634+0000 7f5a6f59e640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a7014e660 0x7f5a7015dbd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800b9e0 tx=0x7f5a6800bea0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.635+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 shutdown_connections 2023-11-28T20:50:16.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.635+0000 7f5a6f59e640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f5a5006af90 0x7f5a5006d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.635+0000 7f5a6f59e640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a7014e660 0x7f5a7015dbd0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.635+0000 7f5a6f59e640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a7015b080 0x7f5a7015d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.635+0000 7f5a6f59e640 1 --2- 172.21.15.45:0/1251310807 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a700abd70 0x7f5a7014e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.635+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 >> 172.21.15.45:0/1251310807 conn(0x7f5a700cecd0 msgr2=0x7f5a700af790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:16.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.635+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 shutdown_connections 2023-11-28T20:50:16.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.635+0000 7f5a6f59e640 1 -- 172.21.15.45:0/1251310807 wait complete. 2023-11-28T20:50:16.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T20:50:16.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.893+0000 7fa1eeec3640 1 Processor -- start 2023-11-28T20:50:16.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.893+0000 7fa1eeec3640 1 -- start start 2023-11-28T20:50:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.895+0000 7fa1eeec3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1e815b080 0x7fa1e80ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.895+0000 7fa1eeec3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa1e80abd70 0x7fa1e80718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.895+0000 7fa1eeec3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e80ac270 con 0x7fa1e8071e00 2023-11-28T20:50:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.895+0000 7fa1eeec3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e8072270 con 0x7fa1e815b080 2023-11-28T20:50:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.895+0000 7fa1eeec3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e815b830 con 0x7fa1e80abd70 2023-11-28T20:50:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.895+0000 7fa1edec1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1e815b080 0x7fa1e80ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.895+0000 7fa1edec1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1e815b080 0x7fa1e80ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34502/0 (socket says 172.21.15.45:34502) 2023-11-28T20:50:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.895+0000 7fa1edec1640 1 -- 172.21.15.45:0/3568953436 learned_addr learned my addr 172.21.15.45:0/3568953436 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.896+0000 7fa1edec1640 1 -- 172.21.15.45:0/3568953436 >> v2:172.21.15.190:3300/0 conn(0x7fa1e80abd70 msgr2=0x7fa1e80718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.896+0000 7fa1edec1640 1 --2- 172.21.15.45:0/3568953436 >> v2:172.21.15.190:3300/0 conn(0x7fa1e80abd70 0x7fa1e80718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.896+0000 7fa1edec1640 1 -- 172.21.15.45:0/3568953436 >> v1:172.21.15.190:6789/0 conn(0x7fa1e8071e00 legacy=0x7fa1e815eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.896+0000 7fa1edec1640 1 -- 172.21.15.45:0/3568953436 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e815bb10 con 0x7fa1e815b080 2023-11-28T20:50:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.897+0000 7fa1edec1640 1 --2- 172.21.15.45:0/3568953436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e815b080 0x7fa1e80ae370 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa1e806b910 tx=0x7fa1e402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58311ae39ef59152 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.897+0000 7fa1ecebf640 1 -- 172.21.15.45:0/3568953436 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e402ca80 con 0x7fa1e815b080 2023-11-28T20:50:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.897+0000 7fa1ecebf640 1 -- 172.21.15.45:0/3568953436 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e402cbe0 con 0x7fa1e815b080 2023-11-28T20:50:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.897+0000 7fa1ecebf640 1 -- 172.21.15.45:0/3568953436 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40343f0 con 0x7fa1e815b080 2023-11-28T20:50:16.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.898+0000 7fa1eeec3640 1 -- 172.21.15.45:0/3568953436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e815b080 msgr2=0x7fa1e80ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.898+0000 7fa1eeec3640 1 --2- 172.21.15.45:0/3568953436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e815b080 0x7fa1e80ae370 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa1e806b910 tx=0x7fa1e402bee0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.898+0000 7fa1eeec3640 1 -- 172.21.15.45:0/3568953436 shutdown_connections 2023-11-28T20:50:16.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.898+0000 7fa1eeec3640 1 --2- 172.21.15.45:0/3568953436 >> v2:172.21.15.190:3300/0 conn(0x7fa1e80abd70 0x7fa1e80718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.898+0000 7fa1eeec3640 1 --2- 172.21.15.45:0/3568953436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e815b080 0x7fa1e80ae370 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.898+0000 7fa1eeec3640 1 -- 172.21.15.45:0/3568953436 >> 172.21.15.45:0/3568953436 conn(0x7fa1e80cecd0 msgr2=0x7fa1e80cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:16.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.898+0000 7fa1eeec3640 1 -- 172.21.15.45:0/3568953436 shutdown_connections 2023-11-28T20:50:16.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.898+0000 7fa1eeec3640 1 -- 172.21.15.45:0/3568953436 wait complete. 2023-11-28T20:50:16.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.900+0000 7fa1eeec3640 1 Processor -- start 2023-11-28T20:50:16.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.900+0000 7fa1eeec3640 1 -- start start 2023-11-28T20:50:16.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.901+0000 7fa1eeec3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e8071e00 0x7fa1e8157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.901+0000 7fa1eeec3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1e80abd70 0x7fa1e814cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.901+0000 7fa1eeec3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1e815b080 0x7fa1e81514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.901+0000 7fa1eeec3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e809b210 con 0x7fa1e8071e00 2023-11-28T20:50:16.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.901+0000 7fa1eeec3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81521e0 con 0x7fa1e80abd70 2023-11-28T20:50:16.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.901+0000 7fa1eeec3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e8151f00 con 0x7fa1e815b080 2023-11-28T20:50:16.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1edec1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e8071e00 0x7fa1e8157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:16.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1edec1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e8071e00 0x7fa1e8157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34512/0 (socket says 172.21.15.45:34512) 2023-11-28T20:50:16.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1edec1640 1 -- 172.21.15.45:0/4048929261 learned_addr learned my addr 172.21.15.45:0/4048929261 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:16.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1edec1640 1 -- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1e815b080 msgr2=0x7fa1e81514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1ed6c0640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1e80abd70 0x7fa1e814cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:16.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1edec1640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1e815b080 0x7fa1e81514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1edec1640 1 -- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1e80abd70 msgr2=0x7fa1e814cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:16.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1edec1640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1e80abd70 0x7fa1e814cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:16.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1edec1640 1 -- 172.21.15.45:0/4048929261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e4009b30 con 0x7fa1e8071e00 2023-11-28T20:50:16.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1ed6c0640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1e80abd70 0x7fa1e814cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:16.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.902+0000 7fa1edec1640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e8071e00 0x7fa1e8157040 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40093e0 tx=0x7fa1e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:16.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.903+0000 7fa1de7fc640 1 -- 172.21.15.45:0/4048929261 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e403e450 con 0x7fa1e8071e00 2023-11-28T20:50:16.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.903+0000 7fa1de7fc640 1 -- 172.21.15.45:0/4048929261 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e403e5b0 con 0x7fa1e8071e00 2023-11-28T20:50:16.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.903+0000 7fa1de7fc640 1 -- 172.21.15.45:0/4048929261 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e403e960 con 0x7fa1e8071e00 2023-11-28T20:50:16.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.903+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e809abc0 con 0x7fa1e8071e00 2023-11-28T20:50:16.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.903+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e8152b20 con 0x7fa1e8071e00 2023-11-28T20:50:16.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.904+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e8152cb0 con 0x7fa1e8071e00 2023-11-28T20:50:16.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.905+0000 7fa1de7fc640 1 -- 172.21.15.45:0/4048929261 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa1e402cd50 con 0x7fa1e8071e00 2023-11-28T20:50:16.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.906+0000 7fa1de7fc640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa1cc06af60 0x7fa1cc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:16.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.906+0000 7fa1de7fc640 1 -- 172.21.15.45:0/4048929261 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fa1e40ac560 con 0x7fa1e8071e00 2023-11-28T20:50:16.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.907+0000 7fa1ed6c0640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa1cc06af60 0x7fa1cc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:16.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.908+0000 7fa1ed6c0640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa1cc06af60 0x7fa1cc06d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8006fd0 tx=0x7fa1d8008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:16.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:16.911+0000 7fa1de7fc640 1 -- 172.21.15.45:0/4048929261 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1e4077400 con 0x7fa1e8071e00 2023-11-28T20:50:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.113+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fa1e809d0f0 con 0x7fa1e8071e00 2023-11-28T20:50:17.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.119+0000 7fa1de7fc640 1 -- 172.21.15.45:0/4048929261 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa1e403eac0 con 0x7fa1e8071e00 2023-11-28T20:50:17.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa1cc06af60 msgr2=0x7fa1cc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa1cc06af60 0x7fa1cc06d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8006fd0 tx=0x7fa1d8008040 comp rx=0 tx=0).stop 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e8071e00 msgr2=0x7fa1e8157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e8071e00 0x7fa1e8157040 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40093e0 tx=0x7fa1e4004a40 comp rx=0 tx=0).stop 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 shutdown_connections 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa1cc06af60 0x7fa1cc06d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1e815b080 0x7fa1e81514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1e80abd70 0x7fa1e814cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 --2- 172.21.15.45:0/4048929261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e8071e00 0x7fa1e8157040 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 >> 172.21.15.45:0/4048929261 conn(0x7fa1e80cecd0 msgr2=0x7fa1e80ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 shutdown_connections 2023-11-28T20:50:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.122+0000 7fa1eeec3640 1 -- 172.21.15.45:0/4048929261 wait complete. 2023-11-28T20:50:17.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-11-28T20:50:17.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.360+0000 7f2dc8b27640 1 Processor -- start 2023-11-28T20:50:17.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.360+0000 7f2dc8b27640 1 -- start start 2023-11-28T20:50:17.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.362+0000 7f2dc8b27640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dc4155780 0x7f2dc40abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.362+0000 7f2dc8b27640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2dc40ae540 0x7f2dc40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.362+0000 7f2dc8b27640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2dc40ce630 0x7f2dc415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.362+0000 7f2dc8b27640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc40ceac0 con 0x7f2dc40ae540 2023-11-28T20:50:17.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.362+0000 7f2dc8b27640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc415c3c0 con 0x7f2dc40ce630 2023-11-28T20:50:17.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.362+0000 7f2dc8b27640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc4155b60 con 0x7f2dc4155780 2023-11-28T20:50:17.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dbbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2dc40ae540 0x7f2dc40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dbbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2dc40ae540 0x7f2dc40ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34518/0 (socket says 172.21.15.45:34518) 2023-11-28T20:50:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dbbfff640 1 -- 172.21.15.45:0/1572256055 learned_addr learned my addr 172.21.15.45:0/1572256055 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dc37fe640 1 --2- 172.21.15.45:0/1572256055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dc4155780 0x7f2dc40abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dc3fff640 1 --2- 172.21.15.45:0/1572256055 >> v2:172.21.15.190:3300/0 conn(0x7f2dc40ce630 0x7f2dc415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dbbfff640 1 -- 172.21.15.45:0/1572256055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dc4155780 msgr2=0x7f2dc40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dbbfff640 1 --2- 172.21.15.45:0/1572256055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dc4155780 0x7f2dc40abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dbbfff640 1 -- 172.21.15.45:0/1572256055 >> v2:172.21.15.190:3300/0 conn(0x7f2dc40ce630 msgr2=0x7f2dc415ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dbbfff640 1 --2- 172.21.15.45:0/1572256055 >> v2:172.21.15.190:3300/0 conn(0x7f2dc40ce630 0x7f2dc415ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dbbfff640 1 -- 172.21.15.45:0/1572256055 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc415ba90 con 0x7f2dc40ae540 2023-11-28T20:50:17.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.363+0000 7f2dc3fff640 1 --2- 172.21.15.45:0/1572256055 >> v2:172.21.15.190:3300/0 conn(0x7f2dc40ce630 0x7f2dc415ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.364+0000 7f2dbbfff640 1 --2- 172.21.15.45:0/1572256055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ae540 0x7f2dc40ce0f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009870 tx=0x7f2dac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ade3ea97e4f36b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.365+0000 7f2dc2ffd640 1 -- 172.21.15.45:0/1572256055 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac031640 con 0x7f2dc40ae540 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.365+0000 7f2dc2ffd640 1 -- 172.21.15.45:0/1572256055 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac0317a0 con 0x7f2dc40ae540 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.365+0000 7f2dc2ffd640 1 -- 172.21.15.45:0/1572256055 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac031b30 con 0x7f2dc40ae540 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.366+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1572256055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ae540 msgr2=0x7f2dc40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.366+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1572256055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ae540 0x7f2dc40ce0f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009870 tx=0x7f2dac02b8f0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.366+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1572256055 shutdown_connections 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.366+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1572256055 >> v2:172.21.15.190:3300/0 conn(0x7f2dc40ce630 0x7f2dc415ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.366+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1572256055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ae540 0x7f2dc40ce0f0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.366+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1572256055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dc4155780 0x7f2dc40abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.366+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1572256055 >> 172.21.15.45:0/1572256055 conn(0x7f2dc406ab30 msgr2=0x7f2dc40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:17.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.366+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1572256055 shutdown_connections 2023-11-28T20:50:17.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.366+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1572256055 wait complete. 2023-11-28T20:50:17.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.368+0000 7f2dc8b27640 1 Processor -- start 2023-11-28T20:50:17.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.368+0000 7f2dc8b27640 1 -- start start 2023-11-28T20:50:17.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.369+0000 7f2dc8b27640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dc40ae540 0x7f2dc41525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.369+0000 7f2dc8b27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ce630 0x7f2dc4147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.369+0000 7f2dc8b27640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dc4155780 0x7f2dc4147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.369+0000 7f2dc8b27640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc41480e0 con 0x7f2dc40ce630 2023-11-28T20:50:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.369+0000 7f2dc8b27640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc409ae30 con 0x7f2dc40ae540 2023-11-28T20:50:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.369+0000 7f2dc8b27640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc4099d90 con 0x7f2dc4155780 2023-11-28T20:50:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.369+0000 7f2dbbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ce630 0x7f2dc4147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.369+0000 7f2dbbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ce630 0x7f2dc4147660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34528/0 (socket says 172.21.15.45:34528) 2023-11-28T20:50:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.369+0000 7f2dbbfff640 1 -- 172.21.15.45:0/1025927993 learned_addr learned my addr 172.21.15.45:0/1025927993 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dbbfff640 1 -- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dc4155780 msgr2=0x7f2dc4147ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dbbfff640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dc4155780 0x7f2dc4147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dbbfff640 1 -- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dc40ae540 msgr2=0x7f2dc41525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dbbfff640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dc40ae540 0x7f2dc41525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dbbfff640 1 -- 172.21.15.45:0/1025927993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac0094d0 con 0x7f2dc40ce630 2023-11-28T20:50:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dbbfff640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ce630 0x7f2dc4147660 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f2dac000c00 tx=0x7f2dac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dc0ff9640 1 -- 172.21.15.45:0/1025927993 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac03ac60 con 0x7f2dc40ce630 2023-11-28T20:50:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dc0ff9640 1 -- 172.21.15.45:0/1025927993 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac03adc0 con 0x7f2dc40ce630 2023-11-28T20:50:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dc0ff9640 1 -- 172.21.15.45:0/1025927993 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac044630 con 0x7f2dc40ce630 2023-11-28T20:50:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.370+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc409a070 con 0x7f2dc40ce630 2023-11-28T20:50:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.371+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc409a630 con 0x7f2dc40ce630 2023-11-28T20:50:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.372+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc404e920 con 0x7f2dc40ce630 2023-11-28T20:50:17.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.373+0000 7f2dc0ff9640 1 -- 172.21.15.45:0/1025927993 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2dac02f050 con 0x7f2dc40ce630 2023-11-28T20:50:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.374+0000 7f2dc0ff9640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f2da406af60 0x7f2da406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.375+0000 7f2dc0ff9640 1 -- 172.21.15.45:0/1025927993 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f2dac0ab810 con 0x7f2dc40ce630 2023-11-28T20:50:17.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.375+0000 7f2dc37fe640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f2da406af60 0x7f2da406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.376+0000 7f2dc37fe640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f2da406af60 0x7f2da406d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2db4006fd0 tx=0x7f2db4008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:17.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.380+0000 7f2dc0ff9640 1 -- 172.21.15.45:0/1025927993 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dac076640 con 0x7f2dc40ce630 2023-11-28T20:50:17.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.648+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f2dc4154c20 con 0x7f2dc40ce630 2023-11-28T20:50:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.709+0000 7f2dc0ff9640 1 -- 172.21.15.45:0/1025927993 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v603) v1 ==== 154+0+0 (secure 0 0 0) 0x7f2dac031e60 con 0x7f2dc40ce630 2023-11-28T20:50:17.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f2da406af60 msgr2=0x7f2da406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f2da406af60 0x7f2da406d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2db4006fd0 tx=0x7f2db4008040 comp rx=0 tx=0).stop 2023-11-28T20:50:17.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ce630 msgr2=0x7f2dc4147660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ce630 0x7f2dc4147660 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f2dac000c00 tx=0x7f2dac0046c0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 shutdown_connections 2023-11-28T20:50:17.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f2da406af60 0x7f2da406d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dc4155780 0x7f2dc4147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dc40ce630 0x7f2dc4147660 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 --2- 172.21.15.45:0/1025927993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dc40ae540 0x7f2dc41525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 >> 172.21.15.45:0/1025927993 conn(0x7f2dc406ab30 msgr2=0x7f2dc415d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:17.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 shutdown_connections 2023-11-28T20:50:17.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.714+0000 7f2dc8b27640 1 -- 172.21.15.45:0/1025927993 wait complete. 2023-11-28T20:50:17.736 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-11-28T20:50:17.737 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-11-28T20:50:17.737 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T20:50:17.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:17.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.964+0000 7f1328a75640 1 Processor -- start 2023-11-28T20:50:17.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.964+0000 7f1328a75640 1 -- start start 2023-11-28T20:50:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.966+0000 7f1328a75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132415b850 0x7f13240b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.966+0000 7f1328a75640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f13240b1fa0 0x7f13240b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.966+0000 7f1328a75640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132415bda0 con 0x7f13240b35b0 2023-11-28T20:50:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.966+0000 7f1328a75640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f132415ee40 con 0x7f13240b1fa0 2023-11-28T20:50:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.966+0000 7f1328a75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f132415c080 con 0x7f132415b850 2023-11-28T20:50:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.966+0000 7f1321d9b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132415b850 0x7f13240b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.967+0000 7f1321d9b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132415b850 0x7f13240b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:49278/0 (socket says 172.21.15.45:49278) 2023-11-28T20:50:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.967+0000 7f132259c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f13240b35b0 0x7f132415ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:36796/0 (socket says 172.21.15.45:36796) 2023-11-28T20:50:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.967+0000 7f1321d9b640 1 -- 172.21.15.45:0/1333171661 learned_addr learned my addr 172.21.15.45:0/1333171661 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.967+0000 7f1322d9d640 1 --2- 172.21.15.45:0/1333171661 >> v2:172.21.15.190:3300/0 conn(0x7f13240b1fa0 0x7f13240b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.968+0000 7f1321d9b640 1 -- 172.21.15.45:0/1333171661 >> v2:172.21.15.190:3300/0 conn(0x7f13240b1fa0 msgr2=0x7f13240b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:17.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.968+0000 7f1321d9b640 1 --2- 172.21.15.45:0/1333171661 >> v2:172.21.15.190:3300/0 conn(0x7f13240b1fa0 0x7f13240b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.967+0000 7f132159a640 1 -- 172.21.15.45:0/1333171661 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564775707 0 0) 0x7f132415bda0 con 0x7f13240b35b0 2023-11-28T20:50:17.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.968+0000 7f1321d9b640 1 -- 172.21.15.45:0/1333171661 >> v1:172.21.15.190:6789/0 conn(0x7f13240b35b0 legacy=0x7f132415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.968+0000 7f1321d9b640 1 -- 172.21.15.45:0/1333171661 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132415c360 con 0x7f132415b850 2023-11-28T20:50:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.969+0000 7f1321d9b640 1 --2- 172.21.15.45:0/1333171661 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132415b850 0x7f13240b0c30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f13180098a0 tx=0x7f131802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43cf2282988a0d60 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.970+0000 7f132159a640 1 -- 172.21.15.45:0/1333171661 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1318030840 con 0x7f132415b850 2023-11-28T20:50:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.970+0000 7f132159a640 1 -- 172.21.15.45:0/1333171661 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13180309a0 con 0x7f132415b850 2023-11-28T20:50:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.970+0000 7f132159a640 1 -- 172.21.15.45:0/1333171661 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1318030d30 con 0x7f132415b850 2023-11-28T20:50:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.970+0000 7f1328a75640 1 -- 172.21.15.45:0/1333171661 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132415b850 msgr2=0x7f13240b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.970+0000 7f1328a75640 1 --2- 172.21.15.45:0/1333171661 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132415b850 0x7f13240b0c30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f13180098a0 tx=0x7f131802b980 comp rx=0 tx=0).stop 2023-11-28T20:50:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.970+0000 7f1328a75640 1 -- 172.21.15.45:0/1333171661 shutdown_connections 2023-11-28T20:50:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.970+0000 7f1328a75640 1 --2- 172.21.15.45:0/1333171661 >> v2:172.21.15.190:3300/0 conn(0x7f13240b1fa0 0x7f13240b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.970+0000 7f1328a75640 1 --2- 172.21.15.45:0/1333171661 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132415b850 0x7f13240b0c30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.970+0000 7f1328a75640 1 -- 172.21.15.45:0/1333171661 >> 172.21.15.45:0/1333171661 conn(0x7f132406a500 msgr2=0x7f132406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.971+0000 7f1328a75640 1 -- 172.21.15.45:0/1333171661 shutdown_connections 2023-11-28T20:50:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.971+0000 7f1328a75640 1 -- 172.21.15.45:0/1333171661 wait complete. 2023-11-28T20:50:17.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.972+0000 7f1328a75640 1 Processor -- start 2023-11-28T20:50:17.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.973+0000 7f1328a75640 1 -- start start 2023-11-28T20:50:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.973+0000 7f1328a75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13240b1fa0 0x7f13240cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.973+0000 7f1328a75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13240b35b0 0x7f13240ab080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1328a75640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132415b850 0x7f13240ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1328a75640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240cc510 con 0x7f132415b850 2023-11-28T20:50:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1328a75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13241356f0 con 0x7f13240b1fa0 2023-11-28T20:50:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1328a75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1324136550 con 0x7f13240b35b0 2023-11-28T20:50:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1322d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132415b850 0x7f13240ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1322d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132415b850 0x7f13240ab8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34534/0 (socket says 172.21.15.45:34534) 2023-11-28T20:50:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1322d9d640 1 -- 172.21.15.45:0/1033260426 learned_addr learned my addr 172.21.15.45:0/1033260426 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f132259c640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13240b1fa0 0x7f13240cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1321d9b640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13240b35b0 0x7f13240ab080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1322d9d640 1 -- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13240b35b0 msgr2=0x7f13240ab080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1322d9d640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13240b35b0 0x7f13240ab080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1322d9d640 1 -- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13240b1fa0 msgr2=0x7f13240cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1322d9d640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13240b1fa0 0x7f13240cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.974+0000 7f1322d9d640 1 -- 172.21.15.45:0/1033260426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13180094d0 con 0x7f132415b850 2023-11-28T20:50:17.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.975+0000 7f1322d9d640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132415b850 0x7f13240ab8c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f131c009e30 tx=0x7f131c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:17.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.975+0000 7f1312ffd640 1 -- 172.21.15.45:0/1033260426 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c009ca0 con 0x7f132415b850 2023-11-28T20:50:17.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.975+0000 7f1312ffd640 1 -- 172.21.15.45:0/1033260426 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c00f040 con 0x7f132415b850 2023-11-28T20:50:17.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.975+0000 7f1312ffd640 1 -- 172.21.15.45:0/1033260426 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c0146c0 con 0x7f132415b850 2023-11-28T20:50:17.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.975+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1324135940 con 0x7f132415b850 2023-11-28T20:50:17.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.975+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1324135f00 con 0x7f132415b850 2023-11-28T20:50:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.977+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13240733d0 con 0x7f132415b850 2023-11-28T20:50:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.977+0000 7f1312ffd640 1 -- 172.21.15.45:0/1033260426 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f131c014820 con 0x7f132415b850 2023-11-28T20:50:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.978+0000 7f1312ffd640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f130406af90 0x7f130406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.978+0000 7f1312ffd640 1 -- 172.21.15.45:0/1033260426 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f131c08a890 con 0x7f132415b850 2023-11-28T20:50:17.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.979+0000 7f132259c640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f130406af90 0x7f130406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.980+0000 7f132259c640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f130406af90 0x7f130406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1314000f60 tx=0x7f1314007b70 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:17.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:17.983+0000 7f1312ffd640 1 -- 172.21.15.45:0/1033260426 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f131c055620 con 0x7f132415b850 2023-11-28T20:50:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.181+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1324136690 con 0x7f132415b850 2023-11-28T20:50:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.182+0000 7f1312ffd640 1 -- 172.21.15.45:0/1033260426 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+14191 (secure 0 0 0) 0x7f131c018020 con 0x7f132415b850 2023-11-28T20:50:18.185 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:18.185 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":80,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:15.692033+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/3717491008":"2023-11-29T20:50:12.665891+0000","172.21.15.45:6833/3717491008":"2023-11-29T20:50:12.665891+0000","172.21.15.190:0/4256179623":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/4143526042":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/124688230":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/1895312230":"2023-11-29T20:50:08.628199+0000","172.21.15.190:6833/979211100":"2023-11-29T20:50:08.628199+0000","172.21.15.190:6832/979211100":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/1831728153":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/105965464":"2023-11-29T20:50:08.628199+0000","172.21.15.190:0/2346462975":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/1966056956":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6832/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/633536721":"2023-11-29T20:48:28.490206+0000","172.21.15.190:6833/838517151":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/4198236319":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/676976495":"2023-11-29T20:48:28.490206+0000","172.21.15.190:0/3224416347":"2023-11-29T20:48:28.490206+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:18.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f130406af90 msgr2=0x7f130406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f130406af90 0x7f130406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1314000f60 tx=0x7f1314007b70 comp rx=0 tx=0).stop 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132415b850 msgr2=0x7f13240ab8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132415b850 0x7f13240ab8c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f131c009e30 tx=0x7f131c00c490 comp rx=0 tx=0).stop 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 shutdown_connections 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f130406af90 0x7f130406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f132415b850 0x7f13240ab8c0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13240b35b0 0x7f13240ab080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 --2- 172.21.15.45:0/1033260426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13240b1fa0 0x7f13240cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 >> 172.21.15.45:0/1033260426 conn(0x7f132406a500 msgr2=0x7f13240ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 shutdown_connections 2023-11-28T20:50:18.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.185+0000 7f1328a75640 1 -- 172.21.15.45:0/1033260426 wait complete. 2023-11-28T20:50:18.208 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:50:18.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:50:18.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.436+0000 7fa8d5fc9640 1 Processor -- start 2023-11-28T20:50:18.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.437+0000 7fa8d5fc9640 1 -- start start 2023-11-28T20:50:18.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.438+0000 7fa8d5fc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d00cc110 0x7fa8d00cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.438+0000 7fa8d5fc9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa8d006ecc0 0x7fa8d00ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.438+0000 7fa8d5fc9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d006af10 con 0x7fa8d00c3680 2023-11-28T20:50:18.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.438+0000 7fa8d5fc9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d00ce380 con 0x7fa8d006ecc0 2023-11-28T20:50:18.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.438+0000 7fa8d5fc9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00ce620 con 0x7fa8d00cc110 2023-11-28T20:50:18.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.439+0000 7fa8c7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa8d00c3680 0x7fa8d006e5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56758/0 (socket says 172.21.15.45:56758) 2023-11-28T20:50:18.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.439+0000 7fa8c7fff640 1 -- 172.21.15.45:0/2804720822 learned_addr learned my addr 172.21.15.45:0/2804720822 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:18.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.439+0000 7fa8cffff640 1 --2- 172.21.15.45:0/2804720822 >> v2:172.21.15.45:3300/0 conn(0x7fa8d006ecc0 0x7fa8d00ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.439+0000 7fa8cf7fe640 1 --2- 172.21.15.45:0/2804720822 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d00cc110 0x7fa8d00cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.439+0000 7fa8ceffd640 1 -- 172.21.15.45:0/2804720822 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281579753 0 0) 0x7fa8d006af10 con 0x7fa8d00c3680 2023-11-28T20:50:18.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.439+0000 7fa8ceffd640 1 -- 172.21.15.45:0/2804720822 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8bc003520 con 0x7fa8d00c3680 2023-11-28T20:50:18.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.439+0000 7fa8cffff640 1 -- 172.21.15.45:0/2804720822 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d00cc110 msgr2=0x7fa8d00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:18.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.440+0000 7fa8cffff640 1 --2- 172.21.15.45:0/2804720822 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d00cc110 0x7fa8d00cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.440+0000 7fa8cffff640 1 -- 172.21.15.45:0/2804720822 >> v1:172.21.15.45:6789/0 conn(0x7fa8d00c3680 legacy=0x7fa8d006e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.440+0000 7fa8cffff640 1 -- 172.21.15.45:0/2804720822 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d006b1b0 con 0x7fa8d006ecc0 2023-11-28T20:50:18.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.441+0000 7fa8cffff640 1 --2- 172.21.15.45:0/2804720822 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d006ecc0 0x7fa8d00ca590 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009d90 tx=0x7fa8c802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1760d4f34ef68285 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:18.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.441+0000 7fa8ceffd640 1 -- 172.21.15.45:0/2804720822 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c802cc50 con 0x7fa8d006ecc0 2023-11-28T20:50:18.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.441+0000 7fa8ceffd640 1 -- 172.21.15.45:0/2804720822 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c802cdb0 con 0x7fa8d006ecc0 2023-11-28T20:50:18.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.441+0000 7fa8ceffd640 1 -- 172.21.15.45:0/2804720822 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8034660 con 0x7fa8d006ecc0 2023-11-28T20:50:18.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.442+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/2804720822 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d006ecc0 msgr2=0x7fa8d00ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.442+0000 7fa8d5fc9640 1 --2- 172.21.15.45:0/2804720822 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d006ecc0 0x7fa8d00ca590 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009d90 tx=0x7fa8c802bea0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.442+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/2804720822 shutdown_connections 2023-11-28T20:50:18.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.442+0000 7fa8d5fc9640 1 --2- 172.21.15.45:0/2804720822 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d006ecc0 0x7fa8d00ca590 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.442+0000 7fa8d5fc9640 1 --2- 172.21.15.45:0/2804720822 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d00cc110 0x7fa8d00cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.442+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/2804720822 >> 172.21.15.45:0/2804720822 conn(0x7fa8d00cc9c0 msgr2=0x7fa8d00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:18.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.442+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/2804720822 shutdown_connections 2023-11-28T20:50:18.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.442+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/2804720822 wait complete. 2023-11-28T20:50:18.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.444+0000 7fa8d5fc9640 1 Processor -- start 2023-11-28T20:50:18.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.444+0000 7fa8d5fc9640 1 -- start start 2023-11-28T20:50:18.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.445+0000 7fa8d5fc9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8d006ecc0 0x7fa8d015c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.445+0000 7fa8d5fc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d00cc110 0x7fa8d00c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.445+0000 7fa8d5fc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d01f2ba0 0x7fa8d00661f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.445+0000 7fa8d5fc9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d015d6d0 con 0x7fa8d00cc110 2023-11-28T20:50:18.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.445+0000 7fa8d5fc9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00c6980 con 0x7fa8d006ecc0 2023-11-28T20:50:18.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.445+0000 7fa8d5fc9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d015ce30 con 0x7fa8d01f2ba0 2023-11-28T20:50:18.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.445+0000 7fa8c7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d00cc110 0x7fa8d00c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.445+0000 7fa8c7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d00cc110 0x7fa8d00c6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34556/0 (socket says 172.21.15.45:34556) 2023-11-28T20:50:18.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.445+0000 7fa8c7fff640 1 -- 172.21.15.45:0/1127541901 learned_addr learned my addr 172.21.15.45:0/1127541901 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:18.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8cffff640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d01f2ba0 0x7fa8d00661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8cf7fe640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8d006ecc0 0x7fa8d015c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8c7fff640 1 -- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d01f2ba0 msgr2=0x7fa8d00661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8c7fff640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d01f2ba0 0x7fa8d00661f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8c7fff640 1 -- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8d006ecc0 msgr2=0x7fa8d015c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8c7fff640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8d006ecc0 0x7fa8d015c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8c7fff640 1 -- 172.21.15.45:0/1127541901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c8009a70 con 0x7fa8d00cc110 2023-11-28T20:50:18.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8c7fff640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d00cc110 0x7fa8d00c6440 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800c500 tx=0x7fa8b800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:18.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8ccff9640 1 -- 172.21.15.45:0/1127541901 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8005ce0 con 0x7fa8d00cc110 2023-11-28T20:50:18.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8ccff9640 1 -- 172.21.15.45:0/1127541901 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b8005e40 con 0x7fa8d00cc110 2023-11-28T20:50:18.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8ccff9640 1 -- 172.21.15.45:0/1127541901 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8007330 con 0x7fa8d00cc110 2023-11-28T20:50:18.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.446+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d015d110 con 0x7fa8d00cc110 2023-11-28T20:50:18.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.447+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d00b2d80 con 0x7fa8d00cc110 2023-11-28T20:50:18.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.448+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d00b2be0 con 0x7fa8d00cc110 2023-11-28T20:50:18.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.449+0000 7fa8ccff9640 1 -- 172.21.15.45:0/1127541901 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa8b8007490 con 0x7fa8d00cc110 2023-11-28T20:50:18.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.450+0000 7fa8ccff9640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa8b006afe0 0x7fa8b006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.450+0000 7fa8ccff9640 1 -- 172.21.15.45:0/1127541901 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fa8b8088f00 con 0x7fa8d00cc110 2023-11-28T20:50:18.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.450+0000 7fa8cf7fe640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa8b006afe0 0x7fa8b006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.451+0000 7fa8cf7fe640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa8b006afe0 0x7fa8b006d4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009a10 tx=0x7fa8c0008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:18.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.454+0000 7fa8ccff9640 1 -- 172.21.15.45:0/1127541901 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8d00b2be0 con 0x7fa8d00cc110 2023-11-28T20:50:18.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.652+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa8d00b2be0 con 0x7fa8d00cc110 2023-11-28T20:50:18.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.653+0000 7fa8ccff9640 1 -- 172.21.15.45:0/1127541901 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa8d00b2be0 con 0x7fa8d00cc110 2023-11-28T20:50:18.655 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:50:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.655+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa8b006afe0 msgr2=0x7fa8b006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.655+0000 7fa8d5fc9640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa8b006afe0 0x7fa8b006d4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009a10 tx=0x7fa8c0008040 comp rx=0 tx=0).stop 2023-11-28T20:50:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d00cc110 msgr2=0x7fa8d00c6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d00cc110 0x7fa8d00c6440 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800c500 tx=0x7fa8b800c9c0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 shutdown_connections 2023-11-28T20:50:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa8b006afe0 0x7fa8b006d4a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8d01f2ba0 0x7fa8d00661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d00cc110 0x7fa8d00c6440 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 --2- 172.21.15.45:0/1127541901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8d006ecc0 0x7fa8d015c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 >> 172.21.15.45:0/1127541901 conn(0x7fa8d00cc9c0 msgr2=0x7fa8d00cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 shutdown_connections 2023-11-28T20:50:18.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.656+0000 7fa8d5fc9640 1 -- 172.21.15.45:0/1127541901 wait complete. 2023-11-28T20:50:18.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-11-28T20:50:18.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.901+0000 7f7b7ad75640 1 Processor -- start 2023-11-28T20:50:18.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.902+0000 7f7b7ad75640 1 -- start start 2023-11-28T20:50:18.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.903+0000 7f7b7ad75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b7415d550 0x7f7b740cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.903+0000 7f7b7ad75640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7b740d0100 0x7f7b740d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.903+0000 7f7b7ad75640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b740d0560 con 0x7f7b740d1ab0 2023-11-28T20:50:18.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.903+0000 7f7b7ad75640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b740d2230 con 0x7f7b740d0100 2023-11-28T20:50:18.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.903+0000 7f7b7ad75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740d2510 con 0x7f7b7415d550 2023-11-28T20:50:18.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.903+0000 7f7b7a574640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7b740d1ab0 0x7f7b74070b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56770/0 (socket says 172.21.15.45:56770) 2023-11-28T20:50:18.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.903+0000 7f7b7a574640 1 -- 172.21.15.45:0/3316734268 learned_addr learned my addr 172.21.15.45:0/3316734268 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:18.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b79d73640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b7415d550 0x7f7b740cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b79572640 1 --2- 172.21.15.45:0/3316734268 >> v2:172.21.15.190:3300/0 conn(0x7f7b740d0100 0x7f7b740d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022675277 0 0) 0x7f7b740d0560 con 0x7f7b740d1ab0 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c003520 con 0x7f7b740d1ab0 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 295634611 0 0) 0x7f7b5c003520 con 0x7f7b740d1ab0 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b7415d550 msgr2=0x7f7b740cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b7415d550 0x7f7b740cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 >> v2:172.21.15.190:3300/0 conn(0x7f7b740d0100 msgr2=0x7f7b740d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 --2- 172.21.15.45:0/3316734268 >> v2:172.21.15.190:3300/0 conn(0x7f7b740d0100 0x7f7b740d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b740d27f0 con 0x7f7b740d1ab0 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f7b6c0019f0 con 0x7f7b740d1ab0 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 >> v1:172.21.15.45:6789/0 conn(0x7f7b740d1ab0 legacy=0x7f7b74070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b7415d550 0x7f7b5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b5c004d00 0x7f7b5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740d2230 con 0x7f7b5c003970 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740d2510 con 0x7f7b5c004d00 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.904+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c00b980 con 0x7f7b7415d550 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.905+0000 7f7b79d73640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.905+0000 7f7b79572640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b7415d550 0x7f7b5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.905+0000 7f7b79d73640 1 -- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b7415d550 msgr2=0x7f7b5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.905+0000 7f7b7a574640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b5c004d00 0x7f7b5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.905+0000 7f7b79d73640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b7415d550 0x7f7b5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.905+0000 7f7b79d73640 1 -- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b5c004d00 msgr2=0x7f7b5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:18.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.905+0000 7f7b79d73640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b5c004d00 0x7f7b5c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.905+0000 7f7b79d73640 1 -- 172.21.15.45:0/3316734268 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c003520 con 0x7f7b5c003970 2023-11-28T20:50:18.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.905+0000 7f7b79572640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b7415d550 0x7f7b5c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:18.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.906+0000 7f7b79d73640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b5c003d90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f7b64009ac0 tx=0x7f7b5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a60fc354118a4ad server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:18.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6402c6b0 con 0x7f7b5c003970 2023-11-28T20:50:18.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6402c810 con 0x7f7b5c003970 2023-11-28T20:50:18.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b78d71640 1 -- 172.21.15.45:0/3316734268 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6402cba0 con 0x7f7b5c003970 2023-11-28T20:50:18.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b7ad75640 1 -- 172.21.15.45:0/3316734268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 msgr2=0x7f7b5c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b7ad75640 1 --2- 172.21.15.45:0/3316734268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b5c003d90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f7b64009ac0 tx=0x7f7b5c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b7a574640 1 -- 172.21.15.45:0/3316734268 reap_dead start 2023-11-28T20:50:18.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b7ad75640 1 -- 172.21.15.45:0/3316734268 shutdown_connections 2023-11-28T20:50:18.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b7ad75640 1 -- 172.21.15.45:0/3316734268 >> 172.21.15.45:0/3316734268 conn(0x7f7b7406ac70 msgr2=0x7f7b740b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:18.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b7ad75640 1 -- 172.21.15.45:0/3316734268 shutdown_connections 2023-11-28T20:50:18.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.907+0000 7f7b7ad75640 1 -- 172.21.15.45:0/3316734268 wait complete. 2023-11-28T20:50:18.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.909+0000 7f7b7ad75640 1 Processor -- start 2023-11-28T20:50:18.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.909+0000 7f7b7ad75640 1 -- start start 2023-11-28T20:50:18.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.910+0000 7f7b7ad75640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b7415b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.910+0000 7f7b7ad75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b741f84e0 0x7f7b740c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.910+0000 7f7b7ad75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b741f8940 0x7f7b740c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.910+0000 7f7b7ad75640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740b2460 con 0x7f7b5c003970 2023-11-28T20:50:18.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.910+0000 7f7b7ad75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740b2740 con 0x7f7b741f8940 2023-11-28T20:50:18.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.910+0000 7f7b7ad75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740b2a20 con 0x7f7b741f84e0 2023-11-28T20:50:18.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b79d73640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b7415b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b79d73640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b7415b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34580/0 (socket says 172.21.15.45:34580) 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b79d73640 1 -- 172.21.15.45:0/4023026832 learned_addr learned my addr 172.21.15.45:0/4023026832 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b7a574640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b741f8940 0x7f7b740c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b79572640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b741f84e0 0x7f7b740c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b79d73640 1 -- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b741f84e0 msgr2=0x7f7b740c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b79d73640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b741f84e0 0x7f7b740c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b79d73640 1 -- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b741f8940 msgr2=0x7f7b740c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b79d73640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b741f8940 0x7f7b740c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.911+0000 7f7b79d73640 1 -- 172.21.15.45:0/4023026832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b740b2d00 con 0x7f7b5c003970 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.912+0000 7f7b79d73640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b7415b330 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f7b64000c00 tx=0x7f7b64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.912+0000 7f7b727fc640 1 -- 172.21.15.45:0/4023026832 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b64003810 con 0x7f7b5c003970 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.912+0000 7f7b727fc640 1 -- 172.21.15.45:0/4023026832 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b64003990 con 0x7f7b5c003970 2023-11-28T20:50:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.912+0000 7f7b727fc640 1 -- 172.21.15.45:0/4023026832 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b64003d40 con 0x7f7b5c003970 2023-11-28T20:50:18.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.912+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b740b2fe0 con 0x7f7b5c003970 2023-11-28T20:50:18.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.912+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b740b36e0 con 0x7f7b5c003970 2023-11-28T20:50:18.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.913+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7404e920 con 0x7f7b5c003970 2023-11-28T20:50:18.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.914+0000 7f7b727fc640 1 -- 172.21.15.45:0/4023026832 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7b6403b070 con 0x7f7b5c003970 2023-11-28T20:50:18.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.915+0000 7f7b727fc640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f7b6006af90 0x7f7b6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:18.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.915+0000 7f7b727fc640 1 -- 172.21.15.45:0/4023026832 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f7b640ab550 con 0x7f7b5c003970 2023-11-28T20:50:18.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.916+0000 7f7b79572640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f7b6006af90 0x7f7b6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:18.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.917+0000 7f7b79572640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f7b6006af90 0x7f7b6006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7b7415ef30 tx=0x7f7b68008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:18.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:18.920+0000 7f7b727fc640 1 -- 172.21.15.45:0/4023026832 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b640765c0 con 0x7f7b5c003970 2023-11-28T20:50:19.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.189+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f7b7415ca50 con 0x7f7b5c003970 2023-11-28T20:50:19.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:50:19.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:50:19.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.725+0000 7f7b727fc640 1 -- 172.21.15.45:0/4023026832 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v605) v1 ==== 159+0+0 (secure 0 0 0) 0x7f7b64079be0 con 0x7f7b5c003970 2023-11-28T20:50:19.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.728+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f7b6006af90 msgr2=0x7f7b6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:19.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.728+0000 7f7b7ad75640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f7b6006af90 0x7f7b6006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7b7415ef30 tx=0x7f7b68008040 comp rx=0 tx=0).stop 2023-11-28T20:50:19.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 msgr2=0x7f7b7415b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:19.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b7415b330 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f7b64000c00 tx=0x7f7b64004730 comp rx=0 tx=0).stop 2023-11-28T20:50:19.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 shutdown_connections 2023-11-28T20:50:19.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f7b6006af90 0x7f7b6006d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:19.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b741f8940 0x7f7b740c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:19.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b741f84e0 0x7f7b740c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:19.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 --2- 172.21.15.45:0/4023026832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b5c003970 0x7f7b7415b330 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:19.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 >> 172.21.15.45:0/4023026832 conn(0x7f7b7406ac70 msgr2=0x7f7b7415e000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:19.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 shutdown_connections 2023-11-28T20:50:19.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.729+0000 7f7b7ad75640 1 -- 172.21.15.45:0/4023026832 wait complete. 2023-11-28T20:50:19.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T20:50:19.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.984+0000 7f977a98f640 1 Processor -- start 2023-11-28T20:50:19.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.984+0000 7f977a98f640 1 -- start start 2023-11-28T20:50:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.985+0000 7f977a98f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9774155770 0x7f977415c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.986+0000 7f977a98f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97740cdb20 con 0x7f977406c650 2023-11-28T20:50:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.986+0000 7f977a98f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977415c580 con 0x7f97740cdfb0 2023-11-28T20:50:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.986+0000 7f977a98f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97740ccf10 con 0x7f9774155770 2023-11-28T20:50:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.986+0000 7f977998d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9774155770 0x7f977415c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.986+0000 7f977998d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9774155770 0x7f977415c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34584/0 (socket says 172.21.15.45:34584) 2023-11-28T20:50:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.986+0000 7f977998d640 1 -- 172.21.15.45:0/99969275 learned_addr learned my addr 172.21.15.45:0/99969275 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.986+0000 7f977a18e640 1 --1- 172.21.15.45:0/99969275 >> v1:172.21.15.45:6789/0 conn(0x7f977406c650 0x7f97740b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56784/0 (socket says 172.21.15.45:56784) 2023-11-28T20:50:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.987+0000 7f977998d640 1 -- 172.21.15.45:0/99969275 >> v1:172.21.15.190:6789/0 conn(0x7f97740cdfb0 legacy=0x7f977406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.987+0000 7f977998d640 1 -- 172.21.15.45:0/99969275 >> v1:172.21.15.45:6789/0 conn(0x7f977406c650 legacy=0x7f97740b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:19.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.987+0000 7f977998d640 1 -- 172.21.15.45:0/99969275 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97740cd1f0 con 0x7f9774155770 2023-11-28T20:50:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.988+0000 7f977998d640 1 --2- 172.21.15.45:0/99969275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774155770 0x7f977415c040 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9768009e50 tx=0x7f976802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62208c128517a1bf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.988+0000 7f977898b640 1 -- 172.21.15.45:0/99969275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976802ca80 con 0x7f9774155770 2023-11-28T20:50:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.988+0000 7f977898b640 1 -- 172.21.15.45:0/99969275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976802cbe0 con 0x7f9774155770 2023-11-28T20:50:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.988+0000 7f977898b640 1 -- 172.21.15.45:0/99969275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97680343f0 con 0x7f9774155770 2023-11-28T20:50:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.989+0000 7f977a98f640 1 -- 172.21.15.45:0/99969275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774155770 msgr2=0x7f977415c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.989+0000 7f977a98f640 1 --2- 172.21.15.45:0/99969275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774155770 0x7f977415c040 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9768009e50 tx=0x7f976802bee0 comp rx=0 tx=0).stop 2023-11-28T20:50:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.989+0000 7f977a98f640 1 -- 172.21.15.45:0/99969275 shutdown_connections 2023-11-28T20:50:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.989+0000 7f977a98f640 1 --2- 172.21.15.45:0/99969275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774155770 0x7f977415c040 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.989+0000 7f977a98f640 1 -- 172.21.15.45:0/99969275 >> 172.21.15.45:0/99969275 conn(0x7f97740b1050 msgr2=0x7f977406a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.989+0000 7f977a98f640 1 -- 172.21.15.45:0/99969275 shutdown_connections 2023-11-28T20:50:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.989+0000 7f977a98f640 1 -- 172.21.15.45:0/99969275 wait complete. 2023-11-28T20:50:19.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.991+0000 7f977a98f640 1 Processor -- start 2023-11-28T20:50:19.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.991+0000 7f977a98f640 1 -- start start 2023-11-28T20:50:19.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.992+0000 7f977a98f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f977406c650 0x7f9774154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:19.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.992+0000 7f977a98f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97740cdfb0 0x7f9774131520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:19.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.992+0000 7f977a98f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774131a60 0x7f97740ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:19.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.992+0000 7f977a98f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9774155110 con 0x7f9774131a60 2023-11-28T20:50:19.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.992+0000 7f977a98f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97740ac980 con 0x7f977406c650 2023-11-28T20:50:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.992+0000 7f977a98f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9774143e90 con 0x7f97740cdfb0 2023-11-28T20:50:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.992+0000 7f977a18e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774131a60 0x7f97740ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.992+0000 7f977a18e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774131a60 0x7f97740ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34590/0 (socket says 172.21.15.45:34590) 2023-11-28T20:50:19.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.992+0000 7f977a18e640 1 -- 172.21.15.45:0/1529753423 learned_addr learned my addr 172.21.15.45:0/1529753423 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977998d640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f977406c650 0x7f9774154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977a18e640 1 -- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97740cdfb0 msgr2=0x7f9774131520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977918c640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97740cdfb0 0x7f9774131520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977a18e640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97740cdfb0 0x7f9774131520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977a18e640 1 -- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f977406c650 msgr2=0x7f9774154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977a18e640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f977406c650 0x7f9774154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:19.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977a18e640 1 -- 172.21.15.45:0/1529753423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9768009b30 con 0x7f9774131a60 2023-11-28T20:50:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977998d640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f977406c650 0x7f9774154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977918c640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97740cdfb0 0x7f9774131520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f977a18e640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774131a60 0x7f97740ace10 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f976c00c490 tx=0x7f976c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f97727fc640 1 -- 172.21.15.45:0/1529753423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c0074e0 con 0x7f9774131a60 2023-11-28T20:50:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f97727fc640 1 -- 172.21.15.45:0/1529753423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c007640 con 0x7f9774131a60 2023-11-28T20:50:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.993+0000 7f97727fc640 1 -- 172.21.15.45:0/1529753423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c004a00 con 0x7f9774131a60 2023-11-28T20:50:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.994+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97740abd70 con 0x7f9774131a60 2023-11-28T20:50:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.994+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97740ac330 con 0x7f9774131a60 2023-11-28T20:50:19.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.995+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97740ce410 con 0x7f9774131a60 2023-11-28T20:50:19.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.996+0000 7f97727fc640 1 -- 172.21.15.45:0/1529753423 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f976c004b60 con 0x7f9774131a60 2023-11-28T20:50:19.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.997+0000 7f97727fc640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f975806afe0 0x7f975806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:19.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.997+0000 7f97727fc640 1 -- 172.21.15.45:0/1529753423 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f976c0890e0 con 0x7f9774131a60 2023-11-28T20:50:20.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.998+0000 7f977998d640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f975806afe0 0x7f975806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:20.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:19.999+0000 7f977998d640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f975806afe0 0x7f975806d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9768035000 tx=0x7f9768030000 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:20.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.002+0000 7f97727fc640 1 -- 172.21.15.45:0/1529753423 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f97740ce410 con 0x7f9774131a60 2023-11-28T20:50:20.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.196+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f97740ce410 con 0x7f9774131a60 2023-11-28T20:50:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.202+0000 7f97727fc640 1 -- 172.21.15.45:0/1529753423 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f97740ce410 con 0x7f9774131a60 2023-11-28T20:50:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f975806afe0 msgr2=0x7f975806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f975806afe0 0x7f975806d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9768035000 tx=0x7f9768030000 comp rx=0 tx=0).stop 2023-11-28T20:50:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774131a60 msgr2=0x7f97740ace10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774131a60 0x7f97740ace10 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f976c00c490 tx=0x7f976c00c950 comp rx=0 tx=0).stop 2023-11-28T20:50:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 shutdown_connections 2023-11-28T20:50:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f975806afe0 0x7f975806d4a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9774131a60 0x7f97740ace10 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97740cdfb0 0x7f9774131520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 --2- 172.21.15.45:0/1529753423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f977406c650 0x7f9774154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 >> 172.21.15.45:0/1529753423 conn(0x7f97740b1050 msgr2=0x7f977415ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 shutdown_connections 2023-11-28T20:50:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.205+0000 7f977a98f640 1 -- 172.21.15.45:0/1529753423 wait complete. 2023-11-28T20:50:20.228 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:50:20.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:50:20.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:50:20.483 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:50:20.483 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:50:20.484 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:20.548 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:50:20.548 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:50:20.548 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:20.606 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:50:20.606 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:50:20.607 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:20.668 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:50:20.669 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:50:20.669 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:20.728 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:50:20.728 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T20:50:20.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:20.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.965+0000 7fb200801640 1 Processor -- start 2023-11-28T20:50:20.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.965+0000 7fb200801640 1 -- start start 2023-11-28T20:50:20.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.966+0000 7fb200801640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc15d440 0x7fb1fc0cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:20.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.966+0000 7fb200801640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb1fc0b3730 0x7fb1fc099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:20.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.966+0000 7fb200801640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc0cf9d0 con 0x7fb1fc06a500 2023-11-28T20:50:20.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.966+0000 7fb200801640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1fc06b3e0 con 0x7fb1fc0b3730 2023-11-28T20:50:20.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.966+0000 7fb200801640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc15d820 con 0x7fb1fc15d440 2023-11-28T20:50:20.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.967+0000 7fb1fad9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc15d440 0x7fb1fc0cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:20.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.967+0000 7fb1f259c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb1fc0b3730 0x7fb1fc099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:20.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.967+0000 7fb1fb59e640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb1fc06a500 0x7fb1fc0cb5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:36806/0 (socket says 172.21.15.45:36806) 2023-11-28T20:50:20.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.967+0000 7fb1fb59e640 1 -- 172.21.15.45:0/2144353941 learned_addr learned my addr 172.21.15.45:0/2144353941 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:20.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.968+0000 7fb1f259c640 1 -- 172.21.15.45:0/2144353941 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc15d440 msgr2=0x7fb1fc0cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:20.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.968+0000 7fb1f259c640 1 --2- 172.21.15.45:0/2144353941 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc15d440 0x7fb1fc0cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.968+0000 7fb1f259c640 1 -- 172.21.15.45:0/2144353941 >> v1:172.21.15.190:6789/0 conn(0x7fb1fc06a500 legacy=0x7fb1fc0cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:20.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.968+0000 7fb1f259c640 1 -- 172.21.15.45:0/2144353941 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc0cf100 con 0x7fb1fc0b3730 2023-11-28T20:50:20.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.969+0000 7fb1f259c640 1 --2- 172.21.15.45:0/2144353941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1fc0b3730 0x7fb1fc099a40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40097a0 tx=0x7fb1e402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1a3d43d8efcdac server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:20.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.970+0000 7fb1fa59c640 1 -- 172.21.15.45:0/2144353941 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e402d040 con 0x7fb1fc0b3730 2023-11-28T20:50:20.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.970+0000 7fb1fa59c640 1 -- 172.21.15.45:0/2144353941 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e4031810 con 0x7fb1fc0b3730 2023-11-28T20:50:20.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.970+0000 7fb1fa59c640 1 -- 172.21.15.45:0/2144353941 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4031ba0 con 0x7fb1fc0b3730 2023-11-28T20:50:20.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.971+0000 7fb200801640 1 -- 172.21.15.45:0/2144353941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1fc0b3730 msgr2=0x7fb1fc099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:20.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.971+0000 7fb200801640 1 --2- 172.21.15.45:0/2144353941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1fc0b3730 0x7fb1fc099a40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40097a0 tx=0x7fb1e402b8b0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.971+0000 7fb200801640 1 -- 172.21.15.45:0/2144353941 shutdown_connections 2023-11-28T20:50:20.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.971+0000 7fb200801640 1 --2- 172.21.15.45:0/2144353941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1fc0b3730 0x7fb1fc099a40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.971+0000 7fb200801640 1 --2- 172.21.15.45:0/2144353941 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc15d440 0x7fb1fc0cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.971+0000 7fb200801640 1 -- 172.21.15.45:0/2144353941 >> 172.21.15.45:0/2144353941 conn(0x7fb1fc0b2b20 msgr2=0x7fb1fc0b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:20.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.971+0000 7fb200801640 1 -- 172.21.15.45:0/2144353941 shutdown_connections 2023-11-28T20:50:20.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.971+0000 7fb200801640 1 -- 172.21.15.45:0/2144353941 wait complete. 2023-11-28T20:50:20.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.973+0000 7fb200801640 1 Processor -- start 2023-11-28T20:50:20.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.973+0000 7fb200801640 1 -- start start 2023-11-28T20:50:20.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb200801640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc06a500 0x7fb1fc0b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:20.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb200801640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc0b3730 0x7fb1fc1601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:20.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb200801640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1fc15d440 0x7fb1fc147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:20.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb200801640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc158c30 con 0x7fb1fc06a500 2023-11-28T20:50:20.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb200801640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0b6c70 con 0x7fb1fc15d440 2023-11-28T20:50:20.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb200801640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc158e40 con 0x7fb1fc0b3730 2023-11-28T20:50:20.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb1fad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc06a500 0x7fb1fc0b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:20.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb1fad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc06a500 0x7fb1fc0b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34600/0 (socket says 172.21.15.45:34600) 2023-11-28T20:50:20.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb1fad9d640 1 -- 172.21.15.45:0/2100848101 learned_addr learned my addr 172.21.15.45:0/2100848101 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb1f259c640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc0b3730 0x7fb1fc1601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.974+0000 7fb1fb59e640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1fc15d440 0x7fb1fc147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb1fad9d640 1 -- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc0b3730 msgr2=0x7fb1fc1601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb1fad9d640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc0b3730 0x7fb1fc1601a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb1fad9d640 1 -- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1fc15d440 msgr2=0x7fb1fc147b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb1fad9d640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1fc15d440 0x7fb1fc147b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb1fad9d640 1 -- 172.21.15.45:0/2100848101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e4009400 con 0x7fb1fc06a500 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb1fad9d640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc06a500 0x7fb1fc0b6730 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00e9b0 tx=0x7fb1ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb1f3fff640 1 -- 172.21.15.45:0/2100848101 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec009670 con 0x7fb1fc06a500 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb1f3fff640 1 -- 172.21.15.45:0/2100848101 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec00c9a0 con 0x7fb1fc06a500 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc159120 con 0x7fb1fc06a500 2023-11-28T20:50:20.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb1f3fff640 1 -- 172.21.15.45:0/2100848101 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec00cd50 con 0x7fb1fc06a500 2023-11-28T20:50:20.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.975+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc1596e0 con 0x7fb1fc06a500 2023-11-28T20:50:20.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.976+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1fc0b10d0 con 0x7fb1fc06a500 2023-11-28T20:50:20.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.978+0000 7fb1f3fff640 1 -- 172.21.15.45:0/2100848101 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb1ec013470 con 0x7fb1fc06a500 2023-11-28T20:50:20.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.978+0000 7fb1f3fff640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb1dc06af90 0x7fb1dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:20.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.979+0000 7fb1f3fff640 1 -- 172.21.15.45:0/2100848101 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fb1ec089c50 con 0x7fb1fc06a500 2023-11-28T20:50:20.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.979+0000 7fb1f259c640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb1dc06af90 0x7fb1dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:20.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.980+0000 7fb1f259c640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb1dc06af90 0x7fb1dc06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4036000 tx=0x7fb1e403c040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:20.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:20.983+0000 7fb1f3fff640 1 -- 172.21.15.45:0/2100848101 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1ec0549e0 con 0x7fb1fc06a500 2023-11-28T20:50:21.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.215+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1fc09f110 con 0x7fb1fc06a500 2023-11-28T20:50:21.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.216+0000 7fb1f3fff640 1 -- 172.21.15.45:0/2100848101 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fb1ec021e90 con 0x7fb1fc06a500 2023-11-28T20:50:21.218 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:21.218 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5183,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/285399852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":285399852},{"type":"v1","addr":"172.21.15.190:6836","nonce":285399852}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2028545354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2028545354},{"type":"v1","addr":"172.21.15.190:6837","nonce":2028545354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6216,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/784320624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":784320624},{"type":"v1","addr":"172.21.15.45:6834","nonce":784320624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":6630,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1436998341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":1436998341},{"type":"v1","addr":"172.21.15.45:6835","nonce":1436998341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[]} 2023-11-28T20:50:21.218 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 29 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.218+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb1dc06af90 msgr2=0x7fb1dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.218+0000 7fb200801640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb1dc06af90 0x7fb1dc06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4036000 tx=0x7fb1e403c040 comp rx=0 tx=0).stop 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.218+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc06a500 msgr2=0x7fb1fc0b6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.218+0000 7fb200801640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc06a500 0x7fb1fc0b6730 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00e9b0 tx=0x7fb1ec00ee70 comp rx=0 tx=0).stop 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.218+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 shutdown_connections 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.218+0000 7fb200801640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb1dc06af90 0x7fb1dc06d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.218+0000 7fb200801640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1fc15d440 0x7fb1fc147b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.218+0000 7fb200801640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1fc0b3730 0x7fb1fc1601a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.218+0000 7fb200801640 1 --2- 172.21.15.45:0/2100848101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc06a500 0x7fb1fc0b6730 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.219+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 >> 172.21.15.45:0/2100848101 conn(0x7fb1fc0b2b20 msgr2=0x7fb1fc0b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:21.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.219+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 shutdown_connections 2023-11-28T20:50:21.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.219+0000 7fb200801640 1 -- 172.21.15.45:0/2100848101 wait complete. 2023-11-28T20:50:21.239 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T20:50:21.240 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T20:50:21.240 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T20:50:21.240 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:50:21.240 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T20:50:21.240 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T20:50:21.241 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T20:50:21.241 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:50:21.241 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T20:50:21.241 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T20:50:21.241 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T20:50:21.241 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:50:21.242 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T20:50:21.242 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T20:50:21.242 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T20:50:21.242 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:50:21.259 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T20:50:21.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T20:50:21.263 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T20:50:21.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T20:50:21.265 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T20:50:21.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T20:50:21.267 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T20:50:21.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T20:50:21.313 INFO:tasks.ceph.mds.d:Started 2023-11-28T20:50:21.313 DEBUG:teuthology.parallel:result is None 2023-11-28T20:50:21.315 INFO:tasks.ceph.mds.a:Started 2023-11-28T20:50:21.315 DEBUG:teuthology.parallel:result is None 2023-11-28T20:50:21.321 INFO:tasks.ceph.mds.b:Started 2023-11-28T20:50:21.321 DEBUG:teuthology.parallel:result is None 2023-11-28T20:50:21.323 INFO:tasks.ceph.mds.c:Started 2023-11-28T20:50:21.324 DEBUG:teuthology.parallel:result is None 2023-11-28T20:50:21.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T20:50:21.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2023-11-28T20:50:21.480 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T20:50:21.480 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:50:21.481 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:50:21.481 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:50:21.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.523+0000 7f4b76a94640 1 Processor -- start 2023-11-28T20:50:21.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.523+0000 7f4b76a94640 1 -- start start 2023-11-28T20:50:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.524+0000 7f4b76a94640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4b700cf730 0x7f4b7006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.524+0000 7f4b76a94640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 0x7f4b7015f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.524+0000 7f4b76a94640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7015fa80 con 0x7f4b7015b820 2023-11-28T20:50:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.524+0000 7f4b76a94640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b7015fd60 con 0x7f4b700cf730 2023-11-28T20:50:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.524+0000 7f4b76a94640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7006cf40 con 0x7f4b7015edc0 2023-11-28T20:50:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.524+0000 7f4b74830640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4b7015b820 0x7f4b700cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56798/0 (socket says 172.21.15.45:56798) 2023-11-28T20:50:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.524+0000 7f4b74830640 1 -- 172.21.15.45:0/1931936592 learned_addr learned my addr 172.21.15.45:0/1931936592 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.525+0000 7f4b6ffff640 1 --2- 172.21.15.45:0/1931936592 >> v2:172.21.15.190:3300/0 conn(0x7f4b700cf730 0x7f4b7006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.524+0000 7f4b75031640 1 --2- 172.21.15.45:0/1931936592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 0x7f4b7015f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.525+0000 7f4b75031640 1 -- 172.21.15.45:0/1931936592 >> v2:172.21.15.190:3300/0 conn(0x7f4b700cf730 msgr2=0x7f4b7006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:21.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.525+0000 7f4b75031640 1 --2- 172.21.15.45:0/1931936592 >> v2:172.21.15.190:3300/0 conn(0x7f4b700cf730 0x7f4b7006da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.525+0000 7f4b6f7fe640 1 -- 172.21.15.45:0/1931936592 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844655245 0 0) 0x7f4b7015fa80 con 0x7f4b7015b820 2023-11-28T20:50:21.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.525+0000 7f4b75031640 1 -- 172.21.15.45:0/1931936592 >> v1:172.21.15.45:6789/0 conn(0x7f4b7015b820 legacy=0x7f4b700cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:21.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.525+0000 7f4b75031640 1 -- 172.21.15.45:0/1931936592 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7006d220 con 0x7f4b7015edc0 2023-11-28T20:50:21.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.525+0000 7f4b6ffff640 1 --2- 172.21.15.45:0/1931936592 >> v2:172.21.15.190:3300/0 conn(0x7f4b700cf730 0x7f4b7006da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:50:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.526+0000 7f4b75031640 1 --2- 172.21.15.45:0/1931936592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 0x7f4b7015f1e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009d90 tx=0x7f4b6802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77c9954547a979b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b6f7fe640 1 -- 172.21.15.45:0/1931936592 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6802cc50 con 0x7f4b7015edc0 2023-11-28T20:50:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b6f7fe640 1 -- 172.21.15.45:0/1931936592 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6802cdb0 con 0x7f4b7015edc0 2023-11-28T20:50:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b6f7fe640 1 -- 172.21.15.45:0/1931936592 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68034660 con 0x7f4b7015edc0 2023-11-28T20:50:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b76a94640 1 -- 172.21.15.45:0/1931936592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 msgr2=0x7f4b7015f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b76a94640 1 --2- 172.21.15.45:0/1931936592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 0x7f4b7015f1e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009d90 tx=0x7f4b6802bea0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b76a94640 1 -- 172.21.15.45:0/1931936592 shutdown_connections 2023-11-28T20:50:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b76a94640 1 --2- 172.21.15.45:0/1931936592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 0x7f4b7015f1e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b76a94640 1 --2- 172.21.15.45:0/1931936592 >> v2:172.21.15.190:3300/0 conn(0x7f4b700cf730 0x7f4b7006da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b76a94640 1 -- 172.21.15.45:0/1931936592 >> 172.21.15.45:0/1931936592 conn(0x7f4b7006a6e0 msgr2=0x7f4b7006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.527+0000 7f4b76a94640 1 -- 172.21.15.45:0/1931936592 shutdown_connections 2023-11-28T20:50:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.528+0000 7f4b76a94640 1 -- 172.21.15.45:0/1931936592 wait complete. 2023-11-28T20:50:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.529+0000 7f4b76a94640 1 Processor -- start 2023-11-28T20:50:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.529+0000 7f4b76a94640 1 -- start start 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.530+0000 7f4b76a94640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b700cf730 0x7f4b70151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.530+0000 7f4b76a94640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7015b820 0x7f4b700c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.530+0000 7f4b76a94640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 0x7f4b70152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.530+0000 7f4b76a94640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70152590 con 0x7f4b7015b820 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.530+0000 7f4b76a94640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70151ed0 con 0x7f4b700cf730 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.530+0000 7f4b76a94640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701527a0 con 0x7f4b7015edc0 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7015b820 0x7f4b700c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b74830640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b700cf730 0x7f4b70151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7015b820 0x7f4b700c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34624/0 (socket says 172.21.15.45:34624) 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b6ffff640 1 -- 172.21.15.45:0/1555754458 learned_addr learned my addr 172.21.15.45:0/1555754458 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b75031640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 0x7f4b70152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:21.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b6ffff640 1 -- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 msgr2=0x7f4b70152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b6ffff640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 0x7f4b70152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b6ffff640 1 -- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b700cf730 msgr2=0x7f4b70151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b6ffff640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b700cf730 0x7f4b70151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b6ffff640 1 -- 172.21.15.45:0/1555754458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c009400 con 0x7f4b7015b820 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.531+0000 7f4b74830640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b700cf730 0x7f4b70151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.532+0000 7f4b6ffff640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7015b820 0x7f4b700c9e00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00ebe0 tx=0x7f4b5c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.532+0000 7f4b6d7fa640 1 -- 172.21.15.45:0/1555754458 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c009760 con 0x7f4b7015b820 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.532+0000 7f4b6d7fa640 1 -- 172.21.15.45:0/1555754458 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5c00cc30 con 0x7f4b7015b820 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.532+0000 7f4b6d7fa640 1 -- 172.21.15.45:0/1555754458 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c00eee0 con 0x7f4b7015b820 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.532+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b68009a70 con 0x7f4b7015b820 2023-11-28T20:50:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.532+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b700c9600 con 0x7f4b7015b820 2023-11-28T20:50:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.533+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b700c9430 con 0x7f4b7015b820 2023-11-28T20:50:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.534+0000 7f4b6d7fa640 1 -- 172.21.15.45:0/1555754458 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4b5c013470 con 0x7f4b7015b820 2023-11-28T20:50:21.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.535+0000 7f4b6d7fa640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4b5406af60 0x7f4b5406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:21.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.535+0000 7f4b6d7fa640 1 -- 172.21.15.45:0/1555754458 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f4b5c089c00 con 0x7f4b7015b820 2023-11-28T20:50:21.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.536+0000 7f4b74830640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4b5406af60 0x7f4b5406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:21.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.537+0000 7f4b74830640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4b5406af60 0x7f4b5406d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4b64000f60 tx=0x7f4b64009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:21.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.539+0000 7f4b6d7fa640 1 -- 172.21.15.45:0/1555754458 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b5c054ac0 con 0x7f4b7015b820 2023-11-28T20:50:21.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:21.743+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4b70073440 con 0x7f4b7015b820 2023-11-28T20:50:22.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.230+0000 7f4b6d7fa640 1 -- 172.21.15.45:0/1555754458 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v82) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4b5c017080 con 0x7f4b7015b820 2023-11-28T20:50:22.232 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T20:50:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4b5406af60 msgr2=0x7f4b5406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4b5406af60 0x7f4b5406d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4b64000f60 tx=0x7f4b64009040 comp rx=0 tx=0).stop 2023-11-28T20:50:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7015b820 msgr2=0x7f4b700c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7015b820 0x7f4b700c9e00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00ebe0 tx=0x7f4b5c00c520 comp rx=0 tx=0).stop 2023-11-28T20:50:22.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 shutdown_connections 2023-11-28T20:50:22.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f4b5406af60 0x7f4b5406d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4b7015edc0 0x7f4b70152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7015b820 0x7f4b700c9e00 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 --2- 172.21.15.45:0/1555754458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b700cf730 0x7f4b70151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.234+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 >> 172.21.15.45:0/1555754458 conn(0x7f4b7006a6e0 msgr2=0x7f4b70159090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:22.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.235+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 shutdown_connections 2023-11-28T20:50:22.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.235+0000 7f4b76a94640 1 -- 172.21.15.45:0/1555754458 wait complete. 2023-11-28T20:50:22.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T20:50:22.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.463+0000 7feeaa721640 1 Processor -- start 2023-11-28T20:50:22.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.463+0000 7feeaa721640 1 -- start start 2023-11-28T20:50:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.465+0000 7feeaa721640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feea415c050 0x7feea4166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.465+0000 7feeaa721640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea415c670 con 0x7feea40cc2a0 2023-11-28T20:50:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.465+0000 7feeaa721640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea40cda40 con 0x7feea4141600 2023-11-28T20:50:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.465+0000 7feeaa721640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea40ccc80 con 0x7feea415c050 2023-11-28T20:50:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.465+0000 7feea8f1e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7feea40cc2a0 0x7feea415b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56814/0 (socket says 172.21.15.45:56814) 2023-11-28T20:50:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.465+0000 7feea8f1e640 1 -- 172.21.15.45:0/630530189 learned_addr learned my addr 172.21.15.45:0/630530189 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.466+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275199402 0 0) 0x7feea415c670 con 0x7feea40cc2a0 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.466+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee90003520 con 0x7feea40cc2a0 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.466+0000 7feea9f20640 1 --2- 172.21.15.45:0/630530189 >> v2:172.21.15.190:3300/0 conn(0x7feea415c050 0x7feea4166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.466+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2758063386 0 0) 0x7fee90003520 con 0x7feea40cc2a0 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.466+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 >> v2:172.21.15.190:3300/0 conn(0x7feea415c050 msgr2=0x7feea4166020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.466+0000 7fee9bfff640 1 --2- 172.21.15.45:0/630530189 >> v2:172.21.15.190:3300/0 conn(0x7feea415c050 0x7feea4166020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.466+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 >> v1:172.21.15.190:6789/0 conn(0x7feea4141600 legacy=0x7feea40cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.466+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea40ccf60 con 0x7feea40cc2a0 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.466+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fee94001670 con 0x7feea40cc2a0 2023-11-28T20:50:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 >> v1:172.21.15.45:6789/0 conn(0x7feea40cc2a0 legacy=0x7feea415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7fee9bfff640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee900039c0 0x7fee90003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7fee9bfff640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee900047c0 0x7fee90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7fee9bfff640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90005180 0x7fee9000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea40ccc80 con 0x7fee900039c0 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9000c080 con 0x7fee90005180 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9000c240 con 0x7fee900047c0 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7feea971f640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee900039c0 0x7fee90003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7feea9f20640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90005180 0x7fee9000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7feea971f640 1 -- 172.21.15.45:0/630530189 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee900047c0 msgr2=0x7fee90004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.467+0000 7feea971f640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee900047c0 0x7fee90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.468+0000 7feea971f640 1 -- 172.21.15.45:0/630530189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90005180 msgr2=0x7fee9000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.468+0000 7feea971f640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee90005180 0x7fee9000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.468+0000 7feea971f640 1 -- 172.21.15.45:0/630530189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee90003520 con 0x7fee900039c0 2023-11-28T20:50:22.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.468+0000 7feea9f20640 1 -- 172.21.15.45:0/630530189 reap_dead start 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.469+0000 7feea971f640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee900039c0 0x7fee90003de0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7feea0001750 tx=0x7fee900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb75dd6820738cb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.469+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea00052d0 con 0x7fee900039c0 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.469+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea002e050 con 0x7fee900039c0 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.469+0000 7fee9bfff640 1 -- 172.21.15.45:0/630530189 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0003830 con 0x7fee900039c0 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.470+0000 7feeaa721640 1 -- 172.21.15.45:0/630530189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee900039c0 msgr2=0x7fee90003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.470+0000 7feeaa721640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee900039c0 0x7fee90003de0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7feea0001750 tx=0x7fee900057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.470+0000 7feeaa721640 1 -- 172.21.15.45:0/630530189 shutdown_connections 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.470+0000 7feeaa721640 1 --2- 172.21.15.45:0/630530189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee900039c0 0x7fee90003de0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.470+0000 7feeaa721640 1 -- 172.21.15.45:0/630530189 >> 172.21.15.45:0/630530189 conn(0x7feea40b0e70 msgr2=0x7feea40b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.470+0000 7feeaa721640 1 -- 172.21.15.45:0/630530189 shutdown_connections 2023-11-28T20:50:22.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.470+0000 7feeaa721640 1 -- 172.21.15.45:0/630530189 wait complete. 2023-11-28T20:50:22.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.472+0000 7feeaa721640 1 Processor -- start 2023-11-28T20:50:22.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.473+0000 7feeaa721640 1 -- start start 2023-11-28T20:50:22.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.474+0000 7feeaa721640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee900039c0 0x7feea40abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.474+0000 7feeaa721640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feea413f330 0x7feea40ad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.474+0000 7feeaa721640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feea41372a0 0x7feea4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.474+0000 7feeaa721640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea40acf60 con 0x7feea413f330 2023-11-28T20:50:22.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.474+0000 7feeaa721640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea40ac350 con 0x7feea41372a0 2023-11-28T20:50:22.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.474+0000 7feeaa721640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea40ac630 con 0x7fee900039c0 2023-11-28T20:50:22.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.474+0000 7feea8f1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feea413f330 0x7feea40ad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea8f1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feea413f330 0x7feea40ad3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34652/0 (socket says 172.21.15.45:34652) 2023-11-28T20:50:22.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea8f1e640 1 -- 172.21.15.45:0/2318011607 learned_addr learned my addr 172.21.15.45:0/2318011607 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:22.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea971f640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee900039c0 0x7feea40abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea9f20640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feea41372a0 0x7feea4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea8f1e640 1 -- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee900039c0 msgr2=0x7feea40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea8f1e640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee900039c0 0x7feea40abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea8f1e640 1 -- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feea41372a0 msgr2=0x7feea4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea8f1e640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feea41372a0 0x7feea4099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea8f1e640 1 -- 172.21.15.45:0/2318011607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea40ac910 con 0x7feea413f330 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea971f640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee900039c0 0x7feea40abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea9f20640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feea41372a0 0x7feea4099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.475+0000 7feea8f1e640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feea413f330 0x7feea40ad3f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c700 tx=0x7fee9400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.476+0000 7fee99ffb640 1 -- 172.21.15.45:0/2318011607 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee940072d0 con 0x7feea413f330 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.476+0000 7fee99ffb640 1 -- 172.21.15.45:0/2318011607 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee94007430 con 0x7feea413f330 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.476+0000 7fee99ffb640 1 -- 172.21.15.45:0/2318011607 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94004a20 con 0x7feea413f330 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.476+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea40acbf0 con 0x7feea413f330 2023-11-28T20:50:22.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.476+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea4137720 con 0x7feea413f330 2023-11-28T20:50:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.477+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea406daa0 con 0x7feea413f330 2023-11-28T20:50:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.478+0000 7fee99ffb640 1 -- 172.21.15.45:0/2318011607 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fee94004b80 con 0x7feea413f330 2023-11-28T20:50:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.479+0000 7fee99ffb640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee8806af60 0x7fee8806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.479+0000 7fee99ffb640 1 -- 172.21.15.45:0/2318011607 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fee94088a00 con 0x7feea413f330 2023-11-28T20:50:22.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.479+0000 7feea971f640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee8806af60 0x7fee8806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.481+0000 7feea971f640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee8806af60 0x7fee8806d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feea0033000 tx=0x7feea0009030 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.483+0000 7fee99ffb640 1 -- 172.21.15.45:0/2318011607 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee94053b10 con 0x7feea413f330 2023-11-28T20:50:22.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.736+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7feea404e920 con 0x7feea413f330 2023-11-28T20:50:22.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.737+0000 7fee99ffb640 1 -- 172.21.15.45:0/2318011607 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fee94018020 con 0x7feea413f330 2023-11-28T20:50:22.740 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:22.740 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:50:22.740 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T20:50:22.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.740 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T20:50:22.741 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T20:50:22.741 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.741 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:50:22.741 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:50:22.741 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:50:22.741 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.741 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.742 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T20:50:22.742 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T20:50:22.742 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.742 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:50:22.742 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:50:22.742 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:50:22.743 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.743 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.743 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T20:50:22.743 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T20:50:22.743 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.744 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:50:22.744 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:50:22.744 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:50:22.744 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.744 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.744 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T20:50:22.745 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T20:50:22.745 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.745 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:50:22.745 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:50:22.745 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:50:22.745 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.745 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.746 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T20:50:22.746 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T20:50:22.746 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.746 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:50:22.746 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:50:22.746 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.746 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.747 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T20:50:22.747 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T20:50:22.747 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.747 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:50:22.747 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:50:22.747 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.748 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.748 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T20:50:22.748 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T20:50:22.748 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.748 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:50:22.748 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:50:22.749 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.749 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.749 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T20:50:22.749 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T20:50:22.749 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.749 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:50:22.750 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:50:22.750 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.750 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T20:50:22.750 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T20:50:22.750 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.751 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:50:22.751 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:50:22.751 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.751 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.751 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T20:50:22.751 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T20:50:22.752 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.752 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:50:22.752 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:50:22.752 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.752 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.752 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T20:50:22.753 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T20:50:22.753 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.753 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:50:22.753 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:50:22.753 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.753 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.754 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T20:50:22.754 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T20:50:22.754 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.754 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:50:22.754 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:50:22.754 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.755 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T20:50:22.755 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T20:50:22.755 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.755 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:50:22.755 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T20:50:22.755 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T20:50:22.755 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.756 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T20:50:22.756 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T20:50:22.756 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.756 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:50:22.756 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:50:22.756 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:50:22.757 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:50:22.757 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.757 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T20:50:22.757 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T20:50:22.757 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.757 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:50:22.758 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:50:22.758 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:50:22.758 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:50:22.758 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.758 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.759 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:50:22.760 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:50:22.761 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:50:22.761 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.762 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T20:50:22.762 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T20:50:22.762 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:50:22.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:50:22.762 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.763 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:50:22.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:22.763 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T20:50:22.763 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T20:50:22.763 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:50:22.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:50:22.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:50:22.764 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:50:22.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:22.764 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T20:50:22.764 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:50:22.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee8806af60 msgr2=0x7fee8806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee8806af60 0x7fee8806d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feea0033000 tx=0x7feea0009030 comp rx=0 tx=0).stop 2023-11-28T20:50:22.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feea413f330 msgr2=0x7feea40ad3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feea413f330 0x7feea40ad3f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c700 tx=0x7fee9400cbc0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 shutdown_connections 2023-11-28T20:50:22.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fee8806af60 0x7fee8806d420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feea41372a0 0x7feea4099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feea413f330 0x7feea40ad3f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 --2- 172.21.15.45:0/2318011607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee900039c0 0x7feea40abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 >> 172.21.15.45:0/2318011607 conn(0x7feea40b0e70 msgr2=0x7feea415f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:22.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 shutdown_connections 2023-11-28T20:50:22.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.741+0000 7feeaa721640 1 -- 172.21.15.45:0/2318011607 wait complete. 2023-11-28T20:50:22.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:22.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.976+0000 7fc13dfbc640 1 Processor -- start 2023-11-28T20:50:22.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.976+0000 7fc13dfbc640 1 -- start start 2023-11-28T20:50:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.978+0000 7fc13dfbc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc138072200 0x7fc1380b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.978+0000 7fc13dfbc640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc1380713e0 0x7fc1380b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.978+0000 7fc13dfbc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1380b4310 con 0x7fc138146610 2023-11-28T20:50:22.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.978+0000 7fc13dfbc640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1380725e0 con 0x7fc138072200 2023-11-28T20:50:22.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.978+0000 7fc13dfbc640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13806d8d0 con 0x7fc1380713e0 2023-11-28T20:50:22.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.979+0000 7fc13cfba640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc138072200 0x7fc1380b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.979+0000 7fc13cfba640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc138072200 0x7fc1380b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34654/0 (socket says 172.21.15.45:34654) 2023-11-28T20:50:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.979+0000 7fc13cfba640 1 -- 172.21.15.45:0/2199046613 learned_addr learned my addr 172.21.15.45:0/2199046613 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.979+0000 7fc13d7bb640 1 --2- 172.21.15.45:0/2199046613 >> v2:172.21.15.190:3300/0 conn(0x7fc1380713e0 0x7fc1380b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.980+0000 7fc13cfba640 1 -- 172.21.15.45:0/2199046613 >> v2:172.21.15.190:3300/0 conn(0x7fc1380713e0 msgr2=0x7fc1380b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.980+0000 7fc13cfba640 1 --2- 172.21.15.45:0/2199046613 >> v2:172.21.15.190:3300/0 conn(0x7fc1380713e0 0x7fc1380b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.980+0000 7fc13cfba640 1 -- 172.21.15.45:0/2199046613 >> v1:172.21.15.190:6789/0 conn(0x7fc138146610 legacy=0x7fc138160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.980+0000 7fc13cfba640 1 -- 172.21.15.45:0/2199046613 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13806ca20 con 0x7fc138072200 2023-11-28T20:50:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.982+0000 7fc13cfba640 1 --2- 172.21.15.45:0/2199046613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 0x7fc1380b1d90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc13806b390 tx=0x7fc12c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c12a033a56a5d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:22.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.982+0000 7fc1377fe640 1 -- 172.21.15.45:0/2199046613 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c030860 con 0x7fc138072200 2023-11-28T20:50:22.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.982+0000 7fc1377fe640 1 -- 172.21.15.45:0/2199046613 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c0309c0 con 0x7fc138072200 2023-11-28T20:50:22.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.982+0000 7fc1377fe640 1 -- 172.21.15.45:0/2199046613 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c030d50 con 0x7fc138072200 2023-11-28T20:50:22.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.983+0000 7fc13dfbc640 1 -- 172.21.15.45:0/2199046613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 msgr2=0x7fc1380b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.983+0000 7fc13dfbc640 1 --2- 172.21.15.45:0/2199046613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 0x7fc1380b1d90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc13806b390 tx=0x7fc12c02b9a0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.983+0000 7fc13dfbc640 1 -- 172.21.15.45:0/2199046613 shutdown_connections 2023-11-28T20:50:22.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.983+0000 7fc13dfbc640 1 --2- 172.21.15.45:0/2199046613 >> v2:172.21.15.190:3300/0 conn(0x7fc1380713e0 0x7fc1380b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.983+0000 7fc13dfbc640 1 --2- 172.21.15.45:0/2199046613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 0x7fc1380b1d90 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.983+0000 7fc13dfbc640 1 -- 172.21.15.45:0/2199046613 >> 172.21.15.45:0/2199046613 conn(0x7fc13806ab30 msgr2=0x7fc1380b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:22.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.983+0000 7fc13dfbc640 1 -- 172.21.15.45:0/2199046613 shutdown_connections 2023-11-28T20:50:22.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.983+0000 7fc13dfbc640 1 -- 172.21.15.45:0/2199046613 wait complete. 2023-11-28T20:50:22.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.985+0000 7fc13dfbc640 1 Processor -- start 2023-11-28T20:50:22.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.985+0000 7fc13dfbc640 1 -- start start 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc13dfbc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1380713e0 0x7fc1380b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc13dfbc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 0x7fc1381417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc13dfbc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc138146610 0x7fc1381f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc13dfbc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1380b1210 con 0x7fc138072200 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc13dfbc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1381421b0 con 0x7fc1380713e0 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc13dfbc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc138142490 con 0x7fc138146610 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc137fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 0x7fc1381417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc137fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 0x7fc1381417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34660/0 (socket says 172.21.15.45:34660) 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc137fff640 1 -- 172.21.15.45:0/57017631 learned_addr learned my addr 172.21.15.45:0/57017631 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc13cfba640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1380713e0 0x7fc1380b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc137fff640 1 -- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc138146610 msgr2=0x7fc1381f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc13d7bb640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc138146610 0x7fc1381f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc137fff640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc138146610 0x7fc1381f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc137fff640 1 -- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1380713e0 msgr2=0x7fc1380b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.987+0000 7fc137fff640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1380713e0 0x7fc1380b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.988+0000 7fc137fff640 1 -- 172.21.15.45:0/57017631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c0094d0 con 0x7fc138072200 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.988+0000 7fc13cfba640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1380713e0 0x7fc1380b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.988+0000 7fc137fff640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 0x7fc1381417b0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c400 tx=0x7fc12400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.988+0000 7fc1357fa640 1 -- 172.21.15.45:0/57017631 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1240052d0 con 0x7fc138072200 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.988+0000 7fc1357fa640 1 -- 172.21.15.45:0/57017631 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc124005430 con 0x7fc138072200 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.988+0000 7fc1357fa640 1 -- 172.21.15.45:0/57017631 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124003830 con 0x7fc138072200 2023-11-28T20:50:22.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.988+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc138142770 con 0x7fc138072200 2023-11-28T20:50:22.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.988+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13813f990 con 0x7fc138072200 2023-11-28T20:50:22.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.989+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13813fb20 con 0x7fc138072200 2023-11-28T20:50:22.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.990+0000 7fc1357fa640 1 -- 172.21.15.45:0/57017631 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc124003990 con 0x7fc138072200 2023-11-28T20:50:22.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.991+0000 7fc1357fa640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc11c06af60 0x7fc11c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:22.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.991+0000 7fc1357fa640 1 -- 172.21.15.45:0/57017631 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fc124088a50 con 0x7fc138072200 2023-11-28T20:50:22.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.992+0000 7fc13cfba640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc11c06af60 0x7fc11c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:22.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.993+0000 7fc13cfba640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc11c06af60 0x7fc11c06d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc12c036000 tx=0x7fc12c03b040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:22.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:22.995+0000 7fc1357fa640 1 -- 172.21.15.45:0/57017631 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc124053b60 con 0x7fc138072200 2023-11-28T20:50:23.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.191+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1380b42e0 con 0x7fc138072200 2023-11-28T20:50:23.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.193+0000 7fc1357fa640 1 -- 172.21.15.45:0/57017631 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+13050 (secure 0 0 0) 0x7fc124017090 con 0x7fc138072200 2023-11-28T20:50:23.195 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:23.195 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:22.246067+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:23.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.195+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc11c06af60 msgr2=0x7fc11c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.195+0000 7fc13dfbc640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc11c06af60 0x7fc11c06d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc12c036000 tx=0x7fc12c03b040 comp rx=0 tx=0).stop 2023-11-28T20:50:23.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 msgr2=0x7fc1381417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 0x7fc1381417b0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c400 tx=0x7fc12400c8c0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 shutdown_connections 2023-11-28T20:50:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fc11c06af60 0x7fc11c06d420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc138146610 0x7fc1381f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc138072200 0x7fc1381417b0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 --2- 172.21.15.45:0/57017631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1380713e0 0x7fc1380b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 >> 172.21.15.45:0/57017631 conn(0x7fc13806ab30 msgr2=0x7fc13815ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:23.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 shutdown_connections 2023-11-28T20:50:23.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.196+0000 7fc13dfbc640 1 -- 172.21.15.45:0/57017631 wait complete. 2023-11-28T20:50:23.219 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:50:23.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:50:23.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.442+0000 7f91c6f05640 1 Processor -- start 2023-11-28T20:50:23.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.443+0000 7f91c6f05640 1 -- start start 2023-11-28T20:50:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.444+0000 7f91c6f05640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c015e840 0x7f91c006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.444+0000 7f91c6f05640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f91c006ac10 0x7f91c00717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.444+0000 7f91c6f05640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c015ecd0 con 0x7f91c015f720 2023-11-28T20:50:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.444+0000 7f91c6f05640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c0071ce0 con 0x7f91c006ac10 2023-11-28T20:50:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.444+0000 7f91c6f05640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c015f290 con 0x7f91c015e840 2023-11-28T20:50:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.445+0000 7f91c54a2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f91c006ac10 0x7f91c00717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.445+0000 7f91c54a2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f91c006ac10 0x7f91c00717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34664/0 (socket says 172.21.15.45:34664) 2023-11-28T20:50:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.445+0000 7f91c54a2640 1 -- 172.21.15.45:0/1355536162 learned_addr learned my addr 172.21.15.45:0/1355536162 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.445+0000 7f91c4ca1640 1 --2- 172.21.15.45:0/1355536162 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c015e840 0x7f91c006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.445+0000 7f91c54a2640 1 -- 172.21.15.45:0/1355536162 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c015e840 msgr2=0x7f91c006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.445+0000 7f91c54a2640 1 --2- 172.21.15.45:0/1355536162 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c015e840 0x7f91c006c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.445+0000 7f91c54a2640 1 -- 172.21.15.45:0/1355536162 >> v1:172.21.15.190:6789/0 conn(0x7f91c015f720 legacy=0x7f91c006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.445+0000 7f91c54a2640 1 -- 172.21.15.45:0/1355536162 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c015fb60 con 0x7f91c006ac10 2023-11-28T20:50:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.446+0000 7f91c54a2640 1 --2- 172.21.15.45:0/1355536162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c006ac10 0x7f91c00717a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009e50 tx=0x7f91b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a2ade8b605ddc4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.447+0000 7f91bf7fe640 1 -- 172.21.15.45:0/1355536162 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b802ca70 con 0x7f91c006ac10 2023-11-28T20:50:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.447+0000 7f91bf7fe640 1 -- 172.21.15.45:0/1355536162 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b802cbd0 con 0x7f91c006ac10 2023-11-28T20:50:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.447+0000 7f91bf7fe640 1 -- 172.21.15.45:0/1355536162 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b80343f0 con 0x7f91c006ac10 2023-11-28T20:50:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.447+0000 7f91c6f05640 1 -- 172.21.15.45:0/1355536162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c006ac10 msgr2=0x7f91c00717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.447+0000 7f91c6f05640 1 --2- 172.21.15.45:0/1355536162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c006ac10 0x7f91c00717a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009e50 tx=0x7f91b802bed0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.448+0000 7f91c6f05640 1 -- 172.21.15.45:0/1355536162 shutdown_connections 2023-11-28T20:50:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.448+0000 7f91c6f05640 1 --2- 172.21.15.45:0/1355536162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c006ac10 0x7f91c00717a0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.448+0000 7f91c6f05640 1 --2- 172.21.15.45:0/1355536162 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c015e840 0x7f91c006c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.448+0000 7f91c6f05640 1 -- 172.21.15.45:0/1355536162 >> 172.21.15.45:0/1355536162 conn(0x7f91c00ced60 msgr2=0x7f91c00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.448+0000 7f91c6f05640 1 -- 172.21.15.45:0/1355536162 shutdown_connections 2023-11-28T20:50:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.448+0000 7f91c6f05640 1 -- 172.21.15.45:0/1355536162 wait complete. 2023-11-28T20:50:23.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.449+0000 7f91c6f05640 1 Processor -- start 2023-11-28T20:50:23.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.450+0000 7f91c6f05640 1 -- start start 2023-11-28T20:50:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.450+0000 7f91c6f05640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c006ac10 0x7f91c00c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91c6f05640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c015e840 0x7f91c00c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91c6f05640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91c015f720 0x7f91c01f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91c6f05640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c01f3820 con 0x7f91c015e840 2023-11-28T20:50:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91c6f05640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c00c78a0 con 0x7f91c015f720 2023-11-28T20:50:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91c6f05640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c00c8140 con 0x7f91c006ac10 2023-11-28T20:50:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91bffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c015e840 0x7f91c00c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91bffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c015e840 0x7f91c00c4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34670/0 (socket says 172.21.15.45:34670) 2023-11-28T20:50:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91bffff640 1 -- 172.21.15.45:0/1239524333 learned_addr learned my addr 172.21.15.45:0/1239524333 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91c54a2640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91c015f720 0x7f91c01f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91c4ca1640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c006ac10 0x7f91c00c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91bffff640 1 -- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c006ac10 msgr2=0x7f91c00c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91bffff640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c006ac10 0x7f91c00c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91bffff640 1 -- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91c015f720 msgr2=0x7f91c01f6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91bffff640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91c015f720 0x7f91c01f6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.451+0000 7f91bffff640 1 -- 172.21.15.45:0/1239524333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b8009b30 con 0x7f91c015e840 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.452+0000 7f91bffff640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c015e840 0x7f91c00c4b30 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00b3f0 tx=0x7f91ac00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.452+0000 7f91bd7fa640 1 -- 172.21.15.45:0/1239524333 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac011830 con 0x7f91c015e840 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.452+0000 7f91bd7fa640 1 -- 172.21.15.45:0/1239524333 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ac011990 con 0x7f91c015e840 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.452+0000 7f91bd7fa640 1 -- 172.21.15.45:0/1239524333 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac011d80 con 0x7f91c015e840 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.452+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c01f7bf0 con 0x7f91c015e840 2023-11-28T20:50:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.452+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c00c7dd0 con 0x7f91c015e840 2023-11-28T20:50:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.453+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c00cd3d0 con 0x7f91c015e840 2023-11-28T20:50:23.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.454+0000 7f91bd7fa640 1 -- 172.21.15.45:0/1239524333 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f91ac01a400 con 0x7f91c015e840 2023-11-28T20:50:23.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.455+0000 7f91bd7fa640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f91a406af90 0x7f91a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.456+0000 7f91bd7fa640 1 -- 172.21.15.45:0/1239524333 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f91ac089320 con 0x7f91c015e840 2023-11-28T20:50:23.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.456+0000 7f91c4ca1640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f91a406af90 0x7f91a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.457+0000 7f91c4ca1640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f91a406af90 0x7f91a406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f91b4006fd0 tx=0x7f91b4008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:23.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.461+0000 7f91bd7fa640 1 -- 172.21.15.45:0/1239524333 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91ac0544c0 con 0x7f91c015e840 2023-11-28T20:50:23.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.652+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91c006b3a0 con 0x7f91c015e840 2023-11-28T20:50:23.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.652+0000 7f91bd7fa640 1 -- 172.21.15.45:0/1239524333 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91ac057ae0 con 0x7f91c015e840 2023-11-28T20:50:23.655 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:50:23.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f91a406af90 msgr2=0x7f91a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f91a406af90 0x7f91a406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f91b4006fd0 tx=0x7f91b4008040 comp rx=0 tx=0).stop 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c015e840 msgr2=0x7f91c00c4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c015e840 0x7f91c00c4b30 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00b3f0 tx=0x7f91ac00b8b0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 shutdown_connections 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f91a406af90 0x7f91a406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91c015f720 0x7f91c01f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c015e840 0x7f91c00c4b30 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 --2- 172.21.15.45:0/1239524333 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91c006ac10 0x7f91c00c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.655+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 >> 172.21.15.45:0/1239524333 conn(0x7f91c00ced60 msgr2=0x7f91c00cccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:23.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.656+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 shutdown_connections 2023-11-28T20:50:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.656+0000 7f91c6f05640 1 -- 172.21.15.45:0/1239524333 wait complete. 2023-11-28T20:50:23.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T20:50:23.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.904+0000 7fa23ffff640 1 Processor -- start 2023-11-28T20:50:23.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.904+0000 7fa23ffff640 1 -- start start 2023-11-28T20:50:23.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.905+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa240161640 0x7fa24009a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.905+0000 7fa23ffff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2400b45e0 con 0x7fa24006af10 2023-11-28T20:50:23.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.905+0000 7fa23ffff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24006b920 con 0x7fa24015d2f0 2023-11-28T20:50:23.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.905+0000 7fa23ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24006b3c0 con 0x7fa240161640 2023-11-28T20:50:23.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.906+0000 7fa23f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa24006af10 0x7fa2400b6040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42784/0 (socket says 172.21.15.45:42784) 2023-11-28T20:50:23.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.906+0000 7fa23f7fe640 1 -- 172.21.15.45:0/855620859 learned_addr learned my addr 172.21.15.45:0/855620859 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:23.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.906+0000 7fa2367fc640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa240161640 0x7fa24009a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.906+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663255433 0 0) 0x7fa2400b45e0 con 0x7fa24006af10 2023-11-28T20:50:23.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.906+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa22c003520 con 0x7fa24006af10 2023-11-28T20:50:23.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1365243296 0 0) 0x7fa22c003520 con 0x7fa24006af10 2023-11-28T20:50:23.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa240161640 msgr2=0x7fa24009a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa240161640 0x7fa24009a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 >> v1:172.21.15.190:6789/0 conn(0x7fa24015d2f0 legacy=0x7fa240160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24006b640 con 0x7fa24006af10 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fa2380019f0 con 0x7fa24006af10 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 >> v1:172.21.15.45:6789/0 conn(0x7fa24006af10 legacy=0x7fa2400b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa22c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa22c0047c0 0x7fa22c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa240161640 0x7fa22c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24006b3c0 con 0x7fa22c0039a0 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c00bac0 con 0x7fa22c0047c0 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c00bc80 con 0x7fa240161640 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23effd640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa22c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa23f7fe640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa240161640 0x7fa22c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.907+0000 7fa2367fc640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa22c0047c0 0x7fa22c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.908+0000 7fa23effd640 1 -- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa240161640 msgr2=0x7fa22c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:23.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.908+0000 7fa23effd640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa240161640 0x7fa22c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.908+0000 7fa23effd640 1 -- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa22c0047c0 msgr2=0x7fa22c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:23.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.908+0000 7fa23effd640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa22c0047c0 0x7fa22c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.908+0000 7fa23effd640 1 -- 172.21.15.45:0/855620859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22c003520 con 0x7fa22c0039a0 2023-11-28T20:50:23.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.909+0000 7fa23effd640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa22c003dc0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa230001720 tx=0x7fa22c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181d47be742d92f4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:23.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.909+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2300052d0 con 0x7fa22c0039a0 2023-11-28T20:50:23.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.909+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23002e050 con 0x7fa22c0039a0 2023-11-28T20:50:23.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.909+0000 7fa23e7fc640 1 -- 172.21.15.45:0/855620859 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2300035f0 con 0x7fa22c0039a0 2023-11-28T20:50:23.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.910+0000 7fa23ffff640 1 -- 172.21.15.45:0/855620859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 msgr2=0x7fa22c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.910+0000 7fa23ffff640 1 --2- 172.21.15.45:0/855620859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa22c003dc0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa230001720 tx=0x7fa22c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.910+0000 7fa23f7fe640 1 -- 172.21.15.45:0/855620859 reap_dead start 2023-11-28T20:50:23.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.910+0000 7fa23ffff640 1 -- 172.21.15.45:0/855620859 shutdown_connections 2023-11-28T20:50:23.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.910+0000 7fa23ffff640 1 -- 172.21.15.45:0/855620859 >> 172.21.15.45:0/855620859 conn(0x7fa240099a40 msgr2=0x7fa240099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:23.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.910+0000 7fa23ffff640 1 -- 172.21.15.45:0/855620859 shutdown_connections 2023-11-28T20:50:23.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.910+0000 7fa23ffff640 1 -- 172.21.15.45:0/855620859 wait complete. 2023-11-28T20:50:23.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.912+0000 7fa23ffff640 1 Processor -- start 2023-11-28T20:50:23.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.912+0000 7fa23ffff640 1 -- start start 2023-11-28T20:50:23.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.913+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa240148480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.913+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2400c8090 0x7fa240147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.913+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa24006f5b0 0x7fa24006fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.913+0000 7fa23ffff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24014b4e0 con 0x7fa22c0039a0 2023-11-28T20:50:23.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.913+0000 7fa23ffff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24006ff70 con 0x7fa2400c8090 2023-11-28T20:50:23.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.913+0000 7fa23ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2400ca310 con 0x7fa24006f5b0 2023-11-28T20:50:23.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.913+0000 7fa23effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa240148480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.913+0000 7fa23effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa240148480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53382/0 (socket says 172.21.15.45:53382) 2023-11-28T20:50:23.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.913+0000 7fa23effd640 1 -- 172.21.15.45:0/2187463153 learned_addr learned my addr 172.21.15.45:0/2187463153 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23effd640 1 -- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa24006f5b0 msgr2=0x7fa24006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa2367fc640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2400c8090 0x7fa240147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23effd640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa24006f5b0 0x7fa24006fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23effd640 1 -- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2400c8090 msgr2=0x7fa240147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23effd640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2400c8090 0x7fa240147c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23effd640 1 -- 172.21.15.45:0/2187463153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2401489c0 con 0x7fa22c0039a0 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23effd640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa240148480 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa230008fd0 tx=0x7fa230001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23cff9640 1 -- 172.21.15.45:0/2187463153 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2300076c0 con 0x7fa22c0039a0 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23cff9640 1 -- 172.21.15.45:0/2187463153 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa230007a30 con 0x7fa22c0039a0 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23cff9640 1 -- 172.21.15.45:0/2187463153 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa230006be0 con 0x7fa22c0039a0 2023-11-28T20:50:23.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2400c9a40 con 0x7fa22c0039a0 2023-11-28T20:50:23.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.914+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24014b200 con 0x7fa22c0039a0 2023-11-28T20:50:23.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.916+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24009f070 con 0x7fa22c0039a0 2023-11-28T20:50:23.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.916+0000 7fa23cff9640 1 -- 172.21.15.45:0/2187463153 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa230006d40 con 0x7fa22c0039a0 2023-11-28T20:50:23.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.917+0000 7fa23cff9640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa22006af60 0x7fa22006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:23.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.917+0000 7fa23cff9640 1 -- 172.21.15.45:0/2187463153 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa2300ab6d0 con 0x7fa22c0039a0 2023-11-28T20:50:23.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.918+0000 7fa2367fc640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa22006af60 0x7fa22006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:23.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.919+0000 7fa2367fc640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa22006af60 0x7fa22006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2400b6350 tx=0x7fa228008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:23.922+0000 7fa23cff9640 1 -- 172.21.15.45:0/2187463153 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa230076800 con 0x7fa22c0039a0 2023-11-28T20:50:24.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.123+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa24004e920 con 0x7fa22c0039a0 2023-11-28T20:50:24.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.124+0000 7fa23cff9640 1 -- 172.21.15.45:0/2187463153 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v84) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa23004b060 con 0x7fa22c0039a0 2023-11-28T20:50:24.127 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:24.127 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:50:24.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:24.127 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T20:50:24.127 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T20:50:24.127 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:24.127 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:50:24.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa22006af60 msgr2=0x7fa22006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:24.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa22006af60 0x7fa22006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2400b6350 tx=0x7fa228008040 comp rx=0 tx=0).stop 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 msgr2=0x7fa240148480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa240148480 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa230008fd0 tx=0x7fa230001120 comp rx=0 tx=0).stop 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 shutdown_connections 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa22006af60 0x7fa22006d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa24006f5b0 0x7fa24006fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2400c8090 0x7fa240147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 --2- 172.21.15.45:0/2187463153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c0039a0 0x7fa240148480 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 >> 172.21.15.45:0/2187463153 conn(0x7fa240099a40 msgr2=0x7fa2400b65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 shutdown_connections 2023-11-28T20:50:24.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.127+0000 7fa23ffff640 1 -- 172.21.15.45:0/2187463153 wait complete. 2023-11-28T20:50:24.149 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T20:50:24.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T20:50:24.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.371+0000 7ff7ef3f4640 1 Processor -- start 2023-11-28T20:50:24.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.371+0000 7ff7ef3f4640 1 -- start start 2023-11-28T20:50:24.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.372+0000 7ff7ef3f4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff7e815e510 0x7ff7e806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:24.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.372+0000 7ff7ef3f4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e806dde0 0x7ff7e80707f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:24.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.372+0000 7ff7ef3f4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff7e8070d30 0x7ff7e8160df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:24.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.372+0000 7ff7ef3f4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e80711a0 con 0x7ff7e815e510 2023-11-28T20:50:24.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.372+0000 7ff7ef3f4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e80ac8b0 con 0x7ff7e8070d30 2023-11-28T20:50:24.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.372+0000 7ff7ef3f4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e806e2f0 con 0x7ff7e806dde0 2023-11-28T20:50:24.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.372+0000 7ff7ee3f2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff7e815e510 0x7ff7e806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:24.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7ee3f2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff7e815e510 0x7ff7e806d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53394/0 (socket says 172.21.15.45:53394) 2023-11-28T20:50:24.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7ee3f2640 1 -- 172.21.15.45:0/561812396 learned_addr learned my addr 172.21.15.45:0/561812396 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:24.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7ee3f2640 1 -- 172.21.15.45:0/561812396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e806dde0 msgr2=0x7ff7e80707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:24.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7eebf3640 1 --2- 172.21.15.45:0/561812396 >> v2:172.21.15.190:3300/0 conn(0x7ff7e8070d30 0x7ff7e8160df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:24.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7edbf1640 1 --2- 172.21.15.45:0/561812396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e806dde0 0x7ff7e80707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:24.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7ee3f2640 1 --2- 172.21.15.45:0/561812396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e806dde0 0x7ff7e80707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7ee3f2640 1 -- 172.21.15.45:0/561812396 >> v2:172.21.15.190:3300/0 conn(0x7ff7e8070d30 msgr2=0x7ff7e8160df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:24.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7ee3f2640 1 --2- 172.21.15.45:0/561812396 >> v2:172.21.15.190:3300/0 conn(0x7ff7e8070d30 0x7ff7e8160df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7ee3f2640 1 -- 172.21.15.45:0/561812396 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e806e5d0 con 0x7ff7e815e510 2023-11-28T20:50:24.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7edbf1640 1 --2- 172.21.15.45:0/561812396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e806dde0 0x7ff7e80707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:24.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.373+0000 7ff7eebf3640 1 --2- 172.21.15.45:0/561812396 >> v2:172.21.15.190:3300/0 conn(0x7ff7e8070d30 0x7ff7e8160df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:24.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.375+0000 7ff7ee3f2640 1 --2- 172.21.15.45:0/561812396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 0x7ff7e806d8a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80cb3b0 tx=0x7ff7e402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e32b32ecc5a91de server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:24.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.375+0000 7ff7ed3f0640 1 -- 172.21.15.45:0/561812396 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e402c950 con 0x7ff7e815e510 2023-11-28T20:50:24.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.375+0000 7ff7ed3f0640 1 -- 172.21.15.45:0/561812396 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e402cab0 con 0x7ff7e815e510 2023-11-28T20:50:24.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.375+0000 7ff7ed3f0640 1 -- 172.21.15.45:0/561812396 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e402ce40 con 0x7ff7e815e510 2023-11-28T20:50:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.376+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/561812396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 msgr2=0x7ff7e806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.376+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/561812396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 0x7ff7e806d8a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80cb3b0 tx=0x7ff7e402bdb0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.376+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/561812396 shutdown_connections 2023-11-28T20:50:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.376+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/561812396 >> v2:172.21.15.190:3300/0 conn(0x7ff7e8070d30 0x7ff7e8160df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.376+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/561812396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e806dde0 0x7ff7e80707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.376+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/561812396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 0x7ff7e806d8a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.376+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/561812396 >> 172.21.15.45:0/561812396 conn(0x7ff7e80b0da0 msgr2=0x7ff7e80b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.376+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/561812396 shutdown_connections 2023-11-28T20:50:24.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.376+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/561812396 wait complete. 2023-11-28T20:50:24.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.378+0000 7ff7ef3f4640 1 Processor -- start 2023-11-28T20:50:24.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.378+0000 7ff7ef3f4640 1 -- start start 2023-11-28T20:50:24.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7ef3f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7e806dde0 0x7ff7e8155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:24.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7ef3f4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e8070d30 0x7ff7e81537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:24.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7ef3f4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 0x7ff7e8156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:24.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7ef3f4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e81568b0 con 0x7ff7e815e510 2023-11-28T20:50:24.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7ef3f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e8155c30 con 0x7ff7e806dde0 2023-11-28T20:50:24.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7ef3f4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e8155f10 con 0x7ff7e8070d30 2023-11-28T20:50:24.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7eebf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 0x7ff7e8156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:24.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7eebf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 0x7ff7e8156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53406/0 (socket says 172.21.15.45:53406) 2023-11-28T20:50:24.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7eebf3640 1 -- 172.21.15.45:0/4159010913 learned_addr learned my addr 172.21.15.45:0/4159010913 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7edbf1640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e8070d30 0x7ff7e81537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.379+0000 7ff7ee3f2640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7e806dde0 0x7ff7e8155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7eebf3640 1 -- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e8070d30 msgr2=0x7ff7e81537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7eebf3640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e8070d30 0x7ff7e81537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7eebf3640 1 -- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7e806dde0 msgr2=0x7ff7e8155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:24.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7eebf3640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7e806dde0 0x7ff7e8155650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7eebf3640 1 -- 172.21.15.45:0/4159010913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e4009b30 con 0x7ff7e815e510 2023-11-28T20:50:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7edbf1640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e8070d30 0x7ff7e81537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7ee3f2640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7e806dde0 0x7ff7e8155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7eebf3640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 0x7ff7e8156cd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000c7f0 tx=0x7ff7e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7deffd640 1 -- 172.21.15.45:0/4159010913 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0007820 con 0x7ff7e815e510 2023-11-28T20:50:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7deffd640 1 -- 172.21.15.45:0/4159010913 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e0007980 con 0x7ff7e815e510 2023-11-28T20:50:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e81561f0 con 0x7ff7e815e510 2023-11-28T20:50:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7deffd640 1 -- 172.21.15.45:0/4159010913 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0007d70 con 0x7ff7e815e510 2023-11-28T20:50:24.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.380+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e8153d10 con 0x7ff7e815e510 2023-11-28T20:50:24.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.382+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e80733d0 con 0x7ff7e815e510 2023-11-28T20:50:24.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.383+0000 7ff7deffd640 1 -- 172.21.15.45:0/4159010913 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff7e00113f0 con 0x7ff7e815e510 2023-11-28T20:50:24.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.384+0000 7ff7deffd640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7ff7cc06af90 0x7ff7cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:24.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.385+0000 7ff7deffd640 1 -- 172.21.15.45:0/4159010913 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff7e00899f0 con 0x7ff7e815e510 2023-11-28T20:50:24.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.385+0000 7ff7ee3f2640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7ff7cc06af90 0x7ff7cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:24.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.386+0000 7ff7ee3f2640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7ff7cc06af90 0x7ff7cc06d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4035000 tx=0x7ff7e4030000 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:24.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.389+0000 7ff7deffd640 1 -- 172.21.15.45:0/4159010913 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7e0054a80 con 0x7ff7e815e510 2023-11-28T20:50:24.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:24.597+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff7e80b1990 con 0x7ff7e815e510 2023-11-28T20:50:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.321+0000 7ff7deffd640 1 -- 172.21.15.45:0/4159010913 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v85) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff7e001f0b0 con 0x7ff7e815e510 2023-11-28T20:50:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T20:50:25.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7ff7cc06af90 msgr2=0x7ff7cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:25.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7ff7cc06af90 0x7ff7cc06d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4035000 tx=0x7ff7e4030000 comp rx=0 tx=0).stop 2023-11-28T20:50:25.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 msgr2=0x7ff7e8156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:25.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 0x7ff7e8156cd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000c7f0 tx=0x7ff7e000ccb0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 shutdown_connections 2023-11-28T20:50:25.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7ff7cc06af90 0x7ff7cc06d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e815e510 0x7ff7e8156cd0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7e8070d30 0x7ff7e81537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 --2- 172.21.15.45:0/4159010913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7e806dde0 0x7ff7e8155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.325+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 >> 172.21.15.45:0/4159010913 conn(0x7ff7e80b0da0 msgr2=0x7ff7e815c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:25.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.326+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 shutdown_connections 2023-11-28T20:50:25.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.326+0000 7ff7ef3f4640 1 -- 172.21.15.45:0/4159010913 wait complete. 2023-11-28T20:50:25.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T20:50:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.580+0000 7f8211f2e640 1 Processor -- start 2023-11-28T20:50:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.581+0000 7f8211f2e640 1 -- start start 2023-11-28T20:50:25.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.582+0000 7f8211f2e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0cde00 0x7f820c0ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:25.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.582+0000 7f8211f2e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f820c0b6380 0x7f820c0b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:25.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.582+0000 7f8211f2e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820c15d790 con 0x7f820c15d3b0 2023-11-28T20:50:25.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.582+0000 7f8211f2e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c0ce7c0 con 0x7f820c0b6380 2023-11-28T20:50:25.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.582+0000 7f8211f2e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c0d2bf0 con 0x7f820c0cde00 2023-11-28T20:50:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.582+0000 7f821172d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f820c0b6380 0x7f820c0b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.582+0000 7f821172d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f820c0b6380 0x7f820c0b6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53412/0 (socket says 172.21.15.45:53412) 2023-11-28T20:50:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.582+0000 7f821172d640 1 -- 172.21.15.45:0/2238712635 learned_addr learned my addr 172.21.15.45:0/2238712635 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.583+0000 7f820bfff640 1 --2- 172.21.15.45:0/2238712635 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0cde00 0x7f820c0ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.583+0000 7f821172d640 1 -- 172.21.15.45:0/2238712635 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0cde00 msgr2=0x7f820c0ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.583+0000 7f821172d640 1 --2- 172.21.15.45:0/2238712635 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0cde00 0x7f820c0ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.583+0000 7f821172d640 1 -- 172.21.15.45:0/2238712635 >> v1:172.21.15.190:6789/0 conn(0x7f820c15d3b0 legacy=0x7f820c0d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.583+0000 7f821172d640 1 -- 172.21.15.45:0/2238712635 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c0d2ed0 con 0x7f820c0b6380 2023-11-28T20:50:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.584+0000 7f821172d640 1 --2- 172.21.15.45:0/2238712635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c0b6380 0x7f820c0b6810 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8204009e50 tx=0x7f820402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e31e5b5a3154fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.584+0000 7f820b7fe640 1 -- 172.21.15.45:0/2238712635 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820402cb00 con 0x7f820c0b6380 2023-11-28T20:50:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.584+0000 7f820b7fe640 1 -- 172.21.15.45:0/2238712635 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820402cc60 con 0x7f820c0b6380 2023-11-28T20:50:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.585+0000 7f820b7fe640 1 -- 172.21.15.45:0/2238712635 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040343f0 con 0x7f820c0b6380 2023-11-28T20:50:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.585+0000 7f8211f2e640 1 -- 172.21.15.45:0/2238712635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c0b6380 msgr2=0x7f820c0b6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.585+0000 7f8211f2e640 1 --2- 172.21.15.45:0/2238712635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c0b6380 0x7f820c0b6810 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8204009e50 tx=0x7f820402bf60 comp rx=0 tx=0).stop 2023-11-28T20:50:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.585+0000 7f8211f2e640 1 -- 172.21.15.45:0/2238712635 shutdown_connections 2023-11-28T20:50:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.585+0000 7f8211f2e640 1 --2- 172.21.15.45:0/2238712635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c0b6380 0x7f820c0b6810 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.585+0000 7f8211f2e640 1 --2- 172.21.15.45:0/2238712635 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0cde00 0x7f820c0ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.585+0000 7f8211f2e640 1 -- 172.21.15.45:0/2238712635 >> 172.21.15.45:0/2238712635 conn(0x7f820c070f40 msgr2=0x7f820c071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.585+0000 7f8211f2e640 1 -- 172.21.15.45:0/2238712635 shutdown_connections 2023-11-28T20:50:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.585+0000 7f8211f2e640 1 -- 172.21.15.45:0/2238712635 wait complete. 2023-11-28T20:50:25.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.587+0000 7f8211f2e640 1 Processor -- start 2023-11-28T20:50:25.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.587+0000 7f8211f2e640 1 -- start start 2023-11-28T20:50:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.588+0000 7f8211f2e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0b6380 0x7f820c14fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.588+0000 7f8211f2e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f820c0cde00 0x7f820c14bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.588+0000 7f8211f2e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c15d3b0 0x7f820c14dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.588+0000 7f8211f2e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c14c730 con 0x7f820c15d3b0 2023-11-28T20:50:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.588+0000 7f8211f2e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c14ca10 con 0x7f820c0cde00 2023-11-28T20:50:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.588+0000 7f8211f2e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c14ccf0 con 0x7f820c0b6380 2023-11-28T20:50:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f821172d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c15d3b0 0x7f820c14dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f821172d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c15d3b0 0x7f820c14dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53418/0 (socket says 172.21.15.45:53418) 2023-11-28T20:50:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f821172d640 1 -- 172.21.15.45:0/676916053 learned_addr learned my addr 172.21.15.45:0/676916053 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f8210f2c640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0b6380 0x7f820c14fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f820bfff640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f820c0cde00 0x7f820c14bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f821172d640 1 -- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0b6380 msgr2=0x7f820c14fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f821172d640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0b6380 0x7f820c14fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f821172d640 1 -- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f820c0cde00 msgr2=0x7f820c14bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f821172d640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f820c0cde00 0x7f820c14bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f821172d640 1 -- 172.21.15.45:0/676916053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8204009b30 con 0x7f820c15d3b0 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.589+0000 7f820bfff640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f820c0cde00 0x7f820c14bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.590+0000 7f821172d640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c15d3b0 0x7f820c14dab0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8204009e50 tx=0x7f8204004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.590+0000 7f82097fa640 1 -- 172.21.15.45:0/676916053 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204009510 con 0x7f820c15d3b0 2023-11-28T20:50:25.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.590+0000 7f82097fa640 1 -- 172.21.15.45:0/676916053 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820403e450 con 0x7f820c15d3b0 2023-11-28T20:50:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.590+0000 7f82097fa640 1 -- 172.21.15.45:0/676916053 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820403e800 con 0x7f820c15d3b0 2023-11-28T20:50:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.590+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c14cfd0 con 0x7f820c15d3b0 2023-11-28T20:50:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.590+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c14d590 con 0x7f820c15d3b0 2023-11-28T20:50:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.592+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c0cd460 con 0x7f820c15d3b0 2023-11-28T20:50:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.593+0000 7f82097fa640 1 -- 172.21.15.45:0/676916053 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f820403e960 con 0x7f820c15d3b0 2023-11-28T20:50:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.594+0000 7f82097fa640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f81f006af60 0x7f81f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.594+0000 7f82097fa640 1 -- 172.21.15.45:0/676916053 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f82040ac5a0 con 0x7f820c15d3b0 2023-11-28T20:50:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.595+0000 7f8210f2c640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f81f006af60 0x7f81f006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.596+0000 7f8210f2c640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f81f006af60 0x7f81f006d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8200000f60 tx=0x7f8200009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:25.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.600+0000 7f82097fa640 1 -- 172.21.15.45:0/676916053 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82040774b0 con 0x7f820c15d3b0 2023-11-28T20:50:25.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:25.793+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f820c14d780 con 0x7f820c15d3b0 2023-11-28T20:50:26.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.329+0000 7f82097fa640 1 -- 172.21.15.45:0/676916053 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v86) v1 ==== 170+0+0 (secure 0 0 0) 0x7f820407aad0 con 0x7f820c15d3b0 2023-11-28T20:50:26.331 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T20:50:26.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f81f006af60 msgr2=0x7f81f006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f81f006af60 0x7f81f006d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8200000f60 tx=0x7f8200009040 comp rx=0 tx=0).stop 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c15d3b0 msgr2=0x7f820c14dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c15d3b0 0x7f820c14dab0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8204009e50 tx=0x7f8204004a40 comp rx=0 tx=0).stop 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 shutdown_connections 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f81f006af60 0x7f81f006d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c15d3b0 0x7f820c14dab0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f820c0cde00 0x7f820c14bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 --2- 172.21.15.45:0/676916053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f820c0b6380 0x7f820c14fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.332+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 >> 172.21.15.45:0/676916053 conn(0x7f820c070f40 msgr2=0x7f820c0b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.333+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 shutdown_connections 2023-11-28T20:50:26.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.333+0000 7f8211f2e640 1 -- 172.21.15.45:0/676916053 wait complete. 2023-11-28T20:50:26.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T20:50:26.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.559+0000 7f7377291640 1 Processor -- start 2023-11-28T20:50:26.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.559+0000 7f7377291640 1 -- start start 2023-11-28T20:50:26.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.561+0000 7f7377291640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f737015b270 0x7f737006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:26.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.561+0000 7f7377291640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737006d410 con 0x7f73700b3040 2023-11-28T20:50:26.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.561+0000 7f7377291640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737006d6f0 con 0x7f73700b3420 2023-11-28T20:50:26.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.561+0000 7f7377291640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73700ceaa0 con 0x7f737015b270 2023-11-28T20:50:26.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.561+0000 7f737628f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f73700b3040 0x7f737015f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42790/0 (socket says 172.21.15.45:42790) 2023-11-28T20:50:26.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.561+0000 7f737628f640 1 -- 172.21.15.45:0/2251893420 learned_addr learned my addr 172.21.15.45:0/2251893420 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528343154 0 0) 0x7f737006d410 con 0x7f73700b3040 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f735c003520 con 0x7f73700b3040 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f7375a8e640 1 --2- 172.21.15.45:0/2251893420 >> v2:172.21.15.190:3300/0 conn(0x7f737015b270 0x7f737006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2018176405 0 0) 0x7f735c003520 con 0x7f73700b3040 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 >> v2:172.21.15.190:3300/0 conn(0x7f737015b270 msgr2=0x7f737006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f737528d640 1 --2- 172.21.15.45:0/2251893420 >> v2:172.21.15.190:3300/0 conn(0x7f737015b270 0x7f737006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 >> v1:172.21.15.190:6789/0 conn(0x7f73700b3420 legacy=0x7f73700ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737006d9d0 con 0x7f73700b3040 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f736c001670 con 0x7f73700b3040 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.562+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 >> v1:172.21.15.45:6789/0 conn(0x7f73700b3040 legacy=0x7f737015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:26.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f737528d640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c0039c0 0x7f735c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f737528d640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c0047c0 0x7f735c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f737528d640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f735c005380 0x7f735c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c00c230 con 0x7f735c005380 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c00c3a0 con 0x7f735c0047c0 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c00c560 con 0x7f735c0039c0 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f7375a8e640 1 --2- 172.21.15.45:0/2251893420 >> v2:172.21.15.190:3300/0 conn(0x7f737015b270 0x7f737006cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f7376a90640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f735c005380 0x7f735c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f737628f640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c0039c0 0x7f735c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f7375a8e640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c0047c0 0x7f735c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f7376a90640 1 -- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c0039c0 msgr2=0x7f735c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f7376a90640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c0039c0 0x7f735c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f7376a90640 1 -- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c0047c0 msgr2=0x7f735c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:26.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f7376a90640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c0047c0 0x7f735c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f7376a90640 1 -- 172.21.15.45:0/2251893420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c003520 con 0x7f735c005380 2023-11-28T20:50:26.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f7375a8e640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c0047c0 0x7f735c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:26.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.563+0000 7f737628f640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c0039c0 0x7f735c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.565+0000 7f7376a90640 1 -- 172.21.15.45:0/2251893420 reap_dead start 2023-11-28T20:50:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.565+0000 7f7376a90640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f735c005380 0x7f735c00b980 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7368001e50 tx=0x7f735c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c774c903eaa2843d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.565+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368003b70 con 0x7f735c005380 2023-11-28T20:50:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.565+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7368003d10 con 0x7f735c005380 2023-11-28T20:50:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.565+0000 7f737528d640 1 -- 172.21.15.45:0/2251893420 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368005310 con 0x7f735c005380 2023-11-28T20:50:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.566+0000 7f7377291640 1 -- 172.21.15.45:0/2251893420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f735c005380 msgr2=0x7f735c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.566+0000 7f7377291640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f735c005380 0x7f735c00b980 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7368001e50 tx=0x7f735c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.566+0000 7f7377291640 1 -- 172.21.15.45:0/2251893420 shutdown_connections 2023-11-28T20:50:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.566+0000 7f7377291640 1 --2- 172.21.15.45:0/2251893420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f735c005380 0x7f735c00b980 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.566+0000 7f7377291640 1 -- 172.21.15.45:0/2251893420 >> 172.21.15.45:0/2251893420 conn(0x7f73700b1050 msgr2=0x7f73700cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:26.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.566+0000 7f7377291640 1 -- 172.21.15.45:0/2251893420 shutdown_connections 2023-11-28T20:50:26.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.566+0000 7f7377291640 1 -- 172.21.15.45:0/2251893420 wait complete. 2023-11-28T20:50:26.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.568+0000 7f7377291640 1 Processor -- start 2023-11-28T20:50:26.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.568+0000 7f7377291640 1 -- start start 2023-11-28T20:50:26.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7377291640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c005380 0x7f7370144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:26.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7377291640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7370147e00 0x7f73701513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:26.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7377291640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7370145250 0x7f73701456d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:26.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7377291640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7370152790 con 0x7f7370147e00 2023-11-28T20:50:26.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7377291640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7370144dd0 con 0x7f7370145250 2023-11-28T20:50:26.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7377291640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73701518e0 con 0x7f735c005380 2023-11-28T20:50:26.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7375a8e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7370147e00 0x7f73701513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:26.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7375a8e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7370147e00 0x7f73701513a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53432/0 (socket says 172.21.15.45:53432) 2023-11-28T20:50:26.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7375a8e640 1 -- 172.21.15.45:0/4034559203 learned_addr learned my addr 172.21.15.45:0/4034559203 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:26.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f7376a90640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7370145250 0x7f73701456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:26.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.569+0000 7f737628f640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c005380 0x7f7370144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:26.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7375a8e640 1 -- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c005380 msgr2=0x7f7370144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:26.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7375a8e640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c005380 0x7f7370144890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7375a8e640 1 -- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7370145250 msgr2=0x7f73701456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:26.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7375a8e640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7370145250 0x7f73701456d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:26.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7375a8e640 1 -- 172.21.15.45:0/4034559203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7370151150 con 0x7f7370147e00 2023-11-28T20:50:26.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7375a8e640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7370147e00 0x7f73701513a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f7360006c00 tx=0x7f736000f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:26.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7366ffd640 1 -- 172.21.15.45:0/4034559203 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360014890 con 0x7f7370147e00 2023-11-28T20:50:26.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7366ffd640 1 -- 172.21.15.45:0/4034559203 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736000c030 con 0x7f7370147e00 2023-11-28T20:50:26.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7366ffd640 1 -- 172.21.15.45:0/4034559203 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360014bf0 con 0x7f7370147e00 2023-11-28T20:50:26.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.570+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7370151b10 con 0x7f7370147e00 2023-11-28T20:50:26.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.571+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73701520d0 con 0x7f7370147e00 2023-11-28T20:50:26.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.572+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73701528d0 con 0x7f7370147e00 2023-11-28T20:50:26.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.574+0000 7f7366ffd640 1 -- 172.21.15.45:0/4034559203 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7360014d50 con 0x7f7370147e00 2023-11-28T20:50:26.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.575+0000 7f7366ffd640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f735406af60 0x7f735406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:26.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.575+0000 7f7366ffd640 1 -- 172.21.15.45:0/4034559203 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f7360089ff0 con 0x7f7370147e00 2023-11-28T20:50:26.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.576+0000 7f737628f640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f735406af60 0x7f735406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:26.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.577+0000 7f737628f640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f735406af60 0x7f735406d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f736c001eb0 tx=0x7f736c009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:26.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.579+0000 7f7366ffd640 1 -- 172.21.15.45:0/4034559203 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7360054e10 con 0x7f7370147e00 2023-11-28T20:50:26.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:26.810+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f73701528d0 con 0x7f7370147e00 2023-11-28T20:50:27.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.356+0000 7f7366ffd640 1 -- 172.21.15.45:0/4034559203 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v32) v1 ==== 173+0+0 (secure 0 0 0) 0x7f73701528d0 con 0x7f7370147e00 2023-11-28T20:50:27.358 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 9 and data pool 10 2023-11-28T20:50:27.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.362+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f735406af60 msgr2=0x7f735406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:27.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.362+0000 7f7377291640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f735406af60 0x7f735406d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f736c001eb0 tx=0x7f736c009040 comp rx=0 tx=0).stop 2023-11-28T20:50:27.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.362+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7370147e00 msgr2=0x7f73701513a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:27.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.362+0000 7f7377291640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7370147e00 0x7f73701513a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f7360006c00 tx=0x7f736000f9e0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.367+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 shutdown_connections 2023-11-28T20:50:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.367+0000 7f7377291640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f735406af60 0x7f735406d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.367+0000 7f7377291640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7370145250 0x7f73701456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.367+0000 7f7377291640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7370147e00 0x7f73701513a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.367+0000 7f7377291640 1 --2- 172.21.15.45:0/4034559203 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f735c005380 0x7f7370144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.367+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 >> 172.21.15.45:0/4034559203 conn(0x7f73700b1050 msgr2=0x7f73700af550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:27.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.370+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 shutdown_connections 2023-11-28T20:50:27.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.370+0000 7f7377291640 1 -- 172.21.15.45:0/4034559203 wait complete. 2023-11-28T20:50:27.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.591+0000 7fbc4aa07640 1 Processor -- start 2023-11-28T20:50:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.591+0000 7fbc4aa07640 1 -- start start 2023-11-28T20:50:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.593+0000 7fbc4aa07640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbc440b35b0 0x7fbc4415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.593+0000 7fbc4aa07640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4415b850 0x7fbc440b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.593+0000 7fbc4aa07640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4415bda0 con 0x7fbc440b1fa0 2023-11-28T20:50:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.593+0000 7fbc4aa07640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4415ee40 con 0x7fbc440b35b0 2023-11-28T20:50:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.593+0000 7fbc4aa07640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4415c080 con 0x7fbc4415b850 2023-11-28T20:50:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.593+0000 7fbc48fa4640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbc440b1fa0 0x7fbc4415f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42804/0 (socket says 172.21.15.45:42804) 2023-11-28T20:50:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.593+0000 7fbc48fa4640 1 -- 172.21.15.45:0/423329925 learned_addr learned my addr 172.21.15.45:0/423329925 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc3bfff640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4415b850 0x7fbc440b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053910701 0 0) 0x7fbc4415bda0 con 0x7fbc440b1fa0 2023-11-28T20:50:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc43fff640 1 --2- 172.21.15.45:0/423329925 >> v2:172.21.15.190:3300/0 conn(0x7fbc440b35b0 0x7fbc4415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc30003520 con 0x7fbc440b1fa0 2023-11-28T20:50:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 422768241 0 0) 0x7fbc30003520 con 0x7fbc440b1fa0 2023-11-28T20:50:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4415b850 msgr2=0x7fbc440b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4415b850 0x7fbc440b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 >> v2:172.21.15.190:3300/0 conn(0x7fbc440b35b0 msgr2=0x7fbc4415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 --2- 172.21.15.45:0/423329925 >> v2:172.21.15.190:3300/0 conn(0x7fbc440b35b0 0x7fbc4415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4415c360 con 0x7fbc440b1fa0 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fbc3c0019f0 con 0x7fbc440b1fa0 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc3bfff640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4415b850 0x7fbc440b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.594+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 >> v1:172.21.15.45:6789/0 conn(0x7fbc440b1fa0 legacy=0x7fbc4415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc437fe640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc30003970 0x7fbc30003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc437fe640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc300047c0 0x7fbc30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc437fe640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4415b850 0x7fbc30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4415ee40 con 0x7fbc30003970 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4415c080 con 0x7fbc300047c0 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3000ba80 con 0x7fbc4415b850 2023-11-28T20:50:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc43fff640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc30003970 0x7fbc30003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc48fa4640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4415b850 0x7fbc30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc3bfff640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc300047c0 0x7fbc30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc43fff640 1 -- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4415b850 msgr2=0x7fbc30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc43fff640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4415b850 0x7fbc30005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc43fff640 1 -- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc300047c0 msgr2=0x7fbc30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc43fff640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc300047c0 0x7fbc30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc43fff640 1 -- 172.21.15.45:0/423329925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30003520 con 0x7fbc30003970 2023-11-28T20:50:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.595+0000 7fbc3bfff640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc300047c0 0x7fbc30004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:27.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.597+0000 7fbc43fff640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc30003970 0x7fbc30003d90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400ba30 tx=0x7fbc300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10f46245f526b65 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.597+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34032620 con 0x7fbc30003970 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.597+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc34032780 con 0x7fbc30003970 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.597+0000 7fbc437fe640 1 -- 172.21.15.45:0/423329925 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34032b10 con 0x7fbc30003970 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.597+0000 7fbc4aa07640 1 -- 172.21.15.45:0/423329925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc30003970 msgr2=0x7fbc30003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.597+0000 7fbc4aa07640 1 --2- 172.21.15.45:0/423329925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc30003970 0x7fbc30003d90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400ba30 tx=0x7fbc300057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.597+0000 7fbc48fa4640 1 -- 172.21.15.45:0/423329925 reap_dead start 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.597+0000 7fbc4aa07640 1 -- 172.21.15.45:0/423329925 shutdown_connections 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.597+0000 7fbc4aa07640 1 -- 172.21.15.45:0/423329925 >> 172.21.15.45:0/423329925 conn(0x7fbc4406a500 msgr2=0x7fbc4406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.598+0000 7fbc4aa07640 1 -- 172.21.15.45:0/423329925 shutdown_connections 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.598+0000 7fbc4aa07640 1 -- 172.21.15.45:0/423329925 wait complete. 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.599+0000 7fbc4aa07640 1 Processor -- start 2023-11-28T20:50:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.600+0000 7fbc4aa07640 1 -- start start 2023-11-28T20:50:27.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc4aa07640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc30003970 0x7fbc440ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:27.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc4aa07640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc44070330 0x7fbc4415a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc4aa07640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc44071bb0 0x7fbc4413b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc4aa07640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc44071720 con 0x7fbc44071bb0 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc4aa07640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc44070b10 con 0x7fbc30003970 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc4aa07640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc44070df0 con 0x7fbc44070330 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc48fa4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc44071bb0 0x7fbc4413b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc48fa4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc44071bb0 0x7fbc4413b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53458/0 (socket says 172.21.15.45:53458) 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc48fa4640 1 -- 172.21.15.45:0/4001220759 learned_addr learned my addr 172.21.15.45:0/4001220759 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc43fff640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc30003970 0x7fbc440ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc3bfff640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc44070330 0x7fbc4415a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc48fa4640 1 -- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc44070330 msgr2=0x7fbc4415a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc48fa4640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc44070330 0x7fbc4415a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc48fa4640 1 -- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc30003970 msgr2=0x7fbc440ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:27.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc48fa4640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc30003970 0x7fbc440ae120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc48fa4640 1 -- 172.21.15.45:0/4001220759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440710d0 con 0x7fbc44071bb0 2023-11-28T20:50:27.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.601+0000 7fbc3bfff640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc44070330 0x7fbc4415a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:27.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.602+0000 7fbc48fa4640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc44071bb0 0x7fbc4413b400 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbc440cd380 tx=0x7fbc3c007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:27.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.602+0000 7fbc417fa640 1 -- 172.21.15.45:0/4001220759 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c00ea80 con 0x7fbc44071bb0 2023-11-28T20:50:27.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.602+0000 7fbc417fa640 1 -- 172.21.15.45:0/4001220759 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c00ec20 con 0x7fbc44071bb0 2023-11-28T20:50:27.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.602+0000 7fbc417fa640 1 -- 172.21.15.45:0/4001220759 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c013420 con 0x7fbc44071bb0 2023-11-28T20:50:27.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.602+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc440713b0 con 0x7fbc44071bb0 2023-11-28T20:50:27.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.602+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4415a8a0 con 0x7fbc44071bb0 2023-11-28T20:50:27.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.603+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc44071890 con 0x7fbc44071bb0 2023-11-28T20:50:27.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.604+0000 7fbc417fa640 1 -- 172.21.15.45:0/4001220759 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbc3c013580 con 0x7fbc44071bb0 2023-11-28T20:50:27.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.605+0000 7fbc417fa640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbc2406af90 0x7fbc2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:27.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.605+0000 7fbc417fa640 1 -- 172.21.15.45:0/4001220759 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbc3c089ca0 con 0x7fbc44071bb0 2023-11-28T20:50:27.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.606+0000 7fbc43fff640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbc2406af90 0x7fbc2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:27.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.607+0000 7fbc43fff640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbc2406af90 0x7fbc2406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400ba00 tx=0x7fbc3403d040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:27.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.610+0000 7fbc417fa640 1 -- 172.21.15.45:0/4001220759 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc3c054a60 con 0x7fbc44071bb0 2023-11-28T20:50:27.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.802+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc440733d0 con 0x7fbc44071bb0 2023-11-28T20:50:27.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.804+0000 7fbc417fa640 1 -- 172.21.15.45:0/4001220759 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16717 (secure 0 0 0) 0x7fbc3c058080 con 0x7fbc44071bb0 2023-11-28T20:50:27.807 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:27.807 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:26.812395+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:50:24.656511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-28T20:50:25.847745+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:27.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.806+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbc2406af90 msgr2=0x7fbc2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:27.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.806+0000 7fbc4aa07640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbc2406af90 0x7fbc2406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400ba00 tx=0x7fbc3403d040 comp rx=0 tx=0).stop 2023-11-28T20:50:27.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc44071bb0 msgr2=0x7fbc4413b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:27.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc44071bb0 0x7fbc4413b400 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbc440cd380 tx=0x7fbc3c007b10 comp rx=0 tx=0).stop 2023-11-28T20:50:27.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 shutdown_connections 2023-11-28T20:50:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbc2406af90 0x7fbc2406d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc44071bb0 0x7fbc4413b400 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc44070330 0x7fbc4415a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 --2- 172.21.15.45:0/4001220759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc30003970 0x7fbc440ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 >> 172.21.15.45:0/4001220759 conn(0x7fbc4406a500 msgr2=0x7fbc4415d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 shutdown_connections 2023-11-28T20:50:27.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:27.807+0000 7fbc4aa07640 1 -- 172.21.15.45:0/4001220759 wait complete. 2023-11-28T20:50:27.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:28.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.050+0000 7f23c9417640 1 Processor -- start 2023-11-28T20:50:28.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.051+0000 7f23c9417640 1 -- start start 2023-11-28T20:50:28.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.052+0000 7f23c9417640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c415b850 0x7f23c40b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.052+0000 7f23c9417640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f23c40b1fa0 0x7f23c40b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.052+0000 7f23c9417640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c415bda0 con 0x7f23c40b35b0 2023-11-28T20:50:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.052+0000 7f23c9417640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c415ee40 con 0x7f23c40b1fa0 2023-11-28T20:50:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.052+0000 7f23c9417640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c415c080 con 0x7f23c415b850 2023-11-28T20:50:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.053+0000 7f23c27fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c415b850 0x7f23c40b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.053+0000 7f23c27fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c415b850 0x7f23c40b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:50418/0 (socket says 172.21.15.45:50418) 2023-11-28T20:50:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.053+0000 7f23c27fc640 1 -- 172.21.15.45:0/2523976328 learned_addr learned my addr 172.21.15.45:0/2523976328 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.053+0000 7f23c37fe640 1 --2- 172.21.15.45:0/2523976328 >> v2:172.21.15.190:3300/0 conn(0x7f23c40b1fa0 0x7f23c40b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.054+0000 7f23c27fc640 1 -- 172.21.15.45:0/2523976328 >> v2:172.21.15.190:3300/0 conn(0x7f23c40b1fa0 msgr2=0x7f23c40b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.054+0000 7f23c27fc640 1 --2- 172.21.15.45:0/2523976328 >> v2:172.21.15.190:3300/0 conn(0x7f23c40b1fa0 0x7f23c40b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.054+0000 7f23c27fc640 1 -- 172.21.15.45:0/2523976328 >> v1:172.21.15.190:6789/0 conn(0x7f23c40b35b0 legacy=0x7f23c415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.054+0000 7f23c27fc640 1 -- 172.21.15.45:0/2523976328 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c415c360 con 0x7f23c415b850 2023-11-28T20:50:28.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.055+0000 7f23c27fc640 1 --2- 172.21.15.45:0/2523976328 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c415b850 0x7f23c40b0c30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0098a0 tx=0x7f23ac02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c370d111b99bbc85 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.056+0000 7f23c1ffb640 1 -- 172.21.15.45:0/2523976328 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac030840 con 0x7f23c415b850 2023-11-28T20:50:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.056+0000 7f23c1ffb640 1 -- 172.21.15.45:0/2523976328 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac0309a0 con 0x7f23c415b850 2023-11-28T20:50:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.056+0000 7f23c1ffb640 1 -- 172.21.15.45:0/2523976328 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac030d30 con 0x7f23c415b850 2023-11-28T20:50:28.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.057+0000 7f23c9417640 1 -- 172.21.15.45:0/2523976328 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c415b850 msgr2=0x7f23c40b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.057+0000 7f23c9417640 1 --2- 172.21.15.45:0/2523976328 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c415b850 0x7f23c40b0c30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0098a0 tx=0x7f23ac02b980 comp rx=0 tx=0).stop 2023-11-28T20:50:28.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.057+0000 7f23c9417640 1 -- 172.21.15.45:0/2523976328 shutdown_connections 2023-11-28T20:50:28.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.057+0000 7f23c9417640 1 --2- 172.21.15.45:0/2523976328 >> v2:172.21.15.190:3300/0 conn(0x7f23c40b1fa0 0x7f23c40b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.057+0000 7f23c9417640 1 --2- 172.21.15.45:0/2523976328 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c415b850 0x7f23c40b0c30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.057+0000 7f23c9417640 1 -- 172.21.15.45:0/2523976328 >> 172.21.15.45:0/2523976328 conn(0x7f23c406a500 msgr2=0x7f23c406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:28.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.057+0000 7f23c9417640 1 -- 172.21.15.45:0/2523976328 shutdown_connections 2023-11-28T20:50:28.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.057+0000 7f23c9417640 1 -- 172.21.15.45:0/2523976328 wait complete. 2023-11-28T20:50:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.059+0000 7f23c9417640 1 Processor -- start 2023-11-28T20:50:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.059+0000 7f23c9417640 1 -- start start 2023-11-28T20:50:28.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.060+0000 7f23c9417640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c40b1fa0 0x7f23c4071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.060+0000 7f23c9417640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23c40b35b0 0x7f23c40aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.060+0000 7f23c9417640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23c415b850 0x7f23c40aaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.060+0000 7f23c9417640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c40a9880 con 0x7f23c40b35b0 2023-11-28T20:50:28.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.060+0000 7f23c9417640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c4159980 con 0x7f23c415b850 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.060+0000 7f23c9417640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c415a500 con 0x7f23c40b1fa0 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.060+0000 7f23c27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23c40b35b0 0x7f23c40aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.060+0000 7f23c27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23c40b35b0 0x7f23c40aa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53462/0 (socket says 172.21.15.45:53462) 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.060+0000 7f23c27fc640 1 -- 172.21.15.45:0/423047919 learned_addr learned my addr 172.21.15.45:0/423047919 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.061+0000 7f23c2ffd640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c40b1fa0 0x7f23c4071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.061+0000 7f23c37fe640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23c415b850 0x7f23c40aaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.061+0000 7f23c27fc640 1 -- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c40b1fa0 msgr2=0x7f23c4071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.061+0000 7f23c27fc640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c40b1fa0 0x7f23c4071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.061+0000 7f23c27fc640 1 -- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23c415b850 msgr2=0x7f23c40aaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.061+0000 7f23c27fc640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23c415b850 0x7f23c40aaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.061+0000 7f23c27fc640 1 -- 172.21.15.45:0/423047919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0094d0 con 0x7f23c40b35b0 2023-11-28T20:50:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.061+0000 7f23c27fc640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23c40b35b0 0x7f23c40aa5b0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f23ac02bf90 tx=0x7f23ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:28.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.062+0000 7f23bb7fe640 1 -- 172.21.15.45:0/423047919 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac03e070 con 0x7f23c40b35b0 2023-11-28T20:50:28.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.062+0000 7f23bb7fe640 1 -- 172.21.15.45:0/423047919 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac039a10 con 0x7f23c40b35b0 2023-11-28T20:50:28.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.062+0000 7f23bb7fe640 1 -- 172.21.15.45:0/423047919 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac039da0 con 0x7f23c40b35b0 2023-11-28T20:50:28.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.062+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c4159bd0 con 0x7f23c40b35b0 2023-11-28T20:50:28.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.062+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c415a190 con 0x7f23c40b35b0 2023-11-28T20:50:28.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.063+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c40733d0 con 0x7f23c40b35b0 2023-11-28T20:50:28.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.064+0000 7f23bb7fe640 1 -- 172.21.15.45:0/423047919 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f23ac04b400 con 0x7f23c40b35b0 2023-11-28T20:50:28.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.065+0000 7f23bb7fe640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f23a406af60 0x7f23a406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.065+0000 7f23bb7fe640 1 -- 172.21.15.45:0/423047919 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f23ac0abe30 con 0x7f23c40b35b0 2023-11-28T20:50:28.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.066+0000 7f23c2ffd640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f23a406af60 0x7f23a406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.067+0000 7f23c2ffd640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f23a406af60 0x7f23a406d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23c4071d80 tx=0x7f23b4009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:28.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.070+0000 7f23bb7fe640 1 -- 172.21.15.45:0/423047919 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23ac076c10 con 0x7f23c40b35b0 2023-11-28T20:50:28.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2023-11-28T20:50:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.266+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23c415a640 con 0x7f23c40b35b0 2023-11-28T20:50:28.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.269+0000 7f23bb7fe640 1 -- 172.21.15.45:0/423047919 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f23ac034020 con 0x7f23c40b35b0 2023-11-28T20:50:28.271 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:28.272 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:26.812395+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:50:24.656511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-28T20:50:25.847745+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f23a406af60 msgr2=0x7f23a406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f23a406af60 0x7f23a406d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23c4071d80 tx=0x7f23b4009040 comp rx=0 tx=0).stop 2023-11-28T20:50:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23c40b35b0 msgr2=0x7f23c40aa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23c40b35b0 0x7f23c40aa5b0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f23ac02bf90 tx=0x7f23ac0046c0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 shutdown_connections 2023-11-28T20:50:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f23a406af60 0x7f23a406d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23c415b850 0x7f23c40aaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23c40b35b0 0x7f23c40aa5b0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 --2- 172.21.15.45:0/423047919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23c40b1fa0 0x7f23c4071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 >> 172.21.15.45:0/423047919 conn(0x7f23c406a500 msgr2=0x7f23c4070230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 shutdown_connections 2023-11-28T20:50:28.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.272+0000 7f23c9417640 1 -- 172.21.15.45:0/423047919 wait complete. 2023-11-28T20:50:28.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T20:50:28.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.519+0000 7f0402239640 1 Processor -- start 2023-11-28T20:50:28.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.519+0000 7f0402239640 1 -- start start 2023-11-28T20:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.521+0000 7f0402239640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f03fc0ae480 0x7f03fc0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.521+0000 7f0402239640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03fc0708d0 con 0x7f03fc0adfa0 2023-11-28T20:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.521+0000 7f0402239640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03fc0b3bb0 con 0x7f03fc15d4b0 2023-11-28T20:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.521+0000 7f0402239640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03fc0b3e90 con 0x7f03fc0ae480 2023-11-28T20:50:28.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.521+0000 7f0400a36640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f03fc0adfa0 0x7f03fc0abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42810/0 (socket says 172.21.15.45:42810) 2023-11-28T20:50:28.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.521+0000 7f0400a36640 1 -- 172.21.15.45:0/2523404062 learned_addr learned my addr 172.21.15.45:0/2523404062 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:28.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.522+0000 7f0401a38640 1 --2- 172.21.15.45:0/2523404062 >> v2:172.21.15.190:3300/0 conn(0x7f03fc0ae480 0x7f03fc0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.522+0000 7f03f3fff640 1 -- 172.21.15.45:0/2523404062 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340496503 0 0) 0x7f03fc0708d0 con 0x7f03fc0adfa0 2023-11-28T20:50:28.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.522+0000 7f03f3fff640 1 -- 172.21.15.45:0/2523404062 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e8003520 con 0x7f03fc0adfa0 2023-11-28T20:50:28.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.522+0000 7f0401a38640 1 -- 172.21.15.45:0/2523404062 >> v1:172.21.15.190:6789/0 conn(0x7f03fc15d4b0 legacy=0x7f03fc070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.522+0000 7f0401a38640 1 -- 172.21.15.45:0/2523404062 >> v1:172.21.15.45:6789/0 conn(0x7f03fc0adfa0 legacy=0x7f03fc0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.522+0000 7f0401a38640 1 -- 172.21.15.45:0/2523404062 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc070280 con 0x7f03fc0ae480 2023-11-28T20:50:28.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.524+0000 7f0401a38640 1 --2- 172.21.15.45:0/2523404062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc0ae480 0x7f03fc0717e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009d90 tx=0x7f03f402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a76a93e86e8173a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.524+0000 7f03f3fff640 1 -- 172.21.15.45:0/2523404062 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f402cc60 con 0x7f03fc0ae480 2023-11-28T20:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.524+0000 7f03f3fff640 1 -- 172.21.15.45:0/2523404062 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f402cdc0 con 0x7f03fc0ae480 2023-11-28T20:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.524+0000 7f03f3fff640 1 -- 172.21.15.45:0/2523404062 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f4034660 con 0x7f03fc0ae480 2023-11-28T20:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.525+0000 7f0402239640 1 -- 172.21.15.45:0/2523404062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc0ae480 msgr2=0x7f03fc0717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.525+0000 7f0402239640 1 --2- 172.21.15.45:0/2523404062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc0ae480 0x7f03fc0717e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009d90 tx=0x7f03f402beb0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.525+0000 7f0402239640 1 -- 172.21.15.45:0/2523404062 shutdown_connections 2023-11-28T20:50:28.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.525+0000 7f0402239640 1 --2- 172.21.15.45:0/2523404062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc0ae480 0x7f03fc0717e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.525+0000 7f0402239640 1 -- 172.21.15.45:0/2523404062 >> 172.21.15.45:0/2523404062 conn(0x7f03fc0b1050 msgr2=0x7f03fc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:28.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.525+0000 7f0402239640 1 -- 172.21.15.45:0/2523404062 shutdown_connections 2023-11-28T20:50:28.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.525+0000 7f0402239640 1 -- 172.21.15.45:0/2523404062 wait complete. 2023-11-28T20:50:28.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.527+0000 7f0402239640 1 Processor -- start 2023-11-28T20:50:28.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.527+0000 7f0402239640 1 -- start start 2023-11-28T20:50:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0402239640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03fc0adfa0 0x7f03fc143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0402239640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03fc0ae480 0x7f03fc0af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0402239640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc15d4b0 0x7f03fc0ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0402239640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc0af800 con 0x7f03fc0adfa0 2023-11-28T20:50:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0402239640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc09b240 con 0x7f03fc15d4b0 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0402239640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc09b520 con 0x7f03fc0ae480 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0401237640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03fc0adfa0 0x7f03fc143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0401237640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03fc0adfa0 0x7f03fc143350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53468/0 (socket says 172.21.15.45:53468) 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0401237640 1 -- 172.21.15.45:0/587002447 learned_addr learned my addr 172.21.15.45:0/587002447 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0401a38640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc15d4b0 0x7f03fc0ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0400a36640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03fc0ae480 0x7f03fc0af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.528+0000 7f0401237640 1 -- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03fc0ae480 msgr2=0x7f03fc0af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f0401237640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03fc0ae480 0x7f03fc0af2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f0401237640 1 -- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc15d4b0 msgr2=0x7f03fc0ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f0401237640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc15d4b0 0x7f03fc0ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f0401237640 1 -- 172.21.15.45:0/587002447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f4009a70 con 0x7f03fc0adfa0 2023-11-28T20:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f0401a38640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc15d4b0 0x7f03fc0ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f0401237640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03fc0adfa0 0x7f03fc143350 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f03f800c400 tx=0x7f03f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f03f1ffb640 1 -- 172.21.15.45:0/587002447 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f80052d0 con 0x7f03fc0adfa0 2023-11-28T20:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f03f1ffb640 1 -- 172.21.15.45:0/587002447 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f8005450 con 0x7f03fc0adfa0 2023-11-28T20:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f03f1ffb640 1 -- 172.21.15.45:0/587002447 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8007260 con 0x7f03fc0adfa0 2023-11-28T20:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03fc09b800 con 0x7f03fc0adfa0 2023-11-28T20:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.529+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03fc09ae60 con 0x7f03fc0adfa0 2023-11-28T20:50:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.530+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03fc09bd20 con 0x7f03fc0adfa0 2023-11-28T20:50:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.531+0000 7f03f1ffb640 1 -- 172.21.15.45:0/587002447 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f03f80073c0 con 0x7f03fc0adfa0 2023-11-28T20:50:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.532+0000 7f03f1ffb640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f03e006af60 0x7f03e006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.532+0000 7f03f1ffb640 1 -- 172.21.15.45:0/587002447 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f03f8088d70 con 0x7f03fc0adfa0 2023-11-28T20:50:28.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.533+0000 7f0400a36640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f03e006af60 0x7f03e006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.534+0000 7f0400a36640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f03e006af60 0x7f03e006d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03fc157410 tx=0x7f03ec009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.537+0000 7f03f1ffb640 1 -- 172.21.15.45:0/587002447 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03fc09bd20 con 0x7f03fc0adfa0 2023-11-28T20:50:28.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:28.769+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f03fc09aff0 con 0x7f03fc0adfa0 2023-11-28T20:50:29.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.392+0000 7f03f1ffb640 1 -- 172.21.15.45:0/587002447 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) v1 ==== 118+0+0 (secure 0 0 0) 0x7f03f80575c0 con 0x7f03fc0adfa0 2023-11-28T20:50:29.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f03e006af60 msgr2=0x7f03e006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:29.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f03e006af60 0x7f03e006d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03fc157410 tx=0x7f03ec009040 comp rx=0 tx=0).stop 2023-11-28T20:50:29.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03fc0adfa0 msgr2=0x7f03fc143350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:29.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03fc0adfa0 0x7f03fc143350 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f03f800c400 tx=0x7f03f800c8c0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 shutdown_connections 2023-11-28T20:50:29.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f03e006af60 0x7f03e006d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03fc15d4b0 0x7f03fc0ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f03fc0ae480 0x7f03fc0af2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 --2- 172.21.15.45:0/587002447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03fc0adfa0 0x7f03fc143350 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 >> 172.21.15.45:0/587002447 conn(0x7f03fc0b1050 msgr2=0x7f03fc15bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:29.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 shutdown_connections 2023-11-28T20:50:29.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.396+0000 7f0402239640 1 -- 172.21.15.45:0/587002447 wait complete. 2023-11-28T20:50:29.418 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T20:50:29.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T20:50:29.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.647+0000 7fe1eafa0640 1 Processor -- start 2023-11-28T20:50:29.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.648+0000 7fe1eafa0640 1 -- start start 2023-11-28T20:50:29.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.649+0000 7fe1eafa0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40b2290 0x7fe1e40d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:29.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.649+0000 7fe1eafa0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe1e40b1d40 0x7fe1e40d22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:29.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.649+0000 7fe1eafa0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e40b90a0 con 0x7fe1e40d0040 2023-11-28T20:50:29.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.649+0000 7fe1eafa0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e40d1920 con 0x7fe1e40b1d40 2023-11-28T20:50:29.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.649+0000 7fe1eafa0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40b8ae0 con 0x7fe1e40b2290 2023-11-28T20:50:29.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.650+0000 7fe1e979d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40b2290 0x7fe1e40d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:29.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.650+0000 7fe1ea79f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe1e40b1d40 0x7fe1e40d22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:29.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.650+0000 7fe1e9f9e640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe1e40d0040 0x7fe1e40d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:52042/0 (socket says 172.21.15.45:52042) 2023-11-28T20:50:29.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.650+0000 7fe1e9f9e640 1 -- 172.21.15.45:0/1082987662 learned_addr learned my addr 172.21.15.45:0/1082987662 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:29.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.651+0000 7fe1e979d640 1 -- 172.21.15.45:0/1082987662 >> v2:172.21.15.190:3300/0 conn(0x7fe1e40b1d40 msgr2=0x7fe1e40d22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:29.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.651+0000 7fe1e979d640 1 --2- 172.21.15.45:0/1082987662 >> v2:172.21.15.190:3300/0 conn(0x7fe1e40b1d40 0x7fe1e40d22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.651+0000 7fe1e979d640 1 -- 172.21.15.45:0/1082987662 >> v1:172.21.15.190:6789/0 conn(0x7fe1e40d0040 legacy=0x7fe1e40d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:29.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.651+0000 7fe1e979d640 1 -- 172.21.15.45:0/1082987662 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e40b8dc0 con 0x7fe1e40b2290 2023-11-28T20:50:29.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.652+0000 7fe1e979d640 1 --2- 172.21.15.45:0/1082987662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40b2290 0x7fe1e40d1db0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40097a0 tx=0x7fe1d402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cd15e4f57c946e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:29.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.653+0000 7fe1e8f9c640 1 -- 172.21.15.45:0/1082987662 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4030610 con 0x7fe1e40b2290 2023-11-28T20:50:29.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.653+0000 7fe1e8f9c640 1 -- 172.21.15.45:0/1082987662 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d4030770 con 0x7fe1e40b2290 2023-11-28T20:50:29.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.653+0000 7fe1e8f9c640 1 -- 172.21.15.45:0/1082987662 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4030b00 con 0x7fe1e40b2290 2023-11-28T20:50:29.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.654+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1082987662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40b2290 msgr2=0x7fe1e40d1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:29.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.654+0000 7fe1eafa0640 1 --2- 172.21.15.45:0/1082987662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40b2290 0x7fe1e40d1db0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40097a0 tx=0x7fe1d402b960 comp rx=0 tx=0).stop 2023-11-28T20:50:29.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.654+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1082987662 shutdown_connections 2023-11-28T20:50:29.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.654+0000 7fe1eafa0640 1 --2- 172.21.15.45:0/1082987662 >> v2:172.21.15.190:3300/0 conn(0x7fe1e40b1d40 0x7fe1e40d22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.654+0000 7fe1eafa0640 1 --2- 172.21.15.45:0/1082987662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40b2290 0x7fe1e40d1db0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.654+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1082987662 >> 172.21.15.45:0/1082987662 conn(0x7fe1e4070f40 msgr2=0x7fe1e4071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:29.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.654+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1082987662 shutdown_connections 2023-11-28T20:50:29.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.654+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1082987662 wait complete. 2023-11-28T20:50:29.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.656+0000 7fe1eafa0640 1 Processor -- start 2023-11-28T20:50:29.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.657+0000 7fe1eafa0640 1 -- start start 2023-11-28T20:50:29.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.658+0000 7fe1eafa0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40b1d40 0x7fe1e414b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:29.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.658+0000 7fe1eafa0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40b2290 0x7fe1e41e43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:29.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.658+0000 7fe1eafa0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40d0040 0x7fe1e41e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:29.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.658+0000 7fe1eafa0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e414ba80 con 0x7fe1e40b2290 2023-11-28T20:50:29.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.658+0000 7fe1eafa0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e41e4930 con 0x7fe1e40b1d40 2023-11-28T20:50:29.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.658+0000 7fe1eafa0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e41e3f60 con 0x7fe1e40d0040 2023-11-28T20:50:29.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.658+0000 7fe1e979d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40b2290 0x7fe1e41e43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:29.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.658+0000 7fe1e979d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40b2290 0x7fe1e41e43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53470/0 (socket says 172.21.15.45:53470) 2023-11-28T20:50:29.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.658+0000 7fe1e979d640 1 -- 172.21.15.45:0/1648290557 learned_addr learned my addr 172.21.15.45:0/1648290557 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:29.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1e9f9e640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40b1d40 0x7fe1e414b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:29.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1ea79f640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40d0040 0x7fe1e41e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:29.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1e979d640 1 -- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40d0040 msgr2=0x7fe1e41e4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:29.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1e979d640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40d0040 0x7fe1e41e4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1e979d640 1 -- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40b1d40 msgr2=0x7fe1e414b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:29.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1e979d640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40b1d40 0x7fe1e414b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:29.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1e979d640 1 -- 172.21.15.45:0/1648290557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4009400 con 0x7fe1e40b2290 2023-11-28T20:50:29.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1e9f9e640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40b1d40 0x7fe1e414b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:29.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1ea79f640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40d0040 0x7fe1e41e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:29.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1e979d640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40b2290 0x7fe1e41e43f0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe1d402bf70 tx=0x7fe1d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:29.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.659+0000 7fe1da7fc640 1 -- 172.21.15.45:0/1648290557 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d403e040 con 0x7fe1e40b2290 2023-11-28T20:50:29.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.660+0000 7fe1da7fc640 1 -- 172.21.15.45:0/1648290557 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d4039a40 con 0x7fe1e40b2290 2023-11-28T20:50:29.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.660+0000 7fe1da7fc640 1 -- 172.21.15.45:0/1648290557 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4039df0 con 0x7fe1e40b2290 2023-11-28T20:50:29.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.660+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e41e3350 con 0x7fe1e40b2290 2023-11-28T20:50:29.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.660+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e41e3910 con 0x7fe1e40b2290 2023-11-28T20:50:29.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.661+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e41e40d0 con 0x7fe1e40b2290 2023-11-28T20:50:29.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.662+0000 7fe1da7fc640 1 -- 172.21.15.45:0/1648290557 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe1d4043390 con 0x7fe1e40b2290 2023-11-28T20:50:29.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.663+0000 7fe1da7fc640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1c806af60 0x7fe1c806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:29.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.663+0000 7fe1da7fc640 1 -- 172.21.15.45:0/1648290557 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe1d40abbe0 con 0x7fe1e40b2290 2023-11-28T20:50:29.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.663+0000 7fe1e9f9e640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1c806af60 0x7fe1c806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:29.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.665+0000 7fe1e9f9e640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1c806af60 0x7fe1c806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0007fd0 tx=0x7fe1e0009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:29.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.668+0000 7fe1da7fc640 1 -- 172.21.15.45:0/1648290557 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1d40769b0 con 0x7fe1e40b2290 2023-11-28T20:50:29.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:29.908+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe1e4071b30 con 0x7fe1e40b2290 2023-11-28T20:50:30.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.399+0000 7fe1da7fc640 1 -- 172.21.15.45:0/1648290557 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe1d4079fd0 con 0x7fe1e40b2290 2023-11-28T20:50:30.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1c806af60 msgr2=0x7fe1c806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:30.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1c806af60 0x7fe1c806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0007fd0 tx=0x7fe1e0009040 comp rx=0 tx=0).stop 2023-11-28T20:50:30.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40b2290 msgr2=0x7fe1e41e43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:30.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40b2290 0x7fe1e41e43f0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe1d402bf70 tx=0x7fe1d4004730 comp rx=0 tx=0).stop 2023-11-28T20:50:30.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 shutdown_connections 2023-11-28T20:50:30.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1c806af60 0x7fe1c806d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1e40d0040 0x7fe1e41e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1e40b2290 0x7fe1e41e43f0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 --2- 172.21.15.45:0/1648290557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e40b1d40 0x7fe1e414b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.402+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 >> 172.21.15.45:0/1648290557 conn(0x7fe1e4070f40 msgr2=0x7fe1e40d0420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:30.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.403+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 shutdown_connections 2023-11-28T20:50:30.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.403+0000 7fe1eafa0640 1 -- 172.21.15.45:0/1648290557 wait complete. 2023-11-28T20:50:30.425 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T20:50:30.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:30.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.649+0000 7feb04ac3640 1 Processor -- start 2023-11-28T20:50:30.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.649+0000 7feb04ac3640 1 -- start start 2023-11-28T20:50:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.651+0000 7feb04ac3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00072200 0x7feb000b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.651+0000 7feb04ac3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb00146610 0x7feb000b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.651+0000 7feb04ac3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb000b4310 con 0x7feb00146b60 2023-11-28T20:50:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.651+0000 7feb04ac3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb000725e0 con 0x7feb00146610 2023-11-28T20:50:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.651+0000 7feb04ac3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0006d8d0 con 0x7feb00072200 2023-11-28T20:50:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.651+0000 7feafeffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb00146610 0x7feb000b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.651+0000 7feafeffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb00146610 0x7feb000b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53486/0 (socket says 172.21.15.45:53486) 2023-11-28T20:50:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.651+0000 7feafeffd640 1 -- 172.21.15.45:0/2068056535 learned_addr learned my addr 172.21.15.45:0/2068056535 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.652+0000 7feaff7fe640 1 --2- 172.21.15.45:0/2068056535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00072200 0x7feb000b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.652+0000 7feafeffd640 1 -- 172.21.15.45:0/2068056535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00072200 msgr2=0x7feb000b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.652+0000 7feafeffd640 1 --2- 172.21.15.45:0/2068056535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00072200 0x7feb000b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.652+0000 7feafeffd640 1 -- 172.21.15.45:0/2068056535 >> v1:172.21.15.190:6789/0 conn(0x7feb00146b60 legacy=0x7feb00160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.652+0000 7feafeffd640 1 -- 172.21.15.45:0/2068056535 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0006ca20 con 0x7feb00146610 2023-11-28T20:50:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.652+0000 7feaff7fe640 1 --2- 172.21.15.45:0/2068056535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00072200 0x7feb000b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.653+0000 7feafeffd640 1 --2- 172.21.15.45:0/2068056535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 0x7feb000b22d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009e50 tx=0x7feaf002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7114d01e568917b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.653+0000 7feafe7fc640 1 -- 172.21.15.45:0/2068056535 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf002ca80 con 0x7feb00146610 2023-11-28T20:50:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.653+0000 7feafe7fc640 1 -- 172.21.15.45:0/2068056535 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf002cbe0 con 0x7feb00146610 2023-11-28T20:50:30.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.653+0000 7feafe7fc640 1 -- 172.21.15.45:0/2068056535 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf00343f0 con 0x7feb00146610 2023-11-28T20:50:30.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.654+0000 7feb04ac3640 1 -- 172.21.15.45:0/2068056535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 msgr2=0x7feb000b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:30.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.654+0000 7feb04ac3640 1 --2- 172.21.15.45:0/2068056535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 0x7feb000b22d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009e50 tx=0x7feaf002bee0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.654+0000 7feb04ac3640 1 -- 172.21.15.45:0/2068056535 shutdown_connections 2023-11-28T20:50:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.654+0000 7feb04ac3640 1 --2- 172.21.15.45:0/2068056535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 0x7feb000b22d0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.654+0000 7feb04ac3640 1 --2- 172.21.15.45:0/2068056535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00072200 0x7feb000b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.654+0000 7feb04ac3640 1 -- 172.21.15.45:0/2068056535 >> 172.21.15.45:0/2068056535 conn(0x7feb0006ab30 msgr2=0x7feb000b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.654+0000 7feb04ac3640 1 -- 172.21.15.45:0/2068056535 shutdown_connections 2023-11-28T20:50:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.654+0000 7feb04ac3640 1 -- 172.21.15.45:0/2068056535 wait complete. 2023-11-28T20:50:30.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.656+0000 7feb04ac3640 1 Processor -- start 2023-11-28T20:50:30.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.656+0000 7feb04ac3640 1 -- start start 2023-11-28T20:50:30.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.657+0000 7feb04ac3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb00072200 0x7feb0015cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:30.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.657+0000 7feb04ac3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 0x7feb0015ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.657+0000 7feb04ac3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00146b60 0x7feb00140cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.657+0000 7feb04ac3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0015dcc0 con 0x7feb00146610 2023-11-28T20:50:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.657+0000 7feb04ac3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0015dfa0 con 0x7feb00072200 2023-11-28T20:50:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.657+0000 7feb04ac3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0015e280 con 0x7feb00146b60 2023-11-28T20:50:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.657+0000 7feafeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 0x7feb0015ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feafeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 0x7feb0015ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53492/0 (socket says 172.21.15.45:53492) 2023-11-28T20:50:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feafeffd640 1 -- 172.21.15.45:0/4059100132 learned_addr learned my addr 172.21.15.45:0/4059100132 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feaff7fe640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb00072200 0x7feb0015cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feafffff640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00146b60 0x7feb00140cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feafeffd640 1 -- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00146b60 msgr2=0x7feb00140cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feafeffd640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00146b60 0x7feb00140cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feafeffd640 1 -- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb00072200 msgr2=0x7feb0015cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feafeffd640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb00072200 0x7feb0015cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feafeffd640 1 -- 172.21.15.45:0/4059100132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf0009b30 con 0x7feb00146610 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feaff7fe640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb00072200 0x7feb0015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.658+0000 7feafeffd640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 0x7feb0015ed60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009e50 tx=0x7feaf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.659+0000 7feaf7fff640 1 -- 172.21.15.45:0/4059100132 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf003e450 con 0x7feb00146610 2023-11-28T20:50:30.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.659+0000 7feaf7fff640 1 -- 172.21.15.45:0/4059100132 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf003e5b0 con 0x7feb00146610 2023-11-28T20:50:30.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.659+0000 7feaf7fff640 1 -- 172.21.15.45:0/4059100132 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf003e960 con 0x7feb00146610 2023-11-28T20:50:30.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.659+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0015e560 con 0x7feb00146610 2023-11-28T20:50:30.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.659+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0015d0b0 con 0x7feb00146610 2023-11-28T20:50:30.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.660+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0015ea20 con 0x7feb00146610 2023-11-28T20:50:30.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.661+0000 7feaf7fff640 1 -- 172.21.15.45:0/4059100132 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7feaf002cd50 con 0x7feb00146610 2023-11-28T20:50:30.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.662+0000 7feaf7fff640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7feae006af60 0x7feae006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:30.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.662+0000 7feaf7fff640 1 -- 172.21.15.45:0/4059100132 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7feaf00ac3b0 con 0x7feb00146610 2023-11-28T20:50:30.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.663+0000 7feaff7fe640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7feae006af60 0x7feae006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:30.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.664+0000 7feaff7fe640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7feae006af60 0x7feae006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feae8006fd0 tx=0x7feae8008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:30.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.666+0000 7feaf7fff640 1 -- 172.21.15.45:0/4059100132 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feaf0077200 con 0x7feb00146610 2023-11-28T20:50:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.900+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb000b4310 con 0x7feb00146610 2023-11-28T20:50:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.902+0000 7feaf7fff640 1 -- 172.21.15.45:0/4059100132 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4914 (secure 0 0 0) 0x7feb000b4310 con 0x7feb00146610 2023-11-28T20:50:30.904 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:30.904 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5438,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2396470524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2396470524},{"type":"v1","addr":"172.21.15.45:6835","nonce":2396470524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/436103276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":436103276},{"type":"v1","addr":"172.21.15.190:6836","nonce":436103276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/834328551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":834328551},{"type":"v1","addr":"172.21.15.45:6834","nonce":834328551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:50:26.812959+0000","modified":"2023-11-28T20:50:29.910521+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6837/3032907708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3032907708},{"type":"v1","addr":"172.21.15.190:6837","nonce":3032907708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T20:50:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 36 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.904+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7feae006af60 msgr2=0x7feae006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.904+0000 7feb04ac3640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7feae006af60 0x7feae006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feae8006fd0 tx=0x7feae8008040 comp rx=0 tx=0).stop 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.904+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 msgr2=0x7feb0015ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.904+0000 7feb04ac3640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 0x7feb0015ed60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009e50 tx=0x7feaf0004a40 comp rx=0 tx=0).stop 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.904+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 shutdown_connections 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.904+0000 7feb04ac3640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7feae006af60 0x7feae006d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.904+0000 7feb04ac3640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb00146b60 0x7feb00140cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.904+0000 7feb04ac3640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb00146610 0x7feb0015ed60 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.904+0000 7feb04ac3640 1 --2- 172.21.15.45:0/4059100132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb00072200 0x7feb0015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.905+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 >> 172.21.15.45:0/4059100132 conn(0x7feb0006ab30 msgr2=0x7feb000ae410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:30.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.905+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 shutdown_connections 2023-11-28T20:50:30.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:30.905+0000 7feb04ac3640 1 -- 172.21.15.45:0/4059100132 wait complete. 2023-11-28T20:50:30.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.142+0000 7f163cf35640 1 Processor -- start 2023-11-28T20:50:31.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.142+0000 7f163cf35640 1 -- start start 2023-11-28T20:50:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f163cf35640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f163815f110 0x7f16380d1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f163cf35640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f163806e490 0x7f163815f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f163cf35640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 0x7f163815adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f163cf35640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16380cbb90 con 0x7f163815f110 2023-11-28T20:50:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f163cf35640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16380cf4c0 con 0x7f163806e490 2023-11-28T20:50:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f163cf35640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16380cf7a0 con 0x7f163815fa30 2023-11-28T20:50:31.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f16377fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f163815f110 0x7f16380d1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f16377fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f163815f110 0x7f16380d1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53506/0 (socket says 172.21.15.45:53506) 2023-11-28T20:50:31.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f16377fe640 1 -- 172.21.15.45:0/3707334616 learned_addr learned my addr 172.21.15.45:0/3707334616 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:31.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f1636ffd640 1 --2- 172.21.15.45:0/3707334616 >> v2:172.21.15.190:3300/0 conn(0x7f163806e490 0x7f163815f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f1637fff640 1 --2- 172.21.15.45:0/3707334616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 0x7f163815adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.144+0000 7f16377fe640 1 -- 172.21.15.45:0/3707334616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 msgr2=0x7f163815adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.145+0000 7f16377fe640 1 --2- 172.21.15.45:0/3707334616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 0x7f163815adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.145+0000 7f16377fe640 1 -- 172.21.15.45:0/3707334616 >> v2:172.21.15.190:3300/0 conn(0x7f163806e490 msgr2=0x7f163815f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.145+0000 7f16377fe640 1 --2- 172.21.15.45:0/3707334616 >> v2:172.21.15.190:3300/0 conn(0x7f163806e490 0x7f163815f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.145+0000 7f16377fe640 1 -- 172.21.15.45:0/3707334616 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16380cfa80 con 0x7f163815f110 2023-11-28T20:50:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.145+0000 7f1637fff640 1 --2- 172.21.15.45:0/3707334616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 0x7f163815adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:31.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.146+0000 7f16377fe640 1 --2- 172.21.15.45:0/3707334616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 0x7f16380d1a00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f163806d6f0 tx=0x7f162802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e453807cbb003ab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:31.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.146+0000 7f16367fc640 1 -- 172.21.15.45:0/3707334616 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628031870 con 0x7f163815f110 2023-11-28T20:50:31.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.146+0000 7f16367fc640 1 -- 172.21.15.45:0/3707334616 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16280319d0 con 0x7f163815f110 2023-11-28T20:50:31.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.146+0000 7f16367fc640 1 -- 172.21.15.45:0/3707334616 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628031d60 con 0x7f163815f110 2023-11-28T20:50:31.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.147+0000 7f163cf35640 1 -- 172.21.15.45:0/3707334616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 msgr2=0x7f16380d1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.147+0000 7f163cf35640 1 --2- 172.21.15.45:0/3707334616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 0x7f16380d1a00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f163806d6f0 tx=0x7f162802b910 comp rx=0 tx=0).stop 2023-11-28T20:50:31.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.147+0000 7f163cf35640 1 -- 172.21.15.45:0/3707334616 shutdown_connections 2023-11-28T20:50:31.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.147+0000 7f163cf35640 1 --2- 172.21.15.45:0/3707334616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 0x7f163815adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.147+0000 7f163cf35640 1 --2- 172.21.15.45:0/3707334616 >> v2:172.21.15.190:3300/0 conn(0x7f163806e490 0x7f163815f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.147+0000 7f163cf35640 1 --2- 172.21.15.45:0/3707334616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 0x7f16380d1a00 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.147+0000 7f163cf35640 1 -- 172.21.15.45:0/3707334616 >> 172.21.15.45:0/3707334616 conn(0x7f16380d0ee0 msgr2=0x7f16380d12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:31.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.147+0000 7f163cf35640 1 -- 172.21.15.45:0/3707334616 shutdown_connections 2023-11-28T20:50:31.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.147+0000 7f163cf35640 1 -- 172.21.15.45:0/3707334616 wait complete. 2023-11-28T20:50:31.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.149+0000 7f163cf35640 1 Processor -- start 2023-11-28T20:50:31.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.150+0000 7f163cf35640 1 -- start start 2023-11-28T20:50:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f163cf35640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f163806e490 0x7f16381f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f163cf35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 0x7f16380b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f163cf35640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 0x7f16380c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f163cf35640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16380b4710 con 0x7f163815f110 2023-11-28T20:50:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f163cf35640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16380b3b00 con 0x7f163806e490 2023-11-28T20:50:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f163cf35640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16380b3de0 con 0x7f163815fa30 2023-11-28T20:50:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f1636ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 0x7f16380b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f1636ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 0x7f16380b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53508/0 (socket says 172.21.15.45:53508) 2023-11-28T20:50:31.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f1636ffd640 1 -- 172.21.15.45:0/854175068 learned_addr learned my addr 172.21.15.45:0/854175068 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:31.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.151+0000 7f16377fe640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f163806e490 0x7f16381f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f1637fff640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 0x7f16380c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f1636ffd640 1 -- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 msgr2=0x7f16380c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f1636ffd640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 0x7f16380c95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f1636ffd640 1 -- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f163806e490 msgr2=0x7f16381f24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f1636ffd640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f163806e490 0x7f16381f24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f1636ffd640 1 -- 172.21.15.45:0/854175068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16280094d0 con 0x7f163815f110 2023-11-28T20:50:31.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f1636ffd640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 0x7f16380b0940 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f162c00d800 tx=0x7f162c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:31.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f161ffff640 1 -- 172.21.15.45:0/854175068 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00b6c0 con 0x7f163815f110 2023-11-28T20:50:31.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f161ffff640 1 -- 172.21.15.45:0/854175068 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c00b820 con 0x7f163815f110 2023-11-28T20:50:31.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f161ffff640 1 -- 172.21.15.45:0/854175068 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00bbb0 con 0x7f163815f110 2023-11-28T20:50:31.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.152+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16380b40c0 con 0x7f163815f110 2023-11-28T20:50:31.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.153+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16381f2a20 con 0x7f163815f110 2023-11-28T20:50:31.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.154+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16380ce900 con 0x7f163815f110 2023-11-28T20:50:31.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.155+0000 7f161ffff640 1 -- 172.21.15.45:0/854175068 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f162c00bd10 con 0x7f163815f110 2023-11-28T20:50:31.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.155+0000 7f161ffff640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f162406af60 0x7f162406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.156+0000 7f161ffff640 1 -- 172.21.15.45:0/854175068 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f162c089e30 con 0x7f163815f110 2023-11-28T20:50:31.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.156+0000 7f16377fe640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f162406af60 0x7f162406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.157+0000 7f16377fe640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f162406af60 0x7f162406d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1628036000 tx=0x7f162803c040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:31.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.160+0000 7f161ffff640 1 -- 172.21.15.45:0/854175068 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f162c054c90 con 0x7f163815f110 2023-11-28T20:50:31.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.360+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f163809f030 con 0x7f163815f110 2023-11-28T20:50:31.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.363+0000 7f161ffff640 1 -- 172.21.15.45:0/854175068 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16704 (secure 0 0 0) 0x7f162c0582b0 con 0x7f163815f110 2023-11-28T20:50:31.365 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:31.365 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":88,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:28.371523+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:50:24.656511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-28T20:50:25.847745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:31.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.365+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f162406af60 msgr2=0x7f162406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.365+0000 7f163cf35640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f162406af60 0x7f162406d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1628036000 tx=0x7f162803c040 comp rx=0 tx=0).stop 2023-11-28T20:50:31.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.365+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 msgr2=0x7f16380b0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.365+0000 7f163cf35640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 0x7f16380b0940 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f162c00d800 tx=0x7f162c00dcc0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.366+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 shutdown_connections 2023-11-28T20:50:31.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.366+0000 7f163cf35640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f162406af60 0x7f162406d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.366+0000 7f163cf35640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f163815fa30 0x7f16380c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.366+0000 7f163cf35640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163815f110 0x7f16380b0940 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.366+0000 7f163cf35640 1 --2- 172.21.15.45:0/854175068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f163806e490 0x7f16381f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.366+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 >> 172.21.15.45:0/854175068 conn(0x7f16380d0ee0 msgr2=0x7f16380af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:31.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.366+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 shutdown_connections 2023-11-28T20:50:31.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.366+0000 7f163cf35640 1 -- 172.21.15.45:0/854175068 wait complete. 2023-11-28T20:50:31.388 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T20:50:31.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:50:31.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.612+0000 7fcfb4f2c640 1 Processor -- start 2023-11-28T20:50:31.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.612+0000 7fcfb4f2c640 1 -- start start 2023-11-28T20:50:31.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.613+0000 7fcfb4f2c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcfb00cfc00 0x7fcfb00d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.614+0000 7fcfb4f2c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcfb00b64f0 0x7fcfb00b6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.614+0000 7fcfb4f2c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb00d1300 con 0x7fcfb015d580 2023-11-28T20:50:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.614+0000 7fcfb4f2c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb00d1c60 con 0x7fcfb00cfc00 2023-11-28T20:50:31.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.614+0000 7fcfb4f2c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb00d05d0 con 0x7fcfb00b64f0 2023-11-28T20:50:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.614+0000 7fcfa5d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcfb00cfc00 0x7fcfb00d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.614+0000 7fcfa5d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcfb00cfc00 0x7fcfb00d0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53522/0 (socket says 172.21.15.45:53522) 2023-11-28T20:50:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.614+0000 7fcfa5d9b640 1 -- 172.21.15.45:0/3875228967 learned_addr learned my addr 172.21.15.45:0/3875228967 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.614+0000 7fcfaed9d640 1 --2- 172.21.15.45:0/3875228967 >> v2:172.21.15.190:3300/0 conn(0x7fcfb00b64f0 0x7fcfb00b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.615+0000 7fcfa5d9b640 1 -- 172.21.15.45:0/3875228967 >> v2:172.21.15.190:3300/0 conn(0x7fcfb00b64f0 msgr2=0x7fcfb00b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.615+0000 7fcfa5d9b640 1 --2- 172.21.15.45:0/3875228967 >> v2:172.21.15.190:3300/0 conn(0x7fcfb00b64f0 0x7fcfb00b6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.615+0000 7fcfa5d9b640 1 -- 172.21.15.45:0/3875228967 >> v1:172.21.15.45:6789/0 conn(0x7fcfb015d580 legacy=0x7fcfb00d2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.615+0000 7fcfa5d9b640 1 -- 172.21.15.45:0/3875228967 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb00d15a0 con 0x7fcfb00cfc00 2023-11-28T20:50:31.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.615+0000 7fcfadd9b640 1 -- 172.21.15.45:0/3875228967 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651808717 0 0) 0x7fcfb00d1300 con 0x7fcfb015d580 2023-11-28T20:50:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.616+0000 7fcfa5d9b640 1 --2- 172.21.15.45:0/3875228967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb00cfc00 0x7fcfb00d0090 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00cd8c0 tx=0x7fcf9802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67886271b7ce3bc0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.616+0000 7fcfadd9b640 1 -- 172.21.15.45:0/3875228967 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9802cb20 con 0x7fcfb00cfc00 2023-11-28T20:50:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.616+0000 7fcfadd9b640 1 -- 172.21.15.45:0/3875228967 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9802cc80 con 0x7fcfb00cfc00 2023-11-28T20:50:31.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.616+0000 7fcfadd9b640 1 -- 172.21.15.45:0/3875228967 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980343f0 con 0x7fcfb00cfc00 2023-11-28T20:50:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.617+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/3875228967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb00cfc00 msgr2=0x7fcfb00d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.617+0000 7fcfb4f2c640 1 --2- 172.21.15.45:0/3875228967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb00cfc00 0x7fcfb00d0090 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00cd8c0 tx=0x7fcf9802bf80 comp rx=0 tx=0).stop 2023-11-28T20:50:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.617+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/3875228967 shutdown_connections 2023-11-28T20:50:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.617+0000 7fcfb4f2c640 1 --2- 172.21.15.45:0/3875228967 >> v2:172.21.15.190:3300/0 conn(0x7fcfb00b64f0 0x7fcfb00b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.617+0000 7fcfb4f2c640 1 --2- 172.21.15.45:0/3875228967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb00cfc00 0x7fcfb00d0090 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.617+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/3875228967 >> 172.21.15.45:0/3875228967 conn(0x7fcfb00b4e10 msgr2=0x7fcfb00b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.617+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/3875228967 shutdown_connections 2023-11-28T20:50:31.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.617+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/3875228967 wait complete. 2023-11-28T20:50:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.619+0000 7fcfb4f2c640 1 Processor -- start 2023-11-28T20:50:31.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.619+0000 7fcfb4f2c640 1 -- start start 2023-11-28T20:50:31.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.620+0000 7fcfb4f2c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfb00b64f0 0x7fcfb01f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.620+0000 7fcfb4f2c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfb00cfc00 0x7fcfb01f0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.620+0000 7fcfb4f2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb015d580 0x7fcfb00cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.620+0000 7fcfb4f2c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb01f07b0 con 0x7fcfb015d580 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.620+0000 7fcfb4f2c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb015f790 con 0x7fcfb00b64f0 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.620+0000 7fcfb4f2c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb00c8010 con 0x7fcfb00cfc00 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.620+0000 7fcfaed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb015d580 0x7fcfb00cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.620+0000 7fcfaed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb015d580 0x7fcfb00cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53538/0 (socket says 172.21.15.45:53538) 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.620+0000 7fcfaed9d640 1 -- 172.21.15.45:0/1360447148 learned_addr learned my addr 172.21.15.45:0/1360447148 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.621+0000 7fcfaed9d640 1 -- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfb00cfc00 msgr2=0x7fcfb01f0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.621+0000 7fcfaed9d640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfb00cfc00 0x7fcfb01f0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.621+0000 7fcfaed9d640 1 -- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfb00b64f0 msgr2=0x7fcfb01f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.621+0000 7fcfae59c640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfb00b64f0 0x7fcfb01f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.621+0000 7fcfaed9d640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfb00b64f0 0x7fcfb01f3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.621+0000 7fcfaed9d640 1 -- 172.21.15.45:0/1360447148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf98009b30 con 0x7fcfb015d580 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.621+0000 7fcfaed9d640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb015d580 0x7fcfb00cc3c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800d5d0 tx=0x7fcfa800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.621+0000 7fcfa77fe640 1 -- 172.21.15.45:0/1360447148 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80125f0 con 0x7fcfb015d580 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.622+0000 7fcfa77fe640 1 -- 172.21.15.45:0/1360447148 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa8012770 con 0x7fcfb015d580 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.622+0000 7fcfa77fe640 1 -- 172.21.15.45:0/1360447148 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8012b20 con 0x7fcfb015d580 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.622+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb00c82f0 con 0x7fcfb015d580 2023-11-28T20:50:31.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.622+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb00c88b0 con 0x7fcfb015d580 2023-11-28T20:50:31.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.623+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb00b91a0 con 0x7fcfb015d580 2023-11-28T20:50:31.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.624+0000 7fcfa77fe640 1 -- 172.21.15.45:0/1360447148 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fcfa8012c80 con 0x7fcfb015d580 2023-11-28T20:50:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.625+0000 7fcfa77fe640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcf9006af60 0x7fcf9006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.625+0000 7fcfa77fe640 1 -- 172.21.15.45:0/1360447148 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcfa8089e40 con 0x7fcfb015d580 2023-11-28T20:50:31.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.626+0000 7fcfae59c640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcf9006af60 0x7fcf9006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:31.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.627+0000 7fcfae59c640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcf9006af60 0x7fcf9006d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0000f60 tx=0x7fcfa0009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:31.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.629+0000 7fcfa77fe640 1 -- 172.21.15.45:0/1360447148 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfa8054c10 con 0x7fcfb015d580 2023-11-28T20:50:31.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.818+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 --> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcfb00d2080 con 0x7fcf9006af60 2023-11-28T20:50:31.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.846+0000 7fcfa77fe640 1 -- 172.21.15.45:0/1360447148 <== mgr.5327 v2:172.21.15.190:6832/108887886 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286612 (secure 0 0 0) 0x7fcfa0001c50 con 0x7fcf9006af60 2023-11-28T20:50:31.854 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:50:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcf9006af60 msgr2=0x7fcf9006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcf9006af60 0x7fcf9006d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0000f60 tx=0x7fcfa0009040 comp rx=0 tx=0).stop 2023-11-28T20:50:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb015d580 msgr2=0x7fcfb00cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb015d580 0x7fcfb00cc3c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800d5d0 tx=0x7fcfa800da90 comp rx=0 tx=0).stop 2023-11-28T20:50:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 shutdown_connections 2023-11-28T20:50:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fcf9006af60 0x7fcf9006d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb015d580 0x7fcfb00cc3c0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfb00cfc00 0x7fcfb01f0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 --2- 172.21.15.45:0/1360447148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfb00b64f0 0x7fcfb01f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 >> 172.21.15.45:0/1360447148 conn(0x7fcfb00b4e10 msgr2=0x7fcfb00b7650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 shutdown_connections 2023-11-28T20:50:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:31.853+0000 7fcfb4f2c640 1 -- 172.21.15.45:0/1360447148 wait complete. 2023-11-28T20:50:31.876 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-28T20:50:30.666491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":328436,"kb_used_data":9576,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754646284,"statfs":{"total":773094113280,"available":772757794816,"internally_reserved":0,"allocated":9805824,"data_stored":5469916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":326356528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.340840"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546692+0000","last_change":"2023-11-28T20:50:27.395653+0000","last_active":"2023-11-28T20:50:28.546692+0000","last_peered":"2023-11-28T20:50:28.546692+0000","last_clean":"2023-11-28T20:50:28.546692+0000","last_became_active":"2023-11-28T20:50:27.395364+0000","last_became_peered":"2023-11-28T20:50:27.395364+0000","last_unstale":"2023-11-28T20:50:28.546692+0000","last_undegraded":"2023-11-28T20:50:28.546692+0000","last_fullsized":"2023-11-28T20:50:28.546692+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:42:48.727931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546317+0000","last_change":"2023-11-28T20:50:27.395605+0000","last_active":"2023-11-28T20:50:28.546317+0000","last_peered":"2023-11-28T20:50:28.546317+0000","last_clean":"2023-11-28T20:50:28.546317+0000","last_became_active":"2023-11-28T20:50:27.395314+0000","last_became_peered":"2023-11-28T20:50:27.395314+0000","last_unstale":"2023-11-28T20:50:28.546317+0000","last_undegraded":"2023-11-28T20:50:28.546317+0000","last_fullsized":"2023-11-28T20:50:28.546317+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:01:07.388649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547626+0000","last_change":"2023-11-28T20:50:27.395539+0000","last_active":"2023-11-28T20:50:28.547626+0000","last_peered":"2023-11-28T20:50:28.547626+0000","last_clean":"2023-11-28T20:50:28.547626+0000","last_became_active":"2023-11-28T20:50:27.392159+0000","last_became_peered":"2023-11-28T20:50:27.392159+0000","last_unstale":"2023-11-28T20:50:28.547626+0000","last_undegraded":"2023-11-28T20:50:28.547626+0000","last_fullsized":"2023-11-28T20:50:28.547626+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:55:42.771188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389326+0000","last_change":"2023-11-28T20:50:27.395544+0000","last_active":"2023-11-28T20:50:28.389326+0000","last_peered":"2023-11-28T20:50:28.389326+0000","last_clean":"2023-11-28T20:50:28.389326+0000","last_became_active":"2023-11-28T20:50:27.395266+0000","last_became_peered":"2023-11-28T20:50:27.395266+0000","last_unstale":"2023-11-28T20:50:28.389326+0000","last_undegraded":"2023-11-28T20:50:28.389326+0000","last_fullsized":"2023-11-28T20:50:28.389326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:36.275997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548127+0000","last_change":"2023-11-28T20:50:27.399160+0000","last_active":"2023-11-28T20:50:28.548127+0000","last_peered":"2023-11-28T20:50:28.548127+0000","last_clean":"2023-11-28T20:50:28.548127+0000","last_became_active":"2023-11-28T20:50:27.398904+0000","last_became_peered":"2023-11-28T20:50:27.398904+0000","last_unstale":"2023-11-28T20:50:28.548127+0000","last_undegraded":"2023-11-28T20:50:28.548127+0000","last_fullsized":"2023-11-28T20:50:28.548127+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:18.626234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545314+0000","last_change":"2023-11-28T20:50:27.377240+0000","last_active":"2023-11-28T20:50:28.545314+0000","last_peered":"2023-11-28T20:50:28.545314+0000","last_clean":"2023-11-28T20:50:28.545314+0000","last_became_active":"2023-11-28T20:50:27.376847+0000","last_became_peered":"2023-11-28T20:50:27.376847+0000","last_unstale":"2023-11-28T20:50:28.545314+0000","last_undegraded":"2023-11-28T20:50:28.545314+0000","last_fullsized":"2023-11-28T20:50:28.545314+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:47:53.341510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.387257+0000","last_change":"2023-11-28T20:50:27.379160+0000","last_active":"2023-11-28T20:50:28.387257+0000","last_peered":"2023-11-28T20:50:28.387257+0000","last_clean":"2023-11-28T20:50:28.387257+0000","last_became_active":"2023-11-28T20:50:27.378801+0000","last_became_peered":"2023-11-28T20:50:27.378801+0000","last_unstale":"2023-11-28T20:50:28.387257+0000","last_undegraded":"2023-11-28T20:50:28.387257+0000","last_fullsized":"2023-11-28T20:50:28.387257+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:45:25.514682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516557+0000","last_change":"2023-11-28T20:50:27.381383+0000","last_active":"2023-11-28T20:50:28.516557+0000","last_peered":"2023-11-28T20:50:28.516557+0000","last_clean":"2023-11-28T20:50:28.516557+0000","last_became_active":"2023-11-28T20:50:27.379217+0000","last_became_peered":"2023-11-28T20:50:27.379217+0000","last_unstale":"2023-11-28T20:50:28.516557+0000","last_undegraded":"2023-11-28T20:50:28.516557+0000","last_fullsized":"2023-11-28T20:50:28.516557+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:48:28.926200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.549189+0000","last_change":"2023-11-28T20:50:27.393196+0000","last_active":"2023-11-28T20:50:28.549189+0000","last_peered":"2023-11-28T20:50:28.549189+0000","last_clean":"2023-11-28T20:50:28.549189+0000","last_became_active":"2023-11-28T20:50:27.392720+0000","last_became_peered":"2023-11-28T20:50:27.392720+0000","last_unstale":"2023-11-28T20:50:28.549189+0000","last_undegraded":"2023-11-28T20:50:28.549189+0000","last_fullsized":"2023-11-28T20:50:28.549189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:47.085267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548782+0000","last_change":"2023-11-28T20:50:27.395531+0000","last_active":"2023-11-28T20:50:28.548782+0000","last_peered":"2023-11-28T20:50:28.548782+0000","last_clean":"2023-11-28T20:50:28.548782+0000","last_became_active":"2023-11-28T20:50:27.395254+0000","last_became_peered":"2023-11-28T20:50:27.395254+0000","last_unstale":"2023-11-28T20:50:28.548782+0000","last_undegraded":"2023-11-28T20:50:28.548782+0000","last_fullsized":"2023-11-28T20:50:28.548782+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:54.989089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389951+0000","last_change":"2023-11-28T20:50:27.400513+0000","last_active":"2023-11-28T20:50:28.389951+0000","last_peered":"2023-11-28T20:50:28.389951+0000","last_clean":"2023-11-28T20:50:28.389951+0000","last_became_active":"2023-11-28T20:50:27.393358+0000","last_became_peered":"2023-11-28T20:50:27.393358+0000","last_unstale":"2023-11-28T20:50:28.389951+0000","last_undegraded":"2023-11-28T20:50:28.389951+0000","last_fullsized":"2023-11-28T20:50:28.389951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:06.489084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517101+0000","last_change":"2023-11-28T20:50:27.399075+0000","last_active":"2023-11-28T20:50:28.517101+0000","last_peered":"2023-11-28T20:50:28.517101+0000","last_clean":"2023-11-28T20:50:28.517101+0000","last_became_active":"2023-11-28T20:50:27.398825+0000","last_became_peered":"2023-11-28T20:50:27.398825+0000","last_unstale":"2023-11-28T20:50:28.517101+0000","last_undegraded":"2023-11-28T20:50:28.517101+0000","last_fullsized":"2023-11-28T20:50:28.517101+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:21:49.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517095+0000","last_change":"2023-11-28T20:50:27.405174+0000","last_active":"2023-11-28T20:50:28.517095+0000","last_peered":"2023-11-28T20:50:28.517095+0000","last_clean":"2023-11-28T20:50:28.517095+0000","last_became_active":"2023-11-28T20:50:27.404467+0000","last_became_peered":"2023-11-28T20:50:27.404467+0000","last_unstale":"2023-11-28T20:50:28.517095+0000","last_undegraded":"2023-11-28T20:50:28.517095+0000","last_fullsized":"2023-11-28T20:50:28.517095+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:21.780299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546773+0000","last_change":"2023-11-28T20:50:27.392435+0000","last_active":"2023-11-28T20:50:28.546773+0000","last_peered":"2023-11-28T20:50:28.546773+0000","last_clean":"2023-11-28T20:50:28.546773+0000","last_became_active":"2023-11-28T20:50:27.392115+0000","last_became_peered":"2023-11-28T20:50:27.392115+0000","last_unstale":"2023-11-28T20:50:28.546773+0000","last_undegraded":"2023-11-28T20:50:28.546773+0000","last_fullsized":"2023-11-28T20:50:28.546773+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:21.780299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546137+0000","last_change":"2023-11-28T20:50:27.385024+0000","last_active":"2023-11-28T20:50:28.546137+0000","last_peered":"2023-11-28T20:50:28.546137+0000","last_clean":"2023-11-28T20:50:28.546137+0000","last_became_active":"2023-11-28T20:50:27.384754+0000","last_became_peered":"2023-11-28T20:50:27.384754+0000","last_unstale":"2023-11-28T20:50:28.546137+0000","last_undegraded":"2023-11-28T20:50:28.546137+0000","last_fullsized":"2023-11-28T20:50:28.546137+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:21:49.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.386336+0000","last_change":"2023-11-28T20:50:27.382289+0000","last_active":"2023-11-28T20:50:28.386336+0000","last_peered":"2023-11-28T20:50:28.386336+0000","last_clean":"2023-11-28T20:50:28.386336+0000","last_became_active":"2023-11-28T20:50:27.381990+0000","last_became_peered":"2023-11-28T20:50:27.381990+0000","last_unstale":"2023-11-28T20:50:28.386336+0000","last_undegraded":"2023-11-28T20:50:28.386336+0000","last_fullsized":"2023-11-28T20:50:28.386336+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:07:33.146099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546691+0000","last_change":"2023-11-28T20:50:27.403539+0000","last_active":"2023-11-28T20:50:28.546691+0000","last_peered":"2023-11-28T20:50:28.546691+0000","last_clean":"2023-11-28T20:50:28.546691+0000","last_became_active":"2023-11-28T20:50:27.403106+0000","last_became_peered":"2023-11-28T20:50:27.403106+0000","last_unstale":"2023-11-28T20:50:28.546691+0000","last_undegraded":"2023-11-28T20:50:28.546691+0000","last_fullsized":"2023-11-28T20:50:28.546691+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:04:00.856638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389916+0000","last_change":"2023-11-28T20:50:27.397612+0000","last_active":"2023-11-28T20:50:28.389916+0000","last_peered":"2023-11-28T20:50:28.389916+0000","last_clean":"2023-11-28T20:50:28.389916+0000","last_became_active":"2023-11-28T20:50:27.397290+0000","last_became_peered":"2023-11-28T20:50:27.397290+0000","last_unstale":"2023-11-28T20:50:28.389916+0000","last_undegraded":"2023-11-28T20:50:28.389916+0000","last_fullsized":"2023-11-28T20:50:28.389916+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:09.882959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547689+0000","last_change":"2023-11-28T20:50:27.388858+0000","last_active":"2023-11-28T20:50:28.547689+0000","last_peered":"2023-11-28T20:50:28.547689+0000","last_clean":"2023-11-28T20:50:28.547689+0000","last_became_active":"2023-11-28T20:50:27.388547+0000","last_became_peered":"2023-11-28T20:50:27.388547+0000","last_unstale":"2023-11-28T20:50:28.547689+0000","last_undegraded":"2023-11-28T20:50:28.547689+0000","last_fullsized":"2023-11-28T20:50:28.547689+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:28.286051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388960+0000","last_change":"2023-11-28T20:50:27.386241+0000","last_active":"2023-11-28T20:50:28.388960+0000","last_peered":"2023-11-28T20:50:28.388960+0000","last_clean":"2023-11-28T20:50:28.388960+0000","last_became_active":"2023-11-28T20:50:27.385985+0000","last_became_peered":"2023-11-28T20:50:27.385985+0000","last_unstale":"2023-11-28T20:50:28.388960+0000","last_undegraded":"2023-11-28T20:50:28.388960+0000","last_fullsized":"2023-11-28T20:50:28.388960+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:11:33.831894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517130+0000","last_change":"2023-11-28T20:50:27.402514+0000","last_active":"2023-11-28T20:50:28.517130+0000","last_peered":"2023-11-28T20:50:28.517130+0000","last_clean":"2023-11-28T20:50:28.517130+0000","last_became_active":"2023-11-28T20:50:27.399578+0000","last_became_peered":"2023-11-28T20:50:27.399578+0000","last_unstale":"2023-11-28T20:50:28.517130+0000","last_undegraded":"2023-11-28T20:50:28.517130+0000","last_fullsized":"2023-11-28T20:50:28.517130+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:17:56.648761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388598+0000","last_change":"2023-11-28T20:50:27.381861+0000","last_active":"2023-11-28T20:50:28.388598+0000","last_peered":"2023-11-28T20:50:28.388598+0000","last_clean":"2023-11-28T20:50:28.388598+0000","last_became_active":"2023-11-28T20:50:27.381572+0000","last_became_peered":"2023-11-28T20:50:27.381572+0000","last_unstale":"2023-11-28T20:50:28.388598+0000","last_undegraded":"2023-11-28T20:50:28.388598+0000","last_fullsized":"2023-11-28T20:50:28.388598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:18.836466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389428+0000","last_change":"2023-11-28T20:50:27.377228+0000","last_active":"2023-11-28T20:50:28.389428+0000","last_peered":"2023-11-28T20:50:28.389428+0000","last_clean":"2023-11-28T20:50:28.389428+0000","last_became_active":"2023-11-28T20:50:27.375619+0000","last_became_peered":"2023-11-28T20:50:27.375619+0000","last_unstale":"2023-11-28T20:50:28.389428+0000","last_undegraded":"2023-11-28T20:50:28.389428+0000","last_fullsized":"2023-11-28T20:50:28.389428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:05:34.144511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546099+0000","last_change":"2023-11-28T20:50:27.391584+0000","last_active":"2023-11-28T20:50:28.546099+0000","last_peered":"2023-11-28T20:50:28.546099+0000","last_clean":"2023-11-28T20:50:28.546099+0000","last_became_active":"2023-11-28T20:50:27.391273+0000","last_became_peered":"2023-11-28T20:50:27.391273+0000","last_unstale":"2023-11-28T20:50:28.546099+0000","last_undegraded":"2023-11-28T20:50:28.546099+0000","last_fullsized":"2023-11-28T20:50:28.546099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:44.386833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517541+0000","last_change":"2023-11-28T20:50:27.384755+0000","last_active":"2023-11-28T20:50:28.517541+0000","last_peered":"2023-11-28T20:50:28.517541+0000","last_clean":"2023-11-28T20:50:28.517541+0000","last_became_active":"2023-11-28T20:50:27.384493+0000","last_became_peered":"2023-11-28T20:50:27.384493+0000","last_unstale":"2023-11-28T20:50:28.517541+0000","last_undegraded":"2023-11-28T20:50:28.517541+0000","last_fullsized":"2023-11-28T20:50:28.517541+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:54.989089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517275+0000","last_change":"2023-11-28T20:50:27.395102+0000","last_active":"2023-11-28T20:50:28.517275+0000","last_peered":"2023-11-28T20:50:28.517275+0000","last_clean":"2023-11-28T20:50:28.517275+0000","last_became_active":"2023-11-28T20:50:27.394658+0000","last_became_peered":"2023-11-28T20:50:27.394658+0000","last_unstale":"2023-11-28T20:50:28.517275+0000","last_undegraded":"2023-11-28T20:50:28.517275+0000","last_fullsized":"2023-11-28T20:50:28.517275+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:36.701581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389875+0000","last_change":"2023-11-28T20:50:27.390252+0000","last_active":"2023-11-28T20:50:28.389875+0000","last_peered":"2023-11-28T20:50:28.389875+0000","last_clean":"2023-11-28T20:50:28.389875+0000","last_became_active":"2023-11-28T20:50:27.389988+0000","last_became_peered":"2023-11-28T20:50:27.389988+0000","last_unstale":"2023-11-28T20:50:28.389875+0000","last_undegraded":"2023-11-28T20:50:28.389875+0000","last_fullsized":"2023-11-28T20:50:28.389875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:08.611663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546188+0000","last_change":"2023-11-28T20:50:27.393617+0000","last_active":"2023-11-28T20:50:28.546188+0000","last_peered":"2023-11-28T20:50:28.546188+0000","last_clean":"2023-11-28T20:50:28.546188+0000","last_became_active":"2023-11-28T20:50:27.393345+0000","last_became_peered":"2023-11-28T20:50:27.393345+0000","last_unstale":"2023-11-28T20:50:28.546188+0000","last_undegraded":"2023-11-28T20:50:28.546188+0000","last_fullsized":"2023-11-28T20:50:28.546188+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:01:44.632228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547210+0000","last_change":"2023-11-28T20:50:27.398180+0000","last_active":"2023-11-28T20:50:28.547210+0000","last_peered":"2023-11-28T20:50:28.547210+0000","last_clean":"2023-11-28T20:50:28.547210+0000","last_became_active":"2023-11-28T20:50:27.397828+0000","last_became_peered":"2023-11-28T20:50:27.397828+0000","last_unstale":"2023-11-28T20:50:28.547210+0000","last_undegraded":"2023-11-28T20:50:28.547210+0000","last_fullsized":"2023-11-28T20:50:28.547210+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:17:56.648761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516615+0000","last_change":"2023-11-28T20:50:27.387066+0000","last_active":"2023-11-28T20:50:28.516615+0000","last_peered":"2023-11-28T20:50:28.516615+0000","last_clean":"2023-11-28T20:50:28.516615+0000","last_became_active":"2023-11-28T20:50:27.386766+0000","last_became_peered":"2023-11-28T20:50:27.386766+0000","last_unstale":"2023-11-28T20:50:28.516615+0000","last_undegraded":"2023-11-28T20:50:28.516615+0000","last_fullsized":"2023-11-28T20:50:28.516615+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:44.386833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388815+0000","last_change":"2023-11-28T20:50:27.386562+0000","last_active":"2023-11-28T20:50:28.388815+0000","last_peered":"2023-11-28T20:50:28.388815+0000","last_clean":"2023-11-28T20:50:28.388815+0000","last_became_active":"2023-11-28T20:50:27.386280+0000","last_became_peered":"2023-11-28T20:50:27.386280+0000","last_unstale":"2023-11-28T20:50:28.388815+0000","last_undegraded":"2023-11-28T20:50:28.388815+0000","last_fullsized":"2023-11-28T20:50:28.388815+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:33.475972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.544636+0000","last_change":"2023-11-28T20:50:27.380417+0000","last_active":"2023-11-28T20:50:28.544636+0000","last_peered":"2023-11-28T20:50:28.544636+0000","last_clean":"2023-11-28T20:50:28.544636+0000","last_became_active":"2023-11-28T20:50:27.379981+0000","last_became_peered":"2023-11-28T20:50:27.379981+0000","last_unstale":"2023-11-28T20:50:28.544636+0000","last_undegraded":"2023-11-28T20:50:28.544636+0000","last_fullsized":"2023-11-28T20:50:28.544636+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:48:16.088499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547619+0000","last_change":"2023-11-28T20:50:27.400415+0000","last_active":"2023-11-28T20:50:28.547619+0000","last_peered":"2023-11-28T20:50:28.547619+0000","last_clean":"2023-11-28T20:50:28.547619+0000","last_became_active":"2023-11-28T20:50:27.400143+0000","last_became_peered":"2023-11-28T20:50:27.400143+0000","last_unstale":"2023-11-28T20:50:28.547619+0000","last_undegraded":"2023-11-28T20:50:28.547619+0000","last_fullsized":"2023-11-28T20:50:28.547619+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:02:12.299296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"87'4","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546800+0000","last_change":"2023-11-28T20:50:26.369068+0000","last_active":"2023-11-28T20:50:28.546800+0000","last_peered":"2023-11-28T20:50:28.546800+0000","last_clean":"2023-11-28T20:50:28.546800+0000","last_became_active":"2023-11-28T20:50:26.368651+0000","last_became_peered":"2023-11-28T20:50:26.368651+0000","last_unstale":"2023-11-28T20:50:28.546800+0000","last_undegraded":"2023-11-28T20:50:28.546800+0000","last_fullsized":"2023-11-28T20:50:28.546800+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:33:09.154021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.390109+0000","last_change":"2023-11-28T20:50:27.390842+0000","last_active":"2023-11-28T20:50:28.390109+0000","last_peered":"2023-11-28T20:50:28.390109+0000","last_clean":"2023-11-28T20:50:28.390109+0000","last_became_active":"2023-11-28T20:50:27.390545+0000","last_became_peered":"2023-11-28T20:50:27.390545+0000","last_unstale":"2023-11-28T20:50:28.390109+0000","last_undegraded":"2023-11-28T20:50:28.390109+0000","last_fullsized":"2023-11-28T20:50:28.390109+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:47:14.631727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545716+0000","last_change":"2023-11-28T20:50:26.370047+0000","last_active":"2023-11-28T20:50:28.545716+0000","last_peered":"2023-11-28T20:50:28.545716+0000","last_clean":"2023-11-28T20:50:28.545716+0000","last_became_active":"2023-11-28T20:50:26.369740+0000","last_became_peered":"2023-11-28T20:50:26.369740+0000","last_unstale":"2023-11-28T20:50:28.545716+0000","last_undegraded":"2023-11-28T20:50:28.545716+0000","last_fullsized":"2023-11-28T20:50:28.545716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:56.711142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545632+0000","last_change":"2023-11-28T20:50:27.377686+0000","last_active":"2023-11-28T20:50:28.545632+0000","last_peered":"2023-11-28T20:50:28.545632+0000","last_clean":"2023-11-28T20:50:28.545632+0000","last_became_active":"2023-11-28T20:50:27.377351+0000","last_became_peered":"2023-11-28T20:50:27.377351+0000","last_unstale":"2023-11-28T20:50:28.545632+0000","last_undegraded":"2023-11-28T20:50:28.545632+0000","last_fullsized":"2023-11-28T20:50:28.545632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:56:19.429557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545643+0000","last_change":"2023-11-28T20:50:26.374085+0000","last_active":"2023-11-28T20:50:28.545643+0000","last_peered":"2023-11-28T20:50:28.545643+0000","last_clean":"2023-11-28T20:50:28.545643+0000","last_became_active":"2023-11-28T20:50:26.373580+0000","last_became_peered":"2023-11-28T20:50:26.373580+0000","last_unstale":"2023-11-28T20:50:28.545643+0000","last_undegraded":"2023-11-28T20:50:28.545643+0000","last_fullsized":"2023-11-28T20:50:28.545643+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:17:55.635545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548984+0000","last_change":"2023-11-28T20:50:27.398396+0000","last_active":"2023-11-28T20:50:28.548984+0000","last_peered":"2023-11-28T20:50:28.548984+0000","last_clean":"2023-11-28T20:50:28.548984+0000","last_became_active":"2023-11-28T20:50:27.397796+0000","last_became_peered":"2023-11-28T20:50:27.397796+0000","last_unstale":"2023-11-28T20:50:28.548984+0000","last_undegraded":"2023-11-28T20:50:28.548984+0000","last_fullsized":"2023-11-28T20:50:28.548984+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:19:40.943469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545279+0000","last_change":"2023-11-28T20:50:26.360972+0000","last_active":"2023-11-28T20:50:28.545279+0000","last_peered":"2023-11-28T20:50:28.545279+0000","last_clean":"2023-11-28T20:50:28.545279+0000","last_became_active":"2023-11-28T20:50:26.359705+0000","last_became_peered":"2023-11-28T20:50:26.359705+0000","last_unstale":"2023-11-28T20:50:28.545279+0000","last_undegraded":"2023-11-28T20:50:28.545279+0000","last_fullsized":"2023-11-28T20:50:28.545279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:19:39.930253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.386724+0000","last_change":"2023-11-28T20:50:27.387327+0000","last_active":"2023-11-28T20:50:28.386724+0000","last_peered":"2023-11-28T20:50:28.386724+0000","last_clean":"2023-11-28T20:50:28.386724+0000","last_became_active":"2023-11-28T20:50:27.386574+0000","last_became_peered":"2023-11-28T20:50:27.386574+0000","last_unstale":"2023-11-28T20:50:28.386724+0000","last_undegraded":"2023-11-28T20:50:28.386724+0000","last_fullsized":"2023-11-28T20:50:28.386724+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:01.106525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388909+0000","last_change":"2023-11-28T20:50:26.362697+0000","last_active":"2023-11-28T20:50:28.388909+0000","last_peered":"2023-11-28T20:50:28.388909+0000","last_clean":"2023-11-28T20:50:28.388909+0000","last_became_active":"2023-11-28T20:50:26.362321+0000","last_became_peered":"2023-11-28T20:50:26.362321+0000","last_unstale":"2023-11-28T20:50:28.388909+0000","last_undegraded":"2023-11-28T20:50:28.388909+0000","last_fullsized":"2023-11-28T20:50:28.388909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:08.320262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545119+0000","last_change":"2023-11-28T20:50:27.392080+0000","last_active":"2023-11-28T20:50:28.545119+0000","last_peered":"2023-11-28T20:50:28.545119+0000","last_clean":"2023-11-28T20:50:28.545119+0000","last_became_active":"2023-11-28T20:50:27.391753+0000","last_became_peered":"2023-11-28T20:50:27.391753+0000","last_unstale":"2023-11-28T20:50:28.545119+0000","last_undegraded":"2023-11-28T20:50:28.545119+0000","last_fullsized":"2023-11-28T20:50:28.545119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:03:15.320718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388566+0000","last_change":"2023-11-28T20:50:26.346818+0000","last_active":"2023-11-28T20:50:28.388566+0000","last_peered":"2023-11-28T20:50:28.388566+0000","last_clean":"2023-11-28T20:50:28.388566+0000","last_became_active":"2023-11-28T20:50:26.346322+0000","last_became_peered":"2023-11-28T20:50:26.346322+0000","last_unstale":"2023-11-28T20:50:28.388566+0000","last_undegraded":"2023-11-28T20:50:28.388566+0000","last_fullsized":"2023-11-28T20:50:28.388566+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:49:58.751907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547778+0000","last_change":"2023-11-28T20:50:27.400481+0000","last_active":"2023-11-28T20:50:28.547778+0000","last_peered":"2023-11-28T20:50:28.547778+0000","last_clean":"2023-11-28T20:50:28.547778+0000","last_became_active":"2023-11-28T20:50:27.400160+0000","last_became_peered":"2023-11-28T20:50:27.400160+0000","last_unstale":"2023-11-28T20:50:28.547778+0000","last_undegraded":"2023-11-28T20:50:28.547778+0000","last_fullsized":"2023-11-28T20:50:28.547778+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:14.643205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388496+0000","last_change":"2023-11-28T20:50:26.353586+0000","last_active":"2023-11-28T20:50:28.388496+0000","last_peered":"2023-11-28T20:50:28.388496+0000","last_clean":"2023-11-28T20:50:28.388496+0000","last_became_active":"2023-11-28T20:50:26.353263+0000","last_became_peered":"2023-11-28T20:50:26.353263+0000","last_unstale":"2023-11-28T20:50:28.388496+0000","last_undegraded":"2023-11-28T20:50:28.388496+0000","last_fullsized":"2023-11-28T20:50:28.388496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:56.711142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388858+0000","last_change":"2023-11-28T20:50:27.401975+0000","last_active":"2023-11-28T20:50:28.388858+0000","last_peered":"2023-11-28T20:50:28.388858+0000","last_clean":"2023-11-28T20:50:28.388858+0000","last_became_active":"2023-11-28T20:50:27.401559+0000","last_became_peered":"2023-11-28T20:50:27.401559+0000","last_unstale":"2023-11-28T20:50:28.388858+0000","last_undegraded":"2023-11-28T20:50:28.388858+0000","last_fullsized":"2023-11-28T20:50:28.388858+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:03:51.702671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546495+0000","last_change":"2023-11-28T20:50:26.358493+0000","last_active":"2023-11-28T20:50:28.546495+0000","last_peered":"2023-11-28T20:50:28.546495+0000","last_clean":"2023-11-28T20:50:28.546495+0000","last_became_active":"2023-11-28T20:50:26.357981+0000","last_became_peered":"2023-11-28T20:50:26.357981+0000","last_unstale":"2023-11-28T20:50:28.546495+0000","last_undegraded":"2023-11-28T20:50:28.546495+0000","last_fullsized":"2023-11-28T20:50:28.546495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:42:57.331725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546186+0000","last_change":"2023-11-28T20:50:27.385221+0000","last_active":"2023-11-28T20:50:28.546186+0000","last_peered":"2023-11-28T20:50:28.546186+0000","last_clean":"2023-11-28T20:50:28.546186+0000","last_became_active":"2023-11-28T20:50:27.384945+0000","last_became_peered":"2023-11-28T20:50:27.384945+0000","last_unstale":"2023-11-28T20:50:28.546186+0000","last_undegraded":"2023-11-28T20:50:28.546186+0000","last_fullsized":"2023-11-28T20:50:28.546186+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:58:14.924703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.549119+0000","last_change":"2023-11-28T20:50:26.377486+0000","last_active":"2023-11-28T20:50:28.549119+0000","last_peered":"2023-11-28T20:50:28.549119+0000","last_clean":"2023-11-28T20:50:28.549119+0000","last_became_active":"2023-11-28T20:50:26.377174+0000","last_became_peered":"2023-11-28T20:50:26.377174+0000","last_unstale":"2023-11-28T20:50:28.549119+0000","last_undegraded":"2023-11-28T20:50:28.549119+0000","last_fullsized":"2023-11-28T20:50:28.549119+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:16.565083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389021+0000","last_change":"2023-11-28T20:50:27.391658+0000","last_active":"2023-11-28T20:50:28.389021+0000","last_peered":"2023-11-28T20:50:28.389021+0000","last_clean":"2023-11-28T20:50:28.389021+0000","last_became_active":"2023-11-28T20:50:27.391350+0000","last_became_peered":"2023-11-28T20:50:27.391350+0000","last_unstale":"2023-11-28T20:50:28.389021+0000","last_undegraded":"2023-11-28T20:50:28.389021+0000","last_fullsized":"2023-11-28T20:50:28.389021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:23.384849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.390133+0000","last_change":"2023-11-28T20:50:26.367836+0000","last_active":"2023-11-28T20:50:28.390133+0000","last_peered":"2023-11-28T20:50:28.390133+0000","last_clean":"2023-11-28T20:50:28.390133+0000","last_became_active":"2023-11-28T20:50:26.367228+0000","last_became_peered":"2023-11-28T20:50:26.367228+0000","last_unstale":"2023-11-28T20:50:28.390133+0000","last_undegraded":"2023-11-28T20:50:28.390133+0000","last_fullsized":"2023-11-28T20:50:28.390133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:54.714495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548787+0000","last_change":"2023-11-28T20:50:27.394955+0000","last_active":"2023-11-28T20:50:28.548787+0000","last_peered":"2023-11-28T20:50:28.548787+0000","last_clean":"2023-11-28T20:50:28.548787+0000","last_became_active":"2023-11-28T20:50:27.394683+0000","last_became_peered":"2023-11-28T20:50:27.394683+0000","last_unstale":"2023-11-28T20:50:28.548787+0000","last_undegraded":"2023-11-28T20:50:28.548787+0000","last_fullsized":"2023-11-28T20:50:28.548787+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:03:07.832708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"87'3","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547890+0000","last_change":"2023-11-28T20:50:26.375793+0000","last_active":"2023-11-28T20:50:28.547890+0000","last_peered":"2023-11-28T20:50:28.547890+0000","last_clean":"2023-11-28T20:50:28.547890+0000","last_became_active":"2023-11-28T20:50:26.375480+0000","last_became_peered":"2023-11-28T20:50:26.375480+0000","last_unstale":"2023-11-28T20:50:28.547890+0000","last_undegraded":"2023-11-28T20:50:28.547890+0000","last_fullsized":"2023-11-28T20:50:28.547890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:14:31.987430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.390057+0000","last_change":"2023-11-28T20:50:27.388584+0000","last_active":"2023-11-28T20:50:28.390057+0000","last_peered":"2023-11-28T20:50:28.390057+0000","last_clean":"2023-11-28T20:50:28.390057+0000","last_became_active":"2023-11-28T20:50:27.388218+0000","last_became_peered":"2023-11-28T20:50:27.388218+0000","last_unstale":"2023-11-28T20:50:28.390057+0000","last_undegraded":"2023-11-28T20:50:28.390057+0000","last_fullsized":"2023-11-28T20:50:28.390057+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:29:10.316623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"87'6","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546372+0000","last_change":"2023-11-28T20:50:26.364988+0000","last_active":"2023-11-28T20:50:28.546372+0000","last_peered":"2023-11-28T20:50:28.546372+0000","last_clean":"2023-11-28T20:50:28.546372+0000","last_became_active":"2023-11-28T20:50:26.364668+0000","last_became_peered":"2023-11-28T20:50:26.364668+0000","last_unstale":"2023-11-28T20:50:28.546372+0000","last_undegraded":"2023-11-28T20:50:28.546372+0000","last_fullsized":"2023-11-28T20:50:28.546372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:03:47.992576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388915+0000","last_change":"2023-11-28T20:50:27.391981+0000","last_active":"2023-11-28T20:50:28.388915+0000","last_peered":"2023-11-28T20:50:28.388915+0000","last_clean":"2023-11-28T20:50:28.388915+0000","last_became_active":"2023-11-28T20:50:27.391721+0000","last_became_peered":"2023-11-28T20:50:27.391721+0000","last_unstale":"2023-11-28T20:50:28.388915+0000","last_undegraded":"2023-11-28T20:50:28.388915+0000","last_fullsized":"2023-11-28T20:50:28.388915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:36:07.688394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547086+0000","last_change":"2023-11-28T20:50:26.373562+0000","last_active":"2023-11-28T20:50:28.547086+0000","last_peered":"2023-11-28T20:50:28.547086+0000","last_clean":"2023-11-28T20:50:28.547086+0000","last_became_active":"2023-11-28T20:50:26.372795+0000","last_became_peered":"2023-11-28T20:50:26.372795+0000","last_unstale":"2023-11-28T20:50:28.547086+0000","last_undegraded":"2023-11-28T20:50:28.547086+0000","last_fullsized":"2023-11-28T20:50:28.547086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:48:27.912984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516571+0000","last_change":"2023-11-28T20:50:27.390734+0000","last_active":"2023-11-28T20:50:28.516571+0000","last_peered":"2023-11-28T20:50:28.516571+0000","last_clean":"2023-11-28T20:50:28.516571+0000","last_became_active":"2023-11-28T20:50:27.390372+0000","last_became_peered":"2023-11-28T20:50:27.390372+0000","last_unstale":"2023-11-28T20:50:28.516571+0000","last_undegraded":"2023-11-28T20:50:28.516571+0000","last_fullsized":"2023-11-28T20:50:28.516571+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:02.602202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516395+0000","last_change":"2023-11-28T20:50:26.369735+0000","last_active":"2023-11-28T20:50:28.516395+0000","last_peered":"2023-11-28T20:50:28.516395+0000","last_clean":"2023-11-28T20:50:28.516395+0000","last_became_active":"2023-11-28T20:50:26.369425+0000","last_became_peered":"2023-11-28T20:50:26.369425+0000","last_unstale":"2023-11-28T20:50:28.516395+0000","last_undegraded":"2023-11-28T20:50:28.516395+0000","last_fullsized":"2023-11-28T20:50:28.516395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:07.750976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548235+0000","last_change":"2023-11-28T20:50:27.383192+0000","last_active":"2023-11-28T20:50:28.548235+0000","last_peered":"2023-11-28T20:50:28.548235+0000","last_clean":"2023-11-28T20:50:28.548235+0000","last_became_active":"2023-11-28T20:50:27.381824+0000","last_became_peered":"2023-11-28T20:50:27.381824+0000","last_unstale":"2023-11-28T20:50:28.548235+0000","last_undegraded":"2023-11-28T20:50:28.548235+0000","last_fullsized":"2023-11-28T20:50:28.548235+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:36:31.251810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546354+0000","last_change":"2023-11-28T20:50:26.372157+0000","last_active":"2023-11-28T20:50:28.546354+0000","last_peered":"2023-11-28T20:50:28.546354+0000","last_clean":"2023-11-28T20:50:28.546354+0000","last_became_active":"2023-11-28T20:50:26.371068+0000","last_became_peered":"2023-11-28T20:50:26.371068+0000","last_unstale":"2023-11-28T20:50:28.546354+0000","last_undegraded":"2023-11-28T20:50:28.546354+0000","last_fullsized":"2023-11-28T20:50:28.546354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:49:59.121264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545693+0000","last_change":"2023-11-28T20:50:27.392550+0000","last_active":"2023-11-28T20:50:28.545693+0000","last_peered":"2023-11-28T20:50:28.545693+0000","last_clean":"2023-11-28T20:50:28.545693+0000","last_became_active":"2023-11-28T20:50:27.392018+0000","last_became_peered":"2023-11-28T20:50:27.392018+0000","last_unstale":"2023-11-28T20:50:28.545693+0000","last_undegraded":"2023-11-28T20:50:28.545693+0000","last_fullsized":"2023-11-28T20:50:28.545693+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:35.366099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"87'8","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388712+0000","last_change":"2023-11-28T20:50:26.371646+0000","last_active":"2023-11-28T20:50:28.388712+0000","last_peered":"2023-11-28T20:50:28.388712+0000","last_clean":"2023-11-28T20:50:28.388712+0000","last_became_active":"2023-11-28T20:50:26.371248+0000","last_became_peered":"2023-11-28T20:50:26.371248+0000","last_unstale":"2023-11-28T20:50:28.388712+0000","last_undegraded":"2023-11-28T20:50:28.388712+0000","last_fullsized":"2023-11-28T20:50:28.388712+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:56:11.241090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547281+0000","last_change":"2023-11-28T20:50:27.397537+0000","last_active":"2023-11-28T20:50:28.547281+0000","last_peered":"2023-11-28T20:50:28.547281+0000","last_clean":"2023-11-28T20:50:28.547281+0000","last_became_active":"2023-11-28T20:50:27.397220+0000","last_became_peered":"2023-11-28T20:50:27.397220+0000","last_unstale":"2023-11-28T20:50:28.547281+0000","last_undegraded":"2023-11-28T20:50:28.547281+0000","last_fullsized":"2023-11-28T20:50:28.547281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:36.701581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517363+0000","last_change":"2023-11-28T20:50:26.372117+0000","last_active":"2023-11-28T20:50:28.517363+0000","last_peered":"2023-11-28T20:50:28.517363+0000","last_clean":"2023-11-28T20:50:28.517363+0000","last_became_active":"2023-11-28T20:50:26.370855+0000","last_became_peered":"2023-11-28T20:50:26.370855+0000","last_unstale":"2023-11-28T20:50:28.517363+0000","last_undegraded":"2023-11-28T20:50:28.517363+0000","last_fullsized":"2023-11-28T20:50:28.517363+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:10:44.042782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548047+0000","last_change":"2023-11-28T20:50:27.403944+0000","last_active":"2023-11-28T20:50:28.548047+0000","last_peered":"2023-11-28T20:50:28.548047+0000","last_clean":"2023-11-28T20:50:28.548047+0000","last_became_active":"2023-11-28T20:50:27.403709+0000","last_became_peered":"2023-11-28T20:50:27.403709+0000","last_unstale":"2023-11-28T20:50:28.548047+0000","last_undegraded":"2023-11-28T20:50:28.548047+0000","last_fullsized":"2023-11-28T20:50:28.548047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:43:56.162583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548774+0000","last_change":"2023-11-28T20:50:26.377645+0000","last_active":"2023-11-28T20:50:28.548774+0000","last_peered":"2023-11-28T20:50:28.548774+0000","last_clean":"2023-11-28T20:50:28.548774+0000","last_became_active":"2023-11-28T20:50:26.377303+0000","last_became_peered":"2023-11-28T20:50:26.377303+0000","last_unstale":"2023-11-28T20:50:28.548774+0000","last_undegraded":"2023-11-28T20:50:28.548774+0000","last_fullsized":"2023-11-28T20:50:28.548774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:07.750976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.390257+0000","last_change":"2023-11-28T20:50:27.401010+0000","last_active":"2023-11-28T20:50:28.390257+0000","last_peered":"2023-11-28T20:50:28.390257+0000","last_clean":"2023-11-28T20:50:28.390257+0000","last_became_active":"2023-11-28T20:50:27.394283+0000","last_became_peered":"2023-11-28T20:50:27.394283+0000","last_unstale":"2023-11-28T20:50:28.390257+0000","last_undegraded":"2023-11-28T20:50:28.390257+0000","last_fullsized":"2023-11-28T20:50:28.390257+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:27:04.266720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548581+0000","last_change":"2023-11-28T20:50:26.375882+0000","last_active":"2023-11-28T20:50:28.548581+0000","last_peered":"2023-11-28T20:50:28.548581+0000","last_clean":"2023-11-28T20:50:28.548581+0000","last_became_active":"2023-11-28T20:50:26.375573+0000","last_became_peered":"2023-11-28T20:50:26.375573+0000","last_unstale":"2023-11-28T20:50:28.548581+0000","last_undegraded":"2023-11-28T20:50:28.548581+0000","last_fullsized":"2023-11-28T20:50:28.548581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:11:39.294283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548621+0000","last_change":"2023-11-28T20:50:27.384257+0000","last_active":"2023-11-28T20:50:28.548621+0000","last_peered":"2023-11-28T20:50:28.548621+0000","last_clean":"2023-11-28T20:50:28.548621+0000","last_became_active":"2023-11-28T20:50:27.383983+0000","last_became_peered":"2023-11-28T20:50:27.383983+0000","last_unstale":"2023-11-28T20:50:28.548621+0000","last_undegraded":"2023-11-28T20:50:28.548621+0000","last_fullsized":"2023-11-28T20:50:28.548621+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:07:17.175256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388882+0000","last_change":"2023-11-28T20:50:26.361958+0000","last_active":"2023-11-28T20:50:28.388882+0000","last_peered":"2023-11-28T20:50:28.388882+0000","last_clean":"2023-11-28T20:50:28.388882+0000","last_became_active":"2023-11-28T20:50:26.361732+0000","last_became_peered":"2023-11-28T20:50:26.361732+0000","last_unstale":"2023-11-28T20:50:28.388882+0000","last_undegraded":"2023-11-28T20:50:28.388882+0000","last_fullsized":"2023-11-28T20:50:28.388882+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:51:45.548184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.544688+0000","last_change":"2023-11-28T20:50:27.380245+0000","last_active":"2023-11-28T20:50:28.544688+0000","last_peered":"2023-11-28T20:50:28.544688+0000","last_clean":"2023-11-28T20:50:28.544688+0000","last_became_active":"2023-11-28T20:50:27.379633+0000","last_became_peered":"2023-11-28T20:50:27.379633+0000","last_unstale":"2023-11-28T20:50:28.544688+0000","last_undegraded":"2023-11-28T20:50:28.544688+0000","last_fullsized":"2023-11-28T20:50:28.544688+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:41.319338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545169+0000","last_change":"2023-11-28T20:50:26.359911+0000","last_active":"2023-11-28T20:50:28.545169+0000","last_peered":"2023-11-28T20:50:28.545169+0000","last_clean":"2023-11-28T20:50:28.545169+0000","last_became_active":"2023-11-28T20:50:26.359378+0000","last_became_peered":"2023-11-28T20:50:26.359378+0000","last_unstale":"2023-11-28T20:50:28.545169+0000","last_undegraded":"2023-11-28T20:50:28.545169+0000","last_fullsized":"2023-11-28T20:50:28.545169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:35.688365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516523+0000","last_change":"2023-11-28T20:50:27.390044+0000","last_active":"2023-11-28T20:50:28.516523+0000","last_peered":"2023-11-28T20:50:28.516523+0000","last_clean":"2023-11-28T20:50:28.516523+0000","last_became_active":"2023-11-28T20:50:27.382173+0000","last_became_peered":"2023-11-28T20:50:27.382173+0000","last_unstale":"2023-11-28T20:50:28.516523+0000","last_undegraded":"2023-11-28T20:50:28.516523+0000","last_fullsized":"2023-11-28T20:50:28.516523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:42:48.797967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516665+0000","last_change":"2023-11-28T20:50:26.369602+0000","last_active":"2023-11-28T20:50:28.516665+0000","last_peered":"2023-11-28T20:50:28.516665+0000","last_clean":"2023-11-28T20:50:28.516665+0000","last_became_active":"2023-11-28T20:50:26.369273+0000","last_became_peered":"2023-11-28T20:50:26.369273+0000","last_unstale":"2023-11-28T20:50:28.516665+0000","last_undegraded":"2023-11-28T20:50:28.516665+0000","last_fullsized":"2023-11-28T20:50:28.516665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:05:30.047531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548128+0000","last_change":"2023-11-28T20:50:27.391826+0000","last_active":"2023-11-28T20:50:28.548128+0000","last_peered":"2023-11-28T20:50:28.548128+0000","last_clean":"2023-11-28T20:50:28.548128+0000","last_became_active":"2023-11-28T20:50:27.391488+0000","last_became_peered":"2023-11-28T20:50:27.391488+0000","last_unstale":"2023-11-28T20:50:28.548128+0000","last_undegraded":"2023-11-28T20:50:28.548128+0000","last_fullsized":"2023-11-28T20:50:28.548128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:31:29.801467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.544889+0000","last_change":"2023-11-28T20:50:26.368382+0000","last_active":"2023-11-28T20:50:28.544889+0000","last_peered":"2023-11-28T20:50:28.544889+0000","last_clean":"2023-11-28T20:50:28.544889+0000","last_became_active":"2023-11-28T20:50:26.367894+0000","last_became_peered":"2023-11-28T20:50:26.367894+0000","last_unstale":"2023-11-28T20:50:28.544889+0000","last_undegraded":"2023-11-28T20:50:28.544889+0000","last_fullsized":"2023-11-28T20:50:28.544889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:20.767083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545161+0000","last_change":"2023-11-28T20:50:27.397610+0000","last_active":"2023-11-28T20:50:28.545161+0000","last_peered":"2023-11-28T20:50:28.545161+0000","last_clean":"2023-11-28T20:50:28.545161+0000","last_became_active":"2023-11-28T20:50:27.397008+0000","last_became_peered":"2023-11-28T20:50:27.397008+0000","last_unstale":"2023-11-28T20:50:28.545161+0000","last_undegraded":"2023-11-28T20:50:28.545161+0000","last_fullsized":"2023-11-28T20:50:28.545161+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:09.333478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517662+0000","last_change":"2023-11-28T20:50:26.372742+0000","last_active":"2023-11-28T20:50:28.517662+0000","last_peered":"2023-11-28T20:50:28.517662+0000","last_clean":"2023-11-28T20:50:28.517662+0000","last_became_active":"2023-11-28T20:50:26.372474+0000","last_became_peered":"2023-11-28T20:50:26.372474+0000","last_unstale":"2023-11-28T20:50:28.517662+0000","last_undegraded":"2023-11-28T20:50:28.517662+0000","last_fullsized":"2023-11-28T20:50:28.517662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:16.565083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.387412+0000","last_change":"2023-11-28T20:50:27.393641+0000","last_active":"2023-11-28T20:50:28.387412+0000","last_peered":"2023-11-28T20:50:28.387412+0000","last_clean":"2023-11-28T20:50:28.387412+0000","last_became_active":"2023-11-28T20:50:27.393361+0000","last_became_peered":"2023-11-28T20:50:27.393361+0000","last_unstale":"2023-11-28T20:50:28.387412+0000","last_undegraded":"2023-11-28T20:50:28.387412+0000","last_fullsized":"2023-11-28T20:50:28.387412+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:09:40.276726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517642+0000","last_change":"2023-11-28T20:50:26.363037+0000","last_active":"2023-11-28T20:50:28.517642+0000","last_peered":"2023-11-28T20:50:28.517642+0000","last_clean":"2023-11-28T20:50:28.517642+0000","last_became_active":"2023-11-28T20:50:26.361957+0000","last_became_peered":"2023-11-28T20:50:26.361957+0000","last_unstale":"2023-11-28T20:50:28.517642+0000","last_undegraded":"2023-11-28T20:50:28.517642+0000","last_fullsized":"2023-11-28T20:50:28.517642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:36:06.716305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517421+0000","last_change":"2023-11-28T20:50:27.402230+0000","last_active":"2023-11-28T20:50:28.517421+0000","last_peered":"2023-11-28T20:50:28.517421+0000","last_clean":"2023-11-28T20:50:28.517421+0000","last_became_active":"2023-11-28T20:50:27.400924+0000","last_became_peered":"2023-11-28T20:50:27.400924+0000","last_unstale":"2023-11-28T20:50:28.517421+0000","last_undegraded":"2023-11-28T20:50:28.517421+0000","last_fullsized":"2023-11-28T20:50:28.517421+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:55:42.771188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545742+0000","last_change":"2023-11-28T20:50:26.369906+0000","last_active":"2023-11-28T20:50:28.545742+0000","last_peered":"2023-11-28T20:50:28.545742+0000","last_clean":"2023-11-28T20:50:28.545742+0000","last_became_active":"2023-11-28T20:50:26.369573+0000","last_became_peered":"2023-11-28T20:50:26.369573+0000","last_unstale":"2023-11-28T20:50:28.545742+0000","last_undegraded":"2023-11-28T20:50:28.545742+0000","last_fullsized":"2023-11-28T20:50:28.545742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:55:41.757972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548885+0000","last_change":"2023-11-28T20:50:27.396055+0000","last_active":"2023-11-28T20:50:28.548885+0000","last_peered":"2023-11-28T20:50:28.548885+0000","last_clean":"2023-11-28T20:50:28.548885+0000","last_became_active":"2023-11-28T20:50:27.395785+0000","last_became_peered":"2023-11-28T20:50:27.395785+0000","last_unstale":"2023-11-28T20:50:28.548885+0000","last_undegraded":"2023-11-28T20:50:28.548885+0000","last_fullsized":"2023-11-28T20:50:28.548885+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:33.475972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546368+0000","last_change":"2023-11-28T20:50:26.364878+0000","last_active":"2023-11-28T20:50:28.546368+0000","last_peered":"2023-11-28T20:50:28.546368+0000","last_clean":"2023-11-28T20:50:28.546368+0000","last_became_active":"2023-11-28T20:50:26.364592+0000","last_became_peered":"2023-11-28T20:50:26.364592+0000","last_unstale":"2023-11-28T20:50:28.546368+0000","last_undegraded":"2023-11-28T20:50:28.546368+0000","last_fullsized":"2023-11-28T20:50:28.546368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:01.616206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546438+0000","last_change":"2023-11-28T20:50:27.392401+0000","last_active":"2023-11-28T20:50:28.546438+0000","last_peered":"2023-11-28T20:50:28.546438+0000","last_clean":"2023-11-28T20:50:28.546438+0000","last_became_active":"2023-11-28T20:50:27.392141+0000","last_became_peered":"2023-11-28T20:50:27.392141+0000","last_unstale":"2023-11-28T20:50:28.546438+0000","last_undegraded":"2023-11-28T20:50:28.546438+0000","last_fullsized":"2023-11-28T20:50:28.546438+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:56:12.254306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547290+0000","last_change":"2023-11-28T20:50:26.374351+0000","last_active":"2023-11-28T20:50:28.547290+0000","last_peered":"2023-11-28T20:50:28.547290+0000","last_clean":"2023-11-28T20:50:28.547290+0000","last_became_active":"2023-11-28T20:50:26.373855+0000","last_became_peered":"2023-11-28T20:50:26.373855+0000","last_unstale":"2023-11-28T20:50:28.547290+0000","last_undegraded":"2023-11-28T20:50:28.547290+0000","last_fullsized":"2023-11-28T20:50:28.547290+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:01.588986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388915+0000","last_change":"2023-11-28T20:50:27.382619+0000","last_active":"2023-11-28T20:50:28.388915+0000","last_peered":"2023-11-28T20:50:28.388915+0000","last_clean":"2023-11-28T20:50:28.388915+0000","last_became_active":"2023-11-28T20:50:27.382310+0000","last_became_peered":"2023-11-28T20:50:27.382310+0000","last_unstale":"2023-11-28T20:50:28.388915+0000","last_undegraded":"2023-11-28T20:50:28.388915+0000","last_fullsized":"2023-11-28T20:50:28.388915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:43:21.674252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388685+0000","last_change":"2023-11-28T20:50:26.358481+0000","last_active":"2023-11-28T20:50:28.388685+0000","last_peered":"2023-11-28T20:50:28.388685+0000","last_clean":"2023-11-28T20:50:28.388685+0000","last_became_active":"2023-11-28T20:50:26.357789+0000","last_became_peered":"2023-11-28T20:50:26.357789+0000","last_unstale":"2023-11-28T20:50:28.388685+0000","last_undegraded":"2023-11-28T20:50:28.388685+0000","last_fullsized":"2023-11-28T20:50:28.388685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:47:26.140403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545329+0000","last_change":"2023-11-28T20:50:27.394510+0000","last_active":"2023-11-28T20:50:28.545329+0000","last_peered":"2023-11-28T20:50:28.545329+0000","last_clean":"2023-11-28T20:50:28.545329+0000","last_became_active":"2023-11-28T20:50:27.391591+0000","last_became_peered":"2023-11-28T20:50:27.391591+0000","last_unstale":"2023-11-28T20:50:28.545329+0000","last_undegraded":"2023-11-28T20:50:28.545329+0000","last_fullsized":"2023-11-28T20:50:28.545329+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:47:28.185099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388645+0000","last_change":"2023-11-28T20:50:26.353717+0000","last_active":"2023-11-28T20:50:28.388645+0000","last_peered":"2023-11-28T20:50:28.388645+0000","last_clean":"2023-11-28T20:50:28.388645+0000","last_became_active":"2023-11-28T20:50:26.353299+0000","last_became_peered":"2023-11-28T20:50:26.353299+0000","last_unstale":"2023-11-28T20:50:28.388645+0000","last_undegraded":"2023-11-28T20:50:28.388645+0000","last_fullsized":"2023-11-28T20:50:28.388645+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:40.306123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.386232+0000","last_change":"2023-11-28T20:50:27.387954+0000","last_active":"2023-11-28T20:50:28.386232+0000","last_peered":"2023-11-28T20:50:28.386232+0000","last_clean":"2023-11-28T20:50:28.386232+0000","last_became_active":"2023-11-28T20:50:27.387669+0000","last_became_peered":"2023-11-28T20:50:27.387669+0000","last_unstale":"2023-11-28T20:50:28.386232+0000","last_undegraded":"2023-11-28T20:50:28.386232+0000","last_fullsized":"2023-11-28T20:50:28.386232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:28:47.506318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.544792+0000","last_change":"2023-11-28T20:50:26.358155+0000","last_active":"2023-11-28T20:50:28.544792+0000","last_peered":"2023-11-28T20:50:28.544792+0000","last_clean":"2023-11-28T20:50:28.544792+0000","last_became_active":"2023-11-28T20:50:26.357813+0000","last_became_peered":"2023-11-28T20:50:26.357813+0000","last_unstale":"2023-11-28T20:50:28.544792+0000","last_undegraded":"2023-11-28T20:50:28.544792+0000","last_fullsized":"2023-11-28T20:50:28.544792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:27:03.253504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548493+0000","last_change":"2023-11-28T20:50:27.384864+0000","last_active":"2023-11-28T20:50:28.548493+0000","last_peered":"2023-11-28T20:50:28.548493+0000","last_clean":"2023-11-28T20:50:28.548493+0000","last_became_active":"2023-11-28T20:50:27.384565+0000","last_became_peered":"2023-11-28T20:50:27.384565+0000","last_unstale":"2023-11-28T20:50:28.548493+0000","last_undegraded":"2023-11-28T20:50:28.548493+0000","last_fullsized":"2023-11-28T20:50:28.548493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:55.727711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388277+0000","last_change":"2023-11-28T20:50:26.360824+0000","last_active":"2023-11-28T20:50:28.388277+0000","last_peered":"2023-11-28T20:50:28.388277+0000","last_clean":"2023-11-28T20:50:28.388277+0000","last_became_active":"2023-11-28T20:50:26.360330+0000","last_became_peered":"2023-11-28T20:50:26.360330+0000","last_unstale":"2023-11-28T20:50:28.388277+0000","last_undegraded":"2023-11-28T20:50:28.388277+0000","last_fullsized":"2023-11-28T20:50:28.388277+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:48:15.075283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":389,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-11-28T20:50:27.365265+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:50:27.365265+0000","last_peered":"2023-11-28T20:50:27.365265+0000","last_clean":"2023-11-28T20:50:27.365265+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:50:27.365265+0000","last_undegraded":"2023-11-28T20:50:27.365265+0000","last_fullsized":"2023-11-28T20:50:27.365265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575048,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59196,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312644,"statfs":{"total":96636764160,"available":96576147456,"internally_reserved":0,"allocated":1679360,"data_stored":1126689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1919999999999999,"15min":1.194},"min":{"1min":0.79700000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.722,"5min":1.722,"15min":2.23},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.131,"5min":1.153,"15min":1.1799999999999999},"min":{"1min":0.84299999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5169999999999999,"5min":1.681,"15min":1.726},"last":1.141}]},{"osd":1,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1830000000000001,"15min":1.1850000000000001},"min":{"1min":0.83099999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.579,"5min":1.744,"15min":1.744},"last":1.24},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.92100000000000004,"5min":0.624,"15min":0.624},"max":{"1min":1.6679999999999999,"5min":1.835,"15min":2.3599999999999999},"last":1.085}]},{"osd":2,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.2949999999999999,"15min":1.3029999999999999},"min":{"1min":0.97199999999999998,"5min":0.80300000000000005,"15min":0.58799999999999997},"max":{"1min":1.655,"5min":1.8300000000000001,"15min":1.9359999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.292,"15min":1.2869999999999999},"min":{"1min":0.96899999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.675,"5min":1.847,"15min":2.137},"last":1.4770000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2649999999999999,"15min":1.276},"min":{"1min":0.73199999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.577,"5min":1.7729999999999999,"15min":1.8460000000000001},"last":1.9950000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.403,"15min":1.4059999999999999},"min":{"1min":1.1579999999999999,"5min":0.88300000000000001,"15min":0.68300000000000005},"max":{"1min":1.956,"5min":1.956,"15min":2.4220000000000002},"last":1.5189999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.2629999999999999,"15min":1.264},"min":{"1min":0.83999999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7649999999999999,"5min":1.7989999999999999,"15min":1.919},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3200000000000001,"15min":1.329},"min":{"1min":0.93000000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.794,"5min":1.863,"15min":2.0049999999999999},"last":1.8560000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":0.97199999999999998,"5min":0.86499999999999999,"15min":0.67900000000000005},"max":{"1min":1.891,"5min":1.9219999999999999,"15min":2.5449999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.47,"15min":1.46},"min":{"1min":1.1160000000000001,"5min":0.73499999999999999,"15min":0.72199999999999998},"max":{"1min":1.931,"5min":1.9550000000000001,"15min":2.1000000000000001},"last":1.6879999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5649999999999999,"15min":1.548},"min":{"1min":1.1779999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.2389999999999999},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4830000000000001,"15min":1.488},"min":{"1min":1.056,"5min":0.94099999999999995,"15min":0.90800000000000003},"max":{"1min":1.9139999999999999,"5min":1.9870000000000001,"15min":2.3079999999999998},"last":2.0840000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575048,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32232,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339608,"statfs":{"total":96636764160,"available":96603758592,"internally_reserved":0,"allocated":1073152,"data_stored":535579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1040000000000001,"15min":1.1140000000000001},"min":{"1min":0.71799999999999997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.125,"15min":1.1319999999999999},"min":{"1min":0.73099999999999998,"5min":0.63500000000000001,"15min":0.627},"max":{"1min":1.452,"5min":1.7549999999999999,"15min":2.0739999999999998},"last":0.99299999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2130000000000001,"15min":1.2569999999999999},"min":{"1min":0.84399999999999997,"5min":0.67200000000000004,"15min":0.52200000000000002},"max":{"1min":1.728,"5min":1.728,"15min":1.8069999999999999},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.359,"15min":1.3700000000000001},"min":{"1min":0.97199999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6850000000000001,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":1.369}]},{"osd":2,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.2709999999999999,"15min":1.2669999999999999},"min":{"1min":0.89400000000000002,"5min":0.71099999999999997,"15min":0.69799999999999995},"max":{"1min":1.631,"5min":1.6879999999999999,"15min":1.71},"last":1.302},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.355,"15min":1.361},"min":{"1min":0.84999999999999998,"5min":0.73199999999999998,"15min":0.60299999999999998},"max":{"1min":1.6439999999999999,"5min":1.851,"15min":1.851},"last":1.0800000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.2729999999999999,"15min":1.3},"min":{"1min":0.96799999999999997,"5min":0.70099999999999996,"15min":0.68799999999999994},"max":{"1min":1.6200000000000001,"5min":1.821,"15min":2.222},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2549999999999999,"15min":1.2669999999999999},"min":{"1min":0.95399999999999996,"5min":0.83899999999999997,"15min":0.751},"max":{"1min":1.635,"5min":1.6599999999999999,"15min":1.7949999999999999},"last":1.4279999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2509999999999999,"15min":1.2729999999999999},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.704,"5min":1.849,"15min":2.7290000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.286,"15min":1.329},"min":{"1min":0.84099999999999997,"5min":0.80800000000000005,"15min":0.76400000000000001},"max":{"1min":1.752,"5min":2.2010000000000001,"15min":3.2519999999999998},"last":1.2210000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3100000000000001,"15min":1.325},"min":{"1min":0.96999999999999997,"5min":0.754,"15min":0.62},"max":{"1min":1.714,"5min":1.714,"15min":2.0539999999999998},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5129999999999999,"15min":1.5469999999999999},"min":{"1min":1.2709999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.804,"5min":1.9179999999999999,"15min":3.0310000000000001},"last":1.476}]},{"osd":7,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.45,"15min":1.494},"min":{"1min":1.014,"5min":0.97799999999999998,"15min":0.86499999999999999},"max":{"1min":1.8080000000000001,"5min":2.1120000000000001,"15min":3.379},"last":1.627},{"interface":"front","average":{"1min":1.522,"5min":1.4490000000000001,"15min":1.462},"min":{"1min":1.0669999999999999,"5min":0.93300000000000005,"15min":0.84599999999999997},"max":{"1min":1.7669999999999999,"5min":1.8859999999999999,"15min":2.0739999999999998},"last":1.9630000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575048,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1073152,"data_stored":535567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.157,"15min":1.1719999999999999},"min":{"1min":0.81999999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6970000000000001,"5min":1.702,"15min":1.768},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1599999999999999,"15min":1.2},"min":{"1min":0.88500000000000001,"5min":0.441,"15min":0.441},"max":{"1min":1.631,"5min":1.774,"15min":1.996},"last":1.3260000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1459999999999999,"15min":1.1519999999999999},"min":{"1min":0.90500000000000003,"5min":0.57399999999999995,"15min":0.48899999999999999},"max":{"1min":1.548,"5min":1.768,"15min":1.784},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.194,"15min":1.2170000000000001},"min":{"1min":0.88400000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.956},"last":1.1599999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2609999999999999,"15min":1.2649999999999999},"min":{"1min":0.91400000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.7130000000000001,"5min":1.8100000000000001,"15min":1.901},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2789999999999999,"15min":1.292},"min":{"1min":0.76700000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.611,"5min":1.7310000000000001,"15min":1.958},"last":1.226}]},{"osd":3,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3129999999999999,"15min":1.3280000000000001},"min":{"1min":0.93999999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.7290000000000001,"5min":1.9350000000000001,"15min":2.298},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.2529999999999999,"15min":1.2689999999999999},"min":{"1min":1.0009999999999999,"5min":0.64000000000000001,"15min":0.54600000000000004},"max":{"1min":1.667,"5min":1.667,"15min":2.1040000000000001},"last":1.3620000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.2989999999999999,"15min":1.298},"min":{"1min":0.91300000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":2.2429999999999999},"last":0.875},{"interface":"front","average":{"1min":1.494,"5min":1.429,"15min":1.427},"min":{"1min":0.93600000000000005,"5min":0.73399999999999999,"15min":0.68100000000000005},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.4850000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3109999999999999,"15min":1.3240000000000001},"min":{"1min":0.871,"5min":0.5,"15min":0.5},"max":{"1min":1.6180000000000001,"5min":1.845,"15min":2.1600000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.4670000000000001,"15min":1.466},"min":{"1min":1.2829999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.7809999999999999,"5min":1.806,"15min":2.339},"last":1.109}]},{"osd":7,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.397,"15min":1.4039999999999999},"min":{"1min":0.97599999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.988,"5min":1.988,"15min":2.0800000000000001},"last":1.54},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.486,"15min":1.48},"min":{"1min":1.0509999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7589999999999999,"5min":1.929,"15min":2.3849999999999998},"last":1.456}]}]},{"osd":4,"up_from":13,"seq":55834575049,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1081344,"data_stored":537157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.139,"15min":1.129},"min":{"1min":0.80300000000000005,"5min":0.66000000000000003,"15min":0.64400000000000002},"max":{"1min":1.4359999999999999,"5min":1.6659999999999999,"15min":1.879},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1339999999999999,"15min":1.1180000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6850000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.117}]},{"osd":1,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.29,"15min":1.3100000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.74299999999999999},"max":{"1min":1.609,"5min":1.7490000000000001,"15min":1.923},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.375,"15min":1.3520000000000001},"min":{"1min":1.026,"5min":0.86599999999999999,"15min":0.76700000000000002},"max":{"1min":1.768,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.94799999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.232,"15min":1.2110000000000001},"min":{"1min":1.0229999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.589,"5min":1.71,"15min":1.71},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.278,"15min":1.2949999999999999},"min":{"1min":1.083,"5min":0.77400000000000002,"15min":0.77200000000000002},"max":{"1min":1.5189999999999999,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.4970000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.294,"15min":1.2869999999999999},"min":{"1min":0.97499999999999998,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.76,"5min":2.157,"15min":2.157},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.361,"15min":1.377},"min":{"1min":1.087,"5min":0.876,"15min":0.67900000000000005},"max":{"1min":1.718,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.552}]},{"osd":5,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.3260000000000001,"15min":1.278},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.65800000000000003},"max":{"1min":1.655,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.649},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3620000000000001,"15min":1.2969999999999999},"min":{"1min":0.84099999999999997,"5min":0.83799999999999997,"15min":0.69299999999999995},"max":{"1min":2.0579999999999998,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.347}]},{"osd":6,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3979999999999999,"15min":1.401},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.628},"max":{"1min":1.8049999999999999,"5min":2.04,"15min":2.04},"last":1.629},{"interface":"front","average":{"1min":1.526,"5min":1.5780000000000001,"15min":1.5489999999999999},"min":{"1min":1.3120000000000001,"5min":1.0369999999999999,"15min":0.80600000000000005},"max":{"1min":1.8380000000000001,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":1.7290000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4930000000000001,"15min":1.45},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.76200000000000001},"max":{"1min":1.9570000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.538,"15min":1.526},"min":{"1min":1.208,"5min":1.0349999999999999,"15min":0.72299999999999998},"max":{"1min":1.8580000000000001,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.6819999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40552,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":1069056,"data_stored":535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.88700000000000001,"15min":0.91400000000000003},"min":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.296,"5min":4.7210000000000001,"15min":4.7210000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0069999999999999,"15min":1.083},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":3.968},"last":1.4870000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97299999999999998,"15min":0.98899999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.44600000000000001},"max":{"1min":1.593,"5min":2.8460000000000001,"15min":3.9180000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.173,"15min":1.2110000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7190000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.0600000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2470000000000001,"15min":1.2609999999999999},"min":{"1min":0.53100000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.8169999999999999,"5min":3.444,"15min":4.6130000000000004},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.276,"5min":1.3129999999999999,"15min":1.3460000000000001},"min":{"1min":0.63900000000000001,"5min":0.58099999999999996,"15min":0.54400000000000004},"max":{"1min":1.7130000000000001,"5min":3.7050000000000001,"15min":4.5759999999999996},"last":1.643}]},{"osd":4,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3149999999999999,"15min":1.3480000000000001},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.72299999999999998},"max":{"1min":1.7889999999999999,"5min":1.8779999999999999,"15min":4.1779999999999999},"last":1.204},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3500000000000001,"15min":1.3939999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.8160000000000001,"5min":1.891,"15min":4.492},"last":1.1579999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4390000000000001,"15min":1.4590000000000001},"min":{"1min":0.84399999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7250000000000001,"5min":3.7730000000000001,"15min":4.7169999999999996},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.395,"15min":1.4419999999999999},"min":{"1min":0.66800000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0110000000000001,"5min":2.2429999999999999,"15min":4.2889999999999997},"last":1.607}]},{"osd":6,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.4330000000000001,"15min":1.4930000000000001},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.72799999999999998},"max":{"1min":1.8899999999999999,"5min":2.3100000000000001,"15min":5.2279999999999998},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.556,"15min":1.5960000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.855,"5min":3.661,"15min":6.9740000000000002},"last":1.5369999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.484,"15min":1.5489999999999999},"min":{"1min":0.92400000000000004,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.907,"5min":2.1469999999999998,"15min":4.0819999999999999},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.544,"15min":1.609},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.76200000000000001},"max":{"1min":1.976,"5min":2.3919999999999999,"15min":4.1399999999999997},"last":1.2829999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575049,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58600,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313240,"statfs":{"total":96636764160,"available":96576757760,"internally_reserved":0,"allocated":1069056,"data_stored":535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.88300000000000001,"15min":0.91200000000000003},"min":{"1min":0.54200000000000004,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":1.036},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.996,"15min":1.0589999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.486,"5min":1.6419999999999999,"15min":1.9390000000000001},"last":0.86699999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95399999999999996,"15min":0.98499999999999999},"min":{"1min":0.60799999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.403,"5min":1.7669999999999999,"15min":3.2599999999999998},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.099,"15min":1.1499999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.81599999999999995}]},{"osd":3,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1240000000000001,"15min":1.1579999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4770000000000001,"5min":1.774,"15min":2.194},"last":1.139},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2390000000000001,"15min":1.2929999999999999},"min":{"1min":0.53500000000000003,"5min":0.48199999999999998,"15min":0.45600000000000002},"max":{"1min":1.6579999999999999,"5min":1.831,"15min":3.1440000000000001},"last":0.75900000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.389,"15min":1.411},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.853,"5min":2.23,"15min":2.8919999999999999},"last":1.52},{"interface":"front","average":{"1min":1.389,"5min":1.387,"15min":1.4119999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.988,"5min":2.3740000000000001,"15min":2.3809999999999998},"last":1.4350000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.3919999999999999,"15min":1.421},"min":{"1min":0.85099999999999998,"5min":0.82199999999999995,"15min":0.79600000000000004},"max":{"1min":2.327,"5min":2.327,"15min":2.327},"last":1.272},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4650000000000001,"15min":1.496},"min":{"1min":0.96499999999999997,"5min":0.89100000000000001,"15min":0.75800000000000001},"max":{"1min":2.077,"5min":2.1680000000000001,"15min":3.4279999999999999},"last":1.619}]},{"osd":6,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.452,"15min":1.502},"min":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.165,"5min":2.1709999999999998,"15min":2.621},"last":1.323},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.526,"15min":1.5740000000000001},"min":{"1min":0.89000000000000001,"5min":0.82499999999999996,"15min":0.78300000000000003},"max":{"1min":2.6680000000000001,"5min":2.6749999999999998,"15min":3.6269999999999998},"last":1.6830000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.508,"15min":1.542},"min":{"1min":0.94299999999999995,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.54,"15min":1.5900000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.2480000000000002,"5min":2.4849999999999999,"15min":2.7109999999999999},"last":1.5669999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335424,"statfs":{"total":96636764160,"available":96599474176,"internally_reserved":0,"allocated":1097728,"data_stored":537921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.88900000000000001,"15min":0.90400000000000003},"min":{"1min":0.67600000000000005,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4419999999999999,"5min":1.6639999999999999,"15min":1.9259999999999999},"last":1.103},{"interface":"front","average":{"1min":1.03,"5min":1.0589999999999999,"15min":1.077},"min":{"1min":0.72999999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":0.75600000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0389999999999999,"15min":1.089},"min":{"1min":0.66200000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.4850000000000001,"5min":1.619,"15min":2.234},"last":1.248},{"interface":"front","average":{"1min":1.29,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.88300000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.895},"last":1.399}]},{"osd":3,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.26,"15min":1.3160000000000001},"min":{"1min":0.64400000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.748,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.083,"15min":1.1120000000000001},"min":{"1min":0.84199999999999997,"5min":0.37,"15min":0.37},"max":{"1min":1.744,"5min":1.887,"15min":2.3679999999999999},"last":0.84299999999999997}]},{"osd":4,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3220000000000001,"15min":1.337},"min":{"1min":0.97399999999999998,"5min":0.79200000000000004,"15min":0.76600000000000001},"max":{"1min":1.742,"5min":2.1019999999999999,"15min":2.1150000000000002},"last":1.149},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.343,"15min":1.387},"min":{"1min":1.0349999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8799999999999999,"5min":2.1469999999999998,"15min":2.6819999999999999},"last":1.5169999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.4059999999999999,"15min":1.429},"min":{"1min":1.165,"5min":0.81100000000000005,"15min":0.81000000000000005},"max":{"1min":1.8620000000000001,"5min":1.903,"15min":2.1840000000000002},"last":1.647},{"interface":"front","average":{"1min":1.532,"5min":1.4870000000000001,"15min":1.496},"min":{"1min":1.2130000000000001,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.8280000000000001,"5min":2.294,"15min":2.294},"last":1.583}]},{"osd":6,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.3819999999999999,"15min":1.4199999999999999},"min":{"1min":1.0980000000000001,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7749999999999999,"5min":1.8759999999999999,"15min":1.9379999999999999},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.4990000000000001,"15min":1.524},"min":{"1min":1.3120000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.956,"5min":2.1520000000000001,"15min":2.3300000000000001},"last":1.7509999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.542,"15min":1.5640000000000001},"min":{"1min":1.165,"5min":0.753,"15min":0.753},"max":{"1min":2.1429999999999998,"5min":2.218,"15min":2.3359999999999999},"last":1.681},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.504,"15min":1.5169999999999999},"min":{"1min":1.3320000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":2.294},"last":1.464}]}]},{"osd":0,"up_from":13,"seq":55834575049,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36964,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334876,"statfs":{"total":96636764160,"available":96598913024,"internally_reserved":0,"allocated":1662976,"data_stored":1125913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91500000000000004,"15min":0.90800000000000003},"min":{"1min":0.52100000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3240000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.75},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0509999999999999,"15min":1.0800000000000001},"min":{"1min":0.54200000000000004,"5min":0.501,"15min":0.45700000000000002},"max":{"1min":1.534,"5min":1.764,"15min":1.764},"last":1.0580000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.054,"15min":1.1060000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.6080000000000001,"5min":1.6220000000000001,"15min":1.7709999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1950000000000001},"min":{"1min":0.59899999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.722,"5min":1.893,"15min":2.4740000000000002},"last":0.93100000000000005}]},{"osd":3,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0660000000000001,"15min":1.1180000000000001},"min":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.508,"5min":2.081,"15min":2.081},"last":1.008},{"interface":"front","average":{"1min":1.262,"5min":1.24,"15min":1.3149999999999999},"min":{"1min":0.67700000000000005,"5min":0.54400000000000004,"15min":0.52500000000000002},"max":{"1min":1.8180000000000001,"5min":1.8839999999999999,"15min":2.1080000000000001},"last":1.1120000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3979999999999999,"15min":1.417},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.804,"5min":2.012,"15min":2.012},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.385,"15min":1.3740000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.68300000000000005},"max":{"1min":1.7230000000000001,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.1459999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.403,"15min":1.4159999999999999},"min":{"1min":0.84799999999999998,"5min":0.80100000000000005,"15min":0.79400000000000004},"max":{"1min":1.8460000000000001,"5min":2.0350000000000001,"15min":2.528},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.462,"15min":1.484},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.992,"5min":2.0750000000000002,"15min":2.2490000000000001},"last":1.266}]},{"osd":6,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.512,"15min":1.5489999999999999},"min":{"1min":0.78800000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.0419999999999998,"5min":2.1160000000000001,"15min":2.2959999999999998},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.504,"15min":1.5249999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2330000000000001,"5min":2.274,"15min":2.573},"last":1.673}]},{"osd":7,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5720000000000001,"15min":1.591},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.2970000000000002,"5min":2.3610000000000002,"15min":2.4100000000000001},"last":1.734},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5049999999999999,"15min":1.5429999999999999},"min":{"1min":0.875,"5min":0.81699999999999995,"15min":0.73799999999999999},"max":{"1min":1.776,"5min":2.3460000000000001,"15min":2.6640000000000001},"last":1.339}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:50:31.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:50:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.143+0000 7fbb6c105640 1 Processor -- start 2023-11-28T20:50:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.143+0000 7fbb6c105640 1 -- start start 2023-11-28T20:50:32.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.144+0000 7fbb6c105640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbb640b10e0 0x7fbb640b4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.144+0000 7fbb6c105640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6415dcc0 con 0x7fbb641438b0 2023-11-28T20:50:32.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.144+0000 7fbb6c105640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6415e840 con 0x7fbb640b0c30 2023-11-28T20:50:32.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.144+0000 7fbb6c105640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb640b26b0 con 0x7fbb640b10e0 2023-11-28T20:50:32.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.144+0000 7fbb69ea1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbb641438b0 0x7fbb640b1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42840/0 (socket says 172.21.15.45:42840) 2023-11-28T20:50:32.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.144+0000 7fbb69ea1640 1 -- 172.21.15.45:0/1791494902 learned_addr learned my addr 172.21.15.45:0/1791494902 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:32.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.144+0000 7fbb6a6a2640 1 --2- 172.21.15.45:0/1791494902 >> v2:172.21.15.45:3300/0 conn(0x7fbb640b10e0 0x7fbb640b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.145+0000 7fbb68e9f640 1 -- 172.21.15.45:0/1791494902 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733724860 0 0) 0x7fbb6415dcc0 con 0x7fbb641438b0 2023-11-28T20:50:32.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.145+0000 7fbb68e9f640 1 -- 172.21.15.45:0/1791494902 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb4c005730 con 0x7fbb641438b0 2023-11-28T20:50:32.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.145+0000 7fbb6a6a2640 1 -- 172.21.15.45:0/1791494902 >> v1:172.21.15.190:6789/0 conn(0x7fbb640b0c30 legacy=0x7fbb640b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:32.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.145+0000 7fbb6a6a2640 1 -- 172.21.15.45:0/1791494902 >> v1:172.21.15.45:6789/0 conn(0x7fbb641438b0 legacy=0x7fbb640b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.145+0000 7fbb6a6a2640 1 -- 172.21.15.45:0/1791494902 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640b1530 con 0x7fbb640b10e0 2023-11-28T20:50:32.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.146+0000 7fbb6a6a2640 1 --2- 172.21.15.45:0/1791494902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 0x7fbb640b4470 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0097a0 tx=0x7fbb5c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9107780cc2471f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:32.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.146+0000 7fbb68e9f640 1 -- 172.21.15.45:0/1791494902 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c02e070 con 0x7fbb640b10e0 2023-11-28T20:50:32.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.146+0000 7fbb68e9f640 1 -- 172.21.15.45:0/1791494902 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c0315a0 con 0x7fbb640b10e0 2023-11-28T20:50:32.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.147+0000 7fbb68e9f640 1 -- 172.21.15.45:0/1791494902 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c031910 con 0x7fbb640b10e0 2023-11-28T20:50:32.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.147+0000 7fbb6c105640 1 -- 172.21.15.45:0/1791494902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 msgr2=0x7fbb640b4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.147+0000 7fbb6c105640 1 --2- 172.21.15.45:0/1791494902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 0x7fbb640b4470 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0097a0 tx=0x7fbb5c02b840 comp rx=0 tx=0).stop 2023-11-28T20:50:32.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.147+0000 7fbb6c105640 1 -- 172.21.15.45:0/1791494902 shutdown_connections 2023-11-28T20:50:32.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.147+0000 7fbb6c105640 1 --2- 172.21.15.45:0/1791494902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 0x7fbb640b4470 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.147+0000 7fbb6c105640 1 -- 172.21.15.45:0/1791494902 >> 172.21.15.45:0/1791494902 conn(0x7fbb6406a500 msgr2=0x7fbb6406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:32.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.147+0000 7fbb6c105640 1 -- 172.21.15.45:0/1791494902 shutdown_connections 2023-11-28T20:50:32.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.147+0000 7fbb6c105640 1 -- 172.21.15.45:0/1791494902 wait complete. 2023-11-28T20:50:32.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.149+0000 7fbb6c105640 1 Processor -- start 2023-11-28T20:50:32.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.149+0000 7fbb6c105640 1 -- start start 2023-11-28T20:50:32.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.150+0000 7fbb6c105640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb640b0c30 0x7fbb6415c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.150+0000 7fbb6c105640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 0x7fbb64141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.150+0000 7fbb6c105640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb641438b0 0x7fbb641421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.150+0000 7fbb6c105640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640ad5c0 con 0x7fbb640b10e0 2023-11-28T20:50:32.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.150+0000 7fbb6c105640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640ad8a0 con 0x7fbb640b0c30 2023-11-28T20:50:32.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.150+0000 7fbb6c105640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640adb80 con 0x7fbb641438b0 2023-11-28T20:50:32.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.150+0000 7fbb696a0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 0x7fbb64141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.150+0000 7fbb696a0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 0x7fbb64141c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53562/0 (socket says 172.21.15.45:53562) 2023-11-28T20:50:32.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.150+0000 7fbb696a0640 1 -- 172.21.15.45:0/2320176248 learned_addr learned my addr 172.21.15.45:0/2320176248 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:32.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb696a0640 1 -- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb641438b0 msgr2=0x7fbb641421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb6a6a2640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb641438b0 0x7fbb641421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb696a0640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb641438b0 0x7fbb641421a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb69ea1640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb640b0c30 0x7fbb6415c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb696a0640 1 -- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb640b0c30 msgr2=0x7fbb6415c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb696a0640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb640b0c30 0x7fbb6415c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb696a0640 1 -- 172.21.15.45:0/2320176248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c009400 con 0x7fbb640b10e0 2023-11-28T20:50:32.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb69ea1640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb640b0c30 0x7fbb6415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:32.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb696a0640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 0x7fbb64141c60 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800c3c0 tx=0x7fbb5800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:32.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.151+0000 7fbb627fc640 1 -- 172.21.15.45:0/2320176248 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb580052d0 con 0x7fbb640b10e0 2023-11-28T20:50:32.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.152+0000 7fbb627fc640 1 -- 172.21.15.45:0/2320176248 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb58005450 con 0x7fbb640b10e0 2023-11-28T20:50:32.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.152+0000 7fbb627fc640 1 -- 172.21.15.45:0/2320176248 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58007260 con 0x7fbb640b10e0 2023-11-28T20:50:32.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.152+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb640ade60 con 0x7fbb640b10e0 2023-11-28T20:50:32.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.152+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6415c880 con 0x7fbb640b10e0 2023-11-28T20:50:32.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.153+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb640ae380 con 0x7fbb640b10e0 2023-11-28T20:50:32.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.154+0000 7fbb627fc640 1 -- 172.21.15.45:0/2320176248 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbb580073c0 con 0x7fbb640b10e0 2023-11-28T20:50:32.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.155+0000 7fbb627fc640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbb5006af60 0x7fbb5006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.155+0000 7fbb627fc640 1 -- 172.21.15.45:0/2320176248 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbb58088c80 con 0x7fbb640b10e0 2023-11-28T20:50:32.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.156+0000 7fbb69ea1640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbb5006af60 0x7fbb5006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.157+0000 7fbb69ea1640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbb5006af60 0x7fbb5006d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001560 tx=0x7fbb54009040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:32.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.160+0000 7fbb627fc640 1 -- 172.21.15.45:0/2320176248 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb640ae380 con 0x7fbb640b10e0 2023-11-28T20:50:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.351+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 --> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb6415e4d0 con 0x7fbb5006af60 2023-11-28T20:50:32.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.373+0000 7fbb627fc640 1 -- 172.21.15.45:0/2320176248 <== mgr.5327 v2:172.21.15.190:6832/108887886 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286612 (secure 0 0 0) 0x7fbb54001db0 con 0x7fbb5006af60 2023-11-28T20:50:32.379 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:50:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbb5006af60 msgr2=0x7fbb5006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbb5006af60 0x7fbb5006d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001560 tx=0x7fbb54009040 comp rx=0 tx=0).stop 2023-11-28T20:50:32.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 msgr2=0x7fbb64141c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 0x7fbb64141c60 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800c3c0 tx=0x7fbb5800c880 comp rx=0 tx=0).stop 2023-11-28T20:50:32.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 shutdown_connections 2023-11-28T20:50:32.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fbb5006af60 0x7fbb5006d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb641438b0 0x7fbb641421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb640b10e0 0x7fbb64141c60 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 --2- 172.21.15.45:0/2320176248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb640b0c30 0x7fbb6415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 >> 172.21.15.45:0/2320176248 conn(0x7fbb6406a500 msgr2=0x7fbb64070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:32.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 shutdown_connections 2023-11-28T20:50:32.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.379+0000 7fbb6c105640 1 -- 172.21.15.45:0/2320176248 wait complete. 2023-11-28T20:50:32.402 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-28T20:50:30.666491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":328436,"kb_used_data":9576,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754646284,"statfs":{"total":773094113280,"available":772757794816,"internally_reserved":0,"allocated":9805824,"data_stored":5469916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":326356528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.340840"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546692+0000","last_change":"2023-11-28T20:50:27.395653+0000","last_active":"2023-11-28T20:50:28.546692+0000","last_peered":"2023-11-28T20:50:28.546692+0000","last_clean":"2023-11-28T20:50:28.546692+0000","last_became_active":"2023-11-28T20:50:27.395364+0000","last_became_peered":"2023-11-28T20:50:27.395364+0000","last_unstale":"2023-11-28T20:50:28.546692+0000","last_undegraded":"2023-11-28T20:50:28.546692+0000","last_fullsized":"2023-11-28T20:50:28.546692+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:42:48.727931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546317+0000","last_change":"2023-11-28T20:50:27.395605+0000","last_active":"2023-11-28T20:50:28.546317+0000","last_peered":"2023-11-28T20:50:28.546317+0000","last_clean":"2023-11-28T20:50:28.546317+0000","last_became_active":"2023-11-28T20:50:27.395314+0000","last_became_peered":"2023-11-28T20:50:27.395314+0000","last_unstale":"2023-11-28T20:50:28.546317+0000","last_undegraded":"2023-11-28T20:50:28.546317+0000","last_fullsized":"2023-11-28T20:50:28.546317+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:01:07.388649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547626+0000","last_change":"2023-11-28T20:50:27.395539+0000","last_active":"2023-11-28T20:50:28.547626+0000","last_peered":"2023-11-28T20:50:28.547626+0000","last_clean":"2023-11-28T20:50:28.547626+0000","last_became_active":"2023-11-28T20:50:27.392159+0000","last_became_peered":"2023-11-28T20:50:27.392159+0000","last_unstale":"2023-11-28T20:50:28.547626+0000","last_undegraded":"2023-11-28T20:50:28.547626+0000","last_fullsized":"2023-11-28T20:50:28.547626+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:55:42.771188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389326+0000","last_change":"2023-11-28T20:50:27.395544+0000","last_active":"2023-11-28T20:50:28.389326+0000","last_peered":"2023-11-28T20:50:28.389326+0000","last_clean":"2023-11-28T20:50:28.389326+0000","last_became_active":"2023-11-28T20:50:27.395266+0000","last_became_peered":"2023-11-28T20:50:27.395266+0000","last_unstale":"2023-11-28T20:50:28.389326+0000","last_undegraded":"2023-11-28T20:50:28.389326+0000","last_fullsized":"2023-11-28T20:50:28.389326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:35:36.275997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548127+0000","last_change":"2023-11-28T20:50:27.399160+0000","last_active":"2023-11-28T20:50:28.548127+0000","last_peered":"2023-11-28T20:50:28.548127+0000","last_clean":"2023-11-28T20:50:28.548127+0000","last_became_active":"2023-11-28T20:50:27.398904+0000","last_became_peered":"2023-11-28T20:50:27.398904+0000","last_unstale":"2023-11-28T20:50:28.548127+0000","last_undegraded":"2023-11-28T20:50:28.548127+0000","last_fullsized":"2023-11-28T20:50:28.548127+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:18.626234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545314+0000","last_change":"2023-11-28T20:50:27.377240+0000","last_active":"2023-11-28T20:50:28.545314+0000","last_peered":"2023-11-28T20:50:28.545314+0000","last_clean":"2023-11-28T20:50:28.545314+0000","last_became_active":"2023-11-28T20:50:27.376847+0000","last_became_peered":"2023-11-28T20:50:27.376847+0000","last_unstale":"2023-11-28T20:50:28.545314+0000","last_undegraded":"2023-11-28T20:50:28.545314+0000","last_fullsized":"2023-11-28T20:50:28.545314+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:47:53.341510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.387257+0000","last_change":"2023-11-28T20:50:27.379160+0000","last_active":"2023-11-28T20:50:28.387257+0000","last_peered":"2023-11-28T20:50:28.387257+0000","last_clean":"2023-11-28T20:50:28.387257+0000","last_became_active":"2023-11-28T20:50:27.378801+0000","last_became_peered":"2023-11-28T20:50:27.378801+0000","last_unstale":"2023-11-28T20:50:28.387257+0000","last_undegraded":"2023-11-28T20:50:28.387257+0000","last_fullsized":"2023-11-28T20:50:28.387257+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:45:25.514682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516557+0000","last_change":"2023-11-28T20:50:27.381383+0000","last_active":"2023-11-28T20:50:28.516557+0000","last_peered":"2023-11-28T20:50:28.516557+0000","last_clean":"2023-11-28T20:50:28.516557+0000","last_became_active":"2023-11-28T20:50:27.379217+0000","last_became_peered":"2023-11-28T20:50:27.379217+0000","last_unstale":"2023-11-28T20:50:28.516557+0000","last_undegraded":"2023-11-28T20:50:28.516557+0000","last_fullsized":"2023-11-28T20:50:28.516557+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:48:28.926200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.549189+0000","last_change":"2023-11-28T20:50:27.393196+0000","last_active":"2023-11-28T20:50:28.549189+0000","last_peered":"2023-11-28T20:50:28.549189+0000","last_clean":"2023-11-28T20:50:28.549189+0000","last_became_active":"2023-11-28T20:50:27.392720+0000","last_became_peered":"2023-11-28T20:50:27.392720+0000","last_unstale":"2023-11-28T20:50:28.549189+0000","last_undegraded":"2023-11-28T20:50:28.549189+0000","last_fullsized":"2023-11-28T20:50:28.549189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:47.085267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548782+0000","last_change":"2023-11-28T20:50:27.395531+0000","last_active":"2023-11-28T20:50:28.548782+0000","last_peered":"2023-11-28T20:50:28.548782+0000","last_clean":"2023-11-28T20:50:28.548782+0000","last_became_active":"2023-11-28T20:50:27.395254+0000","last_became_peered":"2023-11-28T20:50:27.395254+0000","last_unstale":"2023-11-28T20:50:28.548782+0000","last_undegraded":"2023-11-28T20:50:28.548782+0000","last_fullsized":"2023-11-28T20:50:28.548782+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:54.989089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389951+0000","last_change":"2023-11-28T20:50:27.400513+0000","last_active":"2023-11-28T20:50:28.389951+0000","last_peered":"2023-11-28T20:50:28.389951+0000","last_clean":"2023-11-28T20:50:28.389951+0000","last_became_active":"2023-11-28T20:50:27.393358+0000","last_became_peered":"2023-11-28T20:50:27.393358+0000","last_unstale":"2023-11-28T20:50:28.389951+0000","last_undegraded":"2023-11-28T20:50:28.389951+0000","last_fullsized":"2023-11-28T20:50:28.389951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:06.489084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517101+0000","last_change":"2023-11-28T20:50:27.399075+0000","last_active":"2023-11-28T20:50:28.517101+0000","last_peered":"2023-11-28T20:50:28.517101+0000","last_clean":"2023-11-28T20:50:28.517101+0000","last_became_active":"2023-11-28T20:50:27.398825+0000","last_became_peered":"2023-11-28T20:50:27.398825+0000","last_unstale":"2023-11-28T20:50:28.517101+0000","last_undegraded":"2023-11-28T20:50:28.517101+0000","last_fullsized":"2023-11-28T20:50:28.517101+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:21:49.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517095+0000","last_change":"2023-11-28T20:50:27.405174+0000","last_active":"2023-11-28T20:50:28.517095+0000","last_peered":"2023-11-28T20:50:28.517095+0000","last_clean":"2023-11-28T20:50:28.517095+0000","last_became_active":"2023-11-28T20:50:27.404467+0000","last_became_peered":"2023-11-28T20:50:27.404467+0000","last_unstale":"2023-11-28T20:50:28.517095+0000","last_undegraded":"2023-11-28T20:50:28.517095+0000","last_fullsized":"2023-11-28T20:50:28.517095+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:21.780299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546773+0000","last_change":"2023-11-28T20:50:27.392435+0000","last_active":"2023-11-28T20:50:28.546773+0000","last_peered":"2023-11-28T20:50:28.546773+0000","last_clean":"2023-11-28T20:50:28.546773+0000","last_became_active":"2023-11-28T20:50:27.392115+0000","last_became_peered":"2023-11-28T20:50:27.392115+0000","last_unstale":"2023-11-28T20:50:28.546773+0000","last_undegraded":"2023-11-28T20:50:28.546773+0000","last_fullsized":"2023-11-28T20:50:28.546773+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:21.780299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546137+0000","last_change":"2023-11-28T20:50:27.385024+0000","last_active":"2023-11-28T20:50:28.546137+0000","last_peered":"2023-11-28T20:50:28.546137+0000","last_clean":"2023-11-28T20:50:28.546137+0000","last_became_active":"2023-11-28T20:50:27.384754+0000","last_became_peered":"2023-11-28T20:50:27.384754+0000","last_unstale":"2023-11-28T20:50:28.546137+0000","last_undegraded":"2023-11-28T20:50:28.546137+0000","last_fullsized":"2023-11-28T20:50:28.546137+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:21:49.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.386336+0000","last_change":"2023-11-28T20:50:27.382289+0000","last_active":"2023-11-28T20:50:28.386336+0000","last_peered":"2023-11-28T20:50:28.386336+0000","last_clean":"2023-11-28T20:50:28.386336+0000","last_became_active":"2023-11-28T20:50:27.381990+0000","last_became_peered":"2023-11-28T20:50:27.381990+0000","last_unstale":"2023-11-28T20:50:28.386336+0000","last_undegraded":"2023-11-28T20:50:28.386336+0000","last_fullsized":"2023-11-28T20:50:28.386336+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:07:33.146099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546691+0000","last_change":"2023-11-28T20:50:27.403539+0000","last_active":"2023-11-28T20:50:28.546691+0000","last_peered":"2023-11-28T20:50:28.546691+0000","last_clean":"2023-11-28T20:50:28.546691+0000","last_became_active":"2023-11-28T20:50:27.403106+0000","last_became_peered":"2023-11-28T20:50:27.403106+0000","last_unstale":"2023-11-28T20:50:28.546691+0000","last_undegraded":"2023-11-28T20:50:28.546691+0000","last_fullsized":"2023-11-28T20:50:28.546691+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:04:00.856638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389916+0000","last_change":"2023-11-28T20:50:27.397612+0000","last_active":"2023-11-28T20:50:28.389916+0000","last_peered":"2023-11-28T20:50:28.389916+0000","last_clean":"2023-11-28T20:50:28.389916+0000","last_became_active":"2023-11-28T20:50:27.397290+0000","last_became_peered":"2023-11-28T20:50:27.397290+0000","last_unstale":"2023-11-28T20:50:28.389916+0000","last_undegraded":"2023-11-28T20:50:28.389916+0000","last_fullsized":"2023-11-28T20:50:28.389916+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:53:09.882959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547689+0000","last_change":"2023-11-28T20:50:27.388858+0000","last_active":"2023-11-28T20:50:28.547689+0000","last_peered":"2023-11-28T20:50:28.547689+0000","last_clean":"2023-11-28T20:50:28.547689+0000","last_became_active":"2023-11-28T20:50:27.388547+0000","last_became_peered":"2023-11-28T20:50:27.388547+0000","last_unstale":"2023-11-28T20:50:28.547689+0000","last_undegraded":"2023-11-28T20:50:28.547689+0000","last_fullsized":"2023-11-28T20:50:28.547689+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:28.286051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388960+0000","last_change":"2023-11-28T20:50:27.386241+0000","last_active":"2023-11-28T20:50:28.388960+0000","last_peered":"2023-11-28T20:50:28.388960+0000","last_clean":"2023-11-28T20:50:28.388960+0000","last_became_active":"2023-11-28T20:50:27.385985+0000","last_became_peered":"2023-11-28T20:50:27.385985+0000","last_unstale":"2023-11-28T20:50:28.388960+0000","last_undegraded":"2023-11-28T20:50:28.388960+0000","last_fullsized":"2023-11-28T20:50:28.388960+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:11:33.831894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517130+0000","last_change":"2023-11-28T20:50:27.402514+0000","last_active":"2023-11-28T20:50:28.517130+0000","last_peered":"2023-11-28T20:50:28.517130+0000","last_clean":"2023-11-28T20:50:28.517130+0000","last_became_active":"2023-11-28T20:50:27.399578+0000","last_became_peered":"2023-11-28T20:50:27.399578+0000","last_unstale":"2023-11-28T20:50:28.517130+0000","last_undegraded":"2023-11-28T20:50:28.517130+0000","last_fullsized":"2023-11-28T20:50:28.517130+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:17:56.648761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388598+0000","last_change":"2023-11-28T20:50:27.381861+0000","last_active":"2023-11-28T20:50:28.388598+0000","last_peered":"2023-11-28T20:50:28.388598+0000","last_clean":"2023-11-28T20:50:28.388598+0000","last_became_active":"2023-11-28T20:50:27.381572+0000","last_became_peered":"2023-11-28T20:50:27.381572+0000","last_unstale":"2023-11-28T20:50:28.388598+0000","last_undegraded":"2023-11-28T20:50:28.388598+0000","last_fullsized":"2023-11-28T20:50:28.388598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:18.836466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389428+0000","last_change":"2023-11-28T20:50:27.377228+0000","last_active":"2023-11-28T20:50:28.389428+0000","last_peered":"2023-11-28T20:50:28.389428+0000","last_clean":"2023-11-28T20:50:28.389428+0000","last_became_active":"2023-11-28T20:50:27.375619+0000","last_became_peered":"2023-11-28T20:50:27.375619+0000","last_unstale":"2023-11-28T20:50:28.389428+0000","last_undegraded":"2023-11-28T20:50:28.389428+0000","last_fullsized":"2023-11-28T20:50:28.389428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:05:34.144511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546099+0000","last_change":"2023-11-28T20:50:27.391584+0000","last_active":"2023-11-28T20:50:28.546099+0000","last_peered":"2023-11-28T20:50:28.546099+0000","last_clean":"2023-11-28T20:50:28.546099+0000","last_became_active":"2023-11-28T20:50:27.391273+0000","last_became_peered":"2023-11-28T20:50:27.391273+0000","last_unstale":"2023-11-28T20:50:28.546099+0000","last_undegraded":"2023-11-28T20:50:28.546099+0000","last_fullsized":"2023-11-28T20:50:28.546099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:44.386833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517541+0000","last_change":"2023-11-28T20:50:27.384755+0000","last_active":"2023-11-28T20:50:28.517541+0000","last_peered":"2023-11-28T20:50:28.517541+0000","last_clean":"2023-11-28T20:50:28.517541+0000","last_became_active":"2023-11-28T20:50:27.384493+0000","last_became_peered":"2023-11-28T20:50:27.384493+0000","last_unstale":"2023-11-28T20:50:28.517541+0000","last_undegraded":"2023-11-28T20:50:28.517541+0000","last_fullsized":"2023-11-28T20:50:28.517541+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:56:54.989089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517275+0000","last_change":"2023-11-28T20:50:27.395102+0000","last_active":"2023-11-28T20:50:28.517275+0000","last_peered":"2023-11-28T20:50:28.517275+0000","last_clean":"2023-11-28T20:50:28.517275+0000","last_became_active":"2023-11-28T20:50:27.394658+0000","last_became_peered":"2023-11-28T20:50:27.394658+0000","last_unstale":"2023-11-28T20:50:28.517275+0000","last_undegraded":"2023-11-28T20:50:28.517275+0000","last_fullsized":"2023-11-28T20:50:28.517275+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:36.701581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389875+0000","last_change":"2023-11-28T20:50:27.390252+0000","last_active":"2023-11-28T20:50:28.389875+0000","last_peered":"2023-11-28T20:50:28.389875+0000","last_clean":"2023-11-28T20:50:28.389875+0000","last_became_active":"2023-11-28T20:50:27.389988+0000","last_became_peered":"2023-11-28T20:50:27.389988+0000","last_unstale":"2023-11-28T20:50:28.389875+0000","last_undegraded":"2023-11-28T20:50:28.389875+0000","last_fullsized":"2023-11-28T20:50:28.389875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:08.611663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546188+0000","last_change":"2023-11-28T20:50:27.393617+0000","last_active":"2023-11-28T20:50:28.546188+0000","last_peered":"2023-11-28T20:50:28.546188+0000","last_clean":"2023-11-28T20:50:28.546188+0000","last_became_active":"2023-11-28T20:50:27.393345+0000","last_became_peered":"2023-11-28T20:50:27.393345+0000","last_unstale":"2023-11-28T20:50:28.546188+0000","last_undegraded":"2023-11-28T20:50:28.546188+0000","last_fullsized":"2023-11-28T20:50:28.546188+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:01:44.632228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547210+0000","last_change":"2023-11-28T20:50:27.398180+0000","last_active":"2023-11-28T20:50:28.547210+0000","last_peered":"2023-11-28T20:50:28.547210+0000","last_clean":"2023-11-28T20:50:28.547210+0000","last_became_active":"2023-11-28T20:50:27.397828+0000","last_became_peered":"2023-11-28T20:50:27.397828+0000","last_unstale":"2023-11-28T20:50:28.547210+0000","last_undegraded":"2023-11-28T20:50:28.547210+0000","last_fullsized":"2023-11-28T20:50:28.547210+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:17:56.648761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516615+0000","last_change":"2023-11-28T20:50:27.387066+0000","last_active":"2023-11-28T20:50:28.516615+0000","last_peered":"2023-11-28T20:50:28.516615+0000","last_clean":"2023-11-28T20:50:28.516615+0000","last_became_active":"2023-11-28T20:50:27.386766+0000","last_became_peered":"2023-11-28T20:50:27.386766+0000","last_unstale":"2023-11-28T20:50:28.516615+0000","last_undegraded":"2023-11-28T20:50:28.516615+0000","last_fullsized":"2023-11-28T20:50:28.516615+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:44.386833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388815+0000","last_change":"2023-11-28T20:50:27.386562+0000","last_active":"2023-11-28T20:50:28.388815+0000","last_peered":"2023-11-28T20:50:28.388815+0000","last_clean":"2023-11-28T20:50:28.388815+0000","last_became_active":"2023-11-28T20:50:27.386280+0000","last_became_peered":"2023-11-28T20:50:27.386280+0000","last_unstale":"2023-11-28T20:50:28.388815+0000","last_undegraded":"2023-11-28T20:50:28.388815+0000","last_fullsized":"2023-11-28T20:50:28.388815+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:33.475972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.544636+0000","last_change":"2023-11-28T20:50:27.380417+0000","last_active":"2023-11-28T20:50:28.544636+0000","last_peered":"2023-11-28T20:50:28.544636+0000","last_clean":"2023-11-28T20:50:28.544636+0000","last_became_active":"2023-11-28T20:50:27.379981+0000","last_became_peered":"2023-11-28T20:50:27.379981+0000","last_unstale":"2023-11-28T20:50:28.544636+0000","last_undegraded":"2023-11-28T20:50:28.544636+0000","last_fullsized":"2023-11-28T20:50:28.544636+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:48:16.088499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547619+0000","last_change":"2023-11-28T20:50:27.400415+0000","last_active":"2023-11-28T20:50:28.547619+0000","last_peered":"2023-11-28T20:50:28.547619+0000","last_clean":"2023-11-28T20:50:28.547619+0000","last_became_active":"2023-11-28T20:50:27.400143+0000","last_became_peered":"2023-11-28T20:50:27.400143+0000","last_unstale":"2023-11-28T20:50:28.547619+0000","last_undegraded":"2023-11-28T20:50:28.547619+0000","last_fullsized":"2023-11-28T20:50:28.547619+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:02:12.299296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"87'4","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546800+0000","last_change":"2023-11-28T20:50:26.369068+0000","last_active":"2023-11-28T20:50:28.546800+0000","last_peered":"2023-11-28T20:50:28.546800+0000","last_clean":"2023-11-28T20:50:28.546800+0000","last_became_active":"2023-11-28T20:50:26.368651+0000","last_became_peered":"2023-11-28T20:50:26.368651+0000","last_unstale":"2023-11-28T20:50:28.546800+0000","last_undegraded":"2023-11-28T20:50:28.546800+0000","last_fullsized":"2023-11-28T20:50:28.546800+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:33:09.154021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.390109+0000","last_change":"2023-11-28T20:50:27.390842+0000","last_active":"2023-11-28T20:50:28.390109+0000","last_peered":"2023-11-28T20:50:28.390109+0000","last_clean":"2023-11-28T20:50:28.390109+0000","last_became_active":"2023-11-28T20:50:27.390545+0000","last_became_peered":"2023-11-28T20:50:27.390545+0000","last_unstale":"2023-11-28T20:50:28.390109+0000","last_undegraded":"2023-11-28T20:50:28.390109+0000","last_fullsized":"2023-11-28T20:50:28.390109+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:47:14.631727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545716+0000","last_change":"2023-11-28T20:50:26.370047+0000","last_active":"2023-11-28T20:50:28.545716+0000","last_peered":"2023-11-28T20:50:28.545716+0000","last_clean":"2023-11-28T20:50:28.545716+0000","last_became_active":"2023-11-28T20:50:26.369740+0000","last_became_peered":"2023-11-28T20:50:26.369740+0000","last_unstale":"2023-11-28T20:50:28.545716+0000","last_undegraded":"2023-11-28T20:50:28.545716+0000","last_fullsized":"2023-11-28T20:50:28.545716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:56.711142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545632+0000","last_change":"2023-11-28T20:50:27.377686+0000","last_active":"2023-11-28T20:50:28.545632+0000","last_peered":"2023-11-28T20:50:28.545632+0000","last_clean":"2023-11-28T20:50:28.545632+0000","last_became_active":"2023-11-28T20:50:27.377351+0000","last_became_peered":"2023-11-28T20:50:27.377351+0000","last_unstale":"2023-11-28T20:50:28.545632+0000","last_undegraded":"2023-11-28T20:50:28.545632+0000","last_fullsized":"2023-11-28T20:50:28.545632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:56:19.429557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545643+0000","last_change":"2023-11-28T20:50:26.374085+0000","last_active":"2023-11-28T20:50:28.545643+0000","last_peered":"2023-11-28T20:50:28.545643+0000","last_clean":"2023-11-28T20:50:28.545643+0000","last_became_active":"2023-11-28T20:50:26.373580+0000","last_became_peered":"2023-11-28T20:50:26.373580+0000","last_unstale":"2023-11-28T20:50:28.545643+0000","last_undegraded":"2023-11-28T20:50:28.545643+0000","last_fullsized":"2023-11-28T20:50:28.545643+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:17:55.635545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548984+0000","last_change":"2023-11-28T20:50:27.398396+0000","last_active":"2023-11-28T20:50:28.548984+0000","last_peered":"2023-11-28T20:50:28.548984+0000","last_clean":"2023-11-28T20:50:28.548984+0000","last_became_active":"2023-11-28T20:50:27.397796+0000","last_became_peered":"2023-11-28T20:50:27.397796+0000","last_unstale":"2023-11-28T20:50:28.548984+0000","last_undegraded":"2023-11-28T20:50:28.548984+0000","last_fullsized":"2023-11-28T20:50:28.548984+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:19:40.943469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545279+0000","last_change":"2023-11-28T20:50:26.360972+0000","last_active":"2023-11-28T20:50:28.545279+0000","last_peered":"2023-11-28T20:50:28.545279+0000","last_clean":"2023-11-28T20:50:28.545279+0000","last_became_active":"2023-11-28T20:50:26.359705+0000","last_became_peered":"2023-11-28T20:50:26.359705+0000","last_unstale":"2023-11-28T20:50:28.545279+0000","last_undegraded":"2023-11-28T20:50:28.545279+0000","last_fullsized":"2023-11-28T20:50:28.545279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:19:39.930253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.386724+0000","last_change":"2023-11-28T20:50:27.387327+0000","last_active":"2023-11-28T20:50:28.386724+0000","last_peered":"2023-11-28T20:50:28.386724+0000","last_clean":"2023-11-28T20:50:28.386724+0000","last_became_active":"2023-11-28T20:50:27.386574+0000","last_became_peered":"2023-11-28T20:50:27.386574+0000","last_unstale":"2023-11-28T20:50:28.386724+0000","last_undegraded":"2023-11-28T20:50:28.386724+0000","last_fullsized":"2023-11-28T20:50:28.386724+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:01.106525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388909+0000","last_change":"2023-11-28T20:50:26.362697+0000","last_active":"2023-11-28T20:50:28.388909+0000","last_peered":"2023-11-28T20:50:28.388909+0000","last_clean":"2023-11-28T20:50:28.388909+0000","last_became_active":"2023-11-28T20:50:26.362321+0000","last_became_peered":"2023-11-28T20:50:26.362321+0000","last_unstale":"2023-11-28T20:50:28.388909+0000","last_undegraded":"2023-11-28T20:50:28.388909+0000","last_fullsized":"2023-11-28T20:50:28.388909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:08.320262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545119+0000","last_change":"2023-11-28T20:50:27.392080+0000","last_active":"2023-11-28T20:50:28.545119+0000","last_peered":"2023-11-28T20:50:28.545119+0000","last_clean":"2023-11-28T20:50:28.545119+0000","last_became_active":"2023-11-28T20:50:27.391753+0000","last_became_peered":"2023-11-28T20:50:27.391753+0000","last_unstale":"2023-11-28T20:50:28.545119+0000","last_undegraded":"2023-11-28T20:50:28.545119+0000","last_fullsized":"2023-11-28T20:50:28.545119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:03:15.320718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388566+0000","last_change":"2023-11-28T20:50:26.346818+0000","last_active":"2023-11-28T20:50:28.388566+0000","last_peered":"2023-11-28T20:50:28.388566+0000","last_clean":"2023-11-28T20:50:28.388566+0000","last_became_active":"2023-11-28T20:50:26.346322+0000","last_became_peered":"2023-11-28T20:50:26.346322+0000","last_unstale":"2023-11-28T20:50:28.388566+0000","last_undegraded":"2023-11-28T20:50:28.388566+0000","last_fullsized":"2023-11-28T20:50:28.388566+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:49:58.751907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547778+0000","last_change":"2023-11-28T20:50:27.400481+0000","last_active":"2023-11-28T20:50:28.547778+0000","last_peered":"2023-11-28T20:50:28.547778+0000","last_clean":"2023-11-28T20:50:28.547778+0000","last_became_active":"2023-11-28T20:50:27.400160+0000","last_became_peered":"2023-11-28T20:50:27.400160+0000","last_unstale":"2023-11-28T20:50:28.547778+0000","last_undegraded":"2023-11-28T20:50:28.547778+0000","last_fullsized":"2023-11-28T20:50:28.547778+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:14.643205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388496+0000","last_change":"2023-11-28T20:50:26.353586+0000","last_active":"2023-11-28T20:50:28.388496+0000","last_peered":"2023-11-28T20:50:28.388496+0000","last_clean":"2023-11-28T20:50:28.388496+0000","last_became_active":"2023-11-28T20:50:26.353263+0000","last_became_peered":"2023-11-28T20:50:26.353263+0000","last_unstale":"2023-11-28T20:50:28.388496+0000","last_undegraded":"2023-11-28T20:50:28.388496+0000","last_fullsized":"2023-11-28T20:50:28.388496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:56.711142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388858+0000","last_change":"2023-11-28T20:50:27.401975+0000","last_active":"2023-11-28T20:50:28.388858+0000","last_peered":"2023-11-28T20:50:28.388858+0000","last_clean":"2023-11-28T20:50:28.388858+0000","last_became_active":"2023-11-28T20:50:27.401559+0000","last_became_peered":"2023-11-28T20:50:27.401559+0000","last_unstale":"2023-11-28T20:50:28.388858+0000","last_undegraded":"2023-11-28T20:50:28.388858+0000","last_fullsized":"2023-11-28T20:50:28.388858+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:03:51.702671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546495+0000","last_change":"2023-11-28T20:50:26.358493+0000","last_active":"2023-11-28T20:50:28.546495+0000","last_peered":"2023-11-28T20:50:28.546495+0000","last_clean":"2023-11-28T20:50:28.546495+0000","last_became_active":"2023-11-28T20:50:26.357981+0000","last_became_peered":"2023-11-28T20:50:26.357981+0000","last_unstale":"2023-11-28T20:50:28.546495+0000","last_undegraded":"2023-11-28T20:50:28.546495+0000","last_fullsized":"2023-11-28T20:50:28.546495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:42:57.331725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546186+0000","last_change":"2023-11-28T20:50:27.385221+0000","last_active":"2023-11-28T20:50:28.546186+0000","last_peered":"2023-11-28T20:50:28.546186+0000","last_clean":"2023-11-28T20:50:28.546186+0000","last_became_active":"2023-11-28T20:50:27.384945+0000","last_became_peered":"2023-11-28T20:50:27.384945+0000","last_unstale":"2023-11-28T20:50:28.546186+0000","last_undegraded":"2023-11-28T20:50:28.546186+0000","last_fullsized":"2023-11-28T20:50:28.546186+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:58:14.924703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.549119+0000","last_change":"2023-11-28T20:50:26.377486+0000","last_active":"2023-11-28T20:50:28.549119+0000","last_peered":"2023-11-28T20:50:28.549119+0000","last_clean":"2023-11-28T20:50:28.549119+0000","last_became_active":"2023-11-28T20:50:26.377174+0000","last_became_peered":"2023-11-28T20:50:26.377174+0000","last_unstale":"2023-11-28T20:50:28.549119+0000","last_undegraded":"2023-11-28T20:50:28.549119+0000","last_fullsized":"2023-11-28T20:50:28.549119+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:16.565083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.389021+0000","last_change":"2023-11-28T20:50:27.391658+0000","last_active":"2023-11-28T20:50:28.389021+0000","last_peered":"2023-11-28T20:50:28.389021+0000","last_clean":"2023-11-28T20:50:28.389021+0000","last_became_active":"2023-11-28T20:50:27.391350+0000","last_became_peered":"2023-11-28T20:50:27.391350+0000","last_unstale":"2023-11-28T20:50:28.389021+0000","last_undegraded":"2023-11-28T20:50:28.389021+0000","last_fullsized":"2023-11-28T20:50:28.389021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:23.384849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.390133+0000","last_change":"2023-11-28T20:50:26.367836+0000","last_active":"2023-11-28T20:50:28.390133+0000","last_peered":"2023-11-28T20:50:28.390133+0000","last_clean":"2023-11-28T20:50:28.390133+0000","last_became_active":"2023-11-28T20:50:26.367228+0000","last_became_peered":"2023-11-28T20:50:26.367228+0000","last_unstale":"2023-11-28T20:50:28.390133+0000","last_undegraded":"2023-11-28T20:50:28.390133+0000","last_fullsized":"2023-11-28T20:50:28.390133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:54.714495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548787+0000","last_change":"2023-11-28T20:50:27.394955+0000","last_active":"2023-11-28T20:50:28.548787+0000","last_peered":"2023-11-28T20:50:28.548787+0000","last_clean":"2023-11-28T20:50:28.548787+0000","last_became_active":"2023-11-28T20:50:27.394683+0000","last_became_peered":"2023-11-28T20:50:27.394683+0000","last_unstale":"2023-11-28T20:50:28.548787+0000","last_undegraded":"2023-11-28T20:50:28.548787+0000","last_fullsized":"2023-11-28T20:50:28.548787+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:03:07.832708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"87'3","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547890+0000","last_change":"2023-11-28T20:50:26.375793+0000","last_active":"2023-11-28T20:50:28.547890+0000","last_peered":"2023-11-28T20:50:28.547890+0000","last_clean":"2023-11-28T20:50:28.547890+0000","last_became_active":"2023-11-28T20:50:26.375480+0000","last_became_peered":"2023-11-28T20:50:26.375480+0000","last_unstale":"2023-11-28T20:50:28.547890+0000","last_undegraded":"2023-11-28T20:50:28.547890+0000","last_fullsized":"2023-11-28T20:50:28.547890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:14:31.987430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.390057+0000","last_change":"2023-11-28T20:50:27.388584+0000","last_active":"2023-11-28T20:50:28.390057+0000","last_peered":"2023-11-28T20:50:28.390057+0000","last_clean":"2023-11-28T20:50:28.390057+0000","last_became_active":"2023-11-28T20:50:27.388218+0000","last_became_peered":"2023-11-28T20:50:27.388218+0000","last_unstale":"2023-11-28T20:50:28.390057+0000","last_undegraded":"2023-11-28T20:50:28.390057+0000","last_fullsized":"2023-11-28T20:50:28.390057+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:29:10.316623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"87'6","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546372+0000","last_change":"2023-11-28T20:50:26.364988+0000","last_active":"2023-11-28T20:50:28.546372+0000","last_peered":"2023-11-28T20:50:28.546372+0000","last_clean":"2023-11-28T20:50:28.546372+0000","last_became_active":"2023-11-28T20:50:26.364668+0000","last_became_peered":"2023-11-28T20:50:26.364668+0000","last_unstale":"2023-11-28T20:50:28.546372+0000","last_undegraded":"2023-11-28T20:50:28.546372+0000","last_fullsized":"2023-11-28T20:50:28.546372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:03:47.992576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388915+0000","last_change":"2023-11-28T20:50:27.391981+0000","last_active":"2023-11-28T20:50:28.388915+0000","last_peered":"2023-11-28T20:50:28.388915+0000","last_clean":"2023-11-28T20:50:28.388915+0000","last_became_active":"2023-11-28T20:50:27.391721+0000","last_became_peered":"2023-11-28T20:50:27.391721+0000","last_unstale":"2023-11-28T20:50:28.388915+0000","last_undegraded":"2023-11-28T20:50:28.388915+0000","last_fullsized":"2023-11-28T20:50:28.388915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:36:07.688394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547086+0000","last_change":"2023-11-28T20:50:26.373562+0000","last_active":"2023-11-28T20:50:28.547086+0000","last_peered":"2023-11-28T20:50:28.547086+0000","last_clean":"2023-11-28T20:50:28.547086+0000","last_became_active":"2023-11-28T20:50:26.372795+0000","last_became_peered":"2023-11-28T20:50:26.372795+0000","last_unstale":"2023-11-28T20:50:28.547086+0000","last_undegraded":"2023-11-28T20:50:28.547086+0000","last_fullsized":"2023-11-28T20:50:28.547086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:48:27.912984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516571+0000","last_change":"2023-11-28T20:50:27.390734+0000","last_active":"2023-11-28T20:50:28.516571+0000","last_peered":"2023-11-28T20:50:28.516571+0000","last_clean":"2023-11-28T20:50:28.516571+0000","last_became_active":"2023-11-28T20:50:27.390372+0000","last_became_peered":"2023-11-28T20:50:27.390372+0000","last_unstale":"2023-11-28T20:50:28.516571+0000","last_undegraded":"2023-11-28T20:50:28.516571+0000","last_fullsized":"2023-11-28T20:50:28.516571+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:02.602202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516395+0000","last_change":"2023-11-28T20:50:26.369735+0000","last_active":"2023-11-28T20:50:28.516395+0000","last_peered":"2023-11-28T20:50:28.516395+0000","last_clean":"2023-11-28T20:50:28.516395+0000","last_became_active":"2023-11-28T20:50:26.369425+0000","last_became_peered":"2023-11-28T20:50:26.369425+0000","last_unstale":"2023-11-28T20:50:28.516395+0000","last_undegraded":"2023-11-28T20:50:28.516395+0000","last_fullsized":"2023-11-28T20:50:28.516395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:07.750976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548235+0000","last_change":"2023-11-28T20:50:27.383192+0000","last_active":"2023-11-28T20:50:28.548235+0000","last_peered":"2023-11-28T20:50:28.548235+0000","last_clean":"2023-11-28T20:50:28.548235+0000","last_became_active":"2023-11-28T20:50:27.381824+0000","last_became_peered":"2023-11-28T20:50:27.381824+0000","last_unstale":"2023-11-28T20:50:28.548235+0000","last_undegraded":"2023-11-28T20:50:28.548235+0000","last_fullsized":"2023-11-28T20:50:28.548235+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:36:31.251810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546354+0000","last_change":"2023-11-28T20:50:26.372157+0000","last_active":"2023-11-28T20:50:28.546354+0000","last_peered":"2023-11-28T20:50:28.546354+0000","last_clean":"2023-11-28T20:50:28.546354+0000","last_became_active":"2023-11-28T20:50:26.371068+0000","last_became_peered":"2023-11-28T20:50:26.371068+0000","last_unstale":"2023-11-28T20:50:28.546354+0000","last_undegraded":"2023-11-28T20:50:28.546354+0000","last_fullsized":"2023-11-28T20:50:28.546354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:49:59.121264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545693+0000","last_change":"2023-11-28T20:50:27.392550+0000","last_active":"2023-11-28T20:50:28.545693+0000","last_peered":"2023-11-28T20:50:28.545693+0000","last_clean":"2023-11-28T20:50:28.545693+0000","last_became_active":"2023-11-28T20:50:27.392018+0000","last_became_peered":"2023-11-28T20:50:27.392018+0000","last_unstale":"2023-11-28T20:50:28.545693+0000","last_undegraded":"2023-11-28T20:50:28.545693+0000","last_fullsized":"2023-11-28T20:50:28.545693+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:08:35.366099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"87'8","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388712+0000","last_change":"2023-11-28T20:50:26.371646+0000","last_active":"2023-11-28T20:50:28.388712+0000","last_peered":"2023-11-28T20:50:28.388712+0000","last_clean":"2023-11-28T20:50:28.388712+0000","last_became_active":"2023-11-28T20:50:26.371248+0000","last_became_peered":"2023-11-28T20:50:26.371248+0000","last_unstale":"2023-11-28T20:50:28.388712+0000","last_undegraded":"2023-11-28T20:50:28.388712+0000","last_fullsized":"2023-11-28T20:50:28.388712+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:56:11.241090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547281+0000","last_change":"2023-11-28T20:50:27.397537+0000","last_active":"2023-11-28T20:50:28.547281+0000","last_peered":"2023-11-28T20:50:28.547281+0000","last_clean":"2023-11-28T20:50:28.547281+0000","last_became_active":"2023-11-28T20:50:27.397220+0000","last_became_peered":"2023-11-28T20:50:27.397220+0000","last_unstale":"2023-11-28T20:50:28.547281+0000","last_undegraded":"2023-11-28T20:50:28.547281+0000","last_fullsized":"2023-11-28T20:50:28.547281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:36.701581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517363+0000","last_change":"2023-11-28T20:50:26.372117+0000","last_active":"2023-11-28T20:50:28.517363+0000","last_peered":"2023-11-28T20:50:28.517363+0000","last_clean":"2023-11-28T20:50:28.517363+0000","last_became_active":"2023-11-28T20:50:26.370855+0000","last_became_peered":"2023-11-28T20:50:26.370855+0000","last_unstale":"2023-11-28T20:50:28.517363+0000","last_undegraded":"2023-11-28T20:50:28.517363+0000","last_fullsized":"2023-11-28T20:50:28.517363+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:10:44.042782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548047+0000","last_change":"2023-11-28T20:50:27.403944+0000","last_active":"2023-11-28T20:50:28.548047+0000","last_peered":"2023-11-28T20:50:28.548047+0000","last_clean":"2023-11-28T20:50:28.548047+0000","last_became_active":"2023-11-28T20:50:27.403709+0000","last_became_peered":"2023-11-28T20:50:27.403709+0000","last_unstale":"2023-11-28T20:50:28.548047+0000","last_undegraded":"2023-11-28T20:50:28.548047+0000","last_fullsized":"2023-11-28T20:50:28.548047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:43:56.162583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548774+0000","last_change":"2023-11-28T20:50:26.377645+0000","last_active":"2023-11-28T20:50:28.548774+0000","last_peered":"2023-11-28T20:50:28.548774+0000","last_clean":"2023-11-28T20:50:28.548774+0000","last_became_active":"2023-11-28T20:50:26.377303+0000","last_became_peered":"2023-11-28T20:50:26.377303+0000","last_unstale":"2023-11-28T20:50:28.548774+0000","last_undegraded":"2023-11-28T20:50:28.548774+0000","last_fullsized":"2023-11-28T20:50:28.548774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:07.750976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.390257+0000","last_change":"2023-11-28T20:50:27.401010+0000","last_active":"2023-11-28T20:50:28.390257+0000","last_peered":"2023-11-28T20:50:28.390257+0000","last_clean":"2023-11-28T20:50:28.390257+0000","last_became_active":"2023-11-28T20:50:27.394283+0000","last_became_peered":"2023-11-28T20:50:27.394283+0000","last_unstale":"2023-11-28T20:50:28.390257+0000","last_undegraded":"2023-11-28T20:50:28.390257+0000","last_fullsized":"2023-11-28T20:50:28.390257+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:27:04.266720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"87'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548581+0000","last_change":"2023-11-28T20:50:26.375882+0000","last_active":"2023-11-28T20:50:28.548581+0000","last_peered":"2023-11-28T20:50:28.548581+0000","last_clean":"2023-11-28T20:50:28.548581+0000","last_became_active":"2023-11-28T20:50:26.375573+0000","last_became_peered":"2023-11-28T20:50:26.375573+0000","last_unstale":"2023-11-28T20:50:28.548581+0000","last_undegraded":"2023-11-28T20:50:28.548581+0000","last_fullsized":"2023-11-28T20:50:28.548581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:11:39.294283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548621+0000","last_change":"2023-11-28T20:50:27.384257+0000","last_active":"2023-11-28T20:50:28.548621+0000","last_peered":"2023-11-28T20:50:28.548621+0000","last_clean":"2023-11-28T20:50:28.548621+0000","last_became_active":"2023-11-28T20:50:27.383983+0000","last_became_peered":"2023-11-28T20:50:27.383983+0000","last_unstale":"2023-11-28T20:50:28.548621+0000","last_undegraded":"2023-11-28T20:50:28.548621+0000","last_fullsized":"2023-11-28T20:50:28.548621+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:07:17.175256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388882+0000","last_change":"2023-11-28T20:50:26.361958+0000","last_active":"2023-11-28T20:50:28.388882+0000","last_peered":"2023-11-28T20:50:28.388882+0000","last_clean":"2023-11-28T20:50:28.388882+0000","last_became_active":"2023-11-28T20:50:26.361732+0000","last_became_peered":"2023-11-28T20:50:26.361732+0000","last_unstale":"2023-11-28T20:50:28.388882+0000","last_undegraded":"2023-11-28T20:50:28.388882+0000","last_fullsized":"2023-11-28T20:50:28.388882+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:51:45.548184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.544688+0000","last_change":"2023-11-28T20:50:27.380245+0000","last_active":"2023-11-28T20:50:28.544688+0000","last_peered":"2023-11-28T20:50:28.544688+0000","last_clean":"2023-11-28T20:50:28.544688+0000","last_became_active":"2023-11-28T20:50:27.379633+0000","last_became_peered":"2023-11-28T20:50:27.379633+0000","last_unstale":"2023-11-28T20:50:28.544688+0000","last_undegraded":"2023-11-28T20:50:28.544688+0000","last_fullsized":"2023-11-28T20:50:28.544688+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:41.319338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545169+0000","last_change":"2023-11-28T20:50:26.359911+0000","last_active":"2023-11-28T20:50:28.545169+0000","last_peered":"2023-11-28T20:50:28.545169+0000","last_clean":"2023-11-28T20:50:28.545169+0000","last_became_active":"2023-11-28T20:50:26.359378+0000","last_became_peered":"2023-11-28T20:50:26.359378+0000","last_unstale":"2023-11-28T20:50:28.545169+0000","last_undegraded":"2023-11-28T20:50:28.545169+0000","last_fullsized":"2023-11-28T20:50:28.545169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:35.688365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516523+0000","last_change":"2023-11-28T20:50:27.390044+0000","last_active":"2023-11-28T20:50:28.516523+0000","last_peered":"2023-11-28T20:50:28.516523+0000","last_clean":"2023-11-28T20:50:28.516523+0000","last_became_active":"2023-11-28T20:50:27.382173+0000","last_became_peered":"2023-11-28T20:50:27.382173+0000","last_unstale":"2023-11-28T20:50:28.516523+0000","last_undegraded":"2023-11-28T20:50:28.516523+0000","last_fullsized":"2023-11-28T20:50:28.516523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:42:48.797967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.516665+0000","last_change":"2023-11-28T20:50:26.369602+0000","last_active":"2023-11-28T20:50:28.516665+0000","last_peered":"2023-11-28T20:50:28.516665+0000","last_clean":"2023-11-28T20:50:28.516665+0000","last_became_active":"2023-11-28T20:50:26.369273+0000","last_became_peered":"2023-11-28T20:50:26.369273+0000","last_unstale":"2023-11-28T20:50:28.516665+0000","last_undegraded":"2023-11-28T20:50:28.516665+0000","last_fullsized":"2023-11-28T20:50:28.516665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:05:30.047531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548128+0000","last_change":"2023-11-28T20:50:27.391826+0000","last_active":"2023-11-28T20:50:28.548128+0000","last_peered":"2023-11-28T20:50:28.548128+0000","last_clean":"2023-11-28T20:50:28.548128+0000","last_became_active":"2023-11-28T20:50:27.391488+0000","last_became_peered":"2023-11-28T20:50:27.391488+0000","last_unstale":"2023-11-28T20:50:28.548128+0000","last_undegraded":"2023-11-28T20:50:28.548128+0000","last_fullsized":"2023-11-28T20:50:28.548128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:31:29.801467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.544889+0000","last_change":"2023-11-28T20:50:26.368382+0000","last_active":"2023-11-28T20:50:28.544889+0000","last_peered":"2023-11-28T20:50:28.544889+0000","last_clean":"2023-11-28T20:50:28.544889+0000","last_became_active":"2023-11-28T20:50:26.367894+0000","last_became_peered":"2023-11-28T20:50:26.367894+0000","last_unstale":"2023-11-28T20:50:28.544889+0000","last_undegraded":"2023-11-28T20:50:28.544889+0000","last_fullsized":"2023-11-28T20:50:28.544889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:20.767083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545161+0000","last_change":"2023-11-28T20:50:27.397610+0000","last_active":"2023-11-28T20:50:28.545161+0000","last_peered":"2023-11-28T20:50:28.545161+0000","last_clean":"2023-11-28T20:50:28.545161+0000","last_became_active":"2023-11-28T20:50:27.397008+0000","last_became_peered":"2023-11-28T20:50:27.397008+0000","last_unstale":"2023-11-28T20:50:28.545161+0000","last_undegraded":"2023-11-28T20:50:28.545161+0000","last_fullsized":"2023-11-28T20:50:28.545161+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:09.333478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"87'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517662+0000","last_change":"2023-11-28T20:50:26.372742+0000","last_active":"2023-11-28T20:50:28.517662+0000","last_peered":"2023-11-28T20:50:28.517662+0000","last_clean":"2023-11-28T20:50:28.517662+0000","last_became_active":"2023-11-28T20:50:26.372474+0000","last_became_peered":"2023-11-28T20:50:26.372474+0000","last_unstale":"2023-11-28T20:50:28.517662+0000","last_undegraded":"2023-11-28T20:50:28.517662+0000","last_fullsized":"2023-11-28T20:50:28.517662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:16.565083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.387412+0000","last_change":"2023-11-28T20:50:27.393641+0000","last_active":"2023-11-28T20:50:28.387412+0000","last_peered":"2023-11-28T20:50:28.387412+0000","last_clean":"2023-11-28T20:50:28.387412+0000","last_became_active":"2023-11-28T20:50:27.393361+0000","last_became_peered":"2023-11-28T20:50:27.393361+0000","last_unstale":"2023-11-28T20:50:28.387412+0000","last_undegraded":"2023-11-28T20:50:28.387412+0000","last_fullsized":"2023-11-28T20:50:28.387412+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:09:40.276726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517642+0000","last_change":"2023-11-28T20:50:26.363037+0000","last_active":"2023-11-28T20:50:28.517642+0000","last_peered":"2023-11-28T20:50:28.517642+0000","last_clean":"2023-11-28T20:50:28.517642+0000","last_became_active":"2023-11-28T20:50:26.361957+0000","last_became_peered":"2023-11-28T20:50:26.361957+0000","last_unstale":"2023-11-28T20:50:28.517642+0000","last_undegraded":"2023-11-28T20:50:28.517642+0000","last_fullsized":"2023-11-28T20:50:28.517642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:36:06.716305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.517421+0000","last_change":"2023-11-28T20:50:27.402230+0000","last_active":"2023-11-28T20:50:28.517421+0000","last_peered":"2023-11-28T20:50:28.517421+0000","last_clean":"2023-11-28T20:50:28.517421+0000","last_became_active":"2023-11-28T20:50:27.400924+0000","last_became_peered":"2023-11-28T20:50:27.400924+0000","last_unstale":"2023-11-28T20:50:28.517421+0000","last_undegraded":"2023-11-28T20:50:28.517421+0000","last_fullsized":"2023-11-28T20:50:28.517421+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:55:42.771188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545742+0000","last_change":"2023-11-28T20:50:26.369906+0000","last_active":"2023-11-28T20:50:28.545742+0000","last_peered":"2023-11-28T20:50:28.545742+0000","last_clean":"2023-11-28T20:50:28.545742+0000","last_became_active":"2023-11-28T20:50:26.369573+0000","last_became_peered":"2023-11-28T20:50:26.369573+0000","last_unstale":"2023-11-28T20:50:28.545742+0000","last_undegraded":"2023-11-28T20:50:28.545742+0000","last_fullsized":"2023-11-28T20:50:28.545742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:55:41.757972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548885+0000","last_change":"2023-11-28T20:50:27.396055+0000","last_active":"2023-11-28T20:50:28.548885+0000","last_peered":"2023-11-28T20:50:28.548885+0000","last_clean":"2023-11-28T20:50:28.548885+0000","last_became_active":"2023-11-28T20:50:27.395785+0000","last_became_peered":"2023-11-28T20:50:27.395785+0000","last_unstale":"2023-11-28T20:50:28.548885+0000","last_undegraded":"2023-11-28T20:50:28.548885+0000","last_fullsized":"2023-11-28T20:50:28.548885+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:47:33.475972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546368+0000","last_change":"2023-11-28T20:50:26.364878+0000","last_active":"2023-11-28T20:50:28.546368+0000","last_peered":"2023-11-28T20:50:28.546368+0000","last_clean":"2023-11-28T20:50:28.546368+0000","last_became_active":"2023-11-28T20:50:26.364592+0000","last_became_peered":"2023-11-28T20:50:26.364592+0000","last_unstale":"2023-11-28T20:50:28.546368+0000","last_undegraded":"2023-11-28T20:50:28.546368+0000","last_fullsized":"2023-11-28T20:50:28.546368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:01.616206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.546438+0000","last_change":"2023-11-28T20:50:27.392401+0000","last_active":"2023-11-28T20:50:28.546438+0000","last_peered":"2023-11-28T20:50:28.546438+0000","last_clean":"2023-11-28T20:50:28.546438+0000","last_became_active":"2023-11-28T20:50:27.392141+0000","last_became_peered":"2023-11-28T20:50:27.392141+0000","last_unstale":"2023-11-28T20:50:28.546438+0000","last_undegraded":"2023-11-28T20:50:28.546438+0000","last_fullsized":"2023-11-28T20:50:28.546438+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:56:12.254306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.547290+0000","last_change":"2023-11-28T20:50:26.374351+0000","last_active":"2023-11-28T20:50:28.547290+0000","last_peered":"2023-11-28T20:50:28.547290+0000","last_clean":"2023-11-28T20:50:28.547290+0000","last_became_active":"2023-11-28T20:50:26.373855+0000","last_became_peered":"2023-11-28T20:50:26.373855+0000","last_unstale":"2023-11-28T20:50:28.547290+0000","last_undegraded":"2023-11-28T20:50:28.547290+0000","last_fullsized":"2023-11-28T20:50:28.547290+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:01.588986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388915+0000","last_change":"2023-11-28T20:50:27.382619+0000","last_active":"2023-11-28T20:50:28.388915+0000","last_peered":"2023-11-28T20:50:28.388915+0000","last_clean":"2023-11-28T20:50:28.388915+0000","last_became_active":"2023-11-28T20:50:27.382310+0000","last_became_peered":"2023-11-28T20:50:27.382310+0000","last_unstale":"2023-11-28T20:50:28.388915+0000","last_undegraded":"2023-11-28T20:50:28.388915+0000","last_fullsized":"2023-11-28T20:50:28.388915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:43:21.674252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388685+0000","last_change":"2023-11-28T20:50:26.358481+0000","last_active":"2023-11-28T20:50:28.388685+0000","last_peered":"2023-11-28T20:50:28.388685+0000","last_clean":"2023-11-28T20:50:28.388685+0000","last_became_active":"2023-11-28T20:50:26.357789+0000","last_became_peered":"2023-11-28T20:50:26.357789+0000","last_unstale":"2023-11-28T20:50:28.388685+0000","last_undegraded":"2023-11-28T20:50:28.388685+0000","last_fullsized":"2023-11-28T20:50:28.388685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:47:26.140403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.545329+0000","last_change":"2023-11-28T20:50:27.394510+0000","last_active":"2023-11-28T20:50:28.545329+0000","last_peered":"2023-11-28T20:50:28.545329+0000","last_clean":"2023-11-28T20:50:28.545329+0000","last_became_active":"2023-11-28T20:50:27.391591+0000","last_became_peered":"2023-11-28T20:50:27.391591+0000","last_unstale":"2023-11-28T20:50:28.545329+0000","last_undegraded":"2023-11-28T20:50:28.545329+0000","last_fullsized":"2023-11-28T20:50:28.545329+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:47:28.185099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388645+0000","last_change":"2023-11-28T20:50:26.353717+0000","last_active":"2023-11-28T20:50:28.388645+0000","last_peered":"2023-11-28T20:50:28.388645+0000","last_clean":"2023-11-28T20:50:28.388645+0000","last_became_active":"2023-11-28T20:50:26.353299+0000","last_became_peered":"2023-11-28T20:50:26.353299+0000","last_unstale":"2023-11-28T20:50:28.388645+0000","last_undegraded":"2023-11-28T20:50:28.388645+0000","last_fullsized":"2023-11-28T20:50:28.388645+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:40.306123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.386232+0000","last_change":"2023-11-28T20:50:27.387954+0000","last_active":"2023-11-28T20:50:28.386232+0000","last_peered":"2023-11-28T20:50:28.386232+0000","last_clean":"2023-11-28T20:50:28.386232+0000","last_became_active":"2023-11-28T20:50:27.387669+0000","last_became_peered":"2023-11-28T20:50:27.387669+0000","last_unstale":"2023-11-28T20:50:28.386232+0000","last_undegraded":"2023-11-28T20:50:28.386232+0000","last_fullsized":"2023-11-28T20:50:28.386232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:28:47.506318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.544792+0000","last_change":"2023-11-28T20:50:26.358155+0000","last_active":"2023-11-28T20:50:28.544792+0000","last_peered":"2023-11-28T20:50:28.544792+0000","last_clean":"2023-11-28T20:50:28.544792+0000","last_became_active":"2023-11-28T20:50:26.357813+0000","last_became_peered":"2023-11-28T20:50:26.357813+0000","last_unstale":"2023-11-28T20:50:28.544792+0000","last_undegraded":"2023-11-28T20:50:28.544792+0000","last_fullsized":"2023-11-28T20:50:28.544792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:27:03.253504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.548493+0000","last_change":"2023-11-28T20:50:27.384864+0000","last_active":"2023-11-28T20:50:28.548493+0000","last_peered":"2023-11-28T20:50:28.548493+0000","last_clean":"2023-11-28T20:50:28.548493+0000","last_became_active":"2023-11-28T20:50:27.384565+0000","last_became_peered":"2023-11-28T20:50:27.384565+0000","last_unstale":"2023-11-28T20:50:28.548493+0000","last_undegraded":"2023-11-28T20:50:28.548493+0000","last_fullsized":"2023-11-28T20:50:28.548493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:26.322371+0000","last_clean_scrub_stamp":"2023-11-28T20:50:26.322371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:55.727711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-28T20:50:28.388277+0000","last_change":"2023-11-28T20:50:26.360824+0000","last_active":"2023-11-28T20:50:28.388277+0000","last_peered":"2023-11-28T20:50:28.388277+0000","last_clean":"2023-11-28T20:50:28.388277+0000","last_became_active":"2023-11-28T20:50:26.360330+0000","last_became_peered":"2023-11-28T20:50:26.360330+0000","last_unstale":"2023-11-28T20:50:28.388277+0000","last_undegraded":"2023-11-28T20:50:28.388277+0000","last_fullsized":"2023-11-28T20:50:28.388277+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:50:25.309155+0000","last_clean_scrub_stamp":"2023-11-28T20:50:25.309155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:48:15.075283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":389,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-11-28T20:50:27.365265+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:50:27.365265+0000","last_peered":"2023-11-28T20:50:27.365265+0000","last_clean":"2023-11-28T20:50:27.365265+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:50:27.365265+0000","last_undegraded":"2023-11-28T20:50:27.365265+0000","last_fullsized":"2023-11-28T20:50:27.365265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575048,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59196,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312644,"statfs":{"total":96636764160,"available":96576147456,"internally_reserved":0,"allocated":1679360,"data_stored":1126689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1919999999999999,"15min":1.194},"min":{"1min":0.79700000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.722,"5min":1.722,"15min":2.23},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.131,"5min":1.153,"15min":1.1799999999999999},"min":{"1min":0.84299999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5169999999999999,"5min":1.681,"15min":1.726},"last":1.141}]},{"osd":1,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1830000000000001,"15min":1.1850000000000001},"min":{"1min":0.83099999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.579,"5min":1.744,"15min":1.744},"last":1.24},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.92100000000000004,"5min":0.624,"15min":0.624},"max":{"1min":1.6679999999999999,"5min":1.835,"15min":2.3599999999999999},"last":1.085}]},{"osd":2,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.2949999999999999,"15min":1.3029999999999999},"min":{"1min":0.97199999999999998,"5min":0.80300000000000005,"15min":0.58799999999999997},"max":{"1min":1.655,"5min":1.8300000000000001,"15min":1.9359999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.292,"15min":1.2869999999999999},"min":{"1min":0.96899999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.675,"5min":1.847,"15min":2.137},"last":1.4770000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2649999999999999,"15min":1.276},"min":{"1min":0.73199999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.577,"5min":1.7729999999999999,"15min":1.8460000000000001},"last":1.9950000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.403,"15min":1.4059999999999999},"min":{"1min":1.1579999999999999,"5min":0.88300000000000001,"15min":0.68300000000000005},"max":{"1min":1.956,"5min":1.956,"15min":2.4220000000000002},"last":1.5189999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.2629999999999999,"15min":1.264},"min":{"1min":0.83999999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7649999999999999,"5min":1.7989999999999999,"15min":1.919},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3200000000000001,"15min":1.329},"min":{"1min":0.93000000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.794,"5min":1.863,"15min":2.0049999999999999},"last":1.8560000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":0.97199999999999998,"5min":0.86499999999999999,"15min":0.67900000000000005},"max":{"1min":1.891,"5min":1.9219999999999999,"15min":2.5449999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.47,"15min":1.46},"min":{"1min":1.1160000000000001,"5min":0.73499999999999999,"15min":0.72199999999999998},"max":{"1min":1.931,"5min":1.9550000000000001,"15min":2.1000000000000001},"last":1.6879999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5649999999999999,"15min":1.548},"min":{"1min":1.1779999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.2389999999999999},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4830000000000001,"15min":1.488},"min":{"1min":1.056,"5min":0.94099999999999995,"15min":0.90800000000000003},"max":{"1min":1.9139999999999999,"5min":1.9870000000000001,"15min":2.3079999999999998},"last":2.0840000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575048,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32232,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339608,"statfs":{"total":96636764160,"available":96603758592,"internally_reserved":0,"allocated":1073152,"data_stored":535579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1040000000000001,"15min":1.1140000000000001},"min":{"1min":0.71799999999999997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.125,"15min":1.1319999999999999},"min":{"1min":0.73099999999999998,"5min":0.63500000000000001,"15min":0.627},"max":{"1min":1.452,"5min":1.7549999999999999,"15min":2.0739999999999998},"last":0.99299999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2130000000000001,"15min":1.2569999999999999},"min":{"1min":0.84399999999999997,"5min":0.67200000000000004,"15min":0.52200000000000002},"max":{"1min":1.728,"5min":1.728,"15min":1.8069999999999999},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.359,"15min":1.3700000000000001},"min":{"1min":0.97199999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6850000000000001,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":1.369}]},{"osd":2,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.2709999999999999,"15min":1.2669999999999999},"min":{"1min":0.89400000000000002,"5min":0.71099999999999997,"15min":0.69799999999999995},"max":{"1min":1.631,"5min":1.6879999999999999,"15min":1.71},"last":1.302},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.355,"15min":1.361},"min":{"1min":0.84999999999999998,"5min":0.73199999999999998,"15min":0.60299999999999998},"max":{"1min":1.6439999999999999,"5min":1.851,"15min":1.851},"last":1.0800000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.2729999999999999,"15min":1.3},"min":{"1min":0.96799999999999997,"5min":0.70099999999999996,"15min":0.68799999999999994},"max":{"1min":1.6200000000000001,"5min":1.821,"15min":2.222},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2549999999999999,"15min":1.2669999999999999},"min":{"1min":0.95399999999999996,"5min":0.83899999999999997,"15min":0.751},"max":{"1min":1.635,"5min":1.6599999999999999,"15min":1.7949999999999999},"last":1.4279999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2509999999999999,"15min":1.2729999999999999},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.704,"5min":1.849,"15min":2.7290000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.286,"15min":1.329},"min":{"1min":0.84099999999999997,"5min":0.80800000000000005,"15min":0.76400000000000001},"max":{"1min":1.752,"5min":2.2010000000000001,"15min":3.2519999999999998},"last":1.2210000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3100000000000001,"15min":1.325},"min":{"1min":0.96999999999999997,"5min":0.754,"15min":0.62},"max":{"1min":1.714,"5min":1.714,"15min":2.0539999999999998},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5129999999999999,"15min":1.5469999999999999},"min":{"1min":1.2709999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.804,"5min":1.9179999999999999,"15min":3.0310000000000001},"last":1.476}]},{"osd":7,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.45,"15min":1.494},"min":{"1min":1.014,"5min":0.97799999999999998,"15min":0.86499999999999999},"max":{"1min":1.8080000000000001,"5min":2.1120000000000001,"15min":3.379},"last":1.627},{"interface":"front","average":{"1min":1.522,"5min":1.4490000000000001,"15min":1.462},"min":{"1min":1.0669999999999999,"5min":0.93300000000000005,"15min":0.84599999999999997},"max":{"1min":1.7669999999999999,"5min":1.8859999999999999,"15min":2.0739999999999998},"last":1.9630000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575048,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1073152,"data_stored":535567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.157,"15min":1.1719999999999999},"min":{"1min":0.81999999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6970000000000001,"5min":1.702,"15min":1.768},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1599999999999999,"15min":1.2},"min":{"1min":0.88500000000000001,"5min":0.441,"15min":0.441},"max":{"1min":1.631,"5min":1.774,"15min":1.996},"last":1.3260000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1459999999999999,"15min":1.1519999999999999},"min":{"1min":0.90500000000000003,"5min":0.57399999999999995,"15min":0.48899999999999999},"max":{"1min":1.548,"5min":1.768,"15min":1.784},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.194,"15min":1.2170000000000001},"min":{"1min":0.88400000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.956},"last":1.1599999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2609999999999999,"15min":1.2649999999999999},"min":{"1min":0.91400000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.7130000000000001,"5min":1.8100000000000001,"15min":1.901},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2789999999999999,"15min":1.292},"min":{"1min":0.76700000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.611,"5min":1.7310000000000001,"15min":1.958},"last":1.226}]},{"osd":3,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3129999999999999,"15min":1.3280000000000001},"min":{"1min":0.93999999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.7290000000000001,"5min":1.9350000000000001,"15min":2.298},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.2529999999999999,"15min":1.2689999999999999},"min":{"1min":1.0009999999999999,"5min":0.64000000000000001,"15min":0.54600000000000004},"max":{"1min":1.667,"5min":1.667,"15min":2.1040000000000001},"last":1.3620000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.2989999999999999,"15min":1.298},"min":{"1min":0.91300000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":2.2429999999999999},"last":0.875},{"interface":"front","average":{"1min":1.494,"5min":1.429,"15min":1.427},"min":{"1min":0.93600000000000005,"5min":0.73399999999999999,"15min":0.68100000000000005},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.4850000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3109999999999999,"15min":1.3240000000000001},"min":{"1min":0.871,"5min":0.5,"15min":0.5},"max":{"1min":1.6180000000000001,"5min":1.845,"15min":2.1600000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.4670000000000001,"15min":1.466},"min":{"1min":1.2829999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.7809999999999999,"5min":1.806,"15min":2.339},"last":1.109}]},{"osd":7,"last update":"Tue Nov 28 20:50:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.397,"15min":1.4039999999999999},"min":{"1min":0.97599999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.988,"5min":1.988,"15min":2.0800000000000001},"last":1.54},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.486,"15min":1.48},"min":{"1min":1.0509999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7589999999999999,"5min":1.929,"15min":2.3849999999999998},"last":1.456}]}]},{"osd":4,"up_from":13,"seq":55834575049,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1081344,"data_stored":537157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.139,"15min":1.129},"min":{"1min":0.80300000000000005,"5min":0.66000000000000003,"15min":0.64400000000000002},"max":{"1min":1.4359999999999999,"5min":1.6659999999999999,"15min":1.879},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1339999999999999,"15min":1.1180000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6850000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.117}]},{"osd":1,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.29,"15min":1.3100000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.74299999999999999},"max":{"1min":1.609,"5min":1.7490000000000001,"15min":1.923},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.375,"15min":1.3520000000000001},"min":{"1min":1.026,"5min":0.86599999999999999,"15min":0.76700000000000002},"max":{"1min":1.768,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.94799999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.232,"15min":1.2110000000000001},"min":{"1min":1.0229999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.589,"5min":1.71,"15min":1.71},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.278,"15min":1.2949999999999999},"min":{"1min":1.083,"5min":0.77400000000000002,"15min":0.77200000000000002},"max":{"1min":1.5189999999999999,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.4970000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.294,"15min":1.2869999999999999},"min":{"1min":0.97499999999999998,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.76,"5min":2.157,"15min":2.157},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.361,"15min":1.377},"min":{"1min":1.087,"5min":0.876,"15min":0.67900000000000005},"max":{"1min":1.718,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.552}]},{"osd":5,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.3260000000000001,"15min":1.278},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.65800000000000003},"max":{"1min":1.655,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.649},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3620000000000001,"15min":1.2969999999999999},"min":{"1min":0.84099999999999997,"5min":0.83799999999999997,"15min":0.69299999999999995},"max":{"1min":2.0579999999999998,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.347}]},{"osd":6,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3979999999999999,"15min":1.401},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.628},"max":{"1min":1.8049999999999999,"5min":2.04,"15min":2.04},"last":1.629},{"interface":"front","average":{"1min":1.526,"5min":1.5780000000000001,"15min":1.5489999999999999},"min":{"1min":1.3120000000000001,"5min":1.0369999999999999,"15min":0.80600000000000005},"max":{"1min":1.8380000000000001,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":1.7290000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4930000000000001,"15min":1.45},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.76200000000000001},"max":{"1min":1.9570000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.538,"15min":1.526},"min":{"1min":1.208,"5min":1.0349999999999999,"15min":0.72299999999999998},"max":{"1min":1.8580000000000001,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.6819999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40552,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":1069056,"data_stored":535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.88700000000000001,"15min":0.91400000000000003},"min":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.296,"5min":4.7210000000000001,"15min":4.7210000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0069999999999999,"15min":1.083},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":3.968},"last":1.4870000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97299999999999998,"15min":0.98899999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.44600000000000001},"max":{"1min":1.593,"5min":2.8460000000000001,"15min":3.9180000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.173,"15min":1.2110000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7190000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.0600000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2470000000000001,"15min":1.2609999999999999},"min":{"1min":0.53100000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.8169999999999999,"5min":3.444,"15min":4.6130000000000004},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.276,"5min":1.3129999999999999,"15min":1.3460000000000001},"min":{"1min":0.63900000000000001,"5min":0.58099999999999996,"15min":0.54400000000000004},"max":{"1min":1.7130000000000001,"5min":3.7050000000000001,"15min":4.5759999999999996},"last":1.643}]},{"osd":4,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3149999999999999,"15min":1.3480000000000001},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.72299999999999998},"max":{"1min":1.7889999999999999,"5min":1.8779999999999999,"15min":4.1779999999999999},"last":1.204},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3500000000000001,"15min":1.3939999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.8160000000000001,"5min":1.891,"15min":4.492},"last":1.1579999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4390000000000001,"15min":1.4590000000000001},"min":{"1min":0.84399999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7250000000000001,"5min":3.7730000000000001,"15min":4.7169999999999996},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.395,"15min":1.4419999999999999},"min":{"1min":0.66800000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0110000000000001,"5min":2.2429999999999999,"15min":4.2889999999999997},"last":1.607}]},{"osd":6,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.4330000000000001,"15min":1.4930000000000001},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.72799999999999998},"max":{"1min":1.8899999999999999,"5min":2.3100000000000001,"15min":5.2279999999999998},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.556,"15min":1.5960000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.855,"5min":3.661,"15min":6.9740000000000002},"last":1.5369999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.484,"15min":1.5489999999999999},"min":{"1min":0.92400000000000004,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.907,"5min":2.1469999999999998,"15min":4.0819999999999999},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.544,"15min":1.609},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.76200000000000001},"max":{"1min":1.976,"5min":2.3919999999999999,"15min":4.1399999999999997},"last":1.2829999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575049,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58600,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313240,"statfs":{"total":96636764160,"available":96576757760,"internally_reserved":0,"allocated":1069056,"data_stored":535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.88300000000000001,"15min":0.91200000000000003},"min":{"1min":0.54200000000000004,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":1.036},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.996,"15min":1.0589999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.486,"5min":1.6419999999999999,"15min":1.9390000000000001},"last":0.86699999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95399999999999996,"15min":0.98499999999999999},"min":{"1min":0.60799999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.403,"5min":1.7669999999999999,"15min":3.2599999999999998},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.099,"15min":1.1499999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.81599999999999995}]},{"osd":3,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1240000000000001,"15min":1.1579999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4770000000000001,"5min":1.774,"15min":2.194},"last":1.139},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2390000000000001,"15min":1.2929999999999999},"min":{"1min":0.53500000000000003,"5min":0.48199999999999998,"15min":0.45600000000000002},"max":{"1min":1.6579999999999999,"5min":1.831,"15min":3.1440000000000001},"last":0.75900000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.389,"15min":1.411},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.853,"5min":2.23,"15min":2.8919999999999999},"last":1.52},{"interface":"front","average":{"1min":1.389,"5min":1.387,"15min":1.4119999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.988,"5min":2.3740000000000001,"15min":2.3809999999999998},"last":1.4350000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.3919999999999999,"15min":1.421},"min":{"1min":0.85099999999999998,"5min":0.82199999999999995,"15min":0.79600000000000004},"max":{"1min":2.327,"5min":2.327,"15min":2.327},"last":1.272},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4650000000000001,"15min":1.496},"min":{"1min":0.96499999999999997,"5min":0.89100000000000001,"15min":0.75800000000000001},"max":{"1min":2.077,"5min":2.1680000000000001,"15min":3.4279999999999999},"last":1.619}]},{"osd":6,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.452,"15min":1.502},"min":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.165,"5min":2.1709999999999998,"15min":2.621},"last":1.323},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.526,"15min":1.5740000000000001},"min":{"1min":0.89000000000000001,"5min":0.82499999999999996,"15min":0.78300000000000003},"max":{"1min":2.6680000000000001,"5min":2.6749999999999998,"15min":3.6269999999999998},"last":1.6830000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.508,"15min":1.542},"min":{"1min":0.94299999999999995,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.54,"15min":1.5900000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.2480000000000002,"5min":2.4849999999999999,"15min":2.7109999999999999},"last":1.5669999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335424,"statfs":{"total":96636764160,"available":96599474176,"internally_reserved":0,"allocated":1097728,"data_stored":537921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.88900000000000001,"15min":0.90400000000000003},"min":{"1min":0.67600000000000005,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4419999999999999,"5min":1.6639999999999999,"15min":1.9259999999999999},"last":1.103},{"interface":"front","average":{"1min":1.03,"5min":1.0589999999999999,"15min":1.077},"min":{"1min":0.72999999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":0.75600000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0389999999999999,"15min":1.089},"min":{"1min":0.66200000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.4850000000000001,"5min":1.619,"15min":2.234},"last":1.248},{"interface":"front","average":{"1min":1.29,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.88300000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.895},"last":1.399}]},{"osd":3,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.26,"15min":1.3160000000000001},"min":{"1min":0.64400000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.748,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.083,"15min":1.1120000000000001},"min":{"1min":0.84199999999999997,"5min":0.37,"15min":0.37},"max":{"1min":1.744,"5min":1.887,"15min":2.3679999999999999},"last":0.84299999999999997}]},{"osd":4,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3220000000000001,"15min":1.337},"min":{"1min":0.97399999999999998,"5min":0.79200000000000004,"15min":0.76600000000000001},"max":{"1min":1.742,"5min":2.1019999999999999,"15min":2.1150000000000002},"last":1.149},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.343,"15min":1.387},"min":{"1min":1.0349999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8799999999999999,"5min":2.1469999999999998,"15min":2.6819999999999999},"last":1.5169999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.4059999999999999,"15min":1.429},"min":{"1min":1.165,"5min":0.81100000000000005,"15min":0.81000000000000005},"max":{"1min":1.8620000000000001,"5min":1.903,"15min":2.1840000000000002},"last":1.647},{"interface":"front","average":{"1min":1.532,"5min":1.4870000000000001,"15min":1.496},"min":{"1min":1.2130000000000001,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.8280000000000001,"5min":2.294,"15min":2.294},"last":1.583}]},{"osd":6,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.3819999999999999,"15min":1.4199999999999999},"min":{"1min":1.0980000000000001,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7749999999999999,"5min":1.8759999999999999,"15min":1.9379999999999999},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.4990000000000001,"15min":1.524},"min":{"1min":1.3120000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.956,"5min":2.1520000000000001,"15min":2.3300000000000001},"last":1.7509999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.542,"15min":1.5640000000000001},"min":{"1min":1.165,"5min":0.753,"15min":0.753},"max":{"1min":2.1429999999999998,"5min":2.218,"15min":2.3359999999999999},"last":1.681},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.504,"15min":1.5169999999999999},"min":{"1min":1.3320000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":2.294},"last":1.464}]}]},{"osd":0,"up_from":13,"seq":55834575049,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36964,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334876,"statfs":{"total":96636764160,"available":96598913024,"internally_reserved":0,"allocated":1662976,"data_stored":1125913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91500000000000004,"15min":0.90800000000000003},"min":{"1min":0.52100000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3240000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.75},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0509999999999999,"15min":1.0800000000000001},"min":{"1min":0.54200000000000004,"5min":0.501,"15min":0.45700000000000002},"max":{"1min":1.534,"5min":1.764,"15min":1.764},"last":1.0580000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.054,"15min":1.1060000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.6080000000000001,"5min":1.6220000000000001,"15min":1.7709999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1950000000000001},"min":{"1min":0.59899999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.722,"5min":1.893,"15min":2.4740000000000002},"last":0.93100000000000005}]},{"osd":3,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0660000000000001,"15min":1.1180000000000001},"min":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.508,"5min":2.081,"15min":2.081},"last":1.008},{"interface":"front","average":{"1min":1.262,"5min":1.24,"15min":1.3149999999999999},"min":{"1min":0.67700000000000005,"5min":0.54400000000000004,"15min":0.52500000000000002},"max":{"1min":1.8180000000000001,"5min":1.8839999999999999,"15min":2.1080000000000001},"last":1.1120000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3979999999999999,"15min":1.417},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.804,"5min":2.012,"15min":2.012},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.385,"15min":1.3740000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.68300000000000005},"max":{"1min":1.7230000000000001,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.1459999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.403,"15min":1.4159999999999999},"min":{"1min":0.84799999999999998,"5min":0.80100000000000005,"15min":0.79400000000000004},"max":{"1min":1.8460000000000001,"5min":2.0350000000000001,"15min":2.528},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.462,"15min":1.484},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.992,"5min":2.0750000000000002,"15min":2.2490000000000001},"last":1.266}]},{"osd":6,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.512,"15min":1.5489999999999999},"min":{"1min":0.78800000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.0419999999999998,"5min":2.1160000000000001,"15min":2.2959999999999998},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.504,"15min":1.5249999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2330000000000001,"5min":2.274,"15min":2.573},"last":1.673}]},{"osd":7,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5720000000000001,"15min":1.591},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.2970000000000002,"5min":2.3610000000000002,"15min":2.4100000000000001},"last":1.734},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5049999999999999,"15min":1.5429999999999999},"min":{"1min":0.875,"5min":0.81699999999999995,"15min":0.73799999999999999},"max":{"1min":1.776,"5min":2.3460000000000001,"15min":2.6640000000000001},"last":1.339}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:50:32.408 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T20:50:32.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T20:50:32.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.683+0000 7f813bb90640 1 Processor -- start 2023-11-28T20:50:32.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.683+0000 7f813bb90640 1 -- start start 2023-11-28T20:50:32.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.684+0000 7f813bb90640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f813415e840 0x7f813406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.684+0000 7f813bb90640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813415ecd0 con 0x7f813415f720 2023-11-28T20:50:32.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.684+0000 7f813bb90640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813415f290 con 0x7f813406ac10 2023-11-28T20:50:32.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.684+0000 7f813bb90640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813415fb60 con 0x7f813415e840 2023-11-28T20:50:32.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.684+0000 7f813912b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f813415f720 0x7f813406a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42848/0 (socket says 172.21.15.45:42848) 2023-11-28T20:50:32.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.684+0000 7f813912b640 1 -- 172.21.15.45:0/750004316 learned_addr learned my addr 172.21.15.45:0/750004316 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:32.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054910715 0 0) 0x7f813415ecd0 con 0x7f813415f720 2023-11-28T20:50:32.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f811c003520 con 0x7f813415f720 2023-11-28T20:50:32.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813992c640 1 --2- 172.21.15.45:0/750004316 >> v2:172.21.15.190:3300/0 conn(0x7f813415e840 0x7f813406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2217783877 0 0) 0x7f811c003520 con 0x7f813415f720 2023-11-28T20:50:32.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 >> v2:172.21.15.190:3300/0 conn(0x7f813415e840 msgr2=0x7f813406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:32.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 --2- 172.21.15.45:0/750004316 >> v2:172.21.15.190:3300/0 conn(0x7f813415e840 0x7f813406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 >> v1:172.21.15.190:6789/0 conn(0x7f813406ac10 legacy=0x7f81340cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8134072020 con 0x7f813415f720 2023-11-28T20:50:32.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f81280019f0 con 0x7f813415f720 2023-11-28T20:50:32.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 >> v1:172.21.15.45:6789/0 conn(0x7f813415f720 legacy=0x7f813406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f811c0039c0 0x7f811c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811c0047c0 0x7f811c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f811c0051d0 0x7f811c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.685+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813415fb60 con 0x7f811c0039c0 2023-11-28T20:50:32.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c00c120 con 0x7f811c0047c0 2023-11-28T20:50:32.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811c00c2e0 con 0x7f811c0051d0 2023-11-28T20:50:32.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813992c640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f811c0039c0 0x7f811c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813912b640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811c0047c0 0x7f811c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813992c640 1 -- 172.21.15.45:0/750004316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f811c0051d0 msgr2=0x7f811c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:32.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813992c640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f811c0051d0 0x7f811c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813992c640 1 -- 172.21.15.45:0/750004316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811c0047c0 msgr2=0x7f811c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:32.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813992c640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811c0047c0 0x7f811c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813a12d640 1 -- 172.21.15.45:0/750004316 reap_dead start 2023-11-28T20:50:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813992c640 1 -- 172.21.15.45:0/750004316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c003520 con 0x7f811c0039c0 2023-11-28T20:50:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.686+0000 7f813912b640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811c0047c0 0x7f811c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.687+0000 7f813992c640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f811c0039c0 0x7f811c003de0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f812400ba30 tx=0x7f811c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147a68ea0f12684c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124032660 con 0x7f811c0039c0 2023-11-28T20:50:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81240327c0 con 0x7f811c0039c0 2023-11-28T20:50:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813892a640 1 -- 172.21.15.45:0/750004316 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124032b50 con 0x7f811c0039c0 2023-11-28T20:50:32.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813bb90640 1 -- 172.21.15.45:0/750004316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f811c0039c0 msgr2=0x7f811c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813bb90640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f811c0039c0 0x7f811c003de0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f812400ba30 tx=0x7f811c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813bb90640 1 -- 172.21.15.45:0/750004316 shutdown_connections 2023-11-28T20:50:32.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813bb90640 1 --2- 172.21.15.45:0/750004316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f811c0039c0 0x7f811c003de0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813bb90640 1 -- 172.21.15.45:0/750004316 >> 172.21.15.45:0/750004316 conn(0x7f81340ced60 msgr2=0x7f81340cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:32.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813bb90640 1 -- 172.21.15.45:0/750004316 shutdown_connections 2023-11-28T20:50:32.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.688+0000 7f813bb90640 1 -- 172.21.15.45:0/750004316 wait complete. 2023-11-28T20:50:32.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.690+0000 7f813bb90640 1 Processor -- start 2023-11-28T20:50:32.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.690+0000 7f813bb90640 1 -- start start 2023-11-28T20:50:32.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.691+0000 7f813bb90640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f811c0039c0 0x7f813415a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.691+0000 7f813bb90640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f813415d7f0 0x7f813415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.691+0000 7f813bb90640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81340af730 0x7f81340afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.691+0000 7f813bb90640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340b0120 con 0x7f81340af730 2023-11-28T20:50:32.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.691+0000 7f813bb90640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813415c0f0 con 0x7f813415d7f0 2023-11-28T20:50:32.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.691+0000 7f813bb90640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813415c3d0 con 0x7f811c0039c0 2023-11-28T20:50:32.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813a12d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81340af730 0x7f81340afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813a12d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81340af730 0x7f81340afbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53578/0 (socket says 172.21.15.45:53578) 2023-11-28T20:50:32.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813a12d640 1 -- 172.21.15.45:0/2706626123 learned_addr learned my addr 172.21.15.45:0/2706626123 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:32.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813912b640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f813415d7f0 0x7f813415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813992c640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f811c0039c0 0x7f813415a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813a12d640 1 -- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f811c0039c0 msgr2=0x7f813415a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813a12d640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f811c0039c0 0x7f813415a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813a12d640 1 -- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f813415d7f0 msgr2=0x7f813415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:32.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813a12d640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f813415d7f0 0x7f813415dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:32.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813a12d640 1 -- 172.21.15.45:0/2706626123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813415c6b0 con 0x7f81340af730 2023-11-28T20:50:32.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.692+0000 7f813992c640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f811c0039c0 0x7f813415a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:32.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.693+0000 7f813a12d640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81340af730 0x7f81340afbe0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f81340ae6d0 tx=0x7f812c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:32.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.693+0000 7f81327fc640 1 -- 172.21.15.45:0/2706626123 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c004220 con 0x7f81340af730 2023-11-28T20:50:32.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.693+0000 7f81327fc640 1 -- 172.21.15.45:0/2706626123 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c0043a0 con 0x7f81340af730 2023-11-28T20:50:32.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.693+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813415c990 con 0x7f81340af730 2023-11-28T20:50:32.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.693+0000 7f81327fc640 1 -- 172.21.15.45:0/2706626123 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c007310 con 0x7f81340af730 2023-11-28T20:50:32.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.693+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813415d190 con 0x7f81340af730 2023-11-28T20:50:32.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.694+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813415ceb0 con 0x7f81340af730 2023-11-28T20:50:32.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.695+0000 7f81327fc640 1 -- 172.21.15.45:0/2706626123 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f812c007470 con 0x7f81340af730 2023-11-28T20:50:32.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.696+0000 7f81327fc640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f812006af90 0x7f812006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:32.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.696+0000 7f81327fc640 1 -- 172.21.15.45:0/2706626123 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f812c088960 con 0x7f81340af730 2023-11-28T20:50:32.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.697+0000 7f813992c640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f812006af90 0x7f812006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:32.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.698+0000 7f813992c640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f812006af90 0x7f812006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f812400ba00 tx=0x7f812403d040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:32.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.701+0000 7f81327fc640 1 -- 172.21.15.45:0/2706626123 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f813415ceb0 con 0x7f81340af730 2023-11-28T20:50:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.978+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f813415ceb0 con 0x7f81340af730 2023-11-28T20:50:33.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.983+0000 7f81327fc640 1 -- 172.21.15.45:0/2706626123 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f813415ceb0 con 0x7f81340af730 2023-11-28T20:50:33.196 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T20:50:33.196 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:50:33.196 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T20:50:33.196 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T20:50:33.196 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T20:50:33.197 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T20:50:33.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.986+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f812006af90 msgr2=0x7f812006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.986+0000 7f813bb90640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f812006af90 0x7f812006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f812400ba00 tx=0x7f812403d040 comp rx=0 tx=0).stop 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.986+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81340af730 msgr2=0x7f81340afbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.986+0000 7f813bb90640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81340af730 0x7f81340afbe0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f81340ae6d0 tx=0x7f812c00c820 comp rx=0 tx=0).stop 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.987+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 shutdown_connections 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.987+0000 7f813bb90640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f812006af90 0x7f812006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.987+0000 7f813bb90640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81340af730 0x7f81340afbe0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.987+0000 7f813bb90640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f813415d7f0 0x7f813415dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.987+0000 7f813bb90640 1 --2- 172.21.15.45:0/2706626123 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f811c0039c0 0x7f813415a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.987+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 >> 172.21.15.45:0/2706626123 conn(0x7f81340ced60 msgr2=0x7f813409bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.987+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 shutdown_connections 2023-11-28T20:50:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:32.987+0000 7f813bb90640 1 -- 172.21.15.45:0/2706626123 wait complete. 2023-11-28T20:50:33.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:33.237 INFO:tasks.ceph.mds.d.smithi190.stderr:2023-11-28T20:50:33.235+0000 7f8505e3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:50:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.437+0000 7f8479711640 1 Processor -- start 2023-11-28T20:50:33.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.437+0000 7f8479711640 1 -- start start 2023-11-28T20:50:33.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.439+0000 7f8479711640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84740b3870 0x7f847415db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:33.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.439+0000 7f8479711640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f847406d880 0x7f847415e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:33.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.439+0000 7f8479711640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847415e680 con 0x7f84740721e0 2023-11-28T20:50:33.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.439+0000 7f8479711640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847415a460 con 0x7f84740b3870 2023-11-28T20:50:33.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.439+0000 7f8479711640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847415e960 con 0x7f847406d880 2023-11-28T20:50:33.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.439+0000 7f8473fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84740b3870 0x7f847415db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:33.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.439+0000 7f846b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f84740721e0 0x7f847406d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42854/0 (socket says 172.21.15.45:42854) 2023-11-28T20:50:33.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.439+0000 7f8473fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84740b3870 0x7f847415db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53584/0 (socket says 172.21.15.45:53584) 2023-11-28T20:50:33.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.439+0000 7f846b7fe640 1 -- 172.21.15.45:0/387529241 learned_addr learned my addr 172.21.15.45:0/387529241 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:33.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.440+0000 7f8473fff640 1 -- 172.21.15.45:0/387529241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f847406d880 msgr2=0x7f847415e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:33.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.440+0000 7f8473fff640 1 --2- 172.21.15.45:0/387529241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f847406d880 0x7f847415e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.440+0000 7f8473fff640 1 -- 172.21.15.45:0/387529241 >> v1:172.21.15.45:6789/0 conn(0x7f84740721e0 legacy=0x7f847406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:33.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.440+0000 7f8473fff640 1 -- 172.21.15.45:0/387529241 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8474071700 con 0x7f84740b3870 2023-11-28T20:50:33.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.440+0000 7f84737fe640 1 -- 172.21.15.45:0/387529241 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730599627 0 0) 0x7f847415e680 con 0x7f84740721e0 2023-11-28T20:50:33.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.441+0000 7f8473fff640 1 --2- 172.21.15.45:0/387529241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84740b3870 0x7f847415db10 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8464009d90 tx=0x7f846402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce2a7a7355040bc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.441+0000 7f84737fe640 1 -- 172.21.15.45:0/387529241 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846402cc60 con 0x7f84740b3870 2023-11-28T20:50:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.441+0000 7f84737fe640 1 -- 172.21.15.45:0/387529241 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846402cdc0 con 0x7f84740b3870 2023-11-28T20:50:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.441+0000 7f84737fe640 1 -- 172.21.15.45:0/387529241 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8464034660 con 0x7f84740b3870 2023-11-28T20:50:33.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.442+0000 7f8479711640 1 -- 172.21.15.45:0/387529241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84740b3870 msgr2=0x7f847415db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.442+0000 7f8479711640 1 --2- 172.21.15.45:0/387529241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84740b3870 0x7f847415db10 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8464009d90 tx=0x7f846402beb0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.442+0000 7f8479711640 1 -- 172.21.15.45:0/387529241 shutdown_connections 2023-11-28T20:50:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.442+0000 7f8479711640 1 --2- 172.21.15.45:0/387529241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f847406d880 0x7f847415e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.442+0000 7f8479711640 1 --2- 172.21.15.45:0/387529241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84740b3870 0x7f847415db10 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.442+0000 7f8479711640 1 -- 172.21.15.45:0/387529241 >> 172.21.15.45:0/387529241 conn(0x7f847406ab30 msgr2=0x7f84740b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.442+0000 7f8479711640 1 -- 172.21.15.45:0/387529241 shutdown_connections 2023-11-28T20:50:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.442+0000 7f8479711640 1 -- 172.21.15.45:0/387529241 wait complete. 2023-11-28T20:50:33.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.444+0000 7f8479711640 1 Processor -- start 2023-11-28T20:50:33.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.444+0000 7f8479711640 1 -- start start 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.445+0000 7f8479711640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847406d880 0x7f84741e9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.445+0000 7f8479711640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84740721e0 0x7f84741fa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.445+0000 7f8479711640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84740b3870 0x7f847414dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.445+0000 7f8479711640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847414e1f0 con 0x7f847406d880 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.445+0000 7f8479711640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847414e450 con 0x7f84740721e0 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.445+0000 7f8479711640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847414e730 con 0x7f84740b3870 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.445+0000 7f8473fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847406d880 0x7f84741e9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.445+0000 7f8473fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847406d880 0x7f84741e9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53600/0 (socket says 172.21.15.45:53600) 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.445+0000 7f8473fff640 1 -- 172.21.15.45:0/2615700931 learned_addr learned my addr 172.21.15.45:0/2615700931 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.446+0000 7f8473fff640 1 -- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84740b3870 msgr2=0x7f847414dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:33.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.446+0000 7f8473fff640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84740b3870 0x7f847414dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.446+0000 7f8473fff640 1 -- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84740721e0 msgr2=0x7f84741fa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.446+0000 7f8473fff640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84740721e0 0x7f84741fa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.446+0000 7f8473fff640 1 -- 172.21.15.45:0/2615700931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8464009a70 con 0x7f847406d880 2023-11-28T20:50:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.446+0000 7f8473fff640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847406d880 0x7f84741e9510 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8464009d90 tx=0x7f8464004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.447+0000 7f84717fa640 1 -- 172.21.15.45:0/2615700931 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846403e610 con 0x7f847406d880 2023-11-28T20:50:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.447+0000 7f84717fa640 1 -- 172.21.15.45:0/2615700931 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846403e770 con 0x7f847406d880 2023-11-28T20:50:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.447+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847414ea10 con 0x7f847406d880 2023-11-28T20:50:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.447+0000 7f84717fa640 1 -- 172.21.15.45:0/2615700931 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846403eb20 con 0x7f847406d880 2023-11-28T20:50:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.447+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847414efd0 con 0x7f847406d880 2023-11-28T20:50:33.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.448+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847409d0f0 con 0x7f847406d880 2023-11-28T20:50:33.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.449+0000 7f84717fa640 1 -- 172.21.15.45:0/2615700931 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f846403b050 con 0x7f847406d880 2023-11-28T20:50:33.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.450+0000 7f84717fa640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f845406af90 0x7f845406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:33.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.450+0000 7f84717fa640 1 -- 172.21.15.45:0/2615700931 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f84640ac0f0 con 0x7f847406d880 2023-11-28T20:50:33.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.450+0000 7f846b7fe640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f845406af90 0x7f845406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:33.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.452+0000 7f846b7fe640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f845406af90 0x7f845406d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f845c007fd0 tx=0x7f845c007920 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:33.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.454+0000 7f84717fa640 1 -- 172.21.15.45:0/2615700931 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8464076e30 con 0x7f847406d880 2023-11-28T20:50:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.687+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8474159ef0 con 0x7f847406d880 2023-11-28T20:50:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.688+0000 7f84717fa640 1 -- 172.21.15.45:0/2615700931 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4914 (secure 0 0 0) 0x7f846407a450 con 0x7f847406d880 2023-11-28T20:50:33.691 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:33.691 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5438,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2396470524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2396470524},{"type":"v1","addr":"172.21.15.45:6835","nonce":2396470524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/436103276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":436103276},{"type":"v1","addr":"172.21.15.190:6836","nonce":436103276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/834328551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":834328551},{"type":"v1","addr":"172.21.15.45:6834","nonce":834328551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:50:26.812959+0000","modified":"2023-11-28T20:50:29.910521+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6837/3032907708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3032907708},{"type":"v1","addr":"172.21.15.190:6837","nonce":3032907708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T20:50:33.691 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 36 2023-11-28T20:50:33.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f845406af90 msgr2=0x7f845406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f845406af90 0x7f845406d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f845c007fd0 tx=0x7f845c007920 comp rx=0 tx=0).stop 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847406d880 msgr2=0x7f84741e9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847406d880 0x7f84741e9510 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8464009d90 tx=0x7f8464004a40 comp rx=0 tx=0).stop 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 shutdown_connections 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f845406af90 0x7f845406d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84740b3870 0x7f847414dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84740721e0 0x7f84741fa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 --2- 172.21.15.45:0/2615700931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847406d880 0x7f84741e9510 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 >> 172.21.15.45:0/2615700931 conn(0x7f847406ab30 msgr2=0x7f847406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.691+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 shutdown_connections 2023-11-28T20:50:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:33.692+0000 7f8479711640 1 -- 172.21.15.45:0/2615700931 wait complete. 2023-11-28T20:50:34.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:50:26.812959+0000', 'modified': '2023-11-28T20:50:29.910521+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6729}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6729': {'gid': 6729, 'name': 'd', 'rank': 0, 'incarnation': 32, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.190:6837/3032907708', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6835', 'nonce': 3032907708}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 3032907708}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T20:50:34.115 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:50:34.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-11-28T20:50:34.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.356+0000 7f8058e10640 1 Processor -- start 2023-11-28T20:50:34.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.356+0000 7f8058e10640 1 -- start start 2023-11-28T20:50:34.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.358+0000 7f8058e10640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f805415b820 0x7f805406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.358+0000 7f8058e10640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 0x7f80540cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.358+0000 7f8058e10640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805415fa80 con 0x7f805415edc0 2023-11-28T20:50:34.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.358+0000 7f8058e10640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805415fd60 con 0x7f805415b820 2023-11-28T20:50:34.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.358+0000 7f8058e10640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80540cf9f0 con 0x7f80540cf020 2023-11-28T20:50:34.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f8052d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f805415edc0 0x7f805415f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39590/0 (socket says 172.21.15.45:39590) 2023-11-28T20:50:34.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f8052d9d640 1 -- 172.21.15.45:0/271883637 learned_addr learned my addr 172.21.15.45:0/271883637 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:34.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f805259c640 1 --2- 172.21.15.45:0/271883637 >> v2:172.21.15.190:3300/0 conn(0x7f805415b820 0x7f805406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f804bfff640 1 --2- 172.21.15.45:0/271883637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 0x7f80540cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f8051d9b640 1 -- 172.21.15.45:0/271883637 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965535153 0 0) 0x7f805415fa80 con 0x7f805415edc0 2023-11-28T20:50:34.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f8051d9b640 1 -- 172.21.15.45:0/271883637 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8040003520 con 0x7f805415edc0 2023-11-28T20:50:34.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f805259c640 1 -- 172.21.15.45:0/271883637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 msgr2=0x7f80540cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f8051d9b640 1 -- 172.21.15.45:0/271883637 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 80111474 0 0) 0x7f8040003520 con 0x7f805415edc0 2023-11-28T20:50:34.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f805259c640 1 --2- 172.21.15.45:0/271883637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 0x7f80540cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.359+0000 7f805259c640 1 -- 172.21.15.45:0/271883637 >> v1:172.21.15.45:6789/0 conn(0x7f805415edc0 legacy=0x7f805415f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.360+0000 7f805259c640 1 -- 172.21.15.45:0/271883637 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805406cf40 con 0x7f805415b820 2023-11-28T20:50:34.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.361+0000 7f805259c640 1 --2- 172.21.15.45:0/271883637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f805415b820 0x7f805406da20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f80440097a0 tx=0x7f804402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9ff3a8ebec19ad9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.361+0000 7f8051d9b640 1 -- 172.21.15.45:0/271883637 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c0019f0 con 0x7f805415b820 2023-11-28T20:50:34.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8051d9b640 1 -- 172.21.15.45:0/271883637 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804402e050 con 0x7f805415b820 2023-11-28T20:50:34.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8051d9b640 1 -- 172.21.15.45:0/271883637 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044030730 con 0x7f805415b820 2023-11-28T20:50:34.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8058e10640 1 -- 172.21.15.45:0/271883637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f805415b820 msgr2=0x7f805406da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8058e10640 1 --2- 172.21.15.45:0/271883637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f805415b820 0x7f805406da20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f80440097a0 tx=0x7f804402b790 comp rx=0 tx=0).stop 2023-11-28T20:50:34.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8058e10640 1 -- 172.21.15.45:0/271883637 shutdown_connections 2023-11-28T20:50:34.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8058e10640 1 --2- 172.21.15.45:0/271883637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 0x7f80540cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8058e10640 1 --2- 172.21.15.45:0/271883637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f805415b820 0x7f805406da20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8058e10640 1 -- 172.21.15.45:0/271883637 >> 172.21.15.45:0/271883637 conn(0x7f805406a6e0 msgr2=0x7f805406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:34.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8058e10640 1 -- 172.21.15.45:0/271883637 shutdown_connections 2023-11-28T20:50:34.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.362+0000 7f8058e10640 1 -- 172.21.15.45:0/271883637 wait complete. 2023-11-28T20:50:34.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.364+0000 7f8058e10640 1 Processor -- start 2023-11-28T20:50:34.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.365+0000 7f8058e10640 1 -- start start 2023-11-28T20:50:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.365+0000 7f8058e10640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 0x7f805409c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.365+0000 7f8058e10640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f805415b820 0x7f805414f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8058e10640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805415edc0 0x7f805409a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8058e10640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805409be00 con 0x7f805415edc0 2023-11-28T20:50:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8058e10640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805409b180 con 0x7f805415b820 2023-11-28T20:50:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8058e10640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805409b460 con 0x7f80540cf020 2023-11-28T20:50:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8052d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805415edc0 0x7f805409a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8052d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805415edc0 0x7f805409a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51188/0 (socket says 172.21.15.45:51188) 2023-11-28T20:50:34.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8052d9d640 1 -- 172.21.15.45:0/831415270 learned_addr learned my addr 172.21.15.45:0/831415270 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f805259c640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 0x7f805409c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f804bfff640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f805415b820 0x7f805414f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8052d9d640 1 -- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 msgr2=0x7f805409c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8052d9d640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 0x7f805409c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8052d9d640 1 -- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f805415b820 msgr2=0x7f805414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8052d9d640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f805415b820 0x7f805414f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.366+0000 7f8052d9d640 1 -- 172.21.15.45:0/831415270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8044009400 con 0x7f805415edc0 2023-11-28T20:50:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.367+0000 7f8052d9d640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805415edc0 0x7f805409a8a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f804c00cc70 tx=0x7f804c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.367+0000 7f804affd640 1 -- 172.21.15.45:0/831415270 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c00ce80 con 0x7f805415edc0 2023-11-28T20:50:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.367+0000 7f804affd640 1 -- 172.21.15.45:0/831415270 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c012400 con 0x7f805415edc0 2023-11-28T20:50:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.367+0000 7f804affd640 1 -- 172.21.15.45:0/831415270 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c0127d0 con 0x7f805415edc0 2023-11-28T20:50:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.367+0000 7f8058e10640 1 -- 172.21.15.45:0/831415270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805409b740 con 0x7f805415edc0 2023-11-28T20:50:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.367+0000 7f8058e10640 1 -- 172.21.15.45:0/831415270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805414fbf0 con 0x7f805415edc0 2023-11-28T20:50:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.370+0000 7f804affd640 1 -- 172.21.15.45:0/831415270 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f804c007070 con 0x7f805415edc0 2023-11-28T20:50:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.371+0000 7f804affd640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f803406af90 0x7f803406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.371+0000 7f804affd640 1 -- 172.21.15.45:0/831415270 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f804c089b60 con 0x7f805415edc0 2023-11-28T20:50:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.371+0000 7f805259c640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f803406af90 0x7f803406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.372+0000 7f805b9a3740 1 Processor -- start 2023-11-28T20:50:34.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.372+0000 7f805b9a3740 1 -- start start 2023-11-28T20:50:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 0x55dc8c67a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f805259c640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f803406af90 0x7f803406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8044034000 tx=0x7f804402f000 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c674ba0 0x55dc8c6790a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c6bd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f805b9a3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c5d8880 con 0x55dc8c674ba0 2023-11-28T20:50:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f805b9a3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c5e7ae0 con 0x55dc8c679ea0 2023-11-28T20:50:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f805b9a3740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c5e51d0 con 0x55dc8c6795e0 2023-11-28T20:50:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f804bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c674ba0 0x55dc8c6790a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f804bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c674ba0 0x55dc8c6790a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51194/0 (socket says 172.21.15.45:51194) 2023-11-28T20:50:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f804bfff640 1 -- 172.21.15.45:0/1113470075 learned_addr learned my addr 172.21.15.45:0/1113470075 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.373+0000 7f805259c640 1 --2- 172.21.15.45:0/1113470075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 0x55dc8c67a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f8052d9d640 1 --2- 172.21.15.45:0/1113470075 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c6bd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f804bfff640 1 -- 172.21.15.45:0/1113470075 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 msgr2=0x55dc8c6bd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f804bfff640 1 --2- 172.21.15.45:0/1113470075 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c6bd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f804bfff640 1 -- 172.21.15.45:0/1113470075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 msgr2=0x55dc8c67a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f804bfff640 1 --2- 172.21.15.45:0/1113470075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 0x55dc8c67a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f804bfff640 1 -- 172.21.15.45:0/1113470075 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc8c5a8860 con 0x55dc8c674ba0 2023-11-28T20:50:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f804bfff640 1 --2- 172.21.15.45:0/1113470075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c674ba0 0x55dc8c6790a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f803c009b20 tx=0x7f803c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae878447cc09e36 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f8033fff640 1 -- 172.21.15.45:0/1113470075 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c010dc0 con 0x55dc8c674ba0 2023-11-28T20:50:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f8033fff640 1 -- 172.21.15.45:0/1113470075 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803c0143f0 con 0x55dc8c674ba0 2023-11-28T20:50:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.374+0000 7f8033fff640 1 -- 172.21.15.45:0/1113470075 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c014780 con 0x55dc8c674ba0 2023-11-28T20:50:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.375+0000 7f805b9a3740 1 -- 172.21.15.45:0/1113470075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c674ba0 msgr2=0x55dc8c6790a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.375+0000 7f805b9a3740 1 --2- 172.21.15.45:0/1113470075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c674ba0 0x55dc8c6790a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f803c009b20 tx=0x7f803c009190 comp rx=0 tx=0).stop 2023-11-28T20:50:34.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.375+0000 7f805b9a3740 1 -- 172.21.15.45:0/1113470075 shutdown_connections 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.375+0000 7f805b9a3740 1 --2- 172.21.15.45:0/1113470075 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c6bd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.375+0000 7f805b9a3740 1 --2- 172.21.15.45:0/1113470075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c674ba0 0x55dc8c6790a0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.375+0000 7f805b9a3740 1 --2- 172.21.15.45:0/1113470075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 0x55dc8c67a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.375+0000 7f805b9a3740 1 -- 172.21.15.45:0/1113470075 >> 172.21.15.45:0/1113470075 conn(0x55dc8c63bc70 msgr2=0x55dc8c6bf990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.375+0000 7f805b9a3740 1 -- 172.21.15.45:0/1113470075 shutdown_connections 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.375+0000 7f805b9a3740 1 -- 172.21.15.45:0/1113470075 wait complete. 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.376+0000 7f805b9a3740 1 -- start start 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.376+0000 7f805b9a3740 1 Processor -- start 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.377+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c674ba0 0x55dc8c6c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.377+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c69e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.377+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.377+0000 7f805b9a3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c5e0ac0 con 0x55dc8c679ea0 2023-11-28T20:50:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.377+0000 7f805b9a3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c57d2f0 con 0x55dc8c674ba0 2023-11-28T20:50:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.377+0000 7f805b9a3740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c59ac70 con 0x55dc8c6795e0 2023-11-28T20:50:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.377+0000 7f805b9a3740 10 client.0 _ll_register_callbacks cb 0x55dc8c59f3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:50:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f8052d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f8052d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51210/0 (socket says 172.21.15.45:51210) 2023-11-28T20:50:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f8052d9d640 1 -- 172.21.15.45:0/3190442490 learned_addr learned my addr 172.21.15.45:0/3190442490 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:34.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f805259c640 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c674ba0 0x55dc8c6c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f804bfff640 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c69e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f8052d9d640 1 -- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 msgr2=0x55dc8c69e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f8052d9d640 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c69e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f8052d9d640 1 -- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c674ba0 msgr2=0x55dc8c6c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f8052d9d640 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c674ba0 0x55dc8c6c6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f8052d9d640 1 -- 172.21.15.45:0/3190442490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803c009750 con 0x55dc8c679ea0 2023-11-28T20:50:34.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f8052d9d640 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ee20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f804c001b20 tx=0x7f804c05c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fc4582a67e8b89 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.378+0000 7f80317fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.379+0000 7f80317fa640 1 -- 172.21.15.45:0/3190442490 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c057c50 con 0x55dc8c679ea0 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.379+0000 7f805b9a3740 1 -- 172.21.15.45:0/3190442490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dc8c5d9aa0 con 0x55dc8c679ea0 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.379+0000 7f80317fa640 1 -- 172.21.15.45:0/3190442490 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f804c060450 con 0x55dc8c679ea0 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.379+0000 7f80317fa640 1 -- 172.21.15.45:0/3190442490 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f804c060610 con 0x55dc8c679ea0 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.379+0000 7f805b9a3740 10 client.6855 fetch_fsmap learned FSMap version 36 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.379+0000 7f805b9a3740 1 -- 172.21.15.45:0/3190442490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55dc8c5df140 con 0x55dc8c679ea0 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.379+0000 7f80317fa640 1 -- 172.21.15.45:0/3190442490 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55dc8c5d9aa0 con 0x55dc8c679ea0 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.380+0000 7f80317fa640 1 -- 172.21.15.45:0/3190442490 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 36) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f804c004b60 con 0x55dc8c679ea0 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.380+0000 7f805b9a3740 10 client.6855 fetch_fsmap finished waiting for FSMap version 36 2023-11-28T20:50:34.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.380+0000 7f805b9a3740 10 client.6855 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-28T20:50:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.380+0000 7f805b9a3740 20 client.6855 populate_metadata read hostname 'smithi045' 2023-11-28T20:50:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.380+0000 7f805b9a3740 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] conn(0x55dc8c6b8380 0x55dc8c6dcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.380+0000 7f805b9a3740 4 client.6855 mds_command: new command op to 6729 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:50:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.380+0000 7f805b9a3740 1 -- 172.21.15.45:0/3190442490 --> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dc8c5d97c0 con 0x55dc8c6b8380 2023-11-28T20:50:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.380+0000 7f805259c640 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] conn(0x55dc8c6b8380 0x55dc8c6dcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.381+0000 7f805259c640 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] conn(0x55dc8c6b8380 0x55dc8c6dcc40 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e54f4a3edbed2eea server_cookie=d32185e1af3680bd in_seq=0 out_seq=0 2023-11-28T20:50:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.381+0000 7f80317fa640 10 client.6855 ms_handle_connect on v2:172.21.15.190:6835/3032907708 2023-11-28T20:50:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f80317fa640 1 -- 172.21.15.45:0/3190442490 <== mds.0 v2:172.21.15.190:6835/3032907708 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dc8c5d97c0 con 0x55dc8c6b8380 2023-11-28T20:50:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f80317fa640 10 client.6855 handle_command_reply: tid=0 2023-11-28T20:50:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 client.6855 shutdown 2023-11-28T20:50:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 -- 172.21.15.45:0/3190442490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 msgr2=0x55dc8c69ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ee20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f804c001b20 tx=0x7f804c05c7e0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 -- 172.21.15.45:0/3190442490 shutdown_connections 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] conn(0x55dc8c6b8380 0x55dc8c6dcc40 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ee20 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c69e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 --2- 172.21.15.45:0/3190442490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c674ba0 0x55dc8c6c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f80317fa640 0 client.6855 ms_handle_reset on v2:172.21.15.190:6835/3032907708 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 -- 172.21.15.45:0/3190442490 >> 172.21.15.45:0/3190442490 conn(0x55dc8c63bc70 msgr2=0x55dc8c673b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 -- 172.21.15.45:0/3190442490 shutdown_connections 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 1 -- 172.21.15.45:0/3190442490 wait complete. 2023-11-28T20:50:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.384+0000 7f805b9a3740 20 client.6855 trim_cache size 0 max 16384 2023-11-28T20:50:34.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.402+0000 7f805b9a3740 1 Processor -- start 2023-11-28T20:50:34.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.402+0000 7f805b9a3740 1 -- start start 2023-11-28T20:50:34.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.403+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c674ba0 0x55dc8c69dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.403+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c6795e0 0x55dc8c69e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.403+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.403+0000 7f805b9a3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c343f20 con 0x55dc8c6795e0 2023-11-28T20:50:34.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.403+0000 7f805b9a3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c5e2a40 con 0x55dc8c679ea0 2023-11-28T20:50:34.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.403+0000 7f805b9a3740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c564520 con 0x55dc8c674ba0 2023-11-28T20:50:34.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.403+0000 7f804bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c6795e0 0x55dc8c69e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.403+0000 7f804bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c6795e0 0x55dc8c69e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51220/0 (socket says 172.21.15.45:51220) 2023-11-28T20:50:34.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.403+0000 7f804bfff640 1 -- 172.21.15.45:0/2340678658 learned_addr learned my addr 172.21.15.45:0/2340678658 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:34.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.404+0000 7f8052d9d640 1 --2- 172.21.15.45:0/2340678658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.404+0000 7f804bfff640 1 -- 172.21.15.45:0/2340678658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c674ba0 msgr2=0x55dc8c69dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.404+0000 7f805259c640 1 --2- 172.21.15.45:0/2340678658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c674ba0 0x55dc8c69dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.404+0000 7f804bfff640 1 --2- 172.21.15.45:0/2340678658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c674ba0 0x55dc8c69dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.404+0000 7f804bfff640 1 -- 172.21.15.45:0/2340678658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 msgr2=0x55dc8c69ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.404+0000 7f804bfff640 1 --2- 172.21.15.45:0/2340678658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.404+0000 7f804bfff640 1 -- 172.21.15.45:0/2340678658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc8c57cb80 con 0x55dc8c6795e0 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.404+0000 7f804bfff640 1 --2- 172.21.15.45:0/2340678658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c6795e0 0x55dc8c69e6f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f803c008010 tx=0x7f803c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ab9250e57825ae server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.404+0000 7f8048ff9640 1 -- 172.21.15.45:0/2340678658 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c0095f0 con 0x55dc8c6795e0 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f8048ff9640 1 -- 172.21.15.45:0/2340678658 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803c023040 con 0x55dc8c6795e0 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f8048ff9640 1 -- 172.21.15.45:0/2340678658 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c004040 con 0x55dc8c6795e0 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f805b9a3740 1 -- 172.21.15.45:0/2340678658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c6795e0 msgr2=0x55dc8c69e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f805b9a3740 1 --2- 172.21.15.45:0/2340678658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c6795e0 0x55dc8c69e6f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f803c008010 tx=0x7f803c015000 comp rx=0 tx=0).stop 2023-11-28T20:50:34.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f805b9a3740 1 -- 172.21.15.45:0/2340678658 shutdown_connections 2023-11-28T20:50:34.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f805b9a3740 1 --2- 172.21.15.45:0/2340678658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c679ea0 0x55dc8c69ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f805b9a3740 1 --2- 172.21.15.45:0/2340678658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c6795e0 0x55dc8c69e6f0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f805b9a3740 1 --2- 172.21.15.45:0/2340678658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c674ba0 0x55dc8c69dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f805b9a3740 1 -- 172.21.15.45:0/2340678658 >> 172.21.15.45:0/2340678658 conn(0x55dc8c63bc70 msgr2=0x55dc8c67a280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:34.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f805b9a3740 1 -- 172.21.15.45:0/2340678658 shutdown_connections 2023-11-28T20:50:34.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.405+0000 7f805b9a3740 1 -- 172.21.15.45:0/2340678658 wait complete. 2023-11-28T20:50:34.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.406+0000 7f805b9a3740 1 -- start start 2023-11-28T20:50:34.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.406+0000 7f805b9a3740 1 Processor -- start 2023-11-28T20:50:34.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.407+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c674ba0 0x55dc8c6d8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.407+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c729350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.407+0000 7f805b9a3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c6a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.407+0000 7f805b9a3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c729890 con 0x55dc8c679ea0 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.407+0000 7f805b9a3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c6a10b0 con 0x55dc8c674ba0 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.407+0000 7f805b9a3740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc8c6d8320 con 0x55dc8c6795e0 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.407+0000 7f805b9a3740 10 client.0 _ll_register_callbacks cb 0x7f804c057eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.407+0000 7f8052d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c6a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f8052d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c6a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51232/0 (socket says 172.21.15.45:51232) 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f8052d9d640 1 -- 172.21.15.45:0/427452665 learned_addr learned my addr 172.21.15.45:0/427452665 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f8052d9d640 1 -- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 msgr2=0x55dc8c729350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f8052d9d640 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c729350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f8052d9d640 1 -- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c674ba0 msgr2=0x55dc8c6d8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f8052d9d640 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c674ba0 0x55dc8c6d8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f8052d9d640 1 -- 172.21.15.45:0/427452665 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc8c5a8860 con 0x55dc8c679ea0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f8052d9d640 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c6a19c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f804c058250 tx=0x7f804c0021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8185231c91760c23 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f80317fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f80317fa640 1 -- 172.21.15.45:0/427452665 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c076030 con 0x55dc8c679ea0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.408+0000 7f80317fa640 1 -- 172.21.15.45:0/427452665 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f804c05c170 con 0x55dc8c679ea0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f805b9a3740 1 -- 172.21.15.45:0/427452665 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f803c0143f0 con 0x55dc8c679ea0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f80317fa640 1 -- 172.21.15.45:0/427452665 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f804c058090 con 0x55dc8c679ea0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f805b9a3740 10 client.6861 fetch_fsmap learned FSMap version 36 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f805b9a3740 1 -- 172.21.15.45:0/427452665 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55dc8c5a2170 con 0x55dc8c679ea0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f80317fa640 1 -- 172.21.15.45:0/427452665 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803c0143f0 con 0x55dc8c679ea0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f80317fa640 1 -- 172.21.15.45:0/427452665 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 36) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f804c060b30 con 0x55dc8c679ea0 2023-11-28T20:50:34.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f805b9a3740 10 client.6861 fetch_fsmap finished waiting for FSMap version 36 2023-11-28T20:50:34.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f805b9a3740 10 client.6861 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-28T20:50:34.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f805b9a3740 20 client.6861 populate_metadata read hostname 'smithi045' 2023-11-28T20:50:34.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f805b9a3740 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] conn(0x55dc8c69e740 0x55dc8c69eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f805b9a3740 4 client.6861 mds_command: new command op to 6729 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:50:34.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.409+0000 7f805b9a3740 1 -- 172.21.15.45:0/427452665 --> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55dc8c2e16c0 con 0x55dc8c69e740 2023-11-28T20:50:34.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.410+0000 7f805259c640 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] conn(0x55dc8c69e740 0x55dc8c69eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.411+0000 7f805259c640 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] conn(0x55dc8c69e740 0x55dc8c69eb20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29fc285aab8aa806 server_cookie=44aef9d6e46e488d in_seq=0 out_seq=0 2023-11-28T20:50:34.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.411+0000 7f80317fa640 10 client.6861 ms_handle_connect on v2:172.21.15.190:6835/3032907708 2023-11-28T20:50:34.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f80317fa640 1 -- 172.21.15.45:0/427452665 <== mds.0 v2:172.21.15.190:6835/3032907708 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55dc8c2e16c0 con 0x55dc8c69e740 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f80317fa640 10 client.6861 handle_command_reply: tid=0 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 client.6861 shutdown 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 -- 172.21.15.45:0/427452665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 msgr2=0x55dc8c6a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c6a19c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f804c058250 tx=0x7f804c0021f0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 -- 172.21.15.45:0/427452665 shutdown_connections 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:6835/3032907708,v1:172.21.15.190:6837/3032907708] conn(0x55dc8c69e740 0x55dc8c69eb20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc8c679ea0 0x55dc8c6a19c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc8c6795e0 0x55dc8c729350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 --2- 172.21.15.45:0/427452665 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc8c674ba0 0x55dc8c6d8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f80317fa640 0 client.6861 ms_handle_reset on v2:172.21.15.190:6835/3032907708 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 -- 172.21.15.45:0/427452665 >> 172.21.15.45:0/427452665 conn(0x55dc8c63bc70 msgr2=0x55dc8c69dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:34.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 -- 172.21.15.45:0/427452665 shutdown_connections 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 1 -- 172.21.15.45:0/427452665 wait complete. 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.412+0000 7f805b9a3740 20 client.6861 trim_cache size 0 max 16384 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 6729, 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:50:34.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 7.0537343479999999, 2023-11-28T20:50:34.418 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 36, 2023-11-28T20:50:34.418 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 88, 2023-11-28T20:50:34.418 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 88, 2023-11-28T20:50:34.418 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 12.928076555000001 2023-11-28T20:50:34.418 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:50:34.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 -- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f803406af90 msgr2=0x7f803406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f803406af90 0x7f803406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8044034000 tx=0x7f804402f000 comp rx=0 tx=0).stop 2023-11-28T20:50:34.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 -- 172.21.15.45:0/831415270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805415edc0 msgr2=0x7f805409a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805415edc0 0x7f805409a8a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f804c00cc70 tx=0x7f804c00a430 comp rx=0 tx=0).stop 2023-11-28T20:50:34.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 -- 172.21.15.45:0/831415270 shutdown_connections 2023-11-28T20:50:34.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f803406af90 0x7f803406d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f805415edc0 0x7f805409a8a0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f805415b820 0x7f805414f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 --2- 172.21.15.45:0/831415270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80540cf020 0x7f805409c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 -- 172.21.15.45:0/831415270 >> 172.21.15.45:0/831415270 conn(0x7f805406a6e0 msgr2=0x7f805406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:34.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 -- 172.21.15.45:0/831415270 shutdown_connections 2023-11-28T20:50:34.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.413+0000 7f8058e10640 1 -- 172.21.15.45:0/831415270 wait complete. 2023-11-28T20:50:34.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:50:34.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.642+0000 7fe1da2ef640 1 Processor -- start 2023-11-28T20:50:34.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.642+0000 7fe1da2ef640 1 -- start start 2023-11-28T20:50:34.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.643+0000 7fe1da2ef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d40b3870 0x7fe1d415db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.643+0000 7fe1da2ef640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe1d406d880 0x7fe1d415e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.643+0000 7fe1da2ef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d415e680 con 0x7fe1d40721e0 2023-11-28T20:50:34.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.643+0000 7fe1da2ef640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d415a460 con 0x7fe1d406d880 2023-11-28T20:50:34.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.643+0000 7fe1da2ef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d415e960 con 0x7fe1d40b3870 2023-11-28T20:50:34.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.643+0000 7fe1d8aec640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe1d40721e0 0x7fe1d406d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39606/0 (socket says 172.21.15.45:39606) 2023-11-28T20:50:34.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.643+0000 7fe1d8aec640 1 -- 172.21.15.45:0/1836022059 learned_addr learned my addr 172.21.15.45:0/1836022059 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:34.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.643+0000 7fe1d9aee640 1 --2- 172.21.15.45:0/1836022059 >> v2:172.21.15.45:3300/0 conn(0x7fe1d406d880 0x7fe1d415e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.644+0000 7fe1d92ed640 1 --2- 172.21.15.45:0/1836022059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d40b3870 0x7fe1d415db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.644+0000 7fe1d9aee640 1 -- 172.21.15.45:0/1836022059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d40b3870 msgr2=0x7fe1d415db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:34.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.644+0000 7fe1d9aee640 1 --2- 172.21.15.45:0/1836022059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d40b3870 0x7fe1d415db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.644+0000 7fe1d9aee640 1 -- 172.21.15.45:0/1836022059 >> v1:172.21.15.45:6789/0 conn(0x7fe1d40721e0 legacy=0x7fe1d406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.644+0000 7fe1d9aee640 1 -- 172.21.15.45:0/1836022059 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4071700 con 0x7fe1d406d880 2023-11-28T20:50:34.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.645+0000 7fe1d9aee640 1 --2- 172.21.15.45:0/1836022059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d406d880 0x7fe1d415e050 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009d90 tx=0x7fe1cc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad374fd7600faba server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.645+0000 7fe1cbfff640 1 -- 172.21.15.45:0/1836022059 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc02cc60 con 0x7fe1d406d880 2023-11-28T20:50:34.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.645+0000 7fe1cbfff640 1 -- 172.21.15.45:0/1836022059 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc02cdc0 con 0x7fe1d406d880 2023-11-28T20:50:34.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.645+0000 7fe1cbfff640 1 -- 172.21.15.45:0/1836022059 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc034660 con 0x7fe1d406d880 2023-11-28T20:50:34.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.646+0000 7fe1da2ef640 1 -- 172.21.15.45:0/1836022059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d406d880 msgr2=0x7fe1d415e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.646+0000 7fe1da2ef640 1 --2- 172.21.15.45:0/1836022059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d406d880 0x7fe1d415e050 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009d90 tx=0x7fe1cc02beb0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.646+0000 7fe1da2ef640 1 -- 172.21.15.45:0/1836022059 shutdown_connections 2023-11-28T20:50:34.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.646+0000 7fe1da2ef640 1 --2- 172.21.15.45:0/1836022059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d406d880 0x7fe1d415e050 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.646+0000 7fe1da2ef640 1 --2- 172.21.15.45:0/1836022059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d40b3870 0x7fe1d415db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.646+0000 7fe1da2ef640 1 -- 172.21.15.45:0/1836022059 >> 172.21.15.45:0/1836022059 conn(0x7fe1d406ab30 msgr2=0x7fe1d40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:34.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.646+0000 7fe1da2ef640 1 -- 172.21.15.45:0/1836022059 shutdown_connections 2023-11-28T20:50:34.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.646+0000 7fe1da2ef640 1 -- 172.21.15.45:0/1836022059 wait complete. 2023-11-28T20:50:34.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.648+0000 7fe1da2ef640 1 Processor -- start 2023-11-28T20:50:34.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.648+0000 7fe1da2ef640 1 -- start start 2023-11-28T20:50:34.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.649+0000 7fe1da2ef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d406d880 0x7fe1d4159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.649+0000 7fe1da2ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d40721e0 0x7fe1d4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.649+0000 7fe1da2ef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1d40b3870 0x7fe1d4099f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.649+0000 7fe1da2ef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d4165f70 con 0x7fe1d40721e0 2023-11-28T20:50:34.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.649+0000 7fe1da2ef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d40b2800 con 0x7fe1d40b3870 2023-11-28T20:50:34.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.649+0000 7fe1da2ef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d40b2ae0 con 0x7fe1d406d880 2023-11-28T20:50:34.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.649+0000 7fe1d8aec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d40721e0 0x7fe1d4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.649+0000 7fe1d8aec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d40721e0 0x7fe1d4099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51254/0 (socket says 172.21.15.45:51254) 2023-11-28T20:50:34.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.649+0000 7fe1d8aec640 1 -- 172.21.15.45:0/450379298 learned_addr learned my addr 172.21.15.45:0/450379298 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d92ed640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d406d880 0x7fe1d4159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d9aee640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1d40b3870 0x7fe1d4099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d8aec640 1 -- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d406d880 msgr2=0x7fe1d4159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d8aec640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d406d880 0x7fe1d4159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d8aec640 1 -- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1d40b3870 msgr2=0x7fe1d4099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d8aec640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1d40b3870 0x7fe1d4099f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d8aec640 1 -- 172.21.15.45:0/450379298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc009a70 con 0x7fe1d40721e0 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d92ed640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d406d880 0x7fe1d4159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d9aee640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1d40b3870 0x7fe1d4099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:34.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1d8aec640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d40721e0 0x7fe1d4099a40 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400c4a0 tx=0x7fe1c400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.650+0000 7fe1c9ffb640 1 -- 172.21.15.45:0/450379298 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4005ce0 con 0x7fe1d40721e0 2023-11-28T20:50:34.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.651+0000 7fe1c9ffb640 1 -- 172.21.15.45:0/450379298 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c4005e40 con 0x7fe1d40721e0 2023-11-28T20:50:34.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.651+0000 7fe1c9ffb640 1 -- 172.21.15.45:0/450379298 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4007270 con 0x7fe1d40721e0 2023-11-28T20:50:34.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.651+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d4166210 con 0x7fe1d40721e0 2023-11-28T20:50:34.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.651+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d40b2dc0 con 0x7fe1d40721e0 2023-11-28T20:50:34.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.652+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d4166730 con 0x7fe1d40721e0 2023-11-28T20:50:34.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.653+0000 7fe1c9ffb640 1 -- 172.21.15.45:0/450379298 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe1c4007490 con 0x7fe1d40721e0 2023-11-28T20:50:34.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.654+0000 7fe1c9ffb640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1b806af60 0x7fe1b806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:34.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.654+0000 7fe1c9ffb640 1 -- 172.21.15.45:0/450379298 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe1c4088e40 con 0x7fe1d40721e0 2023-11-28T20:50:34.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.654+0000 7fe1d92ed640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1b806af60 0x7fe1b806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:34.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.656+0000 7fe1d92ed640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1b806af60 0x7fe1b806d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000b820 tx=0x7fe1d0008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:34.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.660+0000 7fe1c9ffb640 1 -- 172.21.15.45:0/450379298 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1d4166730 con 0x7fe1d40721e0 2023-11-28T20:50:34.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.894+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe1d4166730 con 0x7fe1d40721e0 2023-11-28T20:50:34.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.895+0000 7fe1c9ffb640 1 -- 172.21.15.45:0/450379298 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe1d4166730 con 0x7fe1d40721e0 2023-11-28T20:50:34.897 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T20:50:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.897+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1b806af60 msgr2=0x7fe1b806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.897+0000 7fe1da2ef640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1b806af60 0x7fe1b806d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000b820 tx=0x7fe1d0008040 comp rx=0 tx=0).stop 2023-11-28T20:50:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.897+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d40721e0 msgr2=0x7fe1d4099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.897+0000 7fe1da2ef640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d40721e0 0x7fe1d4099a40 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400c4a0 tx=0x7fe1c400c960 comp rx=0 tx=0).stop 2023-11-28T20:50:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.897+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 shutdown_connections 2023-11-28T20:50:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.897+0000 7fe1da2ef640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fe1b806af60 0x7fe1b806d420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.897+0000 7fe1da2ef640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1d40b3870 0x7fe1d4099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.897+0000 7fe1da2ef640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1d40721e0 0x7fe1d4099a40 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.897+0000 7fe1da2ef640 1 --2- 172.21.15.45:0/450379298 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe1d406d880 0x7fe1d4159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:34.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.898+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 >> 172.21.15.45:0/450379298 conn(0x7fe1d406ab30 msgr2=0x7fe1d406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:34.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.898+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 shutdown_connections 2023-11-28T20:50:34.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:34.898+0000 7fe1da2ef640 1 -- 172.21.15.45:0/450379298 wait complete. 2023-11-28T20:50:34.920 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T20:50:34.920 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:50:34.920 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:50:34.920 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T20:50:34.920 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:50:34.920 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T20:50:34.920 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T20:50:34.921 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:50:34.921 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:50:34.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:34.922 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:50:34.982 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:50:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:50:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:50:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:50:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:50:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:50:34.983 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:50:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:50:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:50:34.984 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:50:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:50:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:50:34.984 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:50:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:50:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:50:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:50:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 41308sec preferred_lft 41308sec 2023-11-28T20:50:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:50:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:50:34.985 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:50:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:50:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:50:34.985 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:50:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T20:50:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:50:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:50:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T20:50:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:50:34.986 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:50:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:50:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T20:50:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:50:34.987 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:50:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T20:50:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T20:50:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:50:34.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:34.988 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:50:35.085 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:50:35.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:35.089 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:50:35.146 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:50:35.147 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:50:35.147 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:50:35.147 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:50:35.147 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:50:35.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:35.147 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:50:35.148 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:50:35.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:50:35.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:50:35.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:50:35.148 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:50:35.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:35.347 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:50:35.405 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T20:50:35.405 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T20:50:35.406 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:50:35.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:35.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:50:35.494 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:50:35.495 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T20:50:35.549 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:50:35.549 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:50:35.549 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:50:35.599 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:50:35.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:35.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:50:35.741 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:50:35.742 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:50:35.742 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:50:35.765 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:50:35.765 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:50:35.765 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:50:35.766 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:50:35.766 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:50:35.792 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:50:35.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:50:35.793 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:50:35.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:50:35.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:50:35.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:50:35.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:50:35.793 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:50:35.795 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T20:50:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:50:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:50:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:50:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:50:35.799 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:50:35.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:35.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:50:35.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:35.889 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:50:35.972 INFO:teuthology.orchestra.run.smithi045.stdout:client6873 2023-11-28T20:50:35.972 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:50:35.973 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:36.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T20:50:36.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.267+0000 7fa1b2fff640 1 Processor -- start 2023-11-28T20:50:36.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.267+0000 7fa1b2fff640 1 -- start start 2023-11-28T20:50:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.269+0000 7fa1b2fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1ac159ed0 0x7fa1ac071060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.269+0000 7fa1b2fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac071e80 0x7fa1ac0ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.269+0000 7fa1b2fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa1ac0cef00 0x7fa1ac0cf390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.269+0000 7fa1b2fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ac0cf8d0 con 0x7fa1ac159ed0 2023-11-28T20:50:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.269+0000 7fa1b2fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ac070bd0 con 0x7fa1ac0cef00 2023-11-28T20:50:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.269+0000 7fa1b2fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac0adad0 con 0x7fa1ac071e80 2023-11-28T20:50:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.269+0000 7fa1b1ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1ac159ed0 0x7fa1ac071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.269+0000 7fa1b1ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1ac159ed0 0x7fa1ac071060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51262/0 (socket says 172.21.15.45:51262) 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.269+0000 7fa1b1ffd640 1 -- 172.21.15.45:0/262446710 learned_addr learned my addr 172.21.15.45:0/262446710 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.270+0000 7fa1b17fc640 1 --2- 172.21.15.45:0/262446710 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac071e80 0x7fa1ac0ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.270+0000 7fa1b1ffd640 1 -- 172.21.15.45:0/262446710 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac071e80 msgr2=0x7fa1ac0ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.270+0000 7fa1b27fe640 1 --2- 172.21.15.45:0/262446710 >> v2:172.21.15.190:3300/0 conn(0x7fa1ac0cef00 0x7fa1ac0cf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.270+0000 7fa1b1ffd640 1 --2- 172.21.15.45:0/262446710 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac071e80 0x7fa1ac0ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.270+0000 7fa1b1ffd640 1 -- 172.21.15.45:0/262446710 >> v2:172.21.15.190:3300/0 conn(0x7fa1ac0cef00 msgr2=0x7fa1ac0cf390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.270+0000 7fa1b1ffd640 1 --2- 172.21.15.45:0/262446710 >> v2:172.21.15.190:3300/0 conn(0x7fa1ac0cef00 0x7fa1ac0cf390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.270+0000 7fa1b1ffd640 1 -- 172.21.15.45:0/262446710 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0702a0 con 0x7fa1ac159ed0 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.270+0000 7fa1b17fc640 1 --2- 172.21.15.45:0/262446710 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac071e80 0x7fa1ac0ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:36.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.270+0000 7fa1b27fe640 1 --2- 172.21.15.45:0/262446710 >> v2:172.21.15.190:3300/0 conn(0x7fa1ac0cef00 0x7fa1ac0cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:36.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.271+0000 7fa1b1ffd640 1 --2- 172.21.15.45:0/262446710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac159ed0 0x7fa1ac071060 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac06dde0 tx=0x7fa1a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5def4102156e792d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b0ffb640 1 -- 172.21.15.45:0/262446710 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a80317e0 con 0x7fa1ac159ed0 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b0ffb640 1 -- 172.21.15.45:0/262446710 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a8031940 con 0x7fa1ac159ed0 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b0ffb640 1 -- 172.21.15.45:0/262446710 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8031cd0 con 0x7fa1ac159ed0 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b2fff640 1 -- 172.21.15.45:0/262446710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac159ed0 msgr2=0x7fa1ac071060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/262446710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac159ed0 0x7fa1ac071060 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac06dde0 tx=0x7fa1a802b880 comp rx=0 tx=0).stop 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b2fff640 1 -- 172.21.15.45:0/262446710 shutdown_connections 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/262446710 >> v2:172.21.15.190:3300/0 conn(0x7fa1ac0cef00 0x7fa1ac0cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/262446710 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac071e80 0x7fa1ac0ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/262446710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac159ed0 0x7fa1ac071060 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b2fff640 1 -- 172.21.15.45:0/262446710 >> 172.21.15.45:0/262446710 conn(0x7fa1ac06a710 msgr2=0x7fa1ac06ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:36.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.272+0000 7fa1b2fff640 1 -- 172.21.15.45:0/262446710 shutdown_connections 2023-11-28T20:50:36.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.273+0000 7fa1b2fff640 1 -- 172.21.15.45:0/262446710 wait complete. 2023-11-28T20:50:36.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.274+0000 7fa1b2fff640 1 Processor -- start 2023-11-28T20:50:36.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.274+0000 7fa1b2fff640 1 -- start start 2023-11-28T20:50:36.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.275+0000 7fa1b2fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac071e80 0x7fa1ac09c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.275+0000 7fa1b2fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1ac0cef00 0x7fa1ac0b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.275+0000 7fa1b2fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac159ed0 0x7fa1ac0b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.275+0000 7fa1b2fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac09ca00 con 0x7fa1ac071e80 2023-11-28T20:50:36.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b2fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac14ec60 con 0x7fa1ac0cef00 2023-11-28T20:50:36.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b2fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac099b70 con 0x7fa1ac159ed0 2023-11-28T20:50:36.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b1ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac071e80 0x7fa1ac09c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b1ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac071e80 0x7fa1ac09c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51266/0 (socket says 172.21.15.45:51266) 2023-11-28T20:50:36.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b1ffd640 1 -- 172.21.15.45:0/2341937958 learned_addr learned my addr 172.21.15.45:0/2341937958 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b17fc640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1ac0cef00 0x7fa1ac0b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b1ffd640 1 -- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac159ed0 msgr2=0x7fa1ac0b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b27fe640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac159ed0 0x7fa1ac0b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b1ffd640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac159ed0 0x7fa1ac0b22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b1ffd640 1 -- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1ac0cef00 msgr2=0x7fa1ac0b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b1ffd640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1ac0cef00 0x7fa1ac0b1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.276+0000 7fa1b1ffd640 1 -- 172.21.15.45:0/2341937958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a80094d0 con 0x7fa1ac071e80 2023-11-28T20:50:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.277+0000 7fa1b1ffd640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac071e80 0x7fa1ac09c4c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8000c00 tx=0x7fa1a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.277+0000 7fa1a27fc640 1 -- 172.21.15.45:0/2341937958 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8032880 con 0x7fa1ac071e80 2023-11-28T20:50:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.277+0000 7fa1a27fc640 1 -- 172.21.15.45:0/2341937958 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a8032a00 con 0x7fa1ac071e80 2023-11-28T20:50:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.277+0000 7fa1a27fc640 1 -- 172.21.15.45:0/2341937958 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8032db0 con 0x7fa1ac071e80 2023-11-28T20:50:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.277+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac099e50 con 0x7fa1ac071e80 2023-11-28T20:50:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.277+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac09a410 con 0x7fa1ac071e80 2023-11-28T20:50:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.278+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ac099cb0 con 0x7fa1ac071e80 2023-11-28T20:50:36.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.279+0000 7fa1a27fc640 1 -- 172.21.15.45:0/2341937958 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa1a804d3f0 con 0x7fa1ac071e80 2023-11-28T20:50:36.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.280+0000 7fa1a27fc640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa19006af60 0x7fa19006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.280+0000 7fa1a27fc640 1 -- 172.21.15.45:0/2341937958 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa1a80ab890 con 0x7fa1ac071e80 2023-11-28T20:50:36.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.281+0000 7fa1b17fc640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa19006af60 0x7fa19006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.282+0000 7fa1b17fc640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa19006af60 0x7fa19006d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa19c006fd0 tx=0x7fa19c008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:36.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.285+0000 7fa1a27fc640 1 -- 172.21.15.45:0/2341937958 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1ac099cb0 con 0x7fa1ac071e80 2023-11-28T20:50:36.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.510+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 --> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1ac1621a0 con 0x7fa19006af60 2023-11-28T20:50:36.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.514+0000 7fa1a27fc640 1 -- 172.21.15.45:0/2341937958 <== mgr.5327 v2:172.21.15.190:6832/108887886 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa1ac1621a0 con 0x7fa19006af60 2023-11-28T20:50:36.515 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:50:36.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:50:36.516 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T20:50:36.516 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:50:36.516 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:50:36.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.517+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa19006af60 msgr2=0x7fa19006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:36.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.517+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa19006af60 0x7fa19006d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa19c006fd0 tx=0x7fa19c008040 comp rx=0 tx=0).stop 2023-11-28T20:50:36.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac071e80 msgr2=0x7fa1ac09c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:36.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac071e80 0x7fa1ac09c4c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8000c00 tx=0x7fa1a80046c0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 shutdown_connections 2023-11-28T20:50:36.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fa19006af60 0x7fa19006d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1ac159ed0 0x7fa1ac0b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1ac0cef00 0x7fa1ac0b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 --2- 172.21.15.45:0/2341937958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1ac071e80 0x7fa1ac09c4c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 >> 172.21.15.45:0/2341937958 conn(0x7fa1ac06a710 msgr2=0x7fa1ac14ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:36.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 shutdown_connections 2023-11-28T20:50:36.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.518+0000 7fa1b2fff640 1 -- 172.21.15.45:0/2341937958 wait complete. 2023-11-28T20:50:36.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T20:50:36.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.779+0000 7f21f359e640 1 Processor -- start 2023-11-28T20:50:36.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.779+0000 7f21f359e640 1 -- start start 2023-11-28T20:50:36.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.781+0000 7f21f359e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21f415b080 0x7f21f40ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.781+0000 7f21f359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f40abd70 0x7f21f40718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.781+0000 7f21f359e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f21f4071e00 0x7f21f415eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.781+0000 7f21f359e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f40722b0 con 0x7f21f415b080 2023-11-28T20:50:36.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.781+0000 7f21f359e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f40ac270 con 0x7f21f4071e00 2023-11-28T20:50:36.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.781+0000 7f21f359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f415b830 con 0x7f21f40abd70 2023-11-28T20:50:36.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.782+0000 7f21f259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21f415b080 0x7f21f40ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.782+0000 7f21f259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21f415b080 0x7f21f40ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51278/0 (socket says 172.21.15.45:51278) 2023-11-28T20:50:36.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.782+0000 7f21f259c640 1 -- 172.21.15.45:0/3283889368 learned_addr learned my addr 172.21.15.45:0/3283889368 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:36.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.782+0000 7f21ebfff640 1 --2- 172.21.15.45:0/3283889368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f40abd70 0x7f21f40718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.782+0000 7f21f2d9d640 1 --2- 172.21.15.45:0/3283889368 >> v2:172.21.15.190:3300/0 conn(0x7f21f4071e00 0x7f21f415eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.783+0000 7f21f259c640 1 -- 172.21.15.45:0/3283889368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f40abd70 msgr2=0x7f21f40718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:36.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.783+0000 7f21f259c640 1 --2- 172.21.15.45:0/3283889368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f40abd70 0x7f21f40718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.783+0000 7f21f259c640 1 -- 172.21.15.45:0/3283889368 >> v2:172.21.15.190:3300/0 conn(0x7f21f4071e00 msgr2=0x7f21f415eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:36.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.783+0000 7f21f259c640 1 --2- 172.21.15.45:0/3283889368 >> v2:172.21.15.190:3300/0 conn(0x7f21f4071e00 0x7f21f415eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.783+0000 7f21f259c640 1 -- 172.21.15.45:0/3283889368 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f415bb10 con 0x7f21f415b080 2023-11-28T20:50:36.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.783+0000 7f21f2d9d640 1 --2- 172.21.15.45:0/3283889368 >> v2:172.21.15.190:3300/0 conn(0x7f21f4071e00 0x7f21f415eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:36.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.785+0000 7f21f259c640 1 --2- 172.21.15.45:0/3283889368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f415b080 0x7f21f40ae370 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f21f406e6d0 tx=0x7f21e402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6bc2834fb0db8d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:36.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.785+0000 7f21f1d9b640 1 -- 172.21.15.45:0/3283889368 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031860 con 0x7f21f415b080 2023-11-28T20:50:36.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.785+0000 7f21f1d9b640 1 -- 172.21.15.45:0/3283889368 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40319c0 con 0x7f21f415b080 2023-11-28T20:50:36.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.785+0000 7f21f1d9b640 1 -- 172.21.15.45:0/3283889368 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031d50 con 0x7f21f415b080 2023-11-28T20:50:36.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.786+0000 7f21f359e640 1 -- 172.21.15.45:0/3283889368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f415b080 msgr2=0x7f21f40ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:36.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.786+0000 7f21f359e640 1 --2- 172.21.15.45:0/3283889368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f415b080 0x7f21f40ae370 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f21f406e6d0 tx=0x7f21e402b900 comp rx=0 tx=0).stop 2023-11-28T20:50:36.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.786+0000 7f21f359e640 1 -- 172.21.15.45:0/3283889368 shutdown_connections 2023-11-28T20:50:36.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.786+0000 7f21f359e640 1 --2- 172.21.15.45:0/3283889368 >> v2:172.21.15.190:3300/0 conn(0x7f21f4071e00 0x7f21f415eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.786+0000 7f21f359e640 1 --2- 172.21.15.45:0/3283889368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f40abd70 0x7f21f40718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.786+0000 7f21f359e640 1 --2- 172.21.15.45:0/3283889368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f415b080 0x7f21f40ae370 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.786+0000 7f21f359e640 1 -- 172.21.15.45:0/3283889368 >> 172.21.15.45:0/3283889368 conn(0x7f21f40cecd0 msgr2=0x7f21f40cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:36.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.786+0000 7f21f359e640 1 -- 172.21.15.45:0/3283889368 shutdown_connections 2023-11-28T20:50:36.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.786+0000 7f21f359e640 1 -- 172.21.15.45:0/3283889368 wait complete. 2023-11-28T20:50:36.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.788+0000 7f21f359e640 1 Processor -- start 2023-11-28T20:50:36.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.788+0000 7f21f359e640 1 -- start start 2023-11-28T20:50:36.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.789+0000 7f21f359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f4071e00 0x7f21f4151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.789+0000 7f21f359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21f40abd70 0x7f21f41460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.789+0000 7f21f359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f415b080 0x7f21f414e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.789+0000 7f21f359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f4154060 con 0x7f21f4071e00 2023-11-28T20:50:36.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.789+0000 7f21f359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f409b6c0 con 0x7f21f40abd70 2023-11-28T20:50:36.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.789+0000 7f21f359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f409aaf0 con 0x7f21f415b080 2023-11-28T20:50:36.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.789+0000 7f21f259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f4071e00 0x7f21f4151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f4071e00 0x7f21f4151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51288/0 (socket says 172.21.15.45:51288) 2023-11-28T20:50:36.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f259c640 1 -- 172.21.15.45:0/1393342142 learned_addr learned my addr 172.21.15.45:0/1393342142 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:36.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21ebfff640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21f40abd70 0x7f21f41460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f259c640 1 -- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f415b080 msgr2=0x7f21f414e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:36.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f2d9d640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f415b080 0x7f21f414e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f259c640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f415b080 0x7f21f414e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f259c640 1 -- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21f40abd70 msgr2=0x7f21f41460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f259c640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21f40abd70 0x7f21f41460e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f259c640 1 -- 172.21.15.45:0/1393342142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e40094d0 con 0x7f21f4071e00 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21ebfff640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21f40abd70 0x7f21f41460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f2d9d640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f415b080 0x7f21f414e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.790+0000 7f21f259c640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f4071e00 0x7f21f4151020 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f21e4000c00 tx=0x7f21e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.791+0000 7f21eaffd640 1 -- 172.21.15.45:0/1393342142 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e403ac60 con 0x7f21f4071e00 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.791+0000 7f21eaffd640 1 -- 172.21.15.45:0/1393342142 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e403adc0 con 0x7f21f4071e00 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.791+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f4146620 con 0x7f21f4071e00 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.791+0000 7f21eaffd640 1 -- 172.21.15.45:0/1393342142 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4044630 con 0x7f21f4071e00 2023-11-28T20:50:36.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.791+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f409b070 con 0x7f21f4071e00 2023-11-28T20:50:36.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.792+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f409d0f0 con 0x7f21f4071e00 2023-11-28T20:50:36.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.793+0000 7f21eaffd640 1 -- 172.21.15.45:0/1393342142 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 30) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f21e404d450 con 0x7f21f4071e00 2023-11-28T20:50:36.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.794+0000 7f21eaffd640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f21d406af60 0x7f21d406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:36.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.794+0000 7f21eaffd640 1 -- 172.21.15.45:0/1393342142 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f21e40abef0 con 0x7f21f4071e00 2023-11-28T20:50:36.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.795+0000 7f21ebfff640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f21d406af60 0x7f21d406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:36.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.796+0000 7f21ebfff640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f21d406af60 0x7f21d406d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f21f40715e0 tx=0x7f21dc008040 comp rx=0 tx=0).ready entity=mgr.5327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:36.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:36.798+0000 7f21eaffd640 1 -- 172.21.15.45:0/1393342142 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21e4076c40 con 0x7f21f4071e00 2023-11-28T20:50:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.043+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f21f404e920 con 0x7f21f4071e00 2023-11-28T20:50:37.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.457+0000 7f21eaffd640 1 -- 172.21.15.45:0/1393342142 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v31) v1 ==== 92+0+0 (secure 0 0 0) 0x7f21e407a260 con 0x7f21f4071e00 2023-11-28T20:50:37.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.461+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f21d406af60 msgr2=0x7f21d406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:37.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.461+0000 7f21f359e640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f21d406af60 0x7f21d406d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f21f40715e0 tx=0x7f21dc008040 comp rx=0 tx=0).stop 2023-11-28T20:50:37.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.461+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f4071e00 msgr2=0x7f21f4151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:37.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.461+0000 7f21f359e640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f4071e00 0x7f21f4151020 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f21e4000c00 tx=0x7f21e40046c0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.462+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 shutdown_connections 2023-11-28T20:50:37.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.462+0000 7f21f359e640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f21d406af60 0x7f21d406d420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.462+0000 7f21f359e640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21f415b080 0x7f21f414e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.462+0000 7f21f359e640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21f40abd70 0x7f21f41460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.462+0000 7f21f359e640 1 --2- 172.21.15.45:0/1393342142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21f4071e00 0x7f21f4151020 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.462+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 >> 172.21.15.45:0/1393342142 conn(0x7f21f40cecd0 msgr2=0x7f21f40cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:37.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.463+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 shutdown_connections 2023-11-28T20:50:37.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.463+0000 7f21f359e640 1 -- 172.21.15.45:0/1393342142 wait complete. 2023-11-28T20:50:37.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T20:50:37.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.702+0000 7f9ad8456640 1 Processor -- start 2023-11-28T20:50:37.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.703+0000 7f9ad8456640 1 -- start start 2023-11-28T20:50:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.704+0000 7f9ad8456640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9ad00cfa50 0x7f9ad00cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.704+0000 7f9ad8456640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9ad00b36c0 0x7f9ad00b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.704+0000 7f9ad8456640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad00b0570 con 0x7f9ad015b3c0 2023-11-28T20:50:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.704+0000 7f9ad8456640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad00b08c0 con 0x7f9ad00b36c0 2023-11-28T20:50:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.704+0000 7f9ad8456640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad00cc800 con 0x7f9ad00cfa50 2023-11-28T20:50:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.705+0000 7f9ad69f3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9ad00b36c0 0x7f9ad00b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.705+0000 7f9ad69f3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9ad00b36c0 0x7f9ad00b0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51326/0 (socket says 172.21.15.45:51326) 2023-11-28T20:50:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.705+0000 7f9ad69f3640 1 -- 172.21.15.45:0/3647077940 learned_addr learned my addr 172.21.15.45:0/3647077940 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.705+0000 7f9ad59f1640 1 --2- 172.21.15.45:0/3647077940 >> v2:172.21.15.190:3300/0 conn(0x7f9ad00cfa50 0x7f9ad00cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.705+0000 7f9ad69f3640 1 -- 172.21.15.45:0/3647077940 >> v2:172.21.15.190:3300/0 conn(0x7f9ad00cfa50 msgr2=0x7f9ad00cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.705+0000 7f9ad69f3640 1 --2- 172.21.15.45:0/3647077940 >> v2:172.21.15.190:3300/0 conn(0x7f9ad00cfa50 0x7f9ad00cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.705+0000 7f9ad69f3640 1 -- 172.21.15.45:0/3647077940 >> v1:172.21.15.190:6789/0 conn(0x7f9ad015b3c0 legacy=0x7f9ad00cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.705+0000 7f9ad69f3640 1 -- 172.21.15.45:0/3647077940 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad00ccae0 con 0x7f9ad00b36c0 2023-11-28T20:50:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.705+0000 7f9ad59f1640 1 --2- 172.21.15.45:0/3647077940 >> v2:172.21.15.190:3300/0 conn(0x7f9ad00cfa50 0x7f9ad00cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:37.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.707+0000 7f9ad69f3640 1 --2- 172.21.15.45:0/3647077940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00b36c0 0x7f9ad00b0ce0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00b0500 tx=0x7f9ac802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d29f820c856ed7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:37.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.707+0000 7f9ad51f0640 1 -- 172.21.15.45:0/3647077940 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac802c960 con 0x7f9ad00b36c0 2023-11-28T20:50:37.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.707+0000 7f9ad51f0640 1 -- 172.21.15.45:0/3647077940 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac802cac0 con 0x7f9ad00b36c0 2023-11-28T20:50:37.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.707+0000 7f9ad51f0640 1 -- 172.21.15.45:0/3647077940 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac802ce50 con 0x7f9ad00b36c0 2023-11-28T20:50:37.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.708+0000 7f9ad8456640 1 -- 172.21.15.45:0/3647077940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00b36c0 msgr2=0x7f9ad00b0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:37.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.708+0000 7f9ad8456640 1 --2- 172.21.15.45:0/3647077940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00b36c0 0x7f9ad00b0ce0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00b0500 tx=0x7f9ac802bdc0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.708+0000 7f9ad8456640 1 -- 172.21.15.45:0/3647077940 shutdown_connections 2023-11-28T20:50:37.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.708+0000 7f9ad8456640 1 --2- 172.21.15.45:0/3647077940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00b36c0 0x7f9ad00b0ce0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.708+0000 7f9ad8456640 1 --2- 172.21.15.45:0/3647077940 >> v2:172.21.15.190:3300/0 conn(0x7f9ad00cfa50 0x7f9ad00cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.708+0000 7f9ad8456640 1 -- 172.21.15.45:0/3647077940 >> 172.21.15.45:0/3647077940 conn(0x7f9ad00b2ba0 msgr2=0x7f9ad00b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:37.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.708+0000 7f9ad8456640 1 -- 172.21.15.45:0/3647077940 shutdown_connections 2023-11-28T20:50:37.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.708+0000 7f9ad8456640 1 -- 172.21.15.45:0/3647077940 wait complete. 2023-11-28T20:50:37.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.711+0000 7f9ad8456640 1 Processor -- start 2023-11-28T20:50:37.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.712+0000 7f9ad8456640 1 -- start start 2023-11-28T20:50:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.713+0000 7f9ad8456640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ad00b36c0 0x7f9ad00b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.713+0000 7f9ad8456640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00cfa50 0x7f9ad01f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.713+0000 7f9ad8456640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ad015b3c0 0x7f9ad00c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.713+0000 7f9ad8456640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad01f7350 con 0x7f9ad00cfa50 2023-11-28T20:50:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.713+0000 7f9ad8456640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00b4080 con 0x7f9ad015b3c0 2023-11-28T20:50:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.713+0000 7f9ad8456640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00b4360 con 0x7f9ad00b36c0 2023-11-28T20:50:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad59f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00cfa50 0x7f9ad01f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:37.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad59f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00cfa50 0x7f9ad01f6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51340/0 (socket says 172.21.15.45:51340) 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad59f1640 1 -- 172.21.15.45:0/886668975 learned_addr learned my addr 172.21.15.45:0/886668975 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad69f3640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ad015b3c0 0x7f9ad00c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad61f2640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ad00b36c0 0x7f9ad00b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad59f1640 1 -- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ad00b36c0 msgr2=0x7f9ad00b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad59f1640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ad00b36c0 0x7f9ad00b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad59f1640 1 -- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ad015b3c0 msgr2=0x7f9ad00c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad59f1640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ad015b3c0 0x7f9ad00c5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad59f1640 1 -- 172.21.15.45:0/886668975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac8009b30 con 0x7f9ad00cfa50 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad61f2640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ad00b36c0 0x7f9ad00b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad69f3640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ad015b3c0 0x7f9ad00c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.714+0000 7f9ad59f1640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00cfa50 0x7f9ad01f6e10 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400e7e0 tx=0x7f9ac400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:37.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.715+0000 7f9ac2ffd640 1 -- 172.21.15.45:0/886668975 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4004030 con 0x7f9ad00cfa50 2023-11-28T20:50:37.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.715+0000 7f9ac2ffd640 1 -- 172.21.15.45:0/886668975 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac4004190 con 0x7f9ad00cfa50 2023-11-28T20:50:37.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.715+0000 7f9ac2ffd640 1 -- 172.21.15.45:0/886668975 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4004520 con 0x7f9ad00cfa50 2023-11-28T20:50:37.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.715+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad00b4640 con 0x7f9ad00cfa50 2023-11-28T20:50:37.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.715+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad00b4c00 con 0x7f9ad00cfa50 2023-11-28T20:50:37.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.716+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad00d08f0 con 0x7f9ad00cfa50 2023-11-28T20:50:37.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.717+0000 7f9ac2ffd640 1 -- 172.21.15.45:0/886668975 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 31) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f9ac400c6d0 con 0x7f9ad00cfa50 2023-11-28T20:50:37.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.718+0000 7f9ac2ffd640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f9ab406afb0 0x7f9ab406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:37.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.718+0000 7f9ac2ffd640 1 -- 172.21.15.45:0/886668975 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9ac4089730 con 0x7f9ad00cfa50 2023-11-28T20:50:37.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.718+0000 7f9ad61f2640 1 -- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f9ab406afb0 msgr2=0x7f9ab406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/108887886 2023-11-28T20:50:37.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.718+0000 7f9ad61f2640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f9ab406afb0 0x7f9ab406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:37.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.722+0000 7f9ac2ffd640 1 -- 172.21.15.45:0/886668975 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ac4054480 con 0x7f9ad00cfa50 2023-11-28T20:50:37.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.918+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f9ad00733d0 con 0x7f9ad00cfa50 2023-11-28T20:50:37.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.919+0000 7f9ad61f2640 1 -- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f9ab406afb0 msgr2=0x7f9ab406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/108887886 2023-11-28T20:50:37.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.919+0000 7f9ad61f2640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f9ab406afb0 0x7f9ab406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.923+0000 7f9ac2ffd640 1 -- 172.21.15.45:0/886668975 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9ac4057aa0 con 0x7f9ad00cfa50 2023-11-28T20:50:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.925+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f9ab406afb0 msgr2=0x7f9ab406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.925+0000 7f9ad8456640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f9ab406afb0 0x7f9ab406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.925+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00cfa50 msgr2=0x7f9ad01f6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.925+0000 7f9ad8456640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00cfa50 0x7f9ad01f6e10 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400e7e0 tx=0x7f9ac400eca0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.926+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 shutdown_connections 2023-11-28T20:50:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.926+0000 7f9ad8456640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7f9ab406afb0 0x7f9ab406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.926+0000 7f9ad8456640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ad015b3c0 0x7f9ad00c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.926+0000 7f9ad8456640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ad00cfa50 0x7f9ad01f6e10 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.926+0000 7f9ad8456640 1 --2- 172.21.15.45:0/886668975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ad00b36c0 0x7f9ad00b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.926+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 >> 172.21.15.45:0/886668975 conn(0x7f9ad00b2ba0 msgr2=0x7f9ad00afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.926+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 shutdown_connections 2023-11-28T20:50:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:37.926+0000 7f9ad8456640 1 -- 172.21.15.45:0/886668975 wait complete. 2023-11-28T20:50:37.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T20:50:37.983 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:37.981+0000 7fa644e2e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:50:38.011 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:38.010+0000 7f06e6e15040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:50:38.035 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.033+0000 7fa644e2e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:50:38.067 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:38.066+0000 7f06e6e15040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:50:38.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.159+0000 7fab0bf97640 1 Processor -- start 2023-11-28T20:50:38.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.159+0000 7fab0bf97640 1 -- start start 2023-11-28T20:50:38.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.161+0000 7fab0bf97640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab0406e120 0x7fab0406d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.161+0000 7fab0bf97640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fab0415d6e0 0x7fab0406e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.161+0000 7fab0bf97640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0406ea40 con 0x7fab040c3510 2023-11-28T20:50:38.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.161+0000 7fab0bf97640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0406d680 con 0x7fab0406e120 2023-11-28T20:50:38.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.161+0000 7fab0bf97640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab040cf1a0 con 0x7fab0415d6e0 2023-11-28T20:50:38.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.161+0000 7fab09d33640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab0406e120 0x7fab0406d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.161+0000 7fab0a534640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fab040c3510 0x7fab040661f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39632/0 (socket says 172.21.15.45:39632) 2023-11-28T20:50:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.161+0000 7fab0a534640 1 -- 172.21.15.45:0/1837375778 learned_addr learned my addr 172.21.15.45:0/1837375778 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.162+0000 7fab09532640 1 --2- 172.21.15.45:0/1837375778 >> v2:172.21.15.190:3300/0 conn(0x7fab0415d6e0 0x7fab0406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.162+0000 7fab08d31640 1 -- 172.21.15.45:0/1837375778 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503851256 0 0) 0x7fab0406ea40 con 0x7fab040c3510 2023-11-28T20:50:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.162+0000 7fab09d33640 1 -- 172.21.15.45:0/1837375778 >> v2:172.21.15.190:3300/0 conn(0x7fab0415d6e0 msgr2=0x7fab0406e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.162+0000 7fab09d33640 1 --2- 172.21.15.45:0/1837375778 >> v2:172.21.15.190:3300/0 conn(0x7fab0415d6e0 0x7fab0406e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.162+0000 7fab09d33640 1 -- 172.21.15.45:0/1837375778 >> v1:172.21.15.45:6789/0 conn(0x7fab040c3510 legacy=0x7fab040661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.162+0000 7fab09d33640 1 -- 172.21.15.45:0/1837375778 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040cf480 con 0x7fab0406e120 2023-11-28T20:50:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.163+0000 7fab09d33640 1 --2- 172.21.15.45:0/1837375778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 0x7fab0406d8f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fab0406f1f0 tx=0x7fab0002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83c63ab6879005e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.163+0000 7fab08d31640 1 -- 172.21.15.45:0/1837375778 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0002e070 con 0x7fab0406e120 2023-11-28T20:50:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.163+0000 7fab08d31640 1 -- 172.21.15.45:0/1837375778 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab000315a0 con 0x7fab0406e120 2023-11-28T20:50:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.163+0000 7fab08d31640 1 -- 172.21.15.45:0/1837375778 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00031930 con 0x7fab0406e120 2023-11-28T20:50:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.164+0000 7fab0bf97640 1 -- 172.21.15.45:0/1837375778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 msgr2=0x7fab0406d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.164+0000 7fab0bf97640 1 --2- 172.21.15.45:0/1837375778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 0x7fab0406d8f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fab0406f1f0 tx=0x7fab0002b840 comp rx=0 tx=0).stop 2023-11-28T20:50:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.164+0000 7fab0bf97640 1 -- 172.21.15.45:0/1837375778 shutdown_connections 2023-11-28T20:50:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.164+0000 7fab0bf97640 1 --2- 172.21.15.45:0/1837375778 >> v2:172.21.15.190:3300/0 conn(0x7fab0415d6e0 0x7fab0406e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.164+0000 7fab0bf97640 1 --2- 172.21.15.45:0/1837375778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 0x7fab0406d8f0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.164+0000 7fab0bf97640 1 -- 172.21.15.45:0/1837375778 >> 172.21.15.45:0/1837375778 conn(0x7fab040cc900 msgr2=0x7fab040ccd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.164+0000 7fab0bf97640 1 -- 172.21.15.45:0/1837375778 shutdown_connections 2023-11-28T20:50:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.164+0000 7fab0bf97640 1 -- 172.21.15.45:0/1837375778 wait complete. 2023-11-28T20:50:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.166+0000 7fab0bf97640 1 Processor -- start 2023-11-28T20:50:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.166+0000 7fab0bf97640 1 -- start start 2023-11-28T20:50:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab0bf97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 0x7fab04159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab0bf97640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab040c3510 0x7fab041f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab0bf97640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab040c39b0 0x7fab0414d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab0bf97640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab041f2be0 con 0x7fab0406e120 2023-11-28T20:50:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab0bf97640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab040c4190 con 0x7fab040c39b0 2023-11-28T20:50:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab0bf97640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab040c4470 con 0x7fab040c3510 2023-11-28T20:50:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab09d33640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 0x7fab04159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab09d33640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 0x7fab04159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51354/0 (socket says 172.21.15.45:51354) 2023-11-28T20:50:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab09d33640 1 -- 172.21.15.45:0/1296488422 learned_addr learned my addr 172.21.15.45:0/1296488422 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab09d33640 1 -- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab040c3510 msgr2=0x7fab041f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab09d33640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab040c3510 0x7fab041f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab0a534640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab040c39b0 0x7fab0414d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab09d33640 1 -- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab040c39b0 msgr2=0x7fab0414d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.167+0000 7fab09d33640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab040c39b0 0x7fab0414d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.168+0000 7fab09d33640 1 -- 172.21.15.45:0/1296488422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab00009400 con 0x7fab0406e120 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.168+0000 7fab09d33640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 0x7fab04159420 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fab00009890 tx=0x7fab000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.168+0000 7faafa7fc640 1 -- 172.21.15.45:0/1296488422 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00032840 con 0x7fab0406e120 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.168+0000 7faafa7fc640 1 -- 172.21.15.45:0/1296488422 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab000329a0 con 0x7fab0406e120 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.168+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab040c4750 con 0x7fab0406e120 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.168+0000 7faafa7fc640 1 -- 172.21.15.45:0/1296488422 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00032d30 con 0x7fab0406e120 2023-11-28T20:50:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.168+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab040c4d10 con 0x7fab0406e120 2023-11-28T20:50:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.170+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab040c4f00 con 0x7fab0406e120 2023-11-28T20:50:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.170+0000 7faafa7fc640 1 -- 172.21.15.45:0/1296488422 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 31) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fab0003b3f0 con 0x7fab0406e120 2023-11-28T20:50:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.171+0000 7faafa7fc640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7faae806afe0 0x7faae806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.171+0000 7faafa7fc640 1 -- 172.21.15.45:0/1296488422 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fab000aba30 con 0x7fab0406e120 2023-11-28T20:50:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.171+0000 7fab09532640 1 -- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7faae806afe0 msgr2=0x7faae806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/108887886 2023-11-28T20:50:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.171+0000 7fab09532640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7faae806afe0 0x7faae806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:38.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.176+0000 7faafa7fc640 1 -- 172.21.15.45:0/1296488422 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab00076780 con 0x7fab0406e120 2023-11-28T20:50:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.372+0000 7fab09532640 1 -- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7faae806afe0 msgr2=0x7faae806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/108887886 2023-11-28T20:50:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.372+0000 7fab09532640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7faae806afe0 0x7faae806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:38.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.372+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fab040733d0 con 0x7fab0406e120 2023-11-28T20:50:38.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.377+0000 7faafa7fc640 1 -- 172.21.15.45:0/1296488422 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7fab00079da0 con 0x7fab0406e120 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7faae806afe0 msgr2=0x7faae806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7faae806afe0 0x7faae806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 msgr2=0x7fab04159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 0x7fab04159420 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fab00009890 tx=0x7fab000046c0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 shutdown_connections 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7faae806afe0 0x7faae806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab040c39b0 0x7fab0414d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab040c3510 0x7fab041f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 --2- 172.21.15.45:0/1296488422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab0406e120 0x7fab04159420 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 >> 172.21.15.45:0/1296488422 conn(0x7fab040cc900 msgr2=0x7fab0415b620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:38.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 shutdown_connections 2023-11-28T20:50:38.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.380+0000 7fab0bf97640 1 -- 172.21.15.45:0/1296488422 wait complete. 2023-11-28T20:50:38.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:38.400 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-11-28T20:50:38.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-11-28T20:50:38.485 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.481+0000 7fa644e2e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:50:38.536 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.532+0000 7fa644e2e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:50:38.547 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:38.545+0000 7f06e6e15040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:50:38.584 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.582+0000 7fa644e2e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:50:38.602 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:38.601+0000 7f06e6e15040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:50:38.644 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.641+0000 7fa644e2e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:50:38.655 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:38.654+0000 7f06e6e15040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:50:38.692 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.690+0000 7fa644e2e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:50:38.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.686+0000 7fb0cc5d0640 1 Processor -- start 2023-11-28T20:50:38.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.686+0000 7fb0cc5d0640 1 -- start start 2023-11-28T20:50:38.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.688+0000 7fb0cc5d0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c415b3c0 0x7fb0c4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.688+0000 7fb0cc5d0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb0c40ccb40 0x7fb0c4099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.688+0000 7fb0cc5d0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c40c4f10 con 0x7fb0c40d1b40 2023-11-28T20:50:38.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.688+0000 7fb0cc5d0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c415b7a0 con 0x7fb0c40ccb40 2023-11-28T20:50:38.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.688+0000 7fb0cc5d0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40c4c10 con 0x7fb0c415b3c0 2023-11-28T20:50:38.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.688+0000 7fb0c9b6b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb0c40d1b40 0x7fb0c40c53a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39646/0 (socket says 172.21.15.45:39646) 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.688+0000 7fb0c9b6b640 1 -- 172.21.15.45:0/1171860867 learned_addr learned my addr 172.21.15.45:0/1171860867 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.688+0000 7fb0cab6d640 1 --2- 172.21.15.45:0/1171860867 >> v2:172.21.15.190:3300/0 conn(0x7fb0c40ccb40 0x7fb0c4099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.688+0000 7fb0ca36c640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c415b3c0 0x7fb0c4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968709783 0 0) 0x7fb0c40c4f10 con 0x7fb0c40d1b40 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b0003520 con 0x7fb0c40d1b40 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2998179004 0 0) 0x7fb0b0003520 con 0x7fb0c40d1b40 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c415b3c0 msgr2=0x7fb0c4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c415b3c0 0x7fb0c4099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 >> v2:172.21.15.190:3300/0 conn(0x7fb0c40ccb40 msgr2=0x7fb0c4099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 --2- 172.21.15.45:0/1171860867 >> v2:172.21.15.190:3300/0 conn(0x7fb0c40ccb40 0x7fb0c4099f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40cf600 con 0x7fb0c40d1b40 2023-11-28T20:50:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb0b4001670 con 0x7fb0c40d1b40 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 >> v1:172.21.15.45:6789/0 conn(0x7fb0c40d1b40 legacy=0x7fb0c40c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b0003970 0x7fb0b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c415b3c0 0x7fb0b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b0004d00 0x7fb0b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c415b7a0 con 0x7fb0b0004d00 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40c4c10 con 0x7fb0b0003970 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b000ba00 con 0x7fb0c415b3c0 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.689+0000 7fb0cab6d640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b0004d00 0x7fb0b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.690+0000 7fb0ca36c640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b0003970 0x7fb0b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.690+0000 7fb0c9b6b640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c415b3c0 0x7fb0b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.690+0000 7fb0cab6d640 1 -- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c415b3c0 msgr2=0x7fb0b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.690+0000 7fb0cab6d640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c415b3c0 0x7fb0b00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.690+0000 7fb0cab6d640 1 -- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b0003970 msgr2=0x7fb0b0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:38.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.690+0000 7fb0cab6d640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b0003970 0x7fb0b0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.690+0000 7fb0cab6d640 1 -- 172.21.15.45:0/1171860867 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0003520 con 0x7fb0b0004d00 2023-11-28T20:50:38.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.691+0000 7fb0cab6d640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b0004d00 0x7fb0b0005180 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009fc0 tx=0x7fb0b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3e179a8409d36 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:38.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.691+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc02cc60 con 0x7fb0b0004d00 2023-11-28T20:50:38.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.691+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc00f040 con 0x7fb0b0004d00 2023-11-28T20:50:38.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.691+0000 7fb0c936a640 1 -- 172.21.15.45:0/1171860867 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc034440 con 0x7fb0b0004d00 2023-11-28T20:50:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.692+0000 7fb0cc5d0640 1 -- 172.21.15.45:0/1171860867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b0004d00 msgr2=0x7fb0b0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.692+0000 7fb0cc5d0640 1 --2- 172.21.15.45:0/1171860867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b0004d00 0x7fb0b0005180 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009fc0 tx=0x7fb0b00057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.692+0000 7fb0cab6d640 1 -- 172.21.15.45:0/1171860867 reap_dead start 2023-11-28T20:50:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.692+0000 7fb0cc5d0640 1 -- 172.21.15.45:0/1171860867 shutdown_connections 2023-11-28T20:50:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.692+0000 7fb0cc5d0640 1 -- 172.21.15.45:0/1171860867 >> 172.21.15.45:0/1171860867 conn(0x7fb0c40d1020 msgr2=0x7fb0c40d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.692+0000 7fb0cc5d0640 1 -- 172.21.15.45:0/1171860867 shutdown_connections 2023-11-28T20:50:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.692+0000 7fb0cc5d0640 1 -- 172.21.15.45:0/1171860867 wait complete. 2023-11-28T20:50:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.695+0000 7fb0cc5d0640 1 Processor -- start 2023-11-28T20:50:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.696+0000 7fb0cc5d0640 1 -- start start 2023-11-28T20:50:38.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.697+0000 7fb0cc5d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b0004d00 0x7fb0c40b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.697+0000 7fb0cc5d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c40b6790 0x7fb0c4159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.697+0000 7fb0cc5d0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c41f2460 0x7fb0c41f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.697+0000 7fb0cc5d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40b51f0 con 0x7fb0b0004d00 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.697+0000 7fb0cc5d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40b5a90 con 0x7fb0c40b6790 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.697+0000 7fb0cc5d0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c41f2e20 con 0x7fb0c41f2460 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.697+0000 7fb0ca36c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b0004d00 0x7fb0c40b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.697+0000 7fb0ca36c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b0004d00 0x7fb0c40b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51368/0 (socket says 172.21.15.45:51368) 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.697+0000 7fb0ca36c640 1 -- 172.21.15.45:0/952739814 learned_addr learned my addr 172.21.15.45:0/952739814 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0c9b6b640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c40b6790 0x7fb0c4159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0ca36c640 1 -- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c41f2460 msgr2=0x7fb0c41f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0ca36c640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0c41f2460 0x7fb0c41f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0ca36c640 1 -- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c40b6790 msgr2=0x7fb0c4159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0ca36c640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c40b6790 0x7fb0c4159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0ca36c640 1 -- 172.21.15.45:0/952739814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40b6d00 con 0x7fb0b0004d00 2023-11-28T20:50:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0ca36c640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b0004d00 0x7fb0c40b3ec0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000f5b0 tx=0x7fb0c000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0013810 con 0x7fb0b0004d00 2023-11-28T20:50:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c0013970 con 0x7fb0b0004d00 2023-11-28T20:50:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0013ce0 con 0x7fb0b0004d00 2023-11-28T20:50:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.698+0000 7fb0cc5d0640 1 -- 172.21.15.45:0/952739814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c40b5440 con 0x7fb0b0004d00 2023-11-28T20:50:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.699+0000 7fb0cc5d0640 1 -- 172.21.15.45:0/952739814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c40b4400 con 0x7fb0b0004d00 2023-11-28T20:50:38.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.700+0000 7fb0cc5d0640 1 -- 172.21.15.45:0/952739814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c40733d0 con 0x7fb0b0004d00 2023-11-28T20:50:38.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.701+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 31) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fb0c001f460 con 0x7fb0b0004d00 2023-11-28T20:50:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.702+0000 7fb0baffd640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 0x7fb0a806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.702+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb0c0089d10 con 0x7fb0b0004d00 2023-11-28T20:50:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.702+0000 7fb0c9b6b640 1 -- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 msgr2=0x7fb0a806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/108887886 2023-11-28T20:50:38.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.702+0000 7fb0c9b6b640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 0x7fb0a806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.707+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0c008f020 con 0x7fb0b0004d00 2023-11-28T20:50:38.724 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:38.723+0000 7f06e6e15040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:50:38.777 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:38.776+0000 7f06e6e15040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:50:38.803 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.801+0000 7fa644e2e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:50:38.864 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.862+0000 7fa644e2e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:38.895 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:38.894+0000 7f06e6e15040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:50:38.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.903+0000 7fb0c9b6b640 1 -- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 msgr2=0x7fb0a806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/108887886 2023-11-28T20:50:38.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.903+0000 7fb0c9b6b640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 0x7fb0a806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:38.913 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.911+0000 7fa644e2e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:50:38.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:38.928+0000 7fb0cc5d0640 1 -- 172.21.15.45:0/952739814 --> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0c40cfbc0 con 0x7fb0a806afb0 2023-11-28T20:50:38.961 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:38.960+0000 7f06e6e15040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:38.970 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:38.967+0000 7fa644e2e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:50:39.012 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:39.011+0000 7f06e6e15040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:50:39.020 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:39.018+0000 7fa644e2e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:50:39.072 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:39.071+0000 7f06e6e15040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:50:39.124 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:39.123+0000 7f06e6e15040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:50:39.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:39.304+0000 7fb0c9b6b640 1 -- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 msgr2=0x7fb0a806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/108887886 2023-11-28T20:50:39.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:39.304+0000 7fb0c9b6b640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 0x7fb0a806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T20:50:39.398 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:39.396+0000 7fa644e2e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:50:39.525 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:39.524+0000 7f06e6e15040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:50:39.558 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:39.556+0000 7fa644e2e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:50:39.666 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:39.664+0000 7fa644e2e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:50:39.694 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:39.693+0000 7f06e6e15040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:50:39.716 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:39.714+0000 7fa644e2e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:50:39.808 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:39.807+0000 7f06e6e15040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:50:39.824 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:39.822+0000 7fa644e2e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:39.861 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:39.860+0000 7f06e6e15040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:50:39.929 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:39.927+0000 7fa644e2e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:50:39.974 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:39.973+0000 7f06e6e15040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:40.087 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:40.086+0000 7f06e6e15040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:50:40.088 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:40.085+0000 7fa644e2e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:40.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:40.105+0000 7fb0c9b6b640 1 -- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 msgr2=0x7fb0a806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/108887886 2023-11-28T20:50:40.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:40.105+0000 7fb0c9b6b640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 0x7fb0a806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T20:50:40.137 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:40.134+0000 7fa644e2e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:40.252 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:40.251+0000 7f06e6e15040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:40.256 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:40.253+0000 7fa644e2e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:50:40.304 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:40.303+0000 7f06e6e15040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:40.430 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:40.429+0000 7f06e6e15040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:50:41.078 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.076+0000 7fa644e2e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:50:41.294 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:41.292+0000 7f06e6e15040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:50:41.372 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.370+0000 7fa644e2e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:50:41.430 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.427+0000 7fa644e2e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:50:41.544 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.542+0000 7fa644e2e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:50:41.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:41.567+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 32) v1 ==== 88959+0+0 (secure 0 0 0) 0x7fb0c0054ea0 con 0x7fb0b0004d00 2023-11-28T20:50:41.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:41.567+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 msgr2=0x7fb0a806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:41.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:41.567+0000 7fb0baffd640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/108887886,v1:172.21.15.190:6833/108887886] conn(0x7fb0a806afb0 0x7fb0a806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:41.613 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:41.612+0000 7f06e6e15040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:50:41.677 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:41.676+0000 7f06e6e15040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.722+0000 7fa5c40ef640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.722+0000 7fa5c40ef640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.722+0000 7fa5c40ef640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.722+0000 7fa5c40ef640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.722+0000 7fa5c40ef640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.723+0000 7fa5c10e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.723+0000 7fa5c10e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.723+0000 7fa5c10e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.723+0000 7fa5c10e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:41.723+0000 7fa5c10e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:41.802 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:41.801+0000 7f06e6e15040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:50:42.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.574+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 33) v1 ==== 89086+0+0 (secure 0 0 0) 0x7fb0c001c310 con 0x7fb0b0004d00 2023-11-28T20:50:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.575+0000 7fb0baffd640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fb0a8035570 0x7fb0a8035950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.575+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 --> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0a8035e90 con 0x7fb0a8035570 2023-11-28T20:50:42.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.577+0000 7fb0c9b6b640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fb0a8035570 0x7fb0a8035950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:42.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.580+0000 7fb0c9b6b640 1 --2- 172.21.15.45:0/952739814 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fb0a8035570 0x7fb0a8035950 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40017a0 tx=0x7fb0b4009040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:42.609 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:42.607+0000 7fa5cb0fd640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-11-28T20:50:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.609+0000 7fb0baffd640 1 -- 172.21.15.45:0/952739814 <== mgr.5899 v2:172.21.15.190:6832/2590179685 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fb0a8035e90 con 0x7fb0a8035570 2023-11-28T20:50:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-11-28T20:50:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T20:50:42.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:42.626 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-11-28T20:50:42.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T20:50:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.894+0000 7f1a154c3640 1 Processor -- start 2023-11-28T20:50:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.894+0000 7f1a154c3640 1 -- start start 2023-11-28T20:50:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.896+0000 7f1a154c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a1015b080 0x7f1a100ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.896+0000 7f1a154c3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1a10071fd0 0x7f1a100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.896+0000 7f1a154c3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a100ac270 con 0x7f1a100abd70 2023-11-28T20:50:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.896+0000 7f1a154c3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1015b830 con 0x7f1a10071fd0 2023-11-28T20:50:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.896+0000 7f1a154c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1015bb10 con 0x7f1a1015b080 2023-11-28T20:50:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.897+0000 7f1a14cc2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1a10071fd0 0x7f1a100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.897+0000 7f1a14cc2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1a10071fd0 0x7f1a100702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51376/0 (socket says 172.21.15.45:51376) 2023-11-28T20:50:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.897+0000 7f1a14cc2640 1 -- 172.21.15.45:0/3702353035 learned_addr learned my addr 172.21.15.45:0/3702353035 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.897+0000 7f1a0ffff640 1 --2- 172.21.15.45:0/3702353035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a1015b080 0x7f1a100ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.897+0000 7f1a14cc2640 1 -- 172.21.15.45:0/3702353035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a1015b080 msgr2=0x7f1a100ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.897+0000 7f1a14cc2640 1 --2- 172.21.15.45:0/3702353035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a1015b080 0x7f1a100ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.897+0000 7f1a14cc2640 1 -- 172.21.15.45:0/3702353035 >> v1:172.21.15.190:6789/0 conn(0x7f1a100abd70 legacy=0x7f1a100718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.897+0000 7f1a14cc2640 1 -- 172.21.15.45:0/3702353035 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1015bdf0 con 0x7f1a10071fd0 2023-11-28T20:50:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.899+0000 7f1a14cc2640 1 --2- 172.21.15.45:0/3702353035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a10071fd0 0x7f1a100702c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009e50 tx=0x7f1a0802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18315743bdeb6020 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.899+0000 7f1a0f7fe640 1 -- 172.21.15.45:0/3702353035 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0802cb00 con 0x7f1a10071fd0 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.899+0000 7f1a0f7fe640 1 -- 172.21.15.45:0/3702353035 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0802cc60 con 0x7f1a10071fd0 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.899+0000 7f1a0f7fe640 1 -- 172.21.15.45:0/3702353035 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a080343f0 con 0x7f1a10071fd0 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.899+0000 7f1a154c3640 1 -- 172.21.15.45:0/3702353035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a10071fd0 msgr2=0x7f1a100702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.899+0000 7f1a154c3640 1 --2- 172.21.15.45:0/3702353035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a10071fd0 0x7f1a100702c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009e50 tx=0x7f1a0802bf60 comp rx=0 tx=0).stop 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.900+0000 7f1a154c3640 1 -- 172.21.15.45:0/3702353035 shutdown_connections 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.900+0000 7f1a154c3640 1 --2- 172.21.15.45:0/3702353035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a10071fd0 0x7f1a100702c0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.900+0000 7f1a154c3640 1 --2- 172.21.15.45:0/3702353035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a1015b080 0x7f1a100ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.900+0000 7f1a154c3640 1 -- 172.21.15.45:0/3702353035 >> 172.21.15.45:0/3702353035 conn(0x7f1a100cecd0 msgr2=0x7f1a100cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.900+0000 7f1a154c3640 1 -- 172.21.15.45:0/3702353035 shutdown_connections 2023-11-28T20:50:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.900+0000 7f1a154c3640 1 -- 172.21.15.45:0/3702353035 wait complete. 2023-11-28T20:50:42.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.901+0000 7f1a154c3640 1 Processor -- start 2023-11-28T20:50:42.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.902+0000 7f1a154c3640 1 -- start start 2023-11-28T20:50:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.902+0000 7f1a154c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a10071fd0 0x7f1a101578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a154c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100abd70 0x7f1a10144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a154c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a1015b080 0x7f1a101f0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a154c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1009ba70 con 0x7f1a100abd70 2023-11-28T20:50:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a154c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10158070 con 0x7f1a1015b080 2023-11-28T20:50:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a154c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10157e20 con 0x7f1a10071fd0 2023-11-28T20:50:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100abd70 0x7f1a10144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100abd70 0x7f1a10144510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51388/0 (socket says 172.21.15.45:51388) 2023-11-28T20:50:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a07fff640 1 -- 172.21.15.45:0/3266474350 learned_addr learned my addr 172.21.15.45:0/3266474350 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a14cc2640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a1015b080 0x7f1a101f0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:42.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a0ffff640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a10071fd0 0x7f1a101578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:42.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a07fff640 1 -- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a10071fd0 msgr2=0x7f1a101578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:42.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a07fff640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a10071fd0 0x7f1a101578e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:42.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a07fff640 1 -- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a1015b080 msgr2=0x7f1a101f0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:42.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a07fff640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a1015b080 0x7f1a101f0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:42.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.903+0000 7f1a07fff640 1 -- 172.21.15.45:0/3266474350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a08009b30 con 0x7f1a100abd70 2023-11-28T20:50:42.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.904+0000 7f1a07fff640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100abd70 0x7f1a10144510 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f19f800c400 tx=0x7f19f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:42.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.904+0000 7f1a0d7fa640 1 -- 172.21.15.45:0/3266474350 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f80052d0 con 0x7f1a100abd70 2023-11-28T20:50:42.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.904+0000 7f1a0d7fa640 1 -- 172.21.15.45:0/3266474350 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f8005450 con 0x7f1a100abd70 2023-11-28T20:50:42.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.904+0000 7f1a0d7fa640 1 -- 172.21.15.45:0/3266474350 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8007260 con 0x7f1a100abd70 2023-11-28T20:50:42.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.904+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1009b0d0 con 0x7f1a100abd70 2023-11-28T20:50:42.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.904+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1009b690 con 0x7f1a100abd70 2023-11-28T20:50:42.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.905+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1009bbb0 con 0x7f1a100abd70 2023-11-28T20:50:42.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.906+0000 7f1a0d7fa640 1 -- 172.21.15.45:0/3266474350 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 33) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f19f80073c0 con 0x7f1a100abd70 2023-11-28T20:50:42.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.908+0000 7f1a0d7fa640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f19f006ad60 0x7f19f006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:42.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.908+0000 7f1a0d7fa640 1 -- 172.21.15.45:0/3266474350 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f19f8088970 con 0x7f1a100abd70 2023-11-28T20:50:42.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.908+0000 7f1a0ffff640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f19f006ad60 0x7f19f006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:42.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.909+0000 7f1a0ffff640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f19f006ad60 0x7f19f006d220 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1a00006fd0 tx=0x7f1a00008040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:42.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:42.913+0000 7f1a0d7fa640 1 -- 172.21.15.45:0/3266474350 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a1009bbb0 con 0x7f1a100abd70 2023-11-28T20:50:43.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.157+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f1a1009bbb0 con 0x7f1a100abd70 2023-11-28T20:50:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.625+0000 7f1a0d7fa640 1 -- 172.21.15.45:0/3266474350 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v34) v1 ==== 93+0+0 (secure 0 0 0) 0x7f1a1009bbb0 con 0x7f1a100abd70 2023-11-28T20:50:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f19f006ad60 msgr2=0x7f19f006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f19f006ad60 0x7f19f006d220 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1a00006fd0 tx=0x7f1a00008040 comp rx=0 tx=0).stop 2023-11-28T20:50:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100abd70 msgr2=0x7f1a10144510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100abd70 0x7f1a10144510 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f19f800c400 tx=0x7f19f800c8c0 comp rx=0 tx=0).stop 2023-11-28T20:50:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 shutdown_connections 2023-11-28T20:50:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f19f006ad60 0x7f19f006d220 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a1015b080 0x7f1a101f0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100abd70 0x7f1a10144510 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 --2- 172.21.15.45:0/3266474350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a10071fd0 0x7f1a101578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.629+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 >> 172.21.15.45:0/3266474350 conn(0x7f1a100cecd0 msgr2=0x7f1a100af4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.630+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 shutdown_connections 2023-11-28T20:50:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:43.631+0000 7f1a154c3640 1 -- 172.21.15.45:0/3266474350 wait complete. 2023-11-28T20:50:43.654 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:50:43.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:50:43.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:50:43.845 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:50:43.845 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:50:43.845 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:43.899 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:50:43.899 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:43.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:44.157 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:44.155+0000 7f1b376da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:50:44.172 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:44.171+0000 7f9384615040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:50:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.196+0000 7fa2b5562640 1 Processor -- start 2023-11-28T20:50:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.196+0000 7fa2b5562640 1 -- start start 2023-11-28T20:50:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.198+0000 7fa2b5562640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa2b015b210 0x7fa2b00714a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.198+0000 7fa2b5562640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa2b015c600 0x7fa2b00ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.198+0000 7fa2b5562640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b00c4720 0x7fa2b00ce950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.198+0000 7fa2b5562640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b00719e0 con 0x7fa2b015b210 2023-11-28T20:50:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2b5562640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b015b5f0 con 0x7fa2b015c600 2023-11-28T20:50:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2b5562640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b015c170 con 0x7fa2b00c4720 2023-11-28T20:50:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2affff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa2b015b210 0x7fa2b00714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2affff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa2b015b210 0x7fa2b00714a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52318/0 (socket says 172.21.15.45:52318) 2023-11-28T20:50:44.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2affff640 1 -- 172.21.15.45:0/3967041622 learned_addr learned my addr 172.21.15.45:0/3967041622 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2b4d61640 1 --2- 172.21.15.45:0/3967041622 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b00c4720 0x7fa2b00ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2af7fe640 1 --2- 172.21.15.45:0/3967041622 >> v2:172.21.15.190:3300/0 conn(0x7fa2b015c600 0x7fa2b00ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2affff640 1 -- 172.21.15.45:0/3967041622 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b00c4720 msgr2=0x7fa2b00ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2affff640 1 --2- 172.21.15.45:0/3967041622 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b00c4720 0x7fa2b00ce950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2affff640 1 -- 172.21.15.45:0/3967041622 >> v2:172.21.15.190:3300/0 conn(0x7fa2b015c600 msgr2=0x7fa2b00ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:44.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2affff640 1 --2- 172.21.15.45:0/3967041622 >> v2:172.21.15.190:3300/0 conn(0x7fa2b015c600 0x7fa2b00ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2affff640 1 -- 172.21.15.45:0/3967041622 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0071bc0 con 0x7fa2b015b210 2023-11-28T20:50:44.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.199+0000 7fa2af7fe640 1 --2- 172.21.15.45:0/3967041622 >> v2:172.21.15.190:3300/0 conn(0x7fa2b015c600 0x7fa2b00ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:44.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.200+0000 7fa2affff640 1 --2- 172.21.15.45:0/3967041622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b015b210 0x7fa2b00714a0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00723c0 tx=0x7fa2a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5843a1fa1f9b2fc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.201+0000 7fa2aeffd640 1 -- 172.21.15.45:0/3967041622 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a00317e0 con 0x7fa2b015b210 2023-11-28T20:50:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.201+0000 7fa2aeffd640 1 -- 172.21.15.45:0/3967041622 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a0031940 con 0x7fa2b015b210 2023-11-28T20:50:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.201+0000 7fa2aeffd640 1 -- 172.21.15.45:0/3967041622 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0031cd0 con 0x7fa2b015b210 2023-11-28T20:50:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.201+0000 7fa2b5562640 1 -- 172.21.15.45:0/3967041622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b015b210 msgr2=0x7fa2b00714a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.201+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3967041622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b015b210 0x7fa2b00714a0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00723c0 tx=0x7fa2a002b880 comp rx=0 tx=0).stop 2023-11-28T20:50:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.202+0000 7fa2b5562640 1 -- 172.21.15.45:0/3967041622 shutdown_connections 2023-11-28T20:50:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.202+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3967041622 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b00c4720 0x7fa2b00ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.202+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3967041622 >> v2:172.21.15.190:3300/0 conn(0x7fa2b015c600 0x7fa2b00ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.202+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3967041622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b015b210 0x7fa2b00714a0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.202+0000 7fa2b5562640 1 -- 172.21.15.45:0/3967041622 >> 172.21.15.45:0/3967041622 conn(0x7fa2b00b14b0 msgr2=0x7fa2b00b18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:44.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.202+0000 7fa2b5562640 1 -- 172.21.15.45:0/3967041622 shutdown_connections 2023-11-28T20:50:44.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.202+0000 7fa2b5562640 1 -- 172.21.15.45:0/3967041622 wait complete. 2023-11-28T20:50:44.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.204+0000 7fa2b5562640 1 Processor -- start 2023-11-28T20:50:44.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.204+0000 7fa2b5562640 1 -- start start 2023-11-28T20:50:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2b5562640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b00c4720 0x7fa2b01f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2b5562640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2b015b210 0x7fa2b00b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2b5562640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b015c600 0x7fa2b01f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2b5562640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00b3ad0 con 0x7fa2b00c4720 2023-11-28T20:50:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2b5562640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0154120 con 0x7fa2b015b210 2023-11-28T20:50:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2b5562640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b01f3a70 con 0x7fa2b015c600 2023-11-28T20:50:44.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2affff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b00c4720 0x7fa2b01f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2affff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b00c4720 0x7fa2b01f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52330/0 (socket says 172.21.15.45:52330) 2023-11-28T20:50:44.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2affff640 1 -- 172.21.15.45:0/3523502424 learned_addr learned my addr 172.21.15.45:0/3523502424 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:44.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.205+0000 7fa2b4d61640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b015c600 0x7fa2b01f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2af7fe640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2b015b210 0x7fa2b00b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2affff640 1 -- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b015c600 msgr2=0x7fa2b01f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2affff640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b015c600 0x7fa2b01f3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2affff640 1 -- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2b015b210 msgr2=0x7fa2b00b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2affff640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2b015b210 0x7fa2b00b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2affff640 1 -- 172.21.15.45:0/3523502424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a00094d0 con 0x7fa2b00c4720 2023-11-28T20:50:44.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2affff640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b00c4720 0x7fa2b01f73e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa2a002b8c0 tx=0x7fa2a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:44.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2acff9640 1 -- 172.21.15.45:0/3523502424 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0032880 con 0x7fa2b00c4720 2023-11-28T20:50:44.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2acff9640 1 -- 172.21.15.45:0/3523502424 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a0032a00 con 0x7fa2b00c4720 2023-11-28T20:50:44.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2acff9640 1 -- 172.21.15.45:0/3523502424 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0032db0 con 0x7fa2b00c4720 2023-11-28T20:50:44.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b01f3d50 con 0x7fa2b00c4720 2023-11-28T20:50:44.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.206+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b01543a0 con 0x7fa2b00c4720 2023-11-28T20:50:44.212 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:44.208+0000 7f1b376da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:50:44.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.208+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b0067fb0 con 0x7fa2b00c4720 2023-11-28T20:50:44.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.209+0000 7fa2acff9640 1 -- 172.21.15.45:0/3523502424 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 34) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fa2a004d3f0 con 0x7fa2b00c4720 2023-11-28T20:50:44.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.210+0000 7fa2acff9640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fa29c06af60 0x7fa29c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.210+0000 7fa2acff9640 1 -- 172.21.15.45:0/3523502424 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fa2a00ab960 con 0x7fa2b00c4720 2023-11-28T20:50:44.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.210+0000 7fa2af7fe640 1 -- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fa29c06af60 msgr2=0x7fa29c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:44.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.210+0000 7fa2af7fe640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fa29c06af60 0x7fa29c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:44.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.215+0000 7fa2acff9640 1 -- 172.21.15.45:0/3523502424 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2a00765f0 con 0x7fa2b00c4720 2023-11-28T20:50:44.231 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:44.230+0000 7f9384615040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:50:44.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.410+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2b01f4270 con 0x7fa2b00c4720 2023-11-28T20:50:44.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.411+0000 7fa2af7fe640 1 -- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fa29c06af60 msgr2=0x7fa29c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:44.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.411+0000 7fa2af7fe640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fa29c06af60 0x7fa29c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:44.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.412+0000 7fa2acff9640 1 -- 172.21.15.45:0/3523502424 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+17211 (secure 0 0 0) 0x7fa2a0079c10 con 0x7fa2b00c4720 2023-11-28T20:50:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:44.414 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":89,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:41.547109+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:50:24.656511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-28T20:50:25.847745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1902490820":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1766221473":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1382859164":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6833/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1155029644":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6832/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/2238215747":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/3448409370":"2023-11-29T20:50:41.547044+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fa29c06af60 msgr2=0x7fa29c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fa29c06af60 0x7fa29c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b00c4720 msgr2=0x7fa2b01f73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b00c4720 0x7fa2b01f73e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa2a002b8c0 tx=0x7fa2a00046c0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 shutdown_connections 2023-11-28T20:50:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fa29c06af60 0x7fa29c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2b015c600 0x7fa2b01f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2b015b210 0x7fa2b00b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 --2- 172.21.15.45:0/3523502424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2b00c4720 0x7fa2b01f73e0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 >> 172.21.15.45:0/3523502424 conn(0x7fa2b00b14b0 msgr2=0x7fa2b006da10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 shutdown_connections 2023-11-28T20:50:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.415+0000 7fa2b5562640 1 -- 172.21.15.45:0/3523502424 wait complete. 2023-11-28T20:50:44.435 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:50:24.656511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:50:25.847745+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:50:44.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:50:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.648+0000 7f42b5fde640 1 Processor -- start 2023-11-28T20:50:44.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.648+0000 7f42b5fde640 1 -- start start 2023-11-28T20:50:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.650+0000 7f42b5fde640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f42b00d0240 0x7f42b0168410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.650+0000 7f42b5fde640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b00cdf00 con 0x7f42b00cda70 2023-11-28T20:50:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.650+0000 7f42b5fde640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b00ce1e0 con 0x7f42b015d420 2023-11-28T20:50:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.650+0000 7f42b5fde640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b00ce4c0 con 0x7f42b00d0240 2023-11-28T20:50:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.650+0000 7f42a7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f42b00cda70 0x7f42b00cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45010/0 (socket says 172.21.15.45:45010) 2023-11-28T20:50:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.650+0000 7f42a7fff640 1 -- 172.21.15.45:0/573171905 learned_addr learned my addr 172.21.15.45:0/573171905 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.650+0000 7f42affff640 1 --2- 172.21.15.45:0/573171905 >> v2:172.21.15.190:3300/0 conn(0x7f42b00d0240 0x7f42b0168410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668195547 0 0) 0x7f42b00cdf00 con 0x7f42b00cda70 2023-11-28T20:50:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f429c003520 con 0x7f42b00cda70 2023-11-28T20:50:44.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 303076059 0 0) 0x7f429c003520 con 0x7f42b00cda70 2023-11-28T20:50:44.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 >> v2:172.21.15.190:3300/0 conn(0x7f42b00d0240 msgr2=0x7f42b0168410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:44.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 --2- 172.21.15.45:0/573171905 >> v2:172.21.15.190:3300/0 conn(0x7f42b00d0240 0x7f42b0168410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 >> v1:172.21.15.190:6789/0 conn(0x7f42b015d420 legacy=0x7f42b0161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b00ce7a0 con 0x7f42b00cda70 2023-11-28T20:50:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4298001670 con 0x7f42b00cda70 2023-11-28T20:50:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 >> v1:172.21.15.45:6789/0 conn(0x7f42b00cda70 legacy=0x7f42b00cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f429c0039c0 0x7f429c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f429c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f429c005180 0x7f429c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b00ce4c0 con 0x7f429c0047c0 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c00c080 con 0x7f429c005180 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.651+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429c00c240 con 0x7f429c0039c0 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42a7fff640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f429c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42affff640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f429c005180 0x7f429c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42af7fe640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f429c0039c0 0x7f429c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42a7fff640 1 -- 172.21.15.45:0/573171905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f429c0039c0 msgr2=0x7f429c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42a7fff640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f429c0039c0 0x7f429c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42a7fff640 1 -- 172.21.15.45:0/573171905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f429c005180 msgr2=0x7f429c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42a7fff640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f429c005180 0x7f429c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42a7fff640 1 -- 172.21.15.45:0/573171905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c003520 con 0x7f429c0047c0 2023-11-28T20:50:44.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42affff640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f429c005180 0x7f429c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.652+0000 7f42affff640 1 -- 172.21.15.45:0/573171905 reap_dead start 2023-11-28T20:50:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.653+0000 7f42a7fff640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f429c004c40 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f42b008c660 tx=0x7f429c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14867bf1404cbf04 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4298005ad0 con 0x7f429c0047c0 2023-11-28T20:50:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4298005c50 con 0x7f429c0047c0 2023-11-28T20:50:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42aeffd640 1 -- 172.21.15.45:0/573171905 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429802c790 con 0x7f429c0047c0 2023-11-28T20:50:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42b5fde640 1 -- 172.21.15.45:0/573171905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 msgr2=0x7f429c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42b5fde640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f429c004c40 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f42b008c660 tx=0x7f429c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42b5fde640 1 -- 172.21.15.45:0/573171905 shutdown_connections 2023-11-28T20:50:44.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42b5fde640 1 --2- 172.21.15.45:0/573171905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f429c004c40 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42b5fde640 1 -- 172.21.15.45:0/573171905 >> 172.21.15.45:0/573171905 conn(0x7f42b0070b20 msgr2=0x7f42b0070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:44.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42b5fde640 1 -- 172.21.15.45:0/573171905 shutdown_connections 2023-11-28T20:50:44.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.654+0000 7f42b5fde640 1 -- 172.21.15.45:0/573171905 wait complete. 2023-11-28T20:50:44.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.656+0000 7f42b5fde640 1 Processor -- start 2023-11-28T20:50:44.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.656+0000 7f42b5fde640 1 -- start start 2023-11-28T20:50:44.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.657+0000 7f42b5fde640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f42b01e7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.657+0000 7f42b5fde640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42b015f3e0 0x7f42b0156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.657+0000 7f42b5fde640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42b0160370 0x7f42b00b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.657+0000 7f42b5fde640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b015fb60 con 0x7f429c0047c0 2023-11-28T20:50:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.657+0000 7f42b5fde640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b015f880 con 0x7f42b015f3e0 2023-11-28T20:50:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.657+0000 7f42b5fde640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b01e6840 con 0x7f42b0160370 2023-11-28T20:50:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.657+0000 7f42af7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f42b01e7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.657+0000 7f42af7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f42b01e7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52348/0 (socket says 172.21.15.45:52348) 2023-11-28T20:50:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.657+0000 7f42af7fe640 1 -- 172.21.15.45:0/1420407365 learned_addr learned my addr 172.21.15.45:0/1420407365 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.658+0000 7f42a7fff640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42b015f3e0 0x7f42b0156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.658+0000 7f42affff640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42b0160370 0x7f42b00b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.658+0000 7f42af7fe640 1 -- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42b0160370 msgr2=0x7f42b00b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.658+0000 7f42af7fe640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42b0160370 0x7f42b00b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.658+0000 7f42af7fe640 1 -- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42b015f3e0 msgr2=0x7f42b0156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.658+0000 7f42af7fe640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42b015f3e0 0x7f42b0156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.658+0000 7f42af7fe640 1 -- 172.21.15.45:0/1420407365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b01e6b20 con 0x7f429c0047c0 2023-11-28T20:50:44.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.658+0000 7f42af7fe640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f42b01e7320 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f42a000e550 tx=0x7f42a000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:44.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.659+0000 7f42acff9640 1 -- 172.21.15.45:0/1420407365 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a00052e0 con 0x7f429c0047c0 2023-11-28T20:50:44.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.659+0000 7f42acff9640 1 -- 172.21.15.45:0/1420407365 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a0005440 con 0x7f429c0047c0 2023-11-28T20:50:44.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.659+0000 7f42acff9640 1 -- 172.21.15.45:0/1420407365 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0003810 con 0x7f429c0047c0 2023-11-28T20:50:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.659+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b01e6e00 con 0x7f429c0047c0 2023-11-28T20:50:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.659+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b01609e0 con 0x7f429c0047c0 2023-11-28T20:50:44.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.660+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b01617d0 con 0x7f429c0047c0 2023-11-28T20:50:44.670 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:44.660+0000 7f1b376da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:50:44.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.661+0000 7f42acff9640 1 -- 172.21.15.45:0/1420407365 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 34) v1 ==== 89334+0+0 (secure 0 0 0) 0x7f42a0003970 con 0x7f429c0047c0 2023-11-28T20:50:44.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.662+0000 7f42acff9640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f429006af60 0x7f429006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:44.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.662+0000 7f42acff9640 1 -- 172.21.15.45:0/1420407365 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f42a00897c0 con 0x7f429c0047c0 2023-11-28T20:50:44.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.662+0000 7f42a7fff640 1 -- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f429006af60 msgr2=0x7f429006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:44.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.662+0000 7f42a7fff640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f429006af60 0x7f429006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:44.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.666+0000 7f42acff9640 1 -- 172.21.15.45:0/1420407365 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42a00544b0 con 0x7f429c0047c0 2023-11-28T20:50:44.714 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:44.713+0000 7f9384615040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:50:44.715 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:44.712+0000 7f1b376da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:50:44.764 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:44.762+0000 7f1b376da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:50:44.769 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:44.768+0000 7f9384615040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:50:44.823 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:44.822+0000 7f9384615040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:50:44.825 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:44.823+0000 7f1b376da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:50:44.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.859+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42b00b8c30 con 0x7f429c0047c0 2023-11-28T20:50:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.860+0000 7f42acff9640 1 -- 172.21.15.45:0/1420407365 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f42a0057ad0 con 0x7f429c0047c0 2023-11-28T20:50:44.861 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:50:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f429006af60 msgr2=0x7f429006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f429006af60 0x7f429006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 msgr2=0x7f42b01e7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f42b01e7320 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f42a000e550 tx=0x7f42a000ea10 comp rx=0 tx=0).stop 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 shutdown_connections 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f429006af60 0x7f429006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42b0160370 0x7f42b00b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42b015f3e0 0x7f42b0156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 --2- 172.21.15.45:0/1420407365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f429c0047c0 0x7f42b01e7320 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.862+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 >> 172.21.15.45:0/1420407365 conn(0x7f42b0070b20 msgr2=0x7f42b00d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.863+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 shutdown_connections 2023-11-28T20:50:44.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:44.863+0000 7f42b5fde640 1 -- 172.21.15.45:0/1420407365 wait complete. 2023-11-28T20:50:44.874 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:44.872+0000 7f1b376da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:50:44.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:50:44.893 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:44.890+0000 7f9384615040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:50:44.944 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:44.943+0000 7f9384615040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:50:44.986 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:44.983+0000 7f1b376da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:50:45.047 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:45.045+0000 7f1b376da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:45.064 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:45.062+0000 7f9384615040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:50:45.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.092+0000 7f1b884da640 1 Processor -- start 2023-11-28T20:50:45.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.092+0000 7f1b884da640 1 -- start start 2023-11-28T20:50:45.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.094+0000 7f1b884da640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1b8015d180 0x7f1b8006dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.094+0000 7f1b884da640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b8006e7d0 0x7f1b800ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.094+0000 7f1b884da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 0x7f1b800cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.094+0000 7f1b884da640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b80160fd0 con 0x7f1b8006e7d0 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.094+0000 7f1b884da640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b800b29b0 con 0x7f1b8015d180 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.094+0000 7f1b884da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800b2c00 con 0x7f1b8015d560 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.094+0000 7f1b85a75640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b8006e7d0 0x7f1b800ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.095+0000 7f1b85a75640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b8006e7d0 0x7f1b800ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52364/0 (socket says 172.21.15.45:52364) 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.095+0000 7f1b85a75640 1 -- 172.21.15.45:0/40954999 learned_addr learned my addr 172.21.15.45:0/40954999 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.095+0000 7f1b86276640 1 --2- 172.21.15.45:0/40954999 >> v2:172.21.15.190:3300/0 conn(0x7f1b8015d180 0x7f1b8006dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.095+0000 7f1b86a77640 1 --2- 172.21.15.45:0/40954999 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 0x7f1b800cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.095+0000 7f1b85a75640 1 -- 172.21.15.45:0/40954999 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 msgr2=0x7f1b800cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.095+0000 7f1b85a75640 1 --2- 172.21.15.45:0/40954999 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 0x7f1b800cf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.095+0000 7f1b85a75640 1 -- 172.21.15.45:0/40954999 >> v2:172.21.15.190:3300/0 conn(0x7f1b8015d180 msgr2=0x7f1b8006dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.095+0000 7f1b85a75640 1 --2- 172.21.15.45:0/40954999 >> v2:172.21.15.190:3300/0 conn(0x7f1b8015d180 0x7f1b8006dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.095+0000 7f1b85a75640 1 -- 172.21.15.45:0/40954999 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b800b2ee0 con 0x7f1b8006e7d0 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.097+0000 7f1b85a75640 1 --2- 172.21.15.45:0/40954999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 0x7f1b800ac8b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1b800b2980 tx=0x7f1b7002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45f36abdc839e85 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.097+0000 7f1b85274640 1 -- 172.21.15.45:0/40954999 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70031860 con 0x7f1b8006e7d0 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.097+0000 7f1b85274640 1 -- 172.21.15.45:0/40954999 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b700319c0 con 0x7f1b8006e7d0 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.097+0000 7f1b85274640 1 -- 172.21.15.45:0/40954999 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70031d50 con 0x7f1b8006e7d0 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.098+0000 7f1b884da640 1 -- 172.21.15.45:0/40954999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 msgr2=0x7f1b800ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.098+0000 7f1b884da640 1 --2- 172.21.15.45:0/40954999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 0x7f1b800ac8b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1b800b2980 tx=0x7f1b7002b900 comp rx=0 tx=0).stop 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.098+0000 7f1b884da640 1 -- 172.21.15.45:0/40954999 shutdown_connections 2023-11-28T20:50:45.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.098+0000 7f1b884da640 1 --2- 172.21.15.45:0/40954999 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 0x7f1b800cf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.098+0000 7f1b884da640 1 --2- 172.21.15.45:0/40954999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 0x7f1b800ac8b0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.098+0000 7f1b884da640 1 --2- 172.21.15.45:0/40954999 >> v2:172.21.15.190:3300/0 conn(0x7f1b8015d180 0x7f1b8006dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.098+0000 7f1b884da640 1 -- 172.21.15.45:0/40954999 >> 172.21.15.45:0/40954999 conn(0x7f1b8006a500 msgr2=0x7f1b8006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:45.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.098+0000 7f1b884da640 1 -- 172.21.15.45:0/40954999 shutdown_connections 2023-11-28T20:50:45.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.098+0000 7f1b884da640 1 -- 172.21.15.45:0/40954999 wait complete. 2023-11-28T20:50:45.103 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:45.094+0000 7f1b376da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:50:45.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.100+0000 7f1b884da640 1 Processor -- start 2023-11-28T20:50:45.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.100+0000 7f1b884da640 1 -- start start 2023-11-28T20:50:45.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b884da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 0x7f1b801f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b884da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b8015d180 0x7f1b801f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b884da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 0x7f1b8015ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b884da640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800cee70 con 0x7f1b8006e7d0 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b884da640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8015a530 con 0x7f1b8015d180 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b884da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80151f70 con 0x7f1b8015d560 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b86276640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 0x7f1b801f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b86276640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 0x7f1b801f6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52380/0 (socket says 172.21.15.45:52380) 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b86276640 1 -- 172.21.15.45:0/986311202 learned_addr learned my addr 172.21.15.45:0/986311202 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b85a75640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b8015d180 0x7f1b801f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.101+0000 7f1b86a77640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 0x7f1b8015ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b86276640 1 -- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 msgr2=0x7f1b8015ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b86276640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 0x7f1b8015ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b86276640 1 -- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b8015d180 msgr2=0x7f1b801f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b86276640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b8015d180 0x7f1b801f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b86276640 1 -- 172.21.15.45:0/986311202 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b700094d0 con 0x7f1b8006e7d0 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b86276640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 0x7f1b801f6330 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c007e90 tx=0x7f1b7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b76ffd640 1 -- 172.21.15.45:0/986311202 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c014620 con 0x7f1b8006e7d0 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b76ffd640 1 -- 172.21.15.45:0/986311202 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c014780 con 0x7f1b8006e7d0 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b76ffd640 1 -- 172.21.15.45:0/986311202 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c014b30 con 0x7f1b8006e7d0 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b801f3990 con 0x7f1b8006e7d0 2023-11-28T20:50:45.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.102+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b800cc1d0 con 0x7f1b8006e7d0 2023-11-28T20:50:45.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.104+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8006b490 con 0x7f1b8006e7d0 2023-11-28T20:50:45.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.105+0000 7f1b76ffd640 1 -- 172.21.15.45:0/986311202 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 34) v1 ==== 89334+0+0 (secure 0 0 0) 0x7f1b7c014c90 con 0x7f1b8006e7d0 2023-11-28T20:50:45.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.106+0000 7f1b76ffd640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f1b6406af60 0x7f1b6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.106+0000 7f1b76ffd640 1 -- 172.21.15.45:0/986311202 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f1b7c08a470 con 0x7f1b8006e7d0 2023-11-28T20:50:45.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.106+0000 7f1b85a75640 1 -- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f1b6406af60 msgr2=0x7f1b6406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:45.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.106+0000 7f1b85a75640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f1b6406af60 0x7f1b6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.114+0000 7f1b76ffd640 1 -- 172.21.15.45:0/986311202 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b7c0550e0 con 0x7f1b8006e7d0 2023-11-28T20:50:45.151 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:45.150+0000 7f9384615040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:45.154 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:45.152+0000 7f1b376da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:50:45.203 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:45.201+0000 7f1b376da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:50:45.239 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:45.237+0000 7f9384615040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:50:45.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.307+0000 7f1b85a75640 1 -- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f1b6406af60 msgr2=0x7f1b6406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:45.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.307+0000 7f1b85a75640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f1b6406af60 0x7f1b6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:45.328 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:45.327+0000 7f9384615040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:50:45.382 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:45.381+0000 7f9384615040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:50:45.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.403+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1b800733d0 con 0x7f1b8006e7d0 2023-11-28T20:50:45.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.403+0000 7f1b76ffd640 1 -- 172.21.15.45:0/986311202 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1b7c058700 con 0x7f1b8006e7d0 2023-11-28T20:50:45.405 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:50:45.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f1b6406af60 msgr2=0x7f1b6406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:45.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f1b6406af60 0x7f1b6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 msgr2=0x7f1b801f6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 0x7f1b801f6330 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c007e90 tx=0x7f1b7c00e4b0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 shutdown_connections 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f1b6406af60 0x7f1b6406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b8015d560 0x7f1b8015ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b8015d180 0x7f1b801f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 --2- 172.21.15.45:0/986311202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b8006e7d0 0x7f1b801f6330 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 >> 172.21.15.45:0/986311202 conn(0x7f1b8006a500 msgr2=0x7f1b800adb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 shutdown_connections 2023-11-28T20:50:45.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.406+0000 7f1b884da640 1 -- 172.21.15.45:0/986311202 wait complete. 2023-11-28T20:50:45.426 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:50:45.586 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:45.584+0000 7f1b376da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:50:45.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.681+0000 7f0281148640 1 Processor -- start 2023-11-28T20:50:45.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.681+0000 7f0281148640 1 -- start start 2023-11-28T20:50:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.683+0000 7f0281148640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f027c06e7d0 0x7f027c06dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.683+0000 7f0281148640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c0b29b0 con 0x7f027c15d560 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.683+0000 7f0281148640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c0b2c00 con 0x7f027c15d180 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.683+0000 7f0281148640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027c160fd0 con 0x7f027c06e7d0 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.683+0000 7f027b59e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f027c15d560 0x7f027c0cf5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45012/0 (socket says 172.21.15.45:45012) 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.683+0000 7f027b59e640 1 -- 172.21.15.45:0/942976258 learned_addr learned my addr 172.21.15.45:0/942976258 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f027a59c640 1 --2- 172.21.15.45:0/942976258 >> v2:172.21.15.190:3300/0 conn(0x7f027c06e7d0 0x7f027c06dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139136409 0 0) 0x7f027c0b29b0 con 0x7f027c15d560 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0264003520 con 0x7f027c15d560 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 118291448 0 0) 0x7f0264003520 con 0x7f027c15d560 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 >> v2:172.21.15.190:3300/0 conn(0x7f027c06e7d0 msgr2=0x7f027c06dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 --2- 172.21.15.45:0/942976258 >> v2:172.21.15.190:3300/0 conn(0x7f027c06e7d0 0x7f027c06dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 >> v1:172.21.15.190:6789/0 conn(0x7f027c15d180 legacy=0x7f027c0ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c0b2ee0 con 0x7f027c15d560 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f02740019f0 con 0x7f027c15d560 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 >> v1:172.21.15.45:6789/0 conn(0x7f027c15d560 legacy=0x7f027c0cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0264003970 0x7f0264003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02640047c0 0x7f0264004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0264005180 0x7f026400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c160fd0 con 0x7f0264003970 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026400c020 con 0x7f0264005180 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.684+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026400c1e0 con 0x7f02640047c0 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.685+0000 7f027ad9d640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0264003970 0x7f0264003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.685+0000 7f027a59c640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02640047c0 0x7f0264004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.685+0000 7f027b59e640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0264005180 0x7f026400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.685+0000 7f027ad9d640 1 -- 172.21.15.45:0/942976258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02640047c0 msgr2=0x7f0264004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:45.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.685+0000 7f027ad9d640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02640047c0 0x7f0264004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.685+0000 7f027ad9d640 1 -- 172.21.15.45:0/942976258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0264005180 msgr2=0x7f026400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:45.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.685+0000 7f027ad9d640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0264005180 0x7f026400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.685+0000 7f027ad9d640 1 -- 172.21.15.45:0/942976258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0264003520 con 0x7f0264003970 2023-11-28T20:50:45.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.685+0000 7f027b59e640 1 -- 172.21.15.45:0/942976258 reap_dead start 2023-11-28T20:50:45.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.686+0000 7f027ad9d640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0264003970 0x7f0264003d90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f026c001750 tx=0x7f02640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8cbaa9fa3330397 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:45.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c0052d0 con 0x7f0264003970 2023-11-28T20:50:45.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c02e050 con 0x7f0264003970 2023-11-28T20:50:45.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0279d9b640 1 -- 172.21.15.45:0/942976258 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c003830 con 0x7f0264003970 2023-11-28T20:50:45.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0281148640 1 -- 172.21.15.45:0/942976258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0264003970 msgr2=0x7f0264003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0281148640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0264003970 0x7f0264003d90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f026c001750 tx=0x7f02640057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0281148640 1 -- 172.21.15.45:0/942976258 shutdown_connections 2023-11-28T20:50:45.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0281148640 1 --2- 172.21.15.45:0/942976258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0264003970 0x7f0264003d90 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0281148640 1 -- 172.21.15.45:0/942976258 >> 172.21.15.45:0/942976258 conn(0x7f027c06a500 msgr2=0x7f027c06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:45.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0281148640 1 -- 172.21.15.45:0/942976258 shutdown_connections 2023-11-28T20:50:45.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.687+0000 7f0281148640 1 -- 172.21.15.45:0/942976258 wait complete. 2023-11-28T20:50:45.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.689+0000 7f0281148640 1 Processor -- start 2023-11-28T20:50:45.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.689+0000 7f0281148640 1 -- start start 2023-11-28T20:50:45.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.690+0000 7f0281148640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0264003970 0x7f027c14d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.690+0000 7f0281148640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f027c1f64f0 0x7f027c14a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.690+0000 7f0281148640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f027c15f400 0x7f027c15f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.690+0000 7f0281148640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c160070 con 0x7f027c15f400 2023-11-28T20:50:45.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.690+0000 7f0281148640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c14a750 con 0x7f0264003970 2023-11-28T20:50:45.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.690+0000 7f0281148640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027c15fdc0 con 0x7f027c1f64f0 2023-11-28T20:50:45.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.690+0000 7f027b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f027c15f400 0x7f027c15f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f027c15f400 0x7f027c15f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52398/0 (socket says 172.21.15.45:52398) 2023-11-28T20:50:45.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027b59e640 1 -- 172.21.15.45:0/1438561226 learned_addr learned my addr 172.21.15.45:0/1438561226 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:45.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027a59c640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f027c1f64f0 0x7f027c14a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027ad9d640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0264003970 0x7f027c14d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:45.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027b59e640 1 -- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f027c1f64f0 msgr2=0x7f027c14a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027b59e640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f027c1f64f0 0x7f027c14a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027b59e640 1 -- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0264003970 msgr2=0x7f027c14d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027b59e640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0264003970 0x7f027c14d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027b59e640 1 -- 172.21.15.45:0/1438561226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c15d9c0 con 0x7f027c15f400 2023-11-28T20:50:45.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f027b59e640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f027c15f400 0x7f027c15f880 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0274000f80 tx=0x7f027400df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:45.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.691+0000 7f02637fe640 1 -- 172.21.15.45:0/1438561226 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027400b860 con 0x7f027c15f400 2023-11-28T20:50:45.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.692+0000 7f02637fe640 1 -- 172.21.15.45:0/1438561226 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027400ba00 con 0x7f027c15f400 2023-11-28T20:50:45.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.692+0000 7f02637fe640 1 -- 172.21.15.45:0/1438561226 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027400bdb0 con 0x7f027c15f400 2023-11-28T20:50:45.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.692+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c1f69f0 con 0x7f027c15f400 2023-11-28T20:50:45.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.692+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c1f6fb0 con 0x7f027c15f400 2023-11-28T20:50:45.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.693+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c06b490 con 0x7f027c15f400 2023-11-28T20:50:45.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.694+0000 7f02637fe640 1 -- 172.21.15.45:0/1438561226 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 34) v1 ==== 89334+0+0 (secure 0 0 0) 0x7f0274007110 con 0x7f027c15f400 2023-11-28T20:50:45.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.695+0000 7f02637fe640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f026806af90 0x7f026806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:45.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.695+0000 7f02637fe640 1 -- 172.21.15.45:0/1438561226 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f0274089930 con 0x7f027c15f400 2023-11-28T20:50:45.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.695+0000 7f027ad9d640 1 -- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f026806af90 msgr2=0x7f026806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:45.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.695+0000 7f027ad9d640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f026806af90 0x7f026806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:45.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.699+0000 7f02637fe640 1 -- 172.21.15.45:0/1438561226 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0274054510 con 0x7f027c15f400 2023-11-28T20:50:45.747 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:45.744+0000 7f1b376da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:50:45.798 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:45.797+0000 7f9384615040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:50:45.854 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:45.851+0000 7f1b376da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:50:45.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.890+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f027c0733d0 con 0x7f027c15f400 2023-11-28T20:50:45.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.891+0000 7f02637fe640 1 -- 172.21.15.45:0/1438561226 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0274057b30 con 0x7f027c15f400 2023-11-28T20:50:45.892 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.893+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f026806af90 msgr2=0x7f026806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.893+0000 7f0281148640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f026806af90 0x7f026806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f027c15f400 msgr2=0x7f027c15f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f027c15f400 0x7f027c15f880 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0274000f80 tx=0x7f027400df70 comp rx=0 tx=0).stop 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 shutdown_connections 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f026806af90 0x7f026806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f027c15f400 0x7f027c15f880 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f027c1f64f0 0x7f027c14a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 --2- 172.21.15.45:0/1438561226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0264003970 0x7f027c14d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 >> 172.21.15.45:0/1438561226 conn(0x7f027c06a500 msgr2=0x7f027c0cf580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:45.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 shutdown_connections 2023-11-28T20:50:45.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:45.894+0000 7f0281148640 1 -- 172.21.15.45:0/1438561226 wait complete. 2023-11-28T20:50:45.904 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:45.902+0000 7f1b376da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:50:45.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:50:45.971 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:45.970+0000 7f9384615040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:50:46.011 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:46.009+0000 7f1b376da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:46.088 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:46.087+0000 7f9384615040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:50:46.117 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:46.114+0000 7f1b376da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:50:46.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.124+0000 7fd9f2690640 1 Processor -- start 2023-11-28T20:50:46.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.124+0000 7fd9f2690640 1 -- start start 2023-11-28T20:50:46.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.125+0000 7fd9f2690640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd9ec15b820 0x7fd9ec06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.125+0000 7fd9f2690640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec15f4d0 0x7fd9ec0cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.125+0000 7fd9f2690640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec15fa80 con 0x7fd9ec0cf020 2023-11-28T20:50:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.125+0000 7fd9f2690640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ec15fd60 con 0x7fd9ec15b820 2023-11-28T20:50:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.125+0000 7fd9f2690640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec0cf9f0 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.126+0000 7fd9ebfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd9ec15b820 0x7fd9ec06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.126+0000 7fd9eb7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd9ec0cf020 0x7fd9ec15edc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:33922/0 (socket says 172.21.15.45:33922) 2023-11-28T20:50:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.126+0000 7fd9eb7fe640 1 -- 172.21.15.45:0/1573866436 learned_addr learned my addr 172.21.15.45:0/1573866436 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.126+0000 7fd9f0c2d640 1 --2- 172.21.15.45:0/1573866436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec15f4d0 0x7fd9ec0cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.127+0000 7fd9ebfff640 1 -- 172.21.15.45:0/1573866436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec15f4d0 msgr2=0x7fd9ec0cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.127+0000 7fd9ebfff640 1 --2- 172.21.15.45:0/1573866436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec15f4d0 0x7fd9ec0cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.127+0000 7fd9ebfff640 1 -- 172.21.15.45:0/1573866436 >> v1:172.21.15.190:6789/0 conn(0x7fd9ec0cf020 legacy=0x7fd9ec15edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.127+0000 7fd9ebfff640 1 -- 172.21.15.45:0/1573866436 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec06cf40 con 0x7fd9ec15b820 2023-11-28T20:50:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.127+0000 7fd9f0c2d640 1 --2- 172.21.15.45:0/1573866436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec15f4d0 0x7fd9ec0cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:50:46.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.128+0000 7fd9ebfff640 1 --2- 172.21.15.45:0/1573866436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ec15b820 0x7fd9ec06da20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0097d0 tx=0x7fd9dc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=160a9a38cbfc17bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:46.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.129+0000 7fd9eaffd640 1 -- 172.21.15.45:0/1573866436 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc02d040 con 0x7fd9ec15b820 2023-11-28T20:50:46.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.129+0000 7fd9eaffd640 1 -- 172.21.15.45:0/1573866436 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc031810 con 0x7fd9ec15b820 2023-11-28T20:50:46.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.129+0000 7fd9eaffd640 1 -- 172.21.15.45:0/1573866436 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc031ba0 con 0x7fd9ec15b820 2023-11-28T20:50:46.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.129+0000 7fd9f2690640 1 -- 172.21.15.45:0/1573866436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ec15b820 msgr2=0x7fd9ec06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:46.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.129+0000 7fd9f2690640 1 --2- 172.21.15.45:0/1573866436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ec15b820 0x7fd9ec06da20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0097d0 tx=0x7fd9dc02b8b0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.129+0000 7fd9f2690640 1 -- 172.21.15.45:0/1573866436 shutdown_connections 2023-11-28T20:50:46.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.129+0000 7fd9f2690640 1 --2- 172.21.15.45:0/1573866436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec15f4d0 0x7fd9ec0cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.129+0000 7fd9f2690640 1 --2- 172.21.15.45:0/1573866436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ec15b820 0x7fd9ec06da20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.129+0000 7fd9f2690640 1 -- 172.21.15.45:0/1573866436 >> 172.21.15.45:0/1573866436 conn(0x7fd9ec06a6e0 msgr2=0x7fd9ec06aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.130+0000 7fd9f2690640 1 -- 172.21.15.45:0/1573866436 shutdown_connections 2023-11-28T20:50:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.130+0000 7fd9f2690640 1 -- 172.21.15.45:0/1573866436 wait complete. 2023-11-28T20:50:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.131+0000 7fd9f2690640 1 Processor -- start 2023-11-28T20:50:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.131+0000 7fd9f2690640 1 -- start start 2023-11-28T20:50:46.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.132+0000 7fd9f2690640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec0cf020 0x7fd9ec0b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.132+0000 7fd9f2690640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ec15b820 0x7fd9ec15dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.132+0000 7fd9f2690640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ec15f4d0 0x7fd9ec142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.132+0000 7fd9f2690640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec0b39c0 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.132+0000 7fd9f2690640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec09b1d0 con 0x7fd9ec15b820 2023-11-28T20:50:46.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.132+0000 7fd9f2690640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec09b4b0 con 0x7fd9ec0cf020 2023-11-28T20:50:46.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.133+0000 7fd9f0c2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ec15f4d0 0x7fd9ec142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.133+0000 7fd9f0c2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ec15f4d0 0x7fd9ec142420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52406/0 (socket says 172.21.15.45:52406) 2023-11-28T20:50:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.133+0000 7fd9f0c2d640 1 -- 172.21.15.45:0/4197295453 learned_addr learned my addr 172.21.15.45:0/4197295453 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.133+0000 7fd9eb7fe640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ec15b820 0x7fd9ec15dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.133+0000 7fd9ebfff640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec0cf020 0x7fd9ec0b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9f0c2d640 1 -- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec0cf020 msgr2=0x7fd9ec0b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9f0c2d640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec0cf020 0x7fd9ec0b2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9f0c2d640 1 -- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ec15b820 msgr2=0x7fd9ec15dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9f0c2d640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ec15b820 0x7fd9ec15dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9f0c2d640 1 -- 172.21.15.45:0/4197295453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e4009a70 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9f0c2d640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ec15f4d0 0x7fd9ec142420 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400efc0 tx=0x7fd9e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9e8ff9640 1 -- 172.21.15.45:0/4197295453 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4009cd0 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9e8ff9640 1 -- 172.21.15.45:0/4197295453 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e400f040 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc009400 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9e8ff9640 1 -- 172.21.15.45:0/4197295453 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4003990 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.134+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec09ba70 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.136+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ec073440 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.137+0000 7fd9e8ff9640 1 -- 172.21.15.45:0/4197295453 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 34) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fd9e4014480 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.138+0000 7fd9e8ff9640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fd9d806af60 0x7fd9d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.138+0000 7fd9e8ff9640 1 -- 172.21.15.45:0/4197295453 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fd9e408a840 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.138+0000 7fd9ebfff640 1 -- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fd9d806af60 msgr2=0x7fd9d806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.139+0000 7fd9ebfff640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fd9d806af60 0x7fd9d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:46.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.143+0000 7fd9e8ff9640 1 -- 172.21.15.45:0/4197295453 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9e40553a0 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.145 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:46.144+0000 7f9384615040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:50:46.261 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:46.260+0000 7f9384615040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:46.274 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:46.271+0000 7f1b376da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:46.322 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:46.320+0000 7f1b376da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:46.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.340+0000 7fd9ebfff640 1 -- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fd9d806af60 msgr2=0x7fd9d806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:46.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.340+0000 7fd9ebfff640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fd9d806af60 0x7fd9d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:46.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.350+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd9ec06c8c0 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.351+0000 7fd9e8ff9640 1 -- 172.21.15.45:0/4197295453 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v89) v1 ==== 80+0+476 (secure 0 0 0) 0x7fd9e40589c0 con 0x7fd9ec15f4d0 2023-11-28T20:50:46.352 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1902490820 2023-11-29T20:50:41.547044+0000 2023-11-28T20:50:46.352 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1766221473 2023-11-29T20:50:41.547044+0000 2023-11-28T20:50:46.352 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1382859164 2023-11-29T20:50:41.547044+0000 2023-11-28T20:50:46.353 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/108887886 2023-11-29T20:50:41.547044+0000 2023-11-28T20:50:46.353 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1155029644 2023-11-29T20:50:41.547044+0000 2023-11-28T20:50:46.353 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/108887886 2023-11-29T20:50:41.547044+0000 2023-11-28T20:50:46.353 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2238215747 2023-11-29T20:50:41.547044+0000 2023-11-28T20:50:46.353 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3448409370 2023-11-29T20:50:41.547044+0000 2023-11-28T20:50:46.353 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fd9d806af60 msgr2=0x7fd9d806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fd9d806af60 0x7fd9d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ec15f4d0 msgr2=0x7fd9ec142420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ec15f4d0 0x7fd9ec142420 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400efc0 tx=0x7fd9e400c490 comp rx=0 tx=0).stop 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 shutdown_connections 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7fd9d806af60 0x7fd9d806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ec15f4d0 0x7fd9ec142420 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ec15b820 0x7fd9ec15dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 --2- 172.21.15.45:0/4197295453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ec0cf020 0x7fd9ec0b2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.353+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 >> 172.21.15.45:0/4197295453 conn(0x7fd9ec06a6e0 msgr2=0x7fd9ec0cdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:46.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.354+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 shutdown_connections 2023-11-28T20:50:46.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.354+0000 7fd9f2690640 1 -- 172.21.15.45:0/4197295453 wait complete. 2023-11-28T20:50:46.371 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T20:50:46.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:46.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T20:50:46.378 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:46.376+0000 7f9384615040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:50:46.443 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:46.440+0000 7f1b376da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:50:46.463 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:50:46.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:50:46.463 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T20:50:46.505 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:50:46.506 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:46.554 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:46.552+0000 7f9384615040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:50:46.559 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:50:46.559 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:50:46.559 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:46.606 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:46.605+0000 7f9384615040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:50:46.610 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:50:46.611 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:50:46.611 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:50:46.662 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:50:46.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:46.736 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:46.734+0000 7f9384615040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:50:46.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.873+0000 7f99dffff640 1 Processor -- start 2023-11-28T20:50:46.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.873+0000 7f99dffff640 1 -- start start 2023-11-28T20:50:46.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.875+0000 7f99dffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f99e00711e0 0x7f99e015ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.875+0000 7f99dffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00cf120 0x7f99e006e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.875+0000 7f99dffff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e015dc00 con 0x7f99e0157880 2023-11-28T20:50:46.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.875+0000 7f99dffff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e015dee0 con 0x7f99e00711e0 2023-11-28T20:50:46.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.875+0000 7f99dffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e015e1c0 con 0x7f99e00cf120 2023-11-28T20:50:46.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99deffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f99e0157880 0x7f99e0070ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45018/0 (socket says 172.21.15.45:45018) 2023-11-28T20:50:46.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99deffd640 1 -- 172.21.15.45:0/3484908454 learned_addr learned my addr 172.21.15.45:0/3484908454 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:46.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99de7fc640 1 --2- 172.21.15.45:0/3484908454 >> v2:172.21.15.45:3300/0 conn(0x7f99e00711e0 0x7f99e015ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99df7fe640 1 --2- 172.21.15.45:0/3484908454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00cf120 0x7f99e006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99ddffb640 1 -- 172.21.15.45:0/3484908454 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606639926 0 0) 0x7f99e015dc00 con 0x7f99e0157880 2023-11-28T20:50:46.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99ddffb640 1 -- 172.21.15.45:0/3484908454 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c8003520 con 0x7f99e0157880 2023-11-28T20:50:46.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99de7fc640 1 -- 172.21.15.45:0/3484908454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00cf120 msgr2=0x7f99e006e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:46.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99de7fc640 1 --2- 172.21.15.45:0/3484908454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00cf120 0x7f99e006e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99de7fc640 1 -- 172.21.15.45:0/3484908454 >> v1:172.21.15.45:6789/0 conn(0x7f99e0157880 legacy=0x7f99e0070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:46.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.876+0000 7f99de7fc640 1 -- 172.21.15.45:0/3484908454 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e015e4a0 con 0x7f99e00711e0 2023-11-28T20:50:46.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.877+0000 7f99de7fc640 1 --2- 172.21.15.45:0/3484908454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e00711e0 0x7f99e015ea70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f99d40097a0 tx=0x7f99d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d5b5a3d7ba25e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:46.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.878+0000 7f99ddffb640 1 -- 172.21.15.45:0/3484908454 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d402e070 con 0x7f99e00711e0 2023-11-28T20:50:46.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.878+0000 7f99ddffb640 1 -- 172.21.15.45:0/3484908454 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d40315a0 con 0x7f99e00711e0 2023-11-28T20:50:46.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.878+0000 7f99ddffb640 1 -- 172.21.15.45:0/3484908454 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4031910 con 0x7f99e00711e0 2023-11-28T20:50:46.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.878+0000 7f99dffff640 1 -- 172.21.15.45:0/3484908454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e00711e0 msgr2=0x7f99e015ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:46.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.878+0000 7f99dffff640 1 --2- 172.21.15.45:0/3484908454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e00711e0 0x7f99e015ea70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f99d40097a0 tx=0x7f99d402b840 comp rx=0 tx=0).stop 2023-11-28T20:50:46.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.879+0000 7f99dffff640 1 -- 172.21.15.45:0/3484908454 shutdown_connections 2023-11-28T20:50:46.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.879+0000 7f99dffff640 1 --2- 172.21.15.45:0/3484908454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00cf120 0x7f99e006e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.879+0000 7f99dffff640 1 --2- 172.21.15.45:0/3484908454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e00711e0 0x7f99e015ea70 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.879+0000 7f99dffff640 1 -- 172.21.15.45:0/3484908454 >> 172.21.15.45:0/3484908454 conn(0x7f99e00b14b0 msgr2=0x7f99e00b18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.879+0000 7f99dffff640 1 -- 172.21.15.45:0/3484908454 shutdown_connections 2023-11-28T20:50:46.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.879+0000 7f99dffff640 1 -- 172.21.15.45:0/3484908454 wait complete. 2023-11-28T20:50:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.880+0000 7f99dffff640 1 Processor -- start 2023-11-28T20:50:46.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.881+0000 7f99dffff640 1 -- start start 2023-11-28T20:50:46.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.881+0000 7f99dffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00711e0 0x7f99e0148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.881+0000 7f99dffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99e00cf120 0x7f99e01e3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e0157880 0x7f99e01e5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99dffff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e01e42e0 con 0x7f99e0157880 2023-11-28T20:50:46.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99dffff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e01e45c0 con 0x7f99e00cf120 2023-11-28T20:50:46.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99dffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e01e48a0 con 0x7f99e00711e0 2023-11-28T20:50:46.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99df7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e0157880 0x7f99e01e5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99de7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99e00cf120 0x7f99e01e3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99df7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e0157880 0x7f99e01e5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52420/0 (socket says 172.21.15.45:52420) 2023-11-28T20:50:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99df7fe640 1 -- 172.21.15.45:0/2464490467 learned_addr learned my addr 172.21.15.45:0/2464490467 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99deffd640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00711e0 0x7f99e0148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.882+0000 7f99df7fe640 1 -- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00711e0 msgr2=0x7f99e0148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99df7fe640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00711e0 0x7f99e0148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99df7fe640 1 -- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99e00cf120 msgr2=0x7f99e01e3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99df7fe640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99e00cf120 0x7f99e01e3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99df7fe640 1 -- 172.21.15.45:0/2464490467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4009400 con 0x7f99e0157880 2023-11-28T20:50:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99df7fe640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e0157880 0x7f99e01e5c50 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f99d8007e30 tx=0x7f99d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99c77fe640 1 -- 172.21.15.45:0/2464490467 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8014620 con 0x7f99e0157880 2023-11-28T20:50:46.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99c77fe640 1 -- 172.21.15.45:0/2464490467 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d800ee90 con 0x7f99e0157880 2023-11-28T20:50:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99c77fe640 1 -- 172.21.15.45:0/2464490467 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d80149c0 con 0x7f99e0157880 2023-11-28T20:50:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e0157de0 con 0x7f99e0157880 2023-11-28T20:50:46.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.883+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e01e4e60 con 0x7f99e0157880 2023-11-28T20:50:46.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.885+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e0067fb0 con 0x7f99e0157880 2023-11-28T20:50:46.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.887+0000 7f99c77fe640 1 -- 172.21.15.45:0/2464490467 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 34) v1 ==== 89334+0+0 (secure 0 0 0) 0x7f99d8014b20 con 0x7f99e0157880 2023-11-28T20:50:46.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.888+0000 7f99c77fe640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f99cc06af60 0x7f99cc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:46.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.888+0000 7f99c77fe640 1 -- 172.21.15.45:0/2464490467 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f99d808a390 con 0x7f99e0157880 2023-11-28T20:50:46.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.888+0000 7f99deffd640 1 -- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f99cc06af60 msgr2=0x7f99cc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:46.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.888+0000 7f99deffd640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f99cc06af60 0x7f99cc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:46.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:46.892+0000 7f99c77fe640 1 -- 172.21.15.45:0/2464490467 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99d8055010 con 0x7f99e0157880 2023-11-28T20:50:47.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.089+0000 7f99deffd640 1 -- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f99cc06af60 msgr2=0x7f99cc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:47.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.089+0000 7f99deffd640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f99cc06af60 0x7f99cc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:47.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.125+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99e004e920 con 0x7f99e0157880 2023-11-28T20:50:47.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.126+0000 7f99c77fe640 1 -- 172.21.15.45:0/2464490467 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4914 (secure 0 0 0) 0x7f99d8058630 con 0x7f99e0157880 2023-11-28T20:50:47.127 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:47.127 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5438,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2396470524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2396470524},{"type":"v1","addr":"172.21.15.45:6835","nonce":2396470524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/436103276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":436103276},{"type":"v1","addr":"172.21.15.190:6836","nonce":436103276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/834328551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":834328551},{"type":"v1","addr":"172.21.15.45:6834","nonce":834328551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:50:26.812959+0000","modified":"2023-11-28T20:50:29.910521+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6837/3032907708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3032907708},{"type":"v1","addr":"172.21.15.190:6837","nonce":3032907708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T20:50:47.128 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 36 2023-11-28T20:50:47.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f99cc06af60 msgr2=0x7f99cc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f99cc06af60 0x7f99cc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e0157880 msgr2=0x7f99e01e5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e0157880 0x7f99e01e5c50 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f99d8007e30 tx=0x7f99d800e4b0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 shutdown_connections 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f99cc06af60 0x7f99cc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99e0157880 0x7f99e01e5c50 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99e00cf120 0x7f99e01e3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 --2- 172.21.15.45:0/2464490467 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99e00711e0 0x7f99e0148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 >> 172.21.15.45:0/2464490467 conn(0x7f99e00b14b0 msgr2=0x7f99e015a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 shutdown_connections 2023-11-28T20:50:47.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.129+0000 7f99dffff640 1 -- 172.21.15.45:0/2464490467 wait complete. 2023-11-28T20:50:47.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:47.272 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.269+0000 7f1b376da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:50:47.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.345+0000 7f52f58e5640 1 Processor -- start 2023-11-28T20:50:47.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.345+0000 7f52f58e5640 1 -- start start 2023-11-28T20:50:47.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.347+0000 7f52f58e5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 0x7f52f00ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.347+0000 7f52f58e5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52f00b1340 0x7f52f0071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.347+0000 7f52f58e5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f00ce710 con 0x7f52f00add40 2023-11-28T20:50:47.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.347+0000 7f52f58e5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f00ae5a0 con 0x7f52f00b1340 2023-11-28T20:50:47.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.347+0000 7f52f58e5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0160fe0 con 0x7f52f015d320 2023-11-28T20:50:47.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.347+0000 7f52ef7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52f00b1340 0x7f52f0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.347+0000 7f52ef7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52f00b1340 0x7f52f0071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:33162/0 (socket says 172.21.15.45:33162) 2023-11-28T20:50:47.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.347+0000 7f52ef7fe640 1 -- 172.21.15.45:0/1620073363 learned_addr learned my addr 172.21.15.45:0/1620073363 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:47.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.347+0000 7f52eeffd640 1 --2- 172.21.15.45:0/1620073363 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 0x7f52f00ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.348+0000 7f52ef7fe640 1 -- 172.21.15.45:0/1620073363 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 msgr2=0x7f52f00ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:47.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.348+0000 7f52ef7fe640 1 --2- 172.21.15.45:0/1620073363 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 0x7f52f00ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.348+0000 7f52ef7fe640 1 -- 172.21.15.45:0/1620073363 >> v1:172.21.15.190:6789/0 conn(0x7f52f00add40 legacy=0x7f52f00b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.348+0000 7f52ef7fe640 1 -- 172.21.15.45:0/1620073363 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00ae2c0 con 0x7f52f00b1340 2023-11-28T20:50:47.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.349+0000 7f52ef7fe640 1 --2- 172.21.15.45:0/1620073363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f00b1340 0x7f52f0071190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009e50 tx=0x7f52e802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e46b1b3bd5e90e22 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:47.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.350+0000 7f52ee7fc640 1 -- 172.21.15.45:0/1620073363 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e802cb00 con 0x7f52f00b1340 2023-11-28T20:50:47.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.350+0000 7f52ee7fc640 1 -- 172.21.15.45:0/1620073363 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e802cc60 con 0x7f52f00b1340 2023-11-28T20:50:47.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.350+0000 7f52ee7fc640 1 -- 172.21.15.45:0/1620073363 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e80343f0 con 0x7f52f00b1340 2023-11-28T20:50:47.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.350+0000 7f52f58e5640 1 -- 172.21.15.45:0/1620073363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f00b1340 msgr2=0x7f52f0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.350+0000 7f52f58e5640 1 --2- 172.21.15.45:0/1620073363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f00b1340 0x7f52f0071190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009e50 tx=0x7f52e802bf60 comp rx=0 tx=0).stop 2023-11-28T20:50:47.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.350+0000 7f52f58e5640 1 -- 172.21.15.45:0/1620073363 shutdown_connections 2023-11-28T20:50:47.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.350+0000 7f52f58e5640 1 --2- 172.21.15.45:0/1620073363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f00b1340 0x7f52f0071190 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.350+0000 7f52f58e5640 1 --2- 172.21.15.45:0/1620073363 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 0x7f52f00ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.350+0000 7f52f58e5640 1 -- 172.21.15.45:0/1620073363 >> 172.21.15.45:0/1620073363 conn(0x7f52f006a500 msgr2=0x7f52f006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:47.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.351+0000 7f52f58e5640 1 -- 172.21.15.45:0/1620073363 shutdown_connections 2023-11-28T20:50:47.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.351+0000 7f52f58e5640 1 -- 172.21.15.45:0/1620073363 wait complete. 2023-11-28T20:50:47.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.352+0000 7f52f58e5640 1 Processor -- start 2023-11-28T20:50:47.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.352+0000 7f52f58e5640 1 -- start start 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.353+0000 7f52f58e5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f00add40 0x7f52f0158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.353+0000 7f52f58e5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f00b1340 0x7f52f015da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.353+0000 7f52f58e5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 0x7f52f015df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.353+0000 7f52f58e5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f015f4e0 con 0x7f52f00add40 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.353+0000 7f52f58e5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0158740 con 0x7f52f00b1340 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.353+0000 7f52f58e5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f01589c0 con 0x7f52f015d320 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f00add40 0x7f52f0158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f00add40 0x7f52f0158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52432/0 (socket says 172.21.15.45:52432) 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52eeffd640 1 -- 172.21.15.45:0/1875081375 learned_addr learned my addr 172.21.15.45:0/1875081375 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:47.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52e7fff640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f00b1340 0x7f52f015da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52ef7fe640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 0x7f52f015df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52eeffd640 1 -- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 msgr2=0x7f52f015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52eeffd640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 0x7f52f015df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52eeffd640 1 -- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f00b1340 msgr2=0x7f52f015da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52eeffd640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f00b1340 0x7f52f015da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52eeffd640 1 -- 172.21.15.45:0/1875081375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e8009b30 con 0x7f52f00add40 2023-11-28T20:50:47.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.354+0000 7f52ef7fe640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 0x7f52f015df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:47.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.355+0000 7f52eeffd640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f00add40 0x7f52f0158200 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f52e000e9b0 tx=0x7f52e000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:47.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.355+0000 7f52ecff9640 1 -- 172.21.15.45:0/1875081375 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0009670 con 0x7f52f00add40 2023-11-28T20:50:47.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.355+0000 7f52ecff9640 1 -- 172.21.15.45:0/1875081375 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e000c9a0 con 0x7f52f00add40 2023-11-28T20:50:47.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.355+0000 7f52ecff9640 1 -- 172.21.15.45:0/1875081375 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e000cd50 con 0x7f52f00add40 2023-11-28T20:50:47.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.355+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f0158ca0 con 0x7f52f00add40 2023-11-28T20:50:47.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.355+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f0159260 con 0x7f52f00add40 2023-11-28T20:50:47.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.357+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f00733d0 con 0x7f52f00add40 2023-11-28T20:50:47.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.359+0000 7f52ecff9640 1 -- 172.21.15.45:0/1875081375 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 34) v1 ==== 89334+0+0 (secure 0 0 0) 0x7f52e0013470 con 0x7f52f00add40 2023-11-28T20:50:47.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.359+0000 7f52ecff9640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f52d006af60 0x7f52d006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.359+0000 7f52ecff9640 1 -- 172.21.15.45:0/1875081375 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f52e0089ba0 con 0x7f52f00add40 2023-11-28T20:50:47.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.360+0000 7f52e7fff640 1 -- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f52d006af60 msgr2=0x7f52d006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:47.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.360+0000 7f52e7fff640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f52d006af60 0x7f52d006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:50:47.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.364+0000 7f52ecff9640 1 -- 172.21.15.45:0/1875081375 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52e00549e0 con 0x7f52f00add40 2023-11-28T20:50:47.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.561+0000 7f52e7fff640 1 -- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f52d006af60 msgr2=0x7f52d006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2590179685 2023-11-28T20:50:47.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.561+0000 7f52e7fff640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f52d006af60 0x7f52d006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:50:47.572 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.570+0000 7f1b376da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:50:47.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.576+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52f00ad250 con 0x7f52f00add40 2023-11-28T20:50:47.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.578+0000 7f52ecff9640 1 -- 172.21.15.45:0/1875081375 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+17211 (secure 0 0 0) 0x7f52e0017090 con 0x7f52f00add40 2023-11-28T20:50:47.579 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:47.580 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":89,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:41.547109+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:50:24.656511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-28T20:50:25.847745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1902490820":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1766221473":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1382859164":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6833/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1155029644":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6832/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/2238215747":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/3448409370":"2023-11-29T20:50:41.547044+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:47.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f52d006af60 msgr2=0x7f52d006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:50:47.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f52d006af60 0x7f52d006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f00add40 msgr2=0x7f52f0158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f00add40 0x7f52f0158200 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f52e000e9b0 tx=0x7f52e000ee70 comp rx=0 tx=0).stop 2023-11-28T20:50:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 shutdown_connections 2023-11-28T20:50:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:6832/2590179685,v1:172.21.15.190:6833/2590179685] conn(0x7f52d006af60 0x7f52d006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f015d320 0x7f52f015df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f00b1340 0x7f52f015da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 --2- 172.21.15.45:0/1875081375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f00add40 0x7f52f0158200 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 >> 172.21.15.45:0/1875081375 conn(0x7f52f006a500 msgr2=0x7f52f015bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:47.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 shutdown_connections 2023-11-28T20:50:47.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.581+0000 7f52f58e5640 1 -- 172.21.15.45:0/1875081375 wait complete. 2023-11-28T20:50:47.599 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:50:24.656511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:50:25.847745+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:50:47.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:50:47.608 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:47.607+0000 7f9384615040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:50:47.629 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.626+0000 7f1b376da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:50:47.744 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.741+0000 7f1b376da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:50:47.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.826+0000 7fe491056640 1 Processor -- start 2023-11-28T20:50:47.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.827+0000 7fe491056640 1 -- start start 2023-11-28T20:50:47.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.828+0000 7fe491056640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 0x7fe48c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.828+0000 7fe491056640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe48c15f720 0x7fe48c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.828+0000 7fe491056640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c15ecd0 con 0x7fe48c15e840 2023-11-28T20:50:47.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.828+0000 7fe491056640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c071ce0 con 0x7fe48c15f720 2023-11-28T20:50:47.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.828+0000 7fe491056640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c15fbd0 con 0x7fe48c06ac10 2023-11-28T20:50:47.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.828+0000 7fe48b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe48c15f720 0x7fe48c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.828+0000 7fe483fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 0x7fe48c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.828+0000 7fe48b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe48c15f720 0x7fe48c0717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:33184/0 (socket says 172.21.15.45:33184) 2023-11-28T20:50:47.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.828+0000 7fe48b59e640 1 -- 172.21.15.45:0/1995189902 learned_addr learned my addr 172.21.15.45:0/1995189902 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:47.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.829+0000 7fe48b59e640 1 -- 172.21.15.45:0/1995189902 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 msgr2=0x7fe48c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:47.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.829+0000 7fe48b59e640 1 --2- 172.21.15.45:0/1995189902 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 0x7fe48c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.829+0000 7fe48b59e640 1 -- 172.21.15.45:0/1995189902 >> v1:172.21.15.190:6789/0 conn(0x7fe48c15e840 legacy=0x7fe48c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.829+0000 7fe48b59e640 1 -- 172.21.15.45:0/1995189902 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c15f290 con 0x7fe48c15f720 2023-11-28T20:50:47.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.830+0000 7fe48b59e640 1 --2- 172.21.15.45:0/1995189902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe48c15f720 0x7fe48c0717a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe484009e80 tx=0x7fe48402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84dea410d652321a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:47.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.830+0000 7fe48a59c640 1 -- 172.21.15.45:0/1995189902 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48402caf0 con 0x7fe48c15f720 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.830+0000 7fe48a59c640 1 -- 172.21.15.45:0/1995189902 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48402cc50 con 0x7fe48c15f720 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.830+0000 7fe48a59c640 1 -- 172.21.15.45:0/1995189902 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4840343f0 con 0x7fe48c15f720 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.831+0000 7fe491056640 1 -- 172.21.15.45:0/1995189902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe48c15f720 msgr2=0x7fe48c0717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.831+0000 7fe491056640 1 --2- 172.21.15.45:0/1995189902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe48c15f720 0x7fe48c0717a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe484009e80 tx=0x7fe48402bf50 comp rx=0 tx=0).stop 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.831+0000 7fe491056640 1 -- 172.21.15.45:0/1995189902 shutdown_connections 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.831+0000 7fe491056640 1 --2- 172.21.15.45:0/1995189902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe48c15f720 0x7fe48c0717a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.831+0000 7fe491056640 1 --2- 172.21.15.45:0/1995189902 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 0x7fe48c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.831+0000 7fe491056640 1 -- 172.21.15.45:0/1995189902 >> 172.21.15.45:0/1995189902 conn(0x7fe48c0ced60 msgr2=0x7fe48c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.831+0000 7fe491056640 1 -- 172.21.15.45:0/1995189902 shutdown_connections 2023-11-28T20:50:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.831+0000 7fe491056640 1 -- 172.21.15.45:0/1995189902 wait complete. 2023-11-28T20:50:47.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.832+0000 7fe491056640 1 Processor -- start 2023-11-28T20:50:47.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.833+0000 7fe491056640 1 -- start start 2023-11-28T20:50:47.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.833+0000 7fe491056640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 0x7fe48c0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.833+0000 7fe491056640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe48c15e840 0x7fe48c1596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe491056640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe48c15f720 0x7fe48c0af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:47.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe491056640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c8d60 con 0x7fe48c15e840 2023-11-28T20:50:47.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe491056640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c159c00 con 0x7fe48c15f720 2023-11-28T20:50:47.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe491056640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c74a0 con 0x7fe48c06ac10 2023-11-28T20:50:47.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe483fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe48c15e840 0x7fe48c1596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe483fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe48c15e840 0x7fe48c1596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52446/0 (socket says 172.21.15.45:52446) 2023-11-28T20:50:47.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe483fff640 1 -- 172.21.15.45:0/445291133 learned_addr learned my addr 172.21.15.45:0/445291133 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:47.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe48ad9d640 1 --2- 172.21.15.45:0/445291133 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 0x7fe48c0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe48b59e640 1 --2- 172.21.15.45:0/445291133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe48c15f720 0x7fe48c0af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:47.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe483fff640 1 -- 172.21.15.45:0/445291133 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 msgr2=0x7fe48c0c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe483fff640 1 --2- 172.21.15.45:0/445291133 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 0x7fe48c0c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe483fff640 1 -- 172.21.15.45:0/445291133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe48c15f720 msgr2=0x7fe48c0af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:47.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe483fff640 1 --2- 172.21.15.45:0/445291133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe48c15f720 0x7fe48c0af910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:47.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.834+0000 7fe483fff640 1 -- 172.21.15.45:0/445291133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484009b30 con 0x7fe48c15e840 2023-11-28T20:50:47.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.835+0000 7fe483fff640 1 --2- 172.21.15.45:0/445291133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe48c15e840 0x7fe48c1596c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe47400e9b0 tx=0x7fe47400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:47.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.835+0000 7fe4837fe640 1 -- 172.21.15.45:0/445291133 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474009670 con 0x7fe48c15e840 2023-11-28T20:50:47.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.835+0000 7fe4837fe640 1 -- 172.21.15.45:0/445291133 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47400c9a0 con 0x7fe48c15e840 2023-11-28T20:50:47.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.835+0000 7fe4837fe640 1 -- 172.21.15.45:0/445291133 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47400cd50 con 0x7fe48c15e840 2023-11-28T20:50:47.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.835+0000 7fe491056640 1 -- 172.21.15.45:0/445291133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c0c7780 con 0x7fe48c15e840 2023-11-28T20:50:47.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.835+0000 7fe491056640 1 -- 172.21.15.45:0/445291133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c0c7d40 con 0x7fe48c15e840 2023-11-28T20:50:47.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.836+0000 7fe491056640 1 -- 172.21.15.45:0/445291133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48c0733d0 con 0x7fe48c15e840 2023-11-28T20:50:47.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.838+0000 7fe4837fe640 1 -- 172.21.15.45:0/445291133 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 35) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fe474013470 con 0x7fe48c15e840 2023-11-28T20:50:47.839 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1ab9138640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.840 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1ab9138640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.840 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1ab9138640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.840 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1ab9138640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.840 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1ab9138640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.840 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1abd140640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.840 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1abd140640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.840 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1abd140640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.840 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1abd140640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.841 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:47.836+0000 7f1abd140640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:50:47.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.839+0000 7fe4837fe640 1 -- 172.21.15.45:0/445291133 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6717+0+0 (secure 0 0 0) 0x7fe474089730 con 0x7fe48c15e840 2023-11-28T20:50:47.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:47.844+0000 7fe4837fe640 1 -- 172.21.15.45:0/445291133 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe474054800 con 0x7fe48c15e840 2023-11-28T20:50:47.941 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:47.940+0000 7f9384615040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:50:48.002 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:48.001+0000 7f9384615040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:50:48.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.035+0000 7fe491056640 1 -- 172.21.15.45:0/445291133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe48c070f80 con 0x7fe48c15e840 2023-11-28T20:50:48.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.036+0000 7fe4837fe640 1 -- 172.21.15.45:0/445291133 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe474057e20 con 0x7fe48c15e840 2023-11-28T20:50:48.037 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:50:48.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.039+0000 7fe491056640 1 -- 172.21.15.45:0/445291133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe48c15e840 msgr2=0x7fe48c1596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.039+0000 7fe491056640 1 --2- 172.21.15.45:0/445291133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe48c15e840 0x7fe48c1596c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe47400e9b0 tx=0x7fe47400ee70 comp rx=0 tx=0).stop 2023-11-28T20:50:48.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.039+0000 7fe491056640 1 -- 172.21.15.45:0/445291133 shutdown_connections 2023-11-28T20:50:48.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.039+0000 7fe491056640 1 --2- 172.21.15.45:0/445291133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe48c15f720 0x7fe48c0af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.039+0000 7fe491056640 1 --2- 172.21.15.45:0/445291133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe48c15e840 0x7fe48c1596c0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.039+0000 7fe491056640 1 --2- 172.21.15.45:0/445291133 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe48c06ac10 0x7fe48c0c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.039+0000 7fe491056640 1 -- 172.21.15.45:0/445291133 >> 172.21.15.45:0/445291133 conn(0x7fe48c0ced60 msgr2=0x7fe48c0701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:48.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.039+0000 7fe491056640 1 -- 172.21.15.45:0/445291133 shutdown_connections 2023-11-28T20:50:48.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.039+0000 7fe491056640 1 -- 172.21.15.45:0/445291133 wait complete. 2023-11-28T20:50:48.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:50:48.123 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:50:48.123+0000 7f9384615040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:50:48.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.326+0000 7f4c63fff640 1 Processor -- start 2023-11-28T20:50:48.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.326+0000 7f4c63fff640 1 -- start start 2023-11-28T20:50:48.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.327+0000 7f4c63fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c64066d90 0x7f4c6406b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.328+0000 7f4c63fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4c6406ba50 0x7f4c6415b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.328+0000 7f4c63fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c641409c0 con 0x7f4c6413e170 2023-11-28T20:50:48.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.328+0000 7f4c63fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6406d7f0 con 0x7f4c6406ba50 2023-11-28T20:50:48.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.328+0000 7f4c63fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6406bec0 con 0x7f4c64066d90 2023-11-28T20:50:48.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.328+0000 7f4c62ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4c6413e170 0x7f4c6406d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45030/0 (socket says 172.21.15.45:45030) 2023-11-28T20:50:48.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.328+0000 7f4c62ffd640 1 -- 172.21.15.45:0/4190644830 learned_addr learned my addr 172.21.15.45:0/4190644830 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:48.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.328+0000 7f4c627fc640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c64066d90 0x7f4c6406b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.328+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416417043 0 0) 0x7f4c641409c0 con 0x7f4c6413e170 2023-11-28T20:50:48.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.328+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c50003520 con 0x7f4c6413e170 2023-11-28T20:50:48.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c637fe640 1 --2- 172.21.15.45:0/4190644830 >> v2:172.21.15.190:3300/0 conn(0x7f4c6406ba50 0x7f4c6415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137833602 0 0) 0x7f4c50003520 con 0x7f4c6413e170 2023-11-28T20:50:48.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c64066d90 msgr2=0x7f4c6406b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c64066d90 0x7f4c6406b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 >> v2:172.21.15.190:3300/0 conn(0x7f4c6406ba50 msgr2=0x7f4c6415b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 --2- 172.21.15.45:0/4190644830 >> v2:172.21.15.190:3300/0 conn(0x7f4c6406ba50 0x7f4c6415b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c64140ca0 con 0x7f4c6413e170 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4c540019f0 con 0x7f4c6413e170 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 >> v1:172.21.15.45:6789/0 conn(0x7f4c6413e170 legacy=0x7f4c6406d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c50003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500047c0 0x7f4c50004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c50005140 0x7f4c5000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6406bec0 con 0x7f4c50003970 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5000bec0 con 0x7f4c50005140 2023-11-28T20:50:48.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5000c000 con 0x7f4c500047c0 2023-11-28T20:50:48.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.329+0000 7f4c62ffd640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c50003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.330+0000 7f4c62ffd640 1 -- 172.21.15.45:0/4190644830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500047c0 msgr2=0x7f4c50004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:48.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.330+0000 7f4c62ffd640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500047c0 0x7f4c50004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.330+0000 7f4c62ffd640 1 -- 172.21.15.45:0/4190644830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c50005140 msgr2=0x7f4c5000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:48.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.330+0000 7f4c62ffd640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c50005140 0x7f4c5000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.330+0000 7f4c62ffd640 1 -- 172.21.15.45:0/4190644830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50003520 con 0x7f4c50003970 2023-11-28T20:50:48.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.331+0000 7f4c62ffd640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c50003d90 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400a780 tx=0x7f4c500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfde5e5b8099750 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:48.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.331+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54034630 con 0x7f4c50003970 2023-11-28T20:50:48.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.331+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5402f070 con 0x7f4c50003970 2023-11-28T20:50:48.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.331+0000 7f4c61ffb640 1 -- 172.21.15.45:0/4190644830 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c540349d0 con 0x7f4c50003970 2023-11-28T20:50:48.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.332+0000 7f4c63fff640 1 -- 172.21.15.45:0/4190644830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 msgr2=0x7f4c50003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.332+0000 7f4c63fff640 1 --2- 172.21.15.45:0/4190644830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c50003d90 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400a780 tx=0x7f4c500057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.332+0000 7f4c637fe640 1 -- 172.21.15.45:0/4190644830 reap_dead start 2023-11-28T20:50:48.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.332+0000 7f4c63fff640 1 -- 172.21.15.45:0/4190644830 shutdown_connections 2023-11-28T20:50:48.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.332+0000 7f4c63fff640 1 -- 172.21.15.45:0/4190644830 >> 172.21.15.45:0/4190644830 conn(0x7f4c64071240 msgr2=0x7f4c64071650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:48.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.332+0000 7f4c63fff640 1 -- 172.21.15.45:0/4190644830 shutdown_connections 2023-11-28T20:50:48.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.332+0000 7f4c63fff640 1 -- 172.21.15.45:0/4190644830 wait complete. 2023-11-28T20:50:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.333+0000 7f4c63fff640 1 Processor -- start 2023-11-28T20:50:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.334+0000 7f4c63fff640 1 -- start start 2023-11-28T20:50:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.334+0000 7f4c63fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c64151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c63fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641520e0 0x7f4c6415de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c63fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c64152540 0x7f4c6413e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c63fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c64156940 con 0x7f4c50003970 2023-11-28T20:50:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c63fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6415d220 con 0x7f4c641520e0 2023-11-28T20:50:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c63fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c64155d30 con 0x7f4c64152540 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c62ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c64151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c62ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c64151900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52472/0 (socket says 172.21.15.45:52472) 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c62ffd640 1 -- 172.21.15.45:0/3624658181 learned_addr learned my addr 172.21.15.45:0/3624658181 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c627fc640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641520e0 0x7f4c6415de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c637fe640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c64152540 0x7f4c6413e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c62ffd640 1 -- 172.21.15.45:0/3624658181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c64152540 msgr2=0x7f4c6413e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c62ffd640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c64152540 0x7f4c6413e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c62ffd640 1 -- 172.21.15.45:0/3624658181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641520e0 msgr2=0x7f4c6415de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c62ffd640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641520e0 0x7f4c6415de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c62ffd640 1 -- 172.21.15.45:0/3624658181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c64156010 con 0x7f4c50003970 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.335+0000 7f4c627fc640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641520e0 0x7f4c6415de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.336+0000 7f4c62ffd640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c64151900 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4c5402cad0 tx=0x7f4c54001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.336+0000 7f4c5b7fe640 1 -- 172.21.15.45:0/3624658181 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54004d70 con 0x7f4c50003970 2023-11-28T20:50:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.336+0000 7f4c5b7fe640 1 -- 172.21.15.45:0/3624658181 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c540354c0 con 0x7f4c50003970 2023-11-28T20:50:48.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.336+0000 7f4c5b7fe640 1 -- 172.21.15.45:0/3624658181 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54035870 con 0x7f4c50003970 2023-11-28T20:50:48.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.336+0000 7f4c63fff640 1 -- 172.21.15.45:0/3624658181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c641562f0 con 0x7f4c50003970 2023-11-28T20:50:48.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.336+0000 7f4c63fff640 1 -- 172.21.15.45:0/3624658181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c64151e40 con 0x7f4c50003970 2023-11-28T20:50:48.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.337+0000 7f4c63fff640 1 -- 172.21.15.45:0/3624658181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c64159160 con 0x7f4c50003970 2023-11-28T20:50:48.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.338+0000 7f4c5b7fe640 1 -- 172.21.15.45:0/3624658181 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 35) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f4c54031030 con 0x7f4c50003970 2023-11-28T20:50:48.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.339+0000 7f4c5b7fe640 1 -- 172.21.15.45:0/3624658181 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f4c540abaf0 con 0x7f4c50003970 2023-11-28T20:50:48.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.343+0000 7f4c5b7fe640 1 -- 172.21.15.45:0/3624658181 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c54076c40 con 0x7f4c50003970 2023-11-28T20:50:48.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.540+0000 7f4c63fff640 1 -- 172.21.15.45:0/3624658181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4c6409e100 con 0x7f4c50003970 2023-11-28T20:50:48.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.541+0000 7f4c5b7fe640 1 -- 172.21.15.45:0/3624658181 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4c5407a260 con 0x7f4c50003970 2023-11-28T20:50:48.543 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:50:48.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.544+0000 7f4c63fff640 1 -- 172.21.15.45:0/3624658181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 msgr2=0x7f4c64151900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.544+0000 7f4c63fff640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c64151900 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4c5402cad0 tx=0x7f4c54001360 comp rx=0 tx=0).stop 2023-11-28T20:50:48.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.544+0000 7f4c63fff640 1 -- 172.21.15.45:0/3624658181 shutdown_connections 2023-11-28T20:50:48.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.544+0000 7f4c63fff640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c64152540 0x7f4c6413e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.544+0000 7f4c63fff640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641520e0 0x7f4c6415de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.544+0000 7f4c63fff640 1 --2- 172.21.15.45:0/3624658181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c50003970 0x7f4c64151900 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.544+0000 7f4c63fff640 1 -- 172.21.15.45:0/3624658181 >> 172.21.15.45:0/3624658181 conn(0x7f4c64071240 msgr2=0x7f4c6406f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:48.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.544+0000 7f4c63fff640 1 -- 172.21.15.45:0/3624658181 shutdown_connections 2023-11-28T20:50:48.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.544+0000 7f4c63fff640 1 -- 172.21.15.45:0/3624658181 wait complete. 2023-11-28T20:50:48.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:50:48.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.837+0000 7f39e50c3640 1 Processor -- start 2023-11-28T20:50:48.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.837+0000 7f39e50c3640 1 -- start start 2023-11-28T20:50:48.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.839+0000 7f39e50c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 0x7f39e006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.839+0000 7f39e50c3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f39e015b270 0x7f39e015f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.839+0000 7f39e50c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e015faf0 con 0x7f39e00b3420 2023-11-28T20:50:48.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.839+0000 7f39e50c3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e00ceaa0 con 0x7f39e015b270 2023-11-28T20:50:48.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.839+0000 7f39e50c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e006d410 con 0x7f39e00b3040 2023-11-28T20:50:48.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.839+0000 7f39df7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f39e015b270 0x7f39e015f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.839+0000 7f39e48c2640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f39e00b3420 0x7f39e00b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45032/0 (socket says 172.21.15.45:45032) 2023-11-28T20:50:48.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.839+0000 7f39e48c2640 1 -- 172.21.15.45:0/1926918287 learned_addr learned my addr 172.21.15.45:0/1926918287 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:48.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.840+0000 7f39dffff640 1 --2- 172.21.15.45:0/1926918287 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 0x7f39e006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.840+0000 7f39df7fe640 1 -- 172.21.15.45:0/1926918287 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 msgr2=0x7f39e006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:48.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.840+0000 7f39df7fe640 1 --2- 172.21.15.45:0/1926918287 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 0x7f39e006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.840+0000 7f39df7fe640 1 -- 172.21.15.45:0/1926918287 >> v1:172.21.15.45:6789/0 conn(0x7f39e00b3420 legacy=0x7f39e00b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.840+0000 7f39df7fe640 1 -- 172.21.15.45:0/1926918287 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e006d6f0 con 0x7f39e015b270 2023-11-28T20:50:48.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.840+0000 7f39deffd640 1 -- 172.21.15.45:0/1926918287 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203786816 0 0) 0x7f39e015faf0 con 0x7f39e00b3420 2023-11-28T20:50:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.841+0000 7f39df7fe640 1 --2- 172.21.15.45:0/1926918287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 0x7f39e015f5b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f39d40097a0 tx=0x7f39d402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c53b49405ad05e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.841+0000 7f39deffd640 1 -- 172.21.15.45:0/1926918287 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d402f050 con 0x7f39e015b270 2023-11-28T20:50:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.841+0000 7f39deffd640 1 -- 172.21.15.45:0/1926918287 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d4031700 con 0x7f39e015b270 2023-11-28T20:50:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.841+0000 7f39deffd640 1 -- 172.21.15.45:0/1926918287 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d4031a90 con 0x7f39e015b270 2023-11-28T20:50:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.842+0000 7f39e50c3640 1 -- 172.21.15.45:0/1926918287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 msgr2=0x7f39e015f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.842+0000 7f39e50c3640 1 --2- 172.21.15.45:0/1926918287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 0x7f39e015f5b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f39d40097a0 tx=0x7f39d402b7a0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.842+0000 7f39e50c3640 1 -- 172.21.15.45:0/1926918287 shutdown_connections 2023-11-28T20:50:48.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.842+0000 7f39e50c3640 1 --2- 172.21.15.45:0/1926918287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 0x7f39e015f5b0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.842+0000 7f39e50c3640 1 --2- 172.21.15.45:0/1926918287 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 0x7f39e006cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.842+0000 7f39e50c3640 1 -- 172.21.15.45:0/1926918287 >> 172.21.15.45:0/1926918287 conn(0x7f39e00b1050 msgr2=0x7f39e00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:48.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.842+0000 7f39e50c3640 1 -- 172.21.15.45:0/1926918287 shutdown_connections 2023-11-28T20:50:48.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.842+0000 7f39e50c3640 1 -- 172.21.15.45:0/1926918287 wait complete. 2023-11-28T20:50:48.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.844+0000 7f39e50c3640 1 Processor -- start 2023-11-28T20:50:48.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.844+0000 7f39e50c3640 1 -- start start 2023-11-28T20:50:48.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.845+0000 7f39e50c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 0x7f39e0145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.845+0000 7f39e50c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39e00b3420 0x7f39e0071be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.845+0000 7f39e50c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 0x7f39e015b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.845+0000 7f39e50c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0159610 con 0x7f39e015b270 2023-11-28T20:50:48.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.845+0000 7f39e50c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0158a00 con 0x7f39e00b3420 2023-11-28T20:50:48.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.845+0000 7f39e50c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0158ce0 con 0x7f39e00b3040 2023-11-28T20:50:48.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.845+0000 7f39e48c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 0x7f39e015b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.845+0000 7f39e48c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 0x7f39e015b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52494/0 (socket says 172.21.15.45:52494) 2023-11-28T20:50:48.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.845+0000 7f39e48c2640 1 -- 172.21.15.45:0/3395012450 learned_addr learned my addr 172.21.15.45:0/3395012450 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:48.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39dffff640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 0x7f39e0145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39e48c2640 1 -- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 msgr2=0x7f39e0145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39e48c2640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 0x7f39e0145830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39e48c2640 1 -- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39e00b3420 msgr2=0x7f39e0071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:48.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39e48c2640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39e00b3420 0x7f39e0071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:48.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39e48c2640 1 -- 172.21.15.45:0/3395012450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d4009400 con 0x7f39e015b270 2023-11-28T20:50:48.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39e48c2640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 0x7f39e015b670 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f39d800c7e0 tx=0x7f39d800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:48.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39dcff9640 1 -- 172.21.15.45:0/3395012450 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d800a640 con 0x7f39e015b270 2023-11-28T20:50:48.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39dcff9640 1 -- 172.21.15.45:0/3395012450 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d800a7c0 con 0x7f39e015b270 2023-11-28T20:50:48.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39dcff9640 1 -- 172.21.15.45:0/3395012450 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d800ab70 con 0x7f39e015b270 2023-11-28T20:50:48.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.846+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e0158fc0 con 0x7f39e015b270 2023-11-28T20:50:48.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.847+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e0072120 con 0x7f39e015b270 2023-11-28T20:50:48.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.848+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e0159780 con 0x7f39e015b270 2023-11-28T20:50:48.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.849+0000 7f39dcff9640 1 -- 172.21.15.45:0/3395012450 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 36) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f39d800acd0 con 0x7f39e015b270 2023-11-28T20:50:48.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.850+0000 7f39dcff9640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f39cc06ac90 0x7f39cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:48.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.850+0000 7f39dcff9640 1 -- 172.21.15.45:0/3395012450 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f39d8088d20 con 0x7f39e015b270 2023-11-28T20:50:48.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.850+0000 7f39dffff640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f39cc06ac90 0x7f39cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:48.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.851+0000 7f39dffff640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f39cc06ac90 0x7f39cc06d150 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f39d0006fd0 tx=0x7f39d0008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:48.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:48.854+0000 7f39dcff9640 1 -- 172.21.15.45:0/3395012450 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f39d8053c50 con 0x7f39e015b270 2023-11-28T20:50:49.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.043+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39e004e920 con 0x7f39e015b270 2023-11-28T20:50:49.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.044+0000 7f39dcff9640 1 -- 172.21.15.45:0/3395012450 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f39d8018070 con 0x7f39e015b270 2023-11-28T20:50:49.045 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:50:49.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.046+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f39cc06ac90 msgr2=0x7f39cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.046+0000 7f39e50c3640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f39cc06ac90 0x7f39cc06d150 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f39d0006fd0 tx=0x7f39d0008040 comp rx=0 tx=0).stop 2023-11-28T20:50:49.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.046+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 msgr2=0x7f39e015b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.046+0000 7f39e50c3640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 0x7f39e015b670 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f39d800c7e0 tx=0x7f39d800cca0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.047+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 shutdown_connections 2023-11-28T20:50:49.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.047+0000 7f39e50c3640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f39cc06ac90 0x7f39cc06d150 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.047+0000 7f39e50c3640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39e015b270 0x7f39e015b670 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.047+0000 7f39e50c3640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39e00b3420 0x7f39e0071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.047+0000 7f39e50c3640 1 --2- 172.21.15.45:0/3395012450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39e00b3040 0x7f39e0145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.047+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 >> 172.21.15.45:0/3395012450 conn(0x7f39e00b1050 msgr2=0x7f39e00aea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.047+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 shutdown_connections 2023-11-28T20:50:49.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.047+0000 7f39e50c3640 1 -- 172.21.15.45:0/3395012450 wait complete. 2023-11-28T20:50:49.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.284+0000 7f97ba997640 1 Processor -- start 2023-11-28T20:50:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.284+0000 7f97ba997640 1 -- start start 2023-11-28T20:50:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.285+0000 7f97ba997640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f97b4161960 0x7f97b40b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.285+0000 7f97ba997640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b40b4c40 0x7f97b4070330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.285+0000 7f97ba997640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b40711c0 con 0x7f97b40b0c30 2023-11-28T20:50:49.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.285+0000 7f97ba997640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b4071e40 con 0x7f97b4161960 2023-11-28T20:50:49.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.285+0000 7f97ba997640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40714a0 con 0x7f97b40b4c40 2023-11-28T20:50:49.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.286+0000 7f97b8f34640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f97b40b0c30 0x7f97b4165660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45038/0 (socket says 172.21.15.45:45038) 2023-11-28T20:50:49.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.286+0000 7f97b8f34640 1 -- 172.21.15.45:0/3642607996 learned_addr learned my addr 172.21.15.45:0/3642607996 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.286+0000 7f97b3fff640 1 --2- 172.21.15.45:0/3642607996 >> v2:172.21.15.190:3300/0 conn(0x7f97b4161960 0x7f97b40b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.286+0000 7f97abfff640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b40b4c40 0x7f97b4070330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.286+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393134050 0 0) 0x7f97b40711c0 con 0x7f97b40b0c30 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.286+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a0003520 con 0x7f97b40b0c30 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1810810079 0 0) 0x7f97a0003520 con 0x7f97b40b0c30 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b40b4c40 msgr2=0x7f97b4070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b40b4c40 0x7f97b4070330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 >> v2:172.21.15.190:3300/0 conn(0x7f97b4161960 msgr2=0x7f97b40b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 --2- 172.21.15.45:0/3642607996 >> v2:172.21.15.190:3300/0 conn(0x7f97b4161960 0x7f97b40b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b4071780 con 0x7f97b40b0c30 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f97ac0019f0 con 0x7f97b40b0c30 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 >> v1:172.21.15.45:6789/0 conn(0x7f97b40b0c30 legacy=0x7f97b4165660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b40b4c40 0x7f97a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a0004d00 0x7f97a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b4071e40 con 0x7f97a0003970 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40714a0 con 0x7f97a0004d00 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a000b980 con 0x7f97b40b4c40 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.287+0000 7f97b3fff640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.288+0000 7f97abfff640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b40b4c40 0x7f97a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.288+0000 7f97b3fff640 1 -- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b40b4c40 msgr2=0x7f97a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.288+0000 7f97b3fff640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b40b4c40 0x7f97a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.288+0000 7f97b3fff640 1 -- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a0004d00 msgr2=0x7f97a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.288+0000 7f97b8f34640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a0004d00 0x7f97a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.288+0000 7f97b3fff640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a0004d00 0x7f97a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.288+0000 7f97b3fff640 1 -- 172.21.15.45:0/3642607996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a0003520 con 0x7f97a0003970 2023-11-28T20:50:49.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.288+0000 7f97b8f34640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a0004d00 0x7f97a0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.288+0000 7f97abfff640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b40b4c40 0x7f97a00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.289+0000 7f97b3fff640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97a0003d90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009ac0 tx=0x7f97a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8beb59e16108bac0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.289+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a402c710 con 0x7f97a0003970 2023-11-28T20:50:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.289+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a402c870 con 0x7f97a0003970 2023-11-28T20:50:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.289+0000 7f97b37fe640 1 -- 172.21.15.45:0/3642607996 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a402cc00 con 0x7f97a0003970 2023-11-28T20:50:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.290+0000 7f97ba997640 1 -- 172.21.15.45:0/3642607996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 msgr2=0x7f97a0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.290+0000 7f97ba997640 1 --2- 172.21.15.45:0/3642607996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97a0003d90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009ac0 tx=0x7f97a00057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.290+0000 7f97b8f34640 1 -- 172.21.15.45:0/3642607996 reap_dead start 2023-11-28T20:50:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.290+0000 7f97ba997640 1 -- 172.21.15.45:0/3642607996 shutdown_connections 2023-11-28T20:50:49.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.290+0000 7f97ba997640 1 -- 172.21.15.45:0/3642607996 >> 172.21.15.45:0/3642607996 conn(0x7f97b406a500 msgr2=0x7f97b406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:49.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.290+0000 7f97ba997640 1 -- 172.21.15.45:0/3642607996 shutdown_connections 2023-11-28T20:50:49.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.290+0000 7f97ba997640 1 -- 172.21.15.45:0/3642607996 wait complete. 2023-11-28T20:50:49.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.292+0000 7f97ba997640 1 Processor -- start 2023-11-28T20:50:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.292+0000 7f97ba997640 1 -- start start 2023-11-28T20:50:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.293+0000 7f97ba997640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97b4155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97ba997640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b4164740 0x7f97b41f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97ba997640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40cb940 0x7f97b40cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97ba997640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40cc300 con 0x7f97a0003970 2023-11-28T20:50:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97ba997640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b41f7e50 con 0x7f97b40cb940 2023-11-28T20:50:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97ba997640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b41f8130 con 0x7f97b4164740 2023-11-28T20:50:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97b4155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97b4155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52504/0 (socket says 172.21.15.45:52504) 2023-11-28T20:50:49.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b3fff640 1 -- 172.21.15.45:0/1346693480 learned_addr learned my addr 172.21.15.45:0/1346693480 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:49.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b8f34640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40cb940 0x7f97b40cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97abfff640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b4164740 0x7f97b41f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b3fff640 1 -- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b4164740 msgr2=0x7f97b41f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b3fff640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b4164740 0x7f97b41f7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b3fff640 1 -- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40cb940 msgr2=0x7f97b40cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b3fff640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40cb940 0x7f97b40cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b3fff640 1 -- 172.21.15.45:0/1346693480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b41f8410 con 0x7f97a0003970 2023-11-28T20:50:49.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.294+0000 7f97b8f34640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40cb940 0x7f97b40cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:49.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.295+0000 7f97b3fff640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97b4155350 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f97a402c2b0 tx=0x7f97a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:49.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.295+0000 7f97b17fa640 1 -- 172.21.15.45:0/1346693480 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4003810 con 0x7f97a0003970 2023-11-28T20:50:49.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.295+0000 7f97b17fa640 1 -- 172.21.15.45:0/1346693480 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a402de90 con 0x7f97a0003970 2023-11-28T20:50:49.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.295+0000 7f97b17fa640 1 -- 172.21.15.45:0/1346693480 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4003b70 con 0x7f97a0003970 2023-11-28T20:50:49.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.295+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b415ec80 con 0x7f97a0003970 2023-11-28T20:50:49.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.295+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b41f7a80 con 0x7f97a0003970 2023-11-28T20:50:49.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.297+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b40cecf0 con 0x7f97a0003970 2023-11-28T20:50:49.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.297+0000 7f97b17fa640 1 -- 172.21.15.45:0/1346693480 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 36) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f97a4003cd0 con 0x7f97a0003970 2023-11-28T20:50:49.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.299+0000 7f97b17fa640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f979406acc0 0x7f979406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.299+0000 7f97b17fa640 1 -- 172.21.15.45:0/1346693480 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f97a40ab550 con 0x7f97a0003970 2023-11-28T20:50:49.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.299+0000 7f97abfff640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f979406acc0 0x7f979406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.300+0000 7f97abfff640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f979406acc0 0x7f979406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f97b41646d0 tx=0x7f979c008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:49.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.303+0000 7f97b17fa640 1 -- 172.21.15.45:0/1346693480 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f97a4076480 con 0x7f97a0003970 2023-11-28T20:50:49.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.537+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97b415eac0 con 0x7f97a0003970 2023-11-28T20:50:49.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.539+0000 7f97b17fa640 1 -- 172.21.15.45:0/1346693480 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4914 (secure 0 0 0) 0x7f97a4079aa0 con 0x7f97a0003970 2023-11-28T20:50:49.540 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:49.540 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5438,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2396470524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2396470524},{"type":"v1","addr":"172.21.15.45:6835","nonce":2396470524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/436103276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":436103276},{"type":"v1","addr":"172.21.15.190:6836","nonce":436103276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/834328551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":834328551},{"type":"v1","addr":"172.21.15.45:6834","nonce":834328551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:50:26.812959+0000","modified":"2023-11-28T20:50:29.910521+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6837/3032907708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3032907708},{"type":"v1","addr":"172.21.15.190:6837","nonce":3032907708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T20:50:49.540 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 36 2023-11-28T20:50:49.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f979406acc0 msgr2=0x7f979406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f979406acc0 0x7f979406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f97b41646d0 tx=0x7f979c008040 comp rx=0 tx=0).stop 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 msgr2=0x7f97b4155350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97b4155350 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f97a402c2b0 tx=0x7f97a4004730 comp rx=0 tx=0).stop 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 shutdown_connections 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f979406acc0 0x7f979406d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40cb940 0x7f97b40cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97b4164740 0x7f97b41f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 --2- 172.21.15.45:0/1346693480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97a0003970 0x7f97b4155350 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 >> 172.21.15.45:0/1346693480 conn(0x7f97b406a500 msgr2=0x7f97b40cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.541+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 shutdown_connections 2023-11-28T20:50:49.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.542+0000 7f97ba997640 1 -- 172.21.15.45:0/1346693480 wait complete. 2023-11-28T20:50:49.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:49.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:50:49.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:50:49.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.778+0000 7f3bd089a640 1 Processor -- start 2023-11-28T20:50:49.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.779+0000 7f3bd089a640 1 -- start start 2023-11-28T20:50:49.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.780+0000 7f3bd089a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3bcc0cf010 0x7f3bcc0c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.780+0000 7f3bd089a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc0b3610 0x7f3bcc0c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.780+0000 7f3bd089a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bcc0c4ac0 con 0x7f3bcc099a40 2023-11-28T20:50:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.780+0000 7f3bd089a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bcc0c5080 con 0x7f3bcc0cf010 2023-11-28T20:50:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.780+0000 7f3bd089a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc09a610 con 0x7f3bcc0b3610 2023-11-28T20:50:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bcad9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3bcc099a40 0x7f3bcc099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45050/0 (socket says 172.21.15.45:45050) 2023-11-28T20:50:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bcad9d640 1 -- 172.21.15.45:0/2993532646 learned_addr learned my addr 172.21.15.45:0/2993532646 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bcb59e640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc0b3610 0x7f3bcc0c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bca59c640 1 --2- 172.21.15.45:0/2993532646 >> v2:172.21.15.190:3300/0 conn(0x7f3bcc0cf010 0x7f3bcc0c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015875080 0 0) 0x7f3bcc0c4ac0 con 0x7f3bcc099a40 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb4003520 con 0x7f3bcc099a40 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2728255969 0 0) 0x7f3bb4003520 con 0x7f3bcc099a40 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc0b3610 msgr2=0x7f3bcc0c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bc9d9b640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc0b3610 0x7f3bcc0c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 >> v2:172.21.15.190:3300/0 conn(0x7f3bcc0cf010 msgr2=0x7f3bcc0c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.781+0000 7f3bc9d9b640 1 --2- 172.21.15.45:0/2993532646 >> v2:172.21.15.190:3300/0 conn(0x7f3bcc0cf010 0x7f3bcc0c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc09a8f0 con 0x7f3bcc099a40 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3bbc001670 con 0x7f3bcc099a40 2023-11-28T20:50:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 >> v1:172.21.15.45:6789/0 conn(0x7f3bcc099a40 legacy=0x7f3bcc099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bc9d9b640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb4003970 0x7f3bb4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bc9d9b640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bb40047c0 0x7f3bb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bc9d9b640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bb400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0c5080 con 0x7f3bb4005180 2023-11-28T20:50:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc09a610 con 0x7f3bb4003970 2023-11-28T20:50:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400bec0 con 0x7f3bb40047c0 2023-11-28T20:50:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bcb59e640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc0b3610 0x7f3bcc0c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:50:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bcb59e640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bb400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bca59c640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bb40047c0 0x7f3bb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bcad9d640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb4003970 0x7f3bb4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bcb59e640 1 -- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bb40047c0 msgr2=0x7f3bb4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.782+0000 7f3bcb59e640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bb40047c0 0x7f3bb4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.783+0000 7f3bcb59e640 1 -- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb4003970 msgr2=0x7f3bb4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.783+0000 7f3bcb59e640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb4003970 0x7f3bb4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.783+0000 7f3bcb59e640 1 -- 172.21.15.45:0/2993532646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb4003520 con 0x7f3bb4005180 2023-11-28T20:50:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.783+0000 7f3bcad9d640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb4003970 0x7f3bb4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:49.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.784+0000 7f3bcb59e640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bb400b980 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400e220 tx=0x7f3bb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd067358f0e7b867 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:49.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.784+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc402cce0 con 0x7f3bb4005180 2023-11-28T20:50:49.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.784+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc400f040 con 0x7f3bb4005180 2023-11-28T20:50:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.784+0000 7f3bc9d9b640 1 -- 172.21.15.45:0/2993532646 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4003bb0 con 0x7f3bb4005180 2023-11-28T20:50:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.784+0000 7f3bd089a640 1 -- 172.21.15.45:0/2993532646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 msgr2=0x7f3bb400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.784+0000 7f3bd089a640 1 --2- 172.21.15.45:0/2993532646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bb400b980 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400e220 tx=0x7f3bb40057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.784+0000 7f3bcb59e640 1 -- 172.21.15.45:0/2993532646 reap_dead start 2023-11-28T20:50:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.785+0000 7f3bd089a640 1 -- 172.21.15.45:0/2993532646 shutdown_connections 2023-11-28T20:50:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.785+0000 7f3bd089a640 1 -- 172.21.15.45:0/2993532646 >> 172.21.15.45:0/2993532646 conn(0x7f3bcc0b2b20 msgr2=0x7f3bcc0b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.785+0000 7f3bd089a640 1 -- 172.21.15.45:0/2993532646 shutdown_connections 2023-11-28T20:50:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.785+0000 7f3bd089a640 1 -- 172.21.15.45:0/2993532646 wait complete. 2023-11-28T20:50:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.787+0000 7f3bd089a640 1 Processor -- start 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.787+0000 7f3bd089a640 1 -- start start 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bd089a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bcc1e7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bd089a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bcc153320 0x7f3bcc1e7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bd089a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc140490 0x7f3bcc140910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bd089a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc141410 con 0x7f3bb4005180 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bd089a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc1416f0 con 0x7f3bcc153320 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bd089a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc1419d0 con 0x7f3bcc140490 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bcad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bcc1e7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bcad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bcc1e7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52526/0 (socket says 172.21.15.45:52526) 2023-11-28T20:50:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bcad9d640 1 -- 172.21.15.45:0/3038510002 learned_addr learned my addr 172.21.15.45:0/3038510002 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bcb59e640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc140490 0x7f3bcc140910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bca59c640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bcc153320 0x7f3bcc1e7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bcad9d640 1 -- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc140490 msgr2=0x7f3bcc140910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.788+0000 7f3bcad9d640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc140490 0x7f3bcc140910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.789+0000 7f3bcad9d640 1 -- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bcc153320 msgr2=0x7f3bcc1e7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.789+0000 7f3bcad9d640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bcc153320 0x7f3bcc1e7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.789+0000 7f3bcad9d640 1 -- 172.21.15.45:0/3038510002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc141cb0 con 0x7f3bb4005180 2023-11-28T20:50:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.789+0000 7f3bcad9d640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bcc1e7710 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc000ff0 tx=0x7f3bbc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.789+0000 7f3bb37fe640 1 -- 172.21.15.45:0/3038510002 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc00c870 con 0x7f3bb4005180 2023-11-28T20:50:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.789+0000 7f3bb37fe640 1 -- 172.21.15.45:0/3038510002 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bbc00c9d0 con 0x7f3bb4005180 2023-11-28T20:50:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.789+0000 7f3bb37fe640 1 -- 172.21.15.45:0/3038510002 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc00cd40 con 0x7f3bb4005180 2023-11-28T20:50:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.789+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bcc141f90 con 0x7f3bb4005180 2023-11-28T20:50:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.789+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bcc141030 con 0x7f3bb4005180 2023-11-28T20:50:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.791+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bcc04e920 con 0x7f3bb4005180 2023-11-28T20:50:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.791+0000 7f3bb37fe640 1 -- 172.21.15.45:0/3038510002 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 36) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f3bbc01d460 con 0x7f3bb4005180 2023-11-28T20:50:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.792+0000 7f3bb37fe640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3bb806ac90 0x7f3bb806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.793+0000 7f3bb37fe640 1 -- 172.21.15.45:0/3038510002 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f3bbc089690 con 0x7f3bb4005180 2023-11-28T20:50:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.793+0000 7f3bca59c640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3bb806ac90 0x7f3bb806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.794+0000 7f3bca59c640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3bb806ac90 0x7f3bb806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0cb520 tx=0x7f3bc0008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:49.797+0000 7f3bb37fe640 1 -- 172.21.15.45:0/3038510002 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bbc0545c0 con 0x7f3bb4005180 2023-11-28T20:50:49.819 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:49.816+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:50:50.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.000+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bcc0cd760 con 0x7f3bb4005180 2023-11-28T20:50:50.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.002+0000 7f3bb37fe640 1 -- 172.21.15.45:0/3038510002 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+17719 (secure 0 0 0) 0x7f3bbc057be0 con 0x7f3bb4005180 2023-11-28T20:50:50.004 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:50.004 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":90,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:47.747533+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:50:24.656511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-28T20:50:25.847745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6833/2590179685":"2023-11-29T20:50:47.747393+0000","172.21.15.190:6832/2590179685":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/3115528802":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/2734256196":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/559775284":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/709518420":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/1852415784":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/1902490820":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1766221473":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/2227588570":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/1382859164":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6833/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1155029644":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6832/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/2238215747":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/3448409370":"2023-11-29T20:50:41.547044+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:50.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.005+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3bb806ac90 msgr2=0x7f3bb806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.005+0000 7f3bd089a640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3bb806ac90 0x7f3bb806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0cb520 tx=0x7f3bc0008040 comp rx=0 tx=0).stop 2023-11-28T20:50:50.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 msgr2=0x7f3bcc1e7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bcc1e7710 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc000ff0 tx=0x7f3bbc00ec80 comp rx=0 tx=0).stop 2023-11-28T20:50:50.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 shutdown_connections 2023-11-28T20:50:50.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3bb806ac90 0x7f3bb806d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3bcc140490 0x7f3bcc140910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bcc153320 0x7f3bcc1e7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 --2- 172.21.15.45:0/3038510002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4005180 0x7f3bcc1e7710 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 >> 172.21.15.45:0/3038510002 conn(0x7f3bcc0b2b20 msgr2=0x7f3bcc0cf550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:50.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 shutdown_connections 2023-11-28T20:50:50.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.006+0000 7f3bd089a640 1 -- 172.21.15.45:0/3038510002 wait complete. 2023-11-28T20:50:50.026 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T20:50:50.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:50.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.258+0000 7efcd9e87640 1 Processor -- start 2023-11-28T20:50:50.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.259+0000 7efcd9e87640 1 -- start start 2023-11-28T20:50:50.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.260+0000 7efcd9e87640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd40b35b0 0x7efcd415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.260+0000 7efcd9e87640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efcd40b1340 0x7efcd415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.260+0000 7efcd9e87640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd415bda0 con 0x7efcd415b850 2023-11-28T20:50:50.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.260+0000 7efcd9e87640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd415ee40 con 0x7efcd40b1340 2023-11-28T20:50:50.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.260+0000 7efcd9e87640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd415c080 con 0x7efcd40b35b0 2023-11-28T20:50:50.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.261+0000 7efccbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efcd415b850 0x7efcd40b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45062/0 (socket says 172.21.15.45:45062) 2023-11-28T20:50:50.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.261+0000 7efccbfff640 1 -- 172.21.15.45:0/54754701 learned_addr learned my addr 172.21.15.45:0/54754701 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:50.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.261+0000 7efcd3fff640 1 --2- 172.21.15.45:0/54754701 >> v2:172.21.15.190:3300/0 conn(0x7efcd40b1340 0x7efcd415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.261+0000 7efcd37fe640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd40b35b0 0x7efcd415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.261+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748635257 0 0) 0x7efcd415bda0 con 0x7efcd415b850 2023-11-28T20:50:50.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc0003520 con 0x7efcd415b850 2023-11-28T20:50:50.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1407088665 0 0) 0x7efcc0003520 con 0x7efcd415b850 2023-11-28T20:50:50.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd40b35b0 msgr2=0x7efcd415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:50.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd40b35b0 0x7efcd415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 >> v2:172.21.15.190:3300/0 conn(0x7efcd40b1340 msgr2=0x7efcd415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:50.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 --2- 172.21.15.45:0/54754701 >> v2:172.21.15.190:3300/0 conn(0x7efcd40b1340 0x7efcd415f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd415c360 con 0x7efcd415b850 2023-11-28T20:50:50.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7efcbc001670 con 0x7efcd415b850 2023-11-28T20:50:50.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 >> v1:172.21.15.45:6789/0 conn(0x7efcd415b850 legacy=0x7efcd40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd40b35b0 0x7efcc0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcc0003f40 0x7efcc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcc0004d00 0x7efcc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd415ee40 con 0x7efcc0003f40 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd415c080 con 0x7efcc0004d00 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc000fd90 con 0x7efcd40b35b0 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.262+0000 7efccbfff640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcc0003f40 0x7efcc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.263+0000 7efcd3fff640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcc0004d00 0x7efcc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.263+0000 7efcd37fe640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd40b35b0 0x7efcc0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.263+0000 7efccbfff640 1 -- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd40b35b0 msgr2=0x7efcc0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.263+0000 7efccbfff640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd40b35b0 0x7efcc0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.263+0000 7efccbfff640 1 -- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcc0004d00 msgr2=0x7efcc0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.263+0000 7efccbfff640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcc0004d00 0x7efcc0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.263+0000 7efccbfff640 1 -- 172.21.15.45:0/54754701 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc0003520 con 0x7efcc0003f40 2023-11-28T20:50:50.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.264+0000 7efccbfff640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcc0003f40 0x7efcc00047c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00c110 tx=0x7efcc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1032a0e69a77c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:50.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.264+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc005ad0 con 0x7efcc0003f40 2023-11-28T20:50:50.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.264+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcbc005c50 con 0x7efcc0003f40 2023-11-28T20:50:50.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.264+0000 7efcd2ffd640 1 -- 172.21.15.45:0/54754701 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc02c7e0 con 0x7efcc0003f40 2023-11-28T20:50:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.265+0000 7efcd9e87640 1 -- 172.21.15.45:0/54754701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcc0003f40 msgr2=0x7efcc00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.265+0000 7efcd9e87640 1 --2- 172.21.15.45:0/54754701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcc0003f40 0x7efcc00047c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00c110 tx=0x7efcc00057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.265+0000 7efcd3fff640 1 -- 172.21.15.45:0/54754701 reap_dead start 2023-11-28T20:50:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.265+0000 7efcd9e87640 1 -- 172.21.15.45:0/54754701 shutdown_connections 2023-11-28T20:50:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.265+0000 7efcd9e87640 1 -- 172.21.15.45:0/54754701 >> 172.21.15.45:0/54754701 conn(0x7efcd406a500 msgr2=0x7efcd406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.265+0000 7efcd9e87640 1 -- 172.21.15.45:0/54754701 shutdown_connections 2023-11-28T20:50:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.265+0000 7efcd9e87640 1 -- 172.21.15.45:0/54754701 wait complete. 2023-11-28T20:50:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.267+0000 7efcd9e87640 1 Processor -- start 2023-11-28T20:50:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.267+0000 7efcd9e87640 1 -- start start 2023-11-28T20:50:50.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.268+0000 7efcd9e87640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcc0003f40 0x7efcd415e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.268+0000 7efcd9e87640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd413c410 0x7efcd413b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.268+0000 7efcd9e87640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4141e50 0x7efcd41422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.268+0000 7efcd9e87640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd413d810 con 0x7efcd4141e50 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.268+0000 7efcd9e87640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd4142810 con 0x7efcd413c410 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.268+0000 7efcd9e87640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd413cb90 con 0x7efcc0003f40 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4141e50 0x7efcd41422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4141e50 0x7efcd41422d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52548/0 (socket says 172.21.15.45:52548) 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd3fff640 1 -- 172.21.15.45:0/4202981504 learned_addr learned my addr 172.21.15.45:0/4202981504 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd37fe640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcc0003f40 0x7efcd415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efccbfff640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd413c410 0x7efcd413b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd3fff640 1 -- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcc0003f40 msgr2=0x7efcd415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd3fff640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcc0003f40 0x7efcd415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd3fff640 1 -- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd413c410 msgr2=0x7efcd413b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd3fff640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd413c410 0x7efcd413b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd3fff640 1 -- 172.21.15.45:0/4202981504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd413ce70 con 0x7efcd4141e50 2023-11-28T20:50:50.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd37fe640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcc0003f40 0x7efcd415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.269+0000 7efcd3fff640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4141e50 0x7efcd41422d0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7efcd406e440 tx=0x7efccc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.270+0000 7efcd0ff9640 1 -- 172.21.15.45:0/4202981504 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc00e070 con 0x7efcd4141e50 2023-11-28T20:50:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.270+0000 7efcd0ff9640 1 -- 172.21.15.45:0/4202981504 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc011e10 con 0x7efcd4141e50 2023-11-28T20:50:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.270+0000 7efcd0ff9640 1 -- 172.21.15.45:0/4202981504 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc003990 con 0x7efcd4141e50 2023-11-28T20:50:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.270+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd413d150 con 0x7efcd4141e50 2023-11-28T20:50:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.270+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd413c890 con 0x7efcd4141e50 2023-11-28T20:50:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.271+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd40713e0 con 0x7efcd4141e50 2023-11-28T20:50:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.272+0000 7efcd0ff9640 1 -- 172.21.15.45:0/4202981504 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 36) v1 ==== 89069+0+0 (secure 0 0 0) 0x7efccc003af0 con 0x7efcd4141e50 2023-11-28T20:50:50.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.273+0000 7efcd0ff9640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7efcb406ac90 0x7efcb406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.273+0000 7efcd0ff9640 1 -- 172.21.15.45:0/4202981504 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6717+0+0 (secure 0 0 0) 0x7efccc089da0 con 0x7efcd4141e50 2023-11-28T20:50:50.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.274+0000 7efcd37fe640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7efcb406ac90 0x7efcb406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.275+0000 7efcd37fe640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7efcb406ac90 0x7efcb406d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efcc4006ce0 tx=0x7efcc4008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.278+0000 7efcd0ff9640 1 -- 172.21.15.45:0/4202981504 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efccc054cd0 con 0x7efcd4141e50 2023-11-28T20:50:50.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.331+0000 7efcd0ff9640 1 -- 172.21.15.45:0/4202981504 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7efccc0550a0 con 0x7efcd4141e50 2023-11-28T20:50:50.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.527+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcd40733d0 con 0x7efcd4141e50 2023-11-28T20:50:50.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.528+0000 7efcd0ff9640 1 -- 172.21.15.45:0/4202981504 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4914 (secure 0 0 0) 0x7efccc085f60 con 0x7efcd4141e50 2023-11-28T20:50:50.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:50.529 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5438,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2396470524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2396470524},{"type":"v1","addr":"172.21.15.45:6835","nonce":2396470524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/436103276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":436103276},{"type":"v1","addr":"172.21.15.190:6836","nonce":436103276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/834328551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":834328551},{"type":"v1","addr":"172.21.15.45:6834","nonce":834328551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:50:26.812959+0000","modified":"2023-11-28T20:50:29.910521+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6837/3032907708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3032907708},{"type":"v1","addr":"172.21.15.190:6837","nonce":3032907708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T20:50:50.529 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 36 2023-11-28T20:50:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.530+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7efcb406ac90 msgr2=0x7efcb406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.530+0000 7efcd9e87640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7efcb406ac90 0x7efcb406d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efcc4006ce0 tx=0x7efcc4008040 comp rx=0 tx=0).stop 2023-11-28T20:50:50.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.530+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4141e50 msgr2=0x7efcd41422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.530+0000 7efcd9e87640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4141e50 0x7efcd41422d0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7efcd406e440 tx=0x7efccc0115a0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.531+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 shutdown_connections 2023-11-28T20:50:50.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.531+0000 7efcd9e87640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7efcb406ac90 0x7efcb406d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.531+0000 7efcd9e87640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4141e50 0x7efcd41422d0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.531+0000 7efcd9e87640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd413c410 0x7efcd413b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.531+0000 7efcd9e87640 1 --2- 172.21.15.45:0/4202981504 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcc0003f40 0x7efcd415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.531+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 >> 172.21.15.45:0/4202981504 conn(0x7efcd406a500 msgr2=0x7efcd40b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:50.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.531+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 shutdown_connections 2023-11-28T20:50:50.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.531+0000 7efcd9e87640 1 -- 172.21.15.45:0/4202981504 wait complete. 2023-11-28T20:50:50.553 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:50:50.685 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:50.682+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:50:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.784+0000 7f0cb4886640 1 Processor -- start 2023-11-28T20:50:50.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.784+0000 7f0cb4886640 1 -- start start 2023-11-28T20:50:50.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.786+0000 7f0cb4886640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00cfaf0 0x7f0cb00cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.786+0000 7f0cb4886640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb00d1250 con 0x7f0cb00b6560 2023-11-28T20:50:50.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.786+0000 7f0cb4886640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb00ce6b0 con 0x7f0cb00b69c0 2023-11-28T20:50:50.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.786+0000 7f0cb4886640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00ce990 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.786+0000 7f0ca7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0cb00b6560 0x7f0cb00cdad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45078/0 (socket says 172.21.15.45:45078) 2023-11-28T20:50:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.786+0000 7f0ca7fff640 1 -- 172.21.15.45:0/2434442472 learned_addr learned my addr 172.21.15.45:0/2434442472 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.787+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2434442472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00cfaf0 0x7f0cb00cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.788+0000 7f0caed9d640 1 -- 172.21.15.45:0/2434442472 >> v1:172.21.15.190:6789/0 conn(0x7f0cb00b69c0 legacy=0x7f0cb015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.788+0000 7f0caed9d640 1 -- 172.21.15.45:0/2434442472 >> v1:172.21.15.45:6789/0 conn(0x7f0cb00b6560 legacy=0x7f0cb00cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.788+0000 7f0caed9d640 1 -- 172.21.15.45:0/2434442472 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00d14a0 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.789+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2434442472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00cfaf0 0x7f0cb00cfed0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00097d0 tx=0x7f0ca002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad42ea15d323ab46 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:50.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.794+0000 7f0cae59c640 1 -- 172.21.15.45:0/2434442472 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0031890 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.794+0000 7f0cae59c640 1 -- 172.21.15.45:0/2434442472 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca00319f0 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.794+0000 7f0cae59c640 1 -- 172.21.15.45:0/2434442472 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0031d80 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.795+0000 7f0cb4886640 1 -- 172.21.15.45:0/2434442472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00cfaf0 msgr2=0x7f0cb00cfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.795+0000 7f0cb4886640 1 --2- 172.21.15.45:0/2434442472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00cfaf0 0x7f0cb00cfed0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00097d0 tx=0x7f0ca002b930 comp rx=0 tx=0).stop 2023-11-28T20:50:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.795+0000 7f0cb4886640 1 -- 172.21.15.45:0/2434442472 shutdown_connections 2023-11-28T20:50:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.795+0000 7f0cb4886640 1 --2- 172.21.15.45:0/2434442472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00cfaf0 0x7f0cb00cfed0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.795+0000 7f0cb4886640 1 -- 172.21.15.45:0/2434442472 >> 172.21.15.45:0/2434442472 conn(0x7f0cb00b4c40 msgr2=0x7f0cb00b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.795+0000 7f0cb4886640 1 -- 172.21.15.45:0/2434442472 shutdown_connections 2023-11-28T20:50:50.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.795+0000 7f0cb4886640 1 -- 172.21.15.45:0/2434442472 wait complete. 2023-11-28T20:50:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.797+0000 7f0cb4886640 1 Processor -- start 2023-11-28T20:50:50.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.797+0000 7f0cb4886640 1 -- start start 2023-11-28T20:50:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.798+0000 7f0cb4886640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cb00b6560 0x7f0cb014b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.798+0000 7f0cb4886640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00b69c0 0x7f0cb00cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.798+0000 7f0cb4886640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb00cfaf0 0x7f0cb0159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.798+0000 7f0cb4886640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb014b660 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.798+0000 7f0cb4886640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00cbd30 con 0x7f0cb00b6560 2023-11-28T20:50:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.798+0000 7f0cb4886640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00cc010 con 0x7f0cb00b69c0 2023-11-28T20:50:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0caf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb00cfaf0 0x7f0cb0159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0caf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb00cfaf0 0x7f0cb0159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52556/0 (socket says 172.21.15.45:52556) 2023-11-28T20:50:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0caf59e640 1 -- 172.21.15.45:0/2858618565 learned_addr learned my addr 172.21.15.45:0/2858618565 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cb00b6560 0x7f0cb014b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0ca7fff640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00b69c0 0x7f0cb00cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0caf59e640 1 -- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00b69c0 msgr2=0x7f0cb00cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0caf59e640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00b69c0 0x7f0cb00cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0caf59e640 1 -- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cb00b6560 msgr2=0x7f0cb014b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0caf59e640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cb00b6560 0x7f0cb014b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.799+0000 7f0caf59e640 1 -- 172.21.15.45:0/2858618565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0009400 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.800+0000 7f0caf59e640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb00cfaf0 0x7f0cb0159440 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800c7a0 tx=0x7f0ca800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:50.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.800+0000 7f0ca77fe640 1 -- 172.21.15.45:0/2858618565 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8003b70 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.800+0000 7f0ca77fe640 1 -- 172.21.15.45:0/2858618565 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca8003cf0 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.800+0000 7f0ca77fe640 1 -- 172.21.15.45:0/2858618565 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8003ea0 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.800+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb00cc2f0 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.800+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb00cc8b0 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.801+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb004e920 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.802+0000 7f0ca77fe640 1 -- 172.21.15.45:0/2858618565 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f0ca800aa90 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.803+0000 7f0ca77fe640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f0c9006af00 0x7f0c9006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:50.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.804+0000 7f0ca77fe640 1 -- 172.21.15.45:0/2858618565 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f0ca8089940 con 0x7f0cb00cfaf0 2023-11-28T20:50:50.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.804+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f0c9006af00 0x7f0c9006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:50.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.806+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f0c9006af00 0x7f0c9006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0036000 tx=0x7f0ca003c040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:50.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:50.808+0000 7f0ca77fe640 1 -- 172.21.15.45:0/2858618565 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ca8054510 con 0x7f0cb00cfaf0 2023-11-28T20:50:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.047+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cb00d1030 con 0x7f0cb00cfaf0 2023-11-28T20:50:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.048+0000 7f0ca77fe640 1 -- 172.21.15.45:0/2858618565 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4914 (secure 0 0 0) 0x7f0ca8057b30 con 0x7f0cb00cfaf0 2023-11-28T20:50:51.049 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:51.049 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5438,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2396470524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2396470524},{"type":"v1","addr":"172.21.15.45:6835","nonce":2396470524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/436103276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":436103276},{"type":"v1","addr":"172.21.15.190:6836","nonce":436103276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/834328551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":834328551},{"type":"v1","addr":"172.21.15.45:6834","nonce":834328551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:50:26.812959+0000","modified":"2023-11-28T20:50:29.910521+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6837/3032907708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3032907708},{"type":"v1","addr":"172.21.15.190:6837","nonce":3032907708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T20:50:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 36 2023-11-28T20:50:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.050+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f0c9006af00 msgr2=0x7f0c9006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.050+0000 7f0cb4886640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f0c9006af00 0x7f0c9006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0036000 tx=0x7f0ca003c040 comp rx=0 tx=0).stop 2023-11-28T20:50:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb00cfaf0 msgr2=0x7f0cb0159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb00cfaf0 0x7f0cb0159440 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800c7a0 tx=0x7f0ca800cc60 comp rx=0 tx=0).stop 2023-11-28T20:50:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 shutdown_connections 2023-11-28T20:50:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f0c9006af00 0x7f0c9006d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb00cfaf0 0x7f0cb0159440 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cb00b69c0 0x7f0cb00cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 --2- 172.21.15.45:0/2858618565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cb00b6560 0x7f0cb014b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 >> 172.21.15.45:0/2858618565 conn(0x7f0cb00b4c40 msgr2=0x7f0cb00b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 shutdown_connections 2023-11-28T20:50:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.051+0000 7f0cb4886640 1 -- 172.21.15.45:0/2858618565 wait complete. 2023-11-28T20:50:51.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:51.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.299+0000 7fcb1579a640 1 Processor -- start 2023-11-28T20:50:51.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.300+0000 7fcb1579a640 1 -- start start 2023-11-28T20:50:51.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.301+0000 7fcb1579a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100b5a80 0x7fcb1009a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.301+0000 7fcb1579a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb100b45e0 con 0x7fcb1015d2f0 2023-11-28T20:50:51.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.301+0000 7fcb1579a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1006b920 con 0x7fcb10161640 2023-11-28T20:50:51.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.301+0000 7fcb1579a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1006b640 con 0x7fcb100b5a80 2023-11-28T20:50:51.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.302+0000 7fcb0ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcb1015d2f0 0x7fcb10160f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45084/0 (socket says 172.21.15.45:45084) 2023-11-28T20:50:51.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.302+0000 7fcb0ffff640 1 -- 172.21.15.45:0/1435591966 learned_addr learned my addr 172.21.15.45:0/1435591966 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:51.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.302+0000 7fcb14f99640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100b5a80 0x7fcb1009a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.302+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033351110 0 0) 0x7fcb100b45e0 con 0x7fcb1015d2f0 2023-11-28T20:50:51.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.302+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf8003520 con 0x7fcb1015d2f0 2023-11-28T20:50:51.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2862603255 0 0) 0x7fcaf8003520 con 0x7fcb1015d2f0 2023-11-28T20:50:51.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100b5a80 msgr2=0x7fcb1009a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:51.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100b5a80 0x7fcb1009a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 >> v1:172.21.15.190:6789/0 conn(0x7fcb10161640 legacy=0x7fcb100b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100b3f90 con 0x7fcb1015d2f0 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fcb000019f0 con 0x7fcb1015d2f0 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 >> v1:172.21.15.45:6789/0 conn(0x7fcb1015d2f0 legacy=0x7fcb10160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100b5a80 0x7fcaf80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8003f70 0x7fcaf80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8004d00 0x7fcaf8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1006b640 con 0x7fcaf8004d00 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf800bb50 con 0x7fcaf8003f70 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf800bd10 con 0x7fcb100b5a80 2023-11-28T20:50:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.303+0000 7fcb14f99640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8004d00 0x7fcaf8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.304+0000 7fcb0ffff640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100b5a80 0x7fcaf80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.304+0000 7fcb14f99640 1 -- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100b5a80 msgr2=0x7fcaf80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.304+0000 7fcb14f99640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100b5a80 0x7fcaf80039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.304+0000 7fcb0f7fe640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8003f70 0x7fcaf80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.304+0000 7fcb14f99640 1 -- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8003f70 msgr2=0x7fcaf80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.304+0000 7fcb14f99640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8003f70 0x7fcaf80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.304+0000 7fcb14f99640 1 -- 172.21.15.45:0/1435591966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf8003520 con 0x7fcaf8004d00 2023-11-28T20:50:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.305+0000 7fcb14f99640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8004d00 0x7fcaf8005180 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009a90 tx=0x7fcaf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15de33860698ced1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.305+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0802c680 con 0x7fcaf8004d00 2023-11-28T20:50:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.305+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0802c7e0 con 0x7fcaf8004d00 2023-11-28T20:50:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.305+0000 7fcb0effd640 1 -- 172.21.15.45:0/1435591966 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0802cb70 con 0x7fcaf8004d00 2023-11-28T20:50:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.306+0000 7fcb1579a640 1 -- 172.21.15.45:0/1435591966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8004d00 msgr2=0x7fcaf8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.306+0000 7fcb1579a640 1 --2- 172.21.15.45:0/1435591966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8004d00 0x7fcaf8005180 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009a90 tx=0x7fcaf80057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.306+0000 7fcb14f99640 1 -- 172.21.15.45:0/1435591966 reap_dead start 2023-11-28T20:50:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.306+0000 7fcb1579a640 1 -- 172.21.15.45:0/1435591966 shutdown_connections 2023-11-28T20:50:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.306+0000 7fcb1579a640 1 -- 172.21.15.45:0/1435591966 >> 172.21.15.45:0/1435591966 conn(0x7fcb10099a40 msgr2=0x7fcb10099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.306+0000 7fcb1579a640 1 -- 172.21.15.45:0/1435591966 shutdown_connections 2023-11-28T20:50:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.306+0000 7fcb1579a640 1 -- 172.21.15.45:0/1435591966 wait complete. 2023-11-28T20:50:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.308+0000 7fcb1579a640 1 Processor -- start 2023-11-28T20:50:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.308+0000 7fcb1579a640 1 -- start start 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.309+0000 7fcb1579a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb10146340 0x7fcb101f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.309+0000 7fcb1579a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100c6a30 0x7fcb100cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.309+0000 7fcb1579a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb100d0270 0x7fcb100c4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.309+0000 7fcb1579a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100d06f0 con 0x7fcb10146340 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.309+0000 7fcb1579a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb101e6980 con 0x7fcb100d0270 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.309+0000 7fcb1579a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100c6eb0 con 0x7fcb100c6a30 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.309+0000 7fcb0ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb10146340 0x7fcb101f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb0ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb10146340 0x7fcb101f6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52576/0 (socket says 172.21.15.45:52576) 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb0ffff640 1 -- 172.21.15.45:0/964762169 learned_addr learned my addr 172.21.15.45:0/964762169 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb0f7fe640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100c6a30 0x7fcb100cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb0ffff640 1 -- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100c6a30 msgr2=0x7fcb100cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb0ffff640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100c6a30 0x7fcb100cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb14f99640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb100d0270 0x7fcb100c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb0ffff640 1 -- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb100d0270 msgr2=0x7fcb100c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb0ffff640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb100d0270 0x7fcb100c4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb0ffff640 1 -- 172.21.15.45:0/964762169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb101f7070 con 0x7fcb10146340 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.310+0000 7fcb0ffff640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb10146340 0x7fcb101f6970 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fcb00002180 tx=0x7fcb0000b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.311+0000 7fcb0cff9640 1 -- 172.21.15.45:0/964762169 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb000072d0 con 0x7fcb10146340 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.311+0000 7fcb0cff9640 1 -- 172.21.15.45:0/964762169 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00007430 con 0x7fcb10146340 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.311+0000 7fcb0cff9640 1 -- 172.21.15.45:0/964762169 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb000077a0 con 0x7fcb10146340 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.311+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb101f7350 con 0x7fcb10146340 2023-11-28T20:50:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.311+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb101e6bd0 con 0x7fcb10146340 2023-11-28T20:50:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.312+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb101e6df0 con 0x7fcb10146340 2023-11-28T20:50:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.313+0000 7fcb0cff9640 1 -- 172.21.15.45:0/964762169 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fcb00007900 con 0x7fcb10146340 2023-11-28T20:50:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.314+0000 7fcb0cff9640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fcafc06af00 0x7fcafc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.314+0000 7fcb0cff9640 1 -- 172.21.15.45:0/964762169 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6717+0+0 (secure 0 0 0) 0x7fcb00089ff0 con 0x7fcb10146340 2023-11-28T20:50:51.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.315+0000 7fcb0f7fe640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fcafc06af00 0x7fcafc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.316+0000 7fcb0f7fe640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fcafc06af00 0x7fcafc06d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb101f68d0 tx=0x7fcb04009040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.318+0000 7fcb0cff9640 1 -- 172.21.15.45:0/964762169 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb000bce90 con 0x7fcb10146340 2023-11-28T20:50:51.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.529+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb1009a910 con 0x7fcb10146340 2023-11-28T20:50:51.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.532+0000 7fcb0cff9640 1 -- 172.21.15.45:0/964762169 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+17719 (secure 0 0 0) 0x7fcb00019020 con 0x7fcb10146340 2023-11-28T20:50:51.534 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:50:51.534 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":90,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:47.747533+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:50:24.656511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-28T20:50:25.847745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6833/2590179685":"2023-11-29T20:50:47.747393+0000","172.21.15.190:6832/2590179685":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/3115528802":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/2734256196":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/559775284":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/709518420":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/1852415784":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/1902490820":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1766221473":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/2227588570":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/1382859164":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6833/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1155029644":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6832/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/2238215747":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/3448409370":"2023-11-29T20:50:41.547044+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:50:51.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fcafc06af00 msgr2=0x7fcafc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fcafc06af00 0x7fcafc06d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb101f68d0 tx=0x7fcb04009040 comp rx=0 tx=0).stop 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb10146340 msgr2=0x7fcb101f6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb10146340 0x7fcb101f6970 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fcb00002180 tx=0x7fcb0000b560 comp rx=0 tx=0).stop 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 shutdown_connections 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fcafc06af00 0x7fcafc06d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb100d0270 0x7fcb100c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb100c6a30 0x7fcb100cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 --2- 172.21.15.45:0/964762169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb10146340 0x7fcb101f6970 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 >> 172.21.15.45:0/964762169 conn(0x7fcb10099a40 msgr2=0x7fcb1006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 shutdown_connections 2023-11-28T20:50:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.535+0000 7fcb1579a640 1 -- 172.21.15.45:0/964762169 wait complete. 2023-11-28T20:50:51.554 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:51.551+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:50:51.557 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T20:50:51.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.782+0000 7fd872c13640 1 Processor -- start 2023-11-28T20:50:51.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.782+0000 7fd872c13640 1 -- start start 2023-11-28T20:50:51.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.784+0000 7fd872c13640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd86c160e70 0x7fd86c0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.784+0000 7fd872c13640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86c0ce7d0 con 0x7fd86c15d310 2023-11-28T20:50:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.784+0000 7fd872c13640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86c0d14d0 con 0x7fd86c0ce3a0 2023-11-28T20:50:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.784+0000 7fd872c13640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c1613c0 con 0x7fd86c160e70 2023-11-28T20:50:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.784+0000 7fd871410640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd86c160e70 0x7fd86c0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.785+0000 7fd871410640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd86c160e70 0x7fd86c0cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52586/0 (socket says 172.21.15.45:52586) 2023-11-28T20:50:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.785+0000 7fd871410640 1 -- 172.21.15.45:0/1684546177 learned_addr learned my addr 172.21.15.45:0/1684546177 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.785+0000 7fd871410640 1 -- 172.21.15.45:0/1684546177 >> v1:172.21.15.190:6789/0 conn(0x7fd86c0ce3a0 legacy=0x7fd86c0d19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.785+0000 7fd871410640 1 -- 172.21.15.45:0/1684546177 >> v1:172.21.15.45:6789/0 conn(0x7fd86c15d310 legacy=0x7fd86c0b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.785+0000 7fd871410640 1 -- 172.21.15.45:0/1684546177 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c1616a0 con 0x7fd86c160e70 2023-11-28T20:50:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.786+0000 7fd871410640 1 --2- 172.21.15.45:0/1684546177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c160e70 0x7fd86c0cde60 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009870 tx=0x7fd85c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba52fcebd36cac88 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.787+0000 7fd870c0f640 1 -- 172.21.15.45:0/1684546177 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c031870 con 0x7fd86c160e70 2023-11-28T20:50:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.787+0000 7fd870c0f640 1 -- 172.21.15.45:0/1684546177 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c0319d0 con 0x7fd86c160e70 2023-11-28T20:50:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.787+0000 7fd870c0f640 1 -- 172.21.15.45:0/1684546177 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c031d60 con 0x7fd86c160e70 2023-11-28T20:50:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.788+0000 7fd872c13640 1 -- 172.21.15.45:0/1684546177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c160e70 msgr2=0x7fd86c0cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.788+0000 7fd872c13640 1 --2- 172.21.15.45:0/1684546177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c160e70 0x7fd86c0cde60 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009870 tx=0x7fd85c02b910 comp rx=0 tx=0).stop 2023-11-28T20:50:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.788+0000 7fd872c13640 1 -- 172.21.15.45:0/1684546177 shutdown_connections 2023-11-28T20:50:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.788+0000 7fd872c13640 1 --2- 172.21.15.45:0/1684546177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c160e70 0x7fd86c0cde60 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.788+0000 7fd872c13640 1 -- 172.21.15.45:0/1684546177 >> 172.21.15.45:0/1684546177 conn(0x7fd86c06a500 msgr2=0x7fd86c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.788+0000 7fd872c13640 1 -- 172.21.15.45:0/1684546177 shutdown_connections 2023-11-28T20:50:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.788+0000 7fd872c13640 1 -- 172.21.15.45:0/1684546177 wait complete. 2023-11-28T20:50:51.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.790+0000 7fd872c13640 1 Processor -- start 2023-11-28T20:50:51.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.790+0000 7fd872c13640 1 -- start start 2023-11-28T20:50:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.791+0000 7fd872c13640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd86c0ce3a0 0x7fd86c151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.791+0000 7fd872c13640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c15d310 0x7fd86c15f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.791+0000 7fd872c13640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd86c160e70 0x7fd86c15f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.791+0000 7fd872c13640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0caea0 con 0x7fd86c15d310 2023-11-28T20:50:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.791+0000 7fd872c13640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c157440 con 0x7fd86c160e70 2023-11-28T20:50:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.791+0000 7fd872c13640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0ca290 con 0x7fd86c0ce3a0 2023-11-28T20:50:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.791+0000 7fd871410640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c15d310 0x7fd86c15f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.791+0000 7fd871410640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c15d310 0x7fd86c15f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52602/0 (socket says 172.21.15.45:52602) 2023-11-28T20:50:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.791+0000 7fd871410640 1 -- 172.21.15.45:0/1197609770 learned_addr learned my addr 172.21.15.45:0/1197609770 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.792+0000 7fd871410640 1 -- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd86c0ce3a0 msgr2=0x7fd86c151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.792+0000 7fd871410640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd86c0ce3a0 0x7fd86c151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.792+0000 7fd871410640 1 -- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd86c160e70 msgr2=0x7fd86c15f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.792+0000 7fd872412640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd86c160e70 0x7fd86c15f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.792+0000 7fd871410640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd86c160e70 0x7fd86c15f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.792+0000 7fd871410640 1 -- 172.21.15.45:0/1197609770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c0094d0 con 0x7fd86c15d310 2023-11-28T20:50:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.792+0000 7fd871410640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c15d310 0x7fd86c15f450 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fd85c000c00 tx=0x7fd85c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.793+0000 7fd8627fc640 1 -- 172.21.15.45:0/1197609770 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c031820 con 0x7fd86c15d310 2023-11-28T20:50:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.793+0000 7fd8627fc640 1 -- 172.21.15.45:0/1197609770 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c0443f0 con 0x7fd86c15d310 2023-11-28T20:50:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.793+0000 7fd8627fc640 1 -- 172.21.15.45:0/1197609770 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0447a0 con 0x7fd86c15d310 2023-11-28T20:50:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.793+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c0ca570 con 0x7fd86c15d310 2023-11-28T20:50:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.793+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c0cab30 con 0x7fd86c15d310 2023-11-28T20:50:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.794+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c071b30 con 0x7fd86c15d310 2023-11-28T20:50:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.795+0000 7fd8627fc640 1 -- 172.21.15.45:0/1197609770 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fd85c04d450 con 0x7fd86c15d310 2023-11-28T20:50:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.796+0000 7fd8627fc640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd85006af00 0x7fd85006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.796+0000 7fd8627fc640 1 -- 172.21.15.45:0/1197609770 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6717+0+0 (secure 0 0 0) 0x7fd85c02d040 con 0x7fd86c15d310 2023-11-28T20:50:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.797+0000 7fd871c11640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd85006af00 0x7fd85006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.798+0000 7fd871c11640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd85006af00 0x7fd85006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd868000f60 tx=0x7fd8680079b0 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:51.800+0000 7fd8627fc640 1 -- 172.21.15.45:0/1197609770 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd85c076dd0 con 0x7fd86c15d310 2023-11-28T20:50:52.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.038+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd86c04e920 con 0x7fd86c15d310 2023-11-28T20:50:52.040 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T20:50:52.039+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T20:50:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.043+0000 7fd8627fc640 1 -- 172.21.15.45:0/1197609770 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd85c07a3f0 con 0x7fd86c15d310 2023-11-28T20:50:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T20:50:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.047+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd85006af00 msgr2=0x7fd85006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.047+0000 7fd872c13640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd85006af00 0x7fd85006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd868000f60 tx=0x7fd8680079b0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.047+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c15d310 msgr2=0x7fd86c15f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.047+0000 7fd872c13640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c15d310 0x7fd86c15f450 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fd85c000c00 tx=0x7fd85c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.048+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 shutdown_connections 2023-11-28T20:50:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.048+0000 7fd872c13640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd85006af00 0x7fd85006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.048+0000 7fd872c13640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd86c160e70 0x7fd86c15f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.048+0000 7fd872c13640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd86c15d310 0x7fd86c15f450 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.048+0000 7fd872c13640 1 --2- 172.21.15.45:0/1197609770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd86c0ce3a0 0x7fd86c151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.048+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 >> 172.21.15.45:0/1197609770 conn(0x7fd86c06a500 msgr2=0x7fd86c0cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.048+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 shutdown_connections 2023-11-28T20:50:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.048+0000 7fd872c13640 1 -- 172.21.15.45:0/1197609770 wait complete. 2023-11-28T20:50:52.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T20:50:52.077 INFO:tasks.ceph.mds.d.smithi190.stderr: -3713> 2023-11-28T20:50:33.235+0000 7f8505e3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:50:52.159 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:50:52.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.291+0000 7f403be90640 1 Processor -- start 2023-11-28T20:50:52.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.291+0000 7f403be90640 1 -- start start 2023-11-28T20:50:52.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.292+0000 7f403be90640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f40340c63b0 0x7f40340cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:52.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.292+0000 7f403be90640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340c4f10 con 0x7f403415b3c0 2023-11-28T20:50:52.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.292+0000 7f403be90640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340c4c10 con 0x7f403409a150 2023-11-28T20:50:52.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.292+0000 7f403be90640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403415b7a0 con 0x7f40340c63b0 2023-11-28T20:50:52.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.293+0000 7f4039c2c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f403415b3c0 0x7f4034099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45094/0 (socket says 172.21.15.45:45094) 2023-11-28T20:50:52.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.293+0000 7f4039c2c640 1 -- 172.21.15.45:0/1675392377 learned_addr learned my addr 172.21.15.45:0/1675392377 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.293+0000 7f403a42d640 1 --2- 172.21.15.45:0/1675392377 >> v2:172.21.15.190:3300/0 conn(0x7f40340c63b0 0x7f40340cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.293+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906828101 0 0) 0x7f40340c4f10 con 0x7f403415b3c0 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.293+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020003520 con 0x7f403415b3c0 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1960834155 0 0) 0x7f4020003520 con 0x7f403415b3c0 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 >> v2:172.21.15.190:3300/0 conn(0x7f40340c63b0 msgr2=0x7f40340cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 --2- 172.21.15.45:0/1675392377 >> v2:172.21.15.190:3300/0 conn(0x7f40340c63b0 0x7f40340cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 >> v1:172.21.15.190:6789/0 conn(0x7f403409a150 legacy=0x7f40340cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340cf600 con 0x7f403415b3c0 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f40300019f0 con 0x7f403415b3c0 2023-11-28T20:50:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 >> v1:172.21.15.45:6789/0 conn(0x7f403415b3c0 legacy=0x7f4034099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 --2- 172.21.15.45:0/1675392377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40200039c0 0x7f4020003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 --2- 172.21.15.45:0/1675392377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f4020004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 --2- 172.21.15.45:0/1675392377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4020005180 0x7f402000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403415b7a0 con 0x7f40200047c0 2023-11-28T20:50:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402000c080 con 0x7f40200039c0 2023-11-28T20:50:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402000c240 con 0x7f4020005180 2023-11-28T20:50:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.294+0000 7f403942b640 1 --2- 172.21.15.45:0/1675392377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f4020004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.295+0000 7f403942b640 1 -- 172.21.15.45:0/1675392377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4020005180 msgr2=0x7f402000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.295+0000 7f403942b640 1 --2- 172.21.15.45:0/1675392377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4020005180 0x7f402000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.295+0000 7f403942b640 1 -- 172.21.15.45:0/1675392377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40200039c0 msgr2=0x7f4020003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:52.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.295+0000 7f403942b640 1 --2- 172.21.15.45:0/1675392377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40200039c0 0x7f4020003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.295+0000 7f403942b640 1 -- 172.21.15.45:0/1675392377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020003520 con 0x7f40200047c0 2023-11-28T20:50:52.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.295+0000 7f403a42d640 1 -- 172.21.15.45:0/1675392377 reap_dead start 2023-11-28T20:50:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.296+0000 7f403942b640 1 --2- 172.21.15.45:0/1675392377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f4020004c40 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4024001750 tx=0x7f40200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483d67a64dcb38ae server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.296+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40240052d0 con 0x7f40200047c0 2023-11-28T20:50:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.296+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402402e050 con 0x7f40200047c0 2023-11-28T20:50:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.296+0000 7f4038c2a640 1 -- 172.21.15.45:0/1675392377 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024003830 con 0x7f40200047c0 2023-11-28T20:50:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.297+0000 7f403be90640 1 -- 172.21.15.45:0/1675392377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 msgr2=0x7f4020004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.297+0000 7f403be90640 1 --2- 172.21.15.45:0/1675392377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f4020004c40 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4024001750 tx=0x7f40200057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.297+0000 7f403be90640 1 -- 172.21.15.45:0/1675392377 shutdown_connections 2023-11-28T20:50:52.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.297+0000 7f403be90640 1 --2- 172.21.15.45:0/1675392377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f4020004c40 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.297+0000 7f403be90640 1 -- 172.21.15.45:0/1675392377 >> 172.21.15.45:0/1675392377 conn(0x7f40340d1020 msgr2=0x7f40340d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:52.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.297+0000 7f403be90640 1 -- 172.21.15.45:0/1675392377 shutdown_connections 2023-11-28T20:50:52.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.297+0000 7f403be90640 1 -- 172.21.15.45:0/1675392377 wait complete. 2023-11-28T20:50:52.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.299+0000 7f403be90640 1 Processor -- start 2023-11-28T20:50:52.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.299+0000 7f403be90640 1 -- start start 2023-11-28T20:50:52.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.300+0000 7f403be90640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f40340b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:52.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.300+0000 7f403be90640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f403409b550 0x7f403414f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:52.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.300+0000 7f403be90640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f403414cc90 0x7f403409df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:52.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.300+0000 7f403be90640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40340b57a0 con 0x7f40200047c0 2023-11-28T20:50:52.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.300+0000 7f403be90640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403409bac0 con 0x7f403414cc90 2023-11-28T20:50:52.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.300+0000 7f403be90640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403414fc30 con 0x7f403409b550 2023-11-28T20:50:52.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.300+0000 7f4039c2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f40340b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:52.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.300+0000 7f4039c2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f40340b2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52626/0 (socket says 172.21.15.45:52626) 2023-11-28T20:50:52.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.300+0000 7f4039c2c640 1 -- 172.21.15.45:0/2598805350 learned_addr learned my addr 172.21.15.45:0/2598805350 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f403942b640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f403409b550 0x7f403414f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f403a42d640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f403414cc90 0x7f403409df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f4039c2c640 1 -- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f403409b550 msgr2=0x7f403414f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f4039c2c640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f403409b550 0x7f403414f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f4039c2c640 1 -- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f403414cc90 msgr2=0x7f403409df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f4039c2c640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f403414cc90 0x7f403409df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f4039c2c640 1 -- 172.21.15.45:0/2598805350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340b4e00 con 0x7f40200047c0 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f4039c2c640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f40340b2550 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f4030001350 tx=0x7f403000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f402a7fc640 1 -- 172.21.15.45:0/2598805350 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403000a650 con 0x7f40200047c0 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f402a7fc640 1 -- 172.21.15.45:0/2598805350 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403000a7b0 con 0x7f40200047c0 2023-11-28T20:50:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f402a7fc640 1 -- 172.21.15.45:0/2598805350 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403000aba0 con 0x7f40200047c0 2023-11-28T20:50:52.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.301+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40340b50e0 con 0x7f40200047c0 2023-11-28T20:50:52.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.302+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403414d110 con 0x7f40200047c0 2023-11-28T20:50:52.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.303+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403414d390 con 0x7f40200047c0 2023-11-28T20:50:52.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.304+0000 7f402a7fc640 1 -- 172.21.15.45:0/2598805350 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f403000ad00 con 0x7f40200047c0 2023-11-28T20:50:52.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.305+0000 7f402a7fc640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f401806af00 0x7f401806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:52.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.305+0000 7f402a7fc640 1 -- 172.21.15.45:0/2598805350 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f4030089280 con 0x7f40200047c0 2023-11-28T20:50:52.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.305+0000 7f403942b640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f401806af00 0x7f401806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:52.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.306+0000 7f403942b640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f401806af00 0x7f401806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4024033000 tx=0x7f4024009030 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:52.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.309+0000 7f402a7fc640 1 -- 172.21.15.45:0/2598805350 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4030053e80 con 0x7f40200047c0 2023-11-28T20:50:52.431 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:52.429+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:50:52.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:52.544+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f40340733d0 con 0x7f40200047c0 2023-11-28T20:50:53.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.047+0000 7f402a7fc640 1 -- 172.21.15.45:0/2598805350 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f40300180a0 con 0x7f40200047c0 2023-11-28T20:50:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f401806af00 msgr2=0x7f401806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f401806af00 0x7f401806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4024033000 tx=0x7f4024009030 comp rx=0 tx=0).stop 2023-11-28T20:50:53.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 msgr2=0x7f40340b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:53.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f40340b2550 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f4030001350 tx=0x7f403000cf00 comp rx=0 tx=0).stop 2023-11-28T20:50:53.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 shutdown_connections 2023-11-28T20:50:53.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f401806af00 0x7f401806d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f403414cc90 0x7f403409df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f403409b550 0x7f403414f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 --2- 172.21.15.45:0/2598805350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40200047c0 0x7f40340b2550 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 >> 172.21.15.45:0/2598805350 conn(0x7f40340d1020 msgr2=0x7f4034159d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:53.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 shutdown_connections 2023-11-28T20:50:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.051+0000 7f403be90640 1 -- 172.21.15.45:0/2598805350 wait complete. 2023-11-28T20:50:53.076 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T20:50:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.310+0000 7fe7d359e640 1 Processor -- start 2023-11-28T20:50:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.310+0000 7fe7d359e640 1 -- start start 2023-11-28T20:50:53.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.312+0000 7fe7d359e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7d415d260 0x7fe7d40d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:53.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.312+0000 7fe7d359e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe7d415dd50 0x7fe7d40b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.312+0000 7fe7d359e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d40d1750 con 0x7fe7d40d1300 2023-11-28T20:50:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.312+0000 7fe7d359e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d40b0780 con 0x7fe7d415d260 2023-11-28T20:50:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.312+0000 7fe7d359e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d415e180 con 0x7fe7d415dd50 2023-11-28T20:50:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.312+0000 7fe7d259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7d415d260 0x7fe7d40d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.312+0000 7fe7d259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7d415d260 0x7fe7d40d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52632/0 (socket says 172.21.15.45:52632) 2023-11-28T20:50:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.312+0000 7fe7d259c640 1 -- 172.21.15.45:0/2433488174 learned_addr learned my addr 172.21.15.45:0/2433488174 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.312+0000 7fe7cbfff640 1 --1- 172.21.15.45:0/2433488174 >> v1:172.21.15.45:6789/0 conn(0x7fe7d40d1300 0x7fe7d415d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45098/0 (socket says 172.21.15.45:45098) 2023-11-28T20:50:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.313+0000 7fe7d2d9d640 1 --2- 172.21.15.45:0/2433488174 >> v2:172.21.15.190:3300/0 conn(0x7fe7d415dd50 0x7fe7d40b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.313+0000 7fe7d259c640 1 -- 172.21.15.45:0/2433488174 >> v2:172.21.15.190:3300/0 conn(0x7fe7d415dd50 msgr2=0x7fe7d40b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.313+0000 7fe7d259c640 1 --2- 172.21.15.45:0/2433488174 >> v2:172.21.15.190:3300/0 conn(0x7fe7d415dd50 0x7fe7d40b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.313+0000 7fe7d259c640 1 -- 172.21.15.45:0/2433488174 >> v1:172.21.15.45:6789/0 conn(0x7fe7d40d1300 legacy=0x7fe7d415d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.313+0000 7fe7d259c640 1 -- 172.21.15.45:0/2433488174 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d415e450 con 0x7fe7d415d260 2023-11-28T20:50:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.313+0000 7fe7d1d9b640 1 -- 172.21.15.45:0/2433488174 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956041614 0 0) 0x7fe7bc001110 con 0x7fe7d40d1300 2023-11-28T20:50:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.313+0000 7fe7d2d9d640 1 --2- 172.21.15.45:0/2433488174 >> v2:172.21.15.190:3300/0 conn(0x7fe7d415dd50 0x7fe7d40b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.314+0000 7fe7d259c640 1 --2- 172.21.15.45:0/2433488174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415d260 0x7fe7d40d0dc0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe7d406a830 tx=0x7fe7c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b832b632b1e78de0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.314+0000 7fe7d1d9b640 1 -- 172.21.15.45:0/2433488174 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4030860 con 0x7fe7d415d260 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d1d9b640 1 -- 172.21.15.45:0/2433488174 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c40309c0 con 0x7fe7d415d260 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d1d9b640 1 -- 172.21.15.45:0/2433488174 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4030d50 con 0x7fe7d415d260 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d359e640 1 -- 172.21.15.45:0/2433488174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415d260 msgr2=0x7fe7d40d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d359e640 1 --2- 172.21.15.45:0/2433488174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415d260 0x7fe7d40d0dc0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe7d406a830 tx=0x7fe7c402b9a0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d359e640 1 -- 172.21.15.45:0/2433488174 shutdown_connections 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d359e640 1 --2- 172.21.15.45:0/2433488174 >> v2:172.21.15.190:3300/0 conn(0x7fe7d415dd50 0x7fe7d40b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d359e640 1 --2- 172.21.15.45:0/2433488174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415d260 0x7fe7d40d0dc0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d359e640 1 -- 172.21.15.45:0/2433488174 >> 172.21.15.45:0/2433488174 conn(0x7fe7d4099a40 msgr2=0x7fe7d4099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d359e640 1 -- 172.21.15.45:0/2433488174 shutdown_connections 2023-11-28T20:50:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.315+0000 7fe7d359e640 1 -- 172.21.15.45:0/2433488174 wait complete. 2023-11-28T20:50:53.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.317+0000 7fe7d359e640 1 Processor -- start 2023-11-28T20:50:53.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.317+0000 7fe7d359e640 1 -- start start 2023-11-28T20:50:53.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.318+0000 7fe7d359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7d40d1300 0x7fe7d415a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:53.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.318+0000 7fe7d359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7d415d260 0x7fe7d4157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:53.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.318+0000 7fe7d359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415dd50 0x7fe7d4158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:53.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.318+0000 7fe7d359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d41f7780 con 0x7fe7d415dd50 2023-11-28T20:50:53.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.318+0000 7fe7d359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d41f6b70 con 0x7fe7d415d260 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.318+0000 7fe7d359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d41f6e50 con 0x7fe7d40d1300 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7d2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415dd50 0x7fe7d4158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7d2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415dd50 0x7fe7d4158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52644/0 (socket says 172.21.15.45:52644) 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7d2d9d640 1 -- 172.21.15.45:0/562345311 learned_addr learned my addr 172.21.15.45:0/562345311 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7d259c640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7d40d1300 0x7fe7d415a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7cbfff640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7d415d260 0x7fe7d4157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7d2d9d640 1 -- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7d40d1300 msgr2=0x7fe7d415a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7d2d9d640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7d40d1300 0x7fe7d415a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7d2d9d640 1 -- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7d415d260 msgr2=0x7fe7d4157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7d2d9d640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7d415d260 0x7fe7d4157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7d2d9d640 1 -- 172.21.15.45:0/562345311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40094d0 con 0x7fe7d415dd50 2023-11-28T20:50:53.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.319+0000 7fe7cbfff640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7d415d260 0x7fe7d4157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:53.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.320+0000 7fe7d2d9d640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415dd50 0x7fe7d4158470 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00cd50 tx=0x7fe7cc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:53.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.320+0000 7fe7caffd640 1 -- 172.21.15.45:0/562345311 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc00f680 con 0x7fe7d415dd50 2023-11-28T20:50:53.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.320+0000 7fe7caffd640 1 -- 172.21.15.45:0/562345311 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc00f800 con 0x7fe7d415dd50 2023-11-28T20:50:53.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.320+0000 7fe7caffd640 1 -- 172.21.15.45:0/562345311 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc00fbb0 con 0x7fe7d415dd50 2023-11-28T20:50:53.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.320+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d41f7130 con 0x7fe7d415dd50 2023-11-28T20:50:53.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.320+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d415a640 con 0x7fe7d415dd50 2023-11-28T20:50:53.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.321+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d41f7930 con 0x7fe7d415dd50 2023-11-28T20:50:53.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.322+0000 7fe7caffd640 1 -- 172.21.15.45:0/562345311 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fe7cc00b040 con 0x7fe7d415dd50 2023-11-28T20:50:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.323+0000 7fe7caffd640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe7b406af00 0x7fe7b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.323+0000 7fe7caffd640 1 -- 172.21.15.45:0/562345311 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6803+0+0 (secure 0 0 0) 0x7fe7cc08a620 con 0x7fe7d415dd50 2023-11-28T20:50:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.324+0000 7fe7d259c640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe7b406af00 0x7fe7b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.325+0000 7fe7d259c640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe7b406af00 0x7fe7b406d3c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4036000 tx=0x7fe7c4008030 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:53.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.328+0000 7fe7caffd640 1 -- 172.21.15.45:0/562345311 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7d41f7930 con 0x7fe7d415dd50 2023-11-28T20:50:53.342 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:53.340+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:50:53.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:53.531+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe7d406b2e0 con 0x7fe7d415dd50 2023-11-28T20:50:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.061+0000 7fe7caffd640 1 -- 172.21.15.45:0/562345311 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v92) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe7cc055080 con 0x7fe7d415dd50 2023-11-28T20:50:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T20:50:54.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.065+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe7b406af00 msgr2=0x7fe7b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:54.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.065+0000 7fe7d359e640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe7b406af00 0x7fe7b406d3c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4036000 tx=0x7fe7c4008030 comp rx=0 tx=0).stop 2023-11-28T20:50:54.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.065+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415dd50 msgr2=0x7fe7d4158470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:54.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.065+0000 7fe7d359e640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415dd50 0x7fe7d4158470 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00cd50 tx=0x7fe7cc00a650 comp rx=0 tx=0).stop 2023-11-28T20:50:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.065+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 shutdown_connections 2023-11-28T20:50:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.066+0000 7fe7d359e640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe7b406af00 0x7fe7b406d3c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.066+0000 7fe7d359e640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7d415dd50 0x7fe7d4158470 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.066+0000 7fe7d359e640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7d415d260 0x7fe7d4157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.066+0000 7fe7d359e640 1 --2- 172.21.15.45:0/562345311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7d40d1300 0x7fe7d415a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.066+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 >> 172.21.15.45:0/562345311 conn(0x7fe7d4099a40 msgr2=0x7fe7d40b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:54.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.066+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 shutdown_connections 2023-11-28T20:50:54.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.066+0000 7fe7d359e640 1 -- 172.21.15.45:0/562345311 wait complete. 2023-11-28T20:50:54.098 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T20:50:54.261 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:54.259+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:50:54.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.305+0000 7f603d0c9640 1 Processor -- start 2023-11-28T20:50:54.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.305+0000 7f603d0c9640 1 -- start start 2023-11-28T20:50:54.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.307+0000 7f603d0c9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f603806a500 0x7f603806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:54.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.307+0000 7f603d0c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603806ca60 con 0x7f6038157c10 2023-11-28T20:50:54.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.307+0000 7f603d0c9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603806aea0 con 0x7f603815ddd0 2023-11-28T20:50:54.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.307+0000 7f603d0c9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603815e930 con 0x7f603806a500 2023-11-28T20:50:54.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.307+0000 7f6036d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6038157c10 0x7f603806da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37754/0 (socket says 172.21.15.45:37754) 2023-11-28T20:50:54.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.307+0000 7f6036d9d640 1 -- 172.21.15.45:0/4074106303 learned_addr learned my addr 172.21.15.45:0/4074106303 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:54.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.307+0000 7f603659c640 1 --2- 172.21.15.45:0/4074106303 >> v2:172.21.15.190:3300/0 conn(0x7f603806a500 0x7f603806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:54.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.307+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153539430 0 0) 0x7f603806ca60 con 0x7f6038157c10 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6020003520 con 0x7f6038157c10 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942918218 0 0) 0x7f6020003520 con 0x7f6038157c10 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 >> v2:172.21.15.190:3300/0 conn(0x7f603806a500 msgr2=0x7f603806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 --2- 172.21.15.45:0/4074106303 >> v2:172.21.15.190:3300/0 conn(0x7f603806a500 0x7f603806a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 >> v1:172.21.15.190:6789/0 conn(0x7f603815ddd0 legacy=0x7f603815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603815eb40 con 0x7f6038157c10 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6028001670 con 0x7f6038157c10 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 >> v1:172.21.15.45:6789/0 conn(0x7f6038157c10 legacy=0x7f603806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6020003970 0x7f6020003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60200047c0 0x7f6020004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6020005180 0x7f602000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603815e930 con 0x7f6020003970 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602000c020 con 0x7f60200047c0 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602000c1e0 con 0x7f6020005180 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.308+0000 7f6036d9d640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6020003970 0x7f6020003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.309+0000 7f603659c640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60200047c0 0x7f6020004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.309+0000 7f603759e640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6020005180 0x7f602000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.309+0000 7f6036d9d640 1 -- 172.21.15.45:0/4074106303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6020005180 msgr2=0x7f602000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.309+0000 7f6036d9d640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6020005180 0x7f602000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.309+0000 7f6036d9d640 1 -- 172.21.15.45:0/4074106303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60200047c0 msgr2=0x7f6020004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.309+0000 7f6036d9d640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60200047c0 0x7f6020004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.309+0000 7f6036d9d640 1 -- 172.21.15.45:0/4074106303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6020003520 con 0x7f6020003970 2023-11-28T20:50:54.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.309+0000 7f603759e640 1 -- 172.21.15.45:0/4074106303 reap_dead start 2023-11-28T20:50:54.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.310+0000 7f6036d9d640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6020003970 0x7f6020003d90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f602800c190 tx=0x7f60200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c21003784a6d50c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:54.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.310+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028005ad0 con 0x7f6020003970 2023-11-28T20:50:54.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.310+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6028005c30 con 0x7f6020003970 2023-11-28T20:50:54.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.310+0000 7f6035d9b640 1 -- 172.21.15.45:0/4074106303 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602802cae0 con 0x7f6020003970 2023-11-28T20:50:54.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.311+0000 7f603d0c9640 1 -- 172.21.15.45:0/4074106303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6020003970 msgr2=0x7f6020003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:54.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.311+0000 7f603d0c9640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6020003970 0x7f6020003d90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f602800c190 tx=0x7f60200057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.311+0000 7f603d0c9640 1 -- 172.21.15.45:0/4074106303 shutdown_connections 2023-11-28T20:50:54.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.311+0000 7f603d0c9640 1 --2- 172.21.15.45:0/4074106303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6020003970 0x7f6020003d90 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.311+0000 7f603d0c9640 1 -- 172.21.15.45:0/4074106303 >> 172.21.15.45:0/4074106303 conn(0x7f60380ced60 msgr2=0x7f60380cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:54.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.311+0000 7f603d0c9640 1 -- 172.21.15.45:0/4074106303 shutdown_connections 2023-11-28T20:50:54.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.311+0000 7f603d0c9640 1 -- 172.21.15.45:0/4074106303 wait complete. 2023-11-28T20:50:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.313+0000 7f603d0c9640 1 Processor -- start 2023-11-28T20:50:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.313+0000 7f603d0c9640 1 -- start start 2023-11-28T20:50:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.314+0000 7f603d0c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6020003970 0x7f60380ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.314+0000 7f603d0c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60380ce100 0x7f60380ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.314+0000 7f603d0c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60380c1ac0 0x7f6038159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.314+0000 7f603d0c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380ac460 con 0x7f60380ce100 2023-11-28T20:50:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.314+0000 7f603d0c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380ce590 con 0x7f60380c1ac0 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.314+0000 7f603d0c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60380abe10 con 0x7f6020003970 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.314+0000 7f603659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60380ce100 0x7f60380ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f603659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60380ce100 0x7f60380ac8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48676/0 (socket says 172.21.15.45:48676) 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f603659c640 1 -- 172.21.15.45:0/3629866041 learned_addr learned my addr 172.21.15.45:0/3629866041 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f603659c640 1 -- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6020003970 msgr2=0x7f60380ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f603759e640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60380c1ac0 0x7f6038159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f6036d9d640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6020003970 0x7f60380ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f603659c640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6020003970 0x7f60380ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f603659c640 1 -- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60380c1ac0 msgr2=0x7f6038159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f603659c640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60380c1ac0 0x7f6038159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f603659c640 1 -- 172.21.15.45:0/3629866041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60380ac0f0 con 0x7f60380ce100 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.315+0000 7f603659c640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60380ce100 0x7f60380ac8f0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f602c0116d0 tx=0x7f602c011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.316+0000 7f601f7fe640 1 -- 172.21.15.45:0/3629866041 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c00f9c0 con 0x7f60380ce100 2023-11-28T20:50:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.316+0000 7f601f7fe640 1 -- 172.21.15.45:0/3629866041 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c00fb20 con 0x7f60380ce100 2023-11-28T20:50:54.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.316+0000 7f601f7fe640 1 -- 172.21.15.45:0/3629866041 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c00fe90 con 0x7f60380ce100 2023-11-28T20:50:54.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.316+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60380ce870 con 0x7f60380ce100 2023-11-28T20:50:54.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.316+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60380c14e0 con 0x7f60380ce100 2023-11-28T20:50:54.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.317+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60380ad010 con 0x7f60380ce100 2023-11-28T20:50:54.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.319+0000 7f601f7fe640 1 -- 172.21.15.45:0/3629866041 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f602c015460 con 0x7f60380ce100 2023-11-28T20:50:54.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.320+0000 7f601f7fe640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f602406af00 0x7f602406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:54.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.320+0000 7f601f7fe640 1 -- 172.21.15.45:0/3629866041 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f602c08a140 con 0x7f60380ce100 2023-11-28T20:50:54.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.320+0000 7f6036d9d640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f602406af00 0x7f602406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:54.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.321+0000 7f6036d9d640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f602406af00 0x7f602406d3c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6028034000 tx=0x7f6028009030 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:54.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.323+0000 7f601f7fe640 1 -- 172.21.15.45:0/3629866041 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f602c054f80 con 0x7f60380ce100 2023-11-28T20:50:54.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:54.527+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f60380733d0 con 0x7f60380ce100 2023-11-28T20:50:55.194 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:55.191+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:50:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.327+0000 7f601f7fe640 1 -- 172.21.15.45:0/3629866041 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v93) v1 ==== 169+0+0 (secure 0 0 0) 0x7f602c0585a0 con 0x7f60380ce100 2023-11-28T20:50:55.329 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T20:50:55.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f602406af00 msgr2=0x7f602406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:55.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f602406af00 0x7f602406d3c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6028034000 tx=0x7f6028009030 comp rx=0 tx=0).stop 2023-11-28T20:50:55.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60380ce100 msgr2=0x7f60380ac8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:55.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60380ce100 0x7f60380ac8f0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f602c0116d0 tx=0x7f602c011b90 comp rx=0 tx=0).stop 2023-11-28T20:50:55.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 shutdown_connections 2023-11-28T20:50:55.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f602406af00 0x7f602406d3c0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6028034000 tx=0x7f6028009030 comp rx=0 tx=0).stop 2023-11-28T20:50:55.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60380c1ac0 0x7f6038159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60380ce100 0x7f60380ac8f0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 --2- 172.21.15.45:0/3629866041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6020003970 0x7f60380ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.335+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 >> 172.21.15.45:0/3629866041 conn(0x7f60380ced60 msgr2=0x7f6038155690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:55.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.336+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 shutdown_connections 2023-11-28T20:50:55.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.336+0000 7f603d0c9640 1 -- 172.21.15.45:0/3629866041 wait complete. 2023-11-28T20:50:55.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T20:50:55.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.574+0000 7f1c6ea09640 1 Processor -- start 2023-11-28T20:50:55.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.574+0000 7f1c6ea09640 1 -- start start 2023-11-28T20:50:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.575+0000 7f1c6ea09640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1c68157c10 0x7f1c6806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.575+0000 7f1c6ea09640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6815e4e0 0x7f1c6806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.575+0000 7f1c6ea09640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6806ca60 con 0x7f1c6806a500 2023-11-28T20:50:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.575+0000 7f1c6ea09640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6815eb40 con 0x7f1c68157c10 2023-11-28T20:50:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.575+0000 7f1c6ea09640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c68157ff0 con 0x7f1c6815e4e0 2023-11-28T20:50:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.576+0000 7f1c67fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1c68157c10 0x7f1c6806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.576+0000 7f1c6cfa6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6815e4e0 0x7f1c6806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.576+0000 7f1c67fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1c68157c10 0x7f1c6806da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:53558/0 (socket says 172.21.15.45:53558) 2023-11-28T20:50:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.576+0000 7f1c67fff640 1 -- 172.21.15.45:0/120528869 learned_addr learned my addr 172.21.15.45:0/120528869 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.577+0000 7f1c6cfa6640 1 -- 172.21.15.45:0/120528869 >> v2:172.21.15.190:3300/0 conn(0x7f1c68157c10 msgr2=0x7f1c6806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.577+0000 7f1c6cfa6640 1 --2- 172.21.15.45:0/120528869 >> v2:172.21.15.190:3300/0 conn(0x7f1c68157c10 0x7f1c6806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.577+0000 7f1c6cfa6640 1 -- 172.21.15.45:0/120528869 >> v1:172.21.15.190:6789/0 conn(0x7f1c6806a500 legacy=0x7f1c6815ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.577+0000 7f1c6cfa6640 1 -- 172.21.15.45:0/120528869 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6806aea0 con 0x7f1c6815e4e0 2023-11-28T20:50:55.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.577+0000 7f1c67fff640 1 --2- 172.21.15.45:0/120528869 >> v2:172.21.15.190:3300/0 conn(0x7f1c68157c10 0x7f1c6806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:50:55.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.578+0000 7f1c6cfa6640 1 --2- 172.21.15.45:0/120528869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6815e4e0 0x7f1c6806a960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c600097a0 tx=0x7f1c6002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b774c326c7a027a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:55.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.579+0000 7f1c677fe640 1 -- 172.21.15.45:0/120528869 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6002d040 con 0x7f1c6815e4e0 2023-11-28T20:50:55.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.579+0000 7f1c677fe640 1 -- 172.21.15.45:0/120528869 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c60031810 con 0x7f1c6815e4e0 2023-11-28T20:50:55.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.579+0000 7f1c677fe640 1 -- 172.21.15.45:0/120528869 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60031ba0 con 0x7f1c6815e4e0 2023-11-28T20:50:55.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.580+0000 7f1c6ea09640 1 -- 172.21.15.45:0/120528869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6815e4e0 msgr2=0x7f1c6806a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:55.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.580+0000 7f1c6ea09640 1 --2- 172.21.15.45:0/120528869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6815e4e0 0x7f1c6806a960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c600097a0 tx=0x7f1c6002b8b0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.580+0000 7f1c6ea09640 1 -- 172.21.15.45:0/120528869 shutdown_connections 2023-11-28T20:50:55.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.580+0000 7f1c6ea09640 1 --2- 172.21.15.45:0/120528869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6815e4e0 0x7f1c6806a960 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.580+0000 7f1c6ea09640 1 --2- 172.21.15.45:0/120528869 >> v2:172.21.15.190:3300/0 conn(0x7f1c68157c10 0x7f1c6806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.580+0000 7f1c6ea09640 1 -- 172.21.15.45:0/120528869 >> 172.21.15.45:0/120528869 conn(0x7f1c680ced60 msgr2=0x7f1c680cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:55.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.580+0000 7f1c6ea09640 1 -- 172.21.15.45:0/120528869 shutdown_connections 2023-11-28T20:50:55.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.580+0000 7f1c6ea09640 1 -- 172.21.15.45:0/120528869 wait complete. 2023-11-28T20:50:55.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.582+0000 7f1c6ea09640 1 Processor -- start 2023-11-28T20:50:55.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.582+0000 7f1c6ea09640 1 -- start start 2023-11-28T20:50:55.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.583+0000 7f1c6ea09640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6806a500 0x7f1c680cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:55.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.583+0000 7f1c6ea09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c68157c10 0x7f1c680c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.583+0000 7f1c6ea09640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c6815e4e0 0x7f1c680c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.583+0000 7f1c6ea09640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c680c2df0 con 0x7f1c68157c10 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.583+0000 7f1c6ea09640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c680ad570 con 0x7f1c6815e4e0 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.583+0000 7f1c6ea09640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c68157680 con 0x7f1c6806a500 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.583+0000 7f1c5f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c68157c10 0x7f1c680c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.583+0000 7f1c5f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c68157c10 0x7f1c680c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48682/0 (socket says 172.21.15.45:48682) 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.583+0000 7f1c5f7fe640 1 -- 172.21.15.45:0/2301412630 learned_addr learned my addr 172.21.15.45:0/2301412630 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c5f7fe640 1 -- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6806a500 msgr2=0x7f1c680cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c67fff640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6806a500 0x7f1c680cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c6cfa6640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c6815e4e0 0x7f1c680c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c5f7fe640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6806a500 0x7f1c680cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c5f7fe640 1 -- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c6815e4e0 msgr2=0x7f1c680c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c5f7fe640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c6815e4e0 0x7f1c680c6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c5f7fe640 1 -- 172.21.15.45:0/2301412630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c580094d0 con 0x7f1c68157c10 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c5f7fe640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c68157c10 0x7f1c680c4cd0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000c870 tx=0x7f1c5000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c657fa640 1 -- 172.21.15.45:0/2301412630 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50003b70 con 0x7f1c68157c10 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c657fa640 1 -- 172.21.15.45:0/2301412630 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c50003cf0 con 0x7f1c68157c10 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c657fa640 1 -- 172.21.15.45:0/2301412630 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5000aa90 con 0x7f1c68157c10 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.584+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c60009400 con 0x7f1c68157c10 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.585+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c680ad870 con 0x7f1c68157c10 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.586+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c680c2f30 con 0x7f1c68157c10 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.587+0000 7f1c657fa640 1 -- 172.21.15.45:0/2301412630 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f1c5000abf0 con 0x7f1c68157c10 2023-11-28T20:50:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.588+0000 7f1c657fa640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f1c4806aed0 0x7f1c4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.588+0000 7f1c657fa640 1 -- 172.21.15.45:0/2301412630 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f1c500895f0 con 0x7f1c68157c10 2023-11-28T20:50:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.588+0000 7f1c67fff640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f1c4806aed0 0x7f1c4806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.589+0000 7f1c67fff640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f1c4806aed0 0x7f1c4806d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1c68156e00 tx=0x7f1c58008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:55.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.592+0000 7f1c657fa640 1 -- 172.21.15.45:0/2301412630 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c680c2f30 con 0x7f1c68157c10 2023-11-28T20:50:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.787+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f1c680c2f30 con 0x7f1c68157c10 2023-11-28T20:50:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.793+0000 7f1c657fa640 1 -- 172.21.15.45:0/2301412630 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v25) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1c680c2f30 con 0x7f1c68157c10 2023-11-28T20:50:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f1c4806aed0 msgr2=0x7f1c4806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f1c4806aed0 0x7f1c4806d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1c68156e00 tx=0x7f1c58008040 comp rx=0 tx=0).stop 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c68157c10 msgr2=0x7f1c680c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c68157c10 0x7f1c680c4cd0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000c870 tx=0x7f1c5000cd30 comp rx=0 tx=0).stop 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 shutdown_connections 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f1c4806aed0 0x7f1c4806d390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c6815e4e0 0x7f1c680c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c68157c10 0x7f1c680c4cd0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 --2- 172.21.15.45:0/2301412630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c6806a500 0x7f1c680cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 >> 172.21.15.45:0/2301412630 conn(0x7f1c680ced60 msgr2=0x7f1c6815a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 shutdown_connections 2023-11-28T20:50:55.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:55.796+0000 7f1c6ea09640 1 -- 172.21.15.45:0/2301412630 wait complete. 2023-11-28T20:50:55.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T20:50:56.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.040+0000 7fe42e75c640 1 Processor -- start 2023-11-28T20:50:56.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.040+0000 7fe42e75c640 1 -- start start 2023-11-28T20:50:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.041+0000 7fe42e75c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe42815b080 0x7fe4280ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.041+0000 7fe42e75c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe4280abd70 0x7fe4280718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.042+0000 7fe42e75c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 0x7fe42815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.042+0000 7fe42e75c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4280722b0 con 0x7fe4280abd70 2023-11-28T20:50:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.042+0000 7fe42e75c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4280ac270 con 0x7fe42815b080 2023-11-28T20:50:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.042+0000 7fe42e75c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42815b830 con 0x7fe428071e00 2023-11-28T20:50:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.042+0000 7fe42cf59640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe4280abd70 0x7fe4280718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.042+0000 7fe42cf59640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe4280abd70 0x7fe4280718c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48690/0 (socket says 172.21.15.45:48690) 2023-11-28T20:50:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.042+0000 7fe42cf59640 1 -- 172.21.15.45:0/4107834989 learned_addr learned my addr 172.21.15.45:0/4107834989 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.042+0000 7fe42d75a640 1 --2- 172.21.15.45:0/4107834989 >> v2:172.21.15.190:3300/0 conn(0x7fe42815b080 0x7fe4280ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.042+0000 7fe42df5b640 1 --2- 172.21.15.45:0/4107834989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 0x7fe42815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.043+0000 7fe42cf59640 1 -- 172.21.15.45:0/4107834989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 msgr2=0x7fe42815eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.043+0000 7fe42cf59640 1 --2- 172.21.15.45:0/4107834989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 0x7fe42815eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.043+0000 7fe42cf59640 1 -- 172.21.15.45:0/4107834989 >> v2:172.21.15.190:3300/0 conn(0x7fe42815b080 msgr2=0x7fe4280ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.043+0000 7fe42cf59640 1 --2- 172.21.15.45:0/4107834989 >> v2:172.21.15.190:3300/0 conn(0x7fe42815b080 0x7fe4280ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.043+0000 7fe42cf59640 1 -- 172.21.15.45:0/4107834989 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42815bb10 con 0x7fe4280abd70 2023-11-28T20:50:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.043+0000 7fe42d75a640 1 --2- 172.21.15.45:0/4107834989 >> v2:172.21.15.190:3300/0 conn(0x7fe42815b080 0x7fe4280ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:50:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.044+0000 7fe42cf59640 1 --2- 172.21.15.45:0/4107834989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4280abd70 0x7fe4280718c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe42806bfa0 tx=0x7fe41802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ec3aaf86517ab6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:56.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.044+0000 7fe41ffff640 1 -- 172.21.15.45:0/4107834989 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe418031870 con 0x7fe4280abd70 2023-11-28T20:50:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.044+0000 7fe41ffff640 1 -- 172.21.15.45:0/4107834989 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4180319d0 con 0x7fe4280abd70 2023-11-28T20:50:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.044+0000 7fe41ffff640 1 -- 172.21.15.45:0/4107834989 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe418031d60 con 0x7fe4280abd70 2023-11-28T20:50:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.045+0000 7fe42e75c640 1 -- 172.21.15.45:0/4107834989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4280abd70 msgr2=0x7fe4280718c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.045+0000 7fe42e75c640 1 --2- 172.21.15.45:0/4107834989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4280abd70 0x7fe4280718c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe42806bfa0 tx=0x7fe41802b910 comp rx=0 tx=0).stop 2023-11-28T20:50:56.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.045+0000 7fe42e75c640 1 -- 172.21.15.45:0/4107834989 shutdown_connections 2023-11-28T20:50:56.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.045+0000 7fe42e75c640 1 --2- 172.21.15.45:0/4107834989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 0x7fe42815eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.045+0000 7fe42e75c640 1 --2- 172.21.15.45:0/4107834989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4280abd70 0x7fe4280718c0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.045+0000 7fe42e75c640 1 --2- 172.21.15.45:0/4107834989 >> v2:172.21.15.190:3300/0 conn(0x7fe42815b080 0x7fe4280ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.045+0000 7fe42e75c640 1 -- 172.21.15.45:0/4107834989 >> 172.21.15.45:0/4107834989 conn(0x7fe4280cecd0 msgr2=0x7fe4280cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:56.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.045+0000 7fe42e75c640 1 -- 172.21.15.45:0/4107834989 shutdown_connections 2023-11-28T20:50:56.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.045+0000 7fe42e75c640 1 -- 172.21.15.45:0/4107834989 wait complete. 2023-11-28T20:50:56.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.047+0000 7fe42e75c640 1 Processor -- start 2023-11-28T20:50:56.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.048+0000 7fe42e75c640 1 -- start start 2023-11-28T20:50:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42e75c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 0x7fe428142620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42e75c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4280abd70 0x7fe428153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42e75c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe42815b080 0x7fe428159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42e75c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe428150bf0 con 0x7fe42815b080 2023-11-28T20:50:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42e75c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281f68c0 con 0x7fe4280abd70 2023-11-28T20:50:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42e75c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281f6ba0 con 0x7fe428071e00 2023-11-28T20:50:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42df5b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe42815b080 0x7fe428159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42df5b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe42815b080 0x7fe428159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48700/0 (socket says 172.21.15.45:48700) 2023-11-28T20:50:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42df5b640 1 -- 172.21.15.45:0/114452161 learned_addr learned my addr 172.21.15.45:0/114452161 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:56.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.049+0000 7fe42d75a640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 0x7fe428142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe42cf59640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4280abd70 0x7fe428153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe42df5b640 1 -- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 msgr2=0x7fe428142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe42df5b640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 0x7fe428142620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe42df5b640 1 -- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4280abd70 msgr2=0x7fe428153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe42df5b640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4280abd70 0x7fe428153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe42df5b640 1 -- 172.21.15.45:0/114452161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4180094d0 con 0x7fe42815b080 2023-11-28T20:50:56.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe42cf59640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4280abd70 0x7fe428153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:50:56.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe42df5b640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe42815b080 0x7fe428159880 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe420007e60 tx=0x7fe42000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:56.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe41dffb640 1 -- 172.21.15.45:0/114452161 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe420014620 con 0x7fe42815b080 2023-11-28T20:50:56.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe41dffb640 1 -- 172.21.15.45:0/114452161 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42000ee90 con 0x7fe42815b080 2023-11-28T20:50:56.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4281f6e80 con 0x7fe42815b080 2023-11-28T20:50:56.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.050+0000 7fe41dffb640 1 -- 172.21.15.45:0/114452161 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4200149e0 con 0x7fe42815b080 2023-11-28T20:50:56.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.051+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4281f64e0 con 0x7fe42815b080 2023-11-28T20:50:56.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.052+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4281f73a0 con 0x7fe42815b080 2023-11-28T20:50:56.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.053+0000 7fe41dffb640 1 -- 172.21.15.45:0/114452161 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fe420014b40 con 0x7fe42815b080 2023-11-28T20:50:56.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.054+0000 7fe41dffb640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe40c06aed0 0x7fe40c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.054+0000 7fe41dffb640 1 -- 172.21.15.45:0/114452161 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fe42008a110 con 0x7fe42815b080 2023-11-28T20:50:56.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.054+0000 7fe42d75a640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe40c06aed0 0x7fe40c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.056+0000 7fe42d75a640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe40c06aed0 0x7fe40c06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe424006fd0 tx=0x7fe424008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:56.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.058+0000 7fe41dffb640 1 -- 172.21.15.45:0/114452161 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe420055070 con 0x7fe42815b080 2023-11-28T20:50:56.122 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:56.119+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:50:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.255+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe42809d0f0 con 0x7fe42815b080 2023-11-28T20:50:56.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.261+0000 7fe41dffb640 1 -- 172.21.15.45:0/114452161 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe420058690 con 0x7fe42815b080 2023-11-28T20:50:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.263+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe40c06aed0 msgr2=0x7fe40c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.263+0000 7fe42e75c640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe40c06aed0 0x7fe40c06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe424006fd0 tx=0x7fe424008040 comp rx=0 tx=0).stop 2023-11-28T20:50:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.263+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe42815b080 msgr2=0x7fe428159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.263+0000 7fe42e75c640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe42815b080 0x7fe428159880 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe420007e60 tx=0x7fe42000e4b0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.264+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 shutdown_connections 2023-11-28T20:50:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.264+0000 7fe42e75c640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe40c06aed0 0x7fe40c06d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.264+0000 7fe42e75c640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe42815b080 0x7fe428159880 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.264+0000 7fe42e75c640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4280abd70 0x7fe428153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.264+0000 7fe42e75c640 1 --2- 172.21.15.45:0/114452161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe428071e00 0x7fe428142620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.264+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 >> 172.21.15.45:0/114452161 conn(0x7fe4280cecd0 msgr2=0x7fe4280ce460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:56.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.264+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 shutdown_connections 2023-11-28T20:50:56.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.264+0000 7fe42e75c640 1 -- 172.21.15.45:0/114452161 wait complete. 2023-11-28T20:50:56.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T20:50:56.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.507+0000 7f5e765ae640 1 Processor -- start 2023-11-28T20:50:56.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.508+0000 7f5e765ae640 1 -- start start 2023-11-28T20:50:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.510+0000 7f5e765ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700d1fe0 0x7f5e700d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.510+0000 7f5e765ae640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e700b2930 con 0x7f5e700cf940 2023-11-28T20:50:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.510+0000 7f5e765ae640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e700b2650 con 0x7f5e700cfd20 2023-11-28T20:50:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.510+0000 7f5e765ae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700ce240 con 0x7f5e700d1fe0 2023-11-28T20:50:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.510+0000 7f5e755ac640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5e700cf940 0x7f5e70161800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37758/0 (socket says 172.21.15.45:37758) 2023-11-28T20:50:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.510+0000 7f5e755ac640 1 -- 172.21.15.45:0/216199582 learned_addr learned my addr 172.21.15.45:0/216199582 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.510+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479056731 0 0) 0x7f5e700b2930 con 0x7f5e700cf940 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e75dad640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700d1fe0 0x7f5e700d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5c003520 con 0x7f5e700cf940 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1444190104 0 0) 0x7f5e5c003520 con 0x7f5e700cf940 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700d1fe0 msgr2=0x7f5e700d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700d1fe0 0x7f5e700d2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 >> v1:172.21.15.190:6789/0 conn(0x7f5e700cfd20 legacy=0x7f5e700d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e700ce520 con 0x7f5e700cf940 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5e6c001670 con 0x7f5e700cf940 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 >> v1:172.21.15.45:6789/0 conn(0x7f5e700cf940 legacy=0x7f5e70161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c0039a0 0x7f5e5c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700d1fe0 0x7f5e5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c004d00 0x7f5e5c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700ce240 con 0x7f5e5c004d00 2023-11-28T20:50:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c00bbc0 con 0x7f5e5c0039a0 2023-11-28T20:50:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.511+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c00bd80 con 0x7f5e700d1fe0 2023-11-28T20:50:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.512+0000 7f5e75dad640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c004d00 0x7f5e5c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.512+0000 7f5e755ac640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c0039a0 0x7f5e5c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.512+0000 7f5e74dab640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700d1fe0 0x7f5e5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.512+0000 7f5e75dad640 1 -- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700d1fe0 msgr2=0x7f5e5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.512+0000 7f5e75dad640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700d1fe0 0x7f5e5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.512+0000 7f5e75dad640 1 -- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c0039a0 msgr2=0x7f5e5c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.512+0000 7f5e75dad640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c0039a0 0x7f5e5c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.512+0000 7f5e75dad640 1 -- 172.21.15.45:0/216199582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c003520 con 0x7f5e5c004d00 2023-11-28T20:50:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.514+0000 7f5e75dad640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c004d00 0x7f5e5c005160 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009fc0 tx=0x7f5e5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657789ac0002800b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.514+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6802cc20 con 0x7f5e5c004d00 2023-11-28T20:50:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.514+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6800f040 con 0x7f5e5c004d00 2023-11-28T20:50:56.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.514+0000 7f5e67fff640 1 -- 172.21.15.45:0/216199582 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68034440 con 0x7f5e5c004d00 2023-11-28T20:50:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.514+0000 7f5e765ae640 1 -- 172.21.15.45:0/216199582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c004d00 msgr2=0x7f5e5c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.514+0000 7f5e765ae640 1 --2- 172.21.15.45:0/216199582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c004d00 0x7f5e5c005160 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009fc0 tx=0x7f5e5c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.514+0000 7f5e75dad640 1 -- 172.21.15.45:0/216199582 reap_dead start 2023-11-28T20:50:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.515+0000 7f5e765ae640 1 -- 172.21.15.45:0/216199582 shutdown_connections 2023-11-28T20:50:56.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.515+0000 7f5e765ae640 1 -- 172.21.15.45:0/216199582 >> 172.21.15.45:0/216199582 conn(0x7f5e7006a500 msgr2=0x7f5e7006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.515+0000 7f5e765ae640 1 -- 172.21.15.45:0/216199582 shutdown_connections 2023-11-28T20:50:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.515+0000 7f5e765ae640 1 -- 172.21.15.45:0/216199582 wait complete. 2023-11-28T20:50:56.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.517+0000 7f5e765ae640 1 Processor -- start 2023-11-28T20:50:56.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.517+0000 7f5e765ae640 1 -- start start 2023-11-28T20:50:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.518+0000 7f5e765ae640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c004d00 0x7f5e701514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.518+0000 7f5e765ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e700b7520 0x7f5e700b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.518+0000 7f5e765ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700c8430 0x7f5e700c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.518+0000 7f5e765ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700b7ee0 con 0x7f5e700b7520 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.518+0000 7f5e765ae640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700c6a30 con 0x7f5e5c004d00 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.518+0000 7f5e765ae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700c8df0 con 0x7f5e700c8430 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.518+0000 7f5e74dab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e700b7520 0x7f5e700b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e74dab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e700b7520 0x7f5e700b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48724/0 (socket says 172.21.15.45:48724) 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e74dab640 1 -- 172.21.15.45:0/653545551 learned_addr learned my addr 172.21.15.45:0/653545551 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e755ac640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c004d00 0x7f5e701514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e75dad640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700c8430 0x7f5e700c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e74dab640 1 -- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700c8430 msgr2=0x7f5e700c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e74dab640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700c8430 0x7f5e700c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e74dab640 1 -- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c004d00 msgr2=0x7f5e701514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e74dab640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c004d00 0x7f5e701514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e74dab640 1 -- 172.21.15.45:0/653545551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e701519e0 con 0x7f5e700b7520 2023-11-28T20:50:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.519+0000 7f5e74dab640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e700b7520 0x7f5e700b79a0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000e550 tx=0x7f5e6000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.520+0000 7f5e65ffb640 1 -- 172.21.15.45:0/653545551 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e600052e0 con 0x7f5e700b7520 2023-11-28T20:50:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.520+0000 7f5e65ffb640 1 -- 172.21.15.45:0/653545551 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e60005440 con 0x7f5e700b7520 2023-11-28T20:50:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.520+0000 7f5e65ffb640 1 -- 172.21.15.45:0/653545551 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60003810 con 0x7f5e700b7520 2023-11-28T20:50:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.520+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e700c6c80 con 0x7f5e700b7520 2023-11-28T20:50:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.520+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e700c7240 con 0x7f5e700b7520 2023-11-28T20:50:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.521+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e70071b30 con 0x7f5e700b7520 2023-11-28T20:50:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.522+0000 7f5e65ffb640 1 -- 172.21.15.45:0/653545551 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f5e60003970 con 0x7f5e700b7520 2023-11-28T20:50:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.523+0000 7f5e65ffb640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5e5406aed0 0x7f5e5406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.524+0000 7f5e65ffb640 1 -- 172.21.15.45:0/653545551 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f5e600895a0 con 0x7f5e700b7520 2023-11-28T20:50:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.524+0000 7f5e755ac640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5e5406aed0 0x7f5e5406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.525+0000 7f5e755ac640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5e5406aed0 0x7f5e5406d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c001bc0 tx=0x7f5e6c009040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:56.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.528+0000 7f5e65ffb640 1 -- 172.21.15.45:0/653545551 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e60054470 con 0x7f5e700b7520 2023-11-28T20:50:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.725+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f5e7004e920 con 0x7f5e700b7520 2023-11-28T20:50:56.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.732+0000 7f5e65ffb640 1 -- 172.21.15.45:0/653545551 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v27) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5e60057a90 con 0x7f5e700b7520 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.735+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5e5406aed0 msgr2=0x7f5e5406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.735+0000 7f5e765ae640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5e5406aed0 0x7f5e5406d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c001bc0 tx=0x7f5e6c009040 comp rx=0 tx=0).stop 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.735+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e700b7520 msgr2=0x7f5e700b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.735+0000 7f5e765ae640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e700b7520 0x7f5e700b79a0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000e550 tx=0x7f5e6000ea10 comp rx=0 tx=0).stop 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.736+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 shutdown_connections 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.736+0000 7f5e765ae640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5e5406aed0 0x7f5e5406d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.736+0000 7f5e765ae640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e700c8430 0x7f5e700c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.736+0000 7f5e765ae640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e700b7520 0x7f5e700b79a0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.736+0000 7f5e765ae640 1 --2- 172.21.15.45:0/653545551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c004d00 0x7f5e701514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.736+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 >> 172.21.15.45:0/653545551 conn(0x7f5e7006a500 msgr2=0x7f5e700b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.736+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 shutdown_connections 2023-11-28T20:50:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.736+0000 7f5e765ae640 1 -- 172.21.15.45:0/653545551 wait complete. 2023-11-28T20:50:56.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-11-28T20:50:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.983+0000 7f347189b640 1 Processor -- start 2023-11-28T20:50:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.984+0000 7f347189b640 1 -- start start 2023-11-28T20:50:56.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.986+0000 7f347189b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c1434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.986+0000 7f347189b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f346c0703b0 0x7f346c070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.986+0000 7f347189b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c161df0 con 0x7f346c1402f0 2023-11-28T20:50:56.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.986+0000 7f347189b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346c143a00 con 0x7f346c0703b0 2023-11-28T20:50:56.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.986+0000 7f347189b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c1612d0 con 0x7f346c160e70 2023-11-28T20:50:56.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.986+0000 7f346a7fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c1434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.986+0000 7f346b7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f346c0703b0 0x7f346c070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.987+0000 7f346a7fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c1434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:50538/0 (socket says 172.21.15.45:50538) 2023-11-28T20:50:56.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.987+0000 7f346a7fc640 1 -- 172.21.15.45:0/1664100909 learned_addr learned my addr 172.21.15.45:0/1664100909 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:56.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.987+0000 7f346b7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f346c0703b0 0x7f346c070840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:53626/0 (socket says 172.21.15.45:53626) 2023-11-28T20:50:56.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.987+0000 7f346a7fc640 1 -- 172.21.15.45:0/1664100909 >> v2:172.21.15.190:3300/0 conn(0x7f346c0703b0 msgr2=0x7f346c070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:50:56.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.987+0000 7f346a7fc640 1 --2- 172.21.15.45:0/1664100909 >> v2:172.21.15.190:3300/0 conn(0x7f346c0703b0 0x7f346c070840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.987+0000 7f346a7fc640 1 -- 172.21.15.45:0/1664100909 >> v1:172.21.15.190:6789/0 conn(0x7f346c1402f0 legacy=0x7f346c1409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.987+0000 7f346a7fc640 1 -- 172.21.15.45:0/1664100909 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c1406d0 con 0x7f346c160e70 2023-11-28T20:50:56.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.987+0000 7f346b7fe640 1 --2- 172.21.15.45:0/1664100909 >> v2:172.21.15.190:3300/0 conn(0x7f346c0703b0 0x7f346c070840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:50:56.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.989+0000 7f346a7fc640 1 --2- 172.21.15.45:0/1664100909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c1434c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3454009870 tx=0x7f345402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef210187036eaa1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:56.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.989+0000 7f3469ffb640 1 -- 172.21.15.45:0/1664100909 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454031640 con 0x7f346c160e70 2023-11-28T20:50:56.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.989+0000 7f3469ffb640 1 -- 172.21.15.45:0/1664100909 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34540317a0 con 0x7f346c160e70 2023-11-28T20:50:56.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.989+0000 7f3469ffb640 1 -- 172.21.15.45:0/1664100909 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454031b30 con 0x7f346c160e70 2023-11-28T20:50:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.990+0000 7f347189b640 1 -- 172.21.15.45:0/1664100909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 msgr2=0x7f346c1434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.990+0000 7f347189b640 1 --2- 172.21.15.45:0/1664100909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c1434c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3454009870 tx=0x7f345402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.990+0000 7f347189b640 1 -- 172.21.15.45:0/1664100909 shutdown_connections 2023-11-28T20:50:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.990+0000 7f347189b640 1 --2- 172.21.15.45:0/1664100909 >> v2:172.21.15.190:3300/0 conn(0x7f346c0703b0 0x7f346c070840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.990+0000 7f347189b640 1 --2- 172.21.15.45:0/1664100909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c1434c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.990+0000 7f347189b640 1 -- 172.21.15.45:0/1664100909 >> 172.21.15.45:0/1664100909 conn(0x7f346c06e7c0 msgr2=0x7f346c06ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.991+0000 7f347189b640 1 -- 172.21.15.45:0/1664100909 shutdown_connections 2023-11-28T20:50:56.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.991+0000 7f347189b640 1 -- 172.21.15.45:0/1664100909 wait complete. 2023-11-28T20:50:56.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.992+0000 7f347189b640 1 Processor -- start 2023-11-28T20:50:56.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.993+0000 7f347189b640 1 -- start start 2023-11-28T20:50:56.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f347189b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f346c0703b0 0x7f346c0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f347189b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346c1402f0 0x7f346c0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f347189b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f347189b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0ce3d0 con 0x7f346c1402f0 2023-11-28T20:50:56.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f347189b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0d0b20 con 0x7f346c0703b0 2023-11-28T20:50:56.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f347189b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0cc9f0 con 0x7f346c160e70 2023-11-28T20:50:56.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346c1402f0 0x7f346c0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346c1402f0 0x7f346c0cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48732/0 (socket says 172.21.15.45:48732) 2023-11-28T20:50:56.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346a7fc640 1 -- 172.21.15.45:0/377769939 learned_addr learned my addr 172.21.15.45:0/377769939 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346b7fe640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346a7fc640 1 -- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 msgr2=0x7f346c141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346a7fc640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c141ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346a7fc640 1 -- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f346c0703b0 msgr2=0x7f346c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346affd640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f346c0703b0 0x7f346c0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346a7fc640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f346c0703b0 0x7f346c0d05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.994+0000 7f346a7fc640 1 -- 172.21.15.45:0/377769939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3464009400 con 0x7f346c1402f0 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.995+0000 7f346a7fc640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346c1402f0 0x7f346c0cc4b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3454009870 tx=0x7f34540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.995+0000 7f3463fff640 1 -- 172.21.15.45:0/377769939 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345403a930 con 0x7f346c1402f0 2023-11-28T20:50:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.995+0000 7f3463fff640 1 -- 172.21.15.45:0/377769939 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345403aa90 con 0x7f346c1402f0 2023-11-28T20:50:56.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.995+0000 7f3463fff640 1 -- 172.21.15.45:0/377769939 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345403ae40 con 0x7f346c1402f0 2023-11-28T20:50:56.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.995+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34540094d0 con 0x7f346c1402f0 2023-11-28T20:50:56.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.995+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346c0cab00 con 0x7f346c1402f0 2023-11-28T20:50:56.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.997+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346c0ce560 con 0x7f346c1402f0 2023-11-28T20:50:56.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.998+0000 7f3463fff640 1 -- 172.21.15.45:0/377769939 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f345402f050 con 0x7f346c1402f0 2023-11-28T20:50:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.998+0000 7f3463fff640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f344c06af00 0x7f344c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:50:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.999+0000 7f3463fff640 1 -- 172.21.15.45:0/377769939 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f34540ab7c0 con 0x7f346c1402f0 2023-11-28T20:50:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:56.999+0000 7f346affd640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f344c06af00 0x7f344c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:50:57.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.001+0000 7f346affd640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f344c06af00 0x7f344c06d3c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f345c007fd0 tx=0x7f345c009040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:50:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.003+0000 7f3463fff640 1 -- 172.21.15.45:0/377769939 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f346c0ce560 con 0x7f346c1402f0 2023-11-28T20:50:57.048 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:57.046+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:50:57.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.277+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f346c0ce560 con 0x7f346c1402f0 2023-11-28T20:50:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.341+0000 7f3463fff640 1 -- 172.21.15.45:0/377769939 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v639) v1 ==== 156+0+0 (secure 0 0 0) 0x7f346c0ce560 con 0x7f346c1402f0 2023-11-28T20:50:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f344c06af00 msgr2=0x7f344c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f344c06af00 0x7f344c06d3c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f345c007fd0 tx=0x7f345c009040 comp rx=0 tx=0).stop 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346c1402f0 msgr2=0x7f346c0cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346c1402f0 0x7f346c0cc4b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3454009870 tx=0x7f34540046c0 comp rx=0 tx=0).stop 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 shutdown_connections 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f344c06af00 0x7f344c06d3c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f346c160e70 0x7f346c141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f346c1402f0 0x7f346c0cc4b0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 --2- 172.21.15.45:0/377769939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f346c0703b0 0x7f346c0d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.344+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 >> 172.21.15.45:0/377769939 conn(0x7f346c06e7c0 msgr2=0x7f346c143b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.345+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 shutdown_connections 2023-11-28T20:50:57.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.345+0000 7f347189b640 1 -- 172.21.15.45:0/377769939 wait complete. 2023-11-28T20:50:57.366 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-11-28T20:50:57.366 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-11-28T20:50:57.367 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T20:50:57.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:50:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.597+0000 7f3b0cf27640 1 Processor -- start 2023-11-28T20:51:03.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.597+0000 7f3b0cf27640 1 -- start start 2023-11-28T20:51:03.344 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:57.994+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:03.395 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:58.982+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:03.395 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:50:59.978+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:03.395 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:00.996+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:03.395 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:02.013+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:03.396 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:03.029+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:03.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.598+0000 7f3b0cf27640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3b0815d550 0x7f3b080cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.598+0000 7f3b0cf27640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b080d1c80 0x7f3b080b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.599+0000 7f3b0cf27640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b080d2230 con 0x7f3b080d0100 2023-11-28T20:51:03.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.599+0000 7f3b0cf27640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b080d0590 con 0x7f3b0815d550 2023-11-28T20:51:03.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.599+0000 7f3b0cf27640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080d2510 con 0x7f3b080d1c80 2023-11-28T20:51:03.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.599+0000 7f3afeffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3b080d0100 0x7f3b080d1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37764/0 (socket says 172.21.15.45:37764) 2023-11-28T20:51:03.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.599+0000 7f3afeffd640 1 -- 172.21.15.45:0/3778311674 learned_addr learned my addr 172.21.15.45:0/3778311674 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:03.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.599+0000 7f3b077fe640 1 --2- 172.21.15.45:0/3778311674 >> v2:172.21.15.190:3300/0 conn(0x7f3b0815d550 0x7f3b080cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.599+0000 7f3b07fff640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b080d1c80 0x7f3b080b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.599+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253416453 0 0) 0x7f3b080d2230 con 0x7f3b080d0100 2023-11-28T20:51:03.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af4003520 con 0x7f3b080d0100 2023-11-28T20:51:03.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2589531854 0 0) 0x7f3af4003520 con 0x7f3b080d0100 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b080d1c80 msgr2=0x7f3b080b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b080d1c80 0x7f3b080b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 >> v2:172.21.15.190:3300/0 conn(0x7f3b0815d550 msgr2=0x7f3b080cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 --2- 172.21.15.45:0/3778311674 >> v2:172.21.15.190:3300/0 conn(0x7f3b0815d550 0x7f3b080cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b080d27f0 con 0x7f3b080d0100 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3af0001690 con 0x7f3b080d0100 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b07fff640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b080d1c80 0x7f3b080b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 >> v1:172.21.15.45:6789/0 conn(0x7f3b080d0100 legacy=0x7f3b080d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3af4003970 0x7f3af4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b080d1c80 0x7f3af40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3af4004d00 0x7f3af4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080d0590 con 0x7f3af4004d00 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080d2510 con 0x7f3af4003970 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af400ba30 con 0x7f3b080d1c80 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b077fe640 1 --2- 172.21.15.45:0/3778311674 >> v2:172.21.15.190:3300/0 conn(0x7f3b0815d550 0x7f3b080cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.600+0000 7f3b07fff640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3af4004d00 0x7f3af4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.601+0000 7f3b07fff640 1 -- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b080d1c80 msgr2=0x7f3af40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.601+0000 7f3b07fff640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b080d1c80 0x7f3af40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.601+0000 7f3b07fff640 1 -- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3af4003970 msgr2=0x7f3af4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.601+0000 7f3b07fff640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3af4003970 0x7f3af4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.601+0000 7f3b07fff640 1 -- 172.21.15.45:0/3778311674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af4003520 con 0x7f3af4004d00 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.602+0000 7f3b07fff640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3af4004d00 0x7f3af4005180 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f3b000098a0 tx=0x7f3af40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e4fc757963cdb8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.602+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00004230 con 0x7f3af4004d00 2023-11-28T20:51:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.602+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00004390 con 0x7f3af4004d00 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.602+0000 7f3b06ffd640 1 -- 172.21.15.45:0/3778311674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00003830 con 0x7f3af4004d00 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.603+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3778311674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3af4004d00 msgr2=0x7f3af4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.603+0000 7f3b0cf27640 1 --2- 172.21.15.45:0/3778311674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3af4004d00 0x7f3af4005180 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f3b000098a0 tx=0x7f3af40057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.603+0000 7f3b07fff640 1 -- 172.21.15.45:0/3778311674 reap_dead start 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.603+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3778311674 shutdown_connections 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.603+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3778311674 >> 172.21.15.45:0/3778311674 conn(0x7f3b0806ac70 msgr2=0x7f3b080b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.603+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3778311674 shutdown_connections 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.603+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3778311674 wait complete. 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.605+0000 7f3b0cf27640 1 Processor -- start 2023-11-28T20:51:03.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.605+0000 7f3b0cf27640 1 -- start start 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.606+0000 7f3b0cf27640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b0815afd0 0x7f3b080cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.606+0000 7f3b0cf27640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b080cb630 0x7f3b080aac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.606+0000 7f3b0cf27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b080ab140 0x7f3b080ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.606+0000 7f3b0cf27640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080cc100 con 0x7f3b080ab140 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.606+0000 7f3b0cf27640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080cbe20 con 0x7f3b080cb630 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.606+0000 7f3b0cf27640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080aa120 con 0x7f3b0815afd0 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.606+0000 7f3b07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b080ab140 0x7f3b080ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.606+0000 7f3b07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b080ab140 0x7f3b080ab5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48748/0 (socket says 172.21.15.45:48748) 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.606+0000 7f3b07fff640 1 -- 172.21.15.45:0/3819705339 learned_addr learned my addr 172.21.15.45:0/3819705339 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b077fe640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b0815afd0 0x7f3b080cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3afeffd640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b080cb630 0x7f3b080aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b07fff640 1 -- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b0815afd0 msgr2=0x7f3b080cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b07fff640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b0815afd0 0x7f3b080cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b07fff640 1 -- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b080cb630 msgr2=0x7f3b080aac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b07fff640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b080cb630 0x7f3b080aac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b07fff640 1 -- 172.21.15.45:0/3819705339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b080aa400 con 0x7f3b080ab140 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b07fff640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b080ab140 0x7f3b080ab5c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f3b000099c0 tx=0x7f3b00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b04ff9640 1 -- 172.21.15.45:0/3819705339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00030660 con 0x7f3b080ab140 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b04ff9640 1 -- 172.21.15.45:0/3819705339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b000307c0 con 0x7f3b080ab140 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b080cc100 con 0x7f3b080ab140 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.607+0000 7f3b04ff9640 1 -- 172.21.15.45:0/3819705339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00030b30 con 0x7f3b080ab140 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.608+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b080cbab0 con 0x7f3b080ab140 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.609+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b080c6b50 con 0x7f3b080ab140 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.610+0000 7f3b04ff9640 1 -- 172.21.15.45:0/3819705339 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f3b00030c90 con 0x7f3b080ab140 2023-11-28T20:51:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.611+0000 7f3b04ff9640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3ae806aed0 0x7f3ae806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.612+0000 7f3b04ff9640 1 -- 172.21.15.45:0/3819705339 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f3b00000e40 con 0x7f3b080ab140 2023-11-28T20:51:03.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.612+0000 7f3b077fe640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3ae806aed0 0x7f3ae806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.613+0000 7f3b077fe640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3ae806aed0 0x7f3ae806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009ed0 tx=0x7f3af800b040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:03.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.616+0000 7f3b04ff9640 1 -- 172.21.15.45:0/3819705339 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b00076410 con 0x7f3b080ab140 2023-11-28T20:51:03.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.823+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b080a9dd0 con 0x7f3b080ab140 2023-11-28T20:51:03.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.825+0000 7f3b04ff9640 1 -- 172.21.15.45:0/3819705339 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+14198 (secure 0 0 0) 0x7f3b00006e10 con 0x7f3b080ab140 2023-11-28T20:51:03.432 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:03.432 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":93,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:50:55.323854+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6837/3032907708":"2023-11-29T20:50:52.039578+0000","172.21.15.190:6835/3032907708":"2023-11-29T20:50:52.039578+0000","172.21.15.190:6833/2590179685":"2023-11-29T20:50:47.747393+0000","172.21.15.190:6832/2590179685":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/3115528802":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/2734256196":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/559775284":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/709518420":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/1852415784":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/1902490820":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1766221473":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/2227588570":"2023-11-29T20:50:47.747393+0000","172.21.15.190:0/1382859164":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6833/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/1155029644":"2023-11-29T20:50:41.547044+0000","172.21.15.190:6832/108887886":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/2238215747":"2023-11-29T20:50:41.547044+0000","172.21.15.190:0/3448409370":"2023-11-29T20:50:41.547044+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:03.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3ae806aed0 msgr2=0x7f3ae806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3ae806aed0 0x7f3ae806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009ed0 tx=0x7f3af800b040 comp rx=0 tx=0).stop 2023-11-28T20:51:03.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b080ab140 msgr2=0x7f3b080ab5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b080ab140 0x7f3b080ab5c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f3b000099c0 tx=0x7f3b00004730 comp rx=0 tx=0).stop 2023-11-28T20:51:03.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 shutdown_connections 2023-11-28T20:51:03.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f3ae806aed0 0x7f3ae806d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b080ab140 0x7f3b080ab5c0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b080cb630 0x7f3b080aac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 --2- 172.21.15.45:0/3819705339 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b0815afd0 0x7f3b080cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.828+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 >> 172.21.15.45:0/3819705339 conn(0x7f3b0806ac70 msgr2=0x7f3b080c8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:03.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.829+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 shutdown_connections 2023-11-28T20:51:03.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:50:57.829+0000 7f3b0cf27640 1 -- 172.21.15.45:0/3819705339 wait complete. 2023-11-28T20:51:03.434 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:51:03.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:51:03.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.675+0000 7fe9ba0f8640 1 Processor -- start 2023-11-28T20:51:03.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.676+0000 7fe9ba0f8640 1 -- start start 2023-11-28T20:51:03.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.677+0000 7fe9ba0f8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe9b415dfe0 0x7fe9b415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.677+0000 7fe9ba0f8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe9b4070e70 0x7fe9b40b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.677+0000 7fe9ba0f8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b40b2580 con 0x7fe9b4070a10 2023-11-28T20:51:03.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.677+0000 7fe9ba0f8640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b406ca10 con 0x7fe9b415dfe0 2023-11-28T20:51:03.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.677+0000 7fe9ba0f8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b4071310 con 0x7fe9b4070e70 2023-11-28T20:51:03.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.678+0000 7fe9b90f6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe9b415dfe0 0x7fe9b415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.678+0000 7fe9b90f6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe9b415dfe0 0x7fe9b415e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48756/0 (socket says 172.21.15.45:48756) 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.678+0000 7fe9b90f6640 1 -- 172.21.15.45:0/3515639441 learned_addr learned my addr 172.21.15.45:0/3515639441 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.678+0000 7fe9b98f7640 1 --2- 172.21.15.45:0/3515639441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe9b4070e70 0x7fe9b40b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.678+0000 7fe9b90f6640 1 -- 172.21.15.45:0/3515639441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe9b4070e70 msgr2=0x7fe9b40b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.678+0000 7fe9b90f6640 1 --2- 172.21.15.45:0/3515639441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe9b4070e70 0x7fe9b40b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.678+0000 7fe9b90f6640 1 -- 172.21.15.45:0/3515639441 >> v1:172.21.15.45:6789/0 conn(0x7fe9b4070a10 legacy=0x7fe9b40ad250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.678+0000 7fe9b90f6640 1 -- 172.21.15.45:0/3515639441 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b40715f0 con 0x7fe9b415dfe0 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.679+0000 7fe9b90f6640 1 --2- 172.21.15.45:0/3515639441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 0x7fe9b415e3c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009e50 tx=0x7fe9b002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1a187f6e17b741 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.679+0000 7fe9abfff640 1 -- 172.21.15.45:0/3515639441 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b002cb90 con 0x7fe9b415dfe0 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9abfff640 1 -- 172.21.15.45:0/3515639441 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b002ccf0 con 0x7fe9b415dfe0 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9abfff640 1 -- 172.21.15.45:0/3515639441 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0034610 con 0x7fe9b415dfe0 2023-11-28T20:51:03.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/3515639441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 msgr2=0x7fe9b415e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9ba0f8640 1 --2- 172.21.15.45:0/3515639441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 0x7fe9b415e3c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009e50 tx=0x7fe9b002bff0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/3515639441 shutdown_connections 2023-11-28T20:51:03.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9ba0f8640 1 --2- 172.21.15.45:0/3515639441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe9b4070e70 0x7fe9b40b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9ba0f8640 1 --2- 172.21.15.45:0/3515639441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 0x7fe9b415e3c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/3515639441 >> 172.21.15.45:0/3515639441 conn(0x7fe9b40cf4d0 msgr2=0x7fe9b40cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:03.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/3515639441 shutdown_connections 2023-11-28T20:51:03.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.680+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/3515639441 wait complete. 2023-11-28T20:51:03.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.682+0000 7fe9ba0f8640 1 Processor -- start 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.682+0000 7fe9ba0f8640 1 -- start start 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.683+0000 7fe9ba0f8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe9b4070a10 0x7fe9b40cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.683+0000 7fe9ba0f8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9b4070e70 0x7fe9b413e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.683+0000 7fe9ba0f8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 0x7fe9b4155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.683+0000 7fe9ba0f8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b41e7160 con 0x7fe9b415dfe0 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.683+0000 7fe9ba0f8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b413ed70 con 0x7fe9b4070e70 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.683+0000 7fe9ba0f8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b4152480 con 0x7fe9b4070a10 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.683+0000 7fe9b98f7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 0x7fe9b4155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b98f7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 0x7fe9b4155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48768/0 (socket says 172.21.15.45:48768) 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b98f7640 1 -- 172.21.15.45:0/58327515 learned_addr learned my addr 172.21.15.45:0/58327515 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b98f7640 1 -- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe9b4070a10 msgr2=0x7fe9b40cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b98f7640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe9b4070a10 0x7fe9b40cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b98f7640 1 -- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9b4070e70 msgr2=0x7fe9b413e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b88f5640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9b4070e70 0x7fe9b413e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b98f7640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9b4070e70 0x7fe9b413e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b98f7640 1 -- 172.21.15.45:0/58327515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b0009b30 con 0x7fe9b415dfe0 2023-11-28T20:51:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b88f5640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9b4070e70 0x7fe9b413e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.684+0000 7fe9b98f7640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 0x7fe9b4155940 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac00d5a0 tx=0x7fe9ac00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.685+0000 7fe9a9ffb640 1 -- 172.21.15.45:0/58327515 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac0125f0 con 0x7fe9b415dfe0 2023-11-28T20:51:03.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.685+0000 7fe9a9ffb640 1 -- 172.21.15.45:0/58327515 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac012750 con 0x7fe9b415dfe0 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.685+0000 7fe9a9ffb640 1 -- 172.21.15.45:0/58327515 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac012b00 con 0x7fe9b415dfe0 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.685+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b41e7370 con 0x7fe9b415dfe0 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.685+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b413f0d0 con 0x7fe9b415dfe0 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.686+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b40723c0 con 0x7fe9b415dfe0 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.687+0000 7fe9a9ffb640 1 -- 172.21.15.45:0/58327515 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fe9ac012c60 con 0x7fe9b415dfe0 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.688+0000 7fe9a9ffb640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe99806aed0 0x7fe99806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.688+0000 7fe9a9ffb640 1 -- 172.21.15.45:0/58327515 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fe9ac089cc0 con 0x7fe9b415dfe0 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.689+0000 7fe9b90f6640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe99806aed0 0x7fe99806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.690+0000 7fe9b90f6640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe99806aed0 0x7fe99806d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0035000 tx=0x7fe9b002e000 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:03.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.693+0000 7fe9a9ffb640 1 -- 172.21.15.45:0/58327515 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9ac090080 con 0x7fe9b415dfe0 2023-11-28T20:51:03.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.896+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe9b404e920 con 0x7fe9b415dfe0 2023-11-28T20:51:03.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.897+0000 7fe9a9ffb640 1 -- 172.21.15.45:0/58327515 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe9ac054b90 con 0x7fe9b415dfe0 2023-11-28T20:51:03.900 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:51:03.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.899+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe99806aed0 msgr2=0x7fe99806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.899+0000 7fe9ba0f8640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe99806aed0 0x7fe99806d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0035000 tx=0x7fe9b002e000 comp rx=0 tx=0).stop 2023-11-28T20:51:03.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 msgr2=0x7fe9b4155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:03.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 0x7fe9b4155940 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac00d5a0 tx=0x7fe9ac00da60 comp rx=0 tx=0).stop 2023-11-28T20:51:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 shutdown_connections 2023-11-28T20:51:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe99806aed0 0x7fe99806d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9b415dfe0 0x7fe9b4155940 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9b4070e70 0x7fe9b413e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 --2- 172.21.15.45:0/58327515 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe9b4070a10 0x7fe9b40cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 >> 172.21.15.45:0/58327515 conn(0x7fe9b40cf4d0 msgr2=0x7fe9b406eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 shutdown_connections 2023-11-28T20:51:03.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:03.900+0000 7fe9ba0f8640 1 -- 172.21.15.45:0/58327515 wait complete. 2023-11-28T20:51:03.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-11-28T20:51:04.033 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:04.030+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:04.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.146+0000 7faa551f1640 1 Processor -- start 2023-11-28T20:51:04.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.146+0000 7faa551f1640 1 -- start start 2023-11-28T20:51:04.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.148+0000 7faa551f1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa50144bf0 0x7faa5015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:04.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.148+0000 7faa551f1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa5006e6e0 0x7faa5006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:04.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.148+0000 7faa551f1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5006f070 con 0x7faa500ce490 2023-11-28T20:51:04.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.148+0000 7faa551f1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa500c6440 con 0x7faa50144bf0 2023-11-28T20:51:04.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.148+0000 7faa551f1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa50144fd0 con 0x7faa5006e6e0 2023-11-28T20:51:04.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.148+0000 7faa4ed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa50144bf0 0x7faa5015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:04.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.148+0000 7faa4ed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa50144bf0 0x7faa5015c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45556/0 (socket says 172.21.15.45:45556) 2023-11-28T20:51:04.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.148+0000 7faa4ed9d640 1 -- 172.21.15.45:0/1508884423 learned_addr learned my addr 172.21.15.45:0/1508884423 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:04.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.149+0000 7faa4f59e640 1 --2- 172.21.15.45:0/1508884423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa5006e6e0 0x7faa5006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:04.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.149+0000 7faa4ed9d640 1 -- 172.21.15.45:0/1508884423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa5006e6e0 msgr2=0x7faa5006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:04.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.149+0000 7faa4ed9d640 1 --2- 172.21.15.45:0/1508884423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa5006e6e0 0x7faa5006eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:04.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.149+0000 7faa4ed9d640 1 -- 172.21.15.45:0/1508884423 >> v1:172.21.15.190:6789/0 conn(0x7faa500ce490 legacy=0x7faa500c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:04.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.149+0000 7faa4ed9d640 1 -- 172.21.15.45:0/1508884423 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa500cebd0 con 0x7faa50144bf0 2023-11-28T20:51:04.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.150+0000 7faa4ed9d640 1 --2- 172.21.15.45:0/1508884423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa50144bf0 0x7faa5015c080 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7faa40009e50 tx=0x7faa4002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75376ed63fbf8dfc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.150+0000 7faa4e59c640 1 -- 172.21.15.45:0/1508884423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4002cb00 con 0x7faa50144bf0 2023-11-28T20:51:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa4e59c640 1 -- 172.21.15.45:0/1508884423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4002cc60 con 0x7faa50144bf0 2023-11-28T20:51:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa4e59c640 1 -- 172.21.15.45:0/1508884423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa400343f0 con 0x7faa50144bf0 2023-11-28T20:51:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa551f1640 1 -- 172.21.15.45:0/1508884423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa50144bf0 msgr2=0x7faa5015c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa551f1640 1 --2- 172.21.15.45:0/1508884423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa50144bf0 0x7faa5015c080 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7faa40009e50 tx=0x7faa4002bf60 comp rx=0 tx=0).stop 2023-11-28T20:51:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa551f1640 1 -- 172.21.15.45:0/1508884423 shutdown_connections 2023-11-28T20:51:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa551f1640 1 --2- 172.21.15.45:0/1508884423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa5006e6e0 0x7faa5006eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa551f1640 1 --2- 172.21.15.45:0/1508884423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa50144bf0 0x7faa5015c080 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa551f1640 1 -- 172.21.15.45:0/1508884423 >> 172.21.15.45:0/1508884423 conn(0x7faa5006c710 msgr2=0x7faa5006caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:04.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa551f1640 1 -- 172.21.15.45:0/1508884423 shutdown_connections 2023-11-28T20:51:04.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.151+0000 7faa551f1640 1 -- 172.21.15.45:0/1508884423 wait complete. 2023-11-28T20:51:04.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.153+0000 7faa551f1640 1 Processor -- start 2023-11-28T20:51:04.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.153+0000 7faa551f1640 1 -- start start 2023-11-28T20:51:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.154+0000 7faa551f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa5006e6e0 0x7faa5015daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.154+0000 7faa551f1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa500ce490 0x7faa5006aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.154+0000 7faa551f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa50144bf0 0x7faa501e5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.154+0000 7faa551f1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5006b880 con 0x7faa5006e6e0 2023-11-28T20:51:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.154+0000 7faa551f1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5006bb60 con 0x7faa50144bf0 2023-11-28T20:51:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.154+0000 7faa551f1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5006be40 con 0x7faa500ce490 2023-11-28T20:51:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa5006e6e0 0x7faa5015daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa5006e6e0 0x7faa5015daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45566/0 (socket says 172.21.15.45:45566) 2023-11-28T20:51:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4ed9d640 1 -- 172.21.15.45:0/2854361911 learned_addr learned my addr 172.21.15.45:0/2854361911 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4659c640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa500ce490 0x7faa5006aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4ed9d640 1 -- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa500ce490 msgr2=0x7faa5006aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4f59e640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa50144bf0 0x7faa501e5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4ed9d640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa500ce490 0x7faa5006aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:04.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4ed9d640 1 -- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa50144bf0 msgr2=0x7faa501e5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:04.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4ed9d640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa50144bf0 0x7faa501e5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:04.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4ed9d640 1 -- 172.21.15.45:0/2854361911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40009b30 con 0x7faa5006e6e0 2023-11-28T20:51:04.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4659c640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa500ce490 0x7faa5006aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:04.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.155+0000 7faa4f59e640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa50144bf0 0x7faa501e5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:04.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.156+0000 7faa4ed9d640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa5006e6e0 0x7faa5015daf0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7faa40000f80 tx=0x7faa40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:04.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.156+0000 7faa47fff640 1 -- 172.21.15.45:0/2854361911 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40009510 con 0x7faa5006e6e0 2023-11-28T20:51:04.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.156+0000 7faa47fff640 1 -- 172.21.15.45:0/2854361911 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4003e450 con 0x7faa5006e6e0 2023-11-28T20:51:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.156+0000 7faa47fff640 1 -- 172.21.15.45:0/2854361911 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4003e800 con 0x7faa5006e6e0 2023-11-28T20:51:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.156+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa500c8b60 con 0x7faa5006e6e0 2023-11-28T20:51:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.156+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5015e030 con 0x7faa5006e6e0 2023-11-28T20:51:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.157+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa500733d0 con 0x7faa5006e6e0 2023-11-28T20:51:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.158+0000 7faa47fff640 1 -- 172.21.15.45:0/2854361911 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7faa4003e960 con 0x7faa5006e6e0 2023-11-28T20:51:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.159+0000 7faa47fff640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faa3006aed0 0x7faa3006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.159+0000 7faa47fff640 1 -- 172.21.15.45:0/2854361911 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7faa400ac4c0 con 0x7faa5006e6e0 2023-11-28T20:51:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.160+0000 7faa4659c640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faa3006aed0 0x7faa3006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.161+0000 7faa4659c640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faa3006aed0 0x7faa3006d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faa500cabf0 tx=0x7faa38009040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:04.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.164+0000 7faa47fff640 1 -- 172.21.15.45:0/2854361911 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa40077640 con 0x7faa5006e6e0 2023-11-28T20:51:04.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:04.448+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7faa50144410 con 0x7faa5006e6e0 2023-11-28T20:51:05.065 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:05.062+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:05.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.382+0000 7faa47fff640 1 -- 172.21.15.45:0/2854361911 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v645) v1 ==== 169+0+0 (secure 0 0 0) 0x7faa4007ac60 con 0x7faa5006e6e0 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faa3006aed0 msgr2=0x7faa3006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faa3006aed0 0x7faa3006d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faa500cabf0 tx=0x7faa38009040 comp rx=0 tx=0).stop 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa5006e6e0 msgr2=0x7faa5015daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa5006e6e0 0x7faa5015daf0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7faa40000f80 tx=0x7faa40004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 shutdown_connections 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faa3006aed0 0x7faa3006d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa50144bf0 0x7faa501e5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa500ce490 0x7faa5006aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 --2- 172.21.15.45:0/2854361911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa5006e6e0 0x7faa5015daf0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.386+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 >> 172.21.15.45:0/2854361911 conn(0x7faa5006c710 msgr2=0x7faa500b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:05.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.387+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 shutdown_connections 2023-11-28T20:51:05.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.387+0000 7faa551f1640 1 -- 172.21.15.45:0/2854361911 wait complete. 2023-11-28T20:51:05.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T20:51:05.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.643+0000 7f5211a71640 1 Processor -- start 2023-11-28T20:51:05.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.644+0000 7f5211a71640 1 -- start start 2023-11-28T20:51:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.645+0000 7f5211a71640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f520c0cdfb0 0x7f520c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.645+0000 7f5211a71640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f520c06c480 0x7f520c0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.645+0000 7f5211a71640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c0cdb20 con 0x7f520c155770 2023-11-28T20:51:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.645+0000 7f5211a71640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520c06c8b0 con 0x7f520c06c480 2023-11-28T20:51:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.645+0000 7f5211a71640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520c0ccf10 con 0x7f520c0cdfb0 2023-11-28T20:51:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.646+0000 7f5211270640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f520c06c480 0x7f520c0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.646+0000 7f5211270640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f520c06c480 0x7f520c0b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45582/0 (socket says 172.21.15.45:45582) 2023-11-28T20:51:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.646+0000 7f5211270640 1 -- 172.21.15.45:0/787212557 learned_addr learned my addr 172.21.15.45:0/787212557 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.646+0000 7f520bfff640 1 --2- 172.21.15.45:0/787212557 >> v2:172.21.15.190:3300/0 conn(0x7f520c0cdfb0 0x7f520c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.646+0000 7f5211270640 1 -- 172.21.15.45:0/787212557 >> v2:172.21.15.190:3300/0 conn(0x7f520c0cdfb0 msgr2=0x7f520c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.646+0000 7f5211270640 1 --2- 172.21.15.45:0/787212557 >> v2:172.21.15.190:3300/0 conn(0x7f520c0cdfb0 0x7f520c06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.646+0000 7f5211270640 1 -- 172.21.15.45:0/787212557 >> v1:172.21.15.190:6789/0 conn(0x7f520c155770 legacy=0x7f520c15c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.646+0000 7f5211270640 1 -- 172.21.15.45:0/787212557 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c0cd1f0 con 0x7f520c06c480 2023-11-28T20:51:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.647+0000 7f5211270640 1 --2- 172.21.15.45:0/787212557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 0x7f520c0b20e0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f520c072480 tx=0x7f520402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f3d772ebbd2ae5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.648+0000 7f520b7fe640 1 -- 172.21.15.45:0/787212557 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520402c9f0 con 0x7f520c06c480 2023-11-28T20:51:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.648+0000 7f520b7fe640 1 -- 172.21.15.45:0/787212557 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520402cb50 con 0x7f520c06c480 2023-11-28T20:51:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.648+0000 7f520b7fe640 1 -- 172.21.15.45:0/787212557 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52040343f0 con 0x7f520c06c480 2023-11-28T20:51:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.649+0000 7f5211a71640 1 -- 172.21.15.45:0/787212557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 msgr2=0x7f520c0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.649+0000 7f5211a71640 1 --2- 172.21.15.45:0/787212557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 0x7f520c0b20e0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f520c072480 tx=0x7f520402be50 comp rx=0 tx=0).stop 2023-11-28T20:51:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.649+0000 7f5211a71640 1 -- 172.21.15.45:0/787212557 shutdown_connections 2023-11-28T20:51:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.649+0000 7f5211a71640 1 --2- 172.21.15.45:0/787212557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 0x7f520c0b20e0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.649+0000 7f5211a71640 1 --2- 172.21.15.45:0/787212557 >> v2:172.21.15.190:3300/0 conn(0x7f520c0cdfb0 0x7f520c06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.649+0000 7f5211a71640 1 -- 172.21.15.45:0/787212557 >> 172.21.15.45:0/787212557 conn(0x7f520c0b1050 msgr2=0x7f520c06a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:05.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.649+0000 7f5211a71640 1 -- 172.21.15.45:0/787212557 shutdown_connections 2023-11-28T20:51:05.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.649+0000 7f5211a71640 1 -- 172.21.15.45:0/787212557 wait complete. 2023-11-28T20:51:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.651+0000 7f5211a71640 1 Processor -- start 2023-11-28T20:51:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.651+0000 7f5211a71640 1 -- start start 2023-11-28T20:51:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.652+0000 7f5211a71640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 0x7f520c14e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.652+0000 7f5211a71640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c0cdfb0 0x7f520c09c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.652+0000 7f5211a71640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c155770 0x7f520c153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.652+0000 7f5211a71640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c09b610 con 0x7f520c06c480 2023-11-28T20:51:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.652+0000 7f5211a71640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c1571d0 con 0x7f520c0cdfb0 2023-11-28T20:51:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.652+0000 7f5211a71640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c09aa90 con 0x7f520c155770 2023-11-28T20:51:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.652+0000 7f5210a6f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 0x7f520c14e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.652+0000 7f5210a6f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 0x7f520c14e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45590/0 (socket says 172.21.15.45:45590) 2023-11-28T20:51:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.652+0000 7f5210a6f640 1 -- 172.21.15.45:0/2125867827 learned_addr learned my addr 172.21.15.45:0/2125867827 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f520bfff640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c0cdfb0 0x7f520c09c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f5211270640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c155770 0x7f520c153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f5210a6f640 1 -- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c155770 msgr2=0x7f520c153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f5210a6f640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c155770 0x7f520c153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f5210a6f640 1 -- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c0cdfb0 msgr2=0x7f520c09c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f5210a6f640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c0cdfb0 0x7f520c09c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f5210a6f640 1 -- 172.21.15.45:0/2125867827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5204009b30 con 0x7f520c06c480 2023-11-28T20:51:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f520bfff640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c0cdfb0 0x7f520c09c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f5210a6f640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 0x7f520c14e6b0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f520000b3f0 tx=0x7f520000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.653+0000 7f52097fa640 1 -- 172.21.15.45:0/2125867827 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200011830 con 0x7f520c06c480 2023-11-28T20:51:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.654+0000 7f52097fa640 1 -- 172.21.15.45:0/2125867827 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5200011990 con 0x7f520c06c480 2023-11-28T20:51:05.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.654+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c09ad70 con 0x7f520c06c480 2023-11-28T20:51:05.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.654+0000 7f52097fa640 1 -- 172.21.15.45:0/2125867827 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200011d40 con 0x7f520c06c480 2023-11-28T20:51:05.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.654+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c09b330 con 0x7f520c06c480 2023-11-28T20:51:05.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.655+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c09d0f0 con 0x7f520c06c480 2023-11-28T20:51:05.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.656+0000 7f52097fa640 1 -- 172.21.15.45:0/2125867827 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f520001a400 con 0x7f520c06c480 2023-11-28T20:51:05.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.657+0000 7f52097fa640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f51f006aed0 0x7f51f006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:05.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.658+0000 7f52097fa640 1 -- 172.21.15.45:0/2125867827 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f5200089580 con 0x7f520c06c480 2023-11-28T20:51:05.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.658+0000 7f520bfff640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f51f006aed0 0x7f51f006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.659+0000 7f520bfff640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f51f006aed0 0x7f51f006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f520c0674c0 tx=0x7f51f8008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:05.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.662+0000 7f52097fa640 1 -- 172.21.15.45:0/2125867827 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5200054550 con 0x7f520c06c480 2023-11-28T20:51:05.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.866+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f520c04e920 con 0x7f520c06c480 2023-11-28T20:51:05.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.872+0000 7f52097fa640 1 -- 172.21.15.45:0/2125867827 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5200057b70 con 0x7f520c06c480 2023-11-28T20:51:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.875+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f51f006aed0 msgr2=0x7f51f006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.875+0000 7f5211a71640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f51f006aed0 0x7f51f006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f520c0674c0 tx=0x7f51f8008040 comp rx=0 tx=0).stop 2023-11-28T20:51:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 msgr2=0x7f520c14e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 0x7f520c14e6b0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f520000b3f0 tx=0x7f520000b8b0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 shutdown_connections 2023-11-28T20:51:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f51f006aed0 0x7f51f006d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c155770 0x7f520c153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c0cdfb0 0x7f520c09c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 --2- 172.21.15.45:0/2125867827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06c480 0x7f520c14e6b0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 >> 172.21.15.45:0/2125867827 conn(0x7f520c0b1050 msgr2=0x7f520c15a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 shutdown_connections 2023-11-28T20:51:05.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:05.876+0000 7f5211a71640 1 -- 172.21.15.45:0/2125867827 wait complete. 2023-11-28T20:51:05.898 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:51:05.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:51:06.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:51:06.077 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:06.075+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:06.150 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:51:06.151 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:51:06.151 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:06.213 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:51:06.214 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:51:06.214 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:06.273 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:51:06.274 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:51:06.274 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:06.338 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:51:06.338 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:51:06.338 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:06.400 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:51:06.400 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T20:51:06.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:06.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.632+0000 7fa5ceb78640 1 Processor -- start 2023-11-28T20:51:06.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.632+0000 7fa5ceb78640 1 -- start start 2023-11-28T20:51:06.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.634+0000 7fa5ceb78640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa5c80cc090 0x7fa5c8150000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:06.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.634+0000 7fa5ceb78640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c80ceaa0 con 0x7fa5c8070ee0 2023-11-28T20:51:06.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.634+0000 7fa5ceb78640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c815a9a0 con 0x7fa5c80cb3b0 2023-11-28T20:51:06.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.634+0000 7fa5ceb78640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c8159da0 con 0x7fa5c80cc090 2023-11-28T20:51:06.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.634+0000 7fa5cd115640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa5c80cc090 0x7fa5c8150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:06.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.634+0000 7fa5cc914640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa5c8070ee0 0x7fa5c80ce390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47176/0 (socket says 172.21.15.45:47176) 2023-11-28T20:51:06.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.634+0000 7fa5cc914640 1 -- 172.21.15.45:0/4099451491 learned_addr learned my addr 172.21.15.45:0/4099451491 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:06.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.635+0000 7fa5cd115640 1 -- 172.21.15.45:0/4099451491 >> v1:172.21.15.190:6789/0 conn(0x7fa5c80cb3b0 legacy=0x7fa5c80cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:06.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.635+0000 7fa5cd115640 1 -- 172.21.15.45:0/4099451491 >> v1:172.21.15.45:6789/0 conn(0x7fa5c8070ee0 legacy=0x7fa5c80ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:06.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.635+0000 7fa5cd115640 1 -- 172.21.15.45:0/4099451491 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c815a070 con 0x7fa5c80cc090 2023-11-28T20:51:06.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.636+0000 7fa5cd115640 1 --2- 172.21.15.45:0/4099451491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c80cc090 0x7fa5c8150000 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00097a0 tx=0x7fa5c002db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d754e074aebaaf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:06.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.636+0000 7fa5c77fe640 1 -- 172.21.15.45:0/4099451491 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c002e920 con 0x7fa5c80cc090 2023-11-28T20:51:06.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.636+0000 7fa5c77fe640 1 -- 172.21.15.45:0/4099451491 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c002ea80 con 0x7fa5c80cc090 2023-11-28T20:51:06.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.636+0000 7fa5c77fe640 1 -- 172.21.15.45:0/4099451491 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c002ee10 con 0x7fa5c80cc090 2023-11-28T20:51:06.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.637+0000 7fa5ceb78640 1 -- 172.21.15.45:0/4099451491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c80cc090 msgr2=0x7fa5c8150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:06.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.637+0000 7fa5ceb78640 1 --2- 172.21.15.45:0/4099451491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c80cc090 0x7fa5c8150000 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00097a0 tx=0x7fa5c002db70 comp rx=0 tx=0).stop 2023-11-28T20:51:06.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.637+0000 7fa5ceb78640 1 -- 172.21.15.45:0/4099451491 shutdown_connections 2023-11-28T20:51:06.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.637+0000 7fa5ceb78640 1 --2- 172.21.15.45:0/4099451491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c80cc090 0x7fa5c8150000 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:06.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.637+0000 7fa5ceb78640 1 -- 172.21.15.45:0/4099451491 >> 172.21.15.45:0/4099451491 conn(0x7fa5c806a500 msgr2=0x7fa5c806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:06.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.637+0000 7fa5ceb78640 1 -- 172.21.15.45:0/4099451491 shutdown_connections 2023-11-28T20:51:06.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.637+0000 7fa5ceb78640 1 -- 172.21.15.45:0/4099451491 wait complete. 2023-11-28T20:51:06.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.639+0000 7fa5ceb78640 1 Processor -- start 2023-11-28T20:51:06.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.639+0000 7fa5ceb78640 1 -- start start 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.640+0000 7fa5ceb78640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c8070ee0 0x7fa5c81458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.640+0000 7fa5ceb78640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5c80cb3b0 0x7fa5c815c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.640+0000 7fa5ceb78640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5c80cc090 0x7fa5c815c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.640+0000 7fa5ceb78640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81f3810 con 0x7fa5c8070ee0 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.640+0000 7fa5ceb78640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81f3af0 con 0x7fa5c80cb3b0 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.640+0000 7fa5ceb78640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81f3dd0 con 0x7fa5c80cc090 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.640+0000 7fa5cc914640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c8070ee0 0x7fa5c81458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.640+0000 7fa5cc914640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c8070ee0 0x7fa5c81458b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45616/0 (socket says 172.21.15.45:45616) 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.640+0000 7fa5cc914640 1 -- 172.21.15.45:0/1115537291 learned_addr learned my addr 172.21.15.45:0/1115537291 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.641+0000 7fa5cc914640 1 -- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5c80cc090 msgr2=0x7fa5c815c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:06.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.641+0000 7fa5cc914640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5c80cc090 0x7fa5c815c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:06.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.641+0000 7fa5cc914640 1 -- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5c80cb3b0 msgr2=0x7fa5c815c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:06.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.641+0000 7fa5cc914640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5c80cb3b0 0x7fa5c815c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:06.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.641+0000 7fa5cc914640 1 -- 172.21.15.45:0/1115537291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c0009400 con 0x7fa5c8070ee0 2023-11-28T20:51:06.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.641+0000 7fa5cc914640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c8070ee0 0x7fa5c81458b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800cba0 tx=0x7fa5b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:06.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.641+0000 7fa5c57fa640 1 -- 172.21.15.45:0/1115537291 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800ad40 con 0x7fa5c8070ee0 2023-11-28T20:51:06.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.641+0000 7fa5c57fa640 1 -- 172.21.15.45:0/1115537291 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8012400 con 0x7fa5c8070ee0 2023-11-28T20:51:06.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.641+0000 7fa5c57fa640 1 -- 172.21.15.45:0/1115537291 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800aea0 con 0x7fa5c8070ee0 2023-11-28T20:51:06.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.642+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c81f40b0 con 0x7fa5c8070ee0 2023-11-28T20:51:06.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.642+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c81f34c0 con 0x7fa5c8070ee0 2023-11-28T20:51:06.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.643+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c81f42a0 con 0x7fa5c8070ee0 2023-11-28T20:51:06.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.644+0000 7fa5c57fa640 1 -- 172.21.15.45:0/1115537291 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fa5b8007070 con 0x7fa5c8070ee0 2023-11-28T20:51:06.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.645+0000 7fa5c57fa640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa5a806aed0 0x7fa5a806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:06.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.645+0000 7fa5c57fa640 1 -- 172.21.15.45:0/1115537291 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fa5b8089470 con 0x7fa5c8070ee0 2023-11-28T20:51:06.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.646+0000 7fa5c7fff640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa5a806aed0 0x7fa5a806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:06.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.647+0000 7fa5c7fff640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa5a806aed0 0x7fa5a806d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0000f60 tx=0x7fa5b00079b0 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.649+0000 7fa5c57fa640 1 -- 172.21.15.45:0/1115537291 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5b80544f0 con 0x7fa5c8070ee0 2023-11-28T20:51:06.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.897+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5c81f3950 con 0x7fa5c8070ee0 2023-11-28T20:51:06.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.898+0000 7fa5c57fa640 1 -- 172.21.15.45:0/1115537291 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fa5b8017e80 con 0x7fa5c8070ee0 2023-11-28T20:51:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:06.901 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5438,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2396470524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2396470524},{"type":"v1","addr":"172.21.15.45:6835","nonce":2396470524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/436103276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":436103276},{"type":"v1","addr":"172.21.15.190:6836","nonce":436103276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/438721770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":438721770},{"type":"v1","addr":"172.21.15.190:6837","nonce":438721770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/834328551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":834328551},{"type":"v1","addr":"172.21.15.45:6834","nonce":834328551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[]} 2023-11-28T20:51:06.901 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 38 2023-11-28T20:51:06.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa5a806aed0 msgr2=0x7fa5a806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:06.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa5a806aed0 0x7fa5a806d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0000f60 tx=0x7fa5b00079b0 comp rx=0 tx=0).stop 2023-11-28T20:51:06.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c8070ee0 msgr2=0x7fa5c81458b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:06.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c8070ee0 0x7fa5c81458b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800cba0 tx=0x7fa5b800a430 comp rx=0 tx=0).stop 2023-11-28T20:51:06.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 shutdown_connections 2023-11-28T20:51:06.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa5a806aed0 0x7fa5a806d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:06.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5c80cc090 0x7fa5c815c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:06.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5c80cb3b0 0x7fa5c815c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:06.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 --2- 172.21.15.45:0/1115537291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5c8070ee0 0x7fa5c81458b0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:06.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 >> 172.21.15.45:0/1115537291 conn(0x7fa5c806a500 msgr2=0x7fa5c8148b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:06.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 shutdown_connections 2023-11-28T20:51:06.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:06.901+0000 7fa5ceb78640 1 -- 172.21.15.45:0/1115537291 wait complete. 2023-11-28T20:51:06.924 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T20:51:06.924 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T20:51:06.924 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T20:51:06.925 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:51:06.925 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T20:51:06.925 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T20:51:06.925 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T20:51:06.925 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:51:06.926 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T20:51:06.926 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T20:51:06.926 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T20:51:06.926 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:51:06.926 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T20:51:06.927 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T20:51:06.927 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T20:51:06.927 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:51:06.943 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T20:51:06.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T20:51:06.946 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T20:51:06.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T20:51:06.947 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T20:51:06.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T20:51:06.961 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T20:51:06.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T20:51:06.997 INFO:tasks.ceph.mds.a:Started 2023-11-28T20:51:07.000 DEBUG:teuthology.parallel:result is None 2023-11-28T20:51:07.008 INFO:tasks.ceph.mds.c:Started 2023-11-28T20:51:07.008 DEBUG:teuthology.parallel:result is None 2023-11-28T20:51:07.009 INFO:tasks.ceph.mds.d:Started 2023-11-28T20:51:07.009 DEBUG:teuthology.parallel:result is None 2023-11-28T20:51:07.017 INFO:tasks.ceph.mds.b:Started 2023-11-28T20:51:07.017 DEBUG:teuthology.parallel:result is None 2023-11-28T20:51:07.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T20:51:07.056 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:07.053+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:07.172 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:51:07.172 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:51:07.177 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:51:07.183 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T20:51:07.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.222+0000 7fb45eff5640 1 Processor -- start 2023-11-28T20:51:07.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.222+0000 7fb45eff5640 1 -- start start 2023-11-28T20:51:07.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.223+0000 7fb45eff5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb45815b820 0x7fb45806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:07.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.223+0000 7fb45eff5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb45815f4d0 0x7fb4580cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:07.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.223+0000 7fb45eff5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45815fa80 con 0x7fb4580cf020 2023-11-28T20:51:07.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.223+0000 7fb45eff5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45815fd60 con 0x7fb45815f4d0 2023-11-28T20:51:07.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.223+0000 7fb45eff5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4580cf9f0 con 0x7fb45815b820 2023-11-28T20:51:07.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb45d592640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb45815f4d0 0x7fb4580cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:07.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb457fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb4580cf020 0x7fb45815edc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47200/0 (socket says 172.21.15.45:47200) 2023-11-28T20:51:07.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb457fff640 1 -- 172.21.15.45:0/2929679615 learned_addr learned my addr 172.21.15.45:0/2929679615 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:07.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb45cd91640 1 --2- 172.21.15.45:0/2929679615 >> v2:172.21.15.190:3300/0 conn(0x7fb45815b820 0x7fb45806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:07.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb45d592640 1 -- 172.21.15.45:0/2929679615 >> v2:172.21.15.190:3300/0 conn(0x7fb45815b820 msgr2=0x7fb45806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:07.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb45d592640 1 --2- 172.21.15.45:0/2929679615 >> v2:172.21.15.190:3300/0 conn(0x7fb45815b820 0x7fb45806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:07.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb45d592640 1 -- 172.21.15.45:0/2929679615 >> v1:172.21.15.45:6789/0 conn(0x7fb4580cf020 legacy=0x7fb45815edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:07.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb45d592640 1 -- 172.21.15.45:0/2929679615 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45806cf40 con 0x7fb45815f4d0 2023-11-28T20:51:07.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb4577fe640 1 -- 172.21.15.45:0/2929679615 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940826966 0 0) 0x7fb45815fa80 con 0x7fb4580cf020 2023-11-28T20:51:07.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.224+0000 7fb45cd91640 1 --2- 172.21.15.45:0/2929679615 >> v2:172.21.15.190:3300/0 conn(0x7fb45815b820 0x7fb45806da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:07.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.225+0000 7fb45d592640 1 --2- 172.21.15.45:0/2929679615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 0x7fb4580cf4b0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb4580b40e0 tx=0x7fb45002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee7dcd695094319 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.226+0000 7fb4577fe640 1 -- 172.21.15.45:0/2929679615 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45002cc80 con 0x7fb45815f4d0 2023-11-28T20:51:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.226+0000 7fb4577fe640 1 -- 172.21.15.45:0/2929679615 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45002cde0 con 0x7fb45815f4d0 2023-11-28T20:51:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.226+0000 7fb4577fe640 1 -- 172.21.15.45:0/2929679615 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb450034660 con 0x7fb45815f4d0 2023-11-28T20:51:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.226+0000 7fb45eff5640 1 -- 172.21.15.45:0/2929679615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 msgr2=0x7fb4580cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.226+0000 7fb45eff5640 1 --2- 172.21.15.45:0/2929679615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 0x7fb4580cf4b0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb4580b40e0 tx=0x7fb45002bed0 comp rx=0 tx=0).stop 2023-11-28T20:51:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.226+0000 7fb45eff5640 1 -- 172.21.15.45:0/2929679615 shutdown_connections 2023-11-28T20:51:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.226+0000 7fb45eff5640 1 --2- 172.21.15.45:0/2929679615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 0x7fb4580cf4b0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.226+0000 7fb45eff5640 1 --2- 172.21.15.45:0/2929679615 >> v2:172.21.15.190:3300/0 conn(0x7fb45815b820 0x7fb45806da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:07.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.226+0000 7fb45eff5640 1 -- 172.21.15.45:0/2929679615 >> 172.21.15.45:0/2929679615 conn(0x7fb45806a6e0 msgr2=0x7fb45806aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:07.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.227+0000 7fb45eff5640 1 -- 172.21.15.45:0/2929679615 shutdown_connections 2023-11-28T20:51:07.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.227+0000 7fb45eff5640 1 -- 172.21.15.45:0/2929679615 wait complete. 2023-11-28T20:51:07.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.228+0000 7fb45eff5640 1 Processor -- start 2023-11-28T20:51:07.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.229+0000 7fb45eff5640 1 -- start start 2023-11-28T20:51:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.229+0000 7fb45eff5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4580cf020 0x7fb4581f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.229+0000 7fb45eff5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb45815b820 0x7fb45809b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45eff5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 0x7fb45809a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45eff5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4580c9100 con 0x7fb45815f4d0 2023-11-28T20:51:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45eff5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45809c240 con 0x7fb4580cf020 2023-11-28T20:51:07.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45eff5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4580c9370 con 0x7fb45815b820 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45d592640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 0x7fb45809a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45d592640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 0x7fb45809a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45668/0 (socket says 172.21.15.45:45668) 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45d592640 1 -- 172.21.15.45:0/185147596 learned_addr learned my addr 172.21.15.45:0/185147596 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45cd91640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4580cf020 0x7fb4581f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb457fff640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb45815b820 0x7fb45809b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45d592640 1 -- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb45815b820 msgr2=0x7fb45809b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45d592640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb45815b820 0x7fb45809b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45d592640 1 -- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4580cf020 msgr2=0x7fb4581f3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45d592640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4580cf020 0x7fb4581f3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:07.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.230+0000 7fb45d592640 1 -- 172.21.15.45:0/185147596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb450009a70 con 0x7fb45815f4d0 2023-11-28T20:51:07.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.231+0000 7fb45d592640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 0x7fb45809a990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb45002c4e0 tx=0x7fb450004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:07.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.231+0000 7fb4557fa640 1 -- 172.21.15.45:0/185147596 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45003e610 con 0x7fb45815f4d0 2023-11-28T20:51:07.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.231+0000 7fb4557fa640 1 -- 172.21.15.45:0/185147596 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45003e770 con 0x7fb45815f4d0 2023-11-28T20:51:07.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.231+0000 7fb4557fa640 1 -- 172.21.15.45:0/185147596 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45003eb20 con 0x7fb45815f4d0 2023-11-28T20:51:07.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.231+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4580c9650 con 0x7fb45815f4d0 2023-11-28T20:51:07.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.231+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45809bea0 con 0x7fb45815f4d0 2023-11-28T20:51:07.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.232+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4580c9870 con 0x7fb45815f4d0 2023-11-28T20:51:07.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.233+0000 7fb4557fa640 1 -- 172.21.15.45:0/185147596 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fb45003b050 con 0x7fb45815f4d0 2023-11-28T20:51:07.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.234+0000 7fb4557fa640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb43c06aed0 0x7fb43c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:07.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.234+0000 7fb4557fa640 1 -- 172.21.15.45:0/185147596 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fb4500abe20 con 0x7fb45815f4d0 2023-11-28T20:51:07.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.234+0000 7fb45cd91640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb43c06aed0 0x7fb43c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:07.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.235+0000 7fb45cd91640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb43c06aed0 0x7fb43c06d390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb44c006e80 tx=0x7fb44c008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:07.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.239+0000 7fb4557fa640 1 -- 172.21.15.45:0/185147596 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb450076ec0 con 0x7fb45815f4d0 2023-11-28T20:51:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:07.457+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb458073440 con 0x7fb45815f4d0 2023-11-28T20:51:08.055 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:08.051+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:08.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.448+0000 7fb4557fa640 1 -- 172.21.15.45:0/185147596 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v98) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb450039080 con 0x7fb45815f4d0 2023-11-28T20:51:08.450 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T20:51:08.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb43c06aed0 msgr2=0x7fb43c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:08.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb43c06aed0 0x7fb43c06d390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb44c006e80 tx=0x7fb44c008040 comp rx=0 tx=0).stop 2023-11-28T20:51:08.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 msgr2=0x7fb45809a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:08.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 0x7fb45809a990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb45002c4e0 tx=0x7fb450004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:08.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 shutdown_connections 2023-11-28T20:51:08.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb43c06aed0 0x7fb43c06d390 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb45815f4d0 0x7fb45809a990 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb45815b820 0x7fb45809b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 --2- 172.21.15.45:0/185147596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4580cf020 0x7fb4581f3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 >> 172.21.15.45:0/185147596 conn(0x7fb45806a6e0 msgr2=0x7fb45815cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:08.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 shutdown_connections 2023-11-28T20:51:08.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.452+0000 7fb45eff5640 1 -- 172.21.15.45:0/185147596 wait complete. 2023-11-28T20:51:08.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T20:51:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.693+0000 7ff090339640 1 Processor -- start 2023-11-28T20:51:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.694+0000 7ff090339640 1 -- start start 2023-11-28T20:51:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.695+0000 7ff090339640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0880b35b0 0x7ff08815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.695+0000 7ff090339640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff08815b850 0x7ff0880b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.695+0000 7ff090339640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b1fa0 0x7ff0880b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.695+0000 7ff090339640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08815ee40 con 0x7ff0880b35b0 2023-11-28T20:51:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.695+0000 7ff090339640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08815bda0 con 0x7ff08815b850 2023-11-28T20:51:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.695+0000 7ff090339640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08815c080 con 0x7ff0880b1fa0 2023-11-28T20:51:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08e0d5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0880b35b0 0x7ff08815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08e0d5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0880b35b0 0x7ff08815ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45672/0 (socket says 172.21.15.45:45672) 2023-11-28T20:51:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08e0d5640 1 -- 172.21.15.45:0/4224342290 learned_addr learned my addr 172.21.15.45:0/4224342290 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08d8d4640 1 --2- 172.21.15.45:0/4224342290 >> v2:172.21.15.190:3300/0 conn(0x7ff08815b850 0x7ff0880b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08e8d6640 1 --2- 172.21.15.45:0/4224342290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b1fa0 0x7ff0880b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08e0d5640 1 -- 172.21.15.45:0/4224342290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b1fa0 msgr2=0x7ff0880b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08e0d5640 1 --2- 172.21.15.45:0/4224342290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b1fa0 0x7ff0880b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08e0d5640 1 -- 172.21.15.45:0/4224342290 >> v2:172.21.15.190:3300/0 conn(0x7ff08815b850 msgr2=0x7ff0880b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08e0d5640 1 --2- 172.21.15.45:0/4224342290 >> v2:172.21.15.190:3300/0 conn(0x7ff08815b850 0x7ff0880b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.696+0000 7ff08e0d5640 1 -- 172.21.15.45:0/4224342290 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08815c360 con 0x7ff0880b35b0 2023-11-28T20:51:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.698+0000 7ff08e0d5640 1 --2- 172.21.15.45:0/4224342290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0880b35b0 0x7ff08815ca10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7ff0880cd380 tx=0x7ff08402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cd13fb928442ec server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.698+0000 7ff08d0d3640 1 -- 172.21.15.45:0/4224342290 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084031870 con 0x7ff0880b35b0 2023-11-28T20:51:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.698+0000 7ff08d0d3640 1 -- 172.21.15.45:0/4224342290 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0840319d0 con 0x7ff0880b35b0 2023-11-28T20:51:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.698+0000 7ff08d0d3640 1 -- 172.21.15.45:0/4224342290 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084031d60 con 0x7ff0880b35b0 2023-11-28T20:51:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.699+0000 7ff090339640 1 -- 172.21.15.45:0/4224342290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0880b35b0 msgr2=0x7ff08815ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.699+0000 7ff090339640 1 --2- 172.21.15.45:0/4224342290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0880b35b0 0x7ff08815ca10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7ff0880cd380 tx=0x7ff08402b910 comp rx=0 tx=0).stop 2023-11-28T20:51:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.699+0000 7ff090339640 1 -- 172.21.15.45:0/4224342290 shutdown_connections 2023-11-28T20:51:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.699+0000 7ff090339640 1 --2- 172.21.15.45:0/4224342290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b1fa0 0x7ff0880b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.699+0000 7ff090339640 1 --2- 172.21.15.45:0/4224342290 >> v2:172.21.15.190:3300/0 conn(0x7ff08815b850 0x7ff0880b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.699+0000 7ff090339640 1 --2- 172.21.15.45:0/4224342290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0880b35b0 0x7ff08815ca10 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.699+0000 7ff090339640 1 -- 172.21.15.45:0/4224342290 >> 172.21.15.45:0/4224342290 conn(0x7ff08806a500 msgr2=0x7ff08806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.699+0000 7ff090339640 1 -- 172.21.15.45:0/4224342290 shutdown_connections 2023-11-28T20:51:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.699+0000 7ff090339640 1 -- 172.21.15.45:0/4224342290 wait complete. 2023-11-28T20:51:08.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.702+0000 7ff090339640 1 Processor -- start 2023-11-28T20:51:08.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.702+0000 7ff090339640 1 -- start start 2023-11-28T20:51:08.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff090339640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0880b1fa0 0x7ff088159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:08.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff090339640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b35b0 0x7ff088141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:08.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff090339640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08815b850 0x7ff0881417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:08.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff090339640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881f1e00 con 0x7ff08815b850 2023-11-28T20:51:08.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff090339640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881f45d0 con 0x7ff0880b1fa0 2023-11-28T20:51:08.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff090339640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881f48b0 con 0x7ff0880b35b0 2023-11-28T20:51:08.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff08e8d6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08815b850 0x7ff0881417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:08.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff08e8d6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08815b850 0x7ff0881417c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45680/0 (socket says 172.21.15.45:45680) 2023-11-28T20:51:08.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff08e8d6640 1 -- 172.21.15.45:0/65048632 learned_addr learned my addr 172.21.15.45:0/65048632 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:08.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff08e0d5640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0880b1fa0 0x7ff088159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:08.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.703+0000 7ff08d8d4640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b35b0 0x7ff088141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:08.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.704+0000 7ff08e8d6640 1 -- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b35b0 msgr2=0x7ff088141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:08.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.704+0000 7ff08e8d6640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b35b0 0x7ff088141280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.704+0000 7ff08e8d6640 1 -- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0880b1fa0 msgr2=0x7ff088159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:08.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.704+0000 7ff08e8d6640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0880b1fa0 0x7ff088159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:08.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.704+0000 7ff08e8d6640 1 -- 172.21.15.45:0/65048632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0840094d0 con 0x7ff08815b850 2023-11-28T20:51:08.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.704+0000 7ff08e0d5640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0880b1fa0 0x7ff088159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:08.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.704+0000 7ff08e8d6640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08815b850 0x7ff0881417c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ff08000cd60 tx=0x7ff08000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:08.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.704+0000 7ff07effd640 1 -- 172.21.15.45:0/65048632 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080003b90 con 0x7ff08815b850 2023-11-28T20:51:08.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.704+0000 7ff07effd640 1 -- 172.21.15.45:0/65048632 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff080003d10 con 0x7ff08815b850 2023-11-28T20:51:08.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.705+0000 7ff07effd640 1 -- 172.21.15.45:0/65048632 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080004260 con 0x7ff08815b850 2023-11-28T20:51:08.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.705+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0881f4b90 con 0x7ff08815b850 2023-11-28T20:51:08.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.705+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff088158280 con 0x7ff08815b850 2023-11-28T20:51:08.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.706+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0880733d0 con 0x7ff08815b850 2023-11-28T20:51:08.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.707+0000 7ff07effd640 1 -- 172.21.15.45:0/65048632 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7ff0800043c0 con 0x7ff08815b850 2023-11-28T20:51:08.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.708+0000 7ff07effd640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff06c06aed0 0x7ff06c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:08.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.708+0000 7ff07effd640 1 -- 172.21.15.45:0/65048632 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff080012070 con 0x7ff08815b850 2023-11-28T20:51:08.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.709+0000 7ff08e0d5640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff06c06aed0 0x7ff06c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:08.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.710+0000 7ff08e0d5640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff06c06aed0 0x7ff06c06d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff084036000 tx=0x7ff08403c040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:08.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.713+0000 7ff07effd640 1 -- 172.21.15.45:0/65048632 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff080055040 con 0x7ff08815b850 2023-11-28T20:51:08.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.978+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff0880cd3b0 con 0x7ff08815b850 2023-11-28T20:51:08.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.980+0000 7ff07effd640 1 -- 172.21.15.45:0/65048632 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff080058660 con 0x7ff08815b850 2023-11-28T20:51:08.982 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:08.982 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:51:08.982 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T20:51:08.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.983 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T20:51:08.983 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T20:51:08.983 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.983 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:08.983 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:08.984 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:08.984 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.984 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.985 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T20:51:08.985 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T20:51:08.985 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.985 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:08.985 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:08.985 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:08.986 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.986 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.987 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T20:51:08.987 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T20:51:08.987 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:08.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:08.988 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:08.988 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.988 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T20:51:08.988 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T20:51:08.988 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.989 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:08.989 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:08.989 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:08.989 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.989 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.989 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T20:51:08.989 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T20:51:08.990 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.990 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:08.990 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:08.990 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.990 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T20:51:08.990 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.991 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:08.992 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:08.992 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.992 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.992 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T20:51:08.992 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T20:51:08.992 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.993 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:08.993 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:08.993 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.993 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.993 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T20:51:08.994 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T20:51:08.994 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.994 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:08.994 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:08.994 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.994 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.995 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T20:51:08.995 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T20:51:08.995 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.995 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:08.995 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:08.995 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.996 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.996 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T20:51:08.996 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T20:51:08.997 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.997 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:08.997 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:08.997 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.998 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T20:51:08.998 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T20:51:08.998 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.998 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:08.998 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T20:51:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T20:51:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:08.999 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T20:51:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T20:51:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:09.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:09.001 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T20:51:09.001 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T20:51:09.001 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:09.001 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:09.001 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:51:09.001 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:51:09.002 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:51:09.002 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:09.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:09.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:09.002 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T20:51:09.002 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T20:51:09.003 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:09.003 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:09.003 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:51:09.003 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:51:09.003 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:51:09.003 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:09.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:09.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:09.004 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T20:51:09.004 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T20:51:09.004 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:09.004 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:09.004 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:51:09.004 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:09.005 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:09.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:09.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:09.006 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T20:51:09.006 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T20:51:09.006 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:09.006 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:51:09.006 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:51:09.006 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:09.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:09.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:09.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:09.007 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T20:51:09.007 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T20:51:09.007 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:09.007 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:51:09.007 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:51:09.007 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:09.008 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:09.008 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:09.008 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T20:51:09.008 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:51:09.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.982+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff06c06aed0 msgr2=0x7ff06c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.982+0000 7ff090339640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff06c06aed0 0x7ff06c06d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff084036000 tx=0x7ff08403c040 comp rx=0 tx=0).stop 2023-11-28T20:51:09.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08815b850 msgr2=0x7ff0881417c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08815b850 0x7ff0881417c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ff08000cd60 tx=0x7ff08000a660 comp rx=0 tx=0).stop 2023-11-28T20:51:09.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 shutdown_connections 2023-11-28T20:51:09.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff06c06aed0 0x7ff06c06d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08815b850 0x7ff0881417c0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0880b35b0 0x7ff088141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 --2- 172.21.15.45:0/65048632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0880b1fa0 0x7ff088159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 >> 172.21.15.45:0/65048632 conn(0x7ff08806a500 msgr2=0x7ff08815d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:09.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 shutdown_connections 2023-11-28T20:51:09.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:08.983+0000 7ff090339640 1 -- 172.21.15.45:0/65048632 wait complete. 2023-11-28T20:51:09.012 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:09.103 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:09.097+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:09.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.245+0000 7fa09a656640 1 Processor -- start 2023-11-28T20:51:09.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.245+0000 7fa09a656640 1 -- start start 2023-11-28T20:51:09.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.247+0000 7fa09a656640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa0940aec60 0x7fa09406cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.247+0000 7fa09a656640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0940cb9f0 0x7fa09406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.247+0000 7fa09a656640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0940ce110 con 0x7fa094070ba0 2023-11-28T20:51:09.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.247+0000 7fa09a656640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0940ccb90 con 0x7fa0940cb9f0 2023-11-28T20:51:09.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.247+0000 7fa09a656640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0940ce3f0 con 0x7fa0940aec60 2023-11-28T20:51:09.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.248+0000 7fa098bf3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0940cb9f0 0x7fa09406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.248+0000 7fa098bf3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0940cb9f0 0x7fa09406d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45688/0 (socket says 172.21.15.45:45688) 2023-11-28T20:51:09.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.248+0000 7fa098bf3640 1 -- 172.21.15.45:0/2080635949 learned_addr learned my addr 172.21.15.45:0/2080635949 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:09.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.248+0000 7fa0937fe640 1 --2- 172.21.15.45:0/2080635949 >> v2:172.21.15.190:3300/0 conn(0x7fa0940aec60 0x7fa09406cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.248+0000 7fa098bf3640 1 -- 172.21.15.45:0/2080635949 >> v2:172.21.15.190:3300/0 conn(0x7fa0940aec60 msgr2=0x7fa09406cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:09.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.248+0000 7fa098bf3640 1 --2- 172.21.15.45:0/2080635949 >> v2:172.21.15.190:3300/0 conn(0x7fa0940aec60 0x7fa09406cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.248+0000 7fa098bf3640 1 -- 172.21.15.45:0/2080635949 >> v1:172.21.15.190:6789/0 conn(0x7fa094070ba0 legacy=0x7fa094070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:09.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.248+0000 7fa098bf3640 1 -- 172.21.15.45:0/2080635949 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940ce6d0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.250+0000 7fa098bf3640 1 --2- 172.21.15.45:0/2080635949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 0x7fa09406d100 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa09406c930 tx=0x7fa08c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd853f5cf7f4be11 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:09.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.250+0000 7fa092ffd640 1 -- 172.21.15.45:0/2080635949 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c0317e0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.250+0000 7fa092ffd640 1 -- 172.21.15.45:0/2080635949 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c031940 con 0x7fa0940cb9f0 2023-11-28T20:51:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.250+0000 7fa092ffd640 1 -- 172.21.15.45:0/2080635949 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c031cd0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.251+0000 7fa09a656640 1 -- 172.21.15.45:0/2080635949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 msgr2=0x7fa09406d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.251+0000 7fa09a656640 1 --2- 172.21.15.45:0/2080635949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 0x7fa09406d100 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa09406c930 tx=0x7fa08c02b880 comp rx=0 tx=0).stop 2023-11-28T20:51:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.251+0000 7fa09a656640 1 -- 172.21.15.45:0/2080635949 shutdown_connections 2023-11-28T20:51:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.251+0000 7fa09a656640 1 --2- 172.21.15.45:0/2080635949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 0x7fa09406d100 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.251+0000 7fa09a656640 1 --2- 172.21.15.45:0/2080635949 >> v2:172.21.15.190:3300/0 conn(0x7fa0940aec60 0x7fa09406cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.251+0000 7fa09a656640 1 -- 172.21.15.45:0/2080635949 >> 172.21.15.45:0/2080635949 conn(0x7fa0940b1350 msgr2=0x7fa0940b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:09.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.251+0000 7fa09a656640 1 -- 172.21.15.45:0/2080635949 shutdown_connections 2023-11-28T20:51:09.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.251+0000 7fa09a656640 1 -- 172.21.15.45:0/2080635949 wait complete. 2023-11-28T20:51:09.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.253+0000 7fa09a656640 1 Processor -- start 2023-11-28T20:51:09.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.253+0000 7fa09a656640 1 -- start start 2023-11-28T20:51:09.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.254+0000 7fa09a656640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094070ba0 0x7fa0941f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.254+0000 7fa09a656640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0940aec60 0x7fa09415bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.254+0000 7fa09a656640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 0x7fa094150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.254+0000 7fa09a656640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0941f66c0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.254+0000 7fa09a656640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09413c490 con 0x7fa094070ba0 2023-11-28T20:51:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.254+0000 7fa09a656640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09413c770 con 0x7fa0940aec60 2023-11-28T20:51:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.254+0000 7fa098bf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 0x7fa094150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.254+0000 7fa098bf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 0x7fa094150190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45698/0 (socket says 172.21.15.45:45698) 2023-11-28T20:51:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.254+0000 7fa098bf3640 1 -- 172.21.15.45:0/2798289688 learned_addr learned my addr 172.21.15.45:0/2798289688 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.255+0000 7fa093fff640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094070ba0 0x7fa0941f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.255+0000 7fa0937fe640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0940aec60 0x7fa09415bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.255+0000 7fa098bf3640 1 -- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0940aec60 msgr2=0x7fa09415bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.255+0000 7fa098bf3640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0940aec60 0x7fa09415bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.255+0000 7fa098bf3640 1 -- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094070ba0 msgr2=0x7fa0941f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.255+0000 7fa098bf3640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094070ba0 0x7fa0941f6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.255+0000 7fa098bf3640 1 -- 172.21.15.45:0/2798289688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c0094d0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.255+0000 7fa093fff640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094070ba0 0x7fa0941f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.255+0000 7fa098bf3640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 0x7fa094150190 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fa08c0090b0 tx=0x7fa08c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.256+0000 7fa090ff9640 1 -- 172.21.15.45:0/2798289688 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c032880 con 0x7fa0940cb9f0 2023-11-28T20:51:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.256+0000 7fa090ff9640 1 -- 172.21.15.45:0/2798289688 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c032a00 con 0x7fa0940cb9f0 2023-11-28T20:51:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.256+0000 7fa090ff9640 1 -- 172.21.15.45:0/2798289688 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c032db0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.256+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09413ca50 con 0x7fa0940cb9f0 2023-11-28T20:51:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.256+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09413d010 con 0x7fa0940cb9f0 2023-11-28T20:51:09.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.257+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09406e5c0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.258+0000 7fa090ff9640 1 -- 172.21.15.45:0/2798289688 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fa08c04d3f0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.259+0000 7fa090ff9640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa08006af00 0x7fa08006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.259+0000 7fa090ff9640 1 -- 172.21.15.45:0/2798289688 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa08c0ab250 con 0x7fa0940cb9f0 2023-11-28T20:51:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.259+0000 7fa093fff640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa08006af00 0x7fa08006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.261+0000 7fa093fff640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa08006af00 0x7fa08006d3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa09415e380 tx=0x7fa084009040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.263+0000 7fa090ff9640 1 -- 172.21.15.45:0/2798289688 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa08c0763a0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.466+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0940733d0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.468+0000 7fa090ff9640 1 -- 172.21.15.45:0/2798289688 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+13051 (secure 0 0 0) 0x7fa08c0799c0 con 0x7fa0940cb9f0 2023-11-28T20:51:09.470 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:09.470 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":98,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:08.439676+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:09.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa08006af00 msgr2=0x7fa08006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa08006af00 0x7fa08006d3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa09415e380 tx=0x7fa084009040 comp rx=0 tx=0).stop 2023-11-28T20:51:09.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 msgr2=0x7fa094150190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 0x7fa094150190 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fa08c0090b0 tx=0x7fa08c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 shutdown_connections 2023-11-28T20:51:09.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fa08006af00 0x7fa08006d3c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0940cb9f0 0x7fa094150190 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0940aec60 0x7fa09415bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 --2- 172.21.15.45:0/2798289688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094070ba0 0x7fa0941f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 >> 172.21.15.45:0/2798289688 conn(0x7fa0940b1350 msgr2=0x7fa0940af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:09.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 shutdown_connections 2023-11-28T20:51:09.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.471+0000 7fa09a656640 1 -- 172.21.15.45:0/2798289688 wait complete. 2023-11-28T20:51:09.494 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:51:09.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:51:09.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.723+0000 7f58cbb89640 1 Processor -- start 2023-11-28T20:51:09.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.724+0000 7f58cbb89640 1 -- start start 2023-11-28T20:51:09.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.726+0000 7f58cbb89640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f58c406ac10 0x7f58c406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.726+0000 7f58cbb89640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58c415f720 0x7f58c40717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.726+0000 7f58cbb89640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c415ecd0 con 0x7f58c415e840 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.726+0000 7f58cbb89640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c4071ce0 con 0x7f58c415f720 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.726+0000 7f58cbb89640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c415fbd0 con 0x7f58c406ac10 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.726+0000 7f58ca126640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58c415f720 0x7f58c40717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.726+0000 7f58ca126640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58c415f720 0x7f58c40717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45714/0 (socket says 172.21.15.45:45714) 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.726+0000 7f58ca126640 1 -- 172.21.15.45:0/4286307228 learned_addr learned my addr 172.21.15.45:0/4286307228 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.727+0000 7f58c9124640 1 --2- 172.21.15.45:0/4286307228 >> v2:172.21.15.190:3300/0 conn(0x7f58c406ac10 0x7f58c406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.727+0000 7f58ca126640 1 -- 172.21.15.45:0/4286307228 >> v2:172.21.15.190:3300/0 conn(0x7f58c406ac10 msgr2=0x7f58c406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:09.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.727+0000 7f58ca126640 1 --2- 172.21.15.45:0/4286307228 >> v2:172.21.15.190:3300/0 conn(0x7f58c406ac10 0x7f58c406c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.727+0000 7f58ca126640 1 -- 172.21.15.45:0/4286307228 >> v1:172.21.15.190:6789/0 conn(0x7f58c415e840 legacy=0x7f58c406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:09.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.727+0000 7f58ca126640 1 -- 172.21.15.45:0/4286307228 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c415f290 con 0x7f58c415f720 2023-11-28T20:51:09.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.728+0000 7f58ca126640 1 --2- 172.21.15.45:0/4286307228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 0x7f58c40717a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f58c406e6d0 tx=0x7f58bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5bfeb8f9f0ff54 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:09.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.728+0000 7f58c8923640 1 -- 172.21.15.45:0/4286307228 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc02ca80 con 0x7f58c415f720 2023-11-28T20:51:09.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.728+0000 7f58c8923640 1 -- 172.21.15.45:0/4286307228 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58bc02cbe0 con 0x7f58c415f720 2023-11-28T20:51:09.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.728+0000 7f58c8923640 1 -- 172.21.15.45:0/4286307228 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0343f0 con 0x7f58c415f720 2023-11-28T20:51:09.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.729+0000 7f58cbb89640 1 -- 172.21.15.45:0/4286307228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 msgr2=0x7f58c40717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.729+0000 7f58cbb89640 1 --2- 172.21.15.45:0/4286307228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 0x7f58c40717a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f58c406e6d0 tx=0x7f58bc02bee0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.729+0000 7f58cbb89640 1 -- 172.21.15.45:0/4286307228 shutdown_connections 2023-11-28T20:51:09.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.729+0000 7f58cbb89640 1 --2- 172.21.15.45:0/4286307228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 0x7f58c40717a0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.729+0000 7f58cbb89640 1 --2- 172.21.15.45:0/4286307228 >> v2:172.21.15.190:3300/0 conn(0x7f58c406ac10 0x7f58c406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.729+0000 7f58cbb89640 1 -- 172.21.15.45:0/4286307228 >> 172.21.15.45:0/4286307228 conn(0x7f58c40ced60 msgr2=0x7f58c40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:09.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.729+0000 7f58cbb89640 1 -- 172.21.15.45:0/4286307228 shutdown_connections 2023-11-28T20:51:09.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.729+0000 7f58cbb89640 1 -- 172.21.15.45:0/4286307228 wait complete. 2023-11-28T20:51:09.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.731+0000 7f58cbb89640 1 Processor -- start 2023-11-28T20:51:09.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.731+0000 7f58cbb89640 1 -- start start 2023-11-28T20:51:09.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.732+0000 7f58cbb89640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58c406ac10 0x7f58c40c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.732+0000 7f58cbb89640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58c415e840 0x7f58c40b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.732+0000 7f58cbb89640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 0x7f58c40b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.732+0000 7f58cbb89640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c40c66e0 con 0x7f58c415f720 2023-11-28T20:51:09.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.732+0000 7f58cbb89640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c40c69c0 con 0x7f58c406ac10 2023-11-28T20:51:09.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.732+0000 7f58cbb89640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c40c6ca0 con 0x7f58c415e840 2023-11-28T20:51:09.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.732+0000 7f58ca126640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 0x7f58c40b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.732+0000 7f58ca126640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 0x7f58c40b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45724/0 (socket says 172.21.15.45:45724) 2023-11-28T20:51:09.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.732+0000 7f58ca126640 1 -- 172.21.15.45:0/37405862 learned_addr learned my addr 172.21.15.45:0/37405862 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:09.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.733+0000 7f58ca126640 1 -- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58c415e840 msgr2=0x7f58c40b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.733+0000 7f58ca126640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58c415e840 0x7f58c40b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.733+0000 7f58ca126640 1 -- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58c406ac10 msgr2=0x7f58c40c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.733+0000 7f58ca126640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58c406ac10 0x7f58c40c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.733+0000 7f58ca126640 1 -- 172.21.15.45:0/37405862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc009b30 con 0x7f58c415f720 2023-11-28T20:51:09.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.733+0000 7f58ca126640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 0x7f58c40b34d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f58bc036040 tx=0x7f58bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:09.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.733+0000 7f58ba7fc640 1 -- 172.21.15.45:0/37405862 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc009510 con 0x7f58c415f720 2023-11-28T20:51:09.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.734+0000 7f58ba7fc640 1 -- 172.21.15.45:0/37405862 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58bc03e450 con 0x7f58c415f720 2023-11-28T20:51:09.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.734+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c40c6f80 con 0x7f58c415f720 2023-11-28T20:51:09.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.734+0000 7f58ba7fc640 1 -- 172.21.15.45:0/37405862 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc03e800 con 0x7f58c415f720 2023-11-28T20:51:09.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.734+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c40c7780 con 0x7f58c415f720 2023-11-28T20:51:09.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.735+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c40c7440 con 0x7f58c415f720 2023-11-28T20:51:09.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.736+0000 7f58ba7fc640 1 -- 172.21.15.45:0/37405862 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f58bc02cd50 con 0x7f58c415f720 2023-11-28T20:51:09.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.737+0000 7f58ba7fc640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f58a806aed0 0x7f58a806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:09.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.737+0000 7f58ba7fc640 1 -- 172.21.15.45:0/37405862 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f58bc0abff0 con 0x7f58c415f720 2023-11-28T20:51:09.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.737+0000 7f58c9925640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f58a806aed0 0x7f58a806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:09.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.739+0000 7f58c9925640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f58a806aed0 0x7f58a806d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f58c0003cc0 tx=0x7f58c00079b0 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:09.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.741+0000 7f58ba7fc640 1 -- 172.21.15.45:0/37405862 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58bc0dee90 con 0x7f58c415f720 2023-11-28T20:51:09.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.951+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58c40733d0 con 0x7f58c415f720 2023-11-28T20:51:09.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.952+0000 7f58ba7fc640 1 -- 172.21.15.45:0/37405862 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58bc043020 con 0x7f58c415f720 2023-11-28T20:51:09.955 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:51:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f58a806aed0 msgr2=0x7f58a806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f58a806aed0 0x7f58a806d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f58c0003cc0 tx=0x7f58c00079b0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 msgr2=0x7f58c40b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 0x7f58c40b34d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f58bc036040 tx=0x7f58bc004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 shutdown_connections 2023-11-28T20:51:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f58a806aed0 0x7f58a806d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c415f720 0x7f58c40b34d0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58c415e840 0x7f58c40b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 --2- 172.21.15.45:0/37405862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58c406ac10 0x7f58c40c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:09.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 >> 172.21.15.45:0/37405862 conn(0x7f58c40ced60 msgr2=0x7f58c40cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:09.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.955+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 shutdown_connections 2023-11-28T20:51:09.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:09.956+0000 7f58cbb89640 1 -- 172.21.15.45:0/37405862 wait complete. 2023-11-28T20:51:09.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T20:51:10.121 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:10.118+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:10.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.205+0000 7fefe08cf640 1 Processor -- start 2023-11-28T20:51:10.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.205+0000 7fefe08cf640 1 -- start start 2023-11-28T20:51:10.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.207+0000 7fefe08cf640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fefdc15b850 0x7fefdc0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.207+0000 7fefe08cf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefdc15bda0 con 0x7fefdc0b1fa0 2023-11-28T20:51:10.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.207+0000 7fefe08cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefdc15c080 con 0x7fefdc0b35b0 2023-11-28T20:51:10.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.207+0000 7fefe08cf640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefdc15ee40 con 0x7fefdc15b850 2023-11-28T20:51:10.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.207+0000 7fefdad9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fefdc0b1fa0 0x7fefdc15f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47212/0 (socket says 172.21.15.45:47212) 2023-11-28T20:51:10.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.207+0000 7fefdad9d640 1 -- 172.21.15.45:0/3187575687 learned_addr learned my addr 172.21.15.45:0/3187575687 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:10.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.207+0000 7fefd3fff640 1 --2- 172.21.15.45:0/3187575687 >> v2:172.21.15.190:3300/0 conn(0x7fefdc15b850 0x7fefdc0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666529914 0 0) 0x7fefdc15bda0 con 0x7fefdc0b1fa0 2023-11-28T20:51:10.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc8003520 con 0x7fefdc0b1fa0 2023-11-28T20:51:10.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2633052758 0 0) 0x7fefc8003520 con 0x7fefdc0b1fa0 2023-11-28T20:51:10.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 >> v2:172.21.15.190:3300/0 conn(0x7fefdc15b850 msgr2=0x7fefdc0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:10.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 --2- 172.21.15.45:0/3187575687 >> v2:172.21.15.190:3300/0 conn(0x7fefdc15b850 0x7fefdc0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 >> v1:172.21.15.190:6789/0 conn(0x7fefdc0b35b0 legacy=0x7fefdc15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc15c360 con 0x7fefdc0b1fa0 2023-11-28T20:51:10.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fefd40019f0 con 0x7fefdc0b1fa0 2023-11-28T20:51:10.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 >> v1:172.21.15.45:6789/0 conn(0x7fefdc0b1fa0 legacy=0x7fefdc15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefc8003970 0x7fefc8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefc80047c0 0x7fefc8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefc800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc15ee40 con 0x7fefc8005180 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc800c020 con 0x7fefc8003970 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.208+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc800c1e0 con 0x7fefc80047c0 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.209+0000 7fefdad9d640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefc800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.209+0000 7fefda59c640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefc8003970 0x7fefc8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.209+0000 7fefd3fff640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefc80047c0 0x7fefc8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.209+0000 7fefdad9d640 1 -- 172.21.15.45:0/3187575687 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefc80047c0 msgr2=0x7fefc8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.209+0000 7fefdad9d640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefc80047c0 0x7fefc8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.209+0000 7fefdad9d640 1 -- 172.21.15.45:0/3187575687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefc8003970 msgr2=0x7fefc8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:10.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.209+0000 7fefdad9d640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefc8003970 0x7fefc8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.209+0000 7fefdad9d640 1 -- 172.21.15.45:0/3187575687 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc8003520 con 0x7fefc8005180 2023-11-28T20:51:10.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.209+0000 7fefd3fff640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefc80047c0 0x7fefc8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:10.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.210+0000 7fefdad9d640 1 -- 172.21.15.45:0/3187575687 reap_dead start 2023-11-28T20:51:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.210+0000 7fefdad9d640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefc800b980 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fefd400a800 tx=0x7fefc80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e3ed6fce655391 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.211+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd40072d0 con 0x7fefc8005180 2023-11-28T20:51:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.211+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd402f070 con 0x7fefc8005180 2023-11-28T20:51:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.211+0000 7fefd9d9b640 1 -- 172.21.15.45:0/3187575687 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd4007670 con 0x7fefc8005180 2023-11-28T20:51:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.211+0000 7fefe08cf640 1 -- 172.21.15.45:0/3187575687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 msgr2=0x7fefc800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.211+0000 7fefe08cf640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefc800b980 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fefd400a800 tx=0x7fefc80057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.211+0000 7fefe08cf640 1 -- 172.21.15.45:0/3187575687 shutdown_connections 2023-11-28T20:51:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.211+0000 7fefe08cf640 1 --2- 172.21.15.45:0/3187575687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefc800b980 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.211+0000 7fefe08cf640 1 -- 172.21.15.45:0/3187575687 >> 172.21.15.45:0/3187575687 conn(0x7fefdc06a500 msgr2=0x7fefdc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.212+0000 7fefe08cf640 1 -- 172.21.15.45:0/3187575687 shutdown_connections 2023-11-28T20:51:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.212+0000 7fefe08cf640 1 -- 172.21.15.45:0/3187575687 wait complete. 2023-11-28T20:51:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.214+0000 7fefe08cf640 1 Processor -- start 2023-11-28T20:51:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.214+0000 7fefe08cf640 1 -- start start 2023-11-28T20:51:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefe08cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefdc0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefe08cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefdc0c94a0 0x7fefdc0c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefe08cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefdc1357d0 0x7fefdc135c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefe08cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc15ac60 con 0x7fefc8005180 2023-11-28T20:51:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefe08cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc0c9e60 con 0x7fefdc1357d0 2023-11-28T20:51:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefe08cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc137130 con 0x7fefdc0c94a0 2023-11-28T20:51:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefda59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefdc0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefda59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefdc0c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45732/0 (socket says 172.21.15.45:45732) 2023-11-28T20:51:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefda59c640 1 -- 172.21.15.45:0/4176314096 learned_addr learned my addr 172.21.15.45:0/4176314096 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefda59c640 1 -- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefdc0c94a0 msgr2=0x7fefdc0c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefda59c640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefdc0c94a0 0x7fefdc0c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefda59c640 1 -- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefdc1357d0 msgr2=0x7fefdc135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefda59c640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefdc1357d0 0x7fefdc135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.215+0000 7fefda59c640 1 -- 172.21.15.45:0/4176314096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc1375c0 con 0x7fefc8005180 2023-11-28T20:51:10.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.216+0000 7fefda59c640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefdc0c8cc0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e4b0 tx=0x7fefcc00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:10.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.216+0000 7fefd2ffd640 1 -- 172.21.15.45:0/4176314096 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc0052e0 con 0x7fefc8005180 2023-11-28T20:51:10.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.216+0000 7fefd2ffd640 1 -- 172.21.15.45:0/4176314096 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefcc005440 con 0x7fefc8005180 2023-11-28T20:51:10.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.216+0000 7fefd2ffd640 1 -- 172.21.15.45:0/4176314096 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc003830 con 0x7fefc8005180 2023-11-28T20:51:10.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.216+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefdc136520 con 0x7fefc8005180 2023-11-28T20:51:10.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.216+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefdc136ae0 con 0x7fefc8005180 2023-11-28T20:51:10.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.217+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefdc0733d0 con 0x7fefc8005180 2023-11-28T20:51:10.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.218+0000 7fefd2ffd640 1 -- 172.21.15.45:0/4176314096 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fefcc003990 con 0x7fefc8005180 2023-11-28T20:51:10.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.219+0000 7fefd2ffd640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fefbc06af20 0x7fefbc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.219+0000 7fefd2ffd640 1 -- 172.21.15.45:0/4176314096 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fefcc0891b0 con 0x7fefc8005180 2023-11-28T20:51:10.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.220+0000 7fefd3fff640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fefbc06af20 0x7fefbc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.221+0000 7fefd3fff640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fefbc06af20 0x7fefbc06d3e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fefdc06ae70 tx=0x7fefc4008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:10.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.224+0000 7fefd2ffd640 1 -- 172.21.15.45:0/4176314096 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fefcc054400 con 0x7fefc8005180 2023-11-28T20:51:10.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.430+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fefdc0cd3b0 con 0x7fefc8005180 2023-11-28T20:51:10.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.433+0000 7fefd2ffd640 1 -- 172.21.15.45:0/4176314096 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v98) v1 ==== 84+0+65 (secure 0 0 0) 0x7fefcc057a20 con 0x7fefc8005180 2023-11-28T20:51:10.435 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:10.435 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:51:10.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:10.436 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T20:51:10.436 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T20:51:10.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:10.436 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:51:10.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fefbc06af20 msgr2=0x7fefbc06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fefbc06af20 0x7fefbc06d3e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fefdc06ae70 tx=0x7fefc4008040 comp rx=0 tx=0).stop 2023-11-28T20:51:10.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 msgr2=0x7fefdc0c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefdc0c8cc0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e4b0 tx=0x7fefcc00e970 comp rx=0 tx=0).stop 2023-11-28T20:51:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 shutdown_connections 2023-11-28T20:51:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fefbc06af20 0x7fefbc06d3e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefdc1357d0 0x7fefdc135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefdc0c94a0 0x7fefdc0c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 --2- 172.21.15.45:0/4176314096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefc8005180 0x7fefdc0c8cc0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 >> 172.21.15.45:0/4176314096 conn(0x7fefdc06a500 msgr2=0x7fefdc070330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 shutdown_connections 2023-11-28T20:51:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.436+0000 7fefe08cf640 1 -- 172.21.15.45:0/4176314096 wait complete. 2023-11-28T20:51:10.457 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T20:51:10.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T20:51:10.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.693+0000 7fd3f7d78640 1 Processor -- start 2023-11-28T20:51:10.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.693+0000 7fd3f7d78640 1 -- start start 2023-11-28T20:51:10.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.695+0000 7fd3f7d78640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015e840 0x7fd3f006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.695+0000 7fd3f7d78640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f015ecd0 con 0x7fd3f006ac10 2023-11-28T20:51:10.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.695+0000 7fd3f7d78640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f015f290 con 0x7fd3f015f720 2023-11-28T20:51:10.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.695+0000 7fd3f7d78640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f015fb60 con 0x7fd3f015e840 2023-11-28T20:51:10.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.695+0000 7fd3f6315640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd3f006ac10 0x7fd3f00cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47218/0 (socket says 172.21.15.45:47218) 2023-11-28T20:51:10.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.695+0000 7fd3f6315640 1 -- 172.21.15.45:0/1430373463 learned_addr learned my addr 172.21.15.45:0/1430373463 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:10.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.696+0000 7fd3f5b14640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015e840 0x7fd3f006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.696+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885497715 0 0) 0x7fd3f015ecd0 con 0x7fd3f006ac10 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.696+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3dc003520 con 0x7fd3f006ac10 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.696+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1164060117 0 0) 0x7fd3dc003520 con 0x7fd3f006ac10 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.696+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015e840 msgr2=0x7fd3f006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.696+0000 7fd3f4b12640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015e840 0x7fd3f006c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.696+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 >> v1:172.21.15.190:6789/0 conn(0x7fd3f015f720 legacy=0x7fd3f006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.696+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f0072020 con 0x7fd3f006ac10 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.696+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd3e80019f0 con 0x7fd3f006ac10 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 >> v1:172.21.15.45:6789/0 conn(0x7fd3f006ac10 legacy=0x7fd3f00cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f4b12640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc0039a0 0x7fd3dc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f4b12640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015e840 0x7fd3dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f4b12640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f015fb60 con 0x7fd3dc004d00 2023-11-28T20:51:10.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc00ba30 con 0x7fd3dc0039a0 2023-11-28T20:51:10.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc00bbf0 con 0x7fd3f015e840 2023-11-28T20:51:10.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f6315640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f5b14640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc0039a0 0x7fd3dc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f5313640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015e840 0x7fd3dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f6315640 1 -- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015e840 msgr2=0x7fd3dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:10.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f6315640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015e840 0x7fd3dc0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f6315640 1 -- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc0039a0 msgr2=0x7fd3dc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:10.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f6315640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3dc0039a0 0x7fd3dc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.697+0000 7fd3f6315640 1 -- 172.21.15.45:0/1430373463 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc003520 con 0x7fd3dc004d00 2023-11-28T20:51:10.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.699+0000 7fd3f6315640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3dc005180 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800a800 tx=0x7fd3dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546c447ece5a26e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:10.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.699+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e80072d0 con 0x7fd3dc004d00 2023-11-28T20:51:10.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.699+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e802f070 con 0x7fd3dc004d00 2023-11-28T20:51:10.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.699+0000 7fd3f4b12640 1 -- 172.21.15.45:0/1430373463 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e80076b0 con 0x7fd3dc004d00 2023-11-28T20:51:10.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.700+0000 7fd3f7d78640 1 -- 172.21.15.45:0/1430373463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 msgr2=0x7fd3dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.700+0000 7fd3f7d78640 1 --2- 172.21.15.45:0/1430373463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3dc005180 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800a800 tx=0x7fd3dc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.700+0000 7fd3f6315640 1 -- 172.21.15.45:0/1430373463 reap_dead start 2023-11-28T20:51:10.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.700+0000 7fd3f7d78640 1 -- 172.21.15.45:0/1430373463 shutdown_connections 2023-11-28T20:51:10.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.700+0000 7fd3f7d78640 1 -- 172.21.15.45:0/1430373463 >> 172.21.15.45:0/1430373463 conn(0x7fd3f00ced60 msgr2=0x7fd3f00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:10.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.700+0000 7fd3f7d78640 1 -- 172.21.15.45:0/1430373463 shutdown_connections 2023-11-28T20:51:10.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.700+0000 7fd3f7d78640 1 -- 172.21.15.45:0/1430373463 wait complete. 2023-11-28T20:51:10.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.702+0000 7fd3f7d78640 1 Processor -- start 2023-11-28T20:51:10.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.702+0000 7fd3f7d78640 1 -- start start 2023-11-28T20:51:10.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f7d78640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3f00ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f7d78640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3f015bc90 0x7fd3f00b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f7d78640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015c110 0x7fd3f00b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f7d78640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f00c5c50 con 0x7fd3dc004d00 2023-11-28T20:51:10.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f7d78640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f015d6b0 con 0x7fd3f015bc90 2023-11-28T20:51:10.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f7d78640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f015c610 con 0x7fd3f015c110 2023-11-28T20:51:10.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f5b14640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3f00ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f5b14640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3f00ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45752/0 (socket says 172.21.15.45:45752) 2023-11-28T20:51:10.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f5b14640 1 -- 172.21.15.45:0/2367653288 learned_addr learned my addr 172.21.15.45:0/2367653288 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f5b14640 1 -- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015c110 msgr2=0x7fd3f00b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f5b14640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015c110 0x7fd3f00b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f5313640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3f015bc90 0x7fd3f00b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f5b14640 1 -- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3f015bc90 msgr2=0x7fd3f00b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f5b14640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3f015bc90 0x7fd3f00b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.703+0000 7fd3f5b14640 1 -- 172.21.15.45:0/2367653288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f015c8f0 con 0x7fd3dc004d00 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.704+0000 7fd3f5b14640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3f00ca510 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0115b0 tx=0x7fd3ec011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.704+0000 7fd3e67fc640 1 -- 172.21.15.45:0/2367653288 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec015870 con 0x7fd3dc004d00 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.704+0000 7fd3e67fc640 1 -- 172.21.15.45:0/2367653288 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec0159d0 con 0x7fd3dc004d00 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.704+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f015cbd0 con 0x7fd3dc004d00 2023-11-28T20:51:10.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.704+0000 7fd3e67fc640 1 -- 172.21.15.45:0/2367653288 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec015d40 con 0x7fd3dc004d00 2023-11-28T20:51:10.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.704+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f015d190 con 0x7fd3dc004d00 2023-11-28T20:51:10.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.705+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f00719b0 con 0x7fd3dc004d00 2023-11-28T20:51:10.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.706+0000 7fd3e67fc640 1 -- 172.21.15.45:0/2367653288 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fd3ec00f480 con 0x7fd3dc004d00 2023-11-28T20:51:10.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.707+0000 7fd3e67fc640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd3d406aed0 0x7fd3d406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:10.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.707+0000 7fd3e67fc640 1 -- 172.21.15.45:0/2367653288 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd3ec089ce0 con 0x7fd3dc004d00 2023-11-28T20:51:10.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.708+0000 7fd3f5313640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd3d406aed0 0x7fd3d406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:10.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.709+0000 7fd3f5313640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd3d406aed0 0x7fd3d406d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd3f01f6860 tx=0x7fd3e0009040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:10.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.712+0000 7fd3e67fc640 1 -- 172.21.15.45:0/2367653288 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3ec054f40 con 0x7fd3dc004d00 2023-11-28T20:51:10.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:10.915+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd3f00733d0 con 0x7fd3dc004d00 2023-11-28T20:51:11.176 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:11.173+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:11.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.446+0000 7fd3e67fc640 1 -- 172.21.15.45:0/2367653288 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v99) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd3ec058560 con 0x7fd3dc004d00 2023-11-28T20:51:11.449 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T20:51:11.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd3d406aed0 msgr2=0x7fd3d406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd3d406aed0 0x7fd3d406d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd3f01f6860 tx=0x7fd3e0009040 comp rx=0 tx=0).stop 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 msgr2=0x7fd3f00ca510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3f00ca510 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0115b0 tx=0x7fd3ec011a70 comp rx=0 tx=0).stop 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 shutdown_connections 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fd3d406aed0 0x7fd3d406d390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3f015c110 0x7fd3f00b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3f015bc90 0x7fd3f00b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 --2- 172.21.15.45:0/2367653288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3dc004d00 0x7fd3f00ca510 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 >> 172.21.15.45:0/2367653288 conn(0x7fd3f00ced60 msgr2=0x7fd3f00ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:11.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 shutdown_connections 2023-11-28T20:51:11.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.450+0000 7fd3f7d78640 1 -- 172.21.15.45:0/2367653288 wait complete. 2023-11-28T20:51:11.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T20:51:11.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.695+0000 7fb443b51640 1 Processor -- start 2023-11-28T20:51:11.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.695+0000 7fb443b51640 1 -- start start 2023-11-28T20:51:11.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.697+0000 7fb443b51640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15e840 0x7fb43c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:11.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.697+0000 7fb443b51640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb43c15f720 0x7fb43c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:11.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.697+0000 7fb443b51640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43c15ecd0 con 0x7fb43c06a500 2023-11-28T20:51:11.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.697+0000 7fb443b51640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43c071ce0 con 0x7fb43c15f720 2023-11-28T20:51:11.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.697+0000 7fb443b51640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c15f290 con 0x7fb43c15e840 2023-11-28T20:51:11.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.697+0000 7fb4410ec640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb43c15f720 0x7fb43c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:11.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.697+0000 7fb4410ec640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb43c15f720 0x7fb43c0717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45754/0 (socket says 172.21.15.45:45754) 2023-11-28T20:51:11.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.697+0000 7fb4410ec640 1 -- 172.21.15.45:0/1522548961 learned_addr learned my addr 172.21.15.45:0/1522548961 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:11.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.698+0000 7fb4418ed640 1 --2- 172.21.15.45:0/1522548961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15e840 0x7fb43c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:11.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.698+0000 7fb4410ec640 1 -- 172.21.15.45:0/1522548961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15e840 msgr2=0x7fb43c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:11.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.698+0000 7fb4410ec640 1 --2- 172.21.15.45:0/1522548961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15e840 0x7fb43c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:11.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.698+0000 7fb4410ec640 1 -- 172.21.15.45:0/1522548961 >> v1:172.21.15.190:6789/0 conn(0x7fb43c06a500 legacy=0x7fb43c0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:11.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.698+0000 7fb4410ec640 1 -- 172.21.15.45:0/1522548961 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43c15fb60 con 0x7fb43c15f720 2023-11-28T20:51:11.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.700+0000 7fb4410ec640 1 --2- 172.21.15.45:0/1522548961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15f720 0x7fb43c0717a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0098a0 tx=0x7fb42c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7caad61378a685 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:11.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.700+0000 7fb4408eb640 1 -- 172.21.15.45:0/1522548961 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c030840 con 0x7fb43c15f720 2023-11-28T20:51:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.700+0000 7fb4408eb640 1 -- 172.21.15.45:0/1522548961 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c0309a0 con 0x7fb43c15f720 2023-11-28T20:51:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.700+0000 7fb4408eb640 1 -- 172.21.15.45:0/1522548961 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c030d30 con 0x7fb43c15f720 2023-11-28T20:51:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.701+0000 7fb443b51640 1 -- 172.21.15.45:0/1522548961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15f720 msgr2=0x7fb43c0717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.701+0000 7fb443b51640 1 --2- 172.21.15.45:0/1522548961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15f720 0x7fb43c0717a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0098a0 tx=0x7fb42c02b980 comp rx=0 tx=0).stop 2023-11-28T20:51:11.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.701+0000 7fb443b51640 1 -- 172.21.15.45:0/1522548961 shutdown_connections 2023-11-28T20:51:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.701+0000 7fb443b51640 1 --2- 172.21.15.45:0/1522548961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15f720 0x7fb43c0717a0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.701+0000 7fb443b51640 1 --2- 172.21.15.45:0/1522548961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15e840 0x7fb43c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.701+0000 7fb443b51640 1 -- 172.21.15.45:0/1522548961 >> 172.21.15.45:0/1522548961 conn(0x7fb43c0ced60 msgr2=0x7fb43c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.701+0000 7fb443b51640 1 -- 172.21.15.45:0/1522548961 shutdown_connections 2023-11-28T20:51:11.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.701+0000 7fb443b51640 1 -- 172.21.15.45:0/1522548961 wait complete. 2023-11-28T20:51:11.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.703+0000 7fb443b51640 1 Processor -- start 2023-11-28T20:51:11.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.703+0000 7fb443b51640 1 -- start start 2023-11-28T20:51:11.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb443b51640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb43c06a500 0x7fb43c0b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:11.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb443b51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15e840 0x7fb43c1f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:11.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb443b51640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15f720 0x7fb43c1f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:11.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb443b51640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c0b38c0 con 0x7fb43c15e840 2023-11-28T20:51:11.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb443b51640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c0c57a0 con 0x7fb43c06a500 2023-11-28T20:51:11.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb443b51640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c0b22e0 con 0x7fb43c15f720 2023-11-28T20:51:11.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb4410ec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15e840 0x7fb43c1f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:11.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb4410ec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15e840 0x7fb43c1f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45760/0 (socket says 172.21.15.45:45760) 2023-11-28T20:51:11.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb4410ec640 1 -- 172.21.15.45:0/383190248 learned_addr learned my addr 172.21.15.45:0/383190248 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:11.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb4420ee640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15f720 0x7fb43c1f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:11.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.704+0000 7fb4418ed640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb43c06a500 0x7fb43c0b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:11.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb4410ec640 1 -- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15f720 msgr2=0x7fb43c1f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:11.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb4410ec640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15f720 0x7fb43c1f71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:11.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb4410ec640 1 -- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb43c06a500 msgr2=0x7fb43c0b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:11.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb4410ec640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb43c06a500 0x7fb43c0b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:11.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb4410ec640 1 -- 172.21.15.45:0/383190248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c0094d0 con 0x7fb43c15e840 2023-11-28T20:51:11.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb4410ec640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15e840 0x7fb43c1f6c90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb42c000c00 tx=0x7fb42c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:11.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb4327fc640 1 -- 172.21.15.45:0/383190248 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c039a90 con 0x7fb43c15e840 2023-11-28T20:51:11.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb4327fc640 1 -- 172.21.15.45:0/383190248 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c039bf0 con 0x7fb43c15e840 2023-11-28T20:51:11.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb4327fc640 1 -- 172.21.15.45:0/383190248 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c043410 con 0x7fb43c15e840 2023-11-28T20:51:11.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.705+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43c0b25c0 con 0x7fb43c15e840 2023-11-28T20:51:11.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.706+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43c0b2b80 con 0x7fb43c15e840 2023-11-28T20:51:11.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.707+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43c0733d0 con 0x7fb43c15e840 2023-11-28T20:51:11.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.708+0000 7fb4327fc640 1 -- 172.21.15.45:0/383190248 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fb42c02f390 con 0x7fb43c15e840 2023-11-28T20:51:11.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.709+0000 7fb4327fc640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb42006aed0 0x7fb42006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:11.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.709+0000 7fb4327fc640 1 -- 172.21.15.45:0/383190248 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5547+0+0 (secure 0 0 0) 0x7fb42c0abd30 con 0x7fb43c15e840 2023-11-28T20:51:11.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.710+0000 7fb4418ed640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb42006aed0 0x7fb42006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:11.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.711+0000 7fb4418ed640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb42006aed0 0x7fb42006d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb438006fd0 tx=0x7fb438008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:11.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.714+0000 7fb4327fc640 1 -- 172.21.15.45:0/383190248 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb42c076f20 con 0x7fb43c15e840 2023-11-28T20:51:11.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:11.918+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb43c0c9120 con 0x7fb43c15e840 2023-11-28T20:51:12.238 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:12.235+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:12.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.461+0000 7fb4327fc640 1 -- 172.21.15.45:0/383190248 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v100) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb42c07a540 con 0x7fb43c15e840 2023-11-28T20:51:12.463 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T20:51:12.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.465+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb42006aed0 msgr2=0x7fb42006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:12.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.465+0000 7fb443b51640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb42006aed0 0x7fb42006d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb438006fd0 tx=0x7fb438008040 comp rx=0 tx=0).stop 2023-11-28T20:51:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.465+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15e840 msgr2=0x7fb43c1f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.465+0000 7fb443b51640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15e840 0x7fb43c1f6c90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb42c000c00 tx=0x7fb42c004730 comp rx=0 tx=0).stop 2023-11-28T20:51:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.466+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 shutdown_connections 2023-11-28T20:51:12.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.466+0000 7fb443b51640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb42006aed0 0x7fb42006d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.466+0000 7fb443b51640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb43c15f720 0x7fb43c1f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.466+0000 7fb443b51640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb43c15e840 0x7fb43c1f6c90 secure :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb42c000c00 tx=0x7fb42c004730 comp rx=0 tx=0).stop 2023-11-28T20:51:12.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.466+0000 7fb443b51640 1 --2- 172.21.15.45:0/383190248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb43c06a500 0x7fb43c0b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.466+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 >> 172.21.15.45:0/383190248 conn(0x7fb43c0ced60 msgr2=0x7fb43c06da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:12.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.467+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 shutdown_connections 2023-11-28T20:51:12.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.467+0000 7fb443b51640 1 -- 172.21.15.45:0/383190248 wait complete. 2023-11-28T20:51:12.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T20:51:12.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.699+0000 7fba1f397640 1 Processor -- start 2023-11-28T20:51:12.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.699+0000 7fba1f397640 1 -- start start 2023-11-28T20:51:12.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.701+0000 7fba1f397640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba180aec60 0x7fba1815c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:12.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.701+0000 7fba1f397640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806cbe0 0x7fba1806e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:12.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.701+0000 7fba1f397640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fba1806e740 0x7fba180cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:12.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.701+0000 7fba1f397640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1815c5d0 con 0x7fba180aec60 2023-11-28T20:51:12.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.701+0000 7fba1f397640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1806d160 con 0x7fba1806e740 2023-11-28T20:51:12.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.701+0000 7fba1f397640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1806d440 con 0x7fba1806cbe0 2023-11-28T20:51:12.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.701+0000 7fba1d133640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba180aec60 0x7fba1815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:12.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.702+0000 7fba1d133640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba180aec60 0x7fba1815c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45762/0 (socket says 172.21.15.45:45762) 2023-11-28T20:51:12.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.702+0000 7fba1d133640 1 -- 172.21.15.45:0/3856786670 learned_addr learned my addr 172.21.15.45:0/3856786670 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:12.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.702+0000 7fba1d934640 1 --2- 172.21.15.45:0/3856786670 >> v2:172.21.15.190:3300/0 conn(0x7fba1806e740 0x7fba180cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:12.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.702+0000 7fba1c932640 1 --2- 172.21.15.45:0/3856786670 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806cbe0 0x7fba1806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:12.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.702+0000 7fba1d133640 1 -- 172.21.15.45:0/3856786670 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806cbe0 msgr2=0x7fba1806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:12.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.702+0000 7fba1d133640 1 --2- 172.21.15.45:0/3856786670 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806cbe0 0x7fba1806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.702+0000 7fba1d133640 1 -- 172.21.15.45:0/3856786670 >> v2:172.21.15.190:3300/0 conn(0x7fba1806e740 msgr2=0x7fba180cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:12.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.702+0000 7fba1d133640 1 --2- 172.21.15.45:0/3856786670 >> v2:172.21.15.190:3300/0 conn(0x7fba1806e740 0x7fba180cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.702+0000 7fba1d133640 1 -- 172.21.15.45:0/3856786670 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1806d720 con 0x7fba180aec60 2023-11-28T20:51:12.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.703+0000 7fba1d133640 1 --2- 172.21.15.45:0/3856786670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba180aec60 0x7fba1815c090 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fba180cdd60 tx=0x7fba1402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965f3ad4b2a4665c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.704+0000 7fba0bfff640 1 -- 172.21.15.45:0/3856786670 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba140317e0 con 0x7fba180aec60 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.704+0000 7fba0bfff640 1 -- 172.21.15.45:0/3856786670 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba14031940 con 0x7fba180aec60 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.704+0000 7fba0bfff640 1 -- 172.21.15.45:0/3856786670 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14031cd0 con 0x7fba180aec60 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.704+0000 7fba1f397640 1 -- 172.21.15.45:0/3856786670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba180aec60 msgr2=0x7fba1815c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.704+0000 7fba1f397640 1 --2- 172.21.15.45:0/3856786670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba180aec60 0x7fba1815c090 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fba180cdd60 tx=0x7fba1402b880 comp rx=0 tx=0).stop 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.705+0000 7fba1f397640 1 -- 172.21.15.45:0/3856786670 shutdown_connections 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.705+0000 7fba1f397640 1 --2- 172.21.15.45:0/3856786670 >> v2:172.21.15.190:3300/0 conn(0x7fba1806e740 0x7fba180cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.705+0000 7fba1f397640 1 --2- 172.21.15.45:0/3856786670 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806cbe0 0x7fba1806e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.705+0000 7fba1f397640 1 --2- 172.21.15.45:0/3856786670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba180aec60 0x7fba1815c090 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.705+0000 7fba1f397640 1 -- 172.21.15.45:0/3856786670 >> 172.21.15.45:0/3856786670 conn(0x7fba1806a500 msgr2=0x7fba1806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.705+0000 7fba1f397640 1 -- 172.21.15.45:0/3856786670 shutdown_connections 2023-11-28T20:51:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.705+0000 7fba1f397640 1 -- 172.21.15.45:0/3856786670 wait complete. 2023-11-28T20:51:12.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.706+0000 7fba1f397640 1 Processor -- start 2023-11-28T20:51:12.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.707+0000 7fba1f397640 1 -- start start 2023-11-28T20:51:12.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.707+0000 7fba1f397640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba1806cbe0 0x7fba181f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1f397640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806e740 0x7fba180aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1f397640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba180aec60 0x7fba180ab0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1f397640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1813d180 con 0x7fba1806cbe0 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1f397640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1813f2e0 con 0x7fba180aec60 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1f397640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1813f5c0 con 0x7fba1806e740 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1d133640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba1806cbe0 0x7fba181f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1d133640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba1806cbe0 0x7fba181f5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45764/0 (socket says 172.21.15.45:45764) 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1d133640 1 -- 172.21.15.45:0/3007417711 learned_addr learned my addr 172.21.15.45:0/3007417711 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1c932640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806e740 0x7fba180aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1d934640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba180aec60 0x7fba180ab0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:12.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1d133640 1 -- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806e740 msgr2=0x7fba180aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1d133640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806e740 0x7fba180aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1d133640 1 -- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba180aec60 msgr2=0x7fba180ab0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1d133640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba180aec60 0x7fba180ab0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.708+0000 7fba1d133640 1 -- 172.21.15.45:0/3007417711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba140094d0 con 0x7fba1806cbe0 2023-11-28T20:51:12.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.709+0000 7fba1d133640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba1806cbe0 0x7fba181f5b00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fba14000c00 tx=0x7fba140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:12.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.709+0000 7fba09ffb640 1 -- 172.21.15.45:0/3007417711 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14032880 con 0x7fba1806cbe0 2023-11-28T20:51:12.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.709+0000 7fba09ffb640 1 -- 172.21.15.45:0/3007417711 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba14032a20 con 0x7fba1806cbe0 2023-11-28T20:51:12.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.709+0000 7fba09ffb640 1 -- 172.21.15.45:0/3007417711 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14032dd0 con 0x7fba1806cbe0 2023-11-28T20:51:12.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.709+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1813f8a0 con 0x7fba1806cbe0 2023-11-28T20:51:12.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.709+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1813fe60 con 0x7fba1806cbe0 2023-11-28T20:51:12.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.710+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba180a8660 con 0x7fba1806cbe0 2023-11-28T20:51:12.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.712+0000 7fba09ffb640 1 -- 172.21.15.45:0/3007417711 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fba1404e450 con 0x7fba1806cbe0 2023-11-28T20:51:12.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.713+0000 7fba09ffb640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb9fc06aed0 0x7fb9fc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:12.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.713+0000 7fba09ffb640 1 -- 172.21.15.45:0/3007417711 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fba140ab9e0 con 0x7fba1806cbe0 2023-11-28T20:51:12.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.713+0000 7fba1c932640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb9fc06aed0 0x7fb9fc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:12.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.715+0000 7fba1c932640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb9fc06aed0 0x7fb9fc06d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba1806e370 tx=0x7fba0c008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:12.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.718+0000 7fba09ffb640 1 -- 172.21.15.45:0/3007417711 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba140768f0 con 0x7fba1806cbe0 2023-11-28T20:51:12.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:12.951+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fba180733d0 con 0x7fba1806cbe0 2023-11-28T20:51:13.320 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:13.317+0000 7f1ad1a29640 -1 client.5576: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5917:228a80e3-f364-4864-86fc-00aabbe0cffc:v1:172.21.15.190:0/1852415784] 2023-11-28T20:51:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.467+0000 7fba09ffb640 1 -- 172.21.15.45:0/3007417711 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v41) v1 ==== 174+0+0 (secure 0 0 0) 0x7fba14079f10 con 0x7fba1806cbe0 2023-11-28T20:51:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 11 and data pool 12 2023-11-28T20:51:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb9fc06aed0 msgr2=0x7fb9fc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb9fc06aed0 0x7fb9fc06d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba1806e370 tx=0x7fba0c008040 comp rx=0 tx=0).stop 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba1806cbe0 msgr2=0x7fba181f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba1806cbe0 0x7fba181f5b00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fba14000c00 tx=0x7fba140046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 shutdown_connections 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fb9fc06aed0 0x7fb9fc06d390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba180aec60 0x7fba180ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba1806e740 0x7fba180aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 --2- 172.21.15.45:0/3007417711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba1806cbe0 0x7fba181f5b00 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.471+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 >> 172.21.15.45:0/3007417711 conn(0x7fba1806a500 msgr2=0x7fba180b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.472+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 shutdown_connections 2023-11-28T20:51:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.472+0000 7fba1f397640 1 -- 172.21.15.45:0/3007417711 wait complete. 2023-11-28T20:51:13.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:13.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.708+0000 7ff18c8f3640 1 Processor -- start 2023-11-28T20:51:13.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.708+0000 7ff18c8f3640 1 -- start start 2023-11-28T20:51:13.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.709+0000 7ff18c8f3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff1880b35b0 0x7ff18815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:13.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.709+0000 7ff18c8f3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1880b1340 0x7ff18815f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:13.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.709+0000 7ff18c8f3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18815bda0 con 0x7ff18815b850 2023-11-28T20:51:13.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.709+0000 7ff18c8f3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18815ee40 con 0x7ff1880b1340 2023-11-28T20:51:13.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.709+0000 7ff18c8f3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18815c080 con 0x7ff1880b35b0 2023-11-28T20:51:13.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.710+0000 7ff186d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1880b1340 0x7ff18815f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:13.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.710+0000 7ff186d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1880b1340 0x7ff18815f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45778/0 (socket says 172.21.15.45:45778) 2023-11-28T20:51:13.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.710+0000 7ff186d9d640 1 -- 172.21.15.45:0/966416175 learned_addr learned my addr 172.21.15.45:0/966416175 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:13.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.710+0000 7ff18659c640 1 --2- 172.21.15.45:0/966416175 >> v2:172.21.15.190:3300/0 conn(0x7ff1880b35b0 0x7ff18815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:13.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.710+0000 7ff186d9d640 1 -- 172.21.15.45:0/966416175 >> v2:172.21.15.190:3300/0 conn(0x7ff1880b35b0 msgr2=0x7ff18815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:13.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.710+0000 7ff186d9d640 1 --2- 172.21.15.45:0/966416175 >> v2:172.21.15.190:3300/0 conn(0x7ff1880b35b0 0x7ff18815ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.710+0000 7ff186d9d640 1 -- 172.21.15.45:0/966416175 >> v1:172.21.15.190:6789/0 conn(0x7ff18815b850 legacy=0x7ff1880b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:13.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.710+0000 7ff186d9d640 1 -- 172.21.15.45:0/966416175 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18815c360 con 0x7ff1880b1340 2023-11-28T20:51:13.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.710+0000 7ff18659c640 1 --2- 172.21.15.45:0/966416175 >> v2:172.21.15.190:3300/0 conn(0x7ff1880b35b0 0x7ff18815ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:13.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.711+0000 7ff186d9d640 1 --2- 172.21.15.45:0/966416175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 0x7ff18815f290 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ff18806e7f0 tx=0x7ff18002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c2b7ce0c5a5641 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:13.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.712+0000 7ff185d9b640 1 -- 172.21.15.45:0/966416175 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18002ca80 con 0x7ff1880b1340 2023-11-28T20:51:13.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.712+0000 7ff185d9b640 1 -- 172.21.15.45:0/966416175 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18002cbe0 con 0x7ff1880b1340 2023-11-28T20:51:13.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.712+0000 7ff185d9b640 1 -- 172.21.15.45:0/966416175 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1800343f0 con 0x7ff1880b1340 2023-11-28T20:51:13.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.712+0000 7ff18c8f3640 1 -- 172.21.15.45:0/966416175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 msgr2=0x7ff18815f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:13.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.712+0000 7ff18c8f3640 1 --2- 172.21.15.45:0/966416175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 0x7ff18815f290 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ff18806e7f0 tx=0x7ff18002bee0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.712+0000 7ff18c8f3640 1 -- 172.21.15.45:0/966416175 shutdown_connections 2023-11-28T20:51:13.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.712+0000 7ff18c8f3640 1 --2- 172.21.15.45:0/966416175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 0x7ff18815f290 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.712+0000 7ff18c8f3640 1 --2- 172.21.15.45:0/966416175 >> v2:172.21.15.190:3300/0 conn(0x7ff1880b35b0 0x7ff18815ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.712+0000 7ff18c8f3640 1 -- 172.21.15.45:0/966416175 >> 172.21.15.45:0/966416175 conn(0x7ff18806a500 msgr2=0x7ff18806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:13.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.713+0000 7ff18c8f3640 1 -- 172.21.15.45:0/966416175 shutdown_connections 2023-11-28T20:51:13.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.713+0000 7ff18c8f3640 1 -- 172.21.15.45:0/966416175 wait complete. 2023-11-28T20:51:13.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.714+0000 7ff18c8f3640 1 Processor -- start 2023-11-28T20:51:13.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.715+0000 7ff18c8f3640 1 -- start start 2023-11-28T20:51:13.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.715+0000 7ff18c8f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 0x7ff188138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:13.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.715+0000 7ff18c8f3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1880b35b0 0x7ff1880addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:13.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.715+0000 7ff18c8f3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff18815b850 0x7ff1880abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:13.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18c8f3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880a84a0 con 0x7ff1880b1340 2023-11-28T20:51:13.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18c8f3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880a9300 con 0x7ff1880b35b0 2023-11-28T20:51:13.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18c8f3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff188139200 con 0x7ff18815b850 2023-11-28T20:51:13.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 0x7ff188138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 0x7ff188138cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45788/0 (socket says 172.21.15.45:45788) 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18659c640 1 -- 172.21.15.45:0/1157184211 learned_addr learned my addr 172.21.15.45:0/1157184211 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff186d9d640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff18815b850 0x7ff1880abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff17ffff640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1880b35b0 0x7ff1880addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18659c640 1 -- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff18815b850 msgr2=0x7ff1880abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18659c640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff18815b850 0x7ff1880abd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18659c640 1 -- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1880b35b0 msgr2=0x7ff1880addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18659c640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1880b35b0 0x7ff1880addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff18659c640 1 -- 172.21.15.45:0/1157184211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180009b30 con 0x7ff1880b1340 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.716+0000 7ff17ffff640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1880b35b0 0x7ff1880addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.717+0000 7ff18659c640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 0x7ff188138cc0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff17800c820 tx=0x7ff17800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.717+0000 7ff17effd640 1 -- 172.21.15.45:0/1157184211 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178007a30 con 0x7ff1880b1340 2023-11-28T20:51:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.717+0000 7ff17effd640 1 -- 172.21.15.45:0/1157184211 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff178007b90 con 0x7ff1880b1340 2023-11-28T20:51:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.717+0000 7ff17effd640 1 -- 172.21.15.45:0/1157184211 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17800ce70 con 0x7ff1880b1340 2023-11-28T20:51:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.717+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1880a86f0 con 0x7ff1880b1340 2023-11-28T20:51:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.717+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1880a8cb0 con 0x7ff1880b1340 2023-11-28T20:51:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.718+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1880733d0 con 0x7ff1880b1340 2023-11-28T20:51:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.719+0000 7ff17effd640 1 -- 172.21.15.45:0/1157184211 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7ff1780113f0 con 0x7ff1880b1340 2023-11-28T20:51:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.720+0000 7ff17effd640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff16806aed0 0x7ff16806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.720+0000 7ff17effd640 1 -- 172.21.15.45:0/1157184211 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff178089b10 con 0x7ff1880b1340 2023-11-28T20:51:13.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.721+0000 7ff17ffff640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff16806aed0 0x7ff16806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:13.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.722+0000 7ff17ffff640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff16806aed0 0x7ff16806d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff18806a9c0 tx=0x7ff170008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:13.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.725+0000 7ff17effd640 1 -- 172.21.15.45:0/1157184211 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff178054880 con 0x7ff1880b1340 2023-11-28T20:51:13.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.921+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1880cecd0 con 0x7ff1880b1340 2023-11-28T20:51:13.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.924+0000 7ff17effd640 1 -- 172.21.15.45:0/1157184211 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16789 (secure 0 0 0) 0x7ff178057ea0 con 0x7ff1880b1340 2023-11-28T20:51:13.926 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:13.927 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:12.953260+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:10.974716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:11.975222+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.926+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff16806aed0 msgr2=0x7ff16806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.926+0000 7ff18c8f3640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff16806aed0 0x7ff16806d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff18806a9c0 tx=0x7ff170008040 comp rx=0 tx=0).stop 2023-11-28T20:51:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 msgr2=0x7ff188138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 0x7ff188138cc0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff17800c820 tx=0x7ff17800cc20 comp rx=0 tx=0).stop 2023-11-28T20:51:13.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 shutdown_connections 2023-11-28T20:51:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ff16806aed0 0x7ff16806d390 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff18815b850 0x7ff1880abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1880b35b0 0x7ff1880addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 --2- 172.21.15.45:0/1157184211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1880b1340 0x7ff188138cc0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 >> 172.21.15.45:0/1157184211 conn(0x7ff18806a500 msgr2=0x7ff18815ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:13.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 shutdown_connections 2023-11-28T20:51:13.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:13.927+0000 7ff18c8f3640 1 -- 172.21.15.45:0/1157184211 wait complete. 2023-11-28T20:51:13.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:14.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.171+0000 7f5be01d5640 1 Processor -- start 2023-11-28T20:51:14.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.172+0000 7f5be01d5640 1 -- start start 2023-11-28T20:51:14.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.173+0000 7f5be01d5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5bd80b35b0 0x7f5bd815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.173+0000 7f5be01d5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bd815b850 0x7f5bd80b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.174+0000 7f5be01d5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd815bda0 con 0x7f5bd80b1fa0 2023-11-28T20:51:14.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.174+0000 7f5be01d5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd815ee40 con 0x7f5bd80b35b0 2023-11-28T20:51:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.174+0000 7f5be01d5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd815c080 con 0x7f5bd815b850 2023-11-28T20:51:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.174+0000 7f5bde772640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5bd80b1fa0 0x7f5bd815f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48850/0 (socket says 172.21.15.45:48850) 2023-11-28T20:51:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.174+0000 7f5bde772640 1 -- 172.21.15.45:0/2123919903 learned_addr learned my addr 172.21.15.45:0/2123919903 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.174+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891220350 0 0) 0x7f5bd815bda0 con 0x7f5bd80b1fa0 2023-11-28T20:51:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.174+0000 7f5bddf71640 1 --2- 172.21.15.45:0/2123919903 >> v2:172.21.15.190:3300/0 conn(0x7f5bd80b35b0 0x7f5bd815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.174+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0003520 con 0x7f5bd80b1fa0 2023-11-28T20:51:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdd770640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bd815b850 0x7f5bd80b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 362868983 0 0) 0x7f5bc0003520 con 0x7f5bd80b1fa0 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bd815b850 msgr2=0x7f5bd80b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bd815b850 0x7f5bd80b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 >> v2:172.21.15.190:3300/0 conn(0x7f5bd80b35b0 msgr2=0x7f5bd815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 --2- 172.21.15.45:0/2123919903 >> v2:172.21.15.190:3300/0 conn(0x7f5bd80b35b0 0x7f5bd815ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd815c360 con 0x7f5bd80b1fa0 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5bd0001670 con 0x7f5bd80b1fa0 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 >> v1:172.21.15.45:6789/0 conn(0x7f5bd80b1fa0 legacy=0x7f5bd815f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc0003970 0x7f5bc0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bd815b850 0x7f5bc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bc0004d00 0x7f5bc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd815ee40 con 0x7f5bc0003970 2023-11-28T20:51:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd815c080 con 0x7f5bc0004d00 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc000bae0 con 0x7f5bd815b850 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.175+0000 7f5bddf71640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc0003970 0x7f5bc0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.176+0000 7f5bdd770640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bd815b850 0x7f5bc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.176+0000 7f5bde772640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bc0004d00 0x7f5bc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.176+0000 7f5bddf71640 1 -- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bd815b850 msgr2=0x7f5bc00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.176+0000 7f5bddf71640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bd815b850 0x7f5bc00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.176+0000 7f5bddf71640 1 -- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bc0004d00 msgr2=0x7f5bc0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.176+0000 7f5bddf71640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bc0004d00 0x7f5bc0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.176+0000 7f5bddf71640 1 -- 172.21.15.45:0/2123919903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc0003520 con 0x7f5bc0003970 2023-11-28T20:51:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.177+0000 7f5bddf71640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc0003970 0x7f5bc0003d90 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8007360 tx=0x7f5bc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9133daab81e5588f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:14.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.177+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc802ccf0 con 0x7f5bc0003970 2023-11-28T20:51:14.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.177+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc802ce50 con 0x7f5bc0003970 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.177+0000 7f5bdcf6f640 1 -- 172.21.15.45:0/2123919903 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8034660 con 0x7f5bc0003970 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.178+0000 7f5be01d5640 1 -- 172.21.15.45:0/2123919903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc0003970 msgr2=0x7f5bc0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.178+0000 7f5be01d5640 1 --2- 172.21.15.45:0/2123919903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bc0003970 0x7f5bc0003d90 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8007360 tx=0x7f5bc00057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.178+0000 7f5bde772640 1 -- 172.21.15.45:0/2123919903 reap_dead start 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.178+0000 7f5be01d5640 1 -- 172.21.15.45:0/2123919903 shutdown_connections 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.178+0000 7f5be01d5640 1 -- 172.21.15.45:0/2123919903 >> 172.21.15.45:0/2123919903 conn(0x7f5bd806a500 msgr2=0x7f5bd806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.178+0000 7f5be01d5640 1 -- 172.21.15.45:0/2123919903 shutdown_connections 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.178+0000 7f5be01d5640 1 -- 172.21.15.45:0/2123919903 wait complete. 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.180+0000 7f5be01d5640 1 Processor -- start 2023-11-28T20:51:14.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.180+0000 7f5be01d5640 1 -- start start 2023-11-28T20:51:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.181+0000 7f5be01d5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bc0003970 0x7f5bd815a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.181+0000 7f5be01d5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bd8071de0 0x7f5bd80adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.181+0000 7f5be01d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd81f5d80 0x7f5bd81f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.181+0000 7f5be01d5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd81f6740 con 0x7f5bd81f5d80 2023-11-28T20:51:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.181+0000 7f5be01d5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd8071720 con 0x7f5bd8071de0 2023-11-28T20:51:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.181+0000 7f5be01d5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd8159d90 con 0x7f5bc0003970 2023-11-28T20:51:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.181+0000 7f5bde772640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd81f5d80 0x7f5bd81f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.181+0000 7f5bde772640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd81f5d80 0x7f5bd81f6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44732/0 (socket says 172.21.15.45:44732) 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.181+0000 7f5bde772640 1 -- 172.21.15.45:0/1604795423 learned_addr learned my addr 172.21.15.45:0/1604795423 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bddf71640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bc0003970 0x7f5bd815a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bdd770640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bd8071de0 0x7f5bd80adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bde772640 1 -- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bc0003970 msgr2=0x7f5bd815a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bde772640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bc0003970 0x7f5bd815a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bde772640 1 -- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bd8071de0 msgr2=0x7f5bd80adaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bde772640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bd8071de0 0x7f5bd80adaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bde772640 1 -- 172.21.15.45:0/1604795423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd815a070 con 0x7f5bd81f5d80 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bde772640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd81f5d80 0x7f5bd81f6200 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80cd380 tx=0x7f5bd000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bd67fc640 1 -- 172.21.15.45:0/1604795423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0007230 con 0x7f5bd81f5d80 2023-11-28T20:51:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bd67fc640 1 -- 172.21.15.45:0/1604795423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd00073b0 con 0x7f5bd81f5d80 2023-11-28T20:51:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5bd67fc640 1 -- 172.21.15.45:0/1604795423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd000dec0 con 0x7f5bd81f5d80 2023-11-28T20:51:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.182+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd815a350 con 0x7f5bd81f5d80 2023-11-28T20:51:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.183+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd8143600 con 0x7f5bd81f5d80 2023-11-28T20:51:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.184+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd815efb0 con 0x7f5bd81f5d80 2023-11-28T20:51:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.185+0000 7f5bd67fc640 1 -- 172.21.15.45:0/1604795423 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f5bd00047e0 con 0x7f5bd81f5d80 2023-11-28T20:51:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.186+0000 7f5bd67fc640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5bc406aed0 0x7f5bc406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.186+0000 7f5bd67fc640 1 -- 172.21.15.45:0/1604795423 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5bd0089690 con 0x7f5bd81f5d80 2023-11-28T20:51:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.186+0000 7f5bddf71640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5bc406aed0 0x7f5bc406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.188+0000 7f5bddf71640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5bc406aed0 0x7f5bc406d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8036000 tx=0x7f5bc8030000 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.190+0000 7f5bd67fc640 1 -- 172.21.15.45:0/1604795423 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bd0054370 con 0x7f5bd81f5d80 2023-11-28T20:51:14.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.394+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bd815a540 con 0x7f5bd81f5d80 2023-11-28T20:51:14.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.396+0000 7f5bd67fc640 1 -- 172.21.15.45:0/1604795423 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f5bd0057990 con 0x7f5bd81f5d80 2023-11-28T20:51:14.401 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:14.401 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:12.953260+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:10.974716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:11.975222+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:14.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5bc406aed0 msgr2=0x7f5bc406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5bc406aed0 0x7f5bc406d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8036000 tx=0x7f5bc8030000 comp rx=0 tx=0).stop 2023-11-28T20:51:14.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd81f5d80 msgr2=0x7f5bd81f6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd81f5d80 0x7f5bd81f6200 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80cd380 tx=0x7f5bd000db80 comp rx=0 tx=0).stop 2023-11-28T20:51:14.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 shutdown_connections 2023-11-28T20:51:14.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f5bc406aed0 0x7f5bc406d390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd81f5d80 0x7f5bd81f6200 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bd8071de0 0x7f5bd80adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 --2- 172.21.15.45:0/1604795423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5bc0003970 0x7f5bd815a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 >> 172.21.15.45:0/1604795423 conn(0x7f5bd806a500 msgr2=0x7f5bd815d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:14.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.398+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 shutdown_connections 2023-11-28T20:51:14.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.399+0000 7f5be01d5640 1 -- 172.21.15.45:0/1604795423 wait complete. 2023-11-28T20:51:14.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T20:51:14.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.651+0000 7f0464f5c640 1 Processor -- start 2023-11-28T20:51:14.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.652+0000 7f0464f5c640 1 -- start start 2023-11-28T20:51:14.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.653+0000 7f0464f5c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f046006b510 0x7f04600aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.653+0000 7f0464f5c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04600b3760 0x7f04600b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.653+0000 7f0464f5c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046015e630 con 0x7f046015d620 2023-11-28T20:51:14.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.653+0000 7f0464f5c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04600ada70 con 0x7f046006b510 2023-11-28T20:51:14.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.653+0000 7f0464f5c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046015dd90 con 0x7f04600b3760 2023-11-28T20:51:14.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.653+0000 7f045effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f046015d620 0x7f046006b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48860/0 (socket says 172.21.15.45:48860) 2023-11-28T20:51:14.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.653+0000 7f045effd640 1 -- 172.21.15.45:0/3559207692 learned_addr learned my addr 172.21.15.45:0/3559207692 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045f7fe640 1 --2- 172.21.15.45:0/3559207692 >> v2:172.21.15.190:3300/0 conn(0x7f046006b510 0x7f04600aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941222808 0 0) 0x7f046015e630 con 0x7f046015d620 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045ffff640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04600b3760 0x7f04600b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0448003520 con 0x7f046015d620 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746952488 0 0) 0x7f0448003520 con 0x7f046015d620 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04600b3760 msgr2=0x7f04600b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04600b3760 0x7f04600b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 >> v2:172.21.15.190:3300/0 conn(0x7f046006b510 msgr2=0x7f04600aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 --2- 172.21.15.45:0/3559207692 >> v2:172.21.15.190:3300/0 conn(0x7f046006b510 0x7f04600aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046015e070 con 0x7f046015d620 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f0454001670 con 0x7f046015d620 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 >> v1:172.21.15.45:6789/0 conn(0x7f046015d620 legacy=0x7f046006b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0448003970 0x7f0448003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f0448004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0448005180 0x7f044800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04600ada70 con 0x7f04480047c0 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046015dd90 con 0x7f0448003970 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.654+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044800bf70 con 0x7f0448005180 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045ffff640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04600b3760 0x7f04600b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:14.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045effd640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f0448004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045ffff640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0448005180 0x7f044800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045f7fe640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0448003970 0x7f0448003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045effd640 1 -- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0448005180 msgr2=0x7f044800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045effd640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0448005180 0x7f044800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045effd640 1 -- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0448003970 msgr2=0x7f0448003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045effd640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0448003970 0x7f0448003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045effd640 1 -- 172.21.15.45:0/3559207692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448003520 con 0x7f04480047c0 2023-11-28T20:51:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045ffff640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0448005180 0x7f044800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.655+0000 7f045f7fe640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0448003970 0x7f0448003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:14.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.656+0000 7f045effd640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f0448004c40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f045400c110 tx=0x7f04480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a07ecee3489789 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:14.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454005ad0 con 0x7f04480047c0 2023-11-28T20:51:14.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0454005c50 con 0x7f04480047c0 2023-11-28T20:51:14.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f045e7fc640 1 -- 172.21.15.45:0/3559207692 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045402c830 con 0x7f04480047c0 2023-11-28T20:51:14.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f0464f5c640 1 -- 172.21.15.45:0/3559207692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 msgr2=0x7f0448004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f0464f5c640 1 --2- 172.21.15.45:0/3559207692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f0448004c40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f045400c110 tx=0x7f04480057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f045ffff640 1 -- 172.21.15.45:0/3559207692 reap_dead start 2023-11-28T20:51:14.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f0464f5c640 1 -- 172.21.15.45:0/3559207692 shutdown_connections 2023-11-28T20:51:14.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f0464f5c640 1 -- 172.21.15.45:0/3559207692 >> 172.21.15.45:0/3559207692 conn(0x7f04600712d0 msgr2=0x7f04600716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:14.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f0464f5c640 1 -- 172.21.15.45:0/3559207692 shutdown_connections 2023-11-28T20:51:14.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.657+0000 7f0464f5c640 1 -- 172.21.15.45:0/3559207692 wait complete. 2023-11-28T20:51:14.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.659+0000 7f0464f5c640 1 Processor -- start 2023-11-28T20:51:14.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.659+0000 7f0464f5c640 1 -- start start 2023-11-28T20:51:14.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.660+0000 7f0464f5c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f046013b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.660+0000 7f0464f5c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f046013de20 0x7f046013bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.660+0000 7f0464f5c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f046015c250 0x7f04600b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.660+0000 7f0464f5c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046013c310 con 0x7f04480047c0 2023-11-28T20:51:14.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.660+0000 7f0464f5c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046013e2a0 con 0x7f046013de20 2023-11-28T20:51:14.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.660+0000 7f0464f5c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046013c540 con 0x7f046015c250 2023-11-28T20:51:14.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f046013b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f046013b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44756/0 (socket says 172.21.15.45:44756) 2023-11-28T20:51:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045f7fe640 1 -- 172.21.15.45:0/1729662188 learned_addr learned my addr 172.21.15.45:0/1729662188 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045effd640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f046013de20 0x7f046013bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045ffff640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f046015c250 0x7f04600b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045f7fe640 1 -- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f046015c250 msgr2=0x7f04600b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045f7fe640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f046015c250 0x7f04600b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045f7fe640 1 -- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f046013de20 msgr2=0x7f046013bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045f7fe640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f046013de20 0x7f046013bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045f7fe640 1 -- 172.21.15.45:0/1729662188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046013c820 con 0x7f04480047c0 2023-11-28T20:51:14.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.661+0000 7f045effd640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f046013de20 0x7f046013bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:14.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.662+0000 7f045f7fe640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f046013b7b0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f045000f150 tx=0x7f045000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:14.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.662+0000 7f0447fff640 1 -- 172.21.15.45:0/1729662188 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045001a070 con 0x7f04480047c0 2023-11-28T20:51:14.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.662+0000 7f0447fff640 1 -- 172.21.15.45:0/1729662188 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450003990 con 0x7f04480047c0 2023-11-28T20:51:14.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.662+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046015e630 con 0x7f04480047c0 2023-11-28T20:51:14.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.662+0000 7f0447fff640 1 -- 172.21.15.45:0/1729662188 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450009c80 con 0x7f04480047c0 2023-11-28T20:51:14.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.662+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046015ebf0 con 0x7f04480047c0 2023-11-28T20:51:14.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.663+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046015ed80 con 0x7f04480047c0 2023-11-28T20:51:14.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.664+0000 7f0447fff640 1 -- 172.21.15.45:0/1729662188 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f0450003d00 con 0x7f04480047c0 2023-11-28T20:51:14.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.665+0000 7f0447fff640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f044c06aed0 0x7f044c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:14.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.665+0000 7f0447fff640 1 -- 172.21.15.45:0/1729662188 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f045008a570 con 0x7f04480047c0 2023-11-28T20:51:14.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.666+0000 7f045effd640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f044c06aed0 0x7f044c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:14.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.667+0000 7f045effd640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f044c06aed0 0x7f044c06d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0454034000 tx=0x7f0454007000 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:14.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.670+0000 7f0447fff640 1 -- 172.21.15.45:0/1729662188 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0450055340 con 0x7f04480047c0 2023-11-28T20:51:14.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:14.904+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f046009e100 con 0x7f04480047c0 2023-11-28T20:51:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.502+0000 7f0447fff640 1 -- 172.21.15.45:0/1729662188 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0450058960 con 0x7f04480047c0 2023-11-28T20:51:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f044c06aed0 msgr2=0x7f044c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:15.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f044c06aed0 0x7f044c06d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0454034000 tx=0x7f0454007000 comp rx=0 tx=0).stop 2023-11-28T20:51:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 msgr2=0x7f046013b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f046013b7b0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f045000f150 tx=0x7f045000f610 comp rx=0 tx=0).stop 2023-11-28T20:51:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 shutdown_connections 2023-11-28T20:51:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f044c06aed0 0x7f044c06d390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f046015c250 0x7f04600b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f046013de20 0x7f046013bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 --2- 172.21.15.45:0/1729662188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04480047c0 0x7f046013b7b0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 >> 172.21.15.45:0/1729662188 conn(0x7f04600712d0 msgr2=0x7f04601446b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.506+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 shutdown_connections 2023-11-28T20:51:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.507+0000 7f0464f5c640 1 -- 172.21.15.45:0/1729662188 wait complete. 2023-11-28T20:51:15.531 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T20:51:15.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T20:51:15.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.761+0000 7ffaa3fff640 1 Processor -- start 2023-11-28T20:51:15.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.761+0000 7ffaa3fff640 1 -- start start 2023-11-28T20:51:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.762+0000 7ffaa3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffaa40b39d0 0x7ffaa40717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.762+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 0x7ffaa40cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:15.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.762+0000 7ffaa3fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa40724d0 con 0x7ffaa40b3db0 2023-11-28T20:51:15.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.762+0000 7ffaa3fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa4071d20 con 0x7ffaa40b39d0 2023-11-28T20:51:15.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.762+0000 7ffaa3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa40cc000 con 0x7ffaa40cb630 2023-11-28T20:51:15.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.763+0000 7ffaa2ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffaa40b39d0 0x7ffaa40717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:15.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.763+0000 7ffaa2ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffaa40b39d0 0x7ffaa40717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44766/0 (socket says 172.21.15.45:44766) 2023-11-28T20:51:15.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.763+0000 7ffaa2ffd640 1 -- 172.21.15.45:0/4159767062 learned_addr learned my addr 172.21.15.45:0/4159767062 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:15.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.763+0000 7ffaa37fe640 1 --2- 172.21.15.45:0/4159767062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 0x7ffaa40cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.763+0000 7ffaa2ffd640 1 -- 172.21.15.45:0/4159767062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 msgr2=0x7ffaa40cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.763+0000 7ffaa2ffd640 1 --2- 172.21.15.45:0/4159767062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 0x7ffaa40cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.763+0000 7ffaa2ffd640 1 -- 172.21.15.45:0/4159767062 >> v1:172.21.15.190:6789/0 conn(0x7ffaa40b3db0 legacy=0x7ffaa406bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.763+0000 7ffaa2ffd640 1 -- 172.21.15.45:0/4159767062 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa406c670 con 0x7ffaa40b39d0 2023-11-28T20:51:15.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.765+0000 7ffaa2ffd640 1 --2- 172.21.15.45:0/4159767062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 0x7ffaa40717e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009e50 tx=0x7ffa9402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50896a9e99494686 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:15.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.765+0000 7ffaa1ffb640 1 -- 172.21.15.45:0/4159767062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9402ca70 con 0x7ffaa40b39d0 2023-11-28T20:51:15.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.765+0000 7ffaa1ffb640 1 -- 172.21.15.45:0/4159767062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9402cbd0 con 0x7ffaa40b39d0 2023-11-28T20:51:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.765+0000 7ffaa1ffb640 1 -- 172.21.15.45:0/4159767062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa940343f0 con 0x7ffaa40b39d0 2023-11-28T20:51:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.766+0000 7ffaa3fff640 1 -- 172.21.15.45:0/4159767062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 msgr2=0x7ffaa40717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.766+0000 7ffaa3fff640 1 --2- 172.21.15.45:0/4159767062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 0x7ffaa40717e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009e50 tx=0x7ffa9402bed0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.766+0000 7ffaa3fff640 1 -- 172.21.15.45:0/4159767062 shutdown_connections 2023-11-28T20:51:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.766+0000 7ffaa3fff640 1 --2- 172.21.15.45:0/4159767062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 0x7ffaa40cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.766+0000 7ffaa3fff640 1 --2- 172.21.15.45:0/4159767062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 0x7ffaa40717e0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.766+0000 7ffaa3fff640 1 -- 172.21.15.45:0/4159767062 >> 172.21.15.45:0/4159767062 conn(0x7ffaa40b1050 msgr2=0x7ffaa40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.766+0000 7ffaa3fff640 1 -- 172.21.15.45:0/4159767062 shutdown_connections 2023-11-28T20:51:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.766+0000 7ffaa3fff640 1 -- 172.21.15.45:0/4159767062 wait complete. 2023-11-28T20:51:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.768+0000 7ffaa3fff640 1 Processor -- start 2023-11-28T20:51:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.768+0000 7ffaa3fff640 1 -- start start 2023-11-28T20:51:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.769+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 0x7ffaa4148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.769+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffaa40b3db0 0x7ffaa415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.769+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 0x7ffaa41f76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.769+0000 7ffaa3fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa41631a0 con 0x7ffaa40b39d0 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.769+0000 7ffaa3fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4163480 con 0x7ffaa40b3db0 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.769+0000 7ffaa3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4163760 con 0x7ffaa40cb630 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 0x7ffaa4148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 0x7ffaa4148dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44778/0 (socket says 172.21.15.45:44778) 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa2ffd640 1 -- 172.21.15.45:0/366156671 learned_addr learned my addr 172.21.15.45:0/366156671 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa37fe640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 0x7ffaa41f76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa27fc640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffaa40b3db0 0x7ffaa415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa2ffd640 1 -- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 msgr2=0x7ffaa41f76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa2ffd640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 0x7ffaa41f76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa2ffd640 1 -- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffaa40b3db0 msgr2=0x7ffaa415a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa2ffd640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffaa40b3db0 0x7ffaa415a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.770+0000 7ffaa2ffd640 1 -- 172.21.15.45:0/366156671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa94009b30 con 0x7ffaa40b39d0 2023-11-28T20:51:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.771+0000 7ffaa2ffd640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 0x7ffaa4148dd0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ffa94000f80 tx=0x7ffa94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.771+0000 7ffa9b7fe640 1 -- 172.21.15.45:0/366156671 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94009510 con 0x7ffaa40b39d0 2023-11-28T20:51:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.771+0000 7ffa9b7fe640 1 -- 172.21.15.45:0/366156671 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9403e450 con 0x7ffaa40b39d0 2023-11-28T20:51:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.771+0000 7ffa9b7fe640 1 -- 172.21.15.45:0/366156671 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9403e800 con 0x7ffaa40b39d0 2023-11-28T20:51:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.771+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa4163a40 con 0x7ffaa40b39d0 2023-11-28T20:51:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.771+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa4149310 con 0x7ffaa40b39d0 2023-11-28T20:51:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.772+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa409d0f0 con 0x7ffaa40b39d0 2023-11-28T20:51:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.773+0000 7ffa9b7fe640 1 -- 172.21.15.45:0/366156671 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 37) v1 ==== 89277+0+0 (secure 0 0 0) 0x7ffa9403e960 con 0x7ffaa40b39d0 2023-11-28T20:51:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.774+0000 7ffa9b7fe640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ffa8406aed0 0x7ffa8406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.775+0000 7ffa9b7fe640 1 -- 172.21.15.45:0/366156671 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7ffa940ac540 con 0x7ffaa40b39d0 2023-11-28T20:51:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.775+0000 7ffaa27fc640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ffa8406aed0 0x7ffa8406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.776+0000 7ffaa27fc640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ffa8406aed0 0x7ffa8406d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffaa415c210 tx=0x7ffa8c009040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:15.779+0000 7ffa9b7fe640 1 -- 172.21.15.45:0/366156671 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa94077640 con 0x7ffaa40b39d0 2023-11-28T20:51:16.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.013+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ffaa41615e0 con 0x7ffaa40b39d0 2023-11-28T20:51:16.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.515+0000 7ffa9b7fe640 1 -- 172.21.15.45:0/366156671 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7ffa9407ac60 con 0x7ffaa40b39d0 2023-11-28T20:51:16.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ffa8406aed0 msgr2=0x7ffa8406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:16.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ffa8406aed0 0x7ffa8406d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffaa415c210 tx=0x7ffa8c009040 comp rx=0 tx=0).stop 2023-11-28T20:51:16.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 msgr2=0x7ffaa4148dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:16.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 0x7ffaa4148dd0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ffa94000f80 tx=0x7ffa94004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:16.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 shutdown_connections 2023-11-28T20:51:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7ffa8406aed0 0x7ffa8406d390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffaa40cb630 0x7ffaa41f76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffaa40b3db0 0x7ffaa415a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 --2- 172.21.15.45:0/366156671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa40b39d0 0x7ffaa4148dd0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 >> 172.21.15.45:0/366156671 conn(0x7ffaa40b1050 msgr2=0x7ffaa4070a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 shutdown_connections 2023-11-28T20:51:16.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.519+0000 7ffaa3fff640 1 -- 172.21.15.45:0/366156671 wait complete. 2023-11-28T20:51:16.545 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T20:51:16.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:16.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.768+0000 7faccad9e640 1 Processor -- start 2023-11-28T20:51:16.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.769+0000 7faccad9e640 1 -- start start 2023-11-28T20:51:16.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.770+0000 7faccad9e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7facc40cfed0 0x7facc40cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:16.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.770+0000 7faccad9e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40ce010 0x7facc415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:16.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.770+0000 7faccad9e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40d1250 con 0x7facc40cfaf0 2023-11-28T20:51:16.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.770+0000 7faccad9e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc40b6c70 con 0x7facc40cfed0 2023-11-28T20:51:16.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.770+0000 7faccad9e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc40ce4a0 con 0x7facc40ce010 2023-11-28T20:51:16.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.771+0000 7facca59d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40ce010 0x7facc415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:16.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.771+0000 7facca59d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40ce010 0x7facc415cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:37330/0 (socket says 172.21.15.45:37330) 2023-11-28T20:51:16.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.771+0000 7facca59d640 1 -- 172.21.15.45:0/2505234013 learned_addr learned my addr 172.21.15.45:0/2505234013 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:16.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.771+0000 7facc959b640 1 --2- 172.21.15.45:0/2505234013 >> v2:172.21.15.190:3300/0 conn(0x7facc40cfed0 0x7facc40cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:16.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.772+0000 7facca59d640 1 -- 172.21.15.45:0/2505234013 >> v2:172.21.15.190:3300/0 conn(0x7facc40cfed0 msgr2=0x7facc40cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:16.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.772+0000 7facca59d640 1 --2- 172.21.15.45:0/2505234013 >> v2:172.21.15.190:3300/0 conn(0x7facc40cfed0 0x7facc40cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:16.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.772+0000 7facca59d640 1 -- 172.21.15.45:0/2505234013 >> v1:172.21.15.190:6789/0 conn(0x7facc40cfaf0 legacy=0x7facc40b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:16.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.772+0000 7facca59d640 1 -- 172.21.15.45:0/2505234013 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40ce6b0 con 0x7facc40ce010 2023-11-28T20:51:16.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.773+0000 7facca59d640 1 --2- 172.21.15.45:0/2505234013 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40ce010 0x7facc415cf60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7facbc009e50 tx=0x7facbc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59ea79f1fa2f78c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:16.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7facc8d9a640 1 -- 172.21.15.45:0/2505234013 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc02cb00 con 0x7facc40ce010 2023-11-28T20:51:16.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7facc8d9a640 1 -- 172.21.15.45:0/2505234013 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc02cc60 con 0x7facc40ce010 2023-11-28T20:51:16.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7facc8d9a640 1 -- 172.21.15.45:0/2505234013 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc0343f0 con 0x7facc40ce010 2023-11-28T20:51:16.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7faccad9e640 1 -- 172.21.15.45:0/2505234013 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40ce010 msgr2=0x7facc415cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7faccad9e640 1 --2- 172.21.15.45:0/2505234013 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40ce010 0x7facc415cf60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7facbc009e50 tx=0x7facbc02bf60 comp rx=0 tx=0).stop 2023-11-28T20:51:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7faccad9e640 1 -- 172.21.15.45:0/2505234013 shutdown_connections 2023-11-28T20:51:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7faccad9e640 1 --2- 172.21.15.45:0/2505234013 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40ce010 0x7facc415cf60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7faccad9e640 1 --2- 172.21.15.45:0/2505234013 >> v2:172.21.15.190:3300/0 conn(0x7facc40cfed0 0x7facc40cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7faccad9e640 1 -- 172.21.15.45:0/2505234013 >> 172.21.15.45:0/2505234013 conn(0x7facc40b4c40 msgr2=0x7facc40b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.774+0000 7faccad9e640 1 -- 172.21.15.45:0/2505234013 shutdown_connections 2023-11-28T20:51:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.775+0000 7faccad9e640 1 -- 172.21.15.45:0/2505234013 wait complete. 2023-11-28T20:51:16.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.776+0000 7faccad9e640 1 Processor -- start 2023-11-28T20:51:16.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.777+0000 7faccad9e640 1 -- start start 2023-11-28T20:51:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.777+0000 7faccad9e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc40ce010 0x7facc414b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.777+0000 7faccad9e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc40cfaf0 0x7facc40cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.777+0000 7faccad9e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40cfed0 0x7facc40cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7faccad9e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc41e7e70 con 0x7facc40ce010 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7faccad9e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc41e8150 con 0x7facc40cfaf0 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7faccad9e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc41e8430 con 0x7facc40cfed0 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc9d9c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc40ce010 0x7facc414b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc9d9c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc40ce010 0x7facc414b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44788/0 (socket says 172.21.15.45:44788) 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc9d9c640 1 -- 172.21.15.45:0/23123762 learned_addr learned my addr 172.21.15.45:0/23123762 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc959b640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc40cfaf0 0x7facc40cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facca59d640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40cfed0 0x7facc40cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc9d9c640 1 -- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40cfed0 msgr2=0x7facc40cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc9d9c640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40cfed0 0x7facc40cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc9d9c640 1 -- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc40cfaf0 msgr2=0x7facc40cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc9d9c640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc40cfaf0 0x7facc40cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc9d9c640 1 -- 172.21.15.45:0/23123762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc009b30 con 0x7facc40ce010 2023-11-28T20:51:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.778+0000 7facc959b640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc40cfaf0 0x7facc40cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.779+0000 7facc9d9c640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc40ce010 0x7facc414b4e0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7facc000c400 tx=0x7facc000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.779+0000 7facb67fc640 1 -- 172.21.15.45:0/23123762 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc00052d0 con 0x7facc40ce010 2023-11-28T20:51:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.779+0000 7facb67fc640 1 -- 172.21.15.45:0/23123762 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc0005450 con 0x7facc40ce010 2023-11-28T20:51:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.779+0000 7facb67fc640 1 -- 172.21.15.45:0/23123762 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc0007260 con 0x7facc40ce010 2023-11-28T20:51:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.779+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc41e8710 con 0x7facc40ce010 2023-11-28T20:51:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.779+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc414ba20 con 0x7facc40ce010 2023-11-28T20:51:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.780+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc404e920 con 0x7facc40ce010 2023-11-28T20:51:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.781+0000 7facb67fc640 1 -- 172.21.15.45:0/23123762 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7facc00073c0 con 0x7facc40ce010 2023-11-28T20:51:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.782+0000 7facb67fc640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faca806af60 0x7faca806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.782+0000 7facb67fc640 1 -- 172.21.15.45:0/23123762 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7facc0088d00 con 0x7facc40ce010 2023-11-28T20:51:16.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.783+0000 7facc959b640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faca806af60 0x7faca806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:16.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.784+0000 7facc959b640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faca806af60 0x7faca806d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7facc41f6ea0 tx=0x7facb8008040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:16.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:16.787+0000 7facb67fc640 1 -- 172.21.15.45:0/23123762 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facc0053b30 con 0x7facc40ce010 2023-11-28T20:51:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.021+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facc40d1030 con 0x7facc40ce010 2023-11-28T20:51:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.022+0000 7facb67fc640 1 -- 172.21.15.45:0/23123762 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4918 (secure 0 0 0) 0x7facc0057150 con 0x7facc40ce010 2023-11-28T20:51:17.025 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:17.025 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3492018158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3492018158},{"type":"v1","addr":"172.21.15.45:6833","nonce":3492018158}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6121,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2276329291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2276329291},{"type":"v1","addr":"172.21.15.190:6835","nonce":2276329291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6130,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1261857840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1261857840},{"type":"v1","addr":"172.21.15.190:6837","nonce":1261857840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:12.953862+0000","modified":"2023-11-28T20:51:16.015607+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7179},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7179":{"gid":7179,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/862206408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":862206408},{"type":"v1","addr":"172.21.15.45:6835","nonce":862206408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-28T20:51:17.025 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 44 2023-11-28T20:51:17.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faca806af60 msgr2=0x7faca806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faca806af60 0x7faca806d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7facc41f6ea0 tx=0x7facb8008040 comp rx=0 tx=0).stop 2023-11-28T20:51:17.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc40ce010 msgr2=0x7facc414b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc40ce010 0x7facc414b4e0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7facc000c400 tx=0x7facc000c8c0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 shutdown_connections 2023-11-28T20:51:17.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7faca806af60 0x7faca806d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facc40cfed0 0x7facc40cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc40cfaf0 0x7facc40cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 --2- 172.21.15.45:0/23123762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc40ce010 0x7facc414b4e0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 >> 172.21.15.45:0/23123762 conn(0x7facc40b4c40 msgr2=0x7facc40b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:17.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 shutdown_connections 2023-11-28T20:51:17.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.025+0000 7faccad9e640 1 -- 172.21.15.45:0/23123762 wait complete. 2023-11-28T20:51:17.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:17.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.264+0000 7f49844f3640 1 Processor -- start 2023-11-28T20:51:17.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.264+0000 7f49844f3640 1 -- start start 2023-11-28T20:51:17.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.265+0000 7f49844f3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f497c06a500 0x7f497c0b6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.265+0000 7f49844f3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c06aa70 con 0x7f497c071230 2023-11-28T20:51:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.265+0000 7f49844f3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c06ad50 con 0x7f497c160e70 2023-11-28T20:51:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.265+0000 7f49844f3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f497c06b030 con 0x7f497c06a500 2023-11-28T20:51:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.266+0000 7f4981a8e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f497c071230 0x7f497c0d1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48874/0 (socket says 172.21.15.45:48874) 2023-11-28T20:51:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.266+0000 7f4981a8e640 1 -- 172.21.15.45:0/3021824879 learned_addr learned my addr 172.21.15.45:0/3021824879 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.266+0000 7f4982a90640 1 --2- 172.21.15.45:0/3021824879 >> v2:172.21.15.190:3300/0 conn(0x7f497c06a500 0x7f497c0b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.266+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924235820 0 0) 0x7f497c06aa70 con 0x7f497c071230 2023-11-28T20:51:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.266+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4968003520 con 0x7f497c071230 2023-11-28T20:51:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 823177076 0 0) 0x7f4968003520 con 0x7f497c071230 2023-11-28T20:51:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 >> v2:172.21.15.190:3300/0 conn(0x7f497c06a500 msgr2=0x7f497c0b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 --2- 172.21.15.45:0/3021824879 >> v2:172.21.15.190:3300/0 conn(0x7f497c06a500 0x7f497c0b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 >> v1:172.21.15.190:6789/0 conn(0x7f497c160e70 legacy=0x7f497c070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c06b310 con 0x7f497c071230 2023-11-28T20:51:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f496c001670 con 0x7f497c071230 2023-11-28T20:51:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 >> v1:172.21.15.45:6789/0 conn(0x7f497c071230 legacy=0x7f497c0d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 --2- 172.21.15.45:0/3021824879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49680039c0 0x7f4968003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 --2- 172.21.15.45:0/3021824879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49680047c0 0x7f4968004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 --2- 172.21.15.45:0/3021824879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4968005180 0x7f496800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c06b030 con 0x7f49680039c0 2023-11-28T20:51:17.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496800c080 con 0x7f49680047c0 2023-11-28T20:51:17.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496800c240 con 0x7f4968005180 2023-11-28T20:51:17.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498228f640 1 --2- 172.21.15.45:0/3021824879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49680039c0 0x7f4968003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498228f640 1 -- 172.21.15.45:0/3021824879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4968005180 msgr2=0x7f496800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498228f640 1 --2- 172.21.15.45:0/3021824879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4968005180 0x7f496800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498228f640 1 -- 172.21.15.45:0/3021824879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49680047c0 msgr2=0x7f4968004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498228f640 1 --2- 172.21.15.45:0/3021824879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49680047c0 0x7f4968004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f498228f640 1 -- 172.21.15.45:0/3021824879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4968003520 con 0x7f49680039c0 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.267+0000 7f4982a90640 1 -- 172.21.15.45:0/3021824879 reap_dead start 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.269+0000 7f498228f640 1 --2- 172.21.15.45:0/3021824879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49680039c0 0x7f4968003de0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f4978001e50 tx=0x7f49680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e56f555a4dba10 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.269+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978003b70 con 0x7f49680039c0 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.269+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4978003cd0 con 0x7f49680039c0 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.269+0000 7f498128d640 1 -- 172.21.15.45:0/3021824879 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978005310 con 0x7f49680039c0 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.269+0000 7f49844f3640 1 -- 172.21.15.45:0/3021824879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49680039c0 msgr2=0x7f4968003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.269+0000 7f49844f3640 1 --2- 172.21.15.45:0/3021824879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49680039c0 0x7f4968003de0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f4978001e50 tx=0x7f49680057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.269+0000 7f49844f3640 1 -- 172.21.15.45:0/3021824879 shutdown_connections 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.269+0000 7f49844f3640 1 --2- 172.21.15.45:0/3021824879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49680039c0 0x7f4968003de0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.270+0000 7f49844f3640 1 -- 172.21.15.45:0/3021824879 >> 172.21.15.45:0/3021824879 conn(0x7f497c0cfc60 msgr2=0x7f497c0d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:17.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.270+0000 7f49844f3640 1 -- 172.21.15.45:0/3021824879 shutdown_connections 2023-11-28T20:51:17.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.270+0000 7f49844f3640 1 -- 172.21.15.45:0/3021824879 wait complete. 2023-11-28T20:51:17.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.271+0000 7f49844f3640 1 Processor -- start 2023-11-28T20:51:17.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.272+0000 7f49844f3640 1 -- start start 2023-11-28T20:51:17.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.272+0000 7f49844f3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49680039c0 0x7f497c156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f49844f3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f497c1f3bb0 0x7f497c1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f49844f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c159be0 0x7f497c15a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f49844f3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c15a9c0 con 0x7f497c159be0 2023-11-28T20:51:17.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f49844f3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c15ac80 con 0x7f497c1f3bb0 2023-11-28T20:51:17.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f49844f3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497c15af60 con 0x7f49680039c0 2023-11-28T20:51:17.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f4982a90640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c159be0 0x7f497c15a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:17.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f4982a90640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c159be0 0x7f497c15a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44804/0 (socket says 172.21.15.45:44804) 2023-11-28T20:51:17.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f4982a90640 1 -- 172.21.15.45:0/2152001426 learned_addr learned my addr 172.21.15.45:0/2152001426 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f4982a90640 1 -- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49680039c0 msgr2=0x7f497c156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f4982a90640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49680039c0 0x7f497c156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f4982a90640 1 -- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f497c1f3bb0 msgr2=0x7f497c1f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f4982a90640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f497c1f3bb0 0x7f497c1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.273+0000 7f4982a90640 1 -- 172.21.15.45:0/2152001426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c15b240 con 0x7f497c159be0 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.274+0000 7f4982a90640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c159be0 0x7f497c15a060 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f497c0d2e60 tx=0x7f497400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.274+0000 7f4972ffd640 1 -- 172.21.15.45:0/2152001426 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497400c620 con 0x7f497c159be0 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.274+0000 7f4972ffd640 1 -- 172.21.15.45:0/2152001426 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497400c780 con 0x7f497c159be0 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.274+0000 7f4972ffd640 1 -- 172.21.15.45:0/2152001426 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497400cb30 con 0x7f497c159be0 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.274+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c15b520 con 0x7f497c159be0 2023-11-28T20:51:17.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.274+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c0cc480 con 0x7f497c159be0 2023-11-28T20:51:17.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.275+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c15ba40 con 0x7f497c159be0 2023-11-28T20:51:17.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.276+0000 7f4972ffd640 1 -- 172.21.15.45:0/2152001426 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f497400cc90 con 0x7f497c159be0 2023-11-28T20:51:17.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.277+0000 7f4972ffd640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f496006af90 0x7f496006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.277+0000 7f4972ffd640 1 -- 172.21.15.45:0/2152001426 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f49740897a0 con 0x7f497c159be0 2023-11-28T20:51:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.278+0000 7f498228f640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f496006af90 0x7f496006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:17.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.279+0000 7f498228f640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f496006af90 0x7f496006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4978034000 tx=0x7f4978030000 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:17.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.282+0000 7f4972ffd640 1 -- 172.21.15.45:0/2152001426 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f497c15ba40 con 0x7f497c159be0 2023-11-28T20:51:17.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.476+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f497c15ba40 con 0x7f497c159be0 2023-11-28T20:51:17.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.479+0000 7f4972ffd640 1 -- 172.21.15.45:0/2152001426 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16838 (secure 0 0 0) 0x7f497c15ba40 con 0x7f497c159be0 2023-11-28T20:51:17.481 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:17.481 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":102,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:14.488259+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:10.974716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:11.975222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1852415784":"2023-11-28T21:51:13.642235+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f496006af90 msgr2=0x7f496006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f496006af90 0x7f496006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4978034000 tx=0x7f4978030000 comp rx=0 tx=0).stop 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c159be0 msgr2=0x7f497c15a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c159be0 0x7f497c15a060 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f497c0d2e60 tx=0x7f497400ec00 comp rx=0 tx=0).stop 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 shutdown_connections 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f496006af90 0x7f496006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c159be0 0x7f497c15a060 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f497c1f3bb0 0x7f497c1f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 --2- 172.21.15.45:0/2152001426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49680039c0 0x7f497c156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.481+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 >> 172.21.15.45:0/2152001426 conn(0x7f497c0cfc60 msgr2=0x7f497c0b8190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:17.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.482+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 shutdown_connections 2023-11-28T20:51:17.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.482+0000 7f49844f3640 1 -- 172.21.15.45:0/2152001426 wait complete. 2023-11-28T20:51:17.507 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T20:51:17.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:51:17.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.728+0000 7f154a2dc640 1 Processor -- start 2023-11-28T20:51:17.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.729+0000 7f154a2dc640 1 -- start start 2023-11-28T20:51:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.730+0000 7f154a2dc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15440ce560 0x7f15440d2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.730+0000 7f154a2dc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15440d1a50 con 0x7f15440d1470 2023-11-28T20:51:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.730+0000 7f154a2dc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15440d1d30 con 0x7f1544160710 2023-11-28T20:51:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.730+0000 7f154a2dc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15440d2010 con 0x7f15440ce560 2023-11-28T20:51:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.730+0000 7f1548ad9640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f15440d1470 0x7f15440cde50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48880/0 (socket says 172.21.15.45:48880) 2023-11-28T20:51:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.730+0000 7f1548ad9640 1 -- 172.21.15.45:0/1981817886 learned_addr learned my addr 172.21.15.45:0/1981817886 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243395504 0 0) 0x7f15440d1a50 con 0x7f15440d1470 2023-11-28T20:51:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1530003520 con 0x7f15440d1470 2023-11-28T20:51:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f1549adb640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15440ce560 0x7f15440d2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4060012481 0 0) 0x7f1530003520 con 0x7f15440d1470 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15440ce560 msgr2=0x7f15440d2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15440ce560 0x7f15440d2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 >> v1:172.21.15.190:6789/0 conn(0x7f1544160710 legacy=0x7f15440b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15440d22f0 con 0x7f15440d1470 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f1534001670 con 0x7f15440d1470 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 >> v1:172.21.15.45:6789/0 conn(0x7f15440d1470 legacy=0x7f15440cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15300039a0 0x7f1530003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f1530004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15300052a0 0x7f153000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15440d2010 con 0x7f15300047c0 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153000c0d0 con 0x7f15300039a0 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153000c290 con 0x7f15300052a0 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.731+0000 7f1549adb640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15440ce560 0x7f15440d2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.732+0000 7f1548ad9640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f1530004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.732+0000 7f1549adb640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15300052a0 0x7f153000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.732+0000 7f1548ad9640 1 -- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15300052a0 msgr2=0x7f153000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.732+0000 7f1548ad9640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15300052a0 0x7f153000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.732+0000 7f1548ad9640 1 -- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15300039a0 msgr2=0x7f1530003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.732+0000 7f1548ad9640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15300039a0 0x7f1530003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.732+0000 7f1548ad9640 1 -- 172.21.15.45:0/1981817886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1530003520 con 0x7f15300047c0 2023-11-28T20:51:17.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.733+0000 7f1548ad9640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f1530004c40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f153400c240 tx=0x7f15300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0fed4c2f0b2730 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:17.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.733+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534005ad0 con 0x7f15300047c0 2023-11-28T20:51:17.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.733+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1534005c30 con 0x7f15300047c0 2023-11-28T20:51:17.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.733+0000 7f153bfff640 1 -- 172.21.15.45:0/1981817886 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153402c8b0 con 0x7f15300047c0 2023-11-28T20:51:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.734+0000 7f154a2dc640 1 -- 172.21.15.45:0/1981817886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 msgr2=0x7f1530004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.734+0000 7f154a2dc640 1 --2- 172.21.15.45:0/1981817886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f1530004c40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f153400c240 tx=0x7f15300057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.734+0000 7f1549adb640 1 -- 172.21.15.45:0/1981817886 reap_dead start 2023-11-28T20:51:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.734+0000 7f154a2dc640 1 -- 172.21.15.45:0/1981817886 shutdown_connections 2023-11-28T20:51:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.734+0000 7f154a2dc640 1 -- 172.21.15.45:0/1981817886 >> 172.21.15.45:0/1981817886 conn(0x7f154406a500 msgr2=0x7f154406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.734+0000 7f154a2dc640 1 -- 172.21.15.45:0/1981817886 shutdown_connections 2023-11-28T20:51:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.734+0000 7f154a2dc640 1 -- 172.21.15.45:0/1981817886 wait complete. 2023-11-28T20:51:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.736+0000 7f154a2dc640 1 Processor -- start 2023-11-28T20:51:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.736+0000 7f154a2dc640 1 -- start start 2023-11-28T20:51:17.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.737+0000 7f154a2dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f15441f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.737+0000 7f154a2dc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15440b52f0 0x7f1544158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.737+0000 7f154a2dc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15440b5770 0x7f15441586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.737+0000 7f154a2dc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15440b2b10 con 0x7f15300047c0 2023-11-28T20:51:17.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.737+0000 7f154a2dc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1544154db0 con 0x7f15440b5770 2023-11-28T20:51:17.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.737+0000 7f154a2dc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1544159080 con 0x7f15440b52f0 2023-11-28T20:51:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.737+0000 7f15492da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f15441f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.737+0000 7f15492da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f15441f2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44820/0 (socket says 172.21.15.45:44820) 2023-11-28T20:51:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.737+0000 7f15492da640 1 -- 172.21.15.45:0/419295303 learned_addr learned my addr 172.21.15.45:0/419295303 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f15492da640 1 -- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15440b52f0 msgr2=0x7f1544158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f15492da640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15440b52f0 0x7f1544158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f15492da640 1 -- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15440b5770 msgr2=0x7f15441586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f15492da640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15440b5770 0x7f15441586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f15492da640 1 -- 172.21.15.45:0/419295303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1544159360 con 0x7f15300047c0 2023-11-28T20:51:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f15492da640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f15441f2a50 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f154000c840 tx=0x7f154000cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f1539ffb640 1 -- 172.21.15.45:0/419295303 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1540003b70 con 0x7f15300047c0 2023-11-28T20:51:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f1539ffb640 1 -- 172.21.15.45:0/419295303 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1540003cf0 con 0x7f15300047c0 2023-11-28T20:51:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f1539ffb640 1 -- 172.21.15.45:0/419295303 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154000aa90 con 0x7f15300047c0 2023-11-28T20:51:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.738+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1544159640 con 0x7f15300047c0 2023-11-28T20:51:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.739+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15441f2f90 con 0x7f15300047c0 2023-11-28T20:51:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.740+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1544071b30 con 0x7f15300047c0 2023-11-28T20:51:17.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.741+0000 7f1539ffb640 1 -- 172.21.15.45:0/419295303 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f154000abf0 con 0x7f15300047c0 2023-11-28T20:51:17.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.742+0000 7f1539ffb640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f152806af90 0x7f152806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:17.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.742+0000 7f1539ffb640 1 -- 172.21.15.45:0/419295303 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f15400897d0 con 0x7f15300047c0 2023-11-28T20:51:17.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.743+0000 7f1548ad9640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f152806af90 0x7f152806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:17.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.744+0000 7f1548ad9640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f152806af90 0x7f152806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1534034000 tx=0x7f1534007000 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:17.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.747+0000 7f1539ffb640 1 -- 172.21.15.45:0/419295303 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15400545f0 con 0x7f15300047c0 2023-11-28T20:51:17.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.932+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 --> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1544160af0 con 0x7f152806af90 2023-11-28T20:51:17.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.961+0000 7f1539ffb640 1 -- 172.21.15.45:0/419295303 <== mgr.5959 v2:172.21.15.190:6832/970930168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286766 (secure 0 0 0) 0x7f1534005870 con 0x7f152806af90 2023-11-28T20:51:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:51:17.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f152806af90 msgr2=0x7f152806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f152806af90 0x7f152806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1534034000 tx=0x7f1534007000 comp rx=0 tx=0).stop 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 msgr2=0x7f15441f2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f15441f2a50 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f154000c840 tx=0x7f154000cd00 comp rx=0 tx=0).stop 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 shutdown_connections 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f152806af90 0x7f152806d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15440b5770 0x7f15441586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15440b52f0 0x7f1544158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 --2- 172.21.15.45:0/419295303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15300047c0 0x7f15441f2a50 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.967+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 >> 172.21.15.45:0/419295303 conn(0x7f154406a500 msgr2=0x7f15440b6540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.968+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 shutdown_connections 2023-11-28T20:51:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:17.968+0000 7f154a2dc640 1 -- 172.21.15.45:0/419295303 wait complete. 2023-11-28T20:51:17.991 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-11-28T20:51:17.793993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":570,"num_read_kb":1050,"num_write":290,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2635,"ondisk_log_size":2635,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":339348,"kb_used_data":10772,"kb_used_omap":12,"kb_used_meta":328435,"kb_avail":754635372,"statfs":{"total":773094113280,"available":772746620928,"internally_reserved":0,"allocated":11030528,"data_stored":6106554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":336318000},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001917"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.518119+0000","last_change":"2023-11-28T20:51:13.515389+0000","last_active":"2023-11-28T20:51:13.518119+0000","last_peered":"2023-11-28T20:51:13.518119+0000","last_clean":"2023-11-28T20:51:13.518119+0000","last_became_active":"2023-11-28T20:51:13.515100+0000","last_became_peered":"2023-11-28T20:51:13.515100+0000","last_unstale":"2023-11-28T20:51:13.518119+0000","last_undegraded":"2023-11-28T20:51:13.518119+0000","last_fullsized":"2023-11-28T20:51:13.518119+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:18:42.410562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506897+0000","last_change":"2023-11-28T20:51:13.503822+0000","last_active":"2023-11-28T20:51:13.506897+0000","last_peered":"2023-11-28T20:51:13.506897+0000","last_clean":"2023-11-28T20:51:13.506897+0000","last_became_active":"2023-11-28T20:51:13.503546+0000","last_became_peered":"2023-11-28T20:51:13.503546+0000","last_unstale":"2023-11-28T20:51:13.506897+0000","last_undegraded":"2023-11-28T20:51:13.506897+0000","last_fullsized":"2023-11-28T20:51:13.506897+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:39:26.295263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.509803+0000","last_change":"2023-11-28T20:51:13.507476+0000","last_active":"2023-11-28T20:51:13.509803+0000","last_peered":"2023-11-28T20:51:13.509803+0000","last_clean":"2023-11-28T20:51:13.509803+0000","last_became_active":"2023-11-28T20:51:13.507042+0000","last_became_peered":"2023-11-28T20:51:13.507042+0000","last_unstale":"2023-11-28T20:51:13.509803+0000","last_undegraded":"2023-11-28T20:51:13.509803+0000","last_fullsized":"2023-11-28T20:51:13.509803+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:02.952326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517355+0000","last_change":"2023-11-28T20:51:13.516325+0000","last_active":"2023-11-28T20:51:13.517355+0000","last_peered":"2023-11-28T20:51:13.517355+0000","last_clean":"2023-11-28T20:51:13.517355+0000","last_became_active":"2023-11-28T20:51:13.513015+0000","last_became_peered":"2023-11-28T20:51:13.513015+0000","last_unstale":"2023-11-28T20:51:13.517355+0000","last_undegraded":"2023-11-28T20:51:13.517355+0000","last_fullsized":"2023-11-28T20:51:13.517355+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:34:22.392589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.497652+0000","last_change":"2023-11-28T20:51:13.492454+0000","last_active":"2023-11-28T20:51:13.497652+0000","last_peered":"2023-11-28T20:51:13.497652+0000","last_clean":"2023-11-28T20:51:13.497652+0000","last_became_active":"2023-11-28T20:51:13.492144+0000","last_became_peered":"2023-11-28T20:51:13.492144+0000","last_unstale":"2023-11-28T20:51:13.497652+0000","last_undegraded":"2023-11-28T20:51:13.497652+0000","last_fullsized":"2023-11-28T20:51:13.497652+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:26:37.646917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514320+0000","last_change":"2023-11-28T20:51:13.510642+0000","last_active":"2023-11-28T20:51:13.514320+0000","last_peered":"2023-11-28T20:51:13.514320+0000","last_clean":"2023-11-28T20:51:13.514320+0000","last_became_active":"2023-11-28T20:51:13.510143+0000","last_became_peered":"2023-11-28T20:51:13.510143+0000","last_unstale":"2023-11-28T20:51:13.514320+0000","last_undegraded":"2023-11-28T20:51:13.514320+0000","last_fullsized":"2023-11-28T20:51:13.514320+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:39:26.295263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522161+0000","last_change":"2023-11-28T20:51:13.520701+0000","last_active":"2023-11-28T20:51:13.522161+0000","last_peered":"2023-11-28T20:51:13.522161+0000","last_clean":"2023-11-28T20:51:13.522161+0000","last_became_active":"2023-11-28T20:51:13.520389+0000","last_became_peered":"2023-11-28T20:51:13.520389+0000","last_unstale":"2023-11-28T20:51:13.522161+0000","last_undegraded":"2023-11-28T20:51:13.522161+0000","last_fullsized":"2023-11-28T20:51:13.522161+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:05:55.771224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.507710+0000","last_change":"2023-11-28T20:51:13.495833+0000","last_active":"2023-11-28T20:51:13.507710+0000","last_peered":"2023-11-28T20:51:13.507710+0000","last_clean":"2023-11-28T20:51:13.507710+0000","last_became_active":"2023-11-28T20:51:13.493644+0000","last_became_peered":"2023-11-28T20:51:13.493644+0000","last_unstale":"2023-11-28T20:51:13.507710+0000","last_undegraded":"2023-11-28T20:51:13.507710+0000","last_fullsized":"2023-11-28T20:51:13.507710+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:00.768908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.516606+0000","last_change":"2023-11-28T20:51:13.511246+0000","last_active":"2023-11-28T20:51:13.516606+0000","last_peered":"2023-11-28T20:51:13.516606+0000","last_clean":"2023-11-28T20:51:13.516606+0000","last_became_active":"2023-11-28T20:51:13.509466+0000","last_became_peered":"2023-11-28T20:51:13.509466+0000","last_unstale":"2023-11-28T20:51:13.516606+0000","last_undegraded":"2023-11-28T20:51:13.516606+0000","last_fullsized":"2023-11-28T20:51:13.516606+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:24.987055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.519964+0000","last_change":"2023-11-28T20:51:13.516783+0000","last_active":"2023-11-28T20:51:13.519964+0000","last_peered":"2023-11-28T20:51:13.519964+0000","last_clean":"2023-11-28T20:51:13.519964+0000","last_became_active":"2023-11-28T20:51:13.516508+0000","last_became_peered":"2023-11-28T20:51:13.516508+0000","last_unstale":"2023-11-28T20:51:13.519964+0000","last_undegraded":"2023-11-28T20:51:13.519964+0000","last_fullsized":"2023-11-28T20:51:13.519964+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:18.509118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506936+0000","last_change":"2023-11-28T20:51:13.505792+0000","last_active":"2023-11-28T20:51:13.506936+0000","last_peered":"2023-11-28T20:51:13.506936+0000","last_clean":"2023-11-28T20:51:13.506936+0000","last_became_active":"2023-11-28T20:51:13.504698+0000","last_became_peered":"2023-11-28T20:51:13.504698+0000","last_unstale":"2023-11-28T20:51:13.506936+0000","last_undegraded":"2023-11-28T20:51:13.506936+0000","last_fullsized":"2023-11-28T20:51:13.506936+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:19:40.944865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520761+0000","last_change":"2023-11-28T20:51:13.518586+0000","last_active":"2023-11-28T20:51:13.520761+0000","last_peered":"2023-11-28T20:51:13.520761+0000","last_clean":"2023-11-28T20:51:13.520761+0000","last_became_active":"2023-11-28T20:51:13.517764+0000","last_became_peered":"2023-11-28T20:51:13.517764+0000","last_unstale":"2023-11-28T20:51:13.520761+0000","last_undegraded":"2023-11-28T20:51:13.520761+0000","last_fullsized":"2023-11-28T20:51:13.520761+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:13:06.103102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520417+0000","last_change":"2023-11-28T20:51:13.508273+0000","last_active":"2023-11-28T20:51:13.520417+0000","last_peered":"2023-11-28T20:51:13.520417+0000","last_clean":"2023-11-28T20:51:13.520417+0000","last_became_active":"2023-11-28T20:51:13.507975+0000","last_became_peered":"2023-11-28T20:51:13.507975+0000","last_unstale":"2023-11-28T20:51:13.520417+0000","last_undegraded":"2023-11-28T20:51:13.520417+0000","last_fullsized":"2023-11-28T20:51:13.520417+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:32:16.543758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514300+0000","last_change":"2023-11-28T20:51:13.510641+0000","last_active":"2023-11-28T20:51:13.514300+0000","last_peered":"2023-11-28T20:51:13.514300+0000","last_clean":"2023-11-28T20:51:13.514300+0000","last_became_active":"2023-11-28T20:51:13.510128+0000","last_became_peered":"2023-11-28T20:51:13.510128+0000","last_unstale":"2023-11-28T20:51:13.514300+0000","last_undegraded":"2023-11-28T20:51:13.514300+0000","last_fullsized":"2023-11-28T20:51:13.514300+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:19:40.944865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.500188+0000","last_change":"2023-11-28T20:51:13.495228+0000","last_active":"2023-11-28T20:51:13.500188+0000","last_peered":"2023-11-28T20:51:13.500188+0000","last_clean":"2023-11-28T20:51:13.500188+0000","last_became_active":"2023-11-28T20:51:13.494949+0000","last_became_peered":"2023-11-28T20:51:13.494949+0000","last_unstale":"2023-11-28T20:51:13.500188+0000","last_undegraded":"2023-11-28T20:51:13.500188+0000","last_fullsized":"2023-11-28T20:51:13.500188+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:18:42.410562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506588+0000","last_change":"2023-11-28T20:51:13.503352+0000","last_active":"2023-11-28T20:51:13.506588+0000","last_peered":"2023-11-28T20:51:13.506588+0000","last_clean":"2023-11-28T20:51:13.506588+0000","last_became_active":"2023-11-28T20:51:13.503007+0000","last_became_peered":"2023-11-28T20:51:13.503007+0000","last_unstale":"2023-11-28T20:51:13.506588+0000","last_undegraded":"2023-11-28T20:51:13.506588+0000","last_fullsized":"2023-11-28T20:51:13.506588+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:13.967025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522708+0000","last_change":"2023-11-28T20:51:13.520195+0000","last_active":"2023-11-28T20:51:13.522708+0000","last_peered":"2023-11-28T20:51:13.522708+0000","last_clean":"2023-11-28T20:51:13.522708+0000","last_became_active":"2023-11-28T20:51:13.519610+0000","last_became_peered":"2023-11-28T20:51:13.519610+0000","last_unstale":"2023-11-28T20:51:13.522708+0000","last_undegraded":"2023-11-28T20:51:13.522708+0000","last_fullsized":"2023-11-28T20:51:13.522708+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:23:38.003752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.516833+0000","last_change":"2023-11-28T20:51:13.515728+0000","last_active":"2023-11-28T20:51:13.516833+0000","last_peered":"2023-11-28T20:51:13.516833+0000","last_clean":"2023-11-28T20:51:13.516833+0000","last_became_active":"2023-11-28T20:51:13.515386+0000","last_became_peered":"2023-11-28T20:51:13.515386+0000","last_unstale":"2023-11-28T20:51:13.516833+0000","last_undegraded":"2023-11-28T20:51:13.516833+0000","last_fullsized":"2023-11-28T20:51:13.516833+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:47.176949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.523975+0000","last_change":"2023-11-28T20:51:13.523111+0000","last_active":"2023-11-28T20:51:13.523975+0000","last_peered":"2023-11-28T20:51:13.523975+0000","last_clean":"2023-11-28T20:51:13.523975+0000","last_became_active":"2023-11-28T20:51:13.522747+0000","last_became_peered":"2023-11-28T20:51:13.522747+0000","last_unstale":"2023-11-28T20:51:13.523975+0000","last_undegraded":"2023-11-28T20:51:13.523975+0000","last_fullsized":"2023-11-28T20:51:13.523975+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:07.963984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.496121+0000","last_change":"2023-11-28T20:51:13.489157+0000","last_active":"2023-11-28T20:51:13.496121+0000","last_peered":"2023-11-28T20:51:13.496121+0000","last_clean":"2023-11-28T20:51:13.496121+0000","last_became_active":"2023-11-28T20:51:13.488825+0000","last_became_peered":"2023-11-28T20:51:13.488825+0000","last_unstale":"2023-11-28T20:51:13.496121+0000","last_undegraded":"2023-11-28T20:51:13.496121+0000","last_fullsized":"2023-11-28T20:51:13.496121+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:44:42.288287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.516805+0000","last_change":"2023-11-28T20:51:13.511525+0000","last_active":"2023-11-28T20:51:13.516805+0000","last_peered":"2023-11-28T20:51:13.516805+0000","last_clean":"2023-11-28T20:51:13.516805+0000","last_became_active":"2023-11-28T20:51:13.510254+0000","last_became_peered":"2023-11-28T20:51:13.510254+0000","last_unstale":"2023-11-28T20:51:13.516805+0000","last_undegraded":"2023-11-28T20:51:13.516805+0000","last_fullsized":"2023-11-28T20:51:13.516805+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:34:07.276346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.496210+0000","last_change":"2023-11-28T20:51:13.488397+0000","last_active":"2023-11-28T20:51:13.496210+0000","last_peered":"2023-11-28T20:51:13.496210+0000","last_clean":"2023-11-28T20:51:13.496210+0000","last_became_active":"2023-11-28T20:51:13.488107+0000","last_became_peered":"2023-11-28T20:51:13.488107+0000","last_unstale":"2023-11-28T20:51:13.496210+0000","last_undegraded":"2023-11-28T20:51:13.496210+0000","last_fullsized":"2023-11-28T20:51:13.496210+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:50:38.763001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.512124+0000","last_change":"2023-11-28T20:51:13.506491+0000","last_active":"2023-11-28T20:51:13.512124+0000","last_peered":"2023-11-28T20:51:13.512124+0000","last_clean":"2023-11-28T20:51:13.512124+0000","last_became_active":"2023-11-28T20:51:13.506133+0000","last_became_peered":"2023-11-28T20:51:13.506133+0000","last_unstale":"2023-11-28T20:51:13.512124+0000","last_undegraded":"2023-11-28T20:51:13.512124+0000","last_fullsized":"2023-11-28T20:51:13.512124+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:05:55.771224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.518611+0000","last_change":"2023-11-28T20:51:13.510261+0000","last_active":"2023-11-28T20:51:13.518611+0000","last_peered":"2023-11-28T20:51:13.518611+0000","last_clean":"2023-11-28T20:51:13.518611+0000","last_became_active":"2023-11-28T20:51:13.508621+0000","last_became_peered":"2023-11-28T20:51:13.508621+0000","last_unstale":"2023-11-28T20:51:13.518611+0000","last_undegraded":"2023-11-28T20:51:13.518611+0000","last_fullsized":"2023-11-28T20:51:13.518611+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:11:44.887484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.504933+0000","last_change":"2023-11-28T20:51:13.501559+0000","last_active":"2023-11-28T20:51:13.504933+0000","last_peered":"2023-11-28T20:51:13.504933+0000","last_clean":"2023-11-28T20:51:13.504933+0000","last_became_active":"2023-11-28T20:51:13.500638+0000","last_became_peered":"2023-11-28T20:51:13.500638+0000","last_unstale":"2023-11-28T20:51:13.504933+0000","last_undegraded":"2023-11-28T20:51:13.504933+0000","last_fullsized":"2023-11-28T20:51:13.504933+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:28.433626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.507360+0000","last_change":"2023-11-28T20:51:13.506396+0000","last_active":"2023-11-28T20:51:13.507360+0000","last_peered":"2023-11-28T20:51:13.507360+0000","last_clean":"2023-11-28T20:51:13.507360+0000","last_became_active":"2023-11-28T20:51:13.506115+0000","last_became_peered":"2023-11-28T20:51:13.506115+0000","last_unstale":"2023-11-28T20:51:13.507360+0000","last_undegraded":"2023-11-28T20:51:13.507360+0000","last_fullsized":"2023-11-28T20:51:13.507360+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:08.706931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.505278+0000","last_change":"2023-11-28T20:51:13.502500+0000","last_active":"2023-11-28T20:51:13.505278+0000","last_peered":"2023-11-28T20:51:13.505278+0000","last_clean":"2023-11-28T20:51:13.505278+0000","last_became_active":"2023-11-28T20:51:13.502174+0000","last_became_peered":"2023-11-28T20:51:13.502174+0000","last_unstale":"2023-11-28T20:51:13.505278+0000","last_undegraded":"2023-11-28T20:51:13.505278+0000","last_fullsized":"2023-11-28T20:51:13.505278+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:06:52.066200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522218+0000","last_change":"2023-11-28T20:51:13.501613+0000","last_active":"2023-11-28T20:51:13.522218+0000","last_peered":"2023-11-28T20:51:13.522218+0000","last_clean":"2023-11-28T20:51:13.522218+0000","last_became_active":"2023-11-28T20:51:13.501334+0000","last_became_peered":"2023-11-28T20:51:13.501334+0000","last_unstale":"2023-11-28T20:51:13.522218+0000","last_undegraded":"2023-11-28T20:51:13.522218+0000","last_fullsized":"2023-11-28T20:51:13.522218+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:32:15.927170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517250+0000","last_change":"2023-11-28T20:51:13.508308+0000","last_active":"2023-11-28T20:51:13.517250+0000","last_peered":"2023-11-28T20:51:13.517250+0000","last_clean":"2023-11-28T20:51:13.517250+0000","last_became_active":"2023-11-28T20:51:13.507861+0000","last_became_peered":"2023-11-28T20:51:13.507861+0000","last_unstale":"2023-11-28T20:51:13.517250+0000","last_undegraded":"2023-11-28T20:51:13.517250+0000","last_fullsized":"2023-11-28T20:51:13.517250+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:45:19.402469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.507924+0000","last_change":"2023-11-28T20:51:13.505705+0000","last_active":"2023-11-28T20:51:13.507924+0000","last_peered":"2023-11-28T20:51:13.507924+0000","last_clean":"2023-11-28T20:51:13.507924+0000","last_became_active":"2023-11-28T20:51:13.504825+0000","last_became_peered":"2023-11-28T20:51:13.504825+0000","last_unstale":"2023-11-28T20:51:13.507924+0000","last_undegraded":"2023-11-28T20:51:13.507924+0000","last_fullsized":"2023-11-28T20:51:13.507924+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:19:40.944865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514408+0000","last_change":"2023-11-28T20:51:13.512999+0000","last_active":"2023-11-28T20:51:13.514408+0000","last_peered":"2023-11-28T20:51:13.514408+0000","last_clean":"2023-11-28T20:51:13.514408+0000","last_became_active":"2023-11-28T20:51:13.512717+0000","last_became_peered":"2023-11-28T20:51:13.512717+0000","last_unstale":"2023-11-28T20:51:13.514408+0000","last_undegraded":"2023-11-28T20:51:13.514408+0000","last_fullsized":"2023-11-28T20:51:13.514408+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:51:09.007211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.513900+0000","last_change":"2023-11-28T20:51:13.508634+0000","last_active":"2023-11-28T20:51:13.513900+0000","last_peered":"2023-11-28T20:51:13.513900+0000","last_clean":"2023-11-28T20:51:13.513900+0000","last_became_active":"2023-11-28T20:51:13.508364+0000","last_became_peered":"2023-11-28T20:51:13.508364+0000","last_unstale":"2023-11-28T20:51:13.513900+0000","last_undegraded":"2023-11-28T20:51:13.513900+0000","last_fullsized":"2023-11-28T20:51:13.513900+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:18.509118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.515825+0000","last_change":"2023-11-28T20:51:13.515101+0000","last_active":"2023-11-28T20:51:13.515825+0000","last_peered":"2023-11-28T20:51:13.515825+0000","last_clean":"2023-11-28T20:51:13.515825+0000","last_became_active":"2023-11-28T20:51:13.512097+0000","last_became_peered":"2023-11-28T20:51:13.512097+0000","last_unstale":"2023-11-28T20:51:13.515825+0000","last_undegraded":"2023-11-28T20:51:13.515825+0000","last_fullsized":"2023-11-28T20:51:13.515825+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:26:13.953999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"101'4","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.521748+0000","last_change":"2023-11-28T20:51:12.484239+0000","last_active":"2023-11-28T20:51:13.521748+0000","last_peered":"2023-11-28T20:51:13.521748+0000","last_clean":"2023-11-28T20:51:13.521748+0000","last_became_active":"2023-11-28T20:51:12.483657+0000","last_became_peered":"2023-11-28T20:51:12.483657+0000","last_unstale":"2023-11-28T20:51:13.521748+0000","last_undegraded":"2023-11-28T20:51:13.521748+0000","last_fullsized":"2023-11-28T20:51:13.521748+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:01:58.181623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517047+0000","last_change":"2023-11-28T20:51:13.506785+0000","last_active":"2023-11-28T20:51:13.517047+0000","last_peered":"2023-11-28T20:51:13.517047+0000","last_clean":"2023-11-28T20:51:13.517047+0000","last_became_active":"2023-11-28T20:51:13.506384+0000","last_became_peered":"2023-11-28T20:51:13.506384+0000","last_unstale":"2023-11-28T20:51:13.517047+0000","last_undegraded":"2023-11-28T20:51:13.517047+0000","last_fullsized":"2023-11-28T20:51:13.517047+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:34:22.392589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.515680+0000","last_change":"2023-11-28T20:51:12.506096+0000","last_active":"2023-11-28T20:51:13.515680+0000","last_peered":"2023-11-28T20:51:13.515680+0000","last_clean":"2023-11-28T20:51:13.515680+0000","last_became_active":"2023-11-28T20:51:12.505812+0000","last_became_peered":"2023-11-28T20:51:12.505812+0000","last_unstale":"2023-11-28T20:51:13.515680+0000","last_undegraded":"2023-11-28T20:51:13.515680+0000","last_fullsized":"2023-11-28T20:51:13.515680+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:34:21.386513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514157+0000","last_change":"2023-11-28T20:51:13.508333+0000","last_active":"2023-11-28T20:51:13.514157+0000","last_peered":"2023-11-28T20:51:13.514157+0000","last_clean":"2023-11-28T20:51:13.514157+0000","last_became_active":"2023-11-28T20:51:13.508037+0000","last_became_peered":"2023-11-28T20:51:13.508037+0000","last_unstale":"2023-11-28T20:51:13.514157+0000","last_undegraded":"2023-11-28T20:51:13.514157+0000","last_fullsized":"2023-11-28T20:51:13.514157+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:51:37.137247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522224+0000","last_change":"2023-11-28T20:51:12.501461+0000","last_active":"2023-11-28T20:51:13.522224+0000","last_peered":"2023-11-28T20:51:13.522224+0000","last_clean":"2023-11-28T20:51:13.522224+0000","last_became_active":"2023-11-28T20:51:12.500691+0000","last_became_peered":"2023-11-28T20:51:12.500691+0000","last_unstale":"2023-11-28T20:51:13.522224+0000","last_undegraded":"2023-11-28T20:51:13.522224+0000","last_fullsized":"2023-11-28T20:51:13.522224+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:27.427550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.516827+0000","last_change":"2023-11-28T20:51:13.511702+0000","last_active":"2023-11-28T20:51:13.516827+0000","last_peered":"2023-11-28T20:51:13.516827+0000","last_clean":"2023-11-28T20:51:13.516827+0000","last_became_active":"2023-11-28T20:51:13.510597+0000","last_became_peered":"2023-11-28T20:51:13.510597+0000","last_unstale":"2023-11-28T20:51:13.516827+0000","last_undegraded":"2023-11-28T20:51:13.516827+0000","last_fullsized":"2023-11-28T20:51:13.516827+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:20.697835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520881+0000","last_change":"2023-11-28T20:51:12.502881+0000","last_active":"2023-11-28T20:51:13.520881+0000","last_peered":"2023-11-28T20:51:13.520881+0000","last_clean":"2023-11-28T20:51:13.520881+0000","last_became_active":"2023-11-28T20:51:12.502586+0000","last_became_peered":"2023-11-28T20:51:12.502586+0000","last_unstale":"2023-11-28T20:51:13.520881+0000","last_undegraded":"2023-11-28T20:51:13.520881+0000","last_fullsized":"2023-11-28T20:51:13.520881+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:13.254831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506075+0000","last_change":"2023-11-28T20:51:13.501884+0000","last_active":"2023-11-28T20:51:13.506075+0000","last_peered":"2023-11-28T20:51:13.506075+0000","last_clean":"2023-11-28T20:51:13.506075+0000","last_became_active":"2023-11-28T20:51:13.501591+0000","last_became_peered":"2023-11-28T20:51:13.501591+0000","last_unstale":"2023-11-28T20:51:13.506075+0000","last_undegraded":"2023-11-28T20:51:13.506075+0000","last_fullsized":"2023-11-28T20:51:13.506075+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:42:11.298316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.511829+0000","last_change":"2023-11-28T20:51:12.497543+0000","last_active":"2023-11-28T20:51:13.511829+0000","last_peered":"2023-11-28T20:51:13.511829+0000","last_clean":"2023-11-28T20:51:13.511829+0000","last_became_active":"2023-11-28T20:51:12.497239+0000","last_became_peered":"2023-11-28T20:51:12.497239+0000","last_unstale":"2023-11-28T20:51:13.511829+0000","last_undegraded":"2023-11-28T20:51:13.511829+0000","last_fullsized":"2023-11-28T20:51:13.511829+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:54.794319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.501291+0000","last_change":"2023-11-28T20:51:13.499305+0000","last_active":"2023-11-28T20:51:13.501291+0000","last_peered":"2023-11-28T20:51:13.501291+0000","last_clean":"2023-11-28T20:51:13.501291+0000","last_became_active":"2023-11-28T20:51:13.498732+0000","last_became_peered":"2023-11-28T20:51:13.498732+0000","last_unstale":"2023-11-28T20:51:13.501291+0000","last_undegraded":"2023-11-28T20:51:13.501291+0000","last_fullsized":"2023-11-28T20:51:13.501291+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:47:57.031660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.505768+0000","last_change":"2023-11-28T20:51:12.490486+0000","last_active":"2023-11-28T20:51:13.505768+0000","last_peered":"2023-11-28T20:51:13.505768+0000","last_clean":"2023-11-28T20:51:13.505768+0000","last_became_active":"2023-11-28T20:51:12.490169+0000","last_became_peered":"2023-11-28T20:51:12.490169+0000","last_unstale":"2023-11-28T20:51:13.505768+0000","last_undegraded":"2023-11-28T20:51:13.505768+0000","last_fullsized":"2023-11-28T20:51:13.505768+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:06:41.223231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.519805+0000","last_change":"2023-11-28T20:51:13.512002+0000","last_active":"2023-11-28T20:51:13.519805+0000","last_peered":"2023-11-28T20:51:13.519805+0000","last_clean":"2023-11-28T20:51:13.519805+0000","last_became_active":"2023-11-28T20:51:13.508935+0000","last_became_peered":"2023-11-28T20:51:13.508935+0000","last_unstale":"2023-11-28T20:51:13.519805+0000","last_undegraded":"2023-11-28T20:51:13.519805+0000","last_fullsized":"2023-11-28T20:51:13.519805+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:29.635174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.484633+0000","last_change":"2023-11-28T20:51:12.497787+0000","last_active":"2023-11-28T20:51:13.484633+0000","last_peered":"2023-11-28T20:51:13.484633+0000","last_clean":"2023-11-28T20:51:13.484633+0000","last_became_active":"2023-11-28T20:51:12.497167+0000","last_became_peered":"2023-11-28T20:51:12.497167+0000","last_unstale":"2023-11-28T20:51:13.484633+0000","last_undegraded":"2023-11-28T20:51:13.484633+0000","last_fullsized":"2023-11-28T20:51:13.484633+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:50.172662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.524282+0000","last_change":"2023-11-28T20:51:13.523581+0000","last_active":"2023-11-28T20:51:13.524282+0000","last_peered":"2023-11-28T20:51:13.524282+0000","last_clean":"2023-11-28T20:51:13.524282+0000","last_became_active":"2023-11-28T20:51:13.523158+0000","last_became_peered":"2023-11-28T20:51:13.523158+0000","last_unstale":"2023-11-28T20:51:13.524282+0000","last_undegraded":"2023-11-28T20:51:13.524282+0000","last_fullsized":"2023-11-28T20:51:13.524282+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:52:15.633702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.480521+0000","last_change":"2023-11-28T20:51:12.487193+0000","last_active":"2023-11-28T20:51:13.480521+0000","last_peered":"2023-11-28T20:51:13.480521+0000","last_clean":"2023-11-28T20:51:13.480521+0000","last_became_active":"2023-11-28T20:51:12.486541+0000","last_became_peered":"2023-11-28T20:51:12.486541+0000","last_unstale":"2023-11-28T20:51:13.480521+0000","last_undegraded":"2023-11-28T20:51:13.480521+0000","last_fullsized":"2023-11-28T20:51:13.480521+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:13.405678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.499824+0000","last_change":"2023-11-28T20:51:13.492702+0000","last_active":"2023-11-28T20:51:13.499824+0000","last_peered":"2023-11-28T20:51:13.499824+0000","last_clean":"2023-11-28T20:51:13.499824+0000","last_became_active":"2023-11-28T20:51:13.492449+0000","last_became_peered":"2023-11-28T20:51:13.492449+0000","last_unstale":"2023-11-28T20:51:13.499824+0000","last_undegraded":"2023-11-28T20:51:13.499824+0000","last_fullsized":"2023-11-28T20:51:13.499824+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:13.967025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.481343+0000","last_change":"2023-11-28T20:51:12.506534+0000","last_active":"2023-11-28T20:51:13.481343+0000","last_peered":"2023-11-28T20:51:13.481343+0000","last_clean":"2023-11-28T20:51:13.481343+0000","last_became_active":"2023-11-28T20:51:12.505899+0000","last_became_peered":"2023-11-28T20:51:12.505899+0000","last_unstale":"2023-11-28T20:51:13.481343+0000","last_undegraded":"2023-11-28T20:51:13.481343+0000","last_fullsized":"2023-11-28T20:51:13.481343+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:29.409766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520945+0000","last_change":"2023-11-28T20:51:13.519416+0000","last_active":"2023-11-28T20:51:13.520945+0000","last_peered":"2023-11-28T20:51:13.520945+0000","last_clean":"2023-11-28T20:51:13.520945+0000","last_became_active":"2023-11-28T20:51:13.518667+0000","last_became_peered":"2023-11-28T20:51:13.518667+0000","last_unstale":"2023-11-28T20:51:13.520945+0000","last_undegraded":"2023-11-28T20:51:13.520945+0000","last_fullsized":"2023-11-28T20:51:13.520945+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:18:42.410562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.492991+0000","last_change":"2023-11-28T20:51:12.497171+0000","last_active":"2023-11-28T20:51:13.492991+0000","last_peered":"2023-11-28T20:51:13.492991+0000","last_clean":"2023-11-28T20:51:13.492991+0000","last_became_active":"2023-11-28T20:51:12.496868+0000","last_became_peered":"2023-11-28T20:51:12.496868+0000","last_unstale":"2023-11-28T20:51:13.492991+0000","last_undegraded":"2023-11-28T20:51:13.492991+0000","last_fullsized":"2023-11-28T20:51:13.492991+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:06:51.060124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.495825+0000","last_change":"2023-11-28T20:51:13.493179+0000","last_active":"2023-11-28T20:51:13.495825+0000","last_peered":"2023-11-28T20:51:13.495825+0000","last_clean":"2023-11-28T20:51:13.495825+0000","last_became_active":"2023-11-28T20:51:13.492909+0000","last_became_peered":"2023-11-28T20:51:13.492909+0000","last_unstale":"2023-11-28T20:51:13.495825+0000","last_undegraded":"2023-11-28T20:51:13.495825+0000","last_fullsized":"2023-11-28T20:51:13.495825+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:28.433626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"101'3","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520714+0000","last_change":"2023-11-28T20:51:12.489353+0000","last_active":"2023-11-28T20:51:13.520714+0000","last_peered":"2023-11-28T20:51:13.520714+0000","last_clean":"2023-11-28T20:51:13.520714+0000","last_became_active":"2023-11-28T20:51:12.488259+0000","last_became_peered":"2023-11-28T20:51:12.488259+0000","last_unstale":"2023-11-28T20:51:13.520714+0000","last_undegraded":"2023-11-28T20:51:13.520714+0000","last_fullsized":"2023-11-28T20:51:13.520714+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:43:43.464568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.503954+0000","last_change":"2023-11-28T20:51:13.503120+0000","last_active":"2023-11-28T20:51:13.503954+0000","last_peered":"2023-11-28T20:51:13.503954+0000","last_clean":"2023-11-28T20:51:13.503954+0000","last_became_active":"2023-11-28T20:51:13.502844+0000","last_became_peered":"2023-11-28T20:51:13.502844+0000","last_unstale":"2023-11-28T20:51:13.503954+0000","last_undegraded":"2023-11-28T20:51:13.503954+0000","last_fullsized":"2023-11-28T20:51:13.503954+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:15:45.314242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"101'6","reported_seq":17,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.525615+0000","last_change":"2023-11-28T20:51:12.501097+0000","last_active":"2023-11-28T20:51:13.525615+0000","last_peered":"2023-11-28T20:51:13.525615+0000","last_clean":"2023-11-28T20:51:13.525615+0000","last_became_active":"2023-11-28T20:51:12.499789+0000","last_became_peered":"2023-11-28T20:51:12.499789+0000","last_unstale":"2023-11-28T20:51:13.525615+0000","last_undegraded":"2023-11-28T20:51:13.525615+0000","last_fullsized":"2023-11-28T20:51:13.525615+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:09:57.520400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506892+0000","last_change":"2023-11-28T20:51:13.504409+0000","last_active":"2023-11-28T20:51:13.506892+0000","last_peered":"2023-11-28T20:51:13.506892+0000","last_clean":"2023-11-28T20:51:13.506892+0000","last_became_active":"2023-11-28T20:51:13.504135+0000","last_became_peered":"2023-11-28T20:51:13.504135+0000","last_unstale":"2023-11-28T20:51:13.506892+0000","last_undegraded":"2023-11-28T20:51:13.506892+0000","last_fullsized":"2023-11-28T20:51:13.506892+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:13.967025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.512905+0000","last_change":"2023-11-28T20:51:12.495185+0000","last_active":"2023-11-28T20:51:13.512905+0000","last_peered":"2023-11-28T20:51:13.512905+0000","last_clean":"2023-11-28T20:51:13.512905+0000","last_became_active":"2023-11-28T20:51:12.494721+0000","last_became_peered":"2023-11-28T20:51:12.494721+0000","last_unstale":"2023-11-28T20:51:13.512905+0000","last_undegraded":"2023-11-28T20:51:13.512905+0000","last_fullsized":"2023-11-28T20:51:13.512905+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:59:32.995662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.507853+0000","last_change":"2023-11-28T20:51:13.500041+0000","last_active":"2023-11-28T20:51:13.507853+0000","last_peered":"2023-11-28T20:51:13.507853+0000","last_clean":"2023-11-28T20:51:13.507853+0000","last_became_active":"2023-11-28T20:51:13.499771+0000","last_became_peered":"2023-11-28T20:51:13.499771+0000","last_unstale":"2023-11-28T20:51:13.507853+0000","last_undegraded":"2023-11-28T20:51:13.507853+0000","last_fullsized":"2023-11-28T20:51:13.507853+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:05.680077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.480062+0000","last_change":"2023-11-28T20:51:12.502262+0000","last_active":"2023-11-28T20:51:13.480062+0000","last_peered":"2023-11-28T20:51:13.480062+0000","last_clean":"2023-11-28T20:51:13.480062+0000","last_became_active":"2023-11-28T20:51:12.501984+0000","last_became_peered":"2023-11-28T20:51:12.501984+0000","last_unstale":"2023-11-28T20:51:13.480062+0000","last_undegraded":"2023-11-28T20:51:13.480062+0000","last_fullsized":"2023-11-28T20:51:13.480062+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:42:31.025256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517275+0000","last_change":"2023-11-28T20:51:13.511951+0000","last_active":"2023-11-28T20:51:13.517275+0000","last_peered":"2023-11-28T20:51:13.517275+0000","last_clean":"2023-11-28T20:51:13.517275+0000","last_became_active":"2023-11-28T20:51:13.511635+0000","last_became_peered":"2023-11-28T20:51:13.511635+0000","last_unstale":"2023-11-28T20:51:13.517275+0000","last_undegraded":"2023-11-28T20:51:13.517275+0000","last_fullsized":"2023-11-28T20:51:13.517275+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:29.121136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.481992+0000","last_change":"2023-11-28T20:51:12.483682+0000","last_active":"2023-11-28T20:51:13.481992+0000","last_peered":"2023-11-28T20:51:13.481992+0000","last_clean":"2023-11-28T20:51:13.481992+0000","last_became_active":"2023-11-28T20:51:12.483335+0000","last_became_peered":"2023-11-28T20:51:12.483335+0000","last_unstale":"2023-11-28T20:51:13.481992+0000","last_undegraded":"2023-11-28T20:51:13.481992+0000","last_fullsized":"2023-11-28T20:51:13.481992+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:33:03.745861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520876+0000","last_change":"2023-11-28T20:51:13.514841+0000","last_active":"2023-11-28T20:51:13.520876+0000","last_peered":"2023-11-28T20:51:13.520876+0000","last_clean":"2023-11-28T20:51:13.520876+0000","last_became_active":"2023-11-28T20:51:13.514502+0000","last_became_peered":"2023-11-28T20:51:13.514502+0000","last_unstale":"2023-11-28T20:51:13.520876+0000","last_undegraded":"2023-11-28T20:51:13.520876+0000","last_fullsized":"2023-11-28T20:51:13.520876+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:20.697835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"101'8","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.524583+0000","last_change":"2023-11-28T20:51:12.493381+0000","last_active":"2023-11-28T20:51:13.524583+0000","last_peered":"2023-11-28T20:51:13.524583+0000","last_clean":"2023-11-28T20:51:13.524583+0000","last_became_active":"2023-11-28T20:51:12.493106+0000","last_became_peered":"2023-11-28T20:51:12.493106+0000","last_unstale":"2023-11-28T20:51:13.524583+0000","last_undegraded":"2023-11-28T20:51:13.524583+0000","last_fullsized":"2023-11-28T20:51:13.524583+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:14:07.789156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.497591+0000","last_change":"2023-11-28T20:51:13.492090+0000","last_active":"2023-11-28T20:51:13.497591+0000","last_peered":"2023-11-28T20:51:13.497591+0000","last_clean":"2023-11-28T20:51:13.497591+0000","last_became_active":"2023-11-28T20:51:13.488443+0000","last_became_peered":"2023-11-28T20:51:13.488443+0000","last_unstale":"2023-11-28T20:51:13.497591+0000","last_undegraded":"2023-11-28T20:51:13.497591+0000","last_fullsized":"2023-11-28T20:51:13.497591+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:42:11.298316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514828+0000","last_change":"2023-11-28T20:51:12.503430+0000","last_active":"2023-11-28T20:51:13.514828+0000","last_peered":"2023-11-28T20:51:13.514828+0000","last_clean":"2023-11-28T20:51:13.514828+0000","last_became_active":"2023-11-28T20:51:12.503187+0000","last_became_peered":"2023-11-28T20:51:12.503187+0000","last_unstale":"2023-11-28T20:51:13.514828+0000","last_undegraded":"2023-11-28T20:51:13.514828+0000","last_fullsized":"2023-11-28T20:51:13.514828+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:50.053211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522293+0000","last_change":"2023-11-28T20:51:13.518392+0000","last_active":"2023-11-28T20:51:13.522293+0000","last_peered":"2023-11-28T20:51:13.522293+0000","last_clean":"2023-11-28T20:51:13.522293+0000","last_became_active":"2023-11-28T20:51:13.518109+0000","last_became_peered":"2023-11-28T20:51:13.518109+0000","last_unstale":"2023-11-28T20:51:13.522293+0000","last_undegraded":"2023-11-28T20:51:13.522293+0000","last_fullsized":"2023-11-28T20:51:13.522293+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:57:18.348611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.515732+0000","last_change":"2023-11-28T20:51:12.493220+0000","last_active":"2023-11-28T20:51:13.515732+0000","last_peered":"2023-11-28T20:51:13.515732+0000","last_clean":"2023-11-28T20:51:13.515732+0000","last_became_active":"2023-11-28T20:51:12.492626+0000","last_became_peered":"2023-11-28T20:51:12.492626+0000","last_unstale":"2023-11-28T20:51:13.515732+0000","last_undegraded":"2023-11-28T20:51:13.515732+0000","last_fullsized":"2023-11-28T20:51:13.515732+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:50.264502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522505+0000","last_change":"2023-11-28T20:51:13.517975+0000","last_active":"2023-11-28T20:51:13.522505+0000","last_peered":"2023-11-28T20:51:13.522505+0000","last_clean":"2023-11-28T20:51:13.522505+0000","last_became_active":"2023-11-28T20:51:13.513725+0000","last_became_peered":"2023-11-28T20:51:13.513725+0000","last_unstale":"2023-11-28T20:51:13.522505+0000","last_undegraded":"2023-11-28T20:51:13.522505+0000","last_fullsized":"2023-11-28T20:51:13.522505+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:41.759604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522913+0000","last_change":"2023-11-28T20:51:12.486812+0000","last_active":"2023-11-28T20:51:13.522913+0000","last_peered":"2023-11-28T20:51:13.522913+0000","last_clean":"2023-11-28T20:51:13.522913+0000","last_became_active":"2023-11-28T20:51:12.486513+0000","last_became_peered":"2023-11-28T20:51:12.486513+0000","last_unstale":"2023-11-28T20:51:13.522913+0000","last_undegraded":"2023-11-28T20:51:13.522913+0000","last_fullsized":"2023-11-28T20:51:13.522913+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:02:57.418923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520269+0000","last_change":"2023-11-28T20:51:13.518992+0000","last_active":"2023-11-28T20:51:13.520269+0000","last_peered":"2023-11-28T20:51:13.520269+0000","last_clean":"2023-11-28T20:51:13.520269+0000","last_became_active":"2023-11-28T20:51:13.515701+0000","last_became_peered":"2023-11-28T20:51:13.515701+0000","last_unstale":"2023-11-28T20:51:13.520269+0000","last_undegraded":"2023-11-28T20:51:13.520269+0000","last_fullsized":"2023-11-28T20:51:13.520269+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:12:11.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.476301+0000","last_change":"2023-11-28T20:51:12.492296+0000","last_active":"2023-11-28T20:51:13.476301+0000","last_peered":"2023-11-28T20:51:13.476301+0000","last_clean":"2023-11-28T20:51:13.476301+0000","last_became_active":"2023-11-28T20:51:12.492018+0000","last_became_peered":"2023-11-28T20:51:12.492018+0000","last_unstale":"2023-11-28T20:51:13.476301+0000","last_undegraded":"2023-11-28T20:51:13.476301+0000","last_fullsized":"2023-11-28T20:51:13.476301+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:23:16.583514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.518096+0000","last_change":"2023-11-28T20:51:13.511100+0000","last_active":"2023-11-28T20:51:13.518096+0000","last_peered":"2023-11-28T20:51:13.518096+0000","last_clean":"2023-11-28T20:51:13.518096+0000","last_became_active":"2023-11-28T20:51:13.505852+0000","last_became_peered":"2023-11-28T20:51:13.505852+0000","last_unstale":"2023-11-28T20:51:13.518096+0000","last_undegraded":"2023-11-28T20:51:13.518096+0000","last_fullsized":"2023-11-28T20:51:13.518096+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:30.415842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.513994+0000","last_change":"2023-11-28T20:51:12.488412+0000","last_active":"2023-11-28T20:51:13.513994+0000","last_peered":"2023-11-28T20:51:13.513994+0000","last_clean":"2023-11-28T20:51:13.513994+0000","last_became_active":"2023-11-28T20:51:12.488158+0000","last_became_peered":"2023-11-28T20:51:12.488158+0000","last_unstale":"2023-11-28T20:51:13.513994+0000","last_undegraded":"2023-11-28T20:51:13.513994+0000","last_fullsized":"2023-11-28T20:51:13.513994+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:11:43.881408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.521107+0000","last_change":"2023-11-28T20:51:13.518710+0000","last_active":"2023-11-28T20:51:13.521107+0000","last_peered":"2023-11-28T20:51:13.521107+0000","last_clean":"2023-11-28T20:51:13.521107+0000","last_became_active":"2023-11-28T20:51:13.518389+0000","last_became_peered":"2023-11-28T20:51:13.518389+0000","last_unstale":"2023-11-28T20:51:13.521107+0000","last_undegraded":"2023-11-28T20:51:13.521107+0000","last_fullsized":"2023-11-28T20:51:13.521107+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:51.059287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.491281+0000","last_change":"2023-11-28T20:51:12.494355+0000","last_active":"2023-11-28T20:51:13.491281+0000","last_peered":"2023-11-28T20:51:13.491281+0000","last_clean":"2023-11-28T20:51:13.491281+0000","last_became_active":"2023-11-28T20:51:12.493788+0000","last_became_peered":"2023-11-28T20:51:12.493788+0000","last_unstale":"2023-11-28T20:51:13.491281+0000","last_undegraded":"2023-11-28T20:51:13.491281+0000","last_fullsized":"2023-11-28T20:51:13.491281+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:45:18.396393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.513758+0000","last_change":"2023-11-28T20:51:13.507310+0000","last_active":"2023-11-28T20:51:13.513758+0000","last_peered":"2023-11-28T20:51:13.513758+0000","last_clean":"2023-11-28T20:51:13.513758+0000","last_became_active":"2023-11-28T20:51:13.506966+0000","last_became_peered":"2023-11-28T20:51:13.506966+0000","last_unstale":"2023-11-28T20:51:13.513758+0000","last_undegraded":"2023-11-28T20:51:13.513758+0000","last_fullsized":"2023-11-28T20:51:13.513758+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:51:37.137247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.497984+0000","last_change":"2023-11-28T20:51:12.490523+0000","last_active":"2023-11-28T20:51:13.497984+0000","last_peered":"2023-11-28T20:51:13.497984+0000","last_clean":"2023-11-28T20:51:13.497984+0000","last_became_active":"2023-11-28T20:51:12.490313+0000","last_became_peered":"2023-11-28T20:51:12.490313+0000","last_unstale":"2023-11-28T20:51:13.497984+0000","last_undegraded":"2023-11-28T20:51:13.497984+0000","last_fullsized":"2023-11-28T20:51:13.497984+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:49:10.910467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.518977+0000","last_change":"2023-11-28T20:51:13.512380+0000","last_active":"2023-11-28T20:51:13.518977+0000","last_peered":"2023-11-28T20:51:13.518977+0000","last_clean":"2023-11-28T20:51:13.518977+0000","last_became_active":"2023-11-28T20:51:13.512113+0000","last_became_peered":"2023-11-28T20:51:13.512113+0000","last_unstale":"2023-11-28T20:51:13.518977+0000","last_undegraded":"2023-11-28T20:51:13.518977+0000","last_fullsized":"2023-11-28T20:51:13.518977+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:52:15.633702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.509276+0000","last_change":"2023-11-28T20:51:12.496968+0000","last_active":"2023-11-28T20:51:13.509276+0000","last_peered":"2023-11-28T20:51:13.509276+0000","last_clean":"2023-11-28T20:51:13.509276+0000","last_became_active":"2023-11-28T20:51:12.496632+0000","last_became_peered":"2023-11-28T20:51:12.496632+0000","last_unstale":"2023-11-28T20:51:13.509276+0000","last_undegraded":"2023-11-28T20:51:13.509276+0000","last_fullsized":"2023-11-28T20:51:13.509276+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:42:31.025256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.513759+0000","last_change":"2023-11-28T20:51:13.507737+0000","last_active":"2023-11-28T20:51:13.513759+0000","last_peered":"2023-11-28T20:51:13.513759+0000","last_clean":"2023-11-28T20:51:13.513759+0000","last_became_active":"2023-11-28T20:51:13.507404+0000","last_became_peered":"2023-11-28T20:51:13.507404+0000","last_unstale":"2023-11-28T20:51:13.513759+0000","last_undegraded":"2023-11-28T20:51:13.513759+0000","last_fullsized":"2023-11-28T20:51:13.513759+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:29.635174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.486525+0000","last_change":"2023-11-28T20:51:12.492307+0000","last_active":"2023-11-28T20:51:13.486525+0000","last_peered":"2023-11-28T20:51:13.486525+0000","last_clean":"2023-11-28T20:51:13.486525+0000","last_became_active":"2023-11-28T20:51:12.491941+0000","last_became_peered":"2023-11-28T20:51:12.491941+0000","last_unstale":"2023-11-28T20:51:13.486525+0000","last_undegraded":"2023-11-28T20:51:13.486525+0000","last_fullsized":"2023-11-28T20:51:13.486525+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:12:10.438657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514137+0000","last_change":"2023-11-28T20:51:13.507726+0000","last_active":"2023-11-28T20:51:13.514137+0000","last_peered":"2023-11-28T20:51:13.514137+0000","last_clean":"2023-11-28T20:51:13.514137+0000","last_became_active":"2023-11-28T20:51:13.506946+0000","last_became_peered":"2023-11-28T20:51:13.506946+0000","last_unstale":"2023-11-28T20:51:13.514137+0000","last_undegraded":"2023-11-28T20:51:13.514137+0000","last_fullsized":"2023-11-28T20:51:13.514137+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:05.680077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.483498+0000","last_change":"2023-11-28T20:51:12.504344+0000","last_active":"2023-11-28T20:51:13.483498+0000","last_peered":"2023-11-28T20:51:13.483498+0000","last_clean":"2023-11-28T20:51:13.483498+0000","last_became_active":"2023-11-28T20:51:12.503986+0000","last_became_peered":"2023-11-28T20:51:12.503986+0000","last_unstale":"2023-11-28T20:51:13.483498+0000","last_undegraded":"2023-11-28T20:51:13.483498+0000","last_fullsized":"2023-11-28T20:51:13.483498+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:42:10.292240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506894+0000","last_change":"2023-11-28T20:51:13.505478+0000","last_active":"2023-11-28T20:51:13.506894+0000","last_peered":"2023-11-28T20:51:13.506894+0000","last_clean":"2023-11-28T20:51:13.506894+0000","last_became_active":"2023-11-28T20:51:13.505198+0000","last_became_peered":"2023-11-28T20:51:13.505198+0000","last_unstale":"2023-11-28T20:51:13.506894+0000","last_undegraded":"2023-11-28T20:51:13.506894+0000","last_fullsized":"2023-11-28T20:51:13.506894+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:07.963984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.483080+0000","last_change":"2023-11-28T20:51:12.506141+0000","last_active":"2023-11-28T20:51:13.483080+0000","last_peered":"2023-11-28T20:51:13.483080+0000","last_clean":"2023-11-28T20:51:13.483080+0000","last_became_active":"2023-11-28T20:51:12.505844+0000","last_became_peered":"2023-11-28T20:51:12.505844+0000","last_unstale":"2023-11-28T20:51:13.483080+0000","last_undegraded":"2023-11-28T20:51:13.483080+0000","last_fullsized":"2023-11-28T20:51:13.483080+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:50.172662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517105+0000","last_change":"2023-11-28T20:51:13.510215+0000","last_active":"2023-11-28T20:51:13.517105+0000","last_peered":"2023-11-28T20:51:13.517105+0000","last_clean":"2023-11-28T20:51:13.517105+0000","last_became_active":"2023-11-28T20:51:13.506598+0000","last_became_peered":"2023-11-28T20:51:13.506598+0000","last_unstale":"2023-11-28T20:51:13.517105+0000","last_undegraded":"2023-11-28T20:51:13.517105+0000","last_fullsized":"2023-11-28T20:51:13.517105+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:07:09.331469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.487937+0000","last_change":"2023-11-28T20:51:12.501734+0000","last_active":"2023-11-28T20:51:13.487937+0000","last_peered":"2023-11-28T20:51:13.487937+0000","last_clean":"2023-11-28T20:51:13.487937+0000","last_became_active":"2023-11-28T20:51:12.501323+0000","last_became_peered":"2023-11-28T20:51:12.501323+0000","last_unstale":"2023-11-28T20:51:13.487937+0000","last_undegraded":"2023-11-28T20:51:13.487937+0000","last_fullsized":"2023-11-28T20:51:13.487937+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:22:37.776214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.523680+0000","last_change":"2023-11-28T20:51:13.521710+0000","last_active":"2023-11-28T20:51:13.523680+0000","last_peered":"2023-11-28T20:51:13.523680+0000","last_clean":"2023-11-28T20:51:13.523680+0000","last_became_active":"2023-11-28T20:51:13.521407+0000","last_became_peered":"2023-11-28T20:51:13.521407+0000","last_unstale":"2023-11-28T20:51:13.523680+0000","last_undegraded":"2023-11-28T20:51:13.523680+0000","last_fullsized":"2023-11-28T20:51:13.523680+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:24.987055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.479950+0000","last_change":"2023-11-28T20:51:12.483895+0000","last_active":"2023-11-28T20:51:13.479950+0000","last_peered":"2023-11-28T20:51:13.479950+0000","last_clean":"2023-11-28T20:51:13.479950+0000","last_became_active":"2023-11-28T20:51:12.483535+0000","last_became_peered":"2023-11-28T20:51:12.483535+0000","last_unstale":"2023-11-28T20:51:13.479950+0000","last_undegraded":"2023-11-28T20:51:13.479950+0000","last_fullsized":"2023-11-28T20:51:13.479950+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:56:57.373934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.499940+0000","last_change":"2023-11-28T20:51:13.496809+0000","last_active":"2023-11-28T20:51:13.499940+0000","last_peered":"2023-11-28T20:51:13.499940+0000","last_clean":"2023-11-28T20:51:13.499940+0000","last_became_active":"2023-11-28T20:51:13.496494+0000","last_became_peered":"2023-11-28T20:51:13.496494+0000","last_unstale":"2023-11-28T20:51:13.499940+0000","last_undegraded":"2023-11-28T20:51:13.499940+0000","last_fullsized":"2023-11-28T20:51:13.499940+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:21.262203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.500936+0000","last_change":"2023-11-28T20:51:12.490611+0000","last_active":"2023-11-28T20:51:13.500936+0000","last_peered":"2023-11-28T20:51:13.500936+0000","last_clean":"2023-11-28T20:51:13.500936+0000","last_became_active":"2023-11-28T20:51:12.490179+0000","last_became_peered":"2023-11-28T20:51:12.490179+0000","last_unstale":"2023-11-28T20:51:13.500936+0000","last_undegraded":"2023-11-28T20:51:13.500936+0000","last_fullsized":"2023-11-28T20:51:13.500936+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:34:06.270270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.493037+0000","last_change":"2023-11-28T20:51:13.491602+0000","last_active":"2023-11-28T20:51:13.493037+0000","last_peered":"2023-11-28T20:51:13.493037+0000","last_clean":"2023-11-28T20:51:13.493037+0000","last_became_active":"2023-11-28T20:51:13.491254+0000","last_became_peered":"2023-11-28T20:51:13.491254+0000","last_unstale":"2023-11-28T20:51:13.493037+0000","last_undegraded":"2023-11-28T20:51:13.493037+0000","last_fullsized":"2023-11-28T20:51:13.493037+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:26.633298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.473928+0000","last_change":"2023-11-28T20:51:12.494925+0000","last_active":"2023-11-28T20:51:13.473928+0000","last_peered":"2023-11-28T20:51:13.473928+0000","last_clean":"2023-11-28T20:51:13.473928+0000","last_became_active":"2023-11-28T20:51:12.494324+0000","last_became_peered":"2023-11-28T20:51:12.494324+0000","last_unstale":"2023-11-28T20:51:13.473928+0000","last_undegraded":"2023-11-28T20:51:13.473928+0000","last_fullsized":"2023-11-28T20:51:13.473928+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:32.738648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.498187+0000","last_change":"2023-11-28T20:51:13.494390+0000","last_active":"2023-11-28T20:51:13.498187+0000","last_peered":"2023-11-28T20:51:13.498187+0000","last_clean":"2023-11-28T20:51:13.498187+0000","last_became_active":"2023-11-28T20:51:13.491441+0000","last_became_peered":"2023-11-28T20:51:13.491441+0000","last_unstale":"2023-11-28T20:51:13.498187+0000","last_undegraded":"2023-11-28T20:51:13.498187+0000","last_fullsized":"2023-11-28T20:51:13.498187+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:47:57.031660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.492006+0000","last_change":"2023-11-28T20:51:12.495437+0000","last_active":"2023-11-28T20:51:13.492006+0000","last_peered":"2023-11-28T20:51:13.492006+0000","last_clean":"2023-11-28T20:51:13.492006+0000","last_became_active":"2023-11-28T20:51:12.494371+0000","last_became_peered":"2023-11-28T20:51:12.494371+0000","last_unstale":"2023-11-28T20:51:13.492006+0000","last_undegraded":"2023-11-28T20:51:13.492006+0000","last_fullsized":"2023-11-28T20:51:13.492006+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:14:07.789156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"101'2597","reported_seq":2875,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.639384+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:51:13.639384+0000","last_peered":"2023-11-28T20:51:13.639384+0000","last_clean":"2023-11-28T20:51:13.639384+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:51:13.639384+0000","last_undegraded":"2023-11-28T20:51:13.639384+0000","last_fullsized":"2023-11-28T20:51:13.639384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":2597,"log_dups_size":0,"ondisk_log_size":2597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":570,"num_read_kb":1050,"num_write":246,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":570,"num_read_kb":1050,"num_write":246,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2597,"ondisk_log_size":2597,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575056,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55184,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316656,"statfs":{"total":96636764160,"available":96580255744,"internally_reserved":0,"allocated":1830912,"data_stored":1211202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1919999999999999,"15min":1.194},"min":{"1min":0.79700000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.722,"5min":1.722,"15min":2.23},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.153,"15min":1.1799999999999999},"min":{"1min":0.84299999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5169999999999999,"5min":1.681,"15min":1.726},"last":0.84899999999999998}]},{"osd":1,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1830000000000001,"15min":1.1850000000000001},"min":{"1min":0.83099999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.579,"5min":1.744,"15min":1.744},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.92100000000000004,"5min":0.624,"15min":0.624},"max":{"1min":1.6679999999999999,"5min":1.835,"15min":2.3599999999999999},"last":0.747}]},{"osd":2,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.2949999999999999,"15min":1.3029999999999999},"min":{"1min":0.97199999999999998,"5min":0.80300000000000005,"15min":0.58799999999999997},"max":{"1min":1.655,"5min":1.8300000000000001,"15min":1.9359999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.292,"15min":1.2869999999999999},"min":{"1min":0.96899999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.675,"5min":1.847,"15min":2.137},"last":1.105}]},{"osd":3,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2649999999999999,"15min":1.276},"min":{"1min":0.73199999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.577,"5min":1.7729999999999999,"15min":1.8460000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.403,"15min":1.4059999999999999},"min":{"1min":1.1579999999999999,"5min":0.88300000000000001,"15min":0.68300000000000005},"max":{"1min":1.956,"5min":1.956,"15min":2.4220000000000002},"last":1.5920000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.2629999999999999,"15min":1.264},"min":{"1min":0.83999999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7649999999999999,"5min":1.7989999999999999,"15min":1.919},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3200000000000001,"15min":1.329},"min":{"1min":0.93000000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.794,"5min":1.863,"15min":2.0049999999999999},"last":1.294}]},{"osd":5,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":0.97199999999999998,"5min":0.86499999999999999,"15min":0.67900000000000005},"max":{"1min":1.891,"5min":1.9219999999999999,"15min":2.5449999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.47,"15min":1.46},"min":{"1min":1.1160000000000001,"5min":0.73499999999999999,"15min":0.72199999999999998},"max":{"1min":1.931,"5min":1.9550000000000001,"15min":2.1000000000000001},"last":1.24}]},{"osd":6,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5649999999999999,"15min":1.548},"min":{"1min":1.1779999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.2389999999999999},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4830000000000001,"15min":1.488},"min":{"1min":1.056,"5min":0.94099999999999995,"15min":0.90800000000000003},"max":{"1min":1.9139999999999999,"5min":1.9870000000000001,"15min":2.3079999999999998},"last":1.5169999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575056,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1040000000000001,"15min":1.1140000000000001},"min":{"1min":0.71799999999999997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.125,"15min":1.1319999999999999},"min":{"1min":0.73099999999999998,"5min":0.63500000000000001,"15min":0.627},"max":{"1min":1.452,"5min":1.7549999999999999,"15min":2.0739999999999998},"last":0.80400000000000005}]},{"osd":1,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2130000000000001,"15min":1.2569999999999999},"min":{"1min":0.84399999999999997,"5min":0.67200000000000004,"15min":0.52200000000000002},"max":{"1min":1.728,"5min":1.728,"15min":1.8069999999999999},"last":1.341},{"interface":"front","average":{"1min":1.409,"5min":1.359,"15min":1.3700000000000001},"min":{"1min":0.97199999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6850000000000001,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":1.431}]},{"osd":2,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.2709999999999999,"15min":1.2669999999999999},"min":{"1min":0.89400000000000002,"5min":0.71099999999999997,"15min":0.69799999999999995},"max":{"1min":1.631,"5min":1.6879999999999999,"15min":1.71},"last":1.286},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.355,"15min":1.361},"min":{"1min":0.84999999999999998,"5min":0.73199999999999998,"15min":0.60299999999999998},"max":{"1min":1.6439999999999999,"5min":1.851,"15min":1.851},"last":1.113}]},{"osd":3,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.2729999999999999,"15min":1.3},"min":{"1min":0.96799999999999997,"5min":0.70099999999999996,"15min":0.68799999999999994},"max":{"1min":1.6200000000000001,"5min":1.821,"15min":2.222},"last":1.23},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2549999999999999,"15min":1.2669999999999999},"min":{"1min":0.95399999999999996,"5min":0.83899999999999997,"15min":0.751},"max":{"1min":1.635,"5min":1.6599999999999999,"15min":1.7949999999999999},"last":1.171}]},{"osd":4,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2509999999999999,"15min":1.2729999999999999},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.704,"5min":1.849,"15min":2.7290000000000001},"last":1.647},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.286,"15min":1.329},"min":{"1min":0.84099999999999997,"5min":0.80800000000000005,"15min":0.76400000000000001},"max":{"1min":1.752,"5min":2.2010000000000001,"15min":3.2519999999999998},"last":1.597}]},{"osd":5,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3100000000000001,"15min":1.325},"min":{"1min":0.96999999999999997,"5min":0.754,"15min":0.62},"max":{"1min":1.714,"5min":1.714,"15min":2.0539999999999998},"last":1.383},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5129999999999999,"15min":1.5469999999999999},"min":{"1min":1.2709999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.804,"5min":1.9179999999999999,"15min":3.0310000000000001},"last":1.5369999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.45,"15min":1.494},"min":{"1min":1.014,"5min":0.97799999999999998,"15min":0.86499999999999999},"max":{"1min":1.8080000000000001,"5min":2.1120000000000001,"15min":3.379},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.522,"5min":1.4490000000000001,"15min":1.462},"min":{"1min":1.0669999999999999,"5min":0.93300000000000005,"15min":0.84599999999999997},"max":{"1min":1.7669999999999999,"5min":1.8859999999999999,"15min":2.0739999999999998},"last":1.488}]}]},{"osd":5,"up_from":13,"seq":55834575056,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50428,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321412,"statfs":{"total":96636764160,"available":96585125888,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.159,"15min":1.175},"min":{"1min":0.72199999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.702,"5min":1.702,"15min":1.768},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1160000000000001,"15min":1.1879999999999999},"min":{"1min":0.56200000000000006,"5min":0.441,"15min":0.441},"max":{"1min":1.6060000000000001,"5min":1.631,"15min":1.829},"last":1.0129999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.115,"15min":1.1499999999999999},"min":{"1min":0.67200000000000004,"5min":0.57399999999999995,"15min":0.48899999999999999},"max":{"1min":1.587,"5min":1.768,"15min":1.768},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.208,"5min":1.204,"15min":1.218},"min":{"1min":0.72399999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5860000000000001,"5min":1.8879999999999999,"15min":1.956},"last":1.389}]},{"osd":2,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2529999999999999,"15min":1.268},"min":{"1min":0.68600000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.325},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.226,"15min":1.292},"min":{"1min":0.76900000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.958},"last":1.4279999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.276,"15min":1.3169999999999999},"min":{"1min":0.77700000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.379,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.2490000000000001,"15min":1.2689999999999999},"min":{"1min":0.83299999999999996,"5min":0.64000000000000001,"15min":0.54600000000000004},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.2430000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.284,"15min":1.2949999999999999},"min":{"1min":0.875,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.772,"5min":1.772,"15min":1.8380000000000001},"last":1.347},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.45,"15min":1.4370000000000001},"min":{"1min":1.0760000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.86399999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3029999999999999,"15min":1.325},"min":{"1min":0.82199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":2.093},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.464,"15min":1.464},"min":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.056}]},{"osd":7,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.421,"15min":1.4179999999999999},"min":{"1min":1.0349999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.466,"15min":1.4830000000000001},"min":{"1min":1.0620000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.958,"5min":1.958,"15min":2.1829999999999998},"last":1.28}]}]},{"osd":4,"up_from":13,"seq":55834575057,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1257472,"data_stored":622550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.139,"15min":1.129},"min":{"1min":0.80300000000000005,"5min":0.66000000000000003,"15min":0.64400000000000002},"max":{"1min":1.4359999999999999,"5min":1.6659999999999999,"15min":1.879},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1339999999999999,"15min":1.1180000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6850000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.65300000000000002}]},{"osd":1,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.29,"15min":1.3100000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.74299999999999999},"max":{"1min":1.609,"5min":1.7490000000000001,"15min":1.923},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.327,"5min":1.375,"15min":1.3520000000000001},"min":{"1min":1.026,"5min":0.86599999999999999,"15min":0.76700000000000002},"max":{"1min":1.768,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.80600000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.232,"15min":1.2110000000000001},"min":{"1min":1.0229999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.589,"5min":1.71,"15min":1.71},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.278,"15min":1.2949999999999999},"min":{"1min":1.083,"5min":0.77400000000000002,"15min":0.77200000000000002},"max":{"1min":1.5189999999999999,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":0.85399999999999998}]},{"osd":3,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.294,"15min":1.2869999999999999},"min":{"1min":0.97499999999999998,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.76,"5min":2.157,"15min":2.157},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.395,"5min":1.361,"15min":1.377},"min":{"1min":1.087,"5min":0.876,"15min":0.67900000000000005},"max":{"1min":1.718,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.97999999999999998}]},{"osd":5,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.3260000000000001,"15min":1.278},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.65800000000000003},"max":{"1min":1.655,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.083},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3620000000000001,"15min":1.2969999999999999},"min":{"1min":0.84099999999999997,"5min":0.83799999999999997,"15min":0.69299999999999995},"max":{"1min":2.0579999999999998,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.02}]},{"osd":6,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3979999999999999,"15min":1.401},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.628},"max":{"1min":1.8049999999999999,"5min":2.04,"15min":2.04},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.5780000000000001,"15min":1.5489999999999999},"min":{"1min":1.3120000000000001,"5min":1.0369999999999999,"15min":0.80600000000000005},"max":{"1min":1.8380000000000001,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":1.151}]},{"osd":7,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4930000000000001,"15min":1.45},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.76200000000000001},"max":{"1min":1.9570000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.532,"5min":1.538,"15min":1.526},"min":{"1min":1.208,"5min":1.0349999999999999,"15min":0.72299999999999998},"max":{"1min":1.8580000000000001,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.133}]}]},{"osd":3,"up_from":13,"seq":55834575057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40700,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331140,"statfs":{"total":96636764160,"available":96595087360,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.873,"15min":0.90600000000000003},"min":{"1min":0.45500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.6000000000000001,"5min":4.7210000000000001,"15min":4.7210000000000001},"last":1.036},{"interface":"front","average":{"1min":0.875,"5min":0.96499999999999997,"15min":1.071},"min":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.4870000000000001,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":0.82399999999999995}]},{"osd":1,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.95199999999999996,"15min":0.97499999999999998},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4730000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.745},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.1499999999999999,"15min":1.1990000000000001},"min":{"1min":0.58499999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5289999999999999,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.67100000000000004}]},{"osd":2,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.1839999999999999,"15min":1.2470000000000001},"min":{"1min":0.55800000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4490000000000001,"5min":3.444,"15min":3.444},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2769999999999999,"15min":1.335},"min":{"1min":0.66800000000000004,"5min":0.58099999999999996,"15min":0.54400000000000004},"max":{"1min":1.7230000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":1.454}]},{"osd":4,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.298,"15min":1.3420000000000001},"min":{"1min":0.84499999999999997,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.6890000000000001,"5min":1.8779999999999999,"15min":2.9420000000000002},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.327,"15min":1.3839999999999999},"min":{"1min":0.86399999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.1389999999999998},"last":1.1299999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4259999999999999,"15min":1.452},"min":{"1min":0.88,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.8959999999999999,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":1.101},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.375,"15min":1.4330000000000001},"min":{"1min":0.79800000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9670000000000001,"5min":2.2429999999999999,"15min":3.1520000000000001},"last":0.98199999999999998}]},{"osd":6,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3979999999999999,"15min":1.4790000000000001},"min":{"1min":0.86099999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.3100000000000001},"last":1.24},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.5309999999999999,"15min":1.577},"min":{"1min":0.91100000000000003,"5min":0.749,"15min":0.749},"max":{"1min":1.7729999999999999,"5min":3.661,"15min":3.661},"last":1.4119999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.45,"15min":1.5369999999999999},"min":{"1min":0.73399999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.863,"5min":2.1469999999999998,"15min":3.266},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.514,"15min":1.601},"min":{"1min":0.90200000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.1269999999999998,"5min":2.3439999999999999,"15min":2.3919999999999999},"last":1.512}]}]},{"osd":2,"up_from":13,"seq":55834575057,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58748,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313092,"statfs":{"total":96636764160,"available":96576606208,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.88300000000000001,"15min":0.91200000000000003},"min":{"1min":0.54200000000000004,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.624},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.996,"15min":1.0589999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.486,"5min":1.6419999999999999,"15min":1.9390000000000001},"last":0.877}]},{"osd":1,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95399999999999996,"15min":0.98499999999999999},"min":{"1min":0.60799999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.403,"5min":1.7669999999999999,"15min":3.2599999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.099,"15min":1.1499999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.99199999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1240000000000001,"15min":1.1579999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4770000000000001,"5min":1.774,"15min":2.194},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2390000000000001,"15min":1.2929999999999999},"min":{"1min":0.53500000000000003,"5min":0.48199999999999998,"15min":0.45600000000000002},"max":{"1min":1.6579999999999999,"5min":1.831,"15min":3.1440000000000001},"last":1.0329999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.389,"15min":1.411},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.853,"5min":2.23,"15min":2.8919999999999999},"last":1.276},{"interface":"front","average":{"1min":1.389,"5min":1.387,"15min":1.4119999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.988,"5min":2.3740000000000001,"15min":2.3809999999999998},"last":1.5800000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.3919999999999999,"15min":1.421},"min":{"1min":0.85099999999999998,"5min":0.82199999999999995,"15min":0.79600000000000004},"max":{"1min":2.327,"5min":2.327,"15min":2.327},"last":1.218},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4650000000000001,"15min":1.496},"min":{"1min":0.96499999999999997,"5min":0.89100000000000001,"15min":0.75800000000000001},"max":{"1min":2.077,"5min":2.1680000000000001,"15min":3.4279999999999999},"last":1.486}]},{"osd":6,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.452,"15min":1.502},"min":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.165,"5min":2.1709999999999998,"15min":2.621},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.526,"15min":1.5740000000000001},"min":{"1min":0.89000000000000001,"5min":0.82499999999999996,"15min":0.78300000000000003},"max":{"1min":2.6680000000000001,"5min":2.6749999999999998,"15min":3.6269999999999998},"last":1.6279999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.508,"15min":1.542},"min":{"1min":0.94299999999999995,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.54,"15min":1.5900000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.2480000000000002,"5min":2.4849999999999999,"15min":2.7109999999999999},"last":1.796}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36540,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335300,"statfs":{"total":96636764160,"available":96599347200,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.88900000000000001,"15min":0.90400000000000003},"min":{"1min":0.67600000000000005,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4419999999999999,"5min":1.6639999999999999,"15min":1.9259999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.0589999999999999,"15min":1.077},"min":{"1min":0.72999999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":0.88600000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0389999999999999,"15min":1.089},"min":{"1min":0.66200000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.4850000000000001,"5min":1.619,"15min":2.234},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.29,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.88300000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.895},"last":1.111}]},{"osd":3,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.26,"15min":1.3160000000000001},"min":{"1min":0.64400000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.748,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.083,"15min":1.1120000000000001},"min":{"1min":0.84199999999999997,"5min":0.37,"15min":0.37},"max":{"1min":1.744,"5min":1.887,"15min":2.3679999999999999},"last":1.7490000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3220000000000001,"15min":1.337},"min":{"1min":0.97399999999999998,"5min":0.79200000000000004,"15min":0.76600000000000001},"max":{"1min":1.742,"5min":2.1019999999999999,"15min":2.1150000000000002},"last":1.49},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.343,"15min":1.387},"min":{"1min":1.0349999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8799999999999999,"5min":2.1469999999999998,"15min":2.6819999999999999},"last":1.296}]},{"osd":5,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.4059999999999999,"15min":1.429},"min":{"1min":1.165,"5min":0.81100000000000005,"15min":0.81000000000000005},"max":{"1min":1.8620000000000001,"5min":1.903,"15min":2.1840000000000002},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.4870000000000001,"15min":1.496},"min":{"1min":1.2130000000000001,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.8280000000000001,"5min":2.294,"15min":2.294},"last":1.238}]},{"osd":6,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.3819999999999999,"15min":1.4199999999999999},"min":{"1min":1.0980000000000001,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7749999999999999,"5min":1.8759999999999999,"15min":1.9379999999999999},"last":1.643},{"interface":"front","average":{"1min":1.554,"5min":1.4990000000000001,"15min":1.524},"min":{"1min":1.3120000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.956,"5min":2.1520000000000001,"15min":2.3300000000000001},"last":1.425}]},{"osd":7,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.542,"15min":1.5640000000000001},"min":{"1min":1.165,"5min":0.753,"15min":0.753},"max":{"1min":2.1429999999999998,"5min":2.218,"15min":2.3359999999999999},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.504,"15min":1.5169999999999999},"min":{"1min":1.3320000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":2.294},"last":1.1870000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575057,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32960,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338880,"statfs":{"total":96636764160,"available":96603013120,"internally_reserved":0,"allocated":1818624,"data_stored":1204107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91500000000000004,"15min":0.90800000000000003},"min":{"1min":0.52100000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3240000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0509999999999999,"15min":1.0800000000000001},"min":{"1min":0.54200000000000004,"5min":0.501,"15min":0.45700000000000002},"max":{"1min":1.534,"5min":1.764,"15min":1.764},"last":0.93100000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.054,"15min":1.1060000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.6080000000000001,"5min":1.6220000000000001,"15min":1.7709999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1950000000000001},"min":{"1min":0.59899999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.722,"5min":1.893,"15min":2.4740000000000002},"last":1.4319999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0660000000000001,"15min":1.1180000000000001},"min":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.508,"5min":2.081,"15min":2.081},"last":0.996},{"interface":"front","average":{"1min":1.262,"5min":1.24,"15min":1.3149999999999999},"min":{"1min":0.67700000000000005,"5min":0.54400000000000004,"15min":0.52500000000000002},"max":{"1min":1.8180000000000001,"5min":1.8839999999999999,"15min":2.1080000000000001},"last":1.512}]},{"osd":4,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3979999999999999,"15min":1.417},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.804,"5min":2.012,"15min":2.012},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.385,"15min":1.3740000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.68300000000000005},"max":{"1min":1.7230000000000001,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.4650000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.403,"15min":1.4159999999999999},"min":{"1min":0.84799999999999998,"5min":0.80100000000000005,"15min":0.79400000000000004},"max":{"1min":1.8460000000000001,"5min":2.0350000000000001,"15min":2.528},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.462,"15min":1.484},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.992,"5min":2.0750000000000002,"15min":2.2490000000000001},"last":1.409}]},{"osd":6,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.512,"15min":1.5489999999999999},"min":{"1min":0.78800000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.0419999999999998,"5min":2.1160000000000001,"15min":2.2959999999999998},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.504,"15min":1.5249999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2330000000000001,"5min":2.274,"15min":2.573},"last":1.2509999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5720000000000001,"15min":1.591},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.2970000000000002,"5min":2.3610000000000002,"15min":2.4100000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5049999999999999,"15min":1.5429999999999999},"min":{"1min":0.875,"5min":0.81699999999999995,"15min":0.73799999999999999},"max":{"1min":1.776,"5min":2.3460000000000001,"15min":2.6640000000000001},"last":1.2010000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:51:17.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:51:18.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.259+0000 7f33bd547640 1 Processor -- start 2023-11-28T20:51:18.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.259+0000 7f33bd547640 1 -- start start 2023-11-28T20:51:18.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.261+0000 7f33bd547640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d02d0 0x7f33b80d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.261+0000 7f33bd547640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f33b80d1ab0 0x7f33b80b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.261+0000 7f33bd547640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b80d2230 con 0x7f33b815d550 2023-11-28T20:51:18.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.261+0000 7f33bd547640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b80d2510 con 0x7f33b80d1ab0 2023-11-28T20:51:18.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.261+0000 7f33bd547640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80d27f0 con 0x7f33b80d02d0 2023-11-28T20:51:18.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.261+0000 7f33bcd46640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f33b80d1ab0 0x7f33b80b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.262+0000 7f33bcd46640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f33b80d1ab0 0x7f33b80b6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44826/0 (socket says 172.21.15.45:44826) 2023-11-28T20:51:18.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.262+0000 7f33bcd46640 1 -- 172.21.15.45:0/3280384633 learned_addr learned my addr 172.21.15.45:0/3280384633 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:18.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.262+0000 7f33affff640 1 --2- 172.21.15.45:0/3280384633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d02d0 0x7f33b80d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.262+0000 7f33bcd46640 1 -- 172.21.15.45:0/3280384633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d02d0 msgr2=0x7f33b80d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:18.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.262+0000 7f33bcd46640 1 --2- 172.21.15.45:0/3280384633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d02d0 0x7f33b80d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.262+0000 7f33bcd46640 1 -- 172.21.15.45:0/3280384633 >> v1:172.21.15.45:6789/0 conn(0x7f33b815d550 legacy=0x7f33b80cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.262+0000 7f33bcd46640 1 -- 172.21.15.45:0/3280384633 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80b6a30 con 0x7f33b80d1ab0 2023-11-28T20:51:18.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.263+0000 7f33bcd46640 1 --2- 172.21.15.45:0/3280384633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b80d1ab0 0x7f33b80b6f50 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009e50 tx=0x7f33b002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9320dc8c585e08a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:18.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.263+0000 7f33b77fe640 1 -- 172.21.15.45:0/3280384633 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b002cb00 con 0x7f33b80d1ab0 2023-11-28T20:51:18.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33b77fe640 1 -- 172.21.15.45:0/3280384633 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b002cc60 con 0x7f33b80d1ab0 2023-11-28T20:51:18.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33b77fe640 1 -- 172.21.15.45:0/3280384633 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b00343f0 con 0x7f33b80d1ab0 2023-11-28T20:51:18.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33bd547640 1 -- 172.21.15.45:0/3280384633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b80d1ab0 msgr2=0x7f33b80b6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33bd547640 1 --2- 172.21.15.45:0/3280384633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b80d1ab0 0x7f33b80b6f50 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009e50 tx=0x7f33b002bf60 comp rx=0 tx=0).stop 2023-11-28T20:51:18.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33bd547640 1 -- 172.21.15.45:0/3280384633 shutdown_connections 2023-11-28T20:51:18.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33bd547640 1 --2- 172.21.15.45:0/3280384633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b80d1ab0 0x7f33b80b6f50 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33bd547640 1 --2- 172.21.15.45:0/3280384633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d02d0 0x7f33b80d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33bd547640 1 -- 172.21.15.45:0/3280384633 >> 172.21.15.45:0/3280384633 conn(0x7f33b806ac70 msgr2=0x7f33b80b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:18.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33bd547640 1 -- 172.21.15.45:0/3280384633 shutdown_connections 2023-11-28T20:51:18.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.264+0000 7f33bd547640 1 -- 172.21.15.45:0/3280384633 wait complete. 2023-11-28T20:51:18.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.266+0000 7f33bd547640 1 Processor -- start 2023-11-28T20:51:18.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.266+0000 7f33bd547640 1 -- start start 2023-11-28T20:51:18.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.267+0000 7f33bd547640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33b80d02d0 0x7f33b8156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.267+0000 7f33bd547640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d1ab0 0x7f33b815e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.267+0000 7f33bd547640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b815d550 0x7f33b815e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.267+0000 7f33bd547640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b81590f0 con 0x7f33b815d550 2023-11-28T20:51:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.267+0000 7f33bd547640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80ca440 con 0x7f33b80d02d0 2023-11-28T20:51:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.267+0000 7f33bd547640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b8158e10 con 0x7f33b80d1ab0 2023-11-28T20:51:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.267+0000 7f33bcd46640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b815d550 0x7f33b815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.267+0000 7f33bcd46640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b815d550 0x7f33b815e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44838/0 (socket says 172.21.15.45:44838) 2023-11-28T20:51:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.267+0000 7f33bcd46640 1 -- 172.21.15.45:0/541946650 learned_addr learned my addr 172.21.15.45:0/541946650 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.268+0000 7f33bcd46640 1 -- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d1ab0 msgr2=0x7f33b815e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.268+0000 7f33affff640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d1ab0 0x7f33b815e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.268+0000 7f33bcd46640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d1ab0 0x7f33b815e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.268+0000 7f33bcd46640 1 -- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33b80d02d0 msgr2=0x7f33b8156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.268+0000 7f33b7fff640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33b80d02d0 0x7f33b8156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.268+0000 7f33bcd46640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33b80d02d0 0x7f33b8156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.268+0000 7f33bcd46640 1 -- 172.21.15.45:0/541946650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b0009b30 con 0x7f33b815d550 2023-11-28T20:51:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.268+0000 7f33b7fff640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33b80d02d0 0x7f33b8156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.268+0000 7f33bcd46640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b815d550 0x7f33b815e650 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009e50 tx=0x7f33b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:18.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.269+0000 7f33b57fa640 1 -- 172.21.15.45:0/541946650 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b0009510 con 0x7f33b815d550 2023-11-28T20:51:18.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.269+0000 7f33b57fa640 1 -- 172.21.15.45:0/541946650 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b003e450 con 0x7f33b815d550 2023-11-28T20:51:18.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.269+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b80ca6e0 con 0x7f33b815d550 2023-11-28T20:51:18.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.269+0000 7f33b57fa640 1 -- 172.21.15.45:0/541946650 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b003e800 con 0x7f33b815d550 2023-11-28T20:51:18.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.269+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b8156650 con 0x7f33b815d550 2023-11-28T20:51:18.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.270+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b80ca2a0 con 0x7f33b815d550 2023-11-28T20:51:18.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.271+0000 7f33b57fa640 1 -- 172.21.15.45:0/541946650 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f33b003e960 con 0x7f33b815d550 2023-11-28T20:51:18.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.272+0000 7f33b57fa640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f339806af60 0x7f339806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.272+0000 7f33b57fa640 1 -- 172.21.15.45:0/541946650 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f33b00ac690 con 0x7f33b815d550 2023-11-28T20:51:18.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.273+0000 7f33b7fff640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f339806af60 0x7f339806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.274+0000 7f33b7fff640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f339806af60 0x7f339806d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f33a8000f60 tx=0x7f33a8009040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:18.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.277+0000 7f33b57fa640 1 -- 172.21.15.45:0/541946650 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33b80ca2a0 con 0x7f33b815d550 2023-11-28T20:51:18.438 INFO:tasks.ceph.mds.c.smithi045.stderr:2023-11-28T20:51:18.436+0000 7ff5d20b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:51:18.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.469+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 --> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33b815d930 con 0x7f339806af60 2023-11-28T20:51:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.491+0000 7f33b57fa640 1 -- 172.21.15.45:0/541946650 <== mgr.5959 v2:172.21.15.190:6832/970930168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286766 (secure 0 0 0) 0x7f33a80013e0 con 0x7f339806af60 2023-11-28T20:51:18.496 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:51:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.497+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f339806af60 msgr2=0x7f339806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.497+0000 7f33bd547640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f339806af60 0x7f339806d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f33a8000f60 tx=0x7f33a8009040 comp rx=0 tx=0).stop 2023-11-28T20:51:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.497+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b815d550 msgr2=0x7f33b815e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.497+0000 7f33bd547640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b815d550 0x7f33b815e650 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009e50 tx=0x7f33b0004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.498+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 shutdown_connections 2023-11-28T20:51:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.498+0000 7f33bd547640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f339806af60 0x7f339806d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.498+0000 7f33bd547640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33b815d550 0x7f33b815e650 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.498+0000 7f33bd547640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33b80d1ab0 0x7f33b815e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.498+0000 7f33bd547640 1 --2- 172.21.15.45:0/541946650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33b80d02d0 0x7f33b8156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.498+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 >> 172.21.15.45:0/541946650 conn(0x7f33b806ac70 msgr2=0x7f33b80d32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.498+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 shutdown_connections 2023-11-28T20:51:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.498+0000 7f33bd547640 1 -- 172.21.15.45:0/541946650 wait complete. 2023-11-28T20:51:18.519 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-11-28T20:51:17.793993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":570,"num_read_kb":1050,"num_write":290,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2635,"ondisk_log_size":2635,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":339348,"kb_used_data":10772,"kb_used_omap":12,"kb_used_meta":328435,"kb_avail":754635372,"statfs":{"total":773094113280,"available":772746620928,"internally_reserved":0,"allocated":11030528,"data_stored":6106554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":336318000},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001917"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.518119+0000","last_change":"2023-11-28T20:51:13.515389+0000","last_active":"2023-11-28T20:51:13.518119+0000","last_peered":"2023-11-28T20:51:13.518119+0000","last_clean":"2023-11-28T20:51:13.518119+0000","last_became_active":"2023-11-28T20:51:13.515100+0000","last_became_peered":"2023-11-28T20:51:13.515100+0000","last_unstale":"2023-11-28T20:51:13.518119+0000","last_undegraded":"2023-11-28T20:51:13.518119+0000","last_fullsized":"2023-11-28T20:51:13.518119+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:18:42.410562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506897+0000","last_change":"2023-11-28T20:51:13.503822+0000","last_active":"2023-11-28T20:51:13.506897+0000","last_peered":"2023-11-28T20:51:13.506897+0000","last_clean":"2023-11-28T20:51:13.506897+0000","last_became_active":"2023-11-28T20:51:13.503546+0000","last_became_peered":"2023-11-28T20:51:13.503546+0000","last_unstale":"2023-11-28T20:51:13.506897+0000","last_undegraded":"2023-11-28T20:51:13.506897+0000","last_fullsized":"2023-11-28T20:51:13.506897+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:39:26.295263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.509803+0000","last_change":"2023-11-28T20:51:13.507476+0000","last_active":"2023-11-28T20:51:13.509803+0000","last_peered":"2023-11-28T20:51:13.509803+0000","last_clean":"2023-11-28T20:51:13.509803+0000","last_became_active":"2023-11-28T20:51:13.507042+0000","last_became_peered":"2023-11-28T20:51:13.507042+0000","last_unstale":"2023-11-28T20:51:13.509803+0000","last_undegraded":"2023-11-28T20:51:13.509803+0000","last_fullsized":"2023-11-28T20:51:13.509803+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:02.952326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517355+0000","last_change":"2023-11-28T20:51:13.516325+0000","last_active":"2023-11-28T20:51:13.517355+0000","last_peered":"2023-11-28T20:51:13.517355+0000","last_clean":"2023-11-28T20:51:13.517355+0000","last_became_active":"2023-11-28T20:51:13.513015+0000","last_became_peered":"2023-11-28T20:51:13.513015+0000","last_unstale":"2023-11-28T20:51:13.517355+0000","last_undegraded":"2023-11-28T20:51:13.517355+0000","last_fullsized":"2023-11-28T20:51:13.517355+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:34:22.392589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.497652+0000","last_change":"2023-11-28T20:51:13.492454+0000","last_active":"2023-11-28T20:51:13.497652+0000","last_peered":"2023-11-28T20:51:13.497652+0000","last_clean":"2023-11-28T20:51:13.497652+0000","last_became_active":"2023-11-28T20:51:13.492144+0000","last_became_peered":"2023-11-28T20:51:13.492144+0000","last_unstale":"2023-11-28T20:51:13.497652+0000","last_undegraded":"2023-11-28T20:51:13.497652+0000","last_fullsized":"2023-11-28T20:51:13.497652+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:26:37.646917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514320+0000","last_change":"2023-11-28T20:51:13.510642+0000","last_active":"2023-11-28T20:51:13.514320+0000","last_peered":"2023-11-28T20:51:13.514320+0000","last_clean":"2023-11-28T20:51:13.514320+0000","last_became_active":"2023-11-28T20:51:13.510143+0000","last_became_peered":"2023-11-28T20:51:13.510143+0000","last_unstale":"2023-11-28T20:51:13.514320+0000","last_undegraded":"2023-11-28T20:51:13.514320+0000","last_fullsized":"2023-11-28T20:51:13.514320+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:39:26.295263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522161+0000","last_change":"2023-11-28T20:51:13.520701+0000","last_active":"2023-11-28T20:51:13.522161+0000","last_peered":"2023-11-28T20:51:13.522161+0000","last_clean":"2023-11-28T20:51:13.522161+0000","last_became_active":"2023-11-28T20:51:13.520389+0000","last_became_peered":"2023-11-28T20:51:13.520389+0000","last_unstale":"2023-11-28T20:51:13.522161+0000","last_undegraded":"2023-11-28T20:51:13.522161+0000","last_fullsized":"2023-11-28T20:51:13.522161+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:05:55.771224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.507710+0000","last_change":"2023-11-28T20:51:13.495833+0000","last_active":"2023-11-28T20:51:13.507710+0000","last_peered":"2023-11-28T20:51:13.507710+0000","last_clean":"2023-11-28T20:51:13.507710+0000","last_became_active":"2023-11-28T20:51:13.493644+0000","last_became_peered":"2023-11-28T20:51:13.493644+0000","last_unstale":"2023-11-28T20:51:13.507710+0000","last_undegraded":"2023-11-28T20:51:13.507710+0000","last_fullsized":"2023-11-28T20:51:13.507710+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:00.768908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.516606+0000","last_change":"2023-11-28T20:51:13.511246+0000","last_active":"2023-11-28T20:51:13.516606+0000","last_peered":"2023-11-28T20:51:13.516606+0000","last_clean":"2023-11-28T20:51:13.516606+0000","last_became_active":"2023-11-28T20:51:13.509466+0000","last_became_peered":"2023-11-28T20:51:13.509466+0000","last_unstale":"2023-11-28T20:51:13.516606+0000","last_undegraded":"2023-11-28T20:51:13.516606+0000","last_fullsized":"2023-11-28T20:51:13.516606+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:24.987055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.519964+0000","last_change":"2023-11-28T20:51:13.516783+0000","last_active":"2023-11-28T20:51:13.519964+0000","last_peered":"2023-11-28T20:51:13.519964+0000","last_clean":"2023-11-28T20:51:13.519964+0000","last_became_active":"2023-11-28T20:51:13.516508+0000","last_became_peered":"2023-11-28T20:51:13.516508+0000","last_unstale":"2023-11-28T20:51:13.519964+0000","last_undegraded":"2023-11-28T20:51:13.519964+0000","last_fullsized":"2023-11-28T20:51:13.519964+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:18.509118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506936+0000","last_change":"2023-11-28T20:51:13.505792+0000","last_active":"2023-11-28T20:51:13.506936+0000","last_peered":"2023-11-28T20:51:13.506936+0000","last_clean":"2023-11-28T20:51:13.506936+0000","last_became_active":"2023-11-28T20:51:13.504698+0000","last_became_peered":"2023-11-28T20:51:13.504698+0000","last_unstale":"2023-11-28T20:51:13.506936+0000","last_undegraded":"2023-11-28T20:51:13.506936+0000","last_fullsized":"2023-11-28T20:51:13.506936+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:19:40.944865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520761+0000","last_change":"2023-11-28T20:51:13.518586+0000","last_active":"2023-11-28T20:51:13.520761+0000","last_peered":"2023-11-28T20:51:13.520761+0000","last_clean":"2023-11-28T20:51:13.520761+0000","last_became_active":"2023-11-28T20:51:13.517764+0000","last_became_peered":"2023-11-28T20:51:13.517764+0000","last_unstale":"2023-11-28T20:51:13.520761+0000","last_undegraded":"2023-11-28T20:51:13.520761+0000","last_fullsized":"2023-11-28T20:51:13.520761+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:13:06.103102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520417+0000","last_change":"2023-11-28T20:51:13.508273+0000","last_active":"2023-11-28T20:51:13.520417+0000","last_peered":"2023-11-28T20:51:13.520417+0000","last_clean":"2023-11-28T20:51:13.520417+0000","last_became_active":"2023-11-28T20:51:13.507975+0000","last_became_peered":"2023-11-28T20:51:13.507975+0000","last_unstale":"2023-11-28T20:51:13.520417+0000","last_undegraded":"2023-11-28T20:51:13.520417+0000","last_fullsized":"2023-11-28T20:51:13.520417+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:32:16.543758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514300+0000","last_change":"2023-11-28T20:51:13.510641+0000","last_active":"2023-11-28T20:51:13.514300+0000","last_peered":"2023-11-28T20:51:13.514300+0000","last_clean":"2023-11-28T20:51:13.514300+0000","last_became_active":"2023-11-28T20:51:13.510128+0000","last_became_peered":"2023-11-28T20:51:13.510128+0000","last_unstale":"2023-11-28T20:51:13.514300+0000","last_undegraded":"2023-11-28T20:51:13.514300+0000","last_fullsized":"2023-11-28T20:51:13.514300+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:19:40.944865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.500188+0000","last_change":"2023-11-28T20:51:13.495228+0000","last_active":"2023-11-28T20:51:13.500188+0000","last_peered":"2023-11-28T20:51:13.500188+0000","last_clean":"2023-11-28T20:51:13.500188+0000","last_became_active":"2023-11-28T20:51:13.494949+0000","last_became_peered":"2023-11-28T20:51:13.494949+0000","last_unstale":"2023-11-28T20:51:13.500188+0000","last_undegraded":"2023-11-28T20:51:13.500188+0000","last_fullsized":"2023-11-28T20:51:13.500188+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:18:42.410562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506588+0000","last_change":"2023-11-28T20:51:13.503352+0000","last_active":"2023-11-28T20:51:13.506588+0000","last_peered":"2023-11-28T20:51:13.506588+0000","last_clean":"2023-11-28T20:51:13.506588+0000","last_became_active":"2023-11-28T20:51:13.503007+0000","last_became_peered":"2023-11-28T20:51:13.503007+0000","last_unstale":"2023-11-28T20:51:13.506588+0000","last_undegraded":"2023-11-28T20:51:13.506588+0000","last_fullsized":"2023-11-28T20:51:13.506588+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:13.967025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522708+0000","last_change":"2023-11-28T20:51:13.520195+0000","last_active":"2023-11-28T20:51:13.522708+0000","last_peered":"2023-11-28T20:51:13.522708+0000","last_clean":"2023-11-28T20:51:13.522708+0000","last_became_active":"2023-11-28T20:51:13.519610+0000","last_became_peered":"2023-11-28T20:51:13.519610+0000","last_unstale":"2023-11-28T20:51:13.522708+0000","last_undegraded":"2023-11-28T20:51:13.522708+0000","last_fullsized":"2023-11-28T20:51:13.522708+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:23:38.003752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.516833+0000","last_change":"2023-11-28T20:51:13.515728+0000","last_active":"2023-11-28T20:51:13.516833+0000","last_peered":"2023-11-28T20:51:13.516833+0000","last_clean":"2023-11-28T20:51:13.516833+0000","last_became_active":"2023-11-28T20:51:13.515386+0000","last_became_peered":"2023-11-28T20:51:13.515386+0000","last_unstale":"2023-11-28T20:51:13.516833+0000","last_undegraded":"2023-11-28T20:51:13.516833+0000","last_fullsized":"2023-11-28T20:51:13.516833+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:47.176949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.523975+0000","last_change":"2023-11-28T20:51:13.523111+0000","last_active":"2023-11-28T20:51:13.523975+0000","last_peered":"2023-11-28T20:51:13.523975+0000","last_clean":"2023-11-28T20:51:13.523975+0000","last_became_active":"2023-11-28T20:51:13.522747+0000","last_became_peered":"2023-11-28T20:51:13.522747+0000","last_unstale":"2023-11-28T20:51:13.523975+0000","last_undegraded":"2023-11-28T20:51:13.523975+0000","last_fullsized":"2023-11-28T20:51:13.523975+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:07.963984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.496121+0000","last_change":"2023-11-28T20:51:13.489157+0000","last_active":"2023-11-28T20:51:13.496121+0000","last_peered":"2023-11-28T20:51:13.496121+0000","last_clean":"2023-11-28T20:51:13.496121+0000","last_became_active":"2023-11-28T20:51:13.488825+0000","last_became_peered":"2023-11-28T20:51:13.488825+0000","last_unstale":"2023-11-28T20:51:13.496121+0000","last_undegraded":"2023-11-28T20:51:13.496121+0000","last_fullsized":"2023-11-28T20:51:13.496121+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:44:42.288287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.516805+0000","last_change":"2023-11-28T20:51:13.511525+0000","last_active":"2023-11-28T20:51:13.516805+0000","last_peered":"2023-11-28T20:51:13.516805+0000","last_clean":"2023-11-28T20:51:13.516805+0000","last_became_active":"2023-11-28T20:51:13.510254+0000","last_became_peered":"2023-11-28T20:51:13.510254+0000","last_unstale":"2023-11-28T20:51:13.516805+0000","last_undegraded":"2023-11-28T20:51:13.516805+0000","last_fullsized":"2023-11-28T20:51:13.516805+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:34:07.276346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.496210+0000","last_change":"2023-11-28T20:51:13.488397+0000","last_active":"2023-11-28T20:51:13.496210+0000","last_peered":"2023-11-28T20:51:13.496210+0000","last_clean":"2023-11-28T20:51:13.496210+0000","last_became_active":"2023-11-28T20:51:13.488107+0000","last_became_peered":"2023-11-28T20:51:13.488107+0000","last_unstale":"2023-11-28T20:51:13.496210+0000","last_undegraded":"2023-11-28T20:51:13.496210+0000","last_fullsized":"2023-11-28T20:51:13.496210+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:50:38.763001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.512124+0000","last_change":"2023-11-28T20:51:13.506491+0000","last_active":"2023-11-28T20:51:13.512124+0000","last_peered":"2023-11-28T20:51:13.512124+0000","last_clean":"2023-11-28T20:51:13.512124+0000","last_became_active":"2023-11-28T20:51:13.506133+0000","last_became_peered":"2023-11-28T20:51:13.506133+0000","last_unstale":"2023-11-28T20:51:13.512124+0000","last_undegraded":"2023-11-28T20:51:13.512124+0000","last_fullsized":"2023-11-28T20:51:13.512124+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:05:55.771224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.518611+0000","last_change":"2023-11-28T20:51:13.510261+0000","last_active":"2023-11-28T20:51:13.518611+0000","last_peered":"2023-11-28T20:51:13.518611+0000","last_clean":"2023-11-28T20:51:13.518611+0000","last_became_active":"2023-11-28T20:51:13.508621+0000","last_became_peered":"2023-11-28T20:51:13.508621+0000","last_unstale":"2023-11-28T20:51:13.518611+0000","last_undegraded":"2023-11-28T20:51:13.518611+0000","last_fullsized":"2023-11-28T20:51:13.518611+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:11:44.887484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.504933+0000","last_change":"2023-11-28T20:51:13.501559+0000","last_active":"2023-11-28T20:51:13.504933+0000","last_peered":"2023-11-28T20:51:13.504933+0000","last_clean":"2023-11-28T20:51:13.504933+0000","last_became_active":"2023-11-28T20:51:13.500638+0000","last_became_peered":"2023-11-28T20:51:13.500638+0000","last_unstale":"2023-11-28T20:51:13.504933+0000","last_undegraded":"2023-11-28T20:51:13.504933+0000","last_fullsized":"2023-11-28T20:51:13.504933+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:28.433626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.507360+0000","last_change":"2023-11-28T20:51:13.506396+0000","last_active":"2023-11-28T20:51:13.507360+0000","last_peered":"2023-11-28T20:51:13.507360+0000","last_clean":"2023-11-28T20:51:13.507360+0000","last_became_active":"2023-11-28T20:51:13.506115+0000","last_became_peered":"2023-11-28T20:51:13.506115+0000","last_unstale":"2023-11-28T20:51:13.507360+0000","last_undegraded":"2023-11-28T20:51:13.507360+0000","last_fullsized":"2023-11-28T20:51:13.507360+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:08.706931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.505278+0000","last_change":"2023-11-28T20:51:13.502500+0000","last_active":"2023-11-28T20:51:13.505278+0000","last_peered":"2023-11-28T20:51:13.505278+0000","last_clean":"2023-11-28T20:51:13.505278+0000","last_became_active":"2023-11-28T20:51:13.502174+0000","last_became_peered":"2023-11-28T20:51:13.502174+0000","last_unstale":"2023-11-28T20:51:13.505278+0000","last_undegraded":"2023-11-28T20:51:13.505278+0000","last_fullsized":"2023-11-28T20:51:13.505278+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:06:52.066200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522218+0000","last_change":"2023-11-28T20:51:13.501613+0000","last_active":"2023-11-28T20:51:13.522218+0000","last_peered":"2023-11-28T20:51:13.522218+0000","last_clean":"2023-11-28T20:51:13.522218+0000","last_became_active":"2023-11-28T20:51:13.501334+0000","last_became_peered":"2023-11-28T20:51:13.501334+0000","last_unstale":"2023-11-28T20:51:13.522218+0000","last_undegraded":"2023-11-28T20:51:13.522218+0000","last_fullsized":"2023-11-28T20:51:13.522218+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:32:15.927170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517250+0000","last_change":"2023-11-28T20:51:13.508308+0000","last_active":"2023-11-28T20:51:13.517250+0000","last_peered":"2023-11-28T20:51:13.517250+0000","last_clean":"2023-11-28T20:51:13.517250+0000","last_became_active":"2023-11-28T20:51:13.507861+0000","last_became_peered":"2023-11-28T20:51:13.507861+0000","last_unstale":"2023-11-28T20:51:13.517250+0000","last_undegraded":"2023-11-28T20:51:13.517250+0000","last_fullsized":"2023-11-28T20:51:13.517250+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:45:19.402469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.507924+0000","last_change":"2023-11-28T20:51:13.505705+0000","last_active":"2023-11-28T20:51:13.507924+0000","last_peered":"2023-11-28T20:51:13.507924+0000","last_clean":"2023-11-28T20:51:13.507924+0000","last_became_active":"2023-11-28T20:51:13.504825+0000","last_became_peered":"2023-11-28T20:51:13.504825+0000","last_unstale":"2023-11-28T20:51:13.507924+0000","last_undegraded":"2023-11-28T20:51:13.507924+0000","last_fullsized":"2023-11-28T20:51:13.507924+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:19:40.944865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514408+0000","last_change":"2023-11-28T20:51:13.512999+0000","last_active":"2023-11-28T20:51:13.514408+0000","last_peered":"2023-11-28T20:51:13.514408+0000","last_clean":"2023-11-28T20:51:13.514408+0000","last_became_active":"2023-11-28T20:51:13.512717+0000","last_became_peered":"2023-11-28T20:51:13.512717+0000","last_unstale":"2023-11-28T20:51:13.514408+0000","last_undegraded":"2023-11-28T20:51:13.514408+0000","last_fullsized":"2023-11-28T20:51:13.514408+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:51:09.007211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.513900+0000","last_change":"2023-11-28T20:51:13.508634+0000","last_active":"2023-11-28T20:51:13.513900+0000","last_peered":"2023-11-28T20:51:13.513900+0000","last_clean":"2023-11-28T20:51:13.513900+0000","last_became_active":"2023-11-28T20:51:13.508364+0000","last_became_peered":"2023-11-28T20:51:13.508364+0000","last_unstale":"2023-11-28T20:51:13.513900+0000","last_undegraded":"2023-11-28T20:51:13.513900+0000","last_fullsized":"2023-11-28T20:51:13.513900+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:18.509118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.515825+0000","last_change":"2023-11-28T20:51:13.515101+0000","last_active":"2023-11-28T20:51:13.515825+0000","last_peered":"2023-11-28T20:51:13.515825+0000","last_clean":"2023-11-28T20:51:13.515825+0000","last_became_active":"2023-11-28T20:51:13.512097+0000","last_became_peered":"2023-11-28T20:51:13.512097+0000","last_unstale":"2023-11-28T20:51:13.515825+0000","last_undegraded":"2023-11-28T20:51:13.515825+0000","last_fullsized":"2023-11-28T20:51:13.515825+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:26:13.953999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"101'4","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.521748+0000","last_change":"2023-11-28T20:51:12.484239+0000","last_active":"2023-11-28T20:51:13.521748+0000","last_peered":"2023-11-28T20:51:13.521748+0000","last_clean":"2023-11-28T20:51:13.521748+0000","last_became_active":"2023-11-28T20:51:12.483657+0000","last_became_peered":"2023-11-28T20:51:12.483657+0000","last_unstale":"2023-11-28T20:51:13.521748+0000","last_undegraded":"2023-11-28T20:51:13.521748+0000","last_fullsized":"2023-11-28T20:51:13.521748+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:01:58.181623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517047+0000","last_change":"2023-11-28T20:51:13.506785+0000","last_active":"2023-11-28T20:51:13.517047+0000","last_peered":"2023-11-28T20:51:13.517047+0000","last_clean":"2023-11-28T20:51:13.517047+0000","last_became_active":"2023-11-28T20:51:13.506384+0000","last_became_peered":"2023-11-28T20:51:13.506384+0000","last_unstale":"2023-11-28T20:51:13.517047+0000","last_undegraded":"2023-11-28T20:51:13.517047+0000","last_fullsized":"2023-11-28T20:51:13.517047+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:34:22.392589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.515680+0000","last_change":"2023-11-28T20:51:12.506096+0000","last_active":"2023-11-28T20:51:13.515680+0000","last_peered":"2023-11-28T20:51:13.515680+0000","last_clean":"2023-11-28T20:51:13.515680+0000","last_became_active":"2023-11-28T20:51:12.505812+0000","last_became_peered":"2023-11-28T20:51:12.505812+0000","last_unstale":"2023-11-28T20:51:13.515680+0000","last_undegraded":"2023-11-28T20:51:13.515680+0000","last_fullsized":"2023-11-28T20:51:13.515680+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:34:21.386513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514157+0000","last_change":"2023-11-28T20:51:13.508333+0000","last_active":"2023-11-28T20:51:13.514157+0000","last_peered":"2023-11-28T20:51:13.514157+0000","last_clean":"2023-11-28T20:51:13.514157+0000","last_became_active":"2023-11-28T20:51:13.508037+0000","last_became_peered":"2023-11-28T20:51:13.508037+0000","last_unstale":"2023-11-28T20:51:13.514157+0000","last_undegraded":"2023-11-28T20:51:13.514157+0000","last_fullsized":"2023-11-28T20:51:13.514157+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:51:37.137247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522224+0000","last_change":"2023-11-28T20:51:12.501461+0000","last_active":"2023-11-28T20:51:13.522224+0000","last_peered":"2023-11-28T20:51:13.522224+0000","last_clean":"2023-11-28T20:51:13.522224+0000","last_became_active":"2023-11-28T20:51:12.500691+0000","last_became_peered":"2023-11-28T20:51:12.500691+0000","last_unstale":"2023-11-28T20:51:13.522224+0000","last_undegraded":"2023-11-28T20:51:13.522224+0000","last_fullsized":"2023-11-28T20:51:13.522224+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:27.427550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.516827+0000","last_change":"2023-11-28T20:51:13.511702+0000","last_active":"2023-11-28T20:51:13.516827+0000","last_peered":"2023-11-28T20:51:13.516827+0000","last_clean":"2023-11-28T20:51:13.516827+0000","last_became_active":"2023-11-28T20:51:13.510597+0000","last_became_peered":"2023-11-28T20:51:13.510597+0000","last_unstale":"2023-11-28T20:51:13.516827+0000","last_undegraded":"2023-11-28T20:51:13.516827+0000","last_fullsized":"2023-11-28T20:51:13.516827+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:20.697835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520881+0000","last_change":"2023-11-28T20:51:12.502881+0000","last_active":"2023-11-28T20:51:13.520881+0000","last_peered":"2023-11-28T20:51:13.520881+0000","last_clean":"2023-11-28T20:51:13.520881+0000","last_became_active":"2023-11-28T20:51:12.502586+0000","last_became_peered":"2023-11-28T20:51:12.502586+0000","last_unstale":"2023-11-28T20:51:13.520881+0000","last_undegraded":"2023-11-28T20:51:13.520881+0000","last_fullsized":"2023-11-28T20:51:13.520881+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:13.254831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506075+0000","last_change":"2023-11-28T20:51:13.501884+0000","last_active":"2023-11-28T20:51:13.506075+0000","last_peered":"2023-11-28T20:51:13.506075+0000","last_clean":"2023-11-28T20:51:13.506075+0000","last_became_active":"2023-11-28T20:51:13.501591+0000","last_became_peered":"2023-11-28T20:51:13.501591+0000","last_unstale":"2023-11-28T20:51:13.506075+0000","last_undegraded":"2023-11-28T20:51:13.506075+0000","last_fullsized":"2023-11-28T20:51:13.506075+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:42:11.298316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.511829+0000","last_change":"2023-11-28T20:51:12.497543+0000","last_active":"2023-11-28T20:51:13.511829+0000","last_peered":"2023-11-28T20:51:13.511829+0000","last_clean":"2023-11-28T20:51:13.511829+0000","last_became_active":"2023-11-28T20:51:12.497239+0000","last_became_peered":"2023-11-28T20:51:12.497239+0000","last_unstale":"2023-11-28T20:51:13.511829+0000","last_undegraded":"2023-11-28T20:51:13.511829+0000","last_fullsized":"2023-11-28T20:51:13.511829+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:54.794319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.501291+0000","last_change":"2023-11-28T20:51:13.499305+0000","last_active":"2023-11-28T20:51:13.501291+0000","last_peered":"2023-11-28T20:51:13.501291+0000","last_clean":"2023-11-28T20:51:13.501291+0000","last_became_active":"2023-11-28T20:51:13.498732+0000","last_became_peered":"2023-11-28T20:51:13.498732+0000","last_unstale":"2023-11-28T20:51:13.501291+0000","last_undegraded":"2023-11-28T20:51:13.501291+0000","last_fullsized":"2023-11-28T20:51:13.501291+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:47:57.031660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.505768+0000","last_change":"2023-11-28T20:51:12.490486+0000","last_active":"2023-11-28T20:51:13.505768+0000","last_peered":"2023-11-28T20:51:13.505768+0000","last_clean":"2023-11-28T20:51:13.505768+0000","last_became_active":"2023-11-28T20:51:12.490169+0000","last_became_peered":"2023-11-28T20:51:12.490169+0000","last_unstale":"2023-11-28T20:51:13.505768+0000","last_undegraded":"2023-11-28T20:51:13.505768+0000","last_fullsized":"2023-11-28T20:51:13.505768+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:06:41.223231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.519805+0000","last_change":"2023-11-28T20:51:13.512002+0000","last_active":"2023-11-28T20:51:13.519805+0000","last_peered":"2023-11-28T20:51:13.519805+0000","last_clean":"2023-11-28T20:51:13.519805+0000","last_became_active":"2023-11-28T20:51:13.508935+0000","last_became_peered":"2023-11-28T20:51:13.508935+0000","last_unstale":"2023-11-28T20:51:13.519805+0000","last_undegraded":"2023-11-28T20:51:13.519805+0000","last_fullsized":"2023-11-28T20:51:13.519805+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:29.635174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.484633+0000","last_change":"2023-11-28T20:51:12.497787+0000","last_active":"2023-11-28T20:51:13.484633+0000","last_peered":"2023-11-28T20:51:13.484633+0000","last_clean":"2023-11-28T20:51:13.484633+0000","last_became_active":"2023-11-28T20:51:12.497167+0000","last_became_peered":"2023-11-28T20:51:12.497167+0000","last_unstale":"2023-11-28T20:51:13.484633+0000","last_undegraded":"2023-11-28T20:51:13.484633+0000","last_fullsized":"2023-11-28T20:51:13.484633+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:50.172662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.524282+0000","last_change":"2023-11-28T20:51:13.523581+0000","last_active":"2023-11-28T20:51:13.524282+0000","last_peered":"2023-11-28T20:51:13.524282+0000","last_clean":"2023-11-28T20:51:13.524282+0000","last_became_active":"2023-11-28T20:51:13.523158+0000","last_became_peered":"2023-11-28T20:51:13.523158+0000","last_unstale":"2023-11-28T20:51:13.524282+0000","last_undegraded":"2023-11-28T20:51:13.524282+0000","last_fullsized":"2023-11-28T20:51:13.524282+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:52:15.633702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.480521+0000","last_change":"2023-11-28T20:51:12.487193+0000","last_active":"2023-11-28T20:51:13.480521+0000","last_peered":"2023-11-28T20:51:13.480521+0000","last_clean":"2023-11-28T20:51:13.480521+0000","last_became_active":"2023-11-28T20:51:12.486541+0000","last_became_peered":"2023-11-28T20:51:12.486541+0000","last_unstale":"2023-11-28T20:51:13.480521+0000","last_undegraded":"2023-11-28T20:51:13.480521+0000","last_fullsized":"2023-11-28T20:51:13.480521+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:13.405678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.499824+0000","last_change":"2023-11-28T20:51:13.492702+0000","last_active":"2023-11-28T20:51:13.499824+0000","last_peered":"2023-11-28T20:51:13.499824+0000","last_clean":"2023-11-28T20:51:13.499824+0000","last_became_active":"2023-11-28T20:51:13.492449+0000","last_became_peered":"2023-11-28T20:51:13.492449+0000","last_unstale":"2023-11-28T20:51:13.499824+0000","last_undegraded":"2023-11-28T20:51:13.499824+0000","last_fullsized":"2023-11-28T20:51:13.499824+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:13.967025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.481343+0000","last_change":"2023-11-28T20:51:12.506534+0000","last_active":"2023-11-28T20:51:13.481343+0000","last_peered":"2023-11-28T20:51:13.481343+0000","last_clean":"2023-11-28T20:51:13.481343+0000","last_became_active":"2023-11-28T20:51:12.505899+0000","last_became_peered":"2023-11-28T20:51:12.505899+0000","last_unstale":"2023-11-28T20:51:13.481343+0000","last_undegraded":"2023-11-28T20:51:13.481343+0000","last_fullsized":"2023-11-28T20:51:13.481343+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:29.409766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520945+0000","last_change":"2023-11-28T20:51:13.519416+0000","last_active":"2023-11-28T20:51:13.520945+0000","last_peered":"2023-11-28T20:51:13.520945+0000","last_clean":"2023-11-28T20:51:13.520945+0000","last_became_active":"2023-11-28T20:51:13.518667+0000","last_became_peered":"2023-11-28T20:51:13.518667+0000","last_unstale":"2023-11-28T20:51:13.520945+0000","last_undegraded":"2023-11-28T20:51:13.520945+0000","last_fullsized":"2023-11-28T20:51:13.520945+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:18:42.410562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.492991+0000","last_change":"2023-11-28T20:51:12.497171+0000","last_active":"2023-11-28T20:51:13.492991+0000","last_peered":"2023-11-28T20:51:13.492991+0000","last_clean":"2023-11-28T20:51:13.492991+0000","last_became_active":"2023-11-28T20:51:12.496868+0000","last_became_peered":"2023-11-28T20:51:12.496868+0000","last_unstale":"2023-11-28T20:51:13.492991+0000","last_undegraded":"2023-11-28T20:51:13.492991+0000","last_fullsized":"2023-11-28T20:51:13.492991+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:06:51.060124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.495825+0000","last_change":"2023-11-28T20:51:13.493179+0000","last_active":"2023-11-28T20:51:13.495825+0000","last_peered":"2023-11-28T20:51:13.495825+0000","last_clean":"2023-11-28T20:51:13.495825+0000","last_became_active":"2023-11-28T20:51:13.492909+0000","last_became_peered":"2023-11-28T20:51:13.492909+0000","last_unstale":"2023-11-28T20:51:13.495825+0000","last_undegraded":"2023-11-28T20:51:13.495825+0000","last_fullsized":"2023-11-28T20:51:13.495825+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:28.433626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"101'3","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520714+0000","last_change":"2023-11-28T20:51:12.489353+0000","last_active":"2023-11-28T20:51:13.520714+0000","last_peered":"2023-11-28T20:51:13.520714+0000","last_clean":"2023-11-28T20:51:13.520714+0000","last_became_active":"2023-11-28T20:51:12.488259+0000","last_became_peered":"2023-11-28T20:51:12.488259+0000","last_unstale":"2023-11-28T20:51:13.520714+0000","last_undegraded":"2023-11-28T20:51:13.520714+0000","last_fullsized":"2023-11-28T20:51:13.520714+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:43:43.464568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.503954+0000","last_change":"2023-11-28T20:51:13.503120+0000","last_active":"2023-11-28T20:51:13.503954+0000","last_peered":"2023-11-28T20:51:13.503954+0000","last_clean":"2023-11-28T20:51:13.503954+0000","last_became_active":"2023-11-28T20:51:13.502844+0000","last_became_peered":"2023-11-28T20:51:13.502844+0000","last_unstale":"2023-11-28T20:51:13.503954+0000","last_undegraded":"2023-11-28T20:51:13.503954+0000","last_fullsized":"2023-11-28T20:51:13.503954+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:15:45.314242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"101'6","reported_seq":17,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.525615+0000","last_change":"2023-11-28T20:51:12.501097+0000","last_active":"2023-11-28T20:51:13.525615+0000","last_peered":"2023-11-28T20:51:13.525615+0000","last_clean":"2023-11-28T20:51:13.525615+0000","last_became_active":"2023-11-28T20:51:12.499789+0000","last_became_peered":"2023-11-28T20:51:12.499789+0000","last_unstale":"2023-11-28T20:51:13.525615+0000","last_undegraded":"2023-11-28T20:51:13.525615+0000","last_fullsized":"2023-11-28T20:51:13.525615+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:09:57.520400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506892+0000","last_change":"2023-11-28T20:51:13.504409+0000","last_active":"2023-11-28T20:51:13.506892+0000","last_peered":"2023-11-28T20:51:13.506892+0000","last_clean":"2023-11-28T20:51:13.506892+0000","last_became_active":"2023-11-28T20:51:13.504135+0000","last_became_peered":"2023-11-28T20:51:13.504135+0000","last_unstale":"2023-11-28T20:51:13.506892+0000","last_undegraded":"2023-11-28T20:51:13.506892+0000","last_fullsized":"2023-11-28T20:51:13.506892+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:13.967025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.512905+0000","last_change":"2023-11-28T20:51:12.495185+0000","last_active":"2023-11-28T20:51:13.512905+0000","last_peered":"2023-11-28T20:51:13.512905+0000","last_clean":"2023-11-28T20:51:13.512905+0000","last_became_active":"2023-11-28T20:51:12.494721+0000","last_became_peered":"2023-11-28T20:51:12.494721+0000","last_unstale":"2023-11-28T20:51:13.512905+0000","last_undegraded":"2023-11-28T20:51:13.512905+0000","last_fullsized":"2023-11-28T20:51:13.512905+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:59:32.995662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.507853+0000","last_change":"2023-11-28T20:51:13.500041+0000","last_active":"2023-11-28T20:51:13.507853+0000","last_peered":"2023-11-28T20:51:13.507853+0000","last_clean":"2023-11-28T20:51:13.507853+0000","last_became_active":"2023-11-28T20:51:13.499771+0000","last_became_peered":"2023-11-28T20:51:13.499771+0000","last_unstale":"2023-11-28T20:51:13.507853+0000","last_undegraded":"2023-11-28T20:51:13.507853+0000","last_fullsized":"2023-11-28T20:51:13.507853+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:05.680077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.480062+0000","last_change":"2023-11-28T20:51:12.502262+0000","last_active":"2023-11-28T20:51:13.480062+0000","last_peered":"2023-11-28T20:51:13.480062+0000","last_clean":"2023-11-28T20:51:13.480062+0000","last_became_active":"2023-11-28T20:51:12.501984+0000","last_became_peered":"2023-11-28T20:51:12.501984+0000","last_unstale":"2023-11-28T20:51:13.480062+0000","last_undegraded":"2023-11-28T20:51:13.480062+0000","last_fullsized":"2023-11-28T20:51:13.480062+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:42:31.025256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517275+0000","last_change":"2023-11-28T20:51:13.511951+0000","last_active":"2023-11-28T20:51:13.517275+0000","last_peered":"2023-11-28T20:51:13.517275+0000","last_clean":"2023-11-28T20:51:13.517275+0000","last_became_active":"2023-11-28T20:51:13.511635+0000","last_became_peered":"2023-11-28T20:51:13.511635+0000","last_unstale":"2023-11-28T20:51:13.517275+0000","last_undegraded":"2023-11-28T20:51:13.517275+0000","last_fullsized":"2023-11-28T20:51:13.517275+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:29.121136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.481992+0000","last_change":"2023-11-28T20:51:12.483682+0000","last_active":"2023-11-28T20:51:13.481992+0000","last_peered":"2023-11-28T20:51:13.481992+0000","last_clean":"2023-11-28T20:51:13.481992+0000","last_became_active":"2023-11-28T20:51:12.483335+0000","last_became_peered":"2023-11-28T20:51:12.483335+0000","last_unstale":"2023-11-28T20:51:13.481992+0000","last_undegraded":"2023-11-28T20:51:13.481992+0000","last_fullsized":"2023-11-28T20:51:13.481992+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:33:03.745861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520876+0000","last_change":"2023-11-28T20:51:13.514841+0000","last_active":"2023-11-28T20:51:13.520876+0000","last_peered":"2023-11-28T20:51:13.520876+0000","last_clean":"2023-11-28T20:51:13.520876+0000","last_became_active":"2023-11-28T20:51:13.514502+0000","last_became_peered":"2023-11-28T20:51:13.514502+0000","last_unstale":"2023-11-28T20:51:13.520876+0000","last_undegraded":"2023-11-28T20:51:13.520876+0000","last_fullsized":"2023-11-28T20:51:13.520876+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:20.697835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"101'8","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.524583+0000","last_change":"2023-11-28T20:51:12.493381+0000","last_active":"2023-11-28T20:51:13.524583+0000","last_peered":"2023-11-28T20:51:13.524583+0000","last_clean":"2023-11-28T20:51:13.524583+0000","last_became_active":"2023-11-28T20:51:12.493106+0000","last_became_peered":"2023-11-28T20:51:12.493106+0000","last_unstale":"2023-11-28T20:51:13.524583+0000","last_undegraded":"2023-11-28T20:51:13.524583+0000","last_fullsized":"2023-11-28T20:51:13.524583+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:14:07.789156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.497591+0000","last_change":"2023-11-28T20:51:13.492090+0000","last_active":"2023-11-28T20:51:13.497591+0000","last_peered":"2023-11-28T20:51:13.497591+0000","last_clean":"2023-11-28T20:51:13.497591+0000","last_became_active":"2023-11-28T20:51:13.488443+0000","last_became_peered":"2023-11-28T20:51:13.488443+0000","last_unstale":"2023-11-28T20:51:13.497591+0000","last_undegraded":"2023-11-28T20:51:13.497591+0000","last_fullsized":"2023-11-28T20:51:13.497591+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:42:11.298316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514828+0000","last_change":"2023-11-28T20:51:12.503430+0000","last_active":"2023-11-28T20:51:13.514828+0000","last_peered":"2023-11-28T20:51:13.514828+0000","last_clean":"2023-11-28T20:51:13.514828+0000","last_became_active":"2023-11-28T20:51:12.503187+0000","last_became_peered":"2023-11-28T20:51:12.503187+0000","last_unstale":"2023-11-28T20:51:13.514828+0000","last_undegraded":"2023-11-28T20:51:13.514828+0000","last_fullsized":"2023-11-28T20:51:13.514828+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:50.053211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522293+0000","last_change":"2023-11-28T20:51:13.518392+0000","last_active":"2023-11-28T20:51:13.522293+0000","last_peered":"2023-11-28T20:51:13.522293+0000","last_clean":"2023-11-28T20:51:13.522293+0000","last_became_active":"2023-11-28T20:51:13.518109+0000","last_became_peered":"2023-11-28T20:51:13.518109+0000","last_unstale":"2023-11-28T20:51:13.522293+0000","last_undegraded":"2023-11-28T20:51:13.522293+0000","last_fullsized":"2023-11-28T20:51:13.522293+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:57:18.348611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.515732+0000","last_change":"2023-11-28T20:51:12.493220+0000","last_active":"2023-11-28T20:51:13.515732+0000","last_peered":"2023-11-28T20:51:13.515732+0000","last_clean":"2023-11-28T20:51:13.515732+0000","last_became_active":"2023-11-28T20:51:12.492626+0000","last_became_peered":"2023-11-28T20:51:12.492626+0000","last_unstale":"2023-11-28T20:51:13.515732+0000","last_undegraded":"2023-11-28T20:51:13.515732+0000","last_fullsized":"2023-11-28T20:51:13.515732+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:50.264502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522505+0000","last_change":"2023-11-28T20:51:13.517975+0000","last_active":"2023-11-28T20:51:13.522505+0000","last_peered":"2023-11-28T20:51:13.522505+0000","last_clean":"2023-11-28T20:51:13.522505+0000","last_became_active":"2023-11-28T20:51:13.513725+0000","last_became_peered":"2023-11-28T20:51:13.513725+0000","last_unstale":"2023-11-28T20:51:13.522505+0000","last_undegraded":"2023-11-28T20:51:13.522505+0000","last_fullsized":"2023-11-28T20:51:13.522505+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:41.759604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"101'1","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.522913+0000","last_change":"2023-11-28T20:51:12.486812+0000","last_active":"2023-11-28T20:51:13.522913+0000","last_peered":"2023-11-28T20:51:13.522913+0000","last_clean":"2023-11-28T20:51:13.522913+0000","last_became_active":"2023-11-28T20:51:12.486513+0000","last_became_peered":"2023-11-28T20:51:12.486513+0000","last_unstale":"2023-11-28T20:51:13.522913+0000","last_undegraded":"2023-11-28T20:51:13.522913+0000","last_fullsized":"2023-11-28T20:51:13.522913+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:02:57.418923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.520269+0000","last_change":"2023-11-28T20:51:13.518992+0000","last_active":"2023-11-28T20:51:13.520269+0000","last_peered":"2023-11-28T20:51:13.520269+0000","last_clean":"2023-11-28T20:51:13.520269+0000","last_became_active":"2023-11-28T20:51:13.515701+0000","last_became_peered":"2023-11-28T20:51:13.515701+0000","last_unstale":"2023-11-28T20:51:13.520269+0000","last_undegraded":"2023-11-28T20:51:13.520269+0000","last_fullsized":"2023-11-28T20:51:13.520269+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:12:11.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.476301+0000","last_change":"2023-11-28T20:51:12.492296+0000","last_active":"2023-11-28T20:51:13.476301+0000","last_peered":"2023-11-28T20:51:13.476301+0000","last_clean":"2023-11-28T20:51:13.476301+0000","last_became_active":"2023-11-28T20:51:12.492018+0000","last_became_peered":"2023-11-28T20:51:12.492018+0000","last_unstale":"2023-11-28T20:51:13.476301+0000","last_undegraded":"2023-11-28T20:51:13.476301+0000","last_fullsized":"2023-11-28T20:51:13.476301+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:23:16.583514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.518096+0000","last_change":"2023-11-28T20:51:13.511100+0000","last_active":"2023-11-28T20:51:13.518096+0000","last_peered":"2023-11-28T20:51:13.518096+0000","last_clean":"2023-11-28T20:51:13.518096+0000","last_became_active":"2023-11-28T20:51:13.505852+0000","last_became_peered":"2023-11-28T20:51:13.505852+0000","last_unstale":"2023-11-28T20:51:13.518096+0000","last_undegraded":"2023-11-28T20:51:13.518096+0000","last_fullsized":"2023-11-28T20:51:13.518096+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:30.415842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.513994+0000","last_change":"2023-11-28T20:51:12.488412+0000","last_active":"2023-11-28T20:51:13.513994+0000","last_peered":"2023-11-28T20:51:13.513994+0000","last_clean":"2023-11-28T20:51:13.513994+0000","last_became_active":"2023-11-28T20:51:12.488158+0000","last_became_peered":"2023-11-28T20:51:12.488158+0000","last_unstale":"2023-11-28T20:51:13.513994+0000","last_undegraded":"2023-11-28T20:51:13.513994+0000","last_fullsized":"2023-11-28T20:51:13.513994+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:11:43.881408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.521107+0000","last_change":"2023-11-28T20:51:13.518710+0000","last_active":"2023-11-28T20:51:13.521107+0000","last_peered":"2023-11-28T20:51:13.521107+0000","last_clean":"2023-11-28T20:51:13.521107+0000","last_became_active":"2023-11-28T20:51:13.518389+0000","last_became_peered":"2023-11-28T20:51:13.518389+0000","last_unstale":"2023-11-28T20:51:13.521107+0000","last_undegraded":"2023-11-28T20:51:13.521107+0000","last_fullsized":"2023-11-28T20:51:13.521107+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:51.059287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.491281+0000","last_change":"2023-11-28T20:51:12.494355+0000","last_active":"2023-11-28T20:51:13.491281+0000","last_peered":"2023-11-28T20:51:13.491281+0000","last_clean":"2023-11-28T20:51:13.491281+0000","last_became_active":"2023-11-28T20:51:12.493788+0000","last_became_peered":"2023-11-28T20:51:12.493788+0000","last_unstale":"2023-11-28T20:51:13.491281+0000","last_undegraded":"2023-11-28T20:51:13.491281+0000","last_fullsized":"2023-11-28T20:51:13.491281+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:45:18.396393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.513758+0000","last_change":"2023-11-28T20:51:13.507310+0000","last_active":"2023-11-28T20:51:13.513758+0000","last_peered":"2023-11-28T20:51:13.513758+0000","last_clean":"2023-11-28T20:51:13.513758+0000","last_became_active":"2023-11-28T20:51:13.506966+0000","last_became_peered":"2023-11-28T20:51:13.506966+0000","last_unstale":"2023-11-28T20:51:13.513758+0000","last_undegraded":"2023-11-28T20:51:13.513758+0000","last_fullsized":"2023-11-28T20:51:13.513758+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:51:37.137247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.497984+0000","last_change":"2023-11-28T20:51:12.490523+0000","last_active":"2023-11-28T20:51:13.497984+0000","last_peered":"2023-11-28T20:51:13.497984+0000","last_clean":"2023-11-28T20:51:13.497984+0000","last_became_active":"2023-11-28T20:51:12.490313+0000","last_became_peered":"2023-11-28T20:51:12.490313+0000","last_unstale":"2023-11-28T20:51:13.497984+0000","last_undegraded":"2023-11-28T20:51:13.497984+0000","last_fullsized":"2023-11-28T20:51:13.497984+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:49:10.910467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.518977+0000","last_change":"2023-11-28T20:51:13.512380+0000","last_active":"2023-11-28T20:51:13.518977+0000","last_peered":"2023-11-28T20:51:13.518977+0000","last_clean":"2023-11-28T20:51:13.518977+0000","last_became_active":"2023-11-28T20:51:13.512113+0000","last_became_peered":"2023-11-28T20:51:13.512113+0000","last_unstale":"2023-11-28T20:51:13.518977+0000","last_undegraded":"2023-11-28T20:51:13.518977+0000","last_fullsized":"2023-11-28T20:51:13.518977+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:52:15.633702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"101'2","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.509276+0000","last_change":"2023-11-28T20:51:12.496968+0000","last_active":"2023-11-28T20:51:13.509276+0000","last_peered":"2023-11-28T20:51:13.509276+0000","last_clean":"2023-11-28T20:51:13.509276+0000","last_became_active":"2023-11-28T20:51:12.496632+0000","last_became_peered":"2023-11-28T20:51:12.496632+0000","last_unstale":"2023-11-28T20:51:13.509276+0000","last_undegraded":"2023-11-28T20:51:13.509276+0000","last_fullsized":"2023-11-28T20:51:13.509276+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:42:31.025256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.513759+0000","last_change":"2023-11-28T20:51:13.507737+0000","last_active":"2023-11-28T20:51:13.513759+0000","last_peered":"2023-11-28T20:51:13.513759+0000","last_clean":"2023-11-28T20:51:13.513759+0000","last_became_active":"2023-11-28T20:51:13.507404+0000","last_became_peered":"2023-11-28T20:51:13.507404+0000","last_unstale":"2023-11-28T20:51:13.513759+0000","last_undegraded":"2023-11-28T20:51:13.513759+0000","last_fullsized":"2023-11-28T20:51:13.513759+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:29.635174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.486525+0000","last_change":"2023-11-28T20:51:12.492307+0000","last_active":"2023-11-28T20:51:13.486525+0000","last_peered":"2023-11-28T20:51:13.486525+0000","last_clean":"2023-11-28T20:51:13.486525+0000","last_became_active":"2023-11-28T20:51:12.491941+0000","last_became_peered":"2023-11-28T20:51:12.491941+0000","last_unstale":"2023-11-28T20:51:13.486525+0000","last_undegraded":"2023-11-28T20:51:13.486525+0000","last_fullsized":"2023-11-28T20:51:13.486525+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:12:10.438657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.514137+0000","last_change":"2023-11-28T20:51:13.507726+0000","last_active":"2023-11-28T20:51:13.514137+0000","last_peered":"2023-11-28T20:51:13.514137+0000","last_clean":"2023-11-28T20:51:13.514137+0000","last_became_active":"2023-11-28T20:51:13.506946+0000","last_became_peered":"2023-11-28T20:51:13.506946+0000","last_unstale":"2023-11-28T20:51:13.514137+0000","last_undegraded":"2023-11-28T20:51:13.514137+0000","last_fullsized":"2023-11-28T20:51:13.514137+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:05.680077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.483498+0000","last_change":"2023-11-28T20:51:12.504344+0000","last_active":"2023-11-28T20:51:13.483498+0000","last_peered":"2023-11-28T20:51:13.483498+0000","last_clean":"2023-11-28T20:51:13.483498+0000","last_became_active":"2023-11-28T20:51:12.503986+0000","last_became_peered":"2023-11-28T20:51:12.503986+0000","last_unstale":"2023-11-28T20:51:13.483498+0000","last_undegraded":"2023-11-28T20:51:13.483498+0000","last_fullsized":"2023-11-28T20:51:13.483498+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:42:10.292240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.506894+0000","last_change":"2023-11-28T20:51:13.505478+0000","last_active":"2023-11-28T20:51:13.506894+0000","last_peered":"2023-11-28T20:51:13.506894+0000","last_clean":"2023-11-28T20:51:13.506894+0000","last_became_active":"2023-11-28T20:51:13.505198+0000","last_became_peered":"2023-11-28T20:51:13.505198+0000","last_unstale":"2023-11-28T20:51:13.506894+0000","last_undegraded":"2023-11-28T20:51:13.506894+0000","last_fullsized":"2023-11-28T20:51:13.506894+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:07.963984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.483080+0000","last_change":"2023-11-28T20:51:12.506141+0000","last_active":"2023-11-28T20:51:13.483080+0000","last_peered":"2023-11-28T20:51:13.483080+0000","last_clean":"2023-11-28T20:51:13.483080+0000","last_became_active":"2023-11-28T20:51:12.505844+0000","last_became_peered":"2023-11-28T20:51:12.505844+0000","last_unstale":"2023-11-28T20:51:13.483080+0000","last_undegraded":"2023-11-28T20:51:13.483080+0000","last_fullsized":"2023-11-28T20:51:13.483080+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:50.172662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.517105+0000","last_change":"2023-11-28T20:51:13.510215+0000","last_active":"2023-11-28T20:51:13.517105+0000","last_peered":"2023-11-28T20:51:13.517105+0000","last_clean":"2023-11-28T20:51:13.517105+0000","last_became_active":"2023-11-28T20:51:13.506598+0000","last_became_peered":"2023-11-28T20:51:13.506598+0000","last_unstale":"2023-11-28T20:51:13.517105+0000","last_undegraded":"2023-11-28T20:51:13.517105+0000","last_fullsized":"2023-11-28T20:51:13.517105+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:07:09.331469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.487937+0000","last_change":"2023-11-28T20:51:12.501734+0000","last_active":"2023-11-28T20:51:13.487937+0000","last_peered":"2023-11-28T20:51:13.487937+0000","last_clean":"2023-11-28T20:51:13.487937+0000","last_became_active":"2023-11-28T20:51:12.501323+0000","last_became_peered":"2023-11-28T20:51:12.501323+0000","last_unstale":"2023-11-28T20:51:13.487937+0000","last_undegraded":"2023-11-28T20:51:13.487937+0000","last_fullsized":"2023-11-28T20:51:13.487937+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:22:37.776214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.523680+0000","last_change":"2023-11-28T20:51:13.521710+0000","last_active":"2023-11-28T20:51:13.523680+0000","last_peered":"2023-11-28T20:51:13.523680+0000","last_clean":"2023-11-28T20:51:13.523680+0000","last_became_active":"2023-11-28T20:51:13.521407+0000","last_became_peered":"2023-11-28T20:51:13.521407+0000","last_unstale":"2023-11-28T20:51:13.523680+0000","last_undegraded":"2023-11-28T20:51:13.523680+0000","last_fullsized":"2023-11-28T20:51:13.523680+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:24.987055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.479950+0000","last_change":"2023-11-28T20:51:12.483895+0000","last_active":"2023-11-28T20:51:13.479950+0000","last_peered":"2023-11-28T20:51:13.479950+0000","last_clean":"2023-11-28T20:51:13.479950+0000","last_became_active":"2023-11-28T20:51:12.483535+0000","last_became_peered":"2023-11-28T20:51:12.483535+0000","last_unstale":"2023-11-28T20:51:13.479950+0000","last_undegraded":"2023-11-28T20:51:13.479950+0000","last_fullsized":"2023-11-28T20:51:13.479950+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:56:57.373934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.499940+0000","last_change":"2023-11-28T20:51:13.496809+0000","last_active":"2023-11-28T20:51:13.499940+0000","last_peered":"2023-11-28T20:51:13.499940+0000","last_clean":"2023-11-28T20:51:13.499940+0000","last_became_active":"2023-11-28T20:51:13.496494+0000","last_became_peered":"2023-11-28T20:51:13.496494+0000","last_unstale":"2023-11-28T20:51:13.499940+0000","last_undegraded":"2023-11-28T20:51:13.499940+0000","last_fullsized":"2023-11-28T20:51:13.499940+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:21.262203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.500936+0000","last_change":"2023-11-28T20:51:12.490611+0000","last_active":"2023-11-28T20:51:13.500936+0000","last_peered":"2023-11-28T20:51:13.500936+0000","last_clean":"2023-11-28T20:51:13.500936+0000","last_became_active":"2023-11-28T20:51:12.490179+0000","last_became_peered":"2023-11-28T20:51:12.490179+0000","last_unstale":"2023-11-28T20:51:13.500936+0000","last_undegraded":"2023-11-28T20:51:13.500936+0000","last_fullsized":"2023-11-28T20:51:13.500936+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:34:06.270270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.493037+0000","last_change":"2023-11-28T20:51:13.491602+0000","last_active":"2023-11-28T20:51:13.493037+0000","last_peered":"2023-11-28T20:51:13.493037+0000","last_clean":"2023-11-28T20:51:13.493037+0000","last_became_active":"2023-11-28T20:51:13.491254+0000","last_became_peered":"2023-11-28T20:51:13.491254+0000","last_unstale":"2023-11-28T20:51:13.493037+0000","last_undegraded":"2023-11-28T20:51:13.493037+0000","last_fullsized":"2023-11-28T20:51:13.493037+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:26.633298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.473928+0000","last_change":"2023-11-28T20:51:12.494925+0000","last_active":"2023-11-28T20:51:13.473928+0000","last_peered":"2023-11-28T20:51:13.473928+0000","last_clean":"2023-11-28T20:51:13.473928+0000","last_became_active":"2023-11-28T20:51:12.494324+0000","last_became_peered":"2023-11-28T20:51:12.494324+0000","last_unstale":"2023-11-28T20:51:13.473928+0000","last_undegraded":"2023-11-28T20:51:13.473928+0000","last_fullsized":"2023-11-28T20:51:13.473928+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:32.738648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.498187+0000","last_change":"2023-11-28T20:51:13.494390+0000","last_active":"2023-11-28T20:51:13.498187+0000","last_peered":"2023-11-28T20:51:13.498187+0000","last_clean":"2023-11-28T20:51:13.498187+0000","last_became_active":"2023-11-28T20:51:13.491441+0000","last_became_peered":"2023-11-28T20:51:13.491441+0000","last_unstale":"2023-11-28T20:51:13.498187+0000","last_undegraded":"2023-11-28T20:51:13.498187+0000","last_fullsized":"2023-11-28T20:51:13.498187+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:12.448075+0000","last_clean_scrub_stamp":"2023-11-28T20:51:12.448075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:47:57.031660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.492006+0000","last_change":"2023-11-28T20:51:12.495437+0000","last_active":"2023-11-28T20:51:13.492006+0000","last_peered":"2023-11-28T20:51:13.492006+0000","last_clean":"2023-11-28T20:51:13.492006+0000","last_became_active":"2023-11-28T20:51:12.494371+0000","last_became_peered":"2023-11-28T20:51:12.494371+0000","last_unstale":"2023-11-28T20:51:13.492006+0000","last_undegraded":"2023-11-28T20:51:13.492006+0000","last_fullsized":"2023-11-28T20:51:13.492006+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:11.441999+0000","last_clean_scrub_stamp":"2023-11-28T20:51:11.441999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:14:07.789156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"101'2597","reported_seq":2875,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-11-28T20:51:13.639384+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:51:13.639384+0000","last_peered":"2023-11-28T20:51:13.639384+0000","last_clean":"2023-11-28T20:51:13.639384+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:51:13.639384+0000","last_undegraded":"2023-11-28T20:51:13.639384+0000","last_fullsized":"2023-11-28T20:51:13.639384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":2597,"log_dups_size":0,"ondisk_log_size":2597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":570,"num_read_kb":1050,"num_write":246,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":570,"num_read_kb":1050,"num_write":246,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2597,"ondisk_log_size":2597,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575056,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55184,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316656,"statfs":{"total":96636764160,"available":96580255744,"internally_reserved":0,"allocated":1830912,"data_stored":1211202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1919999999999999,"15min":1.194},"min":{"1min":0.79700000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.722,"5min":1.722,"15min":2.23},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.153,"15min":1.1799999999999999},"min":{"1min":0.84299999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5169999999999999,"5min":1.681,"15min":1.726},"last":0.84899999999999998}]},{"osd":1,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1830000000000001,"15min":1.1850000000000001},"min":{"1min":0.83099999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.579,"5min":1.744,"15min":1.744},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.92100000000000004,"5min":0.624,"15min":0.624},"max":{"1min":1.6679999999999999,"5min":1.835,"15min":2.3599999999999999},"last":0.747}]},{"osd":2,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.2949999999999999,"15min":1.3029999999999999},"min":{"1min":0.97199999999999998,"5min":0.80300000000000005,"15min":0.58799999999999997},"max":{"1min":1.655,"5min":1.8300000000000001,"15min":1.9359999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.292,"15min":1.2869999999999999},"min":{"1min":0.96899999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.675,"5min":1.847,"15min":2.137},"last":1.105}]},{"osd":3,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2649999999999999,"15min":1.276},"min":{"1min":0.73199999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.577,"5min":1.7729999999999999,"15min":1.8460000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.403,"15min":1.4059999999999999},"min":{"1min":1.1579999999999999,"5min":0.88300000000000001,"15min":0.68300000000000005},"max":{"1min":1.956,"5min":1.956,"15min":2.4220000000000002},"last":1.5920000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.2629999999999999,"15min":1.264},"min":{"1min":0.83999999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7649999999999999,"5min":1.7989999999999999,"15min":1.919},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3200000000000001,"15min":1.329},"min":{"1min":0.93000000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.794,"5min":1.863,"15min":2.0049999999999999},"last":1.294}]},{"osd":5,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":0.97199999999999998,"5min":0.86499999999999999,"15min":0.67900000000000005},"max":{"1min":1.891,"5min":1.9219999999999999,"15min":2.5449999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.47,"15min":1.46},"min":{"1min":1.1160000000000001,"5min":0.73499999999999999,"15min":0.72199999999999998},"max":{"1min":1.931,"5min":1.9550000000000001,"15min":2.1000000000000001},"last":1.24}]},{"osd":6,"last update":"Tue Nov 28 20:50:12 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5649999999999999,"15min":1.548},"min":{"1min":1.1779999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.2389999999999999},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4830000000000001,"15min":1.488},"min":{"1min":1.056,"5min":0.94099999999999995,"15min":0.90800000000000003},"max":{"1min":1.9139999999999999,"5min":1.9870000000000001,"15min":2.3079999999999998},"last":1.5169999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575056,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1040000000000001,"15min":1.1140000000000001},"min":{"1min":0.71799999999999997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.125,"15min":1.1319999999999999},"min":{"1min":0.73099999999999998,"5min":0.63500000000000001,"15min":0.627},"max":{"1min":1.452,"5min":1.7549999999999999,"15min":2.0739999999999998},"last":0.80400000000000005}]},{"osd":1,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2130000000000001,"15min":1.2569999999999999},"min":{"1min":0.84399999999999997,"5min":0.67200000000000004,"15min":0.52200000000000002},"max":{"1min":1.728,"5min":1.728,"15min":1.8069999999999999},"last":1.341},{"interface":"front","average":{"1min":1.409,"5min":1.359,"15min":1.3700000000000001},"min":{"1min":0.97199999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6850000000000001,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":1.431}]},{"osd":2,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.2709999999999999,"15min":1.2669999999999999},"min":{"1min":0.89400000000000002,"5min":0.71099999999999997,"15min":0.69799999999999995},"max":{"1min":1.631,"5min":1.6879999999999999,"15min":1.71},"last":1.286},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.355,"15min":1.361},"min":{"1min":0.84999999999999998,"5min":0.73199999999999998,"15min":0.60299999999999998},"max":{"1min":1.6439999999999999,"5min":1.851,"15min":1.851},"last":1.113}]},{"osd":3,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.2729999999999999,"15min":1.3},"min":{"1min":0.96799999999999997,"5min":0.70099999999999996,"15min":0.68799999999999994},"max":{"1min":1.6200000000000001,"5min":1.821,"15min":2.222},"last":1.23},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2549999999999999,"15min":1.2669999999999999},"min":{"1min":0.95399999999999996,"5min":0.83899999999999997,"15min":0.751},"max":{"1min":1.635,"5min":1.6599999999999999,"15min":1.7949999999999999},"last":1.171}]},{"osd":4,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2509999999999999,"15min":1.2729999999999999},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.704,"5min":1.849,"15min":2.7290000000000001},"last":1.647},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.286,"15min":1.329},"min":{"1min":0.84099999999999997,"5min":0.80800000000000005,"15min":0.76400000000000001},"max":{"1min":1.752,"5min":2.2010000000000001,"15min":3.2519999999999998},"last":1.597}]},{"osd":5,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3100000000000001,"15min":1.325},"min":{"1min":0.96999999999999997,"5min":0.754,"15min":0.62},"max":{"1min":1.714,"5min":1.714,"15min":2.0539999999999998},"last":1.383},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5129999999999999,"15min":1.5469999999999999},"min":{"1min":1.2709999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.804,"5min":1.9179999999999999,"15min":3.0310000000000001},"last":1.5369999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.45,"15min":1.494},"min":{"1min":1.014,"5min":0.97799999999999998,"15min":0.86499999999999999},"max":{"1min":1.8080000000000001,"5min":2.1120000000000001,"15min":3.379},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.522,"5min":1.4490000000000001,"15min":1.462},"min":{"1min":1.0669999999999999,"5min":0.93300000000000005,"15min":0.84599999999999997},"max":{"1min":1.7669999999999999,"5min":1.8859999999999999,"15min":2.0739999999999998},"last":1.488}]}]},{"osd":5,"up_from":13,"seq":55834575056,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50428,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321412,"statfs":{"total":96636764160,"available":96585125888,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.159,"15min":1.175},"min":{"1min":0.72199999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.702,"5min":1.702,"15min":1.768},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1160000000000001,"15min":1.1879999999999999},"min":{"1min":0.56200000000000006,"5min":0.441,"15min":0.441},"max":{"1min":1.6060000000000001,"5min":1.631,"15min":1.829},"last":1.0129999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.115,"15min":1.1499999999999999},"min":{"1min":0.67200000000000004,"5min":0.57399999999999995,"15min":0.48899999999999999},"max":{"1min":1.587,"5min":1.768,"15min":1.768},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.208,"5min":1.204,"15min":1.218},"min":{"1min":0.72399999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5860000000000001,"5min":1.8879999999999999,"15min":1.956},"last":1.389}]},{"osd":2,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2529999999999999,"15min":1.268},"min":{"1min":0.68600000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.325},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.226,"15min":1.292},"min":{"1min":0.76900000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.958},"last":1.4279999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.276,"15min":1.3169999999999999},"min":{"1min":0.77700000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.379,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.2490000000000001,"15min":1.2689999999999999},"min":{"1min":0.83299999999999996,"5min":0.64000000000000001,"15min":0.54600000000000004},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.2430000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.284,"15min":1.2949999999999999},"min":{"1min":0.875,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.772,"5min":1.772,"15min":1.8380000000000001},"last":1.347},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.45,"15min":1.4370000000000001},"min":{"1min":1.0760000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.86399999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3029999999999999,"15min":1.325},"min":{"1min":0.82199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":2.093},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.464,"15min":1.464},"min":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.056}]},{"osd":7,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.421,"15min":1.4179999999999999},"min":{"1min":1.0349999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.466,"15min":1.4830000000000001},"min":{"1min":1.0620000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.958,"5min":1.958,"15min":2.1829999999999998},"last":1.28}]}]},{"osd":4,"up_from":13,"seq":55834575057,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1257472,"data_stored":622550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.139,"15min":1.129},"min":{"1min":0.80300000000000005,"5min":0.66000000000000003,"15min":0.64400000000000002},"max":{"1min":1.4359999999999999,"5min":1.6659999999999999,"15min":1.879},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1339999999999999,"15min":1.1180000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6850000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.65300000000000002}]},{"osd":1,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.29,"15min":1.3100000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.74299999999999999},"max":{"1min":1.609,"5min":1.7490000000000001,"15min":1.923},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.327,"5min":1.375,"15min":1.3520000000000001},"min":{"1min":1.026,"5min":0.86599999999999999,"15min":0.76700000000000002},"max":{"1min":1.768,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.80600000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.232,"15min":1.2110000000000001},"min":{"1min":1.0229999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.589,"5min":1.71,"15min":1.71},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.278,"15min":1.2949999999999999},"min":{"1min":1.083,"5min":0.77400000000000002,"15min":0.77200000000000002},"max":{"1min":1.5189999999999999,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":0.85399999999999998}]},{"osd":3,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.294,"15min":1.2869999999999999},"min":{"1min":0.97499999999999998,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.76,"5min":2.157,"15min":2.157},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.395,"5min":1.361,"15min":1.377},"min":{"1min":1.087,"5min":0.876,"15min":0.67900000000000005},"max":{"1min":1.718,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.97999999999999998}]},{"osd":5,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.3260000000000001,"15min":1.278},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.65800000000000003},"max":{"1min":1.655,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.083},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3620000000000001,"15min":1.2969999999999999},"min":{"1min":0.84099999999999997,"5min":0.83799999999999997,"15min":0.69299999999999995},"max":{"1min":2.0579999999999998,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.02}]},{"osd":6,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3979999999999999,"15min":1.401},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.628},"max":{"1min":1.8049999999999999,"5min":2.04,"15min":2.04},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.5780000000000001,"15min":1.5489999999999999},"min":{"1min":1.3120000000000001,"5min":1.0369999999999999,"15min":0.80600000000000005},"max":{"1min":1.8380000000000001,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":1.151}]},{"osd":7,"last update":"Tue Nov 28 20:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4930000000000001,"15min":1.45},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.76200000000000001},"max":{"1min":1.9570000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.532,"5min":1.538,"15min":1.526},"min":{"1min":1.208,"5min":1.0349999999999999,"15min":0.72299999999999998},"max":{"1min":1.8580000000000001,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.133}]}]},{"osd":3,"up_from":13,"seq":55834575057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40700,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331140,"statfs":{"total":96636764160,"available":96595087360,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.873,"15min":0.90600000000000003},"min":{"1min":0.45500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.6000000000000001,"5min":4.7210000000000001,"15min":4.7210000000000001},"last":1.036},{"interface":"front","average":{"1min":0.875,"5min":0.96499999999999997,"15min":1.071},"min":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.4870000000000001,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":0.82399999999999995}]},{"osd":1,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.95199999999999996,"15min":0.97499999999999998},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4730000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.745},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.1499999999999999,"15min":1.1990000000000001},"min":{"1min":0.58499999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5289999999999999,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.67100000000000004}]},{"osd":2,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.1839999999999999,"15min":1.2470000000000001},"min":{"1min":0.55800000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4490000000000001,"5min":3.444,"15min":3.444},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2769999999999999,"15min":1.335},"min":{"1min":0.66800000000000004,"5min":0.58099999999999996,"15min":0.54400000000000004},"max":{"1min":1.7230000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":1.454}]},{"osd":4,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.298,"15min":1.3420000000000001},"min":{"1min":0.84499999999999997,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.6890000000000001,"5min":1.8779999999999999,"15min":2.9420000000000002},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.327,"15min":1.3839999999999999},"min":{"1min":0.86399999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.1389999999999998},"last":1.1299999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4259999999999999,"15min":1.452},"min":{"1min":0.88,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.8959999999999999,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":1.101},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.375,"15min":1.4330000000000001},"min":{"1min":0.79800000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9670000000000001,"5min":2.2429999999999999,"15min":3.1520000000000001},"last":0.98199999999999998}]},{"osd":6,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3979999999999999,"15min":1.4790000000000001},"min":{"1min":0.86099999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.3100000000000001},"last":1.24},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.5309999999999999,"15min":1.577},"min":{"1min":0.91100000000000003,"5min":0.749,"15min":0.749},"max":{"1min":1.7729999999999999,"5min":3.661,"15min":3.661},"last":1.4119999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.45,"15min":1.5369999999999999},"min":{"1min":0.73399999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.863,"5min":2.1469999999999998,"15min":3.266},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.514,"15min":1.601},"min":{"1min":0.90200000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.1269999999999998,"5min":2.3439999999999999,"15min":2.3919999999999999},"last":1.512}]}]},{"osd":2,"up_from":13,"seq":55834575057,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58748,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313092,"statfs":{"total":96636764160,"available":96576606208,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.88300000000000001,"15min":0.91200000000000003},"min":{"1min":0.54200000000000004,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.624},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.996,"15min":1.0589999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.486,"5min":1.6419999999999999,"15min":1.9390000000000001},"last":0.877}]},{"osd":1,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95399999999999996,"15min":0.98499999999999999},"min":{"1min":0.60799999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.403,"5min":1.7669999999999999,"15min":3.2599999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.099,"15min":1.1499999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.99199999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1240000000000001,"15min":1.1579999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4770000000000001,"5min":1.774,"15min":2.194},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2390000000000001,"15min":1.2929999999999999},"min":{"1min":0.53500000000000003,"5min":0.48199999999999998,"15min":0.45600000000000002},"max":{"1min":1.6579999999999999,"5min":1.831,"15min":3.1440000000000001},"last":1.0329999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.389,"15min":1.411},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.853,"5min":2.23,"15min":2.8919999999999999},"last":1.276},{"interface":"front","average":{"1min":1.389,"5min":1.387,"15min":1.4119999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.988,"5min":2.3740000000000001,"15min":2.3809999999999998},"last":1.5800000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.3919999999999999,"15min":1.421},"min":{"1min":0.85099999999999998,"5min":0.82199999999999995,"15min":0.79600000000000004},"max":{"1min":2.327,"5min":2.327,"15min":2.327},"last":1.218},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4650000000000001,"15min":1.496},"min":{"1min":0.96499999999999997,"5min":0.89100000000000001,"15min":0.75800000000000001},"max":{"1min":2.077,"5min":2.1680000000000001,"15min":3.4279999999999999},"last":1.486}]},{"osd":6,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.452,"15min":1.502},"min":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.165,"5min":2.1709999999999998,"15min":2.621},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.526,"15min":1.5740000000000001},"min":{"1min":0.89000000000000001,"5min":0.82499999999999996,"15min":0.78300000000000003},"max":{"1min":2.6680000000000001,"5min":2.6749999999999998,"15min":3.6269999999999998},"last":1.6279999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.508,"15min":1.542},"min":{"1min":0.94299999999999995,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.54,"15min":1.5900000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.2480000000000002,"5min":2.4849999999999999,"15min":2.7109999999999999},"last":1.796}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36540,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335300,"statfs":{"total":96636764160,"available":96599347200,"internally_reserved":0,"allocated":1224704,"data_stored":613739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.88900000000000001,"15min":0.90400000000000003},"min":{"1min":0.67600000000000005,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4419999999999999,"5min":1.6639999999999999,"15min":1.9259999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.0589999999999999,"15min":1.077},"min":{"1min":0.72999999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":0.88600000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0389999999999999,"15min":1.089},"min":{"1min":0.66200000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.4850000000000001,"5min":1.619,"15min":2.234},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.29,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.88300000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.895},"last":1.111}]},{"osd":3,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.26,"15min":1.3160000000000001},"min":{"1min":0.64400000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.748,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.083,"15min":1.1120000000000001},"min":{"1min":0.84199999999999997,"5min":0.37,"15min":0.37},"max":{"1min":1.744,"5min":1.887,"15min":2.3679999999999999},"last":1.7490000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3220000000000001,"15min":1.337},"min":{"1min":0.97399999999999998,"5min":0.79200000000000004,"15min":0.76600000000000001},"max":{"1min":1.742,"5min":2.1019999999999999,"15min":2.1150000000000002},"last":1.49},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.343,"15min":1.387},"min":{"1min":1.0349999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8799999999999999,"5min":2.1469999999999998,"15min":2.6819999999999999},"last":1.296}]},{"osd":5,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.4059999999999999,"15min":1.429},"min":{"1min":1.165,"5min":0.81100000000000005,"15min":0.81000000000000005},"max":{"1min":1.8620000000000001,"5min":1.903,"15min":2.1840000000000002},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.4870000000000001,"15min":1.496},"min":{"1min":1.2130000000000001,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.8280000000000001,"5min":2.294,"15min":2.294},"last":1.238}]},{"osd":6,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.3819999999999999,"15min":1.4199999999999999},"min":{"1min":1.0980000000000001,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7749999999999999,"5min":1.8759999999999999,"15min":1.9379999999999999},"last":1.643},{"interface":"front","average":{"1min":1.554,"5min":1.4990000000000001,"15min":1.524},"min":{"1min":1.3120000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.956,"5min":2.1520000000000001,"15min":2.3300000000000001},"last":1.425}]},{"osd":7,"last update":"Tue Nov 28 20:50:13 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.542,"15min":1.5640000000000001},"min":{"1min":1.165,"5min":0.753,"15min":0.753},"max":{"1min":2.1429999999999998,"5min":2.218,"15min":2.3359999999999999},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.504,"15min":1.5169999999999999},"min":{"1min":1.3320000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":2.294},"last":1.1870000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575057,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32960,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338880,"statfs":{"total":96636764160,"available":96603013120,"internally_reserved":0,"allocated":1818624,"data_stored":1204107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91500000000000004,"15min":0.90800000000000003},"min":{"1min":0.52100000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3240000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0509999999999999,"15min":1.0800000000000001},"min":{"1min":0.54200000000000004,"5min":0.501,"15min":0.45700000000000002},"max":{"1min":1.534,"5min":1.764,"15min":1.764},"last":0.93100000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.054,"15min":1.1060000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.6080000000000001,"5min":1.6220000000000001,"15min":1.7709999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1950000000000001},"min":{"1min":0.59899999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.722,"5min":1.893,"15min":2.4740000000000002},"last":1.4319999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0660000000000001,"15min":1.1180000000000001},"min":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.508,"5min":2.081,"15min":2.081},"last":0.996},{"interface":"front","average":{"1min":1.262,"5min":1.24,"15min":1.3149999999999999},"min":{"1min":0.67700000000000005,"5min":0.54400000000000004,"15min":0.52500000000000002},"max":{"1min":1.8180000000000001,"5min":1.8839999999999999,"15min":2.1080000000000001},"last":1.512}]},{"osd":4,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3979999999999999,"15min":1.417},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.804,"5min":2.012,"15min":2.012},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.385,"15min":1.3740000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.68300000000000005},"max":{"1min":1.7230000000000001,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.4650000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.403,"15min":1.4159999999999999},"min":{"1min":0.84799999999999998,"5min":0.80100000000000005,"15min":0.79400000000000004},"max":{"1min":1.8460000000000001,"5min":2.0350000000000001,"15min":2.528},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.462,"15min":1.484},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.992,"5min":2.0750000000000002,"15min":2.2490000000000001},"last":1.409}]},{"osd":6,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.512,"15min":1.5489999999999999},"min":{"1min":0.78800000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.0419999999999998,"5min":2.1160000000000001,"15min":2.2959999999999998},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.504,"15min":1.5249999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2330000000000001,"5min":2.274,"15min":2.573},"last":1.2509999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:50:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5720000000000001,"15min":1.591},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.2970000000000002,"5min":2.3610000000000002,"15min":2.4100000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5049999999999999,"15min":1.5429999999999999},"min":{"1min":0.875,"5min":0.81699999999999995,"15min":0.73799999999999999},"max":{"1min":1.776,"5min":2.3460000000000001,"15min":2.6640000000000001},"last":1.2010000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:51:18.524 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T20:51:18.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T20:51:18.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.794+0000 7fbcbf348640 1 Processor -- start 2023-11-28T20:51:18.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.794+0000 7fbcbf348640 1 -- start start 2023-11-28T20:51:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.796+0000 7fbcbf348640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbcb806ac10 0x7fbcb806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.796+0000 7fbcbf348640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815f720 0x7fbcb80717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.796+0000 7fbcbf348640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb815ecd0 con 0x7fbcb815e840 2023-11-28T20:51:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.796+0000 7fbcbf348640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb8071ce0 con 0x7fbcb806ac10 2023-11-28T20:51:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.796+0000 7fbcbf348640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb815fbd0 con 0x7fbcb815f720 2023-11-28T20:51:18.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.796+0000 7fbcbd0e4640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbcb815e840 0x7fbcb806a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48898/0 (socket says 172.21.15.45:48898) 2023-11-28T20:51:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.796+0000 7fbcbd0e4640 1 -- 172.21.15.45:0/2418883982 learned_addr learned my addr 172.21.15.45:0/2418883982 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.796+0000 7fbcbc8e3640 1 --2- 172.21.15.45:0/2418883982 >> v2:172.21.15.45:3300/0 conn(0x7fbcb806ac10 0x7fbcb806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.797+0000 7fbcbd8e5640 1 --2- 172.21.15.45:0/2418883982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815f720 0x7fbcb80717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.797+0000 7fbcaffff640 1 -- 172.21.15.45:0/2418883982 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703792427 0 0) 0x7fbcb815ecd0 con 0x7fbcb815e840 2023-11-28T20:51:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.797+0000 7fbcbc8e3640 1 -- 172.21.15.45:0/2418883982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815f720 msgr2=0x7fbcb80717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.797+0000 7fbcbc8e3640 1 --2- 172.21.15.45:0/2418883982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815f720 0x7fbcb80717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.797+0000 7fbcbc8e3640 1 -- 172.21.15.45:0/2418883982 >> v1:172.21.15.45:6789/0 conn(0x7fbcb815e840 legacy=0x7fbcb806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.797+0000 7fbcbc8e3640 1 -- 172.21.15.45:0/2418883982 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb815f290 con 0x7fbcb806ac10 2023-11-28T20:51:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.797+0000 7fbcbd8e5640 1 --2- 172.21.15.45:0/2418883982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815f720 0x7fbcb80717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:18.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.798+0000 7fbcbc8e3640 1 --2- 172.21.15.45:0/2418883982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 0x7fbcb806c350 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fbca80097a0 tx=0x7fbca802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dbf2a5a6beeedd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:18.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.798+0000 7fbcaffff640 1 -- 172.21.15.45:0/2418883982 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca802d040 con 0x7fbcb806ac10 2023-11-28T20:51:18.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.798+0000 7fbcaffff640 1 -- 172.21.15.45:0/2418883982 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca8031810 con 0x7fbcb806ac10 2023-11-28T20:51:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.798+0000 7fbcaffff640 1 -- 172.21.15.45:0/2418883982 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8031ba0 con 0x7fbcb806ac10 2023-11-28T20:51:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.799+0000 7fbcbf348640 1 -- 172.21.15.45:0/2418883982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 msgr2=0x7fbcb806c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.799+0000 7fbcbf348640 1 --2- 172.21.15.45:0/2418883982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 0x7fbcb806c350 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fbca80097a0 tx=0x7fbca802b8b0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.799+0000 7fbcbf348640 1 -- 172.21.15.45:0/2418883982 shutdown_connections 2023-11-28T20:51:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.799+0000 7fbcbf348640 1 --2- 172.21.15.45:0/2418883982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815f720 0x7fbcb80717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.799+0000 7fbcbf348640 1 --2- 172.21.15.45:0/2418883982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 0x7fbcb806c350 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.799+0000 7fbcbf348640 1 -- 172.21.15.45:0/2418883982 >> 172.21.15.45:0/2418883982 conn(0x7fbcb80ced60 msgr2=0x7fbcb80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.799+0000 7fbcbf348640 1 -- 172.21.15.45:0/2418883982 shutdown_connections 2023-11-28T20:51:18.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.799+0000 7fbcbf348640 1 -- 172.21.15.45:0/2418883982 wait complete. 2023-11-28T20:51:18.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.801+0000 7fbcbf348640 1 Processor -- start 2023-11-28T20:51:18.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.802+0000 7fbcbf348640 1 -- start start 2023-11-28T20:51:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.803+0000 7fbcbf348640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 0x7fbcb80c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.803+0000 7fbcbf348640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815e840 0x7fbcb80b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.803+0000 7fbcbf348640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcb815f720 0x7fbcb80b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.803+0000 7fbcbf348640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80ca0a0 con 0x7fbcb806ac10 2023-11-28T20:51:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.803+0000 7fbcbf348640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80ca380 con 0x7fbcb815f720 2023-11-28T20:51:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.803+0000 7fbcbf348640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80ca660 con 0x7fbcb815e840 2023-11-28T20:51:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.803+0000 7fbcbd0e4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 0x7fbcb80c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.803+0000 7fbcbd0e4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 0x7fbcb80c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44852/0 (socket says 172.21.15.45:44852) 2023-11-28T20:51:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.803+0000 7fbcbd0e4640 1 -- 172.21.15.45:0/2801819748 learned_addr learned my addr 172.21.15.45:0/2801819748 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcbc8e3640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815e840 0x7fbcb80b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcbd0e4640 1 -- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815e840 msgr2=0x7fbcb80b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcbd0e4640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815e840 0x7fbcb80b2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcbd0e4640 1 -- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcb815f720 msgr2=0x7fbcb80b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcbd0e4640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcb815f720 0x7fbcb80b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcbd0e4640 1 -- 172.21.15.45:0/2801819748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8009400 con 0x7fbcb806ac10 2023-11-28T20:51:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcbc8e3640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815e840 0x7fbcb80b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:18.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcbd0e4640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 0x7fbcb80c4cd0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c890 tx=0x7fbcb400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcadffb640 1 -- 172.21.15.45:0/2801819748 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb400a640 con 0x7fbcb806ac10 2023-11-28T20:51:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcadffb640 1 -- 172.21.15.45:0/2801819748 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb400a7c0 con 0x7fbcb806ac10 2023-11-28T20:51:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb80ca940 con 0x7fbcb806ac10 2023-11-28T20:51:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.804+0000 7fbcadffb640 1 -- 172.21.15.45:0/2801819748 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb400ab70 con 0x7fbcb806ac10 2023-11-28T20:51:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.805+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb80c5210 con 0x7fbcb806ac10 2023-11-28T20:51:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.806+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb80c5400 con 0x7fbcb806ac10 2023-11-28T20:51:18.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.806+0000 7fbcadffb640 1 -- 172.21.15.45:0/2801819748 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbcb400acd0 con 0x7fbcb806ac10 2023-11-28T20:51:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.807+0000 7fbcadffb640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fbc9c06af60 0x7fbc9c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.808+0000 7fbcadffb640 1 -- 172.21.15.45:0/2801819748 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7fbcb4088ec0 con 0x7fbcb806ac10 2023-11-28T20:51:18.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.808+0000 7fbcbc8e3640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fbc9c06af60 0x7fbc9c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:18.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.809+0000 7fbcbc8e3640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fbc9c06af60 0x7fbc9c06d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbca8036000 tx=0x7fbca803c040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:18.814+0000 7fbcadffb640 1 -- 172.21.15.45:0/2801819748 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbcb4053c50 con 0x7fbcb806ac10 2023-11-28T20:51:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.068+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbcb80cae60 con 0x7fbcb806ac10 2023-11-28T20:51:19.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.073+0000 7fbcadffb640 1 -- 172.21.15.45:0/2801819748 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7fbcb80cae60 con 0x7fbcb806ac10 2023-11-28T20:51:19.075 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T20:51:19.075 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:51:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T20:51:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T20:51:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T20:51:19.076 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T20:51:19.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fbc9c06af60 msgr2=0x7fbc9c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:19.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fbc9c06af60 0x7fbc9c06d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbca8036000 tx=0x7fbca803c040 comp rx=0 tx=0).stop 2023-11-28T20:51:19.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 msgr2=0x7fbcb80c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:19.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 0x7fbcb80c4cd0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c890 tx=0x7fbcb400cd50 comp rx=0 tx=0).stop 2023-11-28T20:51:19.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 shutdown_connections 2023-11-28T20:51:19.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fbc9c06af60 0x7fbc9c06d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcb815f720 0x7fbcb80b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbcb815e840 0x7fbcb80b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 --2- 172.21.15.45:0/2801819748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcb806ac10 0x7fbcb80c4cd0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.075+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 >> 172.21.15.45:0/2801819748 conn(0x7fbcb80ced60 msgr2=0x7fbcb80af080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.076+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 shutdown_connections 2023-11-28T20:51:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.076+0000 7fbcbf348640 1 -- 172.21.15.45:0/2801819748 wait complete. 2023-11-28T20:51:19.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:19.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.315+0000 7f3701ca3640 1 Processor -- start 2023-11-28T20:51:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.315+0000 7f3701ca3640 1 -- start start 2023-11-28T20:51:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.317+0000 7f3701ca3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc0b24a0 0x7f36fc0ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.317+0000 7f3701ca3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f36fc0ae590 0x7f36fc0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.317+0000 7f3701ca3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc0abf10 con 0x7f36fc14b090 2023-11-28T20:51:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.317+0000 7f3701ca3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36fc06db90 con 0x7f36fc0ae590 2023-11-28T20:51:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.317+0000 7f3701ca3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc06d2f0 con 0x7f36fc0b24a0 2023-11-28T20:51:19.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.318+0000 7f3700ca1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f36fc14b090 0x7f36fc0b1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48908/0 (socket says 172.21.15.45:48908) 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.318+0000 7f3700ca1640 1 -- 172.21.15.45:0/1670628508 learned_addr learned my addr 172.21.15.45:0/1670628508 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.318+0000 7f36fbfff640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc0b24a0 0x7f36fc0ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.318+0000 7f37014a2640 1 --2- 172.21.15.45:0/1670628508 >> v2:172.21.15.190:3300/0 conn(0x7f36fc0ae590 0x7f36fc0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.318+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128066890 0 0) 0x7f36fc0abf10 con 0x7f36fc14b090 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.318+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ec003520 con 0x7f36fc14b090 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502912730 0 0) 0x7f36ec003520 con 0x7f36fc14b090 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc0b24a0 msgr2=0x7f36fc0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc0b24a0 0x7f36fc0ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 >> v2:172.21.15.190:3300/0 conn(0x7f36fc0ae590 msgr2=0x7f36fc0ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 --2- 172.21.15.45:0/1670628508 >> v2:172.21.15.190:3300/0 conn(0x7f36fc0ae590 0x7f36fc0ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc06d5d0 con 0x7f36fc14b090 2023-11-28T20:51:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f36f0001670 con 0x7f36fc14b090 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 >> v1:172.21.15.45:6789/0 conn(0x7f36fc14b090 legacy=0x7f36fc0b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc0b24a0 0x7f36ec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36ec004d00 0x7f36ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc06db90 con 0x7f36ec003f40 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc06d2f0 con 0x7f36ec004d00 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec00b980 con 0x7f36fc0b24a0 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fbfff640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f3700ca1640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc0b24a0 0x7f36ec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fbfff640 1 -- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc0b24a0 msgr2=0x7f36ec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fbfff640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc0b24a0 0x7f36ec003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f36fbfff640 1 -- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36ec004d00 msgr2=0x7f36ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:19.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.319+0000 7f37014a2640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36ec004d00 0x7f36ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:19.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.320+0000 7f36fbfff640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36ec004d00 0x7f36ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.320+0000 7f36fbfff640 1 -- 172.21.15.45:0/1670628508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec003520 con 0x7f36ec003f40 2023-11-28T20:51:19.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.320+0000 7f37014a2640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36ec004d00 0x7f36ec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:19.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.321+0000 7f36fbfff640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36ec0047c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009af0 tx=0x7f36ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fcfb77bbd17c08 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:19.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.321+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e802c6b0 con 0x7f36ec003f40 2023-11-28T20:51:19.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.321+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e802c810 con 0x7f36ec003f40 2023-11-28T20:51:19.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.321+0000 7f36fb7fe640 1 -- 172.21.15.45:0/1670628508 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e802cba0 con 0x7f36ec003f40 2023-11-28T20:51:19.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.321+0000 7f3701ca3640 1 -- 172.21.15.45:0/1670628508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 msgr2=0x7f36ec0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:19.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.321+0000 7f3701ca3640 1 --2- 172.21.15.45:0/1670628508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36ec0047c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009af0 tx=0x7f36ec0057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.321+0000 7f37014a2640 1 -- 172.21.15.45:0/1670628508 reap_dead start 2023-11-28T20:51:19.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.322+0000 7f3701ca3640 1 -- 172.21.15.45:0/1670628508 shutdown_connections 2023-11-28T20:51:19.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.322+0000 7f3701ca3640 1 -- 172.21.15.45:0/1670628508 >> 172.21.15.45:0/1670628508 conn(0x7f36fc06ab30 msgr2=0x7f36fc0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:19.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.322+0000 7f3701ca3640 1 -- 172.21.15.45:0/1670628508 shutdown_connections 2023-11-28T20:51:19.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.322+0000 7f3701ca3640 1 -- 172.21.15.45:0/1670628508 wait complete. 2023-11-28T20:51:19.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.323+0000 7f3701ca3640 1 Processor -- start 2023-11-28T20:51:19.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.323+0000 7f3701ca3640 1 -- start start 2023-11-28T20:51:19.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.324+0000 7f3701ca3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36fc146350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:19.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.324+0000 7f3701ca3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36fc149120 0x7f36fc1495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:19.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.324+0000 7f3701ca3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc160b20 0x7f36fc160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:19.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.324+0000 7f3701ca3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc1f8370 con 0x7f36ec003f40 2023-11-28T20:51:19.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.324+0000 7f3701ca3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc149ae0 con 0x7f36fc149120 2023-11-28T20:51:19.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f3701ca3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc1faa30 con 0x7f36fc160b20 2023-11-28T20:51:19.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f3700ca1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36fc146350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:19.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f3700ca1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36fc146350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44876/0 (socket says 172.21.15.45:44876) 2023-11-28T20:51:19.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f3700ca1640 1 -- 172.21.15.45:0/3171990412 learned_addr learned my addr 172.21.15.45:0/3171990412 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:19.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f37014a2640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc160b20 0x7f36fc160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:19.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f3700ca1640 1 -- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc160b20 msgr2=0x7f36fc160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:19.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f3700ca1640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc160b20 0x7f36fc160f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f3700ca1640 1 -- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36fc149120 msgr2=0x7f36fc1495a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:19.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f3700ca1640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36fc149120 0x7f36fc1495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.325+0000 7f3700ca1640 1 -- 172.21.15.45:0/3171990412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc1fad10 con 0x7f36ec003f40 2023-11-28T20:51:19.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.326+0000 7f3700ca1640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36fc146350 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e730 tx=0x7f36f000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:19.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.326+0000 7f36f97fa640 1 -- 172.21.15.45:0/3171990412 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f000ee40 con 0x7f36ec003f40 2023-11-28T20:51:19.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.326+0000 7f36f97fa640 1 -- 172.21.15.45:0/3171990412 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f000c8d0 con 0x7f36ec003f40 2023-11-28T20:51:19.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.326+0000 7f36f97fa640 1 -- 172.21.15.45:0/3171990412 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f000cc80 con 0x7f36ec003f40 2023-11-28T20:51:19.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.326+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36fc1faff0 con 0x7f36ec003f40 2023-11-28T20:51:19.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.326+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36fc1614a0 con 0x7f36ec003f40 2023-11-28T20:51:19.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.327+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36fc09d0f0 con 0x7f36ec003f40 2023-11-28T20:51:19.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.328+0000 7f36f97fa640 1 -- 172.21.15.45:0/3171990412 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f36f001d460 con 0x7f36ec003f40 2023-11-28T20:51:19.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.329+0000 7f36f97fa640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f36e006afb0 0x7f36e006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:19.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.329+0000 7f36f97fa640 1 -- 172.21.15.45:0/3171990412 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f36f00898a0 con 0x7f36ec003f40 2023-11-28T20:51:19.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.330+0000 7f36fbfff640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f36e006afb0 0x7f36e006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:19.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.331+0000 7f36fbfff640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f36e006afb0 0x7f36e006d470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f36e8036000 tx=0x7f36e802e000 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:19.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.333+0000 7f36f97fa640 1 -- 172.21.15.45:0/3171990412 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36f0054740 con 0x7f36ec003f40 2023-11-28T20:51:19.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.570+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36fc072510 con 0x7f36ec003f40 2023-11-28T20:51:19.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.570+0000 7f36f97fa640 1 -- 172.21.15.45:0/3171990412 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f36f0057d60 con 0x7f36ec003f40 2023-11-28T20:51:19.573 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:19.573 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3492018158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3492018158},{"type":"v1","addr":"172.21.15.45:6833","nonce":3492018158}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6121,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2276329291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2276329291},{"type":"v1","addr":"172.21.15.190:6835","nonce":2276329291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6130,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1261857840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1261857840},{"type":"v1","addr":"172.21.15.190:6837","nonce":1261857840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:12.953862+0000","modified":"2023-11-28T20:51:16.015607+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7179},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7179":{"gid":7179,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/862206408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":862206408},{"type":"v1","addr":"172.21.15.45:6835","nonce":862206408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-28T20:51:19.574 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 44 2023-11-28T20:51:19.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f36e006afb0 msgr2=0x7f36e006d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:19.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f36e006afb0 0x7f36e006d470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f36e8036000 tx=0x7f36e802e000 comp rx=0 tx=0).stop 2023-11-28T20:51:19.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 msgr2=0x7f36fc146350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:19.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36fc146350 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e730 tx=0x7f36f000ebf0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 shutdown_connections 2023-11-28T20:51:19.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f36e006afb0 0x7f36e006d470 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f36fc160b20 0x7f36fc160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36fc149120 0x7f36fc1495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 --2- 172.21.15.45:0/3171990412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36ec003f40 0x7f36fc146350 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:19.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 >> 172.21.15.45:0/3171990412 conn(0x7f36fc06ab30 msgr2=0x7f36fc0af170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:19.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 shutdown_connections 2023-11-28T20:51:19.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:19.573+0000 7f3701ca3640 1 -- 172.21.15.45:0/3171990412 wait complete. 2023-11-28T20:51:19.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:51:19.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:51:19.996 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:51:12.953862+0000', 'modified': '2023-11-28T20:51:16.015607+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7179}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7179': {'gid': 7179, 'name': 'c', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/862206408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6834', 'nonce': 862206408}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 862206408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T20:51:19.997 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:51:19.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T20:51:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.229+0000 7fe47b2e1640 1 Processor -- start 2023-11-28T20:51:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.229+0000 7fe47b2e1640 1 -- start start 2023-11-28T20:51:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.231+0000 7fe47b2e1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe474161540 0x7fe4740b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.231+0000 7fe47b2e1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4740cf610 0x7fe4740704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.231+0000 7fe47b2e1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47406c6e0 con 0x7fe47406c1d0 2023-11-28T20:51:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.231+0000 7fe47b2e1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47406c9c0 con 0x7fe474161540 2023-11-28T20:51:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.231+0000 7fe47b2e1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47406cca0 con 0x7fe4740cf610 2023-11-28T20:51:20.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.232+0000 7fe47a2df640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe474161540 0x7fe4740b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.232+0000 7fe47a2df640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe474161540 0x7fe4740b3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44878/0 (socket says 172.21.15.45:44878) 2023-11-28T20:51:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.232+0000 7fe47a2df640 1 -- 172.21.15.45:0/3385729557 learned_addr learned my addr 172.21.15.45:0/3385729557 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.232+0000 7fe47aae0640 1 --2- 172.21.15.45:0/3385729557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4740cf610 0x7fe4740704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.232+0000 7fe47a2df640 1 -- 172.21.15.45:0/3385729557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4740cf610 msgr2=0x7fe4740704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.232+0000 7fe47a2df640 1 --2- 172.21.15.45:0/3385729557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4740cf610 0x7fe4740704e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.232+0000 7fe47a2df640 1 -- 172.21.15.45:0/3385729557 >> v1:172.21.15.190:6789/0 conn(0x7fe47406c1d0 legacy=0x7fe4740cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.232+0000 7fe47a2df640 1 -- 172.21.15.45:0/3385729557 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47406cf80 con 0x7fe474161540 2023-11-28T20:51:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.234+0000 7fe47a2df640 1 --2- 172.21.15.45:0/3385729557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe474161540 0x7fe4740b3cf0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fe470009870 tx=0x7fe47002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc62b98f998a6dd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.234+0000 7fe4792dd640 1 -- 172.21.15.45:0/3385729557 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470031640 con 0x7fe474161540 2023-11-28T20:51:20.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.234+0000 7fe4792dd640 1 -- 172.21.15.45:0/3385729557 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4700317a0 con 0x7fe474161540 2023-11-28T20:51:20.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.234+0000 7fe4792dd640 1 -- 172.21.15.45:0/3385729557 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470031b30 con 0x7fe474161540 2023-11-28T20:51:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.235+0000 7fe47b2e1640 1 -- 172.21.15.45:0/3385729557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe474161540 msgr2=0x7fe4740b3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.235+0000 7fe47b2e1640 1 --2- 172.21.15.45:0/3385729557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe474161540 0x7fe4740b3cf0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fe470009870 tx=0x7fe47002b8f0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.235+0000 7fe47b2e1640 1 -- 172.21.15.45:0/3385729557 shutdown_connections 2023-11-28T20:51:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.235+0000 7fe47b2e1640 1 --2- 172.21.15.45:0/3385729557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4740cf610 0x7fe4740704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.235+0000 7fe47b2e1640 1 --2- 172.21.15.45:0/3385729557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe474161540 0x7fe4740b3cf0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.235+0000 7fe47b2e1640 1 -- 172.21.15.45:0/3385729557 >> 172.21.15.45:0/3385729557 conn(0x7fe47406a710 msgr2=0x7fe47406ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:20.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.235+0000 7fe47b2e1640 1 -- 172.21.15.45:0/3385729557 shutdown_connections 2023-11-28T20:51:20.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.235+0000 7fe47b2e1640 1 -- 172.21.15.45:0/3385729557 wait complete. 2023-11-28T20:51:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.237+0000 7fe47b2e1640 1 Processor -- start 2023-11-28T20:51:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.237+0000 7fe47b2e1640 1 -- start start 2023-11-28T20:51:20.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.238+0000 7fe47b2e1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe47406c1d0 0x7fe4741634f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.238+0000 7fe47b2e1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4740cf610 0x7fe4741507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.238+0000 7fe47b2e1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe474161540 0x7fe474150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.238+0000 7fe47b2e1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe474163d20 con 0x7fe4740cf610 2023-11-28T20:51:20.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.238+0000 7fe47b2e1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe474164000 con 0x7fe474161540 2023-11-28T20:51:20.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.238+0000 7fe47b2e1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4741642e0 con 0x7fe47406c1d0 2023-11-28T20:51:20.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.238+0000 7fe479ade640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4740cf610 0x7fe4741507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.238+0000 7fe479ade640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4740cf610 0x7fe4741507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44894/0 (socket says 172.21.15.45:44894) 2023-11-28T20:51:20.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.238+0000 7fe479ade640 1 -- 172.21.15.45:0/1058698403 learned_addr learned my addr 172.21.15.45:0/1058698403 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe479ade640 1 -- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe47406c1d0 msgr2=0x7fe4741634f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe47a2df640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe47406c1d0 0x7fe4741634f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe479ade640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe47406c1d0 0x7fe4741634f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe47aae0640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe474161540 0x7fe474150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe479ade640 1 -- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe474161540 msgr2=0x7fe474150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe479ade640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe474161540 0x7fe474150cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe479ade640 1 -- 172.21.15.45:0/1058698403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4700094d0 con 0x7fe4740cf610 2023-11-28T20:51:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe479ade640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4740cf610 0x7fe4741507b0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c400 tx=0x7fe46400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe46affd640 1 -- 172.21.15.45:0/1058698403 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4640052d0 con 0x7fe4740cf610 2023-11-28T20:51:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe46affd640 1 -- 172.21.15.45:0/1058698403 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464005450 con 0x7fe4740cf610 2023-11-28T20:51:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe47b2e1640 1 -- 172.21.15.45:0/1058698403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4741645c0 con 0x7fe4740cf610 2023-11-28T20:51:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.239+0000 7fe46affd640 1 -- 172.21.15.45:0/1058698403 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464007260 con 0x7fe4740cf610 2023-11-28T20:51:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.240+0000 7fe47b2e1640 1 -- 172.21.15.45:0/1058698403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe474163a30 con 0x7fe4740cf610 2023-11-28T20:51:20.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.242+0000 7fe46affd640 1 -- 172.21.15.45:0/1058698403 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe4640073c0 con 0x7fe4740cf610 2023-11-28T20:51:20.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.243+0000 7fe46affd640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe45806af60 0x7fe45806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.243+0000 7fe46affd640 1 -- 172.21.15.45:0/1058698403 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7fe464088da0 con 0x7fe4740cf610 2023-11-28T20:51:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.244+0000 7fe47f1b9740 1 Processor -- start 2023-11-28T20:51:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.244+0000 7fe47a2df640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe45806af60 0x7fe45806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.244+0000 7fe47f1b9740 1 -- start start 2023-11-28T20:51:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 0x563260ca2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 0x563260ca1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563260c00880 con 0x563260ca17b0 2023-11-28T20:51:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563260c0fae0 con 0x563260c9cd70 2023-11-28T20:51:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x563260c0d1d0 con 0x563260ca2070 2023-11-28T20:51:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47a2df640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe45806af60 0x7fe45806d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe470036000 tx=0x7fe47003c040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47aae0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47aae0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44906/0 (socket says 172.21.15.45:44906) 2023-11-28T20:51:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.245+0000 7fe47aae0640 1 -- 172.21.15.45:0/2714237663 learned_addr learned my addr 172.21.15.45:0/2714237663 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe47a2df640 1 --2- 172.21.15.45:0/2714237663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 0x563260ca2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe479ade640 1 --2- 172.21.15.45:0/2714237663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 0x563260ca1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe47aae0640 1 -- 172.21.15.45:0/2714237663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 msgr2=0x563260ca2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe47aae0640 1 --2- 172.21.15.45:0/2714237663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 0x563260ca2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe47aae0640 1 -- 172.21.15.45:0/2714237663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 msgr2=0x563260ca1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe47aae0640 1 --2- 172.21.15.45:0/2714237663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 0x563260ca1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe47aae0640 1 -- 172.21.15.45:0/2714237663 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563260bd0860 con 0x563260ca17b0 2023-11-28T20:51:20.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe47aae0640 1 --2- 172.21.15.45:0/2714237663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5730 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009fa0 tx=0x7fe46c0077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec674400ada9ae45 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe457fff640 1 -- 172.21.15.45:0/2714237663 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c013780 con 0x563260ca17b0 2023-11-28T20:51:20.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe457fff640 1 -- 172.21.15.45:0/2714237663 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c00f050 con 0x563260ca17b0 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.246+0000 7fe457fff640 1 -- 172.21.15.45:0/2714237663 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c013b20 con 0x563260ca17b0 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.247+0000 7fe47f1b9740 1 -- 172.21.15.45:0/2714237663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 msgr2=0x563260ce5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.247+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/2714237663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5730 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009fa0 tx=0x7fe46c0077c0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.247+0000 7fe47f1b9740 1 -- 172.21.15.45:0/2714237663 shutdown_connections 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.247+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/2714237663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5730 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.247+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/2714237663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 0x563260ca1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.247+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/2714237663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 0x563260ca2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.247+0000 7fe47f1b9740 1 -- 172.21.15.45:0/2714237663 >> 172.21.15.45:0/2714237663 conn(0x563260c63c70 msgr2=0x563260ce7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.247+0000 7fe47f1b9740 1 -- 172.21.15.45:0/2714237663 shutdown_connections 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.247+0000 7fe47f1b9740 1 -- 172.21.15.45:0/2714237663 wait complete. 2023-11-28T20:51:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.248+0000 7fe47f1b9740 1 -- start start 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.248+0000 7fe47f1b9740 1 Processor -- start 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.249+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260c9cd70 0x563260cfb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.249+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260ca17b0 0x563260cec230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.249+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260cc7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.249+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563260c08ac0 con 0x563260ca2070 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.249+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563260ba52f0 con 0x563260ca17b0 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.249+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x563260bc2c70 con 0x563260c9cd70 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47f1b9740 10 client.0 _ll_register_callbacks cb 0x563260bc73b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47aae0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260cc7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47aae0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260cc7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44914/0 (socket says 172.21.15.45:44914) 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47aae0640 1 -- 172.21.15.45:0/1284609795 learned_addr learned my addr 172.21.15.45:0/1284609795 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47a2df640 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260c9cd70 0x563260cfb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe479ade640 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260ca17b0 0x563260cec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47aae0640 1 -- 172.21.15.45:0/1284609795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260c9cd70 msgr2=0x563260cfb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47aae0640 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260c9cd70 0x563260cfb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47aae0640 1 -- 172.21.15.45:0/1284609795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260ca17b0 msgr2=0x563260cec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47aae0640 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260ca17b0 0x563260cec230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.250+0000 7fe47aae0640 1 -- 172.21.15.45:0/1284609795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46c009ba0 con 0x563260ca2070 2023-11-28T20:51:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.251+0000 7fe47aae0640 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260cc7a20 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009450 tx=0x7fe46c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db418dbdec875c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.251+0000 7fe4557fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:51:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.251+0000 7fe4557fa640 1 -- 172.21.15.45:0/1284609795 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c009610 con 0x563260ca2070 2023-11-28T20:51:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.251+0000 7fe4557fa640 1 -- 172.21.15.45:0/1284609795 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7fe46c029c50 con 0x563260ca2070 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.251+0000 7fe47f1b9740 1 -- 172.21.15.45:0/1284609795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563260c01aa0 con 0x563260ca2070 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.251+0000 7fe4557fa640 1 -- 172.21.15.45:0/1284609795 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe46c003e60 con 0x563260ca2070 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.251+0000 7fe47f1b9740 10 client.7314 fetch_fsmap learned FSMap version 44 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.251+0000 7fe47f1b9740 1 -- 172.21.15.45:0/1284609795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x563260c07140 con 0x563260ca2070 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.252+0000 7fe4557fa640 1 -- 172.21.15.45:0/1284609795 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563260c01aa0 con 0x563260ca2070 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.252+0000 7fe4557fa640 1 -- 172.21.15.45:0/1284609795 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 44) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe46c0045f0 con 0x563260ca2070 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.252+0000 7fe47f1b9740 10 client.7314 fetch_fsmap finished waiting for FSMap version 44 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.252+0000 7fe47f1b9740 10 client.7314 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.252+0000 7fe47f1b9740 20 client.7314 populate_metadata read hostname 'smithi045' 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.252+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] conn(0x563260cca470 0x563260cca850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.252+0000 7fe47f1b9740 4 client.7314 mds_command: new command op to 7179 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:51:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.252+0000 7fe47f1b9740 1 -- 172.21.15.45:0/1284609795 --> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563260c017c0 con 0x563260cca470 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.252+0000 7fe47a2df640 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] conn(0x563260cca470 0x563260cca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.253+0000 7fe47a2df640 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] conn(0x563260cca470 0x563260cca850 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66a81c58b31b2ca0 server_cookie=97e453e63ef19b7a in_seq=0 out_seq=0 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.253+0000 7fe4557fa640 10 client.7314 ms_handle_connect on v2:172.21.15.45:6834/862206408 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.253+0000 7fe4557fa640 1 -- 172.21.15.45:0/1284609795 <== mds.0 v2:172.21.15.45:6834/862206408 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563260c017c0 con 0x563260cca470 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.253+0000 7fe4557fa640 10 client.7314 handle_command_reply: tid=0 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 client.7314 shutdown 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 -- 172.21.15.45:0/1284609795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 msgr2=0x563260cc7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260cc7a20 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009450 tx=0x7fe46c004ab0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 -- 172.21.15.45:0/1284609795 shutdown_connections 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] conn(0x563260cca470 0x563260cca850 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260cc7a20 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260ca17b0 0x563260cec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/1284609795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260c9cd70 0x563260cfb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 -- 172.21.15.45:0/1284609795 >> 172.21.15.45:0/1284609795 conn(0x563260c63c70 msgr2=0x563260ceaca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe4557fa640 0 client.7314 ms_handle_reset on v2:172.21.15.45:6834/862206408 2023-11-28T20:51:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 -- 172.21.15.45:0/1284609795 shutdown_connections 2023-11-28T20:51:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 1 -- 172.21.15.45:0/1284609795 wait complete. 2023-11-28T20:51:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.254+0000 7fe47f1b9740 20 client.7314 trim_cache size 0 max 16384 2023-11-28T20:51:20.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.271+0000 7fe47f1b9740 1 Processor -- start 2023-11-28T20:51:20.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.272+0000 7fe47f1b9740 1 -- start start 2023-11-28T20:51:20.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.272+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260c9cd70 0x563260ccaa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.272+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260ca17b0 0x563260ce6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.272+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260ce6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.272+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56326096bf20 con 0x563260ca2070 2023-11-28T20:51:20.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.272+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563260cccfb0 con 0x563260ca17b0 2023-11-28T20:51:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.273+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x563260ccc4f0 con 0x563260c9cd70 2023-11-28T20:51:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.273+0000 7fe47aae0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260ce6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.273+0000 7fe47aae0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260ce6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44920/0 (socket says 172.21.15.45:44920) 2023-11-28T20:51:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.273+0000 7fe47aae0640 1 -- 172.21.15.45:0/3295657136 learned_addr learned my addr 172.21.15.45:0/3295657136 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.273+0000 7fe47aae0640 1 -- 172.21.15.45:0/3295657136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260c9cd70 msgr2=0x563260ccaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.273+0000 7fe47aae0640 1 --2- 172.21.15.45:0/3295657136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260c9cd70 0x563260ccaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.273+0000 7fe47aae0640 1 -- 172.21.15.45:0/3295657136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260ca17b0 msgr2=0x563260ce6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.273+0000 7fe47aae0640 1 --2- 172.21.15.45:0/3295657136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260ca17b0 0x563260ce6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.273+0000 7fe47aae0640 1 -- 172.21.15.45:0/3295657136 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563260ba4b80 con 0x563260ca2070 2023-11-28T20:51:20.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.274+0000 7fe47aae0640 1 --2- 172.21.15.45:0/3295657136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260ce6fa0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe46c007800 tx=0x7fe46c021070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0f76a6d03020bc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.274+0000 7fe468ff9640 1 -- 172.21.15.45:0/3295657136 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c036070 con 0x563260ca2070 2023-11-28T20:51:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.274+0000 7fe468ff9640 1 -- 172.21.15.45:0/3295657136 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c023070 con 0x563260ca2070 2023-11-28T20:51:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.274+0000 7fe468ff9640 1 -- 172.21.15.45:0/3295657136 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c009610 con 0x563260ca2070 2023-11-28T20:51:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.274+0000 7fe47f1b9740 1 -- 172.21.15.45:0/3295657136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 msgr2=0x563260ce6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.274+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/3295657136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260ce6fa0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe46c007800 tx=0x7fe46c021070 comp rx=0 tx=0).stop 2023-11-28T20:51:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.274+0000 7fe47f1b9740 1 -- 172.21.15.45:0/3295657136 shutdown_connections 2023-11-28T20:51:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.275+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/3295657136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca2070 0x563260ce6fa0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.275+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/3295657136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260ca17b0 0x563260ce6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.275+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/3295657136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260c9cd70 0x563260ccaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.275+0000 7fe47f1b9740 1 -- 172.21.15.45:0/3295657136 >> 172.21.15.45:0/3295657136 conn(0x563260c63c70 msgr2=0x563260cec0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.275+0000 7fe47f1b9740 1 -- 172.21.15.45:0/3295657136 shutdown_connections 2023-11-28T20:51:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.275+0000 7fe47f1b9740 1 -- 172.21.15.45:0/3295657136 wait complete. 2023-11-28T20:51:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.275+0000 7fe47f1b9740 1 -- start start 2023-11-28T20:51:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.276+0000 7fe47f1b9740 1 Processor -- start 2023-11-28T20:51:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.276+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 0x563260ce5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe47f1b9740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 0x563260ceaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563260cc89e0 con 0x563260ca17b0 2023-11-28T20:51:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563260ccadd0 con 0x563260c9cd70 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe47f1b9740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x563260d528c0 con 0x563260ca2070 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe47f1b9740 10 client.0 _ll_register_callbacks cb 0x7fe46c009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe479ade640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe479ade640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44932/0 (socket says 172.21.15.45:44932) 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe479ade640 1 -- 172.21.15.45:0/52348293 learned_addr learned my addr 172.21.15.45:0/52348293 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe47a2df640 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 0x563260ce5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe47aae0640 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 0x563260ceaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe479ade640 1 -- 172.21.15.45:0/52348293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 msgr2=0x563260ceaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe479ade640 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 0x563260ceaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe479ade640 1 -- 172.21.15.45:0/52348293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 msgr2=0x563260ce5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe479ade640 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 0x563260ce5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe479ade640 1 -- 172.21.15.45:0/52348293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563260c07140 con 0x563260ca17b0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.277+0000 7fe47aae0640 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 0x563260ceaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.278+0000 7fe4557fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.278+0000 7fe479ade640 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5a70 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe46400cfd0 tx=0x7fe4640077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa42db74eda30aa8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.278+0000 7fe4557fa640 1 -- 172.21.15.45:0/52348293 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46405faf0 con 0x563260ca17b0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.278+0000 7fe4557fa640 1 -- 172.21.15.45:0/52348293 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7fe46405f660 con 0x563260ca17b0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.278+0000 7fe47f1b9740 1 -- 172.21.15.45:0/52348293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe46c00f050 con 0x563260ca17b0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.278+0000 7fe4557fa640 1 -- 172.21.15.45:0/52348293 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe46405f820 con 0x563260ca17b0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.278+0000 7fe47f1b9740 10 client.7320 fetch_fsmap learned FSMap version 44 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.278+0000 7fe47f1b9740 1 -- 172.21.15.45:0/52348293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x563260bca170 con 0x563260ca17b0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.278+0000 7fe4557fa640 1 -- 172.21.15.45:0/52348293 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c00f050 con 0x563260ca17b0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe4557fa640 1 -- 172.21.15.45:0/52348293 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 44) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe464058b20 con 0x563260ca17b0 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe47f1b9740 10 client.7320 fetch_fsmap finished waiting for FSMap version 44 2023-11-28T20:51:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe47f1b9740 10 client.7320 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:51:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe47f1b9740 20 client.7320 populate_metadata read hostname 'smithi045' 2023-11-28T20:51:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] conn(0x563260ceb1e0 0x563260d00dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe47f1b9740 4 client.7320 mds_command: new command op to 7179 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:51:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe47f1b9740 1 -- 172.21.15.45:0/52348293 --> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5632609096c0 con 0x563260ceb1e0 2023-11-28T20:51:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe47a2df640 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] conn(0x563260ceb1e0 0x563260d00dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe47a2df640 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] conn(0x563260ceb1e0 0x563260d00dd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea2b9158fbfc5110 server_cookie=57f78dd41b23dc1c in_seq=0 out_seq=0 2023-11-28T20:51:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.279+0000 7fe4557fa640 10 client.7320 ms_handle_connect on v2:172.21.15.45:6834/862206408 2023-11-28T20:51:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe4557fa640 1 -- 172.21.15.45:0/52348293 <== mds.0 v2:172.21.15.45:6834/862206408 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5632609096c0 con 0x563260ceb1e0 2023-11-28T20:51:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe4557fa640 10 client.7320 handle_command_reply: tid=0 2023-11-28T20:51:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 client.7320 shutdown 2023-11-28T20:51:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 -- 172.21.15.45:0/52348293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 msgr2=0x563260ce5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5a70 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe46400cfd0 tx=0x7fe4640077b0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 -- 172.21.15.45:0/52348293 shutdown_connections 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.45:6834/862206408,v1:172.21.15.45:6835/862206408] conn(0x563260ceb1e0 0x563260d00dd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x563260ca2070 0x563260ceaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563260ca17b0 0x563260ce5a70 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 --2- 172.21.15.45:0/52348293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563260c9cd70 0x563260ce5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe4557fa640 0 client.7320 ms_handle_reset on v2:172.21.15.45:6834/862206408 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 -- 172.21.15.45:0/52348293 >> 172.21.15.45:0/52348293 conn(0x563260c63c70 msgr2=0x563260c9bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 -- 172.21.15.45:0/52348293 shutdown_connections 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 1 -- 172.21.15.45:0/52348293 wait complete. 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.280+0000 7fe47f1b9740 20 client.7320 trim_cache size 0 max 16384 2023-11-28T20:51:20.286 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 7179, 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.8103838200000002, 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 44, 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 102, 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 102, 2023-11-28T20:51:20.287 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 13.098840234000001 2023-11-28T20:51:20.288 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:51:20.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 -- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe45806af60 msgr2=0x7fe45806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe45806af60 0x7fe45806d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe470036000 tx=0x7fe47003c040 comp rx=0 tx=0).stop 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 -- 172.21.15.45:0/1058698403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4740cf610 msgr2=0x7fe4741507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4740cf610 0x7fe4741507b0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c400 tx=0x7fe46400c8c0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 -- 172.21.15.45:0/1058698403 shutdown_connections 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe45806af60 0x7fe45806d420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe474161540 0x7fe474150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4740cf610 0x7fe4741507b0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 --2- 172.21.15.45:0/1058698403 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe47406c1d0 0x7fe4741634f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 -- 172.21.15.45:0/1058698403 >> 172.21.15.45:0/1058698403 conn(0x7fe47406a710 msgr2=0x7fe4740ac430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 -- 172.21.15.45:0/1058698403 shutdown_connections 2023-11-28T20:51:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.281+0000 7fe47b2e1640 1 -- 172.21.15.45:0/1058698403 wait complete. 2023-11-28T20:51:20.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:51:20.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.523+0000 7f9c17148640 1 Processor -- start 2023-11-28T20:51:20.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.523+0000 7f9c17148640 1 -- start start 2023-11-28T20:51:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.525+0000 7f9c17148640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c1015d320 0x7f9c100ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.525+0000 7f9c17148640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9c100b1340 0x7f9c10071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.525+0000 7f9c17148640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c100ce710 con 0x7f9c100add40 2023-11-28T20:51:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.525+0000 7f9c17148640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c100ae5a0 con 0x7f9c100b1340 2023-11-28T20:51:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.525+0000 7f9c17148640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c10160fe0 con 0x7f9c1015d320 2023-11-28T20:51:20.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.526+0000 7f9c14ee4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c1015d320 0x7f9c100ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.526+0000 7f9c14ee4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c1015d320 0x7f9c100ad800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:37524/0 (socket says 172.21.15.45:37524) 2023-11-28T20:51:20.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.526+0000 7f9c14ee4640 1 -- 172.21.15.45:0/2997699112 learned_addr learned my addr 172.21.15.45:0/2997699112 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:20.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.526+0000 7f9c156e5640 1 --2- 172.21.15.45:0/2997699112 >> v2:172.21.15.190:3300/0 conn(0x7f9c100b1340 0x7f9c10071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.527+0000 7f9c14ee4640 1 -- 172.21.15.45:0/2997699112 >> v2:172.21.15.190:3300/0 conn(0x7f9c100b1340 msgr2=0x7f9c10071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.527+0000 7f9c14ee4640 1 --2- 172.21.15.45:0/2997699112 >> v2:172.21.15.190:3300/0 conn(0x7f9c100b1340 0x7f9c10071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.527+0000 7f9c14ee4640 1 -- 172.21.15.45:0/2997699112 >> v1:172.21.15.190:6789/0 conn(0x7f9c100add40 legacy=0x7f9c100b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.527+0000 7f9c14ee4640 1 -- 172.21.15.45:0/2997699112 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c100ae2c0 con 0x7f9c1015d320 2023-11-28T20:51:20.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.528+0000 7f9c14ee4640 1 --2- 172.21.15.45:0/2997699112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c1015d320 0x7f9c100ad800 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009870 tx=0x7f9c0402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=230222283618fec5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.533+0000 7f9c0f7fe640 1 -- 172.21.15.45:0/2997699112 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04030840 con 0x7f9c1015d320 2023-11-28T20:51:20.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.533+0000 7f9c0f7fe640 1 -- 172.21.15.45:0/2997699112 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c040309a0 con 0x7f9c1015d320 2023-11-28T20:51:20.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.533+0000 7f9c0f7fe640 1 -- 172.21.15.45:0/2997699112 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04030d30 con 0x7f9c1015d320 2023-11-28T20:51:20.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.534+0000 7f9c17148640 1 -- 172.21.15.45:0/2997699112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c1015d320 msgr2=0x7f9c100ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.534+0000 7f9c17148640 1 --2- 172.21.15.45:0/2997699112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c1015d320 0x7f9c100ad800 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009870 tx=0x7f9c0402b980 comp rx=0 tx=0).stop 2023-11-28T20:51:20.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.534+0000 7f9c17148640 1 -- 172.21.15.45:0/2997699112 shutdown_connections 2023-11-28T20:51:20.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.534+0000 7f9c17148640 1 --2- 172.21.15.45:0/2997699112 >> v2:172.21.15.190:3300/0 conn(0x7f9c100b1340 0x7f9c10071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.534+0000 7f9c17148640 1 --2- 172.21.15.45:0/2997699112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c1015d320 0x7f9c100ad800 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.534+0000 7f9c17148640 1 -- 172.21.15.45:0/2997699112 >> 172.21.15.45:0/2997699112 conn(0x7f9c1006a500 msgr2=0x7f9c1006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:20.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.534+0000 7f9c17148640 1 -- 172.21.15.45:0/2997699112 shutdown_connections 2023-11-28T20:51:20.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.534+0000 7f9c17148640 1 -- 172.21.15.45:0/2997699112 wait complete. 2023-11-28T20:51:20.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.537+0000 7f9c17148640 1 Processor -- start 2023-11-28T20:51:20.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.537+0000 7f9c17148640 1 -- start start 2023-11-28T20:51:20.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.538+0000 7f9c17148640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c100add40 0x7f9c101518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.538+0000 7f9c17148640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c100b1340 0x7f9c101549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.538+0000 7f9c17148640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c1015d320 0x7f9c10154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.538+0000 7f9c17148640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c101529d0 con 0x7f9c100b1340 2023-11-28T20:51:20.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.538+0000 7f9c17148640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1015e9e0 con 0x7f9c1015d320 2023-11-28T20:51:20.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.538+0000 7f9c17148640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1015ecc0 con 0x7f9c100add40 2023-11-28T20:51:20.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.538+0000 7f9c0ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c100b1340 0x7f9c101549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c0ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c100b1340 0x7f9c101549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44942/0 (socket says 172.21.15.45:44942) 2023-11-28T20:51:20.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c0ffff640 1 -- 172.21.15.45:0/2361607167 learned_addr learned my addr 172.21.15.45:0/2361607167 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:20.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c14ee4640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c100add40 0x7f9c101518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c156e5640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c1015d320 0x7f9c10154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c0ffff640 1 -- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c100add40 msgr2=0x7f9c101518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c0ffff640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c100add40 0x7f9c101518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c0ffff640 1 -- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c1015d320 msgr2=0x7f9c10154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c0ffff640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c1015d320 0x7f9c10154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c0ffff640 1 -- 172.21.15.45:0/2361607167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c040094d0 con 0x7f9c100b1340 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.539+0000 7f9c0ffff640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c100b1340 0x7f9c101549a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000c4d0 tx=0x7f9c0000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.540+0000 7f9c0d7fa640 1 -- 172.21.15.45:0/2361607167 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c000052d0 con 0x7f9c100b1340 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.540+0000 7f9c0d7fa640 1 -- 172.21.15.45:0/2361607167 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c00005430 con 0x7f9c100b1340 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.540+0000 7f9c0d7fa640 1 -- 172.21.15.45:0/2361607167 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00012850 con 0x7f9c100b1340 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.540+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1015efa0 con 0x7f9c100b1340 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.540+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1015e420 con 0x7f9c100b1340 2023-11-28T20:51:20.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.541+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c10152b40 con 0x7f9c100b1340 2023-11-28T20:51:20.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.542+0000 7f9c0d7fa640 1 -- 172.21.15.45:0/2361607167 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9c000129b0 con 0x7f9c100b1340 2023-11-28T20:51:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.543+0000 7f9c0d7fa640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f9bfc06af60 0x7f9bfc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.543+0000 7f9c0d7fa640 1 -- 172.21.15.45:0/2361607167 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f9c00089460 con 0x7f9c100b1340 2023-11-28T20:51:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.544+0000 7f9c14ee4640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f9bfc06af60 0x7f9bfc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:20.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.545+0000 7f9c14ee4640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f9bfc06af60 0x7f9bfc06d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9c04036000 tx=0x7f9c0403b040 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:20.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.548+0000 7f9c0d7fa640 1 -- 172.21.15.45:0/2361607167 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c10152b40 con 0x7f9c100b1340 2023-11-28T20:51:20.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.780+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9c10152b10 con 0x7f9c100b1340 2023-11-28T20:51:20.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.781+0000 7f9c0d7fa640 1 -- 172.21.15.45:0/2361607167 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9c00054200 con 0x7f9c100b1340 2023-11-28T20:51:20.783 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f9bfc06af60 msgr2=0x7f9bfc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f9bfc06af60 0x7f9bfc06d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9c04036000 tx=0x7f9c0403b040 comp rx=0 tx=0).stop 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c100b1340 msgr2=0x7f9c101549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c100b1340 0x7f9c101549a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000c4d0 tx=0x7f9c0000c990 comp rx=0 tx=0).stop 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 shutdown_connections 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f9bfc06af60 0x7f9bfc06d420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c1015d320 0x7f9c10154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c100b1340 0x7f9c101549a0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 --2- 172.21.15.45:0/2361607167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c100add40 0x7f9c101518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.783+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 >> 172.21.15.45:0/2361607167 conn(0x7f9c1006a500 msgr2=0x7f9c100ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:20.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.784+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 shutdown_connections 2023-11-28T20:51:20.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:20.784+0000 7f9c17148640 1 -- 172.21.15.45:0/2361607167 wait complete. 2023-11-28T20:51:20.805 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T20:51:20.805 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:51:20.805 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:51:20.806 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T20:51:20.806 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:51:20.806 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T20:51:20.806 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T20:51:20.806 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:51:20.806 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:51:20.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:20.807 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:51:20.863 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:51:20.863 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:51:20.863 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:51:20.864 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:51:20.865 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:51:20.865 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:51:20.865 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:51:20.865 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 41262sec preferred_lft 41262sec 2023-11-28T20:51:20.865 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:51:20.865 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:51:20.866 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:51:20.866 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:51:20.866 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:51:20.866 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:51:20.866 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T20:51:20.866 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:51:20.866 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:51:20.866 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T20:51:20.867 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:51:20.867 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:51:20.867 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:51:20.867 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T20:51:20.867 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:51:20.867 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:51:20.867 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T20:51:20.867 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T20:51:20.867 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:51:20.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:20.868 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:51:20.970 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:51:20.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:20.975 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:51:21.034 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:51:21.035 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:51:21.035 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:51:21.035 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:51:21.035 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:51:21.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:21.035 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:51:21.035 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:51:21.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:51:21.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:51:21.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:51:21.036 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:51:21.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:21.238 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:51:21.295 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T20:51:21.295 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T20:51:21.296 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:51:21.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:21.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:51:21.403 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:51:21.403 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T20:51:21.462 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:51:21.463 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:51:21.463 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:51:21.523 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:51:21.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:21.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:51:21.675 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:51:21.676 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:51:21.676 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:51:21.701 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:51:21.701 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:51:21.701 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:51:21.701 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:51:21.701 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:51:21.730 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:51:21.730 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:51:21.730 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:51:21.730 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:51:21.730 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:51:21.730 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:51:21.730 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:51:21.731 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:51:21.734 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T20:51:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:51:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:51:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:51:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:51:21.739 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:51:21.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:21.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:51:21.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:21.828 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:51:21.900 INFO:teuthology.orchestra.run.smithi045.stdout:client6214 2023-11-28T20:51:21.901 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:51:21.901 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:21.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T20:51:22.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.185+0000 7feeea02e640 1 Processor -- start 2023-11-28T20:51:22.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.185+0000 7feeea02e640 1 -- start start 2023-11-28T20:51:22.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.187+0000 7feeea02e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feee415afc0 0x7feee406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.187+0000 7feeea02e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee406bc20 0x7feee415b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.187+0000 7feeea02e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee406d770 con 0x7feee40af300 2023-11-28T20:51:22.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.187+0000 7feeea02e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee40af790 con 0x7feee415afc0 2023-11-28T20:51:22.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.187+0000 7feeea02e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee40af160 con 0x7feee406bc20 2023-11-28T20:51:22.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.187+0000 7feee902c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feee415afc0 0x7feee406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.187+0000 7feee902c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feee415afc0 0x7feee406f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44952/0 (socket says 172.21.15.45:44952) 2023-11-28T20:51:22.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.187+0000 7feee902c640 1 -- 172.21.15.45:0/2172665555 learned_addr learned my addr 172.21.15.45:0/2172665555 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:22.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.188+0000 7feee982d640 1 --2- 172.21.15.45:0/2172665555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee406bc20 0x7feee415b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.188+0000 7feee902c640 1 -- 172.21.15.45:0/2172665555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee406bc20 msgr2=0x7feee415b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:22.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.188+0000 7feee902c640 1 --2- 172.21.15.45:0/2172665555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee406bc20 0x7feee415b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.188+0000 7feee902c640 1 -- 172.21.15.45:0/2172665555 >> v1:172.21.15.190:6789/0 conn(0x7feee40af300 legacy=0x7feee406b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:22.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.188+0000 7feee902c640 1 -- 172.21.15.45:0/2172665555 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee406d160 con 0x7feee415afc0 2023-11-28T20:51:22.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.190+0000 7feee902c640 1 --2- 172.21.15.45:0/2172665555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 0x7feee406f840 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7feee0009e50 tx=0x7feee002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b4c978815b29bc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.190+0000 7feedbfff640 1 -- 172.21.15.45:0/2172665555 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee002ca70 con 0x7feee415afc0 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.190+0000 7feedbfff640 1 -- 172.21.15.45:0/2172665555 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee002cbd0 con 0x7feee415afc0 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.190+0000 7feedbfff640 1 -- 172.21.15.45:0/2172665555 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee00343f0 con 0x7feee415afc0 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.190+0000 7feeea02e640 1 -- 172.21.15.45:0/2172665555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 msgr2=0x7feee406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.190+0000 7feeea02e640 1 --2- 172.21.15.45:0/2172665555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 0x7feee406f840 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7feee0009e50 tx=0x7feee002bed0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.191+0000 7feeea02e640 1 -- 172.21.15.45:0/2172665555 shutdown_connections 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.191+0000 7feeea02e640 1 --2- 172.21.15.45:0/2172665555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee406bc20 0x7feee415b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.191+0000 7feeea02e640 1 --2- 172.21.15.45:0/2172665555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 0x7feee406f840 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.191+0000 7feeea02e640 1 -- 172.21.15.45:0/2172665555 >> 172.21.15.45:0/2172665555 conn(0x7feee4071220 msgr2=0x7feee4071630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.191+0000 7feeea02e640 1 -- 172.21.15.45:0/2172665555 shutdown_connections 2023-11-28T20:51:22.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.191+0000 7feeea02e640 1 -- 172.21.15.45:0/2172665555 wait complete. 2023-11-28T20:51:22.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.193+0000 7feeea02e640 1 Processor -- start 2023-11-28T20:51:22.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.193+0000 7feeea02e640 1 -- start start 2023-11-28T20:51:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feeea02e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee406bc20 0x7feee414c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feeea02e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee40af300 0x7feee414fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feeea02e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 0x7feee4150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feeea02e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee4151e10 con 0x7feee415afc0 2023-11-28T20:51:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feeea02e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee41520f0 con 0x7feee406bc20 2023-11-28T20:51:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feeea02e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee41523d0 con 0x7feee40af300 2023-11-28T20:51:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feee982d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 0x7feee4150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feee982d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 0x7feee4150e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44954/0 (socket says 172.21.15.45:44954) 2023-11-28T20:51:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feee982d640 1 -- 172.21.15.45:0/2998246118 learned_addr learned my addr 172.21.15.45:0/2998246118 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:22.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feee902c640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee406bc20 0x7feee414c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.194+0000 7feee882b640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee40af300 0x7feee414fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feee982d640 1 -- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee40af300 msgr2=0x7feee414fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feee982d640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee40af300 0x7feee414fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feee982d640 1 -- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee406bc20 msgr2=0x7feee414c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feee982d640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee406bc20 0x7feee414c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feee982d640 1 -- 172.21.15.45:0/2998246118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0009b30 con 0x7feee415afc0 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feee982d640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 0x7feee4150e70 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7feedc00d5d0 tx=0x7feedc00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feed9ffb640 1 -- 172.21.15.45:0/2998246118 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc0125f0 con 0x7feee415afc0 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feed9ffb640 1 -- 172.21.15.45:0/2998246118 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feedc012750 con 0x7feee415afc0 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feed9ffb640 1 -- 172.21.15.45:0/2998246118 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc012b00 con 0x7feee415afc0 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee40b2ea0 con 0x7feee415afc0 2023-11-28T20:51:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.195+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee414cc50 con 0x7feee415afc0 2023-11-28T20:51:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.197+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee40ad390 con 0x7feee415afc0 2023-11-28T20:51:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.198+0000 7feed9ffb640 1 -- 172.21.15.45:0/2998246118 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7feedc012c60 con 0x7feee415afc0 2023-11-28T20:51:22.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.198+0000 7feed9ffb640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7feec806af60 0x7feec806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.199+0000 7feed9ffb640 1 -- 172.21.15.45:0/2998246118 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7feedc089f10 con 0x7feee415afc0 2023-11-28T20:51:22.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.199+0000 7feee902c640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7feec806af60 0x7feec806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.200+0000 7feee902c640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7feec806af60 0x7feec806d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feee0035000 tx=0x7feee0030000 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.203+0000 7feed9ffb640 1 -- 172.21.15.45:0/2998246118 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feee40ad390 con 0x7feee415afc0 2023-11-28T20:51:22.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.420+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 --> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7feee40aeb30 con 0x7feec806af60 2023-11-28T20:51:22.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.424+0000 7feed9ffb640 1 -- 172.21.15.45:0/2998246118 <== mgr.5959 v2:172.21.15.190:6832/970930168 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7feee0044080 con 0x7feec806af60 2023-11-28T20:51:22.426 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:51:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T20:51:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:22.426 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:51:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.426+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7feec806af60 msgr2=0x7feec806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.426+0000 7feeea02e640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7feec806af60 0x7feec806d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feee0035000 tx=0x7feee0030000 comp rx=0 tx=0).stop 2023-11-28T20:51:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.426+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 msgr2=0x7feee4150e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.426+0000 7feeea02e640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 0x7feee4150e70 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7feedc00d5d0 tx=0x7feedc00da90 comp rx=0 tx=0).stop 2023-11-28T20:51:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.426+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 shutdown_connections 2023-11-28T20:51:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.426+0000 7feeea02e640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7feec806af60 0x7feec806d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.427+0000 7feeea02e640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee415afc0 0x7feee4150e70 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.427+0000 7feeea02e640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee40af300 0x7feee414fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.427+0000 7feeea02e640 1 --2- 172.21.15.45:0/2998246118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee406bc20 0x7feee414c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.427+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 >> 172.21.15.45:0/2998246118 conn(0x7feee4071220 msgr2=0x7feee40b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.427+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 shutdown_connections 2023-11-28T20:51:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.427+0000 7feeea02e640 1 -- 172.21.15.45:0/2998246118 wait complete. 2023-11-28T20:51:22.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T20:51:22.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.678+0000 7f2c95743640 1 Processor -- start 2023-11-28T20:51:22.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.678+0000 7f2c95743640 1 -- start start 2023-11-28T20:51:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c95743640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 0x7f2c900ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c95743640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c900abd70 0x7f2c900718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c95743640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2c90071e00 0x7f2c9015eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c95743640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c900722b0 con 0x7f2c900abd70 2023-11-28T20:51:22.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c95743640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c900ac270 con 0x7f2c90071e00 2023-11-28T20:51:22.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c95743640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9015b830 con 0x7f2c9015b080 2023-11-28T20:51:22.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c87fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c900abd70 0x7f2c900718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c87fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c900abd70 0x7f2c900718c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44960/0 (socket says 172.21.15.45:44960) 2023-11-28T20:51:22.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c87fff640 1 -- 172.21.15.45:0/2613101577 learned_addr learned my addr 172.21.15.45:0/2613101577 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.680+0000 7f2c94f42640 1 --2- 172.21.15.45:0/2613101577 >> v2:172.21.15.190:3300/0 conn(0x7f2c90071e00 0x7f2c9015eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.681+0000 7f2c8ffff640 1 --2- 172.21.15.45:0/2613101577 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 0x7f2c900ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.681+0000 7f2c87fff640 1 -- 172.21.15.45:0/2613101577 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 msgr2=0x7f2c900ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.681+0000 7f2c87fff640 1 --2- 172.21.15.45:0/2613101577 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 0x7f2c900ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.681+0000 7f2c87fff640 1 -- 172.21.15.45:0/2613101577 >> v2:172.21.15.190:3300/0 conn(0x7f2c90071e00 msgr2=0x7f2c9015eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.681+0000 7f2c87fff640 1 --2- 172.21.15.45:0/2613101577 >> v2:172.21.15.190:3300/0 conn(0x7f2c90071e00 0x7f2c9015eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.681+0000 7f2c87fff640 1 -- 172.21.15.45:0/2613101577 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9015bb10 con 0x7f2c900abd70 2023-11-28T20:51:22.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.682+0000 7f2c87fff640 1 --2- 172.21.15.45:0/2613101577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c900abd70 0x7f2c900718c0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2c9006e6d0 tx=0x7f2c78034120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dbabf2e07d123a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:22.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.682+0000 7f2c8f7fe640 1 -- 172.21.15.45:0/2613101577 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7803c440 con 0x7f2c900abd70 2023-11-28T20:51:22.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.682+0000 7f2c8f7fe640 1 -- 172.21.15.45:0/2613101577 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7803c5a0 con 0x7f2c900abd70 2023-11-28T20:51:22.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.682+0000 7f2c8f7fe640 1 -- 172.21.15.45:0/2613101577 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7803c930 con 0x7f2c900abd70 2023-11-28T20:51:22.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.683+0000 7f2c95743640 1 -- 172.21.15.45:0/2613101577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c900abd70 msgr2=0x7f2c900718c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:22.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.683+0000 7f2c95743640 1 --2- 172.21.15.45:0/2613101577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c900abd70 0x7f2c900718c0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2c9006e6d0 tx=0x7f2c78034120 comp rx=0 tx=0).stop 2023-11-28T20:51:22.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.683+0000 7f2c95743640 1 -- 172.21.15.45:0/2613101577 shutdown_connections 2023-11-28T20:51:22.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.683+0000 7f2c95743640 1 --2- 172.21.15.45:0/2613101577 >> v2:172.21.15.190:3300/0 conn(0x7f2c90071e00 0x7f2c9015eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.683+0000 7f2c95743640 1 --2- 172.21.15.45:0/2613101577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c900abd70 0x7f2c900718c0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.683+0000 7f2c95743640 1 --2- 172.21.15.45:0/2613101577 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 0x7f2c900ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.683+0000 7f2c95743640 1 -- 172.21.15.45:0/2613101577 >> 172.21.15.45:0/2613101577 conn(0x7f2c900cecd0 msgr2=0x7f2c900cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:22.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.683+0000 7f2c95743640 1 -- 172.21.15.45:0/2613101577 shutdown_connections 2023-11-28T20:51:22.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.683+0000 7f2c95743640 1 -- 172.21.15.45:0/2613101577 wait complete. 2023-11-28T20:51:22.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.685+0000 7f2c95743640 1 Processor -- start 2023-11-28T20:51:22.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.685+0000 7f2c95743640 1 -- start start 2023-11-28T20:51:22.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.686+0000 7f2c95743640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c90071e00 0x7f2c90154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.686+0000 7f2c95743640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c900abd70 0x7f2c901515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.686+0000 7f2c95743640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 0x7f2c9009b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.686+0000 7f2c95743640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c901536a0 con 0x7f2c90071e00 2023-11-28T20:51:22.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.686+0000 7f2c95743640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c901538f0 con 0x7f2c900abd70 2023-11-28T20:51:22.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.686+0000 7f2c95743640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c90153bd0 con 0x7f2c9015b080 2023-11-28T20:51:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.686+0000 7f2c8ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c90071e00 0x7f2c90154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c8ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c90071e00 0x7f2c90154990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44970/0 (socket says 172.21.15.45:44970) 2023-11-28T20:51:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c8ffff640 1 -- 172.21.15.45:0/3858880944 learned_addr learned my addr 172.21.15.45:0/3858880944 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c94f42640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 0x7f2c9009b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c87fff640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c900abd70 0x7f2c901515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c8ffff640 1 -- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 msgr2=0x7f2c9009b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c8ffff640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 0x7f2c9009b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c8ffff640 1 -- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c900abd70 msgr2=0x7f2c901515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:22.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c8ffff640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c900abd70 0x7f2c901515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:22.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c8ffff640 1 -- 172.21.15.45:0/3858880944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c780094d0 con 0x7f2c90071e00 2023-11-28T20:51:22.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.687+0000 7f2c8ffff640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c90071e00 0x7f2c90154990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000d860 tx=0x7f2c8000dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:22.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.688+0000 7f2c8d7fa640 1 -- 172.21.15.45:0/3858880944 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8000de50 con 0x7f2c90071e00 2023-11-28T20:51:22.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.688+0000 7f2c8d7fa640 1 -- 172.21.15.45:0/3858880944 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8000c940 con 0x7f2c90071e00 2023-11-28T20:51:22.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.688+0000 7f2c8d7fa640 1 -- 172.21.15.45:0/3858880944 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8000ccf0 con 0x7f2c90071e00 2023-11-28T20:51:22.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.688+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c90153eb0 con 0x7f2c90071e00 2023-11-28T20:51:22.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.688+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c90154470 con 0x7f2c90071e00 2023-11-28T20:51:22.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.689+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9009d0f0 con 0x7f2c90071e00 2023-11-28T20:51:22.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.690+0000 7f2c8d7fa640 1 -- 172.21.15.45:0/3858880944 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 38) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2c80015410 con 0x7f2c90071e00 2023-11-28T20:51:22.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.691+0000 7f2c8d7fa640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2c7006af60 0x7f2c7006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:22.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.692+0000 7f2c8d7fa640 1 -- 172.21.15.45:0/3858880944 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f2c80089180 con 0x7f2c90071e00 2023-11-28T20:51:22.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.692+0000 7f2c87fff640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2c7006af60 0x7f2c7006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:22.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.693+0000 7f2c87fff640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2c7006af60 0x7f2c7006d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c7803e000 tx=0x7f2c78038000 comp rx=0 tx=0).ready entity=mgr.5959 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:22.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.696+0000 7f2c8d7fa640 1 -- 172.21.15.45:0/3858880944 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c80053f50 con 0x7f2c90071e00 2023-11-28T20:51:22.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:22.938+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f2c9004e920 con 0x7f2c90071e00 2023-11-28T20:51:23.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.553+0000 7f2c8d7fa640 1 -- 172.21.15.45:0/3858880944 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v39) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2c80057570 con 0x7f2c90071e00 2023-11-28T20:51:23.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.556+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2c7006af60 msgr2=0x7f2c7006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:23.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.556+0000 7f2c95743640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2c7006af60 0x7f2c7006d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c7803e000 tx=0x7f2c78038000 comp rx=0 tx=0).stop 2023-11-28T20:51:23.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.556+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c90071e00 msgr2=0x7f2c90154990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:23.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.556+0000 7f2c95743640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c90071e00 0x7f2c90154990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000d860 tx=0x7f2c8000dd20 comp rx=0 tx=0).stop 2023-11-28T20:51:23.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.557+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 shutdown_connections 2023-11-28T20:51:23.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.557+0000 7f2c95743640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2c7006af60 0x7f2c7006d420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.557+0000 7f2c95743640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9015b080 0x7f2c9009b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.557+0000 7f2c95743640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c900abd70 0x7f2c901515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.557+0000 7f2c95743640 1 --2- 172.21.15.45:0/3858880944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c90071e00 0x7f2c90154990 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.557+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 >> 172.21.15.45:0/3858880944 conn(0x7f2c900cecd0 msgr2=0x7f2c900afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:23.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.557+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 shutdown_connections 2023-11-28T20:51:23.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.557+0000 7f2c95743640 1 -- 172.21.15.45:0/3858880944 wait complete. 2023-11-28T20:51:23.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T20:51:23.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.789+0000 7f2a53b78640 1 Processor -- start 2023-11-28T20:51:23.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.789+0000 7f2a53b78640 1 -- start start 2023-11-28T20:51:23.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.791+0000 7f2a53b78640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c15e840 0x7f2a4c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:23.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.791+0000 7f2a53b78640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a4c15f720 0x7f2a4c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:23.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.791+0000 7f2a53b78640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2a4c06a500 0x7f2a4c06a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:23.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.791+0000 7f2a53b78640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c071ce0 con 0x7f2a4c15f720 2023-11-28T20:51:23.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.791+0000 7f2a53b78640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c06aed0 con 0x7f2a4c06a500 2023-11-28T20:51:23.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.791+0000 7f2a53b78640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c15ecd0 con 0x7f2a4c15e840 2023-11-28T20:51:23.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.791+0000 7f2a51113640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a4c15f720 0x7f2a4c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:23.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.792+0000 7f2a51113640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a4c15f720 0x7f2a4c0717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44998/0 (socket says 172.21.15.45:44998) 2023-11-28T20:51:23.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.792+0000 7f2a51113640 1 -- 172.21.15.45:0/54875204 learned_addr learned my addr 172.21.15.45:0/54875204 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.792+0000 7f2a51914640 1 --2- 172.21.15.45:0/54875204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c15e840 0x7f2a4c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.792+0000 7f2a52115640 1 --2- 172.21.15.45:0/54875204 >> v2:172.21.15.190:3300/0 conn(0x7f2a4c06a500 0x7f2a4c06a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.792+0000 7f2a51113640 1 -- 172.21.15.45:0/54875204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c15e840 msgr2=0x7f2a4c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.792+0000 7f2a51113640 1 --2- 172.21.15.45:0/54875204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c15e840 0x7f2a4c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.792+0000 7f2a51113640 1 -- 172.21.15.45:0/54875204 >> v2:172.21.15.190:3300/0 conn(0x7f2a4c06a500 msgr2=0x7f2a4c06a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.792+0000 7f2a51113640 1 --2- 172.21.15.45:0/54875204 >> v2:172.21.15.190:3300/0 conn(0x7f2a4c06a500 0x7f2a4c06a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.792+0000 7f2a51113640 1 -- 172.21.15.45:0/54875204 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c15f290 con 0x7f2a4c15f720 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.793+0000 7f2a51113640 1 --2- 172.21.15.45:0/54875204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15f720 0x7f2a4c0717a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c009870 tx=0x7f2a3c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd96eeffaa13d65 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.794+0000 7f2a50912640 1 -- 172.21.15.45:0/54875204 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c031640 con 0x7f2a4c15f720 2023-11-28T20:51:23.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.794+0000 7f2a50912640 1 -- 172.21.15.45:0/54875204 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c0317a0 con 0x7f2a4c15f720 2023-11-28T20:51:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.794+0000 7f2a50912640 1 -- 172.21.15.45:0/54875204 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c031b30 con 0x7f2a4c15f720 2023-11-28T20:51:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.794+0000 7f2a53b78640 1 -- 172.21.15.45:0/54875204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15f720 msgr2=0x7f2a4c0717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.794+0000 7f2a53b78640 1 --2- 172.21.15.45:0/54875204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15f720 0x7f2a4c0717a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c009870 tx=0x7f2a3c02b8f0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.795+0000 7f2a53b78640 1 -- 172.21.15.45:0/54875204 shutdown_connections 2023-11-28T20:51:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.795+0000 7f2a53b78640 1 --2- 172.21.15.45:0/54875204 >> v2:172.21.15.190:3300/0 conn(0x7f2a4c06a500 0x7f2a4c06a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.795+0000 7f2a53b78640 1 --2- 172.21.15.45:0/54875204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15f720 0x7f2a4c0717a0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.795+0000 7f2a53b78640 1 --2- 172.21.15.45:0/54875204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c15e840 0x7f2a4c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.795+0000 7f2a53b78640 1 -- 172.21.15.45:0/54875204 >> 172.21.15.45:0/54875204 conn(0x7f2a4c0ced60 msgr2=0x7f2a4c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.795+0000 7f2a53b78640 1 -- 172.21.15.45:0/54875204 shutdown_connections 2023-11-28T20:51:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.795+0000 7f2a53b78640 1 -- 172.21.15.45:0/54875204 wait complete. 2023-11-28T20:51:23.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.796+0000 7f2a53b78640 1 Processor -- start 2023-11-28T20:51:23.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.797+0000 7f2a53b78640 1 -- start start 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a53b78640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c06a500 0x7f2a4c0b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a53b78640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15e840 0x7f2a4c156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a53b78640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a4c15f720 0x7f2a4c0c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a53b78640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c7130 con 0x7f2a4c15e840 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a53b78640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c7410 con 0x7f2a4c15f720 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a53b78640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c76f0 con 0x7f2a4c06a500 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51113640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15e840 0x7f2a4c156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51113640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15e840 0x7f2a4c156400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45006/0 (socket says 172.21.15.45:45006) 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51113640 1 -- 172.21.15.45:0/1476883639 learned_addr learned my addr 172.21.15.45:0/1476883639 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51914640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c06a500 0x7f2a4c0b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a52115640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a4c15f720 0x7f2a4c0c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51113640 1 -- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c06a500 msgr2=0x7f2a4c0b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51113640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c06a500 0x7f2a4c0b19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51113640 1 -- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a4c15f720 msgr2=0x7f2a4c0c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51113640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a4c15f720 0x7f2a4c0c81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51113640 1 -- 172.21.15.45:0/1476883639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c0094d0 con 0x7f2a4c15e840 2023-11-28T20:51:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.798+0000 7f2a51914640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c06a500 0x7f2a4c0b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.799+0000 7f2a51113640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15e840 0x7f2a4c156400 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c000c00 tx=0x7f2a3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.799+0000 7f2a427fc640 1 -- 172.21.15.45:0/1476883639 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c03ac60 con 0x7f2a4c15e840 2023-11-28T20:51:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.799+0000 7f2a427fc640 1 -- 172.21.15.45:0/1476883639 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c03adc0 con 0x7f2a4c15e840 2023-11-28T20:51:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.799+0000 7f2a427fc640 1 -- 172.21.15.45:0/1476883639 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c044630 con 0x7f2a4c15e840 2023-11-28T20:51:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.799+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4c0c79d0 con 0x7f2a4c15e840 2023-11-28T20:51:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.799+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4c0b1ee0 con 0x7f2a4c15e840 2023-11-28T20:51:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.800+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a4c0c7ef0 con 0x7f2a4c15e840 2023-11-28T20:51:23.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.802+0000 7f2a427fc640 1 -- 172.21.15.45:0/1476883639 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 39) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f2a3c02f050 con 0x7f2a4c15e840 2023-11-28T20:51:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.802+0000 7f2a427fc640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2a3006afb0 0x7f2a3006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.803+0000 7f2a427fc640 1 -- 172.21.15.45:0/1476883639 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f2a3c0ab930 con 0x7f2a4c15e840 2023-11-28T20:51:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.803+0000 7f2a51914640 1 -- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2a3006afb0 msgr2=0x7f2a3006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/970930168 2023-11-28T20:51:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.803+0000 7f2a51914640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2a3006afb0 0x7f2a3006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:23.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:23.811+0000 7f2a427fc640 1 -- 172.21.15.45:0/1476883639 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a4c0c7ef0 con 0x7f2a4c15e840 2023-11-28T20:51:24.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.003+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2a4c0c7ef0 con 0x7f2a4c15e840 2023-11-28T20:51:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.004+0000 7f2a51914640 1 -- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2a3006afb0 msgr2=0x7f2a3006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/970930168 2023-11-28T20:51:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.004+0000 7f2a51914640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2a3006afb0 0x7f2a3006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:51:24.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.010+0000 7f2a427fc640 1 -- 172.21.15.45:0/1476883639 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2a4c0c7ef0 con 0x7f2a4c15e840 2023-11-28T20:51:24.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.013+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2a3006afb0 msgr2=0x7f2a3006d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:24.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.013+0000 7f2a53b78640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2a3006afb0 0x7f2a3006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15e840 msgr2=0x7f2a4c156400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15e840 0x7f2a4c156400 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c000c00 tx=0x7f2a3c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 shutdown_connections 2023-11-28T20:51:24.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f2a3006afb0 0x7f2a3006d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a4c15f720 0x7f2a4c0c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a4c15e840 0x7f2a4c156400 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 --2- 172.21.15.45:0/1476883639 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a4c06a500 0x7f2a4c0b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 >> 172.21.15.45:0/1476883639 conn(0x7f2a4c0ced60 msgr2=0x7f2a4c159100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:24.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 shutdown_connections 2023-11-28T20:51:24.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.014+0000 7f2a53b78640 1 -- 172.21.15.45:0/1476883639 wait complete. 2023-11-28T20:51:24.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T20:51:24.096 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:24.094+0000 7f6cf7232040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:51:24.123 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:24.121+0000 7f2a28a2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:51:24.148 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:24.146+0000 7f6cf7232040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:51:24.180 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:24.178+0000 7f2a28a2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:51:24.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.264+0000 7f067765d640 1 Processor -- start 2023-11-28T20:51:24.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.265+0000 7f067765d640 1 -- start start 2023-11-28T20:51:24.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.266+0000 7f067765d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f067006d880 0x7f067006b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.266+0000 7f067765d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067006a5a0 con 0x7f06700ca500 2023-11-28T20:51:24.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.266+0000 7f067765d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06700caa60 con 0x7f06700cc270 2023-11-28T20:51:24.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.266+0000 7f067765d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067006a810 con 0x7f067006d880 2023-11-28T20:51:24.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.266+0000 7f0675bfa640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f06700ca500 0x7f067006b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35370/0 (socket says 172.21.15.45:35370) 2023-11-28T20:51:24.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.266+0000 7f0675bfa640 1 -- 172.21.15.45:0/2153143080 learned_addr learned my addr 172.21.15.45:0/2153143080 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:24.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0674bf8640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f067006d880 0x7f067006b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136050224 0 0) 0x7f067006a5a0 con 0x7f06700ca500 2023-11-28T20:51:24.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f065c003520 con 0x7f06700ca500 2023-11-28T20:51:24.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2170852235 0 0) 0x7f065c003520 con 0x7f06700ca500 2023-11-28T20:51:24.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f067006d880 msgr2=0x7f067006b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:24.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f067006d880 0x7f067006b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 >> v1:172.21.15.190:6789/0 conn(0x7f06700cc270 legacy=0x7f067006d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067006aaf0 con 0x7f06700ca500 2023-11-28T20:51:24.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f0668001670 con 0x7f06700ca500 2023-11-28T20:51:24.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 >> v1:172.21.15.45:6789/0 conn(0x7f06700ca500 legacy=0x7f067006b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f065c0039a0 0x7f065c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f065c004300 0x7f065c0047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.267+0000 7f0667fff640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f065c005080 0x7f065c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067006a810 con 0x7f065c005080 2023-11-28T20:51:24.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c00e1b0 con 0x7f065c004300 2023-11-28T20:51:24.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c00e370 con 0x7f065c0039a0 2023-11-28T20:51:24.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0674bf8640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f067006d880 0x7f067006b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:51:24.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0675bfa640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f065c005080 0x7f065c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f06753f9640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f065c0039a0 0x7f065c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0674bf8640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f065c004300 0x7f065c0047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0675bfa640 1 -- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f065c0039a0 msgr2=0x7f065c003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:24.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0675bfa640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f065c0039a0 0x7f065c003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0675bfa640 1 -- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f065c004300 msgr2=0x7f065c0047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:24.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0675bfa640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f065c004300 0x7f065c0047e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.268+0000 7f0675bfa640 1 -- 172.21.15.45:0/2153143080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c003520 con 0x7f065c005080 2023-11-28T20:51:24.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.270+0000 7f0675bfa640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f065c005080 0x7f065c00b980 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f06700ca990 tx=0x7f065c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541e224fbc463802 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:24.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.270+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668005ad0 con 0x7f065c005080 2023-11-28T20:51:24.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.270+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0668005c50 con 0x7f065c005080 2023-11-28T20:51:24.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.270+0000 7f0667fff640 1 -- 172.21.15.45:0/2153143080 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066802c890 con 0x7f065c005080 2023-11-28T20:51:24.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.271+0000 7f067765d640 1 -- 172.21.15.45:0/2153143080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f065c005080 msgr2=0x7f065c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.271+0000 7f067765d640 1 --2- 172.21.15.45:0/2153143080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f065c005080 0x7f065c00b980 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f06700ca990 tx=0x7f065c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.271+0000 7f0675bfa640 1 -- 172.21.15.45:0/2153143080 reap_dead start 2023-11-28T20:51:24.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.271+0000 7f067765d640 1 -- 172.21.15.45:0/2153143080 shutdown_connections 2023-11-28T20:51:24.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.271+0000 7f067765d640 1 -- 172.21.15.45:0/2153143080 >> 172.21.15.45:0/2153143080 conn(0x7f06700cc930 msgr2=0x7f06700ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:24.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.271+0000 7f067765d640 1 -- 172.21.15.45:0/2153143080 shutdown_connections 2023-11-28T20:51:24.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.271+0000 7f067765d640 1 -- 172.21.15.45:0/2153143080 wait complete. 2023-11-28T20:51:24.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.273+0000 7f067765d640 1 Processor -- start 2023-11-28T20:51:24.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.273+0000 7f067765d640 1 -- start start 2023-11-28T20:51:24.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f067765d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f065c005080 0x7f06700b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f067765d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06700a07b0 0x7f06700b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f067765d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06701f3540 0x7f06700afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f067765d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06700b5470 con 0x7f06700a07b0 2023-11-28T20:51:24.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f067765d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06700b5750 con 0x7f065c005080 2023-11-28T20:51:24.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f067765d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06700b5a30 con 0x7f06701f3540 2023-11-28T20:51:24.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f0674bf8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06700a07b0 0x7f06700b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f0674bf8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06700a07b0 0x7f06700b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32906/0 (socket says 172.21.15.45:32906) 2023-11-28T20:51:24.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f0674bf8640 1 -- 172.21.15.45:0/775942479 learned_addr learned my addr 172.21.15.45:0/775942479 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:24.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f06753f9640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f065c005080 0x7f06700b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.274+0000 7f0675bfa640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06701f3540 0x7f06700afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0674bf8640 1 -- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06701f3540 msgr2=0x7f06700afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0674bf8640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06701f3540 0x7f06700afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0674bf8640 1 -- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f065c005080 msgr2=0x7f06700b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0674bf8640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f065c005080 0x7f06700b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0674bf8640 1 -- 172.21.15.45:0/775942479 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06701f3020 con 0x7f06700a07b0 2023-11-28T20:51:24.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0675bfa640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06701f3540 0x7f06700afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:24.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0674bf8640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06700a07b0 0x7f06700b3360 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f0660010800 tx=0x7f0660010cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:24.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0665ffb640 1 -- 172.21.15.45:0/775942479 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066000eb60 con 0x7f06700a07b0 2023-11-28T20:51:24.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0665ffb640 1 -- 172.21.15.45:0/775942479 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066000ecc0 con 0x7f06700a07b0 2023-11-28T20:51:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f0665ffb640 1 -- 172.21.15.45:0/775942479 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066001c400 con 0x7f06700a07b0 2023-11-28T20:51:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06701f2d30 con 0x7f06700a07b0 2023-11-28T20:51:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.275+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06700a0c30 con 0x7f06700a07b0 2023-11-28T20:51:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.277+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06700733d0 con 0x7f06700a07b0 2023-11-28T20:51:24.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.278+0000 7f0665ffb640 1 -- 172.21.15.45:0/775942479 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 39) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f066001c560 con 0x7f06700a07b0 2023-11-28T20:51:24.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.279+0000 7f0665ffb640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f065406afb0 0x7f065406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.280+0000 7f0665ffb640 1 -- 172.21.15.45:0/775942479 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f0660089fd0 con 0x7f06700a07b0 2023-11-28T20:51:24.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.280+0000 7f06753f9640 1 -- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f065406afb0 msgr2=0x7f065406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/970930168 2023-11-28T20:51:24.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.280+0000 7f06753f9640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f065406afb0 0x7f065406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:24.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.284+0000 7f0665ffb640 1 -- 172.21.15.45:0/775942479 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0660054ea0 con 0x7f06700a07b0 2023-11-28T20:51:24.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.476+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f06701f31b0 con 0x7f06700a07b0 2023-11-28T20:51:24.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.481+0000 7f06753f9640 1 -- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f065406afb0 msgr2=0x7f065406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/970930168 2023-11-28T20:51:24.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.481+0000 7f06753f9640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f065406afb0 0x7f065406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:51:24.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.481+0000 7f0665ffb640 1 -- 172.21.15.45:0/775942479 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f06600584c0 con 0x7f06700a07b0 2023-11-28T20:51:24.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f065406afb0 msgr2=0x7f065406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:24.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f065406afb0 0x7f065406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06700a07b0 msgr2=0x7f06700b3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06700a07b0 0x7f06700b3360 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f0660010800 tx=0x7f0660010cc0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 shutdown_connections 2023-11-28T20:51:24.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7f065406afb0 0x7f065406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06701f3540 0x7f06700afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06700a07b0 0x7f06700b3360 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 --2- 172.21.15.45:0/775942479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f065c005080 0x7f06700b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 >> 172.21.15.45:0/775942479 conn(0x7f06700cc930 msgr2=0x7f067006d170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:24.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 shutdown_connections 2023-11-28T20:51:24.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.484+0000 7f067765d640 1 -- 172.21.15.45:0/775942479 wait complete. 2023-11-28T20:51:24.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-11-28T20:51:24.598 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:24.596+0000 7f6cf7232040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:51:24.649 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:24.647+0000 7f6cf7232040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:51:24.662 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:24.660+0000 7f2a28a2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:51:24.699 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:24.697+0000 7f6cf7232040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:51:24.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.710+0000 7fe707fff640 1 Processor -- start 2023-11-28T20:51:24.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.710+0000 7fe707fff640 1 -- start start 2023-11-28T20:51:24.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.712+0000 7fe707fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 0x7fe7080cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.712+0000 7fe707fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7080ce3a0 0x7fe708070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.712+0000 7fe707fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7080ce800 con 0x7fe708071010 2023-11-28T20:51:24.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.712+0000 7fe707fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70815d510 con 0x7fe7080ce3a0 2023-11-28T20:51:24.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.712+0000 7fe707fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70815e130 con 0x7fe70815d130 2023-11-28T20:51:24.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.712+0000 7fe6fe7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7080ce3a0 0x7fe708070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.713+0000 7fe6fe7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7080ce3a0 0x7fe708070ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32912/0 (socket says 172.21.15.45:32912) 2023-11-28T20:51:24.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.713+0000 7fe6fe7fc640 1 -- 172.21.15.45:0/2934204312 learned_addr learned my addr 172.21.15.45:0/2934204312 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:24.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.713+0000 7fe706ffd640 1 --2- 172.21.15.45:0/2934204312 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 0x7fe7080cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.713+0000 7fe6fe7fc640 1 -- 172.21.15.45:0/2934204312 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 msgr2=0x7fe7080cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:24.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.713+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2934204312 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 0x7fe7080cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.713+0000 7fe6fe7fc640 1 -- 172.21.15.45:0/2934204312 >> v1:172.21.15.190:6789/0 conn(0x7fe708071010 legacy=0x7fe70806cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.713+0000 7fe6fe7fc640 1 -- 172.21.15.45:0/2934204312 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70815e410 con 0x7fe7080ce3a0 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.713+0000 7fe706ffd640 1 --2- 172.21.15.45:0/2934204312 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 0x7fe7080cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.714+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2934204312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7080ce3a0 0x7fe708070ad0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0009870 tx=0x7fe6f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7054b71fd25dd77 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.715+0000 7fe7067fc640 1 -- 172.21.15.45:0/2934204312 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0031640 con 0x7fe7080ce3a0 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.715+0000 7fe7067fc640 1 -- 172.21.15.45:0/2934204312 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f00317a0 con 0x7fe7080ce3a0 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.715+0000 7fe7067fc640 1 -- 172.21.15.45:0/2934204312 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0031b30 con 0x7fe7080ce3a0 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.715+0000 7fe707fff640 1 -- 172.21.15.45:0/2934204312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7080ce3a0 msgr2=0x7fe708070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.715+0000 7fe707fff640 1 --2- 172.21.15.45:0/2934204312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7080ce3a0 0x7fe708070ad0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0009870 tx=0x7fe6f002b8f0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.716+0000 7fe707fff640 1 -- 172.21.15.45:0/2934204312 shutdown_connections 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.716+0000 7fe707fff640 1 --2- 172.21.15.45:0/2934204312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7080ce3a0 0x7fe708070ad0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.716+0000 7fe707fff640 1 --2- 172.21.15.45:0/2934204312 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 0x7fe7080cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.716+0000 7fe707fff640 1 -- 172.21.15.45:0/2934204312 >> 172.21.15.45:0/2934204312 conn(0x7fe7080661f0 msgr2=0x7fe7080665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:24.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.716+0000 7fe707fff640 1 -- 172.21.15.45:0/2934204312 shutdown_connections 2023-11-28T20:51:24.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.716+0000 7fe707fff640 1 -- 172.21.15.45:0/2934204312 wait complete. 2023-11-28T20:51:24.722 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:24.716+0000 7f2a28a2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.717+0000 7fe707fff640 1 Processor -- start 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.717+0000 7fe707fff640 1 -- start start 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.718+0000 7fe707fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe708071010 0x7fe708148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.718+0000 7fe707fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7080ce3a0 0x7fe708156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.718+0000 7fe707fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 0x7fe70815aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.718+0000 7fe707fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe708149420 con 0x7fe708071010 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.718+0000 7fe707fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe708159740 con 0x7fe7080ce3a0 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.718+0000 7fe707fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe708159a20 con 0x7fe70815d130 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe706ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe708071010 0x7fe708148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe706ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe708071010 0x7fe708148ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32926/0 (socket says 172.21.15.45:32926) 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe706ffd640 1 -- 172.21.15.45:0/2825242570 learned_addr learned my addr 172.21.15.45:0/2825242570 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe7077fe640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 0x7fe70815aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7080ce3a0 0x7fe708156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe706ffd640 1 -- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 msgr2=0x7fe70815aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe706ffd640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe70815d130 0x7fe70815aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe706ffd640 1 -- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7080ce3a0 msgr2=0x7fe708156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe706ffd640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7080ce3a0 0x7fe708156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe706ffd640 1 -- 172.21.15.45:0/2825242570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f00094d0 con 0x7fe708071010 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.719+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7080ce3a0 0x7fe708156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.720+0000 7fe706ffd640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe708071010 0x7fe708148ee0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fe6f800ce00 tx=0x7fe6f800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.720+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f800f890 con 0x7fe708071010 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.720+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f800f9f0 con 0x7fe708071010 2023-11-28T20:51:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.720+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f800fdc0 con 0x7fe708071010 2023-11-28T20:51:24.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.720+0000 7fe707fff640 1 -- 172.21.15.45:0/2825242570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe708159d00 con 0x7fe708071010 2023-11-28T20:51:24.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.720+0000 7fe707fff640 1 -- 172.21.15.45:0/2825242570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70815a2c0 con 0x7fe708071010 2023-11-28T20:51:24.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.721+0000 7fe707fff640 1 -- 172.21.15.45:0/2825242570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe708072380 con 0x7fe708071010 2023-11-28T20:51:24.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.722+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 39) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fe6f800b040 con 0x7fe708071010 2023-11-28T20:51:24.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.723+0000 7fe6fffff640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 0x7fe6e806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:24.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.724+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6072+0+0 (secure 0 0 0) 0x7fe6f808a560 con 0x7fe708071010 2023-11-28T20:51:24.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.724+0000 7fe6fe7fc640 1 -- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 msgr2=0x7fe6e806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/970930168 2023-11-28T20:51:24.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.724+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 0x7fe6e806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:24.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.728+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6f8055160 con 0x7fe708071010 2023-11-28T20:51:24.760 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:24.758+0000 7f6cf7232040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:51:24.772 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:24.770+0000 7f2a28a2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:51:24.809 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:24.807+0000 7f6cf7232040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:51:24.837 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:24.835+0000 7f2a28a2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:51:24.889 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:24.886+0000 7f2a28a2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:51:24.920 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:24.918+0000 7f6cf7232040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:51:24.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.925+0000 7fe6fe7fc640 1 -- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 msgr2=0x7fe6e806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/970930168 2023-11-28T20:51:24.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.925+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 0x7fe6e806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:51:24.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:24.944+0000 7fe707fff640 1 -- 172.21.15.45:0/2825242570 --> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe70815ecb0 con 0x7fe6e806afe0 2023-11-28T20:51:24.982 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:24.980+0000 7f6cf7232040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:51:25.005 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:25.003+0000 7f2a28a2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:51:25.031 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:25.029+0000 7f6cf7232040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:51:25.072 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:25.069+0000 7f2a28a2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:51:25.090 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:25.086+0000 7f6cf7232040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:51:25.123 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:25.121+0000 7f2a28a2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:51:25.138 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:25.135+0000 7f6cf7232040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:51:25.182 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:25.180+0000 7f2a28a2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:51:25.235 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:25.233+0000 7f2a28a2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:51:25.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:25.326+0000 7fe6fe7fc640 1 -- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 msgr2=0x7fe6e806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/970930168 2023-11-28T20:51:25.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:25.326+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 0x7fe6e806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T20:51:25.517 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:25.516+0000 7f6cf7232040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:51:25.637 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:25.635+0000 7f2a28a2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:51:25.674 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:25.673+0000 7f6cf7232040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:51:25.785 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:25.784+0000 7f6cf7232040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:51:25.806 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:25.804+0000 7f2a28a2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:51:25.835 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:25.834+0000 7f6cf7232040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:51:25.919 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:25.917+0000 7f2a28a2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:51:25.943 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:25.942+0000 7f6cf7232040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:51:25.971 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:25.969+0000 7f2a28a2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:51:26.050 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:26.049+0000 7f6cf7232040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:51:26.085 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:26.082+0000 7f2a28a2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:51:26.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:26.127+0000 7fe6fe7fc640 1 -- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 msgr2=0x7fe6e806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/970930168 2023-11-28T20:51:26.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:26.128+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 0x7fe6e806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T20:51:26.197 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:26.195+0000 7f2a28a2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:51:26.206 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:26.205+0000 7f6cf7232040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:51:26.255 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:26.253+0000 7f6cf7232040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:51:26.362 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:26.359+0000 7f2a28a2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:51:26.373 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:26.372+0000 7f6cf7232040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:51:26.414 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:26.411+0000 7f2a28a2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:51:26.539 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:26.536+0000 7f2a28a2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:51:27.194 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.193+0000 7f6cf7232040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:51:27.423 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:27.421+0000 7f2a28a2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:51:27.489 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.488+0000 7f6cf7232040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:51:27.546 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.545+0000 7f6cf7232040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:51:27.660 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.659+0000 7f6cf7232040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:51:27.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:27.679+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 40) v1 ==== 88959+0+0 (secure 0 0 0) 0x7fe6f8055530 con 0x7fe708071010 2023-11-28T20:51:27.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:27.680+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 msgr2=0x7fe6e806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:27.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:27.680+0000 7fe6fffff640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/970930168,v1:172.21.15.190:6833/970930168] conn(0x7fe6e806afe0 0x7fe6e806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:27.747 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:27.744+0000 7f2a28a2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:51:27.817 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:27.814+0000 7f2a28a2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:51:27.866 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c765e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.866 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c765e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.866 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c765e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.866 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c765e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.866 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c765e9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.867 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c735e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.867 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c735e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.867 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c735e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.867 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c735e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.867 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:27.865+0000 7f6c735e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:27.940 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:27.938+0000 7f2a28a2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:51:28.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.689+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 41) v1 ==== 89086+0+0 (secure 0 0 0) 0x7fe6f8056c40 con 0x7fe708071010 2023-11-28T20:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.690+0000 7fe6fffff640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fe6e80355a0 0x7fe6e8035980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:28.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.690+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 --> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6e8035ec0 con 0x7fe6e80355a0 2023-11-28T20:51:28.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.694+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fe6e80355a0 0x7fe6e8035980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:28.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.702+0000 7fe6fe7fc640 1 --2- 172.21.15.45:0/2825242570 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fe6e80355a0 0x7fe6e8035980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0036000 tx=0x7fe6f0008030 comp rx=0 tx=0).ready entity=mgr.5771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:28.726 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:28.725+0000 7f6c7d5f7640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-11-28T20:51:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.725+0000 7fe6fffff640 1 -- 172.21.15.45:0/2825242570 <== mgr.5771 v2:172.21.15.190:6832/1373961985 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fe6e8035ec0 con 0x7fe6e80355a0 2023-11-28T20:51:28.727 INFO:teuthology.orchestra.run.smithi045.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-11-28T20:51:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T20:51:28.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T20:51:28.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.964+0000 7fe5ee953640 1 Processor -- start 2023-11-28T20:51:28.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.964+0000 7fe5ee953640 1 -- start start 2023-11-28T20:51:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.965+0000 7fe5ee953640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe5e8157c10 0x7fe5e806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.965+0000 7fe5ee953640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5e806a500 0x7fe5e806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.965+0000 7fe5ee953640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5e815ddd0 0x7fe5e815e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.965+0000 7fe5ee953640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e806ca60 con 0x7fe5e815ddd0 2023-11-28T20:51:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.965+0000 7fe5ee953640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e806aea0 con 0x7fe5e8157c10 2023-11-28T20:51:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.965+0000 7fe5ee953640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e815eb40 con 0x7fe5e806a500 2023-11-28T20:51:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5ecef0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5e815ddd0 0x7fe5e815e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5ecef0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5e815ddd0 0x7fe5e815e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32942/0 (socket says 172.21.15.45:32942) 2023-11-28T20:51:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5ecef0640 1 -- 172.21.15.45:0/219495023 learned_addr learned my addr 172.21.15.45:0/219495023 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:28.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/219495023 >> v2:172.21.15.190:3300/0 conn(0x7fe5e8157c10 0x7fe5e806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5e77fe640 1 --2- 172.21.15.45:0/219495023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5e806a500 0x7fe5e806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5ecef0640 1 -- 172.21.15.45:0/219495023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5e806a500 msgr2=0x7fe5e806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5ecef0640 1 --2- 172.21.15.45:0/219495023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5e806a500 0x7fe5e806a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5ecef0640 1 -- 172.21.15.45:0/219495023 >> v2:172.21.15.190:3300/0 conn(0x7fe5e8157c10 msgr2=0x7fe5e806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5ecef0640 1 --2- 172.21.15.45:0/219495023 >> v2:172.21.15.190:3300/0 conn(0x7fe5e8157c10 0x7fe5e806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.966+0000 7fe5ecef0640 1 -- 172.21.15.45:0/219495023 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8157ff0 con 0x7fe5e815ddd0 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.967+0000 7fe5ecef0640 1 --2- 172.21.15.45:0/219495023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e815ddd0 0x7fe5e815e260 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80c9540 tx=0x7fe5e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8b544600688bfe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5e6ffd640 1 -- 172.21.15.45:0/219495023 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0031870 con 0x7fe5e815ddd0 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5e6ffd640 1 -- 172.21.15.45:0/219495023 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e00319d0 con 0x7fe5e815ddd0 2023-11-28T20:51:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5e6ffd640 1 -- 172.21.15.45:0/219495023 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0031d60 con 0x7fe5e815ddd0 2023-11-28T20:51:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5ee953640 1 -- 172.21.15.45:0/219495023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e815ddd0 msgr2=0x7fe5e815e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5ee953640 1 --2- 172.21.15.45:0/219495023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e815ddd0 0x7fe5e815e260 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80c9540 tx=0x7fe5e002b910 comp rx=0 tx=0).stop 2023-11-28T20:51:28.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5ee953640 1 -- 172.21.15.45:0/219495023 shutdown_connections 2023-11-28T20:51:28.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5ee953640 1 --2- 172.21.15.45:0/219495023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e815ddd0 0x7fe5e815e260 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:28.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5ee953640 1 --2- 172.21.15.45:0/219495023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5e806a500 0x7fe5e806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:28.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5ee953640 1 --2- 172.21.15.45:0/219495023 >> v2:172.21.15.190:3300/0 conn(0x7fe5e8157c10 0x7fe5e806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:28.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5ee953640 1 -- 172.21.15.45:0/219495023 >> 172.21.15.45:0/219495023 conn(0x7fe5e80ced60 msgr2=0x7fe5e80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:28.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.968+0000 7fe5ee953640 1 -- 172.21.15.45:0/219495023 shutdown_connections 2023-11-28T20:51:28.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.969+0000 7fe5ee953640 1 -- 172.21.15.45:0/219495023 wait complete. 2023-11-28T20:51:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.971+0000 7fe5ee953640 1 Processor -- start 2023-11-28T20:51:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.971+0000 7fe5ee953640 1 -- start start 2023-11-28T20:51:28.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.972+0000 7fe5ee953640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e806a500 0x7fe5e809ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:28.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.972+0000 7fe5ee953640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5e8157c10 0x7fe5e80ad640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:28.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.972+0000 7fe5ee953640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5e815ddd0 0x7fe5e81581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:28.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.972+0000 7fe5ee953640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e809be50 con 0x7fe5e806a500 2023-11-28T20:51:28.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.972+0000 7fe5ee953640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e809b240 con 0x7fe5e815ddd0 2023-11-28T20:51:28.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.972+0000 7fe5ee953640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e809b520 con 0x7fe5e8157c10 2023-11-28T20:51:28.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.972+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e806a500 0x7fe5e809ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:28.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.972+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e806a500 0x7fe5e809ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32954/0 (socket says 172.21.15.45:32954) 2023-11-28T20:51:28.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.972+0000 7fe5e7fff640 1 -- 172.21.15.45:0/3857083823 learned_addr learned my addr 172.21.15.45:0/3857083823 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:28.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5ecef0640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5e815ddd0 0x7fe5e81581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:28.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5e7fff640 1 -- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5e8157c10 msgr2=0x7fe5e80ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:28.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5e8157c10 0x7fe5e80ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:28.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5e7fff640 1 -- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5e815ddd0 msgr2=0x7fe5e81581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:28.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5e815ddd0 0x7fe5e81581d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:28.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5e7fff640 1 -- 172.21.15.45:0/3857083823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00094d0 con 0x7fe5e806a500 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e806a500 0x7fe5e809ab80 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8007ef0 tx=0x7fe5d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/3857083823 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8014620 con 0x7fe5e806a500 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/3857083823 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8014780 con 0x7fe5e806a500 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/3857083823 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8014b30 con 0x7fe5e806a500 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.973+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e809b800 con 0x7fe5e806a500 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.974+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e80c1ad0 con 0x7fe5e806a500 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.975+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e80733d0 con 0x7fe5e806a500 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.976+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/3857083823 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 41) v1 ==== 89086+0+0 (secure 0 0 0) 0x7fe5d8014c90 con 0x7fe5e806a500 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.976+0000 7fe5e4ff9640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fe5d406ad10 0x7fe5d406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.977+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/3857083823 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6416+0+0 (secure 0 0 0) 0x7fe5d8089e80 con 0x7fe5e806a500 2023-11-28T20:51:28.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.977+0000 7fe5e77fe640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fe5d406ad10 0x7fe5d406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:28.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.978+0000 7fe5e77fe640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fe5d406ad10 0x7fe5d406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc008040 comp rx=0 tx=0).ready entity=mgr.5771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:28.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:28.981+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/3857083823 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5d8054fb0 con 0x7fe5e806a500 2023-11-28T20:51:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.226+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fe5e809cea0 con 0x7fe5e806a500 2023-11-28T20:51:29.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.741+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/3857083823 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v42) v1 ==== 93+0+0 (secure 0 0 0) 0x7fe5d80585d0 con 0x7fe5e806a500 2023-11-28T20:51:29.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.748+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/3857083823 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 42) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fe5d8055380 con 0x7fe5e806a500 2023-11-28T20:51:29.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.748+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fe5d406ad10 msgr2=0x7fe5d406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:29.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.748+0000 7fe5ee953640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fe5d406ad10 0x7fe5d406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc008040 comp rx=0 tx=0).stop 2023-11-28T20:51:29.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.748+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e806a500 msgr2=0x7fe5e809ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:29.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.748+0000 7fe5ee953640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e806a500 0x7fe5e809ab80 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8007ef0 tx=0x7fe5d800e4b0 comp rx=0 tx=0).stop 2023-11-28T20:51:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.749+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 shutdown_connections 2023-11-28T20:51:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.749+0000 7fe5ee953640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fe5d406ad10 0x7fe5d406d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.749+0000 7fe5ee953640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5e815ddd0 0x7fe5e81581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.749+0000 7fe5ee953640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5e8157c10 0x7fe5e80ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.749+0000 7fe5ee953640 1 --2- 172.21.15.45:0/3857083823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5e806a500 0x7fe5e809ab80 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.749+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 >> 172.21.15.45:0/3857083823 conn(0x7fe5e80ced60 msgr2=0x7fe5e80ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.749+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 shutdown_connections 2023-11-28T20:51:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:29.749+0000 7fe5ee953640 1 -- 172.21.15.45:0/3857083823 wait complete. 2023-11-28T20:51:29.773 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:51:29.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:51:29.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:51:30.008 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:51:30.008 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:51:30.008 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:30.065 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:51:30.065 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:30.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:30.279 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:30.278+0000 7f55a5f53040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:51:30.291 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:30.289+0000 7f5b36e0e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:51:30.332 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:30.331+0000 7f55a5f53040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:51:30.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.334+0000 7fed5ffff640 1 Processor -- start 2023-11-28T20:51:30.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.334+0000 7fed5ffff640 1 -- start start 2023-11-28T20:51:30.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed6015d550 0x7fed600cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5ffff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fed600d0100 0x7fed600d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed600d1ab0 0x7fed60070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5ffff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed600d0560 con 0x7fed600d1ab0 2023-11-28T20:51:30.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5ffff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed600d2230 con 0x7fed600d0100 2023-11-28T20:51:30.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed600d2510 con 0x7fed6015d550 2023-11-28T20:51:30.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5f7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed600d1ab0 0x7fed60070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5f7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed600d1ab0 0x7fed60070b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32974/0 (socket says 172.21.15.45:32974) 2023-11-28T20:51:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5f7fe640 1 -- 172.21.15.45:0/1649462672 learned_addr learned my addr 172.21.15.45:0/1649462672 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5e7fc640 1 --2- 172.21.15.45:0/1649462672 >> v2:172.21.15.190:3300/0 conn(0x7fed600d0100 0x7fed600d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.336+0000 7fed5effd640 1 --2- 172.21.15.45:0/1649462672 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed6015d550 0x7fed600cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.337+0000 7fed5f7fe640 1 -- 172.21.15.45:0/1649462672 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed6015d550 msgr2=0x7fed600cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:30.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.337+0000 7fed5f7fe640 1 --2- 172.21.15.45:0/1649462672 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed6015d550 0x7fed600cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.337+0000 7fed5f7fe640 1 -- 172.21.15.45:0/1649462672 >> v2:172.21.15.190:3300/0 conn(0x7fed600d0100 msgr2=0x7fed600d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:30.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.337+0000 7fed5f7fe640 1 --2- 172.21.15.45:0/1649462672 >> v2:172.21.15.190:3300/0 conn(0x7fed600d0100 0x7fed600d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.337+0000 7fed5f7fe640 1 -- 172.21.15.45:0/1649462672 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed600d27f0 con 0x7fed600d1ab0 2023-11-28T20:51:30.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.338+0000 7fed5f7fe640 1 --2- 172.21.15.45:0/1649462672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 0x7fed60070b20 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fed600d0d50 tx=0x7fed5802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ac35510113ada0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:30.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.339+0000 7fed5dffb640 1 -- 172.21.15.45:0/1649462672 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5802ca70 con 0x7fed600d1ab0 2023-11-28T20:51:30.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.339+0000 7fed5dffb640 1 -- 172.21.15.45:0/1649462672 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5802cbd0 con 0x7fed600d1ab0 2023-11-28T20:51:30.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.339+0000 7fed5dffb640 1 -- 172.21.15.45:0/1649462672 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed580343f0 con 0x7fed600d1ab0 2023-11-28T20:51:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.339+0000 7fed5ffff640 1 -- 172.21.15.45:0/1649462672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 msgr2=0x7fed60070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.339+0000 7fed5ffff640 1 --2- 172.21.15.45:0/1649462672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 0x7fed60070b20 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fed600d0d50 tx=0x7fed5802bed0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.340+0000 7fed5ffff640 1 -- 172.21.15.45:0/1649462672 shutdown_connections 2023-11-28T20:51:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.340+0000 7fed5ffff640 1 --2- 172.21.15.45:0/1649462672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 0x7fed60070b20 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.340+0000 7fed5ffff640 1 --2- 172.21.15.45:0/1649462672 >> v2:172.21.15.190:3300/0 conn(0x7fed600d0100 0x7fed600d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.340+0000 7fed5ffff640 1 --2- 172.21.15.45:0/1649462672 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed6015d550 0x7fed600cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.340+0000 7fed5ffff640 1 -- 172.21.15.45:0/1649462672 >> 172.21.15.45:0/1649462672 conn(0x7fed6006ac70 msgr2=0x7fed600b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.340+0000 7fed5ffff640 1 -- 172.21.15.45:0/1649462672 shutdown_connections 2023-11-28T20:51:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.340+0000 7fed5ffff640 1 -- 172.21.15.45:0/1649462672 wait complete. 2023-11-28T20:51:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.341+0000 7fed5ffff640 1 Processor -- start 2023-11-28T20:51:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.342+0000 7fed5ffff640 1 -- start start 2023-11-28T20:51:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed600d0100 0x7fed600cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 0x7fed600cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6015d550 0x7fed60156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5ffff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed600cd950 con 0x7fed600d1ab0 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5ffff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed601576d0 con 0x7fed6015d550 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed600cb4c0 con 0x7fed600d0100 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5e7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 0x7fed600cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5e7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 0x7fed600cd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32988/0 (socket says 172.21.15.45:32988) 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5e7fc640 1 -- 172.21.15.45:0/4085311096 learned_addr learned my addr 172.21.15.45:0/4085311096 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5effd640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed600d0100 0x7fed600cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5f7fe640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6015d550 0x7fed60156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5e7fc640 1 -- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed600d0100 msgr2=0x7fed600cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5e7fc640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed600d0100 0x7fed600cac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5e7fc640 1 -- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6015d550 msgr2=0x7fed60156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5e7fc640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6015d550 0x7fed60156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.343+0000 7fed5e7fc640 1 -- 172.21.15.45:0/4085311096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed58009b30 con 0x7fed600d1ab0 2023-11-28T20:51:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.344+0000 7fed5e7fc640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 0x7fed600cd410 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fed5400d800 tx=0x7fed5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.344+0000 7fed47fff640 1 -- 172.21.15.45:0/4085311096 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed540096a0 con 0x7fed600d1ab0 2023-11-28T20:51:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.344+0000 7fed47fff640 1 -- 172.21.15.45:0/4085311096 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed54009800 con 0x7fed600d1ab0 2023-11-28T20:51:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.344+0000 7fed47fff640 1 -- 172.21.15.45:0/4085311096 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed54009b90 con 0x7fed600d1ab0 2023-11-28T20:51:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.344+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed60156c70 con 0x7fed600d1ab0 2023-11-28T20:51:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.344+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed60157080 con 0x7fed600d1ab0 2023-11-28T20:51:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.346+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed60156ee0 con 0x7fed600d1ab0 2023-11-28T20:51:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.346+0000 7fed47fff640 1 -- 172.21.15.45:0/4085311096 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 42) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fed54009cf0 con 0x7fed600d1ab0 2023-11-28T20:51:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.347+0000 7fed47fff640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fed4c06af90 0x7fed4c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.347+0000 7fed47fff640 1 -- 172.21.15.45:0/4085311096 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6416+0+0 (secure 0 0 0) 0x7fed54089f40 con 0x7fed600d1ab0 2023-11-28T20:51:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.347+0000 7fed5effd640 1 -- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fed4c06af90 msgr2=0x7fed4c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.347+0000 7fed5effd640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fed4c06af90 0x7fed4c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:30.351 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:30.349+0000 7f5b36e0e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:51:30.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.354+0000 7fed47fff640 1 -- 172.21.15.45:0/4085311096 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed60156ee0 con 0x7fed600d1ab0 2023-11-28T20:51:30.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.548+0000 7fed5effd640 1 -- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fed4c06af90 msgr2=0x7fed4c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:30.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.548+0000 7fed5effd640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fed4c06af90 0x7fed4c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:51:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.568+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed60156ee0 con 0x7fed600d1ab0 2023-11-28T20:51:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.571+0000 7fed47fff640 1 -- 172.21.15.45:0/4085311096 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+17346 (secure 0 0 0) 0x7fed60156ee0 con 0x7fed600d1ab0 2023-11-28T20:51:30.573 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:30.573 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":103,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:27.664546+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:10.974716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:11.975222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3103043064":"2023-11-29T20:51:27.664495+0000","172.21.15.190:6832/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3436414960":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3227357091":"2023-11-29T20:51:27.664495+0000","172.21.15.190:6833/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/4025400682":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2627631957":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2067877259":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/1852415784":"2023-11-28T21:51:13.642235+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.573+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fed4c06af90 msgr2=0x7fed4c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fed4c06af90 0x7fed4c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 msgr2=0x7fed600cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 0x7fed600cd410 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fed5400d800 tx=0x7fed5400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 shutdown_connections 2023-11-28T20:51:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fed4c06af90 0x7fed4c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6015d550 0x7fed60156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed600d1ab0 0x7fed600cd410 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 --2- 172.21.15.45:0/4085311096 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed600d0100 0x7fed600cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 >> 172.21.15.45:0/4085311096 conn(0x7fed6006ac70 msgr2=0x7fed600d2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:30.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 shutdown_connections 2023-11-28T20:51:30.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.574+0000 7fed5ffff640 1 -- 172.21.15.45:0/4085311096 wait complete. 2023-11-28T20:51:30.594 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:51:10.974716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:51:11.975222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:51:30.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:51:30.781 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:30.780+0000 7f55a5f53040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:51:30.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.808+0000 7f21bcae5640 1 Processor -- start 2023-11-28T20:51:30.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.809+0000 7f21bcae5640 1 -- start start 2023-11-28T20:51:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.810+0000 7f21bcae5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21b80661f0 0x7f21b8066650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.810+0000 7f21bcae5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b80cd710 con 0x7f21b815b6d0 2023-11-28T20:51:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.810+0000 7f21bcae5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b8066b90 con 0x7f21b806a500 2023-11-28T20:51:30.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.810+0000 7f21bcae5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b80cc140 con 0x7f21b80661f0 2023-11-28T20:51:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.810+0000 7f21affff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21b80661f0 0x7f21b8066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.810+0000 7f21b659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f21b815b6d0 0x7f21b80ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35378/0 (socket says 172.21.15.45:35378) 2023-11-28T20:51:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.810+0000 7f21affff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21b80661f0 0x7f21b8066650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33002/0 (socket says 172.21.15.45:33002) 2023-11-28T20:51:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.810+0000 7f21b659c640 1 -- 172.21.15.45:0/1248733162 learned_addr learned my addr 172.21.15.45:0/1248733162 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.811+0000 7f21affff640 1 -- 172.21.15.45:0/1248733162 >> v1:172.21.15.190:6789/0 conn(0x7f21b806a500 legacy=0x7f21b806a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.811+0000 7f21affff640 1 -- 172.21.15.45:0/1248733162 >> v1:172.21.15.45:6789/0 conn(0x7f21b815b6d0 legacy=0x7f21b80ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:30.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.811+0000 7f21affff640 1 -- 172.21.15.45:0/1248733162 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80cb220 con 0x7f21b80661f0 2023-11-28T20:51:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.812+0000 7f21affff640 1 --2- 172.21.15.45:0/1248733162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b80661f0 0x7f21b8066650 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f21a00097a0 tx=0x7f21a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67351b9202a0267 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.812+0000 7f21b5d9b640 1 -- 172.21.15.45:0/1248733162 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a002e070 con 0x7f21b80661f0 2023-11-28T20:51:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.813+0000 7f21b5d9b640 1 -- 172.21.15.45:0/1248733162 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a00315a0 con 0x7f21b80661f0 2023-11-28T20:51:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.813+0000 7f21b5d9b640 1 -- 172.21.15.45:0/1248733162 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0031930 con 0x7f21b80661f0 2023-11-28T20:51:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.813+0000 7f21bcae5640 1 -- 172.21.15.45:0/1248733162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b80661f0 msgr2=0x7f21b8066650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.813+0000 7f21bcae5640 1 --2- 172.21.15.45:0/1248733162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b80661f0 0x7f21b8066650 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f21a00097a0 tx=0x7f21a002b840 comp rx=0 tx=0).stop 2023-11-28T20:51:30.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.813+0000 7f21bcae5640 1 -- 172.21.15.45:0/1248733162 shutdown_connections 2023-11-28T20:51:30.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.813+0000 7f21bcae5640 1 --2- 172.21.15.45:0/1248733162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b80661f0 0x7f21b8066650 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.813+0000 7f21bcae5640 1 -- 172.21.15.45:0/1248733162 >> 172.21.15.45:0/1248733162 conn(0x7f21b80cc930 msgr2=0x7f21b80ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:30.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.813+0000 7f21bcae5640 1 -- 172.21.15.45:0/1248733162 shutdown_connections 2023-11-28T20:51:30.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.813+0000 7f21bcae5640 1 -- 172.21.15.45:0/1248733162 wait complete. 2023-11-28T20:51:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.815+0000 7f21bcae5640 1 Processor -- start 2023-11-28T20:51:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.815+0000 7f21bcae5640 1 -- start start 2023-11-28T20:51:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.816+0000 7f21bcae5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21b80661f0 0x7f21b80c3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.816+0000 7f21bcae5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b806a500 0x7f21b80c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.816+0000 7f21bcae5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b815b6d0 0x7f21b80c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.816+0000 7f21bcae5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80c98b0 con 0x7f21b806a500 2023-11-28T20:51:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.816+0000 7f21bcae5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b8158600 con 0x7f21b815b6d0 2023-11-28T20:51:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.816+0000 7f21bcae5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80c89e0 con 0x7f21b80661f0 2023-11-28T20:51:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.816+0000 7f21affff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b806a500 0x7f21b80c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.816+0000 7f21affff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b806a500 0x7f21b80c5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33014/0 (socket says 172.21.15.45:33014) 2023-11-28T20:51:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.816+0000 7f21affff640 1 -- 172.21.15.45:0/1401403605 learned_addr learned my addr 172.21.15.45:0/1401403605 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:30.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21b659c640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21b80661f0 0x7f21b80c3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21b6d9d640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b815b6d0 0x7f21b80c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:30.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21affff640 1 -- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21b80661f0 msgr2=0x7f21b80c3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:30.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21affff640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21b80661f0 0x7f21b80c3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21affff640 1 -- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b815b6d0 msgr2=0x7f21b80c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21affff640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b815b6d0 0x7f21b80c6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21affff640 1 -- 172.21.15.45:0/1401403605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0009400 con 0x7f21b806a500 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21b659c640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21b80661f0 0x7f21b80c3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21b6d9d640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b815b6d0 0x7f21b80c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21affff640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b806a500 0x7f21b80c5e00 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f21a0034040 tx=0x7f21a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.817+0000 7f21aeffd640 1 -- 172.21.15.45:0/1401403605 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0032900 con 0x7f21b806a500 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.818+0000 7f21aeffd640 1 -- 172.21.15.45:0/1401403605 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a0032a60 con 0x7f21b806a500 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.818+0000 7f21aeffd640 1 -- 172.21.15.45:0/1401403605 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0032e10 con 0x7f21b806a500 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.818+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b80c6880 con 0x7f21b806a500 2023-11-28T20:51:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.818+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b80c8f10 con 0x7f21b806a500 2023-11-28T20:51:30.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.819+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b80733d0 con 0x7f21b806a500 2023-11-28T20:51:30.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.820+0000 7f21aeffd640 1 -- 172.21.15.45:0/1401403605 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 42) v1 ==== 89334+0+0 (secure 0 0 0) 0x7f21a003b3f0 con 0x7f21b806a500 2023-11-28T20:51:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.821+0000 7f21aeffd640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f219806af60 0x7f219806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.821+0000 7f21aeffd640 1 -- 172.21.15.45:0/1401403605 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6416+0+0 (secure 0 0 0) 0x7f21a00abc20 con 0x7f21b806a500 2023-11-28T20:51:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.821+0000 7f21b659c640 1 -- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f219806af60 msgr2=0x7f219806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.821+0000 7f21b659c640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f219806af60 0x7f219806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:30.825+0000 7f21aeffd640 1 -- 172.21.15.45:0/1401403605 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21a0076800 con 0x7f21b806a500 2023-11-28T20:51:30.834 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:30.833+0000 7f55a5f53040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:51:30.858 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:30.854+0000 7f5b36e0e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:51:30.884 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:30.883+0000 7f55a5f53040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:51:30.911 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:30.909+0000 7f5b36e0e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:51:30.945 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:30.944+0000 7f55a5f53040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:51:30.964 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:30.962+0000 7f5b36e0e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:51:30.994 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:30.993+0000 7f55a5f53040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:51:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.013+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21b809d860 con 0x7f21b806a500 2023-11-28T20:51:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.014+0000 7f21aeffd640 1 -- 172.21.15.45:0/1401403605 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21a0079e20 con 0x7f21b806a500 2023-11-28T20:51:31.016 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:51:31.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f219806af60 msgr2=0x7f219806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f219806af60 0x7f219806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b806a500 msgr2=0x7f21b80c5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b806a500 0x7f21b80c5e00 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f21a0034040 tx=0x7f21a00046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 shutdown_connections 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f219806af60 0x7f219806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b815b6d0 0x7f21b80c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21b806a500 0x7f21b80c5e00 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 --2- 172.21.15.45:0/1401403605 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21b80661f0 0x7f21b80c3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 >> 172.21.15.45:0/1401403605 conn(0x7f21b80cc930 msgr2=0x7f21b80afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:31.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 shutdown_connections 2023-11-28T20:51:31.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.016+0000 7f21bcae5640 1 -- 172.21.15.45:0/1401403605 wait complete. 2023-11-28T20:51:31.028 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:31.026+0000 7f5b36e0e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:51:31.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:51:31.081 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:31.079+0000 7f5b36e0e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:51:31.104 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:31.103+0000 7f55a5f53040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:51:31.166 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:31.165+0000 7f55a5f53040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:51:31.200 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:31.198+0000 7f5b36e0e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:51:31.216 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:31.214+0000 7f55a5f53040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:51:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.249+0000 7fba9dba9640 1 Processor -- start 2023-11-28T20:51:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.250+0000 7fba9dba9640 1 -- start start 2023-11-28T20:51:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.251+0000 7fba9dba9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9806ac10 0x7fba9815e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.251+0000 7fba9dba9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fba98070830 0x7fba9806cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.251+0000 7fba9dba9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9815eb40 con 0x7fba9815a1e0 2023-11-28T20:51:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.251+0000 7fba9dba9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba980ae670 con 0x7fba98070830 2023-11-28T20:51:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.251+0000 7fba9dba9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba98070d60 con 0x7fba9806ac10 2023-11-28T20:51:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.252+0000 7fba977fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fba9815a1e0 0x7fba9806a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35386/0 (socket says 172.21.15.45:35386) 2023-11-28T20:51:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.252+0000 7fba977fe640 1 -- 172.21.15.45:0/2025119405 learned_addr learned my addr 172.21.15.45:0/2025119405 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.252+0000 7fba97fff640 1 --2- 172.21.15.45:0/2025119405 >> v2:172.21.15.190:3300/0 conn(0x7fba98070830 0x7fba9806cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.252+0000 7fba8ffff640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9806ac10 0x7fba9815e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.252+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499449233 0 0) 0x7fba88001110 con 0x7fba9815a1e0 2023-11-28T20:51:31.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.252+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba84005730 con 0x7fba9815a1e0 2023-11-28T20:51:31.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.252+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197401271 0 0) 0x7fba84005730 con 0x7fba9815a1e0 2023-11-28T20:51:31.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9806ac10 msgr2=0x7fba9815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:31.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9806ac10 0x7fba9815e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 >> v2:172.21.15.190:3300/0 conn(0x7fba98070830 msgr2=0x7fba9806cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:31.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 --2- 172.21.15.45:0/2025119405 >> v2:172.21.15.190:3300/0 conn(0x7fba98070830 0x7fba9806cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba98071040 con 0x7fba9815a1e0 2023-11-28T20:51:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fba88001800 con 0x7fba9815a1e0 2023-11-28T20:51:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 >> v1:172.21.15.45:6789/0 conn(0x7fba9815a1e0 legacy=0x7fba9806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba84005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba840069d0 0x7fba84006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9806ac10 0x7fba84007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba980ae670 con 0x7fba84005b80 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba98070d60 con 0x7fba840069d0 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8400dbe0 con 0x7fba9806ac10 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba977fe640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba84005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba97fff640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9806ac10 0x7fba84007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.253+0000 7fba8ffff640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba840069d0 0x7fba84006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.254+0000 7fba977fe640 1 -- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9806ac10 msgr2=0x7fba84007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.254+0000 7fba977fe640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9806ac10 0x7fba84007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.254+0000 7fba977fe640 1 -- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba840069d0 msgr2=0x7fba84006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.254+0000 7fba977fe640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba840069d0 0x7fba84006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.254+0000 7fba977fe640 1 -- 172.21.15.45:0/2025119405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84005730 con 0x7fba84005b80 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.255+0000 7fba977fe640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba84005fa0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fba8800c270 tx=0x7fba84007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77f15c352e2c28c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.255+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88005ad0 con 0x7fba84005b80 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.255+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba88005c30 con 0x7fba84005b80 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.255+0000 7fba96ffd640 1 -- 172.21.15.45:0/2025119405 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8802c8b0 con 0x7fba84005b80 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.256+0000 7fba9dba9640 1 -- 172.21.15.45:0/2025119405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 msgr2=0x7fba84005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.256+0000 7fba9dba9640 1 --2- 172.21.15.45:0/2025119405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba84005fa0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fba8800c270 tx=0x7fba84007a00 comp rx=0 tx=0).stop 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.256+0000 7fba97fff640 1 -- 172.21.15.45:0/2025119405 reap_dead start 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.256+0000 7fba9dba9640 1 -- 172.21.15.45:0/2025119405 shutdown_connections 2023-11-28T20:51:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.256+0000 7fba9dba9640 1 -- 172.21.15.45:0/2025119405 >> 172.21.15.45:0/2025119405 conn(0x7fba980cb780 msgr2=0x7fba980cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:31.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.256+0000 7fba9dba9640 1 -- 172.21.15.45:0/2025119405 shutdown_connections 2023-11-28T20:51:31.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.256+0000 7fba9dba9640 1 -- 172.21.15.45:0/2025119405 wait complete. 2023-11-28T20:51:31.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.258+0000 7fba9dba9640 1 Processor -- start 2023-11-28T20:51:31.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.258+0000 7fba9dba9640 1 -- start start 2023-11-28T20:51:31.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.259+0000 7fba9dba9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba9814e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.259+0000 7fba9dba9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba9815a5c0 0x7fba9815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.259+0000 7fba9dba9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9815c2c0 0x7fba9815c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.259+0000 7fba9dba9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9815d2b0 con 0x7fba84005b80 2023-11-28T20:51:31.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.259+0000 7fba9dba9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba980c2b90 con 0x7fba9815a5c0 2023-11-28T20:51:31.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.259+0000 7fba9dba9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba980c2e30 con 0x7fba9815c2c0 2023-11-28T20:51:31.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.259+0000 7fba977fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba9814e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.259+0000 7fba977fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba9814e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33036/0 (socket says 172.21.15.45:33036) 2023-11-28T20:51:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.259+0000 7fba977fe640 1 -- 172.21.15.45:0/4219391025 learned_addr learned my addr 172.21.15.45:0/4219391025 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.260+0000 7fba8ffff640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba9815a5c0 0x7fba9815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.260+0000 7fba97fff640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9815c2c0 0x7fba9815c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.260+0000 7fba977fe640 1 -- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9815c2c0 msgr2=0x7fba9815c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.260+0000 7fba977fe640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9815c2c0 0x7fba9815c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.260+0000 7fba977fe640 1 -- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba9815a5c0 msgr2=0x7fba9815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.260+0000 7fba977fe640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba9815a5c0 0x7fba9815aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.260+0000 7fba977fe640 1 -- 172.21.15.45:0/4219391025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba980c3110 con 0x7fba84005b80 2023-11-28T20:51:31.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.260+0000 7fba977fe640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba9814e040 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fba8800c270 tx=0x7fba980718e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:31.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.261+0000 7fba94ff9640 1 -- 172.21.15.45:0/4219391025 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88039070 con 0x7fba84005b80 2023-11-28T20:51:31.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.261+0000 7fba94ff9640 1 -- 172.21.15.45:0/4219391025 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8802ccc0 con 0x7fba84005b80 2023-11-28T20:51:31.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.261+0000 7fba94ff9640 1 -- 172.21.15.45:0/4219391025 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8803c040 con 0x7fba84005b80 2023-11-28T20:51:31.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.261+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba980c33f0 con 0x7fba84005b80 2023-11-28T20:51:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.261+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba980c39b0 con 0x7fba84005b80 2023-11-28T20:51:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.262+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba980c3b40 con 0x7fba84005b80 2023-11-28T20:51:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.263+0000 7fba94ff9640 1 -- 172.21.15.45:0/4219391025 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 42) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fba88032070 con 0x7fba84005b80 2023-11-28T20:51:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.264+0000 7fba94ff9640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fba7806af60 0x7fba7806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.265+0000 7fba94ff9640 1 -- 172.21.15.45:0/4219391025 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6416+0+0 (secure 0 0 0) 0x7fba880aaf20 con 0x7fba84005b80 2023-11-28T20:51:31.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.265+0000 7fba8ffff640 1 -- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fba7806af60 msgr2=0x7fba7806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:31.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.265+0000 7fba8ffff640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fba7806af60 0x7fba7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:31.275 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:31.267+0000 7f5b36e0e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:51:31.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.269+0000 7fba94ff9640 1 -- 172.21.15.45:0/4219391025 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba88075b80 con 0x7fba84005b80 2023-11-28T20:51:31.276 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:31.273+0000 7f55a5f53040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:51:31.322 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:31.319+0000 7f5b36e0e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:51:31.324 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:31.323+0000 7f55a5f53040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:51:31.381 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:31.379+0000 7f5b36e0e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:51:31.434 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:31.432+0000 7f5b36e0e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:51:31.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.457+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba9815eb40 con 0x7fba84005b80 2023-11-28T20:51:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.458+0000 7fba94ff9640 1 -- 172.21.15.45:0/4219391025 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) v1 ==== 104+0+11 (secure 0 0 0) 0x7fba9815eb40 con 0x7fba84005b80 2023-11-28T20:51:31.460 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:51:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fba7806af60 msgr2=0x7fba7806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fba7806af60 0x7fba7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 msgr2=0x7fba9814e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba9814e040 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fba8800c270 tx=0x7fba980718e0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 shutdown_connections 2023-11-28T20:51:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba9815c2c0 0x7fba9815c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7fba7806af60 0x7fba7806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba9815a5c0 0x7fba9815aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 --2- 172.21.15.45:0/4219391025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba84005b80 0x7fba9814e040 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 >> 172.21.15.45:0/4219391025 conn(0x7fba980cb780 msgr2=0x7fba9806a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 shutdown_connections 2023-11-28T20:51:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.461+0000 7fba9dba9640 1 -- 172.21.15.45:0/4219391025 wait complete. 2023-11-28T20:51:31.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:51:31.705 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:31.704+0000 7f55a5f53040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:51:31.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.744+0000 7f5857147640 1 Processor -- start 2023-11-28T20:51:31.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.745+0000 7f5857147640 1 -- start start 2023-11-28T20:51:31.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.746+0000 7f5857147640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500b2290 0x7f58500d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.746+0000 7f5857147640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58500b90a0 con 0x7f58500b1d40 2023-11-28T20:51:31.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.746+0000 7f5857147640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58500d1920 con 0x7f58500d0040 2023-11-28T20:51:31.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.746+0000 7f5857147640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58500b8ae0 con 0x7f58500b2290 2023-11-28T20:51:31.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.746+0000 7f5856946640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f58500b1d40 0x7f58500d22f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35398/0 (socket says 172.21.15.45:35398) 2023-11-28T20:51:31.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.746+0000 7f5856946640 1 -- 172.21.15.45:0/1999521193 learned_addr learned my addr 172.21.15.45:0/1999521193 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:31.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.747+0000 7f5855944640 1 --2- 172.21.15.45:0/1999521193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500b2290 0x7f58500d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.747+0000 7f5855944640 1 -- 172.21.15.45:0/1999521193 >> v1:172.21.15.190:6789/0 conn(0x7f58500d0040 legacy=0x7f58500d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.747+0000 7f5855944640 1 -- 172.21.15.45:0/1999521193 >> v1:172.21.15.45:6789/0 conn(0x7f58500b1d40 legacy=0x7f58500d22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.747+0000 7f5855944640 1 -- 172.21.15.45:0/1999521193 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58500b8dc0 con 0x7f58500b2290 2023-11-28T20:51:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.749+0000 7f5855944640 1 --2- 172.21.15.45:0/1999521193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500b2290 0x7f58500d1db0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f58400097d0 tx=0x7f584002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69316dafb36c3760 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.751+0000 7f5855143640 1 -- 172.21.15.45:0/1999521193 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840030880 con 0x7f58500b2290 2023-11-28T20:51:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.751+0000 7f5855143640 1 -- 172.21.15.45:0/1999521193 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58400309e0 con 0x7f58500b2290 2023-11-28T20:51:31.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.751+0000 7f5855143640 1 -- 172.21.15.45:0/1999521193 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840030d70 con 0x7f58500b2290 2023-11-28T20:51:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.752+0000 7f5857147640 1 -- 172.21.15.45:0/1999521193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500b2290 msgr2=0x7f58500d1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.752+0000 7f5857147640 1 --2- 172.21.15.45:0/1999521193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500b2290 0x7f58500d1db0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f58400097d0 tx=0x7f584002b9c0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.752+0000 7f5857147640 1 -- 172.21.15.45:0/1999521193 shutdown_connections 2023-11-28T20:51:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.752+0000 7f5857147640 1 --2- 172.21.15.45:0/1999521193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500b2290 0x7f58500d1db0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.752+0000 7f5857147640 1 -- 172.21.15.45:0/1999521193 >> 172.21.15.45:0/1999521193 conn(0x7f5850070f40 msgr2=0x7f5850071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.753+0000 7f5857147640 1 -- 172.21.15.45:0/1999521193 shutdown_connections 2023-11-28T20:51:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.753+0000 7f5857147640 1 -- 172.21.15.45:0/1999521193 wait complete. 2023-11-28T20:51:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.754+0000 7f5857147640 1 Processor -- start 2023-11-28T20:51:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.754+0000 7f5857147640 1 -- start start 2023-11-28T20:51:31.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.755+0000 7f5857147640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58500b1d40 0x7f5850151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.755+0000 7f5857147640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58500b2290 0x7f5850158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.755+0000 7f5857147640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500d0040 0x7f58500b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.755+0000 7f5857147640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5850159760 con 0x7f58500b2290 2023-11-28T20:51:31.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.755+0000 7f5857147640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5850151f70 con 0x7f58500b1d40 2023-11-28T20:51:31.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.755+0000 7f5857147640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5850158b50 con 0x7f58500d0040 2023-11-28T20:51:31.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5855944640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58500b2290 0x7f5850158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5855944640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58500b2290 0x7f5850158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33040/0 (socket says 172.21.15.45:33040) 2023-11-28T20:51:31.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5855944640 1 -- 172.21.15.45:0/4162859236 learned_addr learned my addr 172.21.15.45:0/4162859236 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:31.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5856145640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58500b1d40 0x7f5850151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5856946640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500d0040 0x7f58500b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5855944640 1 -- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500d0040 msgr2=0x7f58500b7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5855944640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500d0040 0x7f58500b7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5855944640 1 -- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58500b1d40 msgr2=0x7f5850151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5855944640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58500b1d40 0x7f5850151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5855944640 1 -- 172.21.15.45:0/4162859236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5840009400 con 0x7f58500b2290 2023-11-28T20:51:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.756+0000 7f5856946640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500d0040 0x7f58500b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.757+0000 7f5855944640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58500b2290 0x7f5850158100 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f584002bfd0 tx=0x7f5840004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.757+0000 7f5846ffd640 1 -- 172.21.15.45:0/4162859236 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584003e040 con 0x7f58500b2290 2023-11-28T20:51:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.757+0000 7f5846ffd640 1 -- 172.21.15.45:0/4162859236 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5840030820 con 0x7f58500b2290 2023-11-28T20:51:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.757+0000 7f5846ffd640 1 -- 172.21.15.45:0/4162859236 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840043680 con 0x7f58500b2290 2023-11-28T20:51:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.757+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5850158e30 con 0x7f58500b2290 2023-11-28T20:51:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.757+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58501593f0 con 0x7f58500b2290 2023-11-28T20:51:31.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.758+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5850159910 con 0x7f58500b2290 2023-11-28T20:51:31.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.759+0000 7f5846ffd640 1 -- 172.21.15.45:0/4162859236 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 42) v1 ==== 89334+0+0 (secure 0 0 0) 0x7f58400423e0 con 0x7f58500b2290 2023-11-28T20:51:31.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.761+0000 7f5846ffd640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f583406af60 0x7f583406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:31.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.761+0000 7f5846ffd640 1 -- 172.21.15.45:0/4162859236 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6416+0+0 (secure 0 0 0) 0x7f58400ac080 con 0x7f58500b2290 2023-11-28T20:51:31.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.761+0000 7f5856145640 1 -- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f583406af60 msgr2=0x7f583406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:31.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.761+0000 7f5856145640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f583406af60 0x7f583406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.765+0000 7f5846ffd640 1 -- 172.21.15.45:0/4162859236 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5850159910 con 0x7f58500b2290 2023-11-28T20:51:31.843 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:31.841+0000 7f5b36e0e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:51:31.865 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:31.864+0000 7f55a5f53040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:51:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.957+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5850159910 con 0x7f58500b2290 2023-11-28T20:51:31.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.958+0000 7f5846ffd640 1 -- 172.21.15.45:0/4162859236 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5840034020 con 0x7f58500b2290 2023-11-28T20:51:31.960 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:51:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f583406af60 msgr2=0x7f583406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f583406af60 0x7f583406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58500b2290 msgr2=0x7f5850158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58500b2290 0x7f5850158100 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f584002bfd0 tx=0x7f5840004730 comp rx=0 tx=0).stop 2023-11-28T20:51:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 shutdown_connections 2023-11-28T20:51:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58500d0040 0x7f58500b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58500b2290 0x7f5850158100 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f583406af60 0x7f583406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 --2- 172.21.15.45:0/4162859236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58500b1d40 0x7f5850151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 >> 172.21.15.45:0/4162859236 conn(0x7f5850070f40 msgr2=0x7f585015c800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 shutdown_connections 2023-11-28T20:51:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:31.961+0000 7f5857147640 1 -- 172.21.15.45:0/4162859236 wait complete. 2023-11-28T20:51:31.972 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:31.971+0000 7f55a5f53040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:51:31.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:51:32.014 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:32.012+0000 7f5b36e0e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:51:32.022 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:32.021+0000 7f55a5f53040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:51:32.129 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:32.127+0000 7f55a5f53040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:51:32.131 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:32.129+0000 7f5b36e0e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:51:32.184 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:32.182+0000 7f5b36e0e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:51:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.189+0000 7ff060c75640 1 Processor -- start 2023-11-28T20:51:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.189+0000 7ff060c75640 1 -- start start 2023-11-28T20:51:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.191+0000 7ff060c75640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff05c158a70 0x7ff05c0cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.191+0000 7ff060c75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c0cff80 0x7ff05c15c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.191+0000 7ff060c75640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c0d03d0 con 0x7ff05c0cdfb0 2023-11-28T20:51:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.191+0000 7ff060c75640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05c15ec10 con 0x7ff05c158a70 2023-11-28T20:51:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.191+0000 7ff060c75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c15eeb0 con 0x7ff05c0cff80 2023-11-28T20:51:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.191+0000 7ff053fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff05c0cdfb0 0x7ff05c15b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35412/0 (socket says 172.21.15.45:35412) 2023-11-28T20:51:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.191+0000 7ff053fff640 1 -- 172.21.15.45:0/1473201766 learned_addr learned my addr 172.21.15.45:0/1473201766 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.191+0000 7ff05a59c640 1 --2- 172.21.15.45:0/1473201766 >> v2:172.21.15.190:3300/0 conn(0x7ff05c158a70 0x7ff05c0cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.192+0000 7ff05ad9d640 1 --2- 172.21.15.45:0/1473201766 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c0cff80 0x7ff05c15c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.192+0000 7ff059d9b640 1 -- 172.21.15.45:0/1473201766 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207540133 0 0) 0x7ff044001110 con 0x7ff05c0cdfb0 2023-11-28T20:51:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.192+0000 7ff059d9b640 1 -- 172.21.15.45:0/1473201766 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff048003520 con 0x7ff05c0cdfb0 2023-11-28T20:51:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.192+0000 7ff05a59c640 1 -- 172.21.15.45:0/1473201766 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c0cff80 msgr2=0x7ff05c15c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.192+0000 7ff05a59c640 1 --2- 172.21.15.45:0/1473201766 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c0cff80 0x7ff05c15c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.192+0000 7ff05a59c640 1 -- 172.21.15.45:0/1473201766 >> v1:172.21.15.45:6789/0 conn(0x7ff05c0cdfb0 legacy=0x7ff05c15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.192+0000 7ff059d9b640 1 -- 172.21.15.45:0/1473201766 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4234542961 0 0) 0x7ff048003520 con 0x7ff05c0cdfb0 2023-11-28T20:51:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.192+0000 7ff05a59c640 1 -- 172.21.15.45:0/1473201766 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c15f190 con 0x7ff05c158a70 2023-11-28T20:51:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.193+0000 7ff05a59c640 1 --2- 172.21.15.45:0/1473201766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff05c158a70 0x7ff05c0cfa40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0097a0 tx=0x7ff04c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=261598938f523f80 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.194+0000 7ff059d9b640 1 -- 172.21.15.45:0/1473201766 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff044001820 con 0x7ff05c158a70 2023-11-28T20:51:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.194+0000 7ff059d9b640 1 -- 172.21.15.45:0/1473201766 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04c02e050 con 0x7ff05c158a70 2023-11-28T20:51:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.194+0000 7ff059d9b640 1 -- 172.21.15.45:0/1473201766 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04c030710 con 0x7ff05c158a70 2023-11-28T20:51:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.195+0000 7ff060c75640 1 -- 172.21.15.45:0/1473201766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff05c158a70 msgr2=0x7ff05c0cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.195+0000 7ff060c75640 1 --2- 172.21.15.45:0/1473201766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff05c158a70 0x7ff05c0cfa40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0097a0 tx=0x7ff04c02b790 comp rx=0 tx=0).stop 2023-11-28T20:51:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.195+0000 7ff060c75640 1 -- 172.21.15.45:0/1473201766 shutdown_connections 2023-11-28T20:51:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.195+0000 7ff060c75640 1 --2- 172.21.15.45:0/1473201766 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c0cff80 0x7ff05c15c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.195+0000 7ff060c75640 1 --2- 172.21.15.45:0/1473201766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff05c158a70 0x7ff05c0cfa40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.195+0000 7ff060c75640 1 -- 172.21.15.45:0/1473201766 >> 172.21.15.45:0/1473201766 conn(0x7ff05c06a920 msgr2=0x7ff05c06ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.195+0000 7ff060c75640 1 -- 172.21.15.45:0/1473201766 shutdown_connections 2023-11-28T20:51:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.195+0000 7ff060c75640 1 -- 172.21.15.45:0/1473201766 wait complete. 2023-11-28T20:51:32.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.197+0000 7ff060c75640 1 Processor -- start 2023-11-28T20:51:32.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.197+0000 7ff060c75640 1 -- start start 2023-11-28T20:51:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff060c75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff05c0cdfb0 0x7ff05c0ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff060c75640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05c0cff80 0x7ff05c0b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff060c75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c158a70 0x7ff05c153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff060c75640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c0b3de0 con 0x7ff05c0cff80 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff060c75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c143e20 con 0x7ff05c0cdfb0 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff060c75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c153a50 con 0x7ff05c158a70 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff053fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05c0cff80 0x7ff05c0b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff053fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05c0cff80 0x7ff05c0b38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33056/0 (socket says 172.21.15.45:33056) 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff053fff640 1 -- 172.21.15.45:0/1648160830 learned_addr learned my addr 172.21.15.45:0/1648160830 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff05a59c640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff05c0cdfb0 0x7ff05c0ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff05ad9d640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c158a70 0x7ff05c153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.198+0000 7ff053fff640 1 -- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c158a70 msgr2=0x7ff05c153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff053fff640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c158a70 0x7ff05c153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff053fff640 1 -- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff05c0cdfb0 msgr2=0x7ff05c0ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff053fff640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff05c0cdfb0 0x7ff05c0ad290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff053fff640 1 -- 172.21.15.45:0/1648160830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c009400 con 0x7ff05c0cff80 2023-11-28T20:51:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff053fff640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05c0cff80 0x7ff05c0b38a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff04400c780 tx=0x7ff04400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff052ffd640 1 -- 172.21.15.45:0/1648160830 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff044005ad0 con 0x7ff05c0cff80 2023-11-28T20:51:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff052ffd640 1 -- 172.21.15.45:0/1648160830 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff044005c30 con 0x7ff05c0cff80 2023-11-28T20:51:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff052ffd640 1 -- 172.21.15.45:0/1648160830 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff044007070 con 0x7ff05c0cff80 2023-11-28T20:51:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05c0ad8a0 con 0x7ff05c0cff80 2023-11-28T20:51:32.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.199+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05c0ade60 con 0x7ff05c0cff80 2023-11-28T20:51:32.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.201+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05c143ff0 con 0x7ff05c0cff80 2023-11-28T20:51:32.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.201+0000 7ff052ffd640 1 -- 172.21.15.45:0/1648160830 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 42) v1 ==== 89334+0+0 (secure 0 0 0) 0x7ff0440071d0 con 0x7ff05c0cff80 2023-11-28T20:51:32.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.202+0000 7ff052ffd640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff03c06af60 0x7ff03c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:32.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.202+0000 7ff052ffd640 1 -- 172.21.15.45:0/1648160830 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6416+0+0 (secure 0 0 0) 0x7ff044088e60 con 0x7ff05c0cff80 2023-11-28T20:51:32.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.202+0000 7ff05a59c640 1 -- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff03c06af60 msgr2=0x7ff03c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:32.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.202+0000 7ff05a59c640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff03c06af60 0x7ff03c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:32.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.206+0000 7ff052ffd640 1 -- 172.21.15.45:0/1648160830 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff05c143ff0 con 0x7ff05c0cff80 2023-11-28T20:51:32.238 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:32.237+0000 7f55a5f53040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:51:32.301 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:32.299+0000 7f5b36e0e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:51:32.395 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:32.393+0000 7f55a5f53040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:51:32.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.403+0000 7ff05a59c640 1 -- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff03c06af60 msgr2=0x7ff03c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:32.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.403+0000 7ff05a59c640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff03c06af60 0x7ff03c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:51:32.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.411+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff05c0d03d0 con 0x7ff05c0cff80 2023-11-28T20:51:32.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.412+0000 7ff052ffd640 1 -- 172.21.15.45:0/1648160830 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 9 entries v103) v1 ==== 80+0+535 (secure 0 0 0) 0x7ff05c0d03d0 con 0x7ff05c0cff80 2023-11-28T20:51:32.414 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:32.412+0000 7f5b36e0e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:51:32.414 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3103043064 2023-11-29T20:51:27.664495+0000 2023-11-28T20:51:32.415 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/970930168 2023-11-29T20:51:27.664495+0000 2023-11-28T20:51:32.415 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3436414960 2023-11-29T20:51:27.664495+0000 2023-11-28T20:51:32.415 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3227357091 2023-11-29T20:51:27.664495+0000 2023-11-28T20:51:32.415 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/970930168 2023-11-29T20:51:27.664495+0000 2023-11-28T20:51:32.415 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/4025400682 2023-11-29T20:51:27.664495+0000 2023-11-28T20:51:32.415 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2627631957 2023-11-29T20:51:27.664495+0000 2023-11-28T20:51:32.415 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2067877259 2023-11-29T20:51:27.664495+0000 2023-11-28T20:51:32.415 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1852415784 2023-11-28T21:51:13.642235+0000 2023-11-28T20:51:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:listed 9 entries 2023-11-28T20:51:32.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff03c06af60 msgr2=0x7ff03c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:32.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff03c06af60 0x7ff03c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05c0cff80 msgr2=0x7ff05c0b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:32.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05c0cff80 0x7ff05c0b38a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff04400c780 tx=0x7ff04400cc40 comp rx=0 tx=0).stop 2023-11-28T20:51:32.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 shutdown_connections 2023-11-28T20:51:32.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff05c158a70 0x7ff05c153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05c0cff80 0x7ff05c0b38a0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff03c06af60 0x7ff03c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 --2- 172.21.15.45:0/1648160830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff05c0cdfb0 0x7ff05c0ad290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:32.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 >> 172.21.15.45:0/1648160830 conn(0x7ff05c06a920 msgr2=0x7ff05c15b760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:32.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 shutdown_connections 2023-11-28T20:51:32.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:32.415+0000 7ff060c75640 1 -- 172.21.15.45:0/1648160830 wait complete. 2023-11-28T20:51:32.433 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T20:51:32.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:32.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T20:51:32.443 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:32.442+0000 7f55a5f53040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:51:32.562 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:32.561+0000 7f55a5f53040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:51:32.581 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:51:32.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:51:32.581 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T20:51:32.589 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:32.587+0000 7f5b36e0e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:51:32.635 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:51:32.635 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:32.645 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:32.643+0000 7f5b36e0e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:51:32.690 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:51:32.690 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:51:32.690 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:32.747 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:51:32.748 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:51:32.748 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:32.776 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:32.773+0000 7f5b36e0e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:51:32.802 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:51:32.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:33.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.012+0000 7ff152ad7640 1 Processor -- start 2023-11-28T20:51:33.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.012+0000 7ff152ad7640 1 -- start start 2023-11-28T20:51:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.014+0000 7ff152ad7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff14c15b850 0x7ff14c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.014+0000 7ff152ad7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c15bda0 con 0x7ff14c0b35b0 2023-11-28T20:51:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.014+0000 7ff152ad7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c15c080 con 0x7ff14c0b1fa0 2023-11-28T20:51:33.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.014+0000 7ff152ad7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c15ee40 con 0x7ff14c15b850 2023-11-28T20:51:33.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.014+0000 7ff150873640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff14c0b35b0 0x7ff14c15ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35414/0 (socket says 172.21.15.45:35414) 2023-11-28T20:51:33.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.014+0000 7ff150873640 1 -- 172.21.15.45:0/2914784783 learned_addr learned my addr 172.21.15.45:0/2914784783 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:33.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14bfff640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff14c15b850 0x7ff14c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673721507 0 0) 0x7ff14c15bda0 con 0x7ff14c0b35b0 2023-11-28T20:51:33.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c003520 con 0x7ff14c0b35b0 2023-11-28T20:51:33.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459102838 0 0) 0x7ff14c15c080 con 0x7ff14c0b1fa0 2023-11-28T20:51:33.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14c15bda0 con 0x7ff14c0b1fa0 2023-11-28T20:51:33.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1987581740 0 0) 0x7ff13c003520 con 0x7ff14c0b35b0 2023-11-28T20:51:33.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff14c15b850 msgr2=0x7ff14c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:33.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff14c15b850 0x7ff14c0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 >> v1:172.21.15.190:6789/0 conn(0x7ff14c0b1fa0 legacy=0x7ff14c15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c15c360 con 0x7ff14c0b35b0 2023-11-28T20:51:33.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ff140001670 con 0x7ff14c0b35b0 2023-11-28T20:51:33.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 >> v1:172.21.15.45:6789/0 conn(0x7ff14c0b35b0 legacy=0x7ff14c15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.015+0000 7ff14b7fe640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff13c003b30 0x7ff13c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff14b7fe640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff14c15b850 0x7ff13c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff14b7fe640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff13c004d00 0x7ff13c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c15ee40 con 0x7ff13c004d00 2023-11-28T20:51:33.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c00ba30 con 0x7ff13c003b30 2023-11-28T20:51:33.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c00bbf0 con 0x7ff14c15b850 2023-11-28T20:51:33.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff151074640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff13c004d00 0x7ff13c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff150873640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff13c003b30 0x7ff13c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff14bfff640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff14c15b850 0x7ff13c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff151074640 1 -- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff14c15b850 msgr2=0x7ff13c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:33.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff151074640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff14c15b850 0x7ff13c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff151074640 1 -- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff13c003b30 msgr2=0x7ff13c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:33.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff151074640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff13c003b30 0x7ff13c003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.016+0000 7ff151074640 1 -- 172.21.15.45:0/2914784783 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c003520 con 0x7ff13c004d00 2023-11-28T20:51:33.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.017+0000 7ff151074640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff13c004d00 0x7ff13c005180 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7ff144001fa0 tx=0x7ff13c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3feafc1fb9e397 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:33.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.018+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144003b70 con 0x7ff13c004d00 2023-11-28T20:51:33.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.018+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff144003cd0 con 0x7ff13c004d00 2023-11-28T20:51:33.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.018+0000 7ff14b7fe640 1 -- 172.21.15.45:0/2914784783 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1440052f0 con 0x7ff13c004d00 2023-11-28T20:51:33.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.018+0000 7ff152ad7640 1 -- 172.21.15.45:0/2914784783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff13c004d00 msgr2=0x7ff13c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.018+0000 7ff152ad7640 1 --2- 172.21.15.45:0/2914784783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff13c004d00 0x7ff13c005180 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7ff144001fa0 tx=0x7ff13c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.018+0000 7ff151074640 1 -- 172.21.15.45:0/2914784783 reap_dead start 2023-11-28T20:51:33.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.018+0000 7ff152ad7640 1 -- 172.21.15.45:0/2914784783 shutdown_connections 2023-11-28T20:51:33.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.018+0000 7ff152ad7640 1 -- 172.21.15.45:0/2914784783 >> 172.21.15.45:0/2914784783 conn(0x7ff14c06a500 msgr2=0x7ff14c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:33.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.019+0000 7ff152ad7640 1 -- 172.21.15.45:0/2914784783 shutdown_connections 2023-11-28T20:51:33.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.019+0000 7ff152ad7640 1 -- 172.21.15.45:0/2914784783 wait complete. 2023-11-28T20:51:33.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.020+0000 7ff152ad7640 1 Processor -- start 2023-11-28T20:51:33.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.020+0000 7ff152ad7640 1 -- start start 2023-11-28T20:51:33.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.021+0000 7ff152ad7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff13c004d00 0x7ff14c1518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.021+0000 7ff152ad7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff14c151df0 0x7ff14c1f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.021+0000 7ff152ad7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff14c1f3880 0x7ff14c1f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.021+0000 7ff152ad7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c148e30 con 0x7ff14c1f3880 2023-11-28T20:51:33.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.021+0000 7ff152ad7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c152270 con 0x7ff14c151df0 2023-11-28T20:51:33.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.021+0000 7ff152ad7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c1f4240 con 0x7ff13c004d00 2023-11-28T20:51:33.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff151074640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff14c1f3880 0x7ff14c1f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff151074640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff14c1f3880 0x7ff14c1f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33082/0 (socket says 172.21.15.45:33082) 2023-11-28T20:51:33.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff151074640 1 -- 172.21.15.45:0/1190270905 learned_addr learned my addr 172.21.15.45:0/1190270905 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:33.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff150873640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff13c004d00 0x7ff14c1518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff14bfff640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff14c151df0 0x7ff14c1f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff151074640 1 -- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff13c004d00 msgr2=0x7ff14c1518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff151074640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff13c004d00 0x7ff14c1518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff151074640 1 -- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff14c151df0 msgr2=0x7ff14c1f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff151074640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff14c151df0 0x7ff14c1f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.022+0000 7ff151074640 1 -- 172.21.15.45:0/1190270905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0ac8b0 con 0x7ff14c1f3880 2023-11-28T20:51:33.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.023+0000 7ff151074640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff14c1f3880 0x7ff14c1f3d00 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7ff144005040 tx=0x7ff144001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:33.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.023+0000 7ff1497fa640 1 -- 172.21.15.45:0/1190270905 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144005600 con 0x7ff14c1f3880 2023-11-28T20:51:33.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.023+0000 7ff1497fa640 1 -- 172.21.15.45:0/1190270905 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff144005760 con 0x7ff14c1f3880 2023-11-28T20:51:33.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.023+0000 7ff1497fa640 1 -- 172.21.15.45:0/1190270905 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14402c740 con 0x7ff14c1f3880 2023-11-28T20:51:33.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.023+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c152470 con 0x7ff14c1f3880 2023-11-28T20:51:33.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.023+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c152a30 con 0x7ff14c1f3880 2023-11-28T20:51:33.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.024+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c0aca40 con 0x7ff14c1f3880 2023-11-28T20:51:33.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.025+0000 7ff1497fa640 1 -- 172.21.15.45:0/1190270905 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 42) v1 ==== 89334+0+0 (secure 0 0 0) 0x7ff14402c8a0 con 0x7ff14c1f3880 2023-11-28T20:51:33.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.026+0000 7ff1497fa640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff13006af90 0x7ff13006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.026+0000 7ff1497fa640 1 -- 172.21.15.45:0/1190270905 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6416+0+0 (secure 0 0 0) 0x7ff1440aa320 con 0x7ff14c1f3880 2023-11-28T20:51:33.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.026+0000 7ff150873640 1 -- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff13006af90 msgr2=0x7ff13006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:33.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.026+0000 7ff150873640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff13006af90 0x7ff13006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:33.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.030+0000 7ff1497fa640 1 -- 172.21.15.45:0/1190270905 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff144075140 con 0x7ff14c1f3880 2023-11-28T20:51:33.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.227+0000 7ff150873640 1 -- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff13006af90 msgr2=0x7ff13006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:33.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.227+0000 7ff150873640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff13006af90 0x7ff13006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:51:33.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.261+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff14c0aca40 con 0x7ff14c1f3880 2023-11-28T20:51:33.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.262+0000 7ff1497fa640 1 -- 172.21.15.45:0/1190270905 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4918 (secure 0 0 0) 0x7ff144078760 con 0x7ff14c1f3880 2023-11-28T20:51:33.264 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:33.265 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3492018158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3492018158},{"type":"v1","addr":"172.21.15.45:6833","nonce":3492018158}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6121,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2276329291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2276329291},{"type":"v1","addr":"172.21.15.190:6835","nonce":2276329291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6130,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1261857840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1261857840},{"type":"v1","addr":"172.21.15.190:6837","nonce":1261857840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:12.953862+0000","modified":"2023-11-28T20:51:16.015607+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7179},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7179":{"gid":7179,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/862206408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":862206408},{"type":"v1","addr":"172.21.15.45:6835","nonce":862206408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-28T20:51:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 44 2023-11-28T20:51:33.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff13006af90 msgr2=0x7ff13006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:33.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff13006af90 0x7ff13006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff14c1f3880 msgr2=0x7ff14c1f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff14c1f3880 0x7ff14c1f3d00 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7ff144005040 tx=0x7ff144001650 comp rx=0 tx=0).stop 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 shutdown_connections 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff14c1f3880 0x7ff14c1f3d00 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7ff13006af90 0x7ff13006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff14c151df0 0x7ff14c1f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 --2- 172.21.15.45:0/1190270905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff13c004d00 0x7ff14c1518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 >> 172.21.15.45:0/1190270905 conn(0x7ff14c06a500 msgr2=0x7ff14c15a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 shutdown_connections 2023-11-28T20:51:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.265+0000 7ff152ad7640 1 -- 172.21.15.45:0/1190270905 wait complete. 2023-11-28T20:51:33.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:33.397 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.395+0000 7f55a5f53040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:51:33.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.480+0000 7f93994f8640 1 Processor -- start 2023-11-28T20:51:33.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.480+0000 7f93994f8640 1 -- start start 2023-11-28T20:51:33.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.481+0000 7f93994f8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940ce800 0x7f9394165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.481+0000 7f93994f8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939415c3c0 con 0x7f9394155780 2023-11-28T20:51:33.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.482+0000 7f93994f8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9394155b60 con 0x7f93940ae540 2023-11-28T20:51:33.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.482+0000 7f93994f8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939415ba90 con 0x7f93940ce800 2023-11-28T20:51:33.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.482+0000 7f9393fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9394155780 0x7f93940abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35426/0 (socket says 172.21.15.45:35426) 2023-11-28T20:51:33.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.482+0000 7f9393fff640 1 -- 172.21.15.45:0/2873817195 learned_addr learned my addr 172.21.15.45:0/2873817195 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:33.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.482+0000 7f9398cf7640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940ce800 0x7f9394165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.482+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039755250 0 0) 0x7f939415c3c0 con 0x7f9394155780 2023-11-28T20:51:33.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.482+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9380003520 con 0x7f9394155780 2023-11-28T20:51:33.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 663095254 0 0) 0x7f9380003520 con 0x7f9394155780 2023-11-28T20:51:33.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940ce800 msgr2=0x7f9394165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:33.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940ce800 0x7f9394165e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 >> v1:172.21.15.190:6789/0 conn(0x7f93940ae540 legacy=0x7f93940ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939415bd70 con 0x7f9394155780 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f9384001670 con 0x7f9394155780 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 >> v1:172.21.15.45:6789/0 conn(0x7f9394155780 legacy=0x7f93940abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93800039a0 0x7f9380003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800047c0 0x7f9380004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940ce800 0x7f9380005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939415ba90 con 0x7f93800039a0 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938000fe40 con 0x7f93800047c0 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9380010000 con 0x7f93940ce800 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f9393fff640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93800039a0 0x7f9380003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.483+0000 7f938bfff640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800047c0 0x7f9380004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.484+0000 7f9398cf7640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940ce800 0x7f9380005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.484+0000 7f9393fff640 1 -- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940ce800 msgr2=0x7f9380005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.484+0000 7f9393fff640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940ce800 0x7f9380005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.484+0000 7f9393fff640 1 -- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800047c0 msgr2=0x7f9380004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.484+0000 7f9393fff640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800047c0 0x7f9380004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.484+0000 7f9393fff640 1 -- 172.21.15.45:0/2873817195 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9380003520 con 0x7f93800039a0 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.485+0000 7f9393fff640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93800039a0 0x7f9380003d80 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f938400c160 tx=0x7f93800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19f4a9a3b3b117d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.485+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384005ad0 con 0x7f93800039a0 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.485+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9384005c30 con 0x7f93800039a0 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.485+0000 7f93937fe640 1 -- 172.21.15.45:0/2873817195 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938402c7c0 con 0x7f93800039a0 2023-11-28T20:51:33.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.486+0000 7f93994f8640 1 -- 172.21.15.45:0/2873817195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93800039a0 msgr2=0x7f9380003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.486+0000 7f93994f8640 1 --2- 172.21.15.45:0/2873817195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93800039a0 0x7f9380003d80 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f938400c160 tx=0x7f93800057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.486+0000 7f9398cf7640 1 -- 172.21.15.45:0/2873817195 reap_dead start 2023-11-28T20:51:33.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.486+0000 7f93994f8640 1 -- 172.21.15.45:0/2873817195 shutdown_connections 2023-11-28T20:51:33.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.486+0000 7f93994f8640 1 -- 172.21.15.45:0/2873817195 >> 172.21.15.45:0/2873817195 conn(0x7f939406ab30 msgr2=0x7f93940b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:33.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.486+0000 7f93994f8640 1 -- 172.21.15.45:0/2873817195 shutdown_connections 2023-11-28T20:51:33.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.486+0000 7f93994f8640 1 -- 172.21.15.45:0/2873817195 wait complete. 2023-11-28T20:51:33.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.488+0000 7f93994f8640 1 Processor -- start 2023-11-28T20:51:33.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.488+0000 7f93994f8640 1 -- start start 2023-11-28T20:51:33.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f93994f8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800039a0 0x7f939413d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f93994f8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940af410 0x7f939409a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f93994f8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940af890 0x7f939409a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f93994f8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939414cf20 con 0x7f93940af410 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f93994f8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93940b0370 con 0x7f93800039a0 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f93994f8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939413dce0 con 0x7f93940af890 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f938bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940af410 0x7f939409a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f938bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940af410 0x7f939409a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33098/0 (socket says 172.21.15.45:33098) 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f938bfff640 1 -- 172.21.15.45:0/2675642421 learned_addr learned my addr 172.21.15.45:0/2675642421 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.489+0000 7f9393fff640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800039a0 0x7f939413d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f9398cf7640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940af890 0x7f939409a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f938bfff640 1 -- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940af890 msgr2=0x7f939409a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f938bfff640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940af890 0x7f939409a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f938bfff640 1 -- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800039a0 msgr2=0x7f939413d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f938bfff640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800039a0 0x7f939413d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f938bfff640 1 -- 172.21.15.45:0/2675642421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93940afdb0 con 0x7f93940af410 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f9393fff640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800039a0 0x7f939413d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f938bfff640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940af410 0x7f939409a350 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f937c00f830 tx=0x7f937c00fcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f93917fa640 1 -- 172.21.15.45:0/2675642421 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937c0014c0 con 0x7f93940af410 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f93917fa640 1 -- 172.21.15.45:0/2675642421 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937c0052e0 con 0x7f93940af410 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f93917fa640 1 -- 172.21.15.45:0/2675642421 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937c00d870 con 0x7f93940af410 2023-11-28T20:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.490+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93940b0090 con 0x7f93940af410 2023-11-28T20:51:33.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.491+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9394151030 con 0x7f93940af410 2023-11-28T20:51:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.492+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939404e920 con 0x7f93940af410 2023-11-28T20:51:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.493+0000 7f93917fa640 1 -- 172.21.15.45:0/2675642421 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 42) v1 ==== 89334+0+0 (secure 0 0 0) 0x7f937c00d9d0 con 0x7f93940af410 2023-11-28T20:51:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.494+0000 7f93917fa640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f937406af60 0x7f937406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.494+0000 7f93917fa640 1 -- 172.21.15.45:0/2675642421 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6416+0+0 (secure 0 0 0) 0x7f937c089a60 con 0x7f93940af410 2023-11-28T20:51:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.494+0000 7f9393fff640 1 -- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f937406af60 msgr2=0x7f937406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:33.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.494+0000 7f9393fff640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f937406af60 0x7f937406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:51:33.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.498+0000 7f93917fa640 1 -- 172.21.15.45:0/2675642421 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f937c054750 con 0x7f93940af410 2023-11-28T20:51:33.640 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:33.638+0000 7f5b36e0e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:51:33.691 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.689+0000 7f55a5f53040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:51:33.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.692+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f939406c100 con 0x7f93940af410 2023-11-28T20:51:33.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.695+0000 7f93917fa640 1 -- 172.21.15.45:0/2675642421 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+17346 (secure 0 0 0) 0x7f937c057d70 con 0x7f93940af410 2023-11-28T20:51:33.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.695+0000 7f9393fff640 1 -- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f937406af60 msgr2=0x7f937406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1373961985 2023-11-28T20:51:33.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.695+0000 7f9393fff640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f937406af60 0x7f937406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:51:33.698 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:33.698 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":103,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:27.664546+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:10.974716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:11.975222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3103043064":"2023-11-29T20:51:27.664495+0000","172.21.15.190:6832/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3436414960":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3227357091":"2023-11-29T20:51:27.664495+0000","172.21.15.190:6833/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/4025400682":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2627631957":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2067877259":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/1852415784":"2023-11-28T21:51:13.642235+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:33.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f937406af60 msgr2=0x7f937406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:51:33.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f937406af60 0x7f937406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940af410 msgr2=0x7f939409a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940af410 0x7f939409a350 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f937c00f830 tx=0x7f937c00fcf0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 shutdown_connections 2023-11-28T20:51:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93940af890 0x7f939409a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93940af410 0x7f939409a350 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:6832/1373961985,v1:172.21.15.190:6833/1373961985] conn(0x7f937406af60 0x7f937406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 --2- 172.21.15.45:0/2675642421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93800039a0 0x7f939413d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 >> 172.21.15.45:0/2675642421 conn(0x7f939406ab30 msgr2=0x7f939415d510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:33.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 shutdown_connections 2023-11-28T20:51:33.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.698+0000 7f93994f8640 1 -- 172.21.15.45:0/2675642421 wait complete. 2023-11-28T20:51:33.719 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:51:10.974716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:51:11.975222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:51:33.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:51:33.747 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.746+0000 7f55a5f53040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:51:33.861 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.860+0000 7f55a5f53040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:51:33.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.924+0000 7faa0a547640 1 Processor -- start 2023-11-28T20:51:33.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.924+0000 7faa0a547640 1 -- start start 2023-11-28T20:51:33.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.925+0000 7faa0a547640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b2290 0x7faa040d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.925+0000 7faa0a547640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa040b90a0 con 0x7faa040b1d40 2023-11-28T20:51:33.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.925+0000 7faa0a547640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa040d1920 con 0x7faa040d0040 2023-11-28T20:51:33.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.925+0000 7faa0a547640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa040b8ae0 con 0x7faa040b2290 2023-11-28T20:51:33.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.926+0000 7faa08d44640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b2290 0x7faa040d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.926+0000 7faa09d46640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faa040b1d40 0x7faa040d22f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54890/0 (socket says 172.21.15.45:54890) 2023-11-28T20:51:33.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.926+0000 7faa09545640 1 --1- 172.21.15.45:0/3794947668 >> v1:172.21.15.190:6789/0 conn(0x7faa040d0040 0x7faa040d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:33750/0 (socket says 172.21.15.45:33750) 2023-11-28T20:51:33.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.926+0000 7faa09d46640 1 -- 172.21.15.45:0/3794947668 learned_addr learned my addr 172.21.15.45:0/3794947668 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.926+0000 7faa08d44640 1 -- 172.21.15.45:0/3794947668 >> v1:172.21.15.190:6789/0 conn(0x7faa040d0040 legacy=0x7faa040d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.926+0000 7faa08d44640 1 -- 172.21.15.45:0/3794947668 >> v1:172.21.15.45:6789/0 conn(0x7faa040b1d40 legacy=0x7faa040d22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.926+0000 7faa08d44640 1 -- 172.21.15.45:0/3794947668 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa040b8dc0 con 0x7faa040b2290 2023-11-28T20:51:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.926+0000 7faa03fff640 1 -- 172.21.15.45:0/3794947668 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370714605 0 0) 0x7faa040b90a0 con 0x7faa040b1d40 2023-11-28T20:51:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.927+0000 7faa08d44640 1 --2- 172.21.15.45:0/3794947668 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b2290 0x7faa040d1db0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009750 tx=0x7fa9f802b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0335c77b72ca2f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.928+0000 7faa03fff640 1 -- 172.21.15.45:0/3794947668 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8031710 con 0x7faa040b2290 2023-11-28T20:51:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.928+0000 7faa03fff640 1 -- 172.21.15.45:0/3794947668 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f802f050 con 0x7faa040b2290 2023-11-28T20:51:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.928+0000 7faa03fff640 1 -- 172.21.15.45:0/3794947668 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8031a90 con 0x7faa040b2290 2023-11-28T20:51:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.929+0000 7faa0a547640 1 -- 172.21.15.45:0/3794947668 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b2290 msgr2=0x7faa040d1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.929+0000 7faa0a547640 1 --2- 172.21.15.45:0/3794947668 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b2290 0x7faa040d1db0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009750 tx=0x7fa9f802b7b0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.929+0000 7faa0a547640 1 -- 172.21.15.45:0/3794947668 shutdown_connections 2023-11-28T20:51:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.929+0000 7faa0a547640 1 --2- 172.21.15.45:0/3794947668 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b2290 0x7faa040d1db0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.929+0000 7faa0a547640 1 -- 172.21.15.45:0/3794947668 >> 172.21.15.45:0/3794947668 conn(0x7faa04070f40 msgr2=0x7faa04071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.929+0000 7faa0a547640 1 -- 172.21.15.45:0/3794947668 shutdown_connections 2023-11-28T20:51:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.929+0000 7faa0a547640 1 -- 172.21.15.45:0/3794947668 wait complete. 2023-11-28T20:51:33.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.931+0000 7faa0a547640 1 Processor -- start 2023-11-28T20:51:33.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.931+0000 7faa0a547640 1 -- start start 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa0a547640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b1d40 0x7faa041465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa0a547640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa040b2290 0x7faa041f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa0a547640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa040d0040 0x7faa040cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa0a547640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa040c94e0 con 0x7faa040b2290 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa0a547640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa04156fc0 con 0x7faa040d0040 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa0a547640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa04146b00 con 0x7faa040b1d40 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa08d44640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa040b2290 0x7faa041f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa09545640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b1d40 0x7faa041465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa09d46640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa040d0040 0x7faa040cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa09545640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b1d40 0x7faa041465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:55950/0 (socket says 172.21.15.45:55950) 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa09545640 1 -- 172.21.15.45:0/3448455512 learned_addr learned my addr 172.21.15.45:0/3448455512 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa09545640 1 -- 172.21.15.45:0/3448455512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa040d0040 msgr2=0x7faa040cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa09545640 1 --2- 172.21.15.45:0/3448455512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa040d0040 0x7faa040cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa09545640 1 -- 172.21.15.45:0/3448455512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa040b2290 msgr2=0x7faa041f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa09545640 1 --2- 172.21.15.45:0/3448455512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa040b2290 0x7faa041f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.932+0000 7faa09545640 1 -- 172.21.15.45:0/3448455512 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f8009400 con 0x7faa040b1d40 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.933+0000 7faa08d44640 1 --2- 172.21.15.45:0/3448455512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa040b2290 0x7faa041f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.933+0000 7faa09545640 1 --2- 172.21.15.45:0/3448455512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b1d40 0x7faa041465c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400c780 tx=0x7fa9f400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.933+0000 7faa01ffb640 1 -- 172.21.15.45:0/3448455512 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40072d0 con 0x7faa040b1d40 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.933+0000 7faa01ffb640 1 -- 172.21.15.45:0/3448455512 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f4007450 con 0x7faa040b1d40 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.933+0000 7faa01ffb640 1 -- 172.21.15.45:0/3448455512 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40047e0 con 0x7faa040b1d40 2023-11-28T20:51:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.933+0000 7faa0a547640 1 -- 172.21.15.45:0/3448455512 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa040c8440 con 0x7faa040b1d40 2023-11-28T20:51:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.933+0000 7faa0a547640 1 -- 172.21.15.45:0/3448455512 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa040c8a00 con 0x7faa040b1d40 2023-11-28T20:51:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.934+0000 7faa0a547640 1 -- 172.21.15.45:0/3448455512 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa04071b30 con 0x7faa040b1d40 2023-11-28T20:51:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.935+0000 7faa01ffb640 1 -- 172.21.15.45:0/3448455512 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 43) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fa9f4004960 con 0x7faa040b1d40 2023-11-28T20:51:33.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.936+0000 7faa01ffb640 1 -- 172.21.15.45:0/3448455512 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6760+0+0 (secure 0 0 0) 0x7fa9f4088880 con 0x7faa040b1d40 2023-11-28T20:51:33.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:33.942+0000 7faa01ffb640 1 -- 172.21.15.45:0/3448455512 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9f4053890 con 0x7faa040b1d40 2023-11-28T20:51:33.958 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.957+0000 7f55289b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.958 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.957+0000 7f55289b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.958 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.957+0000 7f55289b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.958 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.957+0000 7f55289b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.958 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.957+0000 7f55289b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.959 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.958+0000 7f552a9b5640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.959 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.958+0000 7f552a9b5640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.959 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.958+0000 7f552a9b5640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.959 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.958+0000 7f552a9b5640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.959 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:33.958+0000 7f552a9b5640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:51:33.968 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:33.966+0000 7f5b36e0e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:51:34.028 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:34.025+0000 7f5b36e0e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:51:34.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.134+0000 7faa0a547640 1 -- 172.21.15.45:0/3448455512 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa0404e920 con 0x7faa040b1d40 2023-11-28T20:51:34.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.136+0000 7faa01ffb640 1 -- 172.21.15.45:0/3448455512 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9f4018020 con 0x7faa040b1d40 2023-11-28T20:51:34.138 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:51:34.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.138+0000 7faa0a547640 1 -- 172.21.15.45:0/3448455512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b1d40 msgr2=0x7faa041465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:34.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.138+0000 7faa0a547640 1 --2- 172.21.15.45:0/3448455512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b1d40 0x7faa041465c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400c780 tx=0x7fa9f400cc40 comp rx=0 tx=0).stop 2023-11-28T20:51:34.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.138+0000 7faa0a547640 1 -- 172.21.15.45:0/3448455512 shutdown_connections 2023-11-28T20:51:34.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.138+0000 7faa0a547640 1 --2- 172.21.15.45:0/3448455512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa040d0040 0x7faa040cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.138+0000 7faa0a547640 1 --2- 172.21.15.45:0/3448455512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa040b2290 0x7faa041f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.138+0000 7faa0a547640 1 --2- 172.21.15.45:0/3448455512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa040b1d40 0x7faa041465c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.138+0000 7faa0a547640 1 -- 172.21.15.45:0/3448455512 >> 172.21.15.45:0/3448455512 conn(0x7faa04070f40 msgr2=0x7faa040b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:34.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.138+0000 7faa0a547640 1 -- 172.21.15.45:0/3448455512 shutdown_connections 2023-11-28T20:51:34.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.138+0000 7faa0a547640 1 -- 172.21.15.45:0/3448455512 wait complete. 2023-11-28T20:51:34.148 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:51:34.146+0000 7f5b36e0e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:51:34.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:51:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.373+0000 7f3f04bbc640 1 Processor -- start 2023-11-28T20:51:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.373+0000 7f3f04bbc640 1 -- start start 2023-11-28T20:51:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.375+0000 7f3f04bbc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f00153fc0 0x7f3f001543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.375+0000 7f3f04bbc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f0006b510 0x7f3f0006b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.375+0000 7f3f04bbc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0015da00 con 0x7f3f000b2e10 2023-11-28T20:51:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.375+0000 7f3f04bbc640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f001548e0 con 0x7f3f00153fc0 2023-11-28T20:51:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.375+0000 7f3f04bbc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0006bed0 con 0x7f3f0006b510 2023-11-28T20:51:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.375+0000 7f3eff7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f00153fc0 0x7f3f001543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.375+0000 7f3eff7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3f00153fc0 0x7f3f001543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58286/0 (socket says 172.21.15.45:58286) 2023-11-28T20:51:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.375+0000 7f3eff7fe640 1 -- 172.21.15.45:0/265663155 learned_addr learned my addr 172.21.15.45:0/265663155 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.376+0000 7f3efeffd640 1 --2- 172.21.15.45:0/265663155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f0006b510 0x7f3f0006b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.376+0000 7f3eff7fe640 1 -- 172.21.15.45:0/265663155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f0006b510 msgr2=0x7f3f0006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:34.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.376+0000 7f3eff7fe640 1 --2- 172.21.15.45:0/265663155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f0006b510 0x7f3f0006b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.376+0000 7f3eff7fe640 1 -- 172.21.15.45:0/265663155 >> v1:172.21.15.190:6789/0 conn(0x7f3f000b2e10 legacy=0x7f3f000b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:34.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.376+0000 7f3eff7fe640 1 -- 172.21.15.45:0/265663155 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0015dce0 con 0x7f3f00153fc0 2023-11-28T20:51:34.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.376+0000 7f3efeffd640 1 --2- 172.21.15.45:0/265663155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f0006b510 0x7f3f0006b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.377+0000 7f3eff7fe640 1 --2- 172.21.15.45:0/265663155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f00153fc0 0x7f3f001543a0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00098a0 tx=0x7f3ef002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d12236b0f87132 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.377+0000 7f3efe7fc640 1 -- 172.21.15.45:0/265663155 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0031640 con 0x7f3f00153fc0 2023-11-28T20:51:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.377+0000 7f3efe7fc640 1 -- 172.21.15.45:0/265663155 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef00317a0 con 0x7f3f00153fc0 2023-11-28T20:51:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.377+0000 7f3efe7fc640 1 -- 172.21.15.45:0/265663155 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0031b30 con 0x7f3f00153fc0 2023-11-28T20:51:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.378+0000 7f3f04bbc640 1 -- 172.21.15.45:0/265663155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f00153fc0 msgr2=0x7f3f001543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.378+0000 7f3f04bbc640 1 --2- 172.21.15.45:0/265663155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f00153fc0 0x7f3f001543a0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00098a0 tx=0x7f3ef002b8f0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.378+0000 7f3f04bbc640 1 -- 172.21.15.45:0/265663155 shutdown_connections 2023-11-28T20:51:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.378+0000 7f3f04bbc640 1 --2- 172.21.15.45:0/265663155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f0006b510 0x7f3f0006b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.378+0000 7f3f04bbc640 1 --2- 172.21.15.45:0/265663155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f00153fc0 0x7f3f001543a0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.378+0000 7f3f04bbc640 1 -- 172.21.15.45:0/265663155 >> 172.21.15.45:0/265663155 conn(0x7f3f000712d0 msgr2=0x7f3f000716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.378+0000 7f3f04bbc640 1 -- 172.21.15.45:0/265663155 shutdown_connections 2023-11-28T20:51:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.378+0000 7f3f04bbc640 1 -- 172.21.15.45:0/265663155 wait complete. 2023-11-28T20:51:34.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.380+0000 7f3f04bbc640 1 Processor -- start 2023-11-28T20:51:34.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.380+0000 7f3f04bbc640 1 -- start start 2023-11-28T20:51:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3f04bbc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0006b510 0x7f3f00142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3f04bbc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000b2e10 0x7f3f0014c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3f04bbc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f00153fc0 0x7f3f000cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3f04bbc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0014b230 con 0x7f3f000b2e10 2023-11-28T20:51:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3f04bbc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f001444f0 con 0x7f3f0006b510 2023-11-28T20:51:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3f04bbc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0014b480 con 0x7f3f00153fc0 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3efeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000b2e10 0x7f3f0014c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3efeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000b2e10 0x7f3f0014c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58300/0 (socket says 172.21.15.45:58300) 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3efeffd640 1 -- 172.21.15.45:0/227668568 learned_addr learned my addr 172.21.15.45:0/227668568 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.381+0000 7f3eff7fe640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0006b510 0x7f3f00142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3efffff640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f00153fc0 0x7f3f000cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3efeffd640 1 -- 172.21.15.45:0/227668568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f00153fc0 msgr2=0x7f3f000cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3efeffd640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f00153fc0 0x7f3f000cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3efeffd640 1 -- 172.21.15.45:0/227668568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0006b510 msgr2=0x7f3f00142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3efeffd640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0006b510 0x7f3f00142ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3efeffd640 1 -- 172.21.15.45:0/227668568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef00094d0 con 0x7f3f000b2e10 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3efffff640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f00153fc0 0x7f3f000cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:34.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3efeffd640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000b2e10 0x7f3f0014c520 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c7f0 tx=0x7f3ef400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3ee7fff640 1 -- 172.21.15.45:0/227668568 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4007820 con 0x7f3f000b2e10 2023-11-28T20:51:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3ee7fff640 1 -- 172.21.15.45:0/227668568 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef4007980 con 0x7f3f000b2e10 2023-11-28T20:51:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3ee7fff640 1 -- 172.21.15.45:0/227668568 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4007cf0 con 0x7f3f000b2e10 2023-11-28T20:51:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.382+0000 7f3f04bbc640 1 -- 172.21.15.45:0/227668568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0014b760 con 0x7f3f000b2e10 2023-11-28T20:51:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.383+0000 7f3f04bbc640 1 -- 172.21.15.45:0/227668568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0014bd20 con 0x7f3f000b2e10 2023-11-28T20:51:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.384+0000 7f3f04bbc640 1 -- 172.21.15.45:0/227668568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0009e100 con 0x7f3f000b2e10 2023-11-28T20:51:34.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.385+0000 7f3ee7fff640 1 -- 172.21.15.45:0/227668568 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 43) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f3ef40113f0 con 0x7f3f000b2e10 2023-11-28T20:51:34.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.386+0000 7f3ee7fff640 1 -- 172.21.15.45:0/227668568 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6760+0+0 (secure 0 0 0) 0x7f3ef40898c0 con 0x7f3f000b2e10 2023-11-28T20:51:34.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.391+0000 7f3ee7fff640 1 -- 172.21.15.45:0/227668568 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ef4054930 con 0x7f3f000b2e10 2023-11-28T20:51:34.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.581+0000 7f3f04bbc640 1 -- 172.21.15.45:0/227668568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3f0004e920 con 0x7f3f000b2e10 2023-11-28T20:51:34.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.582+0000 7f3ee7fff640 1 -- 172.21.15.45:0/227668568 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ef4057f50 con 0x7f3f000b2e10 2023-11-28T20:51:34.584 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:51:34.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.585+0000 7f3f04bbc640 1 -- 172.21.15.45:0/227668568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000b2e10 msgr2=0x7f3f0014c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:34.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.585+0000 7f3f04bbc640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000b2e10 0x7f3f0014c520 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c7f0 tx=0x7f3ef400ccb0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.585+0000 7f3f04bbc640 1 -- 172.21.15.45:0/227668568 shutdown_connections 2023-11-28T20:51:34.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.585+0000 7f3f04bbc640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f00153fc0 0x7f3f000cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.585+0000 7f3f04bbc640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f000b2e10 0x7f3f0014c520 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.585+0000 7f3f04bbc640 1 --2- 172.21.15.45:0/227668568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f0006b510 0x7f3f00142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.585+0000 7f3f04bbc640 1 -- 172.21.15.45:0/227668568 >> 172.21.15.45:0/227668568 conn(0x7f3f000712d0 msgr2=0x7f3f000acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:34.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.585+0000 7f3f04bbc640 1 -- 172.21.15.45:0/227668568 shutdown_connections 2023-11-28T20:51:34.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.585+0000 7f3f04bbc640 1 -- 172.21.15.45:0/227668568 wait complete. 2023-11-28T20:51:34.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:51:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.824+0000 7fee86c91640 1 Processor -- start 2023-11-28T20:51:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.825+0000 7fee86c91640 1 -- start start 2023-11-28T20:51:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.826+0000 7fee86c91640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fee80159fc0 0x7fee800b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.827+0000 7fee86c91640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee80071ac0 0x7fee800b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.827+0000 7fee86c91640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee800b27a0 con 0x7fee80072010 2023-11-28T20:51:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.827+0000 7fee86c91640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee80071450 con 0x7fee80159fc0 2023-11-28T20:51:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.827+0000 7fee86c91640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8009ceb0 con 0x7fee80071ac0 2023-11-28T20:51:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.827+0000 7fee86490640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fee80072010 0x7fee8015dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54906/0 (socket says 172.21.15.45:54906) 2023-11-28T20:51:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.827+0000 7fee86490640 1 -- 172.21.15.45:0/3419169229 learned_addr learned my addr 172.21.15.45:0/3419169229 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:34.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.827+0000 7fee85c8f640 1 --2- 172.21.15.45:0/3419169229 >> v2:172.21.15.190:3300/0 conn(0x7fee80159fc0 0x7fee800b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.827+0000 7fee8548e640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee80071ac0 0x7fee800b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772212662 0 0) 0x7fee800b27a0 con 0x7fee80072010 2023-11-28T20:51:34.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee6c003520 con 0x7fee80072010 2023-11-28T20:51:34.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4150844468 0 0) 0x7fee6c003520 con 0x7fee80072010 2023-11-28T20:51:34.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee80071ac0 msgr2=0x7fee800b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee80071ac0 0x7fee800b2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 >> v2:172.21.15.190:3300/0 conn(0x7fee80159fc0 msgr2=0x7fee800b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 --2- 172.21.15.45:0/3419169229 >> v2:172.21.15.190:3300/0 conn(0x7fee80159fc0 0x7fee800b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee800b2cb0 con 0x7fee80072010 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fee78001670 con 0x7fee80072010 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 >> v1:172.21.15.45:6789/0 conn(0x7fee80072010 legacy=0x7fee8015dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee80071ac0 0x7fee6c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c003ef0 0x7fee6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee6c004d00 0x7fee6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee80071450 con 0x7fee6c004d00 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8009ceb0 con 0x7fee6c003ef0 2023-11-28T20:51:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.828+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c00b9d0 con 0x7fee80071ac0 2023-11-28T20:51:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.829+0000 7fee86490640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee6c004d00 0x7fee6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.829+0000 7fee85c8f640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee80071ac0 0x7fee6c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.829+0000 7fee8548e640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c003ef0 0x7fee6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.829+0000 7fee86490640 1 -- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee80071ac0 msgr2=0x7fee6c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.829+0000 7fee86490640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee80071ac0 0x7fee6c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.829+0000 7fee86490640 1 -- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c003ef0 msgr2=0x7fee6c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.829+0000 7fee86490640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c003ef0 0x7fee6c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.829+0000 7fee86490640 1 -- 172.21.15.45:0/3419169229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c003520 con 0x7fee6c004d00 2023-11-28T20:51:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.829+0000 7fee8548e640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c003ef0 0x7fee6c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:34.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.830+0000 7fee86490640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee6c004d00 0x7fee6c005180 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fee7800c2c0 tx=0x7fee6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2712d9e470f394ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.830+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee78005ad0 con 0x7fee6c004d00 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.830+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee78005c30 con 0x7fee6c004d00 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.830+0000 7fee84c8d640 1 -- 172.21.15.45:0/3419169229 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7802c8b0 con 0x7fee6c004d00 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.831+0000 7fee86c91640 1 -- 172.21.15.45:0/3419169229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee6c004d00 msgr2=0x7fee6c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.831+0000 7fee86c91640 1 --2- 172.21.15.45:0/3419169229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee6c004d00 0x7fee6c005180 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fee7800c2c0 tx=0x7fee6c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.831+0000 7fee86490640 1 -- 172.21.15.45:0/3419169229 reap_dead start 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.831+0000 7fee86c91640 1 -- 172.21.15.45:0/3419169229 shutdown_connections 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.831+0000 7fee86c91640 1 -- 172.21.15.45:0/3419169229 >> 172.21.15.45:0/3419169229 conn(0x7fee800b1050 msgr2=0x7fee800cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.831+0000 7fee86c91640 1 -- 172.21.15.45:0/3419169229 shutdown_connections 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.831+0000 7fee86c91640 1 -- 172.21.15.45:0/3419169229 wait complete. 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.833+0000 7fee86c91640 1 Processor -- start 2023-11-28T20:51:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.833+0000 7fee86c91640 1 -- start start 2023-11-28T20:51:34.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.834+0000 7fee86c91640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c004d00 0x7fee801540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.834+0000 7fee86c91640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8014cba0 0x7fee8015b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.834+0000 7fee86c91640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee8014d020 0x7fee8015c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.834+0000 7fee86c91640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8015ba80 con 0x7fee8014d020 2023-11-28T20:51:34.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.834+0000 7fee86c91640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee80099a40 con 0x7fee6c004d00 2023-11-28T20:51:34.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.834+0000 7fee86c91640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee80099d20 con 0x7fee8014cba0 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.834+0000 7fee86490640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee8014d020 0x7fee8015c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee86490640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee8014d020 0x7fee8015c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58320/0 (socket says 172.21.15.45:58320) 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee86490640 1 -- 172.21.15.45:0/3020817378 learned_addr learned my addr 172.21.15.45:0/3020817378 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee85c8f640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c004d00 0x7fee801540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee8548e640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8014cba0 0x7fee8015b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee86490640 1 -- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8014cba0 msgr2=0x7fee8015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee86490640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8014cba0 0x7fee8015b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee86490640 1 -- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c004d00 msgr2=0x7fee801540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee86490640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c004d00 0x7fee801540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee86490640 1 -- 172.21.15.45:0/3020817378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8009a000 con 0x7fee8014d020 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.835+0000 7fee86490640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee8014d020 0x7fee8015c6e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fee78006fd0 tx=0x7fee78001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.836+0000 7fee767fc640 1 -- 172.21.15.45:0/3020817378 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7803c040 con 0x7fee8014d020 2023-11-28T20:51:34.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.836+0000 7fee767fc640 1 -- 172.21.15.45:0/3020817378 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee78005a40 con 0x7fee8014d020 2023-11-28T20:51:34.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.836+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8009a2e0 con 0x7fee8014d020 2023-11-28T20:51:34.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.836+0000 7fee767fc640 1 -- 172.21.15.45:0/3020817378 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee78007070 con 0x7fee8014d020 2023-11-28T20:51:34.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.836+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee80154610 con 0x7fee8014d020 2023-11-28T20:51:34.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.837+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8004e920 con 0x7fee8014d020 2023-11-28T20:51:34.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.838+0000 7fee767fc640 1 -- 172.21.15.45:0/3020817378 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 43) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fee78032070 con 0x7fee8014d020 2023-11-28T20:51:34.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.839+0000 7fee767fc640 1 -- 172.21.15.45:0/3020817378 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6760+0+0 (secure 0 0 0) 0x7fee780aa520 con 0x7fee8014d020 2023-11-28T20:51:34.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.843+0000 7fee767fc640 1 -- 172.21.15.45:0/3020817378 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee78075600 con 0x7fee8014d020 2023-11-28T20:51:34.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.886+0000 7fee767fc640 1 -- 172.21.15.45:0/3020817378 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 44) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fee780750b0 con 0x7fee8014d020 2023-11-28T20:51:34.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.887+0000 7fee767fc640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee64070b40 0x7fee64072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:34.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.890+0000 7fee85c8f640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee64070b40 0x7fee64072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:34.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:34.898+0000 7fee85c8f640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee64070b40 0x7fee64072f30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fee7c007380 tx=0x7fee7c00b040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.037+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fee8015e160 con 0x7fee8014d020 2023-11-28T20:51:35.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.038+0000 7fee767fc640 1 -- 172.21.15.45:0/3020817378 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) v1 ==== 100+0+11 (secure 0 0 0) 0x7fee7808e680 con 0x7fee8014d020 2023-11-28T20:51:35.040 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:51:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.040+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee64070b40 msgr2=0x7fee64072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.040+0000 7fee86c91640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee64070b40 0x7fee64072f30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fee7c007380 tx=0x7fee7c00b040 comp rx=0 tx=0).stop 2023-11-28T20:51:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee8014d020 msgr2=0x7fee8015c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee8014d020 0x7fee8015c6e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fee78006fd0 tx=0x7fee78001a00 comp rx=0 tx=0).stop 2023-11-28T20:51:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 shutdown_connections 2023-11-28T20:51:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee64070b40 0x7fee64072f30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee8014d020 0x7fee8015c6e0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8014cba0 0x7fee8015b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 --2- 172.21.15.45:0/3020817378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6c004d00 0x7fee801540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 >> 172.21.15.45:0/3020817378 conn(0x7fee800b1050 msgr2=0x7fee800affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 shutdown_connections 2023-11-28T20:51:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.041+0000 7fee86c91640 1 -- 172.21.15.45:0/3020817378 wait complete. 2023-11-28T20:51:35.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:35.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.276+0000 7f4cd6d53640 1 Processor -- start 2023-11-28T20:51:35.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.276+0000 7f4cd6d53640 1 -- start start 2023-11-28T20:51:35.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.278+0000 7f4cd6d53640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4cd00b35b0 0x7f4cd015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.278+0000 7f4cd6d53640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd015b850 0x7f4cd00b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.278+0000 7f4cd6d53640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd015bda0 con 0x7f4cd00b1fa0 2023-11-28T20:51:35.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.278+0000 7f4cd6d53640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd015ee40 con 0x7f4cd00b35b0 2023-11-28T20:51:35.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.278+0000 7f4cd6d53640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd015c080 con 0x7f4cd015b850 2023-11-28T20:51:35.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.278+0000 7f4cd52f0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4cd00b1fa0 0x7f4cd015f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54918/0 (socket says 172.21.15.45:54918) 2023-11-28T20:51:35.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.278+0000 7f4cd52f0640 1 -- 172.21.15.45:0/2974995188 learned_addr learned my addr 172.21.15.45:0/2974995188 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:35.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123250419 0 0) 0x7f4cd015bda0 con 0x7f4cd00b1fa0 2023-11-28T20:51:35.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc0003520 con 0x7f4cd00b1fa0 2023-11-28T20:51:35.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4cd4aef640 1 --2- 172.21.15.45:0/2974995188 >> v2:172.21.15.190:3300/0 conn(0x7f4cd00b35b0 0x7f4cd015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccffff640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd015b850 0x7f4cd00b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4227845111 0 0) 0x7f4cc0003520 con 0x7f4cd00b1fa0 2023-11-28T20:51:35.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd015b850 msgr2=0x7f4cd00b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:35.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd015b850 0x7f4cd00b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 >> v2:172.21.15.190:3300/0 conn(0x7f4cd00b35b0 msgr2=0x7f4cd015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:35.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 --2- 172.21.15.45:0/2974995188 >> v2:172.21.15.190:3300/0 conn(0x7f4cd00b35b0 0x7f4cd015ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd015c360 con 0x7f4cd00b1fa0 2023-11-28T20:51:35.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4cc80019f0 con 0x7f4cd00b1fa0 2023-11-28T20:51:35.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 >> v1:172.21.15.45:6789/0 conn(0x7f4cd00b1fa0 legacy=0x7f4cd015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc0003970 0x7f4cc0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc00047c0 0x7f4cc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd015b850 0x7f4cc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.279+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd015ee40 con 0x7f4cc0003970 2023-11-28T20:51:35.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd015c080 con 0x7f4cc00047c0 2023-11-28T20:51:35.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc000bab0 con 0x7f4cd015b850 2023-11-28T20:51:35.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4cd4aef640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc0003970 0x7f4cc0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4ccffff640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc00047c0 0x7f4cc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4cd52f0640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd015b850 0x7f4cc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4cd4aef640 1 -- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd015b850 msgr2=0x7f4cc0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:35.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4cd4aef640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd015b850 0x7f4cc0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4cd4aef640 1 -- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc00047c0 msgr2=0x7f4cc0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:35.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4cd4aef640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc00047c0 0x7f4cc0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4cd4aef640 1 -- 172.21.15.45:0/2974995188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc0003520 con 0x7f4cc0003970 2023-11-28T20:51:35.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.280+0000 7f4cd52f0640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd015b850 0x7f4cc0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:35.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4cd4aef640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc0003970 0x7f4cc0003d90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400ba00 tx=0x7f4cc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf711fbe39f74c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:35.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4032630 con 0x7f4cc0003970 2023-11-28T20:51:35.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc4032790 con 0x7f4cc0003970 2023-11-28T20:51:35.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4ccf7fe640 1 -- 172.21.15.45:0/2974995188 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4032b20 con 0x7f4cc0003970 2023-11-28T20:51:35.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4cd6d53640 1 -- 172.21.15.45:0/2974995188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc0003970 msgr2=0x7f4cc0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4cd6d53640 1 --2- 172.21.15.45:0/2974995188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cc0003970 0x7f4cc0003d90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400ba00 tx=0x7f4cc00057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4cd52f0640 1 -- 172.21.15.45:0/2974995188 reap_dead start 2023-11-28T20:51:35.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4cd6d53640 1 -- 172.21.15.45:0/2974995188 shutdown_connections 2023-11-28T20:51:35.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4cd6d53640 1 -- 172.21.15.45:0/2974995188 >> 172.21.15.45:0/2974995188 conn(0x7f4cd006a500 msgr2=0x7f4cd006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:35.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.282+0000 7f4cd6d53640 1 -- 172.21.15.45:0/2974995188 shutdown_connections 2023-11-28T20:51:35.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.283+0000 7f4cd6d53640 1 -- 172.21.15.45:0/2974995188 wait complete. 2023-11-28T20:51:35.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.284+0000 7f4cd6d53640 1 Processor -- start 2023-11-28T20:51:35.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.284+0000 7f4cd6d53640 1 -- start start 2023-11-28T20:51:35.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.285+0000 7f4cd6d53640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc0003970 0x7f4cd0138a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.285+0000 7f4cd6d53640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd0155640 0x7f4cd00ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.285+0000 7f4cd6d53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cd015d0d0 0x7f4cd015d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.285+0000 7f4cd6d53640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd015dac0 con 0x7f4cd015d0d0 2023-11-28T20:51:35.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.285+0000 7f4cd6d53640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd01f6420 con 0x7f4cc0003970 2023-11-28T20:51:35.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.285+0000 7f4cd6d53640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd01f6f60 con 0x7f4cd0155640 2023-11-28T20:51:35.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd52f0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cd015d0d0 0x7f4cd015d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd52f0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cd015d0d0 0x7f4cd015d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58334/0 (socket says 172.21.15.45:58334) 2023-11-28T20:51:35.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd52f0640 1 -- 172.21.15.45:0/57886058 learned_addr learned my addr 172.21.15.45:0/57886058 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4ccffff640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd0155640 0x7f4cd00ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd4aef640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc0003970 0x7f4cd0138a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd52f0640 1 -- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd0155640 msgr2=0x7f4cd00ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd52f0640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd0155640 0x7f4cd00ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd52f0640 1 -- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc0003970 msgr2=0x7f4cd0138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd52f0640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc0003970 0x7f4cd0138a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd52f0640 1 -- 172.21.15.45:0/57886058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd00ae3c0 con 0x7f4cd015d0d0 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.286+0000 7f4cd52f0640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cd015d0d0 0x7f4cd015d580 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00b2920 tx=0x7f4cc800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.287+0000 7f4ccd7fa640 1 -- 172.21.15.45:0/57886058 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8004d60 con 0x7f4cd015d0d0 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.287+0000 7f4ccd7fa640 1 -- 172.21.15.45:0/57886058 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc800bb80 con 0x7f4cd015d0d0 2023-11-28T20:51:35.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.287+0000 7f4ccd7fa640 1 -- 172.21.15.45:0/57886058 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8007070 con 0x7f4cd015d0d0 2023-11-28T20:51:35.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.287+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd01f6630 con 0x7f4cd015d0d0 2023-11-28T20:51:35.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.287+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd01f6bf0 con 0x7f4cd015d0d0 2023-11-28T20:51:35.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.288+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd01f70d0 con 0x7f4cd015d0d0 2023-11-28T20:51:35.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.289+0000 7f4ccd7fa640 1 -- 172.21.15.45:0/57886058 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 44) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f4cc80071d0 con 0x7f4cd015d0d0 2023-11-28T20:51:35.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.290+0000 7f4ccd7fa640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4cb406acc0 0x7f4cb406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.290+0000 7f4ccd7fa640 1 -- 172.21.15.45:0/57886058 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6760+0+0 (secure 0 0 0) 0x7f4cc8089a80 con 0x7f4cd015d0d0 2023-11-28T20:51:35.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.291+0000 7f4cd4aef640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4cb406acc0 0x7f4cb406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.292+0000 7f4cd4aef640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4cb406acc0 0x7f4cb406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0004740 tx=0x7f4cc403d040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:35.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.294+0000 7f4ccd7fa640 1 -- 172.21.15.45:0/57886058 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4cc8054980 con 0x7f4cd015d0d0 2023-11-28T20:51:35.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.526+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd00733d0 con 0x7f4cd015d0d0 2023-11-28T20:51:35.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.528+0000 7f4ccd7fa640 1 -- 172.21.15.45:0/57886058 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f4cc8057fa0 con 0x7f4cd015d0d0 2023-11-28T20:51:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:35.530 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3492018158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3492018158},{"type":"v1","addr":"172.21.15.45:6833","nonce":3492018158}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6121,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2276329291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2276329291},{"type":"v1","addr":"172.21.15.190:6835","nonce":2276329291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6130,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1261857840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1261857840},{"type":"v1","addr":"172.21.15.190:6837","nonce":1261857840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:12.953862+0000","modified":"2023-11-28T20:51:16.015607+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7179},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7179":{"gid":7179,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/862206408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":862206408},{"type":"v1","addr":"172.21.15.45:6835","nonce":862206408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-28T20:51:35.530 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 44 2023-11-28T20:51:35.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4cb406acc0 msgr2=0x7f4cb406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4cb406acc0 0x7f4cb406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0004740 tx=0x7f4cc403d040 comp rx=0 tx=0).stop 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cd015d0d0 msgr2=0x7f4cd015d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cd015d0d0 0x7f4cd015d580 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00b2920 tx=0x7f4cc800b4a0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 shutdown_connections 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4cb406acc0 0x7f4cb406d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cd015d0d0 0x7f4cd015d580 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cd0155640 0x7f4cd00ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 --2- 172.21.15.45:0/57886058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc0003970 0x7f4cd0138a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.530+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 >> 172.21.15.45:0/57886058 conn(0x7f4cd006a500 msgr2=0x7f4cd006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.531+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 shutdown_connections 2023-11-28T20:51:35.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.531+0000 7f4cd6d53640 1 -- 172.21.15.45:0/57886058 wait complete. 2023-11-28T20:51:35.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:35.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.769+0000 7fe034c91640 1 Processor -- start 2023-11-28T20:51:35.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.769+0000 7fe034c91640 1 -- start start 2023-11-28T20:51:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.770+0000 7fe034c91640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe0300ce800 0x7fe030165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.770+0000 7fe034c91640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03015c3c0 con 0x7fe030155780 2023-11-28T20:51:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.770+0000 7fe034c91640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe030155b60 con 0x7fe0300ae540 2023-11-28T20:51:35.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.770+0000 7fe034c91640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe03015ba90 con 0x7fe0300ce800 2023-11-28T20:51:35.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.770+0000 7fe02f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe030155780 0x7fe0300abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54932/0 (socket says 172.21.15.45:54932) 2023-11-28T20:51:35.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.770+0000 7fe02f7fe640 1 -- 172.21.15.45:0/2649959440 learned_addr learned my addr 172.21.15.45:0/2649959440 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:35.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02ffff640 1 --2- 172.21.15.45:0/2649959440 >> v2:172.21.15.190:3300/0 conn(0x7fe0300ce800 0x7fe030165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335092991 0 0) 0x7fe03015c3c0 con 0x7fe030155780 2023-11-28T20:51:35.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c003520 con 0x7fe030155780 2023-11-28T20:51:35.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2887925934 0 0) 0x7fe01c003520 con 0x7fe030155780 2023-11-28T20:51:35.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 >> v2:172.21.15.190:3300/0 conn(0x7fe0300ce800 msgr2=0x7fe030165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:35.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 --2- 172.21.15.45:0/2649959440 >> v2:172.21.15.190:3300/0 conn(0x7fe0300ce800 0x7fe030165e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 >> v1:172.21.15.190:6789/0 conn(0x7fe0300ae540 legacy=0x7fe0300ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03015bd70 con 0x7fe030155780 2023-11-28T20:51:35.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fe020001670 con 0x7fe030155780 2023-11-28T20:51:35.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 >> v1:172.21.15.45:6789/0 conn(0x7fe030155780 legacy=0x7fe0300abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe01c0039c0 0x7fe01c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0047c0 0x7fe01c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe01c005180 0x7fe01c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03015ba90 con 0x7fe01c0047c0 2023-11-28T20:51:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c00c080 con 0x7fe01c0039c0 2023-11-28T20:51:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.771+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c00c240 con 0x7fe01c005180 2023-11-28T20:51:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.772+0000 7fe027fff640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0047c0 0x7fe01c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.772+0000 7fe02f7fe640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe01c0039c0 0x7fe01c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.772+0000 7fe02ffff640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe01c005180 0x7fe01c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.772+0000 7fe027fff640 1 -- 172.21.15.45:0/2649959440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe01c005180 msgr2=0x7fe01c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:35.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.772+0000 7fe027fff640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe01c005180 0x7fe01c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.772+0000 7fe027fff640 1 -- 172.21.15.45:0/2649959440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe01c0039c0 msgr2=0x7fe01c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:35.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.772+0000 7fe027fff640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe01c0039c0 0x7fe01c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.772+0000 7fe027fff640 1 -- 172.21.15.45:0/2649959440 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c003520 con 0x7fe01c0047c0 2023-11-28T20:51:35.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.772+0000 7fe02ffff640 1 -- 172.21.15.45:0/2649959440 reap_dead start 2023-11-28T20:51:35.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.774+0000 7fe027fff640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0047c0 0x7fe01c004c40 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe018001750 tx=0x7fe01c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139cf280fb76720f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.774+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0180052d0 con 0x7fe01c0047c0 2023-11-28T20:51:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.774+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01802e050 con 0x7fe01c0047c0 2023-11-28T20:51:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.774+0000 7fe02effd640 1 -- 172.21.15.45:0/2649959440 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018003830 con 0x7fe01c0047c0 2023-11-28T20:51:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.774+0000 7fe034c91640 1 -- 172.21.15.45:0/2649959440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0047c0 msgr2=0x7fe01c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.774+0000 7fe034c91640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0047c0 0x7fe01c004c40 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe018001750 tx=0x7fe01c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.775+0000 7fe034c91640 1 -- 172.21.15.45:0/2649959440 shutdown_connections 2023-11-28T20:51:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.775+0000 7fe034c91640 1 --2- 172.21.15.45:0/2649959440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0047c0 0x7fe01c004c40 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.775+0000 7fe034c91640 1 -- 172.21.15.45:0/2649959440 >> 172.21.15.45:0/2649959440 conn(0x7fe03006ab30 msgr2=0x7fe0300b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.775+0000 7fe034c91640 1 -- 172.21.15.45:0/2649959440 shutdown_connections 2023-11-28T20:51:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.775+0000 7fe034c91640 1 -- 172.21.15.45:0/2649959440 wait complete. 2023-11-28T20:51:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.777+0000 7fe034c91640 1 Processor -- start 2023-11-28T20:51:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.777+0000 7fe034c91640 1 -- start start 2023-11-28T20:51:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe034c91640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe01c0047c0 0x7fe030143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe034c91640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0301512a0 0x7fe030071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe034c91640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe03009aa50 0x7fe03009aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe034c91640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03009a630 con 0x7fe0301512a0 2023-11-28T20:51:35.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe034c91640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe030143fd0 con 0x7fe01c0047c0 2023-11-28T20:51:35.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe034c91640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe030099a40 con 0x7fe03009aa50 2023-11-28T20:51:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe027fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0301512a0 0x7fe030071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe027fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0301512a0 0x7fe030071550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58352/0 (socket says 172.21.15.45:58352) 2023-11-28T20:51:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe027fff640 1 -- 172.21.15.45:0/1903734190 learned_addr learned my addr 172.21.15.45:0/1903734190 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe02f7fe640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe01c0047c0 0x7fe030143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.778+0000 7fe02ffff640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe03009aa50 0x7fe03009aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe027fff640 1 -- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe03009aa50 msgr2=0x7fe03009aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe027fff640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe03009aa50 0x7fe03009aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe027fff640 1 -- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe01c0047c0 msgr2=0x7fe030143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe027fff640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe01c0047c0 0x7fe030143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe027fff640 1 -- 172.21.15.45:0/1903734190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe030071a90 con 0x7fe0301512a0 2023-11-28T20:51:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe027fff640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0301512a0 0x7fe030071550 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fe018005040 tx=0x7fe018001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe02cff9640 1 -- 172.21.15.45:0/1903734190 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018007a30 con 0x7fe0301512a0 2023-11-28T20:51:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe02cff9640 1 -- 172.21.15.45:0/1903734190 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0180069e0 con 0x7fe0301512a0 2023-11-28T20:51:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe030099c90 con 0x7fe0301512a0 2023-11-28T20:51:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.779+0000 7fe02cff9640 1 -- 172.21.15.45:0/1903734190 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018006db0 con 0x7fe0301512a0 2023-11-28T20:51:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.780+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03009a250 con 0x7fe0301512a0 2023-11-28T20:51:35.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.781+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03015b720 con 0x7fe0301512a0 2023-11-28T20:51:35.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.782+0000 7fe02cff9640 1 -- 172.21.15.45:0/1903734190 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 44) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fe0180035f0 con 0x7fe0301512a0 2023-11-28T20:51:35.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.782+0000 7fe02cff9640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fe01006acc0 0x7fe01006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:35.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.783+0000 7fe02cff9640 1 -- 172.21.15.45:0/1903734190 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6760+0+0 (secure 0 0 0) 0x7fe0180ab820 con 0x7fe0301512a0 2023-11-28T20:51:35.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.783+0000 7fe02f7fe640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fe01006acc0 0x7fe01006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:35.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.784+0000 7fe02f7fe640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fe01006acc0 0x7fe01006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe020001050 tx=0x7fe020009040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:35.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.787+0000 7fe02cff9640 1 -- 172.21.15.45:0/1903734190 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe03015b720 con 0x7fe0301512a0 2023-11-28T20:51:35.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.902+0000 7fe02cff9640 1 -- 172.21.15.45:0/1903734190 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fe018076a60 con 0x7fe0301512a0 2023-11-28T20:51:35.939 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:35.937+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:35.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:35.997+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe03015b720 con 0x7fe0301512a0 2023-11-28T20:51:36.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.000+0000 7fe02cff9640 1 -- 172.21.15.45:0/1903734190 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+17854 (secure 0 0 0) 0x7fe03015b720 con 0x7fe0301512a0 2023-11-28T20:51:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:36.003 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":104,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:33.864472+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:10.974716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:11.975222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1724154865":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/412684916":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/514984079":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/2474029917":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/3103043064":"2023-11-29T20:51:27.664495+0000","172.21.15.190:6833/1373961985":"2023-11-29T20:51:33.864326+0000","172.21.15.190:6832/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.190:6832/1373961985":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/3436414960":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3227357091":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3114762982":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/1429551158":"2023-11-29T20:51:33.864326+0000","172.21.15.190:6833/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/4025400682":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2627631957":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2067877259":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/1852415784":"2023-11-28T21:51:13.642235+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:36.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fe01006acc0 msgr2=0x7fe01006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fe01006acc0 0x7fe01006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe020001050 tx=0x7fe020009040 comp rx=0 tx=0).stop 2023-11-28T20:51:36.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0301512a0 msgr2=0x7fe030071550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0301512a0 0x7fe030071550 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fe018005040 tx=0x7fe018001120 comp rx=0 tx=0).stop 2023-11-28T20:51:36.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 shutdown_connections 2023-11-28T20:51:36.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fe01006acc0 0x7fe01006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe03009aa50 0x7fe03009aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0301512a0 0x7fe030071550 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 --2- 172.21.15.45:0/1903734190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe01c0047c0 0x7fe030143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 >> 172.21.15.45:0/1903734190 conn(0x7fe03006ab30 msgr2=0x7fe03006d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:36.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 shutdown_connections 2023-11-28T20:51:36.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.003+0000 7fe034c91640 1 -- 172.21.15.45:0/1903734190 wait complete. 2023-11-28T20:51:36.024 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T20:51:36.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:36.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.233+0000 7f23898e6640 1 Processor -- start 2023-11-28T20:51:36.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.233+0000 7f23898e6640 1 -- start start 2023-11-28T20:51:36.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.235+0000 7f23898e6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23840ae540 0x7f23840ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.235+0000 7f23898e6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f23840ce630 0x7f23840b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.235+0000 7f23898e6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238415c3c0 con 0x7f2384155780 2023-11-28T20:51:36.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.235+0000 7f23898e6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23840ceac0 con 0x7f23840ae540 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.235+0000 7f23898e6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2384155b60 con 0x7f23840ce630 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.235+0000 7f2383fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23840ae540 0x7f23840ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.235+0000 7f23888e4640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2384155780 0x7f23840abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54942/0 (socket says 172.21.15.45:54942) 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.236+0000 7f2383fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23840ae540 0x7f23840ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58360/0 (socket says 172.21.15.45:58360) 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.236+0000 7f23888e4640 1 -- 172.21.15.45:0/1599753906 learned_addr learned my addr 172.21.15.45:0/1599753906 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.236+0000 7f23890e5640 1 --2- 172.21.15.45:0/1599753906 >> v2:172.21.15.190:3300/0 conn(0x7f23840ce630 0x7f23840b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.236+0000 7f2383fff640 1 -- 172.21.15.45:0/1599753906 >> v2:172.21.15.190:3300/0 conn(0x7f23840ce630 msgr2=0x7f23840b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.236+0000 7f2383fff640 1 --2- 172.21.15.45:0/1599753906 >> v2:172.21.15.190:3300/0 conn(0x7f23840ce630 0x7f23840b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.236+0000 7f2383fff640 1 -- 172.21.15.45:0/1599753906 >> v1:172.21.15.45:6789/0 conn(0x7f2384155780 legacy=0x7f23840abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.236+0000 7f2383fff640 1 -- 172.21.15.45:0/1599753906 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238415ba90 con 0x7f23840ae540 2023-11-28T20:51:36.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.236+0000 7f23890e5640 1 --2- 172.21.15.45:0/1599753906 >> v2:172.21.15.190:3300/0 conn(0x7f23840ce630 0x7f23840b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:36.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.237+0000 7f2383fff640 1 --2- 172.21.15.45:0/1599753906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ae540 0x7f23840ce0f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f23840b1690 tx=0x7f237002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9178fae6f55f1c89 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.238+0000 7f23837fe640 1 -- 172.21.15.45:0/1599753906 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370031830 con 0x7f23840ae540 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.238+0000 7f23837fe640 1 -- 172.21.15.45:0/1599753906 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2370031990 con 0x7f23840ae540 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.238+0000 7f23837fe640 1 -- 172.21.15.45:0/1599753906 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370031d20 con 0x7f23840ae540 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.238+0000 7f23898e6640 1 -- 172.21.15.45:0/1599753906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ae540 msgr2=0x7f23840ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.238+0000 7f23898e6640 1 --2- 172.21.15.45:0/1599753906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ae540 0x7f23840ce0f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f23840b1690 tx=0x7f237002b8d0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.238+0000 7f23898e6640 1 -- 172.21.15.45:0/1599753906 shutdown_connections 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.238+0000 7f23898e6640 1 --2- 172.21.15.45:0/1599753906 >> v2:172.21.15.190:3300/0 conn(0x7f23840ce630 0x7f23840b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.238+0000 7f23898e6640 1 --2- 172.21.15.45:0/1599753906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ae540 0x7f23840ce0f0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.238+0000 7f23898e6640 1 -- 172.21.15.45:0/1599753906 >> 172.21.15.45:0/1599753906 conn(0x7f238406ab30 msgr2=0x7f23840b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.239+0000 7f23898e6640 1 -- 172.21.15.45:0/1599753906 shutdown_connections 2023-11-28T20:51:36.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.239+0000 7f23898e6640 1 -- 172.21.15.45:0/1599753906 wait complete. 2023-11-28T20:51:36.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.240+0000 7f23898e6640 1 Processor -- start 2023-11-28T20:51:36.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.240+0000 7f23898e6640 1 -- start start 2023-11-28T20:51:36.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.241+0000 7f23898e6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23840ae540 0x7f23841457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.241+0000 7f23898e6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ce630 0x7f238409bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.241+0000 7f23898e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384155780 0x7f2384151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.241+0000 7f23898e6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2384150df0 con 0x7f23840ce630 2023-11-28T20:51:36.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.241+0000 7f23898e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2384151280 con 0x7f2384155780 2023-11-28T20:51:36.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.241+0000 7f23898e6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23841501e0 con 0x7f23840ae540 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f2383fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ce630 0x7f238409bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f2383fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ce630 0x7f238409bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58370/0 (socket says 172.21.15.45:58370) 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f2383fff640 1 -- 172.21.15.45:0/42605387 learned_addr learned my addr 172.21.15.45:0/42605387 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f23890e5640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384155780 0x7f2384151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f23888e4640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23840ae540 0x7f23841457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f2383fff640 1 -- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23840ae540 msgr2=0x7f23841457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f2383fff640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23840ae540 0x7f23841457d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f2383fff640 1 -- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384155780 msgr2=0x7f2384151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f2383fff640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384155780 0x7f2384151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f2383fff640 1 -- 172.21.15.45:0/42605387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2370009400 con 0x7f23840ce630 2023-11-28T20:51:36.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f23888e4640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23840ae540 0x7f23841457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:36.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.242+0000 7f23890e5640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384155780 0x7f2384151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:36.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.243+0000 7f2383fff640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ce630 0x7f238409bdc0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f2370000c00 tx=0x7f23700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:36.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.243+0000 7f23817fa640 1 -- 172.21.15.45:0/42605387 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237003ba70 con 0x7f23840ce630 2023-11-28T20:51:36.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.243+0000 7f23817fa640 1 -- 172.21.15.45:0/42605387 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237003bbd0 con 0x7f23840ce630 2023-11-28T20:51:36.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.243+0000 7f23817fa640 1 -- 172.21.15.45:0/42605387 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700453f0 con 0x7f23840ce630 2023-11-28T20:51:36.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.243+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23841504c0 con 0x7f23840ce630 2023-11-28T20:51:36.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.243+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2384150a80 con 0x7f23840ce630 2023-11-28T20:51:36.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.244+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238404e920 con 0x7f23840ce630 2023-11-28T20:51:36.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.245+0000 7f23817fa640 1 -- 172.21.15.45:0/42605387 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f2370045550 con 0x7f23840ce630 2023-11-28T20:51:36.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.246+0000 7f23817fa640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f236806aed0 0x7f236806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.246+0000 7f23817fa640 1 -- 172.21.15.45:0/42605387 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6760+0+0 (secure 0 0 0) 0x7f23700abfe0 con 0x7f23840ce630 2023-11-28T20:51:36.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.247+0000 7f23888e4640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f236806aed0 0x7f236806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.249+0000 7f23888e4640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f236806aed0 0x7f236806d390 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2378000f60 tx=0x7f2378009040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:36.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.251+0000 7f23817fa640 1 -- 172.21.15.45:0/42605387 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23700dee90 con 0x7f23840ce630 2023-11-28T20:51:36.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.484+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f238406ccc0 con 0x7f23840ce630 2023-11-28T20:51:36.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.487+0000 7f23817fa640 1 -- 172.21.15.45:0/42605387 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f2370035020 con 0x7f23840ce630 2023-11-28T20:51:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:36.489 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3492018158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3492018158},{"type":"v1","addr":"172.21.15.45:6833","nonce":3492018158}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6121,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2276329291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2276329291},{"type":"v1","addr":"172.21.15.190:6835","nonce":2276329291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6130,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1261857840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1261857840},{"type":"v1","addr":"172.21.15.190:6837","nonce":1261857840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:12.953862+0000","modified":"2023-11-28T20:51:16.015607+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7179},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7179":{"gid":7179,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/862206408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":862206408},{"type":"v1","addr":"172.21.15.45:6835","nonce":862206408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-28T20:51:36.489 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 44 2023-11-28T20:51:36.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f236806aed0 msgr2=0x7f236806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f236806aed0 0x7f236806d390 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2378000f60 tx=0x7f2378009040 comp rx=0 tx=0).stop 2023-11-28T20:51:36.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ce630 msgr2=0x7f238409bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ce630 0x7f238409bdc0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f2370000c00 tx=0x7f23700046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 shutdown_connections 2023-11-28T20:51:36.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f236806aed0 0x7f236806d390 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384155780 0x7f2384151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23840ce630 0x7f238409bdc0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 --2- 172.21.15.45:0/42605387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23840ae540 0x7f23841457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 >> 172.21.15.45:0/42605387 conn(0x7f238406ab30 msgr2=0x7f23840ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:36.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.489+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 shutdown_connections 2023-11-28T20:51:36.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.490+0000 7f23898e6640 1 -- 172.21.15.45:0/42605387 wait complete. 2023-11-28T20:51:36.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:36.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.723+0000 7f56c1d75640 1 Processor -- start 2023-11-28T20:51:36.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.723+0000 7f56c1d75640 1 -- start start 2023-11-28T20:51:36.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1d75640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f56bc155780 0x7f56bc0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1d75640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56bc0ce800 0x7f56bc0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1d75640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56bc15c3c0 con 0x7f56bc0ae540 2023-11-28T20:51:36.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1d75640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56bc155b60 con 0x7f56bc0ce800 2023-11-28T20:51:36.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1d75640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56bc15ba90 con 0x7f56bc155780 2023-11-28T20:51:36.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1574640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56bc0ce800 0x7f56bc0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1574640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56bc0ce800 0x7f56bc0b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58384/0 (socket says 172.21.15.45:58384) 2023-11-28T20:51:36.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1574640 1 -- 172.21.15.45:0/3529258582 learned_addr learned my addr 172.21.15.45:0/3529258582 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:36.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c0d73640 1 --2- 172.21.15.45:0/3529258582 >> v2:172.21.15.190:3300/0 conn(0x7f56bc155780 0x7f56bc0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1574640 1 -- 172.21.15.45:0/3529258582 >> v2:172.21.15.190:3300/0 conn(0x7f56bc155780 msgr2=0x7f56bc0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1574640 1 --2- 172.21.15.45:0/3529258582 >> v2:172.21.15.190:3300/0 conn(0x7f56bc155780 0x7f56bc0abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1574640 1 -- 172.21.15.45:0/3529258582 >> v1:172.21.15.190:6789/0 conn(0x7f56bc0ae540 legacy=0x7f56bc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.726+0000 7f56c1574640 1 -- 172.21.15.45:0/3529258582 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc15bd70 con 0x7f56bc0ce800 2023-11-28T20:51:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.727+0000 7f56c1574640 1 --2- 172.21.15.45:0/3529258582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0ce800 0x7f56bc0b20a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f56bc06e6d0 tx=0x7f56b402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2edb67c3d1beec0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.728+0000 7f56bb7fe640 1 -- 172.21.15.45:0/3529258582 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b402cb20 con 0x7f56bc0ce800 2023-11-28T20:51:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.728+0000 7f56bb7fe640 1 -- 172.21.15.45:0/3529258582 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b402cc80 con 0x7f56bc0ce800 2023-11-28T20:51:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.728+0000 7f56bb7fe640 1 -- 172.21.15.45:0/3529258582 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b40343f0 con 0x7f56bc0ce800 2023-11-28T20:51:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.728+0000 7f56c1d75640 1 -- 172.21.15.45:0/3529258582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0ce800 msgr2=0x7f56bc0b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.728+0000 7f56c1d75640 1 --2- 172.21.15.45:0/3529258582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0ce800 0x7f56bc0b20a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f56bc06e6d0 tx=0x7f56b402bf80 comp rx=0 tx=0).stop 2023-11-28T20:51:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.729+0000 7f56c1d75640 1 -- 172.21.15.45:0/3529258582 shutdown_connections 2023-11-28T20:51:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.729+0000 7f56c1d75640 1 --2- 172.21.15.45:0/3529258582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc0ce800 0x7f56bc0b20a0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.729+0000 7f56c1d75640 1 --2- 172.21.15.45:0/3529258582 >> v2:172.21.15.190:3300/0 conn(0x7f56bc155780 0x7f56bc0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.729+0000 7f56c1d75640 1 -- 172.21.15.45:0/3529258582 >> 172.21.15.45:0/3529258582 conn(0x7f56bc06ab30 msgr2=0x7f56bc0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:36.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.729+0000 7f56c1d75640 1 -- 172.21.15.45:0/3529258582 shutdown_connections 2023-11-28T20:51:36.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.729+0000 7f56c1d75640 1 -- 172.21.15.45:0/3529258582 wait complete. 2023-11-28T20:51:36.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.730+0000 7f56c1d75640 1 Processor -- start 2023-11-28T20:51:36.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.731+0000 7f56c1d75640 1 -- start start 2023-11-28T20:51:36.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.731+0000 7f56c1d75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0ae540 0x7f56bc09b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.731+0000 7f56c1d75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc0ce800 0x7f56bc099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1d75640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc155780 0x7f56bc09c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1d75640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc152180 con 0x7f56bc155780 2023-11-28T20:51:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1d75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc099f80 con 0x7f56bc0ce800 2023-11-28T20:51:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1d75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc09adc0 con 0x7f56bc0ae540 2023-11-28T20:51:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1574640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc155780 0x7f56bc09c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1574640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc155780 0x7f56bc09c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58400/0 (socket says 172.21.15.45:58400) 2023-11-28T20:51:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1574640 1 -- 172.21.15.45:0/2017077296 learned_addr learned my addr 172.21.15.45:0/2017077296 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56bbfff640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc0ce800 0x7f56bc099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1574640 1 -- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0ae540 msgr2=0x7f56bc09b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c0d73640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0ae540 0x7f56bc09b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1574640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0ae540 0x7f56bc09b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1574640 1 -- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc0ce800 msgr2=0x7f56bc099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1574640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc0ce800 0x7f56bc099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1574640 1 -- 172.21.15.45:0/2017077296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4009b30 con 0x7f56bc155780 2023-11-28T20:51:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56bbfff640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc0ce800 0x7f56bc099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1574640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc155780 0x7f56bc09c090 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009f40 tx=0x7f56b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56b97fa640 1 -- 172.21.15.45:0/2017077296 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b4007e70 con 0x7f56bc155780 2023-11-28T20:51:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56b97fa640 1 -- 172.21.15.45:0/2017077296 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b4009510 con 0x7f56bc155780 2023-11-28T20:51:36.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56bc09a1b0 con 0x7f56bc155780 2023-11-28T20:51:36.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56b97fa640 1 -- 172.21.15.45:0/2017077296 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b403e690 con 0x7f56bc155780 2023-11-28T20:51:36.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.733+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56bc09a770 con 0x7f56bc155780 2023-11-28T20:51:36.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.734+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56bc04e920 con 0x7f56bc155780 2023-11-28T20:51:36.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.735+0000 7f56b97fa640 1 -- 172.21.15.45:0/2017077296 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f56b403e7f0 con 0x7f56bc155780 2023-11-28T20:51:36.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.736+0000 7f56b97fa640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f56a006aed0 0x7f56a006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:36.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.736+0000 7f56b97fa640 1 -- 172.21.15.45:0/2017077296 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6760+0+0 (secure 0 0 0) 0x7f56b40ac9d0 con 0x7f56bc155780 2023-11-28T20:51:36.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.737+0000 7f56c0d73640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f56a006aed0 0x7f56a006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:36.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.738+0000 7f56c0d73640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f56a006aed0 0x7f56a006d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006fd0 tx=0x7f56b0008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:36.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.742+0000 7f56b97fa640 1 -- 172.21.15.45:0/2017077296 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56b4077570 con 0x7f56bc155780 2023-11-28T20:51:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.980+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56bc06d9a0 con 0x7f56bc155780 2023-11-28T20:51:36.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.981+0000 7f56b97fa640 1 -- 172.21.15.45:0/2017077296 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f56b407ab90 con 0x7f56bc155780 2023-11-28T20:51:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:36.982 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3492018158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3492018158},{"type":"v1","addr":"172.21.15.45:6833","nonce":3492018158}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6121,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2276329291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2276329291},{"type":"v1","addr":"172.21.15.190:6835","nonce":2276329291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6130,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1261857840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1261857840},{"type":"v1","addr":"172.21.15.190:6837","nonce":1261857840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:12.953862+0000","modified":"2023-11-28T20:51:16.015607+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7179},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7179":{"gid":7179,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/862206408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":862206408},{"type":"v1","addr":"172.21.15.45:6835","nonce":862206408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-28T20:51:36.983 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 44 2023-11-28T20:51:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.983+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f56a006aed0 msgr2=0x7f56a006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.983+0000 7f56c1d75640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f56a006aed0 0x7f56a006d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006fd0 tx=0x7f56b0008040 comp rx=0 tx=0).stop 2023-11-28T20:51:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.983+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc155780 msgr2=0x7f56bc09c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.983+0000 7f56c1d75640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc155780 0x7f56bc09c090 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009f40 tx=0x7f56b4004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.984+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 shutdown_connections 2023-11-28T20:51:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.984+0000 7f56c1d75640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f56a006aed0 0x7f56a006d390 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.984+0000 7f56c1d75640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56bc155780 0x7f56bc09c090 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.984+0000 7f56c1d75640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56bc0ce800 0x7f56bc099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.984+0000 7f56c1d75640 1 --2- 172.21.15.45:0/2017077296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56bc0ae540 0x7f56bc09b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.984+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 >> 172.21.15.45:0/2017077296 conn(0x7f56bc06ab30 msgr2=0x7f56bc06ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.984+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 shutdown_connections 2023-11-28T20:51:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:36.984+0000 7f56c1d75640 1 -- 172.21.15.45:0/2017077296 wait complete. 2023-11-28T20:51:36.989 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:36.987+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:37.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:37.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.222+0000 7fca4fe1c640 1 Processor -- start 2023-11-28T20:51:37.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.223+0000 7fca4fe1c640 1 -- start start 2023-11-28T20:51:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.224+0000 7fca4fe1c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca48146b60 0x7fca4806b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.224+0000 7fca4fe1c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fca480b2f90 0x7fca480b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.225+0000 7fca4fe1c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4806b340 con 0x7fca48160fd0 2023-11-28T20:51:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.225+0000 7fca4fe1c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca480b3950 con 0x7fca480b2f90 2023-11-28T20:51:37.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.225+0000 7fca4fe1c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca48146f40 con 0x7fca48146b60 2023-11-28T20:51:37.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.225+0000 7fca4e3b9640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fca48160fd0 0x7fca48161460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54944/0 (socket says 172.21.15.45:54944) 2023-11-28T20:51:37.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.225+0000 7fca4e3b9640 1 -- 172.21.15.45:0/1128795637 learned_addr learned my addr 172.21.15.45:0/1128795637 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:37.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.225+0000 7fca4dbb8640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca48146b60 0x7fca4806b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.225+0000 7fca4d3b7640 1 --2- 172.21.15.45:0/1128795637 >> v2:172.21.15.190:3300/0 conn(0x7fca480b2f90 0x7fca480b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.225+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766236019 0 0) 0x7fca4806b340 con 0x7fca48160fd0 2023-11-28T20:51:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.225+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca34003520 con 0x7fca48160fd0 2023-11-28T20:51:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 900808441 0 0) 0x7fca34003520 con 0x7fca48160fd0 2023-11-28T20:51:37.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca48146b60 msgr2=0x7fca4806b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:37.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca48146b60 0x7fca4806b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 >> v2:172.21.15.190:3300/0 conn(0x7fca480b2f90 msgr2=0x7fca480b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:37.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 --2- 172.21.15.45:0/1128795637 >> v2:172.21.15.190:3300/0 conn(0x7fca480b2f90 0x7fca480b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca48161d40 con 0x7fca48160fd0 2023-11-28T20:51:37.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fca40001670 con 0x7fca48160fd0 2023-11-28T20:51:37.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 >> v1:172.21.15.45:6789/0 conn(0x7fca48160fd0 legacy=0x7fca48161460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca34003970 0x7fca34003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca48146b60 0x7fca340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34004d00 0x7fca34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca480b3950 con 0x7fca34004d00 2023-11-28T20:51:37.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca48146f40 con 0x7fca34003970 2023-11-28T20:51:37.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3400b980 con 0x7fca48146b60 2023-11-28T20:51:37.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.226+0000 7fca4e3b9640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34004d00 0x7fca34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.227+0000 7fca4d3b7640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca48146b60 0x7fca340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.227+0000 7fca4dbb8640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca34003970 0x7fca34003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.227+0000 7fca4e3b9640 1 -- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca48146b60 msgr2=0x7fca340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:37.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.227+0000 7fca4e3b9640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca48146b60 0x7fca340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.227+0000 7fca4e3b9640 1 -- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca34003970 msgr2=0x7fca34003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:37.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.227+0000 7fca4e3b9640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca34003970 0x7fca34003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.227+0000 7fca4e3b9640 1 -- 172.21.15.45:0/1128795637 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca34003520 con 0x7fca34004d00 2023-11-28T20:51:37.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.228+0000 7fca4e3b9640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34004d00 0x7fca34005180 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c210 tx=0x7fca340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461ec7fe053e17f2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:37.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.228+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4002cae0 con 0x7fca34004d00 2023-11-28T20:51:37.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.228+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4002cc40 con 0x7fca34004d00 2023-11-28T20:51:37.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.228+0000 7fca4cbb6640 1 -- 172.21.15.45:0/1128795637 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca400058d0 con 0x7fca34004d00 2023-11-28T20:51:37.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.229+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1128795637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34004d00 msgr2=0x7fca34005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.229+0000 7fca4fe1c640 1 --2- 172.21.15.45:0/1128795637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca34004d00 0x7fca34005180 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c210 tx=0x7fca340057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.229+0000 7fca4e3b9640 1 -- 172.21.15.45:0/1128795637 reap_dead start 2023-11-28T20:51:37.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.229+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1128795637 shutdown_connections 2023-11-28T20:51:37.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.229+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1128795637 >> 172.21.15.45:0/1128795637 conn(0x7fca4806c8f0 msgr2=0x7fca4806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:37.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.229+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1128795637 shutdown_connections 2023-11-28T20:51:37.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.229+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1128795637 wait complete. 2023-11-28T20:51:37.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.231+0000 7fca4fe1c640 1 Processor -- start 2023-11-28T20:51:37.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.231+0000 7fca4fe1c640 1 -- start start 2023-11-28T20:51:37.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4fe1c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca34004d00 0x7fca4815cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4fe1c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca480d09d0 0x7fca480cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4fe1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca480cbc50 0x7fca480cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4fe1c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca480cc610 con 0x7fca480cbc50 2023-11-28T20:51:37.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4fe1c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca480ce730 con 0x7fca480d09d0 2023-11-28T20:51:37.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4fe1c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4815d070 con 0x7fca34004d00 2023-11-28T20:51:37.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4e3b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca480cbc50 0x7fca480cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4e3b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca480cbc50 0x7fca480cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58430/0 (socket says 172.21.15.45:58430) 2023-11-28T20:51:37.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4e3b9640 1 -- 172.21.15.45:0/1403709000 learned_addr learned my addr 172.21.15.45:0/1403709000 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:37.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4dbb8640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca34004d00 0x7fca4815cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.232+0000 7fca4d3b7640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca480d09d0 0x7fca480cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca4e3b9640 1 -- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca34004d00 msgr2=0x7fca4815cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca4e3b9640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca34004d00 0x7fca4815cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca4e3b9640 1 -- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca480d09d0 msgr2=0x7fca480cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca4e3b9640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca480d09d0 0x7fca480cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca4e3b9640 1 -- 172.21.15.45:0/1403709000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca480cb5a0 con 0x7fca480cbc50 2023-11-28T20:51:37.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca4d3b7640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca480d09d0 0x7fca480cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:37.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca4e3b9640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca480cbc50 0x7fca480cc0d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c330 tx=0x7fca40001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:37.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca3e7fc640 1 -- 172.21.15.45:0/1403709000 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4003c040 con 0x7fca480cbc50 2023-11-28T20:51:37.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca3e7fc640 1 -- 172.21.15.45:0/1403709000 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca40001630 con 0x7fca480cbc50 2023-11-28T20:51:37.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca3e7fc640 1 -- 172.21.15.45:0/1403709000 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40007270 con 0x7fca480cbc50 2023-11-28T20:51:37.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca480ce0e0 con 0x7fca480cbc50 2023-11-28T20:51:37.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.233+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca480cb220 con 0x7fca480cbc50 2023-11-28T20:51:37.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.235+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca480733d0 con 0x7fca480cbc50 2023-11-28T20:51:37.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.235+0000 7fca3e7fc640 1 -- 172.21.15.45:0/1403709000 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fca40032070 con 0x7fca480cbc50 2023-11-28T20:51:37.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.236+0000 7fca3e7fc640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fca2c06aed0 0x7fca2c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.236+0000 7fca3e7fc640 1 -- 172.21.15.45:0/1403709000 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6760+0+0 (secure 0 0 0) 0x7fca400aa7e0 con 0x7fca480cbc50 2023-11-28T20:51:37.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.237+0000 7fca4dbb8640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fca2c06aed0 0x7fca2c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.238+0000 7fca4dbb8640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fca2c06aed0 0x7fca2c06d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca44009fd0 tx=0x7fca4400b040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:37.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.241+0000 7fca3e7fc640 1 -- 172.21.15.45:0/1403709000 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca40075380 con 0x7fca480cbc50 2023-11-28T20:51:37.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.434+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca48070ca0 con 0x7fca480cbc50 2023-11-28T20:51:37.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.436+0000 7fca3e7fc640 1 -- 172.21.15.45:0/1403709000 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+17854 (secure 0 0 0) 0x7fca400789a0 con 0x7fca480cbc50 2023-11-28T20:51:37.437 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:37.438 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":104,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:33.864472+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:10.974716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:11.975222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1724154865":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/412684916":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/514984079":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/2474029917":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/3103043064":"2023-11-29T20:51:27.664495+0000","172.21.15.190:6833/1373961985":"2023-11-29T20:51:33.864326+0000","172.21.15.190:6832/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.190:6832/1373961985":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/3436414960":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3227357091":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3114762982":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/1429551158":"2023-11-29T20:51:33.864326+0000","172.21.15.190:6833/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/4025400682":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2627631957":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2067877259":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/1852415784":"2023-11-28T21:51:13.642235+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:37.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.438+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fca2c06aed0 msgr2=0x7fca2c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.438+0000 7fca4fe1c640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fca2c06aed0 0x7fca2c06d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca44009fd0 tx=0x7fca4400b040 comp rx=0 tx=0).stop 2023-11-28T20:51:37.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.438+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca480cbc50 msgr2=0x7fca480cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.438+0000 7fca4fe1c640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca480cbc50 0x7fca480cc0d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c330 tx=0x7fca40001a00 comp rx=0 tx=0).stop 2023-11-28T20:51:37.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.439+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 shutdown_connections 2023-11-28T20:51:37.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.439+0000 7fca4fe1c640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fca2c06aed0 0x7fca2c06d390 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.439+0000 7fca4fe1c640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca480cbc50 0x7fca480cc0d0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.439+0000 7fca4fe1c640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca480d09d0 0x7fca480cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.439+0000 7fca4fe1c640 1 --2- 172.21.15.45:0/1403709000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca34004d00 0x7fca4815cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.439+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 >> 172.21.15.45:0/1403709000 conn(0x7fca4806c8f0 msgr2=0x7fca4815d900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:37.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.439+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 shutdown_connections 2023-11-28T20:51:37.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.439+0000 7fca4fe1c640 1 -- 172.21.15.45:0/1403709000 wait complete. 2023-11-28T20:51:37.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T20:51:37.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.692+0000 7fb751c07640 1 Processor -- start 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.692+0000 7fb751c07640 1 -- start start 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.694+0000 7fb751c07640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb74c15b530 0x7fb74c0cc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.694+0000 7fb751c07640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb74c0cd640 0x7fb74c0c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.694+0000 7fb751c07640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c15b910 con 0x7fb74c0cedd0 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.694+0000 7fb751c07640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c09a5f0 con 0x7fb74c15b530 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.694+0000 7fb751c07640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c09a8d0 con 0x7fb74c0cd640 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.694+0000 7fb74b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb74c15b530 0x7fb74c0cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.694+0000 7fb74b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb74c15b530 0x7fb74c0cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58444/0 (socket says 172.21.15.45:58444) 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.694+0000 7fb74b7fe640 1 -- 172.21.15.45:0/3775652007 learned_addr learned my addr 172.21.15.45:0/3775652007 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.694+0000 7fb743fff640 1 --2- 172.21.15.45:0/3775652007 >> v2:172.21.15.190:3300/0 conn(0x7fb74c0cd640 0x7fb74c0c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.695+0000 7fb74b7fe640 1 -- 172.21.15.45:0/3775652007 >> v2:172.21.15.190:3300/0 conn(0x7fb74c0cd640 msgr2=0x7fb74c0c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.695+0000 7fb74b7fe640 1 --2- 172.21.15.45:0/3775652007 >> v2:172.21.15.190:3300/0 conn(0x7fb74c0cd640 0x7fb74c0c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.695+0000 7fb74b7fe640 1 -- 172.21.15.45:0/3775652007 >> v1:172.21.15.190:6789/0 conn(0x7fb74c0cedd0 legacy=0x7fb74c0cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.695+0000 7fb74b7fe640 1 -- 172.21.15.45:0/3775652007 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0cc090 con 0x7fb74c15b530 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.696+0000 7fb74b7fe640 1 --2- 172.21.15.45:0/3775652007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c15b530 0x7fb74c0cc890 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fb74c072d10 tx=0x7fb73c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1888d5e31d3c10f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.696+0000 7fb74affd640 1 -- 172.21.15.45:0/3775652007 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c031870 con 0x7fb74c15b530 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.696+0000 7fb74affd640 1 -- 172.21.15.45:0/3775652007 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c0319d0 con 0x7fb74c15b530 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.696+0000 7fb74affd640 1 -- 172.21.15.45:0/3775652007 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c031d60 con 0x7fb74c15b530 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.697+0000 7fb751c07640 1 -- 172.21.15.45:0/3775652007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c15b530 msgr2=0x7fb74c0cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.697+0000 7fb751c07640 1 --2- 172.21.15.45:0/3775652007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c15b530 0x7fb74c0cc890 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fb74c072d10 tx=0x7fb73c02b910 comp rx=0 tx=0).stop 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.697+0000 7fb751c07640 1 -- 172.21.15.45:0/3775652007 shutdown_connections 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.697+0000 7fb751c07640 1 --2- 172.21.15.45:0/3775652007 >> v2:172.21.15.190:3300/0 conn(0x7fb74c0cd640 0x7fb74c0c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.697+0000 7fb751c07640 1 --2- 172.21.15.45:0/3775652007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c15b530 0x7fb74c0cc890 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.697+0000 7fb751c07640 1 -- 172.21.15.45:0/3775652007 >> 172.21.15.45:0/3775652007 conn(0x7fb74c0b2ae0 msgr2=0x7fb74c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:37.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.697+0000 7fb751c07640 1 -- 172.21.15.45:0/3775652007 shutdown_connections 2023-11-28T20:51:37.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.697+0000 7fb751c07640 1 -- 172.21.15.45:0/3775652007 wait complete. 2023-11-28T20:51:37.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.699+0000 7fb751c07640 1 Processor -- start 2023-11-28T20:51:37.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.699+0000 7fb751c07640 1 -- start start 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.700+0000 7fb751c07640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c0cd640 0x7fb74c0b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.700+0000 7fb751c07640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb74c0cedd0 0x7fb74c158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.700+0000 7fb751c07640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c15b530 0x7fb74c158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.700+0000 7fb751c07640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0b6d20 con 0x7fb74c0cd640 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.700+0000 7fb751c07640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0b6a40 con 0x7fb74c15b530 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.700+0000 7fb751c07640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1f4460 con 0x7fb74c0cedd0 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c0cd640 0x7fb74c0b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c0cd640 0x7fb74c0b6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58454/0 (socket says 172.21.15.45:58454) 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74b7fe640 1 -- 172.21.15.45:0/1957386289 learned_addr learned my addr 172.21.15.45:0/1957386289 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74bfff640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c15b530 0x7fb74c158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb743fff640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb74c0cedd0 0x7fb74c158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74b7fe640 1 -- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb74c0cedd0 msgr2=0x7fb74c158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74b7fe640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb74c0cedd0 0x7fb74c158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74b7fe640 1 -- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c15b530 msgr2=0x7fb74c158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74b7fe640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c15b530 0x7fb74c158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74b7fe640 1 -- 172.21.15.45:0/1957386289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c0094d0 con 0x7fb74c0cd640 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.701+0000 7fb74bfff640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c15b530 0x7fb74c158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.702+0000 7fb74b7fe640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c0cd640 0x7fb74c0b6460 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fb73c000c00 tx=0x7fb73c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.702+0000 7fb748ff9640 1 -- 172.21.15.45:0/1957386289 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c03abe0 con 0x7fb74c0cd640 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.702+0000 7fb748ff9640 1 -- 172.21.15.45:0/1957386289 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c03ad40 con 0x7fb74c0cd640 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.702+0000 7fb748ff9640 1 -- 172.21.15.45:0/1957386289 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c044610 con 0x7fb74c0cd640 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.702+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c1f4740 con 0x7fb74c0cd640 2023-11-28T20:51:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.702+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c1f4f40 con 0x7fb74c0cd640 2023-11-28T20:51:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.703+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c1f4c60 con 0x7fb74c0cd640 2023-11-28T20:51:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.704+0000 7fb748ff9640 1 -- 172.21.15.45:0/1957386289 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fb73c04d450 con 0x7fb74c0cd640 2023-11-28T20:51:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.705+0000 7fb748ff9640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb72c06aed0 0x7fb72c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.705+0000 7fb748ff9640 1 -- 172.21.15.45:0/1957386289 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6760+0+0 (secure 0 0 0) 0x7fb73c02d040 con 0x7fb74c0cd640 2023-11-28T20:51:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.706+0000 7fb743fff640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb72c06aed0 0x7fb72c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:37.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.707+0000 7fb743fff640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb72c06aed0 0x7fb72c06d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0c6a50 tx=0x7fb734008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:37.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.709+0000 7fb748ff9640 1 -- 172.21.15.45:0/1957386289 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb74c1f4c60 con 0x7fb74c0cd640 2023-11-28T20:51:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.958+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb74c15b910 con 0x7fb74c0cd640 2023-11-28T20:51:37.960 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T20:51:37.959+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T20:51:37.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.963+0000 7fb748ff9640 1 -- 172.21.15.45:0/1957386289 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb74c15b910 con 0x7fb74c0cd640 2023-11-28T20:51:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T20:51:37.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb72c06aed0 msgr2=0x7fb72c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb72c06aed0 0x7fb72c06d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0c6a50 tx=0x7fb734008040 comp rx=0 tx=0).stop 2023-11-28T20:51:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c0cd640 msgr2=0x7fb74c0b6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c0cd640 0x7fb74c0b6460 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fb73c000c00 tx=0x7fb73c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 shutdown_connections 2023-11-28T20:51:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb72c06aed0 0x7fb72c06d390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb74c15b530 0x7fb74c158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb74c0cedd0 0x7fb74c158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 --2- 172.21.15.45:0/1957386289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb74c0cd640 0x7fb74c0b6460 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.967+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 >> 172.21.15.45:0/1957386289 conn(0x7fb74c0b2ae0 msgr2=0x7fb74c0c59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.968+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 shutdown_connections 2023-11-28T20:51:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:37.968+0000 7fb751c07640 1 -- 172.21.15.45:0/1957386289 wait complete. 2023-11-28T20:51:37.976 INFO:tasks.ceph.mds.c.smithi045.stderr: -2669> 2023-11-28T20:51:18.436+0000 7ff5d20b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:51:37.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T20:51:38.038 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:51:38.048 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:38.045+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.186+0000 7fb79b298640 1 Processor -- start 2023-11-28T20:51:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.186+0000 7fb79b298640 1 -- start start 2023-11-28T20:51:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.187+0000 7fb79b298640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb7940ac7f0 0x7fb7940710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.188+0000 7fb79b298640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7940addf0 con 0x7fb7940ad7e0 2023-11-28T20:51:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.188+0000 7fb79b298640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb794070c20 con 0x7fb79406dbc0 2023-11-28T20:51:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.188+0000 7fb79b298640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7940acc60 con 0x7fb7940ac7f0 2023-11-28T20:51:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.188+0000 7fb799a95640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb7940ad7e0 0x7fb7940b1d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54958/0 (socket says 172.21.15.45:54958) 2023-11-28T20:51:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.188+0000 7fb799a95640 1 -- 172.21.15.45:0/2374338715 learned_addr learned my addr 172.21.15.45:0/2374338715 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.188+0000 7fb79aa97640 1 --2- 172.21.15.45:0/2374338715 >> v2:172.21.15.45:3300/0 conn(0x7fb7940ac7f0 0x7fb7940710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.189+0000 7fb79aa97640 1 -- 172.21.15.45:0/2374338715 >> v1:172.21.15.190:6789/0 conn(0x7fb79406dbc0 legacy=0x7fb79406dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.189+0000 7fb79aa97640 1 -- 172.21.15.45:0/2374338715 >> v1:172.21.15.45:6789/0 conn(0x7fb7940ad7e0 legacy=0x7fb7940b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.189+0000 7fb799294640 1 -- 172.21.15.45:0/2374338715 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330676027 0 0) 0x7fb7940addf0 con 0x7fb7940ad7e0 2023-11-28T20:51:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.189+0000 7fb79aa97640 1 -- 172.21.15.45:0/2374338715 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7940702f0 con 0x7fb7940ac7f0 2023-11-28T20:51:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.190+0000 7fb79aa97640 1 --2- 172.21.15.45:0/2374338715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ac7f0 0x7fb7940710b0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009d90 tx=0x7fb78c02e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa2a9684fb89067 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:38.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.190+0000 7fb799294640 1 -- 172.21.15.45:0/2374338715 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c02ee70 con 0x7fb7940ac7f0 2023-11-28T20:51:38.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.190+0000 7fb799294640 1 -- 172.21.15.45:0/2374338715 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c036440 con 0x7fb7940ac7f0 2023-11-28T20:51:38.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.190+0000 7fb799294640 1 -- 172.21.15.45:0/2374338715 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c0367d0 con 0x7fb7940ac7f0 2023-11-28T20:51:38.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.191+0000 7fb79b298640 1 -- 172.21.15.45:0/2374338715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ac7f0 msgr2=0x7fb7940710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:38.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.191+0000 7fb79b298640 1 --2- 172.21.15.45:0/2374338715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ac7f0 0x7fb7940710b0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009d90 tx=0x7fb78c02e0c0 comp rx=0 tx=0).stop 2023-11-28T20:51:38.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.191+0000 7fb79b298640 1 -- 172.21.15.45:0/2374338715 shutdown_connections 2023-11-28T20:51:38.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.191+0000 7fb79b298640 1 --2- 172.21.15.45:0/2374338715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ac7f0 0x7fb7940710b0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:38.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.191+0000 7fb79b298640 1 -- 172.21.15.45:0/2374338715 >> 172.21.15.45:0/2374338715 conn(0x7fb7940cecd0 msgr2=0x7fb7940cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:38.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.191+0000 7fb79b298640 1 -- 172.21.15.45:0/2374338715 shutdown_connections 2023-11-28T20:51:38.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.191+0000 7fb79b298640 1 -- 172.21.15.45:0/2374338715 wait complete. 2023-11-28T20:51:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.193+0000 7fb79b298640 1 Processor -- start 2023-11-28T20:51:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.193+0000 7fb79b298640 1 -- start start 2023-11-28T20:51:38.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.194+0000 7fb79b298640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb79406dbc0 0x7fb794142750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:38.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.194+0000 7fb79b298640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7940ac7f0 0x7fb79413d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:38.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.194+0000 7fb79b298640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ad7e0 0x7fb79413ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:38.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.194+0000 7fb79b298640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79413dd30 con 0x7fb7940ad7e0 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.194+0000 7fb79b298640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79413e010 con 0x7fb79406dbc0 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.194+0000 7fb79b298640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79413e2f0 con 0x7fb7940ac7f0 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.194+0000 7fb79aa97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ad7e0 0x7fb79413ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.195+0000 7fb79aa97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ad7e0 0x7fb79413ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58498/0 (socket says 172.21.15.45:58498) 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.195+0000 7fb79aa97640 1 -- 172.21.15.45:0/3083910278 learned_addr learned my addr 172.21.15.45:0/3083910278 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.195+0000 7fb79aa97640 1 -- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7940ac7f0 msgr2=0x7fb79413d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.195+0000 7fb79aa97640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7940ac7f0 0x7fb79413d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.195+0000 7fb79aa97640 1 -- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb79406dbc0 msgr2=0x7fb794142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.195+0000 7fb79a296640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb79406dbc0 0x7fb794142750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.195+0000 7fb79aa97640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb79406dbc0 0x7fb794142750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.195+0000 7fb79aa97640 1 -- 172.21.15.45:0/3083910278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c009a70 con 0x7fb7940ad7e0 2023-11-28T20:51:38.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.195+0000 7fb79aa97640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ad7e0 0x7fb79413ff10 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009d90 tx=0x7fb78c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:38.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.196+0000 7fb786ffd640 1 -- 172.21.15.45:0/3083910278 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c003b90 con 0x7fb7940ad7e0 2023-11-28T20:51:38.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.196+0000 7fb786ffd640 1 -- 172.21.15.45:0/3083910278 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c003cf0 con 0x7fb7940ad7e0 2023-11-28T20:51:38.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.196+0000 7fb786ffd640 1 -- 172.21.15.45:0/3083910278 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c004250 con 0x7fb7940ad7e0 2023-11-28T20:51:38.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.196+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb79413e5d0 con 0x7fb7940ad7e0 2023-11-28T20:51:38.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.196+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb79409a2c0 con 0x7fb7940ad7e0 2023-11-28T20:51:38.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.197+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79413eaf0 con 0x7fb7940ad7e0 2023-11-28T20:51:38.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.198+0000 7fb786ffd640 1 -- 172.21.15.45:0/3083910278 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fb78c03d070 con 0x7fb7940ad7e0 2023-11-28T20:51:38.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.199+0000 7fb786ffd640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb77806aed0 0x7fb77806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:38.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.199+0000 7fb786ffd640 1 -- 172.21.15.45:0/3083910278 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6846+0+0 (secure 0 0 0) 0x7fb78c0ac290 con 0x7fb7940ad7e0 2023-11-28T20:51:38.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.200+0000 7fb79a296640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb77806aed0 0x7fb77806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:38.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.201+0000 7fb79a296640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb77806aed0 0x7fb77806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb790000f60 tx=0x7fb790009040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:38.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.203+0000 7fb786ffd640 1 -- 172.21.15.45:0/3083910278 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb79413eaf0 con 0x7fb7940ad7e0 2023-11-28T20:51:38.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.443+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb79413eaf0 con 0x7fb7940ad7e0 2023-11-28T20:51:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.967+0000 7fb786ffd640 1 -- 172.21.15.45:0/3083910278 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb79413eaf0 con 0x7fb7940ad7e0 2023-11-28T20:51:38.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb77806aed0 msgr2=0x7fb77806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:38.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb77806aed0 0x7fb77806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb790000f60 tx=0x7fb790009040 comp rx=0 tx=0).stop 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ad7e0 msgr2=0x7fb79413ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ad7e0 0x7fb79413ff10 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009d90 tx=0x7fb78c004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 shutdown_connections 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb77806aed0 0x7fb77806d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7940ad7e0 0x7fb79413ff10 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7940ac7f0 0x7fb79413d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 --2- 172.21.15.45:0/3083910278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb79406dbc0 0x7fb794142750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 >> 172.21.15.45:0/3083910278 conn(0x7fb7940cecd0 msgr2=0x7fb794159660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 shutdown_connections 2023-11-28T20:51:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:38.970+0000 7fb79b298640 1 -- 172.21.15.45:0/3083910278 wait complete. 2023-11-28T20:51:38.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T20:51:39.133 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:39.131+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:39.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.225+0000 7f8ba0017640 1 Processor -- start 2023-11-28T20:51:39.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.225+0000 7f8ba0017640 1 -- start start 2023-11-28T20:51:39.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.227+0000 7f8ba0017640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8b9815e840 0x7f8b9806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:39.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.227+0000 7f8ba0017640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9815ecd0 con 0x7f8b9815f720 2023-11-28T20:51:39.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.227+0000 7f8ba0017640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9815f290 con 0x7f8b9806ac10 2023-11-28T20:51:39.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.227+0000 7f8ba0017640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b9815fb60 con 0x7f8b9815e840 2023-11-28T20:51:39.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.227+0000 7f8b9ddb3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8b9815e840 0x7f8b9806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:39.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.227+0000 7f8b9ddb3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8b9815e840 0x7f8b9806c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58514/0 (socket says 172.21.15.45:58514) 2023-11-28T20:51:39.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.227+0000 7f8b9ddb3640 1 -- 172.21.15.45:0/3552310707 learned_addr learned my addr 172.21.15.45:0/3552310707 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:39.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.228+0000 7f8b9ddb3640 1 -- 172.21.15.45:0/3552310707 >> v1:172.21.15.190:6789/0 conn(0x7f8b9806ac10 legacy=0x7f8b980cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:39.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.228+0000 7f8b9ddb3640 1 -- 172.21.15.45:0/3552310707 >> v1:172.21.15.45:6789/0 conn(0x7f8b9815f720 legacy=0x7f8b9806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:39.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.228+0000 7f8b9ddb3640 1 -- 172.21.15.45:0/3552310707 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98072020 con 0x7f8b9815e840 2023-11-28T20:51:39.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.229+0000 7f8b9ddb3640 1 --2- 172.21.15.45:0/3552310707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815e840 0x7f8b9806c350 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009870 tx=0x7f8b9402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bad6c131ea9fda server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:39.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.229+0000 7f8b9cdb1640 1 -- 172.21.15.45:0/3552310707 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94030860 con 0x7f8b9815e840 2023-11-28T20:51:39.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.229+0000 7f8b9cdb1640 1 -- 172.21.15.45:0/3552310707 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b940309c0 con 0x7f8b9815e840 2023-11-28T20:51:39.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.229+0000 7f8b9cdb1640 1 -- 172.21.15.45:0/3552310707 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94030d50 con 0x7f8b9815e840 2023-11-28T20:51:39.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.230+0000 7f8ba0017640 1 -- 172.21.15.45:0/3552310707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815e840 msgr2=0x7f8b9806c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:39.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.230+0000 7f8ba0017640 1 --2- 172.21.15.45:0/3552310707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815e840 0x7f8b9806c350 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009870 tx=0x7f8b9402b9a0 comp rx=0 tx=0).stop 2023-11-28T20:51:39.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.230+0000 7f8ba0017640 1 -- 172.21.15.45:0/3552310707 shutdown_connections 2023-11-28T20:51:39.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.230+0000 7f8ba0017640 1 --2- 172.21.15.45:0/3552310707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815e840 0x7f8b9806c350 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:39.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.230+0000 7f8ba0017640 1 -- 172.21.15.45:0/3552310707 >> 172.21.15.45:0/3552310707 conn(0x7f8b980ced60 msgr2=0x7f8b980cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:39.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.230+0000 7f8ba0017640 1 -- 172.21.15.45:0/3552310707 shutdown_connections 2023-11-28T20:51:39.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.230+0000 7f8ba0017640 1 -- 172.21.15.45:0/3552310707 wait complete. 2023-11-28T20:51:39.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.233+0000 7f8ba0017640 1 Processor -- start 2023-11-28T20:51:39.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.233+0000 7f8ba0017640 1 -- start start 2023-11-28T20:51:39.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8ba0017640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b9806ac10 0x7f8b9815c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:39.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8ba0017640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b9815e840 0x7f8b981f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:39.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8ba0017640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815f720 0x7f8b981e4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:39.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8ba0017640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980b2ab0 con 0x7f8b9815f720 2023-11-28T20:51:39.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8ba0017640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9815ca00 con 0x7f8b9806ac10 2023-11-28T20:51:39.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8ba0017640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9815ccd0 con 0x7f8b9815e840 2023-11-28T20:51:39.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8b9e5b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815f720 0x7f8b981e4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8b9e5b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815f720 0x7f8b981e4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58518/0 (socket says 172.21.15.45:58518) 2023-11-28T20:51:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8b9e5b4640 1 -- 172.21.15.45:0/3840539642 learned_addr learned my addr 172.21.15.45:0/3840539642 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:39.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.234+0000 7f8b9ddb3640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b9806ac10 0x7f8b9815c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:39.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b9d5b2640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b9815e840 0x7f8b981f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:39.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b9e5b4640 1 -- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b9815e840 msgr2=0x7f8b981f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:39.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b9e5b4640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b9815e840 0x7f8b981f3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b9e5b4640 1 -- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b9806ac10 msgr2=0x7f8b9815c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b9e5b4640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b9806ac10 0x7f8b9815c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b9e5b4640 1 -- 172.21.15.45:0/3840539642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b940094d0 con 0x7f8b9815f720 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b9ddb3640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b9806ac10 0x7f8b9815c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b9e5b4640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815f720 0x7f8b981e4ad0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000c780 tx=0x7f8b9000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/3840539642 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90004b10 con 0x7f8b9815f720 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/3840539642 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b90004c70 con 0x7f8b9815f720 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/3840539642 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9000aa90 con 0x7f8b9815f720 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9815cfb0 con 0x7f8b9815f720 2023-11-28T20:51:39.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.235+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b981f4430 con 0x7f8b9815f720 2023-11-28T20:51:39.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.237+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9815d4d0 con 0x7f8b9815f720 2023-11-28T20:51:39.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.237+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/3840539642 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f8b9000abf0 con 0x7f8b9815f720 2023-11-28T20:51:39.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.239+0000 7f8b8e7fc640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f8b7c06aed0 0x7f8b7c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:39.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.239+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/3840539642 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6846+0+0 (secure 0 0 0) 0x7f8b90089c60 con 0x7f8b9815f720 2023-11-28T20:51:39.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.240+0000 7f8b9ddb3640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f8b7c06aed0 0x7f8b7c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:39.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.241+0000 7f8b9ddb3640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f8b7c06aed0 0x7f8b7c06d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8b94036000 tx=0x7f8b9403b040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:39.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.244+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/3840539642 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b900547b0 con 0x7f8b9815f720 2023-11-28T20:51:39.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:39.448+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8b980b27f0 con 0x7f8b9815f720 2023-11-28T20:51:40.271 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:40.268+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:40.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.350+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/3840539642 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v106) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8b90057dd0 con 0x7f8b9815f720 2023-11-28T20:51:40.352 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T20:51:40.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.354+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f8b7c06aed0 msgr2=0x7f8b7c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:40.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.354+0000 7f8ba0017640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f8b7c06aed0 0x7f8b7c06d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8b94036000 tx=0x7f8b9403b040 comp rx=0 tx=0).stop 2023-11-28T20:51:40.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.354+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815f720 msgr2=0x7f8b981e4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:40.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.354+0000 7f8ba0017640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815f720 0x7f8b981e4ad0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000c780 tx=0x7f8b9000cc40 comp rx=0 tx=0).stop 2023-11-28T20:51:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.355+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 shutdown_connections 2023-11-28T20:51:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.355+0000 7f8ba0017640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f8b7c06aed0 0x7f8b7c06d390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.355+0000 7f8ba0017640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9815f720 0x7f8b981e4ad0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.355+0000 7f8ba0017640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b9815e840 0x7f8b981f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.355+0000 7f8ba0017640 1 --2- 172.21.15.45:0/3840539642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b9806ac10 0x7f8b9815c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.355+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 >> 172.21.15.45:0/3840539642 conn(0x7f8b980ced60 msgr2=0x7f8b980cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:40.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.355+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 shutdown_connections 2023-11-28T20:51:40.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.355+0000 7f8ba0017640 1 -- 172.21.15.45:0/3840539642 wait complete. 2023-11-28T20:51:40.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T20:51:40.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.588+0000 7fa5a081a640 1 Processor -- start 2023-11-28T20:51:40.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.588+0000 7fa5a081a640 1 -- start start 2023-11-28T20:51:40.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.589+0000 7fa5a081a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c06e140 0x7fa59c0ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:40.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.589+0000 7fa5a081a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59c071490 con 0x7fa59c0ca5c0 2023-11-28T20:51:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.589+0000 7fa5a081a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59c0716e0 con 0x7fa59c159f50 2023-11-28T20:51:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.589+0000 7fa5a081a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c0719c0 con 0x7fa59c06e140 2023-11-28T20:51:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.590+0000 7fa59bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa59c0ca5c0 0x7fa59c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54980/0 (socket says 172.21.15.45:54980) 2023-11-28T20:51:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.590+0000 7fa59bfff640 1 -- 172.21.15.45:0/1599405027 learned_addr learned my addr 172.21.15.45:0/1599405027 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:40.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.590+0000 7fa59affd640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c06e140 0x7fa59c0ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:40.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.590+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476997098 0 0) 0x7fa59c071490 con 0x7fa59c0ca5c0 2023-11-28T20:51:40.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.590+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa584003520 con 0x7fa59c0ca5c0 2023-11-28T20:51:40.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869681873 0 0) 0x7fa584003520 con 0x7fa59c0ca5c0 2023-11-28T20:51:40.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c06e140 msgr2=0x7fa59c0ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:40.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c06e140 0x7fa59c0ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 >> v1:172.21.15.190:6789/0 conn(0x7fa59c159f50 legacy=0x7fa59c06da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c071ca0 con 0x7fa59c0ca5c0 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fa5940019f0 con 0x7fa59c0ca5c0 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 >> v1:172.21.15.45:6789/0 conn(0x7fa59c0ca5c0 legacy=0x7fa59c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5840039a0 0x7fa584003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c06e140 0x7fa5840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa584005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c0719c0 con 0x7fa584004d00 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58400dc70 con 0x7fa5840039a0 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58400de30 con 0x7fa59c06e140 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59bfff640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa584005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:40.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.591+0000 7fa59b7fe640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5840039a0 0x7fa584003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.592+0000 7fa59affd640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c06e140 0x7fa5840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.592+0000 7fa59bfff640 1 -- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c06e140 msgr2=0x7fa5840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.592+0000 7fa59bfff640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c06e140 0x7fa5840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.592+0000 7fa59bfff640 1 -- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5840039a0 msgr2=0x7fa584003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.592+0000 7fa59bfff640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5840039a0 0x7fa584003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.592+0000 7fa59bfff640 1 -- 172.21.15.45:0/1599405027 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa584003520 con 0x7fa584004d00 2023-11-28T20:51:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.592+0000 7fa59b7fe640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5840039a0 0x7fa584003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.593+0000 7fa59bfff640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa584005180 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fa594002230 tx=0x7fa5840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917da7cc5b328765 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.593+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594034630 con 0x7fa584004d00 2023-11-28T20:51:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.593+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59402f070 con 0x7fa584004d00 2023-11-28T20:51:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.593+0000 7fa59a7fc640 1 -- 172.21.15.45:0/1599405027 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594034a10 con 0x7fa584004d00 2023-11-28T20:51:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.594+0000 7fa5a081a640 1 -- 172.21.15.45:0/1599405027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 msgr2=0x7fa584005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.594+0000 7fa5a081a640 1 --2- 172.21.15.45:0/1599405027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa584005180 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fa594002230 tx=0x7fa5840057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.594+0000 7fa59bfff640 1 -- 172.21.15.45:0/1599405027 reap_dead start 2023-11-28T20:51:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.594+0000 7fa5a081a640 1 -- 172.21.15.45:0/1599405027 shutdown_connections 2023-11-28T20:51:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.594+0000 7fa5a081a640 1 -- 172.21.15.45:0/1599405027 >> 172.21.15.45:0/1599405027 conn(0x7fa59c0cec30 msgr2=0x7fa59c0cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.594+0000 7fa5a081a640 1 -- 172.21.15.45:0/1599405027 shutdown_connections 2023-11-28T20:51:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.594+0000 7fa5a081a640 1 -- 172.21.15.45:0/1599405027 wait complete. 2023-11-28T20:51:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.596+0000 7fa5a081a640 1 Processor -- start 2023-11-28T20:51:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.596+0000 7fa5a081a640 1 -- start start 2023-11-28T20:51:40.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.597+0000 7fa5a081a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa59c14fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:40.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.597+0000 7fa5a081a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c0c6470 0x7fa59c0c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:40.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.597+0000 7fa5a081a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa59c13ba70 0x7fa59c13bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:40.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.597+0000 7fa5a081a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c0c6e30 con 0x7fa584004d00 2023-11-28T20:51:40.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.597+0000 7fa5a081a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c14fa40 con 0x7fa59c13ba70 2023-11-28T20:51:40.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.597+0000 7fa5a081a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c150410 con 0x7fa59c0c6470 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa59c14fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa59c14fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58540/0 (socket says 172.21.15.45:58540) 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59b7fe640 1 -- 172.21.15.45:0/2605024124 learned_addr learned my addr 172.21.15.45:0/2605024124 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59affd640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c0c6470 0x7fa59c0c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59bfff640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa59c13ba70 0x7fa59c13bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59b7fe640 1 -- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c0c6470 msgr2=0x7fa59c0c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59b7fe640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c0c6470 0x7fa59c0c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59b7fe640 1 -- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa59c13ba70 msgr2=0x7fa59c13bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59b7fe640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa59c13ba70 0x7fa59c13bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:40.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.598+0000 7fa59b7fe640 1 -- 172.21.15.45:0/2605024124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c0cab70 con 0x7fa584004d00 2023-11-28T20:51:40.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.599+0000 7fa59b7fe640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa59c14fed0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fa58c00e550 tx=0x7fa58c00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:40.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.599+0000 7fa583fff640 1 -- 172.21.15.45:0/2605024124 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c0053a0 con 0x7fa584004d00 2023-11-28T20:51:40.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.599+0000 7fa583fff640 1 -- 172.21.15.45:0/2605024124 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c0035f0 con 0x7fa584004d00 2023-11-28T20:51:40.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.599+0000 7fa583fff640 1 -- 172.21.15.45:0/2605024124 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c003980 con 0x7fa584004d00 2023-11-28T20:51:40.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.599+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59c14ee30 con 0x7fa584004d00 2023-11-28T20:51:40.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.599+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59c14f3f0 con 0x7fa584004d00 2023-11-28T20:51:40.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.600+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59c0733d0 con 0x7fa584004d00 2023-11-28T20:51:40.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.601+0000 7fa583fff640 1 -- 172.21.15.45:0/2605024124 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fa58c003ae0 con 0x7fa584004d00 2023-11-28T20:51:40.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.603+0000 7fa583fff640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fa58806aed0 0x7fa58806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:40.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.603+0000 7fa583fff640 1 -- 172.21.15.45:0/2605024124 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6408+0+0 (secure 0 0 0) 0x7fa58c089800 con 0x7fa584004d00 2023-11-28T20:51:40.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.603+0000 7fa59affd640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fa58806aed0 0x7fa58806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:40.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.604+0000 7fa59affd640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fa58806aed0 0x7fa58806d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa59c13b940 tx=0x7fa590008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:40.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.608+0000 7fa583fff640 1 -- 172.21.15.45:0/2605024124 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa58c054510 con 0x7fa584004d00 2023-11-28T20:51:40.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:40.813+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa59c1580b0 con 0x7fa584004d00 2023-11-28T20:51:41.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.366+0000 7fa583fff640 1 -- 172.21.15.45:0/2605024124 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v107) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa58c057b30 con 0x7fa584004d00 2023-11-28T20:51:41.368 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T20:51:41.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.371+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fa58806aed0 msgr2=0x7fa58806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:41.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.371+0000 7fa5a081a640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fa58806aed0 0x7fa58806d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa59c13b940 tx=0x7fa590008040 comp rx=0 tx=0).stop 2023-11-28T20:51:41.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.371+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 msgr2=0x7fa59c14fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:41.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.371+0000 7fa5a081a640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa59c14fed0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fa58c00e550 tx=0x7fa58c00ea10 comp rx=0 tx=0).stop 2023-11-28T20:51:41.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.372+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 shutdown_connections 2023-11-28T20:51:41.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.372+0000 7fa5a081a640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fa58806aed0 0x7fa58806d390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.372+0000 7fa5a081a640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa59c13ba70 0x7fa59c13bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.372+0000 7fa5a081a640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa59c0c6470 0x7fa59c0c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.372+0000 7fa5a081a640 1 --2- 172.21.15.45:0/2605024124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa584004d00 0x7fa59c14fed0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.372+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 >> 172.21.15.45:0/2605024124 conn(0x7fa59c0cec30 msgr2=0x7fa59c0aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:41.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.372+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 shutdown_connections 2023-11-28T20:51:41.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.373+0000 7fa5a081a640 1 -- 172.21.15.45:0/2605024124 wait complete. 2023-11-28T20:51:41.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T20:51:41.409 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:41.406+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:41.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.614+0000 7fdd83300640 1 Processor -- start 2023-11-28T20:51:41.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.614+0000 7fdd83300640 1 -- start start 2023-11-28T20:51:41.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.615+0000 7fdd83300640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0ce600 0x7fdd7c06a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.615+0000 7fdd83300640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd7c06aa40 0x7fdd7c0c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.616+0000 7fdd83300640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c06e030 con 0x7fdd7c06dc50 2023-11-28T20:51:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.616+0000 7fdd83300640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7c06aed0 con 0x7fdd7c06aa40 2023-11-28T20:51:41.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.616+0000 7fdd83300640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c0ccf60 con 0x7fdd7c0ce600 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.616+0000 7fdd8109c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdd7c06dc50 0x7fdd7c0cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54988/0 (socket says 172.21.15.45:54988) 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.616+0000 7fdd8109c640 1 -- 172.21.15.45:0/4056941561 learned_addr learned my addr 172.21.15.45:0/4056941561 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.616+0000 7fdd8189d640 1 --2- 172.21.15.45:0/4056941561 >> v2:172.21.15.190:3300/0 conn(0x7fdd7c06aa40 0x7fdd7c0c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.616+0000 7fdd8089b640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0ce600 0x7fdd7c06a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.616+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719812827 0 0) 0x7fdd78001490 con 0x7fdd7c06dc50 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd68003520 con 0x7fdd7c06dc50 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 618187726 0 0) 0x7fdd68003520 con 0x7fdd7c06dc50 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0ce600 msgr2=0x7fdd7c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0ce600 0x7fdd7c06a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 >> v2:172.21.15.190:3300/0 conn(0x7fdd7c06aa40 msgr2=0x7fdd7c0c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 --2- 172.21.15.45:0/4056941561 >> v2:172.21.15.190:3300/0 conn(0x7fdd7c06aa40 0x7fdd7c0c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c0cc930 con 0x7fdd7c06dc50 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fdd78001b80 con 0x7fdd7c06dc50 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 >> v1:172.21.15.45:6789/0 conn(0x7fdd7c06dc50 legacy=0x7fdd7c0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd68003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd680047c0 0x7fdd68004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd68005180 0x7fdd6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c06aed0 con 0x7fdd68003970 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c0ccf60 con 0x7fdd680047c0 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6800bf70 con 0x7fdd68005180 2023-11-28T20:51:41.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd8089b640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0ce600 0x7fdd7c06a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:41.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.617+0000 7fdd8109c640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd68003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:41.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.618+0000 7fdd8109c640 1 -- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd68005180 msgr2=0x7fdd6800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:41.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.618+0000 7fdd8109c640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd68005180 0x7fdd6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.618+0000 7fdd8109c640 1 -- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd680047c0 msgr2=0x7fdd68004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:41.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.618+0000 7fdd8109c640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd680047c0 0x7fdd68004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.618+0000 7fdd8109c640 1 -- 172.21.15.45:0/4056941561 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd68003520 con 0x7fdd68003970 2023-11-28T20:51:41.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.619+0000 7fdd8109c640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd68003d90 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800a780 tx=0x7fdd680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28ecb478f4f0d1a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:41.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.619+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd78034630 con 0x7fdd68003970 2023-11-28T20:51:41.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.619+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7802f070 con 0x7fdd68003970 2023-11-28T20:51:41.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.619+0000 7fdd73fff640 1 -- 172.21.15.45:0/4056941561 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd780349f0 con 0x7fdd68003970 2023-11-28T20:51:41.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.620+0000 7fdd83300640 1 -- 172.21.15.45:0/4056941561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 msgr2=0x7fdd68003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:41.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.620+0000 7fdd83300640 1 --2- 172.21.15.45:0/4056941561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd68003d90 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800a780 tx=0x7fdd680057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.620+0000 7fdd8189d640 1 -- 172.21.15.45:0/4056941561 reap_dead start 2023-11-28T20:51:41.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.620+0000 7fdd83300640 1 -- 172.21.15.45:0/4056941561 shutdown_connections 2023-11-28T20:51:41.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.620+0000 7fdd83300640 1 -- 172.21.15.45:0/4056941561 >> 172.21.15.45:0/4056941561 conn(0x7fdd7c0ced60 msgr2=0x7fdd7c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:41.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.620+0000 7fdd83300640 1 -- 172.21.15.45:0/4056941561 shutdown_connections 2023-11-28T20:51:41.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.620+0000 7fdd83300640 1 -- 172.21.15.45:0/4056941561 wait complete. 2023-11-28T20:51:41.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.622+0000 7fdd83300640 1 Processor -- start 2023-11-28T20:51:41.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.622+0000 7fdd83300640 1 -- start start 2023-11-28T20:51:41.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.623+0000 7fdd83300640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd7c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:41.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.623+0000 7fdd83300640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0c5f80 0x7fdd7c0c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:41.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.623+0000 7fdd83300640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd7c09bbd0 0x7fdd7c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:41.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.623+0000 7fdd83300640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c09ab30 con 0x7fdd68003970 2023-11-28T20:51:41.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.623+0000 7fdd83300640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c09ae10 con 0x7fdd7c09bbd0 2023-11-28T20:51:41.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.623+0000 7fdd83300640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c09b0f0 con 0x7fdd7c0c5f80 2023-11-28T20:51:41.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8109c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd7c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:41.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8109c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd7c1521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58566/0 (socket says 172.21.15.45:58566) 2023-11-28T20:51:41.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8109c640 1 -- 172.21.15.45:0/78573833 learned_addr learned my addr 172.21.15.45:0/78573833 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:41.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8189d640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd7c09bbd0 0x7fdd7c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:41.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8109c640 1 -- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0c5f80 msgr2=0x7fdd7c0c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:41.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8089b640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0c5f80 0x7fdd7c0c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:41.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8109c640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0c5f80 0x7fdd7c0c6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8109c640 1 -- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd7c09bbd0 msgr2=0x7fdd7c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:41.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8109c640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd7c09bbd0 0x7fdd7c0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8109c640 1 -- 172.21.15.45:0/78573833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c09b3d0 con 0x7fdd68003970 2023-11-28T20:51:41.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.624+0000 7fdd8109c640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd7c1521a0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800a0b0 tx=0x7fdd78001e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:41.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.625+0000 7fdd71ffb640 1 -- 172.21.15.45:0/78573833 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd78004c20 con 0x7fdd68003970 2023-11-28T20:51:41.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.625+0000 7fdd71ffb640 1 -- 172.21.15.45:0/78573833 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd78004d80 con 0x7fdd68003970 2023-11-28T20:51:41.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.625+0000 7fdd71ffb640 1 -- 172.21.15.45:0/78573833 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd78034670 con 0x7fdd68003970 2023-11-28T20:51:41.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.625+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7c09b6b0 con 0x7fdd68003970 2023-11-28T20:51:41.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.625+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c0c6d80 con 0x7fdd68003970 2023-11-28T20:51:41.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.626+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7c0733d0 con 0x7fdd68003970 2023-11-28T20:51:41.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.627+0000 7fdd71ffb640 1 -- 172.21.15.45:0/78573833 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fdd78031030 con 0x7fdd68003970 2023-11-28T20:51:41.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.628+0000 7fdd71ffb640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fdd6006af00 0x7fdd6006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:41.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.628+0000 7fdd71ffb640 1 -- 172.21.15.45:0/78573833 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7fdd780ab790 con 0x7fdd68003970 2023-11-28T20:51:41.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.629+0000 7fdd8089b640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fdd6006af00 0x7fdd6006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:41.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.630+0000 7fdd8089b640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fdd6006af00 0x7fdd6006d3c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0b2500 tx=0x7fdd6c008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:41.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.633+0000 7fdd71ffb640 1 -- 172.21.15.45:0/78573833 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd78076520 con 0x7fdd68003970 2023-11-28T20:51:41.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.842+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fdd7c06e030 con 0x7fdd68003970 2023-11-28T20:51:41.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.848+0000 7fdd71ffb640 1 -- 172.21.15.45:0/78573833 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v31) v1 ==== 120+0+0 (secure 0 0 0) 0x7fdd7c06e030 con 0x7fdd68003970 2023-11-28T20:51:41.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fdd6006af00 msgr2=0x7fdd6006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:41.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fdd6006af00 0x7fdd6006d3c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0b2500 tx=0x7fdd6c008040 comp rx=0 tx=0).stop 2023-11-28T20:51:41.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 msgr2=0x7fdd7c1521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:41.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd7c1521a0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800a0b0 tx=0x7fdd78001e40 comp rx=0 tx=0).stop 2023-11-28T20:51:41.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 shutdown_connections 2023-11-28T20:51:41.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fdd6006af00 0x7fdd6006d3c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd7c09bbd0 0x7fdd7c0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7c0c5f80 0x7fdd7c0c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 --2- 172.21.15.45:0/78573833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd68003970 0x7fdd7c1521a0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:41.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 >> 172.21.15.45:0/78573833 conn(0x7fdd7c0ced60 msgr2=0x7fdd7c160ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:41.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 shutdown_connections 2023-11-28T20:51:41.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:41.851+0000 7fdd83300640 1 -- 172.21.15.45:0/78573833 wait complete. 2023-11-28T20:51:41.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T20:51:42.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.097+0000 7f9fb3423640 1 Processor -- start 2023-11-28T20:51:42.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.097+0000 7f9fb3423640 1 -- start start 2023-11-28T20:51:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb3423640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac15e840 0x7f9fac06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb3423640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9fac15f720 0x7f9fac0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb3423640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9fac06a500 0x7f9fac06a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb3423640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fac071ce0 con 0x7f9fac15f720 2023-11-28T20:51:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb3423640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fac06aed0 con 0x7f9fac06a500 2023-11-28T20:51:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb3423640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac15ecd0 con 0x7f9fac15e840 2023-11-28T20:51:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb09be640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9fac15f720 0x7f9fac0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb09be640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9fac15f720 0x7f9fac0717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58568/0 (socket says 172.21.15.45:58568) 2023-11-28T20:51:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb09be640 1 -- 172.21.15.45:0/2410583521 learned_addr learned my addr 172.21.15.45:0/2410583521 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb19c0640 1 --2- 172.21.15.45:0/2410583521 >> v2:172.21.15.190:3300/0 conn(0x7f9fac06a500 0x7f9fac06a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.099+0000 7f9fb11bf640 1 --2- 172.21.15.45:0/2410583521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac15e840 0x7f9fac06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.100+0000 7f9fb09be640 1 -- 172.21.15.45:0/2410583521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac15e840 msgr2=0x7f9fac06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.100+0000 7f9fb09be640 1 --2- 172.21.15.45:0/2410583521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac15e840 0x7f9fac06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.100+0000 7f9fb09be640 1 -- 172.21.15.45:0/2410583521 >> v2:172.21.15.190:3300/0 conn(0x7f9fac06a500 msgr2=0x7f9fac06a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.100+0000 7f9fb09be640 1 --2- 172.21.15.45:0/2410583521 >> v2:172.21.15.190:3300/0 conn(0x7f9fac06a500 0x7f9fac06a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.100+0000 7f9fb09be640 1 -- 172.21.15.45:0/2410583521 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fac15f290 con 0x7f9fac15f720 2023-11-28T20:51:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.101+0000 7f9fb09be640 1 --2- 172.21.15.45:0/2410583521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15f720 0x7f9fac0717a0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009870 tx=0x7f9f9c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cdfa070a7bf4778 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:42.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.101+0000 7f9fa3fff640 1 -- 172.21.15.45:0/2410583521 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c031640 con 0x7f9fac15f720 2023-11-28T20:51:42.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.101+0000 7f9fa3fff640 1 -- 172.21.15.45:0/2410583521 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9c0317a0 con 0x7f9fac15f720 2023-11-28T20:51:42.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.101+0000 7f9fa3fff640 1 -- 172.21.15.45:0/2410583521 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c031b30 con 0x7f9fac15f720 2023-11-28T20:51:42.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.102+0000 7f9fb3423640 1 -- 172.21.15.45:0/2410583521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15f720 msgr2=0x7f9fac0717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.102+0000 7f9fb3423640 1 --2- 172.21.15.45:0/2410583521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15f720 0x7f9fac0717a0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009870 tx=0x7f9f9c02b8f0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.102+0000 7f9fb3423640 1 -- 172.21.15.45:0/2410583521 shutdown_connections 2023-11-28T20:51:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.102+0000 7f9fb3423640 1 --2- 172.21.15.45:0/2410583521 >> v2:172.21.15.190:3300/0 conn(0x7f9fac06a500 0x7f9fac06a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.102+0000 7f9fb3423640 1 --2- 172.21.15.45:0/2410583521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15f720 0x7f9fac0717a0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.102+0000 7f9fb3423640 1 --2- 172.21.15.45:0/2410583521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac15e840 0x7f9fac06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.102+0000 7f9fb3423640 1 -- 172.21.15.45:0/2410583521 >> 172.21.15.45:0/2410583521 conn(0x7f9fac0ced60 msgr2=0x7f9fac0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.102+0000 7f9fb3423640 1 -- 172.21.15.45:0/2410583521 shutdown_connections 2023-11-28T20:51:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.102+0000 7f9fb3423640 1 -- 172.21.15.45:0/2410583521 wait complete. 2023-11-28T20:51:42.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.104+0000 7f9fb3423640 1 Processor -- start 2023-11-28T20:51:42.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.104+0000 7f9fb3423640 1 -- start start 2023-11-28T20:51:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.105+0000 7f9fb3423640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac06a500 0x7f9fac156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.105+0000 7f9fb3423640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15e840 0x7f9fac0c2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.105+0000 7f9fb3423640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fac15f720 0x7f9fac14e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.105+0000 7f9fb3423640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac0c2f90 con 0x7f9fac15e840 2023-11-28T20:51:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.105+0000 7f9fb3423640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac09c840 con 0x7f9fac15f720 2023-11-28T20:51:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.105+0000 7f9fb3423640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac0c51c0 con 0x7f9fac06a500 2023-11-28T20:51:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.105+0000 7f9fb09be640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15e840 0x7f9fac0c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.105+0000 7f9fb09be640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15e840 0x7f9fac0c2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58570/0 (socket says 172.21.15.45:58570) 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.105+0000 7f9fb09be640 1 -- 172.21.15.45:0/1710313275 learned_addr learned my addr 172.21.15.45:0/1710313275 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.106+0000 7f9fb11bf640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac06a500 0x7f9fac156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.106+0000 7f9fb19c0640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fac15f720 0x7f9fac14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.106+0000 7f9fb09be640 1 -- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac06a500 msgr2=0x7f9fac156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.106+0000 7f9fb09be640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac06a500 0x7f9fac156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.106+0000 7f9fb09be640 1 -- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fac15f720 msgr2=0x7f9fac14e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.106+0000 7f9fb09be640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fac15f720 0x7f9fac14e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.106+0000 7f9fb09be640 1 -- 172.21.15.45:0/1710313275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c0094d0 con 0x7f9fac15e840 2023-11-28T20:51:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.106+0000 7f9fb19c0640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fac15f720 0x7f9fac14e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.106+0000 7f9fb09be640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15e840 0x7f9fac0c2890 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c000c00 tx=0x7f9f9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.107+0000 7f9fa1ffb640 1 -- 172.21.15.45:0/1710313275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c03a8b0 con 0x7f9fac15e840 2023-11-28T20:51:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.107+0000 7f9fa1ffb640 1 -- 172.21.15.45:0/1710313275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9c03aa10 con 0x7f9fac15e840 2023-11-28T20:51:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.107+0000 7f9fa1ffb640 1 -- 172.21.15.45:0/1710313275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c03adc0 con 0x7f9fac15e840 2023-11-28T20:51:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.107+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fac0c54a0 con 0x7f9fac15e840 2023-11-28T20:51:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.107+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fac14e6a0 con 0x7f9fac15e840 2023-11-28T20:51:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.108+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fac0b19a0 con 0x7f9fac15e840 2023-11-28T20:51:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.109+0000 7f9fa1ffb640 1 -- 172.21.15.45:0/1710313275 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f9f9c02f050 con 0x7f9fac15e840 2023-11-28T20:51:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.110+0000 7f9fa1ffb640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f9f9006aed0 0x7f9f9006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.110+0000 7f9fa1ffb640 1 -- 172.21.15.45:0/1710313275 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7f9f9c0ab750 con 0x7f9fac15e840 2023-11-28T20:51:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.110+0000 7f9fb11bf640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f9f9006aed0 0x7f9f9006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.111+0000 7f9fb11bf640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f9f9006aed0 0x7f9f9006d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9fac1569d0 tx=0x7f9fa8008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:42.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.115+0000 7f9fa1ffb640 1 -- 172.21.15.45:0/1710313275 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fac0b19a0 con 0x7f9fac15e840 2023-11-28T20:51:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.313+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9fac0b1d80 con 0x7f9fac15e840 2023-11-28T20:51:42.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.319+0000 7f9fa1ffb640 1 -- 172.21.15.45:0/1710313275 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9fac0b1d80 con 0x7f9fac15e840 2023-11-28T20:51:42.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f9f9006aed0 msgr2=0x7f9f9006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f9f9006aed0 0x7f9f9006d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9fac1569d0 tx=0x7f9fa8008040 comp rx=0 tx=0).stop 2023-11-28T20:51:42.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15e840 msgr2=0x7f9fac0c2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15e840 0x7f9fac0c2890 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c000c00 tx=0x7f9f9c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 shutdown_connections 2023-11-28T20:51:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f9f9006aed0 0x7f9f9006d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fac15f720 0x7f9fac14e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fac15e840 0x7f9fac0c2890 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 --2- 172.21.15.45:0/1710313275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fac06a500 0x7f9fac156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 >> 172.21.15.45:0/1710313275 conn(0x7f9fac0ced60 msgr2=0x7f9fac0ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 shutdown_connections 2023-11-28T20:51:42.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.322+0000 7f9fb3423640 1 -- 172.21.15.45:0/1710313275 wait complete. 2023-11-28T20:51:42.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T20:51:42.514 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:42.511+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:42.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.572+0000 7fb369abb640 1 Processor -- start 2023-11-28T20:51:42.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.573+0000 7fb369abb640 1 -- start start 2023-11-28T20:51:42.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.574+0000 7fb369abb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb36415e840 0x7fb36406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.574+0000 7fb369abb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb36406ac10 0x7fb3640717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.574+0000 7fb369abb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36415ecd0 con 0x7fb36415f720 2023-11-28T20:51:42.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.574+0000 7fb369abb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb364071ce0 con 0x7fb36406ac10 2023-11-28T20:51:42.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.574+0000 7fb369abb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36415f290 con 0x7fb36415e840 2023-11-28T20:51:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.574+0000 7fb363fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb36406ac10 0x7fb3640717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.574+0000 7fb363fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb36406ac10 0x7fb3640717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58582/0 (socket says 172.21.15.45:58582) 2023-11-28T20:51:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.574+0000 7fb363fff640 1 -- 172.21.15.45:0/2494791973 learned_addr learned my addr 172.21.15.45:0/2494791973 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.575+0000 7fb3637fe640 1 --2- 172.21.15.45:0/2494791973 >> v2:172.21.15.190:3300/0 conn(0x7fb36415e840 0x7fb36406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.575+0000 7fb363fff640 1 -- 172.21.15.45:0/2494791973 >> v2:172.21.15.190:3300/0 conn(0x7fb36415e840 msgr2=0x7fb36406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.575+0000 7fb363fff640 1 --2- 172.21.15.45:0/2494791973 >> v2:172.21.15.190:3300/0 conn(0x7fb36415e840 0x7fb36406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.575+0000 7fb363fff640 1 -- 172.21.15.45:0/2494791973 >> v1:172.21.15.190:6789/0 conn(0x7fb36415f720 legacy=0x7fb36406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:42.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.575+0000 7fb363fff640 1 -- 172.21.15.45:0/2494791973 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36415fb60 con 0x7fb36406ac10 2023-11-28T20:51:42.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.576+0000 7fb363fff640 1 --2- 172.21.15.45:0/2494791973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 0x7fb3640717a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fb3640cb5c0 tx=0x7fb35c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960b17d58c6989aa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:42.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.576+0000 7fb362ffd640 1 -- 172.21.15.45:0/2494791973 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c02ca80 con 0x7fb36406ac10 2023-11-28T20:51:42.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.576+0000 7fb362ffd640 1 -- 172.21.15.45:0/2494791973 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c02cbe0 con 0x7fb36406ac10 2023-11-28T20:51:42.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.576+0000 7fb362ffd640 1 -- 172.21.15.45:0/2494791973 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c0343f0 con 0x7fb36406ac10 2023-11-28T20:51:42.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.577+0000 7fb369abb640 1 -- 172.21.15.45:0/2494791973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 msgr2=0x7fb3640717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.577+0000 7fb369abb640 1 --2- 172.21.15.45:0/2494791973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 0x7fb3640717a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fb3640cb5c0 tx=0x7fb35c02bee0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.577+0000 7fb369abb640 1 -- 172.21.15.45:0/2494791973 shutdown_connections 2023-11-28T20:51:42.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.577+0000 7fb369abb640 1 --2- 172.21.15.45:0/2494791973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 0x7fb3640717a0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.577+0000 7fb369abb640 1 --2- 172.21.15.45:0/2494791973 >> v2:172.21.15.190:3300/0 conn(0x7fb36415e840 0x7fb36406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.577+0000 7fb369abb640 1 -- 172.21.15.45:0/2494791973 >> 172.21.15.45:0/2494791973 conn(0x7fb3640ced60 msgr2=0x7fb3640cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:42.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.577+0000 7fb369abb640 1 -- 172.21.15.45:0/2494791973 shutdown_connections 2023-11-28T20:51:42.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.577+0000 7fb369abb640 1 -- 172.21.15.45:0/2494791973 wait complete. 2023-11-28T20:51:42.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.579+0000 7fb369abb640 1 Processor -- start 2023-11-28T20:51:42.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.579+0000 7fb369abb640 1 -- start start 2023-11-28T20:51:42.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.580+0000 7fb369abb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 0x7fb36409c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.580+0000 7fb369abb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36415e840 0x7fb3640c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.580+0000 7fb369abb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36415f720 0x7fb3640b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.580+0000 7fb369abb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36409c8b0 con 0x7fb36406ac10 2023-11-28T20:51:42.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.580+0000 7fb369abb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3640c7990 con 0x7fb36415e840 2023-11-28T20:51:42.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.580+0000 7fb369abb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3640c7c30 con 0x7fb36415f720 2023-11-28T20:51:42.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.580+0000 7fb3637fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 0x7fb36409c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb3637fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 0x7fb36409c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58594/0 (socket says 172.21.15.45:58594) 2023-11-28T20:51:42.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb3637fe640 1 -- 172.21.15.45:0/2746653883 learned_addr learned my addr 172.21.15.45:0/2746653883 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:42.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb363fff640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36415f720 0x7fb3640b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb35bfff640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36415e840 0x7fb3640c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb3637fe640 1 -- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36415f720 msgr2=0x7fb3640b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb3637fe640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36415f720 0x7fb3640b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb3637fe640 1 -- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36415e840 msgr2=0x7fb3640c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb3637fe640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36415e840 0x7fb3640c86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb3637fe640 1 -- 172.21.15.45:0/2746653883 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c009b30 con 0x7fb36406ac10 2023-11-28T20:51:42.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb35bfff640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36415e840 0x7fb3640c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:42.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb3637fe640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 0x7fb36409c370 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fb35400d800 tx=0x7fb35400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:42.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.581+0000 7fb360ff9640 1 -- 172.21.15.45:0/2746653883 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35400b6c0 con 0x7fb36406ac10 2023-11-28T20:51:42.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.582+0000 7fb360ff9640 1 -- 172.21.15.45:0/2746653883 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35400b820 con 0x7fb36406ac10 2023-11-28T20:51:42.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.582+0000 7fb360ff9640 1 -- 172.21.15.45:0/2746653883 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35400bbb0 con 0x7fb36406ac10 2023-11-28T20:51:42.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.582+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3640c64f0 con 0x7fb36406ac10 2023-11-28T20:51:42.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.582+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3640c7ed0 con 0x7fb36406ac10 2023-11-28T20:51:42.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.583+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3640733d0 con 0x7fb36406ac10 2023-11-28T20:51:42.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.584+0000 7fb360ff9640 1 -- 172.21.15.45:0/2746653883 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fb35400bd10 con 0x7fb36406ac10 2023-11-28T20:51:42.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.585+0000 7fb360ff9640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb34406af00 0x7fb34406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:42.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.585+0000 7fb360ff9640 1 -- 172.21.15.45:0/2746653883 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7fb354089bf0 con 0x7fb36406ac10 2023-11-28T20:51:42.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.585+0000 7fb35bfff640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb34406af00 0x7fb34406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:42.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.587+0000 7fb35bfff640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb34406af00 0x7fb34406d3c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb3640ad2a0 tx=0x7fb34c008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:42.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.590+0000 7fb360ff9640 1 -- 172.21.15.45:0/2746653883 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb354054c20 con 0x7fb36406ac10 2023-11-28T20:51:42.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.788+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fb3640ce2b0 con 0x7fb36406ac10 2023-11-28T20:51:42.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.794+0000 7fb360ff9640 1 -- 172.21.15.45:0/2746653883 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v33) v1 ==== 115+0+0 (secure 0 0 0) 0x7fb354058240 con 0x7fb36406ac10 2023-11-28T20:51:42.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb34406af00 msgr2=0x7fb34406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb34406af00 0x7fb34406d3c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb3640ad2a0 tx=0x7fb34c008040 comp rx=0 tx=0).stop 2023-11-28T20:51:42.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 msgr2=0x7fb36409c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:42.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 0x7fb36409c370 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fb35400d800 tx=0x7fb35400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 shutdown_connections 2023-11-28T20:51:42.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb34406af00 0x7fb34406d3c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb36415f720 0x7fb3640b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36415e840 0x7fb3640c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 --2- 172.21.15.45:0/2746653883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb36406ac10 0x7fb36409c370 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:42.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 >> 172.21.15.45:0/2746653883 conn(0x7fb3640ced60 msgr2=0x7fb3640700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:42.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 shutdown_connections 2023-11-28T20:51:42.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:42.797+0000 7fb369abb640 1 -- 172.21.15.45:0/2746653883 wait complete. 2023-11-28T20:51:42.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-11-28T20:51:43.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.045+0000 7fee5f375640 1 Processor -- start 2023-11-28T20:51:43.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.046+0000 7fee5f375640 1 -- start start 2023-11-28T20:51:43.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.047+0000 7fee5f375640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee580ae540 0x7fee580ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.047+0000 7fee5f375640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5815c3c0 con 0x7fee580ce630 2023-11-28T20:51:43.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.047+0000 7fee5f375640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee58155b60 con 0x7fee58155780 2023-11-28T20:51:43.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.047+0000 7fee5f375640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee580ceac0 con 0x7fee580ae540 2023-11-28T20:51:43.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.048+0000 7fee5eb74640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fee580ce630 0x7fee580b20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54992/0 (socket says 172.21.15.45:54992) 2023-11-28T20:51:43.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.048+0000 7fee5eb74640 1 -- 172.21.15.45:0/2054418718 learned_addr learned my addr 172.21.15.45:0/2054418718 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:43.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.048+0000 7fee5db72640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee580ae540 0x7fee580ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.048+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867088550 0 0) 0x7fee5815c3c0 con 0x7fee580ce630 2023-11-28T20:51:43.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.048+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee44003520 con 0x7fee580ce630 2023-11-28T20:51:43.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1656917581 0 0) 0x7fee44003520 con 0x7fee580ce630 2023-11-28T20:51:43.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee580ae540 msgr2=0x7fee580ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:43.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee580ae540 0x7fee580ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 >> v1:172.21.15.190:6789/0 conn(0x7fee58155780 legacy=0x7fee580abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5815ba90 con 0x7fee580ce630 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fee500019f0 con 0x7fee580ce630 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 >> v1:172.21.15.45:6789/0 conn(0x7fee580ce630 legacy=0x7fee580b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee440039a0 0x7fee44003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee580ae540 0x7fee44005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee580ceac0 con 0x7fee440047c0 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4400ba60 con 0x7fee440039a0 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4400bc20 con 0x7fee580ae540 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5db72640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.049+0000 7fee5e373640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee440039a0 0x7fee44003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.050+0000 7fee5eb74640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee580ae540 0x7fee44005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.050+0000 7fee5db72640 1 -- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee580ae540 msgr2=0x7fee44005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:43.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.050+0000 7fee5db72640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee580ae540 0x7fee44005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.050+0000 7fee5db72640 1 -- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee440039a0 msgr2=0x7fee44003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:43.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.050+0000 7fee5db72640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee440039a0 0x7fee44003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.050+0000 7fee5db72640 1 -- 172.21.15.45:0/2054418718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44003520 con 0x7fee440047c0 2023-11-28T20:51:43.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.051+0000 7fee5db72640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee44004c40 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fee48009a90 tx=0x7fee440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb165d2b0626bc86 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:43.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.051+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4802c680 con 0x7fee440047c0 2023-11-28T20:51:43.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.051+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4802c7e0 con 0x7fee440047c0 2023-11-28T20:51:43.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.051+0000 7fee5d371640 1 -- 172.21.15.45:0/2054418718 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4802cb70 con 0x7fee440047c0 2023-11-28T20:51:43.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.052+0000 7fee5f375640 1 -- 172.21.15.45:0/2054418718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 msgr2=0x7fee44004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.052+0000 7fee5f375640 1 --2- 172.21.15.45:0/2054418718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee44004c40 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fee48009a90 tx=0x7fee440057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.052+0000 7fee5eb74640 1 -- 172.21.15.45:0/2054418718 reap_dead start 2023-11-28T20:51:43.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.052+0000 7fee5f375640 1 -- 172.21.15.45:0/2054418718 shutdown_connections 2023-11-28T20:51:43.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.052+0000 7fee5f375640 1 -- 172.21.15.45:0/2054418718 >> 172.21.15.45:0/2054418718 conn(0x7fee5806ab30 msgr2=0x7fee580b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:43.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.052+0000 7fee5f375640 1 -- 172.21.15.45:0/2054418718 shutdown_connections 2023-11-28T20:51:43.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.052+0000 7fee5f375640 1 -- 172.21.15.45:0/2054418718 wait complete. 2023-11-28T20:51:43.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.054+0000 7fee5f375640 1 Processor -- start 2023-11-28T20:51:43.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.054+0000 7fee5f375640 1 -- start start 2023-11-28T20:51:43.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.055+0000 7fee5f375640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee58143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.055+0000 7fee5f375640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee5814fc20 0x7fee58153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.055+0000 7fee5f375640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee58154030 0x7fee58150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.055+0000 7fee5f375640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee581544b0 con 0x7fee440047c0 2023-11-28T20:51:43.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.055+0000 7fee5f375640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee58143cd0 con 0x7fee5814fc20 2023-11-28T20:51:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.055+0000 7fee5f375640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5814f790 con 0x7fee58154030 2023-11-28T20:51:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.055+0000 7fee5e373640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee58143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.055+0000 7fee5e373640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee58143790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58614/0 (socket says 172.21.15.45:58614) 2023-11-28T20:51:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.055+0000 7fee5e373640 1 -- 172.21.15.45:0/597877954 learned_addr learned my addr 172.21.15.45:0/597877954 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.056+0000 7fee5db72640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee5814fc20 0x7fee58153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.056+0000 7fee5eb74640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee58154030 0x7fee58150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.056+0000 7fee5e373640 1 -- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee58154030 msgr2=0x7fee58150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.056+0000 7fee5e373640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee58154030 0x7fee58150770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.056+0000 7fee5e373640 1 -- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee5814fc20 msgr2=0x7fee58153af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.056+0000 7fee5e373640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee5814fc20 0x7fee58153af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.056+0000 7fee5e373640 1 -- 172.21.15.45:0/597877954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5814eb80 con 0x7fee440047c0 2023-11-28T20:51:43.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.056+0000 7fee5db72640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee5814fc20 0x7fee58153af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:43.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.056+0000 7fee5e373640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee58143790 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fee5400e550 tx=0x7fee5400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.057+0000 7fee4effd640 1 -- 172.21.15.45:0/597877954 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee540052e0 con 0x7fee440047c0 2023-11-28T20:51:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.057+0000 7fee4effd640 1 -- 172.21.15.45:0/597877954 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee54005440 con 0x7fee440047c0 2023-11-28T20:51:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.057+0000 7fee4effd640 1 -- 172.21.15.45:0/597877954 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee54003810 con 0x7fee440047c0 2023-11-28T20:51:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.057+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5814ee60 con 0x7fee440047c0 2023-11-28T20:51:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.057+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5814f420 con 0x7fee440047c0 2023-11-28T20:51:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.058+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5804e920 con 0x7fee440047c0 2023-11-28T20:51:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.059+0000 7fee4effd640 1 -- 172.21.15.45:0/597877954 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fee54003970 con 0x7fee440047c0 2023-11-28T20:51:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.060+0000 7fee4effd640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee3c06aed0 0x7fee3c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.060+0000 7fee4effd640 1 -- 172.21.15.45:0/597877954 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7fee540895d0 con 0x7fee440047c0 2023-11-28T20:51:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.061+0000 7fee5db72640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee3c06aed0 0x7fee3c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.062+0000 7fee5db72640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee3c06aed0 0x7fee3c06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fee48036000 tx=0x7fee48030000 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:43.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.064+0000 7fee4effd640 1 -- 172.21.15.45:0/597877954 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee54054470 con 0x7fee440047c0 2023-11-28T20:51:43.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.343+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fee5815e650 con 0x7fee440047c0 2023-11-28T20:51:43.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.372+0000 7fee4effd640 1 -- 172.21.15.45:0/597877954 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v678) v1 ==== 166+0+0 (secure 0 0 0) 0x7fee54057a90 con 0x7fee440047c0 2023-11-28T20:51:43.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee3c06aed0 msgr2=0x7fee3c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee3c06aed0 0x7fee3c06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fee48036000 tx=0x7fee48030000 comp rx=0 tx=0).stop 2023-11-28T20:51:43.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 msgr2=0x7fee58143790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee58143790 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fee5400e550 tx=0x7fee5400ea10 comp rx=0 tx=0).stop 2023-11-28T20:51:43.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 shutdown_connections 2023-11-28T20:51:43.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fee3c06aed0 0x7fee3c06d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee58154030 0x7fee58150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee5814fc20 0x7fee58153af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 --2- 172.21.15.45:0/597877954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee440047c0 0x7fee58143790 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 >> 172.21.15.45:0/597877954 conn(0x7fee5806ab30 msgr2=0x7fee5815d640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:43.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 shutdown_connections 2023-11-28T20:51:43.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.376+0000 7fee5f375640 1 -- 172.21.15.45:0/597877954 wait complete. 2023-11-28T20:51:43.399 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-11-28T20:51:43.399 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-11-28T20:51:43.399 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T20:51:43.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.621+0000 7fcafd527640 1 Processor -- start 2023-11-28T20:51:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.621+0000 7fcafd527640 1 -- start start 2023-11-28T20:51:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.623+0000 7fcafd527640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf8155780 0x7fcaf80abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.623+0000 7fcafd527640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcaf80ce800 0x7fcaf80b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.623+0000 7fcafd527640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf815c3c0 con 0x7fcaf80ae540 2023-11-28T20:51:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.623+0000 7fcafd527640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf8155b60 con 0x7fcaf80ce800 2023-11-28T20:51:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.623+0000 7fcafd527640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf815ba90 con 0x7fcaf8155780 2023-11-28T20:51:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.623+0000 7fcafcd26640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcaf80ce800 0x7fcaf80b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.623+0000 7fcafcd26640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcaf80ce800 0x7fcaf80b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58624/0 (socket says 172.21.15.45:58624) 2023-11-28T20:51:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.623+0000 7fcafcd26640 1 -- 172.21.15.45:0/3601041422 learned_addr learned my addr 172.21.15.45:0/3601041422 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.624+0000 7fcaf7fff640 1 --2- 172.21.15.45:0/3601041422 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf8155780 0x7fcaf80abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.624+0000 7fcafcd26640 1 -- 172.21.15.45:0/3601041422 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf8155780 msgr2=0x7fcaf80abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.624+0000 7fcafcd26640 1 --2- 172.21.15.45:0/3601041422 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf8155780 0x7fcaf80abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.624+0000 7fcafcd26640 1 -- 172.21.15.45:0/3601041422 >> v1:172.21.15.190:6789/0 conn(0x7fcaf80ae540 legacy=0x7fcaf80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.624+0000 7fcafcd26640 1 -- 172.21.15.45:0/3601041422 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf815bd70 con 0x7fcaf80ce800 2023-11-28T20:51:43.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.625+0000 7fcafcd26640 1 --2- 172.21.15.45:0/3601041422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf80ce800 0x7fcaf80b20a0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0009e50 tx=0x7fcaf002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8024d35fe2d14717 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:43.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.625+0000 7fcaf6ffd640 1 -- 172.21.15.45:0/3601041422 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf002ca70 con 0x7fcaf80ce800 2023-11-28T20:51:43.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.625+0000 7fcaf6ffd640 1 -- 172.21.15.45:0/3601041422 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf002cbd0 con 0x7fcaf80ce800 2023-11-28T20:51:43.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.625+0000 7fcaf6ffd640 1 -- 172.21.15.45:0/3601041422 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf00343f0 con 0x7fcaf80ce800 2023-11-28T20:51:43.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.626+0000 7fcafd527640 1 -- 172.21.15.45:0/3601041422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf80ce800 msgr2=0x7fcaf80b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.626+0000 7fcafd527640 1 --2- 172.21.15.45:0/3601041422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf80ce800 0x7fcaf80b20a0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0009e50 tx=0x7fcaf002bed0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.626+0000 7fcafd527640 1 -- 172.21.15.45:0/3601041422 shutdown_connections 2023-11-28T20:51:43.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.626+0000 7fcafd527640 1 --2- 172.21.15.45:0/3601041422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf80ce800 0x7fcaf80b20a0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.626+0000 7fcafd527640 1 --2- 172.21.15.45:0/3601041422 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf8155780 0x7fcaf80abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.626+0000 7fcafd527640 1 -- 172.21.15.45:0/3601041422 >> 172.21.15.45:0/3601041422 conn(0x7fcaf806ab30 msgr2=0x7fcaf80b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:43.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.626+0000 7fcafd527640 1 -- 172.21.15.45:0/3601041422 shutdown_connections 2023-11-28T20:51:43.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.626+0000 7fcafd527640 1 -- 172.21.15.45:0/3601041422 wait complete. 2023-11-28T20:51:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.629+0000 7fcafd527640 1 Processor -- start 2023-11-28T20:51:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.629+0000 7fcafd527640 1 -- start start 2023-11-28T20:51:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.630+0000 7fcafd527640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf80ae540 0x7fcaf8153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.630+0000 7fcafd527640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf80ce800 0x7fcaf81e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.630+0000 7fcafd527640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8155780 0x7fcaf81e6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.630+0000 7fcafd527640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf81e6870 con 0x7fcaf8155780 2023-11-28T20:51:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.630+0000 7fcafd527640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf8159de0 con 0x7fcaf80ce800 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.630+0000 7fcafd527640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf815ac30 con 0x7fcaf80ae540 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.630+0000 7fcafcd26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8155780 0x7fcaf81e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.630+0000 7fcafcd26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8155780 0x7fcaf81e6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58628/0 (socket says 172.21.15.45:58628) 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.630+0000 7fcafcd26640 1 -- 172.21.15.45:0/1757254647 learned_addr learned my addr 172.21.15.45:0/1757254647 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.631+0000 7fcaf77fe640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf80ce800 0x7fcaf81e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.631+0000 7fcaf7fff640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf80ae540 0x7fcaf8153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.631+0000 7fcafcd26640 1 -- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf80ae540 msgr2=0x7fcaf8153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.631+0000 7fcafcd26640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf80ae540 0x7fcaf8153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.631+0000 7fcafcd26640 1 -- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf80ce800 msgr2=0x7fcaf81e6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.631+0000 7fcafcd26640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf80ce800 0x7fcaf81e6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.631+0000 7fcafcd26640 1 -- 172.21.15.45:0/1757254647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf0009b30 con 0x7fcaf8155780 2023-11-28T20:51:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.631+0000 7fcafcd26640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8155780 0x7fcaf81e6330 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0009e50 tx=0x7fcaf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.632+0000 7fcaf4ff9640 1 -- 172.21.15.45:0/1757254647 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf003e450 con 0x7fcaf8155780 2023-11-28T20:51:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.632+0000 7fcaf4ff9640 1 -- 172.21.15.45:0/1757254647 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf003e5b0 con 0x7fcaf8155780 2023-11-28T20:51:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.632+0000 7fcaf4ff9640 1 -- 172.21.15.45:0/1757254647 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf003e960 con 0x7fcaf8155780 2023-11-28T20:51:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.632+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf815af10 con 0x7fcaf8155780 2023-11-28T20:51:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.632+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf815b1f0 con 0x7fcaf8155780 2023-11-28T20:51:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.633+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf81e6aa0 con 0x7fcaf8155780 2023-11-28T20:51:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.634+0000 7fcaf4ff9640 1 -- 172.21.15.45:0/1757254647 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fcaf003eac0 con 0x7fcaf8155780 2023-11-28T20:51:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.635+0000 7fcaf4ff9640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fcae406aed0 0x7fcae406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.635+0000 7fcaf4ff9640 1 -- 172.21.15.45:0/1757254647 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7fcaf00ac600 con 0x7fcaf8155780 2023-11-28T20:51:43.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.636+0000 7fcaf7fff640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fcae406aed0 0x7fcae406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:43.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.637+0000 7fcaf7fff640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fcae406aed0 0x7fcae406d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006fd0 tx=0x7fcae8008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:43.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.639+0000 7fcaf4ff9640 1 -- 172.21.15.45:0/1757254647 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcaf00774a0 con 0x7fcaf8155780 2023-11-28T20:51:43.645 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:43.642+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:43.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.843+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcaf804e920 con 0x7fcaf8155780 2023-11-28T20:51:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.845+0000 7fcaf4ff9640 1 -- 172.21.15.45:0/1757254647 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+14266 (secure 0 0 0) 0x7fcaf007aac0 con 0x7fcaf8155780 2023-11-28T20:51:43.846 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:43.846 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":107,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:41.363023+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6834/862206408":"2023-11-29T20:51:37.959623+0000","172.21.15.190:0/1724154865":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/412684916":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/514984079":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/2474029917":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/3103043064":"2023-11-29T20:51:27.664495+0000","172.21.15.190:6833/1373961985":"2023-11-29T20:51:33.864326+0000","172.21.15.190:6832/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.45:6835/862206408":"2023-11-29T20:51:37.959623+0000","172.21.15.190:6832/1373961985":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/3436414960":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3227357091":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/3114762982":"2023-11-29T20:51:33.864326+0000","172.21.15.190:0/1429551158":"2023-11-29T20:51:33.864326+0000","172.21.15.190:6833/970930168":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/4025400682":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2627631957":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/2067877259":"2023-11-29T20:51:27.664495+0000","172.21.15.190:0/1852415784":"2023-11-28T21:51:13.642235+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fcae406aed0 msgr2=0x7fcae406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fcae406aed0 0x7fcae406d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006fd0 tx=0x7fcae8008040 comp rx=0 tx=0).stop 2023-11-28T20:51:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8155780 msgr2=0x7fcaf81e6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8155780 0x7fcaf81e6330 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0009e50 tx=0x7fcaf0004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 shutdown_connections 2023-11-28T20:51:43.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fcae406aed0 0x7fcae406d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf8155780 0x7fcaf81e6330 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf80ce800 0x7fcaf81e6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 --2- 172.21.15.45:0/1757254647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf80ae540 0x7fcaf8153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:43.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 >> 172.21.15.45:0/1757254647 conn(0x7fcaf806ab30 msgr2=0x7fcaf806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:43.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 shutdown_connections 2023-11-28T20:51:43.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:43.848+0000 7fcafd527640 1 -- 172.21.15.45:0/1757254647 wait complete. 2023-11-28T20:51:43.871 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:51:43.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:51:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.096+0000 7f4d7eead640 1 Processor -- start 2023-11-28T20:51:44.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.096+0000 7f4d7eead640 1 -- start start 2023-11-28T20:51:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.098+0000 7f4d7eead640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4d78159720 0x7f4d780b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.098+0000 7f4d7eead640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d78071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.098+0000 7f4d7eead640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d780cc120 con 0x7f4d7806dfa0 2023-11-28T20:51:44.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.098+0000 7f4d7eead640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d78072500 con 0x7f4d78159720 2023-11-28T20:51:44.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.098+0000 7f4d7eead640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7806e430 con 0x7f4d78071b90 2023-11-28T20:51:44.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.099+0000 7f4d7deab640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4d78159720 0x7f4d780b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.099+0000 7f4d7e6ac640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d78071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.099+0000 7f4d7e6ac640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d78071fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:48970/0 (socket says 172.21.15.45:48970) 2023-11-28T20:51:44.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.099+0000 7f4d7deab640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4d78159720 0x7f4d780b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:40628/0 (socket says 172.21.15.45:40628) 2023-11-28T20:51:44.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.099+0000 7f4d7e6ac640 1 -- 172.21.15.45:0/1416280118 learned_addr learned my addr 172.21.15.45:0/1416280118 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:44.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.100+0000 7f4d7e6ac640 1 -- 172.21.15.45:0/1416280118 >> v2:172.21.15.190:3300/0 conn(0x7f4d78159720 msgr2=0x7f4d780b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:44.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.100+0000 7f4d7e6ac640 1 --2- 172.21.15.45:0/1416280118 >> v2:172.21.15.190:3300/0 conn(0x7f4d78159720 0x7f4d780b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.100+0000 7f4d7e6ac640 1 -- 172.21.15.45:0/1416280118 >> v1:172.21.15.190:6789/0 conn(0x7f4d7806dfa0 legacy=0x7f4d780701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:44.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.100+0000 7f4d7e6ac640 1 -- 172.21.15.45:0/1416280118 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780b2420 con 0x7f4d78071b90 2023-11-28T20:51:44.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.102+0000 7f4d7e6ac640 1 --2- 172.21.15.45:0/1416280118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d78071fc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000d150 tx=0x7f4d7002e160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f3790e57536bb1b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:44.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.102+0000 7f4d7cea9640 1 -- 172.21.15.45:0/1416280118 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70038400 con 0x7f4d78071b90 2023-11-28T20:51:44.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.102+0000 7f4d7cea9640 1 -- 172.21.15.45:0/1416280118 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d70032070 con 0x7f4d78071b90 2023-11-28T20:51:44.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.102+0000 7f4d7cea9640 1 -- 172.21.15.45:0/1416280118 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70038780 con 0x7f4d78071b90 2023-11-28T20:51:44.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.103+0000 7f4d7eead640 1 -- 172.21.15.45:0/1416280118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 msgr2=0x7f4d78071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:44.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.103+0000 7f4d7eead640 1 --2- 172.21.15.45:0/1416280118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d78071fc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000d150 tx=0x7f4d7002e160 comp rx=0 tx=0).stop 2023-11-28T20:51:44.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.103+0000 7f4d7eead640 1 -- 172.21.15.45:0/1416280118 shutdown_connections 2023-11-28T20:51:44.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.103+0000 7f4d7eead640 1 --2- 172.21.15.45:0/1416280118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d78071fc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.103+0000 7f4d7eead640 1 --2- 172.21.15.45:0/1416280118 >> v2:172.21.15.190:3300/0 conn(0x7f4d78159720 0x7f4d780b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.103+0000 7f4d7eead640 1 -- 172.21.15.45:0/1416280118 >> 172.21.15.45:0/1416280118 conn(0x7f4d780b1050 msgr2=0x7f4d780cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:44.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.103+0000 7f4d7eead640 1 -- 172.21.15.45:0/1416280118 shutdown_connections 2023-11-28T20:51:44.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.103+0000 7f4d7eead640 1 -- 172.21.15.45:0/1416280118 wait complete. 2023-11-28T20:51:44.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.105+0000 7f4d7eead640 1 Processor -- start 2023-11-28T20:51:44.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.105+0000 7f4d7eead640 1 -- start start 2023-11-28T20:51:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.106+0000 7f4d7eead640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d7806dfa0 0x7f4d780af140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.106+0000 7f4d7eead640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d7809af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.106+0000 7f4d7eead640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d78159720 0x7f4d7809b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.106+0000 7f4d7eead640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7809ab50 con 0x7f4d78159720 2023-11-28T20:51:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.106+0000 7f4d7eead640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d780c8240 con 0x7f4d7806dfa0 2023-11-28T20:51:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.106+0000 7f4d7eead640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d78099ed0 con 0x7f4d78071b90 2023-11-28T20:51:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7e6ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d78159720 0x7f4d7809b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7e6ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d78159720 0x7f4d7809b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35756/0 (socket says 172.21.15.45:35756) 2023-11-28T20:51:44.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7e6ac640 1 -- 172.21.15.45:0/2269762420 learned_addr learned my addr 172.21.15.45:0/2269762420 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:44.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7d6aa640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d7809af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7deab640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d7806dfa0 0x7f4d780af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7e6ac640 1 -- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 msgr2=0x7f4d7809af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7e6ac640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d7809af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7e6ac640 1 -- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d7806dfa0 msgr2=0x7f4d780af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7e6ac640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d7806dfa0 0x7f4d780af140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.107+0000 7f4d7e6ac640 1 -- 172.21.15.45:0/2269762420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7000bd40 con 0x7f4d78159720 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.108+0000 7f4d7e6ac640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d78159720 0x7f4d7809b4b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f4d7003a040 tx=0x7f4d70004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.108+0000 7f4d6e7fc640 1 -- 172.21.15.45:0/2269762420 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7000b720 con 0x7f4d78159720 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.108+0000 7f4d6e7fc640 1 -- 172.21.15.45:0/2269762420 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d70042430 con 0x7f4d78159720 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.108+0000 7f4d6e7fc640 1 -- 172.21.15.45:0/2269762420 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d700427e0 con 0x7f4d78159720 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.108+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7809a1b0 con 0x7f4d78159720 2023-11-28T20:51:44.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.108+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7809a770 con 0x7f4d78159720 2023-11-28T20:51:44.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.110+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d7804e920 con 0x7f4d78159720 2023-11-28T20:51:44.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.110+0000 7f4d6e7fc640 1 -- 172.21.15.45:0/2269762420 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f4d70042940 con 0x7f4d78159720 2023-11-28T20:51:44.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.112+0000 7f4d6e7fc640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4d5c06aed0 0x7f4d5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.112+0000 7f4d6e7fc640 1 -- 172.21.15.45:0/2269762420 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7f4d700ac310 con 0x7f4d78159720 2023-11-28T20:51:44.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.113+0000 7f4d7deab640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4d5c06aed0 0x7f4d5c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.114+0000 7f4d7deab640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4d5c06aed0 0x7f4d5c06d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009a30 tx=0x7f4d74008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:44.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.116+0000 7f4d6e7fc640 1 -- 172.21.15.45:0/2269762420 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d700771b0 con 0x7f4d78159720 2023-11-28T20:51:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.324+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d780ce310 con 0x7f4d78159720 2023-11-28T20:51:44.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.324+0000 7f4d6e7fc640 1 -- 172.21.15.45:0/2269762420 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4d7007a7d0 con 0x7f4d78159720 2023-11-28T20:51:44.326 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:51:44.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4d5c06aed0 msgr2=0x7f4d5c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4d5c06aed0 0x7f4d5c06d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009a30 tx=0x7f4d74008040 comp rx=0 tx=0).stop 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d78159720 msgr2=0x7f4d7809b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d78159720 0x7f4d7809b4b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f4d7003a040 tx=0x7f4d70004ab0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 shutdown_connections 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f4d5c06aed0 0x7f4d5c06d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d78159720 0x7f4d7809b4b0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d78071b90 0x7f4d7809af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 --2- 172.21.15.45:0/2269762420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d7806dfa0 0x7f4d780af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.327+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 >> 172.21.15.45:0/2269762420 conn(0x7f4d780b1050 msgr2=0x7f4d78151a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:44.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.328+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 shutdown_connections 2023-11-28T20:51:44.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.328+0000 7f4d7eead640 1 -- 172.21.15.45:0/2269762420 wait complete. 2023-11-28T20:51:44.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-11-28T20:51:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.588+0000 7f9581d27640 1 Processor -- start 2023-11-28T20:51:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.588+0000 7f9581d27640 1 -- start start 2023-11-28T20:51:44.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.589+0000 7f9581d27640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f957c155780 0x7f957c0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581d27640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f957c0ce800 0x7f957c0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581d27640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957c15c3c0 con 0x7f957c0ae540 2023-11-28T20:51:44.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581d27640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f957c155b60 con 0x7f957c0ce800 2023-11-28T20:51:44.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581d27640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f957c15ba90 con 0x7f957c155780 2023-11-28T20:51:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581526640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f957c0ce800 0x7f957c0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581526640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f957c0ce800 0x7f957c0b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35762/0 (socket says 172.21.15.45:35762) 2023-11-28T20:51:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581526640 1 -- 172.21.15.45:0/2656168120 learned_addr learned my addr 172.21.15.45:0/2656168120 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9580d25640 1 --2- 172.21.15.45:0/2656168120 >> v2:172.21.15.190:3300/0 conn(0x7f957c155780 0x7f957c0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581526640 1 -- 172.21.15.45:0/2656168120 >> v2:172.21.15.190:3300/0 conn(0x7f957c155780 msgr2=0x7f957c0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581526640 1 --2- 172.21.15.45:0/2656168120 >> v2:172.21.15.190:3300/0 conn(0x7f957c155780 0x7f957c0abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581526640 1 -- 172.21.15.45:0/2656168120 >> v1:172.21.15.190:6789/0 conn(0x7f957c0ae540 legacy=0x7f957c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:44.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9581526640 1 -- 172.21.15.45:0/2656168120 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c15bd70 con 0x7f957c0ce800 2023-11-28T20:51:44.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.590+0000 7f9580d25640 1 --2- 172.21.15.45:0/2656168120 >> v2:172.21.15.190:3300/0 conn(0x7f957c155780 0x7f957c0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:44.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.592+0000 7f9581526640 1 --2- 172.21.15.45:0/2656168120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ce800 0x7f957c0b20a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f957c06e6d0 tx=0x7f957402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748ffcfe60171ce0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.592+0000 7f957b7fe640 1 -- 172.21.15.45:0/2656168120 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957402ca80 con 0x7f957c0ce800 2023-11-28T20:51:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.592+0000 7f957b7fe640 1 -- 172.21.15.45:0/2656168120 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957402cbe0 con 0x7f957c0ce800 2023-11-28T20:51:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.592+0000 7f957b7fe640 1 -- 172.21.15.45:0/2656168120 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95740343f0 con 0x7f957c0ce800 2023-11-28T20:51:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.593+0000 7f9581d27640 1 -- 172.21.15.45:0/2656168120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ce800 msgr2=0x7f957c0b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.593+0000 7f9581d27640 1 --2- 172.21.15.45:0/2656168120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ce800 0x7f957c0b20a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f957c06e6d0 tx=0x7f957402bee0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.593+0000 7f9581d27640 1 -- 172.21.15.45:0/2656168120 shutdown_connections 2023-11-28T20:51:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.593+0000 7f9581d27640 1 --2- 172.21.15.45:0/2656168120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ce800 0x7f957c0b20a0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.593+0000 7f9581d27640 1 --2- 172.21.15.45:0/2656168120 >> v2:172.21.15.190:3300/0 conn(0x7f957c155780 0x7f957c0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.593+0000 7f9581d27640 1 -- 172.21.15.45:0/2656168120 >> 172.21.15.45:0/2656168120 conn(0x7f957c06ab30 msgr2=0x7f957c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.593+0000 7f9581d27640 1 -- 172.21.15.45:0/2656168120 shutdown_connections 2023-11-28T20:51:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.593+0000 7f9581d27640 1 -- 172.21.15.45:0/2656168120 wait complete. 2023-11-28T20:51:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.595+0000 7f9581d27640 1 Processor -- start 2023-11-28T20:51:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.595+0000 7f9581d27640 1 -- start start 2023-11-28T20:51:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.596+0000 7f9581d27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ae540 0x7f957c0c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.596+0000 7f9581d27640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f957c0ce800 0x7f957c0701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.596+0000 7f9581d27640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f957c155780 0x7f957c15a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.596+0000 7f9581d27640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c070700 con 0x7f957c0ae540 2023-11-28T20:51:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.596+0000 7f9581d27640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c0709d0 con 0x7f957c0ce800 2023-11-28T20:51:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.596+0000 7f9581d27640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957c070cb0 con 0x7f957c155780 2023-11-28T20:51:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.596+0000 7f9580d25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ae540 0x7f957c0c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.596+0000 7f9580d25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ae540 0x7f957c0c47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35770/0 (socket says 172.21.15.45:35770) 2023-11-28T20:51:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.596+0000 7f9580d25640 1 -- 172.21.15.45:0/3828310187 learned_addr learned my addr 172.21.15.45:0/3828310187 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f9580d25640 1 -- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f957c155780 msgr2=0x7f957c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f9580d25640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f957c155780 0x7f957c15a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f9580d25640 1 -- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f957c0ce800 msgr2=0x7f957c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f9580d25640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f957c0ce800 0x7f957c0701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f9580d25640 1 -- 172.21.15.45:0/3828310187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9574009b30 con 0x7f957c0ae540 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f9580d25640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ae540 0x7f957c0c47a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f956c00c7f0 tx=0x7f956c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f95797fa640 1 -- 172.21.15.45:0/3828310187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c007820 con 0x7f957c0ae540 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f95797fa640 1 -- 172.21.15.45:0/3828310187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c007980 con 0x7f957c0ae540 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f95797fa640 1 -- 172.21.15.45:0/3828310187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c007cf0 con 0x7f957c0ae540 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957c070f90 con 0x7f957c0ae540 2023-11-28T20:51:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.597+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957c071550 con 0x7f957c0ae540 2023-11-28T20:51:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.599+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957c04e920 con 0x7f957c0ae540 2023-11-28T20:51:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.600+0000 7f95797fa640 1 -- 172.21.15.45:0/3828310187 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f956c0113f0 con 0x7f957c0ae540 2023-11-28T20:51:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.601+0000 7f95797fa640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f956806aed0 0x7f956806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.601+0000 7f95797fa640 1 -- 172.21.15.45:0/3828310187 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7f956c089c00 con 0x7f957c0ae540 2023-11-28T20:51:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.601+0000 7f957bfff640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f956806aed0 0x7f956806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:44.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.603+0000 7f957bfff640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f956806aed0 0x7f956806d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f957c06d760 tx=0x7f9570007990 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.605+0000 7f95797fa640 1 -- 172.21.15.45:0/3828310187 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f956c08f020 con 0x7f957c0ae540 2023-11-28T20:51:44.760 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:44.757+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:44.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:44.889+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f957c15d5b0 con 0x7f957c0ae540 2023-11-28T20:51:45.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.385+0000 7f95797fa640 1 -- 172.21.15.45:0/3828310187 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v680) v1 ==== 186+0+0 (secure 0 0 0) 0x7f956c01f020 con 0x7f957c0ae540 2023-11-28T20:51:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.388+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f956806aed0 msgr2=0x7f956806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.388+0000 7f9581d27640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f956806aed0 0x7f956806d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f957c06d760 tx=0x7f9570007990 comp rx=0 tx=0).stop 2023-11-28T20:51:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.388+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ae540 msgr2=0x7f957c0c47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.388+0000 7f9581d27640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ae540 0x7f957c0c47a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f956c00c7f0 tx=0x7f956c00ccb0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.388+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 shutdown_connections 2023-11-28T20:51:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.388+0000 7f9581d27640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f956806aed0 0x7f956806d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.388+0000 7f9581d27640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f957c155780 0x7f957c15a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.389+0000 7f9581d27640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f957c0ce800 0x7f957c0701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.389+0000 7f9581d27640 1 --2- 172.21.15.45:0/3828310187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0ae540 0x7f957c0c47a0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.389+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 >> 172.21.15.45:0/3828310187 conn(0x7f957c06ab30 msgr2=0x7f957c15c870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.389+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 shutdown_connections 2023-11-28T20:51:45.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.389+0000 7f9581d27640 1 -- 172.21.15.45:0/3828310187 wait complete. 2023-11-28T20:51:45.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T20:51:45.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.638+0000 7f18b3741640 1 Processor -- start 2023-11-28T20:51:45.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.639+0000 7f18b3741640 1 -- start start 2023-11-28T20:51:45.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.640+0000 7f18b3741640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac06ac10 0x7f18ac06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:45.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.641+0000 7f18b3741640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac15ecd0 con 0x7f18ac15f720 2023-11-28T20:51:45.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.641+0000 7f18b3741640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac15f290 con 0x7f18ac15e840 2023-11-28T20:51:45.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.641+0000 7f18b3741640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac15fbd0 con 0x7f18ac06ac10 2023-11-28T20:51:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.641+0000 7f18b1cde640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f18ac15f720 0x7f18ac0cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39906/0 (socket says 172.21.15.45:39906) 2023-11-28T20:51:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.641+0000 7f18b1cde640 1 -- 172.21.15.45:0/1412699645 learned_addr learned my addr 172.21.15.45:0/1412699645 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.641+0000 7f18b0cdc640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac06ac10 0x7f18ac06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683165033 0 0) 0x7f18ac15ecd0 con 0x7f18ac15f720 2023-11-28T20:51:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898003520 con 0x7f18ac15f720 2023-11-28T20:51:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441817754 0 0) 0x7f1898003520 con 0x7f18ac15f720 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac06ac10 msgr2=0x7f18ac06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac06ac10 0x7f18ac06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 >> v1:172.21.15.190:6789/0 conn(0x7f18ac15e840 legacy=0x7f18ac06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac072020 con 0x7f18ac15f720 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f18a4001670 con 0x7f18ac15f720 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 >> v1:172.21.15.45:6789/0 conn(0x7f18ac15f720 legacy=0x7f18ac0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac06ac10 0x7f18980039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898003f20 0x7f18980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f1898005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac15fbd0 con 0x7f1898004d00 2023-11-28T20:51:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189800fe70 con 0x7f1898003f20 2023-11-28T20:51:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.642+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1898010030 con 0x7f18ac06ac10 2023-11-28T20:51:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.643+0000 7f18b1cde640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f1898005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.643+0000 7f18b14dd640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac06ac10 0x7f18980039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.643+0000 7f18b0cdc640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898003f20 0x7f18980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.643+0000 7f18b1cde640 1 -- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac06ac10 msgr2=0x7f18980039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.643+0000 7f18b1cde640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac06ac10 0x7f18980039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.643+0000 7f18b1cde640 1 -- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898003f20 msgr2=0x7f18980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.643+0000 7f18b1cde640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898003f20 0x7f18980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.643+0000 7f18b1cde640 1 -- 172.21.15.45:0/1412699645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1898003520 con 0x7f1898004d00 2023-11-28T20:51:45.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.645+0000 7f18b1cde640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f1898005180 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f18a400c240 tx=0x7f18980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89583bf5da324de3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.645+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4005ad0 con 0x7f1898004d00 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.645+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a4005c30 con 0x7f1898004d00 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.645+0000 7f18a3fff640 1 -- 172.21.15.45:0/1412699645 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a402caf0 con 0x7f1898004d00 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.645+0000 7f18b3741640 1 -- 172.21.15.45:0/1412699645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 msgr2=0x7f1898005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.645+0000 7f18b3741640 1 --2- 172.21.15.45:0/1412699645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f1898005180 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f18a400c240 tx=0x7f18980057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.646+0000 7f18b1cde640 1 -- 172.21.15.45:0/1412699645 reap_dead start 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.646+0000 7f18b3741640 1 -- 172.21.15.45:0/1412699645 shutdown_connections 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.646+0000 7f18b3741640 1 -- 172.21.15.45:0/1412699645 >> 172.21.15.45:0/1412699645 conn(0x7f18ac0ced60 msgr2=0x7f18ac0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.646+0000 7f18b3741640 1 -- 172.21.15.45:0/1412699645 shutdown_connections 2023-11-28T20:51:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.646+0000 7f18b3741640 1 -- 172.21.15.45:0/1412699645 wait complete. 2023-11-28T20:51:45.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.648+0000 7f18b3741640 1 Processor -- start 2023-11-28T20:51:45.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.648+0000 7f18b3741640 1 -- start start 2023-11-28T20:51:45.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.649+0000 7f18b3741640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f18ac0b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:45.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.649+0000 7f18b3741640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac156d80 0x7f18ac0c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:45.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.649+0000 7f18b3741640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac0c5220 0x7f18ac0af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:45.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.649+0000 7f18b3741640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0c6900 con 0x7f1898004d00 2023-11-28T20:51:45.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.649+0000 7f18b3741640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0b1f90 con 0x7f18ac156d80 2023-11-28T20:51:45.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.649+0000 7f18b3741640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0c5fd0 con 0x7f18ac0c5220 2023-11-28T20:51:45.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.649+0000 7f18b14dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f18ac0b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:45.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.649+0000 7f18b14dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f18ac0b1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35788/0 (socket says 172.21.15.45:35788) 2023-11-28T20:51:45.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.649+0000 7f18b14dd640 1 -- 172.21.15.45:0/2379143494 learned_addr learned my addr 172.21.15.45:0/2379143494 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:45.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b1cde640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac0c5220 0x7f18ac0af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b14dd640 1 -- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac0c5220 msgr2=0x7f18ac0af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b14dd640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac0c5220 0x7f18ac0af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b0cdc640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac156d80 0x7f18ac0c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b14dd640 1 -- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac156d80 msgr2=0x7f18ac0c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b14dd640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac156d80 0x7f18ac0c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b14dd640 1 -- 172.21.15.45:0/2379143494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac0c62b0 con 0x7f1898004d00 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b14dd640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f18ac0b1a50 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f18a800e8c0 tx=0x7f18a800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18a1ffb640 1 -- 172.21.15.45:0/2379143494 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8003b80 con 0x7f1898004d00 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18a1ffb640 1 -- 172.21.15.45:0/2379143494 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a8003ce0 con 0x7f1898004d00 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac0c6590 con 0x7f1898004d00 2023-11-28T20:51:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18a1ffb640 1 -- 172.21.15.45:0/2379143494 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a80052f0 con 0x7f1898004d00 2023-11-28T20:51:45.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.650+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac157c50 con 0x7f1898004d00 2023-11-28T20:51:45.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.652+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac0733d0 con 0x7f1898004d00 2023-11-28T20:51:45.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.653+0000 7f18a1ffb640 1 -- 172.21.15.45:0/2379143494 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f18a8005450 con 0x7f1898004d00 2023-11-28T20:51:45.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.654+0000 7f18a1ffb640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f189006aed0 0x7f189006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:45.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.654+0000 7f18a1ffb640 1 -- 172.21.15.45:0/2379143494 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7f18a8089970 con 0x7f1898004d00 2023-11-28T20:51:45.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.654+0000 7f18b0cdc640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f189006aed0 0x7f189006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:45.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.655+0000 7f18b0cdc640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f189006aed0 0x7f189006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0b46e0 tx=0x7f189c008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.658+0000 7f18a1ffb640 1 -- 172.21.15.45:0/2379143494 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18a8054880 con 0x7f1898004d00 2023-11-28T20:51:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.863+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f18ac0ce2b0 con 0x7f1898004d00 2023-11-28T20:51:45.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.871+0000 7f18a1ffb640 1 -- 172.21.15.45:0/2379143494 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f18a801a310 con 0x7f1898004d00 2023-11-28T20:51:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f189006aed0 msgr2=0x7f189006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f189006aed0 0x7f189006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0b46e0 tx=0x7f189c008040 comp rx=0 tx=0).stop 2023-11-28T20:51:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 msgr2=0x7f18ac0b1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f18ac0b1a50 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f18a800e8c0 tx=0x7f18a800ed80 comp rx=0 tx=0).stop 2023-11-28T20:51:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 shutdown_connections 2023-11-28T20:51:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f189006aed0 0x7f189006d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18ac0c5220 0x7f18ac0af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac156d80 0x7f18ac0c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 --2- 172.21.15.45:0/2379143494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1898004d00 0x7f18ac0b1a50 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 >> 172.21.15.45:0/2379143494 conn(0x7f18ac0ced60 msgr2=0x7f18ac0717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.874+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 shutdown_connections 2023-11-28T20:51:45.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:45.875+0000 7f18b3741640 1 -- 172.21.15.45:0/2379143494 wait complete. 2023-11-28T20:51:45.897 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:51:45.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:51:45.897 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:45.893+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:46.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:51:46.151 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:51:46.152 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:51:46.152 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:46.217 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:51:46.217 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:51:46.217 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:46.278 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:51:46.279 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:51:46.279 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:46.340 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:51:46.340 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:51:46.340 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:51:46.398 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:51:46.398 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T20:51:46.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:46.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.636+0000 7faa76ae6640 1 Processor -- start 2023-11-28T20:51:46.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.636+0000 7faa76ae6640 1 -- start start 2023-11-28T20:51:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.638+0000 7faa76ae6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faa700ae540 0x7faa700ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.638+0000 7faa76ae6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7015c3c0 con 0x7faa70155780 2023-11-28T20:51:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.638+0000 7faa76ae6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa70155b60 con 0x7faa700ce630 2023-11-28T20:51:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.638+0000 7faa76ae6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa700ceac0 con 0x7faa700ae540 2023-11-28T20:51:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.638+0000 7faa75ae4640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faa70155780 0x7faa700abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39918/0 (socket says 172.21.15.45:39918) 2023-11-28T20:51:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.638+0000 7faa75ae4640 1 -- 172.21.15.45:0/1909840895 learned_addr learned my addr 172.21.15.45:0/1909840895 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.638+0000 7faa752e3640 1 --2- 172.21.15.45:0/1909840895 >> v2:172.21.15.45:3300/0 conn(0x7faa700ae540 0x7faa700ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.639+0000 7faa752e3640 1 -- 172.21.15.45:0/1909840895 >> v1:172.21.15.190:6789/0 conn(0x7faa700ce630 legacy=0x7faa700b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.639+0000 7faa752e3640 1 -- 172.21.15.45:0/1909840895 >> v1:172.21.15.45:6789/0 conn(0x7faa70155780 legacy=0x7faa700abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.639+0000 7faa74ae2640 1 -- 172.21.15.45:0/1909840895 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037270871 0 0) 0x7faa7015c3c0 con 0x7faa70155780 2023-11-28T20:51:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.639+0000 7faa752e3640 1 -- 172.21.15.45:0/1909840895 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7015ba90 con 0x7faa700ae540 2023-11-28T20:51:46.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.640+0000 7faa752e3640 1 --2- 172.21.15.45:0/1909840895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ae540 0x7faa700ce0f0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7faa600097a0 tx=0x7faa6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc217d3184cde3c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:46.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.640+0000 7faa74ae2640 1 -- 172.21.15.45:0/1909840895 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60031830 con 0x7faa700ae540 2023-11-28T20:51:46.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.640+0000 7faa74ae2640 1 -- 172.21.15.45:0/1909840895 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa60031990 con 0x7faa700ae540 2023-11-28T20:51:46.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.640+0000 7faa74ae2640 1 -- 172.21.15.45:0/1909840895 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60031d20 con 0x7faa700ae540 2023-11-28T20:51:46.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.641+0000 7faa76ae6640 1 -- 172.21.15.45:0/1909840895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ae540 msgr2=0x7faa700ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:46.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.641+0000 7faa76ae6640 1 --2- 172.21.15.45:0/1909840895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ae540 0x7faa700ce0f0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7faa600097a0 tx=0x7faa6002b8d0 comp rx=0 tx=0).stop 2023-11-28T20:51:46.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.641+0000 7faa76ae6640 1 -- 172.21.15.45:0/1909840895 shutdown_connections 2023-11-28T20:51:46.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.641+0000 7faa76ae6640 1 --2- 172.21.15.45:0/1909840895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ae540 0x7faa700ce0f0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:46.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.641+0000 7faa76ae6640 1 -- 172.21.15.45:0/1909840895 >> 172.21.15.45:0/1909840895 conn(0x7faa7006ab30 msgr2=0x7faa700b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:46.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.641+0000 7faa76ae6640 1 -- 172.21.15.45:0/1909840895 shutdown_connections 2023-11-28T20:51:46.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.641+0000 7faa76ae6640 1 -- 172.21.15.45:0/1909840895 wait complete. 2023-11-28T20:51:46.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.643+0000 7faa76ae6640 1 Processor -- start 2023-11-28T20:51:46.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.643+0000 7faa76ae6640 1 -- start start 2023-11-28T20:51:46.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.644+0000 7faa76ae6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa700ae540 0x7faa701534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:46.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.644+0000 7faa76ae6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ce630 0x7faa700c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:46.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.644+0000 7faa76ae6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa70155780 0x7faa70099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:46.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.644+0000 7faa76ae6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa70099f80 con 0x7faa700ce630 2023-11-28T20:51:46.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.644+0000 7faa76ae6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7009b200 con 0x7faa70155780 2023-11-28T20:51:46.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.644+0000 7faa76ae6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa700c5680 con 0x7faa700ae540 2023-11-28T20:51:46.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.644+0000 7faa752e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ce630 0x7faa700c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:46.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.644+0000 7faa752e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ce630 0x7faa700c42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35792/0 (socket says 172.21.15.45:35792) 2023-11-28T20:51:46.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.644+0000 7faa752e3640 1 -- 172.21.15.45:0/588611731 learned_addr learned my addr 172.21.15.45:0/588611731 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa75ae4640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa700ae540 0x7faa701534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa762e5640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa70155780 0x7faa70099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa752e3640 1 -- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa700ae540 msgr2=0x7faa701534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa752e3640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa700ae540 0x7faa701534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa752e3640 1 -- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa70155780 msgr2=0x7faa70099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa752e3640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa70155780 0x7faa70099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa752e3640 1 -- 172.21.15.45:0/588611731 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa60009400 con 0x7faa700ce630 2023-11-28T20:51:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa75ae4640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa700ae540 0x7faa701534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa762e5640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa70155780 0x7faa70099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.645+0000 7faa752e3640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ce630 0x7faa700c42c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7faa600098c0 tx=0x7faa600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.646+0000 7faa667fc640 1 -- 172.21.15.45:0/588611731 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6003ba70 con 0x7faa700ce630 2023-11-28T20:51:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.646+0000 7faa667fc640 1 -- 172.21.15.45:0/588611731 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6003bbf0 con 0x7faa700ce630 2023-11-28T20:51:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.646+0000 7faa667fc640 1 -- 172.21.15.45:0/588611731 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa600453f0 con 0x7faa700ce630 2023-11-28T20:51:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.646+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7009a170 con 0x7faa700ce630 2023-11-28T20:51:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.646+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7009a730 con 0x7faa700ce630 2023-11-28T20:51:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.647+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa700b1030 con 0x7faa700ce630 2023-11-28T20:51:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.648+0000 7faa667fc640 1 -- 172.21.15.45:0/588611731 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7faa60045550 con 0x7faa700ce630 2023-11-28T20:51:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.649+0000 7faa667fc640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7faa5406aed0 0x7faa5406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.649+0000 7faa667fc640 1 -- 172.21.15.45:0/588611731 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7faa600abd30 con 0x7faa700ce630 2023-11-28T20:51:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.650+0000 7faa75ae4640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7faa5406aed0 0x7faa5406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:46.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.651+0000 7faa75ae4640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7faa5406aed0 0x7faa5406d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa6c000f60 tx=0x7faa6c009040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:46.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.653+0000 7faa667fc640 1 -- 172.21.15.45:0/588611731 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa60076be0 con 0x7faa700ce630 2023-11-28T20:51:46.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.901+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa7004e920 con 0x7faa700ce630 2023-11-28T20:51:46.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.902+0000 7faa667fc640 1 -- 172.21.15.45:0/588611731 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+3705 (secure 0 0 0) 0x7faa6007a200 con 0x7faa700ce630 2023-11-28T20:51:46.904 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:46.904 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":46,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3492018158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3492018158},{"type":"v1","addr":"172.21.15.45:6833","nonce":3492018158}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6121,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2276329291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2276329291},{"type":"v1","addr":"172.21.15.190:6835","nonce":2276329291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6130,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1261857840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1261857840},{"type":"v1","addr":"172.21.15.190:6837","nonce":1261857840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":7539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2704602907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2704602907},{"type":"v1","addr":"172.21.15.45:6835","nonce":2704602907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[]} 2023-11-28T20:51:46.904 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 46 2023-11-28T20:51:46.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7faa5406aed0 msgr2=0x7faa5406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7faa5406aed0 0x7faa5406d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa6c000f60 tx=0x7faa6c009040 comp rx=0 tx=0).stop 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ce630 msgr2=0x7faa700c42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ce630 0x7faa700c42c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7faa600098c0 tx=0x7faa600046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 shutdown_connections 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7faa5406aed0 0x7faa5406d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa70155780 0x7faa70099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa700ce630 0x7faa700c42c0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 --2- 172.21.15.45:0/588611731 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa700ae540 0x7faa701534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 >> 172.21.15.45:0/588611731 conn(0x7faa7006ab30 msgr2=0x7faa7006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.905+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 shutdown_connections 2023-11-28T20:51:46.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:46.906+0000 7faa76ae6640 1 -- 172.21.15.45:0/588611731 wait complete. 2023-11-28T20:51:46.930 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T20:51:46.930 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T20:51:46.930 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T20:51:46.930 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:51:46.931 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T20:51:46.931 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T20:51:46.931 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T20:51:46.931 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:51:46.931 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T20:51:46.931 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T20:51:46.932 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T20:51:46.932 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:51:46.932 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T20:51:46.932 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T20:51:46.933 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T20:51:46.933 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:51:46.949 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T20:51:46.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T20:51:46.950 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T20:51:46.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T20:51:46.959 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T20:51:46.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T20:51:46.960 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T20:51:46.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T20:51:46.984 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:46.981+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:46.992 INFO:tasks.ceph.mds.b:Started 2023-11-28T20:51:46.993 DEBUG:teuthology.parallel:result is None 2023-11-28T20:51:46.998 INFO:tasks.ceph.mds.a:Started 2023-11-28T20:51:46.998 DEBUG:teuthology.parallel:result is None 2023-11-28T20:51:47.000 INFO:tasks.ceph.mds.d:Started 2023-11-28T20:51:47.000 DEBUG:teuthology.parallel:result is None 2023-11-28T20:51:47.005 INFO:tasks.ceph.mds.c:Started 2023-11-28T20:51:47.005 DEBUG:teuthology.parallel:result is None 2023-11-28T20:51:47.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T20:51:47.159 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:51:47.159 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T20:51:47.172 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:51:47.172 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:51:47.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.205+0000 7f69fd8ad640 1 Processor -- start 2023-11-28T20:51:47.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.205+0000 7f69fd8ad640 1 -- start start 2023-11-28T20:51:47.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.207+0000 7f69fd8ad640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f69f80cf730 0x7f69f806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:47.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.207+0000 7f69fd8ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f69f815edc0 0x7f69f815f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:47.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.207+0000 7f69fd8ad640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f815fa80 con 0x7f69f815b820 2023-11-28T20:51:47.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.207+0000 7f69fd8ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f815fd60 con 0x7f69f815edc0 2023-11-28T20:51:47.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.207+0000 7f69fd8ad640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f806cf40 con 0x7f69f80cf730 2023-11-28T20:51:47.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.207+0000 7f69f77fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f69f815edc0 0x7f69f815f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:47.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.207+0000 7f69f77fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f69f815edc0 0x7f69f815f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35824/0 (socket says 172.21.15.45:35824) 2023-11-28T20:51:47.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.207+0000 7f69f77fe640 1 -- 172.21.15.45:0/1381039238 learned_addr learned my addr 172.21.15.45:0/1381039238 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:47.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.208+0000 7f69effff640 1 --2- 172.21.15.45:0/1381039238 >> v2:172.21.15.190:3300/0 conn(0x7f69f80cf730 0x7f69f806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:47.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.208+0000 7f69f77fe640 1 -- 172.21.15.45:0/1381039238 >> v2:172.21.15.190:3300/0 conn(0x7f69f80cf730 msgr2=0x7f69f806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:47.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.208+0000 7f69f77fe640 1 --2- 172.21.15.45:0/1381039238 >> v2:172.21.15.190:3300/0 conn(0x7f69f80cf730 0x7f69f806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:47.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.208+0000 7f69f77fe640 1 -- 172.21.15.45:0/1381039238 >> v1:172.21.15.190:6789/0 conn(0x7f69f815b820 legacy=0x7f69f80cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:47.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.208+0000 7f69f77fe640 1 -- 172.21.15.45:0/1381039238 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f806d220 con 0x7f69f815edc0 2023-11-28T20:51:47.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.209+0000 7f69f77fe640 1 --2- 172.21.15.45:0/1381039238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 0x7f69f815f1e0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f69f809cf90 tx=0x7f69f002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6229eb295375a1c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:47.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.209+0000 7f69f67fc640 1 -- 172.21.15.45:0/1381039238 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f002cb20 con 0x7f69f815edc0 2023-11-28T20:51:47.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.209+0000 7f69f67fc640 1 -- 172.21.15.45:0/1381039238 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f002cc80 con 0x7f69f815edc0 2023-11-28T20:51:47.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.209+0000 7f69f67fc640 1 -- 172.21.15.45:0/1381039238 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f00343f0 con 0x7f69f815edc0 2023-11-28T20:51:47.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.210+0000 7f69fd8ad640 1 -- 172.21.15.45:0/1381039238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 msgr2=0x7f69f815f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:47.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.210+0000 7f69fd8ad640 1 --2- 172.21.15.45:0/1381039238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 0x7f69f815f1e0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f69f809cf90 tx=0x7f69f002bf80 comp rx=0 tx=0).stop 2023-11-28T20:51:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.210+0000 7f69fd8ad640 1 -- 172.21.15.45:0/1381039238 shutdown_connections 2023-11-28T20:51:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.210+0000 7f69fd8ad640 1 --2- 172.21.15.45:0/1381039238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 0x7f69f815f1e0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.210+0000 7f69fd8ad640 1 --2- 172.21.15.45:0/1381039238 >> v2:172.21.15.190:3300/0 conn(0x7f69f80cf730 0x7f69f806da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.210+0000 7f69fd8ad640 1 -- 172.21.15.45:0/1381039238 >> 172.21.15.45:0/1381039238 conn(0x7f69f806a6e0 msgr2=0x7f69f806aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.210+0000 7f69fd8ad640 1 -- 172.21.15.45:0/1381039238 shutdown_connections 2023-11-28T20:51:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.210+0000 7f69fd8ad640 1 -- 172.21.15.45:0/1381039238 wait complete. 2023-11-28T20:51:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.212+0000 7f69fd8ad640 1 Processor -- start 2023-11-28T20:51:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.212+0000 7f69fd8ad640 1 -- start start 2023-11-28T20:51:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.213+0000 7f69fd8ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69f80cf730 0x7f69f8150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.213+0000 7f69fd8ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69f815b820 0x7f69f80ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.213+0000 7f69fd8ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 0x7f69f81f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.213+0000 7f69fd8ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80acb20 con 0x7f69f815edc0 2023-11-28T20:51:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.213+0000 7f69fd8ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80abf10 con 0x7f69f815b820 2023-11-28T20:51:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.213+0000 7f69fd8ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80ac1f0 con 0x7f69f80cf730 2023-11-28T20:51:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.213+0000 7f69f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 0x7f69f81f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.213+0000 7f69f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 0x7f69f81f6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35830/0 (socket says 172.21.15.45:35830) 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.213+0000 7f69f77fe640 1 -- 172.21.15.45:0/4041054267 learned_addr learned my addr 172.21.15.45:0/4041054267 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.214+0000 7f69effff640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69f815b820 0x7f69f80ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.214+0000 7f69f6ffd640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69f80cf730 0x7f69f8150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.214+0000 7f69f77fe640 1 -- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69f80cf730 msgr2=0x7f69f8150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.214+0000 7f69f77fe640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69f80cf730 0x7f69f8150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.214+0000 7f69f77fe640 1 -- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69f815b820 msgr2=0x7f69f80ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.214+0000 7f69f77fe640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69f815b820 0x7f69f80ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.214+0000 7f69f77fe640 1 -- 172.21.15.45:0/4041054267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0009b30 con 0x7f69f815edc0 2023-11-28T20:51:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.214+0000 7f69f77fe640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 0x7f69f81f6b10 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f69f0036040 tx=0x7f69f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:47.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.214+0000 7f69f4ff9640 1 -- 172.21.15.45:0/4041054267 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f0009510 con 0x7f69f815edc0 2023-11-28T20:51:47.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.215+0000 7f69f4ff9640 1 -- 172.21.15.45:0/4041054267 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f002c920 con 0x7f69f815edc0 2023-11-28T20:51:47.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.215+0000 7f69f4ff9640 1 -- 172.21.15.45:0/4041054267 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f003e670 con 0x7f69f815edc0 2023-11-28T20:51:47.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.215+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f80ac4d0 con 0x7f69f815edc0 2023-11-28T20:51:47.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.215+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f8147e00 con 0x7f69f815edc0 2023-11-28T20:51:47.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.216+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f80accd0 con 0x7f69f815edc0 2023-11-28T20:51:47.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.217+0000 7f69f4ff9640 1 -- 172.21.15.45:0/4041054267 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f69f003e7d0 con 0x7f69f815edc0 2023-11-28T20:51:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.218+0000 7f69f4ff9640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f69d806af00 0x7f69d806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.218+0000 7f69f4ff9640 1 -- 172.21.15.45:0/4041054267 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5994+0+0 (secure 0 0 0) 0x7f69f00abe60 con 0x7f69f815edc0 2023-11-28T20:51:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.218+0000 7f69f6ffd640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f69d806af00 0x7f69d806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.219+0000 7f69f6ffd640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f69d806af00 0x7f69d806d3c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69f815c1f0 tx=0x7f69e8009040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.223+0000 7f69f4ff9640 1 -- 172.21.15.45:0/4041054267 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69f0076e40 con 0x7f69f815edc0 2023-11-28T20:51:47.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:47.445+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f69f8073440 con 0x7f69f815edc0 2023-11-28T20:51:48.118 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:48.115+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.431+0000 7f69f4ff9640 1 -- 172.21.15.45:0/4041054267 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v112) v1 ==== 97+0+0 (secure 0 0 0) 0x7f69f007a460 con 0x7f69f815edc0 2023-11-28T20:51:48.433 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T20:51:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f69d806af00 msgr2=0x7f69d806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f69d806af00 0x7f69d806d3c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69f815c1f0 tx=0x7f69e8009040 comp rx=0 tx=0).stop 2023-11-28T20:51:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 msgr2=0x7f69f81f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 0x7f69f81f6b10 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f69f0036040 tx=0x7f69f0004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 shutdown_connections 2023-11-28T20:51:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f69d806af00 0x7f69d806d3c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f815edc0 0x7f69f81f6b10 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69f815b820 0x7f69f80ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 --2- 172.21.15.45:0/4041054267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69f80cf730 0x7f69f8150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 >> 172.21.15.45:0/4041054267 conn(0x7f69f806a6e0 msgr2=0x7f69f80ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.435+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 shutdown_connections 2023-11-28T20:51:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.436+0000 7f69fd8ad640 1 -- 172.21.15.45:0/4041054267 wait complete. 2023-11-28T20:51:48.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T20:51:48.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.687+0000 7f6c9f59e640 1 Processor -- start 2023-11-28T20:51:48.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.687+0000 7f6c9f59e640 1 -- start start 2023-11-28T20:51:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.688+0000 7f6c9f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00cbb90 0x7f6ca00cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.688+0000 7f6c9f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca00703d0 con 0x7f6ca015d250 2023-11-28T20:51:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.688+0000 7f6c9f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca00706b0 con 0x7f6ca00b2a50 2023-11-28T20:51:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.688+0000 7f6c9f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca00cc580 con 0x7f6ca00cbb90 2023-11-28T20:51:48.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.689+0000 7f6c9e59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6ca015d250 0x7f6ca0071190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39946/0 (socket says 172.21.15.45:39946) 2023-11-28T20:51:48.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.689+0000 7f6c9e59c640 1 -- 172.21.15.45:0/431854680 learned_addr learned my addr 172.21.15.45:0/431854680 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:48.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.689+0000 7f6c9ed9d640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00cbb90 0x7f6ca00cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:48.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.689+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150660795 0 0) 0x7f6ca00703d0 con 0x7f6ca015d250 2023-11-28T20:51:48.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.689+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c8c003520 con 0x7f6ca015d250 2023-11-28T20:51:48.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1139778570 0 0) 0x7f6c8c003520 con 0x7f6ca015d250 2023-11-28T20:51:48.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00cbb90 msgr2=0x7f6ca00cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00cbb90 0x7f6ca00cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 >> v1:172.21.15.190:6789/0 conn(0x7f6ca00b2a50 legacy=0x7f6ca0071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca0070990 con 0x7f6ca015d250 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6c90001670 con 0x7f6ca015d250 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 >> v1:172.21.15.45:6789/0 conn(0x7f6ca015d250 legacy=0x7f6ca0071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00cbb90 0x7f6c8c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8c003f20 0x7f6c8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c004d00 0x7f6c8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca00cc580 con 0x7f6c8c003f20 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c00bb10 con 0x7f6c8c004d00 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c00bcd0 con 0x7f6ca00cbb90 2023-11-28T20:51:48.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.690+0000 7f6c97fff640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8c003f20 0x7f6c8c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:48.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.691+0000 7f6c9e59c640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00cbb90 0x7f6c8c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:48.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.691+0000 7f6c9ed9d640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c004d00 0x7f6c8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:48.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.691+0000 7f6c97fff640 1 -- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00cbb90 msgr2=0x7f6c8c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:48.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.691+0000 7f6c97fff640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00cbb90 0x7f6c8c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.691+0000 7f6c97fff640 1 -- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c004d00 msgr2=0x7f6c8c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:48.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.691+0000 7f6c97fff640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c004d00 0x7f6c8c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.691+0000 7f6c97fff640 1 -- 172.21.15.45:0/431854680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c003520 con 0x7f6c8c003f20 2023-11-28T20:51:48.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.692+0000 7f6c97fff640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8c003f20 0x7f6c8c0047c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f6c88001720 tx=0x7f6c8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344e24141b0d62bb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:48.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.692+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880052d0 con 0x7f6c8c003f20 2023-11-28T20:51:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.692+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8802e050 con 0x7f6c8c003f20 2023-11-28T20:51:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.692+0000 7f6c9dd9b640 1 -- 172.21.15.45:0/431854680 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880035f0 con 0x7f6c8c003f20 2023-11-28T20:51:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.693+0000 7f6c9f59e640 1 -- 172.21.15.45:0/431854680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8c003f20 msgr2=0x7f6c8c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.693+0000 7f6c9f59e640 1 --2- 172.21.15.45:0/431854680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c8c003f20 0x7f6c8c0047c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f6c88001720 tx=0x7f6c8c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.693+0000 7f6c9ed9d640 1 -- 172.21.15.45:0/431854680 reap_dead start 2023-11-28T20:51:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.693+0000 7f6c9f59e640 1 -- 172.21.15.45:0/431854680 shutdown_connections 2023-11-28T20:51:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.693+0000 7f6c9f59e640 1 -- 172.21.15.45:0/431854680 >> 172.21.15.45:0/431854680 conn(0x7f6ca00b0e70 msgr2=0x7f6ca00b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.693+0000 7f6c9f59e640 1 -- 172.21.15.45:0/431854680 shutdown_connections 2023-11-28T20:51:48.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.693+0000 7f6c9f59e640 1 -- 172.21.15.45:0/431854680 wait complete. 2023-11-28T20:51:48.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.696+0000 7f6c9f59e640 1 Processor -- start 2023-11-28T20:51:48.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.696+0000 7f6c9f59e640 1 -- start start 2023-11-28T20:51:48.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.697+0000 7f6c9f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c003f20 0x7f6ca00ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:48.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.697+0000 7f6c9f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00c8850 0x7f6ca009c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:48.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.697+0000 7f6c9f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ca0099bd0 0x7f6ca009a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.697+0000 7f6c9f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca009a5c0 con 0x7f6ca0099bd0 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.697+0000 7f6c9f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca00c99d0 con 0x7f6c8c003f20 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.697+0000 7f6c9f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca00c8dc0 con 0x7f6ca00c8850 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.697+0000 7f6c9ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ca0099bd0 0x7f6ca009a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.697+0000 7f6c9ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ca0099bd0 0x7f6ca009a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35850/0 (socket says 172.21.15.45:35850) 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.697+0000 7f6c9ed9d640 1 -- 172.21.15.45:0/69173103 learned_addr learned my addr 172.21.15.45:0/69173103 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c9e59c640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c003f20 0x7f6ca00ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c97fff640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00c8850 0x7f6ca009c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c9ed9d640 1 -- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00c8850 msgr2=0x7f6ca009c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c9ed9d640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00c8850 0x7f6ca009c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c9ed9d640 1 -- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c003f20 msgr2=0x7f6ca00ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:48.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c9ed9d640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c003f20 0x7f6ca00ad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c9ed9d640 1 -- 172.21.15.45:0/69173103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca00c90a0 con 0x7f6ca0099bd0 2023-11-28T20:51:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c9e59c640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c003f20 0x7f6ca00ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c9ed9d640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ca0099bd0 0x7f6ca009a080 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00cc750 tx=0x7f6c980115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c96ffd640 1 -- 172.21.15.45:0/69173103 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800e070 con 0x7f6ca0099bd0 2023-11-28T20:51:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c96ffd640 1 -- 172.21.15.45:0/69173103 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c98011e10 con 0x7f6ca0099bd0 2023-11-28T20:51:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c96ffd640 1 -- 172.21.15.45:0/69173103 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003990 con 0x7f6ca0099bd0 2023-11-28T20:51:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.698+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca00c9380 con 0x7f6ca0099bd0 2023-11-28T20:51:48.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.699+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca015d630 con 0x7f6ca0099bd0 2023-11-28T20:51:48.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.700+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca00c9b40 con 0x7f6ca0099bd0 2023-11-28T20:51:48.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.701+0000 7f6c96ffd640 1 -- 172.21.15.45:0/69173103 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f6c98003af0 con 0x7f6ca0099bd0 2023-11-28T20:51:48.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.702+0000 7f6c96ffd640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6c8006aed0 0x7f6c8006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:48.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.702+0000 7f6c96ffd640 1 -- 172.21.15.45:0/69173103 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6c98089c30 con 0x7f6ca0099bd0 2023-11-28T20:51:48.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.703+0000 7f6c9e59c640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6c8006aed0 0x7f6c8006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:48.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.704+0000 7f6c9e59c640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6c8006aed0 0x7f6c8006d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6c90001050 tx=0x7f6c90001d40 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:48.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.707+0000 7f6c96ffd640 1 -- 172.21.15.45:0/69173103 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c98054d80 con 0x7f6ca0099bd0 2023-11-28T20:51:48.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.977+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6ca004e920 con 0x7f6ca0099bd0 2023-11-28T20:51:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.978+0000 7f6c96ffd640 1 -- 172.21.15.45:0/69173103 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6c980583a0 con 0x7f6ca0099bd0 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T20:51:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T20:51:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T20:51:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T20:51:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T20:51:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T20:51:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T20:51:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T20:51:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T20:51:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T20:51:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:51:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:51:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T20:51:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T20:51:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T20:51:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T20:51:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T20:51:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T20:51:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:51:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:51:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:51:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:51:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:51:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T20:51:48.997 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:51:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6c8006aed0 msgr2=0x7f6c8006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6c8006aed0 0x7f6c8006d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6c90001050 tx=0x7f6c90001d40 comp rx=0 tx=0).stop 2023-11-28T20:51:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ca0099bd0 msgr2=0x7f6ca009a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ca0099bd0 0x7f6ca009a080 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00cc750 tx=0x7f6c980115a0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 shutdown_connections 2023-11-28T20:51:48.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6c8006aed0 0x7f6c8006d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ca0099bd0 0x7f6ca009a080 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ca00c8850 0x7f6ca009c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 --2- 172.21.15.45:0/69173103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c8c003f20 0x7f6ca00ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:48.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 >> 172.21.15.45:0/69173103 conn(0x7f6ca00b0e70 msgr2=0x7f6ca0071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:48.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 shutdown_connections 2023-11-28T20:51:48.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:48.982+0000 7f6c9f59e640 1 -- 172.21.15.45:0/69173103 wait complete. 2023-11-28T20:51:49.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.231+0000 7f7aff0aa640 1 Processor -- start 2023-11-28T20:51:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.232+0000 7f7aff0aa640 1 -- start start 2023-11-28T20:51:49.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.234+0000 7f7aff0aa640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7af80b24a0 0x7f7af80ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.234+0000 7f7aff0aa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7af80ae590 0x7f7af80ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7aff0aa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af80abf10 con 0x7f7af814b090 2023-11-28T20:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7aff0aa640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af806db90 con 0x7f7af80ae590 2023-11-28T20:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7aff0aa640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af806d2f0 con 0x7f7af80b24a0 2023-11-28T20:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7afe0a8640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7af814b090 0x7f7af80b1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39954/0 (socket says 172.21.15.45:39954) 2023-11-28T20:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7afe0a8640 1 -- 172.21.15.45:0/869945920 learned_addr learned my addr 172.21.15.45:0/869945920 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7afe8a9640 1 --2- 172.21.15.45:0/869945920 >> v2:172.21.15.45:3300/0 conn(0x7f7af80ae590 0x7f7af80ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7afd8a7640 1 --2- 172.21.15.45:0/869945920 >> v2:172.21.15.190:3300/0 conn(0x7f7af80b24a0 0x7f7af80ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7afd0a6640 1 -- 172.21.15.45:0/869945920 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618582692 0 0) 0x7f7af80abf10 con 0x7f7af814b090 2023-11-28T20:51:49.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7afe8a9640 1 -- 172.21.15.45:0/869945920 >> v2:172.21.15.190:3300/0 conn(0x7f7af80b24a0 msgr2=0x7f7af80ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7afe8a9640 1 --2- 172.21.15.45:0/869945920 >> v2:172.21.15.190:3300/0 conn(0x7f7af80b24a0 0x7f7af80ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.235+0000 7f7afe8a9640 1 -- 172.21.15.45:0/869945920 >> v1:172.21.15.45:6789/0 conn(0x7f7af814b090 legacy=0x7f7af80b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.236+0000 7f7afe8a9640 1 -- 172.21.15.45:0/869945920 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af806d5d0 con 0x7f7af80ae590 2023-11-28T20:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.236+0000 7f7afd8a7640 1 --2- 172.21.15.45:0/869945920 >> v2:172.21.15.190:3300/0 conn(0x7f7af80b24a0 0x7f7af80ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.237+0000 7f7afe8a9640 1 --2- 172.21.15.45:0/869945920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 0x7f7af80ac8e0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f7af80ced90 tx=0x7f7af002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193a7ccc38c51425 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.237+0000 7f7afd0a6640 1 -- 172.21.15.45:0/869945920 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af002cc70 con 0x7f7af80ae590 2023-11-28T20:51:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.237+0000 7f7afd0a6640 1 -- 172.21.15.45:0/869945920 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af002cdd0 con 0x7f7af80ae590 2023-11-28T20:51:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.237+0000 7f7afd0a6640 1 -- 172.21.15.45:0/869945920 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0034660 con 0x7f7af80ae590 2023-11-28T20:51:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.238+0000 7f7aff0aa640 1 -- 172.21.15.45:0/869945920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 msgr2=0x7f7af80ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.238+0000 7f7aff0aa640 1 --2- 172.21.15.45:0/869945920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 0x7f7af80ac8e0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f7af80ced90 tx=0x7f7af002bec0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.238+0000 7f7aff0aa640 1 -- 172.21.15.45:0/869945920 shutdown_connections 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.238+0000 7f7aff0aa640 1 --2- 172.21.15.45:0/869945920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 0x7f7af80ac8e0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.238+0000 7f7aff0aa640 1 --2- 172.21.15.45:0/869945920 >> v2:172.21.15.190:3300/0 conn(0x7f7af80b24a0 0x7f7af80ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.238+0000 7f7aff0aa640 1 -- 172.21.15.45:0/869945920 >> 172.21.15.45:0/869945920 conn(0x7f7af806ab30 msgr2=0x7f7af80b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.238+0000 7f7aff0aa640 1 -- 172.21.15.45:0/869945920 shutdown_connections 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.238+0000 7f7aff0aa640 1 -- 172.21.15.45:0/869945920 wait complete. 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.240+0000 7f7aff0aa640 1 Processor -- start 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.240+0000 7f7aff0aa640 1 -- start start 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7aff0aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 0x7f7af809bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7aff0aa640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7af80b24a0 0x7f7af8145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7aff0aa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7af814b090 0x7f7af80aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7aff0aa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af809c0e0 con 0x7f7af80ae590 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7aff0aa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af8099d70 con 0x7f7af814b090 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7aff0aa640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af809a050 con 0x7f7af80b24a0 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afe0a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 0x7f7af809bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afe0a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 0x7f7af809bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35872/0 (socket says 172.21.15.45:35872) 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afe0a8640 1 -- 172.21.15.45:0/643782419 learned_addr learned my addr 172.21.15.45:0/643782419 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afd8a7640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7af80b24a0 0x7f7af8145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afe8a9640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7af814b090 0x7f7af80aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afe0a8640 1 -- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7af80b24a0 msgr2=0x7f7af8145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afe0a8640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7af80b24a0 0x7f7af8145eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afe0a8640 1 -- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7af814b090 msgr2=0x7f7af80aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afe0a8640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7af814b090 0x7f7af80aff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afe0a8640 1 -- 172.21.15.45:0/643782419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af0009a70 con 0x7f7af80ae590 2023-11-28T20:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.241+0000 7f7afd8a7640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7af80b24a0 0x7f7af8145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.242+0000 7f7afe0a8640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 0x7f7af809bba0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f7af400c430 tx=0x7f7af400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.242+0000 7f7aeeffd640 1 -- 172.21.15.45:0/643782419 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af40074e0 con 0x7f7af80ae590 2023-11-28T20:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.242+0000 7f7aeeffd640 1 -- 172.21.15.45:0/643782419 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af4007660 con 0x7f7af80ae590 2023-11-28T20:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.242+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af809a330 con 0x7f7af80ae590 2023-11-28T20:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.242+0000 7f7aeeffd640 1 -- 172.21.15.45:0/643782419 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4004a20 con 0x7f7af80ae590 2023-11-28T20:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.242+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af809a8f0 con 0x7f7af80ae590 2023-11-28T20:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.243+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af81f7910 con 0x7f7af80ae590 2023-11-28T20:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.245+0000 7f7aeeffd640 1 -- 172.21.15.45:0/643782419 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f7af4004b80 con 0x7f7af80ae590 2023-11-28T20:51:49.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.246+0000 7f7aeeffd640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f7adc06aed0 0x7f7adc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.247+0000 7f7aeeffd640 1 -- 172.21.15.45:0/643782419 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f7af40892b0 con 0x7f7af80ae590 2023-11-28T20:51:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.247+0000 7f7afd8a7640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f7adc06aed0 0x7f7adc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.248+0000 7f7afd8a7640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f7adc06aed0 0x7f7adc06d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7af80b12f0 tx=0x7f7ae8008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.251+0000 7f7aeeffd640 1 -- 172.21.15.45:0/643782419 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7af4054580 con 0x7f7af80ae590 2023-11-28T20:51:49.292 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:49.289+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:49.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.452+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7af809d0f0 con 0x7f7af80ae590 2023-11-28T20:51:49.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.454+0000 7f7aeeffd640 1 -- 172.21.15.45:0/643782419 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+13060 (secure 0 0 0) 0x7f7af4057ba0 con 0x7f7af80ae590 2023-11-28T20:51:49.455 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:49.455 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":112,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:48.428021+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f7adc06aed0 msgr2=0x7f7adc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f7adc06aed0 0x7f7adc06d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7af80b12f0 tx=0x7f7ae8008040 comp rx=0 tx=0).stop 2023-11-28T20:51:49.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 msgr2=0x7f7af809bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 0x7f7af809bba0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f7af400c430 tx=0x7f7af400c8f0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 shutdown_connections 2023-11-28T20:51:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f7adc06aed0 0x7f7adc06d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7af814b090 0x7f7af80aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7af80b24a0 0x7f7af8145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 --2- 172.21.15.45:0/643782419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7af80ae590 0x7f7af809bba0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 >> 172.21.15.45:0/643782419 conn(0x7f7af806ab30 msgr2=0x7f7af80af260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:49.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 shutdown_connections 2023-11-28T20:51:49.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.456+0000 7f7aff0aa640 1 -- 172.21.15.45:0/643782419 wait complete. 2023-11-28T20:51:49.480 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:51:49.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:51:49.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.693+0000 7fd769988640 1 Processor -- start 2023-11-28T20:51:49.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.693+0000 7fd769988640 1 -- start start 2023-11-28T20:51:49.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.694+0000 7fd769988640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd76415d4b0 0x7fd764070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd769988640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd7640adfa0 0x7fd7640717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd769988640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640708d0 con 0x7fd7640712a0 2023-11-28T20:51:49.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd769988640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7640b3bb0 con 0x7fd7640adfa0 2023-11-28T20:51:49.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd769988640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7640b3e90 con 0x7fd76415d4b0 2023-11-28T20:51:49.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd769187640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd7640adfa0 0x7fd7640717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd763fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd7640712a0 0x7fd7640abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39958/0 (socket says 172.21.15.45:39958) 2023-11-28T20:51:49.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd763fff640 1 -- 172.21.15.45:0/689048929 learned_addr learned my addr 172.21.15.45:0/689048929 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:49.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd768986640 1 --2- 172.21.15.45:0/689048929 >> v2:172.21.15.190:3300/0 conn(0x7fd76415d4b0 0x7fd764070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd769187640 1 -- 172.21.15.45:0/689048929 >> v2:172.21.15.190:3300/0 conn(0x7fd76415d4b0 msgr2=0x7fd764070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:49.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.695+0000 7fd769187640 1 --2- 172.21.15.45:0/689048929 >> v2:172.21.15.190:3300/0 conn(0x7fd76415d4b0 0x7fd764070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.696+0000 7fd769187640 1 -- 172.21.15.45:0/689048929 >> v1:172.21.15.45:6789/0 conn(0x7fd7640712a0 legacy=0x7fd7640abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.696+0000 7fd769187640 1 -- 172.21.15.45:0/689048929 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd764070280 con 0x7fd7640adfa0 2023-11-28T20:51:49.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.697+0000 7fd769187640 1 --2- 172.21.15.45:0/689048929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7640adfa0 0x7fd7640717e0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd7640ce4a0 tx=0x7fd75c02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3fc4fc823d8aaf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:49.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.697+0000 7fd7637fe640 1 -- 172.21.15.45:0/689048929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c02cbe0 con 0x7fd7640adfa0 2023-11-28T20:51:49.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.697+0000 7fd7637fe640 1 -- 172.21.15.45:0/689048929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c02cd40 con 0x7fd7640adfa0 2023-11-28T20:51:49.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.697+0000 7fd7637fe640 1 -- 172.21.15.45:0/689048929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c034660 con 0x7fd7640adfa0 2023-11-28T20:51:49.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.698+0000 7fd769988640 1 -- 172.21.15.45:0/689048929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7640adfa0 msgr2=0x7fd7640717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.698+0000 7fd769988640 1 --2- 172.21.15.45:0/689048929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7640adfa0 0x7fd7640717e0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd7640ce4a0 tx=0x7fd75c02be30 comp rx=0 tx=0).stop 2023-11-28T20:51:49.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.698+0000 7fd769988640 1 -- 172.21.15.45:0/689048929 shutdown_connections 2023-11-28T20:51:49.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.698+0000 7fd769988640 1 --2- 172.21.15.45:0/689048929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7640adfa0 0x7fd7640717e0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.698+0000 7fd769988640 1 --2- 172.21.15.45:0/689048929 >> v2:172.21.15.190:3300/0 conn(0x7fd76415d4b0 0x7fd764070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.698+0000 7fd769988640 1 -- 172.21.15.45:0/689048929 >> 172.21.15.45:0/689048929 conn(0x7fd7640b1050 msgr2=0x7fd7640cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:49.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.698+0000 7fd769988640 1 -- 172.21.15.45:0/689048929 shutdown_connections 2023-11-28T20:51:49.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.698+0000 7fd769988640 1 -- 172.21.15.45:0/689048929 wait complete. 2023-11-28T20:51:49.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.700+0000 7fd769988640 1 Processor -- start 2023-11-28T20:51:49.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.700+0000 7fd769988640 1 -- start start 2023-11-28T20:51:49.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd769988640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7640712a0 0x7fd7641549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd769988640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7640adfa0 0x7fd76406d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd769988640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd76415d4b0 0x7fd76406d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd769988640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd764156fb0 con 0x7fd76415d4b0 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd769988640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd764140420 con 0x7fd7640adfa0 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd769988640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd764156610 con 0x7fd7640712a0 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd769187640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd76415d4b0 0x7fd76406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd769187640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd76415d4b0 0x7fd76406d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35882/0 (socket says 172.21.15.45:35882) 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd769187640 1 -- 172.21.15.45:0/916359333 learned_addr learned my addr 172.21.15.45:0/916359333 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.701+0000 7fd763fff640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7640adfa0 0x7fd76406d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd768986640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7640712a0 0x7fd7641549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd769187640 1 -- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7640712a0 msgr2=0x7fd7641549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd769187640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7640712a0 0x7fd7641549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd769187640 1 -- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7640adfa0 msgr2=0x7fd76406d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd769187640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7640adfa0 0x7fd76406d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd769187640 1 -- 172.21.15.45:0/916359333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c009a70 con 0x7fd76415d4b0 2023-11-28T20:51:49.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd769187640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd76415d4b0 0x7fd76406d620 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fd75c037040 tx=0x7fd75c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:49.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd7617fa640 1 -- 172.21.15.45:0/916359333 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c03e610 con 0x7fd76415d4b0 2023-11-28T20:51:49.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd7617fa640 1 -- 172.21.15.45:0/916359333 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c03e770 con 0x7fd76415d4b0 2023-11-28T20:51:49.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd7617fa640 1 -- 172.21.15.45:0/916359333 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c03eb00 con 0x7fd76415d4b0 2023-11-28T20:51:49.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.702+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7641568f0 con 0x7fd76415d4b0 2023-11-28T20:51:49.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.703+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd764154ef0 con 0x7fd76415d4b0 2023-11-28T20:51:49.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.704+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7641570f0 con 0x7fd76415d4b0 2023-11-28T20:51:49.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.705+0000 7fd7617fa640 1 -- 172.21.15.45:0/916359333 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fd75c03b050 con 0x7fd76415d4b0 2023-11-28T20:51:49.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.706+0000 7fd7617fa640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd74806aed0 0x7fd74806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:49.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.706+0000 7fd7617fa640 1 -- 172.21.15.45:0/916359333 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd75c0abd60 con 0x7fd76415d4b0 2023-11-28T20:51:49.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.707+0000 7fd768986640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd74806aed0 0x7fd74806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:49.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.708+0000 7fd768986640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd74806aed0 0x7fd74806d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd758006fd0 tx=0x7fd758008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:49.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.711+0000 7fd7617fa640 1 -- 172.21.15.45:0/916359333 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd75c0770c0 con 0x7fd76415d4b0 2023-11-28T20:51:49.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.907+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd76409d0f0 con 0x7fd76415d4b0 2023-11-28T20:51:49.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.908+0000 7fd7617fa640 1 -- 172.21.15.45:0/916359333 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd75c04e020 con 0x7fd76415d4b0 2023-11-28T20:51:49.909 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:51:49.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd74806aed0 msgr2=0x7fd74806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd74806aed0 0x7fd74806d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd758006fd0 tx=0x7fd758008040 comp rx=0 tx=0).stop 2023-11-28T20:51:49.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd76415d4b0 msgr2=0x7fd76406d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:49.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd76415d4b0 0x7fd76406d620 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fd75c037040 tx=0x7fd75c004a40 comp rx=0 tx=0).stop 2023-11-28T20:51:49.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 shutdown_connections 2023-11-28T20:51:49.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd74806aed0 0x7fd74806d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd76415d4b0 0x7fd76406d620 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7640adfa0 0x7fd76406d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 --2- 172.21.15.45:0/916359333 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7640712a0 0x7fd7641549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:49.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 >> 172.21.15.45:0/916359333 conn(0x7fd7640b1050 msgr2=0x7fd7640cd410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:49.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 shutdown_connections 2023-11-28T20:51:49.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:49.911+0000 7fd769988640 1 -- 172.21.15.45:0/916359333 wait complete. 2023-11-28T20:51:49.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:51:49.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:51:49.934 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T20:51:50.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.195+0000 7fd3f1c3d640 1 Processor -- start 2023-11-28T20:51:50.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.195+0000 7fd3f1c3d640 1 -- start start 2023-11-28T20:51:50.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.197+0000 7fd3f1c3d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 0x7fd3ec06a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.197+0000 7fd3f1c3d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd3ec0b37c0 0x7fd3ec0cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.197+0000 7fd3f1c3d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec0cfe00 con 0x7fd3ec15d500 2023-11-28T20:51:50.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.197+0000 7fd3f1c3d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ec06ac50 con 0x7fd3ec0b37c0 2023-11-28T20:51:50.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.197+0000 7fd3f1c3d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0cb830 con 0x7fd3ec06b500 2023-11-28T20:51:50.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.197+0000 7fd3ebfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd3ec0b37c0 0x7fd3ec0cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.198+0000 7fd3ebfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd3ec0b37c0 0x7fd3ec0cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:40810/0 (socket says 172.21.15.45:40810) 2023-11-28T20:51:50.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.198+0000 7fd3ebfff640 1 -- 172.21.15.45:0/2081900450 learned_addr learned my addr 172.21.15.45:0/2081900450 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:50.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.198+0000 7fd3eaffd640 1 --2- 172.21.15.45:0/2081900450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 0x7fd3ec06a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.198+0000 7fd3ebfff640 1 -- 172.21.15.45:0/2081900450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 msgr2=0x7fd3ec06a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:50.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.198+0000 7fd3ebfff640 1 --2- 172.21.15.45:0/2081900450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 0x7fd3ec06a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.198+0000 7fd3ebfff640 1 -- 172.21.15.45:0/2081900450 >> v1:172.21.15.190:6789/0 conn(0x7fd3ec15d500 legacy=0x7fd3ec06adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:50.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.199+0000 7fd3ebfff640 1 -- 172.21.15.45:0/2081900450 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec0d0050 con 0x7fd3ec0b37c0 2023-11-28T20:51:50.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.200+0000 7fd3ebfff640 1 --2- 172.21.15.45:0/2081900450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3ec0b37c0 0x7fd3ec0cbfc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009870 tx=0x7fd3e402b740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b19d2e58c629b41b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:50.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.201+0000 7fd3ea7fc640 1 -- 172.21.15.45:0/2081900450 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4030710 con 0x7fd3ec0b37c0 2023-11-28T20:51:50.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.201+0000 7fd3ea7fc640 1 -- 172.21.15.45:0/2081900450 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e402e050 con 0x7fd3ec0b37c0 2023-11-28T20:51:50.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.201+0000 7fd3ea7fc640 1 -- 172.21.15.45:0/2081900450 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4030a90 con 0x7fd3ec0b37c0 2023-11-28T20:51:50.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.202+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/2081900450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3ec0b37c0 msgr2=0x7fd3ec0cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:50.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.202+0000 7fd3f1c3d640 1 --2- 172.21.15.45:0/2081900450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3ec0b37c0 0x7fd3ec0cbfc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009870 tx=0x7fd3e402b740 comp rx=0 tx=0).stop 2023-11-28T20:51:50.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.202+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/2081900450 shutdown_connections 2023-11-28T20:51:50.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.202+0000 7fd3f1c3d640 1 --2- 172.21.15.45:0/2081900450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3ec0b37c0 0x7fd3ec0cbfc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.202+0000 7fd3f1c3d640 1 --2- 172.21.15.45:0/2081900450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 0x7fd3ec06a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.202+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/2081900450 >> 172.21.15.45:0/2081900450 conn(0x7fd3ec0b2c30 msgr2=0x7fd3ec0b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.202+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/2081900450 shutdown_connections 2023-11-28T20:51:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.202+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/2081900450 wait complete. 2023-11-28T20:51:50.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.204+0000 7fd3f1c3d640 1 Processor -- start 2023-11-28T20:51:50.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.204+0000 7fd3f1c3d640 1 -- start start 2023-11-28T20:51:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.205+0000 7fd3f1c3d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 0x7fd3ec0b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3f1c3d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3ec0b37c0 0x7fd3ec1f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3f1c3d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ec15d500 0x7fd3ec1f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3f1c3d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0c9bd0 con 0x7fd3ec15d500 2023-11-28T20:51:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3f1c3d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0c5ef0 con 0x7fd3ec0b37c0 2023-11-28T20:51:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3f1c3d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0c9050 con 0x7fd3ec06b500 2023-11-28T20:51:50.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3ebfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ec15d500 0x7fd3ec1f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3ebfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ec15d500 0x7fd3ec1f6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35890/0 (socket says 172.21.15.45:35890) 2023-11-28T20:51:50.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3ebfff640 1 -- 172.21.15.45:0/71766106 learned_addr learned my addr 172.21.15.45:0/71766106 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3eaffd640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3ec0b37c0 0x7fd3ec1f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3eb7fe640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 0x7fd3ec0b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.206+0000 7fd3ebfff640 1 -- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 msgr2=0x7fd3ec0b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3ebfff640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 0x7fd3ec0b5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3ebfff640 1 -- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3ec0b37c0 msgr2=0x7fd3ec1f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3ebfff640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3ec0b37c0 0x7fd3ec1f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3ebfff640 1 -- 172.21.15.45:0/71766106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e40094d0 con 0x7fd3ec15d500 2023-11-28T20:51:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3ebfff640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ec15d500 0x7fd3ec1f6aa0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009870 tx=0x7fd3e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3e3fff640 1 -- 172.21.15.45:0/71766106 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e403b810 con 0x7fd3ec15d500 2023-11-28T20:51:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3e3fff640 1 -- 172.21.15.45:0/71766106 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e403b990 con 0x7fd3ec15d500 2023-11-28T20:51:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3e3fff640 1 -- 172.21.15.45:0/71766106 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e403bd40 con 0x7fd3ec15d500 2023-11-28T20:51:50.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ec0c9330 con 0x7fd3ec15d500 2023-11-28T20:51:50.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.207+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ec0c98f0 con 0x7fd3ec15d500 2023-11-28T20:51:50.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.209+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ec06be50 con 0x7fd3ec15d500 2023-11-28T20:51:50.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.210+0000 7fd3e3fff640 1 -- 172.21.15.45:0/71766106 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fd3e4039040 con 0x7fd3ec15d500 2023-11-28T20:51:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.211+0000 7fd3e3fff640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd3cc06aed0 0x7fd3cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.211+0000 7fd3e3fff640 1 -- 172.21.15.45:0/71766106 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd3e40aad10 con 0x7fd3ec15d500 2023-11-28T20:51:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.211+0000 7fd3eb7fe640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd3cc06aed0 0x7fd3cc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.213+0000 7fd3eb7fe640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd3cc06aed0 0x7fd3cc06d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc007fd0 tx=0x7fd3dc009040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.215+0000 7fd3e3fff640 1 -- 172.21.15.45:0/71766106 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3e40b0020 con 0x7fd3ec15d500 2023-11-28T20:51:50.417 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:50.414+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.426+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd3ec0733d0 con 0x7fd3ec15d500 2023-11-28T20:51:50.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.427+0000 7fd3e3fff640 1 -- 172.21.15.45:0/71766106 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v112) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd3e404b090 con 0x7fd3ec15d500 2023-11-28T20:51:50.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:50.473 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:51:50.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:51:50.473 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T20:51:50.473 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T20:51:50.473 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:51:50.474 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:51:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd3cc06aed0 msgr2=0x7fd3cc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd3cc06aed0 0x7fd3cc06d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc007fd0 tx=0x7fd3dc009040 comp rx=0 tx=0).stop 2023-11-28T20:51:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ec15d500 msgr2=0x7fd3ec1f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ec15d500 0x7fd3ec1f6aa0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009870 tx=0x7fd3e40046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 shutdown_connections 2023-11-28T20:51:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd3cc06aed0 0x7fd3cc06d390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ec15d500 0x7fd3ec1f6aa0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3ec0b37c0 0x7fd3ec1f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 --2- 172.21.15.45:0/71766106 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3ec06b500 0x7fd3ec0b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 >> 172.21.15.45:0/71766106 conn(0x7fd3ec0b2c30 msgr2=0x7fd3ec0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 shutdown_connections 2023-11-28T20:51:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.430+0000 7fd3f1c3d640 1 -- 172.21.15.45:0/71766106 wait complete. 2023-11-28T20:51:50.477 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T20:51:50.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T20:51:50.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.715+0000 7fc1c5fd0640 1 Processor -- start 2023-11-28T20:51:50.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.715+0000 7fc1c5fd0640 1 -- start start 2023-11-28T20:51:50.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.717+0000 7fc1c5fd0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc1c006dc50 0x7fc1c00cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.717+0000 7fc1c5fd0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.717+0000 7fc1c5fd0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1c006aa40 0x7fc1c00c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.717+0000 7fc1c5fd0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c006aed0 con 0x7fc1c006aa40 2023-11-28T20:51:50.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.717+0000 7fc1c5fd0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c006e030 con 0x7fc1c006dc50 2023-11-28T20:51:50.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.717+0000 7fc1c5fd0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00ccf60 con 0x7fc1c00ce600 2023-11-28T20:51:50.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.717+0000 7fc1bffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1c006aa40 0x7fc1c00c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.717+0000 7fc1bffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1c006aa40 0x7fc1c00c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35906/0 (socket says 172.21.15.45:35906) 2023-11-28T20:51:50.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.717+0000 7fc1bffff640 1 -- 172.21.15.45:0/3370293283 learned_addr learned my addr 172.21.15.45:0/3370293283 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:50.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.718+0000 7fc1bf7fe640 1 --2- 172.21.15.45:0/3370293283 >> v2:172.21.15.190:3300/0 conn(0x7fc1c006dc50 0x7fc1c00cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.718+0000 7fc1beffd640 1 --2- 172.21.15.45:0/3370293283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.718+0000 7fc1bffff640 1 -- 172.21.15.45:0/3370293283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 msgr2=0x7fc1c006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:50.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.718+0000 7fc1bffff640 1 --2- 172.21.15.45:0/3370293283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c006a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.718+0000 7fc1bffff640 1 -- 172.21.15.45:0/3370293283 >> v2:172.21.15.190:3300/0 conn(0x7fc1c006dc50 msgr2=0x7fc1c00cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.718+0000 7fc1bffff640 1 --2- 172.21.15.45:0/3370293283 >> v2:172.21.15.190:3300/0 conn(0x7fc1c006dc50 0x7fc1c00cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.718+0000 7fc1bffff640 1 -- 172.21.15.45:0/3370293283 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c00cd930 con 0x7fc1c006aa40 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.718+0000 7fc1beffd640 1 --2- 172.21.15.45:0/3370293283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.718+0000 7fc1bf7fe640 1 --2- 172.21.15.45:0/3370293283 >> v2:172.21.15.190:3300/0 conn(0x7fc1c006dc50 0x7fc1c00cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.719+0000 7fc1bffff640 1 --2- 172.21.15.45:0/3370293283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 0x7fc1c00c93b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fc1c006b230 tx=0x7fc1b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e116a439a6bafc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.719+0000 7fc1be7fc640 1 -- 172.21.15.45:0/3370293283 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b802ca80 con 0x7fc1c006aa40 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.719+0000 7fc1be7fc640 1 -- 172.21.15.45:0/3370293283 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b802cbe0 con 0x7fc1c006aa40 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.719+0000 7fc1be7fc640 1 -- 172.21.15.45:0/3370293283 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80343f0 con 0x7fc1c006aa40 2023-11-28T20:51:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.720+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3370293283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 msgr2=0x7fc1c00c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.720+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3370293283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 0x7fc1c00c93b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fc1c006b230 tx=0x7fc1b802bee0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.720+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3370293283 shutdown_connections 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.720+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3370293283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 0x7fc1c00c93b0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.720+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3370293283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.720+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3370293283 >> v2:172.21.15.190:3300/0 conn(0x7fc1c006dc50 0x7fc1c00cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.720+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3370293283 >> 172.21.15.45:0/3370293283 conn(0x7fc1c00ced60 msgr2=0x7fc1c00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.720+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3370293283 shutdown_connections 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.720+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3370293283 wait complete. 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.722+0000 7fc1c5fd0640 1 Processor -- start 2023-11-28T20:51:50.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.722+0000 7fc1c5fd0640 1 -- start start 2023-11-28T20:51:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.723+0000 7fc1c5fd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 0x7fc1c015d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.723+0000 7fc1c5fd0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c006dc50 0x7fc1c00c6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.723+0000 7fc1c5fd0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c00b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.723+0000 7fc1c5fd0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00c6d50 con 0x7fc1c006aa40 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.723+0000 7fc1c5fd0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01f6c20 con 0x7fc1c006dc50 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.723+0000 7fc1c5fd0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00c3220 con 0x7fc1c00ce600 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 0x7fc1c015d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 0x7fc1c015d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35912/0 (socket says 172.21.15.45:35912) 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bf7fe640 1 -- 172.21.15.45:0/3706890360 learned_addr learned my addr 172.21.15.45:0/3706890360 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1beffd640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c006dc50 0x7fc1c00c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bffff640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c00b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bf7fe640 1 -- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 msgr2=0x7fc1c00b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bf7fe640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c00b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bf7fe640 1 -- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c006dc50 msgr2=0x7fc1c00c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bf7fe640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c006dc50 0x7fc1c00c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bf7fe640 1 -- 172.21.15.45:0/3706890360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8009b30 con 0x7fc1c006aa40 2023-11-28T20:51:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bffff640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c00b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.724+0000 7fc1bf7fe640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 0x7fc1c015d800 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c7c0 tx=0x7fc1b000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.725+0000 7fc1a7fff640 1 -- 172.21.15.45:0/3706890360 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b00079d0 con 0x7fc1c006aa40 2023-11-28T20:51:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.725+0000 7fc1a7fff640 1 -- 172.21.15.45:0/3706890360 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b0007b30 con 0x7fc1c006aa40 2023-11-28T20:51:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.725+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c00c2930 con 0x7fc1c006aa40 2023-11-28T20:51:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.725+0000 7fc1a7fff640 1 -- 172.21.15.45:0/3706890360 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b000ce70 con 0x7fc1c006aa40 2023-11-28T20:51:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.725+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c00c2eb0 con 0x7fc1c006aa40 2023-11-28T20:51:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.726+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c00733d0 con 0x7fc1c006aa40 2023-11-28T20:51:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.727+0000 7fc1a7fff640 1 -- 172.21.15.45:0/3706890360 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fc1b00113f0 con 0x7fc1c006aa40 2023-11-28T20:51:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.728+0000 7fc1a7fff640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc1ac06af00 0x7fc1ac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.728+0000 7fc1a7fff640 1 -- 172.21.15.45:0/3706890360 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fc1b0089690 con 0x7fc1c006aa40 2023-11-28T20:51:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.729+0000 7fc1beffd640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc1ac06af00 0x7fc1ac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.730+0000 7fc1beffd640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc1ac06af00 0x7fc1ac06d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400a990 tx=0x7fc1b4008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:50.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.733+0000 7fc1a7fff640 1 -- 172.21.15.45:0/3706890360 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1b0054750 con 0x7fc1c006aa40 2023-11-28T20:51:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:50.934+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc1c00b3350 con 0x7fc1c006aa40 2023-11-28T20:51:51.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.461+0000 7fc1a7fff640 1 -- 172.21.15.45:0/3706890360 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v113) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc1b0057d70 con 0x7fc1c006aa40 2023-11-28T20:51:51.463 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T20:51:51.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.466+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc1ac06af00 msgr2=0x7fc1ac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:51.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.466+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc1ac06af00 0x7fc1ac06d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400a990 tx=0x7fc1b4008040 comp rx=0 tx=0).stop 2023-11-28T20:51:51.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.466+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 msgr2=0x7fc1c015d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:51.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.466+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 0x7fc1c015d800 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c7c0 tx=0x7fc1b000cc80 comp rx=0 tx=0).stop 2023-11-28T20:51:51.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.467+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 shutdown_connections 2023-11-28T20:51:51.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.467+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc1ac06af00 0x7fc1ac06d3c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.467+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1c00ce600 0x7fc1c00b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.467+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c006dc50 0x7fc1c00c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.467+0000 7fc1c5fd0640 1 --2- 172.21.15.45:0/3706890360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c006aa40 0x7fc1c015d800 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.467+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 >> 172.21.15.45:0/3706890360 conn(0x7fc1c00ced60 msgr2=0x7fc1c00add80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:51.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.468+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 shutdown_connections 2023-11-28T20:51:51.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.468+0000 7fc1c5fd0640 1 -- 172.21.15.45:0/3706890360 wait complete. 2023-11-28T20:51:51.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T20:51:51.619 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:51.616+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:51.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.721+0000 7fde9ede3640 1 Processor -- start 2023-11-28T20:51:51.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.721+0000 7fde9ede3640 1 -- start start 2023-11-28T20:51:51.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.723+0000 7fde9ede3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fde9806bc20 0x7fde9815ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:51.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.723+0000 7fde9ede3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde980af4c0 con 0x7fde98072310 2023-11-28T20:51:51.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.723+0000 7fde9ede3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9809df30 con 0x7fde9806ef70 2023-11-28T20:51:51.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.723+0000 7fde9ede3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9815b790 con 0x7fde9806bc20 2023-11-28T20:51:51.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.723+0000 7fde9d380640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fde98072310 0x7fde9815f2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39970/0 (socket says 172.21.15.45:39970) 2023-11-28T20:51:51.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.723+0000 7fde9d380640 1 -- 172.21.15.45:0/3565303719 learned_addr learned my addr 172.21.15.45:0/3565303719 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.723+0000 7fde97fff640 1 --2- 172.21.15.45:0/3565303719 >> v2:172.21.15.190:3300/0 conn(0x7fde9806bc20 0x7fde9815ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.724+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877885671 0 0) 0x7fde980af4c0 con 0x7fde98072310 2023-11-28T20:51:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.724+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde84003520 con 0x7fde98072310 2023-11-28T20:51:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.724+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 829480306 0 0) 0x7fde84003520 con 0x7fde98072310 2023-11-28T20:51:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.724+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 >> v2:172.21.15.190:3300/0 conn(0x7fde9806bc20 msgr2=0x7fde9815ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.724+0000 7fde977fe640 1 --2- 172.21.15.45:0/3565303719 >> v2:172.21.15.190:3300/0 conn(0x7fde9806bc20 0x7fde9815ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.724+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 >> v1:172.21.15.190:6789/0 conn(0x7fde9806ef70 legacy=0x7fde9806b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.724+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde980af710 con 0x7fde98072310 2023-11-28T20:51:51.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.724+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fde90001670 con 0x7fde98072310 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 >> v1:172.21.15.45:6789/0 conn(0x7fde98072310 legacy=0x7fde9815f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde977fe640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde84003970 0x7fde84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde977fe640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde840047c0 0x7fde84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde977fe640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde84005180 0x7fde8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9815b790 con 0x7fde84005180 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8400c120 con 0x7fde84003970 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8400c2e0 con 0x7fde840047c0 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde9d380640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde84005180 0x7fde8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde9cb7f640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde84003970 0x7fde84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.725+0000 7fde97fff640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde840047c0 0x7fde84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:51.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.726+0000 7fde9d380640 1 -- 172.21.15.45:0/3565303719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde840047c0 msgr2=0x7fde84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:51.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.726+0000 7fde9d380640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde840047c0 0x7fde84004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.726+0000 7fde9d380640 1 -- 172.21.15.45:0/3565303719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde84003970 msgr2=0x7fde84003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:51.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.726+0000 7fde9d380640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde84003970 0x7fde84003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.726+0000 7fde9d380640 1 -- 172.21.15.45:0/3565303719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde84003520 con 0x7fde84005180 2023-11-28T20:51:51.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.727+0000 7fde9d380640 1 -- 172.21.15.45:0/3565303719 reap_dead start 2023-11-28T20:51:51.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.727+0000 7fde9d380640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde84005180 0x7fde8400b980 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fde9000c240 tx=0x7fde840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b6d73f0b0127d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:51.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.727+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde90005ad0 con 0x7fde84005180 2023-11-28T20:51:51.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.727+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde90005c70 con 0x7fde84005180 2023-11-28T20:51:51.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.727+0000 7fde977fe640 1 -- 172.21.15.45:0/3565303719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9002c8b0 con 0x7fde84005180 2023-11-28T20:51:51.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.728+0000 7fde9ede3640 1 -- 172.21.15.45:0/3565303719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde84005180 msgr2=0x7fde8400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:51.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.728+0000 7fde9ede3640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde84005180 0x7fde8400b980 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fde9000c240 tx=0x7fde840057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.728+0000 7fde9ede3640 1 -- 172.21.15.45:0/3565303719 shutdown_connections 2023-11-28T20:51:51.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.728+0000 7fde9ede3640 1 --2- 172.21.15.45:0/3565303719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde84005180 0x7fde8400b980 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.728+0000 7fde9ede3640 1 -- 172.21.15.45:0/3565303719 >> 172.21.15.45:0/3565303719 conn(0x7fde98071310 msgr2=0x7fde98071720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:51.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.728+0000 7fde9ede3640 1 -- 172.21.15.45:0/3565303719 shutdown_connections 2023-11-28T20:51:51.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.728+0000 7fde9ede3640 1 -- 172.21.15.45:0/3565303719 wait complete. 2023-11-28T20:51:51.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.730+0000 7fde9ede3640 1 Processor -- start 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.730+0000 7fde9ede3640 1 -- start start 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.731+0000 7fde9ede3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde84005180 0x7fde9815a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.731+0000 7fde9ede3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde98147600 0x7fde98148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.731+0000 7fde9ede3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde981490e0 0x7fde981f0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.731+0000 7fde9ede3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde98147b00 con 0x7fde98147600 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.731+0000 7fde9ede3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde98147de0 con 0x7fde981490e0 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.731+0000 7fde9ede3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde981480c0 con 0x7fde84005180 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.731+0000 7fde97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde98147600 0x7fde98148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.731+0000 7fde97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde98147600 0x7fde98148ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35936/0 (socket says 172.21.15.45:35936) 2023-11-28T20:51:51.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.731+0000 7fde97fff640 1 -- 172.21.15.45:0/2734396587 learned_addr learned my addr 172.21.15.45:0/2734396587 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.732+0000 7fde9cb7f640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde84005180 0x7fde9815a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.732+0000 7fde9d380640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde981490e0 0x7fde981f0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.732+0000 7fde97fff640 1 -- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde84005180 msgr2=0x7fde9815a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.732+0000 7fde97fff640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde84005180 0x7fde9815a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.732+0000 7fde97fff640 1 -- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde981490e0 msgr2=0x7fde981f0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.732+0000 7fde97fff640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde981490e0 0x7fde981f0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.732+0000 7fde97fff640 1 -- 172.21.15.45:0/2734396587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde981483a0 con 0x7fde98147600 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.732+0000 7fde97fff640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde98147600 0x7fde98148ba0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fde80011540 tx=0x7fde80011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.733+0000 7fde957fa640 1 -- 172.21.15.45:0/2734396587 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80015930 con 0x7fde98147600 2023-11-28T20:51:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.733+0000 7fde957fa640 1 -- 172.21.15.45:0/2734396587 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde80015a90 con 0x7fde98147600 2023-11-28T20:51:51.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.733+0000 7fde957fa640 1 -- 172.21.15.45:0/2734396587 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80015e60 con 0x7fde98147600 2023-11-28T20:51:51.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.733+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde98148680 con 0x7fde98147600 2023-11-28T20:51:51.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.733+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde981f1060 con 0x7fde98147600 2023-11-28T20:51:51.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.734+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde98148810 con 0x7fde98147600 2023-11-28T20:51:51.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.736+0000 7fde957fa640 1 -- 172.21.15.45:0/2734396587 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fde8000f460 con 0x7fde98147600 2023-11-28T20:51:51.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.737+0000 7fde957fa640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fde7806af20 0x7fde7806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:51.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.737+0000 7fde957fa640 1 -- 172.21.15.45:0/2734396587 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5547+0+0 (secure 0 0 0) 0x7fde8008a020 con 0x7fde98147600 2023-11-28T20:51:51.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.738+0000 7fde9cb7f640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fde7806af20 0x7fde7806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:51.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.739+0000 7fde9cb7f640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fde7806af20 0x7fde7806d3e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde88007fd0 tx=0x7fde88009040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:51.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.741+0000 7fde957fa640 1 -- 172.21.15.45:0/2734396587 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde80055290 con 0x7fde98147600 2023-11-28T20:51:51.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:51.945+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fde980733d0 con 0x7fde98147600 2023-11-28T20:51:52.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.468+0000 7fde957fa640 1 -- 172.21.15.45:0/2734396587 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v114) v1 ==== 170+0+0 (secure 0 0 0) 0x7fde800588b0 con 0x7fde98147600 2023-11-28T20:51:52.470 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T20:51:52.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.472+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fde7806af20 msgr2=0x7fde7806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.472+0000 7fde9ede3640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fde7806af20 0x7fde7806d3e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde88007fd0 tx=0x7fde88009040 comp rx=0 tx=0).stop 2023-11-28T20:51:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.472+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde98147600 msgr2=0x7fde98148ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.472+0000 7fde9ede3640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde98147600 0x7fde98148ba0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fde80011540 tx=0x7fde80011a00 comp rx=0 tx=0).stop 2023-11-28T20:51:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.473+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 shutdown_connections 2023-11-28T20:51:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.473+0000 7fde9ede3640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fde7806af20 0x7fde7806d3e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.473+0000 7fde9ede3640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde981490e0 0x7fde981f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.473+0000 7fde9ede3640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde98147600 0x7fde98148ba0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.473+0000 7fde9ede3640 1 --2- 172.21.15.45:0/2734396587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde84005180 0x7fde9815a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.473+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 >> 172.21.15.45:0/2734396587 conn(0x7fde98071310 msgr2=0x7fde9806b370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:52.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.473+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 shutdown_connections 2023-11-28T20:51:52.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.473+0000 7fde9ede3640 1 -- 172.21.15.45:0/2734396587 wait complete. 2023-11-28T20:51:52.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T20:51:52.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.702+0000 7fd57b388640 1 Processor -- start 2023-11-28T20:51:52.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.703+0000 7fd57b388640 1 -- start start 2023-11-28T20:51:52.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.704+0000 7fd57b388640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd5740b3040 0x7fd57406cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:52.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.704+0000 7fd57b388640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5740b3420 0x7fd5740ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:52.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.704+0000 7fd57b388640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57406d410 con 0x7fd57415b270 2023-11-28T20:51:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.704+0000 7fd57b388640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5740ceaa0 con 0x7fd5740b3040 2023-11-28T20:51:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.704+0000 7fd57b388640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57406d6f0 con 0x7fd5740b3420 2023-11-28T20:51:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.704+0000 7fd579b85640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd57415b270 0x7fd57415f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39978/0 (socket says 172.21.15.45:39978) 2023-11-28T20:51:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.704+0000 7fd579b85640 1 -- 172.21.15.45:0/249216542 learned_addr learned my addr 172.21.15.45:0/249216542 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd57a386640 1 --2- 172.21.15.45:0/249216542 >> v2:172.21.15.190:3300/0 conn(0x7fd5740b3040 0x7fd57406cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd57ab87640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5740b3420 0x7fd5740ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308520228 0 0) 0x7fd57406d410 con 0x7fd57415b270 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd55c003520 con 0x7fd57415b270 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2152765111 0 0) 0x7fd55c003520 con 0x7fd57415b270 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5740b3420 msgr2=0x7fd5740ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd579384640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5740b3420 0x7fd5740ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 >> v2:172.21.15.190:3300/0 conn(0x7fd5740b3040 msgr2=0x7fd57406cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd579384640 1 --2- 172.21.15.45:0/249216542 >> v2:172.21.15.190:3300/0 conn(0x7fd5740b3040 0x7fd57406cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57406d9d0 con 0x7fd57415b270 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.705+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd568001670 con 0x7fd57415b270 2023-11-28T20:51:52.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 >> v1:172.21.15.45:6789/0 conn(0x7fd57415b270 legacy=0x7fd57415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd579384640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd55c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd579384640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c0047c0 0x7fd55c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd579384640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5740b3420 0x7fd55c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57406d6f0 con 0x7fd55c003970 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c00b980 con 0x7fd55c0047c0 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c00bb40 con 0x7fd5740b3420 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd57a386640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd55c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd57a386640 1 -- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5740b3420 msgr2=0x7fd55c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd57a386640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5740b3420 0x7fd55c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd57a386640 1 -- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c0047c0 msgr2=0x7fd55c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd579b85640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c0047c0 0x7fd55c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:52.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd57a386640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c0047c0 0x7fd55c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.706+0000 7fd57a386640 1 -- 172.21.15.45:0/249216542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c003520 con 0x7fd55c003970 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.708+0000 7fd57a386640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd55c003d90 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fd564009ac0 tx=0x7fd55c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297779e51fea1907 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.708+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56402c710 con 0x7fd55c003970 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.708+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56402c870 con 0x7fd55c003970 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.708+0000 7fd579384640 1 -- 172.21.15.45:0/249216542 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56402cc00 con 0x7fd55c003970 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.708+0000 7fd57b388640 1 -- 172.21.15.45:0/249216542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 msgr2=0x7fd55c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.708+0000 7fd57b388640 1 --2- 172.21.15.45:0/249216542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd55c003d90 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fd564009ac0 tx=0x7fd55c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.708+0000 7fd57ab87640 1 -- 172.21.15.45:0/249216542 reap_dead start 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.709+0000 7fd57b388640 1 -- 172.21.15.45:0/249216542 shutdown_connections 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.709+0000 7fd57b388640 1 -- 172.21.15.45:0/249216542 >> 172.21.15.45:0/249216542 conn(0x7fd5740b1050 msgr2=0x7fd5740cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.709+0000 7fd57b388640 1 -- 172.21.15.45:0/249216542 shutdown_connections 2023-11-28T20:51:52.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.709+0000 7fd57b388640 1 -- 172.21.15.45:0/249216542 wait complete. 2023-11-28T20:51:52.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.711+0000 7fd57b388640 1 Processor -- start 2023-11-28T20:51:52.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.711+0000 7fd57b388640 1 -- start start 2023-11-28T20:51:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57b388640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd574153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57b388640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd574070cf0 0x7fd57415ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:52.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57b388640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd574150390 0x7fd574150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:52.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57b388640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5741542f0 con 0x7fd55c003970 2023-11-28T20:51:52.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57b388640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd574150d50 con 0x7fd574070cf0 2023-11-28T20:51:52.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57b388640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd574153770 con 0x7fd574150390 2023-11-28T20:51:52.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57a386640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd574153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:52.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57a386640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd574153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35958/0 (socket says 172.21.15.45:35958) 2023-11-28T20:51:52.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57a386640 1 -- 172.21.15.45:0/1629159931 learned_addr learned my addr 172.21.15.45:0/1629159931 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:52.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd579b85640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd574070cf0 0x7fd57415ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:52.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57ab87640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd574150390 0x7fd574150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:52.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.712+0000 7fd57a386640 1 -- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd574150390 msgr2=0x7fd574150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:52.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.713+0000 7fd57a386640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd574150390 0x7fd574150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.713+0000 7fd57a386640 1 -- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd574070cf0 msgr2=0x7fd57415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:52.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.713+0000 7fd57a386640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd574070cf0 0x7fd57415ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:52.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.713+0000 7fd57a386640 1 -- 172.21.15.45:0/1629159931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd574153a50 con 0x7fd55c003970 2023-11-28T20:51:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.713+0000 7fd57a386640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd574153140 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fd56402c2b0 tx=0x7fd564004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.713+0000 7fd572ffd640 1 -- 172.21.15.45:0/1629159931 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564003810 con 0x7fd55c003970 2023-11-28T20:51:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.713+0000 7fd572ffd640 1 -- 172.21.15.45:0/1629159931 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56402de90 con 0x7fd55c003970 2023-11-28T20:51:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.713+0000 7fd572ffd640 1 -- 172.21.15.45:0/1629159931 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564003bb0 con 0x7fd55c003970 2023-11-28T20:51:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.713+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd574153d30 con 0x7fd55c003970 2023-11-28T20:51:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.714+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57409ae40 con 0x7fd55c003970 2023-11-28T20:51:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.715+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57409d0f0 con 0x7fd55c003970 2023-11-28T20:51:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.716+0000 7fd572ffd640 1 -- 172.21.15.45:0/1629159931 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fd564003d10 con 0x7fd55c003970 2023-11-28T20:51:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.717+0000 7fd572ffd640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd56006af00 0x7fd56006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.717+0000 7fd572ffd640 1 -- 172.21.15.45:0/1629159931 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fd5640ab680 con 0x7fd55c003970 2023-11-28T20:51:52.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.718+0000 7fd579b85640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd56006af00 0x7fd56006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:52.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.719+0000 7fd579b85640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd56006af00 0x7fd56006d3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd568000c00 tx=0x7fd568009040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:52.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.722+0000 7fd572ffd640 1 -- 172.21.15.45:0/1629159931 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5640765f0 con 0x7fd55c003970 2023-11-28T20:51:52.806 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:52.803+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:52.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:52.967+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd5740b2c10 con 0x7fd55c003970 2023-11-28T20:51:53.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.524+0000 7fd572ffd640 1 -- 172.21.15.45:0/1629159931 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v48) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd564079c10 con 0x7fd55c003970 2023-11-28T20:51:53.527 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 13 and data pool 14 2023-11-28T20:51:53.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.532+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd56006af00 msgr2=0x7fd56006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:53.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.532+0000 7fd57b388640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd56006af00 0x7fd56006d3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd568000c00 tx=0x7fd568009040 comp rx=0 tx=0).stop 2023-11-28T20:51:53.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.532+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 msgr2=0x7fd574153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:53.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.532+0000 7fd57b388640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd574153140 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fd56402c2b0 tx=0x7fd564004730 comp rx=0 tx=0).stop 2023-11-28T20:51:53.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.536+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 shutdown_connections 2023-11-28T20:51:53.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.536+0000 7fd57b388640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd56006af00 0x7fd56006d3c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.536+0000 7fd57b388640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd574150390 0x7fd574150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.536+0000 7fd57b388640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd574070cf0 0x7fd57415ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.536+0000 7fd57b388640 1 --2- 172.21.15.45:0/1629159931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd55c003970 0x7fd574153140 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.536+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 >> 172.21.15.45:0/1629159931 conn(0x7fd5740b1050 msgr2=0x7fd5740af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:53.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.537+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 shutdown_connections 2023-11-28T20:51:53.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.537+0000 7fd57b388640 1 -- 172.21.15.45:0/1629159931 wait complete. 2023-11-28T20:51:53.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:53.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.759+0000 7f0f152ef640 1 Processor -- start 2023-11-28T20:51:53.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.759+0000 7f0f152ef640 1 -- start start 2023-11-28T20:51:53.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.760+0000 7f0f152ef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f100b3870 0x7f0f1015db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:53.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.760+0000 7f0f152ef640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0f100721e0 0x7f0f1015e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:53.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.760+0000 7f0f152ef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1015e680 con 0x7f0f1006d170 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.760+0000 7f0f152ef640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1015a460 con 0x7f0f100721e0 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.760+0000 7f0f152ef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1015e960 con 0x7f0f100b3870 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.761+0000 7f0f14aee640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0f1006d170 0x7f0f100701c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39990/0 (socket says 172.21.15.45:39990) 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.761+0000 7f0f14aee640 1 -- 172.21.15.45:0/1466573313 learned_addr learned my addr 172.21.15.45:0/1466573313 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.761+0000 7f0f0f7fe640 1 --2- 172.21.15.45:0/1466573313 >> v2:172.21.15.190:3300/0 conn(0x7f0f100721e0 0x7f0f1015e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.761+0000 7f0f0ffff640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f100b3870 0x7f0f1015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.761+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001474454 0 0) 0x7f0f1015e680 con 0x7f0f1006d170 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.761+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0efc003520 con 0x7f0f1006d170 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1981823532 0 0) 0x7f0efc003520 con 0x7f0f1006d170 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f100b3870 msgr2=0x7f0f1015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f100b3870 0x7f0f1015db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 >> v2:172.21.15.190:3300/0 conn(0x7f0f100721e0 msgr2=0x7f0f1015e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:53.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 --2- 172.21.15.45:0/1466573313 >> v2:172.21.15.190:3300/0 conn(0x7f0f100721e0 0x7f0f1015e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f10071700 con 0x7f0f1006d170 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f0f08001670 con 0x7f0f1006d170 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 >> v1:172.21.15.45:6789/0 conn(0x7f0f1006d170 legacy=0x7f0f100701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f100b3870 0x7f0efc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc003ef0 0x7f0efc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0efc004d00 0x7f0efc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1015e960 con 0x7f0efc003ef0 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc00b980 con 0x7f0efc004d00 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc00bac0 con 0x7f0f100b3870 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.762+0000 7f0f0f7fe640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc003ef0 0x7f0efc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.763+0000 7f0f0f7fe640 1 -- 172.21.15.45:0/1466573313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f100b3870 msgr2=0x7f0efc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:53.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.763+0000 7f0f0f7fe640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f100b3870 0x7f0efc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.763+0000 7f0f0f7fe640 1 -- 172.21.15.45:0/1466573313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0efc004d00 msgr2=0x7f0efc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:53.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.763+0000 7f0f0f7fe640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0efc004d00 0x7f0efc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.763+0000 7f0f0f7fe640 1 -- 172.21.15.45:0/1466573313 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc003520 con 0x7f0efc003ef0 2023-11-28T20:51:53.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.764+0000 7f0f0f7fe640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc003ef0 0x7f0efc0047c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009650 tx=0x7f0efc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3c376157c04ef8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:53.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.764+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef802e8e0 con 0x7f0efc003ef0 2023-11-28T20:51:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.764+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef802ea40 con 0x7f0efc003ef0 2023-11-28T20:51:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.764+0000 7f0f0effd640 1 -- 172.21.15.45:0/1466573313 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef802edd0 con 0x7f0efc003ef0 2023-11-28T20:51:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.765+0000 7f0f152ef640 1 -- 172.21.15.45:0/1466573313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc003ef0 msgr2=0x7f0efc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.765+0000 7f0f152ef640 1 --2- 172.21.15.45:0/1466573313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc003ef0 0x7f0efc0047c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009650 tx=0x7f0efc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.765+0000 7f0f14aee640 1 -- 172.21.15.45:0/1466573313 reap_dead start 2023-11-28T20:51:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.765+0000 7f0f152ef640 1 -- 172.21.15.45:0/1466573313 shutdown_connections 2023-11-28T20:51:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.765+0000 7f0f152ef640 1 -- 172.21.15.45:0/1466573313 >> 172.21.15.45:0/1466573313 conn(0x7f0f1006ab30 msgr2=0x7f0f100b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.765+0000 7f0f152ef640 1 -- 172.21.15.45:0/1466573313 shutdown_connections 2023-11-28T20:51:53.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.765+0000 7f0f152ef640 1 -- 172.21.15.45:0/1466573313 wait complete. 2023-11-28T20:51:53.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.767+0000 7f0f152ef640 1 Processor -- start 2023-11-28T20:51:53.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.767+0000 7f0f152ef640 1 -- start start 2023-11-28T20:51:53.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.768+0000 7f0f152ef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0efc003ef0 0x7f0f1009afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.768+0000 7f0f152ef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f1014fbf0 0x7f0f101584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.768+0000 7f0f152ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f1009c650 0x7f0f1009cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.768+0000 7f0f152ef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1009c1c0 con 0x7f0f1009c650 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.768+0000 7f0f152ef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1009b5b0 con 0x7f0efc003ef0 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.768+0000 7f0f152ef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1009b890 con 0x7f0f1014fbf0 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.769+0000 7f0f14aee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f1009c650 0x7f0f1009cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.769+0000 7f0f14aee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f1009c650 0x7f0f1009cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35972/0 (socket says 172.21.15.45:35972) 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.769+0000 7f0f14aee640 1 -- 172.21.15.45:0/768632681 learned_addr learned my addr 172.21.15.45:0/768632681 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.769+0000 7f0f0f7fe640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f1014fbf0 0x7f0f101584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.769+0000 7f0f14aee640 1 -- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f1014fbf0 msgr2=0x7f0f101584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.769+0000 7f0f14aee640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f1014fbf0 0x7f0f101584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.769+0000 7f0f14aee640 1 -- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0efc003ef0 msgr2=0x7f0f1009afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:53.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.769+0000 7f0f14aee640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0efc003ef0 0x7f0f1009afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.769+0000 7f0f14aee640 1 -- 172.21.15.45:0/768632681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1009bb70 con 0x7f0f1009c650 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.770+0000 7f0f14aee640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f1009c650 0x7f0f1009cad0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f0f1015eaa0 tx=0x7f0f0800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.770+0000 7f0f0cff9640 1 -- 172.21.15.45:0/768632681 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08007440 con 0x7f0f1009c650 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.770+0000 7f0f0cff9640 1 -- 172.21.15.45:0/768632681 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f080075a0 con 0x7f0f1009c650 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.770+0000 7f0f0cff9640 1 -- 172.21.15.45:0/768632681 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08004a20 con 0x7f0f1009c650 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.770+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1009be50 con 0x7f0f1009c650 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.770+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f10158a30 con 0x7f0f1009c650 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.771+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f10066f40 con 0x7f0f1009c650 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.772+0000 7f0f0cff9640 1 -- 172.21.15.45:0/768632681 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f0f08004b80 con 0x7f0f1009c650 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.773+0000 7f0f0cff9640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f0ef006af00 0x7f0ef006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:53.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.773+0000 7f0f0cff9640 1 -- 172.21.15.45:0/768632681 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0f08088990 con 0x7f0f1009c650 2023-11-28T20:51:53.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.774+0000 7f0f0ffff640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f0ef006af00 0x7f0ef006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:53.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.775+0000 7f0f0ffff640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f0ef006af00 0x7f0ef006d3c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0f000070b0 tx=0x7f0f0000d040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:53.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.778+0000 7f0f0cff9640 1 -- 172.21.15.45:0/768632681 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f08053910 con 0x7f0f1009c650 2023-11-28T20:51:53.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.982+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f10158bc0 con 0x7f0f1009c650 2023-11-28T20:51:53.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.985+0000 7f0f0cff9640 1 -- 172.21.15.45:0/768632681 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f0f08056f30 con 0x7f0f1009c650 2023-11-28T20:51:53.986 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:53.986 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":115,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:52.969270+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:50.993341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:52.004867+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.987+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f0ef006af00 msgr2=0x7f0ef006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.987+0000 7f0f152ef640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f0ef006af00 0x7f0ef006d3c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0f000070b0 tx=0x7f0f0000d040 comp rx=0 tx=0).stop 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.987+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f1009c650 msgr2=0x7f0f1009cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.987+0000 7f0f152ef640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f1009c650 0x7f0f1009cad0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f0f1015eaa0 tx=0x7f0f0800c960 comp rx=0 tx=0).stop 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.987+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 shutdown_connections 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.987+0000 7f0f152ef640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f0ef006af00 0x7f0ef006d3c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.987+0000 7f0f152ef640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f1009c650 0x7f0f1009cad0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.987+0000 7f0f152ef640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f1014fbf0 0x7f0f101584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.987+0000 7f0f152ef640 1 --2- 172.21.15.45:0/768632681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0efc003ef0 0x7f0f1009afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.988+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 >> 172.21.15.45:0/768632681 conn(0x7f0f1006ab30 msgr2=0x7f0f100b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.988+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 shutdown_connections 2023-11-28T20:51:53.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:53.988+0000 7f0f152ef640 1 -- 172.21.15.45:0/768632681 wait complete. 2023-11-28T20:51:54.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:54.021 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:54.018+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:54.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.227+0000 7f7880bdb640 1 Processor -- start 2023-11-28T20:51:54.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.228+0000 7f7880bdb640 1 -- start start 2023-11-28T20:51:54.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.229+0000 7f7880bdb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c0cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.229+0000 7f7880bdb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f787c0d1c80 0x7f787c0b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.229+0000 7f7880bdb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c0d0560 con 0x7f787c0d0100 2023-11-28T20:51:54.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.229+0000 7f7880bdb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f787c0d2230 con 0x7f787c0d1c80 2023-11-28T20:51:54.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.229+0000 7f7880bdb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787c0d2510 con 0x7f787c15d550 2023-11-28T20:51:54.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.230+0000 7f787b7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c0cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.230+0000 7f787b7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c0cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:49872/0 (socket says 172.21.15.45:49872) 2023-11-28T20:51:54.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.230+0000 7f787b7fe640 1 -- 172.21.15.45:0/1891125811 learned_addr learned my addr 172.21.15.45:0/1891125811 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:54.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.230+0000 7f787bfff640 1 --2- 172.21.15.45:0/1891125811 >> v2:172.21.15.190:3300/0 conn(0x7f787c0d1c80 0x7f787c0b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.231+0000 7f787bfff640 1 -- 172.21.15.45:0/1891125811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 msgr2=0x7f787c0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:54.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.231+0000 7f787bfff640 1 --2- 172.21.15.45:0/1891125811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c0cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.231+0000 7f787bfff640 1 -- 172.21.15.45:0/1891125811 >> v1:172.21.15.190:6789/0 conn(0x7f787c0d0100 legacy=0x7f787c0d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.231+0000 7f787bfff640 1 -- 172.21.15.45:0/1891125811 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787c0d27f0 con 0x7f787c0d1c80 2023-11-28T20:51:54.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.231+0000 7f787b7fe640 1 --2- 172.21.15.45:0/1891125811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:51:54.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.232+0000 7f787bfff640 1 --2- 172.21.15.45:0/1891125811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d1c80 0x7f787c0b6f50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7874009d90 tx=0x7f787402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3656b34092ca8d2e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:54.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.232+0000 7f787affd640 1 -- 172.21.15.45:0/1891125811 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787402cc60 con 0x7f787c0d1c80 2023-11-28T20:51:54.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.232+0000 7f787affd640 1 -- 172.21.15.45:0/1891125811 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787402cdc0 con 0x7f787c0d1c80 2023-11-28T20:51:54.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.232+0000 7f787affd640 1 -- 172.21.15.45:0/1891125811 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7874034660 con 0x7f787c0d1c80 2023-11-28T20:51:54.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.233+0000 7f7880bdb640 1 -- 172.21.15.45:0/1891125811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d1c80 msgr2=0x7f787c0b6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.233+0000 7f7880bdb640 1 --2- 172.21.15.45:0/1891125811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d1c80 0x7f787c0b6f50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7874009d90 tx=0x7f787402beb0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.233+0000 7f7880bdb640 1 -- 172.21.15.45:0/1891125811 shutdown_connections 2023-11-28T20:51:54.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.233+0000 7f7880bdb640 1 --2- 172.21.15.45:0/1891125811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d1c80 0x7f787c0b6f50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.233+0000 7f7880bdb640 1 --2- 172.21.15.45:0/1891125811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.233+0000 7f7880bdb640 1 -- 172.21.15.45:0/1891125811 >> 172.21.15.45:0/1891125811 conn(0x7f787c06ac70 msgr2=0x7f787c0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:54.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.233+0000 7f7880bdb640 1 -- 172.21.15.45:0/1891125811 shutdown_connections 2023-11-28T20:51:54.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.233+0000 7f7880bdb640 1 -- 172.21.15.45:0/1891125811 wait complete. 2023-11-28T20:51:54.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.235+0000 7f7880bdb640 1 Processor -- start 2023-11-28T20:51:54.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.235+0000 7f7880bdb640 1 -- start start 2023-11-28T20:51:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.236+0000 7f7880bdb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d0100 0x7f787c149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.236+0000 7f7880bdb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0d1c80 0x7f787c0c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.236+0000 7f7880bdb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.236+0000 7f7880bdb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c157940 con 0x7f787c0d1c80 2023-11-28T20:51:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.236+0000 7f7880bdb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c152b50 con 0x7f787c0d0100 2023-11-28T20:51:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.236+0000 7f7880bdb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787c156d30 con 0x7f787c15d550 2023-11-28T20:51:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f7872ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0d1c80 0x7f787c0c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f7872ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0d1c80 0x7f787c0c8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52464/0 (socket says 172.21.15.45:52464) 2023-11-28T20:51:54.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f7872ffd640 1 -- 172.21.15.45:0/474138349 learned_addr learned my addr 172.21.15.45:0/474138349 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f787b7fe640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d0100 0x7f787c149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f787bfff640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f7872ffd640 1 -- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 msgr2=0x7f787c157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f7872ffd640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f7872ffd640 1 -- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d0100 msgr2=0x7f787c149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f7872ffd640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d0100 0x7f787c149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f7872ffd640 1 -- 172.21.15.45:0/474138349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c0094d0 con 0x7f787c0d1c80 2023-11-28T20:51:54.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.237+0000 7f787b7fe640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d0100 0x7f787c149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:54.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.238+0000 7f7872ffd640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0d1c80 0x7f787c0c8f10 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f786400c3f0 tx=0x7f786400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:54.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.238+0000 7f7878ff9640 1 -- 172.21.15.45:0/474138349 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78640052d0 con 0x7f787c0d1c80 2023-11-28T20:51:54.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.238+0000 7f7878ff9640 1 -- 172.21.15.45:0/474138349 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7864005450 con 0x7f787c0d1c80 2023-11-28T20:51:54.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.238+0000 7f7878ff9640 1 -- 172.21.15.45:0/474138349 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864007260 con 0x7f787c0d1c80 2023-11-28T20:51:54.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.238+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7874009a70 con 0x7f787c0d1c80 2023-11-28T20:51:54.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.238+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787c1572f0 con 0x7f787c0d1c80 2023-11-28T20:51:54.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.239+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787c04e920 con 0x7f787c0d1c80 2023-11-28T20:51:54.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.240+0000 7f7878ff9640 1 -- 172.21.15.45:0/474138349 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f78640073c0 con 0x7f787c0d1c80 2023-11-28T20:51:54.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.242+0000 7f7878ff9640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f785c06aed0 0x7f785c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.242+0000 7f7878ff9640 1 -- 172.21.15.45:0/474138349 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7864088bf0 con 0x7f787c0d1c80 2023-11-28T20:51:54.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.243+0000 7f787b7fe640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f785c06aed0 0x7f785c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.244+0000 7f787b7fe640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f785c06aed0 0x7f785c06d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f786c009db0 tx=0x7f786c008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:54.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.247+0000 7f7878ff9640 1 -- 172.21.15.45:0/474138349 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7864053a70 con 0x7f787c0d1c80 2023-11-28T20:51:54.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.455+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f787c0c6a20 con 0x7f787c0d1c80 2023-11-28T20:51:54.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.457+0000 7f7878ff9640 1 -- 172.21.15.45:0/474138349 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f7864057090 con 0x7f787c0d1c80 2023-11-28T20:51:54.458 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:54.460 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":115,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:52.969270+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:50.993341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:52.004867+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f785c06aed0 msgr2=0x7f785c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f785c06aed0 0x7f785c06d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f786c009db0 tx=0x7f786c008040 comp rx=0 tx=0).stop 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0d1c80 msgr2=0x7f787c0c8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0d1c80 0x7f787c0c8f10 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f786400c3f0 tx=0x7f786400c8b0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 shutdown_connections 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f785c06aed0 0x7f785c06d390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c15d550 0x7f787c157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0d1c80 0x7f787c0c8f10 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 --2- 172.21.15.45:0/474138349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c0d0100 0x7f787c149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 >> 172.21.15.45:0/474138349 conn(0x7f787c06ac70 msgr2=0x7f787c15e630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.460+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 shutdown_connections 2023-11-28T20:51:54.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.461+0000 7f7880bdb640 1 -- 172.21.15.45:0/474138349 wait complete. 2023-11-28T20:51:54.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T20:51:54.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.712+0000 7f6cf0c48640 1 Processor -- start 2023-11-28T20:51:54.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.712+0000 7f6cf0c48640 1 -- start start 2023-11-28T20:51:54.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.713+0000 7f6cf0c48640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6cec0cf820 0x7f6cec099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.713+0000 7f6cf0c48640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6cec099f80 0x7f6cec0b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.713+0000 7f6cf0c48640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec0b0970 con 0x7f6cec15b600 2023-11-28T20:51:54.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.713+0000 7f6cf0c48640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec0cc5c0 con 0x7f6cec0cf820 2023-11-28T20:51:54.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.713+0000 7f6cf0c48640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec0cfc80 con 0x7f6cec099f80 2023-11-28T20:51:54.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.713+0000 7f6cea59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6cec15b600 0x7f6cec0cbeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44140/0 (socket says 172.21.15.45:44140) 2023-11-28T20:51:54.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.713+0000 7f6cea59c640 1 -- 172.21.15.45:0/2176451620 learned_addr learned my addr 172.21.15.45:0/2176451620 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:54.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.713+0000 7f6ce3fff640 1 --2- 172.21.15.45:0/2176451620 >> v2:172.21.15.45:3300/0 conn(0x7f6cec0cf820 0x7f6cec099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.714+0000 7f6cead9d640 1 --2- 172.21.15.45:0/2176451620 >> v2:172.21.15.190:3300/0 conn(0x7f6cec099f80 0x7f6cec0b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.714+0000 7f6ce3fff640 1 -- 172.21.15.45:0/2176451620 >> v2:172.21.15.190:3300/0 conn(0x7f6cec099f80 msgr2=0x7f6cec0b13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:54.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.714+0000 7f6ce3fff640 1 --2- 172.21.15.45:0/2176451620 >> v2:172.21.15.190:3300/0 conn(0x7f6cec099f80 0x7f6cec0b13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.714+0000 7f6ce3fff640 1 -- 172.21.15.45:0/2176451620 >> v1:172.21.15.45:6789/0 conn(0x7f6cec15b600 legacy=0x7f6cec0cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.714+0000 7f6ce3fff640 1 -- 172.21.15.45:0/2176451620 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0b0f30 con 0x7f6cec0cf820 2023-11-28T20:51:54.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.714+0000 7f6ce9d9b640 1 -- 172.21.15.45:0/2176451620 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729900828 0 0) 0x7f6cec0b0970 con 0x7f6cec15b600 2023-11-28T20:51:54.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.714+0000 7f6cead9d640 1 --2- 172.21.15.45:0/2176451620 >> v2:172.21.15.190:3300/0 conn(0x7f6cec099f80 0x7f6cec0b13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:54.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.715+0000 7f6ce3fff640 1 --2- 172.21.15.45:0/2176451620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec0cf820 0x7f6cec099a40 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f6cec071b30 tx=0x7f6cd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2b413444e1778c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:54.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.716+0000 7f6ce9d9b640 1 -- 172.21.15.45:0/2176451620 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4031830 con 0x7f6cec0cf820 2023-11-28T20:51:54.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.716+0000 7f6ce9d9b640 1 -- 172.21.15.45:0/2176451620 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4031990 con 0x7f6cec0cf820 2023-11-28T20:51:54.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.716+0000 7f6ce9d9b640 1 -- 172.21.15.45:0/2176451620 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4031d20 con 0x7f6cec0cf820 2023-11-28T20:51:54.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.717+0000 7f6cf0c48640 1 -- 172.21.15.45:0/2176451620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec0cf820 msgr2=0x7f6cec099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.717+0000 7f6cf0c48640 1 --2- 172.21.15.45:0/2176451620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec0cf820 0x7f6cec099a40 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f6cec071b30 tx=0x7f6cd402b8d0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.717+0000 7f6cf0c48640 1 -- 172.21.15.45:0/2176451620 shutdown_connections 2023-11-28T20:51:54.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.717+0000 7f6cf0c48640 1 --2- 172.21.15.45:0/2176451620 >> v2:172.21.15.190:3300/0 conn(0x7f6cec099f80 0x7f6cec0b13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.717+0000 7f6cf0c48640 1 --2- 172.21.15.45:0/2176451620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec0cf820 0x7f6cec099a40 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.717+0000 7f6cf0c48640 1 -- 172.21.15.45:0/2176451620 >> 172.21.15.45:0/2176451620 conn(0x7f6cec0b2b20 msgr2=0x7f6cec0b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:54.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.717+0000 7f6cf0c48640 1 -- 172.21.15.45:0/2176451620 shutdown_connections 2023-11-28T20:51:54.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.717+0000 7f6cf0c48640 1 -- 172.21.15.45:0/2176451620 wait complete. 2023-11-28T20:51:54.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.719+0000 7f6cf0c48640 1 Processor -- start 2023-11-28T20:51:54.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.719+0000 7f6cf0c48640 1 -- start start 2023-11-28T20:51:54.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.720+0000 7f6cf0c48640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cec099f80 0x7f6cec14b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.720+0000 7f6cf0c48640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cec0cf820 0x7f6cec0c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.720+0000 7f6cf0c48640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec15b600 0x7f6cec0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.720+0000 7f6cf0c48640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0c5b90 con 0x7f6cec15b600 2023-11-28T20:51:54.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.720+0000 7f6cf0c48640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0c4dc0 con 0x7f6cec0cf820 2023-11-28T20:51:54.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.720+0000 7f6cf0c48640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0c5010 con 0x7f6cec099f80 2023-11-28T20:51:54.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.720+0000 7f6cead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec15b600 0x7f6cec0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.720+0000 7f6cead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec15b600 0x7f6cec0ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52472/0 (socket says 172.21.15.45:52472) 2023-11-28T20:51:54.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.720+0000 7f6cead9d640 1 -- 172.21.15.45:0/3183251249 learned_addr learned my addr 172.21.15.45:0/3183251249 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.721+0000 7f6ce3fff640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cec0cf820 0x7f6cec0c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.721+0000 7f6cead9d640 1 -- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cec099f80 msgr2=0x7f6cec14b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.721+0000 7f6cead9d640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cec099f80 0x7f6cec14b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.721+0000 7f6cead9d640 1 -- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cec0cf820 msgr2=0x7f6cec0c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.721+0000 7f6cead9d640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cec0cf820 0x7f6cec0c8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.721+0000 7f6cead9d640 1 -- 172.21.15.45:0/3183251249 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd4009400 con 0x7f6cec15b600 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.721+0000 7f6cead9d640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec15b600 0x7f6cec0ccd00 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400cd50 tx=0x7f6ce400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.721+0000 7f6ce2ffd640 1 -- 172.21.15.45:0/3183251249 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce400f680 con 0x7f6cec15b600 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.722+0000 7f6ce2ffd640 1 -- 172.21.15.45:0/3183251249 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce400f800 con 0x7f6cec15b600 2023-11-28T20:51:54.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.722+0000 7f6ce2ffd640 1 -- 172.21.15.45:0/3183251249 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce400fbb0 con 0x7f6cec15b600 2023-11-28T20:51:54.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.722+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec0c52f0 con 0x7f6cec15b600 2023-11-28T20:51:54.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.722+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec0c58b0 con 0x7f6cec15b600 2023-11-28T20:51:54.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.723+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec0733d0 con 0x7f6cec15b600 2023-11-28T20:51:54.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.724+0000 7f6ce2ffd640 1 -- 172.21.15.45:0/3183251249 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f6ce400b040 con 0x7f6cec15b600 2023-11-28T20:51:54.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.725+0000 7f6ce2ffd640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6ccc06aed0 0x7f6ccc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:54.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.725+0000 7f6ce2ffd640 1 -- 172.21.15.45:0/3183251249 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6ce408a290 con 0x7f6cec15b600 2023-11-28T20:51:54.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.726+0000 7f6cea59c640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6ccc06aed0 0x7f6ccc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:54.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.727+0000 7f6cea59c640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6ccc06aed0 0x7f6ccc06d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc007fd0 tx=0x7f6cdc007920 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:54.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.730+0000 7f6ce2ffd640 1 -- 172.21.15.45:0/3183251249 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ce4055100 con 0x7f6cec15b600 2023-11-28T20:51:54.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:54.965+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6cec0b6fb0 con 0x7f6cec15b600 2023-11-28T20:51:55.248 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:55.245+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:55.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.536+0000 7f6ce2ffd640 1 -- 172.21.15.45:0/3183251249 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6ce4058720 con 0x7f6cec15b600 2023-11-28T20:51:55.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6ccc06aed0 msgr2=0x7f6ccc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6ccc06aed0 0x7f6ccc06d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc007fd0 tx=0x7f6cdc007920 comp rx=0 tx=0).stop 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec15b600 msgr2=0x7f6cec0ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec15b600 0x7f6cec0ccd00 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400cd50 tx=0x7f6ce400a650 comp rx=0 tx=0).stop 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 shutdown_connections 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6ccc06aed0 0x7f6ccc06d390 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cec15b600 0x7f6cec0ccd00 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6cec0cf820 0x7f6cec0c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 --2- 172.21.15.45:0/3183251249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6cec099f80 0x7f6cec14b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 >> 172.21.15.45:0/3183251249 conn(0x7f6cec0b2b20 msgr2=0x7f6cec0b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.540+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 shutdown_connections 2023-11-28T20:51:55.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.541+0000 7f6cf0c48640 1 -- 172.21.15.45:0/3183251249 wait complete. 2023-11-28T20:51:55.563 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T20:51:55.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T20:51:55.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.788+0000 7fef9aca6640 1 Processor -- start 2023-11-28T20:51:55.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.788+0000 7fef9aca6640 1 -- start start 2023-11-28T20:51:55.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.789+0000 7fef9aca6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fef9415d420 0x7fef94161a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:55.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.789+0000 7fef9aca6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef940d0240 0x7fef940d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:55.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.789+0000 7fef9aca6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef94161fd0 con 0x7fef940cda70 2023-11-28T20:51:55.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.789+0000 7fef9aca6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef940cdf00 con 0x7fef940d0240 2023-11-28T20:51:55.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.789+0000 7fef9aca6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef940ce1e0 con 0x7fef9415d420 2023-11-28T20:51:55.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.789+0000 7fef99243640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef940d0240 0x7fef940d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:55.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.790+0000 7fef99243640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fef940d0240 0x7fef940d1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52476/0 (socket says 172.21.15.45:52476) 2023-11-28T20:51:55.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.790+0000 7fef99243640 1 -- 172.21.15.45:0/3339154939 learned_addr learned my addr 172.21.15.45:0/3339154939 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:55.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.790+0000 7fef98a42640 1 --2- 172.21.15.45:0/3339154939 >> v2:172.21.15.190:3300/0 conn(0x7fef9415d420 0x7fef94161a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:55.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.790+0000 7fef99243640 1 -- 172.21.15.45:0/3339154939 >> v2:172.21.15.190:3300/0 conn(0x7fef9415d420 msgr2=0x7fef94161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:55.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.790+0000 7fef99243640 1 --2- 172.21.15.45:0/3339154939 >> v2:172.21.15.190:3300/0 conn(0x7fef9415d420 0x7fef94161a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:55.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.790+0000 7fef99243640 1 -- 172.21.15.45:0/3339154939 >> v1:172.21.15.190:6789/0 conn(0x7fef940cda70 legacy=0x7fef940cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:55.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.790+0000 7fef99243640 1 -- 172.21.15.45:0/3339154939 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef940ce4c0 con 0x7fef940d0240 2023-11-28T20:51:55.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.790+0000 7fef98a42640 1 --2- 172.21.15.45:0/3339154939 >> v2:172.21.15.190:3300/0 conn(0x7fef9415d420 0x7fef94161a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.791+0000 7fef99243640 1 --2- 172.21.15.45:0/3339154939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef940d0240 0x7fef940d1aa0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fef94067220 tx=0x7fef8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c2ee0d9e2b1905 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.791+0000 7fef937fe640 1 -- 172.21.15.45:0/3339154939 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c031870 con 0x7fef940d0240 2023-11-28T20:51:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef937fe640 1 -- 172.21.15.45:0/3339154939 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c0319d0 con 0x7fef940d0240 2023-11-28T20:51:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef937fe640 1 -- 172.21.15.45:0/3339154939 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c031d60 con 0x7fef940d0240 2023-11-28T20:51:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef9aca6640 1 -- 172.21.15.45:0/3339154939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef940d0240 msgr2=0x7fef940d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef9aca6640 1 --2- 172.21.15.45:0/3339154939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef940d0240 0x7fef940d1aa0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fef94067220 tx=0x7fef8c02b910 comp rx=0 tx=0).stop 2023-11-28T20:51:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef9aca6640 1 -- 172.21.15.45:0/3339154939 shutdown_connections 2023-11-28T20:51:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef9aca6640 1 --2- 172.21.15.45:0/3339154939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef940d0240 0x7fef940d1aa0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef9aca6640 1 --2- 172.21.15.45:0/3339154939 >> v2:172.21.15.190:3300/0 conn(0x7fef9415d420 0x7fef94161a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef9aca6640 1 -- 172.21.15.45:0/3339154939 >> 172.21.15.45:0/3339154939 conn(0x7fef94070b20 msgr2=0x7fef94070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef9aca6640 1 -- 172.21.15.45:0/3339154939 shutdown_connections 2023-11-28T20:51:55.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.792+0000 7fef9aca6640 1 -- 172.21.15.45:0/3339154939 wait complete. 2023-11-28T20:51:55.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.794+0000 7fef9aca6640 1 Processor -- start 2023-11-28T20:51:55.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.794+0000 7fef9aca6640 1 -- start start 2023-11-28T20:51:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.795+0000 7fef9aca6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef940d0240 0x7fef940cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.795+0000 7fef9aca6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9415d420 0x7fef940cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.795+0000 7fef9aca6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef941518b0 0x7fef9415dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.795+0000 7fef9aca6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9415f610 con 0x7fef941518b0 2023-11-28T20:51:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.795+0000 7fef9aca6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9415e990 con 0x7fef940d0240 2023-11-28T20:51:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.795+0000 7fef9aca6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9415ec70 con 0x7fef9415d420 2023-11-28T20:51:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef99243640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef941518b0 0x7fef9415dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef99243640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef941518b0 0x7fef9415dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52490/0 (socket says 172.21.15.45:52490) 2023-11-28T20:51:55.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef99243640 1 -- 172.21.15.45:0/2738934062 learned_addr learned my addr 172.21.15.45:0/2738934062 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef98a42640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef940d0240 0x7fef940cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef93fff640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9415d420 0x7fef940cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef99243640 1 -- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9415d420 msgr2=0x7fef940cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef99243640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9415d420 0x7fef940cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef99243640 1 -- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef940d0240 msgr2=0x7fef940cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef99243640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef940d0240 0x7fef940cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.796+0000 7fef99243640 1 -- 172.21.15.45:0/2738934062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c0094d0 con 0x7fef941518b0 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.797+0000 7fef99243640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef941518b0 0x7fef9415dc40 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0097c0 tx=0x7fef8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.797+0000 7fef917fa640 1 -- 172.21.15.45:0/2738934062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c03add0 con 0x7fef941518b0 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.797+0000 7fef917fa640 1 -- 172.21.15.45:0/2738934062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c0443f0 con 0x7fef941518b0 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.797+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9415ef50 con 0x7fef941518b0 2023-11-28T20:51:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.797+0000 7fef917fa640 1 -- 172.21.15.45:0/2738934062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0447a0 con 0x7fef941518b0 2023-11-28T20:51:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.797+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9415e180 con 0x7fef941518b0 2023-11-28T20:51:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.799+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef940733d0 con 0x7fef941518b0 2023-11-28T20:51:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.799+0000 7fef917fa640 1 -- 172.21.15.45:0/2738934062 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fef8c04d450 con 0x7fef941518b0 2023-11-28T20:51:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.800+0000 7fef917fa640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fef8006aed0 0x7fef8006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.800+0000 7fef917fa640 1 -- 172.21.15.45:0/2738934062 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fef8c02d040 con 0x7fef941518b0 2023-11-28T20:51:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.801+0000 7fef98a42640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fef8006aed0 0x7fef8006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.802+0000 7fef98a42640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fef8006aed0 0x7fef8006d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fef84007510 tx=0x7fef840074a0 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:55.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:55.805+0000 7fef917fa640 1 -- 172.21.15.45:0/2738934062 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef8c076e60 con 0x7fef941518b0 2023-11-28T20:51:56.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.054+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fef9415c3b0 con 0x7fef941518b0 2023-11-28T20:51:56.496 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:56.492+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:56.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.539+0000 7fef917fa640 1 -- 172.21.15.45:0/2738934062 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7fef8c035020 con 0x7fef941518b0 2023-11-28T20:51:56.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.542+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fef8006aed0 msgr2=0x7fef8006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:56.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.542+0000 7fef9aca6640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fef8006aed0 0x7fef8006d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fef84007510 tx=0x7fef840074a0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.542+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef941518b0 msgr2=0x7fef9415dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:56.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.542+0000 7fef9aca6640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef941518b0 0x7fef9415dc40 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0097c0 tx=0x7fef8c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.543+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 shutdown_connections 2023-11-28T20:51:56.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.543+0000 7fef9aca6640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fef8006aed0 0x7fef8006d390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.543+0000 7fef9aca6640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef941518b0 0x7fef9415dc40 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.543+0000 7fef9aca6640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef9415d420 0x7fef940cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.543+0000 7fef9aca6640 1 --2- 172.21.15.45:0/2738934062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef940d0240 0x7fef940cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.543+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 >> 172.21.15.45:0/2738934062 conn(0x7fef94070b20 msgr2=0x7fef94070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:56.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.543+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 shutdown_connections 2023-11-28T20:51:56.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.543+0000 7fef9aca6640 1 -- 172.21.15.45:0/2738934062 wait complete. 2023-11-28T20:51:56.567 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T20:51:56.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:56.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.788+0000 7f37ab49b640 1 Processor -- start 2023-11-28T20:51:56.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.788+0000 7f37ab49b640 1 -- start start 2023-11-28T20:51:56.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.790+0000 7f37ab49b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f37a4160e70 0x7f37a40b6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:56.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.790+0000 7f37ab49b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 0x7f37a40d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:56.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.790+0000 7f37ab49b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a406aa70 con 0x7f37a406a500 2023-11-28T20:51:56.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.790+0000 7f37ab49b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a406ad50 con 0x7f37a4160e70 2023-11-28T20:51:56.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.790+0000 7f37ab49b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a406b030 con 0x7f37a4071230 2023-11-28T20:51:56.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.790+0000 7f37a9237640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f37a4160e70 0x7f37a40b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:56.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.790+0000 7f37a9a38640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 0x7f37a40d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:56.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.790+0000 7f37a9a38640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 0x7f37a40d1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:49918/0 (socket says 172.21.15.45:49918) 2023-11-28T20:51:56.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.790+0000 7f37a9a38640 1 -- 172.21.15.45:0/3402492869 learned_addr learned my addr 172.21.15.45:0/3402492869 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:56.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.791+0000 7f37a9237640 1 -- 172.21.15.45:0/3402492869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 msgr2=0x7f37a40d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:56.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.791+0000 7f37a9237640 1 --2- 172.21.15.45:0/3402492869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 0x7f37a40d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.791+0000 7f37a9237640 1 -- 172.21.15.45:0/3402492869 >> v1:172.21.15.190:6789/0 conn(0x7f37a406a500 legacy=0x7f37a4070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:56.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.791+0000 7f37a9237640 1 -- 172.21.15.45:0/3402492869 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a406b310 con 0x7f37a4160e70 2023-11-28T20:51:56.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.793+0000 7f37a9237640 1 --2- 172.21.15.45:0/3402492869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37a4160e70 0x7f37a40b6940 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009d90 tx=0x7f37a002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2be2fb2d562593f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:56.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.793+0000 7f379bfff640 1 -- 172.21.15.45:0/3402492869 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a002cc50 con 0x7f37a4160e70 2023-11-28T20:51:56.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.793+0000 7f379bfff640 1 -- 172.21.15.45:0/3402492869 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a002cdb0 con 0x7f37a4160e70 2023-11-28T20:51:56.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.793+0000 7f379bfff640 1 -- 172.21.15.45:0/3402492869 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0034660 con 0x7f37a4160e70 2023-11-28T20:51:56.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.794+0000 7f37ab49b640 1 -- 172.21.15.45:0/3402492869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37a4160e70 msgr2=0x7f37a40b6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:56.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.794+0000 7f37ab49b640 1 --2- 172.21.15.45:0/3402492869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37a4160e70 0x7f37a40b6940 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009d90 tx=0x7f37a002bea0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.794+0000 7f37ab49b640 1 -- 172.21.15.45:0/3402492869 shutdown_connections 2023-11-28T20:51:56.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.794+0000 7f37ab49b640 1 --2- 172.21.15.45:0/3402492869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 0x7f37a40d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.794+0000 7f37ab49b640 1 --2- 172.21.15.45:0/3402492869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37a4160e70 0x7f37a40b6940 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.794+0000 7f37ab49b640 1 -- 172.21.15.45:0/3402492869 >> 172.21.15.45:0/3402492869 conn(0x7f37a40cfc60 msgr2=0x7f37a40d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:56.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.794+0000 7f37ab49b640 1 -- 172.21.15.45:0/3402492869 shutdown_connections 2023-11-28T20:51:56.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.794+0000 7f37ab49b640 1 -- 172.21.15.45:0/3402492869 wait complete. 2023-11-28T20:51:56.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.796+0000 7f37ab49b640 1 Processor -- start 2023-11-28T20:51:56.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.796+0000 7f37ab49b640 1 -- start start 2023-11-28T20:51:56.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.797+0000 7f37ab49b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37a406a500 0x7f37a415a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.797+0000 7f37ab49b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 0x7f37a414b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.797+0000 7f37ab49b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a4160e70 0x7f37a41f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.797+0000 7f37ab49b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4148ea0 con 0x7f37a4160e70 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.797+0000 7f37ab49b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a41f6e10 con 0x7f37a406a500 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.797+0000 7f37ab49b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a415a970 con 0x7f37a4071230 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a9a38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a4160e70 0x7f37a41f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a9a38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a4160e70 0x7f37a41f72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52502/0 (socket says 172.21.15.45:52502) 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a9a38640 1 -- 172.21.15.45:0/3177812197 learned_addr learned my addr 172.21.15.45:0/3177812197 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a9237640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37a406a500 0x7f37a415a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a8a36640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 0x7f37a414b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:56.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a9a38640 1 -- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 msgr2=0x7f37a414b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:56.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a9a38640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 0x7f37a414b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a9a38640 1 -- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37a406a500 msgr2=0x7f37a415a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:56.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a9a38640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37a406a500 0x7f37a415a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:56.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.798+0000 7f37a9a38640 1 -- 172.21.15.45:0/3177812197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a0009a70 con 0x7f37a4160e70 2023-11-28T20:51:56.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.799+0000 7f37a9a38640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a4160e70 0x7f37a41f72a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f379c00e840 tx=0x7f379c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:56.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.799+0000 7f3799ffb640 1 -- 172.21.15.45:0/3177812197 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c009660 con 0x7f37a4160e70 2023-11-28T20:51:56.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.799+0000 7f3799ffb640 1 -- 172.21.15.45:0/3177812197 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c00c940 con 0x7f37a4160e70 2023-11-28T20:51:56.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.799+0000 7f3799ffb640 1 -- 172.21.15.45:0/3177812197 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c00cd10 con 0x7f37a4160e70 2023-11-28T20:51:56.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.799+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a41f6ae0 con 0x7f37a4160e70 2023-11-28T20:51:56.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.799+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a414bd00 con 0x7f37a4160e70 2023-11-28T20:51:56.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.801+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a40733d0 con 0x7f37a4160e70 2023-11-28T20:51:56.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.801+0000 7f3799ffb640 1 -- 172.21.15.45:0/3177812197 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f379c013470 con 0x7f37a4160e70 2023-11-28T20:51:56.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.802+0000 7f3799ffb640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f378806af00 0x7f378806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:56.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.803+0000 7f3799ffb640 1 -- 172.21.15.45:0/3177812197 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f379c089d10 con 0x7f37a4160e70 2023-11-28T20:51:56.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.803+0000 7f37a9237640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f378806af00 0x7f378806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:56.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.804+0000 7f37a9237640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f378806af00 0x7f378806d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37a0036000 tx=0x7f37a0030000 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:56.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:56.807+0000 7f3799ffb640 1 -- 172.21.15.45:0/3177812197 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f379c08f020 con 0x7f37a4160e70 2023-11-28T20:51:57.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.054+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37a41f6f50 con 0x7f37a4160e70 2023-11-28T20:51:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.056+0000 7f3799ffb640 1 -- 172.21.15.45:0/3177812197 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f379c017090 con 0x7f37a4160e70 2023-11-28T20:51:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:57.057 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2348343744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2348343744},{"type":"v1","addr":"172.21.15.190:6836","nonce":2348343744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6439,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1864791063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":1864791063},{"type":"v1","addr":"172.21.15.190:6837","nonce":1864791063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3003510210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3003510210},{"type":"v1","addr":"172.21.15.45:6835","nonce":3003510210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:52.969776+0000","modified":"2023-11-28T20:51:56.055296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6448":{"gid":6448,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2489162496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2489162496},{"type":"v1","addr":"172.21.15.45:6834","nonce":2489162496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-28T20:51:57.058 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 51 2023-11-28T20:51:57.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f378806af00 msgr2=0x7f378806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f378806af00 0x7f378806d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37a0036000 tx=0x7f37a0030000 comp rx=0 tx=0).stop 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a4160e70 msgr2=0x7f37a41f72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a4160e70 0x7f37a41f72a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f379c00e840 tx=0x7f379c00ed00 comp rx=0 tx=0).stop 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 shutdown_connections 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f378806af00 0x7f378806d3c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a4160e70 0x7f37a41f72a0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f37a4071230 0x7f37a414b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 --2- 172.21.15.45:0/3177812197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37a406a500 0x7f37a415a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 >> 172.21.15.45:0/3177812197 conn(0x7f37a40cfc60 msgr2=0x7f37a40d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 shutdown_connections 2023-11-28T20:51:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.059+0000 7f37ab49b640 1 -- 172.21.15.45:0/3177812197 wait complete. 2023-11-28T20:51:57.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:51:57.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.303+0000 7f12abfff640 1 Processor -- start 2023-11-28T20:51:57.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.304+0000 7f12abfff640 1 -- start start 2023-11-28T20:51:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.305+0000 7f12abfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12ac14b090 0x7f12ac0b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.305+0000 7f12abfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f12ac0ae590 0x7f12ac0b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.305+0000 7f12abfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f12ac06c060 0x7f12ac165250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.305+0000 7f12abfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12ac0abf10 con 0x7f12ac06c060 2023-11-28T20:51:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.305+0000 7f12abfff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12ac06db90 con 0x7f12ac0ae590 2023-11-28T20:51:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.305+0000 7f12abfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac06d2f0 con 0x7f12ac14b090 2023-11-28T20:51:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12ab7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f12ac06c060 0x7f12ac165250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12ab7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f12ac06c060 0x7f12ac165250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52510/0 (socket says 172.21.15.45:52510) 2023-11-28T20:51:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12ab7fe640 1 -- 172.21.15.45:0/2040922369 learned_addr learned my addr 172.21.15.45:0/2040922369 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12aa7fc640 1 --2- 172.21.15.45:0/2040922369 >> v2:172.21.15.190:3300/0 conn(0x7f12ac0ae590 0x7f12ac0b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12aaffd640 1 --2- 172.21.15.45:0/2040922369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12ac14b090 0x7f12ac0b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12ab7fe640 1 -- 172.21.15.45:0/2040922369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12ac14b090 msgr2=0x7f12ac0b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12ab7fe640 1 --2- 172.21.15.45:0/2040922369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12ac14b090 0x7f12ac0b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12ab7fe640 1 -- 172.21.15.45:0/2040922369 >> v2:172.21.15.190:3300/0 conn(0x7f12ac0ae590 msgr2=0x7f12ac0b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12ab7fe640 1 --2- 172.21.15.45:0/2040922369 >> v2:172.21.15.190:3300/0 conn(0x7f12ac0ae590 0x7f12ac0b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.306+0000 7f12ab7fe640 1 -- 172.21.15.45:0/2040922369 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac06d5d0 con 0x7f12ac06c060 2023-11-28T20:51:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.308+0000 7f12ab7fe640 1 --2- 172.21.15.45:0/2040922369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 0x7f12ac165250 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f12ac06b410 tx=0x7f12a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63907f7db940463 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.308+0000 7f12a9ffb640 1 -- 172.21.15.45:0/2040922369 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a402ca70 con 0x7f12ac06c060 2023-11-28T20:51:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.308+0000 7f12a9ffb640 1 -- 172.21.15.45:0/2040922369 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a402cbd0 con 0x7f12ac06c060 2023-11-28T20:51:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.308+0000 7f12a9ffb640 1 -- 172.21.15.45:0/2040922369 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40343f0 con 0x7f12ac06c060 2023-11-28T20:51:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.309+0000 7f12abfff640 1 -- 172.21.15.45:0/2040922369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 msgr2=0x7f12ac165250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.309+0000 7f12abfff640 1 --2- 172.21.15.45:0/2040922369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 0x7f12ac165250 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f12ac06b410 tx=0x7f12a402bed0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.309+0000 7f12abfff640 1 -- 172.21.15.45:0/2040922369 shutdown_connections 2023-11-28T20:51:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.309+0000 7f12abfff640 1 --2- 172.21.15.45:0/2040922369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 0x7f12ac165250 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.309+0000 7f12abfff640 1 --2- 172.21.15.45:0/2040922369 >> v2:172.21.15.190:3300/0 conn(0x7f12ac0ae590 0x7f12ac0b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.309+0000 7f12abfff640 1 --2- 172.21.15.45:0/2040922369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12ac14b090 0x7f12ac0b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.309+0000 7f12abfff640 1 -- 172.21.15.45:0/2040922369 >> 172.21.15.45:0/2040922369 conn(0x7f12ac06ab30 msgr2=0x7f12ac0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.309+0000 7f12abfff640 1 -- 172.21.15.45:0/2040922369 shutdown_connections 2023-11-28T20:51:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.309+0000 7f12abfff640 1 -- 172.21.15.45:0/2040922369 wait complete. 2023-11-28T20:51:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.311+0000 7f12abfff640 1 Processor -- start 2023-11-28T20:51:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.311+0000 7f12abfff640 1 -- start start 2023-11-28T20:51:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12abfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 0x7f12ac146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12abfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac0ae590 0x7f12ac1ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12abfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12ac14b090 0x7f12ac149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12abfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac1630b0 con 0x7f12ac06c060 2023-11-28T20:51:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12abfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac1ed620 con 0x7f12ac0ae590 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12abfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac1ed900 con 0x7f12ac14b090 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12aaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 0x7f12ac146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12aaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 0x7f12ac146020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52518/0 (socket says 172.21.15.45:52518) 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12aaffd640 1 -- 172.21.15.45:0/4158558133 learned_addr learned my addr 172.21.15.45:0/4158558133 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.312+0000 7f12aa7fc640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac0ae590 0x7f12ac1ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12aaffd640 1 -- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12ac14b090 msgr2=0x7f12ac149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12aaffd640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12ac14b090 0x7f12ac149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12aaffd640 1 -- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac0ae590 msgr2=0x7f12ac1ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12aaffd640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac0ae590 0x7f12ac1ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12aaffd640 1 -- 172.21.15.45:0/4158558133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a4009b30 con 0x7f12ac06c060 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12aaffd640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 0x7f12ac146020 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f129c00d5f0 tx=0x7f129c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12937fe640 1 -- 172.21.15.45:0/4158558133 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c0125f0 con 0x7f12ac06c060 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12937fe640 1 -- 172.21.15.45:0/4158558133 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c012750 con 0x7f12ac06c060 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac1edbe0 con 0x7f12ac06c060 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12937fe640 1 -- 172.21.15.45:0/4158558133 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c012b00 con 0x7f12ac06c060 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.313+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ac146560 con 0x7f12ac06c060 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.315+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ac1eddd0 con 0x7f12ac06c060 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.316+0000 7f12937fe640 1 -- 172.21.15.45:0/4158558133 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f129c012c60 con 0x7f12ac06c060 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.317+0000 7f12937fe640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f129806aed0 0x7f129806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.317+0000 7f12937fe640 1 -- 172.21.15.45:0/4158558133 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f129c089d90 con 0x7f12ac06c060 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.318+0000 7f12aa7fc640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f129806aed0 0x7f129806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.319+0000 7f12aa7fc640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f129806aed0 0x7f129806d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0ace50 tx=0x7f12a0008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:57.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.322+0000 7f12937fe640 1 -- 172.21.15.45:0/4158558133 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f129c054b80 con 0x7f12ac06c060 2023-11-28T20:51:57.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.520+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12ac09d0f0 con 0x7f12ac06c060 2023-11-28T20:51:57.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.523+0000 7f12937fe640 1 -- 172.21.15.45:0/4158558133 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16716 (secure 0 0 0) 0x7f129c0581a0 con 0x7f12ac06c060 2023-11-28T20:51:57.525 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:57.525 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":116,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:51:54.522974+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:50.993341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:52.004867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:51:57.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.525+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f129806aed0 msgr2=0x7f129806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.525+0000 7f12abfff640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f129806aed0 0x7f129806d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0ace50 tx=0x7f12a0008040 comp rx=0 tx=0).stop 2023-11-28T20:51:57.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.525+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 msgr2=0x7f12ac146020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.525+0000 7f12abfff640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 0x7f12ac146020 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f129c00d5f0 tx=0x7f129c00dab0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.526+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 shutdown_connections 2023-11-28T20:51:57.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.526+0000 7f12abfff640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f129806aed0 0x7f129806d390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.526+0000 7f12abfff640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12ac14b090 0x7f12ac149070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.526+0000 7f12abfff640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac0ae590 0x7f12ac1ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.526+0000 7f12abfff640 1 --2- 172.21.15.45:0/4158558133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12ac06c060 0x7f12ac146020 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.526+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 >> 172.21.15.45:0/4158558133 conn(0x7f12ac06ab30 msgr2=0x7f12ac0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:57.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.526+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 shutdown_connections 2023-11-28T20:51:57.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.526+0000 7f12abfff640 1 -- 172.21.15.45:0/4158558133 wait complete. 2023-11-28T20:51:57.548 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T20:51:57.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:51:57.701 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:57.698+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:57.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.762+0000 7fc81b810640 1 Processor -- start 2023-11-28T20:51:57.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.763+0000 7fc81b810640 1 -- start start 2023-11-28T20:51:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.764+0000 7fc81b810640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8141402f0 0x7fc814160e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.764+0000 7fc81b810640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc8141434c0 0x7fc8141409f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.764+0000 7fc81b810640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc814161df0 con 0x7fc814140f30 2023-11-28T20:51:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.764+0000 7fc81b810640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8141406d0 con 0x7fc8141402f0 2023-11-28T20:51:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.764+0000 7fc81b810640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc814143950 con 0x7fc8141434c0 2023-11-28T20:51:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.765+0000 7fc8195ac640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8141402f0 0x7fc814160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.765+0000 7fc8195ac640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc8141402f0 0x7fc814160e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52522/0 (socket says 172.21.15.45:52522) 2023-11-28T20:51:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.765+0000 7fc8195ac640 1 -- 172.21.15.45:0/3088704885 learned_addr learned my addr 172.21.15.45:0/3088704885 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.765+0000 7fc818dab640 1 --2- 172.21.15.45:0/3088704885 >> v2:172.21.15.190:3300/0 conn(0x7fc8141434c0 0x7fc8141409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.765+0000 7fc8195ac640 1 -- 172.21.15.45:0/3088704885 >> v2:172.21.15.190:3300/0 conn(0x7fc8141434c0 msgr2=0x7fc8141409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.765+0000 7fc8195ac640 1 --2- 172.21.15.45:0/3088704885 >> v2:172.21.15.190:3300/0 conn(0x7fc8141434c0 0x7fc8141409f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.765+0000 7fc8195ac640 1 -- 172.21.15.45:0/3088704885 >> v1:172.21.15.190:6789/0 conn(0x7fc814140f30 legacy=0x7fc8140703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.765+0000 7fc8195ac640 1 -- 172.21.15.45:0/3088704885 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc814161550 con 0x7fc8141402f0 2023-11-28T20:51:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.765+0000 7fc818dab640 1 --2- 172.21.15.45:0/3088704885 >> v2:172.21.15.190:3300/0 conn(0x7fc8141434c0 0x7fc8141409f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.766+0000 7fc8195ac640 1 --2- 172.21.15.45:0/3088704885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 0x7fc814160e70 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fc8140b3d50 tx=0x7fc81002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9341ccf33d14da5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.767+0000 7fc80bfff640 1 -- 172.21.15.45:0/3088704885 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8100317e0 con 0x7fc8141402f0 2023-11-28T20:51:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.767+0000 7fc80bfff640 1 -- 172.21.15.45:0/3088704885 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc810031940 con 0x7fc8141402f0 2023-11-28T20:51:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.767+0000 7fc80bfff640 1 -- 172.21.15.45:0/3088704885 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810031cd0 con 0x7fc8141402f0 2023-11-28T20:51:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.767+0000 7fc81b810640 1 -- 172.21.15.45:0/3088704885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 msgr2=0x7fc814160e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.767+0000 7fc81b810640 1 --2- 172.21.15.45:0/3088704885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 0x7fc814160e70 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fc8140b3d50 tx=0x7fc81002b880 comp rx=0 tx=0).stop 2023-11-28T20:51:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.767+0000 7fc81b810640 1 -- 172.21.15.45:0/3088704885 shutdown_connections 2023-11-28T20:51:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.767+0000 7fc81b810640 1 --2- 172.21.15.45:0/3088704885 >> v2:172.21.15.190:3300/0 conn(0x7fc8141434c0 0x7fc8141409f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.767+0000 7fc81b810640 1 --2- 172.21.15.45:0/3088704885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 0x7fc814160e70 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.767+0000 7fc81b810640 1 -- 172.21.15.45:0/3088704885 >> 172.21.15.45:0/3088704885 conn(0x7fc81406e7c0 msgr2=0x7fc81406ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.768+0000 7fc81b810640 1 -- 172.21.15.45:0/3088704885 shutdown_connections 2023-11-28T20:51:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.768+0000 7fc81b810640 1 -- 172.21.15.45:0/3088704885 wait complete. 2023-11-28T20:51:57.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.769+0000 7fc81b810640 1 Processor -- start 2023-11-28T20:51:57.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.769+0000 7fc81b810640 1 -- start start 2023-11-28T20:51:57.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.770+0000 7fc81b810640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 0x7fc8140ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.770+0000 7fc81b810640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814140f30 0x7fc81415fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.770+0000 7fc81b810640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8141434c0 0x7fc81413e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.770+0000 7fc81b810640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8140cb2f0 con 0x7fc8141402f0 2023-11-28T20:51:57.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.770+0000 7fc81b810640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8140cb5d0 con 0x7fc814140f30 2023-11-28T20:51:57.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.770+0000 7fc81b810640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8140cb8b0 con 0x7fc8141434c0 2023-11-28T20:51:57.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.771+0000 7fc8195ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 0x7fc8140ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.771+0000 7fc8195ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 0x7fc8140ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52534/0 (socket says 172.21.15.45:52534) 2023-11-28T20:51:57.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.771+0000 7fc8195ac640 1 -- 172.21.15.45:0/1087831032 learned_addr learned my addr 172.21.15.45:0/1087831032 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:57.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.771+0000 7fc818dab640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814140f30 0x7fc81415fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.771+0000 7fc8195ac640 1 -- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8141434c0 msgr2=0x7fc81413e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.771+0000 7fc8195ac640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8141434c0 0x7fc81413e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.771+0000 7fc8195ac640 1 -- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814140f30 msgr2=0x7fc81415fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:57.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.771+0000 7fc8195ac640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814140f30 0x7fc81415fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:57.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.771+0000 7fc8195ac640 1 -- 172.21.15.45:0/1087831032 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8100094d0 con 0x7fc8141402f0 2023-11-28T20:51:57.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.772+0000 7fc8195ac640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 0x7fc8140ccce0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fc8100090b0 tx=0x7fc8100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:57.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.772+0000 7fc809ffb640 1 -- 172.21.15.45:0/1087831032 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810032980 con 0x7fc8141402f0 2023-11-28T20:51:57.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.772+0000 7fc809ffb640 1 -- 172.21.15.45:0/1087831032 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc810032b00 con 0x7fc8141402f0 2023-11-28T20:51:57.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.772+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8140cbb90 con 0x7fc8141402f0 2023-11-28T20:51:57.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.772+0000 7fc809ffb640 1 -- 172.21.15.45:0/1087831032 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8100453f0 con 0x7fc8141402f0 2023-11-28T20:51:57.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.772+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8140cd220 con 0x7fc8141402f0 2023-11-28T20:51:57.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.773+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8140733d0 con 0x7fc8141402f0 2023-11-28T20:51:57.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.774+0000 7fc809ffb640 1 -- 172.21.15.45:0/1087831032 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fc81004d3f0 con 0x7fc8141402f0 2023-11-28T20:51:57.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.775+0000 7fc809ffb640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc7f806af00 0x7fc7f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:57.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.775+0000 7fc809ffb640 1 -- 172.21.15.45:0/1087831032 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc8100ab750 con 0x7fc8141402f0 2023-11-28T20:51:57.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.776+0000 7fc818dab640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc7f806af00 0x7fc7f806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:57.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.777+0000 7fc818dab640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc7f806af00 0x7fc7f806d3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc804006fd0 tx=0x7fc804008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:57.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.780+0000 7fc809ffb640 1 -- 172.21.15.45:0/1087831032 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8100764c0 con 0x7fc8141402f0 2023-11-28T20:51:57.895 INFO:tasks.ceph.mds.a.smithi045.stderr:2023-11-28T20:51:57.894+0000 7f30d6d6f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:51:57.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:57.985+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 --> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc81406fe90 con 0x7fc7f806af00 2023-11-28T20:51:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.019+0000 7fc809ffb640 1 -- 172.21.15.45:0/1087831032 <== mgr.5828 v2:172.21.15.190:6832/4159844301 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286978 (secure 0 0 0) 0x7fc80400c040 con 0x7fc7f806af00 2023-11-28T20:51:58.028 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:58.030 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:51:58.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc7f806af00 msgr2=0x7fc7f806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc7f806af00 0x7fc7f806d3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc804006fd0 tx=0x7fc804008040 comp rx=0 tx=0).stop 2023-11-28T20:51:58.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 msgr2=0x7fc8140ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 0x7fc8140ccce0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fc8100090b0 tx=0x7fc8100046c0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 shutdown_connections 2023-11-28T20:51:58.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fc7f806af00 0x7fc7f806d3c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8141434c0 0x7fc81413e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814140f30 0x7fc81415fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 --2- 172.21.15.45:0/1087831032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8141402f0 0x7fc8140ccce0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 >> 172.21.15.45:0/1087831032 conn(0x7fc81406e7c0 msgr2=0x7fc8140d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:58.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 shutdown_connections 2023-11-28T20:51:58.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.027+0000 7fc81b810640 1 -- 172.21.15.45:0/1087831032 wait complete. 2023-11-28T20:51:58.051 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-11-28T20:51:57.907751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":710,"num_read_kb":1360,"num_write":315,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4324,"ondisk_log_size":4324,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":357500,"kb_used_data":12296,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754617220,"statfs":{"total":773094113280,"available":772728033280,"internally_reserved":0,"allocated":12591104,"data_stored":6891564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":353357360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002160"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.553028+0000","last_change":"2023-11-28T20:51:53.544286+0000","last_active":"2023-11-28T20:51:53.553028+0000","last_peered":"2023-11-28T20:51:53.553028+0000","last_clean":"2023-11-28T20:51:53.553028+0000","last_became_active":"2023-11-28T20:51:53.543924+0000","last_became_peered":"2023-11-28T20:51:53.543924+0000","last_unstale":"2023-11-28T20:51:53.553028+0000","last_undegraded":"2023-11-28T20:51:53.553028+0000","last_fullsized":"2023-11-28T20:51:53.553028+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:44:45.759419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544909+0000","last_change":"2023-11-28T20:51:53.546281+0000","last_active":"2023-11-28T20:51:54.544909+0000","last_peered":"2023-11-28T20:51:54.544909+0000","last_clean":"2023-11-28T20:51:54.544909+0000","last_became_active":"2023-11-28T20:51:53.545147+0000","last_became_peered":"2023-11-28T20:51:53.545147+0000","last_unstale":"2023-11-28T20:51:54.544909+0000","last_undegraded":"2023-11-28T20:51:54.544909+0000","last_fullsized":"2023-11-28T20:51:54.544909+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:44:45.759419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.542606+0000","last_change":"2023-11-28T20:51:53.540245+0000","last_active":"2023-11-28T20:51:53.542606+0000","last_peered":"2023-11-28T20:51:53.542606+0000","last_clean":"2023-11-28T20:51:53.542606+0000","last_became_active":"2023-11-28T20:51:53.539525+0000","last_became_peered":"2023-11-28T20:51:53.539525+0000","last_unstale":"2023-11-28T20:51:53.542606+0000","last_undegraded":"2023-11-28T20:51:53.542606+0000","last_fullsized":"2023-11-28T20:51:53.542606+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:57:38.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.552628+0000","last_change":"2023-11-28T20:51:53.537281+0000","last_active":"2023-11-28T20:51:53.552628+0000","last_peered":"2023-11-28T20:51:53.552628+0000","last_clean":"2023-11-28T20:51:53.552628+0000","last_became_active":"2023-11-28T20:51:53.536974+0000","last_became_peered":"2023-11-28T20:51:53.536974+0000","last_unstale":"2023-11-28T20:51:53.552628+0000","last_undegraded":"2023-11-28T20:51:53.552628+0000","last_fullsized":"2023-11-28T20:51:53.552628+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:44.399354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543053+0000","last_change":"2023-11-28T20:51:53.534749+0000","last_active":"2023-11-28T20:51:54.543053+0000","last_peered":"2023-11-28T20:51:54.543053+0000","last_clean":"2023-11-28T20:51:54.543053+0000","last_became_active":"2023-11-28T20:51:53.534175+0000","last_became_peered":"2023-11-28T20:51:53.534175+0000","last_unstale":"2023-11-28T20:51:54.543053+0000","last_undegraded":"2023-11-28T20:51:54.543053+0000","last_fullsized":"2023-11-28T20:51:54.543053+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:13.224682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.549213+0000","last_change":"2023-11-28T20:51:53.546009+0000","last_active":"2023-11-28T20:51:53.549213+0000","last_peered":"2023-11-28T20:51:53.549213+0000","last_clean":"2023-11-28T20:51:53.549213+0000","last_became_active":"2023-11-28T20:51:53.545720+0000","last_became_peered":"2023-11-28T20:51:53.545720+0000","last_unstale":"2023-11-28T20:51:53.549213+0000","last_undegraded":"2023-11-28T20:51:53.549213+0000","last_fullsized":"2023-11-28T20:51:53.549213+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:06.029430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.544998+0000","last_change":"2023-11-28T20:51:53.542990+0000","last_active":"2023-11-28T20:51:53.544998+0000","last_peered":"2023-11-28T20:51:53.544998+0000","last_clean":"2023-11-28T20:51:53.544998+0000","last_became_active":"2023-11-28T20:51:53.542642+0000","last_became_peered":"2023-11-28T20:51:53.542642+0000","last_unstale":"2023-11-28T20:51:53.544998+0000","last_undegraded":"2023-11-28T20:51:53.544998+0000","last_fullsized":"2023-11-28T20:51:53.544998+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:44:45.759419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.545781+0000","last_change":"2023-11-28T20:51:53.531982+0000","last_active":"2023-11-28T20:51:53.545781+0000","last_peered":"2023-11-28T20:51:53.545781+0000","last_clean":"2023-11-28T20:51:53.545781+0000","last_became_active":"2023-11-28T20:51:53.531534+0000","last_became_peered":"2023-11-28T20:51:53.531534+0000","last_unstale":"2023-11-28T20:51:53.545781+0000","last_undegraded":"2023-11-28T20:51:53.545781+0000","last_fullsized":"2023-11-28T20:51:53.545781+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:42.893430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544402+0000","last_change":"2023-11-28T20:51:53.530638+0000","last_active":"2023-11-28T20:51:54.544402+0000","last_peered":"2023-11-28T20:51:54.544402+0000","last_clean":"2023-11-28T20:51:54.544402+0000","last_became_active":"2023-11-28T20:51:53.528389+0000","last_became_peered":"2023-11-28T20:51:53.528389+0000","last_unstale":"2023-11-28T20:51:54.544402+0000","last_undegraded":"2023-11-28T20:51:54.544402+0000","last_fullsized":"2023-11-28T20:51:54.544402+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:49:07.458753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535709+0000","last_change":"2023-11-28T20:51:53.531803+0000","last_active":"2023-11-28T20:51:54.535709+0000","last_peered":"2023-11-28T20:51:54.535709+0000","last_clean":"2023-11-28T20:51:54.535709+0000","last_became_active":"2023-11-28T20:51:53.531478+0000","last_became_peered":"2023-11-28T20:51:53.531478+0000","last_unstale":"2023-11-28T20:51:54.535709+0000","last_undegraded":"2023-11-28T20:51:54.535709+0000","last_fullsized":"2023-11-28T20:51:54.535709+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:17.150958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542913+0000","last_change":"2023-11-28T20:51:53.541664+0000","last_active":"2023-11-28T20:51:54.542913+0000","last_peered":"2023-11-28T20:51:54.542913+0000","last_clean":"2023-11-28T20:51:54.542913+0000","last_became_active":"2023-11-28T20:51:53.541376+0000","last_became_peered":"2023-11-28T20:51:53.541376+0000","last_unstale":"2023-11-28T20:51:54.542913+0000","last_undegraded":"2023-11-28T20:51:54.542913+0000","last_fullsized":"2023-11-28T20:51:54.542913+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:21:07.082883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535286+0000","last_change":"2023-11-28T20:51:53.544372+0000","last_active":"2023-11-28T20:51:54.535286+0000","last_peered":"2023-11-28T20:51:54.535286+0000","last_clean":"2023-11-28T20:51:54.535286+0000","last_became_active":"2023-11-28T20:51:53.544089+0000","last_became_peered":"2023-11-28T20:51:53.544089+0000","last_unstale":"2023-11-28T20:51:54.535286+0000","last_undegraded":"2023-11-28T20:51:54.535286+0000","last_fullsized":"2023-11-28T20:51:54.535286+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:35.784935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.550866+0000","last_change":"2023-11-28T20:51:53.537622+0000","last_active":"2023-11-28T20:51:53.550866+0000","last_peered":"2023-11-28T20:51:53.550866+0000","last_clean":"2023-11-28T20:51:53.550866+0000","last_became_active":"2023-11-28T20:51:53.532788+0000","last_became_peered":"2023-11-28T20:51:53.532788+0000","last_unstale":"2023-11-28T20:51:53.550866+0000","last_undegraded":"2023-11-28T20:51:53.550866+0000","last_fullsized":"2023-11-28T20:51:53.550866+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:37:33.868935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542058+0000","last_change":"2023-11-28T20:51:53.546780+0000","last_active":"2023-11-28T20:51:54.542058+0000","last_peered":"2023-11-28T20:51:54.542058+0000","last_clean":"2023-11-28T20:51:54.542058+0000","last_became_active":"2023-11-28T20:51:53.546503+0000","last_became_peered":"2023-11-28T20:51:53.546503+0000","last_unstale":"2023-11-28T20:51:54.542058+0000","last_undegraded":"2023-11-28T20:51:54.542058+0000","last_fullsized":"2023-11-28T20:51:54.542058+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:15:47.919713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557685+0000","last_change":"2023-11-28T20:51:53.553491+0000","last_active":"2023-11-28T20:51:53.557685+0000","last_peered":"2023-11-28T20:51:53.557685+0000","last_clean":"2023-11-28T20:51:53.557685+0000","last_became_active":"2023-11-28T20:51:53.553186+0000","last_became_peered":"2023-11-28T20:51:53.553186+0000","last_unstale":"2023-11-28T20:51:53.557685+0000","last_undegraded":"2023-11-28T20:51:53.557685+0000","last_fullsized":"2023-11-28T20:51:53.557685+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:51.868674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.555291+0000","last_change":"2023-11-28T20:51:53.544132+0000","last_active":"2023-11-28T20:51:53.555291+0000","last_peered":"2023-11-28T20:51:53.555291+0000","last_clean":"2023-11-28T20:51:53.555291+0000","last_became_active":"2023-11-28T20:51:53.543821+0000","last_became_peered":"2023-11-28T20:51:53.543821+0000","last_unstale":"2023-11-28T20:51:53.555291+0000","last_undegraded":"2023-11-28T20:51:53.555291+0000","last_fullsized":"2023-11-28T20:51:53.555291+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:55:54.519619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.551156+0000","last_change":"2023-11-28T20:51:53.550005+0000","last_active":"2023-11-28T20:51:53.551156+0000","last_peered":"2023-11-28T20:51:53.551156+0000","last_clean":"2023-11-28T20:51:53.551156+0000","last_became_active":"2023-11-28T20:51:53.549647+0000","last_became_peered":"2023-11-28T20:51:53.549647+0000","last_unstale":"2023-11-28T20:51:53.551156+0000","last_undegraded":"2023-11-28T20:51:53.551156+0000","last_fullsized":"2023-11-28T20:51:53.551156+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:35:24.953834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.553083+0000","last_change":"2023-11-28T20:51:53.543792+0000","last_active":"2023-11-28T20:51:53.553083+0000","last_peered":"2023-11-28T20:51:53.553083+0000","last_clean":"2023-11-28T20:51:53.553083+0000","last_became_active":"2023-11-28T20:51:53.543440+0000","last_became_peered":"2023-11-28T20:51:53.543440+0000","last_unstale":"2023-11-28T20:51:53.553083+0000","last_undegraded":"2023-11-28T20:51:53.553083+0000","last_fullsized":"2023-11-28T20:51:53.553083+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:09.278310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543282+0000","last_change":"2023-11-28T20:51:53.547334+0000","last_active":"2023-11-28T20:51:54.543282+0000","last_peered":"2023-11-28T20:51:54.543282+0000","last_clean":"2023-11-28T20:51:54.543282+0000","last_became_active":"2023-11-28T20:51:53.547069+0000","last_became_peered":"2023-11-28T20:51:53.547069+0000","last_unstale":"2023-11-28T20:51:54.543282+0000","last_undegraded":"2023-11-28T20:51:54.543282+0000","last_fullsized":"2023-11-28T20:51:54.543282+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:12.700814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557743+0000","last_change":"2023-11-28T20:51:53.553071+0000","last_active":"2023-11-28T20:51:53.557743+0000","last_peered":"2023-11-28T20:51:53.557743+0000","last_clean":"2023-11-28T20:51:53.557743+0000","last_became_active":"2023-11-28T20:51:53.552746+0000","last_became_peered":"2023-11-28T20:51:53.552746+0000","last_unstale":"2023-11-28T20:51:53.557743+0000","last_undegraded":"2023-11-28T20:51:53.557743+0000","last_fullsized":"2023-11-28T20:51:53.557743+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:11.195412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543389+0000","last_change":"2023-11-28T20:51:53.531880+0000","last_active":"2023-11-28T20:51:54.543389+0000","last_peered":"2023-11-28T20:51:54.543389+0000","last_clean":"2023-11-28T20:51:54.543389+0000","last_became_active":"2023-11-28T20:51:53.530832+0000","last_became_peered":"2023-11-28T20:51:53.530832+0000","last_unstale":"2023-11-28T20:51:54.543389+0000","last_undegraded":"2023-11-28T20:51:54.543389+0000","last_fullsized":"2023-11-28T20:51:54.543389+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:49:42.227913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542288+0000","last_change":"2023-11-28T20:51:53.533580+0000","last_active":"2023-11-28T20:51:54.542288+0000","last_peered":"2023-11-28T20:51:54.542288+0000","last_clean":"2023-11-28T20:51:54.542288+0000","last_became_active":"2023-11-28T20:51:53.533194+0000","last_became_peered":"2023-11-28T20:51:53.533194+0000","last_unstale":"2023-11-28T20:51:54.542288+0000","last_undegraded":"2023-11-28T20:51:54.542288+0000","last_fullsized":"2023-11-28T20:51:54.542288+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:19:22.788175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.555727+0000","last_change":"2023-11-28T20:51:53.552798+0000","last_active":"2023-11-28T20:51:53.555727+0000","last_peered":"2023-11-28T20:51:53.555727+0000","last_clean":"2023-11-28T20:51:53.555727+0000","last_became_active":"2023-11-28T20:51:53.552588+0000","last_became_peered":"2023-11-28T20:51:53.552588+0000","last_unstale":"2023-11-28T20:51:53.555727+0000","last_undegraded":"2023-11-28T20:51:53.555727+0000","last_fullsized":"2023-11-28T20:51:53.555727+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:09:04.070926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.539459+0000","last_change":"2023-11-28T20:51:53.533528+0000","last_active":"2023-11-28T20:51:53.539459+0000","last_peered":"2023-11-28T20:51:53.539459+0000","last_clean":"2023-11-28T20:51:53.539459+0000","last_became_active":"2023-11-28T20:51:53.533183+0000","last_became_peered":"2023-11-28T20:51:53.533183+0000","last_unstale":"2023-11-28T20:51:53.539459+0000","last_undegraded":"2023-11-28T20:51:53.539459+0000","last_fullsized":"2023-11-28T20:51:53.539459+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:46.909178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.545172+0000","last_change":"2023-11-28T20:51:53.529982+0000","last_active":"2023-11-28T20:51:54.545172+0000","last_peered":"2023-11-28T20:51:54.545172+0000","last_clean":"2023-11-28T20:51:54.545172+0000","last_became_active":"2023-11-28T20:51:53.529685+0000","last_became_peered":"2023-11-28T20:51:53.529685+0000","last_unstale":"2023-11-28T20:51:54.545172+0000","last_undegraded":"2023-11-28T20:51:54.545172+0000","last_fullsized":"2023-11-28T20:51:54.545172+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:26:07.345244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542434+0000","last_change":"2023-11-28T20:51:53.533307+0000","last_active":"2023-11-28T20:51:54.542434+0000","last_peered":"2023-11-28T20:51:54.542434+0000","last_clean":"2023-11-28T20:51:54.542434+0000","last_became_active":"2023-11-28T20:51:53.532972+0000","last_became_peered":"2023-11-28T20:51:53.532972+0000","last_unstale":"2023-11-28T20:51:54.542434+0000","last_undegraded":"2023-11-28T20:51:54.542434+0000","last_fullsized":"2023-11-28T20:51:54.542434+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:02.840995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535791+0000","last_change":"2023-11-28T20:51:53.541833+0000","last_active":"2023-11-28T20:51:54.535791+0000","last_peered":"2023-11-28T20:51:54.535791+0000","last_clean":"2023-11-28T20:51:54.535791+0000","last_became_active":"2023-11-28T20:51:53.539098+0000","last_became_peered":"2023-11-28T20:51:53.539098+0000","last_unstale":"2023-11-28T20:51:54.535791+0000","last_undegraded":"2023-11-28T20:51:54.535791+0000","last_fullsized":"2023-11-28T20:51:54.535791+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:52:55.647413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.552608+0000","last_change":"2023-11-28T20:51:53.533209+0000","last_active":"2023-11-28T20:51:53.552608+0000","last_peered":"2023-11-28T20:51:53.552608+0000","last_clean":"2023-11-28T20:51:53.552608+0000","last_became_active":"2023-11-28T20:51:53.532903+0000","last_became_peered":"2023-11-28T20:51:53.532903+0000","last_unstale":"2023-11-28T20:51:53.552608+0000","last_undegraded":"2023-11-28T20:51:53.552608+0000","last_fullsized":"2023-11-28T20:51:53.552608+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:43.717714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.555945+0000","last_change":"2023-11-28T20:51:53.554484+0000","last_active":"2023-11-28T20:51:53.555945+0000","last_peered":"2023-11-28T20:51:53.555945+0000","last_clean":"2023-11-28T20:51:53.555945+0000","last_became_active":"2023-11-28T20:51:53.554196+0000","last_became_peered":"2023-11-28T20:51:53.554196+0000","last_unstale":"2023-11-28T20:51:53.555945+0000","last_undegraded":"2023-11-28T20:51:53.555945+0000","last_fullsized":"2023-11-28T20:51:53.555945+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:06.560961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542728+0000","last_change":"2023-11-28T20:51:53.527930+0000","last_active":"2023-11-28T20:51:54.542728+0000","last_peered":"2023-11-28T20:51:54.542728+0000","last_clean":"2023-11-28T20:51:54.542728+0000","last_became_active":"2023-11-28T20:51:53.523324+0000","last_became_peered":"2023-11-28T20:51:53.523324+0000","last_unstale":"2023-11-28T20:51:54.542728+0000","last_undegraded":"2023-11-28T20:51:54.542728+0000","last_fullsized":"2023-11-28T20:51:54.542728+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:57:08.910602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543642+0000","last_change":"2023-11-28T20:51:53.533882+0000","last_active":"2023-11-28T20:51:54.543642+0000","last_peered":"2023-11-28T20:51:54.543642+0000","last_clean":"2023-11-28T20:51:54.543642+0000","last_became_active":"2023-11-28T20:51:53.533503+0000","last_became_peered":"2023-11-28T20:51:53.533503+0000","last_unstale":"2023-11-28T20:51:54.543642+0000","last_undegraded":"2023-11-28T20:51:54.543642+0000","last_fullsized":"2023-11-28T20:51:54.543642+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:46.909178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.539864+0000","last_change":"2023-11-28T20:51:53.538125+0000","last_active":"2023-11-28T20:51:53.539864+0000","last_peered":"2023-11-28T20:51:53.539864+0000","last_clean":"2023-11-28T20:51:53.539864+0000","last_became_active":"2023-11-28T20:51:53.537744+0000","last_became_peered":"2023-11-28T20:51:53.537744+0000","last_unstale":"2023-11-28T20:51:53.539864+0000","last_undegraded":"2023-11-28T20:51:53.539864+0000","last_fullsized":"2023-11-28T20:51:53.539864+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:59.208311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.549143+0000","last_change":"2023-11-28T20:51:53.542050+0000","last_active":"2023-11-28T20:51:53.549143+0000","last_peered":"2023-11-28T20:51:53.549143+0000","last_clean":"2023-11-28T20:51:53.549143+0000","last_became_active":"2023-11-28T20:51:53.541810+0000","last_became_peered":"2023-11-28T20:51:53.541810+0000","last_unstale":"2023-11-28T20:51:53.549143+0000","last_undegraded":"2023-11-28T20:51:53.549143+0000","last_fullsized":"2023-11-28T20:51:53.549143+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:50:04.881346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"115'4","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.566334+0000","last_change":"2023-11-28T20:51:52.509651+0000","last_active":"2023-11-28T20:51:53.566334+0000","last_peered":"2023-11-28T20:51:53.566334+0000","last_clean":"2023-11-28T20:51:53.566334+0000","last_became_active":"2023-11-28T20:51:52.509187+0000","last_became_peered":"2023-11-28T20:51:52.509187+0000","last_unstale":"2023-11-28T20:51:53.566334+0000","last_undegraded":"2023-11-28T20:51:53.566334+0000","last_fullsized":"2023-11-28T20:51:53.566334+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:15:58.123873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557879+0000","last_change":"2023-11-28T20:51:53.554699+0000","last_active":"2023-11-28T20:51:53.557879+0000","last_peered":"2023-11-28T20:51:53.557879+0000","last_clean":"2023-11-28T20:51:53.557879+0000","last_became_active":"2023-11-28T20:51:53.554389+0000","last_became_peered":"2023-11-28T20:51:53.554389+0000","last_unstale":"2023-11-28T20:51:53.557879+0000","last_undegraded":"2023-11-28T20:51:53.557879+0000","last_fullsized":"2023-11-28T20:51:53.557879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:31.154886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"115'1","reported_seq":13,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542920+0000","last_change":"2023-11-28T20:51:52.502164+0000","last_active":"2023-11-28T20:51:54.542920+0000","last_peered":"2023-11-28T20:51:54.542920+0000","last_clean":"2023-11-28T20:51:54.542920+0000","last_became_active":"2023-11-28T20:51:52.501281+0000","last_became_peered":"2023-11-28T20:51:52.501281+0000","last_unstale":"2023-11-28T20:51:54.542920+0000","last_undegraded":"2023-11-28T20:51:54.542920+0000","last_fullsized":"2023-11-28T20:51:54.542920+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:33.887418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.548261+0000","last_change":"2023-11-28T20:51:53.546939+0000","last_active":"2023-11-28T20:51:53.548261+0000","last_peered":"2023-11-28T20:51:53.548261+0000","last_clean":"2023-11-28T20:51:53.548261+0000","last_became_active":"2023-11-28T20:51:53.546662+0000","last_became_peered":"2023-11-28T20:51:53.546662+0000","last_unstale":"2023-11-28T20:51:53.548261+0000","last_undegraded":"2023-11-28T20:51:53.548261+0000","last_fullsized":"2023-11-28T20:51:53.548261+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:09.278310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.561423+0000","last_change":"2023-11-28T20:51:52.508214+0000","last_active":"2023-11-28T20:51:53.561423+0000","last_peered":"2023-11-28T20:51:53.561423+0000","last_clean":"2023-11-28T20:51:53.561423+0000","last_became_active":"2023-11-28T20:51:52.507929+0000","last_became_peered":"2023-11-28T20:51:52.507929+0000","last_unstale":"2023-11-28T20:51:53.561423+0000","last_undegraded":"2023-11-28T20:51:53.561423+0000","last_fullsized":"2023-11-28T20:51:53.561423+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:57:37.832266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.544662+0000","last_change":"2023-11-28T20:51:53.541488+0000","last_active":"2023-11-28T20:51:53.544662+0000","last_peered":"2023-11-28T20:51:53.544662+0000","last_clean":"2023-11-28T20:51:53.544662+0000","last_became_active":"2023-11-28T20:51:53.541130+0000","last_became_peered":"2023-11-28T20:51:53.541130+0000","last_unstale":"2023-11-28T20:51:53.544662+0000","last_undegraded":"2023-11-28T20:51:53.544662+0000","last_fullsized":"2023-11-28T20:51:53.544662+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:45:00.161448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.565682+0000","last_change":"2023-11-28T20:51:52.507786+0000","last_active":"2023-11-28T20:51:53.565682+0000","last_peered":"2023-11-28T20:51:53.565682+0000","last_clean":"2023-11-28T20:51:53.565682+0000","last_became_active":"2023-11-28T20:51:52.507514+0000","last_became_peered":"2023-11-28T20:51:52.507514+0000","last_unstale":"2023-11-28T20:51:53.565682+0000","last_undegraded":"2023-11-28T20:51:53.565682+0000","last_fullsized":"2023-11-28T20:51:53.565682+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:25.257706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.550506+0000","last_change":"2023-11-28T20:51:53.540465+0000","last_active":"2023-11-28T20:51:53.550506+0000","last_peered":"2023-11-28T20:51:53.550506+0000","last_clean":"2023-11-28T20:51:53.550506+0000","last_became_active":"2023-11-28T20:51:53.536764+0000","last_became_peered":"2023-11-28T20:51:53.536764+0000","last_unstale":"2023-11-28T20:51:53.550506+0000","last_undegraded":"2023-11-28T20:51:53.550506+0000","last_fullsized":"2023-11-28T20:51:53.550506+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:26.273894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"115'1","reported_seq":13,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543641+0000","last_change":"2023-11-28T20:51:52.492944+0000","last_active":"2023-11-28T20:51:54.543641+0000","last_peered":"2023-11-28T20:51:54.543641+0000","last_clean":"2023-11-28T20:51:54.543641+0000","last_became_active":"2023-11-28T20:51:52.492350+0000","last_became_peered":"2023-11-28T20:51:52.492350+0000","last_unstale":"2023-11-28T20:51:54.543641+0000","last_undegraded":"2023-11-28T20:51:54.543641+0000","last_fullsized":"2023-11-28T20:51:54.543641+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:43.383167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.547568+0000","last_change":"2023-11-28T20:51:53.544104+0000","last_active":"2023-11-28T20:51:53.547568+0000","last_peered":"2023-11-28T20:51:53.547568+0000","last_clean":"2023-11-28T20:51:53.547568+0000","last_became_active":"2023-11-28T20:51:53.543784+0000","last_became_peered":"2023-11-28T20:51:53.543784+0000","last_unstale":"2023-11-28T20:51:53.547568+0000","last_undegraded":"2023-11-28T20:51:53.547568+0000","last_fullsized":"2023-11-28T20:51:53.547568+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:00:30.572025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.565729+0000","last_change":"2023-11-28T20:51:52.488209+0000","last_active":"2023-11-28T20:51:53.565729+0000","last_peered":"2023-11-28T20:51:53.565729+0000","last_clean":"2023-11-28T20:51:53.565729+0000","last_became_active":"2023-11-28T20:51:52.487787+0000","last_became_peered":"2023-11-28T20:51:52.487787+0000","last_unstale":"2023-11-28T20:51:53.565729+0000","last_undegraded":"2023-11-28T20:51:53.565729+0000","last_fullsized":"2023-11-28T20:51:53.565729+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:00:29.555837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.537344+0000","last_change":"2023-11-28T20:51:53.534963+0000","last_active":"2023-11-28T20:51:53.537344+0000","last_peered":"2023-11-28T20:51:53.537344+0000","last_clean":"2023-11-28T20:51:53.537344+0000","last_became_active":"2023-11-28T20:51:53.534666+0000","last_became_peered":"2023-11-28T20:51:53.534666+0000","last_unstale":"2023-11-28T20:51:53.537344+0000","last_undegraded":"2023-11-28T20:51:53.537344+0000","last_fullsized":"2023-11-28T20:51:53.537344+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:21:56.699930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.536036+0000","last_change":"2023-11-28T20:51:52.503213+0000","last_active":"2023-11-28T20:51:53.536036+0000","last_peered":"2023-11-28T20:51:53.536036+0000","last_clean":"2023-11-28T20:51:53.536036+0000","last_became_active":"2023-11-28T20:51:52.502952+0000","last_became_peered":"2023-11-28T20:51:52.502952+0000","last_unstale":"2023-11-28T20:51:53.536036+0000","last_undegraded":"2023-11-28T20:51:53.536036+0000","last_fullsized":"2023-11-28T20:51:53.536036+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:23.244332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.548724+0000","last_change":"2023-11-28T20:51:53.535283+0000","last_active":"2023-11-28T20:51:53.548724+0000","last_peered":"2023-11-28T20:51:53.548724+0000","last_clean":"2023-11-28T20:51:53.548724+0000","last_became_active":"2023-11-28T20:51:53.534988+0000","last_became_peered":"2023-11-28T20:51:53.534988+0000","last_unstale":"2023-11-28T20:51:53.548724+0000","last_undegraded":"2023-11-28T20:51:53.548724+0000","last_fullsized":"2023-11-28T20:51:53.548724+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:15:59.140061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.526278+0000","last_change":"2023-11-28T20:51:52.510669+0000","last_active":"2023-11-28T20:51:53.526278+0000","last_peered":"2023-11-28T20:51:53.526278+0000","last_clean":"2023-11-28T20:51:53.526278+0000","last_became_active":"2023-11-28T20:51:52.507728+0000","last_became_peered":"2023-11-28T20:51:52.507728+0000","last_unstale":"2023-11-28T20:51:53.526278+0000","last_undegraded":"2023-11-28T20:51:53.526278+0000","last_fullsized":"2023-11-28T20:51:53.526278+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:45.892991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543362+0000","last_change":"2023-11-28T20:51:53.540615+0000","last_active":"2023-11-28T20:51:54.543362+0000","last_peered":"2023-11-28T20:51:54.543362+0000","last_clean":"2023-11-28T20:51:54.543362+0000","last_became_active":"2023-11-28T20:51:53.540282+0000","last_became_peered":"2023-11-28T20:51:53.540282+0000","last_unstale":"2023-11-28T20:51:54.543362+0000","last_undegraded":"2023-11-28T20:51:54.543362+0000","last_fullsized":"2023-11-28T20:51:54.543362+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:23.863772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.545244+0000","last_change":"2023-11-28T20:51:52.500440+0000","last_active":"2023-11-28T20:51:54.545244+0000","last_peered":"2023-11-28T20:51:54.545244+0000","last_clean":"2023-11-28T20:51:54.545244+0000","last_became_active":"2023-11-28T20:51:52.500003+0000","last_became_peered":"2023-11-28T20:51:52.500003+0000","last_unstale":"2023-11-28T20:51:54.545244+0000","last_undegraded":"2023-11-28T20:51:54.545244+0000","last_fullsized":"2023-11-28T20:51:54.545244+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:57:41.133836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.540750+0000","last_change":"2023-11-28T20:51:53.539220+0000","last_active":"2023-11-28T20:51:53.540750+0000","last_peered":"2023-11-28T20:51:53.540750+0000","last_clean":"2023-11-28T20:51:53.540750+0000","last_became_active":"2023-11-28T20:51:53.538922+0000","last_became_peered":"2023-11-28T20:51:53.538922+0000","last_unstale":"2023-11-28T20:51:53.540750+0000","last_undegraded":"2023-11-28T20:51:53.540750+0000","last_fullsized":"2023-11-28T20:51:53.540750+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:21:56.699930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.531870+0000","last_change":"2023-11-28T20:51:52.485182+0000","last_active":"2023-11-28T20:51:53.531870+0000","last_peered":"2023-11-28T20:51:53.531870+0000","last_clean":"2023-11-28T20:51:53.531870+0000","last_became_active":"2023-11-28T20:51:52.484534+0000","last_became_peered":"2023-11-28T20:51:52.484534+0000","last_unstale":"2023-11-28T20:51:53.531870+0000","last_undegraded":"2023-11-28T20:51:53.531870+0000","last_fullsized":"2023-11-28T20:51:53.531870+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:26:06.329057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.554854+0000","last_change":"2023-11-28T20:51:53.547157+0000","last_active":"2023-11-28T20:51:53.554854+0000","last_peered":"2023-11-28T20:51:53.554854+0000","last_clean":"2023-11-28T20:51:53.554854+0000","last_became_active":"2023-11-28T20:51:53.546863+0000","last_became_peered":"2023-11-28T20:51:53.546863+0000","last_unstale":"2023-11-28T20:51:53.554854+0000","last_undegraded":"2023-11-28T20:51:53.554854+0000","last_fullsized":"2023-11-28T20:51:53.554854+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:40:25.263861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"115'3","reported_seq":15,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543762+0000","last_change":"2023-11-28T20:51:52.497727+0000","last_active":"2023-11-28T20:51:54.543762+0000","last_peered":"2023-11-28T20:51:54.543762+0000","last_clean":"2023-11-28T20:51:54.543762+0000","last_became_active":"2023-11-28T20:51:52.497394+0000","last_became_peered":"2023-11-28T20:51:52.497394+0000","last_unstale":"2023-11-28T20:51:54.543762+0000","last_undegraded":"2023-11-28T20:51:54.543762+0000","last_fullsized":"2023-11-28T20:51:54.543762+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:44:13.921193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.554246+0000","last_change":"2023-11-28T20:51:53.533344+0000","last_active":"2023-11-28T20:51:53.554246+0000","last_peered":"2023-11-28T20:51:53.554246+0000","last_clean":"2023-11-28T20:51:53.554246+0000","last_became_active":"2023-11-28T20:51:53.533066+0000","last_became_peered":"2023-11-28T20:51:53.533066+0000","last_unstale":"2023-11-28T20:51:53.554246+0000","last_undegraded":"2023-11-28T20:51:53.554246+0000","last_fullsized":"2023-11-28T20:51:53.554246+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:06:57.200161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"115'6","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.570678+0000","last_change":"2023-11-28T20:51:52.507964+0000","last_active":"2023-11-28T20:51:53.570678+0000","last_peered":"2023-11-28T20:51:53.570678+0000","last_clean":"2023-11-28T20:51:53.570678+0000","last_became_active":"2023-11-28T20:51:52.507673+0000","last_became_peered":"2023-11-28T20:51:52.507673+0000","last_unstale":"2023-11-28T20:51:53.570678+0000","last_undegraded":"2023-11-28T20:51:53.570678+0000","last_fullsized":"2023-11-28T20:51:53.570678+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:02:41.250452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.545760+0000","last_change":"2023-11-28T20:51:53.531801+0000","last_active":"2023-11-28T20:51:53.545760+0000","last_peered":"2023-11-28T20:51:53.545760+0000","last_clean":"2023-11-28T20:51:53.545760+0000","last_became_active":"2023-11-28T20:51:53.531153+0000","last_became_peered":"2023-11-28T20:51:53.531153+0000","last_unstale":"2023-11-28T20:51:53.545760+0000","last_undegraded":"2023-11-28T20:51:53.545760+0000","last_fullsized":"2023-11-28T20:51:53.545760+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:34.903606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.564179+0000","last_change":"2023-11-28T20:51:52.511046+0000","last_active":"2023-11-28T20:51:53.564179+0000","last_peered":"2023-11-28T20:51:53.564179+0000","last_clean":"2023-11-28T20:51:53.564179+0000","last_became_active":"2023-11-28T20:51:52.510527+0000","last_became_peered":"2023-11-28T20:51:52.510527+0000","last_unstale":"2023-11-28T20:51:53.564179+0000","last_undegraded":"2023-11-28T20:51:53.564179+0000","last_fullsized":"2023-11-28T20:51:53.564179+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:01.844967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.541952+0000","last_change":"2023-11-28T20:51:53.536735+0000","last_active":"2023-11-28T20:51:54.541952+0000","last_peered":"2023-11-28T20:51:54.541952+0000","last_clean":"2023-11-28T20:51:54.541952+0000","last_became_active":"2023-11-28T20:51:53.536386+0000","last_became_peered":"2023-11-28T20:51:53.536386+0000","last_unstale":"2023-11-28T20:51:54.541952+0000","last_undegraded":"2023-11-28T20:51:54.541952+0000","last_fullsized":"2023-11-28T20:51:54.541952+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:23:15.786207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.519194+0000","last_change":"2023-11-28T20:51:52.507937+0000","last_active":"2023-11-28T20:51:53.519194+0000","last_peered":"2023-11-28T20:51:53.519194+0000","last_clean":"2023-11-28T20:51:53.519194+0000","last_became_active":"2023-11-28T20:51:52.507550+0000","last_became_peered":"2023-11-28T20:51:52.507550+0000","last_unstale":"2023-11-28T20:51:53.519194+0000","last_undegraded":"2023-11-28T20:51:53.519194+0000","last_fullsized":"2023-11-28T20:51:53.519194+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:57:01.544189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535179+0000","last_change":"2023-11-28T20:51:53.532485+0000","last_active":"2023-11-28T20:51:54.535179+0000","last_peered":"2023-11-28T20:51:54.535179+0000","last_clean":"2023-11-28T20:51:54.535179+0000","last_became_active":"2023-11-28T20:51:53.532137+0000","last_became_peered":"2023-11-28T20:51:53.532137+0000","last_unstale":"2023-11-28T20:51:54.535179+0000","last_undegraded":"2023-11-28T20:51:54.535179+0000","last_fullsized":"2023-11-28T20:51:54.535179+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:09.648885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.510878+0000","last_change":"2023-11-28T20:51:52.502989+0000","last_active":"2023-11-28T20:51:53.510878+0000","last_peered":"2023-11-28T20:51:53.510878+0000","last_clean":"2023-11-28T20:51:53.510878+0000","last_became_active":"2023-11-28T20:51:52.501113+0000","last_became_peered":"2023-11-28T20:51:52.501113+0000","last_unstale":"2023-11-28T20:51:53.510878+0000","last_undegraded":"2023-11-28T20:51:53.510878+0000","last_fullsized":"2023-11-28T20:51:53.510878+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:13:05.430726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535581+0000","last_change":"2023-11-28T20:51:53.536365+0000","last_active":"2023-11-28T20:51:54.535581+0000","last_peered":"2023-11-28T20:51:54.535581+0000","last_clean":"2023-11-28T20:51:54.535581+0000","last_became_active":"2023-11-28T20:51:53.536070+0000","last_became_peered":"2023-11-28T20:51:53.536070+0000","last_unstale":"2023-11-28T20:51:54.535581+0000","last_undegraded":"2023-11-28T20:51:54.535581+0000","last_fullsized":"2023-11-28T20:51:54.535581+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:33:05.000766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"115'8","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.567793+0000","last_change":"2023-11-28T20:51:52.488061+0000","last_active":"2023-11-28T20:51:53.567793+0000","last_peered":"2023-11-28T20:51:53.567793+0000","last_clean":"2023-11-28T20:51:53.567793+0000","last_became_active":"2023-11-28T20:51:52.487740+0000","last_became_peered":"2023-11-28T20:51:52.487740+0000","last_unstale":"2023-11-28T20:51:53.567793+0000","last_undegraded":"2023-11-28T20:51:53.567793+0000","last_fullsized":"2023-11-28T20:51:53.567793+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:21:55.994002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.545873+0000","last_change":"2023-11-28T20:51:53.542013+0000","last_active":"2023-11-28T20:51:53.545873+0000","last_peered":"2023-11-28T20:51:53.545873+0000","last_clean":"2023-11-28T20:51:53.545873+0000","last_became_active":"2023-11-28T20:51:53.541455+0000","last_became_peered":"2023-11-28T20:51:53.541455+0000","last_unstale":"2023-11-28T20:51:53.545873+0000","last_undegraded":"2023-11-28T20:51:53.545873+0000","last_fullsized":"2023-11-28T20:51:53.545873+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:59:03.064971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.561769+0000","last_change":"2023-11-28T20:51:52.506560+0000","last_active":"2023-11-28T20:51:53.561769+0000","last_peered":"2023-11-28T20:51:53.561769+0000","last_clean":"2023-11-28T20:51:53.561769+0000","last_became_active":"2023-11-28T20:51:52.505794+0000","last_became_peered":"2023-11-28T20:51:52.505794+0000","last_unstale":"2023-11-28T20:51:53.561769+0000","last_undegraded":"2023-11-28T20:51:53.561769+0000","last_fullsized":"2023-11-28T20:51:53.561769+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:21:55.994002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.539550+0000","last_change":"2023-11-28T20:51:53.535786+0000","last_active":"2023-11-28T20:51:53.539550+0000","last_peered":"2023-11-28T20:51:53.539550+0000","last_clean":"2023-11-28T20:51:53.539550+0000","last_became_active":"2023-11-28T20:51:53.535325+0000","last_became_peered":"2023-11-28T20:51:53.535325+0000","last_unstale":"2023-11-28T20:51:53.539550+0000","last_undegraded":"2023-11-28T20:51:53.539550+0000","last_fullsized":"2023-11-28T20:51:53.539550+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:42.893430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.561175+0000","last_change":"2023-11-28T20:51:52.507128+0000","last_active":"2023-11-28T20:51:53.561175+0000","last_peered":"2023-11-28T20:51:53.561175+0000","last_clean":"2023-11-28T20:51:53.561175+0000","last_became_active":"2023-11-28T20:51:52.506856+0000","last_became_peered":"2023-11-28T20:51:52.506856+0000","last_unstale":"2023-11-28T20:51:53.561175+0000","last_undegraded":"2023-11-28T20:51:53.561175+0000","last_fullsized":"2023-11-28T20:51:53.561175+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:23.244332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542734+0000","last_change":"2023-11-28T20:51:53.539900+0000","last_active":"2023-11-28T20:51:54.542734+0000","last_peered":"2023-11-28T20:51:54.542734+0000","last_clean":"2023-11-28T20:51:54.542734+0000","last_became_active":"2023-11-28T20:51:53.539578+0000","last_became_peered":"2023-11-28T20:51:53.539578+0000","last_unstale":"2023-11-28T20:51:54.542734+0000","last_undegraded":"2023-11-28T20:51:54.542734+0000","last_fullsized":"2023-11-28T20:51:54.542734+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:58:21.128503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"115'1","reported_seq":13,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544000+0000","last_change":"2023-11-28T20:51:52.489432+0000","last_active":"2023-11-28T20:51:54.544000+0000","last_peered":"2023-11-28T20:51:54.544000+0000","last_clean":"2023-11-28T20:51:54.544000+0000","last_became_active":"2023-11-28T20:51:52.487841+0000","last_became_peered":"2023-11-28T20:51:52.487841+0000","last_unstale":"2023-11-28T20:51:54.544000+0000","last_undegraded":"2023-11-28T20:51:54.544000+0000","last_fullsized":"2023-11-28T20:51:54.544000+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:54.049426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.537960+0000","last_change":"2023-11-28T20:51:53.535321+0000","last_active":"2023-11-28T20:51:53.537960+0000","last_peered":"2023-11-28T20:51:53.537960+0000","last_clean":"2023-11-28T20:51:53.537960+0000","last_became_active":"2023-11-28T20:51:53.535061+0000","last_became_peered":"2023-11-28T20:51:53.535061+0000","last_unstale":"2023-11-28T20:51:53.537960+0000","last_undegraded":"2023-11-28T20:51:53.537960+0000","last_fullsized":"2023-11-28T20:51:53.537960+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:55.838302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"115'2","reported_seq":14,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544754+0000","last_change":"2023-11-28T20:51:52.500521+0000","last_active":"2023-11-28T20:51:54.544754+0000","last_peered":"2023-11-28T20:51:54.544754+0000","last_clean":"2023-11-28T20:51:54.544754+0000","last_became_active":"2023-11-28T20:51:52.498314+0000","last_became_peered":"2023-11-28T20:51:52.498314+0000","last_unstale":"2023-11-28T20:51:54.544754+0000","last_undegraded":"2023-11-28T20:51:54.544754+0000","last_fullsized":"2023-11-28T20:51:54.544754+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:43:23.678537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543726+0000","last_change":"2023-11-28T20:51:53.543024+0000","last_active":"2023-11-28T20:51:54.543726+0000","last_peered":"2023-11-28T20:51:54.543726+0000","last_clean":"2023-11-28T20:51:54.543726+0000","last_became_active":"2023-11-28T20:51:53.542650+0000","last_became_peered":"2023-11-28T20:51:53.542650+0000","last_unstale":"2023-11-28T20:51:54.543726+0000","last_undegraded":"2023-11-28T20:51:54.543726+0000","last_fullsized":"2023-11-28T20:51:54.543726+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:09.278310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535338+0000","last_change":"2023-11-28T20:51:52.485518+0000","last_active":"2023-11-28T20:51:54.535338+0000","last_peered":"2023-11-28T20:51:54.535338+0000","last_clean":"2023-11-28T20:51:54.535338+0000","last_became_active":"2023-11-28T20:51:52.485128+0000","last_became_peered":"2023-11-28T20:51:52.485128+0000","last_unstale":"2023-11-28T20:51:54.535338+0000","last_undegraded":"2023-11-28T20:51:54.535338+0000","last_fullsized":"2023-11-28T20:51:54.535338+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:10:37.523999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544499+0000","last_change":"2023-11-28T20:51:53.533892+0000","last_active":"2023-11-28T20:51:54.544499+0000","last_peered":"2023-11-28T20:51:54.544499+0000","last_clean":"2023-11-28T20:51:54.544499+0000","last_became_active":"2023-11-28T20:51:53.533567+0000","last_became_peered":"2023-11-28T20:51:53.533567+0000","last_unstale":"2023-11-28T20:51:54.544499+0000","last_undegraded":"2023-11-28T20:51:54.544499+0000","last_fullsized":"2023-11-28T20:51:54.544499+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:57:38.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543880+0000","last_change":"2023-11-28T20:51:52.509352+0000","last_active":"2023-11-28T20:51:54.543880+0000","last_peered":"2023-11-28T20:51:54.543880+0000","last_clean":"2023-11-28T20:51:54.543880+0000","last_became_active":"2023-11-28T20:51:52.509097+0000","last_became_peered":"2023-11-28T20:51:52.509097+0000","last_unstale":"2023-11-28T20:51:54.543880+0000","last_undegraded":"2023-11-28T20:51:54.543880+0000","last_fullsized":"2023-11-28T20:51:54.543880+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:26:59.101603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.558843+0000","last_change":"2023-11-28T20:51:53.556522+0000","last_active":"2023-11-28T20:51:53.558843+0000","last_peered":"2023-11-28T20:51:53.558843+0000","last_clean":"2023-11-28T20:51:53.558843+0000","last_became_active":"2023-11-28T20:51:53.555042+0000","last_became_peered":"2023-11-28T20:51:53.555042+0000","last_unstale":"2023-11-28T20:51:53.558843+0000","last_undegraded":"2023-11-28T20:51:53.558843+0000","last_fullsized":"2023-11-28T20:51:53.558843+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:32:37.046552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535010+0000","last_change":"2023-11-28T20:51:52.502790+0000","last_active":"2023-11-28T20:51:54.535010+0000","last_peered":"2023-11-28T20:51:54.535010+0000","last_clean":"2023-11-28T20:51:54.535010+0000","last_became_active":"2023-11-28T20:51:52.502466+0000","last_became_peered":"2023-11-28T20:51:52.502466+0000","last_unstale":"2023-11-28T20:51:54.535010+0000","last_undegraded":"2023-11-28T20:51:54.535010+0000","last_fullsized":"2023-11-28T20:51:54.535010+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:53.258431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.539498+0000","last_change":"2023-11-28T20:51:53.535512+0000","last_active":"2023-11-28T20:51:53.539498+0000","last_peered":"2023-11-28T20:51:53.539498+0000","last_clean":"2023-11-28T20:51:53.539498+0000","last_became_active":"2023-11-28T20:51:53.534118+0000","last_became_peered":"2023-11-28T20:51:53.534118+0000","last_unstale":"2023-11-28T20:51:53.539498+0000","last_undegraded":"2023-11-28T20:51:53.539498+0000","last_fullsized":"2023-11-28T20:51:53.539498+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:29.714777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.563932+0000","last_change":"2023-11-28T20:51:52.507737+0000","last_active":"2023-11-28T20:51:53.563932+0000","last_peered":"2023-11-28T20:51:53.563932+0000","last_clean":"2023-11-28T20:51:53.563932+0000","last_became_active":"2023-11-28T20:51:52.507406+0000","last_became_peered":"2023-11-28T20:51:52.507406+0000","last_unstale":"2023-11-28T20:51:53.563932+0000","last_undegraded":"2023-11-28T20:51:53.563932+0000","last_fullsized":"2023-11-28T20:51:53.563932+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:58.590400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.547495+0000","last_change":"2023-11-28T20:51:53.542993+0000","last_active":"2023-11-28T20:51:53.547495+0000","last_peered":"2023-11-28T20:51:53.547495+0000","last_clean":"2023-11-28T20:51:53.547495+0000","last_became_active":"2023-11-28T20:51:53.542752+0000","last_became_peered":"2023-11-28T20:51:53.542752+0000","last_unstale":"2023-11-28T20:51:53.547495+0000","last_undegraded":"2023-11-28T20:51:53.547495+0000","last_fullsized":"2023-11-28T20:51:53.547495+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:46:45.164961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543165+0000","last_change":"2023-11-28T20:51:52.495385+0000","last_active":"2023-11-28T20:51:54.543165+0000","last_peered":"2023-11-28T20:51:54.543165+0000","last_clean":"2023-11-28T20:51:54.543165+0000","last_became_active":"2023-11-28T20:51:52.494891+0000","last_became_peered":"2023-11-28T20:51:52.494891+0000","last_unstale":"2023-11-28T20:51:54.543165+0000","last_undegraded":"2023-11-28T20:51:54.543165+0000","last_fullsized":"2023-11-28T20:51:54.543165+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:06:56.183974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557900+0000","last_change":"2023-11-28T20:51:53.554804+0000","last_active":"2023-11-28T20:51:53.557900+0000","last_peered":"2023-11-28T20:51:53.557900+0000","last_clean":"2023-11-28T20:51:53.557900+0000","last_became_active":"2023-11-28T20:51:53.554323+0000","last_became_peered":"2023-11-28T20:51:53.554323+0000","last_unstale":"2023-11-28T20:51:53.557900+0000","last_undegraded":"2023-11-28T20:51:53.557900+0000","last_fullsized":"2023-11-28T20:51:53.557900+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:04:56.995653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544537+0000","last_change":"2023-11-28T20:51:52.510461+0000","last_active":"2023-11-28T20:51:54.544537+0000","last_peered":"2023-11-28T20:51:54.544537+0000","last_clean":"2023-11-28T20:51:54.544537+0000","last_became_active":"2023-11-28T20:51:52.509630+0000","last_became_peered":"2023-11-28T20:51:52.509630+0000","last_unstale":"2023-11-28T20:51:54.544537+0000","last_undegraded":"2023-11-28T20:51:54.544537+0000","last_fullsized":"2023-11-28T20:51:54.544537+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:36:27.725428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.545124+0000","last_change":"2023-11-28T20:51:53.534614+0000","last_active":"2023-11-28T20:51:54.545124+0000","last_peered":"2023-11-28T20:51:54.545124+0000","last_clean":"2023-11-28T20:51:54.545124+0000","last_became_active":"2023-11-28T20:51:53.534323+0000","last_became_peered":"2023-11-28T20:51:53.534323+0000","last_unstale":"2023-11-28T20:51:54.545124+0000","last_undegraded":"2023-11-28T20:51:54.545124+0000","last_fullsized":"2023-11-28T20:51:54.545124+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:40:25.263861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.525913+0000","last_change":"2023-11-28T20:51:52.503666+0000","last_active":"2023-11-28T20:51:53.525913+0000","last_peered":"2023-11-28T20:51:53.525913+0000","last_clean":"2023-11-28T20:51:53.525913+0000","last_became_active":"2023-11-28T20:51:52.503280+0000","last_became_peered":"2023-11-28T20:51:52.503280+0000","last_unstale":"2023-11-28T20:51:53.525913+0000","last_undegraded":"2023-11-28T20:51:53.525913+0000","last_fullsized":"2023-11-28T20:51:53.525913+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:50.852486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535284+0000","last_change":"2023-11-28T20:51:53.539474+0000","last_active":"2023-11-28T20:51:54.535284+0000","last_peered":"2023-11-28T20:51:54.535284+0000","last_clean":"2023-11-28T20:51:54.535284+0000","last_became_active":"2023-11-28T20:51:53.539143+0000","last_became_peered":"2023-11-28T20:51:53.539143+0000","last_unstale":"2023-11-28T20:51:54.535284+0000","last_undegraded":"2023-11-28T20:51:54.535284+0000","last_fullsized":"2023-11-28T20:51:54.535284+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:45.693788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.528801+0000","last_change":"2023-11-28T20:51:52.505376+0000","last_active":"2023-11-28T20:51:53.528801+0000","last_peered":"2023-11-28T20:51:53.528801+0000","last_clean":"2023-11-28T20:51:53.528801+0000","last_became_active":"2023-11-28T20:51:52.504325+0000","last_became_peered":"2023-11-28T20:51:52.504325+0000","last_unstale":"2023-11-28T20:51:53.528801+0000","last_undegraded":"2023-11-28T20:51:53.528801+0000","last_fullsized":"2023-11-28T20:51:53.528801+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:26.208118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557785+0000","last_change":"2023-11-28T20:51:53.554859+0000","last_active":"2023-11-28T20:51:53.557785+0000","last_peered":"2023-11-28T20:51:53.557785+0000","last_clean":"2023-11-28T20:51:53.557785+0000","last_became_active":"2023-11-28T20:51:53.554439+0000","last_became_peered":"2023-11-28T20:51:53.554439+0000","last_unstale":"2023-11-28T20:51:53.557785+0000","last_undegraded":"2023-11-28T20:51:53.557785+0000","last_fullsized":"2023-11-28T20:51:53.557785+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:59.606587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544555+0000","last_change":"2023-11-28T20:51:52.503843+0000","last_active":"2023-11-28T20:51:54.544555+0000","last_peered":"2023-11-28T20:51:54.544555+0000","last_clean":"2023-11-28T20:51:54.544555+0000","last_became_active":"2023-11-28T20:51:52.499139+0000","last_became_peered":"2023-11-28T20:51:52.499139+0000","last_unstale":"2023-11-28T20:51:54.544555+0000","last_undegraded":"2023-11-28T20:51:54.544555+0000","last_fullsized":"2023-11-28T20:51:54.544555+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:15:53.716050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.548425+0000","last_change":"2023-11-28T20:51:53.547064+0000","last_active":"2023-11-28T20:51:53.548425+0000","last_peered":"2023-11-28T20:51:53.548425+0000","last_clean":"2023-11-28T20:51:53.548425+0000","last_became_active":"2023-11-28T20:51:53.546800+0000","last_became_peered":"2023-11-28T20:51:53.546800+0000","last_unstale":"2023-11-28T20:51:53.548425+0000","last_undegraded":"2023-11-28T20:51:53.548425+0000","last_fullsized":"2023-11-28T20:51:53.548425+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:47:47.219724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.526120+0000","last_change":"2023-11-28T20:51:52.498124+0000","last_active":"2023-11-28T20:51:53.526120+0000","last_peered":"2023-11-28T20:51:53.526120+0000","last_clean":"2023-11-28T20:51:53.526120+0000","last_became_active":"2023-11-28T20:51:52.497564+0000","last_became_peered":"2023-11-28T20:51:52.497564+0000","last_unstale":"2023-11-28T20:51:53.526120+0000","last_undegraded":"2023-11-28T20:51:53.526120+0000","last_fullsized":"2023-11-28T20:51:53.526120+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:43.383167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.547470+0000","last_change":"2023-11-28T20:51:53.543934+0000","last_active":"2023-11-28T20:51:53.547470+0000","last_peered":"2023-11-28T20:51:53.547470+0000","last_clean":"2023-11-28T20:51:53.547470+0000","last_became_active":"2023-11-28T20:51:53.543663+0000","last_became_peered":"2023-11-28T20:51:53.543663+0000","last_unstale":"2023-11-28T20:51:53.547470+0000","last_undegraded":"2023-11-28T20:51:53.547470+0000","last_fullsized":"2023-11-28T20:51:53.547470+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:23:14.887409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.526854+0000","last_change":"2023-11-28T20:51:52.511212+0000","last_active":"2023-11-28T20:51:53.526854+0000","last_peered":"2023-11-28T20:51:53.526854+0000","last_clean":"2023-11-28T20:51:53.526854+0000","last_became_active":"2023-11-28T20:51:52.510917+0000","last_became_peered":"2023-11-28T20:51:52.510917+0000","last_unstale":"2023-11-28T20:51:53.526854+0000","last_undegraded":"2023-11-28T20:51:53.526854+0000","last_fullsized":"2023-11-28T20:51:53.526854+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:40:24.247674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.546174+0000","last_change":"2023-11-28T20:51:53.545387+0000","last_active":"2023-11-28T20:51:53.546174+0000","last_peered":"2023-11-28T20:51:53.546174+0000","last_clean":"2023-11-28T20:51:53.546174+0000","last_became_active":"2023-11-28T20:51:53.545100+0000","last_became_peered":"2023-11-28T20:51:53.545100+0000","last_unstale":"2023-11-28T20:51:53.546174+0000","last_undegraded":"2023-11-28T20:51:53.546174+0000","last_fullsized":"2023-11-28T20:51:53.546174+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:26:07.345244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.510743+0000","last_change":"2023-11-28T20:51:52.502373+0000","last_active":"2023-11-28T20:51:53.510743+0000","last_peered":"2023-11-28T20:51:53.510743+0000","last_clean":"2023-11-28T20:51:53.510743+0000","last_became_active":"2023-11-28T20:51:52.501029+0000","last_became_peered":"2023-11-28T20:51:52.501029+0000","last_unstale":"2023-11-28T20:51:53.510743+0000","last_undegraded":"2023-11-28T20:51:53.510743+0000","last_fullsized":"2023-11-28T20:51:53.510743+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:44:13.921193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"115'4286","reported_seq":4629,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.894325+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:51:54.894325+0000","last_peered":"2023-11-28T20:51:54.894325+0000","last_clean":"2023-11-28T20:51:54.894325+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:51:54.894325+0000","last_undegraded":"2023-11-28T20:51:54.894325+0000","last_fullsized":"2023-11-28T20:51:54.894325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":710,"num_read_kb":1360,"num_write":271,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":710,"num_read_kb":1360,"num_write":271,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4286,"ondisk_log_size":4286,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575063,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63680,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308160,"statfs":{"total":96636764160,"available":96571555840,"internally_reserved":0,"allocated":2019328,"data_stored":1303675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.1519999999999999,"15min":1.1879999999999999},"min":{"1min":0.66400000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.6679999999999999,"5min":1.722,"15min":1.786},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.1379999999999999,"15min":1.1779999999999999},"min":{"1min":0.72199999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.64600000000000002}]},{"osd":1,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.163,"15min":1.1899999999999999},"min":{"1min":0.72599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.744},"last":1.504},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.226,"15min":1.248},"min":{"1min":0.747,"5min":0.624,"15min":0.624},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.889},"last":1.4550000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2729999999999999,"15min":1.302},"min":{"1min":0.873,"5min":0.80300000000000005,"15min":0.58799999999999997},"max":{"1min":1.6919999999999999,"5min":1.8300000000000001,"15min":1.9359999999999999},"last":0.87},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2929999999999999,"15min":1.294},"min":{"1min":0.92500000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.921,"5min":1.921,"15min":1.921},"last":1.129}]},{"osd":3,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.2809999999999999},"min":{"1min":0.755,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.234},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.385,"15min":1.4039999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.68300000000000005},"max":{"1min":1.6579999999999999,"5min":1.956,"15min":1.956},"last":1.1659999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3,"15min":1.27},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.212},{"interface":"front","average":{"1min":1.343,"5min":1.3320000000000001,"15min":1.327},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.8560000000000001,"5min":1.863,"15min":1.9970000000000001},"last":1.7589999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4019999999999999,"15min":1.3979999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.7909999999999999,"5min":1.9219999999999999,"15min":2.048},"last":1.335},{"interface":"front","average":{"1min":1.47,"5min":1.4790000000000001,"15min":1.4610000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.3089999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.569,"15min":1.5509999999999999},"min":{"1min":0.85799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.431},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5009999999999999,"15min":1.4890000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.1699999999999999},"last":1.395}]}]},{"osd":6,"up_from":13,"seq":55834575063,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1437696,"data_stored":715467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.075,"15min":1.115},"min":{"1min":0.65900000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.6319999999999999,"5min":1.665,"15min":1.7350000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.095,"15min":1.129},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8260000000000001},"last":1.2989999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1910000000000001,"15min":1.2609999999999999},"min":{"1min":0.76100000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.706,"5min":1.728,"15min":1.8069999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.325,"15min":1.375},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.7889999999999999,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.94099999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.2450000000000001,"15min":1.2749999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5620000000000001,"5min":1.6879999999999999,"15min":1.71},"last":1.135},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3160000000000001,"15min":1.3620000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.784,"5min":1.851,"15min":1.851},"last":0.995}]},{"osd":3,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.2709999999999999,"15min":1.2949999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6699999999999999,"5min":1.821,"15min":1.9299999999999999},"last":1.071},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.256,"15min":1.284},"min":{"1min":0.879,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7949999999999999},"last":0.78900000000000003}]},{"osd":4,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.2849999999999999,"15min":1.272},"min":{"1min":0.81499999999999995,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.7529999999999999,"5min":1.849,"15min":2.7290000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3009999999999999,"15min":1.3280000000000001},"min":{"1min":0.89200000000000002,"5min":0.80800000000000005,"15min":0.80100000000000005},"max":{"1min":1.9179999999999999,"5min":2.2010000000000001,"15min":3.2519999999999998},"last":1.4570000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.3149999999999999,"15min":1.333},"min":{"1min":0.91000000000000003,"5min":0.754,"15min":0.62},"max":{"1min":1.857,"5min":1.857,"15min":2.0539999999999998},"last":1.353},{"interface":"front","average":{"1min":1.45,"5min":1.4990000000000001,"15min":1.5429999999999999},"min":{"1min":0.93400000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.8340000000000001,"5min":1.9179999999999999,"15min":3.0310000000000001},"last":1.4039999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4379999999999999,"15min":1.482},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.86499999999999999},"max":{"1min":1.9790000000000001,"5min":2.1120000000000001,"15min":3.379},"last":1.544},{"interface":"front","average":{"1min":1.423,"5min":1.456,"15min":1.4630000000000001},"min":{"1min":0.96099999999999997,"5min":0.93300000000000005,"15min":0.84599999999999997},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":2.0739999999999998},"last":1.498}]}]},{"osd":5,"up_from":13,"seq":55834575063,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50612,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321228,"statfs":{"total":96636764160,"available":96584937472,"internally_reserved":0,"allocated":1417216,"data_stored":713251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.159,"15min":1.175},"min":{"1min":0.72199999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.702,"5min":1.702,"15min":1.768},"last":0.75},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1160000000000001,"15min":1.1879999999999999},"min":{"1min":0.56200000000000006,"5min":0.441,"15min":0.441},"max":{"1min":1.6060000000000001,"5min":1.631,"15min":1.829},"last":1.024}]},{"osd":1,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.115,"15min":1.1499999999999999},"min":{"1min":0.67200000000000004,"5min":0.57399999999999995,"15min":0.48899999999999999},"max":{"1min":1.587,"5min":1.768,"15min":1.768},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.204,"15min":1.218},"min":{"1min":0.72399999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5860000000000001,"5min":1.8879999999999999,"15min":1.956},"last":1.206}]},{"osd":2,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2529999999999999,"15min":1.268},"min":{"1min":0.68600000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.387},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.226,"15min":1.292},"min":{"1min":0.76900000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.958},"last":1.149}]},{"osd":3,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.276,"15min":1.3169999999999999},"min":{"1min":0.77700000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.379,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.2490000000000001,"15min":1.2689999999999999},"min":{"1min":0.83299999999999996,"5min":0.64000000000000001,"15min":0.54600000000000004},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.575}]},{"osd":4,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.284,"15min":1.2949999999999999},"min":{"1min":0.875,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.772,"5min":1.772,"15min":1.8380000000000001},"last":1.29},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.45,"15min":1.4370000000000001},"min":{"1min":1.0760000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.6259999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3029999999999999,"15min":1.325},"min":{"1min":0.82199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":2.093},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.464,"15min":1.464},"min":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.752}]},{"osd":7,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.421,"15min":1.4179999999999999},"min":{"1min":1.0349999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.466,"15min":1.4830000000000001},"min":{"1min":1.0620000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.958,"5min":1.958,"15min":2.1829999999999998},"last":1.254}]}]},{"osd":4,"up_from":13,"seq":55834575064,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36740,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335100,"statfs":{"total":96636764160,"available":96599142400,"internally_reserved":0,"allocated":1425408,"data_stored":713411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0840000000000001,"15min":1.137},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3919999999999999,"5min":1.605,"15min":1.879},"last":1.387},{"interface":"front","average":{"1min":1.026,"5min":1.1020000000000001,"15min":1.121},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.554,"5min":1.891,"15min":1.9750000000000001},"last":0.83299999999999996}]},{"osd":1,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.264,"15min":1.3169999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.542,"5min":1.734,"15min":1.923},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.331,"15min":1.355},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.6779999999999999,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.70499999999999996}]},{"osd":2,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.224,"15min":1.212},"min":{"1min":0.76800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":1.117},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.258,"15min":1.2989999999999999},"min":{"1min":0.84399999999999997,"5min":0.77400000000000002,"15min":0.77200000000000002},"max":{"1min":1.6299999999999999,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.581}]},{"osd":3,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.292,"15min":1.29},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.9630000000000001,"5min":2.157,"15min":2.157},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.3540000000000001,"15min":1.375},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.67900000000000005},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.206}]},{"osd":5,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3089999999999999,"15min":1.2789999999999999},"min":{"1min":0.86699999999999999,"5min":0.86099999999999999,"15min":0.65800000000000003},"max":{"1min":1.9750000000000001,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.706},{"interface":"front","average":{"1min":1.347,"5min":1.339,"15min":1.296},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.79500000000000004},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.6779999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.403,"15min":1.393},"min":{"1min":0.872,"5min":0.872,"15min":0.72899999999999998},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.5569999999999999,"15min":1.552},"min":{"1min":1.147,"5min":1.0369999999999999,"15min":0.91500000000000004},"max":{"1min":2.1349999999999998,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":1.7949999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.4950000000000001,"15min":1.456},"min":{"1min":0.94899999999999995,"5min":0.91300000000000003,"15min":0.85299999999999998},"max":{"1min":2.2490000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.573,"5min":1.5409999999999999,"15min":1.5269999999999999},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.95399999999999996},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.7609999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575064,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40884,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330956,"statfs":{"total":96636764160,"available":96594898944,"internally_reserved":0,"allocated":1417216,"data_stored":713251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.873,"15min":0.90600000000000003},"min":{"1min":0.45500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.6000000000000001,"5min":4.7210000000000001,"15min":4.7210000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.96499999999999997,"15min":1.071},"min":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.4870000000000001,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":0.76400000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.95199999999999996,"15min":0.97499999999999998},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4730000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.1499999999999999,"15min":1.1990000000000001},"min":{"1min":0.58499999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5289999999999999,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.69399999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.1839999999999999,"15min":1.2470000000000001},"min":{"1min":0.55800000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4490000000000001,"5min":3.444,"15min":3.444},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.196,"5min":1.2769999999999999,"15min":1.335},"min":{"1min":0.66800000000000004,"5min":0.58099999999999996,"15min":0.54400000000000004},"max":{"1min":1.7230000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":1.085}]},{"osd":4,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.298,"15min":1.3420000000000001},"min":{"1min":0.84499999999999997,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.6890000000000001,"5min":1.8779999999999999,"15min":2.9420000000000002},"last":1.145},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.327,"15min":1.3839999999999999},"min":{"1min":0.86399999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.1389999999999998},"last":1.208}]},{"osd":5,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4259999999999999,"15min":1.452},"min":{"1min":0.88,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.8959999999999999,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.375,"15min":1.4330000000000001},"min":{"1min":0.79800000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9670000000000001,"5min":2.2429999999999999,"15min":3.1520000000000001},"last":1.3640000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3979999999999999,"15min":1.4790000000000001},"min":{"1min":0.86099999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.3100000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.5309999999999999,"15min":1.577},"min":{"1min":0.91100000000000003,"5min":0.749,"15min":0.749},"max":{"1min":1.7729999999999999,"5min":3.661,"15min":3.661},"last":1.6259999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.45,"15min":1.5369999999999999},"min":{"1min":0.73399999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.863,"5min":2.1469999999999998,"15min":3.266},"last":1.514},{"interface":"front","average":{"1min":1.462,"5min":1.514,"15min":1.601},"min":{"1min":0.90200000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.1269999999999998,"5min":2.3439999999999999,"15min":2.3919999999999999},"last":1.4099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575064,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58932,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312908,"statfs":{"total":96636764160,"available":96576417792,"internally_reserved":0,"allocated":1417216,"data_stored":713251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.85199999999999998,"15min":0.90900000000000003},"min":{"1min":0.373,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.0589999999999999,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.96299999999999997,"15min":1.0620000000000001},"min":{"1min":0.51200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.569,"5min":1.6419999999999999,"15min":1.9390000000000001},"last":0.92400000000000004}]},{"osd":1,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.93200000000000005,"15min":0.98499999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.611,"5min":1.7669999999999999,"15min":3.2599999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.054,"15min":1.1479999999999999},"min":{"1min":0.53700000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.675,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.68899999999999995}]},{"osd":3,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.0900000000000001,"15min":1.157},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.5549999999999999,"5min":1.774,"15min":2.194},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1719999999999999,"15min":1.292},"min":{"1min":0.55600000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":3.1440000000000001},"last":0.48499999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.345,"15min":1.411},"min":{"1min":0.874,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0920000000000001,"5min":2.23,"15min":2.8919999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.335,"5min":1.375,"15min":1.4159999999999999},"min":{"1min":0.90900000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.901,"5min":2.3740000000000001,"15min":2.3809999999999998},"last":1.409}]},{"osd":5,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3520000000000001,"15min":1.4179999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.681,"5min":2.327,"15min":2.327},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.4419999999999999,"15min":1.4970000000000001},"min":{"1min":1.0289999999999999,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.153,"5min":2.1680000000000001,"15min":3.4279999999999999},"last":1.077}]},{"osd":6,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.4279999999999999,"15min":1.502},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.8520000000000001,"5min":2.1709999999999998,"15min":2.621},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.4950000000000001,"15min":1.575},"min":{"1min":0.95599999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.2000000000000002,"5min":2.6749999999999998,"15min":3.6269999999999998},"last":1.036}]},{"osd":7,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4850000000000001,"15min":1.5389999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.7210000000000001,"5min":2.367,"15min":2.367},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.508,"15min":1.587},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.835,"5min":2.4849999999999999,"15min":2.7109999999999999},"last":1.444}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1421312,"data_stored":713321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.875,"15min":0.90100000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.9259999999999999},"last":0.873},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.97699999999999998,"15min":1.071},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.375,"5min":2.028,"15min":2.028},"last":1.3360000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0309999999999999,"15min":1.0880000000000001},"min":{"1min":0.53200000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.48,"5min":1.534,"15min":2.234},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.1539999999999999,"15min":1.206},"min":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.8360000000000001,"5min":1.9199999999999999,"15min":2.895},"last":1.429}]},{"osd":3,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.212,"15min":1.3129999999999999},"min":{"1min":0.58199999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.7849999999999999,"5min":1.893,"15min":2.0459999999999998},"last":1.389},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.044,"15min":1.113},"min":{"1min":0.45100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.7490000000000001,"5min":1.887,"15min":2.3679999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.266,"15min":1.3380000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.657,"5min":1.742,"15min":2.1150000000000002},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3480000000000001,"15min":1.389},"min":{"1min":0.72399999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":2.6819999999999999},"last":1.4970000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3879999999999999,"15min":1.425},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.8109999999999999,"5min":1.903,"15min":2.1840000000000002},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.4350000000000001,"15min":1.496},"min":{"1min":0.89700000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.0219999999999998,"5min":2.1000000000000001,"15min":2.294},"last":1.1719999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.355,"15min":1.4199999999999999},"min":{"1min":0.92300000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7470000000000001,"5min":1.8380000000000001,"15min":1.9379999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.4690000000000001,"15min":1.518},"min":{"1min":0.86199999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.873,"5min":2.1520000000000001,"15min":2.3300000000000001},"last":1.6000000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.49,"15min":1.5580000000000001},"min":{"1min":0.80500000000000005,"5min":0.753,"15min":0.753},"max":{"1min":1.9430000000000001,"5min":2.177,"15min":2.3359999999999999},"last":1.472},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.446,"15min":1.512},"min":{"1min":0.80500000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":2.294},"last":1.2809999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575064,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37332,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334508,"statfs":{"total":96636764160,"available":96598536192,"internally_reserved":0,"allocated":2035712,"data_stored":1305937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.89900000000000002,"15min":0.90600000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.2729999999999999,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.016,"15min":1.085},"min":{"1min":0.60299999999999998,"5min":0.501,"15min":0.501},"max":{"1min":1.6619999999999999,"5min":1.6879999999999999,"15min":1.764},"last":1.171}]},{"osd":2,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.02,"15min":1.109},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.6000000000000001,"5min":1.6220000000000001,"15min":1.7709999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.127,"15min":1.1910000000000001},"min":{"1min":0.67800000000000005,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4510000000000001,"5min":1.893,"15min":1.9550000000000001},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.014,"15min":1.1080000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.325,"5min":2.081,"15min":2.081},"last":1.099},{"interface":"front","average":{"1min":1.115,"5min":1.1990000000000001,"15min":1.3080000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.7030000000000001,"5min":1.8180000000000001,"15min":2.1080000000000001},"last":1.5209999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.363,"15min":1.421},"min":{"1min":0.879,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.802,"5min":2.012,"15min":2.012},"last":1.712},{"interface":"front","average":{"1min":1.286,"5min":1.3400000000000001,"15min":1.377},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.7509999999999999,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.48}]},{"osd":5,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3720000000000001,"15min":1.4099999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4330000000000001,"15min":1.486},"min":{"1min":0.90500000000000003,"5min":0.621,"15min":0.621},"max":{"1min":2.0129999999999999,"5min":2.0750000000000002,"15min":2.2490000000000001},"last":1.3600000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.472,"15min":1.5469999999999999},"min":{"1min":0.73799999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.8919999999999999,"5min":2.1160000000000001,"15min":2.2959999999999998},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.476,"15min":1.5229999999999999},"min":{"1min":0.93899999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.956,"5min":2.274,"15min":2.4209999999999998},"last":1.9079999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.532,"15min":1.589},"min":{"1min":0.93500000000000005,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.8049999999999999,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.9830000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.4650000000000001,"15min":1.5349999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.73799999999999999},"max":{"1min":1.996,"5min":2.3460000000000001,"15min":2.524},"last":1.6120000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:51:58.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:51:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.325+0000 7f3b99151640 1 Processor -- start 2023-11-28T20:51:58.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.325+0000 7f3b99151640 1 -- start start 2023-11-28T20:51:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.327+0000 7f3b99151640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3b9415b850 0x7f3b940b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.327+0000 7f3b99151640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9415bda0 con 0x7f3b940b1fa0 2023-11-28T20:51:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.327+0000 7f3b99151640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9415c080 con 0x7f3b940b35b0 2023-11-28T20:51:58.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.327+0000 7f3b99151640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9415ee40 con 0x7f3b9415b850 2023-11-28T20:51:58.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.328+0000 7f3b9359e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3b940b1fa0 0x7f3b9415f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44154/0 (socket says 172.21.15.45:44154) 2023-11-28T20:51:58.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.328+0000 7f3b9359e640 1 -- 172.21.15.45:0/1803653863 learned_addr learned my addr 172.21.15.45:0/1803653863 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:58.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.328+0000 7f3b8a59c640 1 --2- 172.21.15.45:0/1803653863 >> v2:172.21.15.190:3300/0 conn(0x7f3b9415b850 0x7f3b940b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.328+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714931570 0 0) 0x7f3b9415bda0 con 0x7f3b940b1fa0 2023-11-28T20:51:58.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.328+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b80003520 con 0x7f3b940b1fa0 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3575523674 0 0) 0x7f3b80003520 con 0x7f3b940b1fa0 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 >> v2:172.21.15.190:3300/0 conn(0x7f3b9415b850 msgr2=0x7f3b940b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 --2- 172.21.15.45:0/1803653863 >> v2:172.21.15.190:3300/0 conn(0x7f3b9415b850 0x7f3b940b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 >> v1:172.21.15.190:6789/0 conn(0x7f3b940b35b0 legacy=0x7f3b9415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9415c360 con 0x7f3b940b1fa0 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3b8c001670 con 0x7f3b940b1fa0 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 >> v1:172.21.15.45:6789/0 conn(0x7f3b940b1fa0 legacy=0x7f3b9415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80003970 0x7f3b80003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b800047c0 0x7f3b80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9415ee40 con 0x7f3b80005180 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8000c020 con 0x7f3b80003970 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8000c1e0 con 0x7f3b800047c0 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b9359e640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.329+0000 7f3b92d9d640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80003970 0x7f3b80003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.330+0000 7f3b9359e640 1 -- 172.21.15.45:0/1803653863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b800047c0 msgr2=0x7f3b80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.330+0000 7f3b9359e640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b800047c0 0x7f3b80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.330+0000 7f3b9359e640 1 -- 172.21.15.45:0/1803653863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80003970 msgr2=0x7f3b80003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.330+0000 7f3b9359e640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80003970 0x7f3b80003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.330+0000 7f3b9359e640 1 -- 172.21.15.45:0/1803653863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b80003520 con 0x7f3b80005180 2023-11-28T20:51:58.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.330+0000 7f3b9359e640 1 -- 172.21.15.45:0/1803653863 reap_dead start 2023-11-28T20:51:58.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.331+0000 7f3b9359e640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b8000b980 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00c210 tx=0x7f3b800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ca8f575fea9f87 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:58.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.331+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c005ad0 con 0x7f3b80005180 2023-11-28T20:51:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.331+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c005c30 con 0x7f3b80005180 2023-11-28T20:51:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.331+0000 7f3b9259c640 1 -- 172.21.15.45:0/1803653863 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c02c890 con 0x7f3b80005180 2023-11-28T20:51:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.331+0000 7f3b99151640 1 -- 172.21.15.45:0/1803653863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 msgr2=0x7f3b8000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.331+0000 7f3b99151640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b8000b980 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00c210 tx=0x7f3b800057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.332+0000 7f3b99151640 1 -- 172.21.15.45:0/1803653863 shutdown_connections 2023-11-28T20:51:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.332+0000 7f3b99151640 1 --2- 172.21.15.45:0/1803653863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b8000b980 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.332+0000 7f3b99151640 1 -- 172.21.15.45:0/1803653863 >> 172.21.15.45:0/1803653863 conn(0x7f3b9406a500 msgr2=0x7f3b9406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.332+0000 7f3b99151640 1 -- 172.21.15.45:0/1803653863 shutdown_connections 2023-11-28T20:51:58.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.332+0000 7f3b99151640 1 -- 172.21.15.45:0/1803653863 wait complete. 2023-11-28T20:51:58.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.334+0000 7f3b99151640 1 Processor -- start 2023-11-28T20:51:58.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.334+0000 7f3b99151640 1 -- start start 2023-11-28T20:51:58.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b99151640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b9415a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b99151640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b941f2b90 0x7f3b941f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b99151640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b940b0b90 0x7f3b940b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b99151640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940b1550 con 0x7f3b80005180 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b99151640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941f36c0 con 0x7f3b941f2b90 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b99151640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941f39a0 con 0x7f3b940b0b90 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b92d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b9415a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b92d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b9415a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52550/0 (socket says 172.21.15.45:52550) 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b92d9d640 1 -- 172.21.15.45:0/1603739472 learned_addr learned my addr 172.21.15.45:0/1603739472 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b8a59c640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b941f2b90 0x7f3b941f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b9359e640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b940b0b90 0x7f3b940b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.335+0000 7f3b92d9d640 1 -- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b940b0b90 msgr2=0x7f3b940b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b92d9d640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b940b0b90 0x7f3b940b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b92d9d640 1 -- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b941f2b90 msgr2=0x7f3b941f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b92d9d640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b941f2b90 0x7f3b941f3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b92d9d640 1 -- 172.21.15.45:0/1603739472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b941f3f80 con 0x7f3b80005180 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b92d9d640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b9415a0d0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400e820 tx=0x7f3b8400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b8bfff640 1 -- 172.21.15.45:0/1603739472 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84003b80 con 0x7f3b80005180 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b8bfff640 1 -- 172.21.15.45:0/1603739472 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b84003ce0 con 0x7f3b80005180 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b8bfff640 1 -- 172.21.15.45:0/1603739472 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b840052f0 con 0x7f3b80005180 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b941f4260 con 0x7f3b80005180 2023-11-28T20:51:58.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.336+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b94143880 con 0x7f3b80005180 2023-11-28T20:51:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.337+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b940733d0 con 0x7f3b80005180 2023-11-28T20:51:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.339+0000 7f3b8bfff640 1 -- 172.21.15.45:0/1603739472 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f3b84005450 con 0x7f3b80005180 2023-11-28T20:51:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.339+0000 7f3b8bfff640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f3b7406aed0 0x7f3b7406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.340+0000 7f3b8bfff640 1 -- 172.21.15.45:0/1603739472 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3b84089900 con 0x7f3b80005180 2023-11-28T20:51:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.340+0000 7f3b8a59c640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f3b7406aed0 0x7f3b7406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.341+0000 7f3b8a59c640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f3b7406aed0 0x7f3b7406d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b9406d330 tx=0x7f3b7c008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:58.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.344+0000 7f3b8bfff640 1 -- 172.21.15.45:0/1603739472 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b84054790 con 0x7f3b80005180 2023-11-28T20:51:58.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.537+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 --> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b940b3990 con 0x7f3b7406aed0 2023-11-28T20:51:58.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.563+0000 7f3b8bfff640 1 -- 172.21.15.45:0/1603739472 <== mgr.5828 v2:172.21.15.190:6832/4159844301 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286978 (secure 0 0 0) 0x7f3b7c0107c0 con 0x7f3b7406aed0 2023-11-28T20:51:58.569 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:58.570 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:51:58.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f3b7406aed0 msgr2=0x7f3b7406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f3b7406aed0 0x7f3b7406d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b9406d330 tx=0x7f3b7c008040 comp rx=0 tx=0).stop 2023-11-28T20:51:58.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 msgr2=0x7f3b9415a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b9415a0d0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400e820 tx=0x7f3b8400ece0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 shutdown_connections 2023-11-28T20:51:58.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f3b7406aed0 0x7f3b7406d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b940b0b90 0x7f3b940b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b941f2b90 0x7f3b941f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 --2- 172.21.15.45:0/1603739472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b80005180 0x7f3b9415a0d0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 >> 172.21.15.45:0/1603739472 conn(0x7f3b9406a500 msgr2=0x7f3b94071190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.570+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 shutdown_connections 2023-11-28T20:51:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.571+0000 7f3b99151640 1 -- 172.21.15.45:0/1603739472 wait complete. 2023-11-28T20:51:58.593 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-11-28T20:51:57.907751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":710,"num_read_kb":1360,"num_write":315,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4324,"ondisk_log_size":4324,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":357500,"kb_used_data":12296,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754617220,"statfs":{"total":773094113280,"available":772728033280,"internally_reserved":0,"allocated":12591104,"data_stored":6891564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":353357360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002160"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.553028+0000","last_change":"2023-11-28T20:51:53.544286+0000","last_active":"2023-11-28T20:51:53.553028+0000","last_peered":"2023-11-28T20:51:53.553028+0000","last_clean":"2023-11-28T20:51:53.553028+0000","last_became_active":"2023-11-28T20:51:53.543924+0000","last_became_peered":"2023-11-28T20:51:53.543924+0000","last_unstale":"2023-11-28T20:51:53.553028+0000","last_undegraded":"2023-11-28T20:51:53.553028+0000","last_fullsized":"2023-11-28T20:51:53.553028+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:44:45.759419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544909+0000","last_change":"2023-11-28T20:51:53.546281+0000","last_active":"2023-11-28T20:51:54.544909+0000","last_peered":"2023-11-28T20:51:54.544909+0000","last_clean":"2023-11-28T20:51:54.544909+0000","last_became_active":"2023-11-28T20:51:53.545147+0000","last_became_peered":"2023-11-28T20:51:53.545147+0000","last_unstale":"2023-11-28T20:51:54.544909+0000","last_undegraded":"2023-11-28T20:51:54.544909+0000","last_fullsized":"2023-11-28T20:51:54.544909+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:44:45.759419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.542606+0000","last_change":"2023-11-28T20:51:53.540245+0000","last_active":"2023-11-28T20:51:53.542606+0000","last_peered":"2023-11-28T20:51:53.542606+0000","last_clean":"2023-11-28T20:51:53.542606+0000","last_became_active":"2023-11-28T20:51:53.539525+0000","last_became_peered":"2023-11-28T20:51:53.539525+0000","last_unstale":"2023-11-28T20:51:53.542606+0000","last_undegraded":"2023-11-28T20:51:53.542606+0000","last_fullsized":"2023-11-28T20:51:53.542606+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:57:38.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.552628+0000","last_change":"2023-11-28T20:51:53.537281+0000","last_active":"2023-11-28T20:51:53.552628+0000","last_peered":"2023-11-28T20:51:53.552628+0000","last_clean":"2023-11-28T20:51:53.552628+0000","last_became_active":"2023-11-28T20:51:53.536974+0000","last_became_peered":"2023-11-28T20:51:53.536974+0000","last_unstale":"2023-11-28T20:51:53.552628+0000","last_undegraded":"2023-11-28T20:51:53.552628+0000","last_fullsized":"2023-11-28T20:51:53.552628+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:44.399354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543053+0000","last_change":"2023-11-28T20:51:53.534749+0000","last_active":"2023-11-28T20:51:54.543053+0000","last_peered":"2023-11-28T20:51:54.543053+0000","last_clean":"2023-11-28T20:51:54.543053+0000","last_became_active":"2023-11-28T20:51:53.534175+0000","last_became_peered":"2023-11-28T20:51:53.534175+0000","last_unstale":"2023-11-28T20:51:54.543053+0000","last_undegraded":"2023-11-28T20:51:54.543053+0000","last_fullsized":"2023-11-28T20:51:54.543053+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:13.224682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.549213+0000","last_change":"2023-11-28T20:51:53.546009+0000","last_active":"2023-11-28T20:51:53.549213+0000","last_peered":"2023-11-28T20:51:53.549213+0000","last_clean":"2023-11-28T20:51:53.549213+0000","last_became_active":"2023-11-28T20:51:53.545720+0000","last_became_peered":"2023-11-28T20:51:53.545720+0000","last_unstale":"2023-11-28T20:51:53.549213+0000","last_undegraded":"2023-11-28T20:51:53.549213+0000","last_fullsized":"2023-11-28T20:51:53.549213+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:06.029430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.544998+0000","last_change":"2023-11-28T20:51:53.542990+0000","last_active":"2023-11-28T20:51:53.544998+0000","last_peered":"2023-11-28T20:51:53.544998+0000","last_clean":"2023-11-28T20:51:53.544998+0000","last_became_active":"2023-11-28T20:51:53.542642+0000","last_became_peered":"2023-11-28T20:51:53.542642+0000","last_unstale":"2023-11-28T20:51:53.544998+0000","last_undegraded":"2023-11-28T20:51:53.544998+0000","last_fullsized":"2023-11-28T20:51:53.544998+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:44:45.759419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.545781+0000","last_change":"2023-11-28T20:51:53.531982+0000","last_active":"2023-11-28T20:51:53.545781+0000","last_peered":"2023-11-28T20:51:53.545781+0000","last_clean":"2023-11-28T20:51:53.545781+0000","last_became_active":"2023-11-28T20:51:53.531534+0000","last_became_peered":"2023-11-28T20:51:53.531534+0000","last_unstale":"2023-11-28T20:51:53.545781+0000","last_undegraded":"2023-11-28T20:51:53.545781+0000","last_fullsized":"2023-11-28T20:51:53.545781+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:42.893430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544402+0000","last_change":"2023-11-28T20:51:53.530638+0000","last_active":"2023-11-28T20:51:54.544402+0000","last_peered":"2023-11-28T20:51:54.544402+0000","last_clean":"2023-11-28T20:51:54.544402+0000","last_became_active":"2023-11-28T20:51:53.528389+0000","last_became_peered":"2023-11-28T20:51:53.528389+0000","last_unstale":"2023-11-28T20:51:54.544402+0000","last_undegraded":"2023-11-28T20:51:54.544402+0000","last_fullsized":"2023-11-28T20:51:54.544402+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:49:07.458753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535709+0000","last_change":"2023-11-28T20:51:53.531803+0000","last_active":"2023-11-28T20:51:54.535709+0000","last_peered":"2023-11-28T20:51:54.535709+0000","last_clean":"2023-11-28T20:51:54.535709+0000","last_became_active":"2023-11-28T20:51:53.531478+0000","last_became_peered":"2023-11-28T20:51:53.531478+0000","last_unstale":"2023-11-28T20:51:54.535709+0000","last_undegraded":"2023-11-28T20:51:54.535709+0000","last_fullsized":"2023-11-28T20:51:54.535709+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:17.150958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542913+0000","last_change":"2023-11-28T20:51:53.541664+0000","last_active":"2023-11-28T20:51:54.542913+0000","last_peered":"2023-11-28T20:51:54.542913+0000","last_clean":"2023-11-28T20:51:54.542913+0000","last_became_active":"2023-11-28T20:51:53.541376+0000","last_became_peered":"2023-11-28T20:51:53.541376+0000","last_unstale":"2023-11-28T20:51:54.542913+0000","last_undegraded":"2023-11-28T20:51:54.542913+0000","last_fullsized":"2023-11-28T20:51:54.542913+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:21:07.082883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535286+0000","last_change":"2023-11-28T20:51:53.544372+0000","last_active":"2023-11-28T20:51:54.535286+0000","last_peered":"2023-11-28T20:51:54.535286+0000","last_clean":"2023-11-28T20:51:54.535286+0000","last_became_active":"2023-11-28T20:51:53.544089+0000","last_became_peered":"2023-11-28T20:51:53.544089+0000","last_unstale":"2023-11-28T20:51:54.535286+0000","last_undegraded":"2023-11-28T20:51:54.535286+0000","last_fullsized":"2023-11-28T20:51:54.535286+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:35.784935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.550866+0000","last_change":"2023-11-28T20:51:53.537622+0000","last_active":"2023-11-28T20:51:53.550866+0000","last_peered":"2023-11-28T20:51:53.550866+0000","last_clean":"2023-11-28T20:51:53.550866+0000","last_became_active":"2023-11-28T20:51:53.532788+0000","last_became_peered":"2023-11-28T20:51:53.532788+0000","last_unstale":"2023-11-28T20:51:53.550866+0000","last_undegraded":"2023-11-28T20:51:53.550866+0000","last_fullsized":"2023-11-28T20:51:53.550866+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:37:33.868935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542058+0000","last_change":"2023-11-28T20:51:53.546780+0000","last_active":"2023-11-28T20:51:54.542058+0000","last_peered":"2023-11-28T20:51:54.542058+0000","last_clean":"2023-11-28T20:51:54.542058+0000","last_became_active":"2023-11-28T20:51:53.546503+0000","last_became_peered":"2023-11-28T20:51:53.546503+0000","last_unstale":"2023-11-28T20:51:54.542058+0000","last_undegraded":"2023-11-28T20:51:54.542058+0000","last_fullsized":"2023-11-28T20:51:54.542058+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:15:47.919713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557685+0000","last_change":"2023-11-28T20:51:53.553491+0000","last_active":"2023-11-28T20:51:53.557685+0000","last_peered":"2023-11-28T20:51:53.557685+0000","last_clean":"2023-11-28T20:51:53.557685+0000","last_became_active":"2023-11-28T20:51:53.553186+0000","last_became_peered":"2023-11-28T20:51:53.553186+0000","last_unstale":"2023-11-28T20:51:53.557685+0000","last_undegraded":"2023-11-28T20:51:53.557685+0000","last_fullsized":"2023-11-28T20:51:53.557685+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:51.868674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.555291+0000","last_change":"2023-11-28T20:51:53.544132+0000","last_active":"2023-11-28T20:51:53.555291+0000","last_peered":"2023-11-28T20:51:53.555291+0000","last_clean":"2023-11-28T20:51:53.555291+0000","last_became_active":"2023-11-28T20:51:53.543821+0000","last_became_peered":"2023-11-28T20:51:53.543821+0000","last_unstale":"2023-11-28T20:51:53.555291+0000","last_undegraded":"2023-11-28T20:51:53.555291+0000","last_fullsized":"2023-11-28T20:51:53.555291+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:55:54.519619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.551156+0000","last_change":"2023-11-28T20:51:53.550005+0000","last_active":"2023-11-28T20:51:53.551156+0000","last_peered":"2023-11-28T20:51:53.551156+0000","last_clean":"2023-11-28T20:51:53.551156+0000","last_became_active":"2023-11-28T20:51:53.549647+0000","last_became_peered":"2023-11-28T20:51:53.549647+0000","last_unstale":"2023-11-28T20:51:53.551156+0000","last_undegraded":"2023-11-28T20:51:53.551156+0000","last_fullsized":"2023-11-28T20:51:53.551156+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:35:24.953834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.553083+0000","last_change":"2023-11-28T20:51:53.543792+0000","last_active":"2023-11-28T20:51:53.553083+0000","last_peered":"2023-11-28T20:51:53.553083+0000","last_clean":"2023-11-28T20:51:53.553083+0000","last_became_active":"2023-11-28T20:51:53.543440+0000","last_became_peered":"2023-11-28T20:51:53.543440+0000","last_unstale":"2023-11-28T20:51:53.553083+0000","last_undegraded":"2023-11-28T20:51:53.553083+0000","last_fullsized":"2023-11-28T20:51:53.553083+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:09.278310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543282+0000","last_change":"2023-11-28T20:51:53.547334+0000","last_active":"2023-11-28T20:51:54.543282+0000","last_peered":"2023-11-28T20:51:54.543282+0000","last_clean":"2023-11-28T20:51:54.543282+0000","last_became_active":"2023-11-28T20:51:53.547069+0000","last_became_peered":"2023-11-28T20:51:53.547069+0000","last_unstale":"2023-11-28T20:51:54.543282+0000","last_undegraded":"2023-11-28T20:51:54.543282+0000","last_fullsized":"2023-11-28T20:51:54.543282+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:12.700814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557743+0000","last_change":"2023-11-28T20:51:53.553071+0000","last_active":"2023-11-28T20:51:53.557743+0000","last_peered":"2023-11-28T20:51:53.557743+0000","last_clean":"2023-11-28T20:51:53.557743+0000","last_became_active":"2023-11-28T20:51:53.552746+0000","last_became_peered":"2023-11-28T20:51:53.552746+0000","last_unstale":"2023-11-28T20:51:53.557743+0000","last_undegraded":"2023-11-28T20:51:53.557743+0000","last_fullsized":"2023-11-28T20:51:53.557743+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:11.195412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543389+0000","last_change":"2023-11-28T20:51:53.531880+0000","last_active":"2023-11-28T20:51:54.543389+0000","last_peered":"2023-11-28T20:51:54.543389+0000","last_clean":"2023-11-28T20:51:54.543389+0000","last_became_active":"2023-11-28T20:51:53.530832+0000","last_became_peered":"2023-11-28T20:51:53.530832+0000","last_unstale":"2023-11-28T20:51:54.543389+0000","last_undegraded":"2023-11-28T20:51:54.543389+0000","last_fullsized":"2023-11-28T20:51:54.543389+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:49:42.227913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542288+0000","last_change":"2023-11-28T20:51:53.533580+0000","last_active":"2023-11-28T20:51:54.542288+0000","last_peered":"2023-11-28T20:51:54.542288+0000","last_clean":"2023-11-28T20:51:54.542288+0000","last_became_active":"2023-11-28T20:51:53.533194+0000","last_became_peered":"2023-11-28T20:51:53.533194+0000","last_unstale":"2023-11-28T20:51:54.542288+0000","last_undegraded":"2023-11-28T20:51:54.542288+0000","last_fullsized":"2023-11-28T20:51:54.542288+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:19:22.788175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.555727+0000","last_change":"2023-11-28T20:51:53.552798+0000","last_active":"2023-11-28T20:51:53.555727+0000","last_peered":"2023-11-28T20:51:53.555727+0000","last_clean":"2023-11-28T20:51:53.555727+0000","last_became_active":"2023-11-28T20:51:53.552588+0000","last_became_peered":"2023-11-28T20:51:53.552588+0000","last_unstale":"2023-11-28T20:51:53.555727+0000","last_undegraded":"2023-11-28T20:51:53.555727+0000","last_fullsized":"2023-11-28T20:51:53.555727+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:09:04.070926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.539459+0000","last_change":"2023-11-28T20:51:53.533528+0000","last_active":"2023-11-28T20:51:53.539459+0000","last_peered":"2023-11-28T20:51:53.539459+0000","last_clean":"2023-11-28T20:51:53.539459+0000","last_became_active":"2023-11-28T20:51:53.533183+0000","last_became_peered":"2023-11-28T20:51:53.533183+0000","last_unstale":"2023-11-28T20:51:53.539459+0000","last_undegraded":"2023-11-28T20:51:53.539459+0000","last_fullsized":"2023-11-28T20:51:53.539459+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:46.909178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.545172+0000","last_change":"2023-11-28T20:51:53.529982+0000","last_active":"2023-11-28T20:51:54.545172+0000","last_peered":"2023-11-28T20:51:54.545172+0000","last_clean":"2023-11-28T20:51:54.545172+0000","last_became_active":"2023-11-28T20:51:53.529685+0000","last_became_peered":"2023-11-28T20:51:53.529685+0000","last_unstale":"2023-11-28T20:51:54.545172+0000","last_undegraded":"2023-11-28T20:51:54.545172+0000","last_fullsized":"2023-11-28T20:51:54.545172+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:26:07.345244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542434+0000","last_change":"2023-11-28T20:51:53.533307+0000","last_active":"2023-11-28T20:51:54.542434+0000","last_peered":"2023-11-28T20:51:54.542434+0000","last_clean":"2023-11-28T20:51:54.542434+0000","last_became_active":"2023-11-28T20:51:53.532972+0000","last_became_peered":"2023-11-28T20:51:53.532972+0000","last_unstale":"2023-11-28T20:51:54.542434+0000","last_undegraded":"2023-11-28T20:51:54.542434+0000","last_fullsized":"2023-11-28T20:51:54.542434+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:02.840995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535791+0000","last_change":"2023-11-28T20:51:53.541833+0000","last_active":"2023-11-28T20:51:54.535791+0000","last_peered":"2023-11-28T20:51:54.535791+0000","last_clean":"2023-11-28T20:51:54.535791+0000","last_became_active":"2023-11-28T20:51:53.539098+0000","last_became_peered":"2023-11-28T20:51:53.539098+0000","last_unstale":"2023-11-28T20:51:54.535791+0000","last_undegraded":"2023-11-28T20:51:54.535791+0000","last_fullsized":"2023-11-28T20:51:54.535791+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:52:55.647413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.552608+0000","last_change":"2023-11-28T20:51:53.533209+0000","last_active":"2023-11-28T20:51:53.552608+0000","last_peered":"2023-11-28T20:51:53.552608+0000","last_clean":"2023-11-28T20:51:53.552608+0000","last_became_active":"2023-11-28T20:51:53.532903+0000","last_became_peered":"2023-11-28T20:51:53.532903+0000","last_unstale":"2023-11-28T20:51:53.552608+0000","last_undegraded":"2023-11-28T20:51:53.552608+0000","last_fullsized":"2023-11-28T20:51:53.552608+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:43.717714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.555945+0000","last_change":"2023-11-28T20:51:53.554484+0000","last_active":"2023-11-28T20:51:53.555945+0000","last_peered":"2023-11-28T20:51:53.555945+0000","last_clean":"2023-11-28T20:51:53.555945+0000","last_became_active":"2023-11-28T20:51:53.554196+0000","last_became_peered":"2023-11-28T20:51:53.554196+0000","last_unstale":"2023-11-28T20:51:53.555945+0000","last_undegraded":"2023-11-28T20:51:53.555945+0000","last_fullsized":"2023-11-28T20:51:53.555945+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:06.560961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542728+0000","last_change":"2023-11-28T20:51:53.527930+0000","last_active":"2023-11-28T20:51:54.542728+0000","last_peered":"2023-11-28T20:51:54.542728+0000","last_clean":"2023-11-28T20:51:54.542728+0000","last_became_active":"2023-11-28T20:51:53.523324+0000","last_became_peered":"2023-11-28T20:51:53.523324+0000","last_unstale":"2023-11-28T20:51:54.542728+0000","last_undegraded":"2023-11-28T20:51:54.542728+0000","last_fullsized":"2023-11-28T20:51:54.542728+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:57:08.910602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543642+0000","last_change":"2023-11-28T20:51:53.533882+0000","last_active":"2023-11-28T20:51:54.543642+0000","last_peered":"2023-11-28T20:51:54.543642+0000","last_clean":"2023-11-28T20:51:54.543642+0000","last_became_active":"2023-11-28T20:51:53.533503+0000","last_became_peered":"2023-11-28T20:51:53.533503+0000","last_unstale":"2023-11-28T20:51:54.543642+0000","last_undegraded":"2023-11-28T20:51:54.543642+0000","last_fullsized":"2023-11-28T20:51:54.543642+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:46.909178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.539864+0000","last_change":"2023-11-28T20:51:53.538125+0000","last_active":"2023-11-28T20:51:53.539864+0000","last_peered":"2023-11-28T20:51:53.539864+0000","last_clean":"2023-11-28T20:51:53.539864+0000","last_became_active":"2023-11-28T20:51:53.537744+0000","last_became_peered":"2023-11-28T20:51:53.537744+0000","last_unstale":"2023-11-28T20:51:53.539864+0000","last_undegraded":"2023-11-28T20:51:53.539864+0000","last_fullsized":"2023-11-28T20:51:53.539864+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:59.208311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.549143+0000","last_change":"2023-11-28T20:51:53.542050+0000","last_active":"2023-11-28T20:51:53.549143+0000","last_peered":"2023-11-28T20:51:53.549143+0000","last_clean":"2023-11-28T20:51:53.549143+0000","last_became_active":"2023-11-28T20:51:53.541810+0000","last_became_peered":"2023-11-28T20:51:53.541810+0000","last_unstale":"2023-11-28T20:51:53.549143+0000","last_undegraded":"2023-11-28T20:51:53.549143+0000","last_fullsized":"2023-11-28T20:51:53.549143+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:50:04.881346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"115'4","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.566334+0000","last_change":"2023-11-28T20:51:52.509651+0000","last_active":"2023-11-28T20:51:53.566334+0000","last_peered":"2023-11-28T20:51:53.566334+0000","last_clean":"2023-11-28T20:51:53.566334+0000","last_became_active":"2023-11-28T20:51:52.509187+0000","last_became_peered":"2023-11-28T20:51:52.509187+0000","last_unstale":"2023-11-28T20:51:53.566334+0000","last_undegraded":"2023-11-28T20:51:53.566334+0000","last_fullsized":"2023-11-28T20:51:53.566334+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:15:58.123873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557879+0000","last_change":"2023-11-28T20:51:53.554699+0000","last_active":"2023-11-28T20:51:53.557879+0000","last_peered":"2023-11-28T20:51:53.557879+0000","last_clean":"2023-11-28T20:51:53.557879+0000","last_became_active":"2023-11-28T20:51:53.554389+0000","last_became_peered":"2023-11-28T20:51:53.554389+0000","last_unstale":"2023-11-28T20:51:53.557879+0000","last_undegraded":"2023-11-28T20:51:53.557879+0000","last_fullsized":"2023-11-28T20:51:53.557879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:31.154886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"115'1","reported_seq":13,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542920+0000","last_change":"2023-11-28T20:51:52.502164+0000","last_active":"2023-11-28T20:51:54.542920+0000","last_peered":"2023-11-28T20:51:54.542920+0000","last_clean":"2023-11-28T20:51:54.542920+0000","last_became_active":"2023-11-28T20:51:52.501281+0000","last_became_peered":"2023-11-28T20:51:52.501281+0000","last_unstale":"2023-11-28T20:51:54.542920+0000","last_undegraded":"2023-11-28T20:51:54.542920+0000","last_fullsized":"2023-11-28T20:51:54.542920+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:33.887418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.548261+0000","last_change":"2023-11-28T20:51:53.546939+0000","last_active":"2023-11-28T20:51:53.548261+0000","last_peered":"2023-11-28T20:51:53.548261+0000","last_clean":"2023-11-28T20:51:53.548261+0000","last_became_active":"2023-11-28T20:51:53.546662+0000","last_became_peered":"2023-11-28T20:51:53.546662+0000","last_unstale":"2023-11-28T20:51:53.548261+0000","last_undegraded":"2023-11-28T20:51:53.548261+0000","last_fullsized":"2023-11-28T20:51:53.548261+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:09.278310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.561423+0000","last_change":"2023-11-28T20:51:52.508214+0000","last_active":"2023-11-28T20:51:53.561423+0000","last_peered":"2023-11-28T20:51:53.561423+0000","last_clean":"2023-11-28T20:51:53.561423+0000","last_became_active":"2023-11-28T20:51:52.507929+0000","last_became_peered":"2023-11-28T20:51:52.507929+0000","last_unstale":"2023-11-28T20:51:53.561423+0000","last_undegraded":"2023-11-28T20:51:53.561423+0000","last_fullsized":"2023-11-28T20:51:53.561423+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:57:37.832266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.544662+0000","last_change":"2023-11-28T20:51:53.541488+0000","last_active":"2023-11-28T20:51:53.544662+0000","last_peered":"2023-11-28T20:51:53.544662+0000","last_clean":"2023-11-28T20:51:53.544662+0000","last_became_active":"2023-11-28T20:51:53.541130+0000","last_became_peered":"2023-11-28T20:51:53.541130+0000","last_unstale":"2023-11-28T20:51:53.544662+0000","last_undegraded":"2023-11-28T20:51:53.544662+0000","last_fullsized":"2023-11-28T20:51:53.544662+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:45:00.161448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.565682+0000","last_change":"2023-11-28T20:51:52.507786+0000","last_active":"2023-11-28T20:51:53.565682+0000","last_peered":"2023-11-28T20:51:53.565682+0000","last_clean":"2023-11-28T20:51:53.565682+0000","last_became_active":"2023-11-28T20:51:52.507514+0000","last_became_peered":"2023-11-28T20:51:52.507514+0000","last_unstale":"2023-11-28T20:51:53.565682+0000","last_undegraded":"2023-11-28T20:51:53.565682+0000","last_fullsized":"2023-11-28T20:51:53.565682+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:25.257706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.550506+0000","last_change":"2023-11-28T20:51:53.540465+0000","last_active":"2023-11-28T20:51:53.550506+0000","last_peered":"2023-11-28T20:51:53.550506+0000","last_clean":"2023-11-28T20:51:53.550506+0000","last_became_active":"2023-11-28T20:51:53.536764+0000","last_became_peered":"2023-11-28T20:51:53.536764+0000","last_unstale":"2023-11-28T20:51:53.550506+0000","last_undegraded":"2023-11-28T20:51:53.550506+0000","last_fullsized":"2023-11-28T20:51:53.550506+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:26.273894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"115'1","reported_seq":13,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543641+0000","last_change":"2023-11-28T20:51:52.492944+0000","last_active":"2023-11-28T20:51:54.543641+0000","last_peered":"2023-11-28T20:51:54.543641+0000","last_clean":"2023-11-28T20:51:54.543641+0000","last_became_active":"2023-11-28T20:51:52.492350+0000","last_became_peered":"2023-11-28T20:51:52.492350+0000","last_unstale":"2023-11-28T20:51:54.543641+0000","last_undegraded":"2023-11-28T20:51:54.543641+0000","last_fullsized":"2023-11-28T20:51:54.543641+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:43.383167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.547568+0000","last_change":"2023-11-28T20:51:53.544104+0000","last_active":"2023-11-28T20:51:53.547568+0000","last_peered":"2023-11-28T20:51:53.547568+0000","last_clean":"2023-11-28T20:51:53.547568+0000","last_became_active":"2023-11-28T20:51:53.543784+0000","last_became_peered":"2023-11-28T20:51:53.543784+0000","last_unstale":"2023-11-28T20:51:53.547568+0000","last_undegraded":"2023-11-28T20:51:53.547568+0000","last_fullsized":"2023-11-28T20:51:53.547568+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:00:30.572025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.565729+0000","last_change":"2023-11-28T20:51:52.488209+0000","last_active":"2023-11-28T20:51:53.565729+0000","last_peered":"2023-11-28T20:51:53.565729+0000","last_clean":"2023-11-28T20:51:53.565729+0000","last_became_active":"2023-11-28T20:51:52.487787+0000","last_became_peered":"2023-11-28T20:51:52.487787+0000","last_unstale":"2023-11-28T20:51:53.565729+0000","last_undegraded":"2023-11-28T20:51:53.565729+0000","last_fullsized":"2023-11-28T20:51:53.565729+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:00:29.555837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.537344+0000","last_change":"2023-11-28T20:51:53.534963+0000","last_active":"2023-11-28T20:51:53.537344+0000","last_peered":"2023-11-28T20:51:53.537344+0000","last_clean":"2023-11-28T20:51:53.537344+0000","last_became_active":"2023-11-28T20:51:53.534666+0000","last_became_peered":"2023-11-28T20:51:53.534666+0000","last_unstale":"2023-11-28T20:51:53.537344+0000","last_undegraded":"2023-11-28T20:51:53.537344+0000","last_fullsized":"2023-11-28T20:51:53.537344+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:21:56.699930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.536036+0000","last_change":"2023-11-28T20:51:52.503213+0000","last_active":"2023-11-28T20:51:53.536036+0000","last_peered":"2023-11-28T20:51:53.536036+0000","last_clean":"2023-11-28T20:51:53.536036+0000","last_became_active":"2023-11-28T20:51:52.502952+0000","last_became_peered":"2023-11-28T20:51:52.502952+0000","last_unstale":"2023-11-28T20:51:53.536036+0000","last_undegraded":"2023-11-28T20:51:53.536036+0000","last_fullsized":"2023-11-28T20:51:53.536036+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:23.244332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.548724+0000","last_change":"2023-11-28T20:51:53.535283+0000","last_active":"2023-11-28T20:51:53.548724+0000","last_peered":"2023-11-28T20:51:53.548724+0000","last_clean":"2023-11-28T20:51:53.548724+0000","last_became_active":"2023-11-28T20:51:53.534988+0000","last_became_peered":"2023-11-28T20:51:53.534988+0000","last_unstale":"2023-11-28T20:51:53.548724+0000","last_undegraded":"2023-11-28T20:51:53.548724+0000","last_fullsized":"2023-11-28T20:51:53.548724+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:15:59.140061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.526278+0000","last_change":"2023-11-28T20:51:52.510669+0000","last_active":"2023-11-28T20:51:53.526278+0000","last_peered":"2023-11-28T20:51:53.526278+0000","last_clean":"2023-11-28T20:51:53.526278+0000","last_became_active":"2023-11-28T20:51:52.507728+0000","last_became_peered":"2023-11-28T20:51:52.507728+0000","last_unstale":"2023-11-28T20:51:53.526278+0000","last_undegraded":"2023-11-28T20:51:53.526278+0000","last_fullsized":"2023-11-28T20:51:53.526278+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:45.892991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543362+0000","last_change":"2023-11-28T20:51:53.540615+0000","last_active":"2023-11-28T20:51:54.543362+0000","last_peered":"2023-11-28T20:51:54.543362+0000","last_clean":"2023-11-28T20:51:54.543362+0000","last_became_active":"2023-11-28T20:51:53.540282+0000","last_became_peered":"2023-11-28T20:51:53.540282+0000","last_unstale":"2023-11-28T20:51:54.543362+0000","last_undegraded":"2023-11-28T20:51:54.543362+0000","last_fullsized":"2023-11-28T20:51:54.543362+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:55:23.863772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.545244+0000","last_change":"2023-11-28T20:51:52.500440+0000","last_active":"2023-11-28T20:51:54.545244+0000","last_peered":"2023-11-28T20:51:54.545244+0000","last_clean":"2023-11-28T20:51:54.545244+0000","last_became_active":"2023-11-28T20:51:52.500003+0000","last_became_peered":"2023-11-28T20:51:52.500003+0000","last_unstale":"2023-11-28T20:51:54.545244+0000","last_undegraded":"2023-11-28T20:51:54.545244+0000","last_fullsized":"2023-11-28T20:51:54.545244+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:57:41.133836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.540750+0000","last_change":"2023-11-28T20:51:53.539220+0000","last_active":"2023-11-28T20:51:53.540750+0000","last_peered":"2023-11-28T20:51:53.540750+0000","last_clean":"2023-11-28T20:51:53.540750+0000","last_became_active":"2023-11-28T20:51:53.538922+0000","last_became_peered":"2023-11-28T20:51:53.538922+0000","last_unstale":"2023-11-28T20:51:53.540750+0000","last_undegraded":"2023-11-28T20:51:53.540750+0000","last_fullsized":"2023-11-28T20:51:53.540750+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:21:56.699930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.531870+0000","last_change":"2023-11-28T20:51:52.485182+0000","last_active":"2023-11-28T20:51:53.531870+0000","last_peered":"2023-11-28T20:51:53.531870+0000","last_clean":"2023-11-28T20:51:53.531870+0000","last_became_active":"2023-11-28T20:51:52.484534+0000","last_became_peered":"2023-11-28T20:51:52.484534+0000","last_unstale":"2023-11-28T20:51:53.531870+0000","last_undegraded":"2023-11-28T20:51:53.531870+0000","last_fullsized":"2023-11-28T20:51:53.531870+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:26:06.329057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.554854+0000","last_change":"2023-11-28T20:51:53.547157+0000","last_active":"2023-11-28T20:51:53.554854+0000","last_peered":"2023-11-28T20:51:53.554854+0000","last_clean":"2023-11-28T20:51:53.554854+0000","last_became_active":"2023-11-28T20:51:53.546863+0000","last_became_peered":"2023-11-28T20:51:53.546863+0000","last_unstale":"2023-11-28T20:51:53.554854+0000","last_undegraded":"2023-11-28T20:51:53.554854+0000","last_fullsized":"2023-11-28T20:51:53.554854+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:40:25.263861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"115'3","reported_seq":15,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543762+0000","last_change":"2023-11-28T20:51:52.497727+0000","last_active":"2023-11-28T20:51:54.543762+0000","last_peered":"2023-11-28T20:51:54.543762+0000","last_clean":"2023-11-28T20:51:54.543762+0000","last_became_active":"2023-11-28T20:51:52.497394+0000","last_became_peered":"2023-11-28T20:51:52.497394+0000","last_unstale":"2023-11-28T20:51:54.543762+0000","last_undegraded":"2023-11-28T20:51:54.543762+0000","last_fullsized":"2023-11-28T20:51:54.543762+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:44:13.921193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.554246+0000","last_change":"2023-11-28T20:51:53.533344+0000","last_active":"2023-11-28T20:51:53.554246+0000","last_peered":"2023-11-28T20:51:53.554246+0000","last_clean":"2023-11-28T20:51:53.554246+0000","last_became_active":"2023-11-28T20:51:53.533066+0000","last_became_peered":"2023-11-28T20:51:53.533066+0000","last_unstale":"2023-11-28T20:51:53.554246+0000","last_undegraded":"2023-11-28T20:51:53.554246+0000","last_fullsized":"2023-11-28T20:51:53.554246+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:06:57.200161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"115'6","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.570678+0000","last_change":"2023-11-28T20:51:52.507964+0000","last_active":"2023-11-28T20:51:53.570678+0000","last_peered":"2023-11-28T20:51:53.570678+0000","last_clean":"2023-11-28T20:51:53.570678+0000","last_became_active":"2023-11-28T20:51:52.507673+0000","last_became_peered":"2023-11-28T20:51:52.507673+0000","last_unstale":"2023-11-28T20:51:53.570678+0000","last_undegraded":"2023-11-28T20:51:53.570678+0000","last_fullsized":"2023-11-28T20:51:53.570678+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:02:41.250452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.545760+0000","last_change":"2023-11-28T20:51:53.531801+0000","last_active":"2023-11-28T20:51:53.545760+0000","last_peered":"2023-11-28T20:51:53.545760+0000","last_clean":"2023-11-28T20:51:53.545760+0000","last_became_active":"2023-11-28T20:51:53.531153+0000","last_became_peered":"2023-11-28T20:51:53.531153+0000","last_unstale":"2023-11-28T20:51:53.545760+0000","last_undegraded":"2023-11-28T20:51:53.545760+0000","last_fullsized":"2023-11-28T20:51:53.545760+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:34.903606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.564179+0000","last_change":"2023-11-28T20:51:52.511046+0000","last_active":"2023-11-28T20:51:53.564179+0000","last_peered":"2023-11-28T20:51:53.564179+0000","last_clean":"2023-11-28T20:51:53.564179+0000","last_became_active":"2023-11-28T20:51:52.510527+0000","last_became_peered":"2023-11-28T20:51:52.510527+0000","last_unstale":"2023-11-28T20:51:53.564179+0000","last_undegraded":"2023-11-28T20:51:53.564179+0000","last_fullsized":"2023-11-28T20:51:53.564179+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:01.844967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.541952+0000","last_change":"2023-11-28T20:51:53.536735+0000","last_active":"2023-11-28T20:51:54.541952+0000","last_peered":"2023-11-28T20:51:54.541952+0000","last_clean":"2023-11-28T20:51:54.541952+0000","last_became_active":"2023-11-28T20:51:53.536386+0000","last_became_peered":"2023-11-28T20:51:53.536386+0000","last_unstale":"2023-11-28T20:51:54.541952+0000","last_undegraded":"2023-11-28T20:51:54.541952+0000","last_fullsized":"2023-11-28T20:51:54.541952+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:23:15.786207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.519194+0000","last_change":"2023-11-28T20:51:52.507937+0000","last_active":"2023-11-28T20:51:53.519194+0000","last_peered":"2023-11-28T20:51:53.519194+0000","last_clean":"2023-11-28T20:51:53.519194+0000","last_became_active":"2023-11-28T20:51:52.507550+0000","last_became_peered":"2023-11-28T20:51:52.507550+0000","last_unstale":"2023-11-28T20:51:53.519194+0000","last_undegraded":"2023-11-28T20:51:53.519194+0000","last_fullsized":"2023-11-28T20:51:53.519194+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:57:01.544189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535179+0000","last_change":"2023-11-28T20:51:53.532485+0000","last_active":"2023-11-28T20:51:54.535179+0000","last_peered":"2023-11-28T20:51:54.535179+0000","last_clean":"2023-11-28T20:51:54.535179+0000","last_became_active":"2023-11-28T20:51:53.532137+0000","last_became_peered":"2023-11-28T20:51:53.532137+0000","last_unstale":"2023-11-28T20:51:54.535179+0000","last_undegraded":"2023-11-28T20:51:54.535179+0000","last_fullsized":"2023-11-28T20:51:54.535179+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:09.648885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.510878+0000","last_change":"2023-11-28T20:51:52.502989+0000","last_active":"2023-11-28T20:51:53.510878+0000","last_peered":"2023-11-28T20:51:53.510878+0000","last_clean":"2023-11-28T20:51:53.510878+0000","last_became_active":"2023-11-28T20:51:52.501113+0000","last_became_peered":"2023-11-28T20:51:52.501113+0000","last_unstale":"2023-11-28T20:51:53.510878+0000","last_undegraded":"2023-11-28T20:51:53.510878+0000","last_fullsized":"2023-11-28T20:51:53.510878+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:13:05.430726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535581+0000","last_change":"2023-11-28T20:51:53.536365+0000","last_active":"2023-11-28T20:51:54.535581+0000","last_peered":"2023-11-28T20:51:54.535581+0000","last_clean":"2023-11-28T20:51:54.535581+0000","last_became_active":"2023-11-28T20:51:53.536070+0000","last_became_peered":"2023-11-28T20:51:53.536070+0000","last_unstale":"2023-11-28T20:51:54.535581+0000","last_undegraded":"2023-11-28T20:51:54.535581+0000","last_fullsized":"2023-11-28T20:51:54.535581+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:33:05.000766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"115'8","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.567793+0000","last_change":"2023-11-28T20:51:52.488061+0000","last_active":"2023-11-28T20:51:53.567793+0000","last_peered":"2023-11-28T20:51:53.567793+0000","last_clean":"2023-11-28T20:51:53.567793+0000","last_became_active":"2023-11-28T20:51:52.487740+0000","last_became_peered":"2023-11-28T20:51:52.487740+0000","last_unstale":"2023-11-28T20:51:53.567793+0000","last_undegraded":"2023-11-28T20:51:53.567793+0000","last_fullsized":"2023-11-28T20:51:53.567793+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:21:55.994002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.545873+0000","last_change":"2023-11-28T20:51:53.542013+0000","last_active":"2023-11-28T20:51:53.545873+0000","last_peered":"2023-11-28T20:51:53.545873+0000","last_clean":"2023-11-28T20:51:53.545873+0000","last_became_active":"2023-11-28T20:51:53.541455+0000","last_became_peered":"2023-11-28T20:51:53.541455+0000","last_unstale":"2023-11-28T20:51:53.545873+0000","last_undegraded":"2023-11-28T20:51:53.545873+0000","last_fullsized":"2023-11-28T20:51:53.545873+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:59:03.064971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"115'1","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.561769+0000","last_change":"2023-11-28T20:51:52.506560+0000","last_active":"2023-11-28T20:51:53.561769+0000","last_peered":"2023-11-28T20:51:53.561769+0000","last_clean":"2023-11-28T20:51:53.561769+0000","last_became_active":"2023-11-28T20:51:52.505794+0000","last_became_peered":"2023-11-28T20:51:52.505794+0000","last_unstale":"2023-11-28T20:51:53.561769+0000","last_undegraded":"2023-11-28T20:51:53.561769+0000","last_fullsized":"2023-11-28T20:51:53.561769+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:21:55.994002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.539550+0000","last_change":"2023-11-28T20:51:53.535786+0000","last_active":"2023-11-28T20:51:53.539550+0000","last_peered":"2023-11-28T20:51:53.539550+0000","last_clean":"2023-11-28T20:51:53.539550+0000","last_became_active":"2023-11-28T20:51:53.535325+0000","last_became_peered":"2023-11-28T20:51:53.535325+0000","last_unstale":"2023-11-28T20:51:53.539550+0000","last_undegraded":"2023-11-28T20:51:53.539550+0000","last_fullsized":"2023-11-28T20:51:53.539550+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:42.893430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.561175+0000","last_change":"2023-11-28T20:51:52.507128+0000","last_active":"2023-11-28T20:51:53.561175+0000","last_peered":"2023-11-28T20:51:53.561175+0000","last_clean":"2023-11-28T20:51:53.561175+0000","last_became_active":"2023-11-28T20:51:52.506856+0000","last_became_peered":"2023-11-28T20:51:52.506856+0000","last_unstale":"2023-11-28T20:51:53.561175+0000","last_undegraded":"2023-11-28T20:51:53.561175+0000","last_fullsized":"2023-11-28T20:51:53.561175+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:23.244332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.542734+0000","last_change":"2023-11-28T20:51:53.539900+0000","last_active":"2023-11-28T20:51:54.542734+0000","last_peered":"2023-11-28T20:51:54.542734+0000","last_clean":"2023-11-28T20:51:54.542734+0000","last_became_active":"2023-11-28T20:51:53.539578+0000","last_became_peered":"2023-11-28T20:51:53.539578+0000","last_unstale":"2023-11-28T20:51:54.542734+0000","last_undegraded":"2023-11-28T20:51:54.542734+0000","last_fullsized":"2023-11-28T20:51:54.542734+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:58:21.128503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"115'1","reported_seq":13,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544000+0000","last_change":"2023-11-28T20:51:52.489432+0000","last_active":"2023-11-28T20:51:54.544000+0000","last_peered":"2023-11-28T20:51:54.544000+0000","last_clean":"2023-11-28T20:51:54.544000+0000","last_became_active":"2023-11-28T20:51:52.487841+0000","last_became_peered":"2023-11-28T20:51:52.487841+0000","last_unstale":"2023-11-28T20:51:54.544000+0000","last_undegraded":"2023-11-28T20:51:54.544000+0000","last_fullsized":"2023-11-28T20:51:54.544000+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:54.049426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.537960+0000","last_change":"2023-11-28T20:51:53.535321+0000","last_active":"2023-11-28T20:51:53.537960+0000","last_peered":"2023-11-28T20:51:53.537960+0000","last_clean":"2023-11-28T20:51:53.537960+0000","last_became_active":"2023-11-28T20:51:53.535061+0000","last_became_peered":"2023-11-28T20:51:53.535061+0000","last_unstale":"2023-11-28T20:51:53.537960+0000","last_undegraded":"2023-11-28T20:51:53.537960+0000","last_fullsized":"2023-11-28T20:51:53.537960+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:55.838302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"115'2","reported_seq":14,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544754+0000","last_change":"2023-11-28T20:51:52.500521+0000","last_active":"2023-11-28T20:51:54.544754+0000","last_peered":"2023-11-28T20:51:54.544754+0000","last_clean":"2023-11-28T20:51:54.544754+0000","last_became_active":"2023-11-28T20:51:52.498314+0000","last_became_peered":"2023-11-28T20:51:52.498314+0000","last_unstale":"2023-11-28T20:51:54.544754+0000","last_undegraded":"2023-11-28T20:51:54.544754+0000","last_fullsized":"2023-11-28T20:51:54.544754+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:43:23.678537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543726+0000","last_change":"2023-11-28T20:51:53.543024+0000","last_active":"2023-11-28T20:51:54.543726+0000","last_peered":"2023-11-28T20:51:54.543726+0000","last_clean":"2023-11-28T20:51:54.543726+0000","last_became_active":"2023-11-28T20:51:53.542650+0000","last_became_peered":"2023-11-28T20:51:53.542650+0000","last_unstale":"2023-11-28T20:51:54.543726+0000","last_undegraded":"2023-11-28T20:51:54.543726+0000","last_fullsized":"2023-11-28T20:51:54.543726+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:09.278310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535338+0000","last_change":"2023-11-28T20:51:52.485518+0000","last_active":"2023-11-28T20:51:54.535338+0000","last_peered":"2023-11-28T20:51:54.535338+0000","last_clean":"2023-11-28T20:51:54.535338+0000","last_became_active":"2023-11-28T20:51:52.485128+0000","last_became_peered":"2023-11-28T20:51:52.485128+0000","last_unstale":"2023-11-28T20:51:54.535338+0000","last_undegraded":"2023-11-28T20:51:54.535338+0000","last_fullsized":"2023-11-28T20:51:54.535338+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:10:37.523999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544499+0000","last_change":"2023-11-28T20:51:53.533892+0000","last_active":"2023-11-28T20:51:54.544499+0000","last_peered":"2023-11-28T20:51:54.544499+0000","last_clean":"2023-11-28T20:51:54.544499+0000","last_became_active":"2023-11-28T20:51:53.533567+0000","last_became_peered":"2023-11-28T20:51:53.533567+0000","last_unstale":"2023-11-28T20:51:54.544499+0000","last_undegraded":"2023-11-28T20:51:54.544499+0000","last_fullsized":"2023-11-28T20:51:54.544499+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:57:38.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543880+0000","last_change":"2023-11-28T20:51:52.509352+0000","last_active":"2023-11-28T20:51:54.543880+0000","last_peered":"2023-11-28T20:51:54.543880+0000","last_clean":"2023-11-28T20:51:54.543880+0000","last_became_active":"2023-11-28T20:51:52.509097+0000","last_became_peered":"2023-11-28T20:51:52.509097+0000","last_unstale":"2023-11-28T20:51:54.543880+0000","last_undegraded":"2023-11-28T20:51:54.543880+0000","last_fullsized":"2023-11-28T20:51:54.543880+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:26:59.101603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.558843+0000","last_change":"2023-11-28T20:51:53.556522+0000","last_active":"2023-11-28T20:51:53.558843+0000","last_peered":"2023-11-28T20:51:53.558843+0000","last_clean":"2023-11-28T20:51:53.558843+0000","last_became_active":"2023-11-28T20:51:53.555042+0000","last_became_peered":"2023-11-28T20:51:53.555042+0000","last_unstale":"2023-11-28T20:51:53.558843+0000","last_undegraded":"2023-11-28T20:51:53.558843+0000","last_fullsized":"2023-11-28T20:51:53.558843+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:32:37.046552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535010+0000","last_change":"2023-11-28T20:51:52.502790+0000","last_active":"2023-11-28T20:51:54.535010+0000","last_peered":"2023-11-28T20:51:54.535010+0000","last_clean":"2023-11-28T20:51:54.535010+0000","last_became_active":"2023-11-28T20:51:52.502466+0000","last_became_peered":"2023-11-28T20:51:52.502466+0000","last_unstale":"2023-11-28T20:51:54.535010+0000","last_undegraded":"2023-11-28T20:51:54.535010+0000","last_fullsized":"2023-11-28T20:51:54.535010+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:53.258431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.539498+0000","last_change":"2023-11-28T20:51:53.535512+0000","last_active":"2023-11-28T20:51:53.539498+0000","last_peered":"2023-11-28T20:51:53.539498+0000","last_clean":"2023-11-28T20:51:53.539498+0000","last_became_active":"2023-11-28T20:51:53.534118+0000","last_became_peered":"2023-11-28T20:51:53.534118+0000","last_unstale":"2023-11-28T20:51:53.539498+0000","last_undegraded":"2023-11-28T20:51:53.539498+0000","last_fullsized":"2023-11-28T20:51:53.539498+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:29.714777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"115'2","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.563932+0000","last_change":"2023-11-28T20:51:52.507737+0000","last_active":"2023-11-28T20:51:53.563932+0000","last_peered":"2023-11-28T20:51:53.563932+0000","last_clean":"2023-11-28T20:51:53.563932+0000","last_became_active":"2023-11-28T20:51:52.507406+0000","last_became_peered":"2023-11-28T20:51:52.507406+0000","last_unstale":"2023-11-28T20:51:53.563932+0000","last_undegraded":"2023-11-28T20:51:53.563932+0000","last_fullsized":"2023-11-28T20:51:53.563932+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:58.590400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.547495+0000","last_change":"2023-11-28T20:51:53.542993+0000","last_active":"2023-11-28T20:51:53.547495+0000","last_peered":"2023-11-28T20:51:53.547495+0000","last_clean":"2023-11-28T20:51:53.547495+0000","last_became_active":"2023-11-28T20:51:53.542752+0000","last_became_peered":"2023-11-28T20:51:53.542752+0000","last_unstale":"2023-11-28T20:51:53.547495+0000","last_undegraded":"2023-11-28T20:51:53.547495+0000","last_fullsized":"2023-11-28T20:51:53.547495+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:46:45.164961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.543165+0000","last_change":"2023-11-28T20:51:52.495385+0000","last_active":"2023-11-28T20:51:54.543165+0000","last_peered":"2023-11-28T20:51:54.543165+0000","last_clean":"2023-11-28T20:51:54.543165+0000","last_became_active":"2023-11-28T20:51:52.494891+0000","last_became_peered":"2023-11-28T20:51:52.494891+0000","last_unstale":"2023-11-28T20:51:54.543165+0000","last_undegraded":"2023-11-28T20:51:54.543165+0000","last_fullsized":"2023-11-28T20:51:54.543165+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:06:56.183974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557900+0000","last_change":"2023-11-28T20:51:53.554804+0000","last_active":"2023-11-28T20:51:53.557900+0000","last_peered":"2023-11-28T20:51:53.557900+0000","last_clean":"2023-11-28T20:51:53.557900+0000","last_became_active":"2023-11-28T20:51:53.554323+0000","last_became_peered":"2023-11-28T20:51:53.554323+0000","last_unstale":"2023-11-28T20:51:53.557900+0000","last_undegraded":"2023-11-28T20:51:53.557900+0000","last_fullsized":"2023-11-28T20:51:53.557900+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:04:56.995653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544537+0000","last_change":"2023-11-28T20:51:52.510461+0000","last_active":"2023-11-28T20:51:54.544537+0000","last_peered":"2023-11-28T20:51:54.544537+0000","last_clean":"2023-11-28T20:51:54.544537+0000","last_became_active":"2023-11-28T20:51:52.509630+0000","last_became_peered":"2023-11-28T20:51:52.509630+0000","last_unstale":"2023-11-28T20:51:54.544537+0000","last_undegraded":"2023-11-28T20:51:54.544537+0000","last_fullsized":"2023-11-28T20:51:54.544537+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:36:27.725428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.545124+0000","last_change":"2023-11-28T20:51:53.534614+0000","last_active":"2023-11-28T20:51:54.545124+0000","last_peered":"2023-11-28T20:51:54.545124+0000","last_clean":"2023-11-28T20:51:54.545124+0000","last_became_active":"2023-11-28T20:51:53.534323+0000","last_became_peered":"2023-11-28T20:51:53.534323+0000","last_unstale":"2023-11-28T20:51:54.545124+0000","last_undegraded":"2023-11-28T20:51:54.545124+0000","last_fullsized":"2023-11-28T20:51:54.545124+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:40:25.263861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.525913+0000","last_change":"2023-11-28T20:51:52.503666+0000","last_active":"2023-11-28T20:51:53.525913+0000","last_peered":"2023-11-28T20:51:53.525913+0000","last_clean":"2023-11-28T20:51:53.525913+0000","last_became_active":"2023-11-28T20:51:52.503280+0000","last_became_peered":"2023-11-28T20:51:52.503280+0000","last_unstale":"2023-11-28T20:51:53.525913+0000","last_undegraded":"2023-11-28T20:51:53.525913+0000","last_fullsized":"2023-11-28T20:51:53.525913+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:50.852486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.535284+0000","last_change":"2023-11-28T20:51:53.539474+0000","last_active":"2023-11-28T20:51:54.535284+0000","last_peered":"2023-11-28T20:51:54.535284+0000","last_clean":"2023-11-28T20:51:54.535284+0000","last_became_active":"2023-11-28T20:51:53.539143+0000","last_became_peered":"2023-11-28T20:51:53.539143+0000","last_unstale":"2023-11-28T20:51:54.535284+0000","last_undegraded":"2023-11-28T20:51:54.535284+0000","last_fullsized":"2023-11-28T20:51:54.535284+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:45.693788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.528801+0000","last_change":"2023-11-28T20:51:52.505376+0000","last_active":"2023-11-28T20:51:53.528801+0000","last_peered":"2023-11-28T20:51:53.528801+0000","last_clean":"2023-11-28T20:51:53.528801+0000","last_became_active":"2023-11-28T20:51:52.504325+0000","last_became_peered":"2023-11-28T20:51:52.504325+0000","last_unstale":"2023-11-28T20:51:53.528801+0000","last_undegraded":"2023-11-28T20:51:53.528801+0000","last_fullsized":"2023-11-28T20:51:53.528801+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:59:26.208118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.557785+0000","last_change":"2023-11-28T20:51:53.554859+0000","last_active":"2023-11-28T20:51:53.557785+0000","last_peered":"2023-11-28T20:51:53.557785+0000","last_clean":"2023-11-28T20:51:53.557785+0000","last_became_active":"2023-11-28T20:51:53.554439+0000","last_became_peered":"2023-11-28T20:51:53.554439+0000","last_unstale":"2023-11-28T20:51:53.557785+0000","last_undegraded":"2023-11-28T20:51:53.557785+0000","last_fullsized":"2023-11-28T20:51:53.557785+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:59.606587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.544555+0000","last_change":"2023-11-28T20:51:52.503843+0000","last_active":"2023-11-28T20:51:54.544555+0000","last_peered":"2023-11-28T20:51:54.544555+0000","last_clean":"2023-11-28T20:51:54.544555+0000","last_became_active":"2023-11-28T20:51:52.499139+0000","last_became_peered":"2023-11-28T20:51:52.499139+0000","last_unstale":"2023-11-28T20:51:54.544555+0000","last_undegraded":"2023-11-28T20:51:54.544555+0000","last_fullsized":"2023-11-28T20:51:54.544555+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:15:53.716050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.548425+0000","last_change":"2023-11-28T20:51:53.547064+0000","last_active":"2023-11-28T20:51:53.548425+0000","last_peered":"2023-11-28T20:51:53.548425+0000","last_clean":"2023-11-28T20:51:53.548425+0000","last_became_active":"2023-11-28T20:51:53.546800+0000","last_became_peered":"2023-11-28T20:51:53.546800+0000","last_unstale":"2023-11-28T20:51:53.548425+0000","last_undegraded":"2023-11-28T20:51:53.548425+0000","last_fullsized":"2023-11-28T20:51:53.548425+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:47:47.219724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.526120+0000","last_change":"2023-11-28T20:51:52.498124+0000","last_active":"2023-11-28T20:51:53.526120+0000","last_peered":"2023-11-28T20:51:53.526120+0000","last_clean":"2023-11-28T20:51:53.526120+0000","last_became_active":"2023-11-28T20:51:52.497564+0000","last_became_peered":"2023-11-28T20:51:52.497564+0000","last_unstale":"2023-11-28T20:51:53.526120+0000","last_undegraded":"2023-11-28T20:51:53.526120+0000","last_fullsized":"2023-11-28T20:51:53.526120+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:43.383167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.547470+0000","last_change":"2023-11-28T20:51:53.543934+0000","last_active":"2023-11-28T20:51:53.547470+0000","last_peered":"2023-11-28T20:51:53.547470+0000","last_clean":"2023-11-28T20:51:53.547470+0000","last_became_active":"2023-11-28T20:51:53.543663+0000","last_became_peered":"2023-11-28T20:51:53.543663+0000","last_unstale":"2023-11-28T20:51:53.547470+0000","last_undegraded":"2023-11-28T20:51:53.547470+0000","last_fullsized":"2023-11-28T20:51:53.547470+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:23:14.887409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.526854+0000","last_change":"2023-11-28T20:51:52.511212+0000","last_active":"2023-11-28T20:51:53.526854+0000","last_peered":"2023-11-28T20:51:53.526854+0000","last_clean":"2023-11-28T20:51:53.526854+0000","last_became_active":"2023-11-28T20:51:52.510917+0000","last_became_peered":"2023-11-28T20:51:52.510917+0000","last_unstale":"2023-11-28T20:51:53.526854+0000","last_undegraded":"2023-11-28T20:51:53.526854+0000","last_fullsized":"2023-11-28T20:51:53.526854+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:40:24.247674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.546174+0000","last_change":"2023-11-28T20:51:53.545387+0000","last_active":"2023-11-28T20:51:53.546174+0000","last_peered":"2023-11-28T20:51:53.546174+0000","last_clean":"2023-11-28T20:51:53.546174+0000","last_became_active":"2023-11-28T20:51:53.545100+0000","last_became_peered":"2023-11-28T20:51:53.545100+0000","last_unstale":"2023-11-28T20:51:53.546174+0000","last_undegraded":"2023-11-28T20:51:53.546174+0000","last_fullsized":"2023-11-28T20:51:53.546174+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:52.461786+0000","last_clean_scrub_stamp":"2023-11-28T20:51:52.461786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:26:07.345244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:53.510743+0000","last_change":"2023-11-28T20:51:52.502373+0000","last_active":"2023-11-28T20:51:53.510743+0000","last_peered":"2023-11-28T20:51:53.510743+0000","last_clean":"2023-11-28T20:51:53.510743+0000","last_became_active":"2023-11-28T20:51:52.501029+0000","last_became_peered":"2023-11-28T20:51:52.501029+0000","last_unstale":"2023-11-28T20:51:53.510743+0000","last_undegraded":"2023-11-28T20:51:53.510743+0000","last_fullsized":"2023-11-28T20:51:53.510743+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:51:51.445598+0000","last_clean_scrub_stamp":"2023-11-28T20:51:51.445598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:44:13.921193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"115'4286","reported_seq":4629,"reported_epoch":115,"state":"active+clean","last_fresh":"2023-11-28T20:51:54.894325+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:51:54.894325+0000","last_peered":"2023-11-28T20:51:54.894325+0000","last_clean":"2023-11-28T20:51:54.894325+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:51:54.894325+0000","last_undegraded":"2023-11-28T20:51:54.894325+0000","last_fullsized":"2023-11-28T20:51:54.894325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":710,"num_read_kb":1360,"num_write":271,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":710,"num_read_kb":1360,"num_write":271,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4286,"ondisk_log_size":4286,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575063,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63680,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308160,"statfs":{"total":96636764160,"available":96571555840,"internally_reserved":0,"allocated":2019328,"data_stored":1303675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.1519999999999999,"15min":1.1879999999999999},"min":{"1min":0.66400000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.6679999999999999,"5min":1.722,"15min":1.786},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.1379999999999999,"15min":1.1779999999999999},"min":{"1min":0.72199999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.64600000000000002}]},{"osd":1,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.163,"15min":1.1899999999999999},"min":{"1min":0.72599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.744},"last":1.504},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.226,"15min":1.248},"min":{"1min":0.747,"5min":0.624,"15min":0.624},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.889},"last":1.4550000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2729999999999999,"15min":1.302},"min":{"1min":0.873,"5min":0.80300000000000005,"15min":0.58799999999999997},"max":{"1min":1.6919999999999999,"5min":1.8300000000000001,"15min":1.9359999999999999},"last":0.87},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2929999999999999,"15min":1.294},"min":{"1min":0.92500000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.921,"5min":1.921,"15min":1.921},"last":1.129}]},{"osd":3,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.2809999999999999},"min":{"1min":0.755,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.234},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.385,"15min":1.4039999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.68300000000000005},"max":{"1min":1.6579999999999999,"5min":1.956,"15min":1.956},"last":1.1659999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3,"15min":1.27},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.212},{"interface":"front","average":{"1min":1.343,"5min":1.3320000000000001,"15min":1.327},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.8560000000000001,"5min":1.863,"15min":1.9970000000000001},"last":1.7589999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4019999999999999,"15min":1.3979999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.7909999999999999,"5min":1.9219999999999999,"15min":2.048},"last":1.335},{"interface":"front","average":{"1min":1.47,"5min":1.4790000000000001,"15min":1.4610000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.3089999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.569,"15min":1.5509999999999999},"min":{"1min":0.85799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.431},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5009999999999999,"15min":1.4890000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.1699999999999999},"last":1.395}]}]},{"osd":6,"up_from":13,"seq":55834575063,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1437696,"data_stored":715467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.075,"15min":1.115},"min":{"1min":0.65900000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.6319999999999999,"5min":1.665,"15min":1.7350000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.095,"15min":1.129},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8260000000000001},"last":1.2989999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1910000000000001,"15min":1.2609999999999999},"min":{"1min":0.76100000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.706,"5min":1.728,"15min":1.8069999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.325,"15min":1.375},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.7889999999999999,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.94099999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.2450000000000001,"15min":1.2749999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5620000000000001,"5min":1.6879999999999999,"15min":1.71},"last":1.135},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3160000000000001,"15min":1.3620000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.784,"5min":1.851,"15min":1.851},"last":0.995}]},{"osd":3,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.2709999999999999,"15min":1.2949999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6699999999999999,"5min":1.821,"15min":1.9299999999999999},"last":1.071},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.256,"15min":1.284},"min":{"1min":0.879,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7949999999999999},"last":0.78900000000000003}]},{"osd":4,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.2849999999999999,"15min":1.272},"min":{"1min":0.81499999999999995,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.7529999999999999,"5min":1.849,"15min":2.7290000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3009999999999999,"15min":1.3280000000000001},"min":{"1min":0.89200000000000002,"5min":0.80800000000000005,"15min":0.80100000000000005},"max":{"1min":1.9179999999999999,"5min":2.2010000000000001,"15min":3.2519999999999998},"last":1.4570000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.3149999999999999,"15min":1.333},"min":{"1min":0.91000000000000003,"5min":0.754,"15min":0.62},"max":{"1min":1.857,"5min":1.857,"15min":2.0539999999999998},"last":1.353},{"interface":"front","average":{"1min":1.45,"5min":1.4990000000000001,"15min":1.5429999999999999},"min":{"1min":0.93400000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.8340000000000001,"5min":1.9179999999999999,"15min":3.0310000000000001},"last":1.4039999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4379999999999999,"15min":1.482},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.86499999999999999},"max":{"1min":1.9790000000000001,"5min":2.1120000000000001,"15min":3.379},"last":1.544},{"interface":"front","average":{"1min":1.423,"5min":1.456,"15min":1.4630000000000001},"min":{"1min":0.96099999999999997,"5min":0.93300000000000005,"15min":0.84599999999999997},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":2.0739999999999998},"last":1.498}]}]},{"osd":5,"up_from":13,"seq":55834575063,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50612,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321228,"statfs":{"total":96636764160,"available":96584937472,"internally_reserved":0,"allocated":1417216,"data_stored":713251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.159,"15min":1.175},"min":{"1min":0.72199999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.702,"5min":1.702,"15min":1.768},"last":0.75},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1160000000000001,"15min":1.1879999999999999},"min":{"1min":0.56200000000000006,"5min":0.441,"15min":0.441},"max":{"1min":1.6060000000000001,"5min":1.631,"15min":1.829},"last":1.024}]},{"osd":1,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.115,"15min":1.1499999999999999},"min":{"1min":0.67200000000000004,"5min":0.57399999999999995,"15min":0.48899999999999999},"max":{"1min":1.587,"5min":1.768,"15min":1.768},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.204,"15min":1.218},"min":{"1min":0.72399999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5860000000000001,"5min":1.8879999999999999,"15min":1.956},"last":1.206}]},{"osd":2,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2529999999999999,"15min":1.268},"min":{"1min":0.68600000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.387},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.226,"15min":1.292},"min":{"1min":0.76900000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.958},"last":1.149}]},{"osd":3,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.276,"15min":1.3169999999999999},"min":{"1min":0.77700000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.379,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.2490000000000001,"15min":1.2689999999999999},"min":{"1min":0.83299999999999996,"5min":0.64000000000000001,"15min":0.54600000000000004},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.575}]},{"osd":4,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.284,"15min":1.2949999999999999},"min":{"1min":0.875,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.772,"5min":1.772,"15min":1.8380000000000001},"last":1.29},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.45,"15min":1.4370000000000001},"min":{"1min":1.0760000000000001,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.6259999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3029999999999999,"15min":1.325},"min":{"1min":0.82199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":2.093},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.464,"15min":1.464},"min":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.752}]},{"osd":7,"last update":"Tue Nov 28 20:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.421,"15min":1.4179999999999999},"min":{"1min":1.0349999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.466,"15min":1.4830000000000001},"min":{"1min":1.0620000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.958,"5min":1.958,"15min":2.1829999999999998},"last":1.254}]}]},{"osd":4,"up_from":13,"seq":55834575064,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36740,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335100,"statfs":{"total":96636764160,"available":96599142400,"internally_reserved":0,"allocated":1425408,"data_stored":713411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0840000000000001,"15min":1.137},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3919999999999999,"5min":1.605,"15min":1.879},"last":1.387},{"interface":"front","average":{"1min":1.026,"5min":1.1020000000000001,"15min":1.121},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.554,"5min":1.891,"15min":1.9750000000000001},"last":0.83299999999999996}]},{"osd":1,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.264,"15min":1.3169999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.542,"5min":1.734,"15min":1.923},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.331,"15min":1.355},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.6779999999999999,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.70499999999999996}]},{"osd":2,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.224,"15min":1.212},"min":{"1min":0.76800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":1.117},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.258,"15min":1.2989999999999999},"min":{"1min":0.84399999999999997,"5min":0.77400000000000002,"15min":0.77200000000000002},"max":{"1min":1.6299999999999999,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.581}]},{"osd":3,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.292,"15min":1.29},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.9630000000000001,"5min":2.157,"15min":2.157},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.3540000000000001,"15min":1.375},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.67900000000000005},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.206}]},{"osd":5,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3089999999999999,"15min":1.2789999999999999},"min":{"1min":0.86699999999999999,"5min":0.86099999999999999,"15min":0.65800000000000003},"max":{"1min":1.9750000000000001,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.706},{"interface":"front","average":{"1min":1.347,"5min":1.339,"15min":1.296},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.79500000000000004},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.6779999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.403,"15min":1.393},"min":{"1min":0.872,"5min":0.872,"15min":0.72899999999999998},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.5569999999999999,"15min":1.552},"min":{"1min":1.147,"5min":1.0369999999999999,"15min":0.91500000000000004},"max":{"1min":2.1349999999999998,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":1.7949999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.4950000000000001,"15min":1.456},"min":{"1min":0.94899999999999995,"5min":0.91300000000000003,"15min":0.85299999999999998},"max":{"1min":2.2490000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.573,"5min":1.5409999999999999,"15min":1.5269999999999999},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.95399999999999996},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.7609999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575064,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40884,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330956,"statfs":{"total":96636764160,"available":96594898944,"internally_reserved":0,"allocated":1417216,"data_stored":713251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.873,"15min":0.90600000000000003},"min":{"1min":0.45500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.6000000000000001,"5min":4.7210000000000001,"15min":4.7210000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.96499999999999997,"15min":1.071},"min":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.4870000000000001,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":0.76400000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.95199999999999996,"15min":0.97499999999999998},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4730000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.1499999999999999,"15min":1.1990000000000001},"min":{"1min":0.58499999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5289999999999999,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.69399999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.1839999999999999,"15min":1.2470000000000001},"min":{"1min":0.55800000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4490000000000001,"5min":3.444,"15min":3.444},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.196,"5min":1.2769999999999999,"15min":1.335},"min":{"1min":0.66800000000000004,"5min":0.58099999999999996,"15min":0.54400000000000004},"max":{"1min":1.7230000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":1.085}]},{"osd":4,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.298,"15min":1.3420000000000001},"min":{"1min":0.84499999999999997,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.6890000000000001,"5min":1.8779999999999999,"15min":2.9420000000000002},"last":1.145},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.327,"15min":1.3839999999999999},"min":{"1min":0.86399999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.1389999999999998},"last":1.208}]},{"osd":5,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4259999999999999,"15min":1.452},"min":{"1min":0.88,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.8959999999999999,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.375,"15min":1.4330000000000001},"min":{"1min":0.79800000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9670000000000001,"5min":2.2429999999999999,"15min":3.1520000000000001},"last":1.3640000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3979999999999999,"15min":1.4790000000000001},"min":{"1min":0.86099999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.3100000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.5309999999999999,"15min":1.577},"min":{"1min":0.91100000000000003,"5min":0.749,"15min":0.749},"max":{"1min":1.7729999999999999,"5min":3.661,"15min":3.661},"last":1.6259999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.45,"15min":1.5369999999999999},"min":{"1min":0.73399999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.863,"5min":2.1469999999999998,"15min":3.266},"last":1.514},{"interface":"front","average":{"1min":1.462,"5min":1.514,"15min":1.601},"min":{"1min":0.90200000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.1269999999999998,"5min":2.3439999999999999,"15min":2.3919999999999999},"last":1.4099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575064,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58932,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312908,"statfs":{"total":96636764160,"available":96576417792,"internally_reserved":0,"allocated":1417216,"data_stored":713251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.85199999999999998,"15min":0.90900000000000003},"min":{"1min":0.373,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.0589999999999999,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.96299999999999997,"15min":1.0620000000000001},"min":{"1min":0.51200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.569,"5min":1.6419999999999999,"15min":1.9390000000000001},"last":0.92400000000000004}]},{"osd":1,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.93200000000000005,"15min":0.98499999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.611,"5min":1.7669999999999999,"15min":3.2599999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.054,"15min":1.1479999999999999},"min":{"1min":0.53700000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.675,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.68899999999999995}]},{"osd":3,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.0900000000000001,"15min":1.157},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.5549999999999999,"5min":1.774,"15min":2.194},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1719999999999999,"15min":1.292},"min":{"1min":0.55600000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":3.1440000000000001},"last":0.48499999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.345,"15min":1.411},"min":{"1min":0.874,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0920000000000001,"5min":2.23,"15min":2.8919999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.335,"5min":1.375,"15min":1.4159999999999999},"min":{"1min":0.90900000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.901,"5min":2.3740000000000001,"15min":2.3809999999999998},"last":1.409}]},{"osd":5,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3520000000000001,"15min":1.4179999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.681,"5min":2.327,"15min":2.327},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.4419999999999999,"15min":1.4970000000000001},"min":{"1min":1.0289999999999999,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.153,"5min":2.1680000000000001,"15min":3.4279999999999999},"last":1.077}]},{"osd":6,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.4279999999999999,"15min":1.502},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.8520000000000001,"5min":2.1709999999999998,"15min":2.621},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.4950000000000001,"15min":1.575},"min":{"1min":0.95599999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.2000000000000002,"5min":2.6749999999999998,"15min":3.6269999999999998},"last":1.036}]},{"osd":7,"last update":"Tue Nov 28 20:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4850000000000001,"15min":1.5389999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.7210000000000001,"5min":2.367,"15min":2.367},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.508,"15min":1.587},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.835,"5min":2.4849999999999999,"15min":2.7109999999999999},"last":1.444}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1421312,"data_stored":713321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.875,"15min":0.90100000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.9259999999999999},"last":0.873},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.97699999999999998,"15min":1.071},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.375,"5min":2.028,"15min":2.028},"last":1.3360000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0309999999999999,"15min":1.0880000000000001},"min":{"1min":0.53200000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.48,"5min":1.534,"15min":2.234},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.1539999999999999,"15min":1.206},"min":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.8360000000000001,"5min":1.9199999999999999,"15min":2.895},"last":1.429}]},{"osd":3,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.212,"15min":1.3129999999999999},"min":{"1min":0.58199999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.7849999999999999,"5min":1.893,"15min":2.0459999999999998},"last":1.389},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.044,"15min":1.113},"min":{"1min":0.45100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.7490000000000001,"5min":1.887,"15min":2.3679999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.266,"15min":1.3380000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.657,"5min":1.742,"15min":2.1150000000000002},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3480000000000001,"15min":1.389},"min":{"1min":0.72399999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":2.6819999999999999},"last":1.4970000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3879999999999999,"15min":1.425},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.8109999999999999,"5min":1.903,"15min":2.1840000000000002},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.4350000000000001,"15min":1.496},"min":{"1min":0.89700000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.0219999999999998,"5min":2.1000000000000001,"15min":2.294},"last":1.1719999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.355,"15min":1.4199999999999999},"min":{"1min":0.92300000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7470000000000001,"5min":1.8380000000000001,"15min":1.9379999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.4690000000000001,"15min":1.518},"min":{"1min":0.86199999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.873,"5min":2.1520000000000001,"15min":2.3300000000000001},"last":1.6000000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.49,"15min":1.5580000000000001},"min":{"1min":0.80500000000000005,"5min":0.753,"15min":0.753},"max":{"1min":1.9430000000000001,"5min":2.177,"15min":2.3359999999999999},"last":1.472},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.446,"15min":1.512},"min":{"1min":0.80500000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":2.294},"last":1.2809999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575064,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37332,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334508,"statfs":{"total":96636764160,"available":96598536192,"internally_reserved":0,"allocated":2035712,"data_stored":1305937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.89900000000000002,"15min":0.90600000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.2729999999999999,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.016,"15min":1.085},"min":{"1min":0.60299999999999998,"5min":0.501,"15min":0.501},"max":{"1min":1.6619999999999999,"5min":1.6879999999999999,"15min":1.764},"last":1.171}]},{"osd":2,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.02,"15min":1.109},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.6000000000000001,"5min":1.6220000000000001,"15min":1.7709999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.127,"15min":1.1910000000000001},"min":{"1min":0.67800000000000005,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4510000000000001,"5min":1.893,"15min":1.9550000000000001},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.014,"15min":1.1080000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.325,"5min":2.081,"15min":2.081},"last":1.099},{"interface":"front","average":{"1min":1.115,"5min":1.1990000000000001,"15min":1.3080000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.7030000000000001,"5min":1.8180000000000001,"15min":2.1080000000000001},"last":1.5209999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.363,"15min":1.421},"min":{"1min":0.879,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.802,"5min":2.012,"15min":2.012},"last":1.712},{"interface":"front","average":{"1min":1.286,"5min":1.3400000000000001,"15min":1.377},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.7509999999999999,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.48}]},{"osd":5,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3720000000000001,"15min":1.4099999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4330000000000001,"15min":1.486},"min":{"1min":0.90500000000000003,"5min":0.621,"15min":0.621},"max":{"1min":2.0129999999999999,"5min":2.0750000000000002,"15min":2.2490000000000001},"last":1.3600000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.472,"15min":1.5469999999999999},"min":{"1min":0.73799999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.8919999999999999,"5min":2.1160000000000001,"15min":2.2959999999999998},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.476,"15min":1.5229999999999999},"min":{"1min":0.93899999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.956,"5min":2.274,"15min":2.4209999999999998},"last":1.9079999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:51:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.532,"15min":1.589},"min":{"1min":0.93500000000000005,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.8049999999999999,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.9830000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.4650000000000001,"15min":1.5349999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.73799999999999999},"max":{"1min":1.996,"5min":2.3460000000000001,"15min":2.524},"last":1.6120000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:51:58.597 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T20:51:58.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T20:51:58.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.880+0000 7f009b26b640 1 Processor -- start 2023-11-28T20:51:58.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.880+0000 7f009b26b640 1 -- start start 2023-11-28T20:51:58.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.882+0000 7f009b26b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f009406dc50 0x7f00940cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.882+0000 7f009b26b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00940ce600 0x7f009406a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.882+0000 7f009b26b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009406e030 con 0x7f009406aa40 2023-11-28T20:51:58.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.882+0000 7f009b26b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009406aed0 con 0x7f009406dc50 2023-11-28T20:51:58.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.882+0000 7f009b26b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00940ccf60 con 0x7f00940ce600 2023-11-28T20:51:58.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.882+0000 7f0099808640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f009406aa40 0x7f00940c93b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44170/0 (socket says 172.21.15.45:44170) 2023-11-28T20:51:58.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.882+0000 7f0099808640 1 -- 172.21.15.45:0/1632598387 learned_addr learned my addr 172.21.15.45:0/1632598387 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:51:58.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.882+0000 7f0099007640 1 --2- 172.21.15.45:0/1632598387 >> v2:172.21.15.190:3300/0 conn(0x7f009406dc50 0x7f00940cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.882+0000 7f0098806640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00940ce600 0x7f009406a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427461510 0 0) 0x7f008c001490 con 0x7f009406aa40 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0080003520 con 0x7f009406aa40 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804894136 0 0) 0x7f0080003520 con 0x7f009406aa40 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00940ce600 msgr2=0x7f009406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00940ce600 0x7f009406a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 >> v2:172.21.15.190:3300/0 conn(0x7f009406dc50 msgr2=0x7f00940cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 --2- 172.21.15.45:0/1632598387 >> v2:172.21.15.190:3300/0 conn(0x7f009406dc50 0x7f00940cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00940cd930 con 0x7f009406aa40 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f008c001b80 con 0x7f009406aa40 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 >> v1:172.21.15.45:6789/0 conn(0x7f009406aa40 legacy=0x7f00940c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0080003970 0x7f0080003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00940ce600 0x7f00800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0080004d00 0x7f0080005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009406aed0 con 0x7f0080004d00 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.883+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00940ccf60 con 0x7f0080003970 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008000b980 con 0x7f00940ce600 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f0099808640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0080004d00 0x7f0080005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f0099007640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0080003970 0x7f0080003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f0098806640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00940ce600 0x7f00800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f0099808640 1 -- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00940ce600 msgr2=0x7f00800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f0099808640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00940ce600 0x7f00800047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f0099808640 1 -- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0080003970 msgr2=0x7f0080003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f0099808640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0080003970 0x7f0080003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f0099808640 1 -- 172.21.15.45:0/1632598387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0080003520 con 0x7f0080004d00 2023-11-28T20:51:58.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.884+0000 7f0099007640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0080003970 0x7f0080003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:51:58.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.885+0000 7f0099808640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0080004d00 0x7f0080005180 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f008c00a840 tx=0x7f00800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958727a4625d07f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c034630 con 0x7f0080004d00 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c02f070 con 0x7f0080004d00 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f008bfff640 1 -- 172.21.15.45:0/1632598387 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c0349f0 con 0x7f0080004d00 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f009b26b640 1 -- 172.21.15.45:0/1632598387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0080004d00 msgr2=0x7f0080005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f009b26b640 1 --2- 172.21.15.45:0/1632598387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0080004d00 0x7f0080005180 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f008c00a840 tx=0x7f00800057f0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f0099808640 1 -- 172.21.15.45:0/1632598387 reap_dead start 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f009b26b640 1 -- 172.21.15.45:0/1632598387 shutdown_connections 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f009b26b640 1 -- 172.21.15.45:0/1632598387 >> 172.21.15.45:0/1632598387 conn(0x7f00940ced60 msgr2=0x7f00940cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f009b26b640 1 -- 172.21.15.45:0/1632598387 shutdown_connections 2023-11-28T20:51:58.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.886+0000 7f009b26b640 1 -- 172.21.15.45:0/1632598387 wait complete. 2023-11-28T20:51:58.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.888+0000 7f009b26b640 1 Processor -- start 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.888+0000 7f009b26b640 1 -- start start 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.889+0000 7f009b26b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0080004d00 0x7f0094154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.889+0000 7f009b26b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00940b1940 0x7f009409c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.889+0000 7f009b26b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00941e7210 0x7f00941e7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.889+0000 7f009b26b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0094155230 con 0x7f00940b1940 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.889+0000 7f009b26b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009409c880 con 0x7f00941e7210 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.889+0000 7f009b26b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00940b1f00 con 0x7f0080004d00 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0098806640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00940b1940 0x7f009409c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0098806640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00940b1940 0x7f009409c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52572/0 (socket says 172.21.15.45:52572) 2023-11-28T20:51:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0098806640 1 -- 172.21.15.45:0/4155357874 learned_addr learned my addr 172.21.15.45:0/4155357874 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:58.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0099007640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0080004d00 0x7f0094154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0099808640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00941e7210 0x7f00941e7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0098806640 1 -- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0080004d00 msgr2=0x7f0094154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0098806640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0080004d00 0x7f0094154cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0098806640 1 -- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00941e7210 msgr2=0x7f00941e7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:58.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0098806640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00941e7210 0x7f00941e7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:58.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0098806640 1 -- 172.21.15.45:0/4155357874 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00940b21e0 con 0x7f00940b1940 2023-11-28T20:51:58.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.890+0000 7f0098806640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00940b1940 0x7f009409c340 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f0084010860 tx=0x7f0084010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:58.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.891+0000 7f0089ffb640 1 -- 172.21.15.45:0/4155357874 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084009610 con 0x7f00940b1940 2023-11-28T20:51:58.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.891+0000 7f0089ffb640 1 -- 172.21.15.45:0/4155357874 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008400eb60 con 0x7f00940b1940 2023-11-28T20:51:58.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.891+0000 7f0089ffb640 1 -- 172.21.15.45:0/4155357874 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008401c400 con 0x7f00940b1940 2023-11-28T20:51:58.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.891+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00940b24c0 con 0x7f00940b1940 2023-11-28T20:51:58.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.891+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00940b2a80 con 0x7f00940b1940 2023-11-28T20:51:58.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.892+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00940733d0 con 0x7f00940b1940 2023-11-28T20:51:58.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.893+0000 7f0089ffb640 1 -- 172.21.15.45:0/4155357874 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f008401c560 con 0x7f00940b1940 2023-11-28T20:51:58.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.894+0000 7f0089ffb640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f007806aed0 0x7f007806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:58.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.894+0000 7f0089ffb640 1 -- 172.21.15.45:0/4155357874 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0084089d60 con 0x7f00940b1940 2023-11-28T20:51:58.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.895+0000 7f0099007640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f007806aed0 0x7f007806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:58.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.896+0000 7f0099007640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f007806aed0 0x7f007806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0090009a40 tx=0x7f0090008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:58.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:58.899+0000 7f0089ffb640 1 -- 172.21.15.45:0/4155357874 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0084054d20 con 0x7f00940b1940 2023-11-28T20:51:58.937 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:51:58.933+0000 7f55402a0640 -1 client.6331: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5789:5e03f14b-9fe7-48be-b73f-94f0877537c7:v1:172.21.15.190:0/1724154865] 2023-11-28T20:51:59.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.171+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f009406a820 con 0x7f00940b1940 2023-11-28T20:51:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.176+0000 7f0089ffb640 1 -- 172.21.15.45:0/4155357874 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f0084058340 con 0x7f00940b1940 2023-11-28T20:51:59.177 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T20:51:59.178 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:51:59.178 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T20:51:59.178 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T20:51:59.178 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T20:51:59.178 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.179+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f007806aed0 msgr2=0x7f007806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.179+0000 7f009b26b640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f007806aed0 0x7f007806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0090009a40 tx=0x7f0090008040 comp rx=0 tx=0).stop 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.179+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00940b1940 msgr2=0x7f009409c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.179+0000 7f009b26b640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00940b1940 0x7f009409c340 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f0084010860 tx=0x7f0084010d20 comp rx=0 tx=0).stop 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.180+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 shutdown_connections 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.180+0000 7f009b26b640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f007806aed0 0x7f007806d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.180+0000 7f009b26b640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00941e7210 0x7f00941e7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.180+0000 7f009b26b640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00940b1940 0x7f009409c340 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.180+0000 7f009b26b640 1 --2- 172.21.15.45:0/4155357874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0080004d00 0x7f0094154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.180+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 >> 172.21.15.45:0/4155357874 conn(0x7f00940ced60 msgr2=0x7f00940af830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:59.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.180+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 shutdown_connections 2023-11-28T20:51:59.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.180+0000 7f009b26b640 1 -- 172.21.15.45:0/4155357874 wait complete. 2023-11-28T20:51:59.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:51:59.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.427+0000 7fd976c89640 1 Processor -- start 2023-11-28T20:51:59.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.427+0000 7fd976c89640 1 -- start start 2023-11-28T20:51:59.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.429+0000 7fd976c89640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd97015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:59.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.429+0000 7fd976c89640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd9700b1340 0x7fd97015f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:59.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.429+0000 7fd976c89640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97015bda0 con 0x7fd97015b850 2023-11-28T20:51:59.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.429+0000 7fd976c89640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97015ee40 con 0x7fd9700b1340 2023-11-28T20:51:59.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.429+0000 7fd976c89640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97015c080 con 0x7fd9700b35b0 2023-11-28T20:51:59.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.430+0000 7fd974a25640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd97015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:59.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.430+0000 7fd975226640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd9700b1340 0x7fd97015f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:59.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.430+0000 7fd975226640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd9700b1340 0x7fd97015f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:37706/0 (socket says 172.21.15.45:37706) 2023-11-28T20:51:59.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.430+0000 7fd975226640 1 -- 172.21.15.45:0/2418519699 learned_addr learned my addr 172.21.15.45:0/2418519699 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:59.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.430+0000 7fd974a25640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd97015ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:50022/0 (socket says 172.21.15.45:50022) 2023-11-28T20:51:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.431+0000 7fd975226640 1 -- 172.21.15.45:0/2418519699 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 msgr2=0x7fd97015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:51:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.431+0000 7fd975226640 1 --2- 172.21.15.45:0/2418519699 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd97015ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.431+0000 7fd975226640 1 -- 172.21.15.45:0/2418519699 >> v1:172.21.15.190:6789/0 conn(0x7fd97015b850 legacy=0x7fd9700b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.431+0000 7fd975226640 1 -- 172.21.15.45:0/2418519699 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97015c360 con 0x7fd9700b1340 2023-11-28T20:51:59.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.431+0000 7fd974a25640 1 --2- 172.21.15.45:0/2418519699 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd97015ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:51:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.432+0000 7fd975226640 1 --2- 172.21.15.45:0/2418519699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 0x7fd97015f290 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd968009870 tx=0x7fd96802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7b49bfd660b8f96 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd9677fe640 1 -- 172.21.15.45:0/2418519699 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968030840 con 0x7fd9700b1340 2023-11-28T20:51:59.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd9677fe640 1 -- 172.21.15.45:0/2418519699 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9680309a0 con 0x7fd9700b1340 2023-11-28T20:51:59.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd9677fe640 1 -- 172.21.15.45:0/2418519699 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968030d30 con 0x7fd9700b1340 2023-11-28T20:51:59.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd976c89640 1 -- 172.21.15.45:0/2418519699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 msgr2=0x7fd97015f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:59.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd976c89640 1 --2- 172.21.15.45:0/2418519699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 0x7fd97015f290 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd968009870 tx=0x7fd96802b980 comp rx=0 tx=0).stop 2023-11-28T20:51:59.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd976c89640 1 -- 172.21.15.45:0/2418519699 shutdown_connections 2023-11-28T20:51:59.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd976c89640 1 --2- 172.21.15.45:0/2418519699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 0x7fd97015f290 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd976c89640 1 --2- 172.21.15.45:0/2418519699 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd97015ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd976c89640 1 -- 172.21.15.45:0/2418519699 >> 172.21.15.45:0/2418519699 conn(0x7fd97006a500 msgr2=0x7fd97006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:59.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd976c89640 1 -- 172.21.15.45:0/2418519699 shutdown_connections 2023-11-28T20:51:59.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.433+0000 7fd976c89640 1 -- 172.21.15.45:0/2418519699 wait complete. 2023-11-28T20:51:59.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.435+0000 7fd976c89640 1 Processor -- start 2023-11-28T20:51:59.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.435+0000 7fd976c89640 1 -- start start 2023-11-28T20:51:59.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.436+0000 7fd976c89640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 0x7fd97015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:59.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.436+0000 7fd976c89640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd970156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:59.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.436+0000 7fd976c89640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd97015b850 0x7fd9701573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:59.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.436+0000 7fd976c89640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd970159da0 con 0x7fd97015b850 2023-11-28T20:51:59.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.436+0000 7fd976c89640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97015a080 con 0x7fd9700b1340 2023-11-28T20:51:59.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.436+0000 7fd976c89640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97015a360 con 0x7fd9700b35b0 2023-11-28T20:51:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd975226640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd97015b850 0x7fd9701573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd975226640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd97015b850 0x7fd9701573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52584/0 (socket says 172.21.15.45:52584) 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd975226640 1 -- 172.21.15.45:0/139241414 learned_addr learned my addr 172.21.15.45:0/139241414 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd974a25640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 0x7fd97015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd967fff640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd970156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd975226640 1 -- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 msgr2=0x7fd970156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd975226640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd970156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd975226640 1 -- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 msgr2=0x7fd97015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd975226640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 0x7fd97015ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd975226640 1 -- 172.21.15.45:0/139241414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c009a70 con 0x7fd97015b850 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.437+0000 7fd974a25640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 0x7fd97015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.438+0000 7fd975226640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd97015b850 0x7fd9701573b0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fd968009870 tx=0x7fd968004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.438+0000 7fd9657fa640 1 -- 172.21.15.45:0/139241414 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968039c90 con 0x7fd97015b850 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.438+0000 7fd9657fa640 1 -- 172.21.15.45:0/139241414 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd968039e10 con 0x7fd97015b850 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.438+0000 7fd9657fa640 1 -- 172.21.15.45:0/139241414 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968043640 con 0x7fd97015b850 2023-11-28T20:51:59.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.438+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9680094d0 con 0x7fd97015b850 2023-11-28T20:51:59.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.438+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97015a920 con 0x7fd97015b850 2023-11-28T20:51:59.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.440+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9700733d0 con 0x7fd97015b850 2023-11-28T20:51:59.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.440+0000 7fd9657fa640 1 -- 172.21.15.45:0/139241414 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fd9680437a0 con 0x7fd97015b850 2023-11-28T20:51:59.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.441+0000 7fd9657fa640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd95c06aed0 0x7fd95c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:51:59.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.441+0000 7fd9657fa640 1 -- 172.21.15.45:0/139241414 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd96803e070 con 0x7fd97015b850 2023-11-28T20:51:59.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.442+0000 7fd974a25640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd95c06aed0 0x7fd95c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:51:59.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.444+0000 7fd974a25640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd95c06aed0 0x7fd95c06d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd96c007640 tx=0x7fd96c00b040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:51:59.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.446+0000 7fd9657fa640 1 -- 172.21.15.45:0/139241414 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9680776b0 con 0x7fd97015b850 2023-11-28T20:51:59.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.682+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9700cd3b0 con 0x7fd97015b850 2023-11-28T20:51:59.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.683+0000 7fd9657fa640 1 -- 172.21.15.45:0/139241414 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fd96807acd0 con 0x7fd97015b850 2023-11-28T20:51:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:51:59.684 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2348343744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2348343744},{"type":"v1","addr":"172.21.15.190:6836","nonce":2348343744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6439,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1864791063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":1864791063},{"type":"v1","addr":"172.21.15.190:6837","nonce":1864791063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3003510210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3003510210},{"type":"v1","addr":"172.21.15.45:6835","nonce":3003510210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:52.969776+0000","modified":"2023-11-28T20:51:56.055296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6448":{"gid":6448,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2489162496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2489162496},{"type":"v1","addr":"172.21.15.45:6834","nonce":2489162496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-28T20:51:59.685 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 51 2023-11-28T20:51:59.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd95c06aed0 msgr2=0x7fd95c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:59.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd95c06aed0 0x7fd95c06d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd96c007640 tx=0x7fd96c00b040 comp rx=0 tx=0).stop 2023-11-28T20:51:59.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd97015b850 msgr2=0x7fd9701573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:51:59.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd97015b850 0x7fd9701573b0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fd968009870 tx=0x7fd968004730 comp rx=0 tx=0).stop 2023-11-28T20:51:59.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 shutdown_connections 2023-11-28T20:51:59.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fd95c06aed0 0x7fd95c06d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd97015b850 0x7fd9701573b0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9700b35b0 0x7fd970156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 --2- 172.21.15.45:0/139241414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9700b1340 0x7fd97015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:51:59.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 >> 172.21.15.45:0/139241414 conn(0x7fd97006a500 msgr2=0x7fd9700aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:51:59.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 shutdown_connections 2023-11-28T20:51:59.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:51:59.686+0000 7fd976c89640 1 -- 172.21.15.45:0/139241414 wait complete. 2023-11-28T20:52:00.110 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:51:52.969776+0000', 'modified': '2023-11-28T20:51:56.055296+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6448}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6448': {'gid': 6448, 'name': 'a', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6834/2489162496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 2489162496}, {'type': 'v1', 'addr': '172.21.15.45:6834', 'nonce': 2489162496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T20:52:00.110 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:52:00.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-28T20:52:00.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.349+0000 7fb53f59e640 1 Processor -- start 2023-11-28T20:52:00.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.349+0000 7fb53f59e640 1 -- start start 2023-11-28T20:52:00.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.350+0000 7fb53f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb540160e70 0x7fb5400cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.350+0000 7fb53f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400d14d0 con 0x7fb5400ce3a0 2023-11-28T20:52:00.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.350+0000 7fb53f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5401613c0 con 0x7fb54015d310 2023-11-28T20:52:00.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.350+0000 7fb53f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400ce830 con 0x7fb540160e70 2023-11-28T20:52:00.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.351+0000 7fb53ed9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb5400ce3a0 0x7fb5400d19d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44176/0 (socket says 172.21.15.45:44176) 2023-11-28T20:52:00.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.351+0000 7fb53ed9d640 1 -- 172.21.15.45:0/4083603864 learned_addr learned my addr 172.21.15.45:0/4083603864 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:52:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.351+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb540160e70 0x7fb5400cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.351+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725616348 0 0) 0x7fb5400d14d0 con 0x7fb5400ce3a0 2023-11-28T20:52:00.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.351+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb52c003520 con 0x7fb5400ce3a0 2023-11-28T20:52:00.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2143377257 0 0) 0x7fb52c003520 con 0x7fb5400ce3a0 2023-11-28T20:52:00.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb540160e70 msgr2=0x7fb5400cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb540160e70 0x7fb5400cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 >> v1:172.21.15.190:6789/0 conn(0x7fb54015d310 legacy=0x7fb5400b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5401616a0 con 0x7fb5400ce3a0 2023-11-28T20:52:00.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb538001670 con 0x7fb5400ce3a0 2023-11-28T20:52:00.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 >> v1:172.21.15.45:6789/0 conn(0x7fb5400ce3a0 legacy=0x7fb5400d19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb52c0039a0 0x7fb52c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb52c0047c0 0x7fb52c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb540160e70 0x7fb52c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400ce830 con 0x7fb52c0047c0 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c00bb10 con 0x7fb52c0039a0 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c00bcd0 con 0x7fb540160e70 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.352+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb52c0047c0 0x7fb52c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.353+0000 7fb53e59c640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb52c0039a0 0x7fb52c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.353+0000 7fb53dd9b640 1 -- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb540160e70 msgr2=0x7fb52c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.353+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb540160e70 0x7fb52c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.353+0000 7fb53dd9b640 1 -- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb52c0039a0 msgr2=0x7fb52c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.353+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb52c0039a0 0x7fb52c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.353+0000 7fb53dd9b640 1 -- 172.21.15.45:0/4083603864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c003520 con 0x7fb52c0047c0 2023-11-28T20:52:00.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.353+0000 7fb53e59c640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb52c0039a0 0x7fb52c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:00.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.354+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb52c0047c0 0x7fb52c004c40 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fb528009ac0 tx=0x7fb52c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81a343a8050c8a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.354+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52802c710 con 0x7fb52c0047c0 2023-11-28T20:52:00.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.354+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52802c870 con 0x7fb52c0047c0 2023-11-28T20:52:00.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.354+0000 7fb53d59a640 1 -- 172.21.15.45:0/4083603864 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52802cc00 con 0x7fb52c0047c0 2023-11-28T20:52:00.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.355+0000 7fb53f59e640 1 -- 172.21.15.45:0/4083603864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb52c0047c0 msgr2=0x7fb52c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.355+0000 7fb53f59e640 1 --2- 172.21.15.45:0/4083603864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb52c0047c0 0x7fb52c004c40 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fb528009ac0 tx=0x7fb52c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.355+0000 7fb53ed9d640 1 -- 172.21.15.45:0/4083603864 reap_dead start 2023-11-28T20:52:00.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.355+0000 7fb53f59e640 1 -- 172.21.15.45:0/4083603864 shutdown_connections 2023-11-28T20:52:00.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.355+0000 7fb53f59e640 1 -- 172.21.15.45:0/4083603864 >> 172.21.15.45:0/4083603864 conn(0x7fb54006a500 msgr2=0x7fb54006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:00.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.355+0000 7fb53f59e640 1 -- 172.21.15.45:0/4083603864 shutdown_connections 2023-11-28T20:52:00.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.355+0000 7fb53f59e640 1 -- 172.21.15.45:0/4083603864 wait complete. 2023-11-28T20:52:00.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.357+0000 7fb53f59e640 1 Processor -- start 2023-11-28T20:52:00.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.357+0000 7fb53f59e640 1 -- start start 2023-11-28T20:52:00.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.358+0000 7fb53f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb52c0047c0 0x7fb5401443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.358+0000 7fb53f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb54014ee70 0x7fb5400b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.358+0000 7fb53f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5400b3420 0x7fb5400b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.358+0000 7fb53f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54015dd00 con 0x7fb54014ee70 2023-11-28T20:52:00.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.358+0000 7fb53f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb540151ae0 con 0x7fb5400b3420 2023-11-28T20:52:00.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.358+0000 7fb53f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb540151dc0 con 0x7fb52c0047c0 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.358+0000 7fb53dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb54014ee70 0x7fb5400b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb54014ee70 0x7fb5400b2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52594/0 (socket says 172.21.15.45:52594) 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53dd9b640 1 -- 172.21.15.45:0/183320791 learned_addr learned my addr 172.21.15.45:0/183320791 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5400b3420 0x7fb5400b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53e59c640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb52c0047c0 0x7fb5401443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53dd9b640 1 -- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb52c0047c0 msgr2=0x7fb5401443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb52c0047c0 0x7fb5401443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53dd9b640 1 -- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5400b3420 msgr2=0x7fb5400b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5400b3420 0x7fb5400b38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53dd9b640 1 -- 172.21.15.45:0/183320791 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5401520a0 con 0x7fb54014ee70 2023-11-28T20:52:00.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53e59c640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb52c0047c0 0x7fb5401443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:00.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.359+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb54014ee70 0x7fb5400b2ee0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fb52802c2b0 tx=0x7fb528004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.360+0000 7fb536ffd640 1 -- 172.21.15.45:0/183320791 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528003810 con 0x7fb54014ee70 2023-11-28T20:52:00.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.360+0000 7fb536ffd640 1 -- 172.21.15.45:0/183320791 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb528003970 con 0x7fb54014ee70 2023-11-28T20:52:00.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.360+0000 7fb536ffd640 1 -- 172.21.15.45:0/183320791 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528003d00 con 0x7fb54014ee70 2023-11-28T20:52:00.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.360+0000 7fb53f59e640 1 -- 172.21.15.45:0/183320791 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54015d920 con 0x7fb54014ee70 2023-11-28T20:52:00.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.360+0000 7fb53f59e640 1 -- 172.21.15.45:0/183320791 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb540150840 con 0x7fb54014ee70 2023-11-28T20:52:00.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.362+0000 7fb536ffd640 1 -- 172.21.15.45:0/183320791 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fb528040450 con 0x7fb54014ee70 2023-11-28T20:52:00.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.363+0000 7fb536ffd640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb52006af00 0x7fb52006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.363+0000 7fb536ffd640 1 -- 172.21.15.45:0/183320791 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb5280ab700 con 0x7fb54014ee70 2023-11-28T20:52:00.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.364+0000 7fb53e59c640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb52006af00 0x7fb52006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.364+0000 7fb547840740 1 Processor -- start 2023-11-28T20:52:00.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.364+0000 7fb547840740 1 -- start start 2023-11-28T20:52:00.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb53e59c640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb52006af00 0x7fb52006d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb530007fd0 tx=0x7fb530009040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb547840740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591bab070 0x559591bab450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb547840740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 0x559591baa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb547840740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591baa7b0 0x559591bee730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb547840740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559591b09880 con 0x559591baa7b0 2023-11-28T20:52:00.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb547840740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559591b18ae0 con 0x559591bab070 2023-11-28T20:52:00.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb547840740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x559591b161d0 con 0x559591ba5d70 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb53ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591baa7b0 0x559591bee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb53ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591baa7b0 0x559591bee730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52600/0 (socket says 172.21.15.45:52600) 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb53ed9d640 1 -- 172.21.15.45:0/2770677446 learned_addr learned my addr 172.21.15.45:0/2770677446 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.365+0000 7fb53e59c640 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591bab070 0x559591bab450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 0x559591baa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb53ed9d640 1 -- 172.21.15.45:0/2770677446 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 msgr2=0x559591baa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 0x559591baa270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb53ed9d640 1 -- 172.21.15.45:0/2770677446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591bab070 msgr2=0x559591bab450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591bab070 0x559591bab450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb53ed9d640 1 -- 172.21.15.45:0/2770677446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559591ad9860 con 0x559591baa7b0 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb53e59c640 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591bab070 0x559591bab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591baa7b0 0x559591bee730 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fb52c004740 tx=0x7fb538005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bc9d3442f90eaa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb51ffff640 1 -- 172.21.15.45:0/2770677446 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53800e950 con 0x559591baa7b0 2023-11-28T20:52:00.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb51ffff640 1 -- 172.21.15.45:0/2770677446 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53800ead0 con 0x559591baa7b0 2023-11-28T20:52:00.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.366+0000 7fb51ffff640 1 -- 172.21.15.45:0/2770677446 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53800ee80 con 0x559591baa7b0 2023-11-28T20:52:00.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.367+0000 7fb547840740 1 -- 172.21.15.45:0/2770677446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591baa7b0 msgr2=0x559591bee730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.367+0000 7fb547840740 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591baa7b0 0x559591bee730 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fb52c004740 tx=0x7fb538005bc0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.367+0000 7fb547840740 1 -- 172.21.15.45:0/2770677446 shutdown_connections 2023-11-28T20:52:00.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.367+0000 7fb547840740 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591baa7b0 0x559591bee730 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.367+0000 7fb547840740 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 0x559591baa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.367+0000 7fb547840740 1 --2- 172.21.15.45:0/2770677446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591bab070 0x559591bab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.367+0000 7fb547840740 1 -- 172.21.15.45:0/2770677446 >> 172.21.15.45:0/2770677446 conn(0x559591b6cc70 msgr2=0x559591bf0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.367+0000 7fb547840740 1 -- 172.21.15.45:0/2770677446 shutdown_connections 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.367+0000 7fb547840740 1 -- 172.21.15.45:0/2770677446 wait complete. 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.368+0000 7fb547840740 1 -- start start 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.368+0000 7fb547840740 1 Processor -- start 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.369+0000 7fb547840740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 0x559591bd3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.369+0000 7fb547840740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 0x559591bd41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.369+0000 7fb547840740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bd4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.369+0000 7fb547840740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559591b11ac0 con 0x559591bab070 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.369+0000 7fb547840740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559591aae2f0 con 0x559591baa7b0 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.369+0000 7fb547840740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x559591acbc70 con 0x559591ba5d70 2023-11-28T20:52:00.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.369+0000 7fb547840740 10 client.0 _ll_register_callbacks cb 0x559591ad03b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bd4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bd4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52608/0 (socket says 172.21.15.45:52608) 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53ed9d640 1 -- 172.21.15.45:0/2134823208 learned_addr learned my addr 172.21.15.45:0/2134823208 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 0x559591bd41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53e59c640 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 0x559591bd3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53ed9d640 1 -- 172.21.15.45:0/2134823208 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 msgr2=0x559591bd3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 0x559591bd3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53ed9d640 1 -- 172.21.15.45:0/2134823208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 msgr2=0x559591bd41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 0x559591bd41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53ed9d640 1 -- 172.21.15.45:0/2134823208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53800f040 con 0x559591bab070 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 0x559591bd41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bd4710 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fb5400d0cf0 tx=0x7fb538000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e152bb8877f73ef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.370+0000 7fb51d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb51d7fa640 1 -- 172.21.15.45:0/2134823208 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538014640 con 0x559591bab070 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb51d7fa640 1 -- 172.21.15.45:0/2134823208 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb53801c070 con 0x559591bab070 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb547840740 1 -- 172.21.15.45:0/2134823208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559591b0aaa0 con 0x559591bab070 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb51d7fa640 1 -- 172.21.15.45:0/2134823208 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5380147a0 con 0x559591bab070 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb547840740 10 client.7752 fetch_fsmap learned FSMap version 51 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb547840740 1 -- 172.21.15.45:0/2134823208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x559591b10140 con 0x559591bab070 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb51d7fa640 1 -- 172.21.15.45:0/2134823208 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559591b0aaa0 con 0x559591bab070 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb51d7fa640 1 -- 172.21.15.45:0/2134823208 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb538014900 con 0x559591bab070 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb547840740 10 client.7752 fetch_fsmap finished waiting for FSMap version 51 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb547840740 10 client.7752 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T20:52:00.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb547840740 20 client.7752 populate_metadata read hostname 'smithi045' 2023-11-28T20:52:00.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb547840740 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] conn(0x559591bd0b10 0x559591c0df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb547840740 4 client.7752 mds_command: new command op to 6448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:52:00.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.371+0000 7fb547840740 1 -- 172.21.15.45:0/2134823208 --> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559591b0a7c0 con 0x559591bd0b10 2023-11-28T20:52:00.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.372+0000 7fb53e59c640 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] conn(0x559591bd0b10 0x559591c0df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.372+0000 7fb53e59c640 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] conn(0x559591bd0b10 0x559591c0df80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=200f43ad940bfcd6 server_cookie=6890530c1438d8f8 in_seq=0 out_seq=0 2023-11-28T20:52:00.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.372+0000 7fb51d7fa640 10 client.7752 ms_handle_connect on v2:172.21.15.45:6832/2489162496 2023-11-28T20:52:00.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb51d7fa640 1 -- 172.21.15.45:0/2134823208 <== mds.0 v2:172.21.15.45:6832/2489162496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559591b0a7c0 con 0x559591bd0b10 2023-11-28T20:52:00.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb51d7fa640 10 client.7752 handle_command_reply: tid=0 2023-11-28T20:52:00.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb547840740 1 client.7752 shutdown 2023-11-28T20:52:00.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb547840740 1 -- 172.21.15.45:0/2134823208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 msgr2=0x559591bd4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb547840740 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bd4710 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fb5400d0cf0 tx=0x7fb538000ed0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb547840740 1 -- 172.21.15.45:0/2134823208 shutdown_connections 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb547840740 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bd4710 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb547840740 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] conn(0x559591bd0b10 0x559591c0df80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb547840740 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 0x559591bd41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb547840740 1 --2- 172.21.15.45:0/2134823208 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591ba5d70 0x559591bd3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb51d7fa640 0 client.7752 ms_handle_reset on v2:172.21.15.45:6832/2489162496 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.373+0000 7fb547840740 1 -- 172.21.15.45:0/2134823208 >> 172.21.15.45:0/2134823208 conn(0x559591b6cc70 msgr2=0x559591bedd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.374+0000 7fb547840740 1 -- 172.21.15.45:0/2134823208 shutdown_connections 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.374+0000 7fb547840740 1 -- 172.21.15.45:0/2134823208 wait complete. 2023-11-28T20:52:00.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.374+0000 7fb547840740 20 client.7752 trim_cache size 0 max 16384 2023-11-28T20:52:00.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.391+0000 7fb547840740 1 Processor -- start 2023-11-28T20:52:00.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.391+0000 7fb547840740 1 -- start start 2023-11-28T20:52:00.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.392+0000 7fb547840740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591ba5d70 0x559591c10130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.392+0000 7fb547840740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 0x559591babf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.392+0000 7fb547840740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591bab070 0x559591bd2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.392+0000 7fb547840740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559591874f20 con 0x559591ba5d70 2023-11-28T20:52:00.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.392+0000 7fb547840740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559591bd9cb0 con 0x559591baa7b0 2023-11-28T20:52:00.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.392+0000 7fb547840740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x559591ac5e30 con 0x559591bab070 2023-11-28T20:52:00.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.392+0000 7fb53e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591ba5d70 0x559591c10130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.392+0000 7fb53e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591ba5d70 0x559591c10130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52614/0 (socket says 172.21.15.45:52614) 2023-11-28T20:52:00.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.392+0000 7fb53e59c640 1 -- 172.21.15.45:0/3481150843 learned_addr learned my addr 172.21.15.45:0/3481150843 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:00.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.393+0000 7fb53e59c640 1 -- 172.21.15.45:0/3481150843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591bab070 msgr2=0x559591bd2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.393+0000 7fb53e59c640 1 --2- 172.21.15.45:0/3481150843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591bab070 0x559591bd2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.393+0000 7fb53e59c640 1 -- 172.21.15.45:0/3481150843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 msgr2=0x559591babf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.393+0000 7fb53e59c640 1 --2- 172.21.15.45:0/3481150843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 0x559591babf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.393+0000 7fb53e59c640 1 -- 172.21.15.45:0/3481150843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559591aadb80 con 0x559591ba5d70 2023-11-28T20:52:00.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.393+0000 7fb53e59c640 1 --2- 172.21.15.45:0/3481150843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591ba5d70 0x559591c10130 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x559591bd0650 tx=0x7fb5300021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eab87a1e3fee501 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.393+0000 7fb534ff9640 1 -- 172.21.15.45:0/3481150843 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530027070 con 0x559591ba5d70 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.393+0000 7fb534ff9640 1 -- 172.21.15.45:0/3481150843 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53001b040 con 0x559591ba5d70 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.393+0000 7fb534ff9640 1 -- 172.21.15.45:0/3481150843 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53000c050 con 0x559591ba5d70 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.394+0000 7fb547840740 1 -- 172.21.15.45:0/3481150843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591ba5d70 msgr2=0x559591c10130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.394+0000 7fb547840740 1 --2- 172.21.15.45:0/3481150843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591ba5d70 0x559591c10130 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x559591bd0650 tx=0x7fb5300021f0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.394+0000 7fb547840740 1 -- 172.21.15.45:0/3481150843 shutdown_connections 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.394+0000 7fb547840740 1 --2- 172.21.15.45:0/3481150843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591bab070 0x559591bd2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.394+0000 7fb547840740 1 --2- 172.21.15.45:0/3481150843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591baa7b0 0x559591babf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.394+0000 7fb547840740 1 --2- 172.21.15.45:0/3481150843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591ba5d70 0x559591c10130 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.394+0000 7fb547840740 1 -- 172.21.15.45:0/3481150843 >> 172.21.15.45:0/3481150843 conn(0x559591b6cc70 msgr2=0x559591bc1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.394+0000 7fb547840740 1 -- 172.21.15.45:0/3481150843 shutdown_connections 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.394+0000 7fb547840740 1 -- 172.21.15.45:0/3481150843 wait complete. 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.395+0000 7fb547840740 1 -- start start 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.395+0000 7fb547840740 1 Processor -- start 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.396+0000 7fb547840740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591ba5d70 0x559591c0a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.396+0000 7fb547840740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591baa7b0 0x559591c0fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.396+0000 7fb547840740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bf3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.396+0000 7fb547840740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559591bd1a60 con 0x559591bab070 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.396+0000 7fb547840740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559591bd1f20 con 0x559591ba5d70 2023-11-28T20:52:00.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.396+0000 7fb547840740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x559591c1a3e0 con 0x559591baa7b0 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.396+0000 7fb547840740 10 client.0 _ll_register_callbacks cb 0x7fb538001cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.396+0000 7fb53ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bf3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bf3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52622/0 (socket says 172.21.15.45:52622) 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53ed9d640 1 -- 172.21.15.45:0/97257989 learned_addr learned my addr 172.21.15.45:0/97257989 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53e59c640 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591ba5d70 0x559591c0a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53dd9b640 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591baa7b0 0x559591c0fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53ed9d640 1 -- 172.21.15.45:0/97257989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591baa7b0 msgr2=0x559591c0fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591baa7b0 0x559591c0fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53ed9d640 1 -- 172.21.15.45:0/97257989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591ba5d70 msgr2=0x559591c0a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591ba5d70 0x559591c0a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53ed9d640 1 -- 172.21.15.45:0/97257989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5401613c0 con 0x559591bab070 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb53ed9d640 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bf3ca0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb538026040 tx=0x7fb538000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfa6119961402f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.397+0000 7fb51d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb51d7fa640 1 -- 172.21.15.45:0/97257989 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5380016f0 con 0x559591bab070 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb51d7fa640 1 -- 172.21.15.45:0/97257989 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb538001510 con 0x559591bab070 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb547840740 1 -- 172.21.15.45:0/97257989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb53800ead0 con 0x559591bab070 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb51d7fa640 1 -- 172.21.15.45:0/97257989 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb53800ee50 con 0x559591bab070 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb547840740 10 client.7758 fetch_fsmap learned FSMap version 51 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb547840740 1 -- 172.21.15.45:0/97257989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x559591ad3170 con 0x559591bab070 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb51d7fa640 1 -- 172.21.15.45:0/97257989 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb538035020 con 0x559591bab070 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb51d7fa640 1 -- 172.21.15.45:0/97257989 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb538021a70 con 0x559591bab070 2023-11-28T20:52:00.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb547840740 10 client.7758 fetch_fsmap finished waiting for FSMap version 51 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb547840740 10 client.7758 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb547840740 20 client.7758 populate_metadata read hostname 'smithi045' 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb547840740 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] conn(0x559591bf43d0 0x559591bce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb547840740 4 client.7758 mds_command: new command op to 6448 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.398+0000 7fb547840740 1 -- 172.21.15.45:0/97257989 --> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5595918126c0 con 0x559591bf43d0 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.399+0000 7fb53e59c640 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] conn(0x559591bf43d0 0x559591bce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.399+0000 7fb53e59c640 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] conn(0x559591bf43d0 0x559591bce7f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64d66a72de750c46 server_cookie=3883ef1e59807796 in_seq=0 out_seq=0 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.399+0000 7fb51d7fa640 10 client.7758 ms_handle_connect on v2:172.21.15.45:6832/2489162496 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.399+0000 7fb51d7fa640 1 -- 172.21.15.45:0/97257989 <== mds.0 v2:172.21.15.45:6832/2489162496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5595918126c0 con 0x559591bf43d0 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.399+0000 7fb51d7fa640 10 client.7758 handle_command_reply: tid=0 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.399+0000 7fb547840740 1 client.7758 shutdown 2023-11-28T20:52:00.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 -- 172.21.15.45:0/97257989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 msgr2=0x559591bf3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bf3ca0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb538026040 tx=0x7fb538000f40 comp rx=0 tx=0).stop 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 -- 172.21.15.45:0/97257989 shutdown_connections 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559591bab070 0x559591bf3ca0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x559591baa7b0 0x559591c0fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.45:6832/2489162496,v1:172.21.15.45:6834/2489162496] conn(0x559591bf43d0 0x559591bce7f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 --2- 172.21.15.45:0/97257989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559591ba5d70 0x559591c0a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 -- 172.21.15.45:0/97257989 >> 172.21.15.45:0/97257989 conn(0x559591b6cc70 msgr2=0x559591ba4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb51d7fa640 0 client.7758 ms_handle_reset on v2:172.21.15.45:6832/2489162496 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 -- 172.21.15.45:0/97257989 shutdown_connections 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 1 -- 172.21.15.45:0/97257989 wait complete. 2023-11-28T20:52:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.400+0000 7fb547840740 20 client.7758 trim_cache size 0 max 16384 2023-11-28T20:52:00.410 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:52:00.410 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:52:00.410 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:52:00.410 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 6448, 2023-11-28T20:52:00.410 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:52:00.410 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:52:00.410 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:52:00.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.8656170440000004, 2023-11-28T20:52:00.411 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 51, 2023-11-28T20:52:00.411 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 116, 2023-11-28T20:52:00.411 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 116, 2023-11-28T20:52:00.411 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 13.226005515000001 2023-11-28T20:52:00.411 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:52:00.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 -- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb52006af00 msgr2=0x7fb52006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb52006af00 0x7fb52006d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb530007fd0 tx=0x7fb530009040 comp rx=0 tx=0).stop 2023-11-28T20:52:00.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 -- 172.21.15.45:0/183320791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb54014ee70 msgr2=0x7fb5400b2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb54014ee70 0x7fb5400b2ee0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fb52802c2b0 tx=0x7fb528004730 comp rx=0 tx=0).stop 2023-11-28T20:52:00.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 -- 172.21.15.45:0/183320791 shutdown_connections 2023-11-28T20:52:00.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fb52006af00 0x7fb52006d3c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5400b3420 0x7fb5400b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb54014ee70 0x7fb5400b2ee0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 --2- 172.21.15.45:0/183320791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb52c0047c0 0x7fb5401443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 -- 172.21.15.45:0/183320791 >> 172.21.15.45:0/183320791 conn(0x7fb54006a500 msgr2=0x7fb5400d2730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:00.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 -- 172.21.15.45:0/183320791 shutdown_connections 2023-11-28T20:52:00.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.401+0000 7fb53f59e640 1 -- 172.21.15.45:0/183320791 wait complete. 2023-11-28T20:52:00.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:52:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.636+0000 7f80d28ff640 1 Processor -- start 2023-11-28T20:52:00.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.636+0000 7f80d28ff640 1 -- start start 2023-11-28T20:52:00.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.637+0000 7f80d28ff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f80cc155780 0x7f80cc0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.637+0000 7f80d28ff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc15c3c0 con 0x7f80cc0ae540 2023-11-28T20:52:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.637+0000 7f80d28ff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc155b60 con 0x7f80cc0ce800 2023-11-28T20:52:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.637+0000 7f80d28ff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80cc15ba90 con 0x7f80cc155780 2023-11-28T20:52:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.637+0000 7f80d18fd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f80cc155780 0x7f80cc0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.638+0000 7f80d18fd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f80cc155780 0x7f80cc0abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52638/0 (socket says 172.21.15.45:52638) 2023-11-28T20:52:00.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.638+0000 7f80d18fd640 1 -- 172.21.15.45:0/13484649 learned_addr learned my addr 172.21.15.45:0/13484649 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.638+0000 7f80d18fd640 1 -- 172.21.15.45:0/13484649 >> v1:172.21.15.190:6789/0 conn(0x7f80cc0ce800 legacy=0x7f80cc0b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.638+0000 7f80d18fd640 1 -- 172.21.15.45:0/13484649 >> v1:172.21.15.45:6789/0 conn(0x7f80cc0ae540 legacy=0x7f80cc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.638+0000 7f80d18fd640 1 -- 172.21.15.45:0/13484649 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc15bd70 con 0x7f80cc155780 2023-11-28T20:52:00.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.639+0000 7f80d18fd640 1 --2- 172.21.15.45:0/13484649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc155780 0x7f80cc0abd70 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009e50 tx=0x7f80c002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8764fb53d0f499 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.640+0000 7f80d08fb640 1 -- 172.21.15.45:0/13484649 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c002cb10 con 0x7f80cc155780 2023-11-28T20:52:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.640+0000 7f80d08fb640 1 -- 172.21.15.45:0/13484649 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c002cc70 con 0x7f80cc155780 2023-11-28T20:52:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.640+0000 7f80d08fb640 1 -- 172.21.15.45:0/13484649 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c00343f0 con 0x7f80cc155780 2023-11-28T20:52:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.640+0000 7f80d28ff640 1 -- 172.21.15.45:0/13484649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc155780 msgr2=0x7f80cc0abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.640+0000 7f80d28ff640 1 --2- 172.21.15.45:0/13484649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc155780 0x7f80cc0abd70 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009e50 tx=0x7f80c002bf70 comp rx=0 tx=0).stop 2023-11-28T20:52:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.640+0000 7f80d28ff640 1 -- 172.21.15.45:0/13484649 shutdown_connections 2023-11-28T20:52:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.640+0000 7f80d28ff640 1 --2- 172.21.15.45:0/13484649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc155780 0x7f80cc0abd70 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.640+0000 7f80d28ff640 1 -- 172.21.15.45:0/13484649 >> 172.21.15.45:0/13484649 conn(0x7f80cc06ab30 msgr2=0x7f80cc0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.641+0000 7f80d28ff640 1 -- 172.21.15.45:0/13484649 shutdown_connections 2023-11-28T20:52:00.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.641+0000 7f80d28ff640 1 -- 172.21.15.45:0/13484649 wait complete. 2023-11-28T20:52:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.642+0000 7f80d28ff640 1 Processor -- start 2023-11-28T20:52:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.642+0000 7f80d28ff640 1 -- start start 2023-11-28T20:52:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.643+0000 7f80d28ff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80cc0ae540 0x7f80cc1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.643+0000 7f80d28ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0ce800 0x7f80cc099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.643+0000 7f80d28ff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80cc155780 0x7f80cc0afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d28ff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc09b010 con 0x7f80cc0ce800 2023-11-28T20:52:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d28ff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc158600 con 0x7f80cc155780 2023-11-28T20:52:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d28ff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc09a390 con 0x7f80cc0ae540 2023-11-28T20:52:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d10fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0ce800 0x7f80cc099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d10fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0ce800 0x7f80cc099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52654/0 (socket says 172.21.15.45:52654) 2023-11-28T20:52:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d10fc640 1 -- 172.21.15.45:0/4238826550 learned_addr learned my addr 172.21.15.45:0/4238826550 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d10fc640 1 -- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80cc0ae540 msgr2=0x7f80cc1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d10fc640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80cc0ae540 0x7f80cc1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d10fc640 1 -- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80cc155780 msgr2=0x7f80cc0afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d10fc640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80cc155780 0x7f80cc0afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d10fc640 1 -- 172.21.15.45:0/4238826550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c0009b30 con 0x7f80cc0ce800 2023-11-28T20:52:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.644+0000 7f80d10fc640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0ce800 0x7f80cc099a40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f80b800c450 tx=0x7f80b800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.645+0000 7f80ca7fc640 1 -- 172.21.15.45:0/4238826550 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b80052d0 con 0x7f80cc0ce800 2023-11-28T20:52:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.645+0000 7f80ca7fc640 1 -- 172.21.15.45:0/4238826550 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b8005430 con 0x7f80cc0ce800 2023-11-28T20:52:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.645+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80cc09a670 con 0x7f80cc0ce800 2023-11-28T20:52:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.645+0000 7f80ca7fc640 1 -- 172.21.15.45:0/4238826550 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b8012870 con 0x7f80cc0ce800 2023-11-28T20:52:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.645+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80cc09ac30 con 0x7f80cc0ce800 2023-11-28T20:52:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.646+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80cc09b150 con 0x7f80cc0ce800 2023-11-28T20:52:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.648+0000 7f80ca7fc640 1 -- 172.21.15.45:0/4238826550 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f80b80129d0 con 0x7f80cc0ce800 2023-11-28T20:52:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.648+0000 7f80ca7fc640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f80b006aed0 0x7f80b006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.649+0000 7f80ca7fc640 1 -- 172.21.15.45:0/4238826550 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f80b8089330 con 0x7f80cc0ce800 2023-11-28T20:52:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.649+0000 7f80d18fd640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f80b006aed0 0x7f80b006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.650+0000 7f80d18fd640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f80b006aed0 0x7f80b006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f80c0035000 tx=0x7f80c002e000 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.653+0000 7f80ca7fc640 1 -- 172.21.15.45:0/4238826550 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80cc09b150 con 0x7f80cc0ce800 2023-11-28T20:52:00.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.889+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f80cc09b150 con 0x7f80cc0ce800 2023-11-28T20:52:00.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.890+0000 7f80ca7fc640 1 -- 172.21.15.45:0/4238826550 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) v1 ==== 53+0+73 (secure 0 0 0) 0x7f80cc09b150 con 0x7f80cc0ce800 2023-11-28T20:52:00.892 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T20:52:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.892+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f80b006aed0 msgr2=0x7f80b006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f80b006aed0 0x7f80b006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f80c0035000 tx=0x7f80c002e000 comp rx=0 tx=0).stop 2023-11-28T20:52:00.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0ce800 msgr2=0x7f80cc099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0ce800 0x7f80cc099a40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f80b800c450 tx=0x7f80b800c910 comp rx=0 tx=0).stop 2023-11-28T20:52:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 shutdown_connections 2023-11-28T20:52:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f80b006aed0 0x7f80b006d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80cc155780 0x7f80cc0afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80cc0ce800 0x7f80cc099a40 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 --2- 172.21.15.45:0/4238826550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80cc0ae540 0x7f80cc1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 >> 172.21.15.45:0/4238826550 conn(0x7f80cc06ab30 msgr2=0x7f80cc15d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 shutdown_connections 2023-11-28T20:52:00.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:00.893+0000 7f80d28ff640 1 -- 172.21.15.45:0/4238826550 wait complete. 2023-11-28T20:52:00.913 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T20:52:00.913 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:52:00.913 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:52:00.913 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T20:52:00.913 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:52:00.913 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T20:52:00.913 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T20:52:00.913 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:52:00.914 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:52:00.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:52:00.914 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:52:00.970 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:52:00.970 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:52:00.970 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:52:00.970 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:52:00.971 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 41222sec preferred_lft 41222sec 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T20:52:00.972 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:52:00.973 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:52:00.973 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:52:00.973 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T20:52:00.973 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:52:00.973 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:52:00.973 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T20:52:00.973 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T20:52:00.973 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:52:00.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:52:00.974 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:52:01.073 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:52:01.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:52:01.077 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:52:01.136 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:52:01.136 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:52:01.136 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:52:01.137 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:52:01.137 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:52:01.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:52:01.138 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:52:01.138 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:52:01.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:52:01.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:52:01.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:52:01.138 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:52:01.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:52:01.337 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:52:01.392 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T20:52:01.392 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T20:52:01.393 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:52:01.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:52:01.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:52:01.494 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:52:01.495 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T20:52:01.556 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:52:01.556 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:52:01.556 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:52:01.607 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:52:01.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:52:01.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:52:01.753 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:52:01.753 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:52:01.754 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:52:01.776 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:52:01.776 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:52:01.776 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:52:01.776 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:52:01.777 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:52:01.803 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:52:01.804 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:52:01.804 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:52:01.804 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:52:01.804 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:52:01.804 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:52:01.804 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:52:01.804 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:52:01.806 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T20:52:01.806 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:52:01.806 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:52:01.806 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:52:01.806 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:52:01.810 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:52:01.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:52:01.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:52:01.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:52:01.909 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:52:01.974 INFO:teuthology.orchestra.run.smithi045.stdout:client6556 2023-11-28T20:52:01.975 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:52:01.975 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:52:02.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T20:52:02.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.304+0000 7f6b2ce1c640 1 Processor -- start 2023-11-28T20:52:02.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.305+0000 7f6b2ce1c640 1 -- start start 2023-11-28T20:52:02.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.307+0000 7f6b2ce1c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6b280cf730 0x7f6b2806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.307+0000 7f6b2ce1c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6b2815edc0 0x7f6b2815f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.307+0000 7f6b2ce1c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2815fa80 con 0x7f6b2815b820 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.307+0000 7f6b2ce1c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b2815fd60 con 0x7f6b2815edc0 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.307+0000 7f6b2ce1c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b2806cf40 con 0x7f6b280cf730 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.307+0000 7f6b26d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6b2815edc0 0x7f6b2815f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.307+0000 7f6b26d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6b2815edc0 0x7f6b2815f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52662/0 (socket says 172.21.15.45:52662) 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.307+0000 7f6b26d9d640 1 -- 172.21.15.45:0/2990801051 learned_addr learned my addr 172.21.15.45:0/2990801051 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.307+0000 7f6b25d9b640 1 --2- 172.21.15.45:0/2990801051 >> v2:172.21.15.190:3300/0 conn(0x7f6b280cf730 0x7f6b2806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.308+0000 7f6b26d9d640 1 -- 172.21.15.45:0/2990801051 >> v2:172.21.15.190:3300/0 conn(0x7f6b280cf730 msgr2=0x7f6b2806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.308+0000 7f6b26d9d640 1 --2- 172.21.15.45:0/2990801051 >> v2:172.21.15.190:3300/0 conn(0x7f6b280cf730 0x7f6b2806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.308+0000 7f6b26d9d640 1 -- 172.21.15.45:0/2990801051 >> v1:172.21.15.190:6789/0 conn(0x7f6b2815b820 legacy=0x7f6b280cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.308+0000 7f6b26d9d640 1 -- 172.21.15.45:0/2990801051 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2806d220 con 0x7f6b2815edc0 2023-11-28T20:52:02.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.309+0000 7f6b26d9d640 1 --2- 172.21.15.45:0/2990801051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 0x7f6b2815f1e0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f6b2809cf90 tx=0x7f6b2002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e33fe58563a784 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.309+0000 7f6b2559a640 1 -- 172.21.15.45:0/2990801051 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2002ca80 con 0x7f6b2815edc0 2023-11-28T20:52:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.309+0000 7f6b2559a640 1 -- 172.21.15.45:0/2990801051 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2002cbe0 con 0x7f6b2815edc0 2023-11-28T20:52:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.309+0000 7f6b2559a640 1 -- 172.21.15.45:0/2990801051 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b200343f0 con 0x7f6b2815edc0 2023-11-28T20:52:02.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.310+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/2990801051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 msgr2=0x7f6b2815f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.310+0000 7f6b2ce1c640 1 --2- 172.21.15.45:0/2990801051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 0x7f6b2815f1e0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f6b2809cf90 tx=0x7f6b2002bee0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.310+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/2990801051 shutdown_connections 2023-11-28T20:52:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.310+0000 7f6b2ce1c640 1 --2- 172.21.15.45:0/2990801051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 0x7f6b2815f1e0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.310+0000 7f6b2ce1c640 1 --2- 172.21.15.45:0/2990801051 >> v2:172.21.15.190:3300/0 conn(0x7f6b280cf730 0x7f6b2806da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.310+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/2990801051 >> 172.21.15.45:0/2990801051 conn(0x7f6b2806a6e0 msgr2=0x7f6b2806aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.310+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/2990801051 shutdown_connections 2023-11-28T20:52:02.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.310+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/2990801051 wait complete. 2023-11-28T20:52:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.312+0000 7f6b2ce1c640 1 Processor -- start 2023-11-28T20:52:02.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.312+0000 7f6b2ce1c640 1 -- start start 2023-11-28T20:52:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.313+0000 7f6b2ce1c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b280cf730 0x7f6b281f0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.313+0000 7f6b2ce1c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2815b820 0x7f6b28158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.313+0000 7f6b2ce1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 0x7f6b281588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.313+0000 7f6b2ce1c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b281f0960 con 0x7f6b2815edc0 2023-11-28T20:52:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.313+0000 7f6b2ce1c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b281520b0 con 0x7f6b280cf730 2023-11-28T20:52:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.313+0000 7f6b2ce1c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28152390 con 0x7f6b2815b820 2023-11-28T20:52:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.313+0000 7f6b26d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 0x7f6b281588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.313+0000 7f6b26d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 0x7f6b281588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52670/0 (socket says 172.21.15.45:52670) 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.313+0000 7f6b26d9d640 1 -- 172.21.15.45:0/1063612223 learned_addr learned my addr 172.21.15.45:0/1063612223 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.314+0000 7f6b25d9b640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2815b820 0x7f6b28158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.314+0000 7f6b26d9d640 1 -- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2815b820 msgr2=0x7f6b28158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.314+0000 7f6b26d9d640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2815b820 0x7f6b28158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.314+0000 7f6b26d9d640 1 -- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b280cf730 msgr2=0x7f6b281f0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.314+0000 7f6b26d9d640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b280cf730 0x7f6b281f0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.314+0000 7f6b26d9d640 1 -- 172.21.15.45:0/1063612223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b20009b30 con 0x7f6b2815edc0 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.314+0000 7f6b26d9d640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 0x7f6b281588b0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f6b20036040 tx=0x7f6b20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.314+0000 7f6b16ffd640 1 -- 172.21.15.45:0/1063612223 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b20009510 con 0x7f6b2815edc0 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.315+0000 7f6b16ffd640 1 -- 172.21.15.45:0/1063612223 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2003e450 con 0x7f6b2815edc0 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.315+0000 7f6b16ffd640 1 -- 172.21.15.45:0/1063612223 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2003e7e0 con 0x7f6b2815edc0 2023-11-28T20:52:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.315+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b281f41e0 con 0x7f6b2815edc0 2023-11-28T20:52:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.315+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b281f47a0 con 0x7f6b2815edc0 2023-11-28T20:52:02.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.316+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b281f4040 con 0x7f6b2815edc0 2023-11-28T20:52:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.317+0000 7f6b16ffd640 1 -- 172.21.15.45:0/1063612223 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 45) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f6b2002cd50 con 0x7f6b2815edc0 2023-11-28T20:52:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.318+0000 7f6b16ffd640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6b0806aed0 0x7f6b0806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.318+0000 7f6b16ffd640 1 -- 172.21.15.45:0/1063612223 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f6b200ac400 con 0x7f6b2815edc0 2023-11-28T20:52:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.319+0000 7f6b2659c640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6b0806aed0 0x7f6b0806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.320+0000 7f6b2659c640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6b0806aed0 0x7f6b0806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b18001560 tx=0x7f6b18007990 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.322+0000 7f6b16ffd640 1 -- 172.21.15.45:0/1063612223 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b281f4040 con 0x7f6b2815edc0 2023-11-28T20:52:02.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.543+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 --> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b28067eb0 con 0x7f6b0806aed0 2023-11-28T20:52:02.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.548+0000 7f6b16ffd640 1 -- 172.21.15.45:0/1063612223 <== mgr.5828 v2:172.21.15.190:6832/4159844301 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6b28067eb0 con 0x7f6b0806aed0 2023-11-28T20:52:02.549 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:52:02.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:52:02.549 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T20:52:02.549 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:52:02.550 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:52:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6b0806aed0 msgr2=0x7f6b0806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6b0806aed0 0x7f6b0806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b18001560 tx=0x7f6b18007990 comp rx=0 tx=0).stop 2023-11-28T20:52:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 msgr2=0x7f6b281588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:02.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 0x7f6b281588b0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f6b20036040 tx=0x7f6b20004a40 comp rx=0 tx=0).stop 2023-11-28T20:52:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 shutdown_connections 2023-11-28T20:52:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f6b0806aed0 0x7f6b0806d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b2815edc0 0x7f6b281588b0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b2815b820 0x7f6b28158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 --2- 172.21.15.45:0/1063612223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b280cf730 0x7f6b281f0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 >> 172.21.15.45:0/1063612223 conn(0x7f6b2806a6e0 msgr2=0x7f6b28159770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 shutdown_connections 2023-11-28T20:52:02.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.551+0000 7f6b2ce1c640 1 -- 172.21.15.45:0/1063612223 wait complete. 2023-11-28T20:52:02.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T20:52:02.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.805+0000 7f2a7f009640 1 Processor -- start 2023-11-28T20:52:02.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.805+0000 7f2a7f009640 1 -- start start 2023-11-28T20:52:02.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.807+0000 7f2a7f009640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a7815d060 0x7f2a780ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.807+0000 7f2a7f009640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2a7806bc20 0x7f2a780b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.807+0000 7f2a7f009640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a780af3e0 con 0x7f2a78161900 2023-11-28T20:52:02.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.807+0000 7f2a7f009640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a78160e20 con 0x7f2a7806bc20 2023-11-28T20:52:02.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.807+0000 7f2a7f009640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a78161100 con 0x7f2a7815d060 2023-11-28T20:52:02.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.807+0000 7f2a7d806640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2a78161900 0x7f2a7806b510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44196/0 (socket says 172.21.15.45:44196) 2023-11-28T20:52:02.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.807+0000 7f2a7d806640 1 -- 172.21.15.45:0/3511807851 learned_addr learned my addr 172.21.15.45:0/3511807851 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:52:02.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7e808640 1 --2- 172.21.15.45:0/3511807851 >> v2:172.21.15.190:3300/0 conn(0x7f2a7806bc20 0x7f2a780b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7e007640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a7815d060 0x7f2a780ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141936534 0 0) 0x7f2a780af3e0 con 0x7f2a78161900 2023-11-28T20:52:02.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a64003520 con 0x7f2a78161900 2023-11-28T20:52:02.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3636675642 0 0) 0x7f2a64003520 con 0x7f2a78161900 2023-11-28T20:52:02.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a7815d060 msgr2=0x7f2a780ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:02.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a7815d060 0x7f2a780ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 >> v2:172.21.15.190:3300/0 conn(0x7f2a7806bc20 msgr2=0x7f2a780b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:02.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 --2- 172.21.15.45:0/3511807851 >> v2:172.21.15.190:3300/0 conn(0x7f2a7806bc20 0x7f2a780b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a781613e0 con 0x7f2a78161900 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f2a68001670 con 0x7f2a78161900 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.808+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 >> v1:172.21.15.45:6789/0 conn(0x7f2a78161900 legacy=0x7f2a7806b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d005640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a64003970 0x7f2a64003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d005640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640047c0 0x7f2a64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d005640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a7815d060 0x7f2a64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a78160e20 con 0x7f2a640047c0 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a78161100 con 0x7f2a64003970 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6400b980 con 0x7f2a7815d060 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7e808640 1 --2- 172.21.15.45:0/3511807851 >> v2:172.21.15.190:3300/0 conn(0x7f2a7806bc20 0x7f2a780b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:52:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d806640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640047c0 0x7f2a64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7e007640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a64003970 0x7f2a64003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d806640 1 -- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a7815d060 msgr2=0x7f2a64005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d806640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a7815d060 0x7f2a64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d806640 1 -- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a64003970 msgr2=0x7f2a64003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d806640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a64003970 0x7f2a64003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.809+0000 7f2a7d806640 1 -- 172.21.15.45:0/3511807851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a64003520 con 0x7f2a640047c0 2023-11-28T20:52:02.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.811+0000 7f2a7d806640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640047c0 0x7f2a64004c40 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c210 tx=0x7f2a640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98729150b1b0eb0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:02.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.811+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68005ad0 con 0x7f2a640047c0 2023-11-28T20:52:02.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.811+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a68005c30 con 0x7f2a640047c0 2023-11-28T20:52:02.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.811+0000 7f2a7d005640 1 -- 172.21.15.45:0/3511807851 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6802c8d0 con 0x7f2a640047c0 2023-11-28T20:52:02.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.812+0000 7f2a7f009640 1 -- 172.21.15.45:0/3511807851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640047c0 msgr2=0x7f2a64004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:02.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.812+0000 7f2a7f009640 1 --2- 172.21.15.45:0/3511807851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a640047c0 0x7f2a64004c40 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c210 tx=0x7f2a640057f0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.812+0000 7f2a7e808640 1 -- 172.21.15.45:0/3511807851 reap_dead start 2023-11-28T20:52:02.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.812+0000 7f2a7f009640 1 -- 172.21.15.45:0/3511807851 shutdown_connections 2023-11-28T20:52:02.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.812+0000 7f2a7f009640 1 -- 172.21.15.45:0/3511807851 >> 172.21.15.45:0/3511807851 conn(0x7f2a78071290 msgr2=0x7f2a780716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:02.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.812+0000 7f2a7f009640 1 -- 172.21.15.45:0/3511807851 shutdown_connections 2023-11-28T20:52:02.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.812+0000 7f2a7f009640 1 -- 172.21.15.45:0/3511807851 wait complete. 2023-11-28T20:52:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.814+0000 7f2a7f009640 1 Processor -- start 2023-11-28T20:52:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.814+0000 7f2a7f009640 1 -- start start 2023-11-28T20:52:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.815+0000 7f2a7f009640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a640047c0 0x7f2a7814bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.815+0000 7f2a7f009640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a781f2bf0 0x7f2a781f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.815+0000 7f2a7f009640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a781f66f0 0x7f2a780b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.815+0000 7f2a7f009640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7815af70 con 0x7f2a781f66f0 2023-11-28T20:52:02.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.815+0000 7f2a7f009640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a781f6c70 con 0x7f2a640047c0 2023-11-28T20:52:02.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.815+0000 7f2a7f009640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a781f6f50 con 0x7f2a781f2bf0 2023-11-28T20:52:02.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e808640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a781f66f0 0x7f2a780b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e808640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a781f66f0 0x7f2a780b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52684/0 (socket says 172.21.15.45:52684) 2023-11-28T20:52:02.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e808640 1 -- 172.21.15.45:0/3692557237 learned_addr learned my addr 172.21.15.45:0/3692557237 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:02.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7d806640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a781f2bf0 0x7f2a781f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e007640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a640047c0 0x7f2a7814bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e808640 1 -- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a781f2bf0 msgr2=0x7f2a781f61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e808640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a781f2bf0 0x7f2a781f61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e808640 1 -- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a640047c0 msgr2=0x7f2a7814bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e808640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a640047c0 0x7f2a7814bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e808640 1 -- 172.21.15.45:0/3692557237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7815b180 con 0x7f2a781f66f0 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7e007640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a640047c0 0x7f2a7814bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.816+0000 7f2a7d806640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a781f2bf0 0x7f2a781f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.817+0000 7f2a7e808640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a781f66f0 0x7f2a780b0370 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f2a7806fe10 tx=0x7f2a7000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.817+0000 7f2a6effd640 1 -- 172.21.15.45:0/3692557237 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7000de90 con 0x7f2a781f66f0 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.817+0000 7f2a6effd640 1 -- 172.21.15.45:0/3692557237 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7000f040 con 0x7f2a781f66f0 2023-11-28T20:52:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.817+0000 7f2a6effd640 1 -- 172.21.15.45:0/3692557237 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a70003bd0 con 0x7f2a781f66f0 2023-11-28T20:52:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.817+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7815b460 con 0x7f2a781f66f0 2023-11-28T20:52:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.817+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7814c040 con 0x7f2a781f66f0 2023-11-28T20:52:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.818+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a78073840 con 0x7f2a781f66f0 2023-11-28T20:52:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.820+0000 7f2a6effd640 1 -- 172.21.15.45:0/3692557237 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 46) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2a70003d30 con 0x7f2a781f66f0 2023-11-28T20:52:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.820+0000 7f2a6effd640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f2a5c06af60 0x7f2a5c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.820+0000 7f2a6effd640 1 -- 172.21.15.45:0/3692557237 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f2a70089a70 con 0x7f2a781f66f0 2023-11-28T20:52:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.821+0000 7f2a7e007640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f2a5c06af60 0x7f2a5c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:02.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.822+0000 7f2a7e007640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f2a5c06af60 0x7f2a5c06d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2a74006ce0 tx=0x7f2a74008040 comp rx=0 tx=0).ready entity=mgr.5828 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:02.825+0000 7f2a6effd640 1 -- 172.21.15.45:0/3692557237 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a700546f0 con 0x7f2a781f66f0 2023-11-28T20:52:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.069+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f2a781f7110 con 0x7f2a781f66f0 2023-11-28T20:52:03.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.605+0000 7f2a6effd640 1 -- 172.21.15.45:0/3692557237 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v47) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2a70057d10 con 0x7f2a781f66f0 2023-11-28T20:52:03.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.609+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f2a5c06af60 msgr2=0x7f2a5c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:03.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.609+0000 7f2a7f009640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f2a5c06af60 0x7f2a5c06d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2a74006ce0 tx=0x7f2a74008040 comp rx=0 tx=0).stop 2023-11-28T20:52:03.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.609+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a781f66f0 msgr2=0x7f2a780b0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:03.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.609+0000 7f2a7f009640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a781f66f0 0x7f2a780b0370 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f2a7806fe10 tx=0x7f2a7000d4a0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.610+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 shutdown_connections 2023-11-28T20:52:03.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.610+0000 7f2a7f009640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f2a5c06af60 0x7f2a5c06d420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.610+0000 7f2a7f009640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a781f66f0 0x7f2a780b0370 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.610+0000 7f2a7f009640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a781f2bf0 0x7f2a781f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.610+0000 7f2a7f009640 1 --2- 172.21.15.45:0/3692557237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a640047c0 0x7f2a7814bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.610+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 >> 172.21.15.45:0/3692557237 conn(0x7f2a78071290 msgr2=0x7f2a781586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:03.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.611+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 shutdown_connections 2023-11-28T20:52:03.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.611+0000 7f2a7f009640 1 -- 172.21.15.45:0/3692557237 wait complete. 2023-11-28T20:52:03.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T20:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.838+0000 7f6836609640 1 Processor -- start 2023-11-28T20:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.839+0000 7f6836609640 1 -- start start 2023-11-28T20:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.840+0000 7f6836609640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f68300d0040 0x7f68300d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.840+0000 7f6836609640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68300b90a0 con 0x7f68300b2290 2023-11-28T20:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.840+0000 7f6836609640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68300d1920 con 0x7f68300b1d40 2023-11-28T20:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.840+0000 7f6836609640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68300d3e60 con 0x7f68300d0040 2023-11-28T20:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.840+0000 7f6834e06640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f68300b2290 0x7f68300d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44216/0 (socket says 172.21.15.45:44216) 2023-11-28T20:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6834e06640 1 -- 172.21.15.45:0/4075202217 learned_addr learned my addr 172.21.15.45:0/4075202217 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6835607640 1 --2- 172.21.15.45:0/4075202217 >> v2:172.21.15.190:3300/0 conn(0x7f68300d0040 0x7f68300d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524492218 0 0) 0x7f68300b90a0 con 0x7f68300b2290 2023-11-28T20:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f681c003520 con 0x7f68300b2290 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4256959848 0 0) 0x7f681c003520 con 0x7f68300b2290 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 >> v2:172.21.15.190:3300/0 conn(0x7f68300d0040 msgr2=0x7f68300d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6827fff640 1 --2- 172.21.15.45:0/4075202217 >> v2:172.21.15.190:3300/0 conn(0x7f68300d0040 0x7f68300d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 >> v1:172.21.15.190:6789/0 conn(0x7f68300b1d40 legacy=0x7f68300d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68300d2dc0 con 0x7f68300b2290 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.841+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6820001670 con 0x7f68300b2290 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 >> v1:172.21.15.45:6789/0 conn(0x7f68300b2290 legacy=0x7f68300d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6827fff640 1 --2- 172.21.15.45:0/4075202217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c0039c0 0x7f681c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6827fff640 1 --2- 172.21.15.45:0/4075202217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f681c0047c0 0x7f681c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6827fff640 1 --2- 172.21.15.45:0/4075202217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f681c005180 0x7f681c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68300d3e60 con 0x7f681c0039c0 2023-11-28T20:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c00c060 con 0x7f681c005180 2023-11-28T20:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c00c220 con 0x7f681c0047c0 2023-11-28T20:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6835607640 1 --2- 172.21.15.45:0/4075202217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c0039c0 0x7f681c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6835607640 1 -- 172.21.15.45:0/4075202217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f681c0047c0 msgr2=0x7f681c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6835607640 1 --2- 172.21.15.45:0/4075202217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f681c0047c0 0x7f681c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6835607640 1 -- 172.21.15.45:0/4075202217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f681c005180 msgr2=0x7f681c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6835607640 1 --2- 172.21.15.45:0/4075202217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f681c005180 0x7f681c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6835607640 1 -- 172.21.15.45:0/4075202217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c003520 con 0x7f681c0039c0 2023-11-28T20:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.842+0000 7f6835e08640 1 -- 172.21.15.45:0/4075202217 reap_dead start 2023-11-28T20:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.843+0000 7f6835607640 1 --2- 172.21.15.45:0/4075202217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c0039c0 0x7f681c003de0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f682c009fc0 tx=0x7f681c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16009af8abaf213 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.844+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c02cc60 con 0x7f681c0039c0 2023-11-28T20:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.844+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682c00f040 con 0x7f681c0039c0 2023-11-28T20:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.844+0000 7f6827fff640 1 -- 172.21.15.45:0/4075202217 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c034440 con 0x7f681c0039c0 2023-11-28T20:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.844+0000 7f6836609640 1 -- 172.21.15.45:0/4075202217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c0039c0 msgr2=0x7f681c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.844+0000 7f6836609640 1 --2- 172.21.15.45:0/4075202217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c0039c0 0x7f681c003de0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f682c009fc0 tx=0x7f681c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.844+0000 7f6836609640 1 -- 172.21.15.45:0/4075202217 shutdown_connections 2023-11-28T20:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.844+0000 7f6836609640 1 --2- 172.21.15.45:0/4075202217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c0039c0 0x7f681c003de0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.844+0000 7f6836609640 1 -- 172.21.15.45:0/4075202217 >> 172.21.15.45:0/4075202217 conn(0x7f6830070f40 msgr2=0x7f6830071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.845+0000 7f6836609640 1 -- 172.21.15.45:0/4075202217 shutdown_connections 2023-11-28T20:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.845+0000 7f6836609640 1 -- 172.21.15.45:0/4075202217 wait complete. 2023-11-28T20:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.846+0000 7f6836609640 1 Processor -- start 2023-11-28T20:52:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.847+0000 7f6836609640 1 -- start start 2023-11-28T20:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6836609640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f681c0039c0 0x7f6830157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6836609640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68300c8720 0x7f68300c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6836609640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68301f32d0 0x7f68301f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6836609640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68300c90e0 con 0x7f68300c8720 2023-11-28T20:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6836609640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6830157980 con 0x7f68301f32d0 2023-11-28T20:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6836609640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68300c7280 con 0x7f681c0039c0 2023-11-28T20:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6834e06640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68300c8720 0x7f68300c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6834e06640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68300c8720 0x7f68300c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52702/0 (socket says 172.21.15.45:52702) 2023-11-28T20:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6834e06640 1 -- 172.21.15.45:0/255410647 learned_addr learned my addr 172.21.15.45:0/255410647 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6835607640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f681c0039c0 0x7f6830157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.848+0000 7f6835e08640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68301f32d0 0x7f68301f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6834e06640 1 -- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f681c0039c0 msgr2=0x7f6830157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6834e06640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f681c0039c0 0x7f6830157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6834e06640 1 -- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68301f32d0 msgr2=0x7f68301f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6834e06640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68301f32d0 0x7f68301f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6834e06640 1 -- 172.21.15.45:0/255410647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68300c7560 con 0x7f68300c8720 2023-11-28T20:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6835607640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f681c0039c0 0x7f6830157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6835e08640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68301f32d0 0x7f68301f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6834e06640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68300c8720 0x7f68300c8ba0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f682000c730 tx=0x7f682000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6825ffb640 1 -- 172.21.15.45:0/255410647 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68200072d0 con 0x7f68300c8720 2023-11-28T20:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6825ffb640 1 -- 172.21.15.45:0/255410647 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6820007430 con 0x7f68300c8720 2023-11-28T20:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6825ffb640 1 -- 172.21.15.45:0/255410647 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68200047e0 con 0x7f68300c8720 2023-11-28T20:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.849+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68300c7840 con 0x7f68300c8720 2023-11-28T20:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.850+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68300c7e00 con 0x7f68300c8720 2023-11-28T20:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.851+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68300c8020 con 0x7f68300c8720 2023-11-28T20:52:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.852+0000 7f6825ffb640 1 -- 172.21.15.45:0/255410647 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 47) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f6820004b50 con 0x7f68300c8720 2023-11-28T20:52:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.853+0000 7f6825ffb640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f681406afb0 0x7f681406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.853+0000 7f6825ffb640 1 -- 172.21.15.45:0/255410647 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f6820088fa0 con 0x7f68300c8720 2023-11-28T20:52:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.853+0000 7f6835607640 1 -- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f681406afb0 msgr2=0x7f681406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/4159844301 2023-11-28T20:52:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.854+0000 7f6835607640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f681406afb0 0x7f681406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:52:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:03.858+0000 7f6825ffb640 1 -- 172.21.15.45:0/255410647 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68200bbe90 con 0x7f68300c8720 2023-11-28T20:52:04.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.053+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f6830071b30 con 0x7f68300c8720 2023-11-28T20:52:04.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.055+0000 7f6835607640 1 -- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f681406afb0 msgr2=0x7f681406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/4159844301 2023-11-28T20:52:04.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.055+0000 7f6835607640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f681406afb0 0x7f681406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:52:04.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.058+0000 7f6825ffb640 1 -- 172.21.15.45:0/255410647 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6820018070 con 0x7f68300c8720 2023-11-28T20:52:04.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f681406afb0 msgr2=0x7f681406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:52:04.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f681406afb0 0x7f681406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68300c8720 msgr2=0x7f68300c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68300c8720 0x7f68300c8ba0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f682000c730 tx=0x7f682000cbf0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 shutdown_connections 2023-11-28T20:52:04.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f681406afb0 0x7f681406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68301f32d0 0x7f68301f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68300c8720 0x7f68300c8ba0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 --2- 172.21.15.45:0/255410647 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f681c0039c0 0x7f6830157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 >> 172.21.15.45:0/255410647 conn(0x7f6830070f40 msgr2=0x7f68300b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:04.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 shutdown_connections 2023-11-28T20:52:04.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.061+0000 7f6836609640 1 -- 172.21.15.45:0/255410647 wait complete. 2023-11-28T20:52:04.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T20:52:04.149 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:04.148+0000 7fae53a4c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:52:04.154 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:04.152+0000 7f1e1902b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:52:04.202 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:04.201+0000 7fae53a4c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:52:04.210 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:04.209+0000 7f1e1902b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:52:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.295+0000 7f93545e3640 1 Processor -- start 2023-11-28T20:52:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.295+0000 7f93545e3640 1 -- start start 2023-11-28T20:52:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.296+0000 7f93545e3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f934c0cda70 0x7f934c0cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.296+0000 7f93545e3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c0d0070 0x7f934c0d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.297+0000 7f93545e3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934c0d04e0 con 0x7f934c15d420 2023-11-28T20:52:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.297+0000 7f93545e3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934c0cdf00 con 0x7f934c0cda70 2023-11-28T20:52:04.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.297+0000 7f93545e3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0ce1e0 con 0x7f934c0d0070 2023-11-28T20:52:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.297+0000 7f9351b7e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f934c0cda70 0x7f934c0cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.297+0000 7f9351b7e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f934c0cda70 0x7f934c0cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:52584/0 (socket says 172.21.15.45:52584) 2023-11-28T20:52:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.297+0000 7f9351b7e640 1 -- 172.21.15.45:0/2300760716 learned_addr learned my addr 172.21.15.45:0/2300760716 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.297+0000 7f9352b80640 1 --2- 172.21.15.45:0/2300760716 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c0d0070 0x7f934c0d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.298+0000 7f9351b7e640 1 -- 172.21.15.45:0/2300760716 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c0d0070 msgr2=0x7f934c0d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.298+0000 7f9351b7e640 1 --2- 172.21.15.45:0/2300760716 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c0d0070 0x7f934c0d1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.298+0000 7f9351b7e640 1 -- 172.21.15.45:0/2300760716 >> v1:172.21.15.190:6789/0 conn(0x7f934c15d420 legacy=0x7f934c161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.298+0000 7f9351b7e640 1 -- 172.21.15.45:0/2300760716 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c0ce4c0 con 0x7f934c0cda70 2023-11-28T20:52:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.299+0000 7f9351b7e640 1 --2- 172.21.15.45:0/2300760716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0cda70 0x7f934c0cfb30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f934c08b880 tx=0x7f933c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf272892eb23a10 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.299+0000 7f935137d640 1 -- 172.21.15.45:0/2300760716 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c031830 con 0x7f934c0cda70 2023-11-28T20:52:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.299+0000 7f935137d640 1 -- 172.21.15.45:0/2300760716 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933c031990 con 0x7f934c0cda70 2023-11-28T20:52:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.299+0000 7f935137d640 1 -- 172.21.15.45:0/2300760716 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c031d20 con 0x7f934c0cda70 2023-11-28T20:52:04.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.300+0000 7f93545e3640 1 -- 172.21.15.45:0/2300760716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0cda70 msgr2=0x7f934c0cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.300+0000 7f93545e3640 1 --2- 172.21.15.45:0/2300760716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0cda70 0x7f934c0cfb30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f934c08b880 tx=0x7f933c02b8d0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.300+0000 7f93545e3640 1 -- 172.21.15.45:0/2300760716 shutdown_connections 2023-11-28T20:52:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.300+0000 7f93545e3640 1 --2- 172.21.15.45:0/2300760716 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c0d0070 0x7f934c0d1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.300+0000 7f93545e3640 1 --2- 172.21.15.45:0/2300760716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0cda70 0x7f934c0cfb30 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.300+0000 7f93545e3640 1 -- 172.21.15.45:0/2300760716 >> 172.21.15.45:0/2300760716 conn(0x7f934c070b20 msgr2=0x7f934c070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.300+0000 7f93545e3640 1 -- 172.21.15.45:0/2300760716 shutdown_connections 2023-11-28T20:52:04.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.300+0000 7f93545e3640 1 -- 172.21.15.45:0/2300760716 wait complete. 2023-11-28T20:52:04.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.302+0000 7f93545e3640 1 Processor -- start 2023-11-28T20:52:04.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.302+0000 7f93545e3640 1 -- start start 2023-11-28T20:52:04.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f93545e3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0d0070 0x7f934c0b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f93545e3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c15d420 0x7f934c0b7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f93545e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f934c154930 0x7f934c154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f93545e3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c155300 con 0x7f934c154930 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f93545e3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0cad40 con 0x7f934c0d0070 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f93545e3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c160af0 con 0x7f934c15d420 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f9352b80640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f934c154930 0x7f934c154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f9352b80640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f934c154930 0x7f934c154dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55298/0 (socket says 172.21.15.45:55298) 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f9352b80640 1 -- 172.21.15.45:0/1755821471 learned_addr learned my addr 172.21.15.45:0/1755821471 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.304+0000 7f9351b7e640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c15d420 0x7f934c0b7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f9352b80640 1 -- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c15d420 msgr2=0x7f934c0b7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f935237f640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0d0070 0x7f934c0b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f9352b80640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c15d420 0x7f934c0b7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f9352b80640 1 -- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0d0070 msgr2=0x7f934c0b29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f9352b80640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0d0070 0x7f934c0b29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f9352b80640 1 -- 172.21.15.45:0/1755821471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0094d0 con 0x7f934c154930 2023-11-28T20:52:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f935237f640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0d0070 0x7f934c0b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f9352b80640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f934c154930 0x7f934c154dc0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f934400efc0 tx=0x7f934400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f9342ffd640 1 -- 172.21.15.45:0/1755821471 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344014480 con 0x7f934c154930 2023-11-28T20:52:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f9342ffd640 1 -- 172.21.15.45:0/1755821471 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934400f040 con 0x7f934c154930 2023-11-28T20:52:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934c0ca410 con 0x7f934c154930 2023-11-28T20:52:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f9342ffd640 1 -- 172.21.15.45:0/1755821471 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344014820 con 0x7f934c154930 2023-11-28T20:52:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.305+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934c0ca9d0 con 0x7f934c154930 2023-11-28T20:52:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.307+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934c0733d0 con 0x7f934c154930 2023-11-28T20:52:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.308+0000 7f9342ffd640 1 -- 172.21.15.45:0/1755821471 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 47) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f9344014980 con 0x7f934c154930 2023-11-28T20:52:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.309+0000 7f9342ffd640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f933006afb0 0x7f933006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.309+0000 7f9342ffd640 1 -- 172.21.15.45:0/1755821471 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f934408a6d0 con 0x7f934c154930 2023-11-28T20:52:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.309+0000 7f935237f640 1 -- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f933006afb0 msgr2=0x7f933006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/4159844301 2023-11-28T20:52:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.309+0000 7f935237f640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f933006afb0 0x7f933006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:52:04.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.317+0000 7f9342ffd640 1 -- 172.21.15.45:0/1755821471 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93440553e0 con 0x7f934c154930 2023-11-28T20:52:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.510+0000 7f935237f640 1 -- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f933006afb0 msgr2=0x7f933006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/4159844301 2023-11-28T20:52:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.510+0000 7f935237f640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f933006afb0 0x7f933006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:52:04.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.518+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f934c0cae80 con 0x7f934c154930 2023-11-28T20:52:04.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.525+0000 7f9342ffd640 1 -- 172.21.15.45:0/1755821471 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9344058a00 con 0x7f934c154930 2023-11-28T20:52:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f933006afb0 msgr2=0x7f933006d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:52:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f933006afb0 0x7f933006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f934c154930 msgr2=0x7f934c154dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f934c154930 0x7f934c154dc0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f934400efc0 tx=0x7f934400c490 comp rx=0 tx=0).stop 2023-11-28T20:52:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 shutdown_connections 2023-11-28T20:52:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7f933006afb0 0x7f933006d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f934c154930 0x7f934c154dc0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f934c15d420 0x7f934c0b7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 --2- 172.21.15.45:0/1755821471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f934c0d0070 0x7f934c0b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 >> 172.21.15.45:0/1755821471 conn(0x7f934c070b20 msgr2=0x7f934c0d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 shutdown_connections 2023-11-28T20:52:04.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.528+0000 7f93545e3640 1 -- 172.21.15.45:0/1755821471 wait complete. 2023-11-28T20:52:04.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2023-11-28T20:52:04.647 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:04.645+0000 7fae53a4c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:52:04.694 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:04.692+0000 7f1e1902b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:52:04.698 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:04.697+0000 7fae53a4c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:52:04.748 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:04.746+0000 7fae53a4c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:52:04.750 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:04.748+0000 7f1e1902b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:52:04.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.753+0000 7fbed8948640 1 Processor -- start 2023-11-28T20:52:04.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.753+0000 7fbed8948640 1 -- start start 2023-11-28T20:52:04.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.755+0000 7fbed8948640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbed40d24f0 0x7fbed40d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.755+0000 7fbed8948640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed4160e80 con 0x7fbed40ce0e0 2023-11-28T20:52:04.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.755+0000 7fbed8948640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed40d1220 con 0x7fbed4160aa0 2023-11-28T20:52:04.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.755+0000 7fbed8948640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed40d2e70 con 0x7fbed40d24f0 2023-11-28T20:52:04.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.755+0000 7fbed2d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbed40ce0e0 0x7fbed40ce570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34776/0 (socket says 172.21.15.45:34776) 2023-11-28T20:52:04.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.755+0000 7fbed2d9d640 1 -- 172.21.15.45:0/3830722627 learned_addr learned my addr 172.21.15.45:0/3830722627 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:52:04.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.755+0000 7fbec9d9b640 1 --2- 172.21.15.45:0/3830722627 >> v2:172.21.15.190:3300/0 conn(0x7fbed40d24f0 0x7fbed40d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.755+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203529550 0 0) 0x7fbed4160e80 con 0x7fbed40ce0e0 2023-11-28T20:52:04.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbec0003520 con 0x7fbed40ce0e0 2023-11-28T20:52:04.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2876647971 0 0) 0x7fbec0003520 con 0x7fbed40ce0e0 2023-11-28T20:52:04.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 >> v2:172.21.15.190:3300/0 conn(0x7fbed40d24f0 msgr2=0x7fbed40d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:04.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 --2- 172.21.15.45:0/3830722627 >> v2:172.21.15.190:3300/0 conn(0x7fbed40d24f0 0x7fbed40d2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 >> v1:172.21.15.190:6789/0 conn(0x7fbed4160aa0 legacy=0x7fbed40b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed40d1470 con 0x7fbed40ce0e0 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fbecc001670 con 0x7fbed40ce0e0 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 >> v1:172.21.15.45:6789/0 conn(0x7fbed40ce0e0 legacy=0x7fbed40ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbec00039c0 0x7fbec0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec00047c0 0x7fbec0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec0005180 0x7fbec000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40d2e70 con 0x7fbec00047c0 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec000c060 con 0x7fbec0005180 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec000c220 con 0x7fbec00039c0 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.756+0000 7fbec9d9b640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec00047c0 0x7fbec0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.757+0000 7fbed2d9d640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec0005180 0x7fbec000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.757+0000 7fbed259c640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbec00039c0 0x7fbec0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.757+0000 7fbec9d9b640 1 -- 172.21.15.45:0/3830722627 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbec00039c0 msgr2=0x7fbec0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:04.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.757+0000 7fbec9d9b640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbec00039c0 0x7fbec0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.757+0000 7fbec9d9b640 1 -- 172.21.15.45:0/3830722627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec0005180 msgr2=0x7fbec000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:04.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.757+0000 7fbec9d9b640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec0005180 0x7fbec000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.757+0000 7fbec9d9b640 1 -- 172.21.15.45:0/3830722627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec0003520 con 0x7fbec00047c0 2023-11-28T20:52:04.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.757+0000 7fbed2d9d640 1 -- 172.21.15.45:0/3830722627 reap_dead start 2023-11-28T20:52:04.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.758+0000 7fbec9d9b640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec00047c0 0x7fbec0004c40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009ac0 tx=0x7fbec00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabb42412d0c2cb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:04.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.758+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc02c6b0 con 0x7fbec00047c0 2023-11-28T20:52:04.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.758+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbebc02c810 con 0x7fbec00047c0 2023-11-28T20:52:04.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.758+0000 7fbed1d9b640 1 -- 172.21.15.45:0/3830722627 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc02cba0 con 0x7fbec00047c0 2023-11-28T20:52:04.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.759+0000 7fbed8948640 1 -- 172.21.15.45:0/3830722627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec00047c0 msgr2=0x7fbec0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.759+0000 7fbed8948640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec00047c0 0x7fbec0004c40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009ac0 tx=0x7fbec00057f0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.759+0000 7fbed8948640 1 -- 172.21.15.45:0/3830722627 shutdown_connections 2023-11-28T20:52:04.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.759+0000 7fbed8948640 1 --2- 172.21.15.45:0/3830722627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbec00047c0 0x7fbec0004c40 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.759+0000 7fbed8948640 1 -- 172.21.15.45:0/3830722627 >> 172.21.15.45:0/3830722627 conn(0x7fbed4070b20 msgr2=0x7fbed4070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:04.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.759+0000 7fbed8948640 1 -- 172.21.15.45:0/3830722627 shutdown_connections 2023-11-28T20:52:04.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.759+0000 7fbed8948640 1 -- 172.21.15.45:0/3830722627 wait complete. 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.761+0000 7fbed8948640 1 Processor -- start 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.761+0000 7fbed8948640 1 -- start start 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbed8948640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec00047c0 0x7fbed4153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbed8948640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed4152640 0x7fbed4149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbed8948640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbed40c96a0 0x7fbed40c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbed8948640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40c9280 con 0x7fbed4152640 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbed8948640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40c8600 con 0x7fbec00047c0 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbed8948640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40c88e0 con 0x7fbed40c96a0 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbec9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed4152640 0x7fbed4149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbec9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed4152640 0x7fbed4149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55310/0 (socket says 172.21.15.45:55310) 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbec9d9b640 1 -- 172.21.15.45:0/2798017543 learned_addr learned my addr 172.21.15.45:0/2798017543 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbed2d9d640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbed40c96a0 0x7fbed40c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.762+0000 7fbed259c640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec00047c0 0x7fbed4153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbec9d9b640 1 -- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbed40c96a0 msgr2=0x7fbed40c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbec9d9b640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbed40c96a0 0x7fbed40c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbec9d9b640 1 -- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec00047c0 msgr2=0x7fbed4153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbec9d9b640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec00047c0 0x7fbed4153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbec9d9b640 1 -- 172.21.15.45:0/2798017543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed40c8bc0 con 0x7fbed4152640 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbed259c640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec00047c0 0x7fbed4153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:04.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbec9d9b640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed4152640 0x7fbed4149580 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fbebc02c250 tx=0x7fbebc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:04.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc003810 con 0x7fbed4152640 2023-11-28T20:52:04.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbebc003970 con 0x7fbed4152640 2023-11-28T20:52:04.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc003d00 con 0x7fbed4152640 2023-11-28T20:52:04.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed40c8ea0 con 0x7fbed4152640 2023-11-28T20:52:04.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.763+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed4154140 con 0x7fbed4152640 2023-11-28T20:52:04.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.764+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed40733d0 con 0x7fbed4152640 2023-11-28T20:52:04.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.765+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 47) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fbebc034440 con 0x7fbed4152640 2023-11-28T20:52:04.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.766+0000 7fbecb7fe640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 0x7fbeb406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:04.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.766+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6072+0+0 (secure 0 0 0) 0x7fbebc0abc80 con 0x7fbed4152640 2023-11-28T20:52:04.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.767+0000 7fbed259c640 1 -- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 msgr2=0x7fbeb406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/4159844301 2023-11-28T20:52:04.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.767+0000 7fbed259c640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 0x7fbeb406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:52:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.775+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbebc076b50 con 0x7fbed4152640 2023-11-28T20:52:04.809 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:04.806+0000 7f1e1902b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:52:04.809 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:04.808+0000 7fae53a4c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:52:04.857 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:04.856+0000 7fae53a4c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:52:04.873 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:04.871+0000 7f1e1902b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:52:04.924 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:04.922+0000 7f1e1902b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:52:04.967 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:04.966+0000 7fae53a4c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:52:04.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.968+0000 7fbed259c640 1 -- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 msgr2=0x7fbeb406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/4159844301 2023-11-28T20:52:04.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.968+0000 7fbed259c640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 0x7fbeb406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:52:04.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:04.993+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 --> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbed40d1d10 con 0x7fbeb406afb0 2023-11-28T20:52:05.031 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:05.028+0000 7fae53a4c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:52:05.042 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:05.040+0000 7f1e1902b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:52:05.079 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:05.078+0000 7fae53a4c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:52:05.109 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:05.106+0000 7f1e1902b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:52:05.136 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:05.135+0000 7fae53a4c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:52:05.160 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:05.158+0000 7f1e1902b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:52:05.186 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:05.185+0000 7fae53a4c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:52:05.219 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:05.217+0000 7f1e1902b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:52:05.271 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:05.269+0000 7f1e1902b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:52:05.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:05.369+0000 7fbed259c640 1 -- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 msgr2=0x7fbeb406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/4159844301 2023-11-28T20:52:05.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:05.369+0000 7fbed259c640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 0x7fbeb406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T20:52:05.568 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:05.567+0000 7fae53a4c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:52:05.673 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:05.670+0000 7f1e1902b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:52:05.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:05.724+0000 7fae53a4c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:52:05.832 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:05.831+0000 7fae53a4c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:52:05.840 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:05.838+0000 7f1e1902b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:52:05.882 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:05.880+0000 7fae53a4c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:52:05.953 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:05.950+0000 7f1e1902b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:52:05.988 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:05.987+0000 7fae53a4c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:52:06.005 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:06.003+0000 7f1e1902b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:52:06.095 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:06.094+0000 7fae53a4c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:52:06.118 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:06.116+0000 7f1e1902b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:52:06.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:06.170+0000 7fbed259c640 1 -- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 msgr2=0x7fbeb406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/4159844301 2023-11-28T20:52:06.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:06.170+0000 7fbed259c640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 0x7fbeb406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T20:52:06.230 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:06.228+0000 7f1e1902b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:52:06.250 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:06.249+0000 7fae53a4c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:52:06.298 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:06.297+0000 7fae53a4c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:52:06.395 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:06.392+0000 7f1e1902b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:52:06.417 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:06.416+0000 7fae53a4c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:52:06.446 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:06.444+0000 7f1e1902b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:52:06.571 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:06.569+0000 7f1e1902b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:52:07.242 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.240+0000 7fae53a4c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:52:07.458 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:07.456+0000 7f1e1902b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:52:07.535 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.533+0000 7fae53a4c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:52:07.594 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.590+0000 7fae53a4c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:52:07.704 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.703+0000 7fae53a4c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:52:07.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:07.719+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 48) v1 ==== 88959+0+0 (secure 0 0 0) 0x7fbebc076f20 con 0x7fbed4152640 2023-11-28T20:52:07.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:07.720+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 msgr2=0x7fbeb406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:52:07.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:07.720+0000 7fbecb7fe640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/4159844301,v1:172.21.15.190:6833/4159844301] conn(0x7fbeb406afb0 0x7fbeb406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:07.776 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:07.774+0000 7f1e1902b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:52:07.839 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:07.837+0000 7f1e1902b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:52:07.874 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.873+0000 7fadd354d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.874 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.873+0000 7fadd354d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.874 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.873+0000 7fadd354d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.875 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.873+0000 7fadd354d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.875 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.873+0000 7fadd354d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.878 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.877+0000 7fadd0547640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.878 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.877+0000 7fadd0547640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.878 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.877+0000 7fadd0547640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.878 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.877+0000 7fadd0547640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.879 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:52:07.877+0000 7fadd0547640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:52:07.963 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:52:07.961+0000 7f1e1902b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:52:08.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.732+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 49) v1 ==== 89086+0+0 (secure 0 0 0) 0x7fbebc08e4b0 con 0x7fbed4152640 2023-11-28T20:52:08.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.733+0000 7fbecb7fe640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fbeb4035850 0x7fbeb4075440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:08.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.733+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 --> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbeb4035c30 con 0x7fbeb4035850 2023-11-28T20:52:08.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.736+0000 7fbed259c640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fbeb4035850 0x7fbeb4075440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:08.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.740+0000 7fbed259c640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fbeb4035850 0x7fbeb4075440 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbed4069a70 tx=0x7fbec4009040 comp rx=0 tx=0).ready entity=mgr.6038 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:08.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.889+0000 7fbecb7fe640 1 -- 172.21.15.45:0/2798017543 <== mgr.6038 v2:172.21.15.190:6832/1902260958 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fbeb4035c30 con 0x7fbeb4035850 2023-11-28T20:52:08.892 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /bad-path 2023-11-28T20:52:08.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.893+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fbeb4035850 msgr2=0x7fbeb4075440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.893+0000 7fbed8948640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fbeb4035850 0x7fbeb4075440 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbed4069a70 tx=0x7fbec4009040 comp rx=0 tx=0).stop 2023-11-28T20:52:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.893+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed4152640 msgr2=0x7fbed4149580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.893+0000 7fbed8948640 1 --2- 172.21.15.45:0/2798017543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed4152640 0x7fbed4149580 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fbebc02c250 tx=0x7fbebc004730 comp rx=0 tx=0).stop 2023-11-28T20:52:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.893+0000 7fbed2d9d640 1 -- 172.21.15.45:0/2798017543 reap_dead start 2023-11-28T20:52:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.893+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 shutdown_connections 2023-11-28T20:52:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.893+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 >> 172.21.15.45:0/2798017543 conn(0x7fbed4070b20 msgr2=0x7fbed40cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:08.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.894+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 shutdown_connections 2023-11-28T20:52:08.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:08.894+0000 7fbed8948640 1 -- 172.21.15.45:0/2798017543 wait complete. 2023-11-28T20:52:08.917 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-11-28T20:52:08.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-11-28T20:52:09.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.139+0000 7ff8d33c9640 1 Processor -- start 2023-11-28T20:52:09.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.140+0000 7ff8d33c9640 1 -- start start 2023-11-28T20:52:09.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.141+0000 7ff8d33c9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff8cc0b2600 0x7ff8cc0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:09.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.141+0000 7ff8d33c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:09.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.142+0000 7ff8d33c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc0cebf0 con 0x7ff8cc0d2010 2023-11-28T20:52:09.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.142+0000 7ff8d33c9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8cc0d2850 con 0x7ff8cc0b2600 2023-11-28T20:52:09.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.142+0000 7ff8d33c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc0d2b30 con 0x7ff8cc0ce7b0 2023-11-28T20:52:09.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.142+0000 7ff8d1165640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff8cc0b2600 0x7ff8cc0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:09.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.142+0000 7ff8d1966640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff8cc0d2010 0x7ff8cc0b4c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34782/0 (socket says 172.21.15.45:34782) 2023-11-28T20:52:09.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.142+0000 7ff8d1966640 1 -- 172.21.15.45:0/1482150843 learned_addr learned my addr 172.21.15.45:0/1482150843 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:52:09.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.142+0000 7ff8d0964640 1 --2- 172.21.15.45:0/1482150843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:09.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.142+0000 7ff8d1165640 1 -- 172.21.15.45:0/1482150843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 msgr2=0x7ff8cc0d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:52:09.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.143+0000 7ff8cbfff640 1 -- 172.21.15.45:0/1482150843 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198552165 0 0) 0x7ff8cc0cebf0 con 0x7ff8cc0d2010 2023-11-28T20:52:09.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.143+0000 7ff8d1165640 1 --2- 172.21.15.45:0/1482150843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.143+0000 7ff8d1165640 1 -- 172.21.15.45:0/1482150843 >> v1:172.21.15.45:6789/0 conn(0x7ff8cc0d2010 legacy=0x7ff8cc0b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.143+0000 7ff8d1165640 1 -- 172.21.15.45:0/1482150843 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc0b6bb0 con 0x7ff8cc0b2600 2023-11-28T20:52:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.143+0000 7ff8d0964640 1 --2- 172.21.15.45:0/1482150843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.144+0000 7ff8d1165640 1 --2- 172.21.15.45:0/1482150843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0b2600 0x7ff8cc0ce270 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009d90 tx=0x7ff8bc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4267ce6ed0a58227 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.144+0000 7ff8cbfff640 1 -- 172.21.15.45:0/1482150843 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc02cc50 con 0x7ff8cc0b2600 2023-11-28T20:52:09.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.144+0000 7ff8cbfff640 1 -- 172.21.15.45:0/1482150843 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8bc02cdb0 con 0x7ff8cc0b2600 2023-11-28T20:52:09.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.144+0000 7ff8cbfff640 1 -- 172.21.15.45:0/1482150843 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc034660 con 0x7ff8cc0b2600 2023-11-28T20:52:09.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.145+0000 7ff8d33c9640 1 -- 172.21.15.45:0/1482150843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0b2600 msgr2=0x7ff8cc0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:09.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.145+0000 7ff8d33c9640 1 --2- 172.21.15.45:0/1482150843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0b2600 0x7ff8cc0ce270 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009d90 tx=0x7ff8bc02bea0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.145+0000 7ff8d33c9640 1 -- 172.21.15.45:0/1482150843 shutdown_connections 2023-11-28T20:52:09.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.145+0000 7ff8d33c9640 1 --2- 172.21.15.45:0/1482150843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.145+0000 7ff8d33c9640 1 --2- 172.21.15.45:0/1482150843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0b2600 0x7ff8cc0ce270 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.145+0000 7ff8d33c9640 1 -- 172.21.15.45:0/1482150843 >> 172.21.15.45:0/1482150843 conn(0x7ff8cc070b20 msgr2=0x7ff8cc070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:09.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.145+0000 7ff8d33c9640 1 -- 172.21.15.45:0/1482150843 shutdown_connections 2023-11-28T20:52:09.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.145+0000 7ff8d33c9640 1 -- 172.21.15.45:0/1482150843 wait complete. 2023-11-28T20:52:09.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.147+0000 7ff8d33c9640 1 Processor -- start 2023-11-28T20:52:09.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.147+0000 7ff8d33c9640 1 -- start start 2023-11-28T20:52:09.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.148+0000 7ff8d33c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8cc0b2600 0x7ff8cc1e6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:09.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.148+0000 7ff8d33c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:09.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.148+0000 7ff8d33c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0d2010 0x7ff8cc1e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:09.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.148+0000 7ff8d33c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc1e8850 con 0x7ff8cc0d2010 2023-11-28T20:52:09.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.148+0000 7ff8d33c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc1e8570 con 0x7ff8cc0b2600 2023-11-28T20:52:09.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.148+0000 7ff8d33c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc1e61d0 con 0x7ff8cc0ce7b0 2023-11-28T20:52:09.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.148+0000 7ff8d1966640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0d2010 0x7ff8cc1e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:09.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.148+0000 7ff8d1966640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0d2010 0x7ff8cc1e7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55316/0 (socket says 172.21.15.45:55316) 2023-11-28T20:52:09.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.148+0000 7ff8d1966640 1 -- 172.21.15.45:0/2098988866 learned_addr learned my addr 172.21.15.45:0/2098988866 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:52:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d0964640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d1165640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8cc0b2600 0x7ff8cc1e6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d1966640 1 -- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 msgr2=0x7ff8cc0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d1966640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d1966640 1 -- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8cc0b2600 msgr2=0x7ff8cc1e6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d1966640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8cc0b2600 0x7ff8cc1e6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d1966640 1 -- 172.21.15.45:0/2098988866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc009a70 con 0x7ff8cc0d2010 2023-11-28T20:52:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d0964640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:52:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d1966640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0d2010 0x7ff8cc1e7a00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4001eb0 tx=0x7ff8c400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:09.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8c9ffb640 1 -- 172.21.15.45:0/2098988866 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40074e0 con 0x7ff8cc0d2010 2023-11-28T20:52:09.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8c9ffb640 1 -- 172.21.15.45:0/2098988866 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c4007640 con 0x7ff8cc0d2010 2023-11-28T20:52:09.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8c9ffb640 1 -- 172.21.15.45:0/2098988866 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4004a00 con 0x7ff8cc0d2010 2023-11-28T20:52:09.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.149+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc0d2620 con 0x7ff8cc0d2010 2023-11-28T20:52:09.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.150+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc1e6790 con 0x7ff8cc0d2010 2023-11-28T20:52:09.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.151+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8cc1e6030 con 0x7ff8cc0d2010 2023-11-28T20:52:09.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.152+0000 7ff8c9ffb640 1 -- 172.21.15.45:0/2098988866 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 49) v1 ==== 89086+0+0 (secure 0 0 0) 0x7ff8c4004b60 con 0x7ff8cc0d2010 2023-11-28T20:52:09.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.153+0000 7ff8c9ffb640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff8b806ace0 0x7ff8b806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:52:09.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.153+0000 7ff8c9ffb640 1 -- 172.21.15.45:0/2098988866 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6416+0+0 (secure 0 0 0) 0x7ff8c4089400 con 0x7ff8cc0d2010 2023-11-28T20:52:09.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.153+0000 7ff8d1165640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff8b806ace0 0x7ff8b806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:52:09.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.154+0000 7ff8d1165640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff8b806ace0 0x7ff8b806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc036000 tx=0x7ff8bc030000 comp rx=0 tx=0).ready entity=mgr.6038 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:52:09.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.157+0000 7ff8c9ffb640 1 -- 172.21.15.45:0/2098988866 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8cc1e6030 con 0x7ff8cc0d2010 2023-11-28T20:52:09.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.373+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 --> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff8cc0d0a30 con 0x7ff8b806ace0 2023-11-28T20:52:09.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.390+0000 7ff8c9ffb640 1 -- 172.21.15.45:0/2098988866 <== mgr.6038 v2:172.21.15.190:6832/1902260958 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+287 (secure 0 0 0) 0x7ff8cc0d0a30 con 0x7ff8b806ace0 2023-11-28T20:52:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:52:09.392 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:52:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-11-28T20:52:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.392+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff8b806ace0 msgr2=0x7ff8b806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.392+0000 7ff8d33c9640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff8b806ace0 0x7ff8b806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc036000 tx=0x7ff8bc030000 comp rx=0 tx=0).stop 2023-11-28T20:52:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.392+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0d2010 msgr2=0x7ff8cc1e7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:52:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.392+0000 7ff8d33c9640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0d2010 0x7ff8cc1e7a00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4001eb0 tx=0x7ff8c400c870 comp rx=0 tx=0).stop 2023-11-28T20:52:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.393+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 shutdown_connections 2023-11-28T20:52:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.393+0000 7ff8d33c9640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff8b806ace0 0x7ff8b806d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.393+0000 7ff8d33c9640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8cc0d2010 0x7ff8cc1e7a00 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.393+0000 7ff8d33c9640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8cc0ce7b0 0x7ff8cc0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.393+0000 7ff8d33c9640 1 --2- 172.21.15.45:0/2098988866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8cc0b2600 0x7ff8cc1e6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:52:09.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.393+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 >> 172.21.15.45:0/2098988866 conn(0x7ff8cc070b20 msgr2=0x7ff8cc0b4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:52:09.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.393+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 shutdown_connections 2023-11-28T20:52:09.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:52:09.393+0000 7ff8d33c9640 1 -- 172.21.15.45:0/2098988866 wait complete. 2023-11-28T20:52:09.413 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:52:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-11-28T20:52:20.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:52:20.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:52:50.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:52:50.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:53:00.913 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:00.911+0000 7fadc2d2c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:01.418 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:00.911+0000 7fadc2d2c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:01.419 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:00.911+0000 7fadc2d2c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:01.419 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:00.911+0000 7fadc2d2c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:01.419 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:00.911+0000 7fadc2d2c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:09.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-11-28T20:53:09.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.645+0000 7ffae3fff640 1 Processor -- start 2023-11-28T20:53:09.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.646+0000 7ffae3fff640 1 -- start start 2023-11-28T20:53:09.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.647+0000 7ffae3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae415cf10 0x7ffae40d1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:09.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.647+0000 7ffae3fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae40d02b0 con 0x7ffae415d2f0 2023-11-28T20:53:09.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.647+0000 7ffae3fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae40d1720 con 0x7ffae40ce000 2023-11-28T20:53:09.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.647+0000 7ffae3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae40d1a00 con 0x7ffae415cf10 2023-11-28T20:53:09.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.648+0000 7ffae27fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ffae415d2f0 0x7ffae40cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51408/0 (socket says 172.21.15.45:51408) 2023-11-28T20:53:09.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.648+0000 7ffae27fc640 1 -- 172.21.15.45:0/1782000830 learned_addr learned my addr 172.21.15.45:0/1782000830 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:09.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.648+0000 7ffae2ffd640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae415cf10 0x7ffae40d1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:09.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.648+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546259243 0 0) 0x7ffae40d02b0 con 0x7ffae415d2f0 2023-11-28T20:53:09.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.648+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0003520 con 0x7ffae415d2f0 2023-11-28T20:53:09.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.648+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 113804490 0 0) 0x7ffad0003520 con 0x7ffae415d2f0 2023-11-28T20:53:09.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae415cf10 msgr2=0x7ffae40d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:09.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae415cf10 0x7ffae40d1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 >> v1:172.21.15.190:6789/0 conn(0x7ffae40ce000 legacy=0x7ffae40d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:09.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae40d1ce0 con 0x7ffae415d2f0 2023-11-28T20:53:09.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ffacc0017f0 con 0x7ffae415d2f0 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 >> v1:172.21.15.45:6789/0 conn(0x7ffae415d2f0 legacy=0x7ffae40cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae415cf10 0x7ffad00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffad00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffad0004d00 0x7ffad0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae40d1a00 con 0x7ffad0003f20 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad000ba60 con 0x7ffad0004d00 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad000bc20 con 0x7ffae415cf10 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae27fc640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffad00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae37fe640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffad0004d00 0x7ffad0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae27fc640 1 -- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae415cf10 msgr2=0x7ffad00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.649+0000 7ffae2ffd640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae415cf10 0x7ffad00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:09.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.650+0000 7ffae27fc640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae415cf10 0x7ffad00039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.650+0000 7ffae27fc640 1 -- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffad0004d00 msgr2=0x7ffad0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:09.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.650+0000 7ffae27fc640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffad0004d00 0x7ffad0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.650+0000 7ffae27fc640 1 -- 172.21.15.45:0/1782000830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad0003520 con 0x7ffad0003f20 2023-11-28T20:53:09.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.650+0000 7ffae37fe640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffad0004d00 0x7ffad0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:09.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.651+0000 7ffae27fc640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffad00047c0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00c160 tx=0x7ffad00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab2408c2e905437 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:09.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.651+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc007270 con 0x7ffad0003f20 2023-11-28T20:53:09.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.651+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc0073f0 con 0x7ffad0003f20 2023-11-28T20:53:09.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.651+0000 7ffae1ffb640 1 -- 172.21.15.45:0/1782000830 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc02c870 con 0x7ffad0003f20 2023-11-28T20:53:09.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.652+0000 7ffae3fff640 1 -- 172.21.15.45:0/1782000830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 msgr2=0x7ffad00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:09.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.652+0000 7ffae3fff640 1 --2- 172.21.15.45:0/1782000830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffad00047c0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00c160 tx=0x7ffad00057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.652+0000 7ffae37fe640 1 -- 172.21.15.45:0/1782000830 reap_dead start 2023-11-28T20:53:09.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.652+0000 7ffae3fff640 1 -- 172.21.15.45:0/1782000830 shutdown_connections 2023-11-28T20:53:09.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.652+0000 7ffae3fff640 1 -- 172.21.15.45:0/1782000830 >> 172.21.15.45:0/1782000830 conn(0x7ffae4070f40 msgr2=0x7ffae4071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:09.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.652+0000 7ffae3fff640 1 -- 172.21.15.45:0/1782000830 shutdown_connections 2023-11-28T20:53:09.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.652+0000 7ffae3fff640 1 -- 172.21.15.45:0/1782000830 wait complete. 2023-11-28T20:53:09.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.654+0000 7ffae3fff640 1 Processor -- start 2023-11-28T20:53:09.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.654+0000 7ffae3fff640 1 -- start start 2023-11-28T20:53:09.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffae40ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:09.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae40cc5d0 0x7ffae415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:09.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae415f8a0 0x7ffae415fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:09.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae3fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae4160260 con 0x7ffad0003f20 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae3fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae414fda0 con 0x7ffae415f8a0 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae40ca6a0 con 0x7ffae40cc5d0 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffae40ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffae40ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48054/0 (socket says 172.21.15.45:48054) 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae2ffd640 1 -- 172.21.15.45:0/1654281184 learned_addr learned my addr 172.21.15.45:0/1654281184 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae37fe640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae415f8a0 0x7ffae415fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.656+0000 7ffae27fc640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae40cc5d0 0x7ffae415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffae2ffd640 1 -- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae40cc5d0 msgr2=0x7ffae415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffae2ffd640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae40cc5d0 0x7ffae415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffae2ffd640 1 -- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae415f8a0 msgr2=0x7ffae415fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffae2ffd640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae415f8a0 0x7ffae415fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffae2ffd640 1 -- 172.21.15.45:0/1654281184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae415aef0 con 0x7ffad0003f20 2023-11-28T20:53:09.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffae2ffd640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffae40ca160 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7ffad400f380 tx=0x7ffad400f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:09.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffadb7fe640 1 -- 172.21.15.45:0/1654281184 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4017600 con 0x7ffad0003f20 2023-11-28T20:53:09.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffadb7fe640 1 -- 172.21.15.45:0/1654281184 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad4011040 con 0x7ffad0003f20 2023-11-28T20:53:09.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffadb7fe640 1 -- 172.21.15.45:0/1654281184 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad40179c0 con 0x7ffad0003f20 2023-11-28T20:53:09.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.657+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae40cd0c0 con 0x7ffad0003f20 2023-11-28T20:53:09.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.658+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae40ccb00 con 0x7ffad0003f20 2023-11-28T20:53:09.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.659+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae415c5c0 con 0x7ffad0003f20 2023-11-28T20:53:09.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.660+0000 7ffadb7fe640 1 -- 172.21.15.45:0/1654281184 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 52) v1 ==== 89464+0+0 (secure 0 0 0) 0x7ffad4017b20 con 0x7ffad0003f20 2023-11-28T20:53:09.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.661+0000 7ffadb7fe640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ffac406b0d0 0x7ffac406d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:09.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.661+0000 7ffadb7fe640 1 -- 172.21.15.45:0/1654281184 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ffad408a3b0 con 0x7ffad0003f20 2023-11-28T20:53:09.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.661+0000 7ffae27fc640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ffac406b0d0 0x7ffac406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:09.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.663+0000 7ffae27fc640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ffac406b0d0 0x7ffac406d590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffacc034000 tx=0x7ffacc009030 comp rx=0 tx=0).ready entity=mgr.6038 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:09.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.665+0000 7ffadb7fe640 1 -- 172.21.15.45:0/1654281184 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffad4054ca0 con 0x7ffad0003f20 2023-11-28T20:53:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.882+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 --> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffae4160c60 con 0x7ffac406b0d0 2023-11-28T20:53:09.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.905+0000 7ffadb7fe640 1 -- 172.21.15.45:0/1654281184 <== mgr.6038 v2:172.21.15.190:6832/1902260958 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7ffacc007070 con 0x7ffac406b0d0 2023-11-28T20:53:09.908 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:09.908 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:52:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.908+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ffac406b0d0 msgr2=0x7ffac406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.908+0000 7ffae3fff640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ffac406b0d0 0x7ffac406d590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffacc034000 tx=0x7ffacc009030 comp rx=0 tx=0).stop 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.908+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 msgr2=0x7ffae40ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.908+0000 7ffae3fff640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffae40ca160 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7ffad400f380 tx=0x7ffad400f840 comp rx=0 tx=0).stop 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.908+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 shutdown_connections 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.908+0000 7ffae3fff640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ffac406b0d0 0x7ffac406d590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.908+0000 7ffae3fff640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae415f8a0 0x7ffae415fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.908+0000 7ffae3fff640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffae40cc5d0 0x7ffae415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.909+0000 7ffae3fff640 1 --2- 172.21.15.45:0/1654281184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffad0003f20 0x7ffae40ca160 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:09.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.909+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 >> 172.21.15.45:0/1654281184 conn(0x7ffae4070f40 msgr2=0x7ffae40b32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:09.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.909+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 shutdown_connections 2023-11-28T20:53:09.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:09.909+0000 7ffae3fff640 1 -- 172.21.15.45:0/1654281184 wait complete. 2023-11-28T20:53:09.931 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:52:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-11-28T20:53:09.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2023-11-28T20:53:10.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.146+0000 7f7569b54640 1 Processor -- start 2023-11-28T20:53:10.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.146+0000 7f7569b54640 1 -- start start 2023-11-28T20:53:10.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.148+0000 7f7569b54640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f75640ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.148+0000 7f7569b54640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7564071fd0 0x7f75640702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.148+0000 7f7569b54640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640ac270 con 0x7f75640abd70 2023-11-28T20:53:10.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.148+0000 7f7569b54640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756415b830 con 0x7f7564071fd0 2023-11-28T20:53:10.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.148+0000 7f7569b54640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756415bb10 con 0x7f756415b080 2023-11-28T20:53:10.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.149+0000 7f7568b52640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f75640ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.149+0000 7f7568b52640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f75640ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:41816/0 (socket says 172.21.15.45:41816) 2023-11-28T20:53:10.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.149+0000 7f7568b52640 1 -- 172.21.15.45:0/1877518779 learned_addr learned my addr 172.21.15.45:0/1877518779 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:10.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.149+0000 7f7569353640 1 --2- 172.21.15.45:0/1877518779 >> v2:172.21.15.190:3300/0 conn(0x7f7564071fd0 0x7f75640702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.150+0000 7f7568b52640 1 -- 172.21.15.45:0/1877518779 >> v2:172.21.15.190:3300/0 conn(0x7f7564071fd0 msgr2=0x7f75640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:10.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.150+0000 7f7568b52640 1 --2- 172.21.15.45:0/1877518779 >> v2:172.21.15.190:3300/0 conn(0x7f7564071fd0 0x7f75640702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.150+0000 7f7568b52640 1 -- 172.21.15.45:0/1877518779 >> v1:172.21.15.190:6789/0 conn(0x7f75640abd70 legacy=0x7f75640718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.150+0000 7f75637fe640 1 -- 172.21.15.45:0/1877518779 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951018719 0 0) 0x7f75640ac270 con 0x7f75640abd70 2023-11-28T20:53:10.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.150+0000 7f7568b52640 1 -- 172.21.15.45:0/1877518779 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756415bdf0 con 0x7f756415b080 2023-11-28T20:53:10.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.152+0000 7f7568b52640 1 --2- 172.21.15.45:0/1877518779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f75640ae370 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7558009870 tx=0x7f755802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4485f6285bb0ec72 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:10.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.152+0000 7f75637fe640 1 -- 172.21.15.45:0/1877518779 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558030840 con 0x7f756415b080 2023-11-28T20:53:10.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.152+0000 7f75637fe640 1 -- 172.21.15.45:0/1877518779 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75580309a0 con 0x7f756415b080 2023-11-28T20:53:10.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.153+0000 7f75637fe640 1 -- 172.21.15.45:0/1877518779 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558030d30 con 0x7f756415b080 2023-11-28T20:53:10.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.153+0000 7f7569b54640 1 -- 172.21.15.45:0/1877518779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 msgr2=0x7f75640ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.153+0000 7f7569b54640 1 --2- 172.21.15.45:0/1877518779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f75640ae370 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7558009870 tx=0x7f755802b980 comp rx=0 tx=0).stop 2023-11-28T20:53:10.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.153+0000 7f7569b54640 1 -- 172.21.15.45:0/1877518779 shutdown_connections 2023-11-28T20:53:10.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.153+0000 7f7569b54640 1 --2- 172.21.15.45:0/1877518779 >> v2:172.21.15.190:3300/0 conn(0x7f7564071fd0 0x7f75640702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.153+0000 7f7569b54640 1 --2- 172.21.15.45:0/1877518779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f75640ae370 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.153+0000 7f7569b54640 1 -- 172.21.15.45:0/1877518779 >> 172.21.15.45:0/1877518779 conn(0x7f75640cecd0 msgr2=0x7f75640cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:10.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.153+0000 7f7569b54640 1 -- 172.21.15.45:0/1877518779 shutdown_connections 2023-11-28T20:53:10.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.154+0000 7f7569b54640 1 -- 172.21.15.45:0/1877518779 wait complete. 2023-11-28T20:53:10.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.155+0000 7f7569b54640 1 Processor -- start 2023-11-28T20:53:10.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.155+0000 7f7569b54640 1 -- start start 2023-11-28T20:53:10.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.156+0000 7f7569b54640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7564071fd0 0x7f75640b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.156+0000 7f7569b54640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75640abd70 0x7f7564150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.156+0000 7f7569b54640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f7564151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.156+0000 7f7569b54640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75640b35a0 con 0x7f7564071fd0 2023-11-28T20:53:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.156+0000 7f7569b54640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564150e40 con 0x7f75640abd70 2023-11-28T20:53:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.156+0000 7f7569b54640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75640b2990 con 0x7f756415b080 2023-11-28T20:53:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7568b52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7564071fd0 0x7f75640b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7568b52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7564071fd0 0x7f75640b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48066/0 (socket says 172.21.15.45:48066) 2023-11-28T20:53:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7568b52640 1 -- 172.21.15.45:0/3434772445 learned_addr learned my addr 172.21.15.45:0/3434772445 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7569353640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f7564151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7563fff640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75640abd70 0x7f7564150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7568b52640 1 -- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 msgr2=0x7f7564151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7568b52640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f7564151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7568b52640 1 -- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75640abd70 msgr2=0x7f7564150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7568b52640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75640abd70 0x7f7564150900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7568b52640 1 -- 172.21.15.45:0/3434772445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75580094d0 con 0x7f7564071fd0 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7569353640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f7564151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.157+0000 7f7563fff640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75640abd70 0x7f7564150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.158+0000 7f7568b52640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7564071fd0 0x7f75640b3a30 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f7558000c00 tx=0x7f7558004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.158+0000 7f75617fa640 1 -- 172.21.15.45:0/3434772445 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558039a90 con 0x7f7564071fd0 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.158+0000 7f75617fa640 1 -- 172.21.15.45:0/3434772445 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558039bf0 con 0x7f7564071fd0 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.158+0000 7f75617fa640 1 -- 172.21.15.45:0/3434772445 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558043410 con 0x7f7564071fd0 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.158+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75640b2c70 con 0x7f7564071fd0 2023-11-28T20:53:10.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.158+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75640b3230 con 0x7f7564071fd0 2023-11-28T20:53:10.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.159+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756409d0f0 con 0x7f7564071fd0 2023-11-28T20:53:10.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.160+0000 7f75617fa640 1 -- 172.21.15.45:0/3434772445 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 52) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f755802f390 con 0x7f7564071fd0 2023-11-28T20:53:10.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.161+0000 7f75617fa640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f754806b0d0 0x7f754806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.161+0000 7f75617fa640 1 -- 172.21.15.45:0/3434772445 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f75580abff0 con 0x7f7564071fd0 2023-11-28T20:53:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.162+0000 7f7563fff640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f754806b0d0 0x7f754806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.163+0000 7f7563fff640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f754806b0d0 0x7f754806d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75640cdb00 tx=0x7f7550009040 comp rx=0 tx=0).ready entity=mgr.6038 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:10.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.166+0000 7f75617fa640 1 -- 172.21.15.45:0/3434772445 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75580dee90 con 0x7f7564071fd0 2023-11-28T20:53:10.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.384+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 --> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7f75640b33c0 con 0x7f754806b0d0 2023-11-28T20:53:10.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.543+0000 7f75617fa640 1 -- 172.21.15.45:0/3434772445 <== mgr.6038 v2:172.21.15.190:6832/1902260958 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7f75640b33c0 con 0x7f754806b0d0 2023-11-28T20:53:10.545 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /bad-path 2023-11-28T20:53:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.546+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f754806b0d0 msgr2=0x7f754806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.546+0000 7f7569b54640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f754806b0d0 0x7f754806d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75640cdb00 tx=0x7f7550009040 comp rx=0 tx=0).stop 2023-11-28T20:53:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.546+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7564071fd0 msgr2=0x7f75640b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.546+0000 7f7569b54640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7564071fd0 0x7f75640b3a30 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f7558000c00 tx=0x7f7558004730 comp rx=0 tx=0).stop 2023-11-28T20:53:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.547+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 shutdown_connections 2023-11-28T20:53:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.547+0000 7f7569b54640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f754806b0d0 0x7f754806d590 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.547+0000 7f7569b54640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f756415b080 0x7f7564151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.547+0000 7f7569b54640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75640abd70 0x7f7564150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.547+0000 7f7569b54640 1 --2- 172.21.15.45:0/3434772445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7564071fd0 0x7f75640b3a30 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.547+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 >> 172.21.15.45:0/3434772445 conn(0x7f75640cecd0 msgr2=0x7f75640ad870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:10.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.547+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 shutdown_connections 2023-11-28T20:53:10.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.547+0000 7f7569b54640 1 -- 172.21.15.45:0/3434772445 wait complete. 2023-11-28T20:53:10.574 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-11-28T20:53:10.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T20:53:10.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.804+0000 7f196187e640 1 Processor -- start 2023-11-28T20:53:10.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.804+0000 7f196187e640 1 -- start start 2023-11-28T20:53:10.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.805+0000 7f196187e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f195c099a40 0x7f195c15bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.805+0000 7f196187e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f195c15c310 0x7f195c0cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.805+0000 7f196187e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c0cf230 con 0x7f195c0c7890 2023-11-28T20:53:10.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.805+0000 7f196187e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c0c7c70 con 0x7f195c099a40 2023-11-28T20:53:10.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.805+0000 7f196187e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c0cf0a0 con 0x7f195c15c310 2023-11-28T20:53:10.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.805+0000 7f195bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f195c099a40 0x7f195c15bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.805+0000 7f196087c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f195c0c7890 0x7f195c15b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51412/0 (socket says 172.21.15.45:51412) 2023-11-28T20:53:10.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.806+0000 7f196087c640 1 -- 172.21.15.45:0/3415282106 learned_addr learned my addr 172.21.15.45:0/3415282106 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:10.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.806+0000 7f196107d640 1 --2- 172.21.15.45:0/3415282106 >> v2:172.21.15.190:3300/0 conn(0x7f195c15c310 0x7f195c0cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.806+0000 7f195bfff640 1 -- 172.21.15.45:0/3415282106 >> v2:172.21.15.190:3300/0 conn(0x7f195c15c310 msgr2=0x7f195c0cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:10.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.806+0000 7f195bfff640 1 --2- 172.21.15.45:0/3415282106 >> v2:172.21.15.190:3300/0 conn(0x7f195c15c310 0x7f195c0cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.806+0000 7f195bfff640 1 -- 172.21.15.45:0/3415282106 >> v1:172.21.15.45:6789/0 conn(0x7f195c0c7890 legacy=0x7f195c15b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.806+0000 7f195b7fe640 1 -- 172.21.15.45:0/3415282106 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166951811 0 0) 0x7f195c0cf230 con 0x7f195c0c7890 2023-11-28T20:53:10.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.806+0000 7f195bfff640 1 -- 172.21.15.45:0/3415282106 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c0d1d80 con 0x7f195c099a40 2023-11-28T20:53:10.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.807+0000 7f195bfff640 1 --2- 172.21.15.45:0/3415282106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 0x7f195c15bdd0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f195c0723d0 tx=0x7f195002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb61bb810d5db2d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:10.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.807+0000 7f195b7fe640 1 -- 172.21.15.45:0/3415282106 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950031710 con 0x7f195c099a40 2023-11-28T20:53:10.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f195b7fe640 1 -- 172.21.15.45:0/3415282106 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195002f050 con 0x7f195c099a40 2023-11-28T20:53:10.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f195b7fe640 1 -- 172.21.15.45:0/3415282106 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950031a90 con 0x7f195c099a40 2023-11-28T20:53:10.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f196187e640 1 -- 172.21.15.45:0/3415282106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 msgr2=0x7f195c15bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f196187e640 1 --2- 172.21.15.45:0/3415282106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 0x7f195c15bdd0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f195c0723d0 tx=0x7f195002b7b0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f196187e640 1 -- 172.21.15.45:0/3415282106 shutdown_connections 2023-11-28T20:53:10.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f196187e640 1 --2- 172.21.15.45:0/3415282106 >> v2:172.21.15.190:3300/0 conn(0x7f195c15c310 0x7f195c0cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f196187e640 1 --2- 172.21.15.45:0/3415282106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 0x7f195c15bdd0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f196187e640 1 -- 172.21.15.45:0/3415282106 >> 172.21.15.45:0/3415282106 conn(0x7f195c0d0f10 msgr2=0x7f195c0d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:10.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f196187e640 1 -- 172.21.15.45:0/3415282106 shutdown_connections 2023-11-28T20:53:10.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.808+0000 7f196187e640 1 -- 172.21.15.45:0/3415282106 wait complete. 2023-11-28T20:53:10.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.810+0000 7f196187e640 1 Processor -- start 2023-11-28T20:53:10.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.810+0000 7f196187e640 1 -- start start 2023-11-28T20:53:10.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.811+0000 7f196187e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 0x7f195c14aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.811+0000 7f196187e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f195c0c7890 0x7f195c0b6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.811+0000 7f196187e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f195c15c310 0x7f195c0b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.811+0000 7f196187e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c14af70 con 0x7f195c099a40 2023-11-28T20:53:10.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.811+0000 7f196187e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c14c160 con 0x7f195c15c310 2023-11-28T20:53:10.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.811+0000 7f196187e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c14b550 con 0x7f195c0c7890 2023-11-28T20:53:10.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.811+0000 7f196087c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 0x7f195c14aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196087c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 0x7f195c14aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48088/0 (socket says 172.21.15.45:48088) 2023-11-28T20:53:10.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196087c640 1 -- 172.21.15.45:0/99254655 learned_addr learned my addr 172.21.15.45:0/99254655 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:10.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196107d640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f195c15c310 0x7f195c0b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196087c640 1 -- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f195c0c7890 msgr2=0x7f195c0b6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196087c640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f195c0c7890 0x7f195c0b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196087c640 1 -- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f195c15c310 msgr2=0x7f195c0b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196087c640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f195c15c310 0x7f195c0b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196087c640 1 -- 172.21.15.45:0/99254655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1950009400 con 0x7f195c099a40 2023-11-28T20:53:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196107d640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f195c15c310 0x7f195c0b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.812+0000 7f196087c640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 0x7f195c14aa30 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f194c00c7c0 tx=0x7f194c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.813+0000 7f19597fa640 1 -- 172.21.15.45:0/99254655 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c00a640 con 0x7f195c099a40 2023-11-28T20:53:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.813+0000 7f19597fa640 1 -- 172.21.15.45:0/99254655 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c00a7c0 con 0x7f195c099a40 2023-11-28T20:53:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.813+0000 7f19597fa640 1 -- 172.21.15.45:0/99254655 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c00ab70 con 0x7f195c099a40 2023-11-28T20:53:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.813+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c14b830 con 0x7f195c099a40 2023-11-28T20:53:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.813+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c14bdf0 con 0x7f195c099a40 2023-11-28T20:53:10.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.814+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195c09f180 con 0x7f195c099a40 2023-11-28T20:53:10.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.815+0000 7f19597fa640 1 -- 172.21.15.45:0/99254655 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 52) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f194c00acd0 con 0x7f195c099a40 2023-11-28T20:53:10.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.816+0000 7f19597fa640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f194806b0d0 0x7f194806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:10.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.816+0000 7f19597fa640 1 -- 172.21.15.45:0/99254655 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f194c089440 con 0x7f195c099a40 2023-11-28T20:53:10.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.817+0000 7f195bfff640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f194806b0d0 0x7f194806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:10.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.818+0000 7f195bfff640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f194806b0d0 0x7f194806d590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1950035000 tx=0x7f1950030000 comp rx=0 tx=0).ready entity=mgr.6038 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:10.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:10.822+0000 7f19597fa640 1 -- 172.21.15.45:0/99254655 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f194c053db0 con 0x7f195c099a40 2023-11-28T20:53:11.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.070+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f195c04e920 con 0x7f195c099a40 2023-11-28T20:53:11.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.474+0000 7f19597fa640 1 -- 172.21.15.45:0/99254655 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v53) v1 ==== 93+0+0 (secure 0 0 0) 0x7f194c0180a0 con 0x7f195c099a40 2023-11-28T20:53:11.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f194806b0d0 msgr2=0x7f194806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:11.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f194806b0d0 0x7f194806d590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1950035000 tx=0x7f1950030000 comp rx=0 tx=0).stop 2023-11-28T20:53:11.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 msgr2=0x7f195c14aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:11.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 0x7f195c14aa30 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f194c00c7c0 tx=0x7f194c00cc80 comp rx=0 tx=0).stop 2023-11-28T20:53:11.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 shutdown_connections 2023-11-28T20:53:11.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f194806b0d0 0x7f194806d590 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:11.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f195c15c310 0x7f195c0b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:11.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f195c0c7890 0x7f195c0b6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:11.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 --2- 172.21.15.45:0/99254655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f195c099a40 0x7f195c14aa30 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:11.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 >> 172.21.15.45:0/99254655 conn(0x7f195c0d0f10 msgr2=0x7f195c0cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:11.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.478+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 shutdown_connections 2023-11-28T20:53:11.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:11.479+0000 7f196187e640 1 -- 172.21.15.45:0/99254655 wait complete. 2023-11-28T20:53:11.503 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:53:11.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:53:11.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:53:11.705 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:53:11.705 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:53:11.705 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:11.762 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:53:11.762 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:11.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:53:12.012 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.011+0000 7f684642c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:53:12.037 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:12.035+0000 7fe3bec21040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:53:12.064 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.063+0000 7f684642c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:53:12.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.067+0000 7ff066217640 1 Processor -- start 2023-11-28T20:53:12.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.068+0000 7ff066217640 1 -- start start 2023-11-28T20:53:12.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.069+0000 7ff066217640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600cdf10 0x7ff06006a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.069+0000 7ff066217640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0600c3880 con 0x7ff0600ce460 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.069+0000 7ff066217640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0600cd7c0 con 0x7ff0600661f0 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.069+0000 7ff066217640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06015b960 con 0x7ff0600cdf10 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.070+0000 7ff05ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff0600ce460 0x7ff06006abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51422/0 (socket says 172.21.15.45:51422) 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.070+0000 7ff05ffff640 1 -- 172.21.15.45:0/3332738000 learned_addr learned my addr 172.21.15.45:0/3332738000 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.070+0000 7ff057fff640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600cdf10 0x7ff06006a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.070+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388226291 0 0) 0x7ff058001490 con 0x7ff0600ce460 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.070+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff04c003520 con 0x7ff0600ce460 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993128575 0 0) 0x7ff0600cd7c0 con 0x7ff0600661f0 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff058001490 con 0x7ff0600661f0 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4203075539 0 0) 0x7ff04c003520 con 0x7ff0600ce460 2023-11-28T20:53:12.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600cdf10 msgr2=0x7ff06006a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600cdf10 0x7ff06006a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 >> v1:172.21.15.190:6789/0 conn(0x7ff0600661f0 legacy=0x7ff0600665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0600cd490 con 0x7ff0600ce460 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ff058001b80 con 0x7ff0600ce460 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 >> v1:172.21.15.45:6789/0 conn(0x7ff0600ce460 legacy=0x7ff06006abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600cdf10 0x7ff04c003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff04c0040e0 0x7ff04c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c004d00 0x7ff04c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06015b960 con 0x7ff04c0040e0 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c00bbe0 con 0x7ff04c004d00 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c00bda0 con 0x7ff0600cdf10 2023-11-28T20:53:12.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff057fff640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff04c0040e0 0x7ff04c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.071+0000 7ff05ffff640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c004d00 0x7ff04c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.072+0000 7ff05f7fe640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600cdf10 0x7ff04c003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.072+0000 7ff057fff640 1 -- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600cdf10 msgr2=0x7ff04c003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.072+0000 7ff057fff640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600cdf10 0x7ff04c003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.072+0000 7ff057fff640 1 -- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c004d00 msgr2=0x7ff04c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.072+0000 7ff057fff640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c004d00 0x7ff04c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.072+0000 7ff057fff640 1 -- 172.21.15.45:0/3332738000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c003520 con 0x7ff04c0040e0 2023-11-28T20:53:12.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.072+0000 7ff05f7fe640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600cdf10 0x7ff04c003b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.074+0000 7ff057fff640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff04c0040e0 0x7ff04c0047c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ff048009a90 tx=0x7ff04c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a1563d8fc6272e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.074+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04802c680 con 0x7ff04c0040e0 2023-11-28T20:53:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.074+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04802c7e0 con 0x7ff04c0040e0 2023-11-28T20:53:12.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.074+0000 7ff05effd640 1 -- 172.21.15.45:0/3332738000 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04802cb90 con 0x7ff04c0040e0 2023-11-28T20:53:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.075+0000 7ff066217640 1 -- 172.21.15.45:0/3332738000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff04c0040e0 msgr2=0x7ff04c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.075+0000 7ff066217640 1 --2- 172.21.15.45:0/3332738000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff04c0040e0 0x7ff04c0047c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ff048009a90 tx=0x7ff04c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.075+0000 7ff05ffff640 1 -- 172.21.15.45:0/3332738000 reap_dead start 2023-11-28T20:53:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.075+0000 7ff066217640 1 -- 172.21.15.45:0/3332738000 shutdown_connections 2023-11-28T20:53:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.075+0000 7ff066217640 1 -- 172.21.15.45:0/3332738000 >> 172.21.15.45:0/3332738000 conn(0x7ff0600cc970 msgr2=0x7ff0600ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.075+0000 7ff066217640 1 -- 172.21.15.45:0/3332738000 shutdown_connections 2023-11-28T20:53:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.075+0000 7ff066217640 1 -- 172.21.15.45:0/3332738000 wait complete. 2023-11-28T20:53:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.078+0000 7ff066217640 1 Processor -- start 2023-11-28T20:53:12.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.078+0000 7ff066217640 1 -- start start 2023-11-28T20:53:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.079+0000 7ff066217640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c0040e0 0x7ff06015a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.079+0000 7ff066217640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600c4430 0x7ff0600c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff066217640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0600c5e80 0x7ff0600c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff066217640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0600c6840 con 0x7ff0600c5e80 2023-11-28T20:53:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff066217640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0600c8610 con 0x7ff04c0040e0 2023-11-28T20:53:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff066217640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0600c4c60 con 0x7ff0600c4430 2023-11-28T20:53:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0600c5e80 0x7ff0600c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0600c5e80 0x7ff0600c6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48102/0 (socket says 172.21.15.45:48102) 2023-11-28T20:53:12.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05ffff640 1 -- 172.21.15.45:0/2082999241 learned_addr learned my addr 172.21.15.45:0/2082999241 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:12.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05f7fe640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c0040e0 0x7ff06015a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff057fff640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600c4430 0x7ff0600c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05ffff640 1 -- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600c4430 msgr2=0x7ff0600c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05ffff640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600c4430 0x7ff0600c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05ffff640 1 -- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c0040e0 msgr2=0x7ff06015a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05ffff640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c0040e0 0x7ff06015a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05ffff640 1 -- 172.21.15.45:0/2082999241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0600c4f40 con 0x7ff0600c5e80 2023-11-28T20:53:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff05f7fe640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c0040e0 0x7ff06015a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.080+0000 7ff057fff640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600c4430 0x7ff0600c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.081+0000 7ff05ffff640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0600c5e80 0x7ff0600c6300 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ff060073bd0 tx=0x7ff05800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.081+0000 7ff05cff9640 1 -- 172.21.15.45:0/2082999241 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05800ec10 con 0x7ff0600c5e80 2023-11-28T20:53:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.081+0000 7ff05cff9640 1 -- 172.21.15.45:0/2082999241 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05800ed70 con 0x7ff0600c5e80 2023-11-28T20:53:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.081+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0600c88b0 con 0x7ff0600c5e80 2023-11-28T20:53:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.081+0000 7ff05cff9640 1 -- 172.21.15.45:0/2082999241 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05800eee0 con 0x7ff0600c5e80 2023-11-28T20:53:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.081+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06015a5a0 con 0x7ff0600c5e80 2023-11-28T20:53:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.082+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff060072bd0 con 0x7ff0600c5e80 2023-11-28T20:53:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.083+0000 7ff05cff9640 1 -- 172.21.15.45:0/2082999241 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) v1 ==== 89447+0+0 (secure 0 0 0) 0x7ff058013660 con 0x7ff0600c5e80 2023-11-28T20:53:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.085+0000 7ff05cff9640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff04006b080 0x7ff04006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.085+0000 7ff05cff9640 1 -- 172.21.15.45:0/2082999241 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff058089fa0 con 0x7ff0600c5e80 2023-11-28T20:53:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.085+0000 7ff05f7fe640 1 -- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff04006b080 msgr2=0x7ff04006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:12.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.085+0000 7ff05f7fe640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff04006b080 0x7ff04006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.088+0000 7ff05cff9640 1 -- 172.21.15.45:0/2082999241 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff058054a80 con 0x7ff0600c5e80 2023-11-28T20:53:12.098 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:12.096+0000 7fe3bec21040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:53:12.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.281+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0600c3880 con 0x7ff0600c5e80 2023-11-28T20:53:12.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.284+0000 7ff05cff9640 1 -- 172.21.15.45:0/2082999241 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+17482 (secure 0 0 0) 0x7ff0600c3880 con 0x7ff0600c5e80 2023-11-28T20:53:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:12.292 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":120,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:09.443025+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:50.993341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:52.004867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":7}]},{"pgid":"14.2a","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6833/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:6832/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3239477442":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3809276606":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3450487150":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2166104670":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/4069818189":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2641549859":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/1724154865":"2023-11-28T21:51:59.768228+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:53:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.286+0000 7ff05f7fe640 1 -- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff04006b080 msgr2=0x7ff04006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.286+0000 7ff05f7fe640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff04006b080 0x7ff04006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:53:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff04006b080 msgr2=0x7ff04006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff04006b080 0x7ff04006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0600c5e80 msgr2=0x7ff0600c6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0600c5e80 0x7ff0600c6300 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ff060073bd0 tx=0x7ff05800e530 comp rx=0 tx=0).stop 2023-11-28T20:53:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 shutdown_connections 2023-11-28T20:53:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7ff04006b080 0x7ff04006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0600c5e80 0x7ff0600c6300 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0600c4430 0x7ff0600c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 --2- 172.21.15.45:0/2082999241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff04c0040e0 0x7ff06015a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 >> 172.21.15.45:0/2082999241 conn(0x7ff0600cc970 msgr2=0x7ff060071d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 shutdown_connections 2023-11-28T20:53:12.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.287+0000 7ff066217640 1 -- 172.21.15.45:0/2082999241 wait complete. 2023-11-28T20:53:12.309 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:51:50.993341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:51:52.004867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:53:12.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:53:12.515 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.514+0000 7f684642c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:53:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.514+0000 7efc16b16640 1 Processor -- start 2023-11-28T20:53:12.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.515+0000 7efc16b16640 1 -- start start 2023-11-28T20:53:12.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.516+0000 7efc16b16640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efc1015d060 0x7efc100ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.516+0000 7efc16b16640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efc10161900 0x7efc1006b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.516+0000 7efc16b16640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc100af3e0 con 0x7efc100b4ab0 2023-11-28T20:53:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.516+0000 7efc16b16640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc10160e20 con 0x7efc10161900 2023-11-28T20:53:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.516+0000 7efc16b16640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc10161100 con 0x7efc1015d060 2023-11-28T20:53:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.517+0000 7efc15313640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efc10161900 0x7efc1006b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.517+0000 7efc16315640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efc100b4ab0 0x7efc100661f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51436/0 (socket says 172.21.15.45:51436) 2023-11-28T20:53:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.517+0000 7efc16315640 1 -- 172.21.15.45:0/82429607 learned_addr learned my addr 172.21.15.45:0/82429607 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.517+0000 7efc15b14640 1 --2- 172.21.15.45:0/82429607 >> v2:172.21.15.190:3300/0 conn(0x7efc1015d060 0x7efc100ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.517+0000 7efc15313640 1 -- 172.21.15.45:0/82429607 >> v2:172.21.15.190:3300/0 conn(0x7efc1015d060 msgr2=0x7efc100ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.517+0000 7efc15313640 1 --2- 172.21.15.45:0/82429607 >> v2:172.21.15.190:3300/0 conn(0x7efc1015d060 0x7efc100ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.517+0000 7efc15313640 1 -- 172.21.15.45:0/82429607 >> v1:172.21.15.45:6789/0 conn(0x7efc100b4ab0 legacy=0x7efc100661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.517+0000 7efc15313640 1 -- 172.21.15.45:0/82429607 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc101613e0 con 0x7efc10161900 2023-11-28T20:53:12.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.517+0000 7efc14b12640 1 -- 172.21.15.45:0/82429607 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559071721 0 0) 0x7efc100af3e0 con 0x7efc100b4ab0 2023-11-28T20:53:12.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.519+0000 7efc15313640 1 --2- 172.21.15.45:0/82429607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc10161900 0x7efc1006b510 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7efc100af7e0 tx=0x7efc0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5dccdc26f32293 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:12.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.519+0000 7efc14b12640 1 -- 172.21.15.45:0/82429607 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00031830 con 0x7efc10161900 2023-11-28T20:53:12.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.519+0000 7efc14b12640 1 -- 172.21.15.45:0/82429607 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc00031990 con 0x7efc10161900 2023-11-28T20:53:12.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.519+0000 7efc14b12640 1 -- 172.21.15.45:0/82429607 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00031d20 con 0x7efc10161900 2023-11-28T20:53:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.520+0000 7efc16b16640 1 -- 172.21.15.45:0/82429607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc10161900 msgr2=0x7efc1006b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.520+0000 7efc16b16640 1 --2- 172.21.15.45:0/82429607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc10161900 0x7efc1006b510 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7efc100af7e0 tx=0x7efc0002b8d0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.520+0000 7efc16b16640 1 -- 172.21.15.45:0/82429607 shutdown_connections 2023-11-28T20:53:12.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.520+0000 7efc16b16640 1 --2- 172.21.15.45:0/82429607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc10161900 0x7efc1006b510 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.520+0000 7efc16b16640 1 --2- 172.21.15.45:0/82429607 >> v2:172.21.15.190:3300/0 conn(0x7efc1015d060 0x7efc100ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.520+0000 7efc16b16640 1 -- 172.21.15.45:0/82429607 >> 172.21.15.45:0/82429607 conn(0x7efc10071290 msgr2=0x7efc100716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.520+0000 7efc16b16640 1 -- 172.21.15.45:0/82429607 shutdown_connections 2023-11-28T20:53:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.520+0000 7efc16b16640 1 -- 172.21.15.45:0/82429607 wait complete. 2023-11-28T20:53:12.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.522+0000 7efc16b16640 1 Processor -- start 2023-11-28T20:53:12.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.522+0000 7efc16b16640 1 -- start start 2023-11-28T20:53:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc16b16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc100b4ab0 0x7efc1013f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc16b16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc1015d060 0x7efc1015e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc16b16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc10161900 0x7efc1014e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc16b16640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1014ce90 con 0x7efc1015d060 2023-11-28T20:53:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc16b16640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1014d820 con 0x7efc100b4ab0 2023-11-28T20:53:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc16b16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1014db00 con 0x7efc10161900 2023-11-28T20:53:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc15313640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc1015d060 0x7efc1015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc15313640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc1015d060 0x7efc1015e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48120/0 (socket says 172.21.15.45:48120) 2023-11-28T20:53:12.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc15313640 1 -- 172.21.15.45:0/726976089 learned_addr learned my addr 172.21.15.45:0/726976089 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc15b14640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc100b4ab0 0x7efc1013f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.523+0000 7efc16315640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc10161900 0x7efc1014e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc15313640 1 -- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc10161900 msgr2=0x7efc1014e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc15313640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc10161900 0x7efc1014e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc15313640 1 -- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc100b4ab0 msgr2=0x7efc1013f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc15313640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc100b4ab0 0x7efc1013f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc15313640 1 -- 172.21.15.45:0/726976089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00009400 con 0x7efc1015d060 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc15b14640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc100b4ab0 0x7efc1013f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc15313640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc1015d060 0x7efc1015e080 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7efc00000c00 tx=0x7efc000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:12.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc067fc640 1 -- 172.21.15.45:0/726976089 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0003aa60 con 0x7efc1015d060 2023-11-28T20:53:12.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc067fc640 1 -- 172.21.15.45:0/726976089 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0003abe0 con 0x7efc1015d060 2023-11-28T20:53:12.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc067fc640 1 -- 172.21.15.45:0/726976089 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000443f0 con 0x7efc1015d060 2023-11-28T20:53:12.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.524+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1014dde0 con 0x7efc1015d060 2023-11-28T20:53:12.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.525+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1014e3a0 con 0x7efc1015d060 2023-11-28T20:53:12.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.526+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc10073840 con 0x7efc1015d060 2023-11-28T20:53:12.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.526+0000 7efc067fc640 1 -- 172.21.15.45:0/726976089 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) v1 ==== 89447+0+0 (secure 0 0 0) 0x7efc0004d450 con 0x7efc1015d060 2023-11-28T20:53:12.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.527+0000 7efc067fc640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7efbf406b080 0x7efbf406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.527+0000 7efc067fc640 1 -- 172.21.15.45:0/726976089 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efc000ac0d0 con 0x7efc1015d060 2023-11-28T20:53:12.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.527+0000 7efc15b14640 1 -- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7efbf406b080 msgr2=0x7efbf406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:12.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.527+0000 7efc15b14640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7efbf406b080 0x7efbf406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:12.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.532+0000 7efc067fc640 1 -- 172.21.15.45:0/726976089 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc00076930 con 0x7efc1015d060 2023-11-28T20:53:12.570 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.569+0000 7f684642c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:53:12.580 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:12.578+0000 7fe3bec21040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:53:12.619 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.618+0000 7f684642c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:53:12.635 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:12.633+0000 7fe3bec21040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:53:12.680 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.679+0000 7f684642c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:53:12.688 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:12.686+0000 7fe3bec21040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:53:12.728 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.727+0000 7f684642c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:53:12.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.728+0000 7efc15b14640 1 -- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7efbf406b080 msgr2=0x7efbf406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:12.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.728+0000 7efc15b14640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7efbf406b080 0x7efbf406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:53:12.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.729+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc1015c660 con 0x7efc1015d060 2023-11-28T20:53:12.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.730+0000 7efc067fc640 1 -- 172.21.15.45:0/726976089 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc00035020 con 0x7efc1015d060 2023-11-28T20:53:12.732 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:53:12.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7efbf406b080 msgr2=0x7efbf406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:12.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7efbf406b080 0x7efbf406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc1015d060 msgr2=0x7efc1015e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc1015d060 0x7efc1015e080 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7efc00000c00 tx=0x7efc000046c0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 shutdown_connections 2023-11-28T20:53:12.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7efbf406b080 0x7efbf406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc10161900 0x7efc1014e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc1015d060 0x7efc1015e080 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 --2- 172.21.15.45:0/726976089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc100b4ab0 0x7efc1013f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 >> 172.21.15.45:0/726976089 conn(0x7efc10071290 msgr2=0x7efc1006e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:12.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 shutdown_connections 2023-11-28T20:53:12.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.733+0000 7efc16b16640 1 -- 172.21.15.45:0/726976089 wait complete. 2023-11-28T20:53:12.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:53:12.758 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:12.752+0000 7fe3bec21040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:53:12.807 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:12.805+0000 7fe3bec21040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:53:12.838 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.837+0000 7f684642c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:53:12.900 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.899+0000 7f684642c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:53:12.927 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:12.925+0000 7fe3bec21040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:53:12.949 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:12.948+0000 7f684642c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:53:12.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.960+0000 7f9995390640 1 Processor -- start 2023-11-28T20:53:12.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.960+0000 7f9995390640 1 -- start start 2023-11-28T20:53:12.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.962+0000 7f9995390640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9990160aa0 0x7f99900b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.962+0000 7f9995390640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9990160e80 con 0x7f99900ce7f0 2023-11-28T20:53:12.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.962+0000 7f9995390640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900d1220 con 0x7f99900d24f0 2023-11-28T20:53:12.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.962+0000 7f9995390640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b6960 con 0x7f9990160aa0 2023-11-28T20:53:12.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f99900ce7f0 0x7f99901612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51440/0 (socket says 172.21.15.45:51440) 2023-11-28T20:53:12.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998f7fe640 1 -- 172.21.15.45:0/3718797719 learned_addr learned my addr 172.21.15.45:0/3718797719 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:12.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998effd640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9990160aa0 0x7f99900b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434182105 0 0) 0x7f9990160e80 con 0x7f99900ce7f0 2023-11-28T20:53:12.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997c003520 con 0x7f99900ce7f0 2023-11-28T20:53:12.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359433458 0 0) 0x7f99900d1220 con 0x7f99900d24f0 2023-11-28T20:53:12.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9990160e80 con 0x7f99900d24f0 2023-11-28T20:53:12.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638477385 0 0) 0x7f997c003520 con 0x7f99900ce7f0 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9990160aa0 msgr2=0x7f99900b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.963+0000 7f998e7fc640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9990160aa0 0x7f99900b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 >> v1:172.21.15.190:6789/0 conn(0x7f99900d24f0 legacy=0x7f99900ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99900d1470 con 0x7f99900ce7f0 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f99880019f0 con 0x7f99900ce7f0 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 >> v1:172.21.15.45:6789/0 conn(0x7f99900ce7f0 legacy=0x7f99901612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f997c003b30 0x7f997c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9990160aa0 0x7f997c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f997c004d00 0x7f997c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b6960 con 0x7f997c004d00 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c00ba30 con 0x7f997c003b30 2023-11-28T20:53:12.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997c00bbf0 con 0x7f9990160aa0 2023-11-28T20:53:12.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998f7fe640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f997c004d00 0x7f997c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f998effd640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f997c003b30 0x7f997c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.964+0000 7f9987fff640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9990160aa0 0x7f997c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.965+0000 7f998f7fe640 1 -- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9990160aa0 msgr2=0x7f997c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:12.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.965+0000 7f998f7fe640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9990160aa0 0x7f997c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.965+0000 7f998f7fe640 1 -- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f997c003b30 msgr2=0x7f997c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:12.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.965+0000 7f998f7fe640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f997c003b30 0x7f997c003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.965+0000 7f998f7fe640 1 -- 172.21.15.45:0/3718797719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997c003520 con 0x7f997c004d00 2023-11-28T20:53:12.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.965+0000 7f998effd640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f997c003b30 0x7f997c003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:12.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.966+0000 7f998f7fe640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f997c004d00 0x7f997c005180 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f99900cec60 tx=0x7f997c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=507f8b90894aa270 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:12.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.966+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988034630 con 0x7f997c004d00 2023-11-28T20:53:12.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.966+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998802f070 con 0x7f997c004d00 2023-11-28T20:53:12.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.966+0000 7f998e7fc640 1 -- 172.21.15.45:0/3718797719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99880349f0 con 0x7f997c004d00 2023-11-28T20:53:12.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.967+0000 7f9995390640 1 -- 172.21.15.45:0/3718797719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f997c004d00 msgr2=0x7f997c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.967+0000 7f9995390640 1 --2- 172.21.15.45:0/3718797719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f997c004d00 0x7f997c005180 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f99900cec60 tx=0x7f997c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.967+0000 7f998f7fe640 1 -- 172.21.15.45:0/3718797719 reap_dead start 2023-11-28T20:53:12.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.967+0000 7f9995390640 1 -- 172.21.15.45:0/3718797719 shutdown_connections 2023-11-28T20:53:12.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.967+0000 7f9995390640 1 -- 172.21.15.45:0/3718797719 >> 172.21.15.45:0/3718797719 conn(0x7f9990070b20 msgr2=0x7f9990070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:12.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.967+0000 7f9995390640 1 -- 172.21.15.45:0/3718797719 shutdown_connections 2023-11-28T20:53:12.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.967+0000 7f9995390640 1 -- 172.21.15.45:0/3718797719 wait complete. 2023-11-28T20:53:12.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.969+0000 7f9995390640 1 Processor -- start 2023-11-28T20:53:12.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.969+0000 7f9995390640 1 -- start start 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.970+0000 7f9995390640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f997c004d00 0x7f999015c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.970+0000 7f9995390640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f999015d390 0x7f99901f3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.970+0000 7f9995390640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f999015f9b0 0x7f99900b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.970+0000 7f9995390640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b45e0 con 0x7f999015f9b0 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.970+0000 7f9995390640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999015cc00 con 0x7f999015d390 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.970+0000 7f9995390640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b49d0 con 0x7f997c004d00 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.970+0000 7f998f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f999015f9b0 0x7f99900b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.970+0000 7f998f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f999015f9b0 0x7f99900b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48150/0 (socket says 172.21.15.45:48150) 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.970+0000 7f998f7fe640 1 -- 172.21.15.45:0/2907504708 learned_addr learned my addr 172.21.15.45:0/2907504708 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.971+0000 7f9987fff640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f999015d390 0x7f99901f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.971+0000 7f998effd640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f997c004d00 0x7f999015c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:12.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.971+0000 7f998f7fe640 1 -- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f997c004d00 msgr2=0x7f999015c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.971+0000 7f998f7fe640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f997c004d00 0x7f999015c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.971+0000 7f998f7fe640 1 -- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f999015d390 msgr2=0x7f99901f3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:12.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.971+0000 7f998f7fe640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f999015d390 0x7f99901f3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:12.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.971+0000 7f998f7fe640 1 -- 172.21.15.45:0/2907504708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999015d890 con 0x7f999015f9b0 2023-11-28T20:53:12.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.971+0000 7f998f7fe640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f999015f9b0 0x7f99900b4e90 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9988005c20 tx=0x7f9988001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:12.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.972+0000 7f99877fe640 1 -- 172.21.15.45:0/2907504708 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988034840 con 0x7f999015f9b0 2023-11-28T20:53:12.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.972+0000 7f99877fe640 1 -- 172.21.15.45:0/2907504708 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99880349a0 con 0x7f999015f9b0 2023-11-28T20:53:12.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.972+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999015db70 con 0x7f999015f9b0 2023-11-28T20:53:12.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.972+0000 7f99877fe640 1 -- 172.21.15.45:0/2907504708 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988034d10 con 0x7f999015f9b0 2023-11-28T20:53:12.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.972+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99901f3860 con 0x7f999015f9b0 2023-11-28T20:53:12.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.973+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99900733d0 con 0x7f999015f9b0 2023-11-28T20:53:12.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.974+0000 7f99877fe640 1 -- 172.21.15.45:0/2907504708 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f9988031030 con 0x7f999015f9b0 2023-11-28T20:53:12.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.975+0000 7f99877fe640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f997006b0b0 0x7f997006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:12.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.975+0000 7f99877fe640 1 -- 172.21.15.45:0/2907504708 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f99880abfe0 con 0x7f999015f9b0 2023-11-28T20:53:12.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.975+0000 7f998effd640 1 -- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f997006b0b0 msgr2=0x7f997006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:12.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.975+0000 7f998effd640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f997006b0b0 0x7f997006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:12.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:12.979+0000 7f99877fe640 1 -- 172.21.15.45:0/2907504708 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9988076b20 con 0x7f999015f9b0 2023-11-28T20:53:12.996 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:12.994+0000 7fe3bec21040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:53:13.008 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:13.007+0000 7f684642c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:53:13.049 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:13.047+0000 7fe3bec21040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:53:13.058 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:13.057+0000 7f684642c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:53:13.109 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:13.107+0000 7fe3bec21040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:53:13.162 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:13.160+0000 7fe3bec21040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:53:13.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.170+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f99900733d0 con 0x7f999015f9b0 2023-11-28T20:53:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.171+0000 7f99877fe640 1 -- 172.21.15.45:0/2907504708 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f998807a140 con 0x7f999015f9b0 2023-11-28T20:53:13.173 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f997006b0b0 msgr2=0x7f997006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f997006b0b0 0x7f997006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f999015f9b0 msgr2=0x7f99900b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f999015f9b0 0x7f99900b4e90 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9988005c20 tx=0x7f9988001cb0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 shutdown_connections 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f997006b0b0 0x7f997006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f999015f9b0 0x7f99900b4e90 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f999015d390 0x7f99901f3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 --2- 172.21.15.45:0/2907504708 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f997c004d00 0x7f999015c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 >> 172.21.15.45:0/2907504708 conn(0x7f9990070b20 msgr2=0x7f99900cab70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:13.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 shutdown_connections 2023-11-28T20:53:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.174+0000 7f9995390640 1 -- 172.21.15.45:0/2907504708 wait complete. 2023-11-28T20:53:13.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:53:13.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.406+0000 7fc721ded640 1 Processor -- start 2023-11-28T20:53:13.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.407+0000 7fc721ded640 1 -- start start 2023-11-28T20:53:13.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.408+0000 7fc721ded640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc71c0af070 0x7fc71c0acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.408+0000 7fc721ded640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c15db60 con 0x7fc71c06e8e0 2023-11-28T20:53:13.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.408+0000 7fc721ded640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c0ad2c0 con 0x7fc71c06b510 2023-11-28T20:53:13.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.408+0000 7fc721ded640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c15e700 con 0x7fc71c0af070 2023-11-28T20:53:13.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.409+0000 7fc720deb640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc71c06e8e0 0x7fc71c15dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51446/0 (socket says 172.21.15.45:51446) 2023-11-28T20:53:13.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.409+0000 7fc720deb640 1 -- 172.21.15.45:0/1722710153 learned_addr learned my addr 172.21.15.45:0/1722710153 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:13.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.409+0000 7fc71bfff640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc71c0af070 0x7fc71c0acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.409+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510329094 0 0) 0x7fc71c15db60 con 0x7fc71c06e8e0 2023-11-28T20:53:13.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.409+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70c003520 con 0x7fc71c06e8e0 2023-11-28T20:53:13.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.409+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524474207 0 0) 0x7fc71c0ad2c0 con 0x7fc71c06b510 2023-11-28T20:53:13.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.409+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc71c15db60 con 0x7fc71c06b510 2023-11-28T20:53:13.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098246516 0 0) 0x7fc70c003520 con 0x7fc71c06e8e0 2023-11-28T20:53:13.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc71c0af070 msgr2=0x7fc71c0acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:13.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc71c0af070 0x7fc71c0acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 >> v1:172.21.15.190:6789/0 conn(0x7fc71c06b510 legacy=0x7fc71c0b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c0af4f0 con 0x7fc71c06e8e0 2023-11-28T20:53:13.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc710001670 con 0x7fc71c06e8e0 2023-11-28T20:53:13.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 >> v1:172.21.15.45:6789/0 conn(0x7fc71c06e8e0 legacy=0x7fc71c15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c003b30 0x7fc70c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc71c0af070 0x7fc70c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc70c004d00 0x7fc70c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c15e700 con 0x7fc70c003b30 2023-11-28T20:53:13.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c00ba30 con 0x7fc70c004d00 2023-11-28T20:53:13.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c00bbf0 con 0x7fc71c0af070 2023-11-28T20:53:13.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc720deb640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c003b30 0x7fc70c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc7215ec640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc70c004d00 0x7fc70c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc720deb640 1 -- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc71c0af070 msgr2=0x7fc70c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.410+0000 7fc71bfff640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc71c0af070 0x7fc70c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.411+0000 7fc720deb640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc71c0af070 0x7fc70c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.411+0000 7fc720deb640 1 -- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc70c004d00 msgr2=0x7fc70c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:13.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.411+0000 7fc720deb640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc70c004d00 0x7fc70c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.411+0000 7fc720deb640 1 -- 172.21.15.45:0/1722710153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c003520 con 0x7fc70c003b30 2023-11-28T20:53:13.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.411+0000 7fc7215ec640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc70c004d00 0x7fc70c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.412+0000 7fc720deb640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c003b30 0x7fc70c003f30 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc710007f80 tx=0x7fc70c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec633c449c8a3de9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.412+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710005b90 con 0x7fc70c003b30 2023-11-28T20:53:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.412+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc710005d10 con 0x7fc70c003b30 2023-11-28T20:53:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.412+0000 7fc71b7fe640 1 -- 172.21.15.45:0/1722710153 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7100072b0 con 0x7fc70c003b30 2023-11-28T20:53:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.412+0000 7fc721ded640 1 -- 172.21.15.45:0/1722710153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c003b30 msgr2=0x7fc70c003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.412+0000 7fc721ded640 1 --2- 172.21.15.45:0/1722710153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c003b30 0x7fc70c003f30 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc710007f80 tx=0x7fc70c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.413+0000 7fc7215ec640 1 -- 172.21.15.45:0/1722710153 reap_dead start 2023-11-28T20:53:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.413+0000 7fc721ded640 1 -- 172.21.15.45:0/1722710153 shutdown_connections 2023-11-28T20:53:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.413+0000 7fc721ded640 1 -- 172.21.15.45:0/1722710153 >> 172.21.15.45:0/1722710153 conn(0x7fc71c071240 msgr2=0x7fc71c071650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.413+0000 7fc721ded640 1 -- 172.21.15.45:0/1722710153 shutdown_connections 2023-11-28T20:53:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.413+0000 7fc721ded640 1 -- 172.21.15.45:0/1722710153 wait complete. 2023-11-28T20:53:13.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.415+0000 7fc721ded640 1 Processor -- start 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.415+0000 7fc721ded640 1 -- start start 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.416+0000 7fc721ded640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc70c003b30 0x7fc71c1f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.416+0000 7fc721ded640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc71c1f33f0 0x7fc71c151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.416+0000 7fc721ded640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c153270 0x7fc71c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.416+0000 7fc721ded640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c15e9e0 con 0x7fc71c1f33f0 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.416+0000 7fc721ded640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c1511c0 con 0x7fc71c153270 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.416+0000 7fc721ded640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c06e4d0 con 0x7fc70c003b30 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.416+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc71c1f33f0 0x7fc71c151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.416+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc71c1f33f0 0x7fc71c151560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48166/0 (socket says 172.21.15.45:48166) 2023-11-28T20:53:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.416+0000 7fc71bfff640 1 -- 172.21.15.45:0/1872906725 learned_addr learned my addr 172.21.15.45:0/1872906725 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.417+0000 7fc7215ec640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c153270 0x7fc71c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.417+0000 7fc720deb640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc70c003b30 0x7fc71c1f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.417+0000 7fc71bfff640 1 -- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc70c003b30 msgr2=0x7fc71c1f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.417+0000 7fc71bfff640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc70c003b30 0x7fc71c1f2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.417+0000 7fc71bfff640 1 -- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c153270 msgr2=0x7fc71c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.417+0000 7fc71bfff640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c153270 0x7fc71c1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.417+0000 7fc71bfff640 1 -- 172.21.15.45:0/1872906725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c151aa0 con 0x7fc71c1f33f0 2023-11-28T20:53:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.417+0000 7fc71bfff640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc71c1f33f0 0x7fc71c151560 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc708010890 tx=0x7fc708010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.417+0000 7fc7197fa640 1 -- 172.21.15.45:0/1872906725 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70800eb00 con 0x7fc71c1f33f0 2023-11-28T20:53:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.418+0000 7fc7197fa640 1 -- 172.21.15.45:0/1872906725 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70800ec80 con 0x7fc71c1f33f0 2023-11-28T20:53:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.418+0000 7fc7197fa640 1 -- 172.21.15.45:0/1872906725 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc708010e80 con 0x7fc71c1f33f0 2023-11-28T20:53:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.418+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c1f3870 con 0x7fc71c1f33f0 2023-11-28T20:53:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.418+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c151fb0 con 0x7fc71c1f33f0 2023-11-28T20:53:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.419+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c09e100 con 0x7fc71c1f33f0 2023-11-28T20:53:13.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.420+0000 7fc7197fa640 1 -- 172.21.15.45:0/1872906725 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) v1 ==== 89447+0+0 (secure 0 0 0) 0x7fc7080143f0 con 0x7fc71c1f33f0 2023-11-28T20:53:13.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.421+0000 7fc7197fa640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fc70006b0b0 0x7fc70006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.421+0000 7fc7197fa640 1 -- 172.21.15.45:0/1872906725 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc708089d10 con 0x7fc71c1f33f0 2023-11-28T20:53:13.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.421+0000 7fc720deb640 1 -- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fc70006b0b0 msgr2=0x7fc70006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:13.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.421+0000 7fc720deb640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fc70006b0b0 0x7fc70006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:13.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.425+0000 7fc7197fa640 1 -- 172.21.15.45:0/1872906725 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc70808f020 con 0x7fc71c1f33f0 2023-11-28T20:53:13.440 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:13.439+0000 7f684642c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:53:13.607 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:13.606+0000 7f684642c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:53:13.620 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:13.618+0000 7fe3bec21040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:53:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.622+0000 7fc720deb640 1 -- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fc70006b0b0 msgr2=0x7fc70006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.622+0000 7fc720deb640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fc70006b0b0 0x7fc70006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:53:13.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.664+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc71c1f3870 con 0x7fc71c1f33f0 2023-11-28T20:53:13.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.665+0000 7fc7197fa640 1 -- 172.21.15.45:0/1872906725 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc71c1f3870 con 0x7fc71c1f33f0 2023-11-28T20:53:13.667 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:53:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.667+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fc70006b0b0 msgr2=0x7fc70006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.667+0000 7fc721ded640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fc70006b0b0 0x7fc70006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.667+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc71c1f33f0 msgr2=0x7fc71c151560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.667+0000 7fc721ded640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc71c1f33f0 0x7fc71c151560 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc708010890 tx=0x7fc708010d50 comp rx=0 tx=0).stop 2023-11-28T20:53:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.667+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 shutdown_connections 2023-11-28T20:53:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.667+0000 7fc721ded640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7fc70006b0b0 0x7fc70006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.667+0000 7fc721ded640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c153270 0x7fc71c1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.667+0000 7fc721ded640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc71c1f33f0 0x7fc71c151560 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.668+0000 7fc721ded640 1 --2- 172.21.15.45:0/1872906725 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc70c003b30 0x7fc71c1f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.668+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 >> 172.21.15.45:0/1872906725 conn(0x7fc71c071240 msgr2=0x7fc71c0b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.668+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 shutdown_connections 2023-11-28T20:53:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.668+0000 7fc721ded640 1 -- 172.21.15.45:0/1872906725 wait complete. 2023-11-28T20:53:13.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:53:13.715 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:13.714+0000 7f684642c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:53:13.765 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:13.764+0000 7f684642c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:53:13.795 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:13.793+0000 7fe3bec21040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:53:13.874 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:13.873+0000 7f684642c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:53:13.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.892+0000 7f209992c640 1 Processor -- start 2023-11-28T20:53:13.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.893+0000 7f209992c640 1 -- start start 2023-11-28T20:53:13.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.895+0000 7f209992c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f209415b820 0x7f209406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.895+0000 7f209992c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f209415f4d0 0x7f20940cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.895+0000 7f209992c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20940cf9c0 con 0x7f20940cf020 2023-11-28T20:53:13.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.895+0000 7f209992c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f209415fd60 con 0x7f209415f4d0 2023-11-28T20:53:13.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.895+0000 7f209992c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209415fa80 con 0x7f209415b820 2023-11-28T20:53:13.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.895+0000 7f20937fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f209415f4d0 0x7f20940cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.895+0000 7f2092ffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f209415b820 0x7f209406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.895+0000 7f20937fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f209415f4d0 0x7f20940cf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:36160/0 (socket says 172.21.15.45:36160) 2023-11-28T20:53:13.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.895+0000 7f20937fe640 1 -- 172.21.15.45:0/3089428012 learned_addr learned my addr 172.21.15.45:0/3089428012 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.896+0000 7f20937fe640 1 -- 172.21.15.45:0/3089428012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f209415b820 msgr2=0x7f209406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.896+0000 7f20937fe640 1 --2- 172.21.15.45:0/3089428012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f209415b820 0x7f209406da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.896+0000 7f20937fe640 1 -- 172.21.15.45:0/3089428012 >> v1:172.21.15.190:6789/0 conn(0x7f20940cf020 legacy=0x7f209415edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.896+0000 7f20937fe640 1 -- 172.21.15.45:0/3089428012 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209406cf40 con 0x7f209415f4d0 2023-11-28T20:53:13.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.897+0000 7f20937fe640 1 --2- 172.21.15.45:0/3089428012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415f4d0 0x7f20940cf480 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f208c009e50 tx=0x7f208c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f97f7c305b6a8994 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:13.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f20927fc640 1 -- 172.21.15.45:0/3089428012 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c02cb00 con 0x7f209415f4d0 2023-11-28T20:53:13.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f20927fc640 1 -- 172.21.15.45:0/3089428012 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208c02cc60 con 0x7f209415f4d0 2023-11-28T20:53:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f20927fc640 1 -- 172.21.15.45:0/3089428012 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c0343f0 con 0x7f209415f4d0 2023-11-28T20:53:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f209992c640 1 -- 172.21.15.45:0/3089428012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415f4d0 msgr2=0x7f20940cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f209992c640 1 --2- 172.21.15.45:0/3089428012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415f4d0 0x7f20940cf480 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f208c009e50 tx=0x7f208c02bf60 comp rx=0 tx=0).stop 2023-11-28T20:53:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f209992c640 1 -- 172.21.15.45:0/3089428012 shutdown_connections 2023-11-28T20:53:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f209992c640 1 --2- 172.21.15.45:0/3089428012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415f4d0 0x7f20940cf480 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f209992c640 1 --2- 172.21.15.45:0/3089428012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f209415b820 0x7f209406da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f209992c640 1 -- 172.21.15.45:0/3089428012 >> 172.21.15.45:0/3089428012 conn(0x7f209406a6e0 msgr2=0x7f209406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.898+0000 7f209992c640 1 -- 172.21.15.45:0/3089428012 shutdown_connections 2023-11-28T20:53:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.899+0000 7f209992c640 1 -- 172.21.15.45:0/3089428012 wait complete. 2023-11-28T20:53:13.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.900+0000 7f209992c640 1 Processor -- start 2023-11-28T20:53:13.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.900+0000 7f209992c640 1 -- start start 2023-11-28T20:53:13.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.901+0000 7f209992c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20940cf020 0x7f20940c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.901+0000 7f209992c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415b820 0x7f20941477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.901+0000 7f209992c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209415f4d0 0x7f20940b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.901+0000 7f209992c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209414ffb0 con 0x7f209415f4d0 2023-11-28T20:53:13.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.901+0000 7f209992c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20940c9120 con 0x7f209415b820 2023-11-28T20:53:13.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.901+0000 7f209992c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20940c9400 con 0x7f20940cf020 2023-11-28T20:53:13.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f20937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209415f4d0 0x7f20940b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f20937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209415f4d0 0x7f20940b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38494/0 (socket says 172.21.15.45:38494) 2023-11-28T20:53:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f20937fe640 1 -- 172.21.15.45:0/2349266454 learned_addr learned my addr 172.21.15.45:0/2349266454 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f2092ffd640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20940cf020 0x7f20940c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f208bfff640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415b820 0x7f20941477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:13.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f20937fe640 1 -- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20940cf020 msgr2=0x7f20940c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f20937fe640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20940cf020 0x7f20940c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f20937fe640 1 -- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415b820 msgr2=0x7f20941477f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:13.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f20937fe640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415b820 0x7f20941477f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:13.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f20937fe640 1 -- 172.21.15.45:0/2349266454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c009b30 con 0x7f209415f4d0 2023-11-28T20:53:13.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f208bfff640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415b820 0x7f20941477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:13.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.902+0000 7f2092ffd640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20940cf020 0x7f20940c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.903+0000 7f20937fe640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209415f4d0 0x7f20940b2de0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f208c036040 tx=0x7f208c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.903+0000 7f2090ff9640 1 -- 172.21.15.45:0/2349266454 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c009510 con 0x7f209415f4d0 2023-11-28T20:53:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.903+0000 7f2090ff9640 1 -- 172.21.15.45:0/2349266454 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208c03e450 con 0x7f209415f4d0 2023-11-28T20:53:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.903+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20940c96e0 con 0x7f209415f4d0 2023-11-28T20:53:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.903+0000 7f2090ff9640 1 -- 172.21.15.45:0/2349266454 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c03e820 con 0x7f209415f4d0 2023-11-28T20:53:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.903+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20940c9ca0 con 0x7f209415f4d0 2023-11-28T20:53:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.905+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2094073440 con 0x7f209415f4d0 2023-11-28T20:53:13.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.905+0000 7f2090ff9640 1 -- 172.21.15.45:0/2349266454 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f208c03e980 con 0x7f209415f4d0 2023-11-28T20:53:13.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.906+0000 7f2090ff9640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f207406b080 0x7f207406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:13.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.906+0000 7f2090ff9640 1 -- 172.21.15.45:0/2349266454 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f208c0acb00 con 0x7f209415f4d0 2023-11-28T20:53:13.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.907+0000 7f2092ffd640 1 -- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f207406b080 msgr2=0x7f207406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:13.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.907+0000 7f2092ffd640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f207406b080 0x7f207406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:13.911 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:13.910+0000 7fe3bec21040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:53:13.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:13.911+0000 7f2090ff9640 1 -- 172.21.15.45:0/2349266454 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f208c0aee60 con 0x7f209415f4d0 2023-11-28T20:53:13.982 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:13.980+0000 7fe3bec21040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:53:13.983 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:13.981+0000 7f684642c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:53:14.100 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:14.097+0000 7fe3bec21040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:53:14.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.108+0000 7f2092ffd640 1 -- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f207406b080 msgr2=0x7f207406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:14.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.108+0000 7f2092ffd640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f207406b080 0x7f207406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:53:14.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.132+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f209406c8c0 con 0x7f209415f4d0 2023-11-28T20:53:14.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.133+0000 7f2090ff9640 1 -- 172.21.15.45:0/2349266454 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 9 entries v120) v1 ==== 80+0+537 (secure 0 0 0) 0x7f208c077470 con 0x7f209415f4d0 2023-11-28T20:53:14.135 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/4159844301 2023-11-29T20:52:07.707914+0000 2023-11-28T20:53:14.135 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/4159844301 2023-11-29T20:52:07.707914+0000 2023-11-28T20:53:14.135 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3239477442 2023-11-29T20:52:07.707914+0000 2023-11-28T20:53:14.135 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3809276606 2023-11-29T20:52:07.707914+0000 2023-11-28T20:53:14.135 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3450487150 2023-11-29T20:52:07.707914+0000 2023-11-28T20:53:14.135 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2166104670 2023-11-29T20:52:07.707914+0000 2023-11-28T20:53:14.135 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/4069818189 2023-11-29T20:52:07.707914+0000 2023-11-28T20:53:14.135 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2641549859 2023-11-29T20:52:07.707914+0000 2023-11-28T20:53:14.135 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1724154865 2023-11-28T21:51:59.768228+0000 2023-11-28T20:53:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:listed 9 entries 2023-11-28T20:53:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.135+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f207406b080 msgr2=0x7f207406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.135+0000 7f209992c640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f207406b080 0x7f207406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.135+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209415f4d0 msgr2=0x7f20940b2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.135+0000 7f209992c640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209415f4d0 0x7f20940b2de0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f208c036040 tx=0x7f208c004a40 comp rx=0 tx=0).stop 2023-11-28T20:53:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.135+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 shutdown_connections 2023-11-28T20:53:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.136+0000 7f209992c640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f207406b080 0x7f207406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.136+0000 7f209992c640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f209415f4d0 0x7f20940b2de0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.136+0000 7f209992c640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f209415b820 0x7f20941477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.136+0000 7f209992c640 1 --2- 172.21.15.45:0/2349266454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f20940cf020 0x7f20940c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.136+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 >> 172.21.15.45:0/2349266454 conn(0x7f209406a6e0 msgr2=0x7f209415d060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:14.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.136+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 shutdown_connections 2023-11-28T20:53:14.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.136+0000 7f209992c640 1 -- 172.21.15.45:0/2349266454 wait complete. 2023-11-28T20:53:14.139 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:14.138+0000 7f684642c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:53:14.155 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T20:53:14.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:14.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T20:53:14.188 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:14.187+0000 7f684642c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:53:14.219 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:14.217+0000 7fe3bec21040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:53:14.255 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:53:14.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:14.256 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T20:53:14.302 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:53:14.302 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:14.310 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:14.309+0000 7f684642c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:53:14.356 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:53:14.356 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:53:14.357 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:14.390 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:14.388+0000 7fe3bec21040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:53:14.414 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:53:14.415 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:53:14.415 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:14.442 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:14.440+0000 7fe3bec21040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:53:14.468 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:53:14.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:53:14.571 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:14.569+0000 7fe3bec21040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:53:14.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.686+0000 7f6248683640 1 Processor -- start 2023-11-28T20:53:14.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.686+0000 7f6248683640 1 -- start start 2023-11-28T20:53:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.688+0000 7f6248683640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f624015d250 0x7f62400b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.688+0000 7f6248683640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f62400b58e0 0x7f62400b6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.688+0000 7f6248683640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62400d2240 con 0x7f62400d26d0 2023-11-28T20:53:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.688+0000 7f6248683640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62400d2fc0 con 0x7f62400b58e0 2023-11-28T20:53:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.688+0000 7f6248683640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62400d1630 con 0x7f624015d250 2023-11-28T20:53:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.688+0000 7f6245c1e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f62400d26d0 0x7f62400b51d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37952/0 (socket says 172.21.15.45:37952) 2023-11-28T20:53:14.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.688+0000 7f6245c1e640 1 -- 172.21.15.45:0/384363992 learned_addr learned my addr 172.21.15.45:0/384363992 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.688+0000 7f624641f640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f624015d250 0x7f62400b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:14.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.688+0000 7f6246c20640 1 --2- 172.21.15.45:0/384363992 >> v2:172.21.15.190:3300/0 conn(0x7f62400b58e0 0x7f62400b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380819153 0 0) 0x7f62400d2240 con 0x7f62400d26d0 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f622c003520 con 0x7f62400d26d0 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3886561307 0 0) 0x7f622c003520 con 0x7f62400d26d0 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f624015d250 msgr2=0x7f62400b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f624015d250 0x7f62400b8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 >> v2:172.21.15.190:3300/0 conn(0x7f62400b58e0 msgr2=0x7f62400b6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 --2- 172.21.15.45:0/384363992 >> v2:172.21.15.190:3300/0 conn(0x7f62400b58e0 0x7f62400b6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62400d1910 con 0x7f62400d26d0 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6230001670 con 0x7f62400d26d0 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 >> v1:172.21.15.45:6789/0 conn(0x7f62400d26d0 legacy=0x7f62400b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:14.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c003970 0x7f622c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f624015d250 0x7f622c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c004d00 0x7f622c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62400d2fc0 con 0x7f622c004d00 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62400d1630 con 0x7f622c003970 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.689+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c00b980 con 0x7f624015d250 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.690+0000 7f6246c20640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c004d00 0x7f622c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.690+0000 7f6245c1e640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f624015d250 0x7f622c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.690+0000 7f624641f640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c003970 0x7f622c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.690+0000 7f6246c20640 1 -- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f624015d250 msgr2=0x7f622c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:14.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.690+0000 7f6246c20640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f624015d250 0x7f622c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.690+0000 7f6246c20640 1 -- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c003970 msgr2=0x7f622c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:14.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.690+0000 7f6246c20640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c003970 0x7f622c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.690+0000 7f6246c20640 1 -- 172.21.15.45:0/384363992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c003520 con 0x7f622c004d00 2023-11-28T20:53:14.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.690+0000 7f624641f640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c003970 0x7f622c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:14.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.691+0000 7f6246c20640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c004d00 0x7f622c005180 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f6238009f90 tx=0x7f622c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d5205e272e3091 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:14.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623802cc20 con 0x7f622c004d00 2023-11-28T20:53:14.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623800f040 con 0x7f622c004d00 2023-11-28T20:53:14.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f624541d640 1 -- 172.21.15.45:0/384363992 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6238034440 con 0x7f622c004d00 2023-11-28T20:53:14.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f6248683640 1 -- 172.21.15.45:0/384363992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c004d00 msgr2=0x7f622c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f6248683640 1 --2- 172.21.15.45:0/384363992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c004d00 0x7f622c005180 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f6238009f90 tx=0x7f622c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f6246c20640 1 -- 172.21.15.45:0/384363992 reap_dead start 2023-11-28T20:53:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f6248683640 1 -- 172.21.15.45:0/384363992 shutdown_connections 2023-11-28T20:53:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f6248683640 1 -- 172.21.15.45:0/384363992 >> 172.21.15.45:0/384363992 conn(0x7f6240070f40 msgr2=0x7f6240071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f6248683640 1 -- 172.21.15.45:0/384363992 shutdown_connections 2023-11-28T20:53:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.692+0000 7f6248683640 1 -- 172.21.15.45:0/384363992 wait complete. 2023-11-28T20:53:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.694+0000 7f6248683640 1 Processor -- start 2023-11-28T20:53:14.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.694+0000 7f6248683640 1 -- start start 2023-11-28T20:53:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.695+0000 7f6248683640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f622c004d00 0x7f62400c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.695+0000 7f6248683640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f624015f0d0 0x7f624015f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.695+0000 7f6248683640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62400ca2c0 0x7f62400ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:14.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.695+0000 7f6248683640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6240151480 con 0x7f624015f0d0 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.695+0000 7f6248683640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62400cac80 con 0x7f62400ca2c0 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.695+0000 7f6248683640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62400c5df0 con 0x7f622c004d00 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.695+0000 7f6245c1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f624015f0d0 0x7f624015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.695+0000 7f6245c1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f624015f0d0 0x7f624015f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38514/0 (socket says 172.21.15.45:38514) 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.695+0000 7f6245c1e640 1 -- 172.21.15.45:0/242705675 learned_addr learned my addr 172.21.15.45:0/242705675 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f624641f640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f622c004d00 0x7f62400c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f6246c20640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62400ca2c0 0x7f62400ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f6245c1e640 1 -- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f622c004d00 msgr2=0x7f62400c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f6245c1e640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f622c004d00 0x7f62400c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f6245c1e640 1 -- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62400ca2c0 msgr2=0x7f62400ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f6245c1e640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62400ca2c0 0x7f62400ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f6245c1e640 1 -- 172.21.15.45:0/242705675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624015fbe0 con 0x7f624015f0d0 2023-11-28T20:53:14.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f6246c20640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62400ca2c0 0x7f62400ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f6245c1e640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f624015f0d0 0x7f624015f550 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f623000e7b0 tx=0x7f623000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.696+0000 7f6236ffd640 1 -- 172.21.15.45:0/242705675 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623000c870 con 0x7f624015f0d0 2023-11-28T20:53:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.697+0000 7f6236ffd640 1 -- 172.21.15.45:0/242705675 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623000c9d0 con 0x7f624015f0d0 2023-11-28T20:53:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.697+0000 7f6236ffd640 1 -- 172.21.15.45:0/242705675 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623000cd80 con 0x7f624015f0d0 2023-11-28T20:53:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.697+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624015fec0 con 0x7f624015f0d0 2023-11-28T20:53:14.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.697+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6240151970 con 0x7f624015f0d0 2023-11-28T20:53:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.698+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62400d0640 con 0x7f624015f0d0 2023-11-28T20:53:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.699+0000 7f6236ffd640 1 -- 172.21.15.45:0/242705675 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f623001d460 con 0x7f624015f0d0 2023-11-28T20:53:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.700+0000 7f6236ffd640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f622406b0b0 0x7f622406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.700+0000 7f6236ffd640 1 -- 172.21.15.45:0/242705675 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6230089be0 con 0x7f624015f0d0 2023-11-28T20:53:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.700+0000 7f624641f640 1 -- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f622406b0b0 msgr2=0x7f622406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:14.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.700+0000 7f624641f640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f622406b0b0 0x7f622406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:14.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.705+0000 7f6236ffd640 1 -- 172.21.15.45:0/242705675 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6230054750 con 0x7f624015f0d0 2023-11-28T20:53:14.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.901+0000 7f624641f640 1 -- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f622406b0b0 msgr2=0x7f622406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:14.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.901+0000 7f624641f640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f622406b0b0 0x7f622406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:53:14.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.928+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62401600b0 con 0x7f624015f0d0 2023-11-28T20:53:14.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.930+0000 7f6236ffd640 1 -- 172.21.15.45:0/242705675 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f62401600b0 con 0x7f624015f0d0 2023-11-28T20:53:14.932 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:14.932 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2348343744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2348343744},{"type":"v1","addr":"172.21.15.190:6836","nonce":2348343744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6439,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1864791063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":1864791063},{"type":"v1","addr":"172.21.15.190:6837","nonce":1864791063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3003510210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3003510210},{"type":"v1","addr":"172.21.15.45:6835","nonce":3003510210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:52.969776+0000","modified":"2023-11-28T20:51:56.055296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6448":{"gid":6448,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2489162496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2489162496},{"type":"v1","addr":"172.21.15.45:6834","nonce":2489162496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-28T20:53:14.932 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 51 2023-11-28T20:53:14.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f622406b0b0 msgr2=0x7f622406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:14.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f622406b0b0 0x7f622406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f624015f0d0 msgr2=0x7f624015f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f624015f0d0 0x7f624015f550 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f623000e7b0 tx=0x7f623000ec70 comp rx=0 tx=0).stop 2023-11-28T20:53:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 shutdown_connections 2023-11-28T20:53:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f622406b0b0 0x7f622406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62400ca2c0 0x7f62400ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f624015f0d0 0x7f624015f550 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 --2- 172.21.15.45:0/242705675 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f622c004d00 0x7f62400c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 >> 172.21.15.45:0/242705675 conn(0x7f6240070f40 msgr2=0x7f62400c79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 shutdown_connections 2023-11-28T20:53:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:14.932+0000 7f6248683640 1 -- 172.21.15.45:0/242705675 wait complete. 2023-11-28T20:53:14.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:53:15.142 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.141+0000 7f684642c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:53:15.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.149+0000 7f5002e94640 1 Processor -- start 2023-11-28T20:53:15.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.149+0000 7f5002e94640 1 -- start start 2023-11-28T20:53:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.150+0000 7f5002e94640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4ffc15bd10 0x7f4ffc15c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.150+0000 7f5002e94640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc0701d0 0x7f4ffc13f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.150+0000 7f5002e94640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc070660 con 0x7f4ffc06fd60 2023-11-28T20:53:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.150+0000 7f5002e94640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc15c630 con 0x7f4ffc15bd10 2023-11-28T20:53:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.150+0000 7f5002e94640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc13e540 con 0x7f4ffc0701d0 2023-11-28T20:53:15.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.150+0000 7f5000c30640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4ffc15bd10 0x7f4ffc15c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.151+0000 7f5000c30640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4ffc15bd10 0x7f4ffc15c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38530/0 (socket says 172.21.15.45:38530) 2023-11-28T20:53:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.151+0000 7f5000c30640 1 -- 172.21.15.45:0/3392683558 learned_addr learned my addr 172.21.15.45:0/3392683558 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.151+0000 7f5001431640 1 --2- 172.21.15.45:0/3392683558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc0701d0 0x7f4ffc13f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.151+0000 7f5000c30640 1 -- 172.21.15.45:0/3392683558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc0701d0 msgr2=0x7f4ffc13f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.151+0000 7f5000c30640 1 --2- 172.21.15.45:0/3392683558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc0701d0 0x7f4ffc13f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.151+0000 7f5000c30640 1 -- 172.21.15.45:0/3392683558 >> v1:172.21.15.190:6789/0 conn(0x7f4ffc06fd60 legacy=0x7f4ffc140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.151+0000 7f5000c30640 1 -- 172.21.15.45:0/3392683558 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc070940 con 0x7f4ffc15bd10 2023-11-28T20:53:15.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.151+0000 7f5001431640 1 --2- 172.21.15.45:0/3392683558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc0701d0 0x7f4ffc13f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:15.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.152+0000 7f5000c30640 1 --2- 172.21.15.45:0/3392683558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc15bd10 0x7f4ffc15c0f0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00098a0 tx=0x7f4ff002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6210c84cf0254f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.152+0000 7f4ffb7fe640 1 -- 172.21.15.45:0/3392683558 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0031640 con 0x7f4ffc15bd10 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f4ffb7fe640 1 -- 172.21.15.45:0/3392683558 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff00317a0 con 0x7f4ffc15bd10 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f4ffb7fe640 1 -- 172.21.15.45:0/3392683558 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0031b30 con 0x7f4ffc15bd10 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f5002e94640 1 -- 172.21.15.45:0/3392683558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc15bd10 msgr2=0x7f4ffc15c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f5002e94640 1 --2- 172.21.15.45:0/3392683558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc15bd10 0x7f4ffc15c0f0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00098a0 tx=0x7f4ff002b8f0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f5002e94640 1 -- 172.21.15.45:0/3392683558 shutdown_connections 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f5002e94640 1 --2- 172.21.15.45:0/3392683558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc0701d0 0x7f4ffc13f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f5002e94640 1 --2- 172.21.15.45:0/3392683558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc15bd10 0x7f4ffc15c0f0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f5002e94640 1 -- 172.21.15.45:0/3392683558 >> 172.21.15.45:0/3392683558 conn(0x7f4ffc06e7c0 msgr2=0x7f4ffc06ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f5002e94640 1 -- 172.21.15.45:0/3392683558 shutdown_connections 2023-11-28T20:53:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.153+0000 7f5002e94640 1 -- 172.21.15.45:0/3392683558 wait complete. 2023-11-28T20:53:15.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.156+0000 7f5002e94640 1 Processor -- start 2023-11-28T20:53:15.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.156+0000 7f5002e94640 1 -- start start 2023-11-28T20:53:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.158+0000 7f5002e94640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc06fd60 0x7f4ffc1f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.158+0000 7f5002e94640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc15bd10 0x7f4ffc1f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.158+0000 7f5002e94640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ffc1503f0 0x7f4ffc15d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.158+0000 7f5002e94640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc1f4150 con 0x7f4ffc06fd60 2023-11-28T20:53:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.158+0000 7f5002e94640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc1557f0 con 0x7f4ffc1503f0 2023-11-28T20:53:15.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.158+0000 7f5002e94640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc155ad0 con 0x7f4ffc15bd10 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.158+0000 7f5000c30640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc06fd60 0x7f4ffc1f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.158+0000 7f5000c30640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc06fd60 0x7f4ffc1f72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38538/0 (socket says 172.21.15.45:38538) 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.158+0000 7f5000c30640 1 -- 172.21.15.45:0/1463651271 learned_addr learned my addr 172.21.15.45:0/1463651271 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f5001431640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ffc1503f0 0x7f4ffc15d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f5000c30640 1 -- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc15bd10 msgr2=0x7f4ffc1f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f4ffbfff640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc15bd10 0x7f4ffc1f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f5000c30640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc15bd10 0x7f4ffc1f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f5000c30640 1 -- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ffc1503f0 msgr2=0x7f4ffc15d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f5000c30640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ffc1503f0 0x7f4ffc15d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f5000c30640 1 -- 172.21.15.45:0/1463651271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00094d0 con 0x7f4ffc06fd60 2023-11-28T20:53:15.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f5001431640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ffc1503f0 0x7f4ffc15d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:15.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f4ffbfff640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc15bd10 0x7f4ffc1f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:15.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f5000c30640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc06fd60 0x7f4ffc1f72c0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0000c00 tx=0x7f4ff00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:15.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.159+0000 7f4ff97fa640 1 -- 172.21.15.45:0/1463651271 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff003ac60 con 0x7f4ffc06fd60 2023-11-28T20:53:15.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.160+0000 7f4ff97fa640 1 -- 172.21.15.45:0/1463651271 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff003adc0 con 0x7f4ffc06fd60 2023-11-28T20:53:15.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.160+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc1f67e0 con 0x7f4ffc06fd60 2023-11-28T20:53:15.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.160+0000 7f4ff97fa640 1 -- 172.21.15.45:0/1463651271 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0044610 con 0x7f4ffc06fd60 2023-11-28T20:53:15.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.160+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc1f6da0 con 0x7f4ffc06fd60 2023-11-28T20:53:15.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.161+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc13e820 con 0x7f4ffc06fd60 2023-11-28T20:53:15.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.162+0000 7f4ff97fa640 1 -- 172.21.15.45:0/1463651271 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f4ff002f050 con 0x7f4ffc06fd60 2023-11-28T20:53:15.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.163+0000 7f4ff97fa640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f4fe006b080 0x7f4fe006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.163+0000 7f4ff97fa640 1 -- 172.21.15.45:0/1463651271 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4ff00abf70 con 0x7f4ffc06fd60 2023-11-28T20:53:15.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.163+0000 7f4ffbfff640 1 -- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f4fe006b080 msgr2=0x7f4fe006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:15.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.163+0000 7f4ffbfff640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f4fe006b080 0x7f4fe006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.167+0000 7f4ff97fa640 1 -- 172.21.15.45:0/1463651271 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ff00767d0 con 0x7f4ffc06fd60 2023-11-28T20:53:15.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.359+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ffc1f6f90 con 0x7f4ffc06fd60 2023-11-28T20:53:15.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.362+0000 7f4ff97fa640 1 -- 172.21.15.45:0/1463651271 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+17482 (secure 0 0 0) 0x7f4ff0035090 con 0x7f4ffc06fd60 2023-11-28T20:53:15.364 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:15.365 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":120,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:09.443025+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:50.993341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:52.004867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":7}]},{"pgid":"14.2a","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6833/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:6832/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3239477442":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3809276606":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3450487150":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2166104670":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/4069818189":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2641549859":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/1724154865":"2023-11-28T21:51:59.768228+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:53:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.364+0000 7f4ffbfff640 1 -- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f4fe006b080 msgr2=0x7f4fe006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:15.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.364+0000 7f4ffbfff640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f4fe006b080 0x7f4fe006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:53:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f4fe006b080 msgr2=0x7f4fe006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:15.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f4fe006b080 0x7f4fe006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc06fd60 msgr2=0x7f4ffc1f72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc06fd60 0x7f4ffc1f72c0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0000c00 tx=0x7f4ff00046c0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 shutdown_connections 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f4fe006b080 0x7f4fe006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ffc1503f0 0x7f4ffc15d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ffc15bd10 0x7f4ffc1f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 --2- 172.21.15.45:0/1463651271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc06fd60 0x7f4ffc1f72c0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 >> 172.21.15.45:0/1463651271 conn(0x7f4ffc06e7c0 msgr2=0x7f4ffc15c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 shutdown_connections 2023-11-28T20:53:15.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.365+0000 7f5002e94640 1 -- 172.21.15.45:0/1463651271 wait complete. 2023-11-28T20:53:15.385 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:51:50.993341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:51:52.004867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:53:15.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:53:15.420 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:15.418+0000 7fe3bec21040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:53:15.437 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.436+0000 7f684642c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:53:15.497 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.496+0000 7f684642c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:53:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.577+0000 7f90a0e61640 1 Processor -- start 2023-11-28T20:53:15.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.578+0000 7f90a0e61640 1 -- start start 2023-11-28T20:53:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.579+0000 7f90a0e61640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c06c350 0x7f909c15f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.579+0000 7f90a0e61640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c15ecd0 con 0x7f909c15e840 2023-11-28T20:53:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.579+0000 7f90a0e61640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c15f290 con 0x7f909c06ac10 2023-11-28T20:53:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.579+0000 7f90a0e61640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c072020 con 0x7f909c06c350 2023-11-28T20:53:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.579+0000 7f909a59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f909c15e840 0x7f909c06a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37964/0 (socket says 172.21.15.45:37964) 2023-11-28T20:53:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.579+0000 7f909a59c640 1 -- 172.21.15.45:0/274424071 learned_addr learned my addr 172.21.15.45:0/274424071 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909ad9d640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c06c350 0x7f909c15f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042307505 0 0) 0x7f909c15ecd0 con 0x7f909c15e840 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9088003520 con 0x7f909c15e840 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1200391793 0 0) 0x7f9088003520 con 0x7f909c15e840 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c06c350 msgr2=0x7f909c15f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c06c350 0x7f909c15f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 >> v1:172.21.15.190:6789/0 conn(0x7f909c06ac10 legacy=0x7f909c0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c072300 con 0x7f909c15e840 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f908c0019f0 con 0x7f909c15e840 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 >> v1:172.21.15.45:6789/0 conn(0x7f909c15e840 legacy=0x7f909c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.580+0000 7f909959a640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90880039a0 0x7f9088003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909959a640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c06c350 0x7f90880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909959a640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9088004d00 0x7f9088005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c072020 con 0x7f9088004d00 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908800ba90 con 0x7f90880039a0 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908800bc50 con 0x7f909c06c350 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909ad9d640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9088004d00 0x7f9088005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909a59c640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90880039a0 0x7f9088003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f9099d9b640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c06c350 0x7f90880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909ad9d640 1 -- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c06c350 msgr2=0x7f90880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909ad9d640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c06c350 0x7f90880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909ad9d640 1 -- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90880039a0 msgr2=0x7f9088003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909ad9d640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90880039a0 0x7f9088003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.581+0000 7f909ad9d640 1 -- 172.21.15.45:0/274424071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9088003520 con 0x7f9088004d00 2023-11-28T20:53:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.582+0000 7f909ad9d640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9088004d00 0x7f9088005180 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f9094009a90 tx=0x7f90880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74abcbaba2e361fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.583+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909402c680 con 0x7f9088004d00 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.583+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909402c7e0 con 0x7f9088004d00 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.583+0000 7f909959a640 1 -- 172.21.15.45:0/274424071 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909402cb70 con 0x7f9088004d00 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.583+0000 7f90a0e61640 1 -- 172.21.15.45:0/274424071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9088004d00 msgr2=0x7f9088005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.583+0000 7f90a0e61640 1 --2- 172.21.15.45:0/274424071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9088004d00 0x7f9088005180 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f9094009a90 tx=0x7f90880057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.583+0000 7f909ad9d640 1 -- 172.21.15.45:0/274424071 reap_dead start 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.583+0000 7f90a0e61640 1 -- 172.21.15.45:0/274424071 shutdown_connections 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.583+0000 7f90a0e61640 1 -- 172.21.15.45:0/274424071 >> 172.21.15.45:0/274424071 conn(0x7f909c0ced60 msgr2=0x7f909c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.583+0000 7f90a0e61640 1 -- 172.21.15.45:0/274424071 shutdown_connections 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.584+0000 7f90a0e61640 1 -- 172.21.15.45:0/274424071 wait complete. 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.585+0000 7f90a0e61640 1 Processor -- start 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.585+0000 7f90a0e61640 1 -- start start 2023-11-28T20:53:15.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.586+0000 7f90a0e61640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9088004d00 0x7f909c0cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.586+0000 7f90a0e61640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c155fb0 0x7f909c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.586+0000 7f90a0e61640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c070600 0x7f909c09bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.586+0000 7f90a0e61640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c070a80 con 0x7f909c155fb0 2023-11-28T20:53:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.586+0000 7f90a0e61640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c099a40 con 0x7f9088004d00 2023-11-28T20:53:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.586+0000 7f90a0e61640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c0c4160 con 0x7f909c070600 2023-11-28T20:53:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9099d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c155fb0 0x7f909c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9099d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c155fb0 0x7f909c0700c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38548/0 (socket says 172.21.15.45:38548) 2023-11-28T20:53:15.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9099d9b640 1 -- 172.21.15.45:0/112519007 learned_addr learned my addr 172.21.15.45:0/112519007 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:15.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f909ad9d640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c070600 0x7f909c09bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f909a59c640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9088004d00 0x7f909c0cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9099d9b640 1 -- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c070600 msgr2=0x7f909c09bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9099d9b640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c070600 0x7f909c09bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9099d9b640 1 -- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9088004d00 msgr2=0x7f909c0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9099d9b640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9088004d00 0x7f909c0cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9099d9b640 1 -- 172.21.15.45:0/112519007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c0b2e00 con 0x7f909c155fb0 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f909a59c640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9088004d00 0x7f909c0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9099d9b640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c155fb0 0x7f909c0700c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f908400e4c0 tx=0x7f908400e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.587+0000 7f9092ffd640 1 -- 172.21.15.45:0/112519007 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90840052e0 con 0x7f909c155fb0 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.588+0000 7f9092ffd640 1 -- 172.21.15.45:0/112519007 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084005440 con 0x7f909c155fb0 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.588+0000 7f9092ffd640 1 -- 172.21.15.45:0/112519007 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084003830 con 0x7f909c155fb0 2023-11-28T20:53:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.588+0000 7f90a0e61640 1 -- 172.21.15.45:0/112519007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c0b30e0 con 0x7f909c155fb0 2023-11-28T20:53:15.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.588+0000 7f90a0e61640 1 -- 172.21.15.45:0/112519007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c0b36a0 con 0x7f909c155fb0 2023-11-28T20:53:15.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.589+0000 7f90a0e61640 1 -- 172.21.15.45:0/112519007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909c0ad640 con 0x7f909c155fb0 2023-11-28T20:53:15.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.590+0000 7f9092ffd640 1 -- 172.21.15.45:0/112519007 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 53) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f9084003990 con 0x7f909c155fb0 2023-11-28T20:53:15.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.591+0000 7f9092ffd640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f907c06b0b0 0x7f907c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:15.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.591+0000 7f9092ffd640 1 -- 172.21.15.45:0/112519007 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9084089a40 con 0x7f909c155fb0 2023-11-28T20:53:15.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.591+0000 7f909a59c640 1 -- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f907c06b0b0 msgr2=0x7f907c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1902260958 2023-11-28T20:53:15.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.591+0000 7f909a59c640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f907c06b0b0 0x7f907c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:15.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.595+0000 7f9092ffd640 1 -- 172.21.15.45:0/112519007 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f908408be60 con 0x7f909c155fb0 2023-11-28T20:53:15.612 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.611+0000 7f684642c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:53:15.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.624+0000 7f9092ffd640 1 -- 172.21.15.45:0/112519007 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 54) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f9084010db0 con 0x7f909c155fb0 2023-11-28T20:53:15.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.624+0000 7f9092ffd640 1 -- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f907c06b0b0 msgr2=0x7f907c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:15.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.624+0000 7f9092ffd640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f907c06b0b0 0x7f907c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.698 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.697+0000 7f67c778b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.699 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.697+0000 7f67c778b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.699 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.697+0000 7f67c778b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.699 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.697+0000 7f67c778b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.699 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.697+0000 7f67c778b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.699 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.698+0000 7f67caf92640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.699 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.698+0000 7f67caf92640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.700 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.698+0000 7f67caf92640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.700 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.698+0000 7f67caf92640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.700 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:15.698+0000 7f67caf92640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:15.753 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:15.750+0000 7fe3bec21040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:53:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.792+0000 7f90a0e61640 1 -- 172.21.15.45:0/112519007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f909c0733d0 con 0x7f909c155fb0 2023-11-28T20:53:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.793+0000 7f9092ffd640 1 -- 172.21.15.45:0/112519007 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7f90840744d0 con 0x7f909c155fb0 2023-11-28T20:53:15.795 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:53:15.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.795+0000 7f90a0e61640 1 -- 172.21.15.45:0/112519007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c155fb0 msgr2=0x7f909c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:15.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.795+0000 7f90a0e61640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c155fb0 0x7f909c0700c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f908400e4c0 tx=0x7f908400e980 comp rx=0 tx=0).stop 2023-11-28T20:53:15.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.795+0000 7f90a0e61640 1 -- 172.21.15.45:0/112519007 shutdown_connections 2023-11-28T20:53:15.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.795+0000 7f90a0e61640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:6832/1902260958,v1:172.21.15.190:6833/1902260958] conn(0x7f907c06b0b0 0x7f907c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.795+0000 7f90a0e61640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f909c070600 0x7f909c09bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.795+0000 7f90a0e61640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c155fb0 0x7f909c0700c0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.795+0000 7f90a0e61640 1 --2- 172.21.15.45:0/112519007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9088004d00 0x7f909c0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:15.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.795+0000 7f90a0e61640 1 -- 172.21.15.45:0/112519007 >> 172.21.15.45:0/112519007 conn(0x7f909c0ced60 msgr2=0x7f909c06a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:15.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.796+0000 7f90a0e61640 1 -- 172.21.15.45:0/112519007 shutdown_connections 2023-11-28T20:53:15.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:15.796+0000 7f90a0e61640 1 -- 172.21.15.45:0/112519007 wait complete. 2023-11-28T20:53:15.814 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:15.812+0000 7fe3bec21040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:53:15.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:53:15.934 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:15.932+0000 7fe3bec21040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:53:16.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.077+0000 7f657b13c640 1 Processor -- start 2023-11-28T20:53:16.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.077+0000 7f657b13c640 1 -- start start 2023-11-28T20:53:16.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.078+0000 7f657b13c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6574157c10 0x7f657406da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f657b13c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f657406a500 0x7f657406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f657b13c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657406ca60 con 0x7f657415ddd0 2023-11-28T20:53:16.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f657b13c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657415e930 con 0x7f6574157c10 2023-11-28T20:53:16.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f657b13c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657415eb40 con 0x7f657406a500 2023-11-28T20:53:16.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f6578ed8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6574157c10 0x7f657406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f6578ed8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6574157c10 0x7f657406da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38564/0 (socket says 172.21.15.45:38564) 2023-11-28T20:53:16.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f6578ed8640 1 -- 172.21.15.45:0/228172986 learned_addr learned my addr 172.21.15.45:0/228172986 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:16.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f6578ed8640 1 -- 172.21.15.45:0/228172986 >> v2:172.21.15.190:3300/0 conn(0x7f657406a500 msgr2=0x7f657406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:16.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f6578ed8640 1 --2- 172.21.15.45:0/228172986 >> v2:172.21.15.190:3300/0 conn(0x7f657406a500 0x7f657406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f6578ed8640 1 -- 172.21.15.45:0/228172986 >> v1:172.21.15.190:6789/0 conn(0x7f657415ddd0 legacy=0x7f657415e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:16.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.079+0000 7f6578ed8640 1 -- 172.21.15.45:0/228172986 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657406aea0 con 0x7f6574157c10 2023-11-28T20:53:16.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.081+0000 7f6578ed8640 1 --2- 172.21.15.45:0/228172986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 0x7f657406da50 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f65740cfe40 tx=0x7f657002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54be020a39f9540f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:16.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.081+0000 7f656b7fe640 1 -- 172.21.15.45:0/228172986 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570031870 con 0x7f6574157c10 2023-11-28T20:53:16.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.081+0000 7f656b7fe640 1 -- 172.21.15.45:0/228172986 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65700319d0 con 0x7f6574157c10 2023-11-28T20:53:16.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.081+0000 7f656b7fe640 1 -- 172.21.15.45:0/228172986 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570031d60 con 0x7f6574157c10 2023-11-28T20:53:16.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.082+0000 7f657b13c640 1 -- 172.21.15.45:0/228172986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 msgr2=0x7f657406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.082+0000 7f657b13c640 1 --2- 172.21.15.45:0/228172986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 0x7f657406da50 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f65740cfe40 tx=0x7f657002b910 comp rx=0 tx=0).stop 2023-11-28T20:53:16.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.082+0000 7f657b13c640 1 -- 172.21.15.45:0/228172986 shutdown_connections 2023-11-28T20:53:16.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.082+0000 7f657b13c640 1 --2- 172.21.15.45:0/228172986 >> v2:172.21.15.190:3300/0 conn(0x7f657406a500 0x7f657406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.082+0000 7f657b13c640 1 --2- 172.21.15.45:0/228172986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 0x7f657406da50 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.082+0000 7f657b13c640 1 -- 172.21.15.45:0/228172986 >> 172.21.15.45:0/228172986 conn(0x7f65740ced60 msgr2=0x7f65740cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:16.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.082+0000 7f657b13c640 1 -- 172.21.15.45:0/228172986 shutdown_connections 2023-11-28T20:53:16.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.082+0000 7f657b13c640 1 -- 172.21.15.45:0/228172986 wait complete. 2023-11-28T20:53:16.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.084+0000 7f657b13c640 1 Processor -- start 2023-11-28T20:53:16.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.085+0000 7f657b13c640 1 -- start start 2023-11-28T20:53:16.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.086+0000 7f657b13c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f657406a500 0x7f65740c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.086+0000 7f657b13c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 0x7f6574150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.086+0000 7f657b13c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657415ddd0 0x7f6574151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.086+0000 7f657b13c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65740c8360 con 0x7f6574157c10 2023-11-28T20:53:16.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.086+0000 7f657b13c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65740c8590 con 0x7f657406a500 2023-11-28T20:53:16.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.086+0000 7f657b13c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65740cc910 con 0x7f657415ddd0 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.086+0000 7f656bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 0x7f6574150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.086+0000 7f656bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 0x7f6574150f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38578/0 (socket says 172.21.15.45:38578) 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.086+0000 7f656bfff640 1 -- 172.21.15.45:0/1674744372 learned_addr learned my addr 172.21.15.45:0/1674744372 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.087+0000 7f6578ed8640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f657406a500 0x7f65740c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.087+0000 7f65796d9640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657415ddd0 0x7f6574151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.087+0000 7f656bfff640 1 -- 172.21.15.45:0/1674744372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657415ddd0 msgr2=0x7f6574151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.087+0000 7f656bfff640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657415ddd0 0x7f6574151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.087+0000 7f656bfff640 1 -- 172.21.15.45:0/1674744372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f657406a500 msgr2=0x7f65740c6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.087+0000 7f656bfff640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f657406a500 0x7f65740c6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.087+0000 7f656bfff640 1 -- 172.21.15.45:0/1674744372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65700094d0 con 0x7f6574157c10 2023-11-28T20:53:16.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.087+0000 7f65796d9640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657415ddd0 0x7f6574151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:16.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.087+0000 7f656bfff640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 0x7f6574150f10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f656400b410 tx=0x7f656400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:16.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.088+0000 7f65697fa640 1 -- 172.21.15.45:0/1674744372 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564011830 con 0x7f6574157c10 2023-11-28T20:53:16.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.088+0000 7f65697fa640 1 -- 172.21.15.45:0/1674744372 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65640119b0 con 0x7f6574157c10 2023-11-28T20:53:16.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.088+0000 7f65697fa640 1 -- 172.21.15.45:0/1674744372 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564011d60 con 0x7f6574157c10 2023-11-28T20:53:16.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.088+0000 7f657b13c640 1 -- 172.21.15.45:0/1674744372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65740c8830 con 0x7f6574157c10 2023-11-28T20:53:16.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.088+0000 7f657b13c640 1 -- 172.21.15.45:0/1674744372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65740ccbb0 con 0x7f6574157c10 2023-11-28T20:53:16.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.089+0000 7f657b13c640 1 -- 172.21.15.45:0/1674744372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65740ccda0 con 0x7f6574157c10 2023-11-28T20:53:16.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.091+0000 7f65697fa640 1 -- 172.21.15.45:0/1674744372 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 54) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f656401a400 con 0x7f6574157c10 2023-11-28T20:53:16.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.092+0000 7f65697fa640 1 -- 172.21.15.45:0/1674744372 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6962+0+0 (secure 0 0 0) 0x7f6564089770 con 0x7f6574157c10 2023-11-28T20:53:16.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.097+0000 7f65697fa640 1 -- 172.21.15.45:0/1674744372 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65640546d0 con 0x7f6574157c10 2023-11-28T20:53:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.290+0000 7f657b13c640 1 -- 172.21.15.45:0/1674744372 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f65740733d0 con 0x7f6574157c10 2023-11-28T20:53:16.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.291+0000 7f65697fa640 1 -- 172.21.15.45:0/1674744372 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6564015020 con 0x7f6574157c10 2023-11-28T20:53:16.293 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:53:16.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.293+0000 7f657b13c640 1 -- 172.21.15.45:0/1674744372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 msgr2=0x7f6574150f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.293+0000 7f657b13c640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 0x7f6574150f10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f656400b410 tx=0x7f656400b8d0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.293+0000 7f657b13c640 1 -- 172.21.15.45:0/1674744372 shutdown_connections 2023-11-28T20:53:16.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.293+0000 7f657b13c640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657415ddd0 0x7f6574151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.293+0000 7f657b13c640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6574157c10 0x7f6574150f10 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.293+0000 7f657b13c640 1 --2- 172.21.15.45:0/1674744372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f657406a500 0x7f65740c6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.293+0000 7f657b13c640 1 -- 172.21.15.45:0/1674744372 >> 172.21.15.45:0/1674744372 conn(0x7f65740ced60 msgr2=0x7f65740aeff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:16.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.293+0000 7f657b13c640 1 -- 172.21.15.45:0/1674744372 shutdown_connections 2023-11-28T20:53:16.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.294+0000 7f657b13c640 1 -- 172.21.15.45:0/1674744372 wait complete. 2023-11-28T20:53:16.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:53:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.542+0000 7f4446a94640 1 Processor -- start 2023-11-28T20:53:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.542+0000 7f4446a94640 1 -- start start 2023-11-28T20:53:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.543+0000 7f4446a94640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4440099a40 0x7f444006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.544+0000 7f4446a94640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f44400cbd30 0x7f44400cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.544+0000 7f4446a94640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44400d1fc0 con 0x7f44400aff90 2023-11-28T20:53:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.544+0000 7f4446a94640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f444006a910 con 0x7f44400cbd30 2023-11-28T20:53:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.544+0000 7f4446a94640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44400cb8a0 con 0x7f4440099a40 2023-11-28T20:53:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.544+0000 7f4444830640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4440099a40 0x7f444006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.544+0000 7f4444830640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4440099a40 0x7f444006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:59092/0 (socket says 172.21.15.45:59092) 2023-11-28T20:53:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.544+0000 7f4444830640 1 -- 172.21.15.45:0/358591248 learned_addr learned my addr 172.21.15.45:0/358591248 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.544+0000 7f4445031640 1 --2- 172.21.15.45:0/358591248 >> v2:172.21.15.190:3300/0 conn(0x7f44400cbd30 0x7f44400cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.545+0000 7f4444830640 1 -- 172.21.15.45:0/358591248 >> v2:172.21.15.190:3300/0 conn(0x7f44400cbd30 msgr2=0x7f44400cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.545+0000 7f4444830640 1 --2- 172.21.15.45:0/358591248 >> v2:172.21.15.190:3300/0 conn(0x7f44400cbd30 0x7f44400cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.545+0000 7f4444830640 1 -- 172.21.15.45:0/358591248 >> v1:172.21.15.190:6789/0 conn(0x7f44400aff90 legacy=0x7f4440099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.545+0000 7f4444830640 1 -- 172.21.15.45:0/358591248 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444006a630 con 0x7f4440099a40 2023-11-28T20:53:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.546+0000 7f4444830640 1 --2- 172.21.15.45:0/358591248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4440099a40 0x7f444006b4d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f443c009e80 tx=0x7f443c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39ae03f5b854c593 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f44377fe640 1 -- 172.21.15.45:0/358591248 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c02ca50 con 0x7f4440099a40 2023-11-28T20:53:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f44377fe640 1 -- 172.21.15.45:0/358591248 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c02cbb0 con 0x7f4440099a40 2023-11-28T20:53:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f44377fe640 1 -- 172.21.15.45:0/358591248 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0343f0 con 0x7f4440099a40 2023-11-28T20:53:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f4446a94640 1 -- 172.21.15.45:0/358591248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4440099a40 msgr2=0x7f444006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f4446a94640 1 --2- 172.21.15.45:0/358591248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4440099a40 0x7f444006b4d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f443c009e80 tx=0x7f443c02beb0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f4446a94640 1 -- 172.21.15.45:0/358591248 shutdown_connections 2023-11-28T20:53:16.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f4446a94640 1 --2- 172.21.15.45:0/358591248 >> v2:172.21.15.190:3300/0 conn(0x7f44400cbd30 0x7f44400cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f4446a94640 1 --2- 172.21.15.45:0/358591248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4440099a40 0x7f444006b4d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f4446a94640 1 -- 172.21.15.45:0/358591248 >> 172.21.15.45:0/358591248 conn(0x7f44400d1020 msgr2=0x7f44400d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:16.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f4446a94640 1 -- 172.21.15.45:0/358591248 shutdown_connections 2023-11-28T20:53:16.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.547+0000 7f4446a94640 1 -- 172.21.15.45:0/358591248 wait complete. 2023-11-28T20:53:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.549+0000 7f4446a94640 1 Processor -- start 2023-11-28T20:53:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.549+0000 7f4446a94640 1 -- start start 2023-11-28T20:53:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.550+0000 7f4446a94640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4440099a40 0x7f44401f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.550+0000 7f4446a94640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44400aff90 0x7f44401f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.550+0000 7f4446a94640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44400cbd30 0x7f44400b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.550+0000 7f4446a94640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44400b6140 con 0x7f4440099a40 2023-11-28T20:53:16.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.550+0000 7f4446a94640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44401f3ec0 con 0x7f44400cbd30 2023-11-28T20:53:16.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.550+0000 7f4446a94640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44401f3620 con 0x7f44400aff90 2023-11-28T20:53:16.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.550+0000 7f4444830640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4440099a40 0x7f44401f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.550+0000 7f4444830640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4440099a40 0x7f44401f40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38584/0 (socket says 172.21.15.45:38584) 2023-11-28T20:53:16.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.550+0000 7f4444830640 1 -- 172.21.15.45:0/3561579894 learned_addr learned my addr 172.21.15.45:0/3561579894 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:16.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4444830640 1 -- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44400aff90 msgr2=0x7f44401f2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4437fff640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44400aff90 0x7f44401f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4445031640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44400cbd30 0x7f44400b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4444830640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44400aff90 0x7f44401f2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4444830640 1 -- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44400cbd30 msgr2=0x7f44400b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4444830640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44400cbd30 0x7f44400b5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4444830640 1 -- 172.21.15.45:0/3561579894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c009b30 con 0x7f4440099a40 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4437fff640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44400aff90 0x7f44401f2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4445031640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44400cbd30 0x7f44400b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.551+0000 7f4444830640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4440099a40 0x7f44401f40e0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f443c036040 tx=0x7f443c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.552+0000 7f44357fa640 1 -- 172.21.15.45:0/3561579894 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c009510 con 0x7f4440099a40 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.552+0000 7f44357fa640 1 -- 172.21.15.45:0/3561579894 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c03e450 con 0x7f4440099a40 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.552+0000 7f44357fa640 1 -- 172.21.15.45:0/3561579894 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c03e7e0 con 0x7f4440099a40 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.552+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44401f3900 con 0x7f4440099a40 2023-11-28T20:53:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.552+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44400b3db0 con 0x7f4440099a40 2023-11-28T20:53:16.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.553+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44400733d0 con 0x7f4440099a40 2023-11-28T20:53:16.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.554+0000 7f44357fa640 1 -- 172.21.15.45:0/3561579894 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 54) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f443c03e940 con 0x7f4440099a40 2023-11-28T20:53:16.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.555+0000 7f44357fa640 1 -- 172.21.15.45:0/3561579894 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6962+0+0 (secure 0 0 0) 0x7f443c0ac430 con 0x7f4440099a40 2023-11-28T20:53:16.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.559+0000 7f44357fa640 1 -- 172.21.15.45:0/3561579894 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f443c0772d0 con 0x7f4440099a40 2023-11-28T20:53:16.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.634+0000 7f44357fa640 1 -- 172.21.15.45:0/3561579894 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 55) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f443c076d80 con 0x7f4440099a40 2023-11-28T20:53:16.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.635+0000 7f44357fa640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f442c070dc0 0x7f442c0731b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:16.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.636+0000 7f4437fff640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f442c070dc0 0x7f442c0731b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:16.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.650+0000 7f4437fff640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f442c070dc0 0x7f442c0731b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f44400c7cb0 tx=0x7f4430009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:16.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.759+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44400665c0 con 0x7f4440099a40 2023-11-28T20:53:16.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.761+0000 7f44357fa640 1 -- 172.21.15.45:0/3561579894 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) v1 ==== 100+0+11 (secure 0 0 0) 0x7f443c07eb70 con 0x7f4440099a40 2023-11-28T20:53:16.763 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:53:16.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.763+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f442c070dc0 msgr2=0x7f442c0731b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.763+0000 7f4446a94640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f442c070dc0 0x7f442c0731b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f44400c7cb0 tx=0x7f4430009040 comp rx=0 tx=0).stop 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.763+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4440099a40 msgr2=0x7f44401f40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.763+0000 7f4446a94640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4440099a40 0x7f44401f40e0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f443c036040 tx=0x7f443c004a40 comp rx=0 tx=0).stop 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.764+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 shutdown_connections 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.764+0000 7f4446a94640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f442c070dc0 0x7f442c0731b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.764+0000 7f4446a94640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44400cbd30 0x7f44400b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.764+0000 7f4446a94640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44400aff90 0x7f44401f2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.764+0000 7f4446a94640 1 --2- 172.21.15.45:0/3561579894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4440099a40 0x7f44401f40e0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.764+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 >> 172.21.15.45:0/3561579894 conn(0x7f44400d1020 msgr2=0x7f444006aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.764+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 shutdown_connections 2023-11-28T20:53:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:16.764+0000 7f4446a94640 1 -- 172.21.15.45:0/3561579894 wait complete. 2023-11-28T20:53:16.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:53:17.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.010+0000 7fe00caad640 1 Processor -- start 2023-11-28T20:53:17.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.011+0000 7fe00caad640 1 -- start start 2023-11-28T20:53:17.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.013+0000 7fe00caad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe0080b1340 0x7fe00815f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.013+0000 7fe00caad640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00815bda0 con 0x7fe0080b35b0 2023-11-28T20:53:17.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.013+0000 7fe00caad640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00815c080 con 0x7fe00815b850 2023-11-28T20:53:17.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.013+0000 7fe00caad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00815ee40 con 0x7fe0080b1340 2023-11-28T20:53:17.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.013+0000 7fe006d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe0080b1340 0x7fe00815f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.013+0000 7fe00659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe0080b35b0 0x7fe00815ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37966/0 (socket says 172.21.15.45:37966) 2023-11-28T20:53:17.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.013+0000 7fe006d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe0080b1340 0x7fe00815f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38594/0 (socket says 172.21.15.45:38594) 2023-11-28T20:53:17.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.013+0000 7fe00659c640 1 -- 172.21.15.45:0/2409295495 learned_addr learned my addr 172.21.15.45:0/2409295495 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:17.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.014+0000 7fe006d9d640 1 -- 172.21.15.45:0/2409295495 >> v1:172.21.15.190:6789/0 conn(0x7fe00815b850 legacy=0x7fe0080b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:17.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.014+0000 7fe006d9d640 1 -- 172.21.15.45:0/2409295495 >> v1:172.21.15.45:6789/0 conn(0x7fe0080b35b0 legacy=0x7fe00815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.014+0000 7fe006d9d640 1 -- 172.21.15.45:0/2409295495 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00815c360 con 0x7fe0080b1340 2023-11-28T20:53:17.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.015+0000 7fe006d9d640 1 --2- 172.21.15.45:0/2409295495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b1340 0x7fe00815f290 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fe000009d90 tx=0x7fe00002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc27f49055b6780e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:17.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.015+0000 7fe005d9b640 1 -- 172.21.15.45:0/2409295495 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00002cc60 con 0x7fe0080b1340 2023-11-28T20:53:17.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.015+0000 7fe005d9b640 1 -- 172.21.15.45:0/2409295495 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00002cdc0 con 0x7fe0080b1340 2023-11-28T20:53:17.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.015+0000 7fe005d9b640 1 -- 172.21.15.45:0/2409295495 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000034660 con 0x7fe0080b1340 2023-11-28T20:53:17.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.016+0000 7fe00caad640 1 -- 172.21.15.45:0/2409295495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b1340 msgr2=0x7fe00815f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.016+0000 7fe00caad640 1 --2- 172.21.15.45:0/2409295495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b1340 0x7fe00815f290 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fe000009d90 tx=0x7fe00002beb0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.016+0000 7fe00caad640 1 -- 172.21.15.45:0/2409295495 shutdown_connections 2023-11-28T20:53:17.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.016+0000 7fe00caad640 1 --2- 172.21.15.45:0/2409295495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b1340 0x7fe00815f290 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.016+0000 7fe00caad640 1 -- 172.21.15.45:0/2409295495 >> 172.21.15.45:0/2409295495 conn(0x7fe00806a500 msgr2=0x7fe00806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:17.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.016+0000 7fe00caad640 1 -- 172.21.15.45:0/2409295495 shutdown_connections 2023-11-28T20:53:17.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.016+0000 7fe00caad640 1 -- 172.21.15.45:0/2409295495 wait complete. 2023-11-28T20:53:17.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.018+0000 7fe00caad640 1 Processor -- start 2023-11-28T20:53:17.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.018+0000 7fe00caad640 1 -- start start 2023-11-28T20:53:17.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.019+0000 7fe00caad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0080b1340 0x7fe0081f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.019+0000 7fe00caad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b35b0 0x7fe0081f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.019+0000 7fe00caad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe00815b850 0x7fe0081f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.019+0000 7fe00caad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe008156600 con 0x7fe0080b35b0 2023-11-28T20:53:17.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.019+0000 7fe00caad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe008159130 con 0x7fe00815b850 2023-11-28T20:53:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.019+0000 7fe00caad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00813d830 con 0x7fe0080b1340 2023-11-28T20:53:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.019+0000 7fdffffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b35b0 0x7fe0081f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.019+0000 7fdffffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b35b0 0x7fe0081f4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38596/0 (socket says 172.21.15.45:38596) 2023-11-28T20:53:17.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.019+0000 7fdffffff640 1 -- 172.21.15.45:0/3736827229 learned_addr learned my addr 172.21.15.45:0/3736827229 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.020+0000 7fe006d9d640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe00815b850 0x7fe0081f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.020+0000 7fe00659c640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0080b1340 0x7fe0081f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.020+0000 7fdffffff640 1 -- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0080b1340 msgr2=0x7fe0081f07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.020+0000 7fdffffff640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0080b1340 0x7fe0081f07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.020+0000 7fdffffff640 1 -- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe00815b850 msgr2=0x7fe0081f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.020+0000 7fdffffff640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe00815b850 0x7fe0081f4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.020+0000 7fdffffff640 1 -- 172.21.15.45:0/3736827229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe000009a70 con 0x7fe0080b35b0 2023-11-28T20:53:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.020+0000 7fdffffff640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b35b0 0x7fe0081f4630 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fdff000b3f0 tx=0x7fdff000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.021+0000 7fdffeffd640 1 -- 172.21.15.45:0/3736827229 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff0011830 con 0x7fe0080b35b0 2023-11-28T20:53:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.021+0000 7fdffeffd640 1 -- 172.21.15.45:0/3736827229 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff0011990 con 0x7fe0080b35b0 2023-11-28T20:53:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.021+0000 7fdffeffd640 1 -- 172.21.15.45:0/3736827229 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff0011d40 con 0x7fe0080b35b0 2023-11-28T20:53:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.021+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0081f51a0 con 0x7fe0080b35b0 2023-11-28T20:53:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.021+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe008156290 con 0x7fe0080b35b0 2023-11-28T20:53:17.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.022+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0081567b0 con 0x7fe0080b35b0 2023-11-28T20:53:17.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.023+0000 7fdffeffd640 1 -- 172.21.15.45:0/3736827229 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fdff001a400 con 0x7fe0080b35b0 2023-11-28T20:53:17.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.024+0000 7fdffeffd640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fdfe806acc0 0x7fdfe806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.024+0000 7fdffeffd640 1 -- 172.21.15.45:0/3736827229 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6962+0+0 (secure 0 0 0) 0x7fdff00894b0 con 0x7fe0080b35b0 2023-11-28T20:53:17.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.025+0000 7fe00659c640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fdfe806acc0 0x7fdfe806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.026+0000 7fe00659c640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fdfe806acc0 0x7fdfe806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdff8005880 tx=0x7fdff8009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:17.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.029+0000 7fdffeffd640 1 -- 172.21.15.45:0/3736827229 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0081567b0 con 0x7fe0080b35b0 2023-11-28T20:53:17.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.262+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0081567b0 con 0x7fe0080b35b0 2023-11-28T20:53:17.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.263+0000 7fdffeffd640 1 -- 172.21.15.45:0/3736827229 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fe0081567b0 con 0x7fe0080b35b0 2023-11-28T20:53:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:17.266 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2348343744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2348343744},{"type":"v1","addr":"172.21.15.190:6836","nonce":2348343744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6439,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1864791063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":1864791063},{"type":"v1","addr":"172.21.15.190:6837","nonce":1864791063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3003510210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3003510210},{"type":"v1","addr":"172.21.15.45:6835","nonce":3003510210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:52.969776+0000","modified":"2023-11-28T20:51:56.055296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6448":{"gid":6448,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2489162496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2489162496},{"type":"v1","addr":"172.21.15.45:6834","nonce":2489162496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-28T20:53:17.266 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 51 2023-11-28T20:53:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fdfe806acc0 msgr2=0x7fdfe806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fdfe806acc0 0x7fdfe806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdff8005880 tx=0x7fdff8009040 comp rx=0 tx=0).stop 2023-11-28T20:53:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b35b0 msgr2=0x7fe0081f4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b35b0 0x7fe0081f4630 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fdff000b3f0 tx=0x7fdff000b8b0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 shutdown_connections 2023-11-28T20:53:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fdfe806acc0 0x7fdfe806d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe00815b850 0x7fe0081f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0080b35b0 0x7fe0081f4630 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 --2- 172.21.15.45:0/3736827229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0080b1340 0x7fe0081f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 >> 172.21.15.45:0/3736827229 conn(0x7fe00806a500 msgr2=0x7fe008144740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 shutdown_connections 2023-11-28T20:53:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.266+0000 7fe00caad640 1 -- 172.21.15.45:0/3736827229 wait complete. 2023-11-28T20:53:17.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:53:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.516+0000 7f907f59e640 1 Processor -- start 2023-11-28T20:53:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.517+0000 7f907f59e640 1 -- start start 2023-11-28T20:53:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.518+0000 7f907f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908014b090 0x7f90800b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.518+0000 7f907f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f90800ad3b0 0x7f90800b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.518+0000 7f907f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90800abf10 con 0x7f90800ae590 2023-11-28T20:53:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.519+0000 7f907f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908006db90 con 0x7f90800ad3b0 2023-11-28T20:53:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.519+0000 7f907f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908006d2f0 con 0x7f908014b090 2023-11-28T20:53:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.519+0000 7f9077fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f90800ae590 0x7f9080165250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37968/0 (socket says 172.21.15.45:37968) 2023-11-28T20:53:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.519+0000 7f9077fff640 1 -- 172.21.15.45:0/2355468556 learned_addr learned my addr 172.21.15.45:0/2355468556 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.519+0000 7f907ed9d640 1 --2- 172.21.15.45:0/2355468556 >> v2:172.21.15.190:3300/0 conn(0x7f90800ad3b0 0x7f90800b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.519+0000 7f907e59c640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908014b090 0x7f90800b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584849588 0 0) 0x7f90800abf10 con 0x7f90800ae590 2023-11-28T20:53:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f906c003520 con 0x7f90800ae590 2023-11-28T20:53:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3611249335 0 0) 0x7f906c003520 con 0x7f90800ae590 2023-11-28T20:53:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908014b090 msgr2=0x7f90800b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908014b090 0x7f90800b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 >> v2:172.21.15.190:3300/0 conn(0x7f90800ad3b0 msgr2=0x7f90800b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 --2- 172.21.15.45:0/2355468556 >> v2:172.21.15.190:3300/0 conn(0x7f90800ad3b0 0x7f90800b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908006d5d0 con 0x7f90800ae590 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f9068001670 con 0x7f90800ae590 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 >> v1:172.21.15.45:6789/0 conn(0x7f90800ae590 legacy=0x7f9080165250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f906c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f906c0047c0 0x7f906c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908014b090 0x7f906c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908006db90 con 0x7f906c003970 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908006d2f0 con 0x7f906c0047c0 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.520+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906c00b9d0 con 0x7f908014b090 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.521+0000 7f907e59c640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f906c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.521+0000 7f9077fff640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f906c0047c0 0x7f906c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.521+0000 7f907ed9d640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908014b090 0x7f906c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.521+0000 7f907e59c640 1 -- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908014b090 msgr2=0x7f906c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.521+0000 7f907e59c640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908014b090 0x7f906c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.521+0000 7f907e59c640 1 -- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f906c0047c0 msgr2=0x7f906c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.521+0000 7f907e59c640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f906c0047c0 0x7f906c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.521+0000 7f907e59c640 1 -- 172.21.15.45:0/2355468556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c003520 con 0x7f906c003970 2023-11-28T20:53:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.522+0000 7f907e59c640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f906c003d90 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f907000ba30 tx=0x7f906c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb49d00b88cf66d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070032640 con 0x7f906c003970 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90700327a0 con 0x7f906c003970 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907dd9b640 1 -- 172.21.15.45:0/2355468556 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070032b70 con 0x7f906c003970 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907f59e640 1 -- 172.21.15.45:0/2355468556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 msgr2=0x7f906c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907f59e640 1 --2- 172.21.15.45:0/2355468556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f906c003d90 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f907000ba30 tx=0x7f906c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907ed9d640 1 -- 172.21.15.45:0/2355468556 reap_dead start 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907f59e640 1 -- 172.21.15.45:0/2355468556 shutdown_connections 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907f59e640 1 -- 172.21.15.45:0/2355468556 >> 172.21.15.45:0/2355468556 conn(0x7f908006ab30 msgr2=0x7f90800b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907f59e640 1 -- 172.21.15.45:0/2355468556 shutdown_connections 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.523+0000 7f907f59e640 1 -- 172.21.15.45:0/2355468556 wait complete. 2023-11-28T20:53:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.525+0000 7f907f59e640 1 Processor -- start 2023-11-28T20:53:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.525+0000 7f907f59e640 1 -- start start 2023-11-28T20:53:17.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.526+0000 7f907f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f908009beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.526+0000 7f907f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908009ad00 0x7f908014fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.526+0000 7f907f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f908009b180 0x7f9080150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.526+0000 7f907f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908009ba90 con 0x7f906c003970 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.526+0000 7f907f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9080144160 con 0x7f908009b180 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.526+0000 7f907f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90801edc40 con 0x7f908009ad00 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f907e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f908009beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f907e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f908009beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38608/0 (socket says 172.21.15.45:38608) 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f907e59c640 1 -- 172.21.15.45:0/1284571373 learned_addr learned my addr 172.21.15.45:0/1284571373 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f9077fff640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908009ad00 0x7f908014fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f907ed9d640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f908009b180 0x7f9080150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f907e59c640 1 -- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908009ad00 msgr2=0x7f908014fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f907e59c640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908009ad00 0x7f908014fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f907e59c640 1 -- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f908009b180 msgr2=0x7f9080150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f907e59c640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f908009b180 0x7f9080150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.527+0000 7f907e59c640 1 -- 172.21.15.45:0/1284571373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9080143950 con 0x7f906c003970 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.528+0000 7f907e59c640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f908009beb0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f907000bb50 tx=0x7f9070004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.528+0000 7f9076ffd640 1 -- 172.21.15.45:0/1284571373 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070004220 con 0x7f906c003970 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.528+0000 7f9076ffd640 1 -- 172.21.15.45:0/1284571373 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9070004380 con 0x7f906c003970 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.528+0000 7f9076ffd640 1 -- 172.21.15.45:0/1284571373 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907003ba30 con 0x7f906c003970 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.528+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908009c3f0 con 0x7f906c003970 2023-11-28T20:53:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.528+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9080143e80 con 0x7f906c003970 2023-11-28T20:53:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.529+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908009d0f0 con 0x7f906c003970 2023-11-28T20:53:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.530+0000 7f9076ffd640 1 -- 172.21.15.45:0/1284571373 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 55) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f907003bb90 con 0x7f906c003970 2023-11-28T20:53:17.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.531+0000 7f9076ffd640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f906006acc0 0x7f906006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:17.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.531+0000 7f9076ffd640 1 -- 172.21.15.45:0/1284571373 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6962+0+0 (secure 0 0 0) 0x7f90700ac090 con 0x7f906c003970 2023-11-28T20:53:17.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.532+0000 7f9077fff640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f906006acc0 0x7f906006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:17.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.533+0000 7f9077fff640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f906006acc0 0x7f906006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9068001eb0 tx=0x7f9068009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:17.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.535+0000 7f9076ffd640 1 -- 172.21.15.45:0/1284571373 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9070076e30 con 0x7f906c003970 2023-11-28T20:53:17.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.656+0000 7f9076ffd640 1 -- 172.21.15.45:0/1284571373 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 56) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f9070077200 con 0x7f906c003970 2023-11-28T20:53:17.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.747+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f908009d0f0 con 0x7f906c003970 2023-11-28T20:53:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.750+0000 7f9076ffd640 1 -- 172.21.15.45:0/1284571373 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+18117 (secure 0 0 0) 0x7f9070091320 con 0x7f906c003970 2023-11-28T20:53:17.753 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:17.753 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":121,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:15.615033+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:50.993341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:52.004867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":7}]},{"pgid":"14.2a","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6833/1902260958":"2023-11-29T20:53:15.614996+0000","172.21.15.190:6832/1902260958":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/2251272392":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/1361952934":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/2675112319":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/915400531":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/3484290831":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/1265166084":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/1549713842":"2023-11-29T20:53:15.614996+0000","172.21.15.190:6833/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:6832/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3239477442":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3809276606":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3450487150":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2166104670":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3808290090":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/4069818189":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2641549859":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/1724154865":"2023-11-28T21:51:59.768228+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:53:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.752+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f906006acc0 msgr2=0x7f906006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.752+0000 7f907f59e640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f906006acc0 0x7f906006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9068001eb0 tx=0x7f9068009040 comp rx=0 tx=0).stop 2023-11-28T20:53:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 msgr2=0x7f908009beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f908009beb0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f907000bb50 tx=0x7f9070004730 comp rx=0 tx=0).stop 2023-11-28T20:53:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 shutdown_connections 2023-11-28T20:53:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f906006acc0 0x7f906006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f908009b180 0x7f9080150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f908009ad00 0x7f908014fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 --2- 172.21.15.45:0/1284571373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f906c003970 0x7f908009beb0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:17.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 >> 172.21.15.45:0/1284571373 conn(0x7f908006ab30 msgr2=0x7f90800af490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:17.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 shutdown_connections 2023-11-28T20:53:17.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.753+0000 7f907f59e640 1 -- 172.21.15.45:0/1284571373 wait complete. 2023-11-28T20:53:17.776 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T20:53:17.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:53:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.996+0000 7f1139b88640 1 Processor -- start 2023-11-28T20:53:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.997+0000 7f1139b88640 1 -- start start 2023-11-28T20:53:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.998+0000 7f1139b88640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f11340b24a0 0x7f11340703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.998+0000 7f1139b88640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 0x7f1134070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.998+0000 7f1139b88640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340b4310 con 0x7f1134072200 2023-11-28T20:53:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.998+0000 7f1139b88640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11340725e0 con 0x7f11340b24a0 2023-11-28T20:53:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.998+0000 7f1139b88640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113406d8d0 con 0x7f1134146610 2023-11-28T20:53:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.999+0000 7f1133fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f11340b24a0 0x7f11340703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.999+0000 7f1139387640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 0x7f1134070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.999+0000 7f1133fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f11340b24a0 0x7f11340703d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:36312/0 (socket says 172.21.15.45:36312) 2023-11-28T20:53:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:17.999+0000 7f1133fff640 1 -- 172.21.15.45:0/3957160634 learned_addr learned my addr 172.21.15.45:0/3957160634 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.000+0000 7f1133fff640 1 -- 172.21.15.45:0/3957160634 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 msgr2=0x7f1134070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.000+0000 7f1133fff640 1 --2- 172.21.15.45:0/3957160634 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 0x7f1134070910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.000+0000 7f1133fff640 1 -- 172.21.15.45:0/3957160634 >> v1:172.21.15.190:6789/0 conn(0x7f1134072200 legacy=0x7f11340b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.000+0000 7f1133fff640 1 -- 172.21.15.45:0/3957160634 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113406ca20 con 0x7f11340b24a0 2023-11-28T20:53:18.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.001+0000 7f1133fff640 1 --2- 172.21.15.45:0/3957160634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 0x7f11340703d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1120009870 tx=0x7f112002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dc446086a93ee53 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.002+0000 7f11337fe640 1 -- 172.21.15.45:0/3957160634 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120030840 con 0x7f11340b24a0 2023-11-28T20:53:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.002+0000 7f11337fe640 1 -- 172.21.15.45:0/3957160634 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11200309a0 con 0x7f11340b24a0 2023-11-28T20:53:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.002+0000 7f11337fe640 1 -- 172.21.15.45:0/3957160634 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120030d30 con 0x7f11340b24a0 2023-11-28T20:53:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.003+0000 7f1139b88640 1 -- 172.21.15.45:0/3957160634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 msgr2=0x7f11340703d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.003+0000 7f1139b88640 1 --2- 172.21.15.45:0/3957160634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 0x7f11340703d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1120009870 tx=0x7f112002b980 comp rx=0 tx=0).stop 2023-11-28T20:53:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.003+0000 7f1139b88640 1 -- 172.21.15.45:0/3957160634 shutdown_connections 2023-11-28T20:53:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.003+0000 7f1139b88640 1 --2- 172.21.15.45:0/3957160634 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 0x7f1134070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.003+0000 7f1139b88640 1 --2- 172.21.15.45:0/3957160634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 0x7f11340703d0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.003+0000 7f1139b88640 1 -- 172.21.15.45:0/3957160634 >> 172.21.15.45:0/3957160634 conn(0x7f113406ab30 msgr2=0x7f11340b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.003+0000 7f1139b88640 1 -- 172.21.15.45:0/3957160634 shutdown_connections 2023-11-28T20:53:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.003+0000 7f1139b88640 1 -- 172.21.15.45:0/3957160634 wait complete. 2023-11-28T20:53:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.005+0000 7f1139b88640 1 Processor -- start 2023-11-28T20:53:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.005+0000 7f1139b88640 1 -- start start 2023-11-28T20:53:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1139b88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1134072200 0x7f1134099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1139b88640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 0x7f11341345c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1139b88640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 0x7f113409be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1139b88640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113409ae80 con 0x7f1134072200 2023-11-28T20:53:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1139b88640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113409a270 con 0x7f11340b24a0 2023-11-28T20:53:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1139b88640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113409a550 con 0x7f1134146610 2023-11-28T20:53:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1138b86640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1134072200 0x7f1134099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1138b86640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1134072200 0x7f1134099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38622/0 (socket says 172.21.15.45:38622) 2023-11-28T20:53:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1138b86640 1 -- 172.21.15.45:0/803301401 learned_addr learned my addr 172.21.15.45:0/803301401 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1133fff640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 0x7f11341345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.007+0000 7f1139387640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 0x7f113409be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f1138b86640 1 -- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 msgr2=0x7f113409be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f1138b86640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 0x7f113409be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f1138b86640 1 -- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 msgr2=0x7f11341345c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f1138b86640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 0x7f11341345c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f1138b86640 1 -- 172.21.15.45:0/803301401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11200094d0 con 0x7f1134072200 2023-11-28T20:53:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f1133fff640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 0x7f11341345c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f1139387640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 0x7f113409be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f1138b86640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1134072200 0x7f1134099a40 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f112800c450 tx=0x7f112800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f11317fa640 1 -- 172.21.15.45:0/803301401 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11280052d0 con 0x7f1134072200 2023-11-28T20:53:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f11317fa640 1 -- 172.21.15.45:0/803301401 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1128005430 con 0x7f1134072200 2023-11-28T20:53:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f11317fa640 1 -- 172.21.15.45:0/803301401 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1128012870 con 0x7f1134072200 2023-11-28T20:53:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.008+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113409a830 con 0x7f1134072200 2023-11-28T20:53:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.009+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1134099f80 con 0x7f1134072200 2023-11-28T20:53:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.010+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113409d0f0 con 0x7f1134072200 2023-11-28T20:53:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.011+0000 7f11317fa640 1 -- 172.21.15.45:0/803301401 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 56) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f11280129d0 con 0x7f1134072200 2023-11-28T20:53:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.012+0000 7f11317fa640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f111806af00 0x7f111806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.012+0000 7f11317fa640 1 -- 172.21.15.45:0/803301401 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6962+0+0 (secure 0 0 0) 0x7f1128089610 con 0x7f1134072200 2023-11-28T20:53:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.012+0000 7f1133fff640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f111806af00 0x7f111806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.014+0000 7f1133fff640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f111806af00 0x7f111806d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1120036000 tx=0x7f1120008030 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.016+0000 7f11317fa640 1 -- 172.21.15.45:0/803301401 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1128054050 con 0x7f1134072200 2023-11-28T20:53:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.251+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11340b4310 con 0x7f1134072200 2023-11-28T20:53:18.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.252+0000 7f11317fa640 1 -- 172.21.15.45:0/803301401 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f11340b4310 con 0x7f1134072200 2023-11-28T20:53:18.255 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:18.255 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2348343744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2348343744},{"type":"v1","addr":"172.21.15.190:6836","nonce":2348343744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6439,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1864791063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":1864791063},{"type":"v1","addr":"172.21.15.190:6837","nonce":1864791063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3003510210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3003510210},{"type":"v1","addr":"172.21.15.45:6835","nonce":3003510210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:52.969776+0000","modified":"2023-11-28T20:51:56.055296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6448":{"gid":6448,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2489162496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2489162496},{"type":"v1","addr":"172.21.15.45:6834","nonce":2489162496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-28T20:53:18.255 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 51 2023-11-28T20:53:18.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f111806af00 msgr2=0x7f111806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f111806af00 0x7f111806d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1120036000 tx=0x7f1120008030 comp rx=0 tx=0).stop 2023-11-28T20:53:18.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1134072200 msgr2=0x7f1134099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1134072200 0x7f1134099a40 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f112800c450 tx=0x7f112800c910 comp rx=0 tx=0).stop 2023-11-28T20:53:18.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 shutdown_connections 2023-11-28T20:53:18.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f111806af00 0x7f111806d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1134146610 0x7f113409be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11340b24a0 0x7f11341345c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 --2- 172.21.15.45:0/803301401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1134072200 0x7f1134099a40 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 >> 172.21.15.45:0/803301401 conn(0x7f113406ab30 msgr2=0x7f11340ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:18.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 shutdown_connections 2023-11-28T20:53:18.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.255+0000 7f1139b88640 1 -- 172.21.15.45:0/803301401 wait complete. 2023-11-28T20:53:18.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:53:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.507+0000 7f47d7489640 1 Processor -- start 2023-11-28T20:53:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.507+0000 7f47d7489640 1 -- start start 2023-11-28T20:53:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.509+0000 7f47d7489640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d015d320 0x7f47d00ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.509+0000 7f47d7489640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f47d00add40 0x7f47d00b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.509+0000 7f47d7489640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d00ce710 con 0x7f47d00b1170 2023-11-28T20:53:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.509+0000 7f47d7489640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d00ae5a0 con 0x7f47d00add40 2023-11-28T20:53:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.509+0000 7f47d7489640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d00b1600 con 0x7f47d015d320 2023-11-28T20:53:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.510+0000 7f47d5225640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d015d320 0x7f47d00ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.510+0000 7f47d5225640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d015d320 0x7f47d00ad800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:59174/0 (socket says 172.21.15.45:59174) 2023-11-28T20:53:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.510+0000 7f47d5225640 1 -- 172.21.15.45:0/4149702545 learned_addr learned my addr 172.21.15.45:0/4149702545 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.510+0000 7f47d4a24640 1 --2- 172.21.15.45:0/4149702545 >> v2:172.21.15.190:3300/0 conn(0x7f47d00add40 0x7f47d00b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.511+0000 7f47d4a24640 1 -- 172.21.15.45:0/4149702545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d015d320 msgr2=0x7f47d00ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.511+0000 7f47d4a24640 1 --2- 172.21.15.45:0/4149702545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d015d320 0x7f47d00ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.511+0000 7f47d4a24640 1 -- 172.21.15.45:0/4149702545 >> v1:172.21.15.190:6789/0 conn(0x7f47d00b1170 legacy=0x7f47d00cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.511+0000 7f47d4a24640 1 -- 172.21.15.45:0/4149702545 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d00721a0 con 0x7f47d00add40 2023-11-28T20:53:18.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.512+0000 7f47d4a24640 1 --2- 172.21.15.45:0/4149702545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47d00add40 0x7f47d00b0c30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f47c00097a0 tx=0x7f47c002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9931377477ed11 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:18.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.512+0000 7f47c7fff640 1 -- 172.21.15.45:0/4149702545 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c002d040 con 0x7f47d00add40 2023-11-28T20:53:18.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.513+0000 7f47c7fff640 1 -- 172.21.15.45:0/4149702545 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c0031810 con 0x7f47d00add40 2023-11-28T20:53:18.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.513+0000 7f47c7fff640 1 -- 172.21.15.45:0/4149702545 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c0031ba0 con 0x7f47d00add40 2023-11-28T20:53:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.513+0000 7f47d7489640 1 -- 172.21.15.45:0/4149702545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47d00add40 msgr2=0x7f47d00b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.513+0000 7f47d7489640 1 --2- 172.21.15.45:0/4149702545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47d00add40 0x7f47d00b0c30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f47c00097a0 tx=0x7f47c002b8b0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.513+0000 7f47d7489640 1 -- 172.21.15.45:0/4149702545 shutdown_connections 2023-11-28T20:53:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.513+0000 7f47d7489640 1 --2- 172.21.15.45:0/4149702545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47d00add40 0x7f47d00b0c30 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.513+0000 7f47d7489640 1 --2- 172.21.15.45:0/4149702545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d015d320 0x7f47d00ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.513+0000 7f47d7489640 1 -- 172.21.15.45:0/4149702545 >> 172.21.15.45:0/4149702545 conn(0x7f47d006a500 msgr2=0x7f47d006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.514+0000 7f47d7489640 1 -- 172.21.15.45:0/4149702545 shutdown_connections 2023-11-28T20:53:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.514+0000 7f47d7489640 1 -- 172.21.15.45:0/4149702545 wait complete. 2023-11-28T20:53:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.515+0000 7f47d7489640 1 Processor -- start 2023-11-28T20:53:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.515+0000 7f47d7489640 1 -- start start 2023-11-28T20:53:18.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.516+0000 7f47d7489640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47d00add40 0x7f47d015e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.516+0000 7f47d7489640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d00b1170 0x7f47d00afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.516+0000 7f47d7489640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015d320 0x7f47d00aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.516+0000 7f47d7489640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d0160320 con 0x7f47d015d320 2023-11-28T20:53:18.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.516+0000 7f47d7489640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d014df00 con 0x7f47d00add40 2023-11-28T20:53:18.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.516+0000 7f47d7489640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d014e1e0 con 0x7f47d00b1170 2023-11-28T20:53:18.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.517+0000 7f47d5a26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015d320 0x7f47d00aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.517+0000 7f47d5a26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015d320 0x7f47d00aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38632/0 (socket says 172.21.15.45:38632) 2023-11-28T20:53:18.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.517+0000 7f47d5a26640 1 -- 172.21.15.45:0/3089346815 learned_addr learned my addr 172.21.15.45:0/3089346815 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:18.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.517+0000 7f47d5225640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47d00add40 0x7f47d015e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.517+0000 7f47d4a24640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d00b1170 0x7f47d00afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47d5a26640 1 -- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d00b1170 msgr2=0x7f47d00afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47d5a26640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d00b1170 0x7f47d00afa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47d5a26640 1 -- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47d00add40 msgr2=0x7f47d015e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47d5a26640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47d00add40 0x7f47d015e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47d5a26640 1 -- 172.21.15.45:0/3089346815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c0009400 con 0x7f47d015d320 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47d5a26640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015d320 0x7f47d00aff40 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f47c800c7a0 tx=0x7f47c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47c5ffb640 1 -- 172.21.15.45:0/3089346815 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8003b70 con 0x7f47d015d320 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47c5ffb640 1 -- 172.21.15.45:0/3089346815 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c8003cf0 con 0x7f47d015d320 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d014e4c0 con 0x7f47d015d320 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47c5ffb640 1 -- 172.21.15.45:0/3089346815 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8003ea0 con 0x7f47d015d320 2023-11-28T20:53:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.518+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d014ea80 con 0x7f47d015d320 2023-11-28T20:53:18.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.519+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d014ec70 con 0x7f47d015d320 2023-11-28T20:53:18.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.521+0000 7f47c5ffb640 1 -- 172.21.15.45:0/3089346815 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 56) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f47c800aa90 con 0x7f47d015d320 2023-11-28T20:53:18.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.521+0000 7f47c5ffb640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f47b406aed0 0x7f47b406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:18.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.522+0000 7f47c5ffb640 1 -- 172.21.15.45:0/3089346815 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6962+0+0 (secure 0 0 0) 0x7f47c8089900 con 0x7f47d015d320 2023-11-28T20:53:18.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.522+0000 7f47d5225640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f47b406aed0 0x7f47b406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:18.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.523+0000 7f47d5225640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f47b406aed0 0x7f47b406d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009b30 tx=0x7f47cc009240 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:18.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.526+0000 7f47c5ffb640 1 -- 172.21.15.45:0/3089346815 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47c8054480 con 0x7f47d015d320 2023-11-28T20:53:18.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.762+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47d00733d0 con 0x7f47d015d320 2023-11-28T20:53:19.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.763+0000 7f47c5ffb640 1 -- 172.21.15.45:0/3089346815 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f47c8057aa0 con 0x7f47d015d320 2023-11-28T20:53:19.709 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:19.709 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2348343744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2348343744},{"type":"v1","addr":"172.21.15.190:6836","nonce":2348343744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6439,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1864791063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":1864791063},{"type":"v1","addr":"172.21.15.190:6837","nonce":1864791063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3003510210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3003510210},{"type":"v1","addr":"172.21.15.45:6835","nonce":3003510210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:51:52.969776+0000","modified":"2023-11-28T20:51:56.055296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6448":{"gid":6448,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2489162496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2489162496},{"type":"v1","addr":"172.21.15.45:6834","nonce":2489162496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-11-28T20:53:19.711 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 51 2023-11-28T20:53:19.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f47b406aed0 msgr2=0x7f47b406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:19.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f47b406aed0 0x7f47b406d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009b30 tx=0x7f47cc009240 comp rx=0 tx=0).stop 2023-11-28T20:53:19.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015d320 msgr2=0x7f47d00aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:19.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015d320 0x7f47d00aff40 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f47c800c7a0 tx=0x7f47c800cc60 comp rx=0 tx=0).stop 2023-11-28T20:53:19.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 shutdown_connections 2023-11-28T20:53:19.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f47b406aed0 0x7f47b406d390 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:19.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d015d320 0x7f47d00aff40 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:19.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47d00b1170 0x7f47d00afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:19.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 --2- 172.21.15.45:0/3089346815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47d00add40 0x7f47d015e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:19.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 >> 172.21.15.45:0/3089346815 conn(0x7f47d006a500 msgr2=0x7f47d00ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:19.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 shutdown_connections 2023-11-28T20:53:19.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:18.766+0000 7f47d7489640 1 -- 172.21.15.45:0/3089346815 wait complete. 2023-11-28T20:53:19.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:53:19.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.950+0000 7fb919d3c640 1 Processor -- start 2023-11-28T20:53:19.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.950+0000 7fb919d3c640 1 -- start start 2023-11-28T20:53:19.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.952+0000 7fb919d3c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb9140b0c30 0x7fb9140b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:19.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.952+0000 7fb919d3c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9140b3d60 0x7fb9140b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:19.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.952+0000 7fb919d3c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91415dcc0 con 0x7fb9141438b0 2023-11-28T20:53:19.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.952+0000 7fb919d3c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9140b1620 con 0x7fb9140b0c30 2023-11-28T20:53:19.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.952+0000 7fb919d3c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91415e840 con 0x7fb9140b3d60 2023-11-28T20:53:19.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.952+0000 7fb9137fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb9141438b0 0x7fb9140b1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37984/0 (socket says 172.21.15.45:37984) 2023-11-28T20:53:19.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.952+0000 7fb9137fe640 1 -- 172.21.15.45:0/1494684115 learned_addr learned my addr 172.21.15.45:0/1494684115 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:19.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.952+0000 7fb912ffd640 1 --2- 172.21.15.45:0/1494684115 >> v2:172.21.15.45:3300/0 conn(0x7fb9140b0c30 0x7fb9140b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:19.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.953+0000 7fb913fff640 1 --2- 172.21.15.45:0/1494684115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9140b3d60 0x7fb9140b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:19.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.953+0000 7fb9127fc640 1 -- 172.21.15.45:0/1494684115 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065907481 0 0) 0x7fb91415dcc0 con 0x7fb9141438b0 2023-11-28T20:53:19.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.953+0000 7fb9127fc640 1 -- 172.21.15.45:0/1494684115 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8fc003520 con 0x7fb9141438b0 2023-11-28T20:53:19.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.953+0000 7fb912ffd640 1 -- 172.21.15.45:0/1494684115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9140b3d60 msgr2=0x7fb9140b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:19.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.953+0000 7fb912ffd640 1 --2- 172.21.15.45:0/1494684115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9140b3d60 0x7fb9140b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:19.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.953+0000 7fb912ffd640 1 -- 172.21.15.45:0/1494684115 >> v1:172.21.15.45:6789/0 conn(0x7fb9141438b0 legacy=0x7fb9140b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:19.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.953+0000 7fb912ffd640 1 -- 172.21.15.45:0/1494684115 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140b26b0 con 0x7fb9140b0c30 2023-11-28T20:53:19.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.954+0000 7fb912ffd640 1 --2- 172.21.15.45:0/1494684115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b0c30 0x7fb9140b10e0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fb9080097a0 tx=0x7fb90802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5993e20c9194df2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:19.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.954+0000 7fb9127fc640 1 -- 172.21.15.45:0/1494684115 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9080316f0 con 0x7fb9140b0c30 2023-11-28T20:53:19.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.954+0000 7fb9127fc640 1 -- 172.21.15.45:0/1494684115 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb908031850 con 0x7fb9140b0c30 2023-11-28T20:53:19.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.954+0000 7fb9127fc640 1 -- 172.21.15.45:0/1494684115 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908031be0 con 0x7fb9140b0c30 2023-11-28T20:53:19.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.955+0000 7fb919d3c640 1 -- 172.21.15.45:0/1494684115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b0c30 msgr2=0x7fb9140b10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:19.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.955+0000 7fb919d3c640 1 --2- 172.21.15.45:0/1494684115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b0c30 0x7fb9140b10e0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fb9080097a0 tx=0x7fb90802b790 comp rx=0 tx=0).stop 2023-11-28T20:53:19.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.955+0000 7fb919d3c640 1 -- 172.21.15.45:0/1494684115 shutdown_connections 2023-11-28T20:53:19.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.955+0000 7fb919d3c640 1 --2- 172.21.15.45:0/1494684115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9140b3d60 0x7fb9140b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:19.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.955+0000 7fb919d3c640 1 --2- 172.21.15.45:0/1494684115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b0c30 0x7fb9140b10e0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:19.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.955+0000 7fb919d3c640 1 -- 172.21.15.45:0/1494684115 >> 172.21.15.45:0/1494684115 conn(0x7fb91406a500 msgr2=0x7fb91406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:19.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.955+0000 7fb919d3c640 1 -- 172.21.15.45:0/1494684115 shutdown_connections 2023-11-28T20:53:19.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.955+0000 7fb919d3c640 1 -- 172.21.15.45:0/1494684115 wait complete. 2023-11-28T20:53:19.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.957+0000 7fb919d3c640 1 Processor -- start 2023-11-28T20:53:19.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.957+0000 7fb919d3c640 1 -- start start 2023-11-28T20:53:19.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.958+0000 7fb919d3c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9140b0c30 0x7fb91413ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:19.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.958+0000 7fb919d3c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b3d60 0x7fb9141f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:19.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.958+0000 7fb919d3c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9141438b0 0x7fb9140712d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:19.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.958+0000 7fb919d3c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91413ba00 con 0x7fb9140b3d60 2023-11-28T20:53:19.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.958+0000 7fb919d3c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91413c2f0 con 0x7fb9141438b0 2023-11-28T20:53:19.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.958+0000 7fb919d3c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91413d5e0 con 0x7fb9140b0c30 2023-11-28T20:53:19.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.958+0000 7fb912ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b3d60 0x7fb9141f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:19.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.958+0000 7fb912ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b3d60 0x7fb9141f3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38654/0 (socket says 172.21.15.45:38654) 2023-11-28T20:53:19.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.958+0000 7fb912ffd640 1 -- 172.21.15.45:0/1537380512 learned_addr learned my addr 172.21.15.45:0/1537380512 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:19.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.959+0000 7fb912ffd640 1 -- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9140b0c30 msgr2=0x7fb91413ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:19.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.959+0000 7fb912ffd640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9140b0c30 0x7fb91413ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:19.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.959+0000 7fb912ffd640 1 -- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9141438b0 msgr2=0x7fb9140712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:19.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.959+0000 7fb912ffd640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9141438b0 0x7fb9140712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:19.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.959+0000 7fb912ffd640 1 -- 172.21.15.45:0/1537380512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb908009400 con 0x7fb9140b3d60 2023-11-28T20:53:19.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.959+0000 7fb912ffd640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b3d60 0x7fb9141f3e50 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb908000c00 tx=0x7fb9080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:19.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.959+0000 7fb8fbfff640 1 -- 172.21.15.45:0/1537380512 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908034680 con 0x7fb9140b3d60 2023-11-28T20:53:19.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.960+0000 7fb8fbfff640 1 -- 172.21.15.45:0/1537380512 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9080347e0 con 0x7fb9140b3d60 2023-11-28T20:53:19.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.960+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb91413c5a0 con 0x7fb9140b3d60 2023-11-28T20:53:19.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.960+0000 7fb8fbfff640 1 -- 172.21.15.45:0/1537380512 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908034b90 con 0x7fb9140b3d60 2023-11-28T20:53:19.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.960+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91413cde0 con 0x7fb9140b3d60 2023-11-28T20:53:19.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.961+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9140733d0 con 0x7fb9140b3d60 2023-11-28T20:53:19.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.962+0000 7fb8fbfff640 1 -- 172.21.15.45:0/1537380512 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 56) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fb90803b040 con 0x7fb9140b3d60 2023-11-28T20:53:19.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.963+0000 7fb8fbfff640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb90006aed0 0x7fb90006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:19.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.963+0000 7fb8fbfff640 1 -- 172.21.15.45:0/1537380512 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6962+0+0 (secure 0 0 0) 0x7fb9080ab8e0 con 0x7fb9140b3d60 2023-11-28T20:53:19.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.964+0000 7fb9137fe640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb90006aed0 0x7fb90006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:19.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.965+0000 7fb9137fe640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb90006aed0 0x7fb90006d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb904001610 tx=0x7fb904007930 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:19.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:19.967+0000 7fb8fbfff640 1 -- 172.21.15.45:0/1537380512 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb908076330 con 0x7fb9140b3d60 2023-11-28T20:53:20.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.164+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9140ae9c0 con 0x7fb9140b3d60 2023-11-28T20:53:20.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.167+0000 7fb8fbfff640 1 -- 172.21.15.45:0/1537380512 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+18117 (secure 0 0 0) 0x7fb908079950 con 0x7fb9140b3d60 2023-11-28T20:53:20.170 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:20.170 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":121,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:15.615033+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:51:50.993341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-11-28T20:51:52.004867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":7}]},{"pgid":"14.2a","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6833/1902260958":"2023-11-29T20:53:15.614996+0000","172.21.15.190:6832/1902260958":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/2251272392":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/1361952934":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/2675112319":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/915400531":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/3484290831":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/1265166084":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/1549713842":"2023-11-29T20:53:15.614996+0000","172.21.15.190:6833/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:6832/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3239477442":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3809276606":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3450487150":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2166104670":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3808290090":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/4069818189":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2641549859":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/1724154865":"2023-11-28T21:51:59.768228+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:53:20.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.169+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb90006aed0 msgr2=0x7fb90006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.169+0000 7fb919d3c640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb90006aed0 0x7fb90006d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb904001610 tx=0x7fb904007930 comp rx=0 tx=0).stop 2023-11-28T20:53:20.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b3d60 msgr2=0x7fb9141f3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b3d60 0x7fb9141f3e50 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb908000c00 tx=0x7fb9080046c0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 shutdown_connections 2023-11-28T20:53:20.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb90006aed0 0x7fb90006d390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9141438b0 0x7fb9140712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9140b3d60 0x7fb9141f3e50 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 --2- 172.21.15.45:0/1537380512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9140b0c30 0x7fb91413ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 >> 172.21.15.45:0/1537380512 conn(0x7fb91406a500 msgr2=0x7fb91415a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:20.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 shutdown_connections 2023-11-28T20:53:20.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.170+0000 7fb919d3c640 1 -- 172.21.15.45:0/1537380512 wait complete. 2023-11-28T20:53:20.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T20:53:20.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:53:20.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:53:20.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.400+0000 7fbdd6ce0640 1 Processor -- start 2023-11-28T20:53:20.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.401+0000 7fbdd6ce0640 1 -- start start 2023-11-28T20:53:20.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.403+0000 7fbdd6ce0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbdd0070b20 0x7fbdd0070f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.403+0000 7fbdd6ce0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd00714c0 con 0x7fbdd00b65a0 2023-11-28T20:53:20.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.403+0000 7fbdd6ce0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd00d1c60 con 0x7fbdd015d620 2023-11-28T20:53:20.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.403+0000 7fbdd6ce0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd015da00 con 0x7fbdd0070b20 2023-11-28T20:53:20.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.403+0000 7fbdcffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbdd0070b20 0x7fbdd0070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.403+0000 7fbdcffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbdd0070b20 0x7fbdd0070f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38658/0 (socket says 172.21.15.45:38658) 2023-11-28T20:53:20.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.403+0000 7fbdd527d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbdd00b65a0 0x7fbdd00b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37988/0 (socket says 172.21.15.45:37988) 2023-11-28T20:53:20.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.403+0000 7fbdcffff640 1 -- 172.21.15.45:0/1725445073 learned_addr learned my addr 172.21.15.45:0/1725445073 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:20.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.404+0000 7fbdcffff640 1 -- 172.21.15.45:0/1725445073 >> v1:172.21.15.190:6789/0 conn(0x7fbdd015d620 legacy=0x7fbdd00d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:20.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.404+0000 7fbdcffff640 1 -- 172.21.15.45:0/1725445073 >> v1:172.21.15.45:6789/0 conn(0x7fbdd00b65a0 legacy=0x7fbdd00b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.404+0000 7fbdcf7fe640 1 -- 172.21.15.45:0/1725445073 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807324966 0 0) 0x7fbdd00714c0 con 0x7fbdd00b65a0 2023-11-28T20:53:20.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.404+0000 7fbdcffff640 1 -- 172.21.15.45:0/1725445073 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd00d2800 con 0x7fbdd0070b20 2023-11-28T20:53:20.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.406+0000 7fbdcffff640 1 --2- 172.21.15.45:0/1725445073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd0070b20 0x7fbdd0070f80 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc009870 tx=0x7fbdbc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d33515313ae03b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:20.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.406+0000 7fbdcf7fe640 1 -- 172.21.15.45:0/1725445073 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc030860 con 0x7fbdd0070b20 2023-11-28T20:53:20.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.406+0000 7fbdcf7fe640 1 -- 172.21.15.45:0/1725445073 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdbc0309c0 con 0x7fbdd0070b20 2023-11-28T20:53:20.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.406+0000 7fbdcf7fe640 1 -- 172.21.15.45:0/1725445073 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc030d50 con 0x7fbdd0070b20 2023-11-28T20:53:20.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.407+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1725445073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd0070b20 msgr2=0x7fbdd0070f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.407+0000 7fbdd6ce0640 1 --2- 172.21.15.45:0/1725445073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd0070b20 0x7fbdd0070f80 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc009870 tx=0x7fbdbc02b9a0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.407+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1725445073 shutdown_connections 2023-11-28T20:53:20.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.407+0000 7fbdd6ce0640 1 --2- 172.21.15.45:0/1725445073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd0070b20 0x7fbdd0070f80 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.407+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1725445073 >> 172.21.15.45:0/1725445073 conn(0x7fbdd00b5310 msgr2=0x7fbdd00cfc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:20.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.407+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1725445073 shutdown_connections 2023-11-28T20:53:20.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.407+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1725445073 wait complete. 2023-11-28T20:53:20.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.409+0000 7fbdd6ce0640 1 Processor -- start 2023-11-28T20:53:20.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.409+0000 7fbdd6ce0640 1 -- start start 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.410+0000 7fbdd6ce0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd0070b20 0x7fbdd0153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.410+0000 7fbdd6ce0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd00b65a0 0x7fbdd01f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.410+0000 7fbdd6ce0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd015d620 0x7fbdd00b8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.410+0000 7fbdd6ce0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0152f40 con 0x7fbdd015d620 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.410+0000 7fbdd6ce0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0153220 con 0x7fbdd00b65a0 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.410+0000 7fbdd6ce0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0153500 con 0x7fbdd0070b20 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.410+0000 7fbdd527d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd015d620 0x7fbdd00b8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdd527d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd015d620 0x7fbdd00b8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38662/0 (socket says 172.21.15.45:38662) 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdd527d640 1 -- 172.21.15.45:0/1675822581 learned_addr learned my addr 172.21.15.45:0/1675822581 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdcffff640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd00b65a0 0x7fbdd01f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdd4a7c640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd0070b20 0x7fbdd0153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdd527d640 1 -- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd0070b20 msgr2=0x7fbdd0153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdd527d640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd0070b20 0x7fbdd0153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdd527d640 1 -- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd00b65a0 msgr2=0x7fbdd01f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdd527d640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd00b65a0 0x7fbdd01f6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdd527d640 1 -- 172.21.15.45:0/1675822581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc0094d0 con 0x7fbdd015d620 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.411+0000 7fbdd527d640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd015d620 0x7fbdd00b8280 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c7e0 tx=0x7fbdc800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.412+0000 7fbdcd7fa640 1 -- 172.21.15.45:0/1675822581 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800a640 con 0x7fbdd015d620 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.412+0000 7fbdcd7fa640 1 -- 172.21.15.45:0/1675822581 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc800a7c0 con 0x7fbdd015d620 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.412+0000 7fbdcd7fa640 1 -- 172.21.15.45:0/1675822581 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800ab70 con 0x7fbdd015d620 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.412+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd01537e0 con 0x7fbdd015d620 2023-11-28T20:53:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.412+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd01f6cd0 con 0x7fbdd015d620 2023-11-28T20:53:20.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.413+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd0153d00 con 0x7fbdd015d620 2023-11-28T20:53:20.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.414+0000 7fbdcd7fa640 1 -- 172.21.15.45:0/1675822581 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbdc800acd0 con 0x7fbdd015d620 2023-11-28T20:53:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.415+0000 7fbdcd7fa640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbdb406af60 0x7fbdb406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.415+0000 7fbdcd7fa640 1 -- 172.21.15.45:0/1675822581 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6962+0+0 (secure 0 0 0) 0x7fbdc80891f0 con 0x7fbdd015d620 2023-11-28T20:53:20.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.416+0000 7fbdd4a7c640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbdb406af60 0x7fbdb406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.416+0000 7fbdd4a7c640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbdb406af60 0x7fbdb406d420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4006fd0 tx=0x7fbdc4008040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:20.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.420+0000 7fbdcd7fa640 1 -- 172.21.15.45:0/1675822581 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbdd0153d00 con 0x7fbdd015d620 2023-11-28T20:53:20.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.648+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbdd0153d00 con 0x7fbdd015d620 2023-11-28T20:53:20.651 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T20:53:20.649+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T20:53:20.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.652+0000 7fbdcd7fa640 1 -- 172.21.15.45:0/1675822581 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbdd0153d00 con 0x7fbdd015d620 2023-11-28T20:53:20.655 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T20:53:20.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.655+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbdb406af60 msgr2=0x7fbdb406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.655+0000 7fbdd6ce0640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbdb406af60 0x7fbdb406d420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4006fd0 tx=0x7fbdc4008040 comp rx=0 tx=0).stop 2023-11-28T20:53:20.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.655+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd015d620 msgr2=0x7fbdd00b8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.655+0000 7fbdd6ce0640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd015d620 0x7fbdd00b8280 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c7e0 tx=0x7fbdc800cca0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.656+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 shutdown_connections 2023-11-28T20:53:20.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.656+0000 7fbdd6ce0640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbdb406af60 0x7fbdb406d420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.656+0000 7fbdd6ce0640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdd015d620 0x7fbdd00b8280 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.656+0000 7fbdd6ce0640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd00b65a0 0x7fbdd01f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.656+0000 7fbdd6ce0640 1 --2- 172.21.15.45:0/1675822581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd0070b20 0x7fbdd0153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.656+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 >> 172.21.15.45:0/1675822581 conn(0x7fbdd00b5310 msgr2=0x7fbdd015eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:20.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.656+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 shutdown_connections 2023-11-28T20:53:20.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.656+0000 7fbdd6ce0640 1 -- 172.21.15.45:0/1675822581 wait complete. 2023-11-28T20:53:20.678 INFO:tasks.ceph.mds.a.smithi045.stderr: -7031> 2023-11-28T20:51:57.894+0000 7f30d6d6f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:53:20.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T20:53:20.729 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:53:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.873+0000 7f838a610640 1 Processor -- start 2023-11-28T20:53:20.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.873+0000 7f838a610640 1 -- start start 2023-11-28T20:53:20.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.875+0000 7f838a610640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f838406ac10 0x7f838406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.875+0000 7f838a610640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838415ecd0 con 0x7f838415e840 2023-11-28T20:53:20.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.875+0000 7f838a610640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838415f290 con 0x7f838415f720 2023-11-28T20:53:20.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.875+0000 7f838a610640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838415fbd0 con 0x7f838406ac10 2023-11-28T20:53:20.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.875+0000 7f8383fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f838415e840 0x7f838406a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37998/0 (socket says 172.21.15.45:37998) 2023-11-28T20:53:20.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.875+0000 7f8383fff640 1 -- 172.21.15.45:0/2927009689 learned_addr learned my addr 172.21.15.45:0/2927009689 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536912258 0 0) 0x7f838415ecd0 con 0x7f838415e840 2023-11-28T20:53:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8370003520 con 0x7f838415e840 2023-11-28T20:53:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f837bfff640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f838406ac10 0x7f838406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 134707396 0 0) 0x7f8370003520 con 0x7f838415e840 2023-11-28T20:53:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f838406ac10 msgr2=0x7f838406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f838406ac10 0x7f838406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 >> v1:172.21.15.190:6789/0 conn(0x7f838415f720 legacy=0x7f83840cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8384072020 con 0x7f838415e840 2023-11-28T20:53:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f8374001670 con 0x7f838415e840 2023-11-28T20:53:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 >> v1:172.21.15.45:6789/0 conn(0x7f838415e840 legacy=0x7f838406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83700039a0 0x7f8370003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f838406ac10 0x7f83700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8370004d00 0x7f8370005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838415fbd0 con 0x7f8370004d00 2023-11-28T20:53:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837000ff20 con 0x7f83700039a0 2023-11-28T20:53:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.876+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83700100b0 con 0x7f838406ac10 2023-11-28T20:53:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.877+0000 7f8388bad640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8370004d00 0x7f8370005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.877+0000 7f837bfff640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f838406ac10 0x7f83700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.877+0000 7f8383fff640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83700039a0 0x7f8370003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.877+0000 7f8388bad640 1 -- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f838406ac10 msgr2=0x7f83700047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.877+0000 7f8388bad640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f838406ac10 0x7f83700047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.877+0000 7f8388bad640 1 -- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83700039a0 msgr2=0x7f8370003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.877+0000 7f8388bad640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83700039a0 0x7f8370003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.877+0000 7f8388bad640 1 -- 172.21.15.45:0/2927009689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8370003520 con 0x7f8370004d00 2023-11-28T20:53:20.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.878+0000 7f8388bad640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8370004d00 0x7f8370005180 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f837c001e80 tx=0x7f83700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a77a3221f3adec server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:20.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.878+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c003b70 con 0x7f8370004d00 2023-11-28T20:53:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.878+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c003cd0 con 0x7f8370004d00 2023-11-28T20:53:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.878+0000 7f83837fe640 1 -- 172.21.15.45:0/2927009689 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c005310 con 0x7f8370004d00 2023-11-28T20:53:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.879+0000 7f838a610640 1 -- 172.21.15.45:0/2927009689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8370004d00 msgr2=0x7f8370005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.879+0000 7f838a610640 1 --2- 172.21.15.45:0/2927009689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8370004d00 0x7f8370005180 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f837c001e80 tx=0x7f83700057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.879+0000 7f8388bad640 1 -- 172.21.15.45:0/2927009689 reap_dead start 2023-11-28T20:53:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.879+0000 7f838a610640 1 -- 172.21.15.45:0/2927009689 shutdown_connections 2023-11-28T20:53:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.879+0000 7f838a610640 1 -- 172.21.15.45:0/2927009689 >> 172.21.15.45:0/2927009689 conn(0x7f83840ced60 msgr2=0x7f83840cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.879+0000 7f838a610640 1 -- 172.21.15.45:0/2927009689 shutdown_connections 2023-11-28T20:53:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.879+0000 7f838a610640 1 -- 172.21.15.45:0/2927009689 wait complete. 2023-11-28T20:53:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.881+0000 7f838a610640 1 Processor -- start 2023-11-28T20:53:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.881+0000 7f838a610640 1 -- start start 2023-11-28T20:53:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.882+0000 7f838a610640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370004d00 0x7f838409c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.882+0000 7f838a610640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83840b3d80 0x7f83841593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.882+0000 7f838a610640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83840b4200 0x7f8384159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.882+0000 7f838a610640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838415b2b0 con 0x7f83840b4200 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.882+0000 7f838a610640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838415afd0 con 0x7f8370004d00 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.882+0000 7f838a610640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83840afea0 con 0x7f83840b3d80 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.882+0000 7f8388bad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83840b4200 0x7f8384159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.882+0000 7f8388bad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83840b4200 0x7f8384159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38688/0 (socket says 172.21.15.45:38688) 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.882+0000 7f8388bad640 1 -- 172.21.15.45:0/1746270278 learned_addr learned my addr 172.21.15.45:0/1746270278 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f8383fff640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370004d00 0x7f838409c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f8388bad640 1 -- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83840b3d80 msgr2=0x7f83841593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f837bfff640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83840b3d80 0x7f83841593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f8388bad640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83840b3d80 0x7f83841593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f8388bad640 1 -- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370004d00 msgr2=0x7f838409c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f8388bad640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370004d00 0x7f838409c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f8388bad640 1 -- 172.21.15.45:0/1746270278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83840b0180 con 0x7f83840b4200 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f8388bad640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83840b4200 0x7f8384159900 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f837c005040 tx=0x7f837c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f83817fa640 1 -- 172.21.15.45:0/1746270278 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c03e070 con 0x7f83840b4200 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f83817fa640 1 -- 172.21.15.45:0/1746270278 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c007db0 con 0x7f83840b4200 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f83817fa640 1 -- 172.21.15.45:0/1746270278 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c006d70 con 0x7f83840b4200 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83840b0460 con 0x7f83840b4200 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.883+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838415ab60 con 0x7f83840b4200 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.884+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83840afd00 con 0x7f83840b4200 2023-11-28T20:53:20.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.885+0000 7f83817fa640 1 -- 172.21.15.45:0/1746270278 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f837c007030 con 0x7f83840b4200 2023-11-28T20:53:20.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.886+0000 7f83817fa640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f836406af60 0x7f836406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:20.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.886+0000 7f83817fa640 1 -- 172.21.15.45:0/1746270278 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 7048+0+0 (secure 0 0 0) 0x7f837c0ab090 con 0x7f83840b4200 2023-11-28T20:53:20.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.887+0000 7f8383fff640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f836406af60 0x7f836406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:20.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.888+0000 7f8383fff640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f836406af60 0x7f836406d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8374001c50 tx=0x7f8374000f10 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:20.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:20.890+0000 7f83817fa640 1 -- 172.21.15.45:0/1746270278 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83840afd00 con 0x7f83840b4200 2023-11-28T20:53:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.115+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f83840afd00 con 0x7f83840b4200 2023-11-28T20:53:21.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.670+0000 7f83817fa640 1 -- 172.21.15.45:0/1746270278 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) v1 ==== 104+0+0 (secure 0 0 0) 0x7f83840afd00 con 0x7f83840b4200 2023-11-28T20:53:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f836406af60 msgr2=0x7f836406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f836406af60 0x7f836406d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8374001c50 tx=0x7f8374000f10 comp rx=0 tx=0).stop 2023-11-28T20:53:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83840b4200 msgr2=0x7f8384159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83840b4200 0x7f8384159900 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f837c005040 tx=0x7f837c0014a0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 shutdown_connections 2023-11-28T20:53:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f836406af60 0x7f836406d420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83840b4200 0x7f8384159900 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83840b3d80 0x7f83841593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 --2- 172.21.15.45:0/1746270278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370004d00 0x7f838409c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 >> 172.21.15.45:0/1746270278 conn(0x7f83840ced60 msgr2=0x7f83840ac810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 shutdown_connections 2023-11-28T20:53:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.674+0000 7f838a610640 1 -- 172.21.15.45:0/1746270278 wait complete. 2023-11-28T20:53:21.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T20:53:21.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.970+0000 7f4100133640 1 Processor -- start 2023-11-28T20:53:21.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.970+0000 7f4100133640 1 -- start start 2023-11-28T20:53:21.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.972+0000 7f4100133640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f806c350 0x7f40f815f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:21.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.972+0000 7f4100133640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f815ecd0 con 0x7f40f815e840 2023-11-28T20:53:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.972+0000 7f4100133640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f815f290 con 0x7f40f806ac10 2023-11-28T20:53:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.972+0000 7f4100133640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8072020 con 0x7f40f806c350 2023-11-28T20:53:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.972+0000 7f40fdecf640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f40f815e840 0x7f40f806a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38012/0 (socket says 172.21.15.45:38012) 2023-11-28T20:53:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.972+0000 7f40fdecf640 1 -- 172.21.15.45:0/3957149211 learned_addr learned my addr 172.21.15.45:0/3957149211 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:21.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.972+0000 7f40fe6d0640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f806c350 0x7f40f815f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600671805 0 0) 0x7f40f815ecd0 con 0x7f40f815e840 2023-11-28T20:53:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e4003520 con 0x7f40f815e840 2023-11-28T20:53:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575532696 0 0) 0x7f40e4003520 con 0x7f40f815e840 2023-11-28T20:53:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f806c350 msgr2=0x7f40f815f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f806c350 0x7f40f815f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 >> v1:172.21.15.190:6789/0 conn(0x7f40f806ac10 legacy=0x7f40f80cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f8072300 con 0x7f40f815e840 2023-11-28T20:53:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f40f4001670 con 0x7f40f815e840 2023-11-28T20:53:21.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 >> v1:172.21.15.45:6789/0 conn(0x7f40f815e840 legacy=0x7f40f806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f806c350 0x7f40e40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e4003f20 0x7f40e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4004d00 0x7f40e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8072020 con 0x7f40e4003f20 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e400ba60 con 0x7f40e4004d00 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.973+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e400bc20 con 0x7f40f806c350 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.974+0000 7f40fd6ce640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e4003f20 0x7f40e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.974+0000 7f40fe6d0640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4004d00 0x7f40e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.974+0000 7f40fdecf640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f806c350 0x7f40e40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.974+0000 7f40fd6ce640 1 -- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f806c350 msgr2=0x7f40e40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.974+0000 7f40fd6ce640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f806c350 0x7f40e40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.974+0000 7f40fd6ce640 1 -- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4004d00 msgr2=0x7f40e4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:21.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.974+0000 7f40fd6ce640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4004d00 0x7f40e4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.974+0000 7f40fd6ce640 1 -- 172.21.15.45:0/3957149211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e4003520 con 0x7f40e4003f20 2023-11-28T20:53:21.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.974+0000 7f40fe6d0640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4004d00 0x7f40e4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:21.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.975+0000 7f40fd6ce640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e4003f20 0x7f40e40047c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f40e8001750 tx=0x7f40e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d66343e2c6e5f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:21.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e80052d0 con 0x7f40e4003f20 2023-11-28T20:53:21.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e802e050 con 0x7f40e4003f20 2023-11-28T20:53:21.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f40fcecd640 1 -- 172.21.15.45:0/3957149211 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8003830 con 0x7f40e4003f20 2023-11-28T20:53:21.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f4100133640 1 -- 172.21.15.45:0/3957149211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e4003f20 msgr2=0x7f40e40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:21.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f4100133640 1 --2- 172.21.15.45:0/3957149211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e4003f20 0x7f40e40047c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f40e8001750 tx=0x7f40e40057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f40fe6d0640 1 -- 172.21.15.45:0/3957149211 reap_dead start 2023-11-28T20:53:21.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f4100133640 1 -- 172.21.15.45:0/3957149211 shutdown_connections 2023-11-28T20:53:21.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f4100133640 1 -- 172.21.15.45:0/3957149211 >> 172.21.15.45:0/3957149211 conn(0x7f40f80ced60 msgr2=0x7f40f80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:21.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f4100133640 1 -- 172.21.15.45:0/3957149211 shutdown_connections 2023-11-28T20:53:21.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.976+0000 7f4100133640 1 -- 172.21.15.45:0/3957149211 wait complete. 2023-11-28T20:53:21.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.978+0000 7f4100133640 1 Processor -- start 2023-11-28T20:53:21.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.978+0000 7f4100133640 1 -- start start 2023-11-28T20:53:21.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.979+0000 7f4100133640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4003f20 0x7f40f81f1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:21.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.979+0000 7f4100133640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f815d910 0x7f40f81f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.979+0000 7f4100133640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40f81f4d20 0x7f40f81f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.979+0000 7f4100133640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80c2920 con 0x7f40f81f4d20 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.979+0000 7f4100133640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f81f17b0 con 0x7f40e4003f20 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.979+0000 7f4100133640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f81f56e0 con 0x7f40f815d910 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fe6d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40f81f4d20 0x7f40f81f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fe6d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40f81f4d20 0x7f40f81f51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38706/0 (socket says 172.21.15.45:38706) 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fe6d0640 1 -- 172.21.15.45:0/2710638192 learned_addr learned my addr 172.21.15.45:0/2710638192 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fd6ce640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f815d910 0x7f40f81f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fdecf640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4003f20 0x7f40f81f1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:21.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fe6d0640 1 -- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f815d910 msgr2=0x7f40f81f47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:21.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fe6d0640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f815d910 0x7f40f81f47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:21.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fe6d0640 1 -- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4003f20 msgr2=0x7f40f81f1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:21.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fe6d0640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4003f20 0x7f40f81f1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fe6d0640 1 -- 172.21.15.45:0/2710638192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f81f1a30 con 0x7f40f81f4d20 2023-11-28T20:53:22.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.980+0000 7f40fd6ce640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f815d910 0x7f40f81f47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:22.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.981+0000 7f40fe6d0640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40f81f4d20 0x7f40f81f51a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f40f80ae6d0 tx=0x7f40f0010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:22.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.981+0000 7f40ee7fc640 1 -- 172.21.15.45:0/2710638192 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f000e050 con 0x7f40f81f4d20 2023-11-28T20:53:22.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.981+0000 7f40ee7fc640 1 -- 172.21.15.45:0/2710638192 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f000f040 con 0x7f40f81f4d20 2023-11-28T20:53:22.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.981+0000 7f40ee7fc640 1 -- 172.21.15.45:0/2710638192 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0003990 con 0x7f40f81f4d20 2023-11-28T20:53:22.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.981+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f81f1d10 con 0x7f40f81f4d20 2023-11-28T20:53:22.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.981+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f81f22d0 con 0x7f40f81f4d20 2023-11-28T20:53:22.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.982+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f80733d0 con 0x7f40f81f4d20 2023-11-28T20:53:22.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.983+0000 7f40ee7fc640 1 -- 172.21.15.45:0/2710638192 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f40f0015440 con 0x7f40f81f4d20 2023-11-28T20:53:22.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.984+0000 7f40ee7fc640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f40dc06af60 0x7f40dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:22.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.984+0000 7f40ee7fc640 1 -- 172.21.15.45:0/2710638192 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 7048+0+0 (secure 0 0 0) 0x7f40f008a7e0 con 0x7f40f81f4d20 2023-11-28T20:53:22.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.985+0000 7f40fdecf640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f40dc06af60 0x7f40dc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:22.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.986+0000 7f40fdecf640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f40dc06af60 0x7f40dc06d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f40f4001c90 tx=0x7f40f4000ff0 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:22.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:21.988+0000 7f40ee7fc640 1 -- 172.21.15.45:0/2710638192 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40f0055120 con 0x7f40f81f4d20 2023-11-28T20:53:22.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.182+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f40f80ce2b0 con 0x7f40f81f4d20 2023-11-28T20:53:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.682+0000 7f40ee7fc640 1 -- 172.21.15.45:0/2710638192 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v123) v1 ==== 181+0+0 (secure 0 0 0) 0x7f40f0058740 con 0x7f40f81f4d20 2023-11-28T20:53:22.685 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T20:53:22.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.688+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f40dc06af60 msgr2=0x7f40dc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:22.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.688+0000 7f4100133640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f40dc06af60 0x7f40dc06d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f40f4001c90 tx=0x7f40f4000ff0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.688+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40f81f4d20 msgr2=0x7f40f81f51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:22.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.688+0000 7f4100133640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40f81f4d20 0x7f40f81f51a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f40f80ae6d0 tx=0x7f40f0010590 comp rx=0 tx=0).stop 2023-11-28T20:53:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.689+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 shutdown_connections 2023-11-28T20:53:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.689+0000 7f4100133640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f40dc06af60 0x7f40dc06d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.689+0000 7f4100133640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40f81f4d20 0x7f40f81f51a0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.689+0000 7f4100133640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40f815d910 0x7f40f81f47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.689+0000 7f4100133640 1 --2- 172.21.15.45:0/2710638192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40e4003f20 0x7f40f81f1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.689+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 >> 172.21.15.45:0/2710638192 conn(0x7f40f80ced60 msgr2=0x7f40f806a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:22.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.691+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 shutdown_connections 2023-11-28T20:53:22.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.692+0000 7f4100133640 1 -- 172.21.15.45:0/2710638192 wait complete. 2023-11-28T20:53:22.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T20:53:22.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.921+0000 7fbb78f47640 1 Processor -- start 2023-11-28T20:53:22.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.922+0000 7fbb78f47640 1 -- start start 2023-11-28T20:53:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.923+0000 7fbb78f47640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 0x7fbb740ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.923+0000 7fbb78f47640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7415d790 con 0x7fbb740b6380 2023-11-28T20:53:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.923+0000 7fbb78f47640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740ce7c0 con 0x7fbb7415d3b0 2023-11-28T20:53:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.923+0000 7fbb78f47640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740d2bf0 con 0x7fbb740cde00 2023-11-28T20:53:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.924+0000 7fbb73fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbb740b6380 0x7fbb740b6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38022/0 (socket says 172.21.15.45:38022) 2023-11-28T20:53:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.924+0000 7fbb73fff640 1 -- 172.21.15.45:0/2583080007 learned_addr learned my addr 172.21.15.45:0/2583080007 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.924+0000 7fbb6bfff640 1 --2- 172.21.15.45:0/2583080007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 0x7fbb740ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.924+0000 7fbb72ffd640 1 -- 172.21.15.45:0/2583080007 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132206015 0 0) 0x7fbb7415d790 con 0x7fbb740b6380 2023-11-28T20:53:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.925+0000 7fbb72ffd640 1 -- 172.21.15.45:0/2583080007 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb60003520 con 0x7fbb740b6380 2023-11-28T20:53:22.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.925+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2583080007 >> v1:172.21.15.190:6789/0 conn(0x7fbb7415d3b0 legacy=0x7fbb740d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:22.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.925+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2583080007 >> v1:172.21.15.45:6789/0 conn(0x7fbb740b6380 legacy=0x7fbb740b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:22.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.925+0000 7fbb72ffd640 1 -- 172.21.15.45:0/2583080007 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 304954227 0 0) 0x7fbb60003520 con 0x7fbb740b6380 2023-11-28T20:53:22.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.925+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2583080007 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740d2ed0 con 0x7fbb740cde00 2023-11-28T20:53:22.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.926+0000 7fbb6bfff640 1 --2- 172.21.15.45:0/2583080007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 0x7fbb740ce280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009750 tx=0x7fbb5c02b5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a616de903682bd2e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:22.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.927+0000 7fbb72ffd640 1 -- 172.21.15.45:0/2583080007 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0019f0 con 0x7fbb740cde00 2023-11-28T20:53:22.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.927+0000 7fbb72ffd640 1 -- 172.21.15.45:0/2583080007 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c02e050 con 0x7fbb740cde00 2023-11-28T20:53:22.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.927+0000 7fbb72ffd640 1 -- 172.21.15.45:0/2583080007 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c030730 con 0x7fbb740cde00 2023-11-28T20:53:22.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.928+0000 7fbb78f47640 1 -- 172.21.15.45:0/2583080007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 msgr2=0x7fbb740ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:22.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.928+0000 7fbb78f47640 1 --2- 172.21.15.45:0/2583080007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 0x7fbb740ce280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009750 tx=0x7fbb5c02b5e0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.928+0000 7fbb78f47640 1 -- 172.21.15.45:0/2583080007 shutdown_connections 2023-11-28T20:53:22.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.928+0000 7fbb78f47640 1 --2- 172.21.15.45:0/2583080007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 0x7fbb740ce280 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.928+0000 7fbb78f47640 1 -- 172.21.15.45:0/2583080007 >> 172.21.15.45:0/2583080007 conn(0x7fbb74070f40 msgr2=0x7fbb74071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:22.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.928+0000 7fbb78f47640 1 -- 172.21.15.45:0/2583080007 shutdown_connections 2023-11-28T20:53:22.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.928+0000 7fbb78f47640 1 -- 172.21.15.45:0/2583080007 wait complete. 2023-11-28T20:53:22.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.931+0000 7fbb78f47640 1 Processor -- start 2023-11-28T20:53:22.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.931+0000 7fbb78f47640 1 -- start start 2023-11-28T20:53:22.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.932+0000 7fbb78f47640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb740b6380 0x7fbb740c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:22.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.932+0000 7fbb78f47640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 0x7fbb741f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:22.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.932+0000 7fbb78f47640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415d3b0 0x7fbb740c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:22.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.932+0000 7fbb78f47640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7415fd90 con 0x7fbb7415d3b0 2023-11-28T20:53:22.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.932+0000 7fbb78f47640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb741f3700 con 0x7fbb740b6380 2023-11-28T20:53:22.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.932+0000 7fbb78f47640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7415fab0 con 0x7fbb740cde00 2023-11-28T20:53:22.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb73fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415d3b0 0x7fbb740c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:22.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb73fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415d3b0 0x7fbb740c6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38716/0 (socket says 172.21.15.45:38716) 2023-11-28T20:53:22.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb73fff640 1 -- 172.21.15.45:0/1311476547 learned_addr learned my addr 172.21.15.45:0/1311476547 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:22.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb737fe640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb740b6380 0x7fbb740c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:22.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb6bfff640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 0x7fbb741f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb73fff640 1 -- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 msgr2=0x7fbb741f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb73fff640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 0x7fbb741f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb73fff640 1 -- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb740b6380 msgr2=0x7fbb740c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb73fff640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb740b6380 0x7fbb740c9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb73fff640 1 -- 172.21.15.45:0/1311476547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c009400 con 0x7fbb7415d3b0 2023-11-28T20:53:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.933+0000 7fbb737fe640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb740b6380 0x7fbb740c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.934+0000 7fbb73fff640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415d3b0 0x7fbb740c6e70 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00cac0 tx=0x7fbb6c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.934+0000 7fbb70ff9640 1 -- 172.21.15.45:0/1311476547 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00a700 con 0x7fbb7415d3b0 2023-11-28T20:53:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.934+0000 7fbb70ff9640 1 -- 172.21.15.45:0/1311476547 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c00a860 con 0x7fbb7415d3b0 2023-11-28T20:53:22.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.934+0000 7fbb70ff9640 1 -- 172.21.15.45:0/1311476547 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00ac50 con 0x7fbb7415d3b0 2023-11-28T20:53:22.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.934+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb740cb2f0 con 0x7fbb7415d3b0 2023-11-28T20:53:22.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.934+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb740cb8b0 con 0x7fbb7415d3b0 2023-11-28T20:53:22.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.935+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb740cb150 con 0x7fbb7415d3b0 2023-11-28T20:53:22.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.936+0000 7fbb70ff9640 1 -- 172.21.15.45:0/1311476547 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbb6c0070d0 con 0x7fbb7415d3b0 2023-11-28T20:53:22.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.937+0000 7fbb70ff9640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbb5406afb0 0x7fbb5406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:22.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.938+0000 7fbb70ff9640 1 -- 172.21.15.45:0/1311476547 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6610+0+0 (secure 0 0 0) 0x7fbb6c089530 con 0x7fbb7415d3b0 2023-11-28T20:53:22.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.938+0000 7fbb737fe640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbb5406afb0 0x7fbb5406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.939+0000 7fbb737fe640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbb5406afb0 0x7fbb5406d470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbb64007fd0 tx=0x7fbb64009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:22.943+0000 7fbb70ff9640 1 -- 172.21.15.45:0/1311476547 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb740cb150 con 0x7fbb7415d3b0 2023-11-28T20:53:23.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.138+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbb740cb150 con 0x7fbb7415d3b0 2023-11-28T20:53:23.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.705+0000 7fbb70ff9640 1 -- 172.21.15.45:0/1311476547 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v124) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbb740cb150 con 0x7fbb7415d3b0 2023-11-28T20:53:23.707 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T20:53:23.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.710+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbb5406afb0 msgr2=0x7fbb5406d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:23.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.710+0000 7fbb78f47640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbb5406afb0 0x7fbb5406d470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbb64007fd0 tx=0x7fbb64009040 comp rx=0 tx=0).stop 2023-11-28T20:53:23.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.710+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415d3b0 msgr2=0x7fbb740c6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:23.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.710+0000 7fbb78f47640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415d3b0 0x7fbb740c6e70 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00cac0 tx=0x7fbb6c00cf80 comp rx=0 tx=0).stop 2023-11-28T20:53:23.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.712+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 shutdown_connections 2023-11-28T20:53:23.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.712+0000 7fbb78f47640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbb5406afb0 0x7fbb5406d470 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:23.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.712+0000 7fbb78f47640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415d3b0 0x7fbb740c6e70 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:23.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.712+0000 7fbb78f47640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740cde00 0x7fbb741f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:23.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.712+0000 7fbb78f47640 1 --2- 172.21.15.45:0/1311476547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb740b6380 0x7fbb740c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:23.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.712+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 >> 172.21.15.45:0/1311476547 conn(0x7fbb74070f40 msgr2=0x7fbb740b51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:23.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.714+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 shutdown_connections 2023-11-28T20:53:23.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.715+0000 7fbb78f47640 1 -- 172.21.15.45:0/1311476547 wait complete. 2023-11-28T20:53:23.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T20:53:23.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.944+0000 7f8507339640 1 Processor -- start 2023-11-28T20:53:23.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.944+0000 7f8507339640 1 -- start start 2023-11-28T20:53:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.946+0000 7f8507339640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f850006ac10 0x7f850006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.946+0000 7f8507339640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850015ecd0 con 0x7f850015e840 2023-11-28T20:53:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.946+0000 7f8507339640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850015f290 con 0x7f850015f720 2023-11-28T20:53:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.946+0000 7f8507339640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850015fbd0 con 0x7f850006ac10 2023-11-28T20:53:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.946+0000 7f85050d5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f850015e840 0x7f850006a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42674/0 (socket says 172.21.15.45:42674) 2023-11-28T20:53:23.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.946+0000 7f85050d5640 1 -- 172.21.15.45:0/3019030218 learned_addr learned my addr 172.21.15.45:0/3019030218 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.946+0000 7f85048d4640 1 --2- 172.21.15.45:0/3019030218 >> v2:172.21.15.45:3300/0 conn(0x7f850006ac10 0x7f850006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.947+0000 7f84f7fff640 1 -- 172.21.15.45:0/3019030218 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73585682 0 0) 0x7f850015ecd0 con 0x7f850015e840 2023-11-28T20:53:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.947+0000 7f84f7fff640 1 -- 172.21.15.45:0/3019030218 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84ec003520 con 0x7f850015e840 2023-11-28T20:53:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.947+0000 7f85048d4640 1 -- 172.21.15.45:0/3019030218 >> v1:172.21.15.190:6789/0 conn(0x7f850015f720 legacy=0x7f85000cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:23.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.947+0000 7f85048d4640 1 -- 172.21.15.45:0/3019030218 >> v1:172.21.15.45:6789/0 conn(0x7f850015e840 legacy=0x7f850006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:23.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.947+0000 7f85048d4640 1 -- 172.21.15.45:0/3019030218 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8500072020 con 0x7f850006ac10 2023-11-28T20:53:23.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.948+0000 7f85048d4640 1 --2- 172.21.15.45:0/3019030218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 0x7f850006c350 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f84f00097a0 tx=0x7f84f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41168b88d9ec7851 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f84f7fff640 1 -- 172.21.15.45:0/3019030218 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0031830 con 0x7f850006ac10 2023-11-28T20:53:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f84f7fff640 1 -- 172.21.15.45:0/3019030218 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f0031990 con 0x7f850006ac10 2023-11-28T20:53:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f84f7fff640 1 -- 172.21.15.45:0/3019030218 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0031d20 con 0x7f850006ac10 2023-11-28T20:53:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f8507339640 1 -- 172.21.15.45:0/3019030218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 msgr2=0x7f850006c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:23.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f8507339640 1 --2- 172.21.15.45:0/3019030218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 0x7f850006c350 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f84f00097a0 tx=0x7f84f002b8d0 comp rx=0 tx=0).stop 2023-11-28T20:53:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f8507339640 1 -- 172.21.15.45:0/3019030218 shutdown_connections 2023-11-28T20:53:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f8507339640 1 --2- 172.21.15.45:0/3019030218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 0x7f850006c350 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f8507339640 1 -- 172.21.15.45:0/3019030218 >> 172.21.15.45:0/3019030218 conn(0x7f85000ced60 msgr2=0x7f85000cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f8507339640 1 -- 172.21.15.45:0/3019030218 shutdown_connections 2023-11-28T20:53:23.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.949+0000 7f8507339640 1 -- 172.21.15.45:0/3019030218 wait complete. 2023-11-28T20:53:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.951+0000 7f8507339640 1 Processor -- start 2023-11-28T20:53:23.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.951+0000 7f8507339640 1 -- start start 2023-11-28T20:53:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.952+0000 7f8507339640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 0x7f85000c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.952+0000 7f8507339640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f850015e840 0x7f85001e4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.952+0000 7f8507339640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f850015f720 0x7f85001e5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.952+0000 7f8507339640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85001e5d60 con 0x7f850006ac10 2023-11-28T20:53:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.952+0000 7f8507339640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850015b040 con 0x7f850015f720 2023-11-28T20:53:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.952+0000 7f8507339640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85001e5550 con 0x7f850015e840 2023-11-28T20:53:23.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.952+0000 7f85050d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 0x7f85000c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85050d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 0x7f85000c1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59024/0 (socket says 172.21.15.45:59024) 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85050d5640 1 -- 172.21.15.45:0/1119198767 learned_addr learned my addr 172.21.15.45:0/1119198767 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85048d4640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f850015e840 0x7f85001e4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85058d6640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f850015f720 0x7f85001e5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85050d5640 1 -- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f850015e840 msgr2=0x7f85001e4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85050d5640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f850015e840 0x7f85001e4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85050d5640 1 -- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f850015f720 msgr2=0x7f85001e5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85050d5640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f850015f720 0x7f85001e5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85050d5640 1 -- 172.21.15.45:0/1119198767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f0009400 con 0x7f850006ac10 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85058d6640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f850015f720 0x7f85001e5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:23.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.953+0000 7f85050d5640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 0x7f85000c1a10 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00c500 tx=0x7f84fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.954+0000 7f84f5ffb640 1 -- 172.21.15.45:0/1119198767 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc005ce0 con 0x7f850006ac10 2023-11-28T20:53:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.954+0000 7f84f5ffb640 1 -- 172.21.15.45:0/1119198767 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc005e40 con 0x7f850006ac10 2023-11-28T20:53:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.954+0000 7f84f5ffb640 1 -- 172.21.15.45:0/1119198767 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc007330 con 0x7f850006ac10 2023-11-28T20:53:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.954+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85000c2a30 con 0x7f850006ac10 2023-11-28T20:53:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.954+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85001e5a80 con 0x7f850006ac10 2023-11-28T20:53:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.955+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85000733d0 con 0x7f850006ac10 2023-11-28T20:53:23.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.956+0000 7f84f5ffb640 1 -- 172.21.15.45:0/1119198767 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f84fc007490 con 0x7f850006ac10 2023-11-28T20:53:23.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.957+0000 7f84f5ffb640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84e406af60 0x7f84e406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:23.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.957+0000 7f84f5ffb640 1 -- 172.21.15.45:0/1119198767 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7f84fc088ef0 con 0x7f850006ac10 2023-11-28T20:53:23.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.958+0000 7f85048d4640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84e406af60 0x7f84e406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:23.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.959+0000 7f85048d4640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84e406af60 0x7f84e406d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f84f0036000 tx=0x7f84f0008030 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:23.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:23.962+0000 7f84f5ffb640 1 -- 172.21.15.45:0/1119198767 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84fc053e70 con 0x7f850006ac10 2023-11-28T20:53:24.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.162+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f850006a6a0 con 0x7f850006ac10 2023-11-28T20:53:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.168+0000 7f84f5ffb640 1 -- 172.21.15.45:0/1119198767 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v37) v1 ==== 120+0+0 (secure 0 0 0) 0x7f84fc057490 con 0x7f850006ac10 2023-11-28T20:53:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84e406af60 msgr2=0x7f84e406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84e406af60 0x7f84e406d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f84f0036000 tx=0x7f84f0008030 comp rx=0 tx=0).stop 2023-11-28T20:53:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 msgr2=0x7f85000c1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 0x7f85000c1a10 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00c500 tx=0x7f84fc00c9c0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 shutdown_connections 2023-11-28T20:53:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84e406af60 0x7f84e406d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f850015f720 0x7f85001e5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f850015e840 0x7f85001e4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 --2- 172.21.15.45:0/1119198767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850006ac10 0x7f85000c1a10 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 >> 172.21.15.45:0/1119198767 conn(0x7f85000ced60 msgr2=0x7f85000701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 shutdown_connections 2023-11-28T20:53:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.171+0000 7f8507339640 1 -- 172.21.15.45:0/1119198767 wait complete. 2023-11-28T20:53:24.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T20:53:24.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.412+0000 7fc49a178640 1 Processor -- start 2023-11-28T20:53:24.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.413+0000 7fc49a178640 1 -- start start 2023-11-28T20:53:24.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.414+0000 7fc49a178640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc49415e840 0x7fc49406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.414+0000 7fc49a178640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc49406ac10 0x7fc4940717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.414+0000 7fc49a178640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49415ecd0 con 0x7fc49415f720 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.414+0000 7fc49a178640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc494071ce0 con 0x7fc49406ac10 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.414+0000 7fc49a178640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49415f290 con 0x7fc49415e840 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.414+0000 7fc493fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc49406ac10 0x7fc4940717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.415+0000 7fc493fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc49406ac10 0x7fc4940717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59026/0 (socket says 172.21.15.45:59026) 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.415+0000 7fc493fff640 1 -- 172.21.15.45:0/4197304573 learned_addr learned my addr 172.21.15.45:0/4197304573 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.415+0000 7fc4937fe640 1 --2- 172.21.15.45:0/4197304573 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc49415e840 0x7fc49406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.415+0000 7fc493fff640 1 -- 172.21.15.45:0/4197304573 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc49415e840 msgr2=0x7fc49406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.415+0000 7fc493fff640 1 --2- 172.21.15.45:0/4197304573 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc49415e840 0x7fc49406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.415+0000 7fc493fff640 1 -- 172.21.15.45:0/4197304573 >> v1:172.21.15.45:6789/0 conn(0x7fc49415f720 legacy=0x7fc49406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.415+0000 7fc493fff640 1 -- 172.21.15.45:0/4197304573 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49415fb60 con 0x7fc49406ac10 2023-11-28T20:53:24.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.416+0000 7fc493fff640 1 --2- 172.21.15.45:0/4197304573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 0x7fc4940717a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009870 tx=0x7fc48c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392da6fbe3a5f6cb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.417+0000 7fc4927fc640 1 -- 172.21.15.45:0/4197304573 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c030840 con 0x7fc49406ac10 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.417+0000 7fc4927fc640 1 -- 172.21.15.45:0/4197304573 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48c0309a0 con 0x7fc49406ac10 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.417+0000 7fc4927fc640 1 -- 172.21.15.45:0/4197304573 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c030d30 con 0x7fc49406ac10 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.417+0000 7fc49a178640 1 -- 172.21.15.45:0/4197304573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 msgr2=0x7fc4940717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.417+0000 7fc49a178640 1 --2- 172.21.15.45:0/4197304573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 0x7fc4940717a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009870 tx=0x7fc48c02b980 comp rx=0 tx=0).stop 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.417+0000 7fc49a178640 1 -- 172.21.15.45:0/4197304573 shutdown_connections 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.417+0000 7fc49a178640 1 --2- 172.21.15.45:0/4197304573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 0x7fc4940717a0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.417+0000 7fc49a178640 1 --2- 172.21.15.45:0/4197304573 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc49415e840 0x7fc49406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.417+0000 7fc49a178640 1 -- 172.21.15.45:0/4197304573 >> 172.21.15.45:0/4197304573 conn(0x7fc4940ced60 msgr2=0x7fc4940cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:24.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.418+0000 7fc49a178640 1 -- 172.21.15.45:0/4197304573 shutdown_connections 2023-11-28T20:53:24.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.418+0000 7fc49a178640 1 -- 172.21.15.45:0/4197304573 wait complete. 2023-11-28T20:53:24.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.420+0000 7fc49a178640 1 Processor -- start 2023-11-28T20:53:24.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.420+0000 7fc49a178640 1 -- start start 2023-11-28T20:53:24.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc49a178640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 0x7fc4940c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc49a178640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc49415e840 0x7fc4940c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc49a178640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc49415f720 0x7fc4941f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc49a178640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc494157b30 con 0x7fc49406ac10 2023-11-28T20:53:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc49a178640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49415aa60 con 0x7fc49415e840 2023-11-28T20:53:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc49a178640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49415acb0 con 0x7fc49415f720 2023-11-28T20:53:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc4937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 0x7fc4940c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc4937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 0x7fc4940c8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59040/0 (socket says 172.21.15.45:59040) 2023-11-28T20:53:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc4937fe640 1 -- 172.21.15.45:0/2502006359 learned_addr learned my addr 172.21.15.45:0/2502006359 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc4937fe640 1 -- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc49415f720 msgr2=0x7fc4941f6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.421+0000 7fc4937fe640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc49415f720 0x7fc4941f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.422+0000 7fc4937fe640 1 -- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc49415e840 msgr2=0x7fc4940c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.422+0000 7fc4937fe640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc49415e840 0x7fc4940c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.422+0000 7fc4937fe640 1 -- 172.21.15.45:0/2502006359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c0094d0 con 0x7fc49406ac10 2023-11-28T20:53:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.422+0000 7fc4937fe640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 0x7fc4940c8000 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fc484007ef0 tx=0x7fc48400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.422+0000 7fc48bfff640 1 -- 172.21.15.45:0/2502006359 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484014620 con 0x7fc49406ac10 2023-11-28T20:53:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.422+0000 7fc48bfff640 1 -- 172.21.15.45:0/2502006359 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc484014780 con 0x7fc49406ac10 2023-11-28T20:53:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.422+0000 7fc48bfff640 1 -- 172.21.15.45:0/2502006359 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484014b30 con 0x7fc49406ac10 2023-11-28T20:53:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.422+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49415af90 con 0x7fc49406ac10 2023-11-28T20:53:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.422+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49415b550 con 0x7fc49406ac10 2023-11-28T20:53:24.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.424+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4940733d0 con 0x7fc49406ac10 2023-11-28T20:53:24.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.425+0000 7fc48bfff640 1 -- 172.21.15.45:0/2502006359 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc484014c90 con 0x7fc49406ac10 2023-11-28T20:53:24.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.426+0000 7fc48bfff640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc47406af60 0x7fc47406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.426+0000 7fc48bfff640 1 -- 172.21.15.45:0/2502006359 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7fc48408a300 con 0x7fc49406ac10 2023-11-28T20:53:24.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.427+0000 7fc492ffd640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc47406af60 0x7fc47406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.429+0000 7fc492ffd640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc47406af60 0x7fc47406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc47c000f60 tx=0x7fc47c0079b0 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:24.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.431+0000 7fc48bfff640 1 -- 172.21.15.45:0/2502006359 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc484055130 con 0x7fc49406ac10 2023-11-28T20:53:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.626+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc4940ce2b0 con 0x7fc49406ac10 2023-11-28T20:53:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.633+0000 7fc48bfff640 1 -- 172.21.15.45:0/2502006359 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc484058750 con 0x7fc49406ac10 2023-11-28T20:53:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc47406af60 msgr2=0x7fc47406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc47406af60 0x7fc47406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc47c000f60 tx=0x7fc47c0079b0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 msgr2=0x7fc4940c8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 0x7fc4940c8000 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fc484007ef0 tx=0x7fc48400e4b0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 shutdown_connections 2023-11-28T20:53:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc47406af60 0x7fc47406d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc49415f720 0x7fc4941f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc49415e840 0x7fc4940c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 --2- 172.21.15.45:0/2502006359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc49406ac10 0x7fc4940c8000 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 >> 172.21.15.45:0/2502006359 conn(0x7fc4940ced60 msgr2=0x7fc4940adfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 shutdown_connections 2023-11-28T20:53:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.636+0000 7fc49a178640 1 -- 172.21.15.45:0/2502006359 wait complete. 2023-11-28T20:53:24.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T20:53:24.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.878+0000 7fde373c9640 1 Processor -- start 2023-11-28T20:53:24.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.878+0000 7fde373c9640 1 -- start start 2023-11-28T20:53:24.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.879+0000 7fde373c9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fde3006dc50 0x7fde300cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.879+0000 7fde373c9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fde3006ac10 0x7fde300c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.879+0000 7fde373c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3006e030 con 0x7fde300ce600 2023-11-28T20:53:24.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.879+0000 7fde373c9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde300ccf60 con 0x7fde3006ac10 2023-11-28T20:53:24.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.879+0000 7fde373c9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde300cd930 con 0x7fde3006dc50 2023-11-28T20:53:24.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.880+0000 7fde35966640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fde3006ac10 0x7fde300c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.880+0000 7fde34964640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fde300ce600 0x7fde3006a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42690/0 (socket says 172.21.15.45:42690) 2023-11-28T20:53:24.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.880+0000 7fde34964640 1 -- 172.21.15.45:0/3289186634 learned_addr learned my addr 172.21.15.45:0/3289186634 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:24.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.880+0000 7fde35165640 1 --2- 172.21.15.45:0/3289186634 >> v2:172.21.15.190:3300/0 conn(0x7fde3006dc50 0x7fde300cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.880+0000 7fde35966640 1 -- 172.21.15.45:0/3289186634 >> v2:172.21.15.190:3300/0 conn(0x7fde3006dc50 msgr2=0x7fde300cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:24.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.880+0000 7fde35966640 1 --2- 172.21.15.45:0/3289186634 >> v2:172.21.15.190:3300/0 conn(0x7fde3006dc50 0x7fde300cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.880+0000 7fde35966640 1 -- 172.21.15.45:0/3289186634 >> v1:172.21.15.45:6789/0 conn(0x7fde300ce600 legacy=0x7fde3006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.880+0000 7fde35966640 1 -- 172.21.15.45:0/3289186634 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde300cc930 con 0x7fde3006ac10 2023-11-28T20:53:24.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.881+0000 7fde35966640 1 --2- 172.21.15.45:0/3289186634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde3006ac10 0x7fde300c93b0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fde3006b230 tx=0x7fde2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d6e2d63e32c076 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.882+0000 7fde2ffff640 1 -- 172.21.15.45:0/3289186634 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde28031830 con 0x7fde3006ac10 2023-11-28T20:53:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.882+0000 7fde2ffff640 1 -- 172.21.15.45:0/3289186634 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde28031990 con 0x7fde3006ac10 2023-11-28T20:53:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.882+0000 7fde2ffff640 1 -- 172.21.15.45:0/3289186634 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde28031d20 con 0x7fde3006ac10 2023-11-28T20:53:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.882+0000 7fde373c9640 1 -- 172.21.15.45:0/3289186634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde3006ac10 msgr2=0x7fde300c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.882+0000 7fde373c9640 1 --2- 172.21.15.45:0/3289186634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde3006ac10 0x7fde300c93b0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fde3006b230 tx=0x7fde2802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.883+0000 7fde373c9640 1 -- 172.21.15.45:0/3289186634 shutdown_connections 2023-11-28T20:53:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.883+0000 7fde373c9640 1 --2- 172.21.15.45:0/3289186634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde3006ac10 0x7fde300c93b0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.883+0000 7fde373c9640 1 --2- 172.21.15.45:0/3289186634 >> v2:172.21.15.190:3300/0 conn(0x7fde3006dc50 0x7fde300cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.883+0000 7fde373c9640 1 -- 172.21.15.45:0/3289186634 >> 172.21.15.45:0/3289186634 conn(0x7fde300ced60 msgr2=0x7fde300cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:24.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.883+0000 7fde373c9640 1 -- 172.21.15.45:0/3289186634 shutdown_connections 2023-11-28T20:53:24.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.883+0000 7fde373c9640 1 -- 172.21.15.45:0/3289186634 wait complete. 2023-11-28T20:53:24.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.884+0000 7fde373c9640 1 Processor -- start 2023-11-28T20:53:24.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.885+0000 7fde373c9640 1 -- start start 2023-11-28T20:53:24.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde373c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde3006ac10 0x7fde30157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde373c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde3006dc50 0x7fde3009b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde373c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde300ce600 0x7fde300c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.891 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde373c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde30158410 con 0x7fde300ce600 2023-11-28T20:53:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde373c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde300c63d0 con 0x7fde3006dc50 2023-11-28T20:53:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde373c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde300c66b0 con 0x7fde3006ac10 2023-11-28T20:53:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde35966640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde300ce600 0x7fde300c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde35966640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde300ce600 0x7fde300c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59058/0 (socket says 172.21.15.45:59058) 2023-11-28T20:53:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde35966640 1 -- 172.21.15.45:0/1569617354 learned_addr learned my addr 172.21.15.45:0/1569617354 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde35165640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde3006ac10 0x7fde30157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde34964640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde3006dc50 0x7fde3009b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde35966640 1 -- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde3006ac10 msgr2=0x7fde30157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde35966640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde3006ac10 0x7fde30157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde35966640 1 -- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde3006dc50 msgr2=0x7fde3009b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde35966640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde3006dc50 0x7fde3009b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.886+0000 7fde35966640 1 -- 172.21.15.45:0/1569617354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde28009400 con 0x7fde300ce600 2023-11-28T20:53:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.887+0000 7fde35966640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde300ce600 0x7fde300c7470 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fde28034040 tx=0x7fde280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.887+0000 7fde2dffb640 1 -- 172.21.15.45:0/1569617354 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2803a830 con 0x7fde300ce600 2023-11-28T20:53:24.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.887+0000 7fde2dffb640 1 -- 172.21.15.45:0/1569617354 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2803a9b0 con 0x7fde300ce600 2023-11-28T20:53:24.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.887+0000 7fde2dffb640 1 -- 172.21.15.45:0/1569617354 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2803ad60 con 0x7fde300ce600 2023-11-28T20:53:24.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.887+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde300c6990 con 0x7fde300ce600 2023-11-28T20:53:24.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.887+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde300c6f50 con 0x7fde300ce600 2023-11-28T20:53:24.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.888+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde300c7140 con 0x7fde300ce600 2023-11-28T20:53:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.889+0000 7fde2dffb640 1 -- 172.21.15.45:0/1569617354 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fde280443f0 con 0x7fde300ce600 2023-11-28T20:53:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.890+0000 7fde2dffb640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fde1406af90 0x7fde1406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.890+0000 7fde2dffb640 1 -- 172.21.15.45:0/1569617354 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7fde280abca0 con 0x7fde300ce600 2023-11-28T20:53:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.891+0000 7fde35165640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fde1406af90 0x7fde1406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.892+0000 7fde35165640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fde1406af90 0x7fde1406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde24009f30 tx=0x7fde2400b040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:24.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:24.895+0000 7fde2dffb640 1 -- 172.21.15.45:0/1569617354 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde280769d0 con 0x7fde300ce600 2023-11-28T20:53:25.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.099+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fde300733d0 con 0x7fde300ce600 2023-11-28T20:53:25.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.104+0000 7fde2dffb640 1 -- 172.21.15.45:0/1569617354 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v39) v1 ==== 115+0+0 (secure 0 0 0) 0x7fde28035070 con 0x7fde300ce600 2023-11-28T20:53:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fde1406af90 msgr2=0x7fde1406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fde1406af90 0x7fde1406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde24009f30 tx=0x7fde2400b040 comp rx=0 tx=0).stop 2023-11-28T20:53:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde300ce600 msgr2=0x7fde300c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde300ce600 0x7fde300c7470 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fde28034040 tx=0x7fde280046c0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 shutdown_connections 2023-11-28T20:53:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fde1406af90 0x7fde1406d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde300ce600 0x7fde300c7470 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde3006dc50 0x7fde3009b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 --2- 172.21.15.45:0/1569617354 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde3006ac10 0x7fde30157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 >> 172.21.15.45:0/1569617354 conn(0x7fde300ced60 msgr2=0x7fde3006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.107+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 shutdown_connections 2023-11-28T20:53:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.108+0000 7fde373c9640 1 -- 172.21.15.45:0/1569617354 wait complete. 2023-11-28T20:53:25.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-11-28T20:53:25.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.346+0000 7f1bc759e640 1 Processor -- start 2023-11-28T20:53:25.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.346+0000 7f1bc759e640 1 -- start start 2023-11-28T20:53:25.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.347+0000 7f1bc759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc815d550 0x7f1bc80cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1bc80d1c80 0x7f1bc80b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc759e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc80d0560 con 0x7f1bc80d0100 2023-11-28T20:53:25.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc759e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc80d2230 con 0x7f1bc80d1c80 2023-11-28T20:53:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc80d2510 con 0x7f1bc815d550 2023-11-28T20:53:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc6d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1bc80d1c80 0x7f1bc80b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc6d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1bc80d1c80 0x7f1bc80b6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59070/0 (socket says 172.21.15.45:59070) 2023-11-28T20:53:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc6d9d640 1 -- 172.21.15.45:0/2879433132 learned_addr learned my addr 172.21.15.45:0/2879433132 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc659c640 1 --2- 172.21.15.45:0/2879433132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc815d550 0x7f1bc80cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc6d9d640 1 -- 172.21.15.45:0/2879433132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc815d550 msgr2=0x7f1bc80cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:25.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc6d9d640 1 --2- 172.21.15.45:0/2879433132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc815d550 0x7f1bc80cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc6d9d640 1 -- 172.21.15.45:0/2879433132 >> v1:172.21.15.190:6789/0 conn(0x7f1bc80d0100 legacy=0x7f1bc80d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.348+0000 7f1bc6d9d640 1 -- 172.21.15.45:0/2879433132 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc80d27f0 con 0x7f1bc80d1c80 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.350+0000 7f1bc6d9d640 1 --2- 172.21.15.45:0/2879433132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 0x7f1bc80b6f50 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009870 tx=0x7f1bc002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb68eab15c96f72 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.350+0000 7f1bc559a640 1 -- 172.21.15.45:0/2879433132 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0031640 con 0x7f1bc80d1c80 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.350+0000 7f1bc559a640 1 -- 172.21.15.45:0/2879433132 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc00317a0 con 0x7f1bc80d1c80 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.350+0000 7f1bc559a640 1 -- 172.21.15.45:0/2879433132 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0031b30 con 0x7f1bc80d1c80 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.351+0000 7f1bc759e640 1 -- 172.21.15.45:0/2879433132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 msgr2=0x7f1bc80b6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.351+0000 7f1bc759e640 1 --2- 172.21.15.45:0/2879433132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 0x7f1bc80b6f50 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009870 tx=0x7f1bc002b8f0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.351+0000 7f1bc759e640 1 -- 172.21.15.45:0/2879433132 shutdown_connections 2023-11-28T20:53:25.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.351+0000 7f1bc759e640 1 --2- 172.21.15.45:0/2879433132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 0x7f1bc80b6f50 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.351+0000 7f1bc759e640 1 --2- 172.21.15.45:0/2879433132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc815d550 0x7f1bc80cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.351+0000 7f1bc759e640 1 -- 172.21.15.45:0/2879433132 >> 172.21.15.45:0/2879433132 conn(0x7f1bc806ac70 msgr2=0x7f1bc80b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:25.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.351+0000 7f1bc759e640 1 -- 172.21.15.45:0/2879433132 shutdown_connections 2023-11-28T20:53:25.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.351+0000 7f1bc759e640 1 -- 172.21.15.45:0/2879433132 wait complete. 2023-11-28T20:53:25.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.353+0000 7f1bc759e640 1 Processor -- start 2023-11-28T20:53:25.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.353+0000 7f1bc759e640 1 -- start start 2023-11-28T20:53:25.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc80d0100 0x7f1bc80cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 0x7f1bc81f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc815d550 0x7f1bc81f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc81f3f60 con 0x7f1bc80d1c80 2023-11-28T20:53:25.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc80cd280 con 0x7f1bc815d550 2023-11-28T20:53:25.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc80cd560 con 0x7f1bc80d0100 2023-11-28T20:53:25.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 0x7f1bc81f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 0x7f1bc81f2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59082/0 (socket says 172.21.15.45:59082) 2023-11-28T20:53:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc5d9b640 1 -- 172.21.15.45:0/2383349967 learned_addr learned my addr 172.21.15.45:0/2383349967 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc659c640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc80d0100 0x7f1bc80cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.354+0000 7f1bc6d9d640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc815d550 0x7f1bc81f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bc5d9b640 1 -- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc80d0100 msgr2=0x7f1bc80cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bc5d9b640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc80d0100 0x7f1bc80cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bc5d9b640 1 -- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc815d550 msgr2=0x7f1bc81f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bc5d9b640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc815d550 0x7f1bc81f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bc5d9b640 1 -- 172.21.15.45:0/2383349967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc00094d0 con 0x7f1bc80d1c80 2023-11-28T20:53:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bc5d9b640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 0x7f1bc81f2da0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000c430 tx=0x7f1bb000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bbeffd640 1 -- 172.21.15.45:0/2383349967 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0011790 con 0x7f1bc80d1c80 2023-11-28T20:53:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bbeffd640 1 -- 172.21.15.45:0/2383349967 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb00118f0 con 0x7f1bc80d1c80 2023-11-28T20:53:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bbeffd640 1 -- 172.21.15.45:0/2383349967 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0011ce0 con 0x7f1bc80d1c80 2023-11-28T20:53:25.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.355+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc80cd840 con 0x7f1bc80d1c80 2023-11-28T20:53:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.356+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc80b70f0 con 0x7f1bc80d1c80 2023-11-28T20:53:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.357+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc804e920 con 0x7f1bc80d1c80 2023-11-28T20:53:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.358+0000 7f1bbeffd640 1 -- 172.21.15.45:0/2383349967 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1bb001c460 con 0x7f1bc80d1c80 2023-11-28T20:53:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.359+0000 7f1bbeffd640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f1ba806af60 0x7f1ba806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.359+0000 7f1bbeffd640 1 -- 172.21.15.45:0/2383349967 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7f1bb00898b0 con 0x7f1bc80d1c80 2023-11-28T20:53:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.359+0000 7f1bc659c640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f1ba806af60 0x7f1ba806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.361+0000 7f1bc659c640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f1ba806af60 0x7f1ba806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80070b0 tx=0x7f1bb8007040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:25.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.363+0000 7f1bbeffd640 1 -- 172.21.15.45:0/2383349967 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bb00546f0 con 0x7f1bc80d1c80 2023-11-28T20:53:25.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.628+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f1bc815c950 con 0x7f1bc80d1c80 2023-11-28T20:53:25.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.710+0000 7f1bbeffd640 1 -- 172.21.15.45:0/2383349967 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v746) v1 ==== 183+0+0 (secure 0 0 0) 0x7f1bb0057d10 con 0x7f1bc80d1c80 2023-11-28T20:53:25.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f1ba806af60 msgr2=0x7f1ba806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f1ba806af60 0x7f1ba806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80070b0 tx=0x7f1bb8007040 comp rx=0 tx=0).stop 2023-11-28T20:53:25.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 msgr2=0x7f1bc81f2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 0x7f1bc81f2da0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000c430 tx=0x7f1bb000c8f0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 shutdown_connections 2023-11-28T20:53:25.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f1ba806af60 0x7f1ba806d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bc815d550 0x7f1bc81f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bc80d1c80 0x7f1bc81f2da0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 --2- 172.21.15.45:0/2383349967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bc80d0100 0x7f1bc80cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.713+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 >> 172.21.15.45:0/2383349967 conn(0x7f1bc806ac70 msgr2=0x7f1bc815e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:25.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.714+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 shutdown_connections 2023-11-28T20:53:25.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.714+0000 7f1bc759e640 1 -- 172.21.15.45:0/2383349967 wait complete. 2023-11-28T20:53:25.737 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-11-28T20:53:25.737 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-11-28T20:53:25.737 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T20:53:25.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:53:25.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.965+0000 7f31cb59e640 1 Processor -- start 2023-11-28T20:53:25.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.965+0000 7f31cb59e640 1 -- start start 2023-11-28T20:53:25.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.967+0000 7f31cb59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31cc0cfaf0 0x7f31cc0cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.967+0000 7f31cb59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31cc0b6560 0x7f31cc0cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.967+0000 7f31cb59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f31cc0b69f0 0x7f31cc0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.967+0000 7f31cb59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31cc0d1250 con 0x7f31cc0cfaf0 2023-11-28T20:53:25.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.967+0000 7f31cb59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31cc0ce6b0 con 0x7f31cc0b69f0 2023-11-28T20:53:25.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.967+0000 7f31cb59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0ce990 con 0x7f31cc0b6560 2023-11-28T20:53:25.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.967+0000 7f31ca59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31cc0cfaf0 0x7f31cc0cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.967+0000 7f31ca59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31cc0cfaf0 0x7f31cc0cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59094/0 (socket says 172.21.15.45:59094) 2023-11-28T20:53:25.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.967+0000 7f31ca59c640 1 -- 172.21.15.45:0/2850092436 learned_addr learned my addr 172.21.15.45:0/2850092436 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.968+0000 7f31ca59c640 1 -- 172.21.15.45:0/2850092436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31cc0b6560 msgr2=0x7f31cc0cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.968+0000 7f31ca59c640 1 --2- 172.21.15.45:0/2850092436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31cc0b6560 0x7f31cc0cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.968+0000 7f31ca59c640 1 -- 172.21.15.45:0/2850092436 >> v2:172.21.15.190:3300/0 conn(0x7f31cc0b69f0 msgr2=0x7f31cc0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.968+0000 7f31ca59c640 1 --2- 172.21.15.45:0/2850092436 >> v2:172.21.15.190:3300/0 conn(0x7f31cc0b69f0 0x7f31cc0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.968+0000 7f31ca59c640 1 -- 172.21.15.45:0/2850092436 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc0d14a0 con 0x7f31cc0cfaf0 2023-11-28T20:53:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.969+0000 7f31ca59c640 1 --2- 172.21.15.45:0/2850092436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0cfaf0 0x7f31cc0cfed0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0b92a0 tx=0x7f31bc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741f76a6aad13efc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:25.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.969+0000 7f31c9d9b640 1 -- 172.21.15.45:0/2850092436 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc031880 con 0x7f31cc0cfaf0 2023-11-28T20:53:25.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.969+0000 7f31c9d9b640 1 -- 172.21.15.45:0/2850092436 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31bc0319e0 con 0x7f31cc0cfaf0 2023-11-28T20:53:25.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.969+0000 7f31c9d9b640 1 -- 172.21.15.45:0/2850092436 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc031d70 con 0x7f31cc0cfaf0 2023-11-28T20:53:25.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.970+0000 7f31cb59e640 1 -- 172.21.15.45:0/2850092436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0cfaf0 msgr2=0x7f31cc0cfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.970+0000 7f31cb59e640 1 --2- 172.21.15.45:0/2850092436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0cfaf0 0x7f31cc0cfed0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0b92a0 tx=0x7f31bc02b920 comp rx=0 tx=0).stop 2023-11-28T20:53:25.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.970+0000 7f31cb59e640 1 -- 172.21.15.45:0/2850092436 shutdown_connections 2023-11-28T20:53:25.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.970+0000 7f31cb59e640 1 --2- 172.21.15.45:0/2850092436 >> v2:172.21.15.190:3300/0 conn(0x7f31cc0b69f0 0x7f31cc0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.970+0000 7f31cb59e640 1 --2- 172.21.15.45:0/2850092436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31cc0b6560 0x7f31cc0cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.970+0000 7f31cb59e640 1 --2- 172.21.15.45:0/2850092436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0cfaf0 0x7f31cc0cfed0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.970+0000 7f31cb59e640 1 -- 172.21.15.45:0/2850092436 >> 172.21.15.45:0/2850092436 conn(0x7f31cc0b4c40 msgr2=0x7f31cc0b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:25.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.970+0000 7f31cb59e640 1 -- 172.21.15.45:0/2850092436 shutdown_connections 2023-11-28T20:53:25.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.970+0000 7f31cb59e640 1 -- 172.21.15.45:0/2850092436 wait complete. 2023-11-28T20:53:25.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.972+0000 7f31cb59e640 1 Processor -- start 2023-11-28T20:53:25.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.973+0000 7f31cb59e640 1 -- start start 2023-11-28T20:53:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.974+0000 7f31cb59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0b6560 0x7f31cc14e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.974+0000 7f31cb59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31cc0cfaf0 0x7f31cc15f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.974+0000 7f31cb59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31cc15f7e0 0x7f31cc0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.974+0000 7f31cb59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc15fc70 con 0x7f31cc0b6560 2023-11-28T20:53:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.974+0000 7f31cb59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc14ea70 con 0x7f31cc15f7e0 2023-11-28T20:53:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.974+0000 7f31cb59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0c8650 con 0x7f31cc0cfaf0 2023-11-28T20:53:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.974+0000 7f31ca59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0b6560 0x7f31cc14e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.974+0000 7f31ca59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0b6560 0x7f31cc14e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59102/0 (socket says 172.21.15.45:59102) 2023-11-28T20:53:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.974+0000 7f31ca59c640 1 -- 172.21.15.45:0/118814445 learned_addr learned my addr 172.21.15.45:0/118814445 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.975+0000 7f31ca59c640 1 -- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31cc0cfaf0 msgr2=0x7f31cc15f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.975+0000 7f31c3fff640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31cc0cfaf0 0x7f31cc15f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.975+0000 7f31cad9d640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31cc15f7e0 0x7f31cc0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.975+0000 7f31ca59c640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31cc0cfaf0 0x7f31cc15f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.975+0000 7f31ca59c640 1 -- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31cc15f7e0 msgr2=0x7f31cc0c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.975+0000 7f31ca59c640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31cc15f7e0 0x7f31cc0c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.975+0000 7f31ca59c640 1 -- 172.21.15.45:0/118814445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0094d0 con 0x7f31cc0b6560 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.975+0000 7f31cad9d640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31cc15f7e0 0x7f31cc0c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.975+0000 7f31ca59c640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0b6560 0x7f31cc14e530 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f31bc000c00 tx=0x7f31bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.976+0000 7f31c2ffd640 1 -- 172.21.15.45:0/118814445 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc03ad30 con 0x7f31cc0b6560 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.976+0000 7f31c2ffd640 1 -- 172.21.15.45:0/118814445 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31bc03ae90 con 0x7f31cc0b6560 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.976+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31cc0c7a40 con 0x7f31cc0b6560 2023-11-28T20:53:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.976+0000 7f31c2ffd640 1 -- 172.21.15.45:0/118814445 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc044630 con 0x7f31cc0b6560 2023-11-28T20:53:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.976+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc0c8000 con 0x7f31cc0b6560 2023-11-28T20:53:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.977+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31cc04e920 con 0x7f31cc0b6560 2023-11-28T20:53:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.978+0000 7f31c2ffd640 1 -- 172.21.15.45:0/118814445 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f31bc04d450 con 0x7f31cc0b6560 2023-11-28T20:53:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.979+0000 7f31c2ffd640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f31ac06af90 0x7f31ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.979+0000 7f31c2ffd640 1 -- 172.21.15.45:0/118814445 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7f31bc0ac170 con 0x7f31cc0b6560 2023-11-28T20:53:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.980+0000 7f31c3fff640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f31ac06af90 0x7f31ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.981+0000 7f31c3fff640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f31ac06af90 0x7f31ac06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0c9e80 tx=0x7f31b4008040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:25.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:25.983+0000 7f31c2ffd640 1 -- 172.21.15.45:0/118814445 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31bc076e10 con 0x7f31cc0b6560 2023-11-28T20:53:26.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.182+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31cc0d1030 con 0x7f31cc0b6560 2023-11-28T20:53:26.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.184+0000 7f31c2ffd640 1 -- 172.21.15.45:0/118814445 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+14397 (secure 0 0 0) 0x7f31bc07a430 con 0x7f31cc0b6560 2023-11-28T20:53:26.186 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:26.187 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":124,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:23.695406+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6834/2489162496":"2023-11-29T20:53:20.650122+0000","172.21.15.190:6833/1902260958":"2023-11-29T20:53:15.614996+0000","172.21.15.190:6832/1902260958":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/2251272392":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/1361952934":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/2675112319":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/915400531":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/3484290831":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/1265166084":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/1549713842":"2023-11-29T20:53:15.614996+0000","172.21.15.190:6833/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:6832/4159844301":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3239477442":"2023-11-29T20:52:07.707914+0000","172.21.15.45:6832/2489162496":"2023-11-29T20:53:20.650122+0000","172.21.15.190:0/3809276606":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3450487150":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2166104670":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/3808290090":"2023-11-29T20:53:15.614996+0000","172.21.15.190:0/4069818189":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/2641549859":"2023-11-29T20:52:07.707914+0000","172.21.15.190:0/1724154865":"2023-11-28T21:51:59.768228+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:53:26.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f31ac06af90 msgr2=0x7f31ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f31ac06af90 0x7f31ac06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0c9e80 tx=0x7f31b4008040 comp rx=0 tx=0).stop 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0b6560 msgr2=0x7f31cc14e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0b6560 0x7f31cc14e530 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f31bc000c00 tx=0x7f31bc0046c0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 shutdown_connections 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f31ac06af90 0x7f31ac06d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31cc15f7e0 0x7f31cc0c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f31cc0cfaf0 0x7f31cc15f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 --2- 172.21.15.45:0/118814445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31cc0b6560 0x7f31cc14e530 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 >> 172.21.15.45:0/118814445 conn(0x7f31cc0b4c40 msgr2=0x7f31cc0b4230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 shutdown_connections 2023-11-28T20:53:26.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.187+0000 7f31cb59e640 1 -- 172.21.15.45:0/118814445 wait complete. 2023-11-28T20:53:26.209 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:53:26.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:53:26.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.441+0000 7f0cdebcb640 1 Processor -- start 2023-11-28T20:53:26.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.442+0000 7f0cdebcb640 1 -- start start 2023-11-28T20:53:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.443+0000 7f0cdebcb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cd80b3040 0x7f0cd806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.443+0000 7f0cdebcb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0cd815b270 0x7f0cd815f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.443+0000 7f0cdebcb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd815faf0 con 0x7f0cd80b3420 2023-11-28T20:53:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.443+0000 7f0cdebcb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd80ceaa0 con 0x7f0cd815b270 2023-11-28T20:53:26.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.443+0000 7f0cdebcb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd806d410 con 0x7f0cd80b3040 2023-11-28T20:53:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.443+0000 7f0cdd3c8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0cd815b270 0x7f0cd815f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.443+0000 7f0cdd3c8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0cd815b270 0x7f0cd815f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59108/0 (socket says 172.21.15.45:59108) 2023-11-28T20:53:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.443+0000 7f0cdd3c8640 1 -- 172.21.15.45:0/3523252108 learned_addr learned my addr 172.21.15.45:0/3523252108 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.444+0000 7f0cddbc9640 1 --2- 172.21.15.45:0/3523252108 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cd80b3040 0x7f0cd806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.444+0000 7f0cdd3c8640 1 -- 172.21.15.45:0/3523252108 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cd80b3040 msgr2=0x7f0cd806cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.444+0000 7f0cdd3c8640 1 --2- 172.21.15.45:0/3523252108 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cd80b3040 0x7f0cd806cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.444+0000 7f0cdd3c8640 1 -- 172.21.15.45:0/3523252108 >> v1:172.21.15.190:6789/0 conn(0x7f0cd80b3420 legacy=0x7f0cd80b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:26.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.444+0000 7f0cdd3c8640 1 -- 172.21.15.45:0/3523252108 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd806d6f0 con 0x7f0cd815b270 2023-11-28T20:53:26.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.445+0000 7f0cdd3c8640 1 --2- 172.21.15.45:0/3523252108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd815b270 0x7f0cd815f5b0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009870 tx=0x7f0cc802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905a47f3eced0825 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:26.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.445+0000 7f0cdcbc7640 1 -- 172.21.15.45:0/3523252108 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8031740 con 0x7f0cd815b270 2023-11-28T20:53:26.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.445+0000 7f0cdcbc7640 1 -- 172.21.15.45:0/3523252108 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc80318a0 con 0x7f0cd815b270 2023-11-28T20:53:26.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.445+0000 7f0cdcbc7640 1 -- 172.21.15.45:0/3523252108 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8031c30 con 0x7f0cd815b270 2023-11-28T20:53:26.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.446+0000 7f0cdebcb640 1 -- 172.21.15.45:0/3523252108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd815b270 msgr2=0x7f0cd815f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.446+0000 7f0cdebcb640 1 --2- 172.21.15.45:0/3523252108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd815b270 0x7f0cd815f5b0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009870 tx=0x7f0cc802b7e0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.446+0000 7f0cdebcb640 1 -- 172.21.15.45:0/3523252108 shutdown_connections 2023-11-28T20:53:26.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.446+0000 7f0cdebcb640 1 --2- 172.21.15.45:0/3523252108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd815b270 0x7f0cd815f5b0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.446+0000 7f0cdebcb640 1 --2- 172.21.15.45:0/3523252108 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cd80b3040 0x7f0cd806cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.446+0000 7f0cdebcb640 1 -- 172.21.15.45:0/3523252108 >> 172.21.15.45:0/3523252108 conn(0x7f0cd80b1050 msgr2=0x7f0cd80cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:26.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.446+0000 7f0cdebcb640 1 -- 172.21.15.45:0/3523252108 shutdown_connections 2023-11-28T20:53:26.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.446+0000 7f0cdebcb640 1 -- 172.21.15.45:0/3523252108 wait complete. 2023-11-28T20:53:26.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.448+0000 7f0cdebcb640 1 Processor -- start 2023-11-28T20:53:26.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.448+0000 7f0cdebcb640 1 -- start start 2023-11-28T20:53:26.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.449+0000 7f0cdebcb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cd80b3040 0x7f0cd815d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.449+0000 7f0cdebcb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd80b3420 0x7f0cd814d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.449+0000 7f0cdebcb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cd815b270 0x7f0cd815b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.449+0000 7f0cdebcb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd815c0a0 con 0x7f0cd80b3420 2023-11-28T20:53:26.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.449+0000 7f0cdebcb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd815c380 con 0x7f0cd815b270 2023-11-28T20:53:26.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.449+0000 7f0cdebcb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd815c660 con 0x7f0cd80b3040 2023-11-28T20:53:26.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.449+0000 7f0cdd3c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd80b3420 0x7f0cd814d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cdd3c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd80b3420 0x7f0cd814d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59110/0 (socket says 172.21.15.45:59110) 2023-11-28T20:53:26.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cdd3c8640 1 -- 172.21.15.45:0/2089188140 learned_addr learned my addr 172.21.15.45:0/2089188140 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:26.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cdd3c8640 1 -- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cd80b3040 msgr2=0x7f0cd815d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cdd3c8640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cd80b3040 0x7f0cd815d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cde3ca640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cd815b270 0x7f0cd815b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cdd3c8640 1 -- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cd815b270 msgr2=0x7f0cd815b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cdd3c8640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cd815b270 0x7f0cd815b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cdd3c8640 1 -- 172.21.15.45:0/2089188140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc80094d0 con 0x7f0cd80b3420 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cde3ca640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cd815b270 0x7f0cd815b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.450+0000 7f0cdd3c8640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd80b3420 0x7f0cd814d870 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009870 tx=0x7f0cc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.451+0000 7f0cce7fc640 1 -- 172.21.15.45:0/2089188140 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8033c70 con 0x7f0cd80b3420 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.451+0000 7f0cce7fc640 1 -- 172.21.15.45:0/2089188140 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc8033dd0 con 0x7f0cd80b3420 2023-11-28T20:53:26.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.451+0000 7f0cce7fc640 1 -- 172.21.15.45:0/2089188140 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc803c640 con 0x7f0cd80b3420 2023-11-28T20:53:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.451+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd815c940 con 0x7f0cd80b3420 2023-11-28T20:53:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.451+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd81596b0 con 0x7f0cd80b3420 2023-11-28T20:53:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.452+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8159840 con 0x7f0cd80b3420 2023-11-28T20:53:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.453+0000 7f0cce7fc640 1 -- 172.21.15.45:0/2089188140 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f0cc803a040 con 0x7f0cd80b3420 2023-11-28T20:53:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.454+0000 7f0cce7fc640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f0cbc06af60 0x7f0cbc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.454+0000 7f0cce7fc640 1 -- 172.21.15.45:0/2089188140 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7f0cc80ab430 con 0x7f0cd80b3420 2023-11-28T20:53:26.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.455+0000 7f0cddbc9640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f0cbc06af60 0x7f0cbc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.456+0000 7f0cddbc9640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f0cbc06af60 0x7f0cbc06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4006fd0 tx=0x7f0cd4008040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.459+0000 7f0cce7fc640 1 -- 172.21.15.45:0/2089188140 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cc80761e0 con 0x7f0cd80b3420 2023-11-28T20:53:26.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.653+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0cd815ce60 con 0x7f0cd80b3420 2023-11-28T20:53:26.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.654+0000 7f0cce7fc640 1 -- 172.21.15.45:0/2089188140 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0cd815ce60 con 0x7f0cd80b3420 2023-11-28T20:53:26.656 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.656+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f0cbc06af60 msgr2=0x7f0cbc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.656+0000 7f0cdebcb640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f0cbc06af60 0x7f0cbc06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4006fd0 tx=0x7f0cd4008040 comp rx=0 tx=0).stop 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.656+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd80b3420 msgr2=0x7f0cd814d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.656+0000 7f0cdebcb640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd80b3420 0x7f0cd814d870 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009870 tx=0x7f0cc80046c0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.657+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 shutdown_connections 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.657+0000 7f0cdebcb640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f0cbc06af60 0x7f0cbc06d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.657+0000 7f0cdebcb640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cd815b270 0x7f0cd815b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.657+0000 7f0cdebcb640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cd80b3420 0x7f0cd814d870 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.657+0000 7f0cdebcb640 1 --2- 172.21.15.45:0/2089188140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cd80b3040 0x7f0cd815d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.657+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 >> 172.21.15.45:0/2089188140 conn(0x7f0cd80b1050 msgr2=0x7f0cd80ae990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:26.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.657+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 shutdown_connections 2023-11-28T20:53:26.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.657+0000 7f0cdebcb640 1 -- 172.21.15.45:0/2089188140 wait complete. 2023-11-28T20:53:26.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-11-28T20:53:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.906+0000 7fb6020ff640 1 Processor -- start 2023-11-28T20:53:26.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.906+0000 7fb6020ff640 1 -- start start 2023-11-28T20:53:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.907+0000 7fb6020ff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc15d2f0 0x7fb5fc09a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.907+0000 7fb6020ff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc0b45e0 con 0x7fb5fc161640 2023-11-28T20:53:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.907+0000 7fb6020ff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc06b920 con 0x7fb5fc06af10 2023-11-28T20:53:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.907+0000 7fb6020ff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc06b640 con 0x7fb5fc15d2f0 2023-11-28T20:53:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.908+0000 7fb6018fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb5fc161640 0x7fb5fc0b6040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42698/0 (socket says 172.21.15.45:42698) 2023-11-28T20:53:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.908+0000 7fb6018fe640 1 -- 172.21.15.45:0/4014394076 learned_addr learned my addr 172.21.15.45:0/4014394076 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.908+0000 7fb6010fd640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc15d2f0 0x7fb5fc09a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.908+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380924178 0 0) 0x7fb5fc0b45e0 con 0x7fb5fc161640 2023-11-28T20:53:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.908+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e8003520 con 0x7fb5fc161640 2023-11-28T20:53:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 89754898 0 0) 0x7fb5e8003520 con 0x7fb5fc161640 2023-11-28T20:53:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc15d2f0 msgr2=0x7fb5fc09a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc15d2f0 0x7fb5fc09a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 >> v1:172.21.15.190:6789/0 conn(0x7fb5fc06af10 legacy=0x7fb5fc160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0b3f90 con 0x7fb5fc161640 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb5f40019f0 con 0x7fb5fc161640 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 >> v1:172.21.15.45:6789/0 conn(0x7fb5fc161640 legacy=0x7fb5fc0b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc15d2f0 0x7fb5e80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8003f70 0x7fb5e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8004d00 0x7fb5e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc06b640 con 0x7fb5e8003f70 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e800bb50 con 0x7fb5e8004d00 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e800bd10 con 0x7fb5fc15d2f0 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.909+0000 7fb6008fc640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8003f70 0x7fb5e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.910+0000 7fb6010fd640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc15d2f0 0x7fb5e80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.910+0000 7fb6018fe640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8004d00 0x7fb5e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.910+0000 7fb6008fc640 1 -- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc15d2f0 msgr2=0x7fb5e80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.910+0000 7fb6008fc640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc15d2f0 0x7fb5e80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.910+0000 7fb6008fc640 1 -- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8004d00 msgr2=0x7fb5e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.910+0000 7fb6008fc640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8004d00 0x7fb5e8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.910+0000 7fb6008fc640 1 -- 172.21.15.45:0/4014394076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e8003520 con 0x7fb5e8003f70 2023-11-28T20:53:26.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.910+0000 7fb6018fe640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8004d00 0x7fb5e8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.911+0000 7fb6008fc640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8003f70 0x7fb5e80047c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0001750 tx=0x7fb5e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21bd884fbdc0d74 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.911+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00052d0 con 0x7fb5e8003f70 2023-11-28T20:53:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.911+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f002e050 con 0x7fb5e8003f70 2023-11-28T20:53:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.911+0000 7fb5effff640 1 -- 172.21.15.45:0/4014394076 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0003830 con 0x7fb5e8003f70 2023-11-28T20:53:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.912+0000 7fb6020ff640 1 -- 172.21.15.45:0/4014394076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8003f70 msgr2=0x7fb5e80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.912+0000 7fb6020ff640 1 --2- 172.21.15.45:0/4014394076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8003f70 0x7fb5e80047c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0001750 tx=0x7fb5e80057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.912+0000 7fb6018fe640 1 -- 172.21.15.45:0/4014394076 reap_dead start 2023-11-28T20:53:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.912+0000 7fb6020ff640 1 -- 172.21.15.45:0/4014394076 shutdown_connections 2023-11-28T20:53:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.912+0000 7fb6020ff640 1 -- 172.21.15.45:0/4014394076 >> 172.21.15.45:0/4014394076 conn(0x7fb5fc099a40 msgr2=0x7fb5fc099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:26.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.912+0000 7fb6020ff640 1 -- 172.21.15.45:0/4014394076 shutdown_connections 2023-11-28T20:53:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.912+0000 7fb6020ff640 1 -- 172.21.15.45:0/4014394076 wait complete. 2023-11-28T20:53:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.914+0000 7fb6020ff640 1 Processor -- start 2023-11-28T20:53:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.914+0000 7fb6020ff640 1 -- start start 2023-11-28T20:53:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.915+0000 7fb6020ff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8003f70 0x7fb5fc1f7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.915+0000 7fb6020ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5fc14a460 0x7fb5fc15ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.915+0000 7fb6020ff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc0cc670 0x7fb5fc0ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.915+0000 7fb6020ff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc0cc1e0 con 0x7fb5fc14a460 2023-11-28T20:53:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.915+0000 7fb6020ff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc14a8e0 con 0x7fb5e8003f70 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.915+0000 7fb6020ff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc0cb5d0 con 0x7fb5fc0cc670 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.915+0000 7fb6008fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5fc14a460 0x7fb5fc15ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.915+0000 7fb6008fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5fc14a460 0x7fb5fc15ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59120/0 (socket says 172.21.15.45:59120) 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.915+0000 7fb6008fc640 1 -- 172.21.15.45:0/1412527733 learned_addr learned my addr 172.21.15.45:0/1412527733 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb6018fe640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc0cc670 0x7fb5fc0ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb6010fd640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8003f70 0x7fb5fc1f7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb6008fc640 1 -- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc0cc670 msgr2=0x7fb5fc0ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb6008fc640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc0cc670 0x7fb5fc0ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb6008fc640 1 -- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8003f70 msgr2=0x7fb5fc1f7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb6008fc640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8003f70 0x7fb5fc1f7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb6008fc640 1 -- 172.21.15.45:0/1412527733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0cb8b0 con 0x7fb5fc14a460 2023-11-28T20:53:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb6008fc640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5fc14a460 0x7fb5fc15ffb0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0034040 tx=0x7fb5f0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb5edffb640 1 -- 172.21.15.45:0/1412527733 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f003c630 con 0x7fb5fc14a460 2023-11-28T20:53:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb5edffb640 1 -- 172.21.15.45:0/1412527733 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f003c790 con 0x7fb5fc14a460 2023-11-28T20:53:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.916+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc0cbb90 con 0x7fb5fc14a460 2023-11-28T20:53:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.917+0000 7fb5edffb640 1 -- 172.21.15.45:0/1412527733 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f003cb40 con 0x7fb5fc14a460 2023-11-28T20:53:26.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.917+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc1604f0 con 0x7fb5fc14a460 2023-11-28T20:53:26.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.918+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5fc0cc350 con 0x7fb5fc14a460 2023-11-28T20:53:26.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.919+0000 7fb5edffb640 1 -- 172.21.15.45:0/1412527733 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb5f003cca0 con 0x7fb5fc14a460 2023-11-28T20:53:26.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.920+0000 7fb5edffb640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb5e006af60 0x7fb5e006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:26.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.920+0000 7fb5edffb640 1 -- 172.21.15.45:0/1412527733 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7fb5f00abd60 con 0x7fb5fc14a460 2023-11-28T20:53:26.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.920+0000 7fb6010fd640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb5e006af60 0x7fb5e006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:26.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.922+0000 7fb6010fd640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb5e006af60 0x7fb5e006d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8009950 tx=0x7fb5f8008040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:26.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:26.924+0000 7fb5edffb640 1 -- 172.21.15.45:0/1412527733 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5f0076bb0 con 0x7fb5fc14a460 2023-11-28T20:53:27.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.195+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fb5fc09f070 con 0x7fb5fc14a460 2023-11-28T20:53:27.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.715+0000 7fb5edffb640 1 -- 172.21.15.45:0/1412527733 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v748) v1 ==== 151+0+0 (secure 0 0 0) 0x7fb5f004c020 con 0x7fb5fc14a460 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb5e006af60 msgr2=0x7fb5e006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb5e006af60 0x7fb5e006d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8009950 tx=0x7fb5f8008040 comp rx=0 tx=0).stop 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5fc14a460 msgr2=0x7fb5fc15ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5fc14a460 0x7fb5fc15ffb0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0034040 tx=0x7fb5f0001120 comp rx=0 tx=0).stop 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 shutdown_connections 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb5e006af60 0x7fb5e006d420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5fc0cc670 0x7fb5fc0ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5fc14a460 0x7fb5fc15ffb0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 --2- 172.21.15.45:0/1412527733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5e8003f70 0x7fb5fc1f7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.719+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 >> 172.21.15.45:0/1412527733 conn(0x7fb5fc099a40 msgr2=0x7fb5fc0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.720+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 shutdown_connections 2023-11-28T20:53:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.720+0000 7fb6020ff640 1 -- 172.21.15.45:0/1412527733 wait complete. 2023-11-28T20:53:27.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T20:53:27.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.963+0000 7faac209b640 1 Processor -- start 2023-11-28T20:53:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.963+0000 7faac209b640 1 -- start start 2023-11-28T20:53:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.965+0000 7faac209b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc15e840 0x7faabc06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.965+0000 7faac209b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faabc15ecd0 con 0x7faabc06ac10 2023-11-28T20:53:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.965+0000 7faac209b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faabc15f290 con 0x7faabc15f720 2023-11-28T20:53:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.965+0000 7faac209b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faabc15fb60 con 0x7faabc15e840 2023-11-28T20:53:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.966+0000 7faabbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faabc06ac10 0x7faabc0cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42702/0 (socket says 172.21.15.45:42702) 2023-11-28T20:53:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.966+0000 7faabbfff640 1 -- 172.21.15.45:0/1149981488 learned_addr learned my addr 172.21.15.45:0/1149981488 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.966+0000 7faabb7fe640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc15e840 0x7faabc06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.966+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291900397 0 0) 0x7faabc15ecd0 con 0x7faabc06ac10 2023-11-28T20:53:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.966+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa8003520 con 0x7faabc06ac10 2023-11-28T20:53:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.966+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203097267 0 0) 0x7faaa8003520 con 0x7faabc06ac10 2023-11-28T20:53:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.966+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc15e840 msgr2=0x7faabc06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.966+0000 7faabaffd640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc15e840 0x7faabc06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.966+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 >> v1:172.21.15.190:6789/0 conn(0x7faabc15f720 legacy=0x7faabc06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc072020 con 0x7faabc06ac10 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7faab40019f0 con 0x7faabc06ac10 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 >> v1:172.21.15.45:6789/0 conn(0x7faabc06ac10 legacy=0x7faabc0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabaffd640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faaa8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabaffd640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaa80047c0 0x7faaa8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabaffd640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc15e840 0x7faaa8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc15fb60 con 0x7faaa80039a0 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa800ba60 con 0x7faaa80047c0 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa800bc20 con 0x7faabc15e840 2023-11-28T20:53:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabb7fe640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faaa8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabbfff640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc15e840 0x7faaa8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faab3fff640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaa80047c0 0x7faaa8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabb7fe640 1 -- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc15e840 msgr2=0x7faaa8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.967+0000 7faabb7fe640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc15e840 0x7faaa8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.968+0000 7faabb7fe640 1 -- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaa80047c0 msgr2=0x7faaa8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.968+0000 7faabb7fe640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaa80047c0 0x7faaa8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.968+0000 7faabb7fe640 1 -- 172.21.15.45:0/1149981488 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa8003520 con 0x7faaa80039a0 2023-11-28T20:53:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.968+0000 7faabbfff640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc15e840 0x7faaa8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.968+0000 7faab3fff640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaa80047c0 0x7faaa8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.969+0000 7faabb7fe640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faaa8003dc0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7faaac009a90 tx=0x7faaa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489d52bc7ed1b112 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.969+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac02c680 con 0x7faaa80039a0 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.969+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac02c7e0 con 0x7faaa80039a0 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.969+0000 7faabaffd640 1 -- 172.21.15.45:0/1149981488 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac02cb70 con 0x7faaa80039a0 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.970+0000 7faac209b640 1 -- 172.21.15.45:0/1149981488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 msgr2=0x7faaa8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.970+0000 7faac209b640 1 --2- 172.21.15.45:0/1149981488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faaa8003dc0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7faaac009a90 tx=0x7faaa80057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.970+0000 7faabbfff640 1 -- 172.21.15.45:0/1149981488 reap_dead start 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.970+0000 7faac209b640 1 -- 172.21.15.45:0/1149981488 shutdown_connections 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.970+0000 7faac209b640 1 -- 172.21.15.45:0/1149981488 >> 172.21.15.45:0/1149981488 conn(0x7faabc0ced60 msgr2=0x7faabc0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.970+0000 7faac209b640 1 -- 172.21.15.45:0/1149981488 shutdown_connections 2023-11-28T20:53:27.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.970+0000 7faac209b640 1 -- 172.21.15.45:0/1149981488 wait complete. 2023-11-28T20:53:27.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.972+0000 7faac209b640 1 Processor -- start 2023-11-28T20:53:27.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.972+0000 7faac209b640 1 -- start start 2023-11-28T20:53:27.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.973+0000 7faac209b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faabc0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:27.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.973+0000 7faac209b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc0c67d0 0x7faabc148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:27.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.973+0000 7faac209b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faabc1494d0 0x7faabc15cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:27.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.973+0000 7faac209b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc149f30 con 0x7faaa80039a0 2023-11-28T20:53:27.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.973+0000 7faac209b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc149970 con 0x7faabc1494d0 2023-11-28T20:53:27.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.973+0000 7faac209b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faabc149c50 con 0x7faabc0c67d0 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.973+0000 7faabb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faabc0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.973+0000 7faabb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faabc0c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59132/0 (socket says 172.21.15.45:59132) 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.973+0000 7faabb7fe640 1 -- 172.21.15.45:0/3385413530 learned_addr learned my addr 172.21.15.45:0/3385413530 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faabbfff640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faabc1494d0 0x7faabc15cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faab3fff640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc0c67d0 0x7faabc148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faabb7fe640 1 -- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc0c67d0 msgr2=0x7faabc148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faabb7fe640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc0c67d0 0x7faabc148f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faabb7fe640 1 -- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faabc1494d0 msgr2=0x7faabc15cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faabb7fe640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faabc1494d0 0x7faabc15cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faabb7fe640 1 -- 172.21.15.45:0/3385413530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc0c5fd0 con 0x7faaa80039a0 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faabbfff640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faabc1494d0 0x7faabc15cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:27.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faabb7fe640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faabc0c8ce0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7faaac02c220 tx=0x7faaac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:27.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.974+0000 7faab8ff9640 1 -- 172.21.15.45:0/3385413530 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac003810 con 0x7faaa80039a0 2023-11-28T20:53:27.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.975+0000 7faab8ff9640 1 -- 172.21.15.45:0/3385413530 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac003970 con 0x7faaa80039a0 2023-11-28T20:53:27.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.975+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faabc0c62b0 con 0x7faaa80039a0 2023-11-28T20:53:27.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.975+0000 7faab8ff9640 1 -- 172.21.15.45:0/3385413530 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac003ce0 con 0x7faaa80039a0 2023-11-28T20:53:27.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.975+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faabc15a150 con 0x7faaa80039a0 2023-11-28T20:53:27.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.976+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faabc15a340 con 0x7faaa80039a0 2023-11-28T20:53:27.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.977+0000 7faab8ff9640 1 -- 172.21.15.45:0/3385413530 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7faaac034440 con 0x7faaa80039a0 2023-11-28T20:53:27.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.978+0000 7faab8ff9640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7faa9c06af60 0x7faa9c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:27.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.978+0000 7faab8ff9640 1 -- 172.21.15.45:0/3385413530 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7faaac0ab7d0 con 0x7faaa80039a0 2023-11-28T20:53:27.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.978+0000 7faab3fff640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7faa9c06af60 0x7faa9c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:27.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.979+0000 7faab3fff640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7faa9c06af60 0x7faa9c06d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faaa4007fd0 tx=0x7faaa4009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:27.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:27.982+0000 7faab8ff9640 1 -- 172.21.15.45:0/3385413530 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faabc15a340 con 0x7faaa80039a0 2023-11-28T20:53:28.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.180+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7faabc15a340 con 0x7faaa80039a0 2023-11-28T20:53:28.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.186+0000 7faab8ff9640 1 -- 172.21.15.45:0/3385413530 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7faabc15a340 con 0x7faaa80039a0 2023-11-28T20:53:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7faa9c06af60 msgr2=0x7faa9c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7faa9c06af60 0x7faa9c06d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faaa4007fd0 tx=0x7faaa4009040 comp rx=0 tx=0).stop 2023-11-28T20:53:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 msgr2=0x7faabc0c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faabc0c8ce0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7faaac02c220 tx=0x7faaac004730 comp rx=0 tx=0).stop 2023-11-28T20:53:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 shutdown_connections 2023-11-28T20:53:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7faa9c06af60 0x7faa9c06d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faabc1494d0 0x7faabc15cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:28.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faabc0c67d0 0x7faabc148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:28.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 --2- 172.21.15.45:0/3385413530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa80039a0 0x7faabc0c8ce0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:28.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 >> 172.21.15.45:0/3385413530 conn(0x7faabc0ced60 msgr2=0x7faabc0afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:28.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 shutdown_connections 2023-11-28T20:53:28.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.189+0000 7faac209b640 1 -- 172.21.15.45:0/3385413530 wait complete. 2023-11-28T20:53:28.211 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:53:28.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:53:28.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:53:28.457 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:53:28.457 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:53:28.457 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:28.520 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:53:28.520 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:53:28.520 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:28.581 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:53:28.581 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:53:28.581 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:28.644 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:53:28.644 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:53:28.644 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:28.704 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:53:28.704 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T20:53:28.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:53:28.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.942+0000 7f7883f6e640 1 Processor -- start 2023-11-28T20:53:28.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.942+0000 7f7883f6e640 1 -- start start 2023-11-28T20:53:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f7883f6e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f787c1438b0 0x7f787c0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f7883f6e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f787c0b24e0 0x7f787c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f7883f6e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c15dcc0 con 0x7f787c0b1170 2023-11-28T20:53:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f7883f6e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f787c0b1600 con 0x7f787c1438b0 2023-11-28T20:53:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f7883f6e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f787c15e840 con 0x7f787c0b24e0 2023-11-28T20:53:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f7881d0a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f787c1438b0 0x7f787c0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:28.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f7881d0a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f787c1438b0 0x7f787c0b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59148/0 (socket says 172.21.15.45:59148) 2023-11-28T20:53:28.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f7881d0a640 1 -- 172.21.15.45:0/3736057771 learned_addr learned my addr 172.21.15.45:0/3736057771 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:28.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f788250b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f787c0b1170 0x7f787c0ccdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42718/0 (socket says 172.21.15.45:42718) 2023-11-28T20:53:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.944+0000 7f7881509640 1 --2- 172.21.15.45:0/3736057771 >> v2:172.21.15.190:3300/0 conn(0x7f787c0b24e0 0x7f787c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.945+0000 7f7881d0a640 1 -- 172.21.15.45:0/3736057771 >> v2:172.21.15.190:3300/0 conn(0x7f787c0b24e0 msgr2=0x7f787c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.945+0000 7f7881d0a640 1 --2- 172.21.15.45:0/3736057771 >> v2:172.21.15.190:3300/0 conn(0x7f787c0b24e0 0x7f787c0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.945+0000 7f7881d0a640 1 -- 172.21.15.45:0/3736057771 >> v1:172.21.15.45:6789/0 conn(0x7f787c0b1170 legacy=0x7f787c0ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.945+0000 7f7881d0a640 1 -- 172.21.15.45:0/3736057771 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787c15df10 con 0x7f787c1438b0 2023-11-28T20:53:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.945+0000 7f7881509640 1 --2- 172.21.15.45:0/3736057771 >> v2:172.21.15.190:3300/0 conn(0x7f787c0b24e0 0x7f787c0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.946+0000 7f7881d0a640 1 --2- 172.21.15.45:0/3736057771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c1438b0 0x7f787c0b1fa0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f787c0ce2b0 tx=0x7f787802e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed336bdbe112870a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.946+0000 7f7880d08640 1 -- 172.21.15.45:0/3736057771 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787802ea00 con 0x7f787c1438b0 2023-11-28T20:53:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.946+0000 7f7880d08640 1 -- 172.21.15.45:0/3736057771 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787802eb60 con 0x7f787c1438b0 2023-11-28T20:53:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.946+0000 7f7880d08640 1 -- 172.21.15.45:0/3736057771 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7878009600 con 0x7f787c1438b0 2023-11-28T20:53:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.946+0000 7f7883f6e640 1 -- 172.21.15.45:0/3736057771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c1438b0 msgr2=0x7f787c0b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.947+0000 7f7883f6e640 1 --2- 172.21.15.45:0/3736057771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c1438b0 0x7f787c0b1fa0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f787c0ce2b0 tx=0x7f787802e060 comp rx=0 tx=0).stop 2023-11-28T20:53:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.947+0000 7f7883f6e640 1 -- 172.21.15.45:0/3736057771 shutdown_connections 2023-11-28T20:53:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.947+0000 7f7883f6e640 1 --2- 172.21.15.45:0/3736057771 >> v2:172.21.15.190:3300/0 conn(0x7f787c0b24e0 0x7f787c0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.947+0000 7f7883f6e640 1 --2- 172.21.15.45:0/3736057771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c1438b0 0x7f787c0b1fa0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.947+0000 7f7883f6e640 1 -- 172.21.15.45:0/3736057771 >> 172.21.15.45:0/3736057771 conn(0x7f787c06a500 msgr2=0x7f787c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.947+0000 7f7883f6e640 1 -- 172.21.15.45:0/3736057771 shutdown_connections 2023-11-28T20:53:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.947+0000 7f7883f6e640 1 -- 172.21.15.45:0/3736057771 wait complete. 2023-11-28T20:53:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.948+0000 7f7883f6e640 1 Processor -- start 2023-11-28T20:53:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.949+0000 7f7883f6e640 1 -- start start 2023-11-28T20:53:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7883f6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0b1170 0x7f787c1f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7883f6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c0b24e0 0x7f787c0b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7883f6e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c1438b0 0x7f787c0b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7883f6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c13eaf0 con 0x7f787c0b1170 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7883f6e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c13dee0 con 0x7f787c1438b0 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7883f6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787c13e1c0 con 0x7f787c0b24e0 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881d0a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0b1170 0x7f787c1f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881d0a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0b1170 0x7f787c1f6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59160/0 (socket says 172.21.15.45:59160) 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881d0a640 1 -- 172.21.15.45:0/3808114242 learned_addr learned my addr 172.21.15.45:0/3808114242 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881d0a640 1 -- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c0b24e0 msgr2=0x7f787c0b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881509640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c0b24e0 0x7f787c0b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881d0a640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c0b24e0 0x7f787c0b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881d0a640 1 -- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c1438b0 msgr2=0x7f787c0b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881d0a640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c1438b0 0x7f787c0b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881d0a640 1 -- 172.21.15.45:0/3808114242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787800bd40 con 0x7f787c0b1170 2023-11-28T20:53:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.950+0000 7f7881509640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c0b24e0 0x7f787c0b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.951+0000 7f7881d0a640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0b1170 0x7f787c1f6200 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f7878038040 tx=0x7f7878004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.951+0000 7f78727fc640 1 -- 172.21.15.45:0/3808114242 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787802ea00 con 0x7f787c0b1170 2023-11-28T20:53:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.951+0000 7f78727fc640 1 -- 172.21.15.45:0/3808114242 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7878036820 con 0x7f787c0b1170 2023-11-28T20:53:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.951+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787c13e4a0 con 0x7f787c0b1170 2023-11-28T20:53:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.951+0000 7f78727fc640 1 -- 172.21.15.45:0/3808114242 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7878036bd0 con 0x7f787c0b1170 2023-11-28T20:53:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.951+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787c1f6740 con 0x7f787c0b1170 2023-11-28T20:53:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.952+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787c13edd0 con 0x7f787c0b1170 2023-11-28T20:53:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.953+0000 7f78727fc640 1 -- 172.21.15.45:0/3808114242 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7878036d30 con 0x7f787c0b1170 2023-11-28T20:53:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.954+0000 7f78727fc640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f786006af60 0x7f786006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.954+0000 7f78727fc640 1 -- 172.21.15.45:0/3808114242 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7f78780acf20 con 0x7f787c0b1170 2023-11-28T20:53:28.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.955+0000 7f7881509640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f786006af60 0x7f786006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.956+0000 7f7881509640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f786006af60 0x7f786006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f786c006fd0 tx=0x7f786c008040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:28.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:28.959+0000 7f78727fc640 1 -- 172.21.15.45:0/3808114242 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7878077e00 con 0x7f787c0b1170 2023-11-28T20:53:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.192+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f787c0733d0 con 0x7f787c0b1170 2023-11-28T20:53:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.193+0000 7f78727fc640 1 -- 172.21.15.45:0/3808114242 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f787807b420 con 0x7f787c0b1170 2023-11-28T20:53:29.196 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:29.196 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/2348343744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2348343744},{"type":"v1","addr":"172.21.15.190:6836","nonce":2348343744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6439,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1864791063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":1864791063},{"type":"v1","addr":"172.21.15.190:6837","nonce":1864791063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3003510210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3003510210},{"type":"v1","addr":"172.21.15.45:6835","nonce":3003510210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8004,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/1869859357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1869859357},{"type":"v1","addr":"172.21.15.45:6834","nonce":1869859357}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[]} 2023-11-28T20:53:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 53 2023-11-28T20:53:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f786006af60 msgr2=0x7f786006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f786006af60 0x7f786006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f786c006fd0 tx=0x7f786c008040 comp rx=0 tx=0).stop 2023-11-28T20:53:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0b1170 msgr2=0x7f787c1f6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0b1170 0x7f787c1f6200 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f7878038040 tx=0x7f7878004a40 comp rx=0 tx=0).stop 2023-11-28T20:53:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 shutdown_connections 2023-11-28T20:53:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f786006af60 0x7f786006d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f787c1438b0 0x7f787c0b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f787c0b24e0 0x7f787c0b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 --2- 172.21.15.45:0/3808114242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f787c0b1170 0x7f787c1f6200 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 >> 172.21.15.45:0/3808114242 conn(0x7f787c06a500 msgr2=0x7f787c141390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.196+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 shutdown_connections 2023-11-28T20:53:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.197+0000 7f7883f6e640 1 -- 172.21.15.45:0/3808114242 wait complete. 2023-11-28T20:53:29.217 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T20:53:29.217 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T20:53:29.217 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T20:53:29.217 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:53:29.217 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T20:53:29.217 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T20:53:29.218 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T20:53:29.218 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:53:29.218 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T20:53:29.218 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T20:53:29.218 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T20:53:29.218 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:53:29.219 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T20:53:29.219 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T20:53:29.219 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T20:53:29.219 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:53:29.237 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T20:53:29.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T20:53:29.240 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T20:53:29.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T20:53:29.242 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T20:53:29.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T20:53:29.248 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T20:53:29.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T20:53:29.291 INFO:tasks.ceph.mds.b:Started 2023-11-28T20:53:29.292 DEBUG:teuthology.parallel:result is None 2023-11-28T20:53:29.297 INFO:tasks.ceph.mds.a:Started 2023-11-28T20:53:29.298 DEBUG:teuthology.parallel:result is None 2023-11-28T20:53:29.299 INFO:tasks.ceph.mds.d:Started 2023-11-28T20:53:29.299 DEBUG:teuthology.parallel:result is None 2023-11-28T20:53:29.305 INFO:tasks.ceph.mds.c:Started 2023-11-28T20:53:29.306 DEBUG:teuthology.parallel:result is None 2023-11-28T20:53:29.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T20:53:29.468 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T20:53:29.468 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:53:29.469 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:53:29.469 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:53:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.499+0000 7efe6bdea640 1 Processor -- start 2023-11-28T20:53:29.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.499+0000 7efe6bdea640 1 -- start start 2023-11-28T20:53:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe6bdea640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efe640cc3a0 0x7efe6406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe6bdea640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efe64165e50 0x7efe640cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe6bdea640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe640cf020 0x7efe640cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe6bdea640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe640cc780 con 0x7efe64165e50 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe6bdea640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe640cf9f0 con 0x7efe640cc3a0 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe6bdea640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe640b2d30 con 0x7efe640cf020 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe69385640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efe64165e50 0x7efe640cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe69385640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efe64165e50 0x7efe640cb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59178/0 (socket says 172.21.15.45:59178) 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe69385640 1 -- 172.21.15.45:0/3797329879 learned_addr learned my addr 172.21.15.45:0/3797329879 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe6a387640 1 --2- 172.21.15.45:0/3797329879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe640cf020 0x7efe640cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe69b86640 1 --2- 172.21.15.45:0/3797329879 >> v2:172.21.15.190:3300/0 conn(0x7efe640cc3a0 0x7efe6406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.501+0000 7efe69385640 1 -- 172.21.15.45:0/3797329879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe640cf020 msgr2=0x7efe640cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.502+0000 7efe69385640 1 --2- 172.21.15.45:0/3797329879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe640cf020 0x7efe640cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.502+0000 7efe69385640 1 -- 172.21.15.45:0/3797329879 >> v2:172.21.15.190:3300/0 conn(0x7efe640cc3a0 msgr2=0x7efe6406d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.502+0000 7efe69385640 1 --2- 172.21.15.45:0/3797329879 >> v2:172.21.15.190:3300/0 conn(0x7efe640cc3a0 0x7efe6406d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.502+0000 7efe69385640 1 -- 172.21.15.45:0/3797329879 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe640b2fd0 con 0x7efe64165e50 2023-11-28T20:53:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.503+0000 7efe69385640 1 --2- 172.21.15.45:0/3797329879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe64165e50 0x7efe640cb3f0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7efe6406b470 tx=0x7efe5402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea56d97c2ea4d47 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.503+0000 7efe68b84640 1 -- 172.21.15.45:0/3797329879 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe54031870 con 0x7efe64165e50 2023-11-28T20:53:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.503+0000 7efe68b84640 1 -- 172.21.15.45:0/3797329879 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe540319d0 con 0x7efe64165e50 2023-11-28T20:53:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.503+0000 7efe68b84640 1 -- 172.21.15.45:0/3797329879 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe54031d60 con 0x7efe64165e50 2023-11-28T20:53:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.503+0000 7efe6bdea640 1 -- 172.21.15.45:0/3797329879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe64165e50 msgr2=0x7efe640cb3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.503+0000 7efe6bdea640 1 --2- 172.21.15.45:0/3797329879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe64165e50 0x7efe640cb3f0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7efe6406b470 tx=0x7efe5402b910 comp rx=0 tx=0).stop 2023-11-28T20:53:29.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.504+0000 7efe6bdea640 1 -- 172.21.15.45:0/3797329879 shutdown_connections 2023-11-28T20:53:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.504+0000 7efe6bdea640 1 --2- 172.21.15.45:0/3797329879 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe640cf020 0x7efe640cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.504+0000 7efe6bdea640 1 --2- 172.21.15.45:0/3797329879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe64165e50 0x7efe640cb3f0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.504+0000 7efe6bdea640 1 --2- 172.21.15.45:0/3797329879 >> v2:172.21.15.190:3300/0 conn(0x7efe640cc3a0 0x7efe6406d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.504+0000 7efe6bdea640 1 -- 172.21.15.45:0/3797329879 >> 172.21.15.45:0/3797329879 conn(0x7efe6406a6e0 msgr2=0x7efe6406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.504+0000 7efe6bdea640 1 -- 172.21.15.45:0/3797329879 shutdown_connections 2023-11-28T20:53:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.504+0000 7efe6bdea640 1 -- 172.21.15.45:0/3797329879 wait complete. 2023-11-28T20:53:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.506+0000 7efe6bdea640 1 Processor -- start 2023-11-28T20:53:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.506+0000 7efe6bdea640 1 -- start start 2023-11-28T20:53:29.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe6bdea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe640cc3a0 0x7efe641565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:29.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe6bdea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe640cf020 0x7efe6415c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:29.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe6bdea640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe64165e50 0x7efe6415c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:29.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe6bdea640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe64156af0 con 0x7efe640cf020 2023-11-28T20:53:29.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe6bdea640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe64162e90 con 0x7efe640cc3a0 2023-11-28T20:53:29.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe6bdea640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe64162280 con 0x7efe64165e50 2023-11-28T20:53:29.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe69b86640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe640cc3a0 0x7efe641565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:29.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe69385640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe640cf020 0x7efe6415c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:29.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe69b86640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe640cc3a0 0x7efe641565b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:51046/0 (socket says 172.21.15.45:51046) 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe69b86640 1 -- 172.21.15.45:0/499978539 learned_addr learned my addr 172.21.15.45:0/499978539 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.507+0000 7efe6a387640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe64165e50 0x7efe6415c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe69385640 1 -- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe64165e50 msgr2=0x7efe6415c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe69385640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe64165e50 0x7efe6415c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe69385640 1 -- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe640cc3a0 msgr2=0x7efe641565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe69385640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe640cc3a0 0x7efe641565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe69385640 1 -- 172.21.15.45:0/499978539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe540094d0 con 0x7efe640cf020 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe69b86640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe640cc3a0 0x7efe641565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe69385640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe640cf020 0x7efe6415c1d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7efe54033000 tx=0x7efe540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe5a7fc640 1 -- 172.21.15.45:0/499978539 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5403ac60 con 0x7efe640cf020 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe5a7fc640 1 -- 172.21.15.45:0/499978539 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5403ade0 con 0x7efe640cf020 2023-11-28T20:53:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe64162560 con 0x7efe640cf020 2023-11-28T20:53:29.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.508+0000 7efe5a7fc640 1 -- 172.21.15.45:0/499978539 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe54044630 con 0x7efe640cf020 2023-11-28T20:53:29.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.509+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe64162b20 con 0x7efe640cf020 2023-11-28T20:53:29.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.510+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe64073440 con 0x7efe640cf020 2023-11-28T20:53:29.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.511+0000 7efe5a7fc640 1 -- 172.21.15.45:0/499978539 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7efe5404d450 con 0x7efe640cf020 2023-11-28T20:53:29.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.512+0000 7efe5a7fc640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7efe4806af60 0x7efe4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:29.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.512+0000 7efe5a7fc640 1 -- 172.21.15.45:0/499978539 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6080+0+0 (secure 0 0 0) 0x7efe5402d040 con 0x7efe640cf020 2023-11-28T20:53:29.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.513+0000 7efe69b86640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7efe4806af60 0x7efe4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:29.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.513+0000 7efe69b86640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7efe4806af60 0x7efe4806d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe600112c0 tx=0x7efe60009190 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:29.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.516+0000 7efe5a7fc640 1 -- 172.21.15.45:0/499978539 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe54076aa0 con 0x7efe640cf020 2023-11-28T20:53:29.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:29.730+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7efe64162fd0 con 0x7efe640cf020 2023-11-28T20:53:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.752+0000 7efe5a7fc640 1 -- 172.21.15.45:0/499978539 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v129) v1 ==== 97+0+0 (secure 0 0 0) 0x7efe5407a0c0 con 0x7efe640cf020 2023-11-28T20:53:30.755 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T20:53:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7efe4806af60 msgr2=0x7efe4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7efe4806af60 0x7efe4806d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe600112c0 tx=0x7efe60009190 comp rx=0 tx=0).stop 2023-11-28T20:53:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe640cf020 msgr2=0x7efe6415c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe640cf020 0x7efe6415c1d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7efe54033000 tx=0x7efe540046c0 comp rx=0 tx=0).stop 2023-11-28T20:53:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 shutdown_connections 2023-11-28T20:53:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7efe4806af60 0x7efe4806d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe64165e50 0x7efe6415c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe640cf020 0x7efe6415c1d0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 --2- 172.21.15.45:0/499978539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe640cc3a0 0x7efe641565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 >> 172.21.15.45:0/499978539 conn(0x7efe6406a6e0 msgr2=0x7efe6406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 shutdown_connections 2023-11-28T20:53:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.756+0000 7efe6bdea640 1 -- 172.21.15.45:0/499978539 wait complete. 2023-11-28T20:53:30.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T20:53:31.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.998+0000 7f5cf18e1640 1 Processor -- start 2023-11-28T20:53:31.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:30.998+0000 7f5cf18e1640 1 -- start start 2023-11-28T20:53:31.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.000+0000 7f5cf18e1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0cc2a0 0x7f5cec15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.000+0000 7f5cf18e1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec15c670 con 0x7f5cec15be80 2023-11-28T20:53:31.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.000+0000 7f5cf18e1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec0cda40 con 0x7f5cec141600 2023-11-28T20:53:31.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.000+0000 7f5cf18e1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec0ccc80 con 0x7f5cec0cc2a0 2023-11-28T20:53:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.001+0000 7f5cf10e0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5cec15be80 0x7f5cec15c860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42742/0 (socket says 172.21.15.45:42742) 2023-11-28T20:53:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.001+0000 7f5cf10e0640 1 -- 172.21.15.45:0/154314676 learned_addr learned my addr 172.21.15.45:0/154314676 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.001+0000 7f5cebfff640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0cc2a0 0x7f5cec15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.001+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106696047 0 0) 0x7f5cec15c670 con 0x7f5cec15be80 2023-11-28T20:53:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.001+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cdc003520 con 0x7f5cec15be80 2023-11-28T20:53:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237400257 0 0) 0x7f5cec0cda40 con 0x7f5cec141600 2023-11-28T20:53:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cec15c670 con 0x7f5cec141600 2023-11-28T20:53:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4071495285 0 0) 0x7f5cdc003520 con 0x7f5cec15be80 2023-11-28T20:53:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0cc2a0 msgr2=0x7f5cec15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0cc2a0 0x7f5cec15b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 >> v1:172.21.15.190:6789/0 conn(0x7f5cec141600 legacy=0x7f5cec0cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec0ccf60 con 0x7f5cec15be80 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5ce4001970 con 0x7f5cec15be80 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 >> v1:172.21.15.45:6789/0 conn(0x7f5cec15be80 legacy=0x7f5cec15c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cdc003b30 0x7f5cdc003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc0047c0 0x7f5cdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0cc2a0 0x7f5cdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec0ccc80 con 0x7f5cdc003b30 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc00ba30 con 0x7f5cdc0047c0 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc00bbf0 con 0x7f5cec0cc2a0 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.002+0000 7f5cf08df640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cdc003b30 0x7f5cdc003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.003+0000 7f5cebfff640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc0047c0 0x7f5cdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.003+0000 7f5cf08df640 1 -- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0cc2a0 msgr2=0x7f5cdc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.003+0000 7f5cf10e0640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0cc2a0 0x7f5cdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.003+0000 7f5cf08df640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0cc2a0 0x7f5cdc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.003+0000 7f5cf08df640 1 -- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc0047c0 msgr2=0x7f5cdc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.003+0000 7f5cf08df640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc0047c0 0x7f5cdc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.003+0000 7f5cf08df640 1 -- 172.21.15.45:0/154314676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc003520 con 0x7f5cdc003b30 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.003+0000 7f5cebfff640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc0047c0 0x7f5cdc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.004+0000 7f5cf08df640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cdc003b30 0x7f5cdc003f30 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00018d0 tx=0x7f5cdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cecdfc152fc881 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.004+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce00052d0 con 0x7f5cdc003b30 2023-11-28T20:53:31.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.004+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce0005430 con 0x7f5cdc003b30 2023-11-28T20:53:31.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.004+0000 7f5ceb7fe640 1 -- 172.21.15.45:0/154314676 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0003810 con 0x7f5cdc003b30 2023-11-28T20:53:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.005+0000 7f5cf18e1640 1 -- 172.21.15.45:0/154314676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cdc003b30 msgr2=0x7f5cdc003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.005+0000 7f5cf18e1640 1 --2- 172.21.15.45:0/154314676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cdc003b30 0x7f5cdc003f30 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00018d0 tx=0x7f5cdc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.005+0000 7f5cf10e0640 1 -- 172.21.15.45:0/154314676 reap_dead start 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.005+0000 7f5cf18e1640 1 -- 172.21.15.45:0/154314676 shutdown_connections 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.005+0000 7f5cf18e1640 1 -- 172.21.15.45:0/154314676 >> 172.21.15.45:0/154314676 conn(0x7f5cec0b0e70 msgr2=0x7f5cec0b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.005+0000 7f5cf18e1640 1 -- 172.21.15.45:0/154314676 shutdown_connections 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.005+0000 7f5cf18e1640 1 -- 172.21.15.45:0/154314676 wait complete. 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.007+0000 7f5cf18e1640 1 Processor -- start 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.007+0000 7f5cf18e1640 1 -- start start 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.008+0000 7f5cf18e1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc003b30 0x7f5cec15a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.008+0000 7f5cf18e1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0afcd0 0x7f5cec099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.008+0000 7f5cf18e1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec0b2440 0x7f5cec0b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.008+0000 7f5cf18e1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec0b2e00 con 0x7f5cec0b2440 2023-11-28T20:53:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.008+0000 7f5cf18e1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec09a370 con 0x7f5cdc003b30 2023-11-28T20:53:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.008+0000 7f5cf18e1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec09a650 con 0x7f5cec0afcd0 2023-11-28T20:53:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.008+0000 7f5cf10e0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec0b2440 0x7f5cec0b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cf10e0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec0b2440 0x7f5cec0b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59204/0 (socket says 172.21.15.45:59204) 2023-11-28T20:53:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cf10e0640 1 -- 172.21.15.45:0/4224511620 learned_addr learned my addr 172.21.15.45:0/4224511620 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cebfff640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0afcd0 0x7f5cec099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cf08df640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc003b30 0x7f5cec15a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cf10e0640 1 -- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0afcd0 msgr2=0x7f5cec099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cf10e0640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0afcd0 0x7f5cec099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cf10e0640 1 -- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc003b30 msgr2=0x7f5cec15a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cf10e0640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc003b30 0x7f5cec15a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cf10e0640 1 -- 172.21.15.45:0/4224511620 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec09a930 con 0x7f5cec0b2440 2023-11-28T20:53:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.009+0000 7f5cf10e0640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec0b2440 0x7f5cec0b28c0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f5cec15c260 tx=0x7f5ce40021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.010+0000 7f5ce97fa640 1 -- 172.21.15.45:0/4224511620 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce400b800 con 0x7f5cec0b2440 2023-11-28T20:53:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.010+0000 7f5ce97fa640 1 -- 172.21.15.45:0/4224511620 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce400b960 con 0x7f5cec0b2440 2023-11-28T20:53:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.010+0000 7f5ce97fa640 1 -- 172.21.15.45:0/4224511620 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce400bd30 con 0x7f5cec0b2440 2023-11-28T20:53:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.010+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cec09ac10 con 0x7f5cec0b2440 2023-11-28T20:53:31.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.010+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cec099f80 con 0x7f5cec0b2440 2023-11-28T20:53:31.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.011+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cec15ea90 con 0x7f5cec0b2440 2023-11-28T20:53:31.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.012+0000 7f5ce97fa640 1 -- 172.21.15.45:0/4224511620 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5ce4007070 con 0x7f5cec0b2440 2023-11-28T20:53:31.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.013+0000 7f5ce97fa640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5cd006af90 0x7f5cd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.013+0000 7f5ce97fa640 1 -- 172.21.15.45:0/4224511620 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f5ce40898b0 con 0x7f5cec0b2440 2023-11-28T20:53:31.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.014+0000 7f5cf08df640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5cd006af90 0x7f5cd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.015+0000 7f5cf08df640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5cd006af90 0x7f5cd006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0034000 tx=0x7f5ce0031000 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:31.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.018+0000 7f5ce97fa640 1 -- 172.21.15.45:0/4224511620 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ce4054950 con 0x7f5cec0b2440 2023-11-28T20:53:31.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.281+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5cec04e920 con 0x7f5cec0b2440 2023-11-28T20:53:31.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.282+0000 7f5ce97fa640 1 -- 172.21.15.45:0/4224511620 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5ce4057f70 con 0x7f5cec0b2440 2023-11-28T20:53:31.285 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:31.285 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:53:31.285 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T20:53:31.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.285 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T20:53:31.285 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T20:53:31.285 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.285 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:53:31.285 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:53:31.286 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.289 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T20:53:31.290 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.291 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.292 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T20:53:31.292 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T20:53:31.292 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.293 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:53:31.293 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:53:31.293 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.293 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.294 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T20:53:31.294 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T20:53:31.294 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.294 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:53:31.294 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:53:31.294 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.294 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.295 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T20:53:31.295 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T20:53:31.295 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.295 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:53:31.295 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:53:31.295 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.295 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.296 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T20:53:31.296 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T20:53:31.296 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.296 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:53:31.296 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:53:31.296 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.297 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.300 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T20:53:31.301 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T20:53:31.302 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T20:53:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:53:31.304 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:53:31.304 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:53:31.304 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:53:31.304 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.304 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:31.304 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T20:53:31.304 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:53:31.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5cd006af90 msgr2=0x7f5cd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5cd006af90 0x7f5cd006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0034000 tx=0x7f5ce0031000 comp rx=0 tx=0).stop 2023-11-28T20:53:31.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec0b2440 msgr2=0x7f5cec0b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec0b2440 0x7f5cec0b28c0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f5cec15c260 tx=0x7f5ce40021d0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 shutdown_connections 2023-11-28T20:53:31.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5cd006af90 0x7f5cd006d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec0b2440 0x7f5cec0b28c0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cec0afcd0 0x7f5cec099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 --2- 172.21.15.45:0/4224511620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cdc003b30 0x7f5cec15a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 >> 172.21.15.45:0/4224511620 conn(0x7f5cec0b0e70 msgr2=0x7f5cec0adb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:31.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 shutdown_connections 2023-11-28T20:53:31.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.285+0000 7f5cf18e1640 1 -- 172.21.15.45:0/4224511620 wait complete. 2023-11-28T20:53:31.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:53:31.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.530+0000 7f8bd63e6640 1 Processor -- start 2023-11-28T20:53:31.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.530+0000 7f8bd63e6640 1 -- start start 2023-11-28T20:53:31.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.532+0000 7f8bd63e6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd00ae540 0x7f8bd00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.532+0000 7f8bd63e6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8bd00ce630 0x7f8bd00b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.532+0000 7f8bd63e6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd015c3c0 con 0x7f8bd0155780 2023-11-28T20:53:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.532+0000 7f8bd63e6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd00ceac0 con 0x7f8bd00ce630 2023-11-28T20:53:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.532+0000 7f8bd63e6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0155b60 con 0x7f8bd00ae540 2023-11-28T20:53:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.532+0000 7f8bd5be5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8bd00ce630 0x7f8bd00b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.532+0000 7f8bd5be5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8bd00ce630 0x7f8bd00b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59216/0 (socket says 172.21.15.45:59216) 2023-11-28T20:53:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.532+0000 7f8bd5be5640 1 -- 172.21.15.45:0/2833091904 learned_addr learned my addr 172.21.15.45:0/2833091904 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.532+0000 7f8bd53e4640 1 --1- 172.21.15.45:0/2833091904 >> v1:172.21.15.45:6789/0 conn(0x7f8bd0155780 0x7f8bd00abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42746/0 (socket says 172.21.15.45:42746) 2023-11-28T20:53:31.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.533+0000 7f8bd4be3640 1 --2- 172.21.15.45:0/2833091904 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd00ae540 0x7f8bd00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.533+0000 7f8bd5be5640 1 -- 172.21.15.45:0/2833091904 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd00ae540 msgr2=0x7f8bd00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.533+0000 7f8bd5be5640 1 --2- 172.21.15.45:0/2833091904 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd00ae540 0x7f8bd00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.533+0000 7f8bd5be5640 1 -- 172.21.15.45:0/2833091904 >> v1:172.21.15.45:6789/0 conn(0x7f8bd0155780 legacy=0x7f8bd00abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.533+0000 7f8bd5be5640 1 -- 172.21.15.45:0/2833091904 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd015ba90 con 0x7f8bd00ce630 2023-11-28T20:53:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.534+0000 7f8bd5be5640 1 --2- 172.21.15.45:0/2833091904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ce630 0x7f8bd00b20a0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009e80 tx=0x7f8bc802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c103738b8f231f4c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:31.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bc7fff640 1 -- 172.21.15.45:0/2833091904 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc802caf0 con 0x7f8bd00ce630 2023-11-28T20:53:31.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bc7fff640 1 -- 172.21.15.45:0/2833091904 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc802cc50 con 0x7f8bd00ce630 2023-11-28T20:53:31.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bc7fff640 1 -- 172.21.15.45:0/2833091904 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc80343f0 con 0x7f8bd00ce630 2023-11-28T20:53:31.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bd63e6640 1 -- 172.21.15.45:0/2833091904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ce630 msgr2=0x7f8bd00b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bd63e6640 1 --2- 172.21.15.45:0/2833091904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ce630 0x7f8bd00b20a0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009e80 tx=0x7f8bc802bf50 comp rx=0 tx=0).stop 2023-11-28T20:53:31.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bd63e6640 1 -- 172.21.15.45:0/2833091904 shutdown_connections 2023-11-28T20:53:31.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bd63e6640 1 --2- 172.21.15.45:0/2833091904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ce630 0x7f8bd00b20a0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bd63e6640 1 --2- 172.21.15.45:0/2833091904 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd00ae540 0x7f8bd00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bd63e6640 1 -- 172.21.15.45:0/2833091904 >> 172.21.15.45:0/2833091904 conn(0x7f8bd006ab30 msgr2=0x7f8bd00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:31.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bd63e6640 1 -- 172.21.15.45:0/2833091904 shutdown_connections 2023-11-28T20:53:31.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.535+0000 7f8bd63e6640 1 -- 172.21.15.45:0/2833091904 wait complete. 2023-11-28T20:53:31.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.537+0000 7f8bd63e6640 1 Processor -- start 2023-11-28T20:53:31.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.537+0000 7f8bd63e6640 1 -- start start 2023-11-28T20:53:31.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.538+0000 7f8bd63e6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ae540 0x7f8bd0145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.538+0000 7f8bd63e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bd00ce630 0x7f8bd0070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.538+0000 7f8bd63e6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0155780 0x7f8bd00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.538+0000 7f8bd63e6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0150410 con 0x7f8bd00ae540 2023-11-28T20:53:31.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.538+0000 7f8bd63e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd014f3e0 con 0x7f8bd00ce630 2023-11-28T20:53:31.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.538+0000 7f8bd63e6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0146480 con 0x7f8bd0155780 2023-11-28T20:53:31.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd53e4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ae540 0x7f8bd0145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd53e4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ae540 0x7f8bd0145f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59224/0 (socket says 172.21.15.45:59224) 2023-11-28T20:53:31.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd53e4640 1 -- 172.21.15.45:0/1787262830 learned_addr learned my addr 172.21.15.45:0/1787262830 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd5be5640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0155780 0x7f8bd00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd4be3640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bd00ce630 0x7f8bd0070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd53e4640 1 -- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0155780 msgr2=0x7f8bd00710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd53e4640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0155780 0x7f8bd00710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd53e4640 1 -- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bd00ce630 msgr2=0x7f8bd0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd53e4640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bd00ce630 0x7f8bd0070bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd53e4640 1 -- 172.21.15.45:0/1787262830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8009b30 con 0x7f8bd00ae540 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.539+0000 7f8bd53e4640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ae540 0x7f8bd0145f40 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00c3e0 tx=0x7f8bcc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.540+0000 7f8bc5ffb640 1 -- 172.21.15.45:0/1787262830 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc0074e0 con 0x7f8bd00ae540 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.540+0000 7f8bc5ffb640 1 -- 172.21.15.45:0/1787262830 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc007640 con 0x7f8bd00ae540 2023-11-28T20:53:31.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.540+0000 7f8bc5ffb640 1 -- 172.21.15.45:0/1787262830 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc0049e0 con 0x7f8bd00ae540 2023-11-28T20:53:31.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.540+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd009b4d0 con 0x7f8bd00ae540 2023-11-28T20:53:31.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.540+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd014f820 con 0x7f8bd00ae540 2023-11-28T20:53:31.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.541+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd0066f40 con 0x7f8bd00ae540 2023-11-28T20:53:31.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.542+0000 7f8bc5ffb640 1 -- 172.21.15.45:0/1787262830 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8bcc004b40 con 0x7f8bd00ae540 2023-11-28T20:53:31.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.543+0000 7f8bc5ffb640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f8bb406af60 0x7f8bb406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.543+0000 7f8bc5ffb640 1 -- 172.21.15.45:0/1787262830 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8bcc088ee0 con 0x7f8bd00ae540 2023-11-28T20:53:31.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.544+0000 7f8bd4be3640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f8bb406af60 0x7f8bb406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.545+0000 7f8bd4be3640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f8bb406af60 0x7f8bb406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0071930 tx=0x7f8bc0008040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:31.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.548+0000 7f8bc5ffb640 1 -- 172.21.15.45:0/1787262830 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bcc054120 con 0x7f8bd00ae540 2023-11-28T20:53:31.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.744+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bd004e920 con 0x7f8bd00ae540 2023-11-28T20:53:31.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.745+0000 7f8bc5ffb640 1 -- 172.21.15.45:0/1787262830 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+13060 (secure 0 0 0) 0x7f8bcc057740 con 0x7f8bd00ae540 2023-11-28T20:53:31.748 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:31.748 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":129,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:30.749845+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:53:31.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f8bb406af60 msgr2=0x7f8bb406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f8bb406af60 0x7f8bb406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0071930 tx=0x7f8bc0008040 comp rx=0 tx=0).stop 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ae540 msgr2=0x7f8bd0145f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ae540 0x7f8bd0145f40 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00c3e0 tx=0x7f8bcc00c8a0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 shutdown_connections 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f8bb406af60 0x7f8bb406d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0155780 0x7f8bd00710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bd00ce630 0x7f8bd0070bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 --2- 172.21.15.45:0/1787262830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd00ae540 0x7f8bd0145f40 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.748+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 >> 172.21.15.45:0/1787262830 conn(0x7f8bd006ab30 msgr2=0x7f8bd00ac840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.749+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 shutdown_connections 2023-11-28T20:53:31.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.749+0000 7f8bd63e6640 1 -- 172.21.15.45:0/1787262830 wait complete. 2023-11-28T20:53:31.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:53:31.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:53:31.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.988+0000 7fd7645a9640 1 Processor -- start 2023-11-28T20:53:31.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.989+0000 7fd7645a9640 1 -- start start 2023-11-28T20:53:31.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.990+0000 7fd7645a9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c15e840 0x7fd75c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.990+0000 7fd7645a9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75c15ecd0 con 0x7fd75c15f720 2023-11-28T20:53:31.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.990+0000 7fd7645a9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75c15f290 con 0x7fd75c06ac10 2023-11-28T20:53:31.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.990+0000 7fd7645a9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c15fb60 con 0x7fd75c15e840 2023-11-28T20:53:31.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.991+0000 7fd761b44640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd75c15f720 0x7fd75c06a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42754/0 (socket says 172.21.15.45:42754) 2023-11-28T20:53:31.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.991+0000 7fd761b44640 1 -- 172.21.15.45:0/2883805865 learned_addr learned my addr 172.21.15.45:0/2883805865 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:31.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.991+0000 7fd762345640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c15e840 0x7fd75c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.991+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718217528 0 0) 0x7fd75c15ecd0 con 0x7fd75c15f720 2023-11-28T20:53:31.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.991+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c003520 con 0x7fd75c15f720 2023-11-28T20:53:31.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789191307 0 0) 0x7fd74c003520 con 0x7fd75c15f720 2023-11-28T20:53:31.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c15e840 msgr2=0x7fd75c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:31.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c15e840 0x7fd75c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 >> v1:172.21.15.190:6789/0 conn(0x7fd75c06ac10 legacy=0x7fd75c0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c072020 con 0x7fd75c15f720 2023-11-28T20:53:31.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd748001670 con 0x7fd75c15f720 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 >> v1:172.21.15.45:6789/0 conn(0x7fd75c15f720 legacy=0x7fd75c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd74c0039a0 0x7fd74c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c15e840 0x7fd74c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd74c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c15fb60 con 0x7fd74c004d00 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c00ba60 con 0x7fd74c0039a0 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c00bc20 con 0x7fd75c15e840 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd762b46640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd74c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.992+0000 7fd761b44640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c15e840 0x7fd74c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.993+0000 7fd762345640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd74c0039a0 0x7fd74c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.993+0000 7fd762b46640 1 -- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c15e840 msgr2=0x7fd74c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:31.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.993+0000 7fd762b46640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c15e840 0x7fd74c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.993+0000 7fd762b46640 1 -- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd74c0039a0 msgr2=0x7fd74c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.993+0000 7fd762b46640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd74c0039a0 0x7fd74c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.993+0000 7fd762b46640 1 -- 172.21.15.45:0/2883805865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c003520 con 0x7fd74c004d00 2023-11-28T20:53:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.994+0000 7fd762b46640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd74c005180 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fd754001720 tx=0x7fd74c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ebe3f41fd8ab4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.994+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7540052d0 con 0x7fd74c004d00 2023-11-28T20:53:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.994+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75402e050 con 0x7fd74c004d00 2023-11-28T20:53:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.994+0000 7fd761343640 1 -- 172.21.15.45:0/2883805865 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7540035f0 con 0x7fd74c004d00 2023-11-28T20:53:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.995+0000 7fd7645a9640 1 -- 172.21.15.45:0/2883805865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 msgr2=0x7fd74c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.995+0000 7fd7645a9640 1 --2- 172.21.15.45:0/2883805865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd74c005180 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fd754001720 tx=0x7fd74c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.995+0000 7fd762b46640 1 -- 172.21.15.45:0/2883805865 reap_dead start 2023-11-28T20:53:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.995+0000 7fd7645a9640 1 -- 172.21.15.45:0/2883805865 shutdown_connections 2023-11-28T20:53:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.995+0000 7fd7645a9640 1 -- 172.21.15.45:0/2883805865 >> 172.21.15.45:0/2883805865 conn(0x7fd75c0ced60 msgr2=0x7fd75c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.995+0000 7fd7645a9640 1 -- 172.21.15.45:0/2883805865 shutdown_connections 2023-11-28T20:53:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.995+0000 7fd7645a9640 1 -- 172.21.15.45:0/2883805865 wait complete. 2023-11-28T20:53:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.997+0000 7fd7645a9640 1 Processor -- start 2023-11-28T20:53:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.997+0000 7fd7645a9640 1 -- start start 2023-11-28T20:53:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.998+0000 7fd7645a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd75c156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.998+0000 7fd7645a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd75c0af910 0x7fd75c0afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.998+0000 7fd7645a9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c0b2cd0 0x7fd75c0b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.998+0000 7fd7645a9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c0b02d0 con 0x7fd74c004d00 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.998+0000 7fd7645a9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c0b3690 con 0x7fd75c0af910 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.998+0000 7fd7645a9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c0b3940 con 0x7fd75c0b2cd0 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.998+0000 7fd762345640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd75c156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.998+0000 7fd762345640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd75c156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59252/0 (socket says 172.21.15.45:59252) 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.998+0000 7fd762345640 1 -- 172.21.15.45:0/462134776 learned_addr learned my addr 172.21.15.45:0/462134776 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd762345640 1 -- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c0b2cd0 msgr2=0x7fd75c0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd762345640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c0b2cd0 0x7fd75c0b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd762345640 1 -- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd75c0af910 msgr2=0x7fd75c0afd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd762345640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd75c0af910 0x7fd75c0afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd762345640 1 -- 172.21.15.45:0/462134776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c0b3c20 con 0x7fd74c004d00 2023-11-28T20:53:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd762345640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd75c156270 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fd750007f30 tx=0x7fd750011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd75affd640 1 -- 172.21.15.45:0/462134776 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75000e030 con 0x7fd74c004d00 2023-11-28T20:53:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd75affd640 1 -- 172.21.15.45:0/462134776 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd750016470 con 0x7fd74c004d00 2023-11-28T20:53:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd75affd640 1 -- 172.21.15.45:0/462134776 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750016820 con 0x7fd74c004d00 2023-11-28T20:53:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:31.999+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75c0b3f00 con 0x7fd74c004d00 2023-11-28T20:53:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.000+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75c0b44c0 con 0x7fd74c004d00 2023-11-28T20:53:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.001+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75c0733d0 con 0x7fd74c004d00 2023-11-28T20:53:32.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.002+0000 7fd75affd640 1 -- 172.21.15.45:0/462134776 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd750016980 con 0x7fd74c004d00 2023-11-28T20:53:32.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.003+0000 7fd75affd640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd74006af60 0x7fd74006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.003+0000 7fd75affd640 1 -- 172.21.15.45:0/462134776 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd75000f070 con 0x7fd74c004d00 2023-11-28T20:53:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.003+0000 7fd761b44640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd74006af60 0x7fd74006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.004+0000 7fd761b44640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd74006af60 0x7fd74006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd748001050 tx=0x7fd748000f60 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:32.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.007+0000 7fd75affd640 1 -- 172.21.15.45:0/462134776 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7500556b0 con 0x7fd74c004d00 2023-11-28T20:53:32.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.203+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd75c0ce2b0 con 0x7fd74c004d00 2023-11-28T20:53:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.204+0000 7fd75affd640 1 -- 172.21.15.45:0/462134776 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd75008a070 con 0x7fd74c004d00 2023-11-28T20:53:32.206 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.206+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd74006af60 msgr2=0x7fd74006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.206+0000 7fd7645a9640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd74006af60 0x7fd74006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd748001050 tx=0x7fd748000f60 comp rx=0 tx=0).stop 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.206+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 msgr2=0x7fd75c156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.206+0000 7fd7645a9640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd75c156270 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fd750007f30 tx=0x7fd750011660 comp rx=0 tx=0).stop 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.207+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 shutdown_connections 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.207+0000 7fd7645a9640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd74006af60 0x7fd74006d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.207+0000 7fd7645a9640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd75c0b2cd0 0x7fd75c0b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.207+0000 7fd7645a9640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd75c0af910 0x7fd75c0afd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.207+0000 7fd7645a9640 1 --2- 172.21.15.45:0/462134776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd74c004d00 0x7fd75c156270 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.207+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 >> 172.21.15.45:0/462134776 conn(0x7fd75c0ced60 msgr2=0x7fd75c0aef00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.207+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 shutdown_connections 2023-11-28T20:53:32.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.207+0000 7fd7645a9640 1 -- 172.21.15.45:0/462134776 wait complete. 2023-11-28T20:53:32.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T20:53:32.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.452+0000 7fd7c7190640 1 Processor -- start 2023-11-28T20:53:32.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.452+0000 7fd7c7190640 1 -- start start 2023-11-28T20:53:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.454+0000 7fd7c7190640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c00b35b0 0x7fd7c015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.454+0000 7fd7c7190640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd7c015b850 0x7fd7c00b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.454+0000 7fd7c7190640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c015bda0 con 0x7fd7c00b1fa0 2023-11-28T20:53:32.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.454+0000 7fd7c7190640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c015ee40 con 0x7fd7c015b850 2023-11-28T20:53:32.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.454+0000 7fd7c7190640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c015c080 con 0x7fd7c00b35b0 2023-11-28T20:53:32.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.455+0000 7fd7c4f2c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c00b35b0 0x7fd7c015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.455+0000 7fd7c4f2c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c00b35b0 0x7fd7c015ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:59870/0 (socket says 172.21.15.45:59870) 2023-11-28T20:53:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.455+0000 7fd7c4f2c640 1 -- 172.21.15.45:0/1463501173 learned_addr learned my addr 172.21.15.45:0/1463501173 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.455+0000 7fd7bffff640 1 --2- 172.21.15.45:0/1463501173 >> v2:172.21.15.190:3300/0 conn(0x7fd7c015b850 0x7fd7c00b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.455+0000 7fd7c4f2c640 1 -- 172.21.15.45:0/1463501173 >> v2:172.21.15.190:3300/0 conn(0x7fd7c015b850 msgr2=0x7fd7c00b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.455+0000 7fd7c4f2c640 1 --2- 172.21.15.45:0/1463501173 >> v2:172.21.15.190:3300/0 conn(0x7fd7c015b850 0x7fd7c00b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.455+0000 7fd7c4f2c640 1 -- 172.21.15.45:0/1463501173 >> v1:172.21.15.190:6789/0 conn(0x7fd7c00b1fa0 legacy=0x7fd7c015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.456+0000 7fd7c4f2c640 1 -- 172.21.15.45:0/1463501173 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c015c360 con 0x7fd7c00b35b0 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.457+0000 7fd7c4f2c640 1 --2- 172.21.15.45:0/1463501173 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c00b35b0 0x7fd7c015ca10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009e80 tx=0x7fd7b002bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f932aaf4f6276dc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.457+0000 7fd7bf7fe640 1 -- 172.21.15.45:0/1463501173 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b002cae0 con 0x7fd7c00b35b0 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.457+0000 7fd7bf7fe640 1 -- 172.21.15.45:0/1463501173 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b002cc40 con 0x7fd7c00b35b0 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.457+0000 7fd7bf7fe640 1 -- 172.21.15.45:0/1463501173 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b00343f0 con 0x7fd7c00b35b0 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.458+0000 7fd7c7190640 1 -- 172.21.15.45:0/1463501173 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c00b35b0 msgr2=0x7fd7c015ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.458+0000 7fd7c7190640 1 --2- 172.21.15.45:0/1463501173 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c00b35b0 0x7fd7c015ca10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009e80 tx=0x7fd7b002bf40 comp rx=0 tx=0).stop 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.458+0000 7fd7c7190640 1 -- 172.21.15.45:0/1463501173 shutdown_connections 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.458+0000 7fd7c7190640 1 --2- 172.21.15.45:0/1463501173 >> v2:172.21.15.190:3300/0 conn(0x7fd7c015b850 0x7fd7c00b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.458+0000 7fd7c7190640 1 --2- 172.21.15.45:0/1463501173 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c00b35b0 0x7fd7c015ca10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.458+0000 7fd7c7190640 1 -- 172.21.15.45:0/1463501173 >> 172.21.15.45:0/1463501173 conn(0x7fd7c006a500 msgr2=0x7fd7c006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.458+0000 7fd7c7190640 1 -- 172.21.15.45:0/1463501173 shutdown_connections 2023-11-28T20:53:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.459+0000 7fd7c7190640 1 -- 172.21.15.45:0/1463501173 wait complete. 2023-11-28T20:53:32.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.461+0000 7fd7c7190640 1 Processor -- start 2023-11-28T20:53:32.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.461+0000 7fd7c7190640 1 -- start start 2023-11-28T20:53:32.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.462+0000 7fd7c7190640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7c00b1fa0 0x7fd7c01f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.462+0000 7fd7c7190640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7c00b35b0 0x7fd7c01f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.462+0000 7fd7c7190640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c015b850 0x7fd7c01f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.462+0000 7fd7c7190640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c00a8c10 con 0x7fd7c00b35b0 2023-11-28T20:53:32.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.462+0000 7fd7c7190640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c013f490 con 0x7fd7c00b1fa0 2023-11-28T20:53:32.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.462+0000 7fd7c7190640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c013ebc0 con 0x7fd7c015b850 2023-11-28T20:53:32.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7bffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7c00b35b0 0x7fd7c01f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7bffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7c00b35b0 0x7fd7c01f69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59260/0 (socket says 172.21.15.45:59260) 2023-11-28T20:53:32.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7bffff640 1 -- 172.21.15.45:0/3070768080 learned_addr learned my addr 172.21.15.45:0/3070768080 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7c572d640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c015b850 0x7fd7c01f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7c4f2c640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7c00b1fa0 0x7fd7c01f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7bffff640 1 -- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c015b850 msgr2=0x7fd7c01f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7bffff640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c015b850 0x7fd7c01f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7bffff640 1 -- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7c00b1fa0 msgr2=0x7fd7c01f6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7bffff640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7c00b1fa0 0x7fd7c01f6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7bffff640 1 -- 172.21.15.45:0/3070768080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b0009b30 con 0x7fd7c00b35b0 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7c4f2c640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7c00b1fa0 0x7fd7c01f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.463+0000 7fd7c572d640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c015b850 0x7fd7c01f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.464+0000 7fd7bffff640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7c00b35b0 0x7fd7c01f69f0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400e980 tx=0x7fd7b400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.464+0000 7fd7bd7fa640 1 -- 172.21.15.45:0/3070768080 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4009670 con 0x7fd7c00b35b0 2023-11-28T20:53:32.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.464+0000 7fd7bd7fa640 1 -- 172.21.15.45:0/3070768080 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b400ca00 con 0x7fd7c00b35b0 2023-11-28T20:53:32.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.464+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c00a8930 con 0x7fd7c00b35b0 2023-11-28T20:53:32.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.464+0000 7fd7bd7fa640 1 -- 172.21.15.45:0/3070768080 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400cd90 con 0x7fd7c00b35b0 2023-11-28T20:53:32.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.464+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c013f120 con 0x7fd7c00b35b0 2023-11-28T20:53:32.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.466+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c00733d0 con 0x7fd7c00b35b0 2023-11-28T20:53:32.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.466+0000 7fd7bd7fa640 1 -- 172.21.15.45:0/3070768080 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd7b4013470 con 0x7fd7c00b35b0 2023-11-28T20:53:32.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.467+0000 7fd7bd7fa640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd7ac06af60 0x7fd7ac06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.467+0000 7fd7bd7fa640 1 -- 172.21.15.45:0/3070768080 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd7b4089830 con 0x7fd7c00b35b0 2023-11-28T20:53:32.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.468+0000 7fd7c4f2c640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd7ac06af60 0x7fd7ac06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.469+0000 7fd7c4f2c640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd7ac06af60 0x7fd7ac06d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0035000 tx=0x7fd7b0030000 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:32.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.472+0000 7fd7bd7fa640 1 -- 172.21.15.45:0/3070768080 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7b40549e0 con 0x7fd7c00b35b0 2023-11-28T20:53:32.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.671+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd7c015b100 con 0x7fd7c00b35b0 2023-11-28T20:53:32.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.672+0000 7fd7bd7fa640 1 -- 172.21.15.45:0/3070768080 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v129) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd7b4017090 con 0x7fd7c00b35b0 2023-11-28T20:53:32.675 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:32.675 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:53:32.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:32.675 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T20:53:32.675 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T20:53:32.676 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:32.676 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:53:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd7ac06af60 msgr2=0x7fd7ac06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd7ac06af60 0x7fd7ac06d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0035000 tx=0x7fd7b0030000 comp rx=0 tx=0).stop 2023-11-28T20:53:32.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7c00b35b0 msgr2=0x7fd7c01f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7c00b35b0 0x7fd7c01f69f0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400e980 tx=0x7fd7b400ee40 comp rx=0 tx=0).stop 2023-11-28T20:53:32.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 shutdown_connections 2023-11-28T20:53:32.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd7ac06af60 0x7fd7ac06d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7c015b850 0x7fd7c01f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7c00b35b0 0x7fd7c01f69f0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 --2- 172.21.15.45:0/3070768080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7c00b1fa0 0x7fd7c01f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 >> 172.21.15.45:0/3070768080 conn(0x7fd7c006a500 msgr2=0x7fd7c0144770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:32.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 shutdown_connections 2023-11-28T20:53:32.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.675+0000 7fd7c7190640 1 -- 172.21.15.45:0/3070768080 wait complete. 2023-11-28T20:53:32.697 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T20:53:32.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T20:53:32.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.928+0000 7f23249cf640 1 Processor -- start 2023-11-28T20:53:32.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.928+0000 7f23249cf640 1 -- start start 2023-11-28T20:53:32.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.929+0000 7f23249cf640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f23200cc900 0x7f23200adb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.929+0000 7f23249cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23200ae0b0 0x7f2320070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.929+0000 7f23249cf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23200acad0 con 0x7f23201607c0 2023-11-28T20:53:32.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.929+0000 7f23249cf640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23200ac8c0 con 0x7f23200cc900 2023-11-28T20:53:32.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.929+0000 7f23249cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23200acdb0 con 0x7f23200ae0b0 2023-11-28T20:53:32.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.929+0000 7f231ed9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f23201607c0 0x7f23200cd720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42758/0 (socket says 172.21.15.45:42758) 2023-11-28T20:53:32.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231ed9d640 1 -- 172.21.15.45:0/1216583167 learned_addr learned my addr 172.21.15.45:0/1216583167 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:32.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231f59e640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23200ae0b0 0x7f2320070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231e59c640 1 --2- 172.21.15.45:0/1216583167 >> v2:172.21.15.190:3300/0 conn(0x7f23200cc900 0x7f23200adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665257175 0 0) 0x7f23200acad0 con 0x7f23201607c0 2023-11-28T20:53:32.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2308003520 con 0x7f23201607c0 2023-11-28T20:53:32.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083966830 0 0) 0x7f2308003520 con 0x7f23201607c0 2023-11-28T20:53:32.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23200ae0b0 msgr2=0x7f2320070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:32.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231dd9b640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23200ae0b0 0x7f2320070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 >> v2:172.21.15.190:3300/0 conn(0x7f23200cc900 msgr2=0x7f23200adb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:32.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231dd9b640 1 --2- 172.21.15.45:0/1216583167 >> v2:172.21.15.190:3300/0 conn(0x7f23200cc900 0x7f23200adb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23200ad090 con 0x7f23201607c0 2023-11-28T20:53:32.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.930+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f2310001670 con 0x7f23201607c0 2023-11-28T20:53:32.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 >> v1:172.21.15.45:6789/0 conn(0x7f23201607c0 legacy=0x7f23200cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231dd9b640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2308003970 0x7f2308003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231dd9b640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080047c0 0x7f2308004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231dd9b640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23200ae0b0 0x7f2308005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23200ac8c0 con 0x7f23080047c0 2023-11-28T20:53:32.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23200acdb0 con 0x7f2308003970 2023-11-28T20:53:32.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230800ba00 con 0x7f23200ae0b0 2023-11-28T20:53:32.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231e59c640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080047c0 0x7f2308004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231ed9d640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2308003970 0x7f2308003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231f59e640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23200ae0b0 0x7f2308005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231e59c640 1 -- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23200ae0b0 msgr2=0x7f2308005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:32.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231e59c640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23200ae0b0 0x7f2308005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231e59c640 1 -- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2308003970 msgr2=0x7f2308003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:32.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231e59c640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2308003970 0x7f2308003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.931+0000 7f231e59c640 1 -- 172.21.15.45:0/1216583167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2308003520 con 0x7f23080047c0 2023-11-28T20:53:32.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.933+0000 7f231e59c640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080047c0 0x7f2308004c40 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f2314009ac0 tx=0x7f23080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab57fc538da73a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:32.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.933+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231402c710 con 0x7f23080047c0 2023-11-28T20:53:32.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.933+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231402c870 con 0x7f23080047c0 2023-11-28T20:53:32.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.933+0000 7f231dd9b640 1 -- 172.21.15.45:0/1216583167 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231402cc00 con 0x7f23080047c0 2023-11-28T20:53:32.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.934+0000 7f23249cf640 1 -- 172.21.15.45:0/1216583167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080047c0 msgr2=0x7f2308004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.934+0000 7f23249cf640 1 --2- 172.21.15.45:0/1216583167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23080047c0 0x7f2308004c40 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f2314009ac0 tx=0x7f23080057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.934+0000 7f231f59e640 1 -- 172.21.15.45:0/1216583167 reap_dead start 2023-11-28T20:53:32.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.934+0000 7f23249cf640 1 -- 172.21.15.45:0/1216583167 shutdown_connections 2023-11-28T20:53:32.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.934+0000 7f23249cf640 1 -- 172.21.15.45:0/1216583167 >> 172.21.15.45:0/1216583167 conn(0x7f23200cec30 msgr2=0x7f23200cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:32.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.934+0000 7f23249cf640 1 -- 172.21.15.45:0/1216583167 shutdown_connections 2023-11-28T20:53:32.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.934+0000 7f23249cf640 1 -- 172.21.15.45:0/1216583167 wait complete. 2023-11-28T20:53:32.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.936+0000 7f23249cf640 1 Processor -- start 2023-11-28T20:53:32.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.936+0000 7f23249cf640 1 -- start start 2023-11-28T20:53:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.937+0000 7f23249cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23080047c0 0x7f23201f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.937+0000 7f23249cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23200c4ce0 0x7f2320158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.937+0000 7f23249cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23200c5160 0x7f23200b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.937+0000 7f23249cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23201f7b80 con 0x7f23200c4ce0 2023-11-28T20:53:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.937+0000 7f23249cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2320159a10 con 0x7f23200c5160 2023-11-28T20:53:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.937+0000 7f23249cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2320159730 con 0x7f23080047c0 2023-11-28T20:53:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.937+0000 7f231e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23200c4ce0 0x7f2320158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.937+0000 7f231e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23200c4ce0 0x7f2320158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59278/0 (socket says 172.21.15.45:59278) 2023-11-28T20:53:32.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.937+0000 7f231e59c640 1 -- 172.21.15.45:0/3339287332 learned_addr learned my addr 172.21.15.45:0/3339287332 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:32.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f231e59c640 1 -- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23080047c0 msgr2=0x7f23201f7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f231ed9d640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23080047c0 0x7f23201f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f231e59c640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23080047c0 0x7f23201f7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f231f59e640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23200c5160 0x7f23200b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f231e59c640 1 -- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23200c5160 msgr2=0x7f23200b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:32.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f231e59c640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23200c5160 0x7f23200b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:32.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f231e59c640 1 -- 172.21.15.45:0/3339287332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23201f74c0 con 0x7f23200c4ce0 2023-11-28T20:53:32.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f231e59c640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23200c4ce0 0x7f2320158c50 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f2314037040 tx=0x7f2314004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f23077fe640 1 -- 172.21.15.45:0/3339287332 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314003810 con 0x7f23200c4ce0 2023-11-28T20:53:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f23077fe640 1 -- 172.21.15.45:0/3339287332 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231402de90 con 0x7f23200c4ce0 2023-11-28T20:53:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f23077fe640 1 -- 172.21.15.45:0/3339287332 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314003b70 con 0x7f23200c4ce0 2023-11-28T20:53:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.938+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23201f77a0 con 0x7f23200c4ce0 2023-11-28T20:53:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.939+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2320157200 con 0x7f23200c4ce0 2023-11-28T20:53:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.940+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23201f7df0 con 0x7f23200c4ce0 2023-11-28T20:53:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.941+0000 7f23077fe640 1 -- 172.21.15.45:0/3339287332 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2314003cd0 con 0x7f23200c4ce0 2023-11-28T20:53:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.942+0000 7f23077fe640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f230c06af60 0x7f230c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:32.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.942+0000 7f23077fe640 1 -- 172.21.15.45:0/3339287332 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f23140ab5e0 con 0x7f23200c4ce0 2023-11-28T20:53:32.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.942+0000 7f231ed9d640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f230c06af60 0x7f230c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:32.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.943+0000 7f231ed9d640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f230c06af60 0x7f230c06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2310001eb0 tx=0x7f2310000ed0 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:32.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:32.946+0000 7f23077fe640 1 -- 172.21.15.45:0/3339287332 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2314076780 con 0x7f23200c4ce0 2023-11-28T20:53:33.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.141+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f23200733d0 con 0x7f23200c4ce0 2023-11-28T20:53:33.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.778+0000 7f23077fe640 1 -- 172.21.15.45:0/3339287332 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v130) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2314079da0 con 0x7f23200c4ce0 2023-11-28T20:53:33.781 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T20:53:33.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f230c06af60 msgr2=0x7f230c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:33.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f230c06af60 0x7f230c06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2310001eb0 tx=0x7f2310000ed0 comp rx=0 tx=0).stop 2023-11-28T20:53:33.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23200c4ce0 msgr2=0x7f2320158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23200c4ce0 0x7f2320158c50 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f2314037040 tx=0x7f2314004730 comp rx=0 tx=0).stop 2023-11-28T20:53:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 shutdown_connections 2023-11-28T20:53:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f230c06af60 0x7f230c06d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23200c5160 0x7f23200b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23200c4ce0 0x7f2320158c50 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 --2- 172.21.15.45:0/3339287332 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23080047c0 0x7f23201f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 >> 172.21.15.45:0/3339287332 conn(0x7f23200cec30 msgr2=0x7f23200ae490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 shutdown_connections 2023-11-28T20:53:33.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:33.782+0000 7f23249cf640 1 -- 172.21.15.45:0/3339287332 wait complete. 2023-11-28T20:53:33.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T20:53:34.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.030+0000 7f35467ae640 1 Processor -- start 2023-11-28T20:53:34.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.031+0000 7f35467ae640 1 -- start start 2023-11-28T20:53:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.032+0000 7f35467ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f354006c7a0 0x7f354006de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.032+0000 7f35467ae640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354006e3a0 con 0x7f3540070030 2023-11-28T20:53:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.032+0000 7f35467ae640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35400c9d20 con 0x7f354015b190 2023-11-28T20:53:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.032+0000 7f35467ae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354015b570 con 0x7f354006c7a0 2023-11-28T20:53:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.032+0000 7f3544fab640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3540070030 0x7f354006c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54930/0 (socket says 172.21.15.45:54930) 2023-11-28T20:53:34.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.032+0000 7f3544fab640 1 -- 172.21.15.45:0/3101980154 learned_addr learned my addr 172.21.15.45:0/3101980154 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:34.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3545fad640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f354006c7a0 0x7f354006de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:34.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419208681 0 0) 0x7f3530001110 con 0x7f3540070030 2023-11-28T20:53:34.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f352c003520 con 0x7f3540070030 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3370009653 0 0) 0x7f352c003520 con 0x7f3540070030 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f354006c7a0 msgr2=0x7f354006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f354006c7a0 0x7f354006de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 >> v1:172.21.15.190:6789/0 conn(0x7f354015b190 legacy=0x7f35400b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3540071c50 con 0x7f3540070030 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3530001800 con 0x7f3540070030 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 >> v1:172.21.15.45:6789/0 conn(0x7f3540070030 legacy=0x7f354006c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f352c0039a0 0x7f352c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f354006c7a0 0x7f352c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f352c004d00 0x7f352c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354015b570 con 0x7f352c004d00 2023-11-28T20:53:34.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352c00ba60 con 0x7f352c0039a0 2023-11-28T20:53:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.033+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352c00bc20 con 0x7f354006c7a0 2023-11-28T20:53:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.034+0000 7f3545fad640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f352c004d00 0x7f352c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.034+0000 7f35457ac640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f352c0039a0 0x7f352c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.034+0000 7f3544fab640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f354006c7a0 0x7f352c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.034+0000 7f3545fad640 1 -- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f354006c7a0 msgr2=0x7f352c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.034+0000 7f3545fad640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f354006c7a0 0x7f352c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.034+0000 7f3545fad640 1 -- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f352c0039a0 msgr2=0x7f352c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.034+0000 7f3545fad640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f352c0039a0 0x7f352c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.034+0000 7f3545fad640 1 -- 172.21.15.45:0/3101980154 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c003520 con 0x7f352c004d00 2023-11-28T20:53:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.035+0000 7f3545fad640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f352c004d00 0x7f352c005180 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f3538009fc0 tx=0x7f352c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aad34b7580c8727 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.035+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353802cc20 con 0x7f352c004d00 2023-11-28T20:53:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.036+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353800f040 con 0x7f352c004d00 2023-11-28T20:53:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.036+0000 7f3537fff640 1 -- 172.21.15.45:0/3101980154 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3538034440 con 0x7f352c004d00 2023-11-28T20:53:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.036+0000 7f35467ae640 1 -- 172.21.15.45:0/3101980154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f352c004d00 msgr2=0x7f352c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:34.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.036+0000 7f35467ae640 1 --2- 172.21.15.45:0/3101980154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f352c004d00 0x7f352c005180 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f3538009fc0 tx=0x7f352c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.036+0000 7f3545fad640 1 -- 172.21.15.45:0/3101980154 reap_dead start 2023-11-28T20:53:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.036+0000 7f35467ae640 1 -- 172.21.15.45:0/3101980154 shutdown_connections 2023-11-28T20:53:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.036+0000 7f35467ae640 1 -- 172.21.15.45:0/3101980154 >> 172.21.15.45:0/3101980154 conn(0x7f35400cec30 msgr2=0x7f35400cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.036+0000 7f35467ae640 1 -- 172.21.15.45:0/3101980154 shutdown_connections 2023-11-28T20:53:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.037+0000 7f35467ae640 1 -- 172.21.15.45:0/3101980154 wait complete. 2023-11-28T20:53:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.038+0000 7f35467ae640 1 Processor -- start 2023-11-28T20:53:34.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.038+0000 7f35467ae640 1 -- start start 2023-11-28T20:53:34.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.039+0000 7f35467ae640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f352c004d00 0x7f35401587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:34.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.039+0000 7f35467ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35401f2cf0 0x7f35401511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.039+0000 7f35467ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3540154480 0x7f3540154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.039+0000 7f35467ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3540154e40 con 0x7f35401f2cf0 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.039+0000 7f35467ae640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35401f3540 con 0x7f352c004d00 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.039+0000 7f35467ae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35401f3820 con 0x7f3540154480 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f3544fab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35401f2cf0 0x7f35401511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f3544fab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35401f2cf0 0x7f35401511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49546/0 (socket says 172.21.15.45:49546) 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f3544fab640 1 -- 172.21.15.45:0/2517144734 learned_addr learned my addr 172.21.15.45:0/2517144734 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f35457ac640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f352c004d00 0x7f35401587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f3545fad640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3540154480 0x7f3540154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f3544fab640 1 -- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3540154480 msgr2=0x7f3540154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f3544fab640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3540154480 0x7f3540154900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f3544fab640 1 -- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f352c004d00 msgr2=0x7f35401587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:34.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f3544fab640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f352c004d00 0x7f35401587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.040+0000 7f3544fab640 1 -- 172.21.15.45:0/2517144734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35401f3b00 con 0x7f35401f2cf0 2023-11-28T20:53:34.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.041+0000 7f3544fab640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35401f2cf0 0x7f35401511f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f353000d810 tx=0x7f353000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:34.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.041+0000 7f3535ffb640 1 -- 172.21.15.45:0/2517144734 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35300074e0 con 0x7f35401f2cf0 2023-11-28T20:53:34.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.041+0000 7f3535ffb640 1 -- 172.21.15.45:0/2517144734 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3530007640 con 0x7f35401f2cf0 2023-11-28T20:53:34.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.041+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35401f3de0 con 0x7f35401f2cf0 2023-11-28T20:53:34.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.041+0000 7f3535ffb640 1 -- 172.21.15.45:0/2517144734 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530004a00 con 0x7f35401f2cf0 2023-11-28T20:53:34.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.041+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3540151730 con 0x7f35401f2cf0 2023-11-28T20:53:34.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.042+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354006bca0 con 0x7f35401f2cf0 2023-11-28T20:53:34.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.044+0000 7f3535ffb640 1 -- 172.21.15.45:0/2517144734 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3530004b60 con 0x7f35401f2cf0 2023-11-28T20:53:34.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.045+0000 7f3535ffb640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f352406af60 0x7f352406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:34.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.045+0000 7f3535ffb640 1 -- 172.21.15.45:0/2517144734 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f3530089c30 con 0x7f35401f2cf0 2023-11-28T20:53:34.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.045+0000 7f35457ac640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f352406af60 0x7f352406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.047+0000 7f35457ac640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f352406af60 0x7f352406d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f353c007fd0 tx=0x7f353c009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:34.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.049+0000 7f3535ffb640 1 -- 172.21.15.45:0/2517144734 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3530054b40 con 0x7f35401f2cf0 2023-11-28T20:53:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.244+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f354006e3a0 con 0x7f35401f2cf0 2023-11-28T20:53:34.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.794+0000 7f3535ffb640 1 -- 172.21.15.45:0/2517144734 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v131) v1 ==== 170+0+0 (secure 0 0 0) 0x7f354006e3a0 con 0x7f35401f2cf0 2023-11-28T20:53:34.797 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T20:53:34.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.797+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f352406af60 msgr2=0x7f352406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:34.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.797+0000 7f35467ae640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f352406af60 0x7f352406d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f353c007fd0 tx=0x7f353c009040 comp rx=0 tx=0).stop 2023-11-28T20:53:34.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.797+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35401f2cf0 msgr2=0x7f35401511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:34.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.797+0000 7f35467ae640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35401f2cf0 0x7f35401511f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f353000d810 tx=0x7f353000dcd0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.798+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 shutdown_connections 2023-11-28T20:53:34.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.798+0000 7f35467ae640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f352406af60 0x7f352406d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.798+0000 7f35467ae640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3540154480 0x7f3540154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.798+0000 7f35467ae640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35401f2cf0 0x7f35401511f0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.798+0000 7f35467ae640 1 --2- 172.21.15.45:0/2517144734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f352c004d00 0x7f35401587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:34.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.798+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 >> 172.21.15.45:0/2517144734 conn(0x7f35400cec30 msgr2=0x7f3540068240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:34.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.798+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 shutdown_connections 2023-11-28T20:53:34.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:34.799+0000 7f35467ae640 1 -- 172.21.15.45:0/2517144734 wait complete. 2023-11-28T20:53:34.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T20:53:35.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.029+0000 7fbbc5d63640 1 Processor -- start 2023-11-28T20:53:35.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.029+0000 7fbbc5d63640 1 -- start start 2023-11-28T20:53:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.030+0000 7fbbc5d63640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc00717e0 0x7fbbc00b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbc5d63640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbbc00cb630 0x7fbbc00cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbc5d63640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc00724d0 con 0x7fbbc00b39d0 2023-11-28T20:53:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbc5d63640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc00b42f0 con 0x7fbbc00cb630 2023-11-28T20:53:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbc5d63640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00cc000 con 0x7fbbc00717e0 2023-11-28T20:53:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbc4d61640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbbc00b39d0 0x7fbbc006bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54942/0 (socket says 172.21.15.45:54942) 2023-11-28T20:53:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbc4d61640 1 -- 172.21.15.45:0/2850648813 learned_addr learned my addr 172.21.15.45:0/2850648813 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054328656 0 0) 0x7fbbc00724d0 con 0x7fbbc00b39d0 2023-11-28T20:53:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb0003520 con 0x7fbbc00b39d0 2023-11-28T20:53:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbbffff640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc00717e0 0x7fbbc00b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.031+0000 7fbbc5562640 1 --2- 172.21.15.45:0/2850648813 >> v2:172.21.15.190:3300/0 conn(0x7fbbc00cb630 0x7fbbc00cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911106120 0 0) 0x7fbbb0003520 con 0x7fbbc00b39d0 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc00717e0 msgr2=0x7fbbc00b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc00717e0 0x7fbbc00b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 >> v2:172.21.15.190:3300/0 conn(0x7fbbc00cb630 msgr2=0x7fbbc00cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 --2- 172.21.15.45:0/2850648813 >> v2:172.21.15.190:3300/0 conn(0x7fbbc00cb630 0x7fbbc00cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc006c670 con 0x7fbbc00b39d0 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fbbb4001670 con 0x7fbbc00b39d0 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 >> v1:172.21.15.45:6789/0 conn(0x7fbbc00b39d0 legacy=0x7fbbc006bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc00717e0 0x7fbbb0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb0003f40 0x7fbbb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbb0004d00 0x7fbbb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00cc000 con 0x7fbbb0003f40 2023-11-28T20:53:35.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb000b9b0 con 0x7fbbb0004d00 2023-11-28T20:53:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb000bb70 con 0x7fbbc00717e0 2023-11-28T20:53:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.032+0000 7fbbbffff640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb0003f40 0x7fbbb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.033+0000 7fbbc4d61640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc00717e0 0x7fbbb0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.033+0000 7fbbbffff640 1 -- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc00717e0 msgr2=0x7fbbb0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.033+0000 7fbbbffff640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc00717e0 0x7fbbb0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.033+0000 7fbbc5562640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbb0004d00 0x7fbbb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.033+0000 7fbbbffff640 1 -- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbb0004d00 msgr2=0x7fbbb0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.033+0000 7fbbbffff640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbb0004d00 0x7fbbb0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.033+0000 7fbbbffff640 1 -- 172.21.15.45:0/2850648813 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb0003520 con 0x7fbbb0003f40 2023-11-28T20:53:35.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.034+0000 7fbbbffff640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb0003f40 0x7fbbb00047c0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009ac0 tx=0x7fbbb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e897f4687e01e06d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:35.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac02c6b0 con 0x7fbbb0003f40 2023-11-28T20:53:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac02c810 con 0x7fbbb0003f40 2023-11-28T20:53:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbbf7fe640 1 -- 172.21.15.45:0/2850648813 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac02cba0 con 0x7fbbb0003f40 2023-11-28T20:53:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbc5d63640 1 -- 172.21.15.45:0/2850648813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb0003f40 msgr2=0x7fbbb00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbc5d63640 1 --2- 172.21.15.45:0/2850648813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbb0003f40 0x7fbbb00047c0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009ac0 tx=0x7fbbb00057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbc5562640 1 -- 172.21.15.45:0/2850648813 reap_dead start 2023-11-28T20:53:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbc5d63640 1 -- 172.21.15.45:0/2850648813 shutdown_connections 2023-11-28T20:53:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbc5d63640 1 -- 172.21.15.45:0/2850648813 >> 172.21.15.45:0/2850648813 conn(0x7fbbc00b1050 msgr2=0x7fbbc00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbc5d63640 1 -- 172.21.15.45:0/2850648813 shutdown_connections 2023-11-28T20:53:35.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.035+0000 7fbbc5d63640 1 -- 172.21.15.45:0/2850648813 wait complete. 2023-11-28T20:53:35.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.037+0000 7fbbc5d63640 1 Processor -- start 2023-11-28T20:53:35.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.037+0000 7fbbc5d63640 1 -- start start 2023-11-28T20:53:35.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.038+0000 7fbbc5d63640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbb0003f40 0x7fbbc0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:35.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.038+0000 7fbbc5d63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01646c0 0x7fbbc01f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:35.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.038+0000 7fbbc5d63640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc009c800 0x7fbbc00af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:35.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.038+0000 7fbbc5d63640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc009b530 con 0x7fbbc01646c0 2023-11-28T20:53:35.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.038+0000 7fbbc5d63640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc009b760 con 0x7fbbb0003f40 2023-11-28T20:53:35.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.038+0000 7fbbc5d63640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc009ba40 con 0x7fbbc009c800 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbbffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01646c0 0x7fbbc01f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbbffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01646c0 0x7fbbc01f6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49558/0 (socket says 172.21.15.45:49558) 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbbffff640 1 -- 172.21.15.45:0/1170245576 learned_addr learned my addr 172.21.15.45:0/1170245576 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbc5562640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc009c800 0x7fbbc00af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbc4d61640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbb0003f40 0x7fbbc0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbbffff640 1 -- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc009c800 msgr2=0x7fbbc00af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbbffff640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc009c800 0x7fbbc00af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbbffff640 1 -- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbb0003f40 msgr2=0x7fbbc0099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbbffff640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbb0003f40 0x7fbbc0099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.039+0000 7fbbbffff640 1 -- 172.21.15.45:0/1170245576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc009bd20 con 0x7fbbc01646c0 2023-11-28T20:53:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.040+0000 7fbbbffff640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01646c0 0x7fbbc01f6770 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fbbac000c00 tx=0x7fbbac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.040+0000 7fbbbd7fa640 1 -- 172.21.15.45:0/1170245576 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac003810 con 0x7fbbc01646c0 2023-11-28T20:53:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.040+0000 7fbbbd7fa640 1 -- 172.21.15.45:0/1170245576 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac003990 con 0x7fbbc01646c0 2023-11-28T20:53:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.040+0000 7fbbbd7fa640 1 -- 172.21.15.45:0/1170245576 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac003d40 con 0x7fbbc01646c0 2023-11-28T20:53:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.040+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc009c000 con 0x7fbbc01646c0 2023-11-28T20:53:35.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.040+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc0099f80 con 0x7fbbc01646c0 2023-11-28T20:53:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.041+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc009a110 con 0x7fbbc01646c0 2023-11-28T20:53:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.043+0000 7fbbbd7fa640 1 -- 172.21.15.45:0/1170245576 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbbac03b070 con 0x7fbbc01646c0 2023-11-28T20:53:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.044+0000 7fbbbd7fa640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbba406af90 0x7fbba406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.044+0000 7fbbbd7fa640 1 -- 172.21.15.45:0/1170245576 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fbbac0ab690 con 0x7fbbc01646c0 2023-11-28T20:53:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.044+0000 7fbbc4d61640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbba406af90 0x7fbba406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.045+0000 7fbbc4d61640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbba406af90 0x7fbba406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4000c00 tx=0x7fbbb4000f10 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:35.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.047+0000 7fbbbd7fa640 1 -- 172.21.15.45:0/1170245576 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbbac0766e0 con 0x7fbbc01646c0 2023-11-28T20:53:35.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.279+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fbbc009b530 con 0x7fbbc01646c0 2023-11-28T20:53:35.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.823+0000 7fbbbd7fa640 1 -- 172.21.15.45:0/1170245576 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v55) v1 ==== 174+0+0 (secure 0 0 0) 0x7fbbac079d00 con 0x7fbbc01646c0 2023-11-28T20:53:35.831 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 15 and data pool 16 2023-11-28T20:53:35.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.835+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbba406af90 msgr2=0x7fbba406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:35.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.835+0000 7fbbc5d63640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbba406af90 0x7fbba406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4000c00 tx=0x7fbbb4000f10 comp rx=0 tx=0).stop 2023-11-28T20:53:35.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.835+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01646c0 msgr2=0x7fbbc01f6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:35.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.835+0000 7fbbc5d63640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01646c0 0x7fbbc01f6770 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fbbac000c00 tx=0x7fbbac004730 comp rx=0 tx=0).stop 2023-11-28T20:53:35.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.836+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 shutdown_connections 2023-11-28T20:53:35.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.836+0000 7fbbc5d63640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbba406af90 0x7fbba406d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.836+0000 7fbbc5d63640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbc009c800 0x7fbbc00af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.836+0000 7fbbc5d63640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01646c0 0x7fbbc01f6770 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.836+0000 7fbbc5d63640 1 --2- 172.21.15.45:0/1170245576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbb0003f40 0x7fbbc0099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:35.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.836+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 >> 172.21.15.45:0/1170245576 conn(0x7fbbc00b1050 msgr2=0x7fbbc006bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:35.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.837+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 shutdown_connections 2023-11-28T20:53:35.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:35.837+0000 7fbbc5d63640 1 -- 172.21.15.45:0/1170245576 wait complete. 2023-11-28T20:53:35.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:53:36.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.056+0000 7f21dc0c9640 1 Processor -- start 2023-11-28T20:53:36.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.056+0000 7f21dc0c9640 1 -- start start 2023-11-28T20:53:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.058+0000 7f21dc0c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40b1340 0x7f21d415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.058+0000 7f21dc0c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d415bda0 con 0x7f21d40b35b0 2023-11-28T20:53:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.058+0000 7f21dc0c9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d415c080 con 0x7f21d415b850 2023-11-28T20:53:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.058+0000 7f21dc0c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d415ee40 con 0x7f21d40b1340 2023-11-28T20:53:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.058+0000 7f21d9e65640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f21d40b35b0 0x7f21d415ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54958/0 (socket says 172.21.15.45:54958) 2023-11-28T20:53:36.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.058+0000 7f21d9e65640 1 -- 172.21.15.45:0/323824129 learned_addr learned my addr 172.21.15.45:0/323824129 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21da666640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40b1340 0x7f21d415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174450620 0 0) 0x7f21d415bda0 con 0x7f21d40b35b0 2023-11-28T20:53:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c0003520 con 0x7f21d40b35b0 2023-11-28T20:53:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3224831479 0 0) 0x7f21c0003520 con 0x7f21d40b35b0 2023-11-28T20:53:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40b1340 msgr2=0x7f21d415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40b1340 0x7f21d415f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 >> v1:172.21.15.190:6789/0 conn(0x7f21d415b850 legacy=0x7f21d40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d415c360 con 0x7f21d40b35b0 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f21d0001670 con 0x7f21d40b35b0 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 >> v1:172.21.15.45:6789/0 conn(0x7f21d40b35b0 legacy=0x7f21d415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40b1340 0x7f21c00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21c0003f20 0x7f21c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c0004d00 0x7f21c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d415ee40 con 0x7f21c0003f20 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c000ba60 con 0x7f21c0004d00 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.059+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c000bc20 con 0x7f21d40b1340 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21d9664640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21c0003f20 0x7f21c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21d9e65640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40b1340 0x7f21c00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21da666640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c0004d00 0x7f21c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21d9664640 1 -- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40b1340 msgr2=0x7f21c00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21d9664640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40b1340 0x7f21c00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21d9664640 1 -- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c0004d00 msgr2=0x7f21c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21d9664640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c0004d00 0x7f21c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21d9664640 1 -- 172.21.15.45:0/323824129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c0003520 con 0x7f21c0003f20 2023-11-28T20:53:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21d9e65640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40b1340 0x7f21c00039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:36.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.060+0000 7f21da666640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c0004d00 0x7f21c0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:36.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.061+0000 7f21d9664640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21c0003f20 0x7f21c00047c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f21c4001720 tx=0x7f21c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7260a506848f5d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:36.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c40052d0 con 0x7f21c0003f20 2023-11-28T20:53:36.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c402e050 con 0x7f21c0003f20 2023-11-28T20:53:36.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21d8e63640 1 -- 172.21.15.45:0/323824129 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4003830 con 0x7f21c0003f20 2023-11-28T20:53:36.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21dc0c9640 1 -- 172.21.15.45:0/323824129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21c0003f20 msgr2=0x7f21c00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21dc0c9640 1 --2- 172.21.15.45:0/323824129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21c0003f20 0x7f21c00047c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f21c4001720 tx=0x7f21c00057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21da666640 1 -- 172.21.15.45:0/323824129 reap_dead start 2023-11-28T20:53:36.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21dc0c9640 1 -- 172.21.15.45:0/323824129 shutdown_connections 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21dc0c9640 1 -- 172.21.15.45:0/323824129 >> 172.21.15.45:0/323824129 conn(0x7f21d406a500 msgr2=0x7f21d406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21dc0c9640 1 -- 172.21.15.45:0/323824129 shutdown_connections 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.062+0000 7f21dc0c9640 1 -- 172.21.15.45:0/323824129 wait complete. 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.064+0000 7f21dc0c9640 1 Processor -- start 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.064+0000 7f21dc0c9640 1 -- start start 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.065+0000 7f21dc0c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c0003f20 0x7f21d40ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.065+0000 7f21dc0c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d4141920 0x7f21d40ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.065+0000 7f21dc0c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40af950 0x7f21d40afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.065+0000 7f21dc0c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40b0310 con 0x7f21d4141920 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.065+0000 7f21dc0c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40adca0 con 0x7f21c0003f20 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.065+0000 7f21dc0c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40acdf0 con 0x7f21d40af950 2023-11-28T20:53:36.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.065+0000 7f21d9664640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d4141920 0x7f21d40ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.066+0000 7f21d9664640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d4141920 0x7f21d40ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49584/0 (socket says 172.21.15.45:49584) 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.066+0000 7f21d9664640 1 -- 172.21.15.45:0/1656821828 learned_addr learned my addr 172.21.15.45:0/1656821828 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.066+0000 7f21d9664640 1 -- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40af950 msgr2=0x7f21d40afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.066+0000 7f21d9664640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40af950 0x7f21d40afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.066+0000 7f21d9664640 1 -- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c0003f20 msgr2=0x7f21d40ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.066+0000 7f21d9664640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c0003f20 0x7f21d40ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.066+0000 7f21d9664640 1 -- 172.21.15.45:0/1656821828 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d40ad090 con 0x7f21d4141920 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.066+0000 7f21d9664640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d4141920 0x7f21d40ac8b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005040 tx=0x7f21c4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.066+0000 7f21ca7fc640 1 -- 172.21.15.45:0/1656821828 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4007a30 con 0x7f21d4141920 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.067+0000 7f21ca7fc640 1 -- 172.21.15.45:0/1656821828 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c4007640 con 0x7f21d4141920 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.067+0000 7f21ca7fc640 1 -- 172.21.15.45:0/1656821828 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4006be0 con 0x7f21d4141920 2023-11-28T20:53:36.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.067+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d40ad370 con 0x7f21d4141920 2023-11-28T20:53:36.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.067+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d40ad930 con 0x7f21d4141920 2023-11-28T20:53:36.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.068+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d40adac0 con 0x7f21d4141920 2023-11-28T20:53:36.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.069+0000 7f21ca7fc640 1 -- 172.21.15.45:0/1656821828 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f21c4006d40 con 0x7f21d4141920 2023-11-28T20:53:36.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.070+0000 7f21ca7fc640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f21b806af60 0x7f21b806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.070+0000 7f21ca7fc640 1 -- 172.21.15.45:0/1656821828 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f21c40ab8e0 con 0x7f21d4141920 2023-11-28T20:53:36.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.071+0000 7f21d9e65640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f21b806af60 0x7f21b806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.072+0000 7f21d9e65640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f21b806af60 0x7f21b806d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21d0001eb0 tx=0x7f21d0009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:36.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.075+0000 7f21ca7fc640 1 -- 172.21.15.45:0/1656821828 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21c40766b0 con 0x7f21d4141920 2023-11-28T20:53:36.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.271+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21d40733d0 con 0x7f21d4141920 2023-11-28T20:53:36.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.272+0000 7f21ca7fc640 1 -- 172.21.15.45:0/1656821828 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f21c404b020 con 0x7f21d4141920 2023-11-28T20:53:36.275 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:36.275 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":132,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:35.281281+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:53:33.200533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-11-28T20:53:34.300717+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:53:36.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f21b806af60 msgr2=0x7f21b806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f21b806af60 0x7f21b806d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21d0001eb0 tx=0x7f21d0009040 comp rx=0 tx=0).stop 2023-11-28T20:53:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d4141920 msgr2=0x7f21d40ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d4141920 0x7f21d40ac8b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005040 tx=0x7f21c4001120 comp rx=0 tx=0).stop 2023-11-28T20:53:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 shutdown_connections 2023-11-28T20:53:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f21b806af60 0x7f21b806d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21d40af950 0x7f21d40afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21d4141920 0x7f21d40ac8b0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 --2- 172.21.15.45:0/1656821828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21c0003f20 0x7f21d40ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 >> 172.21.15.45:0/1656821828 conn(0x7f21d406a500 msgr2=0x7f21d40b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 shutdown_connections 2023-11-28T20:53:36.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.275+0000 7f21dc0c9640 1 -- 172.21.15.45:0/1656821828 wait complete. 2023-11-28T20:53:36.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:53:36.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.513+0000 7feeeced4640 1 Processor -- start 2023-11-28T20:53:36.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.513+0000 7feeeced4640 1 -- start start 2023-11-28T20:53:36.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.515+0000 7feeeced4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feee8070b20 0x7feee8070f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.515+0000 7feeeced4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee80d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.515+0000 7feeeced4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee80714c0 con 0x7feee8160c80 2023-11-28T20:53:36.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.515+0000 7feeeced4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee80d28f0 con 0x7feee8070b20 2023-11-28T20:53:36.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.515+0000 7feeeced4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee80ce930 con 0x7feee80d1f20 2023-11-28T20:53:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.516+0000 7feee6d9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee80d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.516+0000 7feeddd9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feee8070b20 0x7feee8070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.516+0000 7feee6d9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee80d23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:47586/0 (socket says 172.21.15.45:47586) 2023-11-28T20:53:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.516+0000 7feee6d9d640 1 -- 172.21.15.45:0/3960270410 learned_addr learned my addr 172.21.15.45:0/3960270410 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.516+0000 7feee659c640 1 --1- 172.21.15.45:0/3960270410 >> v1:172.21.15.190:6789/0 conn(0x7feee8160c80 0x7feee80b6640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:40548/0 (socket says 172.21.15.45:40548) 2023-11-28T20:53:36.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.517+0000 7feee6d9d640 1 -- 172.21.15.45:0/3960270410 >> v2:172.21.15.190:3300/0 conn(0x7feee8070b20 msgr2=0x7feee8070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:36.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.517+0000 7feee6d9d640 1 --2- 172.21.15.45:0/3960270410 >> v2:172.21.15.190:3300/0 conn(0x7feee8070b20 0x7feee8070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.517+0000 7feee6d9d640 1 -- 172.21.15.45:0/3960270410 >> v1:172.21.15.190:6789/0 conn(0x7feee8160c80 legacy=0x7feee80b6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.517+0000 7feee6d9d640 1 -- 172.21.15.45:0/3960270410 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee80d1440 con 0x7feee80d1f20 2023-11-28T20:53:36.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.518+0000 7feee6d9d640 1 --2- 172.21.15.45:0/3960270410 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee80d23b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7feee0009e50 tx=0x7feee002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e7b71b47ae0683 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:36.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feee5d9b640 1 -- 172.21.15.45:0/3960270410 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee002cb00 con 0x7feee80d1f20 2023-11-28T20:53:36.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feee5d9b640 1 -- 172.21.15.45:0/3960270410 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee002cc60 con 0x7feee80d1f20 2023-11-28T20:53:36.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feee5d9b640 1 -- 172.21.15.45:0/3960270410 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee00343f0 con 0x7feee80d1f20 2023-11-28T20:53:36.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feeeced4640 1 -- 172.21.15.45:0/3960270410 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 msgr2=0x7feee80d23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feeeced4640 1 --2- 172.21.15.45:0/3960270410 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee80d23b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7feee0009e50 tx=0x7feee002bf60 comp rx=0 tx=0).stop 2023-11-28T20:53:36.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feeeced4640 1 -- 172.21.15.45:0/3960270410 shutdown_connections 2023-11-28T20:53:36.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feeeced4640 1 --2- 172.21.15.45:0/3960270410 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee80d23b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feeeced4640 1 --2- 172.21.15.45:0/3960270410 >> v2:172.21.15.190:3300/0 conn(0x7feee8070b20 0x7feee8070f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feeeced4640 1 -- 172.21.15.45:0/3960270410 >> 172.21.15.45:0/3960270410 conn(0x7feee80cfcb0 msgr2=0x7feee80d00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:36.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.519+0000 7feeeced4640 1 -- 172.21.15.45:0/3960270410 shutdown_connections 2023-11-28T20:53:36.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.520+0000 7feeeced4640 1 -- 172.21.15.45:0/3960270410 wait complete. 2023-11-28T20:53:36.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.521+0000 7feeeced4640 1 Processor -- start 2023-11-28T20:53:36.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.521+0000 7feeeced4640 1 -- start start 2023-11-28T20:53:36.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.522+0000 7feeeced4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee8070b20 0x7feee80caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.522+0000 7feeeced4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee8157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.522+0000 7feeeced4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee8160c80 0x7feee81f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.522+0000 7feeeced4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee8157e60 con 0x7feee8070b20 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.522+0000 7feeeced4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee81f39f0 con 0x7feee8160c80 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.522+0000 7feeeced4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee8158870 con 0x7feee80d1f20 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee8070b20 0x7feee80caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee8070b20 0x7feee80caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49590/0 (socket says 172.21.15.45:49590) 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee659c640 1 -- 172.21.15.45:0/3756700908 learned_addr learned my addr 172.21.15.45:0/3756700908 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feeddd9b640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee8157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee6d9d640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee8160c80 0x7feee81f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee659c640 1 -- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 msgr2=0x7feee8157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee659c640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee8157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee659c640 1 -- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee8160c80 msgr2=0x7feee81f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee659c640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee8160c80 0x7feee81f34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee659c640 1 -- 172.21.15.45:0/3756700908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0009b30 con 0x7feee8070b20 2023-11-28T20:53:36.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feee6d9d640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee8160c80 0x7feee81f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:36.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.523+0000 7feeddd9b640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee8157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:36.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.524+0000 7feee659c640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee8070b20 0x7feee80caf40 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7feed800c4d0 tx=0x7feed800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:36.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.524+0000 7feedf7fe640 1 -- 172.21.15.45:0/3756700908 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed8007480 con 0x7feee8070b20 2023-11-28T20:53:36.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.524+0000 7feedf7fe640 1 -- 172.21.15.45:0/3756700908 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed80075e0 con 0x7feee8070b20 2023-11-28T20:53:36.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.524+0000 7feedf7fe640 1 -- 172.21.15.45:0/3756700908 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed80049e0 con 0x7feee8070b20 2023-11-28T20:53:36.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.524+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee80cb780 con 0x7feee8070b20 2023-11-28T20:53:36.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.524+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee80cbd40 con 0x7feee8070b20 2023-11-28T20:53:36.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.525+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee80733d0 con 0x7feee8070b20 2023-11-28T20:53:36.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.526+0000 7feedf7fe640 1 -- 172.21.15.45:0/3756700908 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7feed8004b40 con 0x7feee8070b20 2023-11-28T20:53:36.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.527+0000 7feedf7fe640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7feec806af60 0x7feec806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.527+0000 7feedf7fe640 1 -- 172.21.15.45:0/3756700908 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6029+0+0 (secure 0 0 0) 0x7feed8088950 con 0x7feee8070b20 2023-11-28T20:53:36.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.528+0000 7feeddd9b640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7feec806af60 0x7feec806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.529+0000 7feeddd9b640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7feec806af60 0x7feec806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feee8071950 tx=0x7feed0008040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:36.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.532+0000 7feedf7fe640 1 -- 172.21.15.45:0/3756700908 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feed8053860 con 0x7feee8070b20 2023-11-28T20:53:36.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.727+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feee80d1110 con 0x7feee8070b20 2023-11-28T20:53:36.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.731+0000 7feedf7fe640 1 -- 172.21.15.45:0/3756700908 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16717 (secure 0 0 0) 0x7feed8018020 con 0x7feee8070b20 2023-11-28T20:53:36.733 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:36.734 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":132,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:35.281281+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:53:33.200533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-11-28T20:53:34.300717+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:53:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7feec806af60 msgr2=0x7feec806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7feec806af60 0x7feec806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feee8071950 tx=0x7feed0008040 comp rx=0 tx=0).stop 2023-11-28T20:53:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee8070b20 msgr2=0x7feee80caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee8070b20 0x7feee80caf40 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7feed800c4d0 tx=0x7feed800c990 comp rx=0 tx=0).stop 2023-11-28T20:53:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 shutdown_connections 2023-11-28T20:53:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7feec806af60 0x7feec806d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee8160c80 0x7feee81f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee80d1f20 0x7feee8157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 --2- 172.21.15.45:0/3756700908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee8070b20 0x7feee80caf40 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.733+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 >> 172.21.15.45:0/3756700908 conn(0x7feee80cfcb0 msgr2=0x7feee80b4520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.734+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 shutdown_connections 2023-11-28T20:53:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.734+0000 7feeeced4640 1 -- 172.21.15.45:0/3756700908 wait complete. 2023-11-28T20:53:36.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T20:53:36.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.979+0000 7f13e12d7640 1 Processor -- start 2023-11-28T20:53:36.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.979+0000 7f13e12d7640 1 -- start start 2023-11-28T20:53:36.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.981+0000 7f13e12d7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13dc0cdca0 0x7f13dc06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.981+0000 7f13e12d7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc06d410 con 0x7f13dc0b3040 2023-11-28T20:53:36.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.981+0000 7f13e12d7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc06d6f0 con 0x7f13dc15b270 2023-11-28T20:53:36.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.981+0000 7f13e12d7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13dc0ceaa0 con 0x7f13dc0cdca0 2023-11-28T20:53:36.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.981+0000 7f13e0ad6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13dc0cdca0 0x7f13dc06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.981+0000 7f13dbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f13dc0b3040 0x7f13dc15f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54972/0 (socket says 172.21.15.45:54972) 2023-11-28T20:53:36.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.981+0000 7f13dbfff640 1 -- 172.21.15.45:0/3227103984 learned_addr learned my addr 172.21.15.45:0/3227103984 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.982+0000 7f13e0ad6640 1 -- 172.21.15.45:0/3227103984 >> v1:172.21.15.190:6789/0 conn(0x7f13dc15b270 legacy=0x7f13dc0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.982+0000 7f13e0ad6640 1 -- 172.21.15.45:0/3227103984 >> v1:172.21.15.45:6789/0 conn(0x7f13dc0b3040 legacy=0x7f13dc15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.982+0000 7f13e0ad6640 1 -- 172.21.15.45:0/3227103984 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc06d9d0 con 0x7f13dc0cdca0 2023-11-28T20:53:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.982+0000 7f13daffd640 1 -- 172.21.15.45:0/3227103984 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162613676 0 0) 0x7f13dc06d410 con 0x7f13dc0b3040 2023-11-28T20:53:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.983+0000 7f13e0ad6640 1 --2- 172.21.15.45:0/3227103984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 0x7f13dc06cda0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f13d4009d90 tx=0x7f13d402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71479fca65b273e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.983+0000 7f13daffd640 1 -- 172.21.15.45:0/3227103984 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d402cb40 con 0x7f13dc0cdca0 2023-11-28T20:53:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.983+0000 7f13daffd640 1 -- 172.21.15.45:0/3227103984 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d402cca0 con 0x7f13dc0cdca0 2023-11-28T20:53:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.983+0000 7f13daffd640 1 -- 172.21.15.45:0/3227103984 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4034660 con 0x7f13dc0cdca0 2023-11-28T20:53:36.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.984+0000 7f13e12d7640 1 -- 172.21.15.45:0/3227103984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 msgr2=0x7f13dc06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.984+0000 7f13e12d7640 1 --2- 172.21.15.45:0/3227103984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 0x7f13dc06cda0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f13d4009d90 tx=0x7f13d402bd90 comp rx=0 tx=0).stop 2023-11-28T20:53:36.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.984+0000 7f13e12d7640 1 -- 172.21.15.45:0/3227103984 shutdown_connections 2023-11-28T20:53:36.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.984+0000 7f13e12d7640 1 --2- 172.21.15.45:0/3227103984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 0x7f13dc06cda0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.984+0000 7f13e12d7640 1 -- 172.21.15.45:0/3227103984 >> 172.21.15.45:0/3227103984 conn(0x7f13dc0b1050 msgr2=0x7f13dc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:36.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.984+0000 7f13e12d7640 1 -- 172.21.15.45:0/3227103984 shutdown_connections 2023-11-28T20:53:36.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.984+0000 7f13e12d7640 1 -- 172.21.15.45:0/3227103984 wait complete. 2023-11-28T20:53:36.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.986+0000 7f13e12d7640 1 Processor -- start 2023-11-28T20:53:36.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.986+0000 7f13e12d7640 1 -- start start 2023-11-28T20:53:36.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.987+0000 7f13e12d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13dc0b3040 0x7f13dc14eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.987+0000 7f13e12d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 0x7f13dc15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.987+0000 7f13e12d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13dc15b270 0x7f13dc14d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.987+0000 7f13e12d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0ad690 con 0x7f13dc0cdca0 2023-11-28T20:53:36.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.987+0000 7f13e12d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc14f170 con 0x7f13dc0b3040 2023-11-28T20:53:36.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.987+0000 7f13e12d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0ad8e0 con 0x7f13dc15b270 2023-11-28T20:53:36.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.987+0000 7f13db7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 0x7f13dc15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13db7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 0x7f13dc15b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49608/0 (socket says 172.21.15.45:49608) 2023-11-28T20:53:36.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13db7fe640 1 -- 172.21.15.45:0/2739194449 learned_addr learned my addr 172.21.15.45:0/2739194449 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:36.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13dbfff640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13dc0b3040 0x7f13dc14eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13e0ad6640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13dc15b270 0x7f13dc14d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13db7fe640 1 -- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13dc15b270 msgr2=0x7f13dc14d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13db7fe640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13dc15b270 0x7f13dc14d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13db7fe640 1 -- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13dc0b3040 msgr2=0x7f13dc14eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:36.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13db7fe640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13dc0b3040 0x7f13dc14eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:36.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13db7fe640 1 -- 172.21.15.45:0/2739194449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d4009a70 con 0x7f13dc0cdca0 2023-11-28T20:53:36.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.988+0000 7f13db7fe640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 0x7f13dc15b6a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f13d000b3c0 tx=0x7f13d000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:36.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.989+0000 7f13d8ff9640 1 -- 172.21.15.45:0/2739194449 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0011830 con 0x7f13dc0cdca0 2023-11-28T20:53:36.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.989+0000 7f13d8ff9640 1 -- 172.21.15.45:0/2739194449 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0011990 con 0x7f13dc0cdca0 2023-11-28T20:53:36.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.989+0000 7f13d8ff9640 1 -- 172.21.15.45:0/2739194449 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0011d80 con 0x7f13dc0cdca0 2023-11-28T20:53:36.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.989+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc0adbc0 con 0x7f13dc0cdca0 2023-11-28T20:53:36.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.989+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc14f940 con 0x7f13dc0cdca0 2023-11-28T20:53:36.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.990+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc0ce490 con 0x7f13dc0cdca0 2023-11-28T20:53:36.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.991+0000 7f13d8ff9640 1 -- 172.21.15.45:0/2739194449 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f13d001b460 con 0x7f13dc0cdca0 2023-11-28T20:53:36.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.993+0000 7f13d8ff9640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f13c806af60 0x7f13c806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:36.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.993+0000 7f13d8ff9640 1 -- 172.21.15.45:0/2739194449 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f13d0089500 con 0x7f13dc0cdca0 2023-11-28T20:53:36.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.994+0000 7f13dbfff640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f13c806af60 0x7f13c806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:36.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.995+0000 7f13dbfff640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f13c806af60 0x7f13c806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13cc000f60 tx=0x7f13cc007b50 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:36.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:36.997+0000 7f13d8ff9640 1 -- 172.21.15.45:0/2739194449 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13d00542c0 con 0x7f13dc0cdca0 2023-11-28T20:53:37.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.224+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f13dc0addb0 con 0x7f13dc0cdca0 2023-11-28T20:53:37.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.839+0000 7f13d8ff9640 1 -- 172.21.15.45:0/2739194449 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v58) v1 ==== 118+0+0 (secure 0 0 0) 0x7f13d00578e0 con 0x7f13dc0cdca0 2023-11-28T20:53:37.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f13c806af60 msgr2=0x7f13c806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:37.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f13c806af60 0x7f13c806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13cc000f60 tx=0x7f13cc007b50 comp rx=0 tx=0).stop 2023-11-28T20:53:37.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 msgr2=0x7f13dc15b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:37.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 0x7f13dc15b6a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f13d000b3c0 tx=0x7f13d000b880 comp rx=0 tx=0).stop 2023-11-28T20:53:37.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 shutdown_connections 2023-11-28T20:53:37.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f13c806af60 0x7f13c806d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:37.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13dc15b270 0x7f13dc14d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:37.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13dc0cdca0 0x7f13dc15b6a0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:37.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 --2- 172.21.15.45:0/2739194449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13dc0b3040 0x7f13dc14eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:37.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.843+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 >> 172.21.15.45:0/2739194449 conn(0x7f13dc0b1050 msgr2=0x7f13dc0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:37.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.844+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 shutdown_connections 2023-11-28T20:53:37.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:37.844+0000 7f13e12d7640 1 -- 172.21.15.45:0/2739194449 wait complete. 2023-11-28T20:53:37.866 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T20:53:37.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T20:53:38.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.094+0000 7fc5ab59e640 1 Processor -- start 2023-11-28T20:53:38.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.094+0000 7fc5ab59e640 1 -- start start 2023-11-28T20:53:38.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.096+0000 7fc5ab59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc5ac15b270 0x7fc5ac06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.096+0000 7fc5ab59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc5ac0b3420 0x7fc5ac0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.096+0000 7fc5ab59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ac06d410 con 0x7fc5ac0b3040 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.096+0000 7fc5ab59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ac0ceaa0 con 0x7fc5ac15b270 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.096+0000 7fc5ab59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ac06d6f0 con 0x7fc5ac0b3420 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.096+0000 7fc5a3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc5ac15b270 0x7fc5ac06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.096+0000 7fc5a3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc5ac15b270 0x7fc5ac06cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49616/0 (socket says 172.21.15.45:49616) 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.096+0000 7fc5a3fff640 1 -- 172.21.15.45:0/3659008973 learned_addr learned my addr 172.21.15.45:0/3659008973 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.097+0000 7fc5a3fff640 1 -- 172.21.15.45:0/3659008973 >> v2:172.21.15.190:3300/0 conn(0x7fc5ac0b3420 msgr2=0x7fc5ac0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.097+0000 7fc5aad9d640 1 --2- 172.21.15.45:0/3659008973 >> v2:172.21.15.190:3300/0 conn(0x7fc5ac0b3420 0x7fc5ac0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.097+0000 7fc5a3fff640 1 --2- 172.21.15.45:0/3659008973 >> v2:172.21.15.190:3300/0 conn(0x7fc5ac0b3420 0x7fc5ac0ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:38.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.097+0000 7fc5a3fff640 1 -- 172.21.15.45:0/3659008973 >> v1:172.21.15.190:6789/0 conn(0x7fc5ac0b3040 legacy=0x7fc5ac15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.097+0000 7fc5a3fff640 1 -- 172.21.15.45:0/3659008973 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac06d9d0 con 0x7fc5ac15b270 2023-11-28T20:53:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.097+0000 7fc5aad9d640 1 --2- 172.21.15.45:0/3659008973 >> v2:172.21.15.190:3300/0 conn(0x7fc5ac0b3420 0x7fc5ac0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.099+0000 7fc5a3fff640 1 --2- 172.21.15.45:0/3659008973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac15b270 0x7fc5ac06cda0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0adfd0 tx=0x7fc59402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7882d259caeca0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.099+0000 7fc5a9d9b640 1 -- 172.21.15.45:0/3659008973 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5940317e0 con 0x7fc5ac15b270 2023-11-28T20:53:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.099+0000 7fc5a9d9b640 1 -- 172.21.15.45:0/3659008973 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc594031940 con 0x7fc5ac15b270 2023-11-28T20:53:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.099+0000 7fc5a9d9b640 1 -- 172.21.15.45:0/3659008973 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594031cd0 con 0x7fc5ac15b270 2023-11-28T20:53:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.100+0000 7fc5ab59e640 1 -- 172.21.15.45:0/3659008973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac15b270 msgr2=0x7fc5ac06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.100+0000 7fc5ab59e640 1 --2- 172.21.15.45:0/3659008973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac15b270 0x7fc5ac06cda0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0adfd0 tx=0x7fc59402b880 comp rx=0 tx=0).stop 2023-11-28T20:53:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.100+0000 7fc5ab59e640 1 -- 172.21.15.45:0/3659008973 shutdown_connections 2023-11-28T20:53:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.100+0000 7fc5ab59e640 1 --2- 172.21.15.45:0/3659008973 >> v2:172.21.15.190:3300/0 conn(0x7fc5ac0b3420 0x7fc5ac0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.100+0000 7fc5ab59e640 1 --2- 172.21.15.45:0/3659008973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac15b270 0x7fc5ac06cda0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.100+0000 7fc5ab59e640 1 -- 172.21.15.45:0/3659008973 >> 172.21.15.45:0/3659008973 conn(0x7fc5ac0b1050 msgr2=0x7fc5ac0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.100+0000 7fc5ab59e640 1 -- 172.21.15.45:0/3659008973 shutdown_connections 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.100+0000 7fc5ab59e640 1 -- 172.21.15.45:0/3659008973 wait complete. 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.102+0000 7fc5ab59e640 1 Processor -- start 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.102+0000 7fc5ab59e640 1 -- start start 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.103+0000 7fc5ab59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac0b3040 0x7fc5ac09acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.103+0000 7fc5ab59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5ac0b3420 0x7fc5ac14e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.103+0000 7fc5ab59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5ac15b270 0x7fc5ac15cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.103+0000 7fc5ab59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac14ee70 con 0x7fc5ac0b3040 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.103+0000 7fc5ab59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac09b360 con 0x7fc5ac0b3420 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.103+0000 7fc5ab59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac09c1c0 con 0x7fc5ac15b270 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.103+0000 7fc5aa59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac0b3040 0x7fc5ac09acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.103+0000 7fc5aa59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac0b3040 0x7fc5ac09acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49618/0 (socket says 172.21.15.45:49618) 2023-11-28T20:53:38.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.103+0000 7fc5aa59c640 1 -- 172.21.15.45:0/2320312132 learned_addr learned my addr 172.21.15.45:0/2320312132 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:38.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5aa59c640 1 -- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5ac15b270 msgr2=0x7fc5ac15cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:38.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5a3fff640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5ac0b3420 0x7fc5ac14e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:38.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5aa59c640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5ac15b270 0x7fc5ac15cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:38.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5aa59c640 1 -- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5ac0b3420 msgr2=0x7fc5ac14e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:38.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5aa59c640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5ac0b3420 0x7fc5ac14e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:38.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5aa59c640 1 -- 172.21.15.45:0/2320312132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5940094d0 con 0x7fc5ac0b3040 2023-11-28T20:53:38.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5a3fff640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5ac0b3420 0x7fc5ac14e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5aa59c640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac0b3040 0x7fc5ac09acf0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fc59c00c450 tx=0x7fc59c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5a2ffd640 1 -- 172.21.15.45:0/2320312132 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59c0052d0 con 0x7fc5ac0b3040 2023-11-28T20:53:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5a2ffd640 1 -- 172.21.15.45:0/2320312132 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59c005430 con 0x7fc5ac0b3040 2023-11-28T20:53:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5a2ffd640 1 -- 172.21.15.45:0/2320312132 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59c012870 con 0x7fc5ac0b3040 2023-11-28T20:53:38.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ac09c4a0 con 0x7fc5ac0b3040 2023-11-28T20:53:38.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.104+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ac14be30 con 0x7fc5ac0b3040 2023-11-28T20:53:38.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.106+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ac09b4d0 con 0x7fc5ac0b3040 2023-11-28T20:53:38.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.107+0000 7fc5a2ffd640 1 -- 172.21.15.45:0/2320312132 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc59c0129d0 con 0x7fc5ac0b3040 2023-11-28T20:53:38.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.108+0000 7fc5a2ffd640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc58c06af60 0x7fc58c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:38.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.108+0000 7fc5a2ffd640 1 -- 172.21.15.45:0/2320312132 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc59c089420 con 0x7fc5ac0b3040 2023-11-28T20:53:38.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.109+0000 7fc5a3fff640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc58c06af60 0x7fc58c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:38.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.111+0000 7fc5a3fff640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc58c06af60 0x7fc58c06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc594036000 tx=0x7fc59403d040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:38.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.114+0000 7fc5a2ffd640 1 -- 172.21.15.45:0/2320312132 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc59c054270 con 0x7fc5ac0b3040 2023-11-28T20:53:38.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.345+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc5ac09d0f0 con 0x7fc5ac0b3040 2023-11-28T20:53:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.851+0000 7fc5a2ffd640 1 -- 172.21.15.45:0/2320312132 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc59c057890 con 0x7fc5ac0b3040 2023-11-28T20:53:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.854+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc58c06af60 msgr2=0x7fc58c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.854+0000 7fc5ab59e640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc58c06af60 0x7fc58c06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc594036000 tx=0x7fc59403d040 comp rx=0 tx=0).stop 2023-11-28T20:53:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac0b3040 msgr2=0x7fc5ac09acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac0b3040 0x7fc5ac09acf0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fc59c00c450 tx=0x7fc59c00c910 comp rx=0 tx=0).stop 2023-11-28T20:53:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 shutdown_connections 2023-11-28T20:53:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fc58c06af60 0x7fc58c06d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5ac15b270 0x7fc5ac15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5ac0b3420 0x7fc5ac14e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 --2- 172.21.15.45:0/2320312132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5ac0b3040 0x7fc5ac09acf0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 >> 172.21.15.45:0/2320312132 conn(0x7fc5ac0b1050 msgr2=0x7fc5ac0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 shutdown_connections 2023-11-28T20:53:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:38.855+0000 7fc5ab59e640 1 -- 172.21.15.45:0/2320312132 wait complete. 2023-11-28T20:53:38.879 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T20:53:38.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:53:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.103+0000 7ff2d899e640 1 Processor -- start 2023-11-28T20:53:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.103+0000 7ff2d899e640 1 -- start start 2023-11-28T20:53:39.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.104+0000 7ff2d899e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff2d40ae540 0x7ff2d40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.104+0000 7ff2d899e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d415c3c0 con 0x7ff2d40ce630 2023-11-28T20:53:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.104+0000 7ff2d899e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d4155b60 con 0x7ff2d4155780 2023-11-28T20:53:39.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.104+0000 7ff2d899e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d40ceac0 con 0x7ff2d40ae540 2023-11-28T20:53:39.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.104+0000 7ff2d3fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff2d40ce630 0x7ff2d40b20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54988/0 (socket says 172.21.15.45:54988) 2023-11-28T20:53:39.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.104+0000 7ff2d3fff640 1 -- 172.21.15.45:0/240867550 learned_addr learned my addr 172.21.15.45:0/240867550 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:39.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.104+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489869938 0 0) 0x7ff2d415c3c0 con 0x7ff2d40ce630 2023-11-28T20:53:39.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.104+0000 7ff2cbfff640 1 --2- 172.21.15.45:0/240867550 >> v2:172.21.15.190:3300/0 conn(0x7ff2d40ae540 0x7ff2d40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.104+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c0003520 con 0x7ff2d40ce630 2023-11-28T20:53:39.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592878000 0 0) 0x7ff2c0003520 con 0x7ff2d40ce630 2023-11-28T20:53:39.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 >> v2:172.21.15.190:3300/0 conn(0x7ff2d40ae540 msgr2=0x7ff2d40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:39.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 --2- 172.21.15.45:0/240867550 >> v2:172.21.15.190:3300/0 conn(0x7ff2d40ae540 0x7ff2d40ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 >> v1:172.21.15.190:6789/0 conn(0x7ff2d4155780 legacy=0x7ff2d40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d415ba90 con 0x7ff2d40ce630 2023-11-28T20:53:39.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ff2cc0019f0 con 0x7ff2d40ce630 2023-11-28T20:53:39.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 >> v1:172.21.15.45:6789/0 conn(0x7ff2d40ce630 legacy=0x7ff2d40b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c0003970 0x7ff2c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2c00047c0 0x7ff2c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2c0005180 0x7ff2c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d40ceac0 con 0x7ff2c0005180 2023-11-28T20:53:39.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c000c0d0 con 0x7ff2c0003970 2023-11-28T20:53:39.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c000c290 con 0x7ff2c00047c0 2023-11-28T20:53:39.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.105+0000 7ff2d3fff640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2c0005180 0x7ff2c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.106+0000 7ff2d37fe640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c0003970 0x7ff2c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.106+0000 7ff2cbfff640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2c00047c0 0x7ff2c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.106+0000 7ff2d3fff640 1 -- 172.21.15.45:0/240867550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2c00047c0 msgr2=0x7ff2c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:39.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.106+0000 7ff2d3fff640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2c00047c0 0x7ff2c0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.106+0000 7ff2d3fff640 1 -- 172.21.15.45:0/240867550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c0003970 msgr2=0x7ff2c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:39.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.106+0000 7ff2d3fff640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c0003970 0x7ff2c0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.106+0000 7ff2d3fff640 1 -- 172.21.15.45:0/240867550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c0003520 con 0x7ff2c0005180 2023-11-28T20:53:39.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.107+0000 7ff2d3fff640 1 -- 172.21.15.45:0/240867550 reap_dead start 2023-11-28T20:53:39.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.107+0000 7ff2d3fff640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2c0005180 0x7ff2c000b980 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00a830 tx=0x7ff2c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b715e9b522ef731 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:39.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.107+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc0072d0 con 0x7ff2c0005180 2023-11-28T20:53:39.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.107+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc02f070 con 0x7ff2c0005180 2023-11-28T20:53:39.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.107+0000 7ff2d2ffd640 1 -- 172.21.15.45:0/240867550 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc0076b0 con 0x7ff2c0005180 2023-11-28T20:53:39.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.108+0000 7ff2d899e640 1 -- 172.21.15.45:0/240867550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2c0005180 msgr2=0x7ff2c000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.108+0000 7ff2d899e640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2c0005180 0x7ff2c000b980 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00a830 tx=0x7ff2c00057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.108+0000 7ff2d899e640 1 -- 172.21.15.45:0/240867550 shutdown_connections 2023-11-28T20:53:39.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.108+0000 7ff2d899e640 1 --2- 172.21.15.45:0/240867550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2c0005180 0x7ff2c000b980 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.108+0000 7ff2d899e640 1 -- 172.21.15.45:0/240867550 >> 172.21.15.45:0/240867550 conn(0x7ff2d406ab30 msgr2=0x7ff2d40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.108+0000 7ff2d899e640 1 -- 172.21.15.45:0/240867550 shutdown_connections 2023-11-28T20:53:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.108+0000 7ff2d899e640 1 -- 172.21.15.45:0/240867550 wait complete. 2023-11-28T20:53:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.110+0000 7ff2d899e640 1 Processor -- start 2023-11-28T20:53:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.110+0000 7ff2d899e640 1 -- start start 2023-11-28T20:53:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.111+0000 7ff2d899e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c0005180 0x7ff2d414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.111+0000 7ff2d899e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d4153c60 0x7ff2d41f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.111+0000 7ff2d899e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2d409a2e0 0x7ff2d40afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.111+0000 7ff2d899e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d4099e50 con 0x7ff2d4153c60 2023-11-28T20:53:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.111+0000 7ff2d899e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d414cd40 con 0x7ff2c0005180 2023-11-28T20:53:39.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.111+0000 7ff2d899e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d41f37e0 con 0x7ff2d409a2e0 2023-11-28T20:53:39.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.111+0000 7ff2cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d4153c60 0x7ff2d41f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.111+0000 7ff2cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d4153c60 0x7ff2d41f2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49628/0 (socket says 172.21.15.45:49628) 2023-11-28T20:53:39.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.112+0000 7ff2cbfff640 1 -- 172.21.15.45:0/2427092605 learned_addr learned my addr 172.21.15.45:0/2427092605 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:39.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.112+0000 7ff2d37fe640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c0005180 0x7ff2d414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.112+0000 7ff2d3fff640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2d409a2e0 0x7ff2d40afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.112+0000 7ff2cbfff640 1 -- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2d409a2e0 msgr2=0x7ff2d40afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.112+0000 7ff2cbfff640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2d409a2e0 0x7ff2d40afa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.112+0000 7ff2cbfff640 1 -- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c0005180 msgr2=0x7ff2d414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.112+0000 7ff2cbfff640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c0005180 0x7ff2d414c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.112+0000 7ff2cbfff640 1 -- 172.21.15.45:0/2427092605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d41f3ac0 con 0x7ff2d4153c60 2023-11-28T20:53:39.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.112+0000 7ff2cbfff640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d4153c60 0x7ff2d41f2f10 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc00e7e0 tx=0x7ff2bc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:39.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.113+0000 7ff2d0ff9640 1 -- 172.21.15.45:0/2427092605 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc00c6d0 con 0x7ff2d4153c60 2023-11-28T20:53:39.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.113+0000 7ff2d0ff9640 1 -- 172.21.15.45:0/2427092605 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2bc00c830 con 0x7ff2d4153c60 2023-11-28T20:53:39.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.113+0000 7ff2d0ff9640 1 -- 172.21.15.45:0/2427092605 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc00cba0 con 0x7ff2d4153c60 2023-11-28T20:53:39.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.113+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d41f3da0 con 0x7ff2d4153c60 2023-11-28T20:53:39.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.113+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d41f3450 con 0x7ff2d4153c60 2023-11-28T20:53:39.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.114+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d409a000 con 0x7ff2d4153c60 2023-11-28T20:53:39.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.115+0000 7ff2d0ff9640 1 -- 172.21.15.45:0/2427092605 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff2bc00cd00 con 0x7ff2d4153c60 2023-11-28T20:53:39.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.117+0000 7ff2d0ff9640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7ff2b406af90 0x7ff2b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.117+0000 7ff2d0ff9640 1 -- 172.21.15.45:0/2427092605 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff2bc0897d0 con 0x7ff2d4153c60 2023-11-28T20:53:39.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.118+0000 7ff2d37fe640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7ff2b406af90 0x7ff2b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.118+0000 7ff2d37fe640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7ff2b406af90 0x7ff2b406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4007fd0 tx=0x7ff2c4009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:39.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.121+0000 7ff2d0ff9640 1 -- 172.21.15.45:0/2427092605 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2d409a000 con 0x7ff2d4153c60 2023-11-28T20:53:39.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.351+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2d4099f90 con 0x7ff2d4153c60 2023-11-28T20:53:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.353+0000 7ff2d0ff9640 1 -- 172.21.15.45:0/2427092605 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4918 (secure 0 0 0) 0x7ff2bc054510 con 0x7ff2d4153c60 2023-11-28T20:53:39.355 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:39.356 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/544145878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":544145878},{"type":"v1","addr":"172.21.15.190:6836","nonce":544145878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2464692926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2464692926},{"type":"v1","addr":"172.21.15.45:6835","nonce":2464692926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2048455360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2048455360},{"type":"v1","addr":"172.21.15.190:6837","nonce":2048455360}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:53:35.282168+0000","modified":"2023-11-28T20:53:38.346980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2464465186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2464465186},{"type":"v1","addr":"172.21.15.45:6834","nonce":2464465186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-28T20:53:39.356 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 59 2023-11-28T20:53:39.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.356+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7ff2b406af90 msgr2=0x7ff2b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.356+0000 7ff2d899e640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7ff2b406af90 0x7ff2b406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4007fd0 tx=0x7ff2c4009040 comp rx=0 tx=0).stop 2023-11-28T20:53:39.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.356+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d4153c60 msgr2=0x7ff2d41f2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.356+0000 7ff2d899e640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d4153c60 0x7ff2d41f2f10 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc00e7e0 tx=0x7ff2bc00eca0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.357+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 shutdown_connections 2023-11-28T20:53:39.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.357+0000 7ff2d899e640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7ff2b406af90 0x7ff2b406d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.357+0000 7ff2d899e640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2d409a2e0 0x7ff2d40afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.357+0000 7ff2d899e640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d4153c60 0x7ff2d41f2f10 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.357+0000 7ff2d899e640 1 --2- 172.21.15.45:0/2427092605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c0005180 0x7ff2d414c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.357+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 >> 172.21.15.45:0/2427092605 conn(0x7ff2d406ab30 msgr2=0x7ff2d415a160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:39.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.357+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 shutdown_connections 2023-11-28T20:53:39.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.357+0000 7ff2d899e640 1 -- 172.21.15.45:0/2427092605 wait complete. 2023-11-28T20:53:39.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:53:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.603+0000 7fb1e359e640 1 Processor -- start 2023-11-28T20:53:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.603+0000 7fb1e359e640 1 -- start start 2023-11-28T20:53:39.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.605+0000 7fb1e359e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb1e4155780 0x7fb1e40abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.605+0000 7fb1e359e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb1e40ae540 0x7fb1e40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.605+0000 7fb1e359e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e415c3c0 con 0x7fb1e40ce630 2023-11-28T20:53:39.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.605+0000 7fb1e359e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e40ceac0 con 0x7fb1e4155780 2023-11-28T20:53:39.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.605+0000 7fb1e359e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e4155b60 con 0x7fb1e40ae540 2023-11-28T20:53:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.606+0000 7fb1e2d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb1e40ce630 0x7fb1e415ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54990/0 (socket says 172.21.15.45:54990) 2023-11-28T20:53:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.606+0000 7fb1e2d9d640 1 -- 172.21.15.45:0/1112384843 learned_addr learned my addr 172.21.15.45:0/1112384843 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.606+0000 7fb1e259c640 1 --2- 172.21.15.45:0/1112384843 >> v2:172.21.15.45:3300/0 conn(0x7fb1e4155780 0x7fb1e40abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.606+0000 7fb1dbfff640 1 --2- 172.21.15.45:0/1112384843 >> v2:172.21.15.190:3300/0 conn(0x7fb1e40ae540 0x7fb1e40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.606+0000 7fb1e1d9b640 1 -- 172.21.15.45:0/1112384843 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766550818 0 0) 0x7fb1e415c3c0 con 0x7fb1e40ce630 2023-11-28T20:53:39.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.606+0000 7fb1e259c640 1 -- 172.21.15.45:0/1112384843 >> v2:172.21.15.190:3300/0 conn(0x7fb1e40ae540 msgr2=0x7fb1e40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:39.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.606+0000 7fb1e259c640 1 --2- 172.21.15.45:0/1112384843 >> v2:172.21.15.190:3300/0 conn(0x7fb1e40ae540 0x7fb1e40ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.606+0000 7fb1e259c640 1 -- 172.21.15.45:0/1112384843 >> v1:172.21.15.45:6789/0 conn(0x7fb1e40ce630 legacy=0x7fb1e415ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.606+0000 7fb1e259c640 1 -- 172.21.15.45:0/1112384843 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e415ba90 con 0x7fb1e4155780 2023-11-28T20:53:39.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.607+0000 7fb1e259c640 1 --2- 172.21.15.45:0/1112384843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e4155780 0x7fb1e40abd70 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40cfec0 tx=0x7fb1d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce9697aac738266 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e1d9b640 1 -- 172.21.15.45:0/1112384843 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4031830 con 0x7fb1e4155780 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e1d9b640 1 -- 172.21.15.45:0/1112384843 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d4031990 con 0x7fb1e4155780 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e1d9b640 1 -- 172.21.15.45:0/1112384843 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4031d20 con 0x7fb1e4155780 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e359e640 1 -- 172.21.15.45:0/1112384843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e4155780 msgr2=0x7fb1e40abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e359e640 1 --2- 172.21.15.45:0/1112384843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e4155780 0x7fb1e40abd70 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40cfec0 tx=0x7fb1d402b8d0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e359e640 1 -- 172.21.15.45:0/1112384843 shutdown_connections 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e359e640 1 --2- 172.21.15.45:0/1112384843 >> v2:172.21.15.190:3300/0 conn(0x7fb1e40ae540 0x7fb1e40ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e359e640 1 --2- 172.21.15.45:0/1112384843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e4155780 0x7fb1e40abd70 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e359e640 1 -- 172.21.15.45:0/1112384843 >> 172.21.15.45:0/1112384843 conn(0x7fb1e406ab30 msgr2=0x7fb1e40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e359e640 1 -- 172.21.15.45:0/1112384843 shutdown_connections 2023-11-28T20:53:39.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.608+0000 7fb1e359e640 1 -- 172.21.15.45:0/1112384843 wait complete. 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.610+0000 7fb1e359e640 1 Processor -- start 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.610+0000 7fb1e359e640 1 -- start start 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.611+0000 7fb1e359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e40ae540 0x7fb1e41416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.611+0000 7fb1e359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1e40ce630 0x7fb1e409c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.611+0000 7fb1e359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1e4155780 0x7fb1e4157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.611+0000 7fb1e359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e4141c10 con 0x7fb1e40ae540 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.611+0000 7fb1e359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e41568b0 con 0x7fb1e40ce630 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.611+0000 7fb1e359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e4156b90 con 0x7fb1e4155780 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1e259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e40ae540 0x7fb1e41416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1e259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e40ae540 0x7fb1e41416d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49650/0 (socket says 172.21.15.45:49650) 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1e259c640 1 -- 172.21.15.45:0/4175311946 learned_addr learned my addr 172.21.15.45:0/4175311946 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1e2d9d640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1e4155780 0x7fb1e4157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1dbfff640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1e40ce630 0x7fb1e409c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1e259c640 1 -- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1e4155780 msgr2=0x7fb1e4157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1e259c640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1e4155780 0x7fb1e4157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1e259c640 1 -- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1e40ce630 msgr2=0x7fb1e409c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1e259c640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1e40ce630 0x7fb1e409c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.612+0000 7fb1e259c640 1 -- 172.21.15.45:0/4175311946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d4009400 con 0x7fb1e40ae540 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.613+0000 7fb1e259c640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e40ae540 0x7fb1e41416d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4000c00 tx=0x7fb1d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.613+0000 7fb1daffd640 1 -- 172.21.15.45:0/4175311946 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d403aa60 con 0x7fb1e40ae540 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.613+0000 7fb1daffd640 1 -- 172.21.15.45:0/4175311946 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d403abe0 con 0x7fb1e40ae540 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.613+0000 7fb1daffd640 1 -- 172.21.15.45:0/4175311946 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d40443f0 con 0x7fb1e40ae540 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.613+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e4156e70 con 0x7fb1e40ae540 2023-11-28T20:53:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.613+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e4157430 con 0x7fb1e40ae540 2023-11-28T20:53:39.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.614+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e40abf70 con 0x7fb1e40ae540 2023-11-28T20:53:39.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.615+0000 7fb1daffd640 1 -- 172.21.15.45:0/4175311946 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb1d404d450 con 0x7fb1e40ae540 2023-11-28T20:53:39.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.616+0000 7fb1daffd640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb1c406af60 0x7fb1c406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:39.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.616+0000 7fb1daffd640 1 -- 172.21.15.45:0/4175311946 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb1d40abb70 con 0x7fb1e40ae540 2023-11-28T20:53:39.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.617+0000 7fb1dbfff640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb1c406af60 0x7fb1c406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:39.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.618+0000 7fb1dbfff640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb1c406af60 0x7fb1c406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4141f80 tx=0x7fb1cc008040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:39.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.621+0000 7fb1daffd640 1 -- 172.21.15.45:0/4175311946 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1d40768c0 con 0x7fb1e40ae540 2023-11-28T20:53:39.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.815+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e404e920 con 0x7fb1e40ae540 2023-11-28T20:53:39.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.818+0000 7fb1daffd640 1 -- 172.21.15.45:0/4175311946 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+16704 (secure 0 0 0) 0x7fb1d4035070 con 0x7fb1e40ae540 2023-11-28T20:53:39.819 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:39.819 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":133,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:36.799419+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:53:33.200533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-11-28T20:53:34.300717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:53:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.820+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb1c406af60 msgr2=0x7fb1c406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.820+0000 7fb1e359e640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb1c406af60 0x7fb1c406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4141f80 tx=0x7fb1cc008040 comp rx=0 tx=0).stop 2023-11-28T20:53:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.820+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e40ae540 msgr2=0x7fb1e41416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.820+0000 7fb1e359e640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e40ae540 0x7fb1e41416d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4000c00 tx=0x7fb1d40046c0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.821+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 shutdown_connections 2023-11-28T20:53:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.821+0000 7fb1e359e640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fb1c406af60 0x7fb1c406d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.821+0000 7fb1e359e640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1e4155780 0x7fb1e4157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.821+0000 7fb1e359e640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1e40ce630 0x7fb1e409c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.821+0000 7fb1e359e640 1 --2- 172.21.15.45:0/4175311946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1e40ae540 0x7fb1e41416d0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:39.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.821+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 >> 172.21.15.45:0/4175311946 conn(0x7fb1e406ab30 msgr2=0x7fb1e40b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:39.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.821+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 shutdown_connections 2023-11-28T20:53:39.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:39.821+0000 7fb1e359e640 1 -- 172.21.15.45:0/4175311946 wait complete. 2023-11-28T20:53:39.841 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T20:53:39.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:53:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.066+0000 7fbfa8c4b640 1 Processor -- start 2023-11-28T20:53:40.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.066+0000 7fbfa8c4b640 1 -- start start 2023-11-28T20:53:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.068+0000 7fbfa8c4b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 0x7fbfa40abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.068+0000 7fbfa8c4b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbfa40ae540 0x7fbfa40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.068+0000 7fbfa8c4b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbfa40ce630 0x7fbfa415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.068+0000 7fbfa8c4b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa40ceac0 con 0x7fbfa40ce630 2023-11-28T20:53:40.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.068+0000 7fbfa8c4b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa415c3c0 con 0x7fbfa40ae540 2023-11-28T20:53:40.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.068+0000 7fbfa8c4b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4155b60 con 0x7fbfa4155780 2023-11-28T20:53:40.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.068+0000 7fbfa3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbfa40ce630 0x7fbfa415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.068+0000 7fbfa3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbfa40ce630 0x7fbfa415ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49654/0 (socket says 172.21.15.45:49654) 2023-11-28T20:53:40.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.068+0000 7fbfa3fff640 1 -- 172.21.15.45:0/567217538 learned_addr learned my addr 172.21.15.45:0/567217538 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:40.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.069+0000 7fbfa37fe640 1 --2- 172.21.15.45:0/567217538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 0x7fbfa40abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.069+0000 7fbf9bfff640 1 --2- 172.21.15.45:0/567217538 >> v2:172.21.15.190:3300/0 conn(0x7fbfa40ae540 0x7fbfa40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.069+0000 7fbfa3fff640 1 -- 172.21.15.45:0/567217538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 msgr2=0x7fbfa40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:40.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.069+0000 7fbfa3fff640 1 --2- 172.21.15.45:0/567217538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 0x7fbfa40abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.069+0000 7fbfa3fff640 1 -- 172.21.15.45:0/567217538 >> v2:172.21.15.190:3300/0 conn(0x7fbfa40ae540 msgr2=0x7fbfa40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.069+0000 7fbfa3fff640 1 --2- 172.21.15.45:0/567217538 >> v2:172.21.15.190:3300/0 conn(0x7fbfa40ae540 0x7fbfa40ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.069+0000 7fbfa3fff640 1 -- 172.21.15.45:0/567217538 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa415ba90 con 0x7fbfa40ce630 2023-11-28T20:53:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.069+0000 7fbf9bfff640 1 --2- 172.21.15.45:0/567217538 >> v2:172.21.15.190:3300/0 conn(0x7fbfa40ae540 0x7fbfa40ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:40.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.070+0000 7fbfa3fff640 1 --2- 172.21.15.45:0/567217538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ce630 0x7fbfa415ed30 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009e50 tx=0x7fbf9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703124a813f84be3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.070+0000 7fbfa2ffd640 1 -- 172.21.15.45:0/567217538 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c02ca70 con 0x7fbfa40ce630 2023-11-28T20:53:40.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.070+0000 7fbfa2ffd640 1 -- 172.21.15.45:0/567217538 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c02cbd0 con 0x7fbfa40ce630 2023-11-28T20:53:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.070+0000 7fbfa2ffd640 1 -- 172.21.15.45:0/567217538 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0343f0 con 0x7fbfa40ce630 2023-11-28T20:53:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.071+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/567217538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ce630 msgr2=0x7fbfa415ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.071+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/567217538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ce630 0x7fbfa415ed30 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009e50 tx=0x7fbf9c02bed0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.071+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/567217538 shutdown_connections 2023-11-28T20:53:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.071+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/567217538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ce630 0x7fbfa415ed30 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.071+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/567217538 >> v2:172.21.15.190:3300/0 conn(0x7fbfa40ae540 0x7fbfa40ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.071+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/567217538 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 0x7fbfa40abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.071+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/567217538 >> 172.21.15.45:0/567217538 conn(0x7fbfa406ab30 msgr2=0x7fbfa40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.071+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/567217538 shutdown_connections 2023-11-28T20:53:40.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.071+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/567217538 wait complete. 2023-11-28T20:53:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.073+0000 7fbfa8c4b640 1 Processor -- start 2023-11-28T20:53:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.073+0000 7fbfa8c4b640 1 -- start start 2023-11-28T20:53:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.074+0000 7fbfa8c4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ae540 0x7fbfa4140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.074+0000 7fbfa8c4b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa40ce630 0x7fbfa40afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.074+0000 7fbfa8c4b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 0x7fbfa40b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.074+0000 7fbfa8c4b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40b00f0 con 0x7fbfa40ae540 2023-11-28T20:53:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.074+0000 7fbfa8c4b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40af010 con 0x7fbfa40ce630 2023-11-28T20:53:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.074+0000 7fbfa8c4b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4140a30 con 0x7fbfa4155780 2023-11-28T20:53:40.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ae540 0x7fbfa4140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ae540 0x7fbfa4140380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49662/0 (socket says 172.21.15.45:49662) 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa37fe640 1 -- 172.21.15.45:0/4159862623 learned_addr learned my addr 172.21.15.45:0/4159862623 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa3fff640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 0x7fbfa40b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbf9bfff640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa40ce630 0x7fbfa40afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa37fe640 1 -- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 msgr2=0x7fbfa40b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa37fe640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 0x7fbfa40b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa37fe640 1 -- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa40ce630 msgr2=0x7fbfa40afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa37fe640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa40ce630 0x7fbfa40afbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa37fe640 1 -- 172.21.15.45:0/4159862623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c009b30 con 0x7fbfa40ae540 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbf9bfff640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa40ce630 0x7fbfa40afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.075+0000 7fbfa37fe640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ae540 0x7fbfa4140380 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400d5f0 tx=0x7fbf9400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.076+0000 7fbfa0ff9640 1 -- 172.21.15.45:0/4159862623 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf940125f0 con 0x7fbfa40ae540 2023-11-28T20:53:40.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.076+0000 7fbfa0ff9640 1 -- 172.21.15.45:0/4159862623 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf94012750 con 0x7fbfa40ae540 2023-11-28T20:53:40.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.076+0000 7fbfa0ff9640 1 -- 172.21.15.45:0/4159862623 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94012b00 con 0x7fbfa40ae540 2023-11-28T20:53:40.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.076+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa4140d10 con 0x7fbfa40ae540 2023-11-28T20:53:40.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.076+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa41412d0 con 0x7fbfa40ae540 2023-11-28T20:53:40.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.077+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa404e920 con 0x7fbfa40ae540 2023-11-28T20:53:40.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.078+0000 7fbfa0ff9640 1 -- 172.21.15.45:0/4159862623 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbf94012c60 con 0x7fbfa40ae540 2023-11-28T20:53:40.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.079+0000 7fbfa0ff9640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbf8406af90 0x7fbf8406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.079+0000 7fbfa0ff9640 1 -- 172.21.15.45:0/4159862623 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbf94089d20 con 0x7fbfa40ae540 2023-11-28T20:53:40.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.080+0000 7fbf9bfff640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbf8406af90 0x7fbf8406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.081+0000 7fbf9bfff640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbf8406af90 0x7fbf8406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbfa406d2d0 tx=0x7fbf8c008040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:40.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.083+0000 7fbfa0ff9640 1 -- 172.21.15.45:0/4159862623 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf9408e070 con 0x7fbfa40ae540 2023-11-28T20:53:40.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.274+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 --> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbfa415b740 con 0x7fbf8406af90 2023-11-28T20:53:40.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.303+0000 7fbfa0ff9640 1 -- 172.21.15.45:0/4159862623 <== mgr.6655 v2:172.21.15.190:6832/3959324769 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287194 (secure 0 0 0) 0x7fbf8c00c040 con 0x7fbf8406af90 2023-11-28T20:53:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:40.310 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:53:40.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbf8406af90 msgr2=0x7fbf8406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbf8406af90 0x7fbf8406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbfa406d2d0 tx=0x7fbf8c008040 comp rx=0 tx=0).stop 2023-11-28T20:53:40.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ae540 msgr2=0x7fbfa4140380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ae540 0x7fbfa4140380 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400d5f0 tx=0x7fbf9400dab0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 shutdown_connections 2023-11-28T20:53:40.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fbf8406af90 0x7fbf8406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa4155780 0x7fbfa40b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa40ce630 0x7fbfa40afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 --2- 172.21.15.45:0/4159862623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfa40ae540 0x7fbfa4140380 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.309+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 >> 172.21.15.45:0/4159862623 conn(0x7fbfa406ab30 msgr2=0x7fbfa414e350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:40.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.310+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 shutdown_connections 2023-11-28T20:53:40.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.310+0000 7fbfa8c4b640 1 -- 172.21.15.45:0/4159862623 wait complete. 2023-11-28T20:53:40.332 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-11-28T20:53:39.654069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":997,"num_read_kb":1979,"num_write":375,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4750,"ondisk_log_size":4750,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":350836,"kb_used_data":13976,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754623884,"statfs":{"total":773094113280,"available":772734857216,"internally_reserved":0,"allocated":14311424,"data_stored":7798006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":344837680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.872249"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.853502+0000","last_change":"2023-11-28T20:53:35.852739+0000","last_active":"2023-11-28T20:53:35.853502+0000","last_peered":"2023-11-28T20:53:35.853502+0000","last_clean":"2023-11-28T20:53:35.853502+0000","last_became_active":"2023-11-28T20:53:35.849844+0000","last_became_peered":"2023-11-28T20:53:35.849844+0000","last_unstale":"2023-11-28T20:53:35.853502+0000","last_undegraded":"2023-11-28T20:53:35.853502+0000","last_fullsized":"2023-11-28T20:53:35.853502+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:06:32.653220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.840589+0000","last_change":"2023-11-28T20:53:35.834922+0000","last_active":"2023-11-28T20:53:35.840589+0000","last_peered":"2023-11-28T20:53:35.840589+0000","last_clean":"2023-11-28T20:53:35.840589+0000","last_became_active":"2023-11-28T20:53:35.834671+0000","last_became_peered":"2023-11-28T20:53:35.834671+0000","last_unstale":"2023-11-28T20:53:35.840589+0000","last_undegraded":"2023-11-28T20:53:35.840589+0000","last_fullsized":"2023-11-28T20:53:35.840589+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:41:48.627407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.849976+0000","last_change":"2023-11-28T20:53:35.838626+0000","last_active":"2023-11-28T20:53:35.849976+0000","last_peered":"2023-11-28T20:53:35.849976+0000","last_clean":"2023-11-28T20:53:35.849976+0000","last_became_active":"2023-11-28T20:53:35.836237+0000","last_became_peered":"2023-11-28T20:53:35.836237+0000","last_unstale":"2023-11-28T20:53:35.849976+0000","last_undegraded":"2023-11-28T20:53:35.849976+0000","last_fullsized":"2023-11-28T20:53:35.849976+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:32.617542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.851868+0000","last_change":"2023-11-28T20:53:35.841472+0000","last_active":"2023-11-28T20:53:35.851868+0000","last_peered":"2023-11-28T20:53:35.851868+0000","last_clean":"2023-11-28T20:53:35.851868+0000","last_became_active":"2023-11-28T20:53:35.841120+0000","last_became_peered":"2023-11-28T20:53:35.841120+0000","last_unstale":"2023-11-28T20:53:35.851868+0000","last_undegraded":"2023-11-28T20:53:35.851868+0000","last_fullsized":"2023-11-28T20:53:35.851868+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:56:52.886021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841378+0000","last_change":"2023-11-28T20:53:35.834266+0000","last_active":"2023-11-28T20:53:35.841378+0000","last_peered":"2023-11-28T20:53:35.841378+0000","last_clean":"2023-11-28T20:53:35.841378+0000","last_became_active":"2023-11-28T20:53:35.833311+0000","last_became_peered":"2023-11-28T20:53:35.833311+0000","last_unstale":"2023-11-28T20:53:35.841378+0000","last_undegraded":"2023-11-28T20:53:35.841378+0000","last_fullsized":"2023-11-28T20:53:35.841378+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:34.248688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.840440+0000","last_change":"2023-11-28T20:53:35.834650+0000","last_active":"2023-11-28T20:53:35.840440+0000","last_peered":"2023-11-28T20:53:35.840440+0000","last_clean":"2023-11-28T20:53:35.840440+0000","last_became_active":"2023-11-28T20:53:35.833712+0000","last_became_peered":"2023-11-28T20:53:35.833712+0000","last_unstale":"2023-11-28T20:53:35.840440+0000","last_undegraded":"2023-11-28T20:53:35.840440+0000","last_fullsized":"2023-11-28T20:53:35.840440+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:21:04.742706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841121+0000","last_change":"2023-11-28T20:53:35.836961+0000","last_active":"2023-11-28T20:53:35.841121+0000","last_peered":"2023-11-28T20:53:35.841121+0000","last_clean":"2023-11-28T20:53:35.841121+0000","last_became_active":"2023-11-28T20:53:35.836668+0000","last_became_peered":"2023-11-28T20:53:35.836668+0000","last_unstale":"2023-11-28T20:53:35.841121+0000","last_undegraded":"2023-11-28T20:53:35.841121+0000","last_fullsized":"2023-11-28T20:53:35.841121+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:17:36.299170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.839952+0000","last_change":"2023-11-28T20:53:35.837333+0000","last_active":"2023-11-28T20:53:35.839952+0000","last_peered":"2023-11-28T20:53:35.839952+0000","last_clean":"2023-11-28T20:53:35.839952+0000","last_became_active":"2023-11-28T20:53:35.836533+0000","last_became_peered":"2023-11-28T20:53:35.836533+0000","last_unstale":"2023-11-28T20:53:35.839952+0000","last_undegraded":"2023-11-28T20:53:35.839952+0000","last_fullsized":"2023-11-28T20:53:35.839952+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:39:16.187369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.855412+0000","last_change":"2023-11-28T20:53:35.853355+0000","last_active":"2023-11-28T20:53:35.855412+0000","last_peered":"2023-11-28T20:53:35.855412+0000","last_clean":"2023-11-28T20:53:35.855412+0000","last_became_active":"2023-11-28T20:53:35.852724+0000","last_became_peered":"2023-11-28T20:53:35.852724+0000","last_unstale":"2023-11-28T20:53:35.855412+0000","last_undegraded":"2023-11-28T20:53:35.855412+0000","last_fullsized":"2023-11-28T20:53:35.855412+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:33.776878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.842277+0000","last_change":"2023-11-28T20:53:35.835427+0000","last_active":"2023-11-28T20:53:35.842277+0000","last_peered":"2023-11-28T20:53:35.842277+0000","last_clean":"2023-11-28T20:53:35.842277+0000","last_became_active":"2023-11-28T20:53:35.835135+0000","last_became_peered":"2023-11-28T20:53:35.835135+0000","last_unstale":"2023-11-28T20:53:35.842277+0000","last_undegraded":"2023-11-28T20:53:35.842277+0000","last_fullsized":"2023-11-28T20:53:35.842277+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:19:30.296129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.839068+0000","last_change":"2023-11-28T20:53:35.834185+0000","last_active":"2023-11-28T20:53:35.839068+0000","last_peered":"2023-11-28T20:53:35.839068+0000","last_clean":"2023-11-28T20:53:35.839068+0000","last_became_active":"2023-11-28T20:53:35.833907+0000","last_became_peered":"2023-11-28T20:53:35.833907+0000","last_unstale":"2023-11-28T20:53:35.839068+0000","last_undegraded":"2023-11-28T20:53:35.839068+0000","last_fullsized":"2023-11-28T20:53:35.839068+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:04:29.608154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.843407+0000","last_change":"2023-11-28T20:53:35.842302+0000","last_active":"2023-11-28T20:53:35.843407+0000","last_peered":"2023-11-28T20:53:35.843407+0000","last_clean":"2023-11-28T20:53:35.843407+0000","last_became_active":"2023-11-28T20:53:35.842012+0000","last_became_peered":"2023-11-28T20:53:35.842012+0000","last_unstale":"2023-11-28T20:53:35.843407+0000","last_undegraded":"2023-11-28T20:53:35.843407+0000","last_fullsized":"2023-11-28T20:53:35.843407+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:37:38.636899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.848755+0000","last_change":"2023-11-28T20:53:35.839463+0000","last_active":"2023-11-28T20:53:35.848755+0000","last_peered":"2023-11-28T20:53:35.848755+0000","last_clean":"2023-11-28T20:53:35.848755+0000","last_became_active":"2023-11-28T20:53:35.839191+0000","last_became_peered":"2023-11-28T20:53:35.839191+0000","last_unstale":"2023-11-28T20:53:35.848755+0000","last_undegraded":"2023-11-28T20:53:35.848755+0000","last_fullsized":"2023-11-28T20:53:35.848755+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:19.520202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.844226+0000","last_change":"2023-11-28T20:53:35.842783+0000","last_active":"2023-11-28T20:53:35.844226+0000","last_peered":"2023-11-28T20:53:35.844226+0000","last_clean":"2023-11-28T20:53:35.844226+0000","last_became_active":"2023-11-28T20:53:35.842015+0000","last_became_peered":"2023-11-28T20:53:35.842015+0000","last_unstale":"2023-11-28T20:53:35.844226+0000","last_undegraded":"2023-11-28T20:53:35.844226+0000","last_fullsized":"2023-11-28T20:53:35.844226+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:15:27.519053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.840467+0000","last_change":"2023-11-28T20:53:35.837702+0000","last_active":"2023-11-28T20:53:35.840467+0000","last_peered":"2023-11-28T20:53:35.840467+0000","last_clean":"2023-11-28T20:53:35.840467+0000","last_became_active":"2023-11-28T20:53:35.836427+0000","last_became_peered":"2023-11-28T20:53:35.836427+0000","last_unstale":"2023-11-28T20:53:35.840467+0000","last_undegraded":"2023-11-28T20:53:35.840467+0000","last_fullsized":"2023-11-28T20:53:35.840467+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:34.815897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835098+0000","last_change":"2023-11-28T20:53:35.833376+0000","last_active":"2023-11-28T20:53:35.835098+0000","last_peered":"2023-11-28T20:53:35.835098+0000","last_clean":"2023-11-28T20:53:35.835098+0000","last_became_active":"2023-11-28T20:53:35.833064+0000","last_became_peered":"2023-11-28T20:53:35.833064+0000","last_unstale":"2023-11-28T20:53:35.835098+0000","last_undegraded":"2023-11-28T20:53:35.835098+0000","last_fullsized":"2023-11-28T20:53:35.835098+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:08.301254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850087+0000","last_change":"2023-11-28T20:53:35.845140+0000","last_active":"2023-11-28T20:53:35.850087+0000","last_peered":"2023-11-28T20:53:35.850087+0000","last_clean":"2023-11-28T20:53:35.850087+0000","last_became_active":"2023-11-28T20:53:35.844482+0000","last_became_peered":"2023-11-28T20:53:35.844482+0000","last_unstale":"2023-11-28T20:53:35.850087+0000","last_undegraded":"2023-11-28T20:53:35.850087+0000","last_fullsized":"2023-11-28T20:53:35.850087+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:10:31.647188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846086+0000","last_change":"2023-11-28T20:53:35.843456+0000","last_active":"2023-11-28T20:53:35.846086+0000","last_peered":"2023-11-28T20:53:35.846086+0000","last_clean":"2023-11-28T20:53:35.846086+0000","last_became_active":"2023-11-28T20:53:35.843070+0000","last_became_peered":"2023-11-28T20:53:35.843070+0000","last_unstale":"2023-11-28T20:53:35.846086+0000","last_undegraded":"2023-11-28T20:53:35.846086+0000","last_fullsized":"2023-11-28T20:53:35.846086+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:13.510882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850870+0000","last_change":"2023-11-28T20:53:35.849865+0000","last_active":"2023-11-28T20:53:35.850870+0000","last_peered":"2023-11-28T20:53:35.850870+0000","last_clean":"2023-11-28T20:53:35.850870+0000","last_became_active":"2023-11-28T20:53:35.846752+0000","last_became_peered":"2023-11-28T20:53:35.846752+0000","last_unstale":"2023-11-28T20:53:35.850870+0000","last_undegraded":"2023-11-28T20:53:35.850870+0000","last_fullsized":"2023-11-28T20:53:35.850870+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:15:19.399253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.831167+0000","last_change":"2023-11-28T20:53:35.827047+0000","last_active":"2023-11-28T20:53:35.831167+0000","last_peered":"2023-11-28T20:53:35.831167+0000","last_clean":"2023-11-28T20:53:35.831167+0000","last_became_active":"2023-11-28T20:53:35.826494+0000","last_became_peered":"2023-11-28T20:53:35.826494+0000","last_unstale":"2023-11-28T20:53:35.831167+0000","last_undegraded":"2023-11-28T20:53:35.831167+0000","last_fullsized":"2023-11-28T20:53:35.831167+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:27:32.599929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.837616+0000","last_change":"2023-11-28T20:53:35.834662+0000","last_active":"2023-11-28T20:53:35.837616+0000","last_peered":"2023-11-28T20:53:35.837616+0000","last_clean":"2023-11-28T20:53:35.837616+0000","last_became_active":"2023-11-28T20:53:35.833862+0000","last_became_peered":"2023-11-28T20:53:35.833862+0000","last_unstale":"2023-11-28T20:53:35.837616+0000","last_undegraded":"2023-11-28T20:53:35.837616+0000","last_fullsized":"2023-11-28T20:53:35.837616+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:57.255815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.820855+0000","last_change":"2023-11-28T20:53:35.814684+0000","last_active":"2023-11-28T20:53:35.820855+0000","last_peered":"2023-11-28T20:53:35.820855+0000","last_clean":"2023-11-28T20:53:35.820855+0000","last_became_active":"2023-11-28T20:53:35.813927+0000","last_became_peered":"2023-11-28T20:53:35.813927+0000","last_unstale":"2023-11-28T20:53:35.820855+0000","last_undegraded":"2023-11-28T20:53:35.820855+0000","last_fullsized":"2023-11-28T20:53:35.820855+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:31.127376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.843322+0000","last_change":"2023-11-28T20:53:35.834592+0000","last_active":"2023-11-28T20:53:35.843322+0000","last_peered":"2023-11-28T20:53:35.843322+0000","last_clean":"2023-11-28T20:53:35.843322+0000","last_became_active":"2023-11-28T20:53:35.834246+0000","last_became_peered":"2023-11-28T20:53:35.834246+0000","last_unstale":"2023-11-28T20:53:35.843322+0000","last_undegraded":"2023-11-28T20:53:35.843322+0000","last_fullsized":"2023-11-28T20:53:35.843322+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:43:04.039584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.847971+0000","last_change":"2023-11-28T20:53:35.844787+0000","last_active":"2023-11-28T20:53:35.847971+0000","last_peered":"2023-11-28T20:53:35.847971+0000","last_clean":"2023-11-28T20:53:35.847971+0000","last_became_active":"2023-11-28T20:53:35.844514+0000","last_became_peered":"2023-11-28T20:53:35.844514+0000","last_unstale":"2023-11-28T20:53:35.847971+0000","last_undegraded":"2023-11-28T20:53:35.847971+0000","last_fullsized":"2023-11-28T20:53:35.847971+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:33.776878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.832733+0000","last_change":"2023-11-28T20:53:35.828275+0000","last_active":"2023-11-28T20:53:35.832733+0000","last_peered":"2023-11-28T20:53:35.832733+0000","last_clean":"2023-11-28T20:53:35.832733+0000","last_became_active":"2023-11-28T20:53:35.827251+0000","last_became_peered":"2023-11-28T20:53:35.827251+0000","last_unstale":"2023-11-28T20:53:35.832733+0000","last_undegraded":"2023-11-28T20:53:35.832733+0000","last_fullsized":"2023-11-28T20:53:35.832733+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:42:40.422048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.842615+0000","last_change":"2023-11-28T20:53:35.840731+0000","last_active":"2023-11-28T20:53:35.842615+0000","last_peered":"2023-11-28T20:53:35.842615+0000","last_clean":"2023-11-28T20:53:35.842615+0000","last_became_active":"2023-11-28T20:53:35.840152+0000","last_became_peered":"2023-11-28T20:53:35.840152+0000","last_unstale":"2023-11-28T20:53:35.842615+0000","last_undegraded":"2023-11-28T20:53:35.842615+0000","last_fullsized":"2023-11-28T20:53:35.842615+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:31.311166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835107+0000","last_change":"2023-11-28T20:53:35.828179+0000","last_active":"2023-11-28T20:53:35.835107+0000","last_peered":"2023-11-28T20:53:35.835107+0000","last_clean":"2023-11-28T20:53:35.835107+0000","last_became_active":"2023-11-28T20:53:35.827873+0000","last_became_peered":"2023-11-28T20:53:35.827873+0000","last_unstale":"2023-11-28T20:53:35.835107+0000","last_undegraded":"2023-11-28T20:53:35.835107+0000","last_fullsized":"2023-11-28T20:53:35.835107+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:39:53.846804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841387+0000","last_change":"2023-11-28T20:53:35.836673+0000","last_active":"2023-11-28T20:53:35.841387+0000","last_peered":"2023-11-28T20:53:35.841387+0000","last_clean":"2023-11-28T20:53:35.841387+0000","last_became_active":"2023-11-28T20:53:35.836401+0000","last_became_peered":"2023-11-28T20:53:35.836401+0000","last_unstale":"2023-11-28T20:53:35.841387+0000","last_undegraded":"2023-11-28T20:53:35.841387+0000","last_fullsized":"2023-11-28T20:53:35.841387+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:44:54.363477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.833629+0000","last_change":"2023-11-28T20:53:35.830744+0000","last_active":"2023-11-28T20:53:35.833629+0000","last_peered":"2023-11-28T20:53:35.833629+0000","last_clean":"2023-11-28T20:53:35.833629+0000","last_became_active":"2023-11-28T20:53:35.830405+0000","last_became_peered":"2023-11-28T20:53:35.830405+0000","last_unstale":"2023-11-28T20:53:35.833629+0000","last_undegraded":"2023-11-28T20:53:35.833629+0000","last_fullsized":"2023-11-28T20:53:35.833629+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:08:39.518595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.831083+0000","last_change":"2023-11-28T20:53:35.827527+0000","last_active":"2023-11-28T20:53:35.831083+0000","last_peered":"2023-11-28T20:53:35.831083+0000","last_clean":"2023-11-28T20:53:35.831083+0000","last_became_active":"2023-11-28T20:53:35.827184+0000","last_became_peered":"2023-11-28T20:53:35.827184+0000","last_unstale":"2023-11-28T20:53:35.831083+0000","last_undegraded":"2023-11-28T20:53:35.831083+0000","last_fullsized":"2023-11-28T20:53:35.831083+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:34.248688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850005+0000","last_change":"2023-11-28T20:53:35.848368+0000","last_active":"2023-11-28T20:53:35.850005+0000","last_peered":"2023-11-28T20:53:35.850005+0000","last_clean":"2023-11-28T20:53:35.850005+0000","last_became_active":"2023-11-28T20:53:35.848146+0000","last_became_peered":"2023-11-28T20:53:35.848146+0000","last_unstale":"2023-11-28T20:53:35.850005+0000","last_undegraded":"2023-11-28T20:53:35.850005+0000","last_fullsized":"2023-11-28T20:53:35.850005+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:43:13.391431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.834794+0000","last_change":"2023-11-28T20:53:35.832100+0000","last_active":"2023-11-28T20:53:35.834794+0000","last_peered":"2023-11-28T20:53:35.834794+0000","last_clean":"2023-11-28T20:53:35.834794+0000","last_became_active":"2023-11-28T20:53:35.831788+0000","last_became_peered":"2023-11-28T20:53:35.831788+0000","last_unstale":"2023-11-28T20:53:35.834794+0000","last_undegraded":"2023-11-28T20:53:35.834794+0000","last_fullsized":"2023-11-28T20:53:35.834794+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:55:17.222039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.849418+0000","last_change":"2023-11-28T20:53:35.840108+0000","last_active":"2023-11-28T20:53:35.849418+0000","last_peered":"2023-11-28T20:53:35.849418+0000","last_clean":"2023-11-28T20:53:35.849418+0000","last_became_active":"2023-11-28T20:53:35.839791+0000","last_became_peered":"2023-11-28T20:53:35.839791+0000","last_unstale":"2023-11-28T20:53:35.849418+0000","last_undegraded":"2023-11-28T20:53:35.849418+0000","last_fullsized":"2023-11-28T20:53:35.849418+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:09:08.233912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"132'4","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862313+0000","last_change":"2023-11-28T20:53:34.837653+0000","last_active":"2023-11-28T20:53:35.862313+0000","last_peered":"2023-11-28T20:53:35.862313+0000","last_clean":"2023-11-28T20:53:35.862313+0000","last_became_active":"2023-11-28T20:53:34.837411+0000","last_became_peered":"2023-11-28T20:53:34.837411+0000","last_unstale":"2023-11-28T20:53:35.862313+0000","last_undegraded":"2023-11-28T20:53:35.862313+0000","last_fullsized":"2023-11-28T20:53:35.862313+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:49:52.291969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841301+0000","last_change":"2023-11-28T20:53:35.838999+0000","last_active":"2023-11-28T20:53:35.841301+0000","last_peered":"2023-11-28T20:53:35.841301+0000","last_clean":"2023-11-28T20:53:35.841301+0000","last_became_active":"2023-11-28T20:53:35.838711+0000","last_became_peered":"2023-11-28T20:53:35.838711+0000","last_unstale":"2023-11-28T20:53:35.841301+0000","last_undegraded":"2023-11-28T20:53:35.841301+0000","last_fullsized":"2023-11-28T20:53:35.841301+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:13:53.513846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860303+0000","last_change":"2023-11-28T20:53:34.823960+0000","last_active":"2023-11-28T20:53:35.860303+0000","last_peered":"2023-11-28T20:53:35.860303+0000","last_clean":"2023-11-28T20:53:35.860303+0000","last_became_active":"2023-11-28T20:53:34.823693+0000","last_became_peered":"2023-11-28T20:53:34.823693+0000","last_unstale":"2023-11-28T20:53:35.860303+0000","last_undegraded":"2023-11-28T20:53:35.860303+0000","last_fullsized":"2023-11-28T20:53:35.860303+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:58:50.214016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.832615+0000","last_change":"2023-11-28T20:53:35.827998+0000","last_active":"2023-11-28T20:53:35.832615+0000","last_peered":"2023-11-28T20:53:35.832615+0000","last_clean":"2023-11-28T20:53:35.832615+0000","last_became_active":"2023-11-28T20:53:35.827682+0000","last_became_peered":"2023-11-28T20:53:35.827682+0000","last_unstale":"2023-11-28T20:53:35.832615+0000","last_undegraded":"2023-11-28T20:53:35.832615+0000","last_fullsized":"2023-11-28T20:53:35.832615+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:31.127376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.859024+0000","last_change":"2023-11-28T20:53:34.832425+0000","last_active":"2023-11-28T20:53:35.859024+0000","last_peered":"2023-11-28T20:53:35.859024+0000","last_clean":"2023-11-28T20:53:35.859024+0000","last_became_active":"2023-11-28T20:53:34.828892+0000","last_became_peered":"2023-11-28T20:53:34.828892+0000","last_unstale":"2023-11-28T20:53:35.859024+0000","last_undegraded":"2023-11-28T20:53:35.859024+0000","last_fullsized":"2023-11-28T20:53:35.859024+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:06:20.193558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835832+0000","last_change":"2023-11-28T20:53:35.833686+0000","last_active":"2023-11-28T20:53:35.835832+0000","last_peered":"2023-11-28T20:53:35.835832+0000","last_clean":"2023-11-28T20:53:35.835832+0000","last_became_active":"2023-11-28T20:53:35.833072+0000","last_became_peered":"2023-11-28T20:53:35.833072+0000","last_unstale":"2023-11-28T20:53:35.835832+0000","last_undegraded":"2023-11-28T20:53:35.835832+0000","last_fullsized":"2023-11-28T20:53:35.835832+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:57.757815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862691+0000","last_change":"2023-11-28T20:53:34.825479+0000","last_active":"2023-11-28T20:53:35.862691+0000","last_peered":"2023-11-28T20:53:35.862691+0000","last_clean":"2023-11-28T20:53:35.862691+0000","last_became_active":"2023-11-28T20:53:34.824886+0000","last_became_peered":"2023-11-28T20:53:34.824886+0000","last_unstale":"2023-11-28T20:53:35.862691+0000","last_undegraded":"2023-11-28T20:53:35.862691+0000","last_fullsized":"2023-11-28T20:53:35.862691+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:02.261990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.851887+0000","last_change":"2023-11-28T20:53:35.847560+0000","last_active":"2023-11-28T20:53:35.851887+0000","last_peered":"2023-11-28T20:53:35.851887+0000","last_clean":"2023-11-28T20:53:35.851887+0000","last_became_active":"2023-11-28T20:53:35.847249+0000","last_became_peered":"2023-11-28T20:53:35.847249+0000","last_unstale":"2023-11-28T20:53:35.851887+0000","last_undegraded":"2023-11-28T20:53:35.851887+0000","last_fullsized":"2023-11-28T20:53:35.851887+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:15:27.519053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.858048+0000","last_change":"2023-11-28T20:53:34.837342+0000","last_active":"2023-11-28T20:53:35.858048+0000","last_peered":"2023-11-28T20:53:35.858048+0000","last_clean":"2023-11-28T20:53:35.858048+0000","last_became_active":"2023-11-28T20:53:34.836196+0000","last_became_peered":"2023-11-28T20:53:34.836196+0000","last_unstale":"2023-11-28T20:53:35.858048+0000","last_undegraded":"2023-11-28T20:53:35.858048+0000","last_fullsized":"2023-11-28T20:53:35.858048+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:18.505183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850036+0000","last_change":"2023-11-28T20:53:35.846037+0000","last_active":"2023-11-28T20:53:35.850036+0000","last_peered":"2023-11-28T20:53:35.850036+0000","last_clean":"2023-11-28T20:53:35.850036+0000","last_became_active":"2023-11-28T20:53:35.836620+0000","last_became_peered":"2023-11-28T20:53:35.836620+0000","last_unstale":"2023-11-28T20:53:35.850036+0000","last_undegraded":"2023-11-28T20:53:35.850036+0000","last_fullsized":"2023-11-28T20:53:35.850036+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:25:37.115772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862380+0000","last_change":"2023-11-28T20:53:34.830070+0000","last_active":"2023-11-28T20:53:35.862380+0000","last_peered":"2023-11-28T20:53:35.862380+0000","last_clean":"2023-11-28T20:53:35.862380+0000","last_became_active":"2023-11-28T20:53:34.829736+0000","last_became_peered":"2023-11-28T20:53:34.829736+0000","last_unstale":"2023-11-28T20:53:35.862380+0000","last_undegraded":"2023-11-28T20:53:35.862380+0000","last_fullsized":"2023-11-28T20:53:35.862380+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:16.173574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.844307+0000","last_change":"2023-11-28T20:53:35.842530+0000","last_active":"2023-11-28T20:53:35.844307+0000","last_peered":"2023-11-28T20:53:35.844307+0000","last_clean":"2023-11-28T20:53:35.844307+0000","last_became_active":"2023-11-28T20:53:35.842226+0000","last_became_peered":"2023-11-28T20:53:35.842226+0000","last_unstale":"2023-11-28T20:53:35.844307+0000","last_undegraded":"2023-11-28T20:53:35.844307+0000","last_fullsized":"2023-11-28T20:53:35.844307+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:50:19.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.820426+0000","last_change":"2023-11-28T20:53:34.825641+0000","last_active":"2023-11-28T20:53:35.820426+0000","last_peered":"2023-11-28T20:53:35.820426+0000","last_clean":"2023-11-28T20:53:35.820426+0000","last_became_active":"2023-11-28T20:53:34.825237+0000","last_became_peered":"2023-11-28T20:53:34.825237+0000","last_unstale":"2023-11-28T20:53:35.820426+0000","last_undegraded":"2023-11-28T20:53:35.820426+0000","last_fullsized":"2023-11-28T20:53:35.820426+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:19:46.092231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.851903+0000","last_change":"2023-11-28T20:53:35.844312+0000","last_active":"2023-11-28T20:53:35.851903+0000","last_peered":"2023-11-28T20:53:35.851903+0000","last_clean":"2023-11-28T20:53:35.851903+0000","last_became_active":"2023-11-28T20:53:35.844009+0000","last_became_peered":"2023-11-28T20:53:35.844009+0000","last_unstale":"2023-11-28T20:53:35.851903+0000","last_undegraded":"2023-11-28T20:53:35.851903+0000","last_fullsized":"2023-11-28T20:53:35.851903+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:09.910426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.822185+0000","last_change":"2023-11-28T20:53:34.820152+0000","last_active":"2023-11-28T20:53:35.822185+0000","last_peered":"2023-11-28T20:53:35.822185+0000","last_clean":"2023-11-28T20:53:35.822185+0000","last_became_active":"2023-11-28T20:53:34.819839+0000","last_became_peered":"2023-11-28T20:53:34.819839+0000","last_unstale":"2023-11-28T20:53:35.822185+0000","last_undegraded":"2023-11-28T20:53:35.822185+0000","last_fullsized":"2023-11-28T20:53:35.822185+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:33.800878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846640+0000","last_change":"2023-11-28T20:53:35.839179+0000","last_active":"2023-11-28T20:53:35.846640+0000","last_peered":"2023-11-28T20:53:35.846640+0000","last_clean":"2023-11-28T20:53:35.846640+0000","last_became_active":"2023-11-28T20:53:35.836378+0000","last_became_peered":"2023-11-28T20:53:35.836378+0000","last_unstale":"2023-11-28T20:53:35.846640+0000","last_undegraded":"2023-11-28T20:53:35.846640+0000","last_fullsized":"2023-11-28T20:53:35.846640+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:22:57.476588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.822275+0000","last_change":"2023-11-28T20:53:34.830803+0000","last_active":"2023-11-28T20:53:35.822275+0000","last_peered":"2023-11-28T20:53:35.822275+0000","last_clean":"2023-11-28T20:53:35.822275+0000","last_became_active":"2023-11-28T20:53:34.826684+0000","last_became_peered":"2023-11-28T20:53:34.826684+0000","last_unstale":"2023-11-28T20:53:35.822275+0000","last_undegraded":"2023-11-28T20:53:35.822275+0000","last_fullsized":"2023-11-28T20:53:35.822275+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:35:15.873444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841352+0000","last_change":"2023-11-28T20:53:35.833993+0000","last_active":"2023-11-28T20:53:35.841352+0000","last_peered":"2023-11-28T20:53:35.841352+0000","last_clean":"2023-11-28T20:53:35.841352+0000","last_became_active":"2023-11-28T20:53:35.833671+0000","last_became_peered":"2023-11-28T20:53:35.833671+0000","last_unstale":"2023-11-28T20:53:35.841352+0000","last_undegraded":"2023-11-28T20:53:35.841352+0000","last_fullsized":"2023-11-28T20:53:35.841352+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:41:48.627407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.813830+0000","last_change":"2023-11-28T20:53:34.829792+0000","last_active":"2023-11-28T20:53:35.813830+0000","last_peered":"2023-11-28T20:53:35.813830+0000","last_clean":"2023-11-28T20:53:35.813830+0000","last_became_active":"2023-11-28T20:53:34.828811+0000","last_became_peered":"2023-11-28T20:53:34.828811+0000","last_unstale":"2023-11-28T20:53:35.813830+0000","last_undegraded":"2023-11-28T20:53:35.813830+0000","last_fullsized":"2023-11-28T20:53:35.813830+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:03:21.752817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.839268+0000","last_change":"2023-11-28T20:53:35.834360+0000","last_active":"2023-11-28T20:53:35.839268+0000","last_peered":"2023-11-28T20:53:35.839268+0000","last_clean":"2023-11-28T20:53:35.839268+0000","last_became_active":"2023-11-28T20:53:35.834090+0000","last_became_peered":"2023-11-28T20:53:35.834090+0000","last_unstale":"2023-11-28T20:53:35.839268+0000","last_undegraded":"2023-11-28T20:53:35.839268+0000","last_fullsized":"2023-11-28T20:53:35.839268+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:26.036147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"132'3","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860342+0000","last_change":"2023-11-28T20:53:34.825945+0000","last_active":"2023-11-28T20:53:35.860342+0000","last_peered":"2023-11-28T20:53:35.860342+0000","last_clean":"2023-11-28T20:53:35.860342+0000","last_became_active":"2023-11-28T20:53:34.824621+0000","last_became_peered":"2023-11-28T20:53:34.824621+0000","last_unstale":"2023-11-28T20:53:35.860342+0000","last_undegraded":"2023-11-28T20:53:35.860342+0000","last_fullsized":"2023-11-28T20:53:35.860342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:56:51.871002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.838369+0000","last_change":"2023-11-28T20:53:35.833020+0000","last_active":"2023-11-28T20:53:35.838369+0000","last_peered":"2023-11-28T20:53:35.838369+0000","last_clean":"2023-11-28T20:53:35.838369+0000","last_became_active":"2023-11-28T20:53:35.831333+0000","last_became_peered":"2023-11-28T20:53:35.831333+0000","last_unstale":"2023-11-28T20:53:35.838369+0000","last_undegraded":"2023-11-28T20:53:35.838369+0000","last_fullsized":"2023-11-28T20:53:35.838369+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:37:38.636899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"132'6","reported_seq":17,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862145+0000","last_change":"2023-11-28T20:53:34.829678+0000","last_active":"2023-11-28T20:53:35.862145+0000","last_peered":"2023-11-28T20:53:35.862145+0000","last_clean":"2023-11-28T20:53:35.862145+0000","last_became_active":"2023-11-28T20:53:34.829287+0000","last_became_peered":"2023-11-28T20:53:34.829287+0000","last_unstale":"2023-11-28T20:53:35.862145+0000","last_undegraded":"2023-11-28T20:53:35.862145+0000","last_fullsized":"2023-11-28T20:53:35.862145+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:19:29.281109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846717+0000","last_change":"2023-11-28T20:53:35.845380+0000","last_active":"2023-11-28T20:53:35.846717+0000","last_peered":"2023-11-28T20:53:35.846717+0000","last_clean":"2023-11-28T20:53:35.846717+0000","last_became_active":"2023-11-28T20:53:35.845079+0000","last_became_peered":"2023-11-28T20:53:35.845079+0000","last_unstale":"2023-11-28T20:53:35.846717+0000","last_undegraded":"2023-11-28T20:53:35.846717+0000","last_fullsized":"2023-11-28T20:53:35.846717+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:36:07.856406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860985+0000","last_change":"2023-11-28T20:53:34.814256+0000","last_active":"2023-11-28T20:53:35.860985+0000","last_peered":"2023-11-28T20:53:35.860985+0000","last_clean":"2023-11-28T20:53:35.860985+0000","last_became_active":"2023-11-28T20:53:34.813926+0000","last_became_peered":"2023-11-28T20:53:34.813926+0000","last_unstale":"2023-11-28T20:53:35.860985+0000","last_undegraded":"2023-11-28T20:53:35.860985+0000","last_fullsized":"2023-11-28T20:53:35.860985+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:30.296146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.844655+0000","last_change":"2023-11-28T20:53:35.842525+0000","last_active":"2023-11-28T20:53:35.844655+0000","last_peered":"2023-11-28T20:53:35.844655+0000","last_clean":"2023-11-28T20:53:35.844655+0000","last_became_active":"2023-11-28T20:53:35.842240+0000","last_became_peered":"2023-11-28T20:53:35.842240+0000","last_unstale":"2023-11-28T20:53:35.844655+0000","last_undegraded":"2023-11-28T20:53:35.844655+0000","last_fullsized":"2023-11-28T20:53:35.844655+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:22:52.747986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.828971+0000","last_change":"2023-11-28T20:53:34.823919+0000","last_active":"2023-11-28T20:53:35.828971+0000","last_peered":"2023-11-28T20:53:35.828971+0000","last_clean":"2023-11-28T20:53:35.828971+0000","last_became_active":"2023-11-28T20:53:34.823663+0000","last_became_peered":"2023-11-28T20:53:34.823663+0000","last_unstale":"2023-11-28T20:53:35.828971+0000","last_undegraded":"2023-11-28T20:53:35.828971+0000","last_fullsized":"2023-11-28T20:53:35.828971+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:00.263143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.839418+0000","last_change":"2023-11-28T20:53:35.835067+0000","last_active":"2023-11-28T20:53:35.839418+0000","last_peered":"2023-11-28T20:53:35.839418+0000","last_clean":"2023-11-28T20:53:35.839418+0000","last_became_active":"2023-11-28T20:53:35.833326+0000","last_became_peered":"2023-11-28T20:53:35.833326+0000","last_unstale":"2023-11-28T20:53:35.839418+0000","last_undegraded":"2023-11-28T20:53:35.839418+0000","last_fullsized":"2023-11-28T20:53:35.839418+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:17:30.238147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.820032+0000","last_change":"2023-11-28T20:53:34.829509+0000","last_active":"2023-11-28T20:53:35.820032+0000","last_peered":"2023-11-28T20:53:35.820032+0000","last_clean":"2023-11-28T20:53:35.820032+0000","last_became_active":"2023-11-28T20:53:34.829103+0000","last_became_peered":"2023-11-28T20:53:34.829103+0000","last_unstale":"2023-11-28T20:53:35.820032+0000","last_undegraded":"2023-11-28T20:53:35.820032+0000","last_fullsized":"2023-11-28T20:53:35.820032+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:51:36.369028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850110+0000","last_change":"2023-11-28T20:53:35.845392+0000","last_active":"2023-11-28T20:53:35.850110+0000","last_peered":"2023-11-28T20:53:35.850110+0000","last_clean":"2023-11-28T20:53:35.850110+0000","last_became_active":"2023-11-28T20:53:35.844870+0000","last_became_peered":"2023-11-28T20:53:35.844870+0000","last_unstale":"2023-11-28T20:53:35.850110+0000","last_undegraded":"2023-11-28T20:53:35.850110+0000","last_fullsized":"2023-11-28T20:53:35.850110+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:46:34.712897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"132'8","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862165+0000","last_change":"2023-11-28T20:53:34.828747+0000","last_active":"2023-11-28T20:53:35.862165+0000","last_peered":"2023-11-28T20:53:35.862165+0000","last_clean":"2023-11-28T20:53:35.862165+0000","last_became_active":"2023-11-28T20:53:34.828471+0000","last_became_peered":"2023-11-28T20:53:34.828471+0000","last_unstale":"2023-11-28T20:53:35.862165+0000","last_undegraded":"2023-11-28T20:53:35.862165+0000","last_fullsized":"2023-11-28T20:53:35.862165+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:44:26.538531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835822+0000","last_change":"2023-11-28T20:53:35.833822+0000","last_active":"2023-11-28T20:53:35.835822+0000","last_peered":"2023-11-28T20:53:35.835822+0000","last_clean":"2023-11-28T20:53:35.835822+0000","last_became_active":"2023-11-28T20:53:35.833508+0000","last_became_peered":"2023-11-28T20:53:35.833508+0000","last_unstale":"2023-11-28T20:53:35.835822+0000","last_undegraded":"2023-11-28T20:53:35.835822+0000","last_fullsized":"2023-11-28T20:53:35.835822+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:39.328623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.857183+0000","last_change":"2023-11-28T20:53:34.836319+0000","last_active":"2023-11-28T20:53:35.857183+0000","last_peered":"2023-11-28T20:53:35.857183+0000","last_clean":"2023-11-28T20:53:35.857183+0000","last_became_active":"2023-11-28T20:53:34.835521+0000","last_became_peered":"2023-11-28T20:53:34.835521+0000","last_unstale":"2023-11-28T20:53:35.857183+0000","last_undegraded":"2023-11-28T20:53:35.857183+0000","last_fullsized":"2023-11-28T20:53:35.857183+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:40.719594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846692+0000","last_change":"2023-11-28T20:53:35.844184+0000","last_active":"2023-11-28T20:53:35.846692+0000","last_peered":"2023-11-28T20:53:35.846692+0000","last_clean":"2023-11-28T20:53:35.846692+0000","last_became_active":"2023-11-28T20:53:35.843890+0000","last_became_peered":"2023-11-28T20:53:35.843890+0000","last_unstale":"2023-11-28T20:53:35.846692+0000","last_undegraded":"2023-11-28T20:53:35.846692+0000","last_fullsized":"2023-11-28T20:53:35.846692+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:36:44.724734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.858410+0000","last_change":"2023-11-28T20:53:34.827922+0000","last_active":"2023-11-28T20:53:35.858410+0000","last_peered":"2023-11-28T20:53:35.858410+0000","last_clean":"2023-11-28T20:53:35.858410+0000","last_became_active":"2023-11-28T20:53:34.827614+0000","last_became_peered":"2023-11-28T20:53:34.827614+0000","last_unstale":"2023-11-28T20:53:35.858410+0000","last_undegraded":"2023-11-28T20:53:35.858410+0000","last_fullsized":"2023-11-28T20:53:35.858410+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:53.267683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.834719+0000","last_change":"2023-11-28T20:53:35.827063+0000","last_active":"2023-11-28T20:53:35.834719+0000","last_peered":"2023-11-28T20:53:35.834719+0000","last_clean":"2023-11-28T20:53:35.834719+0000","last_became_active":"2023-11-28T20:53:35.825634+0000","last_became_peered":"2023-11-28T20:53:35.825634+0000","last_unstale":"2023-11-28T20:53:35.834719+0000","last_undegraded":"2023-11-28T20:53:35.834719+0000","last_fullsized":"2023-11-28T20:53:35.834719+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:27:32.599929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.811062+0000","last_change":"2023-11-28T20:53:34.834697+0000","last_active":"2023-11-28T20:53:35.811062+0000","last_peered":"2023-11-28T20:53:35.811062+0000","last_clean":"2023-11-28T20:53:35.811062+0000","last_became_active":"2023-11-28T20:53:34.834372+0000","last_became_peered":"2023-11-28T20:53:34.834372+0000","last_unstale":"2023-11-28T20:53:35.811062+0000","last_undegraded":"2023-11-28T20:53:35.811062+0000","last_fullsized":"2023-11-28T20:53:35.811062+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:37:56.208280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.852648+0000","last_change":"2023-11-28T20:53:35.850306+0000","last_active":"2023-11-28T20:53:35.852648+0000","last_peered":"2023-11-28T20:53:35.852648+0000","last_clean":"2023-11-28T20:53:35.852648+0000","last_became_active":"2023-11-28T20:53:35.849614+0000","last_became_peered":"2023-11-28T20:53:35.849614+0000","last_unstale":"2023-11-28T20:53:35.852648+0000","last_undegraded":"2023-11-28T20:53:35.852648+0000","last_fullsized":"2023-11-28T20:53:35.852648+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:05:41.061989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860348+0000","last_change":"2023-11-28T20:53:34.814296+0000","last_active":"2023-11-28T20:53:35.860348+0000","last_peered":"2023-11-28T20:53:35.860348+0000","last_clean":"2023-11-28T20:53:35.860348+0000","last_became_active":"2023-11-28T20:53:34.814030+0000","last_became_peered":"2023-11-28T20:53:34.814030+0000","last_unstale":"2023-11-28T20:53:35.860348+0000","last_undegraded":"2023-11-28T20:53:35.860348+0000","last_fullsized":"2023-11-28T20:53:35.860348+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:22:48.386297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.852304+0000","last_change":"2023-11-28T20:53:35.850046+0000","last_active":"2023-11-28T20:53:35.852304+0000","last_peered":"2023-11-28T20:53:35.852304+0000","last_clean":"2023-11-28T20:53:35.852304+0000","last_became_active":"2023-11-28T20:53:35.847288+0000","last_became_peered":"2023-11-28T20:53:35.847288+0000","last_unstale":"2023-11-28T20:53:35.852304+0000","last_undegraded":"2023-11-28T20:53:35.852304+0000","last_fullsized":"2023-11-28T20:53:35.852304+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:35:39.884963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.818932+0000","last_change":"2023-11-28T20:53:34.821897+0000","last_active":"2023-11-28T20:53:35.818932+0000","last_peered":"2023-11-28T20:53:35.818932+0000","last_clean":"2023-11-28T20:53:35.818932+0000","last_became_active":"2023-11-28T20:53:34.821618+0000","last_became_peered":"2023-11-28T20:53:34.821618+0000","last_unstale":"2023-11-28T20:53:35.818932+0000","last_undegraded":"2023-11-28T20:53:35.818932+0000","last_fullsized":"2023-11-28T20:53:35.818932+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:14:47.750327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.852013+0000","last_change":"2023-11-28T20:53:35.844896+0000","last_active":"2023-11-28T20:53:35.852013+0000","last_peered":"2023-11-28T20:53:35.852013+0000","last_clean":"2023-11-28T20:53:35.852013+0000","last_became_active":"2023-11-28T20:53:35.844642+0000","last_became_peered":"2023-11-28T20:53:35.844642+0000","last_unstale":"2023-11-28T20:53:35.852013+0000","last_undegraded":"2023-11-28T20:53:35.852013+0000","last_fullsized":"2023-11-28T20:53:35.852013+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:54.282703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.825270+0000","last_change":"2023-11-28T20:53:34.832781+0000","last_active":"2023-11-28T20:53:35.825270+0000","last_peered":"2023-11-28T20:53:35.825270+0000","last_clean":"2023-11-28T20:53:35.825270+0000","last_became_active":"2023-11-28T20:53:34.832108+0000","last_became_peered":"2023-11-28T20:53:34.832108+0000","last_unstale":"2023-11-28T20:53:35.825270+0000","last_undegraded":"2023-11-28T20:53:35.825270+0000","last_fullsized":"2023-11-28T20:53:35.825270+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:44:32.615441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.837205+0000","last_change":"2023-11-28T20:53:35.835702+0000","last_active":"2023-11-28T20:53:35.837205+0000","last_peered":"2023-11-28T20:53:35.837205+0000","last_clean":"2023-11-28T20:53:35.837205+0000","last_became_active":"2023-11-28T20:53:35.835420+0000","last_became_peered":"2023-11-28T20:53:35.835420+0000","last_unstale":"2023-11-28T20:53:35.837205+0000","last_undegraded":"2023-11-28T20:53:35.837205+0000","last_fullsized":"2023-11-28T20:53:35.837205+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:24:58.104641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.823267+0000","last_change":"2023-11-28T20:53:34.820508+0000","last_active":"2023-11-28T20:53:35.823267+0000","last_peered":"2023-11-28T20:53:35.823267+0000","last_clean":"2023-11-28T20:53:35.823267+0000","last_became_active":"2023-11-28T20:53:34.820182+0000","last_became_peered":"2023-11-28T20:53:34.820182+0000","last_unstale":"2023-11-28T20:53:35.823267+0000","last_undegraded":"2023-11-28T20:53:35.823267+0000","last_fullsized":"2023-11-28T20:53:35.823267+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:21:04.091589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.853533+0000","last_change":"2023-11-28T20:53:35.852061+0000","last_active":"2023-11-28T20:53:35.853533+0000","last_peered":"2023-11-28T20:53:35.853533+0000","last_clean":"2023-11-28T20:53:35.853533+0000","last_became_active":"2023-11-28T20:53:35.851798+0000","last_became_peered":"2023-11-28T20:53:35.851798+0000","last_unstale":"2023-11-28T20:53:35.853533+0000","last_undegraded":"2023-11-28T20:53:35.853533+0000","last_fullsized":"2023-11-28T20:53:35.853533+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:30:56.956173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860800+0000","last_change":"2023-11-28T20:53:34.815883+0000","last_active":"2023-11-28T20:53:35.860800+0000","last_peered":"2023-11-28T20:53:35.860800+0000","last_clean":"2023-11-28T20:53:35.860800+0000","last_became_active":"2023-11-28T20:53:34.814723+0000","last_became_peered":"2023-11-28T20:53:34.814723+0000","last_unstale":"2023-11-28T20:53:35.860800+0000","last_undegraded":"2023-11-28T20:53:35.860800+0000","last_fullsized":"2023-11-28T20:53:35.860800+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:06.204609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841631+0000","last_change":"2023-11-28T20:53:35.839542+0000","last_active":"2023-11-28T20:53:35.841631+0000","last_peered":"2023-11-28T20:53:35.841631+0000","last_clean":"2023-11-28T20:53:35.841631+0000","last_became_active":"2023-11-28T20:53:35.839237+0000","last_became_peered":"2023-11-28T20:53:35.839237+0000","last_unstale":"2023-11-28T20:53:35.841631+0000","last_undegraded":"2023-11-28T20:53:35.841631+0000","last_fullsized":"2023-11-28T20:53:35.841631+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:07.219629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.814312+0000","last_change":"2023-11-28T20:53:34.826741+0000","last_active":"2023-11-28T20:53:35.814312+0000","last_peered":"2023-11-28T20:53:35.814312+0000","last_clean":"2023-11-28T20:53:35.814312+0000","last_became_active":"2023-11-28T20:53:34.826411+0000","last_became_peered":"2023-11-28T20:53:34.826411+0000","last_unstale":"2023-11-28T20:53:35.814312+0000","last_undegraded":"2023-11-28T20:53:35.814312+0000","last_fullsized":"2023-11-28T20:53:35.814312+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:44:36.874589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.843301+0000","last_change":"2023-11-28T20:53:35.835914+0000","last_active":"2023-11-28T20:53:35.843301+0000","last_peered":"2023-11-28T20:53:35.843301+0000","last_clean":"2023-11-28T20:53:35.843301+0000","last_became_active":"2023-11-28T20:53:35.835579+0000","last_became_peered":"2023-11-28T20:53:35.835579+0000","last_unstale":"2023-11-28T20:53:35.843301+0000","last_undegraded":"2023-11-28T20:53:35.843301+0000","last_fullsized":"2023-11-28T20:53:35.843301+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:08.301254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.825271+0000","last_change":"2023-11-28T20:53:34.835803+0000","last_active":"2023-11-28T20:53:35.825271+0000","last_peered":"2023-11-28T20:53:35.825271+0000","last_clean":"2023-11-28T20:53:35.825271+0000","last_became_active":"2023-11-28T20:53:34.835317+0000","last_became_peered":"2023-11-28T20:53:34.835317+0000","last_unstale":"2023-11-28T20:53:35.825271+0000","last_undegraded":"2023-11-28T20:53:35.825271+0000","last_fullsized":"2023-11-28T20:53:35.825271+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:53:49.750751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835641+0000","last_change":"2023-11-28T20:53:35.832186+0000","last_active":"2023-11-28T20:53:35.835641+0000","last_peered":"2023-11-28T20:53:35.835641+0000","last_clean":"2023-11-28T20:53:35.835641+0000","last_became_active":"2023-11-28T20:53:35.831565+0000","last_became_peered":"2023-11-28T20:53:35.831565+0000","last_unstale":"2023-11-28T20:53:35.835641+0000","last_undegraded":"2023-11-28T20:53:35.835641+0000","last_fullsized":"2023-11-28T20:53:35.835641+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:13:09.488951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.805018+0000","last_change":"2023-11-28T20:53:34.834367+0000","last_active":"2023-11-28T20:53:35.805018+0000","last_peered":"2023-11-28T20:53:35.805018+0000","last_clean":"2023-11-28T20:53:35.805018+0000","last_became_active":"2023-11-28T20:53:34.834019+0000","last_became_peered":"2023-11-28T20:53:34.834019+0000","last_unstale":"2023-11-28T20:53:35.805018+0000","last_undegraded":"2023-11-28T20:53:35.805018+0000","last_fullsized":"2023-11-28T20:53:35.805018+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:40.719594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.849499+0000","last_change":"2023-11-28T20:53:35.842065+0000","last_active":"2023-11-28T20:53:35.849499+0000","last_peered":"2023-11-28T20:53:35.849499+0000","last_clean":"2023-11-28T20:53:35.849499+0000","last_became_active":"2023-11-28T20:53:35.841773+0000","last_became_peered":"2023-11-28T20:53:35.841773+0000","last_unstale":"2023-11-28T20:53:35.849499+0000","last_undegraded":"2023-11-28T20:53:35.849499+0000","last_fullsized":"2023-11-28T20:53:35.849499+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:38:11.060050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.821139+0000","last_change":"2023-11-28T20:53:34.836093+0000","last_active":"2023-11-28T20:53:35.821139+0000","last_peered":"2023-11-28T20:53:35.821139+0000","last_clean":"2023-11-28T20:53:35.821139+0000","last_became_active":"2023-11-28T20:53:34.835797+0000","last_became_peered":"2023-11-28T20:53:34.835797+0000","last_unstale":"2023-11-28T20:53:35.821139+0000","last_undegraded":"2023-11-28T20:53:35.821139+0000","last_fullsized":"2023-11-28T20:53:35.821139+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:58:38.485097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841338+0000","last_change":"2023-11-28T20:53:35.837810+0000","last_active":"2023-11-28T20:53:35.841338+0000","last_peered":"2023-11-28T20:53:35.841338+0000","last_clean":"2023-11-28T20:53:35.841338+0000","last_became_active":"2023-11-28T20:53:35.837527+0000","last_became_peered":"2023-11-28T20:53:35.837527+0000","last_unstale":"2023-11-28T20:53:35.841338+0000","last_undegraded":"2023-11-28T20:53:35.841338+0000","last_fullsized":"2023-11-28T20:53:35.841338+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:00:52.844681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.825134+0000","last_change":"2023-11-28T20:53:34.829508+0000","last_active":"2023-11-28T20:53:35.825134+0000","last_peered":"2023-11-28T20:53:35.825134+0000","last_clean":"2023-11-28T20:53:35.825134+0000","last_became_active":"2023-11-28T20:53:34.829045+0000","last_became_peered":"2023-11-28T20:53:34.829045+0000","last_unstale":"2023-11-28T20:53:35.825134+0000","last_undegraded":"2023-11-28T20:53:35.825134+0000","last_fullsized":"2023-11-28T20:53:35.825134+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:00:02.431917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.838672+0000","last_change":"2023-11-28T20:53:35.837327+0000","last_active":"2023-11-28T20:53:35.838672+0000","last_peered":"2023-11-28T20:53:35.838672+0000","last_clean":"2023-11-28T20:53:35.838672+0000","last_became_active":"2023-11-28T20:53:35.837067+0000","last_became_peered":"2023-11-28T20:53:35.837067+0000","last_unstale":"2023-11-28T20:53:35.838672+0000","last_undegraded":"2023-11-28T20:53:35.838672+0000","last_fullsized":"2023-11-28T20:53:35.838672+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:09.910426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.825543+0000","last_change":"2023-11-28T20:53:34.833368+0000","last_active":"2023-11-28T20:53:35.825543+0000","last_peered":"2023-11-28T20:53:35.825543+0000","last_clean":"2023-11-28T20:53:35.825543+0000","last_became_active":"2023-11-28T20:53:34.832510+0000","last_became_peered":"2023-11-28T20:53:34.832510+0000","last_unstale":"2023-11-28T20:53:35.825543+0000","last_undegraded":"2023-11-28T20:53:35.825543+0000","last_fullsized":"2023-11-28T20:53:35.825543+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:02.261990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835235+0000","last_change":"2023-11-28T20:53:35.830174+0000","last_active":"2023-11-28T20:53:35.835235+0000","last_peered":"2023-11-28T20:53:35.835235+0000","last_clean":"2023-11-28T20:53:35.835235+0000","last_became_active":"2023-11-28T20:53:35.828266+0000","last_became_peered":"2023-11-28T20:53:35.828266+0000","last_unstale":"2023-11-28T20:53:35.835235+0000","last_undegraded":"2023-11-28T20:53:35.835235+0000","last_fullsized":"2023-11-28T20:53:35.835235+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:07:30.282009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.824757+0000","last_change":"2023-11-28T20:53:34.824657+0000","last_active":"2023-11-28T20:53:35.824757+0000","last_peered":"2023-11-28T20:53:35.824757+0000","last_clean":"2023-11-28T20:53:35.824757+0000","last_became_active":"2023-11-28T20:53:34.824365+0000","last_became_peered":"2023-11-28T20:53:34.824365+0000","last_unstale":"2023-11-28T20:53:35.824757+0000","last_undegraded":"2023-11-28T20:53:35.824757+0000","last_fullsized":"2023-11-28T20:53:35.824757+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:44.144409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846632+0000","last_change":"2023-11-28T20:53:35.844727+0000","last_active":"2023-11-28T20:53:35.846632+0000","last_peered":"2023-11-28T20:53:35.846632+0000","last_clean":"2023-11-28T20:53:35.846632+0000","last_became_active":"2023-11-28T20:53:35.844394+0000","last_became_peered":"2023-11-28T20:53:35.844394+0000","last_unstale":"2023-11-28T20:53:35.846632+0000","last_undegraded":"2023-11-28T20:53:35.846632+0000","last_fullsized":"2023-11-28T20:53:35.846632+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:32:43.243619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860428+0000","last_change":"2023-11-28T20:53:34.822269+0000","last_active":"2023-11-28T20:53:35.860428+0000","last_peered":"2023-11-28T20:53:35.860428+0000","last_clean":"2023-11-28T20:53:35.860428+0000","last_became_active":"2023-11-28T20:53:34.821985+0000","last_became_peered":"2023-11-28T20:53:34.821985+0000","last_unstale":"2023-11-28T20:53:35.860428+0000","last_undegraded":"2023-11-28T20:53:35.860428+0000","last_fullsized":"2023-11-28T20:53:35.860428+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:54.528095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"121'4712","reported_seq":5157,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-28T20:53:34.821427+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:53:34.821427+0000","last_peered":"2023-11-28T20:53:34.821427+0000","last_clean":"2023-11-28T20:53:34.821427+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:53:34.821427+0000","last_undegraded":"2023-11-28T20:53:34.821427+0000","last_fullsized":"2023-11-28T20:53:34.821427+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":997,"num_read_kb":1979,"num_write":331,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":997,"num_read_kb":1979,"num_write":331,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4712,"ondisk_log_size":4712,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47248,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324592,"statfs":{"total":96636764160,"available":96588382208,"internally_reserved":0,"allocated":2236416,"data_stored":1417297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1519999999999999,"15min":1.1839999999999999},"min":{"1min":0.75700000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.6910000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.04},{"interface":"front","average":{"1min":1.056,"5min":1.0980000000000001,"15min":1.1599999999999999},"min":{"1min":0.72999999999999998,"5min":0.64300000000000002,"15min":0.58299999999999996},"max":{"1min":1.502,"5min":1.8,"15min":1.8},"last":1.431}]},{"osd":1,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.1659999999999999,"15min":1.1830000000000001},"min":{"1min":0.74299999999999999,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5720000000000001,"5min":1.633,"15min":1.744},"last":1.228},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2609999999999999,"15min":1.2589999999999999},"min":{"1min":0.81699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.752,"5min":1.8540000000000001,"15min":1.889},"last":0.82099999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.2809999999999999,"15min":1.2929999999999999},"min":{"1min":0.93899999999999995,"5min":0.59699999999999998,"15min":0.58799999999999997},"max":{"1min":1.667,"5min":1.736,"15min":1.9359999999999999},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3120000000000001,"15min":1.296},"min":{"1min":1.0469999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.571,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.3069999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.254,"15min":1.272},"min":{"1min":0.95899999999999996,"5min":0.73199999999999998,"15min":0.54200000000000004},"max":{"1min":1.8859999999999999,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.4099999999999999,"15min":1.409},"min":{"1min":1.0469999999999999,"5min":0.77000000000000002,"15min":0.68300000000000005},"max":{"1min":1.9550000000000001,"5min":1.956,"15min":1.956},"last":1.1719999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.296,"15min":1.2769999999999999},"min":{"1min":0.94799999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.849,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.325,"15min":1.321},"min":{"1min":0.93300000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.99,"5min":1.99,"15min":1.9970000000000001},"last":1.7190000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.3999999999999999,"15min":1.403},"min":{"1min":0.77000000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8759999999999999,"5min":1.891,"15min":2.048},"last":1.5},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4610000000000001,"15min":1.462},"min":{"1min":0.99099999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.71,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.651}]},{"osd":6,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.5700000000000001,"15min":1.554},"min":{"1min":1.25,"5min":0.85799999999999998,"15min":0.69499999999999995},"max":{"1min":1.9359999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.766},{"interface":"front","average":{"1min":1.48,"5min":1.4810000000000001,"15min":1.48},"min":{"1min":0.94099999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.7969999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575082,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334880,"statfs":{"total":96636764160,"available":96598917120,"internally_reserved":0,"allocated":1654784,"data_stored":828985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0469999999999999,"15min":1.095},"min":{"1min":0.745,"5min":0.58099999999999996,"15min":0.53500000000000003},"max":{"1min":1.5900000000000001,"5min":1.6319999999999999,"15min":1.7350000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.091,"15min":1.1299999999999999},"min":{"1min":0.69699999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4059999999999999,"5min":1.8069999999999999,"15min":1.8260000000000001},"last":0.73899999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.196,"15min":1.24},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.651,"5min":1.728,"15min":1.8069999999999999},"last":1.595},{"interface":"front","average":{"1min":1.385,"5min":1.292,"15min":1.349},"min":{"1min":1.1100000000000001,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.6759999999999999,"5min":1.7889999999999999,"15min":2.0169999999999999},"last":1.1359999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.2470000000000001,"15min":1.2829999999999999},"min":{"1min":0.88300000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":1.55},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.3200000000000001,"15min":1.361},"min":{"1min":1.2190000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.837,"5min":1.851,"15min":1.851},"last":1.3440000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.264,"15min":1.28},"min":{"1min":0.75,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.9299999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.222,"5min":1.258,"15min":1.2789999999999999},"min":{"1min":1.002,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.7450000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.1519999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.266,"15min":1.246},"min":{"1min":0.78300000000000003,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.6479999999999999,"5min":1.7529999999999999,"15min":2.089},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.3089999999999999,"15min":1.3049999999999999},"min":{"1min":0.92400000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.798,"5min":1.9179999999999999,"15min":2.2010000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.351,"15min":1.337},"min":{"1min":0.95999999999999996,"5min":0.88800000000000001,"15min":0.754},"max":{"1min":1.879,"5min":1.879,"15min":1.97},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.514,"15min":1.528},"min":{"1min":1.0669999999999999,"5min":0.93400000000000005,"15min":0.78000000000000003},"max":{"1min":1.901,"5min":1.9179999999999999,"15min":1.958},"last":1.6719999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4399999999999999,"15min":1.46},"min":{"1min":0.94999999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.849,"5min":1.9790000000000001,"15min":2.3759999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.4550000000000001,"15min":1.4530000000000001},"min":{"1min":1.1599999999999999,"5min":0.96099999999999997,"15min":0.92900000000000005},"max":{"1min":1.9259999999999999,"5min":1.9630000000000001,"15min":2.0739999999999998},"last":1.403}]}]},{"osd":5,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50824,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321016,"statfs":{"total":96636764160,"available":96584720384,"internally_reserved":0,"allocated":1634304,"data_stored":826769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1519999999999999,"15min":1.1639999999999999},"min":{"1min":0.77600000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.4860000000000002,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0660000000000001,"15min":1.159},"min":{"1min":0.60299999999999998,"5min":0.56200000000000006,"15min":0.441},"max":{"1min":1.587,"5min":1.631,"15min":1.829},"last":1.1140000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1160000000000001,"15min":1.1539999999999999},"min":{"1min":0.63100000000000001,"5min":0.61499999999999999,"15min":0.57399999999999995},"max":{"1min":1.619,"5min":1.768,"15min":1.768},"last":1.171},{"interface":"front","average":{"1min":1.323,"5min":1.2090000000000001,"15min":1.2170000000000001},"min":{"1min":0.82199999999999995,"5min":0.69299999999999995,"15min":0.55900000000000005},"max":{"1min":2.282,"5min":2.282,"15min":2.282},"last":1.6100000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.2689999999999999,"15min":1.2749999999999999},"min":{"1min":0.90500000000000003,"5min":0.63600000000000001,"15min":0.53200000000000003},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.26},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2110000000000001,"15min":1.278},"min":{"1min":0.85599999999999998,"5min":0.71799999999999997,"15min":0.66600000000000004},"max":{"1min":1.641,"5min":1.8220000000000001,"15min":1.958},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.28,"15min":1.3140000000000001},"min":{"1min":0.93400000000000005,"5min":0.77700000000000002,"15min":0.71099999999999997},"max":{"1min":1.6839999999999999,"5min":1.7290000000000001,"15min":1.9350000000000001},"last":1.216},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2609999999999999,"15min":1.268},"min":{"1min":0.91800000000000004,"5min":0.68300000000000005,"15min":0.64000000000000001},"max":{"1min":1.5920000000000001,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.397}]},{"osd":4,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3069999999999999,"15min":1.302},"min":{"1min":0.755,"5min":0.73799999999999999,"15min":0.67500000000000004},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.488,"5min":1.47,"15min":1.4399999999999999},"min":{"1min":1.0669999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.9249999999999998,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":1.3620000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.3280000000000001,"15min":1.329},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.5},"max":{"1min":1.6000000000000001,"5min":1.9630000000000001,"15min":2.093},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.474,"15min":1.4750000000000001},"min":{"1min":1.054,"5min":0.84999999999999998,"15min":0.69499999999999995},"max":{"1min":1.6819999999999999,"5min":1.998,"15min":1.998},"last":1.639}]},{"osd":7,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.4690000000000001,"15min":1.4350000000000001},"min":{"1min":1.0609999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":3.319,"5min":3.319,"15min":3.319},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.476,"15min":1.4910000000000001},"min":{"1min":0.99299999999999999,"5min":0.98099999999999998,"15min":0.60499999999999998},"max":{"1min":1.782,"5min":1.9790000000000001,"15min":2.1829999999999998},"last":1.3260000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36948,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334892,"statfs":{"total":96636764160,"available":96598929408,"internally_reserved":0,"allocated":1642496,"data_stored":826861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0620000000000001,"15min":1.1220000000000001},"min":{"1min":0.69199999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.409,"5min":1.605,"15min":1.879},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.095,"15min":1.117},"min":{"1min":0.86299999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.5449999999999999,"5min":1.6850000000000001,"15min":1.9750000000000001},"last":1.016}]},{"osd":1,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.228,"15min":1.2969999999999999},"min":{"1min":0.79600000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.923},"last":1.044},{"interface":"front","average":{"1min":1.327,"5min":1.2849999999999999,"15min":1.3420000000000001},"min":{"1min":0.91200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7070000000000001,"5min":1.845,"15min":2.2679999999999998},"last":0.95699999999999996}]},{"osd":2,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.196,"15min":1.21},"min":{"1min":0.90000000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.7250000000000001,"5min":1.78,"15min":1.78},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2549999999999999,"15min":1.2869999999999999},"min":{"1min":0.85499999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.623,"5min":1.837,"15min":2.6949999999999998},"last":1.163}]},{"osd":3,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.236,"15min":1.2789999999999999},"min":{"1min":1.0269999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.6519999999999999,"5min":1.9630000000000001,"15min":2.157},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3380000000000001,"15min":1.361},"min":{"1min":1.0249999999999999,"5min":0.5,"15min":0.5},"max":{"1min":1.673,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.4370000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.284,"15min":1.2849999999999999},"min":{"1min":0.79900000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.4279999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.288,"15min":1.292},"min":{"1min":0.84299999999999997,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.655,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.097}]},{"osd":6,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.393,"15min":1.3859999999999999},"min":{"1min":0.90000000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.8400000000000001,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.347},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.53,"15min":1.55},"min":{"1min":1.0980000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.8899999999999999,"5min":2.1349999999999998,"15min":3.7000000000000002},"last":1.478}]},{"osd":7,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4390000000000001,"15min":1.4510000000000001},"min":{"1min":1.056,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.9430000000000001,"5min":2.2490000000000001,"15min":2.4990000000000001},"last":1.2},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.518,"15min":1.512},"min":{"1min":1.115,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.0430000000000001,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.2190000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575083,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41096,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330744,"statfs":{"total":96636764160,"available":96594681856,"internally_reserved":0,"allocated":1638400,"data_stored":827351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.85299999999999998,"15min":0.90400000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.3300000000000001,"5min":1.6000000000000001,"15min":4.7210000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.02,"5min":0.93899999999999995,"15min":1.0469999999999999},"min":{"1min":0.59299999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.5740000000000001,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.92700000000000005,"15min":0.96999999999999997},"min":{"1min":0.501,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.351,"5min":1.9770000000000001,"15min":2.8460000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.256,"5min":1.0900000000000001,"15min":1.1859999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.7170000000000001,"5min":1.7190000000000001,"15min":3.7410000000000001},"last":0.79500000000000004}]},{"osd":2,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.1399999999999999,"15min":1.2290000000000001},"min":{"1min":0.54300000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":3.444},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.236,"15min":1.3300000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":3.7050000000000001},"last":0.626}]},{"osd":4,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.2869999999999999,"15min":1.3420000000000001},"min":{"1min":1.0209999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.956,"5min":1.956,"15min":2.9420000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.3440000000000001,"15min":1.377},"min":{"1min":0.80300000000000005,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":1.014}]},{"osd":5,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.3859999999999999,"15min":1.4450000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":3.7730000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.522,"5min":1.383,"15min":1.4339999999999999},"min":{"1min":0.69699999999999995,"5min":0.66800000000000004,"15min":0.53500000000000003},"max":{"1min":1.825,"5min":2.0110000000000001,"15min":3.1520000000000001},"last":0.877}]},{"osd":6,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.407,"15min":1.4670000000000001},"min":{"1min":0.77200000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.3100000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.585,"5min":1.47,"15min":1.5680000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.016,"5min":2.0840000000000001,"15min":3.661},"last":0.97599999999999998}]},{"osd":7,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.425,"15min":1.5309999999999999},"min":{"1min":0.753,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.847,"5min":1.923,"15min":3.266},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.5269999999999999,"15min":1.587},"min":{"1min":1.046,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.145,"5min":2.2080000000000002,"15min":2.3919999999999999},"last":1.0700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575083,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59144,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312696,"statfs":{"total":96636764160,"available":96576200704,"internally_reserved":0,"allocated":1638400,"data_stored":826803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.86699999999999999,"15min":0.90200000000000002},"min":{"1min":0.61699999999999999,"5min":0.373,"15min":0.28399999999999997},"max":{"1min":1.5569999999999999,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.024,"15min":1.0580000000000001},"min":{"1min":0.64500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.7270000000000001,"5min":4.8490000000000002,"15min":4.8490000000000002},"last":0.60699999999999998}]},{"osd":1,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.004,"15min":0.998},"min":{"1min":0.68700000000000006,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.7529999999999999,"5min":1.8160000000000001,"15min":3.2599999999999998},"last":0.498},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0609999999999999,"15min":1.137},"min":{"1min":0.72499999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5529999999999999,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.39000000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.129,"15min":1.167},"min":{"1min":0.60899999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.665,"5min":4.9349999999999996,"15min":4.9349999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.252,"15min":1.29},"min":{"1min":0.75800000000000001,"5min":0.48499999999999999,"15min":0.48199999999999998},"max":{"1min":1.746,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":0.70899999999999996}]},{"osd":4,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4019999999999999,"15min":1.4159999999999999},"min":{"1min":1.03,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.784,"5min":3.6499999999999999,"15min":3.6499999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.468,"15min":1.4399999999999999},"min":{"1min":1.1519999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8360000000000001,"5min":4.6479999999999997,"15min":4.6479999999999997},"last":0.998}]},{"osd":5,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.3999999999999999,"15min":1.423},"min":{"1min":1.137,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.8160000000000001,"5min":2.492,"15min":2.492},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.49,"15min":1.5009999999999999},"min":{"1min":1.1950000000000001,"5min":0.92400000000000004,"15min":0.89100000000000001},"max":{"1min":1.8740000000000001,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":0.76800000000000002}]},{"osd":6,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.504,"15min":1.5149999999999999},"min":{"1min":1.127,"5min":0.747,"15min":0.747},"max":{"1min":1.8720000000000001,"5min":4.5270000000000001,"15min":4.5270000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":1.1080000000000001,"5min":0.89000000000000001,"15min":0.82499999999999996},"max":{"1min":1.972,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.083}]},{"osd":7,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.494,"15min":1.532},"min":{"1min":0.99399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.1160000000000001,"5min":2.641,"15min":2.641},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5589999999999999,"15min":1.593},"min":{"1min":1.1659999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.9410000000000001,"5min":4.7530000000000001,"15min":4.7530000000000001},"last":0.81799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36940,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334900,"statfs":{"total":96636764160,"available":96598937600,"internally_reserved":0,"allocated":1638400,"data_stored":826803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.92400000000000004,"15min":0.91700000000000004},"min":{"1min":0.71699999999999997,"5min":0.311,"15min":0.311},"max":{"1min":1.742,"5min":1.8009999999999999,"15min":1.9259999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.014,"15min":1.083},"min":{"1min":0.79400000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.5669999999999999,"5min":2.028,"15min":2.028},"last":0.96099999999999997}]},{"osd":2,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.083,"15min":1.095},"min":{"1min":0.71399999999999997,"5min":0.51100000000000001,"15min":0.371},"max":{"1min":1.766,"5min":1.766,"15min":2.234},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.244,"5min":1.198,"15min":1.2090000000000001},"min":{"1min":0.56000000000000005,"5min":0.53700000000000003,"15min":0.52800000000000002},"max":{"1min":1.7629999999999999,"5min":1.9199999999999999,"15min":2.895},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.26,"15min":1.3069999999999999},"min":{"1min":0.75600000000000001,"5min":0.58199999999999996,"15min":0.46700000000000003},"max":{"1min":1.7450000000000001,"5min":1.8779999999999999,"15min":2.0459999999999998},"last":0.752},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.0820000000000001,"15min":1.1160000000000001},"min":{"1min":0.86299999999999999,"5min":0.45100000000000001,"15min":0.37},"max":{"1min":1.585,"5min":1.9790000000000001,"15min":2.3679999999999999},"last":0.77600000000000002}]},{"osd":4,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.3160000000000001,"15min":1.349},"min":{"1min":0.996,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":2.1150000000000002},"last":1.131},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.409,"15min":1.403},"min":{"1min":1.081,"5min":0.72399999999999998,"15min":0.71899999999999997},"max":{"1min":1.774,"5min":1.8819999999999999,"15min":2.6819999999999999},"last":1.0640000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.444,"15min":1.4410000000000001},"min":{"1min":0.86599999999999999,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.8839999999999999,"5min":2.0190000000000001,"15min":2.1840000000000002},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.474,"15min":1.494},"min":{"1min":1.1120000000000001,"5min":0.89700000000000002,"15min":0.89600000000000002},"max":{"1min":1.885,"5min":2.0219999999999998,"15min":2.294},"last":1.2889999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.393,"15min":1.4259999999999999},"min":{"1min":1.028,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.607},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.522,"15min":1.532},"min":{"1min":0.98599999999999999,"5min":0.81200000000000006,"15min":0.77100000000000002},"max":{"1min":2.0049999999999999,"5min":2.1059999999999999,"15min":2.3300000000000001},"last":1.4670000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.534,"15min":1.5649999999999999},"min":{"1min":1.198,"5min":0.80500000000000005,"15min":0.753},"max":{"1min":1.9750000000000001,"5min":2.1429999999999998,"15min":2.3359999999999999},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.4770000000000001,"15min":1.5229999999999999},"min":{"1min":1.1459999999999999,"5min":0.80500000000000005,"15min":0.72799999999999998},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.294},"last":1.6859999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41676,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330164,"statfs":{"total":96636764160,"available":96594087936,"internally_reserved":0,"allocated":2228224,"data_stored":1417137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.85599999999999998,"15min":0.89200000000000002},"min":{"1min":0.32800000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.208,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.0389999999999999,"15min":1.0900000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.57399999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.004,"15min":1.101},"min":{"1min":0.41699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.504},{"interface":"front","average":{"1min":1.111,"5min":1.1060000000000001,"15min":1.1819999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.782,"5min":1.893,"15min":1.9159999999999999},"last":0.64700000000000002}]},{"osd":3,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.038,"15min":1.111},"min":{"1min":0.60599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.6390000000000002,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.347,"5min":1.1950000000000001,"15min":1.304},"min":{"1min":0.66000000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.55300000000000005}]},{"osd":4,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.343,"15min":1.4199999999999999},"min":{"1min":0.82299999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.712,"5min":1.804,"15min":2.012},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.337,"15min":1.387},"min":{"1min":0.95399999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":4.5119999999999996,"5min":4.5119999999999996,"15min":4.5119999999999996},"last":0.879}]},{"osd":5,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.389,"15min":1.419},"min":{"1min":0.89900000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.9379999999999997,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.4279999999999999,"15min":1.4830000000000001},"min":{"1min":0.97299999999999998,"5min":0.621,"15min":0.621},"max":{"1min":2.6139999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.0720000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.482,"15min":1.546},"min":{"1min":1.02,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.466,"15min":1.522},"min":{"1min":0.94999999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.0609999999999999,"5min":2.274,"15min":2.4209999999999998},"last":0.95399999999999996}]},{"osd":7,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.544,"15min":1.593},"min":{"1min":0.873,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1749999999999998,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.48,"15min":1.5429999999999999},"min":{"1min":1.0640000000000001,"5min":0.80300000000000005,"15min":0.73799999999999999},"max":{"1min":4.7359999999999998,"5min":4.7359999999999998,"15min":4.7359999999999998},"last":1.2909999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:53:40.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:53:40.411 INFO:tasks.ceph.mds.a.smithi045.stderr:2023-11-28T20:53:40.410+0000 7f4758e3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:53:40.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.600+0000 7fd90f2d1640 1 Processor -- start 2023-11-28T20:53:40.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.601+0000 7fd90f2d1640 1 -- start start 2023-11-28T20:53:40.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.602+0000 7fd90f2d1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd9080ce1e0 0x7fd90815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.602+0000 7fd90f2d1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9080d1250 con 0x7fd9080cfaf0 2023-11-28T20:53:40.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.602+0000 7fd90f2d1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9080ce6b0 con 0x7fd9080cfed0 2023-11-28T20:53:40.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.602+0000 7fd90f2d1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9080b6c70 con 0x7fd9080ce1e0 2023-11-28T20:53:40.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.602+0000 7fd90e2cf640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd9080cfaf0 0x7fd9080b6560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54998/0 (socket says 172.21.15.45:54998) 2023-11-28T20:53:40.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.602+0000 7fd90e2cf640 1 -- 172.21.15.45:0/3580021285 learned_addr learned my addr 172.21.15.45:0/3580021285 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:40.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.603+0000 7fd90ead0640 1 --2- 172.21.15.45:0/3580021285 >> v2:172.21.15.45:3300/0 conn(0x7fd9080ce1e0 0x7fd90815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.603+0000 7fd90d2cd640 1 -- 172.21.15.45:0/3580021285 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829656490 0 0) 0x7fd9080d1250 con 0x7fd9080cfaf0 2023-11-28T20:53:40.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.603+0000 7fd90d2cd640 1 -- 172.21.15.45:0/3580021285 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f4003520 con 0x7fd9080cfaf0 2023-11-28T20:53:40.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.603+0000 7fd90ead0640 1 -- 172.21.15.45:0/3580021285 >> v1:172.21.15.190:6789/0 conn(0x7fd9080cfed0 legacy=0x7fd9080cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.603+0000 7fd90ead0640 1 -- 172.21.15.45:0/3580021285 >> v1:172.21.15.45:6789/0 conn(0x7fd9080cfaf0 legacy=0x7fd9080b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.603+0000 7fd90ead0640 1 -- 172.21.15.45:0/3580021285 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080ce990 con 0x7fd9080ce1e0 2023-11-28T20:53:40.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.605+0000 7fd90ead0640 1 --2- 172.21.15.45:0/3580021285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080ce1e0 0x7fd90815cf60 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fd900009d90 tx=0x7fd90002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d0c1cd15853c85 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:40.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.605+0000 7fd90d2cd640 1 -- 172.21.15.45:0/3580021285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90002cc60 con 0x7fd9080ce1e0 2023-11-28T20:53:40.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.605+0000 7fd90d2cd640 1 -- 172.21.15.45:0/3580021285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90002cdc0 con 0x7fd9080ce1e0 2023-11-28T20:53:40.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.605+0000 7fd90d2cd640 1 -- 172.21.15.45:0/3580021285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900034660 con 0x7fd9080ce1e0 2023-11-28T20:53:40.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.606+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3580021285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080ce1e0 msgr2=0x7fd90815cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.606+0000 7fd90f2d1640 1 --2- 172.21.15.45:0/3580021285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080ce1e0 0x7fd90815cf60 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fd900009d90 tx=0x7fd90002beb0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.606+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3580021285 shutdown_connections 2023-11-28T20:53:40.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.606+0000 7fd90f2d1640 1 --2- 172.21.15.45:0/3580021285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080ce1e0 0x7fd90815cf60 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.606+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3580021285 >> 172.21.15.45:0/3580021285 conn(0x7fd9080b4c40 msgr2=0x7fd9080b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:40.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.606+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3580021285 shutdown_connections 2023-11-28T20:53:40.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.606+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3580021285 wait complete. 2023-11-28T20:53:40.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.608+0000 7fd90f2d1640 1 Processor -- start 2023-11-28T20:53:40.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.608+0000 7fd90f2d1640 1 -- start start 2023-11-28T20:53:40.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.609+0000 7fd90f2d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9080ce1e0 0x7fd908151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.609+0000 7fd90f2d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080cfaf0 0x7fd90815d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.609+0000 7fd90f2d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9080cfed0 0x7fd9080cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.609+0000 7fd90f2d1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080cc290 con 0x7fd9080cfaf0 2023-11-28T20:53:40.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.609+0000 7fd90f2d1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080cb9d0 con 0x7fd9080cfed0 2023-11-28T20:53:40.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.609+0000 7fd90f2d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd908151b80 con 0x7fd9080ce1e0 2023-11-28T20:53:40.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.609+0000 7fd90dace640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080cfaf0 0x7fd90815d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.609+0000 7fd90dace640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080cfaf0 0x7fd90815d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49680/0 (socket says 172.21.15.45:49680) 2023-11-28T20:53:40.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.609+0000 7fd90dace640 1 -- 172.21.15.45:0/3065474518 learned_addr learned my addr 172.21.15.45:0/3065474518 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:40.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90e2cf640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9080ce1e0 0x7fd908151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90ead0640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9080cfed0 0x7fd9080cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90dace640 1 -- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9080ce1e0 msgr2=0x7fd908151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90dace640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9080ce1e0 0x7fd908151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90dace640 1 -- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9080cfed0 msgr2=0x7fd9080cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90dace640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9080cfed0 0x7fd9080cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90dace640 1 -- 172.21.15.45:0/3065474518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900009a70 con 0x7fd9080cfaf0 2023-11-28T20:53:40.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90e2cf640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9080ce1e0 0x7fd908151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:40.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90ead0640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9080cfed0 0x7fd9080cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:40.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90dace640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080cfaf0 0x7fd90815d920 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800c400 tx=0x7fd8f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:40.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd8feffd640 1 -- 172.21.15.45:0/3065474518 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80052d0 con 0x7fd9080cfaf0 2023-11-28T20:53:40.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd8feffd640 1 -- 172.21.15.45:0/3065474518 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f8005430 con 0x7fd9080cfaf0 2023-11-28T20:53:40.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd8feffd640 1 -- 172.21.15.45:0/3065474518 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8003830 con 0x7fd9080cfaf0 2023-11-28T20:53:40.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.610+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9080cafc0 con 0x7fd9080cfaf0 2023-11-28T20:53:40.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.611+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9080cbf20 con 0x7fd9080cfaf0 2023-11-28T20:53:40.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.612+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9080cc0b0 con 0x7fd9080cfaf0 2023-11-28T20:53:40.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.613+0000 7fd8feffd640 1 -- 172.21.15.45:0/3065474518 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd8f8003990 con 0x7fd9080cfaf0 2023-11-28T20:53:40.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.614+0000 7fd8feffd640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd8ec06af90 0x7fd8ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:40.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.614+0000 7fd8feffd640 1 -- 172.21.15.45:0/3065474518 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd8f8088d50 con 0x7fd9080cfaf0 2023-11-28T20:53:40.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.615+0000 7fd90e2cf640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd8ec06af90 0x7fd8ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:40.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.616+0000 7fd90e2cf640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd8ec06af90 0x7fd8ec06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd904007fd0 tx=0x7fd904009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:40.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.619+0000 7fd8feffd640 1 -- 172.21.15.45:0/3065474518 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8f808c030 con 0x7fd9080cfaf0 2023-11-28T20:53:40.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.808+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 --> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9080d1780 con 0x7fd8ec06af90 2023-11-28T20:53:40.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.831+0000 7fd8feffd640 1 -- 172.21.15.45:0/3065474518 <== mgr.6655 v2:172.21.15.190:6832/3959324769 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287194 (secure 0 0 0) 0x7fd9040013f0 con 0x7fd8ec06af90 2023-11-28T20:53:40.837 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:53:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd8ec06af90 msgr2=0x7fd8ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd8ec06af90 0x7fd8ec06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd904007fd0 tx=0x7fd904009040 comp rx=0 tx=0).stop 2023-11-28T20:53:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080cfaf0 msgr2=0x7fd90815d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080cfaf0 0x7fd90815d920 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800c400 tx=0x7fd8f800c8c0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 shutdown_connections 2023-11-28T20:53:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7fd8ec06af90 0x7fd8ec06d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9080cfed0 0x7fd9080cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9080cfaf0 0x7fd90815d920 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 --2- 172.21.15.45:0/3065474518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9080ce1e0 0x7fd908151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 >> 172.21.15.45:0/3065474518 conn(0x7fd9080b4c40 msgr2=0x7fd9080b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 shutdown_connections 2023-11-28T20:53:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:40.837+0000 7fd90f2d1640 1 -- 172.21.15.45:0/3065474518 wait complete. 2023-11-28T20:53:40.860 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-11-28T20:53:39.654069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":997,"num_read_kb":1979,"num_write":375,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4750,"ondisk_log_size":4750,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":350836,"kb_used_data":13976,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754623884,"statfs":{"total":773094113280,"available":772734857216,"internally_reserved":0,"allocated":14311424,"data_stored":7798006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":344837680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.872249"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.853502+0000","last_change":"2023-11-28T20:53:35.852739+0000","last_active":"2023-11-28T20:53:35.853502+0000","last_peered":"2023-11-28T20:53:35.853502+0000","last_clean":"2023-11-28T20:53:35.853502+0000","last_became_active":"2023-11-28T20:53:35.849844+0000","last_became_peered":"2023-11-28T20:53:35.849844+0000","last_unstale":"2023-11-28T20:53:35.853502+0000","last_undegraded":"2023-11-28T20:53:35.853502+0000","last_fullsized":"2023-11-28T20:53:35.853502+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:06:32.653220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.840589+0000","last_change":"2023-11-28T20:53:35.834922+0000","last_active":"2023-11-28T20:53:35.840589+0000","last_peered":"2023-11-28T20:53:35.840589+0000","last_clean":"2023-11-28T20:53:35.840589+0000","last_became_active":"2023-11-28T20:53:35.834671+0000","last_became_peered":"2023-11-28T20:53:35.834671+0000","last_unstale":"2023-11-28T20:53:35.840589+0000","last_undegraded":"2023-11-28T20:53:35.840589+0000","last_fullsized":"2023-11-28T20:53:35.840589+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:41:48.627407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.849976+0000","last_change":"2023-11-28T20:53:35.838626+0000","last_active":"2023-11-28T20:53:35.849976+0000","last_peered":"2023-11-28T20:53:35.849976+0000","last_clean":"2023-11-28T20:53:35.849976+0000","last_became_active":"2023-11-28T20:53:35.836237+0000","last_became_peered":"2023-11-28T20:53:35.836237+0000","last_unstale":"2023-11-28T20:53:35.849976+0000","last_undegraded":"2023-11-28T20:53:35.849976+0000","last_fullsized":"2023-11-28T20:53:35.849976+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:32.617542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.851868+0000","last_change":"2023-11-28T20:53:35.841472+0000","last_active":"2023-11-28T20:53:35.851868+0000","last_peered":"2023-11-28T20:53:35.851868+0000","last_clean":"2023-11-28T20:53:35.851868+0000","last_became_active":"2023-11-28T20:53:35.841120+0000","last_became_peered":"2023-11-28T20:53:35.841120+0000","last_unstale":"2023-11-28T20:53:35.851868+0000","last_undegraded":"2023-11-28T20:53:35.851868+0000","last_fullsized":"2023-11-28T20:53:35.851868+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:56:52.886021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841378+0000","last_change":"2023-11-28T20:53:35.834266+0000","last_active":"2023-11-28T20:53:35.841378+0000","last_peered":"2023-11-28T20:53:35.841378+0000","last_clean":"2023-11-28T20:53:35.841378+0000","last_became_active":"2023-11-28T20:53:35.833311+0000","last_became_peered":"2023-11-28T20:53:35.833311+0000","last_unstale":"2023-11-28T20:53:35.841378+0000","last_undegraded":"2023-11-28T20:53:35.841378+0000","last_fullsized":"2023-11-28T20:53:35.841378+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:34.248688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.840440+0000","last_change":"2023-11-28T20:53:35.834650+0000","last_active":"2023-11-28T20:53:35.840440+0000","last_peered":"2023-11-28T20:53:35.840440+0000","last_clean":"2023-11-28T20:53:35.840440+0000","last_became_active":"2023-11-28T20:53:35.833712+0000","last_became_peered":"2023-11-28T20:53:35.833712+0000","last_unstale":"2023-11-28T20:53:35.840440+0000","last_undegraded":"2023-11-28T20:53:35.840440+0000","last_fullsized":"2023-11-28T20:53:35.840440+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:21:04.742706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841121+0000","last_change":"2023-11-28T20:53:35.836961+0000","last_active":"2023-11-28T20:53:35.841121+0000","last_peered":"2023-11-28T20:53:35.841121+0000","last_clean":"2023-11-28T20:53:35.841121+0000","last_became_active":"2023-11-28T20:53:35.836668+0000","last_became_peered":"2023-11-28T20:53:35.836668+0000","last_unstale":"2023-11-28T20:53:35.841121+0000","last_undegraded":"2023-11-28T20:53:35.841121+0000","last_fullsized":"2023-11-28T20:53:35.841121+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:17:36.299170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.839952+0000","last_change":"2023-11-28T20:53:35.837333+0000","last_active":"2023-11-28T20:53:35.839952+0000","last_peered":"2023-11-28T20:53:35.839952+0000","last_clean":"2023-11-28T20:53:35.839952+0000","last_became_active":"2023-11-28T20:53:35.836533+0000","last_became_peered":"2023-11-28T20:53:35.836533+0000","last_unstale":"2023-11-28T20:53:35.839952+0000","last_undegraded":"2023-11-28T20:53:35.839952+0000","last_fullsized":"2023-11-28T20:53:35.839952+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:39:16.187369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.855412+0000","last_change":"2023-11-28T20:53:35.853355+0000","last_active":"2023-11-28T20:53:35.855412+0000","last_peered":"2023-11-28T20:53:35.855412+0000","last_clean":"2023-11-28T20:53:35.855412+0000","last_became_active":"2023-11-28T20:53:35.852724+0000","last_became_peered":"2023-11-28T20:53:35.852724+0000","last_unstale":"2023-11-28T20:53:35.855412+0000","last_undegraded":"2023-11-28T20:53:35.855412+0000","last_fullsized":"2023-11-28T20:53:35.855412+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:33.776878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.842277+0000","last_change":"2023-11-28T20:53:35.835427+0000","last_active":"2023-11-28T20:53:35.842277+0000","last_peered":"2023-11-28T20:53:35.842277+0000","last_clean":"2023-11-28T20:53:35.842277+0000","last_became_active":"2023-11-28T20:53:35.835135+0000","last_became_peered":"2023-11-28T20:53:35.835135+0000","last_unstale":"2023-11-28T20:53:35.842277+0000","last_undegraded":"2023-11-28T20:53:35.842277+0000","last_fullsized":"2023-11-28T20:53:35.842277+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:19:30.296129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.839068+0000","last_change":"2023-11-28T20:53:35.834185+0000","last_active":"2023-11-28T20:53:35.839068+0000","last_peered":"2023-11-28T20:53:35.839068+0000","last_clean":"2023-11-28T20:53:35.839068+0000","last_became_active":"2023-11-28T20:53:35.833907+0000","last_became_peered":"2023-11-28T20:53:35.833907+0000","last_unstale":"2023-11-28T20:53:35.839068+0000","last_undegraded":"2023-11-28T20:53:35.839068+0000","last_fullsized":"2023-11-28T20:53:35.839068+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:04:29.608154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.843407+0000","last_change":"2023-11-28T20:53:35.842302+0000","last_active":"2023-11-28T20:53:35.843407+0000","last_peered":"2023-11-28T20:53:35.843407+0000","last_clean":"2023-11-28T20:53:35.843407+0000","last_became_active":"2023-11-28T20:53:35.842012+0000","last_became_peered":"2023-11-28T20:53:35.842012+0000","last_unstale":"2023-11-28T20:53:35.843407+0000","last_undegraded":"2023-11-28T20:53:35.843407+0000","last_fullsized":"2023-11-28T20:53:35.843407+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:37:38.636899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.848755+0000","last_change":"2023-11-28T20:53:35.839463+0000","last_active":"2023-11-28T20:53:35.848755+0000","last_peered":"2023-11-28T20:53:35.848755+0000","last_clean":"2023-11-28T20:53:35.848755+0000","last_became_active":"2023-11-28T20:53:35.839191+0000","last_became_peered":"2023-11-28T20:53:35.839191+0000","last_unstale":"2023-11-28T20:53:35.848755+0000","last_undegraded":"2023-11-28T20:53:35.848755+0000","last_fullsized":"2023-11-28T20:53:35.848755+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:19.520202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.844226+0000","last_change":"2023-11-28T20:53:35.842783+0000","last_active":"2023-11-28T20:53:35.844226+0000","last_peered":"2023-11-28T20:53:35.844226+0000","last_clean":"2023-11-28T20:53:35.844226+0000","last_became_active":"2023-11-28T20:53:35.842015+0000","last_became_peered":"2023-11-28T20:53:35.842015+0000","last_unstale":"2023-11-28T20:53:35.844226+0000","last_undegraded":"2023-11-28T20:53:35.844226+0000","last_fullsized":"2023-11-28T20:53:35.844226+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:15:27.519053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.840467+0000","last_change":"2023-11-28T20:53:35.837702+0000","last_active":"2023-11-28T20:53:35.840467+0000","last_peered":"2023-11-28T20:53:35.840467+0000","last_clean":"2023-11-28T20:53:35.840467+0000","last_became_active":"2023-11-28T20:53:35.836427+0000","last_became_peered":"2023-11-28T20:53:35.836427+0000","last_unstale":"2023-11-28T20:53:35.840467+0000","last_undegraded":"2023-11-28T20:53:35.840467+0000","last_fullsized":"2023-11-28T20:53:35.840467+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:34.815897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835098+0000","last_change":"2023-11-28T20:53:35.833376+0000","last_active":"2023-11-28T20:53:35.835098+0000","last_peered":"2023-11-28T20:53:35.835098+0000","last_clean":"2023-11-28T20:53:35.835098+0000","last_became_active":"2023-11-28T20:53:35.833064+0000","last_became_peered":"2023-11-28T20:53:35.833064+0000","last_unstale":"2023-11-28T20:53:35.835098+0000","last_undegraded":"2023-11-28T20:53:35.835098+0000","last_fullsized":"2023-11-28T20:53:35.835098+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:08.301254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850087+0000","last_change":"2023-11-28T20:53:35.845140+0000","last_active":"2023-11-28T20:53:35.850087+0000","last_peered":"2023-11-28T20:53:35.850087+0000","last_clean":"2023-11-28T20:53:35.850087+0000","last_became_active":"2023-11-28T20:53:35.844482+0000","last_became_peered":"2023-11-28T20:53:35.844482+0000","last_unstale":"2023-11-28T20:53:35.850087+0000","last_undegraded":"2023-11-28T20:53:35.850087+0000","last_fullsized":"2023-11-28T20:53:35.850087+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:10:31.647188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846086+0000","last_change":"2023-11-28T20:53:35.843456+0000","last_active":"2023-11-28T20:53:35.846086+0000","last_peered":"2023-11-28T20:53:35.846086+0000","last_clean":"2023-11-28T20:53:35.846086+0000","last_became_active":"2023-11-28T20:53:35.843070+0000","last_became_peered":"2023-11-28T20:53:35.843070+0000","last_unstale":"2023-11-28T20:53:35.846086+0000","last_undegraded":"2023-11-28T20:53:35.846086+0000","last_fullsized":"2023-11-28T20:53:35.846086+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:13.510882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850870+0000","last_change":"2023-11-28T20:53:35.849865+0000","last_active":"2023-11-28T20:53:35.850870+0000","last_peered":"2023-11-28T20:53:35.850870+0000","last_clean":"2023-11-28T20:53:35.850870+0000","last_became_active":"2023-11-28T20:53:35.846752+0000","last_became_peered":"2023-11-28T20:53:35.846752+0000","last_unstale":"2023-11-28T20:53:35.850870+0000","last_undegraded":"2023-11-28T20:53:35.850870+0000","last_fullsized":"2023-11-28T20:53:35.850870+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:15:19.399253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.831167+0000","last_change":"2023-11-28T20:53:35.827047+0000","last_active":"2023-11-28T20:53:35.831167+0000","last_peered":"2023-11-28T20:53:35.831167+0000","last_clean":"2023-11-28T20:53:35.831167+0000","last_became_active":"2023-11-28T20:53:35.826494+0000","last_became_peered":"2023-11-28T20:53:35.826494+0000","last_unstale":"2023-11-28T20:53:35.831167+0000","last_undegraded":"2023-11-28T20:53:35.831167+0000","last_fullsized":"2023-11-28T20:53:35.831167+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:27:32.599929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.837616+0000","last_change":"2023-11-28T20:53:35.834662+0000","last_active":"2023-11-28T20:53:35.837616+0000","last_peered":"2023-11-28T20:53:35.837616+0000","last_clean":"2023-11-28T20:53:35.837616+0000","last_became_active":"2023-11-28T20:53:35.833862+0000","last_became_peered":"2023-11-28T20:53:35.833862+0000","last_unstale":"2023-11-28T20:53:35.837616+0000","last_undegraded":"2023-11-28T20:53:35.837616+0000","last_fullsized":"2023-11-28T20:53:35.837616+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:57.255815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.820855+0000","last_change":"2023-11-28T20:53:35.814684+0000","last_active":"2023-11-28T20:53:35.820855+0000","last_peered":"2023-11-28T20:53:35.820855+0000","last_clean":"2023-11-28T20:53:35.820855+0000","last_became_active":"2023-11-28T20:53:35.813927+0000","last_became_peered":"2023-11-28T20:53:35.813927+0000","last_unstale":"2023-11-28T20:53:35.820855+0000","last_undegraded":"2023-11-28T20:53:35.820855+0000","last_fullsized":"2023-11-28T20:53:35.820855+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:31.127376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.843322+0000","last_change":"2023-11-28T20:53:35.834592+0000","last_active":"2023-11-28T20:53:35.843322+0000","last_peered":"2023-11-28T20:53:35.843322+0000","last_clean":"2023-11-28T20:53:35.843322+0000","last_became_active":"2023-11-28T20:53:35.834246+0000","last_became_peered":"2023-11-28T20:53:35.834246+0000","last_unstale":"2023-11-28T20:53:35.843322+0000","last_undegraded":"2023-11-28T20:53:35.843322+0000","last_fullsized":"2023-11-28T20:53:35.843322+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:43:04.039584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.847971+0000","last_change":"2023-11-28T20:53:35.844787+0000","last_active":"2023-11-28T20:53:35.847971+0000","last_peered":"2023-11-28T20:53:35.847971+0000","last_clean":"2023-11-28T20:53:35.847971+0000","last_became_active":"2023-11-28T20:53:35.844514+0000","last_became_peered":"2023-11-28T20:53:35.844514+0000","last_unstale":"2023-11-28T20:53:35.847971+0000","last_undegraded":"2023-11-28T20:53:35.847971+0000","last_fullsized":"2023-11-28T20:53:35.847971+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:33.776878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.832733+0000","last_change":"2023-11-28T20:53:35.828275+0000","last_active":"2023-11-28T20:53:35.832733+0000","last_peered":"2023-11-28T20:53:35.832733+0000","last_clean":"2023-11-28T20:53:35.832733+0000","last_became_active":"2023-11-28T20:53:35.827251+0000","last_became_peered":"2023-11-28T20:53:35.827251+0000","last_unstale":"2023-11-28T20:53:35.832733+0000","last_undegraded":"2023-11-28T20:53:35.832733+0000","last_fullsized":"2023-11-28T20:53:35.832733+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:42:40.422048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.842615+0000","last_change":"2023-11-28T20:53:35.840731+0000","last_active":"2023-11-28T20:53:35.842615+0000","last_peered":"2023-11-28T20:53:35.842615+0000","last_clean":"2023-11-28T20:53:35.842615+0000","last_became_active":"2023-11-28T20:53:35.840152+0000","last_became_peered":"2023-11-28T20:53:35.840152+0000","last_unstale":"2023-11-28T20:53:35.842615+0000","last_undegraded":"2023-11-28T20:53:35.842615+0000","last_fullsized":"2023-11-28T20:53:35.842615+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:31.311166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835107+0000","last_change":"2023-11-28T20:53:35.828179+0000","last_active":"2023-11-28T20:53:35.835107+0000","last_peered":"2023-11-28T20:53:35.835107+0000","last_clean":"2023-11-28T20:53:35.835107+0000","last_became_active":"2023-11-28T20:53:35.827873+0000","last_became_peered":"2023-11-28T20:53:35.827873+0000","last_unstale":"2023-11-28T20:53:35.835107+0000","last_undegraded":"2023-11-28T20:53:35.835107+0000","last_fullsized":"2023-11-28T20:53:35.835107+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:39:53.846804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841387+0000","last_change":"2023-11-28T20:53:35.836673+0000","last_active":"2023-11-28T20:53:35.841387+0000","last_peered":"2023-11-28T20:53:35.841387+0000","last_clean":"2023-11-28T20:53:35.841387+0000","last_became_active":"2023-11-28T20:53:35.836401+0000","last_became_peered":"2023-11-28T20:53:35.836401+0000","last_unstale":"2023-11-28T20:53:35.841387+0000","last_undegraded":"2023-11-28T20:53:35.841387+0000","last_fullsized":"2023-11-28T20:53:35.841387+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:44:54.363477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.833629+0000","last_change":"2023-11-28T20:53:35.830744+0000","last_active":"2023-11-28T20:53:35.833629+0000","last_peered":"2023-11-28T20:53:35.833629+0000","last_clean":"2023-11-28T20:53:35.833629+0000","last_became_active":"2023-11-28T20:53:35.830405+0000","last_became_peered":"2023-11-28T20:53:35.830405+0000","last_unstale":"2023-11-28T20:53:35.833629+0000","last_undegraded":"2023-11-28T20:53:35.833629+0000","last_fullsized":"2023-11-28T20:53:35.833629+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:08:39.518595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.831083+0000","last_change":"2023-11-28T20:53:35.827527+0000","last_active":"2023-11-28T20:53:35.831083+0000","last_peered":"2023-11-28T20:53:35.831083+0000","last_clean":"2023-11-28T20:53:35.831083+0000","last_became_active":"2023-11-28T20:53:35.827184+0000","last_became_peered":"2023-11-28T20:53:35.827184+0000","last_unstale":"2023-11-28T20:53:35.831083+0000","last_undegraded":"2023-11-28T20:53:35.831083+0000","last_fullsized":"2023-11-28T20:53:35.831083+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:51:34.248688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850005+0000","last_change":"2023-11-28T20:53:35.848368+0000","last_active":"2023-11-28T20:53:35.850005+0000","last_peered":"2023-11-28T20:53:35.850005+0000","last_clean":"2023-11-28T20:53:35.850005+0000","last_became_active":"2023-11-28T20:53:35.848146+0000","last_became_peered":"2023-11-28T20:53:35.848146+0000","last_unstale":"2023-11-28T20:53:35.850005+0000","last_undegraded":"2023-11-28T20:53:35.850005+0000","last_fullsized":"2023-11-28T20:53:35.850005+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:43:13.391431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.834794+0000","last_change":"2023-11-28T20:53:35.832100+0000","last_active":"2023-11-28T20:53:35.834794+0000","last_peered":"2023-11-28T20:53:35.834794+0000","last_clean":"2023-11-28T20:53:35.834794+0000","last_became_active":"2023-11-28T20:53:35.831788+0000","last_became_peered":"2023-11-28T20:53:35.831788+0000","last_unstale":"2023-11-28T20:53:35.834794+0000","last_undegraded":"2023-11-28T20:53:35.834794+0000","last_fullsized":"2023-11-28T20:53:35.834794+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:55:17.222039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.849418+0000","last_change":"2023-11-28T20:53:35.840108+0000","last_active":"2023-11-28T20:53:35.849418+0000","last_peered":"2023-11-28T20:53:35.849418+0000","last_clean":"2023-11-28T20:53:35.849418+0000","last_became_active":"2023-11-28T20:53:35.839791+0000","last_became_peered":"2023-11-28T20:53:35.839791+0000","last_unstale":"2023-11-28T20:53:35.849418+0000","last_undegraded":"2023-11-28T20:53:35.849418+0000","last_fullsized":"2023-11-28T20:53:35.849418+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:09:08.233912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"132'4","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862313+0000","last_change":"2023-11-28T20:53:34.837653+0000","last_active":"2023-11-28T20:53:35.862313+0000","last_peered":"2023-11-28T20:53:35.862313+0000","last_clean":"2023-11-28T20:53:35.862313+0000","last_became_active":"2023-11-28T20:53:34.837411+0000","last_became_peered":"2023-11-28T20:53:34.837411+0000","last_unstale":"2023-11-28T20:53:35.862313+0000","last_undegraded":"2023-11-28T20:53:35.862313+0000","last_fullsized":"2023-11-28T20:53:35.862313+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:49:52.291969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841301+0000","last_change":"2023-11-28T20:53:35.838999+0000","last_active":"2023-11-28T20:53:35.841301+0000","last_peered":"2023-11-28T20:53:35.841301+0000","last_clean":"2023-11-28T20:53:35.841301+0000","last_became_active":"2023-11-28T20:53:35.838711+0000","last_became_peered":"2023-11-28T20:53:35.838711+0000","last_unstale":"2023-11-28T20:53:35.841301+0000","last_undegraded":"2023-11-28T20:53:35.841301+0000","last_fullsized":"2023-11-28T20:53:35.841301+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:13:53.513846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860303+0000","last_change":"2023-11-28T20:53:34.823960+0000","last_active":"2023-11-28T20:53:35.860303+0000","last_peered":"2023-11-28T20:53:35.860303+0000","last_clean":"2023-11-28T20:53:35.860303+0000","last_became_active":"2023-11-28T20:53:34.823693+0000","last_became_peered":"2023-11-28T20:53:34.823693+0000","last_unstale":"2023-11-28T20:53:35.860303+0000","last_undegraded":"2023-11-28T20:53:35.860303+0000","last_fullsized":"2023-11-28T20:53:35.860303+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:58:50.214016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.832615+0000","last_change":"2023-11-28T20:53:35.827998+0000","last_active":"2023-11-28T20:53:35.832615+0000","last_peered":"2023-11-28T20:53:35.832615+0000","last_clean":"2023-11-28T20:53:35.832615+0000","last_became_active":"2023-11-28T20:53:35.827682+0000","last_became_peered":"2023-11-28T20:53:35.827682+0000","last_unstale":"2023-11-28T20:53:35.832615+0000","last_undegraded":"2023-11-28T20:53:35.832615+0000","last_fullsized":"2023-11-28T20:53:35.832615+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:31.127376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.859024+0000","last_change":"2023-11-28T20:53:34.832425+0000","last_active":"2023-11-28T20:53:35.859024+0000","last_peered":"2023-11-28T20:53:35.859024+0000","last_clean":"2023-11-28T20:53:35.859024+0000","last_became_active":"2023-11-28T20:53:34.828892+0000","last_became_peered":"2023-11-28T20:53:34.828892+0000","last_unstale":"2023-11-28T20:53:35.859024+0000","last_undegraded":"2023-11-28T20:53:35.859024+0000","last_fullsized":"2023-11-28T20:53:35.859024+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:06:20.193558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835832+0000","last_change":"2023-11-28T20:53:35.833686+0000","last_active":"2023-11-28T20:53:35.835832+0000","last_peered":"2023-11-28T20:53:35.835832+0000","last_clean":"2023-11-28T20:53:35.835832+0000","last_became_active":"2023-11-28T20:53:35.833072+0000","last_became_peered":"2023-11-28T20:53:35.833072+0000","last_unstale":"2023-11-28T20:53:35.835832+0000","last_undegraded":"2023-11-28T20:53:35.835832+0000","last_fullsized":"2023-11-28T20:53:35.835832+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:30:57.757815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862691+0000","last_change":"2023-11-28T20:53:34.825479+0000","last_active":"2023-11-28T20:53:35.862691+0000","last_peered":"2023-11-28T20:53:35.862691+0000","last_clean":"2023-11-28T20:53:35.862691+0000","last_became_active":"2023-11-28T20:53:34.824886+0000","last_became_peered":"2023-11-28T20:53:34.824886+0000","last_unstale":"2023-11-28T20:53:35.862691+0000","last_undegraded":"2023-11-28T20:53:35.862691+0000","last_fullsized":"2023-11-28T20:53:35.862691+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:02.261990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.851887+0000","last_change":"2023-11-28T20:53:35.847560+0000","last_active":"2023-11-28T20:53:35.851887+0000","last_peered":"2023-11-28T20:53:35.851887+0000","last_clean":"2023-11-28T20:53:35.851887+0000","last_became_active":"2023-11-28T20:53:35.847249+0000","last_became_peered":"2023-11-28T20:53:35.847249+0000","last_unstale":"2023-11-28T20:53:35.851887+0000","last_undegraded":"2023-11-28T20:53:35.851887+0000","last_fullsized":"2023-11-28T20:53:35.851887+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:15:27.519053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.858048+0000","last_change":"2023-11-28T20:53:34.837342+0000","last_active":"2023-11-28T20:53:35.858048+0000","last_peered":"2023-11-28T20:53:35.858048+0000","last_clean":"2023-11-28T20:53:35.858048+0000","last_became_active":"2023-11-28T20:53:34.836196+0000","last_became_peered":"2023-11-28T20:53:34.836196+0000","last_unstale":"2023-11-28T20:53:35.858048+0000","last_undegraded":"2023-11-28T20:53:35.858048+0000","last_fullsized":"2023-11-28T20:53:35.858048+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:32:18.505183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850036+0000","last_change":"2023-11-28T20:53:35.846037+0000","last_active":"2023-11-28T20:53:35.850036+0000","last_peered":"2023-11-28T20:53:35.850036+0000","last_clean":"2023-11-28T20:53:35.850036+0000","last_became_active":"2023-11-28T20:53:35.836620+0000","last_became_peered":"2023-11-28T20:53:35.836620+0000","last_unstale":"2023-11-28T20:53:35.850036+0000","last_undegraded":"2023-11-28T20:53:35.850036+0000","last_fullsized":"2023-11-28T20:53:35.850036+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:25:37.115772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862380+0000","last_change":"2023-11-28T20:53:34.830070+0000","last_active":"2023-11-28T20:53:35.862380+0000","last_peered":"2023-11-28T20:53:35.862380+0000","last_clean":"2023-11-28T20:53:35.862380+0000","last_became_active":"2023-11-28T20:53:34.829736+0000","last_became_peered":"2023-11-28T20:53:34.829736+0000","last_unstale":"2023-11-28T20:53:35.862380+0000","last_undegraded":"2023-11-28T20:53:35.862380+0000","last_fullsized":"2023-11-28T20:53:35.862380+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:16.173574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.844307+0000","last_change":"2023-11-28T20:53:35.842530+0000","last_active":"2023-11-28T20:53:35.844307+0000","last_peered":"2023-11-28T20:53:35.844307+0000","last_clean":"2023-11-28T20:53:35.844307+0000","last_became_active":"2023-11-28T20:53:35.842226+0000","last_became_peered":"2023-11-28T20:53:35.842226+0000","last_unstale":"2023-11-28T20:53:35.844307+0000","last_undegraded":"2023-11-28T20:53:35.844307+0000","last_fullsized":"2023-11-28T20:53:35.844307+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:50:19.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.820426+0000","last_change":"2023-11-28T20:53:34.825641+0000","last_active":"2023-11-28T20:53:35.820426+0000","last_peered":"2023-11-28T20:53:35.820426+0000","last_clean":"2023-11-28T20:53:35.820426+0000","last_became_active":"2023-11-28T20:53:34.825237+0000","last_became_peered":"2023-11-28T20:53:34.825237+0000","last_unstale":"2023-11-28T20:53:35.820426+0000","last_undegraded":"2023-11-28T20:53:35.820426+0000","last_fullsized":"2023-11-28T20:53:35.820426+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:19:46.092231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.851903+0000","last_change":"2023-11-28T20:53:35.844312+0000","last_active":"2023-11-28T20:53:35.851903+0000","last_peered":"2023-11-28T20:53:35.851903+0000","last_clean":"2023-11-28T20:53:35.851903+0000","last_became_active":"2023-11-28T20:53:35.844009+0000","last_became_peered":"2023-11-28T20:53:35.844009+0000","last_unstale":"2023-11-28T20:53:35.851903+0000","last_undegraded":"2023-11-28T20:53:35.851903+0000","last_fullsized":"2023-11-28T20:53:35.851903+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:09.910426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.822185+0000","last_change":"2023-11-28T20:53:34.820152+0000","last_active":"2023-11-28T20:53:35.822185+0000","last_peered":"2023-11-28T20:53:35.822185+0000","last_clean":"2023-11-28T20:53:35.822185+0000","last_became_active":"2023-11-28T20:53:34.819839+0000","last_became_peered":"2023-11-28T20:53:34.819839+0000","last_unstale":"2023-11-28T20:53:35.822185+0000","last_undegraded":"2023-11-28T20:53:35.822185+0000","last_fullsized":"2023-11-28T20:53:35.822185+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:33.800878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846640+0000","last_change":"2023-11-28T20:53:35.839179+0000","last_active":"2023-11-28T20:53:35.846640+0000","last_peered":"2023-11-28T20:53:35.846640+0000","last_clean":"2023-11-28T20:53:35.846640+0000","last_became_active":"2023-11-28T20:53:35.836378+0000","last_became_peered":"2023-11-28T20:53:35.836378+0000","last_unstale":"2023-11-28T20:53:35.846640+0000","last_undegraded":"2023-11-28T20:53:35.846640+0000","last_fullsized":"2023-11-28T20:53:35.846640+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:22:57.476588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.822275+0000","last_change":"2023-11-28T20:53:34.830803+0000","last_active":"2023-11-28T20:53:35.822275+0000","last_peered":"2023-11-28T20:53:35.822275+0000","last_clean":"2023-11-28T20:53:35.822275+0000","last_became_active":"2023-11-28T20:53:34.826684+0000","last_became_peered":"2023-11-28T20:53:34.826684+0000","last_unstale":"2023-11-28T20:53:35.822275+0000","last_undegraded":"2023-11-28T20:53:35.822275+0000","last_fullsized":"2023-11-28T20:53:35.822275+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:35:15.873444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841352+0000","last_change":"2023-11-28T20:53:35.833993+0000","last_active":"2023-11-28T20:53:35.841352+0000","last_peered":"2023-11-28T20:53:35.841352+0000","last_clean":"2023-11-28T20:53:35.841352+0000","last_became_active":"2023-11-28T20:53:35.833671+0000","last_became_peered":"2023-11-28T20:53:35.833671+0000","last_unstale":"2023-11-28T20:53:35.841352+0000","last_undegraded":"2023-11-28T20:53:35.841352+0000","last_fullsized":"2023-11-28T20:53:35.841352+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:41:48.627407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.813830+0000","last_change":"2023-11-28T20:53:34.829792+0000","last_active":"2023-11-28T20:53:35.813830+0000","last_peered":"2023-11-28T20:53:35.813830+0000","last_clean":"2023-11-28T20:53:35.813830+0000","last_became_active":"2023-11-28T20:53:34.828811+0000","last_became_peered":"2023-11-28T20:53:34.828811+0000","last_unstale":"2023-11-28T20:53:35.813830+0000","last_undegraded":"2023-11-28T20:53:35.813830+0000","last_fullsized":"2023-11-28T20:53:35.813830+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:03:21.752817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.839268+0000","last_change":"2023-11-28T20:53:35.834360+0000","last_active":"2023-11-28T20:53:35.839268+0000","last_peered":"2023-11-28T20:53:35.839268+0000","last_clean":"2023-11-28T20:53:35.839268+0000","last_became_active":"2023-11-28T20:53:35.834090+0000","last_became_peered":"2023-11-28T20:53:35.834090+0000","last_unstale":"2023-11-28T20:53:35.839268+0000","last_undegraded":"2023-11-28T20:53:35.839268+0000","last_fullsized":"2023-11-28T20:53:35.839268+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:26.036147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"132'3","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860342+0000","last_change":"2023-11-28T20:53:34.825945+0000","last_active":"2023-11-28T20:53:35.860342+0000","last_peered":"2023-11-28T20:53:35.860342+0000","last_clean":"2023-11-28T20:53:35.860342+0000","last_became_active":"2023-11-28T20:53:34.824621+0000","last_became_peered":"2023-11-28T20:53:34.824621+0000","last_unstale":"2023-11-28T20:53:35.860342+0000","last_undegraded":"2023-11-28T20:53:35.860342+0000","last_fullsized":"2023-11-28T20:53:35.860342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:56:51.871002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.838369+0000","last_change":"2023-11-28T20:53:35.833020+0000","last_active":"2023-11-28T20:53:35.838369+0000","last_peered":"2023-11-28T20:53:35.838369+0000","last_clean":"2023-11-28T20:53:35.838369+0000","last_became_active":"2023-11-28T20:53:35.831333+0000","last_became_peered":"2023-11-28T20:53:35.831333+0000","last_unstale":"2023-11-28T20:53:35.838369+0000","last_undegraded":"2023-11-28T20:53:35.838369+0000","last_fullsized":"2023-11-28T20:53:35.838369+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:37:38.636899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"132'6","reported_seq":17,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862145+0000","last_change":"2023-11-28T20:53:34.829678+0000","last_active":"2023-11-28T20:53:35.862145+0000","last_peered":"2023-11-28T20:53:35.862145+0000","last_clean":"2023-11-28T20:53:35.862145+0000","last_became_active":"2023-11-28T20:53:34.829287+0000","last_became_peered":"2023-11-28T20:53:34.829287+0000","last_unstale":"2023-11-28T20:53:35.862145+0000","last_undegraded":"2023-11-28T20:53:35.862145+0000","last_fullsized":"2023-11-28T20:53:35.862145+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:19:29.281109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846717+0000","last_change":"2023-11-28T20:53:35.845380+0000","last_active":"2023-11-28T20:53:35.846717+0000","last_peered":"2023-11-28T20:53:35.846717+0000","last_clean":"2023-11-28T20:53:35.846717+0000","last_became_active":"2023-11-28T20:53:35.845079+0000","last_became_peered":"2023-11-28T20:53:35.845079+0000","last_unstale":"2023-11-28T20:53:35.846717+0000","last_undegraded":"2023-11-28T20:53:35.846717+0000","last_fullsized":"2023-11-28T20:53:35.846717+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:36:07.856406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860985+0000","last_change":"2023-11-28T20:53:34.814256+0000","last_active":"2023-11-28T20:53:35.860985+0000","last_peered":"2023-11-28T20:53:35.860985+0000","last_clean":"2023-11-28T20:53:35.860985+0000","last_became_active":"2023-11-28T20:53:34.813926+0000","last_became_peered":"2023-11-28T20:53:34.813926+0000","last_unstale":"2023-11-28T20:53:35.860985+0000","last_undegraded":"2023-11-28T20:53:35.860985+0000","last_fullsized":"2023-11-28T20:53:35.860985+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:30.296146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.844655+0000","last_change":"2023-11-28T20:53:35.842525+0000","last_active":"2023-11-28T20:53:35.844655+0000","last_peered":"2023-11-28T20:53:35.844655+0000","last_clean":"2023-11-28T20:53:35.844655+0000","last_became_active":"2023-11-28T20:53:35.842240+0000","last_became_peered":"2023-11-28T20:53:35.842240+0000","last_unstale":"2023-11-28T20:53:35.844655+0000","last_undegraded":"2023-11-28T20:53:35.844655+0000","last_fullsized":"2023-11-28T20:53:35.844655+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:22:52.747986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.828971+0000","last_change":"2023-11-28T20:53:34.823919+0000","last_active":"2023-11-28T20:53:35.828971+0000","last_peered":"2023-11-28T20:53:35.828971+0000","last_clean":"2023-11-28T20:53:35.828971+0000","last_became_active":"2023-11-28T20:53:34.823663+0000","last_became_peered":"2023-11-28T20:53:34.823663+0000","last_unstale":"2023-11-28T20:53:35.828971+0000","last_undegraded":"2023-11-28T20:53:35.828971+0000","last_fullsized":"2023-11-28T20:53:35.828971+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:00.263143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.839418+0000","last_change":"2023-11-28T20:53:35.835067+0000","last_active":"2023-11-28T20:53:35.839418+0000","last_peered":"2023-11-28T20:53:35.839418+0000","last_clean":"2023-11-28T20:53:35.839418+0000","last_became_active":"2023-11-28T20:53:35.833326+0000","last_became_peered":"2023-11-28T20:53:35.833326+0000","last_unstale":"2023-11-28T20:53:35.839418+0000","last_undegraded":"2023-11-28T20:53:35.839418+0000","last_fullsized":"2023-11-28T20:53:35.839418+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:17:30.238147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.820032+0000","last_change":"2023-11-28T20:53:34.829509+0000","last_active":"2023-11-28T20:53:35.820032+0000","last_peered":"2023-11-28T20:53:35.820032+0000","last_clean":"2023-11-28T20:53:35.820032+0000","last_became_active":"2023-11-28T20:53:34.829103+0000","last_became_peered":"2023-11-28T20:53:34.829103+0000","last_unstale":"2023-11-28T20:53:35.820032+0000","last_undegraded":"2023-11-28T20:53:35.820032+0000","last_fullsized":"2023-11-28T20:53:35.820032+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:51:36.369028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.850110+0000","last_change":"2023-11-28T20:53:35.845392+0000","last_active":"2023-11-28T20:53:35.850110+0000","last_peered":"2023-11-28T20:53:35.850110+0000","last_clean":"2023-11-28T20:53:35.850110+0000","last_became_active":"2023-11-28T20:53:35.844870+0000","last_became_peered":"2023-11-28T20:53:35.844870+0000","last_unstale":"2023-11-28T20:53:35.850110+0000","last_undegraded":"2023-11-28T20:53:35.850110+0000","last_fullsized":"2023-11-28T20:53:35.850110+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:46:34.712897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"132'8","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.862165+0000","last_change":"2023-11-28T20:53:34.828747+0000","last_active":"2023-11-28T20:53:35.862165+0000","last_peered":"2023-11-28T20:53:35.862165+0000","last_clean":"2023-11-28T20:53:35.862165+0000","last_became_active":"2023-11-28T20:53:34.828471+0000","last_became_peered":"2023-11-28T20:53:34.828471+0000","last_unstale":"2023-11-28T20:53:35.862165+0000","last_undegraded":"2023-11-28T20:53:35.862165+0000","last_fullsized":"2023-11-28T20:53:35.862165+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:44:26.538531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835822+0000","last_change":"2023-11-28T20:53:35.833822+0000","last_active":"2023-11-28T20:53:35.835822+0000","last_peered":"2023-11-28T20:53:35.835822+0000","last_clean":"2023-11-28T20:53:35.835822+0000","last_became_active":"2023-11-28T20:53:35.833508+0000","last_became_peered":"2023-11-28T20:53:35.833508+0000","last_unstale":"2023-11-28T20:53:35.835822+0000","last_undegraded":"2023-11-28T20:53:35.835822+0000","last_fullsized":"2023-11-28T20:53:35.835822+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:39.328623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.857183+0000","last_change":"2023-11-28T20:53:34.836319+0000","last_active":"2023-11-28T20:53:35.857183+0000","last_peered":"2023-11-28T20:53:35.857183+0000","last_clean":"2023-11-28T20:53:35.857183+0000","last_became_active":"2023-11-28T20:53:34.835521+0000","last_became_peered":"2023-11-28T20:53:34.835521+0000","last_unstale":"2023-11-28T20:53:35.857183+0000","last_undegraded":"2023-11-28T20:53:35.857183+0000","last_fullsized":"2023-11-28T20:53:35.857183+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:40.719594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846692+0000","last_change":"2023-11-28T20:53:35.844184+0000","last_active":"2023-11-28T20:53:35.846692+0000","last_peered":"2023-11-28T20:53:35.846692+0000","last_clean":"2023-11-28T20:53:35.846692+0000","last_became_active":"2023-11-28T20:53:35.843890+0000","last_became_peered":"2023-11-28T20:53:35.843890+0000","last_unstale":"2023-11-28T20:53:35.846692+0000","last_undegraded":"2023-11-28T20:53:35.846692+0000","last_fullsized":"2023-11-28T20:53:35.846692+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:36:44.724734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"132'1","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.858410+0000","last_change":"2023-11-28T20:53:34.827922+0000","last_active":"2023-11-28T20:53:35.858410+0000","last_peered":"2023-11-28T20:53:35.858410+0000","last_clean":"2023-11-28T20:53:35.858410+0000","last_became_active":"2023-11-28T20:53:34.827614+0000","last_became_peered":"2023-11-28T20:53:34.827614+0000","last_unstale":"2023-11-28T20:53:35.858410+0000","last_undegraded":"2023-11-28T20:53:35.858410+0000","last_fullsized":"2023-11-28T20:53:35.858410+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:53.267683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.834719+0000","last_change":"2023-11-28T20:53:35.827063+0000","last_active":"2023-11-28T20:53:35.834719+0000","last_peered":"2023-11-28T20:53:35.834719+0000","last_clean":"2023-11-28T20:53:35.834719+0000","last_became_active":"2023-11-28T20:53:35.825634+0000","last_became_peered":"2023-11-28T20:53:35.825634+0000","last_unstale":"2023-11-28T20:53:35.834719+0000","last_undegraded":"2023-11-28T20:53:35.834719+0000","last_fullsized":"2023-11-28T20:53:35.834719+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:27:32.599929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.811062+0000","last_change":"2023-11-28T20:53:34.834697+0000","last_active":"2023-11-28T20:53:35.811062+0000","last_peered":"2023-11-28T20:53:35.811062+0000","last_clean":"2023-11-28T20:53:35.811062+0000","last_became_active":"2023-11-28T20:53:34.834372+0000","last_became_peered":"2023-11-28T20:53:34.834372+0000","last_unstale":"2023-11-28T20:53:35.811062+0000","last_undegraded":"2023-11-28T20:53:35.811062+0000","last_fullsized":"2023-11-28T20:53:35.811062+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:37:56.208280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.852648+0000","last_change":"2023-11-28T20:53:35.850306+0000","last_active":"2023-11-28T20:53:35.852648+0000","last_peered":"2023-11-28T20:53:35.852648+0000","last_clean":"2023-11-28T20:53:35.852648+0000","last_became_active":"2023-11-28T20:53:35.849614+0000","last_became_peered":"2023-11-28T20:53:35.849614+0000","last_unstale":"2023-11-28T20:53:35.852648+0000","last_undegraded":"2023-11-28T20:53:35.852648+0000","last_fullsized":"2023-11-28T20:53:35.852648+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:05:41.061989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860348+0000","last_change":"2023-11-28T20:53:34.814296+0000","last_active":"2023-11-28T20:53:35.860348+0000","last_peered":"2023-11-28T20:53:35.860348+0000","last_clean":"2023-11-28T20:53:35.860348+0000","last_became_active":"2023-11-28T20:53:34.814030+0000","last_became_peered":"2023-11-28T20:53:34.814030+0000","last_unstale":"2023-11-28T20:53:35.860348+0000","last_undegraded":"2023-11-28T20:53:35.860348+0000","last_fullsized":"2023-11-28T20:53:35.860348+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:22:48.386297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.852304+0000","last_change":"2023-11-28T20:53:35.850046+0000","last_active":"2023-11-28T20:53:35.852304+0000","last_peered":"2023-11-28T20:53:35.852304+0000","last_clean":"2023-11-28T20:53:35.852304+0000","last_became_active":"2023-11-28T20:53:35.847288+0000","last_became_peered":"2023-11-28T20:53:35.847288+0000","last_unstale":"2023-11-28T20:53:35.852304+0000","last_undegraded":"2023-11-28T20:53:35.852304+0000","last_fullsized":"2023-11-28T20:53:35.852304+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:35:39.884963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.818932+0000","last_change":"2023-11-28T20:53:34.821897+0000","last_active":"2023-11-28T20:53:35.818932+0000","last_peered":"2023-11-28T20:53:35.818932+0000","last_clean":"2023-11-28T20:53:35.818932+0000","last_became_active":"2023-11-28T20:53:34.821618+0000","last_became_peered":"2023-11-28T20:53:34.821618+0000","last_unstale":"2023-11-28T20:53:35.818932+0000","last_undegraded":"2023-11-28T20:53:35.818932+0000","last_fullsized":"2023-11-28T20:53:35.818932+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:14:47.750327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.852013+0000","last_change":"2023-11-28T20:53:35.844896+0000","last_active":"2023-11-28T20:53:35.852013+0000","last_peered":"2023-11-28T20:53:35.852013+0000","last_clean":"2023-11-28T20:53:35.852013+0000","last_became_active":"2023-11-28T20:53:35.844642+0000","last_became_peered":"2023-11-28T20:53:35.844642+0000","last_unstale":"2023-11-28T20:53:35.852013+0000","last_undegraded":"2023-11-28T20:53:35.852013+0000","last_fullsized":"2023-11-28T20:53:35.852013+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:54.282703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.825270+0000","last_change":"2023-11-28T20:53:34.832781+0000","last_active":"2023-11-28T20:53:35.825270+0000","last_peered":"2023-11-28T20:53:35.825270+0000","last_clean":"2023-11-28T20:53:35.825270+0000","last_became_active":"2023-11-28T20:53:34.832108+0000","last_became_peered":"2023-11-28T20:53:34.832108+0000","last_unstale":"2023-11-28T20:53:35.825270+0000","last_undegraded":"2023-11-28T20:53:35.825270+0000","last_fullsized":"2023-11-28T20:53:35.825270+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:44:32.615441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.837205+0000","last_change":"2023-11-28T20:53:35.835702+0000","last_active":"2023-11-28T20:53:35.837205+0000","last_peered":"2023-11-28T20:53:35.837205+0000","last_clean":"2023-11-28T20:53:35.837205+0000","last_became_active":"2023-11-28T20:53:35.835420+0000","last_became_peered":"2023-11-28T20:53:35.835420+0000","last_unstale":"2023-11-28T20:53:35.837205+0000","last_undegraded":"2023-11-28T20:53:35.837205+0000","last_fullsized":"2023-11-28T20:53:35.837205+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:24:58.104641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.823267+0000","last_change":"2023-11-28T20:53:34.820508+0000","last_active":"2023-11-28T20:53:35.823267+0000","last_peered":"2023-11-28T20:53:35.823267+0000","last_clean":"2023-11-28T20:53:35.823267+0000","last_became_active":"2023-11-28T20:53:34.820182+0000","last_became_peered":"2023-11-28T20:53:34.820182+0000","last_unstale":"2023-11-28T20:53:35.823267+0000","last_undegraded":"2023-11-28T20:53:35.823267+0000","last_fullsized":"2023-11-28T20:53:35.823267+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:21:04.091589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.853533+0000","last_change":"2023-11-28T20:53:35.852061+0000","last_active":"2023-11-28T20:53:35.853533+0000","last_peered":"2023-11-28T20:53:35.853533+0000","last_clean":"2023-11-28T20:53:35.853533+0000","last_became_active":"2023-11-28T20:53:35.851798+0000","last_became_peered":"2023-11-28T20:53:35.851798+0000","last_unstale":"2023-11-28T20:53:35.853533+0000","last_undegraded":"2023-11-28T20:53:35.853533+0000","last_fullsized":"2023-11-28T20:53:35.853533+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:30:56.956173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860800+0000","last_change":"2023-11-28T20:53:34.815883+0000","last_active":"2023-11-28T20:53:35.860800+0000","last_peered":"2023-11-28T20:53:35.860800+0000","last_clean":"2023-11-28T20:53:35.860800+0000","last_became_active":"2023-11-28T20:53:34.814723+0000","last_became_peered":"2023-11-28T20:53:34.814723+0000","last_unstale":"2023-11-28T20:53:35.860800+0000","last_undegraded":"2023-11-28T20:53:35.860800+0000","last_fullsized":"2023-11-28T20:53:35.860800+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:06.204609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841631+0000","last_change":"2023-11-28T20:53:35.839542+0000","last_active":"2023-11-28T20:53:35.841631+0000","last_peered":"2023-11-28T20:53:35.841631+0000","last_clean":"2023-11-28T20:53:35.841631+0000","last_became_active":"2023-11-28T20:53:35.839237+0000","last_became_peered":"2023-11-28T20:53:35.839237+0000","last_unstale":"2023-11-28T20:53:35.841631+0000","last_undegraded":"2023-11-28T20:53:35.841631+0000","last_fullsized":"2023-11-28T20:53:35.841631+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:14:07.219629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.814312+0000","last_change":"2023-11-28T20:53:34.826741+0000","last_active":"2023-11-28T20:53:35.814312+0000","last_peered":"2023-11-28T20:53:35.814312+0000","last_clean":"2023-11-28T20:53:35.814312+0000","last_became_active":"2023-11-28T20:53:34.826411+0000","last_became_peered":"2023-11-28T20:53:34.826411+0000","last_unstale":"2023-11-28T20:53:35.814312+0000","last_undegraded":"2023-11-28T20:53:35.814312+0000","last_fullsized":"2023-11-28T20:53:35.814312+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:44:36.874589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.843301+0000","last_change":"2023-11-28T20:53:35.835914+0000","last_active":"2023-11-28T20:53:35.843301+0000","last_peered":"2023-11-28T20:53:35.843301+0000","last_clean":"2023-11-28T20:53:35.843301+0000","last_became_active":"2023-11-28T20:53:35.835579+0000","last_became_peered":"2023-11-28T20:53:35.835579+0000","last_unstale":"2023-11-28T20:53:35.843301+0000","last_undegraded":"2023-11-28T20:53:35.843301+0000","last_fullsized":"2023-11-28T20:53:35.843301+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:08.301254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.825271+0000","last_change":"2023-11-28T20:53:34.835803+0000","last_active":"2023-11-28T20:53:35.825271+0000","last_peered":"2023-11-28T20:53:35.825271+0000","last_clean":"2023-11-28T20:53:35.825271+0000","last_became_active":"2023-11-28T20:53:34.835317+0000","last_became_peered":"2023-11-28T20:53:34.835317+0000","last_unstale":"2023-11-28T20:53:35.825271+0000","last_undegraded":"2023-11-28T20:53:35.825271+0000","last_fullsized":"2023-11-28T20:53:35.825271+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:53:49.750751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835641+0000","last_change":"2023-11-28T20:53:35.832186+0000","last_active":"2023-11-28T20:53:35.835641+0000","last_peered":"2023-11-28T20:53:35.835641+0000","last_clean":"2023-11-28T20:53:35.835641+0000","last_became_active":"2023-11-28T20:53:35.831565+0000","last_became_peered":"2023-11-28T20:53:35.831565+0000","last_unstale":"2023-11-28T20:53:35.835641+0000","last_undegraded":"2023-11-28T20:53:35.835641+0000","last_fullsized":"2023-11-28T20:53:35.835641+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:13:09.488951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.805018+0000","last_change":"2023-11-28T20:53:34.834367+0000","last_active":"2023-11-28T20:53:35.805018+0000","last_peered":"2023-11-28T20:53:35.805018+0000","last_clean":"2023-11-28T20:53:35.805018+0000","last_became_active":"2023-11-28T20:53:34.834019+0000","last_became_peered":"2023-11-28T20:53:34.834019+0000","last_unstale":"2023-11-28T20:53:35.805018+0000","last_undegraded":"2023-11-28T20:53:35.805018+0000","last_fullsized":"2023-11-28T20:53:35.805018+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:47:40.719594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.849499+0000","last_change":"2023-11-28T20:53:35.842065+0000","last_active":"2023-11-28T20:53:35.849499+0000","last_peered":"2023-11-28T20:53:35.849499+0000","last_clean":"2023-11-28T20:53:35.849499+0000","last_became_active":"2023-11-28T20:53:35.841773+0000","last_became_peered":"2023-11-28T20:53:35.841773+0000","last_unstale":"2023-11-28T20:53:35.849499+0000","last_undegraded":"2023-11-28T20:53:35.849499+0000","last_fullsized":"2023-11-28T20:53:35.849499+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:38:11.060050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.821139+0000","last_change":"2023-11-28T20:53:34.836093+0000","last_active":"2023-11-28T20:53:35.821139+0000","last_peered":"2023-11-28T20:53:35.821139+0000","last_clean":"2023-11-28T20:53:35.821139+0000","last_became_active":"2023-11-28T20:53:34.835797+0000","last_became_peered":"2023-11-28T20:53:34.835797+0000","last_unstale":"2023-11-28T20:53:35.821139+0000","last_undegraded":"2023-11-28T20:53:35.821139+0000","last_fullsized":"2023-11-28T20:53:35.821139+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:58:38.485097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.841338+0000","last_change":"2023-11-28T20:53:35.837810+0000","last_active":"2023-11-28T20:53:35.841338+0000","last_peered":"2023-11-28T20:53:35.841338+0000","last_clean":"2023-11-28T20:53:35.841338+0000","last_became_active":"2023-11-28T20:53:35.837527+0000","last_became_peered":"2023-11-28T20:53:35.837527+0000","last_unstale":"2023-11-28T20:53:35.841338+0000","last_undegraded":"2023-11-28T20:53:35.841338+0000","last_fullsized":"2023-11-28T20:53:35.841338+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:00:52.844681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.825134+0000","last_change":"2023-11-28T20:53:34.829508+0000","last_active":"2023-11-28T20:53:35.825134+0000","last_peered":"2023-11-28T20:53:35.825134+0000","last_clean":"2023-11-28T20:53:35.825134+0000","last_became_active":"2023-11-28T20:53:34.829045+0000","last_became_peered":"2023-11-28T20:53:34.829045+0000","last_unstale":"2023-11-28T20:53:35.825134+0000","last_undegraded":"2023-11-28T20:53:35.825134+0000","last_fullsized":"2023-11-28T20:53:35.825134+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:00:02.431917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.838672+0000","last_change":"2023-11-28T20:53:35.837327+0000","last_active":"2023-11-28T20:53:35.838672+0000","last_peered":"2023-11-28T20:53:35.838672+0000","last_clean":"2023-11-28T20:53:35.838672+0000","last_became_active":"2023-11-28T20:53:35.837067+0000","last_became_peered":"2023-11-28T20:53:35.837067+0000","last_unstale":"2023-11-28T20:53:35.838672+0000","last_undegraded":"2023-11-28T20:53:35.838672+0000","last_fullsized":"2023-11-28T20:53:35.838672+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:09.910426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.825543+0000","last_change":"2023-11-28T20:53:34.833368+0000","last_active":"2023-11-28T20:53:35.825543+0000","last_peered":"2023-11-28T20:53:35.825543+0000","last_clean":"2023-11-28T20:53:35.825543+0000","last_became_active":"2023-11-28T20:53:34.832510+0000","last_became_peered":"2023-11-28T20:53:34.832510+0000","last_unstale":"2023-11-28T20:53:35.825543+0000","last_undegraded":"2023-11-28T20:53:35.825543+0000","last_fullsized":"2023-11-28T20:53:35.825543+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:22:02.261990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.835235+0000","last_change":"2023-11-28T20:53:35.830174+0000","last_active":"2023-11-28T20:53:35.835235+0000","last_peered":"2023-11-28T20:53:35.835235+0000","last_clean":"2023-11-28T20:53:35.835235+0000","last_became_active":"2023-11-28T20:53:35.828266+0000","last_became_peered":"2023-11-28T20:53:35.828266+0000","last_unstale":"2023-11-28T20:53:35.835235+0000","last_undegraded":"2023-11-28T20:53:35.835235+0000","last_fullsized":"2023-11-28T20:53:35.835235+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:07:30.282009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.824757+0000","last_change":"2023-11-28T20:53:34.824657+0000","last_active":"2023-11-28T20:53:35.824757+0000","last_peered":"2023-11-28T20:53:35.824757+0000","last_clean":"2023-11-28T20:53:35.824757+0000","last_became_active":"2023-11-28T20:53:34.824365+0000","last_became_peered":"2023-11-28T20:53:34.824365+0000","last_unstale":"2023-11-28T20:53:35.824757+0000","last_undegraded":"2023-11-28T20:53:35.824757+0000","last_fullsized":"2023-11-28T20:53:35.824757+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:44.144409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.846632+0000","last_change":"2023-11-28T20:53:35.844727+0000","last_active":"2023-11-28T20:53:35.846632+0000","last_peered":"2023-11-28T20:53:35.846632+0000","last_clean":"2023-11-28T20:53:35.846632+0000","last_became_active":"2023-11-28T20:53:35.844394+0000","last_became_peered":"2023-11-28T20:53:35.844394+0000","last_unstale":"2023-11-28T20:53:35.846632+0000","last_undegraded":"2023-11-28T20:53:35.846632+0000","last_fullsized":"2023-11-28T20:53:35.846632+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:34.780219+0000","last_clean_scrub_stamp":"2023-11-28T20:53:34.780219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:32:43.243619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"132'2","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-11-28T20:53:35.860428+0000","last_change":"2023-11-28T20:53:34.822269+0000","last_active":"2023-11-28T20:53:35.860428+0000","last_peered":"2023-11-28T20:53:35.860428+0000","last_clean":"2023-11-28T20:53:35.860428+0000","last_became_active":"2023-11-28T20:53:34.821985+0000","last_became_peered":"2023-11-28T20:53:34.821985+0000","last_unstale":"2023-11-28T20:53:35.860428+0000","last_undegraded":"2023-11-28T20:53:35.860428+0000","last_fullsized":"2023-11-28T20:53:35.860428+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:53:33.765199+0000","last_clean_scrub_stamp":"2023-11-28T20:53:33.765199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:54.528095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"121'4712","reported_seq":5157,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-11-28T20:53:34.821427+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:53:34.821427+0000","last_peered":"2023-11-28T20:53:34.821427+0000","last_clean":"2023-11-28T20:53:34.821427+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:53:34.821427+0000","last_undegraded":"2023-11-28T20:53:34.821427+0000","last_fullsized":"2023-11-28T20:53:34.821427+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":997,"num_read_kb":1979,"num_write":331,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":997,"num_read_kb":1979,"num_write":331,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4712,"ondisk_log_size":4712,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47248,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324592,"statfs":{"total":96636764160,"available":96588382208,"internally_reserved":0,"allocated":2236416,"data_stored":1417297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1519999999999999,"15min":1.1839999999999999},"min":{"1min":0.75700000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.6910000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.04},{"interface":"front","average":{"1min":1.056,"5min":1.0980000000000001,"15min":1.1599999999999999},"min":{"1min":0.72999999999999998,"5min":0.64300000000000002,"15min":0.58299999999999996},"max":{"1min":1.502,"5min":1.8,"15min":1.8},"last":1.431}]},{"osd":1,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.1659999999999999,"15min":1.1830000000000001},"min":{"1min":0.74299999999999999,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5720000000000001,"5min":1.633,"15min":1.744},"last":1.228},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2609999999999999,"15min":1.2589999999999999},"min":{"1min":0.81699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.752,"5min":1.8540000000000001,"15min":1.889},"last":0.82099999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.2809999999999999,"15min":1.2929999999999999},"min":{"1min":0.93899999999999995,"5min":0.59699999999999998,"15min":0.58799999999999997},"max":{"1min":1.667,"5min":1.736,"15min":1.9359999999999999},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3120000000000001,"15min":1.296},"min":{"1min":1.0469999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.571,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.3069999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.254,"15min":1.272},"min":{"1min":0.95899999999999996,"5min":0.73199999999999998,"15min":0.54200000000000004},"max":{"1min":1.8859999999999999,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.4099999999999999,"15min":1.409},"min":{"1min":1.0469999999999999,"5min":0.77000000000000002,"15min":0.68300000000000005},"max":{"1min":1.9550000000000001,"5min":1.956,"15min":1.956},"last":1.1719999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.296,"15min":1.2769999999999999},"min":{"1min":0.94799999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.849,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.325,"15min":1.321},"min":{"1min":0.93300000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.99,"5min":1.99,"15min":1.9970000000000001},"last":1.7190000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.3999999999999999,"15min":1.403},"min":{"1min":0.77000000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8759999999999999,"5min":1.891,"15min":2.048},"last":1.5},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4610000000000001,"15min":1.462},"min":{"1min":0.99099999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.71,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.651}]},{"osd":6,"last update":"Tue Nov 28 20:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.5700000000000001,"15min":1.554},"min":{"1min":1.25,"5min":0.85799999999999998,"15min":0.69499999999999995},"max":{"1min":1.9359999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.766},{"interface":"front","average":{"1min":1.48,"5min":1.4810000000000001,"15min":1.48},"min":{"1min":0.94099999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.7969999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575082,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334880,"statfs":{"total":96636764160,"available":96598917120,"internally_reserved":0,"allocated":1654784,"data_stored":828985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0469999999999999,"15min":1.095},"min":{"1min":0.745,"5min":0.58099999999999996,"15min":0.53500000000000003},"max":{"1min":1.5900000000000001,"5min":1.6319999999999999,"15min":1.7350000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.091,"15min":1.1299999999999999},"min":{"1min":0.69699999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4059999999999999,"5min":1.8069999999999999,"15min":1.8260000000000001},"last":0.73899999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.196,"15min":1.24},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.651,"5min":1.728,"15min":1.8069999999999999},"last":1.595},{"interface":"front","average":{"1min":1.385,"5min":1.292,"15min":1.349},"min":{"1min":1.1100000000000001,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.6759999999999999,"5min":1.7889999999999999,"15min":2.0169999999999999},"last":1.1359999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.2470000000000001,"15min":1.2829999999999999},"min":{"1min":0.88300000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":1.55},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.3200000000000001,"15min":1.361},"min":{"1min":1.2190000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.837,"5min":1.851,"15min":1.851},"last":1.3440000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.264,"15min":1.28},"min":{"1min":0.75,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.9299999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.222,"5min":1.258,"15min":1.2789999999999999},"min":{"1min":1.002,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.7450000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.1519999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.266,"15min":1.246},"min":{"1min":0.78300000000000003,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.6479999999999999,"5min":1.7529999999999999,"15min":2.089},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.3089999999999999,"15min":1.3049999999999999},"min":{"1min":0.92400000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.798,"5min":1.9179999999999999,"15min":2.2010000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.351,"15min":1.337},"min":{"1min":0.95999999999999996,"5min":0.88800000000000001,"15min":0.754},"max":{"1min":1.879,"5min":1.879,"15min":1.97},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.514,"15min":1.528},"min":{"1min":1.0669999999999999,"5min":0.93400000000000005,"15min":0.78000000000000003},"max":{"1min":1.901,"5min":1.9179999999999999,"15min":1.958},"last":1.6719999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4399999999999999,"15min":1.46},"min":{"1min":0.94999999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.849,"5min":1.9790000000000001,"15min":2.3759999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.4550000000000001,"15min":1.4530000000000001},"min":{"1min":1.1599999999999999,"5min":0.96099999999999997,"15min":0.92900000000000005},"max":{"1min":1.9259999999999999,"5min":1.9630000000000001,"15min":2.0739999999999998},"last":1.403}]}]},{"osd":5,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50824,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321016,"statfs":{"total":96636764160,"available":96584720384,"internally_reserved":0,"allocated":1634304,"data_stored":826769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1519999999999999,"15min":1.1639999999999999},"min":{"1min":0.77600000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.4860000000000002,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0660000000000001,"15min":1.159},"min":{"1min":0.60299999999999998,"5min":0.56200000000000006,"15min":0.441},"max":{"1min":1.587,"5min":1.631,"15min":1.829},"last":1.1140000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1160000000000001,"15min":1.1539999999999999},"min":{"1min":0.63100000000000001,"5min":0.61499999999999999,"15min":0.57399999999999995},"max":{"1min":1.619,"5min":1.768,"15min":1.768},"last":1.171},{"interface":"front","average":{"1min":1.323,"5min":1.2090000000000001,"15min":1.2170000000000001},"min":{"1min":0.82199999999999995,"5min":0.69299999999999995,"15min":0.55900000000000005},"max":{"1min":2.282,"5min":2.282,"15min":2.282},"last":1.6100000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.2689999999999999,"15min":1.2749999999999999},"min":{"1min":0.90500000000000003,"5min":0.63600000000000001,"15min":0.53200000000000003},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.26},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2110000000000001,"15min":1.278},"min":{"1min":0.85599999999999998,"5min":0.71799999999999997,"15min":0.66600000000000004},"max":{"1min":1.641,"5min":1.8220000000000001,"15min":1.958},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.28,"15min":1.3140000000000001},"min":{"1min":0.93400000000000005,"5min":0.77700000000000002,"15min":0.71099999999999997},"max":{"1min":1.6839999999999999,"5min":1.7290000000000001,"15min":1.9350000000000001},"last":1.216},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2609999999999999,"15min":1.268},"min":{"1min":0.91800000000000004,"5min":0.68300000000000005,"15min":0.64000000000000001},"max":{"1min":1.5920000000000001,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.397}]},{"osd":4,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3069999999999999,"15min":1.302},"min":{"1min":0.755,"5min":0.73799999999999999,"15min":0.67500000000000004},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.488,"5min":1.47,"15min":1.4399999999999999},"min":{"1min":1.0669999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.9249999999999998,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":1.3620000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.3280000000000001,"15min":1.329},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.5},"max":{"1min":1.6000000000000001,"5min":1.9630000000000001,"15min":2.093},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.474,"15min":1.4750000000000001},"min":{"1min":1.054,"5min":0.84999999999999998,"15min":0.69499999999999995},"max":{"1min":1.6819999999999999,"5min":1.998,"15min":1.998},"last":1.639}]},{"osd":7,"last update":"Tue Nov 28 20:53:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.4690000000000001,"15min":1.4350000000000001},"min":{"1min":1.0609999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":3.319,"5min":3.319,"15min":3.319},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.476,"15min":1.4910000000000001},"min":{"1min":0.99299999999999999,"5min":0.98099999999999998,"15min":0.60499999999999998},"max":{"1min":1.782,"5min":1.9790000000000001,"15min":2.1829999999999998},"last":1.3260000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36948,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334892,"statfs":{"total":96636764160,"available":96598929408,"internally_reserved":0,"allocated":1642496,"data_stored":826861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0620000000000001,"15min":1.1220000000000001},"min":{"1min":0.69199999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.409,"5min":1.605,"15min":1.879},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.095,"15min":1.117},"min":{"1min":0.86299999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.5449999999999999,"5min":1.6850000000000001,"15min":1.9750000000000001},"last":1.016}]},{"osd":1,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.228,"15min":1.2969999999999999},"min":{"1min":0.79600000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.923},"last":1.044},{"interface":"front","average":{"1min":1.327,"5min":1.2849999999999999,"15min":1.3420000000000001},"min":{"1min":0.91200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7070000000000001,"5min":1.845,"15min":2.2679999999999998},"last":0.95699999999999996}]},{"osd":2,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.196,"15min":1.21},"min":{"1min":0.90000000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.7250000000000001,"5min":1.78,"15min":1.78},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2549999999999999,"15min":1.2869999999999999},"min":{"1min":0.85499999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.623,"5min":1.837,"15min":2.6949999999999998},"last":1.163}]},{"osd":3,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.236,"15min":1.2789999999999999},"min":{"1min":1.0269999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.6519999999999999,"5min":1.9630000000000001,"15min":2.157},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3380000000000001,"15min":1.361},"min":{"1min":1.0249999999999999,"5min":0.5,"15min":0.5},"max":{"1min":1.673,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.4370000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.284,"15min":1.2849999999999999},"min":{"1min":0.79900000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.4279999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.288,"15min":1.292},"min":{"1min":0.84299999999999997,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.655,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.097}]},{"osd":6,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.393,"15min":1.3859999999999999},"min":{"1min":0.90000000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.8400000000000001,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.347},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.53,"15min":1.55},"min":{"1min":1.0980000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.8899999999999999,"5min":2.1349999999999998,"15min":3.7000000000000002},"last":1.478}]},{"osd":7,"last update":"Tue Nov 28 20:53:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4390000000000001,"15min":1.4510000000000001},"min":{"1min":1.056,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.9430000000000001,"5min":2.2490000000000001,"15min":2.4990000000000001},"last":1.2},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.518,"15min":1.512},"min":{"1min":1.115,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.0430000000000001,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.2190000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575083,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41096,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330744,"statfs":{"total":96636764160,"available":96594681856,"internally_reserved":0,"allocated":1638400,"data_stored":827351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.85299999999999998,"15min":0.90400000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.3300000000000001,"5min":1.6000000000000001,"15min":4.7210000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.02,"5min":0.93899999999999995,"15min":1.0469999999999999},"min":{"1min":0.59299999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.5740000000000001,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.92700000000000005,"15min":0.96999999999999997},"min":{"1min":0.501,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.351,"5min":1.9770000000000001,"15min":2.8460000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.256,"5min":1.0900000000000001,"15min":1.1859999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.7170000000000001,"5min":1.7190000000000001,"15min":3.7410000000000001},"last":0.79500000000000004}]},{"osd":2,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.1399999999999999,"15min":1.2290000000000001},"min":{"1min":0.54300000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":3.444},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.236,"15min":1.3300000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":3.7050000000000001},"last":0.626}]},{"osd":4,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.2869999999999999,"15min":1.3420000000000001},"min":{"1min":1.0209999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.956,"5min":1.956,"15min":2.9420000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.3440000000000001,"15min":1.377},"min":{"1min":0.80300000000000005,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":1.014}]},{"osd":5,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.3859999999999999,"15min":1.4450000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":3.7730000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.522,"5min":1.383,"15min":1.4339999999999999},"min":{"1min":0.69699999999999995,"5min":0.66800000000000004,"15min":0.53500000000000003},"max":{"1min":1.825,"5min":2.0110000000000001,"15min":3.1520000000000001},"last":0.877}]},{"osd":6,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.407,"15min":1.4670000000000001},"min":{"1min":0.77200000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.3100000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.585,"5min":1.47,"15min":1.5680000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.016,"5min":2.0840000000000001,"15min":3.661},"last":0.97599999999999998}]},{"osd":7,"last update":"Tue Nov 28 20:53:14 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.425,"15min":1.5309999999999999},"min":{"1min":0.753,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.847,"5min":1.923,"15min":3.266},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.5269999999999999,"15min":1.587},"min":{"1min":1.046,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.145,"5min":2.2080000000000002,"15min":2.3919999999999999},"last":1.0700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575083,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59144,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312696,"statfs":{"total":96636764160,"available":96576200704,"internally_reserved":0,"allocated":1638400,"data_stored":826803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.86699999999999999,"15min":0.90200000000000002},"min":{"1min":0.61699999999999999,"5min":0.373,"15min":0.28399999999999997},"max":{"1min":1.5569999999999999,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.024,"15min":1.0580000000000001},"min":{"1min":0.64500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.7270000000000001,"5min":4.8490000000000002,"15min":4.8490000000000002},"last":0.60699999999999998}]},{"osd":1,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.004,"15min":0.998},"min":{"1min":0.68700000000000006,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.7529999999999999,"5min":1.8160000000000001,"15min":3.2599999999999998},"last":0.498},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0609999999999999,"15min":1.137},"min":{"1min":0.72499999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5529999999999999,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.39000000000000001}]},{"osd":3,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.129,"15min":1.167},"min":{"1min":0.60899999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.665,"5min":4.9349999999999996,"15min":4.9349999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.252,"15min":1.29},"min":{"1min":0.75800000000000001,"5min":0.48499999999999999,"15min":0.48199999999999998},"max":{"1min":1.746,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":0.70899999999999996}]},{"osd":4,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4019999999999999,"15min":1.4159999999999999},"min":{"1min":1.03,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.784,"5min":3.6499999999999999,"15min":3.6499999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.468,"15min":1.4399999999999999},"min":{"1min":1.1519999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8360000000000001,"5min":4.6479999999999997,"15min":4.6479999999999997},"last":0.998}]},{"osd":5,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.3999999999999999,"15min":1.423},"min":{"1min":1.137,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.8160000000000001,"5min":2.492,"15min":2.492},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.49,"15min":1.5009999999999999},"min":{"1min":1.1950000000000001,"5min":0.92400000000000004,"15min":0.89100000000000001},"max":{"1min":1.8740000000000001,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":0.76800000000000002}]},{"osd":6,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.504,"15min":1.5149999999999999},"min":{"1min":1.127,"5min":0.747,"15min":0.747},"max":{"1min":1.8720000000000001,"5min":4.5270000000000001,"15min":4.5270000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":1.1080000000000001,"5min":0.89000000000000001,"15min":0.82499999999999996},"max":{"1min":1.972,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.083}]},{"osd":7,"last update":"Tue Nov 28 20:53:27 2023","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.494,"15min":1.532},"min":{"1min":0.99399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.1160000000000001,"5min":2.641,"15min":2.641},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5589999999999999,"15min":1.593},"min":{"1min":1.1659999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.9410000000000001,"5min":4.7530000000000001,"15min":4.7530000000000001},"last":0.81799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36940,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334900,"statfs":{"total":96636764160,"available":96598937600,"internally_reserved":0,"allocated":1638400,"data_stored":826803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.92400000000000004,"15min":0.91700000000000004},"min":{"1min":0.71699999999999997,"5min":0.311,"15min":0.311},"max":{"1min":1.742,"5min":1.8009999999999999,"15min":1.9259999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.014,"15min":1.083},"min":{"1min":0.79400000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.5669999999999999,"5min":2.028,"15min":2.028},"last":0.96099999999999997}]},{"osd":2,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.083,"15min":1.095},"min":{"1min":0.71399999999999997,"5min":0.51100000000000001,"15min":0.371},"max":{"1min":1.766,"5min":1.766,"15min":2.234},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.244,"5min":1.198,"15min":1.2090000000000001},"min":{"1min":0.56000000000000005,"5min":0.53700000000000003,"15min":0.52800000000000002},"max":{"1min":1.7629999999999999,"5min":1.9199999999999999,"15min":2.895},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.26,"15min":1.3069999999999999},"min":{"1min":0.75600000000000001,"5min":0.58199999999999996,"15min":0.46700000000000003},"max":{"1min":1.7450000000000001,"5min":1.8779999999999999,"15min":2.0459999999999998},"last":0.752},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.0820000000000001,"15min":1.1160000000000001},"min":{"1min":0.86299999999999999,"5min":0.45100000000000001,"15min":0.37},"max":{"1min":1.585,"5min":1.9790000000000001,"15min":2.3679999999999999},"last":0.77600000000000002}]},{"osd":4,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.3160000000000001,"15min":1.349},"min":{"1min":0.996,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":2.1150000000000002},"last":1.131},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.409,"15min":1.403},"min":{"1min":1.081,"5min":0.72399999999999998,"15min":0.71899999999999997},"max":{"1min":1.774,"5min":1.8819999999999999,"15min":2.6819999999999999},"last":1.0640000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.444,"15min":1.4410000000000001},"min":{"1min":0.86599999999999999,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.8839999999999999,"5min":2.0190000000000001,"15min":2.1840000000000002},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.474,"15min":1.494},"min":{"1min":1.1120000000000001,"5min":0.89700000000000002,"15min":0.89600000000000002},"max":{"1min":1.885,"5min":2.0219999999999998,"15min":2.294},"last":1.2889999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.393,"15min":1.4259999999999999},"min":{"1min":1.028,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.607},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.522,"15min":1.532},"min":{"1min":0.98599999999999999,"5min":0.81200000000000006,"15min":0.77100000000000002},"max":{"1min":2.0049999999999999,"5min":2.1059999999999999,"15min":2.3300000000000001},"last":1.4670000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:53:21 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.534,"15min":1.5649999999999999},"min":{"1min":1.198,"5min":0.80500000000000005,"15min":0.753},"max":{"1min":1.9750000000000001,"5min":2.1429999999999998,"15min":2.3359999999999999},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.4770000000000001,"15min":1.5229999999999999},"min":{"1min":1.1459999999999999,"5min":0.80500000000000005,"15min":0.72799999999999998},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.294},"last":1.6859999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41676,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330164,"statfs":{"total":96636764160,"available":96594087936,"internally_reserved":0,"allocated":2228224,"data_stored":1417137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.85599999999999998,"15min":0.89200000000000002},"min":{"1min":0.32800000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.208,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.0389999999999999,"15min":1.0900000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.57399999999999995}]},{"osd":2,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.004,"15min":1.101},"min":{"1min":0.41699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.504},{"interface":"front","average":{"1min":1.111,"5min":1.1060000000000001,"15min":1.1819999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.782,"5min":1.893,"15min":1.9159999999999999},"last":0.64700000000000002}]},{"osd":3,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.038,"15min":1.111},"min":{"1min":0.60599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.6390000000000002,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.347,"5min":1.1950000000000001,"15min":1.304},"min":{"1min":0.66000000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.55300000000000005}]},{"osd":4,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.343,"15min":1.4199999999999999},"min":{"1min":0.82299999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.712,"5min":1.804,"15min":2.012},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.337,"15min":1.387},"min":{"1min":0.95399999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":4.5119999999999996,"5min":4.5119999999999996,"15min":4.5119999999999996},"last":0.879}]},{"osd":5,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.389,"15min":1.419},"min":{"1min":0.89900000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.9379999999999997,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.4279999999999999,"15min":1.4830000000000001},"min":{"1min":0.97299999999999998,"5min":0.621,"15min":0.621},"max":{"1min":2.6139999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.0720000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.482,"15min":1.546},"min":{"1min":1.02,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.466,"15min":1.522},"min":{"1min":0.94999999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.0609999999999999,"5min":2.274,"15min":2.4209999999999998},"last":0.95399999999999996}]},{"osd":7,"last update":"Tue Nov 28 20:52:33 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.544,"15min":1.593},"min":{"1min":0.873,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1749999999999998,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.48,"15min":1.5429999999999999},"min":{"1min":1.0640000000000001,"5min":0.80300000000000005,"15min":0.73799999999999999},"max":{"1min":4.7359999999999998,"5min":4.7359999999999998,"15min":4.7359999999999998},"last":1.2909999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:53:40.864 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T20:53:40.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T20:53:41.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.143+0000 7f382759e640 1 Processor -- start 2023-11-28T20:53:41.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.143+0000 7f382759e640 1 -- start start 2023-11-28T20:53:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f382759e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f38280cbdd0 0x7f3828071320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f382759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3828071860 0x7f38280cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f382759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38280cd260 con 0x7f3828162820 2023-11-28T20:53:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f382759e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3828072110 con 0x7f38280cbdd0 2023-11-28T20:53:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f382759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38280723f0 con 0x7f3828071860 2023-11-28T20:53:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f382659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3828162820 0x7f38280cb6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55010/0 (socket says 172.21.15.45:55010) 2023-11-28T20:53:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f382659c640 1 -- 172.21.15.45:0/3820105140 learned_addr learned my addr 172.21.15.45:0/3820105140 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:41.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f3826d9d640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3828071860 0x7f38280cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f3825d9b640 1 --2- 172.21.15.45:0/3820105140 >> v2:172.21.15.190:3300/0 conn(0x7f38280cbdd0 0x7f3828071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.145+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232617207 0 0) 0x7f38280cd260 con 0x7f3828162820 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3814003520 con 0x7f3828162820 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 31886749 0 0) 0x7f3814003520 con 0x7f3828162820 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3828071860 msgr2=0x7f38280cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3828071860 0x7f38280cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 >> v2:172.21.15.190:3300/0 conn(0x7f38280cbdd0 msgr2=0x7f3828071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 --2- 172.21.15.45:0/3820105140 >> v2:172.21.15.190:3300/0 conn(0x7f38280cbdd0 0x7f3828071320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280ccc10 con 0x7f3828162820 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f38180019f0 con 0x7f3828162820 2023-11-28T20:53:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 >> v1:172.21.15.45:6789/0 conn(0x7f3828162820 legacy=0x7f38280cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3828071860 0x7f3814003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f38140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3814004d00 0x7f3814005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828072110 con 0x7f3814003fa0 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280723f0 con 0x7f3814004d00 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.146+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381400ba80 con 0x7f3828071860 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.147+0000 7f3825d9b640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f38140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.147+0000 7f382659c640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3828071860 0x7f3814003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.147+0000 7f3825d9b640 1 -- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3828071860 msgr2=0x7f3814003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.147+0000 7f3825d9b640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3828071860 0x7f3814003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.147+0000 7f3826d9d640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3814004d00 0x7f3814005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.147+0000 7f3825d9b640 1 -- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3814004d00 msgr2=0x7f3814005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:41.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.147+0000 7f3825d9b640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3814004d00 0x7f3814005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.147+0000 7f3825d9b640 1 -- 172.21.15.45:0/3820105140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3814003520 con 0x7f3814003fa0 2023-11-28T20:53:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.148+0000 7f3825d9b640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f38140047c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f381000ba30 tx=0x7f38140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c7e9110e11a42 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.149+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810032680 con 0x7f3814003fa0 2023-11-28T20:53:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.149+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38100327e0 con 0x7f3814003fa0 2023-11-28T20:53:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.149+0000 7f382559a640 1 -- 172.21.15.45:0/3820105140 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810032b70 con 0x7f3814003fa0 2023-11-28T20:53:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.149+0000 7f382759e640 1 -- 172.21.15.45:0/3820105140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 msgr2=0x7f38140047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.149+0000 7f382759e640 1 --2- 172.21.15.45:0/3820105140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f38140047c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f381000ba30 tx=0x7f38140057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.149+0000 7f3826d9d640 1 -- 172.21.15.45:0/3820105140 reap_dead start 2023-11-28T20:53:41.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.149+0000 7f382759e640 1 -- 172.21.15.45:0/3820105140 shutdown_connections 2023-11-28T20:53:41.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.149+0000 7f382759e640 1 -- 172.21.15.45:0/3820105140 >> 172.21.15.45:0/3820105140 conn(0x7f38280cec70 msgr2=0x7f38280cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:41.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.150+0000 7f382759e640 1 -- 172.21.15.45:0/3820105140 shutdown_connections 2023-11-28T20:53:41.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.150+0000 7f382759e640 1 -- 172.21.15.45:0/3820105140 wait complete. 2023-11-28T20:53:41.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.152+0000 7f382759e640 1 Processor -- start 2023-11-28T20:53:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.152+0000 7f382759e640 1 -- start start 2023-11-28T20:53:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.153+0000 7f382759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f3828147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.153+0000 7f382759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828157ad0 0x7f3828099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.153+0000 7f382759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38280b2b80 0x7f38280b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.153+0000 7f382759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828155df0 con 0x7f3814003fa0 2023-11-28T20:53:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.153+0000 7f382759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280b3540 con 0x7f3828157ad0 2023-11-28T20:53:41.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.153+0000 7f382759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38281483e0 con 0x7f38280b2b80 2023-11-28T20:53:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.153+0000 7f382659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f3828147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.153+0000 7f382659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f3828147ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49712/0 (socket says 172.21.15.45:49712) 2023-11-28T20:53:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.153+0000 7f382659c640 1 -- 172.21.15.45:0/3950006820 learned_addr learned my addr 172.21.15.45:0/3950006820 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f382659c640 1 -- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38280b2b80 msgr2=0x7f38280b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f382659c640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38280b2b80 0x7f38280b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f3825d9b640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828157ad0 0x7f3828099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f382659c640 1 -- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828157ad0 msgr2=0x7f3828099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f382659c640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828157ad0 0x7f3828099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f382659c640 1 -- 172.21.15.45:0/3950006820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382809ab10 con 0x7f3814003fa0 2023-11-28T20:53:41.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f382659c640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f3828147ea0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f3818007ab0 tx=0x7f381800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:41.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f381effd640 1 -- 172.21.15.45:0/3950006820 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818013420 con 0x7f3814003fa0 2023-11-28T20:53:41.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f381effd640 1 -- 172.21.15.45:0/3950006820 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38180135a0 con 0x7f3814003fa0 2023-11-28T20:53:41.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f381effd640 1 -- 172.21.15.45:0/3950006820 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818013950 con 0x7f3814003fa0 2023-11-28T20:53:41.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382809a270 con 0x7f3814003fa0 2023-11-28T20:53:41.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.154+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382809a830 con 0x7f3814003fa0 2023-11-28T20:53:41.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.156+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3828155f30 con 0x7f3814003fa0 2023-11-28T20:53:41.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.156+0000 7f381effd640 1 -- 172.21.15.45:0/3950006820 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3818013ab0 con 0x7f3814003fa0 2023-11-28T20:53:41.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.158+0000 7f381effd640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f380806af90 0x7f380806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.158+0000 7f381effd640 1 -- 172.21.15.45:0/3950006820 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f381808a0e0 con 0x7f3814003fa0 2023-11-28T20:53:41.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.158+0000 7f3825d9b640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f380806af90 0x7f380806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.160+0000 7f3825d9b640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f380806af90 0x7f380806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f381000ba00 tx=0x7f381003d040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:41.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.162+0000 7f381effd640 1 -- 172.21.15.45:0/3950006820 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3828155f30 con 0x7f3814003fa0 2023-11-28T20:53:41.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.427+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3828155f30 con 0x7f3814003fa0 2023-11-28T20:53:41.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.430+0000 7f381effd640 1 -- 172.21.15.45:0/3950006820 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3828155f30 con 0x7f3814003fa0 2023-11-28T20:53:41.432 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T20:53:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:53:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T20:53:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T20:53:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T20:53:41.432 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T20:53:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f380806af90 msgr2=0x7f380806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f380806af90 0x7f380806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f381000ba00 tx=0x7f381003d040 comp rx=0 tx=0).stop 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 msgr2=0x7f3828147ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f3828147ea0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f3818007ab0 tx=0x7f381800e530 comp rx=0 tx=0).stop 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 shutdown_connections 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f380806af90 0x7f380806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38280b2b80 0x7f38280b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828157ad0 0x7f3828099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 --2- 172.21.15.45:0/3950006820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3814003fa0 0x7f3828147ea0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 >> 172.21.15.45:0/3950006820 conn(0x7f38280cec70 msgr2=0x7f382806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 shutdown_connections 2023-11-28T20:53:41.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.433+0000 7f382759e640 1 -- 172.21.15.45:0/3950006820 wait complete. 2023-11-28T20:53:41.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:53:41.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.667+0000 7f5e69f8a640 1 Processor -- start 2023-11-28T20:53:41.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.667+0000 7f5e69f8a640 1 -- start start 2023-11-28T20:53:41.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.669+0000 7f5e69f8a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5e640b35b0 0x7f5e6415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.669+0000 7f5e69f8a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5e640b1340 0x7f5e6415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.669+0000 7f5e69f8a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6415bda0 con 0x7f5e6415b850 2023-11-28T20:53:41.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.669+0000 7f5e69f8a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6415ee40 con 0x7f5e640b35b0 2023-11-28T20:53:41.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.669+0000 7f5e69f8a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6415c080 con 0x7f5e640b1340 2023-11-28T20:53:41.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.669+0000 7f5e5affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5e6415b850 0x7f5e640b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55014/0 (socket says 172.21.15.45:55014) 2023-11-28T20:53:41.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.669+0000 7f5e5affd640 1 -- 172.21.15.45:0/3302358176 learned_addr learned my addr 172.21.15.45:0/3302358176 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:41.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.670+0000 7f5e637fe640 1 --2- 172.21.15.45:0/3302358176 >> v2:172.21.15.45:3300/0 conn(0x7f5e640b35b0 0x7f5e6415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.670+0000 7f5e62ffd640 1 -- 172.21.15.45:0/3302358176 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960182582 0 0) 0x7f5e6415bda0 con 0x7f5e6415b850 2023-11-28T20:53:41.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.670+0000 7f5e62ffd640 1 -- 172.21.15.45:0/3302358176 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e50003520 con 0x7f5e6415b850 2023-11-28T20:53:41.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.670+0000 7f5e63fff640 1 --2- 172.21.15.45:0/3302358176 >> v2:172.21.15.190:3300/0 conn(0x7f5e640b1340 0x7f5e6415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.670+0000 7f5e637fe640 1 -- 172.21.15.45:0/3302358176 >> v2:172.21.15.190:3300/0 conn(0x7f5e640b1340 msgr2=0x7f5e6415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:41.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.670+0000 7f5e637fe640 1 --2- 172.21.15.45:0/3302358176 >> v2:172.21.15.190:3300/0 conn(0x7f5e640b1340 0x7f5e6415f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.670+0000 7f5e637fe640 1 -- 172.21.15.45:0/3302358176 >> v1:172.21.15.45:6789/0 conn(0x7f5e6415b850 legacy=0x7f5e640b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.670+0000 7f5e637fe640 1 -- 172.21.15.45:0/3302358176 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6415c360 con 0x7f5e640b35b0 2023-11-28T20:53:41.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.671+0000 7f5e637fe640 1 --2- 172.21.15.45:0/3302358176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e640b35b0 0x7f5e6415ca10 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f5e64145050 tx=0x7f5e5402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f0a73edd28af17 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.672+0000 7f5e62ffd640 1 -- 172.21.15.45:0/3302358176 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5402cc80 con 0x7f5e640b35b0 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.672+0000 7f5e62ffd640 1 -- 172.21.15.45:0/3302358176 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5402cde0 con 0x7f5e640b35b0 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.672+0000 7f5e62ffd640 1 -- 172.21.15.45:0/3302358176 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54034640 con 0x7f5e640b35b0 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.672+0000 7f5e69f8a640 1 -- 172.21.15.45:0/3302358176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e640b35b0 msgr2=0x7f5e6415ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.672+0000 7f5e69f8a640 1 --2- 172.21.15.45:0/3302358176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e640b35b0 0x7f5e6415ca10 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f5e64145050 tx=0x7f5e5402bed0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.672+0000 7f5e69f8a640 1 -- 172.21.15.45:0/3302358176 shutdown_connections 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.672+0000 7f5e69f8a640 1 --2- 172.21.15.45:0/3302358176 >> v2:172.21.15.190:3300/0 conn(0x7f5e640b1340 0x7f5e6415f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.672+0000 7f5e69f8a640 1 --2- 172.21.15.45:0/3302358176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e640b35b0 0x7f5e6415ca10 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.672+0000 7f5e69f8a640 1 -- 172.21.15.45:0/3302358176 >> 172.21.15.45:0/3302358176 conn(0x7f5e6406a500 msgr2=0x7f5e6406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.673+0000 7f5e69f8a640 1 -- 172.21.15.45:0/3302358176 shutdown_connections 2023-11-28T20:53:41.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.673+0000 7f5e69f8a640 1 -- 172.21.15.45:0/3302358176 wait complete. 2023-11-28T20:53:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.674+0000 7f5e69f8a640 1 Processor -- start 2023-11-28T20:53:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.675+0000 7f5e69f8a640 1 -- start start 2023-11-28T20:53:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.675+0000 7f5e69f8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e640b1340 0x7f5e640cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e69f8a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e640b35b0 0x7f5e64157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e69f8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6415b850 0x7f5e64143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e69f8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e640cbdb0 con 0x7f5e6415b850 2023-11-28T20:53:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e69f8a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6415ad20 con 0x7f5e640b1340 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e69f8a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e641f3450 con 0x7f5e640b35b0 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e63fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6415b850 0x7f5e64143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e63fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6415b850 0x7f5e64143520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49726/0 (socket says 172.21.15.45:49726) 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e63fff640 1 -- 172.21.15.45:0/2150506747 learned_addr learned my addr 172.21.15.45:0/2150506747 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e637fe640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e640b1340 0x7f5e640cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e5affd640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e640b35b0 0x7f5e64157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e63fff640 1 -- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e640b35b0 msgr2=0x7f5e64157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e63fff640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e640b35b0 0x7f5e64157f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e63fff640 1 -- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e640b1340 msgr2=0x7f5e640cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e63fff640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e640b1340 0x7f5e640cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e63fff640 1 -- 172.21.15.45:0/2150506747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e54009a70 con 0x7f5e6415b850 2023-11-28T20:53:41.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.676+0000 7f5e637fe640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e640b1340 0x7f5e640cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:41.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.677+0000 7f5e63fff640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6415b850 0x7f5e64143520 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00c7c0 tx=0x7f5e5c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:41.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.677+0000 7f5e60ff9640 1 -- 172.21.15.45:0/2150506747 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c004220 con 0x7f5e6415b850 2023-11-28T20:53:41.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.677+0000 7f5e60ff9640 1 -- 172.21.15.45:0/2150506747 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5c004380 con 0x7f5e6415b850 2023-11-28T20:53:41.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.677+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e641f3730 con 0x7f5e6415b850 2023-11-28T20:53:41.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.677+0000 7f5e60ff9640 1 -- 172.21.15.45:0/2150506747 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c007840 con 0x7f5e6415b850 2023-11-28T20:53:41.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.677+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e641f3cf0 con 0x7f5e6415b850 2023-11-28T20:53:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.679+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e640733d0 con 0x7f5e6415b850 2023-11-28T20:53:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.679+0000 7f5e60ff9640 1 -- 172.21.15.45:0/2150506747 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5e5c0079a0 con 0x7f5e6415b850 2023-11-28T20:53:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.680+0000 7f5e60ff9640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5e4406af60 0x7f5e4406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.680+0000 7f5e60ff9640 1 -- 172.21.15.45:0/2150506747 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5e5c089610 con 0x7f5e6415b850 2023-11-28T20:53:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.681+0000 7f5e637fe640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5e4406af60 0x7f5e4406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.682+0000 7f5e637fe640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5e4406af60 0x7f5e4406d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5e54036000 tx=0x7f5e54030000 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:41.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.685+0000 7f5e60ff9640 1 -- 172.21.15.45:0/2150506747 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e5c0544b0 con 0x7f5e6415b850 2023-11-28T20:53:41.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.920+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e640cd3b0 con 0x7f5e6415b850 2023-11-28T20:53:41.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.921+0000 7f5e60ff9640 1 -- 172.21.15.45:0/2150506747 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f5e5c057ad0 con 0x7f5e6415b850 2023-11-28T20:53:41.922 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:41.922 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/544145878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":544145878},{"type":"v1","addr":"172.21.15.190:6836","nonce":544145878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2464692926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2464692926},{"type":"v1","addr":"172.21.15.45:6835","nonce":2464692926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2048455360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2048455360},{"type":"v1","addr":"172.21.15.190:6837","nonce":2048455360}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:53:35.282168+0000","modified":"2023-11-28T20:53:38.346980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2464465186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2464465186},{"type":"v1","addr":"172.21.15.45:6834","nonce":2464465186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-28T20:53:41.923 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 59 2023-11-28T20:53:41.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5e4406af60 msgr2=0x7f5e4406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5e4406af60 0x7f5e4406d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5e54036000 tx=0x7f5e54030000 comp rx=0 tx=0).stop 2023-11-28T20:53:41.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6415b850 msgr2=0x7f5e64143520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:41.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6415b850 0x7f5e64143520 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00c7c0 tx=0x7f5e5c00cc80 comp rx=0 tx=0).stop 2023-11-28T20:53:41.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 shutdown_connections 2023-11-28T20:53:41.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f5e4406af60 0x7f5e4406d420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6415b850 0x7f5e64143520 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e640b35b0 0x7f5e64157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 --2- 172.21.15.45:0/2150506747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e640b1340 0x7f5e640cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:41.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 >> 172.21.15.45:0/2150506747 conn(0x7f5e6406a500 msgr2=0x7f5e640aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:41.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 shutdown_connections 2023-11-28T20:53:41.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:41.924+0000 7f5e69f8a640 1 -- 172.21.15.45:0/2150506747 wait complete. 2023-11-28T20:53:42.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:53:35.282168+0000', 'modified': '2023-11-28T20:53:38.346980+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8103}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8103': {'gid': 8103, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6834/2464465186', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 2464465186}, {'type': 'v1', 'addr': '172.21.15.45:6834', 'nonce': 2464465186}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T20:53:42.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:53:42.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-28T20:53:42.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.586+0000 7f84d3fff640 1 Processor -- start 2023-11-28T20:53:42.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.586+0000 7f84d3fff640 1 -- start start 2023-11-28T20:53:42.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.588+0000 7f84d3fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f84d40cf610 0x7f84d40b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.588+0000 7f84d3fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d415d600 con 0x7f84d415d220 2023-11-28T20:53:42.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.588+0000 7f84d3fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d406c6b0 con 0x7f84d406c1d0 2023-11-28T20:53:42.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.588+0000 7f84d3fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d406c990 con 0x7f84d40cf610 2023-11-28T20:53:42.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.589+0000 7f84d2ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f84d415d220 0x7f84d40cef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55022/0 (socket says 172.21.15.45:55022) 2023-11-28T20:53:42.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.589+0000 7f84d2ffd640 1 -- 172.21.15.45:0/3574789219 learned_addr learned my addr 172.21.15.45:0/3574789219 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:42.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.589+0000 7f84d27fc640 1 --2- 172.21.15.45:0/3574789219 >> v2:172.21.15.190:3300/0 conn(0x7f84d40cf610 0x7f84d40b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.589+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313653808 0 0) 0x7f84d415d600 con 0x7f84d415d220 2023-11-28T20:53:42.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.589+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84bc003520 con 0x7f84d415d220 2023-11-28T20:53:42.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740393752 0 0) 0x7f84bc003520 con 0x7f84d415d220 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 >> v2:172.21.15.190:3300/0 conn(0x7f84d40cf610 msgr2=0x7f84d40b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 --2- 172.21.15.45:0/3574789219 >> v2:172.21.15.190:3300/0 conn(0x7f84d40cf610 0x7f84d40b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 >> v1:172.21.15.190:6789/0 conn(0x7f84d406c1d0 legacy=0x7f84d40704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d406cc70 con 0x7f84d415d220 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f84c4001670 con 0x7f84d415d220 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 >> v1:172.21.15.45:6789/0 conn(0x7f84d415d220 legacy=0x7f84d40cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84bc003970 0x7f84bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc0047c0 0x7f84bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84bc005180 0x7f84bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d406c990 con 0x7f84bc003970 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc00c120 con 0x7f84bc0047c0 2023-11-28T20:53:42.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc00c2e0 con 0x7f84bc005180 2023-11-28T20:53:42.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.590+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84bc003970 0x7f84bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.591+0000 7f84d27fc640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc0047c0 0x7f84bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.591+0000 7f84d37fe640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84bc005180 0x7f84bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.591+0000 7f84d2ffd640 1 -- 172.21.15.45:0/3574789219 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84bc005180 msgr2=0x7f84bc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.591+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84bc005180 0x7f84bc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.591+0000 7f84d2ffd640 1 -- 172.21.15.45:0/3574789219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc0047c0 msgr2=0x7f84bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.591+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc0047c0 0x7f84bc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.591+0000 7f84d2ffd640 1 -- 172.21.15.45:0/3574789219 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc003520 con 0x7f84bc003970 2023-11-28T20:53:42.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.591+0000 7f84d37fe640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84bc005180 0x7f84bc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:42.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.591+0000 7f84d37fe640 1 -- 172.21.15.45:0/3574789219 reap_dead start 2023-11-28T20:53:42.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.593+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84bc003970 0x7f84bc003d90 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c240 tx=0x7f84bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38791a984bd40f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.593+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4005ad0 con 0x7f84bc003970 2023-11-28T20:53:42.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.593+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4005c50 con 0x7f84bc003970 2023-11-28T20:53:42.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.593+0000 7f84d1ffb640 1 -- 172.21.15.45:0/3574789219 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c402c8b0 con 0x7f84bc003970 2023-11-28T20:53:42.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.594+0000 7f84d3fff640 1 -- 172.21.15.45:0/3574789219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84bc003970 msgr2=0x7f84bc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.594+0000 7f84d3fff640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84bc003970 0x7f84bc003d90 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c240 tx=0x7f84bc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.594+0000 7f84d3fff640 1 -- 172.21.15.45:0/3574789219 shutdown_connections 2023-11-28T20:53:42.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.594+0000 7f84d3fff640 1 --2- 172.21.15.45:0/3574789219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84bc003970 0x7f84bc003d90 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.594+0000 7f84d3fff640 1 -- 172.21.15.45:0/3574789219 >> 172.21.15.45:0/3574789219 conn(0x7f84d406a710 msgr2=0x7f84d406ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:42.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.594+0000 7f84d3fff640 1 -- 172.21.15.45:0/3574789219 shutdown_connections 2023-11-28T20:53:42.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.594+0000 7f84d3fff640 1 -- 172.21.15.45:0/3574789219 wait complete. 2023-11-28T20:53:42.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.596+0000 7f84d3fff640 1 Processor -- start 2023-11-28T20:53:42.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.596+0000 7f84d3fff640 1 -- start start 2023-11-28T20:53:42.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.597+0000 7f84d3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc003970 0x7f84d4158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.597+0000 7f84d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4159980 0x7f84d4160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.597+0000 7f84d3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84d40b1a50 0x7f84d40b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.597+0000 7f84d3fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40b2410 con 0x7f84d4159980 2023-11-28T20:53:42.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.597+0000 7f84d3fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40ce210 con 0x7f84bc003970 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.597+0000 7f84d3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4159600 con 0x7f84d40b1a50 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.597+0000 7f84d27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4159980 0x7f84d4160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.597+0000 7f84d27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4159980 0x7f84d4160010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49756/0 (socket says 172.21.15.45:49756) 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.597+0000 7f84d27fc640 1 -- 172.21.15.45:0/2342079068 learned_addr learned my addr 172.21.15.45:0/2342079068 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d37fe640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84d40b1a50 0x7f84d40b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc003970 0x7f84d4158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d27fc640 1 -- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84d40b1a50 msgr2=0x7f84d40b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d27fc640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84d40b1a50 0x7f84d40b1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d27fc640 1 -- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc003970 msgr2=0x7f84d4158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d27fc640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc003970 0x7f84d4158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d27fc640 1 -- 172.21.15.45:0/2342079068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d415d850 con 0x7f84d4159980 2023-11-28T20:53:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d37fe640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84d40b1a50 0x7f84d40b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:42.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc003970 0x7f84d4158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:42.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.598+0000 7f84d27fc640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4159980 0x7f84d4160010 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f84c8011aa0 tx=0x7f84c8011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.599+0000 7f84bbfff640 1 -- 172.21.15.45:0/2342079068 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c8009680 con 0x7f84d4159980 2023-11-28T20:53:42.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.599+0000 7f84bbfff640 1 -- 172.21.15.45:0/2342079068 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c800fb80 con 0x7f84d4159980 2023-11-28T20:53:42.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.599+0000 7f84bbfff640 1 -- 172.21.15.45:0/2342079068 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c8014440 con 0x7f84d4159980 2023-11-28T20:53:42.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.599+0000 7f84d3fff640 1 -- 172.21.15.45:0/2342079068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d415db30 con 0x7f84d4159980 2023-11-28T20:53:42.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.599+0000 7f84d3fff640 1 -- 172.21.15.45:0/2342079068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d415e0f0 con 0x7f84d4159980 2023-11-28T20:53:42.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.601+0000 7f84bbfff640 1 -- 172.21.15.45:0/2342079068 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f84c80145a0 con 0x7f84d4159980 2023-11-28T20:53:42.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.603+0000 7f84bbfff640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84c006af60 0x7f84c006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.603+0000 7f84bbfff640 1 -- 172.21.15.45:0/2342079068 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f84c808a0e0 con 0x7f84d4159980 2023-11-28T20:53:42.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.603+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84c006af60 0x7f84c006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.603+0000 7f84dc618740 1 Processor -- start 2023-11-28T20:53:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.604+0000 7f84dc618740 1 -- start start 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84c006af60 0x7f84c006d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84c4034000 tx=0x7f84c4009030 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84dc618740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba6070 0x556f58ba6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84dc618740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba0d70 0x556f58ba5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84dc618740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba57b0 0x556f58be9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84dc618740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f58b04880 con 0x556f58ba6070 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84dc618740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f58b13ae0 con 0x556f58ba0d70 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84dc618740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f58b111d0 con 0x556f58ba57b0 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba6070 0x556f58ba6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba6070 0x556f58ba6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49770/0 (socket says 172.21.15.45:49770) 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84d2ffd640 1 -- 172.21.15.45:0/2640894451 learned_addr learned my addr 172.21.15.45:0/2640894451 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84d37fe640 1 --2- 172.21.15.45:0/2640894451 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba57b0 0x556f58be9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.605+0000 7f84d27fc640 1 --2- 172.21.15.45:0/2640894451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba0d70 0x556f58ba5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.606+0000 7f84d2ffd640 1 -- 172.21.15.45:0/2640894451 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba57b0 msgr2=0x556f58be9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.606+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2640894451 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba57b0 0x556f58be9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.606+0000 7f84d2ffd640 1 -- 172.21.15.45:0/2640894451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba0d70 msgr2=0x556f58ba5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.606+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2640894451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba0d70 0x556f58ba5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.606+0000 7f84d2ffd640 1 -- 172.21.15.45:0/2640894451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f58ad4860 con 0x556f58ba6070 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.606+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2640894451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba6070 0x556f58ba6450 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f84c4001930 tx=0x7f84c4007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22606519c399ec6e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.606+0000 7f84b97fa640 1 -- 172.21.15.45:0/2640894451 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c403c040 con 0x556f58ba6070 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.606+0000 7f84b97fa640 1 -- 172.21.15.45:0/2640894451 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4004e20 con 0x556f58ba6070 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.606+0000 7f84b97fa640 1 -- 172.21.15.45:0/2640894451 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4007270 con 0x556f58ba6070 2023-11-28T20:53:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.607+0000 7f84dc618740 1 -- 172.21.15.45:0/2640894451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba6070 msgr2=0x556f58ba6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.607+0000 7f84dc618740 1 --2- 172.21.15.45:0/2640894451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba6070 0x556f58ba6450 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f84c4001930 tx=0x7f84c4007000 comp rx=0 tx=0).stop 2023-11-28T20:53:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.607+0000 7f84dc618740 1 -- 172.21.15.45:0/2640894451 shutdown_connections 2023-11-28T20:53:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.607+0000 7f84dc618740 1 --2- 172.21.15.45:0/2640894451 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba57b0 0x556f58be9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.607+0000 7f84dc618740 1 --2- 172.21.15.45:0/2640894451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba0d70 0x556f58ba5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.607+0000 7f84dc618740 1 --2- 172.21.15.45:0/2640894451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba6070 0x556f58ba6450 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.607+0000 7f84dc618740 1 -- 172.21.15.45:0/2640894451 >> 172.21.15.45:0/2640894451 conn(0x556f58b67c70 msgr2=0x556f58bebb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.607+0000 7f84dc618740 1 -- 172.21.15.45:0/2640894451 shutdown_connections 2023-11-28T20:53:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.607+0000 7f84dc618740 1 -- 172.21.15.45:0/2640894451 wait complete. 2023-11-28T20:53:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.608+0000 7f84dc618740 1 -- start start 2023-11-28T20:53:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.608+0000 7f84dc618740 1 Processor -- start 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.609+0000 7f84dc618740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba0d70 0x556f58bce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.609+0000 7f84dc618740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba57b0 0x556f58be81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.609+0000 7f84dc618740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba6070 0x556f58bf3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.609+0000 7f84dc618740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f58b0cac0 con 0x556f58ba57b0 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.609+0000 7f84dc618740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f58aa92f0 con 0x556f58ba6070 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.609+0000 7f84dc618740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f58ac6c70 con 0x556f58ba0d70 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.609+0000 7f84dc618740 10 client.0 _ll_register_callbacks cb 0x556f58acb3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba57b0 0x556f58be81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba57b0 0x556f58be81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49780/0 (socket says 172.21.15.45:49780) 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d27fc640 1 -- 172.21.15.45:0/2618366352 learned_addr learned my addr 172.21.15.45:0/2618366352 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba0d70 0x556f58bce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d37fe640 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba6070 0x556f58bf3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d27fc640 1 -- 172.21.15.45:0/2618366352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba0d70 msgr2=0x556f58bce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d27fc640 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba0d70 0x556f58bce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d27fc640 1 -- 172.21.15.45:0/2618366352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba6070 msgr2=0x556f58bf3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d27fc640 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba6070 0x556f58bf3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d27fc640 1 -- 172.21.15.45:0/2618366352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc003520 con 0x556f58ba57b0 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba0d70 0x556f58bce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84d27fc640 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba57b0 0x556f58be81c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f84c801c000 tx=0x7f84c8058de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbe6a6e9f82a4c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.610+0000 7f84b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84b27fc640 1 -- 172.21.15.45:0/2618366352 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c8059100 con 0x556f58ba57b0 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84b27fc640 1 -- 172.21.15.45:0/2618366352 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f84c8058190 con 0x556f58ba57b0 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84dc618740 1 -- 172.21.15.45:0/2618366352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556f58b05aa0 con 0x556f58ba57b0 2023-11-28T20:53:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84b27fc640 1 -- 172.21.15.45:0/2618366352 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84c8058370 con 0x556f58ba57b0 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84dc618740 10 client.8241 fetch_fsmap learned FSMap version 59 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84dc618740 1 -- 172.21.15.45:0/2618366352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x556f58b0b140 con 0x556f58ba57b0 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84b27fc640 1 -- 172.21.15.45:0/2618366352 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556f58b05aa0 con 0x556f58ba57b0 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84b27fc640 1 -- 172.21.15.45:0/2618366352 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 59) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f84c801b020 con 0x556f58ba57b0 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84dc618740 10 client.8241 fetch_fsmap finished waiting for FSMap version 59 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84dc618740 10 client.8241 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84dc618740 20 client.8241 populate_metadata read hostname 'smithi045' 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84dc618740 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] conn(0x556f58bf22d0 0x556f58c08e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84dc618740 4 client.8241 mds_command: new command op to 8103 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.611+0000 7f84dc618740 1 -- 172.21.15.45:0/2618366352 --> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f58b057c0 con 0x556f58bf22d0 2023-11-28T20:53:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.612+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] conn(0x556f58bf22d0 0x556f58c08e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.612+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] conn(0x556f58bf22d0 0x556f58c08e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b78ea1818be2bd68 server_cookie=f06c6c5d7acef447 in_seq=0 out_seq=0 2023-11-28T20:53:42.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.612+0000 7f84b27fc640 10 client.8241 ms_handle_connect on v2:172.21.15.45:6832/2464465186 2023-11-28T20:53:42.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84b27fc640 1 -- 172.21.15.45:0/2618366352 <== mds.0 v2:172.21.15.45:6832/2464465186 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556f58b057c0 con 0x556f58bf22d0 2023-11-28T20:53:42.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84b27fc640 10 client.8241 handle_command_reply: tid=0 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84dc618740 1 client.8241 shutdown 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84dc618740 1 -- 172.21.15.45:0/2618366352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba57b0 msgr2=0x556f58be81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84dc618740 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba57b0 0x556f58be81c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f84c801c000 tx=0x7f84c8058de0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84dc618740 1 -- 172.21.15.45:0/2618366352 shutdown_connections 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84dc618740 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba6070 0x556f58bf3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84dc618740 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] conn(0x556f58bf22d0 0x556f58c08e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84dc618740 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba57b0 0x556f58be81c0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84dc618740 1 --2- 172.21.15.45:0/2618366352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba0d70 0x556f58bce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84b27fc640 0 client.8241 ms_handle_reset on v2:172.21.15.45:6832/2464465186 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.613+0000 7f84dc618740 1 -- 172.21.15.45:0/2618366352 >> 172.21.15.45:0/2618366352 conn(0x556f58b67c70 msgr2=0x556f58be8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.614+0000 7f84dc618740 1 -- 172.21.15.45:0/2618366352 shutdown_connections 2023-11-28T20:53:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.614+0000 7f84dc618740 1 -- 172.21.15.45:0/2618366352 wait complete. 2023-11-28T20:53:42.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.614+0000 7f84dc618740 20 client.8241 trim_cache size 0 max 16384 2023-11-28T20:53:42.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.631+0000 7f84dc618740 1 Processor -- start 2023-11-28T20:53:42.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.631+0000 7f84dc618740 1 -- start start 2023-11-28T20:53:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.632+0000 7f84dc618740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58bebd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.632+0000 7f84dc618740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba57b0 0x556f58bec290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.632+0000 7f84dc618740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 0x556f58c4f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.632+0000 7f84dc618740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5886ff20 con 0x556f58ba0d70 2023-11-28T20:53:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.632+0000 7f84dc618740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f58bef160 con 0x556f58ba57b0 2023-11-28T20:53:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.632+0000 7f84dc618740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f58bef710 con 0x556f58ba6070 2023-11-28T20:53:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.632+0000 7f84d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58bebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.632+0000 7f84d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58bebd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49784/0 (socket says 172.21.15.45:49784) 2023-11-28T20:53:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.632+0000 7f84d2ffd640 1 -- 172.21.15.45:0/1633757935 learned_addr learned my addr 172.21.15.45:0/1633757935 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:42.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.633+0000 7f84d37fe640 1 --2- 172.21.15.45:0/1633757935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 0x556f58c4f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.633+0000 7f84d27fc640 1 --2- 172.21.15.45:0/1633757935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba57b0 0x556f58bec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.633+0000 7f84d2ffd640 1 -- 172.21.15.45:0/1633757935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 msgr2=0x556f58c4f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.633+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/1633757935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 0x556f58c4f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.633+0000 7f84d2ffd640 1 -- 172.21.15.45:0/1633757935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba57b0 msgr2=0x556f58bec290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.633+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/1633757935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba57b0 0x556f58bec290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.633+0000 7f84d2ffd640 1 -- 172.21.15.45:0/1633757935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f58aa8b80 con 0x556f58ba0d70 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.633+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/1633757935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58bebd50 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c210 tx=0x7f84c402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84be7077d2d5f15 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.633+0000 7f84b9ffb640 1 -- 172.21.15.45:0/1633757935 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4052030 con 0x556f58ba0d70 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84b9ffb640 1 -- 172.21.15.45:0/1633757935 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4007140 con 0x556f58ba0d70 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84b9ffb640 1 -- 172.21.15.45:0/1633757935 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c403c040 con 0x556f58ba0d70 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84dc618740 1 -- 172.21.15.45:0/1633757935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 msgr2=0x556f58bebd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84dc618740 1 --2- 172.21.15.45:0/1633757935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58bebd50 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c210 tx=0x7f84c402e000 comp rx=0 tx=0).stop 2023-11-28T20:53:42.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84dc618740 1 -- 172.21.15.45:0/1633757935 shutdown_connections 2023-11-28T20:53:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84dc618740 1 --2- 172.21.15.45:0/1633757935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 0x556f58c4f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84dc618740 1 --2- 172.21.15.45:0/1633757935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba57b0 0x556f58bec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84dc618740 1 --2- 172.21.15.45:0/1633757935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58bebd50 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84dc618740 1 -- 172.21.15.45:0/1633757935 >> 172.21.15.45:0/1633757935 conn(0x556f58b67c70 msgr2=0x556f58ba6450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84dc618740 1 -- 172.21.15.45:0/1633757935 shutdown_connections 2023-11-28T20:53:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.634+0000 7f84dc618740 1 -- 172.21.15.45:0/1633757935 wait complete. 2023-11-28T20:53:42.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.635+0000 7f84dc618740 1 -- start start 2023-11-28T20:53:42.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.635+0000 7f84dc618740 1 Processor -- start 2023-11-28T20:53:42.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.636+0000 7f84dc618740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58c4e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.636+0000 7f84dc618740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba57b0 0x556f58bd18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.636+0000 7f84dc618740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 0x556f58c08ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.636+0000 7f84dc618740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f58ba0300 con 0x556f58ba0d70 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.636+0000 7f84dc618740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f58c09990 con 0x556f58ba57b0 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.636+0000 7f84dc618740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f58bd0b70 con 0x556f58ba6070 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.636+0000 7f84dc618740 10 client.0 _ll_register_callbacks cb 0x7f84c8058600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58c4e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58c4e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49800/0 (socket says 172.21.15.45:49800) 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d2ffd640 1 -- 172.21.15.45:0/2761433435 learned_addr learned my addr 172.21.15.45:0/2761433435 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d2ffd640 1 -- 172.21.15.45:0/2761433435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 msgr2=0x556f58c08ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d37fe640 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 0x556f58c08ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 0x556f58c08ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d2ffd640 1 -- 172.21.15.45:0/2761433435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba57b0 msgr2=0x556f58bd18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba57b0 0x556f58bd18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d2ffd640 1 -- 172.21.15.45:0/2761433435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f58ad4860 con 0x556f58ba0d70 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84d2ffd640 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58c4e9b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f84c402e040 tx=0x7f84c402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e4ecc583563d32 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.637+0000 7f84b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84b27fc640 1 -- 172.21.15.45:0/2761433435 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c403c040 con 0x556f58ba0d70 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84b27fc640 1 -- 172.21.15.45:0/2761433435 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f84c4055040 con 0x556f58ba0d70 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84dc618740 1 -- 172.21.15.45:0/2761433435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84c4004e20 con 0x556f58ba0d70 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84b27fc640 1 -- 172.21.15.45:0/2761433435 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84c4001530 con 0x556f58ba0d70 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84dc618740 10 client.8247 fetch_fsmap learned FSMap version 59 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84dc618740 1 -- 172.21.15.45:0/2761433435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x556f58ace170 con 0x556f58ba0d70 2023-11-28T20:53:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84b27fc640 1 -- 172.21.15.45:0/2761433435 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4004e20 con 0x556f58ba0d70 2023-11-28T20:53:42.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84b27fc640 1 -- 172.21.15.45:0/2761433435 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 59) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f84c403f030 con 0x556f58ba0d70 2023-11-28T20:53:42.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84dc618740 10 client.8247 fetch_fsmap finished waiting for FSMap version 59 2023-11-28T20:53:42.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84dc618740 10 client.8247 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T20:53:42.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84dc618740 20 client.8247 populate_metadata read hostname 'smithi045' 2023-11-28T20:53:42.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84dc618740 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] conn(0x556f58bead70 0x556f58bebd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84dc618740 4 client.8247 mds_command: new command op to 8103 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:53:42.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.638+0000 7f84dc618740 1 -- 172.21.15.45:0/2761433435 --> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556f5880d6c0 con 0x556f58bead70 2023-11-28T20:53:42.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.639+0000 7f84d27fc640 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] conn(0x556f58bead70 0x556f58bebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.639+0000 7f84b27fc640 10 client.8247 ms_handle_connect on v2:172.21.15.45:6832/2464465186 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.639+0000 7f84d27fc640 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] conn(0x556f58bead70 0x556f58bebd50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd543f07d69f264d server_cookie=7529b4df359e6788 in_seq=0 out_seq=0 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84b27fc640 1 -- 172.21.15.45:0/2761433435 <== mds.0 v2:172.21.15.45:6832/2464465186 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x556f5880d6c0 con 0x556f58bead70 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84b27fc640 10 client.8247 handle_command_reply: tid=0 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 client.8247 shutdown 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 -- 172.21.15.45:0/2761433435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 msgr2=0x556f58c4e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58c4e9b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f84c402e040 tx=0x7f84c402e070 comp rx=0 tx=0).stop 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 -- 172.21.15.45:0/2761433435 shutdown_connections 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x556f58ba6070 0x556f58c08ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f58ba57b0 0x556f58bd18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.45:6832/2464465186,v1:172.21.15.45:6834/2464465186] conn(0x556f58bead70 0x556f58bebd50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 --2- 172.21.15.45:0/2761433435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556f58ba0d70 0x556f58c4e9b0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 -- 172.21.15.45:0/2761433435 >> 172.21.15.45:0/2761433435 conn(0x556f58b67c70 msgr2=0x556f58ba19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:42.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84b27fc640 0 client.8247 ms_handle_reset on v2:172.21.15.45:6832/2464465186 2023-11-28T20:53:42.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 -- 172.21.15.45:0/2761433435 shutdown_connections 2023-11-28T20:53:42.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 1 -- 172.21.15.45:0/2761433435 wait complete. 2023-11-28T20:53:42.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.640+0000 7f84dc618740 20 client.8247 trim_cache size 0 max 16384 2023-11-28T20:53:42.645 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 8103, 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.810794896, 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 59, 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 133, 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 133, 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 13.168184334999999 2023-11-28T20:53:42.646 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:53:42.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 -- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84c006af60 msgr2=0x7f84c006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84c006af60 0x7f84c006d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84c4034000 tx=0x7f84c4009030 comp rx=0 tx=0).stop 2023-11-28T20:53:42.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 -- 172.21.15.45:0/2342079068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4159980 msgr2=0x7f84d4160010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4159980 0x7f84d4160010 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f84c8011aa0 tx=0x7f84c8011f60 comp rx=0 tx=0).stop 2023-11-28T20:53:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 -- 172.21.15.45:0/2342079068 shutdown_connections 2023-11-28T20:53:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f84c006af60 0x7f84c006d420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84d40b1a50 0x7f84d40b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4159980 0x7f84d4160010 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 --2- 172.21.15.45:0/2342079068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84bc003970 0x7f84d4158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 -- 172.21.15.45:0/2342079068 >> 172.21.15.45:0/2342079068 conn(0x7f84d406a710 msgr2=0x7f84d40711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 -- 172.21.15.45:0/2342079068 shutdown_connections 2023-11-28T20:53:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.641+0000 7f84d3fff640 1 -- 172.21.15.45:0/2342079068 wait complete. 2023-11-28T20:53:42.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:53:42.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.886+0000 7f52f359e640 1 Processor -- start 2023-11-28T20:53:42.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.887+0000 7f52f359e640 1 -- start start 2023-11-28T20:53:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.888+0000 7f52f359e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52f415d550 0x7f52f40cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.888+0000 7f52f359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f40d1c80 0x7f52f40b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.888+0000 7f52f359e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f40d2230 con 0x7f52f40d0100 2023-11-28T20:53:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.888+0000 7f52f359e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f40d0590 con 0x7f52f415d550 2023-11-28T20:53:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.888+0000 7f52f359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f40d2510 con 0x7f52f40d1c80 2023-11-28T20:53:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.888+0000 7f52f1d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f52f40d0100 0x7f52f40d1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55032/0 (socket says 172.21.15.45:55032) 2023-11-28T20:53:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.888+0000 7f52f1d9b640 1 -- 172.21.15.45:0/2785122883 learned_addr learned my addr 172.21.15.45:0/2785122883 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f2d9d640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f40d1c80 0x7f52f40b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f259c640 1 --2- 172.21.15.45:0/2785122883 >> v2:172.21.15.190:3300/0 conn(0x7f52f415d550 0x7f52f40cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075962341 0 0) 0x7f52f40d2230 con 0x7f52f40d0100 2023-11-28T20:53:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e0003520 con 0x7f52f40d0100 2023-11-28T20:53:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 970998153 0 0) 0x7f52e0003520 con 0x7f52f40d0100 2023-11-28T20:53:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f40d1c80 msgr2=0x7f52f40b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f40d1c80 0x7f52f40b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 >> v2:172.21.15.190:3300/0 conn(0x7f52f415d550 msgr2=0x7f52f40cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 --2- 172.21.15.45:0/2785122883 >> v2:172.21.15.190:3300/0 conn(0x7f52f415d550 0x7f52f40cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f40d27f0 con 0x7f52f40d0100 2023-11-28T20:53:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f52dc001690 con 0x7f52f40d0100 2023-11-28T20:53:42.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 >> v1:172.21.15.45:6789/0 conn(0x7f52f40d0100 legacy=0x7f52f40d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f40d1c80 0x7f52e0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e0003f40 0x7f52e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52e0004d00 0x7f52e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f40d0590 con 0x7f52e0003f40 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f40d2510 con 0x7f52e0004d00 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.889+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e000b980 con 0x7f52f40d1c80 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.890+0000 7f52f1d9b640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e0003f40 0x7f52e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.890+0000 7f52f259c640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f40d1c80 0x7f52e0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.890+0000 7f52f2d9d640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52e0004d00 0x7f52e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.890+0000 7f52f1d9b640 1 -- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f40d1c80 msgr2=0x7f52e0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.890+0000 7f52f1d9b640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52f40d1c80 0x7f52e0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.890+0000 7f52f1d9b640 1 -- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52e0004d00 msgr2=0x7f52e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.890+0000 7f52f1d9b640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52e0004d00 0x7f52e0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.890+0000 7f52f1d9b640 1 -- 172.21.15.45:0/2785122883 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e0003520 con 0x7f52e0003f40 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.892+0000 7f52f1d9b640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e0003f40 0x7f52e00047c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00c190 tx=0x7f52e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e392b8595cc2904b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.892+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc005ad0 con 0x7f52e0003f40 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.892+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52dc005c50 con 0x7f52e0003f40 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.892+0000 7f52f159a640 1 -- 172.21.15.45:0/2785122883 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc02c7e0 con 0x7f52e0003f40 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.892+0000 7f52f359e640 1 -- 172.21.15.45:0/2785122883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e0003f40 msgr2=0x7f52e00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.892+0000 7f52f359e640 1 --2- 172.21.15.45:0/2785122883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e0003f40 0x7f52e00047c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00c190 tx=0x7f52e00057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.892+0000 7f52f2d9d640 1 -- 172.21.15.45:0/2785122883 reap_dead start 2023-11-28T20:53:42.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.893+0000 7f52f359e640 1 -- 172.21.15.45:0/2785122883 shutdown_connections 2023-11-28T20:53:42.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.893+0000 7f52f359e640 1 -- 172.21.15.45:0/2785122883 >> 172.21.15.45:0/2785122883 conn(0x7f52f406ac70 msgr2=0x7f52f40b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:42.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.893+0000 7f52f359e640 1 -- 172.21.15.45:0/2785122883 shutdown_connections 2023-11-28T20:53:42.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.893+0000 7f52f359e640 1 -- 172.21.15.45:0/2785122883 wait complete. 2023-11-28T20:53:42.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.895+0000 7f52f359e640 1 Processor -- start 2023-11-28T20:53:42.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.895+0000 7f52f359e640 1 -- start start 2023-11-28T20:53:42.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52e0003f40 0x7f52f41f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f4151c20 0x7f52f40b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f40c7e40 0x7f52f40c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f40c7a20 con 0x7f52f40c7e40 2023-11-28T20:53:42.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f40c9d70 con 0x7f52f4151c20 2023-11-28T20:53:42.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f41f7150 con 0x7f52e0003f40 2023-11-28T20:53:42.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f40c7e40 0x7f52f40c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f40c7e40 0x7f52f40c82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49808/0 (socket says 172.21.15.45:49808) 2023-11-28T20:53:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f2d9d640 1 -- 172.21.15.45:0/3926336474 learned_addr learned my addr 172.21.15.45:0/3926336474 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f1d9b640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f4151c20 0x7f52f40b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.896+0000 7f52f259c640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52e0003f40 0x7f52f41f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52f2d9d640 1 -- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52e0003f40 msgr2=0x7f52f41f6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52f2d9d640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52e0003f40 0x7f52f41f6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52f2d9d640 1 -- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f4151c20 msgr2=0x7f52f40b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52f2d9d640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f4151c20 0x7f52f40b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52f2d9d640 1 -- 172.21.15.45:0/3926336474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f40c6da0 con 0x7f52f40c7e40 2023-11-28T20:53:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52f259c640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52e0003f40 0x7f52f41f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52f2d9d640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f40c7e40 0x7f52f40c82c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f52f40c9ec0 tx=0x7f52ec011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52eaffd640 1 -- 172.21.15.45:0/3926336474 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec004500 con 0x7f52f40c7e40 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52eaffd640 1 -- 172.21.15.45:0/3926336474 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec003610 con 0x7f52f40c7e40 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f40c7080 con 0x7f52f40c7e40 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52eaffd640 1 -- 172.21.15.45:0/3926336474 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec0039c0 con 0x7f52f40c7e40 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.897+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f40c7640 con 0x7f52f40c7e40 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.899+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f40c7b60 con 0x7f52f40c7e40 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.900+0000 7f52eaffd640 1 -- 172.21.15.45:0/3926336474 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f52ec003b20 con 0x7f52f40c7e40 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.901+0000 7f52eaffd640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f52d406af60 0x7f52d406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.901+0000 7f52eaffd640 1 -- 172.21.15.45:0/3926336474 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f52ec089be0 con 0x7f52f40c7e40 2023-11-28T20:53:42.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.902+0000 7f52f259c640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f52d406af60 0x7f52d406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:42.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.903+0000 7f52f259c640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f52d406af60 0x7f52d406d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f52e4007440 tx=0x7f52e400b040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:42.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:42.906+0000 7f52eaffd640 1 -- 172.21.15.45:0/3926336474 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52f40c7b60 con 0x7f52f40c7e40 2023-11-28T20:53:43.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.141+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f52f40c7b60 con 0x7f52f40c7e40 2023-11-28T20:53:43.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.142+0000 7f52eaffd640 1 -- 172.21.15.45:0/3926336474 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v59) v1 ==== 53+0+73 (secure 0 0 0) 0x7f52f40c7b60 con 0x7f52f40c7e40 2023-11-28T20:53:43.143 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T20:53:43.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.144+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f52d406af60 msgr2=0x7f52d406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:43.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.144+0000 7f52f359e640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f52d406af60 0x7f52d406d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f52e4007440 tx=0x7f52e400b040 comp rx=0 tx=0).stop 2023-11-28T20:53:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f40c7e40 msgr2=0x7f52f40c82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:43.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f40c7e40 0x7f52f40c82c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f52f40c9ec0 tx=0x7f52ec011920 comp rx=0 tx=0).stop 2023-11-28T20:53:43.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 shutdown_connections 2023-11-28T20:53:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f52d406af60 0x7f52d406d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52f40c7e40 0x7f52f40c82c0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52f4151c20 0x7f52f40b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 --2- 172.21.15.45:0/3926336474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52e0003f40 0x7f52f41f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 >> 172.21.15.45:0/3926336474 conn(0x7f52f406ac70 msgr2=0x7f52f415ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 shutdown_connections 2023-11-28T20:53:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:43.145+0000 7f52f359e640 1 -- 172.21.15.45:0/3926336474 wait complete. 2023-11-28T20:53:43.166 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T20:53:43.166 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:53:43.167 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:53:43.167 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T20:53:43.167 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:53:43.167 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T20:53:43.167 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T20:53:43.167 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:53:43.167 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:53:43.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:43.168 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:53:43.226 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:53:43.226 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:53:43.227 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:53:43.228 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:53:43.228 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 41119sec preferred_lft 41119sec 2023-11-28T20:53:43.228 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:53:43.228 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:53:43.228 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:53:43.228 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:53:43.228 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:53:43.228 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T20:53:43.229 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:53:43.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:43.230 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:53:43.327 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:53:43.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:43.331 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:53:43.390 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:53:43.391 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:53:43.391 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:53:43.391 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:53:43.391 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:53:43.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:43.392 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:53:43.392 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:53:43.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:53:43.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:53:43.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:53:43.393 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:53:43.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:43.591 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:53:43.646 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T20:53:43.646 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T20:53:43.647 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:53:43.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:43.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:53:43.751 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:53:43.751 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T20:53:43.810 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:53:43.810 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:53:43.810 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:53:43.870 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:53:43.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:43.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:53:44.019 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:53:44.019 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:53:44.019 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:53:44.041 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:53:44.041 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:53:44.041 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:53:44.041 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:53:44.041 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:53:44.067 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:53:44.067 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:53:44.067 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:53:44.067 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:53:44.067 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:53:44.067 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:53:44.067 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:53:44.068 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:53:44.069 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T20:53:44.069 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:53:44.069 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:53:44.069 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:53:44.069 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:53:44.074 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:53:44.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:44.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:53:44.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:44.162 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:53:44.232 INFO:teuthology.orchestra.run.smithi045.stdout:client6907 2023-11-28T20:53:44.232 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:53:44.233 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:53:44.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T20:53:44.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.520+0000 7f2c23dd0640 1 Processor -- start 2023-11-28T20:53:44.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.520+0000 7f2c23dd0640 1 -- start start 2023-11-28T20:53:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.522+0000 7f2c23dd0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c1c0b2460 0x7f2c1c06af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.522+0000 7f2c23dd0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c09a150 0x7f2c1c0cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.522+0000 7f2c23dd0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c0cc010 con 0x7f2c1c06b450 2023-11-28T20:53:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.522+0000 7f2c23dd0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1c06b970 con 0x7f2c1c0b2460 2023-11-28T20:53:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.522+0000 7f2c23dd0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0cb6e0 con 0x7f2c1c09a150 2023-11-28T20:53:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.522+0000 7f2c21b6c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c1c0b2460 0x7f2c1c06af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.522+0000 7f2c21b6c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c1c0b2460 0x7f2c1c06af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58358/0 (socket says 172.21.15.45:58358) 2023-11-28T20:53:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.522+0000 7f2c21b6c640 1 -- 172.21.15.45:0/1330691334 learned_addr learned my addr 172.21.15.45:0/1330691334 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.523+0000 7f2c2236d640 1 --2- 172.21.15.45:0/1330691334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c09a150 0x7f2c1c0cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.523+0000 7f2c21b6c640 1 -- 172.21.15.45:0/1330691334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c09a150 msgr2=0x7f2c1c0cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.523+0000 7f2c21b6c640 1 --2- 172.21.15.45:0/1330691334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c09a150 0x7f2c1c0cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.523+0000 7f2c21b6c640 1 -- 172.21.15.45:0/1330691334 >> v1:172.21.15.190:6789/0 conn(0x7f2c1c06b450 legacy=0x7f2c1c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.523+0000 7f2c21b6c640 1 -- 172.21.15.45:0/1330691334 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0cb9c0 con 0x7f2c1c0b2460 2023-11-28T20:53:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.523+0000 7f2c2236d640 1 --2- 172.21.15.45:0/1330691334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c09a150 0x7f2c1c0cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.524+0000 7f2c21b6c640 1 --2- 172.21.15.45:0/1330691334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c0b2460 0x7f2c1c06af10 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009e50 tx=0x7f2c0c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcba27fb50480ffa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.524+0000 7f2c20b6a640 1 -- 172.21.15.45:0/1330691334 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c02ca70 con 0x7f2c1c0b2460 2023-11-28T20:53:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.524+0000 7f2c20b6a640 1 -- 172.21.15.45:0/1330691334 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c02cbd0 con 0x7f2c1c0b2460 2023-11-28T20:53:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.524+0000 7f2c20b6a640 1 -- 172.21.15.45:0/1330691334 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0343f0 con 0x7f2c1c0b2460 2023-11-28T20:53:44.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.525+0000 7f2c23dd0640 1 -- 172.21.15.45:0/1330691334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c0b2460 msgr2=0x7f2c1c06af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:44.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.525+0000 7f2c23dd0640 1 --2- 172.21.15.45:0/1330691334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c0b2460 0x7f2c1c06af10 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009e50 tx=0x7f2c0c02bed0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.525+0000 7f2c23dd0640 1 -- 172.21.15.45:0/1330691334 shutdown_connections 2023-11-28T20:53:44.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.525+0000 7f2c23dd0640 1 --2- 172.21.15.45:0/1330691334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c09a150 0x7f2c1c0cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.525+0000 7f2c23dd0640 1 --2- 172.21.15.45:0/1330691334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c0b2460 0x7f2c1c06af10 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.525+0000 7f2c23dd0640 1 -- 172.21.15.45:0/1330691334 >> 172.21.15.45:0/1330691334 conn(0x7f2c1c0b2ae0 msgr2=0x7f2c1c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:44.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.525+0000 7f2c23dd0640 1 -- 172.21.15.45:0/1330691334 shutdown_connections 2023-11-28T20:53:44.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.525+0000 7f2c23dd0640 1 -- 172.21.15.45:0/1330691334 wait complete. 2023-11-28T20:53:44.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.527+0000 7f2c23dd0640 1 Processor -- start 2023-11-28T20:53:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.527+0000 7f2c23dd0640 1 -- start start 2023-11-28T20:53:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.528+0000 7f2c23dd0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c1c06b450 0x7f2c1c0c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.528+0000 7f2c23dd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c09a150 0x7f2c1c15c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.528+0000 7f2c23dd0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c0b2460 0x7f2c1c0b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.528+0000 7f2c23dd0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0c9c30 con 0x7f2c1c09a150 2023-11-28T20:53:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.528+0000 7f2c23dd0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c15c930 con 0x7f2c1c06b450 2023-11-28T20:53:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.528+0000 7f2c23dd0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0c90b0 con 0x7f2c1c0b2460 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.528+0000 7f2c2136b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c09a150 0x7f2c1c15c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2136b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c09a150 0x7f2c1c15c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58362/0 (socket says 172.21.15.45:58362) 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2136b640 1 -- 172.21.15.45:0/3298152190 learned_addr learned my addr 172.21.15.45:0/3298152190 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2236d640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c0b2460 0x7f2c1c0b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c21b6c640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c1c06b450 0x7f2c1c0c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2136b640 1 -- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c0b2460 msgr2=0x7f2c1c0b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2136b640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c0b2460 0x7f2c1c0b5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2136b640 1 -- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c1c06b450 msgr2=0x7f2c1c0c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2136b640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c1c06b450 0x7f2c1c0c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2136b640 1 -- 172.21.15.45:0/3298152190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c009b30 con 0x7f2c1c09a150 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2236d640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c0b2460 0x7f2c1c0b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:44.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c21b6c640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c1c06b450 0x7f2c1c0c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:44.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.529+0000 7f2c2136b640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c09a150 0x7f2c1c15c3f0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000b3e0 tx=0x7f2c1000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:44.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.530+0000 7f2c1a7fc640 1 -- 172.21.15.45:0/3298152190 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10011830 con 0x7f2c1c09a150 2023-11-28T20:53:44.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.530+0000 7f2c1a7fc640 1 -- 172.21.15.45:0/3298152190 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c100119b0 con 0x7f2c1c09a150 2023-11-28T20:53:44.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.530+0000 7f2c1a7fc640 1 -- 172.21.15.45:0/3298152190 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10011d60 con 0x7f2c1c09a150 2023-11-28T20:53:44.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.530+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c0c9390 con 0x7f2c1c09a150 2023-11-28T20:53:44.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.530+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c0c9950 con 0x7f2c1c09a150 2023-11-28T20:53:44.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.531+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c06bd30 con 0x7f2c1c09a150 2023-11-28T20:53:44.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.532+0000 7f2c1a7fc640 1 -- 172.21.15.45:0/3298152190 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2c1001a400 con 0x7f2c1c09a150 2023-11-28T20:53:44.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.533+0000 7f2c1a7fc640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f2c0806af60 0x7f2c0806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:44.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.533+0000 7f2c1a7fc640 1 -- 172.21.15.45:0/3298152190 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2c10089ab0 con 0x7f2c1c09a150 2023-11-28T20:53:44.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.534+0000 7f2c21b6c640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f2c0806af60 0x7f2c0806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:44.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.535+0000 7f2c21b6c640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f2c0806af60 0x7f2c0806d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c035000 tx=0x7f2c0c030000 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.538+0000 7f2c1a7fc640 1 -- 172.21.15.45:0/3298152190 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c10054880 con 0x7f2c1c09a150 2023-11-28T20:53:44.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.755+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 --> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c1c0b2840 con 0x7f2c0806af60 2023-11-28T20:53:44.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.759+0000 7f2c1a7fc640 1 -- 172.21.15.45:0/3298152190 <== mgr.6655 v2:172.21.15.190:6832/3959324769 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2c0c040450 con 0x7f2c0806af60 2023-11-28T20:53:44.763 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:53:44.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:53:44.763 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T20:53:44.765 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:53:44.765 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:53:44.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.761+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f2c0806af60 msgr2=0x7f2c0806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.761+0000 7f2c23dd0640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f2c0806af60 0x7f2c0806d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c035000 tx=0x7f2c0c030000 comp rx=0 tx=0).stop 2023-11-28T20:53:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.761+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c09a150 msgr2=0x7f2c1c15c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.761+0000 7f2c23dd0640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c09a150 0x7f2c1c15c3f0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000b3e0 tx=0x7f2c1000b8a0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.762+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 shutdown_connections 2023-11-28T20:53:44.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.762+0000 7f2c23dd0640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f2c0806af60 0x7f2c0806d420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.762+0000 7f2c23dd0640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c1c0b2460 0x7f2c1c0b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.762+0000 7f2c23dd0640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c1c09a150 0x7f2c1c15c3f0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.762+0000 7f2c23dd0640 1 --2- 172.21.15.45:0/3298152190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c1c06b450 0x7f2c1c0c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:44.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.762+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 >> 172.21.15.45:0/3298152190 conn(0x7f2c1c0b2ae0 msgr2=0x7f2c1c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:44.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.762+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 shutdown_connections 2023-11-28T20:53:44.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:44.762+0000 7f2c23dd0640 1 -- 172.21.15.45:0/3298152190 wait complete. 2023-11-28T20:53:44.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T20:53:45.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.011+0000 7f3c0b5fb640 1 Processor -- start 2023-11-28T20:53:45.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.011+0000 7f3c0b5fb640 1 -- start start 2023-11-28T20:53:45.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.013+0000 7f3c0b5fb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 0x7f3c0406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.013+0000 7f3c0b5fb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3c0415f720 0x7f3c040717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.013+0000 7f3c0b5fb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0415ecd0 con 0x7f3c0415e840 2023-11-28T20:53:45.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.013+0000 7f3c0b5fb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c04071ce0 con 0x7f3c0415f720 2023-11-28T20:53:45.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.013+0000 7f3c0b5fb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0415fbd0 con 0x7f3c0406ac10 2023-11-28T20:53:45.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.013+0000 7f3c09397640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3c0415e840 0x7f3c0406a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:32992/0 (socket says 172.21.15.45:32992) 2023-11-28T20:53:45.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.013+0000 7f3c09397640 1 -- 172.21.15.45:0/2507697166 learned_addr learned my addr 172.21.15.45:0/2507697166 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:45.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.013+0000 7f3c09b98640 1 --2- 172.21.15.45:0/2507697166 >> v2:172.21.15.45:3300/0 conn(0x7f3c0415f720 0x7f3c040717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.014+0000 7f3c08b96640 1 --2- 172.21.15.45:0/2507697166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 0x7f3c0406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.014+0000 7f3bfbfff640 1 -- 172.21.15.45:0/2507697166 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528613009 0 0) 0x7f3c0415ecd0 con 0x7f3c0415e840 2023-11-28T20:53:45.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.014+0000 7f3bfbfff640 1 -- 172.21.15.45:0/2507697166 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf0003520 con 0x7f3c0415e840 2023-11-28T20:53:45.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.014+0000 7f3c09b98640 1 -- 172.21.15.45:0/2507697166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 msgr2=0x7f3c0406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:45.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.014+0000 7f3c09b98640 1 --2- 172.21.15.45:0/2507697166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 0x7f3c0406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.014+0000 7f3c09b98640 1 -- 172.21.15.45:0/2507697166 >> v1:172.21.15.45:6789/0 conn(0x7f3c0415e840 legacy=0x7f3c0406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.014+0000 7f3c09b98640 1 -- 172.21.15.45:0/2507697166 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0415f290 con 0x7f3c0415f720 2023-11-28T20:53:45.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.015+0000 7f3c09b98640 1 --2- 172.21.15.45:0/2507697166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 0x7f3c040717a0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009dc0 tx=0x7f3bfc02be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ec22d222144049 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:45.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.015+0000 7f3bfbfff640 1 -- 172.21.15.45:0/2507697166 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc02cc30 con 0x7f3c0415f720 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3bfbfff640 1 -- 172.21.15.45:0/2507697166 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc02cd90 con 0x7f3c0415f720 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3bfbfff640 1 -- 172.21.15.45:0/2507697166 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc034660 con 0x7f3c0415f720 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/2507697166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 msgr2=0x7f3c040717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3c0b5fb640 1 --2- 172.21.15.45:0/2507697166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 0x7f3c040717a0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009dc0 tx=0x7f3bfc02be80 comp rx=0 tx=0).stop 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/2507697166 shutdown_connections 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3c0b5fb640 1 --2- 172.21.15.45:0/2507697166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 0x7f3c040717a0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3c0b5fb640 1 --2- 172.21.15.45:0/2507697166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 0x7f3c0406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/2507697166 >> 172.21.15.45:0/2507697166 conn(0x7f3c040ced60 msgr2=0x7f3c040cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/2507697166 shutdown_connections 2023-11-28T20:53:45.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.016+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/2507697166 wait complete. 2023-11-28T20:53:45.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.018+0000 7f3c0b5fb640 1 Processor -- start 2023-11-28T20:53:45.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.018+0000 7f3c0b5fb640 1 -- start start 2023-11-28T20:53:45.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.019+0000 7f3c0b5fb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 0x7f3c040cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.019+0000 7f3c0b5fb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c0415e840 0x7f3c0415d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.019+0000 7f3c0b5fb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 0x7f3c040c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c0b5fb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040cab90 con 0x7f3c0415f720 2023-11-28T20:53:45.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c0b5fb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040c9f80 con 0x7f3c0415e840 2023-11-28T20:53:45.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c0b5fb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040ca260 con 0x7f3c0406ac10 2023-11-28T20:53:45.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c09b98640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 0x7f3c040c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c09b98640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 0x7f3c040c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58374/0 (socket says 172.21.15.45:58374) 2023-11-28T20:53:45.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c09b98640 1 -- 172.21.15.45:0/3755781162 learned_addr learned my addr 172.21.15.45:0/3755781162 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:45.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c09397640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 0x7f3c040cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c08b96640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c0415e840 0x7f3c0415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c09b98640 1 -- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 msgr2=0x7f3c040cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c09b98640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 0x7f3c040cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c09b98640 1 -- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c0415e840 msgr2=0x7f3c0415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c09b98640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c0415e840 0x7f3c0415d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.020+0000 7f3c09b98640 1 -- 172.21.15.45:0/3755781162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc009a70 con 0x7f3c0415f720 2023-11-28T20:53:45.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.021+0000 7f3c09b98640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 0x7f3c040c9910 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc037040 tx=0x7f3bfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:45.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.021+0000 7f3bf9ffb640 1 -- 172.21.15.45:0/3755781162 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc040650 con 0x7f3c0415f720 2023-11-28T20:53:45.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.021+0000 7f3bf9ffb640 1 -- 172.21.15.45:0/3755781162 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc0407b0 con 0x7f3c0415f720 2023-11-28T20:53:45.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.021+0000 7f3bf9ffb640 1 -- 172.21.15.45:0/3755781162 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc040b60 con 0x7f3c0415f720 2023-11-28T20:53:45.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.021+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c040ca540 con 0x7f3c0415f720 2023-11-28T20:53:45.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.021+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c040b46f0 con 0x7f3c0415f720 2023-11-28T20:53:45.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.023+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c040cad40 con 0x7f3c0415f720 2023-11-28T20:53:45.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.024+0000 7f3bf9ffb640 1 -- 172.21.15.45:0/3755781162 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 57) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3bfc040cc0 con 0x7f3c0415f720 2023-11-28T20:53:45.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.024+0000 7f3bf9ffb640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f3be806af60 0x7f3be806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.024+0000 7f3bf9ffb640 1 -- 172.21.15.45:0/3755781162 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3bfc0ac580 con 0x7f3c0415f720 2023-11-28T20:53:45.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.025+0000 7f3c09397640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f3be806af60 0x7f3be806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.026+0000 7f3c09397640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f3be806af60 0x7f3be806d420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c000011e0 tx=0x7f3c00009040 comp rx=0 tx=0).ready entity=mgr.6655 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:45.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.029+0000 7f3bf9ffb640 1 -- 172.21.15.45:0/3755781162 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c040cad40 con 0x7f3c0415f720 2023-11-28T20:53:45.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.271+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f3c04066f10 con 0x7f3c0415f720 2023-11-28T20:53:45.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.447+0000 7f3bf9ffb640 1 -- 172.21.15.45:0/3755781162 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v58) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3bfc0773d0 con 0x7f3c0415f720 2023-11-28T20:53:45.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f3be806af60 msgr2=0x7f3be806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f3be806af60 0x7f3be806d420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c000011e0 tx=0x7f3c00009040 comp rx=0 tx=0).stop 2023-11-28T20:53:45.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 msgr2=0x7f3c040c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 0x7f3c040c9910 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc037040 tx=0x7f3bfc004a40 comp rx=0 tx=0).stop 2023-11-28T20:53:45.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 shutdown_connections 2023-11-28T20:53:45.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f3be806af60 0x7f3be806d420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0415f720 0x7f3c040c9910 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c0415e840 0x7f3c0415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 --2- 172.21.15.45:0/3755781162 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c0406ac10 0x7f3c040cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.451+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 >> 172.21.15.45:0/3755781162 conn(0x7f3c040ced60 msgr2=0x7f3c040ad410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:45.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.452+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 shutdown_connections 2023-11-28T20:53:45.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.452+0000 7f3c0b5fb640 1 -- 172.21.15.45:0/3755781162 wait complete. 2023-11-28T20:53:45.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T20:53:45.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.701+0000 7f2916b99640 1 Processor -- start 2023-11-28T20:53:45.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.701+0000 7f2916b99640 1 -- start start 2023-11-28T20:53:45.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.703+0000 7f2916b99640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f29100b1d40 0x7f29100d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.703+0000 7f2916b99640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29100b90a0 con 0x7f29100d0040 2023-11-28T20:53:45.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.703+0000 7f2916b99640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29100d1920 con 0x7f29100b2290 2023-11-28T20:53:45.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.703+0000 7f2916b99640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29100b8ae0 con 0x7f29100b1d40 2023-11-28T20:53:45.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.703+0000 7f2915b97640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f29100d0040 0x7f29100d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:32994/0 (socket says 172.21.15.45:32994) 2023-11-28T20:53:45.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.703+0000 7f2915b97640 1 -- 172.21.15.45:0/2188889099 learned_addr learned my addr 172.21.15.45:0/2188889099 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2916398640 1 --2- 172.21.15.45:0/2188889099 >> v2:172.21.15.190:3300/0 conn(0x7f29100b1d40 0x7f29100d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595707167 0 0) 0x7f29100b90a0 con 0x7f29100d0040 2023-11-28T20:53:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28fc003520 con 0x7f29100d0040 2023-11-28T20:53:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1461046827 0 0) 0x7f28fc003520 con 0x7f29100d0040 2023-11-28T20:53:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 >> v2:172.21.15.190:3300/0 conn(0x7f29100b1d40 msgr2=0x7f29100d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 --2- 172.21.15.45:0/2188889099 >> v2:172.21.15.190:3300/0 conn(0x7f29100b1d40 0x7f29100d24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 >> v1:172.21.15.190:6789/0 conn(0x7f29100b2290 legacy=0x7f29100d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29100b8dc0 con 0x7f29100d0040 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f290c001670 con 0x7f29100d0040 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 >> v1:172.21.15.45:6789/0 conn(0x7f29100d0040 legacy=0x7f29100d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28fc0039c0 0x7f28fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0047c0 0x7f28fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f28fc005180 0x7f28fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100b8ae0 con 0x7f28fc0039c0 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc00c060 con 0x7f28fc0047c0 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.704+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc00c220 con 0x7f28fc005180 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2915b97640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28fc0039c0 0x7f28fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2915396640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0047c0 0x7f28fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2916398640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f28fc005180 0x7f28fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2915b97640 1 -- 172.21.15.45:0/2188889099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f28fc005180 msgr2=0x7f28fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2915b97640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f28fc005180 0x7f28fc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2915b97640 1 -- 172.21.15.45:0/2188889099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0047c0 msgr2=0x7f28fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2915b97640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0047c0 0x7f28fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2915b97640 1 -- 172.21.15.45:0/2188889099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc003520 con 0x7f28fc0039c0 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2915396640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0047c0 0x7f28fc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.705+0000 7f2916398640 1 -- 172.21.15.45:0/2188889099 reap_dead start 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.706+0000 7f2915b97640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28fc0039c0 0x7f28fc003de0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f290c00c0e0 tx=0x7f28fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4733828ff0ef352 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.707+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c005ad0 con 0x7f28fc0039c0 2023-11-28T20:53:45.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.707+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c005c30 con 0x7f28fc0039c0 2023-11-28T20:53:45.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.707+0000 7f2914b95640 1 -- 172.21.15.45:0/2188889099 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c007070 con 0x7f28fc0039c0 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.707+0000 7f2916b99640 1 -- 172.21.15.45:0/2188889099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28fc0039c0 msgr2=0x7f28fc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.707+0000 7f2916b99640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28fc0039c0 0x7f28fc003de0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f290c00c0e0 tx=0x7f28fc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.708+0000 7f2916b99640 1 -- 172.21.15.45:0/2188889099 shutdown_connections 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.708+0000 7f2916b99640 1 --2- 172.21.15.45:0/2188889099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28fc0039c0 0x7f28fc003de0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.708+0000 7f2916b99640 1 -- 172.21.15.45:0/2188889099 >> 172.21.15.45:0/2188889099 conn(0x7f2910070f40 msgr2=0x7f2910071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.708+0000 7f2916b99640 1 -- 172.21.15.45:0/2188889099 shutdown_connections 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.708+0000 7f2916b99640 1 -- 172.21.15.45:0/2188889099 wait complete. 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.709+0000 7f2916b99640 1 Processor -- start 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.710+0000 7f2916b99640 1 -- start start 2023-11-28T20:53:45.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916b99640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0039c0 0x7f291014a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916b99640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29101579c0 0x7f29100c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916b99640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29100c8090 0x7f29100cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916b99640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c76c0 con 0x7f29100c8090 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916b99640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c6ab0 con 0x7f28fc0039c0 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916b99640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c6d90 con 0x7f29101579c0 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916398640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29100c8090 0x7f29100cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916398640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29100c8090 0x7f29100cbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58434/0 (socket says 172.21.15.45:58434) 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916398640 1 -- 172.21.15.45:0/3408985206 learned_addr learned my addr 172.21.15.45:0/3408985206 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2915b97640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0039c0 0x7f291014a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2915396640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29101579c0 0x7f29100c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:45.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916398640 1 -- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29101579c0 msgr2=0x7f29100c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916398640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29101579c0 0x7f29100c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916398640 1 -- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0039c0 msgr2=0x7f291014a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916398640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0039c0 0x7f291014a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.711+0000 7f2916398640 1 -- 172.21.15.45:0/3408985206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29100c7070 con 0x7f29100c8090 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.712+0000 7f2916398640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29100c8090 0x7f29100cbe70 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f29100b4c30 tx=0x7f290800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.712+0000 7f29067fc640 1 -- 172.21.15.45:0/3408985206 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908016620 con 0x7f29100c8090 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.712+0000 7f29067fc640 1 -- 172.21.15.45:0/3408985206 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29080167a0 con 0x7f29100c8090 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.712+0000 7f29067fc640 1 -- 172.21.15.45:0/3408985206 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908016b50 con 0x7f29100c8090 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.712+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29100c7350 con 0x7f29100c8090 2023-11-28T20:53:45.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.712+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2910157470 con 0x7f29100c8090 2023-11-28T20:53:45.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.714+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29100c7870 con 0x7f29100c8090 2023-11-28T20:53:45.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.714+0000 7f29067fc640 1 -- 172.21.15.45:0/3408985206 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 58) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f2908016cb0 con 0x7f29100c8090 2023-11-28T20:53:45.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.715+0000 7f29067fc640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f28f406afb0 0x7f28f406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:45.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.715+0000 7f29067fc640 1 -- 172.21.15.45:0/3408985206 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f290808a7c0 con 0x7f29100c8090 2023-11-28T20:53:45.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.715+0000 7f2915b97640 1 -- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f28f406afb0 msgr2=0x7f28f406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3959324769 2023-11-28T20:53:45.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.715+0000 7f2915b97640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f28f406afb0 0x7f28f406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:45.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.719+0000 7f29067fc640 1 -- 172.21.15.45:0/3408985206 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29100c7870 con 0x7f29100c8090 2023-11-28T20:53:45.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.916+0000 7f2915b97640 1 -- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f28f406afb0 msgr2=0x7f28f406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3959324769 2023-11-28T20:53:45.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.916+0000 7f2915b97640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f28f406afb0 0x7f28f406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:53:45.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.928+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2910071b30 con 0x7f29100c8090 2023-11-28T20:53:45.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.934+0000 7f29067fc640 1 -- 172.21.15.45:0/3408985206 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2908055470 con 0x7f29100c8090 2023-11-28T20:53:45.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.936+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f28f406afb0 msgr2=0x7f28f406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:45.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.936+0000 7f2916b99640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f28f406afb0 0x7f28f406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.936+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29100c8090 msgr2=0x7f29100cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:45.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.936+0000 7f2916b99640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29100c8090 0x7f29100cbe70 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f29100b4c30 tx=0x7f290800f520 comp rx=0 tx=0).stop 2023-11-28T20:53:45.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.937+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 shutdown_connections 2023-11-28T20:53:45.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.937+0000 7f2916b99640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f28f406afb0 0x7f28f406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.937+0000 7f2916b99640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29100c8090 0x7f29100cbe70 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.937+0000 7f2916b99640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29101579c0 0x7f29100c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.937+0000 7f2916b99640 1 --2- 172.21.15.45:0/3408985206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28fc0039c0 0x7f291014a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:45.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.937+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 >> 172.21.15.45:0/3408985206 conn(0x7f2910070f40 msgr2=0x7f29100b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:45.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.937+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 shutdown_connections 2023-11-28T20:53:45.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:45.937+0000 7f2916b99640 1 -- 172.21.15.45:0/3408985206 wait complete. 2023-11-28T20:53:45.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T20:53:45.984 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:45.982+0000 7fc907e11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:53:46.010 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:46.009+0000 7f6a66032040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:53:46.037 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.035+0000 7fc907e11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:53:46.067 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:46.066+0000 7f6a66032040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:53:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.170+0000 7f5901f54640 1 Processor -- start 2023-11-28T20:53:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.170+0000 7f5901f54640 1 -- start start 2023-11-28T20:53:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.172+0000 7f5901f54640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f58fc15d580 0x7f58fc0cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.172+0000 7f5901f54640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58fc0d0490 con 0x7f58fc0ce750 2023-11-28T20:53:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.172+0000 7f5901f54640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58fc0d1360 con 0x7f58fc0d1e40 2023-11-28T20:53:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.172+0000 7f5901f54640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58fc0d1640 con 0x7f58fc15d580 2023-11-28T20:53:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.172+0000 7f5901753640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f58fc0ce750 0x7f58fc160f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:32998/0 (socket says 172.21.15.45:32998) 2023-11-28T20:53:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.172+0000 7f5901753640 1 -- 172.21.15.45:0/1170503678 learned_addr learned my addr 172.21.15.45:0/1170503678 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:53:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.173+0000 7f5900f52640 1 --2- 172.21.15.45:0/1170503678 >> v2:172.21.15.190:3300/0 conn(0x7f58fc15d580 0x7f58fc0cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.173+0000 7f58fb7fe640 1 -- 172.21.15.45:0/1170503678 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699970338 0 0) 0x7f58fc0d0490 con 0x7f58fc0ce750 2023-11-28T20:53:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.173+0000 7f58fb7fe640 1 -- 172.21.15.45:0/1170503678 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ec003520 con 0x7f58fc0ce750 2023-11-28T20:53:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.173+0000 7f5900f52640 1 -- 172.21.15.45:0/1170503678 >> v1:172.21.15.190:6789/0 conn(0x7f58fc0d1e40 legacy=0x7f58fc0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.173+0000 7f58fb7fe640 1 -- 172.21.15.45:0/1170503678 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4102187230 0 0) 0x7f58ec003520 con 0x7f58fc0ce750 2023-11-28T20:53:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.173+0000 7f5900f52640 1 -- 172.21.15.45:0/1170503678 >> v1:172.21.15.45:6789/0 conn(0x7f58fc0ce750 legacy=0x7f58fc160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.173+0000 7f5900f52640 1 -- 172.21.15.45:0/1170503678 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc0d1920 con 0x7f58fc15d580 2023-11-28T20:53:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.175+0000 7f5900f52640 1 --2- 172.21.15.45:0/1170503678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc15d580 0x7f58fc0cfb00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f58f00097a0 tx=0x7f58f002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6572b3bd85239d95 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:46.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.175+0000 7f58fb7fe640 1 -- 172.21.15.45:0/1170503678 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40019f0 con 0x7f58fc15d580 2023-11-28T20:53:46.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.175+0000 7f58fb7fe640 1 -- 172.21.15.45:0/1170503678 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f002e050 con 0x7f58fc15d580 2023-11-28T20:53:46.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.175+0000 7f58fb7fe640 1 -- 172.21.15.45:0/1170503678 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0030730 con 0x7f58fc15d580 2023-11-28T20:53:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.176+0000 7f5901f54640 1 -- 172.21.15.45:0/1170503678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc15d580 msgr2=0x7f58fc0cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.176+0000 7f5901f54640 1 --2- 172.21.15.45:0/1170503678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc15d580 0x7f58fc0cfb00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f58f00097a0 tx=0x7f58f002b790 comp rx=0 tx=0).stop 2023-11-28T20:53:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.176+0000 7f5901f54640 1 -- 172.21.15.45:0/1170503678 shutdown_connections 2023-11-28T20:53:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.176+0000 7f5901f54640 1 --2- 172.21.15.45:0/1170503678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc15d580 0x7f58fc0cfb00 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.176+0000 7f5901f54640 1 -- 172.21.15.45:0/1170503678 >> 172.21.15.45:0/1170503678 conn(0x7f58fc06a500 msgr2=0x7f58fc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.176+0000 7f5901f54640 1 -- 172.21.15.45:0/1170503678 shutdown_connections 2023-11-28T20:53:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.176+0000 7f5901f54640 1 -- 172.21.15.45:0/1170503678 wait complete. 2023-11-28T20:53:46.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.179+0000 7f5901f54640 1 Processor -- start 2023-11-28T20:53:46.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.179+0000 7f5901f54640 1 -- start start 2023-11-28T20:53:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.180+0000 7f5901f54640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc0ce750 0x7f58fc0b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f5901f54640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58fc0d1e40 0x7f58fc0cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f5901f54640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc15d580 0x7f58fc157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f5901f54640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc15b580 con 0x7f58fc0d1e40 2023-11-28T20:53:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f5901f54640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0c8240 con 0x7f58fc0ce750 2023-11-28T20:53:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f5901f54640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc15e6f0 con 0x7f58fc15d580 2023-11-28T20:53:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58fc0d1e40 0x7f58fc0cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58fc0d1e40 0x7f58fc0cbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58438/0 (socket says 172.21.15.45:58438) 2023-11-28T20:53:46.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f58fbfff640 1 -- 172.21.15.45:0/2650962880 learned_addr learned my addr 172.21.15.45:0/2650962880 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:46.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f5901753640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc15d580 0x7f58fc157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.181+0000 7f5900f52640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc0ce750 0x7f58fc0b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f58fbfff640 1 -- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc15d580 msgr2=0x7f58fc157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f58fbfff640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc15d580 0x7f58fc157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f58fbfff640 1 -- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc0ce750 msgr2=0x7f58fc0b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f58fbfff640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc0ce750 0x7f58fc0b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f58fbfff640 1 -- 172.21.15.45:0/2650962880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f0009400 con 0x7f58fc0d1e40 2023-11-28T20:53:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f5901753640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc15d580 0x7f58fc157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f5900f52640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc0ce750 0x7f58fc0b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:53:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f58fbfff640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58fc0d1e40 0x7f58fc0cbe30 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c420 tx=0x7f58e800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f58f97fa640 1 -- 172.21.15.45:0/2650962880 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e80052d0 con 0x7f58fc0d1e40 2023-11-28T20:53:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f58f97fa640 1 -- 172.21.15.45:0/2650962880 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e8005430 con 0x7f58fc0d1e40 2023-11-28T20:53:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f58f97fa640 1 -- 172.21.15.45:0/2650962880 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8003880 con 0x7f58fc0d1e40 2023-11-28T20:53:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58fc15e9d0 con 0x7f58fc0d1e40 2023-11-28T20:53:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.182+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc15ef90 con 0x7f58fc0d1e40 2023-11-28T20:53:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.184+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58fc071b30 con 0x7f58fc0d1e40 2023-11-28T20:53:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.185+0000 7f58f97fa640 1 -- 172.21.15.45:0/2650962880 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 58) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f58e80039e0 con 0x7f58fc0d1e40 2023-11-28T20:53:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.185+0000 7f58f97fa640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f58e006b000 0x7f58e006d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.186+0000 7f58f97fa640 1 -- 172.21.15.45:0/2650962880 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f58e8088da0 con 0x7f58fc0d1e40 2023-11-28T20:53:46.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.186+0000 7f5900f52640 1 -- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f58e006b000 msgr2=0x7f58e006d4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3959324769 2023-11-28T20:53:46.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.186+0000 7f5900f52640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f58e006b000 0x7f58e006d4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.190+0000 7f58f97fa640 1 -- 172.21.15.45:0/2650962880 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58e8053d70 con 0x7f58fc0d1e40 2023-11-28T20:53:46.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.383+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f58fc0c8380 con 0x7f58fc0d1e40 2023-11-28T20:53:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.387+0000 7f5900f52640 1 -- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f58e006b000 msgr2=0x7f58e006d4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3959324769 2023-11-28T20:53:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.387+0000 7f5900f52640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f58e006b000 0x7f58e006d4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:53:46.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.389+0000 7f58f97fa640 1 -- 172.21.15.45:0/2650962880 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f58e80170c0 con 0x7f58fc0d1e40 2023-11-28T20:53:46.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.391+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f58e006b000 msgr2=0x7f58e006d4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:46.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.391+0000 7f5901f54640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f58e006b000 0x7f58e006d4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.391+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58fc0d1e40 msgr2=0x7f58fc0cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:46.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.391+0000 7f5901f54640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58fc0d1e40 0x7f58fc0cbe30 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c420 tx=0x7f58e800c8e0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.392+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 shutdown_connections 2023-11-28T20:53:46.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.392+0000 7f5901f54640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f58e006b000 0x7f58e006d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.392+0000 7f5901f54640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc15d580 0x7f58fc157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.392+0000 7f5901f54640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58fc0d1e40 0x7f58fc0cbe30 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.392+0000 7f5901f54640 1 --2- 172.21.15.45:0/2650962880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58fc0ce750 0x7f58fc0b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.392+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 >> 172.21.15.45:0/2650962880 conn(0x7f58fc06a500 msgr2=0x7f58fc0cd780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:46.393 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.392+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 shutdown_connections 2023-11-28T20:53:46.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.392+0000 7f5901f54640 1 -- 172.21.15.45:0/2650962880 wait complete. 2023-11-28T20:53:46.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:46.410 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-11-28T20:53:46.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-11-28T20:53:46.501 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.499+0000 7fc907e11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:53:46.550 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:46.549+0000 7f6a66032040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:53:46.553 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.552+0000 7fc907e11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:53:46.603 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.602+0000 7fc907e11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:53:46.605 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:46.604+0000 7f6a66032040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:53:46.659 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:46.658+0000 7f6a66032040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:53:46.663 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.662+0000 7fc907e11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:53:46.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.676+0000 7f511bc56640 1 Processor -- start 2023-11-28T20:53:46.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.676+0000 7f511bc56640 1 -- start start 2023-11-28T20:53:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.677+0000 7f511bc56640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5114160aa0 0x7f51140b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.677+0000 7f511bc56640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f51140d24f0 0x7f51140d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.677+0000 7f511bc56640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f51140ce0e0 0x7f51140ce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.677+0000 7f511bc56640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5114160e80 con 0x7f51140ce0e0 2023-11-28T20:53:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.677+0000 7f511bc56640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51140d1220 con 0x7f51140d24f0 2023-11-28T20:53:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.677+0000 7f511bc56640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51140d2e70 con 0x7f5114160aa0 2023-11-28T20:53:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f511a1f3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f51140ce0e0 0x7f51140ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f511a1f3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f51140ce0e0 0x7f51140ce570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58454/0 (socket says 172.21.15.45:58454) 2023-11-28T20:53:46.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f511a1f3640 1 -- 172.21.15.45:0/986155816 learned_addr learned my addr 172.21.15.45:0/986155816 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f51199f2640 1 --2- 172.21.15.45:0/986155816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5114160aa0 0x7f51140b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f51191f1640 1 --2- 172.21.15.45:0/986155816 >> v2:172.21.15.190:3300/0 conn(0x7f51140d24f0 0x7f51140d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f511a1f3640 1 -- 172.21.15.45:0/986155816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5114160aa0 msgr2=0x7f51140b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f511a1f3640 1 --2- 172.21.15.45:0/986155816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5114160aa0 0x7f51140b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f511a1f3640 1 -- 172.21.15.45:0/986155816 >> v2:172.21.15.190:3300/0 conn(0x7f51140d24f0 msgr2=0x7f51140d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f511a1f3640 1 --2- 172.21.15.45:0/986155816 >> v2:172.21.15.190:3300/0 conn(0x7f51140d24f0 0x7f51140d2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.678+0000 7f511a1f3640 1 -- 172.21.15.45:0/986155816 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51140b6960 con 0x7f51140ce0e0 2023-11-28T20:53:46.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.680+0000 7f511a1f3640 1 --2- 172.21.15.45:0/986155816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140ce0e0 0x7f51140ce570 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f51140b5e00 tx=0x7f510c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d684f9821209c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:46.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.680+0000 7f51189f0640 1 -- 172.21.15.45:0/986155816 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c030860 con 0x7f51140ce0e0 2023-11-28T20:53:46.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.680+0000 7f51189f0640 1 -- 172.21.15.45:0/986155816 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510c0309c0 con 0x7f51140ce0e0 2023-11-28T20:53:46.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.680+0000 7f51189f0640 1 -- 172.21.15.45:0/986155816 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c030d50 con 0x7f51140ce0e0 2023-11-28T20:53:46.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.681+0000 7f511bc56640 1 -- 172.21.15.45:0/986155816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140ce0e0 msgr2=0x7f51140ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:46.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.681+0000 7f511bc56640 1 --2- 172.21.15.45:0/986155816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140ce0e0 0x7f51140ce570 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f51140b5e00 tx=0x7f510c02b9a0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.681+0000 7f511bc56640 1 -- 172.21.15.45:0/986155816 shutdown_connections 2023-11-28T20:53:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.681+0000 7f511bc56640 1 --2- 172.21.15.45:0/986155816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140ce0e0 0x7f51140ce570 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.681+0000 7f511bc56640 1 --2- 172.21.15.45:0/986155816 >> v2:172.21.15.190:3300/0 conn(0x7f51140d24f0 0x7f51140d2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.681+0000 7f511bc56640 1 --2- 172.21.15.45:0/986155816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5114160aa0 0x7f51140b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.681+0000 7f511bc56640 1 -- 172.21.15.45:0/986155816 >> 172.21.15.45:0/986155816 conn(0x7f5114070b20 msgr2=0x7f5114070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.681+0000 7f511bc56640 1 -- 172.21.15.45:0/986155816 shutdown_connections 2023-11-28T20:53:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.681+0000 7f511bc56640 1 -- 172.21.15.45:0/986155816 wait complete. 2023-11-28T20:53:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.683+0000 7f511bc56640 1 Processor -- start 2023-11-28T20:53:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.683+0000 7f511bc56640 1 -- start start 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.684+0000 7f511bc56640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51140ce0e0 0x7f51141506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.684+0000 7f511bc56640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140d24f0 0x7f511415c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.684+0000 7f511bc56640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5114160aa0 0x7f511415e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.684+0000 7f511bc56640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5114150c30 con 0x7f51140d24f0 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.684+0000 7f511bc56640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511415da00 con 0x7f51140ce0e0 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.684+0000 7f511bc56640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511415d160 con 0x7f5114160aa0 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.684+0000 7f51191f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140d24f0 0x7f511415c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.685+0000 7f51191f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140d24f0 0x7f511415c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58458/0 (socket says 172.21.15.45:58458) 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.685+0000 7f51191f1640 1 -- 172.21.15.45:0/3327641823 learned_addr learned my addr 172.21.15.45:0/3327641823 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.685+0000 7f511a1f3640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5114160aa0 0x7f511415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.685+0000 7f51199f2640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51140ce0e0 0x7f51141506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:46.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.685+0000 7f51191f1640 1 -- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5114160aa0 msgr2=0x7f511415e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:46.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.685+0000 7f51191f1640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5114160aa0 0x7f511415e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.685+0000 7f51191f1640 1 -- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51140ce0e0 msgr2=0x7f51141506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:46.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.685+0000 7f51191f1640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51140ce0e0 0x7f51141506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:46.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.685+0000 7f51191f1640 1 -- 172.21.15.45:0/3327641823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c0094d0 con 0x7f51140d24f0 2023-11-28T20:53:46.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.686+0000 7f51191f1640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140d24f0 0x7f511415c6c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f510400d830 tx=0x7f510400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:46.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.686+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51040096a0 con 0x7f51140d24f0 2023-11-28T20:53:46.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.686+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104009800 con 0x7f51140d24f0 2023-11-28T20:53:46.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.686+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104009b90 con 0x7f51140d24f0 2023-11-28T20:53:46.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.686+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511415d440 con 0x7f51140d24f0 2023-11-28T20:53:46.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.686+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51141f4500 con 0x7f51140d24f0 2023-11-28T20:53:46.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.687+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51141f4690 con 0x7f51140d24f0 2023-11-28T20:53:46.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.688+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 58) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f5104014410 con 0x7f51140d24f0 2023-11-28T20:53:46.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.689+0000 7f510a7fc640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 0x7f50f806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:46.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.690+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5104089fc0 con 0x7f51140d24f0 2023-11-28T20:53:46.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.690+0000 7f51199f2640 1 -- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 msgr2=0x7f50f806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3959324769 2023-11-28T20:53:46.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.690+0000 7f51199f2640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 0x7f50f806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:53:46.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.694+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5104054d80 con 0x7f51140d24f0 2023-11-28T20:53:46.713 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.712+0000 7fc907e11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:53:46.748 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:46.747+0000 7f6a66032040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:53:46.806 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:46.804+0000 7f6a66032040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:53:46.823 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.822+0000 7fc907e11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:53:46.885 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.884+0000 7fc907e11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:53:46.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.891+0000 7f51199f2640 1 -- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 msgr2=0x7f50f806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3959324769 2023-11-28T20:53:46.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.891+0000 7f51199f2640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 0x7f50f806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:53:46.927 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:46.925+0000 7f6a66032040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:53:46.934 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.933+0000 7fc907e11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:53:46.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:46.936+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 --> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f51140d1750 con 0x7f50f806afb0 2023-11-28T20:53:46.992 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:46.991+0000 7fc907e11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:53:46.996 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:46.995+0000 7f6a66032040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:53:47.041 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:47.040+0000 7fc907e11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:53:47.051 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:47.050+0000 7f6a66032040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:53:47.115 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:47.114+0000 7f6a66032040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:53:47.170 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:47.169+0000 7f6a66032040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:53:47.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:47.292+0000 7f51199f2640 1 -- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 msgr2=0x7f50f806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3959324769 2023-11-28T20:53:47.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:47.292+0000 7f51199f2640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 0x7f50f806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T20:53:47.416 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:47.415+0000 7fc907e11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:53:47.571 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:47.569+0000 7f6a66032040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:53:47.574 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:47.573+0000 7fc907e11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:53:47.681 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:47.680+0000 7fc907e11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:53:47.731 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:47.730+0000 7fc907e11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:53:47.737 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:47.736+0000 7f6a66032040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:53:47.840 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:47.838+0000 7fc907e11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:53:47.850 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:47.849+0000 7f6a66032040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:53:47.902 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:47.901+0000 7f6a66032040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:53:47.950 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:47.946+0000 7fc907e11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:53:48.015 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:48.014+0000 7f6a66032040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:53:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:48.093+0000 7f51199f2640 1 -- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 msgr2=0x7f50f806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3959324769 2023-11-28T20:53:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:48.093+0000 7f51199f2640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 0x7f50f806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T20:53:48.104 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:48.102+0000 7fc907e11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:53:48.127 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:48.126+0000 7f6a66032040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:53:48.153 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:48.151+0000 7fc907e11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:53:48.271 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:48.269+0000 7fc907e11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:53:48.291 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:48.289+0000 7f6a66032040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:53:48.342 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:48.341+0000 7f6a66032040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:53:48.467 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:48.465+0000 7f6a66032040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:53:49.087 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.085+0000 7fc907e11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:53:49.351 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:49.350+0000 7f6a66032040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:53:49.378 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.377+0000 7fc907e11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:53:49.435 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.434+0000 7fc907e11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:53:49.549 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.548+0000 7fc907e11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:53:49.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:49.570+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 59) v1 ==== 88959+0+0 (secure 0 0 0) 0x7f5104055150 con 0x7f51140d24f0 2023-11-28T20:53:49.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:49.571+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 msgr2=0x7f50f806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:53:49.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:49.571+0000 7f510a7fc640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3959324769,v1:172.21.15.190:6833/3959324769] conn(0x7f50f806afb0 0x7f50f806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:49.678 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:49.677+0000 7f6a66032040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:53:49.735 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc885069640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.735 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc885069640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.736 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc885069640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.736 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc885069640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.736 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc885069640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.736 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc88686c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.736 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc88686c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.736 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc88686c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.736 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc88686c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.736 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:53:49.734+0000 7fc88686c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:53:49.741 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:49.740+0000 7f6a66032040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:53:49.862 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:53:49.861+0000 7f6a66032040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:53:50.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:53:50.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:53:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.575+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 60) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f5104056860 con 0x7f51140d24f0 2023-11-28T20:53:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.575+0000 7f510a7fc640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f50f80358b0 0x7f50f8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.575+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 --> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f50f8035c90 con 0x7f50f80358b0 2023-11-28T20:53:50.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.577+0000 7f51199f2640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f50f80358b0 0x7f50f8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:50.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.585+0000 7f51199f2640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f50f80358b0 0x7f50f8075380 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5110009fd0 tx=0x7f511000b040 comp rx=0 tx=0).ready entity=mgr.6350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:50.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.694+0000 7f510a7fc640 1 -- 172.21.15.45:0/3327641823 <== mgr.6350 v2:172.21.15.190:6832/3157274352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f50f8035c90 con 0x7f50f80358b0 2023-11-28T20:53:50.696 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1 2023-11-28T20:53:50.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.697+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f50f80358b0 msgr2=0x7f50f8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:50.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.697+0000 7f511bc56640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f50f80358b0 0x7f50f8075380 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5110009fd0 tx=0x7f511000b040 comp rx=0 tx=0).stop 2023-11-28T20:53:50.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.697+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140d24f0 msgr2=0x7f511415c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:50.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.697+0000 7f511bc56640 1 --2- 172.21.15.45:0/3327641823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51140d24f0 0x7f511415c6c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f510400d830 tx=0x7f510400dcf0 comp rx=0 tx=0).stop 2023-11-28T20:53:50.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.698+0000 7f511a1f3640 1 -- 172.21.15.45:0/3327641823 reap_dead start 2023-11-28T20:53:50.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.698+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 shutdown_connections 2023-11-28T20:53:50.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.698+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 >> 172.21.15.45:0/3327641823 conn(0x7f5114070b20 msgr2=0x7f5114070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:50.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.698+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 shutdown_connections 2023-11-28T20:53:50.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:50.698+0000 7f511bc56640 1 -- 172.21.15.45:0/3327641823 wait complete. 2023-11-28T20:53:50.720 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-11-28T20:53:50.721 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-11-28-20_54_00 in ~10s... 2023-11-28T20:53:50.721 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-11-28T20:53:50.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-11-28T20:53:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.007+0000 7ff3bb2bb640 1 Processor -- start 2023-11-28T20:53:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.007+0000 7ff3bb2bb640 1 -- start start 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.008+0000 7ff3bb2bb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3b4071fd0 0x7ff3b40ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3bb2bb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff3b40abd70 0x7ff3b40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3bb2bb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b40ac270 con 0x7ff3b415b080 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3bb2bb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b415b830 con 0x7ff3b4071fd0 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3bb2bb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b415bb10 con 0x7ff3b40abd70 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3b9ab8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3b4071fd0 0x7ff3b40ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3b9ab8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff3b4071fd0 0x7ff3b40ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58464/0 (socket says 172.21.15.45:58464) 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3b9ab8640 1 -- 172.21.15.45:0/1876905997 learned_addr learned my addr 172.21.15.45:0/1876905997 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3b9ab8640 1 -- 172.21.15.45:0/1876905997 >> v2:172.21.15.190:3300/0 conn(0x7ff3b40abd70 msgr2=0x7ff3b40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3baaba640 1 --2- 172.21.15.45:0/1876905997 >> v2:172.21.15.190:3300/0 conn(0x7ff3b40abd70 0x7ff3b40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3b9ab8640 1 --2- 172.21.15.45:0/1876905997 >> v2:172.21.15.190:3300/0 conn(0x7ff3b40abd70 0x7ff3b40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:51.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3b9ab8640 1 -- 172.21.15.45:0/1876905997 >> v1:172.21.15.190:6789/0 conn(0x7ff3b415b080 legacy=0x7ff3b40718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:53:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.009+0000 7ff3b9ab8640 1 -- 172.21.15.45:0/1876905997 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b415bdf0 con 0x7ff3b4071fd0 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.011+0000 7ff3b9ab8640 1 --2- 172.21.15.45:0/1876905997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b4071fd0 0x7ff3b40ae370 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4072a60 tx=0x7ff3a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3f827daa249e3d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.011+0000 7ff3b92b7640 1 -- 172.21.15.45:0/1876905997 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4031870 con 0x7ff3b4071fd0 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.011+0000 7ff3b92b7640 1 -- 172.21.15.45:0/1876905997 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a40319d0 con 0x7ff3b4071fd0 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.011+0000 7ff3b92b7640 1 -- 172.21.15.45:0/1876905997 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4031d60 con 0x7ff3b4071fd0 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.012+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/1876905997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b4071fd0 msgr2=0x7ff3b40ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.012+0000 7ff3bb2bb640 1 --2- 172.21.15.45:0/1876905997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b4071fd0 0x7ff3b40ae370 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4072a60 tx=0x7ff3a402b910 comp rx=0 tx=0).stop 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.012+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/1876905997 shutdown_connections 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.012+0000 7ff3bb2bb640 1 --2- 172.21.15.45:0/1876905997 >> v2:172.21.15.190:3300/0 conn(0x7ff3b40abd70 0x7ff3b40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.012+0000 7ff3bb2bb640 1 --2- 172.21.15.45:0/1876905997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b4071fd0 0x7ff3b40ae370 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.012+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/1876905997 >> 172.21.15.45:0/1876905997 conn(0x7ff3b40cecd0 msgr2=0x7ff3b40cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.012+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/1876905997 shutdown_connections 2023-11-28T20:53:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.012+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/1876905997 wait complete. 2023-11-28T20:53:51.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.014+0000 7ff3bb2bb640 1 Processor -- start 2023-11-28T20:53:51.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.014+0000 7ff3bb2bb640 1 -- start start 2023-11-28T20:53:51.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3bb2bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b4071fd0 0x7ff3b409a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:51.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3bb2bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b40abd70 0x7ff3b4152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:51.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3bb2bb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b415b080 0x7ff3b4157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:51.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3bb2bb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b4152ff0 con 0x7ff3b415b080 2023-11-28T20:53:51.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3bb2bb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b409c830 con 0x7ff3b4071fd0 2023-11-28T20:53:51.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3bb2bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b409b360 con 0x7ff3b40abd70 2023-11-28T20:53:51.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3baaba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b415b080 0x7ff3b4157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:51.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3baaba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b415b080 0x7ff3b4157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58472/0 (socket says 172.21.15.45:58472) 2023-11-28T20:53:51.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3baaba640 1 -- 172.21.15.45:0/175587771 learned_addr learned my addr 172.21.15.45:0/175587771 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:53:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3ba2b9640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b4071fd0 0x7ff3b409a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3b9ab8640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b40abd70 0x7ff3b4152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3baaba640 1 -- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b40abd70 msgr2=0x7ff3b4152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3baaba640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b40abd70 0x7ff3b4152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3baaba640 1 -- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b4071fd0 msgr2=0x7ff3b409a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3baaba640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b4071fd0 0x7ff3b409a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3baaba640 1 -- 172.21.15.45:0/175587771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a40094d0 con 0x7ff3b415b080 2023-11-28T20:53:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.015+0000 7ff3ba2b9640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b4071fd0 0x7ff3b409a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:53:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.016+0000 7ff3baaba640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b415b080 0x7ff3b4157000 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00cd40 tx=0x7ff3ac00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.016+0000 7ff3aaffd640 1 -- 172.21.15.45:0/175587771 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac003b90 con 0x7ff3b415b080 2023-11-28T20:53:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.016+0000 7ff3aaffd640 1 -- 172.21.15.45:0/175587771 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac003cf0 con 0x7ff3b415b080 2023-11-28T20:53:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.016+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b409aac0 con 0x7ff3b415b080 2023-11-28T20:53:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.016+0000 7ff3aaffd640 1 -- 172.21.15.45:0/175587771 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac004240 con 0x7ff3b415b080 2023-11-28T20:53:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.016+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b409b080 con 0x7ff3b415b080 2023-11-28T20:53:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.017+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b4066f10 con 0x7ff3b415b080 2023-11-28T20:53:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.018+0000 7ff3aaffd640 1 -- 172.21.15.45:0/175587771 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 60) v1 ==== 89086+0+0 (secure 0 0 0) 0x7ff3ac0043a0 con 0x7ff3b415b080 2023-11-28T20:53:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.019+0000 7ff3aaffd640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7ff39806ace0 0x7ff39806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:53:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.019+0000 7ff3aaffd640 1 -- 172.21.15.45:0/175587771 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7ff3ac012070 con 0x7ff3b415b080 2023-11-28T20:53:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.020+0000 7ff3ba2b9640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7ff39806ace0 0x7ff39806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:53:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.021+0000 7ff3ba2b9640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7ff39806ace0 0x7ff39806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4156830 tx=0x7ff3b0008040 comp rx=0 tx=0).ready entity=mgr.6350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:53:51.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.024+0000 7ff3aaffd640 1 -- 172.21.15.45:0/175587771 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3ac08d030 con 0x7ff3b415b080 2023-11-28T20:53:51.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.240+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 --> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3b415c310 con 0x7ff39806ace0 2023-11-28T20:53:51.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.257+0000 7ff3aaffd640 1 -- 172.21.15.45:0/175587771 <== mgr.6350 v2:172.21.15.190:6832/3157274352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7ff3b000ad80 con 0x7ff39806ace0 2023-11-28T20:53:51.259 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:53:51.259 INFO:teuthology.orchestra.run.smithi045.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-11-28T20:53:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7ff39806ace0 msgr2=0x7ff39806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7ff39806ace0 0x7ff39806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4156830 tx=0x7ff3b0008040 comp rx=0 tx=0).stop 2023-11-28T20:53:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b415b080 msgr2=0x7ff3b4157000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:53:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b415b080 0x7ff3b4157000 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00cd40 tx=0x7ff3ac00a660 comp rx=0 tx=0).stop 2023-11-28T20:53:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 shutdown_connections 2023-11-28T20:53:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7ff39806ace0 0x7ff39806d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:51.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3b415b080 0x7ff3b4157000 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3b40abd70 0x7ff3b4152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 --2- 172.21.15.45:0/175587771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3b4071fd0 0x7ff3b409a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:53:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.260+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 >> 172.21.15.45:0/175587771 conn(0x7ff3b40cecd0 msgr2=0x7ff3b406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:53:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.261+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 shutdown_connections 2023-11-28T20:53:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:53:51.261+0000 7ff3bb2bb640 1 -- 172.21.15.45:0/175587771 wait complete. 2023-11-28T20:53:51.284 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-11-28T20:53:51.284 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2023-11-28T20:53:51.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:51.284 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:53:51.341 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:53:51.342 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:53:51.342 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:53:52.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:52.343 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:53:52.411 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:53:52.411 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:53:52.411 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:53:53.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:53.413 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:53:53.481 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:53:53.481 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:53:53.481 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:53:54.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:54.482 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:53:54.550 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:53:54.550 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:53:54.550 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:53:55.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:55.551 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:53:55.620 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:53:55.620 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:53:55.620 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:53:56.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:56.621 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:53:56.688 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:53:56.688 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:53:56.688 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:53:57.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:57.689 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:53:57.757 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:53:57.757 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:53:57.757 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:53:58.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:58.759 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:53:58.830 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:53:58.830 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:53:58.831 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:53:59.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:53:59.833 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:53:59.901 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T20:53:59.902 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T20:53:59.902 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:54:00.718 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:00.715+0000 7fc87784e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:00.718 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:00.715+0000 7fc87784e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:00.718 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:00.715+0000 7fc87784e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:00.718 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:00.715+0000 7fc87784e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:00.719 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:00.715+0000 7fc87784e640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:00.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:00.903 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:54:00.967 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_54_00_UTC 2023-11-28T20:54:00.969 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-20_54_00_UTC'} 2023-11-28T20:54:00.969 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-11-28-20_54_00_UTC'} 2023-11-28T20:54:00.969 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T20:54:00.969 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-11-28-20_54_00_UTC'] 2023-11-28T20:54:00.969 DEBUG:tasks.cephfs.test_snap_schedules:exec=1701204830.7209399, now = 1701204840.9697876, timo = 10 2023-11-28T20:54:01.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-11-28T20:54:02.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.210+0000 7f605dd75640 1 Processor -- start 2023-11-28T20:54:02.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.210+0000 7f605dd75640 1 -- start start 2023-11-28T20:54:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.212+0000 7f605dd75640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f605815b080 0x7f60580ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.212+0000 7f605dd75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6058071fd0 0x7f60580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.212+0000 7f605dd75640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60580ac270 con 0x7f60580abd70 2023-11-28T20:54:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.212+0000 7f605dd75640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605815b830 con 0x7f605815b080 2023-11-28T20:54:02.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.212+0000 7f605dd75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605815bb10 con 0x7f6058071fd0 2023-11-28T20:54:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.212+0000 7f605d574640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6058071fd0 0x7f60580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.213+0000 7f605d574640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6058071fd0 0x7f60580702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:36154/0 (socket says 172.21.15.45:36154) 2023-11-28T20:54:02.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.213+0000 7f605d574640 1 -- 172.21.15.45:0/3530173653 learned_addr learned my addr 172.21.15.45:0/3530173653 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.213+0000 7f605cd73640 1 --2- 172.21.15.45:0/3530173653 >> v2:172.21.15.190:3300/0 conn(0x7f605815b080 0x7f60580ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.213+0000 7f605d574640 1 -- 172.21.15.45:0/3530173653 >> v2:172.21.15.190:3300/0 conn(0x7f605815b080 msgr2=0x7f60580ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.213+0000 7f605d574640 1 --2- 172.21.15.45:0/3530173653 >> v2:172.21.15.190:3300/0 conn(0x7f605815b080 0x7f60580ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.213+0000 7f605d574640 1 -- 172.21.15.45:0/3530173653 >> v1:172.21.15.190:6789/0 conn(0x7f60580abd70 legacy=0x7f60580718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.213+0000 7f605d574640 1 -- 172.21.15.45:0/3530173653 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605815bdf0 con 0x7f6058071fd0 2023-11-28T20:54:02.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.213+0000 7f60577fe640 1 -- 172.21.15.45:0/3530173653 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436583071 0 0) 0x7f60580ac270 con 0x7f60580abd70 2023-11-28T20:54:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.215+0000 7f605d574640 1 --2- 172.21.15.45:0/3530173653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6058071fd0 0x7f60580702c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6050009870 tx=0x7f605002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39728dde39b69c0b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.215+0000 7f60577fe640 1 -- 172.21.15.45:0/3530173653 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050030840 con 0x7f6058071fd0 2023-11-28T20:54:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.215+0000 7f60577fe640 1 -- 172.21.15.45:0/3530173653 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60500309a0 con 0x7f6058071fd0 2023-11-28T20:54:02.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.215+0000 7f60577fe640 1 -- 172.21.15.45:0/3530173653 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050030d30 con 0x7f6058071fd0 2023-11-28T20:54:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.216+0000 7f605dd75640 1 -- 172.21.15.45:0/3530173653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6058071fd0 msgr2=0x7f60580702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.216+0000 7f605dd75640 1 --2- 172.21.15.45:0/3530173653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6058071fd0 0x7f60580702c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6050009870 tx=0x7f605002b980 comp rx=0 tx=0).stop 2023-11-28T20:54:02.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.216+0000 7f605dd75640 1 -- 172.21.15.45:0/3530173653 shutdown_connections 2023-11-28T20:54:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.216+0000 7f605dd75640 1 --2- 172.21.15.45:0/3530173653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6058071fd0 0x7f60580702c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.216+0000 7f605dd75640 1 --2- 172.21.15.45:0/3530173653 >> v2:172.21.15.190:3300/0 conn(0x7f605815b080 0x7f60580ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.216+0000 7f605dd75640 1 -- 172.21.15.45:0/3530173653 >> 172.21.15.45:0/3530173653 conn(0x7f60580cecd0 msgr2=0x7f60580cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.216+0000 7f605dd75640 1 -- 172.21.15.45:0/3530173653 shutdown_connections 2023-11-28T20:54:02.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.216+0000 7f605dd75640 1 -- 172.21.15.45:0/3530173653 wait complete. 2023-11-28T20:54:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.218+0000 7f605dd75640 1 Processor -- start 2023-11-28T20:54:02.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.218+0000 7f605dd75640 1 -- start start 2023-11-28T20:54:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.219+0000 7f605dd75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6058071fd0 0x7f6058138610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.219+0000 7f605dd75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60580abd70 0x7f6058156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.219+0000 7f605dd75640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605815b080 0x7f605809b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.219+0000 7f605dd75640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605809ac10 con 0x7f605815b080 2023-11-28T20:54:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.219+0000 7f605dd75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6058099f90 con 0x7f6058071fd0 2023-11-28T20:54:02.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.219+0000 7f605dd75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605809a270 con 0x7f60580abd70 2023-11-28T20:54:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.219+0000 7f605d574640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605815b080 0x7f605809b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.219+0000 7f605d574640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605815b080 0x7f605809b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41632/0 (socket says 172.21.15.45:41632) 2023-11-28T20:54:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.219+0000 7f605d574640 1 -- 172.21.15.45:0/456697754 learned_addr learned my addr 172.21.15.45:0/456697754 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.220+0000 7f605cd73640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6058071fd0 0x7f6058138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.220+0000 7f6057fff640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60580abd70 0x7f6058156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.220+0000 7f605d574640 1 -- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60580abd70 msgr2=0x7f6058156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.220+0000 7f605d574640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60580abd70 0x7f6058156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.220+0000 7f605d574640 1 -- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6058071fd0 msgr2=0x7f6058138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:02.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.220+0000 7f605d574640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6058071fd0 0x7f6058138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.220+0000 7f605d574640 1 -- 172.21.15.45:0/456697754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60500094d0 con 0x7f605815b080 2023-11-28T20:54:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.220+0000 7f605d574640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605815b080 0x7f605809b030 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f6050009870 tx=0x7f6050004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.221+0000 7f60557fa640 1 -- 172.21.15.45:0/456697754 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050039c10 con 0x7f605815b080 2023-11-28T20:54:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.221+0000 7f60557fa640 1 -- 172.21.15.45:0/456697754 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6050039d70 con 0x7f605815b080 2023-11-28T20:54:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.221+0000 7f60557fa640 1 -- 172.21.15.45:0/456697754 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050043620 con 0x7f605815b080 2023-11-28T20:54:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.221+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605809a550 con 0x7f605815b080 2023-11-28T20:54:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.221+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6058138b50 con 0x7f605815b080 2023-11-28T20:54:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.222+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605809ad50 con 0x7f605815b080 2023-11-28T20:54:02.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.223+0000 7f60557fa640 1 -- 172.21.15.45:0/456697754 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 62) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f6050043780 con 0x7f605815b080 2023-11-28T20:54:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.224+0000 7f60557fa640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f604406b040 0x7f604406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:02.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.224+0000 7f60557fa640 1 -- 172.21.15.45:0/456697754 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f605003e070 con 0x7f605815b080 2023-11-28T20:54:02.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.225+0000 7f605cd73640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f604406b040 0x7f604406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:02.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.226+0000 7f605cd73640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f604406b040 0x7f604406d500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f60480073f0 tx=0x7f604800b040 comp rx=0 tx=0).ready entity=mgr.6350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:02.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.229+0000 7f60557fa640 1 -- 172.21.15.45:0/456697754 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f605809ad50 con 0x7f605815b080 2023-11-28T20:54:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.435+0000 7f60557fa640 1 -- 172.21.15.45:0/456697754 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 63) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f6050077b00 con 0x7f605815b080 2023-11-28T20:54:02.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.451+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 --> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f605815c310 con 0x7f604406b040 2023-11-28T20:54:02.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.620+0000 7f60557fa640 1 -- 172.21.15.45:0/456697754 <== mgr.6350 v2:172.21.15.190:6832/3157274352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f604800f050 con 0x7f604406b040 2023-11-28T20:54:02.621 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path snap_test_dir1 2023-11-28T20:54:02.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f604406b040 msgr2=0x7f604406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:02.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f604406b040 0x7f604406d500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f60480073f0 tx=0x7f604800b040 comp rx=0 tx=0).stop 2023-11-28T20:54:02.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605815b080 msgr2=0x7f605809b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605815b080 0x7f605809b030 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f6050009870 tx=0x7f6050004730 comp rx=0 tx=0).stop 2023-11-28T20:54:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 shutdown_connections 2023-11-28T20:54:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f604406b040 0x7f604406d500 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f605815b080 0x7f605809b030 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60580abd70 0x7f6058156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 --2- 172.21.15.45:0/456697754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6058071fd0 0x7f6058138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 >> 172.21.15.45:0/456697754 conn(0x7f60580cecd0 msgr2=0x7f60580cdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 shutdown_connections 2023-11-28T20:54:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:02.624+0000 7f605dd75640 1 -- 172.21.15.45:0/456697754 wait complete. 2023-11-28T20:54:02.648 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-11-28T20:54:02.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:02.649 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T20:54:02.707 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_54_00_UTC 2023-11-28T20:54:02.710 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-20_54_00_UTC 2023-11-28T20:54:02.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:02.710 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-20_54_00_UTC) 2023-11-28T20:54:02.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:02.775 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-11-28T20:54:02.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T20:54:03.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.063+0000 7f64ae1c5640 1 Processor -- start 2023-11-28T20:54:03.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.064+0000 7f64ae1c5640 1 -- start start 2023-11-28T20:54:03.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.065+0000 7f64ae1c5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a806ac10 0x7f64a806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:03.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.065+0000 7f64ae1c5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a815ecd0 con 0x7f64a815e840 2023-11-28T20:54:03.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.065+0000 7f64ae1c5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a815f290 con 0x7f64a815f720 2023-11-28T20:54:03.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.065+0000 7f64ae1c5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a815fbd0 con 0x7f64a806ac10 2023-11-28T20:54:03.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.066+0000 7f64a77fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f64a815e840 0x7f64a806a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48132/0 (socket says 172.21.15.45:48132) 2023-11-28T20:54:03.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.066+0000 7f64a77fe640 1 -- 172.21.15.45:0/2393956369 learned_addr learned my addr 172.21.15.45:0/2393956369 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:03.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.066+0000 7f649ffff640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a806ac10 0x7f64a806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.066+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178340416 0 0) 0x7f64a815ecd0 con 0x7f64a815e840 2023-11-28T20:54:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6494003520 con 0x7f64a815e840 2023-11-28T20:54:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4049473241 0 0) 0x7f6494003520 con 0x7f64a815e840 2023-11-28T20:54:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a806ac10 msgr2=0x7f64a806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a806ac10 0x7f64a806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 >> v1:172.21.15.190:6789/0 conn(0x7f64a815f720 legacy=0x7f64a80cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:03.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a8072020 con 0x7f64a815e840 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6498001670 con 0x7f64a815e840 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 >> v1:172.21.15.45:6789/0 conn(0x7f64a815e840 legacy=0x7f64a806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f6494003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a806ac10 0x7f64940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6494004d00 0x7f6494005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a815fbd0 con 0x7f64940039a0 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649400b980 con 0x7f6494004d00 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649400bb40 con 0x7f64a806ac10 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.067+0000 7f64a77fe640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f6494003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.068+0000 7f649ffff640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a806ac10 0x7f64940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.068+0000 7f64a7fff640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6494004d00 0x7f6494005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:03.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.068+0000 7f64a77fe640 1 -- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a806ac10 msgr2=0x7f64940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.068+0000 7f64a77fe640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a806ac10 0x7f64940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.068+0000 7f64a77fe640 1 -- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6494004d00 msgr2=0x7f6494005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.068+0000 7f64a77fe640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6494004d00 0x7f6494005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.068+0000 7f64a77fe640 1 -- 172.21.15.45:0/2393956369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6494003520 con 0x7f64940039a0 2023-11-28T20:54:03.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.068+0000 7f64a7fff640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6494004d00 0x7f6494005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:03.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.069+0000 7f64a77fe640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f6494003d80 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f649800c160 tx=0x7f64940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc285cce4b45d80 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:03.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.070+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498005ad0 con 0x7f64940039a0 2023-11-28T20:54:03.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.070+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6498005c30 con 0x7f64940039a0 2023-11-28T20:54:03.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.070+0000 7f64a6ffd640 1 -- 172.21.15.45:0/2393956369 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649802c7c0 con 0x7f64940039a0 2023-11-28T20:54:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.070+0000 7f64ae1c5640 1 -- 172.21.15.45:0/2393956369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 msgr2=0x7f6494003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.070+0000 7f64ae1c5640 1 --2- 172.21.15.45:0/2393956369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f6494003d80 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f649800c160 tx=0x7f64940057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.070+0000 7f64a7fff640 1 -- 172.21.15.45:0/2393956369 reap_dead start 2023-11-28T20:54:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.071+0000 7f64ae1c5640 1 -- 172.21.15.45:0/2393956369 shutdown_connections 2023-11-28T20:54:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.071+0000 7f64ae1c5640 1 -- 172.21.15.45:0/2393956369 >> 172.21.15.45:0/2393956369 conn(0x7f64a80ced60 msgr2=0x7f64a80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.071+0000 7f64ae1c5640 1 -- 172.21.15.45:0/2393956369 shutdown_connections 2023-11-28T20:54:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.071+0000 7f64ae1c5640 1 -- 172.21.15.45:0/2393956369 wait complete. 2023-11-28T20:54:03.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.072+0000 7f64ae1c5640 1 Processor -- start 2023-11-28T20:54:03.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.073+0000 7f64ae1c5640 1 -- start start 2023-11-28T20:54:03.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64ae1c5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f64a80c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64ae1c5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a80b3400 0x7f64a80c2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64ae1c5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a80c3280 0x7f64a80c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64ae1c5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80b4460 con 0x7f64940039a0 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64ae1c5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a81568c0 con 0x7f64a80b3400 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64ae1c5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80b3bc0 con 0x7f64a80c3280 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f64a80c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f64a80c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41660/0 (socket says 172.21.15.45:41660) 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64a77fe640 1 -- 172.21.15.45:0/3117787607 learned_addr learned my addr 172.21.15.45:0/3117787607 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f649ffff640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a80b3400 0x7f64a80c2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.074+0000 7f64a7fff640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a80c3280 0x7f64a80c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64a77fe640 1 -- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a80c3280 msgr2=0x7f64a80c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64a77fe640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a80c3280 0x7f64a80c3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64a77fe640 1 -- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a80b3400 msgr2=0x7f64a80c2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64a77fe640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a80b3400 0x7f64a80c2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64a77fe640 1 -- 172.21.15.45:0/3117787607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80b3ea0 con 0x7f64940039a0 2023-11-28T20:54:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64a77fe640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f64a80c51e0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f649800c160 tx=0x7f6498001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64a4ff9640 1 -- 172.21.15.45:0/3117787607 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649803c040 con 0x7f64940039a0 2023-11-28T20:54:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64a4ff9640 1 -- 172.21.15.45:0/3117787607 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6498005e90 con 0x7f64940039a0 2023-11-28T20:54:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a80b4180 con 0x7f64940039a0 2023-11-28T20:54:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64a4ff9640 1 -- 172.21.15.45:0/3117787607 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64980072b0 con 0x7f64940039a0 2023-11-28T20:54:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.075+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a80b3880 con 0x7f64940039a0 2023-11-28T20:54:03.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.077+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a80c93b0 con 0x7f64940039a0 2023-11-28T20:54:03.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.077+0000 7f64a4ff9640 1 -- 172.21.15.45:0/3117787607 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 63) v1 ==== 89464+0+0 (secure 0 0 0) 0x7f6498032070 con 0x7f64940039a0 2023-11-28T20:54:03.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.078+0000 7f64a4ff9640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f648806b0d0 0x7f648806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:03.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.079+0000 7f64a4ff9640 1 -- 172.21.15.45:0/3117787607 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f64980ab5c0 con 0x7f64940039a0 2023-11-28T20:54:03.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.079+0000 7f649ffff640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f648806b0d0 0x7f648806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:03.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.080+0000 7f649ffff640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f648806b0d0 0x7f648806d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f64a80ac3d0 tx=0x7f6490008040 comp rx=0 tx=0).ready entity=mgr.6350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:03.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.083+0000 7f64a4ff9640 1 -- 172.21.15.45:0/3117787607 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64a80c93b0 con 0x7f64940039a0 2023-11-28T20:54:03.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.327+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f64a80c93b0 con 0x7f64940039a0 2023-11-28T20:54:03.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.435+0000 7f64a4ff9640 1 -- 172.21.15.45:0/3117787607 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v64) v1 ==== 93+0+0 (secure 0 0 0) 0x7f64a80c93b0 con 0x7f64940039a0 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f648806b0d0 msgr2=0x7f648806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f648806b0d0 0x7f648806d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f64a80ac3d0 tx=0x7f6490008040 comp rx=0 tx=0).stop 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 msgr2=0x7f64a80c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f64a80c51e0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f649800c160 tx=0x7f6498001980 comp rx=0 tx=0).stop 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 shutdown_connections 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64a80c3280 0x7f64a80c3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a80b3400 0x7f64a80c2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f648806b0d0 0x7f648806d590 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 --2- 172.21.15.45:0/3117787607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64940039a0 0x7f64a80c51e0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:03.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.438+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 >> 172.21.15.45:0/3117787607 conn(0x7f64a80ced60 msgr2=0x7f64a80ac450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:03.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.439+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 shutdown_connections 2023-11-28T20:54:03.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:03.439+0000 7f64ae1c5640 1 -- 172.21.15.45:0/3117787607 wait complete. 2023-11-28T20:54:03.465 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:54:03.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:54:03.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:54:03.679 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:54:03.679 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:54:03.680 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:03.738 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:54:03.739 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:03.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:03.969 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:03.967+0000 7ff819e2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:54:04.022 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.019+0000 7ff819e2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:54:04.022 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:04.020+0000 7fcb94c1c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:54:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.043+0000 7f7647a56640 1 Processor -- start 2023-11-28T20:54:04.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.044+0000 7f7647a56640 1 -- start start 2023-11-28T20:54:04.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.045+0000 7f7647a56640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f76400b0c30 0x7f76400b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.045+0000 7f7647a56640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7640071190 0x7f7640071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.046+0000 7f7647a56640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76400ce710 con 0x7f764015d320 2023-11-28T20:54:04.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.046+0000 7f7647a56640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76400ae5a0 con 0x7f7640071190 2023-11-28T20:54:04.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.046+0000 7f7647a56640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7640160fe0 con 0x7f76400b0c30 2023-11-28T20:54:04.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.046+0000 7f76457f2640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f764015d320 0x7f76400ad800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49892/0 (socket says 172.21.15.45:49892) 2023-11-28T20:54:04.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.046+0000 7f76457f2640 1 -- 172.21.15.45:0/3342148102 learned_addr learned my addr 172.21.15.45:0/3342148102 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:04.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.046+0000 7f7645ff3640 1 --2- 172.21.15.45:0/3342148102 >> v2:172.21.15.45:3300/0 conn(0x7f7640071190 0x7f7640071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.046+0000 7f7644ff1640 1 --2- 172.21.15.45:0/3342148102 >> v2:172.21.15.190:3300/0 conn(0x7f76400b0c30 0x7f76400b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.046+0000 7f7637fff640 1 -- 172.21.15.45:0/3342148102 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808202506 0 0) 0x7f76400ce710 con 0x7f764015d320 2023-11-28T20:54:04.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.046+0000 7f7637fff640 1 -- 172.21.15.45:0/3342148102 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f762c003520 con 0x7f764015d320 2023-11-28T20:54:04.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.047+0000 7f7645ff3640 1 -- 172.21.15.45:0/3342148102 >> v2:172.21.15.190:3300/0 conn(0x7f76400b0c30 msgr2=0x7f76400b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:04.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.047+0000 7f7645ff3640 1 --2- 172.21.15.45:0/3342148102 >> v2:172.21.15.190:3300/0 conn(0x7f76400b0c30 0x7f76400b10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.047+0000 7f7645ff3640 1 -- 172.21.15.45:0/3342148102 >> v1:172.21.15.45:6789/0 conn(0x7f764015d320 legacy=0x7f76400ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.047+0000 7f7645ff3640 1 -- 172.21.15.45:0/3342148102 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76400ae2c0 con 0x7f7640071190 2023-11-28T20:54:04.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.048+0000 7f7645ff3640 1 --2- 172.21.15.45:0/3342148102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 0x7f7640071600 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f764006e650 tx=0x7f763802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4decc4b42f1a4e8f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:04.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.049+0000 7f7637fff640 1 -- 172.21.15.45:0/3342148102 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763802cc80 con 0x7f7640071190 2023-11-28T20:54:04.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.049+0000 7f7637fff640 1 -- 172.21.15.45:0/3342148102 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763802cde0 con 0x7f7640071190 2023-11-28T20:54:04.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.049+0000 7f7637fff640 1 -- 172.21.15.45:0/3342148102 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7638034660 con 0x7f7640071190 2023-11-28T20:54:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.049+0000 7f7647a56640 1 -- 172.21.15.45:0/3342148102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 msgr2=0x7f7640071600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.049+0000 7f7647a56640 1 --2- 172.21.15.45:0/3342148102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 0x7f7640071600 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f764006e650 tx=0x7f763802bed0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.050+0000 7f7647a56640 1 -- 172.21.15.45:0/3342148102 shutdown_connections 2023-11-28T20:54:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.050+0000 7f7647a56640 1 --2- 172.21.15.45:0/3342148102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 0x7f7640071600 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.050+0000 7f7647a56640 1 --2- 172.21.15.45:0/3342148102 >> v2:172.21.15.190:3300/0 conn(0x7f76400b0c30 0x7f76400b10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.050+0000 7f7647a56640 1 -- 172.21.15.45:0/3342148102 >> 172.21.15.45:0/3342148102 conn(0x7f764006a500 msgr2=0x7f764006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.050+0000 7f7647a56640 1 -- 172.21.15.45:0/3342148102 shutdown_connections 2023-11-28T20:54:04.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.050+0000 7f7647a56640 1 -- 172.21.15.45:0/3342148102 wait complete. 2023-11-28T20:54:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.051+0000 7f7647a56640 1 Processor -- start 2023-11-28T20:54:04.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.052+0000 7f7647a56640 1 -- start start 2023-11-28T20:54:04.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.052+0000 7f7647a56640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 0x7f76401506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.052+0000 7f7647a56640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400b0c30 0x7f7640151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f7647a56640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f764015d320 0x7f764015fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f7647a56640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76400b2a80 con 0x7f7640071190 2023-11-28T20:54:04.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f7647a56640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76400b1bb0 con 0x7f76400b0c30 2023-11-28T20:54:04.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f7647a56640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7640150be0 con 0x7f764015d320 2023-11-28T20:54:04.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f76457f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 0x7f76401506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f76457f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 0x7f76401506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45860/0 (socket says 172.21.15.45:45860) 2023-11-28T20:54:04.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f76457f2640 1 -- 172.21.15.45:0/1839865777 learned_addr learned my addr 172.21.15.45:0/1839865777 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:04.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f7644ff1640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400b0c30 0x7f7640151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f7645ff3640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f764015d320 0x7f764015fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f76457f2640 1 -- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f764015d320 msgr2=0x7f764015fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f76457f2640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f764015d320 0x7f764015fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f76457f2640 1 -- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400b0c30 msgr2=0x7f7640151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f76457f2640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400b0c30 0x7f7640151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f76457f2640 1 -- 172.21.15.45:0/1839865777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7638009a70 con 0x7f7640071190 2023-11-28T20:54:04.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.053+0000 7f7645ff3640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f764015d320 0x7f764015fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:04.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.054+0000 7f76457f2640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 0x7f76401506a0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f763c00c440 tx=0x7f763c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:04.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.054+0000 7f7635ffb640 1 -- 172.21.15.45:0/1839865777 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c005ce0 con 0x7f7640071190 2023-11-28T20:54:04.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.054+0000 7f7635ffb640 1 -- 172.21.15.45:0/1839865777 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c005e40 con 0x7f7640071190 2023-11-28T20:54:04.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.054+0000 7f7635ffb640 1 -- 172.21.15.45:0/1839865777 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0072b0 con 0x7f7640071190 2023-11-28T20:54:04.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.054+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76400b1e00 con 0x7f7640071190 2023-11-28T20:54:04.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.054+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76400b23c0 con 0x7f7640071190 2023-11-28T20:54:04.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.056+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76400733d0 con 0x7f7640071190 2023-11-28T20:54:04.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.056+0000 7f7635ffb640 1 -- 172.21.15.45:0/1839865777 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f763c007410 con 0x7f7640071190 2023-11-28T20:54:04.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.057+0000 7f7635ffb640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f762406b0b0 0x7f762406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.057+0000 7f7635ffb640 1 -- 172.21.15.45:0/1839865777 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f763c0892b0 con 0x7f7640071190 2023-11-28T20:54:04.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.058+0000 7f7644ff1640 1 -- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f762406b0b0 msgr2=0x7f762406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:04.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.058+0000 7f7644ff1640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f762406b0b0 0x7f762406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:04.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.062+0000 7f7635ffb640 1 -- 172.21.15.45:0/1839865777 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f763c053d80 con 0x7f7640071190 2023-11-28T20:54:04.084 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:04.083+0000 7fcb94c1c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:54:04.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.257+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76400ad250 con 0x7f7640071190 2023-11-28T20:54:04.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.259+0000 7f7635ffb640 1 -- 172.21.15.45:0/1839865777 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+17211 (secure 0 0 0) 0x7f763c0573a0 con 0x7f7640071190 2023-11-28T20:54:04.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.259+0000 7f7644ff1640 1 -- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f762406b0b0 msgr2=0x7f762406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:04.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.259+0000 7f7644ff1640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f762406b0b0 0x7f762406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:04.261 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":134,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:49.553240+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:53:33.200533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-11-28T20:53:34.300717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3135688549":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/369403092":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6833/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6832/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1007604882":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/685209934":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1745589824":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1706225465":"2023-11-29T20:53:49.553172+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:04.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.261+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f762406b0b0 msgr2=0x7f762406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:04.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.261+0000 7f7647a56640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f762406b0b0 0x7f762406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.261+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 msgr2=0x7f76401506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.261+0000 7f7647a56640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 0x7f76401506a0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f763c00c440 tx=0x7f763c00c900 comp rx=0 tx=0).stop 2023-11-28T20:54:04.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.262+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 shutdown_connections 2023-11-28T20:54:04.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.262+0000 7f7647a56640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f764015d320 0x7f764015fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.262+0000 7f7647a56640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400b0c30 0x7f7640151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.262+0000 7f7647a56640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f762406b0b0 0x7f762406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.262+0000 7f7647a56640 1 --2- 172.21.15.45:0/1839865777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7640071190 0x7f76401506a0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.262+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 >> 172.21.15.45:0/1839865777 conn(0x7f764006a500 msgr2=0x7f76400b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:04.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.262+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 shutdown_connections 2023-11-28T20:54:04.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.262+0000 7f7647a56640 1 -- 172.21.15.45:0/1839865777 wait complete. 2023-11-28T20:54:04.281 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:53:33.200533+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:53:34.300717+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:54:04.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:54:04.472 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.469+0000 7ff819e2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:54:04.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.495+0000 7f27cda09640 1 Processor -- start 2023-11-28T20:54:04.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.496+0000 7f27cda09640 1 -- start start 2023-11-28T20:54:04.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.497+0000 7f27cda09640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f27c80d0040 0x7f27c80d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.497+0000 7f27cda09640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c80b90a0 con 0x7f27c80b2290 2023-11-28T20:54:04.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.497+0000 7f27cda09640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c80d1920 con 0x7f27c80b1d40 2023-11-28T20:54:04.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.497+0000 7f27cda09640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c80d3e60 con 0x7f27c80d0040 2023-11-28T20:54:04.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.497+0000 7f27c7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f27c80b2290 0x7f27c80d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49904/0 (socket says 172.21.15.45:49904) 2023-11-28T20:54:04.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.497+0000 7f27c7fff640 1 -- 172.21.15.45:0/3486860711 learned_addr learned my addr 172.21.15.45:0/3486860711 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:04.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27cca07640 1 --2- 172.21.15.45:0/3486860711 >> v2:172.21.15.190:3300/0 conn(0x7f27c80d0040 0x7f27c80d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409691298 0 0) 0x7f27c80b90a0 con 0x7f27c80b2290 2023-11-28T20:54:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b8003520 con 0x7f27c80b2290 2023-11-28T20:54:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2139261390 0 0) 0x7f27b8003520 con 0x7f27c80b2290 2023-11-28T20:54:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 >> v2:172.21.15.190:3300/0 conn(0x7f27c80d0040 msgr2=0x7f27c80d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 --2- 172.21.15.45:0/3486860711 >> v2:172.21.15.190:3300/0 conn(0x7f27c80d0040 0x7f27c80d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 >> v1:172.21.15.190:6789/0 conn(0x7f27c80b1d40 legacy=0x7f27c80d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c80d2dc0 con 0x7f27c80b2290 2023-11-28T20:54:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f27b4001670 con 0x7f27c80b2290 2023-11-28T20:54:04.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 >> v1:172.21.15.45:6789/0 conn(0x7f27c80b2290 legacy=0x7f27c80d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.498+0000 7f27c77fe640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27b80047c0 0x7f27b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27c77fe640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27b8005180 0x7f27b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c80d3e60 con 0x7f27b8003970 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b800c030 con 0x7f27b80047c0 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b800c1f0 con 0x7f27b8005180 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27cca07640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27c7fff640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27b80047c0 0x7f27b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27cca07640 1 -- 172.21.15.45:0/3486860711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27b8005180 msgr2=0x7f27b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:04.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27cca07640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27b8005180 0x7f27b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27cca07640 1 -- 172.21.15.45:0/3486860711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27b80047c0 msgr2=0x7f27b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27cca07640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27b80047c0 0x7f27b8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27cca07640 1 -- 172.21.15.45:0/3486860711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b8003520 con 0x7f27b8003970 2023-11-28T20:54:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.499+0000 7f27cd208640 1 -- 172.21.15.45:0/3486860711 reap_dead start 2023-11-28T20:54:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.500+0000 7f27cca07640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27b8003d90 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f27bc009ac0 tx=0x7f27b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ff3bcc4362ffe8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.500+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc02c710 con 0x7f27b8003970 2023-11-28T20:54:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.500+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc02c870 con 0x7f27b8003970 2023-11-28T20:54:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.501+0000 7f27c77fe640 1 -- 172.21.15.45:0/3486860711 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc02cc00 con 0x7f27b8003970 2023-11-28T20:54:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.501+0000 7f27cda09640 1 -- 172.21.15.45:0/3486860711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 msgr2=0x7f27b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.501+0000 7f27cda09640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27b8003d90 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f27bc009ac0 tx=0x7f27b80057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.501+0000 7f27cda09640 1 -- 172.21.15.45:0/3486860711 shutdown_connections 2023-11-28T20:54:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.501+0000 7f27cda09640 1 --2- 172.21.15.45:0/3486860711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27b8003d90 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.501+0000 7f27cda09640 1 -- 172.21.15.45:0/3486860711 >> 172.21.15.45:0/3486860711 conn(0x7f27c8070f40 msgr2=0x7f27c8071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.501+0000 7f27cda09640 1 -- 172.21.15.45:0/3486860711 shutdown_connections 2023-11-28T20:54:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.501+0000 7f27cda09640 1 -- 172.21.15.45:0/3486860711 wait complete. 2023-11-28T20:54:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.503+0000 7f27cda09640 1 Processor -- start 2023-11-28T20:54:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.503+0000 7f27cda09640 1 -- start start 2023-11-28T20:54:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.504+0000 7f27cda09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27c81593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.504+0000 7f27cda09640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c80c8160 0x7f27c80c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.504+0000 7f27cda09640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27c81552b0 0x7f27c81e8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.504+0000 7f27cda09640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c80cc720 con 0x7f27b8003970 2023-11-28T20:54:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.504+0000 7f27cda09640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c80c7d40 con 0x7f27c80c8160 2023-11-28T20:54:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.504+0000 7f27cda09640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c80c7f50 con 0x7f27c81552b0 2023-11-28T20:54:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.504+0000 7f27cca07640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27c81593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.504+0000 7f27cca07640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27c81593e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45876/0 (socket says 172.21.15.45:45876) 2023-11-28T20:54:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.504+0000 7f27cca07640 1 -- 172.21.15.45:0/2682193362 learned_addr learned my addr 172.21.15.45:0/2682193362 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27cca07640 1 -- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27c81552b0 msgr2=0x7f27c81e8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27cca07640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27c81552b0 0x7f27c81e8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27c7fff640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c80c8160 0x7f27c80c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27cca07640 1 -- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c80c8160 msgr2=0x7f27c80c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27cca07640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c80c8160 0x7f27c80c7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27cca07640 1 -- 172.21.15.45:0/2682193362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c80cc930 con 0x7f27b8003970 2023-11-28T20:54:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27cca07640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27c81593e0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f27bc02c2b0 tx=0x7f27bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27c57fa640 1 -- 172.21.15.45:0/2682193362 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc003810 con 0x7f27b8003970 2023-11-28T20:54:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27c57fa640 1 -- 172.21.15.45:0/2682193362 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc02de90 con 0x7f27b8003970 2023-11-28T20:54:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27c57fa640 1 -- 172.21.15.45:0/2682193362 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc003b70 con 0x7f27b8003970 2023-11-28T20:54:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.505+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c80ccbd0 con 0x7f27b8003970 2023-11-28T20:54:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.506+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c8155760 con 0x7f27b8003970 2023-11-28T20:54:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.507+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c80cd500 con 0x7f27b8003970 2023-11-28T20:54:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.507+0000 7f27c57fa640 1 -- 172.21.15.45:0/2682193362 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f27bc003cd0 con 0x7f27b8003970 2023-11-28T20:54:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.508+0000 7f27c57fa640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f27ac06b0b0 0x7f27ac06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.509+0000 7f27c57fa640 1 -- 172.21.15.45:0/2682193362 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f27bc0abb10 con 0x7f27b8003970 2023-11-28T20:54:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.509+0000 7f27c7fff640 1 -- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f27ac06b0b0 msgr2=0x7f27ac06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.509+0000 7f27c7fff640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f27ac06b0b0 0x7f27ac06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.513+0000 7f27c57fa640 1 -- 172.21.15.45:0/2682193362 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27bc0765b0 con 0x7f27b8003970 2023-11-28T20:54:04.526 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.524+0000 7ff819e2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:54:04.569 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:04.567+0000 7fcb94c1c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:54:04.576 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.574+0000 7ff819e2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:54:04.624 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:04.622+0000 7fcb94c1c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:54:04.637 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.635+0000 7ff819e2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:54:04.677 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:04.675+0000 7fcb94c1c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:54:04.686 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.684+0000 7ff819e2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:54:04.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.705+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27c8071b30 con 0x7f27b8003970 2023-11-28T20:54:04.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.706+0000 7f27c57fa640 1 -- 172.21.15.45:0/2682193362 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27bc079bd0 con 0x7f27b8003970 2023-11-28T20:54:04.707 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:54:04.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f27ac06b0b0 msgr2=0x7f27ac06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:04.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f27ac06b0b0 0x7f27ac06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 msgr2=0x7f27c81593e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27c81593e0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f27bc02c2b0 tx=0x7f27bc004730 comp rx=0 tx=0).stop 2023-11-28T20:54:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 shutdown_connections 2023-11-28T20:54:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27c81552b0 0x7f27c81e8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c80c8160 0x7f27c80c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f27ac06b0b0 0x7f27ac06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 --2- 172.21.15.45:0/2682193362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8003970 0x7f27c81593e0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 >> 172.21.15.45:0/2682193362 conn(0x7f27c8070f40 msgr2=0x7f27c80b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 shutdown_connections 2023-11-28T20:54:04.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.708+0000 7f27cda09640 1 -- 172.21.15.45:0/2682193362 wait complete. 2023-11-28T20:54:04.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:54:04.746 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:04.745+0000 7fcb94c1c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:54:04.796 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.794+0000 7ff819e2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:54:04.800 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:04.799+0000 7fcb94c1c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:54:04.858 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.856+0000 7ff819e2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:04.907 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.905+0000 7ff819e2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:54:04.925 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:04.924+0000 7fcb94c1c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:54:04.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.945+0000 7f057918a640 1 Processor -- start 2023-11-28T20:54:04.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.945+0000 7f057918a640 1 -- start start 2023-11-28T20:54:04.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.946+0000 7f057918a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f057415b6d0 0x7f05740ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.946+0000 7f057918a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05740cd710 con 0x7f05740ce580 2023-11-28T20:54:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.947+0000 7f057918a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05740cb220 con 0x7f0574066900 2023-11-28T20:54:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.947+0000 7f057918a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05740cc140 con 0x7f057415b6d0 2023-11-28T20:54:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.947+0000 7f056bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f05740ce580 0x7f05740661f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49912/0 (socket says 172.21.15.45:49912) 2023-11-28T20:54:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.947+0000 7f056bfff640 1 -- 172.21.15.45:0/2201060860 learned_addr learned my addr 172.21.15.45:0/2201060860 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.947+0000 7f0572d9d640 1 --2- 172.21.15.45:0/2201060860 >> v2:172.21.15.190:3300/0 conn(0x7f057415b6d0 0x7f05740ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.947+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433122476 0 0) 0x7f05740cd710 con 0x7f05740ce580 2023-11-28T20:54:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.947+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0560003520 con 0x7f05740ce580 2023-11-28T20:54:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 346381766 0 0) 0x7f0560003520 con 0x7f05740ce580 2023-11-28T20:54:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 >> v2:172.21.15.190:3300/0 conn(0x7f057415b6d0 msgr2=0x7f05740ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 --2- 172.21.15.45:0/2201060860 >> v2:172.21.15.190:3300/0 conn(0x7f057415b6d0 0x7f05740ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 >> v1:172.21.15.190:6789/0 conn(0x7f0574066900 legacy=0x7f057406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057406b3e0 con 0x7f05740ce580 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f055c001670 con 0x7f05740ce580 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 >> v1:172.21.15.45:6789/0 conn(0x7f05740ce580 legacy=0x7f05740661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0560003970 0x7f0560003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05600047c0 0x7f0560004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f056000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05740cc140 con 0x7f0560005180 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056000c120 con 0x7f0560003970 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056000c2e0 con 0x7f05600047c0 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f057359e640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f056000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.948+0000 7f0572d9d640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0560003970 0x7f0560003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.949+0000 7f056bfff640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05600047c0 0x7f0560004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.949+0000 7f057359e640 1 -- 172.21.15.45:0/2201060860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05600047c0 msgr2=0x7f0560004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:04.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.949+0000 7f057359e640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05600047c0 0x7f0560004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.949+0000 7f057359e640 1 -- 172.21.15.45:0/2201060860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0560003970 msgr2=0x7f0560003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:04.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.949+0000 7f057359e640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0560003970 0x7f0560003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.949+0000 7f057359e640 1 -- 172.21.15.45:0/2201060860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0560003520 con 0x7f0560005180 2023-11-28T20:54:04.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.950+0000 7f057359e640 1 -- 172.21.15.45:0/2201060860 reap_dead start 2023-11-28T20:54:04.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.950+0000 7f057359e640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f056000b980 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f056c001e20 tx=0x7f05600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150dbe5e1e63786b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:04.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.950+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c003b70 con 0x7f0560005180 2023-11-28T20:54:04.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.950+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c003cd0 con 0x7f0560005180 2023-11-28T20:54:04.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.950+0000 7f057259c640 1 -- 172.21.15.45:0/2201060860 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c005310 con 0x7f0560005180 2023-11-28T20:54:04.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.951+0000 7f057918a640 1 -- 172.21.15.45:0/2201060860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 msgr2=0x7f056000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.951+0000 7f057918a640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f056000b980 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f056c001e20 tx=0x7f05600057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.951+0000 7f057918a640 1 -- 172.21.15.45:0/2201060860 shutdown_connections 2023-11-28T20:54:04.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.951+0000 7f057918a640 1 --2- 172.21.15.45:0/2201060860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f056000b980 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.951+0000 7f057918a640 1 -- 172.21.15.45:0/2201060860 >> 172.21.15.45:0/2201060860 conn(0x7f05740cc930 msgr2=0x7f05740ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:04.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.951+0000 7f057918a640 1 -- 172.21.15.45:0/2201060860 shutdown_connections 2023-11-28T20:54:04.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.951+0000 7f057918a640 1 -- 172.21.15.45:0/2201060860 wait complete. 2023-11-28T20:54:04.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.952+0000 7f057918a640 1 Processor -- start 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.953+0000 7f057918a640 1 -- start start 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f057918a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f05741f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f057918a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05740c9cf0 0x7f057409b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f057918a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057409b790 0x7f057409bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f057918a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05740c8fa0 con 0x7f0560005180 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f057918a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057415b270 con 0x7f057409b790 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f057918a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05741f0ef0 con 0x7f05740c9cf0 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f0572d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f05741f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f0572d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f05741f09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45890/0 (socket says 172.21.15.45:45890) 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f0572d9d640 1 -- 172.21.15.45:0/2199798408 learned_addr learned my addr 172.21.15.45:0/2199798408 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f057359e640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057409b790 0x7f057409bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f056bfff640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05740c9cf0 0x7f057409b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:04.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f0572d9d640 1 -- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05740c9cf0 msgr2=0x7f057409b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f0572d9d640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05740c9cf0 0x7f057409b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f0572d9d640 1 -- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057409b790 msgr2=0x7f057409bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f0572d9d640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057409b790 0x7f057409bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f0572d9d640 1 -- 172.21.15.45:0/2199798408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05740c8390 con 0x7f0560005180 2023-11-28T20:54:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f056bfff640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05740c9cf0 0x7f057409b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.954+0000 7f057359e640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057409b790 0x7f057409bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.955+0000 7f0572d9d640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f05741f09b0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0564011540 tx=0x7f0564011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:04.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.955+0000 7f056b7fe640 1 -- 172.21.15.45:0/2199798408 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564015930 con 0x7f0560005180 2023-11-28T20:54:04.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.955+0000 7f056b7fe640 1 -- 172.21.15.45:0/2199798408 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0564015a90 con 0x7f0560005180 2023-11-28T20:54:04.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.955+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05740c8670 con 0x7f0560005180 2023-11-28T20:54:04.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.955+0000 7f056b7fe640 1 -- 172.21.15.45:0/2199798408 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564015e60 con 0x7f0560005180 2023-11-28T20:54:04.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.955+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05740c8c30 con 0x7f0560005180 2023-11-28T20:54:04.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.956+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05740733d0 con 0x7f0560005180 2023-11-28T20:54:04.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.957+0000 7f056b7fe640 1 -- 172.21.15.45:0/2199798408 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f056400f460 con 0x7f0560005180 2023-11-28T20:54:04.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.958+0000 7f056b7fe640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f055406b080 0x7f055406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:04.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.958+0000 7f056b7fe640 1 -- 172.21.15.45:0/2199798408 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f056408a550 con 0x7f0560005180 2023-11-28T20:54:04.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.958+0000 7f056bfff640 1 -- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f055406b080 msgr2=0x7f055406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:04.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.958+0000 7f056bfff640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f055406b080 0x7f055406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:04.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:04.962+0000 7f056b7fe640 1 -- 172.21.15.45:0/2199798408 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0564054f60 con 0x7f0560005180 2023-11-28T20:54:04.966 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:04.964+0000 7ff819e2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:54:04.996 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:04.995+0000 7fcb94c1c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:05.017 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:05.015+0000 7ff819e2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:54:05.050 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:05.049+0000 7fcb94c1c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:54:05.110 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:05.109+0000 7fcb94c1c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:54:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.152+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0574066df0 con 0x7f0560005180 2023-11-28T20:54:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.152+0000 7f056b7fe640 1 -- 172.21.15.45:0/2199798408 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0564058580 con 0x7f0560005180 2023-11-28T20:54:05.154 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f055406b080 msgr2=0x7f055406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f055406b080 0x7f055406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 msgr2=0x7f05741f09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f05741f09b0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0564011540 tx=0x7f0564011a00 comp rx=0 tx=0).stop 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 shutdown_connections 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057409b790 0x7f057409bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05740c9cf0 0x7f057409b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f055406b080 0x7f055406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 --2- 172.21.15.45:0/2199798408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560005180 0x7f05741f09b0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.155+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 >> 172.21.15.45:0/2199798408 conn(0x7f05740cc930 msgr2=0x7f05740b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.156+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 shutdown_connections 2023-11-28T20:54:05.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.156+0000 7f057918a640 1 -- 172.21.15.45:0/2199798408 wait complete. 2023-11-28T20:54:05.164 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:05.163+0000 7fcb94c1c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:54:05.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:54:05.399 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:05.394+0000 7ff819e2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:54:05.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.441+0000 7f4ef8acf640 1 Processor -- start 2023-11-28T20:54:05.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.441+0000 7f4ef8acf640 1 -- start start 2023-11-28T20:54:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.443+0000 7f4ef8acf640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4ef4066900 0x7f4ef40cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.443+0000 7f4ef8acf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40cf500 con 0x7f4ef415d7a0 2023-11-28T20:54:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.443+0000 7f4ef8acf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40cf7e0 con 0x7f4ef406d620 2023-11-28T20:54:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.443+0000 7f4ef8acf640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40cfda0 con 0x7f4ef4066900 2023-11-28T20:54:05.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.443+0000 7f4ef259c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4ef415d7a0 0x7f4ef40661f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49922/0 (socket says 172.21.15.45:49922) 2023-11-28T20:54:05.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.443+0000 7f4ef259c640 1 -- 172.21.15.45:0/3947474975 learned_addr learned my addr 172.21.15.45:0/3947474975 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ee9d9b640 1 --2- 172.21.15.45:0/3947474975 >> v2:172.21.15.190:3300/0 conn(0x7f4ef4066900 0x7f4ef40cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527545335 0 0) 0x7f4ef40cf500 con 0x7f4ef415d7a0 2023-11-28T20:54:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee0003520 con 0x7f4ef415d7a0 2023-11-28T20:54:05.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1978593232 0 0) 0x7f4ee0003520 con 0x7f4ef415d7a0 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 >> v2:172.21.15.190:3300/0 conn(0x7f4ef4066900 msgr2=0x7f4ef40cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 --2- 172.21.15.45:0/3947474975 >> v2:172.21.15.190:3300/0 conn(0x7f4ef4066900 0x7f4ef40cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 >> v1:172.21.15.190:6789/0 conn(0x7f4ef406d620 legacy=0x7f4ef406abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef40cfac0 con 0x7f4ef415d7a0 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4ee40019f0 con 0x7f4ef415d7a0 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 >> v1:172.21.15.45:6789/0 conn(0x7f4ef415d7a0 legacy=0x7f4ef40661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ee00039c0 0x7f4ee0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00047c0 0x7f4ee0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee0005180 0x7f4ee000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40cfda0 con 0x7f4ee00047c0 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee000c080 con 0x7f4ee0005180 2023-11-28T20:54:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.444+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee000c240 con 0x7f4ee00039c0 2023-11-28T20:54:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.445+0000 7f4ee9d9b640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00047c0 0x7f4ee0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.445+0000 7f4ef259c640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ee00039c0 0x7f4ee0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.445+0000 7f4ee9d9b640 1 -- 172.21.15.45:0/3947474975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ee00039c0 msgr2=0x7f4ee0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.445+0000 7f4ee9d9b640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ee00039c0 0x7f4ee0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.445+0000 7f4ee9d9b640 1 -- 172.21.15.45:0/3947474975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee0005180 msgr2=0x7f4ee000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.445+0000 7f4ee9d9b640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee0005180 0x7f4ee000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.445+0000 7f4ee9d9b640 1 -- 172.21.15.45:0/3947474975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0003520 con 0x7f4ee00047c0 2023-11-28T20:54:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.445+0000 7f4ef2d9d640 1 -- 172.21.15.45:0/3947474975 reap_dead start 2023-11-28T20:54:05.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.445+0000 7f4ef259c640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ee00039c0 0x7f4ee0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.446+0000 7f4ee9d9b640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00047c0 0x7f4ee0004c40 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009ac0 tx=0x7f4ee00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93dc9f8fa36ef131 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.446+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc02c6d0 con 0x7f4ee00047c0 2023-11-28T20:54:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.446+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4edc02c830 con 0x7f4ee00047c0 2023-11-28T20:54:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.446+0000 7f4ef1d9b640 1 -- 172.21.15.45:0/3947474975 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc02cbc0 con 0x7f4ee00047c0 2023-11-28T20:54:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.447+0000 7f4ef8acf640 1 -- 172.21.15.45:0/3947474975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00047c0 msgr2=0x7f4ee0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.447+0000 7f4ef8acf640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00047c0 0x7f4ee0004c40 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009ac0 tx=0x7f4ee00057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.447+0000 7f4ef8acf640 1 -- 172.21.15.45:0/3947474975 shutdown_connections 2023-11-28T20:54:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.447+0000 7f4ef8acf640 1 --2- 172.21.15.45:0/3947474975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ee00047c0 0x7f4ee0004c40 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.447+0000 7f4ef8acf640 1 -- 172.21.15.45:0/3947474975 >> 172.21.15.45:0/3947474975 conn(0x7f4ef40cc9c0 msgr2=0x7f4ef40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.447+0000 7f4ef8acf640 1 -- 172.21.15.45:0/3947474975 shutdown_connections 2023-11-28T20:54:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.447+0000 7f4ef8acf640 1 -- 172.21.15.45:0/3947474975 wait complete. 2023-11-28T20:54:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.449+0000 7f4ef8acf640 1 Processor -- start 2023-11-28T20:54:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.449+0000 7f4ef8acf640 1 -- start start 2023-11-28T20:54:05.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.450+0000 7f4ef8acf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee00047c0 0x7f4ef4158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.450+0000 7f4ef8acf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ef40c1ca0 0x7f4ef40c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.450+0000 7f4ef8acf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ef409fbf0 0x7f4ef40a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.450+0000 7f4ef8acf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40a11c0 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.450+0000 7f4ef8acf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40a0920 con 0x7f4ee00047c0 2023-11-28T20:54:05.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.450+0000 7f4ef8acf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40a0c00 con 0x7f4ef409fbf0 2023-11-28T20:54:05.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.450+0000 7f4ee9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ef40c1ca0 0x7f4ef40c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.450+0000 7f4ee9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ef40c1ca0 0x7f4ef40c2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45908/0 (socket says 172.21.15.45:45908) 2023-11-28T20:54:05.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.450+0000 7f4ee9d9b640 1 -- 172.21.15.45:0/160824552 learned_addr learned my addr 172.21.15.45:0/160824552 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:05.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ef259c640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee00047c0 0x7f4ef4158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ee9d9b640 1 -- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ef409fbf0 msgr2=0x7f4ef40a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ef2d9d640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ef409fbf0 0x7f4ef40a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ee9d9b640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ef409fbf0 0x7f4ef40a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ee9d9b640 1 -- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee00047c0 msgr2=0x7f4ef4158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ee9d9b640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee00047c0 0x7f4ef4158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ee9d9b640 1 -- 172.21.15.45:0/160824552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef40a0ee0 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ef259c640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee00047c0 0x7f4ef4158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:05.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ef2d9d640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ef409fbf0 0x7f4ef40a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:05.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4ee9d9b640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ef40c1ca0 0x7f4ef40c2100 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4edc02c270 tx=0x7f4edc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:05.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4eeb7fe640 1 -- 172.21.15.45:0/160824552 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc003810 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4eeb7fe640 1 -- 172.21.15.45:0/160824552 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4edc003970 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.451+0000 7f4eeb7fe640 1 -- 172.21.15.45:0/160824552 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc003ce0 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.452+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef415dfb0 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.452+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef40a05c0 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.453+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef40733d0 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.454+0000 7f4eeb7fe640 1 -- 172.21.15.45:0/160824552 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f4edc040660 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.455+0000 7f4eeb7fe640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f4ed406b080 0x7f4ed406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.455+0000 7f4eeb7fe640 1 -- 172.21.15.45:0/160824552 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f4edc0abe90 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.455+0000 7f4ef259c640 1 -- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f4ed406b080 msgr2=0x7f4ed406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:05.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.455+0000 7f4ef259c640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f4ed406b080 0x7f4ed406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:05.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.459+0000 7f4eeb7fe640 1 -- 172.21.15.45:0/160824552 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4edc076940 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.558 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:05.555+0000 7ff819e2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:54:05.573 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:05.572+0000 7fcb94c1c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:54:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.652+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4ef40b0320 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.653+0000 7f4eeb7fe640 1 -- 172.21.15.45:0/160824552 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4edc0044e0 con 0x7f4ef40c1ca0 2023-11-28T20:54:05.654 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:54:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.655+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f4ed406b080 msgr2=0x7f4ed406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.655+0000 7f4ef8acf640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f4ed406b080 0x7f4ed406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.655+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ef40c1ca0 msgr2=0x7f4ef40c2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.655+0000 7f4ef8acf640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ef40c1ca0 0x7f4ef40c2100 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4edc02c270 tx=0x7f4edc004730 comp rx=0 tx=0).stop 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.655+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 shutdown_connections 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.655+0000 7f4ef8acf640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ef409fbf0 0x7f4ef40a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.655+0000 7f4ef8acf640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f4ed406b080 0x7f4ed406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.655+0000 7f4ef8acf640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ef40c1ca0 0x7f4ef40c2100 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.655+0000 7f4ef8acf640 1 --2- 172.21.15.45:0/160824552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ee00047c0 0x7f4ef4158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.656+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 >> 172.21.15.45:0/160824552 conn(0x7f4ef40cc9c0 msgr2=0x7f4ef406aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.656+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 shutdown_connections 2023-11-28T20:54:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.656+0000 7f4ef8acf640 1 -- 172.21.15.45:0/160824552 wait complete. 2023-11-28T20:54:05.665 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:05.663+0000 7ff819e2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:54:05.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:54:05.715 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:05.713+0000 7ff819e2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:54:05.744 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:05.743+0000 7fcb94c1c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:54:05.822 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:05.820+0000 7ff819e2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:05.859 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:05.858+0000 7fcb94c1c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:54:05.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.871+0000 7f73411d0640 1 Processor -- start 2023-11-28T20:54:05.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.871+0000 7f73411d0640 1 -- start start 2023-11-28T20:54:05.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f73411d0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f733c143f50 0x7f733c0ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f73411d0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f733c1445b0 0x7f733c0ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f73411d0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c0cda80 con 0x7f733c0cea10 2023-11-28T20:54:05.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f73411d0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733c144330 con 0x7f733c1445b0 2023-11-28T20:54:05.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f73411d0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733c0cc2b0 con 0x7f733c143f50 2023-11-28T20:54:05.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f7333fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f733c1445b0 0x7f733c0ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f7333fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f733c1445b0 0x7f733c0ce4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45916/0 (socket says 172.21.15.45:45916) 2023-11-28T20:54:05.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f7333fff640 1 -- 172.21.15.45:0/2153527481 learned_addr learned my addr 172.21.15.45:0/2153527481 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:05.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f733ad9d640 1 --2- 172.21.15.45:0/2153527481 >> v2:172.21.15.190:3300/0 conn(0x7f733c143f50 0x7f733c0ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f7333fff640 1 -- 172.21.15.45:0/2153527481 >> v2:172.21.15.190:3300/0 conn(0x7f733c143f50 msgr2=0x7f733c0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:05.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.873+0000 7f7333fff640 1 --2- 172.21.15.45:0/2153527481 >> v2:172.21.15.190:3300/0 conn(0x7f733c143f50 0x7f733c0ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.874+0000 7f7333fff640 1 -- 172.21.15.45:0/2153527481 >> v1:172.21.15.190:6789/0 conn(0x7f733c0cea10 legacy=0x7f733c0661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:05.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.874+0000 7f7333fff640 1 -- 172.21.15.45:0/2153527481 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c0cc590 con 0x7f733c1445b0 2023-11-28T20:54:05.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.875+0000 7f7333fff640 1 --2- 172.21.15.45:0/2153527481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c1445b0 0x7f733c0ce4d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f733c0cd570 tx=0x7f732402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41eef89e584e01e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:05.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.875+0000 7f733a59c640 1 -- 172.21.15.45:0/2153527481 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324031870 con 0x7f733c1445b0 2023-11-28T20:54:05.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.875+0000 7f733a59c640 1 -- 172.21.15.45:0/2153527481 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73240319d0 con 0x7f733c1445b0 2023-11-28T20:54:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.875+0000 7f733a59c640 1 -- 172.21.15.45:0/2153527481 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324031d60 con 0x7f733c1445b0 2023-11-28T20:54:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.876+0000 7f73411d0640 1 -- 172.21.15.45:0/2153527481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c1445b0 msgr2=0x7f733c0ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.876+0000 7f73411d0640 1 --2- 172.21.15.45:0/2153527481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c1445b0 0x7f733c0ce4d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f733c0cd570 tx=0x7f732402b910 comp rx=0 tx=0).stop 2023-11-28T20:54:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.876+0000 7f73411d0640 1 -- 172.21.15.45:0/2153527481 shutdown_connections 2023-11-28T20:54:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.876+0000 7f73411d0640 1 --2- 172.21.15.45:0/2153527481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c1445b0 0x7f733c0ce4d0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.876+0000 7f73411d0640 1 --2- 172.21.15.45:0/2153527481 >> v2:172.21.15.190:3300/0 conn(0x7f733c143f50 0x7f733c0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.876+0000 7f73411d0640 1 -- 172.21.15.45:0/2153527481 >> 172.21.15.45:0/2153527481 conn(0x7f733c0cc930 msgr2=0x7f733c0ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.876+0000 7f73411d0640 1 -- 172.21.15.45:0/2153527481 shutdown_connections 2023-11-28T20:54:05.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.876+0000 7f73411d0640 1 -- 172.21.15.45:0/2153527481 wait complete. 2023-11-28T20:54:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.878+0000 7f73411d0640 1 Processor -- start 2023-11-28T20:54:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.878+0000 7f73411d0640 1 -- start start 2023-11-28T20:54:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f73411d0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f733c143f50 0x7f733c09d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f73411d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f733c1445b0 0x7f733c09a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f73411d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c09c580 0x7f733c15aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f73411d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c09b570 con 0x7f733c09c580 2023-11-28T20:54:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f73411d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c09b7c0 con 0x7f733c1445b0 2023-11-28T20:54:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f73411d0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c09baa0 con 0x7f733c143f50 2023-11-28T20:54:05.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f733b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c09c580 0x7f733c15aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f733b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c09c580 0x7f733c15aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45926/0 (socket says 172.21.15.45:45926) 2023-11-28T20:54:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f733b59e640 1 -- 172.21.15.45:0/4015896076 learned_addr learned my addr 172.21.15.45:0/4015896076 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f733ad9d640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f733c143f50 0x7f733c09d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f7333fff640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f733c1445b0 0x7f733c09a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f733b59e640 1 -- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f733c143f50 msgr2=0x7f733c09d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f733b59e640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f733c143f50 0x7f733c09d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f733b59e640 1 -- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f733c1445b0 msgr2=0x7f733c09a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:05.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f733b59e640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f733c1445b0 0x7f733c09a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:05.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.879+0000 7f733b59e640 1 -- 172.21.15.45:0/4015896076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73240094d0 con 0x7f733c09c580 2023-11-28T20:54:05.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.880+0000 7f733b59e640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c09c580 0x7f733c15aae0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f733400c700 tx=0x7f733400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:05.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.880+0000 7f73337fe640 1 -- 172.21.15.45:0/4015896076 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733400aa50 con 0x7f733c09c580 2023-11-28T20:54:05.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.880+0000 7f73337fe640 1 -- 172.21.15.45:0/4015896076 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733400abb0 con 0x7f733c09c580 2023-11-28T20:54:05.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.880+0000 7f73337fe640 1 -- 172.21.15.45:0/4015896076 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733400ce10 con 0x7f733c09c580 2023-11-28T20:54:05.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.880+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c09bd80 con 0x7f733c09c580 2023-11-28T20:54:05.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.880+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c09ca30 con 0x7f733c09c580 2023-11-28T20:54:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.882+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c09c2a0 con 0x7f733c09c580 2023-11-28T20:54:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.882+0000 7f73337fe640 1 -- 172.21.15.45:0/4015896076 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f7334012420 con 0x7f733c09c580 2023-11-28T20:54:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.883+0000 7f73337fe640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f731c06b080 0x7f731c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.883+0000 7f73337fe640 1 -- 172.21.15.45:0/4015896076 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f733408a1b0 con 0x7f733c09c580 2023-11-28T20:54:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.884+0000 7f733ad9d640 1 -- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f731c06b080 msgr2=0x7f731c06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:05.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.884+0000 7f733ad9d640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f731c06b080 0x7f731c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:05.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:05.888+0000 7f73337fe640 1 -- 172.21.15.45:0/4015896076 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7334054c50 con 0x7f733c09c580 2023-11-28T20:54:05.930 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:05.928+0000 7ff819e2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:54:05.933 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:05.931+0000 7fcb94c1c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:54:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.084+0000 7f733ad9d640 1 -- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f731c06b080 msgr2=0x7f731c06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.084+0000 7f733ad9d640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f731c06b080 0x7f731c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:06.088 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:06.085+0000 7ff819e2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:06.090 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:06.088+0000 7fcb94c1c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:06.138 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:06.135+0000 7ff819e2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:06.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.161+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f733c073440 con 0x7f733c09c580 2023-11-28T20:54:06.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.162+0000 7f73337fe640 1 -- 172.21.15.45:0/4015896076 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v134) v1 ==== 80+0+476 (secure 0 0 0) 0x7f7334003eb0 con 0x7f733c09c580 2023-11-28T20:54:06.164 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3135688549 2023-11-29T20:53:49.553172+0000 2023-11-28T20:54:06.164 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/369403092 2023-11-29T20:53:49.553172+0000 2023-11-28T20:54:06.165 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/3959324769 2023-11-29T20:53:49.553172+0000 2023-11-28T20:54:06.165 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/3959324769 2023-11-29T20:53:49.553172+0000 2023-11-28T20:54:06.165 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1007604882 2023-11-29T20:53:49.553172+0000 2023-11-28T20:54:06.165 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/685209934 2023-11-29T20:53:49.553172+0000 2023-11-28T20:54:06.165 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1745589824 2023-11-29T20:53:49.553172+0000 2023-11-28T20:54:06.165 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1706225465 2023-11-29T20:53:49.553172+0000 2023-11-28T20:54:06.166 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2023-11-28T20:54:06.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f731c06b080 msgr2=0x7f731c06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:06.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f731c06b080 0x7f731c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:06.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c09c580 msgr2=0x7f733c15aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:06.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c09c580 0x7f733c15aae0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f733400c700 tx=0x7f733400cbc0 comp rx=0 tx=0).stop 2023-11-28T20:54:06.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 shutdown_connections 2023-11-28T20:54:06.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f731c06b080 0x7f731c06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:06.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733c09c580 0x7f733c15aae0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:06.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f733c1445b0 0x7f733c09a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:06.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 --2- 172.21.15.45:0/4015896076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f733c143f50 0x7f733c09d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:06.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 >> 172.21.15.45:0/4015896076 conn(0x7f733c0cc930 msgr2=0x7f733c0b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:06.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 shutdown_connections 2023-11-28T20:54:06.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.166+0000 7f73411d0640 1 -- 172.21.15.45:0/4015896076 wait complete. 2023-11-28T20:54:06.194 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T20:54:06.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:06.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T20:54:06.217 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:06.215+0000 7fcb94c1c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:54:06.258 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:06.256+0000 7ff819e2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:54:06.303 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:54:06.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:06.303 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T20:54:06.386 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:06.383+0000 7fcb94c1c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:06.394 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:54:06.395 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:06.438 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:06.435+0000 7fcb94c1c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:06.448 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:54:06.448 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:54:06.448 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:06.504 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:54:06.504 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:54:06.504 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:06.556 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:54:06.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:06.566 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:06.563+0000 7fcb94c1c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:54:06.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.775+0000 7f7c02cc2640 1 Processor -- start 2023-11-28T20:54:06.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.775+0000 7f7c02cc2640 1 -- start start 2023-11-28T20:54:06.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.776+0000 7f7c02cc2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7bfc0d02d0 0x7f7bfc0d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.776+0000 7f7c02cc2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bfc0d2230 con 0x7f7bfc15d550 2023-11-28T20:54:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.776+0000 7f7c02cc2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bfc0d2510 con 0x7f7bfc0d1ab0 2023-11-28T20:54:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.776+0000 7f7c02cc2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bfc0d27f0 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.776+0000 7f7c014bf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7bfc0d02d0 0x7f7bfc0d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.777+0000 7f7c01cc0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7bfc15d550 0x7f7bfc0cfbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49936/0 (socket says 172.21.15.45:49936) 2023-11-28T20:54:06.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.777+0000 7f7c01cc0640 1 -- 172.21.15.45:0/1396205457 learned_addr learned my addr 172.21.15.45:0/1396205457 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:06.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.777+0000 7f7c014bf640 1 -- 172.21.15.45:0/1396205457 >> v1:172.21.15.190:6789/0 conn(0x7f7bfc0d1ab0 legacy=0x7f7bfc0b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:06.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.777+0000 7f7c00cbe640 1 -- 172.21.15.45:0/1396205457 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324022327 0 0) 0x7f7bfc0d2230 con 0x7f7bfc15d550 2023-11-28T20:54:06.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.777+0000 7f7c014bf640 1 -- 172.21.15.45:0/1396205457 >> v1:172.21.15.45:6789/0 conn(0x7f7bfc15d550 legacy=0x7f7bfc0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:06.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.777+0000 7f7c014bf640 1 -- 172.21.15.45:0/1396205457 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc0b6a30 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.778+0000 7f7c014bf640 1 --2- 172.21.15.45:0/1396205457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 0x7f7bfc0d1570 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f7bec0097a0 tx=0x7f7bec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516155ec551c35f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.778+0000 7f7c00cbe640 1 -- 172.21.15.45:0/1396205457 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec030610 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.779+0000 7f7c00cbe640 1 -- 172.21.15.45:0/1396205457 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec030770 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.779+0000 7f7c00cbe640 1 -- 172.21.15.45:0/1396205457 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec030b00 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.779+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1396205457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 msgr2=0x7f7bfc0d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.779+0000 7f7c02cc2640 1 --2- 172.21.15.45:0/1396205457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 0x7f7bfc0d1570 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f7bec0097a0 tx=0x7f7bec02b960 comp rx=0 tx=0).stop 2023-11-28T20:54:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.779+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1396205457 shutdown_connections 2023-11-28T20:54:06.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.779+0000 7f7c02cc2640 1 --2- 172.21.15.45:0/1396205457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 0x7f7bfc0d1570 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:06.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.779+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1396205457 >> 172.21.15.45:0/1396205457 conn(0x7f7bfc06ac70 msgr2=0x7f7bfc0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:06.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.779+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1396205457 shutdown_connections 2023-11-28T20:54:06.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.779+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1396205457 wait complete. 2023-11-28T20:54:06.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.781+0000 7f7c02cc2640 1 Processor -- start 2023-11-28T20:54:06.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.781+0000 7f7c02cc2640 1 -- start start 2023-11-28T20:54:06.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c02cc2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 0x7f7bfc147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:06.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c02cc2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bfc0d1ab0 0x7f7bfc0c7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:06.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c02cc2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bfc15d550 0x7f7bfc0c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:06.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c02cc2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc148890 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c02cc2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc0c82f0 con 0x7f7bfc0d1ab0 2023-11-28T20:54:06.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c02cc2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc0c85d0 con 0x7f7bfc15d550 2023-11-28T20:54:06.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c01cc0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 0x7f7bfc147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:06.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c01cc0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 0x7f7bfc147f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45934/0 (socket says 172.21.15.45:45934) 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c01cc0640 1 -- 172.21.15.45:0/1831521755 learned_addr learned my addr 172.21.15.45:0/1831521755 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c014bf640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bfc0d1ab0 0x7f7bfc0c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.782+0000 7f7c024c1640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bfc15d550 0x7f7bfc0c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7c01cc0640 1 -- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bfc15d550 msgr2=0x7f7bfc0c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7c01cc0640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bfc15d550 0x7f7bfc0c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7c01cc0640 1 -- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bfc0d1ab0 msgr2=0x7f7bfc0c7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7c01cc0640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bfc0d1ab0 0x7f7bfc0c7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7c01cc0640 1 -- 172.21.15.45:0/1831521755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec009400 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7c01cc0640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 0x7f7bfc147f40 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800c3f0 tx=0x7f7bf800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7bf27fc640 1 -- 172.21.15.45:0/1831521755 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf80074e0 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7bf27fc640 1 -- 172.21.15.45:0/1831521755 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf8007640 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7bf27fc640 1 -- 172.21.15.45:0/1831521755 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8004a00 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bfc148aa0 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.783+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bfc15bc30 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.785+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bfc148fc0 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.785+0000 7f7bf27fc640 1 -- 172.21.15.45:0/1831521755 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f7bf8004b60 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.786+0000 7f7bf27fc640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f7be006b0b0 0x7f7be006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:06.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.786+0000 7f7bf27fc640 1 -- 172.21.15.45:0/1831521755 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f7bf80899b0 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.787+0000 7f7c014bf640 1 -- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f7be006b0b0 msgr2=0x7f7be006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:06.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.787+0000 7f7c014bf640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f7be006b0b0 0x7f7be006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:06.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.790+0000 7f7bf27fc640 1 -- 172.21.15.45:0/1831521755 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bf8054450 con 0x7f7bfc0d02d0 2023-11-28T20:54:06.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.988+0000 7f7c014bf640 1 -- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f7be006b0b0 msgr2=0x7f7be006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:06.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:06.988+0000 7f7c014bf640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f7be006b0b0 0x7f7be006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:07.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.025+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bfc15cd00 con 0x7f7bfc0d02d0 2023-11-28T20:54:07.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.026+0000 7f7bf27fc640 1 -- 172.21.15.45:0/1831521755 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f7bf8057a70 con 0x7f7bfc0d02d0 2023-11-28T20:54:07.030 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:07.030 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/544145878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":544145878},{"type":"v1","addr":"172.21.15.190:6836","nonce":544145878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2464692926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2464692926},{"type":"v1","addr":"172.21.15.45:6835","nonce":2464692926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2048455360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2048455360},{"type":"v1","addr":"172.21.15.190:6837","nonce":2048455360}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:53:35.282168+0000","modified":"2023-11-28T20:53:38.346980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2464465186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2464465186},{"type":"v1","addr":"172.21.15.45:6834","nonce":2464465186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-28T20:54:07.030 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 59 2023-11-28T20:54:07.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f7be006b0b0 msgr2=0x7f7be006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:07.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f7be006b0b0 0x7f7be006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 msgr2=0x7f7bfc147f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 0x7f7bfc147f40 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800c3f0 tx=0x7f7bf800c8b0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 shutdown_connections 2023-11-28T20:54:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bfc15d550 0x7f7bfc0c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bfc0d1ab0 0x7f7bfc0c7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f7be006b0b0 0x7f7be006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 --2- 172.21.15.45:0/1831521755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bfc0d02d0 0x7f7bfc147f40 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 >> 172.21.15.45:0/1831521755 conn(0x7f7bfc06ac70 msgr2=0x7f7bfc15e580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 shutdown_connections 2023-11-28T20:54:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.029+0000 7f7c02cc2640 1 -- 172.21.15.45:0/1831521755 wait complete. 2023-11-28T20:54:07.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:07.109 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.107+0000 7ff819e2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:54:07.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.243+0000 7f6a0a68c640 1 Processor -- start 2023-11-28T20:54:07.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.243+0000 7f6a0a68c640 1 -- start start 2023-11-28T20:54:07.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.245+0000 7f6a0a68c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6a0415d3d0 0x7f6a040af500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.245+0000 7f6a0a68c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0415d7b0 con 0x7f6a040aee40 2023-11-28T20:54:07.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.245+0000 7f6a0a68c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a040b3320 con 0x7f6a0406b510 2023-11-28T20:54:07.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.245+0000 7f6a0a68c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a040aeaf0 con 0x7f6a0415d3d0 2023-11-28T20:54:07.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f6a08e89640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6a040aee40 0x7f6a040b4bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49944/0 (socket says 172.21.15.45:49944) 2023-11-28T20:54:07.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f6a08e89640 1 -- 172.21.15.45:0/2293253300 learned_addr learned my addr 172.21.15.45:0/2293253300 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:07.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f6a0968a640 1 --2- 172.21.15.45:0/2293253300 >> v2:172.21.15.190:3300/0 conn(0x7f6a0415d3d0 0x7f6a040af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589719561 0 0) 0x7f6a0415d7b0 con 0x7f6a040aee40 2023-11-28T20:54:07.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f0003520 con 0x7f6a040aee40 2023-11-28T20:54:07.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415930649 0 0) 0x7f69f0003520 con 0x7f6a040aee40 2023-11-28T20:54:07.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 >> v2:172.21.15.190:3300/0 conn(0x7f6a0415d3d0 msgr2=0x7f6a040af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:07.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f69fbfff640 1 --2- 172.21.15.45:0/2293253300 >> v2:172.21.15.190:3300/0 conn(0x7f6a0415d3d0 0x7f6a040af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 >> v1:172.21.15.190:6789/0 conn(0x7f6a0406b510 legacy=0x7f6a0406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a04072140 con 0x7f6a040aee40 2023-11-28T20:54:07.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.246+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f69f4001670 con 0x7f6a040aee40 2023-11-28T20:54:07.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 >> v1:172.21.15.45:6789/0 conn(0x7f6a040aee40 legacy=0x7f6a040b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f69fbfff640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69f00039c0 0x7f69f0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f69fbfff640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f69f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f69fbfff640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69f0005180 0x7f69f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040aeaf0 con 0x7f69f00047c0 2023-11-28T20:54:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f000c080 con 0x7f69f0005180 2023-11-28T20:54:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f000c240 con 0x7f69f00039c0 2023-11-28T20:54:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f6a08e89640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f69f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f6a0968a640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69f00039c0 0x7f69f0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f6a08e89640 1 -- 172.21.15.45:0/2293253300 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69f00039c0 msgr2=0x7f69f0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f6a08e89640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69f00039c0 0x7f69f0003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f6a08e89640 1 -- 172.21.15.45:0/2293253300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69f0005180 msgr2=0x7f69f000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f6a08e89640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69f0005180 0x7f69f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f6a08e89640 1 -- 172.21.15.45:0/2293253300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0003520 con 0x7f69f00047c0 2023-11-28T20:54:07.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.247+0000 7f6a09e8b640 1 -- 172.21.15.45:0/2293253300 reap_dead start 2023-11-28T20:54:07.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.248+0000 7f6a08e89640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f69f0004c40 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f69f4007fb0 tx=0x7f69f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449d9d6afd1e8ee9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:07.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.249+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4005ad0 con 0x7f69f00047c0 2023-11-28T20:54:07.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.249+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f4005c30 con 0x7f69f00047c0 2023-11-28T20:54:07.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.249+0000 7f69fbfff640 1 -- 172.21.15.45:0/2293253300 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f402c7c0 con 0x7f69f00047c0 2023-11-28T20:54:07.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.249+0000 7f6a0a68c640 1 -- 172.21.15.45:0/2293253300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 msgr2=0x7f69f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.249+0000 7f6a0a68c640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f69f0004c40 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f69f4007fb0 tx=0x7f69f00057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.250+0000 7f6a0a68c640 1 -- 172.21.15.45:0/2293253300 shutdown_connections 2023-11-28T20:54:07.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.250+0000 7f6a0a68c640 1 --2- 172.21.15.45:0/2293253300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f69f0004c40 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.250+0000 7f6a0a68c640 1 -- 172.21.15.45:0/2293253300 >> 172.21.15.45:0/2293253300 conn(0x7f6a040712d0 msgr2=0x7f6a040716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:07.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.250+0000 7f6a0a68c640 1 -- 172.21.15.45:0/2293253300 shutdown_connections 2023-11-28T20:54:07.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.250+0000 7f6a0a68c640 1 -- 172.21.15.45:0/2293253300 wait complete. 2023-11-28T20:54:07.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.251+0000 7f6a0a68c640 1 Processor -- start 2023-11-28T20:54:07.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.252+0000 7f6a0a68c640 1 -- start start 2023-11-28T20:54:07.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0a68c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f6a04155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0a68c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a04158800 0x7f6a040cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0a68c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a04140790 0x7f6a04140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0a68c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040af2f0 con 0x7f69f00047c0 2023-11-28T20:54:07.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0a68c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0415e930 con 0x7f6a04140790 2023-11-28T20:54:07.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0a68c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040cf6f0 con 0x7f6a04158800 2023-11-28T20:54:07.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0968a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f6a04155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0968a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f6a04155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45948/0 (socket says 172.21.15.45:45948) 2023-11-28T20:54:07.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0968a640 1 -- 172.21.15.45:0/236464900 learned_addr learned my addr 172.21.15.45:0/236464900 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:07.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a08e89640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a04158800 0x7f6a040cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0968a640 1 -- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a04158800 msgr2=0x7f6a040cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0968a640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a04158800 0x7f6a040cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0968a640 1 -- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a04140790 msgr2=0x7f6a04140c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0968a640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a04140790 0x7f6a04140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.253+0000 7f6a0968a640 1 -- 172.21.15.45:0/236464900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a04141150 con 0x7f69f00047c0 2023-11-28T20:54:07.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.254+0000 7f6a0968a640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f6a04155950 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6a000107f0 tx=0x7f6a00010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:07.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.254+0000 7f69f9ffb640 1 -- 172.21.15.45:0/236464900 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0000eb60 con 0x7f69f00047c0 2023-11-28T20:54:07.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.254+0000 7f69f9ffb640 1 -- 172.21.15.45:0/236464900 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0000ecc0 con 0x7f69f00047c0 2023-11-28T20:54:07.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.254+0000 7f69f9ffb640 1 -- 172.21.15.45:0/236464900 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00010ea0 con 0x7f69f00047c0 2023-11-28T20:54:07.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.254+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a04158c80 con 0x7f69f00047c0 2023-11-28T20:54:07.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.254+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0415af90 con 0x7f69f00047c0 2023-11-28T20:54:07.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.255+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0409e100 con 0x7f69f00047c0 2023-11-28T20:54:07.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.256+0000 7f69f9ffb640 1 -- 172.21.15.45:0/236464900 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 64) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f6a0001c400 con 0x7f69f00047c0 2023-11-28T20:54:07.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.257+0000 7f69f9ffb640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f69e806b080 0x7f69e806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.257+0000 7f69f9ffb640 1 -- 172.21.15.45:0/236464900 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f6a0008a180 con 0x7f69f00047c0 2023-11-28T20:54:07.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.257+0000 7f6a08e89640 1 -- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f69e806b080 msgr2=0x7f69e806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3157274352 2023-11-28T20:54:07.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.257+0000 7f6a08e89640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f69e806b080 0x7f69e806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:07.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.261+0000 7f69f9ffb640 1 -- 172.21.15.45:0/236464900 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a00054c40 con 0x7f69f00047c0 2023-11-28T20:54:07.405 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.403+0000 7ff819e2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:54:07.418 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:07.416+0000 7fcb94c1c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:54:07.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.454+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a040cc5e0 con 0x7f69f00047c0 2023-11-28T20:54:07.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.455+0000 7f69f9ffb640 1 -- 172.21.15.45:0/236464900 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+17211 (secure 0 0 0) 0x7f6a00058260 con 0x7f69f00047c0 2023-11-28T20:54:07.458 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:07.458 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":134,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:53:49.553240+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:53:33.200533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-11-28T20:53:34.300717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3135688549":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/369403092":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6833/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6832/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1007604882":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/685209934":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1745589824":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1706225465":"2023-11-29T20:53:49.553172+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f69e806b080 msgr2=0x7f69e806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T20:54:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f69e806b080 0x7f69e806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 msgr2=0x7f6a04155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f6a04155950 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6a000107f0 tx=0x7f6a00010cb0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 shutdown_connections 2023-11-28T20:54:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a04140790 0x7f6a04140c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6a04158800 0x7f6a040cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.190:6832/3157274352,v1:172.21.15.190:6833/3157274352] conn(0x7f69e806b080 0x7f69e806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 --2- 172.21.15.45:0/236464900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f00047c0 0x7f6a04155950 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 >> 172.21.15.45:0/236464900 conn(0x7f6a040712d0 msgr2=0x7f6a0415d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 shutdown_connections 2023-11-28T20:54:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.458+0000 7f6a0a68c640 1 -- 172.21.15.45:0/236464900 wait complete. 2023-11-28T20:54:07.461 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.459+0000 7ff819e2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:54:07.479 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:53:33.200533+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:53:34.300717+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:54:07.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:54:07.577 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.575+0000 7ff819e2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:54:07.664 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.662+0000 7ff79e07f640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.665 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.662+0000 7ff79e07f640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.665 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.662+0000 7ff79e07f640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.665 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.662+0000 7ff79e07f640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.665 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.662+0000 7ff79e07f640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.665 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.663+0000 7ff79b87a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.666 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.663+0000 7ff79b87a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.666 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.663+0000 7ff79b87a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.666 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.663+0000 7ff79b87a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.666 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:07.663+0000 7ff79b87a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.683+0000 7f0bb359e640 1 Processor -- start 2023-11-28T20:54:07.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.683+0000 7f0bb359e640 1 -- start start 2023-11-28T20:54:07.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.684+0000 7f0bb359e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0bb40d0040 0x7f0bb40d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.684+0000 7f0bb359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 0x7f0bb40d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.684+0000 7f0bb359e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb40b90a0 con 0x7f0bb40b2290 2023-11-28T20:54:07.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.684+0000 7f0bb359e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb40d1920 con 0x7f0bb40d0040 2023-11-28T20:54:07.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.684+0000 7f0bb359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40d2dc0 con 0x7f0bb40b1d40 2023-11-28T20:54:07.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb2d9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 0x7f0bb40d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb259c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0bb40d0040 0x7f0bb40d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb2d9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 0x7f0bb40d3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:49972/0 (socket says 172.21.15.45:49972) 2023-11-28T20:54:07.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb2d9d640 1 -- 172.21.15.45:0/3591574428 learned_addr learned my addr 172.21.15.45:0/3591574428 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:07.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb1d9b640 1 --1- 172.21.15.45:0/3591574428 >> v1:172.21.15.190:6789/0 conn(0x7f0bb40b2290 0x7f0bb40d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:45548/0 (socket says 172.21.15.45:45548) 2023-11-28T20:54:07.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb259c640 1 -- 172.21.15.45:0/3591574428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 msgr2=0x7f0bb40d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:07.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb259c640 1 --2- 172.21.15.45:0/3591574428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 0x7f0bb40d3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb259c640 1 -- 172.21.15.45:0/3591574428 >> v1:172.21.15.190:6789/0 conn(0x7f0bb40b2290 legacy=0x7f0bb40d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb259c640 1 -- 172.21.15.45:0/3591574428 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb40d0960 con 0x7f0bb40d0040 2023-11-28T20:54:07.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.685+0000 7f0bb159a640 1 -- 172.21.15.45:0/3591574428 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275185330 0 0) 0x7f0bb40b90a0 con 0x7f0bb40b2290 2023-11-28T20:54:07.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.686+0000 7f0bb259c640 1 --2- 172.21.15.45:0/3591574428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bb40d0040 0x7f0bb40d0420 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40097a0 tx=0x7f0ba402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=892d664f627c99bc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:07.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb159a640 1 -- 172.21.15.45:0/3591574428 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4030610 con 0x7f0bb40d0040 2023-11-28T20:54:07.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb159a640 1 -- 172.21.15.45:0/3591574428 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba4030770 con 0x7f0bb40d0040 2023-11-28T20:54:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb159a640 1 -- 172.21.15.45:0/3591574428 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4030b00 con 0x7f0bb40d0040 2023-11-28T20:54:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb359e640 1 -- 172.21.15.45:0/3591574428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bb40d0040 msgr2=0x7f0bb40d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb359e640 1 --2- 172.21.15.45:0/3591574428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bb40d0040 0x7f0bb40d0420 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40097a0 tx=0x7f0ba402b960 comp rx=0 tx=0).stop 2023-11-28T20:54:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb359e640 1 -- 172.21.15.45:0/3591574428 shutdown_connections 2023-11-28T20:54:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb359e640 1 --2- 172.21.15.45:0/3591574428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 0x7f0bb40d3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb359e640 1 --2- 172.21.15.45:0/3591574428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bb40d0040 0x7f0bb40d0420 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb359e640 1 -- 172.21.15.45:0/3591574428 >> 172.21.15.45:0/3591574428 conn(0x7f0bb4070f40 msgr2=0x7f0bb4071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb359e640 1 -- 172.21.15.45:0/3591574428 shutdown_connections 2023-11-28T20:54:07.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.687+0000 7f0bb359e640 1 -- 172.21.15.45:0/3591574428 wait complete. 2023-11-28T20:54:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.689+0000 7f0bb359e640 1 Processor -- start 2023-11-28T20:54:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.689+0000 7f0bb359e640 1 -- start start 2023-11-28T20:54:07.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.690+0000 7f0bb359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 0x7f0bb41f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.690+0000 7f0bb359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bb40b2290 0x7f0bb40cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.690+0000 7f0bb359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb40d0040 0x7f0bb40cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:07.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.690+0000 7f0bb359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4148550 con 0x7f0bb40d0040 2023-11-28T20:54:07.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.690+0000 7f0bb359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40cbcd0 con 0x7f0bb40b2290 2023-11-28T20:54:07.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.690+0000 7f0bb359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40cbfb0 con 0x7f0bb40b1d40 2023-11-28T20:54:07.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb40d0040 0x7f0bb40cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb40d0040 0x7f0bb40cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45950/0 (socket says 172.21.15.45:45950) 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb2d9d640 1 -- 172.21.15.45:0/2888733860 learned_addr learned my addr 172.21.15.45:0/2888733860 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb259c640 1 --2- 172.21.15.45:0/2888733860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 0x7f0bb41f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb1d9b640 1 --2- 172.21.15.45:0/2888733860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bb40b2290 0x7f0bb40cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb2d9d640 1 -- 172.21.15.45:0/2888733860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 msgr2=0x7f0bb41f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb2d9d640 1 --2- 172.21.15.45:0/2888733860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 0x7f0bb41f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb2d9d640 1 -- 172.21.15.45:0/2888733860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bb40b2290 msgr2=0x7f0bb40cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb2d9d640 1 --2- 172.21.15.45:0/2888733860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bb40b2290 0x7f0bb40cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb2d9d640 1 -- 172.21.15.45:0/2888733860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba4009400 con 0x7f0bb40d0040 2023-11-28T20:54:07.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.691+0000 7f0bb2d9d640 1 --2- 172.21.15.45:0/2888733860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb40d0040 0x7f0bb40cb670 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00efc0 tx=0x7f0bac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:07.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.692+0000 7f0ba2ffd640 1 -- 172.21.15.45:0/2888733860 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac009b60 con 0x7f0bb40d0040 2023-11-28T20:54:07.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.692+0000 7f0ba2ffd640 1 -- 172.21.15.45:0/2888733860 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac00f040 con 0x7f0bb40d0040 2023-11-28T20:54:07.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.692+0000 7f0bb359e640 1 -- 172.21.15.45:0/2888733860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb4148760 con 0x7f0bb40d0040 2023-11-28T20:54:07.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.692+0000 7f0ba2ffd640 1 -- 172.21.15.45:0/2888733860 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac003990 con 0x7f0bb40d0040 2023-11-28T20:54:07.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.692+0000 7f0bb359e640 1 -- 172.21.15.45:0/2888733860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb415be60 con 0x7f0bb40d0040 2023-11-28T20:54:07.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.693+0000 7f0bb359e640 1 -- 172.21.15.45:0/2888733860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb4148c80 con 0x7f0bb40d0040 2023-11-28T20:54:07.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.694+0000 7f0ba2ffd640 1 -- 172.21.15.45:0/2888733860 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f0bac003af0 con 0x7f0bb40d0040 2023-11-28T20:54:07.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.695+0000 7f0ba2ffd640 1 -- 172.21.15.45:0/2888733860 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f0bac0899e0 con 0x7f0bb40d0040 2023-11-28T20:54:07.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.699+0000 7f0ba2ffd640 1 -- 172.21.15.45:0/2888733860 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bb4148c80 con 0x7f0bb40d0040 2023-11-28T20:54:07.751 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:07.748+0000 7fcb94c1c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:54:07.811 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:07.808+0000 7fcb94c1c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:54:07.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.899+0000 7f0bb359e640 1 -- 172.21.15.45:0/2888733860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0bb4148c80 con 0x7f0bb40d0040 2023-11-28T20:54:07.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.900+0000 7f0ba2ffd640 1 -- 172.21.15.45:0/2888733860 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0bb4148c80 con 0x7f0bb40d0040 2023-11-28T20:54:07.902 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:54:07.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.902+0000 7f0bb359e640 1 -- 172.21.15.45:0/2888733860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb40d0040 msgr2=0x7f0bb40cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:07.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.902+0000 7f0bb359e640 1 --2- 172.21.15.45:0/2888733860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb40d0040 0x7f0bb40cb670 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00efc0 tx=0x7f0bac00c490 comp rx=0 tx=0).stop 2023-11-28T20:54:07.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.902+0000 7f0bb359e640 1 -- 172.21.15.45:0/2888733860 shutdown_connections 2023-11-28T20:54:07.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.902+0000 7f0bb359e640 1 --2- 172.21.15.45:0/2888733860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb40d0040 0x7f0bb40cb670 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.902+0000 7f0bb359e640 1 --2- 172.21.15.45:0/2888733860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bb40b2290 0x7f0bb40cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.902+0000 7f0bb359e640 1 --2- 172.21.15.45:0/2888733860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0bb40b1d40 0x7f0bb41f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:07.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.902+0000 7f0bb359e640 1 -- 172.21.15.45:0/2888733860 >> 172.21.15.45:0/2888733860 conn(0x7f0bb4070f40 msgr2=0x7f0bb40b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:07.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.902+0000 7f0bb359e640 1 -- 172.21.15.45:0/2888733860 shutdown_connections 2023-11-28T20:54:07.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:07.903+0000 7f0bb359e640 1 -- 172.21.15.45:0/2888733860 wait complete. 2023-11-28T20:54:07.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:54:07.940 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:07.936+0000 7fcb94c1c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:54:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.141+0000 7f6b39535640 1 Processor -- start 2023-11-28T20:54:08.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.141+0000 7f6b39535640 1 -- start start 2023-11-28T20:54:08.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.143+0000 7f6b39535640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6b3415b270 0x7f6b3406cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.143+0000 7f6b39535640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6b340b3420 0x7f6b340ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.143+0000 7f6b39535640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3406d410 con 0x7f6b340b3040 2023-11-28T20:54:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.143+0000 7f6b39535640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b340ceaa0 con 0x7f6b3415b270 2023-11-28T20:54:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.143+0000 7f6b39535640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b3406d6f0 con 0x7f6b340b3420 2023-11-28T20:54:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.143+0000 7f6b2b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6b3415b270 0x7f6b3406cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.143+0000 7f6b2b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6b3415b270 0x7f6b3406cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45952/0 (socket says 172.21.15.45:45952) 2023-11-28T20:54:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.143+0000 7f6b2b7fe640 1 -- 172.21.15.45:0/2291090563 learned_addr learned my addr 172.21.15.45:0/2291090563 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.144+0000 7f6b38d34640 1 --2- 172.21.15.45:0/2291090563 >> v2:172.21.15.190:3300/0 conn(0x7f6b340b3420 0x7f6b340ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.144+0000 7f6b2b7fe640 1 -- 172.21.15.45:0/2291090563 >> v2:172.21.15.190:3300/0 conn(0x7f6b340b3420 msgr2=0x7f6b340ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.144+0000 7f6b2b7fe640 1 --2- 172.21.15.45:0/2291090563 >> v2:172.21.15.190:3300/0 conn(0x7f6b340b3420 0x7f6b340ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.144+0000 7f6b2b7fe640 1 -- 172.21.15.45:0/2291090563 >> v1:172.21.15.190:6789/0 conn(0x7f6b340b3040 legacy=0x7f6b3415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.144+0000 7f6b2b7fe640 1 -- 172.21.15.45:0/2291090563 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3406d9d0 con 0x7f6b3415b270 2023-11-28T20:54:08.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.145+0000 7f6b2b7fe640 1 --2- 172.21.15.45:0/2291090563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b3415b270 0x7f6b3406cda0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f6b340adfd0 tx=0x7f6b1c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bb5f07c1ce5a68 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:08.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.145+0000 7f6b337fe640 1 -- 172.21.15.45:0/2291090563 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c031750 con 0x7f6b3415b270 2023-11-28T20:54:08.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.145+0000 7f6b337fe640 1 -- 172.21.15.45:0/2291090563 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c0318b0 con 0x7f6b3415b270 2023-11-28T20:54:08.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.145+0000 7f6b337fe640 1 -- 172.21.15.45:0/2291090563 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c031c40 con 0x7f6b3415b270 2023-11-28T20:54:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.146+0000 7f6b39535640 1 -- 172.21.15.45:0/2291090563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b3415b270 msgr2=0x7f6b3406cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.146+0000 7f6b39535640 1 --2- 172.21.15.45:0/2291090563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b3415b270 0x7f6b3406cda0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f6b340adfd0 tx=0x7f6b1c02b7f0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.146+0000 7f6b39535640 1 -- 172.21.15.45:0/2291090563 shutdown_connections 2023-11-28T20:54:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.146+0000 7f6b39535640 1 --2- 172.21.15.45:0/2291090563 >> v2:172.21.15.190:3300/0 conn(0x7f6b340b3420 0x7f6b340ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.146+0000 7f6b39535640 1 --2- 172.21.15.45:0/2291090563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b3415b270 0x7f6b3406cda0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.146+0000 7f6b39535640 1 -- 172.21.15.45:0/2291090563 >> 172.21.15.45:0/2291090563 conn(0x7f6b340b1050 msgr2=0x7f6b340cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.146+0000 7f6b39535640 1 -- 172.21.15.45:0/2291090563 shutdown_connections 2023-11-28T20:54:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.147+0000 7f6b39535640 1 -- 172.21.15.45:0/2291090563 wait complete. 2023-11-28T20:54:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.148+0000 7f6b39535640 1 Processor -- start 2023-11-28T20:54:08.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.149+0000 7f6b39535640 1 -- start start 2023-11-28T20:54:08.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.149+0000 7f6b39535640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b340b3040 0x7f6b3414cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b39535640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b340b3420 0x7f6b34155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b39535640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b3415b270 0x7f6b34159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b39535640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3409a760 con 0x7f6b340b3040 2023-11-28T20:54:08.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b39535640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b341562a0 con 0x7f6b3415b270 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b39535640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34099ae0 con 0x7f6b340b3420 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b340b3040 0x7f6b3414cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b340b3040 0x7f6b3414cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45958/0 (socket says 172.21.15.45:45958) 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b33fff640 1 -- 172.21.15.45:0/977406915 learned_addr learned my addr 172.21.15.45:0/977406915 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b2b7fe640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b340b3420 0x7f6b34155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b38d34640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b3415b270 0x7f6b34159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b33fff640 1 -- 172.21.15.45:0/977406915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b340b3420 msgr2=0x7f6b34155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b33fff640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b340b3420 0x7f6b34155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b33fff640 1 -- 172.21.15.45:0/977406915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b3415b270 msgr2=0x7f6b34159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b33fff640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b3415b270 0x7f6b34159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b33fff640 1 -- 172.21.15.45:0/977406915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c0094d0 con 0x7f6b340b3040 2023-11-28T20:54:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.150+0000 7f6b2b7fe640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b340b3420 0x7f6b34155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.151+0000 7f6b33fff640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b340b3040 0x7f6b3414cc40 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400c7c0 tx=0x7f6b2400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.151+0000 7f6b317fa640 1 -- 172.21.15.45:0/977406915 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2400aa50 con 0x7f6b340b3040 2023-11-28T20:54:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.151+0000 7f6b317fa640 1 -- 172.21.15.45:0/977406915 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2400abb0 con 0x7f6b340b3040 2023-11-28T20:54:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.151+0000 7f6b39535640 1 -- 172.21.15.45:0/977406915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b34099dc0 con 0x7f6b340b3040 2023-11-28T20:54:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.151+0000 7f6b317fa640 1 -- 172.21.15.45:0/977406915 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24012420 con 0x7f6b340b3040 2023-11-28T20:54:08.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.151+0000 7f6b39535640 1 -- 172.21.15.45:0/977406915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3409a380 con 0x7f6b340b3040 2023-11-28T20:54:08.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.152+0000 7f6b39535640 1 -- 172.21.15.45:0/977406915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3409d0f0 con 0x7f6b340b3040 2023-11-28T20:54:08.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.153+0000 7f6b317fa640 1 -- 172.21.15.45:0/977406915 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 65) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f6b24012580 con 0x7f6b340b3040 2023-11-28T20:54:08.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.154+0000 7f6b317fa640 1 -- 172.21.15.45:0/977406915 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f6b24089800 con 0x7f6b340b3040 2023-11-28T20:54:08.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.158+0000 7f6b317fa640 1 -- 172.21.15.45:0/977406915 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b240547c0 con 0x7f6b340b3040 2023-11-28T20:54:08.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.353+0000 7f6b39535640 1 -- 172.21.15.45:0/977406915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b341585f0 con 0x7f6b340b3040 2023-11-28T20:54:08.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.355+0000 7f6b317fa640 1 -- 172.21.15.45:0/977406915 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v135) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b24057de0 con 0x7f6b340b3040 2023-11-28T20:54:08.357 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:54:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.358+0000 7f6b39535640 1 -- 172.21.15.45:0/977406915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b340b3040 msgr2=0x7f6b3414cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.358+0000 7f6b39535640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b340b3040 0x7f6b3414cc40 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400c7c0 tx=0x7f6b2400cc80 comp rx=0 tx=0).stop 2023-11-28T20:54:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.358+0000 7f6b39535640 1 -- 172.21.15.45:0/977406915 shutdown_connections 2023-11-28T20:54:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.358+0000 7f6b39535640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b3415b270 0x7f6b34159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.358+0000 7f6b39535640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b340b3420 0x7f6b34155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.358+0000 7f6b39535640 1 --2- 172.21.15.45:0/977406915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b340b3040 0x7f6b3414cc40 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.358+0000 7f6b39535640 1 -- 172.21.15.45:0/977406915 >> 172.21.15.45:0/977406915 conn(0x7f6b340b1050 msgr2=0x7f6b340701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.358+0000 7f6b39535640 1 -- 172.21.15.45:0/977406915 shutdown_connections 2023-11-28T20:54:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.358+0000 7f6b39535640 1 -- 172.21.15.45:0/977406915 wait complete. 2023-11-28T20:54:08.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:54:08.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.606+0000 7f2a6bf89640 1 Processor -- start 2023-11-28T20:54:08.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.606+0000 7f2a6bf89640 1 -- start start 2023-11-28T20:54:08.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.608+0000 7f2a6bf89640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2a641607e0 0x7f2a6406cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.608+0000 7f2a6bf89640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a64070a60 0x7f2a64070ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.608+0000 7f2a6bf89640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a640b30b0 con 0x7f2a64071620 2023-11-28T20:54:08.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.608+0000 7f2a6bf89640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a64071410 con 0x7f2a64070a60 2023-11-28T20:54:08.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.608+0000 7f2a6bf89640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a640ad9d0 con 0x7f2a641607e0 2023-11-28T20:54:08.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.609+0000 7f2a69d25640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2a641607e0 0x7f2a6406cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.609+0000 7f2a69524640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2a64071620 0x7f2a640ad2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49950/0 (socket says 172.21.15.45:49950) 2023-11-28T20:54:08.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.609+0000 7f2a69524640 1 -- 172.21.15.45:0/2518520244 learned_addr learned my addr 172.21.15.45:0/2518520244 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:08.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.610+0000 7f2a6a526640 1 --2- 172.21.15.45:0/2518520244 >> v2:172.21.15.45:3300/0 conn(0x7f2a64070a60 0x7f2a64070ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.610+0000 7f2a69d25640 1 -- 172.21.15.45:0/2518520244 >> v2:172.21.15.45:3300/0 conn(0x7f2a64070a60 msgr2=0x7f2a64070ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:08.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.610+0000 7f2a69d25640 1 --2- 172.21.15.45:0/2518520244 >> v2:172.21.15.45:3300/0 conn(0x7f2a64070a60 0x7f2a64070ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.610+0000 7f2a69d25640 1 -- 172.21.15.45:0/2518520244 >> v1:172.21.15.45:6789/0 conn(0x7f2a64071620 legacy=0x7f2a640ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.610+0000 7f2a69d25640 1 -- 172.21.15.45:0/2518520244 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a640b3300 con 0x7f2a641607e0 2023-11-28T20:54:08.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.612+0000 7f2a69d25640 1 --2- 172.21.15.45:0/2518520244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a641607e0 0x7f2a6406cad0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f2a640d06f0 tx=0x7f2a6002bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4da4299172ee5a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.624+0000 7f2a68d23640 1 -- 172.21.15.45:0/2518520244 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6002cc70 con 0x7f2a641607e0 2023-11-28T20:54:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.625+0000 7f2a68d23640 1 -- 172.21.15.45:0/2518520244 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6002cdd0 con 0x7f2a641607e0 2023-11-28T20:54:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.625+0000 7f2a68d23640 1 -- 172.21.15.45:0/2518520244 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a60034660 con 0x7f2a641607e0 2023-11-28T20:54:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.625+0000 7f2a6bf89640 1 -- 172.21.15.45:0/2518520244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a641607e0 msgr2=0x7f2a6406cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.625+0000 7f2a6bf89640 1 --2- 172.21.15.45:0/2518520244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a641607e0 0x7f2a6406cad0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f2a640d06f0 tx=0x7f2a6002bec0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.625+0000 7f2a6bf89640 1 -- 172.21.15.45:0/2518520244 shutdown_connections 2023-11-28T20:54:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.625+0000 7f2a6bf89640 1 --2- 172.21.15.45:0/2518520244 >> v2:172.21.15.45:3300/0 conn(0x7f2a64070a60 0x7f2a64070ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.626+0000 7f2a6bf89640 1 --2- 172.21.15.45:0/2518520244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a641607e0 0x7f2a6406cad0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.626+0000 7f2a6bf89640 1 -- 172.21.15.45:0/2518520244 >> 172.21.15.45:0/2518520244 conn(0x7f2a640b13a0 msgr2=0x7f2a640b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.626+0000 7f2a6bf89640 1 -- 172.21.15.45:0/2518520244 shutdown_connections 2023-11-28T20:54:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.626+0000 7f2a6bf89640 1 -- 172.21.15.45:0/2518520244 wait complete. 2023-11-28T20:54:08.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.627+0000 7f2a6bf89640 1 Processor -- start 2023-11-28T20:54:08.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.628+0000 7f2a6bf89640 1 -- start start 2023-11-28T20:54:08.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a6bf89640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a64070a60 0x7f2a641f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a6bf89640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a64071620 0x7f2a6415b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a6bf89640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a641607e0 0x7f2a6415bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a6bf89640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6415f010 con 0x7f2a64070a60 2023-11-28T20:54:08.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a6bf89640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a641f5c00 con 0x7f2a641607e0 2023-11-28T20:54:08.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a6bf89640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6415e120 con 0x7f2a64071620 2023-11-28T20:54:08.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a69d25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a64070a60 0x7f2a641f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a6a526640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a641607e0 0x7f2a6415bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a69d25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a64070a60 0x7f2a641f56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45980/0 (socket says 172.21.15.45:45980) 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a69d25640 1 -- 172.21.15.45:0/4104105515 learned_addr learned my addr 172.21.15.45:0/4104105515 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.629+0000 7f2a69524640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a64071620 0x7f2a6415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a69d25640 1 -- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a64071620 msgr2=0x7f2a6415b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a69d25640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a64071620 0x7f2a6415b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a69d25640 1 -- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a641607e0 msgr2=0x7f2a6415bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a69d25640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a641607e0 0x7f2a6415bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a69d25640 1 -- 172.21.15.45:0/4104105515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a60009a70 con 0x7f2a64070a60 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a69d25640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a64070a60 0x7f2a641f56c0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2a6002c4d0 tx=0x7f2a60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a5a7fc640 1 -- 172.21.15.45:0/4104105515 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6003e610 con 0x7f2a64070a60 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a5a7fc640 1 -- 172.21.15.45:0/4104105515 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6003e790 con 0x7f2a64070a60 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a5a7fc640 1 -- 172.21.15.45:0/4104105515 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6003eb40 con 0x7f2a64070a60 2023-11-28T20:54:08.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.630+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6415e400 con 0x7f2a64070a60 2023-11-28T20:54:08.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.631+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6415e9c0 con 0x7f2a64070a60 2023-11-28T20:54:08.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.632+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a640733d0 con 0x7f2a64070a60 2023-11-28T20:54:08.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.633+0000 7f2a5a7fc640 1 -- 172.21.15.45:0/4104105515 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 66) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f2a6003b050 con 0x7f2a64070a60 2023-11-28T20:54:08.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.634+0000 7f2a5a7fc640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f2a4806ac90 0x7f2a4806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:08.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.634+0000 7f2a5a7fc640 1 -- 172.21.15.45:0/4104105515 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f2a600ab7f0 con 0x7f2a64070a60 2023-11-28T20:54:08.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.634+0000 7f2a69524640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f2a4806ac90 0x7f2a4806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:08.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.635+0000 7f2a69524640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f2a4806ac90 0x7f2a4806d150 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2a640713e0 tx=0x7f2a54009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:08.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.639+0000 7f2a5a7fc640 1 -- 172.21.15.45:0/4104105515 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a60076740 con 0x7f2a64070a60 2023-11-28T20:54:08.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.824+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2a6406bb40 con 0x7f2a64070a60 2023-11-28T20:54:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.825+0000 7f2a5a7fc640 1 -- 172.21.15.45:0/4104105515 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v136) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2a60079d60 con 0x7f2a64070a60 2023-11-28T20:54:08.827 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:54:08.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.827+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f2a4806ac90 msgr2=0x7f2a4806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.827+0000 7f2a6bf89640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f2a4806ac90 0x7f2a4806d150 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2a640713e0 tx=0x7f2a54009040 comp rx=0 tx=0).stop 2023-11-28T20:54:08.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.827+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a64070a60 msgr2=0x7f2a641f56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:08.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.827+0000 7f2a6bf89640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a64070a60 0x7f2a641f56c0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2a6002c4d0 tx=0x7f2a60004a40 comp rx=0 tx=0).stop 2023-11-28T20:54:08.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.828+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 shutdown_connections 2023-11-28T20:54:08.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.828+0000 7f2a6bf89640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f2a4806ac90 0x7f2a4806d150 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.828+0000 7f2a6bf89640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a641607e0 0x7f2a6415bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.828+0000 7f2a6bf89640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a64071620 0x7f2a6415b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.828+0000 7f2a6bf89640 1 --2- 172.21.15.45:0/4104105515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a64070a60 0x7f2a641f56c0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:08.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.828+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 >> 172.21.15.45:0/4104105515 conn(0x7f2a640b13a0 msgr2=0x7f2a640af310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:08.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.828+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 shutdown_connections 2023-11-28T20:54:08.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:08.828+0000 7f2a6bf89640 1 -- 172.21.15.45:0/4104105515 wait complete. 2023-11-28T20:54:08.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:09.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.064+0000 7fc1b2f6e640 1 Processor -- start 2023-11-28T20:54:09.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.064+0000 7fc1b2f6e640 1 -- start start 2023-11-28T20:54:09.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.065+0000 7fc1b2f6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1ac0d02d0 0x7fc1ac0d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.065+0000 7fc1b2f6e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc1ac0d1ab0 0x7fc1ac0b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.065+0000 7fc1b2f6e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac0d2230 con 0x7fc1ac15d550 2023-11-28T20:54:09.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.065+0000 7fc1b2f6e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ac0d2510 con 0x7fc1ac0d1ab0 2023-11-28T20:54:09.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.065+0000 7fc1b2f6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0d27f0 con 0x7fc1ac0d02d0 2023-11-28T20:54:09.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.066+0000 7fc1b1f6c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc1ac15d550 0x7fc1ac0cfbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49952/0 (socket says 172.21.15.45:49952) 2023-11-28T20:54:09.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.066+0000 7fc1b1f6c640 1 -- 172.21.15.45:0/1543308753 learned_addr learned my addr 172.21.15.45:0/1543308753 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:09.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.066+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941624180 0 0) 0x7fc1ac0d2230 con 0x7fc1ac15d550 2023-11-28T20:54:09.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.066+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc198003520 con 0x7fc1ac15d550 2023-11-28T20:54:09.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.066+0000 7fc1b176b640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1ac0d02d0 0x7fc1ac0d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.066+0000 7fc1b276d640 1 --2- 172.21.15.45:0/1543308753 >> v2:172.21.15.190:3300/0 conn(0x7fc1ac0d1ab0 0x7fc1ac0b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 967188050 0 0) 0x7fc198003520 con 0x7fc1ac15d550 2023-11-28T20:54:09.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1ac0d02d0 msgr2=0x7fc1ac0d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:09.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1ac0d02d0 0x7fc1ac0d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 >> v2:172.21.15.190:3300/0 conn(0x7fc1ac0d1ab0 msgr2=0x7fc1ac0b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:09.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 --2- 172.21.15.45:0/1543308753 >> v2:172.21.15.190:3300/0 conn(0x7fc1ac0d1ab0 0x7fc1ac0b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac0b6a30 con 0x7fc1ac15d550 2023-11-28T20:54:09.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc1a8001670 con 0x7fc1ac15d550 2023-11-28T20:54:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 >> v1:172.21.15.45:6789/0 conn(0x7fc1ac15d550 legacy=0x7fc1ac0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198003970 0x7fc198003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1980047c0 0x7fc198004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc198005180 0x7fc19800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0d2510 con 0x7fc198003970 2023-11-28T20:54:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0d27f0 con 0x7fc1980047c0 2023-11-28T20:54:09.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19800bff0 con 0x7fc198005180 2023-11-28T20:54:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b176b640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1ac0d02d0 0x7fc1ac0d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.067+0000 7fc1b1f6c640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198003970 0x7fc198003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.068+0000 7fc1b1f6c640 1 -- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc198005180 msgr2=0x7fc19800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.068+0000 7fc1b1f6c640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc198005180 0x7fc19800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.068+0000 7fc1b1f6c640 1 -- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1980047c0 msgr2=0x7fc198004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.068+0000 7fc1b1f6c640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1980047c0 0x7fc198004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.068+0000 7fc1b1f6c640 1 -- 172.21.15.45:0/1543308753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198003520 con 0x7fc198003970 2023-11-28T20:54:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.069+0000 7fc1b1f6c640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198003970 0x7fc198003d90 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c110 tx=0x7fc1980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68805537e0ea0c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:09.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.069+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8005ad0 con 0x7fc198003970 2023-11-28T20:54:09.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.069+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a8005c50 con 0x7fc198003970 2023-11-28T20:54:09.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.069+0000 7fc1b0f6a640 1 -- 172.21.15.45:0/1543308753 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a802c830 con 0x7fc198003970 2023-11-28T20:54:09.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.070+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/1543308753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198003970 msgr2=0x7fc198003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.070+0000 7fc1b2f6e640 1 --2- 172.21.15.45:0/1543308753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198003970 0x7fc198003d90 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c110 tx=0x7fc1980057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.070+0000 7fc1b276d640 1 -- 172.21.15.45:0/1543308753 reap_dead start 2023-11-28T20:54:09.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.070+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/1543308753 shutdown_connections 2023-11-28T20:54:09.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.070+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/1543308753 >> 172.21.15.45:0/1543308753 conn(0x7fc1ac06ac70 msgr2=0x7fc1ac0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:09.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.070+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/1543308753 shutdown_connections 2023-11-28T20:54:09.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.070+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/1543308753 wait complete. 2023-11-28T20:54:09.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.072+0000 7fc1b2f6e640 1 Processor -- start 2023-11-28T20:54:09.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.072+0000 7fc1b2f6e640 1 -- start start 2023-11-28T20:54:09.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.073+0000 7fc1b2f6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc198003970 0x7fc1ac0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.073+0000 7fc1b2f6e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1ac0ca2a0 0x7fc1ac0ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.073+0000 7fc1b2f6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac14d980 0x7fc1ac14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.073+0000 7fc1b2f6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0cb4e0 con 0x7fc1ac14d980 2023-11-28T20:54:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.073+0000 7fc1b2f6e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0cac60 con 0x7fc1ac0ca2a0 2023-11-28T20:54:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.073+0000 7fc1b2f6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0caf20 con 0x7fc198003970 2023-11-28T20:54:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.073+0000 7fc1b276d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac14d980 0x7fc1ac14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.073+0000 7fc1b276d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac14d980 0x7fc1ac14de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46000/0 (socket says 172.21.15.45:46000) 2023-11-28T20:54:09.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.073+0000 7fc1b276d640 1 -- 172.21.15.45:0/2770606065 learned_addr learned my addr 172.21.15.45:0/2770606065 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:09.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b276d640 1 -- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc198003970 msgr2=0x7fc1ac0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b1f6c640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc198003970 0x7fc1ac0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b176b640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1ac0ca2a0 0x7fc1ac0ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b276d640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc198003970 0x7fc1ac0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b276d640 1 -- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1ac0ca2a0 msgr2=0x7fc1ac0ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b276d640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1ac0ca2a0 0x7fc1ac0ca720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b276d640 1 -- 172.21.15.45:0/2770606065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac0cb200 con 0x7fc1ac14d980 2023-11-28T20:54:09.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b1f6c640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc198003970 0x7fc1ac0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:09.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b276d640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac14d980 0x7fc1ac14de10 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0d4290 tx=0x7fc1a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:09.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1a27fc640 1 -- 172.21.15.45:0/2770606065 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a400ed90 con 0x7fc1ac14d980 2023-11-28T20:54:09.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1a27fc640 1 -- 172.21.15.45:0/2770606065 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a4014420 con 0x7fc1ac14d980 2023-11-28T20:54:09.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1a27fc640 1 -- 172.21.15.45:0/2770606065 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a40147d0 con 0x7fc1ac14d980 2023-11-28T20:54:09.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac0c6e10 con 0x7fc1ac14d980 2023-11-28T20:54:09.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.074+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac14e350 con 0x7fc1ac14d980 2023-11-28T20:54:09.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.076+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac0c7330 con 0x7fc1ac14d980 2023-11-28T20:54:09.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.077+0000 7fc1a27fc640 1 -- 172.21.15.45:0/2770606065 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 66) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fc1a4014930 con 0x7fc1ac14d980 2023-11-28T20:54:09.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.077+0000 7fc1a27fc640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc19006acc0 0x7fc19006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.078+0000 7fc1a27fc640 1 -- 172.21.15.45:0/2770606065 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fc1a4089a00 con 0x7fc1ac14d980 2023-11-28T20:54:09.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.078+0000 7fc1b1f6c640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc19006acc0 0x7fc19006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.079+0000 7fc1b1f6c640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc19006acc0 0x7fc19006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c110 tx=0x7fc1a8007000 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:09.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.082+0000 7fc1a27fc640 1 -- 172.21.15.45:0/2770606065 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1ac0c7330 con 0x7fc1ac14d980 2023-11-28T20:54:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.323+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1ac0c7330 con 0x7fc1ac14d980 2023-11-28T20:54:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.325+0000 7fc1a27fc640 1 -- 172.21.15.45:0/2770606065 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fc1ac0c7330 con 0x7fc1ac14d980 2023-11-28T20:54:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:09.327 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/544145878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":544145878},{"type":"v1","addr":"172.21.15.190:6836","nonce":544145878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2464692926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2464692926},{"type":"v1","addr":"172.21.15.45:6835","nonce":2464692926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2048455360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2048455360},{"type":"v1","addr":"172.21.15.190:6837","nonce":2048455360}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:53:35.282168+0000","modified":"2023-11-28T20:53:38.346980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2464465186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2464465186},{"type":"v1","addr":"172.21.15.45:6834","nonce":2464465186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-28T20:54:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 59 2023-11-28T20:54:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.327+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc19006acc0 msgr2=0x7fc19006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.327+0000 7fc1b2f6e640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc19006acc0 0x7fc19006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c110 tx=0x7fc1a8007000 comp rx=0 tx=0).stop 2023-11-28T20:54:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.327+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac14d980 msgr2=0x7fc1ac14de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.327+0000 7fc1b2f6e640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac14d980 0x7fc1ac14de10 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0d4290 tx=0x7fc1a400e4b0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.328+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 shutdown_connections 2023-11-28T20:54:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.328+0000 7fc1b2f6e640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc19006acc0 0x7fc19006d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.328+0000 7fc1b2f6e640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac14d980 0x7fc1ac14de10 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.328+0000 7fc1b2f6e640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1ac0ca2a0 0x7fc1ac0ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.328+0000 7fc1b2f6e640 1 --2- 172.21.15.45:0/2770606065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc198003970 0x7fc1ac0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.328+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 >> 172.21.15.45:0/2770606065 conn(0x7fc1ac06ac70 msgr2=0x7fc1ac0b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.328+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 shutdown_connections 2023-11-28T20:54:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.328+0000 7fc1b2f6e640 1 -- 172.21.15.45:0/2770606065 wait complete. 2023-11-28T20:54:09.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:09.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.575+0000 7f8b350d1640 1 Processor -- start 2023-11-28T20:54:09.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.576+0000 7f8b350d1640 1 -- start start 2023-11-28T20:54:09.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.577+0000 7f8b350d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300b7f60 0x7f8b301681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.577+0000 7f8b350d1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b300d2230 con 0x7f8b300d02d0 2023-11-28T20:54:09.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.577+0000 7f8b350d1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b300d2510 con 0x7f8b3015d550 2023-11-28T20:54:09.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.577+0000 7f8b350d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b300d27f0 con 0x7f8b300b7f60 2023-11-28T20:54:09.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.577+0000 7f8b27fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8b300d02d0 0x7f8b300d1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49964/0 (socket says 172.21.15.45:49964) 2023-11-28T20:54:09.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.577+0000 7f8b27fff640 1 -- 172.21.15.45:0/2057735502 learned_addr learned my addr 172.21.15.45:0/2057735502 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:09.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.578+0000 7f8b348d0640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300b7f60 0x7f8b301681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.578+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259610814 0 0) 0x7f8b300d2230 con 0x7f8b300d02d0 2023-11-28T20:54:09.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.578+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1c003520 con 0x7f8b300d02d0 2023-11-28T20:54:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.578+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342491578 0 0) 0x7f8b1c003520 con 0x7f8b300d02d0 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.578+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300b7f60 msgr2=0x7f8b301681d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.578+0000 7f8b2f7fe640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300b7f60 0x7f8b301681d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 >> v1:172.21.15.190:6789/0 conn(0x7f8b3015d550 legacy=0x7f8b300cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b300b6a30 con 0x7f8b300d02d0 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f8b18001670 con 0x7f8b300d02d0 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 >> v1:172.21.15.45:6789/0 conn(0x7f8b300d02d0 legacy=0x7f8b300d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300b7f60 0x7f8b1c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b1c003f20 0x7f8b1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b300d27f0 con 0x7f8b1c004d00 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c00ba60 con 0x7f8b1c003f20 2023-11-28T20:54:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c00bc20 con 0x7f8b300b7f60 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b348d0640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b27fff640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b1c003f20 0x7f8b1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2ffff640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300b7f60 0x7f8b1c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b348d0640 1 -- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300b7f60 msgr2=0x7f8b1c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b348d0640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300b7f60 0x7f8b1c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b348d0640 1 -- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b1c003f20 msgr2=0x7f8b1c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b348d0640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b1c003f20 0x7f8b1c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b348d0640 1 -- 172.21.15.45:0/2057735502 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1c003520 con 0x7f8b1c004d00 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.579+0000 7f8b2ffff640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300b7f60 0x7f8b1c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.581+0000 7f8b348d0640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b1c005180 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009fc0 tx=0x7f8b1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64722b94f54f8759 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.581+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2802cc20 con 0x7f8b1c004d00 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.581+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2800f040 con 0x7f8b1c004d00 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.581+0000 7f8b2f7fe640 1 -- 172.21.15.45:0/2057735502 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28034440 con 0x7f8b1c004d00 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.582+0000 7f8b350d1640 1 -- 172.21.15.45:0/2057735502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 msgr2=0x7f8b1c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.582+0000 7f8b350d1640 1 --2- 172.21.15.45:0/2057735502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b1c005180 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009fc0 tx=0x7f8b1c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.582+0000 7f8b348d0640 1 -- 172.21.15.45:0/2057735502 reap_dead start 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.582+0000 7f8b350d1640 1 -- 172.21.15.45:0/2057735502 shutdown_connections 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.582+0000 7f8b350d1640 1 -- 172.21.15.45:0/2057735502 >> 172.21.15.45:0/2057735502 conn(0x7f8b3006ac70 msgr2=0x7f8b300b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.582+0000 7f8b350d1640 1 -- 172.21.15.45:0/2057735502 shutdown_connections 2023-11-28T20:54:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.582+0000 7f8b350d1640 1 -- 172.21.15.45:0/2057735502 wait complete. 2023-11-28T20:54:09.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.584+0000 7f8b350d1640 1 Processor -- start 2023-11-28T20:54:09.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.584+0000 7f8b350d1640 1 -- start start 2023-11-28T20:54:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.585+0000 7f8b350d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b300b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.585+0000 7f8b350d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300a9b20 0x7f8b300a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.585+0000 7f8b350d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b301f2dd0 0x7f8b301f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.585+0000 7f8b350d1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b301f3b60 con 0x7f8b1c004d00 2023-11-28T20:54:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.585+0000 7f8b350d1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b300aa4e0 con 0x7f8b301f2dd0 2023-11-28T20:54:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.585+0000 7f8b350d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b301f3e80 con 0x7f8b300a9b20 2023-11-28T20:54:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.585+0000 7f8b2ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b300b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.585+0000 7f8b2ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b300b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46026/0 (socket says 172.21.15.45:46026) 2023-11-28T20:54:09.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.585+0000 7f8b2ffff640 1 -- 172.21.15.45:0/1135760700 learned_addr learned my addr 172.21.15.45:0/1135760700 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b27fff640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300a9b20 0x7f8b300a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b2ffff640 1 -- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300a9b20 msgr2=0x7f8b300a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b2ffff640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300a9b20 0x7f8b300a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b2ffff640 1 -- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b301f2dd0 msgr2=0x7f8b301f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b2ffff640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b301f2dd0 0x7f8b301f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b2ffff640 1 -- 172.21.15.45:0/1135760700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b301f4160 con 0x7f8b1c004d00 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b2ffff640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b300b4430 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000d7a0 tx=0x7f8b2000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b2d7fa640 1 -- 172.21.15.45:0/1135760700 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20017070 con 0x7f8b1c004d00 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b2d7fa640 1 -- 172.21.15.45:0/1135760700 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b20005070 con 0x7f8b1c004d00 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b2d7fa640 1 -- 172.21.15.45:0/1135760700 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20005420 con 0x7f8b1c004d00 2023-11-28T20:54:09.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.586+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b301f4440 con 0x7f8b1c004d00 2023-11-28T20:54:09.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.587+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b300b4970 con 0x7f8b1c004d00 2023-11-28T20:54:09.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.588+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3004e920 con 0x7f8b1c004d00 2023-11-28T20:54:09.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.589+0000 7f8b2d7fa640 1 -- 172.21.15.45:0/1135760700 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 66) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f8b20007030 con 0x7f8b1c004d00 2023-11-28T20:54:09.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.590+0000 7f8b2d7fa640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8b1006ac90 0x7f8b1006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:09.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.590+0000 7f8b2d7fa640 1 -- 172.21.15.45:0/1135760700 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f8b20088fd0 con 0x7f8b1c004d00 2023-11-28T20:54:09.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.591+0000 7f8b27fff640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8b1006ac90 0x7f8b1006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:09.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.592+0000 7f8b27fff640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8b1006ac90 0x7f8b1006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b18001200 tx=0x7f8b18001150 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:09.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.598+0000 7f8b2d7fa640 1 -- 172.21.15.45:0/1135760700 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b20053e70 con 0x7f8b1c004d00 2023-11-28T20:54:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.796+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b3015cb30 con 0x7f8b1c004d00 2023-11-28T20:54:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.797+0000 7f8b2d7fa640 1 -- 172.21.15.45:0/1135760700 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+17954 (secure 0 0 0) 0x7f8b20057490 con 0x7f8b1c004d00 2023-11-28T20:54:09.800 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:09.800 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":136,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:08.585686+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:53:33.200533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-11-28T20:53:34.300717+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":136,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3643991290":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/304862452":"2023-11-29T20:54:07.580192+0000","172.21.15.190:6833/3157274352":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/2315351474":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/259384681":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/3489922358":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/665095987":"2023-11-29T20:54:07.580192+0000","172.21.15.190:6832/3157274352":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/2592164098":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/800294716":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/3135688549":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/369403092":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6833/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6832/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1007604882":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/685209934":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1745589824":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1706225465":"2023-11-29T20:53:49.553172+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8b1006ac90 msgr2=0x7f8b1006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8b1006ac90 0x7f8b1006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b18001200 tx=0x7f8b18001150 comp rx=0 tx=0).stop 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 msgr2=0x7f8b300b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b300b4430 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000d7a0 tx=0x7f8b2000dc60 comp rx=0 tx=0).stop 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 shutdown_connections 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8b1006ac90 0x7f8b1006d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b301f2dd0 0x7f8b301f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b300a9b20 0x7f8b300a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 --2- 172.21.15.45:0/1135760700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b1c004d00 0x7f8b300b4430 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 >> 172.21.15.45:0/1135760700 conn(0x7f8b3006ac70 msgr2=0x7f8b3015dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 shutdown_connections 2023-11-28T20:54:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:09.800+0000 7f8b350d1640 1 -- 172.21.15.45:0/1135760700 wait complete. 2023-11-28T20:54:09.822 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T20:54:09.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:10.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.043+0000 7fe80d659640 1 Processor -- start 2023-11-28T20:54:10.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.043+0000 7fe80d659640 1 -- start start 2023-11-28T20:54:10.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.045+0000 7fe80d659640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe8080b3870 0x7fe80815db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.045+0000 7fe80d659640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe80806d880 0x7fe80815e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.045+0000 7fe80d659640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80815e680 con 0x7fe8080721e0 2023-11-28T20:54:10.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.045+0000 7fe80d659640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80815a460 con 0x7fe8080b3870 2023-11-28T20:54:10.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.045+0000 7fe80d659640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80815e960 con 0x7fe80806d880 2023-11-28T20:54:10.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.045+0000 7fe807fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe8080b3870 0x7fe80815db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.045+0000 7fe807fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe8080b3870 0x7fe80815db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46040/0 (socket says 172.21.15.45:46040) 2023-11-28T20:54:10.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.045+0000 7fe807fff640 1 -- 172.21.15.45:0/2792265021 learned_addr learned my addr 172.21.15.45:0/2792265021 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:10.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.046+0000 7fe807fff640 1 -- 172.21.15.45:0/2792265021 >> v2:172.21.15.190:3300/0 conn(0x7fe80806d880 msgr2=0x7fe80815e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:10.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.046+0000 7fe807fff640 1 --2- 172.21.15.45:0/2792265021 >> v2:172.21.15.190:3300/0 conn(0x7fe80806d880 0x7fe80815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.046+0000 7fe807fff640 1 -- 172.21.15.45:0/2792265021 >> v1:172.21.15.190:6789/0 conn(0x7fe8080721e0 legacy=0x7fe80806d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:10.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.046+0000 7fe807fff640 1 -- 172.21.15.45:0/2792265021 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe808071700 con 0x7fe8080b3870 2023-11-28T20:54:10.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.047+0000 7fe807fff640 1 --2- 172.21.15.45:0/2792265021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8080b3870 0x7fe80815db10 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fe8080cecd0 tx=0x7fe7f802db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dba7f9f47384539 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:10.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.047+0000 7fe8077fe640 1 -- 172.21.15.45:0/2792265021 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f802f900 con 0x7fe8080b3870 2023-11-28T20:54:10.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.047+0000 7fe8077fe640 1 -- 172.21.15.45:0/2792265021 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f802fa60 con 0x7fe8080b3870 2023-11-28T20:54:10.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.047+0000 7fe8077fe640 1 -- 172.21.15.45:0/2792265021 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f802fdf0 con 0x7fe8080b3870 2023-11-28T20:54:10.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.048+0000 7fe80d659640 1 -- 172.21.15.45:0/2792265021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8080b3870 msgr2=0x7fe80815db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.048+0000 7fe80d659640 1 --2- 172.21.15.45:0/2792265021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8080b3870 0x7fe80815db10 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fe8080cecd0 tx=0x7fe7f802db40 comp rx=0 tx=0).stop 2023-11-28T20:54:10.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.048+0000 7fe80d659640 1 -- 172.21.15.45:0/2792265021 shutdown_connections 2023-11-28T20:54:10.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.048+0000 7fe80d659640 1 --2- 172.21.15.45:0/2792265021 >> v2:172.21.15.190:3300/0 conn(0x7fe80806d880 0x7fe80815e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.048+0000 7fe80d659640 1 --2- 172.21.15.45:0/2792265021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8080b3870 0x7fe80815db10 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.048+0000 7fe80d659640 1 -- 172.21.15.45:0/2792265021 >> 172.21.15.45:0/2792265021 conn(0x7fe80806ab30 msgr2=0x7fe8080b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.048+0000 7fe80d659640 1 -- 172.21.15.45:0/2792265021 shutdown_connections 2023-11-28T20:54:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.048+0000 7fe80d659640 1 -- 172.21.15.45:0/2792265021 wait complete. 2023-11-28T20:54:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.049+0000 7fe80d659640 1 Processor -- start 2023-11-28T20:54:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.050+0000 7fe80d659640 1 -- start start 2023-11-28T20:54:10.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.050+0000 7fe80d659640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe80806d880 0x7fe80814bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe80d659640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8080721e0 0x7fe8080cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe80d659640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8080b3870 0x7fe80809bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe80d659640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80809aa90 con 0x7fe80806d880 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe80d659640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80809ac70 con 0x7fe8080b3870 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe80d659640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80809af50 con 0x7fe8080721e0 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe807fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe80806d880 0x7fe80814bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe807fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe80806d880 0x7fe80814bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46042/0 (socket says 172.21.15.45:46042) 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe807fff640 1 -- 172.21.15.45:0/1074521509 learned_addr learned my addr 172.21.15.45:0/1074521509 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe807fff640 1 -- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8080721e0 msgr2=0x7fe8080cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe807fff640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8080721e0 0x7fe8080cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe80ce58640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8080b3870 0x7fe80809bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe807fff640 1 -- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8080b3870 msgr2=0x7fe80809bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe807fff640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8080b3870 0x7fe80809bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe807fff640 1 -- 172.21.15.45:0/1074521509 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f80094d0 con 0x7fe80806d880 2023-11-28T20:54:10.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.051+0000 7fe80ce58640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8080b3870 0x7fe80809bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:10.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.052+0000 7fe807fff640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe80806d880 0x7fe80814bb30 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe7f802db80 tx=0x7fe7f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:10.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.052+0000 7fe8057fa640 1 -- 172.21.15.45:0/1074521509 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8037cb0 con 0x7fe80806d880 2023-11-28T20:54:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.052+0000 7fe8057fa640 1 -- 172.21.15.45:0/1074521509 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f8037e10 con 0x7fe80806d880 2023-11-28T20:54:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.052+0000 7fe8057fa640 1 -- 172.21.15.45:0/1074521509 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80426b0 con 0x7fe80806d880 2023-11-28T20:54:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.052+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80809b230 con 0x7fe80806d880 2023-11-28T20:54:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.052+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80809b7f0 con 0x7fe80806d880 2023-11-28T20:54:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.053+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80809b090 con 0x7fe80806d880 2023-11-28T20:54:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.054+0000 7fe8057fa640 1 -- 172.21.15.45:0/1074521509 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 66) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fe7f8042810 con 0x7fe80806d880 2023-11-28T20:54:10.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.055+0000 7fe8057fa640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe7e806ac90 0x7fe7e806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.055+0000 7fe8057fa640 1 -- 172.21.15.45:0/1074521509 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fe7f8033070 con 0x7fe80806d880 2023-11-28T20:54:10.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.056+0000 7fe7fffff640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe7e806ac90 0x7fe7e806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.057+0000 7fe7fffff640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe7e806ac90 0x7fe7e806d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe80809ca40 tx=0x7fe7f0007990 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.059+0000 7fe8057fa640 1 -- 172.21.15.45:0/1074521509 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe80809b090 con 0x7fe80806d880 2023-11-28T20:54:10.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.286+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe80809b090 con 0x7fe80806d880 2023-11-28T20:54:10.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.288+0000 7fe8057fa640 1 -- 172.21.15.45:0/1074521509 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fe80809b090 con 0x7fe80806d880 2023-11-28T20:54:10.290 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:10.290 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/544145878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":544145878},{"type":"v1","addr":"172.21.15.190:6836","nonce":544145878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2464692926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2464692926},{"type":"v1","addr":"172.21.15.45:6835","nonce":2464692926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2048455360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2048455360},{"type":"v1","addr":"172.21.15.190:6837","nonce":2048455360}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:53:35.282168+0000","modified":"2023-11-28T20:53:38.346980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2464465186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2464465186},{"type":"v1","addr":"172.21.15.45:6834","nonce":2464465186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-28T20:54:10.291 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 59 2023-11-28T20:54:10.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe7e806ac90 msgr2=0x7fe7e806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe7e806ac90 0x7fe7e806d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe80809ca40 tx=0x7fe7f0007990 comp rx=0 tx=0).stop 2023-11-28T20:54:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe80806d880 msgr2=0x7fe80814bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe80806d880 0x7fe80814bb30 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe7f802db80 tx=0x7fe7f80046c0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 shutdown_connections 2023-11-28T20:54:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe7e806ac90 0x7fe7e806d150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8080b3870 0x7fe80809bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8080721e0 0x7fe8080cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 --2- 172.21.15.45:0/1074521509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe80806d880 0x7fe80814bb30 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 >> 172.21.15.45:0/1074521509 conn(0x7fe80806ab30 msgr2=0x7fe80806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.290+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 shutdown_connections 2023-11-28T20:54:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.291+0000 7fe80d659640 1 -- 172.21.15.45:0/1074521509 wait complete. 2023-11-28T20:54:10.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:10.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.517+0000 7f32b6a44640 1 Processor -- start 2023-11-28T20:54:10.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.517+0000 7f32b6a44640 1 -- start start 2023-11-28T20:54:10.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.519+0000 7f32b6a44640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f32b0072200 0x7f32b00b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.519+0000 7f32b6a44640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b00713e0 0x7f32b00b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.519+0000 7f32b6a44640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b00b4310 con 0x7f32b0146610 2023-11-28T20:54:10.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.519+0000 7f32b6a44640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b00725e0 con 0x7f32b0072200 2023-11-28T20:54:10.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.519+0000 7f32b6a44640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b006d8d0 con 0x7f32b00713e0 2023-11-28T20:54:10.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.519+0000 7f32b5241640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f32b0146610 0x7f32b0160e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49980/0 (socket says 172.21.15.45:49980) 2023-11-28T20:54:10.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.519+0000 7f32b5241640 1 -- 172.21.15.45:0/2505256887 learned_addr learned my addr 172.21.15.45:0/2505256887 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b5a42640 1 --2- 172.21.15.45:0/2505256887 >> v2:172.21.15.190:3300/0 conn(0x7f32b0072200 0x7f32b00b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b6243640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b00713e0 0x7f32b00b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129110079 0 0) 0x7f32a0001110 con 0x7f32b0146610 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f329c003520 con 0x7f32b0146610 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1212705249 0 0) 0x7f329c003520 con 0x7f32b0146610 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b00713e0 msgr2=0x7f32b00b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b00713e0 0x7f32b00b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 >> v2:172.21.15.190:3300/0 conn(0x7f32b0072200 msgr2=0x7f32b00b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 --2- 172.21.15.45:0/2505256887 >> v2:172.21.15.190:3300/0 conn(0x7f32b0072200 0x7f32b00b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b006ca20 con 0x7f32b0146610 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f32a0001800 con 0x7f32b0146610 2023-11-28T20:54:10.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 >> v1:172.21.15.45:6789/0 conn(0x7f32b0146610 legacy=0x7f32b0160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b00713e0 0x7f329c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f329c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f329c004d00 0x7f329c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b006d8d0 con 0x7f329c003ef0 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c00b980 con 0x7f329c004d00 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.520+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329c00bac0 con 0x7f32b00713e0 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.521+0000 7f32b5241640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f329c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.521+0000 7f32b5a42640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b00713e0 0x7f329c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.521+0000 7f32b6243640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f329c004d00 0x7f329c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.521+0000 7f32b5241640 1 -- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b00713e0 msgr2=0x7f329c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.521+0000 7f32b5241640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b00713e0 0x7f329c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.521+0000 7f32b5241640 1 -- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f329c004d00 msgr2=0x7f329c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:10.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.521+0000 7f32b5241640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f329c004d00 0x7f329c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.521+0000 7f32b5241640 1 -- 172.21.15.45:0/2505256887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c003520 con 0x7f329c003ef0 2023-11-28T20:54:10.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.522+0000 7f32b5241640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f329c0047c0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f32a000c210 tx=0x7f329c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f45ddaf0ace7ac server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:10.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0005ad0 con 0x7f329c003ef0 2023-11-28T20:54:10.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a0005c50 con 0x7f329c003ef0 2023-11-28T20:54:10.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b4a40640 1 -- 172.21.15.45:0/2505256887 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a002c8d0 con 0x7f329c003ef0 2023-11-28T20:54:10.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b6a44640 1 -- 172.21.15.45:0/2505256887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 msgr2=0x7f329c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b6a44640 1 --2- 172.21.15.45:0/2505256887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f329c0047c0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f32a000c210 tx=0x7f329c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b6243640 1 -- 172.21.15.45:0/2505256887 reap_dead start 2023-11-28T20:54:10.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b6a44640 1 -- 172.21.15.45:0/2505256887 shutdown_connections 2023-11-28T20:54:10.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b6a44640 1 -- 172.21.15.45:0/2505256887 >> 172.21.15.45:0/2505256887 conn(0x7f32b006ab30 msgr2=0x7f32b00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:10.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b6a44640 1 -- 172.21.15.45:0/2505256887 shutdown_connections 2023-11-28T20:54:10.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.523+0000 7f32b6a44640 1 -- 172.21.15.45:0/2505256887 wait complete. 2023-11-28T20:54:10.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.525+0000 7f32b6a44640 1 Processor -- start 2023-11-28T20:54:10.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.525+0000 7f32b6a44640 1 -- start start 2023-11-28T20:54:10.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.526+0000 7f32b6a44640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f32b009ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.526+0000 7f32b6a44640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b015fe30 0x7f32b01602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.526+0000 7f32b6a44640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b01f6520 0x7f32b01f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.526+0000 7f32b6a44640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b009cf70 con 0x7f329c003ef0 2023-11-28T20:54:10.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.526+0000 7f32b6a44640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b01607f0 con 0x7f32b015fe30 2023-11-28T20:54:10.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.526+0000 7f32b6a44640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0140980 con 0x7f32b01f6520 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5a42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f32b009ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5a42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f32b009ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46050/0 (socket says 172.21.15.45:46050) 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5a42640 1 -- 172.21.15.45:0/2182154737 learned_addr learned my addr 172.21.15.45:0/2182154737 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b6243640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b01f6520 0x7f32b01f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5241640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b015fe30 0x7f32b01602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5a42640 1 -- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b01f6520 msgr2=0x7f32b01f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5a42640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b01f6520 0x7f32b01f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5a42640 1 -- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b015fe30 msgr2=0x7f32b01602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5a42640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b015fe30 0x7f32b01602b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5a42640 1 -- 172.21.15.45:0/2182154737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b0141990 con 0x7f329c003ef0 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b6243640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b01f6520 0x7f32b01f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.527+0000 7f32b5a42640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f32b009ca30 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f32ac011600 tx=0x7f32ac011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:10.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.528+0000 7f32a67fc640 1 -- 172.21.15.45:0/2182154737 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac00f920 con 0x7f329c003ef0 2023-11-28T20:54:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.528+0000 7f32a67fc640 1 -- 172.21.15.45:0/2182154737 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ac00fa80 con 0x7f329c003ef0 2023-11-28T20:54:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.528+0000 7f32a67fc640 1 -- 172.21.15.45:0/2182154737 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac00fdf0 con 0x7f329c003ef0 2023-11-28T20:54:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.528+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b0140bd0 con 0x7f329c003ef0 2023-11-28T20:54:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.528+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b0141190 con 0x7f329c003ef0 2023-11-28T20:54:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.529+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b009d0f0 con 0x7f329c003ef0 2023-11-28T20:54:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.530+0000 7f32a67fc640 1 -- 172.21.15.45:0/2182154737 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 67) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f32ac015460 con 0x7f329c003ef0 2023-11-28T20:54:10.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.531+0000 7f32a67fc640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f329406aed0 0x7f329406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:10.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.531+0000 7f32a67fc640 1 -- 172.21.15.45:0/2182154737 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f32ac089f80 con 0x7f329c003ef0 2023-11-28T20:54:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.532+0000 7f32b5241640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f329406aed0 0x7f329406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:10.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.533+0000 7f32b5241640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f329406aed0 0x7f329406d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f32a0034000 tx=0x7f32a0007000 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:10.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.535+0000 7f32a67fc640 1 -- 172.21.15.45:0/2182154737 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32ac054c50 con 0x7f329c003ef0 2023-11-28T20:54:10.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.768+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32b00ce200 con 0x7f329c003ef0 2023-11-28T20:54:10.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.770+0000 7f32a67fc640 1 -- 172.21.15.45:0/2182154737 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f32ac058270 con 0x7f329c003ef0 2023-11-28T20:54:10.772 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:10.772 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/544145878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":544145878},{"type":"v1","addr":"172.21.15.190:6836","nonce":544145878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2464692926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2464692926},{"type":"v1","addr":"172.21.15.45:6835","nonce":2464692926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2048455360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2048455360},{"type":"v1","addr":"172.21.15.190:6837","nonce":2048455360}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:53:35.282168+0000","modified":"2023-11-28T20:53:38.346980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6834/2464465186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2464465186},{"type":"v1","addr":"172.21.15.45:6834","nonce":2464465186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-11-28T20:54:10.772 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 59 2023-11-28T20:54:10.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f329406aed0 msgr2=0x7f329406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f329406aed0 0x7f329406d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f32a0034000 tx=0x7f32a0007000 comp rx=0 tx=0).stop 2023-11-28T20:54:10.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 msgr2=0x7f32b009ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:10.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f32b009ca30 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f32ac011600 tx=0x7f32ac011ac0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 shutdown_connections 2023-11-28T20:54:10.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f329406aed0 0x7f329406d390 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32b01f6520 0x7f32b01f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b015fe30 0x7f32b01602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 --2- 172.21.15.45:0/2182154737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f329c003ef0 0x7f32b009ca30 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:10.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.772+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 >> 172.21.15.45:0/2182154737 conn(0x7f32b006ab30 msgr2=0x7f32b00ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:10.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.773+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 shutdown_connections 2023-11-28T20:54:10.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:10.773+0000 7f32b6a44640 1 -- 172.21.15.45:0/2182154737 wait complete. 2023-11-28T20:54:10.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.021+0000 7f6f42632640 1 Processor -- start 2023-11-28T20:54:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.022+0000 7f6f42632640 1 -- start start 2023-11-28T20:54:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.023+0000 7f6f42632640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f3c155780 0x7f6f3c0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.023+0000 7f6f42632640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6f3c0ce800 0x7f6f3c0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.023+0000 7f6f42632640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c15c3c0 con 0x7f6f3c0ae540 2023-11-28T20:54:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.023+0000 7f6f42632640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f3c155b60 con 0x7f6f3c0ce800 2023-11-28T20:54:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.023+0000 7f6f42632640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c15ba90 con 0x7f6f3c155780 2023-11-28T20:54:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.024+0000 7f6f40e2f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6f3c0ae540 0x7f6f3c0ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49988/0 (socket says 172.21.15.45:49988) 2023-11-28T20:54:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.024+0000 7f6f40e2f640 1 -- 172.21.15.45:0/140371094 learned_addr learned my addr 172.21.15.45:0/140371094 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.024+0000 7f6f41e31640 1 --2- 172.21.15.45:0/140371094 >> v2:172.21.15.45:3300/0 conn(0x7f6f3c0ce800 0x7f6f3c0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.024+0000 7f6f41630640 1 --2- 172.21.15.45:0/140371094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f3c155780 0x7f6f3c0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.024+0000 7f6f41e31640 1 -- 172.21.15.45:0/140371094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f3c155780 msgr2=0x7f6f3c0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:11.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.024+0000 7f6f41e31640 1 --2- 172.21.15.45:0/140371094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f3c155780 0x7f6f3c0abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.024+0000 7f6f33fff640 1 -- 172.21.15.45:0/140371094 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610554554 0 0) 0x7f6f3c15c3c0 con 0x7f6f3c0ae540 2023-11-28T20:54:11.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.024+0000 7f6f41e31640 1 -- 172.21.15.45:0/140371094 >> v1:172.21.15.45:6789/0 conn(0x7f6f3c0ae540 legacy=0x7f6f3c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.024+0000 7f6f41e31640 1 -- 172.21.15.45:0/140371094 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c15bd70 con 0x7f6f3c0ce800 2023-11-28T20:54:11.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.025+0000 7f6f41e31640 1 --2- 172.21.15.45:0/140371094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ce800 0x7f6f3c0b20a0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f6f34009d90 tx=0x7f6f3402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b499a81ed6ab88 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:11.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.026+0000 7f6f33fff640 1 -- 172.21.15.45:0/140371094 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3402cc50 con 0x7f6f3c0ce800 2023-11-28T20:54:11.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.026+0000 7f6f33fff640 1 -- 172.21.15.45:0/140371094 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3402cdb0 con 0x7f6f3c0ce800 2023-11-28T20:54:11.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.026+0000 7f6f33fff640 1 -- 172.21.15.45:0/140371094 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34034660 con 0x7f6f3c0ce800 2023-11-28T20:54:11.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.026+0000 7f6f42632640 1 -- 172.21.15.45:0/140371094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ce800 msgr2=0x7f6f3c0b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.026+0000 7f6f42632640 1 --2- 172.21.15.45:0/140371094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ce800 0x7f6f3c0b20a0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f6f34009d90 tx=0x7f6f3402bea0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.027+0000 7f6f42632640 1 -- 172.21.15.45:0/140371094 shutdown_connections 2023-11-28T20:54:11.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.027+0000 7f6f42632640 1 --2- 172.21.15.45:0/140371094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ce800 0x7f6f3c0b20a0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.027+0000 7f6f42632640 1 --2- 172.21.15.45:0/140371094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f3c155780 0x7f6f3c0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.027+0000 7f6f42632640 1 -- 172.21.15.45:0/140371094 >> 172.21.15.45:0/140371094 conn(0x7f6f3c06ab30 msgr2=0x7f6f3c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.027+0000 7f6f42632640 1 -- 172.21.15.45:0/140371094 shutdown_connections 2023-11-28T20:54:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.027+0000 7f6f42632640 1 -- 172.21.15.45:0/140371094 wait complete. 2023-11-28T20:54:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.028+0000 7f6f42632640 1 Processor -- start 2023-11-28T20:54:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.029+0000 7f6f42632640 1 -- start start 2023-11-28T20:54:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.029+0000 7f6f42632640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ae540 0x7f6f3c1433d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f42632640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f3c0ce800 0x7f6f3c09ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f42632640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f3c155780 0x7f6f3c09bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f42632640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c146fe0 con 0x7f6f3c0ae540 2023-11-28T20:54:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f42632640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c1472c0 con 0x7f6f3c0ce800 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f42632640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c145180 con 0x7f6f3c155780 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f41630640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ae540 0x7f6f3c1433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f41630640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ae540 0x7f6f3c1433d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46072/0 (socket says 172.21.15.45:46072) 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f41630640 1 -- 172.21.15.45:0/376040199 learned_addr learned my addr 172.21.15.45:0/376040199 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f40e2f640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f3c0ce800 0x7f6f3c09ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f41630640 1 -- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f3c155780 msgr2=0x7f6f3c09bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f41630640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f3c155780 0x7f6f3c09bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f41630640 1 -- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f3c0ce800 msgr2=0x7f6f3c09ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f41630640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f3c0ce800 0x7f6f3c09ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f41630640 1 -- 172.21.15.45:0/376040199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f34009a70 con 0x7f6f3c0ae540 2023-11-28T20:54:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.030+0000 7f6f40e2f640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f3c0ce800 0x7f6f3c09ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:11.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.031+0000 7f6f41630640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ae540 0x7f6f3c1433d0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800d620 tx=0x7f6f3800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:11.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.031+0000 7f6f31ffb640 1 -- 172.21.15.45:0/376040199 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3800b620 con 0x7f6f3c0ae540 2023-11-28T20:54:11.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.031+0000 7f6f31ffb640 1 -- 172.21.15.45:0/376040199 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3800b780 con 0x7f6f3c0ae540 2023-11-28T20:54:11.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.031+0000 7f6f31ffb640 1 -- 172.21.15.45:0/376040199 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3800bb30 con 0x7f6f3c0ae540 2023-11-28T20:54:11.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.031+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f3c145460 con 0x7f6f3c0ae540 2023-11-28T20:54:11.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.031+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3c144500 con 0x7f6f3c0ae540 2023-11-28T20:54:11.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.032+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3c04e920 con 0x7f6f3c0ae540 2023-11-28T20:54:11.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.033+0000 7f6f31ffb640 1 -- 172.21.15.45:0/376040199 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 67) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f6f3800bc90 con 0x7f6f3c0ae540 2023-11-28T20:54:11.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.034+0000 7f6f31ffb640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6f2006aed0 0x7f6f2006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.034+0000 7f6f31ffb640 1 -- 172.21.15.45:0/376040199 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f6f38089ff0 con 0x7f6f3c0ae540 2023-11-28T20:54:11.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.035+0000 7f6f40e2f640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6f2006aed0 0x7f6f2006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.036+0000 7f6f40e2f640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6f2006aed0 0x7f6f2006d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c1533c0 tx=0x7f6f2c009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:11.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.039+0000 7f6f31ffb640 1 -- 172.21.15.45:0/376040199 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f38054bc0 con 0x7f6f3c0ae540 2023-11-28T20:54:11.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.233+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f3c0af940 con 0x7f6f3c0ae540 2023-11-28T20:54:11.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.236+0000 7f6f31ffb640 1 -- 172.21.15.45:0/376040199 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+17954 (secure 0 0 0) 0x7f6f380581e0 con 0x7f6f3c0ae540 2023-11-28T20:54:11.239 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:11.239 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":136,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:08.585686+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:53:33.200533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-11-28T20:53:34.300717+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":136,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3643991290":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/304862452":"2023-11-29T20:54:07.580192+0000","172.21.15.190:6833/3157274352":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/2315351474":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/259384681":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/3489922358":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/665095987":"2023-11-29T20:54:07.580192+0000","172.21.15.190:6832/3157274352":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/2592164098":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/800294716":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/3135688549":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/369403092":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6833/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6832/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1007604882":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/685209934":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1745589824":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1706225465":"2023-11-29T20:53:49.553172+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:11.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.238+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6f2006aed0 msgr2=0x7f6f2006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.238+0000 7f6f42632640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6f2006aed0 0x7f6f2006d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c1533c0 tx=0x7f6f2c009040 comp rx=0 tx=0).stop 2023-11-28T20:54:11.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.238+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ae540 msgr2=0x7f6f3c1433d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.238+0000 7f6f42632640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ae540 0x7f6f3c1433d0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800d620 tx=0x7f6f3800dae0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.239+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 shutdown_connections 2023-11-28T20:54:11.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.239+0000 7f6f42632640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6f2006aed0 0x7f6f2006d390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.239+0000 7f6f42632640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f3c155780 0x7f6f3c09bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.239+0000 7f6f42632640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f3c0ce800 0x7f6f3c09ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.239+0000 7f6f42632640 1 --2- 172.21.15.45:0/376040199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f3c0ae540 0x7f6f3c1433d0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.239+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 >> 172.21.15.45:0/376040199 conn(0x7f6f3c06ab30 msgr2=0x7f6f3c0afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:11.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.239+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 shutdown_connections 2023-11-28T20:54:11.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.239+0000 7f6f42632640 1 -- 172.21.15.45:0/376040199 wait complete. 2023-11-28T20:54:11.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T20:54:11.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.481+0000 7fc9572a6640 1 Processor -- start 2023-11-28T20:54:11.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.481+0000 7fc9572a6640 1 -- start start 2023-11-28T20:54:11.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.483+0000 7fc9572a6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc95006cbe0 0x7fc95015d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.483+0000 7fc9572a6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95006d070 con 0x7fc95006e740 2023-11-28T20:54:11.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.483+0000 7fc9572a6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95006d350 con 0x7fc9500cef10 2023-11-28T20:54:11.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.483+0000 7fc9572a6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc95006d630 con 0x7fc95006cbe0 2023-11-28T20:54:11.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.483+0000 7fc954841640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc95006cbe0 0x7fc95015d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.483+0000 7fc954841640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc95006cbe0 0x7fc95015d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46084/0 (socket says 172.21.15.45:46084) 2023-11-28T20:54:11.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.483+0000 7fc954841640 1 -- 172.21.15.45:0/1616970376 learned_addr learned my addr 172.21.15.45:0/1616970376 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:11.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.484+0000 7fc954841640 1 -- 172.21.15.45:0/1616970376 >> v1:172.21.15.190:6789/0 conn(0x7fc9500cef10 legacy=0x7fc9500b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.484+0000 7fc954841640 1 -- 172.21.15.45:0/1616970376 >> v1:172.21.15.45:6789/0 conn(0x7fc95006e740 legacy=0x7fc95006a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.484+0000 7fc954841640 1 -- 172.21.15.45:0/1616970376 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95006d910 con 0x7fc95006cbe0 2023-11-28T20:54:11.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.485+0000 7fc954841640 1 --2- 172.21.15.45:0/1616970376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006cbe0 0x7fc95015d7e0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fc940009870 tx=0x7fc94002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad912ecf7717bf2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:11.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.485+0000 7fc947fff640 1 -- 172.21.15.45:0/1616970376 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940030850 con 0x7fc95006cbe0 2023-11-28T20:54:11.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.485+0000 7fc947fff640 1 -- 172.21.15.45:0/1616970376 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9400309b0 con 0x7fc95006cbe0 2023-11-28T20:54:11.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.485+0000 7fc947fff640 1 -- 172.21.15.45:0/1616970376 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940030d40 con 0x7fc95006cbe0 2023-11-28T20:54:11.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.486+0000 7fc9572a6640 1 -- 172.21.15.45:0/1616970376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006cbe0 msgr2=0x7fc95015d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.486+0000 7fc9572a6640 1 --2- 172.21.15.45:0/1616970376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006cbe0 0x7fc95015d7e0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fc940009870 tx=0x7fc94002b990 comp rx=0 tx=0).stop 2023-11-28T20:54:11.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.486+0000 7fc9572a6640 1 -- 172.21.15.45:0/1616970376 shutdown_connections 2023-11-28T20:54:11.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.486+0000 7fc9572a6640 1 --2- 172.21.15.45:0/1616970376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006cbe0 0x7fc95015d7e0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.486+0000 7fc9572a6640 1 -- 172.21.15.45:0/1616970376 >> 172.21.15.45:0/1616970376 conn(0x7fc9500cf610 msgr2=0x7fc9500cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:11.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.486+0000 7fc9572a6640 1 -- 172.21.15.45:0/1616970376 shutdown_connections 2023-11-28T20:54:11.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.486+0000 7fc9572a6640 1 -- 172.21.15.45:0/1616970376 wait complete. 2023-11-28T20:54:11.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.488+0000 7fc9572a6640 1 Processor -- start 2023-11-28T20:54:11.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.488+0000 7fc9572a6640 1 -- start start 2023-11-28T20:54:11.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.489+0000 7fc9572a6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006e740 0x7fc950143f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.489+0000 7fc9572a6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9500cef10 0x7fc95015e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.489+0000 7fc9572a6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9501f3130 0x7fc95015eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.489+0000 7fc9572a6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc950152e70 con 0x7fc95006e740 2023-11-28T20:54:11.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.489+0000 7fc9572a6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9501e7d40 con 0x7fc9501f3130 2023-11-28T20:54:11.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.489+0000 7fc9572a6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9501e8020 con 0x7fc9500cef10 2023-11-28T20:54:11.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.489+0000 7fc955042640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006e740 0x7fc950143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.489+0000 7fc955042640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006e740 0x7fc950143f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46098/0 (socket says 172.21.15.45:46098) 2023-11-28T20:54:11.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.489+0000 7fc955042640 1 -- 172.21.15.45:0/2966213513 learned_addr learned my addr 172.21.15.45:0/2966213513 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:11.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc955042640 1 -- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9500cef10 msgr2=0x7fc95015e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc955042640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9500cef10 0x7fc95015e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc955042640 1 -- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9501f3130 msgr2=0x7fc95015eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc955042640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9501f3130 0x7fc95015eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc955042640 1 -- 172.21.15.45:0/2966213513 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400094d0 con 0x7fc95006e740 2023-11-28T20:54:11.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc955042640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006e740 0x7fc950143f30 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00c450 tx=0x7fc94c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:11.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc945ffb640 1 -- 172.21.15.45:0/2966213513 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0052d0 con 0x7fc95006e740 2023-11-28T20:54:11.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc945ffb640 1 -- 172.21.15.45:0/2966213513 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c005430 con 0x7fc95006e740 2023-11-28T20:54:11.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc945ffb640 1 -- 172.21.15.45:0/2966213513 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c012870 con 0x7fc95006e740 2023-11-28T20:54:11.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9501e8300 con 0x7fc95006e740 2023-11-28T20:54:11.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.490+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc950144470 con 0x7fc95006e740 2023-11-28T20:54:11.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.491+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9501e8820 con 0x7fc95006e740 2023-11-28T20:54:11.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.492+0000 7fc945ffb640 1 -- 172.21.15.45:0/2966213513 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 67) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fc94c0129d0 con 0x7fc95006e740 2023-11-28T20:54:11.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.493+0000 7fc945ffb640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc93406aed0 0x7fc93406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.493+0000 7fc945ffb640 1 -- 172.21.15.45:0/2966213513 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fc94c089660 con 0x7fc95006e740 2023-11-28T20:54:11.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.494+0000 7fc954841640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc93406aed0 0x7fc93406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.495+0000 7fc954841640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc93406aed0 0x7fc93406d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc940036000 tx=0x7fc94003b040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:11.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.498+0000 7fc945ffb640 1 -- 172.21.15.45:0/2966213513 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc94c054230 con 0x7fc95006e740 2023-11-28T20:54:11.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.641+0000 7fc945ffb640 1 -- 172.21.15.45:0/2966213513 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc94c054600 con 0x7fc95006e740 2023-11-28T20:54:11.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.735+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc950073440 con 0x7fc95006e740 2023-11-28T20:54:11.738 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T20:54:11.736+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T20:54:11.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.740+0000 7fc945ffb640 1 -- 172.21.15.45:0/2966213513 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v60) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc94c0717f0 con 0x7fc95006e740 2023-11-28T20:54:11.742 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T20:54:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.743+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc93406aed0 msgr2=0x7fc93406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.743+0000 7fc9572a6640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc93406aed0 0x7fc93406d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc940036000 tx=0x7fc94003b040 comp rx=0 tx=0).stop 2023-11-28T20:54:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.743+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006e740 msgr2=0x7fc950143f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.743+0000 7fc9572a6640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006e740 0x7fc950143f30 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00c450 tx=0x7fc94c00c910 comp rx=0 tx=0).stop 2023-11-28T20:54:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.744+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 shutdown_connections 2023-11-28T20:54:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.744+0000 7fc9572a6640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc93406aed0 0x7fc93406d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.744+0000 7fc9572a6640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9501f3130 0x7fc95015eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.744+0000 7fc9572a6640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc9500cef10 0x7fc95015e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.744+0000 7fc9572a6640 1 --2- 172.21.15.45:0/2966213513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc95006e740 0x7fc950143f30 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.744+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 >> 172.21.15.45:0/2966213513 conn(0x7fc9500cf610 msgr2=0x7fc95006b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:11.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.744+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 shutdown_connections 2023-11-28T20:54:11.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.744+0000 7fc9572a6640 1 -- 172.21.15.45:0/2966213513 wait complete. 2023-11-28T20:54:11.756 INFO:tasks.ceph.mds.a.smithi045.stderr: -6094> 2023-11-28T20:53:40.410+0000 7f4758e3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:54:11.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T20:54:11.809 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:54:11.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.958+0000 7f485a290640 1 Processor -- start 2023-11-28T20:54:11.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.958+0000 7f485a290640 1 -- start start 2023-11-28T20:54:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.959+0000 7f485a290640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485415e840 0x7f485406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.959+0000 7f485a290640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f485406ac10 0x7f48540717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.959+0000 7f485a290640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485415ecd0 con 0x7f485415f720 2023-11-28T20:54:11.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.959+0000 7f485a290640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4854071ce0 con 0x7f485406ac10 2023-11-28T20:54:11.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.959+0000 7f485a290640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485415f290 con 0x7f485415e840 2023-11-28T20:54:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.960+0000 7f48537fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f485415f720 0x7f485406a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50016/0 (socket says 172.21.15.45:50016) 2023-11-28T20:54:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.960+0000 7f48537fe640 1 -- 172.21.15.45:0/901700431 learned_addr learned my addr 172.21.15.45:0/901700431 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.960+0000 7f485882d640 1 --2- 172.21.15.45:0/901700431 >> v2:172.21.15.190:3300/0 conn(0x7f485406ac10 0x7f48540717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.960+0000 7f4853fff640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485415e840 0x7f485406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.960+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201312854 0 0) 0x7f485415ecd0 con 0x7f485415f720 2023-11-28T20:54:11.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.960+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840003520 con 0x7f485415f720 2023-11-28T20:54:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3586479573 0 0) 0x7f4840003520 con 0x7f485415f720 2023-11-28T20:54:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485415e840 msgr2=0x7f485406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485415e840 0x7f485406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 >> v2:172.21.15.190:3300/0 conn(0x7f485406ac10 msgr2=0x7f48540717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 --2- 172.21.15.45:0/901700431 >> v2:172.21.15.190:3300/0 conn(0x7f485406ac10 0x7f48540717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485415fb60 con 0x7f485415f720 2023-11-28T20:54:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f483c0019f0 con 0x7f485415f720 2023-11-28T20:54:11.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 >> v1:172.21.15.45:6789/0 conn(0x7f485415f720 legacy=0x7f485406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4840003970 0x7f4840003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485415e840 0x7f48400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4840004d00 0x7f4840005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485415f290 con 0x7f4840003970 2023-11-28T20:54:11.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484000b980 con 0x7f4840004d00 2023-11-28T20:54:11.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484000bac0 con 0x7f485415e840 2023-11-28T20:54:11.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.961+0000 7f4853fff640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4840003970 0x7f4840003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.962+0000 7f48537fe640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485415e840 0x7f48400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.962+0000 7f485882d640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4840004d00 0x7f4840005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.962+0000 7f4853fff640 1 -- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485415e840 msgr2=0x7f48400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:11.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.962+0000 7f4853fff640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485415e840 0x7f48400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.962+0000 7f4853fff640 1 -- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4840004d00 msgr2=0x7f4840005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:11.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.962+0000 7f4853fff640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4840004d00 0x7f4840005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.962+0000 7f4853fff640 1 -- 172.21.15.45:0/901700431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4840003520 con 0x7f4840003970 2023-11-28T20:54:11.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.963+0000 7f4853fff640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4840003970 0x7f4840003d90 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4844009ac0 tx=0x7f48400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81db9510e6c2d903 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:11.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484402c6b0 con 0x7f4840003970 2023-11-28T20:54:11.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484402c810 con 0x7f4840003970 2023-11-28T20:54:11.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f4852ffd640 1 -- 172.21.15.45:0/901700431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484402cba0 con 0x7f4840003970 2023-11-28T20:54:11.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f485a290640 1 -- 172.21.15.45:0/901700431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4840003970 msgr2=0x7f4840003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f485a290640 1 --2- 172.21.15.45:0/901700431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4840003970 0x7f4840003d90 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4844009ac0 tx=0x7f48400057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f485882d640 1 -- 172.21.15.45:0/901700431 reap_dead start 2023-11-28T20:54:11.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f485a290640 1 -- 172.21.15.45:0/901700431 shutdown_connections 2023-11-28T20:54:11.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f485a290640 1 -- 172.21.15.45:0/901700431 >> 172.21.15.45:0/901700431 conn(0x7f48540ced60 msgr2=0x7f48540cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:11.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f485a290640 1 -- 172.21.15.45:0/901700431 shutdown_connections 2023-11-28T20:54:11.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.964+0000 7f485a290640 1 -- 172.21.15.45:0/901700431 wait complete. 2023-11-28T20:54:11.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.966+0000 7f485a290640 1 Processor -- start 2023-11-28T20:54:11.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.966+0000 7f485a290640 1 -- start start 2023-11-28T20:54:11.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.967+0000 7f485a290640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48540c8410 0x7f48540c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.967+0000 7f485a290640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48540b3230 0x7f48540b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.967+0000 7f485a290640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485414e540 0x7f485414e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.967+0000 7f485a290640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485414d4a0 con 0x7f48540c8410 2023-11-28T20:54:11.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.967+0000 7f485a290640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485414d780 con 0x7f485414e540 2023-11-28T20:54:11.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.967+0000 7f485a290640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485414da60 con 0x7f48540b3230 2023-11-28T20:54:11.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f4853fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48540c8410 0x7f48540c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f4853fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48540c8410 0x7f48540c87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46138/0 (socket says 172.21.15.45:46138) 2023-11-28T20:54:11.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f4853fff640 1 -- 172.21.15.45:0/3054450519 learned_addr learned my addr 172.21.15.45:0/3054450519 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:11.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f48537fe640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48540b3230 0x7f48540b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f485882d640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485414e540 0x7f485414e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f4853fff640 1 -- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48540b3230 msgr2=0x7f48540b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f4853fff640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48540b3230 0x7f48540b36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f4853fff640 1 -- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485414e540 msgr2=0x7f485414e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:11.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f4853fff640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485414e540 0x7f485414e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:11.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.968+0000 7f4853fff640 1 -- 172.21.15.45:0/3054450519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485414dd40 con 0x7f48540c8410 2023-11-28T20:54:11.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.969+0000 7f4853fff640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48540c8410 0x7f48540c87f0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f48440099e0 tx=0x7f4844004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:11.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.969+0000 7f4850ff9640 1 -- 172.21.15.45:0/3054450519 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844003810 con 0x7f48540c8410 2023-11-28T20:54:11.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.969+0000 7f4850ff9640 1 -- 172.21.15.45:0/3054450519 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4844003970 con 0x7f48540c8410 2023-11-28T20:54:11.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.969+0000 7f4850ff9640 1 -- 172.21.15.45:0/3054450519 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844003ce0 con 0x7f48540c8410 2023-11-28T20:54:11.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.969+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485414e020 con 0x7f48540c8410 2023-11-28T20:54:11.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.969+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485414ef00 con 0x7f48540c8410 2023-11-28T20:54:11.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.970+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485414e1b0 con 0x7f48540c8410 2023-11-28T20:54:11.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.971+0000 7f4850ff9640 1 -- 172.21.15.45:0/3054450519 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f484403e440 con 0x7f48540c8410 2023-11-28T20:54:11.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.972+0000 7f4850ff9640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f483406af90 0x7f483406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:11.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.972+0000 7f4850ff9640 1 -- 172.21.15.45:0/3054450519 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f48440ab5c0 con 0x7f48540c8410 2023-11-28T20:54:11.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.973+0000 7f48537fe640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f483406af90 0x7f483406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:11.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.974+0000 7f48537fe640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f483406af90 0x7f483406d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f483c004bc0 tx=0x7f483c009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:11.977+0000 7f4850ff9640 1 -- 172.21.15.45:0/3054450519 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4844076000 con 0x7f48540c8410 2023-11-28T20:54:12.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.214+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f48540733d0 con 0x7f48540c8410 2023-11-28T20:54:12.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.743+0000 7f4850ff9640 1 -- 172.21.15.45:0/3054450519 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v61) v1 ==== 104+0+0 (secure 0 0 0) 0x7f48440044e0 con 0x7f48540c8410 2023-11-28T20:54:12.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f483406af90 msgr2=0x7f483406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:12.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f483406af90 0x7f483406d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f483c004bc0 tx=0x7f483c009040 comp rx=0 tx=0).stop 2023-11-28T20:54:12.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48540c8410 msgr2=0x7f48540c87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:12.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48540c8410 0x7f48540c87f0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f48440099e0 tx=0x7f4844004730 comp rx=0 tx=0).stop 2023-11-28T20:54:12.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 shutdown_connections 2023-11-28T20:54:12.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f483406af90 0x7f483406d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f485414e540 0x7f485414e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48540b3230 0x7f48540b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:12.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 --2- 172.21.15.45:0/3054450519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48540c8410 0x7f48540c87f0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.747+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 >> 172.21.15.45:0/3054450519 conn(0x7f48540ced60 msgr2=0x7f48540ad3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.748+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 shutdown_connections 2023-11-28T20:54:12.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:12.748+0000 7f485a290640 1 -- 172.21.15.45:0/3054450519 wait complete. 2023-11-28T20:54:12.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T20:54:13.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.040+0000 7f8461ee1640 1 Processor -- start 2023-11-28T20:54:13.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.040+0000 7f8461ee1640 1 -- start start 2023-11-28T20:54:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.042+0000 7f8461ee1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f845c0cbe10 0x7f845c071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.042+0000 7f8461ee1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f845c071a40 0x7f845c06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.042+0000 7f8461ee1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c070e20 con 0x7f845c1448e0 2023-11-28T20:54:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.042+0000 7f8461ee1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845c144cc0 con 0x7f845c071a40 2023-11-28T20:54:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.042+0000 7f8461ee1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845c070280 con 0x7f845c0cbe10 2023-11-28T20:54:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.043+0000 7f8460edf640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f845c1448e0 0x7f845c0cb700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50020/0 (socket says 172.21.15.45:50020) 2023-11-28T20:54:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.043+0000 7f8460edf640 1 -- 172.21.15.45:0/108751750 learned_addr learned my addr 172.21.15.45:0/108751750 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:13.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.043+0000 7f84616e0640 1 --2- 172.21.15.45:0/108751750 >> v2:172.21.15.45:3300/0 conn(0x7f845c071a40 0x7f845c06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:13.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.043+0000 7f845bfff640 1 --2- 172.21.15.45:0/108751750 >> v2:172.21.15.190:3300/0 conn(0x7f845c0cbe10 0x7f845c071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:13.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.043+0000 7f845b7fe640 1 -- 172.21.15.45:0/108751750 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356641249 0 0) 0x7f845c070e20 con 0x7f845c1448e0 2023-11-28T20:54:13.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.043+0000 7f845b7fe640 1 -- 172.21.15.45:0/108751750 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c003520 con 0x7f845c1448e0 2023-11-28T20:54:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.044+0000 7f84616e0640 1 -- 172.21.15.45:0/108751750 >> v2:172.21.15.190:3300/0 conn(0x7f845c0cbe10 msgr2=0x7f845c071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.044+0000 7f84616e0640 1 --2- 172.21.15.45:0/108751750 >> v2:172.21.15.190:3300/0 conn(0x7f845c0cbe10 0x7f845c071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.044+0000 7f84616e0640 1 -- 172.21.15.45:0/108751750 >> v1:172.21.15.45:6789/0 conn(0x7f845c1448e0 legacy=0x7f845c0cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.044+0000 7f84616e0640 1 -- 172.21.15.45:0/108751750 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c0704f0 con 0x7f845c071a40 2023-11-28T20:54:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.045+0000 7f84616e0640 1 --2- 172.21.15.45:0/108751750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c071a40 0x7f845c06cda0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f845c06e340 tx=0x7f845402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437e8e2bc556a6e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.045+0000 7f845b7fe640 1 -- 172.21.15.45:0/108751750 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845402cbf0 con 0x7f845c071a40 2023-11-28T20:54:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.045+0000 7f845b7fe640 1 -- 172.21.15.45:0/108751750 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845402cd50 con 0x7f845c071a40 2023-11-28T20:54:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.045+0000 7f845b7fe640 1 -- 172.21.15.45:0/108751750 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8454034660 con 0x7f845c071a40 2023-11-28T20:54:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.046+0000 7f8461ee1640 1 -- 172.21.15.45:0/108751750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c071a40 msgr2=0x7f845c06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.046+0000 7f8461ee1640 1 --2- 172.21.15.45:0/108751750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c071a40 0x7f845c06cda0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f845c06e340 tx=0x7f845402be40 comp rx=0 tx=0).stop 2023-11-28T20:54:13.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.046+0000 7f8461ee1640 1 -- 172.21.15.45:0/108751750 shutdown_connections 2023-11-28T20:54:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.046+0000 7f8461ee1640 1 --2- 172.21.15.45:0/108751750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c071a40 0x7f845c06cda0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.046+0000 7f8461ee1640 1 --2- 172.21.15.45:0/108751750 >> v2:172.21.15.190:3300/0 conn(0x7f845c0cbe10 0x7f845c071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.046+0000 7f8461ee1640 1 -- 172.21.15.45:0/108751750 >> 172.21.15.45:0/108751750 conn(0x7f845c0cec70 msgr2=0x7f845c0cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.046+0000 7f8461ee1640 1 -- 172.21.15.45:0/108751750 shutdown_connections 2023-11-28T20:54:13.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.046+0000 7f8461ee1640 1 -- 172.21.15.45:0/108751750 wait complete. 2023-11-28T20:54:13.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.048+0000 7f8461ee1640 1 Processor -- start 2023-11-28T20:54:13.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.048+0000 7f8461ee1640 1 -- start start 2023-11-28T20:54:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.049+0000 7f8461ee1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f845c071a40 0x7f845c15a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.049+0000 7f8461ee1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c0cbe10 0x7f845c0b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.049+0000 7f8461ee1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f845c1448e0 0x7f845c06cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.049+0000 7f8461ee1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c1f69f0 con 0x7f845c0cbe10 2023-11-28T20:54:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.049+0000 7f8461ee1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c0ce0d0 con 0x7f845c1448e0 2023-11-28T20:54:13.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.049+0000 7f8461ee1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845c0ce3b0 con 0x7f845c071a40 2023-11-28T20:54:13.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.049+0000 7f845bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c0cbe10 0x7f845c0b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:13.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.049+0000 7f845bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c0cbe10 0x7f845c0b3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46148/0 (socket says 172.21.15.45:46148) 2023-11-28T20:54:13.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.049+0000 7f845bfff640 1 -- 172.21.15.45:0/3263207049 learned_addr learned my addr 172.21.15.45:0/3263207049 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f84616e0640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f845c1448e0 0x7f845c06cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f8460edf640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f845c071a40 0x7f845c15a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f845bfff640 1 -- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f845c071a40 msgr2=0x7f845c15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f845bfff640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f845c071a40 0x7f845c15a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f845bfff640 1 -- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f845c1448e0 msgr2=0x7f845c06cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f845bfff640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f845c1448e0 0x7f845c06cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f845bfff640 1 -- 172.21.15.45:0/3263207049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8454009a70 con 0x7f845c0cbe10 2023-11-28T20:54:13.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f845bfff640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c0cbe10 0x7f845c0b3570 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f844800e9b0 tx=0x7f844800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f84597fa640 1 -- 172.21.15.45:0/3263207049 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448009670 con 0x7f845c0cbe10 2023-11-28T20:54:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f84597fa640 1 -- 172.21.15.45:0/3263207049 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844800c9a0 con 0x7f845c0cbe10 2023-11-28T20:54:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f84597fa640 1 -- 172.21.15.45:0/3263207049 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844800cd50 con 0x7f845c0cbe10 2023-11-28T20:54:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.050+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c1f63a0 con 0x7f845c0cbe10 2023-11-28T20:54:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.051+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c159d80 con 0x7f845c0cbe10 2023-11-28T20:54:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.052+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c1f6ba0 con 0x7f845c0cbe10 2023-11-28T20:54:13.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.053+0000 7f84597fa640 1 -- 172.21.15.45:0/3263207049 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8448013470 con 0x7f845c0cbe10 2023-11-28T20:54:13.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.054+0000 7f84597fa640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f844006af90 0x7f844006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:13.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.054+0000 7f84597fa640 1 -- 172.21.15.45:0/3263207049 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f8448089e90 con 0x7f845c0cbe10 2023-11-28T20:54:13.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.054+0000 7f8460edf640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f844006af90 0x7f844006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:13.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.055+0000 7f8460edf640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f844006af90 0x7f844006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f84500011e0 tx=0x7f8450009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:13.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.058+0000 7f84597fa640 1 -- 172.21.15.45:0/3263207049 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f845c1f6ba0 con 0x7f845c0cbe10 2023-11-28T20:54:13.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.254+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f845c066ed0 con 0x7f845c0cbe10 2023-11-28T20:54:13.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.755+0000 7f84597fa640 1 -- 172.21.15.45:0/3263207049 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v138) v1 ==== 181+0+0 (secure 0 0 0) 0x7f84480170d0 con 0x7f845c0cbe10 2023-11-28T20:54:13.758 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T20:54:13.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.760+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f844006af90 msgr2=0x7f844006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:13.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.760+0000 7f8461ee1640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f844006af90 0x7f844006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f84500011e0 tx=0x7f8450009040 comp rx=0 tx=0).stop 2023-11-28T20:54:13.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.760+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c0cbe10 msgr2=0x7f845c0b3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:13.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.760+0000 7f8461ee1640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c0cbe10 0x7f845c0b3570 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f844800e9b0 tx=0x7f844800ee70 comp rx=0 tx=0).stop 2023-11-28T20:54:13.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.761+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 shutdown_connections 2023-11-28T20:54:13.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.761+0000 7f8461ee1640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f844006af90 0x7f844006d450 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f84500011e0 tx=0x7f8450009040 comp rx=0 tx=0).stop 2023-11-28T20:54:13.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.761+0000 7f8461ee1640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f845c1448e0 0x7f845c06cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:13.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.761+0000 7f8461ee1640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f845c0cbe10 0x7f845c0b3570 secure :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f844800e9b0 tx=0x7f844800ee70 comp rx=0 tx=0).stop 2023-11-28T20:54:13.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.761+0000 7f8461ee1640 1 --2- 172.21.15.45:0/3263207049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f845c071a40 0x7f845c15a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:13.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.761+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 >> 172.21.15.45:0/3263207049 conn(0x7f845c0cec70 msgr2=0x7f845c0cb700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:13.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.762+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 shutdown_connections 2023-11-28T20:54:13.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:13.763+0000 7f8461ee1640 1 -- 172.21.15.45:0/3263207049 wait complete. 2023-11-28T20:54:13.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T20:54:14.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.017+0000 7faac759e640 1 Processor -- start 2023-11-28T20:54:14.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.018+0000 7faac759e640 1 -- start start 2023-11-28T20:54:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.019+0000 7faac759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faac80cde00 0x7faac80ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.019+0000 7faac759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faac80b6380 0x7faac80b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faac759e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac815d790 con 0x7faac815d3b0 2023-11-28T20:54:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faac759e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac80ce7c0 con 0x7faac80cde00 2023-11-28T20:54:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faac759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac80d2bf0 con 0x7faac80b6380 2023-11-28T20:54:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faabffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faac80cde00 0x7faac80ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faabffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faac80cde00 0x7faac80ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40324/0 (socket says 172.21.15.45:40324) 2023-11-28T20:54:14.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faabffff640 1 -- 172.21.15.45:0/3772013908 learned_addr learned my addr 172.21.15.45:0/3772013908 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:14.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faabffff640 1 -- 172.21.15.45:0/3772013908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faac80b6380 msgr2=0x7faac80b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:14.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faabffff640 1 --2- 172.21.15.45:0/3772013908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faac80b6380 0x7faac80b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faabffff640 1 -- 172.21.15.45:0/3772013908 >> v1:172.21.15.190:6789/0 conn(0x7faac815d3b0 legacy=0x7faac80d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:14.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.020+0000 7faabffff640 1 -- 172.21.15.45:0/3772013908 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac80d2ed0 con 0x7faac80cde00 2023-11-28T20:54:14.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.022+0000 7faabffff640 1 --2- 172.21.15.45:0/3772013908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80cde00 0x7faac80ce280 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7faab0009870 tx=0x7faab002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97658f93bbd6a39f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:14.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.022+0000 7faac5d9b640 1 -- 172.21.15.45:0/3772013908 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0031640 con 0x7faac80cde00 2023-11-28T20:54:14.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.022+0000 7faac5d9b640 1 -- 172.21.15.45:0/3772013908 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab00317a0 con 0x7faac80cde00 2023-11-28T20:54:14.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.022+0000 7faac5d9b640 1 -- 172.21.15.45:0/3772013908 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0031b30 con 0x7faac80cde00 2023-11-28T20:54:14.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.022+0000 7faac759e640 1 -- 172.21.15.45:0/3772013908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80cde00 msgr2=0x7faac80ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:14.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.022+0000 7faac759e640 1 --2- 172.21.15.45:0/3772013908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80cde00 0x7faac80ce280 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7faab0009870 tx=0x7faab002b8f0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.023+0000 7faac759e640 1 -- 172.21.15.45:0/3772013908 shutdown_connections 2023-11-28T20:54:14.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.023+0000 7faac759e640 1 --2- 172.21.15.45:0/3772013908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faac80b6380 0x7faac80b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.023+0000 7faac759e640 1 --2- 172.21.15.45:0/3772013908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80cde00 0x7faac80ce280 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.023+0000 7faac759e640 1 -- 172.21.15.45:0/3772013908 >> 172.21.15.45:0/3772013908 conn(0x7faac8070f40 msgr2=0x7faac8071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:14.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.023+0000 7faac759e640 1 -- 172.21.15.45:0/3772013908 shutdown_connections 2023-11-28T20:54:14.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.023+0000 7faac759e640 1 -- 172.21.15.45:0/3772013908 wait complete. 2023-11-28T20:54:14.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.025+0000 7faac759e640 1 Processor -- start 2023-11-28T20:54:14.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.025+0000 7faac759e640 1 -- start start 2023-11-28T20:54:14.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.026+0000 7faac759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80b6380 0x7faac80cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:14.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.026+0000 7faac759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faac80cde00 0x7faac80cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:14.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.026+0000 7faac759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faac815d3b0 0x7faac80b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:14.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.026+0000 7faac759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac8158000 con 0x7faac80b6380 2023-11-28T20:54:14.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.026+0000 7faac759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac80cb850 con 0x7faac815d3b0 2023-11-28T20:54:14.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.026+0000 7faac759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac80cbb30 con 0x7faac80cde00 2023-11-28T20:54:14.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.026+0000 7faac659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80b6380 0x7faac80cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:14.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.026+0000 7faac659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80b6380 0x7faac80cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40326/0 (socket says 172.21.15.45:40326) 2023-11-28T20:54:14.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.026+0000 7faac659c640 1 -- 172.21.15.45:0/501746341 learned_addr learned my addr 172.21.15.45:0/501746341 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:14.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faac659c640 1 -- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faac80cde00 msgr2=0x7faac80cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:14.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faac659c640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faac80cde00 0x7faac80cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faac659c640 1 -- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faac815d3b0 msgr2=0x7faac80b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:14.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faac659c640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faac815d3b0 0x7faac80b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faac659c640 1 -- 172.21.15.45:0/501746341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab00094d0 con 0x7faac80b6380 2023-11-28T20:54:14.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faac659c640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80b6380 0x7faac80cb150 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7faab800b3e0 tx=0x7faab800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:14.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faabeffd640 1 -- 172.21.15.45:0/501746341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab8011830 con 0x7faac80b6380 2023-11-28T20:54:14.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faabeffd640 1 -- 172.21.15.45:0/501746341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab80119b0 con 0x7faac80b6380 2023-11-28T20:54:14.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faabeffd640 1 -- 172.21.15.45:0/501746341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab8011d60 con 0x7faac80b6380 2023-11-28T20:54:14.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac80cbe10 con 0x7faac80b6380 2023-11-28T20:54:14.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.027+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac80cc3d0 con 0x7faac80b6380 2023-11-28T20:54:14.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.029+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac80cc5c0 con 0x7faac80b6380 2023-11-28T20:54:14.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.030+0000 7faabeffd640 1 -- 172.21.15.45:0/501746341 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7faab801a400 con 0x7faac80b6380 2023-11-28T20:54:14.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.031+0000 7faabeffd640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7faaa806af90 0x7faaa806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:14.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.031+0000 7faabeffd640 1 -- 172.21.15.45:0/501746341 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6479+0+0 (secure 0 0 0) 0x7faab8089c30 con 0x7faac80b6380 2023-11-28T20:54:14.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.031+0000 7faabffff640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7faaa806af90 0x7faaa806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:14.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.033+0000 7faabffff640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7faaa806af90 0x7faaa806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faab0036000 tx=0x7faab0030000 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:14.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.035+0000 7faabeffd640 1 -- 172.21.15.45:0/501746341 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faab80548d0 con 0x7faac80b6380 2023-11-28T20:54:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.237+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faac80d2830 con 0x7faac80b6380 2023-11-28T20:54:14.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.759+0000 7faabeffd640 1 -- 172.21.15.45:0/501746341 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v139) v1 ==== 169+0+0 (secure 0 0 0) 0x7faab8015020 con 0x7faac80b6380 2023-11-28T20:54:14.762 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T20:54:14.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7faaa806af90 msgr2=0x7faaa806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:14.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7faaa806af90 0x7faaa806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faab0036000 tx=0x7faab0030000 comp rx=0 tx=0).stop 2023-11-28T20:54:14.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80b6380 msgr2=0x7faac80cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:14.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80b6380 0x7faac80cb150 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7faab800b3e0 tx=0x7faab800b8a0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 shutdown_connections 2023-11-28T20:54:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7faaa806af90 0x7faaa806d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faac815d3b0 0x7faac80b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faac80cde00 0x7faac80cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 --2- 172.21.15.45:0/501746341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac80b6380 0x7faac80cb150 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.764+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 >> 172.21.15.45:0/501746341 conn(0x7faac8070f40 msgr2=0x7faac80b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:14.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.766+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 shutdown_connections 2023-11-28T20:54:14.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:14.766+0000 7faac759e640 1 -- 172.21.15.45:0/501746341 wait complete. 2023-11-28T20:54:14.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T20:54:15.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.009+0000 7efd8759e640 1 Processor -- start 2023-11-28T20:54:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.010+0000 7efd8759e640 1 -- start start 2023-11-28T20:54:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.011+0000 7efd8759e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efd8815b080 0x7efd880ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.011+0000 7efd8759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 0x7efd880702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.011+0000 7efd8759e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880ac270 con 0x7efd880abd70 2023-11-28T20:54:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.011+0000 7efd8759e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd8815b830 con 0x7efd8815b080 2023-11-28T20:54:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.011+0000 7efd8759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8815bb10 con 0x7efd88071fd0 2023-11-28T20:54:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.012+0000 7efd86d9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 0x7efd880702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.012+0000 7efd8659c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efd8815b080 0x7efd880ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.012+0000 7efd85d9b640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7efd880abd70 0x7efd880718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:60540/0 (socket says 172.21.15.45:60540) 2023-11-28T20:54:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.012+0000 7efd85d9b640 1 -- 172.21.15.45:0/969886850 learned_addr learned my addr 172.21.15.45:0/969886850 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.013+0000 7efd8559a640 1 -- 172.21.15.45:0/969886850 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152141760 0 0) 0x7efd880ac270 con 0x7efd880abd70 2023-11-28T20:54:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.013+0000 7efd8559a640 1 -- 172.21.15.45:0/969886850 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd74003520 con 0x7efd880abd70 2023-11-28T20:54:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.013+0000 7efd86d9d640 1 -- 172.21.15.45:0/969886850 >> v2:172.21.15.190:3300/0 conn(0x7efd8815b080 msgr2=0x7efd880ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:15.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.013+0000 7efd86d9d640 1 --2- 172.21.15.45:0/969886850 >> v2:172.21.15.190:3300/0 conn(0x7efd8815b080 0x7efd880ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.013+0000 7efd86d9d640 1 -- 172.21.15.45:0/969886850 >> v1:172.21.15.190:6789/0 conn(0x7efd880abd70 legacy=0x7efd880718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.013+0000 7efd86d9d640 1 -- 172.21.15.45:0/969886850 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8815bdf0 con 0x7efd88071fd0 2023-11-28T20:54:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.015+0000 7efd86d9d640 1 --2- 172.21.15.45:0/969886850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 0x7efd880702c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efd800097a0 tx=0x7efd8002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a889372ceee8b413 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.015+0000 7efd8559a640 1 -- 172.21.15.45:0/969886850 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80031830 con 0x7efd88071fd0 2023-11-28T20:54:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.015+0000 7efd8559a640 1 -- 172.21.15.45:0/969886850 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd80031990 con 0x7efd88071fd0 2023-11-28T20:54:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.015+0000 7efd8559a640 1 -- 172.21.15.45:0/969886850 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80031d20 con 0x7efd88071fd0 2023-11-28T20:54:15.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.016+0000 7efd8759e640 1 -- 172.21.15.45:0/969886850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 msgr2=0x7efd880702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.016+0000 7efd8759e640 1 --2- 172.21.15.45:0/969886850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 0x7efd880702c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efd800097a0 tx=0x7efd8002b8d0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.016+0000 7efd8759e640 1 -- 172.21.15.45:0/969886850 shutdown_connections 2023-11-28T20:54:15.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.016+0000 7efd8759e640 1 --2- 172.21.15.45:0/969886850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 0x7efd880702c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.016+0000 7efd8759e640 1 --2- 172.21.15.45:0/969886850 >> v2:172.21.15.190:3300/0 conn(0x7efd8815b080 0x7efd880ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.016+0000 7efd8759e640 1 -- 172.21.15.45:0/969886850 >> 172.21.15.45:0/969886850 conn(0x7efd880cecd0 msgr2=0x7efd880cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:15.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.016+0000 7efd8759e640 1 -- 172.21.15.45:0/969886850 shutdown_connections 2023-11-28T20:54:15.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.017+0000 7efd8759e640 1 -- 172.21.15.45:0/969886850 wait complete. 2023-11-28T20:54:15.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.018+0000 7efd8759e640 1 Processor -- start 2023-11-28T20:54:15.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.018+0000 7efd8759e640 1 -- start start 2023-11-28T20:54:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.019+0000 7efd8759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 0x7efd881533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.019+0000 7efd8759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd880abd70 0x7efd881f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd8759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd8815b080 0x7efd881f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd8759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd880b3180 con 0x7efd880abd70 2023-11-28T20:54:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd8759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8809aa90 con 0x7efd8815b080 2023-11-28T20:54:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd8759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd880b33d0 con 0x7efd88071fd0 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd85d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd880abd70 0x7efd881f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd85d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd880abd70 0x7efd881f6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40330/0 (socket says 172.21.15.45:40330) 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd85d9b640 1 -- 172.21.15.45:0/592201297 learned_addr learned my addr 172.21.15.45:0/592201297 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd8659c640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 0x7efd881533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd86d9d640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd8815b080 0x7efd881f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd85d9b640 1 -- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 msgr2=0x7efd881533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd85d9b640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 0x7efd881533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd85d9b640 1 -- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd8815b080 msgr2=0x7efd881f6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd85d9b640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd8815b080 0x7efd881f6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.020+0000 7efd85d9b640 1 -- 172.21.15.45:0/592201297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd80009400 con 0x7efd880abd70 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.021+0000 7efd85d9b640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd880abd70 0x7efd881f6840 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7efd7000c530 tx=0x7efd7000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.021+0000 7efd7effd640 1 -- 172.21.15.45:0/592201297 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd70005ce0 con 0x7efd880abd70 2023-11-28T20:54:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.021+0000 7efd7effd640 1 -- 172.21.15.45:0/592201297 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd70005e40 con 0x7efd880abd70 2023-11-28T20:54:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.021+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd880b36b0 con 0x7efd880abd70 2023-11-28T20:54:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.021+0000 7efd7effd640 1 -- 172.21.15.45:0/592201297 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd700072b0 con 0x7efd880abd70 2023-11-28T20:54:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.021+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd880b3c70 con 0x7efd880abd70 2023-11-28T20:54:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.022+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd880b3e60 con 0x7efd880abd70 2023-11-28T20:54:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.023+0000 7efd7effd640 1 -- 172.21.15.45:0/592201297 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7efd70007410 con 0x7efd880abd70 2023-11-28T20:54:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.025+0000 7efd7effd640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7efd6806af60 0x7efd6806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.025+0000 7efd7effd640 1 -- 172.21.15.45:0/592201297 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7efd70088f00 con 0x7efd880abd70 2023-11-28T20:54:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.025+0000 7efd8659c640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7efd6806af60 0x7efd6806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.026+0000 7efd8659c640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7efd6806af60 0x7efd6806d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efd881e7530 tx=0x7efd7800b040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:15.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.029+0000 7efd7effd640 1 -- 172.21.15.45:0/592201297 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd70053de0 con 0x7efd880abd70 2023-11-28T20:54:15.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.226+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7efd8809d0f0 con 0x7efd880abd70 2023-11-28T20:54:15.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.233+0000 7efd7effd640 1 -- 172.21.15.45:0/592201297 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v43) v1 ==== 120+0+0 (secure 0 0 0) 0x7efd700170b0 con 0x7efd880abd70 2023-11-28T20:54:15.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7efd6806af60 msgr2=0x7efd6806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7efd6806af60 0x7efd6806d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efd881e7530 tx=0x7efd7800b040 comp rx=0 tx=0).stop 2023-11-28T20:54:15.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd880abd70 msgr2=0x7efd881f6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd880abd70 0x7efd881f6840 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7efd7000c530 tx=0x7efd7000c9f0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 shutdown_connections 2023-11-28T20:54:15.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7efd6806af60 0x7efd6806d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd8815b080 0x7efd881f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd880abd70 0x7efd881f6840 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 --2- 172.21.15.45:0/592201297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd88071fd0 0x7efd881533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.236+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 >> 172.21.15.45:0/592201297 conn(0x7efd880cecd0 msgr2=0x7efd8806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:15.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.237+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 shutdown_connections 2023-11-28T20:54:15.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.237+0000 7efd8759e640 1 -- 172.21.15.45:0/592201297 wait complete. 2023-11-28T20:54:15.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T20:54:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.478+0000 7fa5e64e0640 1 Processor -- start 2023-11-28T20:54:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.478+0000 7fa5e64e0640 1 -- start start 2023-11-28T20:54:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.480+0000 7fa5e64e0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 0x7fa5e006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.480+0000 7fa5e64e0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa5e015f720 0x7fa5e00717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.480+0000 7fa5e64e0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e015ecd0 con 0x7fa5e006a500 2023-11-28T20:54:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.480+0000 7fa5e64e0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e0071ce0 con 0x7fa5e015f720 2023-11-28T20:54:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.480+0000 7fa5e64e0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e015f290 con 0x7fa5e015e840 2023-11-28T20:54:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.480+0000 7fa5d7fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa5e015f720 0x7fa5e00717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.480+0000 7fa5d7fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa5e015f720 0x7fa5e00717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40334/0 (socket says 172.21.15.45:40334) 2023-11-28T20:54:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.480+0000 7fa5d7fff640 1 -- 172.21.15.45:0/3004445828 learned_addr learned my addr 172.21.15.45:0/3004445828 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.480+0000 7fa5dffff640 1 --2- 172.21.15.45:0/3004445828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 0x7fa5e006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.481+0000 7fa5d7fff640 1 -- 172.21.15.45:0/3004445828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 msgr2=0x7fa5e006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:15.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.481+0000 7fa5d7fff640 1 --2- 172.21.15.45:0/3004445828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 0x7fa5e006c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.481+0000 7fa5d7fff640 1 -- 172.21.15.45:0/3004445828 >> v1:172.21.15.190:6789/0 conn(0x7fa5e006a500 legacy=0x7fa5e00cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.481+0000 7fa5d7fff640 1 -- 172.21.15.45:0/3004445828 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e015fb60 con 0x7fa5e015f720 2023-11-28T20:54:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.482+0000 7fa5d7fff640 1 --2- 172.21.15.45:0/3004445828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 0x7fa5e00717a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80098a0 tx=0x7fa5c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405afe64aa7c3f29 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.482+0000 7fa5df7fe640 1 -- 172.21.15.45:0/3004445828 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8030840 con 0x7fa5e015f720 2023-11-28T20:54:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.482+0000 7fa5df7fe640 1 -- 172.21.15.45:0/3004445828 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c80309a0 con 0x7fa5e015f720 2023-11-28T20:54:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.482+0000 7fa5df7fe640 1 -- 172.21.15.45:0/3004445828 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8030d30 con 0x7fa5e015f720 2023-11-28T20:54:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.483+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3004445828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 msgr2=0x7fa5e00717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.483+0000 7fa5e64e0640 1 --2- 172.21.15.45:0/3004445828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 0x7fa5e00717a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80098a0 tx=0x7fa5c802b980 comp rx=0 tx=0).stop 2023-11-28T20:54:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.483+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3004445828 shutdown_connections 2023-11-28T20:54:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.483+0000 7fa5e64e0640 1 --2- 172.21.15.45:0/3004445828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 0x7fa5e00717a0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.483+0000 7fa5e64e0640 1 --2- 172.21.15.45:0/3004445828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 0x7fa5e006c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.483+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3004445828 >> 172.21.15.45:0/3004445828 conn(0x7fa5e00ced60 msgr2=0x7fa5e00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.483+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3004445828 shutdown_connections 2023-11-28T20:54:15.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.483+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3004445828 wait complete. 2023-11-28T20:54:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.485+0000 7fa5e64e0640 1 Processor -- start 2023-11-28T20:54:15.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.485+0000 7fa5e64e0640 1 -- start start 2023-11-28T20:54:15.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.486+0000 7fa5e64e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5e006a500 0x7fa5e015c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.486+0000 7fa5e64e0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 0x7fa5e015a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.486+0000 7fa5e64e0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 0x7fa5e015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.486+0000 7fa5e64e0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e015dac0 con 0x7fa5e015f720 2023-11-28T20:54:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.486+0000 7fa5e64e0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00b2c20 con 0x7fa5e006a500 2023-11-28T20:54:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.486+0000 7fa5e64e0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e015d220 con 0x7fa5e015e840 2023-11-28T20:54:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5e4a7d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 0x7fa5e015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5e4a7d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 0x7fa5e015ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40340/0 (socket says 172.21.15.45:40340) 2023-11-28T20:54:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5e4a7d640 1 -- 172.21.15.45:0/3375425196 learned_addr learned my addr 172.21.15.45:0/3375425196 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:15.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5d7fff640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 0x7fa5e015a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5dffff640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5e006a500 0x7fa5e015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5e4a7d640 1 -- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 msgr2=0x7fa5e015a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5e4a7d640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 0x7fa5e015a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5e4a7d640 1 -- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5e006a500 msgr2=0x7fa5e015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5e4a7d640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5e006a500 0x7fa5e015c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.487+0000 7fa5e4a7d640 1 -- 172.21.15.45:0/3375425196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c80094d0 con 0x7fa5e015f720 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.488+0000 7fa5e4a7d640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 0x7fa5e015ac90 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800c730 tx=0x7fa5d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.488+0000 7fa5dd7fa640 1 -- 172.21.15.45:0/3375425196 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8003b70 con 0x7fa5e015f720 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.488+0000 7fa5dd7fa640 1 -- 172.21.15.45:0/3375425196 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d8003cf0 con 0x7fa5e015f720 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.488+0000 7fa5dd7fa640 1 -- 172.21.15.45:0/3375425196 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8005310 con 0x7fa5e015f720 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.488+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e015d500 con 0x7fa5e015f720 2023-11-28T20:54:15.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.488+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e015ce80 con 0x7fa5e015f720 2023-11-28T20:54:15.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.489+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e00b29f0 con 0x7fa5e015f720 2023-11-28T20:54:15.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.490+0000 7fa5dd7fa640 1 -- 172.21.15.45:0/3375425196 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa5d8005680 con 0x7fa5e015f720 2023-11-28T20:54:15.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.491+0000 7fa5dd7fa640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa5c006af60 0x7fa5c006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.491+0000 7fa5dd7fa640 1 -- 172.21.15.45:0/3375425196 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fa5d8089910 con 0x7fa5e015f720 2023-11-28T20:54:15.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.492+0000 7fa5dffff640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa5c006af60 0x7fa5c006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.493+0000 7fa5dffff640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa5c006af60 0x7fa5c006d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0006fd0 tx=0x7fa5d0008040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.496+0000 7fa5dd7fa640 1 -- 172.21.15.45:0/3375425196 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5e00b29f0 con 0x7fa5e015f720 2023-11-28T20:54:15.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.699+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa5e00b2d90 con 0x7fa5e015f720 2023-11-28T20:54:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.705+0000 7fa5dd7fa640 1 -- 172.21.15.45:0/3375425196 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa5d8003e60 con 0x7fa5e015f720 2023-11-28T20:54:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.707+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa5c006af60 msgr2=0x7fa5c006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.707+0000 7fa5e64e0640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa5c006af60 0x7fa5c006d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0006fd0 tx=0x7fa5d0008040 comp rx=0 tx=0).stop 2023-11-28T20:54:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.707+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 msgr2=0x7fa5e015ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.707+0000 7fa5e64e0640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 0x7fa5e015ac90 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800c730 tx=0x7fa5d800cbf0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.708+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 shutdown_connections 2023-11-28T20:54:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.708+0000 7fa5e64e0640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa5c006af60 0x7fa5c006d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.708+0000 7fa5e64e0640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5e015f720 0x7fa5e015ac90 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.708+0000 7fa5e64e0640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5e015e840 0x7fa5e015a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.708+0000 7fa5e64e0640 1 --2- 172.21.15.45:0/3375425196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5e006a500 0x7fa5e015c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.708+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 >> 172.21.15.45:0/3375425196 conn(0x7fa5e00ced60 msgr2=0x7fa5e00701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.708+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 shutdown_connections 2023-11-28T20:54:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.708+0000 7fa5e64e0640 1 -- 172.21.15.45:0/3375425196 wait complete. 2023-11-28T20:54:15.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T20:54:15.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.954+0000 7fab75f61640 1 Processor -- start 2023-11-28T20:54:15.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.954+0000 7fab75f61640 1 -- start start 2023-11-28T20:54:15.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.955+0000 7fab75f61640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fab7015b420 0x7fab7006e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.955+0000 7fab75f61640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab7006f510 0x7fab700ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.955+0000 7fab75f61640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7015b800 con 0x7fab700c3690 2023-11-28T20:54:15.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab75f61640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7006a500 con 0x7fab7006f510 2023-11-28T20:54:15.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab75f61640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7006b040 con 0x7fab7015b420 2023-11-28T20:54:15.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab6ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab7006f510 0x7fab700ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab6effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fab700c3690 0x7fab7006b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49242/0 (socket says 172.21.15.45:49242) 2023-11-28T20:54:15.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab6effd640 1 -- 172.21.15.45:0/2919698115 learned_addr learned my addr 172.21.15.45:0/2919698115 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:15.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab6f7fe640 1 --2- 172.21.15.45:0/2919698115 >> v2:172.21.15.190:3300/0 conn(0x7fab7015b420 0x7fab7006e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab6ffff640 1 -- 172.21.15.45:0/2919698115 >> v2:172.21.15.190:3300/0 conn(0x7fab7015b420 msgr2=0x7fab7006e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:15.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab6ffff640 1 --2- 172.21.15.45:0/2919698115 >> v2:172.21.15.190:3300/0 conn(0x7fab7015b420 0x7fab7006e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab6ffff640 1 -- 172.21.15.45:0/2919698115 >> v1:172.21.15.45:6789/0 conn(0x7fab700c3690 legacy=0x7fab7006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.956+0000 7fab6ffff640 1 -- 172.21.15.45:0/2919698115 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7006a710 con 0x7fab7006f510 2023-11-28T20:54:15.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.958+0000 7fab6ffff640 1 --2- 172.21.15.45:0/2919698115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7006f510 0x7fab700ca5c0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fab700cfe10 tx=0x7fab6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6995fd8bc61b615f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:15.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.958+0000 7fab6e7fc640 1 -- 172.21.15.45:0/2919698115 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68031830 con 0x7fab7006f510 2023-11-28T20:54:15.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.958+0000 7fab6e7fc640 1 -- 172.21.15.45:0/2919698115 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab68031990 con 0x7fab7006f510 2023-11-28T20:54:15.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.958+0000 7fab6e7fc640 1 -- 172.21.15.45:0/2919698115 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68031d20 con 0x7fab7006f510 2023-11-28T20:54:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.958+0000 7fab75f61640 1 -- 172.21.15.45:0/2919698115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7006f510 msgr2=0x7fab700ca5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.958+0000 7fab75f61640 1 --2- 172.21.15.45:0/2919698115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7006f510 0x7fab700ca5c0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fab700cfe10 tx=0x7fab6802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.959+0000 7fab75f61640 1 -- 172.21.15.45:0/2919698115 shutdown_connections 2023-11-28T20:54:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.959+0000 7fab75f61640 1 --2- 172.21.15.45:0/2919698115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7006f510 0x7fab700ca5c0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.959+0000 7fab75f61640 1 --2- 172.21.15.45:0/2919698115 >> v2:172.21.15.190:3300/0 conn(0x7fab7015b420 0x7fab7006e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.959+0000 7fab75f61640 1 -- 172.21.15.45:0/2919698115 >> 172.21.15.45:0/2919698115 conn(0x7fab700cc9f0 msgr2=0x7fab700ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.959+0000 7fab75f61640 1 -- 172.21.15.45:0/2919698115 shutdown_connections 2023-11-28T20:54:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.959+0000 7fab75f61640 1 -- 172.21.15.45:0/2919698115 wait complete. 2023-11-28T20:54:15.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.961+0000 7fab75f61640 1 Processor -- start 2023-11-28T20:54:15.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.961+0000 7fab75f61640 1 -- start start 2023-11-28T20:54:15.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab75f61640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7006f510 0x7fab70066790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab75f61640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700c3690 0x7fab70155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab75f61640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7015b420 0x7fab700b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab75f61640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700cc080 con 0x7fab700c3690 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab75f61640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700c44b0 con 0x7fab7006f510 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab75f61640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab700cbda0 con 0x7fab7015b420 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700c3690 0x7fab70155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700c3690 0x7fab70155000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40354/0 (socket says 172.21.15.45:40354) 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6effd640 1 -- 172.21.15.45:0/3880762843 learned_addr learned my addr 172.21.15.45:0/3880762843 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6f7fe640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7006f510 0x7fab70066790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6ffff640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7015b420 0x7fab700b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6effd640 1 -- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7015b420 msgr2=0x7fab700b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6effd640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7015b420 0x7fab700b4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6effd640 1 -- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7006f510 msgr2=0x7fab70066790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:15.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6effd640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7006f510 0x7fab70066790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:15.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.962+0000 7fab6effd640 1 -- 172.21.15.45:0/3880762843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68009400 con 0x7fab700c3690 2023-11-28T20:54:15.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.963+0000 7fab6effd640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700c3690 0x7fab70155000 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fab5800c500 tx=0x7fab5800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.963+0000 7fab67fff640 1 -- 172.21.15.45:0/3880762843 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58005ce0 con 0x7fab700c3690 2023-11-28T20:54:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.963+0000 7fab67fff640 1 -- 172.21.15.45:0/3880762843 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab58005e40 con 0x7fab700c3690 2023-11-28T20:54:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.963+0000 7fab67fff640 1 -- 172.21.15.45:0/3880762843 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab580072f0 con 0x7fab700c3690 2023-11-28T20:54:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.963+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab700c91a0 con 0x7fab700c3690 2023-11-28T20:54:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.963+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab70155540 con 0x7fab700c3690 2023-11-28T20:54:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.965+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab700733d0 con 0x7fab700c3690 2023-11-28T20:54:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.965+0000 7fab67fff640 1 -- 172.21.15.45:0/3880762843 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fab58007510 con 0x7fab700c3690 2023-11-28T20:54:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.966+0000 7fab67fff640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fab5006af60 0x7fab5006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.966+0000 7fab67fff640 1 -- 172.21.15.45:0/3880762843 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fab58088bd0 con 0x7fab700c3690 2023-11-28T20:54:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.967+0000 7fab6f7fe640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fab5006af60 0x7fab5006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.968+0000 7fab6f7fe640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fab5006af60 0x7fab5006d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab60009a60 tx=0x7fab60003990 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:15.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:15.971+0000 7fab67fff640 1 -- 172.21.15.45:0/3880762843 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab58053aa0 con 0x7fab700c3690 2023-11-28T20:54:16.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.171+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fab7015b800 con 0x7fab700c3690 2023-11-28T20:54:16.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.179+0000 7fab67fff640 1 -- 172.21.15.45:0/3880762843 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v45) v1 ==== 115+0+0 (secure 0 0 0) 0x7fab7015b800 con 0x7fab700c3690 2023-11-28T20:54:16.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fab5006af60 msgr2=0x7fab5006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:16.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fab5006af60 0x7fab5006d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab60009a60 tx=0x7fab60003990 comp rx=0 tx=0).stop 2023-11-28T20:54:16.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700c3690 msgr2=0x7fab70155000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:16.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700c3690 0x7fab70155000 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fab5800c500 tx=0x7fab5800c9c0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 shutdown_connections 2023-11-28T20:54:16.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fab5006af60 0x7fab5006d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7015b420 0x7fab700b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700c3690 0x7fab70155000 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 --2- 172.21.15.45:0/3880762843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7006f510 0x7fab70066790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 >> 172.21.15.45:0/3880762843 conn(0x7fab700cc9f0 msgr2=0x7fab70155f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:16.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 shutdown_connections 2023-11-28T20:54:16.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.184+0000 7fab75f61640 1 -- 172.21.15.45:0/3880762843 wait complete. 2023-11-28T20:54:16.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-11-28T20:54:16.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.425+0000 7f98c0839640 1 Processor -- start 2023-11-28T20:54:16.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.426+0000 7f98c0839640 1 -- start start 2023-11-28T20:54:16.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.427+0000 7f98c0839640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f98bc0b3870 0x7f98bc15db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:16.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.427+0000 7f98c0839640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98bc06d880 0x7f98bc15e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:16.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98c0839640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc15e680 con 0x7f98bc0721e0 2023-11-28T20:54:16.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98c0839640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98bc15a460 con 0x7f98bc0b3870 2023-11-28T20:54:16.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98c0839640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc15e960 con 0x7f98bc06d880 2023-11-28T20:54:16.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98bb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f98bc0b3870 0x7f98bc15db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:16.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98bb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f98bc0b3870 0x7f98bc15db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40358/0 (socket says 172.21.15.45:40358) 2023-11-28T20:54:16.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98bb7fe640 1 -- 172.21.15.45:0/1811227180 learned_addr learned my addr 172.21.15.45:0/1811227180 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:16.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98bb7fe640 1 -- 172.21.15.45:0/1811227180 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98bc06d880 msgr2=0x7f98bc15e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:16.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98bb7fe640 1 --2- 172.21.15.45:0/1811227180 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98bc06d880 0x7f98bc15e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98bb7fe640 1 -- 172.21.15.45:0/1811227180 >> v1:172.21.15.190:6789/0 conn(0x7f98bc0721e0 legacy=0x7f98bc06d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:16.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.428+0000 7f98bb7fe640 1 -- 172.21.15.45:0/1811227180 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc071700 con 0x7f98bc0b3870 2023-11-28T20:54:16.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.430+0000 7f98bb7fe640 1 --2- 172.21.15.45:0/1811227180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0b3870 0x7f98bc15db10 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009870 tx=0x7f98ac02db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8665afa2e284cd8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:16.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.430+0000 7f98ba7fc640 1 -- 172.21.15.45:0/1811227180 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac02f8c0 con 0x7f98bc0b3870 2023-11-28T20:54:16.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.430+0000 7f98ba7fc640 1 -- 172.21.15.45:0/1811227180 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac02fa20 con 0x7f98bc0b3870 2023-11-28T20:54:16.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.430+0000 7f98ba7fc640 1 -- 172.21.15.45:0/1811227180 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac02fdb0 con 0x7f98bc0b3870 2023-11-28T20:54:16.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.431+0000 7f98c0839640 1 -- 172.21.15.45:0/1811227180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0b3870 msgr2=0x7f98bc15db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:16.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.431+0000 7f98c0839640 1 --2- 172.21.15.45:0/1811227180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0b3870 0x7f98bc15db10 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009870 tx=0x7f98ac02db00 comp rx=0 tx=0).stop 2023-11-28T20:54:16.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.431+0000 7f98c0839640 1 -- 172.21.15.45:0/1811227180 shutdown_connections 2023-11-28T20:54:16.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.431+0000 7f98c0839640 1 --2- 172.21.15.45:0/1811227180 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98bc06d880 0x7f98bc15e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.431+0000 7f98c0839640 1 --2- 172.21.15.45:0/1811227180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0b3870 0x7f98bc15db10 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.431+0000 7f98c0839640 1 -- 172.21.15.45:0/1811227180 >> 172.21.15.45:0/1811227180 conn(0x7f98bc06ab30 msgr2=0x7f98bc0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:16.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.431+0000 7f98c0839640 1 -- 172.21.15.45:0/1811227180 shutdown_connections 2023-11-28T20:54:16.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.431+0000 7f98c0839640 1 -- 172.21.15.45:0/1811227180 wait complete. 2023-11-28T20:54:16.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.433+0000 7f98c0839640 1 Processor -- start 2023-11-28T20:54:16.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.433+0000 7f98c0839640 1 -- start start 2023-11-28T20:54:16.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.434+0000 7f98c0839640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc06d880 0x7f98bc1e42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:16.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.434+0000 7f98c0839640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0721e0 0x7f98bc1e5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:16.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.434+0000 7f98c0839640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98bc0b3870 0x7f98bc09bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:16.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.434+0000 7f98c0839640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc1e4c40 con 0x7f98bc0721e0 2023-11-28T20:54:16.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.434+0000 7f98c0839640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc1e5140 con 0x7f98bc06d880 2023-11-28T20:54:16.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.434+0000 7f98c0839640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc1e5420 con 0x7f98bc0b3870 2023-11-28T20:54:16.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.434+0000 7f98baffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0721e0 0x7f98bc1e5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:16.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.434+0000 7f98baffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0721e0 0x7f98bc1e5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40360/0 (socket says 172.21.15.45:40360) 2023-11-28T20:54:16.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.434+0000 7f98baffd640 1 -- 172.21.15.45:0/893636767 learned_addr learned my addr 172.21.15.45:0/893636767 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:16.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98baffd640 1 -- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98bc0b3870 msgr2=0x7f98bc09bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:16.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98baffd640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98bc0b3870 0x7f98bc09bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98baffd640 1 -- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc06d880 msgr2=0x7f98bc1e42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:16.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98baffd640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc06d880 0x7f98bc1e42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98baffd640 1 -- 172.21.15.45:0/893636767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac0094d0 con 0x7f98bc0721e0 2023-11-28T20:54:16.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98baffd640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0721e0 0x7f98bc1e5f00 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f98a400c7a0 tx=0x7f98a400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:16.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98b3fff640 1 -- 172.21.15.45:0/893636767 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4003d80 con 0x7f98bc0721e0 2023-11-28T20:54:16.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98b3fff640 1 -- 172.21.15.45:0/893636767 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a4005070 con 0x7f98bc0721e0 2023-11-28T20:54:16.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98b3fff640 1 -- 172.21.15.45:0/893636767 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4005460 con 0x7f98bc0721e0 2023-11-28T20:54:16.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.435+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98bc1e5700 con 0x7f98bc0721e0 2023-11-28T20:54:16.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.436+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc1e4820 con 0x7f98bc0721e0 2023-11-28T20:54:16.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.437+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98bc1e5c20 con 0x7f98bc0721e0 2023-11-28T20:54:16.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.438+0000 7f98b3fff640 1 -- 172.21.15.45:0/893636767 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f98a40055c0 con 0x7f98bc0721e0 2023-11-28T20:54:16.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.439+0000 7f98b3fff640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f989c06af60 0x7f989c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:16.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.439+0000 7f98b3fff640 1 -- 172.21.15.45:0/893636767 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f98a4088d10 con 0x7f98bc0721e0 2023-11-28T20:54:16.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.440+0000 7f98bb7fe640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f989c06af60 0x7f989c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:16.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.441+0000 7f98bb7fe640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f989c06af60 0x7f989c06d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f98ac038000 tx=0x7f98ac032000 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:16.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.444+0000 7f98b3fff640 1 -- 172.21.15.45:0/893636767 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98bc1e5c20 con 0x7f98bc0721e0 2023-11-28T20:54:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.714+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f98bc1e5c20 con 0x7f98bc0721e0 2023-11-28T20:54:16.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.774+0000 7f98b3fff640 1 -- 172.21.15.45:0/893636767 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v786) v1 ==== 148+0+0 (secure 0 0 0) 0x7f98bc1e5c20 con 0x7f98bc0721e0 2023-11-28T20:54:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f989c06af60 msgr2=0x7f989c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f989c06af60 0x7f989c06d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f98ac038000 tx=0x7f98ac032000 comp rx=0 tx=0).stop 2023-11-28T20:54:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0721e0 msgr2=0x7f98bc1e5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0721e0 0x7f98bc1e5f00 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f98a400c7a0 tx=0x7f98a400cc60 comp rx=0 tx=0).stop 2023-11-28T20:54:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 shutdown_connections 2023-11-28T20:54:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f989c06af60 0x7f989c06d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98bc0b3870 0x7f98bc09bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98bc0721e0 0x7f98bc1e5f00 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 --2- 172.21.15.45:0/893636767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc06d880 0x7f98bc1e42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 >> 172.21.15.45:0/893636767 conn(0x7f98bc06ab30 msgr2=0x7f98bc06c320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:16.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 shutdown_connections 2023-11-28T20:54:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:16.778+0000 7f98c0839640 1 -- 172.21.15.45:0/893636767 wait complete. 2023-11-28T20:54:16.801 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-11-28T20:54:16.802 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T20:54:16.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:17.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.031+0000 7f8f1cfac640 1 Processor -- start 2023-11-28T20:54:17.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.031+0000 7f8f1cfac640 1 -- start start 2023-11-28T20:54:17.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.033+0000 7f8f1cfac640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8f18155780 0x7f8f180abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.033+0000 7f8f1cfac640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8f180ae540 0x7f8f180ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.033+0000 7f8f1cfac640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1815c3c0 con 0x7f8f180ce630 2023-11-28T20:54:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.033+0000 7f8f1cfac640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f180ceac0 con 0x7f8f18155780 2023-11-28T20:54:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.033+0000 7f8f1cfac640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f18155b60 con 0x7f8f180ae540 2023-11-28T20:54:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.033+0000 7f8f177fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8f18155780 0x7f8f180abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.034+0000 7f8f177fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8f18155780 0x7f8f180abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40372/0 (socket says 172.21.15.45:40372) 2023-11-28T20:54:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.034+0000 7f8f177fe640 1 -- 172.21.15.45:0/490225307 learned_addr learned my addr 172.21.15.45:0/490225307 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.034+0000 7f8f16ffd640 1 --2- 172.21.15.45:0/490225307 >> v2:172.21.15.190:3300/0 conn(0x7f8f180ae540 0x7f8f180ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.034+0000 7f8f177fe640 1 -- 172.21.15.45:0/490225307 >> v2:172.21.15.190:3300/0 conn(0x7f8f180ae540 msgr2=0x7f8f180ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.034+0000 7f8f177fe640 1 --2- 172.21.15.45:0/490225307 >> v2:172.21.15.190:3300/0 conn(0x7f8f180ae540 0x7f8f180ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.034+0000 7f8f177fe640 1 -- 172.21.15.45:0/490225307 >> v1:172.21.15.190:6789/0 conn(0x7f8f180ce630 legacy=0x7f8f1815ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.034+0000 7f8f177fe640 1 -- 172.21.15.45:0/490225307 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1815ba90 con 0x7f8f18155780 2023-11-28T20:54:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.036+0000 7f8f177fe640 1 --2- 172.21.15.45:0/490225307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 0x7f8f180abd70 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f8f180cfec0 tx=0x7f8f0802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624b617f7cb01a6d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.036+0000 7f8f167fc640 1 -- 172.21.15.45:0/490225307 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0802ca80 con 0x7f8f18155780 2023-11-28T20:54:17.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.036+0000 7f8f167fc640 1 -- 172.21.15.45:0/490225307 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0802cbe0 con 0x7f8f18155780 2023-11-28T20:54:17.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.037+0000 7f8f167fc640 1 -- 172.21.15.45:0/490225307 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f080343f0 con 0x7f8f18155780 2023-11-28T20:54:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.037+0000 7f8f1cfac640 1 -- 172.21.15.45:0/490225307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 msgr2=0x7f8f180abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.037+0000 7f8f1cfac640 1 --2- 172.21.15.45:0/490225307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 0x7f8f180abd70 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f8f180cfec0 tx=0x7f8f0802bee0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.037+0000 7f8f1cfac640 1 -- 172.21.15.45:0/490225307 shutdown_connections 2023-11-28T20:54:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.037+0000 7f8f1cfac640 1 --2- 172.21.15.45:0/490225307 >> v2:172.21.15.190:3300/0 conn(0x7f8f180ae540 0x7f8f180ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.037+0000 7f8f1cfac640 1 --2- 172.21.15.45:0/490225307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 0x7f8f180abd70 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.037+0000 7f8f1cfac640 1 -- 172.21.15.45:0/490225307 >> 172.21.15.45:0/490225307 conn(0x7f8f1806ab30 msgr2=0x7f8f180b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.037+0000 7f8f1cfac640 1 -- 172.21.15.45:0/490225307 shutdown_connections 2023-11-28T20:54:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.037+0000 7f8f1cfac640 1 -- 172.21.15.45:0/490225307 wait complete. 2023-11-28T20:54:17.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.039+0000 7f8f1cfac640 1 Processor -- start 2023-11-28T20:54:17.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.040+0000 7f8f1cfac640 1 -- start start 2023-11-28T20:54:17.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f1cfac640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f180ae540 0x7f8f1814d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f1cfac640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f180ce630 0x7f8f1813f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f1cfac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 0x7f8f1809c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f1cfac640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1809bc30 con 0x7f8f18155780 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f1cfac640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1809acd0 con 0x7f8f180ce630 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f1cfac640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1809afb0 con 0x7f8f180ae540 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f17fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 0x7f8f1809c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f17fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 0x7f8f1809c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40388/0 (socket says 172.21.15.45:40388) 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f17fff640 1 -- 172.21.15.45:0/474647336 learned_addr learned my addr 172.21.15.45:0/474647336 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f177fe640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f180ae540 0x7f8f1814d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.041+0000 7f8f17fff640 1 -- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f180ae540 msgr2=0x7f8f1814d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f17fff640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f180ae540 0x7f8f1814d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f17fff640 1 -- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f180ce630 msgr2=0x7f8f1813f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f16ffd640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f180ce630 0x7f8f1813f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f17fff640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f180ce630 0x7f8f1813f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f17fff640 1 -- 172.21.15.45:0/474647336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f08009b30 con 0x7f8f18155780 2023-11-28T20:54:17.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f17fff640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 0x7f8f1809c050 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000c410 tx=0x7f8f1000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:17.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f0ffff640 1 -- 172.21.15.45:0/474647336 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10011790 con 0x7f8f18155780 2023-11-28T20:54:17.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f0ffff640 1 -- 172.21.15.45:0/474647336 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f100118f0 con 0x7f8f18155780 2023-11-28T20:54:17.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f0ffff640 1 -- 172.21.15.45:0/474647336 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10011ca0 con 0x7f8f18155780 2023-11-28T20:54:17.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1809b290 con 0x7f8f18155780 2023-11-28T20:54:17.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.042+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1809b850 con 0x7f8f18155780 2023-11-28T20:54:17.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.043+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1809bd70 con 0x7f8f18155780 2023-11-28T20:54:17.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.044+0000 7f8f0ffff640 1 -- 172.21.15.45:0/474647336 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8f1001c460 con 0x7f8f18155780 2023-11-28T20:54:17.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.045+0000 7f8f0ffff640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8ef806af60 0x7f8ef806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.045+0000 7f8f0ffff640 1 -- 172.21.15.45:0/474647336 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f8f100896c0 con 0x7f8f18155780 2023-11-28T20:54:17.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.046+0000 7f8f177fe640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8ef806af60 0x7f8ef806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.047+0000 7f8f177fe640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8ef806af60 0x7f8ef806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8f08035000 tx=0x7f8f08030000 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:17.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.050+0000 7f8f0ffff640 1 -- 172.21.15.45:0/474647336 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f1809bd70 con 0x7f8f18155780 2023-11-28T20:54:17.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.246+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f1809bd70 con 0x7f8f18155780 2023-11-28T20:54:17.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.247+0000 7f8f0ffff640 1 -- 172.21.15.45:0/474647336 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+14373 (secure 0 0 0) 0x7f8f1809bd70 con 0x7f8f18155780 2023-11-28T20:54:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:17.250 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":139,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:14.756365+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6834/2464465186":"2023-11-29T20:54:11.737488+0000","172.21.15.45:6832/2464465186":"2023-11-29T20:54:11.737488+0000","172.21.15.190:0/3643991290":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/304862452":"2023-11-29T20:54:07.580192+0000","172.21.15.190:6833/3157274352":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/2315351474":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/259384681":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/3489922358":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/665095987":"2023-11-29T20:54:07.580192+0000","172.21.15.190:6832/3157274352":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/2592164098":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/800294716":"2023-11-29T20:54:07.580192+0000","172.21.15.190:0/3135688549":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/369403092":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6833/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:6832/3959324769":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1007604882":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/685209934":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1745589824":"2023-11-29T20:53:49.553172+0000","172.21.15.190:0/1706225465":"2023-11-29T20:53:49.553172+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:17.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8ef806af60 msgr2=0x7f8ef806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8ef806af60 0x7f8ef806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8f08035000 tx=0x7f8f08030000 comp rx=0 tx=0).stop 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 msgr2=0x7f8f1809c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 0x7f8f1809c050 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000c410 tx=0x7f8f1000c8d0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 shutdown_connections 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f8ef806af60 0x7f8ef806d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f18155780 0x7f8f1809c050 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f180ce630 0x7f8f1813f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 --2- 172.21.15.45:0/474647336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f180ae540 0x7f8f1814d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 >> 172.21.15.45:0/474647336 conn(0x7f8f1806ab30 msgr2=0x7f8f1806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 shutdown_connections 2023-11-28T20:54:17.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.250+0000 7f8f1cfac640 1 -- 172.21.15.45:0/474647336 wait complete. 2023-11-28T20:54:17.273 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:54:17.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:54:17.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.504+0000 7fe3520c2640 1 Processor -- start 2023-11-28T20:54:17.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.504+0000 7fe3520c2640 1 -- start start 2023-11-28T20:54:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.505+0000 7fe3520c2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe34c06e8e0 0x7fe34c0af070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.505+0000 7fe3520c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c0acd80 0x7fe34c15dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.505+0000 7fe3520c2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe34c06b510 0x7fe34c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.505+0000 7fe3520c2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34c06bee0 con 0x7fe34c06b510 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.505+0000 7fe3520c2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34c15db60 con 0x7fe34c06e8e0 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.505+0000 7fe3520c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c06ecc0 con 0x7fe34c0acd80 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3518c1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe34c06b510 0x7fe34c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3518c1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe34c06b510 0x7fe34c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40396/0 (socket says 172.21.15.45:40396) 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3518c1640 1 -- 172.21.15.45:0/2189211490 learned_addr learned my addr 172.21.15.45:0/2189211490 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3510c0640 1 --2- 172.21.15.45:0/2189211490 >> v2:172.21.15.190:3300/0 conn(0x7fe34c06e8e0 0x7fe34c0af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3518c1640 1 -- 172.21.15.45:0/2189211490 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c0acd80 msgr2=0x7fe34c15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3508bf640 1 --2- 172.21.15.45:0/2189211490 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c0acd80 0x7fe34c15dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3518c1640 1 --2- 172.21.15.45:0/2189211490 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c0acd80 0x7fe34c15dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3518c1640 1 -- 172.21.15.45:0/2189211490 >> v2:172.21.15.190:3300/0 conn(0x7fe34c06e8e0 msgr2=0x7fe34c0af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:17.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3518c1640 1 --2- 172.21.15.45:0/2189211490 >> v2:172.21.15.190:3300/0 conn(0x7fe34c06e8e0 0x7fe34c0af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3518c1640 1 -- 172.21.15.45:0/2189211490 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c0ad4f0 con 0x7fe34c06b510 2023-11-28T20:54:17.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.506+0000 7fe3508bf640 1 --2- 172.21.15.45:0/2189211490 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c0acd80 0x7fe34c15dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:17.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.507+0000 7fe3518c1640 1 --2- 172.21.15.45:0/2189211490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06b510 0x7fe34c06b9a0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe34c067280 tx=0x7fe34402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b920aa2f3b277e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:17.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe34bfff640 1 -- 172.21.15.45:0/2189211490 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe344031870 con 0x7fe34c06b510 2023-11-28T20:54:17.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe34bfff640 1 -- 172.21.15.45:0/2189211490 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3440319d0 con 0x7fe34c06b510 2023-11-28T20:54:17.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe34bfff640 1 -- 172.21.15.45:0/2189211490 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe344031d60 con 0x7fe34c06b510 2023-11-28T20:54:17.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe3520c2640 1 -- 172.21.15.45:0/2189211490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06b510 msgr2=0x7fe34c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2189211490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06b510 0x7fe34c06b9a0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe34c067280 tx=0x7fe34402b910 comp rx=0 tx=0).stop 2023-11-28T20:54:17.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe3520c2640 1 -- 172.21.15.45:0/2189211490 shutdown_connections 2023-11-28T20:54:17.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2189211490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06b510 0x7fe34c06b9a0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2189211490 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c0acd80 0x7fe34c15dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2189211490 >> v2:172.21.15.190:3300/0 conn(0x7fe34c06e8e0 0x7fe34c0af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.508+0000 7fe3520c2640 1 -- 172.21.15.45:0/2189211490 >> 172.21.15.45:0/2189211490 conn(0x7fe34c071240 msgr2=0x7fe34c071650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:17.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.509+0000 7fe3520c2640 1 -- 172.21.15.45:0/2189211490 shutdown_connections 2023-11-28T20:54:17.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.509+0000 7fe3520c2640 1 -- 172.21.15.45:0/2189211490 wait complete. 2023-11-28T20:54:17.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.510+0000 7fe3520c2640 1 Processor -- start 2023-11-28T20:54:17.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.511+0000 7fe3520c2640 1 -- start start 2023-11-28T20:54:17.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.511+0000 7fe3520c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c06b510 0x7fe34c149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.511+0000 7fe3520c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06e8e0 0x7fe34c14b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.511+0000 7fe3520c2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe34c0acd80 0x7fe34c149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3520c2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c149990 con 0x7fe34c06e8e0 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3520c2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c14b060 con 0x7fe34c0acd80 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3520c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c0c8ef0 con 0x7fe34c06b510 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3508bf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06e8e0 0x7fe34c14b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3508bf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06e8e0 0x7fe34c14b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40398/0 (socket says 172.21.15.45:40398) 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3508bf640 1 -- 172.21.15.45:0/2024776914 learned_addr learned my addr 172.21.15.45:0/2024776914 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3510c0640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c06b510 0x7fe34c149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3518c1640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe34c0acd80 0x7fe34c149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3508bf640 1 -- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c06b510 msgr2=0x7fe34c149450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3508bf640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c06b510 0x7fe34c149450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3508bf640 1 -- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe34c0acd80 msgr2=0x7fe34c149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3508bf640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe34c0acd80 0x7fe34c149c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3508bf640 1 -- 172.21.15.45:0/2024776914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440094d0 con 0x7fe34c06e8e0 2023-11-28T20:54:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.512+0000 7fe3518c1640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe34c0acd80 0x7fe34c149c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.513+0000 7fe3508bf640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06e8e0 0x7fe34c14b4f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fe33800c7f0 tx=0x7fe33800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.513+0000 7fe349ffb640 1 -- 172.21.15.45:0/2024776914 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338007820 con 0x7fe34c06e8e0 2023-11-28T20:54:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.513+0000 7fe349ffb640 1 -- 172.21.15.45:0/2024776914 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe338007980 con 0x7fe34c06e8e0 2023-11-28T20:54:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.513+0000 7fe349ffb640 1 -- 172.21.15.45:0/2024776914 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338007cf0 con 0x7fe34c06e8e0 2023-11-28T20:54:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.513+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c14ba30 con 0x7fe34c06e8e0 2023-11-28T20:54:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.513+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c14a730 con 0x7fe34c06e8e0 2023-11-28T20:54:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.514+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c09e100 con 0x7fe34c06e8e0 2023-11-28T20:54:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.515+0000 7fe349ffb640 1 -- 172.21.15.45:0/2024776914 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe3380113f0 con 0x7fe34c06e8e0 2023-11-28T20:54:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.517+0000 7fe349ffb640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe33006afb0 0x7fe33006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.517+0000 7fe349ffb640 1 -- 172.21.15.45:0/2024776914 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fe338089c90 con 0x7fe34c06e8e0 2023-11-28T20:54:17.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.518+0000 7fe3510c0640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe33006afb0 0x7fe33006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.519+0000 7fe3510c0640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe33006afb0 0x7fe33006d470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe340009a60 tx=0x7fe340003990 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.522+0000 7fe349ffb640 1 -- 172.21.15.45:0/2024776914 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe338054a20 con 0x7fe34c06e8e0 2023-11-28T20:54:17.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.718+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe34c0ad200 con 0x7fe34c06e8e0 2023-11-28T20:54:17.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.719+0000 7fe349ffb640 1 -- 172.21.15.45:0/2024776914 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v139) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe33801f0a0 con 0x7fe34c06e8e0 2023-11-28T20:54:17.722 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:54:17.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.722+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe33006afb0 msgr2=0x7fe33006d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.722+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe33006afb0 0x7fe33006d470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe340009a60 tx=0x7fe340003990 comp rx=0 tx=0).stop 2023-11-28T20:54:17.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.722+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06e8e0 msgr2=0x7fe34c14b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.722+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06e8e0 0x7fe34c14b4f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fe33800c7f0 tx=0x7fe33800ccb0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.722+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 shutdown_connections 2023-11-28T20:54:17.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.722+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe33006afb0 0x7fe33006d470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.722+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe34c0acd80 0x7fe34c149c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.722+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe34c06e8e0 0x7fe34c14b4f0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.723+0000 7fe3520c2640 1 --2- 172.21.15.45:0/2024776914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe34c06b510 0x7fe34c149450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.723+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 >> 172.21.15.45:0/2024776914 conn(0x7fe34c071240 msgr2=0x7fe34c15c150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:17.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.723+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 shutdown_connections 2023-11-28T20:54:17.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.723+0000 7fe3520c2640 1 -- 172.21.15.45:0/2024776914 wait complete. 2023-11-28T20:54:17.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-11-28T20:54:17.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.968+0000 7f856a561640 1 Processor -- start 2023-11-28T20:54:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.968+0000 7f856a561640 1 -- start start 2023-11-28T20:54:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.970+0000 7f856a561640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f85640b35b0 0x7f856415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.970+0000 7f856a561640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.970+0000 7f856a561640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856415bda0 con 0x7f85640b1fa0 2023-11-28T20:54:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.970+0000 7f856a561640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856415ee40 con 0x7f85640b35b0 2023-11-28T20:54:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.970+0000 7f856a561640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856415c080 con 0x7f856415b850 2023-11-28T20:54:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.970+0000 7f855bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.970+0000 7f855bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:57468/0 (socket says 172.21.15.45:57468) 2023-11-28T20:54:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.970+0000 7f855bfff640 1 -- 172.21.15.45:0/4014011898 learned_addr learned my addr 172.21.15.45:0/4014011898 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.971+0000 7f8563fff640 1 --2- 172.21.15.45:0/4014011898 >> v2:172.21.15.190:3300/0 conn(0x7f85640b35b0 0x7f856415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.971+0000 7f855bfff640 1 -- 172.21.15.45:0/4014011898 >> v2:172.21.15.190:3300/0 conn(0x7f85640b35b0 msgr2=0x7f856415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:17.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.971+0000 7f855bfff640 1 --2- 172.21.15.45:0/4014011898 >> v2:172.21.15.190:3300/0 conn(0x7f85640b35b0 0x7f856415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.971+0000 7f855bfff640 1 -- 172.21.15.45:0/4014011898 >> v1:172.21.15.190:6789/0 conn(0x7f85640b1fa0 legacy=0x7f856415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.971+0000 7f855bfff640 1 -- 172.21.15.45:0/4014011898 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856415c360 con 0x7f856415b850 2023-11-28T20:54:17.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.973+0000 7f855bfff640 1 --2- 172.21.15.45:0/4014011898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640b0c30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f854c0098a0 tx=0x7f854c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58892e0fca3df5e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:17.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.973+0000 7f85637fe640 1 -- 172.21.15.45:0/4014011898 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c030840 con 0x7f856415b850 2023-11-28T20:54:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f85637fe640 1 -- 172.21.15.45:0/4014011898 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c0309a0 con 0x7f856415b850 2023-11-28T20:54:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f85637fe640 1 -- 172.21.15.45:0/4014011898 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c030d30 con 0x7f856415b850 2023-11-28T20:54:17.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f856a561640 1 -- 172.21.15.45:0/4014011898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 msgr2=0x7f85640b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f856a561640 1 --2- 172.21.15.45:0/4014011898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640b0c30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f854c0098a0 tx=0x7f854c02b980 comp rx=0 tx=0).stop 2023-11-28T20:54:17.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f856a561640 1 -- 172.21.15.45:0/4014011898 shutdown_connections 2023-11-28T20:54:17.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f856a561640 1 --2- 172.21.15.45:0/4014011898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640b0c30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f856a561640 1 --2- 172.21.15.45:0/4014011898 >> v2:172.21.15.190:3300/0 conn(0x7f85640b35b0 0x7f856415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f856a561640 1 -- 172.21.15.45:0/4014011898 >> 172.21.15.45:0/4014011898 conn(0x7f856406a500 msgr2=0x7f856406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:17.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f856a561640 1 -- 172.21.15.45:0/4014011898 shutdown_connections 2023-11-28T20:54:17.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.974+0000 7f856a561640 1 -- 172.21.15.45:0/4014011898 wait complete. 2023-11-28T20:54:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.976+0000 7f856a561640 1 Processor -- start 2023-11-28T20:54:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.976+0000 7f856a561640 1 -- start start 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.977+0000 7f856a561640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85640b1fa0 0x7f8564143860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.977+0000 7f856a561640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640b35b0 0x7f85640adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.977+0000 7f856a561640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640a9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.977+0000 7f856a561640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85640c4dc0 con 0x7f85640b35b0 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.977+0000 7f856a561640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8564143610 con 0x7f85640b1fa0 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.977+0000 7f856a561640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8564155660 con 0x7f856415b850 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.977+0000 7f855bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640b35b0 0x7f85640adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f855bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640b35b0 0x7f85640adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40406/0 (socket says 172.21.15.45:40406) 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f855bfff640 1 -- 172.21.15.45:0/319775395 learned_addr learned my addr 172.21.15.45:0/319775395 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f8563fff640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85640b1fa0 0x7f8564143860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f8568afe640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640a9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f855bfff640 1 -- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 msgr2=0x7f85640a9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f855bfff640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640a9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f855bfff640 1 -- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85640b1fa0 msgr2=0x7f8564143860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f855bfff640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85640b1fa0 0x7f8564143860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f855bfff640 1 -- 172.21.15.45:0/319775395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c0094d0 con 0x7f85640b35b0 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.978+0000 7f8563fff640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85640b1fa0 0x7f8564143860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.979+0000 7f855bfff640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640b35b0 0x7f85640adef0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f854c02b9c0 tx=0x7f854c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.979+0000 7f85617fa640 1 -- 172.21.15.45:0/319775395 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c03e070 con 0x7f85640b35b0 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.979+0000 7f85617fa640 1 -- 172.21.15.45:0/319775395 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c039a10 con 0x7f85640b35b0 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.979+0000 7f85617fa640 1 -- 172.21.15.45:0/319775395 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c039de0 con 0x7f85640b35b0 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.979+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8564155920 con 0x7f85640b35b0 2023-11-28T20:54:17.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.979+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85640ae430 con 0x7f85640b35b0 2023-11-28T20:54:17.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.980+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85640c4890 con 0x7f85640b35b0 2023-11-28T20:54:17.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.982+0000 7f85617fa640 1 -- 172.21.15.45:0/319775395 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f854c04b400 con 0x7f85640b35b0 2023-11-28T20:54:17.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.982+0000 7f85617fa640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f854406afb0 0x7f854406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:17.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.982+0000 7f85617fa640 1 -- 172.21.15.45:0/319775395 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f854c0abfa0 con 0x7f85640b35b0 2023-11-28T20:54:17.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.983+0000 7f8563fff640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f854406afb0 0x7f854406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:17.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.984+0000 7f8563fff640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f854406afb0 0x7f854406d470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8554009b90 tx=0x7f8554008040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:17.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:17.987+0000 7f85617fa640 1 -- 172.21.15.45:0/319775395 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f854c076ca0 con 0x7f85640b35b0 2023-11-28T20:54:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.249+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f85640733d0 con 0x7f85640b35b0 2023-11-28T20:54:18.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.790+0000 7f85617fa640 1 -- 172.21.15.45:0/319775395 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v788) v1 ==== 163+0+0 (secure 0 0 0) 0x7f854c034020 con 0x7f85640b35b0 2023-11-28T20:54:18.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f854406afb0 msgr2=0x7f854406d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:18.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f854406afb0 0x7f854406d470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8554009b90 tx=0x7f8554008040 comp rx=0 tx=0).stop 2023-11-28T20:54:18.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640b35b0 msgr2=0x7f85640adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640b35b0 0x7f85640adef0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f854c02b9c0 tx=0x7f854c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:54:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 shutdown_connections 2023-11-28T20:54:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f854406afb0 0x7f854406d470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f856415b850 0x7f85640a9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640b35b0 0x7f85640adef0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 --2- 172.21.15.45:0/319775395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85640b1fa0 0x7f8564143860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 >> 172.21.15.45:0/319775395 conn(0x7f856406a500 msgr2=0x7f85640cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 shutdown_connections 2023-11-28T20:54:18.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:18.794+0000 7f856a561640 1 -- 172.21.15.45:0/319775395 wait complete. 2023-11-28T20:54:18.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T20:54:19.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.047+0000 7f91e5b33640 1 Processor -- start 2023-11-28T20:54:19.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.047+0000 7f91e5b33640 1 -- start start 2023-11-28T20:54:19.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.048+0000 7f91e5b33640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e015b3c0 0x7f91e00cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:19.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.048+0000 7f91e5b33640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00b0570 con 0x7f91e00cfa50 2023-11-28T20:54:19.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.048+0000 7f91e5b33640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00cc800 con 0x7f91e00b36c0 2023-11-28T20:54:19.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.048+0000 7f91e5b33640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00b08c0 con 0x7f91e015b3c0 2023-11-28T20:54:19.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.049+0000 7f91dffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f91e00cfa50 0x7f91e00b0ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49252/0 (socket says 172.21.15.45:49252) 2023-11-28T20:54:19.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.049+0000 7f91dffff640 1 -- 172.21.15.45:0/1507020788 learned_addr learned my addr 172.21.15.45:0/1507020788 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:19.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.049+0000 7f91df7fe640 1 --2- 172.21.15.45:0/1507020788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e015b3c0 0x7f91e00cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:19.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.049+0000 7f91de7fc640 1 -- 172.21.15.45:0/1507020788 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582181473 0 0) 0x7f91e00b0570 con 0x7f91e00cfa50 2023-11-28T20:54:19.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.050+0000 7f91de7fc640 1 -- 172.21.15.45:0/1507020788 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91cc003520 con 0x7f91e00cfa50 2023-11-28T20:54:19.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.050+0000 7f91de7fc640 1 -- 172.21.15.45:0/1507020788 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036282648 0 0) 0x7f91e00cc800 con 0x7f91e00b36c0 2023-11-28T20:54:19.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.050+0000 7f91df7fe640 1 -- 172.21.15.45:0/1507020788 >> v1:172.21.15.190:6789/0 conn(0x7f91e00b36c0 legacy=0x7f91e00cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:19.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.050+0000 7f91df7fe640 1 -- 172.21.15.45:0/1507020788 >> v1:172.21.15.45:6789/0 conn(0x7f91e00cfa50 legacy=0x7f91e00b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:19.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.050+0000 7f91df7fe640 1 -- 172.21.15.45:0/1507020788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e00ccae0 con 0x7f91e015b3c0 2023-11-28T20:54:19.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.051+0000 7f91df7fe640 1 --2- 172.21.15.45:0/1507020788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e015b3c0 0x7f91e00cec80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009dc0 tx=0x7f91d002bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3208684043c7dac4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:19.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.052+0000 7f91de7fc640 1 -- 172.21.15.45:0/1507020788 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d002c9f0 con 0x7f91e015b3c0 2023-11-28T20:54:19.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.052+0000 7f91de7fc640 1 -- 172.21.15.45:0/1507020788 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d002cb50 con 0x7f91e015b3c0 2023-11-28T20:54:19.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.052+0000 7f91de7fc640 1 -- 172.21.15.45:0/1507020788 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0034440 con 0x7f91e015b3c0 2023-11-28T20:54:19.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.052+0000 7f91e5b33640 1 -- 172.21.15.45:0/1507020788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e015b3c0 msgr2=0x7f91e00cec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:19.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.052+0000 7f91e5b33640 1 --2- 172.21.15.45:0/1507020788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e015b3c0 0x7f91e00cec80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009dc0 tx=0x7f91d002bcc0 comp rx=0 tx=0).stop 2023-11-28T20:54:19.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.052+0000 7f91e5b33640 1 -- 172.21.15.45:0/1507020788 shutdown_connections 2023-11-28T20:54:19.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.052+0000 7f91e5b33640 1 --2- 172.21.15.45:0/1507020788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e015b3c0 0x7f91e00cec80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:19.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.052+0000 7f91e5b33640 1 -- 172.21.15.45:0/1507020788 >> 172.21.15.45:0/1507020788 conn(0x7f91e00b2ba0 msgr2=0x7f91e00b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:19.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.052+0000 7f91e5b33640 1 -- 172.21.15.45:0/1507020788 shutdown_connections 2023-11-28T20:54:19.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.053+0000 7f91e5b33640 1 -- 172.21.15.45:0/1507020788 wait complete. 2023-11-28T20:54:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.054+0000 7f91e5b33640 1 Processor -- start 2023-11-28T20:54:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.054+0000 7f91e5b33640 1 -- start start 2023-11-28T20:54:19.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91e5b33640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e00b36c0 0x7f91e0158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91e5b33640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91e00cfa50 0x7f91e006b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91e5b33640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e015b3c0 0x7f91e006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91e5b33640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e015d850 con 0x7f91e015b3c0 2023-11-28T20:54:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91e5b33640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e015cc80 con 0x7f91e00cfa50 2023-11-28T20:54:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91e5b33640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e015cf20 con 0x7f91e00b36c0 2023-11-28T20:54:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e015b3c0 0x7f91e006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e015b3c0 0x7f91e006bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40416/0 (socket says 172.21.15.45:40416) 2023-11-28T20:54:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91dffff640 1 -- 172.21.15.45:0/1202896267 learned_addr learned my addr 172.21.15.45:0/1202896267 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91deffd640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91e00cfa50 0x7f91e006b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91dffff640 1 -- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e00b36c0 msgr2=0x7f91e0158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91dffff640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e00b36c0 0x7f91e0158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91dffff640 1 -- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91e00cfa50 msgr2=0x7f91e006b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91dffff640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91e00cfa50 0x7f91e006b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91dffff640 1 -- 172.21.15.45:0/1202896267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0009a70 con 0x7f91e015b3c0 2023-11-28T20:54:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.056+0000 7f91deffd640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91e00cfa50 0x7f91e006b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.057+0000 7f91dffff640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e015b3c0 0x7f91e006bc50 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f91d800c750 tx=0x7f91d800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:19.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.057+0000 7f91d7fff640 1 -- 172.21.15.45:0/1202896267 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8005a90 con 0x7f91e015b3c0 2023-11-28T20:54:19.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.057+0000 7f91d7fff640 1 -- 172.21.15.45:0/1202896267 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d8005bf0 con 0x7f91e015b3c0 2023-11-28T20:54:19.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.057+0000 7f91d7fff640 1 -- 172.21.15.45:0/1202896267 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8007070 con 0x7f91e015b3c0 2023-11-28T20:54:19.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.057+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e015d200 con 0x7f91e015b3c0 2023-11-28T20:54:19.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.057+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e0158c70 con 0x7f91e015b3c0 2023-11-28T20:54:19.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.058+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e015d9c0 con 0x7f91e015b3c0 2023-11-28T20:54:19.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.059+0000 7f91d7fff640 1 -- 172.21.15.45:0/1202896267 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f91d80071d0 con 0x7f91e015b3c0 2023-11-28T20:54:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.061+0000 7f91d7fff640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f91c006af60 0x7f91c006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.061+0000 7f91d7fff640 1 -- 172.21.15.45:0/1202896267 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f91d80890d0 con 0x7f91e015b3c0 2023-11-28T20:54:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.061+0000 7f91df7fe640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f91c006af60 0x7f91c006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:19.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.063+0000 7f91df7fe640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f91c006af60 0x7f91c006d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f91d0036000 tx=0x7f91d002e000 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:19.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.065+0000 7f91d7fff640 1 -- 172.21.15.45:0/1202896267 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91d8053ef0 con 0x7f91e015b3c0 2023-11-28T20:54:19.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.258+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f91e00733d0 con 0x7f91e015b3c0 2023-11-28T20:54:19.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.263+0000 7f91d7fff640 1 -- 172.21.15.45:0/1202896267 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f91d80170c0 con 0x7f91e015b3c0 2023-11-28T20:54:19.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f91c006af60 msgr2=0x7f91c006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:19.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f91c006af60 0x7f91c006d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f91d0036000 tx=0x7f91d002e000 comp rx=0 tx=0).stop 2023-11-28T20:54:19.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e015b3c0 msgr2=0x7f91e006bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e015b3c0 0x7f91e006bc50 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f91d800c750 tx=0x7f91d800cc10 comp rx=0 tx=0).stop 2023-11-28T20:54:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 shutdown_connections 2023-11-28T20:54:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f91c006af60 0x7f91c006d420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91e015b3c0 0x7f91e006bc50 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f91e00cfa50 0x7f91e006b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 --2- 172.21.15.45:0/1202896267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f91e00b36c0 0x7f91e0158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 >> 172.21.15.45:0/1202896267 conn(0x7f91e00b2ba0 msgr2=0x7f91e00cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.266+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 shutdown_connections 2023-11-28T20:54:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:19.267+0000 7f91e5b33640 1 -- 172.21.15.45:0/1202896267 wait complete. 2023-11-28T20:54:19.289 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:54:19.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:54:19.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:54:19.547 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:54:19.547 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:54:19.547 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:19.611 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:54:19.611 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:54:19.611 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:19.670 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:54:19.670 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:54:19.670 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:19.737 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:54:19.738 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:54:19.738 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:19.800 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:54:19.800 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T20:54:19.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:20.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.037+0000 7f3123925640 1 Processor -- start 2023-11-28T20:54:20.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.038+0000 7f3123925640 1 -- start start 2023-11-28T20:54:20.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.039+0000 7f3123925640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f311c0b35b0 0x7f311c15ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.039+0000 7f3123925640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c15b850 0x7f311c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.039+0000 7f3123925640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c15bda0 con 0x7f311c0b1fa0 2023-11-28T20:54:20.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.039+0000 7f3123925640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c15ee40 con 0x7f311c0b35b0 2023-11-28T20:54:20.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.039+0000 7f3123925640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c15c080 con 0x7f311c15b850 2023-11-28T20:54:20.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.040+0000 7f31216c1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f311c0b35b0 0x7f311c15ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.040+0000 7f31216c1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f311c0b35b0 0x7f311c15ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40428/0 (socket says 172.21.15.45:40428) 2023-11-28T20:54:20.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.040+0000 7f31216c1640 1 -- 172.21.15.45:0/3272661554 learned_addr learned my addr 172.21.15.45:0/3272661554 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:20.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.040+0000 7f3120ec0640 1 --2- 172.21.15.45:0/3272661554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c15b850 0x7f311c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.041+0000 7f31216c1640 1 -- 172.21.15.45:0/3272661554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c15b850 msgr2=0x7f311c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:20.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.041+0000 7f31216c1640 1 --2- 172.21.15.45:0/3272661554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c15b850 0x7f311c0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.041+0000 7f31216c1640 1 -- 172.21.15.45:0/3272661554 >> v1:172.21.15.190:6789/0 conn(0x7f311c0b1fa0 legacy=0x7f311c15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:20.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.041+0000 7f31216c1640 1 -- 172.21.15.45:0/3272661554 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c15c360 con 0x7f311c0b35b0 2023-11-28T20:54:20.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.042+0000 7f31216c1640 1 --2- 172.21.15.45:0/3272661554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b35b0 0x7f311c15ca10 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f31180098a0 tx=0x7f311802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13b26025f8eb526 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:20.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3113fff640 1 -- 172.21.15.45:0/3272661554 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118030840 con 0x7f311c0b35b0 2023-11-28T20:54:20.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3113fff640 1 -- 172.21.15.45:0/3272661554 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31180309a0 con 0x7f311c0b35b0 2023-11-28T20:54:20.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3113fff640 1 -- 172.21.15.45:0/3272661554 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118030d30 con 0x7f311c0b35b0 2023-11-28T20:54:20.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3123925640 1 -- 172.21.15.45:0/3272661554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b35b0 msgr2=0x7f311c15ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:20.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3123925640 1 --2- 172.21.15.45:0/3272661554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b35b0 0x7f311c15ca10 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f31180098a0 tx=0x7f311802b980 comp rx=0 tx=0).stop 2023-11-28T20:54:20.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3123925640 1 -- 172.21.15.45:0/3272661554 shutdown_connections 2023-11-28T20:54:20.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3123925640 1 --2- 172.21.15.45:0/3272661554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c15b850 0x7f311c0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3123925640 1 --2- 172.21.15.45:0/3272661554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b35b0 0x7f311c15ca10 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3123925640 1 -- 172.21.15.45:0/3272661554 >> 172.21.15.45:0/3272661554 conn(0x7f311c06a500 msgr2=0x7f311c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:20.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3123925640 1 -- 172.21.15.45:0/3272661554 shutdown_connections 2023-11-28T20:54:20.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.043+0000 7f3123925640 1 -- 172.21.15.45:0/3272661554 wait complete. 2023-11-28T20:54:20.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.045+0000 7f3123925640 1 Processor -- start 2023-11-28T20:54:20.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.045+0000 7f3123925640 1 -- start start 2023-11-28T20:54:20.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.046+0000 7f3123925640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b1fa0 0x7f311c13a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.046+0000 7f3123925640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c0b35b0 0x7f311c071140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.046+0000 7f3123925640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f311c15b850 0x7f311c071680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.046+0000 7f3123925640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0acbb0 con 0x7f311c0b1fa0 2023-11-28T20:54:20.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.046+0000 7f3123925640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c13ad40 con 0x7f311c15b850 2023-11-28T20:54:20.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.046+0000 7f3123925640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c157f20 con 0x7f311c0b35b0 2023-11-28T20:54:20.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f31216c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b1fa0 0x7f311c13a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f31216c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b1fa0 0x7f311c13a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40430/0 (socket says 172.21.15.45:40430) 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f31216c1640 1 -- 172.21.15.45:0/788112803 learned_addr learned my addr 172.21.15.45:0/788112803 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f3121ec2640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f311c15b850 0x7f311c071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f3120ec0640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c0b35b0 0x7f311c071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f31216c1640 1 -- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c0b35b0 msgr2=0x7f311c071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f31216c1640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c0b35b0 0x7f311c071140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f31216c1640 1 -- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f311c15b850 msgr2=0x7f311c071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f31216c1640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f311c15b850 0x7f311c071680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.047+0000 7f31216c1640 1 -- 172.21.15.45:0/788112803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180094d0 con 0x7f311c0b1fa0 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.048+0000 7f31216c1640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b1fa0 0x7f311c13a800 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f3118000c00 tx=0x7f3118004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.048+0000 7f3111ffb640 1 -- 172.21.15.45:0/788112803 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118047070 con 0x7f311c0b1fa0 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.048+0000 7f3111ffb640 1 -- 172.21.15.45:0/788112803 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3118039a90 con 0x7f311c0b1fa0 2023-11-28T20:54:20.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.048+0000 7f3111ffb640 1 -- 172.21.15.45:0/788112803 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118039e20 con 0x7f311c0b1fa0 2023-11-28T20:54:20.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.048+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c158200 con 0x7f311c0b1fa0 2023-11-28T20:54:20.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.048+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c1587c0 con 0x7f311c0b1fa0 2023-11-28T20:54:20.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.049+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c0b44a0 con 0x7f311c0b1fa0 2023-11-28T20:54:20.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.050+0000 7f3111ffb640 1 -- 172.21.15.45:0/788112803 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f311804c410 con 0x7f311c0b1fa0 2023-11-28T20:54:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.051+0000 7f3111ffb640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f310006af60 0x7f310006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.051+0000 7f3111ffb640 1 -- 172.21.15.45:0/788112803 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f311803e070 con 0x7f311c0b1fa0 2023-11-28T20:54:20.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.052+0000 7f3120ec0640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f310006af60 0x7f310006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.053+0000 7f3120ec0640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f310006af60 0x7f310006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f310c006fd0 tx=0x7f310c008040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:20.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.056+0000 7f3111ffb640 1 -- 172.21.15.45:0/788112803 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3118076ce0 con 0x7f311c0b1fa0 2023-11-28T20:54:20.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.288+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f311c0b2480 con 0x7f311c0b1fa0 2023-11-28T20:54:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.289+0000 7f3111ffb640 1 -- 172.21.15.45:0/788112803 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f311807a300 con 0x7f311c0b1fa0 2023-11-28T20:54:20.291 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:20.291 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":61,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/544145878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":544145878},{"type":"v1","addr":"172.21.15.190:6836","nonce":544145878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2464692926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2464692926},{"type":"v1","addr":"172.21.15.45:6835","nonce":2464692926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2048455360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2048455360},{"type":"v1","addr":"172.21.15.190:6837","nonce":2048455360}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":8502,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2505159105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2505159105},{"type":"v1","addr":"172.21.15.45:6834","nonce":2505159105}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[]} 2023-11-28T20:54:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 61 2023-11-28T20:54:20.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.291+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f310006af60 msgr2=0x7f310006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:20.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.291+0000 7f3123925640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f310006af60 0x7f310006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f310c006fd0 tx=0x7f310c008040 comp rx=0 tx=0).stop 2023-11-28T20:54:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.291+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b1fa0 msgr2=0x7f311c13a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.291+0000 7f3123925640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b1fa0 0x7f311c13a800 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f3118000c00 tx=0x7f3118004730 comp rx=0 tx=0).stop 2023-11-28T20:54:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.291+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 shutdown_connections 2023-11-28T20:54:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.291+0000 7f3123925640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f310006af60 0x7f310006d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.291+0000 7f3123925640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f311c15b850 0x7f311c071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.291+0000 7f3123925640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f311c0b35b0 0x7f311c071140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.291+0000 7f3123925640 1 --2- 172.21.15.45:0/788112803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f311c0b1fa0 0x7f311c13a800 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.292+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 >> 172.21.15.45:0/788112803 conn(0x7f311c06a500 msgr2=0x7f311c15d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:20.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.292+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 shutdown_connections 2023-11-28T20:54:20.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.292+0000 7f3123925640 1 -- 172.21.15.45:0/788112803 wait complete. 2023-11-28T20:54:20.312 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T20:54:20.312 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T20:54:20.312 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T20:54:20.312 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:54:20.312 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T20:54:20.312 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T20:54:20.313 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T20:54:20.313 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:54:20.313 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T20:54:20.313 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T20:54:20.313 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T20:54:20.313 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:54:20.313 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T20:54:20.314 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T20:54:20.314 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T20:54:20.314 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:54:20.328 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T20:54:20.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T20:54:20.332 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T20:54:20.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T20:54:20.338 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T20:54:20.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T20:54:20.340 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T20:54:20.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T20:54:20.380 INFO:tasks.ceph.mds.a:Started 2023-11-28T20:54:20.380 DEBUG:teuthology.parallel:result is None 2023-11-28T20:54:20.388 INFO:tasks.ceph.mds.c:Started 2023-11-28T20:54:20.388 DEBUG:teuthology.parallel:result is None 2023-11-28T20:54:20.389 INFO:tasks.ceph.mds.d:Started 2023-11-28T20:54:20.389 DEBUG:teuthology.parallel:result is None 2023-11-28T20:54:20.395 INFO:tasks.ceph.mds.b:Started 2023-11-28T20:54:20.395 DEBUG:teuthology.parallel:result is None 2023-11-28T20:54:20.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T20:54:20.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-11-28T20:54:20.544 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:54:20.548 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:54:20.561 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:54:20.562 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T20:54:20.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.594+0000 7ffa710f3640 1 Processor -- start 2023-11-28T20:54:20.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.595+0000 7ffa710f3640 1 -- start start 2023-11-28T20:54:20.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.596+0000 7ffa710f3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 0x7ffa6c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.596+0000 7ffa710f3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa6c15edc0 0x7ffa6c15f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.596+0000 7ffa710f3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c15fa80 con 0x7ffa6c15b820 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.596+0000 7ffa710f3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6c15fd60 con 0x7ffa6c15edc0 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.596+0000 7ffa710f3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c06cf40 con 0x7ffa6c0cf730 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.596+0000 7ffa6b59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa6c15edc0 0x7ffa6c15f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.596+0000 7ffa6b59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa6c15edc0 0x7ffa6c15f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40460/0 (socket says 172.21.15.45:40460) 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.596+0000 7ffa6b59e640 1 -- 172.21.15.45:0/3035811744 learned_addr learned my addr 172.21.15.45:0/3035811744 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.597+0000 7ffa6259c640 1 --2- 172.21.15.45:0/3035811744 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 0x7ffa6c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.597+0000 7ffa6b59e640 1 -- 172.21.15.45:0/3035811744 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 msgr2=0x7ffa6c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.597+0000 7ffa6b59e640 1 --2- 172.21.15.45:0/3035811744 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 0x7ffa6c06da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.597+0000 7ffa6b59e640 1 -- 172.21.15.45:0/3035811744 >> v1:172.21.15.190:6789/0 conn(0x7ffa6c15b820 legacy=0x7ffa6c0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.597+0000 7ffa6b59e640 1 -- 172.21.15.45:0/3035811744 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c06d220 con 0x7ffa6c15edc0 2023-11-28T20:54:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.598+0000 7ffa6b59e640 1 --2- 172.21.15.45:0/3035811744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15edc0 0x7ffa6c15f1f0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009870 tx=0x7ffa6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d115fe9bb1a89268 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.598+0000 7ffa6a59c640 1 -- 172.21.15.45:0/3035811744 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64031640 con 0x7ffa6c15edc0 2023-11-28T20:54:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.598+0000 7ffa6a59c640 1 -- 172.21.15.45:0/3035811744 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa640317a0 con 0x7ffa6c15edc0 2023-11-28T20:54:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.598+0000 7ffa6a59c640 1 -- 172.21.15.45:0/3035811744 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64031b30 con 0x7ffa6c15edc0 2023-11-28T20:54:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.599+0000 7ffa710f3640 1 -- 172.21.15.45:0/3035811744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15edc0 msgr2=0x7ffa6c15f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.599+0000 7ffa710f3640 1 --2- 172.21.15.45:0/3035811744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15edc0 0x7ffa6c15f1f0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009870 tx=0x7ffa6402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.599+0000 7ffa710f3640 1 -- 172.21.15.45:0/3035811744 shutdown_connections 2023-11-28T20:54:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.599+0000 7ffa710f3640 1 --2- 172.21.15.45:0/3035811744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15edc0 0x7ffa6c15f1f0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.599+0000 7ffa710f3640 1 --2- 172.21.15.45:0/3035811744 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 0x7ffa6c06da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.599+0000 7ffa710f3640 1 -- 172.21.15.45:0/3035811744 >> 172.21.15.45:0/3035811744 conn(0x7ffa6c06a6e0 msgr2=0x7ffa6c06aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:20.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.599+0000 7ffa710f3640 1 -- 172.21.15.45:0/3035811744 shutdown_connections 2023-11-28T20:54:20.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.599+0000 7ffa710f3640 1 -- 172.21.15.45:0/3035811744 wait complete. 2023-11-28T20:54:20.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.601+0000 7ffa710f3640 1 Processor -- start 2023-11-28T20:54:20.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.601+0000 7ffa710f3640 1 -- start start 2023-11-28T20:54:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa710f3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 0x7ffa6c0c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa710f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15b820 0x7ffa6c0aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa710f3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6c15edc0 0x7ffa6c1f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa710f3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0c8ce0 con 0x7ffa6c15b820 2023-11-28T20:54:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa710f3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0c8fc0 con 0x7ffa6c15edc0 2023-11-28T20:54:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa710f3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0c92a0 con 0x7ffa6c0cf730 2023-11-28T20:54:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa6259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15b820 0x7ffa6c0aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa6259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15b820 0x7ffa6c0aeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40466/0 (socket says 172.21.15.45:40466) 2023-11-28T20:54:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa6259c640 1 -- 172.21.15.45:0/3563423728 learned_addr learned my addr 172.21.15.45:0/3563423728 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa6b59e640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6c15edc0 0x7ffa6c1f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.602+0000 7ffa6ad9d640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 0x7ffa6c0c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa6259c640 1 -- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 msgr2=0x7ffa6c0c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa6259c640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 0x7ffa6c0c8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa6259c640 1 -- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6c15edc0 msgr2=0x7ffa6c1f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa6259c640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6c15edc0 0x7ffa6c1f2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa6259c640 1 -- 172.21.15.45:0/3563423728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa640094d0 con 0x7ffa6c15b820 2023-11-28T20:54:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa6ad9d640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 0x7ffa6c0c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa6b59e640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6c15edc0 0x7ffa6c1f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa6259c640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15b820 0x7ffa6c0aeb60 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400d800 tx=0x7ffa5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa63fff640 1 -- 172.21.15.45:0/3563423728 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5400b6c0 con 0x7ffa6c15b820 2023-11-28T20:54:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa63fff640 1 -- 172.21.15.45:0/3563423728 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5400b820 con 0x7ffa6c15b820 2023-11-28T20:54:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa63fff640 1 -- 172.21.15.45:0/3563423728 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5400bbb0 con 0x7ffa6c15b820 2023-11-28T20:54:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c0c9580 con 0x7ffa6c15b820 2023-11-28T20:54:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.603+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c0c89d0 con 0x7ffa6c15b820 2023-11-28T20:54:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.605+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6c073440 con 0x7ffa6c15b820 2023-11-28T20:54:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.606+0000 7ffa63fff640 1 -- 172.21.15.45:0/3563423728 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ffa5400bd10 con 0x7ffa6c15b820 2023-11-28T20:54:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.607+0000 7ffa63fff640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ffa4c06af60 0x7ffa4c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.607+0000 7ffa63fff640 1 -- 172.21.15.45:0/3563423728 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6065+0+0 (secure 0 0 0) 0x7ffa54089fd0 con 0x7ffa6c15b820 2023-11-28T20:54:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.608+0000 7ffa6ad9d640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ffa4c06af60 0x7ffa4c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.608+0000 7ffa6ad9d640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ffa4c06af60 0x7ffa4c06d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c0045b0 tx=0x7ffa5c009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:20.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.612+0000 7ffa63fff640 1 -- 172.21.15.45:0/3563423728 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa54054cd0 con 0x7ffa6c15b820 2023-11-28T20:54:20.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:54:20.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:54:20.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:20.828+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ffa6c06c8c0 con 0x7ffa6c15b820 2023-11-28T20:54:21.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.830+0000 7ffa63fff640 1 -- 172.21.15.45:0/3563423728 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v144) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffa540582f0 con 0x7ffa6c15b820 2023-11-28T20:54:21.832 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T20:54:21.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.833+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ffa4c06af60 msgr2=0x7ffa4c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:21.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.833+0000 7ffa710f3640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ffa4c06af60 0x7ffa4c06d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c0045b0 tx=0x7ffa5c009040 comp rx=0 tx=0).stop 2023-11-28T20:54:21.836 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.833+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15b820 msgr2=0x7ffa6c0aeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:21.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.833+0000 7ffa710f3640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15b820 0x7ffa6c0aeb60 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400d800 tx=0x7ffa5400dcc0 comp rx=0 tx=0).stop 2023-11-28T20:54:21.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.834+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 shutdown_connections 2023-11-28T20:54:21.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.834+0000 7ffa710f3640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ffa4c06af60 0x7ffa4c06d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:21.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.834+0000 7ffa710f3640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa6c15edc0 0x7ffa6c1f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:21.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.834+0000 7ffa710f3640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa6c15b820 0x7ffa6c0aeb60 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:21.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.834+0000 7ffa710f3640 1 --2- 172.21.15.45:0/3563423728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffa6c0cf730 0x7ffa6c0c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.834+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 >> 172.21.15.45:0/3563423728 conn(0x7ffa6c06a6e0 msgr2=0x7ffa6c0ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.834+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 shutdown_connections 2023-11-28T20:54:21.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:21.834+0000 7ffa710f3640 1 -- 172.21.15.45:0/3563423728 wait complete. 2023-11-28T20:54:21.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T20:54:22.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.130+0000 7feb163c9640 1 Processor -- start 2023-11-28T20:54:22.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.131+0000 7feb163c9640 1 -- start start 2023-11-28T20:54:22.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.132+0000 7feb163c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb10141600 0x7feb100cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.132+0000 7feb163c9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb100cc0d0 0x7feb1015b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.132+0000 7feb163c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1015c670 con 0x7feb1015be80 2023-11-28T20:54:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.132+0000 7feb163c9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb100cc550 con 0x7feb100cc0d0 2023-11-28T20:54:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.132+0000 7feb163c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb100cda40 con 0x7feb10141600 2023-11-28T20:54:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.133+0000 7feb15bc8640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7feb1015be80 0x7feb1015ec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49268/0 (socket says 172.21.15.45:49268) 2023-11-28T20:54:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.133+0000 7feb15bc8640 1 -- 172.21.15.45:0/1630309621 learned_addr learned my addr 172.21.15.45:0/1630309621 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.133+0000 7feb153c7640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb10141600 0x7feb100cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.133+0000 7feb14bc6640 1 --2- 172.21.15.45:0/1630309621 >> v2:172.21.15.190:3300/0 conn(0x7feb100cc0d0 0x7feb1015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.133+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094877773 0 0) 0x7feb1015c670 con 0x7feb1015be80 2023-11-28T20:54:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.133+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feafc003520 con 0x7feb1015be80 2023-11-28T20:54:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3293197233 0 0) 0x7feafc003520 con 0x7feb1015be80 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb10141600 msgr2=0x7feb100cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb10141600 0x7feb100cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 >> v2:172.21.15.190:3300/0 conn(0x7feb100cc0d0 msgr2=0x7feb1015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 --2- 172.21.15.45:0/1630309621 >> v2:172.21.15.190:3300/0 conn(0x7feb100cc0d0 0x7feb1015b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100ccc80 con 0x7feb1015be80 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7feb080019f0 con 0x7feb1015be80 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 >> v1:172.21.15.45:6789/0 conn(0x7feb1015be80 legacy=0x7feb1015ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feafc003970 0x7feafc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feafc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb10141600 0x7feafc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb100cc550 con 0x7feafc0047c0 2023-11-28T20:54:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb100cda40 con 0x7feafc003970 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feafc00b980 con 0x7feb10141600 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.134+0000 7feb14bc6640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feafc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.135+0000 7feb153c7640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feafc003970 0x7feafc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.135+0000 7feb15bc8640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb10141600 0x7feafc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.135+0000 7feb14bc6640 1 -- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb10141600 msgr2=0x7feafc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.135+0000 7feb14bc6640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb10141600 0x7feafc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.135+0000 7feb14bc6640 1 -- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feafc003970 msgr2=0x7feafc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.135+0000 7feb14bc6640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feafc003970 0x7feafc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.135+0000 7feb14bc6640 1 -- 172.21.15.45:0/1630309621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feafc003520 con 0x7feafc0047c0 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.136+0000 7feb14bc6640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feafc004c40 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7feb00009ac0 tx=0x7feafc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd5a4043eb1d03 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.136+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0002c6d0 con 0x7feafc0047c0 2023-11-28T20:54:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.136+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0002c830 con 0x7feafc0047c0 2023-11-28T20:54:22.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.136+0000 7feb07fff640 1 -- 172.21.15.45:0/1630309621 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0002cbc0 con 0x7feafc0047c0 2023-11-28T20:54:22.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.137+0000 7feb163c9640 1 -- 172.21.15.45:0/1630309621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 msgr2=0x7feafc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.137+0000 7feb163c9640 1 --2- 172.21.15.45:0/1630309621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feafc004c40 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7feb00009ac0 tx=0x7feafc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.137+0000 7feb15bc8640 1 -- 172.21.15.45:0/1630309621 reap_dead start 2023-11-28T20:54:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.137+0000 7feb163c9640 1 -- 172.21.15.45:0/1630309621 shutdown_connections 2023-11-28T20:54:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.137+0000 7feb163c9640 1 -- 172.21.15.45:0/1630309621 >> 172.21.15.45:0/1630309621 conn(0x7feb100b0e70 msgr2=0x7feb100b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.137+0000 7feb163c9640 1 -- 172.21.15.45:0/1630309621 shutdown_connections 2023-11-28T20:54:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.137+0000 7feb163c9640 1 -- 172.21.15.45:0/1630309621 wait complete. 2023-11-28T20:54:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.139+0000 7feb163c9640 1 Processor -- start 2023-11-28T20:54:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.139+0000 7feb163c9640 1 -- start start 2023-11-28T20:54:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.140+0000 7feb163c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feb100c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.140+0000 7feb163c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1013f920 0x7feb10159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.140+0000 7feb163c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100c9d10 0x7feb100ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.140+0000 7feb163c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb100cb410 con 0x7feafc0047c0 2023-11-28T20:54:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.140+0000 7feb163c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb101401a0 con 0x7feb100c9d10 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.140+0000 7feb163c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb100c85e0 con 0x7feb1013f920 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb153c7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feb100c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb153c7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feb100c80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40484/0 (socket says 172.21.15.45:40484) 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb153c7640 1 -- 172.21.15.45:0/1878818420 learned_addr learned my addr 172.21.15.45:0/1878818420 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb153c7640 1 -- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1013f920 msgr2=0x7feb10159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb153c7640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1013f920 0x7feb10159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb153c7640 1 -- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100c9d10 msgr2=0x7feb100ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb15bc8640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100c9d10 0x7feb100ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb153c7640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100c9d10 0x7feb100ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb153c7640 1 -- 172.21.15.45:0/1878818420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100c8810 con 0x7feafc0047c0 2023-11-28T20:54:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb15bc8640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100c9d10 0x7feb100ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.141+0000 7feb153c7640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feb100c80a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7feb0c011660 tx=0x7feb0c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.142+0000 7feb05ffb640 1 -- 172.21.15.45:0/1878818420 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00f9c0 con 0x7feafc0047c0 2023-11-28T20:54:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.142+0000 7feb05ffb640 1 -- 172.21.15.45:0/1878818420 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c00fb20 con 0x7feafc0047c0 2023-11-28T20:54:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.142+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb100caa70 con 0x7feafc0047c0 2023-11-28T20:54:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.142+0000 7feb05ffb640 1 -- 172.21.15.45:0/1878818420 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00fe90 con 0x7feafc0047c0 2023-11-28T20:54:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.142+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb100cb030 con 0x7feafc0047c0 2023-11-28T20:54:22.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.143+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1004e920 con 0x7feafc0047c0 2023-11-28T20:54:22.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.144+0000 7feb05ffb640 1 -- 172.21.15.45:0/1878818420 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7feb0c015460 con 0x7feafc0047c0 2023-11-28T20:54:22.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.145+0000 7feb05ffb640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7feaf406af90 0x7feaf406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.146+0000 7feb05ffb640 1 -- 172.21.15.45:0/1878818420 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5205+0+0 (secure 0 0 0) 0x7feb0c089980 con 0x7feafc0047c0 2023-11-28T20:54:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.146+0000 7feb14bc6640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7feaf406af90 0x7feaf406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.147+0000 7feb14bc6640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7feaf406af90 0x7feaf406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb00036000 tx=0x7feb0002e000 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:22.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.151+0000 7feb05ffb640 1 -- 172.21.15.45:0/1878818420 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb0c054b00 con 0x7feafc0047c0 2023-11-28T20:54:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.409+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7feb100ce890 con 0x7feafc0047c0 2023-11-28T20:54:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.410+0000 7feb05ffb640 1 -- 172.21.15.45:0/1878818420 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7feb0c058120 con 0x7feafc0047c0 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:54:22.413 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:54:22.414 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.414 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.414 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T20:54:22.414 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T20:54:22.414 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.414 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:54:22.414 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:54:22.415 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.418 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T20:54:22.418 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T20:54:22.418 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.418 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:54:22.418 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:54:22.418 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T20:54:22.419 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:54:22.420 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:54:22.421 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.421 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T20:54:22.421 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T20:54:22.421 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.421 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:54:22.422 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:54:22.422 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.422 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T20:54:22.422 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T20:54:22.422 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.422 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.424 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T20:54:22.425 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:54:22.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:54:22.428 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:54:22.429 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:54:22.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:22.429 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T20:54:22.429 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:54:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.412+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7feaf406af90 msgr2=0x7feaf406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.412+0000 7feb163c9640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7feaf406af90 0x7feaf406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb00036000 tx=0x7feb0002e000 comp rx=0 tx=0).stop 2023-11-28T20:54:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.412+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 msgr2=0x7feb100c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.412+0000 7feb163c9640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feb100c80a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7feb0c011660 tx=0x7feb0c011b20 comp rx=0 tx=0).stop 2023-11-28T20:54:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.413+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 shutdown_connections 2023-11-28T20:54:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.413+0000 7feb163c9640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7feaf406af90 0x7feaf406d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.413+0000 7feb163c9640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100c9d10 0x7feb100ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.413+0000 7feb163c9640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1013f920 0x7feb10159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.413+0000 7feb163c9640 1 --2- 172.21.15.45:0/1878818420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feafc0047c0 0x7feb100c80a0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.413+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 >> 172.21.15.45:0/1878818420 conn(0x7feb100b0e70 msgr2=0x7feb1015f520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.413+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 shutdown_connections 2023-11-28T20:54:22.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.413+0000 7feb163c9640 1 -- 172.21.15.45:0/1878818420 wait complete. 2023-11-28T20:54:22.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:22.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.649+0000 7ff07030f640 1 Processor -- start 2023-11-28T20:54:22.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.649+0000 7ff07030f640 1 -- start start 2023-11-28T20:54:22.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.651+0000 7ff07030f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0680b35b0 0x7ff06815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.651+0000 7ff07030f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff06815b850 0x7ff0680b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.651+0000 7ff07030f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680b1fa0 0x7ff0680b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.651+0000 7ff07030f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06815ee40 con 0x7ff0680b35b0 2023-11-28T20:54:22.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.651+0000 7ff07030f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06815bda0 con 0x7ff06815b850 2023-11-28T20:54:22.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.651+0000 7ff07030f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06815c080 con 0x7ff0680b1fa0 2023-11-28T20:54:22.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.651+0000 7ff06e0ab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0680b35b0 0x7ff06815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.651+0000 7ff06e0ab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0680b35b0 0x7ff06815ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40490/0 (socket says 172.21.15.45:40490) 2023-11-28T20:54:22.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.651+0000 7ff06e0ab640 1 -- 172.21.15.45:0/1443859612 learned_addr learned my addr 172.21.15.45:0/1443859612 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:22.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.652+0000 7ff06d8aa640 1 --2- 172.21.15.45:0/1443859612 >> v2:172.21.15.190:3300/0 conn(0x7ff06815b850 0x7ff0680b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.652+0000 7ff06e8ac640 1 --2- 172.21.15.45:0/1443859612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680b1fa0 0x7ff0680b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.652+0000 7ff06e0ab640 1 -- 172.21.15.45:0/1443859612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680b1fa0 msgr2=0x7ff0680b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:22.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.652+0000 7ff06e0ab640 1 --2- 172.21.15.45:0/1443859612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680b1fa0 0x7ff0680b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.652+0000 7ff06e0ab640 1 -- 172.21.15.45:0/1443859612 >> v2:172.21.15.190:3300/0 conn(0x7ff06815b850 msgr2=0x7ff0680b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:22.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.652+0000 7ff06e0ab640 1 --2- 172.21.15.45:0/1443859612 >> v2:172.21.15.190:3300/0 conn(0x7ff06815b850 0x7ff0680b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.652+0000 7ff06e0ab640 1 -- 172.21.15.45:0/1443859612 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06815c360 con 0x7ff0680b35b0 2023-11-28T20:54:22.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.652+0000 7ff06d8aa640 1 --2- 172.21.15.45:0/1443859612 >> v2:172.21.15.190:3300/0 conn(0x7ff06815b850 0x7ff0680b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:22.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.652+0000 7ff06e8ac640 1 --2- 172.21.15.45:0/1443859612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680b1fa0 0x7ff0680b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:22.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.653+0000 7ff06e0ab640 1 --2- 172.21.15.45:0/1443859612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b35b0 0x7ff06815ca10 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ff0680cd380 tx=0x7ff06402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93ccb1b8784833c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:22.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.653+0000 7ff06d0a9640 1 -- 172.21.15.45:0/1443859612 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06402ca70 con 0x7ff0680b35b0 2023-11-28T20:54:22.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.653+0000 7ff06d0a9640 1 -- 172.21.15.45:0/1443859612 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06402cbd0 con 0x7ff0680b35b0 2023-11-28T20:54:22.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.653+0000 7ff06d0a9640 1 -- 172.21.15.45:0/1443859612 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0640343f0 con 0x7ff0680b35b0 2023-11-28T20:54:22.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.654+0000 7ff07030f640 1 -- 172.21.15.45:0/1443859612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b35b0 msgr2=0x7ff06815ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.654+0000 7ff07030f640 1 --2- 172.21.15.45:0/1443859612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b35b0 0x7ff06815ca10 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ff0680cd380 tx=0x7ff06402bed0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.654+0000 7ff07030f640 1 -- 172.21.15.45:0/1443859612 shutdown_connections 2023-11-28T20:54:22.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.654+0000 7ff07030f640 1 --2- 172.21.15.45:0/1443859612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680b1fa0 0x7ff0680b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.654+0000 7ff07030f640 1 --2- 172.21.15.45:0/1443859612 >> v2:172.21.15.190:3300/0 conn(0x7ff06815b850 0x7ff0680b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.654+0000 7ff07030f640 1 --2- 172.21.15.45:0/1443859612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b35b0 0x7ff06815ca10 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.654+0000 7ff07030f640 1 -- 172.21.15.45:0/1443859612 >> 172.21.15.45:0/1443859612 conn(0x7ff06806a500 msgr2=0x7ff06806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:22.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.654+0000 7ff07030f640 1 -- 172.21.15.45:0/1443859612 shutdown_connections 2023-11-28T20:54:22.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.654+0000 7ff07030f640 1 -- 172.21.15.45:0/1443859612 wait complete. 2023-11-28T20:54:22.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.656+0000 7ff07030f640 1 Processor -- start 2023-11-28T20:54:22.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.657+0000 7ff07030f640 1 -- start start 2023-11-28T20:54:22.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.658+0000 7ff07030f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b1fa0 0x7ff06815a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.658+0000 7ff07030f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0680b35b0 0x7ff068071e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.658+0000 7ff07030f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815b850 0x7ff0681f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.658+0000 7ff07030f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06815ad40 con 0x7ff0680b1fa0 2023-11-28T20:54:22.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.658+0000 7ff07030f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0681f0a00 con 0x7ff0680b35b0 2023-11-28T20:54:22.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.658+0000 7ff07030f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff068072340 con 0x7ff06815b850 2023-11-28T20:54:22.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.658+0000 7ff06e0ab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b1fa0 0x7ff06815a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.658+0000 7ff06e0ab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b1fa0 0x7ff06815a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40500/0 (socket says 172.21.15.45:40500) 2023-11-28T20:54:22.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.658+0000 7ff06e0ab640 1 -- 172.21.15.45:0/86797881 learned_addr learned my addr 172.21.15.45:0/86797881 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:22.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff06e0ab640 1 -- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815b850 msgr2=0x7ff0681f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff06e0ab640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815b850 0x7ff0681f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff06e0ab640 1 -- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0680b35b0 msgr2=0x7ff068071e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff06e0ab640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0680b35b0 0x7ff068071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff06e0ab640 1 -- 172.21.15.45:0/86797881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff064009b30 con 0x7ff0680b1fa0 2023-11-28T20:54:22.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff06e0ab640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b1fa0 0x7ff06815a800 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7ff064009e20 tx=0x7ff064004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:22.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff05effd640 1 -- 172.21.15.45:0/86797881 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06403e450 con 0x7ff0680b1fa0 2023-11-28T20:54:22.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff05effd640 1 -- 172.21.15.45:0/86797881 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06403e5f0 con 0x7ff0680b1fa0 2023-11-28T20:54:22.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff05effd640 1 -- 172.21.15.45:0/86797881 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06403e9a0 con 0x7ff0680b1fa0 2023-11-28T20:54:22.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff068070440 con 0x7ff0680b1fa0 2023-11-28T20:54:22.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.659+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff068070a00 con 0x7ff0680b1fa0 2023-11-28T20:54:22.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.660+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0680733d0 con 0x7ff0680b1fa0 2023-11-28T20:54:22.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.662+0000 7ff05effd640 1 -- 172.21.15.45:0/86797881 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff06403eb00 con 0x7ff0680b1fa0 2023-11-28T20:54:22.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.663+0000 7ff05effd640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff04c06af60 0x7ff04c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:22.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.663+0000 7ff05effd640 1 -- 172.21.15.45:0/86797881 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5205+0+0 (secure 0 0 0) 0x7ff0640ac560 con 0x7ff0680b1fa0 2023-11-28T20:54:22.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.664+0000 7ff06d8aa640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff04c06af60 0x7ff04c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:22.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.665+0000 7ff06d8aa640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff04c06af60 0x7ff04c06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff058006b10 tx=0x7ff058008040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:22.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.668+0000 7ff05effd640 1 -- 172.21.15.45:0/86797881 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff064077820 con 0x7ff0680b1fa0 2023-11-28T20:54:22.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.865+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0680cc350 con 0x7ff0680b1fa0 2023-11-28T20:54:22.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.867+0000 7ff05effd640 1 -- 172.21.15.45:0/86797881 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+13104 (secure 0 0 0) 0x7ff06403edb0 con 0x7ff0680b1fa0 2023-11-28T20:54:22.870 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:22.870 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":144,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:21.827310+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:22.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff04c06af60 msgr2=0x7ff04c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff04c06af60 0x7ff04c06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff058006b10 tx=0x7ff058008040 comp rx=0 tx=0).stop 2023-11-28T20:54:22.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b1fa0 msgr2=0x7ff06815a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b1fa0 0x7ff06815a800 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7ff064009e20 tx=0x7ff064004a40 comp rx=0 tx=0).stop 2023-11-28T20:54:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 shutdown_connections 2023-11-28T20:54:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff04c06af60 0x7ff04c06d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815b850 0x7ff0681f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0680b35b0 0x7ff068071e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 --2- 172.21.15.45:0/86797881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0680b1fa0 0x7ff06815a800 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 >> 172.21.15.45:0/86797881 conn(0x7ff06806a500 msgr2=0x7ff0680cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 shutdown_connections 2023-11-28T20:54:22.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:22.870+0000 7ff07030f640 1 -- 172.21.15.45:0/86797881 wait complete. 2023-11-28T20:54:22.892 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:54:22.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:54:23.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.118+0000 7f9329d9a640 1 Processor -- start 2023-11-28T20:54:23.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.118+0000 7f9329d9a640 1 -- start start 2023-11-28T20:54:23.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.119+0000 7f9329d9a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93240d0040 0x7f93240d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.119+0000 7f9329d9a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93240b90a0 con 0x7f93240b1d40 2023-11-28T20:54:23.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.119+0000 7f9329d9a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93240d1920 con 0x7f93240b2290 2023-11-28T20:54:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.119+0000 7f9329d9a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93240d3e60 con 0x7f93240d0040 2023-11-28T20:54:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.119+0000 7f9328d98640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93240d0040 0x7f93240d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.120+0000 7f9328d98640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93240d0040 0x7f93240d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40502/0 (socket says 172.21.15.45:40502) 2023-11-28T20:54:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.120+0000 7f9328d98640 1 -- 172.21.15.45:0/2136138106 learned_addr learned my addr 172.21.15.45:0/2136138106 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.120+0000 7f9329599640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f93240b1d40 0x7f93240d3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49274/0 (socket says 172.21.15.45:49274) 2023-11-28T20:54:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.120+0000 7f9328d98640 1 -- 172.21.15.45:0/2136138106 >> v1:172.21.15.190:6789/0 conn(0x7f93240b2290 legacy=0x7f93240d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.120+0000 7f9328d98640 1 -- 172.21.15.45:0/2136138106 >> v1:172.21.15.45:6789/0 conn(0x7f93240b1d40 legacy=0x7f93240d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.120+0000 7f93237fe640 1 -- 172.21.15.45:0/2136138106 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421616062 0 0) 0x7f93240b90a0 con 0x7f93240b1d40 2023-11-28T20:54:23.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.120+0000 7f9328d98640 1 -- 172.21.15.45:0/2136138106 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93240d2dc0 con 0x7f93240d0040 2023-11-28T20:54:23.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.121+0000 7f9328d98640 1 --2- 172.21.15.45:0/2136138106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240d0040 0x7f93240d0420 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f9314009870 tx=0x7f931402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4359af5edcab587f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:23.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.121+0000 7f93237fe640 1 -- 172.21.15.45:0/2136138106 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931402f840 con 0x7f93240d0040 2023-11-28T20:54:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.122+0000 7f93237fe640 1 -- 172.21.15.45:0/2136138106 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931402f9a0 con 0x7f93240d0040 2023-11-28T20:54:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.122+0000 7f93237fe640 1 -- 172.21.15.45:0/2136138106 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931402fd30 con 0x7f93240d0040 2023-11-28T20:54:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.122+0000 7f9329d9a640 1 -- 172.21.15.45:0/2136138106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240d0040 msgr2=0x7f93240d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.122+0000 7f9329d9a640 1 --2- 172.21.15.45:0/2136138106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240d0040 0x7f93240d0420 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f9314009870 tx=0x7f931402ba20 comp rx=0 tx=0).stop 2023-11-28T20:54:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.122+0000 7f9329d9a640 1 -- 172.21.15.45:0/2136138106 shutdown_connections 2023-11-28T20:54:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.122+0000 7f9329d9a640 1 --2- 172.21.15.45:0/2136138106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240d0040 0x7f93240d0420 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.122+0000 7f9329d9a640 1 -- 172.21.15.45:0/2136138106 >> 172.21.15.45:0/2136138106 conn(0x7f9324070f40 msgr2=0x7f9324071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.122+0000 7f9329d9a640 1 -- 172.21.15.45:0/2136138106 shutdown_connections 2023-11-28T20:54:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.122+0000 7f9329d9a640 1 -- 172.21.15.45:0/2136138106 wait complete. 2023-11-28T20:54:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.124+0000 7f9329d9a640 1 Processor -- start 2023-11-28T20:54:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.124+0000 7f9329d9a640 1 -- start start 2023-11-28T20:54:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.125+0000 7f9329d9a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93240b1d40 0x7f93240c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.125+0000 7f9329d9a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240b2290 0x7f93240c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.125+0000 7f9329d9a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93240d0040 0x7f93241e42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.125+0000 7f9329d9a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93240c7b90 con 0x7f93240b2290 2023-11-28T20:54:23.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.125+0000 7f9329d9a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93240c6f10 con 0x7f93240b1d40 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.125+0000 7f9329d9a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93240c71f0 con 0x7f93240d0040 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9323fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240b2290 0x7f93240c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9323fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240b2290 0x7f93240c8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40514/0 (socket says 172.21.15.45:40514) 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9323fff640 1 -- 172.21.15.45:0/1128630431 learned_addr learned my addr 172.21.15.45:0/1128630431 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9328d98640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93240b1d40 0x7f93240c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9323fff640 1 -- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93240d0040 msgr2=0x7f93241e42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9329599640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93240d0040 0x7f93241e42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9323fff640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93240d0040 0x7f93241e42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9323fff640 1 -- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93240b1d40 msgr2=0x7f93240c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9323fff640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93240b1d40 0x7f93240c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9323fff640 1 -- 172.21.15.45:0/1128630431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93140094d0 con 0x7f93240b2290 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.126+0000 7f9328d98640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93240b1d40 0x7f93240c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:23.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.127+0000 7f9323fff640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240b2290 0x7f93240c8fb0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f931800c7c0 tx=0x7f931800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:23.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.127+0000 7f93217fa640 1 -- 172.21.15.45:0/1128630431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931800aa50 con 0x7f93240b2290 2023-11-28T20:54:23.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.127+0000 7f93217fa640 1 -- 172.21.15.45:0/1128630431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931800abb0 con 0x7f93240b2290 2023-11-28T20:54:23.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.127+0000 7f93217fa640 1 -- 172.21.15.45:0/1128630431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318012420 con 0x7f93240b2290 2023-11-28T20:54:23.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.127+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93240c74d0 con 0x7f93240b2290 2023-11-28T20:54:23.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.127+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93240c94f0 con 0x7f93240b2290 2023-11-28T20:54:23.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.128+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93240c7cd0 con 0x7f93240b2290 2023-11-28T20:54:23.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.129+0000 7f93217fa640 1 -- 172.21.15.45:0/1128630431 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9318012580 con 0x7f93240b2290 2023-11-28T20:54:23.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.131+0000 7f93217fa640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f931006af60 0x7f931006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.131+0000 7f93217fa640 1 -- 172.21.15.45:0/1128630431 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f93180895b0 con 0x7f93240b2290 2023-11-28T20:54:23.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.132+0000 7f9328d98640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f931006af60 0x7f931006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.133+0000 7f9328d98640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f931006af60 0x7f931006d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9314036000 tx=0x7f9314039040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:23.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.135+0000 7f93217fa640 1 -- 172.21.15.45:0/1128630431 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93180546a0 con 0x7f93240b2290 2023-11-28T20:54:23.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.327+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9324071b30 con 0x7f93240b2290 2023-11-28T20:54:23.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.328+0000 7f93217fa640 1 -- 172.21.15.45:0/1128630431 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9318057cc0 con 0x7f93240b2290 2023-11-28T20:54:23.331 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:54:23.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.331+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f931006af60 msgr2=0x7f931006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.331+0000 7f9329d9a640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f931006af60 0x7f931006d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9314036000 tx=0x7f9314039040 comp rx=0 tx=0).stop 2023-11-28T20:54:23.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.331+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240b2290 msgr2=0x7f93240c8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.331+0000 7f9329d9a640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240b2290 0x7f93240c8fb0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f931800c7c0 tx=0x7f931800cc80 comp rx=0 tx=0).stop 2023-11-28T20:54:23.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.332+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 shutdown_connections 2023-11-28T20:54:23.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.332+0000 7f9329d9a640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f931006af60 0x7f931006d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.332+0000 7f9329d9a640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93240d0040 0x7f93241e42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.332+0000 7f9329d9a640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93240b2290 0x7f93240c8fb0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.332+0000 7f9329d9a640 1 --2- 172.21.15.45:0/1128630431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93240b1d40 0x7f93240c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.332+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 >> 172.21.15.45:0/1128630431 conn(0x7f9324070f40 msgr2=0x7f93240b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:23.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.332+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 shutdown_connections 2023-11-28T20:54:23.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.332+0000 7f9329d9a640 1 -- 172.21.15.45:0/1128630431 wait complete. 2023-11-28T20:54:23.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T20:54:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.582+0000 7f75e4705640 1 Processor -- start 2023-11-28T20:54:23.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.582+0000 7f75e4705640 1 -- start start 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.584+0000 7f75e4705640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f75dc0b35b0 0x7f75dc15ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.584+0000 7f75e4705640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc15b850 0x7f75dc0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.584+0000 7f75e4705640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc15bda0 con 0x7f75dc0b1fa0 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.584+0000 7f75e4705640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75dc15ee40 con 0x7f75dc0b35b0 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.584+0000 7f75e4705640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc15c080 con 0x7f75dc15b850 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.584+0000 7f75e24a1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f75dc0b35b0 0x7f75dc15ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.584+0000 7f75e24a1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f75dc0b35b0 0x7f75dc15ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40526/0 (socket says 172.21.15.45:40526) 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.584+0000 7f75e24a1640 1 -- 172.21.15.45:0/736439652 learned_addr learned my addr 172.21.15.45:0/736439652 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.584+0000 7f75e2ca2640 1 --1- 172.21.15.45:0/736439652 >> v1:172.21.15.45:6789/0 conn(0x7f75dc0b1fa0 0x7f75dc15f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49282/0 (socket says 172.21.15.45:49282) 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.585+0000 7f75e1ca0640 1 --2- 172.21.15.45:0/736439652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc15b850 0x7f75dc0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.585+0000 7f75e24a1640 1 -- 172.21.15.45:0/736439652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc15b850 msgr2=0x7f75dc0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:23.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.585+0000 7f75e24a1640 1 --2- 172.21.15.45:0/736439652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc15b850 0x7f75dc0b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.585+0000 7f75e24a1640 1 -- 172.21.15.45:0/736439652 >> v1:172.21.15.45:6789/0 conn(0x7f75dc0b1fa0 legacy=0x7f75dc15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.585+0000 7f75e24a1640 1 -- 172.21.15.45:0/736439652 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc15c360 con 0x7f75dc0b35b0 2023-11-28T20:54:23.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.586+0000 7f75e24a1640 1 --2- 172.21.15.45:0/736439652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc0b35b0 0x7f75dc15ca10 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f75d80098a0 tx=0x7f75d802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0cb3a1f19a8f79 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:23.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.586+0000 7f75e149f640 1 -- 172.21.15.45:0/736439652 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8030840 con 0x7f75dc0b35b0 2023-11-28T20:54:23.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.586+0000 7f75e149f640 1 -- 172.21.15.45:0/736439652 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d80309a0 con 0x7f75dc0b35b0 2023-11-28T20:54:23.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.586+0000 7f75e149f640 1 -- 172.21.15.45:0/736439652 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8030d30 con 0x7f75dc0b35b0 2023-11-28T20:54:23.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.587+0000 7f75e4705640 1 -- 172.21.15.45:0/736439652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc0b35b0 msgr2=0x7f75dc15ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.587+0000 7f75e4705640 1 --2- 172.21.15.45:0/736439652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc0b35b0 0x7f75dc15ca10 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f75d80098a0 tx=0x7f75d802b980 comp rx=0 tx=0).stop 2023-11-28T20:54:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.587+0000 7f75e4705640 1 -- 172.21.15.45:0/736439652 shutdown_connections 2023-11-28T20:54:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.587+0000 7f75e4705640 1 --2- 172.21.15.45:0/736439652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc15b850 0x7f75dc0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.587+0000 7f75e4705640 1 --2- 172.21.15.45:0/736439652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc0b35b0 0x7f75dc15ca10 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.587+0000 7f75e4705640 1 -- 172.21.15.45:0/736439652 >> 172.21.15.45:0/736439652 conn(0x7f75dc06a500 msgr2=0x7f75dc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.587+0000 7f75e4705640 1 -- 172.21.15.45:0/736439652 shutdown_connections 2023-11-28T20:54:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.587+0000 7f75e4705640 1 -- 172.21.15.45:0/736439652 wait complete. 2023-11-28T20:54:23.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.589+0000 7f75e4705640 1 Processor -- start 2023-11-28T20:54:23.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.589+0000 7f75e4705640 1 -- start start 2023-11-28T20:54:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.590+0000 7f75e4705640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75dc0b1fa0 0x7f75dc13fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.590+0000 7f75e4705640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc0b35b0 0x7f75dc1f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.590+0000 7f75e4705640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc15b850 0x7f75dc0b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.590+0000 7f75e4705640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc1f63c0 con 0x7f75dc15b850 2023-11-28T20:54:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.590+0000 7f75e4705640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc1f3660 con 0x7f75dc0b1fa0 2023-11-28T20:54:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.590+0000 7f75e4705640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc1f3940 con 0x7f75dc0b35b0 2023-11-28T20:54:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.590+0000 7f75e2ca2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc15b850 0x7f75dc0b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.590+0000 7f75e2ca2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc15b850 0x7f75dc0b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40536/0 (socket says 172.21.15.45:40536) 2023-11-28T20:54:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.590+0000 7f75e2ca2640 1 -- 172.21.15.45:0/3755157632 learned_addr learned my addr 172.21.15.45:0/3755157632 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75e1ca0640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc0b35b0 0x7f75dc1f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75e24a1640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75dc0b1fa0 0x7f75dc13fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75e2ca2640 1 -- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc0b35b0 msgr2=0x7f75dc1f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75e2ca2640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc0b35b0 0x7f75dc1f2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75e2ca2640 1 -- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75dc0b1fa0 msgr2=0x7f75dc13fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75e2ca2640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75dc0b1fa0 0x7f75dc13fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75e2ca2640 1 -- 172.21.15.45:0/3755157632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d80094d0 con 0x7f75dc15b850 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75e2ca2640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc15b850 0x7f75dc0b0e90 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f75d400c7a0 tx=0x7f75d400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75d2ffd640 1 -- 172.21.15.45:0/3755157632 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d400a640 con 0x7f75dc15b850 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75d2ffd640 1 -- 172.21.15.45:0/3755157632 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d400a7a0 con 0x7f75dc15b850 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc1f3c20 con 0x7f75dc15b850 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.591+0000 7f75d2ffd640 1 -- 172.21.15.45:0/3755157632 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d400ab10 con 0x7f75dc15b850 2023-11-28T20:54:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.592+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc1f3350 con 0x7f75dc15b850 2023-11-28T20:54:23.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.593+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75dc0733d0 con 0x7f75dc15b850 2023-11-28T20:54:23.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.594+0000 7f75d2ffd640 1 -- 172.21.15.45:0/3755157632 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f75d400ac70 con 0x7f75dc15b850 2023-11-28T20:54:23.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.595+0000 7f75d2ffd640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f75c006af60 0x7f75c006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:23.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.595+0000 7f75d2ffd640 1 -- 172.21.15.45:0/3755157632 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f75d4088cc0 con 0x7f75dc15b850 2023-11-28T20:54:23.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.596+0000 7f75e24a1640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f75c006af60 0x7f75c006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:23.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.597+0000 7f75e24a1640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f75c006af60 0x7f75c006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75d8036000 tx=0x7f75d803b040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.599+0000 7f75d2ffd640 1 -- 172.21.15.45:0/3755157632 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75d4053f00 con 0x7f75dc15b850 2023-11-28T20:54:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.799+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f75dc0cd3b0 con 0x7f75dc15b850 2023-11-28T20:54:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.800+0000 7f75d2ffd640 1 -- 172.21.15.45:0/3755157632 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v144) v1 ==== 84+0+65 (secure 0 0 0) 0x7f75d4017020 con 0x7f75dc15b850 2023-11-28T20:54:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:23.803 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:54:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T20:54:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T20:54:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:23.803 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:54:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f75c006af60 msgr2=0x7f75c006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f75c006af60 0x7f75c006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75d8036000 tx=0x7f75d803b040 comp rx=0 tx=0).stop 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc15b850 msgr2=0x7f75dc0b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc15b850 0x7f75dc0b0e90 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f75d400c7a0 tx=0x7f75d400cc60 comp rx=0 tx=0).stop 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 shutdown_connections 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f75c006af60 0x7f75c006d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75dc15b850 0x7f75dc0b0e90 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75dc0b35b0 0x7f75dc1f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 --2- 172.21.15.45:0/3755157632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75dc0b1fa0 0x7f75dc13fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 >> 172.21.15.45:0/3755157632 conn(0x7f75dc06a500 msgr2=0x7f75dc0ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.803+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 shutdown_connections 2023-11-28T20:54:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:23.804+0000 7f75e4705640 1 -- 172.21.15.45:0/3755157632 wait complete. 2023-11-28T20:54:23.825 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T20:54:23.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T20:54:24.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.052+0000 7fc4d4a4a640 1 Processor -- start 2023-11-28T20:54:24.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.052+0000 7fc4d4a4a640 1 -- start start 2023-11-28T20:54:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.054+0000 7fc4d4a4a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc4d0070030 0x7fc4d0070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.054+0000 7fc4d4a4a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d00acad0 con 0x7fc4d00cc900 2023-11-28T20:54:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.054+0000 7fc4d4a4a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d00acdb0 con 0x7fc4d01607c0 2023-11-28T20:54:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.054+0000 7fc4d4a4a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d00ac8c0 con 0x7fc4d0070030 2023-11-28T20:54:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.054+0000 7fc4c7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc4d00cc900 0x7fc4d00adb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42438/0 (socket says 172.21.15.45:42438) 2023-11-28T20:54:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.054+0000 7fc4c7fff640 1 -- 172.21.15.45:0/2404790640 learned_addr learned my addr 172.21.15.45:0/2404790640 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4cf59e640 1 --2- 172.21.15.45:0/2404790640 >> v2:172.21.15.190:3300/0 conn(0x7fc4d0070030 0x7fc4d0070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875571316 0 0) 0x7fc4d00acad0 con 0x7fc4d00cc900 2023-11-28T20:54:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4bc003520 con 0x7fc4d00cc900 2023-11-28T20:54:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4078555629 0 0) 0x7fc4bc003520 con 0x7fc4d00cc900 2023-11-28T20:54:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 >> v2:172.21.15.190:3300/0 conn(0x7fc4d0070030 msgr2=0x7fc4d0070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 --2- 172.21.15.45:0/2404790640 >> v2:172.21.15.190:3300/0 conn(0x7fc4d0070030 0x7fc4d0070460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 >> v1:172.21.15.190:6789/0 conn(0x7fc4d01607c0 legacy=0x7fc4d00cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d00ad090 con 0x7fc4d00cc900 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc4b8001670 con 0x7fc4d00cc900 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 >> v1:172.21.15.45:6789/0 conn(0x7fc4d00cc900 legacy=0x7fc4d00adb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc003970 0x7fc4bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.055+0000 7fc4ce59c640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4bc0047c0 0x7fc4bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4ce59c640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4bc005180 0x7fc4bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d00ac8c0 con 0x7fc4bc005180 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc00c120 con 0x7fc4bc003970 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc00c2e0 con 0x7fc4bc0047c0 2023-11-28T20:54:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4cf59e640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4bc005180 0x7fc4bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4ced9d640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc003970 0x7fc4bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4cf59e640 1 -- 172.21.15.45:0/2404790640 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4bc0047c0 msgr2=0x7fc4bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4cf59e640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4bc0047c0 0x7fc4bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4cf59e640 1 -- 172.21.15.45:0/2404790640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc003970 msgr2=0x7fc4bc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4cf59e640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc003970 0x7fc4bc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4cf59e640 1 -- 172.21.15.45:0/2404790640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc003520 con 0x7fc4bc005180 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.056+0000 7fc4ced9d640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc003970 0x7fc4bc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.057+0000 7fc4cf59e640 1 -- 172.21.15.45:0/2404790640 reap_dead start 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.057+0000 7fc4cf59e640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4bc005180 0x7fc4bc00b980 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009fc0 tx=0x7fc4bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d2a27564072794 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c802cc60 con 0x7fc4bc005180 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c800f040 con 0x7fc4bc005180 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4ce59c640 1 -- 172.21.15.45:0/2404790640 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8003990 con 0x7fc4bc005180 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/2404790640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4bc005180 msgr2=0x7fc4bc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4d4a4a640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4bc005180 0x7fc4bc00b980 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009fc0 tx=0x7fc4bc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/2404790640 shutdown_connections 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4d4a4a640 1 --2- 172.21.15.45:0/2404790640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4bc005180 0x7fc4bc00b980 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/2404790640 >> 172.21.15.45:0/2404790640 conn(0x7fc4d00cec30 msgr2=0x7fc4d00cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:24.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/2404790640 shutdown_connections 2023-11-28T20:54:24.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.058+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/2404790640 wait complete. 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.060+0000 7fc4d4a4a640 1 Processor -- start 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.060+0000 7fc4d4a4a640 1 -- start start 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.061+0000 7fc4d4a4a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc005180 0x7fc4d015f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.061+0000 7fc4d4a4a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4d01f5fe0 0x7fc4d01f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.061+0000 7fc4d4a4a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d01f69a0 0x7fc4d015d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4d4a4a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d015c4e0 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4d4a4a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d01f6e20 con 0x7fc4bc005180 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4d4a4a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d015c6c0 con 0x7fc4d01f69a0 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4c7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4d01f5fe0 0x7fc4d01f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4c7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4d01f5fe0 0x7fc4d01f6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48178/0 (socket says 172.21.15.45:48178) 2023-11-28T20:54:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4c7fff640 1 -- 172.21.15.45:0/3079997325 learned_addr learned my addr 172.21.15.45:0/3079997325 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4cf59e640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d01f69a0 0x7fc4d015d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4ced9d640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc005180 0x7fc4d015f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4c7fff640 1 -- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d01f69a0 msgr2=0x7fc4d015d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4c7fff640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d01f69a0 0x7fc4d015d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4c7fff640 1 -- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc005180 msgr2=0x7fc4d015f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4c7fff640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc005180 0x7fc4d015f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.062+0000 7fc4c7fff640 1 -- 172.21.15.45:0/3079997325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d015c9a0 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.063+0000 7fc4c7fff640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4d01f5fe0 0x7fc4d01f6460 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800c760 tx=0x7fc4b800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.063+0000 7fc4c77fe640 1 -- 172.21.15.45:0/3079997325 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b80074e0 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.063+0000 7fc4c77fe640 1 -- 172.21.15.45:0/3079997325 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b8007680 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.063+0000 7fc4c77fe640 1 -- 172.21.15.45:0/3079997325 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8004a00 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.063+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d015cc80 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.063+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d015f610 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.064+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d015d1a0 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.065+0000 7fc4c77fe640 1 -- 172.21.15.45:0/3079997325 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc4b8004b60 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.067+0000 7fc4c77fe640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc4b006af60 0x7fc4b006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.067+0000 7fc4c77fe640 1 -- 172.21.15.45:0/3079997325 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fc4b8088a40 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.067+0000 7fc4ced9d640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc4b006af60 0x7fc4b006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.068+0000 7fc4ced9d640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc4b006af60 0x7fc4b006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0000f60 tx=0x7fc4c0009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.071+0000 7fc4c77fe640 1 -- 172.21.15.45:0/3079997325 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4d015d1a0 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.264+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc4d015d1a0 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.857+0000 7fc4c77fe640 1 -- 172.21.15.45:0/3079997325 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v145) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc4d015d1a0 con 0x7fc4d01f5fe0 2023-11-28T20:54:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T20:54:24.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.861+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc4b006af60 msgr2=0x7fc4b006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:24.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.861+0000 7fc4d4a4a640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc4b006af60 0x7fc4b006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0000f60 tx=0x7fc4c0009040 comp rx=0 tx=0).stop 2023-11-28T20:54:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.861+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4d01f5fe0 msgr2=0x7fc4d01f6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.861+0000 7fc4d4a4a640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4d01f5fe0 0x7fc4d01f6460 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800c760 tx=0x7fc4b800cc20 comp rx=0 tx=0).stop 2023-11-28T20:54:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.862+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 shutdown_connections 2023-11-28T20:54:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.862+0000 7fc4d4a4a640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc4b006af60 0x7fc4b006d420 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0000f60 tx=0x7fc4c0009040 comp rx=0 tx=0).stop 2023-11-28T20:54:24.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.862+0000 7fc4d4a4a640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d01f69a0 0x7fc4d015d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.862+0000 7fc4d4a4a640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4d01f5fe0 0x7fc4d01f6460 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.862+0000 7fc4d4a4a640 1 --2- 172.21.15.45:0/3079997325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4bc005180 0x7fc4d015f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:24.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.862+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 >> 172.21.15.45:0/3079997325 conn(0x7fc4d00cec30 msgr2=0x7fc4d00ada50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:24.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.863+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 shutdown_connections 2023-11-28T20:54:24.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:24.863+0000 7fc4d4a4a640 1 -- 172.21.15.45:0/3079997325 wait complete. 2023-11-28T20:54:24.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T20:54:25.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.107+0000 7ff9b31c9640 1 Processor -- start 2023-11-28T20:54:25.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.107+0000 7ff9b31c9640 1 -- start start 2023-11-28T20:54:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.109+0000 7ff9b31c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 0x7ff9ac0ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.109+0000 7ff9b31c9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff9ac161ba0 0x7ff9ac161ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.109+0000 7ff9b31c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac0cea90 con 0x7ff9ac06c6e0 2023-11-28T20:54:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.109+0000 7ff9b31c9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac0cc800 con 0x7ff9ac161ba0 2023-11-28T20:54:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.109+0000 7ff9b31c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac0ccac0 con 0x7ff9ac06c300 2023-11-28T20:54:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.109+0000 7ff9b29c8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff9ac161ba0 0x7ff9ac161ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.109+0000 7ff9b19c6640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff9ac06c6e0 0x7ff9ac0cdb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42440/0 (socket says 172.21.15.45:42440) 2023-11-28T20:54:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.109+0000 7ff9b19c6640 1 -- 172.21.15.45:0/1635528486 learned_addr learned my addr 172.21.15.45:0/1635528486 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.110+0000 7ff9b21c7640 1 --2- 172.21.15.45:0/1635528486 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 0x7ff9ac0ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.110+0000 7ff9b29c8640 1 -- 172.21.15.45:0/1635528486 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 msgr2=0x7ff9ac0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:25.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.110+0000 7ff9b11c5640 1 -- 172.21.15.45:0/1635528486 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450405645 0 0) 0x7ff99c001110 con 0x7ff9ac06c6e0 2023-11-28T20:54:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.110+0000 7ff9b29c8640 1 --2- 172.21.15.45:0/1635528486 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 0x7ff9ac0ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.110+0000 7ff9b29c8640 1 -- 172.21.15.45:0/1635528486 >> v1:172.21.15.45:6789/0 conn(0x7ff9ac06c6e0 legacy=0x7ff9ac0cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.110+0000 7ff9b29c8640 1 -- 172.21.15.45:0/1635528486 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac0ccda0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.111+0000 7ff9b29c8640 1 --2- 172.21.15.45:0/1635528486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 0x7ff9ac161ff0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009d90 tx=0x7ff9a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965748049f17848e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.111+0000 7ff9b11c5640 1 -- 172.21.15.45:0/1635528486 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a402cc50 con 0x7ff9ac161ba0 2023-11-28T20:54:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.112+0000 7ff9b11c5640 1 -- 172.21.15.45:0/1635528486 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a402cdb0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.112+0000 7ff9b11c5640 1 -- 172.21.15.45:0/1635528486 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4034660 con 0x7ff9ac161ba0 2023-11-28T20:54:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.112+0000 7ff9b31c9640 1 -- 172.21.15.45:0/1635528486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 msgr2=0x7ff9ac161ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.112+0000 7ff9b31c9640 1 --2- 172.21.15.45:0/1635528486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 0x7ff9ac161ff0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009d90 tx=0x7ff9a402bea0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.112+0000 7ff9b31c9640 1 -- 172.21.15.45:0/1635528486 shutdown_connections 2023-11-28T20:54:25.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.112+0000 7ff9b31c9640 1 --2- 172.21.15.45:0/1635528486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 0x7ff9ac161ff0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.112+0000 7ff9b31c9640 1 --2- 172.21.15.45:0/1635528486 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 0x7ff9ac0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.112+0000 7ff9b31c9640 1 -- 172.21.15.45:0/1635528486 >> 172.21.15.45:0/1635528486 conn(0x7ff9ac0cec30 msgr2=0x7ff9ac0cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:25.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.112+0000 7ff9b31c9640 1 -- 172.21.15.45:0/1635528486 shutdown_connections 2023-11-28T20:54:25.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.113+0000 7ff9b31c9640 1 -- 172.21.15.45:0/1635528486 wait complete. 2023-11-28T20:54:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.114+0000 7ff9b31c9640 1 Processor -- start 2023-11-28T20:54:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.115+0000 7ff9b31c9640 1 -- start start 2023-11-28T20:54:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.115+0000 7ff9b31c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 0x7ff9ac1652e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.115+0000 7ff9b31c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac06c6e0 0x7ff9ac1f68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b31c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 0x7ff9ac0c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b31c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac164240 con 0x7ff9ac161ba0 2023-11-28T20:54:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b31c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac164520 con 0x7ff9ac06c6e0 2023-11-28T20:54:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b31c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac164800 con 0x7ff9ac06c300 2023-11-28T20:54:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b29c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 0x7ff9ac0c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b29c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 0x7ff9ac0c0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48202/0 (socket says 172.21.15.45:48202) 2023-11-28T20:54:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b29c8640 1 -- 172.21.15.45:0/2912780099 learned_addr learned my addr 172.21.15.45:0/2912780099 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b21c7640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 0x7ff9ac1652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b19c6640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac06c6e0 0x7ff9ac1f68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b29c8640 1 -- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 msgr2=0x7ff9ac1652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b29c8640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 0x7ff9ac1652e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b29c8640 1 -- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac06c6e0 msgr2=0x7ff9ac1f68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b29c8640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac06c6e0 0x7ff9ac1f68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.116+0000 7ff9b29c8640 1 -- 172.21.15.45:0/2912780099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a4009a70 con 0x7ff9ac161ba0 2023-11-28T20:54:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.117+0000 7ff9b29c8640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 0x7ff9ac0c0dd0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009d90 tx=0x7ff9a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.117+0000 7ff9a2ffd640 1 -- 172.21.15.45:0/2912780099 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4040650 con 0x7ff9ac161ba0 2023-11-28T20:54:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.117+0000 7ff9a2ffd640 1 -- 172.21.15.45:0/2912780099 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a40407d0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.117+0000 7ff9a2ffd640 1 -- 172.21.15.45:0/2912780099 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4040b80 con 0x7ff9ac161ba0 2023-11-28T20:54:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.117+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac164ae0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.117+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac0c0190 con 0x7ff9ac161ba0 2023-11-28T20:54:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.119+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac0733d0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.120+0000 7ff9a2ffd640 1 -- 172.21.15.45:0/2912780099 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff9a4040ce0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.121+0000 7ff9a2ffd640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff99006af60 0x7ff99006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.121+0000 7ff9a2ffd640 1 -- 172.21.15.45:0/2912780099 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5575+0+0 (secure 0 0 0) 0x7ff9a40ac9c0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.122+0000 7ff9b21c7640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff99006af60 0x7ff99006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.123+0000 7ff9b21c7640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff99006af60 0x7ff99006d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8006fd0 tx=0x7ff9a8008040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:25.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.125+0000 7ff9a2ffd640 1 -- 172.21.15.45:0/2912780099 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9a40779c0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.318+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff9ac0b4bb0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.875+0000 7ff9a2ffd640 1 -- 172.21.15.45:0/2912780099 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v146) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff9a407afe0 con 0x7ff9ac161ba0 2023-11-28T20:54:25.878 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T20:54:25.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff99006af60 msgr2=0x7ff99006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:25.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff99006af60 0x7ff99006d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8006fd0 tx=0x7ff9a8008040 comp rx=0 tx=0).stop 2023-11-28T20:54:25.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 msgr2=0x7ff9ac0c0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:25.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 0x7ff9ac0c0dd0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009d90 tx=0x7ff9a4004a40 comp rx=0 tx=0).stop 2023-11-28T20:54:25.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 shutdown_connections 2023-11-28T20:54:25.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7ff99006af60 0x7ff99006d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ac161ba0 0x7ff9ac0c0dd0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac06c6e0 0x7ff9ac1f68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 --2- 172.21.15.45:0/2912780099 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff9ac06c300 0x7ff9ac1652e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:25.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 >> 172.21.15.45:0/2912780099 conn(0x7ff9ac0cec30 msgr2=0x7ff9ac06ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:25.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 shutdown_connections 2023-11-28T20:54:25.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:25.879+0000 7ff9b31c9640 1 -- 172.21.15.45:0/2912780099 wait complete. 2023-11-28T20:54:25.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T20:54:26.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.108+0000 7f51e8cda640 1 Processor -- start 2023-11-28T20:54:26.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.108+0000 7f51e8cda640 1 -- start start 2023-11-28T20:54:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.109+0000 7f51e8cda640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f51e406c350 0x7f51e415f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.109+0000 7f51e8cda640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e415ecd0 con 0x7f51e406ac10 2023-11-28T20:54:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.110+0000 7f51e8cda640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e415f290 con 0x7f51e415e840 2023-11-28T20:54:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.110+0000 7f51e8cda640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e4072020 con 0x7f51e406c350 2023-11-28T20:54:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.110+0000 7f51e2d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f51e406c350 0x7f51e415f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.110+0000 7f51dbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f51e406ac10 0x7f51e40cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42454/0 (socket says 172.21.15.45:42454) 2023-11-28T20:54:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.110+0000 7f51dbfff640 1 -- 172.21.15.45:0/3334320095 learned_addr learned my addr 172.21.15.45:0/3334320095 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:26.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.110+0000 7f51e2d9d640 1 -- 172.21.15.45:0/3334320095 >> v1:172.21.15.190:6789/0 conn(0x7f51e415e840 legacy=0x7f51e406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:26.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.110+0000 7f51e2d9d640 1 -- 172.21.15.45:0/3334320095 >> v1:172.21.15.45:6789/0 conn(0x7f51e406ac10 legacy=0x7f51e40cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:26.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.110+0000 7f51e2d9d640 1 -- 172.21.15.45:0/3334320095 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e4072300 con 0x7f51e406c350 2023-11-28T20:54:26.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.112+0000 7f51e2d9d640 1 --2- 172.21.15.45:0/3334320095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e406c350 0x7f51e415f720 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009d90 tx=0x7f51dc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba3e7c41cfd59f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.112+0000 7f51e1d9b640 1 -- 172.21.15.45:0/3334320095 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc02cc60 con 0x7f51e406c350 2023-11-28T20:54:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.112+0000 7f51e1d9b640 1 -- 172.21.15.45:0/3334320095 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc02cdc0 con 0x7f51e406c350 2023-11-28T20:54:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.112+0000 7f51e1d9b640 1 -- 172.21.15.45:0/3334320095 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc034660 con 0x7f51e406c350 2023-11-28T20:54:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.113+0000 7f51e8cda640 1 -- 172.21.15.45:0/3334320095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e406c350 msgr2=0x7f51e415f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.113+0000 7f51e8cda640 1 --2- 172.21.15.45:0/3334320095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e406c350 0x7f51e415f720 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009d90 tx=0x7f51dc02beb0 comp rx=0 tx=0).stop 2023-11-28T20:54:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.113+0000 7f51e8cda640 1 -- 172.21.15.45:0/3334320095 shutdown_connections 2023-11-28T20:54:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.113+0000 7f51e8cda640 1 --2- 172.21.15.45:0/3334320095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e406c350 0x7f51e415f720 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.113+0000 7f51e8cda640 1 -- 172.21.15.45:0/3334320095 >> 172.21.15.45:0/3334320095 conn(0x7f51e40ced60 msgr2=0x7f51e40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.113+0000 7f51e8cda640 1 -- 172.21.15.45:0/3334320095 shutdown_connections 2023-11-28T20:54:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.113+0000 7f51e8cda640 1 -- 172.21.15.45:0/3334320095 wait complete. 2023-11-28T20:54:26.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.115+0000 7f51e8cda640 1 Processor -- start 2023-11-28T20:54:26.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.115+0000 7f51e8cda640 1 -- start start 2023-11-28T20:54:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51e8cda640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51e406ac10 0x7f51e40c5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51e8cda640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51e406c350 0x7f51e40c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51e8cda640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e415e840 0x7f51e41f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51e8cda640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e40c5100 con 0x7f51e415e840 2023-11-28T20:54:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51e8cda640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e40c44f0 con 0x7f51e406c350 2023-11-28T20:54:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51e8cda640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e40c47d0 con 0x7f51e406ac10 2023-11-28T20:54:26.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51e2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e415e840 0x7f51e41f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:26.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51e2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e415e840 0x7f51e41f3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48218/0 (socket says 172.21.15.45:48218) 2023-11-28T20:54:26.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51e2d9d640 1 -- 172.21.15.45:0/2421922570 learned_addr learned my addr 172.21.15.45:0/2421922570 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:26.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.116+0000 7f51dbfff640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51e406c350 0x7f51e40c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:26.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51e259c640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51e406ac10 0x7f51e40c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:26.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51e2d9d640 1 -- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51e406ac10 msgr2=0x7f51e40c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:26.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51e2d9d640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51e406ac10 0x7f51e40c5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:26.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51e2d9d640 1 -- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51e406c350 msgr2=0x7f51e40c3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:26.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51e2d9d640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51e406c350 0x7f51e40c3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:26.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51e2d9d640 1 -- 172.21.15.45:0/2421922570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc009a70 con 0x7f51e415e840 2023-11-28T20:54:26.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51dbfff640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51e406c350 0x7f51e40c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:26.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51e2d9d640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e415e840 0x7f51e41f3670 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009d90 tx=0x7f51dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:26.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51daffd640 1 -- 172.21.15.45:0/2421922570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc03e610 con 0x7f51e415e840 2023-11-28T20:54:26.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51daffd640 1 -- 172.21.15.45:0/2421922570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc03e770 con 0x7f51e415e840 2023-11-28T20:54:26.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e40c4ab0 con 0x7f51e415e840 2023-11-28T20:54:26.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.117+0000 7f51daffd640 1 -- 172.21.15.45:0/2421922570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc03eb60 con 0x7f51e415e840 2023-11-28T20:54:26.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.118+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e40caa80 con 0x7f51e415e840 2023-11-28T20:54:26.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.119+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e40c52b0 con 0x7f51e415e840 2023-11-28T20:54:26.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.120+0000 7f51daffd640 1 -- 172.21.15.45:0/2421922570 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f51dc03b050 con 0x7f51e415e840 2023-11-28T20:54:26.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.121+0000 7f51daffd640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51c406af60 0x7f51c406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:26.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.121+0000 7f51daffd640 1 -- 172.21.15.45:0/2421922570 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f51dc0ac570 con 0x7f51e415e840 2023-11-28T20:54:26.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.122+0000 7f51e259c640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51c406af60 0x7f51c406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:26.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.123+0000 7f51e259c640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51c406af60 0x7f51c406d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51d400a750 tx=0x7f51d4008040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:26.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.125+0000 7f51daffd640 1 -- 172.21.15.45:0/2421922570 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51dc0774b0 con 0x7f51e415e840 2023-11-28T20:54:26.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.352+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f51e40c52b0 con 0x7f51e415e840 2023-11-28T20:54:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.898+0000 7f51daffd640 1 -- 172.21.15.45:0/2421922570 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v63) v1 ==== 174+0+0 (secure 0 0 0) 0x7f51e40c52b0 con 0x7f51e415e840 2023-11-28T20:54:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 17 and data pool 18 2023-11-28T20:54:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.907+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51c406af60 msgr2=0x7f51c406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.907+0000 7f51e8cda640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51c406af60 0x7f51c406d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51d400a750 tx=0x7f51d4008040 comp rx=0 tx=0).stop 2023-11-28T20:54:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.907+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e415e840 msgr2=0x7f51e41f3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.907+0000 7f51e8cda640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e415e840 0x7f51e41f3670 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009d90 tx=0x7f51dc004a40 comp rx=0 tx=0).stop 2023-11-28T20:54:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.908+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 shutdown_connections 2023-11-28T20:54:26.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.908+0000 7f51e8cda640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51c406af60 0x7f51c406d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.908+0000 7f51e8cda640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e415e840 0x7f51e41f3670 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.908+0000 7f51e8cda640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51e406c350 0x7f51e40c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.908+0000 7f51e8cda640 1 --2- 172.21.15.45:0/2421922570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51e406ac10 0x7f51e40c5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.908+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 >> 172.21.15.45:0/2421922570 conn(0x7f51e40ced60 msgr2=0x7f51e40ad620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.909+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 shutdown_connections 2023-11-28T20:54:26.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:26.909+0000 7f51e8cda640 1 -- 172.21.15.45:0/2421922570 wait complete. 2023-11-28T20:54:26.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:27.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.123+0000 7fdd5fc6b640 1 Processor -- start 2023-11-28T20:54:27.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.124+0000 7fdd5fc6b640 1 -- start start 2023-11-28T20:54:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.125+0000 7fdd5fc6b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdd5815b850 0x7fdd580b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.125+0000 7fdd5fc6b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd580b1fa0 0x7fdd580b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.125+0000 7fdd5fc6b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5815bda0 con 0x7fdd580b35b0 2023-11-28T20:54:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.125+0000 7fdd5fc6b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5815ee40 con 0x7fdd5815b850 2023-11-28T20:54:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.125+0000 7fdd5fc6b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5815c080 con 0x7fdd580b1fa0 2023-11-28T20:54:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5da07640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdd580b35b0 0x7fdd5815ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42462/0 (socket says 172.21.15.45:42462) 2023-11-28T20:54:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5da07640 1 -- 172.21.15.45:0/2026749375 learned_addr learned my addr 172.21.15.45:0/2026749375 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5d206640 1 --2- 172.21.15.45:0/2026749375 >> v2:172.21.15.45:3300/0 conn(0x7fdd5815b850 0x7fdd580b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5ca05640 1 -- 172.21.15.45:0/2026749375 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230487571 0 0) 0x7fdd5815bda0 con 0x7fdd580b35b0 2023-11-28T20:54:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5e208640 1 --2- 172.21.15.45:0/2026749375 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd580b1fa0 0x7fdd580b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5ca05640 1 -- 172.21.15.45:0/2026749375 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd44003520 con 0x7fdd580b35b0 2023-11-28T20:54:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5d206640 1 -- 172.21.15.45:0/2026749375 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd580b1fa0 msgr2=0x7fdd580b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5d206640 1 --2- 172.21.15.45:0/2026749375 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd580b1fa0 0x7fdd580b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5d206640 1 -- 172.21.15.45:0/2026749375 >> v1:172.21.15.45:6789/0 conn(0x7fdd580b35b0 legacy=0x7fdd5815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.126+0000 7fdd5d206640 1 -- 172.21.15.45:0/2026749375 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5815c360 con 0x7fdd5815b850 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.128+0000 7fdd5d206640 1 --2- 172.21.15.45:0/2026749375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5815b850 0x7fdd580b0c30 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fdd480097a0 tx=0x7fdd4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602083fb6cf14266 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.128+0000 7fdd5ca05640 1 -- 172.21.15.45:0/2026749375 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48031830 con 0x7fdd5815b850 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.128+0000 7fdd5ca05640 1 -- 172.21.15.45:0/2026749375 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd48031990 con 0x7fdd5815b850 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.128+0000 7fdd5ca05640 1 -- 172.21.15.45:0/2026749375 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48031d20 con 0x7fdd5815b850 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.128+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/2026749375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5815b850 msgr2=0x7fdd580b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.128+0000 7fdd5fc6b640 1 --2- 172.21.15.45:0/2026749375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5815b850 0x7fdd580b0c30 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fdd480097a0 tx=0x7fdd4802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.129+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/2026749375 shutdown_connections 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.129+0000 7fdd5fc6b640 1 --2- 172.21.15.45:0/2026749375 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd580b1fa0 0x7fdd580b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.129+0000 7fdd5fc6b640 1 --2- 172.21.15.45:0/2026749375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd5815b850 0x7fdd580b0c30 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.129+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/2026749375 >> 172.21.15.45:0/2026749375 conn(0x7fdd5806a500 msgr2=0x7fdd5806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.129+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/2026749375 shutdown_connections 2023-11-28T20:54:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.129+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/2026749375 wait complete. 2023-11-28T20:54:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.131+0000 7fdd5fc6b640 1 Processor -- start 2023-11-28T20:54:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.131+0000 7fdd5fc6b640 1 -- start start 2023-11-28T20:54:27.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.132+0000 7fdd5fc6b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd580b1fa0 0x7fdd580ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.132+0000 7fdd5fc6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd580b35b0 0x7fdd58070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.132+0000 7fdd5fc6b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd5815b850 0x7fdd580711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.132+0000 7fdd5fc6b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580ade00 con 0x7fdd580b35b0 2023-11-28T20:54:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.132+0000 7fdd5fc6b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580cc7a0 con 0x7fdd580b1fa0 2023-11-28T20:54:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.132+0000 7fdd5fc6b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd581e52f0 con 0x7fdd5815b850 2023-11-28T20:54:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.132+0000 7fdd5d206640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd580b35b0 0x7fdd58070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.132+0000 7fdd5d206640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd580b35b0 0x7fdd58070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48230/0 (socket says 172.21.15.45:48230) 2023-11-28T20:54:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.132+0000 7fdd5d206640 1 -- 172.21.15.45:0/1990680222 learned_addr learned my addr 172.21.15.45:0/1990680222 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd5da07640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd580b1fa0 0x7fdd580ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd5d206640 1 -- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd5815b850 msgr2=0x7fdd580711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd5d206640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd5815b850 0x7fdd580711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd5d206640 1 -- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd580b1fa0 msgr2=0x7fdd580ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd5d206640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd580b1fa0 0x7fdd580ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd5d206640 1 -- 172.21.15.45:0/1990680222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd48009400 con 0x7fdd580b35b0 2023-11-28T20:54:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd5da07640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd580b1fa0 0x7fdd580ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd5d206640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd580b35b0 0x7fdd58070c70 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fdd48000c00 tx=0x7fdd480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd4e7fc640 1 -- 172.21.15.45:0/1990680222 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4803a8f0 con 0x7fdd580b35b0 2023-11-28T20:54:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd4e7fc640 1 -- 172.21.15.45:0/1990680222 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4803aa70 con 0x7fdd580b35b0 2023-11-28T20:54:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd580acb00 con 0x7fdd580b35b0 2023-11-28T20:54:27.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.133+0000 7fdd4e7fc640 1 -- 172.21.15.45:0/1990680222 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4803ae20 con 0x7fdd580b35b0 2023-11-28T20:54:27.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.134+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd580ad0c0 con 0x7fdd580b35b0 2023-11-28T20:54:27.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.135+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd580733d0 con 0x7fdd580b35b0 2023-11-28T20:54:27.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.136+0000 7fdd4e7fc640 1 -- 172.21.15.45:0/1990680222 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fdd4804d450 con 0x7fdd580b35b0 2023-11-28T20:54:27.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.138+0000 7fdd4e7fc640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fdd3c06af60 0x7fdd3c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.138+0000 7fdd4e7fc640 1 -- 172.21.15.45:0/1990680222 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd480aba80 con 0x7fdd580b35b0 2023-11-28T20:54:27.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.138+0000 7fdd5da07640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fdd3c06af60 0x7fdd3c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.139+0000 7fdd5da07640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fdd3c06af60 0x7fdd3c06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdd54001560 tx=0x7fdd54009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:27.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.142+0000 7fdd4e7fc640 1 -- 172.21.15.45:0/1990680222 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd480768c0 con 0x7fdd580b35b0 2023-11-28T20:54:27.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-11-28T20:54:27.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.335+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd580cd3b0 con 0x7fdd580b35b0 2023-11-28T20:54:27.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.338+0000 7fdd4e7fc640 1 -- 172.21.15.45:0/1990680222 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+16829 (secure 0 0 0) 0x7fdd48035020 con 0x7fdd580b35b0 2023-11-28T20:54:27.342 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:27.343 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":147,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:26.354862+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:54:24.325054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-11-28T20:54:25.374707+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:27.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fdd3c06af60 msgr2=0x7fdd3c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fdd3c06af60 0x7fdd3c06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdd54001560 tx=0x7fdd54009040 comp rx=0 tx=0).stop 2023-11-28T20:54:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd580b35b0 msgr2=0x7fdd58070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd580b35b0 0x7fdd58070c70 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fdd48000c00 tx=0x7fdd480046c0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 shutdown_connections 2023-11-28T20:54:27.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fdd3c06af60 0x7fdd3c06d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd5815b850 0x7fdd580711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd580b35b0 0x7fdd58070c70 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 --2- 172.21.15.45:0/1990680222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd580b1fa0 0x7fdd580ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 >> 172.21.15.45:0/1990680222 conn(0x7fdd5806a500 msgr2=0x7fdd580cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 shutdown_connections 2023-11-28T20:54:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.341+0000 7fdd5fc6b640 1 -- 172.21.15.45:0/1990680222 wait complete. 2023-11-28T20:54:27.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.587+0000 7f6c7b59e640 1 Processor -- start 2023-11-28T20:54:27.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.587+0000 7f6c7b59e640 1 -- start start 2023-11-28T20:54:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.589+0000 7f6c7b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6c7c0ae540 0x7f6c7c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.589+0000 7f6c7b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 0x7f6c7c0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.589+0000 7f6c7b59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c15c3c0 con 0x7f6c7c155780 2023-11-28T20:54:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.589+0000 7f6c7b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7c0ceac0 con 0x7f6c7c0ae540 2023-11-28T20:54:27.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.589+0000 7f6c7b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c155b60 con 0x7f6c7c0ce630 2023-11-28T20:54:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.590+0000 7f6c73fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6c7c0ae540 0x7f6c7c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.590+0000 7f6c7a59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6c7c155780 0x7f6c7c0abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42476/0 (socket says 172.21.15.45:42476) 2023-11-28T20:54:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.590+0000 7f6c7a59c640 1 -- 172.21.15.45:0/4292001905 learned_addr learned my addr 172.21.15.45:0/4292001905 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:27.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.590+0000 7f6c7ad9d640 1 --2- 172.21.15.45:0/4292001905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 0x7f6c7c0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.590+0000 7f6c79d9b640 1 -- 172.21.15.45:0/4292001905 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940560547 0 0) 0x7f6c7c15c3c0 con 0x7f6c7c155780 2023-11-28T20:54:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.590+0000 7f6c79d9b640 1 -- 172.21.15.45:0/4292001905 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c68003520 con 0x7f6c7c155780 2023-11-28T20:54:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.591+0000 7f6c7ad9d640 1 -- 172.21.15.45:0/4292001905 >> v2:172.21.15.190:3300/0 conn(0x7f6c7c0ae540 msgr2=0x7f6c7c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.591+0000 7f6c7ad9d640 1 --2- 172.21.15.45:0/4292001905 >> v2:172.21.15.190:3300/0 conn(0x7f6c7c0ae540 0x7f6c7c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.591+0000 7f6c7ad9d640 1 -- 172.21.15.45:0/4292001905 >> v1:172.21.15.45:6789/0 conn(0x7f6c7c155780 legacy=0x7f6c7c0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.591+0000 7f6c79d9b640 1 -- 172.21.15.45:0/4292001905 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1028229605 0 0) 0x7f6c68003520 con 0x7f6c7c155780 2023-11-28T20:54:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.591+0000 7f6c7ad9d640 1 -- 172.21.15.45:0/4292001905 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c15ba90 con 0x7f6c7c0ce630 2023-11-28T20:54:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.592+0000 7f6c7ad9d640 1 --2- 172.21.15.45:0/4292001905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 0x7f6c7c0b20a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009d90 tx=0x7f6c7402bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5efaf87a61b3eaef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c79d9b640 1 -- 172.21.15.45:0/4292001905 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c001670 con 0x7f6c7c0ce630 2023-11-28T20:54:27.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c79d9b640 1 -- 172.21.15.45:0/4292001905 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7402c8a0 con 0x7f6c7c0ce630 2023-11-28T20:54:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c79d9b640 1 -- 172.21.15.45:0/4292001905 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7402cc10 con 0x7f6c7c0ce630 2023-11-28T20:54:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c7b59e640 1 -- 172.21.15.45:0/4292001905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 msgr2=0x7f6c7c0b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c7b59e640 1 --2- 172.21.15.45:0/4292001905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 0x7f6c7c0b20a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009d90 tx=0x7f6c7402bd70 comp rx=0 tx=0).stop 2023-11-28T20:54:27.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c7b59e640 1 -- 172.21.15.45:0/4292001905 shutdown_connections 2023-11-28T20:54:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c7b59e640 1 --2- 172.21.15.45:0/4292001905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 0x7f6c7c0b20a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c7b59e640 1 --2- 172.21.15.45:0/4292001905 >> v2:172.21.15.190:3300/0 conn(0x7f6c7c0ae540 0x7f6c7c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c7b59e640 1 -- 172.21.15.45:0/4292001905 >> 172.21.15.45:0/4292001905 conn(0x7f6c7c06ab30 msgr2=0x7f6c7c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.593+0000 7f6c7b59e640 1 -- 172.21.15.45:0/4292001905 shutdown_connections 2023-11-28T20:54:27.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.594+0000 7f6c7b59e640 1 -- 172.21.15.45:0/4292001905 wait complete. 2023-11-28T20:54:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.595+0000 7f6c7b59e640 1 Processor -- start 2023-11-28T20:54:27.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.596+0000 7f6c7b59e640 1 -- start start 2023-11-28T20:54:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c7c0ae540 0x7f6c7c1f4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 0x7f6c7c14b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c7c155780 0x7f6c7c0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c14be00 con 0x7f6c7c155780 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0b03e0 con 0x7f6c7c0ae540 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c09b980 con 0x7f6c7c0ce630 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c7c155780 0x7f6c7c0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c7c155780 0x7f6c7c0afea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48242/0 (socket says 172.21.15.45:48242) 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7ad9d640 1 -- 172.21.15.45:0/2099300237 learned_addr learned my addr 172.21.15.45:0/2099300237 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c73fff640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 0x7f6c7c14b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7a59c640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c7c0ae540 0x7f6c7c1f4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7ad9d640 1 -- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 msgr2=0x7f6c7c14b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7ad9d640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 0x7f6c7c14b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7ad9d640 1 -- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c7c0ae540 msgr2=0x7f6c7c1f4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7ad9d640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c7c0ae540 0x7f6c7c1f4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.597+0000 7f6c7ad9d640 1 -- 172.21.15.45:0/2099300237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74009a70 con 0x7f6c7c155780 2023-11-28T20:54:27.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.598+0000 7f6c7ad9d640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c7c155780 0x7f6c7c0afea0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009eb0 tx=0x7f6c74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:27.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.598+0000 7f6c72ffd640 1 -- 172.21.15.45:0/2099300237 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74007000 con 0x7f6c7c155780 2023-11-28T20:54:27.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.598+0000 7f6c72ffd640 1 -- 172.21.15.45:0/2099300237 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c74009450 con 0x7f6c7c155780 2023-11-28T20:54:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.598+0000 7f6c72ffd640 1 -- 172.21.15.45:0/2099300237 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74004490 con 0x7f6c7c155780 2023-11-28T20:54:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.598+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c09bc60 con 0x7f6c7c155780 2023-11-28T20:54:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.598+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c09c220 con 0x7f6c7c155780 2023-11-28T20:54:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.599+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c14bf80 con 0x7f6c7c155780 2023-11-28T20:54:27.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.601+0000 7f6c72ffd640 1 -- 172.21.15.45:0/2099300237 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6c7403b070 con 0x7f6c7c155780 2023-11-28T20:54:27.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.602+0000 7f6c72ffd640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6c5c06af60 0x7f6c5c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:27.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.602+0000 7f6c72ffd640 1 -- 172.21.15.45:0/2099300237 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6c740ac010 con 0x7f6c7c155780 2023-11-28T20:54:27.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.602+0000 7f6c7a59c640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6c5c06af60 0x7f6c5c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:27.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.604+0000 7f6c7a59c640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6c5c06af60 0x7f6c5c06d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c007a00 tx=0x7f6c6c009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:27.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.605+0000 7f6c72ffd640 1 -- 172.21.15.45:0/2099300237 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c74076f30 con 0x7f6c7c155780 2023-11-28T20:54:27.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.799+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c7c0b1030 con 0x7f6c7c155780 2023-11-28T20:54:27.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.802+0000 7f6c72ffd640 1 -- 172.21.15.45:0/2099300237 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+16829 (secure 0 0 0) 0x7f6c7407a550 con 0x7f6c7c155780 2023-11-28T20:54:27.804 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:27.805 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":147,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:26.354862+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:54:24.325054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-11-28T20:54:25.374707+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:27.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.804+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6c5c06af60 msgr2=0x7f6c5c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.804+0000 7f6c7b59e640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6c5c06af60 0x7f6c5c06d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c007a00 tx=0x7f6c6c009040 comp rx=0 tx=0).stop 2023-11-28T20:54:27.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.804+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c7c155780 msgr2=0x7f6c7c0afea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:27.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.804+0000 7f6c7b59e640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c7c155780 0x7f6c7c0afea0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009eb0 tx=0x7f6c74004a40 comp rx=0 tx=0).stop 2023-11-28T20:54:27.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.805+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 shutdown_connections 2023-11-28T20:54:27.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.805+0000 7f6c7b59e640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f6c5c06af60 0x7f6c5c06d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.805+0000 7f6c7b59e640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c7c155780 0x7f6c7c0afea0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.805+0000 7f6c7b59e640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6c7c0ce630 0x7f6c7c14b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.805+0000 7f6c7b59e640 1 --2- 172.21.15.45:0/2099300237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c7c0ae540 0x7f6c7c1f4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:27.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.805+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 >> 172.21.15.45:0/2099300237 conn(0x7f6c7c06ab30 msgr2=0x7f6c7c0ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:27.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.805+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 shutdown_connections 2023-11-28T20:54:27.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:27.805+0000 7f6c7b59e640 1 -- 172.21.15.45:0/2099300237 wait complete. 2023-11-28T20:54:27.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T20:54:28.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.048+0000 7f9673fff640 1 Processor -- start 2023-11-28T20:54:28.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.048+0000 7f9673fff640 1 -- start start 2023-11-28T20:54:28.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.050+0000 7f9673fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740afd40 0x7f96740cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:28.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.050+0000 7f9673fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740cd270 con 0x7f96740cbd00 2023-11-28T20:54:28.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.050+0000 7f9673fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740cd530 con 0x7f967415e860 2023-11-28T20:54:28.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.050+0000 7f9673fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96740adcb0 con 0x7f96740afd40 2023-11-28T20:54:28.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.050+0000 7f96737fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f96740cbd00 0x7f96740cc170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42488/0 (socket says 172.21.15.45:42488) 2023-11-28T20:54:28.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.050+0000 7f96737fe640 1 -- 172.21.15.45:0/3985114842 learned_addr learned my addr 172.21.15.45:0/3985114842 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:28.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f96727fc640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740afd40 0x7f96740cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:28.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645222585 0 0) 0x7f96740cd270 con 0x7f96740cbd00 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9660003520 con 0x7f96740cbd00 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3049380135 0 0) 0x7f9660003520 con 0x7f96740cbd00 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740afd40 msgr2=0x7f96740cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740afd40 0x7f96740cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 >> v1:172.21.15.190:6789/0 conn(0x7f967415e860 legacy=0x7f96740cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96740cfa70 con 0x7f96740cbd00 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f966c001670 con 0x7f96740cbd00 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 >> v1:172.21.15.45:6789/0 conn(0x7f96740cbd00 legacy=0x7f96740cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740afd40 0x7f96600039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660003f20 0x7f96600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f9660005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:28.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966000bb10 con 0x7f9660004d00 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966000bc50 con 0x7f9660003f20 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.051+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966000be10 con 0x7f96740afd40 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.052+0000 7f96737fe640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f9660005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.052+0000 7f96727fc640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660003f20 0x7f96600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.052+0000 7f9672ffd640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740afd40 0x7f96600039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.052+0000 7f96737fe640 1 -- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740afd40 msgr2=0x7f96600039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.052+0000 7f96737fe640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740afd40 0x7f96600039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.052+0000 7f96737fe640 1 -- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660003f20 msgr2=0x7f96600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.052+0000 7f96737fe640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660003f20 0x7f96600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.052+0000 7f96737fe640 1 -- 172.21.15.45:0/3985114842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660003520 con 0x7f9660004d00 2023-11-28T20:54:28.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.053+0000 7f96737fe640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f9660005180 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f966c00c240 tx=0x7f96600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a581179f6669e7ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:28.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c005ad0 con 0x7f9660004d00 2023-11-28T20:54:28.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c005c30 con 0x7f9660004d00 2023-11-28T20:54:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f9671ffb640 1 -- 172.21.15.45:0/3985114842 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c02c8b0 con 0x7f9660004d00 2023-11-28T20:54:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f9673fff640 1 -- 172.21.15.45:0/3985114842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 msgr2=0x7f9660005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f9673fff640 1 --2- 172.21.15.45:0/3985114842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f9660005180 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f966c00c240 tx=0x7f96600057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f96737fe640 1 -- 172.21.15.45:0/3985114842 reap_dead start 2023-11-28T20:54:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f9673fff640 1 -- 172.21.15.45:0/3985114842 shutdown_connections 2023-11-28T20:54:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f9673fff640 1 -- 172.21.15.45:0/3985114842 >> 172.21.15.45:0/3985114842 conn(0x7f96740b29d0 msgr2=0x7f96740b2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f9673fff640 1 -- 172.21.15.45:0/3985114842 shutdown_connections 2023-11-28T20:54:28.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.054+0000 7f9673fff640 1 -- 172.21.15.45:0/3985114842 wait complete. 2023-11-28T20:54:28.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.056+0000 7f9673fff640 1 Processor -- start 2023-11-28T20:54:28.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.056+0000 7f9673fff640 1 -- start start 2023-11-28T20:54:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.057+0000 7f9673fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f96740b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.057+0000 7f9673fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740b60d0 0x7f96740c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.057+0000 7f9673fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96741f25e0 0x7f96740ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.057+0000 7f9673fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96740c5fb0 con 0x7f9660004d00 2023-11-28T20:54:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.057+0000 7f9673fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96740b68c0 con 0x7f96741f25e0 2023-11-28T20:54:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.057+0000 7f9673fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967406a840 con 0x7f96740b60d0 2023-11-28T20:54:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.057+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f96740b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:28.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f96740b3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48258/0 (socket says 172.21.15.45:48258) 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f9672ffd640 1 -- 172.21.15.45:0/3034829417 learned_addr learned my addr 172.21.15.45:0/3034829417 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f96727fc640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740b60d0 0x7f96740c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f9672ffd640 1 -- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740b60d0 msgr2=0x7f96740c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f9672ffd640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740b60d0 0x7f96740c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f9672ffd640 1 -- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96741f25e0 msgr2=0x7f96740ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f96737fe640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96741f25e0 0x7f96740ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f9672ffd640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96741f25e0 0x7f96740ae5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f9672ffd640 1 -- 172.21.15.45:0/3034829417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967406ab20 con 0x7f9660004d00 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f9672ffd640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f96740b3cf0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f966400d570 tx=0x7f966400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.058+0000 7f966b7fe640 1 -- 172.21.15.45:0/3034829417 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664005480 con 0x7f9660004d00 2023-11-28T20:54:28.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.059+0000 7f966b7fe640 1 -- 172.21.15.45:0/3034829417 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9664003610 con 0x7f9660004d00 2023-11-28T20:54:28.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.059+0000 7f966b7fe640 1 -- 172.21.15.45:0/3034829417 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664003980 con 0x7f9660004d00 2023-11-28T20:54:28.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.059+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967406ae00 con 0x7f9660004d00 2023-11-28T20:54:28.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.059+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967406b3c0 con 0x7f9660004d00 2023-11-28T20:54:28.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.060+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967406b580 con 0x7f9660004d00 2023-11-28T20:54:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.061+0000 7f966b7fe640 1 -- 172.21.15.45:0/3034829417 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9664003ae0 con 0x7f9660004d00 2023-11-28T20:54:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.062+0000 7f966b7fe640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f965406af60 0x7f965406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.062+0000 7f966b7fe640 1 -- 172.21.15.45:0/3034829417 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f96640899d0 con 0x7f9660004d00 2023-11-28T20:54:28.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.062+0000 7f96727fc640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f965406af60 0x7f965406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:28.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.063+0000 7f96727fc640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f965406af60 0x7f965406d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f96740c79d0 tx=0x7f965c00b040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:28.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.066+0000 7f966b7fe640 1 -- 172.21.15.45:0/3034829417 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9664054660 con 0x7f9660004d00 2023-11-28T20:54:28.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.299+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f967409f030 con 0x7f9660004d00 2023-11-28T20:54:28.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.915+0000 7f966b7fe640 1 -- 172.21.15.45:0/3034829417 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v66) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9664001230 con 0x7f9660004d00 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f965406af60 msgr2=0x7f965406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f965406af60 0x7f965406d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f96740c79d0 tx=0x7f965c00b040 comp rx=0 tx=0).stop 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 msgr2=0x7f96740b3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f96740b3cf0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f966400d570 tx=0x7f966400da30 comp rx=0 tx=0).stop 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 shutdown_connections 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f965406af60 0x7f965406d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96741f25e0 0x7f96740ae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96740b60d0 0x7f96740c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 --2- 172.21.15.45:0/3034829417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9660004d00 0x7f96740b3cf0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.920+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 >> 172.21.15.45:0/3034829417 conn(0x7f96740b29d0 msgr2=0x7f96740d0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.921+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 shutdown_connections 2023-11-28T20:54:28.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:28.921+0000 7f9673fff640 1 -- 172.21.15.45:0/3034829417 wait complete. 2023-11-28T20:54:28.945 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T20:54:28.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T20:54:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.176+0000 7f5211f6e640 1 Processor -- start 2023-11-28T20:54:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.176+0000 7f5211f6e640 1 -- start start 2023-11-28T20:54:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.177+0000 7f5211f6e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f520c15e840 0x7f520c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.177+0000 7f5211f6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c15f720 0x7f520c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.177+0000 7f5211f6e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f520c06a500 0x7f520c06a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.177+0000 7f5211f6e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520c071ce0 con 0x7f520c06a500 2023-11-28T20:54:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.177+0000 7f5211f6e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520c06aed0 con 0x7f520c15e840 2023-11-28T20:54:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.177+0000 7f5211f6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c15ecd0 con 0x7f520c15f720 2023-11-28T20:54:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f520c06a500 0x7f520c06a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f520c06a500 0x7f520c06a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48274/0 (socket says 172.21.15.45:48274) 2023-11-28T20:54:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520bfff640 1 -- 172.21.15.45:0/269631368 learned_addr learned my addr 172.21.15.45:0/269631368 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520affd640 1 --2- 172.21.15.45:0/269631368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c15f720 0x7f520c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520b7fe640 1 --2- 172.21.15.45:0/269631368 >> v2:172.21.15.190:3300/0 conn(0x7f520c15e840 0x7f520c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520bfff640 1 -- 172.21.15.45:0/269631368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c15f720 msgr2=0x7f520c0717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520bfff640 1 --2- 172.21.15.45:0/269631368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c15f720 0x7f520c0717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520bfff640 1 -- 172.21.15.45:0/269631368 >> v2:172.21.15.190:3300/0 conn(0x7f520c15e840 msgr2=0x7f520c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520bfff640 1 --2- 172.21.15.45:0/269631368 >> v2:172.21.15.190:3300/0 conn(0x7f520c15e840 0x7f520c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.178+0000 7f520bfff640 1 -- 172.21.15.45:0/269631368 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c15f290 con 0x7f520c06a500 2023-11-28T20:54:29.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.180+0000 7f520bfff640 1 --2- 172.21.15.45:0/269631368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06a500 0x7f520c06a990 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f5204009870 tx=0x7f520402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91671564c7c13ee3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:29.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.180+0000 7f520a7fc640 1 -- 172.21.15.45:0/269631368 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204031640 con 0x7f520c06a500 2023-11-28T20:54:29.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.180+0000 7f520a7fc640 1 -- 172.21.15.45:0/269631368 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52040317a0 con 0x7f520c06a500 2023-11-28T20:54:29.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.180+0000 7f520a7fc640 1 -- 172.21.15.45:0/269631368 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204031b30 con 0x7f520c06a500 2023-11-28T20:54:29.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.180+0000 7f5211f6e640 1 -- 172.21.15.45:0/269631368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06a500 msgr2=0x7f520c06a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:29.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.180+0000 7f5211f6e640 1 --2- 172.21.15.45:0/269631368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06a500 0x7f520c06a990 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f5204009870 tx=0x7f520402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:54:29.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.181+0000 7f5211f6e640 1 -- 172.21.15.45:0/269631368 shutdown_connections 2023-11-28T20:54:29.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.181+0000 7f5211f6e640 1 --2- 172.21.15.45:0/269631368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c06a500 0x7f520c06a990 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:29.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.181+0000 7f5211f6e640 1 --2- 172.21.15.45:0/269631368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c15f720 0x7f520c0717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:29.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.181+0000 7f5211f6e640 1 --2- 172.21.15.45:0/269631368 >> v2:172.21.15.190:3300/0 conn(0x7f520c15e840 0x7f520c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:29.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.181+0000 7f5211f6e640 1 -- 172.21.15.45:0/269631368 >> 172.21.15.45:0/269631368 conn(0x7f520c0ced60 msgr2=0x7f520c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:29.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.181+0000 7f5211f6e640 1 -- 172.21.15.45:0/269631368 shutdown_connections 2023-11-28T20:54:29.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.181+0000 7f5211f6e640 1 -- 172.21.15.45:0/269631368 wait complete. 2023-11-28T20:54:29.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.183+0000 7f5211f6e640 1 Processor -- start 2023-11-28T20:54:29.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.183+0000 7f5211f6e640 1 -- start start 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.184+0000 7f5211f6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c06a500 0x7f520c1f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.184+0000 7f5211f6e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c15e840 0x7f520c0c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.184+0000 7f5211f6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c15f720 0x7f520c0c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.184+0000 7f5211f6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c14fb40 con 0x7f520c15f720 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.184+0000 7f5211f6e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c1f1db0 con 0x7f520c15e840 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.184+0000 7f5211f6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c1f1ad0 con 0x7f520c06a500 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.184+0000 7f520bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c15f720 0x7f520c0c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.184+0000 7f520bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c15f720 0x7f520c0c6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48284/0 (socket says 172.21.15.45:48284) 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.184+0000 7f520bfff640 1 -- 172.21.15.45:0/2693199690 learned_addr learned my addr 172.21.15.45:0/2693199690 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f520b7fe640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c06a500 0x7f520c1f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:29.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f520affd640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c15e840 0x7f520c0c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:29.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f520bfff640 1 -- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c06a500 msgr2=0x7f520c1f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:29.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f520bfff640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c06a500 0x7f520c1f3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:29.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f520bfff640 1 -- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c15e840 msgr2=0x7f520c0c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:29.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f520bfff640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c15e840 0x7f520c0c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:29.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f520bfff640 1 -- 172.21.15.45:0/2693199690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52040094d0 con 0x7f520c15f720 2023-11-28T20:54:29.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f520affd640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c15e840 0x7f520c0c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:29.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f520bfff640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c15f720 0x7f520c0c6370 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f5204009870 tx=0x7f52040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:29.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f5203fff640 1 -- 172.21.15.45:0/2693199690 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520403a930 con 0x7f520c15f720 2023-11-28T20:54:29.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.185+0000 7f5203fff640 1 -- 172.21.15.45:0/2693199690 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520403aab0 con 0x7f520c15f720 2023-11-28T20:54:29.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.186+0000 7f5203fff640 1 -- 172.21.15.45:0/2693199690 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520403ae60 con 0x7f520c15f720 2023-11-28T20:54:29.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.186+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c0c2a10 con 0x7f520c15f720 2023-11-28T20:54:29.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.186+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c0c2fd0 con 0x7f520c15f720 2023-11-28T20:54:29.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.187+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c15eee0 con 0x7f520c15f720 2023-11-28T20:54:29.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.188+0000 7f5203fff640 1 -- 172.21.15.45:0/2693199690 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f520402f050 con 0x7f520c15f720 2023-11-28T20:54:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.189+0000 7f5203fff640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51ec06af60 0x7f51ec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.189+0000 7f5203fff640 1 -- 172.21.15.45:0/2693199690 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f52040ab8a0 con 0x7f520c15f720 2023-11-28T20:54:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.189+0000 7f520b7fe640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51ec06af60 0x7f51ec06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.191+0000 7f520b7fe640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51ec06af60 0x7f51ec06d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f51fc006fd0 tx=0x7f51fc008040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.193+0000 7f5203fff640 1 -- 172.21.15.45:0/2693199690 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5204076530 con 0x7f520c15f720 2023-11-28T20:54:29.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:29.426+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f520c0c2830 con 0x7f520c15f720 2023-11-28T20:54:30.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.428+0000 7f5203fff640 1 -- 172.21.15.45:0/2693199690 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v67) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5204035020 con 0x7f520c15f720 2023-11-28T20:54:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.432+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51ec06af60 msgr2=0x7f51ec06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.432+0000 7f5211f6e640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51ec06af60 0x7f51ec06d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f51fc006fd0 tx=0x7f51fc008040 comp rx=0 tx=0).stop 2023-11-28T20:54:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.432+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c15f720 msgr2=0x7f520c0c6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.432+0000 7f5211f6e640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c15f720 0x7f520c0c6370 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f5204009870 tx=0x7f52040046c0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.433+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 shutdown_connections 2023-11-28T20:54:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.433+0000 7f5211f6e640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f51ec06af60 0x7f51ec06d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.433+0000 7f5211f6e640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f520c15f720 0x7f520c0c6370 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.433+0000 7f5211f6e640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520c15e840 0x7f520c0c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.433+0000 7f5211f6e640 1 --2- 172.21.15.45:0/2693199690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f520c06a500 0x7f520c1f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.433+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 >> 172.21.15.45:0/2693199690 conn(0x7f520c0ced60 msgr2=0x7f520c158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.433+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 shutdown_connections 2023-11-28T20:54:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.433+0000 7f5211f6e640 1 -- 172.21.15.45:0/2693199690 wait complete. 2023-11-28T20:54:30.456 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T20:54:30.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:30.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.674+0000 7febd3fff640 1 Processor -- start 2023-11-28T20:54:30.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.675+0000 7febd3fff640 1 -- start start 2023-11-28T20:54:30.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.676+0000 7febd3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7febd40ae540 0x7febd40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:30.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.676+0000 7febd3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ce630 0x7febd40b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:30.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.676+0000 7febd3fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd415c3c0 con 0x7febd4155780 2023-11-28T20:54:30.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.676+0000 7febd3fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd40ceac0 con 0x7febd40ae540 2023-11-28T20:54:30.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.676+0000 7febd3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd4155b60 con 0x7febd40ce630 2023-11-28T20:54:30.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd27fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7febd40ae540 0x7febd40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:30.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd2ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7febd4155780 0x7febd40abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42494/0 (socket says 172.21.15.45:42494) 2023-11-28T20:54:30.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd27fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7febd40ae540 0x7febd40ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48286/0 (socket says 172.21.15.45:48286) 2023-11-28T20:54:30.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd2ffd640 1 -- 172.21.15.45:0/2788762508 learned_addr learned my addr 172.21.15.45:0/2788762508 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:30.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd37fe640 1 --2- 172.21.15.45:0/2788762508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ce630 0x7febd40b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:30.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd1ffb640 1 -- 172.21.15.45:0/2788762508 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700911474 0 0) 0x7febd415c3c0 con 0x7febd4155780 2023-11-28T20:54:30.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd1ffb640 1 -- 172.21.15.45:0/2788762508 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc003520 con 0x7febd4155780 2023-11-28T20:54:30.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd27fc640 1 -- 172.21.15.45:0/2788762508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ce630 msgr2=0x7febd40b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:30.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd27fc640 1 --2- 172.21.15.45:0/2788762508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ce630 0x7febd40b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd27fc640 1 -- 172.21.15.45:0/2788762508 >> v1:172.21.15.45:6789/0 conn(0x7febd4155780 legacy=0x7febd40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:30.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.677+0000 7febd27fc640 1 -- 172.21.15.45:0/2788762508 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd415ba90 con 0x7febd40ae540 2023-11-28T20:54:30.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.679+0000 7febd27fc640 1 --2- 172.21.15.45:0/2788762508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ae540 0x7febd40ce0f0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7febc80097a0 tx=0x7febc802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea08863c03569a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:30.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.679+0000 7febd1ffb640 1 -- 172.21.15.45:0/2788762508 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8031830 con 0x7febd40ae540 2023-11-28T20:54:30.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.679+0000 7febd1ffb640 1 -- 172.21.15.45:0/2788762508 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc8031990 con 0x7febd40ae540 2023-11-28T20:54:30.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.679+0000 7febd1ffb640 1 -- 172.21.15.45:0/2788762508 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8031d20 con 0x7febd40ae540 2023-11-28T20:54:30.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.680+0000 7febd3fff640 1 -- 172.21.15.45:0/2788762508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ae540 msgr2=0x7febd40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:30.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.680+0000 7febd3fff640 1 --2- 172.21.15.45:0/2788762508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ae540 0x7febd40ce0f0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7febc80097a0 tx=0x7febc802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.680+0000 7febd3fff640 1 -- 172.21.15.45:0/2788762508 shutdown_connections 2023-11-28T20:54:30.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.680+0000 7febd3fff640 1 --2- 172.21.15.45:0/2788762508 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ce630 0x7febd40b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.680+0000 7febd3fff640 1 --2- 172.21.15.45:0/2788762508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ae540 0x7febd40ce0f0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.680+0000 7febd3fff640 1 -- 172.21.15.45:0/2788762508 >> 172.21.15.45:0/2788762508 conn(0x7febd406ab30 msgr2=0x7febd40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.680+0000 7febd3fff640 1 -- 172.21.15.45:0/2788762508 shutdown_connections 2023-11-28T20:54:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.680+0000 7febd3fff640 1 -- 172.21.15.45:0/2788762508 wait complete. 2023-11-28T20:54:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.682+0000 7febd3fff640 1 Processor -- start 2023-11-28T20:54:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.682+0000 7febd3fff640 1 -- start start 2023-11-28T20:54:30.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ae540 0x7febd4144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:30.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ce630 0x7febd406cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:30.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febd4155780 0x7febd406d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:30.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd3fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd41593d0 con 0x7febd40ce630 2023-11-28T20:54:30.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd3fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd4158750 con 0x7febd4155780 2023-11-28T20:54:30.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd4158a30 con 0x7febd40ae540 2023-11-28T20:54:30.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ce630 0x7febd406cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:30.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ce630 0x7febd406cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48294/0 (socket says 172.21.15.45:48294) 2023-11-28T20:54:30.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd27fc640 1 -- 172.21.15.45:0/2252042696 learned_addr learned my addr 172.21.15.45:0/2252042696 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:30.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd2ffd640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ae540 0x7febd4144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:30.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.683+0000 7febd37fe640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febd4155780 0x7febd406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:30.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febd27fc640 1 -- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ae540 msgr2=0x7febd4144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:30.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febd27fc640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ae540 0x7febd4144950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febd27fc640 1 -- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febd4155780 msgr2=0x7febd406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:30.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febd27fc640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febd4155780 0x7febd406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febd27fc640 1 -- 172.21.15.45:0/2252042696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8009400 con 0x7febd40ce630 2023-11-28T20:54:30.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febd27fc640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ce630 0x7febd406cb40 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7febc802bee0 tx=0x7febc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:30.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febbb7fe640 1 -- 172.21.15.45:0/2252042696 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc803a6f0 con 0x7febd40ce630 2023-11-28T20:54:30.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febbb7fe640 1 -- 172.21.15.45:0/2252042696 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc803a870 con 0x7febd40ce630 2023-11-28T20:54:30.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febbb7fe640 1 -- 172.21.15.45:0/2252042696 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc803ac20 con 0x7febd40ce630 2023-11-28T20:54:30.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.684+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd4158d10 con 0x7febd40ce630 2023-11-28T20:54:30.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.685+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd4144e90 con 0x7febd40ce630 2023-11-28T20:54:30.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.686+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd404e920 con 0x7febd40ce630 2023-11-28T20:54:30.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.687+0000 7febbb7fe640 1 -- 172.21.15.45:0/2252042696 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7febc8030380 con 0x7febd40ce630 2023-11-28T20:54:30.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.688+0000 7febbb7fe640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7febc006af60 0x7febc006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:30.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.689+0000 7febbb7fe640 1 -- 172.21.15.45:0/2252042696 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7febc80ab380 con 0x7febd40ce630 2023-11-28T20:54:30.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.689+0000 7febd2ffd640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7febc006af60 0x7febc006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:30.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.690+0000 7febd2ffd640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7febc006af60 0x7febc006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febc4001560 tx=0x7febc4009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:30.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.694+0000 7febbb7fe640 1 -- 172.21.15.45:0/2252042696 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7febc80760a0 con 0x7febd40ce630 2023-11-28T20:54:30.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.929+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febd406c620 con 0x7febd40ce630 2023-11-28T20:54:30.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.930+0000 7febbb7fe640 1 -- 172.21.15.45:0/2252042696 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4918 (secure 0 0 0) 0x7febc80796c0 con 0x7febd40ce630 2023-11-28T20:54:30.932 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:30.932 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/1133315976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1133315976},{"type":"v1","addr":"172.21.15.190:6836","nonce":1133315976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2810069664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2810069664},{"type":"v1","addr":"172.21.15.190:6837","nonce":2810069664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1553174870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1553174870},{"type":"v1","addr":"172.21.15.45:6833","nonce":1553174870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:54:26.355691+0000","modified":"2023-11-28T20:54:29.430641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8595":{"gid":8595,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/684266044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":684266044},{"type":"v1","addr":"172.21.15.45:6835","nonce":684266044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-28T20:54:30.932 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 67 2023-11-28T20:54:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7febc006af60 msgr2=0x7febc006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7febc006af60 0x7febc006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febc4001560 tx=0x7febc4009040 comp rx=0 tx=0).stop 2023-11-28T20:54:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ce630 msgr2=0x7febd406cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ce630 0x7febd406cb40 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7febc802bee0 tx=0x7febc80046c0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 shutdown_connections 2023-11-28T20:54:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7febc006af60 0x7febc006d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febd4155780 0x7febd406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febd40ce630 0x7febd406cb40 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 --2- 172.21.15.45:0/2252042696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febd40ae540 0x7febd4144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:30.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 >> 172.21.15.45:0/2252042696 conn(0x7febd406ab30 msgr2=0x7febd415c870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:30.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 shutdown_connections 2023-11-28T20:54:30.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:30.933+0000 7febd3fff640 1 -- 172.21.15.45:0/2252042696 wait complete. 2023-11-28T20:54:30.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:31.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.169+0000 7f6909e9f640 1 Processor -- start 2023-11-28T20:54:31.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.169+0000 7f6909e9f640 1 -- start start 2023-11-28T20:54:31.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.171+0000 7f6909e9f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6904155780 0x7f69040abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.171+0000 7f6909e9f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f69040ce800 0x7f69040b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.171+0000 7f6909e9f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690415c3c0 con 0x7f69040ae540 2023-11-28T20:54:31.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.171+0000 7f6909e9f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6904155b60 con 0x7f69040ce800 2023-11-28T20:54:31.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.171+0000 7f6909e9f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690415ba90 con 0x7f6904155780 2023-11-28T20:54:31.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.172+0000 7f68fbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f69040ae540 0x7f69040ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42508/0 (socket says 172.21.15.45:42508) 2023-11-28T20:54:31.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.172+0000 7f68fbfff640 1 -- 172.21.15.45:0/2421362954 learned_addr learned my addr 172.21.15.45:0/2421362954 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:31.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.172+0000 7f690969e640 1 --2- 172.21.15.45:0/2421362954 >> v2:172.21.15.190:3300/0 conn(0x7f69040ce800 0x7f69040b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.172+0000 7f6908e9d640 1 --2- 172.21.15.45:0/2421362954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6904155780 0x7f69040abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.172+0000 7f68fb7fe640 1 -- 172.21.15.45:0/2421362954 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656346727 0 0) 0x7f690415c3c0 con 0x7f69040ae540 2023-11-28T20:54:31.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.173+0000 7f68fb7fe640 1 -- 172.21.15.45:0/2421362954 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68ec003520 con 0x7f69040ae540 2023-11-28T20:54:31.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.173+0000 7f690969e640 1 -- 172.21.15.45:0/2421362954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6904155780 msgr2=0x7f69040abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:31.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.173+0000 7f690969e640 1 --2- 172.21.15.45:0/2421362954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6904155780 0x7f69040abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.173+0000 7f690969e640 1 -- 172.21.15.45:0/2421362954 >> v1:172.21.15.45:6789/0 conn(0x7f69040ae540 legacy=0x7f69040ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.173+0000 7f68fb7fe640 1 -- 172.21.15.45:0/2421362954 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157979781 0 0) 0x7f68ec003520 con 0x7f69040ae540 2023-11-28T20:54:31.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.173+0000 7f690969e640 1 -- 172.21.15.45:0/2421362954 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690415bd70 con 0x7f69040ce800 2023-11-28T20:54:31.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.173+0000 7f6908e9d640 1 --2- 172.21.15.45:0/2421362954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6904155780 0x7f69040abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:54:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.175+0000 7f690969e640 1 --2- 172.21.15.45:0/2421362954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69040ce800 0x7f69040b20a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009d90 tx=0x7f68fc02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69acb2f0eb7ec14e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.175+0000 7f68fb7fe640 1 -- 172.21.15.45:0/2421362954 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4001670 con 0x7f69040ce800 2023-11-28T20:54:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.175+0000 7f68fb7fe640 1 -- 172.21.15.45:0/2421362954 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc02c8a0 con 0x7f69040ce800 2023-11-28T20:54:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.175+0000 7f68fb7fe640 1 -- 172.21.15.45:0/2421362954 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc02cc30 con 0x7f69040ce800 2023-11-28T20:54:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.176+0000 7f6909e9f640 1 -- 172.21.15.45:0/2421362954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69040ce800 msgr2=0x7f69040b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.176+0000 7f6909e9f640 1 --2- 172.21.15.45:0/2421362954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69040ce800 0x7f69040b20a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009d90 tx=0x7f68fc02bd70 comp rx=0 tx=0).stop 2023-11-28T20:54:31.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.176+0000 7f6909e9f640 1 -- 172.21.15.45:0/2421362954 shutdown_connections 2023-11-28T20:54:31.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.176+0000 7f6909e9f640 1 --2- 172.21.15.45:0/2421362954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69040ce800 0x7f69040b20a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.176+0000 7f6909e9f640 1 --2- 172.21.15.45:0/2421362954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6904155780 0x7f69040abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.176+0000 7f6909e9f640 1 -- 172.21.15.45:0/2421362954 >> 172.21.15.45:0/2421362954 conn(0x7f690406ab30 msgr2=0x7f69040b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:31.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.176+0000 7f6909e9f640 1 -- 172.21.15.45:0/2421362954 shutdown_connections 2023-11-28T20:54:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.176+0000 7f6909e9f640 1 -- 172.21.15.45:0/2421362954 wait complete. 2023-11-28T20:54:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.179+0000 7f6909e9f640 1 Processor -- start 2023-11-28T20:54:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.179+0000 7f6909e9f640 1 -- start start 2023-11-28T20:54:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.180+0000 7f6909e9f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69040ae540 0x7f690413ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.180+0000 7f6909e9f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69040ce800 0x7f6904142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.180+0000 7f6909e9f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6904155780 0x7f690406c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.180+0000 7f6909e9f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904140c20 con 0x7f69040ae540 2023-11-28T20:54:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.180+0000 7f6909e9f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904143180 con 0x7f6904155780 2023-11-28T20:54:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.180+0000 7f6909e9f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6904143460 con 0x7f69040ce800 2023-11-28T20:54:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.180+0000 7f6908e9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69040ae540 0x7f690413ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f6908e9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69040ae540 0x7f690413ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48302/0 (socket says 172.21.15.45:48302) 2023-11-28T20:54:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f6908e9d640 1 -- 172.21.15.45:0/2526402138 learned_addr learned my addr 172.21.15.45:0/2526402138 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:31.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f68fbfff640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69040ce800 0x7f6904142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f690969e640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6904155780 0x7f690406c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f6908e9d640 1 -- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69040ce800 msgr2=0x7f6904142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f6908e9d640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69040ce800 0x7f6904142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f6908e9d640 1 -- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6904155780 msgr2=0x7f690406c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f6908e9d640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6904155780 0x7f690406c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f6908e9d640 1 -- 172.21.15.45:0/2526402138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6900009400 con 0x7f69040ae540 2023-11-28T20:54:31.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f68fbfff640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69040ce800 0x7f6904142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:31.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f690969e640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6904155780 0x7f690406c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:31.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.181+0000 7f6908e9d640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69040ae540 0x7f690413ff00 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f690000e960 tx=0x7f690000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:31.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.182+0000 7f68f97fa640 1 -- 172.21.15.45:0/2526402138 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690000ca00 con 0x7f69040ae540 2023-11-28T20:54:31.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.182+0000 7f68f97fa640 1 -- 172.21.15.45:0/2526402138 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690000cb60 con 0x7f69040ae540 2023-11-28T20:54:31.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.182+0000 7f68f97fa640 1 -- 172.21.15.45:0/2526402138 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900013470 con 0x7f69040ae540 2023-11-28T20:54:31.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.182+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68fc009a70 con 0x7f69040ae540 2023-11-28T20:54:31.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.182+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69041408b0 con 0x7f69040ae540 2023-11-28T20:54:31.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.183+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69040aca00 con 0x7f69040ae540 2023-11-28T20:54:31.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.184+0000 7f68f97fa640 1 -- 172.21.15.45:0/2526402138 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f69000135d0 con 0x7f69040ae540 2023-11-28T20:54:31.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.185+0000 7f68f97fa640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f68f006af60 0x7f68f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.185+0000 7f68f97fa640 1 -- 172.21.15.45:0/2526402138 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6900089f60 con 0x7f69040ae540 2023-11-28T20:54:31.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.186+0000 7f68fbfff640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f68f006af60 0x7f68f006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.187+0000 7f68fbfff640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f68f006af60 0x7f68f006d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f68f40011e0 tx=0x7f68f4009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:31.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.189+0000 7f68f97fa640 1 -- 172.21.15.45:0/2526402138 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6900054d00 con 0x7f69040ae540 2023-11-28T20:54:31.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.388+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69040ad580 con 0x7f69040ae540 2023-11-28T20:54:31.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.390+0000 7f68f97fa640 1 -- 172.21.15.45:0/2526402138 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16816 (secure 0 0 0) 0x7f6900017090 con 0x7f69040ae540 2023-11-28T20:54:31.393 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:31.393 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":148,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:27.894784+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:54:24.325054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-11-28T20:54:25.374707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:31.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f68f006af60 msgr2=0x7f68f006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f68f006af60 0x7f68f006d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f68f40011e0 tx=0x7f68f4009040 comp rx=0 tx=0).stop 2023-11-28T20:54:31.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69040ae540 msgr2=0x7f690413ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69040ae540 0x7f690413ff00 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f690000e960 tx=0x7f690000ee20 comp rx=0 tx=0).stop 2023-11-28T20:54:31.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 shutdown_connections 2023-11-28T20:54:31.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f68f006af60 0x7f68f006d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6904155780 0x7f690406c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f69040ce800 0x7f6904142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 --2- 172.21.15.45:0/2526402138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69040ae540 0x7f690413ff00 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 >> 172.21.15.45:0/2526402138 conn(0x7f690406ab30 msgr2=0x7f69040b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:31.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 shutdown_connections 2023-11-28T20:54:31.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.393+0000 7f6909e9f640 1 -- 172.21.15.45:0/2526402138 wait complete. 2023-11-28T20:54:31.416 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T20:54:31.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:54:31.442 INFO:tasks.ceph.mds.c.smithi045.stderr:2023-11-28T20:54:31.439+0000 7f934467c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:54:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.645+0000 7f0a03fff640 1 Processor -- start 2023-11-28T20:54:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.645+0000 7f0a03fff640 1 -- start start 2023-11-28T20:54:31.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.646+0000 7f0a03fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0a040ae540 0x7f0a040ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.646+0000 7f0a03fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0415c3c0 con 0x7f0a04155780 2023-11-28T20:54:31.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.646+0000 7f0a03fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a04155b60 con 0x7f0a040ce630 2023-11-28T20:54:31.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.646+0000 7f0a03fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a040ceac0 con 0x7f0a040ae540 2023-11-28T20:54:31.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.647+0000 7f0a02ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0a04155780 0x7f0a040abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42518/0 (socket says 172.21.15.45:42518) 2023-11-28T20:54:31.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.647+0000 7f0a02ffd640 1 -- 172.21.15.45:0/4134009253 learned_addr learned my addr 172.21.15.45:0/4134009253 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:31.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.647+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484060291 0 0) 0x7f0a0415c3c0 con 0x7f0a04155780 2023-11-28T20:54:31.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.647+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ec003520 con 0x7f0a04155780 2023-11-28T20:54:31.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.647+0000 7f0a027fc640 1 --2- 172.21.15.45:0/4134009253 >> v2:172.21.15.190:3300/0 conn(0x7f0a040ae540 0x7f0a040ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.647+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1562791825 0 0) 0x7f09ec003520 con 0x7f0a04155780 2023-11-28T20:54:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.647+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 >> v2:172.21.15.190:3300/0 conn(0x7f0a040ae540 msgr2=0x7f0a040ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.647+0000 7f0a01ffb640 1 --2- 172.21.15.45:0/4134009253 >> v2:172.21.15.190:3300/0 conn(0x7f0a040ae540 0x7f0a040ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.647+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 >> v1:172.21.15.190:6789/0 conn(0x7f0a040ce630 legacy=0x7f0a040b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0415ba90 con 0x7f0a04155780 2023-11-28T20:54:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f09f4001670 con 0x7f0a04155780 2023-11-28T20:54:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 >> v1:172.21.15.45:6789/0 conn(0x7f0a04155780 legacy=0x7f0a040abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a01ffb640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09ec003970 0x7f09ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a01ffb640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09ec0047c0 0x7f09ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a01ffb640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09ec005180 0x7f09ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a040ceac0 con 0x7f09ec003970 2023-11-28T20:54:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec00c020 con 0x7f09ec0047c0 2023-11-28T20:54:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec00c1e0 con 0x7f09ec005180 2023-11-28T20:54:31.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a02ffd640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09ec003970 0x7f09ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a02ffd640 1 -- 172.21.15.45:0/4134009253 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09ec005180 msgr2=0x7f09ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a02ffd640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09ec005180 0x7f09ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a02ffd640 1 -- 172.21.15.45:0/4134009253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09ec0047c0 msgr2=0x7f09ec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a027fc640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09ec0047c0 0x7f09ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a02ffd640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09ec0047c0 0x7f09ec004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a02ffd640 1 -- 172.21.15.45:0/4134009253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec003520 con 0x7f09ec003970 2023-11-28T20:54:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.648+0000 7f0a037fe640 1 -- 172.21.15.45:0/4134009253 reap_dead start 2023-11-28T20:54:31.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.650+0000 7f0a02ffd640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09ec003970 0x7f09ec003d90 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f09f400c240 tx=0x7f09ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b681af8d1e6abbf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:31.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.650+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4005ad0 con 0x7f09ec003970 2023-11-28T20:54:31.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.650+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f4005c50 con 0x7f09ec003970 2023-11-28T20:54:31.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.650+0000 7f0a01ffb640 1 -- 172.21.15.45:0/4134009253 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f402c8b0 con 0x7f09ec003970 2023-11-28T20:54:31.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.650+0000 7f0a03fff640 1 -- 172.21.15.45:0/4134009253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09ec003970 msgr2=0x7f09ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.650+0000 7f0a03fff640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09ec003970 0x7f09ec003d90 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f09f400c240 tx=0x7f09ec0057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.650+0000 7f0a03fff640 1 -- 172.21.15.45:0/4134009253 shutdown_connections 2023-11-28T20:54:31.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.650+0000 7f0a03fff640 1 --2- 172.21.15.45:0/4134009253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09ec003970 0x7f09ec003d90 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.650+0000 7f0a03fff640 1 -- 172.21.15.45:0/4134009253 >> 172.21.15.45:0/4134009253 conn(0x7f0a0406ab30 msgr2=0x7f0a040b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:31.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.651+0000 7f0a03fff640 1 -- 172.21.15.45:0/4134009253 shutdown_connections 2023-11-28T20:54:31.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.651+0000 7f0a03fff640 1 -- 172.21.15.45:0/4134009253 wait complete. 2023-11-28T20:54:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.652+0000 7f0a03fff640 1 Processor -- start 2023-11-28T20:54:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.653+0000 7f0a03fff640 1 -- start start 2023-11-28T20:54:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.653+0000 7f0a03fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09ec003970 0x7f0a04099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.653+0000 7f0a03fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a0414c990 0x7f0a0414f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a03fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a0414f7f0 0x7f0a040afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a03fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0414fc70 con 0x7f0a0414f7f0 2023-11-28T20:54:31.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a03fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04150bd0 con 0x7f0a0414c990 2023-11-28T20:54:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a03fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0414fe50 con 0x7f09ec003970 2023-11-28T20:54:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a037fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a0414f7f0 0x7f0a040afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a037fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a0414f7f0 0x7f0a040afa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48324/0 (socket says 172.21.15.45:48324) 2023-11-28T20:54:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a037fe640 1 -- 172.21.15.45:0/3506582112 learned_addr learned my addr 172.21.15.45:0/3506582112 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a027fc640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a0414c990 0x7f0a0414f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a02ffd640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09ec003970 0x7f0a04099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a037fe640 1 -- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09ec003970 msgr2=0x7f0a04099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a037fe640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09ec003970 0x7f0a04099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a037fe640 1 -- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a0414c990 msgr2=0x7f0a0414f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a037fe640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a0414c990 0x7f0a0414f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.654+0000 7f0a037fe640 1 -- 172.21.15.45:0/3506582112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a04150130 con 0x7f0a0414f7f0 2023-11-28T20:54:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.655+0000 7f0a037fe640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a0414f7f0 0x7f0a040afa50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0a040724b0 tx=0x7f09fc00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.655+0000 7f09ebfff640 1 -- 172.21.15.45:0/3506582112 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc005d40 con 0x7f0a0414f7f0 2023-11-28T20:54:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.655+0000 7f09ebfff640 1 -- 172.21.15.45:0/3506582112 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc00ced0 con 0x7f0a0414f7f0 2023-11-28T20:54:31.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.655+0000 7f09ebfff640 1 -- 172.21.15.45:0/3506582112 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc00a850 con 0x7f0a0414f7f0 2023-11-28T20:54:31.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.655+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a04150410 con 0x7f0a0414f7f0 2023-11-28T20:54:31.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.655+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0414ce10 con 0x7f0a0414f7f0 2023-11-28T20:54:31.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.656+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a04150dc0 con 0x7f0a0414f7f0 2023-11-28T20:54:31.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.657+0000 7f09ebfff640 1 -- 172.21.15.45:0/3506582112 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f09fc00a9b0 con 0x7f0a0414f7f0 2023-11-28T20:54:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.659+0000 7f09ebfff640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f09f006af60 0x7f09f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.659+0000 7f09ebfff640 1 -- 172.21.15.45:0/3506582112 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f09fc0893a0 con 0x7f0a0414f7f0 2023-11-28T20:54:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.659+0000 7f0a02ffd640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f09f006af60 0x7f09f006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.660+0000 7f0a02ffd640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f09f006af60 0x7f09f006d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f09f4034000 tx=0x7f09f4009030 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:31.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.663+0000 7f09ebfff640 1 -- 172.21.15.45:0/3506582112 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a04150dc0 con 0x7f0a0414f7f0 2023-11-28T20:54:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.854+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 --> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a0415b740 con 0x7f09f006af60 2023-11-28T20:54:31.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.883+0000 7f09ebfff640 1 -- 172.21.15.45:0/3506582112 <== mgr.7015 v2:172.21.15.190:6832/2586319130 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287165 (secure 0 0 0) 0x7f09f4001430 con 0x7f09f006af60 2023-11-28T20:54:31.887 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:31.890 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:54:31.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f09f006af60 msgr2=0x7f09f006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f09f006af60 0x7f09f006d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f09f4034000 tx=0x7f09f4009030 comp rx=0 tx=0).stop 2023-11-28T20:54:31.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a0414f7f0 msgr2=0x7f0a040afa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:31.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a0414f7f0 0x7f0a040afa50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0a040724b0 tx=0x7f09fc00cb90 comp rx=0 tx=0).stop 2023-11-28T20:54:31.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 shutdown_connections 2023-11-28T20:54:31.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f09f006af60 0x7f09f006d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a0414f7f0 0x7f0a040afa50 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a0414c990 0x7f0a0414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 --2- 172.21.15.45:0/3506582112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f09ec003970 0x7f0a04099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:31.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 >> 172.21.15.45:0/3506582112 conn(0x7f0a0406ab30 msgr2=0x7f0a0415a160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:31.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.889+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 shutdown_connections 2023-11-28T20:54:31.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:31.890+0000 7f0a03fff640 1 -- 172.21.15.45:0/3506582112 wait complete. 2023-11-28T20:54:31.912 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-11-28T20:54:31.618700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1189,"num_read_kb":2391,"num_write":415,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4772,"ondisk_log_size":4772,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":352436,"kb_used_data":15544,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754622284,"statfs":{"total":773094113280,"available":772733218816,"internally_reserved":0,"allocated":15917056,"data_stored":8630612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":344837680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002289"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.912739+0000","last_change":"2023-11-28T20:54:26.908883+0000","last_active":"2023-11-28T20:54:27.912739+0000","last_peered":"2023-11-28T20:54:27.912739+0000","last_clean":"2023-11-28T20:54:27.912739+0000","last_became_active":"2023-11-28T20:54:26.908466+0000","last_became_peered":"2023-11-28T20:54:26.908466+0000","last_unstale":"2023-11-28T20:54:27.912739+0000","last_undegraded":"2023-11-28T20:54:27.912739+0000","last_fullsized":"2023-11-28T20:54:27.912739+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:09:58.510046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924297+0000","last_change":"2023-11-28T20:54:26.918099+0000","last_active":"2023-11-28T20:54:27.924297+0000","last_peered":"2023-11-28T20:54:27.924297+0000","last_clean":"2023-11-28T20:54:27.924297+0000","last_became_active":"2023-11-28T20:54:26.915982+0000","last_became_peered":"2023-11-28T20:54:26.915982+0000","last_unstale":"2023-11-28T20:54:27.924297+0000","last_undegraded":"2023-11-28T20:54:27.924297+0000","last_fullsized":"2023-11-28T20:54:27.924297+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:22:55.453144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924612+0000","last_change":"2023-11-28T20:54:26.916126+0000","last_active":"2023-11-28T20:54:27.924612+0000","last_peered":"2023-11-28T20:54:27.924612+0000","last_clean":"2023-11-28T20:54:27.924612+0000","last_became_active":"2023-11-28T20:54:26.915833+0000","last_became_peered":"2023-11-28T20:54:26.915833+0000","last_unstale":"2023-11-28T20:54:27.924612+0000","last_undegraded":"2023-11-28T20:54:27.924612+0000","last_fullsized":"2023-11-28T20:54:27.924612+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:50:23.060502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241904+0000","last_change":"2023-11-28T20:54:26.910344+0000","last_active":"2023-11-28T20:54:28.241904+0000","last_peered":"2023-11-28T20:54:28.241904+0000","last_clean":"2023-11-28T20:54:28.241904+0000","last_became_active":"2023-11-28T20:54:26.910052+0000","last_became_peered":"2023-11-28T20:54:26.910052+0000","last_unstale":"2023-11-28T20:54:28.241904+0000","last_undegraded":"2023-11-28T20:54:28.241904+0000","last_fullsized":"2023-11-28T20:54:28.241904+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:09.074880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911223+0000","last_change":"2023-11-28T20:54:26.913154+0000","last_active":"2023-11-28T20:54:27.911223+0000","last_peered":"2023-11-28T20:54:27.911223+0000","last_clean":"2023-11-28T20:54:27.911223+0000","last_became_active":"2023-11-28T20:54:26.912827+0000","last_became_peered":"2023-11-28T20:54:26.912827+0000","last_unstale":"2023-11-28T20:54:27.911223+0000","last_undegraded":"2023-11-28T20:54:27.911223+0000","last_fullsized":"2023-11-28T20:54:27.911223+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:39:54.271922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.912615+0000","last_change":"2023-11-28T20:54:26.909050+0000","last_active":"2023-11-28T20:54:26.912615+0000","last_peered":"2023-11-28T20:54:26.912615+0000","last_clean":"2023-11-28T20:54:26.912615+0000","last_became_active":"2023-11-28T20:54:26.908736+0000","last_became_peered":"2023-11-28T20:54:26.908736+0000","last_unstale":"2023-11-28T20:54:26.912615+0000","last_undegraded":"2023-11-28T20:54:26.912615+0000","last_fullsized":"2023-11-28T20:54:26.912615+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:52:15.491692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.909817+0000","last_change":"2023-11-28T20:54:26.900580+0000","last_active":"2023-11-28T20:54:26.909817+0000","last_peered":"2023-11-28T20:54:26.909817+0000","last_clean":"2023-11-28T20:54:26.909817+0000","last_became_active":"2023-11-28T20:54:26.900259+0000","last_became_peered":"2023-11-28T20:54:26.900259+0000","last_unstale":"2023-11-28T20:54:26.909817+0000","last_undegraded":"2023-11-28T20:54:26.909817+0000","last_fullsized":"2023-11-28T20:54:26.909817+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:14.162518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.923908+0000","last_change":"2023-11-28T20:54:26.916679+0000","last_active":"2023-11-28T20:54:27.923908+0000","last_peered":"2023-11-28T20:54:27.923908+0000","last_clean":"2023-11-28T20:54:27.923908+0000","last_became_active":"2023-11-28T20:54:26.915176+0000","last_became_peered":"2023-11-28T20:54:26.915176+0000","last_unstale":"2023-11-28T20:54:27.923908+0000","last_undegraded":"2023-11-28T20:54:27.923908+0000","last_fullsized":"2023-11-28T20:54:27.923908+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:01.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242495+0000","last_change":"2023-11-28T20:54:26.923096+0000","last_active":"2023-11-28T20:54:28.242495+0000","last_peered":"2023-11-28T20:54:28.242495+0000","last_clean":"2023-11-28T20:54:28.242495+0000","last_became_active":"2023-11-28T20:54:26.922765+0000","last_became_peered":"2023-11-28T20:54:26.922765+0000","last_unstale":"2023-11-28T20:54:28.242495+0000","last_undegraded":"2023-11-28T20:54:28.242495+0000","last_fullsized":"2023-11-28T20:54:28.242495+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:36:56.721066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907978+0000","last_change":"2023-11-28T20:54:26.906479+0000","last_active":"2023-11-28T20:54:27.907978+0000","last_peered":"2023-11-28T20:54:27.907978+0000","last_clean":"2023-11-28T20:54:27.907978+0000","last_became_active":"2023-11-28T20:54:26.906174+0000","last_became_peered":"2023-11-28T20:54:26.906174+0000","last_unstale":"2023-11-28T20:54:27.907978+0000","last_undegraded":"2023-11-28T20:54:27.907978+0000","last_fullsized":"2023-11-28T20:54:27.907978+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:23:21.169904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910961+0000","last_change":"2023-11-28T20:54:26.908426+0000","last_active":"2023-11-28T20:54:27.910961+0000","last_peered":"2023-11-28T20:54:27.910961+0000","last_clean":"2023-11-28T20:54:27.910961+0000","last_became_active":"2023-11-28T20:54:26.907420+0000","last_became_peered":"2023-11-28T20:54:26.907420+0000","last_unstale":"2023-11-28T20:54:27.910961+0000","last_undegraded":"2023-11-28T20:54:27.910961+0000","last_fullsized":"2023-11-28T20:54:27.910961+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:23:21.169904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.912726+0000","last_change":"2023-11-28T20:54:26.904208+0000","last_active":"2023-11-28T20:54:26.912726+0000","last_peered":"2023-11-28T20:54:26.912726+0000","last_clean":"2023-11-28T20:54:26.912726+0000","last_became_active":"2023-11-28T20:54:26.903277+0000","last_became_peered":"2023-11-28T20:54:26.903277+0000","last_unstale":"2023-11-28T20:54:26.912726+0000","last_undegraded":"2023-11-28T20:54:26.912726+0000","last_fullsized":"2023-11-28T20:54:26.912726+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:58:45.024456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907412+0000","last_change":"2023-11-28T20:54:26.899888+0000","last_active":"2023-11-28T20:54:27.907412+0000","last_peered":"2023-11-28T20:54:27.907412+0000","last_clean":"2023-11-28T20:54:27.907412+0000","last_became_active":"2023-11-28T20:54:26.899489+0000","last_became_peered":"2023-11-28T20:54:26.899489+0000","last_unstale":"2023-11-28T20:54:27.907412+0000","last_undegraded":"2023-11-28T20:54:27.907412+0000","last_fullsized":"2023-11-28T20:54:27.907412+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:20.727468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924031+0000","last_change":"2023-11-28T20:54:26.914048+0000","last_active":"2023-11-28T20:54:27.924031+0000","last_peered":"2023-11-28T20:54:27.924031+0000","last_clean":"2023-11-28T20:54:27.924031+0000","last_became_active":"2023-11-28T20:54:26.913780+0000","last_became_peered":"2023-11-28T20:54:26.913780+0000","last_unstale":"2023-11-28T20:54:27.924031+0000","last_undegraded":"2023-11-28T20:54:27.924031+0000","last_fullsized":"2023-11-28T20:54:27.924031+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:21.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910485+0000","last_change":"2023-11-28T20:54:26.907081+0000","last_active":"2023-11-28T20:54:27.910485+0000","last_peered":"2023-11-28T20:54:27.910485+0000","last_clean":"2023-11-28T20:54:27.910485+0000","last_became_active":"2023-11-28T20:54:26.906378+0000","last_became_peered":"2023-11-28T20:54:26.906378+0000","last_unstale":"2023-11-28T20:54:27.910485+0000","last_undegraded":"2023-11-28T20:54:27.910485+0000","last_fullsized":"2023-11-28T20:54:27.910485+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:22.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.901117+0000","last_change":"2023-11-28T20:54:26.892995+0000","last_active":"2023-11-28T20:54:26.901117+0000","last_peered":"2023-11-28T20:54:26.901117+0000","last_clean":"2023-11-28T20:54:26.901117+0000","last_became_active":"2023-11-28T20:54:26.892456+0000","last_became_peered":"2023-11-28T20:54:26.892456+0000","last_unstale":"2023-11-28T20:54:26.901117+0000","last_undegraded":"2023-11-28T20:54:26.901117+0000","last_fullsized":"2023-11-28T20:54:26.901117+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:12.946982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907862+0000","last_change":"2023-11-28T20:54:26.901170+0000","last_active":"2023-11-28T20:54:27.907862+0000","last_peered":"2023-11-28T20:54:27.907862+0000","last_clean":"2023-11-28T20:54:27.907862+0000","last_became_active":"2023-11-28T20:54:26.900805+0000","last_became_peered":"2023-11-28T20:54:26.900805+0000","last_unstale":"2023-11-28T20:54:27.907862+0000","last_undegraded":"2023-11-28T20:54:27.907862+0000","last_fullsized":"2023-11-28T20:54:27.907862+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:22.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924451+0000","last_change":"2023-11-28T20:54:26.906923+0000","last_active":"2023-11-28T20:54:27.924451+0000","last_peered":"2023-11-28T20:54:27.924451+0000","last_clean":"2023-11-28T20:54:27.924451+0000","last_became_active":"2023-11-28T20:54:26.906536+0000","last_became_peered":"2023-11-28T20:54:26.906536+0000","last_unstale":"2023-11-28T20:54:27.924451+0000","last_undegraded":"2023-11-28T20:54:27.924451+0000","last_fullsized":"2023-11-28T20:54:27.924451+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:52:37.714109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.911408+0000","last_change":"2023-11-28T20:54:26.902089+0000","last_active":"2023-11-28T20:54:26.911408+0000","last_peered":"2023-11-28T20:54:26.911408+0000","last_clean":"2023-11-28T20:54:26.911408+0000","last_became_active":"2023-11-28T20:54:26.901319+0000","last_became_peered":"2023-11-28T20:54:26.901319+0000","last_unstale":"2023-11-28T20:54:26.911408+0000","last_undegraded":"2023-11-28T20:54:26.911408+0000","last_fullsized":"2023-11-28T20:54:26.911408+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:52:38.278688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544173+0000","last_change":"2023-11-28T20:54:26.910420+0000","last_active":"2023-11-28T20:54:28.544173+0000","last_peered":"2023-11-28T20:54:28.544173+0000","last_clean":"2023-11-28T20:54:28.544173+0000","last_became_active":"2023-11-28T20:54:26.909924+0000","last_became_peered":"2023-11-28T20:54:26.909924+0000","last_unstale":"2023-11-28T20:54:28.544173+0000","last_undegraded":"2023-11-28T20:54:28.544173+0000","last_fullsized":"2023-11-28T20:54:28.544173+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:26:22.390075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241754+0000","last_change":"2023-11-28T20:54:26.912676+0000","last_active":"2023-11-28T20:54:28.241754+0000","last_peered":"2023-11-28T20:54:28.241754+0000","last_clean":"2023-11-28T20:54:28.241754+0000","last_became_active":"2023-11-28T20:54:26.912125+0000","last_became_peered":"2023-11-28T20:54:26.912125+0000","last_unstale":"2023-11-28T20:54:28.241754+0000","last_undegraded":"2023-11-28T20:54:28.241754+0000","last_fullsized":"2023-11-28T20:54:28.241754+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:07:34.024717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911702+0000","last_change":"2023-11-28T20:54:26.917717+0000","last_active":"2023-11-28T20:54:27.911702+0000","last_peered":"2023-11-28T20:54:27.911702+0000","last_clean":"2023-11-28T20:54:27.911702+0000","last_became_active":"2023-11-28T20:54:26.917309+0000","last_became_peered":"2023-11-28T20:54:26.917309+0000","last_unstale":"2023-11-28T20:54:27.911702+0000","last_undegraded":"2023-11-28T20:54:27.911702+0000","last_fullsized":"2023-11-28T20:54:27.911702+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:35.269079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241673+0000","last_change":"2023-11-28T20:54:26.912098+0000","last_active":"2023-11-28T20:54:28.241673+0000","last_peered":"2023-11-28T20:54:28.241673+0000","last_clean":"2023-11-28T20:54:28.241673+0000","last_became_active":"2023-11-28T20:54:26.911842+0000","last_became_peered":"2023-11-28T20:54:26.911842+0000","last_unstale":"2023-11-28T20:54:28.241673+0000","last_undegraded":"2023-11-28T20:54:28.241673+0000","last_fullsized":"2023-11-28T20:54:28.241673+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:59:54.055073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.909722+0000","last_change":"2023-11-28T20:54:26.893116+0000","last_active":"2023-11-28T20:54:27.909722+0000","last_peered":"2023-11-28T20:54:27.909722+0000","last_clean":"2023-11-28T20:54:27.909722+0000","last_became_active":"2023-11-28T20:54:26.889331+0000","last_became_peered":"2023-11-28T20:54:26.889331+0000","last_unstale":"2023-11-28T20:54:27.909722+0000","last_undegraded":"2023-11-28T20:54:27.909722+0000","last_fullsized":"2023-11-28T20:54:27.909722+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:30:18.390364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.912700+0000","last_change":"2023-11-28T20:54:26.910618+0000","last_active":"2023-11-28T20:54:27.912700+0000","last_peered":"2023-11-28T20:54:27.912700+0000","last_clean":"2023-11-28T20:54:27.912700+0000","last_became_active":"2023-11-28T20:54:26.909860+0000","last_became_peered":"2023-11-28T20:54:26.909860+0000","last_unstale":"2023-11-28T20:54:27.912700+0000","last_undegraded":"2023-11-28T20:54:27.912700+0000","last_fullsized":"2023-11-28T20:54:27.912700+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:59:54.055073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544202+0000","last_change":"2023-11-28T20:54:26.896978+0000","last_active":"2023-11-28T20:54:28.544202+0000","last_peered":"2023-11-28T20:54:28.544202+0000","last_clean":"2023-11-28T20:54:28.544202+0000","last_became_active":"2023-11-28T20:54:26.896602+0000","last_became_peered":"2023-11-28T20:54:26.896602+0000","last_unstale":"2023-11-28T20:54:28.544202+0000","last_undegraded":"2023-11-28T20:54:28.544202+0000","last_fullsized":"2023-11-28T20:54:28.544202+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:43:31.500958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.911429+0000","last_change":"2023-11-28T20:54:26.902152+0000","last_active":"2023-11-28T20:54:26.911429+0000","last_peered":"2023-11-28T20:54:26.911429+0000","last_clean":"2023-11-28T20:54:26.911429+0000","last_became_active":"2023-11-28T20:54:26.901608+0000","last_became_peered":"2023-11-28T20:54:26.901608+0000","last_unstale":"2023-11-28T20:54:26.911429+0000","last_undegraded":"2023-11-28T20:54:26.911429+0000","last_fullsized":"2023-11-28T20:54:26.911429+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:25:48.284752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.904310+0000","last_change":"2023-11-28T20:54:26.897032+0000","last_active":"2023-11-28T20:54:26.904310+0000","last_peered":"2023-11-28T20:54:26.904310+0000","last_clean":"2023-11-28T20:54:26.904310+0000","last_became_active":"2023-11-28T20:54:26.896710+0000","last_became_peered":"2023-11-28T20:54:26.896710+0000","last_unstale":"2023-11-28T20:54:26.904310+0000","last_undegraded":"2023-11-28T20:54:26.904310+0000","last_fullsized":"2023-11-28T20:54:26.904310+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:29.947416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544337+0000","last_change":"2023-11-28T20:54:26.911793+0000","last_active":"2023-11-28T20:54:28.544337+0000","last_peered":"2023-11-28T20:54:28.544337+0000","last_clean":"2023-11-28T20:54:28.544337+0000","last_became_active":"2023-11-28T20:54:26.911603+0000","last_became_peered":"2023-11-28T20:54:26.911603+0000","last_unstale":"2023-11-28T20:54:28.544337+0000","last_undegraded":"2023-11-28T20:54:28.544337+0000","last_fullsized":"2023-11-28T20:54:28.544337+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:45:18.632459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924136+0000","last_change":"2023-11-28T20:54:26.912272+0000","last_active":"2023-11-28T20:54:27.924136+0000","last_peered":"2023-11-28T20:54:27.924136+0000","last_clean":"2023-11-28T20:54:27.924136+0000","last_became_active":"2023-11-28T20:54:26.911890+0000","last_became_peered":"2023-11-28T20:54:26.911890+0000","last_unstale":"2023-11-28T20:54:27.924136+0000","last_undegraded":"2023-11-28T20:54:27.924136+0000","last_fullsized":"2023-11-28T20:54:27.924136+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:27:35.212161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910069+0000","last_change":"2023-11-28T20:54:26.897348+0000","last_active":"2023-11-28T20:54:27.910069+0000","last_peered":"2023-11-28T20:54:27.910069+0000","last_clean":"2023-11-28T20:54:27.910069+0000","last_became_active":"2023-11-28T20:54:26.894117+0000","last_became_peered":"2023-11-28T20:54:26.894117+0000","last_unstale":"2023-11-28T20:54:27.910069+0000","last_undegraded":"2023-11-28T20:54:27.910069+0000","last_fullsized":"2023-11-28T20:54:27.910069+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:30.700360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910045+0000","last_change":"2023-11-28T20:54:26.902218+0000","last_active":"2023-11-28T20:54:27.910045+0000","last_peered":"2023-11-28T20:54:27.910045+0000","last_clean":"2023-11-28T20:54:27.910045+0000","last_became_active":"2023-11-28T20:54:26.901949+0000","last_became_peered":"2023-11-28T20:54:26.901949+0000","last_unstale":"2023-11-28T20:54:27.910045+0000","last_undegraded":"2023-11-28T20:54:27.910045+0000","last_fullsized":"2023-11-28T20:54:27.910045+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:50.901505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907631+0000","last_change":"2023-11-28T20:54:26.907670+0000","last_active":"2023-11-28T20:54:27.907631+0000","last_peered":"2023-11-28T20:54:27.907631+0000","last_clean":"2023-11-28T20:54:27.907631+0000","last_became_active":"2023-11-28T20:54:26.907390+0000","last_became_peered":"2023-11-28T20:54:26.907390+0000","last_unstale":"2023-11-28T20:54:27.907631+0000","last_undegraded":"2023-11-28T20:54:27.907631+0000","last_fullsized":"2023-11-28T20:54:27.907631+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:49.671092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"147'4","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.926975+0000","last_change":"2023-11-28T20:54:25.914951+0000","last_active":"2023-11-28T20:54:26.926975+0000","last_peered":"2023-11-28T20:54:26.926975+0000","last_clean":"2023-11-28T20:54:26.926975+0000","last_became_active":"2023-11-28T20:54:25.914554+0000","last_became_peered":"2023-11-28T20:54:25.914554+0000","last_unstale":"2023-11-28T20:54:26.926975+0000","last_undegraded":"2023-11-28T20:54:26.926975+0000","last_fullsized":"2023-11-28T20:54:26.926975+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:06:03.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.912148+0000","last_change":"2023-11-28T20:54:26.906955+0000","last_active":"2023-11-28T20:54:26.912148+0000","last_peered":"2023-11-28T20:54:26.912148+0000","last_clean":"2023-11-28T20:54:26.912148+0000","last_became_active":"2023-11-28T20:54:26.906649+0000","last_became_peered":"2023-11-28T20:54:26.906649+0000","last_unstale":"2023-11-28T20:54:26.912148+0000","last_undegraded":"2023-11-28T20:54:26.912148+0000","last_fullsized":"2023-11-28T20:54:26.912148+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:14:12.737805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"147'1","reported_seq":13,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907838+0000","last_change":"2023-11-28T20:54:25.912000+0000","last_active":"2023-11-28T20:54:27.907838+0000","last_peered":"2023-11-28T20:54:27.907838+0000","last_clean":"2023-11-28T20:54:27.907838+0000","last_became_active":"2023-11-28T20:54:25.911419+0000","last_became_peered":"2023-11-28T20:54:25.911419+0000","last_unstale":"2023-11-28T20:54:27.907838+0000","last_undegraded":"2023-11-28T20:54:27.907838+0000","last_fullsized":"2023-11-28T20:54:27.907838+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:06:03.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910616+0000","last_change":"2023-11-28T20:54:26.907262+0000","last_active":"2023-11-28T20:54:27.910616+0000","last_peered":"2023-11-28T20:54:27.910616+0000","last_clean":"2023-11-28T20:54:27.910616+0000","last_became_active":"2023-11-28T20:54:26.905353+0000","last_became_peered":"2023-11-28T20:54:26.905353+0000","last_unstale":"2023-11-28T20:54:27.910616+0000","last_undegraded":"2023-11-28T20:54:27.910616+0000","last_fullsized":"2023-11-28T20:54:27.910616+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:11.319501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"147'2","reported_seq":14,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242041+0000","last_change":"2023-11-28T20:54:25.917110+0000","last_active":"2023-11-28T20:54:28.242041+0000","last_peered":"2023-11-28T20:54:28.242041+0000","last_clean":"2023-11-28T20:54:28.242041+0000","last_became_active":"2023-11-28T20:54:25.916814+0000","last_became_peered":"2023-11-28T20:54:25.916814+0000","last_unstale":"2023-11-28T20:54:28.242041+0000","last_undegraded":"2023-11-28T20:54:28.242041+0000","last_fullsized":"2023-11-28T20:54:28.242041+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:30:17.377825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242591+0000","last_change":"2023-11-28T20:54:26.916338+0000","last_active":"2023-11-28T20:54:28.242591+0000","last_peered":"2023-11-28T20:54:28.242591+0000","last_clean":"2023-11-28T20:54:28.242591+0000","last_became_active":"2023-11-28T20:54:26.916054+0000","last_became_peered":"2023-11-28T20:54:26.916054+0000","last_unstale":"2023-11-28T20:54:28.242591+0000","last_undegraded":"2023-11-28T20:54:28.242591+0000","last_fullsized":"2023-11-28T20:54:28.242591+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:06:31.250814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"147'2","reported_seq":14,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907746+0000","last_change":"2023-11-28T20:54:25.904749+0000","last_active":"2023-11-28T20:54:27.907746+0000","last_peered":"2023-11-28T20:54:27.907746+0000","last_clean":"2023-11-28T20:54:27.907746+0000","last_became_active":"2023-11-28T20:54:25.904472+0000","last_became_peered":"2023-11-28T20:54:25.904472+0000","last_unstale":"2023-11-28T20:54:27.907746+0000","last_undegraded":"2023-11-28T20:54:27.907746+0000","last_fullsized":"2023-11-28T20:54:27.907746+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:07.237982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907407+0000","last_change":"2023-11-28T20:54:26.898236+0000","last_active":"2023-11-28T20:54:27.907407+0000","last_peered":"2023-11-28T20:54:27.907407+0000","last_clean":"2023-11-28T20:54:27.907407+0000","last_became_active":"2023-11-28T20:54:26.897958+0000","last_became_peered":"2023-11-28T20:54:26.897958+0000","last_unstale":"2023-11-28T20:54:27.907407+0000","last_undegraded":"2023-11-28T20:54:27.907407+0000","last_fullsized":"2023-11-28T20:54:27.907407+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:30:15.906506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"147'1","reported_seq":13,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.912819+0000","last_change":"2023-11-28T20:54:25.900126+0000","last_active":"2023-11-28T20:54:27.912819+0000","last_peered":"2023-11-28T20:54:27.912819+0000","last_clean":"2023-11-28T20:54:27.912819+0000","last_became_active":"2023-11-28T20:54:25.899820+0000","last_became_peered":"2023-11-28T20:54:25.899820+0000","last_unstale":"2023-11-28T20:54:27.912819+0000","last_undegraded":"2023-11-28T20:54:27.912819+0000","last_fullsized":"2023-11-28T20:54:27.912819+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:36:01.280635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910628+0000","last_change":"2023-11-28T20:54:26.909047+0000","last_active":"2023-11-28T20:54:27.910628+0000","last_peered":"2023-11-28T20:54:27.910628+0000","last_clean":"2023-11-28T20:54:27.910628+0000","last_became_active":"2023-11-28T20:54:26.908742+0000","last_became_peered":"2023-11-28T20:54:26.908742+0000","last_unstale":"2023-11-28T20:54:27.910628+0000","last_undegraded":"2023-11-28T20:54:27.910628+0000","last_fullsized":"2023-11-28T20:54:27.910628+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:10:17.176634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"147'1","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.929797+0000","last_change":"2023-11-28T20:54:25.911939+0000","last_active":"2023-11-28T20:54:26.929797+0000","last_peered":"2023-11-28T20:54:26.929797+0000","last_clean":"2023-11-28T20:54:26.929797+0000","last_became_active":"2023-11-28T20:54:25.911702+0000","last_became_peered":"2023-11-28T20:54:25.911702+0000","last_unstale":"2023-11-28T20:54:26.929797+0000","last_undegraded":"2023-11-28T20:54:26.929797+0000","last_fullsized":"2023-11-28T20:54:26.929797+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:00:24.337281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924170+0000","last_change":"2023-11-28T20:54:26.904686+0000","last_active":"2023-11-28T20:54:27.924170+0000","last_peered":"2023-11-28T20:54:27.924170+0000","last_clean":"2023-11-28T20:54:27.924170+0000","last_became_active":"2023-11-28T20:54:26.904341+0000","last_became_peered":"2023-11-28T20:54:26.904341+0000","last_unstale":"2023-11-28T20:54:27.924170+0000","last_undegraded":"2023-11-28T20:54:27.924170+0000","last_fullsized":"2023-11-28T20:54:27.924170+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:54.343728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910300+0000","last_change":"2023-11-28T20:54:25.911743+0000","last_active":"2023-11-28T20:54:27.910300+0000","last_peered":"2023-11-28T20:54:27.910300+0000","last_clean":"2023-11-28T20:54:27.910300+0000","last_became_active":"2023-11-28T20:54:25.911469+0000","last_became_peered":"2023-11-28T20:54:25.911469+0000","last_unstale":"2023-11-28T20:54:27.910300+0000","last_undegraded":"2023-11-28T20:54:27.910300+0000","last_fullsized":"2023-11-28T20:54:27.910300+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:52.453947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907763+0000","last_change":"2023-11-28T20:54:26.911826+0000","last_active":"2023-11-28T20:54:27.907763+0000","last_peered":"2023-11-28T20:54:27.907763+0000","last_clean":"2023-11-28T20:54:27.907763+0000","last_became_active":"2023-11-28T20:54:26.911315+0000","last_became_peered":"2023-11-28T20:54:26.911315+0000","last_unstale":"2023-11-28T20:54:27.907763+0000","last_undegraded":"2023-11-28T20:54:27.907763+0000","last_fullsized":"2023-11-28T20:54:27.907763+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:42:14.025082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.881503+0000","last_change":"2023-11-28T20:54:25.890612+0000","last_active":"2023-11-28T20:54:26.881503+0000","last_peered":"2023-11-28T20:54:26.881503+0000","last_clean":"2023-11-28T20:54:26.881503+0000","last_became_active":"2023-11-28T20:54:25.890264+0000","last_became_peered":"2023-11-28T20:54:25.890264+0000","last_unstale":"2023-11-28T20:54:26.881503+0000","last_undegraded":"2023-11-28T20:54:26.881503+0000","last_fullsized":"2023-11-28T20:54:26.881503+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:34.326582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.908438+0000","last_change":"2023-11-28T20:54:26.899829+0000","last_active":"2023-11-28T20:54:26.908438+0000","last_peered":"2023-11-28T20:54:26.908438+0000","last_clean":"2023-11-28T20:54:26.908438+0000","last_became_active":"2023-11-28T20:54:26.899431+0000","last_became_peered":"2023-11-28T20:54:26.899431+0000","last_unstale":"2023-11-28T20:54:26.908438+0000","last_undegraded":"2023-11-28T20:54:26.908438+0000","last_fullsized":"2023-11-28T20:54:26.908438+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:05:15.663983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.876400+0000","last_change":"2023-11-28T20:54:25.917006+0000","last_active":"2023-11-28T20:54:26.876400+0000","last_peered":"2023-11-28T20:54:26.876400+0000","last_clean":"2023-11-28T20:54:26.876400+0000","last_became_active":"2023-11-28T20:54:25.916076+0000","last_became_peered":"2023-11-28T20:54:25.916076+0000","last_unstale":"2023-11-28T20:54:26.876400+0000","last_undegraded":"2023-11-28T20:54:26.876400+0000","last_fullsized":"2023-11-28T20:54:26.876400+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:49:17.549764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911531+0000","last_change":"2023-11-28T20:54:26.912681+0000","last_active":"2023-11-28T20:54:27.911531+0000","last_peered":"2023-11-28T20:54:27.911531+0000","last_clean":"2023-11-28T20:54:27.911531+0000","last_became_active":"2023-11-28T20:54:26.911684+0000","last_became_peered":"2023-11-28T20:54:26.911684+0000","last_unstale":"2023-11-28T20:54:27.911531+0000","last_undegraded":"2023-11-28T20:54:27.911531+0000","last_fullsized":"2023-11-28T20:54:27.911531+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:57:46.037079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910858+0000","last_change":"2023-11-28T20:54:25.910213+0000","last_active":"2023-11-28T20:54:27.910858+0000","last_peered":"2023-11-28T20:54:27.910858+0000","last_clean":"2023-11-28T20:54:27.910858+0000","last_became_active":"2023-11-28T20:54:25.909938+0000","last_became_peered":"2023-11-28T20:54:25.909938+0000","last_unstale":"2023-11-28T20:54:27.910858+0000","last_undegraded":"2023-11-28T20:54:27.910858+0000","last_fullsized":"2023-11-28T20:54:27.910858+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:00:48.327923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924591+0000","last_change":"2023-11-28T20:54:26.921276+0000","last_active":"2023-11-28T20:54:27.924591+0000","last_peered":"2023-11-28T20:54:27.924591+0000","last_clean":"2023-11-28T20:54:27.924591+0000","last_became_active":"2023-11-28T20:54:26.920674+0000","last_became_peered":"2023-11-28T20:54:26.920674+0000","last_unstale":"2023-11-28T20:54:27.924591+0000","last_undegraded":"2023-11-28T20:54:27.924591+0000","last_fullsized":"2023-11-28T20:54:27.924591+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:01:36.857880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"147'3","reported_seq":15,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911778+0000","last_change":"2023-11-28T20:54:25.914806+0000","last_active":"2023-11-28T20:54:27.911778+0000","last_peered":"2023-11-28T20:54:27.911778+0000","last_clean":"2023-11-28T20:54:27.911778+0000","last_became_active":"2023-11-28T20:54:25.914512+0000","last_became_peered":"2023-11-28T20:54:25.914512+0000","last_unstale":"2023-11-28T20:54:27.911778+0000","last_undegraded":"2023-11-28T20:54:27.911778+0000","last_fullsized":"2023-11-28T20:54:27.911778+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:09:34.932833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.913017+0000","last_change":"2023-11-28T20:54:26.891699+0000","last_active":"2023-11-28T20:54:27.913017+0000","last_peered":"2023-11-28T20:54:27.913017+0000","last_clean":"2023-11-28T20:54:27.913017+0000","last_became_active":"2023-11-28T20:54:26.891088+0000","last_became_peered":"2023-11-28T20:54:26.891088+0000","last_unstale":"2023-11-28T20:54:27.913017+0000","last_undegraded":"2023-11-28T20:54:27.913017+0000","last_fullsized":"2023-11-28T20:54:27.913017+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:07:34.024717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"147'6","reported_seq":18,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924500+0000","last_change":"2023-11-28T20:54:25.912593+0000","last_active":"2023-11-28T20:54:27.924500+0000","last_peered":"2023-11-28T20:54:27.924500+0000","last_clean":"2023-11-28T20:54:27.924500+0000","last_became_active":"2023-11-28T20:54:25.912298+0000","last_became_peered":"2023-11-28T20:54:25.912298+0000","last_unstale":"2023-11-28T20:54:27.924500+0000","last_undegraded":"2023-11-28T20:54:27.924500+0000","last_fullsized":"2023-11-28T20:54:27.924500+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:19:00.813250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.912592+0000","last_change":"2023-11-28T20:54:26.910666+0000","last_active":"2023-11-28T20:54:26.912592+0000","last_peered":"2023-11-28T20:54:26.912592+0000","last_clean":"2023-11-28T20:54:26.912592+0000","last_became_active":"2023-11-28T20:54:26.910160+0000","last_became_peered":"2023-11-28T20:54:26.910160+0000","last_unstale":"2023-11-28T20:54:26.912592+0000","last_undegraded":"2023-11-28T20:54:26.912592+0000","last_fullsized":"2023-11-28T20:54:26.912592+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:52:29.235644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"147'2","reported_seq":14,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924551+0000","last_change":"2023-11-28T20:54:25.914232+0000","last_active":"2023-11-28T20:54:27.924551+0000","last_peered":"2023-11-28T20:54:27.924551+0000","last_clean":"2023-11-28T20:54:27.924551+0000","last_became_active":"2023-11-28T20:54:25.913929+0000","last_became_peered":"2023-11-28T20:54:25.913929+0000","last_unstale":"2023-11-28T20:54:27.924551+0000","last_undegraded":"2023-11-28T20:54:27.924551+0000","last_fullsized":"2023-11-28T20:54:27.924551+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:24.637432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907357+0000","last_change":"2023-11-28T20:54:26.899880+0000","last_active":"2023-11-28T20:54:27.907357+0000","last_peered":"2023-11-28T20:54:27.907357+0000","last_clean":"2023-11-28T20:54:27.907357+0000","last_became_active":"2023-11-28T20:54:26.899588+0000","last_became_peered":"2023-11-28T20:54:26.899588+0000","last_unstale":"2023-11-28T20:54:27.907357+0000","last_undegraded":"2023-11-28T20:54:27.907357+0000","last_fullsized":"2023-11-28T20:54:27.907357+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:53:32.854588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924347+0000","last_change":"2023-11-28T20:54:25.910564+0000","last_active":"2023-11-28T20:54:27.924347+0000","last_peered":"2023-11-28T20:54:27.924347+0000","last_clean":"2023-11-28T20:54:27.924347+0000","last_became_active":"2023-11-28T20:54:25.910102+0000","last_became_peered":"2023-11-28T20:54:25.910102+0000","last_unstale":"2023-11-28T20:54:27.924347+0000","last_undegraded":"2023-11-28T20:54:27.924347+0000","last_fullsized":"2023-11-28T20:54:27.924347+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:28:14.857308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.543720+0000","last_change":"2023-11-28T20:54:26.909144+0000","last_active":"2023-11-28T20:54:28.543720+0000","last_peered":"2023-11-28T20:54:28.543720+0000","last_clean":"2023-11-28T20:54:28.543720+0000","last_became_active":"2023-11-28T20:54:26.908825+0000","last_became_peered":"2023-11-28T20:54:26.908825+0000","last_unstale":"2023-11-28T20:54:28.543720+0000","last_undegraded":"2023-11-28T20:54:28.543720+0000","last_fullsized":"2023-11-28T20:54:28.543720+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:38:29.715808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924212+0000","last_change":"2023-11-28T20:54:25.914232+0000","last_active":"2023-11-28T20:54:27.924212+0000","last_peered":"2023-11-28T20:54:27.924212+0000","last_clean":"2023-11-28T20:54:27.924212+0000","last_became_active":"2023-11-28T20:54:25.913951+0000","last_became_peered":"2023-11-28T20:54:25.913951+0000","last_unstale":"2023-11-28T20:54:27.924212+0000","last_undegraded":"2023-11-28T20:54:27.924212+0000","last_fullsized":"2023-11-28T20:54:27.924212+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:36:55.708527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.912495+0000","last_change":"2023-11-28T20:54:26.906560+0000","last_active":"2023-11-28T20:54:27.912495+0000","last_peered":"2023-11-28T20:54:27.912495+0000","last_clean":"2023-11-28T20:54:27.912495+0000","last_became_active":"2023-11-28T20:54:26.906242+0000","last_became_peered":"2023-11-28T20:54:26.906242+0000","last_unstale":"2023-11-28T20:54:27.912495+0000","last_undegraded":"2023-11-28T20:54:27.912495+0000","last_fullsized":"2023-11-28T20:54:27.912495+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:38:05.965491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"147'1","reported_seq":13,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907873+0000","last_change":"2023-11-28T20:54:25.907756+0000","last_active":"2023-11-28T20:54:27.907873+0000","last_peered":"2023-11-28T20:54:27.907873+0000","last_clean":"2023-11-28T20:54:27.907873+0000","last_became_active":"2023-11-28T20:54:25.907404+0000","last_became_peered":"2023-11-28T20:54:25.907404+0000","last_unstale":"2023-11-28T20:54:27.907873+0000","last_undegraded":"2023-11-28T20:54:27.907873+0000","last_fullsized":"2023-11-28T20:54:27.907873+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:47:25.267556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911326+0000","last_change":"2023-11-28T20:54:26.912380+0000","last_active":"2023-11-28T20:54:27.911326+0000","last_peered":"2023-11-28T20:54:27.911326+0000","last_clean":"2023-11-28T20:54:27.911326+0000","last_became_active":"2023-11-28T20:54:26.909682+0000","last_became_peered":"2023-11-28T20:54:26.909682+0000","last_unstale":"2023-11-28T20:54:27.911326+0000","last_undegraded":"2023-11-28T20:54:27.911326+0000","last_fullsized":"2023-11-28T20:54:27.911326+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:10:27.576002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"147'2","reported_seq":14,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544223+0000","last_change":"2023-11-28T20:54:25.908240+0000","last_active":"2023-11-28T20:54:28.544223+0000","last_peered":"2023-11-28T20:54:28.544223+0000","last_clean":"2023-11-28T20:54:28.544223+0000","last_became_active":"2023-11-28T20:54:25.907837+0000","last_became_peered":"2023-11-28T20:54:25.907837+0000","last_unstale":"2023-11-28T20:54:28.544223+0000","last_undegraded":"2023-11-28T20:54:28.544223+0000","last_fullsized":"2023-11-28T20:54:28.544223+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:52:24.315057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.905174+0000","last_change":"2023-11-28T20:54:26.898938+0000","last_active":"2023-11-28T20:54:26.905174+0000","last_peered":"2023-11-28T20:54:26.905174+0000","last_clean":"2023-11-28T20:54:26.905174+0000","last_became_active":"2023-11-28T20:54:26.898421+0000","last_became_peered":"2023-11-28T20:54:26.898421+0000","last_unstale":"2023-11-28T20:54:26.905174+0000","last_undegraded":"2023-11-28T20:54:26.905174+0000","last_fullsized":"2023-11-28T20:54:26.905174+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:27:10.365990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"147'1","reported_seq":13,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242177+0000","last_change":"2023-11-28T20:54:25.911705+0000","last_active":"2023-11-28T20:54:28.242177+0000","last_peered":"2023-11-28T20:54:28.242177+0000","last_clean":"2023-11-28T20:54:28.242177+0000","last_became_active":"2023-11-28T20:54:25.911347+0000","last_became_peered":"2023-11-28T20:54:25.911347+0000","last_unstale":"2023-11-28T20:54:28.242177+0000","last_undegraded":"2023-11-28T20:54:28.242177+0000","last_fullsized":"2023-11-28T20:54:28.242177+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:49.888966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.913196+0000","last_change":"2023-11-28T20:54:26.897634+0000","last_active":"2023-11-28T20:54:27.913196+0000","last_peered":"2023-11-28T20:54:27.913196+0000","last_clean":"2023-11-28T20:54:27.913196+0000","last_became_active":"2023-11-28T20:54:26.897181+0000","last_became_peered":"2023-11-28T20:54:26.897181+0000","last_unstale":"2023-11-28T20:54:27.913196+0000","last_undegraded":"2023-11-28T20:54:27.913196+0000","last_fullsized":"2023-11-28T20:54:27.913196+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:09.074880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242241+0000","last_change":"2023-11-28T20:54:25.911644+0000","last_active":"2023-11-28T20:54:28.242241+0000","last_peered":"2023-11-28T20:54:28.242241+0000","last_clean":"2023-11-28T20:54:28.242241+0000","last_became_active":"2023-11-28T20:54:25.911304+0000","last_became_peered":"2023-11-28T20:54:25.911304+0000","last_unstale":"2023-11-28T20:54:28.242241+0000","last_undegraded":"2023-11-28T20:54:28.242241+0000","last_fullsized":"2023-11-28T20:54:28.242241+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:00:48.327923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.905183+0000","last_change":"2023-11-28T20:54:26.898906+0000","last_active":"2023-11-28T20:54:26.905183+0000","last_peered":"2023-11-28T20:54:26.905183+0000","last_clean":"2023-11-28T20:54:26.905183+0000","last_became_active":"2023-11-28T20:54:26.897627+0000","last_became_peered":"2023-11-28T20:54:26.897627+0000","last_unstale":"2023-11-28T20:54:26.905183+0000","last_undegraded":"2023-11-28T20:54:26.905183+0000","last_fullsized":"2023-11-28T20:54:26.905183+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:46.731684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"147'2","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.927775+0000","last_change":"2023-11-28T20:54:25.900134+0000","last_active":"2023-11-28T20:54:26.927775+0000","last_peered":"2023-11-28T20:54:26.927775+0000","last_clean":"2023-11-28T20:54:26.927775+0000","last_became_active":"2023-11-28T20:54:25.899909+0000","last_became_peered":"2023-11-28T20:54:25.899909+0000","last_unstale":"2023-11-28T20:54:26.927775+0000","last_undegraded":"2023-11-28T20:54:26.927775+0000","last_fullsized":"2023-11-28T20:54:26.927775+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:55.877897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924046+0000","last_change":"2023-11-28T20:54:26.908487+0000","last_active":"2023-11-28T20:54:27.924046+0000","last_peered":"2023-11-28T20:54:27.924046+0000","last_clean":"2023-11-28T20:54:27.924046+0000","last_became_active":"2023-11-28T20:54:26.908196+0000","last_became_peered":"2023-11-28T20:54:26.908196+0000","last_unstale":"2023-11-28T20:54:27.924046+0000","last_undegraded":"2023-11-28T20:54:27.924046+0000","last_fullsized":"2023-11-28T20:54:27.924046+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:45:12.736796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241687+0000","last_change":"2023-11-28T20:54:25.908639+0000","last_active":"2023-11-28T20:54:28.241687+0000","last_peered":"2023-11-28T20:54:28.241687+0000","last_clean":"2023-11-28T20:54:28.241687+0000","last_became_active":"2023-11-28T20:54:25.908297+0000","last_became_peered":"2023-11-28T20:54:25.908297+0000","last_unstale":"2023-11-28T20:54:28.241687+0000","last_undegraded":"2023-11-28T20:54:28.241687+0000","last_fullsized":"2023-11-28T20:54:28.241687+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:52.453947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924292+0000","last_change":"2023-11-28T20:54:26.905792+0000","last_active":"2023-11-28T20:54:27.924292+0000","last_peered":"2023-11-28T20:54:27.924292+0000","last_clean":"2023-11-28T20:54:27.924292+0000","last_became_active":"2023-11-28T20:54:26.905380+0000","last_became_peered":"2023-11-28T20:54:26.905380+0000","last_unstale":"2023-11-28T20:54:27.924292+0000","last_undegraded":"2023-11-28T20:54:27.924292+0000","last_fullsized":"2023-11-28T20:54:27.924292+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:46:26.771636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924385+0000","last_change":"2023-11-28T20:54:25.903983+0000","last_active":"2023-11-28T20:54:27.924385+0000","last_peered":"2023-11-28T20:54:27.924385+0000","last_clean":"2023-11-28T20:54:27.924385+0000","last_became_active":"2023-11-28T20:54:25.903538+0000","last_became_peered":"2023-11-28T20:54:25.903538+0000","last_unstale":"2023-11-28T20:54:27.924385+0000","last_undegraded":"2023-11-28T20:54:27.924385+0000","last_fullsized":"2023-11-28T20:54:27.924385+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:06:30.238274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910712+0000","last_change":"2023-11-28T20:54:26.891890+0000","last_active":"2023-11-28T20:54:27.910712+0000","last_peered":"2023-11-28T20:54:27.910712+0000","last_clean":"2023-11-28T20:54:27.910712+0000","last_became_active":"2023-11-28T20:54:26.889899+0000","last_became_peered":"2023-11-28T20:54:26.889899+0000","last_unstale":"2023-11-28T20:54:27.910712+0000","last_undegraded":"2023-11-28T20:54:27.910712+0000","last_fullsized":"2023-11-28T20:54:27.910712+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:47:47.773324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.240922+0000","last_change":"2023-11-28T20:54:25.912307+0000","last_active":"2023-11-28T20:54:28.240922+0000","last_peered":"2023-11-28T20:54:28.240922+0000","last_clean":"2023-11-28T20:54:28.240922+0000","last_became_active":"2023-11-28T20:54:25.912027+0000","last_became_peered":"2023-11-28T20:54:25.912027+0000","last_unstale":"2023-11-28T20:54:28.240922+0000","last_undegraded":"2023-11-28T20:54:28.240922+0000","last_fullsized":"2023-11-28T20:54:28.240922+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:29:15.458403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.900897+0000","last_change":"2023-11-28T20:54:26.895510+0000","last_active":"2023-11-28T20:54:26.900897+0000","last_peered":"2023-11-28T20:54:26.900897+0000","last_clean":"2023-11-28T20:54:26.900897+0000","last_became_active":"2023-11-28T20:54:26.895153+0000","last_became_peered":"2023-11-28T20:54:26.895153+0000","last_unstale":"2023-11-28T20:54:26.900897+0000","last_undegraded":"2023-11-28T20:54:26.900897+0000","last_fullsized":"2023-11-28T20:54:26.900897+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:05.996950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"147'2","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.927811+0000","last_change":"2023-11-28T20:54:25.902328+0000","last_active":"2023-11-28T20:54:26.927811+0000","last_peered":"2023-11-28T20:54:26.927811+0000","last_clean":"2023-11-28T20:54:26.927811+0000","last_became_active":"2023-11-28T20:54:25.902040+0000","last_became_peered":"2023-11-28T20:54:25.902040+0000","last_unstale":"2023-11-28T20:54:26.927811+0000","last_undegraded":"2023-11-28T20:54:26.927811+0000","last_fullsized":"2023-11-28T20:54:26.927811+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:04.854597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907890+0000","last_change":"2023-11-28T20:54:26.914128+0000","last_active":"2023-11-28T20:54:27.907890+0000","last_peered":"2023-11-28T20:54:27.907890+0000","last_clean":"2023-11-28T20:54:27.907890+0000","last_became_active":"2023-11-28T20:54:26.913858+0000","last_became_peered":"2023-11-28T20:54:26.913858+0000","last_unstale":"2023-11-28T20:54:27.907890+0000","last_undegraded":"2023-11-28T20:54:27.907890+0000","last_fullsized":"2023-11-28T20:54:27.907890+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:10:17.176634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.876323+0000","last_change":"2023-11-28T20:54:25.916939+0000","last_active":"2023-11-28T20:54:26.876323+0000","last_peered":"2023-11-28T20:54:26.876323+0000","last_clean":"2023-11-28T20:54:26.876323+0000","last_became_active":"2023-11-28T20:54:25.916431+0000","last_became_peered":"2023-11-28T20:54:25.916431+0000","last_unstale":"2023-11-28T20:54:26.876323+0000","last_undegraded":"2023-11-28T20:54:26.876323+0000","last_fullsized":"2023-11-28T20:54:26.876323+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:37:57.338637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242624+0000","last_change":"2023-11-28T20:54:26.913651+0000","last_active":"2023-11-28T20:54:28.242624+0000","last_peered":"2023-11-28T20:54:28.242624+0000","last_clean":"2023-11-28T20:54:28.242624+0000","last_became_active":"2023-11-28T20:54:26.913381+0000","last_became_peered":"2023-11-28T20:54:26.913381+0000","last_unstale":"2023-11-28T20:54:28.242624+0000","last_undegraded":"2023-11-28T20:54:28.242624+0000","last_fullsized":"2023-11-28T20:54:28.242624+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:25.649971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.876110+0000","last_change":"2023-11-28T20:54:25.908920+0000","last_active":"2023-11-28T20:54:26.876110+0000","last_peered":"2023-11-28T20:54:26.876110+0000","last_clean":"2023-11-28T20:54:26.876110+0000","last_became_active":"2023-11-28T20:54:25.908452+0000","last_became_peered":"2023-11-28T20:54:25.908452+0000","last_unstale":"2023-11-28T20:54:26.876110+0000","last_undegraded":"2023-11-28T20:54:26.876110+0000","last_fullsized":"2023-11-28T20:54:26.876110+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:07:29.380457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544565+0000","last_change":"2023-11-28T20:54:26.913188+0000","last_active":"2023-11-28T20:54:28.544565+0000","last_peered":"2023-11-28T20:54:28.544565+0000","last_clean":"2023-11-28T20:54:28.544565+0000","last_became_active":"2023-11-28T20:54:26.907468+0000","last_became_peered":"2023-11-28T20:54:26.907468+0000","last_unstale":"2023-11-28T20:54:28.544565+0000","last_undegraded":"2023-11-28T20:54:28.544565+0000","last_fullsized":"2023-11-28T20:54:28.544565+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:38.186160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.894728+0000","last_change":"2023-11-28T20:54:25.900497+0000","last_active":"2023-11-28T20:54:26.894728+0000","last_peered":"2023-11-28T20:54:26.894728+0000","last_clean":"2023-11-28T20:54:26.894728+0000","last_became_active":"2023-11-28T20:54:25.900024+0000","last_became_peered":"2023-11-28T20:54:25.900024+0000","last_unstale":"2023-11-28T20:54:26.894728+0000","last_undegraded":"2023-11-28T20:54:26.894728+0000","last_fullsized":"2023-11-28T20:54:26.894728+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:51:11.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.911173+0000","last_change":"2023-11-28T20:54:26.901224+0000","last_active":"2023-11-28T20:54:26.911173+0000","last_peered":"2023-11-28T20:54:26.911173+0000","last_clean":"2023-11-28T20:54:26.911173+0000","last_became_active":"2023-11-28T20:54:26.900674+0000","last_became_peered":"2023-11-28T20:54:26.900674+0000","last_unstale":"2023-11-28T20:54:26.911173+0000","last_undegraded":"2023-11-28T20:54:26.911173+0000","last_fullsized":"2023-11-28T20:54:26.911173+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:46:47.295747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911402+0000","last_change":"2023-11-28T20:54:25.910201+0000","last_active":"2023-11-28T20:54:27.911402+0000","last_peered":"2023-11-28T20:54:27.911402+0000","last_clean":"2023-11-28T20:54:27.911402+0000","last_became_active":"2023-11-28T20:54:25.909522+0000","last_became_peered":"2023-11-28T20:54:25.909522+0000","last_unstale":"2023-11-28T20:54:27.911402+0000","last_undegraded":"2023-11-28T20:54:27.911402+0000","last_fullsized":"2023-11-28T20:54:27.911402+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:42:13.012542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924207+0000","last_change":"2023-11-28T20:54:26.920277+0000","last_active":"2023-11-28T20:54:27.924207+0000","last_peered":"2023-11-28T20:54:27.924207+0000","last_clean":"2023-11-28T20:54:27.924207+0000","last_became_active":"2023-11-28T20:54:26.916252+0000","last_became_peered":"2023-11-28T20:54:26.916252+0000","last_unstale":"2023-11-28T20:54:27.924207+0000","last_undegraded":"2023-11-28T20:54:27.924207+0000","last_fullsized":"2023-11-28T20:54:27.924207+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:55:11.972088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911882+0000","last_change":"2023-11-28T20:54:25.905984+0000","last_active":"2023-11-28T20:54:27.911882+0000","last_peered":"2023-11-28T20:54:27.911882+0000","last_clean":"2023-11-28T20:54:27.911882+0000","last_became_active":"2023-11-28T20:54:25.904885+0000","last_became_peered":"2023-11-28T20:54:25.904885+0000","last_unstale":"2023-11-28T20:54:27.911882+0000","last_undegraded":"2023-11-28T20:54:27.911882+0000","last_fullsized":"2023-11-28T20:54:27.911882+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:19.714929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241475+0000","last_change":"2023-11-28T20:54:26.914220+0000","last_active":"2023-11-28T20:54:28.241475+0000","last_peered":"2023-11-28T20:54:28.241475+0000","last_clean":"2023-11-28T20:54:28.241475+0000","last_became_active":"2023-11-28T20:54:26.913712+0000","last_became_peered":"2023-11-28T20:54:26.913712+0000","last_unstale":"2023-11-28T20:54:28.241475+0000","last_undegraded":"2023-11-28T20:54:28.241475+0000","last_fullsized":"2023-11-28T20:54:28.241475+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:38:05.965491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.883204+0000","last_change":"2023-11-28T20:54:25.890626+0000","last_active":"2023-11-28T20:54:26.883204+0000","last_peered":"2023-11-28T20:54:26.883204+0000","last_clean":"2023-11-28T20:54:26.883204+0000","last_became_active":"2023-11-28T20:54:25.889965+0000","last_became_peered":"2023-11-28T20:54:25.889965+0000","last_unstale":"2023-11-28T20:54:26.883204+0000","last_undegraded":"2023-11-28T20:54:26.883204+0000","last_fullsized":"2023-11-28T20:54:26.883204+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:19.978558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907639+0000","last_change":"2023-11-28T20:54:26.897362+0000","last_active":"2023-11-28T20:54:27.907639+0000","last_peered":"2023-11-28T20:54:27.907639+0000","last_clean":"2023-11-28T20:54:27.907639+0000","last_became_active":"2023-11-28T20:54:26.897057+0000","last_became_peered":"2023-11-28T20:54:26.897057+0000","last_unstale":"2023-11-28T20:54:27.907639+0000","last_undegraded":"2023-11-28T20:54:27.907639+0000","last_fullsized":"2023-11-28T20:54:27.907639+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:36:02.293175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.240863+0000","last_change":"2023-11-28T20:54:25.914230+0000","last_active":"2023-11-28T20:54:28.240863+0000","last_peered":"2023-11-28T20:54:28.240863+0000","last_clean":"2023-11-28T20:54:28.240863+0000","last_became_active":"2023-11-28T20:54:25.913684+0000","last_became_peered":"2023-11-28T20:54:25.913684+0000","last_unstale":"2023-11-28T20:54:28.240863+0000","last_undegraded":"2023-11-28T20:54:28.240863+0000","last_fullsized":"2023-11-28T20:54:28.240863+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:13:16.406084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242090+0000","last_change":"2023-11-28T20:54:26.912385+0000","last_active":"2023-11-28T20:54:28.242090+0000","last_peered":"2023-11-28T20:54:28.242090+0000","last_clean":"2023-11-28T20:54:28.242090+0000","last_became_active":"2023-11-28T20:54:26.911789+0000","last_became_peered":"2023-11-28T20:54:26.911789+0000","last_unstale":"2023-11-28T20:54:28.242090+0000","last_undegraded":"2023-11-28T20:54:28.242090+0000","last_fullsized":"2023-11-28T20:54:28.242090+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:23.314439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"147'8","reported_seq":20,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.923917+0000","last_change":"2023-11-28T20:54:25.914874+0000","last_active":"2023-11-28T20:54:27.923917+0000","last_peered":"2023-11-28T20:54:27.923917+0000","last_clean":"2023-11-28T20:54:27.923917+0000","last_became_active":"2023-11-28T20:54:25.914583+0000","last_became_peered":"2023-11-28T20:54:25.914583+0000","last_unstale":"2023-11-28T20:54:27.923917+0000","last_undegraded":"2023-11-28T20:54:27.923917+0000","last_fullsized":"2023-11-28T20:54:27.923917+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:08.062340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"136'4734","reported_seq":5250,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.895014+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:54:26.895014+0000","last_peered":"2023-11-28T20:54:26.895014+0000","last_clean":"2023-11-28T20:54:26.895014+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:54:26.895014+0000","last_undegraded":"2023-11-28T20:54:26.895014+0000","last_fullsized":"2023-11-28T20:54:26.895014+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1189,"num_read_kb":2391,"num_write":371,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1189,"num_read_kb":2391,"num_write":371,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4734,"ondisk_log_size":4734,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575091,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47456,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324384,"statfs":{"total":96636764160,"available":96588169216,"internally_reserved":0,"allocated":2437120,"data_stored":1518452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.167,"15min":1.1870000000000001},"min":{"1min":0.623,"5min":0.57099999999999995,"15min":0.45600000000000002},"max":{"1min":1.7490000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.087},{"interface":"front","average":{"1min":1.097,"5min":1.089,"15min":1.1499999999999999},"min":{"1min":0.66900000000000004,"5min":0.64300000000000002,"15min":0.58299999999999996},"max":{"1min":1.5660000000000001,"5min":1.8,"15min":1.8},"last":1.4279999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1579999999999999,"15min":1.179},"min":{"1min":0.64600000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5629999999999999,"5min":1.633,"15min":1.744},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.228,"15min":1.252},"min":{"1min":0.71799999999999997,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5900000000000001,"5min":1.8540000000000001,"15min":1.889},"last":1.506}]},{"osd":2,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.272,"15min":1.2889999999999999},"min":{"1min":0.879,"5min":0.59699999999999998,"15min":0.58799999999999997},"max":{"1min":1.714,"5min":1.714,"15min":1.9359999999999999},"last":1.6279999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3280000000000001,"15min":1.298},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.57499999999999996},"max":{"1min":1.77,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.6679999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2470000000000001,"15min":1.27},"min":{"1min":0.76800000000000002,"5min":0.73199999999999998,"15min":0.54200000000000004},"max":{"1min":1.583,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.292},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4139999999999999,"15min":1.4099999999999999},"min":{"1min":1.0609999999999999,"5min":0.77000000000000002,"15min":0.68300000000000005},"max":{"1min":1.7869999999999999,"5min":1.956,"15min":1.956},"last":1.9610000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.335,"15min":1.29},"min":{"1min":0.89800000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0430000000000001,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.573},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.347,"15min":1.3320000000000001},"min":{"1min":1.1399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.877,"5min":1.99,"15min":1.9970000000000001},"last":1.385}]},{"osd":5,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.4390000000000001,"15min":1.4179999999999999},"min":{"1min":1.1919999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.891,"5min":1.891,"15min":2.048},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.496,"15min":1.474},"min":{"1min":1.288,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.7330000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.581,"15min":1.5589999999999999},"min":{"1min":1,"5min":0.85799999999999998,"15min":0.69499999999999995},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.476,"15min":1.4830000000000001},"min":{"1min":1.0640000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7969999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.1850000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575091,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37132,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1830912,"data_stored":927890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0469999999999999,"15min":1.095},"min":{"1min":0.745,"5min":0.58099999999999996,"15min":0.53500000000000003},"max":{"1min":1.5900000000000001,"5min":1.6319999999999999,"15min":1.7350000000000001},"last":1.222},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.091,"15min":1.1299999999999999},"min":{"1min":0.69699999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4059999999999999,"5min":1.8069999999999999,"15min":1.8260000000000001},"last":1.1639999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.196,"15min":1.24},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.651,"5min":1.728,"15min":1.8069999999999999},"last":1.343},{"interface":"front","average":{"1min":1.385,"5min":1.292,"15min":1.349},"min":{"1min":1.1100000000000001,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.6759999999999999,"5min":1.7889999999999999,"15min":2.0169999999999999},"last":1.4910000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.2470000000000001,"15min":1.2829999999999999},"min":{"1min":0.88300000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":1.429},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.3200000000000001,"15min":1.361},"min":{"1min":1.2190000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.837,"5min":1.851,"15min":1.851},"last":1.4019999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.264,"15min":1.28},"min":{"1min":0.75,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.9299999999999999},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.258,"15min":1.2789999999999999},"min":{"1min":1.002,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.7450000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.091}]},{"osd":4,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.266,"15min":1.246},"min":{"1min":0.78300000000000003,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.6479999999999999,"5min":1.7529999999999999,"15min":2.089},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.298,"5min":1.3089999999999999,"15min":1.3049999999999999},"min":{"1min":0.92400000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.798,"5min":1.9179999999999999,"15min":2.2010000000000001},"last":1.27}]},{"osd":5,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.351,"15min":1.337},"min":{"1min":0.95999999999999996,"5min":0.88800000000000001,"15min":0.754},"max":{"1min":1.879,"5min":1.879,"15min":1.97},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.514,"15min":1.528},"min":{"1min":1.0669999999999999,"5min":0.93400000000000005,"15min":0.78000000000000003},"max":{"1min":1.901,"5min":1.9179999999999999,"15min":1.958},"last":1.526}]},{"osd":7,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4399999999999999,"15min":1.46},"min":{"1min":0.94999999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.849,"5min":1.9790000000000001,"15min":2.3759999999999999},"last":1.377},{"interface":"front","average":{"1min":1.425,"5min":1.4550000000000001,"15min":1.4530000000000001},"min":{"1min":1.1599999999999999,"5min":0.96099999999999997,"15min":0.92900000000000005},"max":{"1min":1.9259999999999999,"5min":1.9630000000000001,"15min":2.0739999999999998},"last":1.5489999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575091,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51032,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320808,"statfs":{"total":96636764160,"available":96584507392,"internally_reserved":0,"allocated":1847296,"data_stored":934116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1619999999999999,"15min":1.1639999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.696,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.091},{"interface":"front","average":{"1min":1.056,"5min":1.0660000000000001,"15min":1.1439999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.441},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1160000000000001,"15min":1.147},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.623,"5min":1.623,"15min":1.768},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.2190000000000001,"15min":1.2130000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6100000000000001,"5min":2.282,"15min":2.282},"last":0.97299999999999998}]},{"osd":2,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.302,"15min":1.2829999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.53200000000000003},"max":{"1min":1.893,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.212,"5min":1.21,"15min":1.274},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.66600000000000004},"max":{"1min":1.6240000000000001,"5min":1.8220000000000001,"15min":1.958},"last":1.1279999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.286,"15min":1.3140000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.726,"5min":1.7290000000000001,"15min":1.9350000000000001},"last":1.327},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.2589999999999999,"15min":1.262},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.64000000000000001},"max":{"1min":1.6559999999999999,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.361}]},{"osd":4,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3140000000000001,"15min":1.2989999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.67500000000000004},"max":{"1min":1.746,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.585},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4710000000000001,"15min":1.4379999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.8160000000000001,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":1.284}]},{"osd":6,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3380000000000001,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.5},"max":{"1min":1.966,"5min":1.966,"15min":2.093},"last":1.02},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.474,"15min":1.476},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.69499999999999995},"max":{"1min":1.9219999999999999,"5min":1.998,"15min":1.998},"last":1.661}]},{"osd":7,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.464,"15min":1.4330000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.742,"5min":3.319,"15min":3.319},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.474,"15min":1.4830000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.60499999999999998},"max":{"1min":1.764,"5min":1.9790000000000001,"15min":2.1829999999999998},"last":1.234}]}]},{"osd":4,"up_from":13,"seq":55834575092,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37152,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334688,"statfs":{"total":96636764160,"available":96598720512,"internally_reserved":0,"allocated":1851392,"data_stored":935568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0580000000000001,"15min":1.125},"min":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.825,"5min":1.825,"15min":1.879},"last":1.5},{"interface":"front","average":{"1min":0.995,"5min":1.0720000000000001,"15min":1.1080000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.3839999999999999,"5min":1.6850000000000001,"15min":1.9750000000000001},"last":0.751}]},{"osd":1,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2110000000000001,"15min":1.286},"min":{"1min":0.68999999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.923},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.335,"5min":1.2689999999999999,"15min":1.343},"min":{"1min":0.72099999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7470000000000001,"5min":1.768,"15min":2.2679999999999998},"last":1.546}]},{"osd":2,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2010000000000001,"15min":1.2190000000000001},"min":{"1min":0.56899999999999995,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.7729999999999999,"5min":1.78,"15min":1.78},"last":1.002},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2350000000000001,"15min":1.282},"min":{"1min":0.65000000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.704,"5min":1.704,"15min":2.6949999999999998},"last":2.218}]},{"osd":3,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.2230000000000001,"15min":1.2709999999999999},"min":{"1min":0.63,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.647,"5min":1.9630000000000001,"15min":2.157},"last":4.117},{"interface":"front","average":{"1min":1.3,"5min":1.3149999999999999,"15min":1.3580000000000001},"min":{"1min":0.68700000000000006,"5min":0.5,"15min":0.5},"max":{"1min":1.7509999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":4.843}]},{"osd":5,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.286,"15min":1.292},"min":{"1min":0.65500000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.849,"5min":1.9970000000000001,"15min":2.4279999999999999},"last":3.262},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.2909999999999999,"15min":1.298},"min":{"1min":0.746,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.784,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.3600000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.3979999999999999,"15min":1.395},"min":{"1min":0.96699999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.8899999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.5289999999999999,"15min":1.5509999999999999},"min":{"1min":0.92400000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.9219999999999999,"5min":2.1349999999999998,"15min":3.7000000000000002},"last":3.3079999999999998}]},{"osd":7,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.444,"15min":1.46},"min":{"1min":0.76200000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.9159999999999999,"5min":2.2490000000000001,"15min":2.4990000000000001},"last":3.8620000000000001},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.522,"15min":1.518},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.0510000000000002,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":2.4700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575092,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41312,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330528,"statfs":{"total":96636764160,"available":96594460672,"internally_reserved":0,"allocated":1855488,"data_stored":935602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.85699999999999998,"15min":0.89400000000000002},"min":{"1min":0.47299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.383,"5min":1.6000000000000001,"15min":4.7210000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.92800000000000005,"15min":1.0229999999999999},"min":{"1min":0.39700000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.73,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":0.628}]},{"osd":1,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.92900000000000005,"15min":0.96099999999999997},"min":{"1min":0.495,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6739999999999999,"5min":1.9770000000000001,"15min":2.8460000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.071,"15min":1.1719999999999999},"min":{"1min":0.53100000000000003,"5min":0.379,"15min":0.379},"max":{"1min":1.4219999999999999,"5min":1.7190000000000001,"15min":3.7410000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1140000000000001,"15min":1.216},"min":{"1min":0.65900000000000003,"5min":0.53100000000000003,"15min":0.48499999999999999},"max":{"1min":1.5429999999999999,"5min":1.9019999999999999,"15min":3.444},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.04,"5min":1.2030000000000001,"15min":1.3029999999999999},"min":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.452,"5min":1.9530000000000001,"15min":3.7050000000000001},"last":0.86799999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2849999999999999,"15min":1.335},"min":{"1min":0.76000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8089999999999999,"5min":1.956,"15min":2.9420000000000002},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.3300000000000001,"15min":1.3700000000000001},"min":{"1min":0.91400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.8819999999999999,"5min":2.246,"15min":2.246},"last":1.3140000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3799999999999999,"15min":1.4339999999999999},"min":{"1min":0.88800000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6659999999999999,"5min":1.8959999999999999,"15min":3.7730000000000001},"last":1.417},{"interface":"front","average":{"1min":1.379,"5min":1.387,"15min":1.4330000000000001},"min":{"1min":0.81100000000000005,"5min":0.66800000000000004,"15min":0.53500000000000003},"max":{"1min":1.875,"5min":2.0110000000000001,"15min":3.1520000000000001},"last":1.1879999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.399,"15min":1.458},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.825,"5min":2.0630000000000002,"15min":2.3100000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.46,"15min":1.5509999999999999},"min":{"1min":0.97599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.679,"5min":2.0840000000000001,"15min":3.661},"last":1.4970000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.518},"min":{"1min":0.83199999999999996,"5min":0.73399999999999999,"15min":0.71499999999999997},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":3.266},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.522,"15min":1.575},"min":{"1min":0.996,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.7450000000000001,"5min":2.2080000000000002,"15min":2.3919999999999999},"last":1.216}]}]},{"osd":2,"up_from":13,"seq":55834575092,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":1826816,"data_stored":927308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.86199999999999999,"15min":0.90200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":1.2829999999999999,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.0049999999999999,"15min":1.052},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.387,"5min":4.8490000000000002,"15min":4.8490000000000002},"last":1.216}]},{"osd":1,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.96899999999999997,"15min":0.98499999999999999},"min":{"1min":0.498,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.458,"5min":1.8160000000000001,"15min":3.2599999999999998},"last":1.069},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0309999999999999,"15min":1.1240000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.357,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.97099999999999997}]},{"osd":3,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1160000000000001,"15min":1.161},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.589,"5min":4.9349999999999996,"15min":4.9349999999999996},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1970000000000001,"15min":1.282},"min":{"1min":0.64900000000000002,"5min":0.48499999999999999,"15min":0.48199999999999998},"max":{"1min":1.994,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":1.2769999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3440000000000001,"15min":1.3979999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.673,"5min":3.6499999999999999,"15min":3.6499999999999999},"last":1.538},{"interface":"front","average":{"1min":1.363,"5min":1.448,"15min":1.4350000000000001},"min":{"1min":0.88400000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8340000000000001,"5min":4.6479999999999997,"15min":4.6479999999999997},"last":1.508}]},{"osd":5,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3819999999999999,"15min":1.413},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.923,"5min":2.492,"15min":2.492},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.429,"15min":1.4830000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6399999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.3420000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4810000000000001,"15min":1.508},"min":{"1min":0.79200000000000004,"5min":0.747,"15min":0.747},"max":{"1min":2.153,"5min":4.5270000000000001,"15min":4.5270000000000001},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.5329999999999999,"15min":1.5660000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.82499999999999996},"max":{"1min":1.8560000000000001,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.4730000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4650000000000001,"15min":1.524},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.0720000000000001,"5min":2.641,"15min":2.641},"last":1.78},{"interface":"front","average":{"1min":1.484,"5min":1.548,"15min":1.585},"min":{"1min":0.81799999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.2770000000000001,"5min":4.7530000000000001,"15min":4.7530000000000001},"last":1.6539999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575092,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37144,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334696,"statfs":{"total":96636764160,"available":96598728704,"internally_reserved":0,"allocated":1843200,"data_stored":933978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.90800000000000003,"15min":0.91100000000000003},"min":{"1min":0.38900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.466,"5min":1.8009999999999999,"15min":1.9259999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.98899999999999999,"15min":1.069},"min":{"1min":0.505,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.6120000000000001,"5min":2.028,"15min":2.028},"last":0.90800000000000003}]},{"osd":2,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0389999999999999,"15min":1.0740000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.371},"max":{"1min":1.3959999999999999,"5min":1.766,"15min":2.234},"last":0.498},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.153,"15min":1.1910000000000001},"min":{"1min":0.628,"5min":0.53700000000000003,"15min":0.52800000000000002},"max":{"1min":1.4259999999999999,"5min":1.9199999999999999,"15min":2.895},"last":1.198}]},{"osd":3,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.2110000000000001,"15min":1.2889999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.46700000000000003},"max":{"1min":1.5109999999999999,"5min":1.7949999999999999,"15min":2.0459999999999998},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.0760000000000001,"15min":1.101},"min":{"1min":0.47299999999999998,"5min":0.45100000000000001,"15min":0.37},"max":{"1min":1.756,"5min":1.9790000000000001,"15min":2.3679999999999999},"last":0.52600000000000002}]},{"osd":4,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3080000000000001,"15min":1.339},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5629999999999999,"5min":1.8140000000000001,"15min":2.1150000000000002},"last":1.02},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.383,"15min":1.391},"min":{"1min":0.77000000000000002,"5min":0.72399999999999998,"15min":0.71899999999999997},"max":{"1min":1.776,"5min":1.8819999999999999,"15min":2.6819999999999999},"last":0.81000000000000005}]},{"osd":5,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.4299999999999999,"15min":1.4339999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.9650000000000001,"5min":2.0190000000000001,"15min":2.1840000000000002},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.4450000000000001,"15min":1.482},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.9530000000000001,"5min":2.0219999999999998,"15min":2.294},"last":1.2869999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.399,"15min":1.4179999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.66900000000000004},"max":{"1min":1.772,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.502,"15min":1.524},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.77100000000000002},"max":{"1min":1.9139999999999999,"5min":2.0049999999999999,"15min":2.3300000000000001},"last":0.83899999999999997}]},{"osd":7,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.51,"15min":1.552},"min":{"1min":0.80800000000000005,"5min":0.80500000000000005,"15min":0.753},"max":{"1min":1.8959999999999999,"5min":2.1429999999999998,"15min":2.3359999999999999},"last":1.355},{"interface":"front","average":{"1min":1.47,"5min":1.488,"15min":1.5209999999999999},"min":{"1min":0.98999999999999999,"5min":0.80500000000000005,"15min":0.72799999999999998},"max":{"1min":2.165,"5min":2.2330000000000001,"15min":2.294},"last":1.1519999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575092,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2424832,"data_stored":1517698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.84599999999999997,"15min":0.89400000000000002},"min":{"1min":0.45200000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.3919999999999999,"5min":1.4610000000000001,"15min":2.1440000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.034,"5min":1.05,"15min":1.0860000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4730000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.372}]},{"osd":2,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0109999999999999,"15min":1.099},"min":{"1min":0.504,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.772,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.091,"15min":1.1679999999999999},"min":{"1min":0.496,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.649,"5min":1.782,"15min":1.9159999999999999},"last":0.48099999999999998}]},{"osd":3,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.03,"15min":1.099},"min":{"1min":0.629,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.4079999999999999,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.432},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.3},"min":{"1min":0.55300000000000005,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.6160000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.55000000000000004}]},{"osd":4,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.347,"15min":1.4139999999999999},"min":{"1min":0.72799999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.6140000000000001,"5min":1.804,"15min":2.012},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.353,"15min":1.3839999999999999},"min":{"1min":0.79700000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.698,"5min":4.5119999999999996,"15min":4.5119999999999996},"last":0.94199999999999995}]},{"osd":5,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.4119999999999999,"15min":1.419},"min":{"1min":0.99099999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9159999999999999,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.377,"5min":1.423,"15min":1.4770000000000001},"min":{"1min":0.879,"5min":0.621,"15min":0.621},"max":{"1min":2,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.0409999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.5069999999999999,"15min":1.5449999999999999},"min":{"1min":0.92900000000000005,"5min":0.73799999999999999,"15min":0.73699999999999999},"max":{"1min":2.0750000000000002,"5min":2.524,"15min":2.524},"last":0.871},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.464,"15min":1.5169999999999999},"min":{"1min":0.95399999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.833,"5min":2.274,"15min":2.4209999999999998},"last":1.0629999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.5469999999999999,"15min":1.5880000000000001},"min":{"1min":0.995,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.921,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.48,"15min":1.5349999999999999},"min":{"1min":1.079,"5min":0.80300000000000005,"15min":0.73799999999999999},"max":{"1min":1.8200000000000001,"5min":4.7359999999999998,"15min":4.7359999999999998},"last":0.89700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:54:31.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:54:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.187+0000 7f3a162d7640 1 Processor -- start 2023-11-28T20:54:32.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.187+0000 7f3a162d7640 1 -- start start 2023-11-28T20:54:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a162d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a1015d550 0x7f3a100cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a162d7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3a100d0100 0x7f3a100d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a162d7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3a100d1ab0 0x7f3a10070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a162d7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a100d0560 con 0x7f3a100d1ab0 2023-11-28T20:54:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a162d7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a100d2230 con 0x7f3a100d0100 2023-11-28T20:54:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a162d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100d2510 con 0x7f3a1015d550 2023-11-28T20:54:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a15ad6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3a100d1ab0 0x7f3a10070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a15ad6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3a100d1ab0 0x7f3a10070b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48330/0 (socket says 172.21.15.45:48330) 2023-11-28T20:54:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a15ad6640 1 -- 172.21.15.45:0/4165919129 learned_addr learned my addr 172.21.15.45:0/4165919129 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a152d5640 1 --2- 172.21.15.45:0/4165919129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a1015d550 0x7f3a100cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.189+0000 7f3a14ad4640 1 --2- 172.21.15.45:0/4165919129 >> v2:172.21.15.190:3300/0 conn(0x7f3a100d0100 0x7f3a100d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.190+0000 7f3a15ad6640 1 -- 172.21.15.45:0/4165919129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a1015d550 msgr2=0x7f3a100cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.190+0000 7f3a15ad6640 1 --2- 172.21.15.45:0/4165919129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a1015d550 0x7f3a100cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.190+0000 7f3a15ad6640 1 -- 172.21.15.45:0/4165919129 >> v2:172.21.15.190:3300/0 conn(0x7f3a100d0100 msgr2=0x7f3a100d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.190+0000 7f3a15ad6640 1 --2- 172.21.15.45:0/4165919129 >> v2:172.21.15.190:3300/0 conn(0x7f3a100d0100 0x7f3a100d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.190+0000 7f3a15ad6640 1 -- 172.21.15.45:0/4165919129 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a100d27f0 con 0x7f3a100d1ab0 2023-11-28T20:54:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.191+0000 7f3a15ad6640 1 --2- 172.21.15.45:0/4165919129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d1ab0 0x7f3a10070b20 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f3a100d0d50 tx=0x7f3a0802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c94b12159a0bdbf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.191+0000 7f3a07fff640 1 -- 172.21.15.45:0/4165919129 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0802ca70 con 0x7f3a100d1ab0 2023-11-28T20:54:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.191+0000 7f3a07fff640 1 -- 172.21.15.45:0/4165919129 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0802cbd0 con 0x7f3a100d1ab0 2023-11-28T20:54:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.191+0000 7f3a07fff640 1 -- 172.21.15.45:0/4165919129 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a080343f0 con 0x7f3a100d1ab0 2023-11-28T20:54:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.192+0000 7f3a162d7640 1 -- 172.21.15.45:0/4165919129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d1ab0 msgr2=0x7f3a10070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.192+0000 7f3a162d7640 1 --2- 172.21.15.45:0/4165919129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d1ab0 0x7f3a10070b20 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f3a100d0d50 tx=0x7f3a0802bed0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.192+0000 7f3a162d7640 1 -- 172.21.15.45:0/4165919129 shutdown_connections 2023-11-28T20:54:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.192+0000 7f3a162d7640 1 --2- 172.21.15.45:0/4165919129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d1ab0 0x7f3a10070b20 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.192+0000 7f3a162d7640 1 --2- 172.21.15.45:0/4165919129 >> v2:172.21.15.190:3300/0 conn(0x7f3a100d0100 0x7f3a100d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.192+0000 7f3a162d7640 1 --2- 172.21.15.45:0/4165919129 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a1015d550 0x7f3a100cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.192+0000 7f3a162d7640 1 -- 172.21.15.45:0/4165919129 >> 172.21.15.45:0/4165919129 conn(0x7f3a1006ac70 msgr2=0x7f3a100b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.192+0000 7f3a162d7640 1 -- 172.21.15.45:0/4165919129 shutdown_connections 2023-11-28T20:54:32.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.192+0000 7f3a162d7640 1 -- 172.21.15.45:0/4165919129 wait complete. 2023-11-28T20:54:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.194+0000 7f3a162d7640 1 Processor -- start 2023-11-28T20:54:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.194+0000 7f3a162d7640 1 -- start start 2023-11-28T20:54:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.195+0000 7f3a162d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d0100 0x7f3a100cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.195+0000 7f3a162d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a100d1ab0 0x7f3a10156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.195+0000 7f3a162d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1015d550 0x7f3a1015f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.195+0000 7f3a162d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100cc8b0 con 0x7f3a100d0100 2023-11-28T20:54:32.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.195+0000 7f3a162d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a10157e10 con 0x7f3a1015d550 2023-11-28T20:54:32.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.195+0000 7f3a162d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100cce30 con 0x7f3a100d1ab0 2023-11-28T20:54:32.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.195+0000 7f3a152d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d0100 0x7f3a100cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.195+0000 7f3a152d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d0100 0x7f3a100cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48334/0 (socket says 172.21.15.45:48334) 2023-11-28T20:54:32.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.195+0000 7f3a152d5640 1 -- 172.21.15.45:0/1420892466 learned_addr learned my addr 172.21.15.45:0/1420892466 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:32.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a152d5640 1 -- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a100d1ab0 msgr2=0x7f3a10156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:32.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a14ad4640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a100d1ab0 0x7f3a10156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a152d5640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a100d1ab0 0x7f3a10156ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a15ad6640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1015d550 0x7f3a1015f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a152d5640 1 -- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1015d550 msgr2=0x7f3a1015f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:32.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a152d5640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1015d550 0x7f3a1015f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a152d5640 1 -- 172.21.15.45:0/1420892466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a08009b30 con 0x7f3a100d0100 2023-11-28T20:54:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a152d5640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d0100 0x7f3a100cd2c0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00d620 tx=0x7f3a0c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a05ffb640 1 -- 172.21.15.45:0/1420892466 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00b620 con 0x7f3a100d0100 2023-11-28T20:54:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a05ffb640 1 -- 172.21.15.45:0/1420892466 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c00b780 con 0x7f3a100d0100 2023-11-28T20:54:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a05ffb640 1 -- 172.21.15.45:0/1420892466 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00bb30 con 0x7f3a100d0100 2023-11-28T20:54:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.196+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a10157850 con 0x7f3a100d0100 2023-11-28T20:54:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.197+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a100ccac0 con 0x7f3a100d0100 2023-11-28T20:54:32.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.198+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1004e920 con 0x7f3a100d0100 2023-11-28T20:54:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.199+0000 7f3a05ffb640 1 -- 172.21.15.45:0/1420892466 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3a0c00bc90 con 0x7f3a100d0100 2023-11-28T20:54:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.199+0000 7f3a05ffb640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f39f406af60 0x7f39f406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.200+0000 7f3a05ffb640 1 -- 172.21.15.45:0/1420892466 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3a0c089d30 con 0x7f3a100d0100 2023-11-28T20:54:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.200+0000 7f3a14ad4640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f39f406af60 0x7f39f406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.201+0000 7f3a14ad4640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f39f406af60 0x7f39f406d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a10067480 tx=0x7f3a00008040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.204+0000 7f3a05ffb640 1 -- 172.21.15.45:0/1420892466 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a0c054c00 con 0x7f3a100d0100 2023-11-28T20:54:32.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.398+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 --> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a100d1ef0 con 0x7f39f406af60 2023-11-28T20:54:32.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.420+0000 7f3a05ffb640 1 -- 172.21.15.45:0/1420892466 <== mgr.7015 v2:172.21.15.190:6832/2586319130 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287165 (secure 0 0 0) 0x7f3a0000e4c0 con 0x7f39f406af60 2023-11-28T20:54:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:32.428 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:54:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f39f406af60 msgr2=0x7f39f406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f39f406af60 0x7f39f406d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a10067480 tx=0x7f3a00008040 comp rx=0 tx=0).stop 2023-11-28T20:54:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d0100 msgr2=0x7f3a100cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d0100 0x7f3a100cd2c0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00d620 tx=0x7f3a0c00dae0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 shutdown_connections 2023-11-28T20:54:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f39f406af60 0x7f39f406d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1015d550 0x7f3a1015f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a100d1ab0 0x7f3a10156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 --2- 172.21.15.45:0/1420892466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a100d0100 0x7f3a100cd2c0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 >> 172.21.15.45:0/1420892466 conn(0x7f3a1006ac70 msgr2=0x7f3a100d2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:32.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.427+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 shutdown_connections 2023-11-28T20:54:32.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.428+0000 7f3a162d7640 1 -- 172.21.15.45:0/1420892466 wait complete. 2023-11-28T20:54:32.452 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-11-28T20:54:31.618700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1189,"num_read_kb":2391,"num_write":415,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4772,"ondisk_log_size":4772,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":352436,"kb_used_data":15544,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754622284,"statfs":{"total":773094113280,"available":772733218816,"internally_reserved":0,"allocated":15917056,"data_stored":8630612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":344837680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002289"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.912739+0000","last_change":"2023-11-28T20:54:26.908883+0000","last_active":"2023-11-28T20:54:27.912739+0000","last_peered":"2023-11-28T20:54:27.912739+0000","last_clean":"2023-11-28T20:54:27.912739+0000","last_became_active":"2023-11-28T20:54:26.908466+0000","last_became_peered":"2023-11-28T20:54:26.908466+0000","last_unstale":"2023-11-28T20:54:27.912739+0000","last_undegraded":"2023-11-28T20:54:27.912739+0000","last_fullsized":"2023-11-28T20:54:27.912739+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:09:58.510046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924297+0000","last_change":"2023-11-28T20:54:26.918099+0000","last_active":"2023-11-28T20:54:27.924297+0000","last_peered":"2023-11-28T20:54:27.924297+0000","last_clean":"2023-11-28T20:54:27.924297+0000","last_became_active":"2023-11-28T20:54:26.915982+0000","last_became_peered":"2023-11-28T20:54:26.915982+0000","last_unstale":"2023-11-28T20:54:27.924297+0000","last_undegraded":"2023-11-28T20:54:27.924297+0000","last_fullsized":"2023-11-28T20:54:27.924297+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:22:55.453144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924612+0000","last_change":"2023-11-28T20:54:26.916126+0000","last_active":"2023-11-28T20:54:27.924612+0000","last_peered":"2023-11-28T20:54:27.924612+0000","last_clean":"2023-11-28T20:54:27.924612+0000","last_became_active":"2023-11-28T20:54:26.915833+0000","last_became_peered":"2023-11-28T20:54:26.915833+0000","last_unstale":"2023-11-28T20:54:27.924612+0000","last_undegraded":"2023-11-28T20:54:27.924612+0000","last_fullsized":"2023-11-28T20:54:27.924612+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:50:23.060502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241904+0000","last_change":"2023-11-28T20:54:26.910344+0000","last_active":"2023-11-28T20:54:28.241904+0000","last_peered":"2023-11-28T20:54:28.241904+0000","last_clean":"2023-11-28T20:54:28.241904+0000","last_became_active":"2023-11-28T20:54:26.910052+0000","last_became_peered":"2023-11-28T20:54:26.910052+0000","last_unstale":"2023-11-28T20:54:28.241904+0000","last_undegraded":"2023-11-28T20:54:28.241904+0000","last_fullsized":"2023-11-28T20:54:28.241904+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:09.074880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911223+0000","last_change":"2023-11-28T20:54:26.913154+0000","last_active":"2023-11-28T20:54:27.911223+0000","last_peered":"2023-11-28T20:54:27.911223+0000","last_clean":"2023-11-28T20:54:27.911223+0000","last_became_active":"2023-11-28T20:54:26.912827+0000","last_became_peered":"2023-11-28T20:54:26.912827+0000","last_unstale":"2023-11-28T20:54:27.911223+0000","last_undegraded":"2023-11-28T20:54:27.911223+0000","last_fullsized":"2023-11-28T20:54:27.911223+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:39:54.271922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.912615+0000","last_change":"2023-11-28T20:54:26.909050+0000","last_active":"2023-11-28T20:54:26.912615+0000","last_peered":"2023-11-28T20:54:26.912615+0000","last_clean":"2023-11-28T20:54:26.912615+0000","last_became_active":"2023-11-28T20:54:26.908736+0000","last_became_peered":"2023-11-28T20:54:26.908736+0000","last_unstale":"2023-11-28T20:54:26.912615+0000","last_undegraded":"2023-11-28T20:54:26.912615+0000","last_fullsized":"2023-11-28T20:54:26.912615+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:52:15.491692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.909817+0000","last_change":"2023-11-28T20:54:26.900580+0000","last_active":"2023-11-28T20:54:26.909817+0000","last_peered":"2023-11-28T20:54:26.909817+0000","last_clean":"2023-11-28T20:54:26.909817+0000","last_became_active":"2023-11-28T20:54:26.900259+0000","last_became_peered":"2023-11-28T20:54:26.900259+0000","last_unstale":"2023-11-28T20:54:26.909817+0000","last_undegraded":"2023-11-28T20:54:26.909817+0000","last_fullsized":"2023-11-28T20:54:26.909817+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:14.162518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.923908+0000","last_change":"2023-11-28T20:54:26.916679+0000","last_active":"2023-11-28T20:54:27.923908+0000","last_peered":"2023-11-28T20:54:27.923908+0000","last_clean":"2023-11-28T20:54:27.923908+0000","last_became_active":"2023-11-28T20:54:26.915176+0000","last_became_peered":"2023-11-28T20:54:26.915176+0000","last_unstale":"2023-11-28T20:54:27.923908+0000","last_undegraded":"2023-11-28T20:54:27.923908+0000","last_fullsized":"2023-11-28T20:54:27.923908+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:01.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242495+0000","last_change":"2023-11-28T20:54:26.923096+0000","last_active":"2023-11-28T20:54:28.242495+0000","last_peered":"2023-11-28T20:54:28.242495+0000","last_clean":"2023-11-28T20:54:28.242495+0000","last_became_active":"2023-11-28T20:54:26.922765+0000","last_became_peered":"2023-11-28T20:54:26.922765+0000","last_unstale":"2023-11-28T20:54:28.242495+0000","last_undegraded":"2023-11-28T20:54:28.242495+0000","last_fullsized":"2023-11-28T20:54:28.242495+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:36:56.721066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907978+0000","last_change":"2023-11-28T20:54:26.906479+0000","last_active":"2023-11-28T20:54:27.907978+0000","last_peered":"2023-11-28T20:54:27.907978+0000","last_clean":"2023-11-28T20:54:27.907978+0000","last_became_active":"2023-11-28T20:54:26.906174+0000","last_became_peered":"2023-11-28T20:54:26.906174+0000","last_unstale":"2023-11-28T20:54:27.907978+0000","last_undegraded":"2023-11-28T20:54:27.907978+0000","last_fullsized":"2023-11-28T20:54:27.907978+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:23:21.169904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910961+0000","last_change":"2023-11-28T20:54:26.908426+0000","last_active":"2023-11-28T20:54:27.910961+0000","last_peered":"2023-11-28T20:54:27.910961+0000","last_clean":"2023-11-28T20:54:27.910961+0000","last_became_active":"2023-11-28T20:54:26.907420+0000","last_became_peered":"2023-11-28T20:54:26.907420+0000","last_unstale":"2023-11-28T20:54:27.910961+0000","last_undegraded":"2023-11-28T20:54:27.910961+0000","last_fullsized":"2023-11-28T20:54:27.910961+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:23:21.169904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.912726+0000","last_change":"2023-11-28T20:54:26.904208+0000","last_active":"2023-11-28T20:54:26.912726+0000","last_peered":"2023-11-28T20:54:26.912726+0000","last_clean":"2023-11-28T20:54:26.912726+0000","last_became_active":"2023-11-28T20:54:26.903277+0000","last_became_peered":"2023-11-28T20:54:26.903277+0000","last_unstale":"2023-11-28T20:54:26.912726+0000","last_undegraded":"2023-11-28T20:54:26.912726+0000","last_fullsized":"2023-11-28T20:54:26.912726+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:58:45.024456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907412+0000","last_change":"2023-11-28T20:54:26.899888+0000","last_active":"2023-11-28T20:54:27.907412+0000","last_peered":"2023-11-28T20:54:27.907412+0000","last_clean":"2023-11-28T20:54:27.907412+0000","last_became_active":"2023-11-28T20:54:26.899489+0000","last_became_peered":"2023-11-28T20:54:26.899489+0000","last_unstale":"2023-11-28T20:54:27.907412+0000","last_undegraded":"2023-11-28T20:54:27.907412+0000","last_fullsized":"2023-11-28T20:54:27.907412+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:20.727468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924031+0000","last_change":"2023-11-28T20:54:26.914048+0000","last_active":"2023-11-28T20:54:27.924031+0000","last_peered":"2023-11-28T20:54:27.924031+0000","last_clean":"2023-11-28T20:54:27.924031+0000","last_became_active":"2023-11-28T20:54:26.913780+0000","last_became_peered":"2023-11-28T20:54:26.913780+0000","last_unstale":"2023-11-28T20:54:27.924031+0000","last_undegraded":"2023-11-28T20:54:27.924031+0000","last_fullsized":"2023-11-28T20:54:27.924031+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:21.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910485+0000","last_change":"2023-11-28T20:54:26.907081+0000","last_active":"2023-11-28T20:54:27.910485+0000","last_peered":"2023-11-28T20:54:27.910485+0000","last_clean":"2023-11-28T20:54:27.910485+0000","last_became_active":"2023-11-28T20:54:26.906378+0000","last_became_peered":"2023-11-28T20:54:26.906378+0000","last_unstale":"2023-11-28T20:54:27.910485+0000","last_undegraded":"2023-11-28T20:54:27.910485+0000","last_fullsized":"2023-11-28T20:54:27.910485+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:22.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.901117+0000","last_change":"2023-11-28T20:54:26.892995+0000","last_active":"2023-11-28T20:54:26.901117+0000","last_peered":"2023-11-28T20:54:26.901117+0000","last_clean":"2023-11-28T20:54:26.901117+0000","last_became_active":"2023-11-28T20:54:26.892456+0000","last_became_peered":"2023-11-28T20:54:26.892456+0000","last_unstale":"2023-11-28T20:54:26.901117+0000","last_undegraded":"2023-11-28T20:54:26.901117+0000","last_fullsized":"2023-11-28T20:54:26.901117+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:12.946982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907862+0000","last_change":"2023-11-28T20:54:26.901170+0000","last_active":"2023-11-28T20:54:27.907862+0000","last_peered":"2023-11-28T20:54:27.907862+0000","last_clean":"2023-11-28T20:54:27.907862+0000","last_became_active":"2023-11-28T20:54:26.900805+0000","last_became_peered":"2023-11-28T20:54:26.900805+0000","last_unstale":"2023-11-28T20:54:27.907862+0000","last_undegraded":"2023-11-28T20:54:27.907862+0000","last_fullsized":"2023-11-28T20:54:27.907862+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:30:22.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924451+0000","last_change":"2023-11-28T20:54:26.906923+0000","last_active":"2023-11-28T20:54:27.924451+0000","last_peered":"2023-11-28T20:54:27.924451+0000","last_clean":"2023-11-28T20:54:27.924451+0000","last_became_active":"2023-11-28T20:54:26.906536+0000","last_became_peered":"2023-11-28T20:54:26.906536+0000","last_unstale":"2023-11-28T20:54:27.924451+0000","last_undegraded":"2023-11-28T20:54:27.924451+0000","last_fullsized":"2023-11-28T20:54:27.924451+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:52:37.714109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.911408+0000","last_change":"2023-11-28T20:54:26.902089+0000","last_active":"2023-11-28T20:54:26.911408+0000","last_peered":"2023-11-28T20:54:26.911408+0000","last_clean":"2023-11-28T20:54:26.911408+0000","last_became_active":"2023-11-28T20:54:26.901319+0000","last_became_peered":"2023-11-28T20:54:26.901319+0000","last_unstale":"2023-11-28T20:54:26.911408+0000","last_undegraded":"2023-11-28T20:54:26.911408+0000","last_fullsized":"2023-11-28T20:54:26.911408+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:52:38.278688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544173+0000","last_change":"2023-11-28T20:54:26.910420+0000","last_active":"2023-11-28T20:54:28.544173+0000","last_peered":"2023-11-28T20:54:28.544173+0000","last_clean":"2023-11-28T20:54:28.544173+0000","last_became_active":"2023-11-28T20:54:26.909924+0000","last_became_peered":"2023-11-28T20:54:26.909924+0000","last_unstale":"2023-11-28T20:54:28.544173+0000","last_undegraded":"2023-11-28T20:54:28.544173+0000","last_fullsized":"2023-11-28T20:54:28.544173+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:26:22.390075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241754+0000","last_change":"2023-11-28T20:54:26.912676+0000","last_active":"2023-11-28T20:54:28.241754+0000","last_peered":"2023-11-28T20:54:28.241754+0000","last_clean":"2023-11-28T20:54:28.241754+0000","last_became_active":"2023-11-28T20:54:26.912125+0000","last_became_peered":"2023-11-28T20:54:26.912125+0000","last_unstale":"2023-11-28T20:54:28.241754+0000","last_undegraded":"2023-11-28T20:54:28.241754+0000","last_fullsized":"2023-11-28T20:54:28.241754+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:07:34.024717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911702+0000","last_change":"2023-11-28T20:54:26.917717+0000","last_active":"2023-11-28T20:54:27.911702+0000","last_peered":"2023-11-28T20:54:27.911702+0000","last_clean":"2023-11-28T20:54:27.911702+0000","last_became_active":"2023-11-28T20:54:26.917309+0000","last_became_peered":"2023-11-28T20:54:26.917309+0000","last_unstale":"2023-11-28T20:54:27.911702+0000","last_undegraded":"2023-11-28T20:54:27.911702+0000","last_fullsized":"2023-11-28T20:54:27.911702+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:35.269079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241673+0000","last_change":"2023-11-28T20:54:26.912098+0000","last_active":"2023-11-28T20:54:28.241673+0000","last_peered":"2023-11-28T20:54:28.241673+0000","last_clean":"2023-11-28T20:54:28.241673+0000","last_became_active":"2023-11-28T20:54:26.911842+0000","last_became_peered":"2023-11-28T20:54:26.911842+0000","last_unstale":"2023-11-28T20:54:28.241673+0000","last_undegraded":"2023-11-28T20:54:28.241673+0000","last_fullsized":"2023-11-28T20:54:28.241673+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:59:54.055073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.909722+0000","last_change":"2023-11-28T20:54:26.893116+0000","last_active":"2023-11-28T20:54:27.909722+0000","last_peered":"2023-11-28T20:54:27.909722+0000","last_clean":"2023-11-28T20:54:27.909722+0000","last_became_active":"2023-11-28T20:54:26.889331+0000","last_became_peered":"2023-11-28T20:54:26.889331+0000","last_unstale":"2023-11-28T20:54:27.909722+0000","last_undegraded":"2023-11-28T20:54:27.909722+0000","last_fullsized":"2023-11-28T20:54:27.909722+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:30:18.390364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.912700+0000","last_change":"2023-11-28T20:54:26.910618+0000","last_active":"2023-11-28T20:54:27.912700+0000","last_peered":"2023-11-28T20:54:27.912700+0000","last_clean":"2023-11-28T20:54:27.912700+0000","last_became_active":"2023-11-28T20:54:26.909860+0000","last_became_peered":"2023-11-28T20:54:26.909860+0000","last_unstale":"2023-11-28T20:54:27.912700+0000","last_undegraded":"2023-11-28T20:54:27.912700+0000","last_fullsized":"2023-11-28T20:54:27.912700+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:59:54.055073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544202+0000","last_change":"2023-11-28T20:54:26.896978+0000","last_active":"2023-11-28T20:54:28.544202+0000","last_peered":"2023-11-28T20:54:28.544202+0000","last_clean":"2023-11-28T20:54:28.544202+0000","last_became_active":"2023-11-28T20:54:26.896602+0000","last_became_peered":"2023-11-28T20:54:26.896602+0000","last_unstale":"2023-11-28T20:54:28.544202+0000","last_undegraded":"2023-11-28T20:54:28.544202+0000","last_fullsized":"2023-11-28T20:54:28.544202+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:43:31.500958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.911429+0000","last_change":"2023-11-28T20:54:26.902152+0000","last_active":"2023-11-28T20:54:26.911429+0000","last_peered":"2023-11-28T20:54:26.911429+0000","last_clean":"2023-11-28T20:54:26.911429+0000","last_became_active":"2023-11-28T20:54:26.901608+0000","last_became_peered":"2023-11-28T20:54:26.901608+0000","last_unstale":"2023-11-28T20:54:26.911429+0000","last_undegraded":"2023-11-28T20:54:26.911429+0000","last_fullsized":"2023-11-28T20:54:26.911429+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:25:48.284752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.904310+0000","last_change":"2023-11-28T20:54:26.897032+0000","last_active":"2023-11-28T20:54:26.904310+0000","last_peered":"2023-11-28T20:54:26.904310+0000","last_clean":"2023-11-28T20:54:26.904310+0000","last_became_active":"2023-11-28T20:54:26.896710+0000","last_became_peered":"2023-11-28T20:54:26.896710+0000","last_unstale":"2023-11-28T20:54:26.904310+0000","last_undegraded":"2023-11-28T20:54:26.904310+0000","last_fullsized":"2023-11-28T20:54:26.904310+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:29.947416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544337+0000","last_change":"2023-11-28T20:54:26.911793+0000","last_active":"2023-11-28T20:54:28.544337+0000","last_peered":"2023-11-28T20:54:28.544337+0000","last_clean":"2023-11-28T20:54:28.544337+0000","last_became_active":"2023-11-28T20:54:26.911603+0000","last_became_peered":"2023-11-28T20:54:26.911603+0000","last_unstale":"2023-11-28T20:54:28.544337+0000","last_undegraded":"2023-11-28T20:54:28.544337+0000","last_fullsized":"2023-11-28T20:54:28.544337+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:45:18.632459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924136+0000","last_change":"2023-11-28T20:54:26.912272+0000","last_active":"2023-11-28T20:54:27.924136+0000","last_peered":"2023-11-28T20:54:27.924136+0000","last_clean":"2023-11-28T20:54:27.924136+0000","last_became_active":"2023-11-28T20:54:26.911890+0000","last_became_peered":"2023-11-28T20:54:26.911890+0000","last_unstale":"2023-11-28T20:54:27.924136+0000","last_undegraded":"2023-11-28T20:54:27.924136+0000","last_fullsized":"2023-11-28T20:54:27.924136+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:27:35.212161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910069+0000","last_change":"2023-11-28T20:54:26.897348+0000","last_active":"2023-11-28T20:54:27.910069+0000","last_peered":"2023-11-28T20:54:27.910069+0000","last_clean":"2023-11-28T20:54:27.910069+0000","last_became_active":"2023-11-28T20:54:26.894117+0000","last_became_peered":"2023-11-28T20:54:26.894117+0000","last_unstale":"2023-11-28T20:54:27.910069+0000","last_undegraded":"2023-11-28T20:54:27.910069+0000","last_fullsized":"2023-11-28T20:54:27.910069+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:30.700360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910045+0000","last_change":"2023-11-28T20:54:26.902218+0000","last_active":"2023-11-28T20:54:27.910045+0000","last_peered":"2023-11-28T20:54:27.910045+0000","last_clean":"2023-11-28T20:54:27.910045+0000","last_became_active":"2023-11-28T20:54:26.901949+0000","last_became_peered":"2023-11-28T20:54:26.901949+0000","last_unstale":"2023-11-28T20:54:27.910045+0000","last_undegraded":"2023-11-28T20:54:27.910045+0000","last_fullsized":"2023-11-28T20:54:27.910045+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:50.901505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907631+0000","last_change":"2023-11-28T20:54:26.907670+0000","last_active":"2023-11-28T20:54:27.907631+0000","last_peered":"2023-11-28T20:54:27.907631+0000","last_clean":"2023-11-28T20:54:27.907631+0000","last_became_active":"2023-11-28T20:54:26.907390+0000","last_became_peered":"2023-11-28T20:54:26.907390+0000","last_unstale":"2023-11-28T20:54:27.907631+0000","last_undegraded":"2023-11-28T20:54:27.907631+0000","last_fullsized":"2023-11-28T20:54:27.907631+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:49.671092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"147'4","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.926975+0000","last_change":"2023-11-28T20:54:25.914951+0000","last_active":"2023-11-28T20:54:26.926975+0000","last_peered":"2023-11-28T20:54:26.926975+0000","last_clean":"2023-11-28T20:54:26.926975+0000","last_became_active":"2023-11-28T20:54:25.914554+0000","last_became_peered":"2023-11-28T20:54:25.914554+0000","last_unstale":"2023-11-28T20:54:26.926975+0000","last_undegraded":"2023-11-28T20:54:26.926975+0000","last_fullsized":"2023-11-28T20:54:26.926975+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:06:03.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.912148+0000","last_change":"2023-11-28T20:54:26.906955+0000","last_active":"2023-11-28T20:54:26.912148+0000","last_peered":"2023-11-28T20:54:26.912148+0000","last_clean":"2023-11-28T20:54:26.912148+0000","last_became_active":"2023-11-28T20:54:26.906649+0000","last_became_peered":"2023-11-28T20:54:26.906649+0000","last_unstale":"2023-11-28T20:54:26.912148+0000","last_undegraded":"2023-11-28T20:54:26.912148+0000","last_fullsized":"2023-11-28T20:54:26.912148+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:14:12.737805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"147'1","reported_seq":13,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907838+0000","last_change":"2023-11-28T20:54:25.912000+0000","last_active":"2023-11-28T20:54:27.907838+0000","last_peered":"2023-11-28T20:54:27.907838+0000","last_clean":"2023-11-28T20:54:27.907838+0000","last_became_active":"2023-11-28T20:54:25.911419+0000","last_became_peered":"2023-11-28T20:54:25.911419+0000","last_unstale":"2023-11-28T20:54:27.907838+0000","last_undegraded":"2023-11-28T20:54:27.907838+0000","last_fullsized":"2023-11-28T20:54:27.907838+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:06:03.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910616+0000","last_change":"2023-11-28T20:54:26.907262+0000","last_active":"2023-11-28T20:54:27.910616+0000","last_peered":"2023-11-28T20:54:27.910616+0000","last_clean":"2023-11-28T20:54:27.910616+0000","last_became_active":"2023-11-28T20:54:26.905353+0000","last_became_peered":"2023-11-28T20:54:26.905353+0000","last_unstale":"2023-11-28T20:54:27.910616+0000","last_undegraded":"2023-11-28T20:54:27.910616+0000","last_fullsized":"2023-11-28T20:54:27.910616+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:11.319501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"147'2","reported_seq":14,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242041+0000","last_change":"2023-11-28T20:54:25.917110+0000","last_active":"2023-11-28T20:54:28.242041+0000","last_peered":"2023-11-28T20:54:28.242041+0000","last_clean":"2023-11-28T20:54:28.242041+0000","last_became_active":"2023-11-28T20:54:25.916814+0000","last_became_peered":"2023-11-28T20:54:25.916814+0000","last_unstale":"2023-11-28T20:54:28.242041+0000","last_undegraded":"2023-11-28T20:54:28.242041+0000","last_fullsized":"2023-11-28T20:54:28.242041+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:30:17.377825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242591+0000","last_change":"2023-11-28T20:54:26.916338+0000","last_active":"2023-11-28T20:54:28.242591+0000","last_peered":"2023-11-28T20:54:28.242591+0000","last_clean":"2023-11-28T20:54:28.242591+0000","last_became_active":"2023-11-28T20:54:26.916054+0000","last_became_peered":"2023-11-28T20:54:26.916054+0000","last_unstale":"2023-11-28T20:54:28.242591+0000","last_undegraded":"2023-11-28T20:54:28.242591+0000","last_fullsized":"2023-11-28T20:54:28.242591+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:06:31.250814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"147'2","reported_seq":14,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907746+0000","last_change":"2023-11-28T20:54:25.904749+0000","last_active":"2023-11-28T20:54:27.907746+0000","last_peered":"2023-11-28T20:54:27.907746+0000","last_clean":"2023-11-28T20:54:27.907746+0000","last_became_active":"2023-11-28T20:54:25.904472+0000","last_became_peered":"2023-11-28T20:54:25.904472+0000","last_unstale":"2023-11-28T20:54:27.907746+0000","last_undegraded":"2023-11-28T20:54:27.907746+0000","last_fullsized":"2023-11-28T20:54:27.907746+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:55:07.237982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907407+0000","last_change":"2023-11-28T20:54:26.898236+0000","last_active":"2023-11-28T20:54:27.907407+0000","last_peered":"2023-11-28T20:54:27.907407+0000","last_clean":"2023-11-28T20:54:27.907407+0000","last_became_active":"2023-11-28T20:54:26.897958+0000","last_became_peered":"2023-11-28T20:54:26.897958+0000","last_unstale":"2023-11-28T20:54:27.907407+0000","last_undegraded":"2023-11-28T20:54:27.907407+0000","last_fullsized":"2023-11-28T20:54:27.907407+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:30:15.906506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"147'1","reported_seq":13,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.912819+0000","last_change":"2023-11-28T20:54:25.900126+0000","last_active":"2023-11-28T20:54:27.912819+0000","last_peered":"2023-11-28T20:54:27.912819+0000","last_clean":"2023-11-28T20:54:27.912819+0000","last_became_active":"2023-11-28T20:54:25.899820+0000","last_became_peered":"2023-11-28T20:54:25.899820+0000","last_unstale":"2023-11-28T20:54:27.912819+0000","last_undegraded":"2023-11-28T20:54:27.912819+0000","last_fullsized":"2023-11-28T20:54:27.912819+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:36:01.280635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910628+0000","last_change":"2023-11-28T20:54:26.909047+0000","last_active":"2023-11-28T20:54:27.910628+0000","last_peered":"2023-11-28T20:54:27.910628+0000","last_clean":"2023-11-28T20:54:27.910628+0000","last_became_active":"2023-11-28T20:54:26.908742+0000","last_became_peered":"2023-11-28T20:54:26.908742+0000","last_unstale":"2023-11-28T20:54:27.910628+0000","last_undegraded":"2023-11-28T20:54:27.910628+0000","last_fullsized":"2023-11-28T20:54:27.910628+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:10:17.176634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"147'1","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.929797+0000","last_change":"2023-11-28T20:54:25.911939+0000","last_active":"2023-11-28T20:54:26.929797+0000","last_peered":"2023-11-28T20:54:26.929797+0000","last_clean":"2023-11-28T20:54:26.929797+0000","last_became_active":"2023-11-28T20:54:25.911702+0000","last_became_peered":"2023-11-28T20:54:25.911702+0000","last_unstale":"2023-11-28T20:54:26.929797+0000","last_undegraded":"2023-11-28T20:54:26.929797+0000","last_fullsized":"2023-11-28T20:54:26.929797+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:00:24.337281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924170+0000","last_change":"2023-11-28T20:54:26.904686+0000","last_active":"2023-11-28T20:54:27.924170+0000","last_peered":"2023-11-28T20:54:27.924170+0000","last_clean":"2023-11-28T20:54:27.924170+0000","last_became_active":"2023-11-28T20:54:26.904341+0000","last_became_peered":"2023-11-28T20:54:26.904341+0000","last_unstale":"2023-11-28T20:54:27.924170+0000","last_undegraded":"2023-11-28T20:54:27.924170+0000","last_fullsized":"2023-11-28T20:54:27.924170+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:54.343728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910300+0000","last_change":"2023-11-28T20:54:25.911743+0000","last_active":"2023-11-28T20:54:27.910300+0000","last_peered":"2023-11-28T20:54:27.910300+0000","last_clean":"2023-11-28T20:54:27.910300+0000","last_became_active":"2023-11-28T20:54:25.911469+0000","last_became_peered":"2023-11-28T20:54:25.911469+0000","last_unstale":"2023-11-28T20:54:27.910300+0000","last_undegraded":"2023-11-28T20:54:27.910300+0000","last_fullsized":"2023-11-28T20:54:27.910300+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:52.453947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907763+0000","last_change":"2023-11-28T20:54:26.911826+0000","last_active":"2023-11-28T20:54:27.907763+0000","last_peered":"2023-11-28T20:54:27.907763+0000","last_clean":"2023-11-28T20:54:27.907763+0000","last_became_active":"2023-11-28T20:54:26.911315+0000","last_became_peered":"2023-11-28T20:54:26.911315+0000","last_unstale":"2023-11-28T20:54:27.907763+0000","last_undegraded":"2023-11-28T20:54:27.907763+0000","last_fullsized":"2023-11-28T20:54:27.907763+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:42:14.025082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.881503+0000","last_change":"2023-11-28T20:54:25.890612+0000","last_active":"2023-11-28T20:54:26.881503+0000","last_peered":"2023-11-28T20:54:26.881503+0000","last_clean":"2023-11-28T20:54:26.881503+0000","last_became_active":"2023-11-28T20:54:25.890264+0000","last_became_peered":"2023-11-28T20:54:25.890264+0000","last_unstale":"2023-11-28T20:54:26.881503+0000","last_undegraded":"2023-11-28T20:54:26.881503+0000","last_fullsized":"2023-11-28T20:54:26.881503+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:34.326582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.908438+0000","last_change":"2023-11-28T20:54:26.899829+0000","last_active":"2023-11-28T20:54:26.908438+0000","last_peered":"2023-11-28T20:54:26.908438+0000","last_clean":"2023-11-28T20:54:26.908438+0000","last_became_active":"2023-11-28T20:54:26.899431+0000","last_became_peered":"2023-11-28T20:54:26.899431+0000","last_unstale":"2023-11-28T20:54:26.908438+0000","last_undegraded":"2023-11-28T20:54:26.908438+0000","last_fullsized":"2023-11-28T20:54:26.908438+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:05:15.663983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.876400+0000","last_change":"2023-11-28T20:54:25.917006+0000","last_active":"2023-11-28T20:54:26.876400+0000","last_peered":"2023-11-28T20:54:26.876400+0000","last_clean":"2023-11-28T20:54:26.876400+0000","last_became_active":"2023-11-28T20:54:25.916076+0000","last_became_peered":"2023-11-28T20:54:25.916076+0000","last_unstale":"2023-11-28T20:54:26.876400+0000","last_undegraded":"2023-11-28T20:54:26.876400+0000","last_fullsized":"2023-11-28T20:54:26.876400+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:49:17.549764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911531+0000","last_change":"2023-11-28T20:54:26.912681+0000","last_active":"2023-11-28T20:54:27.911531+0000","last_peered":"2023-11-28T20:54:27.911531+0000","last_clean":"2023-11-28T20:54:27.911531+0000","last_became_active":"2023-11-28T20:54:26.911684+0000","last_became_peered":"2023-11-28T20:54:26.911684+0000","last_unstale":"2023-11-28T20:54:27.911531+0000","last_undegraded":"2023-11-28T20:54:27.911531+0000","last_fullsized":"2023-11-28T20:54:27.911531+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:57:46.037079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910858+0000","last_change":"2023-11-28T20:54:25.910213+0000","last_active":"2023-11-28T20:54:27.910858+0000","last_peered":"2023-11-28T20:54:27.910858+0000","last_clean":"2023-11-28T20:54:27.910858+0000","last_became_active":"2023-11-28T20:54:25.909938+0000","last_became_peered":"2023-11-28T20:54:25.909938+0000","last_unstale":"2023-11-28T20:54:27.910858+0000","last_undegraded":"2023-11-28T20:54:27.910858+0000","last_fullsized":"2023-11-28T20:54:27.910858+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:00:48.327923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924591+0000","last_change":"2023-11-28T20:54:26.921276+0000","last_active":"2023-11-28T20:54:27.924591+0000","last_peered":"2023-11-28T20:54:27.924591+0000","last_clean":"2023-11-28T20:54:27.924591+0000","last_became_active":"2023-11-28T20:54:26.920674+0000","last_became_peered":"2023-11-28T20:54:26.920674+0000","last_unstale":"2023-11-28T20:54:27.924591+0000","last_undegraded":"2023-11-28T20:54:27.924591+0000","last_fullsized":"2023-11-28T20:54:27.924591+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:01:36.857880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"147'3","reported_seq":15,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911778+0000","last_change":"2023-11-28T20:54:25.914806+0000","last_active":"2023-11-28T20:54:27.911778+0000","last_peered":"2023-11-28T20:54:27.911778+0000","last_clean":"2023-11-28T20:54:27.911778+0000","last_became_active":"2023-11-28T20:54:25.914512+0000","last_became_peered":"2023-11-28T20:54:25.914512+0000","last_unstale":"2023-11-28T20:54:27.911778+0000","last_undegraded":"2023-11-28T20:54:27.911778+0000","last_fullsized":"2023-11-28T20:54:27.911778+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:09:34.932833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.913017+0000","last_change":"2023-11-28T20:54:26.891699+0000","last_active":"2023-11-28T20:54:27.913017+0000","last_peered":"2023-11-28T20:54:27.913017+0000","last_clean":"2023-11-28T20:54:27.913017+0000","last_became_active":"2023-11-28T20:54:26.891088+0000","last_became_peered":"2023-11-28T20:54:26.891088+0000","last_unstale":"2023-11-28T20:54:27.913017+0000","last_undegraded":"2023-11-28T20:54:27.913017+0000","last_fullsized":"2023-11-28T20:54:27.913017+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:07:34.024717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"147'6","reported_seq":18,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924500+0000","last_change":"2023-11-28T20:54:25.912593+0000","last_active":"2023-11-28T20:54:27.924500+0000","last_peered":"2023-11-28T20:54:27.924500+0000","last_clean":"2023-11-28T20:54:27.924500+0000","last_became_active":"2023-11-28T20:54:25.912298+0000","last_became_peered":"2023-11-28T20:54:25.912298+0000","last_unstale":"2023-11-28T20:54:27.924500+0000","last_undegraded":"2023-11-28T20:54:27.924500+0000","last_fullsized":"2023-11-28T20:54:27.924500+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:19:00.813250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.912592+0000","last_change":"2023-11-28T20:54:26.910666+0000","last_active":"2023-11-28T20:54:26.912592+0000","last_peered":"2023-11-28T20:54:26.912592+0000","last_clean":"2023-11-28T20:54:26.912592+0000","last_became_active":"2023-11-28T20:54:26.910160+0000","last_became_peered":"2023-11-28T20:54:26.910160+0000","last_unstale":"2023-11-28T20:54:26.912592+0000","last_undegraded":"2023-11-28T20:54:26.912592+0000","last_fullsized":"2023-11-28T20:54:26.912592+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:52:29.235644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"147'2","reported_seq":14,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924551+0000","last_change":"2023-11-28T20:54:25.914232+0000","last_active":"2023-11-28T20:54:27.924551+0000","last_peered":"2023-11-28T20:54:27.924551+0000","last_clean":"2023-11-28T20:54:27.924551+0000","last_became_active":"2023-11-28T20:54:25.913929+0000","last_became_peered":"2023-11-28T20:54:25.913929+0000","last_unstale":"2023-11-28T20:54:27.924551+0000","last_undegraded":"2023-11-28T20:54:27.924551+0000","last_fullsized":"2023-11-28T20:54:27.924551+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:24.637432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907357+0000","last_change":"2023-11-28T20:54:26.899880+0000","last_active":"2023-11-28T20:54:27.907357+0000","last_peered":"2023-11-28T20:54:27.907357+0000","last_clean":"2023-11-28T20:54:27.907357+0000","last_became_active":"2023-11-28T20:54:26.899588+0000","last_became_peered":"2023-11-28T20:54:26.899588+0000","last_unstale":"2023-11-28T20:54:27.907357+0000","last_undegraded":"2023-11-28T20:54:27.907357+0000","last_fullsized":"2023-11-28T20:54:27.907357+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:53:32.854588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924347+0000","last_change":"2023-11-28T20:54:25.910564+0000","last_active":"2023-11-28T20:54:27.924347+0000","last_peered":"2023-11-28T20:54:27.924347+0000","last_clean":"2023-11-28T20:54:27.924347+0000","last_became_active":"2023-11-28T20:54:25.910102+0000","last_became_peered":"2023-11-28T20:54:25.910102+0000","last_unstale":"2023-11-28T20:54:27.924347+0000","last_undegraded":"2023-11-28T20:54:27.924347+0000","last_fullsized":"2023-11-28T20:54:27.924347+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:28:14.857308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.543720+0000","last_change":"2023-11-28T20:54:26.909144+0000","last_active":"2023-11-28T20:54:28.543720+0000","last_peered":"2023-11-28T20:54:28.543720+0000","last_clean":"2023-11-28T20:54:28.543720+0000","last_became_active":"2023-11-28T20:54:26.908825+0000","last_became_peered":"2023-11-28T20:54:26.908825+0000","last_unstale":"2023-11-28T20:54:28.543720+0000","last_undegraded":"2023-11-28T20:54:28.543720+0000","last_fullsized":"2023-11-28T20:54:28.543720+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:38:29.715808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924212+0000","last_change":"2023-11-28T20:54:25.914232+0000","last_active":"2023-11-28T20:54:27.924212+0000","last_peered":"2023-11-28T20:54:27.924212+0000","last_clean":"2023-11-28T20:54:27.924212+0000","last_became_active":"2023-11-28T20:54:25.913951+0000","last_became_peered":"2023-11-28T20:54:25.913951+0000","last_unstale":"2023-11-28T20:54:27.924212+0000","last_undegraded":"2023-11-28T20:54:27.924212+0000","last_fullsized":"2023-11-28T20:54:27.924212+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:36:55.708527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.912495+0000","last_change":"2023-11-28T20:54:26.906560+0000","last_active":"2023-11-28T20:54:27.912495+0000","last_peered":"2023-11-28T20:54:27.912495+0000","last_clean":"2023-11-28T20:54:27.912495+0000","last_became_active":"2023-11-28T20:54:26.906242+0000","last_became_peered":"2023-11-28T20:54:26.906242+0000","last_unstale":"2023-11-28T20:54:27.912495+0000","last_undegraded":"2023-11-28T20:54:27.912495+0000","last_fullsized":"2023-11-28T20:54:27.912495+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:38:05.965491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"147'1","reported_seq":13,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907873+0000","last_change":"2023-11-28T20:54:25.907756+0000","last_active":"2023-11-28T20:54:27.907873+0000","last_peered":"2023-11-28T20:54:27.907873+0000","last_clean":"2023-11-28T20:54:27.907873+0000","last_became_active":"2023-11-28T20:54:25.907404+0000","last_became_peered":"2023-11-28T20:54:25.907404+0000","last_unstale":"2023-11-28T20:54:27.907873+0000","last_undegraded":"2023-11-28T20:54:27.907873+0000","last_fullsized":"2023-11-28T20:54:27.907873+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:47:25.267556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911326+0000","last_change":"2023-11-28T20:54:26.912380+0000","last_active":"2023-11-28T20:54:27.911326+0000","last_peered":"2023-11-28T20:54:27.911326+0000","last_clean":"2023-11-28T20:54:27.911326+0000","last_became_active":"2023-11-28T20:54:26.909682+0000","last_became_peered":"2023-11-28T20:54:26.909682+0000","last_unstale":"2023-11-28T20:54:27.911326+0000","last_undegraded":"2023-11-28T20:54:27.911326+0000","last_fullsized":"2023-11-28T20:54:27.911326+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:10:27.576002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"147'2","reported_seq":14,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544223+0000","last_change":"2023-11-28T20:54:25.908240+0000","last_active":"2023-11-28T20:54:28.544223+0000","last_peered":"2023-11-28T20:54:28.544223+0000","last_clean":"2023-11-28T20:54:28.544223+0000","last_became_active":"2023-11-28T20:54:25.907837+0000","last_became_peered":"2023-11-28T20:54:25.907837+0000","last_unstale":"2023-11-28T20:54:28.544223+0000","last_undegraded":"2023-11-28T20:54:28.544223+0000","last_fullsized":"2023-11-28T20:54:28.544223+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:52:24.315057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.905174+0000","last_change":"2023-11-28T20:54:26.898938+0000","last_active":"2023-11-28T20:54:26.905174+0000","last_peered":"2023-11-28T20:54:26.905174+0000","last_clean":"2023-11-28T20:54:26.905174+0000","last_became_active":"2023-11-28T20:54:26.898421+0000","last_became_peered":"2023-11-28T20:54:26.898421+0000","last_unstale":"2023-11-28T20:54:26.905174+0000","last_undegraded":"2023-11-28T20:54:26.905174+0000","last_fullsized":"2023-11-28T20:54:26.905174+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:27:10.365990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"147'1","reported_seq":13,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242177+0000","last_change":"2023-11-28T20:54:25.911705+0000","last_active":"2023-11-28T20:54:28.242177+0000","last_peered":"2023-11-28T20:54:28.242177+0000","last_clean":"2023-11-28T20:54:28.242177+0000","last_became_active":"2023-11-28T20:54:25.911347+0000","last_became_peered":"2023-11-28T20:54:25.911347+0000","last_unstale":"2023-11-28T20:54:28.242177+0000","last_undegraded":"2023-11-28T20:54:28.242177+0000","last_fullsized":"2023-11-28T20:54:28.242177+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:49.888966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.913196+0000","last_change":"2023-11-28T20:54:26.897634+0000","last_active":"2023-11-28T20:54:27.913196+0000","last_peered":"2023-11-28T20:54:27.913196+0000","last_clean":"2023-11-28T20:54:27.913196+0000","last_became_active":"2023-11-28T20:54:26.897181+0000","last_became_peered":"2023-11-28T20:54:26.897181+0000","last_unstale":"2023-11-28T20:54:27.913196+0000","last_undegraded":"2023-11-28T20:54:27.913196+0000","last_fullsized":"2023-11-28T20:54:27.913196+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:09.074880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242241+0000","last_change":"2023-11-28T20:54:25.911644+0000","last_active":"2023-11-28T20:54:28.242241+0000","last_peered":"2023-11-28T20:54:28.242241+0000","last_clean":"2023-11-28T20:54:28.242241+0000","last_became_active":"2023-11-28T20:54:25.911304+0000","last_became_peered":"2023-11-28T20:54:25.911304+0000","last_unstale":"2023-11-28T20:54:28.242241+0000","last_undegraded":"2023-11-28T20:54:28.242241+0000","last_fullsized":"2023-11-28T20:54:28.242241+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:00:48.327923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.905183+0000","last_change":"2023-11-28T20:54:26.898906+0000","last_active":"2023-11-28T20:54:26.905183+0000","last_peered":"2023-11-28T20:54:26.905183+0000","last_clean":"2023-11-28T20:54:26.905183+0000","last_became_active":"2023-11-28T20:54:26.897627+0000","last_became_peered":"2023-11-28T20:54:26.897627+0000","last_unstale":"2023-11-28T20:54:26.905183+0000","last_undegraded":"2023-11-28T20:54:26.905183+0000","last_fullsized":"2023-11-28T20:54:26.905183+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:39:46.731684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"147'2","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.927775+0000","last_change":"2023-11-28T20:54:25.900134+0000","last_active":"2023-11-28T20:54:26.927775+0000","last_peered":"2023-11-28T20:54:26.927775+0000","last_clean":"2023-11-28T20:54:26.927775+0000","last_became_active":"2023-11-28T20:54:25.899909+0000","last_became_peered":"2023-11-28T20:54:25.899909+0000","last_unstale":"2023-11-28T20:54:26.927775+0000","last_undegraded":"2023-11-28T20:54:26.927775+0000","last_fullsized":"2023-11-28T20:54:26.927775+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:55.877897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924046+0000","last_change":"2023-11-28T20:54:26.908487+0000","last_active":"2023-11-28T20:54:27.924046+0000","last_peered":"2023-11-28T20:54:27.924046+0000","last_clean":"2023-11-28T20:54:27.924046+0000","last_became_active":"2023-11-28T20:54:26.908196+0000","last_became_peered":"2023-11-28T20:54:26.908196+0000","last_unstale":"2023-11-28T20:54:27.924046+0000","last_undegraded":"2023-11-28T20:54:27.924046+0000","last_fullsized":"2023-11-28T20:54:27.924046+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:45:12.736796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241687+0000","last_change":"2023-11-28T20:54:25.908639+0000","last_active":"2023-11-28T20:54:28.241687+0000","last_peered":"2023-11-28T20:54:28.241687+0000","last_clean":"2023-11-28T20:54:28.241687+0000","last_became_active":"2023-11-28T20:54:25.908297+0000","last_became_peered":"2023-11-28T20:54:25.908297+0000","last_unstale":"2023-11-28T20:54:28.241687+0000","last_undegraded":"2023-11-28T20:54:28.241687+0000","last_fullsized":"2023-11-28T20:54:28.241687+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:52.453947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924292+0000","last_change":"2023-11-28T20:54:26.905792+0000","last_active":"2023-11-28T20:54:27.924292+0000","last_peered":"2023-11-28T20:54:27.924292+0000","last_clean":"2023-11-28T20:54:27.924292+0000","last_became_active":"2023-11-28T20:54:26.905380+0000","last_became_peered":"2023-11-28T20:54:26.905380+0000","last_unstale":"2023-11-28T20:54:27.924292+0000","last_undegraded":"2023-11-28T20:54:27.924292+0000","last_fullsized":"2023-11-28T20:54:27.924292+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:46:26.771636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924385+0000","last_change":"2023-11-28T20:54:25.903983+0000","last_active":"2023-11-28T20:54:27.924385+0000","last_peered":"2023-11-28T20:54:27.924385+0000","last_clean":"2023-11-28T20:54:27.924385+0000","last_became_active":"2023-11-28T20:54:25.903538+0000","last_became_peered":"2023-11-28T20:54:25.903538+0000","last_unstale":"2023-11-28T20:54:27.924385+0000","last_undegraded":"2023-11-28T20:54:27.924385+0000","last_fullsized":"2023-11-28T20:54:27.924385+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:06:30.238274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.910712+0000","last_change":"2023-11-28T20:54:26.891890+0000","last_active":"2023-11-28T20:54:27.910712+0000","last_peered":"2023-11-28T20:54:27.910712+0000","last_clean":"2023-11-28T20:54:27.910712+0000","last_became_active":"2023-11-28T20:54:26.889899+0000","last_became_peered":"2023-11-28T20:54:26.889899+0000","last_unstale":"2023-11-28T20:54:27.910712+0000","last_undegraded":"2023-11-28T20:54:27.910712+0000","last_fullsized":"2023-11-28T20:54:27.910712+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:47:47.773324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.240922+0000","last_change":"2023-11-28T20:54:25.912307+0000","last_active":"2023-11-28T20:54:28.240922+0000","last_peered":"2023-11-28T20:54:28.240922+0000","last_clean":"2023-11-28T20:54:28.240922+0000","last_became_active":"2023-11-28T20:54:25.912027+0000","last_became_peered":"2023-11-28T20:54:25.912027+0000","last_unstale":"2023-11-28T20:54:28.240922+0000","last_undegraded":"2023-11-28T20:54:28.240922+0000","last_fullsized":"2023-11-28T20:54:28.240922+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:29:15.458403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.900897+0000","last_change":"2023-11-28T20:54:26.895510+0000","last_active":"2023-11-28T20:54:26.900897+0000","last_peered":"2023-11-28T20:54:26.900897+0000","last_clean":"2023-11-28T20:54:26.900897+0000","last_became_active":"2023-11-28T20:54:26.895153+0000","last_became_peered":"2023-11-28T20:54:26.895153+0000","last_unstale":"2023-11-28T20:54:26.900897+0000","last_undegraded":"2023-11-28T20:54:26.900897+0000","last_fullsized":"2023-11-28T20:54:26.900897+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:05.996950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"147'2","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.927811+0000","last_change":"2023-11-28T20:54:25.902328+0000","last_active":"2023-11-28T20:54:26.927811+0000","last_peered":"2023-11-28T20:54:26.927811+0000","last_clean":"2023-11-28T20:54:26.927811+0000","last_became_active":"2023-11-28T20:54:25.902040+0000","last_became_peered":"2023-11-28T20:54:25.902040+0000","last_unstale":"2023-11-28T20:54:26.927811+0000","last_undegraded":"2023-11-28T20:54:26.927811+0000","last_fullsized":"2023-11-28T20:54:26.927811+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:04.854597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907890+0000","last_change":"2023-11-28T20:54:26.914128+0000","last_active":"2023-11-28T20:54:27.907890+0000","last_peered":"2023-11-28T20:54:27.907890+0000","last_clean":"2023-11-28T20:54:27.907890+0000","last_became_active":"2023-11-28T20:54:26.913858+0000","last_became_peered":"2023-11-28T20:54:26.913858+0000","last_unstale":"2023-11-28T20:54:27.907890+0000","last_undegraded":"2023-11-28T20:54:27.907890+0000","last_fullsized":"2023-11-28T20:54:27.907890+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:10:17.176634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.876323+0000","last_change":"2023-11-28T20:54:25.916939+0000","last_active":"2023-11-28T20:54:26.876323+0000","last_peered":"2023-11-28T20:54:26.876323+0000","last_clean":"2023-11-28T20:54:26.876323+0000","last_became_active":"2023-11-28T20:54:25.916431+0000","last_became_peered":"2023-11-28T20:54:25.916431+0000","last_unstale":"2023-11-28T20:54:26.876323+0000","last_undegraded":"2023-11-28T20:54:26.876323+0000","last_fullsized":"2023-11-28T20:54:26.876323+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:37:57.338637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242624+0000","last_change":"2023-11-28T20:54:26.913651+0000","last_active":"2023-11-28T20:54:28.242624+0000","last_peered":"2023-11-28T20:54:28.242624+0000","last_clean":"2023-11-28T20:54:28.242624+0000","last_became_active":"2023-11-28T20:54:26.913381+0000","last_became_peered":"2023-11-28T20:54:26.913381+0000","last_unstale":"2023-11-28T20:54:28.242624+0000","last_undegraded":"2023-11-28T20:54:28.242624+0000","last_fullsized":"2023-11-28T20:54:28.242624+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:21:25.649971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.876110+0000","last_change":"2023-11-28T20:54:25.908920+0000","last_active":"2023-11-28T20:54:26.876110+0000","last_peered":"2023-11-28T20:54:26.876110+0000","last_clean":"2023-11-28T20:54:26.876110+0000","last_became_active":"2023-11-28T20:54:25.908452+0000","last_became_peered":"2023-11-28T20:54:25.908452+0000","last_unstale":"2023-11-28T20:54:26.876110+0000","last_undegraded":"2023-11-28T20:54:26.876110+0000","last_fullsized":"2023-11-28T20:54:26.876110+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:07:29.380457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.544565+0000","last_change":"2023-11-28T20:54:26.913188+0000","last_active":"2023-11-28T20:54:28.544565+0000","last_peered":"2023-11-28T20:54:28.544565+0000","last_clean":"2023-11-28T20:54:28.544565+0000","last_became_active":"2023-11-28T20:54:26.907468+0000","last_became_peered":"2023-11-28T20:54:26.907468+0000","last_unstale":"2023-11-28T20:54:28.544565+0000","last_undegraded":"2023-11-28T20:54:28.544565+0000","last_fullsized":"2023-11-28T20:54:28.544565+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:38.186160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.894728+0000","last_change":"2023-11-28T20:54:25.900497+0000","last_active":"2023-11-28T20:54:26.894728+0000","last_peered":"2023-11-28T20:54:26.894728+0000","last_clean":"2023-11-28T20:54:26.894728+0000","last_became_active":"2023-11-28T20:54:25.900024+0000","last_became_peered":"2023-11-28T20:54:25.900024+0000","last_unstale":"2023-11-28T20:54:26.894728+0000","last_undegraded":"2023-11-28T20:54:26.894728+0000","last_fullsized":"2023-11-28T20:54:26.894728+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:51:11.214969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.911173+0000","last_change":"2023-11-28T20:54:26.901224+0000","last_active":"2023-11-28T20:54:26.911173+0000","last_peered":"2023-11-28T20:54:26.911173+0000","last_clean":"2023-11-28T20:54:26.911173+0000","last_became_active":"2023-11-28T20:54:26.900674+0000","last_became_peered":"2023-11-28T20:54:26.900674+0000","last_unstale":"2023-11-28T20:54:26.911173+0000","last_undegraded":"2023-11-28T20:54:26.911173+0000","last_fullsized":"2023-11-28T20:54:26.911173+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:46:47.295747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911402+0000","last_change":"2023-11-28T20:54:25.910201+0000","last_active":"2023-11-28T20:54:27.911402+0000","last_peered":"2023-11-28T20:54:27.911402+0000","last_clean":"2023-11-28T20:54:27.911402+0000","last_became_active":"2023-11-28T20:54:25.909522+0000","last_became_peered":"2023-11-28T20:54:25.909522+0000","last_unstale":"2023-11-28T20:54:27.911402+0000","last_undegraded":"2023-11-28T20:54:27.911402+0000","last_fullsized":"2023-11-28T20:54:27.911402+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:42:13.012542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.924207+0000","last_change":"2023-11-28T20:54:26.920277+0000","last_active":"2023-11-28T20:54:27.924207+0000","last_peered":"2023-11-28T20:54:27.924207+0000","last_clean":"2023-11-28T20:54:27.924207+0000","last_became_active":"2023-11-28T20:54:26.916252+0000","last_became_peered":"2023-11-28T20:54:26.916252+0000","last_unstale":"2023-11-28T20:54:27.924207+0000","last_undegraded":"2023-11-28T20:54:27.924207+0000","last_fullsized":"2023-11-28T20:54:27.924207+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:55:11.972088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.911882+0000","last_change":"2023-11-28T20:54:25.905984+0000","last_active":"2023-11-28T20:54:27.911882+0000","last_peered":"2023-11-28T20:54:27.911882+0000","last_clean":"2023-11-28T20:54:27.911882+0000","last_became_active":"2023-11-28T20:54:25.904885+0000","last_became_peered":"2023-11-28T20:54:25.904885+0000","last_unstale":"2023-11-28T20:54:27.911882+0000","last_undegraded":"2023-11-28T20:54:27.911882+0000","last_fullsized":"2023-11-28T20:54:27.911882+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:19.714929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.241475+0000","last_change":"2023-11-28T20:54:26.914220+0000","last_active":"2023-11-28T20:54:28.241475+0000","last_peered":"2023-11-28T20:54:28.241475+0000","last_clean":"2023-11-28T20:54:28.241475+0000","last_became_active":"2023-11-28T20:54:26.913712+0000","last_became_peered":"2023-11-28T20:54:26.913712+0000","last_unstale":"2023-11-28T20:54:28.241475+0000","last_undegraded":"2023-11-28T20:54:28.241475+0000","last_fullsized":"2023-11-28T20:54:28.241475+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:38:05.965491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.883204+0000","last_change":"2023-11-28T20:54:25.890626+0000","last_active":"2023-11-28T20:54:26.883204+0000","last_peered":"2023-11-28T20:54:26.883204+0000","last_clean":"2023-11-28T20:54:26.883204+0000","last_became_active":"2023-11-28T20:54:25.889965+0000","last_became_peered":"2023-11-28T20:54:25.889965+0000","last_unstale":"2023-11-28T20:54:26.883204+0000","last_undegraded":"2023-11-28T20:54:26.883204+0000","last_fullsized":"2023-11-28T20:54:26.883204+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:19.978558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.907639+0000","last_change":"2023-11-28T20:54:26.897362+0000","last_active":"2023-11-28T20:54:27.907639+0000","last_peered":"2023-11-28T20:54:27.907639+0000","last_clean":"2023-11-28T20:54:27.907639+0000","last_became_active":"2023-11-28T20:54:26.897057+0000","last_became_peered":"2023-11-28T20:54:26.897057+0000","last_unstale":"2023-11-28T20:54:27.907639+0000","last_undegraded":"2023-11-28T20:54:27.907639+0000","last_fullsized":"2023-11-28T20:54:27.907639+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:36:02.293175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.240863+0000","last_change":"2023-11-28T20:54:25.914230+0000","last_active":"2023-11-28T20:54:28.240863+0000","last_peered":"2023-11-28T20:54:28.240863+0000","last_clean":"2023-11-28T20:54:28.240863+0000","last_became_active":"2023-11-28T20:54:25.913684+0000","last_became_peered":"2023-11-28T20:54:25.913684+0000","last_unstale":"2023-11-28T20:54:28.240863+0000","last_undegraded":"2023-11-28T20:54:28.240863+0000","last_fullsized":"2023-11-28T20:54:28.240863+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:13:16.406084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":11,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:28.242090+0000","last_change":"2023-11-28T20:54:26.912385+0000","last_active":"2023-11-28T20:54:28.242090+0000","last_peered":"2023-11-28T20:54:28.242090+0000","last_clean":"2023-11-28T20:54:28.242090+0000","last_became_active":"2023-11-28T20:54:26.911789+0000","last_became_peered":"2023-11-28T20:54:26.911789+0000","last_unstale":"2023-11-28T20:54:28.242090+0000","last_undegraded":"2023-11-28T20:54:28.242090+0000","last_fullsized":"2023-11-28T20:54:28.242090+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:25.859128+0000","last_clean_scrub_stamp":"2023-11-28T20:54:25.859128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:23.314439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"147'8","reported_seq":20,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-11-28T20:54:27.923917+0000","last_change":"2023-11-28T20:54:25.914874+0000","last_active":"2023-11-28T20:54:27.923917+0000","last_peered":"2023-11-28T20:54:27.923917+0000","last_clean":"2023-11-28T20:54:27.923917+0000","last_became_active":"2023-11-28T20:54:25.914583+0000","last_became_peered":"2023-11-28T20:54:25.914583+0000","last_unstale":"2023-11-28T20:54:27.923917+0000","last_undegraded":"2023-11-28T20:54:27.923917+0000","last_fullsized":"2023-11-28T20:54:27.923917+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:54:24.846589+0000","last_clean_scrub_stamp":"2023-11-28T20:54:24.846589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:29:08.062340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"136'4734","reported_seq":5250,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-11-28T20:54:26.895014+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:54:26.895014+0000","last_peered":"2023-11-28T20:54:26.895014+0000","last_clean":"2023-11-28T20:54:26.895014+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:54:26.895014+0000","last_undegraded":"2023-11-28T20:54:26.895014+0000","last_fullsized":"2023-11-28T20:54:26.895014+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1189,"num_read_kb":2391,"num_write":371,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1189,"num_read_kb":2391,"num_write":371,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4734,"ondisk_log_size":4734,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575091,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47456,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324384,"statfs":{"total":96636764160,"available":96588169216,"internally_reserved":0,"allocated":2437120,"data_stored":1518452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.167,"15min":1.1870000000000001},"min":{"1min":0.623,"5min":0.57099999999999995,"15min":0.45600000000000002},"max":{"1min":1.7490000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.087},{"interface":"front","average":{"1min":1.097,"5min":1.089,"15min":1.1499999999999999},"min":{"1min":0.66900000000000004,"5min":0.64300000000000002,"15min":0.58299999999999996},"max":{"1min":1.5660000000000001,"5min":1.8,"15min":1.8},"last":1.4279999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1579999999999999,"15min":1.179},"min":{"1min":0.64600000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5629999999999999,"5min":1.633,"15min":1.744},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.228,"15min":1.252},"min":{"1min":0.71799999999999997,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5900000000000001,"5min":1.8540000000000001,"15min":1.889},"last":1.506}]},{"osd":2,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.272,"15min":1.2889999999999999},"min":{"1min":0.879,"5min":0.59699999999999998,"15min":0.58799999999999997},"max":{"1min":1.714,"5min":1.714,"15min":1.9359999999999999},"last":1.6279999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3280000000000001,"15min":1.298},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.57499999999999996},"max":{"1min":1.77,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.6679999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2470000000000001,"15min":1.27},"min":{"1min":0.76800000000000002,"5min":0.73199999999999998,"15min":0.54200000000000004},"max":{"1min":1.583,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.292},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4139999999999999,"15min":1.4099999999999999},"min":{"1min":1.0609999999999999,"5min":0.77000000000000002,"15min":0.68300000000000005},"max":{"1min":1.7869999999999999,"5min":1.956,"15min":1.956},"last":1.9610000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.335,"15min":1.29},"min":{"1min":0.89800000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0430000000000001,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.573},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.347,"15min":1.3320000000000001},"min":{"1min":1.1399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.877,"5min":1.99,"15min":1.9970000000000001},"last":1.385}]},{"osd":5,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.4390000000000001,"15min":1.4179999999999999},"min":{"1min":1.1919999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.891,"5min":1.891,"15min":2.048},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.496,"15min":1.474},"min":{"1min":1.288,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.7330000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.581,"15min":1.5589999999999999},"min":{"1min":1,"5min":0.85799999999999998,"15min":0.69499999999999995},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.476,"15min":1.4830000000000001},"min":{"1min":1.0640000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7969999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.1850000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575091,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37132,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1830912,"data_stored":927890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0469999999999999,"15min":1.095},"min":{"1min":0.745,"5min":0.58099999999999996,"15min":0.53500000000000003},"max":{"1min":1.5900000000000001,"5min":1.6319999999999999,"15min":1.7350000000000001},"last":1.222},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.091,"15min":1.1299999999999999},"min":{"1min":0.69699999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4059999999999999,"5min":1.8069999999999999,"15min":1.8260000000000001},"last":1.1639999999999999}]},{"osd":1,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.196,"15min":1.24},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.651,"5min":1.728,"15min":1.8069999999999999},"last":1.343},{"interface":"front","average":{"1min":1.385,"5min":1.292,"15min":1.349},"min":{"1min":1.1100000000000001,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.6759999999999999,"5min":1.7889999999999999,"15min":2.0169999999999999},"last":1.4910000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.2470000000000001,"15min":1.2829999999999999},"min":{"1min":0.88300000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":1.429},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.3200000000000001,"15min":1.361},"min":{"1min":1.2190000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.837,"5min":1.851,"15min":1.851},"last":1.4019999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.264,"15min":1.28},"min":{"1min":0.75,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.9299999999999999},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.258,"15min":1.2789999999999999},"min":{"1min":1.002,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.7450000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.091}]},{"osd":4,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.266,"15min":1.246},"min":{"1min":0.78300000000000003,"5min":0.76300000000000001,"15min":0.72899999999999998},"max":{"1min":1.6479999999999999,"5min":1.7529999999999999,"15min":2.089},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.298,"5min":1.3089999999999999,"15min":1.3049999999999999},"min":{"1min":0.92400000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.798,"5min":1.9179999999999999,"15min":2.2010000000000001},"last":1.27}]},{"osd":5,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.351,"15min":1.337},"min":{"1min":0.95999999999999996,"5min":0.88800000000000001,"15min":0.754},"max":{"1min":1.879,"5min":1.879,"15min":1.97},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.514,"15min":1.528},"min":{"1min":1.0669999999999999,"5min":0.93400000000000005,"15min":0.78000000000000003},"max":{"1min":1.901,"5min":1.9179999999999999,"15min":1.958},"last":1.526}]},{"osd":7,"last update":"Tue Nov 28 20:53:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4399999999999999,"15min":1.46},"min":{"1min":0.94999999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.849,"5min":1.9790000000000001,"15min":2.3759999999999999},"last":1.377},{"interface":"front","average":{"1min":1.425,"5min":1.4550000000000001,"15min":1.4530000000000001},"min":{"1min":1.1599999999999999,"5min":0.96099999999999997,"15min":0.92900000000000005},"max":{"1min":1.9259999999999999,"5min":1.9630000000000001,"15min":2.0739999999999998},"last":1.5489999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575091,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51032,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320808,"statfs":{"total":96636764160,"available":96584507392,"internally_reserved":0,"allocated":1847296,"data_stored":934116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1619999999999999,"15min":1.1639999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.696,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.091},{"interface":"front","average":{"1min":1.056,"5min":1.0660000000000001,"15min":1.1439999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.441},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1160000000000001,"15min":1.147},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.623,"5min":1.623,"15min":1.768},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.2190000000000001,"15min":1.2130000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6100000000000001,"5min":2.282,"15min":2.282},"last":0.97299999999999998}]},{"osd":2,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.302,"15min":1.2829999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.53200000000000003},"max":{"1min":1.893,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.212,"5min":1.21,"15min":1.274},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.66600000000000004},"max":{"1min":1.6240000000000001,"5min":1.8220000000000001,"15min":1.958},"last":1.1279999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.286,"15min":1.3140000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.726,"5min":1.7290000000000001,"15min":1.9350000000000001},"last":1.327},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.2589999999999999,"15min":1.262},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.64000000000000001},"max":{"1min":1.6559999999999999,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.361}]},{"osd":4,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3140000000000001,"15min":1.2989999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.67500000000000004},"max":{"1min":1.746,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.585},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4710000000000001,"15min":1.4379999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.8160000000000001,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":1.284}]},{"osd":6,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3380000000000001,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.5},"max":{"1min":1.966,"5min":1.966,"15min":2.093},"last":1.02},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.474,"15min":1.476},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.69499999999999995},"max":{"1min":1.9219999999999999,"5min":1.998,"15min":1.998},"last":1.661}]},{"osd":7,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.464,"15min":1.4330000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.742,"5min":3.319,"15min":3.319},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.474,"15min":1.4830000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.60499999999999998},"max":{"1min":1.764,"5min":1.9790000000000001,"15min":2.1829999999999998},"last":1.234}]}]},{"osd":4,"up_from":13,"seq":55834575092,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37152,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334688,"statfs":{"total":96636764160,"available":96598720512,"internally_reserved":0,"allocated":1851392,"data_stored":935568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0580000000000001,"15min":1.125},"min":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.825,"5min":1.825,"15min":1.879},"last":1.5},{"interface":"front","average":{"1min":0.995,"5min":1.0720000000000001,"15min":1.1080000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.3839999999999999,"5min":1.6850000000000001,"15min":1.9750000000000001},"last":0.751}]},{"osd":1,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2110000000000001,"15min":1.286},"min":{"1min":0.68999999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.923},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.335,"5min":1.2689999999999999,"15min":1.343},"min":{"1min":0.72099999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7470000000000001,"5min":1.768,"15min":2.2679999999999998},"last":1.546}]},{"osd":2,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2010000000000001,"15min":1.2190000000000001},"min":{"1min":0.56899999999999995,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.7729999999999999,"5min":1.78,"15min":1.78},"last":1.002},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2350000000000001,"15min":1.282},"min":{"1min":0.65000000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.704,"5min":1.704,"15min":2.6949999999999998},"last":2.218}]},{"osd":3,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.2230000000000001,"15min":1.2709999999999999},"min":{"1min":0.63,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.647,"5min":1.9630000000000001,"15min":2.157},"last":4.117},{"interface":"front","average":{"1min":1.3,"5min":1.3149999999999999,"15min":1.3580000000000001},"min":{"1min":0.68700000000000006,"5min":0.5,"15min":0.5},"max":{"1min":1.7509999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":4.843}]},{"osd":5,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.286,"15min":1.292},"min":{"1min":0.65500000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.849,"5min":1.9970000000000001,"15min":2.4279999999999999},"last":3.262},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.2909999999999999,"15min":1.298},"min":{"1min":0.746,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.784,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.3600000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.3979999999999999,"15min":1.395},"min":{"1min":0.96699999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.8899999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.5289999999999999,"15min":1.5509999999999999},"min":{"1min":0.92400000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.9219999999999999,"5min":2.1349999999999998,"15min":3.7000000000000002},"last":3.3079999999999998}]},{"osd":7,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.444,"15min":1.46},"min":{"1min":0.76200000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.9159999999999999,"5min":2.2490000000000001,"15min":2.4990000000000001},"last":3.8620000000000001},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.522,"15min":1.518},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.0510000000000002,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":2.4700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575092,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41312,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330528,"statfs":{"total":96636764160,"available":96594460672,"internally_reserved":0,"allocated":1855488,"data_stored":935602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.85699999999999998,"15min":0.89400000000000002},"min":{"1min":0.47299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.383,"5min":1.6000000000000001,"15min":4.7210000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.92800000000000005,"15min":1.0229999999999999},"min":{"1min":0.39700000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.73,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":0.628}]},{"osd":1,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.92900000000000005,"15min":0.96099999999999997},"min":{"1min":0.495,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6739999999999999,"5min":1.9770000000000001,"15min":2.8460000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.071,"15min":1.1719999999999999},"min":{"1min":0.53100000000000003,"5min":0.379,"15min":0.379},"max":{"1min":1.4219999999999999,"5min":1.7190000000000001,"15min":3.7410000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1140000000000001,"15min":1.216},"min":{"1min":0.65900000000000003,"5min":0.53100000000000003,"15min":0.48499999999999999},"max":{"1min":1.5429999999999999,"5min":1.9019999999999999,"15min":3.444},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.04,"5min":1.2030000000000001,"15min":1.3029999999999999},"min":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.452,"5min":1.9530000000000001,"15min":3.7050000000000001},"last":0.86799999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2849999999999999,"15min":1.335},"min":{"1min":0.76000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8089999999999999,"5min":1.956,"15min":2.9420000000000002},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.3300000000000001,"15min":1.3700000000000001},"min":{"1min":0.91400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.8819999999999999,"5min":2.246,"15min":2.246},"last":1.3140000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3799999999999999,"15min":1.4339999999999999},"min":{"1min":0.88800000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6659999999999999,"5min":1.8959999999999999,"15min":3.7730000000000001},"last":1.417},{"interface":"front","average":{"1min":1.379,"5min":1.387,"15min":1.4330000000000001},"min":{"1min":0.81100000000000005,"5min":0.66800000000000004,"15min":0.53500000000000003},"max":{"1min":1.875,"5min":2.0110000000000001,"15min":3.1520000000000001},"last":1.1879999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.399,"15min":1.458},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.825,"5min":2.0630000000000002,"15min":2.3100000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.46,"15min":1.5509999999999999},"min":{"1min":0.97599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.679,"5min":2.0840000000000001,"15min":3.661},"last":1.4970000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.518},"min":{"1min":0.83199999999999996,"5min":0.73399999999999999,"15min":0.71499999999999997},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":3.266},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.522,"15min":1.575},"min":{"1min":0.996,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.7450000000000001,"5min":2.2080000000000002,"15min":2.3919999999999999},"last":1.216}]}]},{"osd":2,"up_from":13,"seq":55834575092,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":1826816,"data_stored":927308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.86199999999999999,"15min":0.90200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":1.2829999999999999,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.0049999999999999,"15min":1.052},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.387,"5min":4.8490000000000002,"15min":4.8490000000000002},"last":1.216}]},{"osd":1,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.96899999999999997,"15min":0.98499999999999999},"min":{"1min":0.498,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.458,"5min":1.8160000000000001,"15min":3.2599999999999998},"last":1.069},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0309999999999999,"15min":1.1240000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.357,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.97099999999999997}]},{"osd":3,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1160000000000001,"15min":1.161},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.589,"5min":4.9349999999999996,"15min":4.9349999999999996},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1970000000000001,"15min":1.282},"min":{"1min":0.64900000000000002,"5min":0.48499999999999999,"15min":0.48199999999999998},"max":{"1min":1.994,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":1.2769999999999999}]},{"osd":4,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3440000000000001,"15min":1.3979999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.673,"5min":3.6499999999999999,"15min":3.6499999999999999},"last":1.538},{"interface":"front","average":{"1min":1.363,"5min":1.448,"15min":1.4350000000000001},"min":{"1min":0.88400000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8340000000000001,"5min":4.6479999999999997,"15min":4.6479999999999997},"last":1.508}]},{"osd":5,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3819999999999999,"15min":1.413},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.923,"5min":2.492,"15min":2.492},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.429,"15min":1.4830000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6399999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.3420000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4810000000000001,"15min":1.508},"min":{"1min":0.79200000000000004,"5min":0.747,"15min":0.747},"max":{"1min":2.153,"5min":4.5270000000000001,"15min":4.5270000000000001},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.5329999999999999,"15min":1.5660000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.82499999999999996},"max":{"1min":1.8560000000000001,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.4730000000000001}]},{"osd":7,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4650000000000001,"15min":1.524},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.0720000000000001,"5min":2.641,"15min":2.641},"last":1.78},{"interface":"front","average":{"1min":1.484,"5min":1.548,"15min":1.585},"min":{"1min":0.81799999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.2770000000000001,"5min":4.7530000000000001,"15min":4.7530000000000001},"last":1.6539999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575092,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37144,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334696,"statfs":{"total":96636764160,"available":96598728704,"internally_reserved":0,"allocated":1843200,"data_stored":933978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.90800000000000003,"15min":0.91100000000000003},"min":{"1min":0.38900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.466,"5min":1.8009999999999999,"15min":1.9259999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.98899999999999999,"15min":1.069},"min":{"1min":0.505,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.6120000000000001,"5min":2.028,"15min":2.028},"last":0.90800000000000003}]},{"osd":2,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0389999999999999,"15min":1.0740000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.371},"max":{"1min":1.3959999999999999,"5min":1.766,"15min":2.234},"last":0.498},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.153,"15min":1.1910000000000001},"min":{"1min":0.628,"5min":0.53700000000000003,"15min":0.52800000000000002},"max":{"1min":1.4259999999999999,"5min":1.9199999999999999,"15min":2.895},"last":1.198}]},{"osd":3,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.2110000000000001,"15min":1.2889999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.46700000000000003},"max":{"1min":1.5109999999999999,"5min":1.7949999999999999,"15min":2.0459999999999998},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.0760000000000001,"15min":1.101},"min":{"1min":0.47299999999999998,"5min":0.45100000000000001,"15min":0.37},"max":{"1min":1.756,"5min":1.9790000000000001,"15min":2.3679999999999999},"last":0.52600000000000002}]},{"osd":4,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3080000000000001,"15min":1.339},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5629999999999999,"5min":1.8140000000000001,"15min":2.1150000000000002},"last":1.02},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.383,"15min":1.391},"min":{"1min":0.77000000000000002,"5min":0.72399999999999998,"15min":0.71899999999999997},"max":{"1min":1.776,"5min":1.8819999999999999,"15min":2.6819999999999999},"last":0.81000000000000005}]},{"osd":5,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.4299999999999999,"15min":1.4339999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.9650000000000001,"5min":2.0190000000000001,"15min":2.1840000000000002},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.4450000000000001,"15min":1.482},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.9530000000000001,"5min":2.0219999999999998,"15min":2.294},"last":1.2869999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.399,"15min":1.4179999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.66900000000000004},"max":{"1min":1.772,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.502,"15min":1.524},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.77100000000000002},"max":{"1min":1.9139999999999999,"5min":2.0049999999999999,"15min":2.3300000000000001},"last":0.83899999999999997}]},{"osd":7,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.51,"15min":1.552},"min":{"1min":0.80800000000000005,"5min":0.80500000000000005,"15min":0.753},"max":{"1min":1.8959999999999999,"5min":2.1429999999999998,"15min":2.3359999999999999},"last":1.355},{"interface":"front","average":{"1min":1.47,"5min":1.488,"15min":1.5209999999999999},"min":{"1min":0.98999999999999999,"5min":0.80500000000000005,"15min":0.72799999999999998},"max":{"1min":2.165,"5min":2.2330000000000001,"15min":2.294},"last":1.1519999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575092,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2424832,"data_stored":1517698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.84599999999999997,"15min":0.89400000000000002},"min":{"1min":0.45200000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.3919999999999999,"5min":1.4610000000000001,"15min":2.1440000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.034,"5min":1.05,"15min":1.0860000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4730000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.372}]},{"osd":2,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0109999999999999,"15min":1.099},"min":{"1min":0.504,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.772,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.091,"15min":1.1679999999999999},"min":{"1min":0.496,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.649,"5min":1.782,"15min":1.9159999999999999},"last":0.48099999999999998}]},{"osd":3,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.03,"15min":1.099},"min":{"1min":0.629,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.4079999999999999,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.432},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.3},"min":{"1min":0.55300000000000005,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.6160000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.55000000000000004}]},{"osd":4,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.347,"15min":1.4139999999999999},"min":{"1min":0.72799999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.6140000000000001,"5min":1.804,"15min":2.012},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.353,"15min":1.3839999999999999},"min":{"1min":0.79700000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.698,"5min":4.5119999999999996,"15min":4.5119999999999996},"last":0.94199999999999995}]},{"osd":5,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.4119999999999999,"15min":1.419},"min":{"1min":0.99099999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9159999999999999,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.377,"5min":1.423,"15min":1.4770000000000001},"min":{"1min":0.879,"5min":0.621,"15min":0.621},"max":{"1min":2,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.0409999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.5069999999999999,"15min":1.5449999999999999},"min":{"1min":0.92900000000000005,"5min":0.73799999999999999,"15min":0.73699999999999999},"max":{"1min":2.0750000000000002,"5min":2.524,"15min":2.524},"last":0.871},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.464,"15min":1.5169999999999999},"min":{"1min":0.95399999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.833,"5min":2.274,"15min":2.4209999999999998},"last":1.0629999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:53:38 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.5469999999999999,"15min":1.5880000000000001},"min":{"1min":0.995,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.921,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.48,"15min":1.5349999999999999},"min":{"1min":1.079,"5min":0.80300000000000005,"15min":0.73799999999999999},"max":{"1min":1.8200000000000001,"5min":4.7359999999999998,"15min":4.7359999999999998},"last":0.89700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:54:32.457 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T20:54:32.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T20:54:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.720+0000 7f7e1506b640 1 Processor -- start 2023-11-28T20:54:32.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.720+0000 7f7e1506b640 1 -- start start 2023-11-28T20:54:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.722+0000 7f7e1506b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7e101607e0 0x7f7e1006cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.722+0000 7f7e1506b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e10071620 0x7f7e100ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.722+0000 7f7e1506b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e100b30b0 con 0x7f7e10070a60 2023-11-28T20:54:32.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.722+0000 7f7e1506b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e100b3300 con 0x7f7e101607e0 2023-11-28T20:54:32.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.722+0000 7f7e1506b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100b35e0 con 0x7f7e10071620 2023-11-28T20:54:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.723+0000 7f7e0ed9d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7e101607e0 0x7f7e1006cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.723+0000 7f7e07fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e10071620 0x7f7e100ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.723+0000 7f7e07fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e10071620 0x7f7e100ad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:42344/0 (socket says 172.21.15.45:42344) 2023-11-28T20:54:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.723+0000 7f7e07fff640 1 -- 172.21.15.45:0/2022707600 learned_addr learned my addr 172.21.15.45:0/2022707600 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:32.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.724+0000 7f7e07fff640 1 -- 172.21.15.45:0/2022707600 >> v2:172.21.15.190:3300/0 conn(0x7f7e101607e0 msgr2=0x7f7e1006cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:32.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.724+0000 7f7e07fff640 1 --2- 172.21.15.45:0/2022707600 >> v2:172.21.15.190:3300/0 conn(0x7f7e101607e0 0x7f7e1006cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.724+0000 7f7e07fff640 1 -- 172.21.15.45:0/2022707600 >> v1:172.21.15.190:6789/0 conn(0x7f7e10070a60 legacy=0x7f7e100b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:32.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.724+0000 7f7e07fff640 1 -- 172.21.15.45:0/2022707600 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100b38c0 con 0x7f7e10071620 2023-11-28T20:54:32.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.724+0000 7f7e0ed9d640 1 --2- 172.21.15.45:0/2022707600 >> v2:172.21.15.190:3300/0 conn(0x7f7e101607e0 0x7f7e1006cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:54:32.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.725+0000 7f7e07fff640 1 --2- 172.21.15.45:0/2022707600 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e10071620 0x7f7e100ad2c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009870 tx=0x7f7df802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64e017cdd4f6b99e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.726+0000 7f7e0e59c640 1 -- 172.21.15.45:0/2022707600 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8030840 con 0x7f7e10071620 2023-11-28T20:54:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.726+0000 7f7e0e59c640 1 -- 172.21.15.45:0/2022707600 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df80309a0 con 0x7f7e10071620 2023-11-28T20:54:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.726+0000 7f7e0e59c640 1 -- 172.21.15.45:0/2022707600 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8030d30 con 0x7f7e10071620 2023-11-28T20:54:32.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.727+0000 7f7e1506b640 1 -- 172.21.15.45:0/2022707600 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e10071620 msgr2=0x7f7e100ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:32.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.727+0000 7f7e1506b640 1 --2- 172.21.15.45:0/2022707600 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e10071620 0x7f7e100ad2c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009870 tx=0x7f7df802b980 comp rx=0 tx=0).stop 2023-11-28T20:54:32.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.727+0000 7f7e1506b640 1 -- 172.21.15.45:0/2022707600 shutdown_connections 2023-11-28T20:54:32.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.727+0000 7f7e1506b640 1 --2- 172.21.15.45:0/2022707600 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e10071620 0x7f7e100ad2c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.727+0000 7f7e1506b640 1 --2- 172.21.15.45:0/2022707600 >> v2:172.21.15.190:3300/0 conn(0x7f7e101607e0 0x7f7e1006cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.727+0000 7f7e1506b640 1 -- 172.21.15.45:0/2022707600 >> 172.21.15.45:0/2022707600 conn(0x7f7e100b13a0 msgr2=0x7f7e100b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:32.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.727+0000 7f7e1506b640 1 -- 172.21.15.45:0/2022707600 shutdown_connections 2023-11-28T20:54:32.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.727+0000 7f7e1506b640 1 -- 172.21.15.45:0/2022707600 wait complete. 2023-11-28T20:54:32.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.729+0000 7f7e1506b640 1 Processor -- start 2023-11-28T20:54:32.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.729+0000 7f7e1506b640 1 -- start start 2023-11-28T20:54:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.730+0000 7f7e1506b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e10070a60 0x7f7e10149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.730+0000 7f7e1506b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e10071620 0x7f7e1014cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.730+0000 7f7e1506b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e101607e0 0x7f7e100c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.730+0000 7f7e1506b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100c5600 con 0x7f7e10071620 2023-11-28T20:54:32.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.730+0000 7f7e1506b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1014d300 con 0x7f7e10070a60 2023-11-28T20:54:32.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.730+0000 7f7e1506b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100c49f0 con 0x7f7e101607e0 2023-11-28T20:54:32.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.730+0000 7f7e07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e10071620 0x7f7e1014cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.730+0000 7f7e07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e10071620 0x7f7e1014cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48346/0 (socket says 172.21.15.45:48346) 2023-11-28T20:54:32.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.730+0000 7f7e07fff640 1 -- 172.21.15.45:0/3253663514 learned_addr learned my addr 172.21.15.45:0/3253663514 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:32.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e0ed9d640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e10070a60 0x7f7e10149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e0f59e640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e101607e0 0x7f7e100c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e07fff640 1 -- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e101607e0 msgr2=0x7f7e100c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:32.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e07fff640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e101607e0 0x7f7e100c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e07fff640 1 -- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e10070a60 msgr2=0x7f7e10149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:32.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e07fff640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e10070a60 0x7f7e10149c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:32.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e07fff640 1 -- 172.21.15.45:0/3253663514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00009400 con 0x7f7e10071620 2023-11-28T20:54:32.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e07fff640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e10071620 0x7f7e1014cdc0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009990 tx=0x7f7df8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:32.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e077fe640 1 -- 172.21.15.45:0/3253663514 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8039c90 con 0x7f7e10071620 2023-11-28T20:54:32.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e077fe640 1 -- 172.21.15.45:0/3253663514 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df8039df0 con 0x7f7e10071620 2023-11-28T20:54:32.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e077fe640 1 -- 172.21.15.45:0/3253663514 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8043620 con 0x7f7e10071620 2023-11-28T20:54:32.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.731+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df80094d0 con 0x7f7e10071620 2023-11-28T20:54:32.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.732+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e100c4fb0 con 0x7f7e10071620 2023-11-28T20:54:32.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.733+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e100c57b0 con 0x7f7e10071620 2023-11-28T20:54:32.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.734+0000 7f7e077fe640 1 -- 172.21.15.45:0/3253663514 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7df8043780 con 0x7f7e10071620 2023-11-28T20:54:32.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.735+0000 7f7e077fe640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f7df006af60 0x7f7df006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:32.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.735+0000 7f7e077fe640 1 -- 172.21.15.45:0/3253663514 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7df803e070 con 0x7f7e10071620 2023-11-28T20:54:32.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.735+0000 7f7e0ed9d640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f7df006af60 0x7f7df006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:32.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.736+0000 7f7e0ed9d640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f7df006af60 0x7f7df006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7e101f6a90 tx=0x7f7e00008040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:32.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:32.739+0000 7f7e077fe640 1 -- 172.21.15.45:0/3253663514 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e100c57b0 con 0x7f7e10071620 2023-11-28T20:54:33.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.006+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7e10160340 con 0x7f7e10071620 2023-11-28T20:54:33.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.012+0000 7f7e077fe640 1 -- 172.21.15.45:0/3253663514 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7df8034020 con 0x7f7e10071620 2023-11-28T20:54:33.014 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T20:54:33.015 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:54:33.015 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T20:54:33.015 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T20:54:33.015 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T20:54:33.015 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T20:54:33.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f7df006af60 msgr2=0x7f7df006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:33.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f7df006af60 0x7f7df006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7e101f6a90 tx=0x7f7e00008040 comp rx=0 tx=0).stop 2023-11-28T20:54:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e10071620 msgr2=0x7f7e1014cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e10071620 0x7f7e1014cdc0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009990 tx=0x7f7df8004730 comp rx=0 tx=0).stop 2023-11-28T20:54:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 shutdown_connections 2023-11-28T20:54:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f7df006af60 0x7f7df006d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e101607e0 0x7f7e100c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e10071620 0x7f7e1014cdc0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 --2- 172.21.15.45:0/3253663514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e10070a60 0x7f7e10149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 >> 172.21.15.45:0/3253663514 conn(0x7f7e100b13a0 msgr2=0x7f7e100b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:33.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 shutdown_connections 2023-11-28T20:54:33.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.015+0000 7f7e1506b640 1 -- 172.21.15.45:0/3253663514 wait complete. 2023-11-28T20:54:33.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:33.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.257+0000 7f47b3367640 1 Processor -- start 2023-11-28T20:54:33.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.257+0000 7f47b3367640 1 -- start start 2023-11-28T20:54:33.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.259+0000 7f47b3367640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 0x7f47ac0ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:33.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.259+0000 7f47b3367640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f47ac0ce5f0 0x7f47ac0cfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:33.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.259+0000 7f47b3367640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac0d1110 con 0x7f47ac15d2d0 2023-11-28T20:54:33.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.259+0000 7f47b3367640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ac0d2070 con 0x7f47ac0ce5f0 2023-11-28T20:54:33.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.259+0000 7f47b3367640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac0d13f0 con 0x7f47ac15d6b0 2023-11-28T20:54:33.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.259+0000 7f47b2365640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f47ac15d2d0 0x7f47ac0b63a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42520/0 (socket says 172.21.15.45:42520) 2023-11-28T20:54:33.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.259+0000 7f47b2365640 1 -- 172.21.15.45:0/1411751506 learned_addr learned my addr 172.21.15.45:0/1411751506 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:33.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.259+0000 7f47b2b66640 1 --2- 172.21.15.45:0/1411751506 >> v2:172.21.15.45:3300/0 conn(0x7f47ac0ce5f0 0x7f47ac0cfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:33.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.260+0000 7f47b1b64640 1 --2- 172.21.15.45:0/1411751506 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 0x7f47ac0ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:33.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.260+0000 7f47b1363640 1 -- 172.21.15.45:0/1411751506 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218462705 0 0) 0x7f47ac0d1110 con 0x7f47ac15d2d0 2023-11-28T20:54:33.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.260+0000 7f47b1363640 1 -- 172.21.15.45:0/1411751506 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4798003520 con 0x7f47ac15d2d0 2023-11-28T20:54:33.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.260+0000 7f47b2b66640 1 -- 172.21.15.45:0/1411751506 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 msgr2=0x7f47ac0ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:33.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.260+0000 7f47b2b66640 1 --2- 172.21.15.45:0/1411751506 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 0x7f47ac0ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.260+0000 7f47b2b66640 1 -- 172.21.15.45:0/1411751506 >> v1:172.21.15.45:6789/0 conn(0x7f47ac15d2d0 legacy=0x7f47ac0b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:33.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.260+0000 7f47b2b66640 1 -- 172.21.15.45:0/1411751506 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac0d16d0 con 0x7f47ac0ce5f0 2023-11-28T20:54:33.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.261+0000 7f47b2b66640 1 --2- 172.21.15.45:0/1411751506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac0ce5f0 0x7f47ac0cfbb0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009d90 tx=0x7f47a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2135aba210582831 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b1363640 1 -- 172.21.15.45:0/1411751506 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a402cc50 con 0x7f47ac0ce5f0 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b1363640 1 -- 172.21.15.45:0/1411751506 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a402cdb0 con 0x7f47ac0ce5f0 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b1363640 1 -- 172.21.15.45:0/1411751506 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a4034640 con 0x7f47ac0ce5f0 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b3367640 1 -- 172.21.15.45:0/1411751506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac0ce5f0 msgr2=0x7f47ac0cfbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b3367640 1 --2- 172.21.15.45:0/1411751506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac0ce5f0 0x7f47ac0cfbb0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009d90 tx=0x7f47a402bea0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b3367640 1 -- 172.21.15.45:0/1411751506 shutdown_connections 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b3367640 1 --2- 172.21.15.45:0/1411751506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac0ce5f0 0x7f47ac0cfbb0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b3367640 1 --2- 172.21.15.45:0/1411751506 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 0x7f47ac0ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b3367640 1 -- 172.21.15.45:0/1411751506 >> 172.21.15.45:0/1411751506 conn(0x7f47ac06a500 msgr2=0x7f47ac06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b3367640 1 -- 172.21.15.45:0/1411751506 shutdown_connections 2023-11-28T20:54:33.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.262+0000 7f47b3367640 1 -- 172.21.15.45:0/1411751506 wait complete. 2023-11-28T20:54:33.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.264+0000 7f47b3367640 1 Processor -- start 2023-11-28T20:54:33.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.264+0000 7f47b3367640 1 -- start start 2023-11-28T20:54:33.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.265+0000 7f47b3367640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac0ce5f0 0x7f47ac144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.265+0000 7f47b3367640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac15d2d0 0x7f47ac1f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.265+0000 7f47b3367640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 0x7f47ac0b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.265+0000 7f47b3367640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac15ec90 con 0x7f47ac15d2d0 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.265+0000 7f47b3367640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac1f7220 con 0x7f47ac0ce5f0 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.265+0000 7f47b3367640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac1f7500 con 0x7f47ac15d6b0 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.265+0000 7f47b1b64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac15d2d0 0x7f47ac1f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b1b64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac15d2d0 0x7f47ac1f67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48370/0 (socket says 172.21.15.45:48370) 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b1b64640 1 -- 172.21.15.45:0/4234792390 learned_addr learned my addr 172.21.15.45:0/4234792390 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b2b66640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 0x7f47ac0b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b2365640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac0ce5f0 0x7f47ac144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:33.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b1b64640 1 -- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 msgr2=0x7f47ac0b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b1b64640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 0x7f47ac0b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b1b64640 1 -- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac0ce5f0 msgr2=0x7f47ac144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b1b64640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac0ce5f0 0x7f47ac144aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b1b64640 1 -- 172.21.15.45:0/4234792390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a4009a70 con 0x7f47ac15d2d0 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47b1b64640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac15d2d0 0x7f47ac1f67d0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f479c00d5d0 tx=0x7f479c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.266+0000 7f47a2ffd640 1 -- 172.21.15.45:0/4234792390 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c0125f0 con 0x7f47ac15d2d0 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.267+0000 7f47a2ffd640 1 -- 172.21.15.45:0/4234792390 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479c012750 con 0x7f47ac15d2d0 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.267+0000 7f47a2ffd640 1 -- 172.21.15.45:0/4234792390 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c012ae0 con 0x7f47ac15d2d0 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.267+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ac15ef50 con 0x7f47ac15d2d0 2023-11-28T20:54:33.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.267+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac0c7440 con 0x7f47ac15d2d0 2023-11-28T20:54:33.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.268+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ac071ba0 con 0x7f47ac15d2d0 2023-11-28T20:54:33.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.269+0000 7f47a2ffd640 1 -- 172.21.15.45:0/4234792390 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f479c012c40 con 0x7f47ac15d2d0 2023-11-28T20:54:33.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.270+0000 7f47a2ffd640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f479006af60 0x7f479006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:33.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.270+0000 7f47a2ffd640 1 -- 172.21.15.45:0/4234792390 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f479c089e00 con 0x7f47ac15d2d0 2023-11-28T20:54:33.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.271+0000 7f47b2365640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f479006af60 0x7f479006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:33.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.272+0000 7f47b2365640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f479006af60 0x7f479006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0b5d40 tx=0x7f47a8009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:33.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.274+0000 7f47a2ffd640 1 -- 172.21.15.45:0/4234792390 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f479c054bb0 con 0x7f47ac15d2d0 2023-11-28T20:54:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.507+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47ac0b25c0 con 0x7f47ac15d2d0 2023-11-28T20:54:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.508+0000 7f47a2ffd640 1 -- 172.21.15.45:0/4234792390 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f479c0581d0 con 0x7f47ac15d2d0 2023-11-28T20:54:33.510 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:33.511 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/1133315976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1133315976},{"type":"v1","addr":"172.21.15.190:6836","nonce":1133315976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2810069664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2810069664},{"type":"v1","addr":"172.21.15.190:6837","nonce":2810069664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1553174870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1553174870},{"type":"v1","addr":"172.21.15.45:6833","nonce":1553174870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:54:26.355691+0000","modified":"2023-11-28T20:54:29.430641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8595":{"gid":8595,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/684266044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":684266044},{"type":"v1","addr":"172.21.15.45:6835","nonce":684266044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-28T20:54:33.511 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 67 2023-11-28T20:54:33.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f479006af60 msgr2=0x7f479006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:33.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f479006af60 0x7f479006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0b5d40 tx=0x7f47a8009040 comp rx=0 tx=0).stop 2023-11-28T20:54:33.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac15d2d0 msgr2=0x7f47ac1f67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:33.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac15d2d0 0x7f47ac1f67d0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f479c00d5d0 tx=0x7f479c00da90 comp rx=0 tx=0).stop 2023-11-28T20:54:33.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 shutdown_connections 2023-11-28T20:54:33.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f479006af60 0x7f479006d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f47ac15d6b0 0x7f47ac0b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47ac15d2d0 0x7f47ac1f67d0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 --2- 172.21.15.45:0/4234792390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47ac0ce5f0 0x7f47ac144aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:33.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 >> 172.21.15.45:0/4234792390 conn(0x7f47ac06a500 msgr2=0x7f47ac0d3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:33.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 shutdown_connections 2023-11-28T20:54:33.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:33.511+0000 7f47b3367640 1 -- 172.21.15.45:0/4234792390 wait complete. 2023-11-28T20:54:33.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:54:26.355691+0000', 'modified': '2023-11-28T20:54:29.430641+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8595': {'gid': 8595, 'name': 'c', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/684266044', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6834', 'nonce': 684266044}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 684266044}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T20:54:33.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:54:33.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T20:54:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.170+0000 7fe910996640 1 Processor -- start 2023-11-28T20:54:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.171+0000 7fe910996640 1 -- start start 2023-11-28T20:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.172+0000 7fe910996640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe90c160e70 0x7fe90c0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.172+0000 7fe910996640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c0ce3a0 0x7fe90c0d19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.172+0000 7fe910996640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90c0d14d0 con 0x7fe90c15d310 2023-11-28T20:54:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.172+0000 7fe910996640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90c0ce810 con 0x7fe90c160e70 2023-11-28T20:54:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.172+0000 7fe910996640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c1613c0 con 0x7fe90c0ce3a0 2023-11-28T20:54:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.173+0000 7fe903fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe90c160e70 0x7fe90c0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.173+0000 7fe90b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c0ce3a0 0x7fe90c0d19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.173+0000 7fe90b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c0ce3a0 0x7fe90c0d19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:43230/0 (socket says 172.21.15.45:43230) 2023-11-28T20:54:34.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.173+0000 7fe90b59e640 1 -- 172.21.15.45:0/3570323485 learned_addr learned my addr 172.21.15.45:0/3570323485 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:34.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.173+0000 7fe903fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe90c160e70 0x7fe90c0cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:45144/0 (socket says 172.21.15.45:45144) 2023-11-28T20:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.174+0000 7fe90b59e640 1 -- 172.21.15.45:0/3570323485 >> v2:172.21.15.190:3300/0 conn(0x7fe90c160e70 msgr2=0x7fe90c0cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.174+0000 7fe90b59e640 1 --2- 172.21.15.45:0/3570323485 >> v2:172.21.15.190:3300/0 conn(0x7fe90c160e70 0x7fe90c0cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.174+0000 7fe90b59e640 1 -- 172.21.15.45:0/3570323485 >> v1:172.21.15.190:6789/0 conn(0x7fe90c15d310 legacy=0x7fe90c0b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.174+0000 7fe90b59e640 1 -- 172.21.15.45:0/3570323485 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c1616a0 con 0x7fe90c0ce3a0 2023-11-28T20:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.176+0000 7fe90b59e640 1 --2- 172.21.15.45:0/3570323485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c0ce3a0 0x7fe90c0d19d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe904009e50 tx=0x7fe90402bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97aecc86cb329b28 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.176+0000 7fe90a59c640 1 -- 172.21.15.45:0/3570323485 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90402ca80 con 0x7fe90c0ce3a0 2023-11-28T20:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.176+0000 7fe90a59c640 1 -- 172.21.15.45:0/3570323485 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90402cbe0 con 0x7fe90c0ce3a0 2023-11-28T20:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.176+0000 7fe90a59c640 1 -- 172.21.15.45:0/3570323485 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9040343f0 con 0x7fe90c0ce3a0 2023-11-28T20:54:34.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.177+0000 7fe910996640 1 -- 172.21.15.45:0/3570323485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c0ce3a0 msgr2=0x7fe90c0d19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.177+0000 7fe910996640 1 --2- 172.21.15.45:0/3570323485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c0ce3a0 0x7fe90c0d19d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe904009e50 tx=0x7fe90402bee0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.177+0000 7fe910996640 1 -- 172.21.15.45:0/3570323485 shutdown_connections 2023-11-28T20:54:34.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.177+0000 7fe910996640 1 --2- 172.21.15.45:0/3570323485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c0ce3a0 0x7fe90c0d19d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.177+0000 7fe910996640 1 --2- 172.21.15.45:0/3570323485 >> v2:172.21.15.190:3300/0 conn(0x7fe90c160e70 0x7fe90c0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.177+0000 7fe910996640 1 -- 172.21.15.45:0/3570323485 >> 172.21.15.45:0/3570323485 conn(0x7fe90c06a500 msgr2=0x7fe90c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.177+0000 7fe910996640 1 -- 172.21.15.45:0/3570323485 shutdown_connections 2023-11-28T20:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.177+0000 7fe910996640 1 -- 172.21.15.45:0/3570323485 wait complete. 2023-11-28T20:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.179+0000 7fe910996640 1 Processor -- start 2023-11-28T20:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.179+0000 7fe910996640 1 -- start start 2023-11-28T20:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.180+0000 7fe910996640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe90c0ce3a0 0x7fe90c14a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.180+0000 7fe910996640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c15d310 0x7fe90c15e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.180+0000 7fe910996640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c160e70 0x7fe90c0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.180+0000 7fe910996640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c15f930 con 0x7fe90c15d310 2023-11-28T20:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.180+0000 7fe910996640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c15ed20 con 0x7fe90c0ce3a0 2023-11-28T20:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.180+0000 7fe910996640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c15f000 con 0x7fe90c160e70 2023-11-28T20:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe903fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c15d310 0x7fe90c15e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe903fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c15d310 0x7fe90c15e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33318/0 (socket says 172.21.15.45:33318) 2023-11-28T20:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe903fff640 1 -- 172.21.15.45:0/3678012938 learned_addr learned my addr 172.21.15.45:0/3678012938 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe90c0ce3a0 0x7fe90c14a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe90b59e640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c160e70 0x7fe90c0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe903fff640 1 -- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c160e70 msgr2=0x7fe90c0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe903fff640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c160e70 0x7fe90c0c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe903fff640 1 -- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe90c0ce3a0 msgr2=0x7fe90c14a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe903fff640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe90c0ce3a0 0x7fe90c14a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe903fff640 1 -- 172.21.15.45:0/3678012938 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe904009b30 con 0x7fe90c15d310 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe90b59e640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c160e70 0x7fe90c0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.181+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe90c0ce3a0 0x7fe90c14a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.182+0000 7fe903fff640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c15d310 0x7fe90c15e250 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400e870 tx=0x7fe8f400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.182+0000 7fe9037fe640 1 -- 172.21.15.45:0/3678012938 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40094f0 con 0x7fe90c15d310 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.182+0000 7fe9037fe640 1 -- 172.21.15.45:0/3678012938 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f400c940 con 0x7fe90c15d310 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.182+0000 7fe9037fe640 1 -- 172.21.15.45:0/3678012938 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f400ccf0 con 0x7fe90c15d310 2023-11-28T20:54:34.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.182+0000 7fe910996640 1 -- 172.21.15.45:0/3678012938 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c15f2e0 con 0x7fe90c15d310 2023-11-28T20:54:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.182+0000 7fe910996640 1 -- 172.21.15.45:0/3678012938 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c0b4c20 con 0x7fe90c15d310 2023-11-28T20:54:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.184+0000 7fe9037fe640 1 -- 172.21.15.45:0/3678012938 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe8f4013470 con 0x7fe90c15d310 2023-11-28T20:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.185+0000 7fe9037fe640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe8ec06af90 0x7fe8ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.186+0000 7fe9037fe640 1 -- 172.21.15.45:0/3678012938 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe8f40899e0 con 0x7fe90c15d310 2023-11-28T20:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.186+0000 7fe913e0a740 1 Processor -- start 2023-11-28T20:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.186+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe8ec06af90 0x7fe8ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.186+0000 7fe913e0a740 1 -- start start 2023-11-28T20:54:34.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.187+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 0x55c5a4504450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.187+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe8ec06af90 0x7fe8ec06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc001680 tx=0x7fe8fc009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.187+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 0x55c5a4503270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.187+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.187+0000 7fe913e0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5a4462880 con 0x55c5a45037b0 2023-11-28T20:54:34.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.187+0000 7fe913e0a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5a4471ae0 con 0x55c5a44fed70 2023-11-28T20:54:34.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.187+0000 7fe913e0a740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5a446f1d0 con 0x55c5a4504070 2023-11-28T20:54:34.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33322/0 (socket says 172.21.15.45:33322) 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90b59e640 1 -- 172.21.15.45:0/3921783448 learned_addr learned my addr 172.21.15.45:0/3921783448 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 0x55c5a4504450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe903fff640 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 0x55c5a4503270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90b59e640 1 -- 172.21.15.45:0/3921783448 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 msgr2=0x55c5a4504450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90b59e640 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 0x55c5a4504450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90b59e640 1 -- 172.21.15.45:0/3921783448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 msgr2=0x55c5a4503270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90b59e640 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 0x55c5a4503270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90b59e640 1 -- 172.21.15.45:0/3921783448 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5a4432860 con 0x55c5a45037b0 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe903fff640 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 0x55c5a4503270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:34.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.188+0000 7fe90b59e640 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547730 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fe904009f40 tx=0x7fe90402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c68edc88828356 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe900ff9640 1 -- 172.21.15.45:0/3921783448 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe904040470 con 0x55c5a45037b0 2023-11-28T20:54:34.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe900ff9640 1 -- 172.21.15.45:0/3921783448 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9040405d0 con 0x55c5a45037b0 2023-11-28T20:54:34.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe900ff9640 1 -- 172.21.15.45:0/3921783448 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe904040940 con 0x55c5a45037b0 2023-11-28T20:54:34.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe913e0a740 1 -- 172.21.15.45:0/3921783448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 msgr2=0x55c5a4547730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe913e0a740 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547730 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fe904009f40 tx=0x7fe90402e000 comp rx=0 tx=0).stop 2023-11-28T20:54:34.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe913e0a740 1 -- 172.21.15.45:0/3921783448 shutdown_connections 2023-11-28T20:54:34.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe913e0a740 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547730 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe913e0a740 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 0x55c5a4503270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe913e0a740 1 --2- 172.21.15.45:0/3921783448 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 0x55c5a4504450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe913e0a740 1 -- 172.21.15.45:0/3921783448 >> 172.21.15.45:0/3921783448 conn(0x55c5a44c5c70 msgr2=0x55c5a4549b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.189+0000 7fe913e0a740 1 -- 172.21.15.45:0/3921783448 shutdown_connections 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.190+0000 7fe913e0a740 1 -- 172.21.15.45:0/3921783448 wait complete. 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.190+0000 7fe913e0a740 1 -- start start 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.191+0000 7fe913e0a740 1 Processor -- start 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a44fed70 0x55c5a4551570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a45037b0 0x55c5a4529f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a4504070 0x55c5a4529380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe913e0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5a446aac0 con 0x55c5a4504070 2023-11-28T20:54:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe913e0a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5a44072f0 con 0x55c5a45037b0 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe913e0a740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5a4424c70 con 0x55c5a44fed70 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe913e0a740 10 client.0 _ll_register_callbacks cb 0x55c5a44293b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe90b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a4504070 0x55c5a4529380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe90b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a4504070 0x55c5a4529380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33330/0 (socket says 172.21.15.45:33330) 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe90b59e640 1 -- 172.21.15.45:0/1989696773 learned_addr learned my addr 172.21.15.45:0/1989696773 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a44fed70 0x55c5a4551570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe903fff640 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a45037b0 0x55c5a4529f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe90b59e640 1 -- 172.21.15.45:0/1989696773 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a44fed70 msgr2=0x55c5a4551570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe90b59e640 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a44fed70 0x55c5a4551570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe90b59e640 1 -- 172.21.15.45:0/1989696773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a45037b0 msgr2=0x55c5a4529f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe90b59e640 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a45037b0 0x55c5a4529f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.192+0000 7fe90b59e640 1 -- 172.21.15.45:0/1989696773 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c1616a0 con 0x55c5a4504070 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.193+0000 7fe90b59e640 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a4504070 0x55c5a4529380 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe904043040 tx=0x7fe904004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2daa69af6a4cce server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.193+0000 7fe8e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.193+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/1989696773 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe904040940 con 0x55c5a4504070 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.193+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/1989696773 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe904040aa0 con 0x55c5a4504070 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.193+0000 7fe913e0a740 1 -- 172.21.15.45:0/1989696773 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c5a4463aa0 con 0x55c5a4504070 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.193+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/1989696773 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe904040c60 con 0x55c5a4504070 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.193+0000 7fe913e0a740 10 client.8739 fetch_fsmap learned FSMap version 67 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.193+0000 7fe913e0a740 1 -- 172.21.15.45:0/1989696773 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x55c5a4469140 con 0x55c5a4504070 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.194+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/1989696773 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c5a4463aa0 con 0x55c5a4504070 2023-11-28T20:54:34.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.194+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/1989696773 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 67) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe904047070 con 0x55c5a4504070 2023-11-28T20:54:34.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.194+0000 7fe913e0a740 10 client.8739 fetch_fsmap finished waiting for FSMap version 67 2023-11-28T20:54:34.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.194+0000 7fe913e0a740 10 client.8739 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:54:34.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.194+0000 7fe913e0a740 20 client.8739 populate_metadata read hostname 'smithi045' 2023-11-28T20:54:34.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.194+0000 7fe913e0a740 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] conn(0x55c5a454ef60 0x55c5a45678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.194+0000 7fe913e0a740 4 client.8739 mds_command: new command op to 8595 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:54:34.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.194+0000 7fe913e0a740 1 -- 172.21.15.45:0/1989696773 --> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5a44637c0 con 0x55c5a454ef60 2023-11-28T20:54:34.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.194+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] conn(0x55c5a454ef60 0x55c5a45678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.195+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] conn(0x55c5a454ef60 0x55c5a45678b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a922a1ed9ce211d5 server_cookie=8841dc4316c24bad in_seq=0 out_seq=0 2023-11-28T20:54:34.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.195+0000 7fe8e9ffb640 10 client.8739 ms_handle_connect on v2:172.21.15.45:6834/684266044 2023-11-28T20:54:34.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/1989696773 <== mds.0 v2:172.21.15.45:6834/684266044 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c5a44637c0 con 0x55c5a454ef60 2023-11-28T20:54:34.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe8e9ffb640 10 client.8739 handle_command_reply: tid=0 2023-11-28T20:54:34.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 client.8739 shutdown 2023-11-28T20:54:34.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 -- 172.21.15.45:0/1989696773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a4504070 msgr2=0x55c5a4529380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a4504070 0x55c5a4529380 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe904043040 tx=0x7fe904004ab0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 -- 172.21.15.45:0/1989696773 shutdown_connections 2023-11-28T20:54:34.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] conn(0x55c5a454ef60 0x55c5a45678b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a4504070 0x55c5a4529380 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a45037b0 0x55c5a4529f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 --2- 172.21.15.45:0/1989696773 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a44fed70 0x55c5a4551570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 -- 172.21.15.45:0/1989696773 >> 172.21.15.45:0/1989696773 conn(0x55c5a44c5c70 msgr2=0x55c5a4546d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:34.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe8e9ffb640 0 client.8739 ms_handle_reset on v2:172.21.15.45:6834/684266044 2023-11-28T20:54:34.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 -- 172.21.15.45:0/1989696773 shutdown_connections 2023-11-28T20:54:34.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.196+0000 7fe913e0a740 1 -- 172.21.15.45:0/1989696773 wait complete. 2023-11-28T20:54:34.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.197+0000 7fe913e0a740 20 client.8739 trim_cache size 0 max 16384 2023-11-28T20:54:34.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.214+0000 7fe913e0a740 1 Processor -- start 2023-11-28T20:54:34.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.214+0000 7fe913e0a740 1 -- start start 2023-11-28T20:54:34.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a44fed70 0x55c5a452b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a45037b0 0x55c5a4564140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a4504070 0x55c5a44fa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe913e0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5a41cdf20 con 0x55c5a44fed70 2023-11-28T20:54:34.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe913e0a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5a454f930 con 0x55c5a4504070 2023-11-28T20:54:34.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe913e0a740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5a4530170 con 0x55c5a45037b0 2023-11-28T20:54:34.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe90ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a44fed70 0x55c5a452b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe90ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a44fed70 0x55c5a452b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33340/0 (socket says 172.21.15.45:33340) 2023-11-28T20:54:34.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe90ad9d640 1 -- 172.21.15.45:0/4078078012 learned_addr learned my addr 172.21.15.45:0/4078078012 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:34.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe90ad9d640 1 -- 172.21.15.45:0/4078078012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a45037b0 msgr2=0x55c5a4564140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/4078078012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a45037b0 0x55c5a4564140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe90ad9d640 1 -- 172.21.15.45:0/4078078012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a4504070 msgr2=0x55c5a44fa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/4078078012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a4504070 0x55c5a44fa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.215+0000 7fe90ad9d640 1 -- 172.21.15.45:0/4078078012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5a4406b80 con 0x55c5a44fed70 2023-11-28T20:54:34.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.216+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/4078078012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a44fed70 0x55c5a452b9c0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x55c5a4531f00 tx=0x7fe8fc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba7cdca1cd2e9b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.216+0000 7fe9017fa640 1 -- 172.21.15.45:0/4078078012 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc0148f0 con 0x55c5a44fed70 2023-11-28T20:54:34.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.216+0000 7fe9017fa640 1 -- 172.21.15.45:0/4078078012 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc014a50 con 0x55c5a44fed70 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.216+0000 7fe9017fa640 1 -- 172.21.15.45:0/4078078012 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc014de0 con 0x55c5a44fed70 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 -- 172.21.15.45:0/4078078012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a44fed70 msgr2=0x55c5a452b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 --2- 172.21.15.45:0/4078078012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a44fed70 0x55c5a452b9c0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x55c5a4531f00 tx=0x7fe8fc013000 comp rx=0 tx=0).stop 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 -- 172.21.15.45:0/4078078012 shutdown_connections 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 --2- 172.21.15.45:0/4078078012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a4504070 0x55c5a44fa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 --2- 172.21.15.45:0/4078078012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a45037b0 0x55c5a4564140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 --2- 172.21.15.45:0/4078078012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a44fed70 0x55c5a452b9c0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 -- 172.21.15.45:0/4078078012 >> 172.21.15.45:0/4078078012 conn(0x55c5a44c5c70 msgr2=0x55c5a4504450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 -- 172.21.15.45:0/4078078012 shutdown_connections 2023-11-28T20:54:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 -- 172.21.15.45:0/4078078012 wait complete. 2023-11-28T20:54:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.217+0000 7fe913e0a740 1 -- start start 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.218+0000 7fe913e0a740 1 Processor -- start 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 0x55c5a451c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe913e0a740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 0x55c5a4547d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe913e0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5a454cfd0 con 0x55c5a45037b0 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe913e0a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5a45bce80 con 0x55c5a44fed70 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe913e0a740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5a45bd970 con 0x55c5a4504070 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe913e0a740 10 client.0 _ll_register_callbacks cb 0x7fe904043070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe903fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe903fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33346/0 (socket says 172.21.15.45:33346) 2023-11-28T20:54:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe903fff640 1 -- 172.21.15.45:0/2515389036 learned_addr learned my addr 172.21.15.45:0/2515389036 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe903fff640 1 -- 172.21.15.45:0/2515389036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 msgr2=0x55c5a4547d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe90b59e640 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 0x55c5a4547d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 0x55c5a451c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe903fff640 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 0x55c5a4547d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe903fff640 1 -- 172.21.15.45:0/2515389036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 msgr2=0x55c5a451c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe903fff640 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 0x55c5a451c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.219+0000 7fe903fff640 1 -- 172.21.15.45:0/2515389036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c0d14d0 con 0x55c5a45037b0 2023-11-28T20:54:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.220+0000 7fe903fff640 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547850 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fe8f40581d0 tx=0x7fe8f4057f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196ff90da6d083cb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.220+0000 7fe8e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:54:34.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.220+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/2515389036 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4060540 con 0x55c5a45037b0 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.220+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/2515389036 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe8f40607d0 con 0x55c5a45037b0 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.220+0000 7fe913e0a740 1 -- 172.21.15.45:0/2515389036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9040405d0 con 0x55c5a45037b0 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.220+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/2515389036 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8f4081bb0 con 0x55c5a45037b0 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.220+0000 7fe913e0a740 10 client.8745 fetch_fsmap learned FSMap version 67 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.220+0000 7fe913e0a740 1 -- 172.21.15.45:0/2515389036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x55c5a442c170 con 0x55c5a45037b0 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.221+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/2515389036 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9040405d0 con 0x55c5a45037b0 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.221+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/2515389036 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 67) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe8f4011070 con 0x55c5a45037b0 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.221+0000 7fe913e0a740 10 client.8745 fetch_fsmap finished waiting for FSMap version 67 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.221+0000 7fe913e0a740 10 client.8745 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.221+0000 7fe913e0a740 20 client.8745 populate_metadata read hostname 'smithi045' 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.221+0000 7fe913e0a740 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] conn(0x55c5a45290e0 0x55c5a4549b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.221+0000 7fe913e0a740 4 client.8745 mds_command: new command op to 8595 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.221+0000 7fe913e0a740 1 -- 172.21.15.45:0/2515389036 --> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c5a416b6c0 con 0x55c5a45290e0 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.221+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] conn(0x55c5a45290e0 0x55c5a4549b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe90ad9d640 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] conn(0x55c5a45290e0 0x55c5a4549b60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21d5a8f79d3a240c server_cookie=4d2d811e183b026f in_seq=0 out_seq=0 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe8e9ffb640 10 client.8745 ms_handle_connect on v2:172.21.15.45:6834/684266044 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe8e9ffb640 1 -- 172.21.15.45:0/2515389036 <== mds.0 v2:172.21.15.45:6834/684266044 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55c5a416b6c0 con 0x55c5a45290e0 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe8e9ffb640 10 client.8745 handle_command_reply: tid=0 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 client.8745 shutdown 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 -- 172.21.15.45:0/2515389036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 msgr2=0x55c5a4547850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547850 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fe8f40581d0 tx=0x7fe8f4057f20 comp rx=0 tx=0).stop 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 -- 172.21.15.45:0/2515389036 shutdown_connections 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.45:6834/684266044,v1:172.21.15.45:6835/684266044] conn(0x55c5a45290e0 0x55c5a4549b60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55c5a4504070 0x55c5a4547d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5a45037b0 0x55c5a4547850 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe8e9ffb640 0 client.8745 ms_handle_reset on v2:172.21.15.45:6834/684266044 2023-11-28T20:54:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 --2- 172.21.15.45:0/2515389036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c5a44fed70 0x55c5a451c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 -- 172.21.15.45:0/2515389036 >> 172.21.15.45:0/2515389036 conn(0x55c5a44c5c70 msgr2=0x55c5a44fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 -- 172.21.15.45:0/2515389036 shutdown_connections 2023-11-28T20:54:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.222+0000 7fe913e0a740 1 -- 172.21.15.45:0/2515389036 wait complete. 2023-11-28T20:54:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe913e0a740 20 client.8745 trim_cache size 0 max 16384 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 8595, 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 7.3200221750000001, 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 67, 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 148, 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 148, 2023-11-28T20:54:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 13.668710127000001 2023-11-28T20:54:34.230 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:54:34.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 -- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe8ec06af90 msgr2=0x7fe8ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe8ec06af90 0x7fe8ec06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc001680 tx=0x7fe8fc009040 comp rx=0 tx=0).stop 2023-11-28T20:54:34.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 -- 172.21.15.45:0/3678012938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c15d310 msgr2=0x7fe90c15e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c15d310 0x7fe90c15e250 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400e870 tx=0x7fe8f400ed30 comp rx=0 tx=0).stop 2023-11-28T20:54:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 -- 172.21.15.45:0/3678012938 shutdown_connections 2023-11-28T20:54:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fe8ec06af90 0x7fe8ec06d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe90c160e70 0x7fe90c0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe90c15d310 0x7fe90c15e250 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 --2- 172.21.15.45:0/3678012938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe90c0ce3a0 0x7fe90c14a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 -- 172.21.15.45:0/3678012938 >> 172.21.15.45:0/3678012938 conn(0x7fe90c06a500 msgr2=0x7fe90c0b7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 -- 172.21.15.45:0/3678012938 shutdown_connections 2023-11-28T20:54:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.223+0000 7fe910996640 1 -- 172.21.15.45:0/3678012938 wait complete. 2023-11-28T20:54:34.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:54:34.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.448+0000 7fc629e1e640 1 Processor -- start 2023-11-28T20:54:34.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.448+0000 7fc629e1e640 1 -- start start 2023-11-28T20:54:34.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.450+0000 7fc629e1e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62415b850 0x7fc6240b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.450+0000 7fc629e1e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc6240b1fa0 0x7fc6240b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.450+0000 7fc629e1e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62415bda0 con 0x7fc6240b35b0 2023-11-28T20:54:34.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.450+0000 7fc629e1e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc62415ee40 con 0x7fc6240b1fa0 2023-11-28T20:54:34.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.450+0000 7fc629e1e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc62415c080 con 0x7fc62415b850 2023-11-28T20:54:34.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.450+0000 7fc6237fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc6240b35b0 0x7fc62415ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44648/0 (socket says 172.21.15.45:44648) 2023-11-28T20:54:34.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.451+0000 7fc6237fe640 1 -- 172.21.15.45:0/1869648709 learned_addr learned my addr 172.21.15.45:0/1869648709 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:34.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.451+0000 7fc623fff640 1 --2- 172.21.15.45:0/1869648709 >> v2:172.21.15.190:3300/0 conn(0x7fc6240b1fa0 0x7fc6240b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.451+0000 7fc622ffd640 1 --2- 172.21.15.45:0/1869648709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62415b850 0x7fc6240b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.451+0000 7fc6227fc640 1 -- 172.21.15.45:0/1869648709 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724131003 0 0) 0x7fc62415bda0 con 0x7fc6240b35b0 2023-11-28T20:54:34.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.451+0000 7fc6227fc640 1 -- 172.21.15.45:0/1869648709 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc60c003520 con 0x7fc6240b35b0 2023-11-28T20:54:34.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.452+0000 7fc623fff640 1 -- 172.21.15.45:0/1869648709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62415b850 msgr2=0x7fc6240b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:34.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.452+0000 7fc623fff640 1 --2- 172.21.15.45:0/1869648709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62415b850 0x7fc6240b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.452+0000 7fc623fff640 1 -- 172.21.15.45:0/1869648709 >> v1:172.21.15.45:6789/0 conn(0x7fc6240b35b0 legacy=0x7fc62415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.452+0000 7fc623fff640 1 -- 172.21.15.45:0/1869648709 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62415c360 con 0x7fc6240b1fa0 2023-11-28T20:54:34.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.453+0000 7fc623fff640 1 --2- 172.21.15.45:0/1869648709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6240b1fa0 0x7fc6240b1170 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc61c009dc0 tx=0x7fc61c02be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=830ab189d9f30034 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.454+0000 7fc6227fc640 1 -- 172.21.15.45:0/1869648709 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61c02cc30 con 0x7fc6240b1fa0 2023-11-28T20:54:34.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.454+0000 7fc6227fc640 1 -- 172.21.15.45:0/1869648709 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61c02cd90 con 0x7fc6240b1fa0 2023-11-28T20:54:34.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.454+0000 7fc6227fc640 1 -- 172.21.15.45:0/1869648709 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61c034660 con 0x7fc6240b1fa0 2023-11-28T20:54:34.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.454+0000 7fc629e1e640 1 -- 172.21.15.45:0/1869648709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6240b1fa0 msgr2=0x7fc6240b1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.454+0000 7fc629e1e640 1 --2- 172.21.15.45:0/1869648709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6240b1fa0 0x7fc6240b1170 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc61c009dc0 tx=0x7fc61c02be80 comp rx=0 tx=0).stop 2023-11-28T20:54:34.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.454+0000 7fc629e1e640 1 -- 172.21.15.45:0/1869648709 shutdown_connections 2023-11-28T20:54:34.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.454+0000 7fc629e1e640 1 --2- 172.21.15.45:0/1869648709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6240b1fa0 0x7fc6240b1170 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.454+0000 7fc629e1e640 1 --2- 172.21.15.45:0/1869648709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62415b850 0x7fc6240b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.454+0000 7fc629e1e640 1 -- 172.21.15.45:0/1869648709 >> 172.21.15.45:0/1869648709 conn(0x7fc62406a500 msgr2=0x7fc62406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:34.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.455+0000 7fc629e1e640 1 -- 172.21.15.45:0/1869648709 shutdown_connections 2023-11-28T20:54:34.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.455+0000 7fc629e1e640 1 -- 172.21.15.45:0/1869648709 wait complete. 2023-11-28T20:54:34.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.456+0000 7fc629e1e640 1 Processor -- start 2023-11-28T20:54:34.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.457+0000 7fc629e1e640 1 -- start start 2023-11-28T20:54:34.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.457+0000 7fc629e1e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6240b1fa0 0x7fc624154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc629e1e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6240b35b0 0x7fc6241f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc629e1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc62415b850 0x7fc624071900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc629e1e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6241f7020 con 0x7fc62415b850 2023-11-28T20:54:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc629e1e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6241f7300 con 0x7fc6240b35b0 2023-11-28T20:54:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc629e1e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc624135540 con 0x7fc6240b1fa0 2023-11-28T20:54:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc623fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc62415b850 0x7fc624071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc623fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc62415b850 0x7fc624071900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33350/0 (socket says 172.21.15.45:33350) 2023-11-28T20:54:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc623fff640 1 -- 172.21.15.45:0/1763928227 learned_addr learned my addr 172.21.15.45:0/1763928227 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc622ffd640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6240b35b0 0x7fc6241f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.458+0000 7fc6237fe640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6240b1fa0 0x7fc624154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.459+0000 7fc623fff640 1 -- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6240b1fa0 msgr2=0x7fc624154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.459+0000 7fc623fff640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6240b1fa0 0x7fc624154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.459+0000 7fc623fff640 1 -- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6240b35b0 msgr2=0x7fc6241f65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.459+0000 7fc623fff640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6240b35b0 0x7fc6241f65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.459+0000 7fc623fff640 1 -- 172.21.15.45:0/1763928227 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c009a70 con 0x7fc62415b850 2023-11-28T20:54:34.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.459+0000 7fc623fff640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc62415b850 0x7fc624071900 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fc61c037040 tx=0x7fc61c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.460+0000 7fc60bfff640 1 -- 172.21.15.45:0/1763928227 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61c040650 con 0x7fc62415b850 2023-11-28T20:54:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.460+0000 7fc60bfff640 1 -- 172.21.15.45:0/1763928227 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61c0407b0 con 0x7fc62415b850 2023-11-28T20:54:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.460+0000 7fc60bfff640 1 -- 172.21.15.45:0/1763928227 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61c040b60 con 0x7fc62415b850 2023-11-28T20:54:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.460+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc624135820 con 0x7fc62415b850 2023-11-28T20:54:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.460+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6240c40c0 con 0x7fc62415b850 2023-11-28T20:54:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.461+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6241f7520 con 0x7fc62415b850 2023-11-28T20:54:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.462+0000 7fc60bfff640 1 -- 172.21.15.45:0/1763928227 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc61c040cc0 con 0x7fc62415b850 2023-11-28T20:54:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.463+0000 7fc60bfff640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc61006afb0 0x7fc61006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.463+0000 7fc60bfff640 1 -- 172.21.15.45:0/1763928227 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc61c0ac350 con 0x7fc62415b850 2023-11-28T20:54:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.464+0000 7fc6237fe640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc61006afb0 0x7fc61006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:34.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.465+0000 7fc6237fe640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc61006afb0 0x7fc61006d470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc6140011e0 tx=0x7fc614009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:34.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.468+0000 7fc60bfff640 1 -- 172.21.15.45:0/1763928227 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc61c077420 con 0x7fc62415b850 2023-11-28T20:54:34.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.700+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc6240733d0 con 0x7fc62415b850 2023-11-28T20:54:34.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.701+0000 7fc60bfff640 1 -- 172.21.15.45:0/1763928227 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v67) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc61c009450 con 0x7fc62415b850 2023-11-28T20:54:34.704 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T20:54:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.703+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc61006afb0 msgr2=0x7fc61006d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.703+0000 7fc629e1e640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc61006afb0 0x7fc61006d470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc6140011e0 tx=0x7fc614009040 comp rx=0 tx=0).stop 2023-11-28T20:54:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.703+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc62415b850 msgr2=0x7fc624071900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.703+0000 7fc629e1e640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc62415b850 0x7fc624071900 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fc61c037040 tx=0x7fc61c004a40 comp rx=0 tx=0).stop 2023-11-28T20:54:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.704+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 shutdown_connections 2023-11-28T20:54:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.704+0000 7fc629e1e640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fc61006afb0 0x7fc61006d470 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.704+0000 7fc629e1e640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc62415b850 0x7fc624071900 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.704+0000 7fc629e1e640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6240b35b0 0x7fc6241f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.704+0000 7fc629e1e640 1 --2- 172.21.15.45:0/1763928227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6240b1fa0 0x7fc624154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.704+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 >> 172.21.15.45:0/1763928227 conn(0x7fc62406a500 msgr2=0x7fc624144890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:34.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.704+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 shutdown_connections 2023-11-28T20:54:34.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:34.704+0000 7fc629e1e640 1 -- 172.21.15.45:0/1763928227 wait complete. 2023-11-28T20:54:34.726 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T20:54:34.726 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:54:34.727 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:54:34.727 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T20:54:34.727 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:54:34.727 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T20:54:34.728 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T20:54:34.728 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:54:34.728 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:54:34.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:34.729 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:54:34.786 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:54:34.786 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:54:34.786 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:54:34.787 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:54:34.787 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:54:34.787 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:54:34.787 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:54:34.787 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:54:34.787 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:54:34.787 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:54:34.788 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:54:34.788 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:54:34.788 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:54:34.788 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:54:34.788 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:54:34.788 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:54:34.788 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 41068sec preferred_lft 41068sec 2023-11-28T20:54:34.788 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:54:34.789 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:54:34.789 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:54:34.789 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:54:34.789 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:54:34.789 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:54:34.789 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T20:54:34.789 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:54:34.789 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T20:54:34.790 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:54:34.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:34.792 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:54:34.888 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:54:34.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:34.894 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:54:34.952 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:54:34.953 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:54:34.953 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:54:34.953 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:54:34.953 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:54:34.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:34.953 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:54:34.953 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:54:34.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:54:34.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:54:34.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:54:34.954 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:54:35.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:35.146 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:54:35.203 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T20:54:35.204 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T20:54:35.204 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:54:35.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:35.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:54:35.307 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:54:35.307 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T20:54:35.368 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:54:35.369 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:54:35.369 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:54:35.424 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:54:35.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:35.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:54:35.570 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:54:35.570 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:54:35.570 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:54:35.593 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:54:35.593 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:54:35.593 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:54:35.593 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:54:35.593 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:54:35.624 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:54:35.624 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:54:35.624 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:54:35.624 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:54:35.624 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:54:35.624 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:54:35.624 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:54:35.624 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:54:35.627 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T20:54:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:54:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:54:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:54:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:54:35.631 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:54:35.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:35.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:54:35.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:35.723 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:54:35.793 INFO:teuthology.orchestra.run.smithi045.stdout:client6611 2023-11-28T20:54:35.794 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:54:35.794 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:35.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T20:54:36.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.075+0000 7f243359e640 1 Processor -- start 2023-11-28T20:54:36.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.075+0000 7f243359e640 1 -- start start 2023-11-28T20:54:36.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.076+0000 7f243359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f243415d210 0x7f24340b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.076+0000 7f243359e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24340d11b0 con 0x7f24340cddd0 2023-11-28T20:54:36.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.076+0000 7f243359e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24340d1470 con 0x7f24340b2620 2023-11-28T20:54:36.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.076+0000 7f243359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24340b90c0 con 0x7f243415d210 2023-11-28T20:54:36.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.077+0000 7f2432d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f24340cddd0 0x7f24340d2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44656/0 (socket says 172.21.15.45:44656) 2023-11-28T20:54:36.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.077+0000 7f2432d9d640 1 -- 172.21.15.45:0/3976918701 learned_addr learned my addr 172.21.15.45:0/3976918701 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:36.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.077+0000 7f243259c640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f243415d210 0x7f24340b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.077+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029374495 0 0) 0x7f24340d11b0 con 0x7f24340cddd0 2023-11-28T20:54:36.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.077+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2420003520 con 0x7f24340cddd0 2023-11-28T20:54:36.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4094300897 0 0) 0x7f2420003520 con 0x7f24340cddd0 2023-11-28T20:54:36.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f243415d210 msgr2=0x7f24340b61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:36.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f243415d210 0x7f24340b61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 >> v1:172.21.15.190:6789/0 conn(0x7f24340b2620 legacy=0x7f24340b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24340d1750 con 0x7f24340cddd0 2023-11-28T20:54:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f242c001670 con 0x7f24340cddd0 2023-11-28T20:54:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 >> v1:172.21.15.45:6789/0 conn(0x7f24340cddd0 legacy=0x7f24340d2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24200039a0 0x7f2420003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24200047c0 0x7f2420004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f243415d210 0x7f24200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340b90c0 con 0x7f24200039a0 2023-11-28T20:54:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242000bb40 con 0x7f24200047c0 2023-11-28T20:54:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242000bd00 con 0x7f243415d210 2023-11-28T20:54:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.078+0000 7f243259c640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24200039a0 0x7f2420003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.079+0000 7f243259c640 1 -- 172.21.15.45:0/3976918701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f243415d210 msgr2=0x7f24200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:36.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.079+0000 7f243259c640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f243415d210 0x7f24200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.079+0000 7f243259c640 1 -- 172.21.15.45:0/3976918701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24200047c0 msgr2=0x7f2420004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:36.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.079+0000 7f243259c640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24200047c0 0x7f2420004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.079+0000 7f243259c640 1 -- 172.21.15.45:0/3976918701 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2420003520 con 0x7f24200039a0 2023-11-28T20:54:36.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.080+0000 7f243259c640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24200039a0 0x7f2420003dc0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f2424007360 tx=0x7f24200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9c7101f30d61a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:36.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.080+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242402ccf0 con 0x7f24200039a0 2023-11-28T20:54:36.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.080+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242402ce50 con 0x7f24200039a0 2023-11-28T20:54:36.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.080+0000 7f243159a640 1 -- 172.21.15.45:0/3976918701 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424034660 con 0x7f24200039a0 2023-11-28T20:54:36.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.081+0000 7f243359e640 1 -- 172.21.15.45:0/3976918701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24200039a0 msgr2=0x7f2420003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.081+0000 7f243359e640 1 --2- 172.21.15.45:0/3976918701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24200039a0 0x7f2420003dc0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f2424007360 tx=0x7f24200057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.081+0000 7f2432d9d640 1 -- 172.21.15.45:0/3976918701 reap_dead start 2023-11-28T20:54:36.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.081+0000 7f243359e640 1 -- 172.21.15.45:0/3976918701 shutdown_connections 2023-11-28T20:54:36.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.081+0000 7f243359e640 1 -- 172.21.15.45:0/3976918701 >> 172.21.15.45:0/3976918701 conn(0x7f2434070f40 msgr2=0x7f2434071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:36.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.081+0000 7f243359e640 1 -- 172.21.15.45:0/3976918701 shutdown_connections 2023-11-28T20:54:36.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.081+0000 7f243359e640 1 -- 172.21.15.45:0/3976918701 wait complete. 2023-11-28T20:54:36.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.083+0000 7f243359e640 1 Processor -- start 2023-11-28T20:54:36.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.083+0000 7f243359e640 1 -- start start 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.084+0000 7f243359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24200039a0 0x7f24340c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.084+0000 7f243359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2434154050 0x7f24340c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.084+0000 7f243359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24341538e0 0x7f243415f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.084+0000 7f243359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c86e0 con 0x7f24341538e0 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.084+0000 7f243359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c7470 con 0x7f24200039a0 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.084+0000 7f243359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c7d10 con 0x7f2434154050 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.084+0000 7f2432d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24341538e0 0x7f243415f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.084+0000 7f2432d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24341538e0 0x7f243415f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33366/0 (socket says 172.21.15.45:33366) 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.084+0000 7f2432d9d640 1 -- 172.21.15.45:0/2412312967 learned_addr learned my addr 172.21.15.45:0/2412312967 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f2432d9d640 1 -- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2434154050 msgr2=0x7f24340c81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f2432d9d640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2434154050 0x7f24340c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f2432d9d640 1 -- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24200039a0 msgr2=0x7f24340c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f2432d9d640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24200039a0 0x7f24340c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f2432d9d640 1 -- 172.21.15.45:0/2412312967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24340c98a0 con 0x7f24341538e0 2023-11-28T20:54:36.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f2432d9d640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24341538e0 0x7f243415f430 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f24340b5be0 tx=0x7f242c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f242affd640 1 -- 172.21.15.45:0/2412312967 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c007230 con 0x7f24341538e0 2023-11-28T20:54:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f242affd640 1 -- 172.21.15.45:0/2412312967 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c00ced0 con 0x7f24341538e0 2023-11-28T20:54:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f242affd640 1 -- 172.21.15.45:0/2412312967 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c0075d0 con 0x7f24341538e0 2023-11-28T20:54:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.085+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24340c76c0 con 0x7f24341538e0 2023-11-28T20:54:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.086+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2434154830 con 0x7f24341538e0 2023-11-28T20:54:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.087+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2434071b30 con 0x7f24341538e0 2023-11-28T20:54:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.088+0000 7f242affd640 1 -- 172.21.15.45:0/2412312967 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f242c0047e0 con 0x7f24341538e0 2023-11-28T20:54:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.089+0000 7f242affd640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f241406af60 0x7f241406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.089+0000 7f242affd640 1 -- 172.21.15.45:0/2412312967 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f242c088e40 con 0x7f24341538e0 2023-11-28T20:54:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.090+0000 7f243259c640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f241406af60 0x7f241406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.091+0000 7f243259c640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f241406af60 0x7f241406d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2424036000 tx=0x7f242402e000 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:36.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.093+0000 7f242affd640 1 -- 172.21.15.45:0/2412312967 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f242c053d40 con 0x7f24341538e0 2023-11-28T20:54:36.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.309+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 --> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f24340d1d10 con 0x7f241406af60 2023-11-28T20:54:36.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.313+0000 7f242affd640 1 -- 172.21.15.45:0/2412312967 <== mgr.7015 v2:172.21.15.190:6832/2586319130 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f242402ccc0 con 0x7f241406af60 2023-11-28T20:54:36.315 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:54:36.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:54:36.315 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T20:54:36.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:54:36.315 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:54:36.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.315+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f241406af60 msgr2=0x7f241406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.315+0000 7f243359e640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f241406af60 0x7f241406d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2424036000 tx=0x7f242402e000 comp rx=0 tx=0).stop 2023-11-28T20:54:36.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.315+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24341538e0 msgr2=0x7f243415f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.315+0000 7f243359e640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24341538e0 0x7f243415f430 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f24340b5be0 tx=0x7f242c00cb90 comp rx=0 tx=0).stop 2023-11-28T20:54:36.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.316+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 shutdown_connections 2023-11-28T20:54:36.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.316+0000 7f243359e640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f241406af60 0x7f241406d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.316+0000 7f243359e640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24341538e0 0x7f243415f430 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.316+0000 7f243359e640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2434154050 0x7f24340c81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.316+0000 7f243359e640 1 --2- 172.21.15.45:0/2412312967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24200039a0 0x7f24340c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.316+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 >> 172.21.15.45:0/2412312967 conn(0x7f2434070f40 msgr2=0x7f24340b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:36.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.316+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 shutdown_connections 2023-11-28T20:54:36.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.316+0000 7f243359e640 1 -- 172.21.15.45:0/2412312967 wait complete. 2023-11-28T20:54:36.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T20:54:36.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.572+0000 7f5c77fff640 1 Processor -- start 2023-11-28T20:54:36.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.572+0000 7f5c77fff640 1 -- start start 2023-11-28T20:54:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.573+0000 7f5c77fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5c78071fd0 0x7f5c780ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.574+0000 7f5c77fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c780abd70 0x7f5c780702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.574+0000 7f5c77fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c780ac270 con 0x7f5c7815b080 2023-11-28T20:54:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.574+0000 7f5c77fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7815b830 con 0x7f5c78071fd0 2023-11-28T20:54:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.574+0000 7f5c77fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7815bb10 con 0x7f5c780abd70 2023-11-28T20:54:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.574+0000 7f5c76ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5c7815b080 0x7f5c780718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44662/0 (socket says 172.21.15.45:44662) 2023-11-28T20:54:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.574+0000 7f5c76ffd640 1 -- 172.21.15.45:0/4187713797 learned_addr learned my addr 172.21.15.45:0/4187713797 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:36.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.574+0000 7f5c777fe640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c780abd70 0x7f5c780702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.574+0000 7f5c767fc640 1 --2- 172.21.15.45:0/4187713797 >> v2:172.21.15.190:3300/0 conn(0x7f5c78071fd0 0x7f5c780ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306442224 0 0) 0x7f5c780ac270 con 0x7f5c7815b080 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c64003520 con 0x7f5c7815b080 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 303728648 0 0) 0x7f5c64003520 con 0x7f5c7815b080 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c780abd70 msgr2=0x7f5c780702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c780abd70 0x7f5c780702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 >> v2:172.21.15.190:3300/0 conn(0x7f5c78071fd0 msgr2=0x7f5c780ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 --2- 172.21.15.45:0/4187713797 >> v2:172.21.15.190:3300/0 conn(0x7f5c78071fd0 0x7f5c780ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7815bdf0 con 0x7f5c7815b080 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5c68001a10 con 0x7f5c7815b080 2023-11-28T20:54:36.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 >> v1:172.21.15.45:6789/0 conn(0x7f5c7815b080 legacy=0x7f5c780718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c64003970 0x7f5c64003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c640047c0 0x7f5c64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c780abd70 0x7f5c64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7815b830 con 0x7f5c640047c0 2023-11-28T20:54:36.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7815bb10 con 0x7f5c64003970 2023-11-28T20:54:36.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.575+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6400b980 con 0x7f5c780abd70 2023-11-28T20:54:36.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.576+0000 7f5c767fc640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c640047c0 0x7f5c64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.576+0000 7f5c777fe640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c780abd70 0x7f5c64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.576+0000 7f5c76ffd640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c64003970 0x7f5c64003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.576+0000 7f5c767fc640 1 -- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c780abd70 msgr2=0x7f5c64005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.576+0000 7f5c767fc640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c780abd70 0x7f5c64005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.576+0000 7f5c767fc640 1 -- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c64003970 msgr2=0x7f5c64003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.576+0000 7f5c767fc640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c64003970 0x7f5c64003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.576+0000 7f5c767fc640 1 -- 172.21.15.45:0/4187713797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7815bdf0 con 0x7f5c640047c0 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.576+0000 7f5c76ffd640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c64003970 0x7f5c64003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.577+0000 7f5c767fc640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c640047c0 0x7f5c64004c40 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009ac0 tx=0x7f5c640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc57ddfee8027718 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.578+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6002c710 con 0x7f5c640047c0 2023-11-28T20:54:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.578+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6002c870 con 0x7f5c640047c0 2023-11-28T20:54:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.578+0000 7f5c75ffb640 1 -- 172.21.15.45:0/4187713797 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6002cc00 con 0x7f5c640047c0 2023-11-28T20:54:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.578+0000 7f5c77fff640 1 -- 172.21.15.45:0/4187713797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c640047c0 msgr2=0x7f5c64004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.578+0000 7f5c77fff640 1 --2- 172.21.15.45:0/4187713797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c640047c0 0x7f5c64004c40 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009ac0 tx=0x7f5c640057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.578+0000 7f5c777fe640 1 -- 172.21.15.45:0/4187713797 reap_dead start 2023-11-28T20:54:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.579+0000 7f5c77fff640 1 -- 172.21.15.45:0/4187713797 shutdown_connections 2023-11-28T20:54:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.579+0000 7f5c77fff640 1 -- 172.21.15.45:0/4187713797 >> 172.21.15.45:0/4187713797 conn(0x7f5c780cecd0 msgr2=0x7f5c780cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.579+0000 7f5c77fff640 1 -- 172.21.15.45:0/4187713797 shutdown_connections 2023-11-28T20:54:36.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.579+0000 7f5c77fff640 1 -- 172.21.15.45:0/4187713797 wait complete. 2023-11-28T20:54:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.581+0000 7f5c77fff640 1 Processor -- start 2023-11-28T20:54:36.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.581+0000 7f5c77fff640 1 -- start start 2023-11-28T20:54:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.582+0000 7f5c77fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c640047c0 0x7f5c7814e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c77fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c7814ea80 0x7f5c78134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c77fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c78134aa0 0x7f5c78134f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c77fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7815ac40 con 0x7f5c78134aa0 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c77fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7814ef00 con 0x7f5c640047c0 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c77fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781327b0 con 0x7f5c7814ea80 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c78134aa0 0x7f5c78134f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c78134aa0 0x7f5c78134f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33378/0 (socket says 172.21.15.45:33378) 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c777fe640 1 -- 172.21.15.45:0/1188704611 learned_addr learned my addr 172.21.15.45:0/1188704611 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c76ffd640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c640047c0 0x7f5c7814e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c767fc640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c7814ea80 0x7f5c78134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c777fe640 1 -- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c7814ea80 msgr2=0x7f5c78134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c777fe640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c7814ea80 0x7f5c78134560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c777fe640 1 -- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c640047c0 msgr2=0x7f5c7814e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c777fe640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c640047c0 0x7f5c7814e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.583+0000 7f5c777fe640 1 -- 172.21.15.45:0/1188704611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78133ef0 con 0x7f5c78134aa0 2023-11-28T20:54:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.584+0000 7f5c777fe640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c78134aa0 0x7f5c78134f20 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f5c780ac1a0 tx=0x7f5c70010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.584+0000 7f5c6f7fe640 1 -- 172.21.15.45:0/1188704611 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7000e870 con 0x7f5c78134aa0 2023-11-28T20:54:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.584+0000 7f5c6f7fe640 1 -- 172.21.15.45:0/1188704611 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7000e9f0 con 0x7f5c78134aa0 2023-11-28T20:54:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.584+0000 7f5c6f7fe640 1 -- 172.21.15.45:0/1188704611 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7000eda0 con 0x7f5c78134aa0 2023-11-28T20:54:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.584+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7815a680 con 0x7f5c78134aa0 2023-11-28T20:54:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.584+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c78132440 con 0x7f5c78134aa0 2023-11-28T20:54:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.585+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c781322a0 con 0x7f5c78134aa0 2023-11-28T20:54:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.587+0000 7f5c6f7fe640 1 -- 172.21.15.45:0/1188704611 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 68) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5c70020410 con 0x7f5c78134aa0 2023-11-28T20:54:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.588+0000 7f5c6f7fe640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f5c5806af90 0x7f5c5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.588+0000 7f5c6f7fe640 1 -- 172.21.15.45:0/1188704611 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5c70089050 con 0x7f5c78134aa0 2023-11-28T20:54:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.588+0000 7f5c76ffd640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f5c5806af90 0x7f5c5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:36.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.589+0000 7f5c76ffd640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f5c5806af90 0x7f5c5806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5c68001b80 tx=0x7f5c68009040 comp rx=0 tx=0).ready entity=mgr.7015 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:36.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.592+0000 7f5c6f7fe640 1 -- 172.21.15.45:0/1188704611 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c781322a0 con 0x7f5c78134aa0 2023-11-28T20:54:36.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:36.838+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5c781322a0 con 0x7f5c78134aa0 2023-11-28T20:54:37.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.468+0000 7f5c6f7fe640 1 -- 172.21.15.45:0/1188704611 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v69) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5c781322a0 con 0x7f5c78134aa0 2023-11-28T20:54:37.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f5c5806af90 msgr2=0x7f5c5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:37.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f5c5806af90 0x7f5c5806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5c68001b80 tx=0x7f5c68009040 comp rx=0 tx=0).stop 2023-11-28T20:54:37.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c78134aa0 msgr2=0x7f5c78134f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:37.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c78134aa0 0x7f5c78134f20 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f5c780ac1a0 tx=0x7f5c70010c20 comp rx=0 tx=0).stop 2023-11-28T20:54:37.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 shutdown_connections 2023-11-28T20:54:37.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f5c5806af90 0x7f5c5806d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c78134aa0 0x7f5c78134f20 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c7814ea80 0x7f5c78134560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 --2- 172.21.15.45:0/1188704611 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c640047c0 0x7f5c7814e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.471+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 >> 172.21.15.45:0/1188704611 conn(0x7f5c780cecd0 msgr2=0x7f5c780aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:37.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.472+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 shutdown_connections 2023-11-28T20:54:37.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.473+0000 7f5c77fff640 1 -- 172.21.15.45:0/1188704611 wait complete. 2023-11-28T20:54:37.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T20:54:37.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.724+0000 7fa5ae0c3640 1 Processor -- start 2023-11-28T20:54:37.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.724+0000 7fa5ae0c3640 1 -- start start 2023-11-28T20:54:37.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.726+0000 7fa5ae0c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80b1d40 0x7fa5a80d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:37.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.726+0000 7fa5ae0c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a80b90a0 con 0x7fa5a80d0040 2023-11-28T20:54:37.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.726+0000 7fa5ae0c3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a80d1920 con 0x7fa5a80b2290 2023-11-28T20:54:37.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.726+0000 7fa5ae0c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a80b8ae0 con 0x7fa5a80b1d40 2023-11-28T20:54:37.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.726+0000 7fa5ad0c1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa5a80d0040 0x7fa5a80d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44680/0 (socket says 172.21.15.45:44680) 2023-11-28T20:54:37.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.726+0000 7fa5ad0c1640 1 -- 172.21.15.45:0/1614464685 learned_addr learned my addr 172.21.15.45:0/1614464685 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:37.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa5ad8c2640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80b1d40 0x7fa5a80d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:37.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515109705 0 0) 0x7fa5a80b90a0 con 0x7fa5a80d0040 2023-11-28T20:54:37.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa594003520 con 0x7fa5a80d0040 2023-11-28T20:54:37.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4257588595 0 0) 0x7fa594003520 con 0x7fa5a80d0040 2023-11-28T20:54:37.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80b1d40 msgr2=0x7fa5a80d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:37.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80b1d40 0x7fa5a80d24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 >> v1:172.21.15.190:6789/0 conn(0x7fa5a80b2290 legacy=0x7fa5a80d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:37.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a80b8dc0 con 0x7fa5a80d0040 2023-11-28T20:54:37.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fa5a4001670 con 0x7fa5a80d0040 2023-11-28T20:54:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 >> v1:172.21.15.45:6789/0 conn(0x7fa5a80d0040 legacy=0x7fa5a80d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80b1d40 0x7fa5940039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594003ee0 0x7fa5940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa594004d00 0x7fa594005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a80b8ae0 con 0x7fa594004d00 2023-11-28T20:54:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59400b9e0 con 0x7fa594003ee0 2023-11-28T20:54:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.727+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59400bba0 con 0x7fa5a80b1d40 2023-11-28T20:54:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.728+0000 7fa5ad8c2640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa594004d00 0x7fa594005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:37.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.728+0000 7fa5ac8c0640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594003ee0 0x7fa5940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:37.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.728+0000 7fa5ad0c1640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80b1d40 0x7fa5940039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:37.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.728+0000 7fa5ad8c2640 1 -- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80b1d40 msgr2=0x7fa5940039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:37.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.728+0000 7fa5ad8c2640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80b1d40 0x7fa5940039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.728+0000 7fa5ad8c2640 1 -- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594003ee0 msgr2=0x7fa5940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:37.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.728+0000 7fa5ad8c2640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594003ee0 0x7fa5940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.728+0000 7fa5ad8c2640 1 -- 172.21.15.45:0/1614464685 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa594003520 con 0x7fa594004d00 2023-11-28T20:54:37.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.728+0000 7fa5ac8c0640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594003ee0 0x7fa5940047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:37.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.729+0000 7fa5ad8c2640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa594004d00 0x7fa594005180 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009fc0 tx=0x7fa5940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f7fa2848b4d8ac server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:37.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a002cc20 con 0x7fa594004d00 2023-11-28T20:54:37.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a000f040 con 0x7fa594004d00 2023-11-28T20:54:37.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa59ffff640 1 -- 172.21.15.45:0/1614464685 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0034440 con 0x7fa594004d00 2023-11-28T20:54:37.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1614464685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa594004d00 msgr2=0x7fa594005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:37.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa5ae0c3640 1 --2- 172.21.15.45:0/1614464685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa594004d00 0x7fa594005180 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009fc0 tx=0x7fa5940057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa5ad8c2640 1 -- 172.21.15.45:0/1614464685 reap_dead start 2023-11-28T20:54:37.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1614464685 shutdown_connections 2023-11-28T20:54:37.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1614464685 >> 172.21.15.45:0/1614464685 conn(0x7fa5a8070f40 msgr2=0x7fa5a8071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:37.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1614464685 shutdown_connections 2023-11-28T20:54:37.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.730+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1614464685 wait complete. 2023-11-28T20:54:37.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.732+0000 7fa5ae0c3640 1 Processor -- start 2023-11-28T20:54:37.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.732+0000 7fa5ae0c3640 1 -- start start 2023-11-28T20:54:37.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.733+0000 7fa5ae0c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594004d00 0x7fa5a81f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:37.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.733+0000 7fa5ae0c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5a80ab7a0 0x7fa5a815aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:37.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.733+0000 7fa5ae0c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80c8fc0 0x7fa5a80c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:37.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.733+0000 7fa5ae0c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a80c99b0 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.733+0000 7fa5ae0c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a80ac310 con 0x7fa594004d00 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.733+0000 7fa5ae0c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a80b1b20 con 0x7fa5a80c8fc0 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ac8c0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5a80ab7a0 0x7fa5a815aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ac8c0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5a80ab7a0 0x7fa5a815aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33414/0 (socket says 172.21.15.45:33414) 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ac8c0640 1 -- 172.21.15.45:0/1559765706 learned_addr learned my addr 172.21.15.45:0/1559765706 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ad0c1640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594004d00 0x7fa5a81f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ad8c2640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80c8fc0 0x7fa5a80c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ac8c0640 1 -- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80c8fc0 msgr2=0x7fa5a80c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ac8c0640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80c8fc0 0x7fa5a80c9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ac8c0640 1 -- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594004d00 msgr2=0x7fa5a81f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:37.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ac8c0640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594004d00 0x7fa5a81f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ac8c0640 1 -- 172.21.15.45:0/1559765706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a81f39d0 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.734+0000 7fa5ac8c0640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5a80ab7a0 0x7fa5a815aca0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fa59800e7d0 tx=0x7fa59800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:37.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.735+0000 7fa59dffb640 1 -- 172.21.15.45:0/1559765706 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59800ee80 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.735+0000 7fa59dffb640 1 -- 172.21.15.45:0/1559765706 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5980052e0 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.735+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a80abd50 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.735+0000 7fa59dffb640 1 -- 172.21.15.45:0/1559765706 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59800c8c0 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.735+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a80c6870 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.736+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a8071b30 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.737+0000 7fa59dffb640 1 -- 172.21.15.45:0/1559765706 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fa59800ca20 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.738+0000 7fa59dffb640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa58c06afb0 0x7fa58c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:37.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.738+0000 7fa59dffb640 1 -- 172.21.15.45:0/1559765706 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa598089880 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.738+0000 7fa5ad0c1640 1 -- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa58c06afb0 msgr2=0x7fa58c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2586319130 2023-11-28T20:54:37.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.738+0000 7fa5ad0c1640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa58c06afb0 0x7fa58c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:37.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.746+0000 7fa59dffb640 1 -- 172.21.15.45:0/1559765706 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5980546b0 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.939+0000 7fa5ad0c1640 1 -- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa58c06afb0 msgr2=0x7fa58c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2586319130 2023-11-28T20:54:37.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.939+0000 7fa5ad0c1640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa58c06afb0 0x7fa58c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:37.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.940+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fa5a804e920 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.946+0000 7fa59dffb640 1 -- 172.21.15.45:0/1559765706 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa59801a310 con 0x7fa5a80ab7a0 2023-11-28T20:54:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.948+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa58c06afb0 msgr2=0x7fa58c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.948+0000 7fa5ae0c3640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa58c06afb0 0x7fa58c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.948+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5a80ab7a0 msgr2=0x7fa5a815aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.948+0000 7fa5ae0c3640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5a80ab7a0 0x7fa5a815aca0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fa59800e7d0 tx=0x7fa59800ec90 comp rx=0 tx=0).stop 2023-11-28T20:54:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.948+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 shutdown_connections 2023-11-28T20:54:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.949+0000 7fa5ae0c3640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa58c06afb0 0x7fa58c06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.949+0000 7fa5ae0c3640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a80c8fc0 0x7fa5a80c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.949+0000 7fa5ae0c3640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5a80ab7a0 0x7fa5a815aca0 secure :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fa59800e7d0 tx=0x7fa59800ec90 comp rx=0 tx=0).stop 2023-11-28T20:54:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.949+0000 7fa5ae0c3640 1 --2- 172.21.15.45:0/1559765706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa594004d00 0x7fa5a81f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.949+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 >> 172.21.15.45:0/1559765706 conn(0x7fa5a8070f40 msgr2=0x7fa5a80d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.949+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 shutdown_connections 2023-11-28T20:54:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:37.949+0000 7fa5ae0c3640 1 -- 172.21.15.45:0/1559765706 wait complete. 2023-11-28T20:54:37.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T20:54:37.988 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:37.986+0000 7f65eb015040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:54:38.020 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:38.018+0000 7fc516e3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:54:38.041 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.038+0000 7f65eb015040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:54:38.076 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:38.074+0000 7fc516e3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:54:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.180+0000 7f5895c27640 1 Processor -- start 2023-11-28T20:54:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.180+0000 7f5895c27640 1 -- start start 2023-11-28T20:54:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.182+0000 7f5895c27640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58900ceab0 0x7f589015b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.182+0000 7f5895c27640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58900c4c00 con 0x7f589015b0a0 2023-11-28T20:54:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.182+0000 7f5895c27640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5890070c40 con 0x7f58900c46b0 2023-11-28T20:54:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.182+0000 7f5895c27640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5890070a60 con 0x7f58900ceab0 2023-11-28T20:54:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.182+0000 7f5894c25640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f589015b0a0 0x7f58900ce3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44694/0 (socket says 172.21.15.45:44694) 2023-11-28T20:54:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.182+0000 7f5894c25640 1 -- 172.21.15.45:0/1416662256 learned_addr learned my addr 172.21.15.45:0/1416662256 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.182+0000 7f588ffff640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58900ceab0 0x7f589015b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604427109 0 0) 0x7f58900c4c00 con 0x7f589015b0a0 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5880003520 con 0x7f589015b0a0 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630637760 0 0) 0x7f5890070c40 con 0x7f58900c46b0 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58900c4c00 con 0x7f58900c46b0 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952468718 0 0) 0x7f5880003520 con 0x7f589015b0a0 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58900ceab0 msgr2=0x7f589015b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58900ceab0 0x7f589015b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 >> v1:172.21.15.190:6789/0 conn(0x7f58900c46b0 legacy=0x7f589015b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5890070f20 con 0x7f589015b0a0 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5884001670 con 0x7f589015b0a0 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 >> v1:172.21.15.45:6789/0 conn(0x7f589015b0a0 legacy=0x7f58900ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880003b30 0x7f5880003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58900ceab0 0x7f58800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5880004d00 0x7f5880005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890070a60 con 0x7f5880004d00 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588000ba60 con 0x7f5880003b30 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.183+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588000bc20 con 0x7f58900ceab0 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.184+0000 7f5895426640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5880004d00 0x7f5880005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.184+0000 7f5894c25640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880003b30 0x7f5880003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.184+0000 7f588ffff640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58900ceab0 0x7f58800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.184+0000 7f5895426640 1 -- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58900ceab0 msgr2=0x7f58800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.184+0000 7f5895426640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58900ceab0 0x7f58800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.184+0000 7f5895426640 1 -- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880003b30 msgr2=0x7f5880003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.184+0000 7f5895426640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880003b30 0x7f5880003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.184+0000 7f5895426640 1 -- 172.21.15.45:0/1416662256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5880003520 con 0x7f5880004d00 2023-11-28T20:54:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.186+0000 7f5895426640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5880004d00 0x7f5880005180 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f5888001fd0 tx=0x7f58800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e386df2bc9338098 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.186+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888003b70 con 0x7f5880004d00 2023-11-28T20:54:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.186+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5888003cd0 con 0x7f5880004d00 2023-11-28T20:54:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.186+0000 7f588f7fe640 1 -- 172.21.15.45:0/1416662256 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58880052f0 con 0x7f5880004d00 2023-11-28T20:54:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.186+0000 7f5895c27640 1 -- 172.21.15.45:0/1416662256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5880004d00 msgr2=0x7f5880005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.186+0000 7f5895c27640 1 --2- 172.21.15.45:0/1416662256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5880004d00 0x7f5880005180 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f5888001fd0 tx=0x7f58800057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.186+0000 7f5895426640 1 -- 172.21.15.45:0/1416662256 reap_dead start 2023-11-28T20:54:38.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.187+0000 7f5895c27640 1 -- 172.21.15.45:0/1416662256 shutdown_connections 2023-11-28T20:54:38.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.187+0000 7f5895c27640 1 -- 172.21.15.45:0/1416662256 >> 172.21.15.45:0/1416662256 conn(0x7f58900661f0 msgr2=0x7f58900665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:38.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.187+0000 7f5895c27640 1 -- 172.21.15.45:0/1416662256 shutdown_connections 2023-11-28T20:54:38.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.187+0000 7f5895c27640 1 -- 172.21.15.45:0/1416662256 wait complete. 2023-11-28T20:54:38.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.189+0000 7f5895c27640 1 Processor -- start 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.189+0000 7f5895c27640 1 -- start start 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895c27640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5880004d00 0x7f589014bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895c27640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589015cee0 0x7f5890148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895c27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f589014c640 0x7f589014cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895c27640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589014d490 con 0x7f589014c640 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895c27640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589014d000 con 0x7f589015cee0 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895c27640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589015d650 con 0x7f5880004d00 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895426640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f589014c640 0x7f589014cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895426640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f589014c640 0x7f589014cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33428/0 (socket says 172.21.15.45:33428) 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895426640 1 -- 172.21.15.45:0/2308038335 learned_addr learned my addr 172.21.15.45:0/2308038335 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f588ffff640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589015cee0 0x7f5890148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5894c25640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5880004d00 0x7f589014bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.190+0000 7f5895426640 1 -- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5880004d00 msgr2=0x7f589014bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f5895426640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5880004d00 0x7f589014bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f5895426640 1 -- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589015cee0 msgr2=0x7f5890148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f5895426640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589015cee0 0x7f5890148100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f5895426640 1 -- 172.21.15.45:0/2308038335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589015d930 con 0x7f589014c640 2023-11-28T20:54:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f5895426640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f589014c640 0x7f589014cac0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f5888005040 tx=0x7f5888001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:38.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f588d7fa640 1 -- 172.21.15.45:0/2308038335 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888005600 con 0x7f589014c640 2023-11-28T20:54:38.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f588d7fa640 1 -- 172.21.15.45:0/2308038335 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5888005760 con 0x7f589014c640 2023-11-28T20:54:38.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f588d7fa640 1 -- 172.21.15.45:0/2308038335 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588802c770 con 0x7f589014c640 2023-11-28T20:54:38.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589015dc10 con 0x7f589014c640 2023-11-28T20:54:38.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.191+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589015d360 con 0x7f589014c640 2023-11-28T20:54:38.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.192+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5890072380 con 0x7f589014c640 2023-11-28T20:54:38.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.193+0000 7f588d7fa640 1 -- 172.21.15.45:0/2308038335 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f588802c8d0 con 0x7f589014c640 2023-11-28T20:54:38.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.194+0000 7f588d7fa640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f587406afe0 0x7f587406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.194+0000 7f588d7fa640 1 -- 172.21.15.45:0/2308038335 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f58880aa390 con 0x7f589014c640 2023-11-28T20:54:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.195+0000 7f5894c25640 1 -- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f587406afe0 msgr2=0x7f587406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2586319130 2023-11-28T20:54:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.195+0000 7f5894c25640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f587406afe0 0x7f587406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:38.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.203+0000 7f588d7fa640 1 -- 172.21.15.45:0/2308038335 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5888075280 con 0x7f589014c640 2023-11-28T20:54:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.396+0000 7f5894c25640 1 -- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f587406afe0 msgr2=0x7f587406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2586319130 2023-11-28T20:54:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.396+0000 7f5894c25640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f587406afe0 0x7f587406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.400+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5890072380 con 0x7f589014c640 2023-11-28T20:54:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.405+0000 7f588d7fa640 1 -- 172.21.15.45:0/2308038335 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f58880788a0 con 0x7f589014c640 2023-11-28T20:54:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f587406afe0 msgr2=0x7f587406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f587406afe0 0x7f587406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f589014c640 msgr2=0x7f589014cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f589014c640 0x7f589014cac0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f5888005040 tx=0x7f5888001650 comp rx=0 tx=0).stop 2023-11-28T20:54:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 shutdown_connections 2023-11-28T20:54:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7f587406afe0 0x7f587406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f589014c640 0x7f589014cac0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589015cee0 0x7f5890148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 --2- 172.21.15.45:0/2308038335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5880004d00 0x7f589014bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.407+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 >> 172.21.15.45:0/2308038335 conn(0x7f58900661f0 msgr2=0x7f5890158ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.408+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 shutdown_connections 2023-11-28T20:54:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.408+0000 7f5895c27640 1 -- 172.21.15.45:0/2308038335 wait complete. 2023-11-28T20:54:38.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:38.428 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2023-11-28T20:54:38.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2023-11-28T20:54:38.489 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.487+0000 7f65eb015040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:54:38.542 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.540+0000 7f65eb015040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:54:38.562 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:38.560+0000 7fc516e3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:54:38.591 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.589+0000 7f65eb015040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:54:38.618 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:38.616+0000 7fc516e3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:54:38.652 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.650+0000 7f65eb015040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:54:38.672 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:38.670+0000 7fc516e3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:54:38.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.690+0000 7fa3ff59e640 1 Processor -- start 2023-11-28T20:54:38.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.690+0000 7fa3ff59e640 1 -- start start 2023-11-28T20:54:38.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.692+0000 7fa3ff59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4000ce570 0x7fa400070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.692+0000 7fa3ff59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa400071010 0x7fa40015f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.692+0000 7fa3ff59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40015d510 con 0x7fa40015d130 2023-11-28T20:54:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.692+0000 7fa3ff59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa40015e130 con 0x7fa4000ce570 2023-11-28T20:54:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.692+0000 7fa3ff59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40015e410 con 0x7fa400071010 2023-11-28T20:54:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.693+0000 7fa3fe59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa40015d130 0x7fa4000cde60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44708/0 (socket says 172.21.15.45:44708) 2023-11-28T20:54:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.693+0000 7fa3fe59c640 1 -- 172.21.15.45:0/1404457087 learned_addr learned my addr 172.21.15.45:0/1404457087 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.693+0000 7fa3f7fff640 1 --2- 172.21.15.45:0/1404457087 >> v2:172.21.15.190:3300/0 conn(0x7fa4000ce570 0x7fa400070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.693+0000 7fa3fed9d640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa400071010 0x7fa40015f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.693+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533597198 0 0) 0x7fa40015d510 con 0x7fa40015d130 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.693+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ec003520 con 0x7fa40015d130 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3355115149 0 0) 0x7fa3ec003520 con 0x7fa40015d130 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa400071010 msgr2=0x7fa40015f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa400071010 0x7fa40015f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 >> v2:172.21.15.190:3300/0 conn(0x7fa4000ce570 msgr2=0x7fa400070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 --2- 172.21.15.45:0/1404457087 >> v2:172.21.15.190:3300/0 conn(0x7fa4000ce570 0x7fa400070ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40015e6f0 con 0x7fa40015d130 2023-11-28T20:54:38.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fa3f0001670 con 0x7fa40015d130 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 >> v1:172.21.15.45:6789/0 conn(0x7fa40015d130 legacy=0x7fa4000cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa400071010 0x7fa3ec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3ec003f40 0x7fa3ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3ec004d00 0x7fa3ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40015e130 con 0x7fa3ec003f40 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40015e410 con 0x7fa3ec004d00 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec00b980 con 0x7fa400071010 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fed9d640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3ec004d00 0x7fa3ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3f7fff640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3ec003f40 0x7fa3ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.694+0000 7fa3fe59c640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa400071010 0x7fa3ec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.695+0000 7fa3f7fff640 1 -- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa400071010 msgr2=0x7fa3ec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.695+0000 7fa3f7fff640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa400071010 0x7fa3ec003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.695+0000 7fa3f7fff640 1 -- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3ec004d00 msgr2=0x7fa3ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.695+0000 7fa3f7fff640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3ec004d00 0x7fa3ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.695+0000 7fa3f7fff640 1 -- 172.21.15.45:0/1404457087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec003520 con 0x7fa3ec003f40 2023-11-28T20:54:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.695+0000 7fa3fe59c640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa400071010 0x7fa3ec003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.695+0000 7fa3fed9d640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3ec004d00 0x7fa3ec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.697+0000 7fa3f7fff640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3ec003f40 0x7fa3ec0047c0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009a90 tx=0x7fa3ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7420f4a06eeb6ae server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.697+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e802c680 con 0x7fa3ec003f40 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.697+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e802c7e0 con 0x7fa3ec003f40 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.697+0000 7fa3fdd9b640 1 -- 172.21.15.45:0/1404457087 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e802cb70 con 0x7fa3ec003f40 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.698+0000 7fa3ff59e640 1 -- 172.21.15.45:0/1404457087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3ec003f40 msgr2=0x7fa3ec0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.698+0000 7fa3ff59e640 1 --2- 172.21.15.45:0/1404457087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3ec003f40 0x7fa3ec0047c0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009a90 tx=0x7fa3ec0057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.698+0000 7fa3fed9d640 1 -- 172.21.15.45:0/1404457087 reap_dead start 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.698+0000 7fa3ff59e640 1 -- 172.21.15.45:0/1404457087 shutdown_connections 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.698+0000 7fa3ff59e640 1 -- 172.21.15.45:0/1404457087 >> 172.21.15.45:0/1404457087 conn(0x7fa4000661f0 msgr2=0x7fa4000665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.698+0000 7fa3ff59e640 1 -- 172.21.15.45:0/1404457087 shutdown_connections 2023-11-28T20:54:38.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.698+0000 7fa3ff59e640 1 -- 172.21.15.45:0/1404457087 wait complete. 2023-11-28T20:54:38.702 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.699+0000 7f65eb015040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:54:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.701+0000 7fa3ff59e640 1 Processor -- start 2023-11-28T20:54:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.701+0000 7fa3ff59e640 1 -- start start 2023-11-28T20:54:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.702+0000 7fa3ff59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3ec003f40 0x7fa40013cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.702+0000 7fa3ff59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa400149950 0x7fa400149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.702+0000 7fa3ff59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4001510d0 0x7fa400151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.702+0000 7fa3ff59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40014acb0 con 0x7fa4001510d0 2023-11-28T20:54:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.702+0000 7fa3ff59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40014eb40 con 0x7fa400149950 2023-11-28T20:54:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.702+0000 7fa3ff59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa400151a90 con 0x7fa3ec003f40 2023-11-28T20:54:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3fed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4001510d0 0x7fa400151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3fed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4001510d0 0x7fa400151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33444/0 (socket says 172.21.15.45:33444) 2023-11-28T20:54:38.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3fed9d640 1 -- 172.21.15.45:0/2572902052 learned_addr learned my addr 172.21.15.45:0/2572902052 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3fe59c640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3ec003f40 0x7fa40013cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3f7fff640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa400149950 0x7fa400149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3fed9d640 1 -- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3ec003f40 msgr2=0x7fa40013cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3fed9d640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3ec003f40 0x7fa40013cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3fed9d640 1 -- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa400149950 msgr2=0x7fa400149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3fed9d640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa400149950 0x7fa400149dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.703+0000 7fa3fed9d640 1 -- 172.21.15.45:0/2572902052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40014a410 con 0x7fa4001510d0 2023-11-28T20:54:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.704+0000 7fa3fed9d640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4001510d0 0x7fa400151550 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fa4000b4bb0 tx=0x7fa3f800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.704+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f800e040 con 0x7fa4001510d0 2023-11-28T20:54:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.704+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f800f040 con 0x7fa4001510d0 2023-11-28T20:54:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.704+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8003bd0 con 0x7fa4001510d0 2023-11-28T20:54:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.704+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40014a6f0 con 0x7fa4001510d0 2023-11-28T20:54:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.704+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40013d400 con 0x7fa4001510d0 2023-11-28T20:54:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.706+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40014e8a0 con 0x7fa4001510d0 2023-11-28T20:54:38.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.706+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 69) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fa3f8015430 con 0x7fa4001510d0 2023-11-28T20:54:38.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.707+0000 7fa3f6ffd640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 0x7fa3e006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.708+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa3f8089e20 con 0x7fa4001510d0 2023-11-28T20:54:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.708+0000 7fa3fe59c640 1 -- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 msgr2=0x7fa3e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2586319130 2023-11-28T20:54:38.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.708+0000 7fa3fe59c640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 0x7fa3e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:38.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.716+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa40014e8a0 con 0x7fa4001510d0 2023-11-28T20:54:38.742 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:38.740+0000 7fc516e3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:54:38.794 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:38.792+0000 7fc516e3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:54:38.812 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.810+0000 7f65eb015040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:54:38.875 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.873+0000 7f65eb015040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:38.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.909+0000 7fa3fe59c640 1 -- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 msgr2=0x7fa3e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2586319130 2023-11-28T20:54:38.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.909+0000 7fa3fe59c640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 0x7fa3e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:38.912 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:38.910+0000 7fc516e3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:54:38.924 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.922+0000 7f65eb015040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:54:38.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:38.931+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 --> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4000714c0 con 0x7fa3e006afe0 2023-11-28T20:54:38.979 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:38.977+0000 7fc516e3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:38.982 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:38.980+0000 7f65eb015040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:54:39.031 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:39.029+0000 7fc516e3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:54:39.032 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:39.030+0000 7f65eb015040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:54:39.090 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:39.088+0000 7fc516e3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:54:39.143 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:39.141+0000 7fc516e3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:54:39.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:39.310+0000 7fa3fe59c640 1 -- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 msgr2=0x7fa3e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2586319130 2023-11-28T20:54:39.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:39.310+0000 7fa3fe59c640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 0x7fa3e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T20:54:39.410 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:39.407+0000 7f65eb015040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:54:39.545 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:39.543+0000 7fc516e3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:54:39.571 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:39.569+0000 7f65eb015040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:54:39.678 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:39.675+0000 7f65eb015040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:54:39.713 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:39.711+0000 7fc516e3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:54:39.727 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:39.725+0000 7f65eb015040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:54:39.828 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:39.826+0000 7fc516e3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:54:39.835 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:39.833+0000 7f65eb015040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:39.880 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:39.878+0000 7fc516e3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:54:39.942 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:39.940+0000 7f65eb015040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:54:39.994 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:39.992+0000 7fc516e3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:40.097 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:40.094+0000 7f65eb015040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:40.106 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:40.104+0000 7fc516e3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:54:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:40.111+0000 7fa3fe59c640 1 -- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 msgr2=0x7fa3e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2586319130 2023-11-28T20:54:40.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:40.111+0000 7fa3fe59c640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 0x7fa3e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T20:54:40.146 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:40.143+0000 7f65eb015040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:40.263 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:40.261+0000 7f65eb015040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:54:40.279 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:40.277+0000 7fc516e3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:40.336 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:40.334+0000 7fc516e3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:40.463 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:40.461+0000 7fc516e3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:54:41.085 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.083+0000 7f65eb015040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:54:41.358 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:41.357+0000 7fc516e3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:54:41.379 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.377+0000 7f65eb015040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:54:41.435 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.433+0000 7f65eb015040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:54:41.549 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.546+0000 7f65eb015040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:54:41.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:41.567+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 70) v1 ==== 88959+0+0 (secure 0 0 0) 0x7fa3f80881e0 con 0x7fa4001510d0 2023-11-28T20:54:41.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:41.567+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 msgr2=0x7fa3e006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:41.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:41.567+0000 7fa3f6ffd640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/2586319130,v1:172.21.15.190:6833/2586319130] conn(0x7fa3e006afe0 0x7fa3e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:41.681 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:41.680+0000 7fc516e3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:54:41.712 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f65677bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.712 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f65677bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.712 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f65677bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.713 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f65677bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.713 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f65677bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.713 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f656a7c2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.713 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f656a7c2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.713 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f656a7c2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.713 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f656a7c2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.713 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:41.709+0000 7f656a7c2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:41.745 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:41.743+0000 7fc516e3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:54:41.863 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:41.862+0000 7fc516e3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:54:42.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.578+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 71) v1 ==== 89086+0+0 (secure 0 0 0) 0x7fa3f8054fd0 con 0x7fa4001510d0 2023-11-28T20:54:42.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.580+0000 7fa3f6ffd640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fa3e0035880 0x7fa3e0075410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:42.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.580+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3e0035c60 con 0x7fa3e0035880 2023-11-28T20:54:42.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.582+0000 7fa3fe59c640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fa3e0035880 0x7fa3e0075410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:42.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.598+0000 7fa3fe59c640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fa3e0035880 0x7fa3e0075410 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0000c00 tx=0x7fa3f0001e00 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.734+0000 7fa3f6ffd640 1 -- 172.21.15.45:0/2572902052 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fa3e0035c60 con 0x7fa3e0035880 2023-11-28T20:54:42.735 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1/minutes 2023-11-28T20:54:42.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.738+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fa3e0035880 msgr2=0x7fa3e0075410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:42.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.738+0000 7fa3ff59e640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fa3e0035880 0x7fa3e0075410 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0000c00 tx=0x7fa3f0001e00 comp rx=0 tx=0).stop 2023-11-28T20:54:42.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.738+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4001510d0 msgr2=0x7fa400151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.738+0000 7fa3ff59e640 1 --2- 172.21.15.45:0/2572902052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4001510d0 0x7fa400151550 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fa4000b4bb0 tx=0x7fa3f800f580 comp rx=0 tx=0).stop 2023-11-28T20:54:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.738+0000 7fa3fed9d640 1 -- 172.21.15.45:0/2572902052 reap_dead start 2023-11-28T20:54:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.738+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 shutdown_connections 2023-11-28T20:54:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.738+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 >> 172.21.15.45:0/2572902052 conn(0x7fa4000661f0 msgr2=0x7fa4000b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.738+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 shutdown_connections 2023-11-28T20:54:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:42.738+0000 7fa3ff59e640 1 -- 172.21.15.45:0/2572902052 wait complete. 2023-11-28T20:54:42.763 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2023-11-28T20:54:42.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:42.763 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2023-11-28T20:54:42.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2023-11-28T20:54:43.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.055+0000 7f1f35567640 1 Processor -- start 2023-11-28T20:54:43.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.056+0000 7f1f35567640 1 -- start start 2023-11-28T20:54:43.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.057+0000 7f1f35567640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1f300b2430 0x7f1f300ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.057+0000 7f1f35567640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f30071ac0 0x7f1f300cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.057+0000 7f1f35567640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3009ceb0 con 0x7f1f30159fc0 2023-11-28T20:54:43.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.058+0000 7f1f35567640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f30071450 con 0x7f1f30071ac0 2023-11-28T20:54:43.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.058+0000 7f1f35567640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f300b2cb0 con 0x7f1f300b2430 2023-11-28T20:54:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.058+0000 7f1f34d66640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f30071ac0 0x7f1f300cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.058+0000 7f1f2ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1f30159fc0 0x7f1f300b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44724/0 (socket says 172.21.15.45:44724) 2023-11-28T20:54:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.058+0000 7f1f2ffff640 1 -- 172.21.15.45:0/3914950461 learned_addr learned my addr 172.21.15.45:0/3914950461 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.058+0000 7f1f277fe640 1 --2- 172.21.15.45:0/3914950461 >> v2:172.21.15.190:3300/0 conn(0x7f1f300b2430 0x7f1f300ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.059+0000 7f1f2f7fe640 1 -- 172.21.15.45:0/3914950461 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999321737 0 0) 0x7f1f3009ceb0 con 0x7f1f30159fc0 2023-11-28T20:54:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.059+0000 7f1f34d66640 1 -- 172.21.15.45:0/3914950461 >> v2:172.21.15.190:3300/0 conn(0x7f1f300b2430 msgr2=0x7f1f300ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.059+0000 7f1f34d66640 1 --2- 172.21.15.45:0/3914950461 >> v2:172.21.15.190:3300/0 conn(0x7f1f300b2430 0x7f1f300ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.059+0000 7f1f34d66640 1 -- 172.21.15.45:0/3914950461 >> v1:172.21.15.45:6789/0 conn(0x7f1f30159fc0 legacy=0x7f1f300b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.059+0000 7f1f34d66640 1 -- 172.21.15.45:0/3914950461 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300b29d0 con 0x7f1f30071ac0 2023-11-28T20:54:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.059+0000 7f1f277fe640 1 --2- 172.21.15.45:0/3914950461 >> v2:172.21.15.190:3300/0 conn(0x7f1f300b2430 0x7f1f300ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.060+0000 7f1f34d66640 1 --2- 172.21.15.45:0/3914950461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f30071ac0 0x7f1f300cd0d0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f1f3006b630 tx=0x7f1f2802da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cf3502c407e9e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.060+0000 7f1f2f7fe640 1 -- 172.21.15.45:0/3914950461 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28031870 con 0x7f1f30071ac0 2023-11-28T20:54:43.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.060+0000 7f1f2f7fe640 1 -- 172.21.15.45:0/3914950461 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f280319d0 con 0x7f1f30071ac0 2023-11-28T20:54:43.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.060+0000 7f1f2f7fe640 1 -- 172.21.15.45:0/3914950461 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28031d60 con 0x7f1f30071ac0 2023-11-28T20:54:43.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.061+0000 7f1f35567640 1 -- 172.21.15.45:0/3914950461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f30071ac0 msgr2=0x7f1f300cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:43.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.061+0000 7f1f35567640 1 --2- 172.21.15.45:0/3914950461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f30071ac0 0x7f1f300cd0d0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f1f3006b630 tx=0x7f1f2802da50 comp rx=0 tx=0).stop 2023-11-28T20:54:43.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.061+0000 7f1f35567640 1 -- 172.21.15.45:0/3914950461 shutdown_connections 2023-11-28T20:54:43.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.061+0000 7f1f35567640 1 --2- 172.21.15.45:0/3914950461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f30071ac0 0x7f1f300cd0d0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.061+0000 7f1f35567640 1 --2- 172.21.15.45:0/3914950461 >> v2:172.21.15.190:3300/0 conn(0x7f1f300b2430 0x7f1f300ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.061+0000 7f1f35567640 1 -- 172.21.15.45:0/3914950461 >> 172.21.15.45:0/3914950461 conn(0x7f1f300b1050 msgr2=0x7f1f300cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:43.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.061+0000 7f1f35567640 1 -- 172.21.15.45:0/3914950461 shutdown_connections 2023-11-28T20:54:43.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.061+0000 7f1f35567640 1 -- 172.21.15.45:0/3914950461 wait complete. 2023-11-28T20:54:43.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.064+0000 7f1f35567640 1 Processor -- start 2023-11-28T20:54:43.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.064+0000 7f1f35567640 1 -- start start 2023-11-28T20:54:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.065+0000 7f1f35567640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f30071ac0 0x7f1f3015a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.065+0000 7f1f35567640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f300b2430 0x7f1f3015c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f35567640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f30159fc0 0x7f1f301468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f35567640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3015cb80 con 0x7f1f300b2430 2023-11-28T20:54:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f35567640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3015ac20 con 0x7f1f30159fc0 2023-11-28T20:54:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f35567640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3015af00 con 0x7f1f30071ac0 2023-11-28T20:54:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f277fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f300b2430 0x7f1f3015c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f277fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f300b2430 0x7f1f3015c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33464/0 (socket says 172.21.15.45:33464) 2023-11-28T20:54:43.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f277fe640 1 -- 172.21.15.45:0/321779652 learned_addr learned my addr 172.21.15.45:0/321779652 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f34d66640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f30159fc0 0x7f1f301468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f277fe640 1 -- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f30071ac0 msgr2=0x7f1f3015a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f2ffff640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f30071ac0 0x7f1f3015a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f277fe640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f30071ac0 0x7f1f3015a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f277fe640 1 -- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f30159fc0 msgr2=0x7f1f301468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f277fe640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f30159fc0 0x7f1f301468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.066+0000 7f1f277fe640 1 -- 172.21.15.45:0/321779652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28009400 con 0x7f1f300b2430 2023-11-28T20:54:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.067+0000 7f1f277fe640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f300b2430 0x7f1f3015c640 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800e7e0 tx=0x7f1f1800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.067+0000 7f1f2d7fa640 1 -- 172.21.15.45:0/321779652 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18004030 con 0x7f1f300b2430 2023-11-28T20:54:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.067+0000 7f1f2d7fa640 1 -- 172.21.15.45:0/321779652 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f18004190 con 0x7f1f300b2430 2023-11-28T20:54:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.067+0000 7f1f2d7fa640 1 -- 172.21.15.45:0/321779652 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18004520 con 0x7f1f300b2430 2023-11-28T20:54:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.067+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3015b1e0 con 0x7f1f300b2430 2023-11-28T20:54:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.067+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3015b7a0 con 0x7f1f300b2430 2023-11-28T20:54:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.069+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f30154960 con 0x7f1f300b2430 2023-11-28T20:54:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.069+0000 7f1f2d7fa640 1 -- 172.21.15.45:0/321779652 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 71) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f1f1800c6d0 con 0x7f1f300b2430 2023-11-28T20:54:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.070+0000 7f1f2d7fa640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f1f1006ace0 0x7f1f1006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.070+0000 7f1f2d7fa640 1 -- 172.21.15.45:0/321779652 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1f18089310 con 0x7f1f300b2430 2023-11-28T20:54:43.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.071+0000 7f1f2ffff640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f1f1006ace0 0x7f1f1006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.072+0000 7f1f2ffff640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f1f1006ace0 0x7f1f1006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f200012e0 tx=0x7f1f20009040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:43.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.075+0000 7f1f2d7fa640 1 -- 172.21.15.45:0/321779652 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f18054370 con 0x7f1f300b2430 2023-11-28T20:54:43.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.291+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f3015a3a0 con 0x7f1f1006ace0 2023-11-28T20:54:43.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.448+0000 7f1f2d7fa640 1 -- 172.21.15.45:0/321779652 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f1f20001900 con 0x7f1f1006ace0 2023-11-28T20:54:43.449 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1/hourly 2023-11-28T20:54:43.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.451+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f1f1006ace0 msgr2=0x7f1f1006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:43.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.451+0000 7f1f35567640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f1f1006ace0 0x7f1f1006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f200012e0 tx=0x7f1f20009040 comp rx=0 tx=0).stop 2023-11-28T20:54:43.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.451+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f300b2430 msgr2=0x7f1f3015c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:43.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.451+0000 7f1f35567640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f300b2430 0x7f1f3015c640 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800e7e0 tx=0x7f1f1800eca0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.451+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 shutdown_connections 2023-11-28T20:54:43.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.451+0000 7f1f35567640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f1f1006ace0 0x7f1f1006d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.451+0000 7f1f35567640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f30159fc0 0x7f1f301468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.451+0000 7f1f35567640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f300b2430 0x7f1f3015c640 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.451+0000 7f1f35567640 1 --2- 172.21.15.45:0/321779652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f30071ac0 0x7f1f3015a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.452+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 >> 172.21.15.45:0/321779652 conn(0x7f1f300b1050 msgr2=0x7f1f300ad330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:43.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.452+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 shutdown_connections 2023-11-28T20:54:43.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.452+0000 7f1f35567640 1 -- 172.21.15.45:0/321779652 wait complete. 2023-11-28T20:54:43.475 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2023-11-28T20:54:43.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:43.476 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2023-11-28T20:54:43.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2023-11-28T20:54:43.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.770+0000 7fc6d7602640 1 Processor -- start 2023-11-28T20:54:43.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.770+0000 7fc6d7602640 1 -- start start 2023-11-28T20:54:43.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.772+0000 7fc6d7602640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6d015e840 0x7fc6d006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.772+0000 7fc6d7602640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc6d015f720 0x7fc6d00717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.772+0000 7fc6d7602640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d015ecd0 con 0x7fc6d006a500 2023-11-28T20:54:43.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.772+0000 7fc6d7602640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d0071ce0 con 0x7fc6d015e840 2023-11-28T20:54:43.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.772+0000 7fc6d7602640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d015f290 con 0x7fc6d015f720 2023-11-28T20:54:43.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.772+0000 7fc6d539e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6d015e840 0x7fc6d006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.772+0000 7fc6d539e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc6d015e840 0x7fc6d006c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33480/0 (socket says 172.21.15.45:33480) 2023-11-28T20:54:43.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.772+0000 7fc6d539e640 1 -- 172.21.15.45:0/2757902701 learned_addr learned my addr 172.21.15.45:0/2757902701 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:43.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.772+0000 7fc6d4b9d640 1 --2- 172.21.15.45:0/2757902701 >> v2:172.21.15.190:3300/0 conn(0x7fc6d015f720 0x7fc6d00717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.773+0000 7fc6d539e640 1 -- 172.21.15.45:0/2757902701 >> v2:172.21.15.190:3300/0 conn(0x7fc6d015f720 msgr2=0x7fc6d00717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:43.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.773+0000 7fc6d539e640 1 --2- 172.21.15.45:0/2757902701 >> v2:172.21.15.190:3300/0 conn(0x7fc6d015f720 0x7fc6d00717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.773+0000 7fc6d539e640 1 -- 172.21.15.45:0/2757902701 >> v1:172.21.15.190:6789/0 conn(0x7fc6d006a500 legacy=0x7fc6d00cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:43.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.773+0000 7fc6d539e640 1 -- 172.21.15.45:0/2757902701 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d015fb60 con 0x7fc6d015e840 2023-11-28T20:54:43.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.774+0000 7fc6d539e640 1 --2- 172.21.15.45:0/2757902701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 0x7fc6d006c350 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fc6d006e6d0 tx=0x7fc6c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97f766eb875827e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:43.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.775+0000 7fc6cffff640 1 -- 172.21.15.45:0/2757902701 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0031870 con 0x7fc6d015e840 2023-11-28T20:54:43.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.775+0000 7fc6cffff640 1 -- 172.21.15.45:0/2757902701 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c00319d0 con 0x7fc6d015e840 2023-11-28T20:54:43.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.775+0000 7fc6cffff640 1 -- 172.21.15.45:0/2757902701 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0031d60 con 0x7fc6d015e840 2023-11-28T20:54:43.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.775+0000 7fc6d7602640 1 -- 172.21.15.45:0/2757902701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 msgr2=0x7fc6d006c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:43.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.775+0000 7fc6d7602640 1 --2- 172.21.15.45:0/2757902701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 0x7fc6d006c350 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fc6d006e6d0 tx=0x7fc6c002b910 comp rx=0 tx=0).stop 2023-11-28T20:54:43.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.776+0000 7fc6d7602640 1 -- 172.21.15.45:0/2757902701 shutdown_connections 2023-11-28T20:54:43.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.776+0000 7fc6d7602640 1 --2- 172.21.15.45:0/2757902701 >> v2:172.21.15.190:3300/0 conn(0x7fc6d015f720 0x7fc6d00717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.776+0000 7fc6d7602640 1 --2- 172.21.15.45:0/2757902701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 0x7fc6d006c350 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.776+0000 7fc6d7602640 1 -- 172.21.15.45:0/2757902701 >> 172.21.15.45:0/2757902701 conn(0x7fc6d00ced60 msgr2=0x7fc6d00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:43.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.776+0000 7fc6d7602640 1 -- 172.21.15.45:0/2757902701 shutdown_connections 2023-11-28T20:54:43.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.776+0000 7fc6d7602640 1 -- 172.21.15.45:0/2757902701 wait complete. 2023-11-28T20:54:43.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.777+0000 7fc6d7602640 1 Processor -- start 2023-11-28T20:54:43.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.778+0000 7fc6d7602640 1 -- start start 2023-11-28T20:54:43.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.778+0000 7fc6d7602640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d006a500 0x7fc6d00c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.778+0000 7fc6d7602640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 0x7fc6d00c3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.778+0000 7fc6d7602640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6d015f720 0x7fc6d0156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d7602640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d009b810 con 0x7fc6d015e840 2023-11-28T20:54:43.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d7602640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c3c60 con 0x7fc6d006a500 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d7602640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c2b90 con 0x7fc6d015f720 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d4b9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 0x7fc6d00c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d4b9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 0x7fc6d00c3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33490/0 (socket says 172.21.15.45:33490) 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d4b9d640 1 -- 172.21.15.45:0/1465486073 learned_addr learned my addr 172.21.15.45:0/1465486073 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d539e640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d006a500 0x7fc6d00c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d5b9f640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6d015f720 0x7fc6d0156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d4b9d640 1 -- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6d015f720 msgr2=0x7fc6d0156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d4b9d640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6d015f720 0x7fc6d0156200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d4b9d640 1 -- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d006a500 msgr2=0x7fc6d00c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d4b9d640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d006a500 0x7fc6d00c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.779+0000 7fc6d4b9d640 1 -- 172.21.15.45:0/1465486073 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c00094d0 con 0x7fc6d015e840 2023-11-28T20:54:43.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.780+0000 7fc6d4b9d640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 0x7fc6d00c3720 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400d5f0 tx=0x7fc6c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:43.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.780+0000 7fc6cdffb640 1 -- 172.21.15.45:0/1465486073 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40125f0 con 0x7fc6d015e840 2023-11-28T20:54:43.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.780+0000 7fc6cdffb640 1 -- 172.21.15.45:0/1465486073 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c4012750 con 0x7fc6d015e840 2023-11-28T20:54:43.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.780+0000 7fc6cdffb640 1 -- 172.21.15.45:0/1465486073 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4012b00 con 0x7fc6d015e840 2023-11-28T20:54:43.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.780+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d009ac20 con 0x7fc6d015e840 2023-11-28T20:54:43.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.780+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d009b150 con 0x7fc6d015e840 2023-11-28T20:54:43.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.781+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d009ba80 con 0x7fc6d015e840 2023-11-28T20:54:43.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.782+0000 7fc6cdffb640 1 -- 172.21.15.45:0/1465486073 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 72) v1 ==== 89351+0+0 (secure 0 0 0) 0x7fc6c4012c60 con 0x7fc6d015e840 2023-11-28T20:54:43.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.783+0000 7fc6cdffb640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fc6bc06afb0 0x7fc6bc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:43.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.783+0000 7fc6cdffb640 1 -- 172.21.15.45:0/1465486073 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc6c4089f00 con 0x7fc6d015e840 2023-11-28T20:54:43.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.784+0000 7fc6d539e640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fc6bc06afb0 0x7fc6bc06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.785+0000 7fc6d539e640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fc6bc06afb0 0x7fc6bc06d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0036000 tx=0x7fc6c003c040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:43.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:43.792+0000 7fc6cdffb640 1 -- 172.21.15.45:0/1465486073 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6c4054c50 con 0x7fc6d015e840 2023-11-28T20:54:44.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.010+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6d015ef20 con 0x7fc6bc06afb0 2023-11-28T20:54:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.164+0000 7fc6cdffb640 1 -- 172.21.15.45:0/1465486073 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc6c0031bc0 con 0x7fc6bc06afb0 2023-11-28T20:54:44.165 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1/daily 2023-11-28T20:54:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fc6bc06afb0 msgr2=0x7fc6bc06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fc6bc06afb0 0x7fc6bc06d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0036000 tx=0x7fc6c003c040 comp rx=0 tx=0).stop 2023-11-28T20:54:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 msgr2=0x7fc6d00c3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 0x7fc6d00c3720 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400d5f0 tx=0x7fc6c400dab0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 shutdown_connections 2023-11-28T20:54:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fc6bc06afb0 0x7fc6bc06d470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6d015f720 0x7fc6d0156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d015e840 0x7fc6d00c3720 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 --2- 172.21.15.45:0/1465486073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d006a500 0x7fc6d00c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.167+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 >> 172.21.15.45:0/1465486073 conn(0x7fc6d00ced60 msgr2=0x7fc6d006da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:44.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.168+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 shutdown_connections 2023-11-28T20:54:44.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.168+0000 7fc6d7602640 1 -- 172.21.15.45:0/1465486073 wait complete. 2023-11-28T20:54:44.191 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2023-11-28T20:54:44.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:44.192 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2023-11-28T20:54:44.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2023-11-28T20:54:44.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.485+0000 7f5e84d75640 1 Processor -- start 2023-11-28T20:54:44.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.485+0000 7f5e84d75640 1 -- start start 2023-11-28T20:54:44.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.487+0000 7f5e84d75640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5e80159fc0 0x7f5e800b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:44.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.487+0000 7f5e84d75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800cdba0 0x7f5e800b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:44.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.487+0000 7f5e84d75640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e800b27a0 con 0x7f5e80071ac0 2023-11-28T20:54:44.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.487+0000 7f5e84d75640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e80071450 con 0x7f5e80159fc0 2023-11-28T20:54:44.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.487+0000 7f5e84d75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8009ceb0 con 0x7f5e800cdba0 2023-11-28T20:54:44.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.487+0000 7f5e7effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5e80071ac0 0x7f5e8015dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46440/0 (socket says 172.21.15.45:46440) 2023-11-28T20:54:44.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.487+0000 7f5e7effd640 1 -- 172.21.15.45:0/3507500274 learned_addr learned my addr 172.21.15.45:0/3507500274 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:44.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723968813 0 0) 0x7f5e800b27a0 con 0x7f5e80071ac0 2023-11-28T20:54:44.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7f7fe640 1 --2- 172.21.15.45:0/3507500274 >> v2:172.21.15.190:3300/0 conn(0x7f5e80159fc0 0x7f5e800b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:44.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800cdba0 0x7f5e800b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:44.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e6c003520 con 0x7f5e80071ac0 2023-11-28T20:54:44.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225557255 0 0) 0x7f5e6c003520 con 0x7f5e80071ac0 2023-11-28T20:54:44.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800cdba0 msgr2=0x7f5e800b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800cdba0 0x7f5e800b2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 >> v2:172.21.15.190:3300/0 conn(0x7f5e80159fc0 msgr2=0x7f5e800b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 --2- 172.21.15.45:0/3507500274 >> v2:172.21.15.190:3300/0 conn(0x7f5e80159fc0 0x7f5e800b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e800b2cb0 con 0x7f5e80071ac0 2023-11-28T20:54:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5e680019f0 con 0x7f5e80071ac0 2023-11-28T20:54:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 >> v1:172.21.15.45:6789/0 conn(0x7f5e80071ac0 legacy=0x7f5e8015dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c003970 0x7f5e6c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:44.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800cdba0 0x7f5e6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:44.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6c004d00 0x7f5e6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:44.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80071450 con 0x7f5e6c004d00 2023-11-28T20:54:44.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8009ceb0 con 0x7f5e6c003970 2023-11-28T20:54:44.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.488+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c00bb10 con 0x7f5e800cdba0 2023-11-28T20:54:44.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7f7fe640 1 --2- 172.21.15.45:0/3507500274 >> v2:172.21.15.190:3300/0 conn(0x7f5e80159fc0 0x7f5e800b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6c004d00 0x7f5e6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7f7fe640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c003970 0x7f5e6c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7effd640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800cdba0 0x7f5e6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7ffff640 1 -- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800cdba0 msgr2=0x7f5e6c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800cdba0 0x7f5e6c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7ffff640 1 -- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c003970 msgr2=0x7f5e6c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c003970 0x7f5e6c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7ffff640 1 -- 172.21.15.45:0/3507500274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c003520 con 0x7f5e6c004d00 2023-11-28T20:54:44.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.489+0000 7f5e7effd640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800cdba0 0x7f5e6c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:44.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.491+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6c004d00 0x7f5e6c005180 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800ba00 tx=0x7f5e6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80dda72d2ca0a9bc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:44.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.491+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78032620 con 0x7f5e6c004d00 2023-11-28T20:54:44.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.491+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e78032780 con 0x7f5e6c004d00 2023-11-28T20:54:44.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.491+0000 7f5e7e7fc640 1 -- 172.21.15.45:0/3507500274 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78032b10 con 0x7f5e6c004d00 2023-11-28T20:54:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.491+0000 7f5e84d75640 1 -- 172.21.15.45:0/3507500274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6c004d00 msgr2=0x7f5e6c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.491+0000 7f5e84d75640 1 --2- 172.21.15.45:0/3507500274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e6c004d00 0x7f5e6c005180 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800ba00 tx=0x7f5e6c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.491+0000 7f5e7ffff640 1 -- 172.21.15.45:0/3507500274 reap_dead start 2023-11-28T20:54:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.491+0000 7f5e84d75640 1 -- 172.21.15.45:0/3507500274 shutdown_connections 2023-11-28T20:54:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.491+0000 7f5e84d75640 1 -- 172.21.15.45:0/3507500274 >> 172.21.15.45:0/3507500274 conn(0x7f5e800b1050 msgr2=0x7f5e800cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.492+0000 7f5e84d75640 1 -- 172.21.15.45:0/3507500274 shutdown_connections 2023-11-28T20:54:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.492+0000 7f5e84d75640 1 -- 172.21.15.45:0/3507500274 wait complete. 2023-11-28T20:54:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.493+0000 7f5e84d75640 1 Processor -- start 2023-11-28T20:54:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.494+0000 7f5e84d75640 1 -- start start 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.494+0000 7f5e84d75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c004d00 0x7f5e801508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e84d75640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e8015ad10 0x7f5e8015b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e84d75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800ae110 0x7f5e800ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e84d75640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8015b6d0 con 0x7f5e8015ad10 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e84d75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80150e00 con 0x7f5e6c004d00 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e84d75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800afee0 con 0x7f5e800ae110 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e8015ad10 0x7f5e8015b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e8015ad10 0x7f5e8015b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33630/0 (socket says 172.21.15.45:33630) 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7effd640 1 -- 172.21.15.45:0/10314726 learned_addr learned my addr 172.21.15.45:0/10314726 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7f7fe640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c004d00 0x7f5e801508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7ffff640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800ae110 0x7f5e800ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:44.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7effd640 1 -- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800ae110 msgr2=0x7f5e800ae590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7effd640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800ae110 0x7f5e800ae590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7effd640 1 -- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c004d00 msgr2=0x7f5e801508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7effd640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c004d00 0x7f5e801508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7effd640 1 -- 172.21.15.45:0/10314726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e800aee40 con 0x7f5e8015ad10 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.495+0000 7f5e7f7fe640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c004d00 0x7f5e801508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.496+0000 7f5e7effd640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e8015ad10 0x7f5e8015b190 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f5e68001370 tx=0x7f5e6800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.496+0000 7f5e77fff640 1 -- 172.21.15.45:0/10314726 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68004d60 con 0x7f5e8015ad10 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.496+0000 7f5e77fff640 1 -- 172.21.15.45:0/10314726 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e680070d0 con 0x7f5e8015ad10 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.496+0000 7f5e77fff640 1 -- 172.21.15.45:0/10314726 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68007460 con 0x7f5e8015ad10 2023-11-28T20:54:44.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.496+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e800af120 con 0x7f5e8015ad10 2023-11-28T20:54:44.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.496+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e800af6e0 con 0x7f5e8015ad10 2023-11-28T20:54:44.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.497+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e80070ad0 con 0x7f5e8015ad10 2023-11-28T20:54:44.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.499+0000 7f5e77fff640 1 -- 172.21.15.45:0/10314726 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 72) v1 ==== 89351+0+0 (secure 0 0 0) 0x7f5e680075c0 con 0x7f5e8015ad10 2023-11-28T20:54:44.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.500+0000 7f5e77fff640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f5e6006afe0 0x7f5e6006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:44.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.500+0000 7f5e77fff640 1 -- 172.21.15.45:0/10314726 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f5e68089d30 con 0x7f5e8015ad10 2023-11-28T20:54:44.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.500+0000 7f5e7f7fe640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f5e6006afe0 0x7f5e6006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:44.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.502+0000 7f5e7f7fe640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f5e6006afe0 0x7f5e6006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e70006fd0 tx=0x7f5e70008040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:44.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.505+0000 7f5e77fff640 1 -- 172.21.15.45:0/10314726 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e680548f0 con 0x7f5e8015ad10 2023-11-28T20:54:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.724+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e80071070 con 0x7f5e6006afe0 2023-11-28T20:54:44.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.878+0000 7f5e77fff640 1 -- 172.21.15.45:0/10314726 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f5e7000c040 con 0x7f5e6006afe0 2023-11-28T20:54:44.880 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1/weekly 2023-11-28T20:54:44.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f5e6006afe0 msgr2=0x7f5e6006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:44.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f5e6006afe0 0x7f5e6006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e70006fd0 tx=0x7f5e70008040 comp rx=0 tx=0).stop 2023-11-28T20:54:44.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e8015ad10 msgr2=0x7f5e8015b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:44.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e8015ad10 0x7f5e8015b190 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f5e68001370 tx=0x7f5e6800b560 comp rx=0 tx=0).stop 2023-11-28T20:54:44.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 shutdown_connections 2023-11-28T20:54:44.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f5e6006afe0 0x7f5e6006d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e800ae110 0x7f5e800ae590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e8015ad10 0x7f5e8015b190 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 --2- 172.21.15.45:0/10314726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e6c004d00 0x7f5e801508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:44.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 >> 172.21.15.45:0/10314726 conn(0x7f5e800b1050 msgr2=0x7f5e800b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:44.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.881+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 shutdown_connections 2023-11-28T20:54:44.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:44.882+0000 7f5e84d75640 1 -- 172.21.15.45:0/10314726 wait complete. 2023-11-28T20:54:44.906 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2023-11-28T20:54:44.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:44.906 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2023-11-28T20:54:44.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2023-11-28T20:54:45.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.206+0000 7f464eb6e640 1 Processor -- start 2023-11-28T20:54:45.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.206+0000 7f464eb6e640 1 -- start start 2023-11-28T20:54:45.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.207+0000 7f464eb6e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f464806ac10 0x7f464806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.207+0000 7f464eb6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815f720 0x7f46480717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.207+0000 7f464eb6e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464815ecd0 con 0x7f464815e840 2023-11-28T20:54:45.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.207+0000 7f464eb6e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4648071ce0 con 0x7f464806ac10 2023-11-28T20:54:45.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.207+0000 7f464eb6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464815fbd0 con 0x7f464815f720 2023-11-28T20:54:45.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.208+0000 7f4647fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f464806ac10 0x7f464806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.208+0000 7f464d10b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815f720 0x7f46480717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.208+0000 7f4647fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f464806ac10 0x7f464806c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:58158/0 (socket says 172.21.15.45:58158) 2023-11-28T20:54:45.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.208+0000 7f4647fff640 1 -- 172.21.15.45:0/3292084597 learned_addr learned my addr 172.21.15.45:0/3292084597 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:45.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.209+0000 7f464d10b640 1 -- 172.21.15.45:0/3292084597 >> v2:172.21.15.190:3300/0 conn(0x7f464806ac10 msgr2=0x7f464806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:45.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.209+0000 7f464d10b640 1 --2- 172.21.15.45:0/3292084597 >> v2:172.21.15.190:3300/0 conn(0x7f464806ac10 0x7f464806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.209+0000 7f464d10b640 1 -- 172.21.15.45:0/3292084597 >> v1:172.21.15.190:6789/0 conn(0x7f464815e840 legacy=0x7f464806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:45.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.209+0000 7f464d10b640 1 -- 172.21.15.45:0/3292084597 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464815f290 con 0x7f464815f720 2023-11-28T20:54:45.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.209+0000 7f4647fff640 1 --2- 172.21.15.45:0/3292084597 >> v2:172.21.15.190:3300/0 conn(0x7f464806ac10 0x7f464806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:54:45.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.211+0000 7f464d10b640 1 --2- 172.21.15.45:0/3292084597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815f720 0x7f46480717a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4640009d90 tx=0x7f464002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a4075255295d2fc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:45.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.211+0000 7f46477fe640 1 -- 172.21.15.45:0/3292084597 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464002cc50 con 0x7f464815f720 2023-11-28T20:54:45.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f46477fe640 1 -- 172.21.15.45:0/3292084597 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464002cdb0 con 0x7f464815f720 2023-11-28T20:54:45.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f46477fe640 1 -- 172.21.15.45:0/3292084597 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640034660 con 0x7f464815f720 2023-11-28T20:54:45.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f464eb6e640 1 -- 172.21.15.45:0/3292084597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815f720 msgr2=0x7f46480717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:45.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f464eb6e640 1 --2- 172.21.15.45:0/3292084597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815f720 0x7f46480717a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4640009d90 tx=0x7f464002bea0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f464eb6e640 1 -- 172.21.15.45:0/3292084597 shutdown_connections 2023-11-28T20:54:45.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f464eb6e640 1 --2- 172.21.15.45:0/3292084597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815f720 0x7f46480717a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f464eb6e640 1 --2- 172.21.15.45:0/3292084597 >> v2:172.21.15.190:3300/0 conn(0x7f464806ac10 0x7f464806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f464eb6e640 1 -- 172.21.15.45:0/3292084597 >> 172.21.15.45:0/3292084597 conn(0x7f46480ced60 msgr2=0x7f46480cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:45.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f464eb6e640 1 -- 172.21.15.45:0/3292084597 shutdown_connections 2023-11-28T20:54:45.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.212+0000 7f464eb6e640 1 -- 172.21.15.45:0/3292084597 wait complete. 2023-11-28T20:54:45.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.215+0000 7f464eb6e640 1 Processor -- start 2023-11-28T20:54:45.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.215+0000 7f464eb6e640 1 -- start start 2023-11-28T20:54:45.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.216+0000 7f464eb6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f464806ac10 0x7f464815c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.216+0000 7f464eb6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815e840 0x7f46480c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.216+0000 7f464eb6e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f464815f720 0x7f46480c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.216+0000 7f464eb6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46480cad80 con 0x7f464806ac10 2023-11-28T20:54:45.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.216+0000 7f464eb6e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464815cc80 con 0x7f464815f720 2023-11-28T20:54:45.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.216+0000 7f464eb6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464815b450 con 0x7f464815e840 2023-11-28T20:54:45.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f464c90a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f464806ac10 0x7f464815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f464c90a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f464806ac10 0x7f464815c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33646/0 (socket says 172.21.15.45:33646) 2023-11-28T20:54:45.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f464c90a640 1 -- 172.21.15.45:0/2738924884 learned_addr learned my addr 172.21.15.45:0/2738924884 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:45.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f4647fff640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815e840 0x7f46480c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f464d10b640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f464815f720 0x7f46480c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f464c90a640 1 -- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815e840 msgr2=0x7f46480c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:45.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f464c90a640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815e840 0x7f46480c8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f464c90a640 1 -- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f464815f720 msgr2=0x7f46480c5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:45.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f464c90a640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f464815f720 0x7f46480c5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.217+0000 7f464c90a640 1 -- 172.21.15.45:0/2738924884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46340094d0 con 0x7f464806ac10 2023-11-28T20:54:45.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.218+0000 7f464c90a640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f464806ac10 0x7f464815c740 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f463c00c430 tx=0x7f463c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:45.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.218+0000 7f46457fa640 1 -- 172.21.15.45:0/2738924884 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0052d0 con 0x7f464806ac10 2023-11-28T20:54:45.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.218+0000 7f46457fa640 1 -- 172.21.15.45:0/2738924884 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c005450 con 0x7f464806ac10 2023-11-28T20:54:45.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.218+0000 7f46457fa640 1 -- 172.21.15.45:0/2738924884 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c007260 con 0x7f464806ac10 2023-11-28T20:54:45.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.218+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4640009a70 con 0x7f464806ac10 2023-11-28T20:54:45.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.218+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464815b980 con 0x7f464806ac10 2023-11-28T20:54:45.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.220+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46480733d0 con 0x7f464806ac10 2023-11-28T20:54:45.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.221+0000 7f46457fa640 1 -- 172.21.15.45:0/2738924884 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 72) v1 ==== 89351+0+0 (secure 0 0 0) 0x7f463c0073c0 con 0x7f464806ac10 2023-11-28T20:54:45.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.221+0000 7f46457fa640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f462c06afb0 0x7f462c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.222+0000 7f46457fa640 1 -- 172.21.15.45:0/2738924884 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f463c088e70 con 0x7f464806ac10 2023-11-28T20:54:45.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.222+0000 7f4647fff640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f462c06afb0 0x7f462c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.223+0000 7f4647fff640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f462c06afb0 0x7f462c06d470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f46481f60e0 tx=0x7f4634008040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:45.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.226+0000 7f46457fa640 1 -- 172.21.15.45:0/2738924884 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f463c053be0 con 0x7f464806ac10 2023-11-28T20:54:45.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.441+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f464815ef20 con 0x7f462c06afb0 2023-11-28T20:54:45.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.593+0000 7f46457fa640 1 -- 172.21.15.45:0/2738924884 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f4634011e60 con 0x7f462c06afb0 2023-11-28T20:54:45.595 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1/monthly 2023-11-28T20:54:45.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.597+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f462c06afb0 msgr2=0x7f462c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:45.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.597+0000 7f464eb6e640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f462c06afb0 0x7f462c06d470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f46481f60e0 tx=0x7f4634008040 comp rx=0 tx=0).stop 2023-11-28T20:54:45.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.598+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f464806ac10 msgr2=0x7f464815c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:45.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.598+0000 7f46457fa640 1 -- 172.21.15.45:0/2738924884 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 73) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f463c053fb0 con 0x7f464806ac10 2023-11-28T20:54:45.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.598+0000 7f464eb6e640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f464806ac10 0x7f464815c740 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f463c00c430 tx=0x7f463c00c8f0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.598+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 shutdown_connections 2023-11-28T20:54:45.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.598+0000 7f464eb6e640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f462c06afb0 0x7f462c06d470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.598+0000 7f464eb6e640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f464815f720 0x7f46480c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.598+0000 7f464eb6e640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f464815e840 0x7f46480c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.598+0000 7f464eb6e640 1 --2- 172.21.15.45:0/2738924884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f464806ac10 0x7f464815c740 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.598+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 >> 172.21.15.45:0/2738924884 conn(0x7f46480ced60 msgr2=0x7f46480cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:45.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.599+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 shutdown_connections 2023-11-28T20:54:45.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.599+0000 7f464eb6e640 1 -- 172.21.15.45:0/2738924884 wait complete. 2023-11-28T20:54:45.622 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2023-11-28T20:54:45.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:45.623 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2023-11-28T20:54:45.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2023-11-28T20:54:45.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.898+0000 7fb8bbfff640 1 Processor -- start 2023-11-28T20:54:45.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.898+0000 7fb8bbfff640 1 -- start start 2023-11-28T20:54:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.899+0000 7fb8bbfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb8bc0d0040 0x7fb8bc0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.899+0000 7fb8bbfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8bc0b1d40 0x7fb8bc0d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.899+0000 7fb8bbfff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc0b90a0 con 0x7fb8bc0b2290 2023-11-28T20:54:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.899+0000 7fb8bbfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8bc0d1920 con 0x7fb8bc0b1d40 2023-11-28T20:54:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.899+0000 7fb8bbfff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8bc0d2dc0 con 0x7fb8bc0d0040 2023-11-28T20:54:45.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.899+0000 7fb8bb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8bc0b1d40 0x7fb8bc0d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.899+0000 7fb8bb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8bc0b1d40 0x7fb8bc0d3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33654/0 (socket says 172.21.15.45:33654) 2023-11-28T20:54:45.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.899+0000 7fb8bb7fe640 1 -- 172.21.15.45:0/500587426 learned_addr learned my addr 172.21.15.45:0/500587426 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:45.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.900+0000 7fb8baffd640 1 --2- 172.21.15.45:0/500587426 >> v2:172.21.15.190:3300/0 conn(0x7fb8bc0d0040 0x7fb8bc0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.900+0000 7fb8bb7fe640 1 -- 172.21.15.45:0/500587426 >> v2:172.21.15.190:3300/0 conn(0x7fb8bc0d0040 msgr2=0x7fb8bc0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:45.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.900+0000 7fb8bb7fe640 1 --2- 172.21.15.45:0/500587426 >> v2:172.21.15.190:3300/0 conn(0x7fb8bc0d0040 0x7fb8bc0d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.900+0000 7fb8bb7fe640 1 -- 172.21.15.45:0/500587426 >> v1:172.21.15.190:6789/0 conn(0x7fb8bc0b2290 legacy=0x7fb8bc0d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:45.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.900+0000 7fb8bb7fe640 1 -- 172.21.15.45:0/500587426 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc0d0960 con 0x7fb8bc0b1d40 2023-11-28T20:54:45.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.901+0000 7fb8bb7fe640 1 --2- 172.21.15.45:0/500587426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0b1d40 0x7fb8bc0d3750 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc0b53c0 tx=0x7fb8b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efed4e23c99805c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:45.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.901+0000 7fb8b9ffb640 1 -- 172.21.15.45:0/500587426 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4030860 con 0x7fb8bc0b1d40 2023-11-28T20:54:45.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.901+0000 7fb8b9ffb640 1 -- 172.21.15.45:0/500587426 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b40309c0 con 0x7fb8bc0b1d40 2023-11-28T20:54:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.901+0000 7fb8b9ffb640 1 -- 172.21.15.45:0/500587426 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4030d50 con 0x7fb8bc0b1d40 2023-11-28T20:54:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.902+0000 7fb8bbfff640 1 -- 172.21.15.45:0/500587426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0b1d40 msgr2=0x7fb8bc0d3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.902+0000 7fb8bbfff640 1 --2- 172.21.15.45:0/500587426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0b1d40 0x7fb8bc0d3750 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc0b53c0 tx=0x7fb8b402b9a0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.902+0000 7fb8bbfff640 1 -- 172.21.15.45:0/500587426 shutdown_connections 2023-11-28T20:54:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.902+0000 7fb8bbfff640 1 --2- 172.21.15.45:0/500587426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0b1d40 0x7fb8bc0d3750 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.902+0000 7fb8bbfff640 1 --2- 172.21.15.45:0/500587426 >> v2:172.21.15.190:3300/0 conn(0x7fb8bc0d0040 0x7fb8bc0d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.902+0000 7fb8bbfff640 1 -- 172.21.15.45:0/500587426 >> 172.21.15.45:0/500587426 conn(0x7fb8bc070f40 msgr2=0x7fb8bc071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.902+0000 7fb8bbfff640 1 -- 172.21.15.45:0/500587426 shutdown_connections 2023-11-28T20:54:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.902+0000 7fb8bbfff640 1 -- 172.21.15.45:0/500587426 wait complete. 2023-11-28T20:54:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.904+0000 7fb8bbfff640 1 Processor -- start 2023-11-28T20:54:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.904+0000 7fb8bbfff640 1 -- start start 2023-11-28T20:54:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.905+0000 7fb8bbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8bc0b1d40 0x7fb8bc1f7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.905+0000 7fb8bbfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8bc0b2290 0x7fb8bc15b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.905+0000 7fb8bbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0d0040 0x7fb8bc0c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.905+0000 7fb8bbfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0c87f0 con 0x7fb8bc0d0040 2023-11-28T20:54:45.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.905+0000 7fb8bbfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0c8230 con 0x7fb8bc0b1d40 2023-11-28T20:54:45.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.905+0000 7fb8bbfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0c8510 con 0x7fb8bc0b2290 2023-11-28T20:54:45.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8bb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0d0040 0x7fb8bc0c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8bb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0d0040 0x7fb8bc0c7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33664/0 (socket says 172.21.15.45:33664) 2023-11-28T20:54:45.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8bb7fe640 1 -- 172.21.15.45:0/1093207591 learned_addr learned my addr 172.21.15.45:0/1093207591 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:45.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8baffd640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8bc0b1d40 0x7fb8bc1f7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8ba7fc640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8bc0b2290 0x7fb8bc15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8bb7fe640 1 -- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8bc0b2290 msgr2=0x7fb8bc15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:45.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8bb7fe640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8bc0b2290 0x7fb8bc15b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8bb7fe640 1 -- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8bc0b1d40 msgr2=0x7fb8bc1f7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8bb7fe640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8bc0b1d40 0x7fb8bc1f7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8bb7fe640 1 -- 172.21.15.45:0/1093207591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b40094d0 con 0x7fb8bc0d0040 2023-11-28T20:54:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.906+0000 7fb8bb7fe640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0d0040 0x7fb8bc0c7790 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fb8b402b9e0 tx=0x7fb8b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.907+0000 7fb8b37fe640 1 -- 172.21.15.45:0/1093207591 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4039d90 con 0x7fb8bc0d0040 2023-11-28T20:54:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.907+0000 7fb8b37fe640 1 -- 172.21.15.45:0/1093207591 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b40433f0 con 0x7fb8bc0d0040 2023-11-28T20:54:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.907+0000 7fb8b37fe640 1 -- 172.21.15.45:0/1093207591 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40437a0 con 0x7fb8bc0d0040 2023-11-28T20:54:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.907+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc06c7b0 con 0x7fb8bc0d0040 2023-11-28T20:54:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.907+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc0c9630 con 0x7fb8bc0d0040 2023-11-28T20:54:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.908+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc06ccd0 con 0x7fb8bc0d0040 2023-11-28T20:54:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.909+0000 7fb8b37fe640 1 -- 172.21.15.45:0/1093207591 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 73) v1 ==== 89407+0+0 (secure 0 0 0) 0x7fb8b404c3b0 con 0x7fb8bc0d0040 2023-11-28T20:54:45.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.910+0000 7fb8b37fe640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb89c06b040 0x7fb89c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:45.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.910+0000 7fb8b37fe640 1 -- 172.21.15.45:0/1093207591 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb8b40ac4e0 con 0x7fb8bc0d0040 2023-11-28T20:54:45.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.911+0000 7fb8baffd640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb89c06b040 0x7fb89c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.912+0000 7fb8baffd640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb89c06b040 0x7fb89c06d500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00a970 tx=0x7fb8ac008040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:45.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:45.915+0000 7fb8b37fe640 1 -- 172.21.15.45:0/1093207591 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8b4076f40 con 0x7fb8bc0d0040 2023-11-28T20:54:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.132+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8bc0d15b0 con 0x7fb89c06b040 2023-11-28T20:54:46.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.286+0000 7fb8b37fe640 1 -- 172.21.15.45:0/1093207591 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fb8ac00c050 con 0x7fb89c06b040 2023-11-28T20:54:46.288 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1/yearly 2023-11-28T20:54:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb89c06b040 msgr2=0x7fb89c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb89c06b040 0x7fb89c06d500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00a970 tx=0x7fb8ac008040 comp rx=0 tx=0).stop 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0d0040 msgr2=0x7fb8bc0c7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0d0040 0x7fb8bc0c7790 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fb8b402b9e0 tx=0x7fb8b4004730 comp rx=0 tx=0).stop 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 shutdown_connections 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb89c06b040 0x7fb89c06d500 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc0d0040 0x7fb8bc0c7790 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8bc0b2290 0x7fb8bc15b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 --2- 172.21.15.45:0/1093207591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8bc0b1d40 0x7fb8bc1f7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 >> 172.21.15.45:0/1093207591 conn(0x7fb8bc070f40 msgr2=0x7fb8bc0b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 shutdown_connections 2023-11-28T20:54:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.290+0000 7fb8bbfff640 1 -- 172.21.15.45:0/1093207591 wait complete. 2023-11-28T20:54:46.315 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2023-11-28T20:54:46.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:46.315 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2023-11-28T20:54:46.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2023-11-28T20:54:46.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.608+0000 7f675ad8a640 1 Processor -- start 2023-11-28T20:54:46.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.608+0000 7f675ad8a640 1 -- start start 2023-11-28T20:54:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.609+0000 7f675ad8a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f675415e840 0x7f675406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.609+0000 7f675ad8a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675406ac10 0x7f67540717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.610+0000 7f675ad8a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675415ecd0 con 0x7f675415f720 2023-11-28T20:54:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.610+0000 7f675ad8a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6754071ce0 con 0x7f675415e840 2023-11-28T20:54:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.610+0000 7f675ad8a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675415f290 con 0x7f675406ac10 2023-11-28T20:54:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.610+0000 7f6753fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f675415f720 0x7f675406a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46442/0 (socket says 172.21.15.45:46442) 2023-11-28T20:54:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.610+0000 7f6753fff640 1 -- 172.21.15.45:0/829720544 learned_addr learned my addr 172.21.15.45:0/829720544 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:46.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.610+0000 7f6758b26640 1 --2- 172.21.15.45:0/829720544 >> v2:172.21.15.190:3300/0 conn(0x7f675415e840 0x7f675406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:46.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.610+0000 7f6759327640 1 --2- 172.21.15.45:0/829720544 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675406ac10 0x7f67540717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:46.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.611+0000 7f6759327640 1 -- 172.21.15.45:0/829720544 >> v2:172.21.15.190:3300/0 conn(0x7f675415e840 msgr2=0x7f675406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:46.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.611+0000 7f6759327640 1 --2- 172.21.15.45:0/829720544 >> v2:172.21.15.190:3300/0 conn(0x7f675415e840 0x7f675406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.611+0000 7f6759327640 1 -- 172.21.15.45:0/829720544 >> v1:172.21.15.45:6789/0 conn(0x7f675415f720 legacy=0x7f675406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:46.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.611+0000 7f6759327640 1 -- 172.21.15.45:0/829720544 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675415fb60 con 0x7f675406ac10 2023-11-28T20:54:46.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.611+0000 7f6758b26640 1 --2- 172.21.15.45:0/829720544 >> v2:172.21.15.190:3300/0 conn(0x7f675415e840 0x7f675406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T20:54:46.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.613+0000 7f6759327640 1 --2- 172.21.15.45:0/829720544 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675406ac10 0x7f67540717a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f674c009d90 tx=0x7f674c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4ac8e7be6ce1ee5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:46.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.616+0000 7f67537fe640 1 -- 172.21.15.45:0/829720544 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c02cc60 con 0x7f675406ac10 2023-11-28T20:54:46.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.616+0000 7f67537fe640 1 -- 172.21.15.45:0/829720544 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c02cdc0 con 0x7f675406ac10 2023-11-28T20:54:46.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.616+0000 7f67537fe640 1 -- 172.21.15.45:0/829720544 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c034660 con 0x7f675406ac10 2023-11-28T20:54:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.617+0000 7f675ad8a640 1 -- 172.21.15.45:0/829720544 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675406ac10 msgr2=0x7f67540717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.617+0000 7f675ad8a640 1 --2- 172.21.15.45:0/829720544 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675406ac10 0x7f67540717a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f674c009d90 tx=0x7f674c02beb0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.617+0000 7f675ad8a640 1 -- 172.21.15.45:0/829720544 shutdown_connections 2023-11-28T20:54:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.617+0000 7f675ad8a640 1 --2- 172.21.15.45:0/829720544 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675406ac10 0x7f67540717a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.617+0000 7f675ad8a640 1 --2- 172.21.15.45:0/829720544 >> v2:172.21.15.190:3300/0 conn(0x7f675415e840 0x7f675406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.617+0000 7f675ad8a640 1 -- 172.21.15.45:0/829720544 >> 172.21.15.45:0/829720544 conn(0x7f67540ced60 msgr2=0x7f67540cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.617+0000 7f675ad8a640 1 -- 172.21.15.45:0/829720544 shutdown_connections 2023-11-28T20:54:46.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.617+0000 7f675ad8a640 1 -- 172.21.15.45:0/829720544 wait complete. 2023-11-28T20:54:46.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.619+0000 7f675ad8a640 1 Processor -- start 2023-11-28T20:54:46.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.619+0000 7f675ad8a640 1 -- start start 2023-11-28T20:54:46.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.620+0000 7f675ad8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675406ac10 0x7f675415d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:46.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.620+0000 7f675ad8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675415e840 0x7f675409a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:46.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.620+0000 7f675ad8a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675415f720 0x7f6754156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:46.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.620+0000 7f675ad8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675409a960 con 0x7f675406ac10 2023-11-28T20:54:46.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.620+0000 7f675ad8a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675409a6c0 con 0x7f675415e840 2023-11-28T20:54:46.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.620+0000 7f675ad8a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675415be60 con 0x7f675415f720 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.620+0000 7f6758b26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675406ac10 0x7f675415d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.620+0000 7f6758b26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675406ac10 0x7f675415d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33672/0 (socket says 172.21.15.45:33672) 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.620+0000 7f6758b26640 1 -- 172.21.15.45:0/127761618 learned_addr learned my addr 172.21.15.45:0/127761618 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6753fff640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675415e840 0x7f675409a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6759327640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675415f720 0x7f6754156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6758b26640 1 -- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675415f720 msgr2=0x7f6754156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6758b26640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675415f720 0x7f6754156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6758b26640 1 -- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675415e840 msgr2=0x7f675409a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6758b26640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675415e840 0x7f675409a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6758b26640 1 -- 172.21.15.45:0/127761618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6748009400 con 0x7f675406ac10 2023-11-28T20:54:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6759327640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f675415f720 0x7f6754156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6753fff640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675415e840 0x7f675409a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f6758b26640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675406ac10 0x7f675415d690 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f674800ebe0 tx=0x7f674800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.621+0000 7f67517fa640 1 -- 172.21.15.45:0/127761618 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748009760 con 0x7f675406ac10 2023-11-28T20:54:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.622+0000 7f67517fa640 1 -- 172.21.15.45:0/127761618 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674800cc30 con 0x7f675406ac10 2023-11-28T20:54:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.622+0000 7f67517fa640 1 -- 172.21.15.45:0/127761618 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674800eee0 con 0x7f675406ac10 2023-11-28T20:54:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.622+0000 7f675ad8a640 1 -- 172.21.15.45:0/127761618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c009a70 con 0x7f675406ac10 2023-11-28T20:54:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.622+0000 7f675ad8a640 1 -- 172.21.15.45:0/127761618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675415c420 con 0x7f675406ac10 2023-11-28T20:54:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.623+0000 7f675ad8a640 1 -- 172.21.15.45:0/127761618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67540733d0 con 0x7f675406ac10 2023-11-28T20:54:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.624+0000 7f67517fa640 1 -- 172.21.15.45:0/127761618 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 73) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f6748013470 con 0x7f675406ac10 2023-11-28T20:54:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.625+0000 7f67517fa640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f673806b040 0x7f673806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.626+0000 7f67517fa640 1 -- 172.21.15.45:0/127761618 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6748089eb0 con 0x7f675406ac10 2023-11-28T20:54:46.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.626+0000 7f6753fff640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f673806b040 0x7f673806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:46.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.627+0000 7f6753fff640 1 --2- 172.21.15.45:0/127761618 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f673806b040 0x7f673806d500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f675406d0e0 tx=0x7f6740009040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:46.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.630+0000 7f67517fa640 1 -- 172.21.15.45:0/127761618 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6748054ab0 con 0x7f675406ac10 2023-11-28T20:54:46.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.855+0000 7f675ad8a640 1 -- 172.21.15.45:0/127761618 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7f675415ef20 con 0x7f673806b040 2023-11-28T20:54:46.859 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:46.857+0000 7f65717d0640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-11-28T20:54:46.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:46.859+0000 7f67517fa640 1 -- 172.21.15.45:0/127761618 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7f6740001330 con 0x7f673806b040 2023-11-28T20:54:46.860 INFO:teuthology.orchestra.run.smithi045.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-11-28T20:54:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T20:54:46.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2023-11-28T20:54:47.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.100+0000 7f6d07fff640 1 Processor -- start 2023-11-28T20:54:47.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.101+0000 7f6d07fff640 1 -- start start 2023-11-28T20:54:47.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.102+0000 7f6d07fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d080d0040 0x7f6d080d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.102+0000 7f6d07fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d080b2290 0x7f6d080d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.102+0000 7f6d07fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6d080b1d40 0x7f6d080d3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.103+0000 7f6d07fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d080b90a0 con 0x7f6d080d0040 2023-11-28T20:54:47.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.103+0000 7f6d07fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d080d1920 con 0x7f6d080b1d40 2023-11-28T20:54:47.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.103+0000 7f6d07fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080d0960 con 0x7f6d080b2290 2023-11-28T20:54:47.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.103+0000 7f6d06ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d080d0040 0x7f6d080d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.103+0000 7f6d06ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d080d0040 0x7f6d080d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33682/0 (socket says 172.21.15.45:33682) 2023-11-28T20:54:47.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.103+0000 7f6d06ffd640 1 -- 172.21.15.45:0/744597017 learned_addr learned my addr 172.21.15.45:0/744597017 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:47.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.103+0000 7f6d077fe640 1 --2- 172.21.15.45:0/744597017 >> v2:172.21.15.190:3300/0 conn(0x7f6d080b1d40 0x7f6d080d3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.103+0000 7f6cfffff640 1 --2- 172.21.15.45:0/744597017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d080b2290 0x7f6d080d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.103+0000 7f6d06ffd640 1 -- 172.21.15.45:0/744597017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d080b2290 msgr2=0x7f6d080d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:47.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.104+0000 7f6d06ffd640 1 --2- 172.21.15.45:0/744597017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d080b2290 0x7f6d080d3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.104+0000 7f6d06ffd640 1 -- 172.21.15.45:0/744597017 >> v2:172.21.15.190:3300/0 conn(0x7f6d080b1d40 msgr2=0x7f6d080d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:47.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.104+0000 7f6d06ffd640 1 --2- 172.21.15.45:0/744597017 >> v2:172.21.15.190:3300/0 conn(0x7f6d080b1d40 0x7f6d080d3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.104+0000 7f6d06ffd640 1 -- 172.21.15.45:0/744597017 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d080b8ae0 con 0x7f6d080d0040 2023-11-28T20:54:47.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.104+0000 7f6cfffff640 1 --2- 172.21.15.45:0/744597017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d080b2290 0x7f6d080d3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.105+0000 7f6d06ffd640 1 --2- 172.21.15.45:0/744597017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d080d0040 0x7f6d080d0420 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6d080b5c70 tx=0x7f6cf802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9714fe696a43ed83 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.105+0000 7f6d067fc640 1 -- 172.21.15.45:0/744597017 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8030860 con 0x7f6d080d0040 2023-11-28T20:54:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.105+0000 7f6d067fc640 1 -- 172.21.15.45:0/744597017 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf80309c0 con 0x7f6d080d0040 2023-11-28T20:54:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.105+0000 7f6d067fc640 1 -- 172.21.15.45:0/744597017 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8030d50 con 0x7f6d080d0040 2023-11-28T20:54:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.106+0000 7f6d07fff640 1 -- 172.21.15.45:0/744597017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d080d0040 msgr2=0x7f6d080d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.106+0000 7f6d07fff640 1 --2- 172.21.15.45:0/744597017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d080d0040 0x7f6d080d0420 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6d080b5c70 tx=0x7f6cf802b9a0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.106+0000 7f6d07fff640 1 -- 172.21.15.45:0/744597017 shutdown_connections 2023-11-28T20:54:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.106+0000 7f6d07fff640 1 --2- 172.21.15.45:0/744597017 >> v2:172.21.15.190:3300/0 conn(0x7f6d080b1d40 0x7f6d080d3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.106+0000 7f6d07fff640 1 --2- 172.21.15.45:0/744597017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d080b2290 0x7f6d080d3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.106+0000 7f6d07fff640 1 --2- 172.21.15.45:0/744597017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d080d0040 0x7f6d080d0420 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.106+0000 7f6d07fff640 1 -- 172.21.15.45:0/744597017 >> 172.21.15.45:0/744597017 conn(0x7f6d08070f40 msgr2=0x7f6d08071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.106+0000 7f6d07fff640 1 -- 172.21.15.45:0/744597017 shutdown_connections 2023-11-28T20:54:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.106+0000 7f6d07fff640 1 -- 172.21.15.45:0/744597017 wait complete. 2023-11-28T20:54:47.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.108+0000 7f6d07fff640 1 Processor -- start 2023-11-28T20:54:47.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.108+0000 7f6d07fff640 1 -- start start 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.109+0000 7f6d07fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d080b1d40 0x7f6d08159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.109+0000 7f6d07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d080b2290 0x7f6d08156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.109+0000 7f6d07fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d080d0040 0x7f6d08157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.109+0000 7f6d07fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081f4d20 con 0x7f6d080b2290 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.109+0000 7f6d07fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081f5000 con 0x7f6d080b1d40 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.109+0000 7f6d07fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081f52e0 con 0x7f6d080d0040 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.109+0000 7f6cfffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d080b2290 0x7f6d08156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.109+0000 7f6cfffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d080b2290 0x7f6d08156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33690/0 (socket says 172.21.15.45:33690) 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.109+0000 7f6cfffff640 1 -- 172.21.15.45:0/290549684 learned_addr learned my addr 172.21.15.45:0/290549684 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6d06ffd640 1 --2- 172.21.15.45:0/290549684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d080b1d40 0x7f6d08159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6cfffff640 1 -- 172.21.15.45:0/290549684 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d080d0040 msgr2=0x7f6d08157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6cfffff640 1 --2- 172.21.15.45:0/290549684 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d080d0040 0x7f6d08157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6cfffff640 1 -- 172.21.15.45:0/290549684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d080b1d40 msgr2=0x7f6d08159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6cfffff640 1 --2- 172.21.15.45:0/290549684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d080b1d40 0x7f6d08159a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6cfffff640 1 -- 172.21.15.45:0/290549684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf80094d0 con 0x7f6d080b2290 2023-11-28T20:54:47.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6cfffff640 1 --2- 172.21.15.45:0/290549684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d080b2290 0x7f6d08156690 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000c7f0 tx=0x7f6cf000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6cff7fe640 1 -- 172.21.15.45:0/290549684 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0007a30 con 0x7f6d080b2290 2023-11-28T20:54:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6cff7fe640 1 -- 172.21.15.45:0/290549684 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf0007b90 con 0x7f6d080b2290 2023-11-28T20:54:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6cff7fe640 1 -- 172.21.15.45:0/290549684 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000ce40 con 0x7f6d080b2290 2023-11-28T20:54:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.110+0000 7f6d07fff640 1 -- 172.21.15.45:0/290549684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d081f55c0 con 0x7f6d080b2290 2023-11-28T20:54:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.111+0000 7f6d07fff640 1 -- 172.21.15.45:0/290549684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d08156bd0 con 0x7f6d080b2290 2023-11-28T20:54:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.112+0000 7f6d07fff640 1 -- 172.21.15.45:0/290549684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d081f5ae0 con 0x7f6d080b2290 2023-11-28T20:54:47.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.112+0000 7f6cff7fe640 1 -- 172.21.15.45:0/290549684 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 73) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f6cf00113f0 con 0x7f6d080b2290 2023-11-28T20:54:47.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.113+0000 7f6cff7fe640 1 --2- 172.21.15.45:0/290549684 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f6ce806b040 0x7f6ce806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.113+0000 7f6cff7fe640 1 -- 172.21.15.45:0/290549684 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6cf0089ec0 con 0x7f6d080b2290 2023-11-28T20:54:47.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.114+0000 7f6d06ffd640 1 --2- 172.21.15.45:0/290549684 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f6ce806b040 0x7f6ce806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.115+0000 7f6d06ffd640 1 --2- 172.21.15.45:0/290549684 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f6ce806b040 0x7f6ce806d500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8036000 tx=0x7f6cf803b040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.118+0000 7f6cff7fe640 1 -- 172.21.15.45:0/290549684 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d081f5ae0 con 0x7f6d080b2290 2023-11-28T20:54:47.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.335+0000 7f6d07fff640 1 -- 172.21.15.45:0/290549684 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d080d15b0 con 0x7f6ce806b040 2023-11-28T20:54:47.338 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:47.336+0000 7f65717d0640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-11-28T20:54:47.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.338+0000 7f6cff7fe640 1 -- 172.21.15.45:0/290549684 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6cf8030ca0 con 0x7f6ce806b040 2023-11-28T20:54:47.339 INFO:teuthology.orchestra.run.smithi045.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-11-28T20:54:47.360 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T20:54:47.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2023-11-28T20:54:47.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.597+0000 7f814f4e6640 1 Processor -- start 2023-11-28T20:54:47.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.597+0000 7f814f4e6640 1 -- start start 2023-11-28T20:54:47.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.599+0000 7f814f4e6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f81480ae480 0x7f81480717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.599+0000 7f814f4e6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81480708d0 con 0x7f81480adfa0 2023-11-28T20:54:47.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.599+0000 7f814f4e6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81480b3bb0 con 0x7f814815d4b0 2023-11-28T20:54:47.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.599+0000 7f814f4e6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81480b3e90 con 0x7f81480ae480 2023-11-28T20:54:47.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.600+0000 7f814dce3640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f81480adfa0 0x7f81480abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46444/0 (socket says 172.21.15.45:46444) 2023-11-28T20:54:47.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.600+0000 7f814ece5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f81480ae480 0x7f81480717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.600+0000 7f814dce3640 1 -- 172.21.15.45:0/1134816515 learned_addr learned my addr 172.21.15.45:0/1134816515 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:47.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.600+0000 7f814d4e2640 1 -- 172.21.15.45:0/1134816515 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243879829 0 0) 0x7f81480708d0 con 0x7f81480adfa0 2023-11-28T20:54:47.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.600+0000 7f814d4e2640 1 -- 172.21.15.45:0/1134816515 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8134003520 con 0x7f81480adfa0 2023-11-28T20:54:47.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.600+0000 7f814ece5640 1 -- 172.21.15.45:0/1134816515 >> v1:172.21.15.190:6789/0 conn(0x7f814815d4b0 legacy=0x7f8148070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:47.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.600+0000 7f814ece5640 1 -- 172.21.15.45:0/1134816515 >> v1:172.21.15.45:6789/0 conn(0x7f81480adfa0 legacy=0x7f81480abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:47.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.600+0000 7f814ece5640 1 -- 172.21.15.45:0/1134816515 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8148070280 con 0x7f81480ae480 2023-11-28T20:54:47.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.601+0000 7f814ece5640 1 --2- 172.21.15.45:0/1134816515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81480ae480 0x7f81480717e0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f8140009d90 tx=0x7f814002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63540bd4b88254f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:47.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.602+0000 7f814d4e2640 1 -- 172.21.15.45:0/1134816515 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814002cc60 con 0x7f81480ae480 2023-11-28T20:54:47.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.602+0000 7f814d4e2640 1 -- 172.21.15.45:0/1134816515 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814002cdc0 con 0x7f81480ae480 2023-11-28T20:54:47.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.602+0000 7f814d4e2640 1 -- 172.21.15.45:0/1134816515 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140034660 con 0x7f81480ae480 2023-11-28T20:54:47.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.602+0000 7f814f4e6640 1 -- 172.21.15.45:0/1134816515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81480ae480 msgr2=0x7f81480717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:47.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.602+0000 7f814f4e6640 1 --2- 172.21.15.45:0/1134816515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81480ae480 0x7f81480717e0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f8140009d90 tx=0x7f814002beb0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.602+0000 7f814f4e6640 1 -- 172.21.15.45:0/1134816515 shutdown_connections 2023-11-28T20:54:47.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.603+0000 7f814f4e6640 1 --2- 172.21.15.45:0/1134816515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81480ae480 0x7f81480717e0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.603+0000 7f814f4e6640 1 -- 172.21.15.45:0/1134816515 >> 172.21.15.45:0/1134816515 conn(0x7f81480b1050 msgr2=0x7f81480cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:47.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.603+0000 7f814f4e6640 1 -- 172.21.15.45:0/1134816515 shutdown_connections 2023-11-28T20:54:47.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.603+0000 7f814f4e6640 1 -- 172.21.15.45:0/1134816515 wait complete. 2023-11-28T20:54:47.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.604+0000 7f814f4e6640 1 Processor -- start 2023-11-28T20:54:47.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.605+0000 7f814f4e6640 1 -- start start 2023-11-28T20:54:47.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814f4e6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f81480adfa0 0x7f8148143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814f4e6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81480ae480 0x7f81480af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814f4e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f814815d4b0 0x7f81480ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814f4e6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81480af800 con 0x7f81480ae480 2023-11-28T20:54:47.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814f4e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814809b240 con 0x7f814815d4b0 2023-11-28T20:54:47.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814f4e6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814809b520 con 0x7f81480adfa0 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814dce3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81480ae480 0x7f81480af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814dce3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81480ae480 0x7f81480af2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33708/0 (socket says 172.21.15.45:33708) 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814dce3640 1 -- 172.21.15.45:0/1917201550 learned_addr learned my addr 172.21.15.45:0/1917201550 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814e4e4640 1 --2- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f81480adfa0 0x7f8148143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814ece5640 1 --2- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f814815d4b0 0x7f81480ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814dce3640 1 -- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f81480adfa0 msgr2=0x7f8148143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814dce3640 1 --2- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f81480adfa0 0x7f8148143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814dce3640 1 -- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f814815d4b0 msgr2=0x7f81480ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814dce3640 1 --2- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f814815d4b0 0x7f81480ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814dce3640 1 -- 172.21.15.45:0/1917201550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8140009a70 con 0x7f81480ae480 2023-11-28T20:54:47.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.606+0000 7f814ece5640 1 --2- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f814815d4b0 0x7f81480ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:47.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.607+0000 7f814dce3640 1 --2- 172.21.15.45:0/1917201550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81480ae480 0x7f81480af2c0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f813800c3e0 tx=0x7f813800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:47.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.607+0000 7f813effd640 1 -- 172.21.15.45:0/1917201550 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8138005ce0 con 0x7f81480ae480 2023-11-28T20:54:47.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.607+0000 7f813effd640 1 -- 172.21.15.45:0/1917201550 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8138005e40 con 0x7f81480ae480 2023-11-28T20:54:47.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.607+0000 7f813effd640 1 -- 172.21.15.45:0/1917201550 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81380072d0 con 0x7f81480ae480 2023-11-28T20:54:47.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.607+0000 7f814f4e6640 1 -- 172.21.15.45:0/1917201550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f814809b800 con 0x7f81480ae480 2023-11-28T20:54:47.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.607+0000 7f814f4e6640 1 -- 172.21.15.45:0/1917201550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814809ae60 con 0x7f81480ae480 2023-11-28T20:54:47.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.609+0000 7f814f4e6640 1 -- 172.21.15.45:0/1917201550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814809bd20 con 0x7f81480ae480 2023-11-28T20:54:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.609+0000 7f813effd640 1 -- 172.21.15.45:0/1917201550 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 73) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f81380074f0 con 0x7f81480ae480 2023-11-28T20:54:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.610+0000 7f813effd640 1 --2- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f812c06b040 0x7f812c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:47.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.610+0000 7f813effd640 1 -- 172.21.15.45:0/1917201550 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f81380890f0 con 0x7f81480ae480 2023-11-28T20:54:47.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.611+0000 7f814e4e4640 1 --2- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f812c06b040 0x7f812c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:47.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.612+0000 7f814e4e4640 1 --2- 172.21.15.45:0/1917201550 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f812c06b040 0x7f812c06d500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8144000f60 tx=0x7f8144009040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:47.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.615+0000 7f813effd640 1 -- 172.21.15.45:0/1917201550 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f814809bd20 con 0x7f81480ae480 2023-11-28T20:54:47.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.833+0000 7f814f4e6640 1 -- 172.21.15.45:0/1917201550 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8148071470 con 0x7f812c06b040 2023-11-28T20:54:47.836 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:47.834+0000 7f65717d0640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-11-28T20:54:47.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:47.836+0000 7f813effd640 1 -- 172.21.15.45:0/1917201550 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8144001c70 con 0x7f812c06b040 2023-11-28T20:54:47.837 INFO:teuthology.orchestra.run.smithi045.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-11-28T20:54:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T20:54:47.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2023-11-28T20:54:48.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.087+0000 7f02f0ae6640 1 Processor -- start 2023-11-28T20:54:48.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.087+0000 7f02f0ae6640 1 -- start start 2023-11-28T20:54:48.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.088+0000 7f02f0ae6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f02ec0b2290 0x7f02ec0d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.089+0000 7f02f0ae6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec0b90a0 con 0x7f02ec0d0040 2023-11-28T20:54:48.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.089+0000 7f02f0ae6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec0d1920 con 0x7f02ec0b1d40 2023-11-28T20:54:48.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.089+0000 7f02f0ae6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ec0b8ae0 con 0x7f02ec0b2290 2023-11-28T20:54:48.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.089+0000 7f02eb7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f02ec0d0040 0x7f02ec0d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46456/0 (socket says 172.21.15.45:46456) 2023-11-28T20:54:48.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.089+0000 7f02eb7fe640 1 -- 172.21.15.45:0/3554479898 learned_addr learned my addr 172.21.15.45:0/3554479898 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:48.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.089+0000 7f02eaffd640 1 --2- 172.21.15.45:0/3554479898 >> v2:172.21.15.190:3300/0 conn(0x7f02ec0b2290 0x7f02ec0d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.089+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177011914 0 0) 0x7f02ec0b90a0 con 0x7f02ec0d0040 2023-11-28T20:54:48.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.089+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d8003520 con 0x7f02ec0d0040 2023-11-28T20:54:48.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2706383652 0 0) 0x7f02d8003520 con 0x7f02ec0d0040 2023-11-28T20:54:48.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 >> v2:172.21.15.190:3300/0 conn(0x7f02ec0b2290 msgr2=0x7f02ec0d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:48.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 --2- 172.21.15.45:0/3554479898 >> v2:172.21.15.190:3300/0 conn(0x7f02ec0b2290 0x7f02ec0d1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 >> v1:172.21.15.190:6789/0 conn(0x7f02ec0b1d40 legacy=0x7f02ec0d22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec0b8dc0 con 0x7f02ec0d0040 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f02dc001670 con 0x7f02ec0d0040 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02eaffd640 1 --2- 172.21.15.45:0/3554479898 >> v2:172.21.15.190:3300/0 conn(0x7f02ec0b2290 0x7f02ec0d1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 >> v1:172.21.15.45:6789/0 conn(0x7f02ec0d0040 legacy=0x7f02ec0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8003970 0x7f02d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02d80047c0 0x7f02d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02d8005180 0x7f02d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec0b8ae0 con 0x7f02d80047c0 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d800c0b0 con 0x7f02d8003970 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d800c270 con 0x7f02d8005180 2023-11-28T20:54:48.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.090+0000 7f02eaffd640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02d80047c0 0x7f02d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.091+0000 7f02eb7fe640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8003970 0x7f02d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.091+0000 7f02ebfff640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02d8005180 0x7f02d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.091+0000 7f02eaffd640 1 -- 172.21.15.45:0/3554479898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02d8005180 msgr2=0x7f02d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.091+0000 7f02eaffd640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02d8005180 0x7f02d800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.091+0000 7f02eaffd640 1 -- 172.21.15.45:0/3554479898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8003970 msgr2=0x7f02d8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.091+0000 7f02eaffd640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8003970 0x7f02d8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.091+0000 7f02eaffd640 1 -- 172.21.15.45:0/3554479898 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d8003520 con 0x7f02d80047c0 2023-11-28T20:54:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.091+0000 7f02eb7fe640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d8003970 0x7f02d8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:48.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.091+0000 7f02ebfff640 1 -- 172.21.15.45:0/3554479898 reap_dead start 2023-11-28T20:54:48.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.092+0000 7f02eaffd640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02d80047c0 0x7f02d8004c40 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f02d40073e0 tx=0x7f02d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb904bac8e72f98a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:48.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.092+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d402cd40 con 0x7f02d80047c0 2023-11-28T20:54:48.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.092+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d4034440 con 0x7f02d80047c0 2023-11-28T20:54:48.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.092+0000 7f02ea7fc640 1 -- 172.21.15.45:0/3554479898 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d402cea0 con 0x7f02d80047c0 2023-11-28T20:54:48.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.093+0000 7f02f0ae6640 1 -- 172.21.15.45:0/3554479898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02d80047c0 msgr2=0x7f02d8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:48.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.093+0000 7f02f0ae6640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02d80047c0 0x7f02d8004c40 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f02d40073e0 tx=0x7f02d80057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.093+0000 7f02f0ae6640 1 -- 172.21.15.45:0/3554479898 shutdown_connections 2023-11-28T20:54:48.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.093+0000 7f02f0ae6640 1 --2- 172.21.15.45:0/3554479898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02d80047c0 0x7f02d8004c40 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.093+0000 7f02f0ae6640 1 -- 172.21.15.45:0/3554479898 >> 172.21.15.45:0/3554479898 conn(0x7f02ec070f40 msgr2=0x7f02ec071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:48.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.093+0000 7f02f0ae6640 1 -- 172.21.15.45:0/3554479898 shutdown_connections 2023-11-28T20:54:48.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.093+0000 7f02f0ae6640 1 -- 172.21.15.45:0/3554479898 wait complete. 2023-11-28T20:54:48.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.095+0000 7f02f0ae6640 1 Processor -- start 2023-11-28T20:54:48.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.095+0000 7f02f0ae6640 1 -- start start 2023-11-28T20:54:48.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.096+0000 7f02f0ae6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d80047c0 0x7f02ec0cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.096+0000 7f02f0ae6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec1e32a0 0x7f02ec1f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.096+0000 7f02f0ae6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02ec1f3250 0x7f02ec0ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.096+0000 7f02f0ae6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec0cc6a0 con 0x7f02ec1e32a0 2023-11-28T20:54:48.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.096+0000 7f02f0ae6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec0cba90 con 0x7f02d80047c0 2023-11-28T20:54:48.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.096+0000 7f02f0ae6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec0cbd70 con 0x7f02ec1f3250 2023-11-28T20:54:48.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.096+0000 7f02eaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec1e32a0 0x7f02ec1f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.096+0000 7f02eaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec1e32a0 0x7f02ec1f2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33726/0 (socket says 172.21.15.45:33726) 2023-11-28T20:54:48.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.096+0000 7f02eaffd640 1 -- 172.21.15.45:0/638324218 learned_addr learned my addr 172.21.15.45:0/638324218 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:48.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02eb7fe640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d80047c0 0x7f02ec0cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02ebfff640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02ec1f3250 0x7f02ec0ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02eaffd640 1 -- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02ec1f3250 msgr2=0x7f02ec0ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:48.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02eaffd640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02ec1f3250 0x7f02ec0ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02eaffd640 1 -- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d80047c0 msgr2=0x7f02ec0cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:48.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02eaffd640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d80047c0 0x7f02ec0cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02eaffd640 1 -- 172.21.15.45:0/638324218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec0cc050 con 0x7f02ec1e32a0 2023-11-28T20:54:48.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02eb7fe640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d80047c0 0x7f02ec0cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:48.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02ebfff640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02ec1f3250 0x7f02ec0ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:48.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.097+0000 7f02eaffd640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec1e32a0 0x7f02ec1f2d10 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f02d4037040 tx=0x7f02d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:48.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.098+0000 7f02e3fff640 1 -- 172.21.15.45:0/638324218 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4003b90 con 0x7f02ec1e32a0 2023-11-28T20:54:48.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.098+0000 7f02e3fff640 1 -- 172.21.15.45:0/638324218 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d4003cf0 con 0x7f02ec1e32a0 2023-11-28T20:54:48.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.098+0000 7f02e3fff640 1 -- 172.21.15.45:0/638324218 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4004240 con 0x7f02ec1e32a0 2023-11-28T20:54:48.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.098+0000 7f02f0ae6640 1 -- 172.21.15.45:0/638324218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ec0cc6a0 con 0x7f02ec1e32a0 2023-11-28T20:54:48.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.098+0000 7f02f0ae6640 1 -- 172.21.15.45:0/638324218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ec0cb660 con 0x7f02ec1e32a0 2023-11-28T20:54:48.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.099+0000 7f02f0ae6640 1 -- 172.21.15.45:0/638324218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ec1f3630 con 0x7f02ec1e32a0 2023-11-28T20:54:48.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.100+0000 7f02e3fff640 1 -- 172.21.15.45:0/638324218 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 73) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f02d40043a0 con 0x7f02ec1e32a0 2023-11-28T20:54:48.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.101+0000 7f02e3fff640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f02cc06b040 0x7f02cc06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.101+0000 7f02e3fff640 1 -- 172.21.15.45:0/638324218 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f02d40ac2b0 con 0x7f02ec1e32a0 2023-11-28T20:54:48.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.102+0000 7f02eb7fe640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f02cc06b040 0x7f02cc06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.103+0000 7f02eb7fe640 1 --2- 172.21.15.45:0/638324218 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f02cc06b040 0x7f02cc06d500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0cb0b0 tx=0x7f02dc009040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.105+0000 7f02e3fff640 1 -- 172.21.15.45:0/638324218 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02ec1f3630 con 0x7f02ec1e32a0 2023-11-28T20:54:48.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.321+0000 7f02f0ae6640 1 -- 172.21.15.45:0/638324218 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f02ec0b6250 con 0x7f02cc06b040 2023-11-28T20:54:48.324 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:48.322+0000 7f65717d0640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-11-28T20:54:48.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.324+0000 7f02e3fff640 1 -- 172.21.15.45:0/638324218 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f02dc00d040 con 0x7f02cc06b040 2023-11-28T20:54:48.325 INFO:teuthology.orchestra.run.smithi045.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-11-28T20:54:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T20:54:48.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2023-11-28T20:54:48.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.577+0000 7ff55fc48640 1 Processor -- start 2023-11-28T20:54:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.577+0000 7ff55fc48640 1 -- start start 2023-11-28T20:54:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.579+0000 7ff55fc48640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff558157c10 0x7ff55806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.579+0000 7ff55fc48640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55806a500 0x7ff55806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.579+0000 7ff55fc48640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55806ca60 con 0x7ff55815ddd0 2023-11-28T20:54:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.579+0000 7ff55fc48640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55815eb40 con 0x7ff558157c10 2023-11-28T20:54:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.579+0000 7ff55fc48640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff558157ff0 con 0x7ff55806a500 2023-11-28T20:54:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.579+0000 7ff55e1e5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff55815ddd0 0x7ff55815e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46468/0 (socket says 172.21.15.45:46468) 2023-11-28T20:54:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.579+0000 7ff55e1e5640 1 -- 172.21.15.45:0/491871911 learned_addr learned my addr 172.21.15.45:0/491871911 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141503290 0 0) 0x7ff55806ca60 con 0x7ff55815ddd0 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55d1e3640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55806a500 0x7ff55806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff544003520 con 0x7ff55815ddd0 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55d9e4640 1 --2- 172.21.15.45:0/491871911 >> v2:172.21.15.190:3300/0 conn(0x7ff558157c10 0x7ff55806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 118692955 0 0) 0x7ff544003520 con 0x7ff55815ddd0 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55806a500 msgr2=0x7ff55806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55806a500 0x7ff55806a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 >> v2:172.21.15.190:3300/0 conn(0x7ff558157c10 msgr2=0x7ff55806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 --2- 172.21.15.45:0/491871911 >> v2:172.21.15.190:3300/0 conn(0x7ff558157c10 0x7ff55806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55806aea0 con 0x7ff55815ddd0 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ff5500019f0 con 0x7ff55815ddd0 2023-11-28T20:54:48.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 >> v1:172.21.15.45:6789/0 conn(0x7ff55815ddd0 legacy=0x7ff55815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:48.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff544003970 0x7ff544003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5440042d0 0x7ff5440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55806a500 0x7ff544004db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55815eb40 con 0x7ff5440042d0 2023-11-28T20:54:48.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff558157ff0 con 0x7ff544003970 2023-11-28T20:54:48.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.580+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff544005570 con 0x7ff55806a500 2023-11-28T20:54:48.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.581+0000 7ff55d1e3640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5440042d0 0x7ff5440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.581+0000 7ff55d9e4640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff544003970 0x7ff544003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.581+0000 7ff55e1e5640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55806a500 0x7ff544004db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.581+0000 7ff55d1e3640 1 -- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55806a500 msgr2=0x7ff544004db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.581+0000 7ff55d1e3640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55806a500 0x7ff544004db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.581+0000 7ff55d1e3640 1 -- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff544003970 msgr2=0x7ff544003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.581+0000 7ff55d1e3640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff544003970 0x7ff544003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.581+0000 7ff55d1e3640 1 -- 172.21.15.45:0/491871911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544003520 con 0x7ff5440042d0 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.581+0000 7ff55d9e4640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff544003970 0x7ff544003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.582+0000 7ff55d1e3640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5440042d0 0x7ff5440047c0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7ff548009a90 tx=0x7ff5440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4521b1f43a00dd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.582+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54802c680 con 0x7ff5440042d0 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.582+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54802c7e0 con 0x7ff5440042d0 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.583+0000 7ff55c9e2640 1 -- 172.21.15.45:0/491871911 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54802cb70 con 0x7ff5440042d0 2023-11-28T20:54:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.583+0000 7ff55fc48640 1 -- 172.21.15.45:0/491871911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5440042d0 msgr2=0x7ff5440047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.583+0000 7ff55fc48640 1 --2- 172.21.15.45:0/491871911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5440042d0 0x7ff5440047c0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7ff548009a90 tx=0x7ff5440057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.583+0000 7ff55e1e5640 1 -- 172.21.15.45:0/491871911 reap_dead start 2023-11-28T20:54:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.583+0000 7ff55fc48640 1 -- 172.21.15.45:0/491871911 shutdown_connections 2023-11-28T20:54:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.583+0000 7ff55fc48640 1 -- 172.21.15.45:0/491871911 >> 172.21.15.45:0/491871911 conn(0x7ff5580ced60 msgr2=0x7ff5580cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.583+0000 7ff55fc48640 1 -- 172.21.15.45:0/491871911 shutdown_connections 2023-11-28T20:54:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.583+0000 7ff55fc48640 1 -- 172.21.15.45:0/491871911 wait complete. 2023-11-28T20:54:48.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.585+0000 7ff55fc48640 1 Processor -- start 2023-11-28T20:54:48.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.585+0000 7ff55fc48640 1 -- start start 2023-11-28T20:54:48.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.586+0000 7ff55fc48640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5440042d0 0x7ff558153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.586+0000 7ff55fc48640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580acef0 0x7ff5580ad370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.586+0000 7ff55fc48640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580cca90 0x7ff5580ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.586+0000 7ff55fc48640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580ad8b0 con 0x7ff5580acef0 2023-11-28T20:54:48.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.586+0000 7ff55fc48640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff558159ee0 con 0x7ff5580cca90 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.586+0000 7ff55fc48640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581592d0 con 0x7ff5440042d0 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.586+0000 7ff55d1e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580acef0 0x7ff5580ad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d1e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580acef0 0x7ff5580ad370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33742/0 (socket says 172.21.15.45:33742) 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d1e3640 1 -- 172.21.15.45:0/2573775366 learned_addr learned my addr 172.21.15.45:0/2573775366 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55e1e5640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580cca90 0x7ff5580ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d9e4640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5440042d0 0x7ff558153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d1e3640 1 -- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5440042d0 msgr2=0x7ff558153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d1e3640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5440042d0 0x7ff558153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d1e3640 1 -- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580cca90 msgr2=0x7ff5580ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d1e3640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580cca90 0x7ff5580ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d1e3640 1 -- 172.21.15.45:0/2573775366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5581595b0 con 0x7ff5580acef0 2023-11-28T20:54:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55e1e5640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5580cca90 0x7ff5580ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d9e4640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5440042d0 0x7ff558153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.587+0000 7ff55d1e3640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5580acef0 0x7ff5580ad370 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7ff54802c220 tx=0x7ff548004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.588+0000 7ff54e7fc640 1 -- 172.21.15.45:0/2573775366 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548003810 con 0x7ff5580acef0 2023-11-28T20:54:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.588+0000 7ff54e7fc640 1 -- 172.21.15.45:0/2573775366 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff548003970 con 0x7ff5580acef0 2023-11-28T20:54:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.588+0000 7ff54e7fc640 1 -- 172.21.15.45:0/2573775366 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548003d20 con 0x7ff5580acef0 2023-11-28T20:54:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.588+0000 7ff55fc48640 1 -- 172.21.15.45:0/2573775366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff558159890 con 0x7ff5580acef0 2023-11-28T20:54:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.588+0000 7ff55fc48640 1 -- 172.21.15.45:0/2573775366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff558158f30 con 0x7ff5580acef0 2023-11-28T20:54:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.589+0000 7ff55fc48640 1 -- 172.21.15.45:0/2573775366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5580733d0 con 0x7ff5580acef0 2023-11-28T20:54:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.590+0000 7ff54e7fc640 1 -- 172.21.15.45:0/2573775366 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 73) v1 ==== 89407+0+0 (secure 0 0 0) 0x7ff548034440 con 0x7ff5580acef0 2023-11-28T20:54:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.591+0000 7ff54e7fc640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7ff53c06b040 0x7ff53c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.591+0000 7ff54e7fc640 1 -- 172.21.15.45:0/2573775366 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff5480aba10 con 0x7ff5580acef0 2023-11-28T20:54:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.592+0000 7ff55d9e4640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7ff53c06b040 0x7ff53c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.593+0000 7ff55d9e4640 1 --2- 172.21.15.45:0/2573775366 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7ff53c06b040 0x7ff53c06d500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff554009b20 tx=0x7ff554008040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:48.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.596+0000 7ff54e7fc640 1 -- 172.21.15.45:0/2573775366 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5480767e0 con 0x7ff5580acef0 2023-11-28T20:54:48.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.814+0000 7ff55fc48640 1 -- 172.21.15.45:0/2573775366 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5580ad8b0 con 0x7ff53c06b040 2023-11-28T20:54:48.818 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:48.816+0000 7f65717d0640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-11-28T20:54:48.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:48.817+0000 7ff54e7fc640 1 -- 172.21.15.45:0/2573775366 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7ff5580ad8b0 con 0x7ff53c06b040 2023-11-28T20:54:48.819 INFO:teuthology.orchestra.run.smithi045.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-11-28T20:54:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T20:54:48.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2023-11-28T20:54:49.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.065+0000 7f08fc8c3640 1 Processor -- start 2023-11-28T20:54:49.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.065+0000 7f08fc8c3640 1 -- start start 2023-11-28T20:54:49.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.066+0000 7f08fc8c3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f08f80b1d40 0x7f08f80d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.066+0000 7f08fc8c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f80b90a0 con 0x7f08f80d0040 2023-11-28T20:54:49.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.067+0000 7f08fc8c3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f80d1920 con 0x7f08f80b2290 2023-11-28T20:54:49.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.067+0000 7f08fc8c3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f80b8ae0 con 0x7f08f80b1d40 2023-11-28T20:54:49.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.067+0000 7f08f77fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f08f80d0040 0x7f08f80d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46478/0 (socket says 172.21.15.45:46478) 2023-11-28T20:54:49.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.067+0000 7f08f77fe640 1 -- 172.21.15.45:0/2751756285 learned_addr learned my addr 172.21.15.45:0/2751756285 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:49.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.067+0000 7f08f7fff640 1 --2- 172.21.15.45:0/2751756285 >> v2:172.21.15.190:3300/0 conn(0x7f08f80b1d40 0x7f08f80d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.067+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421415646 0 0) 0x7f08f80b90a0 con 0x7f08f80d0040 2023-11-28T20:54:49.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.067+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e0003520 con 0x7f08f80d0040 2023-11-28T20:54:49.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 724289989 0 0) 0x7f08e0003520 con 0x7f08f80d0040 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 >> v2:172.21.15.190:3300/0 conn(0x7f08f80b1d40 msgr2=0x7f08f80d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 --2- 172.21.15.45:0/2751756285 >> v2:172.21.15.190:3300/0 conn(0x7f08f80b1d40 0x7f08f80d24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 >> v1:172.21.15.190:6789/0 conn(0x7f08f80b2290 legacy=0x7f08f80d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f80b8dc0 con 0x7f08f80d0040 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f08e80019f0 con 0x7f08f80d0040 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 >> v1:172.21.15.45:6789/0 conn(0x7f08f80d0040 legacy=0x7f08f80d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08e0003970 0x7f08e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 0x7f08e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e0005180 0x7f08e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f80b8ae0 con 0x7f08e00047c0 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e000c0b0 con 0x7f08e0005180 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e000c270 con 0x7f08e0003970 2023-11-28T20:54:49.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.068+0000 7f08f6ffd640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 0x7f08e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.069+0000 7f08f77fe640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08e0003970 0x7f08e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.069+0000 7f08f7fff640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e0005180 0x7f08e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.069+0000 7f08f6ffd640 1 -- 172.21.15.45:0/2751756285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08e0003970 msgr2=0x7f08e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:49.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.069+0000 7f08f6ffd640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08e0003970 0x7f08e0003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.069+0000 7f08f6ffd640 1 -- 172.21.15.45:0/2751756285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e0005180 msgr2=0x7f08e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:49.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.069+0000 7f08f6ffd640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e0005180 0x7f08e000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.069+0000 7f08f6ffd640 1 -- 172.21.15.45:0/2751756285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e0003520 con 0x7f08e00047c0 2023-11-28T20:54:49.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.069+0000 7f08f7fff640 1 -- 172.21.15.45:0/2751756285 reap_dead start 2023-11-28T20:54:49.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.070+0000 7f08f6ffd640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 0x7f08e0004c40 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f08ec001750 tx=0x7f08e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bbeae489869075 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:49.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.070+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec0052d0 con 0x7f08e00047c0 2023-11-28T20:54:49.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.070+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec02e050 con 0x7f08e00047c0 2023-11-28T20:54:49.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.070+0000 7f08f67fc640 1 -- 172.21.15.45:0/2751756285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec003830 con 0x7f08e00047c0 2023-11-28T20:54:49.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.071+0000 7f08fc8c3640 1 -- 172.21.15.45:0/2751756285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 msgr2=0x7f08e0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:49.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.071+0000 7f08fc8c3640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 0x7f08e0004c40 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f08ec001750 tx=0x7f08e00057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.071+0000 7f08fc8c3640 1 -- 172.21.15.45:0/2751756285 shutdown_connections 2023-11-28T20:54:49.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.071+0000 7f08fc8c3640 1 --2- 172.21.15.45:0/2751756285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 0x7f08e0004c40 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.071+0000 7f08fc8c3640 1 -- 172.21.15.45:0/2751756285 >> 172.21.15.45:0/2751756285 conn(0x7f08f8070f40 msgr2=0x7f08f8071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:49.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.071+0000 7f08fc8c3640 1 -- 172.21.15.45:0/2751756285 shutdown_connections 2023-11-28T20:54:49.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.071+0000 7f08fc8c3640 1 -- 172.21.15.45:0/2751756285 wait complete. 2023-11-28T20:54:49.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.073+0000 7f08fc8c3640 1 Processor -- start 2023-11-28T20:54:49.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.073+0000 7f08fc8c3640 1 -- start start 2023-11-28T20:54:49.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08fc8c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 0x7f08f81572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08fc8c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08f815a190 0x7f08f815a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08fc8c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08f815adb0 0x7f08f80cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08fc8c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f815b230 con 0x7f08e00047c0 2023-11-28T20:54:49.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08fc8c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f81579f0 con 0x7f08f815a190 2023-11-28T20:54:49.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08fc8c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f80cc1e0 con 0x7f08f815adb0 2023-11-28T20:54:49.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 0x7f08f81572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 0x7f08f81572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33762/0 (socket says 172.21.15.45:33762) 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08f77fe640 1 -- 172.21.15.45:0/2759065742 learned_addr learned my addr 172.21.15.45:0/2759065742 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08f77fe640 1 -- 172.21.15.45:0/2759065742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08f815adb0 msgr2=0x7f08f80cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08f77fe640 1 --2- 172.21.15.45:0/2759065742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08f815adb0 0x7f08f80cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08f77fe640 1 -- 172.21.15.45:0/2759065742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08f815a190 msgr2=0x7f08f815a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08f77fe640 1 --2- 172.21.15.45:0/2759065742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08f815a190 0x7f08f815a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.074+0000 7f08f77fe640 1 -- 172.21.15.45:0/2759065742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f80cb8b0 con 0x7f08e00047c0 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.075+0000 7f08f77fe640 1 --2- 172.21.15.45:0/2759065742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00047c0 0x7f08f81572f0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f08e800de80 tx=0x7f08e800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.075+0000 7f08dffff640 1 -- 172.21.15.45:0/2759065742 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e8004d60 con 0x7f08e00047c0 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.075+0000 7f08dffff640 1 -- 172.21.15.45:0/2759065742 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e800be20 con 0x7f08e00047c0 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.075+0000 7f08dffff640 1 -- 172.21.15.45:0/2759065742 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e8007310 con 0x7f08e00047c0 2023-11-28T20:54:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.075+0000 7f08fc8c3640 1 -- 172.21.15.45:0/2759065742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f80cbb90 con 0x7f08e00047c0 2023-11-28T20:54:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.075+0000 7f08fc8c3640 1 -- 172.21.15.45:0/2759065742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f815a5f0 con 0x7f08e00047c0 2023-11-28T20:54:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.076+0000 7f08fc8c3640 1 -- 172.21.15.45:0/2759065742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f8071b30 con 0x7f08e00047c0 2023-11-28T20:54:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.077+0000 7f08dffff640 1 -- 172.21.15.45:0/2759065742 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 73) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f08e8007470 con 0x7f08e00047c0 2023-11-28T20:54:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.078+0000 7f08dffff640 1 --2- 172.21.15.45:0/2759065742 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f08e406b070 0x7f08e406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.078+0000 7f08dffff640 1 -- 172.21.15.45:0/2759065742 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f08e8089cd0 con 0x7f08e00047c0 2023-11-28T20:54:49.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.079+0000 7f08f6ffd640 1 --2- 172.21.15.45:0/2759065742 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f08e406b070 0x7f08e406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.080+0000 7f08f6ffd640 1 --2- 172.21.15.45:0/2759065742 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f08e406b070 0x7f08e406d530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f08ec033000 tx=0x7f08ec02f000 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:49.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.083+0000 7f08dffff640 1 -- 172.21.15.45:0/2759065742 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08e8054740 con 0x7f08e00047c0 2023-11-28T20:54:49.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.303+0000 7f08fc8c3640 1 -- 172.21.15.45:0/2759065742 --> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f08f80b6250 con 0x7f08e406b070 2023-11-28T20:54:49.306 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:49.304+0000 7f65717d0640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-11-28T20:54:49.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.306+0000 7f08dffff640 1 -- 172.21.15.45:0/2759065742 <== mgr.7273 v2:172.21.15.190:6832/1149934668 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f08ec0052b0 con 0x7f08e406b070 2023-11-28T20:54:49.307 INFO:teuthology.orchestra.run.smithi045.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-11-28T20:54:49.321 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T20:54:49.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:49.321 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2023-11-28T20:54:49.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:49.386 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2023-11-28T20:54:49.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:49.450 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2023-11-28T20:54:49.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:49.512 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2023-11-28T20:54:49.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:49.568 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2023-11-28T20:54:49.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:49.622 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2023-11-28T20:54:49.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:49.681 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2023-11-28T20:54:49.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T20:54:49.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.972+0000 7f96d3e4f640 1 Processor -- start 2023-11-28T20:54:49.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.972+0000 7f96d3e4f640 1 -- start start 2023-11-28T20:54:49.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.974+0000 7f96d3e4f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96cc0ce600 0x7f96cc06a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.974+0000 7f96d3e4f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96cc06aa40 0x7f96cc0c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.974+0000 7f96d3e4f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc06e030 con 0x7f96cc06dc50 2023-11-28T20:54:49.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.974+0000 7f96d3e4f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc06aed0 con 0x7f96cc06aa40 2023-11-28T20:54:49.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.974+0000 7f96d3e4f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc0ccf60 con 0x7f96cc0ce600 2023-11-28T20:54:49.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.975+0000 7f96d23ec640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96cc06aa40 0x7f96cc0c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.975+0000 7f96d23ec640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96cc06aa40 0x7f96cc0c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33776/0 (socket says 172.21.15.45:33776) 2023-11-28T20:54:49.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.975+0000 7f96d23ec640 1 -- 172.21.15.45:0/2264612546 learned_addr learned my addr 172.21.15.45:0/2264612546 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:49.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.975+0000 7f96d13ea640 1 --2- 172.21.15.45:0/2264612546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96cc0ce600 0x7f96cc06a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.975+0000 7f96d23ec640 1 -- 172.21.15.45:0/2264612546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96cc0ce600 msgr2=0x7f96cc06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:49.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.975+0000 7f96d23ec640 1 --2- 172.21.15.45:0/2264612546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96cc0ce600 0x7f96cc06a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.975+0000 7f96d23ec640 1 -- 172.21.15.45:0/2264612546 >> v1:172.21.15.190:6789/0 conn(0x7f96cc06dc50 legacy=0x7f96cc0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:49.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.975+0000 7f96d23ec640 1 -- 172.21.15.45:0/2264612546 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc0cc930 con 0x7f96cc06aa40 2023-11-28T20:54:49.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.976+0000 7f96d23ec640 1 --2- 172.21.15.45:0/2264612546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 0x7f96cc0c93b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009e50 tx=0x7f96c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90793475c91fabf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:49.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d0be9640 1 -- 172.21.15.45:0/2264612546 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c402cb00 con 0x7f96cc06aa40 2023-11-28T20:54:49.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d0be9640 1 -- 172.21.15.45:0/2264612546 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c402cc60 con 0x7f96cc06aa40 2023-11-28T20:54:49.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d0be9640 1 -- 172.21.15.45:0/2264612546 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c40343f0 con 0x7f96cc06aa40 2023-11-28T20:54:49.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2264612546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 msgr2=0x7f96cc0c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:49.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d3e4f640 1 --2- 172.21.15.45:0/2264612546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 0x7f96cc0c93b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009e50 tx=0x7f96c402bf60 comp rx=0 tx=0).stop 2023-11-28T20:54:49.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2264612546 shutdown_connections 2023-11-28T20:54:49.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d3e4f640 1 --2- 172.21.15.45:0/2264612546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 0x7f96cc0c93b0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d3e4f640 1 --2- 172.21.15.45:0/2264612546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96cc0ce600 0x7f96cc06a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2264612546 >> 172.21.15.45:0/2264612546 conn(0x7f96cc0ced60 msgr2=0x7f96cc0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:49.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2264612546 shutdown_connections 2023-11-28T20:54:49.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.977+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2264612546 wait complete. 2023-11-28T20:54:49.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.979+0000 7f96d3e4f640 1 Processor -- start 2023-11-28T20:54:49.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.979+0000 7f96d3e4f640 1 -- start start 2023-11-28T20:54:49.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.980+0000 7f96d3e4f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 0x7f96cc0ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.980+0000 7f96d3e4f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96cc06dc50 0x7f96cc0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.980+0000 7f96d3e4f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96cc0ce600 0x7f96cc0ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.980+0000 7f96d3e4f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc0c6a90 con 0x7f96cc06aa40 2023-11-28T20:54:49.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.980+0000 7f96d3e4f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc0ad560 con 0x7f96cc06dc50 2023-11-28T20:54:49.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.980+0000 7f96d3e4f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc14d4e0 con 0x7f96cc0ce600 2023-11-28T20:54:49.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96d1beb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 0x7f96cc0ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96d1beb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 0x7f96cc0ad020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33792/0 (socket says 172.21.15.45:33792) 2023-11-28T20:54:49.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96d1beb640 1 -- 172.21.15.45:0/2004917742 learned_addr learned my addr 172.21.15.45:0/2004917742 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:49.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96d1beb640 1 -- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96cc0ce600 msgr2=0x7f96cc0ac2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:49.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96d1beb640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96cc0ce600 0x7f96cc0ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96d1beb640 1 -- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96cc06dc50 msgr2=0x7f96cc0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:49.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96d1beb640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96cc06dc50 0x7f96cc0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:49.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96d1beb640 1 -- 172.21.15.45:0/2004917742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c4009b30 con 0x7f96cc06aa40 2023-11-28T20:54:49.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96d1beb640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 0x7f96cc0ad020 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f96c800c450 tx=0x7f96c800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:49.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.981+0000 7f96be7fc640 1 -- 172.21.15.45:0/2004917742 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c80052d0 con 0x7f96cc06aa40 2023-11-28T20:54:49.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.982+0000 7f96be7fc640 1 -- 172.21.15.45:0/2004917742 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c8005430 con 0x7f96cc06aa40 2023-11-28T20:54:49.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.982+0000 7f96be7fc640 1 -- 172.21.15.45:0/2004917742 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c8012870 con 0x7f96cc06aa40 2023-11-28T20:54:49.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.982+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc14d7c0 con 0x7f96cc06aa40 2023-11-28T20:54:49.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.982+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc14dd80 con 0x7f96cc06aa40 2023-11-28T20:54:49.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.983+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc0733d0 con 0x7f96cc06aa40 2023-11-28T20:54:49.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.984+0000 7f96be7fc640 1 -- 172.21.15.45:0/2004917742 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 73) v1 ==== 89407+0+0 (secure 0 0 0) 0x7f96c80129d0 con 0x7f96cc06aa40 2023-11-28T20:54:49.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.985+0000 7f96be7fc640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f96b006b040 0x7f96b006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:49.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.985+0000 7f96be7fc640 1 -- 172.21.15.45:0/2004917742 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f96c80896a0 con 0x7f96cc06aa40 2023-11-28T20:54:49.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.986+0000 7f96d13ea640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f96b006b040 0x7f96b006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:49.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.987+0000 7f96d13ea640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f96b006b040 0x7f96b006d500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f96c0006fd0 tx=0x7f96c0008040 comp rx=0 tx=0).ready entity=mgr.7273 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:49.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:49.989+0000 7f96be7fc640 1 -- 172.21.15.45:0/2004917742 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96c8054170 con 0x7f96cc06aa40 2023-11-28T20:54:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.230+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f96cc06e030 con 0x7f96cc06aa40 2023-11-28T20:54:50.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.449+0000 7f96be7fc640 1 -- 172.21.15.45:0/2004917742 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v74) v1 ==== 93+0+0 (secure 0 0 0) 0x7f96cc06e030 con 0x7f96cc06aa40 2023-11-28T20:54:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.453+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f96b006b040 msgr2=0x7f96b006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.453+0000 7f96d3e4f640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f96b006b040 0x7f96b006d500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f96c0006fd0 tx=0x7f96c0008040 comp rx=0 tx=0).stop 2023-11-28T20:54:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.453+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 msgr2=0x7f96cc0ad020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.453+0000 7f96d3e4f640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 0x7f96cc0ad020 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f96c800c450 tx=0x7f96c800c910 comp rx=0 tx=0).stop 2023-11-28T20:54:50.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.454+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 shutdown_connections 2023-11-28T20:54:50.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.454+0000 7f96d3e4f640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f96b006b040 0x7f96b006d500 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:50.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.454+0000 7f96d3e4f640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96cc0ce600 0x7f96cc0ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:50.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.454+0000 7f96d3e4f640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96cc06dc50 0x7f96cc0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:50.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.454+0000 7f96d3e4f640 1 --2- 172.21.15.45:0/2004917742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc06aa40 0x7f96cc0ad020 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:50.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.454+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 >> 172.21.15.45:0/2004917742 conn(0x7f96cc0ced60 msgr2=0x7f96cc0cdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:50.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.454+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 shutdown_connections 2023-11-28T20:54:50.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:50.455+0000 7f96d3e4f640 1 -- 172.21.15.45:0/2004917742 wait complete. 2023-11-28T20:54:50.477 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:54:50.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:54:50.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:54:50.688 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:54:50.688 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:54:50.688 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:50.742 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:54:50.742 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:50.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:50.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:54:50.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:54:50.983 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:50.982+0000 7f5dd2c12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:54:50.991 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:50.989+0000 7f8936613040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:54:51.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.037+0000 7fb03e6f3640 1 Processor -- start 2023-11-28T20:54:51.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.037+0000 7fb03e6f3640 1 -- start start 2023-11-28T20:54:51.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.038+0000 7fb03e6f3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038146150 0x7fb03806a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.039+0000 7fb03e6f3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0380d1b50 con 0x7fb03806edf0 2023-11-28T20:54:51.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.039+0000 7fb03e6f3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb038146530 con 0x7fb03806aa40 2023-11-28T20:54:51.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.039+0000 7fb03e6f3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0380d0f40 con 0x7fb038146150 2023-11-28T20:54:51.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.039+0000 7fb03cc90640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb03806edf0 0x7fb0380c5ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46496/0 (socket says 172.21.15.45:46496) 2023-11-28T20:54:51.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.039+0000 7fb03cc90640 1 -- 172.21.15.45:0/3429463352 learned_addr learned my addr 172.21.15.45:0/3429463352 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:51.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.039+0000 7fb037fff640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038146150 0x7fb03806a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.039+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235492642 0 0) 0x7fb0380d1b50 con 0x7fb03806edf0 2023-11-28T20:54:51.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb024003520 con 0x7fb03806edf0 2023-11-28T20:54:51.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1183787203 0 0) 0x7fb024003520 con 0x7fb03806edf0 2023-11-28T20:54:51.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038146150 msgr2=0x7fb03806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:51.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038146150 0x7fb03806a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 >> v1:172.21.15.190:6789/0 conn(0x7fb03806aa40 legacy=0x7fb03806e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0380d1220 con 0x7fb03806edf0 2023-11-28T20:54:51.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb0300019f0 con 0x7fb03806edf0 2023-11-28T20:54:51.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 >> v1:172.21.15.45:6789/0 conn(0x7fb03806edf0 legacy=0x7fb0380c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.042 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:51.040+0000 7f5dd2c12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:54:51.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0240039a0 0x7fb024003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038146150 0x7fb0240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb024004d00 0x7fb024005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0380d0f40 con 0x7fb024004d00 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02400ba30 con 0x7fb0240039a0 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02400bbf0 con 0x7fb038146150 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.040+0000 7fb03cc90640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb024004d00 0x7fb024005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.041+0000 7fb037fff640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0240039a0 0x7fb024003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.041+0000 7fb02ffff640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038146150 0x7fb0240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.041+0000 7fb03cc90640 1 -- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038146150 msgr2=0x7fb0240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.041+0000 7fb03cc90640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038146150 0x7fb0240047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.041+0000 7fb03cc90640 1 -- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0240039a0 msgr2=0x7fb024003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:51.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.041+0000 7fb03cc90640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0240039a0 0x7fb024003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.041+0000 7fb03cc90640 1 -- 172.21.15.45:0/3429463352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb024003520 con 0x7fb024004d00 2023-11-28T20:54:51.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.041+0000 7fb02ffff640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038146150 0x7fb0240047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:51.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.042+0000 7fb03cc90640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb024004d00 0x7fb024005180 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fb03000a7a0 tx=0x7fb0240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aec130d95ff0fe9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:51.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.043+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb030034630 con 0x7fb024004d00 2023-11-28T20:54:51.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.043+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03002f070 con 0x7fb024004d00 2023-11-28T20:54:51.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.043+0000 7fb0377fe640 1 -- 172.21.15.45:0/3429463352 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0300349f0 con 0x7fb024004d00 2023-11-28T20:54:51.046 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.043+0000 7f8936613040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:54:51.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.043+0000 7fb03e6f3640 1 -- 172.21.15.45:0/3429463352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb024004d00 msgr2=0x7fb024005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.043+0000 7fb03e6f3640 1 --2- 172.21.15.45:0/3429463352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb024004d00 0x7fb024005180 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fb03000a7a0 tx=0x7fb0240057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.043+0000 7fb03cc90640 1 -- 172.21.15.45:0/3429463352 reap_dead start 2023-11-28T20:54:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.043+0000 7fb03e6f3640 1 -- 172.21.15.45:0/3429463352 shutdown_connections 2023-11-28T20:54:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.043+0000 7fb03e6f3640 1 -- 172.21.15.45:0/3429463352 >> 172.21.15.45:0/3429463352 conn(0x7fb03806c710 msgr2=0x7fb03806caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.044+0000 7fb03e6f3640 1 -- 172.21.15.45:0/3429463352 shutdown_connections 2023-11-28T20:54:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.044+0000 7fb03e6f3640 1 -- 172.21.15.45:0/3429463352 wait complete. 2023-11-28T20:54:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.045+0000 7fb03e6f3640 1 Processor -- start 2023-11-28T20:54:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.045+0000 7fb03e6f3640 1 -- start start 2023-11-28T20:54:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.046+0000 7fb03e6f3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038141aa0 0x7fb0380c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.046+0000 7fb03e6f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb038141e80 0x7fb0380af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.046+0000 7fb03e6f3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0380afe50 0x7fb0380b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.046+0000 7fb03e6f3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0380b02d0 con 0x7fb038141e80 2023-11-28T20:54:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.046+0000 7fb03e6f3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0380b62e0 con 0x7fb0380afe50 2023-11-28T20:54:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.046+0000 7fb03e6f3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0380c8610 con 0x7fb038141aa0 2023-11-28T20:54:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb02ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb038141e80 0x7fb0380af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb02ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb038141e80 0x7fb0380af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33828/0 (socket says 172.21.15.45:33828) 2023-11-28T20:54:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb02ffff640 1 -- 172.21.15.45:0/4245048578 learned_addr learned my addr 172.21.15.45:0/4245048578 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb037fff640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038141aa0 0x7fb0380c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb02ffff640 1 -- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038141aa0 msgr2=0x7fb0380c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb02ffff640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038141aa0 0x7fb0380c80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb02ffff640 1 -- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0380afe50 msgr2=0x7fb0380b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb03cc90640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0380afe50 0x7fb0380b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb02ffff640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0380afe50 0x7fb0380b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb02ffff640 1 -- 172.21.15.45:0/4245048578 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0380b5a40 con 0x7fb038141e80 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb03cc90640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0380afe50 0x7fb0380b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.047+0000 7fb02ffff640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb038141e80 0x7fb0380af910 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fb02000d7c0 tx=0x7fb02000dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.048+0000 7fb0357fa640 1 -- 172.21.15.45:0/4245048578 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020017070 con 0x7fb038141e80 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.048+0000 7fb0357fa640 1 -- 172.21.15.45:0/4245048578 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02000de70 con 0x7fb038141e80 2023-11-28T20:54:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.048+0000 7fb0357fa640 1 -- 172.21.15.45:0/4245048578 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020005480 con 0x7fb038141e80 2023-11-28T20:54:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.048+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0380b5d20 con 0x7fb038141e80 2023-11-28T20:54:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.048+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0381f2ed0 con 0x7fb038141e80 2023-11-28T20:54:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.049+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0380ce5d0 con 0x7fb038141e80 2023-11-28T20:54:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.050+0000 7fb0357fa640 1 -- 172.21.15.45:0/4245048578 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 74) v1 ==== 89390+0+0 (secure 0 0 0) 0x7fb0200067d0 con 0x7fb038141e80 2023-11-28T20:54:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.051+0000 7fb0357fa640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb01806b020 0x7fb01806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.051+0000 7fb0357fa640 1 -- 172.21.15.45:0/4245048578 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb0200896c0 con 0x7fb038141e80 2023-11-28T20:54:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.052+0000 7fb037fff640 1 -- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb01806b020 msgr2=0x7fb01806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.052+0000 7fb037fff640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb01806b020 0x7fb01806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.056+0000 7fb0357fa640 1 -- 172.21.15.45:0/4245048578 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0380ce5d0 con 0x7fb038141e80 2023-11-28T20:54:51.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.247+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0380ce5d0 con 0x7fb038141e80 2023-11-28T20:54:51.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.248+0000 7fb0357fa640 1 -- 172.21.15.45:0/4245048578 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+17324 (secure 0 0 0) 0x7fb0380ce5d0 con 0x7fb038141e80 2023-11-28T20:54:51.250 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:51.250 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":149,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:41.552133+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:54:24.325054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-11-28T20:54:25.374707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1886852041":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/649548475":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/1625344948":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2599111328":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2598605134":"2023-11-29T20:54:41.552066+0000","172.21.15.190:6833/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.190:6832/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/1527814408":"2023-11-29T20:54:41.552066+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:51.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb01806b020 msgr2=0x7fb01806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:51.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb01806b020 0x7fb01806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb038141e80 msgr2=0x7fb0380af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb038141e80 0x7fb0380af910 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fb02000d7c0 tx=0x7fb02000dc80 comp rx=0 tx=0).stop 2023-11-28T20:54:51.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 shutdown_connections 2023-11-28T20:54:51.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb01806b020 0x7fb01806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0380afe50 0x7fb0380b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb038141e80 0x7fb0380af910 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 --2- 172.21.15.45:0/4245048578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb038141aa0 0x7fb0380c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 >> 172.21.15.45:0/4245048578 conn(0x7fb03806c710 msgr2=0x7fb0380b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:51.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 shutdown_connections 2023-11-28T20:54:51.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.251+0000 7fb03e6f3640 1 -- 172.21.15.45:0/4245048578 wait complete. 2023-11-28T20:54:51.273 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:54:24.325054+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:54:25.374707+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:54:51.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:54:51.492 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.489+0000 7f8936613040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:54:51.529 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:51.527+0000 7f5dd2c12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:54:51.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.533+0000 7f081d51a640 1 Processor -- start 2023-11-28T20:54:51.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.533+0000 7f081d51a640 1 -- start start 2023-11-28T20:54:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.535+0000 7f081d51a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180b2290 0x7f08180d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.535+0000 7f081d51a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f08180b1d40 0x7f08180d22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.535+0000 7f081d51a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08180b90a0 con 0x7f08180d0040 2023-11-28T20:54:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.535+0000 7f081d51a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08180d1920 con 0x7f08180b1d40 2023-11-28T20:54:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.535+0000 7f081d51a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08180b8ae0 con 0x7f08180b2290 2023-11-28T20:54:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.535+0000 7f081cd19640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f08180b1d40 0x7f08180d22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.535+0000 7f0817fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f08180d0040 0x7f08180d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46510/0 (socket says 172.21.15.45:46510) 2023-11-28T20:54:51.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.535+0000 7f0817fff640 1 -- 172.21.15.45:0/2434362060 learned_addr learned my addr 172.21.15.45:0/2434362060 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:51.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.535+0000 7f080ffff640 1 --2- 172.21.15.45:0/2434362060 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180b2290 0x7f08180d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.536+0000 7f081cd19640 1 -- 172.21.15.45:0/2434362060 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180b2290 msgr2=0x7f08180d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:51.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.536+0000 7f081cd19640 1 --2- 172.21.15.45:0/2434362060 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180b2290 0x7f08180d1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.536+0000 7f081cd19640 1 -- 172.21.15.45:0/2434362060 >> v1:172.21.15.45:6789/0 conn(0x7f08180d0040 legacy=0x7f08180d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.536+0000 7f081cd19640 1 -- 172.21.15.45:0/2434362060 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08180b8dc0 con 0x7f08180b1d40 2023-11-28T20:54:51.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.536+0000 7f08177fe640 1 -- 172.21.15.45:0/2434362060 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289763661 0 0) 0x7f08180b90a0 con 0x7f08180d0040 2023-11-28T20:54:51.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.537+0000 7f081cd19640 1 --2- 172.21.15.45:0/2434362060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b1d40 0x7f08180d22f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0810009d90 tx=0x7f081002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2159d16d9a2237a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.537+0000 7f08177fe640 1 -- 172.21.15.45:0/2434362060 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081002cce0 con 0x7f08180b1d40 2023-11-28T20:54:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.537+0000 7f08177fe640 1 -- 172.21.15.45:0/2434362060 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081002ce40 con 0x7f08180b1d40 2023-11-28T20:54:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.537+0000 7f08177fe640 1 -- 172.21.15.45:0/2434362060 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810034660 con 0x7f08180b1d40 2023-11-28T20:54:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.538+0000 7f081d51a640 1 -- 172.21.15.45:0/2434362060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b1d40 msgr2=0x7f08180d22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.538+0000 7f081d51a640 1 --2- 172.21.15.45:0/2434362060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b1d40 0x7f08180d22f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0810009d90 tx=0x7f081002bf30 comp rx=0 tx=0).stop 2023-11-28T20:54:51.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.538+0000 7f081d51a640 1 -- 172.21.15.45:0/2434362060 shutdown_connections 2023-11-28T20:54:51.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.538+0000 7f081d51a640 1 --2- 172.21.15.45:0/2434362060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b1d40 0x7f08180d22f0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.538+0000 7f081d51a640 1 --2- 172.21.15.45:0/2434362060 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180b2290 0x7f08180d1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.538+0000 7f081d51a640 1 -- 172.21.15.45:0/2434362060 >> 172.21.15.45:0/2434362060 conn(0x7f0818070f40 msgr2=0x7f0818071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:51.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.538+0000 7f081d51a640 1 -- 172.21.15.45:0/2434362060 shutdown_connections 2023-11-28T20:54:51.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.538+0000 7f081d51a640 1 -- 172.21.15.45:0/2434362060 wait complete. 2023-11-28T20:54:51.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.540+0000 7f081d51a640 1 Processor -- start 2023-11-28T20:54:51.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.540+0000 7f081d51a640 1 -- start start 2023-11-28T20:54:51.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.541+0000 7f081d51a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08180b1d40 0x7f08180b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.541+0000 7f081d51a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b2290 0x7f08180c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.541+0000 7f081d51a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180d0040 0x7f08180c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.541+0000 7f081d51a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08180c94e0 con 0x7f08180b2290 2023-11-28T20:54:51.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.541+0000 7f081d51a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081814ca50 con 0x7f08180b1d40 2023-11-28T20:54:51.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.541+0000 7f081d51a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08180b8000 con 0x7f08180d0040 2023-11-28T20:54:51.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.541+0000 7f080ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b2290 0x7f08180c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.541+0000 7f080ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b2290 0x7f08180c7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33852/0 (socket says 172.21.15.45:33852) 2023-11-28T20:54:51.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.541+0000 7f080ffff640 1 -- 172.21.15.45:0/1328005921 learned_addr learned my addr 172.21.15.45:0/1328005921 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:51.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.542+0000 7f0817fff640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08180b1d40 0x7f08180b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.542+0000 7f081cd19640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180d0040 0x7f08180c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.542+0000 7f080ffff640 1 -- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180d0040 msgr2=0x7f08180c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.542+0000 7f080ffff640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180d0040 0x7f08180c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.542+0000 7f080ffff640 1 -- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08180b1d40 msgr2=0x7f08180b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.542+0000 7f080ffff640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08180b1d40 0x7f08180b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.542+0000 7f080ffff640 1 -- 172.21.15.45:0/1328005921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0810009a70 con 0x7f08180b2290 2023-11-28T20:54:51.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.542+0000 7f080ffff640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b2290 0x7f08180c7790 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f080000d800 tx=0x7f080000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:51.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.543+0000 7f08157fa640 1 -- 172.21.15.45:0/1328005921 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080000b6c0 con 0x7f08180b2290 2023-11-28T20:54:51.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.543+0000 7f08157fa640 1 -- 172.21.15.45:0/1328005921 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080000b820 con 0x7f08180b2290 2023-11-28T20:54:51.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.543+0000 7f08157fa640 1 -- 172.21.15.45:0/1328005921 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080000bbb0 con 0x7f08180b2290 2023-11-28T20:54:51.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.543+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08180c7cd0 con 0x7f08180b2290 2023-11-28T20:54:51.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.543+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08180b76d0 con 0x7f08180b2290 2023-11-28T20:54:51.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.544+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08180b8170 con 0x7f08180b2290 2023-11-28T20:54:51.553 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.542+0000 7f8936613040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:54:51.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.545+0000 7f08157fa640 1 -- 172.21.15.45:0/1328005921 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 74) v1 ==== 89390+0+0 (secure 0 0 0) 0x7f080000bd10 con 0x7f08180b2290 2023-11-28T20:54:51.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.546+0000 7f08157fa640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f07f806aff0 0x7f07f806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.546+0000 7f08157fa640 1 -- 172.21.15.45:0/1328005921 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f080008a0b0 con 0x7f08180b2290 2023-11-28T20:54:51.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.546+0000 7f0817fff640 1 -- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f07f806aff0 msgr2=0x7f07f806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:51.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.546+0000 7f0817fff640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f07f806aff0 0x7f07f806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:51.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.550+0000 7f08157fa640 1 -- 172.21.15.45:0/1328005921 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0800054c20 con 0x7f08180b2290 2023-11-28T20:54:51.587 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:51.585+0000 7f5dd2c12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:54:51.595 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.593+0000 7f8936613040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:54:51.641 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:51.639+0000 7f5dd2c12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:54:51.655 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.653+0000 7f8936613040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:54:51.703 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.701+0000 7f8936613040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:54:51.706 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:51.704+0000 7f5dd2c12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:54:51.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.744+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0818071b30 con 0x7f08180b2290 2023-11-28T20:54:51.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.745+0000 7f08157fa640 1 -- 172.21.15.45:0/1328005921 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0800058240 con 0x7f08180b2290 2023-11-28T20:54:51.746 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:54:51.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f0817fff640 1 -- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f07f806aff0 msgr2=0x7f07f806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:51.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f0817fff640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f07f806aff0 0x7f07f806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:51.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f07f806aff0 msgr2=0x7f07f806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:51.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f081d51a640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f07f806aff0 0x7f07f806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b2290 msgr2=0x7f08180c7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f081d51a640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b2290 0x7f08180c7790 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f080000d800 tx=0x7f080000dcc0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 shutdown_connections 2023-11-28T20:54:51.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f081d51a640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f07f806aff0 0x7f07f806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f081d51a640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f08180d0040 0x7f08180c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f081d51a640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08180b2290 0x7f08180c7790 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.747+0000 7f081d51a640 1 --2- 172.21.15.45:0/1328005921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08180b1d40 0x7f08180b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.748+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 >> 172.21.15.45:0/1328005921 conn(0x7f0818070f40 msgr2=0x7f08180d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:51.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.748+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 shutdown_connections 2023-11-28T20:54:51.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.748+0000 7f081d51a640 1 -- 172.21.15.45:0/1328005921 wait complete. 2023-11-28T20:54:51.759 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:51.758+0000 7f5dd2c12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:54:51.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:54:51.813 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.811+0000 7f8936613040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:54:51.875 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.873+0000 7f8936613040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:51.881 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:51.880+0000 7f5dd2c12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:54:51.924 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.921+0000 7f8936613040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:54:51.950 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:51.949+0000 7f5dd2c12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:51.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.978+0000 7f845dca3640 1 Processor -- start 2023-11-28T20:54:51.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.979+0000 7f845dca3640 1 -- start start 2023-11-28T20:54:51.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.980+0000 7f845dca3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f84580d0040 0x7f84580d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.980+0000 7f845dca3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84580b1d40 0x7f84580d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.980+0000 7f845dca3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84580b90a0 con 0x7f84580b2290 2023-11-28T20:54:51.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.980+0000 7f845dca3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84580d1920 con 0x7f84580b1d40 2023-11-28T20:54:51.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.980+0000 7f845dca3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84580d2dc0 con 0x7f84580d0040 2023-11-28T20:54:51.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f845d4a2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84580b1d40 0x7f84580d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f845cca1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f84580d0040 0x7f84580d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f845d4a2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84580b1d40 0x7f84580d3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33858/0 (socket says 172.21.15.45:33858) 2023-11-28T20:54:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f845cca1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f84580d0040 0x7f84580d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:58386/0 (socket says 172.21.15.45:58386) 2023-11-28T20:54:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f845d4a2640 1 -- 172.21.15.45:0/2749043134 learned_addr learned my addr 172.21.15.45:0/2749043134 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f8457fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f84580b2290 0x7f84580d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46522/0 (socket says 172.21.15.45:46522) 2023-11-28T20:54:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f845d4a2640 1 -- 172.21.15.45:0/2749043134 >> v2:172.21.15.190:3300/0 conn(0x7f84580d0040 msgr2=0x7f84580d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:51.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f845d4a2640 1 --2- 172.21.15.45:0/2749043134 >> v2:172.21.15.190:3300/0 conn(0x7f84580d0040 0x7f84580d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f845d4a2640 1 -- 172.21.15.45:0/2749043134 >> v1:172.21.15.45:6789/0 conn(0x7f84580b2290 legacy=0x7f84580d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f845d4a2640 1 -- 172.21.15.45:0/2749043134 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84580d0960 con 0x7f84580b1d40 2023-11-28T20:54:51.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.981+0000 7f84577fe640 1 -- 172.21.15.45:0/2749043134 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852241834 0 0) 0x7f84580b90a0 con 0x7f84580b2290 2023-11-28T20:54:51.986 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:51.980+0000 7f8936613040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:54:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.982+0000 7f845d4a2640 1 --2- 172.21.15.45:0/2749043134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b1d40 0x7f84580d3750 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f84580b53c0 tx=0x7f845002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb4172d51eb5dd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.983+0000 7f84577fe640 1 -- 172.21.15.45:0/2749043134 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845002cbb0 con 0x7f84580b1d40 2023-11-28T20:54:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.983+0000 7f84577fe640 1 -- 172.21.15.45:0/2749043134 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845002cd10 con 0x7f84580b1d40 2023-11-28T20:54:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.983+0000 7f84577fe640 1 -- 172.21.15.45:0/2749043134 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8450034610 con 0x7f84580b1d40 2023-11-28T20:54:51.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.983+0000 7f845dca3640 1 -- 172.21.15.45:0/2749043134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b1d40 msgr2=0x7f84580d3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.983+0000 7f845dca3640 1 --2- 172.21.15.45:0/2749043134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b1d40 0x7f84580d3750 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f84580b53c0 tx=0x7f845002c010 comp rx=0 tx=0).stop 2023-11-28T20:54:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.984+0000 7f845dca3640 1 -- 172.21.15.45:0/2749043134 shutdown_connections 2023-11-28T20:54:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.984+0000 7f845dca3640 1 --2- 172.21.15.45:0/2749043134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b1d40 0x7f84580d3750 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.984+0000 7f845dca3640 1 --2- 172.21.15.45:0/2749043134 >> v2:172.21.15.190:3300/0 conn(0x7f84580d0040 0x7f84580d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.984+0000 7f845dca3640 1 -- 172.21.15.45:0/2749043134 >> 172.21.15.45:0/2749043134 conn(0x7f8458070f40 msgr2=0x7f8458071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.984+0000 7f845dca3640 1 -- 172.21.15.45:0/2749043134 shutdown_connections 2023-11-28T20:54:51.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.984+0000 7f845dca3640 1 -- 172.21.15.45:0/2749043134 wait complete. 2023-11-28T20:54:51.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.986+0000 7f845dca3640 1 Processor -- start 2023-11-28T20:54:51.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.986+0000 7f845dca3640 1 -- start start 2023-11-28T20:54:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.987+0000 7f845dca3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84580b1d40 0x7f845815b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.987+0000 7f845dca3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b2290 0x7f84580c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.987+0000 7f845dca3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84580d0040 0x7f84580c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.987+0000 7f845dca3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84580c72f0 con 0x7f84580b2290 2023-11-28T20:54:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.987+0000 7f845dca3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84580c75d0 con 0x7f84580b1d40 2023-11-28T20:54:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.987+0000 7f845dca3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84580c78b0 con 0x7f84580d0040 2023-11-28T20:54:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f8457fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b2290 0x7f84580c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f8457fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b2290 0x7f84580c80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33862/0 (socket says 172.21.15.45:33862) 2023-11-28T20:54:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f8457fff640 1 -- 172.21.15.45:0/1513711079 learned_addr learned my addr 172.21.15.45:0/1513711079 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f845cca1640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84580b1d40 0x7f845815b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f845d4a2640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84580d0040 0x7f84580c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f8457fff640 1 -- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84580d0040 msgr2=0x7f84580c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f8457fff640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84580d0040 0x7f84580c8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f8457fff640 1 -- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84580b1d40 msgr2=0x7f845815b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f8457fff640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84580b1d40 0x7f845815b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f8457fff640 1 -- 172.21.15.45:0/1513711079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450009b30 con 0x7f84580b2290 2023-11-28T20:54:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.988+0000 7f845d4a2640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84580d0040 0x7f84580c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.989+0000 7f8457fff640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b2290 0x7f84580c80b0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f844400c4d0 tx=0x7f844400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.989+0000 7f84557fa640 1 -- 172.21.15.45:0/1513711079 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84440074e0 con 0x7f84580b2290 2023-11-28T20:54:51.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.989+0000 7f84557fa640 1 -- 172.21.15.45:0/1513711079 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444007660 con 0x7f84580b2290 2023-11-28T20:54:51.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.989+0000 7f84557fa640 1 -- 172.21.15.45:0/1513711079 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444004a20 con 0x7f84580b2290 2023-11-28T20:54:51.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.989+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84580c7b90 con 0x7f84580b2290 2023-11-28T20:54:51.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.990+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84581e4650 con 0x7f84580b2290 2023-11-28T20:54:51.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.991+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84580c7d80 con 0x7f84580b2290 2023-11-28T20:54:51.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.993+0000 7f84557fa640 1 -- 172.21.15.45:0/1513711079 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 74) v1 ==== 89390+0+0 (secure 0 0 0) 0x7f8444004b80 con 0x7f84580b2290 2023-11-28T20:54:51.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.994+0000 7f84557fa640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f843c06b040 0x7f843c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:51.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.994+0000 7f84557fa640 1 -- 172.21.15.45:0/1513711079 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f8444089220 con 0x7f84580b2290 2023-11-28T20:54:51.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.994+0000 7f845cca1640 1 -- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f843c06b040 msgr2=0x7f843c06d500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:51.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.994+0000 7f845cca1640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f843c06b040 0x7f843c06d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:51.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:51.998+0000 7f84557fa640 1 -- 172.21.15.45:0/1513711079 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8444053e20 con 0x7f84580b2290 2023-11-28T20:54:52.006 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:52.006+0000 7f5dd2c12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:54:52.033 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:52.031+0000 7f8936613040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:54:52.068 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:52.067+0000 7f5dd2c12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:54:52.122 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:52.121+0000 7f5dd2c12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:54:52.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.193+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84580c7430 con 0x7f84580b2290 2023-11-28T20:54:52.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.194+0000 7f84557fa640 1 -- 172.21.15.45:0/1513711079 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8444018020 con 0x7f84580b2290 2023-11-28T20:54:52.195 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:54:52.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.195+0000 7f845cca1640 1 -- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f843c06b040 msgr2=0x7f843c06d500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:52.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.195+0000 7f845cca1640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f843c06b040 0x7f843c06d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:52.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f843c06b040 msgr2=0x7f843c06d500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:52.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f843c06b040 0x7f843c06d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b2290 msgr2=0x7f84580c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b2290 0x7f84580c80b0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f844400c4d0 tx=0x7f844400c990 comp rx=0 tx=0).stop 2023-11-28T20:54:52.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 shutdown_connections 2023-11-28T20:54:52.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f843c06b040 0x7f843c06d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84580d0040 0x7f84580c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84580b2290 0x7f84580c80b0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 --2- 172.21.15.45:0/1513711079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84580b1d40 0x7f845815b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 >> 172.21.15.45:0/1513711079 conn(0x7f8458070f40 msgr2=0x7f84580d3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:52.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 shutdown_connections 2023-11-28T20:54:52.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.196+0000 7f845dca3640 1 -- 172.21.15.45:0/1513711079 wait complete. 2023-11-28T20:54:52.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:54:52.411 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:52.409+0000 7f8936613040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:54:52.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.430+0000 7f93612ad640 1 Processor -- start 2023-11-28T20:54:52.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.431+0000 7f93612ad640 1 -- start start 2023-11-28T20:54:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.432+0000 7f93612ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f935c159780 0x7f935c15d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f93612ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f935c0c9dc0 0x7f935c0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f93612ad640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c0cd6b0 con 0x7f935c0ae030 2023-11-28T20:54:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f93612ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c0ce330 con 0x7f935c159780 2023-11-28T20:54:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f93612ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0cd990 con 0x7f935c0c9dc0 2023-11-28T20:54:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f935affd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f935c159780 0x7f935c15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f935affd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f935c159780 0x7f935c15d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33876/0 (socket says 172.21.15.45:33876) 2023-11-28T20:54:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f935affd640 1 -- 172.21.15.45:0/4192208139 learned_addr learned my addr 172.21.15.45:0/4192208139 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f935affd640 1 -- 172.21.15.45:0/4192208139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f935c0c9dc0 msgr2=0x7f935c0ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f935affd640 1 --2- 172.21.15.45:0/4192208139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f935c0c9dc0 0x7f935c0ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f935affd640 1 -- 172.21.15.45:0/4192208139 >> v1:172.21.15.190:6789/0 conn(0x7f935c0ae030 legacy=0x7f935c15d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:52.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.433+0000 7f935affd640 1 -- 172.21.15.45:0/4192208139 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0cdc70 con 0x7f935c159780 2023-11-28T20:54:52.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.434+0000 7f935affd640 1 --2- 172.21.15.45:0/4192208139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c159780 0x7f935c15d390 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f935c0cfd20 tx=0x7f934c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98fbaeab8324a88a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:52.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.435+0000 7f935a7fc640 1 -- 172.21.15.45:0/4192208139 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c031640 con 0x7f935c159780 2023-11-28T20:54:52.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.435+0000 7f935a7fc640 1 -- 172.21.15.45:0/4192208139 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c0317a0 con 0x7f935c159780 2023-11-28T20:54:52.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.435+0000 7f935a7fc640 1 -- 172.21.15.45:0/4192208139 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c031b30 con 0x7f935c159780 2023-11-28T20:54:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.436+0000 7f93612ad640 1 -- 172.21.15.45:0/4192208139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c159780 msgr2=0x7f935c15d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.436+0000 7f93612ad640 1 --2- 172.21.15.45:0/4192208139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c159780 0x7f935c15d390 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f935c0cfd20 tx=0x7f934c02b8f0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.436+0000 7f93612ad640 1 -- 172.21.15.45:0/4192208139 shutdown_connections 2023-11-28T20:54:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.436+0000 7f93612ad640 1 --2- 172.21.15.45:0/4192208139 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f935c0c9dc0 0x7f935c0ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.436+0000 7f93612ad640 1 --2- 172.21.15.45:0/4192208139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c159780 0x7f935c15d390 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.436+0000 7f93612ad640 1 -- 172.21.15.45:0/4192208139 >> 172.21.15.45:0/4192208139 conn(0x7f935c0ced60 msgr2=0x7f935c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.436+0000 7f93612ad640 1 -- 172.21.15.45:0/4192208139 shutdown_connections 2023-11-28T20:54:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.436+0000 7f93612ad640 1 -- 172.21.15.45:0/4192208139 wait complete. 2023-11-28T20:54:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.438+0000 7f93612ad640 1 Processor -- start 2023-11-28T20:54:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.438+0000 7f93612ad640 1 -- start start 2023-11-28T20:54:52.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.439+0000 7f93612ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0ae030 0x7f935c14db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.439+0000 7f93612ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f935c0c9dc0 0x7f935c14b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.439+0000 7f93612ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935c159780 0x7f935c06c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.439+0000 7f93612ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c14f470 con 0x7f935c0ae030 2023-11-28T20:54:52.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.439+0000 7f93612ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c14f750 con 0x7f935c159780 2023-11-28T20:54:52.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.439+0000 7f93612ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c157f50 con 0x7f935c0c9dc0 2023-11-28T20:54:52.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.439+0000 7f935affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0ae030 0x7f935c14db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.439+0000 7f935affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0ae030 0x7f935c14db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33890/0 (socket says 172.21.15.45:33890) 2023-11-28T20:54:52.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.439+0000 7f935affd640 1 -- 172.21.15.45:0/3243713450 learned_addr learned my addr 172.21.15.45:0/3243713450 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:52.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f935b7fe640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935c159780 0x7f935c06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f93527fc640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f935c0c9dc0 0x7f935c14b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f935affd640 1 -- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f935c0c9dc0 msgr2=0x7f935c14b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f935affd640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f935c0c9dc0 0x7f935c14b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f935affd640 1 -- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935c159780 msgr2=0x7f935c06c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f935affd640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935c159780 0x7f935c06c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f935affd640 1 -- 172.21.15.45:0/3243713450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c0094d0 con 0x7f935c0ae030 2023-11-28T20:54:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f935b7fe640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935c159780 0x7f935c06c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f935affd640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0ae030 0x7f935c14db90 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f934c000c00 tx=0x7f934c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f9353fff640 1 -- 172.21.15.45:0/3243713450 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c03ac60 con 0x7f935c0ae030 2023-11-28T20:54:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f9353fff640 1 -- 172.21.15.45:0/3243713450 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c03ade0 con 0x7f935c0ae030 2023-11-28T20:54:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.440+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c158230 con 0x7f935c0ae030 2023-11-28T20:54:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.441+0000 7f9353fff640 1 -- 172.21.15.45:0/3243713450 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c044630 con 0x7f935c0ae030 2023-11-28T20:54:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.441+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c14e1c0 con 0x7f935c0ae030 2023-11-28T20:54:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.442+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c14e350 con 0x7f935c0ae030 2023-11-28T20:54:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.443+0000 7f9353fff640 1 -- 172.21.15.45:0/3243713450 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 74) v1 ==== 89390+0+0 (secure 0 0 0) 0x7f934c02f050 con 0x7f935c0ae030 2023-11-28T20:54:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.444+0000 7f9353fff640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f933c06aff0 0x7f933c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.444+0000 7f9353fff640 1 -- 172.21.15.45:0/3243713450 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f934c0abf60 con 0x7f935c0ae030 2023-11-28T20:54:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.445+0000 7f93527fc640 1 -- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f933c06aff0 msgr2=0x7f933c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.445+0000 7f93527fc640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f933c06aff0 0x7f933c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.449+0000 7f9353fff640 1 -- 172.21.15.45:0/3243713450 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f934c076920 con 0x7f935c0ae030 2023-11-28T20:54:52.539 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:52.537+0000 7f5dd2c12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:54:52.574 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:52.571+0000 7f8936613040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:54:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.642+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f935c14f8c0 con 0x7f935c0ae030 2023-11-28T20:54:52.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.643+0000 7f9353fff640 1 -- 172.21.15.45:0/3243713450 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+11 (secure 0 0 0) 0x7f934c035020 con 0x7f935c0ae030 2023-11-28T20:54:52.644 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:54:52.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f933c06aff0 msgr2=0x7f933c06d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:52.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f933c06aff0 0x7f933c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0ae030 msgr2=0x7f935c14db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0ae030 0x7f935c14db90 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f934c000c00 tx=0x7f934c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 shutdown_connections 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f933c06aff0 0x7f933c06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935c159780 0x7f935c06c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f935c0c9dc0 0x7f935c14b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 --2- 172.21.15.45:0/3243713450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0ae030 0x7f935c14db90 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 >> 172.21.15.45:0/3243713450 conn(0x7f935c0ced60 msgr2=0x7f935c0afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.645+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 shutdown_connections 2023-11-28T20:54:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.646+0000 7f93612ad640 1 -- 172.21.15.45:0/3243713450 wait complete. 2023-11-28T20:54:52.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T20:54:52.681 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:52.679+0000 7f8936613040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:54:52.711 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:52.710+0000 7f5dd2c12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:54:52.730 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:52.729+0000 7f8936613040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:54:52.827 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:52.826+0000 7f5dd2c12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:54:52.837 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:52.836+0000 7f8936613040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:52.881 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:52.880+0000 7f5dd2c12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:54:52.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.881+0000 7fb1ccd24640 1 Processor -- start 2023-11-28T20:54:52.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.881+0000 7fb1ccd24640 1 -- start start 2023-11-28T20:54:52.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.882+0000 7fb1ccd24640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c8070b20 0x7fb1c8070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.883+0000 7fb1ccd24640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c806b580 con 0x7fb1c815d730 2023-11-28T20:54:52.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.883+0000 7fb1ccd24640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c806b2a0 con 0x7fb1c80b6670 2023-11-28T20:54:52.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.883+0000 7fb1ccd24640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80714f0 con 0x7fb1c8070b20 2023-11-28T20:54:52.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.883+0000 7fb1c659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb1c815d730 0x7fb1c80d2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46524/0 (socket says 172.21.15.45:46524) 2023-11-28T20:54:52.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.883+0000 7fb1c659c640 1 -- 172.21.15.45:0/3566124204 learned_addr learned my addr 172.21.15.45:0/3566124204 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:52.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.883+0000 7fb1bffff640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c8070b20 0x7fb1c8070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.883+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691136329 0 0) 0x7fb1c806b580 con 0x7fb1c815d730 2023-11-28T20:54:52.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.883+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b4003520 con 0x7fb1c815d730 2023-11-28T20:54:52.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 908695682 0 0) 0x7fb1b4003520 con 0x7fb1c815d730 2023-11-28T20:54:52.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c8070b20 msgr2=0x7fb1c8070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:52.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c8070b20 0x7fb1c8070fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 >> v1:172.21.15.190:6789/0 conn(0x7fb1c80b6670 legacy=0x7fb1c80b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c80d1730 con 0x7fb1c815d730 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb1b8001670 con 0x7fb1c815d730 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 >> v1:172.21.15.45:6789/0 conn(0x7fb1c815d730 legacy=0x7fb1c80d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c8070b20 0x7fb1b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4004d00 0x7fb1b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80714f0 con 0x7fb1b40039a0 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b400ba60 con 0x7fb1b4004d00 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b400bc20 con 0x7fb1c8070b20 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.884+0000 7fb1c659c640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.885+0000 7fb1c6d9d640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4004d00 0x7fb1b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.885+0000 7fb1bffff640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c8070b20 0x7fb1b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.885+0000 7fb1c659c640 1 -- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c8070b20 msgr2=0x7fb1b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:52.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.885+0000 7fb1c659c640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c8070b20 0x7fb1b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.885+0000 7fb1c659c640 1 -- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4004d00 msgr2=0x7fb1b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:52.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.885+0000 7fb1c659c640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1b4004d00 0x7fb1b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.885+0000 7fb1c659c640 1 -- 172.21.15.45:0/3566124204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b4003520 con 0x7fb1b40039a0 2023-11-28T20:54:52.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.885+0000 7fb1bffff640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c8070b20 0x7fb1b40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:52.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.886+0000 7fb1c659c640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1b4003dc0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c210 tx=0x7fb1b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ec10ad98c6543c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:52.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.887+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8005ad0 con 0x7fb1b40039a0 2023-11-28T20:54:52.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.887+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b8005c30 con 0x7fb1b40039a0 2023-11-28T20:54:52.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.887+0000 7fb1c5d9b640 1 -- 172.21.15.45:0/3566124204 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b802c890 con 0x7fb1b40039a0 2023-11-28T20:54:52.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.887+0000 7fb1ccd24640 1 -- 172.21.15.45:0/3566124204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 msgr2=0x7fb1b4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.887+0000 7fb1ccd24640 1 --2- 172.21.15.45:0/3566124204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1b4003dc0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c210 tx=0x7fb1b40057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.887+0000 7fb1c6d9d640 1 -- 172.21.15.45:0/3566124204 reap_dead start 2023-11-28T20:54:52.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.888+0000 7fb1ccd24640 1 -- 172.21.15.45:0/3566124204 shutdown_connections 2023-11-28T20:54:52.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.888+0000 7fb1ccd24640 1 -- 172.21.15.45:0/3566124204 >> 172.21.15.45:0/3566124204 conn(0x7fb1c80cfc70 msgr2=0x7fb1c80d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:52.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.888+0000 7fb1ccd24640 1 -- 172.21.15.45:0/3566124204 shutdown_connections 2023-11-28T20:54:52.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.888+0000 7fb1ccd24640 1 -- 172.21.15.45:0/3566124204 wait complete. 2023-11-28T20:54:52.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.889+0000 7fb1ccd24640 1 Processor -- start 2023-11-28T20:54:52.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.889+0000 7fb1ccd24640 1 -- start start 2023-11-28T20:54:52.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.890+0000 7fb1ccd24640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1c80b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.890+0000 7fb1ccd24640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c80a9b80 0x7fb1c80aa000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.890+0000 7fb1ccd24640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1c80adff0 0x7fb1c80ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.890+0000 7fb1ccd24640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80ab370 con 0x7fb1b40039a0 2023-11-28T20:54:52.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.890+0000 7fb1ccd24640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80ae9b0 con 0x7fb1c80adff0 2023-11-28T20:54:52.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.890+0000 7fb1ccd24640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c81f3650 con 0x7fb1c80a9b80 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1c659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1c80b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1c659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1c80b1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33906/0 (socket says 172.21.15.45:33906) 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1c659c640 1 -- 172.21.15.45:0/1277480437 learned_addr learned my addr 172.21.15.45:0/1277480437 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1c6d9d640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1c80adff0 0x7fb1c80ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1bffff640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c80a9b80 0x7fb1c80aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1c659c640 1 -- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c80a9b80 msgr2=0x7fb1c80aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1c659c640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c80a9b80 0x7fb1c80aa000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1c659c640 1 -- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1c80adff0 msgr2=0x7fb1c80ae470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1c659c640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1c80adff0 0x7fb1c80ae470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:52.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.891+0000 7fb1c659c640 1 -- 172.21.15.45:0/1277480437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c80aa6f0 con 0x7fb1b40039a0 2023-11-28T20:54:52.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.892+0000 7fb1c659c640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1c80b1a60 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fb1b802c430 tx=0x7fb1b8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:52.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.892+0000 7fb1beffd640 1 -- 172.21.15.45:0/1277480437 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8039070 con 0x7fb1b40039a0 2023-11-28T20:54:52.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.892+0000 7fb1beffd640 1 -- 172.21.15.45:0/1277480437 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b802ddb0 con 0x7fb1b40039a0 2023-11-28T20:54:52.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.892+0000 7fb1beffd640 1 -- 172.21.15.45:0/1277480437 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b803c040 con 0x7fb1b40039a0 2023-11-28T20:54:52.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.892+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c80aa9d0 con 0x7fb1b40039a0 2023-11-28T20:54:52.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.892+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c80aaf90 con 0x7fb1b40039a0 2023-11-28T20:54:52.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.893+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c8073440 con 0x7fb1b40039a0 2023-11-28T20:54:52.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.895+0000 7fb1beffd640 1 -- 172.21.15.45:0/1277480437 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 74) v1 ==== 89390+0+0 (secure 0 0 0) 0x7fb1b8032070 con 0x7fb1b40039a0 2023-11-28T20:54:52.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.896+0000 7fb1beffd640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb1a806b020 0x7fb1a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:52.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.897+0000 7fb1beffd640 1 -- 172.21.15.45:0/1277480437 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb1b80aaed0 con 0x7fb1b40039a0 2023-11-28T20:54:52.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.897+0000 7fb1bffff640 1 -- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb1a806b020 msgr2=0x7fb1a806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:52.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.897+0000 7fb1bffff640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb1a806b020 0x7fb1a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:52.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:52.901+0000 7fb1beffd640 1 -- 172.21.15.45:0/1277480437 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1b8075a30 con 0x7fb1b40039a0 2023-11-28T20:54:52.945 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:52.944+0000 7f8936613040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:54:53.006 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:53.004+0000 7f5dd2c12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:53.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.098+0000 7fb1bffff640 1 -- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb1a806b020 msgr2=0x7fb1a806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:53.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.098+0000 7fb1bffff640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb1a806b020 0x7fb1a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:53.107 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:53.105+0000 7f8936613040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:53.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.116+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb1c80d1140 con 0x7fb1b40039a0 2023-11-28T20:54:53.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.117+0000 7fb1beffd640 1 -- 172.21.15.45:0/1277480437 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v149) v1 ==== 80+0+477 (secure 0 0 0) 0x7fb1b8046070 con 0x7fb1b40039a0 2023-11-28T20:54:53.118 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1886852041 2023-11-29T20:54:41.552066+0000 2023-11-28T20:54:53.118 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/649548475 2023-11-29T20:54:41.552066+0000 2023-11-28T20:54:53.118 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1625344948 2023-11-29T20:54:41.552066+0000 2023-11-28T20:54:53.118 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2599111328 2023-11-29T20:54:41.552066+0000 2023-11-28T20:54:53.118 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2598605134 2023-11-29T20:54:41.552066+0000 2023-11-28T20:54:53.118 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/2586319130 2023-11-29T20:54:41.552066+0000 2023-11-28T20:54:53.119 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/2586319130 2023-11-29T20:54:41.552066+0000 2023-11-28T20:54:53.119 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1527814408 2023-11-29T20:54:41.552066+0000 2023-11-28T20:54:53.119 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2023-11-28T20:54:53.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb1a806b020 msgr2=0x7fb1a806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb1a806b020 0x7fb1a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 msgr2=0x7fb1c80b1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1c80b1a60 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fb1b802c430 tx=0x7fb1b8001a00 comp rx=0 tx=0).stop 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 shutdown_connections 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7fb1a806b020 0x7fb1a806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1c80adff0 0x7fb1c80ae470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb1c80a9b80 0x7fb1c80aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 --2- 172.21.15.45:0/1277480437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b40039a0 0x7fb1c80b1a60 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 >> 172.21.15.45:0/1277480437 conn(0x7fb1c80cfc70 msgr2=0x7fb1c815e520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 shutdown_connections 2023-11-28T20:54:53.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.119+0000 7fb1ccd24640 1 -- 172.21.15.45:0/1277480437 wait complete. 2023-11-28T20:54:53.122 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:53.120+0000 7f5dd2c12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:54:53.140 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T20:54:53.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:53.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T20:54:53.155 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:53.154+0000 7f8936613040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:53.225 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T20:54:53.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:54:53.225 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T20:54:53.268 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:54:53.268 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:53.278 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:53.275+0000 7f8936613040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:54:53.292 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:53.291+0000 7f5dd2c12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:54:53.326 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:54:53.326 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:54:53.326 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:53.345 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:53.344+0000 7f5dd2c12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:54:53.384 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:54:53.384 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:54:53.384 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:54:53.438 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:54:53.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:53.474 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:53.473+0000 7f5dd2c12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:54:53.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.641+0000 7f15e397d640 1 Processor -- start 2023-11-28T20:54:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.641+0000 7f15e397d640 1 -- start start 2023-11-28T20:54:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.643+0000 7f15e397d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc0ce210 0x7f15dc0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.643+0000 7f15e397d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc161f50 con 0x7f15dc161200 2023-11-28T20:54:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.643+0000 7f15e397d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc0d1570 con 0x7f15dc15d5e0 2023-11-28T20:54:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.643+0000 7f15e397d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0cebe0 con 0x7f15dc0ce210 2023-11-28T20:54:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.643+0000 7f15e1f1a640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f15dc161200 0x7f15dc161670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46534/0 (socket says 172.21.15.45:46534) 2023-11-28T20:54:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.643+0000 7f15e1f1a640 1 -- 172.21.15.45:0/1154241230 learned_addr learned my addr 172.21.15.45:0/1154241230 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.643+0000 7f15e0f18640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc0ce210 0x7f15dc0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.643+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696522237 0 0) 0x7f15dc161f50 con 0x7f15dc161200 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.643+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c8003520 con 0x7f15dc161200 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885616227 0 0) 0x7f15c8003520 con 0x7f15dc161200 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc0ce210 msgr2=0x7f15dc0ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc0ce210 0x7f15dc0ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 >> v1:172.21.15.190:6789/0 conn(0x7f15dc15d5e0 legacy=0x7f15dc0d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc0d1850 con 0x7f15dc161200 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f15d40019f0 con 0x7f15dc161200 2023-11-28T20:54:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 >> v1:172.21.15.45:6789/0 conn(0x7f15dc161200 legacy=0x7f15dc161670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c80039a0 0x7f15c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c80047c0 0x7f15c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc0ce210 0x7f15c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0cebe0 con 0x7f15c80039a0 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c8005510 con 0x7f15c80047c0 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c800bec0 con 0x7f15dc0ce210 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.644+0000 7f15e1719640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c80039a0 0x7f15c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.645+0000 7f15e1719640 1 -- 172.21.15.45:0/1154241230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc0ce210 msgr2=0x7f15c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.645+0000 7f15e1719640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc0ce210 0x7f15c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.645+0000 7f15e1719640 1 -- 172.21.15.45:0/1154241230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c80047c0 msgr2=0x7f15c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.645+0000 7f15e1719640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c80047c0 0x7f15c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.645+0000 7f15e1719640 1 -- 172.21.15.45:0/1154241230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8003520 con 0x7f15c80039a0 2023-11-28T20:54:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.646+0000 7f15e1719640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c80039a0 0x7f15c8003dc0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f15d8001750 tx=0x7f15c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db9da609279d7b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.646+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d80052d0 con 0x7f15c80039a0 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.646+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d802e050 con 0x7f15c80039a0 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.646+0000 7f15d3fff640 1 -- 172.21.15.45:0/1154241230 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d80035f0 con 0x7f15c80039a0 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.646+0000 7f15e397d640 1 -- 172.21.15.45:0/1154241230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c80039a0 msgr2=0x7f15c8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.646+0000 7f15e397d640 1 --2- 172.21.15.45:0/1154241230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c80039a0 0x7f15c8003dc0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f15d8001750 tx=0x7f15c80057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.646+0000 7f15e1f1a640 1 -- 172.21.15.45:0/1154241230 reap_dead start 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.647+0000 7f15e397d640 1 -- 172.21.15.45:0/1154241230 shutdown_connections 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.647+0000 7f15e397d640 1 -- 172.21.15.45:0/1154241230 >> 172.21.15.45:0/1154241230 conn(0x7f15dc070b20 msgr2=0x7f15dc070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.647+0000 7f15e397d640 1 -- 172.21.15.45:0/1154241230 shutdown_connections 2023-11-28T20:54:53.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.647+0000 7f15e397d640 1 -- 172.21.15.45:0/1154241230 wait complete. 2023-11-28T20:54:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.648+0000 7f15e397d640 1 Processor -- start 2023-11-28T20:54:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.648+0000 7f15e397d640 1 -- start start 2023-11-28T20:54:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.649+0000 7f15e397d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c80039a0 0x7f15dc15ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.649+0000 7f15e397d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc15ef10 0x7f15dc0cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.649+0000 7f15e397d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc0cb1f0 0x7f15dc0b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.649+0000 7f15e397d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc1602a0 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.649+0000 7f15e397d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc15de70 con 0x7f15c80039a0 2023-11-28T20:54:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.649+0000 7f15e397d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc15e150 con 0x7f15dc15ef10 2023-11-28T20:54:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1f1a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc0cb1f0 0x7f15dc0b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1f1a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc0cb1f0 0x7f15dc0b1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33918/0 (socket says 172.21.15.45:33918) 2023-11-28T20:54:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1f1a640 1 -- 172.21.15.45:0/3665571715 learned_addr learned my addr 172.21.15.45:0/3665571715 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1719640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c80039a0 0x7f15dc15ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e0f18640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc15ef10 0x7f15dc0cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1f1a640 1 -- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc15ef10 msgr2=0x7f15dc0cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1f1a640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc15ef10 0x7f15dc0cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1f1a640 1 -- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c80039a0 msgr2=0x7f15dc15ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1f1a640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c80039a0 0x7f15dc15ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1f1a640 1 -- 172.21.15.45:0/3665571715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc15e430 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1719640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c80039a0 0x7f15dc15ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e0f18640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc15ef10 0x7f15dc0cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.650+0000 7f15e1f1a640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc0cb1f0 0x7f15dc0b1e10 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f15dc1615e0 tx=0x7f15d400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.651+0000 7f15d1ffb640 1 -- 172.21.15.45:0/3665571715 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d400a610 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.651+0000 7f15d1ffb640 1 -- 172.21.15.45:0/3665571715 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d400a790 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.651+0000 7f15d1ffb640 1 -- 172.21.15.45:0/3665571715 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d400ab40 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.651+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15dc15e710 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.651+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15dc15b170 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.652+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15dc0733d0 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.653+0000 7f15d1ffb640 1 -- 172.21.15.45:0/3665571715 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 74) v1 ==== 89390+0+0 (secure 0 0 0) 0x7f15d400aca0 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.654+0000 7f15d1ffb640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f15c006aff0 0x7f15c006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:53.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.654+0000 7f15d1ffb640 1 -- 172.21.15.45:0/3665571715 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f15d4089530 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.654+0000 7f15e1719640 1 -- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f15c006aff0 msgr2=0x7f15c006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:53.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.654+0000 7f15e1719640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f15c006aff0 0x7f15c006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:53.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.658+0000 7f15d1ffb640 1 -- 172.21.15.45:0/3665571715 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15d4054090 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.855+0000 7f15e1719640 1 -- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f15c006aff0 msgr2=0x7f15c006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.855+0000 7f15e1719640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f15c006aff0 0x7f15c006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:53.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.887+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15dc0d1120 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.888+0000 7f15d1ffb640 1 -- 172.21.15.45:0/3665571715 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f15d40576b0 con 0x7f15dc0cb1f0 2023-11-28T20:54:53.890 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:53.890 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/1133315976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1133315976},{"type":"v1","addr":"172.21.15.190:6836","nonce":1133315976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2810069664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2810069664},{"type":"v1","addr":"172.21.15.190:6837","nonce":2810069664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1553174870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1553174870},{"type":"v1","addr":"172.21.15.45:6833","nonce":1553174870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:54:26.355691+0000","modified":"2023-11-28T20:54:29.430641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8595":{"gid":8595,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/684266044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":684266044},{"type":"v1","addr":"172.21.15.45:6835","nonce":684266044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-28T20:54:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 67 2023-11-28T20:54:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f15c006aff0 msgr2=0x7f15c006d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f15c006aff0 0x7f15c006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc0cb1f0 msgr2=0x7f15dc0b1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc0cb1f0 0x7f15dc0b1e10 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f15dc1615e0 tx=0x7f15d400cf00 comp rx=0 tx=0).stop 2023-11-28T20:54:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 shutdown_connections 2023-11-28T20:54:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f15c006aff0 0x7f15c006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15dc0cb1f0 0x7f15dc0b1e10 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15dc15ef10 0x7f15dc0cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 --2- 172.21.15.45:0/3665571715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c80039a0 0x7f15dc15ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 >> 172.21.15.45:0/3665571715 conn(0x7f15dc070b20 msgr2=0x7f15dc0b87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 shutdown_connections 2023-11-28T20:54:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:53.891+0000 7f15e397d640 1 -- 172.21.15.45:0/3665571715 wait complete. 2023-11-28T20:54:53.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:54.118 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.117+0000 7f8936613040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:54:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.153+0000 7f23cf59e640 1 Processor -- start 2023-11-28T20:54:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.153+0000 7f23cf59e640 1 -- start start 2023-11-28T20:54:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.154+0000 7f23cf59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f23d015d550 0x7f23d00cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.154+0000 7f23cf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d00d0100 0x7f23d00d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.154+0000 7f23cf59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23d00d1ab0 0x7f23d0070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.154+0000 7f23cf59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d00d0560 con 0x7f23d00d1ab0 2023-11-28T20:54:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.154+0000 7f23cf59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d00d2230 con 0x7f23d015d550 2023-11-28T20:54:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.154+0000 7f23cf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00d2510 con 0x7f23d00d0100 2023-11-28T20:54:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ced9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23d00d1ab0 0x7f23d0070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ced9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f23d00d1ab0 0x7f23d0070b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49520/0 (socket says 172.21.15.45:49520) 2023-11-28T20:54:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ced9d640 1 -- 172.21.15.45:0/3868392492 learned_addr learned my addr 172.21.15.45:0/3868392492 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ce59c640 1 --2- 172.21.15.45:0/3868392492 >> v2:172.21.15.190:3300/0 conn(0x7f23d015d550 0x7f23d00cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ced9d640 1 -- 172.21.15.45:0/3868392492 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d00d0100 msgr2=0x7f23d00d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23cdd9b640 1 --2- 172.21.15.45:0/3868392492 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d00d0100 0x7f23d00d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ced9d640 1 --2- 172.21.15.45:0/3868392492 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d00d0100 0x7f23d00d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ced9d640 1 -- 172.21.15.45:0/3868392492 >> v2:172.21.15.190:3300/0 conn(0x7f23d015d550 msgr2=0x7f23d00cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ced9d640 1 --2- 172.21.15.45:0/3868392492 >> v2:172.21.15.190:3300/0 conn(0x7f23d015d550 0x7f23d00cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ced9d640 1 -- 172.21.15.45:0/3868392492 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d00d27f0 con 0x7f23d00d1ab0 2023-11-28T20:54:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.155+0000 7f23ce59c640 1 --2- 172.21.15.45:0/3868392492 >> v2:172.21.15.190:3300/0 conn(0x7f23d015d550 0x7f23d00cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.157+0000 7f23ced9d640 1 --2- 172.21.15.45:0/3868392492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 0x7f23d0070b20 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f23d00b5dc0 tx=0x7f23c802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389450133fd9dd79 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.157+0000 7f23cd59a640 1 -- 172.21.15.45:0/3868392492 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c8031860 con 0x7f23d00d1ab0 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.157+0000 7f23cd59a640 1 -- 172.21.15.45:0/3868392492 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c80319c0 con 0x7f23d00d1ab0 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.157+0000 7f23cd59a640 1 -- 172.21.15.45:0/3868392492 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c8031d50 con 0x7f23d00d1ab0 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.158+0000 7f23cf59e640 1 -- 172.21.15.45:0/3868392492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 msgr2=0x7f23d0070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.158+0000 7f23cf59e640 1 --2- 172.21.15.45:0/3868392492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 0x7f23d0070b20 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f23d00b5dc0 tx=0x7f23c802b900 comp rx=0 tx=0).stop 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.158+0000 7f23cf59e640 1 -- 172.21.15.45:0/3868392492 shutdown_connections 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.158+0000 7f23cf59e640 1 --2- 172.21.15.45:0/3868392492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 0x7f23d0070b20 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.158+0000 7f23cf59e640 1 --2- 172.21.15.45:0/3868392492 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d00d0100 0x7f23d00d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.158+0000 7f23cf59e640 1 --2- 172.21.15.45:0/3868392492 >> v2:172.21.15.190:3300/0 conn(0x7f23d015d550 0x7f23d00cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.158+0000 7f23cf59e640 1 -- 172.21.15.45:0/3868392492 >> 172.21.15.45:0/3868392492 conn(0x7f23d006ac70 msgr2=0x7f23d00b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:54.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.158+0000 7f23cf59e640 1 -- 172.21.15.45:0/3868392492 shutdown_connections 2023-11-28T20:54:54.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.158+0000 7f23cf59e640 1 -- 172.21.15.45:0/3868392492 wait complete. 2023-11-28T20:54:54.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.160+0000 7f23cf59e640 1 Processor -- start 2023-11-28T20:54:54.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.160+0000 7f23cf59e640 1 -- start start 2023-11-28T20:54:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cf59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23d00d0100 0x7f23d0157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 0x7f23d01f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d015d550 0x7f23d00c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cf59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d01f6ec0 con 0x7f23d00d1ab0 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cf59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d01f3eb0 con 0x7f23d00d0100 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00c62b0 con 0x7f23d015d550 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cdd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 0x7f23d01f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cdd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 0x7f23d01f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49536/0 (socket says 172.21.15.45:49536) 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cdd9b640 1 -- 172.21.15.45:0/2683634983 learned_addr learned my addr 172.21.15.45:0/2683634983 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23ce59c640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23d00d0100 0x7f23d0157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23ced9d640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d015d550 0x7f23d00c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cdd9b640 1 -- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d015d550 msgr2=0x7f23d00c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cdd9b640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d015d550 0x7f23d00c5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cdd9b640 1 -- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23d00d0100 msgr2=0x7f23d0157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cdd9b640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23d00d0100 0x7f23d0157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23cdd9b640 1 -- 172.21.15.45:0/2683634983 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80094d0 con 0x7f23d00d1ab0 2023-11-28T20:54:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.161+0000 7f23ce59c640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23d00d0100 0x7f23d0157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.162+0000 7f23cdd9b640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 0x7f23d01f3970 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f23b800c820 tx=0x7f23b800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.162+0000 7f23c6ffd640 1 -- 172.21.15.45:0/2683634983 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8004220 con 0x7f23d00d1ab0 2023-11-28T20:54:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.162+0000 7f23c6ffd640 1 -- 172.21.15.45:0/2683634983 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b8004380 con 0x7f23d00d1ab0 2023-11-28T20:54:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.162+0000 7f23c6ffd640 1 -- 172.21.15.45:0/2683634983 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8007a70 con 0x7f23d00d1ab0 2023-11-28T20:54:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.162+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d01576f0 con 0x7f23d00d1ab0 2023-11-28T20:54:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.162+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d01f6800 con 0x7f23d00d1ab0 2023-11-28T20:54:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.164+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d01f7000 con 0x7f23d00d1ab0 2023-11-28T20:54:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.164+0000 7f23c6ffd640 1 -- 172.21.15.45:0/2683634983 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 74) v1 ==== 89390+0+0 (secure 0 0 0) 0x7f23b8007bd0 con 0x7f23d00d1ab0 2023-11-28T20:54:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.165+0000 7f23c6ffd640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f23b006aff0 0x7f23b006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.165+0000 7f23c6ffd640 1 -- 172.21.15.45:0/2683634983 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f23b8089a80 con 0x7f23d00d1ab0 2023-11-28T20:54:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.165+0000 7f23ce59c640 1 -- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f23b006aff0 msgr2=0x7f23b006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:54.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.165+0000 7f23ce59c640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f23b006aff0 0x7f23b006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:54:54.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.169+0000 7f23c6ffd640 1 -- 172.21.15.45:0/2683634983 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23d01f7000 con 0x7f23d00d1ab0 2023-11-28T20:54:54.358 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:54.357+0000 7f5dd2c12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:54:54.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.366+0000 7f23ce59c640 1 -- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f23b006aff0 msgr2=0x7f23b006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1149934668 2023-11-28T20:54:54.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.366+0000 7f23ce59c640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f23b006aff0 0x7f23b006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:54:54.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.373+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23d01f7000 con 0x7f23d00d1ab0 2023-11-28T20:54:54.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.375+0000 7f23c6ffd640 1 -- 172.21.15.45:0/2683634983 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+17324 (secure 0 0 0) 0x7f23d01f7000 con 0x7f23d00d1ab0 2023-11-28T20:54:54.377 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:54.377 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":149,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:41.552133+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:54:24.325054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-11-28T20:54:25.374707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1886852041":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/649548475":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/1625344948":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2599111328":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2598605134":"2023-11-29T20:54:41.552066+0000","172.21.15.190:6833/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.190:6832/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/1527814408":"2023-11-29T20:54:41.552066+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f23b006aff0 msgr2=0x7f23b006d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:54:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f23b006aff0 0x7f23b006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 msgr2=0x7f23d01f3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 0x7f23d01f3970 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f23b800c820 tx=0x7f23b800cce0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 shutdown_connections 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:6832/1149934668,v1:172.21.15.190:6833/1149934668] conn(0x7f23b006aff0 0x7f23b006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23d015d550 0x7f23d00c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d00d1ab0 0x7f23d01f3970 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 --2- 172.21.15.45:0/2683634983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23d00d0100 0x7f23d0157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 >> 172.21.15.45:0/2683634983 conn(0x7f23d006ac70 msgr2=0x7f23d00ce010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 shutdown_connections 2023-11-28T20:54:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.378+0000 7f23cf59e640 1 -- 172.21.15.45:0/2683634983 wait complete. 2023-11-28T20:54:54.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:54:24.325054+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:54:25.374707+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:54:54.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:54:54.413 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.412+0000 7f8936613040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:54:54.475 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.474+0000 7f8936613040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:54:54.589 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.588+0000 7f8936613040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:54:54.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.612+0000 7fc367c90640 1 Processor -- start 2023-11-28T20:54:54.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.612+0000 7fc367c90640 1 -- start start 2023-11-28T20:54:54.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.614+0000 7fc367c90640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc3600b2b70 0x7fc3600b3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.614+0000 7fc367c90640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600c5f60 con 0x7fc36006c0b0 2023-11-28T20:54:54.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.614+0000 7fc367c90640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600c6240 con 0x7fc360146bc0 2023-11-28T20:54:54.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.614+0000 7fc367c90640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc360146fa0 con 0x7fc3600b2b70 2023-11-28T20:54:54.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.614+0000 7fc36522b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc36006c0b0 0x7fc3600ceb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40538/0 (socket says 172.21.15.45:40538) 2023-11-28T20:54:54.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.615+0000 7fc36522b640 1 -- 172.21.15.45:0/3308512823 learned_addr learned my addr 172.21.15.45:0/3308512823 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:54.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.615+0000 7fc36622d640 1 --2- 172.21.15.45:0/3308512823 >> v2:172.21.15.190:3300/0 conn(0x7fc3600b2b70 0x7fc3600b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.615+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595645237 0 0) 0x7fc3600c5f60 con 0x7fc36006c0b0 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.615+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc34c003520 con 0x7fc36006c0b0 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.615+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1326949870 0 0) 0x7fc34c003520 con 0x7fc36006c0b0 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.615+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 >> v2:172.21.15.190:3300/0 conn(0x7fc3600b2b70 msgr2=0x7fc3600b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.615+0000 7fc364a2a640 1 --2- 172.21.15.45:0/3308512823 >> v2:172.21.15.190:3300/0 conn(0x7fc3600b2b70 0x7fc3600b3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.615+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 >> v1:172.21.15.190:6789/0 conn(0x7fc360146bc0 legacy=0x7fc3601618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.615+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36006a5a0 con 0x7fc36006c0b0 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc3540019f0 con 0x7fc36006c0b0 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 >> v1:172.21.15.45:6789/0 conn(0x7fc36006c0b0 legacy=0x7fc3600ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc364a2a640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc34c0039c0 0x7fc34c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc364a2a640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc34c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc364a2a640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34c0051d0 0x7fc34c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc360146fa0 con 0x7fc34c0047c0 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c00c120 con 0x7fc34c0039c0 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c00c2e0 con 0x7fc34c0051d0 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc36522b640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc34c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc365a2c640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc34c0039c0 0x7fc34c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc36622d640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34c0051d0 0x7fc34c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc36522b640 1 -- 172.21.15.45:0/3308512823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34c0051d0 msgr2=0x7fc34c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc36522b640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc34c0051d0 0x7fc34c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc36522b640 1 -- 172.21.15.45:0/3308512823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc34c0039c0 msgr2=0x7fc34c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc36522b640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc34c0039c0 0x7fc34c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc36522b640 1 -- 172.21.15.45:0/3308512823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c003520 con 0x7fc34c0047c0 2023-11-28T20:54:54.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.616+0000 7fc36622d640 1 -- 172.21.15.45:0/3308512823 reap_dead start 2023-11-28T20:54:54.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.618+0000 7fc36522b640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc34c004c40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc35400a760 tx=0x7fc34c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5290a91f645d5ce5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:54.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.618+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354034630 con 0x7fc34c0047c0 2023-11-28T20:54:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.618+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35402f070 con 0x7fc34c0047c0 2023-11-28T20:54:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.618+0000 7fc364a2a640 1 -- 172.21.15.45:0/3308512823 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354034a50 con 0x7fc34c0047c0 2023-11-28T20:54:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.619+0000 7fc367c90640 1 -- 172.21.15.45:0/3308512823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 msgr2=0x7fc34c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.619+0000 7fc367c90640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc34c004c40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc35400a760 tx=0x7fc34c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.619+0000 7fc367c90640 1 -- 172.21.15.45:0/3308512823 shutdown_connections 2023-11-28T20:54:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.619+0000 7fc367c90640 1 --2- 172.21.15.45:0/3308512823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc34c004c40 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.619+0000 7fc367c90640 1 -- 172.21.15.45:0/3308512823 >> 172.21.15.45:0/3308512823 conn(0x7fc360066820 msgr2=0x7fc36006c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.619+0000 7fc367c90640 1 -- 172.21.15.45:0/3308512823 shutdown_connections 2023-11-28T20:54:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.619+0000 7fc367c90640 1 -- 172.21.15.45:0/3308512823 wait complete. 2023-11-28T20:54:54.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.621+0000 7fc367c90640 1 Processor -- start 2023-11-28T20:54:54.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.621+0000 7fc367c90640 1 -- start start 2023-11-28T20:54:54.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.622+0000 7fc367c90640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc36015c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.622+0000 7fc367c90640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3600cef40 0x7fc3600cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.622+0000 7fc367c90640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc36013e1e0 0x7fc36013e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:54.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.622+0000 7fc367c90640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3600cf900 con 0x7fc34c0047c0 2023-11-28T20:54:54.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.622+0000 7fc367c90640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36013f510 con 0x7fc36013e1e0 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.622+0000 7fc367c90640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3600c9420 con 0x7fc3600cef40 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.622+0000 7fc365a2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc36015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.622+0000 7fc365a2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc36015c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49550/0 (socket says 172.21.15.45:49550) 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.622+0000 7fc365a2c640 1 -- 172.21.15.45:0/2358158878 learned_addr learned my addr 172.21.15.45:0/2358158878 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.623+0000 7fc36522b640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3600cef40 0x7fc3600cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.623+0000 7fc36622d640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc36013e1e0 0x7fc36013e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.623+0000 7fc365a2c640 1 -- 172.21.15.45:0/2358158878 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3600cef40 msgr2=0x7fc3600cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.623+0000 7fc365a2c640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3600cef40 0x7fc3600cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.623+0000 7fc365a2c640 1 -- 172.21.15.45:0/2358158878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc36013e1e0 msgr2=0x7fc36013e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.623+0000 7fc365a2c640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc36013e1e0 0x7fc36013e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.623+0000 7fc365a2c640 1 -- 172.21.15.45:0/2358158878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600c90f0 con 0x7fc34c0047c0 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.623+0000 7fc36622d640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc36013e1e0 0x7fc36013e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:54:54.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.623+0000 7fc365a2c640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc36015c080 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fc35c007810 tx=0x7fc35c00e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:54.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.624+0000 7fc3527fc640 1 -- 172.21.15.45:0/2358158878 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0052e0 con 0x7fc34c0047c0 2023-11-28T20:54:54.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.624+0000 7fc3527fc640 1 -- 172.21.15.45:0/2358158878 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c005440 con 0x7fc34c0047c0 2023-11-28T20:54:54.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.624+0000 7fc367c90640 1 -- 172.21.15.45:0/2358158878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36013ef50 con 0x7fc34c0047c0 2023-11-28T20:54:54.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.624+0000 7fc3527fc640 1 -- 172.21.15.45:0/2358158878 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c003810 con 0x7fc34c0047c0 2023-11-28T20:54:54.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.624+0000 7fc367c90640 1 -- 172.21.15.45:0/2358158878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36013eba0 con 0x7fc34c0047c0 2023-11-28T20:54:54.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.625+0000 7fc367c90640 1 -- 172.21.15.45:0/2358158878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3600733d0 con 0x7fc34c0047c0 2023-11-28T20:54:54.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.627+0000 7fc3527fc640 1 -- 172.21.15.45:0/2358158878 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 75) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fc35c003970 con 0x7fc34c0047c0 2023-11-28T20:54:54.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.628+0000 7fc3527fc640 1 -- 172.21.15.45:0/2358158878 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7fc35c088e20 con 0x7fc34c0047c0 2023-11-28T20:54:54.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.633+0000 7fc3527fc640 1 -- 172.21.15.45:0/2358158878 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc35c053df0 con 0x7fc34c0047c0 2023-11-28T20:54:54.675 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.674+0000 7f88bb44d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.676 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.674+0000 7f88bb44d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.676 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.674+0000 7f88bb44d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.676 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.674+0000 7f88bb44d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.676 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.674+0000 7f88bb44d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.676 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.675+0000 7f88b8447640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.676 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.675+0000 7f88b8447640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.676 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.675+0000 7f88b8447640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.677 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.675+0000 7f88b8447640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.677 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:54:54.675+0000 7f88b8447640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:54:54.703 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:54.702+0000 7f5dd2c12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:54:54.765 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:54.764+0000 7f5dd2c12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:54:54.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.828+0000 7fc367c90640 1 -- 172.21.15.45:0/2358158878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc3600b08f0 con 0x7fc34c0047c0 2023-11-28T20:54:54.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.829+0000 7fc3527fc640 1 -- 172.21.15.45:0/2358158878 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v150) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc35c019020 con 0x7fc34c0047c0 2023-11-28T20:54:54.830 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:54:54.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.832+0000 7fc367c90640 1 -- 172.21.15.45:0/2358158878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 msgr2=0x7fc36015c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:54.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.832+0000 7fc367c90640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc36015c080 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fc35c007810 tx=0x7fc35c00e910 comp rx=0 tx=0).stop 2023-11-28T20:54:54.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.832+0000 7fc367c90640 1 -- 172.21.15.45:0/2358158878 shutdown_connections 2023-11-28T20:54:54.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.832+0000 7fc367c90640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc36013e1e0 0x7fc36013e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.832+0000 7fc367c90640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3600cef40 0x7fc3600cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.832+0000 7fc367c90640 1 --2- 172.21.15.45:0/2358158878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc34c0047c0 0x7fc36015c080 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:54.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.832+0000 7fc367c90640 1 -- 172.21.15.45:0/2358158878 >> 172.21.15.45:0/2358158878 conn(0x7fc360066820 msgr2=0x7fc3600b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:54.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.832+0000 7fc367c90640 1 -- 172.21.15.45:0/2358158878 shutdown_connections 2023-11-28T20:54:54.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:54.832+0000 7fc367c90640 1 -- 172.21.15.45:0/2358158878 wait complete. 2023-11-28T20:54:54.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T20:54:54.888 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:54:54.887+0000 7f5dd2c12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:54:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.054+0000 7f5afa9b5640 1 Processor -- start 2023-11-28T20:54:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.055+0000 7f5afa9b5640 1 -- start start 2023-11-28T20:54:55.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.056+0000 7f5afa9b5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5af40b3040 0x7f5af406cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.056+0000 7f5afa9b5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5af415b270 0x7f5af415f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.056+0000 7f5afa9b5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af40b3420 0x7f5af40b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.056+0000 7f5afa9b5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af40ceaa0 con 0x7f5af415b270 2023-11-28T20:54:55.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.056+0000 7f5afa9b5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af415faf0 con 0x7f5af40b3040 2023-11-28T20:54:55.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.056+0000 7f5afa9b5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af406d410 con 0x7f5af40b3420 2023-11-28T20:54:55.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.056+0000 7f5af91b2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5af415b270 0x7f5af415f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.057+0000 7f5af91b2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5af415b270 0x7f5af415f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49554/0 (socket says 172.21.15.45:49554) 2023-11-28T20:54:55.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.057+0000 7f5af91b2640 1 -- 172.21.15.45:0/402900205 learned_addr learned my addr 172.21.15.45:0/402900205 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:55.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.057+0000 7f5af91b2640 1 -- 172.21.15.45:0/402900205 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af40b3420 msgr2=0x7f5af40b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:55.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.057+0000 7f5af91b2640 1 --2- 172.21.15.45:0/402900205 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af40b3420 0x7f5af40b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.057+0000 7f5af91b2640 1 -- 172.21.15.45:0/402900205 >> v2:172.21.15.190:3300/0 conn(0x7f5af40b3040 msgr2=0x7f5af406cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:55.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.057+0000 7f5af99b3640 1 --2- 172.21.15.45:0/402900205 >> v2:172.21.15.190:3300/0 conn(0x7f5af40b3040 0x7f5af406cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.057+0000 7f5af91b2640 1 --2- 172.21.15.45:0/402900205 >> v2:172.21.15.190:3300/0 conn(0x7f5af40b3040 0x7f5af406cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.057+0000 7f5af91b2640 1 -- 172.21.15.45:0/402900205 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af406d6f0 con 0x7f5af415b270 2023-11-28T20:54:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.058+0000 7f5af91b2640 1 --2- 172.21.15.45:0/402900205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 0x7f5af415f5b0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f5af40b4410 tx=0x7f5ae402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84b3b4ea51c109a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.058+0000 7f5af89b1640 1 -- 172.21.15.45:0/402900205 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4031750 con 0x7f5af415b270 2023-11-28T20:54:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.058+0000 7f5af89b1640 1 -- 172.21.15.45:0/402900205 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae40318b0 con 0x7f5af415b270 2023-11-28T20:54:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.058+0000 7f5af89b1640 1 -- 172.21.15.45:0/402900205 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4031c40 con 0x7f5af415b270 2023-11-28T20:54:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.059+0000 7f5afa9b5640 1 -- 172.21.15.45:0/402900205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 msgr2=0x7f5af415f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.059+0000 7f5afa9b5640 1 --2- 172.21.15.45:0/402900205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 0x7f5af415f5b0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f5af40b4410 tx=0x7f5ae402b7f0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.059+0000 7f5afa9b5640 1 -- 172.21.15.45:0/402900205 shutdown_connections 2023-11-28T20:54:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.059+0000 7f5afa9b5640 1 --2- 172.21.15.45:0/402900205 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af40b3420 0x7f5af40b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.059+0000 7f5afa9b5640 1 --2- 172.21.15.45:0/402900205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 0x7f5af415f5b0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.059+0000 7f5afa9b5640 1 --2- 172.21.15.45:0/402900205 >> v2:172.21.15.190:3300/0 conn(0x7f5af40b3040 0x7f5af406cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.059+0000 7f5afa9b5640 1 -- 172.21.15.45:0/402900205 >> 172.21.15.45:0/402900205 conn(0x7f5af40b1050 msgr2=0x7f5af40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.059+0000 7f5afa9b5640 1 -- 172.21.15.45:0/402900205 shutdown_connections 2023-11-28T20:54:55.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.059+0000 7f5afa9b5640 1 -- 172.21.15.45:0/402900205 wait complete. 2023-11-28T20:54:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.061+0000 7f5afa9b5640 1 Processor -- start 2023-11-28T20:54:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.061+0000 7f5afa9b5640 1 -- start start 2023-11-28T20:54:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.062+0000 7f5afa9b5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af40b3040 0x7f5af4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.062+0000 7f5afa9b5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af40b3420 0x7f5af415c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.062+0000 7f5afa9b5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 0x7f5af415c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.062+0000 7f5afa9b5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af415a6c0 con 0x7f5af415b270 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.062+0000 7f5afa9b5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af415b7f0 con 0x7f5af40b3040 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.062+0000 7f5afa9b5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af415bad0 con 0x7f5af40b3420 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.062+0000 7f5afa1b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 0x7f5af415c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.062+0000 7f5afa1b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 0x7f5af415c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49560/0 (socket says 172.21.15.45:49560) 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.062+0000 7f5afa1b4640 1 -- 172.21.15.45:0/806898758 learned_addr learned my addr 172.21.15.45:0/806898758 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5afa1b4640 1 -- 172.21.15.45:0/806898758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af40b3420 msgr2=0x7f5af415c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5af91b2640 1 --2- 172.21.15.45:0/806898758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af40b3420 0x7f5af415c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5af99b3640 1 --2- 172.21.15.45:0/806898758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af40b3040 0x7f5af4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5afa1b4640 1 --2- 172.21.15.45:0/806898758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af40b3420 0x7f5af415c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5afa1b4640 1 -- 172.21.15.45:0/806898758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af40b3040 msgr2=0x7f5af4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5afa1b4640 1 --2- 172.21.15.45:0/806898758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af40b3040 0x7f5af4099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5afa1b4640 1 -- 172.21.15.45:0/806898758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae40094d0 con 0x7f5af415b270 2023-11-28T20:54:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5afa1b4640 1 --2- 172.21.15.45:0/806898758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 0x7f5af415c810 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00b970 tx=0x7f5aec00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5aea7fc640 1 -- 172.21.15.45:0/806898758 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec00cb40 con 0x7f5af415b270 2023-11-28T20:54:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5aea7fc640 1 -- 172.21.15.45:0/806898758 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec00ccc0 con 0x7f5af415b270 2023-11-28T20:54:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5aea7fc640 1 -- 172.21.15.45:0/806898758 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec014680 con 0x7f5af415b270 2023-11-28T20:54:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.063+0000 7f5afa9b5640 1 -- 172.21.15.45:0/806898758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af415bdb0 con 0x7f5af415b270 2023-11-28T20:54:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.064+0000 7f5afa9b5640 1 -- 172.21.15.45:0/806898758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af414d880 con 0x7f5af415b270 2023-11-28T20:54:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.065+0000 7f5afa9b5640 1 -- 172.21.15.45:0/806898758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af415b650 con 0x7f5af415b270 2023-11-28T20:54:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.066+0000 7f5aea7fc640 1 -- 172.21.15.45:0/806898758 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 75) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f5aec01d3b0 con 0x7f5af415b270 2023-11-28T20:54:55.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.067+0000 7f5aea7fc640 1 -- 172.21.15.45:0/806898758 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f5aec089560 con 0x7f5af415b270 2023-11-28T20:54:55.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.071+0000 7f5aea7fc640 1 -- 172.21.15.45:0/806898758 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5af415b650 con 0x7f5af415b270 2023-11-28T20:54:55.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.259+0000 7f5afa9b5640 1 -- 172.21.15.45:0/806898758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5af415b650 con 0x7f5af415b270 2023-11-28T20:54:55.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.259+0000 7f5aea7fc640 1 -- 172.21.15.45:0/806898758 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v150) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5aec019020 con 0x7f5af415b270 2023-11-28T20:54:55.261 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T20:54:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.262+0000 7f5afa9b5640 1 -- 172.21.15.45:0/806898758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 msgr2=0x7f5af415c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.262+0000 7f5afa9b5640 1 --2- 172.21.15.45:0/806898758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 0x7f5af415c810 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00b970 tx=0x7f5aec00be30 comp rx=0 tx=0).stop 2023-11-28T20:54:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.262+0000 7f5afa9b5640 1 -- 172.21.15.45:0/806898758 shutdown_connections 2023-11-28T20:54:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.262+0000 7f5afa9b5640 1 --2- 172.21.15.45:0/806898758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5af415b270 0x7f5af415c810 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.262+0000 7f5afa9b5640 1 --2- 172.21.15.45:0/806898758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5af40b3420 0x7f5af415c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.262+0000 7f5afa9b5640 1 --2- 172.21.15.45:0/806898758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5af40b3040 0x7f5af4099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.262+0000 7f5afa9b5640 1 -- 172.21.15.45:0/806898758 >> 172.21.15.45:0/806898758 conn(0x7f5af40b1050 msgr2=0x7f5af40ad3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.262+0000 7f5afa9b5640 1 -- 172.21.15.45:0/806898758 shutdown_connections 2023-11-28T20:54:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.262+0000 7f5afa9b5640 1 -- 172.21.15.45:0/806898758 wait complete. 2023-11-28T20:54:55.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T20:54:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.554+0000 7fbf9759e640 1 Processor -- start 2023-11-28T20:54:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.554+0000 7fbf9759e640 1 -- start start 2023-11-28T20:54:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf9759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbf980b3040 0x7fbf9806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf9759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf9815b270 0x7fbf9815f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf9759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9815faf0 con 0x7fbf980b3420 2023-11-28T20:54:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf9759e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf980ceaa0 con 0x7fbf980b3040 2023-11-28T20:54:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf9759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9806d410 con 0x7fbf9815b270 2023-11-28T20:54:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf9659c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbf980b3040 0x7fbf9806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf96d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbf980b3420 0x7fbf980b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40546/0 (socket says 172.21.15.45:40546) 2023-11-28T20:54:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf96d9d640 1 -- 172.21.15.45:0/712834996 learned_addr learned my addr 172.21.15.45:0/712834996 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf9659c640 1 --2- 172.21.15.45:0/712834996 >> v2:172.21.15.45:3300/0 conn(0x7fbf980b3040 0x7fbf9806cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49562/0 (socket says 172.21.15.45:49562) 2023-11-28T20:54:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.556+0000 7fbf95d9b640 1 --2- 172.21.15.45:0/712834996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf9815b270 0x7fbf9815f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.557+0000 7fbf9659c640 1 -- 172.21.15.45:0/712834996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf9815b270 msgr2=0x7fbf9815f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.557+0000 7fbf9659c640 1 --2- 172.21.15.45:0/712834996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf9815b270 0x7fbf9815f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.557+0000 7fbf9659c640 1 -- 172.21.15.45:0/712834996 >> v1:172.21.15.45:6789/0 conn(0x7fbf980b3420 legacy=0x7fbf980b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.557+0000 7fbf9659c640 1 -- 172.21.15.45:0/712834996 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9806d6f0 con 0x7fbf980b3040 2023-11-28T20:54:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.557+0000 7fbf9559a640 1 -- 172.21.15.45:0/712834996 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908494789 0 0) 0x7fbf9815faf0 con 0x7fbf980b3420 2023-11-28T20:54:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.558+0000 7fbf9659c640 1 --2- 172.21.15.45:0/712834996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf980b3040 0x7fbf9806cda0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fbf880097a0 tx=0x7fbf8802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb28f783f098a11 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:55.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.558+0000 7fbf9559a640 1 -- 172.21.15.45:0/712834996 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8802f050 con 0x7fbf980b3040 2023-11-28T20:54:55.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.558+0000 7fbf9559a640 1 -- 172.21.15.45:0/712834996 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf88031700 con 0x7fbf980b3040 2023-11-28T20:54:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.558+0000 7fbf9559a640 1 -- 172.21.15.45:0/712834996 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88031a90 con 0x7fbf980b3040 2023-11-28T20:54:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.559+0000 7fbf9759e640 1 -- 172.21.15.45:0/712834996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf980b3040 msgr2=0x7fbf9806cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.559+0000 7fbf9759e640 1 --2- 172.21.15.45:0/712834996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf980b3040 0x7fbf9806cda0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fbf880097a0 tx=0x7fbf8802b7a0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.559+0000 7fbf9759e640 1 -- 172.21.15.45:0/712834996 shutdown_connections 2023-11-28T20:54:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.559+0000 7fbf9759e640 1 --2- 172.21.15.45:0/712834996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf9815b270 0x7fbf9815f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.559+0000 7fbf9759e640 1 --2- 172.21.15.45:0/712834996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf980b3040 0x7fbf9806cda0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.559+0000 7fbf9759e640 1 -- 172.21.15.45:0/712834996 >> 172.21.15.45:0/712834996 conn(0x7fbf980b1050 msgr2=0x7fbf980cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.559+0000 7fbf9759e640 1 -- 172.21.15.45:0/712834996 shutdown_connections 2023-11-28T20:54:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.559+0000 7fbf9759e640 1 -- 172.21.15.45:0/712834996 wait complete. 2023-11-28T20:54:55.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.561+0000 7fbf9759e640 1 Processor -- start 2023-11-28T20:54:55.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.561+0000 7fbf9759e640 1 -- start start 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.562+0000 7fbf9759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf980b3040 0x7fbf9815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.562+0000 7fbf9759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf980b3420 0x7fbf980c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.562+0000 7fbf9759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9815b270 0x7fbf980c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.562+0000 7fbf9759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9815cc30 con 0x7fbf9815b270 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.562+0000 7fbf9759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9815ce60 con 0x7fbf980b3420 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.562+0000 7fbf9759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9815d140 con 0x7fbf980b3040 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf96d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9815b270 0x7fbf980c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf96d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9815b270 0x7fbf980c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49572/0 (socket says 172.21.15.45:49572) 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf96d9d640 1 -- 172.21.15.45:0/99747646 learned_addr learned my addr 172.21.15.45:0/99747646 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf9659c640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf980b3040 0x7fbf9815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf96d9d640 1 -- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf980b3040 msgr2=0x7fbf9815c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf96d9d640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf980b3040 0x7fbf9815c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf96d9d640 1 -- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf980b3420 msgr2=0x7fbf980c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf96d9d640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf980b3420 0x7fbf980c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf96d9d640 1 -- 172.21.15.45:0/99747646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf88009400 con 0x7fbf9815b270 2023-11-28T20:54:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.563+0000 7fbf96d9d640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9815b270 0x7fbf980c85e0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000c7a0 tx=0x7fbf9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.564+0000 7fbf86ffd640 1 -- 172.21.15.45:0/99747646 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9000a640 con 0x7fbf9815b270 2023-11-28T20:54:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.564+0000 7fbf86ffd640 1 -- 172.21.15.45:0/99747646 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9000a7a0 con 0x7fbf9815b270 2023-11-28T20:54:55.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.564+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9815d420 con 0x7fbf9815b270 2023-11-28T20:54:55.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.564+0000 7fbf86ffd640 1 -- 172.21.15.45:0/99747646 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9000abd0 con 0x7fbf9815b270 2023-11-28T20:54:55.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.564+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9815d9e0 con 0x7fbf9815b270 2023-11-28T20:54:55.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.565+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9815dc00 con 0x7fbf9815b270 2023-11-28T20:54:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.566+0000 7fbf86ffd640 1 -- 172.21.15.45:0/99747646 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 75) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fbf9000ad30 con 0x7fbf9815b270 2023-11-28T20:54:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.567+0000 7fbf86ffd640 1 -- 172.21.15.45:0/99747646 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7fbf90088b40 con 0x7fbf9815b270 2023-11-28T20:54:55.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.571+0000 7fbf86ffd640 1 -- 172.21.15.45:0/99747646 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf90053b10 con 0x7fbf9815b270 2023-11-28T20:54:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.612+0000 7fbf86ffd640 1 -- 172.21.15.45:0/99747646 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 76) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fbf900535c0 con 0x7fbf9815b270 2023-11-28T20:54:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.612+0000 7fbf86ffd640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbf78070ba0 0x7fbf78072f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.616+0000 7fbf9659c640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbf78070ba0 0x7fbf78072f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:55.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.629+0000 7fbf9659c640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbf78070ba0 0x7fbf78072f90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf88035000 tx=0x7fbf88008030 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:55.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.772+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf9804e920 con 0x7fbf9815b270 2023-11-28T20:54:55.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.773+0000 7fbf86ffd640 1 -- 172.21.15.45:0/99747646 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v150) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf900180a0 con 0x7fbf9815b270 2023-11-28T20:54:55.774 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T20:54:55.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.775+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbf78070ba0 msgr2=0x7fbf78072f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.775+0000 7fbf9759e640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbf78070ba0 0x7fbf78072f90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf88035000 tx=0x7fbf88008030 comp rx=0 tx=0).stop 2023-11-28T20:54:55.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.775+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9815b270 msgr2=0x7fbf980c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:55.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.775+0000 7fbf9759e640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9815b270 0x7fbf980c85e0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000c7a0 tx=0x7fbf9000cc60 comp rx=0 tx=0).stop 2023-11-28T20:54:55.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.776+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 shutdown_connections 2023-11-28T20:54:55.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.776+0000 7fbf9759e640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbf78070ba0 0x7fbf78072f90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.776+0000 7fbf9759e640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9815b270 0x7fbf980c85e0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.776+0000 7fbf9759e640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf980b3420 0x7fbf980c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.776+0000 7fbf9759e640 1 --2- 172.21.15.45:0/99747646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf980b3040 0x7fbf9815c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:55.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.776+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 >> 172.21.15.45:0/99747646 conn(0x7fbf980b1050 msgr2=0x7fbf980ac050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:55.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.776+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 shutdown_connections 2023-11-28T20:54:55.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:55.776+0000 7fbf9759e640 1 -- 172.21.15.45:0/99747646 wait complete. 2023-11-28T20:54:55.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:56.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.009+0000 7f88a06ad640 1 Processor -- start 2023-11-28T20:54:56.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.009+0000 7f88a06ad640 1 -- start start 2023-11-28T20:54:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.011+0000 7f88a06ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8898157bd0 0x7f889815da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.011+0000 7f88a06ad640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889815e0f0 con 0x7f88980ac8b0 2023-11-28T20:54:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.012+0000 7f88a06ad640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889815e3d0 con 0x7f88980ad8b0 2023-11-28T20:54:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.012+0000 7f88a06ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889815e990 con 0x7f8898157bd0 2023-11-28T20:54:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.012+0000 7f889e449640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8898157bd0 0x7f889815da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.012+0000 7f889e449640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8898157bd0 0x7f889815da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49580/0 (socket says 172.21.15.45:49580) 2023-11-28T20:54:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.012+0000 7f889e449640 1 -- 172.21.15.45:0/275137593 learned_addr learned my addr 172.21.15.45:0/275137593 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.012+0000 7f889e449640 1 -- 172.21.15.45:0/275137593 >> v1:172.21.15.190:6789/0 conn(0x7f88980ad8b0 legacy=0x7f889815f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.012+0000 7f889e449640 1 -- 172.21.15.45:0/275137593 >> v1:172.21.15.45:6789/0 conn(0x7f88980ac8b0 legacy=0x7f88980ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.012+0000 7f889e449640 1 -- 172.21.15.45:0/275137593 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889815e6b0 con 0x7f8898157bd0 2023-11-28T20:54:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.014+0000 7f889e449640 1 --2- 172.21.15.45:0/275137593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8898157bd0 0x7f889815da10 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f8894009870 tx=0x7f889402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a75e1ead8afe1e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:56.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.014+0000 7f889d447640 1 -- 172.21.15.45:0/275137593 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894031870 con 0x7f8898157bd0 2023-11-28T20:54:56.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.014+0000 7f889d447640 1 -- 172.21.15.45:0/275137593 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88940319d0 con 0x7f8898157bd0 2023-11-28T20:54:56.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.014+0000 7f889d447640 1 -- 172.21.15.45:0/275137593 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894031d60 con 0x7f8898157bd0 2023-11-28T20:54:56.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.014+0000 7f88a06ad640 1 -- 172.21.15.45:0/275137593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8898157bd0 msgr2=0x7f889815da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.014+0000 7f88a06ad640 1 --2- 172.21.15.45:0/275137593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8898157bd0 0x7f889815da10 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f8894009870 tx=0x7f889402b910 comp rx=0 tx=0).stop 2023-11-28T20:54:56.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.015+0000 7f88a06ad640 1 -- 172.21.15.45:0/275137593 shutdown_connections 2023-11-28T20:54:56.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.015+0000 7f88a06ad640 1 --2- 172.21.15.45:0/275137593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8898157bd0 0x7f889815da10 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.015+0000 7f88a06ad640 1 -- 172.21.15.45:0/275137593 >> 172.21.15.45:0/275137593 conn(0x7f88980b1350 msgr2=0x7f88980b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:56.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.015+0000 7f88a06ad640 1 -- 172.21.15.45:0/275137593 shutdown_connections 2023-11-28T20:54:56.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.015+0000 7f88a06ad640 1 -- 172.21.15.45:0/275137593 wait complete. 2023-11-28T20:54:56.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.016+0000 7f88a06ad640 1 Processor -- start 2023-11-28T20:54:56.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.017+0000 7f88a06ad640 1 -- start start 2023-11-28T20:54:56.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f88a06ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88980ac8b0 0x7f8898142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f88a06ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88980ad8b0 0x7f8898145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f88a06ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8898157bd0 0x7f88980ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f88a06ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88980cf710 con 0x7f88980ac8b0 2023-11-28T20:54:56.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f88a06ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88981443f0 con 0x7f8898157bd0 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f88a06ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88981431f0 con 0x7f88980ad8b0 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889e449640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88980ac8b0 0x7f8898142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889e449640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88980ac8b0 0x7f8898142cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49584/0 (socket says 172.21.15.45:49584) 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889e449640 1 -- 172.21.15.45:0/2732997406 learned_addr learned my addr 172.21.15.45:0/2732997406 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889dc48640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88980ad8b0 0x7f8898145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889ec4a640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8898157bd0 0x7f88980ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889e449640 1 -- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88980ad8b0 msgr2=0x7f8898145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889e449640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88980ad8b0 0x7f8898145f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889e449640 1 -- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8898157bd0 msgr2=0x7f88980ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889e449640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8898157bd0 0x7f88980ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889e449640 1 -- 172.21.15.45:0/2732997406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88940094d0 con 0x7f88980ac8b0 2023-11-28T20:54:56.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889dc48640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88980ad8b0 0x7f8898145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.018+0000 7f889ec4a640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8898157bd0 0x7f88980ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.019+0000 7f889e449640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88980ac8b0 0x7f8898142cb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f8894009870 tx=0x7f88940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.019+0000 7f888effd640 1 -- 172.21.15.45:0/2732997406 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894031820 con 0x7f88980ac8b0 2023-11-28T20:54:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.019+0000 7f888effd640 1 -- 172.21.15.45:0/2732997406 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88940443f0 con 0x7f88980ac8b0 2023-11-28T20:54:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.019+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8898146480 con 0x7f88980ac8b0 2023-11-28T20:54:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.019+0000 7f888effd640 1 -- 172.21.15.45:0/2732997406 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88940447a0 con 0x7f88980ac8b0 2023-11-28T20:54:56.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.019+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8898143ac0 con 0x7f88980ac8b0 2023-11-28T20:54:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.020+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88980733d0 con 0x7f88980ac8b0 2023-11-28T20:54:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.022+0000 7f888effd640 1 -- 172.21.15.45:0/2732997406 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 76) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f889404d450 con 0x7f88980ac8b0 2023-11-28T20:54:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.022+0000 7f888effd640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f887c06ac90 0x7f887c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.023+0000 7f888effd640 1 -- 172.21.15.45:0/2732997406 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f889402d040 con 0x7f88980ac8b0 2023-11-28T20:54:56.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.023+0000 7f889dc48640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f887c06ac90 0x7f887c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.024+0000 7f889dc48640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f887c06ac90 0x7f887c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f88980b0300 tx=0x7f8888009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:56.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.027+0000 7f888effd640 1 -- 172.21.15.45:0/2732997406 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8894076d40 con 0x7f88980ac8b0 2023-11-28T20:54:56.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.259+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f889815daa0 con 0x7f88980ac8b0 2023-11-28T20:54:56.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.260+0000 7f888effd640 1 -- 172.21.15.45:0/2732997406 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f889407a360 con 0x7f88980ac8b0 2023-11-28T20:54:56.262 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:56.262 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/1133315976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1133315976},{"type":"v1","addr":"172.21.15.190:6836","nonce":1133315976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2810069664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2810069664},{"type":"v1","addr":"172.21.15.190:6837","nonce":2810069664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1553174870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1553174870},{"type":"v1","addr":"172.21.15.45:6833","nonce":1553174870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:54:26.355691+0000","modified":"2023-11-28T20:54:29.430641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8595":{"gid":8595,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/684266044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":684266044},{"type":"v1","addr":"172.21.15.45:6835","nonce":684266044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-28T20:54:56.263 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 67 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f887c06ac90 msgr2=0x7f887c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f887c06ac90 0x7f887c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f88980b0300 tx=0x7f8888009040 comp rx=0 tx=0).stop 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88980ac8b0 msgr2=0x7f8898142cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88980ac8b0 0x7f8898142cb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f8894009870 tx=0x7f88940046c0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 shutdown_connections 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8898157bd0 0x7f88980ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88980ad8b0 0x7f8898145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f887c06ac90 0x7f887c06d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 --2- 172.21.15.45:0/2732997406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88980ac8b0 0x7f8898142cb0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 >> 172.21.15.45:0/2732997406 conn(0x7f88980b1350 msgr2=0x7f889806d480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.263+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 shutdown_connections 2023-11-28T20:54:56.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.264+0000 7f88a06ad640 1 -- 172.21.15.45:0/2732997406 wait complete. 2023-11-28T20:54:56.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:56.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.505+0000 7f15c359e640 1 Processor -- start 2023-11-28T20:54:56.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.506+0000 7f15c359e640 1 -- start start 2023-11-28T20:54:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.507+0000 7f15c359e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f15c40b3870 0x7f15c415db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.508+0000 7f15c359e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15c40721e0 0x7f15c415e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.508+0000 7f15c359e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c415e680 con 0x7f15c406d170 2023-11-28T20:54:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.508+0000 7f15c359e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c415a460 con 0x7f15c40721e0 2023-11-28T20:54:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.508+0000 7f15c359e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c415e960 con 0x7f15c40b3870 2023-11-28T20:54:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.508+0000 7f15c1d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15c40721e0 0x7f15c415e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.508+0000 7f15c2d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f15c406d170 0x7f15c40701c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40556/0 (socket says 172.21.15.45:40556) 2023-11-28T20:54:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.508+0000 7f15c2d9d640 1 -- 172.21.15.45:0/4162228116 learned_addr learned my addr 172.21.15.45:0/4162228116 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.508+0000 7f15c259c640 1 --2- 172.21.15.45:0/4162228116 >> v2:172.21.15.190:3300/0 conn(0x7f15c40b3870 0x7f15c415db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.509+0000 7f15c1d9b640 1 -- 172.21.15.45:0/4162228116 >> v2:172.21.15.190:3300/0 conn(0x7f15c40b3870 msgr2=0x7f15c415db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.509+0000 7f15c1d9b640 1 --2- 172.21.15.45:0/4162228116 >> v2:172.21.15.190:3300/0 conn(0x7f15c40b3870 0x7f15c415db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.509+0000 7f15c1d9b640 1 -- 172.21.15.45:0/4162228116 >> v1:172.21.15.45:6789/0 conn(0x7f15c406d170 legacy=0x7f15c40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.509+0000 7f15c1d9b640 1 -- 172.21.15.45:0/4162228116 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c4071700 con 0x7f15c40721e0 2023-11-28T20:54:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.510+0000 7f15c1d9b640 1 --2- 172.21.15.45:0/4162228116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 0x7f15c415e050 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f15c406df50 tx=0x7f15b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63c91094bbce36e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.510+0000 7f15c159a640 1 -- 172.21.15.45:0/4162228116 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8031830 con 0x7f15c40721e0 2023-11-28T20:54:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.510+0000 7f15c159a640 1 -- 172.21.15.45:0/4162228116 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b8031990 con 0x7f15c40721e0 2023-11-28T20:54:56.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.510+0000 7f15c159a640 1 -- 172.21.15.45:0/4162228116 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8031d20 con 0x7f15c40721e0 2023-11-28T20:54:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.511+0000 7f15c359e640 1 -- 172.21.15.45:0/4162228116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 msgr2=0x7f15c415e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.511+0000 7f15c359e640 1 --2- 172.21.15.45:0/4162228116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 0x7f15c415e050 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f15c406df50 tx=0x7f15b802b8d0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.511+0000 7f15c359e640 1 -- 172.21.15.45:0/4162228116 shutdown_connections 2023-11-28T20:54:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.511+0000 7f15c359e640 1 --2- 172.21.15.45:0/4162228116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 0x7f15c415e050 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.511+0000 7f15c359e640 1 --2- 172.21.15.45:0/4162228116 >> v2:172.21.15.190:3300/0 conn(0x7f15c40b3870 0x7f15c415db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.511+0000 7f15c359e640 1 -- 172.21.15.45:0/4162228116 >> 172.21.15.45:0/4162228116 conn(0x7f15c406ab30 msgr2=0x7f15c40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.511+0000 7f15c359e640 1 -- 172.21.15.45:0/4162228116 shutdown_connections 2023-11-28T20:54:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.511+0000 7f15c359e640 1 -- 172.21.15.45:0/4162228116 wait complete. 2023-11-28T20:54:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.513+0000 7f15c359e640 1 Processor -- start 2023-11-28T20:54:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.513+0000 7f15c359e640 1 -- start start 2023-11-28T20:54:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.514+0000 7f15c359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15c406d170 0x7f15c4153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.514+0000 7f15c359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 0x7f15c4151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.514+0000 7f15c359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c40b3870 0x7f15c4153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.514+0000 7f15c359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c409a000 con 0x7f15c40721e0 2023-11-28T20:54:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.514+0000 7f15c359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c409a2e0 con 0x7f15c40b3870 2023-11-28T20:54:56.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.514+0000 7f15c359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c409a5c0 con 0x7f15c406d170 2023-11-28T20:54:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.514+0000 7f15c1d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 0x7f15c4151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.514+0000 7f15c1d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 0x7f15c4151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49596/0 (socket says 172.21.15.45:49596) 2023-11-28T20:54:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.514+0000 7f15c1d9b640 1 -- 172.21.15.45:0/3328000474 learned_addr learned my addr 172.21.15.45:0/3328000474 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c259c640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15c406d170 0x7f15c4153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c2d9d640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c40b3870 0x7f15c4153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c1d9b640 1 -- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15c406d170 msgr2=0x7f15c4153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c1d9b640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15c406d170 0x7f15c4153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c1d9b640 1 -- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c40b3870 msgr2=0x7f15c4153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c1d9b640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c40b3870 0x7f15c4153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c1d9b640 1 -- 172.21.15.45:0/3328000474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b8009400 con 0x7f15c40721e0 2023-11-28T20:54:56.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c259c640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15c406d170 0x7f15c4153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c2d9d640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c40b3870 0x7f15c4153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c1d9b640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 0x7f15c4151870 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009890 tx=0x7f15b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15b2ffd640 1 -- 172.21.15.45:0/3328000474 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b803ba70 con 0x7f15c40721e0 2023-11-28T20:54:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15b2ffd640 1 -- 172.21.15.45:0/3328000474 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b803bbf0 con 0x7f15c40721e0 2023-11-28T20:54:56.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c409a8a0 con 0x7f15c40721e0 2023-11-28T20:54:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.515+0000 7f15b2ffd640 1 -- 172.21.15.45:0/3328000474 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8045400 con 0x7f15c40721e0 2023-11-28T20:54:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.516+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c4153b60 con 0x7f15c40721e0 2023-11-28T20:54:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.517+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c409adc0 con 0x7f15c40721e0 2023-11-28T20:54:56.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.518+0000 7f15b2ffd640 1 -- 172.21.15.45:0/3328000474 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 76) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f15b804e440 con 0x7f15c40721e0 2023-11-28T20:54:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.519+0000 7f15b2ffd640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f15a406acc0 0x7f15a406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.519+0000 7f15b2ffd640 1 -- 172.21.15.45:0/3328000474 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f15b80ab9a0 con 0x7f15c40721e0 2023-11-28T20:54:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.520+0000 7f15c259c640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f15a406acc0 0x7f15a406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.521+0000 7f15c259c640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f15a406acc0 0x7f15a406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f15b400b820 tx=0x7f15b4008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:56.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.523+0000 7f15b2ffd640 1 -- 172.21.15.45:0/3328000474 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15c409adc0 con 0x7f15c40721e0 2023-11-28T20:54:56.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.626+0000 7f15b2ffd640 1 -- 172.21.15.45:0/3328000474 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 77) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f15b8076d20 con 0x7f15c40721e0 2023-11-28T20:54:56.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.730+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15c409adc0 con 0x7f15c40721e0 2023-11-28T20:54:56.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.733+0000 7f15b2ffd640 1 -- 172.21.15.45:0/3328000474 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+17895 (secure 0 0 0) 0x7f15c409adc0 con 0x7f15c40721e0 2023-11-28T20:54:56.735 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:56.735 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":150,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:54.592393+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:54:24.325054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-11-28T20:54:25.374707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/279215056":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/3021259469":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/4229179752":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6833/1149934668":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6832/1149934668":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/1886852041":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/1445199812":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/649548475":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/3688544491":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/1625344948":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2599111328":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2598605134":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2976990772":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6833/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/305890924":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6832/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/1527814408":"2023-11-29T20:54:41.552066+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f15a406acc0 msgr2=0x7f15a406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f15a406acc0 0x7f15a406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f15b400b820 tx=0x7f15b4008040 comp rx=0 tx=0).stop 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 msgr2=0x7f15c4151870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 0x7f15c4151870 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009890 tx=0x7f15b80046c0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 shutdown_connections 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c40b3870 0x7f15c4153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f15a406acc0 0x7f15a406d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15c40721e0 0x7f15c4151870 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 --2- 172.21.15.45:0/3328000474 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15c406d170 0x7f15c4153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 >> 172.21.15.45:0/3328000474 conn(0x7f15c406ab30 msgr2=0x7f15c40ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 shutdown_connections 2023-11-28T20:54:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.736+0000 7f15c359e640 1 -- 172.21.15.45:0/3328000474 wait complete. 2023-11-28T20:54:56.756 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T20:54:56.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:56.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.971+0000 7f6732c8a640 1 Processor -- start 2023-11-28T20:54:56.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.971+0000 7f6732c8a640 1 -- start start 2023-11-28T20:54:56.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.973+0000 7f6732c8a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c0cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.973+0000 7f6732c8a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f672c0d0100 0x7f672c0d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.973+0000 7f6732c8a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672c0d0560 con 0x7f672c0d1ab0 2023-11-28T20:54:56.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.973+0000 7f6732c8a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f672c0d2230 con 0x7f672c0d0100 2023-11-28T20:54:56.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.973+0000 7f6732c8a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0d2510 con 0x7f672c15d550 2023-11-28T20:54:56.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.974+0000 7f6731c88640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c0cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.974+0000 7f6731487640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f672c0d0100 0x7f672c0d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.974+0000 7f6731c88640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c0cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:38898/0 (socket says 172.21.15.45:38898) 2023-11-28T20:54:56.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.974+0000 7f6731c88640 1 -- 172.21.15.45:0/1233519088 learned_addr learned my addr 172.21.15.45:0/1233519088 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:56.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.974+0000 7f6731c88640 1 -- 172.21.15.45:0/1233519088 >> v2:172.21.15.190:3300/0 conn(0x7f672c0d0100 msgr2=0x7f672c0d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:56.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.974+0000 7f6731c88640 1 --2- 172.21.15.45:0/1233519088 >> v2:172.21.15.190:3300/0 conn(0x7f672c0d0100 0x7f672c0d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.975+0000 7f6731c88640 1 -- 172.21.15.45:0/1233519088 >> v1:172.21.15.190:6789/0 conn(0x7f672c0d1ab0 legacy=0x7f672c070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.975+0000 7f6731c88640 1 -- 172.21.15.45:0/1233519088 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c0d27f0 con 0x7f672c15d550 2023-11-28T20:54:56.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.976+0000 7f6731c88640 1 --2- 172.21.15.45:0/1233519088 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c0cfbc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6728009870 tx=0x7f672802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21da551382a63829 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:56.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.976+0000 7f6730c86640 1 -- 172.21.15.45:0/1233519088 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728030840 con 0x7f672c15d550 2023-11-28T20:54:56.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6730c86640 1 -- 172.21.15.45:0/1233519088 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67280309a0 con 0x7f672c15d550 2023-11-28T20:54:56.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6730c86640 1 -- 172.21.15.45:0/1233519088 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728030d30 con 0x7f672c15d550 2023-11-28T20:54:56.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6732c8a640 1 -- 172.21.15.45:0/1233519088 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 msgr2=0x7f672c0cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6732c8a640 1 --2- 172.21.15.45:0/1233519088 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c0cfbc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6728009870 tx=0x7f672802b980 comp rx=0 tx=0).stop 2023-11-28T20:54:56.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6732c8a640 1 -- 172.21.15.45:0/1233519088 shutdown_connections 2023-11-28T20:54:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6732c8a640 1 --2- 172.21.15.45:0/1233519088 >> v2:172.21.15.190:3300/0 conn(0x7f672c0d0100 0x7f672c0d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6732c8a640 1 --2- 172.21.15.45:0/1233519088 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c0cfbc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6732c8a640 1 -- 172.21.15.45:0/1233519088 >> 172.21.15.45:0/1233519088 conn(0x7f672c06ac70 msgr2=0x7f672c0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6732c8a640 1 -- 172.21.15.45:0/1233519088 shutdown_connections 2023-11-28T20:54:56.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.977+0000 7f6732c8a640 1 -- 172.21.15.45:0/1233519088 wait complete. 2023-11-28T20:54:56.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.980+0000 7f6732c8a640 1 Processor -- start 2023-11-28T20:54:56.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.980+0000 7f6732c8a640 1 -- start start 2023-11-28T20:54:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.981+0000 7f6732c8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f672c0d0100 0x7f672c0c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.981+0000 7f6732c8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672c0d1ab0 0x7f672c0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.981+0000 7f6732c8a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.981+0000 7f6732c8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0c96f0 con 0x7f672c0d1ab0 2023-11-28T20:54:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.981+0000 7f6732c8a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0c8760 con 0x7f672c0d0100 2023-11-28T20:54:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.981+0000 7f6732c8a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0cd110 con 0x7f672c15d550 2023-11-28T20:54:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.981+0000 7f6731487640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672c0d1ab0 0x7f672c0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.981+0000 7f6731487640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672c0d1ab0 0x7f672c0ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49606/0 (socket says 172.21.15.45:49606) 2023-11-28T20:54:56.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.982+0000 7f6731487640 1 -- 172.21.15.45:0/3409954310 learned_addr learned my addr 172.21.15.45:0/3409954310 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:56.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.982+0000 7f6732489640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.982+0000 7f6731c88640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f672c0d0100 0x7f672c0c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.982+0000 7f6731487640 1 -- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 msgr2=0x7f672c145730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.982+0000 7f6731487640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c145730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.982+0000 7f6731487640 1 -- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f672c0d0100 msgr2=0x7f672c0c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.982+0000 7f6731487640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f672c0d0100 0x7f672c0c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.982+0000 7f6731487640 1 -- 172.21.15.45:0/3409954310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67280094d0 con 0x7f672c0d1ab0 2023-11-28T20:54:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.982+0000 7f6731487640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672c0d1ab0 0x7f672c0ca2a0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e930 tx=0x7f671c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.983+0000 7f67227fc640 1 -- 172.21.15.45:0/3409954310 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c009690 con 0x7f672c0d1ab0 2023-11-28T20:54:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.983+0000 7f67227fc640 1 -- 172.21.15.45:0/3409954310 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671c00ca00 con 0x7f672c0d1ab0 2023-11-28T20:54:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.983+0000 7f67227fc640 1 -- 172.21.15.45:0/3409954310 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c00cdb0 con 0x7f672c0d1ab0 2023-11-28T20:54:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.983+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c0cd3f0 con 0x7f672c0d1ab0 2023-11-28T20:54:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.983+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c0c8c90 con 0x7f672c0d1ab0 2023-11-28T20:54:56.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.984+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672c0c8e80 con 0x7f672c0d1ab0 2023-11-28T20:54:56.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.985+0000 7f67227fc640 1 -- 172.21.15.45:0/3409954310 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 77) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f671c013470 con 0x7f672c0d1ab0 2023-11-28T20:54:56.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.986+0000 7f67227fc640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f671006aed0 0x7f671006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:56.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.986+0000 7f67227fc640 1 -- 172.21.15.45:0/3409954310 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f671c089d00 con 0x7f672c0d1ab0 2023-11-28T20:54:56.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.987+0000 7f6731c88640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f671006aed0 0x7f671006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:56.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.988+0000 7f6731c88640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f671006aed0 0x7f671006d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6728036000 tx=0x7f672803b040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:56.990+0000 7f67227fc640 1 -- 172.21.15.45:0/3409954310 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f671c08d030 con 0x7f672c0d1ab0 2023-11-28T20:54:57.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.221+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f672c04e920 con 0x7f672c0d1ab0 2023-11-28T20:54:57.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.222+0000 7f67227fc640 1 -- 172.21.15.45:0/3409954310 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f671c054ad0 con 0x7f672c0d1ab0 2023-11-28T20:54:57.224 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:57.224 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/1133315976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1133315976},{"type":"v1","addr":"172.21.15.190:6836","nonce":1133315976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2810069664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2810069664},{"type":"v1","addr":"172.21.15.190:6837","nonce":2810069664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1553174870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1553174870},{"type":"v1","addr":"172.21.15.45:6833","nonce":1553174870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:54:26.355691+0000","modified":"2023-11-28T20:54:29.430641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8595":{"gid":8595,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/684266044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":684266044},{"type":"v1","addr":"172.21.15.45:6835","nonce":684266044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-28T20:54:57.224 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 67 2023-11-28T20:54:57.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f671006aed0 msgr2=0x7f671006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f671006aed0 0x7f671006d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6728036000 tx=0x7f672803b040 comp rx=0 tx=0).stop 2023-11-28T20:54:57.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672c0d1ab0 msgr2=0x7f672c0ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672c0d1ab0 0x7f672c0ca2a0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e930 tx=0x7f671c00edf0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 shutdown_connections 2023-11-28T20:54:57.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f672c15d550 0x7f672c145730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f671006aed0 0x7f671006d390 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f672c0d1ab0 0x7f672c0ca2a0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 --2- 172.21.15.45:0/3409954310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f672c0d0100 0x7f672c0c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 >> 172.21.15.45:0/3409954310 conn(0x7f672c06ac70 msgr2=0x7f672c0b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 shutdown_connections 2023-11-28T20:54:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.225+0000 7f6732c8a640 1 -- 172.21.15.45:0/3409954310 wait complete. 2023-11-28T20:54:57.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:54:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.472+0000 7fc151b99640 1 Processor -- start 2023-11-28T20:54:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.472+0000 7fc151b99640 1 -- start start 2023-11-28T20:54:57.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.474+0000 7fc151b99640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc14c15d550 0x7fc14c0cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.474+0000 7fc151b99640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14c0d2230 con 0x7fc14c0d1c80 2023-11-28T20:54:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.474+0000 7fc151b99640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14c0d2510 con 0x7fc14c0d0100 2023-11-28T20:54:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.474+0000 7fc151b99640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14c0d0590 con 0x7fc14c15d550 2023-11-28T20:54:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.474+0000 7fc150b97640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc14c15d550 0x7fc14c0cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.474+0000 7fc150b97640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc14c15d550 0x7fc14c0cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49610/0 (socket says 172.21.15.45:49610) 2023-11-28T20:54:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.474+0000 7fc150b97640 1 -- 172.21.15.45:0/463460570 learned_addr learned my addr 172.21.15.45:0/463460570 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.474+0000 7fc150b97640 1 -- 172.21.15.45:0/463460570 >> v1:172.21.15.190:6789/0 conn(0x7fc14c0d0100 legacy=0x7fc14c0d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.475+0000 7fc150b97640 1 -- 172.21.15.45:0/463460570 >> v1:172.21.15.45:6789/0 conn(0x7fc14c0d1c80 legacy=0x7fc14c0b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.475+0000 7fc150b97640 1 -- 172.21.15.45:0/463460570 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c0d27f0 con 0x7fc14c15d550 2023-11-28T20:54:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.476+0000 7fc150b97640 1 --2- 172.21.15.45:0/463460570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c15d550 0x7fc14c0cfbc0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fc148009e50 tx=0x7fc14802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45fcebbaa7f8180 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:57.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.476+0000 7fc1437fe640 1 -- 172.21.15.45:0/463460570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14802cb10 con 0x7fc14c15d550 2023-11-28T20:54:57.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.476+0000 7fc1437fe640 1 -- 172.21.15.45:0/463460570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14802cc70 con 0x7fc14c15d550 2023-11-28T20:54:57.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.476+0000 7fc1437fe640 1 -- 172.21.15.45:0/463460570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1480343f0 con 0x7fc14c15d550 2023-11-28T20:54:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.477+0000 7fc151b99640 1 -- 172.21.15.45:0/463460570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c15d550 msgr2=0x7fc14c0cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.477+0000 7fc151b99640 1 --2- 172.21.15.45:0/463460570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c15d550 0x7fc14c0cfbc0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fc148009e50 tx=0x7fc14802bf70 comp rx=0 tx=0).stop 2023-11-28T20:54:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.477+0000 7fc151b99640 1 -- 172.21.15.45:0/463460570 shutdown_connections 2023-11-28T20:54:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.477+0000 7fc151b99640 1 --2- 172.21.15.45:0/463460570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c15d550 0x7fc14c0cfbc0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.477+0000 7fc151b99640 1 -- 172.21.15.45:0/463460570 >> 172.21.15.45:0/463460570 conn(0x7fc14c06ac70 msgr2=0x7fc14c0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.477+0000 7fc151b99640 1 -- 172.21.15.45:0/463460570 shutdown_connections 2023-11-28T20:54:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.477+0000 7fc151b99640 1 -- 172.21.15.45:0/463460570 wait complete. 2023-11-28T20:54:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.480+0000 7fc151b99640 1 Processor -- start 2023-11-28T20:54:57.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.480+0000 7fc151b99640 1 -- start start 2023-11-28T20:54:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.481+0000 7fc151b99640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c0d0100 0x7fc14c151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.481+0000 7fc151b99640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc14c0d1c80 0x7fc14c0ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.481+0000 7fc151b99640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc14c15d550 0x7fc14c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.481+0000 7fc151b99640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c156e10 con 0x7fc14c0d0100 2023-11-28T20:54:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.481+0000 7fc151b99640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c156c10 con 0x7fc14c0d1c80 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.481+0000 7fc151b99640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c0ccb20 con 0x7fc14c15d550 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.481+0000 7fc150b97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c0d0100 0x7fc14c151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc150b97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c0d0100 0x7fc14c151420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49616/0 (socket says 172.21.15.45:49616) 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc150b97640 1 -- 172.21.15.45:0/1591434089 learned_addr learned my addr 172.21.15.45:0/1591434089 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc150b97640 1 -- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc14c15d550 msgr2=0x7fc14c0cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc143fff640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc14c0d1c80 0x7fc14c0ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc151398640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc14c15d550 0x7fc14c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc150b97640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc14c15d550 0x7fc14c0cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc150b97640 1 -- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc14c0d1c80 msgr2=0x7fc14c0ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc150b97640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc14c0d1c80 0x7fc14c0ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc150b97640 1 -- 172.21.15.45:0/1591434089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009b30 con 0x7fc14c0d0100 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc143fff640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc14c0d1c80 0x7fc14c0ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc151398640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc14c15d550 0x7fc14c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.482+0000 7fc150b97640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c0d0100 0x7fc14c151420 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc148009e50 tx=0x7fc148004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:57.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.483+0000 7fc1417fa640 1 -- 172.21.15.45:0/1591434089 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14803e450 con 0x7fc14c0d0100 2023-11-28T20:54:57.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.483+0000 7fc1417fa640 1 -- 172.21.15.45:0/1591434089 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14803e5f0 con 0x7fc14c0d0100 2023-11-28T20:54:57.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.483+0000 7fc1417fa640 1 -- 172.21.15.45:0/1591434089 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14803e9a0 con 0x7fc14c0d0100 2023-11-28T20:54:57.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.483+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c0cce00 con 0x7fc14c0d0100 2023-11-28T20:54:57.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.483+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14c0cc800 con 0x7fc14c0d0100 2023-11-28T20:54:57.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.484+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14c0ccc60 con 0x7fc14c0d0100 2023-11-28T20:54:57.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.485+0000 7fc1417fa640 1 -- 172.21.15.45:0/1591434089 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 77) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fc14803eb00 con 0x7fc14c0d0100 2023-11-28T20:54:57.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.486+0000 7fc1417fa640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc13806aed0 0x7fc13806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.486+0000 7fc1417fa640 1 -- 172.21.15.45:0/1591434089 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7fc1480acbf0 con 0x7fc14c0d0100 2023-11-28T20:54:57.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.487+0000 7fc143fff640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc13806aed0 0x7fc13806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.488+0000 7fc143fff640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc13806aed0 0x7fc13806d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc14c151960 tx=0x7fc13c009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:57.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.490+0000 7fc1417fa640 1 -- 172.21.15.45:0/1591434089 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc14c0ccc60 con 0x7fc14c0d0100 2023-11-28T20:54:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.723+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc14c0ccc60 con 0x7fc14c0d0100 2023-11-28T20:54:57.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.724+0000 7fc1417fa640 1 -- 172.21.15.45:0/1591434089 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fc14c0ccc60 con 0x7fc14c0d0100 2023-11-28T20:54:57.726 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:57.726 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/1133315976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1133315976},{"type":"v1","addr":"172.21.15.190:6836","nonce":1133315976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2810069664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2810069664},{"type":"v1","addr":"172.21.15.190:6837","nonce":2810069664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1553174870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1553174870},{"type":"v1","addr":"172.21.15.45:6833","nonce":1553174870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:54:26.355691+0000","modified":"2023-11-28T20:54:29.430641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8595":{"gid":8595,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/684266044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":684266044},{"type":"v1","addr":"172.21.15.45:6835","nonce":684266044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-11-28T20:54:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 67 2023-11-28T20:54:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.727+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc13806aed0 msgr2=0x7fc13806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.727+0000 7fc151b99640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc13806aed0 0x7fc13806d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc14c151960 tx=0x7fc13c009040 comp rx=0 tx=0).stop 2023-11-28T20:54:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.727+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c0d0100 msgr2=0x7fc14c151420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.727+0000 7fc151b99640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c0d0100 0x7fc14c151420 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc148009e50 tx=0x7fc148004a40 comp rx=0 tx=0).stop 2023-11-28T20:54:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.727+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 shutdown_connections 2023-11-28T20:54:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.727+0000 7fc151b99640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc14c15d550 0x7fc14c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.727+0000 7fc151b99640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc14c0d1c80 0x7fc14c0ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.728+0000 7fc151b99640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc13806aed0 0x7fc13806d390 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.728+0000 7fc151b99640 1 --2- 172.21.15.45:0/1591434089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc14c0d0100 0x7fc14c151420 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.728+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 >> 172.21.15.45:0/1591434089 conn(0x7fc14c06ac70 msgr2=0x7fc14c15e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.728+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 shutdown_connections 2023-11-28T20:54:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.728+0000 7fc151b99640 1 -- 172.21.15.45:0/1591434089 wait complete. 2023-11-28T20:54:57.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:54:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.976+0000 7fd9ab59e640 1 Processor -- start 2023-11-28T20:54:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.976+0000 7fd9ab59e640 1 -- start start 2023-11-28T20:54:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.977+0000 7fd9ab59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.977+0000 7fd9ab59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd9ac0b6560 0x7fd9ac0cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.977+0000 7fd9ab59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0d1250 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.977+0000 7fd9ab59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac0ce6b0 con 0x7fd9ac0b6560 2023-11-28T20:54:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.977+0000 7fd9ab59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0ce990 con 0x7fd9ac0cfaf0 2023-11-28T20:54:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.977+0000 7fd9a9d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd9ac0b6560 0x7fd9ac0cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.977+0000 7fd9a9d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd9ac0b6560 0x7fd9ac0cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49626/0 (socket says 172.21.15.45:49626) 2023-11-28T20:54:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.977+0000 7fd9a9d9b640 1 -- 172.21.15.45:0/3942117669 learned_addr learned my addr 172.21.15.45:0/3942117669 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.978+0000 7fd9aa59c640 1 --2- 172.21.15.45:0/3942117669 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.978+0000 7fd9a9d9b640 1 -- 172.21.15.45:0/3942117669 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0cfaf0 msgr2=0x7fd9ac0cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:57.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.978+0000 7fd9a9d9b640 1 --2- 172.21.15.45:0/3942117669 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0cfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.978+0000 7fd9a9d9b640 1 -- 172.21.15.45:0/3942117669 >> v1:172.21.15.45:6789/0 conn(0x7fd9ac0b69c0 legacy=0x7fd9ac15cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.978+0000 7fd9a9d9b640 1 -- 172.21.15.45:0/3942117669 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac0d14a0 con 0x7fd9ac0b6560 2023-11-28T20:54:57.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.978+0000 7fd9aa59c640 1 --2- 172.21.15.45:0/3942117669 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0cfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:57.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.979+0000 7fd9a9d9b640 1 --2- 172.21.15.45:0/3942117669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b6560 0x7fd9ac0cdad0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd994009e50 tx=0x7fd99402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561821d83c4cf0fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.979+0000 7fd9a959a640 1 -- 172.21.15.45:0/3942117669 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99402cb00 con 0x7fd9ac0b6560 2023-11-28T20:54:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.979+0000 7fd9a959a640 1 -- 172.21.15.45:0/3942117669 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99402cc60 con 0x7fd9ac0b6560 2023-11-28T20:54:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.979+0000 7fd9a959a640 1 -- 172.21.15.45:0/3942117669 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9940343f0 con 0x7fd9ac0b6560 2023-11-28T20:54:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.980+0000 7fd9ab59e640 1 -- 172.21.15.45:0/3942117669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b6560 msgr2=0x7fd9ac0cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.980+0000 7fd9ab59e640 1 --2- 172.21.15.45:0/3942117669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b6560 0x7fd9ac0cdad0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd994009e50 tx=0x7fd99402bf60 comp rx=0 tx=0).stop 2023-11-28T20:54:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.980+0000 7fd9ab59e640 1 -- 172.21.15.45:0/3942117669 shutdown_connections 2023-11-28T20:54:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.980+0000 7fd9ab59e640 1 --2- 172.21.15.45:0/3942117669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b6560 0x7fd9ac0cdad0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.980+0000 7fd9ab59e640 1 --2- 172.21.15.45:0/3942117669 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0cfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.980+0000 7fd9ab59e640 1 -- 172.21.15.45:0/3942117669 >> 172.21.15.45:0/3942117669 conn(0x7fd9ac0b4c40 msgr2=0x7fd9ac0b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.980+0000 7fd9ab59e640 1 -- 172.21.15.45:0/3942117669 shutdown_connections 2023-11-28T20:54:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.980+0000 7fd9ab59e640 1 -- 172.21.15.45:0/3942117669 wait complete. 2023-11-28T20:54:57.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.982+0000 7fd9ab59e640 1 Processor -- start 2023-11-28T20:54:57.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.982+0000 7fd9ab59e640 1 -- start start 2023-11-28T20:54:57.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.983+0000 7fd9ab59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0b6560 0x7fd9ac0ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.983+0000 7fd9ab59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b69c0 0x7fd9ac15dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.983+0000 7fd9ab59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.983+0000 7fd9ab59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0cb7d0 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.983+0000 7fd9ab59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0cabc0 con 0x7fd9ac0cfaf0 2023-11-28T20:54:57.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.983+0000 7fd9ab59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0caea0 con 0x7fd9ac0b6560 2023-11-28T20:54:57.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.983+0000 7fd9a9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b69c0 0x7fd9ac15dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9a9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b69c0 0x7fd9ac15dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49634/0 (socket says 172.21.15.45:49634) 2023-11-28T20:54:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9a9d9b640 1 -- 172.21.15.45:0/2943417919 learned_addr learned my addr 172.21.15.45:0/2943417919 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9aa59c640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0b6560 0x7fd9ac0ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9a9d9b640 1 -- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0b6560 msgr2=0x7fd9ac0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9aad9d640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9a9d9b640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0b6560 0x7fd9ac0ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9a9d9b640 1 -- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0cfaf0 msgr2=0x7fd9ac0c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9a9d9b640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9a9d9b640 1 -- 172.21.15.45:0/2943417919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994009b30 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9aa59c640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0b6560 0x7fd9ac0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9aad9d640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.984+0000 7fd9a9d9b640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b69c0 0x7fd9ac15dba0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd994000f80 tx=0x7fd994004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.985+0000 7fd9a2ffd640 1 -- 172.21.15.45:0/2943417919 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99403e450 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.985+0000 7fd9a2ffd640 1 -- 172.21.15.45:0/2943417919 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99403e5f0 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.985+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac0cb180 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.985+0000 7fd9a2ffd640 1 -- 172.21.15.45:0/2943417919 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99403e9a0 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.985+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac0c8bc0 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.986+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac04e920 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.987+0000 7fd9a2ffd640 1 -- 172.21.15.45:0/2943417919 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 77) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fd99403eb00 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.988+0000 7fd9a2ffd640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fd98c06af20 0x7fd98c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.988+0000 7fd9a2ffd640 1 -- 172.21.15.45:0/2943417919 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7fd9940ac9b0 con 0x7fd9ac0b69c0 2023-11-28T20:54:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.989+0000 7fd9aa59c640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fd98c06af20 0x7fd98c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.990+0000 7fd9aa59c640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fd98c06af20 0x7fd98c06d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd99c006fd0 tx=0x7fd99c008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:57.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:57.992+0000 7fd9a2ffd640 1 -- 172.21.15.45:0/2943417919 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd994077660 con 0x7fd9ac0b69c0 2023-11-28T20:54:58.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.186+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9ac0d1030 con 0x7fd9ac0b69c0 2023-11-28T20:54:58.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.188+0000 7fd9a2ffd640 1 -- 172.21.15.45:0/2943417919 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+17895 (secure 0 0 0) 0x7fd99403edb0 con 0x7fd9ac0b69c0 2023-11-28T20:54:58.190 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:54:58.190 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":150,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:54:54.592393+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:54:24.325054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-11-28T20:54:25.374707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/279215056":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/3021259469":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/4229179752":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6833/1149934668":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6832/1149934668":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/1886852041":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/1445199812":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/649548475":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/3688544491":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/1625344948":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2599111328":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2598605134":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2976990772":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6833/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/305890924":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6832/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/1527814408":"2023-11-29T20:54:41.552066+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:54:58.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fd98c06af20 msgr2=0x7fd98c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fd98c06af20 0x7fd98c06d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd99c006fd0 tx=0x7fd99c008040 comp rx=0 tx=0).stop 2023-11-28T20:54:58.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b69c0 msgr2=0x7fd9ac15dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b69c0 0x7fd9ac15dba0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd994000f80 tx=0x7fd994004a40 comp rx=0 tx=0).stop 2023-11-28T20:54:58.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 shutdown_connections 2023-11-28T20:54:58.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0cfaf0 0x7fd9ac0c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fd98c06af20 0x7fd98c06d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac0b69c0 0x7fd9ac15dba0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 --2- 172.21.15.45:0/2943417919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9ac0b6560 0x7fd9ac0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 >> 172.21.15.45:0/2943417919 conn(0x7fd9ac0b4c40 msgr2=0x7fd9ac157b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:58.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 shutdown_connections 2023-11-28T20:54:58.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.191+0000 7fd9ab59e640 1 -- 172.21.15.45:0/2943417919 wait complete. 2023-11-28T20:54:58.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T20:54:58.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.436+0000 7f0754eaa640 1 Processor -- start 2023-11-28T20:54:58.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.436+0000 7f0754eaa640 1 -- start start 2023-11-28T20:54:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.437+0000 7f0754eaa640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f075015b070 0x7f07500ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.437+0000 7f0754eaa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500abd70 con 0x7f07500cf610 2023-11-28T20:54:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.437+0000 7f0754eaa640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500b3ed0 con 0x7f07500b2fb0 2023-11-28T20:54:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.437+0000 7f0754eaa640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075015b450 con 0x7f075015b070 2023-11-28T20:54:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.438+0000 7f074ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07500cf610 0x7f075015eb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40580/0 (socket says 172.21.15.45:40580) 2023-11-28T20:54:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.438+0000 7f074ffff640 1 -- 172.21.15.45:0/354610297 learned_addr learned my addr 172.21.15.45:0/354610297 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:54:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.438+0000 7f074f7fe640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f075015b070 0x7f07500ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.438+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260179322 0 0) 0x7f07500abd70 con 0x7f07500cf610 2023-11-28T20:54:58.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.438+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f073c003520 con 0x7f07500cf610 2023-11-28T20:54:58.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.438+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3815387198 0 0) 0x7f073c003520 con 0x7f07500cf610 2023-11-28T20:54:58.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f075015b070 msgr2=0x7f07500ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:58.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f075015b070 0x7f07500ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 >> v1:172.21.15.190:6789/0 conn(0x7f07500b2fb0 legacy=0x7f07500cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07500abfc0 con 0x7f07500cf610 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f0748001670 con 0x7f07500cf610 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 >> v1:172.21.15.45:6789/0 conn(0x7f07500cf610 legacy=0x7f075015eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f075015b070 0x7f073c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c003f70 0x7f073c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c004d00 0x7f073c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c00bb40 con 0x7f073c004d00 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c00bcb0 con 0x7f073c003f70 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c00be70 con 0x7f075015b070 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074ffff640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c004d00 0x7f073c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074effd640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c003f70 0x7f073c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074f7fe640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f075015b070 0x7f073c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074ffff640 1 -- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f075015b070 msgr2=0x7f073c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074ffff640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f075015b070 0x7f073c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074ffff640 1 -- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c003f70 msgr2=0x7f073c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074ffff640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c003f70 0x7f073c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.439+0000 7f074ffff640 1 -- 172.21.15.45:0/354610297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c003520 con 0x7f073c004d00 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.440+0000 7f074f7fe640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f075015b070 0x7f073c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.441+0000 7f074ffff640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c004d00 0x7f073c005180 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f075006d560 tx=0x7f073c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fad57e3fc43af3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.441+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0748005ad0 con 0x7f073c004d00 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.441+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0748005c30 con 0x7f073c004d00 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.441+0000 7f074e7fc640 1 -- 172.21.15.45:0/354610297 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074802ca00 con 0x7f073c004d00 2023-11-28T20:54:58.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.441+0000 7f0754eaa640 1 -- 172.21.15.45:0/354610297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c004d00 msgr2=0x7f073c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.441+0000 7f0754eaa640 1 --2- 172.21.15.45:0/354610297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c004d00 0x7f073c005180 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f075006d560 tx=0x7f073c0057f0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.441+0000 7f074ffff640 1 -- 172.21.15.45:0/354610297 reap_dead start 2023-11-28T20:54:58.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.442+0000 7f0754eaa640 1 -- 172.21.15.45:0/354610297 shutdown_connections 2023-11-28T20:54:58.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.442+0000 7f0754eaa640 1 -- 172.21.15.45:0/354610297 >> 172.21.15.45:0/354610297 conn(0x7f075006a710 msgr2=0x7f075006ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:58.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.442+0000 7f0754eaa640 1 -- 172.21.15.45:0/354610297 shutdown_connections 2023-11-28T20:54:58.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.442+0000 7f0754eaa640 1 -- 172.21.15.45:0/354610297 wait complete. 2023-11-28T20:54:58.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.443+0000 7f0754eaa640 1 Processor -- start 2023-11-28T20:54:58.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.444+0000 7f0754eaa640 1 -- start start 2023-11-28T20:54:58.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f0754eaa640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f073c004d00 0x7f07501571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f0754eaa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0750150920 0x7f075015e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f0754eaa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0750143820 0x7f075009b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f0754eaa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07501578e0 con 0x7f0750150920 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f0754eaa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075015e5a0 con 0x7f0750143820 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f0754eaa640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075015cf20 con 0x7f073c004d00 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0750150920 0x7f075015e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0750150920 0x7f075015e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49650/0 (socket says 172.21.15.45:49650) 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074effd640 1 -- 172.21.15.45:0/2496037860 learned_addr learned my addr 172.21.15.45:0/2496037860 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074ffff640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0750143820 0x7f075009b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074f7fe640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f073c004d00 0x7f07501571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074effd640 1 -- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f073c004d00 msgr2=0x7f07501571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074effd640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f073c004d00 0x7f07501571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074effd640 1 -- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0750143820 msgr2=0x7f075009b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074effd640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0750143820 0x7f075009b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074effd640 1 -- 172.21.15.45:0/2496037860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075015d2a0 con 0x7f0750150920 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.445+0000 7f074ffff640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0750143820 0x7f075009b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.446+0000 7f074effd640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0750150920 0x7f075015e060 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f073800e850 tx=0x7f073800ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.446+0000 7f0747fff640 1 -- 172.21.15.45:0/2496037860 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738003b80 con 0x7f0750150920 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.446+0000 7f0747fff640 1 -- 172.21.15.45:0/2496037860 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738003ce0 con 0x7f0750150920 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.446+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075015d580 con 0x7f0750150920 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.446+0000 7f0747fff640 1 -- 172.21.15.45:0/2496037860 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380052f0 con 0x7f0750150920 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.446+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075015db40 con 0x7f0750150920 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.447+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075015dcd0 con 0x7f0750150920 2023-11-28T20:54:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.448+0000 7f0747fff640 1 -- 172.21.15.45:0/2496037860 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 77) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f0738005450 con 0x7f0750150920 2023-11-28T20:54:58.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.449+0000 7f0747fff640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f073006aed0 0x7f073006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.449+0000 7f0747fff640 1 -- 172.21.15.45:0/2496037860 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f0738089bf0 con 0x7f0750150920 2023-11-28T20:54:58.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.450+0000 7f074f7fe640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f073006aed0 0x7f073006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.451+0000 7f074f7fe640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f073006aed0 0x7f073006d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07400099c0 tx=0x7f0740008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:58.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.454+0000 7f0747fff640 1 -- 172.21.15.45:0/2496037860 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0738054800 con 0x7f0750150920 2023-11-28T20:54:58.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.650+0000 7f0747fff640 1 -- 172.21.15.45:0/2496037860 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f0738005700 con 0x7f0750150920 2023-11-28T20:54:58.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.694+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f07500ceea0 con 0x7f0750150920 2023-11-28T20:54:58.696 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T20:54:58.695+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T20:54:58.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.698+0000 7f0747fff640 1 -- 172.21.15.45:0/2496037860 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) v1 ==== 161+0+0 (secure 0 0 0) 0x7f07500ceea0 con 0x7f0750150920 2023-11-28T20:54:58.699 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T20:54:58.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f073006aed0 msgr2=0x7f073006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f073006aed0 0x7f073006d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07400099c0 tx=0x7f0740008040 comp rx=0 tx=0).stop 2023-11-28T20:54:58.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0750150920 msgr2=0x7f075015e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0750150920 0x7f075015e060 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f073800e850 tx=0x7f073800ed10 comp rx=0 tx=0).stop 2023-11-28T20:54:58.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 shutdown_connections 2023-11-28T20:54:58.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0750143820 0x7f075009b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f073006aed0 0x7f073006d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0750150920 0x7f075015e060 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 --2- 172.21.15.45:0/2496037860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f073c004d00 0x7f07501571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 >> 172.21.15.45:0/2496037860 conn(0x7f075006a710 msgr2=0x7f07500b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:58.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 shutdown_connections 2023-11-28T20:54:58.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.702+0000 7f0754eaa640 1 -- 172.21.15.45:0/2496037860 wait complete. 2023-11-28T20:54:58.722 INFO:tasks.ceph.mds.c.smithi045.stderr: -8832> 2023-11-28T20:54:31.439+0000 7f934467c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:54:58.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T20:54:58.780 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:54:58.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.935+0000 7fdff5f83640 1 Processor -- start 2023-11-28T20:54:58.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.935+0000 7fdff5f83640 1 -- start start 2023-11-28T20:54:58.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.936+0000 7fdff5f83640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdff006a500 0x7fdff006a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.936+0000 7fdff5f83640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff006ca60 con 0x7fdff015ddd0 2023-11-28T20:54:58.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.936+0000 7fdff5f83640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff006aea0 con 0x7fdff0157c10 2023-11-28T20:54:58.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.936+0000 7fdff5f83640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff015e930 con 0x7fdff006a500 2023-11-28T20:54:58.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.937+0000 7fdfeeffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdff006a500 0x7fdff006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.937+0000 7fdfeeffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdff006a500 0x7fdff006a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49692/0 (socket says 172.21.15.45:49692) 2023-11-28T20:54:58.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.937+0000 7fdfeeffd640 1 -- 172.21.15.45:0/2392623973 learned_addr learned my addr 172.21.15.45:0/2392623973 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:58.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.937+0000 7fdfeffff640 1 --1- 172.21.15.45:0/2392623973 >> v1:172.21.15.45:6789/0 conn(0x7fdff015ddd0 0x7fdff015e220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40600/0 (socket says 172.21.15.45:40600) 2023-11-28T20:54:58.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.937+0000 7fdfeeffd640 1 -- 172.21.15.45:0/2392623973 >> v1:172.21.15.190:6789/0 conn(0x7fdff0157c10 legacy=0x7fdff006da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.937+0000 7fdfeeffd640 1 -- 172.21.15.45:0/2392623973 >> v1:172.21.15.45:6789/0 conn(0x7fdff015ddd0 legacy=0x7fdff015e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.937+0000 7fdfee7fc640 1 -- 172.21.15.45:0/2392623973 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054400077 0 0) 0x7fdff006ca60 con 0x7fdff015ddd0 2023-11-28T20:54:58.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.937+0000 7fdfeeffd640 1 -- 172.21.15.45:0/2392623973 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff015eb40 con 0x7fdff006a500 2023-11-28T20:54:58.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.939+0000 7fdfeeffd640 1 --2- 172.21.15.45:0/2392623973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 0x7fdff006a960 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800bb10 tx=0x7fdfd802e200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd87f8fa7db98b39 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:58.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.939+0000 7fdfee7fc640 1 -- 172.21.15.45:0/2392623973 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8036440 con 0x7fdff006a500 2023-11-28T20:54:58.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.939+0000 7fdfee7fc640 1 -- 172.21.15.45:0/2392623973 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd80365a0 con 0x7fdff006a500 2023-11-28T20:54:58.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.939+0000 7fdfee7fc640 1 -- 172.21.15.45:0/2392623973 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8036930 con 0x7fdff006a500 2023-11-28T20:54:58.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.939+0000 7fdff5f83640 1 -- 172.21.15.45:0/2392623973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 msgr2=0x7fdff006a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.939+0000 7fdff5f83640 1 --2- 172.21.15.45:0/2392623973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 0x7fdff006a960 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800bb10 tx=0x7fdfd802e200 comp rx=0 tx=0).stop 2023-11-28T20:54:58.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.940+0000 7fdff5f83640 1 -- 172.21.15.45:0/2392623973 shutdown_connections 2023-11-28T20:54:58.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.940+0000 7fdff5f83640 1 --2- 172.21.15.45:0/2392623973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 0x7fdff006a960 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.940+0000 7fdff5f83640 1 -- 172.21.15.45:0/2392623973 >> 172.21.15.45:0/2392623973 conn(0x7fdff00ced60 msgr2=0x7fdff00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:58.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.940+0000 7fdff5f83640 1 -- 172.21.15.45:0/2392623973 shutdown_connections 2023-11-28T20:54:58.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.940+0000 7fdff5f83640 1 -- 172.21.15.45:0/2392623973 wait complete. 2023-11-28T20:54:58.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.942+0000 7fdff5f83640 1 Processor -- start 2023-11-28T20:54:58.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.942+0000 7fdff5f83640 1 -- start start 2023-11-28T20:54:58.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.943+0000 7fdff5f83640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 0x7fdff00ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.943+0000 7fdff5f83640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdff0157c10 0x7fdff0156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.943+0000 7fdff5f83640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdff015ddd0 0x7fdff0156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.943+0000 7fdff5f83640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff01571d0 con 0x7fdff006a500 2023-11-28T20:54:58.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.943+0000 7fdff5f83640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff015c7d0 con 0x7fdff015ddd0 2023-11-28T20:54:58.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.943+0000 7fdff5f83640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff015b750 con 0x7fdff0157c10 2023-11-28T20:54:58.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.943+0000 7fdfef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 0x7fdff00ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.943+0000 7fdfef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 0x7fdff00ad030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49704/0 (socket says 172.21.15.45:49704) 2023-11-28T20:54:58.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.943+0000 7fdfef7fe640 1 -- 172.21.15.45:0/3219109860 learned_addr learned my addr 172.21.15.45:0/3219109860 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfeffff640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdff015ddd0 0x7fdff0156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfef7fe640 1 -- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdff0157c10 msgr2=0x7fdff0156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfef7fe640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdff0157c10 0x7fdff0156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfef7fe640 1 -- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdff015ddd0 msgr2=0x7fdff0156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfef7fe640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdff015ddd0 0x7fdff0156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfef7fe640 1 -- 172.21.15.45:0/3219109860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd800b770 con 0x7fdff006a500 2023-11-28T20:54:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfeffff640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdff015ddd0 0x7fdff0156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfef7fe640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 0x7fdff00ad030 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000c870 tx=0x7fdfe000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfe7fff640 1 -- 172.21.15.45:0/3219109860 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0003b70 con 0x7fdff006a500 2023-11-28T20:54:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfe7fff640 1 -- 172.21.15.45:0/3219109860 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe0003cf0 con 0x7fdff006a500 2023-11-28T20:54:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdfe7fff640 1 -- 172.21.15.45:0/3219109860 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe000aa90 con 0x7fdff006a500 2023-11-28T20:54:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff015ba10 con 0x7fdff006a500 2023-11-28T20:54:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.944+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff015bfd0 con 0x7fdff006a500 2023-11-28T20:54:58.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.946+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff00733d0 con 0x7fdff006a500 2023-11-28T20:54:58.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.947+0000 7fdfe7fff640 1 -- 172.21.15.45:0/3219109860 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fdfe000abf0 con 0x7fdff006a500 2023-11-28T20:54:58.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.948+0000 7fdfe7fff640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdfd006af60 0x7fdfd006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:58.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.948+0000 7fdfe7fff640 1 -- 172.21.15.45:0/3219109860 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdfe0089af0 con 0x7fdff006a500 2023-11-28T20:54:58.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.949+0000 7fdfeeffd640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdfd006af60 0x7fdfd006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:58.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.950+0000 7fdfeeffd640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdfd006af60 0x7fdfd006d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8038000 tx=0x7fdfd8030000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:58.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:58.953+0000 7fdfe7fff640 1 -- 172.21.15.45:0/3219109860 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdfe00545f0 con 0x7fdff006a500 2023-11-28T20:54:59.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.190+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdff00ae600 con 0x7fdff006a500 2023-11-28T20:54:59.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.702+0000 7fdfe7fff640 1 -- 172.21.15.45:0/3219109860 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdfe0003e60 con 0x7fdff006a500 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdfd006af60 msgr2=0x7fdfd006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdfd006af60 0x7fdfd006d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8038000 tx=0x7fdfd8030000 comp rx=0 tx=0).stop 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 msgr2=0x7fdff00ad030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 0x7fdff00ad030 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000c870 tx=0x7fdfe000cd30 comp rx=0 tx=0).stop 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 shutdown_connections 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdff015ddd0 0x7fdff0156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdff0157c10 0x7fdff0156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdfd006af60 0x7fdfd006d420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 --2- 172.21.15.45:0/3219109860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdff006a500 0x7fdff00ad030 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.705+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 >> 172.21.15.45:0/3219109860 conn(0x7fdff00ced60 msgr2=0x7fdff00cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:59.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.706+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 shutdown_connections 2023-11-28T20:54:59.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.706+0000 7fdff5f83640 1 -- 172.21.15.45:0/3219109860 wait complete. 2023-11-28T20:54:59.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T20:54:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.954+0000 7fabb4dc9640 1 Processor -- start 2023-11-28T20:54:59.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.954+0000 7fabb4dc9640 1 -- start start 2023-11-28T20:54:59.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.956+0000 7fabb4dc9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fabb006ac10 0x7fabb006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:59.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.956+0000 7fabb4dc9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fabb015f720 0x7fabb00717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:59.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.956+0000 7fabb4dc9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb015ecd0 con 0x7fabb015e840 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.956+0000 7fabb4dc9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb0071ce0 con 0x7fabb015f720 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.956+0000 7fabb4dc9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb015fbd0 con 0x7fabb006ac10 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.956+0000 7fabaed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fabb015f720 0x7fabb00717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.957+0000 7fabaed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fabb015f720 0x7fabb00717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49716/0 (socket says 172.21.15.45:49716) 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.957+0000 7fabaed9d640 1 -- 172.21.15.45:0/865739322 learned_addr learned my addr 172.21.15.45:0/865739322 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.957+0000 7faba7fff640 1 --2- 172.21.15.45:0/865739322 >> v2:172.21.15.190:3300/0 conn(0x7fabb006ac10 0x7fabb006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.957+0000 7fabaed9d640 1 -- 172.21.15.45:0/865739322 >> v2:172.21.15.190:3300/0 conn(0x7fabb006ac10 msgr2=0x7fabb006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.957+0000 7fabaed9d640 1 --2- 172.21.15.45:0/865739322 >> v2:172.21.15.190:3300/0 conn(0x7fabb006ac10 0x7fabb006c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.957+0000 7fabaed9d640 1 -- 172.21.15.45:0/865739322 >> v1:172.21.15.190:6789/0 conn(0x7fabb015e840 legacy=0x7fabb006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:54:59.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.957+0000 7fabaed9d640 1 -- 172.21.15.45:0/865739322 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb015f290 con 0x7fabb015f720 2023-11-28T20:54:59.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.958+0000 7fabaed9d640 1 --2- 172.21.15.45:0/865739322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015f720 0x7fabb00717a0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fabb006e6d0 tx=0x7faba802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad74d0e4eadd099 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:59.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.958+0000 7fabadd9b640 1 -- 172.21.15.45:0/865739322 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba802ca80 con 0x7fabb015f720 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.959+0000 7fabadd9b640 1 -- 172.21.15.45:0/865739322 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba802cbe0 con 0x7fabb015f720 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.959+0000 7fabadd9b640 1 -- 172.21.15.45:0/865739322 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba80343f0 con 0x7fabb015f720 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.959+0000 7fabb4dc9640 1 -- 172.21.15.45:0/865739322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015f720 msgr2=0x7fabb00717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.959+0000 7fabb4dc9640 1 --2- 172.21.15.45:0/865739322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015f720 0x7fabb00717a0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fabb006e6d0 tx=0x7faba802bee0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.959+0000 7fabb4dc9640 1 -- 172.21.15.45:0/865739322 shutdown_connections 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.959+0000 7fabb4dc9640 1 --2- 172.21.15.45:0/865739322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015f720 0x7fabb00717a0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.959+0000 7fabb4dc9640 1 --2- 172.21.15.45:0/865739322 >> v2:172.21.15.190:3300/0 conn(0x7fabb006ac10 0x7fabb006c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.959+0000 7fabb4dc9640 1 -- 172.21.15.45:0/865739322 >> 172.21.15.45:0/865739322 conn(0x7fabb00ced60 msgr2=0x7fabb00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.960+0000 7fabb4dc9640 1 -- 172.21.15.45:0/865739322 shutdown_connections 2023-11-28T20:54:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.960+0000 7fabb4dc9640 1 -- 172.21.15.45:0/865739322 wait complete. 2023-11-28T20:54:59.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.962+0000 7fabb4dc9640 1 Processor -- start 2023-11-28T20:54:59.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.962+0000 7fabb4dc9640 1 -- start start 2023-11-28T20:54:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.963+0000 7fabb4dc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fabb006ac10 0x7fabb00c2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:59.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.963+0000 7fabb4dc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015e840 0x7fabb00b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:59.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.963+0000 7fabb4dc9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb015f720 0x7fabb0154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:59.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.963+0000 7fabb4dc9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb015d7b0 con 0x7fabb015e840 2023-11-28T20:54:59.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.963+0000 7fabb4dc9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb00c4490 con 0x7fabb015f720 2023-11-28T20:54:59.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.963+0000 7fabb4dc9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb00c2f90 con 0x7fabb006ac10 2023-11-28T20:54:59.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7faba7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015e840 0x7fabb00b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:59.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7faba7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015e840 0x7fabb00b35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49732/0 (socket says 172.21.15.45:49732) 2023-11-28T20:54:59.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7faba7fff640 1 -- 172.21.15.45:0/2201301907 learned_addr learned my addr 172.21.15.45:0/2201301907 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7fabae59c640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fabb006ac10 0x7fabb00c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7fabaed9d640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb015f720 0x7fabb0154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7faba7fff640 1 -- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fabb006ac10 msgr2=0x7fabb00c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7faba7fff640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fabb006ac10 0x7fabb00c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7faba7fff640 1 -- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb015f720 msgr2=0x7fabb0154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7faba7fff640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb015f720 0x7fabb0154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7faba7fff640 1 -- 172.21.15.45:0/2201301907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba8009b30 con 0x7fabb015e840 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.964+0000 7fabaed9d640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb015f720 0x7fabb0154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.965+0000 7faba7fff640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015e840 0x7fabb00b35e0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fab9800d800 tx=0x7fab9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:59.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.965+0000 7faba6ffd640 1 -- 172.21.15.45:0/2201301907 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9800b6c0 con 0x7fabb015e840 2023-11-28T20:54:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.965+0000 7faba6ffd640 1 -- 172.21.15.45:0/2201301907 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9800b820 con 0x7fabb015e840 2023-11-28T20:54:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.965+0000 7faba6ffd640 1 -- 172.21.15.45:0/2201301907 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9800bbb0 con 0x7fabb015e840 2023-11-28T20:54:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.965+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb01f6a50 con 0x7fabb015e840 2023-11-28T20:54:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.965+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb01f7010 con 0x7fabb015e840 2023-11-28T20:54:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.966+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb00ce9a0 con 0x7fabb015e840 2023-11-28T20:54:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.967+0000 7faba6ffd640 1 -- 172.21.15.45:0/2201301907 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fab9800bd10 con 0x7fabb015e840 2023-11-28T20:54:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.968+0000 7faba6ffd640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab9006af60 0x7fab9006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:54:59.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.968+0000 7faba6ffd640 1 -- 172.21.15.45:0/2201301907 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fab9808a260 con 0x7fabb015e840 2023-11-28T20:54:59.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.969+0000 7fabae59c640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab9006af60 0x7fab9006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:54:59.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.970+0000 7fabae59c640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab9006af60 0x7fab9006d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faba0004ea0 tx=0x7faba0009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:54:59.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:54:59.973+0000 7faba6ffd640 1 -- 172.21.15.45:0/2201301907 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab98054cd0 con 0x7fabb015e840 2023-11-28T20:55:00.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.166+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fabb00733d0 con 0x7fabb015e840 2023-11-28T20:55:00.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.705+0000 7faba6ffd640 1 -- 172.21.15.45:0/2201301907 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v152) v1 ==== 181+0+0 (secure 0 0 0) 0x7fab980582f0 con 0x7fabb015e840 2023-11-28T20:55:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T20:55:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.709+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab9006af60 msgr2=0x7fab9006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.709+0000 7fabb4dc9640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab9006af60 0x7fab9006d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faba0004ea0 tx=0x7faba0009040 comp rx=0 tx=0).stop 2023-11-28T20:55:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.709+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015e840 msgr2=0x7fabb00b35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:00.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.709+0000 7fabb4dc9640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015e840 0x7fabb00b35e0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fab9800d800 tx=0x7fab9800dcc0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.710+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 shutdown_connections 2023-11-28T20:55:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.710+0000 7fabb4dc9640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb015f720 0x7fabb0154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.710+0000 7fabb4dc9640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab9006af60 0x7fab9006d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.710+0000 7fabb4dc9640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabb015e840 0x7fabb00b35e0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.710+0000 7fabb4dc9640 1 --2- 172.21.15.45:0/2201301907 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fabb006ac10 0x7fabb00c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.710+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 >> 172.21.15.45:0/2201301907 conn(0x7fabb00ced60 msgr2=0x7fabb006db90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.710+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 shutdown_connections 2023-11-28T20:55:00.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.710+0000 7fabb4dc9640 1 -- 172.21.15.45:0/2201301907 wait complete. 2023-11-28T20:55:00.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T20:55:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.954+0000 7f067dc75640 1 Processor -- start 2023-11-28T20:55:00.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.955+0000 7f067dc75640 1 -- start start 2023-11-28T20:55:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.956+0000 7f067dc75640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f06780c1d30 0x7f06780ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.956+0000 7f067dc75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06781653a0 0x7f06780c2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.956+0000 7f067dc75640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f06780712c0 0x7f0678162290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:00.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.956+0000 7f067dc75640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067815f7e0 con 0x7f06780712c0 2023-11-28T20:55:00.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.956+0000 7f067dc75640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06780c2650 con 0x7f06780c1d30 2023-11-28T20:55:00.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.956+0000 7f067dc75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06781658a0 con 0x7f06781653a0 2023-11-28T20:55:00.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f0677fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f06780712c0 0x7f0678162290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f0677fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f06780712c0 0x7f0678162290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49742/0 (socket says 172.21.15.45:49742) 2023-11-28T20:55:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f0677fff640 1 -- 172.21.15.45:0/2479541219 learned_addr learned my addr 172.21.15.45:0/2479541219 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f066ffff640 1 --2- 172.21.15.45:0/2479541219 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06781653a0 0x7f06780c2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f0677fff640 1 -- 172.21.15.45:0/2479541219 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06781653a0 msgr2=0x7f06780c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f0677fff640 1 --2- 172.21.15.45:0/2479541219 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06781653a0 0x7f06780c2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f06777fe640 1 --2- 172.21.15.45:0/2479541219 >> v2:172.21.15.190:3300/0 conn(0x7f06780c1d30 0x7f06780ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f0677fff640 1 -- 172.21.15.45:0/2479541219 >> v2:172.21.15.190:3300/0 conn(0x7f06780c1d30 msgr2=0x7f06780ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f0677fff640 1 --2- 172.21.15.45:0/2479541219 >> v2:172.21.15.190:3300/0 conn(0x7f06780c1d30 0x7f06780ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.957+0000 7f0677fff640 1 -- 172.21.15.45:0/2479541219 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0678165b80 con 0x7f06780712c0 2023-11-28T20:55:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.958+0000 7f0677fff640 1 --2- 172.21.15.45:0/2479541219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06780712c0 0x7f0678162290 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f0670009e50 tx=0x7f067002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7e8b2229ca2f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.959+0000 7f0676ffd640 1 -- 172.21.15.45:0/2479541219 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067002ca70 con 0x7f06780712c0 2023-11-28T20:55:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.959+0000 7f0676ffd640 1 -- 172.21.15.45:0/2479541219 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067002cbd0 con 0x7f06780712c0 2023-11-28T20:55:00.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.959+0000 7f0676ffd640 1 -- 172.21.15.45:0/2479541219 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06700343f0 con 0x7f06780712c0 2023-11-28T20:55:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.960+0000 7f067dc75640 1 -- 172.21.15.45:0/2479541219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06780712c0 msgr2=0x7f0678162290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.960+0000 7f067dc75640 1 --2- 172.21.15.45:0/2479541219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06780712c0 0x7f0678162290 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f0670009e50 tx=0x7f067002bed0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.960+0000 7f067dc75640 1 -- 172.21.15.45:0/2479541219 shutdown_connections 2023-11-28T20:55:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.960+0000 7f067dc75640 1 --2- 172.21.15.45:0/2479541219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06780712c0 0x7f0678162290 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.960+0000 7f067dc75640 1 --2- 172.21.15.45:0/2479541219 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06781653a0 0x7f06780c2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.960+0000 7f067dc75640 1 --2- 172.21.15.45:0/2479541219 >> v2:172.21.15.190:3300/0 conn(0x7f06780c1d30 0x7f06780ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.960+0000 7f067dc75640 1 -- 172.21.15.45:0/2479541219 >> 172.21.15.45:0/2479541219 conn(0x7f06780ced60 msgr2=0x7f06780cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.960+0000 7f067dc75640 1 -- 172.21.15.45:0/2479541219 shutdown_connections 2023-11-28T20:55:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.960+0000 7f067dc75640 1 -- 172.21.15.45:0/2479541219 wait complete. 2023-11-28T20:55:00.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.962+0000 7f067dc75640 1 Processor -- start 2023-11-28T20:55:00.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.962+0000 7f067dc75640 1 -- start start 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.963+0000 7f067dc75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06780712c0 0x7f06781599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.963+0000 7f067dc75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06780c1d30 0x7f067814d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.963+0000 7f067dc75640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06781653a0 0x7f06780acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.963+0000 7f067dc75640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067806c340 con 0x7f06781653a0 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.963+0000 7f067dc75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067814d5f0 con 0x7f06780c1d30 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.963+0000 7f067dc75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06780ad5f0 con 0x7f06780712c0 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.963+0000 7f0677fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06781653a0 0x7f06780acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.963+0000 7f0677fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06781653a0 0x7f06780acda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49750/0 (socket says 172.21.15.45:49750) 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.963+0000 7f0677fff640 1 -- 172.21.15.45:0/2983119335 learned_addr learned my addr 172.21.15.45:0/2983119335 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f06777fe640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06780712c0 0x7f06781599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f0677fff640 1 -- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06780712c0 msgr2=0x7f06781599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f066ffff640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06780c1d30 0x7f067814d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f0677fff640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06780712c0 0x7f06781599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f0677fff640 1 -- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06780c1d30 msgr2=0x7f067814d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f0677fff640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06780c1d30 0x7f067814d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:00.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f0677fff640 1 -- 172.21.15.45:0/2983119335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0670009b30 con 0x7f06781653a0 2023-11-28T20:55:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f0677fff640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06781653a0 0x7f06780acda0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f0670036040 tx=0x7f0670004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f0674ff9640 1 -- 172.21.15.45:0/2983119335 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670009510 con 0x7f06781653a0 2023-11-28T20:55:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f0674ff9640 1 -- 172.21.15.45:0/2983119335 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067003e450 con 0x7f06781653a0 2023-11-28T20:55:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.964+0000 7f0674ff9640 1 -- 172.21.15.45:0/2983119335 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067003e800 con 0x7f06781653a0 2023-11-28T20:55:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.965+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06780ad8d0 con 0x7f06781653a0 2023-11-28T20:55:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.965+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06780ad2e0 con 0x7f06781653a0 2023-11-28T20:55:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.966+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06780addf0 con 0x7f06781653a0 2023-11-28T20:55:00.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.967+0000 7f0674ff9640 1 -- 172.21.15.45:0/2983119335 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f067003e960 con 0x7f06781653a0 2023-11-28T20:55:00.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.968+0000 7f0674ff9640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f065806af60 0x7f065806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.968+0000 7f0674ff9640 1 -- 172.21.15.45:0/2983119335 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6436+0+0 (secure 0 0 0) 0x7f06700ac860 con 0x7f06781653a0 2023-11-28T20:55:00.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.968+0000 7f06777fe640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f065806af60 0x7f065806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:00.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.969+0000 7f06777fe640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f065806af60 0x7f065806d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f066800a990 tx=0x7f0668008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:00.972+0000 7f0674ff9640 1 -- 172.21.15.45:0/2983119335 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0670077530 con 0x7f06781653a0 2023-11-28T20:55:01.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.167+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f06780cbd10 con 0x7f06781653a0 2023-11-28T20:55:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.720+0000 7f0674ff9640 1 -- 172.21.15.45:0/2983119335 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v153) v1 ==== 169+0+0 (secure 0 0 0) 0x7f067007ab50 con 0x7f06781653a0 2023-11-28T20:55:01.722 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T20:55:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.725+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f065806af60 msgr2=0x7f065806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.725+0000 7f067dc75640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f065806af60 0x7f065806d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f066800a990 tx=0x7f0668008040 comp rx=0 tx=0).stop 2023-11-28T20:55:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.725+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06781653a0 msgr2=0x7f06780acda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:01.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.725+0000 7f067dc75640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06781653a0 0x7f06780acda0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f0670036040 tx=0x7f0670004a40 comp rx=0 tx=0).stop 2023-11-28T20:55:01.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.726+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 shutdown_connections 2023-11-28T20:55:01.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.726+0000 7f067dc75640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f065806af60 0x7f065806d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:01.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.726+0000 7f067dc75640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06781653a0 0x7f06780acda0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:01.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.726+0000 7f067dc75640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06780c1d30 0x7f067814d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:01.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.726+0000 7f067dc75640 1 --2- 172.21.15.45:0/2983119335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06780712c0 0x7f06781599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:01.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.726+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 >> 172.21.15.45:0/2983119335 conn(0x7f06780ced60 msgr2=0x7f0678163030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:01.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.729+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 shutdown_connections 2023-11-28T20:55:01.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.730+0000 7f067dc75640 1 -- 172.21.15.45:0/2983119335 wait complete. 2023-11-28T20:55:01.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T20:55:01.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.957+0000 7fcc2f59e640 1 Processor -- start 2023-11-28T20:55:01.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.957+0000 7fcc2f59e640 1 -- start start 2023-11-28T20:55:01.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.959+0000 7fcc2f59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcc300abd70 0x7fcc300ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:01.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.959+0000 7fcc2f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc300ac270 con 0x7fcc3015b080 2023-11-28T20:55:01.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.959+0000 7fcc2f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3015b830 con 0x7fcc30071fd0 2023-11-28T20:55:01.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.959+0000 7fcc2f59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3015bb10 con 0x7fcc300abd70 2023-11-28T20:55:01.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.959+0000 7fcc2e59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcc3015b080 0x7fcc300718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40606/0 (socket says 172.21.15.45:40606) 2023-11-28T20:55:01.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.959+0000 7fcc2e59c640 1 -- 172.21.15.45:0/3953102271 learned_addr learned my addr 172.21.15.45:0/3953102271 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:01.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.959+0000 7fcc2ed9d640 1 --2- 172.21.15.45:0/3953102271 >> v2:172.21.15.45:3300/0 conn(0x7fcc300abd70 0x7fcc300ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.960+0000 7fcc2d59a640 1 -- 172.21.15.45:0/3953102271 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265575851 0 0) 0x7fcc300ac270 con 0x7fcc3015b080 2023-11-28T20:55:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.960+0000 7fcc2d59a640 1 -- 172.21.15.45:0/3953102271 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc18003520 con 0x7fcc3015b080 2023-11-28T20:55:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.960+0000 7fcc2ed9d640 1 -- 172.21.15.45:0/3953102271 >> v1:172.21.15.190:6789/0 conn(0x7fcc30071fd0 legacy=0x7fcc300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.960+0000 7fcc2ed9d640 1 -- 172.21.15.45:0/3953102271 >> v1:172.21.15.45:6789/0 conn(0x7fcc3015b080 legacy=0x7fcc300718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.960+0000 7fcc2ed9d640 1 -- 172.21.15.45:0/3953102271 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3015bdf0 con 0x7fcc300abd70 2023-11-28T20:55:01.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.961+0000 7fcc2ed9d640 1 --2- 172.21.15.45:0/3953102271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc300abd70 0x7fcc300ae370 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009d90 tx=0x7fcc2802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990bb052389d97e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2d59a640 1 -- 172.21.15.45:0/3953102271 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2802cc60 con 0x7fcc300abd70 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2d59a640 1 -- 172.21.15.45:0/3953102271 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2802cdc0 con 0x7fcc300abd70 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2d59a640 1 -- 172.21.15.45:0/3953102271 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28034660 con 0x7fcc300abd70 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3953102271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc300abd70 msgr2=0x7fcc300ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2f59e640 1 --2- 172.21.15.45:0/3953102271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc300abd70 0x7fcc300ae370 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009d90 tx=0x7fcc2802beb0 comp rx=0 tx=0).stop 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3953102271 shutdown_connections 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2f59e640 1 --2- 172.21.15.45:0/3953102271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc300abd70 0x7fcc300ae370 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3953102271 >> 172.21.15.45:0/3953102271 conn(0x7fcc300cecd0 msgr2=0x7fcc300cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3953102271 shutdown_connections 2023-11-28T20:55:01.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.962+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3953102271 wait complete. 2023-11-28T20:55:01.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.964+0000 7fcc2f59e640 1 Processor -- start 2023-11-28T20:55:01.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.964+0000 7fcc2f59e640 1 -- start start 2023-11-28T20:55:01.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.965+0000 7fcc2f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc30071fd0 0x7fcc30153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:01.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.965+0000 7fcc2f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc300abd70 0x7fcc300ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:01.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.965+0000 7fcc2f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3015b080 0x7fcc300ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:01.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.965+0000 7fcc2f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300ae650 con 0x7fcc3015b080 2023-11-28T20:55:01.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.965+0000 7fcc2f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300ae930 con 0x7fcc300abd70 2023-11-28T20:55:01.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.965+0000 7fcc2f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300aec10 con 0x7fcc30071fd0 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3015b080 0x7fcc300ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3015b080 0x7fcc300ade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49766/0 (socket says 172.21.15.45:49766) 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2ed9d640 1 -- 172.21.15.45:0/3420553478 learned_addr learned my addr 172.21.15.45:0/3420553478 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2dd9b640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc300abd70 0x7fcc300ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2e59c640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc30071fd0 0x7fcc30153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2ed9d640 1 -- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc30071fd0 msgr2=0x7fcc30153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2ed9d640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc30071fd0 0x7fcc30153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2ed9d640 1 -- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc300abd70 msgr2=0x7fcc300ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2ed9d640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc300abd70 0x7fcc300ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2ed9d640 1 -- 172.21.15.45:0/3420553478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc28009a70 con 0x7fcc3015b080 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2dd9b640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc300abd70 0x7fcc300ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:01.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.966+0000 7fcc2e59c640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc30071fd0 0x7fcc30153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.967+0000 7fcc2ed9d640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3015b080 0x7fcc300ade20 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009d90 tx=0x7fcc28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:01.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.967+0000 7fcc1effd640 1 -- 172.21.15.45:0/3420553478 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28040650 con 0x7fcc3015b080 2023-11-28T20:55:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.967+0000 7fcc1effd640 1 -- 172.21.15.45:0/3420553478 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc280407b0 con 0x7fcc3015b080 2023-11-28T20:55:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.967+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc300aeef0 con 0x7fcc3015b080 2023-11-28T20:55:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.967+0000 7fcc1effd640 1 -- 172.21.15.45:0/3420553478 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28040b60 con 0x7fcc3015b080 2023-11-28T20:55:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.967+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc300ae360 con 0x7fcc3015b080 2023-11-28T20:55:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.968+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc300af410 con 0x7fcc3015b080 2023-11-28T20:55:01.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.969+0000 7fcc1effd640 1 -- 172.21.15.45:0/3420553478 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fcc28040cc0 con 0x7fcc3015b080 2023-11-28T20:55:01.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.971+0000 7fcc1effd640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fcc1006af60 0x7fcc1006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:01.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.971+0000 7fcc1effd640 1 -- 172.21.15.45:0/3420553478 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7fcc280ac690 con 0x7fcc3015b080 2023-11-28T20:55:01.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.971+0000 7fcc2e59c640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fcc1006af60 0x7fcc1006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:01.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.972+0000 7fcc2e59c640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fcc1006af60 0x7fcc1006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcc20007fd0 tx=0x7fcc20009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:01.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:01.975+0000 7fcc1effd640 1 -- 172.21.15.45:0/3420553478 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc300af410 con 0x7fcc3015b080 2023-11-28T20:55:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.177+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fcc300af410 con 0x7fcc3015b080 2023-11-28T20:55:02.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.183+0000 7fcc1effd640 1 -- 172.21.15.45:0/3420553478 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v49) v1 ==== 120+0+0 (secure 0 0 0) 0x7fcc300af410 con 0x7fcc3015b080 2023-11-28T20:55:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fcc1006af60 msgr2=0x7fcc1006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fcc1006af60 0x7fcc1006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcc20007fd0 tx=0x7fcc20009040 comp rx=0 tx=0).stop 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3015b080 msgr2=0x7fcc300ade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3015b080 0x7fcc300ade20 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009d90 tx=0x7fcc28004a40 comp rx=0 tx=0).stop 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 shutdown_connections 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fcc1006af60 0x7fcc1006d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3015b080 0x7fcc300ade20 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc300abd70 0x7fcc300ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 --2- 172.21.15.45:0/3420553478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc30071fd0 0x7fcc30153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 >> 172.21.15.45:0/3420553478 conn(0x7fcc300cecd0 msgr2=0x7fcc3006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 shutdown_connections 2023-11-28T20:55:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.186+0000 7fcc2f59e640 1 -- 172.21.15.45:0/3420553478 wait complete. 2023-11-28T20:55:02.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T20:55:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.424+0000 7f52d5dfb640 1 Processor -- start 2023-11-28T20:55:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.424+0000 7f52d5dfb640 1 -- start start 2023-11-28T20:55:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.426+0000 7f52d5dfb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52d0071fd0 0x7f52d00ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.426+0000 7f52d5dfb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d00abd70 0x7f52d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.426+0000 7f52d5dfb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d00ac270 con 0x7f52d015b080 2023-11-28T20:55:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.426+0000 7f52d5dfb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d015b830 con 0x7f52d0071fd0 2023-11-28T20:55:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.426+0000 7f52d5dfb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d015bb10 con 0x7f52d00abd70 2023-11-28T20:55:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.426+0000 7f52d4df9640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f52d015b080 0x7f52d00718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40608/0 (socket says 172.21.15.45:40608) 2023-11-28T20:55:02.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.426+0000 7f52d4df9640 1 -- 172.21.15.45:0/3889702444 learned_addr learned my addr 172.21.15.45:0/3889702444 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52d55fa640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d00abd70 0x7f52d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cffff640 1 --2- 172.21.15.45:0/3889702444 >> v2:172.21.15.190:3300/0 conn(0x7f52d0071fd0 0x7f52d00ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230812488 0 0) 0x7f52d00ac270 con 0x7f52d015b080 2023-11-28T20:55:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c0003520 con 0x7f52d015b080 2023-11-28T20:55:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1340611565 0 0) 0x7f52c0003520 con 0x7f52d015b080 2023-11-28T20:55:02.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d00abd70 msgr2=0x7f52d00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d00abd70 0x7f52d00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 >> v2:172.21.15.190:3300/0 conn(0x7f52d0071fd0 msgr2=0x7f52d00ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 --2- 172.21.15.45:0/3889702444 >> v2:172.21.15.190:3300/0 conn(0x7f52d0071fd0 0x7f52d00ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d015bdf0 con 0x7f52d015b080 2023-11-28T20:55:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f52c4001690 con 0x7f52d015b080 2023-11-28T20:55:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 >> v1:172.21.15.45:6789/0 conn(0x7f52d015b080 legacy=0x7f52d00718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c0003970 0x7f52c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d00abd70 0x7f52c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d015b830 con 0x7f52c00047c0 2023-11-28T20:55:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d015bb10 con 0x7f52c0003970 2023-11-28T20:55:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.427+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c000b980 con 0x7f52d00abd70 2023-11-28T20:55:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.428+0000 7f52cffff640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.428+0000 7f52d4df9640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c0003970 0x7f52c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.428+0000 7f52d55fa640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d00abd70 0x7f52c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.428+0000 7f52cffff640 1 -- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d00abd70 msgr2=0x7f52c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.428+0000 7f52cffff640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d00abd70 0x7f52c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.428+0000 7f52cffff640 1 -- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c0003970 msgr2=0x7f52c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.428+0000 7f52cffff640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c0003970 0x7f52c0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.428+0000 7f52cffff640 1 -- 172.21.15.45:0/3889702444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c0003520 con 0x7f52c00047c0 2023-11-28T20:55:02.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.428+0000 7f52d4df9640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c0003970 0x7f52c0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:55:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.430+0000 7f52cffff640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52c0004c40 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009ac0 tx=0x7f52c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558b0adee0c636f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.430+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc02c6d0 con 0x7f52c00047c0 2023-11-28T20:55:02.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.431+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc02c830 con 0x7f52c00047c0 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.431+0000 7f52cf7fe640 1 -- 172.21.15.45:0/3889702444 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc02cbc0 con 0x7f52c00047c0 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.431+0000 7f52d5dfb640 1 -- 172.21.15.45:0/3889702444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 msgr2=0x7f52c0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.431+0000 7f52d5dfb640 1 --2- 172.21.15.45:0/3889702444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52c0004c40 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009ac0 tx=0x7f52c00057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.431+0000 7f52d55fa640 1 -- 172.21.15.45:0/3889702444 reap_dead start 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.431+0000 7f52d5dfb640 1 -- 172.21.15.45:0/3889702444 shutdown_connections 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.431+0000 7f52d5dfb640 1 -- 172.21.15.45:0/3889702444 >> 172.21.15.45:0/3889702444 conn(0x7f52d00cecd0 msgr2=0x7f52d00cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.431+0000 7f52d5dfb640 1 -- 172.21.15.45:0/3889702444 shutdown_connections 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.431+0000 7f52d5dfb640 1 -- 172.21.15.45:0/3889702444 wait complete. 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.433+0000 7f52d5dfb640 1 Processor -- start 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.433+0000 7f52d5dfb640 1 -- start start 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.434+0000 7f52d5dfb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52d01533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.434+0000 7f52d5dfb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d009b160 0x7f52d009b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.434+0000 7f52d5dfb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52d01588c0 0x7f52d0155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.434+0000 7f52d5dfb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d01e9180 con 0x7f52c00047c0 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.434+0000 7f52d5dfb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0153920 con 0x7f52d01588c0 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.434+0000 7f52d5dfb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0157820 con 0x7f52d009b160 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52d4df9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52d01533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52d4df9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52d01533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49784/0 (socket says 172.21.15.45:49784) 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52d4df9640 1 -- 172.21.15.45:0/899810475 learned_addr learned my addr 172.21.15.45:0/899810475 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52cffff640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d009b160 0x7f52d009b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52d55fa640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52d01588c0 0x7f52d0155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52d4df9640 1 -- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d009b160 msgr2=0x7f52d009b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52d4df9640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d009b160 0x7f52d009b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52d4df9640 1 -- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52d01588c0 msgr2=0x7f52d0155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52d4df9640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52d01588c0 0x7f52d0155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52d4df9640 1 -- 172.21.15.45:0/899810475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d0157b00 con 0x7f52c00047c0 2023-11-28T20:55:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.435+0000 7f52cffff640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d009b160 0x7f52d009b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.436+0000 7f52d4df9640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52d01533e0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f52c400e730 tx=0x7f52c400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.436+0000 7f52cd7fa640 1 -- 172.21.15.45:0/899810475 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c400c6c0 con 0x7f52c00047c0 2023-11-28T20:55:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.436+0000 7f52cd7fa640 1 -- 172.21.15.45:0/899810475 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c400c820 con 0x7f52c00047c0 2023-11-28T20:55:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.436+0000 7f52cd7fa640 1 -- 172.21.15.45:0/899810475 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c400cb90 con 0x7f52c00047c0 2023-11-28T20:55:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.436+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d0157de0 con 0x7f52c00047c0 2023-11-28T20:55:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.436+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d01583a0 con 0x7f52c00047c0 2023-11-28T20:55:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.437+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d009d0f0 con 0x7f52c00047c0 2023-11-28T20:55:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.438+0000 7f52cd7fa640 1 -- 172.21.15.45:0/899810475 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f52c4013420 con 0x7f52c00047c0 2023-11-28T20:55:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.440+0000 7f52cd7fa640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f52b406af90 0x7f52b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.440+0000 7f52cd7fa640 1 -- 172.21.15.45:0/899810475 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f52c40897c0 con 0x7f52c00047c0 2023-11-28T20:55:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.440+0000 7f52cffff640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f52b406af90 0x7f52b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.442+0000 7f52cffff640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f52b406af90 0x7f52b406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52bc036000 tx=0x7f52bc030000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:02.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.444+0000 7f52cd7fa640 1 -- 172.21.15.45:0/899810475 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52c4054610 con 0x7f52c00047c0 2023-11-28T20:55:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.643+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f52d004e920 con 0x7f52c00047c0 2023-11-28T20:55:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.650+0000 7f52cd7fa640 1 -- 172.21.15.45:0/899810475 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f52c4057c30 con 0x7f52c00047c0 2023-11-28T20:55:02.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f52b406af90 msgr2=0x7f52b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f52b406af90 0x7f52b406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52bc036000 tx=0x7f52bc030000 comp rx=0 tx=0).stop 2023-11-28T20:55:02.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 msgr2=0x7f52d01533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52d01533e0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f52c400e730 tx=0x7f52c400ebf0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 shutdown_connections 2023-11-28T20:55:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52d01588c0 0x7f52d0155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52d009b160 0x7f52d009b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f52b406af90 0x7f52b406d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 --2- 172.21.15.45:0/899810475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52c00047c0 0x7f52d01533e0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 >> 172.21.15.45:0/899810475 conn(0x7f52d00cecd0 msgr2=0x7f52d00af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 shutdown_connections 2023-11-28T20:55:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.653+0000 7f52d5dfb640 1 -- 172.21.15.45:0/899810475 wait complete. 2023-11-28T20:55:02.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T20:55:02.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.903+0000 7f8e22fd4640 1 Processor -- start 2023-11-28T20:55:02.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.903+0000 7f8e22fd4640 1 -- start start 2023-11-28T20:55:02.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.905+0000 7f8e22fd4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c15d230 0x7f8e1c0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.905+0000 7f8e22fd4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c0d11d0 con 0x7f8e1c0b2640 2023-11-28T20:55:02.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.905+0000 7f8e22fd4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c0d1490 con 0x7f8e1c0cddf0 2023-11-28T20:55:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.905+0000 7f8e22fd4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0b90e0 con 0x7f8e1c15d230 2023-11-28T20:55:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.905+0000 7f8e217d1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8e1c0b2640 0x7f8e1c0b89d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40614/0 (socket says 172.21.15.45:40614) 2023-11-28T20:55:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.905+0000 7f8e217d1640 1 -- 172.21.15.45:0/359316724 learned_addr learned my addr 172.21.15.45:0/359316724 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.905+0000 7f8e21fd2640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c15d230 0x7f8e1c0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.905+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330184020 0 0) 0x7f8e1c0d11d0 con 0x7f8e1c0b2640 2023-11-28T20:55:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.905+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e08003520 con 0x7f8e1c0b2640 2023-11-28T20:55:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 615443485 0 0) 0x7f8e08003520 con 0x7f8e1c0b2640 2023-11-28T20:55:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c15d230 msgr2=0x7f8e1c0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c15d230 0x7f8e1c0b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 >> v1:172.21.15.190:6789/0 conn(0x7f8e1c0cddf0 legacy=0x7f8e1c0d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c0d1770 con 0x7f8e1c0b2640 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f8e0c001670 con 0x7f8e1c0b2640 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 >> v1:172.21.15.45:6789/0 conn(0x7f8e1c0b2640 legacy=0x7f8e1c0b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e08003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e080047c0 0x7f8e08004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c15d230 0x7f8e08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0b90e0 con 0x7f8e080039a0 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0800baf0 con 0x7f8e080047c0 2023-11-28T20:55:02.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0800bcb0 con 0x7f8e1c15d230 2023-11-28T20:55:02.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.906+0000 7f8e21fd2640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e08003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.907+0000 7f8e227d3640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c15d230 0x7f8e08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.907+0000 7f8e21fd2640 1 -- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c15d230 msgr2=0x7f8e08005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:02.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.907+0000 7f8e217d1640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e080047c0 0x7f8e08004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.907+0000 7f8e21fd2640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c15d230 0x7f8e08005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.907+0000 7f8e21fd2640 1 -- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e080047c0 msgr2=0x7f8e08004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:02.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.907+0000 7f8e21fd2640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e080047c0 0x7f8e08004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.907+0000 7f8e21fd2640 1 -- 172.21.15.45:0/359316724 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e08003520 con 0x7f8e080039a0 2023-11-28T20:55:02.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.907+0000 7f8e227d3640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c15d230 0x7f8e08005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:02.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.908+0000 7f8e21fd2640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e08003dc0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009a90 tx=0x7f8e080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09e4ec4301466b9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.908+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1802c680 con 0x7f8e080039a0 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.908+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1802c7e0 con 0x7f8e080039a0 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.908+0000 7f8e20fd0640 1 -- 172.21.15.45:0/359316724 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1802cb70 con 0x7f8e080039a0 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.909+0000 7f8e22fd4640 1 -- 172.21.15.45:0/359316724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 msgr2=0x7f8e08003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.909+0000 7f8e22fd4640 1 --2- 172.21.15.45:0/359316724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e08003dc0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009a90 tx=0x7f8e080057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.909+0000 7f8e227d3640 1 -- 172.21.15.45:0/359316724 reap_dead start 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.909+0000 7f8e22fd4640 1 -- 172.21.15.45:0/359316724 shutdown_connections 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.909+0000 7f8e22fd4640 1 -- 172.21.15.45:0/359316724 >> 172.21.15.45:0/359316724 conn(0x7f8e1c070f40 msgr2=0x7f8e1c071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.909+0000 7f8e22fd4640 1 -- 172.21.15.45:0/359316724 shutdown_connections 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.909+0000 7f8e22fd4640 1 -- 172.21.15.45:0/359316724 wait complete. 2023-11-28T20:55:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.911+0000 7f8e22fd4640 1 Processor -- start 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.911+0000 7f8e22fd4640 1 -- start start 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e22fd4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e1c1e6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e22fd4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c1e6c50 0x7f8e1c1f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e22fd4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e1c1f72f0 0x7f8e1c0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e22fd4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c15f590 con 0x7f8e080039a0 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e22fd4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c1e70d0 con 0x7f8e1c1f72f0 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e22fd4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0cbc90 con 0x7f8e1c1e6c50 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e21fd2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e1c1e6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e21fd2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e1c1e6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49802/0 (socket says 172.21.15.45:49802) 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e21fd2640 1 -- 172.21.15.45:0/3070062641 learned_addr learned my addr 172.21.15.45:0/3070062641 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.912+0000 7f8e217d1640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c1e6c50 0x7f8e1c1f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e21fd2640 1 -- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c1e6c50 msgr2=0x7f8e1c1f6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e21fd2640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c1e6c50 0x7f8e1c1f6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e227d3640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e1c1f72f0 0x7f8e1c0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e21fd2640 1 -- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e1c1f72f0 msgr2=0x7f8e1c0ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e21fd2640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e1c1f72f0 0x7f8e1c0ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e21fd2640 1 -- 172.21.15.45:0/3070062641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c0cbf70 con 0x7f8e080039a0 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e21fd2640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e1c1e6710 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f8e1802c220 tx=0x7f8e18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e127fc640 1 -- 172.21.15.45:0/3070062641 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18003810 con 0x7f8e080039a0 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e127fc640 1 -- 172.21.15.45:0/3070062641 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e18003970 con 0x7f8e080039a0 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e127fc640 1 -- 172.21.15.45:0/3070062641 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18003ce0 con 0x7f8e080039a0 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c0cc250 con 0x7f8e080039a0 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.913+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c0cc810 con 0x7f8e080039a0 2023-11-28T20:55:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.915+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1c0d2d20 con 0x7f8e080039a0 2023-11-28T20:55:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.916+0000 7f8e127fc640 1 -- 172.21.15.45:0/3070062641 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8e1803e400 con 0x7f8e080039a0 2023-11-28T20:55:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.916+0000 7f8e127fc640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8e0006afe0 0x7f8e0006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.917+0000 7f8e127fc640 1 -- 172.21.15.45:0/3070062641 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f8e180ab680 con 0x7f8e080039a0 2023-11-28T20:55:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.917+0000 7f8e217d1640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8e0006afe0 0x7f8e0006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.918+0000 7f8e217d1640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8e0006afe0 0x7f8e0006d4a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0cb8b0 tx=0x7f8e0c009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:02.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:02.921+0000 7f8e127fc640 1 -- 172.21.15.45:0/3070062641 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e18076570 con 0x7f8e080039a0 2023-11-28T20:55:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.114+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f8e1c071b30 con 0x7f8e080039a0 2023-11-28T20:55:03.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.121+0000 7f8e127fc640 1 -- 172.21.15.45:0/3070062641 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v51) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8e180044e0 con 0x7f8e080039a0 2023-11-28T20:55:03.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8e0006afe0 msgr2=0x7f8e0006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8e0006afe0 0x7f8e0006d4a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0cb8b0 tx=0x7f8e0c009040 comp rx=0 tx=0).stop 2023-11-28T20:55:03.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 msgr2=0x7f8e1c1e6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e1c1e6710 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f8e1802c220 tx=0x7f8e18004730 comp rx=0 tx=0).stop 2023-11-28T20:55:03.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 shutdown_connections 2023-11-28T20:55:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e1c1f72f0 0x7f8e1c0ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8e1c1e6c50 0x7f8e1c1f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8e0006afe0 0x7f8e0006d4a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 --2- 172.21.15.45:0/3070062641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e080039a0 0x7f8e1c1e6710 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 >> 172.21.15.45:0/3070062641 conn(0x7f8e1c070f40 msgr2=0x7f8e1c0b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 shutdown_connections 2023-11-28T20:55:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.124+0000 7f8e22fd4640 1 -- 172.21.15.45:0/3070062641 wait complete. 2023-11-28T20:55:03.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-11-28T20:55:03.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.367+0000 7f8d34aef640 1 Processor -- start 2023-11-28T20:55:03.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.367+0000 7f8d34aef640 1 -- start start 2023-11-28T20:55:03.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.368+0000 7f8d34aef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d300d02d0 0x7f8d300d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.368+0000 7f8d34aef640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8d300d1ab0 0x7f8d300b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.369+0000 7f8d34aef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d300d2230 con 0x7f8d3015d550 2023-11-28T20:55:03.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.369+0000 7f8d34aef640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d300d2510 con 0x7f8d300d1ab0 2023-11-28T20:55:03.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.369+0000 7f8d34aef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d300d27f0 con 0x7f8d300d02d0 2023-11-28T20:55:03.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.369+0000 7f8d2ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8d300d1ab0 0x7f8d300b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:03.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.369+0000 7f8d2f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8d3015d550 0x7f8d300cfbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40622/0 (socket says 172.21.15.45:40622) 2023-11-28T20:55:03.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.369+0000 7f8d2f7fe640 1 -- 172.21.15.45:0/4256072954 learned_addr learned my addr 172.21.15.45:0/4256072954 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:03.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.369+0000 7f8d26ffd640 1 --2- 172.21.15.45:0/4256072954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d300d02d0 0x7f8d300d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:03.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.370+0000 7f8d2ffff640 1 -- 172.21.15.45:0/4256072954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d300d02d0 msgr2=0x7f8d300d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:03.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.370+0000 7f8d2ffff640 1 --2- 172.21.15.45:0/4256072954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d300d02d0 0x7f8d300d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.370+0000 7f8d2ffff640 1 -- 172.21.15.45:0/4256072954 >> v1:172.21.15.45:6789/0 conn(0x7f8d3015d550 legacy=0x7f8d300cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.370+0000 7f8d2ffff640 1 -- 172.21.15.45:0/4256072954 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d300b6a30 con 0x7f8d300d1ab0 2023-11-28T20:55:03.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.370+0000 7f8d2effd640 1 -- 172.21.15.45:0/4256072954 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850777383 0 0) 0x7f8d300d2230 con 0x7f8d3015d550 2023-11-28T20:55:03.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.371+0000 7f8d2ffff640 1 --2- 172.21.15.45:0/4256072954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d1ab0 0x7f8d300b6f50 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f8d280097a0 tx=0x7f8d2802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca2de9a86fda2415 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:03.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.371+0000 7f8d2effd640 1 -- 172.21.15.45:0/4256072954 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2802d040 con 0x7f8d300d1ab0 2023-11-28T20:55:03.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.371+0000 7f8d2effd640 1 -- 172.21.15.45:0/4256072954 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d28031810 con 0x7f8d300d1ab0 2023-11-28T20:55:03.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.371+0000 7f8d2effd640 1 -- 172.21.15.45:0/4256072954 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d28031ba0 con 0x7f8d300d1ab0 2023-11-28T20:55:03.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.372+0000 7f8d34aef640 1 -- 172.21.15.45:0/4256072954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d1ab0 msgr2=0x7f8d300b6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.372+0000 7f8d34aef640 1 --2- 172.21.15.45:0/4256072954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d1ab0 0x7f8d300b6f50 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f8d280097a0 tx=0x7f8d2802b8b0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.372+0000 7f8d34aef640 1 -- 172.21.15.45:0/4256072954 shutdown_connections 2023-11-28T20:55:03.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.372+0000 7f8d34aef640 1 --2- 172.21.15.45:0/4256072954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d1ab0 0x7f8d300b6f50 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.372+0000 7f8d34aef640 1 --2- 172.21.15.45:0/4256072954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d300d02d0 0x7f8d300d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.372+0000 7f8d34aef640 1 -- 172.21.15.45:0/4256072954 >> 172.21.15.45:0/4256072954 conn(0x7f8d3006ac70 msgr2=0x7f8d300b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:03.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.372+0000 7f8d34aef640 1 -- 172.21.15.45:0/4256072954 shutdown_connections 2023-11-28T20:55:03.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.372+0000 7f8d34aef640 1 -- 172.21.15.45:0/4256072954 wait complete. 2023-11-28T20:55:03.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.374+0000 7f8d34aef640 1 Processor -- start 2023-11-28T20:55:03.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.374+0000 7f8d34aef640 1 -- start start 2023-11-28T20:55:03.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.375+0000 7f8d34aef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d02d0 0x7f8d3015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.375+0000 7f8d34aef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d300d1ab0 0x7f8d301599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.375+0000 7f8d34aef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d3015d550 0x7f8d300aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.375+0000 7f8d34aef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d300c7150 con 0x7f8d300d02d0 2023-11-28T20:55:03.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.375+0000 7f8d34aef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d300c7430 con 0x7f8d3015d550 2023-11-28T20:55:03.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.375+0000 7f8d34aef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d300c7710 con 0x7f8d300d1ab0 2023-11-28T20:55:03.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.375+0000 7f8d2f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d02d0 0x7f8d3015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:03.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.375+0000 7f8d2f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d02d0 0x7f8d3015acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49820/0 (socket says 172.21.15.45:49820) 2023-11-28T20:55:03.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.375+0000 7f8d2f7fe640 1 -- 172.21.15.45:0/143432063 learned_addr learned my addr 172.21.15.45:0/143432063 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:03.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d2f7fe640 1 -- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d300d1ab0 msgr2=0x7f8d301599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d2f7fe640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d300d1ab0 0x7f8d301599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d2f7fe640 1 -- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d3015d550 msgr2=0x7f8d300aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d2f7fe640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d3015d550 0x7f8d300aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d2f7fe640 1 -- 172.21.15.45:0/143432063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d28009400 con 0x7f8d300d02d0 2023-11-28T20:55:03.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d2f7fe640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d02d0 0x7f8d3015acc0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c7c0 tx=0x7f8d2000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:03.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d2cff9640 1 -- 172.21.15.45:0/143432063 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2000a640 con 0x7f8d300d02d0 2023-11-28T20:55:03.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d2cff9640 1 -- 172.21.15.45:0/143432063 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2000a7c0 con 0x7f8d300d02d0 2023-11-28T20:55:03.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d300c79f0 con 0x7f8d300d02d0 2023-11-28T20:55:03.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.376+0000 7f8d2cff9640 1 -- 172.21.15.45:0/143432063 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2000ab70 con 0x7f8d300d02d0 2023-11-28T20:55:03.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.377+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d30159f30 con 0x7f8d300d02d0 2023-11-28T20:55:03.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.378+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d300c7fc0 con 0x7f8d300d02d0 2023-11-28T20:55:03.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.379+0000 7f8d2cff9640 1 -- 172.21.15.45:0/143432063 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8d2000acd0 con 0x7f8d300d02d0 2023-11-28T20:55:03.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.379+0000 7f8d2cff9640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8d1006af60 0x7f8d1006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.380+0000 7f8d2cff9640 1 -- 172.21.15.45:0/143432063 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f8d20088eb0 con 0x7f8d300d02d0 2023-11-28T20:55:03.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.380+0000 7f8d26ffd640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8d1006af60 0x7f8d1006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:03.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.382+0000 7f8d26ffd640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8d1006af60 0x7f8d1006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8d300c6c30 tx=0x7f8d18008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:03.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.384+0000 7f8d2cff9640 1 -- 172.21.15.45:0/143432063 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d20053d10 con 0x7f8d300d02d0 2023-11-28T20:55:03.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.650+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f8d3004e920 con 0x7f8d300d02d0 2023-11-28T20:55:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.726+0000 7f8d2cff9640 1 -- 172.21.15.45:0/143432063 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v826) v1 ==== 160+0+0 (secure 0 0 0) 0x7f8d20018070 con 0x7f8d300d02d0 2023-11-28T20:55:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.729+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8d1006af60 msgr2=0x7f8d1006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.729+0000 7f8d34aef640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8d1006af60 0x7f8d1006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8d300c6c30 tx=0x7f8d18008040 comp rx=0 tx=0).stop 2023-11-28T20:55:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d02d0 msgr2=0x7f8d3015acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d02d0 0x7f8d3015acc0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c7c0 tx=0x7f8d2000cc80 comp rx=0 tx=0).stop 2023-11-28T20:55:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 shutdown_connections 2023-11-28T20:55:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d3015d550 0x7f8d300aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8d300d1ab0 0x7f8d301599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8d1006af60 0x7f8d1006d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 --2- 172.21.15.45:0/143432063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d300d02d0 0x7f8d3015acc0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 >> 172.21.15.45:0/143432063 conn(0x7f8d3006ac70 msgr2=0x7f8d3015c680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 shutdown_connections 2023-11-28T20:55:03.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.730+0000 7f8d34aef640 1 -- 172.21.15.45:0/143432063 wait complete. 2023-11-28T20:55:03.752 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2023-11-28T20:55:03.752 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-11-28T20:55:03.752 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T20:55:03.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:55:03.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.973+0000 7f79ff375640 1 Processor -- start 2023-11-28T20:55:03.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.973+0000 7f79ff375640 1 -- start start 2023-11-28T20:55:03.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.975+0000 7f79ff375640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f79f80ce800 0x7f79f8165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.975+0000 7f79ff375640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f815c3c0 con 0x7f79f8155780 2023-11-28T20:55:03.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.975+0000 7f79ff375640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f8155b60 con 0x7f79f80ae540 2023-11-28T20:55:03.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.975+0000 7f79ff375640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f815ba90 con 0x7f79f80ce800 2023-11-28T20:55:03.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.975+0000 7f79fe373640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f79f8155780 0x7f79f80abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47142/0 (socket says 172.21.15.45:47142) 2023-11-28T20:55:03.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.975+0000 7f79fe373640 1 -- 172.21.15.45:0/1456577659 learned_addr learned my addr 172.21.15.45:0/1456577659 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:03.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.975+0000 7f79feb74640 1 --2- 172.21.15.45:0/1456577659 >> v2:172.21.15.45:3300/0 conn(0x7f79f80ce800 0x7f79f8165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:03.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.976+0000 7f79feb74640 1 -- 172.21.15.45:0/1456577659 >> v1:172.21.15.190:6789/0 conn(0x7f79f80ae540 legacy=0x7f79f80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:03.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.976+0000 7f79fd371640 1 -- 172.21.15.45:0/1456577659 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063713285 0 0) 0x7f79f815c3c0 con 0x7f79f8155780 2023-11-28T20:55:03.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.976+0000 7f79feb74640 1 -- 172.21.15.45:0/1456577659 >> v1:172.21.15.45:6789/0 conn(0x7f79f8155780 legacy=0x7f79f80abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.976+0000 7f79feb74640 1 -- 172.21.15.45:0/1456577659 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f815bd70 con 0x7f79f80ce800 2023-11-28T20:55:03.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.977+0000 7f79feb74640 1 --2- 172.21.15.45:0/1456577659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 0x7f79f8165e90 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f79f0009d90 tx=0x7f79f002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3584cf5e6eb9acc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:03.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.977+0000 7f79fd371640 1 -- 172.21.15.45:0/1456577659 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f002cc60 con 0x7f79f80ce800 2023-11-28T20:55:03.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.977+0000 7f79fd371640 1 -- 172.21.15.45:0/1456577659 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f002cdc0 con 0x7f79f80ce800 2023-11-28T20:55:03.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.977+0000 7f79fd371640 1 -- 172.21.15.45:0/1456577659 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f0034660 con 0x7f79f80ce800 2023-11-28T20:55:03.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.978+0000 7f79ff375640 1 -- 172.21.15.45:0/1456577659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 msgr2=0x7f79f8165e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.978+0000 7f79ff375640 1 --2- 172.21.15.45:0/1456577659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 0x7f79f8165e90 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f79f0009d90 tx=0x7f79f002beb0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.978+0000 7f79ff375640 1 -- 172.21.15.45:0/1456577659 shutdown_connections 2023-11-28T20:55:03.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.978+0000 7f79ff375640 1 --2- 172.21.15.45:0/1456577659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 0x7f79f8165e90 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.978+0000 7f79ff375640 1 -- 172.21.15.45:0/1456577659 >> 172.21.15.45:0/1456577659 conn(0x7f79f806ab30 msgr2=0x7f79f80b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:03.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.978+0000 7f79ff375640 1 -- 172.21.15.45:0/1456577659 shutdown_connections 2023-11-28T20:55:03.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.978+0000 7f79ff375640 1 -- 172.21.15.45:0/1456577659 wait complete. 2023-11-28T20:55:03.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.980+0000 7f79ff375640 1 Processor -- start 2023-11-28T20:55:03.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.980+0000 7f79ff375640 1 -- start start 2023-11-28T20:55:03.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.981+0000 7f79ff375640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f80ae540 0x7f79f813f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.981+0000 7f79ff375640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 0x7f79f80afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.981+0000 7f79ff375640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79f8155780 0x7f79f80b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.981+0000 7f79ff375640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f81f79b0 con 0x7f79f80ce800 2023-11-28T20:55:03.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.981+0000 7f79ff375640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f8142c60 con 0x7f79f80ae540 2023-11-28T20:55:03.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.981+0000 7f79ff375640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f8141ea0 con 0x7f79f8155780 2023-11-28T20:55:03.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.981+0000 7f79fdb72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 0x7f79f80afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:03.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.981+0000 7f79fdb72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 0x7f79f80afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49684/0 (socket says 172.21.15.45:49684) 2023-11-28T20:55:03.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.981+0000 7f79fdb72640 1 -- 172.21.15.45:0/2140402122 learned_addr learned my addr 172.21.15.45:0/2140402122 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:03.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79fe373640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f80ae540 0x7f79f813f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:03.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79feb74640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79f8155780 0x7f79f80b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:03.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79fdb72640 1 -- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79f8155780 msgr2=0x7f79f80b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79fdb72640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79f8155780 0x7f79f80b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79fdb72640 1 -- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f80ae540 msgr2=0x7f79f813f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79fdb72640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f80ae540 0x7f79f813f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79fdb72640 1 -- 172.21.15.45:0/2140402122 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f0009a70 con 0x7f79f80ce800 2023-11-28T20:55:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79fe373640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f80ae540 0x7f79f813f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79fdb72640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 0x7f79f80afff0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f79e800c460 tx=0x7f79e800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79eeffd640 1 -- 172.21.15.45:0/2140402122 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e8012830 con 0x7f79f80ce800 2023-11-28T20:55:03.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79eeffd640 1 -- 172.21.15.45:0/2140402122 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e80129d0 con 0x7f79f80ce800 2023-11-28T20:55:03.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.982+0000 7f79eeffd640 1 -- 172.21.15.45:0/2140402122 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e8012d80 con 0x7f79f80ce800 2023-11-28T20:55:03.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.983+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f8142180 con 0x7f79f80ce800 2023-11-28T20:55:03.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.983+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f8142740 con 0x7f79f80ce800 2023-11-28T20:55:03.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.984+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f8150ad0 con 0x7f79f80ce800 2023-11-28T20:55:03.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.985+0000 7f79eeffd640 1 -- 172.21.15.45:0/2140402122 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f79e801a3b0 con 0x7f79f80ce800 2023-11-28T20:55:03.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.986+0000 7f79eeffd640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f79dc06af60 0x7f79dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:03.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.986+0000 7f79eeffd640 1 -- 172.21.15.45:0/2140402122 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f79e8089600 con 0x7f79f80ce800 2023-11-28T20:55:03.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.986+0000 7f79fe373640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f79dc06af60 0x7f79dc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:03.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.988+0000 7f79fe373640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f79dc06af60 0x7f79dc06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f79f4000f60 tx=0x7f79f4009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:03.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:03.990+0000 7f79eeffd640 1 -- 172.21.15.45:0/2140402122 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79e80543c0 con 0x7f79f80ce800 2023-11-28T20:55:04.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.186+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79f804e920 con 0x7f79f80ce800 2023-11-28T20:55:04.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.188+0000 7f79eeffd640 1 -- 172.21.15.45:0/2140402122 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+14311 (secure 0 0 0) 0x7f79e80579e0 con 0x7f79f80ce800 2023-11-28T20:55:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:04.190 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":153,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:55:01.717167+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/684266044":"2023-11-29T20:54:58.695195+0000","172.21.15.190:0/279215056":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/3021259469":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/4229179752":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6833/1149934668":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6832/1149934668":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/1886852041":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/1445199812":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/649548475":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/3688544491":"2023-11-29T20:54:54.592257+0000","172.21.15.190:0/1625344948":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2599111328":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2598605134":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/2976990772":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6833/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.190:0/305890924":"2023-11-29T20:54:54.592257+0000","172.21.15.190:6832/2586319130":"2023-11-29T20:54:41.552066+0000","172.21.15.45:6834/684266044":"2023-11-29T20:54:58.695195+0000","172.21.15.190:0/1527814408":"2023-11-29T20:54:41.552066+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:55:04.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f79dc06af60 msgr2=0x7f79dc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f79dc06af60 0x7f79dc06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f79f4000f60 tx=0x7f79f4009040 comp rx=0 tx=0).stop 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 msgr2=0x7f79f80afff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 0x7f79f80afff0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f79e800c460 tx=0x7f79e800c920 comp rx=0 tx=0).stop 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 shutdown_connections 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79f8155780 0x7f79f80b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f79dc06af60 0x7f79dc06d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79f80ce800 0x7f79f80afff0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 --2- 172.21.15.45:0/2140402122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f80ae540 0x7f79f813f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 >> 172.21.15.45:0/2140402122 conn(0x7f79f806ab30 msgr2=0x7f79f80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 shutdown_connections 2023-11-28T20:55:04.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.191+0000 7f79ff375640 1 -- 172.21.15.45:0/2140402122 wait complete. 2023-11-28T20:55:04.210 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:55:04.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:55:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.444+0000 7fab83316640 1 Processor -- start 2023-11-28T20:55:04.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.444+0000 7fab83316640 1 -- start start 2023-11-28T20:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.445+0000 7fab83316640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7c0b3040 0x7fab7c06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.446+0000 7fab83316640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab7c15b270 0x7fab7c15f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.446+0000 7fab83316640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7c15faf0 con 0x7fab7c0b3420 2023-11-28T20:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.446+0000 7fab83316640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7c0ceaa0 con 0x7fab7c15b270 2023-11-28T20:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.446+0000 7fab83316640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c06d410 con 0x7fab7c0b3040 2023-11-28T20:55:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.446+0000 7fab81b13640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab7c15b270 0x7fab7c15f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.446+0000 7fab81b13640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab7c15b270 0x7fab7c15f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49686/0 (socket says 172.21.15.45:49686) 2023-11-28T20:55:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.446+0000 7fab81b13640 1 -- 172.21.15.45:0/3894010207 learned_addr learned my addr 172.21.15.45:0/3894010207 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.446+0000 7fab82314640 1 --2- 172.21.15.45:0/3894010207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7c0b3040 0x7fab7c06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.447+0000 7fab81b13640 1 -- 172.21.15.45:0/3894010207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7c0b3040 msgr2=0x7fab7c06cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.447+0000 7fab81b13640 1 --2- 172.21.15.45:0/3894010207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7c0b3040 0x7fab7c06cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.447+0000 7fab81b13640 1 -- 172.21.15.45:0/3894010207 >> v1:172.21.15.190:6789/0 conn(0x7fab7c0b3420 legacy=0x7fab7c0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.447+0000 7fab81b13640 1 -- 172.21.15.45:0/3894010207 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c06d6f0 con 0x7fab7c15b270 2023-11-28T20:55:04.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.448+0000 7fab81b13640 1 --2- 172.21.15.45:0/3894010207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c15b270 0x7fab7c15f5b0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fab70009870 tx=0x7fab7002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190cff5064685a1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:04.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.448+0000 7fab81312640 1 -- 172.21.15.45:0/3894010207 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab700317c0 con 0x7fab7c15b270 2023-11-28T20:55:04.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.448+0000 7fab81312640 1 -- 172.21.15.45:0/3894010207 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab70031920 con 0x7fab7c15b270 2023-11-28T20:55:04.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.448+0000 7fab81312640 1 -- 172.21.15.45:0/3894010207 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70031cb0 con 0x7fab7c15b270 2023-11-28T20:55:04.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.449+0000 7fab83316640 1 -- 172.21.15.45:0/3894010207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c15b270 msgr2=0x7fab7c15f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.449+0000 7fab83316640 1 --2- 172.21.15.45:0/3894010207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c15b270 0x7fab7c15f5b0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fab70009870 tx=0x7fab7002b860 comp rx=0 tx=0).stop 2023-11-28T20:55:04.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.449+0000 7fab83316640 1 -- 172.21.15.45:0/3894010207 shutdown_connections 2023-11-28T20:55:04.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.449+0000 7fab83316640 1 --2- 172.21.15.45:0/3894010207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c15b270 0x7fab7c15f5b0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.449+0000 7fab83316640 1 --2- 172.21.15.45:0/3894010207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7c0b3040 0x7fab7c06cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.449+0000 7fab83316640 1 -- 172.21.15.45:0/3894010207 >> 172.21.15.45:0/3894010207 conn(0x7fab7c0b1050 msgr2=0x7fab7c0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.449+0000 7fab83316640 1 -- 172.21.15.45:0/3894010207 shutdown_connections 2023-11-28T20:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.449+0000 7fab83316640 1 -- 172.21.15.45:0/3894010207 wait complete. 2023-11-28T20:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.451+0000 7fab83316640 1 Processor -- start 2023-11-28T20:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.451+0000 7fab83316640 1 -- start start 2023-11-28T20:55:04.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.452+0000 7fab83316640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c0b3040 0x7fab7c1e4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.452+0000 7fab83316640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7c0b3420 0x7fab7c14d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.452+0000 7fab83316640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7c15b270 0x7fab7c1e5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.452+0000 7fab83316640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c1e59a0 con 0x7fab7c0b3040 2023-11-28T20:55:04.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.452+0000 7fab83316640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c14d650 con 0x7fab7c15b270 2023-11-28T20:55:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.452+0000 7fab83316640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c1e5790 con 0x7fab7c0b3420 2023-11-28T20:55:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.452+0000 7fab82314640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c0b3040 0x7fab7c1e4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.452+0000 7fab82314640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c0b3040 0x7fab7c1e4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49694/0 (socket says 172.21.15.45:49694) 2023-11-28T20:55:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.452+0000 7fab82314640 1 -- 172.21.15.45:0/1769885301 learned_addr learned my addr 172.21.15.45:0/1769885301 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab82314640 1 -- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7c0b3420 msgr2=0x7fab7c14d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab82314640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7c0b3420 0x7fab7c14d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab82314640 1 -- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7c15b270 msgr2=0x7fab7c1e5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab82b15640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7c15b270 0x7fab7c1e5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab82314640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7c15b270 0x7fab7c1e5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab82314640 1 -- 172.21.15.45:0/1769885301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700094d0 con 0x7fab7c0b3040 2023-11-28T20:55:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab82b15640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7c15b270 0x7fab7c1e5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab82314640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c0b3040 0x7fab7c1e4ab0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00d5d0 tx=0x7fab6c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab7affd640 1 -- 172.21.15.45:0/1769885301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0125f0 con 0x7fab7c0b3040 2023-11-28T20:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab7affd640 1 -- 172.21.15.45:0/1769885301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c012770 con 0x7fab7c0b3040 2023-11-28T20:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab7affd640 1 -- 172.21.15.45:0/1769885301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c012b20 con 0x7fab7c0b3040 2023-11-28T20:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7c15c350 con 0x7fab7c0b3040 2023-11-28T20:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.453+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7c15c910 con 0x7fab7c0b3040 2023-11-28T20:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.455+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7c09d0f0 con 0x7fab7c0b3040 2023-11-28T20:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.455+0000 7fab7affd640 1 -- 172.21.15.45:0/1769885301 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fab6c012c80 con 0x7fab7c0b3040 2023-11-28T20:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.456+0000 7fab7affd640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab6806af60 0x7fab6806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.457+0000 7fab7affd640 1 -- 172.21.15.45:0/1769885301 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7fab6c089d90 con 0x7fab7c0b3040 2023-11-28T20:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.457+0000 7fab81b13640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab6806af60 0x7fab6806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.458+0000 7fab81b13640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab6806af60 0x7fab6806d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fab70036000 tx=0x7fab70030000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.461+0000 7fab7affd640 1 -- 172.21.15.45:0/1769885301 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab6c054be0 con 0x7fab7c0b3040 2023-11-28T20:55:04.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.654+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab7c04e920 con 0x7fab7c0b3040 2023-11-28T20:55:04.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.655+0000 7fab7affd640 1 -- 172.21.15.45:0/1769885301 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v153) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab6c058200 con 0x7fab7c0b3040 2023-11-28T20:55:04.656 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:55:04.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.658+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab6806af60 msgr2=0x7fab6806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.658+0000 7fab83316640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab6806af60 0x7fab6806d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fab70036000 tx=0x7fab70030000 comp rx=0 tx=0).stop 2023-11-28T20:55:04.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.658+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c0b3040 msgr2=0x7fab7c1e4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.658+0000 7fab83316640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c0b3040 0x7fab7c1e4ab0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00d5d0 tx=0x7fab6c00da90 comp rx=0 tx=0).stop 2023-11-28T20:55:04.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.659+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 shutdown_connections 2023-11-28T20:55:04.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.659+0000 7fab83316640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab7c15b270 0x7fab7c1e5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.659+0000 7fab83316640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab7c0b3420 0x7fab7c14d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.659+0000 7fab83316640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fab6806af60 0x7fab6806d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.659+0000 7fab83316640 1 --2- 172.21.15.45:0/1769885301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab7c0b3040 0x7fab7c1e4ab0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.659+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 >> 172.21.15.45:0/1769885301 conn(0x7fab7c0b1050 msgr2=0x7fab7c0ae990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.659+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 shutdown_connections 2023-11-28T20:55:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.659+0000 7fab83316640 1 -- 172.21.15.45:0/1769885301 wait complete. 2023-11-28T20:55:04.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-11-28T20:55:04.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.908+0000 7f93b94ad640 1 Processor -- start 2023-11-28T20:55:04.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.908+0000 7f93b94ad640 1 -- start start 2023-11-28T20:55:04.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b94ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93b40b3870 0x7f93b415db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b94ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b406d880 0x7f93b415e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b94ad640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b415e680 con 0x7f93b40721e0 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b94ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b415a460 con 0x7f93b40b3870 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b94ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b415e960 con 0x7f93b406d880 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93b40b3870 0x7f93b415db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b37fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f93b40721e0 0x7f93b406d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47152/0 (socket says 172.21.15.45:47152) 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b37fe640 1 -- 172.21.15.45:0/3690175244 learned_addr learned my addr 172.21.15.45:0/3690175244 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93b40b3870 0x7f93b415db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49706/0 (socket says 172.21.15.45:49706) 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.910+0000 7f93b8cac640 1 --2- 172.21.15.45:0/3690175244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b406d880 0x7f93b415e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.911+0000 7f93b3fff640 1 -- 172.21.15.45:0/3690175244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b406d880 msgr2=0x7f93b415e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.911+0000 7f93b3fff640 1 --2- 172.21.15.45:0/3690175244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b406d880 0x7f93b415e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.911+0000 7f93b3fff640 1 -- 172.21.15.45:0/3690175244 >> v1:172.21.15.45:6789/0 conn(0x7f93b40721e0 legacy=0x7f93b406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.911+0000 7f93b3fff640 1 -- 172.21.15.45:0/3690175244 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b4071700 con 0x7f93b40b3870 2023-11-28T20:55:04.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.911+0000 7f93b2ffd640 1 -- 172.21.15.45:0/3690175244 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14085389 0 0) 0x7f93b415e680 con 0x7f93b40721e0 2023-11-28T20:55:04.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.912+0000 7f93b3fff640 1 --2- 172.21.15.45:0/3690175244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40b3870 0x7f93b415db10 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009d90 tx=0x7f93a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f50fb87de44cdc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:04.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.912+0000 7f93b2ffd640 1 -- 172.21.15.45:0/3690175244 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a402cc50 con 0x7f93b40b3870 2023-11-28T20:55:04.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.912+0000 7f93b2ffd640 1 -- 172.21.15.45:0/3690175244 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a402cdb0 con 0x7f93b40b3870 2023-11-28T20:55:04.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.912+0000 7f93b2ffd640 1 -- 172.21.15.45:0/3690175244 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a4034660 con 0x7f93b40b3870 2023-11-28T20:55:04.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.913+0000 7f93b94ad640 1 -- 172.21.15.45:0/3690175244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40b3870 msgr2=0x7f93b415db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.913+0000 7f93b94ad640 1 --2- 172.21.15.45:0/3690175244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40b3870 0x7f93b415db10 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009d90 tx=0x7f93a402bea0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.913+0000 7f93b94ad640 1 -- 172.21.15.45:0/3690175244 shutdown_connections 2023-11-28T20:55:04.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.913+0000 7f93b94ad640 1 --2- 172.21.15.45:0/3690175244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b406d880 0x7f93b415e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.913+0000 7f93b94ad640 1 --2- 172.21.15.45:0/3690175244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40b3870 0x7f93b415db10 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.913+0000 7f93b94ad640 1 -- 172.21.15.45:0/3690175244 >> 172.21.15.45:0/3690175244 conn(0x7f93b406ab30 msgr2=0x7f93b40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:04.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.913+0000 7f93b94ad640 1 -- 172.21.15.45:0/3690175244 shutdown_connections 2023-11-28T20:55:04.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.913+0000 7f93b94ad640 1 -- 172.21.15.45:0/3690175244 wait complete. 2023-11-28T20:55:04.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.915+0000 7f93b94ad640 1 Processor -- start 2023-11-28T20:55:04.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.915+0000 7f93b94ad640 1 -- start start 2023-11-28T20:55:04.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.916+0000 7f93b94ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b406d880 0x7f93b41f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.916+0000 7f93b94ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40721e0 0x7f93b4158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.916+0000 7f93b94ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b40b3870 0x7f93b4148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.916+0000 7f93b94ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40b08f0 con 0x7f93b40721e0 2023-11-28T20:55:04.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.916+0000 7f93b94ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40af8e0 con 0x7f93b406d880 2023-11-28T20:55:04.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.916+0000 7f93b94ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b414e8c0 con 0x7f93b40b3870 2023-11-28T20:55:04.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.916+0000 7f93b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40721e0 0x7f93b4158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.916+0000 7f93b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40721e0 0x7f93b4158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49710/0 (socket says 172.21.15.45:49710) 2023-11-28T20:55:04.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.916+0000 7f93b37fe640 1 -- 172.21.15.45:0/2001324598 learned_addr learned my addr 172.21.15.45:0/2001324598 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:04.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.917+0000 7f93b8cac640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b40b3870 0x7f93b4148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.917+0000 7f93b3fff640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b406d880 0x7f93b41f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.917+0000 7f93b37fe640 1 -- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b40b3870 msgr2=0x7f93b4148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.917+0000 7f93b37fe640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b40b3870 0x7f93b4148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.917+0000 7f93b37fe640 1 -- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b406d880 msgr2=0x7f93b41f6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:04.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.917+0000 7f93b37fe640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b406d880 0x7f93b41f6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:04.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.917+0000 7f93b37fe640 1 -- 172.21.15.45:0/2001324598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4009a70 con 0x7f93b40721e0 2023-11-28T20:55:04.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.917+0000 7f93b37fe640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40721e0 0x7f93b4158de0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f93a800c500 tx=0x7f93a800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:04.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.918+0000 7f93b0ff9640 1 -- 172.21.15.45:0/2001324598 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8007560 con 0x7f93b40721e0 2023-11-28T20:55:04.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.918+0000 7f93b0ff9640 1 -- 172.21.15.45:0/2001324598 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a80047e0 con 0x7f93b40721e0 2023-11-28T20:55:04.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.918+0000 7f93b0ff9640 1 -- 172.21.15.45:0/2001324598 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8004b90 con 0x7f93b40721e0 2023-11-28T20:55:04.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.918+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b40afb30 con 0x7f93b40721e0 2023-11-28T20:55:04.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.918+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b40b00f0 con 0x7f93b40721e0 2023-11-28T20:55:04.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.919+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b409d0f0 con 0x7f93b40721e0 2023-11-28T20:55:04.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.920+0000 7f93b0ff9640 1 -- 172.21.15.45:0/2001324598 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f93a8004cf0 con 0x7f93b40721e0 2023-11-28T20:55:04.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.921+0000 7f93b0ff9640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f93a006af60 0x7f93a006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:04.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.921+0000 7f93b0ff9640 1 -- 172.21.15.45:0/2001324598 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f93a8088a90 con 0x7f93b40721e0 2023-11-28T20:55:04.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.922+0000 7f93b3fff640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f93a006af60 0x7f93a006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:04.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.923+0000 7f93b3fff640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f93a006af60 0x7f93a006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f93a4036000 tx=0x7f93a4030000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:04.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:04.925+0000 7f93b0ff9640 1 -- 172.21.15.45:0/2001324598 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93a80538e0 con 0x7f93b40721e0 2023-11-28T20:55:05.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.188+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f93b4099d10 con 0x7f93b40721e0 2023-11-28T20:55:05.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.731+0000 7f93b0ff9640 1 -- 172.21.15.45:0/2001324598 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v828) v1 ==== 181+0+0 (secure 0 0 0) 0x7f93a8018020 con 0x7f93b40721e0 2023-11-28T20:55:05.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f93a006af60 msgr2=0x7f93a006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:05.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f93a006af60 0x7f93a006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f93a4036000 tx=0x7f93a4030000 comp rx=0 tx=0).stop 2023-11-28T20:55:05.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40721e0 msgr2=0x7f93b4158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:05.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40721e0 0x7f93b4158de0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f93a800c500 tx=0x7f93a800c9c0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 shutdown_connections 2023-11-28T20:55:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93b40b3870 0x7f93b4148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f93a006af60 0x7f93a006d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40721e0 0x7f93b4158de0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 --2- 172.21.15.45:0/2001324598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93b406d880 0x7f93b41f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.735+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 >> 172.21.15.45:0/2001324598 conn(0x7f93b406ab30 msgr2=0x7f93b40acb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.736+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 shutdown_connections 2023-11-28T20:55:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.736+0000 7f93b94ad640 1 -- 172.21.15.45:0/2001324598 wait complete. 2023-11-28T20:55:05.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T20:55:05.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.986+0000 7f2dda2da640 1 Processor -- start 2023-11-28T20:55:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.986+0000 7f2dda2da640 1 -- start start 2023-11-28T20:55:05.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.988+0000 7f2dda2da640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2dd40adcb0 0x7f2dd406ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:05.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.988+0000 7f2dda2da640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2dd406f2c0 0x7f2dd406b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:05.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.988+0000 7f2dda2da640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd406f710 con 0x7f2dd415d230 2023-11-28T20:55:05.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.988+0000 7f2dda2da640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd406b1b0 con 0x7f2dd40adcb0 2023-11-28T20:55:05.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.988+0000 7f2dda2da640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd40b4b20 con 0x7f2dd406f2c0 2023-11-28T20:55:05.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.989+0000 7f2dd8ad7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2dd40adcb0 0x7f2dd406ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:05.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.989+0000 7f2dd8ad7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2dd40adcb0 0x7f2dd406ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49720/0 (socket says 172.21.15.45:49720) 2023-11-28T20:55:05.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.989+0000 7f2dd8ad7640 1 -- 172.21.15.45:0/2304354388 learned_addr learned my addr 172.21.15.45:0/2304354388 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:05.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.989+0000 7f2dd9ad9640 1 --2- 172.21.15.45:0/2304354388 >> v2:172.21.15.190:3300/0 conn(0x7f2dd406f2c0 0x7f2dd406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:05.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.989+0000 7f2dd8ad7640 1 -- 172.21.15.45:0/2304354388 >> v2:172.21.15.190:3300/0 conn(0x7f2dd406f2c0 msgr2=0x7f2dd406b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:05.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.989+0000 7f2dd8ad7640 1 --2- 172.21.15.45:0/2304354388 >> v2:172.21.15.190:3300/0 conn(0x7f2dd406f2c0 0x7f2dd406b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.989+0000 7f2dd8ad7640 1 -- 172.21.15.45:0/2304354388 >> v1:172.21.15.190:6789/0 conn(0x7f2dd415d230 legacy=0x7f2dd40b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:05.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.989+0000 7f2dd8ad7640 1 -- 172.21.15.45:0/2304354388 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd406a5a0 con 0x7f2dd40adcb0 2023-11-28T20:55:05.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.990+0000 7f2dd8ad7640 1 --2- 172.21.15.45:0/2304354388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd40adcb0 0x7f2dd406ed80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f2dd406e560 tx=0x7f2dc402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a67008d77166e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:05.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.991+0000 7f2dcbfff640 1 -- 172.21.15.45:0/2304354388 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4031870 con 0x7f2dd40adcb0 2023-11-28T20:55:05.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.991+0000 7f2dcbfff640 1 -- 172.21.15.45:0/2304354388 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc40319d0 con 0x7f2dd40adcb0 2023-11-28T20:55:05.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.991+0000 7f2dcbfff640 1 -- 172.21.15.45:0/2304354388 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4031d60 con 0x7f2dd40adcb0 2023-11-28T20:55:05.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.991+0000 7f2dda2da640 1 -- 172.21.15.45:0/2304354388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd40adcb0 msgr2=0x7f2dd406ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:05.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.991+0000 7f2dda2da640 1 --2- 172.21.15.45:0/2304354388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd40adcb0 0x7f2dd406ed80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f2dd406e560 tx=0x7f2dc402b910 comp rx=0 tx=0).stop 2023-11-28T20:55:05.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.992+0000 7f2dda2da640 1 -- 172.21.15.45:0/2304354388 shutdown_connections 2023-11-28T20:55:05.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.992+0000 7f2dda2da640 1 --2- 172.21.15.45:0/2304354388 >> v2:172.21.15.190:3300/0 conn(0x7f2dd406f2c0 0x7f2dd406b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.992+0000 7f2dda2da640 1 --2- 172.21.15.45:0/2304354388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd40adcb0 0x7f2dd406ed80 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.992+0000 7f2dda2da640 1 -- 172.21.15.45:0/2304354388 >> 172.21.15.45:0/2304354388 conn(0x7f2dd4099a40 msgr2=0x7f2dd4099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:05.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.992+0000 7f2dda2da640 1 -- 172.21.15.45:0/2304354388 shutdown_connections 2023-11-28T20:55:05.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.992+0000 7f2dda2da640 1 -- 172.21.15.45:0/2304354388 wait complete. 2023-11-28T20:55:05.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.994+0000 7f2dda2da640 1 Processor -- start 2023-11-28T20:55:05.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.994+0000 7f2dda2da640 1 -- start start 2023-11-28T20:55:05.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dda2da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dd406f2c0 0x7f2dd41e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dda2da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd40adcb0 0x7f2dd41e8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dda2da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd415d230 0x7f2dd415da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dda2da640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd4153590 con 0x7f2dd415d230 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dda2da640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd40cb770 con 0x7f2dd40adcb0 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dda2da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd40cb140 con 0x7f2dd406f2c0 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd9ad9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd415d230 0x7f2dd415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd9ad9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd415d230 0x7f2dd415da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49730/0 (socket says 172.21.15.45:49730) 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd9ad9640 1 -- 172.21.15.45:0/691191993 learned_addr learned my addr 172.21.15.45:0/691191993 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd8ad7640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd40adcb0 0x7f2dd41e8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd92d8640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dd406f2c0 0x7f2dd41e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd9ad9640 1 -- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dd406f2c0 msgr2=0x7f2dd41e89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd9ad9640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dd406f2c0 0x7f2dd41e89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd9ad9640 1 -- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd40adcb0 msgr2=0x7f2dd41e8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd9ad9640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd40adcb0 0x7f2dd41e8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd9ad9640 1 -- 172.21.15.45:0/691191993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc40094d0 con 0x7f2dd415d230 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd8ad7640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd40adcb0 0x7f2dd41e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.995+0000 7f2dd92d8640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dd406f2c0 0x7f2dd41e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.996+0000 7f2dd9ad9640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd415d230 0x7f2dd415da60 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc007e60 tx=0x7f2dcc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.996+0000 7f2dc9ffb640 1 -- 172.21.15.45:0/691191993 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc014420 con 0x7f2dd415d230 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.996+0000 7f2dc9ffb640 1 -- 172.21.15.45:0/691191993 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc0145a0 con 0x7f2dd415d230 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.996+0000 7f2dc9ffb640 1 -- 172.21.15.45:0/691191993 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc014950 con 0x7f2dd415d230 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.996+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd41e9790 con 0x7f2dd415d230 2023-11-28T20:55:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.996+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd41e9470 con 0x7f2dd415d230 2023-11-28T20:55:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.997+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd40cb900 con 0x7f2dd415d230 2023-11-28T20:55:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.998+0000 7f2dc9ffb640 1 -- 172.21.15.45:0/691191993 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2dcc014ab0 con 0x7f2dd415d230 2023-11-28T20:55:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.999+0000 7f2dc9ffb640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f2db806af90 0x7f2db806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:06.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:05.999+0000 7f2dc9ffb640 1 -- 172.21.15.45:0/691191993 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f2dcc08a270 con 0x7f2dd415d230 2023-11-28T20:55:06.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.000+0000 7f2dd92d8640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f2db806af90 0x7f2db806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:06.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.001+0000 7f2dd92d8640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f2db806af90 0x7f2db806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0003eb0 tx=0x7f2dd0009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:06.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.004+0000 7f2dc9ffb640 1 -- 172.21.15.45:0/691191993 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dcc055030 con 0x7f2dd415d230 2023-11-28T20:55:06.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.198+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2dd4160ba0 con 0x7f2dd415d230 2023-11-28T20:55:06.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.205+0000 7f2dc9ffb640 1 -- 172.21.15.45:0/691191993 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2dcc058650 con 0x7f2dd415d230 2023-11-28T20:55:06.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f2db806af90 msgr2=0x7f2db806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:06.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f2db806af90 0x7f2db806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0003eb0 tx=0x7f2dd0009040 comp rx=0 tx=0).stop 2023-11-28T20:55:06.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd415d230 msgr2=0x7f2dd415da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd415d230 0x7f2dd415da60 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc007e60 tx=0x7f2dcc00e4b0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 shutdown_connections 2023-11-28T20:55:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f2db806af90 0x7f2db806d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd415d230 0x7f2dd415da60 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd40adcb0 0x7f2dd41e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 --2- 172.21.15.45:0/691191993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2dd406f2c0 0x7f2dd41e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.208+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 >> 172.21.15.45:0/691191993 conn(0x7f2dd4099a40 msgr2=0x7f2dd40b4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.209+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 shutdown_connections 2023-11-28T20:55:06.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.209+0000 7f2dda2da640 1 -- 172.21.15.45:0/691191993 wait complete. 2023-11-28T20:55:06.230 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T20:55:06.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:55:06.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T20:55:06.485 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T20:55:06.485 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:55:06.485 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:55:06.549 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:55:06.549 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:55:06.549 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:55:06.607 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:55:06.607 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:55:06.607 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:55:06.671 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T20:55:06.672 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T20:55:06.672 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:55:06.732 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T20:55:06.732 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T20:55:06.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:55:06.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.961+0000 7ffb1cdc9640 1 Processor -- start 2023-11-28T20:55:06.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.961+0000 7ffb1cdc9640 1 -- start start 2023-11-28T20:55:06.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.963+0000 7ffb1cdc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 0x7ffb180b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:06.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.963+0000 7ffb1cdc9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffb180b24e0 0x7ffb180b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:06.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.963+0000 7ffb1cdc9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ffb180b1170 0x7ffb180ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:06.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.963+0000 7ffb1cdc9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1815dcc0 con 0x7ffb180b24e0 2023-11-28T20:55:06.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.963+0000 7ffb1cdc9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1815e840 con 0x7ffb180b1170 2023-11-28T20:55:06.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.963+0000 7ffb1cdc9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb180b15b0 con 0x7ffb181438b0 2023-11-28T20:55:06.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.963+0000 7ffb15d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffb180b24e0 0x7ffb180b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:06.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.963+0000 7ffb15d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffb180b24e0 0x7ffb180b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49734/0 (socket says 172.21.15.45:49734) 2023-11-28T20:55:06.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.963+0000 7ffb15d9b640 1 -- 172.21.15.45:0/2626211601 learned_addr learned my addr 172.21.15.45:0/2626211601 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:06.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.964+0000 7ffb16d9d640 1 --2- 172.21.15.45:0/2626211601 >> v2:172.21.15.190:3300/0 conn(0x7ffb180b1170 0x7ffb180ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:06.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.964+0000 7ffb1659c640 1 --2- 172.21.15.45:0/2626211601 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 0x7ffb180b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:06.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.964+0000 7ffb15d9b640 1 -- 172.21.15.45:0/2626211601 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 msgr2=0x7ffb180b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:06.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.964+0000 7ffb15d9b640 1 --2- 172.21.15.45:0/2626211601 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 0x7ffb180b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.964+0000 7ffb15d9b640 1 -- 172.21.15.45:0/2626211601 >> v2:172.21.15.190:3300/0 conn(0x7ffb180b1170 msgr2=0x7ffb180ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:06.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.964+0000 7ffb15d9b640 1 --2- 172.21.15.45:0/2626211601 >> v2:172.21.15.190:3300/0 conn(0x7ffb180b1170 0x7ffb180ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.964+0000 7ffb15d9b640 1 -- 172.21.15.45:0/2626211601 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1815df10 con 0x7ffb180b24e0 2023-11-28T20:55:06.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.965+0000 7ffb15d9b640 1 --2- 172.21.15.45:0/2626211601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b24e0 0x7ffb180b0c30 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7ffb18158290 tx=0x7ffb0c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1231b346b1521b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:06.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.965+0000 7ffb1559a640 1 -- 172.21.15.45:0/2626211601 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c02c950 con 0x7ffb180b24e0 2023-11-28T20:55:06.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.965+0000 7ffb1559a640 1 -- 172.21.15.45:0/2626211601 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c02cab0 con 0x7ffb180b24e0 2023-11-28T20:55:06.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.965+0000 7ffb1559a640 1 -- 172.21.15.45:0/2626211601 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c02ce40 con 0x7ffb180b24e0 2023-11-28T20:55:06.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.966+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2626211601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b24e0 msgr2=0x7ffb180b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:06.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.966+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2626211601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b24e0 0x7ffb180b0c30 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7ffb18158290 tx=0x7ffb0c02bdb0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.966+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2626211601 shutdown_connections 2023-11-28T20:55:06.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.966+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2626211601 >> v2:172.21.15.190:3300/0 conn(0x7ffb180b1170 0x7ffb180ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.966+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2626211601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b24e0 0x7ffb180b0c30 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.966+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2626211601 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 0x7ffb180b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.966+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2626211601 >> 172.21.15.45:0/2626211601 conn(0x7ffb1806a500 msgr2=0x7ffb1806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:06.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.966+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2626211601 shutdown_connections 2023-11-28T20:55:06.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.966+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2626211601 wait complete. 2023-11-28T20:55:06.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.968+0000 7ffb1cdc9640 1 Processor -- start 2023-11-28T20:55:06.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.968+0000 7ffb1cdc9640 1 -- start start 2023-11-28T20:55:06.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.969+0000 7ffb1cdc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b1170 0x7ffb181402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.969+0000 7ffb1cdc9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb180b24e0 0x7ffb1813a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.969+0000 7ffb1cdc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 0x7ffb1813aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.969+0000 7ffb1cdc9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb181380d0 con 0x7ffb180b1170 2023-11-28T20:55:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.969+0000 7ffb1cdc9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb181383b0 con 0x7ffb180b24e0 2023-11-28T20:55:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.969+0000 7ffb1cdc9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18138690 con 0x7ffb181438b0 2023-11-28T20:55:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb1659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b1170 0x7ffb181402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb1659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b1170 0x7ffb181402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49736/0 (socket says 172.21.15.45:49736) 2023-11-28T20:55:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb1659c640 1 -- 172.21.15.45:0/2989956762 learned_addr learned my addr 172.21.15.45:0/2989956762 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb15d9b640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb180b24e0 0x7ffb1813a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb16d9d640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 0x7ffb1813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb1659c640 1 -- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 msgr2=0x7ffb1813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:06.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb1659c640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 0x7ffb1813aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb1659c640 1 -- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb180b24e0 msgr2=0x7ffb1813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:06.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb1659c640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb180b24e0 0x7ffb1813a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:06.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.970+0000 7ffb1659c640 1 -- 172.21.15.45:0/2989956762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c009b30 con 0x7ffb180b1170 2023-11-28T20:55:06.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.971+0000 7ffb1659c640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b1170 0x7ffb181402a0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800c7f0 tx=0x7ffb0800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:06.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.971+0000 7ffb06ffd640 1 -- 172.21.15.45:0/2989956762 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08004220 con 0x7ffb180b1170 2023-11-28T20:55:06.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.971+0000 7ffb06ffd640 1 -- 172.21.15.45:0/2989956762 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb08004380 con 0x7ffb180b1170 2023-11-28T20:55:06.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.971+0000 7ffb06ffd640 1 -- 172.21.15.45:0/2989956762 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08007840 con 0x7ffb180b1170 2023-11-28T20:55:06.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.971+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb18138970 con 0x7ffb180b1170 2023-11-28T20:55:06.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.971+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb181407e0 con 0x7ffb180b1170 2023-11-28T20:55:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.972+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb180733d0 con 0x7ffb180b1170 2023-11-28T20:55:06.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.973+0000 7ffb06ffd640 1 -- 172.21.15.45:0/2989956762 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ffb08007bb0 con 0x7ffb180b1170 2023-11-28T20:55:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.974+0000 7ffb06ffd640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ffaf806af60 0x7ffaf806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:06.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.975+0000 7ffb06ffd640 1 -- 172.21.15.45:0/2989956762 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7ffb08089830 con 0x7ffb180b1170 2023-11-28T20:55:06.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.975+0000 7ffb15d9b640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ffaf806af60 0x7ffaf806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:06.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.976+0000 7ffb15d9b640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ffaf806af60 0x7ffaf806d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c035000 tx=0x7ffb0c030000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:06.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:06.979+0000 7ffb06ffd640 1 -- 172.21.15.45:0/2989956762 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb08054570 con 0x7ffb180b1170 2023-11-28T20:55:07.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.213+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb1806ae00 con 0x7ffb180b1170 2023-11-28T20:55:07.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.214+0000 7ffb06ffd640 1 -- 172.21.15.45:0/2989956762 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+3705 (secure 0 0 0) 0x7ffb0801ee90 con 0x7ffb180b1170 2023-11-28T20:55:07.215 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:07.215 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/1133315976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1133315976},{"type":"v1","addr":"172.21.15.190:6836","nonce":1133315976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2810069664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2810069664},{"type":"v1","addr":"172.21.15.190:6837","nonce":2810069664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1553174870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1553174870},{"type":"v1","addr":"172.21.15.45:6833","nonce":1553174870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":9057,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1196218691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1196218691},{"type":"v1","addr":"172.21.15.45:6835","nonce":1196218691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[]} 2023-11-28T20:55:07.216 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 69 2023-11-28T20:55:07.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ffaf806af60 msgr2=0x7ffaf806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:07.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ffaf806af60 0x7ffaf806d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c035000 tx=0x7ffb0c030000 comp rx=0 tx=0).stop 2023-11-28T20:55:07.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b1170 msgr2=0x7ffb181402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:07.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b1170 0x7ffb181402a0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800c7f0 tx=0x7ffb0800ccb0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 shutdown_connections 2023-11-28T20:55:07.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb181438b0 0x7ffb1813aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb180b24e0 0x7ffb1813a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ffaf806af60 0x7ffaf806d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 --2- 172.21.15.45:0/2989956762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb180b1170 0x7ffb181402a0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 >> 172.21.15.45:0/2989956762 conn(0x7ffb1806a500 msgr2=0x7ffb1806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:07.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 shutdown_connections 2023-11-28T20:55:07.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.217+0000 7ffb1cdc9640 1 -- 172.21.15.45:0/2989956762 wait complete. 2023-11-28T20:55:07.239 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T20:55:07.239 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T20:55:07.239 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T20:55:07.239 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:55:07.240 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T20:55:07.240 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T20:55:07.240 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T20:55:07.240 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:55:07.240 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T20:55:07.240 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T20:55:07.240 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T20:55:07.241 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:55:07.241 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T20:55:07.241 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T20:55:07.241 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T20:55:07.241 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T20:55:07.256 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T20:55:07.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T20:55:07.257 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T20:55:07.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T20:55:07.261 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T20:55:07.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T20:55:07.262 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T20:55:07.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T20:55:07.302 INFO:tasks.ceph.mds.a:Started 2023-11-28T20:55:07.302 DEBUG:teuthology.parallel:result is None 2023-11-28T20:55:07.309 INFO:tasks.ceph.mds.c:Started 2023-11-28T20:55:07.309 DEBUG:teuthology.parallel:result is None 2023-11-28T20:55:07.310 INFO:tasks.ceph.mds.d:Started 2023-11-28T20:55:07.311 DEBUG:teuthology.parallel:result is None 2023-11-28T20:55:07.317 INFO:tasks.ceph.mds.b:Started 2023-11-28T20:55:07.318 DEBUG:teuthology.parallel:result is None 2023-11-28T20:55:07.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T20:55:07.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-11-28T20:55:07.466 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T20:55:07.474 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T20:55:07.476 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T20:55:07.477 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T20:55:07.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.512+0000 7faf64094640 1 Processor -- start 2023-11-28T20:55:07.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.512+0000 7faf64094640 1 -- start start 2023-11-28T20:55:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.513+0000 7faf64094640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf5c15d730 0x7faf5c0d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.513+0000 7faf64094640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faf5c071230 0x7faf5c0b6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.513+0000 7faf64094640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5c06b580 con 0x7faf5c0d2750 2023-11-28T20:55:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.513+0000 7faf64094640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf5c06b2a0 con 0x7faf5c071230 2023-11-28T20:55:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.513+0000 7faf64094640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf5c0d1730 con 0x7faf5c15d730 2023-11-28T20:55:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.514+0000 7faf62631640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faf5c071230 0x7faf5c0b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.514+0000 7faf62631640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faf5c071230 0x7faf5c0b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49768/0 (socket says 172.21.15.45:49768) 2023-11-28T20:55:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.514+0000 7faf62631640 1 -- 172.21.15.45:0/3947574387 learned_addr learned my addr 172.21.15.45:0/3947574387 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.514+0000 7faf61e30640 1 --2- 172.21.15.45:0/3947574387 >> v2:172.21.15.190:3300/0 conn(0x7faf5c15d730 0x7faf5c0d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:07.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.514+0000 7faf62631640 1 -- 172.21.15.45:0/3947574387 >> v2:172.21.15.190:3300/0 conn(0x7faf5c15d730 msgr2=0x7faf5c0d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:07.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.514+0000 7faf62631640 1 --2- 172.21.15.45:0/3947574387 >> v2:172.21.15.190:3300/0 conn(0x7faf5c15d730 0x7faf5c0d2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.514+0000 7faf62631640 1 -- 172.21.15.45:0/3947574387 >> v1:172.21.15.190:6789/0 conn(0x7faf5c0d2750 legacy=0x7faf5c070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:07.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.514+0000 7faf62631640 1 -- 172.21.15.45:0/3947574387 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c0d1a10 con 0x7faf5c071230 2023-11-28T20:55:07.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.515+0000 7faf62631640 1 --2- 172.21.15.45:0/3947574387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c071230 0x7faf5c0b6670 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7faf5c08b650 tx=0x7faf5402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197c34efecfe792b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:07.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.515+0000 7faf60e2e640 1 -- 172.21.15.45:0/3947574387 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5402cb20 con 0x7faf5c071230 2023-11-28T20:55:07.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf60e2e640 1 -- 172.21.15.45:0/3947574387 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5402cc80 con 0x7faf5c071230 2023-11-28T20:55:07.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf60e2e640 1 -- 172.21.15.45:0/3947574387 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf540343f0 con 0x7faf5c071230 2023-11-28T20:55:07.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf64094640 1 -- 172.21.15.45:0/3947574387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c071230 msgr2=0x7faf5c0b6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:07.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf64094640 1 --2- 172.21.15.45:0/3947574387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c071230 0x7faf5c0b6670 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7faf5c08b650 tx=0x7faf5402bf80 comp rx=0 tx=0).stop 2023-11-28T20:55:07.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf64094640 1 -- 172.21.15.45:0/3947574387 shutdown_connections 2023-11-28T20:55:07.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf64094640 1 --2- 172.21.15.45:0/3947574387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c071230 0x7faf5c0b6670 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf64094640 1 --2- 172.21.15.45:0/3947574387 >> v2:172.21.15.190:3300/0 conn(0x7faf5c15d730 0x7faf5c0d2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf64094640 1 -- 172.21.15.45:0/3947574387 >> 172.21.15.45:0/3947574387 conn(0x7faf5c0cfc70 msgr2=0x7faf5c0d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:07.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf64094640 1 -- 172.21.15.45:0/3947574387 shutdown_connections 2023-11-28T20:55:07.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.516+0000 7faf64094640 1 -- 172.21.15.45:0/3947574387 wait complete. 2023-11-28T20:55:07.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.518+0000 7faf64094640 1 Processor -- start 2023-11-28T20:55:07.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.518+0000 7faf64094640 1 -- start start 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.519+0000 7faf64094640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf5c071230 0x7faf5c14a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.519+0000 7faf64094640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf5c0d2750 0x7faf5c157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.519+0000 7faf64094640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c15d730 0x7faf5c0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.519+0000 7faf64094640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c1f4d40 con 0x7faf5c15d730 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.519+0000 7faf64094640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c1f5020 con 0x7faf5c071230 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.519+0000 7faf64094640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c1f5300 con 0x7faf5c0d2750 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.519+0000 7faf62631640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c15d730 0x7faf5c0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.519+0000 7faf62631640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c15d730 0x7faf5c0ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49770/0 (socket says 172.21.15.45:49770) 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.519+0000 7faf62631640 1 -- 172.21.15.45:0/2679674191 learned_addr learned my addr 172.21.15.45:0/2679674191 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:07.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.520+0000 7faf61e30640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf5c071230 0x7faf5c14a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.520+0000 7faf6162f640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf5c0d2750 0x7faf5c157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.520+0000 7faf62631640 1 -- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf5c0d2750 msgr2=0x7faf5c157f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.520+0000 7faf62631640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf5c0d2750 0x7faf5c157f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.520+0000 7faf62631640 1 -- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf5c071230 msgr2=0x7faf5c14a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.520+0000 7faf62631640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf5c071230 0x7faf5c14a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:07.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.520+0000 7faf62631640 1 -- 172.21.15.45:0/2679674191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf54009b30 con 0x7faf5c15d730 2023-11-28T20:55:07.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.520+0000 7faf62631640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c15d730 0x7faf5c0ccde0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7faf54036040 tx=0x7faf54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:07.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.520+0000 7faf527fc640 1 -- 172.21.15.45:0/2679674191 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf54009510 con 0x7faf5c15d730 2023-11-28T20:55:07.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.521+0000 7faf527fc640 1 -- 172.21.15.45:0/2679674191 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5402c920 con 0x7faf5c15d730 2023-11-28T20:55:07.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.521+0000 7faf527fc640 1 -- 172.21.15.45:0/2679674191 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5403e690 con 0x7faf5c15d730 2023-11-28T20:55:07.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.521+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5c15dc90 con 0x7faf5c15d730 2023-11-28T20:55:07.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.521+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5c1f58c0 con 0x7faf5c15d730 2023-11-28T20:55:07.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.522+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5c0cf830 con 0x7faf5c15d730 2023-11-28T20:55:07.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.523+0000 7faf527fc640 1 -- 172.21.15.45:0/2679674191 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7faf5403e7f0 con 0x7faf5c15d730 2023-11-28T20:55:07.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.524+0000 7faf527fc640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7faf4006af90 0x7faf4006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:07.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.524+0000 7faf527fc640 1 -- 172.21.15.45:0/2679674191 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6022+0+0 (secure 0 0 0) 0x7faf540ac440 con 0x7faf5c15d730 2023-11-28T20:55:07.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.524+0000 7faf61e30640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7faf4006af90 0x7faf4006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:07.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.525+0000 7faf61e30640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7faf4006af90 0x7faf4006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faf58009aa0 tx=0x7faf58009190 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:07.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.529+0000 7faf527fc640 1 -- 172.21.15.45:0/2679674191 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf54077170 con 0x7faf5c15d730 2023-11-28T20:55:07.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:07.736+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7faf5c073440 con 0x7faf5c15d730 2023-11-28T20:55:08.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.227+0000 7faf527fc640 1 -- 172.21.15.45:0/2679674191 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v155) v1 ==== 97+0+0 (secure 0 0 0) 0x7faf5407a790 con 0x7faf5c15d730 2023-11-28T20:55:08.229 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T20:55:08.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7faf4006af90 msgr2=0x7faf4006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:08.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7faf4006af90 0x7faf4006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faf58009aa0 tx=0x7faf58009190 comp rx=0 tx=0).stop 2023-11-28T20:55:08.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c15d730 msgr2=0x7faf5c0ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:08.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c15d730 0x7faf5c0ccde0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7faf54036040 tx=0x7faf54004a40 comp rx=0 tx=0).stop 2023-11-28T20:55:08.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 shutdown_connections 2023-11-28T20:55:08.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7faf4006af90 0x7faf4006d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c15d730 0x7faf5c0ccde0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf5c0d2750 0x7faf5c157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 --2- 172.21.15.45:0/2679674191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf5c071230 0x7faf5c14a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 >> 172.21.15.45:0/2679674191 conn(0x7faf5c0cfc70 msgr2=0x7faf5c0b4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:08.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.231+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 shutdown_connections 2023-11-28T20:55:08.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.232+0000 7faf64094640 1 -- 172.21.15.45:0/2679674191 wait complete. 2023-11-28T20:55:08.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T20:55:08.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.473+0000 7f6f13500640 1 Processor -- start 2023-11-28T20:55:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.474+0000 7f6f13500640 1 -- start start 2023-11-28T20:55:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.475+0000 7f6f13500640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6f0c0b4c40 0x7f6f0c0b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.475+0000 7f6f13500640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c0711c0 con 0x7f6f0c161960 2023-11-28T20:55:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.475+0000 7f6f13500640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c0714a0 con 0x7f6f0c0b0c30 2023-11-28T20:55:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.475+0000 7f6f13500640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0c071e40 con 0x7f6f0c0b4c40 2023-11-28T20:55:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.476+0000 7f6f1129c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6f0c161960 0x7f6f0c070330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47172/0 (socket says 172.21.15.45:47172) 2023-11-28T20:55:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.476+0000 7f6f1129c640 1 -- 172.21.15.45:0/1852193431 learned_addr learned my addr 172.21.15.45:0/1852193431 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.476+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621459774 0 0) 0x7f6f0c0711c0 con 0x7f6f0c161960 2023-11-28T20:55:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.476+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef8003520 con 0x7f6f0c161960 2023-11-28T20:55:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.476+0000 7f6f10a9b640 1 --2- 172.21.15.45:0/1852193431 >> v2:172.21.15.190:3300/0 conn(0x7f6f0c0b4c40 0x7f6f0c0b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.476+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572189390 0 0) 0x7f6ef8003520 con 0x7f6f0c161960 2023-11-28T20:55:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.476+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 >> v2:172.21.15.190:3300/0 conn(0x7f6f0c0b4c40 msgr2=0x7f6f0c0b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.476+0000 7f6f03fff640 1 --2- 172.21.15.45:0/1852193431 >> v2:172.21.15.190:3300/0 conn(0x7f6f0c0b4c40 0x7f6f0c0b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.476+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 >> v1:172.21.15.190:6789/0 conn(0x7f6f0c0b0c30 legacy=0x7f6f0c0b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c071780 con 0x7f6f0c161960 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6f080019f0 con 0x7f6f0c161960 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 >> v1:172.21.15.45:6789/0 conn(0x7f6f0c161960 legacy=0x7f6f0c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f03fff640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6ef8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f03fff640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef80047c0 0x7f6ef8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f03fff640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ef8005250 0x7f6ef800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c071e40 con 0x7f6ef80039c0 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef800c1a0 con 0x7f6ef80047c0 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef800c360 con 0x7f6ef8005250 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f1129c640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6ef8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f1129c640 1 -- 172.21.15.45:0/1852193431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ef8005250 msgr2=0x7f6ef800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:08.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f1129c640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ef8005250 0x7f6ef800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f10a9b640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef80047c0 0x7f6ef8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:08.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f1129c640 1 -- 172.21.15.45:0/1852193431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef80047c0 msgr2=0x7f6ef8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:08.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f1129c640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef80047c0 0x7f6ef8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f11a9d640 1 -- 172.21.15.45:0/1852193431 reap_dead start 2023-11-28T20:55:08.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.477+0000 7f6f1129c640 1 -- 172.21.15.45:0/1852193431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef8003520 con 0x7f6ef80039c0 2023-11-28T20:55:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.479+0000 7f6f1129c640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6ef8003dc0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800a810 tx=0x7f6ef80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8840a10b7519c721 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.479+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08034630 con 0x7f6ef80039c0 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.479+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0802f070 con 0x7f6ef80039c0 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.479+0000 7f6f03fff640 1 -- 172.21.15.45:0/1852193431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08034a10 con 0x7f6ef80039c0 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.479+0000 7f6f13500640 1 -- 172.21.15.45:0/1852193431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 msgr2=0x7f6ef8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.479+0000 7f6f13500640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6ef8003dc0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800a810 tx=0x7f6ef80057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.479+0000 7f6f13500640 1 -- 172.21.15.45:0/1852193431 shutdown_connections 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.479+0000 7f6f13500640 1 --2- 172.21.15.45:0/1852193431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6ef8003dc0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.479+0000 7f6f13500640 1 -- 172.21.15.45:0/1852193431 >> 172.21.15.45:0/1852193431 conn(0x7f6f0c06a500 msgr2=0x7f6f0c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.480+0000 7f6f13500640 1 -- 172.21.15.45:0/1852193431 shutdown_connections 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.480+0000 7f6f13500640 1 -- 172.21.15.45:0/1852193431 wait complete. 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.481+0000 7f6f13500640 1 Processor -- start 2023-11-28T20:55:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.481+0000 7f6f13500640 1 -- start start 2023-11-28T20:55:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.482+0000 7f6f13500640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6f0c1f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:08.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.482+0000 7f6f13500640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f0c153fc0 0x7f6f0c159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.482+0000 7f6f13500640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f0c1faa00 0x7f6f0c0cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.482+0000 7f6f13500640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c1f7580 con 0x7f6ef80039c0 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.482+0000 7f6f13500640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c153950 con 0x7f6f0c153fc0 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.482+0000 7f6f13500640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c159170 con 0x7f6f0c1faa00 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f1129c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6f0c1f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f1129c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6f0c1f6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49776/0 (socket says 172.21.15.45:49776) 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f1129c640 1 -- 172.21.15.45:0/3981585078 learned_addr learned my addr 172.21.15.45:0/3981585078 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f11a9d640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f0c1faa00 0x7f6f0c0cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f10a9b640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f0c153fc0 0x7f6f0c159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f1129c640 1 -- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f0c1faa00 msgr2=0x7f6f0c0cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:08.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f1129c640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f0c1faa00 0x7f6f0c0cb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f1129c640 1 -- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f0c153fc0 msgr2=0x7f6f0c159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f1129c640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f0c153fc0 0x7f6f0c159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f1129c640 1 -- 172.21.15.45:0/3981585078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c154450 con 0x7f6ef80039c0 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.483+0000 7f6f10a9b640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f0c153fc0 0x7f6f0c159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.484+0000 7f6f1129c640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6f0c1f6f20 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800a930 tx=0x7f6f08001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.484+0000 7f6f01ffb640 1 -- 172.21.15.45:0/3981585078 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08042070 con 0x7f6ef80039c0 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.484+0000 7f6f01ffb640 1 -- 172.21.15.45:0/3981585078 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f08001830 con 0x7f6ef80039c0 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.484+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c154730 con 0x7f6ef80039c0 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.484+0000 7f6f01ffb640 1 -- 172.21.15.45:0/3981585078 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08034430 con 0x7f6ef80039c0 2023-11-28T20:55:08.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.484+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0c154cf0 con 0x7f6ef80039c0 2023-11-28T20:55:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.485+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c0cecf0 con 0x7f6ef80039c0 2023-11-28T20:55:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.486+0000 7f6f01ffb640 1 -- 172.21.15.45:0/3981585078 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6f08031030 con 0x7f6ef80039c0 2023-11-28T20:55:08.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.487+0000 7f6f01ffb640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ef006af60 0x7f6ef006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.487+0000 7f6f01ffb640 1 -- 172.21.15.45:0/3981585078 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f6f080aafe0 con 0x7f6ef80039c0 2023-11-28T20:55:08.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.488+0000 7f6f10a9b640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ef006af60 0x7f6ef006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:08.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.489+0000 7f6f10a9b640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ef006af60 0x7f6ef006d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0b41a0 tx=0x7f6efc008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:08.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.492+0000 7f6f01ffb640 1 -- 172.21.15.45:0/3981585078 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f08079b00 con 0x7f6ef80039c0 2023-11-28T20:55:08.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.758+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6f0c0cecf0 con 0x7f6ef80039c0 2023-11-28T20:55:08.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.759+0000 7f6f01ffb640 1 -- 172.21.15.45:0/3981585078 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6f08079690 con 0x7f6ef80039c0 2023-11-28T20:55:08.761 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:08.761 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:55:08.761 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T20:55:08.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T20:55:08.762 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.763 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:55:08.764 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:55:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T20:55:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T20:55:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T20:55:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T20:55:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T20:55:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:55:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:55:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T20:55:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T20:55:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T20:55:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T20:55:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T20:55:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T20:55:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T20:55:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T20:55:08.777 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:55:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ef006af60 msgr2=0x7f6ef006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ef006af60 0x7f6ef006d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0b41a0 tx=0x7f6efc008040 comp rx=0 tx=0).stop 2023-11-28T20:55:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 msgr2=0x7f6f0c1f6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6f0c1f6f20 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800a930 tx=0x7f6f08001cb0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 shutdown_connections 2023-11-28T20:55:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6f0c1faa00 0x7f6f0c0cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f0c153fc0 0x7f6f0c159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ef006af60 0x7f6ef006d420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 --2- 172.21.15.45:0/3981585078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ef80039c0 0x7f6f0c1f6f20 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 >> 172.21.15.45:0/3981585078 conn(0x7f6f0c06a500 msgr2=0x7f6f0c06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 shutdown_connections 2023-11-28T20:55:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:08.762+0000 7f6f13500640 1 -- 172.21.15.45:0/3981585078 wait complete. 2023-11-28T20:55:08.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:55:09.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.012+0000 7feb39e9b640 1 Processor -- start 2023-11-28T20:55:09.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.012+0000 7feb39e9b640 1 -- start start 2023-11-28T20:55:09.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.013+0000 7feb39e9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb341402f0 0x7feb34160e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.013+0000 7feb39e9b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb34161df0 con 0x7feb341434c0 2023-11-28T20:55:09.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.013+0000 7feb39e9b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb341406d0 con 0x7feb340703b0 2023-11-28T20:55:09.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.013+0000 7feb39e9b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb34143950 con 0x7feb341402f0 2023-11-28T20:55:09.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.014+0000 7feb32ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7feb341434c0 0x7feb341409f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47178/0 (socket says 172.21.15.45:47178) 2023-11-28T20:55:09.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.014+0000 7feb32ffd640 1 -- 172.21.15.45:0/3667931115 learned_addr learned my addr 172.21.15.45:0/3667931115 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.014+0000 7feb337fe640 1 --2- 172.21.15.45:0/3667931115 >> v2:172.21.15.190:3300/0 conn(0x7feb341402f0 0x7feb34160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.014+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144924214 0 0) 0x7feb34161df0 con 0x7feb341434c0 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.014+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb14003520 con 0x7feb341434c0 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.014+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3456971805 0 0) 0x7feb14003520 con 0x7feb341434c0 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.014+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 >> v2:172.21.15.190:3300/0 conn(0x7feb341402f0 msgr2=0x7feb34160e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.014+0000 7feb327fc640 1 --2- 172.21.15.45:0/3667931115 >> v2:172.21.15.190:3300/0 conn(0x7feb341402f0 0x7feb34160e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 >> v1:172.21.15.190:6789/0 conn(0x7feb340703b0 legacy=0x7feb340d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb34161550 con 0x7feb341434c0 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7feb1c0019f0 con 0x7feb341434c0 2023-11-28T20:55:09.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 >> v1:172.21.15.45:6789/0 conn(0x7feb341434c0 legacy=0x7feb341409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb14003970 0x7feb14003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb140047c0 0x7feb14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb14005180 0x7feb1400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb34143950 con 0x7feb14003970 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1400c120 con 0x7feb14005180 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1400c2e0 con 0x7feb140047c0 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb337fe640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb14003970 0x7feb14003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb32ffd640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb140047c0 0x7feb14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb33fff640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb14005180 0x7feb1400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb337fe640 1 -- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb140047c0 msgr2=0x7feb14004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:09.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb337fe640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb140047c0 0x7feb14004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb337fe640 1 -- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb14005180 msgr2=0x7feb1400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:09.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb337fe640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb14005180 0x7feb1400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb337fe640 1 -- 172.21.15.45:0/3667931115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb14003520 con 0x7feb14003970 2023-11-28T20:55:09.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb32ffd640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb140047c0 0x7feb14004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:09.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.015+0000 7feb33fff640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb14005180 0x7feb1400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:09.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.016+0000 7feb33fff640 1 -- 172.21.15.45:0/3667931115 reap_dead start 2023-11-28T20:55:09.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.017+0000 7feb337fe640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb14003970 0x7feb14003d90 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ac0 tx=0x7feb140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557546b0b714606c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:09.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.017+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2402c6b0 con 0x7feb14003970 2023-11-28T20:55:09.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.017+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2402c810 con 0x7feb14003970 2023-11-28T20:55:09.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.017+0000 7feb327fc640 1 -- 172.21.15.45:0/3667931115 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2402cba0 con 0x7feb14003970 2023-11-28T20:55:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.017+0000 7feb39e9b640 1 -- 172.21.15.45:0/3667931115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb14003970 msgr2=0x7feb14003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.017+0000 7feb39e9b640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb14003970 0x7feb14003d90 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ac0 tx=0x7feb140057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.018+0000 7feb39e9b640 1 -- 172.21.15.45:0/3667931115 shutdown_connections 2023-11-28T20:55:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.018+0000 7feb39e9b640 1 --2- 172.21.15.45:0/3667931115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb14003970 0x7feb14003d90 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.018+0000 7feb39e9b640 1 -- 172.21.15.45:0/3667931115 >> 172.21.15.45:0/3667931115 conn(0x7feb3406e7c0 msgr2=0x7feb3406ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.018+0000 7feb39e9b640 1 -- 172.21.15.45:0/3667931115 shutdown_connections 2023-11-28T20:55:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.018+0000 7feb39e9b640 1 -- 172.21.15.45:0/3667931115 wait complete. 2023-11-28T20:55:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.020+0000 7feb39e9b640 1 Processor -- start 2023-11-28T20:55:09.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.020+0000 7feb39e9b640 1 -- start start 2023-11-28T20:55:09.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb39e9b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb14003970 0x7feb340ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb39e9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb3415b920 0x7feb340cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb39e9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb34141910 0x7feb34141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb39e9b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb341422d0 con 0x7feb3415b920 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb39e9b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb340d0960 con 0x7feb34141910 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb39e9b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb340cec50 con 0x7feb14003970 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb32ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb3415b920 0x7feb340cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb32ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb3415b920 0x7feb340cc820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49796/0 (socket says 172.21.15.45:49796) 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb32ffd640 1 -- 172.21.15.45:0/2969345507 learned_addr learned my addr 172.21.15.45:0/2969345507 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb337fe640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb14003970 0x7feb340ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb33fff640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb34141910 0x7feb34141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb32ffd640 1 -- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb14003970 msgr2=0x7feb340ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb32ffd640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb14003970 0x7feb340ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb32ffd640 1 -- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb34141910 msgr2=0x7feb34141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb32ffd640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb34141910 0x7feb34141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb32ffd640 1 -- 172.21.15.45:0/2969345507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb340d0090 con 0x7feb3415b920 2023-11-28T20:55:09.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.021+0000 7feb33fff640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb34141910 0x7feb34141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:09.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.022+0000 7feb32ffd640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb3415b920 0x7feb340cc820 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00de80 tx=0x7feb1c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:09.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.022+0000 7feb2bfff640 1 -- 172.21.15.45:0/2969345507 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c004d60 con 0x7feb3415b920 2023-11-28T20:55:09.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.022+0000 7feb2bfff640 1 -- 172.21.15.45:0/2969345507 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c00be80 con 0x7feb3415b920 2023-11-28T20:55:09.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.022+0000 7feb2bfff640 1 -- 172.21.15.45:0/2969345507 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0072d0 con 0x7feb3415b920 2023-11-28T20:55:09.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.022+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb340ce970 con 0x7feb3415b920 2023-11-28T20:55:09.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.022+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb340d05f0 con 0x7feb3415b920 2023-11-28T20:55:09.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.023+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb340b5cc0 con 0x7feb3415b920 2023-11-28T20:55:09.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.024+0000 7feb2bfff640 1 -- 172.21.15.45:0/2969345507 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7feb1c007430 con 0x7feb3415b920 2023-11-28T20:55:09.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.026+0000 7feb2bfff640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7feb1806af60 0x7feb1806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.026+0000 7feb2bfff640 1 -- 172.21.15.45:0/2969345507 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7feb1c089690 con 0x7feb3415b920 2023-11-28T20:55:09.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.026+0000 7feb337fe640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7feb1806af60 0x7feb1806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.027+0000 7feb337fe640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7feb1806af60 0x7feb1806d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feb24036000 tx=0x7feb24030000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:09.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.030+0000 7feb2bfff640 1 -- 172.21.15.45:0/2969345507 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb1c054720 con 0x7feb3415b920 2023-11-28T20:55:09.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.232+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb340d0780 con 0x7feb3415b920 2023-11-28T20:55:09.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.236+0000 7feb2bfff640 1 -- 172.21.15.45:0/2969345507 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+13104 (secure 0 0 0) 0x7feb1c019090 con 0x7feb3415b920 2023-11-28T20:55:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:09.238 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":157,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:55:08.243012+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:55:09.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7feb1806af60 msgr2=0x7feb1806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7feb1806af60 0x7feb1806d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feb24036000 tx=0x7feb24030000 comp rx=0 tx=0).stop 2023-11-28T20:55:09.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb3415b920 msgr2=0x7feb340cc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb3415b920 0x7feb340cc820 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00de80 tx=0x7feb1c00b4a0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 shutdown_connections 2023-11-28T20:55:09.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb34141910 0x7feb34141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7feb1806af60 0x7feb1806d420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb3415b920 0x7feb340cc820 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 --2- 172.21.15.45:0/2969345507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb14003970 0x7feb340ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.238+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 >> 172.21.15.45:0/2969345507 conn(0x7feb3406e7c0 msgr2=0x7feb3406bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:09.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.239+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 shutdown_connections 2023-11-28T20:55:09.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.239+0000 7feb39e9b640 1 -- 172.21.15.45:0/2969345507 wait complete. 2023-11-28T20:55:09.262 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T20:55:09.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T20:55:09.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.491+0000 7f238359e640 1 Processor -- start 2023-11-28T20:55:09.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.491+0000 7f238359e640 1 -- start start 2023-11-28T20:55:09.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.493+0000 7f238359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2384066d90 0x7f238406b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.493+0000 7f238359e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f238406ba50 0x7f238415b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.493+0000 7f238359e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23841409c0 con 0x7f238413e170 2023-11-28T20:55:09.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.493+0000 7f238359e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238406d7f0 con 0x7f238406ba50 2023-11-28T20:55:09.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.493+0000 7f238359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238406bec0 con 0x7f2384066d90 2023-11-28T20:55:09.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.493+0000 7f238259c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f238413e170 0x7f238406d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47184/0 (socket says 172.21.15.45:47184) 2023-11-28T20:55:09.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.493+0000 7f238259c640 1 -- 172.21.15.45:0/3250999340 learned_addr learned my addr 172.21.15.45:0/3250999340 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:09.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.493+0000 7f2382d9d640 1 --2- 172.21.15.45:0/3250999340 >> v2:172.21.15.45:3300/0 conn(0x7f238406ba50 0x7f238415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.493+0000 7f2381d9b640 1 --2- 172.21.15.45:0/3250999340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2384066d90 0x7f238406b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.494+0000 7f238159a640 1 -- 172.21.15.45:0/3250999340 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434028597 0 0) 0x7f23841409c0 con 0x7f238413e170 2023-11-28T20:55:09.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.494+0000 7f238159a640 1 -- 172.21.15.45:0/3250999340 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f236c003520 con 0x7f238413e170 2023-11-28T20:55:09.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.494+0000 7f2382d9d640 1 -- 172.21.15.45:0/3250999340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2384066d90 msgr2=0x7f238406b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:09.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.494+0000 7f2382d9d640 1 --2- 172.21.15.45:0/3250999340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2384066d90 0x7f238406b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.494+0000 7f2382d9d640 1 -- 172.21.15.45:0/3250999340 >> v1:172.21.15.45:6789/0 conn(0x7f238413e170 legacy=0x7f238406d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.494+0000 7f2382d9d640 1 -- 172.21.15.45:0/3250999340 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2384140ca0 con 0x7f238406ba50 2023-11-28T20:55:09.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.495+0000 7f2382d9d640 1 --2- 172.21.15.45:0/3250999340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238406ba50 0x7f238415b540 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f237c0097a0 tx=0x7f237c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8148c8e49dd94fb5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:09.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.495+0000 7f238159a640 1 -- 172.21.15.45:0/3250999340 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c031780 con 0x7f238406ba50 2023-11-28T20:55:09.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.495+0000 7f238159a640 1 -- 172.21.15.45:0/3250999340 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c0318e0 con 0x7f238406ba50 2023-11-28T20:55:09.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.495+0000 7f238159a640 1 -- 172.21.15.45:0/3250999340 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c031c70 con 0x7f238406ba50 2023-11-28T20:55:09.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.496+0000 7f238359e640 1 -- 172.21.15.45:0/3250999340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238406ba50 msgr2=0x7f238415b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.496+0000 7f238359e640 1 --2- 172.21.15.45:0/3250999340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238406ba50 0x7f238415b540 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f237c0097a0 tx=0x7f237c02b820 comp rx=0 tx=0).stop 2023-11-28T20:55:09.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.496+0000 7f238359e640 1 -- 172.21.15.45:0/3250999340 shutdown_connections 2023-11-28T20:55:09.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.496+0000 7f238359e640 1 --2- 172.21.15.45:0/3250999340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238406ba50 0x7f238415b540 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.496+0000 7f238359e640 1 --2- 172.21.15.45:0/3250999340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2384066d90 0x7f238406b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.496+0000 7f238359e640 1 -- 172.21.15.45:0/3250999340 >> 172.21.15.45:0/3250999340 conn(0x7f2384071240 msgr2=0x7f2384071650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:09.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.496+0000 7f238359e640 1 -- 172.21.15.45:0/3250999340 shutdown_connections 2023-11-28T20:55:09.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.496+0000 7f238359e640 1 -- 172.21.15.45:0/3250999340 wait complete. 2023-11-28T20:55:09.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.498+0000 7f238359e640 1 Processor -- start 2023-11-28T20:55:09.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.498+0000 7f238359e640 1 -- start start 2023-11-28T20:55:09.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f238359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f238406ba50 0x7f2384154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f238359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238413e170 0x7f2384134200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f238359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384134740 0x7f23841f1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f238359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238415b6f0 con 0x7f238413e170 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f238359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23841570a0 con 0x7f2384134740 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f238359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2384157380 con 0x7f238406ba50 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2381d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238413e170 0x7f2384134200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2381d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238413e170 0x7f2384134200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49816/0 (socket says 172.21.15.45:49816) 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2381d9b640 1 -- 172.21.15.45:0/3102860111 learned_addr learned my addr 172.21.15.45:0/3102860111 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f238259c640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f238406ba50 0x7f2384154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2382d9d640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384134740 0x7f23841f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2381d9b640 1 -- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f238406ba50 msgr2=0x7f2384154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2381d9b640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f238406ba50 0x7f2384154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2381d9b640 1 -- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384134740 msgr2=0x7f23841f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2381d9b640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384134740 0x7f23841f1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2381d9b640 1 -- 172.21.15.45:0/3102860111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c009400 con 0x7f238413e170 2023-11-28T20:55:09.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.500+0000 7f2382d9d640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384134740 0x7f23841f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:09.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.501+0000 7f2381d9b640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238413e170 0x7f2384134200 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f2378007e40 tx=0x7f237800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:09.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.501+0000 7f2372ffd640 1 -- 172.21.15.45:0/3102860111 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2378014620 con 0x7f238413e170 2023-11-28T20:55:09.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.501+0000 7f2372ffd640 1 -- 172.21.15.45:0/3102860111 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237800ee90 con 0x7f238413e170 2023-11-28T20:55:09.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.501+0000 7f2372ffd640 1 -- 172.21.15.45:0/3102860111 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23780149c0 con 0x7f238413e170 2023-11-28T20:55:09.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.501+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2384157660 con 0x7f238413e170 2023-11-28T20:55:09.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.501+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2384157c20 con 0x7f238413e170 2023-11-28T20:55:09.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.502+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2384157db0 con 0x7f238413e170 2023-11-28T20:55:09.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.503+0000 7f2372ffd640 1 -- 172.21.15.45:0/3102860111 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2378014b20 con 0x7f238413e170 2023-11-28T20:55:09.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.504+0000 7f2372ffd640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f236406af60 0x7f236406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.504+0000 7f2372ffd640 1 -- 172.21.15.45:0/3102860111 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f2378089fc0 con 0x7f238413e170 2023-11-28T20:55:09.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.505+0000 7f238259c640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f236406af60 0x7f236406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.506+0000 7f238259c640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f236406af60 0x7f236406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f23740011e0 tx=0x7f2374009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:09.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.509+0000 7f2372ffd640 1 -- 172.21.15.45:0/3102860111 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23780550b0 con 0x7f238413e170 2023-11-28T20:55:09.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.708+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f238406fce0 con 0x7f238413e170 2023-11-28T20:55:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.709+0000 7f2372ffd640 1 -- 172.21.15.45:0/3102860111 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v157) v1 ==== 93+0+10 (secure 0 0 0) 0x7f23780586d0 con 0x7f238413e170 2023-11-28T20:55:09.711 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T20:55:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.712+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f236406af60 msgr2=0x7f236406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.712+0000 7f238359e640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f236406af60 0x7f236406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f23740011e0 tx=0x7f2374009040 comp rx=0 tx=0).stop 2023-11-28T20:55:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.712+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238413e170 msgr2=0x7f2384134200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.712+0000 7f238359e640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238413e170 0x7f2384134200 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f2378007e40 tx=0x7f237800e4b0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.712+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 shutdown_connections 2023-11-28T20:55:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.712+0000 7f238359e640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2384134740 0x7f23841f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.712+0000 7f238359e640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f236406af60 0x7f236406d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.713+0000 7f238359e640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238413e170 0x7f2384134200 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.713+0000 7f238359e640 1 --2- 172.21.15.45:0/3102860111 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f238406ba50 0x7f2384154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.713+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 >> 172.21.15.45:0/3102860111 conn(0x7f2384071240 msgr2=0x7f238406d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:09.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.713+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 shutdown_connections 2023-11-28T20:55:09.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.713+0000 7f238359e640 1 -- 172.21.15.45:0/3102860111 wait complete. 2023-11-28T20:55:09.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T20:55:09.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.963+0000 7f96b3fff640 1 Processor -- start 2023-11-28T20:55:09.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.963+0000 7f96b3fff640 1 -- start start 2023-11-28T20:55:09.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.965+0000 7f96b3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96b4146610 0x7f96b4070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.965+0000 7f96b3fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b40b4310 con 0x7f96b4072200 2023-11-28T20:55:09.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.965+0000 7f96b3fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b406d8d0 con 0x7f96b40b24a0 2023-11-28T20:55:09.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.965+0000 7f96b3fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b40725e0 con 0x7f96b4146610 2023-11-28T20:55:09.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.965+0000 7f96b37fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96b4146610 0x7f96b4070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.965+0000 7f96b2ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f96b4072200 0x7f96b40b1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47200/0 (socket says 172.21.15.45:47200) 2023-11-28T20:55:09.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.965+0000 7f96b2ffd640 1 -- 172.21.15.45:0/2208817341 learned_addr learned my addr 172.21.15.45:0/2208817341 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:09.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.966+0000 7f96b37fe640 1 -- 172.21.15.45:0/2208817341 >> v1:172.21.15.190:6789/0 conn(0x7f96b40b24a0 legacy=0x7f96b40703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.966+0000 7f96b37fe640 1 -- 172.21.15.45:0/2208817341 >> v1:172.21.15.45:6789/0 conn(0x7f96b4072200 legacy=0x7f96b40b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.966+0000 7f96b37fe640 1 -- 172.21.15.45:0/2208817341 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b406ca20 con 0x7f96b4146610 2023-11-28T20:55:09.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.967+0000 7f96b37fe640 1 --2- 172.21.15.45:0/2208817341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 0x7f96b4070ae0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009d90 tx=0x7f96ac02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9eb0fbca7036198 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:09.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b1ffb640 1 -- 172.21.15.45:0/2208817341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac02ccf0 con 0x7f96b4146610 2023-11-28T20:55:09.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b1ffb640 1 -- 172.21.15.45:0/2208817341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac02ce50 con 0x7f96b4146610 2023-11-28T20:55:09.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b1ffb640 1 -- 172.21.15.45:0/2208817341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac034660 con 0x7f96b4146610 2023-11-28T20:55:09.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b3fff640 1 -- 172.21.15.45:0/2208817341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 msgr2=0x7f96b4070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b3fff640 1 --2- 172.21.15.45:0/2208817341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 0x7f96b4070ae0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009d90 tx=0x7f96ac02bf40 comp rx=0 tx=0).stop 2023-11-28T20:55:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b3fff640 1 -- 172.21.15.45:0/2208817341 shutdown_connections 2023-11-28T20:55:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b3fff640 1 --2- 172.21.15.45:0/2208817341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 0x7f96b4070ae0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b3fff640 1 -- 172.21.15.45:0/2208817341 >> 172.21.15.45:0/2208817341 conn(0x7f96b406ab30 msgr2=0x7f96b40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:09.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b3fff640 1 -- 172.21.15.45:0/2208817341 shutdown_connections 2023-11-28T20:55:09.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.968+0000 7f96b3fff640 1 -- 172.21.15.45:0/2208817341 wait complete. 2023-11-28T20:55:09.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.970+0000 7f96b3fff640 1 Processor -- start 2023-11-28T20:55:09.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.970+0000 7f96b3fff640 1 -- start start 2023-11-28T20:55:09.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.971+0000 7f96b3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96b4072200 0x7f96b4140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.971+0000 7f96b3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b40b24a0 0x7f96b415c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.971+0000 7f96b3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 0x7f96b41602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.971+0000 7f96b3fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b413f0a0 con 0x7f96b4146610 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.971+0000 7f96b3fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b413f2f0 con 0x7f96b40b24a0 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.971+0000 7f96b3fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b413f5d0 con 0x7f96b4072200 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 0x7f96b41602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 0x7f96b41602c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49836/0 (socket says 172.21.15.45:49836) 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b37fe640 1 -- 172.21.15.45:0/2753507286 learned_addr learned my addr 172.21.15.45:0/2753507286 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b27fc640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b40b24a0 0x7f96b415c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b2ffd640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96b4072200 0x7f96b4140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b37fe640 1 -- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96b4072200 msgr2=0x7f96b4140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b37fe640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96b4072200 0x7f96b4140390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b37fe640 1 -- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b40b24a0 msgr2=0x7f96b415c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:09.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b37fe640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b40b24a0 0x7f96b415c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:09.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.972+0000 7f96b37fe640 1 -- 172.21.15.45:0/2753507286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac009a70 con 0x7f96b4146610 2023-11-28T20:55:09.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.973+0000 7f96b37fe640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 0x7f96b41602c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f96ac037040 tx=0x7f96ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.973+0000 7f969b7fe640 1 -- 172.21.15.45:0/2753507286 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac03d600 con 0x7f96b4146610 2023-11-28T20:55:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.973+0000 7f969b7fe640 1 -- 172.21.15.45:0/2753507286 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac03d760 con 0x7f96b4146610 2023-11-28T20:55:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.973+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b413f8b0 con 0x7f96b4146610 2023-11-28T20:55:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.973+0000 7f969b7fe640 1 -- 172.21.15.45:0/2753507286 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac03daf0 con 0x7f96b4146610 2023-11-28T20:55:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.973+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b413fe70 con 0x7f96b4146610 2023-11-28T20:55:09.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.974+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b4140090 con 0x7f96b4146610 2023-11-28T20:55:09.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.975+0000 7f969b7fe640 1 -- 172.21.15.45:0/2753507286 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f96ac03b040 con 0x7f96b4146610 2023-11-28T20:55:09.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.976+0000 7f969b7fe640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f96a006af90 0x7f96a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:09.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.976+0000 7f969b7fe640 1 -- 172.21.15.45:0/2753507286 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f96ac0abc60 con 0x7f96b4146610 2023-11-28T20:55:09.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.977+0000 7f96b2ffd640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f96a006af90 0x7f96a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:09.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.978+0000 7f96b2ffd640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f96a006af90 0x7f96a006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f96a4005880 tx=0x7f96a4009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:09.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:09.981+0000 7f969b7fe640 1 -- 172.21.15.45:0/2753507286 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96ac076de0 con 0x7f96b4146610 2023-11-28T20:55:10.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.176+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f96b409d0f0 con 0x7f96b4146610 2023-11-28T20:55:10.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.176+0000 7f969b7fe640 1 -- 172.21.15.45:0/2753507286 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v157) v1 ==== 84+0+65 (secure 0 0 0) 0x7f96ac04e090 con 0x7f96b4146610 2023-11-28T20:55:10.179 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:10.179 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:55:10.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:10.179 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T20:55:10.179 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T20:55:10.179 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:10.179 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:55:10.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f96a006af90 msgr2=0x7f96a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:10.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f96a006af90 0x7f96a006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f96a4005880 tx=0x7f96a4009040 comp rx=0 tx=0).stop 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 msgr2=0x7f96b41602c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 0x7f96b41602c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f96ac037040 tx=0x7f96ac004a40 comp rx=0 tx=0).stop 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 shutdown_connections 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f96a006af90 0x7f96a006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b4146610 0x7f96b41602c0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b40b24a0 0x7f96b415c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 --2- 172.21.15.45:0/2753507286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96b4072200 0x7f96b4140390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 >> 172.21.15.45:0/2753507286 conn(0x7f96b406ab30 msgr2=0x7f96b4144a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 shutdown_connections 2023-11-28T20:55:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.179+0000 7f96b3fff640 1 -- 172.21.15.45:0/2753507286 wait complete. 2023-11-28T20:55:10.201 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T20:55:10.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T20:55:10.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.434+0000 7fa7d6af3640 1 Processor -- start 2023-11-28T20:55:10.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.434+0000 7fa7d6af3640 1 -- start start 2023-11-28T20:55:10.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.436+0000 7fa7d6af3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d0160aa0 0x7fa7d00b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:10.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.436+0000 7fa7d6af3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa7d00d24f0 0x7fa7d00d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:10.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.436+0000 7fa7d6af3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d0160e80 con 0x7fa7d00ce0e0 2023-11-28T20:55:10.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.436+0000 7fa7d6af3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d00d1220 con 0x7fa7d00d24f0 2023-11-28T20:55:10.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.436+0000 7fa7d6af3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d00d2e70 con 0x7fa7d0160aa0 2023-11-28T20:55:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.436+0000 7fa7d5090640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa7d00ce0e0 0x7fa7d00ce570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47210/0 (socket says 172.21.15.45:47210) 2023-11-28T20:55:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.436+0000 7fa7d5090640 1 -- 172.21.15.45:0/126109116 learned_addr learned my addr 172.21.15.45:0/126109116 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:10.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.436+0000 7fa7d488f640 1 --2- 172.21.15.45:0/126109116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d0160aa0 0x7fa7d00b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:10.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.436+0000 7fa7cffff640 1 --2- 172.21.15.45:0/126109116 >> v2:172.21.15.190:3300/0 conn(0x7fa7d00d24f0 0x7fa7d00d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:10.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.437+0000 7fa7cf7fe640 1 -- 172.21.15.45:0/126109116 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369128970 0 0) 0x7fa7d0160e80 con 0x7fa7d00ce0e0 2023-11-28T20:55:10.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.437+0000 7fa7cf7fe640 1 -- 172.21.15.45:0/126109116 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7bc003520 con 0x7fa7d00ce0e0 2023-11-28T20:55:10.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.437+0000 7fa7d488f640 1 -- 172.21.15.45:0/126109116 >> v2:172.21.15.190:3300/0 conn(0x7fa7d00d24f0 msgr2=0x7fa7d00d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:10.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.437+0000 7fa7cf7fe640 1 -- 172.21.15.45:0/126109116 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2210790663 0 0) 0x7fa7bc003520 con 0x7fa7d00ce0e0 2023-11-28T20:55:10.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.437+0000 7fa7d488f640 1 --2- 172.21.15.45:0/126109116 >> v2:172.21.15.190:3300/0 conn(0x7fa7d00d24f0 0x7fa7d00d2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:10.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.437+0000 7fa7d488f640 1 -- 172.21.15.45:0/126109116 >> v1:172.21.15.45:6789/0 conn(0x7fa7d00ce0e0 legacy=0x7fa7d00ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:10.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.437+0000 7fa7d488f640 1 -- 172.21.15.45:0/126109116 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d00b6960 con 0x7fa7d0160aa0 2023-11-28T20:55:10.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.439+0000 7fa7d488f640 1 --2- 172.21.15.45:0/126109116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d0160aa0 0x7fa7d00b6420 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009d90 tx=0x7fa7c002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d7462aa12232b3f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:10.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.440+0000 7fa7cf7fe640 1 -- 172.21.15.45:0/126109116 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8001670 con 0x7fa7d0160aa0 2023-11-28T20:55:10.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.440+0000 7fa7cf7fe640 1 -- 172.21.15.45:0/126109116 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c002c930 con 0x7fa7d0160aa0 2023-11-28T20:55:10.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.440+0000 7fa7cf7fe640 1 -- 172.21.15.45:0/126109116 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c002cca0 con 0x7fa7d0160aa0 2023-11-28T20:55:10.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.440+0000 7fa7d6af3640 1 -- 172.21.15.45:0/126109116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d0160aa0 msgr2=0x7fa7d00b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:10.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.440+0000 7fa7d6af3640 1 --2- 172.21.15.45:0/126109116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d0160aa0 0x7fa7d00b6420 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009d90 tx=0x7fa7c002be00 comp rx=0 tx=0).stop 2023-11-28T20:55:10.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.440+0000 7fa7d6af3640 1 -- 172.21.15.45:0/126109116 shutdown_connections 2023-11-28T20:55:10.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.440+0000 7fa7d6af3640 1 --2- 172.21.15.45:0/126109116 >> v2:172.21.15.190:3300/0 conn(0x7fa7d00d24f0 0x7fa7d00d2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:10.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.440+0000 7fa7d6af3640 1 --2- 172.21.15.45:0/126109116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d0160aa0 0x7fa7d00b6420 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:10.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.440+0000 7fa7d6af3640 1 -- 172.21.15.45:0/126109116 >> 172.21.15.45:0/126109116 conn(0x7fa7d0070b20 msgr2=0x7fa7d0070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:10.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.441+0000 7fa7d6af3640 1 -- 172.21.15.45:0/126109116 shutdown_connections 2023-11-28T20:55:10.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.441+0000 7fa7d6af3640 1 -- 172.21.15.45:0/126109116 wait complete. 2023-11-28T20:55:10.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.443+0000 7fa7d6af3640 1 Processor -- start 2023-11-28T20:55:10.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.443+0000 7fa7d6af3640 1 -- start start 2023-11-28T20:55:10.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7d6af3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d00ce0e0 0x7fa7d014bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:10.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7d6af3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d00d24f0 0x7fa7d014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:10.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7d6af3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7d0160aa0 0x7fa7d00caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:10.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7d6af3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d00aa110 con 0x7fa7d00d24f0 2023-11-28T20:55:10.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7d6af3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d014c3c0 con 0x7fa7d0160aa0 2023-11-28T20:55:10.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7d6af3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d014f3a0 con 0x7fa7d00ce0e0 2023-11-28T20:55:10.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7cffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d00d24f0 0x7fa7d014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:10.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7cffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d00d24f0 0x7fa7d014ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49846/0 (socket says 172.21.15.45:49846) 2023-11-28T20:55:10.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7cffff640 1 -- 172.21.15.45:0/1536954432 learned_addr learned my addr 172.21.15.45:0/1536954432 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7d488f640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d00ce0e0 0x7fa7d014bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.445+0000 7fa7d5090640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7d0160aa0 0x7fa7d00caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7cffff640 1 -- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d00ce0e0 msgr2=0x7fa7d014bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7cffff640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d00ce0e0 0x7fa7d014bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7cffff640 1 -- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7d0160aa0 msgr2=0x7fa7d00caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7cffff640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7d0160aa0 0x7fa7d00caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7cffff640 1 -- 172.21.15.45:0/1536954432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c0009a70 con 0x7fa7d00d24f0 2023-11-28T20:55:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7d488f640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d00ce0e0 0x7fa7d014bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:55:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7cffff640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d00d24f0 0x7fa7d014ee60 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800e9e0 tx=0x7fa7b800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:10.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7cd7fa640 1 -- 172.21.15.45:0/1536954432 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8009670 con 0x7fa7d00d24f0 2023-11-28T20:55:10.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7cd7fa640 1 -- 172.21.15.45:0/1536954432 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b800c9d0 con 0x7fa7d00d24f0 2023-11-28T20:55:10.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7cd7fa640 1 -- 172.21.15.45:0/1536954432 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b800cd80 con 0x7fa7d00d24f0 2023-11-28T20:55:10.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.446+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d00a9b90 con 0x7fa7d00d24f0 2023-11-28T20:55:10.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.447+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d015d670 con 0x7fa7d00d24f0 2023-11-28T20:55:10.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.448+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d00733d0 con 0x7fa7d00d24f0 2023-11-28T20:55:10.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.449+0000 7fa7cd7fa640 1 -- 172.21.15.45:0/1536954432 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa7b8013470 con 0x7fa7d00d24f0 2023-11-28T20:55:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.450+0000 7fa7cd7fa640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa7b006af60 0x7fa7b006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.450+0000 7fa7cd7fa640 1 -- 172.21.15.45:0/1536954432 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fa7b8089b50 con 0x7fa7d00d24f0 2023-11-28T20:55:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.451+0000 7fa7d488f640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa7b006af60 0x7fa7b006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:10.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.452+0000 7fa7d488f640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa7b006af60 0x7fa7b006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0036000 tx=0x7fa7c0030000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:10.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.455+0000 7fa7cd7fa640 1 -- 172.21.15.45:0/1536954432 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7b8054cd0 con 0x7fa7d00d24f0 2023-11-28T20:55:10.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:10.650+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa7d00d1000 con 0x7fa7d00d24f0 2023-11-28T20:55:11.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.241+0000 7fa7cd7fa640 1 -- 172.21.15.45:0/1536954432 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v158) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa7b80170b0 con 0x7fa7d00d24f0 2023-11-28T20:55:11.244 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T20:55:11.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.245+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa7b006af60 msgr2=0x7fa7b006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:11.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.245+0000 7fa7d6af3640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa7b006af60 0x7fa7b006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0036000 tx=0x7fa7c0030000 comp rx=0 tx=0).stop 2023-11-28T20:55:11.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.245+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d00d24f0 msgr2=0x7fa7d014ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:11.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.245+0000 7fa7d6af3640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d00d24f0 0x7fa7d014ee60 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800e9e0 tx=0x7fa7b800eea0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.246+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 shutdown_connections 2023-11-28T20:55:11.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.246+0000 7fa7d6af3640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7d0160aa0 0x7fa7d00caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.246+0000 7fa7d6af3640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa7b006af60 0x7fa7b006d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.246+0000 7fa7d6af3640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d00d24f0 0x7fa7d014ee60 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.246+0000 7fa7d6af3640 1 --2- 172.21.15.45:0/1536954432 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7d00ce0e0 0x7fa7d014bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.246+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 >> 172.21.15.45:0/1536954432 conn(0x7fa7d0070b20 msgr2=0x7fa7d00b6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:11.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.246+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 shutdown_connections 2023-11-28T20:55:11.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.246+0000 7fa7d6af3640 1 -- 172.21.15.45:0/1536954432 wait complete. 2023-11-28T20:55:11.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T20:55:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.504+0000 7f135759e640 1 Processor -- start 2023-11-28T20:55:11.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.504+0000 7f135759e640 1 -- start start 2023-11-28T20:55:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.506+0000 7f135759e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f13580cc900 0x7f13580adb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.506+0000 7f135759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 0x7f1358070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.506+0000 7f135759e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13580acad0 con 0x7f13581607c0 2023-11-28T20:55:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.506+0000 7f135759e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13580ac8c0 con 0x7f13580cc900 2023-11-28T20:55:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.506+0000 7f135759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13580acdb0 con 0x7f13580ae0b0 2023-11-28T20:55:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.507+0000 7f135659c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f13581607c0 0x7f13580cd720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:43136/0 (socket says 172.21.15.45:43136) 2023-11-28T20:55:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.507+0000 7f135659c640 1 -- 172.21.15.45:0/2163009598 learned_addr learned my addr 172.21.15.45:0/2163009598 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.507+0000 7f1355d9b640 1 --2- 172.21.15.45:0/2163009598 >> v2:172.21.15.190:3300/0 conn(0x7f13580cc900 0x7f13580adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.507+0000 7f1356d9d640 1 --2- 172.21.15.45:0/2163009598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 0x7f1358070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.508+0000 7f135559a640 1 -- 172.21.15.45:0/2163009598 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212265529 0 0) 0x7f13580acad0 con 0x7f13581607c0 2023-11-28T20:55:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.508+0000 7f135559a640 1 -- 172.21.15.45:0/2163009598 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1340003520 con 0x7f13581607c0 2023-11-28T20:55:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.508+0000 7f1355d9b640 1 -- 172.21.15.45:0/2163009598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 msgr2=0x7f1358070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.508+0000 7f1355d9b640 1 --2- 172.21.15.45:0/2163009598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 0x7f1358070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.508+0000 7f1355d9b640 1 -- 172.21.15.45:0/2163009598 >> v1:172.21.15.190:6789/0 conn(0x7f13581607c0 legacy=0x7f13580cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.508+0000 7f1355d9b640 1 -- 172.21.15.45:0/2163009598 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13580ad090 con 0x7f13580cc900 2023-11-28T20:55:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.509+0000 7f1355d9b640 1 --2- 172.21.15.45:0/2163009598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13580cc900 0x7f13580adb70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f134c0097a0 tx=0x7f134c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28f8b6581d346ac3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135559a640 1 -- 172.21.15.45:0/2163009598 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c02e070 con 0x7f13580cc900 2023-11-28T20:55:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135559a640 1 -- 172.21.15.45:0/2163009598 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c0315a0 con 0x7f13580cc900 2023-11-28T20:55:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135559a640 1 -- 172.21.15.45:0/2163009598 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c031910 con 0x7f13580cc900 2023-11-28T20:55:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135759e640 1 -- 172.21.15.45:0/2163009598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13580cc900 msgr2=0x7f13580adb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135759e640 1 --2- 172.21.15.45:0/2163009598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13580cc900 0x7f13580adb70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f134c0097a0 tx=0x7f134c02b840 comp rx=0 tx=0).stop 2023-11-28T20:55:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135759e640 1 -- 172.21.15.45:0/2163009598 shutdown_connections 2023-11-28T20:55:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135759e640 1 --2- 172.21.15.45:0/2163009598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 0x7f1358070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135759e640 1 --2- 172.21.15.45:0/2163009598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13580cc900 0x7f13580adb70 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135759e640 1 -- 172.21.15.45:0/2163009598 >> 172.21.15.45:0/2163009598 conn(0x7f13580cec30 msgr2=0x7f13580cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135759e640 1 -- 172.21.15.45:0/2163009598 shutdown_connections 2023-11-28T20:55:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.510+0000 7f135759e640 1 -- 172.21.15.45:0/2163009598 wait complete. 2023-11-28T20:55:11.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.512+0000 7f135759e640 1 Processor -- start 2023-11-28T20:55:11.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.512+0000 7f135759e640 1 -- start start 2023-11-28T20:55:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.513+0000 7f135759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 0x7f13580c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.513+0000 7f135759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13580cc900 0x7f1358153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.513+0000 7f135759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13581607c0 0x7f135815ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.513+0000 7f135759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13580b1be0 con 0x7f13581607c0 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.513+0000 7f135759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13580b1ec0 con 0x7f13580cc900 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f135759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13580b21a0 con 0x7f13580ae0b0 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f1356d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13581607c0 0x7f135815ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f1356d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13581607c0 0x7f135815ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49854/0 (socket says 172.21.15.45:49854) 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f1356d9d640 1 -- 172.21.15.45:0/4168101119 learned_addr learned my addr 172.21.15.45:0/4168101119 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f1355d9b640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13580cc900 0x7f1358153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f135659c640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 0x7f13580c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f1356d9d640 1 -- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 msgr2=0x7f13580c2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f1356d9d640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 0x7f13580c2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f1356d9d640 1 -- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13580cc900 msgr2=0x7f1358153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:11.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f1356d9d640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13580cc900 0x7f1358153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:11.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.514+0000 7f1356d9d640 1 -- 172.21.15.45:0/4168101119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c009400 con 0x7f13581607c0 2023-11-28T20:55:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.515+0000 7f1356d9d640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13581607c0 0x7f135815ec90 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f135000efc0 tx=0x7f135000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.515+0000 7f1346ffd640 1 -- 172.21.15.45:0/4168101119 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350009ba0 con 0x7f13581607c0 2023-11-28T20:55:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.515+0000 7f1346ffd640 1 -- 172.21.15.45:0/4168101119 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135000f040 con 0x7f13581607c0 2023-11-28T20:55:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.515+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13580b2480 con 0x7f13581607c0 2023-11-28T20:55:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.515+0000 7f1346ffd640 1 -- 172.21.15.45:0/4168101119 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500146c0 con 0x7f13581607c0 2023-11-28T20:55:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.515+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13580b1870 con 0x7f13581607c0 2023-11-28T20:55:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.516+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13580b29a0 con 0x7f13581607c0 2023-11-28T20:55:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.518+0000 7f1346ffd640 1 -- 172.21.15.45:0/4168101119 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1350014820 con 0x7f13581607c0 2023-11-28T20:55:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.519+0000 7f1346ffd640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f133806af60 0x7f133806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.519+0000 7f1346ffd640 1 -- 172.21.15.45:0/4168101119 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f135008aa20 con 0x7f13581607c0 2023-11-28T20:55:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.520+0000 7f135659c640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f133806af60 0x7f133806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.521+0000 7f135659c640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f133806af60 0x7f133806d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f13480011e0 tx=0x7f1348009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:11.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.523+0000 7f1346ffd640 1 -- 172.21.15.45:0/4168101119 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1350055890 con 0x7f13581607c0 2023-11-28T20:55:11.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:11.719+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f13580b29a0 con 0x7f13581607c0 2023-11-28T20:55:12.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.251+0000 7f1346ffd640 1 -- 172.21.15.45:0/4168101119 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v159) v1 ==== 170+0+0 (secure 0 0 0) 0x7f13580b29a0 con 0x7f13581607c0 2023-11-28T20:55:12.253 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T20:55:12.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f133806af60 msgr2=0x7f133806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:12.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f133806af60 0x7f133806d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f13480011e0 tx=0x7f1348009040 comp rx=0 tx=0).stop 2023-11-28T20:55:12.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13581607c0 msgr2=0x7f135815ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:12.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13581607c0 0x7f135815ec90 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f135000efc0 tx=0x7f135000c490 comp rx=0 tx=0).stop 2023-11-28T20:55:12.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 shutdown_connections 2023-11-28T20:55:12.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f133806af60 0x7f133806d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:12.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13581607c0 0x7f135815ec90 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:12.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13580cc900 0x7f1358153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:12.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 --2- 172.21.15.45:0/4168101119 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13580ae0b0 0x7f13580c2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:12.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 >> 172.21.15.45:0/4168101119 conn(0x7f13580cec30 msgr2=0x7f13580af500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:12.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 shutdown_connections 2023-11-28T20:55:12.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.255+0000 7f135759e640 1 -- 172.21.15.45:0/4168101119 wait complete. 2023-11-28T20:55:12.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T20:55:12.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.485+0000 7f342c06b640 1 Processor -- start 2023-11-28T20:55:12.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.485+0000 7f342c06b640 1 -- start start 2023-11-28T20:55:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.486+0000 7f342c06b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3424160aa0 0x7f34240b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.486+0000 7f342c06b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3424160e80 con 0x7f34240ce7f0 2023-11-28T20:55:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.486+0000 7f342c06b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34240d1220 con 0x7f34240d24f0 2023-11-28T20:55:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.486+0000 7f342c06b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34240b6960 con 0x7f3424160aa0 2023-11-28T20:55:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.487+0000 7f3429e07640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3424160aa0 0x7f34240b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.487+0000 7f342a608640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f34240ce7f0 0x7f34241612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47220/0 (socket says 172.21.15.45:47220) 2023-11-28T20:55:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.487+0000 7f342a608640 1 -- 172.21.15.45:0/991862403 learned_addr learned my addr 172.21.15.45:0/991862403 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.487+0000 7f3429e07640 1 -- 172.21.15.45:0/991862403 >> v1:172.21.15.190:6789/0 conn(0x7f34240d24f0 legacy=0x7f34240ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.487+0000 7f3429e07640 1 -- 172.21.15.45:0/991862403 >> v1:172.21.15.45:6789/0 conn(0x7f34240ce7f0 legacy=0x7f34241612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.487+0000 7f3429e07640 1 -- 172.21.15.45:0/991862403 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34240d1470 con 0x7f3424160aa0 2023-11-28T20:55:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.487+0000 7f3428e05640 1 -- 172.21.15.45:0/991862403 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607017539 0 0) 0x7f3424160e80 con 0x7f34240ce7f0 2023-11-28T20:55:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.489+0000 7f3429e07640 1 --2- 172.21.15.45:0/991862403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 0x7f34240b6420 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f34200097a0 tx=0x7f342002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98aa7aa8de0e57fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.489+0000 7f3428e05640 1 -- 172.21.15.45:0/991862403 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420030610 con 0x7f3424160aa0 2023-11-28T20:55:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.489+0000 7f3428e05640 1 -- 172.21.15.45:0/991862403 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3420030770 con 0x7f3424160aa0 2023-11-28T20:55:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.489+0000 7f3428e05640 1 -- 172.21.15.45:0/991862403 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420030b00 con 0x7f3424160aa0 2023-11-28T20:55:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.490+0000 7f342c06b640 1 -- 172.21.15.45:0/991862403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 msgr2=0x7f34240b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.490+0000 7f342c06b640 1 --2- 172.21.15.45:0/991862403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 0x7f34240b6420 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f34200097a0 tx=0x7f342002b960 comp rx=0 tx=0).stop 2023-11-28T20:55:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.490+0000 7f342c06b640 1 -- 172.21.15.45:0/991862403 shutdown_connections 2023-11-28T20:55:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.490+0000 7f342c06b640 1 --2- 172.21.15.45:0/991862403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 0x7f34240b6420 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.490+0000 7f342c06b640 1 -- 172.21.15.45:0/991862403 >> 172.21.15.45:0/991862403 conn(0x7f3424070b20 msgr2=0x7f3424070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.490+0000 7f342c06b640 1 -- 172.21.15.45:0/991862403 shutdown_connections 2023-11-28T20:55:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.490+0000 7f342c06b640 1 -- 172.21.15.45:0/991862403 wait complete. 2023-11-28T20:55:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.492+0000 7f342c06b640 1 Processor -- start 2023-11-28T20:55:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.492+0000 7f342c06b640 1 -- start start 2023-11-28T20:55:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342c06b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34240ce7f0 0x7f342414c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342c06b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34240d24f0 0x7f342414b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342c06b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 0x7f342415f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342c06b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34240c8020 con 0x7f3424160aa0 2023-11-28T20:55:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342c06b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342414ef20 con 0x7f34240d24f0 2023-11-28T20:55:12.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342c06b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34240c76f0 con 0x7f34240ce7f0 2023-11-28T20:55:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342a608640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 0x7f342415f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342a608640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 0x7f342415f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49880/0 (socket says 172.21.15.45:49880) 2023-11-28T20:55:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342a608640 1 -- 172.21.15.45:0/650081179 learned_addr learned my addr 172.21.15.45:0/650081179 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f3429e07640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34240ce7f0 0x7f342414c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f3429606640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34240d24f0 0x7f342414b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342a608640 1 -- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34240ce7f0 msgr2=0x7f342414c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342a608640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34240ce7f0 0x7f342414c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342a608640 1 -- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34240d24f0 msgr2=0x7f342414b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342a608640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34240d24f0 0x7f342414b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.493+0000 7f342a608640 1 -- 172.21.15.45:0/650081179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3420009400 con 0x7f3424160aa0 2023-11-28T20:55:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.494+0000 7f3429606640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34240d24f0 0x7f342414b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.494+0000 7f342a608640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 0x7f342415f3c0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f341c001560 tx=0x7f341c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.494+0000 7f341a7fc640 1 -- 172.21.15.45:0/650081179 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c00a640 con 0x7f3424160aa0 2023-11-28T20:55:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.494+0000 7f341a7fc640 1 -- 172.21.15.45:0/650081179 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c00a7a0 con 0x7f3424160aa0 2023-11-28T20:55:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.494+0000 7f341a7fc640 1 -- 172.21.15.45:0/650081179 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c00ab30 con 0x7f3424160aa0 2023-11-28T20:55:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.494+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34240c79d0 con 0x7f3424160aa0 2023-11-28T20:55:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.494+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342414bc70 con 0x7f3424160aa0 2023-11-28T20:55:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.495+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34240c8190 con 0x7f3424160aa0 2023-11-28T20:55:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.498+0000 7f341a7fc640 1 -- 172.21.15.45:0/650081179 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f341c00ac90 con 0x7f3424160aa0 2023-11-28T20:55:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.499+0000 7f341a7fc640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f340806af60 0x7f340806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.499+0000 7f341a7fc640 1 -- 172.21.15.45:0/650081179 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f341c0891b0 con 0x7f3424160aa0 2023-11-28T20:55:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.499+0000 7f3429e07640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f340806af60 0x7f340806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:12.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.501+0000 7f3429e07640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f340806af60 0x7f340806d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3420036000 tx=0x7f342003b040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.503+0000 7f341a7fc640 1 -- 172.21.15.45:0/650081179 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f341c054130 con 0x7f3424160aa0 2023-11-28T20:55:12.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:12.735+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f34240733d0 con 0x7f3424160aa0 2023-11-28T20:55:13.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.276+0000 7f341a7fc640 1 -- 172.21.15.45:0/650081179 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v72) v1 ==== 174+0+0 (secure 0 0 0) 0x7f341c0180c0 con 0x7f3424160aa0 2023-11-28T20:55:13.279 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 19 and data pool 20 2023-11-28T20:55:13.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f340806af60 msgr2=0x7f340806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f340806af60 0x7f340806d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3420036000 tx=0x7f342003b040 comp rx=0 tx=0).stop 2023-11-28T20:55:13.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 msgr2=0x7f342415f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 0x7f342415f3c0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f341c001560 tx=0x7f341c00cc70 comp rx=0 tx=0).stop 2023-11-28T20:55:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 shutdown_connections 2023-11-28T20:55:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f340806af60 0x7f340806d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3424160aa0 0x7f342415f3c0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34240d24f0 0x7f342414b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 --2- 172.21.15.45:0/650081179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34240ce7f0 0x7f342414c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 >> 172.21.15.45:0/650081179 conn(0x7f3424070b20 msgr2=0x7f34240cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.280+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 shutdown_connections 2023-11-28T20:55:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.281+0000 7f342c06b640 1 -- 172.21.15.45:0/650081179 wait complete. 2023-11-28T20:55:13.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:55:13.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.511+0000 7efd76b1a640 1 Processor -- start 2023-11-28T20:55:13.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.511+0000 7efd76b1a640 1 -- start start 2023-11-28T20:55:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.513+0000 7efd76b1a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efd7015d550 0x7efd700cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.513+0000 7efd76b1a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 0x7efd700d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.513+0000 7efd76b1a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd700d0560 con 0x7efd700d1ab0 2023-11-28T20:55:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.513+0000 7efd76b1a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd700d2230 con 0x7efd7015d550 2023-11-28T20:55:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.513+0000 7efd76b1a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd700d2510 con 0x7efd700d0100 2023-11-28T20:55:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.513+0000 7efd75b18640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efd7015d550 0x7efd700cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.513+0000 7efd75b18640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7efd7015d550 0x7efd700cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49896/0 (socket says 172.21.15.45:49896) 2023-11-28T20:55:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.513+0000 7efd75b18640 1 -- 172.21.15.45:0/4109110935 learned_addr learned my addr 172.21.15.45:0/4109110935 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.514+0000 7efd75317640 1 --2- 172.21.15.45:0/4109110935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 0x7efd700d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.514+0000 7efd75b18640 1 -- 172.21.15.45:0/4109110935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 msgr2=0x7efd700d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.514+0000 7efd75b18640 1 --2- 172.21.15.45:0/4109110935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 0x7efd700d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.514+0000 7efd75b18640 1 -- 172.21.15.45:0/4109110935 >> v1:172.21.15.190:6789/0 conn(0x7efd700d1ab0 legacy=0x7efd70070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.514+0000 7efd75b18640 1 -- 172.21.15.45:0/4109110935 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd700d27f0 con 0x7efd7015d550 2023-11-28T20:55:13.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.515+0000 7efd75b18640 1 --2- 172.21.15.45:0/4109110935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 0x7efd700cfbc0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009870 tx=0x7efd6c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729ada255e65fd24 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.516+0000 7efd74b16640 1 -- 172.21.15.45:0/4109110935 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c031640 con 0x7efd7015d550 2023-11-28T20:55:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.516+0000 7efd74b16640 1 -- 172.21.15.45:0/4109110935 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c0317a0 con 0x7efd7015d550 2023-11-28T20:55:13.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.516+0000 7efd74b16640 1 -- 172.21.15.45:0/4109110935 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c031b30 con 0x7efd7015d550 2023-11-28T20:55:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.516+0000 7efd76b1a640 1 -- 172.21.15.45:0/4109110935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 msgr2=0x7efd700cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.516+0000 7efd76b1a640 1 --2- 172.21.15.45:0/4109110935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 0x7efd700cfbc0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009870 tx=0x7efd6c02b8f0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.516+0000 7efd76b1a640 1 -- 172.21.15.45:0/4109110935 shutdown_connections 2023-11-28T20:55:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.516+0000 7efd76b1a640 1 --2- 172.21.15.45:0/4109110935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 0x7efd700d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.516+0000 7efd76b1a640 1 --2- 172.21.15.45:0/4109110935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 0x7efd700cfbc0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.516+0000 7efd76b1a640 1 -- 172.21.15.45:0/4109110935 >> 172.21.15.45:0/4109110935 conn(0x7efd7006ac70 msgr2=0x7efd700b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:13.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.517+0000 7efd76b1a640 1 -- 172.21.15.45:0/4109110935 shutdown_connections 2023-11-28T20:55:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.517+0000 7efd76b1a640 1 -- 172.21.15.45:0/4109110935 wait complete. 2023-11-28T20:55:13.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.518+0000 7efd76b1a640 1 Processor -- start 2023-11-28T20:55:13.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.519+0000 7efd76b1a640 1 -- start start 2023-11-28T20:55:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd76b1a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 0x7efd7014e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd76b1a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd700d1ab0 0x7efd7014f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd76b1a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 0x7efd700aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd76b1a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd700c9ad0 con 0x7efd7015d550 2023-11-28T20:55:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd76b1a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd700a9cc0 con 0x7efd700d1ab0 2023-11-28T20:55:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd76b1a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd700a9fa0 con 0x7efd700d0100 2023-11-28T20:55:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd76319640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 0x7efd700aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd76319640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 0x7efd700aa7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49900/0 (socket says 172.21.15.45:49900) 2023-11-28T20:55:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd76319640 1 -- 172.21.15.45:0/3525587049 learned_addr learned my addr 172.21.15.45:0/3525587049 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd75b18640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 0x7efd7014e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.520+0000 7efd75317640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd700d1ab0 0x7efd7014f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd76319640 1 -- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 msgr2=0x7efd7014e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd76319640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 0x7efd7014e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd76319640 1 -- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd700d1ab0 msgr2=0x7efd7014f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd76319640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd700d1ab0 0x7efd7014f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd76319640 1 -- 172.21.15.45:0/3525587049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c0094d0 con 0x7efd7015d550 2023-11-28T20:55:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd76319640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 0x7efd700aa7a0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7efd6800c780 tx=0x7efd6800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd667fc640 1 -- 172.21.15.45:0/3525587049 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68004b60 con 0x7efd7015d550 2023-11-28T20:55:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd667fc640 1 -- 172.21.15.45:0/3525587049 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6800ce90 con 0x7efd7015d550 2023-11-28T20:55:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd700aa280 con 0x7efd7015d550 2023-11-28T20:55:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd667fc640 1 -- 172.21.15.45:0/3525587049 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6800a850 con 0x7efd7015d550 2023-11-28T20:55:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.521+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7014f6d0 con 0x7efd7015d550 2023-11-28T20:55:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.522+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd700a9b20 con 0x7efd7015d550 2023-11-28T20:55:13.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.524+0000 7efd667fc640 1 -- 172.21.15.45:0/3525587049 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7efd6800a9b0 con 0x7efd7015d550 2023-11-28T20:55:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.525+0000 7efd667fc640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7efd5406af60 0x7efd5406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.525+0000 7efd667fc640 1 -- 172.21.15.45:0/3525587049 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6057+0+0 (secure 0 0 0) 0x7efd68089ae0 con 0x7efd7015d550 2023-11-28T20:55:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.526+0000 7efd75b18640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7efd5406af60 0x7efd5406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.527+0000 7efd75b18640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7efd5406af60 0x7efd5406d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd6c036000 tx=0x7efd6c03c040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:13.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.530+0000 7efd667fc640 1 -- 172.21.15.45:0/3525587049 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd700a9b20 con 0x7efd7015d550 2023-11-28T20:55:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.718+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd700a9b20 con 0x7efd7015d550 2023-11-28T20:55:13.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.721+0000 7efd667fc640 1 -- 172.21.15.45:0/3525587049 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16777 (secure 0 0 0) 0x7efd700a9b20 con 0x7efd7015d550 2023-11-28T20:55:13.724 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:13.724 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":160,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:55:12.737824+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:55:10.712027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-11-28T20:55:11.779380+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:55:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7efd5406af60 msgr2=0x7efd5406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7efd5406af60 0x7efd5406d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd6c036000 tx=0x7efd6c03c040 comp rx=0 tx=0).stop 2023-11-28T20:55:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 msgr2=0x7efd700aa7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 0x7efd700aa7a0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7efd6800c780 tx=0x7efd6800cc40 comp rx=0 tx=0).stop 2023-11-28T20:55:13.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 shutdown_connections 2023-11-28T20:55:13.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7efd5406af60 0x7efd5406d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7015d550 0x7efd700aa7a0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd700d1ab0 0x7efd7014f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 --2- 172.21.15.45:0/3525587049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd700d0100 0x7efd7014e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 >> 172.21.15.45:0/3525587049 conn(0x7efd7006ac70 msgr2=0x7efd7015e510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:13.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 shutdown_connections 2023-11-28T20:55:13.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.724+0000 7efd76b1a640 1 -- 172.21.15.45:0/3525587049 wait complete. 2023-11-28T20:55:13.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:55:13.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.966+0000 7f1ba1ff1640 1 Processor -- start 2023-11-28T20:55:13.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.966+0000 7f1ba1ff1640 1 -- start start 2023-11-28T20:55:13.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.968+0000 7f1ba1ff1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c15d320 0x7f1b9c0ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.968+0000 7f1ba1ff1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1b9c0b1340 0x7f1b9c071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.968+0000 7f1ba1ff1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b9c0ce710 con 0x7f1b9c0add40 2023-11-28T20:55:13.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.968+0000 7f1ba1ff1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9c0ae5a0 con 0x7f1b9c0b1340 2023-11-28T20:55:13.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.968+0000 7f1ba1ff1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c160fe0 con 0x7f1b9c15d320 2023-11-28T20:55:13.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.968+0000 7f1b93fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1b9c0add40 0x7f1b9c0b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36992/0 (socket says 172.21.15.45:36992) 2023-11-28T20:55:13.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.968+0000 7f1b93fff640 1 -- 172.21.15.45:0/3181602641 learned_addr learned my addr 172.21.15.45:0/3181602641 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:13.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.968+0000 7f1b9b7fe640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c15d320 0x7f1b9c0ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.968+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894163211 0 0) 0x7f1b9c0ce710 con 0x7f1b9c0add40 2023-11-28T20:55:13.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b88003520 con 0x7f1b9c0add40 2023-11-28T20:55:13.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9bfff640 1 --2- 172.21.15.45:0/3181602641 >> v2:172.21.15.190:3300/0 conn(0x7f1b9c0b1340 0x7f1b9c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3514431080 0 0) 0x7f1b88003520 con 0x7f1b9c0add40 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c15d320 msgr2=0x7f1b9c0ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c15d320 0x7f1b9c0ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 >> v2:172.21.15.190:3300/0 conn(0x7f1b9c0b1340 msgr2=0x7f1b9c071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 --2- 172.21.15.45:0/3181602641 >> v2:172.21.15.190:3300/0 conn(0x7f1b9c0b1340 0x7f1b9c071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c0ae2c0 con 0x7f1b9c0add40 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f1b84001690 con 0x7f1b9c0add40 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9b7fe640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c15d320 0x7f1b9c0ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 >> v1:172.21.15.45:6789/0 conn(0x7f1b9c0add40 legacy=0x7f1b9c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b88003970 0x7f1b88003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c15d320 0x7f1b880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b88004d00 0x7f1b88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c160fe0 con 0x7f1b88003970 2023-11-28T20:55:13.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8800b980 con 0x7f1b88004d00 2023-11-28T20:55:13.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8800bb40 con 0x7f1b9c15d320 2023-11-28T20:55:13.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.969+0000 7f1b9b7fe640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b88003970 0x7f1b88003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.970+0000 7f1b9b7fe640 1 -- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c15d320 msgr2=0x7f1b880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:13.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.970+0000 7f1b9b7fe640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c15d320 0x7f1b880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.970+0000 7f1b9b7fe640 1 -- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b88004d00 msgr2=0x7f1b88005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:13.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.970+0000 7f1b9bfff640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b88004d00 0x7f1b88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.970+0000 7f1b9b7fe640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b88004d00 0x7f1b88005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.970+0000 7f1b9b7fe640 1 -- 172.21.15.45:0/3181602641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b88003520 con 0x7f1b88003970 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.971+0000 7f1b9b7fe640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b88003970 0x7f1b88003d90 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0098a0 tx=0x7f1b880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b146c2ae741654f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.971+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c004230 con 0x7f1b88003970 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.971+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8c004390 con 0x7f1b88003970 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.971+0000 7f1b9affd640 1 -- 172.21.15.45:0/3181602641 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c003830 con 0x7f1b88003970 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.972+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/3181602641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b88003970 msgr2=0x7f1b88003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.972+0000 7f1ba1ff1640 1 --2- 172.21.15.45:0/3181602641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b88003970 0x7f1b88003d90 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0098a0 tx=0x7f1b880057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.972+0000 7f1b9bfff640 1 -- 172.21.15.45:0/3181602641 reap_dead start 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.972+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/3181602641 shutdown_connections 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.972+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/3181602641 >> 172.21.15.45:0/3181602641 conn(0x7f1b9c06a500 msgr2=0x7f1b9c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:13.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.972+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/3181602641 shutdown_connections 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.972+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/3181602641 wait complete. 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.974+0000 7f1ba1ff1640 1 Processor -- start 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.974+0000 7f1ba1ff1640 1 -- start start 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1ba1ff1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b88003970 0x7f1b9c1e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1ba1ff1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0b4860 0x7f1b9c1e8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1ba1ff1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c1e8e50 0x7f1b9c1e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1ba1ff1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c1e7130 con 0x7f1b9c0b4860 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1ba1ff1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c15f720 con 0x7f1b88003970 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1ba1ff1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c1e9810 con 0x7f1b9c1e8e50 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b93fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0b4860 0x7f1b9c1e8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b93fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0b4860 0x7f1b9c1e8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41556/0 (socket says 172.21.15.45:41556) 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b93fff640 1 -- 172.21.15.45:0/677088281 learned_addr learned my addr 172.21.15.45:0/677088281 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b9b7fe640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b88003970 0x7f1b9c1e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b9bfff640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c1e8e50 0x7f1b9c1e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b93fff640 1 -- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c1e8e50 msgr2=0x7f1b9c1e92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b93fff640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c1e8e50 0x7f1b9c1e92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b93fff640 1 -- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b88003970 msgr2=0x7f1b9c1e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b93fff640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b88003970 0x7f1b9c1e6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.975+0000 7f1b93fff640 1 -- 172.21.15.45:0/677088281 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c15edf0 con 0x7f1b9c0b4860 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.976+0000 7f1b93fff640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0b4860 0x7f1b9c1e8910 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400c7b0 tx=0x7f1b8400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.976+0000 7f1b98ff9640 1 -- 172.21.15.45:0/677088281 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b840074e0 con 0x7f1b9c0b4860 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.976+0000 7f1b98ff9640 1 -- 172.21.15.45:0/677088281 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b84007640 con 0x7f1b9c0b4860 2023-11-28T20:55:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.976+0000 7f1b98ff9640 1 -- 172.21.15.45:0/677088281 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84004a20 con 0x7f1b9c0b4860 2023-11-28T20:55:13.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.976+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9c15f0d0 con 0x7f1b9c0b4860 2023-11-28T20:55:13.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.976+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9c15fbb0 con 0x7f1b9c0b4860 2023-11-28T20:55:13.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.977+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9c15fdd0 con 0x7f1b9c0b4860 2023-11-28T20:55:13.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.978+0000 7f1b98ff9640 1 -- 172.21.15.45:0/677088281 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1b84004b80 con 0x7f1b9c0b4860 2023-11-28T20:55:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.979+0000 7f1b98ff9640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1b7c06af90 0x7f1b7c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.979+0000 7f1b98ff9640 1 -- 172.21.15.45:0/677088281 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1b84088d60 con 0x7f1b9c0b4860 2023-11-28T20:55:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.980+0000 7f1b9b7fe640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1b7c06af90 0x7f1b7c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.981+0000 7f1b9b7fe640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1b7c06af90 0x7f1b7c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c035000 tx=0x7f1b8c03a040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:13.983+0000 7f1b98ff9640 1 -- 172.21.15.45:0/677088281 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b84053b00 con 0x7f1b9c0b4860 2023-11-28T20:55:14.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.170+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b9c0733d0 con 0x7f1b9c0b4860 2023-11-28T20:55:14.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.174+0000 7f1b98ff9640 1 -- 172.21.15.45:0/677088281 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16777 (secure 0 0 0) 0x7f1b84018020 con 0x7f1b9c0b4860 2023-11-28T20:55:14.176 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:14.176 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":160,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:55:12.737824+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:55:10.712027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-11-28T20:55:11.779380+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:55:14.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-11-28T20:55:14.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1b7c06af90 msgr2=0x7f1b7c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:14.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1b7c06af90 0x7f1b7c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c035000 tx=0x7f1b8c03a040 comp rx=0 tx=0).stop 2023-11-28T20:55:14.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0b4860 msgr2=0x7f1b9c1e8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:14.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0b4860 0x7f1b9c1e8910 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400c7b0 tx=0x7f1b8400cc70 comp rx=0 tx=0).stop 2023-11-28T20:55:14.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 shutdown_connections 2023-11-28T20:55:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b9c1e8e50 0x7f1b9c1e92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1b7c06af90 0x7f1b7c06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0b4860 0x7f1b9c1e8910 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 --2- 172.21.15.45:0/677088281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b88003970 0x7f1b9c1e6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.176+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 >> 172.21.15.45:0/677088281 conn(0x7f1b9c06a500 msgr2=0x7f1b9c0b06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.177+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 shutdown_connections 2023-11-28T20:55:14.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.177+0000 7f1ba1ff1640 1 -- 172.21.15.45:0/677088281 wait complete. 2023-11-28T20:55:14.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T20:55:14.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.426+0000 7f1edd6c3640 1 Processor -- start 2023-11-28T20:55:14.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.426+0000 7f1edd6c3640 1 -- start start 2023-11-28T20:55:14.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.428+0000 7f1edd6c3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1ed80cdca0 0x7f1ed806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:14.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.428+0000 7f1edd6c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ed806d410 con 0x7f1ed80b3040 2023-11-28T20:55:14.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.428+0000 7f1edd6c3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ed806d6f0 con 0x7f1ed815b270 2023-11-28T20:55:14.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.428+0000 7f1edd6c3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ed80ceaa0 con 0x7f1ed80cdca0 2023-11-28T20:55:14.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.428+0000 7f1ed7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1ed80b3040 0x7f1ed815f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36996/0 (socket says 172.21.15.45:36996) 2023-11-28T20:55:14.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.428+0000 7f1ed7fff640 1 -- 172.21.15.45:0/1845839645 learned_addr learned my addr 172.21.15.45:0/1845839645 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:14.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1edcec2640 1 --2- 172.21.15.45:0/1845839645 >> v2:172.21.15.190:3300/0 conn(0x7f1ed80cdca0 0x7f1ed806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:14.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179319080 0 0) 0x7f1ed806d410 con 0x7f1ed80b3040 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec4003520 con 0x7f1ed80b3040 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1100311404 0 0) 0x7f1ec4003520 con 0x7f1ed80b3040 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 >> v2:172.21.15.190:3300/0 conn(0x7f1ed80cdca0 msgr2=0x7f1ed806cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1ed77fe640 1 --2- 172.21.15.45:0/1845839645 >> v2:172.21.15.190:3300/0 conn(0x7f1ed80cdca0 0x7f1ed806cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 >> v1:172.21.15.190:6789/0 conn(0x7f1ed815b270 legacy=0x7f1ed80b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed806d9d0 con 0x7f1ed80b3040 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f1ec8001670 con 0x7f1ed80b3040 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.429+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 >> v1:172.21.15.45:6789/0 conn(0x7f1ed80b3040 legacy=0x7f1ed815f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ed77fe640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ec4003970 0x7f1ec4003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ed77fe640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ec4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:14.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ed77fe640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ec4005180 0x7f1ec400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec400c0a0 con 0x7f1ec40047c0 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec400c210 con 0x7f1ec4003970 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec400c3d0 con 0x7f1ec4005180 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ecffff640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ec4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ed7fff640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ec4003970 0x7f1ec4003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ecffff640 1 -- 172.21.15.45:0/1845839645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ec4005180 msgr2=0x7f1ec400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ecffff640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ec4005180 0x7f1ec400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ecffff640 1 -- 172.21.15.45:0/1845839645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ec4003970 msgr2=0x7f1ec4003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ecffff640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ec4003970 0x7f1ec4003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1edcec2640 1 -- 172.21.15.45:0/1845839645 reap_dead start 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ecffff640 1 -- 172.21.15.45:0/1845839645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec4003520 con 0x7f1ec40047c0 2023-11-28T20:55:14.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.430+0000 7f1ed7fff640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ec4003970 0x7f1ec4003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:14.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.431+0000 7f1ecffff640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ec4004c40 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0001830 tx=0x7f1ec40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae83d60de8d2df3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec002f050 con 0x7f1ec40047c0 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec0033610 con 0x7f1ec40047c0 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1ed77fe640 1 -- 172.21.15.45:0/1845839645 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00339c0 con 0x7f1ec40047c0 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1edd6c3640 1 -- 172.21.15.45:0/1845839645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 msgr2=0x7f1ec4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1edd6c3640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ec4004c40 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0001830 tx=0x7f1ec40057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1edd6c3640 1 -- 172.21.15.45:0/1845839645 shutdown_connections 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1edd6c3640 1 --2- 172.21.15.45:0/1845839645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ec4004c40 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1edd6c3640 1 -- 172.21.15.45:0/1845839645 >> 172.21.15.45:0/1845839645 conn(0x7f1ed80b1050 msgr2=0x7f1ed80cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1edd6c3640 1 -- 172.21.15.45:0/1845839645 shutdown_connections 2023-11-28T20:55:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.432+0000 7f1edd6c3640 1 -- 172.21.15.45:0/1845839645 wait complete. 2023-11-28T20:55:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.434+0000 7f1edd6c3640 1 Processor -- start 2023-11-28T20:55:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.434+0000 7f1edd6c3640 1 -- start start 2023-11-28T20:55:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.435+0000 7f1edd6c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ed81f4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.435+0000 7f1edd6c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ed815d5b0 0x7f1ed80ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.435+0000 7f1edd6c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed80aee90 0x7f1ed8153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.435+0000 7f1edd6c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed8156540 con 0x7f1ec40047c0 2023-11-28T20:55:14.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.435+0000 7f1edd6c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed80af310 con 0x7f1ed80aee90 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.435+0000 7f1edd6c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed81f5040 con 0x7f1ed815d5b0 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.435+0000 7f1ed7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ed81f4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ed7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ed81f4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41572/0 (socket says 172.21.15.45:41572) 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ed7fff640 1 -- 172.21.15.45:0/2366776614 learned_addr learned my addr 172.21.15.45:0/2366776614 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1edcec2640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed80aee90 0x7f1ed8153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ed7fff640 1 -- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ed815d5b0 msgr2=0x7f1ed80ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ecffff640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ed815d5b0 0x7f1ed80ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ed7fff640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ed815d5b0 0x7f1ed80ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ed7fff640 1 -- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed80aee90 msgr2=0x7f1ed8153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ed7fff640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed80aee90 0x7f1ed8153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ed7fff640 1 -- 172.21.15.45:0/2366776614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed815be70 con 0x7f1ec40047c0 2023-11-28T20:55:14.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ed7fff640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ed81f4b00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800d770 tx=0x7f1ec800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:14.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.436+0000 7f1ed57fa640 1 -- 172.21.15.45:0/2366776614 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80072d0 con 0x7f1ec40047c0 2023-11-28T20:55:14.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.437+0000 7f1ed57fa640 1 -- 172.21.15.45:0/2366776614 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec8007430 con 0x7f1ec40047c0 2023-11-28T20:55:14.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.437+0000 7f1ed57fa640 1 -- 172.21.15.45:0/2366776614 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80047e0 con 0x7f1ec40047c0 2023-11-28T20:55:14.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.437+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed81558c0 con 0x7f1ec40047c0 2023-11-28T20:55:14.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.437+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed8155e80 con 0x7f1ec40047c0 2023-11-28T20:55:14.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.438+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed809d0f0 con 0x7f1ec40047c0 2023-11-28T20:55:14.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.439+0000 7f1ed57fa640 1 -- 172.21.15.45:0/2366776614 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1ec800b6b0 con 0x7f1ec40047c0 2023-11-28T20:55:14.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.440+0000 7f1ed57fa640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1eb806af60 0x7f1eb806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:14.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.440+0000 7f1ed57fa640 1 -- 172.21.15.45:0/2366776614 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1ec8089110 con 0x7f1ec40047c0 2023-11-28T20:55:14.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.441+0000 7f1ecffff640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1eb806af60 0x7f1eb806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:14.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.442+0000 7f1ecffff640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1eb806af60 0x7f1eb806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0036000 tx=0x7f1ec0031000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:14.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.446+0000 7f1ed57fa640 1 -- 172.21.15.45:0/2366776614 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ec8053e50 con 0x7f1ec40047c0 2023-11-28T20:55:14.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:14.680+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1ed804e920 con 0x7f1ec40047c0 2023-11-28T20:55:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.449+0000 7f1ed57fa640 1 -- 172.21.15.45:0/2366776614 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v75) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1ec8057470 con 0x7f1ec40047c0 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1eb806af60 msgr2=0x7f1eb806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1eb806af60 0x7f1eb806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0036000 tx=0x7f1ec0031000 comp rx=0 tx=0).stop 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 msgr2=0x7f1ed81f4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ed81f4b00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800d770 tx=0x7f1ec800dc30 comp rx=0 tx=0).stop 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 shutdown_connections 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed80aee90 0x7f1ed8153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ed815d5b0 0x7f1ed80ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f1eb806af60 0x7f1eb806d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 --2- 172.21.15.45:0/2366776614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ec40047c0 0x7f1ed81f4b00 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 >> 172.21.15.45:0/2366776614 conn(0x7f1ed80b1050 msgr2=0x7f1ed80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 shutdown_connections 2023-11-28T20:55:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.454+0000 7f1edd6c3640 1 -- 172.21.15.45:0/2366776614 wait complete. 2023-11-28T20:55:15.478 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T20:55:15.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T20:55:15.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.694+0000 7f1815617640 1 Processor -- start 2023-11-28T20:55:15.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.695+0000 7f1815617640 1 -- start start 2023-11-28T20:55:15.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.696+0000 7f1815617640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1810099a40 0x7f181006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:15.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.696+0000 7f1815617640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100aff90 0x7f1810099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.696+0000 7f1815617640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18100d1fc0 con 0x7f18100cf260 2023-11-28T20:55:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.696+0000 7f1815617640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181006a910 con 0x7f1810099a40 2023-11-28T20:55:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.696+0000 7f1815617640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18100cb8a0 con 0x7f18100aff90 2023-11-28T20:55:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.697+0000 7f180e7fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100aff90 0x7f1810099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.697+0000 7f180e7fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100aff90 0x7f1810099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:53854/0 (socket says 172.21.15.45:53854) 2023-11-28T20:55:15.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.697+0000 7f180e7fc640 1 -- 172.21.15.45:0/1929812529 learned_addr learned my addr 172.21.15.45:0/1929812529 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.697+0000 7f180effd640 1 --2- 172.21.15.45:0/1929812529 >> v2:172.21.15.190:3300/0 conn(0x7f1810099a40 0x7f181006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.698+0000 7f180e7fc640 1 -- 172.21.15.45:0/1929812529 >> v2:172.21.15.190:3300/0 conn(0x7f1810099a40 msgr2=0x7f181006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.698+0000 7f180e7fc640 1 --2- 172.21.15.45:0/1929812529 >> v2:172.21.15.190:3300/0 conn(0x7f1810099a40 0x7f181006b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.698+0000 7f180e7fc640 1 -- 172.21.15.45:0/1929812529 >> v1:172.21.15.190:6789/0 conn(0x7f18100cf260 legacy=0x7f18100cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:15.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.698+0000 7f180e7fc640 1 -- 172.21.15.45:0/1929812529 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181009a360 con 0x7f18100aff90 2023-11-28T20:55:15.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.699+0000 7f180e7fc640 1 --2- 172.21.15.45:0/1929812529 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100aff90 0x7f1810099e20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f17f80098a0 tx=0x7f17f802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=315e60f990e86a8f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:15.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.700+0000 7f180dffb640 1 -- 172.21.15.45:0/1929812529 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8031640 con 0x7f18100aff90 2023-11-28T20:55:15.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.700+0000 7f180dffb640 1 -- 172.21.15.45:0/1929812529 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f80317a0 con 0x7f18100aff90 2023-11-28T20:55:15.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.700+0000 7f180dffb640 1 -- 172.21.15.45:0/1929812529 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8031b30 con 0x7f18100aff90 2023-11-28T20:55:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.700+0000 7f1815617640 1 -- 172.21.15.45:0/1929812529 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100aff90 msgr2=0x7f1810099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.700+0000 7f1815617640 1 --2- 172.21.15.45:0/1929812529 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100aff90 0x7f1810099e20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f17f80098a0 tx=0x7f17f802b8f0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.701+0000 7f1815617640 1 -- 172.21.15.45:0/1929812529 shutdown_connections 2023-11-28T20:55:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.701+0000 7f1815617640 1 --2- 172.21.15.45:0/1929812529 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100aff90 0x7f1810099e20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.701+0000 7f1815617640 1 --2- 172.21.15.45:0/1929812529 >> v2:172.21.15.190:3300/0 conn(0x7f1810099a40 0x7f181006b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.701+0000 7f1815617640 1 -- 172.21.15.45:0/1929812529 >> 172.21.15.45:0/1929812529 conn(0x7f18100d1020 msgr2=0x7f18100d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.701+0000 7f1815617640 1 -- 172.21.15.45:0/1929812529 shutdown_connections 2023-11-28T20:55:15.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.701+0000 7f1815617640 1 -- 172.21.15.45:0/1929812529 wait complete. 2023-11-28T20:55:15.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.702+0000 7f1815617640 1 Processor -- start 2023-11-28T20:55:15.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.703+0000 7f1815617640 1 -- start start 2023-11-28T20:55:15.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.703+0000 7f1815617640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1810099a40 0x7f18100c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:15.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.703+0000 7f1815617640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18100aff90 0x7f18101f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:15.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f1815617640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100cf260 0x7f181015f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:15.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f1815617640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18101f3e90 con 0x7f1810099a40 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f1815617640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181015fec0 con 0x7f18100aff90 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f1815617640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18100c9d60 con 0x7f18100cf260 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1810099a40 0x7f18100c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1810099a40 0x7f18100c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41574/0 (socket says 172.21.15.45:41574) 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180effd640 1 -- 172.21.15.45:0/2662549581 learned_addr learned my addr 172.21.15.45:0/2662549581 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180f7fe640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100cf260 0x7f181015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180e7fc640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18100aff90 0x7f18101f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180effd640 1 -- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100cf260 msgr2=0x7f181015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180effd640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100cf260 0x7f181015f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180effd640 1 -- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18100aff90 msgr2=0x7f18101f3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180effd640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18100aff90 0x7f18101f3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:15.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.704+0000 7f180effd640 1 -- 172.21.15.45:0/2662549581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f80094d0 con 0x7f1810099a40 2023-11-28T20:55:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.705+0000 7f180effd640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1810099a40 0x7f18100c9680 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f180000efc0 tx=0x7f180000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:15.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.705+0000 7f18077fe640 1 -- 172.21.15.45:0/2662549581 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800009b80 con 0x7f1810099a40 2023-11-28T20:55:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.705+0000 7f18077fe640 1 -- 172.21.15.45:0/2662549581 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180000f040 con 0x7f1810099a40 2023-11-28T20:55:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.705+0000 7f18077fe640 1 -- 172.21.15.45:0/2662549581 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18000146a0 con 0x7f1810099a40 2023-11-28T20:55:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.705+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18100ca040 con 0x7f1810099a40 2023-11-28T20:55:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.705+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18100c83d0 con 0x7f1810099a40 2023-11-28T20:55:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.706+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18100733d0 con 0x7f1810099a40 2023-11-28T20:55:15.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.708+0000 7f18077fe640 1 -- 172.21.15.45:0/2662549581 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1800014800 con 0x7f1810099a40 2023-11-28T20:55:15.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.708+0000 7f18077fe640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f17f006af60 0x7f17f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.709+0000 7f18077fe640 1 -- 172.21.15.45:0/2662549581 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f180008a5b0 con 0x7f1810099a40 2023-11-28T20:55:15.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.709+0000 7f180e7fc640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f17f006af60 0x7f17f006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:15.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.710+0000 7f180e7fc640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f17f006af60 0x7f17f006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f17f8036000 tx=0x7f17f803c040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.713+0000 7f18077fe640 1 -- 172.21.15.45:0/2662549581 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18000552e0 con 0x7f1810099a40 2023-11-28T20:55:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:15.947+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f18100ccce0 con 0x7f1810099a40 2023-11-28T20:55:16.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.460+0000 7f18077fe640 1 -- 172.21.15.45:0/2662549581 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1800058900 con 0x7f1810099a40 2023-11-28T20:55:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.463+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f17f006af60 msgr2=0x7f17f006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.463+0000 7f1815617640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f17f006af60 0x7f17f006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f17f8036000 tx=0x7f17f803c040 comp rx=0 tx=0).stop 2023-11-28T20:55:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.463+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1810099a40 msgr2=0x7f18100c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.463+0000 7f1815617640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1810099a40 0x7f18100c9680 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f180000efc0 tx=0x7f180000c490 comp rx=0 tx=0).stop 2023-11-28T20:55:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.464+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 shutdown_connections 2023-11-28T20:55:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.464+0000 7f1815617640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18100cf260 0x7f181015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.464+0000 7f1815617640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18100aff90 0x7f18101f3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.464+0000 7f1815617640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f17f006af60 0x7f17f006d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.464+0000 7f1815617640 1 --2- 172.21.15.45:0/2662549581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1810099a40 0x7f18100c9680 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.464+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 >> 172.21.15.45:0/2662549581 conn(0x7f18100d1020 msgr2=0x7f181006b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:16.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.464+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 shutdown_connections 2023-11-28T20:55:16.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.464+0000 7f1815617640 1 -- 172.21.15.45:0/2662549581 wait complete. 2023-11-28T20:55:16.487 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T20:55:16.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:55:16.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.708+0000 7f8a9f644640 1 Processor -- start 2023-11-28T20:55:16.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.709+0000 7f8a9f644640 1 -- start start 2023-11-28T20:55:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.710+0000 7f8a9f644640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8a980b4c40 0x7f8a980b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.710+0000 7f8a9f644640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a980711c0 con 0x7f8a98161960 2023-11-28T20:55:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.710+0000 7f8a9f644640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a980714a0 con 0x7f8a980b0c30 2023-11-28T20:55:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.710+0000 7f8a9f644640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a98071e40 con 0x7f8a980b4c40 2023-11-28T20:55:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.711+0000 7f8a9cbdf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8a980b4c40 0x7f8a980b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.711+0000 7f8a9cbdf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8a980b4c40 0x7f8a980b2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41590/0 (socket says 172.21.15.45:41590) 2023-11-28T20:55:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.711+0000 7f8a9cbdf640 1 -- 172.21.15.45:0/1031444879 learned_addr learned my addr 172.21.15.45:0/1031444879 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.711+0000 7f8a9cbdf640 1 -- 172.21.15.45:0/1031444879 >> v1:172.21.15.190:6789/0 conn(0x7f8a980b0c30 legacy=0x7f8a980b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.711+0000 7f8a9cbdf640 1 -- 172.21.15.45:0/1031444879 >> v1:172.21.15.45:6789/0 conn(0x7f8a98161960 legacy=0x7f8a98070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.711+0000 7f8a9cbdf640 1 -- 172.21.15.45:0/1031444879 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a98071780 con 0x7f8a980b4c40 2023-11-28T20:55:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.712+0000 7f8a9cbdf640 1 --2- 172.21.15.45:0/1031444879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980b4c40 0x7f8a980b2cd0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f8a88009870 tx=0x7f8a8802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbbbf51eb2e0ceb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.713+0000 7f8a8ffff640 1 -- 172.21.15.45:0/1031444879 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88031860 con 0x7f8a980b4c40 2023-11-28T20:55:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.713+0000 7f8a8ffff640 1 -- 172.21.15.45:0/1031444879 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a880319c0 con 0x7f8a980b4c40 2023-11-28T20:55:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.713+0000 7f8a8ffff640 1 -- 172.21.15.45:0/1031444879 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88031d50 con 0x7f8a980b4c40 2023-11-28T20:55:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.713+0000 7f8a9f644640 1 -- 172.21.15.45:0/1031444879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980b4c40 msgr2=0x7f8a980b2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.713+0000 7f8a9f644640 1 --2- 172.21.15.45:0/1031444879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980b4c40 0x7f8a980b2cd0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f8a88009870 tx=0x7f8a8802b900 comp rx=0 tx=0).stop 2023-11-28T20:55:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.714+0000 7f8a9f644640 1 -- 172.21.15.45:0/1031444879 shutdown_connections 2023-11-28T20:55:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.714+0000 7f8a9f644640 1 --2- 172.21.15.45:0/1031444879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a980b4c40 0x7f8a980b2cd0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.714+0000 7f8a9f644640 1 -- 172.21.15.45:0/1031444879 >> 172.21.15.45:0/1031444879 conn(0x7f8a9806a500 msgr2=0x7f8a9806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.714+0000 7f8a9f644640 1 -- 172.21.15.45:0/1031444879 shutdown_connections 2023-11-28T20:55:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.714+0000 7f8a9f644640 1 -- 172.21.15.45:0/1031444879 wait complete. 2023-11-28T20:55:16.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.716+0000 7f8a9f644640 1 Processor -- start 2023-11-28T20:55:16.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.716+0000 7f8a9f644640 1 -- start start 2023-11-28T20:55:16.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.717+0000 7f8a9f644640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980b0c30 0x7f8a981491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:16.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.717+0000 7f8a9f644640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980b4c40 0x7f8a9814bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:16.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.717+0000 7f8a9f644640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98161960 0x7f8a98148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:16.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.717+0000 7f8a9f644640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9814c1b0 con 0x7f8a98161960 2023-11-28T20:55:16.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.717+0000 7f8a9f644640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98148ec0 con 0x7f8a980b0c30 2023-11-28T20:55:16.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.717+0000 7f8a9f644640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a981497b0 con 0x7f8a980b4c40 2023-11-28T20:55:16.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.718+0000 7f8a9dbe1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98161960 0x7f8a98148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:16.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.718+0000 7f8a9dbe1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98161960 0x7f8a98148980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41602/0 (socket says 172.21.15.45:41602) 2023-11-28T20:55:16.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.718+0000 7f8a9dbe1640 1 -- 172.21.15.45:0/1217964343 learned_addr learned my addr 172.21.15.45:0/1217964343 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:16.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.718+0000 7f8a9d3e0640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980b0c30 0x7f8a981491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:16.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.718+0000 7f8a9cbdf640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980b4c40 0x7f8a9814bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a9dbe1640 1 -- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980b4c40 msgr2=0x7f8a9814bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a9dbe1640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980b4c40 0x7f8a9814bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a9dbe1640 1 -- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980b0c30 msgr2=0x7f8a981491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a9dbe1640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980b0c30 0x7f8a981491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a9dbe1640 1 -- 172.21.15.45:0/1217964343 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a880094d0 con 0x7f8a98161960 2023-11-28T20:55:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a9cbdf640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980b4c40 0x7f8a9814bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:55:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a9dbe1640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98161960 0x7f8a98148980 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000c410 tx=0x7f8a9000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a8dffb640 1 -- 172.21.15.45:0/1217964343 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90011790 con 0x7f8a98161960 2023-11-28T20:55:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a8dffb640 1 -- 172.21.15.45:0/1217964343 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a900118f0 con 0x7f8a98161960 2023-11-28T20:55:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.719+0000 7f8a8dffb640 1 -- 172.21.15.45:0/1217964343 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90011ce0 con 0x7f8a98161960 2023-11-28T20:55:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.720+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a98149a90 con 0x7f8a98161960 2023-11-28T20:55:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.720+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9814a050 con 0x7f8a98161960 2023-11-28T20:55:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.721+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a980cecf0 con 0x7f8a98161960 2023-11-28T20:55:16.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.722+0000 7f8a8dffb640 1 -- 172.21.15.45:0/1217964343 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8a9001c460 con 0x7f8a98161960 2023-11-28T20:55:16.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.723+0000 7f8a8dffb640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8a7c06af60 0x7f8a7c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:16.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.723+0000 7f8a8dffb640 1 -- 172.21.15.45:0/1217964343 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8a900895d0 con 0x7f8a98161960 2023-11-28T20:55:16.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.724+0000 7f8a9d3e0640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8a7c06af60 0x7f8a7c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:16.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.725+0000 7f8a9d3e0640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8a7c06af60 0x7f8a7c06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8a940012e0 tx=0x7f8a94009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:16.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.727+0000 7f8a8dffb640 1 -- 172.21.15.45:0/1217964343 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a90054370 con 0x7f8a98161960 2023-11-28T20:55:16.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.964+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a980733d0 con 0x7f8a98161960 2023-11-28T20:55:16.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.965+0000 7f8a8dffb640 1 -- 172.21.15.45:0/1217964343 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f8a90057990 con 0x7f8a98161960 2023-11-28T20:55:16.968 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:16.968 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7531,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1320359028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1320359028},{"type":"v1","addr":"172.21.15.45:6833","nonce":1320359028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/808323687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":808323687},{"type":"v1","addr":"172.21.15.190:6836","nonce":808323687}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3406194668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3406194668},{"type":"v1","addr":"172.21.15.190:6837","nonce":3406194668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:55:12.738156+0000","modified":"2023-11-28T20:55:15.949737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"c","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/637440036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":637440036},{"type":"v1","addr":"172.21.15.45:6835","nonce":637440036}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-28T20:55:16.968 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 76 2023-11-28T20:55:16.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8a7c06af60 msgr2=0x7f8a7c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:16.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8a7c06af60 0x7f8a7c06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8a940012e0 tx=0x7f8a94009040 comp rx=0 tx=0).stop 2023-11-28T20:55:16.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98161960 msgr2=0x7f8a98148980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:16.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98161960 0x7f8a98148980 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000c410 tx=0x7f8a9000c8d0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 shutdown_connections 2023-11-28T20:55:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f8a7c06af60 0x7f8a7c06d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98161960 0x7f8a98148980 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a980b4c40 0x7f8a9814bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 --2- 172.21.15.45:0/1217964343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980b0c30 0x7f8a981491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 >> 172.21.15.45:0/1217964343 conn(0x7f8a9806a500 msgr2=0x7f8a980cb190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 shutdown_connections 2023-11-28T20:55:16.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:16.968+0000 7f8a9f644640 1 -- 172.21.15.45:0/1217964343 wait complete. 2023-11-28T20:55:16.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T20:55:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.217+0000 7f04fddcf640 1 Processor -- start 2023-11-28T20:55:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.217+0000 7f04fddcf640 1 -- start start 2023-11-28T20:55:17.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.219+0000 7f04fddcf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04f8072200 0x7f04f80b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.219+0000 7f04fddcf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f80b4310 con 0x7f04f8146610 2023-11-28T20:55:17.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.219+0000 7f04fddcf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f806d8d0 con 0x7f04f80713e0 2023-11-28T20:55:17.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.219+0000 7f04fddcf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f80725e0 con 0x7f04f8072200 2023-11-28T20:55:17.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.219+0000 7f04f7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f04f8146610 0x7f04f8160e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37014/0 (socket says 172.21.15.45:37014) 2023-11-28T20:55:17.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.219+0000 7f04f7fff640 1 -- 172.21.15.45:0/3525983470 learned_addr learned my addr 172.21.15.45:0/3525983470 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:17.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04fcdcd640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04f8072200 0x7f04f80b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769122405 0 0) 0x7f04e4001110 con 0x7f04f8146610 2023-11-28T20:55:17.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e8003520 con 0x7f04f8146610 2023-11-28T20:55:17.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3847492894 0 0) 0x7f04e8003520 con 0x7f04f8146610 2023-11-28T20:55:17.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04f8072200 msgr2=0x7f04f80b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:17.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04f8072200 0x7f04f80b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 >> v1:172.21.15.190:6789/0 conn(0x7f04f80713e0 legacy=0x7f04f8161570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f8161c80 con 0x7f04f8146610 2023-11-28T20:55:17.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f04e4001800 con 0x7f04f8146610 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 >> v1:172.21.15.45:6789/0 conn(0x7f04f8146610 legacy=0x7f04f8160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.220+0000 7f04f77fe640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e80039a0 0x7f04e8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04f77fe640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04e80047c0 0x7f04e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04f77fe640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04f8072200 0x7f04e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e800ba30 con 0x7f04e80039a0 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e800bba0 con 0x7f04e80047c0 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e800bd60 con 0x7f04f8072200 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04fcdcd640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e80039a0 0x7f04e8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04f7fff640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04e80047c0 0x7f04e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04fd5ce640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04f8072200 0x7f04e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04fcdcd640 1 -- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04f8072200 msgr2=0x7f04e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:17.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04fcdcd640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04f8072200 0x7f04e8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04fcdcd640 1 -- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04e80047c0 msgr2=0x7f04e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:17.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04fcdcd640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04e80047c0 0x7f04e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04fcdcd640 1 -- 172.21.15.45:0/3525983470 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e8003520 con 0x7f04e80039a0 2023-11-28T20:55:17.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.221+0000 7f04fd5ce640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04f8072200 0x7f04e8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:17.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.222+0000 7f04fcdcd640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e80039a0 0x7f04e8003dc0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f04ec009a90 tx=0x7f04e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc5e100fd594cba server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:17.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.223+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec02c660 con 0x7f04e80039a0 2023-11-28T20:55:17.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.223+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec02c7c0 con 0x7f04e80039a0 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.223+0000 7f04f77fe640 1 -- 172.21.15.45:0/3525983470 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec02cb50 con 0x7f04e80039a0 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.223+0000 7f04fddcf640 1 -- 172.21.15.45:0/3525983470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e80039a0 msgr2=0x7f04e8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.223+0000 7f04fddcf640 1 --2- 172.21.15.45:0/3525983470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e80039a0 0x7f04e8003dc0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f04ec009a90 tx=0x7f04e80057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.223+0000 7f04fd5ce640 1 -- 172.21.15.45:0/3525983470 reap_dead start 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.223+0000 7f04fddcf640 1 -- 172.21.15.45:0/3525983470 shutdown_connections 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.223+0000 7f04fddcf640 1 -- 172.21.15.45:0/3525983470 >> 172.21.15.45:0/3525983470 conn(0x7f04f806ab30 msgr2=0x7f04f80b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.223+0000 7f04fddcf640 1 -- 172.21.15.45:0/3525983470 shutdown_connections 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.224+0000 7f04fddcf640 1 -- 172.21.15.45:0/3525983470 wait complete. 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.225+0000 7f04fddcf640 1 Processor -- start 2023-11-28T20:55:17.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.226+0000 7f04fddcf640 1 -- start start 2023-11-28T20:55:17.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.226+0000 7f04fddcf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04e80039a0 0x7f04f813dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04fddcf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04f815f050 0x7f04f815d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04fddcf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04f813e5b0 0x7f04f81e8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04fddcf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f813ee50 con 0x7f04f815f050 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04fddcf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f815dfb0 con 0x7f04f813e5b0 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04fddcf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f815e290 con 0x7f04e80039a0 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04f7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04f815f050 0x7f04f815d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04f7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04f815f050 0x7f04f815d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41606/0 (socket says 172.21.15.45:41606) 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04f7fff640 1 -- 172.21.15.45:0/4040039489 learned_addr learned my addr 172.21.15.45:0/4040039489 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04fcdcd640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04e80039a0 0x7f04f813dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04fd5ce640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04f813e5b0 0x7f04f81e8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04f7fff640 1 -- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04e80039a0 msgr2=0x7f04f813dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04f7fff640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04e80039a0 0x7f04f813dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04f7fff640 1 -- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04f813e5b0 msgr2=0x7f04f81e8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04f7fff640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04f813e5b0 0x7f04f81e8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.227+0000 7f04f7fff640 1 -- 172.21.15.45:0/4040039489 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f815e570 con 0x7f04f815f050 2023-11-28T20:55:17.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.228+0000 7f04f7fff640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04f815f050 0x7f04f815d8f0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f04e400e7b0 tx=0x7f04e400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:17.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.228+0000 7f04f57fa640 1 -- 172.21.15.45:0/4040039489 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e400c8d0 con 0x7f04f815f050 2023-11-28T20:55:17.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.228+0000 7f04f57fa640 1 -- 172.21.15.45:0/4040039489 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e400ca30 con 0x7f04f815f050 2023-11-28T20:55:17.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.228+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f815e850 con 0x7f04f815f050 2023-11-28T20:55:17.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.228+0000 7f04f57fa640 1 -- 172.21.15.45:0/4040039489 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e400cde0 con 0x7f04f815f050 2023-11-28T20:55:17.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.228+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f81e9070 con 0x7f04f815f050 2023-11-28T20:55:17.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.230+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f815ed70 con 0x7f04f815f050 2023-11-28T20:55:17.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.230+0000 7f04f57fa640 1 -- 172.21.15.45:0/4040039489 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f04e4013420 con 0x7f04f815f050 2023-11-28T20:55:17.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.231+0000 7f04f57fa640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f04dc06af90 0x7f04dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.231+0000 7f04f57fa640 1 -- 172.21.15.45:0/4040039489 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f04e4089460 con 0x7f04f815f050 2023-11-28T20:55:17.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.232+0000 7f04fcdcd640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f04dc06af90 0x7f04dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.233+0000 7f04fcdcd640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f04dc06af90 0x7f04dc06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04ec036000 tx=0x7f04ec030000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:17.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.236+0000 7f04f57fa640 1 -- 172.21.15.45:0/4040039489 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04f815ed70 con 0x7f04f815f050 2023-11-28T20:55:17.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.436+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04f815ed70 con 0x7f04f815f050 2023-11-28T20:55:17.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.439+0000 7f04f57fa640 1 -- 172.21.15.45:0/4040039489 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16764 (secure 0 0 0) 0x7f04f815ed70 con 0x7f04f815f050 2023-11-28T20:55:17.441 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:17.441 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":161,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T20:55:14.293150+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:55:10.712027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-11-28T20:55:11.779380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T20:55:17.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.441+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f04dc06af90 msgr2=0x7f04dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.441+0000 7f04fddcf640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f04dc06af90 0x7f04dc06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04ec036000 tx=0x7f04ec030000 comp rx=0 tx=0).stop 2023-11-28T20:55:17.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.441+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04f815f050 msgr2=0x7f04f815d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.441+0000 7f04fddcf640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04f815f050 0x7f04f815d8f0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f04e400e7b0 tx=0x7f04e400ec70 comp rx=0 tx=0).stop 2023-11-28T20:55:17.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.442+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 shutdown_connections 2023-11-28T20:55:17.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.442+0000 7f04fddcf640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04f813e5b0 0x7f04f81e8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.442+0000 7f04fddcf640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f04dc06af90 0x7f04dc06d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.442+0000 7f04fddcf640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04f815f050 0x7f04f815d8f0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.442+0000 7f04fddcf640 1 --2- 172.21.15.45:0/4040039489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04e80039a0 0x7f04f813dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.442+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 >> 172.21.15.45:0/4040039489 conn(0x7f04f806ab30 msgr2=0x7f04f8144b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:17.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.442+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 shutdown_connections 2023-11-28T20:55:17.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.442+0000 7f04fddcf640 1 -- 172.21.15.45:0/4040039489 wait complete. 2023-11-28T20:55:17.464 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T20:55:17.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:55:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.687+0000 7f9e74d2c640 1 Processor -- start 2023-11-28T20:55:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.688+0000 7f9e74d2c640 1 -- start start 2023-11-28T20:55:17.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.689+0000 7f9e74d2c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e700721e0 0x7f9e7006d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.689+0000 7f9e74d2c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7015e680 con 0x7f9e7006d6b0 2023-11-28T20:55:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.689+0000 7f9e74d2c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7015e960 con 0x7f9e700b3870 2023-11-28T20:55:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.689+0000 7f9e74d2c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7015a460 con 0x7f9e700721e0 2023-11-28T20:55:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.690+0000 7f9e6ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9e7006d6b0 0x7f9e700ac840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37020/0 (socket says 172.21.15.45:37020) 2023-11-28T20:55:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.690+0000 7f9e6ffff640 1 -- 172.21.15.45:0/2468255821 learned_addr learned my addr 172.21.15.45:0/2468255821 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.690+0000 7f9e6effd640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e700721e0 0x7f9e7006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.690+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269058363 0 0) 0x7f9e7015e680 con 0x7f9e7006d6b0 2023-11-28T20:55:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.690+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e58003520 con 0x7f9e7006d6b0 2023-11-28T20:55:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3358659018 0 0) 0x7f9e58003520 con 0x7f9e7006d6b0 2023-11-28T20:55:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e700721e0 msgr2=0x7f9e7006d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e700721e0 0x7f9e7006d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 >> v1:172.21.15.190:6789/0 conn(0x7f9e700b3870 legacy=0x7f9e7015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e70071700 con 0x7f9e7006d6b0 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f9e680019f0 con 0x7f9e7006d6b0 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 >> v1:172.21.15.45:6789/0 conn(0x7f9e7006d6b0 legacy=0x7f9e700ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e580039a0 0x7f9e58003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e700721e0 0x7f9e580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e58004d00 0x7f9e58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5800dc40 con 0x7f9e580039a0 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5800dd80 con 0x7f9e58004d00 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5800df40 con 0x7f9e700721e0 2023-11-28T20:55:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6f7fe640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e580039a0 0x7f9e58003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6effd640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e700721e0 0x7f9e580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.691+0000 7f9e6ffff640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e58004d00 0x7f9e58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.692+0000 7f9e6f7fe640 1 -- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e700721e0 msgr2=0x7f9e580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.692+0000 7f9e6f7fe640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e700721e0 0x7f9e580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.692+0000 7f9e6f7fe640 1 -- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e58004d00 msgr2=0x7f9e58005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.692+0000 7f9e6f7fe640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e58004d00 0x7f9e58005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.692+0000 7f9e6f7fe640 1 -- 172.21.15.45:0/2468255821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e58003520 con 0x7f9e580039a0 2023-11-28T20:55:17.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.692+0000 7f9e6ffff640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e58004d00 0x7f9e58005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.693+0000 7f9e6f7fe640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e580039a0 0x7f9e58003dc0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f9e60001720 tx=0x7f9e580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ab40a2a797911d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.693+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600052d0 con 0x7f9e580039a0 2023-11-28T20:55:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.693+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6002e050 con 0x7f9e580039a0 2023-11-28T20:55:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.693+0000 7f9e6e7fc640 1 -- 172.21.15.45:0/2468255821 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600035f0 con 0x7f9e580039a0 2023-11-28T20:55:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.694+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2468255821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e580039a0 msgr2=0x7f9e58003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.694+0000 7f9e74d2c640 1 --2- 172.21.15.45:0/2468255821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e580039a0 0x7f9e58003dc0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f9e60001720 tx=0x7f9e580057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.694+0000 7f9e6ffff640 1 -- 172.21.15.45:0/2468255821 reap_dead start 2023-11-28T20:55:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.694+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2468255821 shutdown_connections 2023-11-28T20:55:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.694+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2468255821 >> 172.21.15.45:0/2468255821 conn(0x7f9e7006ab30 msgr2=0x7f9e700b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.694+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2468255821 shutdown_connections 2023-11-28T20:55:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.694+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2468255821 wait complete. 2023-11-28T20:55:17.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.696+0000 7f9e74d2c640 1 Processor -- start 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.696+0000 7f9e74d2c640 1 -- start start 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.697+0000 7f9e74d2c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e580039a0 0x7f9e7009c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.697+0000 7f9e74d2c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e7014b520 0x7f9e7014cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.697+0000 7f9e74d2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7014abb0 0x7f9e7014d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.697+0000 7f9e74d2c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7014bc90 con 0x7f9e7014abb0 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.697+0000 7f9e74d2c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7014bf70 con 0x7f9e7014b520 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.697+0000 7f9e74d2c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7014c250 con 0x7f9e580039a0 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.697+0000 7f9e6ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7014abb0 0x7f9e7014d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.697+0000 7f9e6ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7014abb0 0x7f9e7014d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41624/0 (socket says 172.21.15.45:41624) 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.697+0000 7f9e6ffff640 1 -- 172.21.15.45:0/2687157021 learned_addr learned my addr 172.21.15.45:0/2687157021 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e6effd640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e7014b520 0x7f9e7014cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e6f7fe640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e580039a0 0x7f9e7009c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e6ffff640 1 -- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e580039a0 msgr2=0x7f9e7009c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e6ffff640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e580039a0 0x7f9e7009c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e6ffff640 1 -- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e7014b520 msgr2=0x7f9e7014cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e6ffff640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e7014b520 0x7f9e7014cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e6ffff640 1 -- 172.21.15.45:0/2687157021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7014c530 con 0x7f9e7014abb0 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e6effd640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e7014b520 0x7f9e7014cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e6ffff640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7014abb0 0x7f9e7014d5b0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f9e68000f80 tx=0x7f9e6800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e57fff640 1 -- 172.21.15.45:0/2687157021 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800de90 con 0x7f9e7014abb0 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e57fff640 1 -- 172.21.15.45:0/2687157021 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6800bb80 con 0x7f9e7014abb0 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e57fff640 1 -- 172.21.15.45:0/2687157021 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68007070 con 0x7f9e7014abb0 2023-11-28T20:55:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.698+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7014c810 con 0x7f9e7014abb0 2023-11-28T20:55:17.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.699+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7014d270 con 0x7f9e7014abb0 2023-11-28T20:55:17.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.700+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7009d0f0 con 0x7f9e7014abb0 2023-11-28T20:55:17.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.701+0000 7f9e57fff640 1 -- 172.21.15.45:0/2687157021 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9e680071d0 con 0x7f9e7014abb0 2023-11-28T20:55:17.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.702+0000 7f9e57fff640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9e5c06af60 0x7f9e5c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:17.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.702+0000 7f9e57fff640 1 -- 172.21.15.45:0/2687157021 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9e68089c80 con 0x7f9e7014abb0 2023-11-28T20:55:17.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.702+0000 7f9e6f7fe640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9e5c06af60 0x7f9e5c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:17.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.704+0000 7f9e6f7fe640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9e5c06af60 0x7f9e5c06d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9e60033000 tx=0x7f9e60009030 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:17.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.706+0000 7f9e57fff640 1 -- 172.21.15.45:0/2687157021 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e68054920 con 0x7f9e7014abb0 2023-11-28T20:55:17.904 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.901+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 --> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e7015a870 con 0x7f9e5c06af60 2023-11-28T20:55:17.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.931+0000 7f9e57fff640 1 -- 172.21.15.45:0/2687157021 <== mgr.7396 v2:172.21.15.190:6832/226305585 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287362 (secure 0 0 0) 0x7f9e600053d0 con 0x7f9e5c06af60 2023-11-28T20:55:17.937 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:17.939 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:55:17.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9e5c06af60 msgr2=0x7f9e5c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9e5c06af60 0x7f9e5c06d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9e60033000 tx=0x7f9e60009030 comp rx=0 tx=0).stop 2023-11-28T20:55:17.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7014abb0 msgr2=0x7f9e7014d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:17.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7014abb0 0x7f9e7014d5b0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f9e68000f80 tx=0x7f9e6800b4a0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 shutdown_connections 2023-11-28T20:55:17.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9e5c06af60 0x7f9e5c06d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7014abb0 0x7f9e7014d5b0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e7014b520 0x7f9e7014cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 --2- 172.21.15.45:0/2687157021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9e580039a0 0x7f9e7009c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:17.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.936+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 >> 172.21.15.45:0/2687157021 conn(0x7f9e7006ab30 msgr2=0x7f9e700cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:17.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.937+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 shutdown_connections 2023-11-28T20:55:17.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:17.937+0000 7f9e74d2c640 1 -- 172.21.15.45:0/2687157021 wait complete. 2023-11-28T20:55:17.958 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-28T20:55:16.631852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1381,"num_read_kb":2803,"num_write":455,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4794,"ondisk_log_size":4794,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":362100,"kb_used_data":16880,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754612620,"statfs":{"total":773094113280,"available":772723322880,"internally_reserved":0,"allocated":17285120,"data_stored":9322780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":353357360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.387451"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306215+0000","last_change":"2023-11-28T20:55:13.317488+0000","last_active":"2023-11-28T20:55:14.306215+0000","last_peered":"2023-11-28T20:55:14.306215+0000","last_clean":"2023-11-28T20:55:14.306215+0000","last_became_active":"2023-11-28T20:55:13.317204+0000","last_became_peered":"2023-11-28T20:55:13.317204+0000","last_unstale":"2023-11-28T20:55:14.306215+0000","last_undegraded":"2023-11-28T20:55:14.306215+0000","last_fullsized":"2023-11-28T20:55:14.306215+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:56.827248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.311139+0000","last_change":"2023-11-28T20:55:13.311128+0000","last_active":"2023-11-28T20:55:14.311139+0000","last_peered":"2023-11-28T20:55:14.311139+0000","last_clean":"2023-11-28T20:55:14.311139+0000","last_became_active":"2023-11-28T20:55:13.310854+0000","last_became_peered":"2023-11-28T20:55:13.310854+0000","last_unstale":"2023-11-28T20:55:14.311139+0000","last_undegraded":"2023-11-28T20:55:14.311139+0000","last_fullsized":"2023-11-28T20:55:14.311139+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:31.407809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402338+0000","last_change":"2023-11-28T20:55:13.325916+0000","last_active":"2023-11-28T20:55:15.402338+0000","last_peered":"2023-11-28T20:55:15.402338+0000","last_clean":"2023-11-28T20:55:15.402338+0000","last_became_active":"2023-11-28T20:55:13.325488+0000","last_became_peered":"2023-11-28T20:55:13.325488+0000","last_unstale":"2023-11-28T20:55:15.402338+0000","last_undegraded":"2023-11-28T20:55:15.402338+0000","last_fullsized":"2023-11-28T20:55:15.402338+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:06.888471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.310102+0000","last_change":"2023-11-28T20:55:13.309975+0000","last_active":"2023-11-28T20:55:14.310102+0000","last_peered":"2023-11-28T20:55:14.310102+0000","last_clean":"2023-11-28T20:55:14.310102+0000","last_became_active":"2023-11-28T20:55:13.309687+0000","last_became_peered":"2023-11-28T20:55:13.309687+0000","last_unstale":"2023-11-28T20:55:14.310102+0000","last_undegraded":"2023-11-28T20:55:14.310102+0000","last_fullsized":"2023-11-28T20:55:14.310102+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:15:16.330769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399936+0000","last_change":"2023-11-28T20:55:13.307271+0000","last_active":"2023-11-28T20:55:15.399936+0000","last_peered":"2023-11-28T20:55:15.399936+0000","last_clean":"2023-11-28T20:55:15.399936+0000","last_became_active":"2023-11-28T20:55:13.307003+0000","last_became_peered":"2023-11-28T20:55:13.307003+0000","last_unstale":"2023-11-28T20:55:15.399936+0000","last_undegraded":"2023-11-28T20:55:15.399936+0000","last_fullsized":"2023-11-28T20:55:15.399936+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:53.890830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899005+0000","last_change":"2023-11-28T20:55:13.327242+0000","last_active":"2023-11-28T20:55:14.899005+0000","last_peered":"2023-11-28T20:55:14.899005+0000","last_clean":"2023-11-28T20:55:14.899005+0000","last_became_active":"2023-11-28T20:55:13.326922+0000","last_became_peered":"2023-11-28T20:55:13.326922+0000","last_unstale":"2023-11-28T20:55:14.899005+0000","last_undegraded":"2023-11-28T20:55:14.899005+0000","last_fullsized":"2023-11-28T20:55:14.899005+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:27:14.578033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399809+0000","last_change":"2023-11-28T20:55:13.328896+0000","last_active":"2023-11-28T20:55:15.399809+0000","last_peered":"2023-11-28T20:55:15.399809+0000","last_clean":"2023-11-28T20:55:15.399809+0000","last_became_active":"2023-11-28T20:55:13.328474+0000","last_became_peered":"2023-11-28T20:55:13.328474+0000","last_unstale":"2023-11-28T20:55:15.399809+0000","last_undegraded":"2023-11-28T20:55:15.399809+0000","last_fullsized":"2023-11-28T20:55:15.399809+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:06:56.689456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309938+0000","last_change":"2023-11-28T20:55:13.305185+0000","last_active":"2023-11-28T20:55:14.309938+0000","last_peered":"2023-11-28T20:55:14.309938+0000","last_clean":"2023-11-28T20:55:14.309938+0000","last_became_active":"2023-11-28T20:55:13.302555+0000","last_became_peered":"2023-11-28T20:55:13.302555+0000","last_unstale":"2023-11-28T20:55:14.309938+0000","last_undegraded":"2023-11-28T20:55:14.309938+0000","last_fullsized":"2023-11-28T20:55:14.309938+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:09.114299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899535+0000","last_change":"2023-11-28T20:55:13.304329+0000","last_active":"2023-11-28T20:55:14.899535+0000","last_peered":"2023-11-28T20:55:14.899535+0000","last_clean":"2023-11-28T20:55:14.899535+0000","last_became_active":"2023-11-28T20:55:13.303518+0000","last_became_peered":"2023-11-28T20:55:13.303518+0000","last_unstale":"2023-11-28T20:55:14.899535+0000","last_undegraded":"2023-11-28T20:55:14.899535+0000","last_fullsized":"2023-11-28T20:55:14.899535+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:41:10.079803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401656+0000","last_change":"2023-11-28T20:55:13.303138+0000","last_active":"2023-11-28T20:55:15.401656+0000","last_peered":"2023-11-28T20:55:15.401656+0000","last_clean":"2023-11-28T20:55:15.401656+0000","last_became_active":"2023-11-28T20:55:13.302775+0000","last_became_peered":"2023-11-28T20:55:13.302775+0000","last_unstale":"2023-11-28T20:55:15.401656+0000","last_undegraded":"2023-11-28T20:55:15.401656+0000","last_fullsized":"2023-11-28T20:55:15.401656+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:57:52.187967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.310567+0000","last_change":"2023-11-28T20:55:13.325629+0000","last_active":"2023-11-28T20:55:14.310567+0000","last_peered":"2023-11-28T20:55:14.310567+0000","last_clean":"2023-11-28T20:55:14.310567+0000","last_became_active":"2023-11-28T20:55:13.325347+0000","last_became_peered":"2023-11-28T20:55:13.325347+0000","last_unstale":"2023-11-28T20:55:14.310567+0000","last_undegraded":"2023-11-28T20:55:14.310567+0000","last_fullsized":"2023-11-28T20:55:14.310567+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:59.330335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306216+0000","last_change":"2023-11-28T20:55:13.326510+0000","last_active":"2023-11-28T20:55:14.306216+0000","last_peered":"2023-11-28T20:55:14.306216+0000","last_clean":"2023-11-28T20:55:14.306216+0000","last_became_active":"2023-11-28T20:55:13.326218+0000","last_became_peered":"2023-11-28T20:55:13.326218+0000","last_unstale":"2023-11-28T20:55:14.306216+0000","last_undegraded":"2023-11-28T20:55:14.306216+0000","last_fullsized":"2023-11-28T20:55:14.306216+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:02:47.005001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899517+0000","last_change":"2023-11-28T20:55:13.327316+0000","last_active":"2023-11-28T20:55:14.899517+0000","last_peered":"2023-11-28T20:55:14.899517+0000","last_clean":"2023-11-28T20:55:14.899517+0000","last_became_active":"2023-11-28T20:55:13.326939+0000","last_became_peered":"2023-11-28T20:55:13.326939+0000","last_unstale":"2023-11-28T20:55:14.899517+0000","last_undegraded":"2023-11-28T20:55:14.899517+0000","last_fullsized":"2023-11-28T20:55:14.899517+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:56.861515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305657+0000","last_change":"2023-11-28T20:55:13.307180+0000","last_active":"2023-11-28T20:55:14.305657+0000","last_peered":"2023-11-28T20:55:14.305657+0000","last_clean":"2023-11-28T20:55:14.305657+0000","last_became_active":"2023-11-28T20:55:13.303740+0000","last_became_peered":"2023-11-28T20:55:13.303740+0000","last_unstale":"2023-11-28T20:55:14.305657+0000","last_undegraded":"2023-11-28T20:55:14.305657+0000","last_fullsized":"2023-11-28T20:55:14.305657+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:06:50.935582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305352+0000","last_change":"2023-11-28T20:55:13.314500+0000","last_active":"2023-11-28T20:55:14.305352+0000","last_peered":"2023-11-28T20:55:14.305352+0000","last_clean":"2023-11-28T20:55:14.305352+0000","last_became_active":"2023-11-28T20:55:13.314165+0000","last_became_peered":"2023-11-28T20:55:13.314165+0000","last_unstale":"2023-11-28T20:55:14.305352+0000","last_undegraded":"2023-11-28T20:55:14.305352+0000","last_fullsized":"2023-11-28T20:55:14.305352+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:24.662041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.898560+0000","last_change":"2023-11-28T20:55:13.306136+0000","last_active":"2023-11-28T20:55:14.898560+0000","last_peered":"2023-11-28T20:55:14.898560+0000","last_clean":"2023-11-28T20:55:14.898560+0000","last_became_active":"2023-11-28T20:55:13.305829+0000","last_became_peered":"2023-11-28T20:55:13.305829+0000","last_unstale":"2023-11-28T20:55:14.898560+0000","last_undegraded":"2023-11-28T20:55:14.898560+0000","last_fullsized":"2023-11-28T20:55:14.898560+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:24:34.938850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899726+0000","last_change":"2023-11-28T20:55:13.319510+0000","last_active":"2023-11-28T20:55:14.899726+0000","last_peered":"2023-11-28T20:55:14.899726+0000","last_clean":"2023-11-28T20:55:14.899726+0000","last_became_active":"2023-11-28T20:55:13.319197+0000","last_became_peered":"2023-11-28T20:55:13.319197+0000","last_unstale":"2023-11-28T20:55:14.899726+0000","last_undegraded":"2023-11-28T20:55:14.899726+0000","last_fullsized":"2023-11-28T20:55:14.899726+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:09.109449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400561+0000","last_change":"2023-11-28T20:55:13.315912+0000","last_active":"2023-11-28T20:55:15.400561+0000","last_peered":"2023-11-28T20:55:15.400561+0000","last_clean":"2023-11-28T20:55:15.400561+0000","last_became_active":"2023-11-28T20:55:13.315638+0000","last_became_peered":"2023-11-28T20:55:13.315638+0000","last_unstale":"2023-11-28T20:55:15.400561+0000","last_undegraded":"2023-11-28T20:55:15.400561+0000","last_fullsized":"2023-11-28T20:55:15.400561+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:16:48.298816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400696+0000","last_change":"2023-11-28T20:55:13.324886+0000","last_active":"2023-11-28T20:55:15.400696+0000","last_peered":"2023-11-28T20:55:15.400696+0000","last_clean":"2023-11-28T20:55:15.400696+0000","last_became_active":"2023-11-28T20:55:13.324270+0000","last_became_peered":"2023-11-28T20:55:13.324270+0000","last_unstale":"2023-11-28T20:55:15.400696+0000","last_undegraded":"2023-11-28T20:55:15.400696+0000","last_fullsized":"2023-11-28T20:55:15.400696+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:52.992994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400046+0000","last_change":"2023-11-28T20:55:13.316904+0000","last_active":"2023-11-28T20:55:15.400046+0000","last_peered":"2023-11-28T20:55:15.400046+0000","last_clean":"2023-11-28T20:55:15.400046+0000","last_became_active":"2023-11-28T20:55:13.311998+0000","last_became_peered":"2023-11-28T20:55:13.311998+0000","last_unstale":"2023-11-28T20:55:15.400046+0000","last_undegraded":"2023-11-28T20:55:15.400046+0000","last_fullsized":"2023-11-28T20:55:15.400046+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:52.992994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309617+0000","last_change":"2023-11-28T20:55:13.304523+0000","last_active":"2023-11-28T20:55:14.309617+0000","last_peered":"2023-11-28T20:55:14.309617+0000","last_clean":"2023-11-28T20:55:14.309617+0000","last_became_active":"2023-11-28T20:55:13.304240+0000","last_became_peered":"2023-11-28T20:55:13.304240+0000","last_unstale":"2023-11-28T20:55:14.309617+0000","last_undegraded":"2023-11-28T20:55:14.309617+0000","last_fullsized":"2023-11-28T20:55:14.309617+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:53:01.875044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305386+0000","last_change":"2023-11-28T20:55:13.314781+0000","last_active":"2023-11-28T20:55:14.305386+0000","last_peered":"2023-11-28T20:55:14.305386+0000","last_clean":"2023-11-28T20:55:14.305386+0000","last_became_active":"2023-11-28T20:55:13.314499+0000","last_became_peered":"2023-11-28T20:55:13.314499+0000","last_unstale":"2023-11-28T20:55:14.305386+0000","last_undegraded":"2023-11-28T20:55:14.305386+0000","last_fullsized":"2023-11-28T20:55:14.305386+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:06:02.047335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312061+0000","last_change":"2023-11-28T20:55:13.307697+0000","last_active":"2023-11-28T20:55:14.312061+0000","last_peered":"2023-11-28T20:55:14.312061+0000","last_clean":"2023-11-28T20:55:14.312061+0000","last_became_active":"2023-11-28T20:55:13.306852+0000","last_became_peered":"2023-11-28T20:55:13.306852+0000","last_unstale":"2023-11-28T20:55:14.312061+0000","last_undegraded":"2023-11-28T20:55:14.312061+0000","last_fullsized":"2023-11-28T20:55:14.312061+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:21.722474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.898769+0000","last_change":"2023-11-28T20:55:13.302163+0000","last_active":"2023-11-28T20:55:14.898769+0000","last_peered":"2023-11-28T20:55:14.898769+0000","last_clean":"2023-11-28T20:55:14.898769+0000","last_became_active":"2023-11-28T20:55:13.301857+0000","last_became_peered":"2023-11-28T20:55:13.301857+0000","last_unstale":"2023-11-28T20:55:14.898769+0000","last_undegraded":"2023-11-28T20:55:14.898769+0000","last_fullsized":"2023-11-28T20:55:14.898769+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:32:34.418435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402713+0000","last_change":"2023-11-28T20:55:13.324352+0000","last_active":"2023-11-28T20:55:15.402713+0000","last_peered":"2023-11-28T20:55:15.402713+0000","last_clean":"2023-11-28T20:55:15.402713+0000","last_became_active":"2023-11-28T20:55:13.320898+0000","last_became_peered":"2023-11-28T20:55:13.320898+0000","last_unstale":"2023-11-28T20:55:15.402713+0000","last_undegraded":"2023-11-28T20:55:15.402713+0000","last_fullsized":"2023-11-28T20:55:15.402713+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:05:00.230098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402425+0000","last_change":"2023-11-28T20:55:13.323315+0000","last_active":"2023-11-28T20:55:15.402425+0000","last_peered":"2023-11-28T20:55:15.402425+0000","last_clean":"2023-11-28T20:55:15.402425+0000","last_became_active":"2023-11-28T20:55:13.322990+0000","last_became_peered":"2023-11-28T20:55:13.322990+0000","last_unstale":"2023-11-28T20:55:15.402425+0000","last_undegraded":"2023-11-28T20:55:15.402425+0000","last_fullsized":"2023-11-28T20:55:15.402425+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:07:18.524250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400911+0000","last_change":"2023-11-28T20:55:13.322205+0000","last_active":"2023-11-28T20:55:15.400911+0000","last_peered":"2023-11-28T20:55:15.400911+0000","last_clean":"2023-11-28T20:55:15.400911+0000","last_became_active":"2023-11-28T20:55:13.321824+0000","last_became_peered":"2023-11-28T20:55:13.321824+0000","last_unstale":"2023-11-28T20:55:15.400911+0000","last_undegraded":"2023-11-28T20:55:15.400911+0000","last_fullsized":"2023-11-28T20:55:15.400911+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:39.018087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400904+0000","last_change":"2023-11-28T20:55:13.307051+0000","last_active":"2023-11-28T20:55:15.400904+0000","last_peered":"2023-11-28T20:55:15.400904+0000","last_clean":"2023-11-28T20:55:15.400904+0000","last_became_active":"2023-11-28T20:55:13.306732+0000","last_became_peered":"2023-11-28T20:55:13.306732+0000","last_unstale":"2023-11-28T20:55:15.400904+0000","last_undegraded":"2023-11-28T20:55:15.400904+0000","last_fullsized":"2023-11-28T20:55:15.400904+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:42:04.217860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402733+0000","last_change":"2023-11-28T20:55:13.323618+0000","last_active":"2023-11-28T20:55:15.402733+0000","last_peered":"2023-11-28T20:55:15.402733+0000","last_clean":"2023-11-28T20:55:15.402733+0000","last_became_active":"2023-11-28T20:55:13.321109+0000","last_became_peered":"2023-11-28T20:55:13.321109+0000","last_unstale":"2023-11-28T20:55:15.402733+0000","last_undegraded":"2023-11-28T20:55:15.402733+0000","last_fullsized":"2023-11-28T20:55:15.402733+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:46:15.351870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402018+0000","last_change":"2023-11-28T20:55:13.323476+0000","last_active":"2023-11-28T20:55:15.402018+0000","last_peered":"2023-11-28T20:55:15.402018+0000","last_clean":"2023-11-28T20:55:15.402018+0000","last_became_active":"2023-11-28T20:55:13.323147+0000","last_became_peered":"2023-11-28T20:55:13.323147+0000","last_unstale":"2023-11-28T20:55:15.402018+0000","last_undegraded":"2023-11-28T20:55:15.402018+0000","last_fullsized":"2023-11-28T20:55:15.402018+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:51:05.171617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.310431+0000","last_change":"2023-11-28T20:55:13.312033+0000","last_active":"2023-11-28T20:55:14.310431+0000","last_peered":"2023-11-28T20:55:14.310431+0000","last_clean":"2023-11-28T20:55:14.310431+0000","last_became_active":"2023-11-28T20:55:13.304258+0000","last_became_peered":"2023-11-28T20:55:13.304258+0000","last_unstale":"2023-11-28T20:55:14.310431+0000","last_undegraded":"2023-11-28T20:55:14.310431+0000","last_fullsized":"2023-11-28T20:55:14.310431+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:06.888471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399286+0000","last_change":"2023-11-28T20:55:13.298552+0000","last_active":"2023-11-28T20:55:15.399286+0000","last_peered":"2023-11-28T20:55:15.399286+0000","last_clean":"2023-11-28T20:55:15.399286+0000","last_became_active":"2023-11-28T20:55:13.298232+0000","last_became_peered":"2023-11-28T20:55:13.298232+0000","last_unstale":"2023-11-28T20:55:15.399286+0000","last_undegraded":"2023-11-28T20:55:15.399286+0000","last_fullsized":"2023-11-28T20:55:15.399286+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:23.123418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309977+0000","last_change":"2023-11-28T20:55:13.314298+0000","last_active":"2023-11-28T20:55:14.309977+0000","last_peered":"2023-11-28T20:55:14.309977+0000","last_clean":"2023-11-28T20:55:14.309977+0000","last_became_active":"2023-11-28T20:55:13.312522+0000","last_became_peered":"2023-11-28T20:55:13.312522+0000","last_unstale":"2023-11-28T20:55:14.309977+0000","last_undegraded":"2023-11-28T20:55:14.309977+0000","last_fullsized":"2023-11-28T20:55:14.309977+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:49:27.579201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"160'4","reported_seq":16,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402626+0000","last_change":"2023-11-28T20:55:12.289802+0000","last_active":"2023-11-28T20:55:15.402626+0000","last_peered":"2023-11-28T20:55:15.402626+0000","last_clean":"2023-11-28T20:55:15.402626+0000","last_became_active":"2023-11-28T20:55:12.287907+0000","last_became_peered":"2023-11-28T20:55:12.287907+0000","last_unstale":"2023-11-28T20:55:15.402626+0000","last_undegraded":"2023-11-28T20:55:15.402626+0000","last_fullsized":"2023-11-28T20:55:15.402626+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:01.537227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305413+0000","last_change":"2023-11-28T20:55:13.312677+0000","last_active":"2023-11-28T20:55:14.305413+0000","last_peered":"2023-11-28T20:55:14.305413+0000","last_clean":"2023-11-28T20:55:14.305413+0000","last_became_active":"2023-11-28T20:55:13.312348+0000","last_became_peered":"2023-11-28T20:55:13.312348+0000","last_unstale":"2023-11-28T20:55:14.305413+0000","last_undegraded":"2023-11-28T20:55:14.305413+0000","last_fullsized":"2023-11-28T20:55:14.305413+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:16.776349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401165+0000","last_change":"2023-11-28T20:55:12.288018+0000","last_active":"2023-11-28T20:55:15.401165+0000","last_peered":"2023-11-28T20:55:15.401165+0000","last_clean":"2023-11-28T20:55:15.401165+0000","last_became_active":"2023-11-28T20:55:12.287625+0000","last_became_peered":"2023-11-28T20:55:12.287625+0000","last_unstale":"2023-11-28T20:55:15.401165+0000","last_undegraded":"2023-11-28T20:55:15.401165+0000","last_fullsized":"2023-11-28T20:55:15.401165+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:03.522881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309634+0000","last_change":"2023-11-28T20:55:13.311668+0000","last_active":"2023-11-28T20:55:14.309634+0000","last_peered":"2023-11-28T20:55:14.309634+0000","last_clean":"2023-11-28T20:55:14.309634+0000","last_became_active":"2023-11-28T20:55:13.306380+0000","last_became_peered":"2023-11-28T20:55:13.306380+0000","last_unstale":"2023-11-28T20:55:14.309634+0000","last_undegraded":"2023-11-28T20:55:14.309634+0000","last_fullsized":"2023-11-28T20:55:14.309634+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:00.545871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400935+0000","last_change":"2023-11-28T20:55:12.293452+0000","last_active":"2023-11-28T20:55:15.400935+0000","last_peered":"2023-11-28T20:55:15.400935+0000","last_clean":"2023-11-28T20:55:15.400935+0000","last_became_active":"2023-11-28T20:55:12.293148+0000","last_became_peered":"2023-11-28T20:55:12.293148+0000","last_unstale":"2023-11-28T20:55:15.400935+0000","last_undegraded":"2023-11-28T20:55:15.400935+0000","last_fullsized":"2023-11-28T20:55:15.400935+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:41:42.258629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305744+0000","last_change":"2023-11-28T20:55:13.314199+0000","last_active":"2023-11-28T20:55:14.305744+0000","last_peered":"2023-11-28T20:55:14.305744+0000","last_clean":"2023-11-28T20:55:14.305744+0000","last_became_active":"2023-11-28T20:55:13.313854+0000","last_became_peered":"2023-11-28T20:55:13.313854+0000","last_unstale":"2023-11-28T20:55:14.305744+0000","last_undegraded":"2023-11-28T20:55:14.305744+0000","last_fullsized":"2023-11-28T20:55:14.305744+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:38:44.734529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400289+0000","last_change":"2023-11-28T20:55:12.295887+0000","last_active":"2023-11-28T20:55:15.400289+0000","last_peered":"2023-11-28T20:55:15.400289+0000","last_clean":"2023-11-28T20:55:15.400289+0000","last_became_active":"2023-11-28T20:55:12.295365+0000","last_became_peered":"2023-11-28T20:55:12.295365+0000","last_unstale":"2023-11-28T20:55:15.400289+0000","last_undegraded":"2023-11-28T20:55:15.400289+0000","last_fullsized":"2023-11-28T20:55:15.400289+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:40:32.099877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312153+0000","last_change":"2023-11-28T20:55:13.311985+0000","last_active":"2023-11-28T20:55:14.312153+0000","last_peered":"2023-11-28T20:55:14.312153+0000","last_clean":"2023-11-28T20:55:14.312153+0000","last_became_active":"2023-11-28T20:55:13.305170+0000","last_became_peered":"2023-11-28T20:55:13.305170+0000","last_unstale":"2023-11-28T20:55:14.312153+0000","last_undegraded":"2023-11-28T20:55:14.312153+0000","last_fullsized":"2023-11-28T20:55:14.312153+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:07:18.524250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.311676+0000","last_change":"2023-11-28T20:55:12.281457+0000","last_active":"2023-11-28T20:55:14.311676+0000","last_peered":"2023-11-28T20:55:14.311676+0000","last_clean":"2023-11-28T20:55:14.311676+0000","last_became_active":"2023-11-28T20:55:12.279839+0000","last_became_peered":"2023-11-28T20:55:12.279839+0000","last_unstale":"2023-11-28T20:55:14.311676+0000","last_undegraded":"2023-11-28T20:55:14.311676+0000","last_fullsized":"2023-11-28T20:55:14.311676+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:01.537227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312557+0000","last_change":"2023-11-28T20:55:13.302944+0000","last_active":"2023-11-28T20:55:14.312557+0000","last_peered":"2023-11-28T20:55:14.312557+0000","last_clean":"2023-11-28T20:55:14.312557+0000","last_became_active":"2023-11-28T20:55:13.302042+0000","last_became_peered":"2023-11-28T20:55:13.302042+0000","last_unstale":"2023-11-28T20:55:14.312557+0000","last_undegraded":"2023-11-28T20:55:14.312557+0000","last_fullsized":"2023-11-28T20:55:14.312557+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:05:00.230098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399476+0000","last_change":"2023-11-28T20:55:12.280094+0000","last_active":"2023-11-28T20:55:15.399476+0000","last_peered":"2023-11-28T20:55:15.399476+0000","last_clean":"2023-11-28T20:55:15.399476+0000","last_became_active":"2023-11-28T20:55:12.279687+0000","last_became_peered":"2023-11-28T20:55:12.279687+0000","last_unstale":"2023-11-28T20:55:15.399476+0000","last_undegraded":"2023-11-28T20:55:15.399476+0000","last_fullsized":"2023-11-28T20:55:15.399476+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:06:52.524129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400927+0000","last_change":"2023-11-28T20:55:13.315347+0000","last_active":"2023-11-28T20:55:15.400927+0000","last_peered":"2023-11-28T20:55:15.400927+0000","last_clean":"2023-11-28T20:55:15.400927+0000","last_became_active":"2023-11-28T20:55:13.311309+0000","last_became_peered":"2023-11-28T20:55:13.311309+0000","last_unstale":"2023-11-28T20:55:15.400927+0000","last_undegraded":"2023-11-28T20:55:15.400927+0000","last_fullsized":"2023-11-28T20:55:15.400927+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:26:38.540530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.898555+0000","last_change":"2023-11-28T20:55:12.284844+0000","last_active":"2023-11-28T20:55:14.898555+0000","last_peered":"2023-11-28T20:55:14.898555+0000","last_clean":"2023-11-28T20:55:14.898555+0000","last_became_active":"2023-11-28T20:55:12.284084+0000","last_became_peered":"2023-11-28T20:55:12.284084+0000","last_unstale":"2023-11-28T20:55:14.898555+0000","last_undegraded":"2023-11-28T20:55:14.898555+0000","last_fullsized":"2023-11-28T20:55:14.898555+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:17:46.357527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401081+0000","last_change":"2023-11-28T20:55:13.324981+0000","last_active":"2023-11-28T20:55:15.401081+0000","last_peered":"2023-11-28T20:55:15.401081+0000","last_clean":"2023-11-28T20:55:15.401081+0000","last_became_active":"2023-11-28T20:55:13.324660+0000","last_became_peered":"2023-11-28T20:55:13.324660+0000","last_unstale":"2023-11-28T20:55:15.401081+0000","last_undegraded":"2023-11-28T20:55:15.401081+0000","last_fullsized":"2023-11-28T20:55:15.401081+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:59:09.009449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401123+0000","last_change":"2023-11-28T20:55:12.290985+0000","last_active":"2023-11-28T20:55:15.401123+0000","last_peered":"2023-11-28T20:55:15.401123+0000","last_clean":"2023-11-28T20:55:15.401123+0000","last_became_active":"2023-11-28T20:55:12.290684+0000","last_became_peered":"2023-11-28T20:55:12.290684+0000","last_unstale":"2023-11-28T20:55:15.401123+0000","last_undegraded":"2023-11-28T20:55:15.401123+0000","last_fullsized":"2023-11-28T20:55:15.401123+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:37.725264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305287+0000","last_change":"2023-11-28T20:55:13.328921+0000","last_active":"2023-11-28T20:55:14.305287+0000","last_peered":"2023-11-28T20:55:14.305287+0000","last_clean":"2023-11-28T20:55:14.305287+0000","last_became_active":"2023-11-28T20:55:13.328010+0000","last_became_peered":"2023-11-28T20:55:13.328010+0000","last_unstale":"2023-11-28T20:55:14.305287+0000","last_undegraded":"2023-11-28T20:55:14.305287+0000","last_fullsized":"2023-11-28T20:55:14.305287+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:26:34.668104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312519+0000","last_change":"2023-11-28T20:55:12.290633+0000","last_active":"2023-11-28T20:55:14.312519+0000","last_peered":"2023-11-28T20:55:14.312519+0000","last_clean":"2023-11-28T20:55:14.312519+0000","last_became_active":"2023-11-28T20:55:12.290331+0000","last_became_peered":"2023-11-28T20:55:12.290331+0000","last_unstale":"2023-11-28T20:55:14.312519+0000","last_undegraded":"2023-11-28T20:55:14.312519+0000","last_fullsized":"2023-11-28T20:55:14.312519+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:47:32.663939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899627+0000","last_change":"2023-11-28T20:55:13.300749+0000","last_active":"2023-11-28T20:55:14.899627+0000","last_peered":"2023-11-28T20:55:14.899627+0000","last_clean":"2023-11-28T20:55:14.899627+0000","last_became_active":"2023-11-28T20:55:13.300453+0000","last_became_peered":"2023-11-28T20:55:13.300453+0000","last_unstale":"2023-11-28T20:55:14.899627+0000","last_undegraded":"2023-11-28T20:55:14.899627+0000","last_fullsized":"2023-11-28T20:55:14.899627+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:48:12.175159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306130+0000","last_change":"2023-11-28T20:55:12.276031+0000","last_active":"2023-11-28T20:55:14.306130+0000","last_peered":"2023-11-28T20:55:14.306130+0000","last_clean":"2023-11-28T20:55:14.306130+0000","last_became_active":"2023-11-28T20:55:12.275740+0000","last_became_peered":"2023-11-28T20:55:12.275740+0000","last_unstale":"2023-11-28T20:55:14.306130+0000","last_undegraded":"2023-11-28T20:55:14.306130+0000","last_fullsized":"2023-11-28T20:55:14.306130+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:43.026055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402345+0000","last_change":"2023-11-28T20:55:13.323283+0000","last_active":"2023-11-28T20:55:15.402345+0000","last_peered":"2023-11-28T20:55:15.402345+0000","last_clean":"2023-11-28T20:55:15.402345+0000","last_became_active":"2023-11-28T20:55:13.322914+0000","last_became_peered":"2023-11-28T20:55:13.322914+0000","last_unstale":"2023-11-28T20:55:15.402345+0000","last_undegraded":"2023-11-28T20:55:15.402345+0000","last_fullsized":"2023-11-28T20:55:15.402345+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:49:27.579201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"160'3","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401149+0000","last_change":"2023-11-28T20:55:12.294093+0000","last_active":"2023-11-28T20:55:15.401149+0000","last_peered":"2023-11-28T20:55:15.401149+0000","last_clean":"2023-11-28T20:55:15.401149+0000","last_became_active":"2023-11-28T20:55:12.293735+0000","last_became_peered":"2023-11-28T20:55:12.293735+0000","last_unstale":"2023-11-28T20:55:15.401149+0000","last_undegraded":"2023-11-28T20:55:15.401149+0000","last_fullsized":"2023-11-28T20:55:15.401149+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:51.235329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402133+0000","last_change":"2023-11-28T20:55:13.324237+0000","last_active":"2023-11-28T20:55:15.402133+0000","last_peered":"2023-11-28T20:55:15.402133+0000","last_clean":"2023-11-28T20:55:15.402133+0000","last_became_active":"2023-11-28T20:55:13.323963+0000","last_became_peered":"2023-11-28T20:55:13.323963+0000","last_unstale":"2023-11-28T20:55:15.402133+0000","last_undegraded":"2023-11-28T20:55:15.402133+0000","last_fullsized":"2023-11-28T20:55:15.402133+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:09.114299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"160'6","reported_seq":18,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400257+0000","last_change":"2023-11-28T20:55:12.293276+0000","last_active":"2023-11-28T20:55:15.400257+0000","last_peered":"2023-11-28T20:55:15.400257+0000","last_clean":"2023-11-28T20:55:15.400257+0000","last_became_active":"2023-11-28T20:55:12.291148+0000","last_became_peered":"2023-11-28T20:55:12.291148+0000","last_unstale":"2023-11-28T20:55:15.400257+0000","last_undegraded":"2023-11-28T20:55:15.400257+0000","last_fullsized":"2023-11-28T20:55:15.400257+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:06.359290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401256+0000","last_change":"2023-11-28T20:55:13.305345+0000","last_active":"2023-11-28T20:55:15.401256+0000","last_peered":"2023-11-28T20:55:15.401256+0000","last_clean":"2023-11-28T20:55:15.401256+0000","last_became_active":"2023-11-28T20:55:13.304977+0000","last_became_peered":"2023-11-28T20:55:13.304977+0000","last_unstale":"2023-11-28T20:55:15.401256+0000","last_undegraded":"2023-11-28T20:55:15.401256+0000","last_fullsized":"2023-11-28T20:55:15.401256+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:48.664836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305705+0000","last_change":"2023-11-28T20:55:12.277643+0000","last_active":"2023-11-28T20:55:14.305705+0000","last_peered":"2023-11-28T20:55:14.305705+0000","last_clean":"2023-11-28T20:55:14.305705+0000","last_became_active":"2023-11-28T20:55:12.277095+0000","last_became_peered":"2023-11-28T20:55:12.277095+0000","last_unstale":"2023-11-28T20:55:14.305705+0000","last_undegraded":"2023-11-28T20:55:14.305705+0000","last_fullsized":"2023-11-28T20:55:14.305705+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:21.830506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401532+0000","last_change":"2023-11-28T20:55:13.330971+0000","last_active":"2023-11-28T20:55:15.401532+0000","last_peered":"2023-11-28T20:55:15.401532+0000","last_clean":"2023-11-28T20:55:15.401532+0000","last_became_active":"2023-11-28T20:55:13.329615+0000","last_became_peered":"2023-11-28T20:55:13.329615+0000","last_unstale":"2023-11-28T20:55:15.401532+0000","last_undegraded":"2023-11-28T20:55:15.401532+0000","last_fullsized":"2023-11-28T20:55:15.401532+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:26:38.540530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401503+0000","last_change":"2023-11-28T20:55:12.290913+0000","last_active":"2023-11-28T20:55:15.401503+0000","last_peered":"2023-11-28T20:55:15.401503+0000","last_clean":"2023-11-28T20:55:15.401503+0000","last_became_active":"2023-11-28T20:55:12.290566+0000","last_became_peered":"2023-11-28T20:55:12.290566+0000","last_unstale":"2023-11-28T20:55:15.401503+0000","last_undegraded":"2023-11-28T20:55:15.401503+0000","last_fullsized":"2023-11-28T20:55:15.401503+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:53:36.404925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401633+0000","last_change":"2023-11-28T20:55:13.315988+0000","last_active":"2023-11-28T20:55:15.401633+0000","last_peered":"2023-11-28T20:55:15.401633+0000","last_clean":"2023-11-28T20:55:15.401633+0000","last_became_active":"2023-11-28T20:55:13.311743+0000","last_became_peered":"2023-11-28T20:55:13.311743+0000","last_unstale":"2023-11-28T20:55:15.401633+0000","last_undegraded":"2023-11-28T20:55:15.401633+0000","last_fullsized":"2023-11-28T20:55:15.401633+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:18:29.065010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"160'8","reported_seq":20,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899551+0000","last_change":"2023-11-28T20:55:12.291018+0000","last_active":"2023-11-28T20:55:14.899551+0000","last_peered":"2023-11-28T20:55:14.899551+0000","last_clean":"2023-11-28T20:55:14.899551+0000","last_became_active":"2023-11-28T20:55:12.290419+0000","last_became_peered":"2023-11-28T20:55:12.290419+0000","last_unstale":"2023-11-28T20:55:14.899551+0000","last_undegraded":"2023-11-28T20:55:14.899551+0000","last_fullsized":"2023-11-28T20:55:14.899551+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:12.788384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400905+0000","last_change":"2023-11-28T20:55:13.325708+0000","last_active":"2023-11-28T20:55:15.400905+0000","last_peered":"2023-11-28T20:55:15.400905+0000","last_clean":"2023-11-28T20:55:15.400905+0000","last_became_active":"2023-11-28T20:55:13.325417+0000","last_became_peered":"2023-11-28T20:55:13.325417+0000","last_unstale":"2023-11-28T20:55:15.400905+0000","last_undegraded":"2023-11-28T20:55:15.400905+0000","last_fullsized":"2023-11-28T20:55:15.400905+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:13.803544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312624+0000","last_change":"2023-11-28T20:55:12.290412+0000","last_active":"2023-11-28T20:55:14.312624+0000","last_peered":"2023-11-28T20:55:14.312624+0000","last_clean":"2023-11-28T20:55:14.312624+0000","last_became_active":"2023-11-28T20:55:12.290021+0000","last_became_peered":"2023-11-28T20:55:12.290021+0000","last_unstale":"2023-11-28T20:55:14.312624+0000","last_undegraded":"2023-11-28T20:55:14.312624+0000","last_fullsized":"2023-11-28T20:55:14.312624+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:51.365142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399958+0000","last_change":"2023-11-28T20:55:13.328730+0000","last_active":"2023-11-28T20:55:15.399958+0000","last_peered":"2023-11-28T20:55:15.399958+0000","last_clean":"2023-11-28T20:55:15.399958+0000","last_became_active":"2023-11-28T20:55:13.328133+0000","last_became_peered":"2023-11-28T20:55:13.328133+0000","last_unstale":"2023-11-28T20:55:15.399958+0000","last_undegraded":"2023-11-28T20:55:15.399958+0000","last_fullsized":"2023-11-28T20:55:15.399958+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:52:50.218429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402439+0000","last_change":"2023-11-28T20:55:12.286254+0000","last_active":"2023-11-28T20:55:15.402439+0000","last_peered":"2023-11-28T20:55:15.402439+0000","last_clean":"2023-11-28T20:55:15.402439+0000","last_became_active":"2023-11-28T20:55:12.285891+0000","last_became_peered":"2023-11-28T20:55:12.285891+0000","last_unstale":"2023-11-28T20:55:15.402439+0000","last_undegraded":"2023-11-28T20:55:15.402439+0000","last_fullsized":"2023-11-28T20:55:15.402439+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:46:24.622410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400309+0000","last_change":"2023-11-28T20:55:13.320997+0000","last_active":"2023-11-28T20:55:15.400309+0000","last_peered":"2023-11-28T20:55:15.400309+0000","last_clean":"2023-11-28T20:55:15.400309+0000","last_became_active":"2023-11-28T20:55:13.320712+0000","last_became_peered":"2023-11-28T20:55:13.320712+0000","last_unstale":"2023-11-28T20:55:15.400309+0000","last_undegraded":"2023-11-28T20:55:15.400309+0000","last_fullsized":"2023-11-28T20:55:15.400309+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:53.890830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899877+0000","last_change":"2023-11-28T20:55:12.282974+0000","last_active":"2023-11-28T20:55:14.899877+0000","last_peered":"2023-11-28T20:55:14.899877+0000","last_clean":"2023-11-28T20:55:14.899877+0000","last_became_active":"2023-11-28T20:55:12.282702+0000","last_became_peered":"2023-11-28T20:55:12.282702+0000","last_unstale":"2023-11-28T20:55:14.899877+0000","last_undegraded":"2023-11-28T20:55:14.899877+0000","last_fullsized":"2023-11-28T20:55:14.899877+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:49:26.564041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401040+0000","last_change":"2023-11-28T20:55:13.323527+0000","last_active":"2023-11-28T20:55:15.401040+0000","last_peered":"2023-11-28T20:55:15.401040+0000","last_clean":"2023-11-28T20:55:15.401040+0000","last_became_active":"2023-11-28T20:55:13.323209+0000","last_became_peered":"2023-11-28T20:55:13.323209+0000","last_unstale":"2023-11-28T20:55:15.401040+0000","last_undegraded":"2023-11-28T20:55:15.401040+0000","last_fullsized":"2023-11-28T20:55:15.401040+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:00:14.212039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400298+0000","last_change":"2023-11-28T20:55:12.276974+0000","last_active":"2023-11-28T20:55:15.400298+0000","last_peered":"2023-11-28T20:55:15.400298+0000","last_clean":"2023-11-28T20:55:15.400298+0000","last_became_active":"2023-11-28T20:55:12.276642+0000","last_became_peered":"2023-11-28T20:55:12.276642+0000","last_unstale":"2023-11-28T20:55:15.400298+0000","last_undegraded":"2023-11-28T20:55:15.400298+0000","last_fullsized":"2023-11-28T20:55:15.400298+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:52:49.203269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400131+0000","last_change":"2023-11-28T20:55:13.323863+0000","last_active":"2023-11-28T20:55:15.400131+0000","last_peered":"2023-11-28T20:55:15.400131+0000","last_clean":"2023-11-28T20:55:15.400131+0000","last_became_active":"2023-11-28T20:55:13.322679+0000","last_became_peered":"2023-11-28T20:55:13.322679+0000","last_unstale":"2023-11-28T20:55:15.400131+0000","last_undegraded":"2023-11-28T20:55:15.400131+0000","last_fullsized":"2023-11-28T20:55:15.400131+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:39.018087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.311585+0000","last_change":"2023-11-28T20:55:12.289623+0000","last_active":"2023-11-28T20:55:14.311585+0000","last_peered":"2023-11-28T20:55:14.311585+0000","last_clean":"2023-11-28T20:55:14.311585+0000","last_became_active":"2023-11-28T20:55:12.287473+0000","last_became_peered":"2023-11-28T20:55:12.287473+0000","last_unstale":"2023-11-28T20:55:14.311585+0000","last_undegraded":"2023-11-28T20:55:14.311585+0000","last_fullsized":"2023-11-28T20:55:14.311585+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:12.788384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306038+0000","last_change":"2023-11-28T20:55:13.316679+0000","last_active":"2023-11-28T20:55:14.306038+0000","last_peered":"2023-11-28T20:55:14.306038+0000","last_clean":"2023-11-28T20:55:14.306038+0000","last_became_active":"2023-11-28T20:55:13.312502+0000","last_became_peered":"2023-11-28T20:55:13.312502+0000","last_unstale":"2023-11-28T20:55:14.306038+0000","last_undegraded":"2023-11-28T20:55:14.306038+0000","last_fullsized":"2023-11-28T20:55:14.306038+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:50:04.945656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305594+0000","last_change":"2023-11-28T20:55:12.282617+0000","last_active":"2023-11-28T20:55:14.305594+0000","last_peered":"2023-11-28T20:55:14.305594+0000","last_clean":"2023-11-28T20:55:14.305594+0000","last_became_active":"2023-11-28T20:55:12.282139+0000","last_became_peered":"2023-11-28T20:55:12.282139+0000","last_unstale":"2023-11-28T20:55:14.305594+0000","last_undegraded":"2023-11-28T20:55:14.305594+0000","last_fullsized":"2023-11-28T20:55:14.305594+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:25:15.465465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401209+0000","last_change":"2023-11-28T20:55:13.322608+0000","last_active":"2023-11-28T20:55:15.401209+0000","last_peered":"2023-11-28T20:55:15.401209+0000","last_clean":"2023-11-28T20:55:15.401209+0000","last_became_active":"2023-11-28T20:55:13.322295+0000","last_became_peered":"2023-11-28T20:55:13.322295+0000","last_unstale":"2023-11-28T20:55:15.401209+0000","last_undegraded":"2023-11-28T20:55:15.401209+0000","last_fullsized":"2023-11-28T20:55:15.401209+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:28.969319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400036+0000","last_change":"2023-11-28T20:55:12.293622+0000","last_active":"2023-11-28T20:55:15.400036+0000","last_peered":"2023-11-28T20:55:15.400036+0000","last_clean":"2023-11-28T20:55:15.400036+0000","last_became_active":"2023-11-28T20:55:12.290115+0000","last_became_peered":"2023-11-28T20:55:12.290115+0000","last_unstale":"2023-11-28T20:55:15.400036+0000","last_undegraded":"2023-11-28T20:55:15.400036+0000","last_fullsized":"2023-11-28T20:55:15.400036+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:20.707314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312165+0000","last_change":"2023-11-28T20:55:13.321541+0000","last_active":"2023-11-28T20:55:14.312165+0000","last_peered":"2023-11-28T20:55:14.312165+0000","last_clean":"2023-11-28T20:55:14.312165+0000","last_became_active":"2023-11-28T20:55:13.321253+0000","last_became_peered":"2023-11-28T20:55:13.321253+0000","last_unstale":"2023-11-28T20:55:14.312165+0000","last_undegraded":"2023-11-28T20:55:14.312165+0000","last_fullsized":"2023-11-28T20:55:14.312165+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:14:59.121157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899021+0000","last_change":"2023-11-28T20:55:12.283388+0000","last_active":"2023-11-28T20:55:14.899021+0000","last_peered":"2023-11-28T20:55:14.899021+0000","last_clean":"2023-11-28T20:55:14.899021+0000","last_became_active":"2023-11-28T20:55:12.281875+0000","last_became_peered":"2023-11-28T20:55:12.281875+0000","last_unstale":"2023-11-28T20:55:14.899021+0000","last_undegraded":"2023-11-28T20:55:14.899021+0000","last_fullsized":"2023-11-28T20:55:14.899021+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:08.099139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400322+0000","last_change":"2023-11-28T20:55:13.318979+0000","last_active":"2023-11-28T20:55:15.400322+0000","last_peered":"2023-11-28T20:55:15.400322+0000","last_clean":"2023-11-28T20:55:15.400322+0000","last_became_active":"2023-11-28T20:55:13.317840+0000","last_became_peered":"2023-11-28T20:55:13.317840+0000","last_unstale":"2023-11-28T20:55:15.400322+0000","last_undegraded":"2023-11-28T20:55:15.400322+0000","last_fullsized":"2023-11-28T20:55:15.400322+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:59:09.009449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306149+0000","last_change":"2023-11-28T20:55:12.290074+0000","last_active":"2023-11-28T20:55:14.306149+0000","last_peered":"2023-11-28T20:55:14.306149+0000","last_clean":"2023-11-28T20:55:14.306149+0000","last_became_active":"2023-11-28T20:55:12.289628+0000","last_became_peered":"2023-11-28T20:55:12.289628+0000","last_unstale":"2023-11-28T20:55:14.306149+0000","last_undegraded":"2023-11-28T20:55:14.306149+0000","last_fullsized":"2023-11-28T20:55:14.306149+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:53:14.603837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400917+0000","last_change":"2023-11-28T20:55:13.317542+0000","last_active":"2023-11-28T20:55:15.400917+0000","last_peered":"2023-11-28T20:55:15.400917+0000","last_clean":"2023-11-28T20:55:15.400917+0000","last_became_active":"2023-11-28T20:55:13.317210+0000","last_became_peered":"2023-11-28T20:55:13.317210+0000","last_unstale":"2023-11-28T20:55:15.400917+0000","last_undegraded":"2023-11-28T20:55:15.400917+0000","last_fullsized":"2023-11-28T20:55:15.400917+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:59.166427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.313180+0000","last_change":"2023-11-28T20:55:12.291469+0000","last_active":"2023-11-28T20:55:14.313180+0000","last_peered":"2023-11-28T20:55:14.313180+0000","last_clean":"2023-11-28T20:55:14.313180+0000","last_became_active":"2023-11-28T20:55:12.291109+0000","last_became_peered":"2023-11-28T20:55:12.291109+0000","last_unstale":"2023-11-28T20:55:14.313180+0000","last_undegraded":"2023-11-28T20:55:14.313180+0000","last_fullsized":"2023-11-28T20:55:14.313180+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:35:12.838952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399958+0000","last_change":"2023-11-28T20:55:13.327645+0000","last_active":"2023-11-28T20:55:15.399958+0000","last_peered":"2023-11-28T20:55:15.399958+0000","last_clean":"2023-11-28T20:55:15.399958+0000","last_became_active":"2023-11-28T20:55:13.327353+0000","last_became_peered":"2023-11-28T20:55:13.327353+0000","last_unstale":"2023-11-28T20:55:15.399958+0000","last_undegraded":"2023-11-28T20:55:15.399958+0000","last_fullsized":"2023-11-28T20:55:15.399958+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:04.538041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312110+0000","last_change":"2023-11-28T20:55:12.278227+0000","last_active":"2023-11-28T20:55:14.312110+0000","last_peered":"2023-11-28T20:55:14.312110+0000","last_clean":"2023-11-28T20:55:14.312110+0000","last_became_active":"2023-11-28T20:55:12.277831+0000","last_became_peered":"2023-11-28T20:55:12.277831+0000","last_unstale":"2023-11-28T20:55:14.312110+0000","last_undegraded":"2023-11-28T20:55:14.312110+0000","last_fullsized":"2023-11-28T20:55:14.312110+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:46:24.622410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401042+0000","last_change":"2023-11-28T20:55:13.305664+0000","last_active":"2023-11-28T20:55:15.401042+0000","last_peered":"2023-11-28T20:55:15.401042+0000","last_clean":"2023-11-28T20:55:15.401042+0000","last_became_active":"2023-11-28T20:55:13.305383+0000","last_became_peered":"2023-11-28T20:55:13.305383+0000","last_unstale":"2023-11-28T20:55:15.401042+0000","last_undegraded":"2023-11-28T20:55:15.401042+0000","last_fullsized":"2023-11-28T20:55:15.401042+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:26:04.420246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899397+0000","last_change":"2023-11-28T20:55:12.285675+0000","last_active":"2023-11-28T20:55:14.899397+0000","last_peered":"2023-11-28T20:55:14.899397+0000","last_clean":"2023-11-28T20:55:14.899397+0000","last_became_active":"2023-11-28T20:55:12.285034+0000","last_became_peered":"2023-11-28T20:55:12.285034+0000","last_unstale":"2023-11-28T20:55:14.899397+0000","last_undegraded":"2023-11-28T20:55:14.899397+0000","last_fullsized":"2023-11-28T20:55:14.899397+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:05.873311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899484+0000","last_change":"2023-11-28T20:55:13.309965+0000","last_active":"2023-11-28T20:55:14.899484+0000","last_peered":"2023-11-28T20:55:14.899484+0000","last_clean":"2023-11-28T20:55:14.899484+0000","last_became_active":"2023-11-28T20:55:13.309654+0000","last_became_peered":"2023-11-28T20:55:13.309654+0000","last_unstale":"2023-11-28T20:55:14.899484+0000","last_undegraded":"2023-11-28T20:55:14.899484+0000","last_fullsized":"2023-11-28T20:55:14.899484+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:37:17.347224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309988+0000","last_change":"2023-11-28T20:55:12.288039+0000","last_active":"2023-11-28T20:55:14.309988+0000","last_peered":"2023-11-28T20:55:14.309988+0000","last_clean":"2023-11-28T20:55:14.309988+0000","last_became_active":"2023-11-28T20:55:12.287739+0000","last_became_peered":"2023-11-28T20:55:12.287739+0000","last_unstale":"2023-11-28T20:55:14.309988+0000","last_undegraded":"2023-11-28T20:55:14.309988+0000","last_fullsized":"2023-11-28T20:55:14.309988+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:56:51.993515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899760+0000","last_change":"2023-11-28T20:55:13.310162+0000","last_active":"2023-11-28T20:55:14.899760+0000","last_peered":"2023-11-28T20:55:14.899760+0000","last_clean":"2023-11-28T20:55:14.899760+0000","last_became_active":"2023-11-28T20:55:13.309881+0000","last_became_peered":"2023-11-28T20:55:13.309881+0000","last_unstale":"2023-11-28T20:55:14.899760+0000","last_undegraded":"2023-11-28T20:55:14.899760+0000","last_fullsized":"2023-11-28T20:55:14.899760+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:08:10.115481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400548+0000","last_change":"2023-11-28T20:55:12.295683+0000","last_active":"2023-11-28T20:55:15.400548+0000","last_peered":"2023-11-28T20:55:15.400548+0000","last_clean":"2023-11-28T20:55:15.400548+0000","last_became_active":"2023-11-28T20:55:12.295338+0000","last_became_peered":"2023-11-28T20:55:12.295338+0000","last_unstale":"2023-11-28T20:55:15.400548+0000","last_undegraded":"2023-11-28T20:55:15.400548+0000","last_fullsized":"2023-11-28T20:55:15.400548+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:14:58.105997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899963+0000","last_change":"2023-11-28T20:55:13.314675+0000","last_active":"2023-11-28T20:55:14.899963+0000","last_peered":"2023-11-28T20:55:14.899963+0000","last_clean":"2023-11-28T20:55:14.899963+0000","last_became_active":"2023-11-28T20:55:13.313564+0000","last_became_peered":"2023-11-28T20:55:13.313564+0000","last_unstale":"2023-11-28T20:55:14.899963+0000","last_undegraded":"2023-11-28T20:55:14.899963+0000","last_fullsized":"2023-11-28T20:55:14.899963+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:45.696174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.898916+0000","last_change":"2023-11-28T20:55:12.285073+0000","last_active":"2023-11-28T20:55:14.898916+0000","last_peered":"2023-11-28T20:55:14.898916+0000","last_clean":"2023-11-28T20:55:14.898916+0000","last_became_active":"2023-11-28T20:55:12.284386+0000","last_became_peered":"2023-11-28T20:55:12.284386+0000","last_unstale":"2023-11-28T20:55:14.898916+0000","last_undegraded":"2023-11-28T20:55:14.898916+0000","last_fullsized":"2023-11-28T20:55:14.898916+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:46:24.622410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401308+0000","last_change":"2023-11-28T20:55:13.330287+0000","last_active":"2023-11-28T20:55:15.401308+0000","last_peered":"2023-11-28T20:55:15.401308+0000","last_clean":"2023-11-28T20:55:15.401308+0000","last_became_active":"2023-11-28T20:55:13.329974+0000","last_became_peered":"2023-11-28T20:55:13.329974+0000","last_unstale":"2023-11-28T20:55:15.401308+0000","last_undegraded":"2023-11-28T20:55:15.401308+0000","last_fullsized":"2023-11-28T20:55:15.401308+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:16:48.298816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400603+0000","last_change":"2023-11-28T20:55:12.290156+0000","last_active":"2023-11-28T20:55:15.400603+0000","last_peered":"2023-11-28T20:55:15.400603+0000","last_clean":"2023-11-28T20:55:15.400603+0000","last_became_active":"2023-11-28T20:55:12.289837+0000","last_became_peered":"2023-11-28T20:55:12.289837+0000","last_unstale":"2023-11-28T20:55:15.400603+0000","last_undegraded":"2023-11-28T20:55:15.400603+0000","last_fullsized":"2023-11-28T20:55:15.400603+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:06:55.674296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402724+0000","last_change":"2023-11-28T20:55:13.325752+0000","last_active":"2023-11-28T20:55:15.402724+0000","last_peered":"2023-11-28T20:55:15.402724+0000","last_clean":"2023-11-28T20:55:15.402724+0000","last_became_active":"2023-11-28T20:55:13.325075+0000","last_became_peered":"2023-11-28T20:55:13.325075+0000","last_unstale":"2023-11-28T20:55:15.402724+0000","last_undegraded":"2023-11-28T20:55:15.402724+0000","last_fullsized":"2023-11-28T20:55:15.402724+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:44:41.501846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400384+0000","last_change":"2023-11-28T20:55:12.292548+0000","last_active":"2023-11-28T20:55:15.400384+0000","last_peered":"2023-11-28T20:55:15.400384+0000","last_clean":"2023-11-28T20:55:15.400384+0000","last_became_active":"2023-11-28T20:55:12.292242+0000","last_became_peered":"2023-11-28T20:55:12.292242+0000","last_unstale":"2023-11-28T20:55:15.400384+0000","last_undegraded":"2023-11-28T20:55:15.400384+0000","last_fullsized":"2023-11-28T20:55:15.400384+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:15:15.315609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"150'4756","reported_seq":5338,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-11-28T20:55:13.305743+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:55:13.305743+0000","last_peered":"2023-11-28T20:55:13.305743+0000","last_clean":"2023-11-28T20:55:13.305743+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:55:13.305743+0000","last_undegraded":"2023-11-28T20:55:13.305743+0000","last_fullsized":"2023-11-28T20:55:13.305743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1381,"num_read_kb":2803,"num_write":411,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1381,"num_read_kb":2803,"num_write":411,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4756,"ondisk_log_size":4756,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575099,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47620,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324220,"statfs":{"total":96636764160,"available":96588001280,"internally_reserved":0,"allocated":2605056,"data_stored":1604237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.167,"15min":1.1870000000000001},"min":{"1min":0.623,"5min":0.57099999999999995,"15min":0.45600000000000002},"max":{"1min":1.7490000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.089,"15min":1.1499999999999999},"min":{"1min":0.66900000000000004,"5min":0.64300000000000002,"15min":0.58299999999999996},"max":{"1min":1.5660000000000001,"5min":1.8,"15min":1.8},"last":0.77700000000000002}]},{"osd":1,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1579999999999999,"15min":1.179},"min":{"1min":0.64600000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5629999999999999,"5min":1.633,"15min":1.744},"last":1.329},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.228,"15min":1.252},"min":{"1min":0.71799999999999997,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5900000000000001,"5min":1.8540000000000001,"15min":1.889},"last":0.68000000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.272,"15min":1.2889999999999999},"min":{"1min":0.879,"5min":0.59699999999999998,"15min":0.58799999999999997},"max":{"1min":1.714,"5min":1.714,"15min":1.9359999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3280000000000001,"15min":1.298},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.57499999999999996},"max":{"1min":1.77,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.135}]},{"osd":3,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2470000000000001,"15min":1.27},"min":{"1min":0.76800000000000002,"5min":0.73199999999999998,"15min":0.54200000000000004},"max":{"1min":1.583,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4139999999999999,"15min":1.4099999999999999},"min":{"1min":1.0609999999999999,"5min":0.77000000000000002,"15min":0.68300000000000005},"max":{"1min":1.7869999999999999,"5min":1.956,"15min":1.956},"last":1.264}]},{"osd":4,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.335,"15min":1.29},"min":{"1min":0.89800000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0430000000000001,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":0.87},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.347,"15min":1.3320000000000001},"min":{"1min":1.1399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.877,"5min":1.99,"15min":1.9970000000000001},"last":1.2210000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.4390000000000001,"15min":1.4179999999999999},"min":{"1min":1.1919999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.891,"5min":1.891,"15min":2.048},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.496,"15min":1.474},"min":{"1min":1.288,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":0.95199999999999996}]},{"osd":6,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.581,"15min":1.5589999999999999},"min":{"1min":1,"5min":0.85799999999999998,"15min":0.69499999999999995},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.476,"15min":1.4830000000000001},"min":{"1min":1.0640000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7969999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.3879999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575099,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37312,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334528,"statfs":{"total":96636764160,"available":96598556672,"internally_reserved":0,"allocated":2015232,"data_stored":1019487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.0700000000000001,"15min":1.103},"min":{"1min":0.754,"5min":0.58099999999999996,"15min":0.53500000000000003},"max":{"1min":1.653,"5min":1.653,"15min":1.7350000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0720000000000001,"15min":1.123},"min":{"1min":0.67800000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4350000000000001,"5min":1.8069999999999999,"15min":1.8260000000000001},"last":0.46800000000000003}]},{"osd":1,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1950000000000001,"15min":1.2370000000000001},"min":{"1min":0.71399999999999997,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.6799999999999999,"5min":1.728,"15min":1.8069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.3080000000000001,"15min":1.349},"min":{"1min":0.89800000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.7869999999999999,"5min":1.7889999999999999,"15min":2.0169999999999999},"last":0.80400000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.2470000000000001,"15min":1.272},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.55,"5min":1.798,"15min":1.798},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3220000000000001,"15min":1.3520000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.732,"5min":1.837,"15min":1.851},"last":0.71399999999999997}]},{"osd":3,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.248,"15min":1.284},"min":{"1min":0.63300000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.73,"5min":1.8580000000000001,"15min":1.9299999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.139,"5min":1.232,"15min":1.268},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5740000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2529999999999999,"15min":1.2430000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":2.089},"last":0.5},{"interface":"front","average":{"1min":1.246,"5min":1.296,"15min":1.304},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.875,"5min":1.9179999999999999,"15min":2.2010000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.331,"15min":1.3260000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.581,"5min":1.879,"15min":1.97},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.417,"5min":1.494,"15min":1.5189999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.752,"5min":1.901,"15min":1.958},"last":0.747}]},{"osd":7,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.4239999999999999,"15min":1.4530000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.96,"5min":1.9790000000000001,"15min":2.3759999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.429,"15min":1.4470000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6950000000000001,"5min":1.9630000000000001,"15min":2.0739999999999998},"last":0.78400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575099,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51192,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320648,"statfs":{"total":96636764160,"available":96584343552,"internally_reserved":0,"allocated":2011136,"data_stored":1018837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1619999999999999,"15min":1.1639999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.696,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.0660000000000001,"15min":1.1439999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.441},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.25}]},{"osd":1,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1160000000000001,"15min":1.147},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.623,"5min":1.623,"15min":1.768},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.157,"5min":1.2190000000000001,"15min":1.2130000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6100000000000001,"5min":2.282,"15min":2.282},"last":0.80400000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.302,"15min":1.2829999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.53200000000000003},"max":{"1min":1.893,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":0.999},{"interface":"front","average":{"1min":1.212,"5min":1.21,"15min":1.274},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.66600000000000004},"max":{"1min":1.6240000000000001,"5min":1.8220000000000001,"15min":1.958},"last":1.536}]},{"osd":3,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.286,"15min":1.3140000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.726,"5min":1.7290000000000001,"15min":1.9350000000000001},"last":1.456},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.2589999999999999,"15min":1.262},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.64000000000000001},"max":{"1min":1.6559999999999999,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.165}]},{"osd":4,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3140000000000001,"15min":1.2989999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.67500000000000004},"max":{"1min":1.746,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.052},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4710000000000001,"15min":1.4379999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.8160000000000001,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":1.401}]},{"osd":6,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3380000000000001,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.5},"max":{"1min":1.966,"5min":1.966,"15min":2.093},"last":1.591},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.474,"15min":1.476},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.69499999999999995},"max":{"1min":1.9219999999999999,"5min":1.998,"15min":1.998},"last":1.339}]},{"osd":7,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.464,"15min":1.4330000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.742,"5min":3.319,"15min":3.319},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.474,"15min":1.4830000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.60499999999999998},"max":{"1min":1.764,"5min":1.9790000000000001,"15min":2.1829999999999998},"last":1.6499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575100,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37308,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334532,"statfs":{"total":96636764160,"available":96598560768,"internally_reserved":0,"allocated":2007040,"data_stored":1018815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0580000000000001,"15min":1.125},"min":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.825,"5min":1.825,"15min":1.879},"last":1.242},{"interface":"front","average":{"1min":0.995,"5min":1.0720000000000001,"15min":1.1080000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.3839999999999999,"5min":1.6850000000000001,"15min":1.9750000000000001},"last":0.97699999999999998}]},{"osd":1,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2110000000000001,"15min":1.286},"min":{"1min":0.68999999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.923},"last":1.155},{"interface":"front","average":{"1min":1.335,"5min":1.2689999999999999,"15min":1.343},"min":{"1min":0.72099999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7470000000000001,"5min":1.768,"15min":2.2679999999999998},"last":1.373}]},{"osd":2,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2010000000000001,"15min":1.2190000000000001},"min":{"1min":0.56899999999999995,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.7729999999999999,"5min":1.78,"15min":1.78},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2350000000000001,"15min":1.282},"min":{"1min":0.65000000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.704,"5min":1.704,"15min":2.6949999999999998},"last":1.466}]},{"osd":3,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.2230000000000001,"15min":1.2709999999999999},"min":{"1min":0.63,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.647,"5min":1.9630000000000001,"15min":2.157},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.3149999999999999,"15min":1.3580000000000001},"min":{"1min":0.68700000000000006,"5min":0.5,"15min":0.5},"max":{"1min":1.7509999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.694}]},{"osd":5,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.286,"15min":1.292},"min":{"1min":0.65500000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.849,"5min":1.9970000000000001,"15min":2.4279999999999999},"last":1.647},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.2909999999999999,"15min":1.298},"min":{"1min":0.746,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.784,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.4179999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.3979999999999999,"15min":1.395},"min":{"1min":0.96699999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.8899999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.76},{"interface":"front","average":{"1min":1.538,"5min":1.5289999999999999,"15min":1.5509999999999999},"min":{"1min":0.92400000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.9219999999999999,"5min":2.1349999999999998,"15min":3.7000000000000002},"last":1.5589999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.444,"15min":1.46},"min":{"1min":0.76200000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.9159999999999999,"5min":2.2490000000000001,"15min":2.4990000000000001},"last":1.52},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.522,"15min":1.518},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.0510000000000002,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.843}]}]},{"osd":3,"up_from":13,"seq":55834575100,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41476,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330364,"statfs":{"total":96636764160,"available":96594292736,"internally_reserved":0,"allocated":2023424,"data_stored":1020497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.85699999999999998,"15min":0.89400000000000002},"min":{"1min":0.47299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.383,"5min":1.6000000000000001,"15min":4.7210000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.92800000000000005,"15min":1.0229999999999999},"min":{"1min":0.39700000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.73,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":1.1850000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.92900000000000005,"15min":0.96099999999999997},"min":{"1min":0.495,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6739999999999999,"5min":1.9770000000000001,"15min":2.8460000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.071,"15min":1.1719999999999999},"min":{"1min":0.53100000000000003,"5min":0.379,"15min":0.379},"max":{"1min":1.4219999999999999,"5min":1.7190000000000001,"15min":3.7410000000000001},"last":0.98599999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1140000000000001,"15min":1.216},"min":{"1min":0.65900000000000003,"5min":0.53100000000000003,"15min":0.48499999999999999},"max":{"1min":1.5429999999999999,"5min":1.9019999999999999,"15min":3.444},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.04,"5min":1.2030000000000001,"15min":1.3029999999999999},"min":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.452,"5min":1.9530000000000001,"15min":3.7050000000000001},"last":1.4870000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2849999999999999,"15min":1.335},"min":{"1min":0.76000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8089999999999999,"5min":1.956,"15min":2.9420000000000002},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.28,"5min":1.3300000000000001,"15min":1.3700000000000001},"min":{"1min":0.91400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.8819999999999999,"5min":2.246,"15min":2.246},"last":1.429}]},{"osd":5,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3799999999999999,"15min":1.4339999999999999},"min":{"1min":0.88800000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6659999999999999,"5min":1.8959999999999999,"15min":3.7730000000000001},"last":1.27},{"interface":"front","average":{"1min":1.379,"5min":1.387,"15min":1.4330000000000001},"min":{"1min":0.81100000000000005,"5min":0.66800000000000004,"15min":0.53500000000000003},"max":{"1min":1.875,"5min":2.0110000000000001,"15min":3.1520000000000001},"last":1.3029999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.399,"15min":1.458},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.825,"5min":2.0630000000000002,"15min":2.3100000000000001},"last":1.032},{"interface":"front","average":{"1min":1.405,"5min":1.46,"15min":1.5509999999999999},"min":{"1min":0.97599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.679,"5min":2.0840000000000001,"15min":3.661},"last":1.375}]},{"osd":7,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.518},"min":{"1min":0.83199999999999996,"5min":0.73399999999999999,"15min":0.71499999999999997},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":3.266},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.478,"5min":1.522,"15min":1.575},"min":{"1min":0.996,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.7450000000000001,"5min":2.2080000000000002,"15min":2.3919999999999999},"last":1.2210000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575100,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63684,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308156,"statfs":{"total":96636764160,"available":96571551744,"internally_reserved":0,"allocated":2015232,"data_stored":1019431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.86199999999999999,"15min":0.90200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":1.2829999999999999,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.0049999999999999,"15min":1.052},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.387,"5min":4.8490000000000002,"15min":4.8490000000000002},"last":1.637}]},{"osd":1,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.96899999999999997,"15min":0.98499999999999999},"min":{"1min":0.498,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.458,"5min":1.8160000000000001,"15min":3.2599999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0309999999999999,"15min":1.1240000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.357,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.819}]},{"osd":3,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1160000000000001,"15min":1.161},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.589,"5min":4.9349999999999996,"15min":4.9349999999999996},"last":1.9379999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1970000000000001,"15min":1.282},"min":{"1min":0.64900000000000002,"5min":0.48499999999999999,"15min":0.48199999999999998},"max":{"1min":1.994,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":1.288}]},{"osd":4,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3440000000000001,"15min":1.3979999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.673,"5min":3.6499999999999999,"15min":3.6499999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.363,"5min":1.448,"15min":1.4350000000000001},"min":{"1min":0.88400000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8340000000000001,"5min":4.6479999999999997,"15min":4.6479999999999997},"last":2.1419999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3819999999999999,"15min":1.413},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.923,"5min":2.492,"15min":2.492},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.429,"15min":1.4830000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6399999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.6899999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4810000000000001,"15min":1.508},"min":{"1min":0.79200000000000004,"5min":0.747,"15min":0.747},"max":{"1min":2.153,"5min":4.5270000000000001,"15min":4.5270000000000001},"last":2.423},{"interface":"front","average":{"1min":1.355,"5min":1.5329999999999999,"15min":1.5660000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.82499999999999996},"max":{"1min":1.8560000000000001,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.885}]},{"osd":7,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4650000000000001,"15min":1.524},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.0720000000000001,"5min":2.641,"15min":2.641},"last":2.0449999999999999},{"interface":"front","average":{"1min":1.484,"5min":1.548,"15min":1.585},"min":{"1min":0.81799999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.2770000000000001,"5min":4.7530000000000001,"15min":4.7530000000000001},"last":2.3079999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37304,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334536,"statfs":{"total":96636764160,"available":96598564864,"internally_reserved":0,"allocated":2011136,"data_stored":1018885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.90800000000000003,"15min":0.91100000000000003},"min":{"1min":0.38900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.466,"5min":1.8009999999999999,"15min":1.9259999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.98899999999999999,"15min":1.069},"min":{"1min":0.505,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.6120000000000001,"5min":2.028,"15min":2.028},"last":1.2250000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0389999999999999,"15min":1.0740000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.371},"max":{"1min":1.3959999999999999,"5min":1.766,"15min":2.234},"last":1.2949999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.153,"15min":1.1910000000000001},"min":{"1min":0.628,"5min":0.53700000000000003,"15min":0.52800000000000002},"max":{"1min":1.4259999999999999,"5min":1.9199999999999999,"15min":2.895},"last":1.6699999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.2110000000000001,"15min":1.2889999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.46700000000000003},"max":{"1min":1.5109999999999999,"5min":1.7949999999999999,"15min":2.0459999999999998},"last":1.361},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.0760000000000001,"15min":1.101},"min":{"1min":0.47299999999999998,"5min":0.45100000000000001,"15min":0.37},"max":{"1min":1.756,"5min":1.9790000000000001,"15min":2.3679999999999999},"last":1.0920000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3080000000000001,"15min":1.339},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5629999999999999,"5min":1.8140000000000001,"15min":2.1150000000000002},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.383,"15min":1.391},"min":{"1min":0.77000000000000002,"5min":0.72399999999999998,"15min":0.71899999999999997},"max":{"1min":1.776,"5min":1.8819999999999999,"15min":2.6819999999999999},"last":1.518}]},{"osd":5,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.4299999999999999,"15min":1.4339999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.9650000000000001,"5min":2.0190000000000001,"15min":2.1840000000000002},"last":1.728},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.4450000000000001,"15min":1.482},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.9530000000000001,"5min":2.0219999999999998,"15min":2.294},"last":1.4910000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.399,"15min":1.4179999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.66900000000000004},"max":{"1min":1.772,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.806},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.502,"15min":1.524},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.77100000000000002},"max":{"1min":1.9139999999999999,"5min":2.0049999999999999,"15min":2.3300000000000001},"last":1.9299999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.51,"15min":1.552},"min":{"1min":0.80800000000000005,"5min":0.80500000000000005,"15min":0.753},"max":{"1min":1.8959999999999999,"5min":2.1429999999999998,"15min":2.3359999999999999},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.488,"15min":1.5209999999999999},"min":{"1min":0.98999999999999999,"5min":0.80500000000000005,"15min":0.72799999999999998},"max":{"1min":2.165,"5min":2.2330000000000001,"15min":2.294},"last":1.5589999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575100,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46204,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325636,"statfs":{"total":96636764160,"available":96589451264,"internally_reserved":0,"allocated":2596864,"data_stored":1602591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.83099999999999996,"15min":0.89300000000000002},"min":{"1min":0.32000000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.29,"5min":1.3919999999999999,"15min":2.1440000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.0249999999999999,"15min":1.073},"min":{"1min":0.372,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7090000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.98799999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.0129999999999999,"15min":1.091},"min":{"1min":0.50800000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.4299999999999999,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.046},{"interface":"front","average":{"1min":1.034,"5min":1.0660000000000001,"15min":1.1499999999999999},"min":{"1min":0.48099999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.544,"5min":1.782,"15min":1.9159999999999999},"last":0.69899999999999995}]},{"osd":3,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.026,"15min":1.0820000000000001},"min":{"1min":0.432,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.522,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.119,"5min":1.2130000000000001,"15min":1.2829999999999999},"min":{"1min":0.55000000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.7150000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.196}]},{"osd":4,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3420000000000001,"15min":1.4059999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.883,"5min":1.883,"15min":2.012},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.357,"15min":1.383},"min":{"1min":0.72899999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.9890000000000001,"5min":4.5119999999999996,"15min":4.5119999999999996},"last":1.321}]},{"osd":5,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4039999999999999,"15min":1.411},"min":{"1min":0.79200000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.929,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4139999999999999,"15min":1.466},"min":{"1min":0.66800000000000004,"5min":0.621,"15min":0.621},"max":{"1min":2.169,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.377}]},{"osd":6,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4830000000000001,"15min":1.536},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.0590000000000002,"5min":2.524,"15min":2.524},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.458,"15min":1.51},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.972,"5min":2.2330000000000001,"15min":2.4209999999999998},"last":1.177}]},{"osd":7,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.5329999999999999,"15min":1.579},"min":{"1min":0.89300000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.98,"5min":2.2970000000000002,"15min":2.3610000000000002},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.48,"15min":1.524},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.1469999999999998,"5min":4.7359999999999998,"15min":4.7359999999999998},"last":1.474}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:55:17.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T20:55:18.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.224+0000 7f854a0ff640 1 Processor -- start 2023-11-28T20:55:18.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.224+0000 7f854a0ff640 1 -- start start 2023-11-28T20:55:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.226+0000 7f854a0ff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8544072200 0x7f85440b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.226+0000 7f854a0ff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8544146610 0x7f85440b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.226+0000 7f854a0ff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 0x7f8544160e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.226+0000 7f854a0ff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85440725e0 con 0x7f8544072200 2023-11-28T20:55:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.226+0000 7f854a0ff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85440b4310 con 0x7f8544146610 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.226+0000 7f854a0ff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854406d8d0 con 0x7f8544146b60 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.226+0000 7f85490fd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8544072200 0x7f85440b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.226+0000 7f85490fd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8544072200 0x7f85440b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41640/0 (socket says 172.21.15.45:41640) 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.226+0000 7f85490fd640 1 -- 172.21.15.45:0/4105230539 learned_addr learned my addr 172.21.15.45:0/4105230539 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.227+0000 7f85488fc640 1 --2- 172.21.15.45:0/4105230539 >> v2:172.21.15.190:3300/0 conn(0x7f8544146610 0x7f85440b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.227+0000 7f85498fe640 1 --2- 172.21.15.45:0/4105230539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 0x7f8544160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.227+0000 7f85490fd640 1 -- 172.21.15.45:0/4105230539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 msgr2=0x7f8544160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.227+0000 7f85490fd640 1 --2- 172.21.15.45:0/4105230539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 0x7f8544160e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.227+0000 7f85490fd640 1 -- 172.21.15.45:0/4105230539 >> v2:172.21.15.190:3300/0 conn(0x7f8544146610 msgr2=0x7f85440b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.227+0000 7f85490fd640 1 --2- 172.21.15.45:0/4105230539 >> v2:172.21.15.190:3300/0 conn(0x7f8544146610 0x7f85440b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.227+0000 7f85490fd640 1 -- 172.21.15.45:0/4105230539 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854406ca20 con 0x7f8544072200 2023-11-28T20:55:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.228+0000 7f85490fd640 1 --2- 172.21.15.45:0/4105230539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 0x7f85440b1d90 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f854409d0c0 tx=0x7f854002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ac47bf1686d67d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f853bfff640 1 -- 172.21.15.45:0/4105230539 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540031860 con 0x7f8544072200 2023-11-28T20:55:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f853bfff640 1 -- 172.21.15.45:0/4105230539 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85400319c0 con 0x7f8544072200 2023-11-28T20:55:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f853bfff640 1 -- 172.21.15.45:0/4105230539 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540031d50 con 0x7f8544072200 2023-11-28T20:55:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f854a0ff640 1 -- 172.21.15.45:0/4105230539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 msgr2=0x7f85440b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f854a0ff640 1 --2- 172.21.15.45:0/4105230539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 0x7f85440b1d90 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f854409d0c0 tx=0x7f854002b900 comp rx=0 tx=0).stop 2023-11-28T20:55:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f854a0ff640 1 -- 172.21.15.45:0/4105230539 shutdown_connections 2023-11-28T20:55:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f854a0ff640 1 --2- 172.21.15.45:0/4105230539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 0x7f8544160e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f854a0ff640 1 --2- 172.21.15.45:0/4105230539 >> v2:172.21.15.190:3300/0 conn(0x7f8544146610 0x7f85440b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f854a0ff640 1 --2- 172.21.15.45:0/4105230539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 0x7f85440b1d90 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f854a0ff640 1 -- 172.21.15.45:0/4105230539 >> 172.21.15.45:0/4105230539 conn(0x7f854406ab30 msgr2=0x7f85440b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:18.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.229+0000 7f854a0ff640 1 -- 172.21.15.45:0/4105230539 shutdown_connections 2023-11-28T20:55:18.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.230+0000 7f854a0ff640 1 -- 172.21.15.45:0/4105230539 wait complete. 2023-11-28T20:55:18.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.231+0000 7f854a0ff640 1 Processor -- start 2023-11-28T20:55:18.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.231+0000 7f854a0ff640 1 -- start start 2023-11-28T20:55:18.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.232+0000 7f854a0ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 0x7f85441398a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.232+0000 7f854a0ff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8544146610 0x7f854409bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.232+0000 7f854a0ff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 0x7f85440c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f854a0ff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854409c8b0 con 0x7f8544072200 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f854a0ff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854415f330 con 0x7f8544146610 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f854a0ff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85441f3230 con 0x7f8544146b60 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85490fd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 0x7f85441398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85490fd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 0x7f85441398a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41656/0 (socket says 172.21.15.45:41656) 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85490fd640 1 -- 172.21.15.45:0/179451542 learned_addr learned my addr 172.21.15.45:0/179451542 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85488fc640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8544146610 0x7f854409bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85498fe640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 0x7f85440c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85490fd640 1 -- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 msgr2=0x7f85440c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85490fd640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 0x7f85440c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85490fd640 1 -- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8544146610 msgr2=0x7f854409bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85490fd640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8544146610 0x7f854409bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85490fd640 1 -- 172.21.15.45:0/179451542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85400094d0 con 0x7f8544072200 2023-11-28T20:55:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.233+0000 7f85498fe640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 0x7f85440c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.234+0000 7f85490fd640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 0x7f85441398a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f8540000c00 tx=0x7f85400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.234+0000 7f8539ffb640 1 -- 172.21.15.45:0/179451542 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854003ac60 con 0x7f8544072200 2023-11-28T20:55:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.234+0000 7f8539ffb640 1 -- 172.21.15.45:0/179451542 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854003adc0 con 0x7f8544072200 2023-11-28T20:55:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.234+0000 7f8539ffb640 1 -- 172.21.15.45:0/179451542 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400445f0 con 0x7f8544072200 2023-11-28T20:55:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.234+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854415ea00 con 0x7f8544072200 2023-11-28T20:55:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.234+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854415efc0 con 0x7f8544072200 2023-11-28T20:55:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.235+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854409d0f0 con 0x7f8544072200 2023-11-28T20:55:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.236+0000 7f8539ffb640 1 -- 172.21.15.45:0/179451542 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8540044750 con 0x7f8544072200 2023-11-28T20:55:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.237+0000 7f8539ffb640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f852806af60 0x7f852806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.237+0000 7f8539ffb640 1 -- 172.21.15.45:0/179451542 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f85400abef0 con 0x7f8544072200 2023-11-28T20:55:18.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.238+0000 7f85488fc640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f852806af60 0x7f852806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.239+0000 7f85488fc640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f852806af60 0x7f852806d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8544139f90 tx=0x7f8534008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:18.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.242+0000 7f8539ffb640 1 -- 172.21.15.45:0/179451542 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8540076cb0 con 0x7f8544072200 2023-11-28T20:55:18.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.433+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 --> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8544066fb0 con 0x7f852806af60 2023-11-28T20:55:18.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.456+0000 7f8539ffb640 1 -- 172.21.15.45:0/179451542 <== mgr.7396 v2:172.21.15.190:6832/226305585 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287362 (secure 0 0 0) 0x7f8544066fb0 con 0x7f852806af60 2023-11-28T20:55:18.463 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:18.464 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T20:55:18.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f852806af60 msgr2=0x7f852806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f852806af60 0x7f852806d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8544139f90 tx=0x7f8534008040 comp rx=0 tx=0).stop 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 msgr2=0x7f85441398a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 0x7f85441398a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f8540000c00 tx=0x7f85400046c0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 shutdown_connections 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8544146b60 0x7f85440c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8544146610 0x7f854409bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f852806af60 0x7f852806d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 --2- 172.21.15.45:0/179451542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8544072200 0x7f85441398a0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 >> 172.21.15.45:0/179451542 conn(0x7f854406ab30 msgr2=0x7f85440b0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 shutdown_connections 2023-11-28T20:55:18.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.462+0000 7f854a0ff640 1 -- 172.21.15.45:0/179451542 wait complete. 2023-11-28T20:55:18.486 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-28T20:55:16.631852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1381,"num_read_kb":2803,"num_write":455,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4794,"ondisk_log_size":4794,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":362100,"kb_used_data":16880,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754612620,"statfs":{"total":773094113280,"available":772723322880,"internally_reserved":0,"allocated":17285120,"data_stored":9322780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":353357360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.387451"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306215+0000","last_change":"2023-11-28T20:55:13.317488+0000","last_active":"2023-11-28T20:55:14.306215+0000","last_peered":"2023-11-28T20:55:14.306215+0000","last_clean":"2023-11-28T20:55:14.306215+0000","last_became_active":"2023-11-28T20:55:13.317204+0000","last_became_peered":"2023-11-28T20:55:13.317204+0000","last_unstale":"2023-11-28T20:55:14.306215+0000","last_undegraded":"2023-11-28T20:55:14.306215+0000","last_fullsized":"2023-11-28T20:55:14.306215+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:56.827248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.311139+0000","last_change":"2023-11-28T20:55:13.311128+0000","last_active":"2023-11-28T20:55:14.311139+0000","last_peered":"2023-11-28T20:55:14.311139+0000","last_clean":"2023-11-28T20:55:14.311139+0000","last_became_active":"2023-11-28T20:55:13.310854+0000","last_became_peered":"2023-11-28T20:55:13.310854+0000","last_unstale":"2023-11-28T20:55:14.311139+0000","last_undegraded":"2023-11-28T20:55:14.311139+0000","last_fullsized":"2023-11-28T20:55:14.311139+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:31.407809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402338+0000","last_change":"2023-11-28T20:55:13.325916+0000","last_active":"2023-11-28T20:55:15.402338+0000","last_peered":"2023-11-28T20:55:15.402338+0000","last_clean":"2023-11-28T20:55:15.402338+0000","last_became_active":"2023-11-28T20:55:13.325488+0000","last_became_peered":"2023-11-28T20:55:13.325488+0000","last_unstale":"2023-11-28T20:55:15.402338+0000","last_undegraded":"2023-11-28T20:55:15.402338+0000","last_fullsized":"2023-11-28T20:55:15.402338+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:06.888471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.310102+0000","last_change":"2023-11-28T20:55:13.309975+0000","last_active":"2023-11-28T20:55:14.310102+0000","last_peered":"2023-11-28T20:55:14.310102+0000","last_clean":"2023-11-28T20:55:14.310102+0000","last_became_active":"2023-11-28T20:55:13.309687+0000","last_became_peered":"2023-11-28T20:55:13.309687+0000","last_unstale":"2023-11-28T20:55:14.310102+0000","last_undegraded":"2023-11-28T20:55:14.310102+0000","last_fullsized":"2023-11-28T20:55:14.310102+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:15:16.330769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399936+0000","last_change":"2023-11-28T20:55:13.307271+0000","last_active":"2023-11-28T20:55:15.399936+0000","last_peered":"2023-11-28T20:55:15.399936+0000","last_clean":"2023-11-28T20:55:15.399936+0000","last_became_active":"2023-11-28T20:55:13.307003+0000","last_became_peered":"2023-11-28T20:55:13.307003+0000","last_unstale":"2023-11-28T20:55:15.399936+0000","last_undegraded":"2023-11-28T20:55:15.399936+0000","last_fullsized":"2023-11-28T20:55:15.399936+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:53.890830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899005+0000","last_change":"2023-11-28T20:55:13.327242+0000","last_active":"2023-11-28T20:55:14.899005+0000","last_peered":"2023-11-28T20:55:14.899005+0000","last_clean":"2023-11-28T20:55:14.899005+0000","last_became_active":"2023-11-28T20:55:13.326922+0000","last_became_peered":"2023-11-28T20:55:13.326922+0000","last_unstale":"2023-11-28T20:55:14.899005+0000","last_undegraded":"2023-11-28T20:55:14.899005+0000","last_fullsized":"2023-11-28T20:55:14.899005+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:27:14.578033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399809+0000","last_change":"2023-11-28T20:55:13.328896+0000","last_active":"2023-11-28T20:55:15.399809+0000","last_peered":"2023-11-28T20:55:15.399809+0000","last_clean":"2023-11-28T20:55:15.399809+0000","last_became_active":"2023-11-28T20:55:13.328474+0000","last_became_peered":"2023-11-28T20:55:13.328474+0000","last_unstale":"2023-11-28T20:55:15.399809+0000","last_undegraded":"2023-11-28T20:55:15.399809+0000","last_fullsized":"2023-11-28T20:55:15.399809+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:06:56.689456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309938+0000","last_change":"2023-11-28T20:55:13.305185+0000","last_active":"2023-11-28T20:55:14.309938+0000","last_peered":"2023-11-28T20:55:14.309938+0000","last_clean":"2023-11-28T20:55:14.309938+0000","last_became_active":"2023-11-28T20:55:13.302555+0000","last_became_peered":"2023-11-28T20:55:13.302555+0000","last_unstale":"2023-11-28T20:55:14.309938+0000","last_undegraded":"2023-11-28T20:55:14.309938+0000","last_fullsized":"2023-11-28T20:55:14.309938+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:09.114299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899535+0000","last_change":"2023-11-28T20:55:13.304329+0000","last_active":"2023-11-28T20:55:14.899535+0000","last_peered":"2023-11-28T20:55:14.899535+0000","last_clean":"2023-11-28T20:55:14.899535+0000","last_became_active":"2023-11-28T20:55:13.303518+0000","last_became_peered":"2023-11-28T20:55:13.303518+0000","last_unstale":"2023-11-28T20:55:14.899535+0000","last_undegraded":"2023-11-28T20:55:14.899535+0000","last_fullsized":"2023-11-28T20:55:14.899535+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:41:10.079803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401656+0000","last_change":"2023-11-28T20:55:13.303138+0000","last_active":"2023-11-28T20:55:15.401656+0000","last_peered":"2023-11-28T20:55:15.401656+0000","last_clean":"2023-11-28T20:55:15.401656+0000","last_became_active":"2023-11-28T20:55:13.302775+0000","last_became_peered":"2023-11-28T20:55:13.302775+0000","last_unstale":"2023-11-28T20:55:15.401656+0000","last_undegraded":"2023-11-28T20:55:15.401656+0000","last_fullsized":"2023-11-28T20:55:15.401656+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:57:52.187967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.310567+0000","last_change":"2023-11-28T20:55:13.325629+0000","last_active":"2023-11-28T20:55:14.310567+0000","last_peered":"2023-11-28T20:55:14.310567+0000","last_clean":"2023-11-28T20:55:14.310567+0000","last_became_active":"2023-11-28T20:55:13.325347+0000","last_became_peered":"2023-11-28T20:55:13.325347+0000","last_unstale":"2023-11-28T20:55:14.310567+0000","last_undegraded":"2023-11-28T20:55:14.310567+0000","last_fullsized":"2023-11-28T20:55:14.310567+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:47:59.330335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306216+0000","last_change":"2023-11-28T20:55:13.326510+0000","last_active":"2023-11-28T20:55:14.306216+0000","last_peered":"2023-11-28T20:55:14.306216+0000","last_clean":"2023-11-28T20:55:14.306216+0000","last_became_active":"2023-11-28T20:55:13.326218+0000","last_became_peered":"2023-11-28T20:55:13.326218+0000","last_unstale":"2023-11-28T20:55:14.306216+0000","last_undegraded":"2023-11-28T20:55:14.306216+0000","last_fullsized":"2023-11-28T20:55:14.306216+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:02:47.005001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899517+0000","last_change":"2023-11-28T20:55:13.327316+0000","last_active":"2023-11-28T20:55:14.899517+0000","last_peered":"2023-11-28T20:55:14.899517+0000","last_clean":"2023-11-28T20:55:14.899517+0000","last_became_active":"2023-11-28T20:55:13.326939+0000","last_became_peered":"2023-11-28T20:55:13.326939+0000","last_unstale":"2023-11-28T20:55:14.899517+0000","last_undegraded":"2023-11-28T20:55:14.899517+0000","last_fullsized":"2023-11-28T20:55:14.899517+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:56.861515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305657+0000","last_change":"2023-11-28T20:55:13.307180+0000","last_active":"2023-11-28T20:55:14.305657+0000","last_peered":"2023-11-28T20:55:14.305657+0000","last_clean":"2023-11-28T20:55:14.305657+0000","last_became_active":"2023-11-28T20:55:13.303740+0000","last_became_peered":"2023-11-28T20:55:13.303740+0000","last_unstale":"2023-11-28T20:55:14.305657+0000","last_undegraded":"2023-11-28T20:55:14.305657+0000","last_fullsized":"2023-11-28T20:55:14.305657+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:06:50.935582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305352+0000","last_change":"2023-11-28T20:55:13.314500+0000","last_active":"2023-11-28T20:55:14.305352+0000","last_peered":"2023-11-28T20:55:14.305352+0000","last_clean":"2023-11-28T20:55:14.305352+0000","last_became_active":"2023-11-28T20:55:13.314165+0000","last_became_peered":"2023-11-28T20:55:13.314165+0000","last_unstale":"2023-11-28T20:55:14.305352+0000","last_undegraded":"2023-11-28T20:55:14.305352+0000","last_fullsized":"2023-11-28T20:55:14.305352+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:24.662041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.898560+0000","last_change":"2023-11-28T20:55:13.306136+0000","last_active":"2023-11-28T20:55:14.898560+0000","last_peered":"2023-11-28T20:55:14.898560+0000","last_clean":"2023-11-28T20:55:14.898560+0000","last_became_active":"2023-11-28T20:55:13.305829+0000","last_became_peered":"2023-11-28T20:55:13.305829+0000","last_unstale":"2023-11-28T20:55:14.898560+0000","last_undegraded":"2023-11-28T20:55:14.898560+0000","last_fullsized":"2023-11-28T20:55:14.898560+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:24:34.938850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899726+0000","last_change":"2023-11-28T20:55:13.319510+0000","last_active":"2023-11-28T20:55:14.899726+0000","last_peered":"2023-11-28T20:55:14.899726+0000","last_clean":"2023-11-28T20:55:14.899726+0000","last_became_active":"2023-11-28T20:55:13.319197+0000","last_became_peered":"2023-11-28T20:55:13.319197+0000","last_unstale":"2023-11-28T20:55:14.899726+0000","last_undegraded":"2023-11-28T20:55:14.899726+0000","last_fullsized":"2023-11-28T20:55:14.899726+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:09.109449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400561+0000","last_change":"2023-11-28T20:55:13.315912+0000","last_active":"2023-11-28T20:55:15.400561+0000","last_peered":"2023-11-28T20:55:15.400561+0000","last_clean":"2023-11-28T20:55:15.400561+0000","last_became_active":"2023-11-28T20:55:13.315638+0000","last_became_peered":"2023-11-28T20:55:13.315638+0000","last_unstale":"2023-11-28T20:55:15.400561+0000","last_undegraded":"2023-11-28T20:55:15.400561+0000","last_fullsized":"2023-11-28T20:55:15.400561+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:16:48.298816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400696+0000","last_change":"2023-11-28T20:55:13.324886+0000","last_active":"2023-11-28T20:55:15.400696+0000","last_peered":"2023-11-28T20:55:15.400696+0000","last_clean":"2023-11-28T20:55:15.400696+0000","last_became_active":"2023-11-28T20:55:13.324270+0000","last_became_peered":"2023-11-28T20:55:13.324270+0000","last_unstale":"2023-11-28T20:55:15.400696+0000","last_undegraded":"2023-11-28T20:55:15.400696+0000","last_fullsized":"2023-11-28T20:55:15.400696+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:52.992994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400046+0000","last_change":"2023-11-28T20:55:13.316904+0000","last_active":"2023-11-28T20:55:15.400046+0000","last_peered":"2023-11-28T20:55:15.400046+0000","last_clean":"2023-11-28T20:55:15.400046+0000","last_became_active":"2023-11-28T20:55:13.311998+0000","last_became_peered":"2023-11-28T20:55:13.311998+0000","last_unstale":"2023-11-28T20:55:15.400046+0000","last_undegraded":"2023-11-28T20:55:15.400046+0000","last_fullsized":"2023-11-28T20:55:15.400046+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:52.992994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309617+0000","last_change":"2023-11-28T20:55:13.304523+0000","last_active":"2023-11-28T20:55:14.309617+0000","last_peered":"2023-11-28T20:55:14.309617+0000","last_clean":"2023-11-28T20:55:14.309617+0000","last_became_active":"2023-11-28T20:55:13.304240+0000","last_became_peered":"2023-11-28T20:55:13.304240+0000","last_unstale":"2023-11-28T20:55:14.309617+0000","last_undegraded":"2023-11-28T20:55:14.309617+0000","last_fullsized":"2023-11-28T20:55:14.309617+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:53:01.875044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305386+0000","last_change":"2023-11-28T20:55:13.314781+0000","last_active":"2023-11-28T20:55:14.305386+0000","last_peered":"2023-11-28T20:55:14.305386+0000","last_clean":"2023-11-28T20:55:14.305386+0000","last_became_active":"2023-11-28T20:55:13.314499+0000","last_became_peered":"2023-11-28T20:55:13.314499+0000","last_unstale":"2023-11-28T20:55:14.305386+0000","last_undegraded":"2023-11-28T20:55:14.305386+0000","last_fullsized":"2023-11-28T20:55:14.305386+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:06:02.047335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312061+0000","last_change":"2023-11-28T20:55:13.307697+0000","last_active":"2023-11-28T20:55:14.312061+0000","last_peered":"2023-11-28T20:55:14.312061+0000","last_clean":"2023-11-28T20:55:14.312061+0000","last_became_active":"2023-11-28T20:55:13.306852+0000","last_became_peered":"2023-11-28T20:55:13.306852+0000","last_unstale":"2023-11-28T20:55:14.312061+0000","last_undegraded":"2023-11-28T20:55:14.312061+0000","last_fullsized":"2023-11-28T20:55:14.312061+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:21.722474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.898769+0000","last_change":"2023-11-28T20:55:13.302163+0000","last_active":"2023-11-28T20:55:14.898769+0000","last_peered":"2023-11-28T20:55:14.898769+0000","last_clean":"2023-11-28T20:55:14.898769+0000","last_became_active":"2023-11-28T20:55:13.301857+0000","last_became_peered":"2023-11-28T20:55:13.301857+0000","last_unstale":"2023-11-28T20:55:14.898769+0000","last_undegraded":"2023-11-28T20:55:14.898769+0000","last_fullsized":"2023-11-28T20:55:14.898769+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:32:34.418435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402713+0000","last_change":"2023-11-28T20:55:13.324352+0000","last_active":"2023-11-28T20:55:15.402713+0000","last_peered":"2023-11-28T20:55:15.402713+0000","last_clean":"2023-11-28T20:55:15.402713+0000","last_became_active":"2023-11-28T20:55:13.320898+0000","last_became_peered":"2023-11-28T20:55:13.320898+0000","last_unstale":"2023-11-28T20:55:15.402713+0000","last_undegraded":"2023-11-28T20:55:15.402713+0000","last_fullsized":"2023-11-28T20:55:15.402713+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:05:00.230098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402425+0000","last_change":"2023-11-28T20:55:13.323315+0000","last_active":"2023-11-28T20:55:15.402425+0000","last_peered":"2023-11-28T20:55:15.402425+0000","last_clean":"2023-11-28T20:55:15.402425+0000","last_became_active":"2023-11-28T20:55:13.322990+0000","last_became_peered":"2023-11-28T20:55:13.322990+0000","last_unstale":"2023-11-28T20:55:15.402425+0000","last_undegraded":"2023-11-28T20:55:15.402425+0000","last_fullsized":"2023-11-28T20:55:15.402425+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:07:18.524250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400911+0000","last_change":"2023-11-28T20:55:13.322205+0000","last_active":"2023-11-28T20:55:15.400911+0000","last_peered":"2023-11-28T20:55:15.400911+0000","last_clean":"2023-11-28T20:55:15.400911+0000","last_became_active":"2023-11-28T20:55:13.321824+0000","last_became_peered":"2023-11-28T20:55:13.321824+0000","last_unstale":"2023-11-28T20:55:15.400911+0000","last_undegraded":"2023-11-28T20:55:15.400911+0000","last_fullsized":"2023-11-28T20:55:15.400911+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:39.018087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400904+0000","last_change":"2023-11-28T20:55:13.307051+0000","last_active":"2023-11-28T20:55:15.400904+0000","last_peered":"2023-11-28T20:55:15.400904+0000","last_clean":"2023-11-28T20:55:15.400904+0000","last_became_active":"2023-11-28T20:55:13.306732+0000","last_became_peered":"2023-11-28T20:55:13.306732+0000","last_unstale":"2023-11-28T20:55:15.400904+0000","last_undegraded":"2023-11-28T20:55:15.400904+0000","last_fullsized":"2023-11-28T20:55:15.400904+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:42:04.217860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402733+0000","last_change":"2023-11-28T20:55:13.323618+0000","last_active":"2023-11-28T20:55:15.402733+0000","last_peered":"2023-11-28T20:55:15.402733+0000","last_clean":"2023-11-28T20:55:15.402733+0000","last_became_active":"2023-11-28T20:55:13.321109+0000","last_became_peered":"2023-11-28T20:55:13.321109+0000","last_unstale":"2023-11-28T20:55:15.402733+0000","last_undegraded":"2023-11-28T20:55:15.402733+0000","last_fullsized":"2023-11-28T20:55:15.402733+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:46:15.351870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402018+0000","last_change":"2023-11-28T20:55:13.323476+0000","last_active":"2023-11-28T20:55:15.402018+0000","last_peered":"2023-11-28T20:55:15.402018+0000","last_clean":"2023-11-28T20:55:15.402018+0000","last_became_active":"2023-11-28T20:55:13.323147+0000","last_became_peered":"2023-11-28T20:55:13.323147+0000","last_unstale":"2023-11-28T20:55:15.402018+0000","last_undegraded":"2023-11-28T20:55:15.402018+0000","last_fullsized":"2023-11-28T20:55:15.402018+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:51:05.171617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.310431+0000","last_change":"2023-11-28T20:55:13.312033+0000","last_active":"2023-11-28T20:55:14.310431+0000","last_peered":"2023-11-28T20:55:14.310431+0000","last_clean":"2023-11-28T20:55:14.310431+0000","last_became_active":"2023-11-28T20:55:13.304258+0000","last_became_peered":"2023-11-28T20:55:13.304258+0000","last_unstale":"2023-11-28T20:55:14.310431+0000","last_undegraded":"2023-11-28T20:55:14.310431+0000","last_fullsized":"2023-11-28T20:55:14.310431+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:06.888471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399286+0000","last_change":"2023-11-28T20:55:13.298552+0000","last_active":"2023-11-28T20:55:15.399286+0000","last_peered":"2023-11-28T20:55:15.399286+0000","last_clean":"2023-11-28T20:55:15.399286+0000","last_became_active":"2023-11-28T20:55:13.298232+0000","last_became_peered":"2023-11-28T20:55:13.298232+0000","last_unstale":"2023-11-28T20:55:15.399286+0000","last_undegraded":"2023-11-28T20:55:15.399286+0000","last_fullsized":"2023-11-28T20:55:15.399286+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:23.123418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309977+0000","last_change":"2023-11-28T20:55:13.314298+0000","last_active":"2023-11-28T20:55:14.309977+0000","last_peered":"2023-11-28T20:55:14.309977+0000","last_clean":"2023-11-28T20:55:14.309977+0000","last_became_active":"2023-11-28T20:55:13.312522+0000","last_became_peered":"2023-11-28T20:55:13.312522+0000","last_unstale":"2023-11-28T20:55:14.309977+0000","last_undegraded":"2023-11-28T20:55:14.309977+0000","last_fullsized":"2023-11-28T20:55:14.309977+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:49:27.579201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"160'4","reported_seq":16,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402626+0000","last_change":"2023-11-28T20:55:12.289802+0000","last_active":"2023-11-28T20:55:15.402626+0000","last_peered":"2023-11-28T20:55:15.402626+0000","last_clean":"2023-11-28T20:55:15.402626+0000","last_became_active":"2023-11-28T20:55:12.287907+0000","last_became_peered":"2023-11-28T20:55:12.287907+0000","last_unstale":"2023-11-28T20:55:15.402626+0000","last_undegraded":"2023-11-28T20:55:15.402626+0000","last_fullsized":"2023-11-28T20:55:15.402626+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:01.537227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305413+0000","last_change":"2023-11-28T20:55:13.312677+0000","last_active":"2023-11-28T20:55:14.305413+0000","last_peered":"2023-11-28T20:55:14.305413+0000","last_clean":"2023-11-28T20:55:14.305413+0000","last_became_active":"2023-11-28T20:55:13.312348+0000","last_became_peered":"2023-11-28T20:55:13.312348+0000","last_unstale":"2023-11-28T20:55:14.305413+0000","last_undegraded":"2023-11-28T20:55:14.305413+0000","last_fullsized":"2023-11-28T20:55:14.305413+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:16.776349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401165+0000","last_change":"2023-11-28T20:55:12.288018+0000","last_active":"2023-11-28T20:55:15.401165+0000","last_peered":"2023-11-28T20:55:15.401165+0000","last_clean":"2023-11-28T20:55:15.401165+0000","last_became_active":"2023-11-28T20:55:12.287625+0000","last_became_peered":"2023-11-28T20:55:12.287625+0000","last_unstale":"2023-11-28T20:55:15.401165+0000","last_undegraded":"2023-11-28T20:55:15.401165+0000","last_fullsized":"2023-11-28T20:55:15.401165+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:03.522881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309634+0000","last_change":"2023-11-28T20:55:13.311668+0000","last_active":"2023-11-28T20:55:14.309634+0000","last_peered":"2023-11-28T20:55:14.309634+0000","last_clean":"2023-11-28T20:55:14.309634+0000","last_became_active":"2023-11-28T20:55:13.306380+0000","last_became_peered":"2023-11-28T20:55:13.306380+0000","last_unstale":"2023-11-28T20:55:14.309634+0000","last_undegraded":"2023-11-28T20:55:14.309634+0000","last_fullsized":"2023-11-28T20:55:14.309634+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:00.545871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400935+0000","last_change":"2023-11-28T20:55:12.293452+0000","last_active":"2023-11-28T20:55:15.400935+0000","last_peered":"2023-11-28T20:55:15.400935+0000","last_clean":"2023-11-28T20:55:15.400935+0000","last_became_active":"2023-11-28T20:55:12.293148+0000","last_became_peered":"2023-11-28T20:55:12.293148+0000","last_unstale":"2023-11-28T20:55:15.400935+0000","last_undegraded":"2023-11-28T20:55:15.400935+0000","last_fullsized":"2023-11-28T20:55:15.400935+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:41:42.258629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305744+0000","last_change":"2023-11-28T20:55:13.314199+0000","last_active":"2023-11-28T20:55:14.305744+0000","last_peered":"2023-11-28T20:55:14.305744+0000","last_clean":"2023-11-28T20:55:14.305744+0000","last_became_active":"2023-11-28T20:55:13.313854+0000","last_became_peered":"2023-11-28T20:55:13.313854+0000","last_unstale":"2023-11-28T20:55:14.305744+0000","last_undegraded":"2023-11-28T20:55:14.305744+0000","last_fullsized":"2023-11-28T20:55:14.305744+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:38:44.734529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400289+0000","last_change":"2023-11-28T20:55:12.295887+0000","last_active":"2023-11-28T20:55:15.400289+0000","last_peered":"2023-11-28T20:55:15.400289+0000","last_clean":"2023-11-28T20:55:15.400289+0000","last_became_active":"2023-11-28T20:55:12.295365+0000","last_became_peered":"2023-11-28T20:55:12.295365+0000","last_unstale":"2023-11-28T20:55:15.400289+0000","last_undegraded":"2023-11-28T20:55:15.400289+0000","last_fullsized":"2023-11-28T20:55:15.400289+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:40:32.099877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312153+0000","last_change":"2023-11-28T20:55:13.311985+0000","last_active":"2023-11-28T20:55:14.312153+0000","last_peered":"2023-11-28T20:55:14.312153+0000","last_clean":"2023-11-28T20:55:14.312153+0000","last_became_active":"2023-11-28T20:55:13.305170+0000","last_became_peered":"2023-11-28T20:55:13.305170+0000","last_unstale":"2023-11-28T20:55:14.312153+0000","last_undegraded":"2023-11-28T20:55:14.312153+0000","last_fullsized":"2023-11-28T20:55:14.312153+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:07:18.524250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.311676+0000","last_change":"2023-11-28T20:55:12.281457+0000","last_active":"2023-11-28T20:55:14.311676+0000","last_peered":"2023-11-28T20:55:14.311676+0000","last_clean":"2023-11-28T20:55:14.311676+0000","last_became_active":"2023-11-28T20:55:12.279839+0000","last_became_peered":"2023-11-28T20:55:12.279839+0000","last_unstale":"2023-11-28T20:55:14.311676+0000","last_undegraded":"2023-11-28T20:55:14.311676+0000","last_fullsized":"2023-11-28T20:55:14.311676+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:28:01.537227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312557+0000","last_change":"2023-11-28T20:55:13.302944+0000","last_active":"2023-11-28T20:55:14.312557+0000","last_peered":"2023-11-28T20:55:14.312557+0000","last_clean":"2023-11-28T20:55:14.312557+0000","last_became_active":"2023-11-28T20:55:13.302042+0000","last_became_peered":"2023-11-28T20:55:13.302042+0000","last_unstale":"2023-11-28T20:55:14.312557+0000","last_undegraded":"2023-11-28T20:55:14.312557+0000","last_fullsized":"2023-11-28T20:55:14.312557+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:05:00.230098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399476+0000","last_change":"2023-11-28T20:55:12.280094+0000","last_active":"2023-11-28T20:55:15.399476+0000","last_peered":"2023-11-28T20:55:15.399476+0000","last_clean":"2023-11-28T20:55:15.399476+0000","last_became_active":"2023-11-28T20:55:12.279687+0000","last_became_peered":"2023-11-28T20:55:12.279687+0000","last_unstale":"2023-11-28T20:55:15.399476+0000","last_undegraded":"2023-11-28T20:55:15.399476+0000","last_fullsized":"2023-11-28T20:55:15.399476+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:06:52.524129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400927+0000","last_change":"2023-11-28T20:55:13.315347+0000","last_active":"2023-11-28T20:55:15.400927+0000","last_peered":"2023-11-28T20:55:15.400927+0000","last_clean":"2023-11-28T20:55:15.400927+0000","last_became_active":"2023-11-28T20:55:13.311309+0000","last_became_peered":"2023-11-28T20:55:13.311309+0000","last_unstale":"2023-11-28T20:55:15.400927+0000","last_undegraded":"2023-11-28T20:55:15.400927+0000","last_fullsized":"2023-11-28T20:55:15.400927+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:26:38.540530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.898555+0000","last_change":"2023-11-28T20:55:12.284844+0000","last_active":"2023-11-28T20:55:14.898555+0000","last_peered":"2023-11-28T20:55:14.898555+0000","last_clean":"2023-11-28T20:55:14.898555+0000","last_became_active":"2023-11-28T20:55:12.284084+0000","last_became_peered":"2023-11-28T20:55:12.284084+0000","last_unstale":"2023-11-28T20:55:14.898555+0000","last_undegraded":"2023-11-28T20:55:14.898555+0000","last_fullsized":"2023-11-28T20:55:14.898555+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:17:46.357527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401081+0000","last_change":"2023-11-28T20:55:13.324981+0000","last_active":"2023-11-28T20:55:15.401081+0000","last_peered":"2023-11-28T20:55:15.401081+0000","last_clean":"2023-11-28T20:55:15.401081+0000","last_became_active":"2023-11-28T20:55:13.324660+0000","last_became_peered":"2023-11-28T20:55:13.324660+0000","last_unstale":"2023-11-28T20:55:15.401081+0000","last_undegraded":"2023-11-28T20:55:15.401081+0000","last_fullsized":"2023-11-28T20:55:15.401081+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:59:09.009449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401123+0000","last_change":"2023-11-28T20:55:12.290985+0000","last_active":"2023-11-28T20:55:15.401123+0000","last_peered":"2023-11-28T20:55:15.401123+0000","last_clean":"2023-11-28T20:55:15.401123+0000","last_became_active":"2023-11-28T20:55:12.290684+0000","last_became_peered":"2023-11-28T20:55:12.290684+0000","last_unstale":"2023-11-28T20:55:15.401123+0000","last_undegraded":"2023-11-28T20:55:15.401123+0000","last_fullsized":"2023-11-28T20:55:15.401123+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:37.725264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305287+0000","last_change":"2023-11-28T20:55:13.328921+0000","last_active":"2023-11-28T20:55:14.305287+0000","last_peered":"2023-11-28T20:55:14.305287+0000","last_clean":"2023-11-28T20:55:14.305287+0000","last_became_active":"2023-11-28T20:55:13.328010+0000","last_became_peered":"2023-11-28T20:55:13.328010+0000","last_unstale":"2023-11-28T20:55:14.305287+0000","last_undegraded":"2023-11-28T20:55:14.305287+0000","last_fullsized":"2023-11-28T20:55:14.305287+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:26:34.668104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312519+0000","last_change":"2023-11-28T20:55:12.290633+0000","last_active":"2023-11-28T20:55:14.312519+0000","last_peered":"2023-11-28T20:55:14.312519+0000","last_clean":"2023-11-28T20:55:14.312519+0000","last_became_active":"2023-11-28T20:55:12.290331+0000","last_became_peered":"2023-11-28T20:55:12.290331+0000","last_unstale":"2023-11-28T20:55:14.312519+0000","last_undegraded":"2023-11-28T20:55:14.312519+0000","last_fullsized":"2023-11-28T20:55:14.312519+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:47:32.663939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899627+0000","last_change":"2023-11-28T20:55:13.300749+0000","last_active":"2023-11-28T20:55:14.899627+0000","last_peered":"2023-11-28T20:55:14.899627+0000","last_clean":"2023-11-28T20:55:14.899627+0000","last_became_active":"2023-11-28T20:55:13.300453+0000","last_became_peered":"2023-11-28T20:55:13.300453+0000","last_unstale":"2023-11-28T20:55:14.899627+0000","last_undegraded":"2023-11-28T20:55:14.899627+0000","last_fullsized":"2023-11-28T20:55:14.899627+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:48:12.175159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306130+0000","last_change":"2023-11-28T20:55:12.276031+0000","last_active":"2023-11-28T20:55:14.306130+0000","last_peered":"2023-11-28T20:55:14.306130+0000","last_clean":"2023-11-28T20:55:14.306130+0000","last_became_active":"2023-11-28T20:55:12.275740+0000","last_became_peered":"2023-11-28T20:55:12.275740+0000","last_unstale":"2023-11-28T20:55:14.306130+0000","last_undegraded":"2023-11-28T20:55:14.306130+0000","last_fullsized":"2023-11-28T20:55:14.306130+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:43.026055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402345+0000","last_change":"2023-11-28T20:55:13.323283+0000","last_active":"2023-11-28T20:55:15.402345+0000","last_peered":"2023-11-28T20:55:15.402345+0000","last_clean":"2023-11-28T20:55:15.402345+0000","last_became_active":"2023-11-28T20:55:13.322914+0000","last_became_peered":"2023-11-28T20:55:13.322914+0000","last_unstale":"2023-11-28T20:55:15.402345+0000","last_undegraded":"2023-11-28T20:55:15.402345+0000","last_fullsized":"2023-11-28T20:55:15.402345+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:49:27.579201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"160'3","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401149+0000","last_change":"2023-11-28T20:55:12.294093+0000","last_active":"2023-11-28T20:55:15.401149+0000","last_peered":"2023-11-28T20:55:15.401149+0000","last_clean":"2023-11-28T20:55:15.401149+0000","last_became_active":"2023-11-28T20:55:12.293735+0000","last_became_peered":"2023-11-28T20:55:12.293735+0000","last_unstale":"2023-11-28T20:55:15.401149+0000","last_undegraded":"2023-11-28T20:55:15.401149+0000","last_fullsized":"2023-11-28T20:55:15.401149+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:51.235329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402133+0000","last_change":"2023-11-28T20:55:13.324237+0000","last_active":"2023-11-28T20:55:15.402133+0000","last_peered":"2023-11-28T20:55:15.402133+0000","last_clean":"2023-11-28T20:55:15.402133+0000","last_became_active":"2023-11-28T20:55:13.323963+0000","last_became_peered":"2023-11-28T20:55:13.323963+0000","last_unstale":"2023-11-28T20:55:15.402133+0000","last_undegraded":"2023-11-28T20:55:15.402133+0000","last_fullsized":"2023-11-28T20:55:15.402133+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:09.114299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"160'6","reported_seq":18,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400257+0000","last_change":"2023-11-28T20:55:12.293276+0000","last_active":"2023-11-28T20:55:15.400257+0000","last_peered":"2023-11-28T20:55:15.400257+0000","last_clean":"2023-11-28T20:55:15.400257+0000","last_became_active":"2023-11-28T20:55:12.291148+0000","last_became_peered":"2023-11-28T20:55:12.291148+0000","last_unstale":"2023-11-28T20:55:15.400257+0000","last_undegraded":"2023-11-28T20:55:15.400257+0000","last_fullsized":"2023-11-28T20:55:15.400257+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:06.359290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401256+0000","last_change":"2023-11-28T20:55:13.305345+0000","last_active":"2023-11-28T20:55:15.401256+0000","last_peered":"2023-11-28T20:55:15.401256+0000","last_clean":"2023-11-28T20:55:15.401256+0000","last_became_active":"2023-11-28T20:55:13.304977+0000","last_became_peered":"2023-11-28T20:55:13.304977+0000","last_unstale":"2023-11-28T20:55:15.401256+0000","last_undegraded":"2023-11-28T20:55:15.401256+0000","last_fullsized":"2023-11-28T20:55:15.401256+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:53:48.664836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305705+0000","last_change":"2023-11-28T20:55:12.277643+0000","last_active":"2023-11-28T20:55:14.305705+0000","last_peered":"2023-11-28T20:55:14.305705+0000","last_clean":"2023-11-28T20:55:14.305705+0000","last_became_active":"2023-11-28T20:55:12.277095+0000","last_became_peered":"2023-11-28T20:55:12.277095+0000","last_unstale":"2023-11-28T20:55:14.305705+0000","last_undegraded":"2023-11-28T20:55:14.305705+0000","last_fullsized":"2023-11-28T20:55:14.305705+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:21.830506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401532+0000","last_change":"2023-11-28T20:55:13.330971+0000","last_active":"2023-11-28T20:55:15.401532+0000","last_peered":"2023-11-28T20:55:15.401532+0000","last_clean":"2023-11-28T20:55:15.401532+0000","last_became_active":"2023-11-28T20:55:13.329615+0000","last_became_peered":"2023-11-28T20:55:13.329615+0000","last_unstale":"2023-11-28T20:55:15.401532+0000","last_undegraded":"2023-11-28T20:55:15.401532+0000","last_fullsized":"2023-11-28T20:55:15.401532+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:26:38.540530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401503+0000","last_change":"2023-11-28T20:55:12.290913+0000","last_active":"2023-11-28T20:55:15.401503+0000","last_peered":"2023-11-28T20:55:15.401503+0000","last_clean":"2023-11-28T20:55:15.401503+0000","last_became_active":"2023-11-28T20:55:12.290566+0000","last_became_peered":"2023-11-28T20:55:12.290566+0000","last_unstale":"2023-11-28T20:55:15.401503+0000","last_undegraded":"2023-11-28T20:55:15.401503+0000","last_fullsized":"2023-11-28T20:55:15.401503+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:53:36.404925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401633+0000","last_change":"2023-11-28T20:55:13.315988+0000","last_active":"2023-11-28T20:55:15.401633+0000","last_peered":"2023-11-28T20:55:15.401633+0000","last_clean":"2023-11-28T20:55:15.401633+0000","last_became_active":"2023-11-28T20:55:13.311743+0000","last_became_peered":"2023-11-28T20:55:13.311743+0000","last_unstale":"2023-11-28T20:55:15.401633+0000","last_undegraded":"2023-11-28T20:55:15.401633+0000","last_fullsized":"2023-11-28T20:55:15.401633+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:18:29.065010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"160'8","reported_seq":20,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899551+0000","last_change":"2023-11-28T20:55:12.291018+0000","last_active":"2023-11-28T20:55:14.899551+0000","last_peered":"2023-11-28T20:55:14.899551+0000","last_clean":"2023-11-28T20:55:14.899551+0000","last_became_active":"2023-11-28T20:55:12.290419+0000","last_became_peered":"2023-11-28T20:55:12.290419+0000","last_unstale":"2023-11-28T20:55:14.899551+0000","last_undegraded":"2023-11-28T20:55:14.899551+0000","last_fullsized":"2023-11-28T20:55:14.899551+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:12.788384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400905+0000","last_change":"2023-11-28T20:55:13.325708+0000","last_active":"2023-11-28T20:55:15.400905+0000","last_peered":"2023-11-28T20:55:15.400905+0000","last_clean":"2023-11-28T20:55:15.400905+0000","last_became_active":"2023-11-28T20:55:13.325417+0000","last_became_peered":"2023-11-28T20:55:13.325417+0000","last_unstale":"2023-11-28T20:55:15.400905+0000","last_undegraded":"2023-11-28T20:55:15.400905+0000","last_fullsized":"2023-11-28T20:55:15.400905+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:13.803544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312624+0000","last_change":"2023-11-28T20:55:12.290412+0000","last_active":"2023-11-28T20:55:14.312624+0000","last_peered":"2023-11-28T20:55:14.312624+0000","last_clean":"2023-11-28T20:55:14.312624+0000","last_became_active":"2023-11-28T20:55:12.290021+0000","last_became_peered":"2023-11-28T20:55:12.290021+0000","last_unstale":"2023-11-28T20:55:14.312624+0000","last_undegraded":"2023-11-28T20:55:14.312624+0000","last_fullsized":"2023-11-28T20:55:14.312624+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:51.365142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399958+0000","last_change":"2023-11-28T20:55:13.328730+0000","last_active":"2023-11-28T20:55:15.399958+0000","last_peered":"2023-11-28T20:55:15.399958+0000","last_clean":"2023-11-28T20:55:15.399958+0000","last_became_active":"2023-11-28T20:55:13.328133+0000","last_became_peered":"2023-11-28T20:55:13.328133+0000","last_unstale":"2023-11-28T20:55:15.399958+0000","last_undegraded":"2023-11-28T20:55:15.399958+0000","last_fullsized":"2023-11-28T20:55:15.399958+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:52:50.218429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402439+0000","last_change":"2023-11-28T20:55:12.286254+0000","last_active":"2023-11-28T20:55:15.402439+0000","last_peered":"2023-11-28T20:55:15.402439+0000","last_clean":"2023-11-28T20:55:15.402439+0000","last_became_active":"2023-11-28T20:55:12.285891+0000","last_became_peered":"2023-11-28T20:55:12.285891+0000","last_unstale":"2023-11-28T20:55:15.402439+0000","last_undegraded":"2023-11-28T20:55:15.402439+0000","last_fullsized":"2023-11-28T20:55:15.402439+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:46:24.622410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400309+0000","last_change":"2023-11-28T20:55:13.320997+0000","last_active":"2023-11-28T20:55:15.400309+0000","last_peered":"2023-11-28T20:55:15.400309+0000","last_clean":"2023-11-28T20:55:15.400309+0000","last_became_active":"2023-11-28T20:55:13.320712+0000","last_became_peered":"2023-11-28T20:55:13.320712+0000","last_unstale":"2023-11-28T20:55:15.400309+0000","last_undegraded":"2023-11-28T20:55:15.400309+0000","last_fullsized":"2023-11-28T20:55:15.400309+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:53.890830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"160'1","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899877+0000","last_change":"2023-11-28T20:55:12.282974+0000","last_active":"2023-11-28T20:55:14.899877+0000","last_peered":"2023-11-28T20:55:14.899877+0000","last_clean":"2023-11-28T20:55:14.899877+0000","last_became_active":"2023-11-28T20:55:12.282702+0000","last_became_peered":"2023-11-28T20:55:12.282702+0000","last_unstale":"2023-11-28T20:55:14.899877+0000","last_undegraded":"2023-11-28T20:55:14.899877+0000","last_fullsized":"2023-11-28T20:55:14.899877+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:49:26.564041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401040+0000","last_change":"2023-11-28T20:55:13.323527+0000","last_active":"2023-11-28T20:55:15.401040+0000","last_peered":"2023-11-28T20:55:15.401040+0000","last_clean":"2023-11-28T20:55:15.401040+0000","last_became_active":"2023-11-28T20:55:13.323209+0000","last_became_peered":"2023-11-28T20:55:13.323209+0000","last_unstale":"2023-11-28T20:55:15.401040+0000","last_undegraded":"2023-11-28T20:55:15.401040+0000","last_fullsized":"2023-11-28T20:55:15.401040+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:00:14.212039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400298+0000","last_change":"2023-11-28T20:55:12.276974+0000","last_active":"2023-11-28T20:55:15.400298+0000","last_peered":"2023-11-28T20:55:15.400298+0000","last_clean":"2023-11-28T20:55:15.400298+0000","last_became_active":"2023-11-28T20:55:12.276642+0000","last_became_peered":"2023-11-28T20:55:12.276642+0000","last_unstale":"2023-11-28T20:55:15.400298+0000","last_undegraded":"2023-11-28T20:55:15.400298+0000","last_fullsized":"2023-11-28T20:55:15.400298+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:52:49.203269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400131+0000","last_change":"2023-11-28T20:55:13.323863+0000","last_active":"2023-11-28T20:55:15.400131+0000","last_peered":"2023-11-28T20:55:15.400131+0000","last_clean":"2023-11-28T20:55:15.400131+0000","last_became_active":"2023-11-28T20:55:13.322679+0000","last_became_peered":"2023-11-28T20:55:13.322679+0000","last_unstale":"2023-11-28T20:55:15.400131+0000","last_undegraded":"2023-11-28T20:55:15.400131+0000","last_fullsized":"2023-11-28T20:55:15.400131+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:39.018087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.311585+0000","last_change":"2023-11-28T20:55:12.289623+0000","last_active":"2023-11-28T20:55:14.311585+0000","last_peered":"2023-11-28T20:55:14.311585+0000","last_clean":"2023-11-28T20:55:14.311585+0000","last_became_active":"2023-11-28T20:55:12.287473+0000","last_became_peered":"2023-11-28T20:55:12.287473+0000","last_unstale":"2023-11-28T20:55:14.311585+0000","last_undegraded":"2023-11-28T20:55:14.311585+0000","last_fullsized":"2023-11-28T20:55:14.311585+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:34:12.788384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306038+0000","last_change":"2023-11-28T20:55:13.316679+0000","last_active":"2023-11-28T20:55:14.306038+0000","last_peered":"2023-11-28T20:55:14.306038+0000","last_clean":"2023-11-28T20:55:14.306038+0000","last_became_active":"2023-11-28T20:55:13.312502+0000","last_became_peered":"2023-11-28T20:55:13.312502+0000","last_unstale":"2023-11-28T20:55:14.306038+0000","last_undegraded":"2023-11-28T20:55:14.306038+0000","last_fullsized":"2023-11-28T20:55:14.306038+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:50:04.945656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.305594+0000","last_change":"2023-11-28T20:55:12.282617+0000","last_active":"2023-11-28T20:55:14.305594+0000","last_peered":"2023-11-28T20:55:14.305594+0000","last_clean":"2023-11-28T20:55:14.305594+0000","last_became_active":"2023-11-28T20:55:12.282139+0000","last_became_peered":"2023-11-28T20:55:12.282139+0000","last_unstale":"2023-11-28T20:55:14.305594+0000","last_undegraded":"2023-11-28T20:55:14.305594+0000","last_fullsized":"2023-11-28T20:55:14.305594+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:25:15.465465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401209+0000","last_change":"2023-11-28T20:55:13.322608+0000","last_active":"2023-11-28T20:55:15.401209+0000","last_peered":"2023-11-28T20:55:15.401209+0000","last_clean":"2023-11-28T20:55:15.401209+0000","last_became_active":"2023-11-28T20:55:13.322295+0000","last_became_peered":"2023-11-28T20:55:13.322295+0000","last_unstale":"2023-11-28T20:55:15.401209+0000","last_undegraded":"2023-11-28T20:55:15.401209+0000","last_fullsized":"2023-11-28T20:55:15.401209+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:28.969319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400036+0000","last_change":"2023-11-28T20:55:12.293622+0000","last_active":"2023-11-28T20:55:15.400036+0000","last_peered":"2023-11-28T20:55:15.400036+0000","last_clean":"2023-11-28T20:55:15.400036+0000","last_became_active":"2023-11-28T20:55:12.290115+0000","last_became_peered":"2023-11-28T20:55:12.290115+0000","last_unstale":"2023-11-28T20:55:15.400036+0000","last_undegraded":"2023-11-28T20:55:15.400036+0000","last_fullsized":"2023-11-28T20:55:15.400036+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:20.707314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312165+0000","last_change":"2023-11-28T20:55:13.321541+0000","last_active":"2023-11-28T20:55:14.312165+0000","last_peered":"2023-11-28T20:55:14.312165+0000","last_clean":"2023-11-28T20:55:14.312165+0000","last_became_active":"2023-11-28T20:55:13.321253+0000","last_became_peered":"2023-11-28T20:55:13.321253+0000","last_unstale":"2023-11-28T20:55:14.312165+0000","last_undegraded":"2023-11-28T20:55:14.312165+0000","last_fullsized":"2023-11-28T20:55:14.312165+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:14:59.121157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899021+0000","last_change":"2023-11-28T20:55:12.283388+0000","last_active":"2023-11-28T20:55:14.899021+0000","last_peered":"2023-11-28T20:55:14.899021+0000","last_clean":"2023-11-28T20:55:14.899021+0000","last_became_active":"2023-11-28T20:55:12.281875+0000","last_became_peered":"2023-11-28T20:55:12.281875+0000","last_unstale":"2023-11-28T20:55:14.899021+0000","last_undegraded":"2023-11-28T20:55:14.899021+0000","last_fullsized":"2023-11-28T20:55:14.899021+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:03:08.099139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400322+0000","last_change":"2023-11-28T20:55:13.318979+0000","last_active":"2023-11-28T20:55:15.400322+0000","last_peered":"2023-11-28T20:55:15.400322+0000","last_clean":"2023-11-28T20:55:15.400322+0000","last_became_active":"2023-11-28T20:55:13.317840+0000","last_became_peered":"2023-11-28T20:55:13.317840+0000","last_unstale":"2023-11-28T20:55:15.400322+0000","last_undegraded":"2023-11-28T20:55:15.400322+0000","last_fullsized":"2023-11-28T20:55:15.400322+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:59:09.009449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"160'2","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.306149+0000","last_change":"2023-11-28T20:55:12.290074+0000","last_active":"2023-11-28T20:55:14.306149+0000","last_peered":"2023-11-28T20:55:14.306149+0000","last_clean":"2023-11-28T20:55:14.306149+0000","last_became_active":"2023-11-28T20:55:12.289628+0000","last_became_peered":"2023-11-28T20:55:12.289628+0000","last_unstale":"2023-11-28T20:55:14.306149+0000","last_undegraded":"2023-11-28T20:55:14.306149+0000","last_fullsized":"2023-11-28T20:55:14.306149+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:53:14.603837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400917+0000","last_change":"2023-11-28T20:55:13.317542+0000","last_active":"2023-11-28T20:55:15.400917+0000","last_peered":"2023-11-28T20:55:15.400917+0000","last_clean":"2023-11-28T20:55:15.400917+0000","last_became_active":"2023-11-28T20:55:13.317210+0000","last_became_peered":"2023-11-28T20:55:13.317210+0000","last_unstale":"2023-11-28T20:55:15.400917+0000","last_undegraded":"2023-11-28T20:55:15.400917+0000","last_fullsized":"2023-11-28T20:55:15.400917+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:59.166427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.313180+0000","last_change":"2023-11-28T20:55:12.291469+0000","last_active":"2023-11-28T20:55:14.313180+0000","last_peered":"2023-11-28T20:55:14.313180+0000","last_clean":"2023-11-28T20:55:14.313180+0000","last_became_active":"2023-11-28T20:55:12.291109+0000","last_became_peered":"2023-11-28T20:55:12.291109+0000","last_unstale":"2023-11-28T20:55:14.313180+0000","last_undegraded":"2023-11-28T20:55:14.313180+0000","last_fullsized":"2023-11-28T20:55:14.313180+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:35:12.838952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.399958+0000","last_change":"2023-11-28T20:55:13.327645+0000","last_active":"2023-11-28T20:55:15.399958+0000","last_peered":"2023-11-28T20:55:15.399958+0000","last_clean":"2023-11-28T20:55:15.399958+0000","last_became_active":"2023-11-28T20:55:13.327353+0000","last_became_peered":"2023-11-28T20:55:13.327353+0000","last_unstale":"2023-11-28T20:55:15.399958+0000","last_undegraded":"2023-11-28T20:55:15.399958+0000","last_fullsized":"2023-11-28T20:55:15.399958+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:14:04.538041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.312110+0000","last_change":"2023-11-28T20:55:12.278227+0000","last_active":"2023-11-28T20:55:14.312110+0000","last_peered":"2023-11-28T20:55:14.312110+0000","last_clean":"2023-11-28T20:55:14.312110+0000","last_became_active":"2023-11-28T20:55:12.277831+0000","last_became_peered":"2023-11-28T20:55:12.277831+0000","last_unstale":"2023-11-28T20:55:14.312110+0000","last_undegraded":"2023-11-28T20:55:14.312110+0000","last_fullsized":"2023-11-28T20:55:14.312110+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:46:24.622410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401042+0000","last_change":"2023-11-28T20:55:13.305664+0000","last_active":"2023-11-28T20:55:15.401042+0000","last_peered":"2023-11-28T20:55:15.401042+0000","last_clean":"2023-11-28T20:55:15.401042+0000","last_became_active":"2023-11-28T20:55:13.305383+0000","last_became_peered":"2023-11-28T20:55:13.305383+0000","last_unstale":"2023-11-28T20:55:15.401042+0000","last_undegraded":"2023-11-28T20:55:15.401042+0000","last_fullsized":"2023-11-28T20:55:15.401042+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:26:04.420246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899397+0000","last_change":"2023-11-28T20:55:12.285675+0000","last_active":"2023-11-28T20:55:14.899397+0000","last_peered":"2023-11-28T20:55:14.899397+0000","last_clean":"2023-11-28T20:55:14.899397+0000","last_became_active":"2023-11-28T20:55:12.285034+0000","last_became_peered":"2023-11-28T20:55:12.285034+0000","last_unstale":"2023-11-28T20:55:14.899397+0000","last_undegraded":"2023-11-28T20:55:14.899397+0000","last_fullsized":"2023-11-28T20:55:14.899397+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:05.873311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899484+0000","last_change":"2023-11-28T20:55:13.309965+0000","last_active":"2023-11-28T20:55:14.899484+0000","last_peered":"2023-11-28T20:55:14.899484+0000","last_clean":"2023-11-28T20:55:14.899484+0000","last_became_active":"2023-11-28T20:55:13.309654+0000","last_became_peered":"2023-11-28T20:55:13.309654+0000","last_unstale":"2023-11-28T20:55:14.899484+0000","last_undegraded":"2023-11-28T20:55:14.899484+0000","last_fullsized":"2023-11-28T20:55:14.899484+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:37:17.347224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.309988+0000","last_change":"2023-11-28T20:55:12.288039+0000","last_active":"2023-11-28T20:55:14.309988+0000","last_peered":"2023-11-28T20:55:14.309988+0000","last_clean":"2023-11-28T20:55:14.309988+0000","last_became_active":"2023-11-28T20:55:12.287739+0000","last_became_peered":"2023-11-28T20:55:12.287739+0000","last_unstale":"2023-11-28T20:55:14.309988+0000","last_undegraded":"2023-11-28T20:55:14.309988+0000","last_fullsized":"2023-11-28T20:55:14.309988+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:56:51.993515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899760+0000","last_change":"2023-11-28T20:55:13.310162+0000","last_active":"2023-11-28T20:55:14.899760+0000","last_peered":"2023-11-28T20:55:14.899760+0000","last_clean":"2023-11-28T20:55:14.899760+0000","last_became_active":"2023-11-28T20:55:13.309881+0000","last_became_peered":"2023-11-28T20:55:13.309881+0000","last_unstale":"2023-11-28T20:55:14.899760+0000","last_undegraded":"2023-11-28T20:55:14.899760+0000","last_fullsized":"2023-11-28T20:55:14.899760+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:08:10.115481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400548+0000","last_change":"2023-11-28T20:55:12.295683+0000","last_active":"2023-11-28T20:55:15.400548+0000","last_peered":"2023-11-28T20:55:15.400548+0000","last_clean":"2023-11-28T20:55:15.400548+0000","last_became_active":"2023-11-28T20:55:12.295338+0000","last_became_peered":"2023-11-28T20:55:12.295338+0000","last_unstale":"2023-11-28T20:55:15.400548+0000","last_undegraded":"2023-11-28T20:55:15.400548+0000","last_fullsized":"2023-11-28T20:55:15.400548+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:14:58.105997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.899963+0000","last_change":"2023-11-28T20:55:13.314675+0000","last_active":"2023-11-28T20:55:14.899963+0000","last_peered":"2023-11-28T20:55:14.899963+0000","last_clean":"2023-11-28T20:55:14.899963+0000","last_became_active":"2023-11-28T20:55:13.313564+0000","last_became_peered":"2023-11-28T20:55:13.313564+0000","last_unstale":"2023-11-28T20:55:14.899963+0000","last_undegraded":"2023-11-28T20:55:14.899963+0000","last_fullsized":"2023-11-28T20:55:14.899963+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:45.696174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:14.898916+0000","last_change":"2023-11-28T20:55:12.285073+0000","last_active":"2023-11-28T20:55:14.898916+0000","last_peered":"2023-11-28T20:55:14.898916+0000","last_clean":"2023-11-28T20:55:14.898916+0000","last_became_active":"2023-11-28T20:55:12.284386+0000","last_became_peered":"2023-11-28T20:55:12.284386+0000","last_unstale":"2023-11-28T20:55:14.898916+0000","last_undegraded":"2023-11-28T20:55:14.898916+0000","last_fullsized":"2023-11-28T20:55:14.898916+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:46:24.622410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.401308+0000","last_change":"2023-11-28T20:55:13.330287+0000","last_active":"2023-11-28T20:55:15.401308+0000","last_peered":"2023-11-28T20:55:15.401308+0000","last_clean":"2023-11-28T20:55:15.401308+0000","last_became_active":"2023-11-28T20:55:13.329974+0000","last_became_peered":"2023-11-28T20:55:13.329974+0000","last_unstale":"2023-11-28T20:55:15.401308+0000","last_undegraded":"2023-11-28T20:55:15.401308+0000","last_fullsized":"2023-11-28T20:55:15.401308+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:16:48.298816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400603+0000","last_change":"2023-11-28T20:55:12.290156+0000","last_active":"2023-11-28T20:55:15.400603+0000","last_peered":"2023-11-28T20:55:15.400603+0000","last_clean":"2023-11-28T20:55:15.400603+0000","last_became_active":"2023-11-28T20:55:12.289837+0000","last_became_peered":"2023-11-28T20:55:12.289837+0000","last_unstale":"2023-11-28T20:55:15.400603+0000","last_undegraded":"2023-11-28T20:55:15.400603+0000","last_fullsized":"2023-11-28T20:55:15.400603+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:06:55.674296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.402724+0000","last_change":"2023-11-28T20:55:13.325752+0000","last_active":"2023-11-28T20:55:15.402724+0000","last_peered":"2023-11-28T20:55:15.402724+0000","last_clean":"2023-11-28T20:55:15.402724+0000","last_became_active":"2023-11-28T20:55:13.325075+0000","last_became_peered":"2023-11-28T20:55:13.325075+0000","last_unstale":"2023-11-28T20:55:15.402724+0000","last_undegraded":"2023-11-28T20:55:15.402724+0000","last_fullsized":"2023-11-28T20:55:15.402724+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:12.242481+0000","last_clean_scrub_stamp":"2023-11-28T20:55:12.242481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:44:41.501846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-11-28T20:55:15.400384+0000","last_change":"2023-11-28T20:55:12.292548+0000","last_active":"2023-11-28T20:55:15.400384+0000","last_peered":"2023-11-28T20:55:15.400384+0000","last_clean":"2023-11-28T20:55:15.400384+0000","last_became_active":"2023-11-28T20:55:12.292242+0000","last_became_peered":"2023-11-28T20:55:12.292242+0000","last_unstale":"2023-11-28T20:55:15.400384+0000","last_undegraded":"2023-11-28T20:55:15.400384+0000","last_fullsized":"2023-11-28T20:55:15.400384+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:55:11.227321+0000","last_clean_scrub_stamp":"2023-11-28T20:55:11.227321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:15:15.315609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"150'4756","reported_seq":5338,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-11-28T20:55:13.305743+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T20:55:13.305743+0000","last_peered":"2023-11-28T20:55:13.305743+0000","last_clean":"2023-11-28T20:55:13.305743+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T20:55:13.305743+0000","last_undegraded":"2023-11-28T20:55:13.305743+0000","last_fullsized":"2023-11-28T20:55:13.305743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1381,"num_read_kb":2803,"num_write":411,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1381,"num_read_kb":2803,"num_write":411,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4756,"ondisk_log_size":4756,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575099,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47620,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324220,"statfs":{"total":96636764160,"available":96588001280,"internally_reserved":0,"allocated":2605056,"data_stored":1604237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.167,"15min":1.1870000000000001},"min":{"1min":0.623,"5min":0.57099999999999995,"15min":0.45600000000000002},"max":{"1min":1.7490000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.089,"15min":1.1499999999999999},"min":{"1min":0.66900000000000004,"5min":0.64300000000000002,"15min":0.58299999999999996},"max":{"1min":1.5660000000000001,"5min":1.8,"15min":1.8},"last":0.77700000000000002}]},{"osd":1,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1579999999999999,"15min":1.179},"min":{"1min":0.64600000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5629999999999999,"5min":1.633,"15min":1.744},"last":1.329},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.228,"15min":1.252},"min":{"1min":0.71799999999999997,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5900000000000001,"5min":1.8540000000000001,"15min":1.889},"last":0.68000000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.272,"15min":1.2889999999999999},"min":{"1min":0.879,"5min":0.59699999999999998,"15min":0.58799999999999997},"max":{"1min":1.714,"5min":1.714,"15min":1.9359999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3280000000000001,"15min":1.298},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.57499999999999996},"max":{"1min":1.77,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.135}]},{"osd":3,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2470000000000001,"15min":1.27},"min":{"1min":0.76800000000000002,"5min":0.73199999999999998,"15min":0.54200000000000004},"max":{"1min":1.583,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4139999999999999,"15min":1.4099999999999999},"min":{"1min":1.0609999999999999,"5min":0.77000000000000002,"15min":0.68300000000000005},"max":{"1min":1.7869999999999999,"5min":1.956,"15min":1.956},"last":1.264}]},{"osd":4,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.335,"15min":1.29},"min":{"1min":0.89800000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0430000000000001,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":0.87},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.347,"15min":1.3320000000000001},"min":{"1min":1.1399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.877,"5min":1.99,"15min":1.9970000000000001},"last":1.2210000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.4390000000000001,"15min":1.4179999999999999},"min":{"1min":1.1919999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.891,"5min":1.891,"15min":2.048},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.496,"15min":1.474},"min":{"1min":1.288,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":0.95199999999999996}]},{"osd":6,"last update":"Tue Nov 28 20:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.581,"15min":1.5589999999999999},"min":{"1min":1,"5min":0.85799999999999998,"15min":0.69499999999999995},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.476,"15min":1.4830000000000001},"min":{"1min":1.0640000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7969999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.3879999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575099,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37312,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334528,"statfs":{"total":96636764160,"available":96598556672,"internally_reserved":0,"allocated":2015232,"data_stored":1019487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.0700000000000001,"15min":1.103},"min":{"1min":0.754,"5min":0.58099999999999996,"15min":0.53500000000000003},"max":{"1min":1.653,"5min":1.653,"15min":1.7350000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0720000000000001,"15min":1.123},"min":{"1min":0.67800000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4350000000000001,"5min":1.8069999999999999,"15min":1.8260000000000001},"last":0.46800000000000003}]},{"osd":1,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1950000000000001,"15min":1.2370000000000001},"min":{"1min":0.71399999999999997,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.6799999999999999,"5min":1.728,"15min":1.8069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.3080000000000001,"15min":1.349},"min":{"1min":0.89800000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.7869999999999999,"5min":1.7889999999999999,"15min":2.0169999999999999},"last":0.80400000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.2470000000000001,"15min":1.272},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.55,"5min":1.798,"15min":1.798},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3220000000000001,"15min":1.3520000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.732,"5min":1.837,"15min":1.851},"last":0.71399999999999997}]},{"osd":3,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.248,"15min":1.284},"min":{"1min":0.63300000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.73,"5min":1.8580000000000001,"15min":1.9299999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.139,"5min":1.232,"15min":1.268},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5740000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2529999999999999,"15min":1.2430000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":2.089},"last":0.5},{"interface":"front","average":{"1min":1.246,"5min":1.296,"15min":1.304},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.875,"5min":1.9179999999999999,"15min":2.2010000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.331,"15min":1.3260000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.581,"5min":1.879,"15min":1.97},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.417,"5min":1.494,"15min":1.5189999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.752,"5min":1.901,"15min":1.958},"last":0.747}]},{"osd":7,"last update":"Tue Nov 28 20:54:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.4239999999999999,"15min":1.4530000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.96,"5min":1.9790000000000001,"15min":2.3759999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.429,"15min":1.4470000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6950000000000001,"5min":1.9630000000000001,"15min":2.0739999999999998},"last":0.78400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575099,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51192,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320648,"statfs":{"total":96636764160,"available":96584343552,"internally_reserved":0,"allocated":2011136,"data_stored":1018837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1619999999999999,"15min":1.1639999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.696,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.0660000000000001,"15min":1.1439999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.441},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.25}]},{"osd":1,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1160000000000001,"15min":1.147},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.623,"5min":1.623,"15min":1.768},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.157,"5min":1.2190000000000001,"15min":1.2130000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6100000000000001,"5min":2.282,"15min":2.282},"last":0.80400000000000005}]},{"osd":2,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.302,"15min":1.2829999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.53200000000000003},"max":{"1min":1.893,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":0.999},{"interface":"front","average":{"1min":1.212,"5min":1.21,"15min":1.274},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.66600000000000004},"max":{"1min":1.6240000000000001,"5min":1.8220000000000001,"15min":1.958},"last":1.536}]},{"osd":3,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.286,"15min":1.3140000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.726,"5min":1.7290000000000001,"15min":1.9350000000000001},"last":1.456},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.2589999999999999,"15min":1.262},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.64000000000000001},"max":{"1min":1.6559999999999999,"5min":1.6819999999999999,"15min":1.7509999999999999},"last":1.165}]},{"osd":4,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3140000000000001,"15min":1.2989999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.67500000000000004},"max":{"1min":1.746,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.052},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4710000000000001,"15min":1.4379999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.8160000000000001,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":1.401}]},{"osd":6,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3380000000000001,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.5},"max":{"1min":1.966,"5min":1.966,"15min":2.093},"last":1.591},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.474,"15min":1.476},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.69499999999999995},"max":{"1min":1.9219999999999999,"5min":1.998,"15min":1.998},"last":1.339}]},{"osd":7,"last update":"Tue Nov 28 20:54:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.464,"15min":1.4330000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.742,"5min":3.319,"15min":3.319},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.474,"15min":1.4830000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.60499999999999998},"max":{"1min":1.764,"5min":1.9790000000000001,"15min":2.1829999999999998},"last":1.6499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575100,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37308,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334532,"statfs":{"total":96636764160,"available":96598560768,"internally_reserved":0,"allocated":2007040,"data_stored":1018815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0580000000000001,"15min":1.125},"min":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.825,"5min":1.825,"15min":1.879},"last":1.242},{"interface":"front","average":{"1min":0.995,"5min":1.0720000000000001,"15min":1.1080000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.3839999999999999,"5min":1.6850000000000001,"15min":1.9750000000000001},"last":0.97699999999999998}]},{"osd":1,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2110000000000001,"15min":1.286},"min":{"1min":0.68999999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.923},"last":1.155},{"interface":"front","average":{"1min":1.335,"5min":1.2689999999999999,"15min":1.343},"min":{"1min":0.72099999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7470000000000001,"5min":1.768,"15min":2.2679999999999998},"last":1.373}]},{"osd":2,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2010000000000001,"15min":1.2190000000000001},"min":{"1min":0.56899999999999995,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.7729999999999999,"5min":1.78,"15min":1.78},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2350000000000001,"15min":1.282},"min":{"1min":0.65000000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.704,"5min":1.704,"15min":2.6949999999999998},"last":1.466}]},{"osd":3,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.2230000000000001,"15min":1.2709999999999999},"min":{"1min":0.63,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.647,"5min":1.9630000000000001,"15min":2.157},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.3149999999999999,"15min":1.3580000000000001},"min":{"1min":0.68700000000000006,"5min":0.5,"15min":0.5},"max":{"1min":1.7509999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.694}]},{"osd":5,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.286,"15min":1.292},"min":{"1min":0.65500000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.849,"5min":1.9970000000000001,"15min":2.4279999999999999},"last":1.647},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.2909999999999999,"15min":1.298},"min":{"1min":0.746,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.784,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.4179999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.3979999999999999,"15min":1.395},"min":{"1min":0.96699999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.8899999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.76},{"interface":"front","average":{"1min":1.538,"5min":1.5289999999999999,"15min":1.5509999999999999},"min":{"1min":0.92400000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.9219999999999999,"5min":2.1349999999999998,"15min":3.7000000000000002},"last":1.5589999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:54:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.444,"15min":1.46},"min":{"1min":0.76200000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.9159999999999999,"5min":2.2490000000000001,"15min":2.4990000000000001},"last":1.52},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.522,"15min":1.518},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.0510000000000002,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":1.843}]}]},{"osd":3,"up_from":13,"seq":55834575100,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41476,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330364,"statfs":{"total":96636764160,"available":96594292736,"internally_reserved":0,"allocated":2023424,"data_stored":1020497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.85699999999999998,"15min":0.89400000000000002},"min":{"1min":0.47299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.383,"5min":1.6000000000000001,"15min":4.7210000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.92800000000000005,"15min":1.0229999999999999},"min":{"1min":0.39700000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.73,"5min":1.7929999999999999,"15min":2.0950000000000002},"last":1.1850000000000001}]},{"osd":1,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.92900000000000005,"15min":0.96099999999999997},"min":{"1min":0.495,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6739999999999999,"5min":1.9770000000000001,"15min":2.8460000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.071,"15min":1.1719999999999999},"min":{"1min":0.53100000000000003,"5min":0.379,"15min":0.379},"max":{"1min":1.4219999999999999,"5min":1.7190000000000001,"15min":3.7410000000000001},"last":0.98599999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1140000000000001,"15min":1.216},"min":{"1min":0.65900000000000003,"5min":0.53100000000000003,"15min":0.48499999999999999},"max":{"1min":1.5429999999999999,"5min":1.9019999999999999,"15min":3.444},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.04,"5min":1.2030000000000001,"15min":1.3029999999999999},"min":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.452,"5min":1.9530000000000001,"15min":3.7050000000000001},"last":1.4870000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2849999999999999,"15min":1.335},"min":{"1min":0.76000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8089999999999999,"5min":1.956,"15min":2.9420000000000002},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.28,"5min":1.3300000000000001,"15min":1.3700000000000001},"min":{"1min":0.91400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.8819999999999999,"5min":2.246,"15min":2.246},"last":1.429}]},{"osd":5,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3799999999999999,"15min":1.4339999999999999},"min":{"1min":0.88800000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6659999999999999,"5min":1.8959999999999999,"15min":3.7730000000000001},"last":1.27},{"interface":"front","average":{"1min":1.379,"5min":1.387,"15min":1.4330000000000001},"min":{"1min":0.81100000000000005,"5min":0.66800000000000004,"15min":0.53500000000000003},"max":{"1min":1.875,"5min":2.0110000000000001,"15min":3.1520000000000001},"last":1.3029999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.399,"15min":1.458},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.825,"5min":2.0630000000000002,"15min":2.3100000000000001},"last":1.032},{"interface":"front","average":{"1min":1.405,"5min":1.46,"15min":1.5509999999999999},"min":{"1min":0.97599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.679,"5min":2.0840000000000001,"15min":3.661},"last":1.375}]},{"osd":7,"last update":"Tue Nov 28 20:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.518},"min":{"1min":0.83199999999999996,"5min":0.73399999999999999,"15min":0.71499999999999997},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":3.266},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.478,"5min":1.522,"15min":1.575},"min":{"1min":0.996,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.7450000000000001,"5min":2.2080000000000002,"15min":2.3919999999999999},"last":1.2210000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575100,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63684,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308156,"statfs":{"total":96636764160,"available":96571551744,"internally_reserved":0,"allocated":2015232,"data_stored":1019431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.86199999999999999,"15min":0.90200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":1.2829999999999999,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.0049999999999999,"15min":1.052},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.387,"5min":4.8490000000000002,"15min":4.8490000000000002},"last":1.637}]},{"osd":1,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.96899999999999997,"15min":0.98499999999999999},"min":{"1min":0.498,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.458,"5min":1.8160000000000001,"15min":3.2599999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0309999999999999,"15min":1.1240000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.357,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.819}]},{"osd":3,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1160000000000001,"15min":1.161},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.589,"5min":4.9349999999999996,"15min":4.9349999999999996},"last":1.9379999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1970000000000001,"15min":1.282},"min":{"1min":0.64900000000000002,"5min":0.48499999999999999,"15min":0.48199999999999998},"max":{"1min":1.994,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":1.288}]},{"osd":4,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3440000000000001,"15min":1.3979999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.673,"5min":3.6499999999999999,"15min":3.6499999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.363,"5min":1.448,"15min":1.4350000000000001},"min":{"1min":0.88400000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8340000000000001,"5min":4.6479999999999997,"15min":4.6479999999999997},"last":2.1419999999999999}]},{"osd":5,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3819999999999999,"15min":1.413},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.923,"5min":2.492,"15min":2.492},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.429,"15min":1.4830000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6399999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.6899999999999999}]},{"osd":6,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4810000000000001,"15min":1.508},"min":{"1min":0.79200000000000004,"5min":0.747,"15min":0.747},"max":{"1min":2.153,"5min":4.5270000000000001,"15min":4.5270000000000001},"last":2.423},{"interface":"front","average":{"1min":1.355,"5min":1.5329999999999999,"15min":1.5660000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.82499999999999996},"max":{"1min":1.8560000000000001,"5min":5.3499999999999996,"15min":5.3499999999999996},"last":1.885}]},{"osd":7,"last update":"Tue Nov 28 20:54:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4650000000000001,"15min":1.524},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.0720000000000001,"5min":2.641,"15min":2.641},"last":2.0449999999999999},{"interface":"front","average":{"1min":1.484,"5min":1.548,"15min":1.585},"min":{"1min":0.81799999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.2770000000000001,"5min":4.7530000000000001,"15min":4.7530000000000001},"last":2.3079999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37304,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334536,"statfs":{"total":96636764160,"available":96598564864,"internally_reserved":0,"allocated":2011136,"data_stored":1018885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.90800000000000003,"15min":0.91100000000000003},"min":{"1min":0.38900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.466,"5min":1.8009999999999999,"15min":1.9259999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.98899999999999999,"15min":1.069},"min":{"1min":0.505,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.6120000000000001,"5min":2.028,"15min":2.028},"last":1.2250000000000001}]},{"osd":2,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0389999999999999,"15min":1.0740000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.371},"max":{"1min":1.3959999999999999,"5min":1.766,"15min":2.234},"last":1.2949999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.153,"15min":1.1910000000000001},"min":{"1min":0.628,"5min":0.53700000000000003,"15min":0.52800000000000002},"max":{"1min":1.4259999999999999,"5min":1.9199999999999999,"15min":2.895},"last":1.6699999999999999}]},{"osd":3,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.2110000000000001,"15min":1.2889999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.46700000000000003},"max":{"1min":1.5109999999999999,"5min":1.7949999999999999,"15min":2.0459999999999998},"last":1.361},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.0760000000000001,"15min":1.101},"min":{"1min":0.47299999999999998,"5min":0.45100000000000001,"15min":0.37},"max":{"1min":1.756,"5min":1.9790000000000001,"15min":2.3679999999999999},"last":1.0920000000000001}]},{"osd":4,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3080000000000001,"15min":1.339},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5629999999999999,"5min":1.8140000000000001,"15min":2.1150000000000002},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.383,"15min":1.391},"min":{"1min":0.77000000000000002,"5min":0.72399999999999998,"15min":0.71899999999999997},"max":{"1min":1.776,"5min":1.8819999999999999,"15min":2.6819999999999999},"last":1.518}]},{"osd":5,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.4299999999999999,"15min":1.4339999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.9650000000000001,"5min":2.0190000000000001,"15min":2.1840000000000002},"last":1.728},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.4450000000000001,"15min":1.482},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.9530000000000001,"5min":2.0219999999999998,"15min":2.294},"last":1.4910000000000001}]},{"osd":6,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.399,"15min":1.4179999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.66900000000000004},"max":{"1min":1.772,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.806},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.502,"15min":1.524},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.77100000000000002},"max":{"1min":1.9139999999999999,"5min":2.0049999999999999,"15min":2.3300000000000001},"last":1.9299999999999999}]},{"osd":7,"last update":"Tue Nov 28 20:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.51,"15min":1.552},"min":{"1min":0.80800000000000005,"5min":0.80500000000000005,"15min":0.753},"max":{"1min":1.8959999999999999,"5min":2.1429999999999998,"15min":2.3359999999999999},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.488,"15min":1.5209999999999999},"min":{"1min":0.98999999999999999,"5min":0.80500000000000005,"15min":0.72799999999999998},"max":{"1min":2.165,"5min":2.2330000000000001,"15min":2.294},"last":1.5589999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575100,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46204,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325636,"statfs":{"total":96636764160,"available":96589451264,"internally_reserved":0,"allocated":2596864,"data_stored":1602591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.83099999999999996,"15min":0.89300000000000002},"min":{"1min":0.32000000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.29,"5min":1.3919999999999999,"15min":2.1440000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.0249999999999999,"15min":1.073},"min":{"1min":0.372,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7090000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.98799999999999999}]},{"osd":2,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.0129999999999999,"15min":1.091},"min":{"1min":0.50800000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.4299999999999999,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.046},{"interface":"front","average":{"1min":1.034,"5min":1.0660000000000001,"15min":1.1499999999999999},"min":{"1min":0.48099999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.544,"5min":1.782,"15min":1.9159999999999999},"last":0.69899999999999995}]},{"osd":3,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.026,"15min":1.0820000000000001},"min":{"1min":0.432,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.522,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.119,"5min":1.2130000000000001,"15min":1.2829999999999999},"min":{"1min":0.55000000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.7150000000000001,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.196}]},{"osd":4,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3420000000000001,"15min":1.4059999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.883,"5min":1.883,"15min":2.012},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.357,"15min":1.383},"min":{"1min":0.72899999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.9890000000000001,"5min":4.5119999999999996,"15min":4.5119999999999996},"last":1.321}]},{"osd":5,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4039999999999999,"15min":1.411},"min":{"1min":0.79200000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.929,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4139999999999999,"15min":1.466},"min":{"1min":0.66800000000000004,"5min":0.621,"15min":0.621},"max":{"1min":2.169,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.377}]},{"osd":6,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4830000000000001,"15min":1.536},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.0590000000000002,"5min":2.524,"15min":2.524},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.458,"15min":1.51},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.972,"5min":2.2330000000000001,"15min":2.4209999999999998},"last":1.177}]},{"osd":7,"last update":"Tue Nov 28 20:54:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.5329999999999999,"15min":1.579},"min":{"1min":0.89300000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.98,"5min":2.2970000000000002,"15min":2.3610000000000002},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.48,"15min":1.524},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.1469999999999998,"5min":4.7359999999999998,"15min":4.7359999999999998},"last":1.474}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T20:55:18.490 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T20:55:18.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T20:55:18.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.767+0000 7ff1bbb89640 1 Processor -- start 2023-11-28T20:55:18.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.767+0000 7ff1bbb89640 1 -- start start 2023-11-28T20:55:18.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.769+0000 7ff1bbb89640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff1b415d420 0x7ff1b4161a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.769+0000 7ff1bbb89640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1b40cda70 0x7ff1b40cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.769+0000 7ff1bbb89640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b40d04c0 con 0x7ff1b40d0070 2023-11-28T20:55:18.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.769+0000 7ff1bbb89640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b4161fd0 con 0x7ff1b40cda70 2023-11-28T20:55:18.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.769+0000 7ff1bbb89640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b40cdf00 con 0x7ff1b415d420 2023-11-28T20:55:18.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.769+0000 7ff1b9124640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1b40cda70 0x7ff1b40cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.769+0000 7ff1ba126640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff1b40d0070 0x7ff1b40b4c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37024/0 (socket says 172.21.15.45:37024) 2023-11-28T20:55:18.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.769+0000 7ff1ba126640 1 -- 172.21.15.45:0/3284940979 learned_addr learned my addr 172.21.15.45:0/3284940979 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:18.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.769+0000 7ff1b9925640 1 --2- 172.21.15.45:0/3284940979 >> v2:172.21.15.190:3300/0 conn(0x7ff1b415d420 0x7ff1b4161a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.770+0000 7ff1b8923640 1 -- 172.21.15.45:0/3284940979 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018130919 0 0) 0x7ff1b40d04c0 con 0x7ff1b40d0070 2023-11-28T20:55:18.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.770+0000 7ff1b9124640 1 -- 172.21.15.45:0/3284940979 >> v2:172.21.15.190:3300/0 conn(0x7ff1b415d420 msgr2=0x7ff1b4161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:18.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.770+0000 7ff1b9124640 1 --2- 172.21.15.45:0/3284940979 >> v2:172.21.15.190:3300/0 conn(0x7ff1b415d420 0x7ff1b4161a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.770+0000 7ff1b9124640 1 -- 172.21.15.45:0/3284940979 >> v1:172.21.15.45:6789/0 conn(0x7ff1b40d0070 legacy=0x7ff1b40b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:18.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.770+0000 7ff1b9124640 1 -- 172.21.15.45:0/3284940979 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b40ce1e0 con 0x7ff1b40cda70 2023-11-28T20:55:18.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.771+0000 7ff1b9124640 1 --2- 172.21.15.45:0/3284940979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40cda70 0x7ff1b40cfb30 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4067220 tx=0x7ff1a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502acd121350d608 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:18.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.771+0000 7ff1b8923640 1 -- 172.21.15.45:0/3284940979 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4031830 con 0x7ff1b40cda70 2023-11-28T20:55:18.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.771+0000 7ff1b8923640 1 -- 172.21.15.45:0/3284940979 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a4031990 con 0x7ff1b40cda70 2023-11-28T20:55:18.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.771+0000 7ff1b8923640 1 -- 172.21.15.45:0/3284940979 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4031d20 con 0x7ff1b40cda70 2023-11-28T20:55:18.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.772+0000 7ff1bbb89640 1 -- 172.21.15.45:0/3284940979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40cda70 msgr2=0x7ff1b40cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:18.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.772+0000 7ff1bbb89640 1 --2- 172.21.15.45:0/3284940979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40cda70 0x7ff1b40cfb30 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4067220 tx=0x7ff1a402b8d0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.772+0000 7ff1bbb89640 1 -- 172.21.15.45:0/3284940979 shutdown_connections 2023-11-28T20:55:18.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.772+0000 7ff1bbb89640 1 --2- 172.21.15.45:0/3284940979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40cda70 0x7ff1b40cfb30 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.772+0000 7ff1bbb89640 1 --2- 172.21.15.45:0/3284940979 >> v2:172.21.15.190:3300/0 conn(0x7ff1b415d420 0x7ff1b4161a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.772+0000 7ff1bbb89640 1 -- 172.21.15.45:0/3284940979 >> 172.21.15.45:0/3284940979 conn(0x7ff1b4070b20 msgr2=0x7ff1b4070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:18.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.772+0000 7ff1bbb89640 1 -- 172.21.15.45:0/3284940979 shutdown_connections 2023-11-28T20:55:18.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.772+0000 7ff1bbb89640 1 -- 172.21.15.45:0/3284940979 wait complete. 2023-11-28T20:55:18.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.774+0000 7ff1bbb89640 1 Processor -- start 2023-11-28T20:55:18.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.774+0000 7ff1bbb89640 1 -- start start 2023-11-28T20:55:18.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.775+0000 7ff1bbb89640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40d0070 0x7ff1b414c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.775+0000 7ff1bbb89640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1b415d420 0x7ff1b40cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.775+0000 7ff1bbb89640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b40cb1b0 0x7ff1b415fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.775+0000 7ff1bbb89640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b40cb660 con 0x7ff1b40d0070 2023-11-28T20:55:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.775+0000 7ff1bbb89640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b414f490 con 0x7ff1b40cb1b0 2023-11-28T20:55:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.775+0000 7ff1bbb89640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b414cea0 con 0x7ff1b415d420 2023-11-28T20:55:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9925640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40d0070 0x7ff1b414c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9925640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40d0070 0x7ff1b414c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41676/0 (socket says 172.21.15.45:41676) 2023-11-28T20:55:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9925640 1 -- 172.21.15.45:0/1683885525 learned_addr learned my addr 172.21.15.45:0/1683885525 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1ba126640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b40cb1b0 0x7ff1b415fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9124640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1b415d420 0x7ff1b40cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9925640 1 -- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1b415d420 msgr2=0x7ff1b40cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:18.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9925640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1b415d420 0x7ff1b40cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9925640 1 -- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b40cb1b0 msgr2=0x7ff1b415fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:18.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9925640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b40cb1b0 0x7ff1b415fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:18.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9925640 1 -- 172.21.15.45:0/1683885525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a4009400 con 0x7ff1b40d0070 2023-11-28T20:55:18.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9124640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1b415d420 0x7ff1b40cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:18.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.776+0000 7ff1b9925640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40d0070 0x7ff1b414c960 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000d5f0 tx=0x7ff1b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:18.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.777+0000 7ff1aa7fc640 1 -- 172.21.15.45:0/1683885525 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b00125f0 con 0x7ff1b40d0070 2023-11-28T20:55:18.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.777+0000 7ff1aa7fc640 1 -- 172.21.15.45:0/1683885525 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b0012750 con 0x7ff1b40d0070 2023-11-28T20:55:18.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.777+0000 7ff1aa7fc640 1 -- 172.21.15.45:0/1683885525 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0012b00 con 0x7ff1b40d0070 2023-11-28T20:55:18.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.777+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b414f6a0 con 0x7ff1b40d0070 2023-11-28T20:55:18.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.777+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b414fc60 con 0x7ff1b40d0070 2023-11-28T20:55:18.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.778+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b40733d0 con 0x7ff1b40d0070 2023-11-28T20:55:18.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.779+0000 7ff1aa7fc640 1 -- 172.21.15.45:0/1683885525 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff1b0012c60 con 0x7ff1b40d0070 2023-11-28T20:55:18.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.781+0000 7ff1aa7fc640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ff19806af90 0x7ff19806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:18.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.781+0000 7ff1aa7fc640 1 -- 172.21.15.45:0/1683885525 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff1b0089d80 con 0x7ff1b40d0070 2023-11-28T20:55:18.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.782+0000 7ff1b9124640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ff19806af90 0x7ff19806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:18.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.783+0000 7ff1b9124640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ff19806af90 0x7ff19806d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4036000 tx=0x7ff1a4008030 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:18.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:18.786+0000 7ff1aa7fc640 1 -- 172.21.15.45:0/1683885525 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1b0054bb0 con 0x7ff1b40d0070 2023-11-28T20:55:19.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.050+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff1b40d0f90 con 0x7ff1b40d0070 2023-11-28T20:55:19.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.054+0000 7ff1aa7fc640 1 -- 172.21.15.45:0/1683885525 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff1b00581d0 con 0x7ff1b40d0070 2023-11-28T20:55:19.056 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T20:55:19.056 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T20:55:19.056 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T20:55:19.056 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T20:55:19.057 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T20:55:19.057 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T20:55:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.056+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ff19806af90 msgr2=0x7ff19806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.056+0000 7ff1bbb89640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ff19806af90 0x7ff19806d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4036000 tx=0x7ff1a4008030 comp rx=0 tx=0).stop 2023-11-28T20:55:19.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.056+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40d0070 msgr2=0x7ff1b414c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.056+0000 7ff1bbb89640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40d0070 0x7ff1b414c960 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000d5f0 tx=0x7ff1b000dab0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.057+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 shutdown_connections 2023-11-28T20:55:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.057+0000 7ff1bbb89640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b40cb1b0 0x7ff1b415fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.057+0000 7ff1bbb89640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1b415d420 0x7ff1b40cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.057+0000 7ff1bbb89640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7ff19806af90 0x7ff19806d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.057+0000 7ff1bbb89640 1 --2- 172.21.15.45:0/1683885525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1b40d0070 0x7ff1b414c960 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.057+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 >> 172.21.15.45:0/1683885525 conn(0x7ff1b4070b20 msgr2=0x7ff1b40b8150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.057+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 shutdown_connections 2023-11-28T20:55:19.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.057+0000 7ff1bbb89640 1 -- 172.21.15.45:0/1683885525 wait complete. 2023-11-28T20:55:19.076 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T20:55:19.239 INFO:tasks.ceph.mds.c.smithi045.stderr:2023-11-28T20:55:19.236+0000 7f34df61c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T20:55:19.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.283+0000 7f587f59e640 1 Processor -- start 2023-11-28T20:55:19.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.283+0000 7f587f59e640 1 -- start start 2023-11-28T20:55:19.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.285+0000 7f587f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f58800ce470 0x7f58800abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:19.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.285+0000 7f587f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800b3bc0 0x7f588015abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:19.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.285+0000 7f587f59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58800ce850 0x7f588015b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:19.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.285+0000 7f587f59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58800ac610 con 0x7f58800ce850 2023-11-28T20:55:19.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.285+0000 7f587f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588015b9a0 con 0x7f58800ce470 2023-11-28T20:55:19.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.285+0000 7f587f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588015bc80 con 0x7f58800b3bc0 2023-11-28T20:55:19.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.285+0000 7f587ed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58800ce850 0x7f588015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:19.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.285+0000 7f587ed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58800ce850 0x7f588015b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41682/0 (socket says 172.21.15.45:41682) 2023-11-28T20:55:19.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.285+0000 7f587ed9d640 1 -- 172.21.15.45:0/1853483429 learned_addr learned my addr 172.21.15.45:0/1853483429 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:19.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.286+0000 7f587e59c640 1 --2- 172.21.15.45:0/1853483429 >> v2:172.21.15.190:3300/0 conn(0x7f58800ce470 0x7f58800abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:19.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.286+0000 7f5877fff640 1 --2- 172.21.15.45:0/1853483429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800b3bc0 0x7f588015abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:19.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.286+0000 7f587ed9d640 1 -- 172.21.15.45:0/1853483429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800b3bc0 msgr2=0x7f588015abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:19.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.286+0000 7f587ed9d640 1 --2- 172.21.15.45:0/1853483429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800b3bc0 0x7f588015abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.286+0000 7f587ed9d640 1 -- 172.21.15.45:0/1853483429 >> v2:172.21.15.190:3300/0 conn(0x7f58800ce470 msgr2=0x7f58800abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:19.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.286+0000 7f587ed9d640 1 --2- 172.21.15.45:0/1853483429 >> v2:172.21.15.190:3300/0 conn(0x7f58800ce470 0x7f58800abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.286+0000 7f587ed9d640 1 -- 172.21.15.45:0/1853483429 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588015bf60 con 0x7f58800ce850 2023-11-28T20:55:19.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.286+0000 7f5877fff640 1 --2- 172.21.15.45:0/1853483429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800b3bc0 0x7f588015abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:19.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.287+0000 7f587ed9d640 1 --2- 172.21.15.45:0/1853483429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800ce850 0x7f588015b130 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f588006b4b0 tx=0x7f587802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32bda7d7b7ec9e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.288+0000 7f587dd9b640 1 -- 172.21.15.45:0/1853483429 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587802ca70 con 0x7f58800ce850 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.288+0000 7f587dd9b640 1 -- 172.21.15.45:0/1853483429 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587802cbd0 con 0x7f58800ce850 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.288+0000 7f587dd9b640 1 -- 172.21.15.45:0/1853483429 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58780343f0 con 0x7f58800ce850 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.289+0000 7f587f59e640 1 -- 172.21.15.45:0/1853483429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800ce850 msgr2=0x7f588015b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.289+0000 7f587f59e640 1 --2- 172.21.15.45:0/1853483429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800ce850 0x7f588015b130 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f588006b4b0 tx=0x7f587802bed0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.289+0000 7f587f59e640 1 -- 172.21.15.45:0/1853483429 shutdown_connections 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.289+0000 7f587f59e640 1 --2- 172.21.15.45:0/1853483429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800ce850 0x7f588015b130 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.289+0000 7f587f59e640 1 --2- 172.21.15.45:0/1853483429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800b3bc0 0x7f588015abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.289+0000 7f587f59e640 1 --2- 172.21.15.45:0/1853483429 >> v2:172.21.15.190:3300/0 conn(0x7f58800ce470 0x7f58800abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.289+0000 7f587f59e640 1 -- 172.21.15.45:0/1853483429 >> 172.21.15.45:0/1853483429 conn(0x7f588006ab30 msgr2=0x7f58800b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:19.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.289+0000 7f587f59e640 1 -- 172.21.15.45:0/1853483429 shutdown_connections 2023-11-28T20:55:19.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.289+0000 7f587f59e640 1 -- 172.21.15.45:0/1853483429 wait complete. 2023-11-28T20:55:19.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.291+0000 7f587f59e640 1 Processor -- start 2023-11-28T20:55:19.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.291+0000 7f587f59e640 1 -- start start 2023-11-28T20:55:19.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800b3bc0 0x7f588014cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:19.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58800ce470 0x7f5880070690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:19.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800ce850 0x7f5880159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58800af2a0 con 0x7f58800b3bc0 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5880151e20 con 0x7f58800ce470 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58800af4d0 con 0x7f58800ce850 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800b3bc0 0x7f588014cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800b3bc0 0x7f588014cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41684/0 (socket says 172.21.15.45:41684) 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587e59c640 1 -- 172.21.15.45:0/2665116223 learned_addr learned my addr 172.21.15.45:0/2665116223 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f587ed9d640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800ce850 0x7f5880159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.292+0000 7f5877fff640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58800ce470 0x7f5880070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f587e59c640 1 -- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800ce850 msgr2=0x7f5880159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f587e59c640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800ce850 0x7f5880159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f587e59c640 1 -- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58800ce470 msgr2=0x7f5880070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f587e59c640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58800ce470 0x7f5880070690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f587e59c640 1 -- 172.21.15.45:0/2665116223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5878009b30 con 0x7f58800b3bc0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f587e59c640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800b3bc0 0x7f588014cce0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f587000e9b0 tx=0x7f587000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f5876ffd640 1 -- 172.21.15.45:0/2665116223 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870009670 con 0x7f58800b3bc0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f5876ffd640 1 -- 172.21.15.45:0/2665116223 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587000c9a0 con 0x7f58800b3bc0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f5876ffd640 1 -- 172.21.15.45:0/2665116223 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587000cd50 con 0x7f58800b3bc0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58800af7b0 con 0x7f58800b3bc0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.293+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5880152100 con 0x7f58800b3bc0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.294+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588009d0f0 con 0x7f58800b3bc0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.295+0000 7f5876ffd640 1 -- 172.21.15.45:0/2665116223 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5870013470 con 0x7f58800b3bc0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.296+0000 7f5876ffd640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f586006af60 0x7f586006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.296+0000 7f5876ffd640 1 -- 172.21.15.45:0/2665116223 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5870089b10 con 0x7f58800b3bc0 2023-11-28T20:55:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.297+0000 7f5877fff640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f586006af60 0x7f586006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:19.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.298+0000 7f5877fff640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f586006af60 0x7f586006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58800b20a0 tx=0x7f5868008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:19.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.300+0000 7f5876ffd640 1 -- 172.21.15.45:0/2665116223 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5870054940 con 0x7f58800b3bc0 2023-11-28T20:55:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.534+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f588004e920 con 0x7f58800b3bc0 2023-11-28T20:55:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.535+0000 7f5876ffd640 1 -- 172.21.15.45:0/2665116223 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f5870017090 con 0x7f58800b3bc0 2023-11-28T20:55:19.537 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:19.537 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7531,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1320359028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1320359028},{"type":"v1","addr":"172.21.15.45:6833","nonce":1320359028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/808323687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":808323687},{"type":"v1","addr":"172.21.15.190:6836","nonce":808323687}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3406194668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3406194668},{"type":"v1","addr":"172.21.15.190:6837","nonce":3406194668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:55:12.738156+0000","modified":"2023-11-28T20:55:15.949737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"c","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/637440036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":637440036},{"type":"v1","addr":"172.21.15.45:6835","nonce":637440036}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-28T20:55:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 76 2023-11-28T20:55:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f586006af60 msgr2=0x7f586006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:19.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f586006af60 0x7f586006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58800b20a0 tx=0x7f5868008040 comp rx=0 tx=0).stop 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800b3bc0 msgr2=0x7f588014cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800b3bc0 0x7f588014cce0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f587000e9b0 tx=0x7f587000ee70 comp rx=0 tx=0).stop 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 shutdown_connections 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58800ce850 0x7f5880159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58800ce470 0x7f5880070690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f586006af60 0x7f586006d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 --2- 172.21.15.45:0/2665116223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58800b3bc0 0x7f588014cce0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 >> 172.21.15.45:0/2665116223 conn(0x7f588006ab30 msgr2=0x7f5880156f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 shutdown_connections 2023-11-28T20:55:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:19.537+0000 7f587f59e640 1 -- 172.21.15.45:0/2665116223 wait complete. 2023-11-28T20:55:19.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T20:55:12.738156+0000', 'modified': '2023-11-28T20:55:15.949737+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9168}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9168': {'gid': 9168, 'name': 'c', 'rank': 0, 'incarnation': 72, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/637440036', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6834', 'nonce': 637440036}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 637440036}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T20:55:19.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T20:55:19.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T20:55:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.199+0000 7fc128ddb640 1 Processor -- start 2023-11-28T20:55:20.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.199+0000 7fc128ddb640 1 -- start start 2023-11-28T20:55:20.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.201+0000 7fc128ddb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc124159ed0 0x7fc124071060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.201+0000 7fc128ddb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cef00 0x7fc1240cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.201+0000 7fc128ddb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc124070bd0 con 0x7fc124071e80 2023-11-28T20:55:20.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.201+0000 7fc128ddb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1240cf8f0 con 0x7fc124159ed0 2023-11-28T20:55:20.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.201+0000 7fc128ddb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1240ad8e0 con 0x7fc1240cef00 2023-11-28T20:55:20.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.201+0000 7fc1237fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc124159ed0 0x7fc124071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.201+0000 7fc1237fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc124159ed0 0x7fc124071060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41686/0 (socket says 172.21.15.45:41686) 2023-11-28T20:55:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.201+0000 7fc1237fe640 1 -- 172.21.15.45:0/147544584 learned_addr learned my addr 172.21.15.45:0/147544584 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.202+0000 7fc1237fe640 1 -- 172.21.15.45:0/147544584 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cef00 msgr2=0x7fc1240cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.202+0000 7fc123fff640 1 --2- 172.21.15.45:0/147544584 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cef00 0x7fc1240cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.202+0000 7fc1237fe640 1 --2- 172.21.15.45:0/147544584 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cef00 0x7fc1240cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.202+0000 7fc1237fe640 1 -- 172.21.15.45:0/147544584 >> v1:172.21.15.45:6789/0 conn(0x7fc124071e80 legacy=0x7fc1240ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.202+0000 7fc1237fe640 1 -- 172.21.15.45:0/147544584 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1240adad0 con 0x7fc124159ed0 2023-11-28T20:55:20.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.203+0000 7fc1237fe640 1 --2- 172.21.15.45:0/147544584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 0x7fc124071060 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fc114009870 tx=0x7fc11402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d4f1225ef51f13 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.203+0000 7fc122ffd640 1 -- 172.21.15.45:0/147544584 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114031640 con 0x7fc124159ed0 2023-11-28T20:55:20.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.203+0000 7fc122ffd640 1 -- 172.21.15.45:0/147544584 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1140317a0 con 0x7fc124159ed0 2023-11-28T20:55:20.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.203+0000 7fc122ffd640 1 -- 172.21.15.45:0/147544584 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114031b30 con 0x7fc124159ed0 2023-11-28T20:55:20.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.204+0000 7fc128ddb640 1 -- 172.21.15.45:0/147544584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 msgr2=0x7fc124071060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.204+0000 7fc128ddb640 1 --2- 172.21.15.45:0/147544584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 0x7fc124071060 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fc114009870 tx=0x7fc11402b8f0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.204+0000 7fc128ddb640 1 -- 172.21.15.45:0/147544584 shutdown_connections 2023-11-28T20:55:20.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.204+0000 7fc128ddb640 1 --2- 172.21.15.45:0/147544584 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cef00 0x7fc1240cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.204+0000 7fc128ddb640 1 --2- 172.21.15.45:0/147544584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 0x7fc124071060 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.204+0000 7fc128ddb640 1 -- 172.21.15.45:0/147544584 >> 172.21.15.45:0/147544584 conn(0x7fc12406a710 msgr2=0x7fc12406ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.204+0000 7fc128ddb640 1 -- 172.21.15.45:0/147544584 shutdown_connections 2023-11-28T20:55:20.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.204+0000 7fc128ddb640 1 -- 172.21.15.45:0/147544584 wait complete. 2023-11-28T20:55:20.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.207+0000 7fc128ddb640 1 Processor -- start 2023-11-28T20:55:20.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.207+0000 7fc128ddb640 1 -- start start 2023-11-28T20:55:20.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.208+0000 7fc128ddb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc124071e80 0x7fc12414f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.208+0000 7fc128ddb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240cef00 0x7fc1241521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.208+0000 7fc128ddb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 0x7fc124152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.208+0000 7fc128ddb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124153710 con 0x7fc124159ed0 2023-11-28T20:55:20.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.208+0000 7fc128ddb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124153cd0 con 0x7fc1240cef00 2023-11-28T20:55:20.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.208+0000 7fc128ddb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc124152d30 con 0x7fc124071e80 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc123fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 0x7fc124152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc123fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 0x7fc124152700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41694/0 (socket says 172.21.15.45:41694) 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc123fff640 1 -- 172.21.15.45:0/854312411 learned_addr learned my addr 172.21.15.45:0/854312411 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc1237fe640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc124071e80 0x7fc12414f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc11affd640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240cef00 0x7fc1241521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc123fff640 1 -- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc124071e80 msgr2=0x7fc12414f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc123fff640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc124071e80 0x7fc12414f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc123fff640 1 -- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240cef00 msgr2=0x7fc1241521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc123fff640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240cef00 0x7fc1241521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.209+0000 7fc123fff640 1 -- 172.21.15.45:0/854312411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1140094d0 con 0x7fc124159ed0 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.210+0000 7fc123fff640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 0x7fc124152700 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00c820 tx=0x7fc11c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.210+0000 7fc120ff9640 1 -- 172.21.15.45:0/854312411 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c004020 con 0x7fc124159ed0 2023-11-28T20:55:20.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.210+0000 7fc120ff9640 1 -- 172.21.15.45:0/854312411 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c004180 con 0x7fc124159ed0 2023-11-28T20:55:20.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.210+0000 7fc128ddb640 1 -- 172.21.15.45:0/854312411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc124153010 con 0x7fc124159ed0 2023-11-28T20:55:20.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.210+0000 7fc120ff9640 1 -- 172.21.15.45:0/854312411 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c007830 con 0x7fc124159ed0 2023-11-28T20:55:20.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.210+0000 7fc128ddb640 1 -- 172.21.15.45:0/854312411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12409cc40 con 0x7fc124159ed0 2023-11-28T20:55:20.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.212+0000 7fc120ff9640 1 -- 172.21.15.45:0/854312411 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc11c007990 con 0x7fc124159ed0 2023-11-28T20:55:20.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.213+0000 7fc120ff9640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc10406af60 0x7fc10406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.213+0000 7fc120ff9640 1 -- 172.21.15.45:0/854312411 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc11c089780 con 0x7fc124159ed0 2023-11-28T20:55:20.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.214+0000 7fc1237fe640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc10406af60 0x7fc10406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.214+0000 7fc12cbcf740 1 Processor -- start 2023-11-28T20:55:20.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.215+0000 7fc12cbcf740 1 -- start start 2023-11-28T20:55:20.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.215+0000 7fc1237fe640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc10406af60 0x7fc10406d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc114036000 tx=0x7fc11403c040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.215+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf867070 0x55fbaf867450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf861d70 0x55fbaf866270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8aa730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf7c5880 con 0x55fbaf8667b0 2023-11-28T20:55:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf7d4ae0 con 0x55fbaf867070 2023-11-28T20:55:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf7d21d0 con 0x55fbaf861d70 2023-11-28T20:55:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc123fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc123fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8aa730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41706/0 (socket says 172.21.15.45:41706) 2023-11-28T20:55:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc123fff640 1 -- 172.21.15.45:0/2405387381 learned_addr learned my addr 172.21.15.45:0/2405387381 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc1237fe640 1 --2- 172.21.15.45:0/2405387381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf867070 0x55fbaf867450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc11affd640 1 --2- 172.21.15.45:0/2405387381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf861d70 0x55fbaf866270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc123fff640 1 -- 172.21.15.45:0/2405387381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf861d70 msgr2=0x55fbaf866270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc123fff640 1 --2- 172.21.15.45:0/2405387381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf861d70 0x55fbaf866270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc123fff640 1 -- 172.21.15.45:0/2405387381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf867070 msgr2=0x55fbaf867450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc123fff640 1 --2- 172.21.15.45:0/2405387381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf867070 0x55fbaf867450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.216+0000 7fc123fff640 1 -- 172.21.15.45:0/2405387381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbaf795860 con 0x55fbaf8667b0 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.217+0000 7fc123fff640 1 --2- 172.21.15.45:0/2405387381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8aa730 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fc11c05fea0 tx=0x7fc11c0587d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e22269ebfa2a1c50 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.217+0000 7fc1197fa640 1 -- 172.21.15.45:0/2405387381 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0603d0 con 0x55fbaf8667b0 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.217+0000 7fc1197fa640 1 -- 172.21.15.45:0/2405387381 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c0547e0 con 0x55fbaf8667b0 2023-11-28T20:55:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.217+0000 7fc1197fa640 1 -- 172.21.15.45:0/2405387381 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c054950 con 0x55fbaf8667b0 2023-11-28T20:55:20.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.218+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2405387381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 msgr2=0x55fbaf8aa730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.218+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2405387381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8aa730 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fc11c05fea0 tx=0x7fc11c0587d0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.218+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2405387381 shutdown_connections 2023-11-28T20:55:20.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.218+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2405387381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8aa730 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.218+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2405387381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf861d70 0x55fbaf866270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.218+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2405387381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf867070 0x55fbaf867450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.218+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2405387381 >> 172.21.15.45:0/2405387381 conn(0x55fbaf828c70 msgr2=0x55fbaf8acb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:20.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.218+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2405387381 shutdown_connections 2023-11-28T20:55:20.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.218+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2405387381 wait complete. 2023-11-28T20:55:20.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.219+0000 7fc12cbcf740 1 -- start start 2023-11-28T20:55:20.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.219+0000 7fc12cbcf740 1 Processor -- start 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf8b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf8667b0 0x55fbaf8b4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf7cdac0 con 0x55fbaf867070 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf76a2f0 con 0x55fbaf861d70 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf787c70 con 0x55fbaf8667b0 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc12cbcf740 10 client.0 _ll_register_callbacks cb 0x55fbaf78c3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc123fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc123fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41708/0 (socket says 172.21.15.45:41708) 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc123fff640 1 -- 172.21.15.45:0/2007225038 learned_addr learned my addr 172.21.15.45:0/2007225038 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.220+0000 7fc1237fe640 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf8b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc123fff640 1 -- 172.21.15.45:0/2007225038 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf8667b0 msgr2=0x55fbaf8b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc123fff640 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf8667b0 0x55fbaf8b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc123fff640 1 -- 172.21.15.45:0/2007225038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 msgr2=0x55fbaf8b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc123fff640 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf8b18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc123fff640 1 -- 172.21.15.45:0/2007225038 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12409cc40 con 0x55fbaf867070 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc1237fe640 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf8b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc123fff640 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8b4ad0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fc11c057dd0 tx=0x7fc11c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29565d4324b2f95 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc1027fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc1027fc640 1 -- 172.21.15.45:0/2007225038 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c060140 con 0x55fbaf867070 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc1027fc640 1 -- 172.21.15.45:0/2007225038 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc11c054950 con 0x55fbaf867070 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2007225038 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fbaf7c6aa0 con 0x55fbaf867070 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc1027fc640 1 -- 172.21.15.45:0/2007225038 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc11c00a3b0 con 0x55fbaf867070 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.221+0000 7fc12cbcf740 10 client.9306 fetch_fsmap learned FSMap version 76 2023-11-28T20:55:20.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2007225038 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0x55fbaf7cc140 con 0x55fbaf867070 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc1027fc640 1 -- 172.21.15.45:0/2007225038 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c07a080 con 0x55fbaf867070 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc1027fc640 1 -- 172.21.15.45:0/2007225038 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 76) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc11c05e070 con 0x55fbaf867070 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc12cbcf740 10 client.9306 fetch_fsmap finished waiting for FSMap version 76 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc12cbcf740 10 client.9306 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc12cbcf740 20 client.9306 populate_metadata read hostname 'smithi045' 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] conn(0x55fbaf88fca0 0x55fbaf8931d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc12cbcf740 4 client.9306 mds_command: new command op to 9168 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2007225038 --> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fbaf7c67c0 con 0x55fbaf88fca0 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.222+0000 7fc1237fe640 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] conn(0x55fbaf88fca0 0x55fbaf8931d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.223+0000 7fc1237fe640 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] conn(0x55fbaf88fca0 0x55fbaf8931d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7a1ad9af095dd11 server_cookie=522ad2afaa32101c in_seq=0 out_seq=0 2023-11-28T20:55:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.223+0000 7fc1027fc640 10 client.9306 ms_handle_connect on v2:172.21.15.45:6834/637440036 2023-11-28T20:55:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc1027fc640 1 -- 172.21.15.45:0/2007225038 <== mds.0 v2:172.21.15.45:6834/637440036 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fbaf7c67c0 con 0x55fbaf88fca0 2023-11-28T20:55:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc1027fc640 10 client.9306 handle_command_reply: tid=0 2023-11-28T20:55:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 client.9306 shutdown 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2007225038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 msgr2=0x55fbaf8b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8b4ad0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fc11c057dd0 tx=0x7fc11c0046c0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2007225038 shutdown_connections 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] conn(0x55fbaf88fca0 0x55fbaf8931d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8b4ad0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf8667b0 0x55fbaf8b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2007225038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf8b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc1027fc640 0 client.9306 ms_handle_reset on v2:172.21.15.45:6834/637440036 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2007225038 >> 172.21.15.45:0/2007225038 conn(0x55fbaf828c70 msgr2=0x55fbaf8aa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2007225038 shutdown_connections 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.224+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2007225038 wait complete. 2023-11-28T20:55:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.225+0000 7fc12cbcf740 20 client.9306 trim_cache size 0 max 16384 2023-11-28T20:55:20.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.242+0000 7fc12cbcf740 1 Processor -- start 2023-11-28T20:55:20.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.242+0000 7fc12cbcf740 1 -- start start 2023-11-28T20:55:20.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.243+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf85d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.243+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8907c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.243+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf867070 0x55fbaf8b4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.243+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf530f20 con 0x55fbaf8667b0 2023-11-28T20:55:20.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.243+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf85dbb0 con 0x55fbaf861d70 2023-11-28T20:55:20.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.243+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf890d00 con 0x55fbaf867070 2023-11-28T20:55:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.243+0000 7fc11affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8907c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.243+0000 7fc11affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8907c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41714/0 (socket says 172.21.15.45:41714) 2023-11-28T20:55:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.243+0000 7fc11affd640 1 -- 172.21.15.45:0/2672047923 learned_addr learned my addr 172.21.15.45:0/2672047923 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.244+0000 7fc11affd640 1 -- 172.21.15.45:0/2672047923 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf867070 msgr2=0x55fbaf8b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.244+0000 7fc11affd640 1 --2- 172.21.15.45:0/2672047923 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf867070 0x55fbaf8b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.244+0000 7fc11affd640 1 -- 172.21.15.45:0/2672047923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 msgr2=0x55fbaf85d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.244+0000 7fc11affd640 1 --2- 172.21.15.45:0/2672047923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf85d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.244+0000 7fc11affd640 1 -- 172.21.15.45:0/2672047923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbaf769b80 con 0x55fbaf8667b0 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.244+0000 7fc11affd640 1 --2- 172.21.15.45:0/2672047923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8907c0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc10c003f80 tx=0x7fc10c0054e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64bff810db6fd5db server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.244+0000 7fc119ffb640 1 -- 172.21.15.45:0/2672047923 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c005730 con 0x55fbaf8667b0 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.244+0000 7fc119ffb640 1 -- 172.21.15.45:0/2672047923 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c001ae0 con 0x55fbaf8667b0 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.244+0000 7fc119ffb640 1 -- 172.21.15.45:0/2672047923 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c001e70 con 0x55fbaf8667b0 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.245+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2672047923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 msgr2=0x55fbaf8907c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.245+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2672047923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8907c0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc10c003f80 tx=0x7fc10c0054e0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.245+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2672047923 shutdown_connections 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.245+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2672047923 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf867070 0x55fbaf8b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.245+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2672047923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf8667b0 0x55fbaf8907c0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.245+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/2672047923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf85d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.245+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2672047923 >> 172.21.15.45:0/2672047923 conn(0x55fbaf828c70 msgr2=0x55fbaf867450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.245+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2672047923 shutdown_connections 2023-11-28T20:55:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.245+0000 7fc12cbcf740 1 -- 172.21.15.45:0/2672047923 wait complete. 2023-11-28T20:55:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.246+0000 7fc12cbcf740 1 -- start start 2023-11-28T20:55:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.246+0000 7fc12cbcf740 1 Processor -- start 2023-11-28T20:55:20.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf8b0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf8667b0 0x55fbaf88c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc12cbcf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8bb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf88d320 con 0x55fbaf867070 2023-11-28T20:55:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf8922d0 con 0x55fbaf861d70 2023-11-28T20:55:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc12cbcf740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fbaf88e680 con 0x55fbaf8667b0 2023-11-28T20:55:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc12cbcf740 10 client.0 _ll_register_callbacks cb 0x7fc11c06e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T20:55:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc123fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8bb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc123fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8bb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41716/0 (socket says 172.21.15.45:41716) 2023-11-28T20:55:20.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.247+0000 7fc123fff640 1 -- 172.21.15.45:0/1480258733 learned_addr learned my addr 172.21.15.45:0/1480258733 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc1237fe640 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf8b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc11affd640 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf8667b0 0x55fbaf88c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc123fff640 1 -- 172.21.15.45:0/1480258733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf8667b0 msgr2=0x55fbaf88c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc123fff640 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf8667b0 0x55fbaf88c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc123fff640 1 -- 172.21.15.45:0/1480258733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 msgr2=0x55fbaf8b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc123fff640 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf8b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc123fff640 1 -- 172.21.15.45:0/1480258733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10c00f040 con 0x55fbaf867070 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc123fff640 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8bb0f0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0545d0 tx=0x7fc11c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efa54b48db57cc0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc1027fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc1027fc640 1 -- 172.21.15.45:0/1480258733 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c00a3b0 con 0x55fbaf867070 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.248+0000 7fc1027fc640 1 -- 172.21.15.45:0/1480258733 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc11c054950 con 0x55fbaf867070 2023-11-28T20:55:20.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc12cbcf740 1 -- 172.21.15.45:0/1480258733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc11c0547e0 con 0x55fbaf867070 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc1027fc640 1 -- 172.21.15.45:0/1480258733 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc11c007e50 con 0x55fbaf867070 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc12cbcf740 10 client.9312 fetch_fsmap learned FSMap version 76 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc12cbcf740 1 -- 172.21.15.45:0/1480258733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0x55fbaf78f170 con 0x55fbaf867070 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc1027fc640 1 -- 172.21.15.45:0/1480258733 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c0547e0 con 0x55fbaf867070 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc1027fc640 1 -- 172.21.15.45:0/1480258733 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 76) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc11c080070 con 0x55fbaf867070 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc12cbcf740 10 client.9312 fetch_fsmap finished waiting for FSMap version 76 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc12cbcf740 10 client.9312 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc12cbcf740 20 client.9312 populate_metadata read hostname 'smithi045' 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] conn(0x55fbaf8bb630 0x55fbaf8a24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc12cbcf740 4 client.9312 mds_command: new command op to 9168 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc12cbcf740 1 -- 172.21.15.45:0/1480258733 --> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fbaf4ce6c0 con 0x55fbaf8bb630 2023-11-28T20:55:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.249+0000 7fc1237fe640 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] conn(0x55fbaf8bb630 0x55fbaf8a24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.250+0000 7fc1237fe640 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] conn(0x55fbaf8bb630 0x55fbaf8a24f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ceb6590b105a2c4 server_cookie=56d460a646da834b in_seq=0 out_seq=0 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.250+0000 7fc1027fc640 10 client.9312 ms_handle_connect on v2:172.21.15.45:6834/637440036 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.250+0000 7fc1027fc640 1 -- 172.21.15.45:0/1480258733 <== mds.0 v2:172.21.15.45:6834/637440036 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55fbaf4ce6c0 con 0x55fbaf8bb630 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.250+0000 7fc1027fc640 10 client.9312 handle_command_reply: tid=0 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.250+0000 7fc12cbcf740 1 client.9312 shutdown 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.250+0000 7fc12cbcf740 1 -- 172.21.15.45:0/1480258733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 msgr2=0x55fbaf8bb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.250+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8bb0f0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0545d0 tx=0x7fc11c00cf90 comp rx=0 tx=0).stop 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc12cbcf740 1 -- 172.21.15.45:0/1480258733 shutdown_connections 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.45:6834/637440036,v1:172.21.15.45:6835/637440036] conn(0x55fbaf8bb630 0x55fbaf8a24f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbaf867070 0x55fbaf8bb0f0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55fbaf8667b0 0x55fbaf88c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc12cbcf740 1 --2- 172.21.15.45:0/1480258733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fbaf861d70 0x55fbaf8b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc1027fc640 0 client.9312 ms_handle_reset on v2:172.21.15.45:6834/637440036 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc12cbcf740 1 -- 172.21.15.45:0/1480258733 >> 172.21.15.45:0/1480258733 conn(0x55fbaf828c70 msgr2=0x55fbaf8bc450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc12cbcf740 1 -- 172.21.15.45:0/1480258733 shutdown_connections 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc12cbcf740 1 -- 172.21.15.45:0/1480258733 wait complete. 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc12cbcf740 20 client.9312 trim_cache size 0 max 16384 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 9168, 2023-11-28T20:55:20.258 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T20:55:20.259 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T20:55:20.259 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T20:55:20.259 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.9722314699999997, 2023-11-28T20:55:20.259 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 76, 2023-11-28T20:55:20.259 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 161, 2023-11-28T20:55:20.259 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 161, 2023-11-28T20:55:20.259 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 12.773827493000001 2023-11-28T20:55:20.259 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T20:55:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc128ddb640 1 -- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc10406af60 msgr2=0x7fc10406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc128ddb640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc10406af60 0x7fc10406d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc114036000 tx=0x7fc11403c040 comp rx=0 tx=0).stop 2023-11-28T20:55:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc128ddb640 1 -- 172.21.15.45:0/854312411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 msgr2=0x7fc124152700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.251+0000 7fc128ddb640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 0x7fc124152700 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00c820 tx=0x7fc11c00cce0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.252+0000 7fc128ddb640 1 -- 172.21.15.45:0/854312411 shutdown_connections 2023-11-28T20:55:20.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.252+0000 7fc128ddb640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fc10406af60 0x7fc10406d420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.252+0000 7fc128ddb640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc124159ed0 0x7fc124152700 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.252+0000 7fc128ddb640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240cef00 0x7fc1241521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.252+0000 7fc128ddb640 1 --2- 172.21.15.45:0/854312411 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc124071e80 0x7fc12414f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.252+0000 7fc128ddb640 1 -- 172.21.15.45:0/854312411 >> 172.21.15.45:0/854312411 conn(0x7fc12406a710 msgr2=0x7fc1240b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:20.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.252+0000 7fc128ddb640 1 -- 172.21.15.45:0/854312411 shutdown_connections 2023-11-28T20:55:20.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.252+0000 7fc128ddb640 1 -- 172.21.15.45:0/854312411 wait complete. 2023-11-28T20:55:20.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T20:55:20.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.489+0000 7fbd2c8d1640 1 Processor -- start 2023-11-28T20:55:20.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.489+0000 7fbd2c8d1640 1 -- start start 2023-11-28T20:55:20.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.490+0000 7fbd2c8d1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbd280cfbb0 0x7fbd280d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.491+0000 7fbd2c8d1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd280d1110 con 0x7fbd2815d6b0 2023-11-28T20:55:20.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.491+0000 7fbd2c8d1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd280d13f0 con 0x7fbd2815d2d0 2023-11-28T20:55:20.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.491+0000 7fbd2c8d1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd280d2070 con 0x7fbd280cfbb0 2023-11-28T20:55:20.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.491+0000 7fbd1ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbd2815d6b0 0x7fbd280ce0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37042/0 (socket says 172.21.15.45:37042) 2023-11-28T20:55:20.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.491+0000 7fbd1ffff640 1 -- 172.21.15.45:0/4176145082 learned_addr learned my addr 172.21.15.45:0/4176145082 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:20.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.491+0000 7fbd27fff640 1 --2- 172.21.15.45:0/4176145082 >> v2:172.21.15.190:3300/0 conn(0x7fbd280cfbb0 0x7fbd280d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.491+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419755481 0 0) 0x7fbd280d1110 con 0x7fbd2815d6b0 2023-11-28T20:55:20.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.491+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd14003520 con 0x7fbd2815d6b0 2023-11-28T20:55:20.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 942676906 0 0) 0x7fbd14003520 con 0x7fbd2815d6b0 2023-11-28T20:55:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 >> v2:172.21.15.190:3300/0 conn(0x7fbd280cfbb0 msgr2=0x7fbd280d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 --2- 172.21.15.45:0/4176145082 >> v2:172.21.15.190:3300/0 conn(0x7fbd280cfbb0 0x7fbd280d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 >> v1:172.21.15.190:6789/0 conn(0x7fbd2815d2d0 legacy=0x7fbd280b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd280d16d0 con 0x7fbd2815d6b0 2023-11-28T20:55:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fbd10001670 con 0x7fbd2815d6b0 2023-11-28T20:55:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 >> v1:172.21.15.45:6789/0 conn(0x7fbd2815d6b0 legacy=0x7fbd280ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd140039c0 0x7fbd14003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd140047c0 0x7fbd14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd1400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280d2070 con 0x7fbd14005180 2023-11-28T20:55:20.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1400c080 con 0x7fbd140039c0 2023-11-28T20:55:20.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1400c240 con 0x7fbd140047c0 2023-11-28T20:55:20.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.492+0000 7fbd27fff640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd1400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.493+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd140047c0 0x7fbd14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.493+0000 7fbd27fff640 1 -- 172.21.15.45:0/4176145082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd140047c0 msgr2=0x7fbd14004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.493+0000 7fbd27fff640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd140047c0 0x7fbd14004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.493+0000 7fbd27fff640 1 -- 172.21.15.45:0/4176145082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd140039c0 msgr2=0x7fbd14003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:20.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.493+0000 7fbd27fff640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd140039c0 0x7fbd14003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.493+0000 7fbd27fff640 1 -- 172.21.15.45:0/4176145082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd14003520 con 0x7fbd14005180 2023-11-28T20:55:20.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.493+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd140047c0 0x7fbd14004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:20.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.494+0000 7fbd27fff640 1 -- 172.21.15.45:0/4176145082 reap_dead start 2023-11-28T20:55:20.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.494+0000 7fbd27fff640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd1400b980 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009fc0 tx=0x7fbd140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8499f6511d8ef9f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.494+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2002cc60 con 0x7fbd14005180 2023-11-28T20:55:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.494+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2000f040 con 0x7fbd14005180 2023-11-28T20:55:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.494+0000 7fbd26ffd640 1 -- 172.21.15.45:0/4176145082 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd20003990 con 0x7fbd14005180 2023-11-28T20:55:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.495+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4176145082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 msgr2=0x7fbd1400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.495+0000 7fbd2c8d1640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd1400b980 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009fc0 tx=0x7fbd140057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.495+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4176145082 shutdown_connections 2023-11-28T20:55:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.495+0000 7fbd2c8d1640 1 --2- 172.21.15.45:0/4176145082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd1400b980 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.495+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4176145082 >> 172.21.15.45:0/4176145082 conn(0x7fbd2806a500 msgr2=0x7fbd2806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.495+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4176145082 shutdown_connections 2023-11-28T20:55:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.495+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4176145082 wait complete. 2023-11-28T20:55:20.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.497+0000 7fbd2c8d1640 1 Processor -- start 2023-11-28T20:55:20.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.497+0000 7fbd2c8d1640 1 -- start start 2023-11-28T20:55:20.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.498+0000 7fbd2c8d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd2814d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.498+0000 7fbd2c8d1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd280cc2c0 0x7fbd280cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.498+0000 7fbd2c8d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd280c7e20 0x7fbd280c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.498+0000 7fbd2c8d1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280c7a60 con 0x7fbd14005180 2023-11-28T20:55:20.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.498+0000 7fbd2c8d1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280ccc60 con 0x7fbd280cc2c0 2023-11-28T20:55:20.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.498+0000 7fbd2c8d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2814de10 con 0x7fbd280c7e20 2023-11-28T20:55:20.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd277fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd2814d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd277fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd2814d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41726/0 (socket says 172.21.15.45:41726) 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd277fe640 1 -- 172.21.15.45:0/4189735217 learned_addr learned my addr 172.21.15.45:0/4189735217 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd27fff640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd280c7e20 0x7fbd280c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd280cc2c0 0x7fbd280cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd277fe640 1 -- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd280c7e20 msgr2=0x7fbd280c82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd277fe640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd280c7e20 0x7fbd280c82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd277fe640 1 -- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd280cc2c0 msgr2=0x7fbd280cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd277fe640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd280cc2c0 0x7fbd280cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd277fe640 1 -- 172.21.15.45:0/4189735217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd280c6de0 con 0x7fbd14005180 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd27fff640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd280c7e20 0x7fbd280c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd280cc2c0 0x7fbd280cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T20:55:20.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.499+0000 7fbd277fe640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd2814d8d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800c3f0 tx=0x7fbd1800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.500+0000 7fbd24ff9640 1 -- 172.21.15.45:0/4189735217 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd180052d0 con 0x7fbd14005180 2023-11-28T20:55:20.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.500+0000 7fbd24ff9640 1 -- 172.21.15.45:0/4189735217 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd18005470 con 0x7fbd14005180 2023-11-28T20:55:20.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.500+0000 7fbd24ff9640 1 -- 172.21.15.45:0/4189735217 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18007260 con 0x7fbd14005180 2023-11-28T20:55:20.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.500+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd280c70c0 con 0x7fbd14005180 2023-11-28T20:55:20.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.500+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd280c7680 con 0x7fbd14005180 2023-11-28T20:55:20.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.501+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd28071ba0 con 0x7fbd14005180 2023-11-28T20:55:20.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.502+0000 7fbd24ff9640 1 -- 172.21.15.45:0/4189735217 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbd180073c0 con 0x7fbd14005180 2023-11-28T20:55:20.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.503+0000 7fbd24ff9640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbd0806af60 0x7fbd0806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:20.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.503+0000 7fbd24ff9640 1 -- 172.21.15.45:0/4189735217 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbd18088c30 con 0x7fbd14005180 2023-11-28T20:55:20.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.504+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbd0806af60 0x7fbd0806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:20.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.505+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbd0806af60 0x7fbd0806d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbd2815a860 tx=0x7fbd10009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:20.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.508+0000 7fbd24ff9640 1 -- 172.21.15.45:0/4189735217 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd18053a60 con 0x7fbd14005180 2023-11-28T20:55:20.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.745+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbd2804e920 con 0x7fbd14005180 2023-11-28T20:55:20.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.745+0000 7fbd24ff9640 1 -- 172.21.15.45:0/4189735217 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v76) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbd18057080 con 0x7fbd14005180 2023-11-28T20:55:20.748 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbd0806af60 msgr2=0x7fbd0806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbd0806af60 0x7fbd0806d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbd2815a860 tx=0x7fbd10009040 comp rx=0 tx=0).stop 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 msgr2=0x7fbd2814d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd2814d8d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800c3f0 tx=0x7fbd1800c8b0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 shutdown_connections 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd280c7e20 0x7fbd280c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd280cc2c0 0x7fbd280cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fbd0806af60 0x7fbd0806d420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 --2- 172.21.15.45:0/4189735217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd14005180 0x7fbd2814d8d0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 >> 172.21.15.45:0/4189735217 conn(0x7fbd2806a500 msgr2=0x7fbd2815f100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.748+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 shutdown_connections 2023-11-28T20:55:20.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:20.749+0000 7fbd2c8d1640 1 -- 172.21.15.45:0/4189735217 wait complete. 2023-11-28T20:55:20.770 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T20:55:20.771 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T20:55:20.771 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T20:55:20.771 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T20:55:20.771 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T20:55:20.771 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T20:55:20.771 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T20:55:20.771 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T20:55:20.771 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:55:20.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:20.771 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T20:55:20.828 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T20:55:20.828 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T20:55:20.828 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T20:55:20.828 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:55:20.828 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T20:55:20.828 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:55:20.828 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:55:20.828 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:55:20.828 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 41022sec preferred_lft 41022sec 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T20:55:20.829 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T20:55:20.830 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T20:55:20.831 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T20:55:20.831 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T20:55:20.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:20.832 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T20:55:20.934 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T20:55:20.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:20.938 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T20:55:20.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:55:20.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:55:20.998 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T20:55:20.998 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T20:55:20.998 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T20:55:20.998 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T20:55:20.998 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T20:55:20.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:20.999 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T20:55:21.000 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T20:55:21.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T20:55:21.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T20:55:21.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T20:55:21.000 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T20:55:21.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:21.229 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T20:55:21.285 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T20:55:23.844 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T20:55:23.845 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:55:23.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:23.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T20:55:23.960 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:55:23.960 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T20:55:24.018 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T20:55:24.018 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T20:55:24.018 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:55:24.075 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T20:55:24.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:24.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:55:24.232 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:55:24.232 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:55:24.232 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:55:24.255 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:55:24.255 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:55:24.255 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:55:24.255 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:55:24.255 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:55:24.282 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T20:55:24.282 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T20:55:24.282 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T20:55:24.282 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T20:55:24.282 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T20:55:24.282 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T20:55:24.282 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T20:55:24.282 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T20:55:24.284 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T20:55:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T20:55:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T20:55:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T20:55:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T20:55:24.288 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T20:55:24.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:24.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T20:55:24.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:24.383 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T20:55:24.454 INFO:teuthology.orchestra.run.smithi045.stdout:client7648 2023-11-28T20:55:24.455 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T20:55:24.455 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T20:55:24.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T20:55:24.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.735+0000 7f6d0b354640 1 Processor -- start 2023-11-28T20:55:24.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.735+0000 7f6d0b354640 1 -- start start 2023-11-28T20:55:24.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.737+0000 7f6d0b354640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d040cf610 0x7f6d040ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:24.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.737+0000 7f6d0b354640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040b2fb0 0x7f6d0415eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:24.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.737+0000 7f6d0b354640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d040abd70 con 0x7f6d0415b070 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.737+0000 7f6d0b354640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d0415b450 con 0x7f6d040cf610 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.737+0000 7f6d0b354640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040b3ed0 con 0x7f6d040b2fb0 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.738+0000 7f6d09b51640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d040cf610 0x7f6d040ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.738+0000 7f6d09b51640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6d040cf610 0x7f6d040ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55336/0 (socket says 172.21.15.45:55336) 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.738+0000 7f6d09b51640 1 -- 172.21.15.45:0/1052827517 learned_addr learned my addr 172.21.15.45:0/1052827517 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.738+0000 7f6d0ab53640 1 --2- 172.21.15.45:0/1052827517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040b2fb0 0x7f6d0415eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.738+0000 7f6d09b51640 1 -- 172.21.15.45:0/1052827517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040b2fb0 msgr2=0x7f6d0415eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.738+0000 7f6d09b51640 1 --2- 172.21.15.45:0/1052827517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040b2fb0 0x7f6d0415eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.738+0000 7f6d09b51640 1 -- 172.21.15.45:0/1052827517 >> v1:172.21.15.190:6789/0 conn(0x7f6d0415b070 legacy=0x7f6d040cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:24.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.738+0000 7f6d09b51640 1 -- 172.21.15.45:0/1052827517 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040abfc0 con 0x7f6d040cf610 2023-11-28T20:55:24.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.739+0000 7f6d09b51640 1 --2- 172.21.15.45:0/1052827517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040cf610 0x7f6d040ae370 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009870 tx=0x7f6cf402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e8418a9aad279b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:24.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.740+0000 7f6d09350640 1 -- 172.21.15.45:0/1052827517 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40317c0 con 0x7f6d040cf610 2023-11-28T20:55:24.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.740+0000 7f6d09350640 1 -- 172.21.15.45:0/1052827517 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf4031920 con 0x7f6d040cf610 2023-11-28T20:55:24.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.740+0000 7f6d09350640 1 -- 172.21.15.45:0/1052827517 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4031cb0 con 0x7f6d040cf610 2023-11-28T20:55:24.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.740+0000 7f6d0b354640 1 -- 172.21.15.45:0/1052827517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040cf610 msgr2=0x7f6d040ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:24.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.740+0000 7f6d0b354640 1 --2- 172.21.15.45:0/1052827517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040cf610 0x7f6d040ae370 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009870 tx=0x7f6cf402b860 comp rx=0 tx=0).stop 2023-11-28T20:55:24.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.740+0000 7f6d0b354640 1 -- 172.21.15.45:0/1052827517 shutdown_connections 2023-11-28T20:55:24.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.740+0000 7f6d0b354640 1 --2- 172.21.15.45:0/1052827517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040b2fb0 0x7f6d0415eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.741+0000 7f6d0b354640 1 --2- 172.21.15.45:0/1052827517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040cf610 0x7f6d040ae370 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.741+0000 7f6d0b354640 1 -- 172.21.15.45:0/1052827517 >> 172.21.15.45:0/1052827517 conn(0x7f6d0406a710 msgr2=0x7f6d0406ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:24.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.741+0000 7f6d0b354640 1 -- 172.21.15.45:0/1052827517 shutdown_connections 2023-11-28T20:55:24.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.741+0000 7f6d0b354640 1 -- 172.21.15.45:0/1052827517 wait complete. 2023-11-28T20:55:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.743+0000 7f6d0b354640 1 Processor -- start 2023-11-28T20:55:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.743+0000 7f6d0b354640 1 -- start start 2023-11-28T20:55:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0b354640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040b2fb0 0x7f6d0409c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0b354640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040cf610 0x7f6d0415dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0b354640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0415b070 0x7f6d041560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0b354640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d041f4680 con 0x7f6d040b2fb0 2023-11-28T20:55:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0b354640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d041f54e0 con 0x7f6d0415b070 2023-11-28T20:55:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0b354640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d041f4860 con 0x7f6d040cf610 2023-11-28T20:55:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0a352640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040b2fb0 0x7f6d0409c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0a352640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040b2fb0 0x7f6d0409c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55350/0 (socket says 172.21.15.45:55350) 2023-11-28T20:55:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0a352640 1 -- 172.21.15.45:0/3050619097 learned_addr learned my addr 172.21.15.45:0/3050619097 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:24.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d09b51640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040cf610 0x7f6d0415dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:24.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0a352640 1 -- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040cf610 msgr2=0x7f6d0415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:24.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0a352640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040cf610 0x7f6d0415dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0a352640 1 -- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0415b070 msgr2=0x7f6d041560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:24.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0a352640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0415b070 0x7f6d041560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.745+0000 7f6d0a352640 1 -- 172.21.15.45:0/3050619097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf40094d0 con 0x7f6d040b2fb0 2023-11-28T20:55:24.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.746+0000 7f6d0a352640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040b2fb0 0x7f6d0409c250 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000b3f0 tx=0x7f6d0000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:24.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.746+0000 7f6cfaffd640 1 -- 172.21.15.45:0/3050619097 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00011830 con 0x7f6d040b2fb0 2023-11-28T20:55:24.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.746+0000 7f6cfaffd640 1 -- 172.21.15.45:0/3050619097 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d00011990 con 0x7f6d040b2fb0 2023-11-28T20:55:24.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.746+0000 7f6cfaffd640 1 -- 172.21.15.45:0/3050619097 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00011d40 con 0x7f6d040b2fb0 2023-11-28T20:55:24.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.746+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d041f4b40 con 0x7f6d040b2fb0 2023-11-28T20:55:24.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.746+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d041f5100 con 0x7f6d040b2fb0 2023-11-28T20:55:24.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.747+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0406c150 con 0x7f6d040b2fb0 2023-11-28T20:55:24.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.749+0000 7f6cfaffd640 1 -- 172.21.15.45:0/3050619097 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6d0001a400 con 0x7f6d040b2fb0 2023-11-28T20:55:24.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.750+0000 7f6cfaffd640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ce806afb0 0x7f6ce806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:24.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.750+0000 7f6cfaffd640 1 -- 172.21.15.45:0/3050619097 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6d00089760 con 0x7f6d040b2fb0 2023-11-28T20:55:24.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.750+0000 7f6d09b51640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ce806afb0 0x7f6ce806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:24.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.752+0000 7f6d09b51640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ce806afb0 0x7f6ce806d470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4036000 tx=0x7f6cf403d040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.754+0000 7f6cfaffd640 1 -- 172.21.15.45:0/3050619097 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d00054580 con 0x7f6d040b2fb0 2023-11-28T20:55:24.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.975+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 --> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d040b3800 con 0x7f6ce806afb0 2023-11-28T20:55:24.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.979+0000 7f6cfaffd640 1 -- 172.21.15.45:0/3050619097 <== mgr.7396 v2:172.21.15.190:6832/226305585 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6cf4031a90 con 0x7f6ce806afb0 2023-11-28T20:55:24.981 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T20:55:24.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T20:55:24.981 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T20:55:24.981 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T20:55:24.981 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T20:55:24.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ce806afb0 msgr2=0x7f6ce806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ce806afb0 0x7f6ce806d470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4036000 tx=0x7f6cf403d040 comp rx=0 tx=0).stop 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040b2fb0 msgr2=0x7f6d0409c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040b2fb0 0x7f6d0409c250 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000b3f0 tx=0x7f6d0000b8b0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 shutdown_connections 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0415b070 0x7f6d041560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d040cf610 0x7f6d0415dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f6ce806afb0 0x7f6ce806d470 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 --2- 172.21.15.45:0/3050619097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d040b2fb0 0x7f6d0409c250 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 >> 172.21.15.45:0/3050619097 conn(0x7f6d0406a710 msgr2=0x7f6d040ceed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 shutdown_connections 2023-11-28T20:55:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:24.981+0000 7f6d0b354640 1 -- 172.21.15.45:0/3050619097 wait complete. 2023-11-28T20:55:25.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T20:55:25.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.212+0000 7f9dd5de3640 1 Processor -- start 2023-11-28T20:55:25.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.213+0000 7f9dd5de3640 1 -- start start 2023-11-28T20:55:25.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.214+0000 7f9dd5de3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dd015e840 0x7f9dd006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.214+0000 7f9dd5de3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd015ecd0 con 0x7f9dd006ac10 2023-11-28T20:55:25.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.214+0000 7f9dd5de3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd015f290 con 0x7f9dd015f720 2023-11-28T20:55:25.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.214+0000 7f9dd5de3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd015fb60 con 0x7f9dd015e840 2023-11-28T20:55:25.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.214+0000 7f9dcffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9dd006ac10 0x7f9dd00cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34964/0 (socket says 172.21.15.45:34964) 2023-11-28T20:55:25.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.214+0000 7f9dcffff640 1 -- 172.21.15.45:0/421344811 learned_addr learned my addr 172.21.15.45:0/421344811 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:25.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664695804 0 0) 0x7f9dd015ecd0 con 0x7f9dd006ac10 2023-11-28T20:55:25.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dcf7fe640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dd015e840 0x7f9dd006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dbc003520 con 0x7f9dd006ac10 2023-11-28T20:55:25.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4235671346 0 0) 0x7f9dbc003520 con 0x7f9dd006ac10 2023-11-28T20:55:25.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dd015e840 msgr2=0x7f9dd006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:25.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dd015e840 0x7f9dd006c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 >> v1:172.21.15.190:6789/0 conn(0x7f9dd015f720 legacy=0x7f9dd006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd0072020 con 0x7f9dd006ac10 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f9dc8001670 con 0x7f9dd006ac10 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 >> v1:172.21.15.45:6789/0 conn(0x7f9dd006ac10 legacy=0x7f9dd00cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dd015e840 0x7f9dbc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dbc003f20 0x7f9dbc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc004d00 0x7f9dbc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd015fb60 con 0x7f9dbc004d00 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc00ba60 con 0x7f9dbc003f20 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.215+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc00bc20 con 0x7f9dd015e840 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.216+0000 7f9dcffff640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc004d00 0x7f9dbc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.216+0000 7f9dceffd640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dbc003f20 0x7f9dbc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.216+0000 7f9dcf7fe640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dd015e840 0x7f9dbc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.216+0000 7f9dcffff640 1 -- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dd015e840 msgr2=0x7f9dbc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:25.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.216+0000 7f9dcffff640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dd015e840 0x7f9dbc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.216+0000 7f9dcffff640 1 -- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dbc003f20 msgr2=0x7f9dbc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:25.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.216+0000 7f9dcffff640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dbc003f20 0x7f9dbc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.216+0000 7f9dcffff640 1 -- 172.21.15.45:0/421344811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc003520 con 0x7f9dbc004d00 2023-11-28T20:55:25.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.217+0000 7f9dcffff640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc004d00 0x7f9dbc005180 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c240 tx=0x7f9dbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4cdf4c7a7e7855 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:25.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.218+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8005ad0 con 0x7f9dbc004d00 2023-11-28T20:55:25.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.218+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc8005c30 con 0x7f9dbc004d00 2023-11-28T20:55:25.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.218+0000 7f9dce7fc640 1 -- 172.21.15.45:0/421344811 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc802caf0 con 0x7f9dbc004d00 2023-11-28T20:55:25.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.218+0000 7f9dd5de3640 1 -- 172.21.15.45:0/421344811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc004d00 msgr2=0x7f9dbc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.218+0000 7f9dd5de3640 1 --2- 172.21.15.45:0/421344811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc004d00 0x7f9dbc005180 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c240 tx=0x7f9dbc0057f0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.218+0000 7f9dcffff640 1 -- 172.21.15.45:0/421344811 reap_dead start 2023-11-28T20:55:25.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.218+0000 7f9dd5de3640 1 -- 172.21.15.45:0/421344811 shutdown_connections 2023-11-28T20:55:25.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.218+0000 7f9dd5de3640 1 -- 172.21.15.45:0/421344811 >> 172.21.15.45:0/421344811 conn(0x7f9dd00ced60 msgr2=0x7f9dd00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:25.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.219+0000 7f9dd5de3640 1 -- 172.21.15.45:0/421344811 shutdown_connections 2023-11-28T20:55:25.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.219+0000 7f9dd5de3640 1 -- 172.21.15.45:0/421344811 wait complete. 2023-11-28T20:55:25.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.220+0000 7f9dd5de3640 1 Processor -- start 2023-11-28T20:55:25.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.220+0000 7f9dd5de3640 1 -- start start 2023-11-28T20:55:25.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.221+0000 7f9dd5de3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dbc004d00 0x7f9dd015d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.221+0000 7f9dd5de3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dd009bc90 0x7f9dd009c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.221+0000 7f9dd5de3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dd0099b90 0x7f9dd009a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.221+0000 7f9dd5de3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd009c650 con 0x7f9dd009bc90 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.221+0000 7f9dd5de3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd015d8e0 con 0x7f9dd0099b90 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.221+0000 7f9dd5de3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd009a8c0 con 0x7f9dbc004d00 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dd009bc90 0x7f9dd009c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dd009bc90 0x7f9dd009c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55374/0 (socket says 172.21.15.45:55374) 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dceffd640 1 -- 172.21.15.45:0/2394030157 learned_addr learned my addr 172.21.15.45:0/2394030157 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dcffff640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dd0099b90 0x7f9dd009a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dcf7fe640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dbc004d00 0x7f9dd015d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dceffd640 1 -- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dbc004d00 msgr2=0x7f9dd015d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dceffd640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dbc004d00 0x7f9dd015d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dceffd640 1 -- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dd0099b90 msgr2=0x7f9dd009a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dceffd640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dd0099b90 0x7f9dd009a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.222+0000 7f9dceffd640 1 -- 172.21.15.45:0/2394030157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd009aba0 con 0x7f9dd009bc90 2023-11-28T20:55:25.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.223+0000 7f9dceffd640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dd009bc90 0x7f9dd009c110 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f9db800e550 tx=0x7f9db800ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:25.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.223+0000 7f9dc7fff640 1 -- 172.21.15.45:0/2394030157 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db80052e0 con 0x7f9dd009bc90 2023-11-28T20:55:25.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.223+0000 7f9dc7fff640 1 -- 172.21.15.45:0/2394030157 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db8005440 con 0x7f9dd009bc90 2023-11-28T20:55:25.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.223+0000 7f9dc7fff640 1 -- 172.21.15.45:0/2394030157 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8003810 con 0x7f9dd009bc90 2023-11-28T20:55:25.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.223+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd009ae80 con 0x7f9dd009bc90 2023-11-28T20:55:25.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.223+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd009a550 con 0x7f9dd009bc90 2023-11-28T20:55:25.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.225+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd00733d0 con 0x7f9dd009bc90 2023-11-28T20:55:25.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.226+0000 7f9dc7fff640 1 -- 172.21.15.45:0/2394030157 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 78) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9db8003970 con 0x7f9dd009bc90 2023-11-28T20:55:25.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.226+0000 7f9dc7fff640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9db006af90 0x7f9db006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.227+0000 7f9dc7fff640 1 -- 172.21.15.45:0/2394030157 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9db80895b0 con 0x7f9dd009bc90 2023-11-28T20:55:25.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.227+0000 7f9dcf7fe640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9db006af90 0x7f9db006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.228+0000 7f9dcf7fe640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9db006af90 0x7f9db006d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00ac0b0 tx=0x7f9dc000b040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:25.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.231+0000 7f9dc7fff640 1 -- 172.21.15.45:0/2394030157 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9db8054480 con 0x7f9dd009bc90 2023-11-28T20:55:25.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.475+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f9dd00ce2b0 con 0x7f9dd009bc90 2023-11-28T20:55:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.520+0000 7f9dc7fff640 1 -- 172.21.15.45:0/2394030157 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v79) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9db8057aa0 con 0x7f9dd009bc90 2023-11-28T20:55:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.523+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9db006af90 msgr2=0x7f9db006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.523+0000 7f9dd5de3640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9db006af90 0x7f9db006d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00ac0b0 tx=0x7f9dc000b040 comp rx=0 tx=0).stop 2023-11-28T20:55:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.523+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dd009bc90 msgr2=0x7f9dd009c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.523+0000 7f9dd5de3640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dd009bc90 0x7f9dd009c110 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f9db800e550 tx=0x7f9db800ea10 comp rx=0 tx=0).stop 2023-11-28T20:55:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.524+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 shutdown_connections 2023-11-28T20:55:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.524+0000 7f9dd5de3640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dd0099b90 0x7f9dd009a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.524+0000 7f9dd5de3640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f9db006af90 0x7f9db006d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.524+0000 7f9dd5de3640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dd009bc90 0x7f9dd009c110 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.524+0000 7f9dd5de3640 1 --2- 172.21.15.45:0/2394030157 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9dbc004d00 0x7f9dd015d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.524+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 >> 172.21.15.45:0/2394030157 conn(0x7f9dd00ced60 msgr2=0x7f9dd006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.524+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 shutdown_connections 2023-11-28T20:55:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.525+0000 7f9dd5de3640 1 -- 172.21.15.45:0/2394030157 wait complete. 2023-11-28T20:55:25.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T20:55:25.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.742+0000 7fdd6759e640 1 Processor -- start 2023-11-28T20:55:25.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.742+0000 7fdd6759e640 1 -- start start 2023-11-28T20:55:25.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.743+0000 7fdd6759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 0x7fdd680d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.743+0000 7fdd6759e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdd680b2290 0x7fdd680d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.743+0000 7fdd6759e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd680b1d40 0x7fdd680d3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.743+0000 7fdd6759e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd680b90a0 con 0x7fdd680b2290 2023-11-28T20:55:25.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.743+0000 7fdd6759e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd680d1920 con 0x7fdd680b1d40 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.743+0000 7fdd6759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680d0960 con 0x7fdd680d0040 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd5ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdd680b2290 0x7fdd680d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd5ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdd680b2290 0x7fdd680d3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55410/0 (socket says 172.21.15.45:55410) 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd5ffff640 1 -- 172.21.15.45:0/3654850311 learned_addr learned my addr 172.21.15.45:0/3654850311 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd6659c640 1 --2- 172.21.15.45:0/3654850311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 0x7fdd680d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd66d9d640 1 --2- 172.21.15.45:0/3654850311 >> v2:172.21.15.190:3300/0 conn(0x7fdd680b1d40 0x7fdd680d3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd5ffff640 1 -- 172.21.15.45:0/3654850311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 msgr2=0x7fdd680d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd5ffff640 1 --2- 172.21.15.45:0/3654850311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 0x7fdd680d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd5ffff640 1 -- 172.21.15.45:0/3654850311 >> v2:172.21.15.190:3300/0 conn(0x7fdd680b1d40 msgr2=0x7fdd680d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd5ffff640 1 --2- 172.21.15.45:0/3654850311 >> v2:172.21.15.190:3300/0 conn(0x7fdd680b1d40 0x7fdd680d3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.744+0000 7fdd5ffff640 1 -- 172.21.15.45:0/3654850311 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd680b8ae0 con 0x7fdd680b2290 2023-11-28T20:55:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.745+0000 7fdd5ffff640 1 --2- 172.21.15.45:0/3654850311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 0x7fdd680d3040 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fdd680b5c70 tx=0x7fdd5002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecd3da2cde19ebc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd65d9b640 1 -- 172.21.15.45:0/3654850311 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50030860 con 0x7fdd680b2290 2023-11-28T20:55:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd65d9b640 1 -- 172.21.15.45:0/3654850311 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd500309c0 con 0x7fdd680b2290 2023-11-28T20:55:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd65d9b640 1 -- 172.21.15.45:0/3654850311 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50030d50 con 0x7fdd680b2290 2023-11-28T20:55:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd6759e640 1 -- 172.21.15.45:0/3654850311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 msgr2=0x7fdd680d3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd6759e640 1 --2- 172.21.15.45:0/3654850311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 0x7fdd680d3040 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fdd680b5c70 tx=0x7fdd5002b9a0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd6759e640 1 -- 172.21.15.45:0/3654850311 shutdown_connections 2023-11-28T20:55:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd6759e640 1 --2- 172.21.15.45:0/3654850311 >> v2:172.21.15.190:3300/0 conn(0x7fdd680b1d40 0x7fdd680d3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd6759e640 1 --2- 172.21.15.45:0/3654850311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 0x7fdd680d3040 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd6759e640 1 --2- 172.21.15.45:0/3654850311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 0x7fdd680d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.746+0000 7fdd6759e640 1 -- 172.21.15.45:0/3654850311 >> 172.21.15.45:0/3654850311 conn(0x7fdd68070f40 msgr2=0x7fdd68071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.747+0000 7fdd6759e640 1 -- 172.21.15.45:0/3654850311 shutdown_connections 2023-11-28T20:55:25.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.747+0000 7fdd6759e640 1 -- 172.21.15.45:0/3654850311 wait complete. 2023-11-28T20:55:25.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.749+0000 7fdd6759e640 1 Processor -- start 2023-11-28T20:55:25.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.749+0000 7fdd6759e640 1 -- start start 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.750+0000 7fdd6759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd680b1d40 0x7fdd681f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.750+0000 7fdd6759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 0x7fdd680c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.750+0000 7fdd6759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 0x7fdd680c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.750+0000 7fdd6759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680cb220 con 0x7fdd680b2290 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.750+0000 7fdd6759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680ca180 con 0x7fdd680b1d40 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.750+0000 7fdd6759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680ca460 con 0x7fdd680d0040 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.750+0000 7fdd5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 0x7fdd680c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 0x7fdd680c8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55418/0 (socket says 172.21.15.45:55418) 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd5ffff640 1 -- 172.21.15.45:0/1325226873 learned_addr learned my addr 172.21.15.45:0/1325226873 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd66d9d640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 0x7fdd680c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd6659c640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd680b1d40 0x7fdd681f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd5ffff640 1 -- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 msgr2=0x7fdd680c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd5ffff640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 0x7fdd680c8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd5ffff640 1 -- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd680b1d40 msgr2=0x7fdd681f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd5ffff640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd680b1d40 0x7fdd681f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd5ffff640 1 -- 172.21.15.45:0/1325226873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500094d0 con 0x7fdd680b2290 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd66d9d640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 0x7fdd680c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.751+0000 7fdd5ffff640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 0x7fdd680c8910 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdd5002b9e0 tx=0x7fdd50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.752+0000 7fdd5effd640 1 -- 172.21.15.45:0/1325226873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50039d60 con 0x7fdd680b2290 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.752+0000 7fdd5effd640 1 -- 172.21.15.45:0/1325226873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd50043400 con 0x7fdd680b2290 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.752+0000 7fdd5effd640 1 -- 172.21.15.45:0/1325226873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd500437b0 con 0x7fdd680b2290 2023-11-28T20:55:25.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.752+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd680ca740 con 0x7fdd680b2290 2023-11-28T20:55:25.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.752+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd680cad00 con 0x7fdd680b2290 2023-11-28T20:55:25.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.753+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd68071b30 con 0x7fdd680b2290 2023-11-28T20:55:25.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.754+0000 7fdd5effd640 1 -- 172.21.15.45:0/1325226873 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 79) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fdd50043910 con 0x7fdd680b2290 2023-11-28T20:55:25.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.755+0000 7fdd5effd640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdd4806afb0 0x7fdd4806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:25.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.755+0000 7fdd5effd640 1 -- 172.21.15.45:0/1325226873 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd5003e070 con 0x7fdd680b2290 2023-11-28T20:55:25.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.755+0000 7fdd6659c640 1 -- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdd4806afb0 msgr2=0x7fdd4806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/226305585 2023-11-28T20:55:25.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.755+0000 7fdd6659c640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdd4806afb0 0x7fdd4806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:55:25.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.760+0000 7fdd5effd640 1 -- 172.21.15.45:0/1325226873 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd50077b20 con 0x7fdd680b2290 2023-11-28T20:55:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.952+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fdd6804e920 con 0x7fdd680b2290 2023-11-28T20:55:25.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.956+0000 7fdd6659c640 1 -- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdd4806afb0 msgr2=0x7fdd4806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/226305585 2023-11-28T20:55:25.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.956+0000 7fdd6659c640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdd4806afb0 0x7fdd4806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:55:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.957+0000 7fdd5effd640 1 -- 172.21.15.45:0/1325226873 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdd5007b140 con 0x7fdd680b2290 2023-11-28T20:55:25.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdd4806afb0 msgr2=0x7fdd4806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:55:25.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdd4806afb0 0x7fdd4806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 msgr2=0x7fdd680c8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:25.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 0x7fdd680c8910 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdd5002b9e0 tx=0x7fdd50004730 comp rx=0 tx=0).stop 2023-11-28T20:55:25.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 shutdown_connections 2023-11-28T20:55:25.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd680d0040 0x7fdd680c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fdd4806afb0 0x7fdd4806d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd680b2290 0x7fdd680c8910 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 --2- 172.21.15.45:0/1325226873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd680b1d40 0x7fdd681f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:25.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 >> 172.21.15.45:0/1325226873 conn(0x7fdd68070f40 msgr2=0x7fdd6806a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:25.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 shutdown_connections 2023-11-28T20:55:25.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:25.960+0000 7fdd6759e640 1 -- 172.21.15.45:0/1325226873 wait complete. 2023-11-28T20:55:25.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T20:55:26.044 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.042+0000 7f5c0dc21040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:55:26.053 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:26.051+0000 7f4141c0e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T20:55:26.096 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.094+0000 7f5c0dc21040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:55:26.110 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:26.108+0000 7f4141c0e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T20:55:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.199+0000 7fa9817b6640 1 Processor -- start 2023-11-28T20:55:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.199+0000 7fa9817b6640 1 -- start start 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa9817b6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa97c15d180 0x7fa97c06dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa9817b6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa97c06e7d0 0x7fa97c0ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa9817b6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97c0b29b0 con 0x7fa97c15d560 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa9817b6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97c160fd0 con 0x7fa97c06e7d0 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa9817b6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97c0b2c00 con 0x7fa97c15d180 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa97a7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa97c06e7d0 0x7fa97c0ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa97a7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa97c06e7d0 0x7fa97c0ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55420/0 (socket says 172.21.15.45:55420) 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa97a7fc640 1 -- 172.21.15.45:0/1113825785 learned_addr learned my addr 172.21.15.45:0/1113825785 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa97affd640 1 --2- 172.21.15.45:0/1113825785 >> v2:172.21.15.190:3300/0 conn(0x7fa97c15d180 0x7fa97c06dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa97a7fc640 1 -- 172.21.15.45:0/1113825785 >> v2:172.21.15.190:3300/0 conn(0x7fa97c15d180 msgr2=0x7fa97c06dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:26.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa97a7fc640 1 --2- 172.21.15.45:0/1113825785 >> v2:172.21.15.190:3300/0 conn(0x7fa97c15d180 0x7fa97c06dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.201+0000 7fa97a7fc640 1 -- 172.21.15.45:0/1113825785 >> v1:172.21.15.190:6789/0 conn(0x7fa97c15d560 legacy=0x7fa97c0cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:26.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.202+0000 7fa97a7fc640 1 -- 172.21.15.45:0/1113825785 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c0b2ee0 con 0x7fa97c06e7d0 2023-11-28T20:55:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.203+0000 7fa97a7fc640 1 --2- 172.21.15.45:0/1113825785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c06e7d0 0x7fa97c0ac8b0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0b2980 tx=0x7fa97002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6db02978dded26 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.203+0000 7fa979ffb640 1 -- 172.21.15.45:0/1113825785 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970031870 con 0x7fa97c06e7d0 2023-11-28T20:55:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.203+0000 7fa979ffb640 1 -- 172.21.15.45:0/1113825785 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9700319d0 con 0x7fa97c06e7d0 2023-11-28T20:55:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.203+0000 7fa979ffb640 1 -- 172.21.15.45:0/1113825785 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970031d60 con 0x7fa97c06e7d0 2023-11-28T20:55:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.203+0000 7fa9817b6640 1 -- 172.21.15.45:0/1113825785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c06e7d0 msgr2=0x7fa97c0ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.203+0000 7fa9817b6640 1 --2- 172.21.15.45:0/1113825785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c06e7d0 0x7fa97c0ac8b0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0b2980 tx=0x7fa97002b910 comp rx=0 tx=0).stop 2023-11-28T20:55:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.204+0000 7fa9817b6640 1 -- 172.21.15.45:0/1113825785 shutdown_connections 2023-11-28T20:55:26.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.204+0000 7fa9817b6640 1 --2- 172.21.15.45:0/1113825785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c06e7d0 0x7fa97c0ac8b0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.204+0000 7fa9817b6640 1 --2- 172.21.15.45:0/1113825785 >> v2:172.21.15.190:3300/0 conn(0x7fa97c15d180 0x7fa97c06dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.204+0000 7fa9817b6640 1 -- 172.21.15.45:0/1113825785 >> 172.21.15.45:0/1113825785 conn(0x7fa97c06a500 msgr2=0x7fa97c06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.204+0000 7fa9817b6640 1 -- 172.21.15.45:0/1113825785 shutdown_connections 2023-11-28T20:55:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.204+0000 7fa9817b6640 1 -- 172.21.15.45:0/1113825785 wait complete. 2023-11-28T20:55:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.205+0000 7fa9817b6640 1 Processor -- start 2023-11-28T20:55:26.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.206+0000 7fa9817b6640 1 -- start start 2023-11-28T20:55:26.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.206+0000 7fa9817b6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa97c06e7d0 0x7fa97c153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.206+0000 7fa9817b6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa97c15d180 0x7fa97c15f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.206+0000 7fa9817b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c15d560 0x7fa97c151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.206+0000 7fa9817b6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c154450 con 0x7fa97c15d560 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.206+0000 7fa9817b6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c140c70 con 0x7fa97c15d180 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.206+0000 7fa9817b6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c141350 con 0x7fa97c06e7d0 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c15d560 0x7fa97c151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c15d560 0x7fa97c151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55426/0 (socket says 172.21.15.45:55426) 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97b7fe640 1 -- 172.21.15.45:0/3410220323 learned_addr learned my addr 172.21.15.45:0/3410220323 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97affd640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa97c06e7d0 0x7fa97c153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97a7fc640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa97c15d180 0x7fa97c15f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97b7fe640 1 -- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa97c06e7d0 msgr2=0x7fa97c153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97b7fe640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa97c06e7d0 0x7fa97c153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97b7fe640 1 -- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa97c15d180 msgr2=0x7fa97c15f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:26.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97b7fe640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa97c15d180 0x7fa97c15f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.207+0000 7fa97b7fe640 1 -- 172.21.15.45:0/3410220323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9700094d0 con 0x7fa97c15d560 2023-11-28T20:55:26.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.208+0000 7fa97b7fe640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c15d560 0x7fa97c151ef0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fa97400b9e0 tx=0x7fa97400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:26.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.208+0000 7fa9637fe640 1 -- 172.21.15.45:0/3410220323 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97400caa0 con 0x7fa97c15d560 2023-11-28T20:55:26.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.208+0000 7fa9637fe640 1 -- 172.21.15.45:0/3410220323 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97400cc00 con 0x7fa97c15d560 2023-11-28T20:55:26.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.208+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c141630 con 0x7fa97c15d560 2023-11-28T20:55:26.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.208+0000 7fa9637fe640 1 -- 172.21.15.45:0/3410220323 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974014440 con 0x7fa97c15d560 2023-11-28T20:55:26.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.208+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c141bf0 con 0x7fa97c15d560 2023-11-28T20:55:26.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.209+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c06b490 con 0x7fa97c15d560 2023-11-28T20:55:26.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.211+0000 7fa9637fe640 1 -- 172.21.15.45:0/3410220323 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 79) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fa97401d3b0 con 0x7fa97c15d560 2023-11-28T20:55:26.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.212+0000 7fa9637fe640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa96806afb0 0x7fa96806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.212+0000 7fa9637fe640 1 -- 172.21.15.45:0/3410220323 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa974089680 con 0x7fa97c15d560 2023-11-28T20:55:26.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.212+0000 7fa97affd640 1 -- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa96806afb0 msgr2=0x7fa96806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/226305585 2023-11-28T20:55:26.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.212+0000 7fa97affd640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa96806afb0 0x7fa96806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:55:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.221+0000 7fa9637fe640 1 -- 172.21.15.45:0/3410220323 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9740544d0 con 0x7fa97c15d560 2023-11-28T20:55:26.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.413+0000 7fa97affd640 1 -- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa96806afb0 msgr2=0x7fa96806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/226305585 2023-11-28T20:55:26.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.413+0000 7fa97affd640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa96806afb0 0x7fa96806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:55:26.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.418+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa97c0733d0 con 0x7fa97c15d560 2023-11-28T20:55:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.422+0000 7fa9637fe640 1 -- 172.21.15.45:0/3410220323 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa974057af0 con 0x7fa97c15d560 2023-11-28T20:55:26.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.424+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa96806afb0 msgr2=0x7fa96806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:55:26.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.424+0000 7fa9817b6640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa96806afb0 0x7fa96806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.424+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c15d560 msgr2=0x7fa97c151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:26.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.424+0000 7fa9817b6640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c15d560 0x7fa97c151ef0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fa97400b9e0 tx=0x7fa97400bea0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.425+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 shutdown_connections 2023-11-28T20:55:26.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.425+0000 7fa9817b6640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7fa96806afb0 0x7fa96806d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.425+0000 7fa9817b6640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97c15d560 0x7fa97c151ef0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.425+0000 7fa9817b6640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa97c15d180 0x7fa97c15f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.425+0000 7fa9817b6640 1 --2- 172.21.15.45:0/3410220323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa97c06e7d0 0x7fa97c153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.425+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 >> 172.21.15.45:0/3410220323 conn(0x7fa97c06a500 msgr2=0x7fa97c1581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.425+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 shutdown_connections 2023-11-28T20:55:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.425+0000 7fa9817b6640 1 -- 172.21.15.45:0/3410220323 wait complete. 2023-11-28T20:55:26.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T20:55:26.447 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-11-28T20:55:26.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2023-11-28T20:55:26.549 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.547+0000 7f5c0dc21040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:55:26.595 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:26.593+0000 7f4141c0e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T20:55:26.601 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.598+0000 7f5c0dc21040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:55:26.651 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:26.648+0000 7f4141c0e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T20:55:26.651 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.648+0000 7f5c0dc21040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:55:26.704 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:26.702+0000 7f4141c0e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T20:55:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.703+0000 7f61fe30f640 1 Processor -- start 2023-11-28T20:55:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.704+0000 7f61fe30f640 1 -- start start 2023-11-28T20:55:26.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.705+0000 7f61fe30f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61f8160aa0 0x7f61f80b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.705+0000 7f61fe30f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f61f80d24f0 0x7f61f80d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.705+0000 7f61fe30f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f8160e80 con 0x7f61f80ce0e0 2023-11-28T20:55:26.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.705+0000 7f61fe30f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f80d1220 con 0x7f61f80d24f0 2023-11-28T20:55:26.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.705+0000 7f61fe30f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80d2e70 con 0x7f61f8160aa0 2023-11-28T20:55:26.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.705+0000 7f61effff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f61f80d24f0 0x7f61f80d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.705+0000 7f61effff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f61f80d24f0 0x7f61f80d2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55438/0 (socket says 172.21.15.45:55438) 2023-11-28T20:55:26.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.705+0000 7f61effff640 1 -- 172.21.15.45:0/1259196400 learned_addr learned my addr 172.21.15.45:0/1259196400 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:26.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.706+0000 7f61f7fff640 1 --2- 172.21.15.45:0/1259196400 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61f8160aa0 0x7f61f80b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.706+0000 7f61effff640 1 -- 172.21.15.45:0/1259196400 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61f8160aa0 msgr2=0x7f61f80b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:26.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.706+0000 7f61effff640 1 --2- 172.21.15.45:0/1259196400 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61f8160aa0 0x7f61f80b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.706+0000 7f61effff640 1 -- 172.21.15.45:0/1259196400 >> v1:172.21.15.45:6789/0 conn(0x7f61f80ce0e0 legacy=0x7f61f80ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:26.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.706+0000 7f61effff640 1 -- 172.21.15.45:0/1259196400 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f80b6960 con 0x7f61f80d24f0 2023-11-28T20:55:26.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.707+0000 7f61effff640 1 --2- 172.21.15.45:0/1259196400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f80d24f0 0x7f61f80d2930 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009870 tx=0x7f61e002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d3c5eef782ea57 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:26.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.707+0000 7f61f77fe640 1 -- 172.21.15.45:0/1259196400 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e002f830 con 0x7f61f80d24f0 2023-11-28T20:55:26.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.708+0000 7f61f77fe640 1 -- 172.21.15.45:0/1259196400 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e002f990 con 0x7f61f80d24f0 2023-11-28T20:55:26.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.708+0000 7f61f77fe640 1 -- 172.21.15.45:0/1259196400 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e002fd20 con 0x7f61f80d24f0 2023-11-28T20:55:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.708+0000 7f61fe30f640 1 -- 172.21.15.45:0/1259196400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f80d24f0 msgr2=0x7f61f80d2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.708+0000 7f61fe30f640 1 --2- 172.21.15.45:0/1259196400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f80d24f0 0x7f61f80d2930 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009870 tx=0x7f61e002ba10 comp rx=0 tx=0).stop 2023-11-28T20:55:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.708+0000 7f61fe30f640 1 -- 172.21.15.45:0/1259196400 shutdown_connections 2023-11-28T20:55:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.708+0000 7f61fe30f640 1 --2- 172.21.15.45:0/1259196400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f80d24f0 0x7f61f80d2930 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.708+0000 7f61fe30f640 1 --2- 172.21.15.45:0/1259196400 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61f8160aa0 0x7f61f80b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.708+0000 7f61fe30f640 1 -- 172.21.15.45:0/1259196400 >> 172.21.15.45:0/1259196400 conn(0x7f61f8070b20 msgr2=0x7f61f8070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.708+0000 7f61fe30f640 1 -- 172.21.15.45:0/1259196400 shutdown_connections 2023-11-28T20:55:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.709+0000 7f61fe30f640 1 -- 172.21.15.45:0/1259196400 wait complete. 2023-11-28T20:55:26.712 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.710+0000 7f5c0dc21040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:55:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.710+0000 7f61fe30f640 1 Processor -- start 2023-11-28T20:55:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.710+0000 7f61fe30f640 1 -- start start 2023-11-28T20:55:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.711+0000 7f61fe30f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61f80ce0e0 0x7f61f80b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.711+0000 7f61fe30f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61f80d24f0 0x7f61f81f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fe30f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f8160aa0 0x7f61f81f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fe30f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80b58b0 con 0x7f61f8160aa0 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fe30f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f81f3bc0 con 0x7f61f80d24f0 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fe30f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f81f3ea0 con 0x7f61f80ce0e0 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fc8ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f8160aa0 0x7f61f81f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fc8ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f8160aa0 0x7f61f81f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55446/0 (socket says 172.21.15.45:55446) 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fc8ac640 1 -- 172.21.15.45:0/1946084901 learned_addr learned my addr 172.21.15.45:0/1946084901 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61effff640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61f80d24f0 0x7f61f81f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61f7fff640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61f80ce0e0 0x7f61f80b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fc8ac640 1 -- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61f80ce0e0 msgr2=0x7f61f80b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fc8ac640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61f80ce0e0 0x7f61f80b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fc8ac640 1 -- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61f80d24f0 msgr2=0x7f61f81f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fc8ac640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61f80d24f0 0x7f61f81f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.712+0000 7f61fc8ac640 1 -- 172.21.15.45:0/1946084901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e00094d0 con 0x7f61f8160aa0 2023-11-28T20:55:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.713+0000 7f61fc8ac640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f8160aa0 0x7f61f81f3520 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f61f000c7a0 tx=0x7f61f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.713+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0003b70 con 0x7f61f8160aa0 2023-11-28T20:55:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.713+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f0003cf0 con 0x7f61f8160aa0 2023-11-28T20:55:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.713+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0003ea0 con 0x7f61f8160aa0 2023-11-28T20:55:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.713+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f81f41a0 con 0x7f61f8160aa0 2023-11-28T20:55:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.713+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f81f4760 con 0x7f61f8160aa0 2023-11-28T20:55:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.714+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f80733d0 con 0x7f61f8160aa0 2023-11-28T20:55:26.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.716+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 79) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f61f000aa90 con 0x7f61f8160aa0 2023-11-28T20:55:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.717+0000 7f61f57fa640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 0x7f61d806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.717+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f61f0089820 con 0x7f61f8160aa0 2023-11-28T20:55:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.717+0000 7f61f7fff640 1 -- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 msgr2=0x7f61d806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/226305585 2023-11-28T20:55:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.717+0000 7f61f7fff640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 0x7f61d806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T20:55:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.722+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61f00545c0 con 0x7f61f8160aa0 2023-11-28T20:55:26.761 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.759+0000 7f5c0dc21040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:55:26.771 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:26.769+0000 7f4141c0e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T20:55:26.823 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:26.821+0000 7f4141c0e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T20:55:26.872 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.870+0000 7f5c0dc21040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:55:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.918+0000 7f61f7fff640 1 -- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 msgr2=0x7f61d806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/226305585 2023-11-28T20:55:26.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.918+0000 7f61f7fff640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 0x7f61d806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T20:55:26.934 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.932+0000 7f5c0dc21040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:55:26.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:26.935+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 --> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f61f80d1a30 con 0x7f61d806afb0 2023-11-28T20:55:26.940 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:26.938+0000 7f4141c0e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T20:55:26.983 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:26.981+0000 7f5c0dc21040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:55:27.006 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:27.003+0000 7f4141c0e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T20:55:27.041 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:27.039+0000 7f5c0dc21040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:55:27.057 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:27.055+0000 7f4141c0e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T20:55:27.091 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:27.089+0000 7f5c0dc21040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:55:27.116 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:27.114+0000 7f4141c0e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T20:55:27.169 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:27.167+0000 7f4141c0e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T20:55:27.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:27.319+0000 7f61f7fff640 1 -- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 msgr2=0x7f61d806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/226305585 2023-11-28T20:55:27.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:27.319+0000 7f61f7fff640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 0x7f61d806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T20:55:27.470 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:27.469+0000 7f5c0dc21040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:55:27.570 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:27.568+0000 7f4141c0e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T20:55:27.629 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:27.628+0000 7f5c0dc21040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:55:27.737 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:27.735+0000 7f4141c0e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T20:55:27.737 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:27.735+0000 7f5c0dc21040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:55:27.786 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:27.785+0000 7f5c0dc21040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:55:27.850 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:27.848+0000 7f4141c0e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T20:55:27.894 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:27.893+0000 7f5c0dc21040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:55:27.902 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:27.900+0000 7f4141c0e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T20:55:28.000 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:27.998+0000 7f5c0dc21040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:55:28.015 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:28.013+0000 7f4141c0e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:55:28.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:28.120+0000 7f61f7fff640 1 -- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 msgr2=0x7f61d806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/226305585 2023-11-28T20:55:28.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:28.121+0000 7f61f7fff640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 0x7f61d806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T20:55:28.127 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:28.125+0000 7f4141c0e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T20:55:28.157 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:28.155+0000 7f5c0dc21040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:55:28.205 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:28.204+0000 7f5c0dc21040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:55:28.296 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:28.293+0000 7f4141c0e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T20:55:28.328 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:28.327+0000 7f5c0dc21040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:55:28.347 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:28.345+0000 7f4141c0e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T20:55:28.473 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:28.470+0000 7f4141c0e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T20:55:29.157 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.155+0000 7f5c0dc21040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:55:29.351 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:29.349+0000 7f4141c0e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T20:55:29.459 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.458+0000 7f5c0dc21040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:55:29.517 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.515+0000 7f5c0dc21040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:55:29.631 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.630+0000 7f5c0dc21040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:55:29.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:29.650+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 80) v1 ==== 88959+0+0 (secure 0 0 0) 0x7f61f000ad40 con 0x7f61f8160aa0 2023-11-28T20:55:29.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:29.650+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 msgr2=0x7f61d806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T20:55:29.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:29.650+0000 7f61f57fa640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/226305585,v1:172.21.15.190:6833/226305585] conn(0x7f61d806afb0 0x7f61d806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:29.670 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:29.668+0000 7f4141c0e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T20:55:29.735 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:29.733+0000 7f4141c0e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T20:55:29.800 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8a257640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.800 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8a257640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.800 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8a257640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.801 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8a257640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.801 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8a257640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.801 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8d25d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.801 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8d25d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.801 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8d25d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.801 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8d25d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.801 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:55:29.799+0000 7f5b8d25d640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:55:29.862 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T20:55:29.860+0000 7f4141c0e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T20:55:30.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.657+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 81) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f61f0054990 con 0x7f61f8160aa0 2023-11-28T20:55:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.658+0000 7f61f57fa640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f61d80358b0 0x7f61d80753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.658+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 --> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f61d8035c90 con 0x7f61d80358b0 2023-11-28T20:55:30.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.661+0000 7f61f7fff640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f61d80358b0 0x7f61d80753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:30.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.664+0000 7f61f7fff640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f61d80358b0 0x7f61d80753e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f61f80b5ce0 tx=0x7f61e8008040 comp rx=0 tx=0).ready entity=mgr.7663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:30.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.836+0000 7f61f57fa640 1 -- 172.21.15.45:0/1946084901 <== mgr.7663 v2:172.21.15.190:6832/560718832 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f61d8035c90 con 0x7f61d80358b0 2023-11-28T20:55:30.838 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1 2023-11-28T20:55:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.839+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f61d80358b0 msgr2=0x7f61d80753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:30.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.839+0000 7f61fe30f640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f61d80358b0 0x7f61d80753e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f61f80b5ce0 tx=0x7f61e8008040 comp rx=0 tx=0).stop 2023-11-28T20:55:30.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.839+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f8160aa0 msgr2=0x7f61f81f3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:30.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.839+0000 7f61fe30f640 1 --2- 172.21.15.45:0/1946084901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f8160aa0 0x7f61f81f3520 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f61f000c7a0 tx=0x7f61f000cc60 comp rx=0 tx=0).stop 2023-11-28T20:55:30.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.839+0000 7f61fc8ac640 1 -- 172.21.15.45:0/1946084901 reap_dead start 2023-11-28T20:55:30.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.840+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 shutdown_connections 2023-11-28T20:55:30.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.840+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 >> 172.21.15.45:0/1946084901 conn(0x7f61f8070b20 msgr2=0x7f61f815e230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:30.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.840+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 shutdown_connections 2023-11-28T20:55:30.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:30.840+0000 7f61fe30f640 1 -- 172.21.15.45:0/1946084901 wait complete. 2023-11-28T20:55:30.862 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-11-28T20:55:30.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2023-11-28T20:55:31.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.085+0000 7fcfd4261640 1 Processor -- start 2023-11-28T20:55:31.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.085+0000 7fcfd4261640 1 -- start start 2023-11-28T20:55:31.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.087+0000 7fcfd4261640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcfcc06dc50 0x7fcfcc0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.087+0000 7fcfd4261640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc0ce600 0x7fcfcc06a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.087+0000 7fcfd4261640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcfcc06aa40 0x7fcfcc0c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.087+0000 7fcfd4261640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfcc06aed0 con 0x7fcfcc06aa40 2023-11-28T20:55:31.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.087+0000 7fcfd4261640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfcc06e030 con 0x7fcfcc06dc50 2023-11-28T20:55:31.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.087+0000 7fcfd4261640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc0ccf60 con 0x7fcfcc0ce600 2023-11-28T20:55:31.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.088+0000 7fcfd27fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcfcc06aa40 0x7fcfcc0c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.088+0000 7fcfd27fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcfcc06aa40 0x7fcfcc0c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55448/0 (socket says 172.21.15.45:55448) 2023-11-28T20:55:31.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.088+0000 7fcfd27fe640 1 -- 172.21.15.45:0/530737350 learned_addr learned my addr 172.21.15.45:0/530737350 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:31.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.088+0000 7fcfd1ffd640 1 --2- 172.21.15.45:0/530737350 >> v2:172.21.15.190:3300/0 conn(0x7fcfcc06dc50 0x7fcfcc0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.088+0000 7fcfd17fc640 1 --2- 172.21.15.45:0/530737350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc0ce600 0x7fcfcc06a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.089+0000 7fcfd27fe640 1 -- 172.21.15.45:0/530737350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc0ce600 msgr2=0x7fcfcc06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:31.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.089+0000 7fcfd27fe640 1 --2- 172.21.15.45:0/530737350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc0ce600 0x7fcfcc06a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.089+0000 7fcfd27fe640 1 -- 172.21.15.45:0/530737350 >> v2:172.21.15.190:3300/0 conn(0x7fcfcc06dc50 msgr2=0x7fcfcc0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:31.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.089+0000 7fcfd27fe640 1 --2- 172.21.15.45:0/530737350 >> v2:172.21.15.190:3300/0 conn(0x7fcfcc06dc50 0x7fcfcc0cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.089+0000 7fcfd27fe640 1 -- 172.21.15.45:0/530737350 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc0cd930 con 0x7fcfcc06aa40 2023-11-28T20:55:31.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.090+0000 7fcfd27fe640 1 --2- 172.21.15.45:0/530737350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 0x7fcfcc0c93b0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc06b230 tx=0x7fcfc402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742c5311c353f89d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:31.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.090+0000 7fcfd0ffb640 1 -- 172.21.15.45:0/530737350 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc402ca70 con 0x7fcfcc06aa40 2023-11-28T20:55:31.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.090+0000 7fcfd0ffb640 1 -- 172.21.15.45:0/530737350 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc402cbd0 con 0x7fcfcc06aa40 2023-11-28T20:55:31.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.090+0000 7fcfd0ffb640 1 -- 172.21.15.45:0/530737350 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40343f0 con 0x7fcfcc06aa40 2023-11-28T20:55:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.091+0000 7fcfd4261640 1 -- 172.21.15.45:0/530737350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 msgr2=0x7fcfcc0c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.091+0000 7fcfd4261640 1 --2- 172.21.15.45:0/530737350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 0x7fcfcc0c93b0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc06b230 tx=0x7fcfc402bed0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.091+0000 7fcfd4261640 1 -- 172.21.15.45:0/530737350 shutdown_connections 2023-11-28T20:55:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.091+0000 7fcfd4261640 1 --2- 172.21.15.45:0/530737350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 0x7fcfcc0c93b0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.091+0000 7fcfd4261640 1 --2- 172.21.15.45:0/530737350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc0ce600 0x7fcfcc06a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.091+0000 7fcfd4261640 1 --2- 172.21.15.45:0/530737350 >> v2:172.21.15.190:3300/0 conn(0x7fcfcc06dc50 0x7fcfcc0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.091+0000 7fcfd4261640 1 -- 172.21.15.45:0/530737350 >> 172.21.15.45:0/530737350 conn(0x7fcfcc0ced60 msgr2=0x7fcfcc0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.091+0000 7fcfd4261640 1 -- 172.21.15.45:0/530737350 shutdown_connections 2023-11-28T20:55:31.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.092+0000 7fcfd4261640 1 -- 172.21.15.45:0/530737350 wait complete. 2023-11-28T20:55:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.094+0000 7fcfd4261640 1 Processor -- start 2023-11-28T20:55:31.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.094+0000 7fcfd4261640 1 -- start start 2023-11-28T20:55:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd4261640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 0x7fcfcc15e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd4261640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc06dc50 0x7fcfcc0b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd4261640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfcc0ce600 0x7fcfcc0b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd4261640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc1f2d60 con 0x7fcfcc06aa40 2023-11-28T20:55:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd4261640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc1f3d30 con 0x7fcfcc0ce600 2023-11-28T20:55:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd4261640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc0b4a40 con 0x7fcfcc06dc50 2023-11-28T20:55:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd1ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 0x7fcfcc15e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd1ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 0x7fcfcc15e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55462/0 (socket says 172.21.15.45:55462) 2023-11-28T20:55:31.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd1ffd640 1 -- 172.21.15.45:0/1175189434 learned_addr learned my addr 172.21.15.45:0/1175189434 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:31.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd17fc640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc06dc50 0x7fcfcc0b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd1ffd640 1 -- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc06dc50 msgr2=0x7fcfcc0b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd1ffd640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc06dc50 0x7fcfcc0b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd1ffd640 1 -- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfcc0ce600 msgr2=0x7fcfcc0b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd1ffd640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfcc0ce600 0x7fcfcc0b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.096+0000 7fcfd1ffd640 1 -- 172.21.15.45:0/1175189434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc4009b30 con 0x7fcfcc06aa40 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.097+0000 7fcfd1ffd640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 0x7fcfcc15e2f0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800e8c0 tx=0x7fcfc800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.097+0000 7fcfc27fc640 1 -- 172.21.15.45:0/1175189434 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc800c8e0 con 0x7fcfcc06aa40 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.097+0000 7fcfc27fc640 1 -- 172.21.15.45:0/1175189434 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc800ca40 con 0x7fcfcc06aa40 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.097+0000 7fcfc27fc640 1 -- 172.21.15.45:0/1175189434 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc800cdd0 con 0x7fcfcc06aa40 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.097+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc161dc0 con 0x7fcfcc06aa40 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.097+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc1f3250 con 0x7fcfcc06aa40 2023-11-28T20:55:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.098+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc0733d0 con 0x7fcfcc06aa40 2023-11-28T20:55:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.099+0000 7fcfc27fc640 1 -- 172.21.15.45:0/1175189434 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 81) v1 ==== 89086+0+0 (secure 0 0 0) 0x7fcfc8013470 con 0x7fcfcc06aa40 2023-11-28T20:55:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.100+0000 7fcfc27fc640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fcfb006ace0 0x7fcfb006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.101+0000 7fcfc27fc640 1 -- 172.21.15.45:0/1175189434 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fcfc80897e0 con 0x7fcfcc06aa40 2023-11-28T20:55:31.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.101+0000 7fcfd17fc640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fcfb006ace0 0x7fcfb006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.102+0000 7fcfd17fc640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fcfb006ace0 0x7fcfb006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0c8de0 tx=0x7fcfbc008040 comp rx=0 tx=0).ready entity=mgr.7663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:31.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.105+0000 7fcfc27fc640 1 -- 172.21.15.45:0/1175189434 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfc8054840 con 0x7fcfcc06aa40 2023-11-28T20:55:31.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.322+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 --> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfcc0705a0 con 0x7fcfb006ace0 2023-11-28T20:55:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.459+0000 7fcfc27fc640 1 -- 172.21.15.45:0/1175189434 <== mgr.7663 v2:172.21.15.190:6832/560718832 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7fcfbc010870 con 0x7fcfb006ace0 2023-11-28T20:55:31.462 INFO:teuthology.orchestra.run.smithi045.stdout:Retention added to path /snap_test_dir1 2023-11-28T20:55:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.462+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fcfb006ace0 msgr2=0x7fcfb006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.462+0000 7fcfd4261640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fcfb006ace0 0x7fcfb006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0c8de0 tx=0x7fcfbc008040 comp rx=0 tx=0).stop 2023-11-28T20:55:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.462+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 msgr2=0x7fcfcc15e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.462+0000 7fcfd4261640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 0x7fcfcc15e2f0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800e8c0 tx=0x7fcfc800ed80 comp rx=0 tx=0).stop 2023-11-28T20:55:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.463+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 shutdown_connections 2023-11-28T20:55:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.463+0000 7fcfd4261640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfcc0ce600 0x7fcfcc0b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.463+0000 7fcfd4261640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fcfb006ace0 0x7fcfb006d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.463+0000 7fcfd4261640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfcc06dc50 0x7fcfcc0b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.463+0000 7fcfd4261640 1 --2- 172.21.15.45:0/1175189434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfcc06aa40 0x7fcfcc15e2f0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.463+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 >> 172.21.15.45:0/1175189434 conn(0x7fcfcc0ced60 msgr2=0x7fcfcc0add80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.463+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 shutdown_connections 2023-11-28T20:55:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.463+0000 7fcfd4261640 1 -- 172.21.15.45:0/1175189434 wait complete. 2023-11-28T20:55:31.487 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-11-28T20:55:31.487 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-11-28T20:55:31.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2023-11-28T20:55:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.708+0000 7f7ef13ae640 1 Processor -- start 2023-11-28T20:55:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.708+0000 7f7ef13ae640 1 -- start start 2023-11-28T20:55:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.709+0000 7f7ef13ae640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7eec0cfd20 0x7f7eec0d18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.710+0000 7f7ef13ae640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec0b2930 con 0x7f7eec0cf940 2023-11-28T20:55:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.710+0000 7f7ef13ae640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec0b2650 con 0x7f7eec0d1e10 2023-11-28T20:55:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.710+0000 7f7ef13ae640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eec0ce240 con 0x7f7eec0cfd20 2023-11-28T20:55:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.710+0000 7f7eebfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7eec0cf940 0x7f7eec161800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34996/0 (socket says 172.21.15.45:34996) 2023-11-28T20:55:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.710+0000 7f7eebfff640 1 -- 172.21.15.45:0/2162846761 learned_addr learned my addr 172.21.15.45:0/2162846761 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T20:55:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.710+0000 7f7eeb7fe640 1 --2- 172.21.15.45:0/2162846761 >> v2:172.21.15.190:3300/0 conn(0x7f7eec0cfd20 0x7f7eec0d18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.710+0000 7f7eeaffd640 1 -- 172.21.15.45:0/2162846761 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256447176 0 0) 0x7f7eec0b2930 con 0x7f7eec0cf940 2023-11-28T20:55:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.710+0000 7f7eeaffd640 1 -- 172.21.15.45:0/2162846761 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed8003520 con 0x7f7eec0cf940 2023-11-28T20:55:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.711+0000 7f7eeaffd640 1 -- 172.21.15.45:0/2162846761 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558901843 0 0) 0x7f7ed8003520 con 0x7f7eec0cf940 2023-11-28T20:55:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.711+0000 7f7eeb7fe640 1 -- 172.21.15.45:0/2162846761 >> v1:172.21.15.190:6789/0 conn(0x7f7eec0d1e10 legacy=0x7f7eec0d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T20:55:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.711+0000 7f7eeb7fe640 1 -- 172.21.15.45:0/2162846761 >> v1:172.21.15.45:6789/0 conn(0x7f7eec0cf940 legacy=0x7f7eec161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.711+0000 7f7eeb7fe640 1 -- 172.21.15.45:0/2162846761 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec0ce520 con 0x7f7eec0cfd20 2023-11-28T20:55:31.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.712+0000 7f7eeb7fe640 1 --2- 172.21.15.45:0/2162846761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7eec0cfd20 0x7f7eec0d18d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40097a0 tx=0x7f7ed402b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de5940fe721279a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:31.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.713+0000 7f7eeaffd640 1 -- 172.21.15.45:0/2162846761 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc0019f0 con 0x7f7eec0cfd20 2023-11-28T20:55:31.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.713+0000 7f7eeaffd640 1 -- 172.21.15.45:0/2162846761 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed402e050 con 0x7f7eec0cfd20 2023-11-28T20:55:31.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.713+0000 7f7eeaffd640 1 -- 172.21.15.45:0/2162846761 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4030710 con 0x7f7eec0cfd20 2023-11-28T20:55:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.714+0000 7f7ef13ae640 1 -- 172.21.15.45:0/2162846761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7eec0cfd20 msgr2=0x7f7eec0d18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.714+0000 7f7ef13ae640 1 --2- 172.21.15.45:0/2162846761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7eec0cfd20 0x7f7eec0d18d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40097a0 tx=0x7f7ed402b680 comp rx=0 tx=0).stop 2023-11-28T20:55:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.714+0000 7f7ef13ae640 1 -- 172.21.15.45:0/2162846761 shutdown_connections 2023-11-28T20:55:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.714+0000 7f7ef13ae640 1 --2- 172.21.15.45:0/2162846761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7eec0cfd20 0x7f7eec0d18d0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.714+0000 7f7ef13ae640 1 -- 172.21.15.45:0/2162846761 >> 172.21.15.45:0/2162846761 conn(0x7f7eec06a500 msgr2=0x7f7eec06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.714+0000 7f7ef13ae640 1 -- 172.21.15.45:0/2162846761 shutdown_connections 2023-11-28T20:55:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.714+0000 7f7ef13ae640 1 -- 172.21.15.45:0/2162846761 wait complete. 2023-11-28T20:55:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.716+0000 7f7ef13ae640 1 Processor -- start 2023-11-28T20:55:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.716+0000 7f7ef13ae640 1 -- start start 2023-11-28T20:55:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.717+0000 7f7ef13ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eec0cf940 0x7f7eec1f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.717+0000 7f7ef13ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7eec0cfd20 0x7f7eec0aa220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.717+0000 7f7ef13ae640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7eec0d1e10 0x7f7eec0aa760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.717+0000 7f7ef13ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec15be80 con 0x7f7eec0cf940 2023-11-28T20:55:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.717+0000 7f7ef13ae640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec1f43c0 con 0x7f7eec0d1e10 2023-11-28T20:55:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.717+0000 7f7ef13ae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec0ccbf0 con 0x7f7eec0cfd20 2023-11-28T20:55:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.717+0000 7f7eebfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eec0cf940 0x7f7eec1f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.717+0000 7f7eebfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eec0cf940 0x7f7eec1f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55470/0 (socket says 172.21.15.45:55470) 2023-11-28T20:55:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.717+0000 7f7eebfff640 1 -- 172.21.15.45:0/317474976 learned_addr learned my addr 172.21.15.45:0/317474976 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T20:55:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7eeb7fe640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7eec0cfd20 0x7f7eec0aa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7eebfff640 1 -- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7eec0cfd20 msgr2=0x7f7eec0aa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7ef0bad640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7eec0d1e10 0x7f7eec0aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7eebfff640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7eec0cfd20 0x7f7eec0aa220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7eebfff640 1 -- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7eec0d1e10 msgr2=0x7f7eec0aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7eebfff640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7eec0d1e10 0x7f7eec0aa760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7eebfff640 1 -- 172.21.15.45:0/317474976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed4009400 con 0x7f7eec0cf940 2023-11-28T20:55:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7eebfff640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eec0cf940 0x7f7eec1f3e80 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00cc70 tx=0x7f7edc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/317474976 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00ce80 con 0x7f7eec0cf940 2023-11-28T20:55:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/317474976 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc012400 con 0x7f7eec0cf940 2023-11-28T20:55:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/317474976 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc012790 con 0x7f7eec0cf940 2023-11-28T20:55:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.718+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eec0cc070 con 0x7f7eec0cf940 2023-11-28T20:55:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.719+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eec0cc630 con 0x7f7eec0cf940 2023-11-28T20:55:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.720+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eec071b30 con 0x7f7eec0cf940 2023-11-28T20:55:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.721+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/317474976 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 82) v1 ==== 89351+0+0 (secure 0 0 0) 0x7f7edc007070 con 0x7f7eec0cf940 2023-11-28T20:55:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.721+0000 7f7ee8ff9640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f7ecc06afb0 0x7f7ecc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T20:55:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.722+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/317474976 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7edc089bb0 con 0x7f7eec0cf940 2023-11-28T20:55:31.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.722+0000 7f7eeb7fe640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f7ecc06afb0 0x7f7ecc06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T20:55:31.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.724+0000 7f7eeb7fe640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f7ecc06afb0 0x7f7ecc06d470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4034000 tx=0x7f7ed4008030 comp rx=0 tx=0).ready entity=mgr.7663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T20:55:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.726+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/317474976 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7edc054920 con 0x7f7eec0cf940 2023-11-28T20:55:31.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.946+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 --> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7eec0d0180 con 0x7f7ecc06afb0 2023-11-28T20:55:31.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.963+0000 7f7ee8ff9640 1 -- 172.21.15.45:0/317474976 <== mgr.7663 v2:172.21.15.190:6832/560718832 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f7ed4004440 con 0x7f7ecc06afb0 2023-11-28T20:55:31.966 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T20:55:31.966 INFO:teuthology.orchestra.run.smithi045.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-11-28T20:55:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.965+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f7ecc06afb0 msgr2=0x7f7ecc06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.965+0000 7f7ef13ae640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f7ecc06afb0 0x7f7ecc06d470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4034000 tx=0x7f7ed4008030 comp rx=0 tx=0).stop 2023-11-28T20:55:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.965+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eec0cf940 msgr2=0x7f7eec1f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T20:55:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.965+0000 7f7ef13ae640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eec0cf940 0x7f7eec1f3e80 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00cc70 tx=0x7f7edc00a430 comp rx=0 tx=0).stop 2023-11-28T20:55:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.966+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 shutdown_connections 2023-11-28T20:55:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.966+0000 7f7ef13ae640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7eec0d1e10 0x7f7eec0aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.966+0000 7f7ef13ae640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f7ecc06afb0 0x7f7ecc06d470 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.966+0000 7f7ef13ae640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7eec0cfd20 0x7f7eec0aa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.966+0000 7f7ef13ae640 1 --2- 172.21.15.45:0/317474976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eec0cf940 0x7f7eec1f3e80 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T20:55:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.966+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 >> 172.21.15.45:0/317474976 conn(0x7f7eec06a500 msgr2=0x7f7eec0b8240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T20:55:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.966+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 shutdown_connections 2023-11-28T20:55:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T20:55:31.966+0000 7f7ef13ae640 1 -- 172.21.15.45:0/317474976 wait complete. 2023-11-28T20:55:31.986 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-11-28T20:55:31.986 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2023-11-28T20:55:51.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:55:51.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:56:00.485 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:56:00.483+0000 7f5b7c23b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:56:00.485 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:56:00.483+0000 7f5b7c23b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:56:00.485 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:56:00.483+0000 7f5b7c23b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:56:00.485 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:56:00.483+0000 7f5b7c23b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:56:00.485 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T20:56:00.483+0000 7f5b7c23b640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T20:56:21.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:56:21.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:56:51.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:56:51.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:57:21.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:57:21.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:57:51.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:57:51.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:58:21.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:58:21.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:58:52.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:58:52.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:59:22.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:59:22.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:59:52.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T20:59:52.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:00:22.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:00:22.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:00:52.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:00:52.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:01:22.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:01:22.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:01:52.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:01:52.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:02:23.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:02:23.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:02:53.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:02:53.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:03:23.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:03:23.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:03:53.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:03:53.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:04:23.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:04:23.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:04:54.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:04:54.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:05:24.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:05:24.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:05:54.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:05:57.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:06:27.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:06:27.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:06:57.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:06:57.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:07:27.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:07:27.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:07:58.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:08:01.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:08:31.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:08:31.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:09:02.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:09:02.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:09:32.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:09:32.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:10:02.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:10:09.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:10:39.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:10:39.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:11:09.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:11:09.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:11:40.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:11:40.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:12:10.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:12:10.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:12:40.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:12:40.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:13:10.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:13:10.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:13:40.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:13:40.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:14:10.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:14:10.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:14:41.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:14:41.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:15:11.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:15:11.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:15:41.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:15:41.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:16:11.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:16:11.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:16:41.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:16:41.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:17:11.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:17:11.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:17:42.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:17:42.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:18:12.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:18:12.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:18:42.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:18:42.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:19:12.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:19:12.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:19:42.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:19:42.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:20:12.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:20:12.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:20:42.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:20:43.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:21:13.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:21:15.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:21:46.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:21:46.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:22:16.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:22:16.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:22:46.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:22:46.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:23:16.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:23:16.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:23:46.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:23:46.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:24:16.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:24:16.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:24:47.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:24:47.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:25:17.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:25:17.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:25:47.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:25:47.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:26:17.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:26:17.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:26:47.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:26:47.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:27:17.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:27:17.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:27:47.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:27:47.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:28:18.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:28:18.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:28:48.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:28:48.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:29:18.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:29:18.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:29:48.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:29:48.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:30:18.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:30:18.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:30:48.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:30:48.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:31:19.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:31:22.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:31:52.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:31:52.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:32:22.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:32:22.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:32:52.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:32:52.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:33:23.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:33:23.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:33:53.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:33:53.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:34:23.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:34:23.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:34:53.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:34:53.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:35:23.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:35:25.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:35:55.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:35:55.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:36:25.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:36:25.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:36:55.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:36:55.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:37:25.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:37:25.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:37:56.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:37:56.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:38:26.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:38:26.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:38:56.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:38:56.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:39:26.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:39:26.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:39:56.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:39:56.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:40:26.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:40:26.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:40:57.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:40:57.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:41:27.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:41:27.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:41:57.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:41:57.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:42:27.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:42:27.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:42:57.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:42:57.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:43:27.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:43:27.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:43:57.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:43:57.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:44:28.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:44:28.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:44:58.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:44:58.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:45:28.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:45:28.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:45:58.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:45:58.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:46:28.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:46:28.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:46:58.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:46:58.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:47:28.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:47:28.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:47:59.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:47:59.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:48:29.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:48:29.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:48:59.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:48:59.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:49:29.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:49:29.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:49:59.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:49:59.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:50:29.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:50:31.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:51:01.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:51:01.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:51:15.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:15.988 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T21:51:16.058 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_57_00_UTC 2023-11-28T21:51:16.058 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_58_00_UTC 2023-11-28T21:51:16.058 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_59_00_UTC 2023-11-28T21:51:16.058 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_00_00_UTC 2023-11-28T21:51:16.058 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_01_00_UTC 2023-11-28T21:51:16.058 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_02_00_UTC 2023-11-28T21:51:16.058 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_03_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_04_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_05_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_06_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_07_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_08_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_09_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_10_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_11_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_12_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_13_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_14_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_15_00_UTC 2023-11-28T21:51:16.059 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_16_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_17_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_18_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_19_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_20_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_21_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_22_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_23_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_24_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_25_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_26_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_27_00_UTC 2023-11-28T21:51:16.060 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_28_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_29_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_30_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_31_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_32_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_33_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_34_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_35_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_36_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_37_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_38_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_39_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_40_00_UTC 2023-11-28T21:51:16.061 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_41_00_UTC 2023-11-28T21:51:16.062 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_42_00_UTC 2023-11-28T21:51:16.062 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_43_00_UTC 2023-11-28T21:51:16.062 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_44_00_UTC 2023-11-28T21:51:16.062 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_45_00_UTC 2023-11-28T21:51:16.062 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_46_00_UTC 2023-11-28T21:51:16.062 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_47_00_UTC 2023-11-28T21:51:16.062 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_48_00_UTC 2023-11-28T21:51:16.062 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_49_00_UTC 2023-11-28T21:51:16.065 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_50_00_UTC 2023-11-28T21:51:16.065 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_51_00_UTC 2023-11-28T21:51:16.066 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-20_57_00_UTC', 'scheduled-2023-11-28-20_58_00_UTC', 'scheduled-2023-11-28-20_59_00_UTC', 'scheduled-2023-11-28-21_00_00_UTC', 'scheduled-2023-11-28-21_01_00_UTC', 'scheduled-2023-11-28-21_02_00_UTC', 'scheduled-2023-11-28-21_03_00_UTC', 'scheduled-2023-11-28-21_04_00_UTC', 'scheduled-2023-11-28-21_05_00_UTC', 'scheduled-2023-11-28-21_06_00_UTC', 'scheduled-2023-11-28-21_07_00_UTC', 'scheduled-2023-11-28-21_08_00_UTC', 'scheduled-2023-11-28-21_09_00_UTC', 'scheduled-2023-11-28-21_10_00_UTC', 'scheduled-2023-11-28-21_11_00_UTC', 'scheduled-2023-11-28-21_12_00_UTC', 'scheduled-2023-11-28-21_13_00_UTC', 'scheduled-2023-11-28-21_14_00_UTC', 'scheduled-2023-11-28-21_15_00_UTC', 'scheduled-2023-11-28-21_16_00_UTC', 'scheduled-2023-11-28-21_17_00_UTC', 'scheduled-2023-11-28-21_18_00_UTC', 'scheduled-2023-11-28-21_19_00_UTC', 'scheduled-2023-11-28-21_20_00_UTC', 'scheduled-2023-11-28-21_21_00_UTC', 'scheduled-2023-11-28-21_22_00_UTC', 'scheduled-2023-11-28-21_23_00_UTC', 'scheduled-2023-11-28-21_24_00_UTC', 'scheduled-2023-11-28-21_25_00_UTC', 'scheduled-2023-11-28-21_26_00_UTC', 'scheduled-2023-11-28-21_27_00_UTC', 'scheduled-2023-11-28-21_28_00_UTC', 'scheduled-2023-11-28-21_29_00_UTC', 'scheduled-2023-11-28-21_30_00_UTC', 'scheduled-2023-11-28-21_31_00_UTC', 'scheduled-2023-11-28-21_32_00_UTC', 'scheduled-2023-11-28-21_33_00_UTC', 'scheduled-2023-11-28-21_34_00_UTC', 'scheduled-2023-11-28-21_35_00_UTC', 'scheduled-2023-11-28-21_36_00_UTC', 'scheduled-2023-11-28-21_37_00_UTC', 'scheduled-2023-11-28-21_38_00_UTC', 'scheduled-2023-11-28-21_39_00_UTC', 'scheduled-2023-11-28-21_40_00_UTC', 'scheduled-2023-11-28-21_41_00_UTC', 'scheduled-2023-11-28-21_42_00_UTC', 'scheduled-2023-11-28-21_43_00_UTC', 'scheduled-2023-11-28-21_44_00_UTC', 'scheduled-2023-11-28-21_45_00_UTC', 'scheduled-2023-11-28-21_46_00_UTC', 'scheduled-2023-11-28-21_47_00_UTC', 'scheduled-2023-11-28-21_48_00_UTC', 'scheduled-2023-11-28-21_49_00_UTC', 'scheduled-2023-11-28-21_50_00_UTC', 'scheduled-2023-11-28-21_51_00_UTC'] 2023-11-28T21:51:16.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2023-11-28T21:51:16.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.298+0000 7f042eeaa640 1 Processor -- start 2023-11-28T21:51:16.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.298+0000 7f042eeaa640 1 -- start start 2023-11-28T21:51:16.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.300+0000 7f042eeaa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f042815b080 0x7f04280ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.300+0000 7f042eeaa640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0428071fd0 0x7f04280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.300+0000 7f042eeaa640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04280ac270 con 0x7f04280abd70 2023-11-28T21:51:16.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.300+0000 7f042eeaa640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042815b830 con 0x7f042815b080 2023-11-28T21:51:16.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.300+0000 7f042eeaa640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042815bb10 con 0x7f0428071fd0 2023-11-28T21:51:16.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.301+0000 7f042dea8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f042815b080 0x7f04280ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.301+0000 7f042dea8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f042815b080 0x7f04280ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54914/0 (socket says 172.21.15.45:54914) 2023-11-28T21:51:16.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.301+0000 7f042dea8640 1 -- 172.21.15.45:0/3914399246 learned_addr learned my addr 172.21.15.45:0/3914399246 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:16.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.301+0000 7f042dea8640 1 -- 172.21.15.45:0/3914399246 >> v2:172.21.15.190:3300/0 conn(0x7f0428071fd0 msgr2=0x7f04280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:16.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.301+0000 7f042dea8640 1 --2- 172.21.15.45:0/3914399246 >> v2:172.21.15.190:3300/0 conn(0x7f0428071fd0 0x7f04280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.301+0000 7f042dea8640 1 -- 172.21.15.45:0/3914399246 >> v1:172.21.15.190:6789/0 conn(0x7f04280abd70 legacy=0x7f04280718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:16.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.301+0000 7f042dea8640 1 -- 172.21.15.45:0/3914399246 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042815bdf0 con 0x7f042815b080 2023-11-28T21:51:16.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.303+0000 7f042dea8640 1 --2- 172.21.15.45:0/3914399246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f042815b080 0x7f04280ae370 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f0428070250 tx=0x7f042402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247fc69def92ec69 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:16.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.303+0000 7f042cea6640 1 -- 172.21.15.45:0/3914399246 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424031870 con 0x7f042815b080 2023-11-28T21:51:16.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.303+0000 7f042cea6640 1 -- 172.21.15.45:0/3914399246 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04240319d0 con 0x7f042815b080 2023-11-28T21:51:16.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.303+0000 7f042cea6640 1 -- 172.21.15.45:0/3914399246 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424031d60 con 0x7f042815b080 2023-11-28T21:51:16.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.304+0000 7f042eeaa640 1 -- 172.21.15.45:0/3914399246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f042815b080 msgr2=0x7f04280ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.304+0000 7f042eeaa640 1 --2- 172.21.15.45:0/3914399246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f042815b080 0x7f04280ae370 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f0428070250 tx=0x7f042402b910 comp rx=0 tx=0).stop 2023-11-28T21:51:16.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.304+0000 7f042eeaa640 1 -- 172.21.15.45:0/3914399246 shutdown_connections 2023-11-28T21:51:16.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.304+0000 7f042eeaa640 1 --2- 172.21.15.45:0/3914399246 >> v2:172.21.15.190:3300/0 conn(0x7f0428071fd0 0x7f04280702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.304+0000 7f042eeaa640 1 --2- 172.21.15.45:0/3914399246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f042815b080 0x7f04280ae370 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.304+0000 7f042eeaa640 1 -- 172.21.15.45:0/3914399246 >> 172.21.15.45:0/3914399246 conn(0x7f04280cecd0 msgr2=0x7f04280cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:16.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.304+0000 7f042eeaa640 1 -- 172.21.15.45:0/3914399246 shutdown_connections 2023-11-28T21:51:16.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.304+0000 7f042eeaa640 1 -- 172.21.15.45:0/3914399246 wait complete. 2023-11-28T21:51:16.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.306+0000 7f042eeaa640 1 Processor -- start 2023-11-28T21:51:16.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.306+0000 7f042eeaa640 1 -- start start 2023-11-28T21:51:16.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.307+0000 7f042eeaa640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0428071fd0 0x7f0428070510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.307+0000 7f042eeaa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04280abd70 0x7f0428159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.307+0000 7f042eeaa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042815b080 0x7f0428143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.307+0000 7f042eeaa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042809aff0 con 0x7f04280abd70 2023-11-28T21:51:16.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.307+0000 7f042eeaa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042809b570 con 0x7f042815b080 2023-11-28T21:51:16.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.307+0000 7f042eeaa640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0428153e40 con 0x7f0428071fd0 2023-11-28T21:51:16.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042d6a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04280abd70 0x7f0428159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042d6a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04280abd70 0x7f0428159ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54920/0 (socket says 172.21.15.45:54920) 2023-11-28T21:51:16.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042d6a7640 1 -- 172.21.15.45:0/2972417964 learned_addr learned my addr 172.21.15.45:0/2972417964 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:16.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042dea8640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0428071fd0 0x7f0428070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042d6a7640 1 -- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0428071fd0 msgr2=0x7f0428070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042d6a7640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0428071fd0 0x7f0428070510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042d6a7640 1 -- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042815b080 msgr2=0x7f0428143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042d6a7640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042815b080 0x7f0428143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042d6a7640 1 -- 172.21.15.45:0/2972417964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04240094d0 con 0x7f04280abd70 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.308+0000 7f042d6a7640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04280abd70 0x7f0428159ae0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f041800c430 tx=0x7f041800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.309+0000 7f041e7fc640 1 -- 172.21.15.45:0/2972417964 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0418011790 con 0x7f04280abd70 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.309+0000 7f041e7fc640 1 -- 172.21.15.45:0/2972417964 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04180118f0 con 0x7f04280abd70 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.309+0000 7f041e7fc640 1 -- 172.21.15.45:0/2972417964 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0418011ce0 con 0x7f04280abd70 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.309+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0428157de0 con 0x7f04280abd70 2023-11-28T21:51:16.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.309+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0428158640 con 0x7f04280abd70 2023-11-28T21:51:16.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.310+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042809b6b0 con 0x7f04280abd70 2023-11-28T21:51:16.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.311+0000 7f041e7fc640 1 -- 172.21.15.45:0/2972417964 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 85) v1 ==== 89362+0+0 (secure 0 0 0) 0x7f041801c460 con 0x7f04280abd70 2023-11-28T21:51:16.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.312+0000 7f041e7fc640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f040c06b000 0x7f040c06d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.312+0000 7f041e7fc640 1 -- 172.21.15.45:0/2972417964 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f0418089890 con 0x7f04280abd70 2023-11-28T21:51:16.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.313+0000 7f042dea8640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f040c06b000 0x7f040c06d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.314+0000 7f042dea8640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f040c06b000 0x7f040c06d4c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0424036000 tx=0x7f042403c040 comp rx=0 tx=0).ready entity=mgr.7663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:16.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.317+0000 7f041e7fc640 1 -- 172.21.15.45:0/2972417964 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f042809b6b0 con 0x7f04280abd70 2023-11-28T21:51:16.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.534+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 --> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f042815c310 con 0x7f040c06b000 2023-11-28T21:51:16.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.554+0000 7f041e7fc640 1 -- 172.21.15.45:0/2972417964 <== mgr.7663 v2:172.21.15.190:6832/560718832 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f0424031cc0 con 0x7f040c06b000 2023-11-28T21:51:16.556 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:16.556 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:55:30", "first": "2023-11-28T20:56:00", "last": "2023-11-28T21:51:00", "last_pruned": "2023-11-28T21:51:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-11-28T21:51:16.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.556+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f040c06b000 msgr2=0x7f040c06d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.556+0000 7f042eeaa640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f040c06b000 0x7f040c06d4c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0424036000 tx=0x7f042403c040 comp rx=0 tx=0).stop 2023-11-28T21:51:16.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.556+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04280abd70 msgr2=0x7f0428159ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.556+0000 7f042eeaa640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04280abd70 0x7f0428159ae0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f041800c430 tx=0x7f041800c8f0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.556+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 shutdown_connections 2023-11-28T21:51:16.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.556+0000 7f042eeaa640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042815b080 0x7f0428143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.557+0000 7f042eeaa640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04280abd70 0x7f0428159ae0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.557+0000 7f042eeaa640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f040c06b000 0x7f040c06d4c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.557+0000 7f042eeaa640 1 --2- 172.21.15.45:0/2972417964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0428071fd0 0x7f0428070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.557+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 >> 172.21.15.45:0/2972417964 conn(0x7f04280cecd0 msgr2=0x7f04280af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:16.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.557+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 shutdown_connections 2023-11-28T21:51:16.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.557+0000 7f042eeaa640 1 -- 172.21.15.45:0/2972417964 wait complete. 2023-11-28T21:51:16.579 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T20:55:30", "first": "2023-11-28T20:56:00", "last": "2023-11-28T21:51:00", "last_pruned": "2023-11-28T21:51:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-11-28T21:51:16.580 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T20:55:30', 'first': '2023-11-28T20:56:00', 'last': '2023-11-28T21:51:00', 'last_pruned': '2023-11-28T21:51:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-11-28T21:51:16.580 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-11-28T21:51:16.580 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-11-28T21:51:16.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2023-11-28T21:51:16.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.804+0000 7f5983ee7640 1 Processor -- start 2023-11-28T21:51:16.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.804+0000 7f5983ee7640 1 -- start start 2023-11-28T21:51:16.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.806+0000 7f5983ee7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f597c06ac10 0x7f597c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.806+0000 7f5983ee7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f597c15ecd0 con 0x7f597c15f720 2023-11-28T21:51:16.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.806+0000 7f5983ee7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f597c15f290 con 0x7f597c15e840 2023-11-28T21:51:16.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.806+0000 7f5983ee7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597c15fbd0 con 0x7f597c06ac10 2023-11-28T21:51:16.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.806+0000 7f5982484640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f597c15f720 0x7f597c0cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38714/0 (socket says 172.21.15.45:38714) 2023-11-28T21:51:16.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.806+0000 7f5982484640 1 -- 172.21.15.45:0/1987338093 learned_addr learned my addr 172.21.15.45:0/1987338093 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5981482640 1 --2- 172.21.15.45:0/1987338093 >> v2:172.21.15.190:3300/0 conn(0x7f597c06ac10 0x7f597c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420121327 0 0) 0x7f597c15ecd0 con 0x7f597c15f720 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968003520 con 0x7f597c15f720 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 781217726 0 0) 0x7f5968003520 con 0x7f597c15f720 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 >> v2:172.21.15.190:3300/0 conn(0x7f597c06ac10 msgr2=0x7f597c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 --2- 172.21.15.45:0/1987338093 >> v2:172.21.15.190:3300/0 conn(0x7f597c06ac10 0x7f597c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 >> v1:172.21.15.190:6789/0 conn(0x7f597c15e840 legacy=0x7f597c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c072020 con 0x7f597c15f720 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f59740019f0 con 0x7f597c15f720 2023-11-28T21:51:16.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 >> v1:172.21.15.45:6789/0 conn(0x7f597c15f720 legacy=0x7f597c0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5968003970 0x7f5968003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59680047c0 0x7f5968004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5968005180 0x7f596800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c15fbd0 con 0x7f59680047c0 2023-11-28T21:51:16.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596800c020 con 0x7f5968003970 2023-11-28T21:51:16.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.807+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596800c1e0 con 0x7f5968005180 2023-11-28T21:51:16.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5981482640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59680047c0 0x7f5968004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5981c83640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5968003970 0x7f5968003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5982484640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5968005180 0x7f596800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5981482640 1 -- 172.21.15.45:0/1987338093 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5968005180 msgr2=0x7f596800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5981482640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5968005180 0x7f596800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5981482640 1 -- 172.21.15.45:0/1987338093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5968003970 msgr2=0x7f5968003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5981482640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5968003970 0x7f5968003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5981482640 1 -- 172.21.15.45:0/1987338093 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968003520 con 0x7f59680047c0 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5982484640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5968005180 0x7f596800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.808+0000 7f5982484640 1 -- 172.21.15.45:0/1987338093 reap_dead start 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.809+0000 7f5981482640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59680047c0 0x7f5968004c40 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f596c009ac0 tx=0x7f59680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41b0b47869a32d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c02c6b0 con 0x7f59680047c0 2023-11-28T21:51:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c02c810 con 0x7f59680047c0 2023-11-28T21:51:16.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5980c81640 1 -- 172.21.15.45:0/1987338093 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c02cba0 con 0x7f59680047c0 2023-11-28T21:51:16.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5983ee7640 1 -- 172.21.15.45:0/1987338093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59680047c0 msgr2=0x7f5968004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5983ee7640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59680047c0 0x7f5968004c40 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f596c009ac0 tx=0x7f59680057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5983ee7640 1 -- 172.21.15.45:0/1987338093 shutdown_connections 2023-11-28T21:51:16.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5983ee7640 1 --2- 172.21.15.45:0/1987338093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59680047c0 0x7f5968004c40 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5983ee7640 1 -- 172.21.15.45:0/1987338093 >> 172.21.15.45:0/1987338093 conn(0x7f597c0ced60 msgr2=0x7f597c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:16.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5983ee7640 1 -- 172.21.15.45:0/1987338093 shutdown_connections 2023-11-28T21:51:16.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.810+0000 7f5983ee7640 1 -- 172.21.15.45:0/1987338093 wait complete. 2023-11-28T21:51:16.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.812+0000 7f5983ee7640 1 Processor -- start 2023-11-28T21:51:16.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.812+0000 7f5983ee7640 1 -- start start 2023-11-28T21:51:16.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.813+0000 7f5983ee7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59680047c0 0x7f597c15b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.813+0000 7f5983ee7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f597c1470a0 0x7f597c0b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.813+0000 7f5983ee7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f597c0b1bb0 0x7f597c1f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.813+0000 7f5983ee7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c09abc0 con 0x7f597c0b1bb0 2023-11-28T21:51:16.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.813+0000 7f5983ee7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c15b950 con 0x7f597c1470a0 2023-11-28T21:51:16.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.813+0000 7f5983ee7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597c09ae10 con 0x7f59680047c0 2023-11-28T21:51:16.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5982484640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f597c0b1bb0 0x7f597c1f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5982484640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f597c0b1bb0 0x7f597c1f5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54938/0 (socket says 172.21.15.45:54938) 2023-11-28T21:51:16.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5982484640 1 -- 172.21.15.45:0/3351185316 learned_addr learned my addr 172.21.15.45:0/3351185316 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:16.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5981482640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f597c1470a0 0x7f597c0b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5981c83640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59680047c0 0x7f597c15b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5982484640 1 -- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59680047c0 msgr2=0x7f597c15b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5982484640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59680047c0 0x7f597c15b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5982484640 1 -- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f597c1470a0 msgr2=0x7f597c0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5982484640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f597c1470a0 0x7f597c0b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.814+0000 7f5982484640 1 -- 172.21.15.45:0/3351185316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c09b0f0 con 0x7f597c0b1bb0 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.815+0000 7f5982484640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f597c0b1bb0 0x7f597c1f5ea0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f597c15fb00 tx=0x7f597400b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.815+0000 7f59727fc640 1 -- 172.21.15.45:0/3351185316 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597400b8d0 con 0x7f597c0b1bb0 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.815+0000 7f59727fc640 1 -- 172.21.15.45:0/3351185316 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597400ba30 con 0x7f597c0b1bb0 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.815+0000 7f59727fc640 1 -- 172.21.15.45:0/3351185316 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597400bde0 con 0x7f597c0b1bb0 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.815+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597c09b3d0 con 0x7f597c0b1bb0 2023-11-28T21:51:16.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.815+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597c147500 con 0x7f597c0b1bb0 2023-11-28T21:51:16.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.816+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597c09b5c0 con 0x7f597c0b1bb0 2023-11-28T21:51:16.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.817+0000 7f59727fc640 1 -- 172.21.15.45:0/3351185316 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 85) v1 ==== 89362+0+0 (secure 0 0 0) 0x7f5974007070 con 0x7f597c0b1bb0 2023-11-28T21:51:16.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.818+0000 7f59727fc640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f596006afb0 0x7f596006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:16.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.818+0000 7f59727fc640 1 -- 172.21.15.45:0/3351185316 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f597408a040 con 0x7f597c0b1bb0 2023-11-28T21:51:16.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.819+0000 7f5981c83640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f596006afb0 0x7f596006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:16.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.820+0000 7f5981c83640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f596006afb0 0x7f596006d470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5978007fd0 tx=0x7f5978009040 comp rx=0 tx=0).ready entity=mgr.7663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:16.823+0000 7f59727fc640 1 -- 172.21.15.45:0/3351185316 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5974054ac0 con 0x7f597c0b1bb0 2023-11-28T21:51:17.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.045+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 --> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f597c15ef20 con 0x7f596006afb0 2023-11-28T21:51:17.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.212+0000 7f59727fc640 1 -- 172.21.15.45:0/3351185316 <== mgr.7663 v2:172.21.15.190:6832/560718832 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f5978007040 con 0x7f596006afb0 2023-11-28T21:51:17.215 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1 2023-11-28T21:51:17.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f596006afb0 msgr2=0x7f596006d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f596006afb0 0x7f596006d470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5978007fd0 tx=0x7f5978009040 comp rx=0 tx=0).stop 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f597c0b1bb0 msgr2=0x7f597c1f5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f597c0b1bb0 0x7f597c1f5ea0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f597c15fb00 tx=0x7f597400b0d0 comp rx=0 tx=0).stop 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 shutdown_connections 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f597c0b1bb0 0x7f597c1f5ea0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f597c1470a0 0x7f597c0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f596006afb0 0x7f596006d470 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 --2- 172.21.15.45:0/3351185316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59680047c0 0x7f597c15b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 >> 172.21.15.45:0/3351185316 conn(0x7f597c0ced60 msgr2=0x7f597c06dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 shutdown_connections 2023-11-28T21:51:17.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:17.216+0000 7f5983ee7640 1 -- 172.21.15.45:0/3351185316 wait complete. 2023-11-28T21:51:17.239 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-11-28T21:51:17.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.239 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T21:51:17.302 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_57_00_UTC 2023-11-28T21:51:17.303 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_58_00_UTC 2023-11-28T21:51:17.303 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-20_59_00_UTC 2023-11-28T21:51:17.303 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_00_00_UTC 2023-11-28T21:51:17.303 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_01_00_UTC 2023-11-28T21:51:17.303 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_02_00_UTC 2023-11-28T21:51:17.303 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_03_00_UTC 2023-11-28T21:51:17.303 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_04_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_05_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_06_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_07_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_08_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_09_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_10_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_11_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_12_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_13_00_UTC 2023-11-28T21:51:17.304 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_14_00_UTC 2023-11-28T21:51:17.305 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_15_00_UTC 2023-11-28T21:51:17.305 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_16_00_UTC 2023-11-28T21:51:17.305 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_17_00_UTC 2023-11-28T21:51:17.305 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_18_00_UTC 2023-11-28T21:51:17.305 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_19_00_UTC 2023-11-28T21:51:17.305 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_20_00_UTC 2023-11-28T21:51:17.305 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_21_00_UTC 2023-11-28T21:51:17.305 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_22_00_UTC 2023-11-28T21:51:17.305 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_23_00_UTC 2023-11-28T21:51:17.306 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_24_00_UTC 2023-11-28T21:51:17.306 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_25_00_UTC 2023-11-28T21:51:17.306 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_26_00_UTC 2023-11-28T21:51:17.306 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_27_00_UTC 2023-11-28T21:51:17.306 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_28_00_UTC 2023-11-28T21:51:17.306 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_29_00_UTC 2023-11-28T21:51:17.306 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_30_00_UTC 2023-11-28T21:51:17.306 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_31_00_UTC 2023-11-28T21:51:17.306 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_32_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_33_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_34_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_35_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_36_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_37_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_38_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_39_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_40_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_41_00_UTC 2023-11-28T21:51:17.307 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_42_00_UTC 2023-11-28T21:51:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_43_00_UTC 2023-11-28T21:51:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_44_00_UTC 2023-11-28T21:51:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_45_00_UTC 2023-11-28T21:51:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_46_00_UTC 2023-11-28T21:51:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_47_00_UTC 2023-11-28T21:51:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_48_00_UTC 2023-11-28T21:51:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_49_00_UTC 2023-11-28T21:51:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_50_00_UTC 2023-11-28T21:51:17.308 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_51_00_UTC 2023-11-28T21:51:17.310 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-20_57_00_UTC 2023-11-28T21:51:17.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.310 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-20_57_00_UTC) 2023-11-28T21:51:17.388 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-20_58_00_UTC 2023-11-28T21:51:17.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.389 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-20_58_00_UTC) 2023-11-28T21:51:17.463 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-20_59_00_UTC 2023-11-28T21:51:17.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.464 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-20_59_00_UTC) 2023-11-28T21:51:17.538 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_00_00_UTC 2023-11-28T21:51:17.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.539 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_00_00_UTC) 2023-11-28T21:51:17.614 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_01_00_UTC 2023-11-28T21:51:17.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.614 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_01_00_UTC) 2023-11-28T21:51:17.683 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_02_00_UTC 2023-11-28T21:51:17.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.684 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_02_00_UTC) 2023-11-28T21:51:17.762 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_03_00_UTC 2023-11-28T21:51:17.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.762 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_03_00_UTC) 2023-11-28T21:51:17.837 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_04_00_UTC 2023-11-28T21:51:17.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.837 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_04_00_UTC) 2023-11-28T21:51:17.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_05_00_UTC 2023-11-28T21:51:17.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.909 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_05_00_UTC) 2023-11-28T21:51:17.981 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_06_00_UTC 2023-11-28T21:51:17.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:17.982 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_06_00_UTC) 2023-11-28T21:51:18.087 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_07_00_UTC 2023-11-28T21:51:18.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.087 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_07_00_UTC) 2023-11-28T21:51:18.157 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_08_00_UTC 2023-11-28T21:51:18.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.158 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_08_00_UTC) 2023-11-28T21:51:18.223 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_09_00_UTC 2023-11-28T21:51:18.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.223 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_09_00_UTC) 2023-11-28T21:51:18.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_10_00_UTC 2023-11-28T21:51:18.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.296 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_10_00_UTC) 2023-11-28T21:51:18.373 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_11_00_UTC 2023-11-28T21:51:18.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.373 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_11_00_UTC) 2023-11-28T21:51:18.445 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_12_00_UTC 2023-11-28T21:51:18.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.445 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_12_00_UTC) 2023-11-28T21:51:18.508 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_13_00_UTC 2023-11-28T21:51:18.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.508 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_13_00_UTC) 2023-11-28T21:51:18.571 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_14_00_UTC 2023-11-28T21:51:18.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.571 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_14_00_UTC) 2023-11-28T21:51:18.638 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_15_00_UTC 2023-11-28T21:51:18.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.638 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_15_00_UTC) 2023-11-28T21:51:18.706 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_16_00_UTC 2023-11-28T21:51:18.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.706 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_16_00_UTC) 2023-11-28T21:51:18.777 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_17_00_UTC 2023-11-28T21:51:18.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.778 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_17_00_UTC) 2023-11-28T21:51:18.852 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_18_00_UTC 2023-11-28T21:51:18.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.852 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_18_00_UTC) 2023-11-28T21:51:18.927 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_19_00_UTC 2023-11-28T21:51:18.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.927 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_19_00_UTC) 2023-11-28T21:51:18.993 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_20_00_UTC 2023-11-28T21:51:18.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:18.993 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_20_00_UTC) 2023-11-28T21:51:19.102 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_21_00_UTC 2023-11-28T21:51:19.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.102 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_21_00_UTC) 2023-11-28T21:51:19.153 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_22_00_UTC 2023-11-28T21:51:19.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.153 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_22_00_UTC) 2023-11-28T21:51:19.209 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_23_00_UTC 2023-11-28T21:51:19.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.209 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_23_00_UTC) 2023-11-28T21:51:19.268 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_24_00_UTC 2023-11-28T21:51:19.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.269 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_24_00_UTC) 2023-11-28T21:51:19.332 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_25_00_UTC 2023-11-28T21:51:19.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.333 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_25_00_UTC) 2023-11-28T21:51:19.406 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_26_00_UTC 2023-11-28T21:51:19.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.406 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_26_00_UTC) 2023-11-28T21:51:19.475 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_27_00_UTC 2023-11-28T21:51:19.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.476 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_27_00_UTC) 2023-11-28T21:51:19.554 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_28_00_UTC 2023-11-28T21:51:19.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.554 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_28_00_UTC) 2023-11-28T21:51:19.625 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_29_00_UTC 2023-11-28T21:51:19.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.625 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_29_00_UTC) 2023-11-28T21:51:19.675 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_30_00_UTC 2023-11-28T21:51:19.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.675 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_30_00_UTC) 2023-11-28T21:51:19.728 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_31_00_UTC 2023-11-28T21:51:19.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.728 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_31_00_UTC) 2023-11-28T21:51:19.785 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_32_00_UTC 2023-11-28T21:51:19.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.786 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_32_00_UTC) 2023-11-28T21:51:19.855 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_33_00_UTC 2023-11-28T21:51:19.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.856 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_33_00_UTC) 2023-11-28T21:51:19.928 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_34_00_UTC 2023-11-28T21:51:19.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:19.928 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_34_00_UTC) 2023-11-28T21:51:20.004 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_35_00_UTC 2023-11-28T21:51:20.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.004 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_35_00_UTC) 2023-11-28T21:51:20.109 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_36_00_UTC 2023-11-28T21:51:20.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.110 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_36_00_UTC) 2023-11-28T21:51:20.213 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_37_00_UTC 2023-11-28T21:51:20.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.213 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_37_00_UTC) 2023-11-28T21:51:20.281 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_38_00_UTC 2023-11-28T21:51:20.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.281 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_38_00_UTC) 2023-11-28T21:51:20.329 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_39_00_UTC 2023-11-28T21:51:20.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.330 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_39_00_UTC) 2023-11-28T21:51:20.381 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_40_00_UTC 2023-11-28T21:51:20.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.382 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_40_00_UTC) 2023-11-28T21:51:20.436 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_41_00_UTC 2023-11-28T21:51:20.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.436 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_41_00_UTC) 2023-11-28T21:51:20.502 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_42_00_UTC 2023-11-28T21:51:20.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.503 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_42_00_UTC) 2023-11-28T21:51:20.568 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_43_00_UTC 2023-11-28T21:51:20.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.569 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_43_00_UTC) 2023-11-28T21:51:20.642 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_44_00_UTC 2023-11-28T21:51:20.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.642 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_44_00_UTC) 2023-11-28T21:51:20.707 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_45_00_UTC 2023-11-28T21:51:20.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.708 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_45_00_UTC) 2023-11-28T21:51:20.772 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_46_00_UTC 2023-11-28T21:51:20.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.772 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_46_00_UTC) 2023-11-28T21:51:20.840 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_47_00_UTC 2023-11-28T21:51:20.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.840 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_47_00_UTC) 2023-11-28T21:51:20.906 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_48_00_UTC 2023-11-28T21:51:20.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.906 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_48_00_UTC) 2023-11-28T21:51:20.976 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_49_00_UTC 2023-11-28T21:51:20.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:20.977 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_49_00_UTC) 2023-11-28T21:51:21.054 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_50_00_UTC 2023-11-28T21:51:21.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:21.054 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_50_00_UTC) 2023-11-28T21:51:21.171 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-11-28-21_51_00_UTC 2023-11-28T21:51:21.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:21.171 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-11-28-21_51_00_UTC) 2023-11-28T21:51:21.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:21.258 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-11-28T21:51:21.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T21:51:21.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.503+0000 7f93dd013640 1 Processor -- start 2023-11-28T21:51:21.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.504+0000 7f93dd013640 1 -- start start 2023-11-28T21:51:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.505+0000 7f93dd013640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cf940 0x7f93d8161800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:21.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.505+0000 7f93dd013640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93d80cfd20 0x7f93d80b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.505+0000 7f93dd013640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f93d80b5070 0x7f93d80b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.505+0000 7f93dd013640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d80b2930 con 0x7f93d80cfd20 2023-11-28T21:51:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.505+0000 7f93dd013640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d8161d40 con 0x7f93d80b5070 2023-11-28T21:51:21.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.505+0000 7f93dd013640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80b2650 con 0x7f93d80cf940 2023-11-28T21:51:21.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.505+0000 7f93d77fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93d80cfd20 0x7f93d80b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:21.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.505+0000 7f93d77fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93d80cfd20 0x7f93d80b4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54946/0 (socket says 172.21.15.45:54946) 2023-11-28T21:51:21.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.505+0000 7f93d77fe640 1 -- 172.21.15.45:0/72665210 learned_addr learned my addr 172.21.15.45:0/72665210 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:21.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.506+0000 7f93dc812640 1 --2- 172.21.15.45:0/72665210 >> v2:172.21.15.190:3300/0 conn(0x7f93d80b5070 0x7f93d80b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:21.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.506+0000 7f93d7fff640 1 --2- 172.21.15.45:0/72665210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cf940 0x7f93d8161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:21.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.506+0000 7f93d77fe640 1 -- 172.21.15.45:0/72665210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cf940 msgr2=0x7f93d8161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:21.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.506+0000 7f93d77fe640 1 --2- 172.21.15.45:0/72665210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cf940 0x7f93d8161800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:21.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.506+0000 7f93d77fe640 1 -- 172.21.15.45:0/72665210 >> v2:172.21.15.190:3300/0 conn(0x7f93d80b5070 msgr2=0x7f93d80b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:21.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.506+0000 7f93d77fe640 1 --2- 172.21.15.45:0/72665210 >> v2:172.21.15.190:3300/0 conn(0x7f93d80b5070 0x7f93d80b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:21.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.506+0000 7f93d77fe640 1 -- 172.21.15.45:0/72665210 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d80ce240 con 0x7f93d80cfd20 2023-11-28T21:51:21.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.506+0000 7f93d7fff640 1 --2- 172.21.15.45:0/72665210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cf940 0x7f93d8161800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:21.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.506+0000 7f93dc812640 1 --2- 172.21.15.45:0/72665210 >> v2:172.21.15.190:3300/0 conn(0x7f93d80b5070 0x7f93d80b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:21.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.507+0000 7f93d77fe640 1 --2- 172.21.15.45:0/72665210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80cfd20 0x7f93d80b4b30 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f93d80d0c30 tx=0x7f93cc02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaedaac7071af5a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:21.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.507+0000 7f93d6ffd640 1 -- 172.21.15.45:0/72665210 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc031740 con 0x7f93d80cfd20 2023-11-28T21:51:21.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.507+0000 7f93d6ffd640 1 -- 172.21.15.45:0/72665210 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc0318a0 con 0x7f93d80cfd20 2023-11-28T21:51:21.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.507+0000 7f93d6ffd640 1 -- 172.21.15.45:0/72665210 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc031c30 con 0x7f93d80cfd20 2023-11-28T21:51:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.508+0000 7f93dd013640 1 -- 172.21.15.45:0/72665210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80cfd20 msgr2=0x7f93d80b4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.508+0000 7f93dd013640 1 --2- 172.21.15.45:0/72665210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80cfd20 0x7f93d80b4b30 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f93d80d0c30 tx=0x7f93cc02b7e0 comp rx=0 tx=0).stop 2023-11-28T21:51:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.508+0000 7f93dd013640 1 -- 172.21.15.45:0/72665210 shutdown_connections 2023-11-28T21:51:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.508+0000 7f93dd013640 1 --2- 172.21.15.45:0/72665210 >> v2:172.21.15.190:3300/0 conn(0x7f93d80b5070 0x7f93d80b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.508+0000 7f93dd013640 1 --2- 172.21.15.45:0/72665210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80cfd20 0x7f93d80b4b30 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.508+0000 7f93dd013640 1 --2- 172.21.15.45:0/72665210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cf940 0x7f93d8161800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:21.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.508+0000 7f93dd013640 1 -- 172.21.15.45:0/72665210 >> 172.21.15.45:0/72665210 conn(0x7f93d806a500 msgr2=0x7f93d806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.508+0000 7f93dd013640 1 -- 172.21.15.45:0/72665210 shutdown_connections 2023-11-28T21:51:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.508+0000 7f93dd013640 1 -- 172.21.15.45:0/72665210 wait complete. 2023-11-28T21:51:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.510+0000 7f93dd013640 1 Processor -- start 2023-11-28T21:51:21.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.510+0000 7f93dd013640 1 -- start start 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.511+0000 7f93dd013640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80b5070 0x7f93d815ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.511+0000 7f93dd013640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93d80cf940 0x7f93d80c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.511+0000 7f93dd013640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cfd20 0x7f93d81e8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.511+0000 7f93dd013640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d815bf60 con 0x7f93d80b5070 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.511+0000 7f93dd013640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d815c420 con 0x7f93d80cf940 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.511+0000 7f93dd013640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d81e6f90 con 0x7f93d80cfd20 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.511+0000 7f93d7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80b5070 0x7f93d815ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.511+0000 7f93d7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80b5070 0x7f93d815ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54948/0 (socket says 172.21.15.45:54948) 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.511+0000 7f93d7fff640 1 -- 172.21.15.45:0/3942662389 learned_addr learned my addr 172.21.15.45:0/3942662389 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93d7fff640 1 -- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cfd20 msgr2=0x7f93d81e8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93d7fff640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cfd20 0x7f93d81e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93d7fff640 1 -- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93d80cf940 msgr2=0x7f93d80c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93d7fff640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93d80cf940 0x7f93d80c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93d7fff640 1 -- 172.21.15.45:0/3942662389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc0094d0 con 0x7f93d80b5070 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93d7fff640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80b5070 0x7f93d815ba20 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c820 tx=0x7f93c800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93d4ff9640 1 -- 172.21.15.45:0/3942662389 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8007a30 con 0x7f93d80b5070 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93d4ff9640 1 -- 172.21.15.45:0/3942662389 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c8007b90 con 0x7f93d80b5070 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93d4ff9640 1 -- 172.21.15.45:0/3942662389 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c800ce10 con 0x7f93d80b5070 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.512+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d81e7270 con 0x7f93d80b5070 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.513+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d81e7830 con 0x7f93d80b5070 2023-11-28T21:51:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.514+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d8066dd0 con 0x7f93d80b5070 2023-11-28T21:51:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.515+0000 7f93d4ff9640 1 -- 172.21.15.45:0/3942662389 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 85) v1 ==== 89362+0+0 (secure 0 0 0) 0x7f93c80113f0 con 0x7f93d80b5070 2023-11-28T21:51:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.515+0000 7f93d4ff9640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f93c406afe0 0x7f93c406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.516+0000 7f93d4ff9640 1 -- 172.21.15.45:0/3942662389 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f93c8089d60 con 0x7f93d80b5070 2023-11-28T21:51:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.516+0000 7f93d77fe640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f93c406afe0 0x7f93c406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:21.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.517+0000 7f93d77fe640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f93c406afe0 0x7f93c406d4a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93cc035000 tx=0x7f93cc030000 comp rx=0 tx=0).ready entity=mgr.7663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.520+0000 7f93d4ff9640 1 -- 172.21.15.45:0/3942662389 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93c8054840 con 0x7f93d80b5070 2023-11-28T21:51:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:21.765+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f93d8071b30 con 0x7f93d80b5070 2023-11-28T21:51:22.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.094+0000 7f93d4ff9640 1 -- 172.21.15.45:0/3942662389 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v86) v1 ==== 93+0+0 (secure 0 0 0) 0x7f93c8057e60 con 0x7f93d80b5070 2023-11-28T21:51:22.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.097+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f93c406afe0 msgr2=0x7f93c406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:22.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.097+0000 7f93dd013640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f93c406afe0 0x7f93c406d4a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93cc035000 tx=0x7f93cc030000 comp rx=0 tx=0).stop 2023-11-28T21:51:22.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.097+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80b5070 msgr2=0x7f93d815ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:22.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.097+0000 7f93dd013640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80b5070 0x7f93d815ba20 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c820 tx=0x7f93c800cce0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.100+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 shutdown_connections 2023-11-28T21:51:22.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.100+0000 7f93dd013640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f93c406afe0 0x7f93c406d4a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.100+0000 7f93dd013640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f93d80cfd20 0x7f93d81e8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.100+0000 7f93dd013640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93d80cf940 0x7f93d80c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.100+0000 7f93dd013640 1 --2- 172.21.15.45:0/3942662389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d80b5070 0x7f93d815ba20 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.100+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 >> 172.21.15.45:0/3942662389 conn(0x7f93d806a500 msgr2=0x7f93d80b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:22.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.103+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 shutdown_connections 2023-11-28T21:51:22.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.103+0000 7f93dd013640 1 -- 172.21.15.45:0/3942662389 wait complete. 2023-11-28T21:51:22.148 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T21:51:22.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T21:51:22.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T21:51:22.419 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T21:51:22.420 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:51:22.420 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:22.470 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:51:22.470 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:22.553 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:51:22.728 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:22.726+0000 7f5308632040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T21:51:22.740 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:22.739+0000 7f5686e37040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T21:51:22.784 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:22.782+0000 7f5308632040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T21:51:22.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.790+0000 7f704c84b640 1 Processor -- start 2023-11-28T21:51:22.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.790+0000 7f704c84b640 1 -- start start 2023-11-28T21:51:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.791+0000 7f704c84b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7048155780 0x7f70480abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.791+0000 7f704c84b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70480ae540 0x7f70480ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.791+0000 7f704c84b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704815c3c0 con 0x7f70480ce630 2023-11-28T21:51:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.791+0000 7f704c84b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70480ceac0 con 0x7f7048155780 2023-11-28T21:51:22.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.792+0000 7f704c84b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7048155b60 con 0x7f70480ae540 2023-11-28T21:51:22.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.792+0000 7f70477fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7048155780 0x7f70480abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:22.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.792+0000 7f7047fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f70480ce630 0x7f704815ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38738/0 (socket says 172.21.15.45:38738) 2023-11-28T21:51:22.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.792+0000 7f7047fff640 1 -- 172.21.15.45:0/3128497500 learned_addr learned my addr 172.21.15.45:0/3128497500 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:22.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.792+0000 7f703effd640 1 --2- 172.21.15.45:0/3128497500 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70480ae540 0x7f70480ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:22.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.792+0000 7f70477fe640 1 -- 172.21.15.45:0/3128497500 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70480ae540 msgr2=0x7f70480ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:22.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.792+0000 7f70477fe640 1 --2- 172.21.15.45:0/3128497500 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70480ae540 0x7f70480ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.792+0000 7f70477fe640 1 -- 172.21.15.45:0/3128497500 >> v1:172.21.15.45:6789/0 conn(0x7f70480ce630 legacy=0x7f704815ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:22.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.792+0000 7f7046ffd640 1 -- 172.21.15.45:0/3128497500 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725973577 0 0) 0x7f704815c3c0 con 0x7f70480ce630 2023-11-28T21:51:22.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.793+0000 7f70477fe640 1 -- 172.21.15.45:0/3128497500 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704815ba90 con 0x7f7048155780 2023-11-28T21:51:22.799 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:22.795+0000 7f5686e37040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T21:51:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.794+0000 7f70477fe640 1 --2- 172.21.15.45:0/3128497500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7048155780 0x7f70480abd70 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f7038009dc0 tx=0x7f703802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e75881eb2ddb97 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.794+0000 7f7046ffd640 1 -- 172.21.15.45:0/3128497500 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703802cc40 con 0x7f7048155780 2023-11-28T21:51:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f7046ffd640 1 -- 172.21.15.45:0/3128497500 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703802cda0 con 0x7f7048155780 2023-11-28T21:51:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f7046ffd640 1 -- 172.21.15.45:0/3128497500 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038034660 con 0x7f7048155780 2023-11-28T21:51:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f704c84b640 1 -- 172.21.15.45:0/3128497500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7048155780 msgr2=0x7f70480abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f704c84b640 1 --2- 172.21.15.45:0/3128497500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7048155780 0x7f70480abd70 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f7038009dc0 tx=0x7f703802be90 comp rx=0 tx=0).stop 2023-11-28T21:51:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f704c84b640 1 -- 172.21.15.45:0/3128497500 shutdown_connections 2023-11-28T21:51:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f704c84b640 1 --2- 172.21.15.45:0/3128497500 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70480ae540 0x7f70480ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f704c84b640 1 --2- 172.21.15.45:0/3128497500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7048155780 0x7f70480abd70 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f704c84b640 1 -- 172.21.15.45:0/3128497500 >> 172.21.15.45:0/3128497500 conn(0x7f704806ab30 msgr2=0x7f70480b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f704c84b640 1 -- 172.21.15.45:0/3128497500 shutdown_connections 2023-11-28T21:51:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.795+0000 7f704c84b640 1 -- 172.21.15.45:0/3128497500 wait complete. 2023-11-28T21:51:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.797+0000 7f704c84b640 1 Processor -- start 2023-11-28T21:51:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.797+0000 7f704c84b640 1 -- start start 2023-11-28T21:51:22.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f704c84b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70480ae540 0x7f70481f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:22.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f704c84b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70480ce630 0x7f70481fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:22.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f704c84b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7048155780 0x7f704809aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:22.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f704c84b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70481f7710 con 0x7f70480ae540 2023-11-28T21:51:22.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f704c84b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704814c280 con 0x7f70480ce630 2023-11-28T21:51:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f704c84b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70481fbd20 con 0x7f7048155780 2023-11-28T21:51:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f70477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70480ae540 0x7f70481f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f70477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70480ae540 0x7f70481f71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54966/0 (socket says 172.21.15.45:54966) 2023-11-28T21:51:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f70477fe640 1 -- 172.21.15.45:0/2972484183 learned_addr learned my addr 172.21.15.45:0/2972484183 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f7047fff640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7048155780 0x7f704809aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f703effd640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70480ce630 0x7f70481fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f70477fe640 1 -- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7048155780 msgr2=0x7f704809aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.799+0000 7f70477fe640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7048155780 0x7f704809aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.800+0000 7f70477fe640 1 -- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70480ce630 msgr2=0x7f70481fb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.800+0000 7f70477fe640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70480ce630 0x7f70481fb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.800+0000 7f70477fe640 1 -- 172.21.15.45:0/2972484183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038009a70 con 0x7f70480ae540 2023-11-28T21:51:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.800+0000 7f70477fe640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70480ae540 0x7f70481f71d0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f7038009dc0 tx=0x7f7038004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.800+0000 7f7044ff9640 1 -- 172.21.15.45:0/2972484183 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038040650 con 0x7f70480ae540 2023-11-28T21:51:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.800+0000 7f7044ff9640 1 -- 172.21.15.45:0/2972484183 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70380407b0 con 0x7f70480ae540 2023-11-28T21:51:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.800+0000 7f7044ff9640 1 -- 172.21.15.45:0/2972484183 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038040b60 con 0x7f70480ae540 2023-11-28T21:51:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.800+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7048146750 con 0x7f70480ae540 2023-11-28T21:51:22.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.800+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7048146d10 con 0x7f70480ae540 2023-11-28T21:51:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.802+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704804e920 con 0x7f70480ae540 2023-11-28T21:51:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.803+0000 7f7044ff9640 1 -- 172.21.15.45:0/2972484183 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 86) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f7038040cc0 con 0x7f70480ae540 2023-11-28T21:51:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.804+0000 7f7044ff9640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f702806af60 0x7f702806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.804+0000 7f7044ff9640 1 -- 172.21.15.45:0/2972484183 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f70380ac8f0 con 0x7f70480ae540 2023-11-28T21:51:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.804+0000 7f703effd640 1 -- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f702806af60 msgr2=0x7f702806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.804+0000 7f703effd640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f702806af60 0x7f702806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:22.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.808+0000 7f7044ff9640 1 -- 172.21.15.45:0/2972484183 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7038077580 con 0x7f70480ae540 2023-11-28T21:51:22.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:22.997+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7048067410 con 0x7f70480ae540 2023-11-28T21:51:23.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.000+0000 7f7044ff9640 1 -- 172.21.15.45:0/2972484183 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) v1 ==== 74+0+17436 (secure 0 0 0) 0x7f703807aba0 con 0x7f70480ae540 2023-11-28T21:51:23.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":169,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:51:22.080667+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:55:10.712027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-11-28T20:55:11.779380+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":110,"snap_epoch":169,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/943491299":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1767202704":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2782217070":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/290777784":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6833/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6832/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1078115353":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2712902162":"2023-11-29T20:55:29.635279+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":52},{"begin":59,"length":52}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":41,"length":14},{"begin":97,"length":14}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:51:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.002+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f702806af60 msgr2=0x7f702806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.002+0000 7f704c84b640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f702806af60 0x7f702806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.002+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70480ae540 msgr2=0x7f70481f71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.002+0000 7f704c84b640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70480ae540 0x7f70481f71d0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f7038009dc0 tx=0x7f7038004a40 comp rx=0 tx=0).stop 2023-11-28T21:51:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.002+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 shutdown_connections 2023-11-28T21:51:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.002+0000 7f704c84b640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f702806af60 0x7f702806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.002+0000 7f704c84b640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7048155780 0x7f704809aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.002+0000 7f704c84b640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70480ce630 0x7f70481fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.003+0000 7f704c84b640 1 --2- 172.21.15.45:0/2972484183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70480ae540 0x7f70481f71d0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.003+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 >> 172.21.15.45:0/2972484183 conn(0x7f704806ab30 msgr2=0x7f704815d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.003+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 shutdown_connections 2023-11-28T21:51:23.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.003+0000 7f704c84b640 1 -- 172.21.15.45:0/2972484183 wait complete. 2023-11-28T21:51:23.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:55:10.712027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:55:11.779380+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '169', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 110, 'snap_epoch': 169, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T21:51:23.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T21:51:23.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.227+0000 7efcbcf11640 1 Processor -- start 2023-11-28T21:51:23.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.227+0000 7efcbcf11640 1 -- start start 2023-11-28T21:51:23.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.229+0000 7efcbcf11640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efcb815e860 0x7efcb80cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.229+0000 7efcbcf11640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80cbd00 0x7efcb80cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.229+0000 7efcbcf11640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb80cd270 con 0x7efcb80afd40 2023-11-28T21:51:23.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.229+0000 7efcbcf11640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb80adcb0 con 0x7efcb815e860 2023-11-28T21:51:23.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.229+0000 7efcbcf11640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80cd530 con 0x7efcb80cbd00 2023-11-28T21:51:23.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.229+0000 7efcaffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efcb80afd40 0x7efcb80cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38748/0 (socket says 172.21.15.45:38748) 2023-11-28T21:51:23.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.229+0000 7efcaffff640 1 -- 172.21.15.45:0/1226381286 learned_addr learned my addr 172.21.15.45:0/1226381286 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:23.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.229+0000 7efcb7fff640 1 --2- 172.21.15.45:0/1226381286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80cbd00 0x7efcb80cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.229+0000 7efcb77fe640 1 --2- 172.21.15.45:0/1226381286 >> v2:172.21.15.190:3300/0 conn(0x7efcb815e860 0x7efcb80cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.230+0000 7efcb6ffd640 1 -- 172.21.15.45:0/1226381286 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292654776 0 0) 0x7efcb80cd270 con 0x7efcb80afd40 2023-11-28T21:51:23.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.230+0000 7efcb6ffd640 1 -- 172.21.15.45:0/1226381286 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca4003520 con 0x7efcb80afd40 2023-11-28T21:51:23.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.230+0000 7efcb7fff640 1 -- 172.21.15.45:0/1226381286 >> v2:172.21.15.190:3300/0 conn(0x7efcb815e860 msgr2=0x7efcb80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:23.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.230+0000 7efcb7fff640 1 --2- 172.21.15.45:0/1226381286 >> v2:172.21.15.190:3300/0 conn(0x7efcb815e860 0x7efcb80cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.230+0000 7efcb7fff640 1 -- 172.21.15.45:0/1226381286 >> v1:172.21.15.45:6789/0 conn(0x7efcb80afd40 legacy=0x7efcb80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.230+0000 7efcb7fff640 1 -- 172.21.15.45:0/1226381286 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb80cfa70 con 0x7efcb80cbd00 2023-11-28T21:51:23.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.231+0000 7efcb7fff640 1 --2- 172.21.15.45:0/1226381286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80cbd00 0x7efcb80cc170 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009d90 tx=0x7efcb002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b24e0b637b7e44 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:23.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcb6ffd640 1 -- 172.21.15.45:0/1226381286 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb002cc50 con 0x7efcb80cbd00 2023-11-28T21:51:23.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcb6ffd640 1 -- 172.21.15.45:0/1226381286 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb002cdb0 con 0x7efcb80cbd00 2023-11-28T21:51:23.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcb6ffd640 1 -- 172.21.15.45:0/1226381286 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb0034660 con 0x7efcb80cbd00 2023-11-28T21:51:23.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcbcf11640 1 -- 172.21.15.45:0/1226381286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80cbd00 msgr2=0x7efcb80cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcbcf11640 1 --2- 172.21.15.45:0/1226381286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80cbd00 0x7efcb80cc170 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009d90 tx=0x7efcb002bea0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcbcf11640 1 -- 172.21.15.45:0/1226381286 shutdown_connections 2023-11-28T21:51:23.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcbcf11640 1 --2- 172.21.15.45:0/1226381286 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80cbd00 0x7efcb80cc170 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcbcf11640 1 --2- 172.21.15.45:0/1226381286 >> v2:172.21.15.190:3300/0 conn(0x7efcb815e860 0x7efcb80cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcbcf11640 1 -- 172.21.15.45:0/1226381286 >> 172.21.15.45:0/1226381286 conn(0x7efcb80b29d0 msgr2=0x7efcb80b2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:23.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcbcf11640 1 -- 172.21.15.45:0/1226381286 shutdown_connections 2023-11-28T21:51:23.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.232+0000 7efcbcf11640 1 -- 172.21.15.45:0/1226381286 wait complete. 2023-11-28T21:51:23.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.234+0000 7efcbcf11640 1 Processor -- start 2023-11-28T21:51:23.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.234+0000 7efcbcf11640 1 -- start start 2023-11-28T21:51:23.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.235+0000 7efcbcf11640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80afd40 0x7efcb814f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.235+0000 7efcbcf11640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb80cbd00 0x7efcb80d0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.235+0000 7efcbcf11640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcb815e860 0x7efcb80d1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.235+0000 7efcbcf11640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb81f71f0 con 0x7efcb80cbd00 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.235+0000 7efcbcf11640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb806c3e0 con 0x7efcb815e860 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.235+0000 7efcbcf11640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb81f68c0 con 0x7efcb80afd40 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.235+0000 7efcaffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb80cbd00 0x7efcb80d0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.235+0000 7efcaffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb80cbd00 0x7efcb80d0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54978/0 (socket says 172.21.15.45:54978) 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.235+0000 7efcaffff640 1 -- 172.21.15.45:0/920263381 learned_addr learned my addr 172.21.15.45:0/920263381 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcb77fe640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80afd40 0x7efcb814f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcb7fff640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcb815e860 0x7efcb80d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcaffff640 1 -- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80afd40 msgr2=0x7efcb814f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcaffff640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80afd40 0x7efcb814f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcaffff640 1 -- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcb815e860 msgr2=0x7efcb80d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcaffff640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcb815e860 0x7efcb80d1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcaffff640 1 -- 172.21.15.45:0/920263381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0009a70 con 0x7efcb80cbd00 2023-11-28T21:51:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcb7fff640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcb815e860 0x7efcb80d1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcaffff640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb80cbd00 0x7efcb80d0e30 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7efca000c410 tx=0x7efca000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.236+0000 7efcb4ff9640 1 -- 172.21.15.45:0/920263381 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0005ce0 con 0x7efcb80cbd00 2023-11-28T21:51:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.237+0000 7efcb4ff9640 1 -- 172.21.15.45:0/920263381 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca0005e40 con 0x7efcb80cbd00 2023-11-28T21:51:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.237+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb81f6ba0 con 0x7efcb80cbd00 2023-11-28T21:51:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.237+0000 7efcb4ff9640 1 -- 172.21.15.45:0/920263381 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca00072f0 con 0x7efcb80cbd00 2023-11-28T21:51:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.237+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb81f7680 con 0x7efcb80cbd00 2023-11-28T21:51:23.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.238+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb81f73a0 con 0x7efcb80cbd00 2023-11-28T21:51:23.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.240+0000 7efcb4ff9640 1 -- 172.21.15.45:0/920263381 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 86) v1 ==== 89345+0+0 (secure 0 0 0) 0x7efca0007510 con 0x7efcb80cbd00 2023-11-28T21:51:23.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.241+0000 7efcb4ff9640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7efc9806af60 0x7efc9806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.241+0000 7efcb4ff9640 1 -- 172.21.15.45:0/920263381 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6489+0+0 (secure 0 0 0) 0x7efca0089090 con 0x7efcb80cbd00 2023-11-28T21:51:23.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.241+0000 7efcb77fe640 1 -- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7efc9806af60 msgr2=0x7efc9806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:23.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.241+0000 7efcb77fe640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7efc9806af60 0x7efc9806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:23.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.249+0000 7efcb4ff9640 1 -- 172.21.15.45:0/920263381 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efcb81f73a0 con 0x7efcb80cbd00 2023-11-28T21:51:23.258 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.257+0000 7f5686e37040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T21:51:23.284 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.282+0000 7f5308632040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T21:51:23.311 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.310+0000 7f5686e37040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T21:51:23.339 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.337+0000 7f5308632040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T21:51:23.360 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.359+0000 7f5686e37040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T21:51:23.392 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.390+0000 7f5308632040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T21:51:23.420 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.419+0000 7f5686e37040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T21:51:23.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.442+0000 7efcb77fe640 1 -- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7efc9806af60 msgr2=0x7efc9806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:23.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.442+0000 7efcb77fe640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7efc9806af60 0x7efc9806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:51:23.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.446+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efcb809f030 con 0x7efcb80cbd00 2023-11-28T21:51:23.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.447+0000 7efcb4ff9640 1 -- 172.21.15.45:0/920263381 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v170) v1 ==== 93+0+10 (secure 0 0 0) 0x7efca0057920 con 0x7efcb80cbd00 2023-11-28T21:51:23.450 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T21:51:23.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7efc9806af60 msgr2=0x7efc9806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7efc9806af60 0x7efc9806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb80cbd00 msgr2=0x7efcb80d0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb80cbd00 0x7efcb80d0e30 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7efca000c410 tx=0x7efca000c8d0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 shutdown_connections 2023-11-28T21:51:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcb815e860 0x7efcb80d1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb80cbd00 0x7efcb80d0e30 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7efc9806af60 0x7efc9806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 --2- 172.21.15.45:0/920263381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcb80afd40 0x7efcb814f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 >> 172.21.15.45:0/920263381 conn(0x7efcb80b29d0 msgr2=0x7efcb80ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:23.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 shutdown_connections 2023-11-28T21:51:23.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.450+0000 7efcbcf11640 1 -- 172.21.15.45:0/920263381 wait complete. 2023-11-28T21:51:23.457 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.455+0000 7f5308632040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T21:51:23.468 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.467+0000 7f5686e37040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T21:51:23.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T21:51:23.512 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.510+0000 7f5308632040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T21:51:23.585 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.584+0000 7f5686e37040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T21:51:23.631 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.629+0000 7f5308632040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T21:51:23.650 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.647+0000 7f5686e37040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T21:51:23.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.672+0000 7ff89a49b640 1 Processor -- start 2023-11-28T21:51:23.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.673+0000 7ff89a49b640 1 -- start start 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.674+0000 7ff89a49b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff89415b600 0x7ff8940cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.674+0000 7ff89a49b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff8940cbeb0 0x7ff894099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.674+0000 7ff89a49b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff8940cc310 0x7ff894099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.674+0000 7ff89a49b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8940b0970 con 0x7ff8940cc310 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.674+0000 7ff89a49b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8940b0f30 con 0x7ff8940cbeb0 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.674+0000 7ff89a49b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940c49a0 con 0x7ff89415b600 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.674+0000 7ff898a38640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff8940cc310 0x7ff894099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.675+0000 7ff898a38640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff8940cc310 0x7ff894099f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54984/0 (socket says 172.21.15.45:54984) 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.675+0000 7ff898a38640 1 -- 172.21.15.45:0/3217900423 learned_addr learned my addr 172.21.15.45:0/3217900423 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.675+0000 7ff893fff640 1 --2- 172.21.15.45:0/3217900423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff89415b600 0x7ff8940cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.675+0000 7ff88bfff640 1 --2- 172.21.15.45:0/3217900423 >> v2:172.21.15.190:3300/0 conn(0x7ff8940cbeb0 0x7ff894099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.675+0000 7ff898a38640 1 -- 172.21.15.45:0/3217900423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff89415b600 msgr2=0x7ff8940cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.675+0000 7ff898a38640 1 --2- 172.21.15.45:0/3217900423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff89415b600 0x7ff8940cf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.675+0000 7ff898a38640 1 -- 172.21.15.45:0/3217900423 >> v2:172.21.15.190:3300/0 conn(0x7ff8940cbeb0 msgr2=0x7ff894099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.675+0000 7ff898a38640 1 --2- 172.21.15.45:0/3217900423 >> v2:172.21.15.190:3300/0 conn(0x7ff8940cbeb0 0x7ff894099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.675+0000 7ff898a38640 1 -- 172.21.15.45:0/3217900423 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8940cfe90 con 0x7ff8940cc310 2023-11-28T21:51:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.676+0000 7ff898a38640 1 --2- 172.21.15.45:0/3217900423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cc310 0x7ff894099f80 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7ff894071ee0 tx=0x7ff88c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe525333a46e03a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.676+0000 7ff8937fe640 1 -- 172.21.15.45:0/3217900423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c02c9e0 con 0x7ff8940cc310 2023-11-28T21:51:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff8937fe640 1 -- 172.21.15.45:0/3217900423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88c02cb40 con 0x7ff8940cc310 2023-11-28T21:51:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff8937fe640 1 -- 172.21.15.45:0/3217900423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c0343f0 con 0x7ff8940cc310 2023-11-28T21:51:23.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff89a49b640 1 -- 172.21.15.45:0/3217900423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cc310 msgr2=0x7ff894099f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff89a49b640 1 --2- 172.21.15.45:0/3217900423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cc310 0x7ff894099f80 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7ff894071ee0 tx=0x7ff88c02be40 comp rx=0 tx=0).stop 2023-11-28T21:51:23.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff89a49b640 1 -- 172.21.15.45:0/3217900423 shutdown_connections 2023-11-28T21:51:23.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff89a49b640 1 --2- 172.21.15.45:0/3217900423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cc310 0x7ff894099f80 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff89a49b640 1 --2- 172.21.15.45:0/3217900423 >> v2:172.21.15.190:3300/0 conn(0x7ff8940cbeb0 0x7ff894099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff89a49b640 1 --2- 172.21.15.45:0/3217900423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff89415b600 0x7ff8940cf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff89a49b640 1 -- 172.21.15.45:0/3217900423 >> 172.21.15.45:0/3217900423 conn(0x7ff8940b2b20 msgr2=0x7ff8940b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:23.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff89a49b640 1 -- 172.21.15.45:0/3217900423 shutdown_connections 2023-11-28T21:51:23.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.677+0000 7ff89a49b640 1 -- 172.21.15.45:0/3217900423 wait complete. 2023-11-28T21:51:23.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.679+0000 7ff89a49b640 1 Processor -- start 2023-11-28T21:51:23.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.679+0000 7ff89a49b640 1 -- start start 2023-11-28T21:51:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.680+0000 7ff89a49b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cbeb0 0x7ff8940af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.680+0000 7ff89a49b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8940cc310 0x7ff8940c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.680+0000 7ff89a49b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89415b600 0x7ff8940c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.680+0000 7ff89a49b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff894157990 con 0x7ff8940cbeb0 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.680+0000 7ff89a49b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940afdc0 con 0x7ff89415b600 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.680+0000 7ff89a49b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940cafb0 con 0x7ff8940cc310 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff893fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cbeb0 0x7ff8940af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff893fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cbeb0 0x7ff8940af880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54994/0 (socket says 172.21.15.45:54994) 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff893fff640 1 -- 172.21.15.45:0/2738758303 learned_addr learned my addr 172.21.15.45:0/2738758303 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff88bfff640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8940cc310 0x7ff8940c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff898a38640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89415b600 0x7ff8940c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff893fff640 1 -- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8940cc310 msgr2=0x7ff8940c82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff893fff640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8940cc310 0x7ff8940c82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff893fff640 1 -- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89415b600 msgr2=0x7ff8940c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff893fff640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89415b600 0x7ff8940c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff893fff640 1 -- 172.21.15.45:0/2738758303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c009b30 con 0x7ff8940cbeb0 2023-11-28T21:51:23.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff898a38640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89415b600 0x7ff8940c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.681+0000 7ff893fff640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cbeb0 0x7ff8940af880 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7ff88400e870 tx=0x7ff88400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.682+0000 7ff8917fa640 1 -- 172.21.15.45:0/2738758303 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8840094f0 con 0x7ff8940cbeb0 2023-11-28T21:51:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.682+0000 7ff8917fa640 1 -- 172.21.15.45:0/2738758303 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88400c940 con 0x7ff8940cbeb0 2023-11-28T21:51:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.682+0000 7ff8917fa640 1 -- 172.21.15.45:0/2738758303 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88400ccf0 con 0x7ff8940cbeb0 2023-11-28T21:51:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.682+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8940b4820 con 0x7ff8940cbeb0 2023-11-28T21:51:23.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.682+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8940b4de0 con 0x7ff8940cbeb0 2023-11-28T21:51:23.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.683+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8940733d0 con 0x7ff8940cbeb0 2023-11-28T21:51:23.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.684+0000 7ff8917fa640 1 -- 172.21.15.45:0/2738758303 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 86) v1 ==== 89345+0+0 (secure 0 0 0) 0x7ff884013470 con 0x7ff8940cbeb0 2023-11-28T21:51:23.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.685+0000 7ff8917fa640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7ff87406af60 0x7ff87406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:23.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.685+0000 7ff8917fa640 1 -- 172.21.15.45:0/2738758303 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6489+0+0 (secure 0 0 0) 0x7ff884089b20 con 0x7ff8940cbeb0 2023-11-28T21:51:23.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.685+0000 7ff88bfff640 1 -- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7ff87406af60 msgr2=0x7ff87406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:23.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.685+0000 7ff88bfff640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7ff87406af60 0x7ff87406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:23.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.693+0000 7ff8917fa640 1 -- 172.21.15.45:0/2738758303 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8840547c0 con 0x7ff8940cbeb0 2023-11-28T21:51:23.699 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.698+0000 7f5686e37040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T21:51:23.702 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.700+0000 7f5308632040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T21:51:23.757 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.756+0000 7f5686e37040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T21:51:23.758 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.756+0000 7f5308632040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T21:51:23.816 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:23.815+0000 7f5686e37040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T21:51:23.823 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.820+0000 7f5308632040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T21:51:23.876 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:23.873+0000 7f5308632040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T21:51:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.886+0000 7ff88bfff640 1 -- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7ff87406af60 msgr2=0x7ff87406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.886+0000 7ff88bfff640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7ff87406af60 0x7ff87406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:51:23.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.895+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff8940b6fb0 con 0x7ff8940cbeb0 2023-11-28T21:51:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.896+0000 7ff8917fa640 1 -- 172.21.15.45:0/2738758303 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v170) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff884017020 con 0x7ff8940cbeb0 2023-11-28T21:51:23.898 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T21:51:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7ff87406af60 msgr2=0x7ff87406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7ff87406af60 0x7ff87406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cbeb0 msgr2=0x7ff8940af880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cbeb0 0x7ff8940af880 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7ff88400e870 tx=0x7ff88400ed30 comp rx=0 tx=0).stop 2023-11-28T21:51:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 shutdown_connections 2023-11-28T21:51:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89415b600 0x7ff8940c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7ff87406af60 0x7ff87406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8940cc310 0x7ff8940c82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 --2- 172.21.15.45:0/2738758303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8940cbeb0 0x7ff8940af880 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:23.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 >> 172.21.15.45:0/2738758303 conn(0x7ff8940b2b20 msgr2=0x7ff8940cef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:23.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 shutdown_connections 2023-11-28T21:51:23.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:23.898+0000 7ff89a49b640 1 -- 172.21.15.45:0/2738758303 wait complete. 2023-11-28T21:51:23.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T21:51:24.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.186+0000 7f64b8a2b640 1 Processor -- start 2023-11-28T21:51:24.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.186+0000 7f64b8a2b640 1 -- start start 2023-11-28T21:51:24.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.188+0000 7f64b8a2b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64b4157c10 0x7f64b406da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.188+0000 7f64b8a2b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b406ca60 con 0x7f64b415e4e0 2023-11-28T21:51:24.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.188+0000 7f64b8a2b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b415eb40 con 0x7f64b406a500 2023-11-28T21:51:24.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.188+0000 7f64b8a2b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b415e930 con 0x7f64b4157c10 2023-11-28T21:51:24.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.188+0000 7f64b259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64b4157c10 0x7f64b406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:24.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.188+0000 7f64b2d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f64b415e4e0 0x7f64b40c9e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33074/0 (socket says 172.21.15.45:33074) 2023-11-28T21:51:24.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.188+0000 7f64b2d9d640 1 -- 172.21.15.45:0/3665723297 learned_addr learned my addr 172.21.15.45:0/3665723297 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:24.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.189+0000 7f64b259c640 1 -- 172.21.15.45:0/3665723297 >> v1:172.21.15.190:6789/0 conn(0x7f64b406a500 legacy=0x7f64b415ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:24.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.189+0000 7f64b259c640 1 -- 172.21.15.45:0/3665723297 >> v1:172.21.15.45:6789/0 conn(0x7f64b415e4e0 legacy=0x7f64b40c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.189+0000 7f64b259c640 1 -- 172.21.15.45:0/3665723297 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b4157ff0 con 0x7f64b4157c10 2023-11-28T21:51:24.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.189+0000 7f64b159a640 1 -- 172.21.15.45:0/3665723297 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087174565 0 0) 0x7f64b406ca60 con 0x7f64b415e4e0 2023-11-28T21:51:24.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.190+0000 7f64b259c640 1 --2- 172.21.15.45:0/3665723297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b4157c10 0x7f64b406da50 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f64a40097a0 tx=0x7f64a402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c565a490143c9c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:24.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.190+0000 7f64b159a640 1 -- 172.21.15.45:0/3665723297 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4030610 con 0x7f64b4157c10 2023-11-28T21:51:24.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.191+0000 7f64b159a640 1 -- 172.21.15.45:0/3665723297 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a4030770 con 0x7f64b4157c10 2023-11-28T21:51:24.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.191+0000 7f64b159a640 1 -- 172.21.15.45:0/3665723297 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4030b00 con 0x7f64b4157c10 2023-11-28T21:51:24.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.191+0000 7f64b8a2b640 1 -- 172.21.15.45:0/3665723297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b4157c10 msgr2=0x7f64b406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.191+0000 7f64b8a2b640 1 --2- 172.21.15.45:0/3665723297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b4157c10 0x7f64b406da50 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f64a40097a0 tx=0x7f64a402b960 comp rx=0 tx=0).stop 2023-11-28T21:51:24.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.191+0000 7f64b8a2b640 1 -- 172.21.15.45:0/3665723297 shutdown_connections 2023-11-28T21:51:24.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.191+0000 7f64b8a2b640 1 --2- 172.21.15.45:0/3665723297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b4157c10 0x7f64b406da50 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.191+0000 7f64b8a2b640 1 -- 172.21.15.45:0/3665723297 >> 172.21.15.45:0/3665723297 conn(0x7f64b40ced60 msgr2=0x7f64b40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:24.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.192+0000 7f64b8a2b640 1 -- 172.21.15.45:0/3665723297 shutdown_connections 2023-11-28T21:51:24.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.192+0000 7f64b8a2b640 1 -- 172.21.15.45:0/3665723297 wait complete. 2023-11-28T21:51:24.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.193+0000 7f64b8a2b640 1 Processor -- start 2023-11-28T21:51:24.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.194+0000 7f64b8a2b640 1 -- start start 2023-11-28T21:51:24.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b8a2b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64b406a500 0x7f64b41543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b8a2b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64b4157c10 0x7f64b4156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b8a2b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b415e4e0 0x7f64b40cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b8a2b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40abef0 con 0x7f64b415e4e0 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b8a2b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40ac1d0 con 0x7f64b406a500 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b8a2b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40ac4b0 con 0x7f64b4157c10 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b415e4e0 0x7f64b40cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b415e4e0 0x7f64b40cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60852/0 (socket says 172.21.15.45:60852) 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b2d9d640 1 -- 172.21.15.45:0/710276012 learned_addr learned my addr 172.21.15.45:0/710276012 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b1d9b640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64b4157c10 0x7f64b4156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.195+0000 7f64b259c640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64b406a500 0x7f64b41543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b2d9d640 1 -- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64b4157c10 msgr2=0x7f64b4156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b2d9d640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64b4157c10 0x7f64b4156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b2d9d640 1 -- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64b406a500 msgr2=0x7f64b41543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b2d9d640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64b406a500 0x7f64b41543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b2d9d640 1 -- 172.21.15.45:0/710276012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a4009400 con 0x7f64b415e4e0 2023-11-28T21:51:24.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b259c640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64b406a500 0x7f64b41543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:24.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b1d9b640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64b4157c10 0x7f64b4156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:24.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b2d9d640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b415e4e0 0x7f64b40cd930 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00c780 tx=0x7f64ac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:24.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64a2ffd640 1 -- 172.21.15.45:0/710276012 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00ce30 con 0x7f64b415e4e0 2023-11-28T21:51:24.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64a2ffd640 1 -- 172.21.15.45:0/710276012 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac00a640 con 0x7f64b415e4e0 2023-11-28T21:51:24.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64a2ffd640 1 -- 172.21.15.45:0/710276012 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00aa70 con 0x7f64b415e4e0 2023-11-28T21:51:24.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b40ac790 con 0x7f64b415e4e0 2023-11-28T21:51:24.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.196+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b4157030 con 0x7f64b415e4e0 2023-11-28T21:51:24.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.197+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b40accb0 con 0x7f64b415e4e0 2023-11-28T21:51:24.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.198+0000 7f64a2ffd640 1 -- 172.21.15.45:0/710276012 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 86) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f64ac00abd0 con 0x7f64b415e4e0 2023-11-28T21:51:24.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.199+0000 7f64a2ffd640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f649406af60 0x7f649406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.199+0000 7f64a2ffd640 1 -- 172.21.15.45:0/710276012 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f64ac0890c0 con 0x7f64b415e4e0 2023-11-28T21:51:24.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.199+0000 7f64b259c640 1 -- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f649406af60 msgr2=0x7f649406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:24.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.199+0000 7f64b259c640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f649406af60 0x7f649406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:24.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.204+0000 7f64a2ffd640 1 -- 172.21.15.45:0/710276012 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64b40accb0 con 0x7f64b415e4e0 2023-11-28T21:51:24.208 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:24.207+0000 7f5686e37040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T21:51:24.289 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:24.286+0000 7f5308632040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T21:51:24.365 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:24.364+0000 7f5686e37040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T21:51:24.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.389+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f64b40accb0 con 0x7f64b415e4e0 2023-11-28T21:51:24.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.390+0000 7f64a2ffd640 1 -- 172.21.15.45:0/710276012 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v171) v1 ==== 100+0+11 (secure 0 0 0) 0x7f64b40accb0 con 0x7f64b415e4e0 2023-11-28T21:51:24.393 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f649406af60 msgr2=0x7f649406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f649406af60 0x7f649406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b415e4e0 msgr2=0x7f64b40cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b415e4e0 0x7f64b40cd930 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00c780 tx=0x7f64ac00cc40 comp rx=0 tx=0).stop 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 shutdown_connections 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64b415e4e0 0x7f64b40cd930 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f649406af60 0x7f649406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64b4157c10 0x7f64b4156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 --2- 172.21.15.45:0/710276012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64b406a500 0x7f64b41543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 >> 172.21.15.45:0/710276012 conn(0x7f64b40ced60 msgr2=0x7f64b40cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:24.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 shutdown_connections 2023-11-28T21:51:24.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.393+0000 7f64b8a2b640 1 -- 172.21.15.45:0/710276012 wait complete. 2023-11-28T21:51:24.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T21:51:24.460 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:24.458+0000 7f5308632040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T21:51:24.475 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:24.470+0000 7f5686e37040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T21:51:24.529 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:24.528+0000 7f5686e37040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T21:51:24.585 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:24.583+0000 7f5308632040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T21:51:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.623+0000 7fb64dc72640 1 Processor -- start 2023-11-28T21:51:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.623+0000 7fb64dc72640 1 -- start start 2023-11-28T21:51:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.624+0000 7fb64dc72640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815ae00 0x7fb64815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.625+0000 7fb64dc72640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb64815e900 0x7fb64815b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.625+0000 7fb64dc72640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64815f130 con 0x7fb64815b8c0 2023-11-28T21:51:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.625+0000 7fb64dc72640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64815b1e0 con 0x7fb64815e900 2023-11-28T21:51:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.625+0000 7fb64dc72640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64815f410 con 0x7fb64815ae00 2023-11-28T21:51:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.625+0000 7fb64cc70640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815ae00 0x7fb64815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.625+0000 7fb64cc70640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815ae00 0x7fb64815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:54796/0 (socket says 172.21.15.45:54796) 2023-11-28T21:51:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.625+0000 7fb64cc70640 1 -- 172.21.15.45:0/3935291367 learned_addr learned my addr 172.21.15.45:0/3935291367 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.625+0000 7fb647fff640 1 --2- 172.21.15.45:0/3935291367 >> v2:172.21.15.190:3300/0 conn(0x7fb64815e900 0x7fb64815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.626+0000 7fb64cc70640 1 -- 172.21.15.45:0/3935291367 >> v2:172.21.15.190:3300/0 conn(0x7fb64815e900 msgr2=0x7fb64815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.626+0000 7fb64cc70640 1 --2- 172.21.15.45:0/3935291367 >> v2:172.21.15.190:3300/0 conn(0x7fb64815e900 0x7fb64815b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.626+0000 7fb64cc70640 1 -- 172.21.15.45:0/3935291367 >> v1:172.21.15.190:6789/0 conn(0x7fb64815b8c0 legacy=0x7fb6481601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.626+0000 7fb64cc70640 1 -- 172.21.15.45:0/3935291367 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64815f6f0 con 0x7fb64815ae00 2023-11-28T21:51:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.627+0000 7fb64cc70640 1 --2- 172.21.15.45:0/3935291367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815ae00 0x7fb64815d6f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009870 tx=0x7fb63c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=509f0e14322fa713 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.627+0000 7fb6477fe640 1 -- 172.21.15.45:0/3935291367 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c030840 con 0x7fb64815ae00 2023-11-28T21:51:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb6477fe640 1 -- 172.21.15.45:0/3935291367 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63c0309a0 con 0x7fb64815ae00 2023-11-28T21:51:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb6477fe640 1 -- 172.21.15.45:0/3935291367 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c030d30 con 0x7fb64815ae00 2023-11-28T21:51:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb64dc72640 1 -- 172.21.15.45:0/3935291367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815ae00 msgr2=0x7fb64815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb64dc72640 1 --2- 172.21.15.45:0/3935291367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815ae00 0x7fb64815d6f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009870 tx=0x7fb63c02b980 comp rx=0 tx=0).stop 2023-11-28T21:51:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb64dc72640 1 -- 172.21.15.45:0/3935291367 shutdown_connections 2023-11-28T21:51:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb64dc72640 1 --2- 172.21.15.45:0/3935291367 >> v2:172.21.15.190:3300/0 conn(0x7fb64815e900 0x7fb64815b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb64dc72640 1 --2- 172.21.15.45:0/3935291367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815ae00 0x7fb64815d6f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb64dc72640 1 -- 172.21.15.45:0/3935291367 >> 172.21.15.45:0/3935291367 conn(0x7fb6480661f0 msgr2=0x7fb6480665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb64dc72640 1 -- 172.21.15.45:0/3935291367 shutdown_connections 2023-11-28T21:51:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.628+0000 7fb64dc72640 1 -- 172.21.15.45:0/3935291367 wait complete. 2023-11-28T21:51:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.630+0000 7fb64dc72640 1 Processor -- start 2023-11-28T21:51:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.630+0000 7fb64dc72640 1 -- start start 2023-11-28T21:51:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.631+0000 7fb64dc72640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb64815ae00 0x7fb6480b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.631+0000 7fb64dc72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb64815b8c0 0x7fb6480b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.631+0000 7fb64dc72640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815e900 0x7fb6481f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.631+0000 7fb64dc72640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6481f71a0 con 0x7fb64815b8c0 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.631+0000 7fb64dc72640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6480a85d0 con 0x7fb64815ae00 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.631+0000 7fb64dc72640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6480a88b0 con 0x7fb64815e900 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb647fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb64815b8c0 0x7fb6480b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb647fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb64815b8c0 0x7fb6480b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60866/0 (socket says 172.21.15.45:60866) 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb647fff640 1 -- 172.21.15.45:0/2584888701 learned_addr learned my addr 172.21.15.45:0/2584888701 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb64d471640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815e900 0x7fb6481f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb64cc70640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb64815ae00 0x7fb6480b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb647fff640 1 -- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815e900 msgr2=0x7fb6481f6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb647fff640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815e900 0x7fb6481f6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb647fff640 1 -- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb64815ae00 msgr2=0x7fb6480b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb647fff640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb64815ae00 0x7fb6480b2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb647fff640 1 -- 172.21.15.45:0/2584888701 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c0094d0 con 0x7fb64815b8c0 2023-11-28T21:51:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb64d471640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815e900 0x7fb6481f6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.632+0000 7fb647fff640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb64815b8c0 0x7fb6480b3360 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fb63400e870 tx=0x7fb63400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.633+0000 7fb6457fa640 1 -- 172.21.15.45:0/2584888701 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6340094f0 con 0x7fb64815b8c0 2023-11-28T21:51:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.633+0000 7fb6457fa640 1 -- 172.21.15.45:0/2584888701 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63400c940 con 0x7fb64815b8c0 2023-11-28T21:51:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.633+0000 7fb6457fa640 1 -- 172.21.15.45:0/2584888701 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63400ccf0 con 0x7fb64815b8c0 2023-11-28T21:51:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.633+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6480a8b90 con 0x7fb64815b8c0 2023-11-28T21:51:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.633+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6480a9390 con 0x7fb64815b8c0 2023-11-28T21:51:24.638 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:24.636+0000 7f5686e37040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:51:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.634+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb648067fb0 con 0x7fb64815b8c0 2023-11-28T21:51:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.636+0000 7fb6457fa640 1 -- 172.21.15.45:0/2584888701 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 86) v1 ==== 89345+0+0 (secure 0 0 0) 0x7fb634013470 con 0x7fb64815b8c0 2023-11-28T21:51:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.637+0000 7fb6457fa640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fb62c06af60 0x7fb62c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:24.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.637+0000 7fb6457fa640 1 -- 172.21.15.45:0/2584888701 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fb634089b40 con 0x7fb64815b8c0 2023-11-28T21:51:24.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.637+0000 7fb64cc70640 1 -- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fb62c06af60 msgr2=0x7fb62c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:24.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.637+0000 7fb64cc70640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fb62c06af60 0x7fb62c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:24.641 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:24.639+0000 7f5308632040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T21:51:24.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.642+0000 7fb6457fa640 1 -- 172.21.15.45:0/2584888701 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb634054860 con 0x7fb64815b8c0 2023-11-28T21:51:24.744 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:24.743+0000 7f5686e37040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T21:51:24.756 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:24.753+0000 7f5308632040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:51:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.838+0000 7fb64cc70640 1 -- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fb62c06af60 msgr2=0x7fb62c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.838+0000 7fb64cc70640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fb62c06af60 0x7fb62c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:51:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.844+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb648072380 con 0x7fb64815b8c0 2023-11-28T21:51:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.845+0000 7fb6457fa640 1 -- 172.21.15.45:0/2584888701 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v171) v1 ==== 80+0+474 (secure 0 0 0) 0x7fb634021080 con 0x7fb64815b8c0 2023-11-28T21:51:24.847 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/943491299 2023-11-29T20:55:29.635279+0000 2023-11-28T21:51:24.847 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1767202704 2023-11-29T20:55:29.635279+0000 2023-11-28T21:51:24.848 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2782217070 2023-11-29T20:55:29.635279+0000 2023-11-28T21:51:24.848 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/290777784 2023-11-29T20:55:29.635279+0000 2023-11-28T21:51:24.848 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/226305585 2023-11-29T20:55:29.635279+0000 2023-11-28T21:51:24.848 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/226305585 2023-11-29T20:55:29.635279+0000 2023-11-28T21:51:24.848 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1078115353 2023-11-29T20:55:29.635279+0000 2023-11-28T21:51:24.848 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2712902162 2023-11-29T20:55:29.635279+0000 2023-11-28T21:51:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2023-11-28T21:51:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.847+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fb62c06af60 msgr2=0x7fb62c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fb62c06af60 0x7fb62c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb64815b8c0 msgr2=0x7fb6480b3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb64815b8c0 0x7fb6480b3360 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fb63400e870 tx=0x7fb63400ed30 comp rx=0 tx=0).stop 2023-11-28T21:51:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 shutdown_connections 2023-11-28T21:51:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fb62c06af60 0x7fb62c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb64815e900 0x7fb6481f6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb64815b8c0 0x7fb6480b3360 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 --2- 172.21.15.45:0/2584888701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb64815ae00 0x7fb6480b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 >> 172.21.15.45:0/2584888701 conn(0x7fb6480661f0 msgr2=0x7fb64806d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 shutdown_connections 2023-11-28T21:51:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:24.848+0000 7fb64dc72640 1 -- 172.21.15.45:0/2584888701 wait complete. 2023-11-28T21:51:24.867 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T21:51:24.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:24.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T21:51:24.870 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:24.868+0000 7f5308632040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T21:51:24.902 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:24.900+0000 7f5686e37040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:51:24.950 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:24.949+0000 7f5686e37040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T21:51:24.978 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T21:51:24.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:24.978 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T21:51:25.028 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T21:51:25.028 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:25.050 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:25.047+0000 7f5308632040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:51:25.072 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:25.071+0000 7f5686e37040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T21:51:25.086 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T21:51:25.087 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:51:25.087 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:25.101 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:25.099+0000 7f5308632040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T21:51:25.144 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T21:51:25.144 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T21:51:25.144 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:25.197 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T21:51:25.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:51:25.232 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:25.229+0000 7f5308632040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T21:51:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.418+0000 7f87825a9640 1 Processor -- start 2023-11-28T21:51:25.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.418+0000 7f87825a9640 1 -- start start 2023-11-28T21:51:25.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.420+0000 7f87825a9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f877c0b1340 0x7f877c15f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.420+0000 7f87825a9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877c15bda0 con 0x7f877c0b35b0 2023-11-28T21:51:25.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.420+0000 7f87825a9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877c15c080 con 0x7f877c15b850 2023-11-28T21:51:25.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.420+0000 7f87825a9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f877c15ee40 con 0x7f877c0b1340 2023-11-28T21:51:25.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.420+0000 7f877bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f877c0b35b0 0x7f877c15ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33078/0 (socket says 172.21.15.45:33078) 2023-11-28T21:51:25.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.420+0000 7f877bfff640 1 -- 172.21.15.45:0/3366421623 learned_addr learned my addr 172.21.15.45:0/3366421623 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:25.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f8780b46640 1 --2- 172.21.15.45:0/3366421623 >> v2:172.21.15.190:3300/0 conn(0x7f877c0b1340 0x7f877c15f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677296372 0 0) 0x7f877c15bda0 con 0x7f877c0b35b0 2023-11-28T21:51:25.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8768003520 con 0x7f877c0b35b0 2023-11-28T21:51:25.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2113106252 0 0) 0x7f8768003520 con 0x7f877c0b35b0 2023-11-28T21:51:25.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 >> v2:172.21.15.190:3300/0 conn(0x7f877c0b1340 msgr2=0x7f877c15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:25.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 --2- 172.21.15.45:0/3366421623 >> v2:172.21.15.190:3300/0 conn(0x7f877c0b1340 0x7f877c15f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 >> v1:172.21.15.190:6789/0 conn(0x7f877c15b850 legacy=0x7f877c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c15c360 con 0x7f877c0b35b0 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f876c0019f0 con 0x7f877c0b35b0 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 >> v1:172.21.15.45:6789/0 conn(0x7f877c0b35b0 legacy=0x7f877c15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8768003970 0x7f8768003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87680047c0 0x7f8768004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8768005180 0x7f876800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c15ee40 con 0x7f8768005180 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876800c020 con 0x7f87680047c0 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.421+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876800c1e0 con 0x7f8768003970 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.422+0000 7f8780b46640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8768005180 0x7f876800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.422+0000 7f877bfff640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8768003970 0x7f8768003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.422+0000 7f8773fff640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87680047c0 0x7f8768004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.422+0000 7f8780b46640 1 -- 172.21.15.45:0/3366421623 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8768003970 msgr2=0x7f8768003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:25.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.422+0000 7f8780b46640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8768003970 0x7f8768003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.422+0000 7f8780b46640 1 -- 172.21.15.45:0/3366421623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87680047c0 msgr2=0x7f8768004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:25.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.422+0000 7f8780b46640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87680047c0 0x7f8768004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.422+0000 7f8780b46640 1 -- 172.21.15.45:0/3366421623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8768003520 con 0x7f8768005180 2023-11-28T21:51:25.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.422+0000 7f8773fff640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87680047c0 0x7f8768004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:25.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.424+0000 7f8780b46640 1 -- 172.21.15.45:0/3366421623 reap_dead start 2023-11-28T21:51:25.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.424+0000 7f8780b46640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8768005180 0x7f876800b980 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f8774009a90 tx=0x7f87680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3ff1d6e33f58f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:25.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.424+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877402c6b0 con 0x7f8768005180 2023-11-28T21:51:25.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.424+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877402c830 con 0x7f8768005180 2023-11-28T21:51:25.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.424+0000 7f877b7fe640 1 -- 172.21.15.45:0/3366421623 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877402cbe0 con 0x7f8768005180 2023-11-28T21:51:25.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.425+0000 7f87825a9640 1 -- 172.21.15.45:0/3366421623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8768005180 msgr2=0x7f876800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.425+0000 7f87825a9640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8768005180 0x7f876800b980 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f8774009a90 tx=0x7f87680057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.425+0000 7f87825a9640 1 -- 172.21.15.45:0/3366421623 shutdown_connections 2023-11-28T21:51:25.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.425+0000 7f87825a9640 1 --2- 172.21.15.45:0/3366421623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8768005180 0x7f876800b980 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.425+0000 7f87825a9640 1 -- 172.21.15.45:0/3366421623 >> 172.21.15.45:0/3366421623 conn(0x7f877c06a500 msgr2=0x7f877c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:25.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.425+0000 7f87825a9640 1 -- 172.21.15.45:0/3366421623 shutdown_connections 2023-11-28T21:51:25.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.425+0000 7f87825a9640 1 -- 172.21.15.45:0/3366421623 wait complete. 2023-11-28T21:51:25.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.427+0000 7f87825a9640 1 Processor -- start 2023-11-28T21:51:25.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.427+0000 7f87825a9640 1 -- start start 2023-11-28T21:51:25.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f87825a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8768005180 0x7f877c1e5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f87825a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f877c142950 0x7f877c0a9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f87825a9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f877c141e50 0x7f877c143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f87825a9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c1e5750 con 0x7f877c142950 2023-11-28T21:51:25.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f87825a9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c0aadb0 con 0x7f8768005180 2023-11-28T21:51:25.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f87825a9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877c1e5120 con 0x7f877c141e50 2023-11-28T21:51:25.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f8773fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f877c142950 0x7f877c0a9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f8773fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f877c142950 0x7f877c0a9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60884/0 (socket says 172.21.15.45:60884) 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f8773fff640 1 -- 172.21.15.45:0/3917924395 learned_addr learned my addr 172.21.15.45:0/3917924395 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.428+0000 7f8780b46640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f877c141e50 0x7f877c143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f877bfff640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8768005180 0x7f877c1e5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f8773fff640 1 -- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f877c141e50 msgr2=0x7f877c143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f8773fff640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f877c141e50 0x7f877c143560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f8773fff640 1 -- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8768005180 msgr2=0x7f877c1e5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f8773fff640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8768005180 0x7f877c1e5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f8773fff640 1 -- 172.21.15.45:0/3917924395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c0aa7f0 con 0x7f877c142950 2023-11-28T21:51:25.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f8773fff640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f877c142950 0x7f877c0a9ea0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f876400e4b0 tx=0x7f876400e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:25.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f87797fa640 1 -- 172.21.15.45:0/3917924395 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640052e0 con 0x7f877c142950 2023-11-28T21:51:25.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f87797fa640 1 -- 172.21.15.45:0/3917924395 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8764005440 con 0x7f877c142950 2023-11-28T21:51:25.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f87797fa640 1 -- 172.21.15.45:0/3917924395 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764003830 con 0x7f877c142950 2023-11-28T21:51:25.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c0aaad0 con 0x7f877c142950 2023-11-28T21:51:25.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.429+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c0aa3e0 con 0x7f877c142950 2023-11-28T21:51:25.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.430+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c0aa570 con 0x7f877c142950 2023-11-28T21:51:25.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.431+0000 7f87797fa640 1 -- 172.21.15.45:0/3917924395 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 86) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f8764003990 con 0x7f877c142950 2023-11-28T21:51:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.432+0000 7f87797fa640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f875c06af60 0x7f875c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.432+0000 7f87797fa640 1 -- 172.21.15.45:0/3917924395 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f8764089800 con 0x7f877c142950 2023-11-28T21:51:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.433+0000 7f877bfff640 1 -- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f875c06af60 msgr2=0x7f875c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.433+0000 7f877bfff640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f875c06af60 0x7f875c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.437+0000 7f87797fa640 1 -- 172.21.15.45:0/3917924395 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8764054550 con 0x7f877c142950 2023-11-28T21:51:25.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.634+0000 7f877bfff640 1 -- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f875c06af60 msgr2=0x7f875c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:25.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.634+0000 7f877bfff640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f875c06af60 0x7f875c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:51:25.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.668+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f877c0733d0 con 0x7f877c142950 2023-11-28T21:51:25.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.669+0000 7f87797fa640 1 -- 172.21.15.45:0/3917924395 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f8764057b70 con 0x7f877c142950 2023-11-28T21:51:25.671 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:25.672 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7531,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1320359028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1320359028},{"type":"v1","addr":"172.21.15.45:6833","nonce":1320359028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/808323687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":808323687},{"type":"v1","addr":"172.21.15.190:6836","nonce":808323687}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3406194668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3406194668},{"type":"v1","addr":"172.21.15.190:6837","nonce":3406194668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:55:12.738156+0000","modified":"2023-11-28T20:55:15.949737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"c","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/637440036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":637440036},{"type":"v1","addr":"172.21.15.45:6835","nonce":637440036}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-28T21:51:25.672 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 76 2023-11-28T21:51:25.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.672+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f875c06af60 msgr2=0x7f875c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:25.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.672+0000 7f87825a9640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f875c06af60 0x7f875c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.672+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f877c142950 msgr2=0x7f877c0a9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.672+0000 7f87825a9640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f877c142950 0x7f877c0a9ea0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f876400e4b0 tx=0x7f876400e970 comp rx=0 tx=0).stop 2023-11-28T21:51:25.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.673+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 shutdown_connections 2023-11-28T21:51:25.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.673+0000 7f87825a9640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f875c06af60 0x7f875c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.673+0000 7f87825a9640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f877c141e50 0x7f877c143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.673+0000 7f87825a9640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f877c142950 0x7f877c0a9ea0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.673+0000 7f87825a9640 1 --2- 172.21.15.45:0/3917924395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8768005180 0x7f877c1e5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.673+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 >> 172.21.15.45:0/3917924395 conn(0x7f877c06a500 msgr2=0x7f877c144720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:25.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.673+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 shutdown_connections 2023-11-28T21:51:25.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.673+0000 7f87825a9640 1 -- 172.21.15.45:0/3917924395 wait complete. 2023-11-28T21:51:25.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:51:25.908 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:25.907+0000 7f5686e37040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T21:51:25.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.923+0000 7f58673a0640 1 Processor -- start 2023-11-28T21:51:25.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.923+0000 7f58673a0640 1 -- start start 2023-11-28T21:51:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.925+0000 7f58673a0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58600b7f60 0x7f58601681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.925+0000 7f58673a0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58600d2230 con 0x7f58600d02d0 2023-11-28T21:51:25.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.925+0000 7f58673a0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58600d2510 con 0x7f586015d550 2023-11-28T21:51:25.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.925+0000 7f58673a0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58600d27f0 con 0x7f58600b7f60 2023-11-28T21:51:25.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.925+0000 7f5865b9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f58600d02d0 0x7f58600d1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33094/0 (socket says 172.21.15.45:33094) 2023-11-28T21:51:25.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.925+0000 7f5865b9d640 1 -- 172.21.15.45:0/2383536899 learned_addr learned my addr 172.21.15.45:0/2383536899 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:25.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.926+0000 7f5866b9f640 1 --2- 172.21.15.45:0/2383536899 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58600b7f60 0x7f58601681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.926+0000 7f586539c640 1 -- 172.21.15.45:0/2383536899 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719705236 0 0) 0x7f58600d2230 con 0x7f58600d02d0 2023-11-28T21:51:25.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.926+0000 7f586539c640 1 -- 172.21.15.45:0/2383536899 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f584c003520 con 0x7f58600d02d0 2023-11-28T21:51:25.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.926+0000 7f5866b9f640 1 -- 172.21.15.45:0/2383536899 >> v1:172.21.15.190:6789/0 conn(0x7f586015d550 legacy=0x7f58600cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.926+0000 7f5866b9f640 1 -- 172.21.15.45:0/2383536899 >> v1:172.21.15.45:6789/0 conn(0x7f58600d02d0 legacy=0x7f58600d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.926+0000 7f5866b9f640 1 -- 172.21.15.45:0/2383536899 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58600b6a30 con 0x7f58600b7f60 2023-11-28T21:51:25.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.927+0000 7f5866b9f640 1 --2- 172.21.15.45:0/2383536899 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58600b7f60 0x7f58601681d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5858009dc0 tx=0x7f585802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=714a02af1ec61009 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:25.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f586539c640 1 -- 172.21.15.45:0/2383536899 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585802ccd0 con 0x7f58600b7f60 2023-11-28T21:51:25.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f586539c640 1 -- 172.21.15.45:0/2383536899 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585802ce30 con 0x7f58600b7f60 2023-11-28T21:51:25.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f586539c640 1 -- 172.21.15.45:0/2383536899 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858034640 con 0x7f58600b7f60 2023-11-28T21:51:25.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f58673a0640 1 -- 172.21.15.45:0/2383536899 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58600b7f60 msgr2=0x7f58601681d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f58673a0640 1 --2- 172.21.15.45:0/2383536899 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58600b7f60 0x7f58601681d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5858009dc0 tx=0x7f585802bf20 comp rx=0 tx=0).stop 2023-11-28T21:51:25.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f58673a0640 1 -- 172.21.15.45:0/2383536899 shutdown_connections 2023-11-28T21:51:25.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f58673a0640 1 --2- 172.21.15.45:0/2383536899 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58600b7f60 0x7f58601681d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f58673a0640 1 -- 172.21.15.45:0/2383536899 >> 172.21.15.45:0/2383536899 conn(0x7f586006ac70 msgr2=0x7f58600b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:25.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f58673a0640 1 -- 172.21.15.45:0/2383536899 shutdown_connections 2023-11-28T21:51:25.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.928+0000 7f58673a0640 1 -- 172.21.15.45:0/2383536899 wait complete. 2023-11-28T21:51:25.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.930+0000 7f58673a0640 1 Processor -- start 2023-11-28T21:51:25.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.930+0000 7f58673a0640 1 -- start start 2023-11-28T21:51:25.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.931+0000 7f58673a0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58600b7f60 0x7f58601f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.931+0000 7f58673a0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600d02d0 0x7f5860159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.931+0000 7f58673a0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f586015d550 0x7f58601e6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.931+0000 7f58673a0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58601e7490 con 0x7f58600d02d0 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.931+0000 7f58673a0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58601e71b0 con 0x7f58600b7f60 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.931+0000 7f58673a0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58600c83c0 con 0x7f586015d550 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.931+0000 7f5865b9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600d02d0 0x7f5860159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f5865b9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600d02d0 0x7f5860159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60896/0 (socket says 172.21.15.45:60896) 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f5865b9d640 1 -- 172.21.15.45:0/426416260 learned_addr learned my addr 172.21.15.45:0/426416260 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f5866b9f640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f586015d550 0x7f58601e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f586639e640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58600b7f60 0x7f58601f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f5865b9d640 1 -- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f586015d550 msgr2=0x7f58601e6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f5865b9d640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f586015d550 0x7f58601e6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f5865b9d640 1 -- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58600b7f60 msgr2=0x7f58601f35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:25.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f5865b9d640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58600b7f60 0x7f58601f35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:25.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f5865b9d640 1 -- 172.21.15.45:0/426416260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5858009a70 con 0x7f58600d02d0 2023-11-28T21:51:25.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f586639e640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58600b7f60 0x7f58601f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:25.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.932+0000 7f5865b9d640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600d02d0 0x7f5860159050 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f585000c500 tx=0x7f585000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:25.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.933+0000 7f5856ffd640 1 -- 172.21.15.45:0/426416260 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5850005ce0 con 0x7f58600d02d0 2023-11-28T21:51:25.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.933+0000 7f5856ffd640 1 -- 172.21.15.45:0/426416260 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5850005e40 con 0x7f58600d02d0 2023-11-28T21:51:25.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.933+0000 7f5856ffd640 1 -- 172.21.15.45:0/426416260 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5850007330 con 0x7f58600d02d0 2023-11-28T21:51:25.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.933+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58600c86a0 con 0x7f58600d02d0 2023-11-28T21:51:25.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.933+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5860159590 con 0x7f58600d02d0 2023-11-28T21:51:25.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.934+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58600c8220 con 0x7f58600d02d0 2023-11-28T21:51:25.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.935+0000 7f5856ffd640 1 -- 172.21.15.45:0/426416260 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 86) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f5850007490 con 0x7f58600d02d0 2023-11-28T21:51:25.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.936+0000 7f5856ffd640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f584406af90 0x7f584406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:25.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.936+0000 7f5856ffd640 1 -- 172.21.15.45:0/426416260 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f58500890a0 con 0x7f58600d02d0 2023-11-28T21:51:25.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.936+0000 7f586639e640 1 -- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f584406af90 msgr2=0x7f584406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:25.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.936+0000 7f586639e640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f584406af90 0x7f584406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:25.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:25.940+0000 7f5856ffd640 1 -- 172.21.15.45:0/426416260 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58600c8220 con 0x7f58600d02d0 2023-11-28T21:51:26.077 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:26.074+0000 7f5308632040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T21:51:26.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.137+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58600c8220 con 0x7f58600d02d0 2023-11-28T21:51:26.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.137+0000 7f586639e640 1 -- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f584406af90 msgr2=0x7f584406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:26.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.137+0000 7f586639e640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f584406af90 0x7f584406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:51:26.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.139+0000 7f5856ffd640 1 -- 172.21.15.45:0/426416260 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+17435 (secure 0 0 0) 0x7f58600c8220 con 0x7f58600d02d0 2023-11-28T21:51:26.142 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:26.142 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":171,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:51:24.096478+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:55:10.712027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-11-28T20:55:11.779380+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":171,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/943491299":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1767202704":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2782217070":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/290777784":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6833/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6832/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1078115353":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2712902162":"2023-11-29T20:55:29.635279+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f584406af90 msgr2=0x7f584406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f584406af90 0x7f584406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600d02d0 msgr2=0x7f5860159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600d02d0 0x7f5860159050 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f585000c500 tx=0x7f585000c9c0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 shutdown_connections 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7f584406af90 0x7f584406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f586015d550 0x7f58601e6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58600d02d0 0x7f5860159050 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 --2- 172.21.15.45:0/426416260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58600b7f60 0x7f58601f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 >> 172.21.15.45:0/426416260 conn(0x7f586006ac70 msgr2=0x7f586015dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:26.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 shutdown_connections 2023-11-28T21:51:26.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.141+0000 7f58673a0640 1 -- 172.21.15.45:0/426416260 wait complete. 2023-11-28T21:51:26.161 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T20:55:10.712027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T20:55:11.779380+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 171, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T21:51:26.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T21:51:26.203 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.202+0000 7f5686e37040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T21:51:26.260 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.259+0000 7f5686e37040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T21:51:26.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.352+0000 7fd168bf4640 1 Processor -- start 2023-11-28T21:51:26.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.353+0000 7fd168bf4640 1 -- start start 2023-11-28T21:51:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.354+0000 7fd168bf4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640b1d40 0x7fd1640d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.354+0000 7fd168bf4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1640b90a0 con 0x7fd1640d0040 2023-11-28T21:51:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.354+0000 7fd168bf4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1640d1920 con 0x7fd1640b2290 2023-11-28T21:51:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.354+0000 7fd168bf4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640b8ae0 con 0x7fd1640b1d40 2023-11-28T21:51:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.354+0000 7fd1637fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd1640d0040 0x7fd1640d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33102/0 (socket says 172.21.15.45:33102) 2023-11-28T21:51:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.355+0000 7fd1637fe640 1 -- 172.21.15.45:0/132944307 learned_addr learned my addr 172.21.15.45:0/132944307 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:26.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.355+0000 7fd163fff640 1 --2- 172.21.15.45:0/132944307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640b1d40 0x7fd1640d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:26.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.355+0000 7fd162ffd640 1 -- 172.21.15.45:0/132944307 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897969151 0 0) 0x7fd1640b90a0 con 0x7fd1640d0040 2023-11-28T21:51:26.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.355+0000 7fd162ffd640 1 -- 172.21.15.45:0/132944307 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd150003520 con 0x7fd1640d0040 2023-11-28T21:51:26.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.355+0000 7fd163fff640 1 -- 172.21.15.45:0/132944307 >> v1:172.21.15.190:6789/0 conn(0x7fd1640b2290 legacy=0x7fd1640d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.355+0000 7fd163fff640 1 -- 172.21.15.45:0/132944307 >> v1:172.21.15.45:6789/0 conn(0x7fd1640d0040 legacy=0x7fd1640d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.355+0000 7fd163fff640 1 -- 172.21.15.45:0/132944307 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640b8dc0 con 0x7fd1640b1d40 2023-11-28T21:51:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.356+0000 7fd163fff640 1 --2- 172.21.15.45:0/132944307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640b1d40 0x7fd1640d24c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009dc0 tx=0x7fd15c02bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4d4c576069032f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.357+0000 7fd162ffd640 1 -- 172.21.15.45:0/132944307 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c02cd60 con 0x7fd1640b1d40 2023-11-28T21:51:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.357+0000 7fd162ffd640 1 -- 172.21.15.45:0/132944307 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c034440 con 0x7fd1640b1d40 2023-11-28T21:51:26.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.357+0000 7fd162ffd640 1 -- 172.21.15.45:0/132944307 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c0347b0 con 0x7fd1640b1d40 2023-11-28T21:51:26.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.357+0000 7fd168bf4640 1 -- 172.21.15.45:0/132944307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640b1d40 msgr2=0x7fd1640d24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.357+0000 7fd168bf4640 1 --2- 172.21.15.45:0/132944307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640b1d40 0x7fd1640d24c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009dc0 tx=0x7fd15c02bfb0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.358+0000 7fd168bf4640 1 -- 172.21.15.45:0/132944307 shutdown_connections 2023-11-28T21:51:26.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.358+0000 7fd168bf4640 1 --2- 172.21.15.45:0/132944307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640b1d40 0x7fd1640d24c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.358+0000 7fd168bf4640 1 -- 172.21.15.45:0/132944307 >> 172.21.15.45:0/132944307 conn(0x7fd164070f40 msgr2=0x7fd164071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:26.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.358+0000 7fd168bf4640 1 -- 172.21.15.45:0/132944307 shutdown_connections 2023-11-28T21:51:26.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.358+0000 7fd168bf4640 1 -- 172.21.15.45:0/132944307 wait complete. 2023-11-28T21:51:26.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.359+0000 7fd168bf4640 1 Processor -- start 2023-11-28T21:51:26.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.359+0000 7fd168bf4640 1 -- start start 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.360+0000 7fd168bf4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1640b1d40 0x7fd1640c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.360+0000 7fd168bf4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1640b2290 0x7fd1640c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.360+0000 7fd168bf4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640d0040 0x7fd1641f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.360+0000 7fd168bf4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640c83f0 con 0x7fd1640b2290 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.360+0000 7fd168bf4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640ca340 con 0x7fd1640b1d40 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.360+0000 7fd168bf4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640c7b50 con 0x7fd1640d0040 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd15bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1640b2290 0x7fd1640c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd15bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1640b2290 0x7fd1640c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60910/0 (socket says 172.21.15.45:60910) 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd15bfff640 1 -- 172.21.15.45:0/3035510595 learned_addr learned my addr 172.21.15.45:0/3035510595 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd163fff640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640d0040 0x7fd1641f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:26.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd1637fe640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1640b1d40 0x7fd1640c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd15bfff640 1 -- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640d0040 msgr2=0x7fd1641f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd15bfff640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640d0040 0x7fd1641f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd15bfff640 1 -- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1640b1d40 msgr2=0x7fd1640c5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd15bfff640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1640b1d40 0x7fd1640c5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd15bfff640 1 -- 172.21.15.45:0/3035510595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c009a70 con 0x7fd1640b2290 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.361+0000 7fd15bfff640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1640b2290 0x7fd1640c93a0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00c480 tx=0x7fd14c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.362+0000 7fd160ff9640 1 -- 172.21.15.45:0/3035510595 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0052d0 con 0x7fd1640b2290 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.362+0000 7fd160ff9640 1 -- 172.21.15.45:0/3035510595 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c005430 con 0x7fd1640b2290 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.362+0000 7fd168bf4640 1 -- 172.21.15.45:0/3035510595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1640c7e30 con 0x7fd1640b2290 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.362+0000 7fd160ff9640 1 -- 172.21.15.45:0/3035510595 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c012870 con 0x7fd1640b2290 2023-11-28T21:51:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.362+0000 7fd168bf4640 1 -- 172.21.15.45:0/3035510595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1640c98e0 con 0x7fd1640b2290 2023-11-28T21:51:26.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.363+0000 7fd168bf4640 1 -- 172.21.15.45:0/3035510595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd164071b30 con 0x7fd1640b2290 2023-11-28T21:51:26.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.364+0000 7fd160ff9640 1 -- 172.21.15.45:0/3035510595 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 86) v1 ==== 89345+0+0 (secure 0 0 0) 0x7fd14c0129d0 con 0x7fd1640b2290 2023-11-28T21:51:26.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.365+0000 7fd160ff9640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fd14406af60 0x7fd14406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.365+0000 7fd160ff9640 1 -- 172.21.15.45:0/3035510595 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fd14c089520 con 0x7fd1640b2290 2023-11-28T21:51:26.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.365+0000 7fd1637fe640 1 -- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fd14406af60 msgr2=0x7fd14406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/560718832 2023-11-28T21:51:26.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.365+0000 7fd1637fe640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fd14406af60 0x7fd14406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.370+0000 7fd160ff9640 1 -- 172.21.15.45:0/3035510595 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd14c054240 con 0x7fd1640b2290 2023-11-28T21:51:26.374 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.373+0000 7f5686e37040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T21:51:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.387+0000 7fd160ff9640 1 -- 172.21.15.45:0/3035510595 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 87) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fd14c054610 con 0x7fd1640b2290 2023-11-28T21:51:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.387+0000 7fd160ff9640 1 -- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fd14406af60 msgr2=0x7fd14406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.387+0000 7fd160ff9640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fd14406af60 0x7fd14406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.409 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:26.407+0000 7f5308632040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T21:51:26.462 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f5607930640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.463 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f5607930640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.463 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f5607930640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.463 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f5607930640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.463 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f5607930640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.463 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f560a936640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.463 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f560a936640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.463 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f560a936640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.463 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f560a936640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.463 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:26.461+0000 7f560a936640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:51:26.476 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:26.474+0000 7f5308632040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T21:51:26.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.569+0000 7fd168bf4640 1 -- 172.21.15.45:0/3035510595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1640ca480 con 0x7fd1640b2290 2023-11-28T21:51:26.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.570+0000 7fd160ff9640 1 -- 172.21.15.45:0/3035510595 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v172) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd14c071800 con 0x7fd1640b2290 2023-11-28T21:51:26.572 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T21:51:26.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.572+0000 7fd168bf4640 1 -- 172.21.15.45:0/3035510595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1640b2290 msgr2=0x7fd1640c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.572+0000 7fd168bf4640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1640b2290 0x7fd1640c93a0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00c480 tx=0x7fd14c00c940 comp rx=0 tx=0).stop 2023-11-28T21:51:26.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.573+0000 7fd168bf4640 1 -- 172.21.15.45:0/3035510595 shutdown_connections 2023-11-28T21:51:26.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.573+0000 7fd168bf4640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:6832/560718832,v1:172.21.15.190:6833/560718832] conn(0x7fd14406af60 0x7fd14406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.573+0000 7fd168bf4640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1640d0040 0x7fd1641f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.573+0000 7fd168bf4640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1640b2290 0x7fd1640c93a0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.573+0000 7fd168bf4640 1 --2- 172.21.15.45:0/3035510595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1640b1d40 0x7fd1640c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.573+0000 7fd168bf4640 1 -- 172.21.15.45:0/3035510595 >> 172.21.15.45:0/3035510595 conn(0x7fd164070f40 msgr2=0x7fd1640d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:26.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.573+0000 7fd168bf4640 1 -- 172.21.15.45:0/3035510595 shutdown_connections 2023-11-28T21:51:26.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.573+0000 7fd168bf4640 1 -- 172.21.15.45:0/3035510595 wait complete. 2023-11-28T21:51:26.591 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T21:51:26.597 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:26.594+0000 7f5308632040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T21:51:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.805+0000 7f768ab9e640 1 Processor -- start 2023-11-28T21:51:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.805+0000 7f768ab9e640 1 -- start start 2023-11-28T21:51:26.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.806+0000 7f768ab9e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f76840d0040 0x7f76840d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.806+0000 7f768ab9e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840b2290 0x7f76840d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.806+0000 7f768ab9e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76840b90a0 con 0x7f76840b1d40 2023-11-28T21:51:26.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.806+0000 7f768ab9e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76840d1920 con 0x7f76840d0040 2023-11-28T21:51:26.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.806+0000 7f768ab9e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76840d2dc0 con 0x7f76840b2290 2023-11-28T21:51:26.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.807+0000 7f7689b9c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f76840d0040 0x7f76840d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:26.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.807+0000 7f7689b9c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f76840d0040 0x7f76840d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:55486/0 (socket says 172.21.15.45:55486) 2023-11-28T21:51:26.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.807+0000 7f7689b9c640 1 -- 172.21.15.45:0/35971874 learned_addr learned my addr 172.21.15.45:0/35971874 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:26.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.807+0000 7f768939b640 1 --2- 172.21.15.45:0/35971874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840b2290 0x7f76840d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:26.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.808+0000 7f768939b640 1 -- 172.21.15.45:0/35971874 >> v2:172.21.15.190:3300/0 conn(0x7f76840d0040 msgr2=0x7f76840d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:26.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.808+0000 7f768939b640 1 --2- 172.21.15.45:0/35971874 >> v2:172.21.15.190:3300/0 conn(0x7f76840d0040 0x7f76840d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.808+0000 7f768939b640 1 -- 172.21.15.45:0/35971874 >> v1:172.21.15.190:6789/0 conn(0x7f76840b1d40 legacy=0x7f76840d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.808+0000 7f768939b640 1 -- 172.21.15.45:0/35971874 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76840d0960 con 0x7f76840b2290 2023-11-28T21:51:26.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.808+0000 7f7689b9c640 1 --2- 172.21.15.45:0/35971874 >> v2:172.21.15.190:3300/0 conn(0x7f76840d0040 0x7f76840d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T21:51:26.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.810+0000 7f768939b640 1 --2- 172.21.15.45:0/35971874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840b2290 0x7f76840d3040 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f76780097a0 tx=0x7f767802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e467dbed2b9b8a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:26.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.810+0000 7f7688b9a640 1 -- 172.21.15.45:0/35971874 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678030800 con 0x7f76840b2290 2023-11-28T21:51:26.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f7688b9a640 1 -- 172.21.15.45:0/35971874 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7678030960 con 0x7f76840b2290 2023-11-28T21:51:26.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f7688b9a640 1 -- 172.21.15.45:0/35971874 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678030cf0 con 0x7f76840b2290 2023-11-28T21:51:26.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f768ab9e640 1 -- 172.21.15.45:0/35971874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840b2290 msgr2=0x7f76840d3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f768ab9e640 1 --2- 172.21.15.45:0/35971874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840b2290 0x7f76840d3040 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f76780097a0 tx=0x7f767802b940 comp rx=0 tx=0).stop 2023-11-28T21:51:26.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f768ab9e640 1 -- 172.21.15.45:0/35971874 shutdown_connections 2023-11-28T21:51:26.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f768ab9e640 1 --2- 172.21.15.45:0/35971874 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840b2290 0x7f76840d3040 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f768ab9e640 1 --2- 172.21.15.45:0/35971874 >> v2:172.21.15.190:3300/0 conn(0x7f76840d0040 0x7f76840d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f768ab9e640 1 -- 172.21.15.45:0/35971874 >> 172.21.15.45:0/35971874 conn(0x7f7684070f40 msgr2=0x7f7684071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:26.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f768ab9e640 1 -- 172.21.15.45:0/35971874 shutdown_connections 2023-11-28T21:51:26.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.811+0000 7f768ab9e640 1 -- 172.21.15.45:0/35971874 wait complete. 2023-11-28T21:51:26.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.813+0000 7f768ab9e640 1 Processor -- start 2023-11-28T21:51:26.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.813+0000 7f768ab9e640 1 -- start start 2023-11-28T21:51:26.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.814+0000 7f768ab9e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76840b1d40 0x7f76840c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.814+0000 7f768ab9e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840d0040 0x7f76840c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.814+0000 7f768ab9e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7684154080 0x7f76840cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:26.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.814+0000 7f768ab9e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76840c9680 con 0x7f7684154080 2023-11-28T21:51:26.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.814+0000 7f768ab9e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76840c77a0 con 0x7f76840b1d40 2023-11-28T21:51:26.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.814+0000 7f768ab9e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76840c7a80 con 0x7f76840d0040 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.815+0000 7f7689b9c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76840b1d40 0x7f76840c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.815+0000 7f768939b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840d0040 0x7f76840c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.815+0000 7f768a39d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7684154080 0x7f76840cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.815+0000 7f7689b9c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76840b1d40 0x7f76840c7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:55488/0 (socket says 172.21.15.45:55488) 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.815+0000 7f7689b9c640 1 -- 172.21.15.45:0/3577068949 learned_addr learned my addr 172.21.15.45:0/3577068949 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f768a39d640 1 -- 172.21.15.45:0/3577068949 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840d0040 msgr2=0x7f76840c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f768a39d640 1 --2- 172.21.15.45:0/3577068949 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840d0040 0x7f76840c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f768a39d640 1 -- 172.21.15.45:0/3577068949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76840b1d40 msgr2=0x7f76840c7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f768a39d640 1 --2- 172.21.15.45:0/3577068949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76840b1d40 0x7f76840c7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f768a39d640 1 -- 172.21.15.45:0/3577068949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7674009b30 con 0x7f7684154080 2023-11-28T21:51:26.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f768a39d640 1 --2- 172.21.15.45:0/3577068949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7684154080 0x7f76840cc470 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f767c00c390 tx=0x7f767c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:26.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f76827fc640 1 -- 172.21.15.45:0/3577068949 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c0052d0 con 0x7f7684154080 2023-11-28T21:51:26.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f76827fc640 1 -- 172.21.15.45:0/3577068949 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c005430 con 0x7f7684154080 2023-11-28T21:51:26.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f768ab9e640 1 -- 172.21.15.45:0/3577068949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7678009400 con 0x7f7684154080 2023-11-28T21:51:26.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f76827fc640 1 -- 172.21.15.45:0/3577068949 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c003830 con 0x7f7684154080 2023-11-28T21:51:26.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.816+0000 7f768ab9e640 1 -- 172.21.15.45:0/3577068949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76840c8040 con 0x7f7684154080 2023-11-28T21:51:26.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.817+0000 7f768ab9e640 1 -- 172.21.15.45:0/3577068949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7684071b30 con 0x7f7684154080 2023-11-28T21:51:26.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.818+0000 7f76827fc640 1 -- 172.21.15.45:0/3577068949 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 87) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f767c003990 con 0x7f7684154080 2023-11-28T21:51:26.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.819+0000 7f76827fc640 1 -- 172.21.15.45:0/3577068949 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f767c088960 con 0x7f7684154080 2023-11-28T21:51:26.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:26.824+0000 7f76827fc640 1 -- 172.21.15.45:0/3577068949 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f767c053aa0 con 0x7f7684154080 2023-11-28T21:51:27.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.017+0000 7f768ab9e640 1 -- 172.21.15.45:0/3577068949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f768404e920 con 0x7f7684154080 2023-11-28T21:51:27.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.018+0000 7f76827fc640 1 -- 172.21.15.45:0/3577068949 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v172) v1 ==== 104+0+11 (secure 0 0 0) 0x7f767c0570c0 con 0x7f7684154080 2023-11-28T21:51:27.020 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T21:51:27.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.021+0000 7f768ab9e640 1 -- 172.21.15.45:0/3577068949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7684154080 msgr2=0x7f76840cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.021+0000 7f768ab9e640 1 --2- 172.21.15.45:0/3577068949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7684154080 0x7f76840cc470 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f767c00c390 tx=0x7f767c00c850 comp rx=0 tx=0).stop 2023-11-28T21:51:27.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.021+0000 7f768ab9e640 1 -- 172.21.15.45:0/3577068949 shutdown_connections 2023-11-28T21:51:27.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.021+0000 7f768ab9e640 1 --2- 172.21.15.45:0/3577068949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7684154080 0x7f76840cc470 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.021+0000 7f768ab9e640 1 --2- 172.21.15.45:0/3577068949 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76840d0040 0x7f76840c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.021+0000 7f768ab9e640 1 --2- 172.21.15.45:0/3577068949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76840b1d40 0x7f76840c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.021+0000 7f768ab9e640 1 -- 172.21.15.45:0/3577068949 >> 172.21.15.45:0/3577068949 conn(0x7f7684070f40 msgr2=0x7f76840b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:27.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.021+0000 7f768ab9e640 1 -- 172.21.15.45:0/3577068949 shutdown_connections 2023-11-28T21:51:27.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.021+0000 7f768ab9e640 1 -- 172.21.15.45:0/3577068949 wait complete. 2023-11-28T21:51:27.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T21:51:27.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.270+0000 7fad63fff640 1 Processor -- start 2023-11-28T21:51:27.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.270+0000 7fad63fff640 1 -- start start 2023-11-28T21:51:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.272+0000 7fad63fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad64159720 0x7fad640b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.272+0000 7fad63fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad6406dfa0 0x7fad640701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.272+0000 7fad63fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad640cc120 con 0x7fad64071b90 2023-11-28T21:51:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.272+0000 7fad63fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad640b2420 con 0x7fad6406dfa0 2023-11-28T21:51:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.272+0000 7fad63fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6406e400 con 0x7fad64159720 2023-11-28T21:51:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.272+0000 7fad5a7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad6406dfa0 0x7fad640701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.273+0000 7fad5a7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fad6406dfa0 0x7fad640701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60934/0 (socket says 172.21.15.45:60934) 2023-11-28T21:51:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.273+0000 7fad5a7fc640 1 -- 172.21.15.45:0/3217453847 learned_addr learned my addr 172.21.15.45:0/3217453847 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:27.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.273+0000 7fad62ffd640 1 --2- 172.21.15.45:0/3217453847 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad64159720 0x7fad640b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.273+0000 7fad5a7fc640 1 -- 172.21.15.45:0/3217453847 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad64159720 msgr2=0x7fad640b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:27.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.273+0000 7fad5a7fc640 1 --2- 172.21.15.45:0/3217453847 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad64159720 0x7fad640b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.273+0000 7fad5a7fc640 1 -- 172.21.15.45:0/3217453847 >> v1:172.21.15.190:6789/0 conn(0x7fad64071b90 legacy=0x7fad6415d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:27.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.273+0000 7fad5a7fc640 1 -- 172.21.15.45:0/3217453847 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad640cb880 con 0x7fad6406dfa0 2023-11-28T21:51:27.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.274+0000 7fad5a7fc640 1 --2- 172.21.15.45:0/3217453847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 0x7fad640701e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009870 tx=0x7fad4c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2169e908dadd5b87 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:27.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.274+0000 7fad627fc640 1 -- 172.21.15.45:0/3217453847 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c031640 con 0x7fad6406dfa0 2023-11-28T21:51:27.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.275+0000 7fad627fc640 1 -- 172.21.15.45:0/3217453847 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c0317a0 con 0x7fad6406dfa0 2023-11-28T21:51:27.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.275+0000 7fad627fc640 1 -- 172.21.15.45:0/3217453847 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c031b30 con 0x7fad6406dfa0 2023-11-28T21:51:27.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.275+0000 7fad63fff640 1 -- 172.21.15.45:0/3217453847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 msgr2=0x7fad640701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.275+0000 7fad63fff640 1 --2- 172.21.15.45:0/3217453847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 0x7fad640701e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009870 tx=0x7fad4c02b8f0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.276+0000 7fad63fff640 1 -- 172.21.15.45:0/3217453847 shutdown_connections 2023-11-28T21:51:27.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.276+0000 7fad63fff640 1 --2- 172.21.15.45:0/3217453847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 0x7fad640701e0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.276+0000 7fad63fff640 1 --2- 172.21.15.45:0/3217453847 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad64159720 0x7fad640b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.276+0000 7fad63fff640 1 -- 172.21.15.45:0/3217453847 >> 172.21.15.45:0/3217453847 conn(0x7fad640b1050 msgr2=0x7fad640cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:27.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.276+0000 7fad63fff640 1 -- 172.21.15.45:0/3217453847 shutdown_connections 2023-11-28T21:51:27.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.276+0000 7fad63fff640 1 -- 172.21.15.45:0/3217453847 wait complete. 2023-11-28T21:51:27.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.278+0000 7fad63fff640 1 Processor -- start 2023-11-28T21:51:27.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.278+0000 7fad63fff640 1 -- start start 2023-11-28T21:51:27.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad63fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 0x7fad64147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad63fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad64071b90 0x7fad6409a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad63fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad64159720 0x7fad6414a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad63fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad64147cc0 con 0x7fad6406dfa0 2023-11-28T21:51:27.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad63fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6409adc0 con 0x7fad64159720 2023-11-28T21:51:27.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad63fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6409b0a0 con 0x7fad64071b90 2023-11-28T21:51:27.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad62ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 0x7fad64147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad62ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 0x7fad64147780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60946/0 (socket says 172.21.15.45:60946) 2023-11-28T21:51:27.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad62ffd640 1 -- 172.21.15.45:0/807463108 learned_addr learned my addr 172.21.15.45:0/807463108 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:27.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad62ffd640 1 -- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad64071b90 msgr2=0x7fad6409a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad62ffd640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad64071b90 0x7fad6409a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad62ffd640 1 -- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad64159720 msgr2=0x7fad6414a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad62ffd640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad64159720 0x7fad6414a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.280+0000 7fad62ffd640 1 -- 172.21.15.45:0/807463108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c0094d0 con 0x7fad6406dfa0 2023-11-28T21:51:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.281+0000 7fad62ffd640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 0x7fad64147780 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fad5400ccd0 tx=0x7fad5400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.281+0000 7fad5bfff640 1 -- 172.21.15.45:0/807463108 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54003990 con 0x7fad6406dfa0 2023-11-28T21:51:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.281+0000 7fad5bfff640 1 -- 172.21.15.45:0/807463108 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad54003af0 con 0x7fad6406dfa0 2023-11-28T21:51:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.281+0000 7fad5bfff640 1 -- 172.21.15.45:0/807463108 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54004020 con 0x7fad6406dfa0 2023-11-28T21:51:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.281+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6409b380 con 0x7fad6406dfa0 2023-11-28T21:51:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.281+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6409b940 con 0x7fad6406dfa0 2023-11-28T21:51:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.283+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6414b960 con 0x7fad6406dfa0 2023-11-28T21:51:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.283+0000 7fad5bfff640 1 -- 172.21.15.45:0/807463108 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 87) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fad54004180 con 0x7fad6406dfa0 2023-11-28T21:51:27.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.284+0000 7fad5bfff640 1 -- 172.21.15.45:0/807463108 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6789+0+0 (secure 0 0 0) 0x7fad540899a0 con 0x7fad6406dfa0 2023-11-28T21:51:27.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.288+0000 7fad5bfff640 1 -- 172.21.15.45:0/807463108 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad6414b960 con 0x7fad6406dfa0 2023-11-28T21:51:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.399+0000 7fad5bfff640 1 -- 172.21.15.45:0/807463108 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 88) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fad54054420 con 0x7fad6406dfa0 2023-11-28T21:51:27.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.399+0000 7fad5bfff640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fad44070be0 0x7fad44072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.409+0000 7fad5a7fc640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fad44070be0 0x7fad44072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.414+0000 7fad5a7fc640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fad44070be0 0x7fad44072fd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad4c036000 tx=0x7fad4c030000 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:27.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.480+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fad6414b960 con 0x7fad6406dfa0 2023-11-28T21:51:27.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.481+0000 7fad5bfff640 1 -- 172.21.15.45:0/807463108 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v172) v1 ==== 100+0+11 (secure 0 0 0) 0x7fad54074040 con 0x7fad6406dfa0 2023-11-28T21:51:27.483 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fad44070be0 msgr2=0x7fad44072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fad44070be0 0x7fad44072fd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad4c036000 tx=0x7fad4c030000 comp rx=0 tx=0).stop 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 msgr2=0x7fad64147780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 0x7fad64147780 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fad5400ccd0 tx=0x7fad5400a5c0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 shutdown_connections 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fad44070be0 0x7fad44072fd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad64159720 0x7fad6414a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fad64071b90 0x7fad6409a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 --2- 172.21.15.45:0/807463108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad6406dfa0 0x7fad64147780 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 >> 172.21.15.45:0/807463108 conn(0x7fad640b1050 msgr2=0x7fad640cd560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:27.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 shutdown_connections 2023-11-28T21:51:27.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.483+0000 7fad63fff640 1 -- 172.21.15.45:0/807463108 wait complete. 2023-11-28T21:51:27.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:51:27.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.714+0000 7f7d4ed47640 1 Processor -- start 2023-11-28T21:51:27.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.715+0000 7f7d4ed47640 1 -- start start 2023-11-28T21:51:27.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.716+0000 7f7d4ed47640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7d480b24a0 0x7f7d480703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.716+0000 7f7d4ed47640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 0x7f7d48070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.716+0000 7f7d4ed47640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d480b4310 con 0x7f7d48072200 2023-11-28T21:51:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.716+0000 7f7d4ed47640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480725e0 con 0x7f7d480b24a0 2023-11-28T21:51:27.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.716+0000 7f7d4ed47640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4806d8d0 con 0x7f7d48146610 2023-11-28T21:51:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.717+0000 7f7d4e546640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 0x7f7d48070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.717+0000 7f7d4dd45640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f7d48072200 0x7f7d480b1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:39094/0 (socket says 172.21.15.45:39094) 2023-11-28T21:51:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.717+0000 7f7d4dd45640 1 -- 172.21.15.45:0/1643795455 learned_addr learned my addr 172.21.15.45:0/1643795455 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.717+0000 7f7d4d544640 1 --2- 172.21.15.45:0/1643795455 >> v2:172.21.15.190:3300/0 conn(0x7f7d480b24a0 0x7f7d480703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.718+0000 7f7d4e546640 1 -- 172.21.15.45:0/1643795455 >> v2:172.21.15.190:3300/0 conn(0x7f7d480b24a0 msgr2=0x7f7d480703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.718+0000 7f7d4e546640 1 --2- 172.21.15.45:0/1643795455 >> v2:172.21.15.190:3300/0 conn(0x7f7d480b24a0 0x7f7d480703d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.718+0000 7f7d4e546640 1 -- 172.21.15.45:0/1643795455 >> v1:172.21.15.190:6789/0 conn(0x7f7d48072200 legacy=0x7f7d480b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.718+0000 7f7d4e546640 1 -- 172.21.15.45:0/1643795455 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4806ca20 con 0x7f7d48146610 2023-11-28T21:51:27.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.718+0000 7f7d4d544640 1 --2- 172.21.15.45:0/1643795455 >> v2:172.21.15.190:3300/0 conn(0x7f7d480b24a0 0x7f7d480703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T21:51:27.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.719+0000 7f7d4e546640 1 --2- 172.21.15.45:0/1643795455 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 0x7f7d48070910 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009d90 tx=0x7f7d4002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd80cf8d317436fa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.719+0000 7f7d4cd43640 1 -- 172.21.15.45:0/1643795455 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4002cc60 con 0x7f7d48146610 2023-11-28T21:51:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4cd43640 1 -- 172.21.15.45:0/1643795455 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4002cdc0 con 0x7f7d48146610 2023-11-28T21:51:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4cd43640 1 -- 172.21.15.45:0/1643795455 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40034660 con 0x7f7d48146610 2023-11-28T21:51:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1643795455 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 msgr2=0x7f7d48070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4ed47640 1 --2- 172.21.15.45:0/1643795455 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 0x7f7d48070910 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009d90 tx=0x7f7d4002beb0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1643795455 shutdown_connections 2023-11-28T21:51:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4ed47640 1 --2- 172.21.15.45:0/1643795455 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 0x7f7d48070910 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4ed47640 1 --2- 172.21.15.45:0/1643795455 >> v2:172.21.15.190:3300/0 conn(0x7f7d480b24a0 0x7f7d480703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1643795455 >> 172.21.15.45:0/1643795455 conn(0x7f7d4806ab30 msgr2=0x7f7d480b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1643795455 shutdown_connections 2023-11-28T21:51:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.720+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1643795455 wait complete. 2023-11-28T21:51:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.722+0000 7f7d4ed47640 1 Processor -- start 2023-11-28T21:51:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.722+0000 7f7d4ed47640 1 -- start start 2023-11-28T21:51:27.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.723+0000 7f7d4ed47640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d48072200 0x7f7d4815c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.723+0000 7f7d4ed47640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d480b24a0 0x7f7d4815e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.723+0000 7f7d4ed47640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 0x7f7d480ad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.723+0000 7f7d4ed47640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4809a9c0 con 0x7f7d48072200 2023-11-28T21:51:27.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.723+0000 7f7d4ed47640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4815cbe0 con 0x7f7d480b24a0 2023-11-28T21:51:27.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.723+0000 7f7d4ed47640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4813f4b0 con 0x7f7d48146610 2023-11-28T21:51:27.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4dd45640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d48072200 0x7f7d4815c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4dd45640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d48072200 0x7f7d4815c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60948/0 (socket says 172.21.15.45:60948) 2023-11-28T21:51:27.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4dd45640 1 -- 172.21.15.45:0/1856229709 learned_addr learned my addr 172.21.15.45:0/1856229709 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:27.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4e546640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 0x7f7d480ad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4d544640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d480b24a0 0x7f7d4815e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4dd45640 1 -- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 msgr2=0x7f7d480ad5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4dd45640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 0x7f7d480ad5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4dd45640 1 -- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d480b24a0 msgr2=0x7f7d4815e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4dd45640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d480b24a0 0x7f7d4815e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.724+0000 7f7d4dd45640 1 -- 172.21.15.45:0/1856229709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38009400 con 0x7f7d48072200 2023-11-28T21:51:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.725+0000 7f7d4dd45640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d48072200 0x7f7d4815c6a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400c480 tx=0x7f7d4400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.725+0000 7f7d3e7fc640 1 -- 172.21.15.45:0/1856229709 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44005ce0 con 0x7f7d48072200 2023-11-28T21:51:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.725+0000 7f7d3e7fc640 1 -- 172.21.15.45:0/1856229709 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d44005e40 con 0x7f7d48072200 2023-11-28T21:51:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.725+0000 7f7d3e7fc640 1 -- 172.21.15.45:0/1856229709 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d440072f0 con 0x7f7d48072200 2023-11-28T21:51:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.725+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d40009a70 con 0x7f7d48072200 2023-11-28T21:51:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.725+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4815f430 con 0x7f7d48072200 2023-11-28T21:51:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.726+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4809d0f0 con 0x7f7d48072200 2023-11-28T21:51:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.727+0000 7f7d3e7fc640 1 -- 172.21.15.45:0/1856229709 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 88) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f7d44007510 con 0x7f7d48072200 2023-11-28T21:51:27.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.728+0000 7f7d3e7fc640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7d2c06ac90 0x7f7d2c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:27.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.728+0000 7f7d3e7fc640 1 -- 172.21.15.45:0/1856229709 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f7d44088930 con 0x7f7d48072200 2023-11-28T21:51:27.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.729+0000 7f7d4d544640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7d2c06ac90 0x7f7d2c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:27.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.730+0000 7f7d4d544640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7d2c06ac90 0x7f7d2c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d48160540 tx=0x7f7d38008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.733+0000 7f7d3e7fc640 1 -- 172.21.15.45:0/1856229709 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d44053990 con 0x7f7d48072200 2023-11-28T21:51:27.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.965+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d480b4310 con 0x7f7d48072200 2023-11-28T21:51:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.967+0000 7f7d3e7fc640 1 -- 172.21.15.45:0/1856229709 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f7d480b4310 con 0x7f7d48072200 2023-11-28T21:51:27.969 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:27.969 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7531,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1320359028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1320359028},{"type":"v1","addr":"172.21.15.45:6833","nonce":1320359028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/808323687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":808323687},{"type":"v1","addr":"172.21.15.190:6836","nonce":808323687}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3406194668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3406194668},{"type":"v1","addr":"172.21.15.190:6837","nonce":3406194668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:55:12.738156+0000","modified":"2023-11-28T20:55:15.949737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"c","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/637440036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":637440036},{"type":"v1","addr":"172.21.15.45:6835","nonce":637440036}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-28T21:51:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 76 2023-11-28T21:51:27.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7d2c06ac90 msgr2=0x7f7d2c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7d2c06ac90 0x7f7d2c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d48160540 tx=0x7f7d38008040 comp rx=0 tx=0).stop 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d48072200 msgr2=0x7f7d4815c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d48072200 0x7f7d4815c6a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400c480 tx=0x7f7d4400c940 comp rx=0 tx=0).stop 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 shutdown_connections 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7d2c06ac90 0x7f7d2c06d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d48146610 0x7f7d480ad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d480b24a0 0x7f7d4815e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 --2- 172.21.15.45:0/1856229709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d48072200 0x7f7d4815c6a0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 >> 172.21.15.45:0/1856229709 conn(0x7f7d4806ab30 msgr2=0x7f7d480b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 shutdown_connections 2023-11-28T21:51:27.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:27.969+0000 7f7d4ed47640 1 -- 172.21.15.45:0/1856229709 wait complete. 2023-11-28T21:51:27.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:51:28.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.214+0000 7ff6b7375640 1 Processor -- start 2023-11-28T21:51:28.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.215+0000 7ff6b7375640 1 -- start start 2023-11-28T21:51:28.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.216+0000 7ff6b7375640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff6b0072200 0x7ff6b00b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.216+0000 7ff6b7375640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b00b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.217+0000 7ff6b7375640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b00b4310 con 0x7ff6b0146b60 2023-11-28T21:51:28.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.217+0000 7ff6b7375640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b00725e0 con 0x7ff6b0072200 2023-11-28T21:51:28.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.217+0000 7ff6b7375640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b006d8d0 con 0x7ff6b0146610 2023-11-28T21:51:28.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.217+0000 7ff6b5b72640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b00b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.218+0000 7ff6b6373640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff6b0072200 0x7ff6b00b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.218+0000 7ff6b6373640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff6b0072200 0x7ff6b00b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:55528/0 (socket says 172.21.15.45:55528) 2023-11-28T21:51:28.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.218+0000 7ff6b6373640 1 -- 172.21.15.45:0/1735425728 learned_addr learned my addr 172.21.15.45:0/1735425728 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:28.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.218+0000 7ff6b5b72640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b00b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:54898/0 (socket says 172.21.15.45:54898) 2023-11-28T21:51:28.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.218+0000 7ff6b6b74640 1 --1- 172.21.15.45:0/1735425728 >> v1:172.21.15.190:6789/0 conn(0x7ff6b0146b60 0x7ff6b0160e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:39108/0 (socket says 172.21.15.45:39108) 2023-11-28T21:51:28.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.219+0000 7ff6b6373640 1 -- 172.21.15.45:0/1735425728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 msgr2=0x7ff6b00b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:28.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.219+0000 7ff6b6373640 1 --2- 172.21.15.45:0/1735425728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b00b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.219+0000 7ff6b6373640 1 -- 172.21.15.45:0/1735425728 >> v1:172.21.15.190:6789/0 conn(0x7ff6b0146b60 legacy=0x7ff6b0160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.219+0000 7ff6b6373640 1 -- 172.21.15.45:0/1735425728 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b006ca20 con 0x7ff6b0072200 2023-11-28T21:51:28.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.219+0000 7ff6b5b72640 1 --2- 172.21.15.45:0/1735425728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b00b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T21:51:28.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.220+0000 7ff6b6373640 1 --2- 172.21.15.45:0/1735425728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b0072200 0x7ff6b00b1d90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac009870 tx=0x7ff6ac02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92c5e748b291568d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:28.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.220+0000 7ff6b5371640 1 -- 172.21.15.45:0/1735425728 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac030840 con 0x7ff6b0072200 2023-11-28T21:51:28.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.220+0000 7ff6b5371640 1 -- 172.21.15.45:0/1735425728 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac0309a0 con 0x7ff6b0072200 2023-11-28T21:51:28.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.220+0000 7ff6b5371640 1 -- 172.21.15.45:0/1735425728 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac030d30 con 0x7ff6b0072200 2023-11-28T21:51:28.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.221+0000 7ff6b7375640 1 -- 172.21.15.45:0/1735425728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b0072200 msgr2=0x7ff6b00b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.221+0000 7ff6b7375640 1 --2- 172.21.15.45:0/1735425728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b0072200 0x7ff6b00b1d90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac009870 tx=0x7ff6ac02b980 comp rx=0 tx=0).stop 2023-11-28T21:51:28.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.221+0000 7ff6b7375640 1 -- 172.21.15.45:0/1735425728 shutdown_connections 2023-11-28T21:51:28.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.221+0000 7ff6b7375640 1 --2- 172.21.15.45:0/1735425728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b00b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.221+0000 7ff6b7375640 1 --2- 172.21.15.45:0/1735425728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b0072200 0x7ff6b00b1d90 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.221+0000 7ff6b7375640 1 -- 172.21.15.45:0/1735425728 >> 172.21.15.45:0/1735425728 conn(0x7ff6b006ab30 msgr2=0x7ff6b00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:28.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.221+0000 7ff6b7375640 1 -- 172.21.15.45:0/1735425728 shutdown_connections 2023-11-28T21:51:28.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.221+0000 7ff6b7375640 1 -- 172.21.15.45:0/1735425728 wait complete. 2023-11-28T21:51:28.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.223+0000 7ff6b7375640 1 Processor -- start 2023-11-28T21:51:28.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.223+0000 7ff6b7375640 1 -- start start 2023-11-28T21:51:28.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.224+0000 7ff6b7375640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b0072200 0x7ff6b015cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.224+0000 7ff6b7375640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b009a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.224+0000 7ff6b7375640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b0146b60 0x7ff6b01f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.224+0000 7ff6b7375640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b009b350 con 0x7ff6b0146b60 2023-11-28T21:51:28.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.224+0000 7ff6b7375640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b009b630 con 0x7ff6b0072200 2023-11-28T21:51:28.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.224+0000 7ff6b7375640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b009b910 con 0x7ff6b0146610 2023-11-28T21:51:28.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.224+0000 7ff6b6b74640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b0146b60 0x7ff6b01f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b6b74640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b0146b60 0x7ff6b01f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60956/0 (socket says 172.21.15.45:60956) 2023-11-28T21:51:28.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b6b74640 1 -- 172.21.15.45:0/36314516 learned_addr learned my addr 172.21.15.45:0/36314516 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:28.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b5b72640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b009a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b6373640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b0072200 0x7ff6b015cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b6b74640 1 -- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 msgr2=0x7ff6b009a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b6b74640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b009a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b6b74640 1 -- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b0072200 msgr2=0x7ff6b015cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b6b74640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b0072200 0x7ff6b015cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b6b74640 1 -- 172.21.15.45:0/36314516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a0009400 con 0x7ff6b0146b60 2023-11-28T21:51:28.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.225+0000 7ff6b6b74640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b0146b60 0x7ff6b01f3090 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800c730 tx=0x7ff6a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:28.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.226+0000 7ff6a6ffd640 1 -- 172.21.15.45:0/36314516 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a800ce40 con 0x7ff6b0146b60 2023-11-28T21:51:28.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.226+0000 7ff6a6ffd640 1 -- 172.21.15.45:0/36314516 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a800a640 con 0x7ff6b0146b60 2023-11-28T21:51:28.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.226+0000 7ff6a6ffd640 1 -- 172.21.15.45:0/36314516 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a800a9f0 con 0x7ff6b0146b60 2023-11-28T21:51:28.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.226+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ac0094d0 con 0x7ff6b0146b60 2023-11-28T21:51:28.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.226+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b009bed0 con 0x7ff6b0146b60 2023-11-28T21:51:28.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.227+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b0161d90 con 0x7ff6b0146b60 2023-11-28T21:51:28.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.228+0000 7ff6a6ffd640 1 -- 172.21.15.45:0/36314516 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 88) v1 ==== 89069+0+0 (secure 0 0 0) 0x7ff6a800ab50 con 0x7ff6b0146b60 2023-11-28T21:51:28.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.229+0000 7ff6a6ffd640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff69406ac90 0x7ff69406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.229+0000 7ff6a6ffd640 1 -- 172.21.15.45:0/36314516 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6789+0+0 (secure 0 0 0) 0x7ff6a8088b40 con 0x7ff6b0146b60 2023-11-28T21:51:28.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.230+0000 7ff6b6373640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff69406ac90 0x7ff69406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.231+0000 7ff6b6373640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff69406ac90 0x7ff69406d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac036000 tx=0x7ff6ac008030 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:28.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.233+0000 7ff6a6ffd640 1 -- 172.21.15.45:0/36314516 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6a8053ba0 con 0x7ff6b0146b60 2023-11-28T21:51:28.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.412+0000 7ff6a6ffd640 1 -- 172.21.15.45:0/36314516 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 89) v1 ==== 89277+0+0 (secure 0 0 0) 0x7ff6a8053f70 con 0x7ff6b0146b60 2023-11-28T21:51:28.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.432+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6b00b4310 con 0x7ff6b0146b60 2023-11-28T21:51:28.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.435+0000 7ff6a6ffd640 1 -- 172.21.15.45:0/36314516 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+17870 (secure 0 0 0) 0x7ff6b00b4310 con 0x7ff6b0146b60 2023-11-28T21:51:28.438 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:28.438 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":172,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:51:26.378062+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:55:10.712027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-11-28T20:55:11.779380+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":171,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/4079793895":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/17282265":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/932255615":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/3336614716":"2023-11-29T21:51:26.378009+0000","172.21.15.190:6832/560718832":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/4184631271":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/2635607595":"2023-11-29T21:51:26.378009+0000","172.21.15.190:6833/560718832":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/943491299":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1767202704":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2782217070":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/290777784":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6833/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6832/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1078115353":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2712902162":"2023-11-29T20:55:29.635279+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:51:28.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff69406ac90 msgr2=0x7ff69406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff69406ac90 0x7ff69406d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac036000 tx=0x7ff6ac008030 comp rx=0 tx=0).stop 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b0146b60 msgr2=0x7ff6b01f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b0146b60 0x7ff6b01f3090 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800c730 tx=0x7ff6a800cbf0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 shutdown_connections 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff69406ac90 0x7ff69406d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b0146b60 0x7ff6b01f3090 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff6b0146610 0x7ff6b009a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 --2- 172.21.15.45:0/36314516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6b0072200 0x7ff6b015cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 >> 172.21.15.45:0/36314516 conn(0x7ff6b006ab30 msgr2=0x7ff6b00b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 shutdown_connections 2023-11-28T21:51:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.438+0000 7ff6b7375640 1 -- 172.21.15.45:0/36314516 wait complete. 2023-11-28T21:51:28.465 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T21:51:28.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:51:28.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.677+0000 7fc4bfcd4640 1 Processor -- start 2023-11-28T21:51:28.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.677+0000 7fc4bfcd4640 1 -- start start 2023-11-28T21:51:28.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.679+0000 7fc4bfcd4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc4b80b4c40 0x7fc4b80b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.679+0000 7fc4bfcd4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b80711c0 con 0x7fc4b80b0c30 2023-11-28T21:51:28.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.679+0000 7fc4bfcd4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b80714a0 con 0x7fc4b8161960 2023-11-28T21:51:28.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.679+0000 7fc4bfcd4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b8071e40 con 0x7fc4b80b4c40 2023-11-28T21:51:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.679+0000 7fc4be271640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc4b80b0c30 0x7fc4b80b1040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33116/0 (socket says 172.21.15.45:33116) 2023-11-28T21:51:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.679+0000 7fc4be271640 1 -- 172.21.15.45:0/3173013297 learned_addr learned my addr 172.21.15.45:0/3173013297 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bd26f640 1 --2- 172.21.15.45:0/3173013297 >> v2:172.21.15.190:3300/0 conn(0x7fc4b80b4c40 0x7fc4b80b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704914880 0 0) 0x7fc4b80711c0 con 0x7fc4b80b0c30 2023-11-28T21:51:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a4003520 con 0x7fc4b80b0c30 2023-11-28T21:51:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1948120137 0 0) 0x7fc4a4003520 con 0x7fc4b80b0c30 2023-11-28T21:51:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 >> v2:172.21.15.190:3300/0 conn(0x7fc4b80b4c40 msgr2=0x7fc4b80b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bca6e640 1 --2- 172.21.15.45:0/3173013297 >> v2:172.21.15.190:3300/0 conn(0x7fc4b80b4c40 0x7fc4b80b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 >> v1:172.21.15.190:6789/0 conn(0x7fc4b8161960 legacy=0x7fc4b8070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b8071780 con 0x7fc4b80b0c30 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc4b00019f0 con 0x7fc4b80b0c30 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bd26f640 1 --2- 172.21.15.45:0/3173013297 >> v2:172.21.15.190:3300/0 conn(0x7fc4b80b4c40 0x7fc4b80b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.680+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 >> v1:172.21.15.45:6789/0 conn(0x7fc4b80b0c30 legacy=0x7fc4b80b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bca6e640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4a4003970 0x7fc4a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bca6e640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a40047c0 0x7fc4a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bca6e640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a4005180 0x7fc4a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b8071e40 con 0x7fc4a40047c0 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a400c120 con 0x7fc4a4005180 2023-11-28T21:51:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a400c2e0 con 0x7fc4a4003970 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bd26f640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a40047c0 0x7fc4a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4be271640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a4005180 0x7fc4a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bda70640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4a4003970 0x7fc4a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bd26f640 1 -- 172.21.15.45:0/3173013297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4a4003970 msgr2=0x7fc4a4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bd26f640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4a4003970 0x7fc4a4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bd26f640 1 -- 172.21.15.45:0/3173013297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a4005180 msgr2=0x7fc4a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bd26f640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a4005180 0x7fc4a400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bd26f640 1 -- 172.21.15.45:0/3173013297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4003520 con 0x7fc4a40047c0 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4bda70640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4a4003970 0x7fc4a4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.681+0000 7fc4be271640 1 -- 172.21.15.45:0/3173013297 reap_dead start 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.683+0000 7fc4bd26f640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a40047c0 0x7fc4a4004c40 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009c90 tx=0x7fc4a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881cf9372be9860f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.683+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a802c8f0 con 0x7fc4a40047c0 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.683+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a802ca50 con 0x7fc4a40047c0 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.683+0000 7fc4bca6e640 1 -- 172.21.15.45:0/3173013297 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a802ce40 con 0x7fc4a40047c0 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.683+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3173013297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a40047c0 msgr2=0x7fc4a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.683+0000 7fc4bfcd4640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a40047c0 0x7fc4a4004c40 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009c90 tx=0x7fc4a40057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.684+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3173013297 shutdown_connections 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.684+0000 7fc4bfcd4640 1 --2- 172.21.15.45:0/3173013297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a40047c0 0x7fc4a4004c40 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.684+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3173013297 >> 172.21.15.45:0/3173013297 conn(0x7fc4b806a500 msgr2=0x7fc4b806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:28.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.684+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3173013297 shutdown_connections 2023-11-28T21:51:28.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.684+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3173013297 wait complete. 2023-11-28T21:51:28.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.686+0000 7fc4bfcd4640 1 Processor -- start 2023-11-28T21:51:28.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.686+0000 7fc4bfcd4640 1 -- start start 2023-11-28T21:51:28.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bfcd4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a40047c0 0x7fc4b81fa900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bfcd4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4b81f7b00 0x7fc4b80ad380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bfcd4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4b81f7f40 0x7fc4b80ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bfcd4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b80acbb0 con 0x7fc4b81f7b00 2023-11-28T21:51:28.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bfcd4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b81fae40 con 0x7fc4a40047c0 2023-11-28T21:51:28.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bfcd4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b80a85d0 con 0x7fc4b81f7f40 2023-11-28T21:51:28.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bd26f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4b81f7b00 0x7fc4b80ad380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bd26f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4b81f7b00 0x7fc4b80ad380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60966/0 (socket says 172.21.15.45:60966) 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bd26f640 1 -- 172.21.15.45:0/3834113334 learned_addr learned my addr 172.21.15.45:0/3834113334 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bda70640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a40047c0 0x7fc4b81fa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bd26f640 1 -- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4b81f7f40 msgr2=0x7fc4b80ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bd26f640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4b81f7f40 0x7fc4b80ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bd26f640 1 -- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a40047c0 msgr2=0x7fc4b81fa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bd26f640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a40047c0 0x7fc4b81fa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bd26f640 1 -- 172.21.15.45:0/3834113334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b80a88b0 con 0x7fc4b81f7b00 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.687+0000 7fc4bda70640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a40047c0 0x7fc4b81fa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.688+0000 7fc4bd26f640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4b81f7b00 0x7fc4b80ad380 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8037040 tx=0x7fc4a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.688+0000 7fc4ae7fc640 1 -- 172.21.15.45:0/3834113334 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8034a80 con 0x7fc4b81f7b00 2023-11-28T21:51:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.688+0000 7fc4ae7fc640 1 -- 172.21.15.45:0/3834113334 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a8034be0 con 0x7fc4b81f7b00 2023-11-28T21:51:28.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.688+0000 7fc4ae7fc640 1 -- 172.21.15.45:0/3834113334 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004020 con 0x7fc4b81f7b00 2023-11-28T21:51:28.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.688+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b80a8b90 con 0x7fc4b81f7b00 2023-11-28T21:51:28.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.688+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b80abe30 con 0x7fc4b81f7b00 2023-11-28T21:51:28.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.690+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b80cecf0 con 0x7fc4b81f7b00 2023-11-28T21:51:28.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.691+0000 7fc4ae7fc640 1 -- 172.21.15.45:0/3834113334 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 89) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fc4a8004180 con 0x7fc4b81f7b00 2023-11-28T21:51:28.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.692+0000 7fc4ae7fc640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fc49c06aed0 0x7fc49c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:28.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.692+0000 7fc4ae7fc640 1 -- 172.21.15.45:0/3834113334 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6789+0+0 (secure 0 0 0) 0x7fc4a80abed0 con 0x7fc4b81f7b00 2023-11-28T21:51:28.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.692+0000 7fc4bda70640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fc49c06aed0 0x7fc49c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:28.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.694+0000 7fc4bda70640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fc49c06aed0 0x7fc49c06d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4000c00 tx=0x7fc4b4009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:28.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.697+0000 7fc4ae7fc640 1 -- 172.21.15.45:0/3834113334 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4a8076ae0 con 0x7fc4b81f7b00 2023-11-28T21:51:28.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.941+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4b8068930 con 0x7fc4b81f7b00 2023-11-28T21:51:28.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.942+0000 7fc4ae7fc640 1 -- 172.21.15.45:0/3834113334 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fc4a807a100 con 0x7fc4b81f7b00 2023-11-28T21:51:28.945 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:28.945 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7531,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1320359028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1320359028},{"type":"v1","addr":"172.21.15.45:6833","nonce":1320359028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/808323687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":808323687},{"type":"v1","addr":"172.21.15.190:6836","nonce":808323687}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3406194668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3406194668},{"type":"v1","addr":"172.21.15.190:6837","nonce":3406194668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:55:12.738156+0000","modified":"2023-11-28T20:55:15.949737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"c","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/637440036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":637440036},{"type":"v1","addr":"172.21.15.45:6835","nonce":637440036}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-28T21:51:28.945 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 76 2023-11-28T21:51:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.945+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fc49c06aed0 msgr2=0x7fc49c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.945+0000 7fc4bfcd4640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fc49c06aed0 0x7fc49c06d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4000c00 tx=0x7fc4b4009040 comp rx=0 tx=0).stop 2023-11-28T21:51:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.945+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4b81f7b00 msgr2=0x7fc4b80ad380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.945+0000 7fc4bfcd4640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4b81f7b00 0x7fc4b80ad380 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8037040 tx=0x7fc4a8004730 comp rx=0 tx=0).stop 2023-11-28T21:51:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.946+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 shutdown_connections 2023-11-28T21:51:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.946+0000 7fc4bfcd4640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fc49c06aed0 0x7fc49c06d390 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.946+0000 7fc4bfcd4640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4b81f7f40 0x7fc4b80ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.946+0000 7fc4bfcd4640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4b81f7b00 0x7fc4b80ad380 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.946+0000 7fc4bfcd4640 1 --2- 172.21.15.45:0/3834113334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4a40047c0 0x7fc4b81fa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.946+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 >> 172.21.15.45:0/3834113334 conn(0x7fc4b806a500 msgr2=0x7fc4b80b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.946+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 shutdown_connections 2023-11-28T21:51:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:28.946+0000 7fc4bfcd4640 1 -- 172.21.15.45:0/3834113334 wait complete. 2023-11-28T21:51:28.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:51:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.192+0000 7f446b59e640 1 Processor -- start 2023-11-28T21:51:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.192+0000 7f446b59e640 1 -- start start 2023-11-28T21:51:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.194+0000 7f446b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f446c155780 0x7f446c0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.194+0000 7f446b59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c15c3c0 con 0x7f446c0ce800 2023-11-28T21:51:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.194+0000 7f446b59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c155b60 con 0x7f446c0ae540 2023-11-28T21:51:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.194+0000 7f446b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446c15ba90 con 0x7f446c155780 2023-11-28T21:51:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.194+0000 7f446ad9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f446c0ce800 0x7f446c0b20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33124/0 (socket says 172.21.15.45:33124) 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.194+0000 7f446ad9d640 1 -- 172.21.15.45:0/4287607865 learned_addr learned my addr 172.21.15.45:0/4287607865 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.195+0000 7f446a59c640 1 --2- 172.21.15.45:0/4287607865 >> v2:172.21.15.190:3300/0 conn(0x7f446c155780 0x7f446c0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.195+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386955356 0 0) 0x7f446c15c3c0 con 0x7f446c0ce800 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.195+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4458003520 con 0x7f446c0ce800 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.195+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 592613164 0 0) 0x7f4458003520 con 0x7f446c0ce800 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.195+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 >> v2:172.21.15.190:3300/0 conn(0x7f446c155780 msgr2=0x7f446c0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.195+0000 7f4469d9b640 1 --2- 172.21.15.45:0/4287607865 >> v2:172.21.15.190:3300/0 conn(0x7f446c155780 0x7f446c0abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.195+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 >> v1:172.21.15.190:6789/0 conn(0x7f446c0ae540 legacy=0x7f446c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c15bd70 con 0x7f446c0ce800 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f44640019f0 con 0x7f446c0ce800 2023-11-28T21:51:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f446a59c640 1 --2- 172.21.15.45:0/4287607865 >> v2:172.21.15.190:3300/0 conn(0x7f446c155780 0x7f446c0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 >> v1:172.21.15.45:6789/0 conn(0x7f446c0ce800 legacy=0x7f446c0b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4469d9b640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4458003970 0x7f4458003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4469d9b640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44580047c0 0x7f4458004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4469d9b640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458005180 0x7f445800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c15ba90 con 0x7f4458005180 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445800c020 con 0x7f44580047c0 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445800c1e0 con 0x7f4458003970 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f446ad9d640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458005180 0x7f445800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f4461d9b640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44580047c0 0x7f4458004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.196+0000 7f446a59c640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4458003970 0x7f4458003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.197+0000 7f446ad9d640 1 -- 172.21.15.45:0/4287607865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4458003970 msgr2=0x7f4458003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:29.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.197+0000 7f446ad9d640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4458003970 0x7f4458003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.197+0000 7f446ad9d640 1 -- 172.21.15.45:0/4287607865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44580047c0 msgr2=0x7f4458004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.197+0000 7f446ad9d640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44580047c0 0x7f4458004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.197+0000 7f446ad9d640 1 -- 172.21.15.45:0/4287607865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4458003520 con 0x7f4458005180 2023-11-28T21:51:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.198+0000 7f446ad9d640 1 -- 172.21.15.45:0/4287607865 reap_dead start 2023-11-28T21:51:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.198+0000 7f446ad9d640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458005180 0x7f445800b980 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f446400a830 tx=0x7f44580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8422dd320984a6d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.198+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44640072d0 con 0x7f4458005180 2023-11-28T21:51:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.198+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446402f070 con 0x7f4458005180 2023-11-28T21:51:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.198+0000 7f4469d9b640 1 -- 172.21.15.45:0/4287607865 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4464007650 con 0x7f4458005180 2023-11-28T21:51:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.199+0000 7f446b59e640 1 -- 172.21.15.45:0/4287607865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458005180 msgr2=0x7f445800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.199+0000 7f446b59e640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458005180 0x7f445800b980 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f446400a830 tx=0x7f44580057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.199+0000 7f446b59e640 1 -- 172.21.15.45:0/4287607865 shutdown_connections 2023-11-28T21:51:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.199+0000 7f446b59e640 1 --2- 172.21.15.45:0/4287607865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458005180 0x7f445800b980 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.199+0000 7f446b59e640 1 -- 172.21.15.45:0/4287607865 >> 172.21.15.45:0/4287607865 conn(0x7f446c06ab30 msgr2=0x7f446c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.199+0000 7f446b59e640 1 -- 172.21.15.45:0/4287607865 shutdown_connections 2023-11-28T21:51:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.199+0000 7f446b59e640 1 -- 172.21.15.45:0/4287607865 wait complete. 2023-11-28T21:51:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.201+0000 7f446b59e640 1 Processor -- start 2023-11-28T21:51:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.201+0000 7f446b59e640 1 -- start start 2023-11-28T21:51:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4458005180 0x7f446c1f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f446c141660 0x7f446c0b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f446c141ac0 0x7f446c0b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c1f4ee0 con 0x7f446c141ac0 2023-11-28T21:51:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c14ee10 con 0x7f4458005180 2023-11-28T21:51:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446c09bef0 con 0x7f446c141660 2023-11-28T21:51:29.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f446c141ac0 0x7f446c0b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f446c141ac0 0x7f446c0b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60980/0 (socket says 172.21.15.45:60980) 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446ad9d640 1 -- 172.21.15.45:0/1317198475 learned_addr learned my addr 172.21.15.45:0/1317198475 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f4461d9b640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f446c141660 0x7f446c0b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.202+0000 7f446a59c640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4458005180 0x7f446c1f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f446ad9d640 1 -- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f446c141660 msgr2=0x7f446c0b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f446ad9d640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f446c141660 0x7f446c0b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f446ad9d640 1 -- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4458005180 msgr2=0x7f446c1f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f446ad9d640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4458005180 0x7f446c1f49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f446ad9d640 1 -- 172.21.15.45:0/1317198475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c09c1d0 con 0x7f446c141ac0 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f446a59c640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4458005180 0x7f446c1f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:29.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f446ad9d640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f446c141ac0 0x7f446c0b3af0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f4464000f80 tx=0x7f4464001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f44637fe640 1 -- 172.21.15.45:0/1317198475 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4464004d60 con 0x7f446c141ac0 2023-11-28T21:51:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f44637fe640 1 -- 172.21.15.45:0/1317198475 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44640070d0 con 0x7f446c141ac0 2023-11-28T21:51:29.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f44637fe640 1 -- 172.21.15.45:0/1317198475 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4464007440 con 0x7f446c141ac0 2023-11-28T21:51:29.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.203+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c09c4b0 con 0x7f446c141ac0 2023-11-28T21:51:29.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.204+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446c14cf00 con 0x7f446c141ac0 2023-11-28T21:51:29.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.205+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446c14cf00 con 0x7f446c141ac0 2023-11-28T21:51:29.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.206+0000 7f44637fe640 1 -- 172.21.15.45:0/1317198475 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 89) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f4464031030 con 0x7f446c141ac0 2023-11-28T21:51:29.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.206+0000 7f44637fe640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f444c06af00 0x7f444c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.207+0000 7f44637fe640 1 -- 172.21.15.45:0/1317198475 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f44640ab7c0 con 0x7f446c141ac0 2023-11-28T21:51:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.207+0000 7f446a59c640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f444c06af00 0x7f444c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.208+0000 7f446a59c640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f444c06af00 0x7f444c06d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f445c009b10 tx=0x7f445c008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.211+0000 7f44637fe640 1 -- 172.21.15.45:0/1317198475 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44640762c0 con 0x7f446c141ac0 2023-11-28T21:51:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.447+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f446c14cf00 con 0x7f446c141ac0 2023-11-28T21:51:29.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.448+0000 7f44637fe640 1 -- 172.21.15.45:0/1317198475 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f44640798e0 con 0x7f446c141ac0 2023-11-28T21:51:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:29.502 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7531,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1320359028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1320359028},{"type":"v1","addr":"172.21.15.45:6833","nonce":1320359028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/808323687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":808323687},{"type":"v1","addr":"172.21.15.190:6836","nonce":808323687}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3406194668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3406194668},{"type":"v1","addr":"172.21.15.190:6837","nonce":3406194668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T20:55:12.738156+0000","modified":"2023-11-28T20:55:15.949737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"c","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/637440036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":637440036},{"type":"v1","addr":"172.21.15.45:6835","nonce":637440036}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-11-28T21:51:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 76 2023-11-28T21:51:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.450+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f444c06af00 msgr2=0x7f444c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.450+0000 7f446b59e640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f444c06af00 0x7f444c06d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f445c009b10 tx=0x7f445c008040 comp rx=0 tx=0).stop 2023-11-28T21:51:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.450+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f446c141ac0 msgr2=0x7f446c0b3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.450+0000 7f446b59e640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f446c141ac0 0x7f446c0b3af0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f4464000f80 tx=0x7f4464001d80 comp rx=0 tx=0).stop 2023-11-28T21:51:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.451+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 shutdown_connections 2023-11-28T21:51:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.451+0000 7f446b59e640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f444c06af00 0x7f444c06d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.451+0000 7f446b59e640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f446c141ac0 0x7f446c0b3af0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.451+0000 7f446b59e640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f446c141660 0x7f446c0b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.451+0000 7f446b59e640 1 --2- 172.21.15.45:0/1317198475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4458005180 0x7f446c1f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.451+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 >> 172.21.15.45:0/1317198475 conn(0x7f446c06ab30 msgr2=0x7f446c15afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.451+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 shutdown_connections 2023-11-28T21:51:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.451+0000 7f446b59e640 1 -- 172.21.15.45:0/1317198475 wait complete. 2023-11-28T21:51:29.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:51:29.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.731+0000 7f8605c02640 1 Processor -- start 2023-11-28T21:51:29.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.731+0000 7f8605c02640 1 -- start start 2023-11-28T21:51:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.733+0000 7f8605c02640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f86000b24a0 0x7f86000703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.733+0000 7f8605c02640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8600146610 0x7f8600070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.733+0000 7f8605c02640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86000b4310 con 0x7f8600072200 2023-11-28T21:51:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.733+0000 7f8605c02640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86000725e0 con 0x7f86000b24a0 2023-11-28T21:51:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.733+0000 7f8605c02640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860006d8d0 con 0x7f8600146610 2023-11-28T21:51:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.733+0000 7f8604c00640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8600072200 0x7f86000b1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33140/0 (socket says 172.21.15.45:33140) 2023-11-28T21:51:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.733+0000 7f8604c00640 1 -- 172.21.15.45:0/362482920 learned_addr learned my addr 172.21.15.45:0/362482920 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85fffff640 1 --2- 172.21.15.45:0/362482920 >> v2:172.21.15.190:3300/0 conn(0x7f86000b24a0 0x7f86000703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f8605401640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8600146610 0x7f8600070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658818894 0 0) 0x7f85f4001110 con 0x7f8600072200 2023-11-28T21:51:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f0003520 con 0x7f8600072200 2023-11-28T21:51:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1655469266 0 0) 0x7f85f0003520 con 0x7f8600072200 2023-11-28T21:51:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8600146610 msgr2=0x7f8600070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8600146610 0x7f8600070910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 >> v2:172.21.15.190:3300/0 conn(0x7f86000b24a0 msgr2=0x7f86000703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 --2- 172.21.15.45:0/362482920 >> v2:172.21.15.190:3300/0 conn(0x7f86000b24a0 0x7f86000703d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860006ca20 con 0x7f8600072200 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f85f4001800 con 0x7f8600072200 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85fffff640 1 --2- 172.21.15.45:0/362482920 >> v2:172.21.15.190:3300/0 conn(0x7f86000b24a0 0x7f86000703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 >> v1:172.21.15.45:6789/0 conn(0x7f8600072200 legacy=0x7f86000b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f0003970 0x7f85f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f85f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8600146610 0x7f85f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860006d8d0 con 0x7f85f00047c0 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f000b9e0 con 0x7f85f0003970 2023-11-28T21:51:29.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.734+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f000bba0 con 0x7f8600146610 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f85fffff640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f85f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f8604c00640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f0003970 0x7f85f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f8605401640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8600146610 0x7f85f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f85fffff640 1 -- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8600146610 msgr2=0x7f85f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f85fffff640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8600146610 0x7f85f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f85fffff640 1 -- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f0003970 msgr2=0x7f85f0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f85fffff640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f0003970 0x7f85f0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f85fffff640 1 -- 172.21.15.45:0/362482920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f0003520 con 0x7f85f00047c0 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f8605401640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8600146610 0x7f85f0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.735+0000 7f8604c00640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85f0003970 0x7f85f0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.736+0000 7f85fffff640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f85f0004c40 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f85ec006fd0 tx=0x7f85f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239887e957a42c29 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:29.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.737+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec030630 con 0x7f85f00047c0 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.737+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ec030790 con 0x7f85f00047c0 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.737+0000 7f85ff7fe640 1 -- 172.21.15.45:0/362482920 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec030b00 con 0x7f85f00047c0 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.737+0000 7f8605c02640 1 -- 172.21.15.45:0/362482920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 msgr2=0x7f85f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.737+0000 7f8605c02640 1 --2- 172.21.15.45:0/362482920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f85f0004c40 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f85ec006fd0 tx=0x7f85f00057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.737+0000 7f8605401640 1 -- 172.21.15.45:0/362482920 reap_dead start 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.737+0000 7f8605c02640 1 -- 172.21.15.45:0/362482920 shutdown_connections 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.737+0000 7f8605c02640 1 -- 172.21.15.45:0/362482920 >> 172.21.15.45:0/362482920 conn(0x7f860006ab30 msgr2=0x7f86000b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.738+0000 7f8605c02640 1 -- 172.21.15.45:0/362482920 shutdown_connections 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.738+0000 7f8605c02640 1 -- 172.21.15.45:0/362482920 wait complete. 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.739+0000 7f8605c02640 1 Processor -- start 2023-11-28T21:51:29.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.740+0000 7f8605c02640 1 -- start start 2023-11-28T21:51:29.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.740+0000 7f8605c02640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f860015c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.740+0000 7f8605c02640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86000af860 0x7f86000ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.740+0000 7f8605c02640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86000c8a40 0x7f86000c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8605c02640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86000b09a0 con 0x7f85f00047c0 2023-11-28T21:51:29.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8605c02640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86000c9430 con 0x7f86000af860 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8605c02640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86000afd20 con 0x7f86000c8a40 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8604c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f860015c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8604c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f860015c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32778/0 (socket says 172.21.15.45:32778) 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8604c00640 1 -- 172.21.15.45:0/1624246158 learned_addr learned my addr 172.21.15.45:0/1624246158 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8604c00640 1 -- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86000c8a40 msgr2=0x7f86000c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8604c00640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86000c8a40 0x7f86000c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8604c00640 1 -- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86000af860 msgr2=0x7f86000ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8604c00640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86000af860 0x7f86000ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.741+0000 7f8604c00640 1 -- 172.21.15.45:0/1624246158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86000b0000 con 0x7f85f00047c0 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.742+0000 7f8604c00640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f860015c9c0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f85f4007330 tx=0x7f85f40077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.742+0000 7f85fd7fa640 1 -- 172.21.15.45:0/1624246158 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f400e8d0 con 0x7f85f00047c0 2023-11-28T21:51:29.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.742+0000 7f85fd7fa640 1 -- 172.21.15.45:0/1624246158 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f400ea30 con 0x7f85f00047c0 2023-11-28T21:51:29.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.742+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86000b02e0 con 0x7f85f00047c0 2023-11-28T21:51:29.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.742+0000 7f85fd7fa640 1 -- 172.21.15.45:0/1624246158 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f400eda0 con 0x7f85f00047c0 2023-11-28T21:51:29.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.742+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860015cf00 con 0x7f85f00047c0 2023-11-28T21:51:29.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.743+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86000b0ae0 con 0x7f85f00047c0 2023-11-28T21:51:29.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.744+0000 7f85fd7fa640 1 -- 172.21.15.45:0/1624246158 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 89) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f85f401d460 con 0x7f85f00047c0 2023-11-28T21:51:29.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.745+0000 7f85fd7fa640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f85e406aed0 0x7f85e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:29.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.746+0000 7f85fd7fa640 1 -- 172.21.15.45:0/1624246158 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f85f40899f0 con 0x7f85f00047c0 2023-11-28T21:51:29.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.746+0000 7f85fffff640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f85e406aed0 0x7f85e406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:29.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.747+0000 7f85fffff640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f85e406aed0 0x7f85e406d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f85ec036000 tx=0x7f85ec030000 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:29.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.750+0000 7f85fd7fa640 1 -- 172.21.15.45:0/1624246158 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86000b0ae0 con 0x7f85f00047c0 2023-11-28T21:51:29.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.955+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86000b4310 con 0x7f85f00047c0 2023-11-28T21:51:29.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.958+0000 7f85fd7fa640 1 -- 172.21.15.45:0/1624246158 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+17870 (secure 0 0 0) 0x7f86000b4310 con 0x7f85f00047c0 2023-11-28T21:51:29.961 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:29.961 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":172,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:51:26.378062+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-11-28T20:55:10.712027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-11-28T20:55:11.779380+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":171,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/4079793895":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/17282265":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/932255615":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/3336614716":"2023-11-29T21:51:26.378009+0000","172.21.15.190:6832/560718832":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/4184631271":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/2635607595":"2023-11-29T21:51:26.378009+0000","172.21.15.190:6833/560718832":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/943491299":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1767202704":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2782217070":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/290777784":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6833/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6832/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1078115353":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2712902162":"2023-11-29T20:55:29.635279+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:51:29.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f85e406aed0 msgr2=0x7f85e406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f85e406aed0 0x7f85e406d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f85ec036000 tx=0x7f85ec030000 comp rx=0 tx=0).stop 2023-11-28T21:51:29.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 msgr2=0x7f860015c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f860015c9c0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f85f4007330 tx=0x7f85f40077f0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 shutdown_connections 2023-11-28T21:51:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f85e406aed0 0x7f85e406d390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86000c8a40 0x7f86000c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86000af860 0x7f86000ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 --2- 172.21.15.45:0/1624246158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85f00047c0 0x7f860015c9c0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 >> 172.21.15.45:0/1624246158 conn(0x7f860006ab30 msgr2=0x7f86000b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 shutdown_connections 2023-11-28T21:51:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:29.961+0000 7f8605c02640 1 -- 172.21.15.45:0/1624246158 wait complete. 2023-11-28T21:51:29.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T21:51:30.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.194+0000 7ff06ce5c640 1 Processor -- start 2023-11-28T21:51:30.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.195+0000 7ff06ce5c640 1 -- start start 2023-11-28T21:51:30.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.196+0000 7ff06ce5c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815d3e0 0x7ff06806a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.196+0000 7ff06ce5c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0680ce6d0 0x7ff0680d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.196+0000 7ff06ce5c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0680b6390 con 0x7ff0680715a0 2023-11-28T21:51:30.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.196+0000 7ff06ce5c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06806b5d0 con 0x7ff0680ce6d0 2023-11-28T21:51:30.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.196+0000 7ff06ce5c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680d18b0 con 0x7ff06815d3e0 2023-11-28T21:51:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.197+0000 7ff06659c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815d3e0 0x7ff06806a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.197+0000 7ff06659c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815d3e0 0x7ff06806a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:54982/0 (socket says 172.21.15.45:54982) 2023-11-28T21:51:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.197+0000 7ff06659c640 1 -- 172.21.15.45:0/2251022270 learned_addr learned my addr 172.21.15.45:0/2251022270 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.197+0000 7ff066d9d640 1 --2- 172.21.15.45:0/2251022270 >> v2:172.21.15.45:3300/0 conn(0x7ff0680ce6d0 0x7ff0680d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.198+0000 7ff06659c640 1 -- 172.21.15.45:0/2251022270 >> v2:172.21.15.45:3300/0 conn(0x7ff0680ce6d0 msgr2=0x7ff0680d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.198+0000 7ff06659c640 1 --2- 172.21.15.45:0/2251022270 >> v2:172.21.15.45:3300/0 conn(0x7ff0680ce6d0 0x7ff0680d1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.198+0000 7ff06659c640 1 -- 172.21.15.45:0/2251022270 >> v1:172.21.15.190:6789/0 conn(0x7ff0680715a0 legacy=0x7ff0680cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.198+0000 7ff06659c640 1 -- 172.21.15.45:0/2251022270 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06806ad30 con 0x7ff06815d3e0 2023-11-28T21:51:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.199+0000 7ff06659c640 1 --2- 172.21.15.45:0/2251022270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815d3e0 0x7ff06806a500 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff0580098a0 tx=0x7ff05802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b73b5212522ddff server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.200+0000 7ff065d9b640 1 -- 172.21.15.45:0/2251022270 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058031640 con 0x7ff06815d3e0 2023-11-28T21:51:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.200+0000 7ff065d9b640 1 -- 172.21.15.45:0/2251022270 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0580317a0 con 0x7ff06815d3e0 2023-11-28T21:51:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.200+0000 7ff065d9b640 1 -- 172.21.15.45:0/2251022270 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058031b30 con 0x7ff06815d3e0 2023-11-28T21:51:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.200+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2251022270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815d3e0 msgr2=0x7ff06806a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.200+0000 7ff06ce5c640 1 --2- 172.21.15.45:0/2251022270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815d3e0 0x7ff06806a500 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff0580098a0 tx=0x7ff05802b8f0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.201+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2251022270 shutdown_connections 2023-11-28T21:51:30.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.201+0000 7ff06ce5c640 1 --2- 172.21.15.45:0/2251022270 >> v2:172.21.15.45:3300/0 conn(0x7ff0680ce6d0 0x7ff0680d1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.201+0000 7ff06ce5c640 1 --2- 172.21.15.45:0/2251022270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff06815d3e0 0x7ff06806a500 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.201+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2251022270 >> 172.21.15.45:0/2251022270 conn(0x7ff0680cfc70 msgr2=0x7ff0680d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:30.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.201+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2251022270 shutdown_connections 2023-11-28T21:51:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.201+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2251022270 wait complete. 2023-11-28T21:51:30.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.203+0000 7ff06ce5c640 1 Processor -- start 2023-11-28T21:51:30.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.203+0000 7ff06ce5c640 1 -- start start 2023-11-28T21:51:30.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.204+0000 7ff06ce5c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680715a0 0x7ff0681564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.204+0000 7ff06ce5c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0680ce6d0 0x7ff068156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.204+0000 7ff06ce5c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff06815d3e0 0x7ff0681529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.204+0000 7ff06ce5c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06815fa50 con 0x7ff06815d3e0 2023-11-28T21:51:30.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.204+0000 7ff06ce5c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff068154720 con 0x7ff0680ce6d0 2023-11-28T21:51:30.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.204+0000 7ff06ce5c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06815f390 con 0x7ff0680715a0 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff066d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff06815d3e0 0x7ff0681529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff066d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff06815d3e0 0x7ff0681529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32802/0 (socket says 172.21.15.45:32802) 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff066d9d640 1 -- 172.21.15.45:0/2748535416 learned_addr learned my addr 172.21.15.45:0/2748535416 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff06659c640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680715a0 0x7ff0681564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff05ffff640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0680ce6d0 0x7ff068156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff066d9d640 1 -- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680715a0 msgr2=0x7ff0681564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff066d9d640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680715a0 0x7ff0681564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff066d9d640 1 -- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0680ce6d0 msgr2=0x7ff068156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff066d9d640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0680ce6d0 0x7ff068156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff066d9d640 1 -- 172.21.15.45:0/2748535416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0580094d0 con 0x7ff06815d3e0 2023-11-28T21:51:30.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.205+0000 7ff066d9d640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff06815d3e0 0x7ff0681529f0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7ff060009fd0 tx=0x7ff06000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:30.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.206+0000 7ff05effd640 1 -- 172.21.15.45:0/2748535416 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060014420 con 0x7ff06815d3e0 2023-11-28T21:51:30.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.206+0000 7ff05effd640 1 -- 172.21.15.45:0/2748535416 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff060014580 con 0x7ff06815d3e0 2023-11-28T21:51:30.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.206+0000 7ff05effd640 1 -- 172.21.15.45:0/2748535416 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060014930 con 0x7ff06815d3e0 2023-11-28T21:51:30.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.206+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06815f670 con 0x7ff06815d3e0 2023-11-28T21:51:30.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.206+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff068153f60 con 0x7ff06815d3e0 2023-11-28T21:51:30.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.207+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06815fb90 con 0x7ff06815d3e0 2023-11-28T21:51:30.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.208+0000 7ff05effd640 1 -- 172.21.15.45:0/2748535416 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 89) v1 ==== 89277+0+0 (secure 0 0 0) 0x7ff060014a90 con 0x7ff06815d3e0 2023-11-28T21:51:30.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.209+0000 7ff05effd640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff04806aed0 0x7ff04806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.209+0000 7ff05effd640 1 -- 172.21.15.45:0/2748535416 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6789+0+0 (secure 0 0 0) 0x7ff06008a3e0 con 0x7ff06815d3e0 2023-11-28T21:51:30.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.210+0000 7ff06659c640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff04806aed0 0x7ff04806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.211+0000 7ff06659c640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff04806aed0 0x7ff04806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff058036000 tx=0x7ff05803c040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:30.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.214+0000 7ff05effd640 1 -- 172.21.15.45:0/2748535416 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff06815fb90 con 0x7ff06815d3e0 2023-11-28T21:51:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.438+0000 7ff05effd640 1 -- 172.21.15.45:0/2748535416 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff0600553c0 con 0x7ff06815d3e0 2023-11-28T21:51:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.455+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff06815fb90 con 0x7ff06815d3e0 2023-11-28T21:51:30.458 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T21:51:30.456+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T21:51:30.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.459+0000 7ff05effd640 1 -- 172.21.15.45:0/2748535416 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v77) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff06815fb90 con 0x7ff06815d3e0 2023-11-28T21:51:30.461 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T21:51:30.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff04806aed0 msgr2=0x7ff04806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff04806aed0 0x7ff04806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff058036000 tx=0x7ff05803c040 comp rx=0 tx=0).stop 2023-11-28T21:51:30.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff06815d3e0 msgr2=0x7ff0681529f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff06815d3e0 0x7ff0681529f0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7ff060009fd0 tx=0x7ff06000e4b0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 shutdown_connections 2023-11-28T21:51:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff04806aed0 0x7ff04806d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff06815d3e0 0x7ff0681529f0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0680ce6d0 0x7ff068156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 --2- 172.21.15.45:0/2748535416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0680715a0 0x7ff0681564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 >> 172.21.15.45:0/2748535416 conn(0x7ff0680cfc70 msgr2=0x7ff0680b6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 shutdown_connections 2023-11-28T21:51:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.463+0000 7ff06ce5c640 1 -- 172.21.15.45:0/2748535416 wait complete. 2023-11-28T21:51:30.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T21:51:30.536 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T21:51:30.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.685+0000 7f7f00d27640 1 Processor -- start 2023-11-28T21:51:30.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.686+0000 7f7f00d27640 1 -- start start 2023-11-28T21:51:30.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.687+0000 7f7f00d27640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7efc0d1fe0 0x7f7efc0d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.687+0000 7f7f00d27640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc0b2930 con 0x7f7efc0cf940 2023-11-28T21:51:30.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.687+0000 7f7f00d27640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc0b2650 con 0x7f7efc0cfd20 2023-11-28T21:51:30.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.687+0000 7f7f00d27640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7efc0ce240 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.688+0000 7f7efb7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7efc0cf940 0x7f7efc161800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33154/0 (socket says 172.21.15.45:33154) 2023-11-28T21:51:30.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.688+0000 7f7efb7fe640 1 -- 172.21.15.45:0/2146475276 learned_addr learned my addr 172.21.15.45:0/2146475276 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:30.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.688+0000 7f7efbfff640 1 --2- 172.21.15.45:0/2146475276 >> v2:172.21.15.45:3300/0 conn(0x7f7efc0d1fe0 0x7f7efc0d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.688+0000 7f7efaffd640 1 -- 172.21.15.45:0/2146475276 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90736277 0 0) 0x7f7efc0b2930 con 0x7f7efc0cf940 2023-11-28T21:51:30.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.688+0000 7f7efaffd640 1 -- 172.21.15.45:0/2146475276 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee8003520 con 0x7f7efc0cf940 2023-11-28T21:51:30.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.689+0000 7f7efbfff640 1 -- 172.21.15.45:0/2146475276 >> v1:172.21.15.190:6789/0 conn(0x7f7efc0cfd20 legacy=0x7f7efc0d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.689+0000 7f7efbfff640 1 -- 172.21.15.45:0/2146475276 >> v1:172.21.15.45:6789/0 conn(0x7f7efc0cf940 legacy=0x7f7efc161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.689+0000 7f7efbfff640 1 -- 172.21.15.45:0/2146475276 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc0ce520 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.690+0000 7f7efbfff640 1 --2- 172.21.15.45:0/2146475276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 0x7f7efc0d2470 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40097a0 tx=0x7f7ef402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0cc1f74cf41445 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:30.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.690+0000 7f7efaffd640 1 -- 172.21.15.45:0/2146475276 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef402e070 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.690+0000 7f7efaffd640 1 -- 172.21.15.45:0/2146475276 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef40315a0 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.690+0000 7f7efaffd640 1 -- 172.21.15.45:0/2146475276 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4031930 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.691+0000 7f7f00d27640 1 -- 172.21.15.45:0/2146475276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 msgr2=0x7f7efc0d2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.691+0000 7f7f00d27640 1 --2- 172.21.15.45:0/2146475276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 0x7f7efc0d2470 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40097a0 tx=0x7f7ef402b840 comp rx=0 tx=0).stop 2023-11-28T21:51:30.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.691+0000 7f7f00d27640 1 -- 172.21.15.45:0/2146475276 shutdown_connections 2023-11-28T21:51:30.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.691+0000 7f7f00d27640 1 --2- 172.21.15.45:0/2146475276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 0x7f7efc0d2470 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.691+0000 7f7f00d27640 1 -- 172.21.15.45:0/2146475276 >> 172.21.15.45:0/2146475276 conn(0x7f7efc06a500 msgr2=0x7f7efc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:30.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.691+0000 7f7f00d27640 1 -- 172.21.15.45:0/2146475276 shutdown_connections 2023-11-28T21:51:30.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.691+0000 7f7f00d27640 1 -- 172.21.15.45:0/2146475276 wait complete. 2023-11-28T21:51:30.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.693+0000 7f7f00d27640 1 Processor -- start 2023-11-28T21:51:30.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.694+0000 7f7f00d27640 1 -- start start 2023-11-28T21:51:30.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.695+0000 7f7f00d27640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7efc0cf940 0x7f7efc1f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.695+0000 7f7f00d27640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7efc0cfd20 0x7f7efc0ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.695+0000 7f7f00d27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 0x7f7efc0cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.695+0000 7f7f00d27640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc1532e0 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.695+0000 7f7f00d27640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc153530 con 0x7f7efc0cfd20 2023-11-28T21:51:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.695+0000 7f7f00d27640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc153810 con 0x7f7efc0cf940 2023-11-28T21:51:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.695+0000 7f7efbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 0x7f7efc0cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.695+0000 7f7efbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 0x7f7efc0cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32858/0 (socket says 172.21.15.45:32858) 2023-11-28T21:51:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.695+0000 7f7efbfff640 1 -- 172.21.15.45:0/1991100374 learned_addr learned my addr 172.21.15.45:0/1991100374 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:30.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7efb7fe640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7efc0cf940 0x7f7efc1f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7ef2ffd640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7efc0cfd20 0x7f7efc0ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7efbfff640 1 -- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7efc0cf940 msgr2=0x7f7efc1f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7efbfff640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7efc0cf940 0x7f7efc1f2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7efbfff640 1 -- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7efc0cfd20 msgr2=0x7f7efc0ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:30.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7efbfff640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7efc0cfd20 0x7f7efc0ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:30.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7efbfff640 1 -- 172.21.15.45:0/1991100374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4009400 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7ef2ffd640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7efc0cfd20 0x7f7efc0ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7efbfff640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 0x7f7efc0cab00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4034040 tx=0x7f7ef40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7ef8ff9640 1 -- 172.21.15.45:0/1991100374 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4032840 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7ef8ff9640 1 -- 172.21.15.45:0/1991100374 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef40329a0 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7ef8ff9640 1 -- 172.21.15.45:0/1991100374 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4032d50 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.696+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7efc153af0 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.697+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7efc1f2fb0 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.698+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7efc154010 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.699+0000 7f7ef8ff9640 1 -- 172.21.15.45:0/1991100374 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7ef404d3f0 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.700+0000 7f7ef8ff9640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7edc06af60 0x7f7edc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:30.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.700+0000 7f7ef8ff9640 1 -- 172.21.15.45:0/1991100374 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6875+0+0 (secure 0 0 0) 0x7f7ef40ab840 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.701+0000 7f7efb7fe640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7edc06af60 0x7f7edc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:30.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.702+0000 7f7efb7fe640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7edc06af60 0x7f7edc06d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7eec001560 tx=0x7f7eec009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.704+0000 7f7ef8ff9640 1 -- 172.21.15.45:0/1991100374 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7efc154010 con 0x7f7efc0d1fe0 2023-11-28T21:51:30.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:30.950+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7efc154010 con 0x7f7efc0d1fe0 2023-11-28T21:51:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.463+0000 7f7ef8ff9640 1 -- 172.21.15.45:0/1991100374 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v78) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7efc154010 con 0x7f7efc0d1fe0 2023-11-28T21:51:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7edc06af60 msgr2=0x7f7edc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7edc06af60 0x7f7edc06d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7eec001560 tx=0x7f7eec009040 comp rx=0 tx=0).stop 2023-11-28T21:51:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 msgr2=0x7f7efc0cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 0x7f7efc0cab00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4034040 tx=0x7f7ef40046c0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 shutdown_connections 2023-11-28T21:51:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f7edc06af60 0x7f7edc06d420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7efc0d1fe0 0x7f7efc0cab00 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7efc0cfd20 0x7f7efc0ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 --2- 172.21.15.45:0/1991100374 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7efc0cf940 0x7f7efc1f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.466+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 >> 172.21.15.45:0/1991100374 conn(0x7f7efc06a500 msgr2=0x7f7efc0d23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.467+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 shutdown_connections 2023-11-28T21:51:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.467+0000 7f7f00d27640 1 -- 172.21.15.45:0/1991100374 wait complete. 2023-11-28T21:51:31.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T21:51:31.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:51:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.711+0000 7f522ab93640 1 Processor -- start 2023-11-28T21:51:31.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:51:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.712+0000 7f522ab93640 1 -- start start 2023-11-28T21:51:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.713+0000 7f522ab93640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5224162820 0x7f52240cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.713+0000 7f522ab93640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5224071a30 0x7f52240cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.713+0000 7f522ab93640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52240cd260 con 0x7f52240cbc00 2023-11-28T21:51:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.713+0000 7f522ab93640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52240cc080 con 0x7f5224071a30 2023-11-28T21:51:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.713+0000 7f522ab93640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5224071ec0 con 0x7f5224162820 2023-11-28T21:51:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.714+0000 7f522a392640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5224071a30 0x7f52240cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.714+0000 7f522a392640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5224071a30 0x7f52240cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32868/0 (socket says 172.21.15.45:32868) 2023-11-28T21:51:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.714+0000 7f522a392640 1 -- 172.21.15.45:0/195247250 learned_addr learned my addr 172.21.15.45:0/195247250 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.714+0000 7f5229390640 1 --1- 172.21.15.45:0/195247250 >> v1:172.21.15.45:6789/0 conn(0x7f52240cbc00 0x7f5224071320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33158/0 (socket says 172.21.15.45:33158) 2023-11-28T21:51:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.714+0000 7f5229b91640 1 --2- 172.21.15.45:0/195247250 >> v2:172.21.15.190:3300/0 conn(0x7f5224162820 0x7f52240cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.714+0000 7f522a392640 1 -- 172.21.15.45:0/195247250 >> v2:172.21.15.190:3300/0 conn(0x7f5224162820 msgr2=0x7f52240cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:31.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.714+0000 7f522a392640 1 --2- 172.21.15.45:0/195247250 >> v2:172.21.15.190:3300/0 conn(0x7f5224162820 0x7f52240cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.714+0000 7f522a392640 1 -- 172.21.15.45:0/195247250 >> v1:172.21.15.45:6789/0 conn(0x7f52240cbc00 legacy=0x7f5224071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:31.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.714+0000 7f522a392640 1 -- 172.21.15.45:0/195247250 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5224072110 con 0x7f5224071a30 2023-11-28T21:51:31.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.716+0000 7f522a392640 1 --2- 172.21.15.45:0/195247250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224071a30 0x7f52240cd6f0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f52240b1a20 tx=0x7f521c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1aa87f4ff33f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:31.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.716+0000 7f5228b8f640 1 -- 172.21.15.45:0/195247250 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c030860 con 0x7f5224071a30 2023-11-28T21:51:31.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.716+0000 7f5228b8f640 1 -- 172.21.15.45:0/195247250 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c0309c0 con 0x7f5224071a30 2023-11-28T21:51:31.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.716+0000 7f5228b8f640 1 -- 172.21.15.45:0/195247250 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c030d50 con 0x7f5224071a30 2023-11-28T21:51:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.717+0000 7f522ab93640 1 -- 172.21.15.45:0/195247250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224071a30 msgr2=0x7f52240cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.717+0000 7f522ab93640 1 --2- 172.21.15.45:0/195247250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224071a30 0x7f52240cd6f0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f52240b1a20 tx=0x7f521c02b9a0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.717+0000 7f522ab93640 1 -- 172.21.15.45:0/195247250 shutdown_connections 2023-11-28T21:51:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.717+0000 7f522ab93640 1 --2- 172.21.15.45:0/195247250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224071a30 0x7f52240cd6f0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.717+0000 7f522ab93640 1 --2- 172.21.15.45:0/195247250 >> v2:172.21.15.190:3300/0 conn(0x7f5224162820 0x7f52240cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.717+0000 7f522ab93640 1 -- 172.21.15.45:0/195247250 >> 172.21.15.45:0/195247250 conn(0x7f52240cec70 msgr2=0x7f52240cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.717+0000 7f522ab93640 1 -- 172.21.15.45:0/195247250 shutdown_connections 2023-11-28T21:51:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.717+0000 7f522ab93640 1 -- 172.21.15.45:0/195247250 wait complete. 2023-11-28T21:51:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.719+0000 7f522ab93640 1 Processor -- start 2023-11-28T21:51:31.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.719+0000 7f522ab93640 1 -- start start 2023-11-28T21:51:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.720+0000 7f522ab93640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5224071a30 0x7f52240b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.720+0000 7f522ab93640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52240cbc00 0x7f5224152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.720+0000 7f522ab93640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224162820 0x7f5224163c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.720+0000 7f522ab93640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52241fbaa0 con 0x7f5224162820 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.720+0000 7f522ab93640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52241fb7c0 con 0x7f52240cbc00 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.720+0000 7f522ab93640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52240b2d90 con 0x7f5224071a30 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f522a392640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224162820 0x7f5224163c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f522a392640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224162820 0x7f5224163c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32876/0 (socket says 172.21.15.45:32876) 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f522a392640 1 -- 172.21.15.45:0/1000550416 learned_addr learned my addr 172.21.15.45:0/1000550416 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f5229b91640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5224071a30 0x7f52240b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f522a392640 1 -- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5224071a30 msgr2=0x7f52240b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f5229390640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52240cbc00 0x7f5224152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f522a392640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5224071a30 0x7f52240b3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f522a392640 1 -- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52240cbc00 msgr2=0x7f5224152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f522a392640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52240cbc00 0x7f5224152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f522a392640 1 -- 172.21.15.45:0/1000550416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c0094d0 con 0x7f5224162820 2023-11-28T21:51:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f5229390640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52240cbc00 0x7f5224152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.721+0000 7f522a392640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224162820 0x7f5224163c50 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f521c0090b0 tx=0x7f521c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.722+0000 7f52227fc640 1 -- 172.21.15.45:0/1000550416 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c039d50 con 0x7f5224162820 2023-11-28T21:51:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.722+0000 7f52227fc640 1 -- 172.21.15.45:0/1000550416 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c043400 con 0x7f5224162820 2023-11-28T21:51:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.722+0000 7f52227fc640 1 -- 172.21.15.45:0/1000550416 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0437b0 con 0x7f5224162820 2023-11-28T21:51:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.722+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52240b3070 con 0x7f5224162820 2023-11-28T21:51:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.722+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52241532f0 con 0x7f5224162820 2023-11-28T21:51:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.723+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52240b2bf0 con 0x7f5224162820 2023-11-28T21:51:31.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.725+0000 7f52227fc640 1 -- 172.21.15.45:0/1000550416 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f521c043910 con 0x7f5224162820 2023-11-28T21:51:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.726+0000 7f52227fc640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f520806af60 0x7f520806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:31.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.726+0000 7f52227fc640 1 -- 172.21.15.45:0/1000550416 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6875+0+0 (secure 0 0 0) 0x7f521c03e070 con 0x7f5224162820 2023-11-28T21:51:31.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.727+0000 7f5229b91640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f520806af60 0x7f520806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.728+0000 7f5229b91640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f520806af60 0x7f520806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5218009e90 tx=0x7f521800b040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.731+0000 7f52227fc640 1 -- 172.21.15.45:0/1000550416 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52240b2bf0 con 0x7f5224162820 2023-11-28T21:51:31.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:31.929+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52240b3590 con 0x7f5224162820 2023-11-28T21:51:32.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.473+0000 7f52227fc640 1 -- 172.21.15.45:0/1000550416 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v175) v1 ==== 181+0+0 (secure 0 0 0) 0x7f52240b3590 con 0x7f5224162820 2023-11-28T21:51:32.476 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T21:51:32.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f520806af60 msgr2=0x7f520806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:32.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f520806af60 0x7f520806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5218009e90 tx=0x7f521800b040 comp rx=0 tx=0).stop 2023-11-28T21:51:32.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224162820 msgr2=0x7f5224163c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:32.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224162820 0x7f5224163c50 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f521c0090b0 tx=0x7f521c004730 comp rx=0 tx=0).stop 2023-11-28T21:51:32.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 shutdown_connections 2023-11-28T21:51:32.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f520806af60 0x7f520806d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5224162820 0x7f5224163c50 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52240cbc00 0x7f5224152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 --2- 172.21.15.45:0/1000550416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5224071a30 0x7f52240b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.477+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 >> 172.21.15.45:0/1000550416 conn(0x7f52240cec70 msgr2=0x7f52240ac430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:32.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.478+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 shutdown_connections 2023-11-28T21:51:32.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.479+0000 7f522ab93640 1 -- 172.21.15.45:0/1000550416 wait complete. 2023-11-28T21:51:32.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T21:51:32.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.712+0000 7f4824893640 1 Processor -- start 2023-11-28T21:51:32.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.712+0000 7f4824893640 1 -- start start 2023-11-28T21:51:32.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.713+0000 7f4824893640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f482015d3b0 0x7f48200d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:32.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.713+0000 7f4824893640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48200d26e0 0x7f48200cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:32.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.713+0000 7f4824893640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48200ce7f0 con 0x7f48200ce340 2023-11-28T21:51:32.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.713+0000 7f4824893640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482015d790 con 0x7f482015d3b0 2023-11-28T21:51:32.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.713+0000 7f4824893640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48200d2bf0 con 0x7f48200d26e0 2023-11-28T21:51:32.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.714+0000 7f481f59e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f48200ce340 0x7f48200cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33164/0 (socket says 172.21.15.45:33164) 2023-11-28T21:51:32.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.714+0000 7f481f59e640 1 -- 172.21.15.45:0/958199872 learned_addr learned my addr 172.21.15.45:0/958199872 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.714+0000 7f481659c640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48200d26e0 0x7f48200cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.714+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816334249 0 0) 0x7f48200ce7f0 con 0x7f48200ce340 2023-11-28T21:51:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.714+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480c003520 con 0x7f48200ce340 2023-11-28T21:51:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.714+0000 7f481ed9d640 1 --2- 172.21.15.45:0/958199872 >> v2:172.21.15.190:3300/0 conn(0x7f482015d3b0 0x7f48200d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1635812927 0 0) 0x7f480c003520 con 0x7f48200ce340 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48200d26e0 msgr2=0x7f48200cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48200d26e0 0x7f48200cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 >> v2:172.21.15.190:3300/0 conn(0x7f482015d3b0 msgr2=0x7f48200d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 --2- 172.21.15.45:0/958199872 >> v2:172.21.15.190:3300/0 conn(0x7f482015d3b0 0x7f48200d21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48200d2ed0 con 0x7f48200ce340 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f48180019f0 con 0x7f48200ce340 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 >> v1:172.21.15.45:6789/0 conn(0x7f48200ce340 legacy=0x7f48200cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f480c003970 0x7f480c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48200d26e0 0x7f480c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480c004d00 0x7f480c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482015d790 con 0x7f480c004d00 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48200d2bf0 con 0x7f480c003970 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480c00b980 con 0x7f48200d26e0 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.715+0000 7f481f59e640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480c004d00 0x7f480c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.716+0000 7f481659c640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48200d26e0 0x7f480c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.716+0000 7f481ed9d640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f480c003970 0x7f480c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.716+0000 7f481f59e640 1 -- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48200d26e0 msgr2=0x7f480c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.716+0000 7f481f59e640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48200d26e0 0x7f480c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.716+0000 7f481f59e640 1 -- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f480c003970 msgr2=0x7f480c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.716+0000 7f481f59e640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f480c003970 0x7f480c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.716+0000 7f481f59e640 1 -- 172.21.15.45:0/958199872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480c003520 con 0x7f480c004d00 2023-11-28T21:51:32.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.717+0000 7f481f59e640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480c004d00 0x7f480c005180 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f481800a860 tx=0x7f480c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99da334ea156ab32 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.717+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818034630 con 0x7f480c004d00 2023-11-28T21:51:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.717+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481802f070 con 0x7f480c004d00 2023-11-28T21:51:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.717+0000 7f481e59c640 1 -- 172.21.15.45:0/958199872 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48180349f0 con 0x7f480c004d00 2023-11-28T21:51:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.718+0000 7f4824893640 1 -- 172.21.15.45:0/958199872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480c004d00 msgr2=0x7f480c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.718+0000 7f4824893640 1 --2- 172.21.15.45:0/958199872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f480c004d00 0x7f480c005180 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f481800a860 tx=0x7f480c0057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.718+0000 7f481f59e640 1 -- 172.21.15.45:0/958199872 reap_dead start 2023-11-28T21:51:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.718+0000 7f4824893640 1 -- 172.21.15.45:0/958199872 shutdown_connections 2023-11-28T21:51:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.718+0000 7f4824893640 1 -- 172.21.15.45:0/958199872 >> 172.21.15.45:0/958199872 conn(0x7f4820070f40 msgr2=0x7f4820071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.718+0000 7f4824893640 1 -- 172.21.15.45:0/958199872 shutdown_connections 2023-11-28T21:51:32.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.718+0000 7f4824893640 1 -- 172.21.15.45:0/958199872 wait complete. 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.720+0000 7f4824893640 1 Processor -- start 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.720+0000 7f4824893640 1 -- start start 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.721+0000 7f4824893640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f480c004d00 0x7f48201f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.721+0000 7f4824893640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48200cabf0 0x7f48200cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.721+0000 7f4824893640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48200c6870 0x7f48200c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.721+0000 7f4824893640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48200c7230 con 0x7f48200cabf0 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.721+0000 7f4824893640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48200cc250 con 0x7f48200c6870 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.721+0000 7f4824893640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48200cb640 con 0x7f480c004d00 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.721+0000 7f481659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48200cabf0 0x7f48200cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:32.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.721+0000 7f481659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48200cabf0 0x7f48200cb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32886/0 (socket says 172.21.15.45:32886) 2023-11-28T21:51:32.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.721+0000 7f481659c640 1 -- 172.21.15.45:0/1022562273 learned_addr learned my addr 172.21.15.45:0/1022562273 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:32.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f481659c640 1 -- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f480c004d00 msgr2=0x7f48201f2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:32.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f481659c640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f480c004d00 0x7f48201f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f481659c640 1 -- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48200c6870 msgr2=0x7f48200c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:32.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f481659c640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48200c6870 0x7f48200c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:32.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f481659c640 1 -- 172.21.15.45:0/1022562273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48200cb920 con 0x7f48200cabf0 2023-11-28T21:51:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f481659c640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48200cabf0 0x7f48200cb050 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f4808011560 tx=0x7f4808011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f4817fff640 1 -- 172.21.15.45:0/1022562273 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808004220 con 0x7f48200cabf0 2023-11-28T21:51:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f4817fff640 1 -- 172.21.15.45:0/1022562273 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4808004380 con 0x7f48200cabf0 2023-11-28T21:51:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f4817fff640 1 -- 172.21.15.45:0/1022562273 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808015870 con 0x7f48200cabf0 2023-11-28T21:51:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.722+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48200cbc00 con 0x7f48200cabf0 2023-11-28T21:51:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.723+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4820152790 con 0x7f48200cabf0 2023-11-28T21:51:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.723+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48200733d0 con 0x7f48200cabf0 2023-11-28T21:51:32.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.725+0000 7f4817fff640 1 -- 172.21.15.45:0/1022562273 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f48080159d0 con 0x7f48200cabf0 2023-11-28T21:51:32.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.726+0000 7f4817fff640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f480006af60 0x7f480006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:32.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.726+0000 7f4817fff640 1 -- 172.21.15.45:0/1022562273 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f4808089c90 con 0x7f48200cabf0 2023-11-28T21:51:32.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.726+0000 7f481ed9d640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f480006af60 0x7f480006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:32.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.728+0000 7f481ed9d640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f480006af60 0x7f480006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4810009910 tx=0x7f4810008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:32.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.730+0000 7f4817fff640 1 -- 172.21.15.45:0/1022562273 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4808054850 con 0x7f48200cabf0 2023-11-28T21:51:32.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:32.923+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f48200cc6e0 con 0x7f48200cabf0 2023-11-28T21:51:33.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.484+0000 7f4817fff640 1 -- 172.21.15.45:0/1022562273 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v176) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4808022020 con 0x7f48200cabf0 2023-11-28T21:51:33.487 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T21:51:33.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.489+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f480006af60 msgr2=0x7f480006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:33.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.489+0000 7f4824893640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f480006af60 0x7f480006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4810009910 tx=0x7f4810008040 comp rx=0 tx=0).stop 2023-11-28T21:51:33.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.489+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48200cabf0 msgr2=0x7f48200cb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:33.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.489+0000 7f4824893640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48200cabf0 0x7f48200cb050 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f4808011560 tx=0x7f4808011a20 comp rx=0 tx=0).stop 2023-11-28T21:51:33.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.491+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 shutdown_connections 2023-11-28T21:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.491+0000 7f4824893640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f480006af60 0x7f480006d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.491+0000 7f4824893640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48200c6870 0x7f48200c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.491+0000 7f4824893640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48200cabf0 0x7f48200cb050 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.491+0000 7f4824893640 1 --2- 172.21.15.45:0/1022562273 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f480c004d00 0x7f48201f2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.491+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 >> 172.21.15.45:0/1022562273 conn(0x7f4820070f40 msgr2=0x7f482015c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.493+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 shutdown_connections 2023-11-28T21:51:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.493+0000 7f4824893640 1 -- 172.21.15.45:0/1022562273 wait complete. 2023-11-28T21:51:33.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T21:51:33.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.736+0000 7eff2f59e640 1 Processor -- start 2023-11-28T21:51:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.736+0000 7eff2f59e640 1 -- start start 2023-11-28T21:51:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.738+0000 7eff2f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff3015d230 0x7eff300b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.738+0000 7eff2f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 0x7eff300ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.738+0000 7eff2f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff300ce780 con 0x7eff300b2640 2023-11-28T21:51:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.738+0000 7eff2f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff300b90e0 con 0x7eff3015d230 2023-11-28T21:51:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.738+0000 7eff2f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff300d11d0 con 0x7eff300cddf0 2023-11-28T21:51:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.739+0000 7eff2e59c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff3015d230 0x7eff300b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.739+0000 7eff2ed9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 0x7eff300ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.739+0000 7eff2e59c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff3015d230 0x7eff300b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:55694/0 (socket says 172.21.15.45:55694) 2023-11-28T21:51:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.739+0000 7eff2e59c640 1 -- 172.21.15.45:0/464670322 learned_addr learned my addr 172.21.15.45:0/464670322 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.740+0000 7eff2e59c640 1 -- 172.21.15.45:0/464670322 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 msgr2=0x7eff300ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.740+0000 7eff2e59c640 1 --2- 172.21.15.45:0/464670322 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 0x7eff300ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.740+0000 7eff2e59c640 1 -- 172.21.15.45:0/464670322 >> v1:172.21.15.190:6789/0 conn(0x7eff300b2640 legacy=0x7eff300b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.740+0000 7eff2e59c640 1 -- 172.21.15.45:0/464670322 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff300d1490 con 0x7eff3015d230 2023-11-28T21:51:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.741+0000 7eff2e59c640 1 --2- 172.21.15.45:0/464670322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 0x7eff300b61e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7eff20009870 tx=0x7eff2002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3350085ac99e039 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:33.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.742+0000 7eff2dd9b640 1 -- 172.21.15.45:0/464670322 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20030840 con 0x7eff3015d230 2023-11-28T21:51:33.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.742+0000 7eff2dd9b640 1 -- 172.21.15.45:0/464670322 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff200309a0 con 0x7eff3015d230 2023-11-28T21:51:33.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.742+0000 7eff2dd9b640 1 -- 172.21.15.45:0/464670322 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20030d30 con 0x7eff3015d230 2023-11-28T21:51:33.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.742+0000 7eff2f59e640 1 -- 172.21.15.45:0/464670322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 msgr2=0x7eff300b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:33.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.742+0000 7eff2f59e640 1 --2- 172.21.15.45:0/464670322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 0x7eff300b61e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7eff20009870 tx=0x7eff2002b980 comp rx=0 tx=0).stop 2023-11-28T21:51:33.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.742+0000 7eff2f59e640 1 -- 172.21.15.45:0/464670322 shutdown_connections 2023-11-28T21:51:33.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.742+0000 7eff2f59e640 1 --2- 172.21.15.45:0/464670322 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 0x7eff300ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.742+0000 7eff2f59e640 1 --2- 172.21.15.45:0/464670322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 0x7eff300b61e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.742+0000 7eff2f59e640 1 -- 172.21.15.45:0/464670322 >> 172.21.15.45:0/464670322 conn(0x7eff30070f40 msgr2=0x7eff30071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:33.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.743+0000 7eff2f59e640 1 -- 172.21.15.45:0/464670322 shutdown_connections 2023-11-28T21:51:33.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.743+0000 7eff2f59e640 1 -- 172.21.15.45:0/464670322 wait complete. 2023-11-28T21:51:33.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.745+0000 7eff2f59e640 1 Processor -- start 2023-11-28T21:51:33.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.745+0000 7eff2f59e640 1 -- start start 2023-11-28T21:51:33.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.746+0000 7eff2f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff300b2640 0x7eff300c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:33.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 0x7eff3015d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:33.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 0x7eff301faed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:33.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff301603c0 con 0x7eff300b2640 2023-11-28T21:51:33.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff301606a0 con 0x7eff3015d230 2023-11-28T21:51:33.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff300ca6d0 con 0x7eff300cddf0 2023-11-28T21:51:33.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff300b2640 0x7eff300c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:33.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff300b2640 0x7eff300c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32894/0 (socket says 172.21.15.45:32894) 2023-11-28T21:51:33.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2e59c640 1 -- 172.21.15.45:0/1794540217 learned_addr learned my addr 172.21.15.45:0/1794540217 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:33.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff27fff640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 0x7eff3015d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2ed9d640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 0x7eff301faed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2e59c640 1 -- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 msgr2=0x7eff3015d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2e59c640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 0x7eff3015d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2e59c640 1 -- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 msgr2=0x7eff301faed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2e59c640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 0x7eff301faed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff2e59c640 1 -- 172.21.15.45:0/1794540217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff200094d0 con 0x7eff300b2640 2023-11-28T21:51:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.747+0000 7eff27fff640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 0x7eff3015d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.748+0000 7eff2ed9d640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 0x7eff301faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:33.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.748+0000 7eff2e59c640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff300b2640 0x7eff300c96e0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7eff20009870 tx=0x7eff20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:33.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.748+0000 7eff26ffd640 1 -- 172.21.15.45:0/1794540217 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20039a90 con 0x7eff300b2640 2023-11-28T21:51:33.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.748+0000 7eff26ffd640 1 -- 172.21.15.45:0/1794540217 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff20039bf0 con 0x7eff300b2640 2023-11-28T21:51:33.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.748+0000 7eff26ffd640 1 -- 172.21.15.45:0/1794540217 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff200433f0 con 0x7eff300b2640 2023-11-28T21:51:33.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.748+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff300ca9b0 con 0x7eff300b2640 2023-11-28T21:51:33.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.748+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff30160000 con 0x7eff300b2640 2023-11-28T21:51:33.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.750+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff30160190 con 0x7eff300b2640 2023-11-28T21:51:33.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.751+0000 7eff26ffd640 1 -- 172.21.15.45:0/1794540217 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7eff20042390 con 0x7eff300b2640 2023-11-28T21:51:33.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.751+0000 7eff26ffd640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7eff1006af60 0x7eff1006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:33.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.752+0000 7eff26ffd640 1 -- 172.21.15.45:0/1794540217 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7eff200ab970 con 0x7eff300b2640 2023-11-28T21:51:33.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.752+0000 7eff27fff640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7eff1006af60 0x7eff1006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:33.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.753+0000 7eff27fff640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7eff1006af60 0x7eff1006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7eff300b4b90 tx=0x7eff18009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:33.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.756+0000 7eff26ffd640 1 -- 172.21.15.45:0/1794540217 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff200766e0 con 0x7eff300b2640 2023-11-28T21:51:33.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.957+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7eff300caed0 con 0x7eff300b2640 2023-11-28T21:51:33.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.963+0000 7eff26ffd640 1 -- 172.21.15.45:0/1794540217 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v55) v1 ==== 120+0+0 (secure 0 0 0) 0x7eff300caed0 con 0x7eff300b2640 2023-11-28T21:51:33.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7eff1006af60 msgr2=0x7eff1006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:33.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7eff1006af60 0x7eff1006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7eff300b4b90 tx=0x7eff18009040 comp rx=0 tx=0).stop 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff300b2640 msgr2=0x7eff300c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff300b2640 0x7eff300c96e0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7eff20009870 tx=0x7eff20004730 comp rx=0 tx=0).stop 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 shutdown_connections 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7eff1006af60 0x7eff1006d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3015d230 0x7eff301faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff300cddf0 0x7eff3015d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 --2- 172.21.15.45:0/1794540217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff300b2640 0x7eff300c96e0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.966+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 >> 172.21.15.45:0/1794540217 conn(0x7eff30070f40 msgr2=0x7eff300b7590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.967+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 shutdown_connections 2023-11-28T21:51:33.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:33.967+0000 7eff2f59e640 1 -- 172.21.15.45:0/1794540217 wait complete. 2023-11-28T21:51:33.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T21:51:34.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.216+0000 7f9a0f59e640 1 Processor -- start 2023-11-28T21:51:34.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.216+0000 7f9a0f59e640 1 -- start start 2023-11-28T21:51:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.218+0000 7f9a0f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9a10071f20 0x7f9a100b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.218+0000 7f9a0f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1006dbe0 con 0x7f9a100b3490 2023-11-28T21:51:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.218+0000 7f9a0f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a100b3870 con 0x7f9a100b2240 2023-11-28T21:51:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.218+0000 7f9a0f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a1006e110 con 0x7f9a10071f20 2023-11-28T21:51:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.218+0000 7f9a0e59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9a100b3490 0x7f9a1015e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39620/0 (socket says 172.21.15.45:39620) 2023-11-28T21:51:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.218+0000 7f9a0e59c640 1 -- 172.21.15.45:0/3081597021 learned_addr learned my addr 172.21.15.45:0/3081597021 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.219+0000 7f9a05d9b640 1 --2- 172.21.15.45:0/3081597021 >> v2:172.21.15.190:3300/0 conn(0x7f9a10071f20 0x7f9a100b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.219+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135869671 0 0) 0x7f9a1006dbe0 con 0x7f9a100b3490 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.219+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99fc003520 con 0x7f9a100b3490 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1442061249 0 0) 0x7f99fc003520 con 0x7f9a100b3490 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 >> v2:172.21.15.190:3300/0 conn(0x7f9a10071f20 msgr2=0x7f9a100b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 --2- 172.21.15.45:0/3081597021 >> v2:172.21.15.190:3300/0 conn(0x7f9a10071f20 0x7f9a100b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 >> v1:172.21.15.190:6789/0 conn(0x7f9a100b2240 legacy=0x7f9a10160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1006de30 con 0x7f9a100b3490 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f9a00001670 con 0x7f9a100b3490 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 >> v1:172.21.15.45:6789/0 conn(0x7f9a100b3490 legacy=0x7f9a1015e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f99fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99fc0047c0 0x7f99fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99fc005180 0x7f99fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1006e110 con 0x7f99fc003970 2023-11-28T21:51:34.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc00c020 con 0x7f99fc005180 2023-11-28T21:51:34.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc00c1e0 con 0x7f99fc0047c0 2023-11-28T21:51:34.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0e59c640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f99fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.220+0000 7f9a0ed9d640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99fc005180 0x7f99fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.221+0000 7f9a05d9b640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99fc0047c0 0x7f99fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.221+0000 7f9a0e59c640 1 -- 172.21.15.45:0/3081597021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99fc0047c0 msgr2=0x7f99fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.221+0000 7f9a0e59c640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99fc0047c0 0x7f99fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.221+0000 7f9a0e59c640 1 -- 172.21.15.45:0/3081597021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99fc005180 msgr2=0x7f99fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.221+0000 7f9a0e59c640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99fc005180 0x7f99fc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.221+0000 7f9a0e59c640 1 -- 172.21.15.45:0/3081597021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc003520 con 0x7f99fc003970 2023-11-28T21:51:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.221+0000 7f9a0ed9d640 1 -- 172.21.15.45:0/3081597021 reap_dead start 2023-11-28T21:51:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.222+0000 7f9a0e59c640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f99fc003d90 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000c240 tx=0x7f99fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15044b3dc5bed122 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.223+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a00005ad0 con 0x7f99fc003970 2023-11-28T21:51:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.223+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a00005c50 con 0x7f99fc003970 2023-11-28T21:51:34.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.223+0000 7f9a0dd9b640 1 -- 172.21.15.45:0/3081597021 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0002c8b0 con 0x7f99fc003970 2023-11-28T21:51:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.223+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3081597021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 msgr2=0x7f99fc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.223+0000 7f9a0f59e640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f99fc003d90 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000c240 tx=0x7f99fc0057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.223+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3081597021 shutdown_connections 2023-11-28T21:51:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.223+0000 7f9a0f59e640 1 --2- 172.21.15.45:0/3081597021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f99fc003d90 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.223+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3081597021 >> 172.21.15.45:0/3081597021 conn(0x7f9a100cecd0 msgr2=0x7f9a100cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.224+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3081597021 shutdown_connections 2023-11-28T21:51:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.224+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3081597021 wait complete. 2023-11-28T21:51:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.226+0000 7f9a0f59e640 1 Processor -- start 2023-11-28T21:51:34.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.226+0000 7f9a0f59e640 1 -- start start 2023-11-28T21:51:34.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.227+0000 7f9a0f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f9a1009b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.227+0000 7f9a0f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a1015c930 0x7f9a1014a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.227+0000 7f9a0f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a1015cd90 0x7f9a1014a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.227+0000 7f9a0f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1009b3b0 con 0x7f99fc003970 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.227+0000 7f9a0f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1009a730 con 0x7f9a1015c930 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.227+0000 7f9a0f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1009aa10 con 0x7f9a1015cd90 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a0e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f9a1009b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a0e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f9a1009b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42020/0 (socket says 172.21.15.45:42020) 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a0e59c640 1 -- 172.21.15.45:0/3470980037 learned_addr learned my addr 172.21.15.45:0/3470980037 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a05d9b640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a1015c930 0x7f9a1014a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a0ed9d640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a1015cd90 0x7f9a1014a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a0e59c640 1 -- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a1015cd90 msgr2=0x7f9a1014a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a0e59c640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a1015cd90 0x7f9a1014a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a0e59c640 1 -- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a1015c930 msgr2=0x7f9a1014a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a0e59c640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a1015c930 0x7f9a1014a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.228+0000 7f9a0e59c640 1 -- 172.21.15.45:0/3470980037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1009acf0 con 0x7f99fc003970 2023-11-28T21:51:34.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.229+0000 7f9a0e59c640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f9a1009b7d0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f9a0002c450 tx=0x7f9a0002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:34.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.229+0000 7f9a077fe640 1 -- 172.21.15.45:0/3470980037 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0003c040 con 0x7f99fc003970 2023-11-28T21:51:34.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.229+0000 7f9a077fe640 1 -- 172.21.15.45:0/3470980037 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0002cd80 con 0x7f99fc003970 2023-11-28T21:51:34.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.229+0000 7f9a077fe640 1 -- 172.21.15.45:0/3470980037 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a00007290 con 0x7f99fc003970 2023-11-28T21:51:34.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.229+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1009afd0 con 0x7f99fc003970 2023-11-28T21:51:34.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.229+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a10099c20 con 0x7f99fc003970 2023-11-28T21:51:34.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.231+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1009d0f0 con 0x7f99fc003970 2023-11-28T21:51:34.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.232+0000 7f9a077fe640 1 -- 172.21.15.45:0/3470980037 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9a00032070 con 0x7f99fc003970 2023-11-28T21:51:34.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.233+0000 7f9a077fe640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f99f006af90 0x7f99f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.233+0000 7f9a077fe640 1 -- 172.21.15.45:0/3470980037 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f9a000aaa70 con 0x7f99fc003970 2023-11-28T21:51:34.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.234+0000 7f9a05d9b640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f99f006af90 0x7f99f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.235+0000 7f9a05d9b640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f99f006af90 0x7f99f006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9a10161380 tx=0x7f99f8008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:34.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.237+0000 7f9a077fe640 1 -- 172.21.15.45:0/3470980037 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a00075860 con 0x7f99fc003970 2023-11-28T21:51:34.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.434+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9a100ad500 con 0x7f99fc003970 2023-11-28T21:51:34.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.441+0000 7f9a077fe640 1 -- 172.21.15.45:0/3470980037 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9a000450a0 con 0x7f99fc003970 2023-11-28T21:51:34.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.444+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f99f006af90 msgr2=0x7f99f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.444+0000 7f9a0f59e640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f99f006af90 0x7f99f006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9a10161380 tx=0x7f99f8008040 comp rx=0 tx=0).stop 2023-11-28T21:51:34.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.444+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 msgr2=0x7f9a1009b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.444+0000 7f9a0f59e640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f9a1009b7d0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f9a0002c450 tx=0x7f9a0002e070 comp rx=0 tx=0).stop 2023-11-28T21:51:34.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.445+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 shutdown_connections 2023-11-28T21:51:34.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.445+0000 7f9a0f59e640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f99f006af90 0x7f99f006d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.445+0000 7f9a0f59e640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a1015cd90 0x7f9a1014a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.445+0000 7f9a0f59e640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a1015c930 0x7f9a1014a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.445+0000 7f9a0f59e640 1 --2- 172.21.15.45:0/3470980037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f99fc003970 0x7f9a1009b7d0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.445+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 >> 172.21.15.45:0/3470980037 conn(0x7f9a100cecd0 msgr2=0x7f9a100afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:34.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.445+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 shutdown_connections 2023-11-28T21:51:34.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.445+0000 7f9a0f59e640 1 -- 172.21.15.45:0/3470980037 wait complete. 2023-11-28T21:51:34.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T21:51:34.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.692+0000 7f9aaa2c3640 1 Processor -- start 2023-11-28T21:51:34.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.692+0000 7f9aaa2c3640 1 -- start start 2023-11-28T21:51:34.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.694+0000 7f9aaa2c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 0x7f9aa40ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.694+0000 7f9aaa2c3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9aa40abd70 0x7f9aa40718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.694+0000 7f9aaa2c3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9aa4071e00 0x7f9aa415eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.694+0000 7f9aaa2c3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa40722b0 con 0x7f9aa4071e00 2023-11-28T21:51:34.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.694+0000 7f9aaa2c3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa40ac270 con 0x7f9aa40abd70 2023-11-28T21:51:34.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.694+0000 7f9aaa2c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa415b830 con 0x7f9aa415b080 2023-11-28T21:51:34.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.694+0000 7f9aa9ac2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9aa4071e00 0x7f9aa415eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.694+0000 7f9aa9ac2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9aa4071e00 0x7f9aa415eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42036/0 (socket says 172.21.15.45:42036) 2023-11-28T21:51:34.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.694+0000 7f9aa9ac2640 1 -- 172.21.15.45:0/468826549 learned_addr learned my addr 172.21.15.45:0/468826549 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:34.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.695+0000 7f9aa92c1640 1 --2- 172.21.15.45:0/468826549 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 0x7f9aa40ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.695+0000 7f9aa8ac0640 1 --2- 172.21.15.45:0/468826549 >> v2:172.21.15.190:3300/0 conn(0x7f9aa40abd70 0x7f9aa40718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.695+0000 7f9aa9ac2640 1 -- 172.21.15.45:0/468826549 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 msgr2=0x7f9aa40ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:34.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.695+0000 7f9aa9ac2640 1 --2- 172.21.15.45:0/468826549 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 0x7f9aa40ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.695+0000 7f9aa9ac2640 1 -- 172.21.15.45:0/468826549 >> v2:172.21.15.190:3300/0 conn(0x7f9aa40abd70 msgr2=0x7f9aa40718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:34.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.695+0000 7f9aa9ac2640 1 --2- 172.21.15.45:0/468826549 >> v2:172.21.15.190:3300/0 conn(0x7f9aa40abd70 0x7f9aa40718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.695+0000 7f9aa9ac2640 1 -- 172.21.15.45:0/468826549 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa415bb10 con 0x7f9aa4071e00 2023-11-28T21:51:34.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.696+0000 7f9aa9ac2640 1 --2- 172.21.15.45:0/468826549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa4071e00 0x7f9aa415eb10 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f9aa406c030 tx=0x7f9a9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b73420bea4037e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:34.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.696+0000 7f9a9bfff640 1 -- 172.21.15.45:0/468826549 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c02ca70 con 0x7f9aa4071e00 2023-11-28T21:51:34.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.696+0000 7f9a9bfff640 1 -- 172.21.15.45:0/468826549 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9c02cbd0 con 0x7f9aa4071e00 2023-11-28T21:51:34.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.696+0000 7f9a9bfff640 1 -- 172.21.15.45:0/468826549 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c0343f0 con 0x7f9aa4071e00 2023-11-28T21:51:34.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.697+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/468826549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa4071e00 msgr2=0x7f9aa415eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.697+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/468826549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa4071e00 0x7f9aa415eb10 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f9aa406c030 tx=0x7f9a9c02bed0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.697+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/468826549 shutdown_connections 2023-11-28T21:51:34.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.697+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/468826549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa4071e00 0x7f9aa415eb10 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.697+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/468826549 >> v2:172.21.15.190:3300/0 conn(0x7f9aa40abd70 0x7f9aa40718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.697+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/468826549 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 0x7f9aa40ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.697+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/468826549 >> 172.21.15.45:0/468826549 conn(0x7f9aa40cecd0 msgr2=0x7f9aa40cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:34.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.697+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/468826549 shutdown_connections 2023-11-28T21:51:34.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.697+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/468826549 wait complete. 2023-11-28T21:51:34.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.699+0000 7f9aaa2c3640 1 Processor -- start 2023-11-28T21:51:34.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.699+0000 7f9aaa2c3640 1 -- start start 2023-11-28T21:51:34.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.700+0000 7f9aaa2c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9aa4071e00 0x7f9aa41f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.700+0000 7f9aaa2c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa40abd70 0x7f9aa4156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.700+0000 7f9aaa2c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 0x7f9aa4144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.700+0000 7f9aaa2c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa41f7b60 con 0x7f9aa40abd70 2023-11-28T21:51:34.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.700+0000 7f9aaa2c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa41f3570 con 0x7f9aa4071e00 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.700+0000 7f9aaa2c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa414f570 con 0x7f9aa415b080 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.700+0000 7f9aa8ac0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa40abd70 0x7f9aa4156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.700+0000 7f9aa8ac0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa40abd70 0x7f9aa4156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42040/0 (socket says 172.21.15.45:42040) 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.700+0000 7f9aa8ac0640 1 -- 172.21.15.45:0/2886034568 learned_addr learned my addr 172.21.15.45:0/2886034568 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9aa92c1640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9aa4071e00 0x7f9aa41f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9aa9ac2640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 0x7f9aa4144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9aa8ac0640 1 -- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 msgr2=0x7f9aa4144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9aa8ac0640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 0x7f9aa4144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9aa8ac0640 1 -- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9aa4071e00 msgr2=0x7f9aa41f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9aa8ac0640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9aa4071e00 0x7f9aa41f7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9aa8ac0640 1 -- 172.21.15.45:0/2886034568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c009b30 con 0x7f9aa40abd70 2023-11-28T21:51:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9aa8ac0640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa40abd70 0x7f9aa4156070 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400d800 tx=0x7f9a9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9a99ffb640 1 -- 172.21.15.45:0/2886034568 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94009640 con 0x7f9aa40abd70 2023-11-28T21:51:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9a99ffb640 1 -- 172.21.15.45:0/2886034568 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a940097a0 con 0x7f9aa40abd70 2023-11-28T21:51:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9a99ffb640 1 -- 172.21.15.45:0/2886034568 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94009b30 con 0x7f9aa40abd70 2023-11-28T21:51:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.701+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa414f810 con 0x7f9aa40abd70 2023-11-28T21:51:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.702+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa414fdd0 con 0x7f9aa40abd70 2023-11-28T21:51:34.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.703+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa40ad1b0 con 0x7f9aa40abd70 2023-11-28T21:51:34.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.704+0000 7f9a99ffb640 1 -- 172.21.15.45:0/2886034568 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9a94009c90 con 0x7f9aa40abd70 2023-11-28T21:51:34.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.704+0000 7f9a99ffb640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f9a8806af90 0x7f9a8806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:34.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.705+0000 7f9a99ffb640 1 -- 172.21.15.45:0/2886034568 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f9a94089f80 con 0x7f9aa40abd70 2023-11-28T21:51:34.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.705+0000 7f9aa92c1640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f9a8806af90 0x7f9a8806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:34.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.706+0000 7f9aa92c1640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f9a8806af90 0x7f9a8806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0006fd0 tx=0x7f9aa0008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:34.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.709+0000 7f9a99ffb640 1 -- 172.21.15.45:0/2886034568 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a94054e00 con 0x7f9aa40abd70 2023-11-28T21:51:34.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.905+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f9aa409d0f0 con 0x7f9aa40abd70 2023-11-28T21:51:34.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.910+0000 7f9a99ffb640 1 -- 172.21.15.45:0/2886034568 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v57) v1 ==== 115+0+0 (secure 0 0 0) 0x7f9a9400b440 con 0x7f9aa40abd70 2023-11-28T21:51:34.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.912+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f9a8806af90 msgr2=0x7f9a8806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.912+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f9a8806af90 0x7f9a8806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0006fd0 tx=0x7f9aa0008040 comp rx=0 tx=0).stop 2023-11-28T21:51:34.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.912+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa40abd70 msgr2=0x7f9aa4156070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:34.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.912+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa40abd70 0x7f9aa4156070 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400d800 tx=0x7f9a9400dcc0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.913+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 shutdown_connections 2023-11-28T21:51:34.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.913+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f9a8806af90 0x7f9a8806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.913+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9aa415b080 0x7f9aa4144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.913+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa40abd70 0x7f9aa4156070 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.913+0000 7f9aaa2c3640 1 --2- 172.21.15.45:0/2886034568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9aa4071e00 0x7f9aa41f7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:34.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.913+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 >> 172.21.15.45:0/2886034568 conn(0x7f9aa40cecd0 msgr2=0x7f9aa40af680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:34.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.913+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 shutdown_connections 2023-11-28T21:51:34.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:34.913+0000 7f9aaa2c3640 1 -- 172.21.15.45:0/2886034568 wait complete. 2023-11-28T21:51:34.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-11-28T21:51:35.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.154+0000 7fd0b50d7640 1 Processor -- start 2023-11-28T21:51:35.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.154+0000 7fd0b50d7640 1 -- start start 2023-11-28T21:51:35.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.156+0000 7fd0b50d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd0b00ae540 0x7fd0b00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.156+0000 7fd0b50d7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd0b00ce630 0x7fd0b00b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.156+0000 7fd0b50d7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b015c3c0 con 0x7fd0b0155780 2023-11-28T21:51:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.156+0000 7fd0b50d7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b00ceac0 con 0x7fd0b00ce630 2023-11-28T21:51:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.156+0000 7fd0b50d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b0155b60 con 0x7fd0b00ae540 2023-11-28T21:51:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.157+0000 7fd0b48d6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd0b00ce630 0x7fd0b00b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.157+0000 7fd0b48d6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd0b00ce630 0x7fd0b00b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42052/0 (socket says 172.21.15.45:42052) 2023-11-28T21:51:35.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.157+0000 7fd0b48d6640 1 -- 172.21.15.45:0/2928259677 learned_addr learned my addr 172.21.15.45:0/2928259677 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.157+0000 7fd0a77fe640 1 --2- 172.21.15.45:0/2928259677 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd0b00ae540 0x7fd0b00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.157+0000 7fd0b48d6640 1 -- 172.21.15.45:0/2928259677 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd0b00ae540 msgr2=0x7fd0b00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.157+0000 7fd0b48d6640 1 --2- 172.21.15.45:0/2928259677 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd0b00ae540 0x7fd0b00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.157+0000 7fd0b48d6640 1 -- 172.21.15.45:0/2928259677 >> v1:172.21.15.190:6789/0 conn(0x7fd0b0155780 legacy=0x7fd0b00abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.157+0000 7fd0b48d6640 1 -- 172.21.15.45:0/2928259677 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b015ba90 con 0x7fd0b00ce630 2023-11-28T21:51:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.158+0000 7fd0b48d6640 1 --2- 172.21.15.45:0/2928259677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ce630 0x7fd0b00b20a0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009e80 tx=0x7fd0a802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f792a9189cf03aaa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.159+0000 7fd0af7fe640 1 -- 172.21.15.45:0/2928259677 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a802caf0 con 0x7fd0b00ce630 2023-11-28T21:51:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.159+0000 7fd0af7fe640 1 -- 172.21.15.45:0/2928259677 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a802cc50 con 0x7fd0b00ce630 2023-11-28T21:51:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.159+0000 7fd0af7fe640 1 -- 172.21.15.45:0/2928259677 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a80343f0 con 0x7fd0b00ce630 2023-11-28T21:51:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.159+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2928259677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ce630 msgr2=0x7fd0b00b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.159+0000 7fd0b50d7640 1 --2- 172.21.15.45:0/2928259677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ce630 0x7fd0b00b20a0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009e80 tx=0x7fd0a802bf50 comp rx=0 tx=0).stop 2023-11-28T21:51:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.159+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2928259677 shutdown_connections 2023-11-28T21:51:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.159+0000 7fd0b50d7640 1 --2- 172.21.15.45:0/2928259677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ce630 0x7fd0b00b20a0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.159+0000 7fd0b50d7640 1 --2- 172.21.15.45:0/2928259677 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd0b00ae540 0x7fd0b00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.159+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2928259677 >> 172.21.15.45:0/2928259677 conn(0x7fd0b006ab30 msgr2=0x7fd0b00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:35.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.160+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2928259677 shutdown_connections 2023-11-28T21:51:35.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.160+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2928259677 wait complete. 2023-11-28T21:51:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.161+0000 7fd0b50d7640 1 Processor -- start 2023-11-28T21:51:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.162+0000 7fd0b50d7640 1 -- start start 2023-11-28T21:51:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0b50d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ae540 0x7fd0b0145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0b50d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd0b00ce630 0x7fd0b006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0b50d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0b0155780 0x7fd0b006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0b50d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b014fc90 con 0x7fd0b00ae540 2023-11-28T21:51:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0b50d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b015d270 con 0x7fd0b0155780 2023-11-28T21:51:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0b50d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b009c0f0 con 0x7fd0b00ce630 2023-11-28T21:51:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0affff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ae540 0x7fd0b0145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0affff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ae540 0x7fd0b0145820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42066/0 (socket says 172.21.15.45:42066) 2023-11-28T21:51:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0affff640 1 -- 172.21.15.45:0/2323576534 learned_addr learned my addr 172.21.15.45:0/2323576534 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0affff640 1 -- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd0b00ce630 msgr2=0x7fd0b006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0affff640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd0b00ce630 0x7fd0b006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0affff640 1 -- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0b0155780 msgr2=0x7fd0b006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0affff640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0b0155780 0x7fd0b006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.163+0000 7fd0affff640 1 -- 172.21.15.45:0/2323576534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a8009b30 con 0x7fd0b00ae540 2023-11-28T21:51:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.164+0000 7fd0affff640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ae540 0x7fd0b0145820 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000c400 tx=0x7fd0a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:35.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.164+0000 7fd0ad7fa640 1 -- 172.21.15.45:0/2323576534 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a00052d0 con 0x7fd0b00ae540 2023-11-28T21:51:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.164+0000 7fd0ad7fa640 1 -- 172.21.15.45:0/2323576534 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a0005430 con 0x7fd0b00ae540 2023-11-28T21:51:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.164+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b009c3d0 con 0x7fd0b00ae540 2023-11-28T21:51:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.164+0000 7fd0ad7fa640 1 -- 172.21.15.45:0/2323576534 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0003830 con 0x7fd0b00ae540 2023-11-28T21:51:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.164+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b009c990 con 0x7fd0b00ae540 2023-11-28T21:51:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.165+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b014fdd0 con 0x7fd0b00ae540 2023-11-28T21:51:35.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.166+0000 7fd0ad7fa640 1 -- 172.21.15.45:0/2323576534 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd0a0003990 con 0x7fd0b00ae540 2023-11-28T21:51:35.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.168+0000 7fd0ad7fa640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fd09006af60 0x7fd09006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.168+0000 7fd0ad7fa640 1 -- 172.21.15.45:0/2323576534 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fd0a0088db0 con 0x7fd0b00ae540 2023-11-28T21:51:35.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.168+0000 7fd0a77fe640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fd09006af60 0x7fd09006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:35.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.169+0000 7fd0a77fe640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fd09006af60 0x7fd09006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00ce460 tx=0x7fd098008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:35.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.172+0000 7fd0ad7fa640 1 -- 172.21.15.45:0/2323576534 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd0b014fdd0 con 0x7fd0b00ae540 2023-11-28T21:51:35.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.436+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7fd0b014fdd0 con 0x7fd0b00ae540 2023-11-28T21:51:35.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.490+0000 7fd0ad7fa640 1 -- 172.21.15.45:0/2323576534 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2545) v1 ==== 178+0+0 (secure 0 0 0) 0x7fd0a00170f0 con 0x7fd0b00ae540 2023-11-28T21:51:35.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.492+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fd09006af60 msgr2=0x7fd09006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.492+0000 7fd0b50d7640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fd09006af60 0x7fd09006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00ce460 tx=0x7fd098008040 comp rx=0 tx=0).stop 2023-11-28T21:51:35.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.492+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ae540 msgr2=0x7fd0b0145820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.492+0000 7fd0b50d7640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ae540 0x7fd0b0145820 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000c400 tx=0x7fd0a000c8c0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.493+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 shutdown_connections 2023-11-28T21:51:35.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.493+0000 7fd0b50d7640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fd09006af60 0x7fd09006d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.493+0000 7fd0b50d7640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0b0155780 0x7fd0b006ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.493+0000 7fd0b50d7640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd0b00ce630 0x7fd0b006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.493+0000 7fd0b50d7640 1 --2- 172.21.15.45:0/2323576534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0b00ae540 0x7fd0b0145820 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.493+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 >> 172.21.15.45:0/2323576534 conn(0x7fd0b006ab30 msgr2=0x7fd0b0153990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:35.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.493+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 shutdown_connections 2023-11-28T21:51:35.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.493+0000 7fd0b50d7640 1 -- 172.21.15.45:0/2323576534 wait complete. 2023-11-28T21:51:35.514 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-11-28T21:51:35.514 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-11-28T21:51:35.514 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T21:51:35.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:51:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.727+0000 7f6396bed640 1 Processor -- start 2023-11-28T21:51:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.727+0000 7f6396bed640 1 -- start start 2023-11-28T21:51:35.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.729+0000 7f6396bed640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6390155780 0x7f63900abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.729+0000 7f6396bed640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f63900ae540 0x7f63900ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.729+0000 7f6396bed640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f639015c3c0 con 0x7f63900ce630 2023-11-28T21:51:35.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.729+0000 7f6396bed640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63900ceac0 con 0x7f63900ae540 2023-11-28T21:51:35.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.729+0000 7f6396bed640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6390155b60 con 0x7f6390155780 2023-11-28T21:51:35.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.729+0000 7f63953ea640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f63900ae540 0x7f63900ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:35.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.729+0000 7f63953ea640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f63900ae540 0x7f63900ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42080/0 (socket says 172.21.15.45:42080) 2023-11-28T21:51:35.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.729+0000 7f63953ea640 1 -- 172.21.15.45:0/808535370 learned_addr learned my addr 172.21.15.45:0/808535370 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:35.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.730+0000 7f63953ea640 1 -- 172.21.15.45:0/808535370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6390155780 msgr2=0x7f63900abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:35.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.730+0000 7f63953ea640 1 --2- 172.21.15.45:0/808535370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6390155780 0x7f63900abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.730+0000 7f63953ea640 1 -- 172.21.15.45:0/808535370 >> v1:172.21.15.190:6789/0 conn(0x7f63900ce630 legacy=0x7f639015ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:35.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.730+0000 7f63953ea640 1 -- 172.21.15.45:0/808535370 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639015ba90 con 0x7f63900ae540 2023-11-28T21:51:35.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.731+0000 7f63953ea640 1 --2- 172.21.15.45:0/808535370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63900ae540 0x7f63900ce0f0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f63800098a0 tx=0x7f638002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47468a82b34620ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:35.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.731+0000 7f6394be9640 1 -- 172.21.15.45:0/808535370 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380031640 con 0x7f63900ae540 2023-11-28T21:51:35.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.731+0000 7f6394be9640 1 -- 172.21.15.45:0/808535370 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63800317a0 con 0x7f63900ae540 2023-11-28T21:51:35.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.731+0000 7f6394be9640 1 -- 172.21.15.45:0/808535370 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380031b30 con 0x7f63900ae540 2023-11-28T21:51:35.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.732+0000 7f6396bed640 1 -- 172.21.15.45:0/808535370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63900ae540 msgr2=0x7f63900ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.732+0000 7f6396bed640 1 --2- 172.21.15.45:0/808535370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63900ae540 0x7f63900ce0f0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f63800098a0 tx=0x7f638002b8f0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.732+0000 7f6396bed640 1 -- 172.21.15.45:0/808535370 shutdown_connections 2023-11-28T21:51:35.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.732+0000 7f6396bed640 1 --2- 172.21.15.45:0/808535370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63900ae540 0x7f63900ce0f0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.732+0000 7f6396bed640 1 --2- 172.21.15.45:0/808535370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6390155780 0x7f63900abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.732+0000 7f6396bed640 1 -- 172.21.15.45:0/808535370 >> 172.21.15.45:0/808535370 conn(0x7f639006ab30 msgr2=0x7f63900b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:35.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.732+0000 7f6396bed640 1 -- 172.21.15.45:0/808535370 shutdown_connections 2023-11-28T21:51:35.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.732+0000 7f6396bed640 1 -- 172.21.15.45:0/808535370 wait complete. 2023-11-28T21:51:35.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.734+0000 7f6396bed640 1 Processor -- start 2023-11-28T21:51:35.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.734+0000 7f6396bed640 1 -- start start 2023-11-28T21:51:35.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f6396bed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63900ae540 0x7f6390138320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f6396bed640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63900ce630 0x7f6390138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f6396bed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6390155780 0x7f639009ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f6396bed640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639009b5d0 con 0x7f6390155780 2023-11-28T21:51:35.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f6396bed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639014efd0 con 0x7f63900ae540 2023-11-28T21:51:35.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f6396bed640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639014f270 con 0x7f63900ce630 2023-11-28T21:51:35.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f63963ec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6390155780 0x7f639009ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:35.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f63963ec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6390155780 0x7f639009ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42096/0 (socket says 172.21.15.45:42096) 2023-11-28T21:51:35.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f63963ec640 1 -- 172.21.15.45:0/2049735581 learned_addr learned my addr 172.21.15.45:0/2049735581 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:35.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f63953ea640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63900ce630 0x7f6390138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:35.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.736+0000 7f6395beb640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63900ae540 0x7f6390138320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63963ec640 1 -- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63900ce630 msgr2=0x7f6390138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63963ec640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63900ce630 0x7f6390138a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63963ec640 1 -- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63900ae540 msgr2=0x7f6390138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63963ec640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63900ae540 0x7f6390138320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63963ec640 1 -- 172.21.15.45:0/2049735581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63800094d0 con 0x7f6390155780 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63953ea640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63900ce630 0x7f6390138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f6395beb640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63900ae540 0x7f6390138320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63963ec640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6390155780 0x7f639009ba60 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f638800c780 tx=0x7f638800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63867fc640 1 -- 172.21.15.45:0/2049735581 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6388003d80 con 0x7f6390155780 2023-11-28T21:51:35.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63867fc640 1 -- 172.21.15.45:0/2049735581 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638800ce90 con 0x7f6390155780 2023-11-28T21:51:35.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f63867fc640 1 -- 172.21.15.45:0/2049735581 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63880052b0 con 0x7f6390155780 2023-11-28T21:51:35.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639014bd80 con 0x7f6390155780 2023-11-28T21:51:35.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.737+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639014c8a0 con 0x7f6390155780 2023-11-28T21:51:35.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.738+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639014f730 con 0x7f6390155780 2023-11-28T21:51:35.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.739+0000 7f63867fc640 1 -- 172.21.15.45:0/2049735581 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6388005410 con 0x7f6390155780 2023-11-28T21:51:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.740+0000 7f63867fc640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f637406af60 0x7f637406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.740+0000 7f63867fc640 1 -- 172.21.15.45:0/2049735581 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f6388088d80 con 0x7f6390155780 2023-11-28T21:51:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.741+0000 7f6395beb640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f637406af60 0x7f637406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.742+0000 7f6395beb640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f637406af60 0x7f637406d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f638c006fd0 tx=0x7f638c008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.744+0000 7f63867fc640 1 -- 172.21.15.45:0/2049735581 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6388053c00 con 0x7f6390155780 2023-11-28T21:51:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.943+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f639009b710 con 0x7f6390155780 2023-11-28T21:51:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.946+0000 7f63867fc640 1 -- 172.21.15.45:0/2049735581 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+14242 (secure 0 0 0) 0x7f6388017020 con 0x7f6390155780 2023-11-28T21:51:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:35.948 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":176,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:51:33.474824+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/637440036":"2023-11-29T21:51:30.457070+0000","172.21.15.45:6834/637440036":"2023-11-29T21:51:30.457070+0000","172.21.15.190:0/4079793895":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/17282265":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/932255615":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/3336614716":"2023-11-29T21:51:26.378009+0000","172.21.15.190:6832/560718832":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/4184631271":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/2635607595":"2023-11-29T21:51:26.378009+0000","172.21.15.190:6833/560718832":"2023-11-29T21:51:26.378009+0000","172.21.15.190:0/943491299":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1767202704":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2782217070":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/290777784":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6833/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:6832/226305585":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/1078115353":"2023-11-29T20:55:29.635279+0000","172.21.15.190:0/2712902162":"2023-11-29T20:55:29.635279+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:51:35.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f637406af60 msgr2=0x7f637406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f637406af60 0x7f637406d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f638c006fd0 tx=0x7f638c008040 comp rx=0 tx=0).stop 2023-11-28T21:51:35.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6390155780 msgr2=0x7f639009ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:35.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6390155780 0x7f639009ba60 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f638800c780 tx=0x7f638800cc40 comp rx=0 tx=0).stop 2023-11-28T21:51:35.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 shutdown_connections 2023-11-28T21:51:35.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f637406af60 0x7f637406d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6390155780 0x7f639009ba60 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63900ce630 0x7f6390138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 --2- 172.21.15.45:0/2049735581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63900ae540 0x7f6390138320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:35.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.948+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 >> 172.21.15.45:0/2049735581 conn(0x7f639006ab30 msgr2=0x7f63900b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:35.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.949+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 shutdown_connections 2023-11-28T21:51:35.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:35.949+0000 7f6396bed640 1 -- 172.21.15.45:0/2049735581 wait complete. 2023-11-28T21:51:35.970 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T21:51:35.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T21:51:36.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.195+0000 7f6307fff640 1 Processor -- start 2023-11-28T21:51:36.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.195+0000 7f6307fff640 1 -- start start 2023-11-28T21:51:36.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.197+0000 7f6307fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f63080cddf0 0x7f63080ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.197+0000 7f6307fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080ce740 con 0x7f630815d230 2023-11-28T21:51:36.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.197+0000 7f6307fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080d11d0 con 0x7f63080b2640 2023-11-28T21:51:36.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.197+0000 7f6307fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63080b68f0 con 0x7f63080cddf0 2023-11-28T21:51:36.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.198+0000 7f6306ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f630815d230 0x7f63080b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39624/0 (socket says 172.21.15.45:39624) 2023-11-28T21:51:36.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.198+0000 7f6306ffd640 1 -- 172.21.15.45:0/2734355526 learned_addr learned my addr 172.21.15.45:0/2734355526 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:36.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.198+0000 7f63077fe640 1 --2- 172.21.15.45:0/2734355526 >> v2:172.21.15.45:3300/0 conn(0x7f63080cddf0 0x7f63080ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:36.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.198+0000 7f6305ffb640 1 -- 172.21.15.45:0/2734355526 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132723506 0 0) 0x7f63080ce740 con 0x7f630815d230 2023-11-28T21:51:36.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.198+0000 7f63077fe640 1 -- 172.21.15.45:0/2734355526 >> v1:172.21.15.190:6789/0 conn(0x7f63080b2640 legacy=0x7f63080b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:36.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.198+0000 7f63077fe640 1 -- 172.21.15.45:0/2734355526 >> v1:172.21.15.45:6789/0 conn(0x7f630815d230 legacy=0x7f63080b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:36.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.198+0000 7f63077fe640 1 -- 172.21.15.45:0/2734355526 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080d1490 con 0x7f63080cddf0 2023-11-28T21:51:36.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.199+0000 7f63077fe640 1 --2- 172.21.15.45:0/2734355526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080cddf0 0x7f63080ce200 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f6300009d90 tx=0x7f630002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b98ab0ef9e14db server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.199+0000 7f6305ffb640 1 -- 172.21.15.45:0/2734355526 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630002cd00 con 0x7f63080cddf0 2023-11-28T21:51:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.200+0000 7f6305ffb640 1 -- 172.21.15.45:0/2734355526 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630002ce60 con 0x7f63080cddf0 2023-11-28T21:51:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.200+0000 7f6305ffb640 1 -- 172.21.15.45:0/2734355526 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300034660 con 0x7f63080cddf0 2023-11-28T21:51:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.200+0000 7f6307fff640 1 -- 172.21.15.45:0/2734355526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080cddf0 msgr2=0x7f63080ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.200+0000 7f6307fff640 1 --2- 172.21.15.45:0/2734355526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080cddf0 0x7f63080ce200 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f6300009d90 tx=0x7f630002bf50 comp rx=0 tx=0).stop 2023-11-28T21:51:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.200+0000 7f6307fff640 1 -- 172.21.15.45:0/2734355526 shutdown_connections 2023-11-28T21:51:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.200+0000 7f6307fff640 1 --2- 172.21.15.45:0/2734355526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080cddf0 0x7f63080ce200 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.200+0000 7f6307fff640 1 -- 172.21.15.45:0/2734355526 >> 172.21.15.45:0/2734355526 conn(0x7f6308070f40 msgr2=0x7f6308071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:36.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.200+0000 7f6307fff640 1 -- 172.21.15.45:0/2734355526 shutdown_connections 2023-11-28T21:51:36.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.200+0000 7f6307fff640 1 -- 172.21.15.45:0/2734355526 wait complete. 2023-11-28T21:51:36.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.202+0000 7f6307fff640 1 Processor -- start 2023-11-28T21:51:36.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.202+0000 7f6307fff640 1 -- start start 2023-11-28T21:51:36.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.203+0000 7f6307fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080b2640 0x7f6308151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.203+0000 7f6307fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63080cddf0 0x7f63081555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.203+0000 7f6307fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f630815d230 0x7f630815e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.203+0000 7f6307fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080c7b40 con 0x7f63080b2640 2023-11-28T21:51:36.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.203+0000 7f6307fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080c7e20 con 0x7f630815d230 2023-11-28T21:51:36.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.203+0000 7f6307fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63080c8100 con 0x7f63080cddf0 2023-11-28T21:51:36.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f6306ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080b2640 0x7f6308151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:36.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f6306ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080b2640 0x7f6308151a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42116/0 (socket says 172.21.15.45:42116) 2023-11-28T21:51:36.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f6306ffd640 1 -- 172.21.15.45:0/1948499985 learned_addr learned my addr 172.21.15.45:0/1948499985 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:36.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f63077fe640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f630815d230 0x7f630815e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:36.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f63067fc640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63080cddf0 0x7f63081555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:36.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f6306ffd640 1 -- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63080cddf0 msgr2=0x7f63081555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:36.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f6306ffd640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63080cddf0 0x7f63081555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f6306ffd640 1 -- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f630815d230 msgr2=0x7f630815e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:36.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f6306ffd640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f630815d230 0x7f630815e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f6306ffd640 1 -- 172.21.15.45:0/1948499985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6300009a70 con 0x7f63080b2640 2023-11-28T21:51:36.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f63077fe640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f630815d230 0x7f630815e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:36.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.204+0000 7f63067fc640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63080cddf0 0x7f63081555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:36.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.205+0000 7f6306ffd640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080b2640 0x7f6308151a10 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f62f800c430 tx=0x7f62f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:36.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.205+0000 7f62ef7fe640 1 -- 172.21.15.45:0/1948499985 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80074e0 con 0x7f63080b2640 2023-11-28T21:51:36.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.205+0000 7f62ef7fe640 1 -- 172.21.15.45:0/1948499985 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f8007660 con 0x7f63080b2640 2023-11-28T21:51:36.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.205+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63080c8420 con 0x7f63080b2640 2023-11-28T21:51:36.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.205+0000 7f62ef7fe640 1 -- 172.21.15.45:0/1948499985 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8004a20 con 0x7f63080b2640 2023-11-28T21:51:36.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.205+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63080c89e0 con 0x7f63080b2640 2023-11-28T21:51:36.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.206+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308071b30 con 0x7f63080b2640 2023-11-28T21:51:36.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.207+0000 7f62ef7fe640 1 -- 172.21.15.45:0/1948499985 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f62f8004b80 con 0x7f63080b2640 2023-11-28T21:51:36.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.208+0000 7f62ef7fe640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f62f406af60 0x7f62f406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.208+0000 7f62ef7fe640 1 -- 172.21.15.45:0/1948499985 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f62f8089770 con 0x7f63080b2640 2023-11-28T21:51:36.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.209+0000 7f63067fc640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f62f406af60 0x7f62f406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:36.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.210+0000 7f63067fc640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f62f406af60 0x7f62f406d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6308158810 tx=0x7f62fc009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:36.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.213+0000 7f62ef7fe640 1 -- 172.21.15.45:0/1948499985 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62f8054560 con 0x7f63080b2640 2023-11-28T21:51:36.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.408+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f630804e920 con 0x7f63080b2640 2023-11-28T21:51:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.409+0000 7f62ef7fe640 1 -- 172.21.15.45:0/1948499985 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v176) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62f8057b80 con 0x7f63080b2640 2023-11-28T21:51:36.411 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T21:51:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f62f406af60 msgr2=0x7f62f406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f62f406af60 0x7f62f406d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6308158810 tx=0x7f62fc009040 comp rx=0 tx=0).stop 2023-11-28T21:51:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080b2640 msgr2=0x7f6308151a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080b2640 0x7f6308151a10 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f62f800c430 tx=0x7f62f800c8f0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 shutdown_connections 2023-11-28T21:51:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f62f406af60 0x7f62f406d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f630815d230 0x7f630815e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63080cddf0 0x7f63081555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 --2- 172.21.15.45:0/1948499985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63080b2640 0x7f6308151a10 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 >> 172.21.15.45:0/1948499985 conn(0x7f6308070f40 msgr2=0x7f63080b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:36.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 shutdown_connections 2023-11-28T21:51:36.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.412+0000 7f6307fff640 1 -- 172.21.15.45:0/1948499985 wait complete. 2023-11-28T21:51:36.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-11-28T21:51:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.653+0000 7f0d72dc1640 1 Processor -- start 2023-11-28T21:51:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.653+0000 7f0d72dc1640 1 -- start start 2023-11-28T21:51:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.655+0000 7f0d72dc1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0d6c15d140 0x7f0d6c0d1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.655+0000 7f0d72dc1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d520 0x7f0d6c0cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.655+0000 7f0d72dc1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0d2810 con 0x7f0d6c0d1680 2023-11-28T21:51:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.655+0000 7f0d72dc1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c0b71b0 con 0x7f0d6c15d140 2023-11-28T21:51:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.655+0000 7f0d72dc1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0d1c00 con 0x7f0d6c15d520 2023-11-28T21:51:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.655+0000 7f0d71dbf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0d6c15d140 0x7f0d6c0d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.655+0000 7f0d71dbf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0d6c15d140 0x7f0d6c0d1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42118/0 (socket says 172.21.15.45:42118) 2023-11-28T21:51:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.655+0000 7f0d71dbf640 1 -- 172.21.15.45:0/1120039142 learned_addr learned my addr 172.21.15.45:0/1120039142 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.656+0000 7f0d715be640 1 --2- 172.21.15.45:0/1120039142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d520 0x7f0d6c0cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.656+0000 7f0d71dbf640 1 -- 172.21.15.45:0/1120039142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d520 msgr2=0x7f0d6c0cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.656+0000 7f0d71dbf640 1 --2- 172.21.15.45:0/1120039142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d520 0x7f0d6c0cfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.656+0000 7f0d71dbf640 1 -- 172.21.15.45:0/1120039142 >> v1:172.21.15.190:6789/0 conn(0x7f0d6c0d1680 legacy=0x7f0d6c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.656+0000 7f0d71dbf640 1 -- 172.21.15.45:0/1120039142 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c0d1ee0 con 0x7f0d6c15d140 2023-11-28T21:51:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.656+0000 7f0d715be640 1 --2- 172.21.15.45:0/1120039142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d520 0x7f0d6c0cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:36.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.657+0000 7f0d71dbf640 1 --2- 172.21.15.45:0/1120039142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c15d140 0x7f0d6c0d1140 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009870 tx=0x7f0d6802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d622c61ab17080f2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.657+0000 7f0d70dbd640 1 -- 172.21.15.45:0/1120039142 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68031640 con 0x7f0d6c15d140 2023-11-28T21:51:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.657+0000 7f0d70dbd640 1 -- 172.21.15.45:0/1120039142 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d680317a0 con 0x7f0d6c15d140 2023-11-28T21:51:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.657+0000 7f0d70dbd640 1 -- 172.21.15.45:0/1120039142 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68031b30 con 0x7f0d6c15d140 2023-11-28T21:51:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.658+0000 7f0d72dc1640 1 -- 172.21.15.45:0/1120039142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c15d140 msgr2=0x7f0d6c0d1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.658+0000 7f0d72dc1640 1 --2- 172.21.15.45:0/1120039142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c15d140 0x7f0d6c0d1140 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009870 tx=0x7f0d6802b8f0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.658+0000 7f0d72dc1640 1 -- 172.21.15.45:0/1120039142 shutdown_connections 2023-11-28T21:51:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.658+0000 7f0d72dc1640 1 --2- 172.21.15.45:0/1120039142 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d520 0x7f0d6c0cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.658+0000 7f0d72dc1640 1 --2- 172.21.15.45:0/1120039142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c15d140 0x7f0d6c0d1140 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.658+0000 7f0d72dc1640 1 -- 172.21.15.45:0/1120039142 >> 172.21.15.45:0/1120039142 conn(0x7f0d6c06a500 msgr2=0x7f0d6c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.658+0000 7f0d72dc1640 1 -- 172.21.15.45:0/1120039142 shutdown_connections 2023-11-28T21:51:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.658+0000 7f0d72dc1640 1 -- 172.21.15.45:0/1120039142 wait complete. 2023-11-28T21:51:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.660+0000 7f0d72dc1640 1 Processor -- start 2023-11-28T21:51:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.660+0000 7f0d72dc1640 1 -- start start 2023-11-28T21:51:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d72dc1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c0d1680 0x7f0d6c0cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d72dc1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d140 0x7f0d6c0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d72dc1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d6c15d520 0x7f0d6c15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d72dc1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c1482f0 con 0x7f0d6c0d1680 2023-11-28T21:51:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d72dc1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0a9b20 con 0x7f0d6c15d520 2023-11-28T21:51:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d72dc1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c148590 con 0x7f0d6c15d140 2023-11-28T21:51:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d71dbf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c0d1680 0x7f0d6c0cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d71dbf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c0d1680 0x7f0d6c0cd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42134/0 (socket says 172.21.15.45:42134) 2023-11-28T21:51:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d71dbf640 1 -- 172.21.15.45:0/2112977803 learned_addr learned my addr 172.21.15.45:0/2112977803 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.661+0000 7f0d71dbf640 1 -- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d140 msgr2=0x7f0d6c0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d71dbf640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d140 0x7f0d6c0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d71dbf640 1 -- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d6c15d520 msgr2=0x7f0d6c15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d71dbf640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d6c15d520 0x7f0d6c15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d71dbf640 1 -- 172.21.15.45:0/2112977803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d680094d0 con 0x7f0d6c0d1680 2023-11-28T21:51:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d71dbf640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c0d1680 0x7f0d6c0cd200 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009870 tx=0x7f0d680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d627fc640 1 -- 172.21.15.45:0/2112977803 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6803ac60 con 0x7f0d6c0d1680 2023-11-28T21:51:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d627fc640 1 -- 172.21.15.45:0/2112977803 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6803ade0 con 0x7f0d6c0d1680 2023-11-28T21:51:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d627fc640 1 -- 172.21.15.45:0/2112977803 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68044630 con 0x7f0d6c0d1680 2023-11-28T21:51:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c148870 con 0x7f0d6c0d1680 2023-11-28T21:51:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.662+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c0c98e0 con 0x7f0d6c0d1680 2023-11-28T21:51:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.664+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c0c9ad0 con 0x7f0d6c0d1680 2023-11-28T21:51:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.664+0000 7f0d627fc640 1 -- 172.21.15.45:0/2112977803 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f0d6802f050 con 0x7f0d6c0d1680 2023-11-28T21:51:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.665+0000 7f0d627fc640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f0d5006af60 0x7f0d5006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.665+0000 7f0d627fc640 1 -- 172.21.15.45:0/2112977803 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f0d680abaf0 con 0x7f0d6c0d1680 2023-11-28T21:51:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.666+0000 7f0d715be640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f0d5006af60 0x7f0d5006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.667+0000 7f0d715be640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f0d5006af60 0x7f0d5006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c006fd0 tx=0x7f0d5c008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.670+0000 7f0d627fc640 1 -- 172.21.15.45:0/2112977803 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d680769c0 con 0x7f0d6c0d1680 2023-11-28T21:51:36.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:36.937+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f0d6c148a90 con 0x7f0d6c0d1680 2023-11-28T21:51:37.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.837+0000 7f0d627fc640 1 -- 172.21.15.45:0/2112977803 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2547) v1 ==== 177+0+0 (secure 0 0 0) 0x7f0d68079fe0 con 0x7f0d6c0d1680 2023-11-28T21:51:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.840+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f0d5006af60 msgr2=0x7f0d5006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.840+0000 7f0d72dc1640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f0d5006af60 0x7f0d5006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c006fd0 tx=0x7f0d5c008040 comp rx=0 tx=0).stop 2023-11-28T21:51:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.840+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c0d1680 msgr2=0x7f0d6c0cd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:37.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.840+0000 7f0d72dc1640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c0d1680 0x7f0d6c0cd200 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009870 tx=0x7f0d680046c0 comp rx=0 tx=0).stop 2023-11-28T21:51:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.841+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 shutdown_connections 2023-11-28T21:51:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.841+0000 7f0d72dc1640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f0d5006af60 0x7f0d5006d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.841+0000 7f0d72dc1640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d6c15d520 0x7f0d6c15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.841+0000 7f0d72dc1640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d6c15d140 0x7f0d6c0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.841+0000 7f0d72dc1640 1 --2- 172.21.15.45:0/2112977803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d6c0d1680 0x7f0d6c0cd200 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.841+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 >> 172.21.15.45:0/2112977803 conn(0x7f0d6c06a500 msgr2=0x7f0d6c06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.841+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 shutdown_connections 2023-11-28T21:51:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:37.841+0000 7f0d72dc1640 1 -- 172.21.15.45:0/2112977803 wait complete. 2023-11-28T21:51:37.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T21:51:38.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.099+0000 7f1bdb59e640 1 Processor -- start 2023-11-28T21:51:38.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.099+0000 7f1bdb59e640 1 -- start start 2023-11-28T21:51:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdb59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1bdc155770 0x7f1bdc15c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdb59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 0x7f1bdc06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdb59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1bdc06c480 0x7f1bdc0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdb59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bdc06c910 con 0x7f1bdc06c480 2023-11-28T21:51:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdb59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bdc0cdb20 con 0x7f1bdc155770 2023-11-28T21:51:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdb59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc15c580 con 0x7f1bdc0cdfb0 2023-11-28T21:51:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdad9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1bdc06c480 0x7f1bdc0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdad9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1bdc06c480 0x7f1bdc0b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42144/0 (socket says 172.21.15.45:42144) 2023-11-28T21:51:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdad9d640 1 -- 172.21.15.45:0/1464929064 learned_addr learned my addr 172.21.15.45:0/1464929064 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:38.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bda59c640 1 --2- 172.21.15.45:0/1464929064 >> v2:172.21.15.190:3300/0 conn(0x7f1bdc155770 0x7f1bdc15c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:38.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bd3fff640 1 --2- 172.21.15.45:0/1464929064 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 0x7f1bdc06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:38.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdad9d640 1 -- 172.21.15.45:0/1464929064 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 msgr2=0x7f1bdc06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:38.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdad9d640 1 --2- 172.21.15.45:0/1464929064 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 0x7f1bdc06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdad9d640 1 -- 172.21.15.45:0/1464929064 >> v2:172.21.15.190:3300/0 conn(0x7f1bdc155770 msgr2=0x7f1bdc15c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:38.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdad9d640 1 --2- 172.21.15.45:0/1464929064 >> v2:172.21.15.190:3300/0 conn(0x7f1bdc155770 0x7f1bdc15c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bdad9d640 1 -- 172.21.15.45:0/1464929064 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc0ccf10 con 0x7f1bdc06c480 2023-11-28T21:51:38.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.101+0000 7f1bd3fff640 1 --2- 172.21.15.45:0/1464929064 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 0x7f1bdc06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:38.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.102+0000 7f1bdad9d640 1 --2- 172.21.15.45:0/1464929064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc06c480 0x7f1bdc0b20e0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc072430 tx=0x7f1bd402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d193f4ed4b31fee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:38.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.103+0000 7f1bd9d9b640 1 -- 172.21.15.45:0/1464929064 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd402c9e0 con 0x7f1bdc06c480 2023-11-28T21:51:38.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.103+0000 7f1bd9d9b640 1 -- 172.21.15.45:0/1464929064 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd402cb40 con 0x7f1bdc06c480 2023-11-28T21:51:38.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.103+0000 7f1bd9d9b640 1 -- 172.21.15.45:0/1464929064 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd40343f0 con 0x7f1bdc06c480 2023-11-28T21:51:38.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.103+0000 7f1bdb59e640 1 -- 172.21.15.45:0/1464929064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc06c480 msgr2=0x7f1bdc0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:38.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.103+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/1464929064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc06c480 0x7f1bdc0b20e0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc072430 tx=0x7f1bd402be40 comp rx=0 tx=0).stop 2023-11-28T21:51:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.104+0000 7f1bdb59e640 1 -- 172.21.15.45:0/1464929064 shutdown_connections 2023-11-28T21:51:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.104+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/1464929064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc06c480 0x7f1bdc0b20e0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.104+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/1464929064 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 0x7f1bdc06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.104+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/1464929064 >> v2:172.21.15.190:3300/0 conn(0x7f1bdc155770 0x7f1bdc15c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.104+0000 7f1bdb59e640 1 -- 172.21.15.45:0/1464929064 >> 172.21.15.45:0/1464929064 conn(0x7f1bdc0b1050 msgr2=0x7f1bdc06a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.104+0000 7f1bdb59e640 1 -- 172.21.15.45:0/1464929064 shutdown_connections 2023-11-28T21:51:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.104+0000 7f1bdb59e640 1 -- 172.21.15.45:0/1464929064 wait complete. 2023-11-28T21:51:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.106+0000 7f1bdb59e640 1 Processor -- start 2023-11-28T21:51:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.106+0000 7f1bdb59e640 1 -- start start 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdb59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bdc06c480 0x7f1bdc155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdb59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 0x7f1bdc0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdb59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc155770 0x7f1bdc142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdb59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0ac810 con 0x7f1bdc155770 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdb59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0acaf0 con 0x7f1bdc06c480 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdb59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0acdd0 con 0x7f1bdc0cdfb0 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc155770 0x7f1bdc142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc155770 0x7f1bdc142ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42150/0 (socket says 172.21.15.45:42150) 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdad9d640 1 -- 172.21.15.45:0/2934623090 learned_addr learned my addr 172.21.15.45:0/2934623090 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdad9d640 1 -- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 msgr2=0x7f1bdc0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdad9d640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 0x7f1bdc0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bda59c640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bdc06c480 0x7f1bdc155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:38.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdad9d640 1 -- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bdc06c480 msgr2=0x7f1bdc155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:38.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdad9d640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bdc06c480 0x7f1bdc155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.107+0000 7f1bdad9d640 1 -- 172.21.15.45:0/2934623090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd4009b30 con 0x7f1bdc155770 2023-11-28T21:51:38.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.108+0000 7f1bdad9d640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc155770 0x7f1bdc142ca0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4036040 tx=0x7f1bd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:38.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.108+0000 7f1bd2ffd640 1 -- 172.21.15.45:0/2934623090 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4009510 con 0x7f1bdc155770 2023-11-28T21:51:38.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.108+0000 7f1bd2ffd640 1 -- 172.21.15.45:0/2934623090 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd403e450 con 0x7f1bdc155770 2023-11-28T21:51:38.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.108+0000 7f1bd2ffd640 1 -- 172.21.15.45:0/2934623090 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd403e7e0 con 0x7f1bdc155770 2023-11-28T21:51:38.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.108+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bdc0ad0b0 con 0x7f1bdc155770 2023-11-28T21:51:38.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.108+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bdc156300 con 0x7f1bdc155770 2023-11-28T21:51:38.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.110+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bdc156490 con 0x7f1bdc155770 2023-11-28T21:51:38.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.110+0000 7f1bd2ffd640 1 -- 172.21.15.45:0/2934623090 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1bd403e940 con 0x7f1bdc155770 2023-11-28T21:51:38.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.112+0000 7f1bd2ffd640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1bbc06afb0 0x7f1bbc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:38.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.112+0000 7f1bd2ffd640 1 -- 172.21.15.45:0/2934623090 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f1bd40ac730 con 0x7f1bdc155770 2023-11-28T21:51:38.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.112+0000 7f1bda59c640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1bbc06afb0 0x7f1bbc06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:38.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.113+0000 7f1bda59c640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1bbc06afb0 0x7f1bbc06d470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc006fd0 tx=0x7f1bcc008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:38.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.117+0000 7f1bd2ffd640 1 -- 172.21.15.45:0/2934623090 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bd40775a0 con 0x7f1bdc155770 2023-11-28T21:51:38.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.315+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1bdc0ad570 con 0x7f1bdc155770 2023-11-28T21:51:38.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.322+0000 7f1bd2ffd640 1 -- 172.21.15.45:0/2934623090 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1bd407abc0 con 0x7f1bdc155770 2023-11-28T21:51:38.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1bbc06afb0 msgr2=0x7f1bbc06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1bbc06afb0 0x7f1bbc06d470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc006fd0 tx=0x7f1bcc008040 comp rx=0 tx=0).stop 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc155770 msgr2=0x7f1bdc142ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc155770 0x7f1bdc142ca0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4036040 tx=0x7f1bd4004a40 comp rx=0 tx=0).stop 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 shutdown_connections 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1bbc06afb0 0x7f1bbc06d470 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bdc155770 0x7f1bdc142ca0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bdc0cdfb0 0x7f1bdc0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 --2- 172.21.15.45:0/2934623090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bdc06c480 0x7f1bdc155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 >> 172.21.15.45:0/2934623090 conn(0x7f1bdc0b1050 msgr2=0x7f1bdc15b430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 shutdown_connections 2023-11-28T21:51:38.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:38.325+0000 7f1bdb59e640 1 -- 172.21.15.45:0/2934623090 wait complete. 2023-11-28T21:51:38.348 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T21:51:38.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T21:51:38.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T21:51:38.604 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T21:51:38.604 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:51:38.604 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:38.667 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T21:51:38.667 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T21:51:38.667 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:38.730 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T21:51:38.730 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:51:38.731 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:38.795 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T21:51:38.795 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T21:51:38.795 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:38.857 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T21:51:38.857 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T21:51:38.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:51:39.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.095+0000 7ff559ec2640 1 Processor -- start 2023-11-28T21:51:39.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.095+0000 7ff559ec2640 1 -- start start 2023-11-28T21:51:39.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.097+0000 7ff559ec2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff554155780 0x7ff5540abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.097+0000 7ff559ec2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55415c3c0 con 0x7ff5540ae540 2023-11-28T21:51:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.097+0000 7ff559ec2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff554155b60 con 0x7ff5540ce800 2023-11-28T21:51:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.097+0000 7ff559ec2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55415ba90 con 0x7ff554155780 2023-11-28T21:51:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.097+0000 7ff553fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff5540ae540 0x7ff5540ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39636/0 (socket says 172.21.15.45:39636) 2023-11-28T21:51:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.097+0000 7ff553fff640 1 -- 172.21.15.45:0/1031584397 learned_addr learned my addr 172.21.15.45:0/1031584397 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:39.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff558ec0640 1 --2- 172.21.15.45:0/1031584397 >> v2:172.21.15.190:3300/0 conn(0x7ff554155780 0x7ff5540abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737288900 0 0) 0x7ff55415c3c0 con 0x7ff5540ae540 2023-11-28T21:51:39.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53c003520 con 0x7ff5540ae540 2023-11-28T21:51:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3066217749 0 0) 0x7ff53c003520 con 0x7ff5540ae540 2023-11-28T21:51:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 >> v2:172.21.15.190:3300/0 conn(0x7ff554155780 msgr2=0x7ff5540abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 --2- 172.21.15.45:0/1031584397 >> v2:172.21.15.190:3300/0 conn(0x7ff554155780 0x7ff5540abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 >> v1:172.21.15.190:6789/0 conn(0x7ff5540ce800 legacy=0x7ff5540b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55415bd70 con 0x7ff5540ae540 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ff544001670 con 0x7ff5540ae540 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 >> v1:172.21.15.45:6789/0 conn(0x7ff5540ae540 legacy=0x7ff5540ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.098+0000 7ff5537fe640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff53c003970 0x7ff53c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff5537fe640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff53c0047c0 0x7ff53c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff5537fe640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff53c005180 0x7ff53c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55415ba90 con 0x7ff53c0047c0 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c00c020 con 0x7ff53c003970 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c00c1e0 con 0x7ff53c005180 2023-11-28T21:51:39.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff553fff640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff53c0047c0 0x7ff53c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff5596c1640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff53c005180 0x7ff53c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff558ec0640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff53c003970 0x7ff53c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff553fff640 1 -- 172.21.15.45:0/1031584397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff53c005180 msgr2=0x7ff53c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff553fff640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff53c005180 0x7ff53c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff553fff640 1 -- 172.21.15.45:0/1031584397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff53c003970 msgr2=0x7ff53c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff553fff640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff53c003970 0x7ff53c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff553fff640 1 -- 172.21.15.45:0/1031584397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c003520 con 0x7ff53c0047c0 2023-11-28T21:51:39.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.099+0000 7ff5596c1640 1 -- 172.21.15.45:0/1031584397 reap_dead start 2023-11-28T21:51:39.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.101+0000 7ff553fff640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff53c0047c0 0x7ff53c004c40 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c240 tx=0x7ff53c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdee6ae8e7a2f25d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:39.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.101+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544005ad0 con 0x7ff53c0047c0 2023-11-28T21:51:39.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.101+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff544005c30 con 0x7ff53c0047c0 2023-11-28T21:51:39.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.101+0000 7ff5537fe640 1 -- 172.21.15.45:0/1031584397 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54402caf0 con 0x7ff53c0047c0 2023-11-28T21:51:39.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.101+0000 7ff559ec2640 1 -- 172.21.15.45:0/1031584397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff53c0047c0 msgr2=0x7ff53c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.101+0000 7ff559ec2640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff53c0047c0 0x7ff53c004c40 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c240 tx=0x7ff53c0057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.102+0000 7ff559ec2640 1 -- 172.21.15.45:0/1031584397 shutdown_connections 2023-11-28T21:51:39.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.102+0000 7ff559ec2640 1 --2- 172.21.15.45:0/1031584397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff53c0047c0 0x7ff53c004c40 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.102+0000 7ff559ec2640 1 -- 172.21.15.45:0/1031584397 >> 172.21.15.45:0/1031584397 conn(0x7ff55406ab30 msgr2=0x7ff5540b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.102+0000 7ff559ec2640 1 -- 172.21.15.45:0/1031584397 shutdown_connections 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.102+0000 7ff559ec2640 1 -- 172.21.15.45:0/1031584397 wait complete. 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.104+0000 7ff559ec2640 1 Processor -- start 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.104+0000 7ff559ec2640 1 -- start start 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff559ec2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff53c0047c0 0x7ff5540aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff559ec2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5541f7410 0x7ff5540b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff559ec2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55409a330 0x7ff554071210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff559ec2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540cc510 con 0x7ff5541f7410 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff559ec2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540aef40 con 0x7ff53c0047c0 2023-11-28T21:51:39.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff559ec2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5541f7950 con 0x7ff55409a330 2023-11-28T21:51:39.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff553fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5541f7410 0x7ff5540b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff553fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5541f7410 0x7ff5540b0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42176/0 (socket says 172.21.15.45:42176) 2023-11-28T21:51:39.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff553fff640 1 -- 172.21.15.45:0/1811799484 learned_addr learned my addr 172.21.15.45:0/1811799484 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:39.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff553fff640 1 -- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55409a330 msgr2=0x7ff554071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff553fff640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55409a330 0x7ff554071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff558ec0640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff53c0047c0 0x7ff5540aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff553fff640 1 -- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff53c0047c0 msgr2=0x7ff5540aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff553fff640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff53c0047c0 0x7ff5540aea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.105+0000 7ff553fff640 1 -- 172.21.15.45:0/1811799484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5541f7c30 con 0x7ff5541f7410 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.106+0000 7ff553fff640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5541f7410 0x7ff5540b0580 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c240 tx=0x7ff54402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.106+0000 7ff5517fa640 1 -- 172.21.15.45:0/1811799484 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54403c040 con 0x7ff5541f7410 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.106+0000 7ff5517fa640 1 -- 172.21.15.45:0/1811799484 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54402ccc0 con 0x7ff5541f7410 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.106+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5541f7f10 con 0x7ff5541f7410 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.106+0000 7ff5517fa640 1 -- 172.21.15.45:0/1811799484 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544007290 con 0x7ff5541f7410 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.106+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5540b0ac0 con 0x7ff5541f7410 2023-11-28T21:51:39.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.108+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55404e920 con 0x7ff5541f7410 2023-11-28T21:51:39.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.108+0000 7ff5517fa640 1 -- 172.21.15.45:0/1811799484 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff544032070 con 0x7ff5541f7410 2023-11-28T21:51:39.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.109+0000 7ff5517fa640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff54006af60 0x7ff54006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.110+0000 7ff5517fa640 1 -- 172.21.15.45:0/1811799484 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7ff5440aacd0 con 0x7ff5541f7410 2023-11-28T21:51:39.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.110+0000 7ff558ec0640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff54006af60 0x7ff54006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.111+0000 7ff558ec0640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff54006af60 0x7ff54006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff554099cd0 tx=0x7ff548008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:39.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.114+0000 7ff5517fa640 1 -- 172.21.15.45:0/1811799484 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff544075bf0 con 0x7ff5541f7410 2023-11-28T21:51:39.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.348+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5540b2240 con 0x7ff5541f7410 2023-11-28T21:51:39.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.349+0000 7ff5517fa640 1 -- 172.21.15.45:0/1811799484 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+3702 (secure 0 0 0) 0x7ff544079210 con 0x7ff5541f7410 2023-11-28T21:51:39.351 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:39.351 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":78,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7531,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/1320359028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":1320359028},{"type":"v1","addr":"172.21.15.45:6833","nonce":1320359028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6836/808323687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":808323687},{"type":"v1","addr":"172.21.15.190:6836","nonce":808323687}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3406194668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3406194668},{"type":"v1","addr":"172.21.15.190:6837","nonce":3406194668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":9552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3249394188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3249394188},{"type":"v1","addr":"172.21.15.45:6835","nonce":3249394188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[]} 2023-11-28T21:51:39.351 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 78 2023-11-28T21:51:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.352+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff54006af60 msgr2=0x7ff54006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.352+0000 7ff559ec2640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff54006af60 0x7ff54006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff554099cd0 tx=0x7ff548008040 comp rx=0 tx=0).stop 2023-11-28T21:51:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5541f7410 msgr2=0x7ff5540b0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5541f7410 0x7ff5540b0580 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c240 tx=0x7ff54402e070 comp rx=0 tx=0).stop 2023-11-28T21:51:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 shutdown_connections 2023-11-28T21:51:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7ff54006af60 0x7ff54006d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff55409a330 0x7ff554071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5541f7410 0x7ff5540b0580 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 --2- 172.21.15.45:0/1811799484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff53c0047c0 0x7ff5540aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 >> 172.21.15.45:0/1811799484 conn(0x7ff55406ab30 msgr2=0x7ff55415cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 shutdown_connections 2023-11-28T21:51:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.353+0000 7ff559ec2640 1 -- 172.21.15.45:0/1811799484 wait complete. 2023-11-28T21:51:39.373 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T21:51:39.373 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T21:51:39.373 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T21:51:39.373 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T21:51:39.373 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T21:51:39.373 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T21:51:39.374 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T21:51:39.374 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T21:51:39.374 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T21:51:39.374 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T21:51:39.374 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T21:51:39.374 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T21:51:39.374 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T21:51:39.375 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T21:51:39.375 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T21:51:39.375 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T21:51:39.391 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T21:51:39.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T21:51:39.396 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T21:51:39.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T21:51:39.398 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T21:51:39.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T21:51:39.406 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T21:51:39.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T21:51:39.444 INFO:tasks.ceph.mds.d:Started 2023-11-28T21:51:39.444 DEBUG:teuthology.parallel:result is None 2023-11-28T21:51:39.449 INFO:tasks.ceph.mds.b:Started 2023-11-28T21:51:39.450 DEBUG:teuthology.parallel:result is None 2023-11-28T21:51:39.455 INFO:tasks.ceph.mds.c:Started 2023-11-28T21:51:39.455 DEBUG:teuthology.parallel:result is None 2023-11-28T21:51:39.463 INFO:tasks.ceph.mds.a:Started 2023-11-28T21:51:39.463 DEBUG:teuthology.parallel:result is None 2023-11-28T21:51:39.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T21:51:39.606 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T21:51:39.607 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T21:51:39.618 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T21:51:39.623 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T21:51:39.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.661+0000 7fea381be640 1 Processor -- start 2023-11-28T21:51:39.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.662+0000 7fea381be640 1 -- start start 2023-11-28T21:51:39.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.663+0000 7fea381be640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fea3015b820 0x7fea3006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.663+0000 7fea381be640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea300cf020 0x7fea300cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.663+0000 7fea381be640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3015fa80 con 0x7fea3015edc0 2023-11-28T21:51:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.663+0000 7fea381be640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3015fd60 con 0x7fea3015b820 2023-11-28T21:51:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.663+0000 7fea381be640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea300cf9f0 con 0x7fea300cf020 2023-11-28T21:51:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.663+0000 7fea3675b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fea3015edc0 0x7fea3015f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39666/0 (socket says 172.21.15.45:39666) 2023-11-28T21:51:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.663+0000 7fea3675b640 1 -- 172.21.15.45:0/1363744956 learned_addr learned my addr 172.21.15.45:0/1363744956 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.663+0000 7fea35f5a640 1 --2- 172.21.15.45:0/1363744956 >> v2:172.21.15.45:3300/0 conn(0x7fea3015b820 0x7fea3006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.664+0000 7fea35759640 1 --2- 172.21.15.45:0/1363744956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea300cf020 0x7fea300cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.664+0000 7fea34f58640 1 -- 172.21.15.45:0/1363744956 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176887011 0 0) 0x7fea3015fa80 con 0x7fea3015edc0 2023-11-28T21:51:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.664+0000 7fea34f58640 1 -- 172.21.15.45:0/1363744956 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea1c003520 con 0x7fea3015edc0 2023-11-28T21:51:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.664+0000 7fea35f5a640 1 -- 172.21.15.45:0/1363744956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea300cf020 msgr2=0x7fea300cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.664+0000 7fea35f5a640 1 --2- 172.21.15.45:0/1363744956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea300cf020 0x7fea300cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.664+0000 7fea35f5a640 1 -- 172.21.15.45:0/1363744956 >> v1:172.21.15.45:6789/0 conn(0x7fea3015edc0 legacy=0x7fea3015f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.664+0000 7fea35f5a640 1 -- 172.21.15.45:0/1363744956 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3006cf40 con 0x7fea3015b820 2023-11-28T21:51:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.665+0000 7fea35f5a640 1 --2- 172.21.15.45:0/1363744956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015b820 0x7fea3006da20 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0097a0 tx=0x7fea2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf612a208aa7e20a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.665+0000 7fea34f58640 1 -- 172.21.15.45:0/1363744956 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c031830 con 0x7fea3015b820 2023-11-28T21:51:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea34f58640 1 -- 172.21.15.45:0/1363744956 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c031990 con 0x7fea3015b820 2023-11-28T21:51:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea34f58640 1 -- 172.21.15.45:0/1363744956 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c031d20 con 0x7fea3015b820 2023-11-28T21:51:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea381be640 1 -- 172.21.15.45:0/1363744956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015b820 msgr2=0x7fea3006da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea381be640 1 --2- 172.21.15.45:0/1363744956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015b820 0x7fea3006da20 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0097a0 tx=0x7fea2c02b8d0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea381be640 1 -- 172.21.15.45:0/1363744956 shutdown_connections 2023-11-28T21:51:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea381be640 1 --2- 172.21.15.45:0/1363744956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea300cf020 0x7fea300cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea381be640 1 --2- 172.21.15.45:0/1363744956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015b820 0x7fea3006da20 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea381be640 1 -- 172.21.15.45:0/1363744956 >> 172.21.15.45:0/1363744956 conn(0x7fea3006a6e0 msgr2=0x7fea3006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea381be640 1 -- 172.21.15.45:0/1363744956 shutdown_connections 2023-11-28T21:51:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.666+0000 7fea381be640 1 -- 172.21.15.45:0/1363744956 wait complete. 2023-11-28T21:51:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.668+0000 7fea381be640 1 Processor -- start 2023-11-28T21:51:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.668+0000 7fea381be640 1 -- start start 2023-11-28T21:51:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.669+0000 7fea381be640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea300cf020 0x7fea3009ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.669+0000 7fea381be640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea3015b820 0x7fea3014f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.669+0000 7fea381be640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015edc0 0x7fea30150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.669+0000 7fea381be640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3009bf50 con 0x7fea3015edc0 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.669+0000 7fea381be640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea301fadf0 con 0x7fea300cf020 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.669+0000 7fea381be640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3009b600 con 0x7fea3015b820 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea3675b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015edc0 0x7fea30150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea3675b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015edc0 0x7fea30150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42240/0 (socket says 172.21.15.45:42240) 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea3675b640 1 -- 172.21.15.45:0/198008316 learned_addr learned my addr 172.21.15.45:0/198008316 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea35f5a640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea300cf020 0x7fea3009ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea35759640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea3015b820 0x7fea3014f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea3675b640 1 -- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea3015b820 msgr2=0x7fea3014f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea3675b640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea3015b820 0x7fea3014f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea3675b640 1 -- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea300cf020 msgr2=0x7fea3009ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:39.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea3675b640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea300cf020 0x7fea3009ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:39.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea3675b640 1 -- 172.21.15.45:0/198008316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c009400 con 0x7fea3015edc0 2023-11-28T21:51:39.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea35759640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea3015b820 0x7fea3014f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:39.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.670+0000 7fea3675b640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015edc0 0x7fea30150b60 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fea2800cad0 tx=0x7fea2800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:39.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.671+0000 7fea267fc640 1 -- 172.21.15.45:0/198008316 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2800ac30 con 0x7fea3015edc0 2023-11-28T21:51:39.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.671+0000 7fea267fc640 1 -- 172.21.15.45:0/198008316 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2800ad90 con 0x7fea3015edc0 2023-11-28T21:51:39.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.671+0000 7fea267fc640 1 -- 172.21.15.45:0/198008316 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28012620 con 0x7fea3015edc0 2023-11-28T21:51:39.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.671+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3009a000 con 0x7fea3015edc0 2023-11-28T21:51:39.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.671+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3009a390 con 0x7fea3015edc0 2023-11-28T21:51:39.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.672+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3009a9c0 con 0x7fea3015edc0 2023-11-28T21:51:39.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.673+0000 7fea267fc640 1 -- 172.21.15.45:0/198008316 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fea28007070 con 0x7fea3015edc0 2023-11-28T21:51:39.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.674+0000 7fea267fc640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fea1406af60 0x7fea1406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:39.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.674+0000 7fea267fc640 1 -- 172.21.15.45:0/198008316 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fea280897a0 con 0x7fea3015edc0 2023-11-28T21:51:39.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.674+0000 7fea35f5a640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fea1406af60 0x7fea1406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:39.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.675+0000 7fea35f5a640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fea1406af60 0x7fea1406d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fea2c036000 tx=0x7fea2c03c040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:39.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.678+0000 7fea267fc640 1 -- 172.21.15.45:0/198008316 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea280546d0 con 0x7fea3015edc0 2023-11-28T21:51:39.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:39.895+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fea301fafc0 con 0x7fea3015edc0 2023-11-28T21:51:40.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.877+0000 7fea267fc640 1 -- 172.21.15.45:0/198008316 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v181) v1 ==== 97+0+0 (secure 0 0 0) 0x7fea2805e070 con 0x7fea3015edc0 2023-11-28T21:51:40.879 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T21:51:40.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fea1406af60 msgr2=0x7fea1406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:40.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fea1406af60 0x7fea1406d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fea2c036000 tx=0x7fea2c03c040 comp rx=0 tx=0).stop 2023-11-28T21:51:40.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015edc0 msgr2=0x7fea30150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:40.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015edc0 0x7fea30150b60 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fea2800cad0 tx=0x7fea2800cf90 comp rx=0 tx=0).stop 2023-11-28T21:51:40.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 shutdown_connections 2023-11-28T21:51:40.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fea1406af60 0x7fea1406d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:40.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea3015edc0 0x7fea30150b60 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:40.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea3015b820 0x7fea3014f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:40.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 --2- 172.21.15.45:0/198008316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea300cf020 0x7fea3009ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:40.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.881+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 >> 172.21.15.45:0/198008316 conn(0x7fea3006a6e0 msgr2=0x7fea3015e180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:40.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.882+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 shutdown_connections 2023-11-28T21:51:40.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:40.882+0000 7fea381be640 1 -- 172.21.15.45:0/198008316 wait complete. 2023-11-28T21:51:40.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T21:51:41.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.132+0000 7f874f22e640 1 Processor -- start 2023-11-28T21:51:41.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.133+0000 7f874f22e640 1 -- start start 2023-11-28T21:51:41.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.134+0000 7f874f22e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f874815d520 0x7f87480cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.134+0000 7f874f22e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f87480cffc0 0x7f87480b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.134+0000 7f874f22e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87480d2810 con 0x7f874815d140 2023-11-28T21:51:41.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.134+0000 7f874f22e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87480d03d0 con 0x7f874815d520 2023-11-28T21:51:41.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.134+0000 7f874f22e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87480d1c00 con 0x7f87480cffc0 2023-11-28T21:51:41.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.135+0000 7f874da2b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f874815d520 0x7f87480cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.135+0000 7f874da2b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f874815d520 0x7f87480cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42256/0 (socket says 172.21.15.45:42256) 2023-11-28T21:51:41.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.135+0000 7f874da2b640 1 -- 172.21.15.45:0/2396097697 learned_addr learned my addr 172.21.15.45:0/2396097697 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:41.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.135+0000 7f874ea2d640 1 --2- 172.21.15.45:0/2396097697 >> v2:172.21.15.190:3300/0 conn(0x7f87480cffc0 0x7f87480b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.135+0000 7f874da2b640 1 -- 172.21.15.45:0/2396097697 >> v2:172.21.15.190:3300/0 conn(0x7f87480cffc0 msgr2=0x7f87480b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:41.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.135+0000 7f874da2b640 1 --2- 172.21.15.45:0/2396097697 >> v2:172.21.15.190:3300/0 conn(0x7f87480cffc0 0x7f87480b61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.136+0000 7f874da2b640 1 -- 172.21.15.45:0/2396097697 >> v1:172.21.15.190:6789/0 conn(0x7f874815d140 legacy=0x7f87480d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.136+0000 7f874da2b640 1 -- 172.21.15.45:0/2396097697 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87480d1ee0 con 0x7f874815d520 2023-11-28T21:51:41.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.137+0000 7f874da2b640 1 --2- 172.21.15.45:0/2396097697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 0x7f87480cfa80 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f87480d0ee0 tx=0x7f873802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b8e3a745f13c3f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:41.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.138+0000 7f874d22a640 1 -- 172.21.15.45:0/2396097697 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873802f850 con 0x7f874815d520 2023-11-28T21:51:41.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.138+0000 7f874d22a640 1 -- 172.21.15.45:0/2396097697 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873802f9b0 con 0x7f874815d520 2023-11-28T21:51:41.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.138+0000 7f874d22a640 1 -- 172.21.15.45:0/2396097697 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873802fd40 con 0x7f874815d520 2023-11-28T21:51:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.138+0000 7f874f22e640 1 -- 172.21.15.45:0/2396097697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 msgr2=0x7f87480cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.138+0000 7f874f22e640 1 --2- 172.21.15.45:0/2396097697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 0x7f87480cfa80 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f87480d0ee0 tx=0x7f873802ba30 comp rx=0 tx=0).stop 2023-11-28T21:51:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.138+0000 7f874f22e640 1 -- 172.21.15.45:0/2396097697 shutdown_connections 2023-11-28T21:51:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.138+0000 7f874f22e640 1 --2- 172.21.15.45:0/2396097697 >> v2:172.21.15.190:3300/0 conn(0x7f87480cffc0 0x7f87480b61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.138+0000 7f874f22e640 1 --2- 172.21.15.45:0/2396097697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 0x7f87480cfa80 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.138+0000 7f874f22e640 1 -- 172.21.15.45:0/2396097697 >> 172.21.15.45:0/2396097697 conn(0x7f874806a500 msgr2=0x7f874806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.139+0000 7f874f22e640 1 -- 172.21.15.45:0/2396097697 shutdown_connections 2023-11-28T21:51:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.139+0000 7f874f22e640 1 -- 172.21.15.45:0/2396097697 wait complete. 2023-11-28T21:51:41.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.140+0000 7f874f22e640 1 Processor -- start 2023-11-28T21:51:41.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.141+0000 7f874f22e640 1 -- start start 2023-11-28T21:51:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.141+0000 7f874f22e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87480cffc0 0x7f87480b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.141+0000 7f874f22e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f874815d140 0x7f87480b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874f22e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 0x7f874814a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874f22e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480b4630 con 0x7f874815d520 2023-11-28T21:51:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874f22e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480b4910 con 0x7f874815d140 2023-11-28T21:51:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874f22e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87480b4bf0 con 0x7f87480cffc0 2023-11-28T21:51:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874ea2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 0x7f874814a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874ea2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 0x7f874814a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42266/0 (socket says 172.21.15.45:42266) 2023-11-28T21:51:41.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874ea2d640 1 -- 172.21.15.45:0/1588947414 learned_addr learned my addr 172.21.15.45:0/1588947414 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:41.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874da2b640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f874815d140 0x7f87480b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874e22c640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87480cffc0 0x7f87480b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874ea2d640 1 -- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87480cffc0 msgr2=0x7f87480b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874ea2d640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87480cffc0 0x7f87480b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874ea2d640 1 -- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f874815d140 msgr2=0x7f87480b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874ea2d640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f874815d140 0x7f87480b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.142+0000 7f874ea2d640 1 -- 172.21.15.45:0/1588947414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380094d0 con 0x7f874815d520 2023-11-28T21:51:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.143+0000 7f874e22c640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87480cffc0 0x7f87480b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.143+0000 7f874da2b640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f874815d140 0x7f87480b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.143+0000 7f874ea2d640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 0x7f874814a340 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f8740007e60 tx=0x7f874000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.143+0000 7f873effd640 1 -- 172.21.15.45:0/1588947414 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8740014420 con 0x7f874815d520 2023-11-28T21:51:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.143+0000 7f873effd640 1 -- 172.21.15.45:0/1588947414 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87400145a0 con 0x7f874815d520 2023-11-28T21:51:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.143+0000 7f873effd640 1 -- 172.21.15.45:0/1588947414 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8740014950 con 0x7f874815d520 2023-11-28T21:51:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.143+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87481483b0 con 0x7f874815d520 2023-11-28T21:51:41.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.143+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87480b4290 con 0x7f874815d520 2023-11-28T21:51:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.145+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87481488d0 con 0x7f874815d520 2023-11-28T21:51:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.145+0000 7f873effd640 1 -- 172.21.15.45:0/1588947414 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8740014ab0 con 0x7f874815d520 2023-11-28T21:51:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.146+0000 7f873effd640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f872c06af60 0x7f872c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.147+0000 7f873effd640 1 -- 172.21.15.45:0/1588947414 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f8740089e80 con 0x7f874815d520 2023-11-28T21:51:41.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.147+0000 7f874e22c640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f872c06af60 0x7f872c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.148+0000 7f874e22c640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f872c06af60 0x7f872c06d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8744000f60 tx=0x7f8744009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:41.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.151+0000 7f873effd640 1 -- 172.21.15.45:0/1588947414 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87481488d0 con 0x7f874815d520 2023-11-28T21:51:41.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.418+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f87481488d0 con 0x7f874815d520 2023-11-28T21:51:41.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.420+0000 7f873effd640 1 -- 172.21.15.45:0/1588947414 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f87481488d0 con 0x7f874815d520 2023-11-28T21:51:41.421 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:41.421 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T21:51:41.421 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T21:51:41.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.421 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T21:51:41.421 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T21:51:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:51:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T21:51:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T21:51:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T21:51:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T21:51:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T21:51:41.436 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T21:51:41.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f872c06af60 msgr2=0x7f872c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f872c06af60 0x7f872c06d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8744000f60 tx=0x7f8744009040 comp rx=0 tx=0).stop 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 msgr2=0x7f874814a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 0x7f874814a340 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f8740007e60 tx=0x7f874000e4b0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 shutdown_connections 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f872c06af60 0x7f872c06d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f874815d520 0x7f874814a340 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f874815d140 0x7f87480b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 --2- 172.21.15.45:0/1588947414 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f87480cffc0 0x7f87480b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 >> 172.21.15.45:0/1588947414 conn(0x7f874806a500 msgr2=0x7f87480b7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 shutdown_connections 2023-11-28T21:51:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.422+0000 7f874f22e640 1 -- 172.21.15.45:0/1588947414 wait complete. 2023-11-28T21:51:41.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:51:41.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.668+0000 7f3a81e23640 1 Processor -- start 2023-11-28T21:51:41.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.668+0000 7f3a81e23640 1 -- start start 2023-11-28T21:51:41.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.670+0000 7f3a81e23640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3a7c13dfb0 0x7f3a7c15c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.670+0000 7f3a81e23640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c0b2fe0 0x7f3a7c0b3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.670+0000 7f3a81e23640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c070d90 con 0x7f3a7c15ec70 2023-11-28T21:51:41.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.670+0000 7f3a81e23640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7c15f850 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.670+0000 7f3a81e23640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c15b790 con 0x7f3a7c0b2fe0 2023-11-28T21:51:41.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.670+0000 7f3a7b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3a7c13dfb0 0x7f3a7c15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.670+0000 7f3a7b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3a7c13dfb0 0x7f3a7c15c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42282/0 (socket says 172.21.15.45:42282) 2023-11-28T21:51:41.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.670+0000 7f3a7b7fe640 1 -- 172.21.15.45:0/196082820 learned_addr learned my addr 172.21.15.45:0/196082820 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:41.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.671+0000 7f3a73fff640 1 --2- 172.21.15.45:0/196082820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c0b2fe0 0x7f3a7c0b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.671+0000 7f3a7b7fe640 1 -- 172.21.15.45:0/196082820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c0b2fe0 msgr2=0x7f3a7c0b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:41.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.671+0000 7f3a7b7fe640 1 --2- 172.21.15.45:0/196082820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c0b2fe0 0x7f3a7c0b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.671+0000 7f3a7b7fe640 1 -- 172.21.15.45:0/196082820 >> v1:172.21.15.190:6789/0 conn(0x7f3a7c15ec70 legacy=0x7f3a7c06e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:41.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.671+0000 7f3a7b7fe640 1 -- 172.21.15.45:0/196082820 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c0b3980 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.671+0000 7f3a73fff640 1 --2- 172.21.15.45:0/196082820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c0b2fe0 0x7f3a7c0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.672+0000 7f3a7b7fe640 1 --2- 172.21.15.45:0/196082820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 0x7f3a7c15c270 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009870 tx=0x7f3a6c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d180c5e5764941 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.672+0000 7f3a7affd640 1 -- 172.21.15.45:0/196082820 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c031640 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.672+0000 7f3a7affd640 1 -- 172.21.15.45:0/196082820 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6c0317a0 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.672+0000 7f3a7affd640 1 -- 172.21.15.45:0/196082820 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c031b30 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.673+0000 7f3a81e23640 1 -- 172.21.15.45:0/196082820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 msgr2=0x7f3a7c15c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.673+0000 7f3a81e23640 1 --2- 172.21.15.45:0/196082820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 0x7f3a7c15c270 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009870 tx=0x7f3a6c02b8f0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.673+0000 7f3a81e23640 1 -- 172.21.15.45:0/196082820 shutdown_connections 2023-11-28T21:51:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.673+0000 7f3a81e23640 1 --2- 172.21.15.45:0/196082820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c0b2fe0 0x7f3a7c0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.673+0000 7f3a81e23640 1 --2- 172.21.15.45:0/196082820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 0x7f3a7c15c270 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.673+0000 7f3a81e23640 1 -- 172.21.15.45:0/196082820 >> 172.21.15.45:0/196082820 conn(0x7f3a7c06c8f0 msgr2=0x7f3a7c06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:41.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.673+0000 7f3a81e23640 1 -- 172.21.15.45:0/196082820 shutdown_connections 2023-11-28T21:51:41.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.673+0000 7f3a81e23640 1 -- 172.21.15.45:0/196082820 wait complete. 2023-11-28T21:51:41.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.676+0000 7f3a81e23640 1 Processor -- start 2023-11-28T21:51:41.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.676+0000 7f3a81e23640 1 -- start start 2023-11-28T21:51:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.677+0000 7f3a81e23640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a7c0b2fe0 0x7f3a7c06c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.677+0000 7f3a81e23640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 0x7f3a7c15dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.677+0000 7f3a81e23640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c15ec70 0x7f3a7c13cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.677+0000 7f3a81e23640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c15d260 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.677+0000 7f3a81e23640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c1560f0 con 0x7f3a7c0b2fe0 2023-11-28T21:51:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.677+0000 7f3a81e23640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c15e360 con 0x7f3a7c15ec70 2023-11-28T21:51:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 0x7f3a7c15dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 0x7f3a7c15dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42290/0 (socket says 172.21.15.45:42290) 2023-11-28T21:51:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a73fff640 1 -- 172.21.15.45:0/1727728336 learned_addr learned my addr 172.21.15.45:0/1727728336 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a7b7fe640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a7c0b2fe0 0x7f3a7c06c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a73fff640 1 -- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c15ec70 msgr2=0x7f3a7c13cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a7bfff640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c15ec70 0x7f3a7c13cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a73fff640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c15ec70 0x7f3a7c13cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a73fff640 1 -- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a7c0b2fe0 msgr2=0x7f3a7c06c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a73fff640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a7c0b2fe0 0x7f3a7c06c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a73fff640 1 -- 172.21.15.45:0/1727728336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0094d0 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.678+0000 7f3a7bfff640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c15ec70 0x7f3a7c13cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.679+0000 7f3a73fff640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 0x7f3a7c15dcd0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400c7f0 tx=0x7f3a6400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.679+0000 7f3a78ff9640 1 -- 172.21.15.45:0/1727728336 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a64007a30 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.679+0000 7f3a78ff9640 1 -- 172.21.15.45:0/1727728336 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a64007b90 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.679+0000 7f3a78ff9640 1 -- 172.21.15.45:0/1727728336 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6400ce40 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.679+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c15c8c0 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.679+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c15ce80 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.680+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7c15d3a0 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.681+0000 7f3a78ff9640 1 -- 172.21.15.45:0/1727728336 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3a640113f0 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.683+0000 7f3a78ff9640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3a5c06af60 0x7f3a5c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.683+0000 7f3a78ff9640 1 -- 172.21.15.45:0/1727728336 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f3a64089810 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.683+0000 7f3a7b7fe640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3a5c06af60 0x7f3a5c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:41.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.684+0000 7f3a7b7fe640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3a5c06af60 0x7f3a5c06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c036000 tx=0x7f3a6c03c040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:41.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.687+0000 7f3a78ff9640 1 -- 172.21.15.45:0/1727728336 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a7c15d3a0 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.884+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a7c15d3a0 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.886+0000 7f3a78ff9640 1 -- 172.21.15.45:0/1727728336 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+13104 (secure 0 0 0) 0x7f3a7c15d3a0 con 0x7f3a7c13dfb0 2023-11-28T21:51:41.888 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:41.888 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":181,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:51:40.873555+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:51:41.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3a5c06af60 msgr2=0x7f3a5c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3a5c06af60 0x7f3a5c06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c036000 tx=0x7f3a6c03c040 comp rx=0 tx=0).stop 2023-11-28T21:51:41.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 msgr2=0x7f3a7c15dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:41.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 0x7f3a7c15dcd0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400c7f0 tx=0x7f3a6400cbf0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 shutdown_connections 2023-11-28T21:51:41.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3a5c06af60 0x7f3a5c06d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.894 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a7c15ec70 0x7f3a7c13cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a7c13dfb0 0x7f3a7c15dcd0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 --2- 172.21.15.45:0/1727728336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a7c0b2fe0 0x7f3a7c06c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:41.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.889+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 >> 172.21.15.45:0/1727728336 conn(0x7f3a7c06c8f0 msgr2=0x7f3a7c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:41.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.890+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 shutdown_connections 2023-11-28T21:51:41.895 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:41.890+0000 7f3a81e23640 1 -- 172.21.15.45:0/1727728336 wait complete. 2023-11-28T21:51:41.909 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T21:51:41.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T21:51:42.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.131+0000 7f1e4d6e0640 1 Processor -- start 2023-11-28T21:51:42.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.131+0000 7f1e4d6e0640 1 -- start start 2023-11-28T21:51:42.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.133+0000 7f1e4d6e0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480d24f0 0x7f1e480d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.133+0000 7f1e4d6e0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1e480ce0e0 0x7f1e480ce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.133+0000 7f1e4d6e0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e48160e80 con 0x7f1e48160aa0 2023-11-28T21:51:42.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.133+0000 7f1e4d6e0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e480d1220 con 0x7f1e480ce0e0 2023-11-28T21:51:42.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.133+0000 7f1e4d6e0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480d2e70 con 0x7f1e480d24f0 2023-11-28T21:51:42.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.134+0000 7f1e3ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480d24f0 0x7f1e480d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.134+0000 7f1e46ffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f1e48160aa0 0x7f1e480b6420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:35626/0 (socket says 172.21.15.45:35626) 2023-11-28T21:51:42.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.134+0000 7f1e46ffd640 1 -- 172.21.15.45:0/3404334145 learned_addr learned my addr 172.21.15.45:0/3404334145 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:42.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.134+0000 7f1e477fe640 1 --2- 172.21.15.45:0/3404334145 >> v2:172.21.15.190:3300/0 conn(0x7f1e480ce0e0 0x7f1e480ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.135+0000 7f1e3ffff640 1 -- 172.21.15.45:0/3404334145 >> v2:172.21.15.190:3300/0 conn(0x7f1e480ce0e0 msgr2=0x7f1e480ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:42.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.135+0000 7f1e3ffff640 1 --2- 172.21.15.45:0/3404334145 >> v2:172.21.15.190:3300/0 conn(0x7f1e480ce0e0 0x7f1e480ce570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.135+0000 7f1e3ffff640 1 -- 172.21.15.45:0/3404334145 >> v1:172.21.15.190:6789/0 conn(0x7f1e48160aa0 legacy=0x7f1e480b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.135+0000 7f1e3ffff640 1 -- 172.21.15.45:0/3404334145 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e480ceab0 con 0x7f1e480d24f0 2023-11-28T21:51:42.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.135+0000 7f1e467fc640 1 -- 172.21.15.45:0/3404334145 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997118922 0 0) 0x7f1e48160e80 con 0x7f1e48160aa0 2023-11-28T21:51:42.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.136+0000 7f1e3ffff640 1 --2- 172.21.15.45:0/3404334145 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480d24f0 0x7f1e480d2930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1e300097a0 tx=0x7f1e3002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71caf29e3ed2fa49 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:42.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.137+0000 7f1e467fc640 1 -- 172.21.15.45:0/3404334145 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30030610 con 0x7f1e480d24f0 2023-11-28T21:51:42.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.137+0000 7f1e467fc640 1 -- 172.21.15.45:0/3404334145 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30030770 con 0x7f1e480d24f0 2023-11-28T21:51:42.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.137+0000 7f1e467fc640 1 -- 172.21.15.45:0/3404334145 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30030b00 con 0x7f1e480d24f0 2023-11-28T21:51:42.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.138+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/3404334145 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480d24f0 msgr2=0x7f1e480d2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.138+0000 7f1e4d6e0640 1 --2- 172.21.15.45:0/3404334145 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480d24f0 0x7f1e480d2930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1e300097a0 tx=0x7f1e3002b960 comp rx=0 tx=0).stop 2023-11-28T21:51:42.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.138+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/3404334145 shutdown_connections 2023-11-28T21:51:42.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.138+0000 7f1e4d6e0640 1 --2- 172.21.15.45:0/3404334145 >> v2:172.21.15.190:3300/0 conn(0x7f1e480ce0e0 0x7f1e480ce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.138+0000 7f1e4d6e0640 1 --2- 172.21.15.45:0/3404334145 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480d24f0 0x7f1e480d2930 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.138+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/3404334145 >> 172.21.15.45:0/3404334145 conn(0x7f1e48070b20 msgr2=0x7f1e48070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:42.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.138+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/3404334145 shutdown_connections 2023-11-28T21:51:42.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.138+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/3404334145 wait complete. 2023-11-28T21:51:42.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.140+0000 7f1e4d6e0640 1 Processor -- start 2023-11-28T21:51:42.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.140+0000 7f1e4d6e0640 1 -- start start 2023-11-28T21:51:42.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.141+0000 7f1e4d6e0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480ce0e0 0x7f1e48156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.141+0000 7f1e4d6e0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e480d24f0 0x7f1e480c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.141+0000 7f1e4d6e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e48160aa0 0x7f1e480b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.141+0000 7f1e4d6e0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480c9a00 con 0x7f1e480d24f0 2023-11-28T21:51:42.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.141+0000 7f1e4d6e0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480c9ce0 con 0x7f1e48160aa0 2023-11-28T21:51:42.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.141+0000 7f1e4d6e0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e48158820 con 0x7f1e480ce0e0 2023-11-28T21:51:42.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.141+0000 7f1e3ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e480d24f0 0x7f1e480c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.141+0000 7f1e3ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e480d24f0 0x7f1e480c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42304/0 (socket says 172.21.15.45:42304) 2023-11-28T21:51:42.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.141+0000 7f1e3ffff640 1 -- 172.21.15.45:0/2020865518 learned_addr learned my addr 172.21.15.45:0/2020865518 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.142+0000 7f1e477fe640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e48160aa0 0x7f1e480b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.142+0000 7f1e46ffd640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480ce0e0 0x7f1e48156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.142+0000 7f1e3ffff640 1 -- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480ce0e0 msgr2=0x7f1e48156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.142+0000 7f1e3ffff640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480ce0e0 0x7f1e48156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.142+0000 7f1e3ffff640 1 -- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e48160aa0 msgr2=0x7f1e480b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.142+0000 7f1e3ffff640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e48160aa0 0x7f1e480b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.142+0000 7f1e3ffff640 1 -- 172.21.15.45:0/2020865518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30009400 con 0x7f1e480d24f0 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.142+0000 7f1e3ffff640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e480d24f0 0x7f1e480c6d40 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f1e3002bf70 tx=0x7f1e30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.142+0000 7f1e3f7fe640 1 -- 172.21.15.45:0/2020865518 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3003e040 con 0x7f1e480d24f0 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.143+0000 7f1e3f7fe640 1 -- 172.21.15.45:0/2020865518 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30039cb0 con 0x7f1e480d24f0 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.143+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e48158b00 con 0x7f1e480d24f0 2023-11-28T21:51:42.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.143+0000 7f1e3f7fe640 1 -- 172.21.15.45:0/2020865518 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30044630 con 0x7f1e480d24f0 2023-11-28T21:51:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.143+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e480c7280 con 0x7f1e480d24f0 2023-11-28T21:51:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.144+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e48158ce0 con 0x7f1e480d24f0 2023-11-28T21:51:42.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.145+0000 7f1e3f7fe640 1 -- 172.21.15.45:0/2020865518 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1e30043390 con 0x7f1e480d24f0 2023-11-28T21:51:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.146+0000 7f1e3f7fe640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1e2806af60 0x7f1e2806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.146+0000 7f1e3f7fe640 1 -- 172.21.15.45:0/2020865518 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f1e300ab9d0 con 0x7f1e480d24f0 2023-11-28T21:51:42.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.147+0000 7f1e46ffd640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1e2806af60 0x7f1e2806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.148+0000 7f1e46ffd640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1e2806af60 0x7f1e2806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e38000f60 tx=0x7f1e38009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:42.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.151+0000 7f1e3f7fe640 1 -- 172.21.15.45:0/2020865518 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e30076930 con 0x7f1e480d24f0 2023-11-28T21:51:42.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.344+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e480733d0 con 0x7f1e480d24f0 2023-11-28T21:51:42.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.345+0000 7f1e3f7fe640 1 -- 172.21.15.45:0/2020865518 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v181) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e30079f50 con 0x7f1e480d24f0 2023-11-28T21:51:42.347 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T21:51:42.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.347+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1e2806af60 msgr2=0x7f1e2806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.347+0000 7f1e4d6e0640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1e2806af60 0x7f1e2806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e38000f60 tx=0x7f1e38009040 comp rx=0 tx=0).stop 2023-11-28T21:51:42.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.347+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e480d24f0 msgr2=0x7f1e480c6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.348+0000 7f1e4d6e0640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e480d24f0 0x7f1e480c6d40 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f1e3002bf70 tx=0x7f1e30004730 comp rx=0 tx=0).stop 2023-11-28T21:51:42.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.348+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 shutdown_connections 2023-11-28T21:51:42.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.348+0000 7f1e4d6e0640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1e2806af60 0x7f1e2806d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.348+0000 7f1e4d6e0640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e48160aa0 0x7f1e480b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.348+0000 7f1e4d6e0640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e480d24f0 0x7f1e480c6d40 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.348+0000 7f1e4d6e0640 1 --2- 172.21.15.45:0/2020865518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e480ce0e0 0x7f1e48156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.348+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 >> 172.21.15.45:0/2020865518 conn(0x7f1e48070b20 msgr2=0x7f1e4815fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:42.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.348+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 shutdown_connections 2023-11-28T21:51:42.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.348+0000 7f1e4d6e0640 1 -- 172.21.15.45:0/2020865518 wait complete. 2023-11-28T21:51:42.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T21:51:42.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.598+0000 7f3b6b0b4640 1 Processor -- start 2023-11-28T21:51:42.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.598+0000 7f3b6b0b4640 1 -- start start 2023-11-28T21:51:42.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.600+0000 7f3b6b0b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b640ce800 0x7f3b64165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.600+0000 7f3b6b0b4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6415c3c0 con 0x7f3b640ae540 2023-11-28T21:51:42.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.600+0000 7f3b6b0b4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b64155b60 con 0x7f3b64155780 2023-11-28T21:51:42.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.600+0000 7f3b6b0b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6415ba90 con 0x7f3b640ce800 2023-11-28T21:51:42.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.600+0000 7f3b698b1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3b640ae540 0x7f3b640ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39680/0 (socket says 172.21.15.45:39680) 2023-11-28T21:51:42.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.600+0000 7f3b698b1640 1 -- 172.21.15.45:0/1380869696 learned_addr learned my addr 172.21.15.45:0/1380869696 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:42.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.601+0000 7f3b6a8b3640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b640ce800 0x7f3b64165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.601+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587017102 0 0) 0x7f3b6415c3c0 con 0x7f3b640ae540 2023-11-28T21:51:42.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.601+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b50003520 con 0x7f3b640ae540 2023-11-28T21:51:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.601+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3124115752 0 0) 0x7f3b50003520 con 0x7f3b640ae540 2023-11-28T21:51:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.601+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b640ce800 msgr2=0x7f3b64165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b640ce800 0x7f3b64165e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 >> v1:172.21.15.190:6789/0 conn(0x7f3b64155780 legacy=0x7f3b640abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6415bd70 con 0x7f3b640ae540 2023-11-28T21:51:42.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3b54001670 con 0x7f3b640ae540 2023-11-28T21:51:42.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 >> v1:172.21.15.45:6789/0 conn(0x7f3b640ae540 legacy=0x7f3b640ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b500039a0 0x7f3b50003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b640ce800 0x7f3b500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b50004d00 0x7f3b50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6415ba90 con 0x7f3b500039a0 2023-11-28T21:51:42.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5000b9e0 con 0x7f3b50004d00 2023-11-28T21:51:42.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5000bba0 con 0x7f3b640ce800 2023-11-28T21:51:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.602+0000 7f3b6a0b2640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b500039a0 0x7f3b50003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.603+0000 7f3b698b1640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b640ce800 0x7f3b500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.603+0000 7f3b6a0b2640 1 -- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b640ce800 msgr2=0x7f3b500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.603+0000 7f3b6a8b3640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b50004d00 0x7f3b50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.603+0000 7f3b6a0b2640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b640ce800 0x7f3b500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.603+0000 7f3b6a0b2640 1 -- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b50004d00 msgr2=0x7f3b50005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.603+0000 7f3b6a0b2640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b50004d00 0x7f3b50005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.603+0000 7f3b6a0b2640 1 -- 172.21.15.45:0/1380869696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b50003520 con 0x7f3b500039a0 2023-11-28T21:51:42.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.603+0000 7f3b698b1640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b640ce800 0x7f3b500047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.603+0000 7f3b6a8b3640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b50004d00 0x7f3b50005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.604+0000 7f3b6a0b2640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b500039a0 0x7f3b50003d80 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f3b60001860 tx=0x7f3b500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48ecb66035f6801 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.604+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60033640 con 0x7f3b500039a0 2023-11-28T21:51:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.604+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b600337c0 con 0x7f3b500039a0 2023-11-28T21:51:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.604+0000 7f3b690b0640 1 -- 172.21.15.45:0/1380869696 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60033b70 con 0x7f3b500039a0 2023-11-28T21:51:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.605+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/1380869696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b500039a0 msgr2=0x7f3b50003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.605+0000 7f3b6b0b4640 1 --2- 172.21.15.45:0/1380869696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b500039a0 0x7f3b50003d80 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f3b60001860 tx=0x7f3b500057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.605+0000 7f3b6a8b3640 1 -- 172.21.15.45:0/1380869696 reap_dead start 2023-11-28T21:51:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.605+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/1380869696 shutdown_connections 2023-11-28T21:51:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.605+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/1380869696 >> 172.21.15.45:0/1380869696 conn(0x7f3b6406ab30 msgr2=0x7f3b640b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.605+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/1380869696 shutdown_connections 2023-11-28T21:51:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.605+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/1380869696 wait complete. 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.607+0000 7f3b6b0b4640 1 Processor -- start 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.607+0000 7f3b6b0b4640 1 -- start start 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.608+0000 7f3b6b0b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b500039a0 0x7f3b64070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.608+0000 7f3b6b0b4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b6414d870 0x7f3b641f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.608+0000 7f3b6b0b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b6414dcf0 0x7f3b64159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.608+0000 7f3b6b0b4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b640711a0 con 0x7f3b6414dcf0 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.608+0000 7f3b6b0b4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b640701c0 con 0x7f3b6414d870 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.608+0000 7f3b6b0b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b641f3a90 con 0x7f3b500039a0 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b6a8b3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b6414dcf0 0x7f3b64159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b6a8b3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b6414dcf0 0x7f3b64159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42318/0 (socket says 172.21.15.45:42318) 2023-11-28T21:51:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b6a8b3640 1 -- 172.21.15.45:0/2098012229 learned_addr learned my addr 172.21.15.45:0/2098012229 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b6a0b2640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b500039a0 0x7f3b64070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b698b1640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b6414d870 0x7f3b641f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b6a8b3640 1 -- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b500039a0 msgr2=0x7f3b64070c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b6a8b3640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b500039a0 0x7f3b64070c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b6a8b3640 1 -- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b6414d870 msgr2=0x7f3b641f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b6a8b3640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b6414d870 0x7f3b641f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b6a8b3640 1 -- 172.21.15.45:0/2098012229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b641f3d70 con 0x7f3b6414dcf0 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.609+0000 7f3b698b1640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b6414d870 0x7f3b641f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.610+0000 7f3b6a8b3640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b6414dcf0 0x7f3b64159f70 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f3b6406e4d0 tx=0x7f3b5c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.610+0000 7f3b5affd640 1 -- 172.21.15.45:0/2098012229 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c017400 con 0x7f3b6414dcf0 2023-11-28T21:51:42.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.610+0000 7f3b5affd640 1 -- 172.21.15.45:0/2098012229 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c011040 con 0x7f3b6414dcf0 2023-11-28T21:51:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.610+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b64070460 con 0x7f3b6414dcf0 2023-11-28T21:51:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.610+0000 7f3b5affd640 1 -- 172.21.15.45:0/2098012229 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c017780 con 0x7f3b6414dcf0 2023-11-28T21:51:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.610+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b64071cc0 con 0x7f3b6414dcf0 2023-11-28T21:51:42.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.612+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b64070980 con 0x7f3b6414dcf0 2023-11-28T21:51:42.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.612+0000 7f3b5affd640 1 -- 172.21.15.45:0/2098012229 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3b5c0203b0 con 0x7f3b6414dcf0 2023-11-28T21:51:42.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.614+0000 7f3b5affd640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3b4806af60 0x7f3b4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:42.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.614+0000 7f3b5affd640 1 -- 172.21.15.45:0/2098012229 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f3b5c089810 con 0x7f3b6414dcf0 2023-11-28T21:51:42.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.614+0000 7f3b6a0b2640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3b4806af60 0x7f3b4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:42.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.616+0000 7f3b6a0b2640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3b4806af60 0x7f3b4806d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3b60035000 tx=0x7f3b60005040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:42.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.618+0000 7f3b5affd640 1 -- 172.21.15.45:0/2098012229 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b64070980 con 0x7f3b6414dcf0 2023-11-28T21:51:42.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.816+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3b64070980 con 0x7f3b6414dcf0 2023-11-28T21:51:42.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.816+0000 7f3b5affd640 1 -- 172.21.15.45:0/2098012229 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v181) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3b64070980 con 0x7f3b6414dcf0 2023-11-28T21:51:42.818 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:42.818 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T21:51:42.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:42.818 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T21:51:42.818 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T21:51:42.818 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:42.819 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T21:51:42.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3b4806af60 msgr2=0x7f3b4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3b4806af60 0x7f3b4806d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3b60035000 tx=0x7f3b60005040 comp rx=0 tx=0).stop 2023-11-28T21:51:42.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b6414dcf0 msgr2=0x7f3b64159f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:42.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b6414dcf0 0x7f3b64159f70 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f3b6406e4d0 tx=0x7f3b5c00f4a0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 shutdown_connections 2023-11-28T21:51:42.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3b4806af60 0x7f3b4806d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b6414dcf0 0x7f3b64159f70 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b6414d870 0x7f3b641f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 --2- 172.21.15.45:0/2098012229 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b500039a0 0x7f3b64070c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:42.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.819+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 >> 172.21.15.45:0/2098012229 conn(0x7f3b6406ab30 msgr2=0x7f3b6406d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:42.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.820+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 shutdown_connections 2023-11-28T21:51:42.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:42.820+0000 7f3b6b0b4640 1 -- 172.21.15.45:0/2098012229 wait complete. 2023-11-28T21:51:42.841 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T21:51:42.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T21:51:43.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.063+0000 7fb6f57a5640 1 Processor -- start 2023-11-28T21:51:43.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.063+0000 7fb6f57a5640 1 -- start start 2023-11-28T21:51:43.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.064+0000 7fb6f57a5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb6f00cc900 0x7fb6f00adb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.064+0000 7fb6f57a5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f00acad0 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.065+0000 7fb6f57a5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f00acdb0 con 0x7fb6f01607c0 2023-11-28T21:51:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.065+0000 7fb6f57a5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f00ac8c0 con 0x7fb6f00cc900 2023-11-28T21:51:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.065+0000 7fb6ef7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb6f00cc900 0x7fb6f00adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.065+0000 7fb6f4fa4640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb6f00ae0b0 0x7fb6f0070030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39692/0 (socket says 172.21.15.45:39692) 2023-11-28T21:51:43.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.065+0000 7fb6f4fa4640 1 -- 172.21.15.45:0/2334042926 learned_addr learned my addr 172.21.15.45:0/2334042926 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:43.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.065+0000 7fb6ef7fe640 1 -- 172.21.15.45:0/2334042926 >> v1:172.21.15.190:6789/0 conn(0x7fb6f01607c0 legacy=0x7fb6f00cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:43.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.065+0000 7fb6ef7fe640 1 -- 172.21.15.45:0/2334042926 >> v1:172.21.15.45:6789/0 conn(0x7fb6f00ae0b0 legacy=0x7fb6f0070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:43.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.065+0000 7fb6ef7fe640 1 -- 172.21.15.45:0/2334042926 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f00ad090 con 0x7fb6f00cc900 2023-11-28T21:51:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.066+0000 7fb6ef7fe640 1 --2- 172.21.15.45:0/2334042926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00cc900 0x7fb6f00adb70 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80097a0 tx=0x7fb6d802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70bfd970c3e5b99 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.067+0000 7fb6eeffd640 1 -- 172.21.15.45:0/2334042926 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d802e070 con 0x7fb6f00cc900 2023-11-28T21:51:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.067+0000 7fb6eeffd640 1 -- 172.21.15.45:0/2334042926 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d80315a0 con 0x7fb6f00cc900 2023-11-28T21:51:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.067+0000 7fb6eeffd640 1 -- 172.21.15.45:0/2334042926 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8031930 con 0x7fb6f00cc900 2023-11-28T21:51:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.067+0000 7fb6f57a5640 1 -- 172.21.15.45:0/2334042926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00cc900 msgr2=0x7fb6f00adb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.067+0000 7fb6f57a5640 1 --2- 172.21.15.45:0/2334042926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00cc900 0x7fb6f00adb70 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80097a0 tx=0x7fb6d802b840 comp rx=0 tx=0).stop 2023-11-28T21:51:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.068+0000 7fb6f57a5640 1 -- 172.21.15.45:0/2334042926 shutdown_connections 2023-11-28T21:51:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.068+0000 7fb6f57a5640 1 --2- 172.21.15.45:0/2334042926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00cc900 0x7fb6f00adb70 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.068+0000 7fb6f57a5640 1 -- 172.21.15.45:0/2334042926 >> 172.21.15.45:0/2334042926 conn(0x7fb6f00cec30 msgr2=0x7fb6f00cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.068+0000 7fb6f57a5640 1 -- 172.21.15.45:0/2334042926 shutdown_connections 2023-11-28T21:51:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.068+0000 7fb6f57a5640 1 -- 172.21.15.45:0/2334042926 wait complete. 2023-11-28T21:51:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.070+0000 7fb6f57a5640 1 Processor -- start 2023-11-28T21:51:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.070+0000 7fb6f57a5640 1 -- start start 2023-11-28T21:51:43.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.071+0000 7fb6f57a5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00ae0b0 0x7fb6f014ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:43.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.071+0000 7fb6f57a5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f00cc900 0x7fb6f00b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:43.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.071+0000 7fb6f57a5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6f01607c0 0x7fb6f00b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.071+0000 7fb6f57a5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f014c300 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.071+0000 7fb6f57a5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00c58e0 con 0x7fb6f00cc900 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6f57a5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f014c510 con 0x7fb6f01607c0 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00ae0b0 0x7fb6f014ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00ae0b0 0x7fb6f014ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42330/0 (socket says 172.21.15.45:42330) 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6effff640 1 -- 172.21.15.45:0/1301498477 learned_addr learned my addr 172.21.15.45:0/1301498477 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6ef7fe640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f00cc900 0x7fb6f00b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6effff640 1 -- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6f01607c0 msgr2=0x7fb6f00b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6effff640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6f01607c0 0x7fb6f00b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6effff640 1 -- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f00cc900 msgr2=0x7fb6f00b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6effff640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f00cc900 0x7fb6f00b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:43.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6effff640 1 -- 172.21.15.45:0/1301498477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d8009400 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.072+0000 7fb6ef7fe640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f00cc900 0x7fb6f00b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:43.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.073+0000 7fb6effff640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00ae0b0 0x7fb6f014ba20 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c7a0 tx=0x7fb6e000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:43.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.073+0000 7fb6ecff9640 1 -- 172.21.15.45:0/1301498477 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0003d80 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.073+0000 7fb6ecff9640 1 -- 172.21.15.45:0/1301498477 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e0005070 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.073+0000 7fb6ecff9640 1 -- 172.21.15.45:0/1301498477 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0005460 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.073+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f014c7f0 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.073+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f00c74e0 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.074+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f014cd10 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.075+0000 7fb6ecff9640 1 -- 172.21.15.45:0/1301498477 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb6e00055c0 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.076+0000 7fb6ecff9640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fb6d006af90 0x7fb6d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:43.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.076+0000 7fb6ecff9640 1 -- 172.21.15.45:0/1301498477 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fb6e0088ae0 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.077+0000 7fb6ef7fe640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fb6d006af90 0x7fb6d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:43.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.078+0000 7fb6ef7fe640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fb6d006af90 0x7fb6d006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8036000 tx=0x7fb6d803d040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:43.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.082+0000 7fb6ecff9640 1 -- 172.21.15.45:0/1301498477 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6e0053bd0 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.277+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb6f00733d0 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.905+0000 7fb6ecff9640 1 -- 172.21.15.45:0/1301498477 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v182) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb6e00571f0 con 0x7fb6f00ae0b0 2023-11-28T21:51:43.907 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T21:51:43.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fb6d006af90 msgr2=0x7fb6d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fb6d006af90 0x7fb6d006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8036000 tx=0x7fb6d803d040 comp rx=0 tx=0).stop 2023-11-28T21:51:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00ae0b0 msgr2=0x7fb6f014ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00ae0b0 0x7fb6f014ba20 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c7a0 tx=0x7fb6e000cc60 comp rx=0 tx=0).stop 2023-11-28T21:51:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 shutdown_connections 2023-11-28T21:51:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fb6d006af90 0x7fb6d006d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6f01607c0 0x7fb6f00b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f00cc900 0x7fb6f00b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 --2- 172.21.15.45:0/1301498477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6f00ae0b0 0x7fb6f014ba20 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.909+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 >> 172.21.15.45:0/1301498477 conn(0x7fb6f00cec30 msgr2=0x7fb6f0070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.911+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 shutdown_connections 2023-11-28T21:51:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:43.911+0000 7fb6f57a5640 1 -- 172.21.15.45:0/1301498477 wait complete. 2023-11-28T21:51:43.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T21:51:44.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.157+0000 7f96212f1640 1 Processor -- start 2023-11-28T21:51:44.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.157+0000 7f96212f1640 1 -- start start 2023-11-28T21:51:44.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.159+0000 7f96212f1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c1598e0 0x7f961c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:44.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.159+0000 7f96212f1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f961c0c9ba0 0x7f961c0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:44.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.159+0000 7f96212f1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961c06cfb0 con 0x7f961c06e050 2023-11-28T21:51:44.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.159+0000 7f96212f1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f961c0ce030 con 0x7f961c0c9ba0 2023-11-28T21:51:44.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.159+0000 7f96212f1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961c159cc0 con 0x7f961c1598e0 2023-11-28T21:51:44.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.160+0000 7f9620af0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f961c0c9ba0 0x7f961c0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:44.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.160+0000 7f9620af0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f961c0c9ba0 0x7f961c0cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:55826/0 (socket says 172.21.15.45:55826) 2023-11-28T21:51:44.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.160+0000 7f961bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c1598e0 0x7f961c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:44.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.160+0000 7f9620af0640 1 -- 172.21.15.45:0/3856111587 learned_addr learned my addr 172.21.15.45:0/3856111587 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:44.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.161+0000 7f9620af0640 1 -- 172.21.15.45:0/3856111587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c1598e0 msgr2=0x7f961c06c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:44.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.161+0000 7f9620af0640 1 --2- 172.21.15.45:0/3856111587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c1598e0 0x7f961c06c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:44.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.161+0000 7f9620af0640 1 -- 172.21.15.45:0/3856111587 >> v1:172.21.15.190:6789/0 conn(0x7f961c06e050 legacy=0x7f961c0c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:44.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.161+0000 7f9620af0640 1 -- 172.21.15.45:0/3856111587 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961c071180 con 0x7f961c0c9ba0 2023-11-28T21:51:44.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.162+0000 7f9620af0640 1 --2- 172.21.15.45:0/3856111587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c0c9ba0 0x7f961c0cdaf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9614009e80 tx=0x7f961402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=482ba377b144b7ab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:44.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.162+0000 7f961affd640 1 -- 172.21.15.45:0/3856111587 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961402cae0 con 0x7f961c0c9ba0 2023-11-28T21:51:44.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.162+0000 7f961affd640 1 -- 172.21.15.45:0/3856111587 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961402cc40 con 0x7f961c0c9ba0 2023-11-28T21:51:44.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.162+0000 7f961affd640 1 -- 172.21.15.45:0/3856111587 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96140343f0 con 0x7f961c0c9ba0 2023-11-28T21:51:44.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.163+0000 7f96212f1640 1 -- 172.21.15.45:0/3856111587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c0c9ba0 msgr2=0x7f961c0cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.163+0000 7f96212f1640 1 --2- 172.21.15.45:0/3856111587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c0c9ba0 0x7f961c0cdaf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9614009e80 tx=0x7f961402bf40 comp rx=0 tx=0).stop 2023-11-28T21:51:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.163+0000 7f96212f1640 1 -- 172.21.15.45:0/3856111587 shutdown_connections 2023-11-28T21:51:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.163+0000 7f96212f1640 1 --2- 172.21.15.45:0/3856111587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c0c9ba0 0x7f961c0cdaf0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.163+0000 7f96212f1640 1 --2- 172.21.15.45:0/3856111587 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c1598e0 0x7f961c06c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.163+0000 7f96212f1640 1 -- 172.21.15.45:0/3856111587 >> 172.21.15.45:0/3856111587 conn(0x7f961c0cec30 msgr2=0x7f961c0cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.163+0000 7f96212f1640 1 -- 172.21.15.45:0/3856111587 shutdown_connections 2023-11-28T21:51:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.163+0000 7f96212f1640 1 -- 172.21.15.45:0/3856111587 wait complete. 2023-11-28T21:51:44.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.165+0000 7f96212f1640 1 Processor -- start 2023-11-28T21:51:44.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.165+0000 7f96212f1640 1 -- start start 2023-11-28T21:51:44.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.166+0000 7f96212f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f961c06e050 0x7f961c159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:44.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.166+0000 7f96212f1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c0c9ba0 0x7f961c14cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:44.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.166+0000 7f96212f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c1598e0 0x7f961c0b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:44.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.166+0000 7f96212f1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0c77b0 con 0x7f961c06e050 2023-11-28T21:51:44.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.166+0000 7f96212f1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c15a380 con 0x7f961c1598e0 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.166+0000 7f96212f1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0c6e80 con 0x7f961c0c9ba0 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f961bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f961c06e050 0x7f961c159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f961bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f961c06e050 0x7f961c159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34440/0 (socket says 172.21.15.45:34440) 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f961bfff640 1 -- 172.21.15.45:0/3348104565 learned_addr learned my addr 172.21.15.45:0/3348104565 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f9620af0640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c1598e0 0x7f961c0b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f961b7fe640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c0c9ba0 0x7f961c14cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f961bfff640 1 -- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c0c9ba0 msgr2=0x7f961c14cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f961bfff640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c0c9ba0 0x7f961c14cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f961bfff640 1 -- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c1598e0 msgr2=0x7f961c0b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:44.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f961bfff640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c1598e0 0x7f961c0b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:44.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.167+0000 7f961bfff640 1 -- 172.21.15.45:0/3348104565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614009b30 con 0x7f961c06e050 2023-11-28T21:51:44.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.168+0000 7f961bfff640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f961c06e050 0x7f961c159e40 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f960c00e8a0 tx=0x7f960c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:44.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.168+0000 7f9618ff9640 1 -- 172.21.15.45:0/3348104565 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c0094f0 con 0x7f961c06e050 2023-11-28T21:51:44.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.168+0000 7f9618ff9640 1 -- 172.21.15.45:0/3348104565 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c00c940 con 0x7f961c06e050 2023-11-28T21:51:44.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.168+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c0c7160 con 0x7f961c06e050 2023-11-28T21:51:44.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.168+0000 7f9618ff9640 1 -- 172.21.15.45:0/3348104565 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c00ccf0 con 0x7f961c06e050 2023-11-28T21:51:44.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.168+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c0b2b40 con 0x7f961c06e050 2023-11-28T21:51:44.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.169+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c0733d0 con 0x7f961c06e050 2023-11-28T21:51:44.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.171+0000 7f9618ff9640 1 -- 172.21.15.45:0/3348104565 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f960c013470 con 0x7f961c06e050 2023-11-28T21:51:44.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.172+0000 7f9618ff9640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f960806af90 0x7f960806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:44.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.172+0000 7f9618ff9640 1 -- 172.21.15.45:0/3348104565 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f960c089fc0 con 0x7f961c06e050 2023-11-28T21:51:44.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.173+0000 7f961b7fe640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f960806af90 0x7f960806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:44.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.174+0000 7f961b7fe640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f960806af90 0x7f960806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f961c0c2350 tx=0x7f9610009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:44.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.176+0000 7f9618ff9640 1 -- 172.21.15.45:0/3348104565 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f960c054fd0 con 0x7f961c06e050 2023-11-28T21:51:44.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.373+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f961c0b4bb0 con 0x7f961c06e050 2023-11-28T21:51:44.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.912+0000 7f9618ff9640 1 -- 172.21.15.45:0/3348104565 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v183) v1 ==== 170+0+0 (secure 0 0 0) 0x7f960c0585f0 con 0x7f961c06e050 2023-11-28T21:51:44.914 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T21:51:44.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f960806af90 msgr2=0x7f960806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:44.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f960806af90 0x7f960806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f961c0c2350 tx=0x7f9610009040 comp rx=0 tx=0).stop 2023-11-28T21:51:44.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f961c06e050 msgr2=0x7f961c159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:44.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f961c06e050 0x7f961c159e40 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f960c00e8a0 tx=0x7f960c00ed60 comp rx=0 tx=0).stop 2023-11-28T21:51:44.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 shutdown_connections 2023-11-28T21:51:44.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f960806af90 0x7f960806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:44.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c1598e0 0x7f961c0b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:44.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f961c0c9ba0 0x7f961c14cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:44.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 --2- 172.21.15.45:0/3348104565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f961c06e050 0x7f961c159e40 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 >> 172.21.15.45:0/3348104565 conn(0x7f961c0cec30 msgr2=0x7f961c15d540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.917+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 shutdown_connections 2023-11-28T21:51:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:44.918+0000 7f96212f1640 1 -- 172.21.15.45:0/3348104565 wait complete. 2023-11-28T21:51:44.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T21:51:45.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.151+0000 7f077ebdb640 1 Processor -- start 2023-11-28T21:51:45.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.151+0000 7f077ebdb640 1 -- start start 2023-11-28T21:51:45.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.153+0000 7f077ebdb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07780b3040 0x7f077806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:45.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.153+0000 7f077ebdb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077806d410 con 0x7f07780b3420 2023-11-28T21:51:45.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.153+0000 7f077ebdb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077806d6f0 con 0x7f077815b270 2023-11-28T21:51:45.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.153+0000 7f077ebdb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07780ceaa0 con 0x7f07780b3040 2023-11-28T21:51:45.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.153+0000 7f077e3da640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07780b3420 0x7f07780ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60892/0 (socket says 172.21.15.45:60892) 2023-11-28T21:51:45.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.153+0000 7f077e3da640 1 -- 172.21.15.45:0/1808441846 learned_addr learned my addr 172.21.15.45:0/1808441846 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:45.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.153+0000 7f077dbd9640 1 --2- 172.21.15.45:0/1808441846 >> v2:172.21.15.45:3300/0 conn(0x7f07780b3040 0x7f077806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:45.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.154+0000 7f077dbd9640 1 -- 172.21.15.45:0/1808441846 >> v1:172.21.15.190:6789/0 conn(0x7f077815b270 legacy=0x7f077815f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:45.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.154+0000 7f077dbd9640 1 -- 172.21.15.45:0/1808441846 >> v1:172.21.15.45:6789/0 conn(0x7f07780b3420 legacy=0x7f07780ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:45.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.154+0000 7f077dbd9640 1 -- 172.21.15.45:0/1808441846 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077806d9d0 con 0x7f07780b3040 2023-11-28T21:51:45.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.155+0000 7f077dbd9640 1 --2- 172.21.15.45:0/1808441846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3040 0x7f077806cda0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f07740097a0 tx=0x7f077402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0ccee0bd956af server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:45.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.155+0000 7f077cbd7640 1 -- 172.21.15.45:0/1808441846 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077402f050 con 0x7f07780b3040 2023-11-28T21:51:45.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.155+0000 7f077cbd7640 1 -- 172.21.15.45:0/1808441846 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0774031700 con 0x7f07780b3040 2023-11-28T21:51:45.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.155+0000 7f077cbd7640 1 -- 172.21.15.45:0/1808441846 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0774031a90 con 0x7f07780b3040 2023-11-28T21:51:45.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.156+0000 7f077ebdb640 1 -- 172.21.15.45:0/1808441846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3040 msgr2=0x7f077806cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:45.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.156+0000 7f077ebdb640 1 --2- 172.21.15.45:0/1808441846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3040 0x7f077806cda0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f07740097a0 tx=0x7f077402b7a0 comp rx=0 tx=0).stop 2023-11-28T21:51:45.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.156+0000 7f077ebdb640 1 -- 172.21.15.45:0/1808441846 shutdown_connections 2023-11-28T21:51:45.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.156+0000 7f077ebdb640 1 --2- 172.21.15.45:0/1808441846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3040 0x7f077806cda0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:45.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.156+0000 7f077ebdb640 1 -- 172.21.15.45:0/1808441846 >> 172.21.15.45:0/1808441846 conn(0x7f07780b1050 msgr2=0x7f07780cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:45.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.156+0000 7f077ebdb640 1 -- 172.21.15.45:0/1808441846 shutdown_connections 2023-11-28T21:51:45.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.156+0000 7f077ebdb640 1 -- 172.21.15.45:0/1808441846 wait complete. 2023-11-28T21:51:45.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.158+0000 7f077ebdb640 1 Processor -- start 2023-11-28T21:51:45.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.158+0000 7f077ebdb640 1 -- start start 2023-11-28T21:51:45.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.159+0000 7f077ebdb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780b3040 0x7f07781472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:45.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.159+0000 7f077ebdb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3420 0x7f07781571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:45.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.159+0000 7f077ebdb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f077815b270 0x7f0778157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:45.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.159+0000 7f077ebdb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0778159d60 con 0x7f07780b3420 2023-11-28T21:51:45.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.159+0000 7f077ebdb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077815a040 con 0x7f077815b270 2023-11-28T21:51:45.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.159+0000 7f077ebdb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077815a320 con 0x7f07780b3040 2023-11-28T21:51:45.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.159+0000 7f077d3d8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3420 0x7f07781571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:45.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.159+0000 7f077d3d8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3420 0x7f07781571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34456/0 (socket says 172.21.15.45:34456) 2023-11-28T21:51:45.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.159+0000 7f077d3d8640 1 -- 172.21.15.45:0/920354396 learned_addr learned my addr 172.21.15.45:0/920354396 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:45.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.160+0000 7f077e3da640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f077815b270 0x7f0778157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:45.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.160+0000 7f077dbd9640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780b3040 0x7f07781472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:45.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.160+0000 7f077d3d8640 1 -- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780b3040 msgr2=0x7f07781472e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:45.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.160+0000 7f077d3d8640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780b3040 0x7f07781472e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:45.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.160+0000 7f077d3d8640 1 -- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f077815b270 msgr2=0x7f0778157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:45.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.160+0000 7f077d3d8640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f077815b270 0x7f0778157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:45.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.160+0000 7f077d3d8640 1 -- 172.21.15.45:0/920354396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0774009400 con 0x7f07780b3420 2023-11-28T21:51:45.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.160+0000 7f077d3d8640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3420 0x7f07781571e0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f076800b3c0 tx=0x7f076800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:45.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.161+0000 7f076e7fc640 1 -- 172.21.15.45:0/920354396 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768011830 con 0x7f07780b3420 2023-11-28T21:51:45.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.161+0000 7f076e7fc640 1 -- 172.21.15.45:0/920354396 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0768011990 con 0x7f07780b3420 2023-11-28T21:51:45.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.161+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077815a600 con 0x7f07780b3420 2023-11-28T21:51:45.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.161+0000 7f076e7fc640 1 -- 172.21.15.45:0/920354396 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768011d80 con 0x7f07780b3420 2023-11-28T21:51:45.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.161+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0778159910 con 0x7f07780b3420 2023-11-28T21:51:45.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.162+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0778159b30 con 0x7f07780b3420 2023-11-28T21:51:45.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.163+0000 7f076e7fc640 1 -- 172.21.15.45:0/920354396 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f076801b460 con 0x7f07780b3420 2023-11-28T21:51:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.164+0000 7f076e7fc640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f075c06af60 0x7f075c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.164+0000 7f076e7fc640 1 -- 172.21.15.45:0/920354396 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f07680894c0 con 0x7f07780b3420 2023-11-28T21:51:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.165+0000 7f077dbd9640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f075c06af60 0x7f075c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.166+0000 7f077dbd9640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f075c06af60 0x7f075c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0774035000 tx=0x7f0774008030 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:45.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.170+0000 7f076e7fc640 1 -- 172.21.15.45:0/920354396 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07680542c0 con 0x7f07780b3420 2023-11-28T21:51:45.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.398+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f077809d0f0 con 0x7f07780b3420 2023-11-28T21:51:45.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.939+0000 7f076e7fc640 1 -- 172.21.15.45:0/920354396 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v80) v1 ==== 174+0+0 (secure 0 0 0) 0x7f07680578e0 con 0x7f07780b3420 2023-11-28T21:51:45.941 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 21 and data pool 22 2023-11-28T21:51:45.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f075c06af60 msgr2=0x7f075c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:45.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f075c06af60 0x7f075c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0774035000 tx=0x7f0774008030 comp rx=0 tx=0).stop 2023-11-28T21:51:45.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3420 msgr2=0x7f07781571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:45.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3420 0x7f07781571e0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f076800b3c0 tx=0x7f076800b880 comp rx=0 tx=0).stop 2023-11-28T21:51:45.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 shutdown_connections 2023-11-28T21:51:45.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f075c06af60 0x7f075c06d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:45.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f077815b270 0x7f0778157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:45.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07780b3420 0x7f07781571e0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:45.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 --2- 172.21.15.45:0/920354396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780b3040 0x7f07781472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:45.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 >> 172.21.15.45:0/920354396 conn(0x7f07780b1050 msgr2=0x7f07780ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:45.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 shutdown_connections 2023-11-28T21:51:45.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:45.943+0000 7f077ebdb640 1 -- 172.21.15.45:0/920354396 wait complete. 2023-11-28T21:51:45.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:51:46.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.158+0000 7f10b8832640 1 Processor -- start 2023-11-28T21:51:46.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.159+0000 7f10b8832640 1 -- start start 2023-11-28T21:51:46.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.160+0000 7f10b8832640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f10b4155780 0x7f10b40abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.160+0000 7f10b8832640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f10b40ce800 0x7f10b40b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.160+0000 7f10b8832640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b415c3c0 con 0x7f10b40ae540 2023-11-28T21:51:46.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.160+0000 7f10b8832640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b4155b60 con 0x7f10b4155780 2023-11-28T21:51:46.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.160+0000 7f10b8832640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b415ba90 con 0x7f10b40ce800 2023-11-28T21:51:46.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b2ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f10b40ae540 0x7f10b40ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60904/0 (socket says 172.21.15.45:60904) 2023-11-28T21:51:46.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b2ffd640 1 -- 172.21.15.45:0/3400733151 learned_addr learned my addr 172.21.15.45:0/3400733151 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:46.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b37fe640 1 --2- 172.21.15.45:0/3400733151 >> v2:172.21.15.45:3300/0 conn(0x7f10b4155780 0x7f10b40abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b3fff640 1 --2- 172.21.15.45:0/3400733151 >> v2:172.21.15.190:3300/0 conn(0x7f10b40ce800 0x7f10b40b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b27fc640 1 -- 172.21.15.45:0/3400733151 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586401571 0 0) 0x7f10b415c3c0 con 0x7f10b40ae540 2023-11-28T21:51:46.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b27fc640 1 -- 172.21.15.45:0/3400733151 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1094003520 con 0x7f10b40ae540 2023-11-28T21:51:46.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b37fe640 1 -- 172.21.15.45:0/3400733151 >> v2:172.21.15.190:3300/0 conn(0x7f10b40ce800 msgr2=0x7f10b40b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:46.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b37fe640 1 --2- 172.21.15.45:0/3400733151 >> v2:172.21.15.190:3300/0 conn(0x7f10b40ce800 0x7f10b40b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b37fe640 1 -- 172.21.15.45:0/3400733151 >> v1:172.21.15.45:6789/0 conn(0x7f10b40ae540 legacy=0x7f10b40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.161+0000 7f10b37fe640 1 -- 172.21.15.45:0/3400733151 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b415bd70 con 0x7f10b4155780 2023-11-28T21:51:46.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.162+0000 7f10b37fe640 1 --2- 172.21.15.45:0/3400733151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b4155780 0x7f10b40abd70 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f10b40b16c0 tx=0x7f10a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5394856129308d5e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:46.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.163+0000 7f10b27fc640 1 -- 172.21.15.45:0/3400733151 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4031830 con 0x7f10b4155780 2023-11-28T21:51:46.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.163+0000 7f10b27fc640 1 -- 172.21.15.45:0/3400733151 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a4031990 con 0x7f10b4155780 2023-11-28T21:51:46.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.163+0000 7f10b27fc640 1 -- 172.21.15.45:0/3400733151 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4031d20 con 0x7f10b4155780 2023-11-28T21:51:46.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.163+0000 7f10b8832640 1 -- 172.21.15.45:0/3400733151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b4155780 msgr2=0x7f10b40abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.163+0000 7f10b8832640 1 --2- 172.21.15.45:0/3400733151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b4155780 0x7f10b40abd70 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f10b40b16c0 tx=0x7f10a402b8d0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.164+0000 7f10b8832640 1 -- 172.21.15.45:0/3400733151 shutdown_connections 2023-11-28T21:51:46.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.164+0000 7f10b8832640 1 --2- 172.21.15.45:0/3400733151 >> v2:172.21.15.190:3300/0 conn(0x7f10b40ce800 0x7f10b40b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.164+0000 7f10b8832640 1 --2- 172.21.15.45:0/3400733151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b4155780 0x7f10b40abd70 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.164+0000 7f10b8832640 1 -- 172.21.15.45:0/3400733151 >> 172.21.15.45:0/3400733151 conn(0x7f10b406ab30 msgr2=0x7f10b40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.164+0000 7f10b8832640 1 -- 172.21.15.45:0/3400733151 shutdown_connections 2023-11-28T21:51:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.164+0000 7f10b8832640 1 -- 172.21.15.45:0/3400733151 wait complete. 2023-11-28T21:51:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.165+0000 7f10b8832640 1 Processor -- start 2023-11-28T21:51:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.166+0000 7f10b8832640 1 -- start start 2023-11-28T21:51:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b8832640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b40ae540 0x7f10b4071590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b8832640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b40ce800 0x7f10b40c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b8832640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10b4155780 0x7f10b40cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b8832640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40c4c00 con 0x7f10b40ae540 2023-11-28T21:51:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b8832640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40c4e80 con 0x7f10b4155780 2023-11-28T21:51:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b8832640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40c5160 con 0x7f10b40ce800 2023-11-28T21:51:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b40ae540 0x7f10b4071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b40ae540 0x7f10b4071590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34484/0 (socket says 172.21.15.45:34484) 2023-11-28T21:51:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b37fe640 1 -- 172.21.15.45:0/1609485178 learned_addr learned my addr 172.21.15.45:0/1609485178 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b2ffd640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b40ce800 0x7f10b40c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b37fe640 1 -- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b40ce800 msgr2=0x7f10b40c46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b37fe640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b40ce800 0x7f10b40c46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b37fe640 1 -- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10b4155780 msgr2=0x7f10b40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b37fe640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10b4155780 0x7f10b40cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.167+0000 7f10b37fe640 1 -- 172.21.15.45:0/1609485178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a4009400 con 0x7f10b40ae540 2023-11-28T21:51:46.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.168+0000 7f10b37fe640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b40ae540 0x7f10b4071590 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f10a4000c00 tx=0x7f10a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:46.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.168+0000 7f10abfff640 1 -- 172.21.15.45:0/1609485178 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a403a8f0 con 0x7f10b40ae540 2023-11-28T21:51:46.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.168+0000 7f10abfff640 1 -- 172.21.15.45:0/1609485178 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a403aa70 con 0x7f10b40ae540 2023-11-28T21:51:46.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.168+0000 7f10abfff640 1 -- 172.21.15.45:0/1609485178 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a403ae20 con 0x7f10b40ae540 2023-11-28T21:51:46.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.168+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b40c5440 con 0x7f10b40ae540 2023-11-28T21:51:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.168+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b40c5a00 con 0x7f10b40ae540 2023-11-28T21:51:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.169+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b40b1030 con 0x7f10b40ae540 2023-11-28T21:51:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.170+0000 7f10abfff640 1 -- 172.21.15.45:0/1609485178 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f10a404d450 con 0x7f10b40ae540 2023-11-28T21:51:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.171+0000 7f10abfff640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f109806af60 0x7f109806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.172+0000 7f10abfff640 1 -- 172.21.15.45:0/1609485178 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f10a40abaf0 con 0x7f10b40ae540 2023-11-28T21:51:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.172+0000 7f10b2ffd640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f109806af60 0x7f109806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.173+0000 7f10b2ffd640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f109806af60 0x7f109806d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10b40b0710 tx=0x7f109c009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.176+0000 7f10abfff640 1 -- 172.21.15.45:0/1609485178 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10a4076780 con 0x7f10b40ae540 2023-11-28T21:51:46.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.363+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10b404e920 con 0x7f10b40ae540 2023-11-28T21:51:46.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.366+0000 7f10abfff640 1 -- 172.21.15.45:0/1609485178 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+16769 (secure 0 0 0) 0x7f10a4035020 con 0x7f10b40ae540 2023-11-28T21:51:46.368 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:46.368 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":184,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:51:45.399588+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:51:43.335999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-11-28T21:51:44.431458+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:51:46.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.368+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f109806af60 msgr2=0x7f109806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.368+0000 7f10b8832640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f109806af60 0x7f109806d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10b40b0710 tx=0x7f109c009040 comp rx=0 tx=0).stop 2023-11-28T21:51:46.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.368+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b40ae540 msgr2=0x7f10b4071590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.369+0000 7f10b8832640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b40ae540 0x7f10b4071590 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f10a4000c00 tx=0x7f10a40046c0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.369+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 shutdown_connections 2023-11-28T21:51:46.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.369+0000 7f10b8832640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f109806af60 0x7f109806d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.369+0000 7f10b8832640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10b4155780 0x7f10b40cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.369+0000 7f10b8832640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b40ce800 0x7f10b40c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.369+0000 7f10b8832640 1 --2- 172.21.15.45:0/1609485178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b40ae540 0x7f10b4071590 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.369+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 >> 172.21.15.45:0/1609485178 conn(0x7f10b406ab30 msgr2=0x7f10b406d090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:46.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.369+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 shutdown_connections 2023-11-28T21:51:46.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.369+0000 7f10b8832640 1 -- 172.21.15.45:0/1609485178 wait complete. 2023-11-28T21:51:46.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:51:46.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.613+0000 7f7056f0f640 1 Processor -- start 2023-11-28T21:51:46.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.613+0000 7f7056f0f640 1 -- start start 2023-11-28T21:51:46.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.614+0000 7f7056f0f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f70501402f0 0x7f7050160e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.614+0000 7f7056f0f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7050161df0 con 0x7f70501434c0 2023-11-28T21:51:46.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.614+0000 7f7056f0f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70501406d0 con 0x7f70500703b0 2023-11-28T21:51:46.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.614+0000 7f7056f0f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7050143950 con 0x7f70501402f0 2023-11-28T21:51:46.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.615+0000 7f704ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f70501434c0 0x7f70501409f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60910/0 (socket says 172.21.15.45:60910) 2023-11-28T21:51:46.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.615+0000 7f704ffff640 1 -- 172.21.15.45:0/4017596031 learned_addr learned my addr 172.21.15.45:0/4017596031 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:46.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.615+0000 7f7054cab640 1 --2- 172.21.15.45:0/4017596031 >> v2:172.21.15.45:3300/0 conn(0x7f70501402f0 0x7f7050160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.615+0000 7f7054cab640 1 -- 172.21.15.45:0/4017596031 >> v1:172.21.15.190:6789/0 conn(0x7f70500703b0 legacy=0x7f70500d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:46.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.615+0000 7f7054cab640 1 -- 172.21.15.45:0/4017596031 >> v1:172.21.15.45:6789/0 conn(0x7f70501434c0 legacy=0x7f70501409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.615+0000 7f7054cab640 1 -- 172.21.15.45:0/4017596031 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7050161550 con 0x7f70501402f0 2023-11-28T21:51:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.616+0000 7f7054cab640 1 --2- 172.21.15.45:0/4017596031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501402f0 0x7f7050160e70 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f70400097a0 tx=0x7f704002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1254888f8e513c6f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.617+0000 7f704f7fe640 1 -- 172.21.15.45:0/4017596031 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704002e070 con 0x7f70501402f0 2023-11-28T21:51:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.617+0000 7f704f7fe640 1 -- 172.21.15.45:0/4017596031 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70400315a0 con 0x7f70501402f0 2023-11-28T21:51:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.617+0000 7f704f7fe640 1 -- 172.21.15.45:0/4017596031 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040031930 con 0x7f70501402f0 2023-11-28T21:51:46.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.617+0000 7f7056f0f640 1 -- 172.21.15.45:0/4017596031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501402f0 msgr2=0x7f7050160e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.617+0000 7f7056f0f640 1 --2- 172.21.15.45:0/4017596031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501402f0 0x7f7050160e70 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f70400097a0 tx=0x7f704002b840 comp rx=0 tx=0).stop 2023-11-28T21:51:46.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.618+0000 7f7056f0f640 1 -- 172.21.15.45:0/4017596031 shutdown_connections 2023-11-28T21:51:46.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.618+0000 7f7056f0f640 1 --2- 172.21.15.45:0/4017596031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501402f0 0x7f7050160e70 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.618+0000 7f7056f0f640 1 -- 172.21.15.45:0/4017596031 >> 172.21.15.45:0/4017596031 conn(0x7f705006e7c0 msgr2=0x7f705006ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:46.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.618+0000 7f7056f0f640 1 -- 172.21.15.45:0/4017596031 shutdown_connections 2023-11-28T21:51:46.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.618+0000 7f7056f0f640 1 -- 172.21.15.45:0/4017596031 wait complete. 2023-11-28T21:51:46.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.619+0000 7f7056f0f640 1 Processor -- start 2023-11-28T21:51:46.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.620+0000 7f7056f0f640 1 -- start start 2023-11-28T21:51:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f7056f0f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70500703b0 0x7f705014f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f7056f0f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70501402f0 0x7f70500b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f7056f0f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501434c0 0x7f70500b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f7056f0f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705014ff20 con 0x7f70501434c0 2023-11-28T21:51:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f7056f0f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70501e76d0 con 0x7f70500703b0 2023-11-28T21:51:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f7056f0f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7050160730 con 0x7f70501402f0 2023-11-28T21:51:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f70554ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501434c0 0x7f70500b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f70554ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501434c0 0x7f70500b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34512/0 (socket says 172.21.15.45:34512) 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f70554ac640 1 -- 172.21.15.45:0/925179800 learned_addr learned my addr 172.21.15.45:0/925179800 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f7054cab640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70500703b0 0x7f705014f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f704ffff640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70501402f0 0x7f70500b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.621+0000 7f70554ac640 1 -- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70501402f0 msgr2=0x7f70500b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f70554ac640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70501402f0 0x7f70500b4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f70554ac640 1 -- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70500703b0 msgr2=0x7f705014f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f70554ac640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70500703b0 0x7f705014f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f70554ac640 1 -- 172.21.15.45:0/925179800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040009400 con 0x7f70501434c0 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f7054cab640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70500703b0 0x7f705014f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f70554ac640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501434c0 0x7f70500b5530 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f704800b970 tx=0x7f704800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f704d7fa640 1 -- 172.21.15.45:0/925179800 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704800cb40 con 0x7f70501434c0 2023-11-28T21:51:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f704d7fa640 1 -- 172.21.15.45:0/925179800 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704800cca0 con 0x7f70501434c0 2023-11-28T21:51:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70501e6de0 con 0x7f70501434c0 2023-11-28T21:51:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f704d7fa640 1 -- 172.21.15.45:0/925179800 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048014680 con 0x7f70501434c0 2023-11-28T21:51:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.622+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70501e7360 con 0x7f70501434c0 2023-11-28T21:51:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.624+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70501e7860 con 0x7f70501434c0 2023-11-28T21:51:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.625+0000 7f704d7fa640 1 -- 172.21.15.45:0/925179800 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f704801d3b0 con 0x7f70501434c0 2023-11-28T21:51:46.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.626+0000 7f704d7fa640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f703006af60 0x7f703006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:46.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.626+0000 7f704d7fa640 1 -- 172.21.15.45:0/925179800 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7048089730 con 0x7f70501434c0 2023-11-28T21:51:46.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.627+0000 7f7054cab640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f703006af60 0x7f703006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:46.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.628+0000 7f7054cab640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f703006af60 0x7f703006d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7040036000 tx=0x7f704003d040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:46.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.634+0000 7f704d7fa640 1 -- 172.21.15.45:0/925179800 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7048054560 con 0x7f70501434c0 2023-11-28T21:51:46.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.830+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70500733d0 con 0x7f70501434c0 2023-11-28T21:51:46.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.833+0000 7f704d7fa640 1 -- 172.21.15.45:0/925179800 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+16769 (secure 0 0 0) 0x7f7048057b80 con 0x7f70501434c0 2023-11-28T21:51:46.835 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:46.835 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":184,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:51:45.399588+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:51:43.335999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-11-28T21:51:44.431458+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:51:46.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f703006af60 msgr2=0x7f703006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f703006af60 0x7f703006d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7040036000 tx=0x7f704003d040 comp rx=0 tx=0).stop 2023-11-28T21:51:46.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501434c0 msgr2=0x7f70500b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:46.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501434c0 0x7f70500b5530 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f704800b970 tx=0x7f704800be30 comp rx=0 tx=0).stop 2023-11-28T21:51:46.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 shutdown_connections 2023-11-28T21:51:46.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f703006af60 0x7f703006d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70501434c0 0x7f70500b5530 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.838 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70501402f0 0x7f70500b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 --2- 172.21.15.45:0/925179800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70500703b0 0x7f705014f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:46.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.836+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 >> 172.21.15.45:0/925179800 conn(0x7f705006e7c0 msgr2=0x7f705013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:46.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.837+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 shutdown_connections 2023-11-28T21:51:46.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:46.837+0000 7f7056f0f640 1 -- 172.21.15.45:0/925179800 wait complete. 2023-11-28T21:51:46.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T21:51:47.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.083+0000 7f7095a2e640 1 Processor -- start 2023-11-28T21:51:47.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.084+0000 7f7095a2e640 1 -- start start 2023-11-28T21:51:47.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.085+0000 7f7095a2e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70900d0040 0x7f70900d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:47.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.085+0000 7f7095a2e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f70900b2290 0x7f70900d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:47.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.085+0000 7f7095a2e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70900b90a0 con 0x7f70900b1d40 2023-11-28T21:51:47.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.085+0000 7f7095a2e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70900d1920 con 0x7f70900b2290 2023-11-28T21:51:47.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.085+0000 7f7095a2e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70900d2dc0 con 0x7f70900d0040 2023-11-28T21:51:47.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f709522d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f70900b1d40 0x7f70900d3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60924/0 (socket says 172.21.15.45:60924) 2023-11-28T21:51:47.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f709522d640 1 -- 172.21.15.45:0/1097734107 learned_addr learned my addr 172.21.15.45:0/1097734107 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:47.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f708ffff640 1 --2- 172.21.15.45:0/1097734107 >> v2:172.21.15.190:3300/0 conn(0x7f70900b2290 0x7f70900d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:47.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f7094a2c640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70900d0040 0x7f70900d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:47.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619584052 0 0) 0x7f70900b90a0 con 0x7f70900b1d40 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7078003520 con 0x7f70900b1d40 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3206371666 0 0) 0x7f7078003520 con 0x7f70900b1d40 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70900d0040 msgr2=0x7f70900d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f708f7fe640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70900d0040 0x7f70900d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 >> v2:172.21.15.190:3300/0 conn(0x7f70900b2290 msgr2=0x7f70900d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.086+0000 7f708f7fe640 1 --2- 172.21.15.45:0/1097734107 >> v2:172.21.15.190:3300/0 conn(0x7f70900b2290 0x7f70900d3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70900d0960 con 0x7f70900b1d40 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f70880019f0 con 0x7f70900b1d40 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 >> v1:172.21.15.45:6789/0 conn(0x7f70900b1d40 legacy=0x7f70900d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f708f7fe640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7078003970 0x7f7078003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f708f7fe640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70900d0040 0x7f70780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:47.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f708f7fe640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7078004d00 0x7f7078005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:47.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70900d1920 con 0x7f7078004d00 2023-11-28T21:51:47.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70900d2dc0 con 0x7f7078003970 2023-11-28T21:51:47.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707800b9b0 con 0x7f70900d0040 2023-11-28T21:51:47.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.087+0000 7f709522d640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7078004d00 0x7f7078005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:47.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.088+0000 7f708ffff640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70900d0040 0x7f70780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.088+0000 7f7094a2c640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7078003970 0x7f7078003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.088+0000 7f709522d640 1 -- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70900d0040 msgr2=0x7f70780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.088+0000 7f709522d640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70900d0040 0x7f70780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.088+0000 7f709522d640 1 -- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7078003970 msgr2=0x7f7078003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.088+0000 7f709522d640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7078003970 0x7f7078003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.088+0000 7f709522d640 1 -- 172.21.15.45:0/1097734107 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078003520 con 0x7f7078004d00 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.089+0000 7f709522d640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7078004d00 0x7f7078005180 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f708800a780 tx=0x7f70780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a73093842f95316 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7088034630 con 0x7f7078004d00 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708802f070 con 0x7f7078004d00 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f708f7fe640 1 -- 172.21.15.45:0/1097734107 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70880349d0 con 0x7f7078004d00 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f7095a2e640 1 -- 172.21.15.45:0/1097734107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7078004d00 msgr2=0x7f7078005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:47.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f7095a2e640 1 --2- 172.21.15.45:0/1097734107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7078004d00 0x7f7078005180 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f708800a780 tx=0x7f70780057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f709522d640 1 -- 172.21.15.45:0/1097734107 reap_dead start 2023-11-28T21:51:47.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f7095a2e640 1 -- 172.21.15.45:0/1097734107 shutdown_connections 2023-11-28T21:51:47.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f7095a2e640 1 -- 172.21.15.45:0/1097734107 >> 172.21.15.45:0/1097734107 conn(0x7f7090070f40 msgr2=0x7f7090071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:47.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f7095a2e640 1 -- 172.21.15.45:0/1097734107 shutdown_connections 2023-11-28T21:51:47.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.090+0000 7f7095a2e640 1 -- 172.21.15.45:0/1097734107 wait complete. 2023-11-28T21:51:47.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.092+0000 7f7095a2e640 1 Processor -- start 2023-11-28T21:51:47.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.093+0000 7f7095a2e640 1 -- start start 2023-11-28T21:51:47.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f7095a2e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7078004d00 0x7f7090156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f7095a2e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70900c8f60 0x7f7090155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f7095a2e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090156370 0x7f7090158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f7095a2e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70901596b0 con 0x7f70900c8f60 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f7095a2e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70901572a0 con 0x7f7090156370 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f7095a2e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70901568e0 con 0x7f7078004d00 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f708ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70900c8f60 0x7f7090155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f708ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70900c8f60 0x7f7090155790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34528/0 (socket says 172.21.15.45:34528) 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f708ffff640 1 -- 172.21.15.45:0/2202754658 learned_addr learned my addr 172.21.15.45:0/2202754658 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f708ffff640 1 -- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7078004d00 msgr2=0x7f7090156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f709522d640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090156370 0x7f7090158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:47.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f7094a2c640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7078004d00 0x7f7090156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:47.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f708ffff640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7078004d00 0x7f7090156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f708ffff640 1 -- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090156370 msgr2=0x7f7090158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:47.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f708ffff640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090156370 0x7f7090158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.094+0000 7f708ffff640 1 -- 172.21.15.45:0/2202754658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70900c81a0 con 0x7f70900c8f60 2023-11-28T21:51:47.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.095+0000 7f708ffff640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70900c8f60 0x7f7090155790 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f70840116d0 tx=0x7f7084011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:47.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.095+0000 7f708d7fa640 1 -- 172.21.15.45:0/2202754658 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400f9c0 con 0x7f70900c8f60 2023-11-28T21:51:47.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.095+0000 7f708d7fa640 1 -- 172.21.15.45:0/2202754658 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708400fb20 con 0x7f70900c8f60 2023-11-28T21:51:47.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.095+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70900c8480 con 0x7f70900c8f60 2023-11-28T21:51:47.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.095+0000 7f708d7fa640 1 -- 172.21.15.45:0/2202754658 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400fe90 con 0x7f70900c8f60 2023-11-28T21:51:47.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.095+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70900c8a40 con 0x7f70900c8f60 2023-11-28T21:51:47.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.096+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7090071b30 con 0x7f70900c8f60 2023-11-28T21:51:47.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.097+0000 7f708d7fa640 1 -- 172.21.15.45:0/2202754658 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7084015460 con 0x7f70900c8f60 2023-11-28T21:51:47.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.098+0000 7f708d7fa640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f707c06af90 0x7f707c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:47.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.099+0000 7f708d7fa640 1 -- 172.21.15.45:0/2202754658 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f708408a1f0 con 0x7f70900c8f60 2023-11-28T21:51:47.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.099+0000 7f7094a2c640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f707c06af90 0x7f707c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:47.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.101+0000 7f7094a2c640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f707c06af90 0x7f707c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7080009950 tx=0x7f7080008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:47.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.103+0000 7f708d7fa640 1 -- 172.21.15.45:0/2202754658 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7084055020 con 0x7f70900c8f60 2023-11-28T21:51:47.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.341+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f70900b82d0 con 0x7f70900c8f60 2023-11-28T21:51:47.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.961+0000 7f708d7fa640 1 -- 172.21.15.45:0/2202754658 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v83) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7084058640 con 0x7f70900c8f60 2023-11-28T21:51:47.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f707c06af90 msgr2=0x7f707c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:47.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f707c06af90 0x7f707c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7080009950 tx=0x7f7080008040 comp rx=0 tx=0).stop 2023-11-28T21:51:47.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70900c8f60 msgr2=0x7f7090155790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:47.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70900c8f60 0x7f7090155790 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f70840116d0 tx=0x7f7084011b90 comp rx=0 tx=0).stop 2023-11-28T21:51:47.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 shutdown_connections 2023-11-28T21:51:47.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f707c06af90 0x7f707c06d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7090156370 0x7f7090158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70900c8f60 0x7f7090155790 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 --2- 172.21.15.45:0/2202754658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7078004d00 0x7f7090156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:47.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.964+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 >> 172.21.15.45:0/2202754658 conn(0x7f7090070f40 msgr2=0x7f70900ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:47.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.965+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 shutdown_connections 2023-11-28T21:51:47.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:47.965+0000 7f7095a2e640 1 -- 172.21.15.45:0/2202754658 wait complete. 2023-11-28T21:51:47.988 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T21:51:47.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T21:51:48.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.214+0000 7f1ddbae0640 1 Processor -- start 2023-11-28T21:51:48.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.214+0000 7f1ddbae0640 1 -- start start 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.216+0000 7f1ddbae0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1dd415b3c0 0x7f1dd4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.216+0000 7f1ddbae0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1dd40d1b40 0x7f1dd4099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.216+0000 7f1ddbae0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1dd40c53a0 0x7f1dd40c5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.216+0000 7f1ddbae0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd415b7a0 con 0x7f1dd40c53a0 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.216+0000 7f1ddbae0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd40c4f10 con 0x7f1dd40d1b40 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.216+0000 7f1ddbae0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40c4c10 con 0x7f1dd415b3c0 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.216+0000 7f1dda07d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1dd40c53a0 0x7f1dd40c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.216+0000 7f1dda07d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1dd40c53a0 0x7f1dd40c5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34530/0 (socket says 172.21.15.45:34530) 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.216+0000 7f1dda07d640 1 -- 172.21.15.45:0/2387178090 learned_addr learned my addr 172.21.15.45:0/2387178090 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:48.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.217+0000 7f1dda07d640 1 -- 172.21.15.45:0/2387178090 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1dd415b3c0 msgr2=0x7f1dd4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:48.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.217+0000 7f1dda07d640 1 --2- 172.21.15.45:0/2387178090 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1dd415b3c0 0x7f1dd4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.217+0000 7f1dda07d640 1 -- 172.21.15.45:0/2387178090 >> v2:172.21.15.190:3300/0 conn(0x7f1dd40d1b40 msgr2=0x7f1dd4099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:48.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.217+0000 7f1dda07d640 1 --2- 172.21.15.45:0/2387178090 >> v2:172.21.15.190:3300/0 conn(0x7f1dd40d1b40 0x7f1dd4099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.217+0000 7f1dda07d640 1 -- 172.21.15.45:0/2387178090 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd40cf600 con 0x7f1dd40c53a0 2023-11-28T21:51:48.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.218+0000 7f1dda07d640 1 --2- 172.21.15.45:0/2387178090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40c53a0 0x7f1dd40c5830 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f1dd406e470 tx=0x7f1dcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75007a4110e7e57 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:48.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.218+0000 7f1dd887a640 1 -- 172.21.15.45:0/2387178090 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc031870 con 0x7f1dd40c53a0 2023-11-28T21:51:48.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.218+0000 7f1dd887a640 1 -- 172.21.15.45:0/2387178090 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dcc0319d0 con 0x7f1dd40c53a0 2023-11-28T21:51:48.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.218+0000 7f1dd887a640 1 -- 172.21.15.45:0/2387178090 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc031d60 con 0x7f1dd40c53a0 2023-11-28T21:51:48.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.219+0000 7f1ddbae0640 1 -- 172.21.15.45:0/2387178090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40c53a0 msgr2=0x7f1dd40c5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.219+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/2387178090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40c53a0 0x7f1dd40c5830 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f1dd406e470 tx=0x7f1dcc02b910 comp rx=0 tx=0).stop 2023-11-28T21:51:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.219+0000 7f1ddbae0640 1 -- 172.21.15.45:0/2387178090 shutdown_connections 2023-11-28T21:51:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.219+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/2387178090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40c53a0 0x7f1dd40c5830 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.219+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/2387178090 >> v2:172.21.15.190:3300/0 conn(0x7f1dd40d1b40 0x7f1dd4099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.219+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/2387178090 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1dd415b3c0 0x7f1dd4099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.219+0000 7f1ddbae0640 1 -- 172.21.15.45:0/2387178090 >> 172.21.15.45:0/2387178090 conn(0x7f1dd40d1020 msgr2=0x7f1dd40d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.219+0000 7f1ddbae0640 1 -- 172.21.15.45:0/2387178090 shutdown_connections 2023-11-28T21:51:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.219+0000 7f1ddbae0640 1 -- 172.21.15.45:0/2387178090 wait complete. 2023-11-28T21:51:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.221+0000 7f1ddbae0640 1 Processor -- start 2023-11-28T21:51:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.221+0000 7f1ddbae0640 1 -- start start 2023-11-28T21:51:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.222+0000 7f1ddbae0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1dd40c53a0 0x7f1dd40b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.222+0000 7f1ddbae0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40d1b40 0x7f1dd4155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.222+0000 7f1ddbae0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1dd415b3c0 0x7f1dd40c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.222+0000 7f1ddbae0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40b4970 con 0x7f1dd40d1b40 2023-11-28T21:51:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.222+0000 7f1ddbae0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40b4c50 con 0x7f1dd40c53a0 2023-11-28T21:51:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.222+0000 7f1ddbae0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40b4f30 con 0x7f1dd415b3c0 2023-11-28T21:51:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.222+0000 7f1dd907b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40d1b40 0x7f1dd4155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.222+0000 7f1dd907b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40d1b40 0x7f1dd4155000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34532/0 (socket says 172.21.15.45:34532) 2023-11-28T21:51:48.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.222+0000 7f1dd907b640 1 -- 172.21.15.45:0/3359862856 learned_addr learned my addr 172.21.15.45:0/3359862856 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:48.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.223+0000 7f1dd907b640 1 -- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1dd415b3c0 msgr2=0x7f1dd40c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:48.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.223+0000 7f1dd907b640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1dd415b3c0 0x7f1dd40c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.223+0000 7f1dd907b640 1 -- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1dd40c53a0 msgr2=0x7f1dd40b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:48.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.223+0000 7f1dd907b640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1dd40c53a0 0x7f1dd40b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.223+0000 7f1dd907b640 1 -- 172.21.15.45:0/3359862856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc0094d0 con 0x7f1dd40d1b40 2023-11-28T21:51:48.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.223+0000 7f1dd907b640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40d1b40 0x7f1dd4155000 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400b410 tx=0x7f1dc400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:48.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.223+0000 7f1dca7fc640 1 -- 172.21.15.45:0/3359862856 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4011830 con 0x7f1dd40d1b40 2023-11-28T21:51:48.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.223+0000 7f1dca7fc640 1 -- 172.21.15.45:0/3359862856 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc40119b0 con 0x7f1dd40d1b40 2023-11-28T21:51:48.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.223+0000 7f1dca7fc640 1 -- 172.21.15.45:0/3359862856 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4011d60 con 0x7f1dd40d1b40 2023-11-28T21:51:48.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.224+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd40b5210 con 0x7f1dd40d1b40 2023-11-28T21:51:48.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.224+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd40b6a00 con 0x7f1dd40d1b40 2023-11-28T21:51:48.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.225+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd4070a10 con 0x7f1dd40d1b40 2023-11-28T21:51:48.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.226+0000 7f1dca7fc640 1 -- 172.21.15.45:0/3359862856 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1dc401a400 con 0x7f1dd40d1b40 2023-11-28T21:51:48.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.227+0000 7f1dca7fc640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1db806af60 0x7f1db806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:48.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.227+0000 7f1dca7fc640 1 -- 172.21.15.45:0/3359862856 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1dc4089b40 con 0x7f1dd40d1b40 2023-11-28T21:51:48.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.227+0000 7f1dd987c640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1db806af60 0x7f1db806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:48.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.229+0000 7f1dd987c640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1db806af60 0x7f1db806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000ab90 tx=0x7f1dd0009250 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:48.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.231+0000 7f1dca7fc640 1 -- 172.21.15.45:0/3359862856 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1dc40548e0 con 0x7f1dd40d1b40 2023-11-28T21:51:48.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.466+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1dd40733d0 con 0x7f1dd40d1b40 2023-11-28T21:51:48.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.974+0000 7f1dca7fc640 1 -- 172.21.15.45:0/3359862856 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v84) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1dc4015020 con 0x7f1dd40d1b40 2023-11-28T21:51:48.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1db806af60 msgr2=0x7f1db806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:48.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1db806af60 0x7f1db806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000ab90 tx=0x7f1dd0009250 comp rx=0 tx=0).stop 2023-11-28T21:51:48.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40d1b40 msgr2=0x7f1dd4155000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:48.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40d1b40 0x7f1dd4155000 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400b410 tx=0x7f1dc400b8d0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 shutdown_connections 2023-11-28T21:51:48.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f1db806af60 0x7f1db806d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1dd415b3c0 0x7f1dd40c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dd40d1b40 0x7f1dd4155000 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 --2- 172.21.15.45:0/3359862856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1dd40c53a0 0x7f1dd40b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:48.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.979+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 >> 172.21.15.45:0/3359862856 conn(0x7f1dd40d1020 msgr2=0x7f1dd40c5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:48.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.980+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 shutdown_connections 2023-11-28T21:51:48.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:48.980+0000 7f1ddbae0640 1 -- 172.21.15.45:0/3359862856 wait complete. 2023-11-28T21:51:49.002 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T21:51:49.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:51:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.235+0000 7eff0b039640 1 Processor -- start 2023-11-28T21:51:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.235+0000 7eff0b039640 1 -- start start 2023-11-28T21:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.237+0000 7eff0b039640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 0x7eff040ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.237+0000 7eff0b039640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff040ae590 0x7eff040ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.237+0000 7eff0b039640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff040abf10 con 0x7eff0414b090 2023-11-28T21:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.237+0000 7eff0b039640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0406db90 con 0x7eff040ae590 2023-11-28T21:51:49.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.237+0000 7eff0b039640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0406d2f0 con 0x7eff040b24a0 2023-11-28T21:51:49.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.238+0000 7eff0a838640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff040ae590 0x7eff040ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.238+0000 7eff09836640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 0x7eff040ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.238+0000 7eff0a838640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff040ae590 0x7eff040ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:55948/0 (socket says 172.21.15.45:55948) 2023-11-28T21:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.238+0000 7eff0a838640 1 -- 172.21.15.45:0/1189112794 learned_addr learned my addr 172.21.15.45:0/1189112794 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.238+0000 7eff09836640 1 -- 172.21.15.45:0/1189112794 >> v2:172.21.15.190:3300/0 conn(0x7eff040ae590 msgr2=0x7eff040ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.238+0000 7eff09836640 1 --2- 172.21.15.45:0/1189112794 >> v2:172.21.15.190:3300/0 conn(0x7eff040ae590 0x7eff040ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.238+0000 7eff09836640 1 -- 172.21.15.45:0/1189112794 >> v1:172.21.15.190:6789/0 conn(0x7eff0414b090 legacy=0x7eff040b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.238+0000 7eff09836640 1 -- 172.21.15.45:0/1189112794 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0406d5d0 con 0x7eff040b24a0 2023-11-28T21:51:49.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.239+0000 7eff0a838640 1 --2- 172.21.15.45:0/1189112794 >> v2:172.21.15.190:3300/0 conn(0x7eff040ae590 0x7eff040ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T21:51:49.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.240+0000 7eff09836640 1 --2- 172.21.15.45:0/1189112794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 0x7eff040ac3a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efef40097a0 tx=0x7efef402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24d0ced97e3892a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:49.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.240+0000 7eff09035640 1 -- 172.21.15.45:0/1189112794 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef402d040 con 0x7eff040b24a0 2023-11-28T21:51:49.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.240+0000 7eff09035640 1 -- 172.21.15.45:0/1189112794 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4031810 con 0x7eff040b24a0 2023-11-28T21:51:49.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.240+0000 7eff09035640 1 -- 172.21.15.45:0/1189112794 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4031ba0 con 0x7eff040b24a0 2023-11-28T21:51:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.241+0000 7eff0b039640 1 -- 172.21.15.45:0/1189112794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 msgr2=0x7eff040ac3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.241+0000 7eff0b039640 1 --2- 172.21.15.45:0/1189112794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 0x7eff040ac3a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efef40097a0 tx=0x7efef402b8b0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.241+0000 7eff0b039640 1 -- 172.21.15.45:0/1189112794 shutdown_connections 2023-11-28T21:51:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.241+0000 7eff0b039640 1 --2- 172.21.15.45:0/1189112794 >> v2:172.21.15.190:3300/0 conn(0x7eff040ae590 0x7eff040ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.241+0000 7eff0b039640 1 --2- 172.21.15.45:0/1189112794 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 0x7eff040ac3a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.241+0000 7eff0b039640 1 -- 172.21.15.45:0/1189112794 >> 172.21.15.45:0/1189112794 conn(0x7eff0406ab30 msgr2=0x7eff040b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:49.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.241+0000 7eff0b039640 1 -- 172.21.15.45:0/1189112794 shutdown_connections 2023-11-28T21:51:49.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.241+0000 7eff0b039640 1 -- 172.21.15.45:0/1189112794 wait complete. 2023-11-28T21:51:49.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.244+0000 7eff0b039640 1 Processor -- start 2023-11-28T21:51:49.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.244+0000 7eff0b039640 1 -- start start 2023-11-28T21:51:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.245+0000 7eff0b039640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff040ae590 0x7eff04160c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.245+0000 7eff0b039640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 0x7eff04165940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.245+0000 7eff0b039640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff0414b090 0x7eff0406c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0b039640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff04165590 con 0x7eff0414b090 2023-11-28T21:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0b039640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff041611b0 con 0x7eff040ae590 2023-11-28T21:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0b039640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff04164cf0 con 0x7eff040b24a0 2023-11-28T21:51:49.248 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0a838640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff0414b090 0x7eff0406c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0a838640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff0414b090 0x7eff0406c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34546/0 (socket says 172.21.15.45:34546) 2023-11-28T21:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0a838640 1 -- 172.21.15.45:0/3274800112 learned_addr learned my addr 172.21.15.45:0/3274800112 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0a037640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff040ae590 0x7eff04160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff09836640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 0x7eff04165940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0a838640 1 -- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 msgr2=0x7eff04165940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0a838640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 0x7eff04165940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0a838640 1 -- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff040ae590 msgr2=0x7eff04160c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0a838640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff040ae590 0x7eff04160c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.250 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.246+0000 7eff0a838640 1 -- 172.21.15.45:0/3274800112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc009b30 con 0x7eff0414b090 2023-11-28T21:51:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.247+0000 7eff0a838640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff0414b090 0x7eff0406c060 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7efefc00efc0 tx=0x7efefc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.247+0000 7efefaffd640 1 -- 172.21.15.45:0/3274800112 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc018070 con 0x7eff0414b090 2023-11-28T21:51:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.247+0000 7efefaffd640 1 -- 172.21.15.45:0/3274800112 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efefc00f040 con 0x7eff0414b090 2023-11-28T21:51:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.247+0000 7efefaffd640 1 -- 172.21.15.45:0/3274800112 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00ce00 con 0x7eff0414b090 2023-11-28T21:51:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.247+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef4009400 con 0x7eff0414b090 2023-11-28T21:51:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.247+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff041652b0 con 0x7eff0414b090 2023-11-28T21:51:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.248+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0409d0f0 con 0x7eff0414b090 2023-11-28T21:51:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.249+0000 7efefaffd640 1 -- 172.21.15.45:0/3274800112 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7efefc014480 con 0x7eff0414b090 2023-11-28T21:51:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.251+0000 7efefaffd640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7efee806af60 0x7efee806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.251+0000 7efefaffd640 1 -- 172.21.15.45:0/3274800112 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7efefc089d50 con 0x7eff0414b090 2023-11-28T21:51:49.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.252+0000 7eff0a037640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7efee806af60 0x7efee806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.253+0000 7eff0a037640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7efee806af60 0x7efee806d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7eff00000f60 tx=0x7eff00009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.256+0000 7efefaffd640 1 -- 172.21.15.45:0/3274800112 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efefc054a70 con 0x7eff0414b090 2023-11-28T21:51:49.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.493+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff04072510 con 0x7eff0414b090 2023-11-28T21:51:49.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.494+0000 7efefaffd640 1 -- 172.21.15.45:0/3274800112 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4908 (secure 0 0 0) 0x7efefc058090 con 0x7eff0414b090 2023-11-28T21:51:49.495 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:49.496 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7130,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/500262407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":500262407},{"type":"v1","addr":"172.21.15.190:6835","nonce":500262407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/195287870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":195287870},{"type":"v1","addr":"172.21.15.190:6837","nonce":195287870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":9657,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/440031651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":440031651},{"type":"v1","addr":"172.21.15.45:6833","nonce":440031651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:51:45.400395+0000","modified":"2023-11-28T21:51:48.467537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9660":{"gid":9660,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/99455191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":99455191},{"type":"v1","addr":"172.21.15.45:6835","nonce":99455191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-28T21:51:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 84 2023-11-28T21:51:49.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7efee806af60 msgr2=0x7efee806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7efee806af60 0x7efee806d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7eff00000f60 tx=0x7eff00009040 comp rx=0 tx=0).stop 2023-11-28T21:51:49.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff0414b090 msgr2=0x7eff0406c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff0414b090 0x7eff0406c060 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7efefc00efc0 tx=0x7efefc00c490 comp rx=0 tx=0).stop 2023-11-28T21:51:49.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 shutdown_connections 2023-11-28T21:51:49.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7efee806af60 0x7efee806d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff0414b090 0x7eff0406c060 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7eff040b24a0 0x7eff04165940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 --2- 172.21.15.45:0/3274800112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff040ae590 0x7eff04160c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 >> 172.21.15.45:0/3274800112 conn(0x7eff0406ab30 msgr2=0x7eff040b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:49.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 shutdown_connections 2023-11-28T21:51:49.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.497+0000 7eff0b039640 1 -- 172.21.15.45:0/3274800112 wait complete. 2023-11-28T21:51:49.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:51:49.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.744+0000 7face44d7640 1 Processor -- start 2023-11-28T21:51:49.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.744+0000 7face44d7640 1 -- start start 2023-11-28T21:51:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.746+0000 7face44d7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7facdc161960 0x7facdc0b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.746+0000 7face44d7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7facdc0b0c30 0x7facdc0b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.746+0000 7face44d7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facdc0b1580 con 0x7facdc0b4c40 2023-11-28T21:51:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.746+0000 7face44d7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc071e40 con 0x7facdc161960 2023-11-28T21:51:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.746+0000 7face44d7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc0711c0 con 0x7facdc0b0c30 2023-11-28T21:51:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.746+0000 7face2273640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7facdc161960 0x7facdc0b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.747+0000 7face2273640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7facdc161960 0x7facdc0b2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34562/0 (socket says 172.21.15.45:34562) 2023-11-28T21:51:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.747+0000 7face2273640 1 -- 172.21.15.45:0/169153846 learned_addr learned my addr 172.21.15.45:0/169153846 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.747+0000 7face2a74640 1 --2- 172.21.15.45:0/169153846 >> v2:172.21.15.190:3300/0 conn(0x7facdc0b0c30 0x7facdc0b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.747+0000 7face2273640 1 -- 172.21.15.45:0/169153846 >> v2:172.21.15.190:3300/0 conn(0x7facdc0b0c30 msgr2=0x7facdc0b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.747+0000 7face2273640 1 --2- 172.21.15.45:0/169153846 >> v2:172.21.15.190:3300/0 conn(0x7facdc0b0c30 0x7facdc0b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.747+0000 7face2273640 1 -- 172.21.15.45:0/169153846 >> v1:172.21.15.190:6789/0 conn(0x7facdc0b4c40 legacy=0x7facdc070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.747+0000 7face2273640 1 -- 172.21.15.45:0/169153846 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc0714a0 con 0x7facdc161960 2023-11-28T21:51:49.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.749+0000 7face2273640 1 --2- 172.21.15.45:0/169153846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc161960 0x7facdc0b2cd0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7facdc0727c0 tx=0x7facd802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7e0b3b46318143 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:49.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.749+0000 7face1271640 1 -- 172.21.15.45:0/169153846 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80317e0 con 0x7facdc161960 2023-11-28T21:51:49.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.749+0000 7face1271640 1 -- 172.21.15.45:0/169153846 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd8031940 con 0x7facdc161960 2023-11-28T21:51:49.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.749+0000 7face1271640 1 -- 172.21.15.45:0/169153846 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8031cd0 con 0x7facdc161960 2023-11-28T21:51:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.750+0000 7face44d7640 1 -- 172.21.15.45:0/169153846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc161960 msgr2=0x7facdc0b2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.750+0000 7face44d7640 1 --2- 172.21.15.45:0/169153846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc161960 0x7facdc0b2cd0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7facdc0727c0 tx=0x7facd802b880 comp rx=0 tx=0).stop 2023-11-28T21:51:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.750+0000 7face44d7640 1 -- 172.21.15.45:0/169153846 shutdown_connections 2023-11-28T21:51:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.750+0000 7face44d7640 1 --2- 172.21.15.45:0/169153846 >> v2:172.21.15.190:3300/0 conn(0x7facdc0b0c30 0x7facdc0b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.750+0000 7face44d7640 1 --2- 172.21.15.45:0/169153846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc161960 0x7facdc0b2cd0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.750+0000 7face44d7640 1 -- 172.21.15.45:0/169153846 >> 172.21.15.45:0/169153846 conn(0x7facdc06a500 msgr2=0x7facdc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.750+0000 7face44d7640 1 -- 172.21.15.45:0/169153846 shutdown_connections 2023-11-28T21:51:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.750+0000 7face44d7640 1 -- 172.21.15.45:0/169153846 wait complete. 2023-11-28T21:51:49.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.752+0000 7face44d7640 1 Processor -- start 2023-11-28T21:51:49.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.753+0000 7face44d7640 1 -- start start 2023-11-28T21:51:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.754+0000 7face44d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facdc0b0c30 0x7facdc1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.754+0000 7face44d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc0b4c40 0x7facdc152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.754+0000 7face44d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facdc161960 0x7facdc15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.754+0000 7face44d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc159620 con 0x7facdc0b4c40 2023-11-28T21:51:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.754+0000 7face44d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc15e2a0 con 0x7facdc161960 2023-11-28T21:51:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.754+0000 7face44d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc15d970 con 0x7facdc0b0c30 2023-11-28T21:51:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.754+0000 7face1a72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc0b4c40 0x7facdc152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.754+0000 7face1a72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc0b4c40 0x7facdc152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34564/0 (socket says 172.21.15.45:34564) 2023-11-28T21:51:49.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.754+0000 7face1a72640 1 -- 172.21.15.45:0/3809049942 learned_addr learned my addr 172.21.15.45:0/3809049942 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:49.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.755+0000 7face2273640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facdc0b0c30 0x7facdc1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.755+0000 7face2a74640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facdc161960 0x7facdc15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.755+0000 7face1a72640 1 -- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facdc0b0c30 msgr2=0x7facdc1540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.755+0000 7face1a72640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facdc0b0c30 0x7facdc1540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.755+0000 7face1a72640 1 -- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facdc161960 msgr2=0x7facdc15a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.755+0000 7face1a72640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facdc161960 0x7facdc15a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.755+0000 7face1a72640 1 -- 172.21.15.45:0/3809049942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd80094d0 con 0x7facdc0b4c40 2023-11-28T21:51:49.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.755+0000 7face1a72640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc0b4c40 0x7facdc152450 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7faccc00c460 tx=0x7faccc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:49.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.755+0000 7facd2ffd640 1 -- 172.21.15.45:0/3809049942 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc012830 con 0x7facdc0b4c40 2023-11-28T21:51:49.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.756+0000 7facd2ffd640 1 -- 172.21.15.45:0/3809049942 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc0129d0 con 0x7facdc0b4c40 2023-11-28T21:51:49.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.756+0000 7facd2ffd640 1 -- 172.21.15.45:0/3809049942 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc012d80 con 0x7facdc0b4c40 2023-11-28T21:51:49.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.756+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facdc15dc50 con 0x7facdc0b4c40 2023-11-28T21:51:49.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.756+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facdc154630 con 0x7facdc0b4c40 2023-11-28T21:51:49.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.757+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facdc0cecf0 con 0x7facdc0b4c40 2023-11-28T21:51:49.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.758+0000 7facd2ffd640 1 -- 172.21.15.45:0/3809049942 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7faccc01b400 con 0x7facdc0b4c40 2023-11-28T21:51:49.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.760+0000 7facd2ffd640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7facc006af90 0x7facc006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:49.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.760+0000 7facd2ffd640 1 -- 172.21.15.45:0/3809049942 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7faccc089850 con 0x7facdc0b4c40 2023-11-28T21:51:49.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.760+0000 7face2273640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7facc006af90 0x7facc006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:49.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.762+0000 7face2273640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7facc006af90 0x7facc006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7facd8036000 tx=0x7facd803d040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:49.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.764+0000 7facd2ffd640 1 -- 172.21.15.45:0/3809049942 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faccc054680 con 0x7facdc0b4c40 2023-11-28T21:51:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.959+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facdc0733d0 con 0x7facdc0b4c40 2023-11-28T21:51:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.961+0000 7facd2ffd640 1 -- 172.21.15.45:0/3809049942 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16756 (secure 0 0 0) 0x7faccc057ca0 con 0x7facdc0b4c40 2023-11-28T21:51:49.962 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:49.962 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":185,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:51:46.837455+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:51:43.335999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-11-28T21:51:44.431458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:51:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7facc006af90 msgr2=0x7facc006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7facc006af90 0x7facc006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7facd8036000 tx=0x7facd803d040 comp rx=0 tx=0).stop 2023-11-28T21:51:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc0b4c40 msgr2=0x7facdc152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc0b4c40 0x7facdc152450 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7faccc00c460 tx=0x7faccc00c920 comp rx=0 tx=0).stop 2023-11-28T21:51:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 shutdown_connections 2023-11-28T21:51:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7facc006af90 0x7facc006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facdc161960 0x7facdc15a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facdc0b4c40 0x7facdc152450 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 --2- 172.21.15.45:0/3809049942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7facdc0b0c30 0x7facdc1540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.963+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 >> 172.21.15.45:0/3809049942 conn(0x7facdc06a500 msgr2=0x7facdc0b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.964+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 shutdown_connections 2023-11-28T21:51:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:49.964+0000 7face44d7640 1 -- 172.21.15.45:0/3809049942 wait complete. 2023-11-28T21:51:49.987 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T21:51:49.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T21:51:50.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.217+0000 7f3e4789b640 1 Processor -- start 2023-11-28T21:51:50.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.218+0000 7f3e4789b640 1 -- start start 2023-11-28T21:51:50.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.219+0000 7f3e4789b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e4015b850 0x7f3e400b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.219+0000 7f3e4789b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4015bda0 con 0x7f3e400b35b0 2023-11-28T21:51:50.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.219+0000 7f3e4789b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4015c080 con 0x7f3e400b1fa0 2023-11-28T21:51:50.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.219+0000 7f3e4789b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4015ee40 con 0x7f3e4015b850 2023-11-28T21:51:50.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.219+0000 7f3e45637640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3e400b35b0 0x7f3e4015ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60930/0 (socket says 172.21.15.45:60930) 2023-11-28T21:51:50.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.219+0000 7f3e45637640 1 -- 172.21.15.45:0/2791371569 learned_addr learned my addr 172.21.15.45:0/2791371569 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:50.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862494599 0 0) 0x7f3e4015bda0 con 0x7f3e400b35b0 2023-11-28T21:51:50.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e2c003520 con 0x7f3e400b35b0 2023-11-28T21:51:50.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e44e36640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e4015b850 0x7f3e400b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 370856606 0 0) 0x7f3e2c003520 con 0x7f3e400b35b0 2023-11-28T21:51:50.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e4015b850 msgr2=0x7f3e400b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:50.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e4015b850 0x7f3e400b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 >> v1:172.21.15.190:6789/0 conn(0x7f3e400b1fa0 legacy=0x7f3e4015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4015c360 con 0x7f3e400b35b0 2023-11-28T21:51:50.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3e3c001670 con 0x7f3e400b35b0 2023-11-28T21:51:50.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 >> v1:172.21.15.45:6789/0 conn(0x7f3e400b35b0 legacy=0x7f3e4015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e2c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e2c0047c0 0x7f3e2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e4015b850 0x7f3e2c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4015ee40 con 0x7f3e2c0039a0 2023-11-28T21:51:50.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c00bb10 con 0x7f3e2c0047c0 2023-11-28T21:51:50.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.220+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c00bca0 con 0x7f3e4015b850 2023-11-28T21:51:50.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.221+0000 7f3e45637640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e2c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.221+0000 7f3e45637640 1 -- 172.21.15.45:0/2791371569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e4015b850 msgr2=0x7f3e2c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:50.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.221+0000 7f3e45637640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e4015b850 0x7f3e2c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.221+0000 7f3e45637640 1 -- 172.21.15.45:0/2791371569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e2c0047c0 msgr2=0x7f3e2c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:50.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.221+0000 7f3e45637640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e2c0047c0 0x7f3e2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.221+0000 7f3e45637640 1 -- 172.21.15.45:0/2791371569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c003520 con 0x7f3e2c0039a0 2023-11-28T21:51:50.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.222+0000 7f3e45637640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e2c003dc0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c160 tx=0x7f3e2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5121a6bc2c0dc029 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.222+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c005ad0 con 0x7f3e2c0039a0 2023-11-28T21:51:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.223+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c005c50 con 0x7f3e2c0039a0 2023-11-28T21:51:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.223+0000 7f3e37fff640 1 -- 172.21.15.45:0/2791371569 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c02cab0 con 0x7f3e2c0039a0 2023-11-28T21:51:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.223+0000 7f3e4789b640 1 -- 172.21.15.45:0/2791371569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 msgr2=0x7f3e2c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.223+0000 7f3e4789b640 1 --2- 172.21.15.45:0/2791371569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e2c003dc0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c160 tx=0x7f3e2c0057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.223+0000 7f3e45e38640 1 -- 172.21.15.45:0/2791371569 reap_dead start 2023-11-28T21:51:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.223+0000 7f3e4789b640 1 -- 172.21.15.45:0/2791371569 shutdown_connections 2023-11-28T21:51:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.223+0000 7f3e4789b640 1 -- 172.21.15.45:0/2791371569 >> 172.21.15.45:0/2791371569 conn(0x7f3e4006a500 msgr2=0x7f3e4006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.223+0000 7f3e4789b640 1 -- 172.21.15.45:0/2791371569 shutdown_connections 2023-11-28T21:51:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.223+0000 7f3e4789b640 1 -- 172.21.15.45:0/2791371569 wait complete. 2023-11-28T21:51:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.225+0000 7f3e4789b640 1 Processor -- start 2023-11-28T21:51:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.225+0000 7f3e4789b640 1 -- start start 2023-11-28T21:51:50.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.226+0000 7f3e4789b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e401596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.226+0000 7f3e4789b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e40158800 0x7f3e400ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.226+0000 7f3e4789b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e40071050 0x7f3e40142450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.226+0000 7f3e4789b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400a8620 con 0x7f3e2c0039a0 2023-11-28T21:51:50.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.226+0000 7f3e4789b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400a8900 con 0x7f3e40071050 2023-11-28T21:51:50.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.226+0000 7f3e4789b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400a8be0 con 0x7f3e40158800 2023-11-28T21:51:50.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.226+0000 7f3e45637640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e401596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.226+0000 7f3e45637640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e401596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34586/0 (socket says 172.21.15.45:34586) 2023-11-28T21:51:50.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.226+0000 7f3e45637640 1 -- 172.21.15.45:0/3207233454 learned_addr learned my addr 172.21.15.45:0/3207233454 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:50.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e45e38640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e40071050 0x7f3e40142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e45637640 1 -- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e40158800 msgr2=0x7f3e400ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e45637640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e40158800 0x7f3e400ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e45637640 1 -- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e40071050 msgr2=0x7f3e40142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e45637640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e40071050 0x7f3e40142450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e45637640 1 -- 172.21.15.45:0/3207233454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400a8ec0 con 0x7f3e2c0039a0 2023-11-28T21:51:50.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e45637640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e401596a0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c160 tx=0x7f3e3c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:50.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e35ffb640 1 -- 172.21.15.45:0/3207233454 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c039070 con 0x7f3e2c0039a0 2023-11-28T21:51:50.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e35ffb640 1 -- 172.21.15.45:0/3207233454 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c007270 con 0x7f3e2c0039a0 2023-11-28T21:51:50.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.227+0000 7f3e35ffb640 1 -- 172.21.15.45:0/3207233454 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0075e0 con 0x7f3e2c0039a0 2023-11-28T21:51:50.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.228+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e400a91a0 con 0x7f3e2c0039a0 2023-11-28T21:51:50.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.228+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e400abd60 con 0x7f3e2c0039a0 2023-11-28T21:51:50.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.229+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e400a9330 con 0x7f3e2c0039a0 2023-11-28T21:51:50.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.230+0000 7f3e35ffb640 1 -- 172.21.15.45:0/3207233454 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3e3c032070 con 0x7f3e2c0039a0 2023-11-28T21:51:50.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.231+0000 7f3e35ffb640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3e2406af60 0x7f3e2406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.231+0000 7f3e35ffb640 1 -- 172.21.15.45:0/3207233454 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3e3c005e40 con 0x7f3e2c0039a0 2023-11-28T21:51:50.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.231+0000 7f3e44e36640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3e2406af60 0x7f3e2406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.232+0000 7f3e44e36640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3e2406af60 0x7f3e2406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3e30006ce0 tx=0x7f3e30008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:50.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.235+0000 7f3e35ffb640 1 -- 172.21.15.45:0/3207233454 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e3c075b10 con 0x7f3e2c0039a0 2023-11-28T21:51:50.344 INFO:tasks.ceph.mds.a.smithi045.stderr:2023-11-28T21:51:50.343+0000 7f3792864640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T21:51:50.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.429+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 --> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e400b4270 con 0x7f3e2406af60 2023-11-28T21:51:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.457+0000 7f3e35ffb640 1 -- 172.21.15.45:0/3207233454 <== mgr.7741 v2:172.21.15.190:6832/2040970137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286758 (secure 0 0 0) 0x7f3e400b4270 con 0x7f3e2406af60 2023-11-28T21:51:50.463 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T21:51:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3e2406af60 msgr2=0x7f3e2406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3e2406af60 0x7f3e2406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3e30006ce0 tx=0x7f3e30008040 comp rx=0 tx=0).stop 2023-11-28T21:51:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 msgr2=0x7f3e401596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e401596a0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c160 tx=0x7f3e3c001a00 comp rx=0 tx=0).stop 2023-11-28T21:51:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 shutdown_connections 2023-11-28T21:51:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f3e2406af60 0x7f3e2406d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e40071050 0x7f3e40142450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e40158800 0x7f3e400ab820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 --2- 172.21.15.45:0/3207233454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e2c0039a0 0x7f3e401596a0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 >> 172.21.15.45:0/3207233454 conn(0x7f3e4006a500 msgr2=0x7f3e400cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 shutdown_connections 2023-11-28T21:51:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.463+0000 7f3e4789b640 1 -- 172.21.15.45:0/3207233454 wait complete. 2023-11-28T21:51:50.486 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-11-28T21:51:50.421129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1573,"num_read_kb":3215,"num_write":495,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4816,"ondisk_log_size":4816,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":357504,"kb_used_data":19212,"kb_used_omap":12,"kb_used_meta":338163,"kb_avail":754617216,"statfs":{"total":773094113280,"available":772728029184,"internally_reserved":0,"allocated":19673088,"data_stored":10608348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":346279472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002198"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862016+0000","last_change":"2023-11-28T21:51:45.878079+0000","last_active":"2023-11-28T21:51:46.862016+0000","last_peered":"2023-11-28T21:51:46.862016+0000","last_clean":"2023-11-28T21:51:46.862016+0000","last_became_active":"2023-11-28T21:51:45.877139+0000","last_became_peered":"2023-11-28T21:51:45.877139+0000","last_unstale":"2023-11-28T21:51:46.862016+0000","last_undegraded":"2023-11-28T21:51:46.862016+0000","last_fullsized":"2023-11-28T21:51:46.862016+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:03.595603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.858117+0000","last_change":"2023-11-28T21:51:45.870384+0000","last_active":"2023-11-28T21:51:46.858117+0000","last_peered":"2023-11-28T21:51:46.858117+0000","last_clean":"2023-11-28T21:51:46.858117+0000","last_became_active":"2023-11-28T21:51:45.869986+0000","last_became_peered":"2023-11-28T21:51:45.869986+0000","last_unstale":"2023-11-28T21:51:46.858117+0000","last_undegraded":"2023-11-28T21:51:46.858117+0000","last_fullsized":"2023-11-28T21:51:46.858117+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:01:01.721880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857617+0000","last_change":"2023-11-28T21:51:45.877935+0000","last_active":"2023-11-28T21:51:46.857617+0000","last_peered":"2023-11-28T21:51:46.857617+0000","last_clean":"2023-11-28T21:51:46.857617+0000","last_became_active":"2023-11-28T21:51:45.875509+0000","last_became_peered":"2023-11-28T21:51:45.875509+0000","last_unstale":"2023-11-28T21:51:46.857617+0000","last_undegraded":"2023-11-28T21:51:46.857617+0000","last_fullsized":"2023-11-28T21:51:46.857617+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:39.313237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862974+0000","last_change":"2023-11-28T21:51:45.872782+0000","last_active":"2023-11-28T21:51:46.862974+0000","last_peered":"2023-11-28T21:51:46.862974+0000","last_clean":"2023-11-28T21:51:46.862974+0000","last_became_active":"2023-11-28T21:51:45.872417+0000","last_became_peered":"2023-11-28T21:51:45.872417+0000","last_unstale":"2023-11-28T21:51:46.862974+0000","last_undegraded":"2023-11-28T21:51:46.862974+0000","last_fullsized":"2023-11-28T21:51:46.862974+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:15.783536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846342+0000","last_change":"2023-11-28T21:51:45.879034+0000","last_active":"2023-11-28T21:51:46.846342+0000","last_peered":"2023-11-28T21:51:46.846342+0000","last_clean":"2023-11-28T21:51:46.846342+0000","last_became_active":"2023-11-28T21:51:45.877738+0000","last_became_peered":"2023-11-28T21:51:45.877738+0000","last_unstale":"2023-11-28T21:51:46.846342+0000","last_undegraded":"2023-11-28T21:51:46.846342+0000","last_fullsized":"2023-11-28T21:51:46.846342+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:10.716408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846256+0000","last_change":"2023-11-28T21:51:45.876805+0000","last_active":"2023-11-28T21:51:46.846256+0000","last_peered":"2023-11-28T21:51:46.846256+0000","last_clean":"2023-11-28T21:51:46.846256+0000","last_became_active":"2023-11-28T21:51:45.876460+0000","last_became_peered":"2023-11-28T21:51:45.876460+0000","last_unstale":"2023-11-28T21:51:46.846256+0000","last_undegraded":"2023-11-28T21:51:46.846256+0000","last_fullsized":"2023-11-28T21:51:46.846256+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:24:12.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859424+0000","last_change":"2023-11-28T21:51:45.868813+0000","last_active":"2023-11-28T21:51:46.859424+0000","last_peered":"2023-11-28T21:51:46.859424+0000","last_clean":"2023-11-28T21:51:46.859424+0000","last_became_active":"2023-11-28T21:51:45.866088+0000","last_became_peered":"2023-11-28T21:51:45.866088+0000","last_unstale":"2023-11-28T21:51:46.859424+0000","last_undegraded":"2023-11-28T21:51:46.859424+0000","last_fullsized":"2023-11-28T21:51:46.859424+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:36:09.187494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.848924+0000","last_change":"2023-11-28T21:51:45.878796+0000","last_active":"2023-11-28T21:51:46.848924+0000","last_peered":"2023-11-28T21:51:46.848924+0000","last_clean":"2023-11-28T21:51:46.848924+0000","last_became_active":"2023-11-28T21:51:45.878494+0000","last_became_peered":"2023-11-28T21:51:45.878494+0000","last_unstale":"2023-11-28T21:51:46.848924+0000","last_undegraded":"2023-11-28T21:51:46.848924+0000","last_fullsized":"2023-11-28T21:51:46.848924+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:15:10.394505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846629+0000","last_change":"2023-11-28T21:51:45.885567+0000","last_active":"2023-11-28T21:51:46.846629+0000","last_peered":"2023-11-28T21:51:46.846629+0000","last_clean":"2023-11-28T21:51:46.846629+0000","last_became_active":"2023-11-28T21:51:45.885114+0000","last_became_peered":"2023-11-28T21:51:45.885114+0000","last_unstale":"2023-11-28T21:51:46.846629+0000","last_undegraded":"2023-11-28T21:51:46.846629+0000","last_fullsized":"2023-11-28T21:51:46.846629+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:03.743376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.847273+0000","last_change":"2023-11-28T21:51:45.877926+0000","last_active":"2023-11-28T21:51:46.847273+0000","last_peered":"2023-11-28T21:51:46.847273+0000","last_clean":"2023-11-28T21:51:46.847273+0000","last_became_active":"2023-11-28T21:51:45.877630+0000","last_became_peered":"2023-11-28T21:51:45.877630+0000","last_unstale":"2023-11-28T21:51:46.847273+0000","last_undegraded":"2023-11-28T21:51:46.847273+0000","last_fullsized":"2023-11-28T21:51:46.847273+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:17:41.920511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844924+0000","last_change":"2023-11-28T21:51:45.880525+0000","last_active":"2023-11-28T21:51:46.844924+0000","last_peered":"2023-11-28T21:51:46.844924+0000","last_clean":"2023-11-28T21:51:46.844924+0000","last_became_active":"2023-11-28T21:51:45.878656+0000","last_became_peered":"2023-11-28T21:51:45.878656+0000","last_unstale":"2023-11-28T21:51:46.844924+0000","last_undegraded":"2023-11-28T21:51:46.844924+0000","last_fullsized":"2023-11-28T21:51:46.844924+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:27:53.152730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863298+0000","last_change":"2023-11-28T21:51:45.882768+0000","last_active":"2023-11-28T21:51:46.863298+0000","last_peered":"2023-11-28T21:51:46.863298+0000","last_clean":"2023-11-28T21:51:46.863298+0000","last_became_active":"2023-11-28T21:51:45.882489+0000","last_became_peered":"2023-11-28T21:51:45.882489+0000","last_unstale":"2023-11-28T21:51:46.863298+0000","last_undegraded":"2023-11-28T21:51:46.863298+0000","last_fullsized":"2023-11-28T21:51:46.863298+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:53:10.392014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846222+0000","last_change":"2023-11-28T21:51:45.882283+0000","last_active":"2023-11-28T21:51:46.846222+0000","last_peered":"2023-11-28T21:51:46.846222+0000","last_clean":"2023-11-28T21:51:46.846222+0000","last_became_active":"2023-11-28T21:51:45.879568+0000","last_became_peered":"2023-11-28T21:51:45.879568+0000","last_unstale":"2023-11-28T21:51:46.846222+0000","last_undegraded":"2023-11-28T21:51:46.846222+0000","last_fullsized":"2023-11-28T21:51:46.846222+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:43:46.949024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860926+0000","last_change":"2023-11-28T21:51:45.881935+0000","last_active":"2023-11-28T21:51:46.860926+0000","last_peered":"2023-11-28T21:51:46.860926+0000","last_clean":"2023-11-28T21:51:46.860926+0000","last_became_active":"2023-11-28T21:51:45.881106+0000","last_became_peered":"2023-11-28T21:51:45.881106+0000","last_unstale":"2023-11-28T21:51:46.860926+0000","last_undegraded":"2023-11-28T21:51:46.860926+0000","last_fullsized":"2023-11-28T21:51:46.860926+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:02:30.064451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.856893+0000","last_change":"2023-11-28T21:51:45.876791+0000","last_active":"2023-11-28T21:51:46.856893+0000","last_peered":"2023-11-28T21:51:46.856893+0000","last_clean":"2023-11-28T21:51:46.856893+0000","last_became_active":"2023-11-28T21:51:45.876142+0000","last_became_peered":"2023-11-28T21:51:45.876142+0000","last_unstale":"2023-11-28T21:51:46.856893+0000","last_undegraded":"2023-11-28T21:51:46.856893+0000","last_fullsized":"2023-11-28T21:51:46.856893+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:58:40.619635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845753+0000","last_change":"2023-11-28T21:51:45.882304+0000","last_active":"2023-11-28T21:51:46.845753+0000","last_peered":"2023-11-28T21:51:46.845753+0000","last_clean":"2023-11-28T21:51:46.845753+0000","last_became_active":"2023-11-28T21:51:45.880898+0000","last_became_peered":"2023-11-28T21:51:45.880898+0000","last_unstale":"2023-11-28T21:51:46.845753+0000","last_undegraded":"2023-11-28T21:51:46.845753+0000","last_fullsized":"2023-11-28T21:51:46.845753+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:08:42.682615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862341+0000","last_change":"2023-11-28T21:51:45.873499+0000","last_active":"2023-11-28T21:51:46.862341+0000","last_peered":"2023-11-28T21:51:46.862341+0000","last_clean":"2023-11-28T21:51:46.862341+0000","last_became_active":"2023-11-28T21:51:45.873112+0000","last_became_peered":"2023-11-28T21:51:45.873112+0000","last_unstale":"2023-11-28T21:51:46.862341+0000","last_undegraded":"2023-11-28T21:51:46.862341+0000","last_fullsized":"2023-11-28T21:51:46.862341+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:01.025010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845003+0000","last_change":"2023-11-28T21:51:45.876388+0000","last_active":"2023-11-28T21:51:46.845003+0000","last_peered":"2023-11-28T21:51:46.845003+0000","last_clean":"2023-11-28T21:51:46.845003+0000","last_became_active":"2023-11-28T21:51:45.874079+0000","last_became_peered":"2023-11-28T21:51:45.874079+0000","last_unstale":"2023-11-28T21:51:46.845003+0000","last_undegraded":"2023-11-28T21:51:46.845003+0000","last_fullsized":"2023-11-28T21:51:46.845003+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:25:23.311571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862320+0000","last_change":"2023-11-28T21:51:45.878326+0000","last_active":"2023-11-28T21:51:46.862320+0000","last_peered":"2023-11-28T21:51:46.862320+0000","last_clean":"2023-11-28T21:51:46.862320+0000","last_became_active":"2023-11-28T21:51:45.877889+0000","last_became_peered":"2023-11-28T21:51:45.877889+0000","last_unstale":"2023-11-28T21:51:46.862320+0000","last_undegraded":"2023-11-28T21:51:46.862320+0000","last_fullsized":"2023-11-28T21:51:46.862320+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:58.310908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859339+0000","last_change":"2023-11-28T21:51:45.871842+0000","last_active":"2023-11-28T21:51:46.859339+0000","last_peered":"2023-11-28T21:51:46.859339+0000","last_clean":"2023-11-28T21:51:46.859339+0000","last_became_active":"2023-11-28T21:51:45.871422+0000","last_became_peered":"2023-11-28T21:51:45.871422+0000","last_unstale":"2023-11-28T21:51:46.859339+0000","last_undegraded":"2023-11-28T21:51:46.859339+0000","last_fullsized":"2023-11-28T21:51:46.859339+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:49:36.819922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846200+0000","last_change":"2023-11-28T21:51:45.883170+0000","last_active":"2023-11-28T21:51:46.846200+0000","last_peered":"2023-11-28T21:51:46.846200+0000","last_clean":"2023-11-28T21:51:46.846200+0000","last_became_active":"2023-11-28T21:51:45.881790+0000","last_became_peered":"2023-11-28T21:51:45.881790+0000","last_unstale":"2023-11-28T21:51:46.846200+0000","last_undegraded":"2023-11-28T21:51:46.846200+0000","last_fullsized":"2023-11-28T21:51:46.846200+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:58.310908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860259+0000","last_change":"2023-11-28T21:51:45.865242+0000","last_active":"2023-11-28T21:51:46.860259+0000","last_peered":"2023-11-28T21:51:46.860259+0000","last_clean":"2023-11-28T21:51:46.860259+0000","last_became_active":"2023-11-28T21:51:45.864937+0000","last_became_peered":"2023-11-28T21:51:45.864937+0000","last_unstale":"2023-11-28T21:51:46.860259+0000","last_undegraded":"2023-11-28T21:51:46.860259+0000","last_fullsized":"2023-11-28T21:51:46.860259+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:15:57.392428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849030+0000","last_change":"2023-11-28T21:51:45.871786+0000","last_active":"2023-11-28T21:51:46.849030+0000","last_peered":"2023-11-28T21:51:46.849030+0000","last_clean":"2023-11-28T21:51:46.849030+0000","last_became_active":"2023-11-28T21:51:45.871490+0000","last_became_peered":"2023-11-28T21:51:45.871490+0000","last_unstale":"2023-11-28T21:51:46.849030+0000","last_undegraded":"2023-11-28T21:51:46.849030+0000","last_fullsized":"2023-11-28T21:51:46.849030+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:33:53.274607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844413+0000","last_change":"2023-11-28T21:51:45.876606+0000","last_active":"2023-11-28T21:51:46.844413+0000","last_peered":"2023-11-28T21:51:46.844413+0000","last_clean":"2023-11-28T21:51:46.844413+0000","last_became_active":"2023-11-28T21:51:45.876297+0000","last_became_peered":"2023-11-28T21:51:45.876297+0000","last_unstale":"2023-11-28T21:51:46.844413+0000","last_undegraded":"2023-11-28T21:51:46.844413+0000","last_fullsized":"2023-11-28T21:51:46.844413+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:12.261849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862027+0000","last_change":"2023-11-28T21:51:45.868438+0000","last_active":"2023-11-28T21:51:46.862027+0000","last_peered":"2023-11-28T21:51:46.862027+0000","last_clean":"2023-11-28T21:51:46.862027+0000","last_became_active":"2023-11-28T21:51:45.868111+0000","last_became_peered":"2023-11-28T21:51:45.868111+0000","last_unstale":"2023-11-28T21:51:46.862027+0000","last_undegraded":"2023-11-28T21:51:46.862027+0000","last_fullsized":"2023-11-28T21:51:46.862027+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:52:57.310627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863299+0000","last_change":"2023-11-28T21:51:45.872229+0000","last_active":"2023-11-28T21:51:46.863299+0000","last_peered":"2023-11-28T21:51:46.863299+0000","last_clean":"2023-11-28T21:51:46.863299+0000","last_became_active":"2023-11-28T21:51:45.869659+0000","last_became_peered":"2023-11-28T21:51:45.869659+0000","last_unstale":"2023-11-28T21:51:46.863299+0000","last_undegraded":"2023-11-28T21:51:46.863299+0000","last_fullsized":"2023-11-28T21:51:46.863299+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:20.412397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846994+0000","last_change":"2023-11-28T21:51:45.881027+0000","last_active":"2023-11-28T21:51:46.846994+0000","last_peered":"2023-11-28T21:51:46.846994+0000","last_clean":"2023-11-28T21:51:46.846994+0000","last_became_active":"2023-11-28T21:51:45.878851+0000","last_became_peered":"2023-11-28T21:51:45.878851+0000","last_unstale":"2023-11-28T21:51:46.846994+0000","last_undegraded":"2023-11-28T21:51:46.846994+0000","last_fullsized":"2023-11-28T21:51:46.846994+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:43:11.146254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844656+0000","last_change":"2023-11-28T21:51:45.867114+0000","last_active":"2023-11-28T21:51:46.844656+0000","last_peered":"2023-11-28T21:51:46.844656+0000","last_clean":"2023-11-28T21:51:46.844656+0000","last_became_active":"2023-11-28T21:51:45.866735+0000","last_became_peered":"2023-11-28T21:51:45.866735+0000","last_unstale":"2023-11-28T21:51:46.844656+0000","last_undegraded":"2023-11-28T21:51:46.844656+0000","last_fullsized":"2023-11-28T21:51:46.844656+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:07:20.731285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849871+0000","last_change":"2023-11-28T21:51:45.885398+0000","last_active":"2023-11-28T21:51:46.849871+0000","last_peered":"2023-11-28T21:51:46.849871+0000","last_clean":"2023-11-28T21:51:46.849871+0000","last_became_active":"2023-11-28T21:51:45.885081+0000","last_became_peered":"2023-11-28T21:51:45.885081+0000","last_unstale":"2023-11-28T21:51:46.849871+0000","last_undegraded":"2023-11-28T21:51:46.849871+0000","last_fullsized":"2023-11-28T21:51:46.849871+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:38:43.699130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844662+0000","last_change":"2023-11-28T21:51:45.869331+0000","last_active":"2023-11-28T21:51:46.844662+0000","last_peered":"2023-11-28T21:51:46.844662+0000","last_clean":"2023-11-28T21:51:46.844662+0000","last_became_active":"2023-11-28T21:51:45.868975+0000","last_became_peered":"2023-11-28T21:51:45.868975+0000","last_unstale":"2023-11-28T21:51:46.844662+0000","last_undegraded":"2023-11-28T21:51:46.844662+0000","last_fullsized":"2023-11-28T21:51:46.844662+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:19.332000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845926+0000","last_change":"2023-11-28T21:51:45.880541+0000","last_active":"2023-11-28T21:51:46.845926+0000","last_peered":"2023-11-28T21:51:46.845926+0000","last_clean":"2023-11-28T21:51:46.845926+0000","last_became_active":"2023-11-28T21:51:45.880252+0000","last_became_peered":"2023-11-28T21:51:45.880252+0000","last_unstale":"2023-11-28T21:51:46.845926+0000","last_undegraded":"2023-11-28T21:51:46.845926+0000","last_fullsized":"2023-11-28T21:51:46.845926+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:08:53.846283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.848997+0000","last_change":"2023-11-28T21:51:45.887436+0000","last_active":"2023-11-28T21:51:46.848997+0000","last_peered":"2023-11-28T21:51:46.848997+0000","last_clean":"2023-11-28T21:51:46.848997+0000","last_became_active":"2023-11-28T21:51:45.887077+0000","last_became_peered":"2023-11-28T21:51:45.887077+0000","last_unstale":"2023-11-28T21:51:46.848997+0000","last_undegraded":"2023-11-28T21:51:46.848997+0000","last_fullsized":"2023-11-28T21:51:46.848997+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:19:19.212771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857692+0000","last_change":"2023-11-28T21:51:45.866602+0000","last_active":"2023-11-28T21:51:46.857692+0000","last_peered":"2023-11-28T21:51:46.857692+0000","last_clean":"2023-11-28T21:51:46.857692+0000","last_became_active":"2023-11-28T21:51:45.866022+0000","last_became_peered":"2023-11-28T21:51:45.866022+0000","last_unstale":"2023-11-28T21:51:46.857692+0000","last_undegraded":"2023-11-28T21:51:46.857692+0000","last_fullsized":"2023-11-28T21:51:46.857692+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:44.823306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"184'4","reported_seq":16,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846964+0000","last_change":"2023-11-28T21:51:44.938778+0000","last_active":"2023-11-28T21:51:46.846964+0000","last_peered":"2023-11-28T21:51:46.846964+0000","last_clean":"2023-11-28T21:51:46.846964+0000","last_became_active":"2023-11-28T21:51:44.938490+0000","last_became_peered":"2023-11-28T21:51:44.938490+0000","last_unstale":"2023-11-28T21:51:46.846964+0000","last_undegraded":"2023-11-28T21:51:46.846964+0000","last_fullsized":"2023-11-28T21:51:46.846964+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:24:28.548759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857953+0000","last_change":"2023-11-28T21:51:45.874870+0000","last_active":"2023-11-28T21:51:46.857953+0000","last_peered":"2023-11-28T21:51:46.857953+0000","last_clean":"2023-11-28T21:51:46.857953+0000","last_became_active":"2023-11-28T21:51:45.874694+0000","last_became_peered":"2023-11-28T21:51:45.874694+0000","last_unstale":"2023-11-28T21:51:46.857953+0000","last_undegraded":"2023-11-28T21:51:46.857953+0000","last_fullsized":"2023-11-28T21:51:46.857953+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:57.433051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860824+0000","last_change":"2023-11-28T21:51:44.956071+0000","last_active":"2023-11-28T21:51:46.860824+0000","last_peered":"2023-11-28T21:51:46.860824+0000","last_clean":"2023-11-28T21:51:46.860824+0000","last_became_active":"2023-11-28T21:51:44.955513+0000","last_became_peered":"2023-11-28T21:51:44.955513+0000","last_unstale":"2023-11-28T21:51:46.860824+0000","last_undegraded":"2023-11-28T21:51:46.860824+0000","last_fullsized":"2023-11-28T21:51:46.860824+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:32.173902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.850381+0000","last_change":"2023-11-28T21:51:45.875314+0000","last_active":"2023-11-28T21:51:46.850381+0000","last_peered":"2023-11-28T21:51:46.850381+0000","last_clean":"2023-11-28T21:51:46.850381+0000","last_became_active":"2023-11-28T21:51:45.874679+0000","last_became_peered":"2023-11-28T21:51:45.874679+0000","last_unstale":"2023-11-28T21:51:46.850381+0000","last_undegraded":"2023-11-28T21:51:46.850381+0000","last_fullsized":"2023-11-28T21:51:46.850381+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:02:05.357903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862940+0000","last_change":"2023-11-28T21:51:44.956398+0000","last_active":"2023-11-28T21:51:46.862940+0000","last_peered":"2023-11-28T21:51:46.862940+0000","last_clean":"2023-11-28T21:51:46.862940+0000","last_became_active":"2023-11-28T21:51:44.954997+0000","last_became_peered":"2023-11-28T21:51:44.954997+0000","last_unstale":"2023-11-28T21:51:46.862940+0000","last_undegraded":"2023-11-28T21:51:46.862940+0000","last_fullsized":"2023-11-28T21:51:46.862940+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:49:35.804765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860962+0000","last_change":"2023-11-28T21:51:45.875181+0000","last_active":"2023-11-28T21:51:46.860962+0000","last_peered":"2023-11-28T21:51:46.860962+0000","last_clean":"2023-11-28T21:51:46.860962+0000","last_became_active":"2023-11-28T21:51:45.874246+0000","last_became_peered":"2023-11-28T21:51:45.874246+0000","last_unstale":"2023-11-28T21:51:46.860962+0000","last_undegraded":"2023-11-28T21:51:46.860962+0000","last_fullsized":"2023-11-28T21:51:46.860962+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:08:42.682615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857527+0000","last_change":"2023-11-28T21:51:44.940748+0000","last_active":"2023-11-28T21:51:46.857527+0000","last_peered":"2023-11-28T21:51:46.857527+0000","last_clean":"2023-11-28T21:51:46.857527+0000","last_became_active":"2023-11-28T21:51:44.940456+0000","last_became_peered":"2023-11-28T21:51:44.940456+0000","last_unstale":"2023-11-28T21:51:46.857527+0000","last_undegraded":"2023-11-28T21:51:46.857527+0000","last_fullsized":"2023-11-28T21:51:46.857527+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:53:09.376856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.858645+0000","last_change":"2023-11-28T21:51:45.865601+0000","last_active":"2023-11-28T21:51:46.858645+0000","last_peered":"2023-11-28T21:51:46.858645+0000","last_clean":"2023-11-28T21:51:46.858645+0000","last_became_active":"2023-11-28T21:51:45.864793+0000","last_became_peered":"2023-11-28T21:51:45.864793+0000","last_unstale":"2023-11-28T21:51:46.858645+0000","last_undegraded":"2023-11-28T21:51:46.858645+0000","last_fullsized":"2023-11-28T21:51:46.858645+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:32:54.787053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.850130+0000","last_change":"2023-11-28T21:51:44.956999+0000","last_active":"2023-11-28T21:51:46.850130+0000","last_peered":"2023-11-28T21:51:46.850130+0000","last_clean":"2023-11-28T21:51:46.850130+0000","last_became_active":"2023-11-28T21:51:44.956355+0000","last_became_peered":"2023-11-28T21:51:44.956355+0000","last_unstale":"2023-11-28T21:51:46.850130+0000","last_undegraded":"2023-11-28T21:51:46.850130+0000","last_fullsized":"2023-11-28T21:51:46.850130+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:29:51.530533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860975+0000","last_change":"2023-11-28T21:51:45.875060+0000","last_active":"2023-11-28T21:51:46.860975+0000","last_peered":"2023-11-28T21:51:46.860975+0000","last_clean":"2023-11-28T21:51:46.860975+0000","last_became_active":"2023-11-28T21:51:45.873876+0000","last_became_peered":"2023-11-28T21:51:45.873876+0000","last_unstale":"2023-11-28T21:51:46.860975+0000","last_undegraded":"2023-11-28T21:51:46.860975+0000","last_fullsized":"2023-11-28T21:51:46.860975+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:08:00.930788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846700+0000","last_change":"2023-11-28T21:51:44.945722+0000","last_active":"2023-11-28T21:51:46.846700+0000","last_peered":"2023-11-28T21:51:46.846700+0000","last_clean":"2023-11-28T21:51:46.846700+0000","last_became_active":"2023-11-28T21:51:44.945393+0000","last_became_peered":"2023-11-28T21:51:44.945393+0000","last_unstale":"2023-11-28T21:51:46.846700+0000","last_undegraded":"2023-11-28T21:51:46.846700+0000","last_fullsized":"2023-11-28T21:51:46.846700+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:45:47.899500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857241+0000","last_change":"2023-11-28T21:51:45.867180+0000","last_active":"2023-11-28T21:51:46.857241+0000","last_peered":"2023-11-28T21:51:46.857241+0000","last_clean":"2023-11-28T21:51:46.857241+0000","last_became_active":"2023-11-28T21:51:45.866844+0000","last_became_peered":"2023-11-28T21:51:45.866844+0000","last_unstale":"2023-11-28T21:51:46.857241+0000","last_undegraded":"2023-11-28T21:51:46.857241+0000","last_fullsized":"2023-11-28T21:51:46.857241+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:53:06.355914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859846+0000","last_change":"2023-11-28T21:51:44.935312+0000","last_active":"2023-11-28T21:51:46.859846+0000","last_peered":"2023-11-28T21:51:46.859846+0000","last_clean":"2023-11-28T21:51:46.859846+0000","last_became_active":"2023-11-28T21:51:44.935014+0000","last_became_peered":"2023-11-28T21:51:44.935014+0000","last_unstale":"2023-11-28T21:51:46.859846+0000","last_undegraded":"2023-11-28T21:51:46.859846+0000","last_fullsized":"2023-11-28T21:51:46.859846+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:34.679453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860755+0000","last_change":"2023-11-28T21:51:45.879934+0000","last_active":"2023-11-28T21:51:46.860755+0000","last_peered":"2023-11-28T21:51:46.860755+0000","last_clean":"2023-11-28T21:51:46.860755+0000","last_became_active":"2023-11-28T21:51:45.879584+0000","last_became_peered":"2023-11-28T21:51:45.879584+0000","last_unstale":"2023-11-28T21:51:46.860755+0000","last_undegraded":"2023-11-28T21:51:46.860755+0000","last_fullsized":"2023-11-28T21:51:46.860755+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:49.171516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845825+0000","last_change":"2023-11-28T21:51:44.949413+0000","last_active":"2023-11-28T21:51:46.845825+0000","last_peered":"2023-11-28T21:51:46.845825+0000","last_clean":"2023-11-28T21:51:46.845825+0000","last_became_active":"2023-11-28T21:51:44.948658+0000","last_became_peered":"2023-11-28T21:51:44.948658+0000","last_unstale":"2023-11-28T21:51:46.845825+0000","last_undegraded":"2023-11-28T21:51:46.845825+0000","last_fullsized":"2023-11-28T21:51:46.845825+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:37.048471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.850120+0000","last_change":"2023-11-28T21:51:45.873238+0000","last_active":"2023-11-28T21:51:46.850120+0000","last_peered":"2023-11-28T21:51:46.850120+0000","last_clean":"2023-11-28T21:51:46.850120+0000","last_became_active":"2023-11-28T21:51:45.872945+0000","last_became_peered":"2023-11-28T21:51:45.872945+0000","last_unstale":"2023-11-28T21:51:46.850120+0000","last_undegraded":"2023-11-28T21:51:46.850120+0000","last_fullsized":"2023-11-28T21:51:46.850120+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:22:54.646623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861339+0000","last_change":"2023-11-28T21:51:44.955272+0000","last_active":"2023-11-28T21:51:46.861339+0000","last_peered":"2023-11-28T21:51:46.861339+0000","last_clean":"2023-11-28T21:51:46.861339+0000","last_became_active":"2023-11-28T21:51:44.954729+0000","last_became_peered":"2023-11-28T21:51:44.954729+0000","last_unstale":"2023-11-28T21:51:46.861339+0000","last_undegraded":"2023-11-28T21:51:46.861339+0000","last_fullsized":"2023-11-28T21:51:46.861339+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:37.048471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845127+0000","last_change":"2023-11-28T21:51:45.870661+0000","last_active":"2023-11-28T21:51:46.845127+0000","last_peered":"2023-11-28T21:51:46.845127+0000","last_clean":"2023-11-28T21:51:46.845127+0000","last_became_active":"2023-11-28T21:51:45.870347+0000","last_became_peered":"2023-11-28T21:51:45.870347+0000","last_unstale":"2023-11-28T21:51:46.845127+0000","last_undegraded":"2023-11-28T21:51:46.845127+0000","last_fullsized":"2023-11-28T21:51:46.845127+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:30:24.222242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863898+0000","last_change":"2023-11-28T21:51:44.942670+0000","last_active":"2023-11-28T21:51:46.863898+0000","last_peered":"2023-11-28T21:51:46.863898+0000","last_clean":"2023-11-28T21:51:46.863898+0000","last_became_active":"2023-11-28T21:51:44.940785+0000","last_became_peered":"2023-11-28T21:51:44.940785+0000","last_unstale":"2023-11-28T21:51:46.863898+0000","last_undegraded":"2023-11-28T21:51:46.863898+0000","last_fullsized":"2023-11-28T21:51:46.863898+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:14:34.926484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845032+0000","last_change":"2023-11-28T21:51:45.876091+0000","last_active":"2023-11-28T21:51:46.845032+0000","last_peered":"2023-11-28T21:51:46.845032+0000","last_clean":"2023-11-28T21:51:46.845032+0000","last_became_active":"2023-11-28T21:51:45.873605+0000","last_became_peered":"2023-11-28T21:51:45.873605+0000","last_unstale":"2023-11-28T21:51:46.845032+0000","last_undegraded":"2023-11-28T21:51:46.845032+0000","last_fullsized":"2023-11-28T21:51:46.845032+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:10.605728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"184'3","reported_seq":15,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845206+0000","last_change":"2023-11-28T21:51:44.936696+0000","last_active":"2023-11-28T21:51:46.845206+0000","last_peered":"2023-11-28T21:51:46.845206+0000","last_clean":"2023-11-28T21:51:46.845206+0000","last_became_active":"2023-11-28T21:51:44.936426+0000","last_became_peered":"2023-11-28T21:51:44.936426+0000","last_unstale":"2023-11-28T21:51:46.845206+0000","last_undegraded":"2023-11-28T21:51:46.845206+0000","last_fullsized":"2023-11-28T21:51:46.845206+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:32:12.144221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849282+0000","last_change":"2023-11-28T21:51:45.874326+0000","last_active":"2023-11-28T21:51:46.849282+0000","last_peered":"2023-11-28T21:51:46.849282+0000","last_clean":"2023-11-28T21:51:46.849282+0000","last_became_active":"2023-11-28T21:51:45.873633+0000","last_became_peered":"2023-11-28T21:51:45.873633+0000","last_unstale":"2023-11-28T21:51:46.849282+0000","last_undegraded":"2023-11-28T21:51:46.849282+0000","last_fullsized":"2023-11-28T21:51:46.849282+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:58:35.716414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846427+0000","last_change":"2023-11-28T21:51:44.937471+0000","last_active":"2023-11-28T21:51:46.846427+0000","last_peered":"2023-11-28T21:51:46.846427+0000","last_clean":"2023-11-28T21:51:46.846427+0000","last_became_active":"2023-11-28T21:51:44.937096+0000","last_became_peered":"2023-11-28T21:51:44.937096+0000","last_unstale":"2023-11-28T21:51:46.846427+0000","last_undegraded":"2023-11-28T21:51:46.846427+0000","last_fullsized":"2023-11-28T21:51:46.846427+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:55:44.808450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857190+0000","last_change":"2023-11-28T21:51:45.876970+0000","last_active":"2023-11-28T21:51:46.857190+0000","last_peered":"2023-11-28T21:51:46.857190+0000","last_clean":"2023-11-28T21:51:46.857190+0000","last_became_active":"2023-11-28T21:51:45.876518+0000","last_became_peered":"2023-11-28T21:51:45.876518+0000","last_unstale":"2023-11-28T21:51:46.857190+0000","last_undegraded":"2023-11-28T21:51:46.857190+0000","last_fullsized":"2023-11-28T21:51:46.857190+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:37:21.095095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861342+0000","last_change":"2023-11-28T21:51:44.952532+0000","last_active":"2023-11-28T21:51:46.861342+0000","last_peered":"2023-11-28T21:51:46.861342+0000","last_clean":"2023-11-28T21:51:46.861342+0000","last_became_active":"2023-11-28T21:51:44.952240+0000","last_became_peered":"2023-11-28T21:51:44.952240+0000","last_unstale":"2023-11-28T21:51:46.861342+0000","last_undegraded":"2023-11-28T21:51:46.861342+0000","last_fullsized":"2023-11-28T21:51:46.861342+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:09:33.914911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846693+0000","last_change":"2023-11-28T21:51:45.873066+0000","last_active":"2023-11-28T21:51:46.846693+0000","last_peered":"2023-11-28T21:51:46.846693+0000","last_clean":"2023-11-28T21:51:46.846693+0000","last_became_active":"2023-11-28T21:51:45.872695+0000","last_became_peered":"2023-11-28T21:51:45.872695+0000","last_unstale":"2023-11-28T21:51:46.846693+0000","last_undegraded":"2023-11-28T21:51:46.846693+0000","last_fullsized":"2023-11-28T21:51:46.846693+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:48:48.044068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861580+0000","last_change":"2023-11-28T21:51:44.945752+0000","last_active":"2023-11-28T21:51:46.861580+0000","last_peered":"2023-11-28T21:51:46.861580+0000","last_clean":"2023-11-28T21:51:46.861580+0000","last_became_active":"2023-11-28T21:51:44.945423+0000","last_became_peered":"2023-11-28T21:51:44.945423+0000","last_unstale":"2023-11-28T21:51:46.861580+0000","last_undegraded":"2023-11-28T21:51:46.861580+0000","last_fullsized":"2023-11-28T21:51:46.861580+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:54.257461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857318+0000","last_change":"2023-11-28T21:51:45.883636+0000","last_active":"2023-11-28T21:51:46.857318+0000","last_peered":"2023-11-28T21:51:46.857318+0000","last_clean":"2023-11-28T21:51:46.857318+0000","last_became_active":"2023-11-28T21:51:45.883228+0000","last_became_peered":"2023-11-28T21:51:45.883228+0000","last_unstale":"2023-11-28T21:51:46.857318+0000","last_undegraded":"2023-11-28T21:51:46.857318+0000","last_fullsized":"2023-11-28T21:51:46.857318+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:07:43.308822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"184'8","reported_seq":20,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.858867+0000","last_change":"2023-11-28T21:51:44.951036+0000","last_active":"2023-11-28T21:51:46.858867+0000","last_peered":"2023-11-28T21:51:46.858867+0000","last_clean":"2023-11-28T21:51:46.858867+0000","last_became_active":"2023-11-28T21:51:44.950584+0000","last_became_peered":"2023-11-28T21:51:44.950584+0000","last_unstale":"2023-11-28T21:51:46.858867+0000","last_undegraded":"2023-11-28T21:51:46.858867+0000","last_fullsized":"2023-11-28T21:51:46.858867+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:57:36.241338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861016+0000","last_change":"2023-11-28T21:51:45.883146+0000","last_active":"2023-11-28T21:51:46.861016+0000","last_peered":"2023-11-28T21:51:46.861016+0000","last_clean":"2023-11-28T21:51:46.861016+0000","last_became_active":"2023-11-28T21:51:45.880601+0000","last_became_peered":"2023-11-28T21:51:45.880601+0000","last_unstale":"2023-11-28T21:51:46.861016+0000","last_undegraded":"2023-11-28T21:51:46.861016+0000","last_fullsized":"2023-11-28T21:51:46.861016+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:08:53.846283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844729+0000","last_change":"2023-11-28T21:51:44.951766+0000","last_active":"2023-11-28T21:51:46.844729+0000","last_peered":"2023-11-28T21:51:46.844729+0000","last_clean":"2023-11-28T21:51:46.844729+0000","last_became_active":"2023-11-28T21:51:44.951154+0000","last_became_peered":"2023-11-28T21:51:44.951154+0000","last_unstale":"2023-11-28T21:51:46.844729+0000","last_undegraded":"2023-11-28T21:51:46.844729+0000","last_fullsized":"2023-11-28T21:51:46.844729+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:09.211879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862697+0000","last_change":"2023-11-28T21:51:45.878486+0000","last_active":"2023-11-28T21:51:46.862697+0000","last_peered":"2023-11-28T21:51:46.862697+0000","last_clean":"2023-11-28T21:51:46.862697+0000","last_became_active":"2023-11-28T21:51:45.878177+0000","last_became_peered":"2023-11-28T21:51:45.878177+0000","last_unstale":"2023-11-28T21:51:46.862697+0000","last_undegraded":"2023-11-28T21:51:46.862697+0000","last_fullsized":"2023-11-28T21:51:46.862697+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:43:46.949024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861068+0000","last_change":"2023-11-28T21:51:44.956246+0000","last_active":"2023-11-28T21:51:46.861068+0000","last_peered":"2023-11-28T21:51:46.861068+0000","last_clean":"2023-11-28T21:51:46.861068+0000","last_became_active":"2023-11-28T21:51:44.955819+0000","last_became_peered":"2023-11-28T21:51:44.955819+0000","last_unstale":"2023-11-28T21:51:46.861068+0000","last_undegraded":"2023-11-28T21:51:46.861068+0000","last_fullsized":"2023-11-28T21:51:46.861068+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:45:47.899500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863681+0000","last_change":"2023-11-28T21:51:45.880930+0000","last_active":"2023-11-28T21:51:46.863681+0000","last_peered":"2023-11-28T21:51:46.863681+0000","last_clean":"2023-11-28T21:51:46.863681+0000","last_became_active":"2023-11-28T21:51:45.880664+0000","last_became_peered":"2023-11-28T21:51:45.880664+0000","last_unstale":"2023-11-28T21:51:46.863681+0000","last_undegraded":"2023-11-28T21:51:46.863681+0000","last_fullsized":"2023-11-28T21:51:46.863681+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:29:01.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857709+0000","last_change":"2023-11-28T21:51:44.954935+0000","last_active":"2023-11-28T21:51:46.857709+0000","last_peered":"2023-11-28T21:51:46.857709+0000","last_clean":"2023-11-28T21:51:46.857709+0000","last_became_active":"2023-11-28T21:51:44.954537+0000","last_became_peered":"2023-11-28T21:51:44.954537+0000","last_unstale":"2023-11-28T21:51:46.857709+0000","last_undegraded":"2023-11-28T21:51:46.857709+0000","last_fullsized":"2023-11-28T21:51:46.857709+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:21:49.107429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862380+0000","last_change":"2023-11-28T21:51:45.885482+0000","last_active":"2023-11-28T21:51:46.862380+0000","last_peered":"2023-11-28T21:51:46.862380+0000","last_clean":"2023-11-28T21:51:46.862380+0000","last_became_active":"2023-11-28T21:51:45.882481+0000","last_became_peered":"2023-11-28T21:51:45.882481+0000","last_unstale":"2023-11-28T21:51:46.862380+0000","last_undegraded":"2023-11-28T21:51:46.862380+0000","last_fullsized":"2023-11-28T21:51:46.862380+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:58:01.699459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863177+0000","last_change":"2023-11-28T21:51:44.952419+0000","last_active":"2023-11-28T21:51:46.863177+0000","last_peered":"2023-11-28T21:51:46.863177+0000","last_clean":"2023-11-28T21:51:46.863177+0000","last_became_active":"2023-11-28T21:51:44.952131+0000","last_became_peered":"2023-11-28T21:51:44.952131+0000","last_unstale":"2023-11-28T21:51:46.863177+0000","last_undegraded":"2023-11-28T21:51:46.863177+0000","last_fullsized":"2023-11-28T21:51:46.863177+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:59:13.462136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859583+0000","last_change":"2023-11-28T21:51:45.881750+0000","last_active":"2023-11-28T21:51:46.859583+0000","last_peered":"2023-11-28T21:51:46.859583+0000","last_clean":"2023-11-28T21:51:46.859583+0000","last_became_active":"2023-11-28T21:51:45.881456+0000","last_became_peered":"2023-11-28T21:51:45.881456+0000","last_unstale":"2023-11-28T21:51:46.859583+0000","last_undegraded":"2023-11-28T21:51:46.859583+0000","last_fullsized":"2023-11-28T21:51:46.859583+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:24:29.563916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859091+0000","last_change":"2023-11-28T21:51:44.948846+0000","last_active":"2023-11-28T21:51:46.859091+0000","last_peered":"2023-11-28T21:51:46.859091+0000","last_clean":"2023-11-28T21:51:46.859091+0000","last_became_active":"2023-11-28T21:51:44.948425+0000","last_became_peered":"2023-11-28T21:51:44.948425+0000","last_unstale":"2023-11-28T21:51:46.859091+0000","last_undegraded":"2023-11-28T21:51:46.859091+0000","last_fullsized":"2023-11-28T21:51:46.859091+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:32.173902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857062+0000","last_change":"2023-11-28T21:51:45.868425+0000","last_active":"2023-11-28T21:51:46.857062+0000","last_peered":"2023-11-28T21:51:46.857062+0000","last_clean":"2023-11-28T21:51:46.857062+0000","last_became_active":"2023-11-28T21:51:45.868064+0000","last_became_peered":"2023-11-28T21:51:45.868064+0000","last_unstale":"2023-11-28T21:51:46.857062+0000","last_undegraded":"2023-11-28T21:51:46.857062+0000","last_fullsized":"2023-11-28T21:51:46.857062+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:26:19.218196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849502+0000","last_change":"2023-11-28T21:51:44.947351+0000","last_active":"2023-11-28T21:51:46.849502+0000","last_peered":"2023-11-28T21:51:46.849502+0000","last_clean":"2023-11-28T21:51:46.849502+0000","last_became_active":"2023-11-28T21:51:44.947075+0000","last_became_peered":"2023-11-28T21:51:44.947075+0000","last_unstale":"2023-11-28T21:51:46.849502+0000","last_undegraded":"2023-11-28T21:51:46.849502+0000","last_fullsized":"2023-11-28T21:51:46.849502+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:36:35.800471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859960+0000","last_change":"2023-11-28T21:51:45.880030+0000","last_active":"2023-11-28T21:51:46.859960+0000","last_peered":"2023-11-28T21:51:46.859960+0000","last_clean":"2023-11-28T21:51:46.859960+0000","last_became_active":"2023-11-28T21:51:45.879668+0000","last_became_peered":"2023-11-28T21:51:45.879668+0000","last_unstale":"2023-11-28T21:51:46.859960+0000","last_undegraded":"2023-11-28T21:51:46.859960+0000","last_fullsized":"2023-11-28T21:51:46.859960+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:55:45.823607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.858781+0000","last_change":"2023-11-28T21:51:44.945354+0000","last_active":"2023-11-28T21:51:46.858781+0000","last_peered":"2023-11-28T21:51:46.858781+0000","last_clean":"2023-11-28T21:51:46.858781+0000","last_became_active":"2023-11-28T21:51:44.944812+0000","last_became_peered":"2023-11-28T21:51:44.944812+0000","last_unstale":"2023-11-28T21:51:46.858781+0000","last_undegraded":"2023-11-28T21:51:46.858781+0000","last_fullsized":"2023-11-28T21:51:46.858781+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:24:44.899820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857187+0000","last_change":"2023-11-28T21:51:45.877419+0000","last_active":"2023-11-28T21:51:46.857187+0000","last_peered":"2023-11-28T21:51:46.857187+0000","last_clean":"2023-11-28T21:51:46.857187+0000","last_became_active":"2023-11-28T21:51:45.873727+0000","last_became_peered":"2023-11-28T21:51:45.873727+0000","last_unstale":"2023-11-28T21:51:46.857187+0000","last_undegraded":"2023-11-28T21:51:46.857187+0000","last_fullsized":"2023-11-28T21:51:46.857187+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:22:24.545177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849540+0000","last_change":"2023-11-28T21:51:44.940523+0000","last_active":"2023-11-28T21:51:46.849540+0000","last_peered":"2023-11-28T21:51:46.849540+0000","last_clean":"2023-11-28T21:51:46.849540+0000","last_became_active":"2023-11-28T21:51:44.940236+0000","last_became_peered":"2023-11-28T21:51:44.940236+0000","last_unstale":"2023-11-28T21:51:46.849540+0000","last_undegraded":"2023-11-28T21:51:46.849540+0000","last_fullsized":"2023-11-28T21:51:46.849540+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:24:54.137809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846729+0000","last_change":"2023-11-28T21:51:45.879426+0000","last_active":"2023-11-28T21:51:46.846729+0000","last_peered":"2023-11-28T21:51:46.846729+0000","last_clean":"2023-11-28T21:51:46.846729+0000","last_became_active":"2023-11-28T21:51:45.878959+0000","last_became_peered":"2023-11-28T21:51:45.878959+0000","last_unstale":"2023-11-28T21:51:46.846729+0000","last_undegraded":"2023-11-28T21:51:46.846729+0000","last_fullsized":"2023-11-28T21:51:46.846729+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:58:26.996916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862435+0000","last_change":"2023-11-28T21:51:44.953351+0000","last_active":"2023-11-28T21:51:46.862435+0000","last_peered":"2023-11-28T21:51:46.862435+0000","last_clean":"2023-11-28T21:51:46.862435+0000","last_became_active":"2023-11-28T21:51:44.953094+0000","last_became_peered":"2023-11-28T21:51:44.953094+0000","last_unstale":"2023-11-28T21:51:46.862435+0000","last_undegraded":"2023-11-28T21:51:46.862435+0000","last_fullsized":"2023-11-28T21:51:46.862435+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:14:15.482556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862465+0000","last_change":"2023-11-28T21:51:45.885000+0000","last_active":"2023-11-28T21:51:46.862465+0000","last_peered":"2023-11-28T21:51:46.862465+0000","last_clean":"2023-11-28T21:51:46.862465+0000","last_became_active":"2023-11-28T21:51:45.884699+0000","last_became_peered":"2023-11-28T21:51:45.884699+0000","last_unstale":"2023-11-28T21:51:46.862465+0000","last_undegraded":"2023-11-28T21:51:46.862465+0000","last_fullsized":"2023-11-28T21:51:46.862465+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:12.676367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.864045+0000","last_change":"2023-11-28T21:51:44.952870+0000","last_active":"2023-11-28T21:51:46.864045+0000","last_peered":"2023-11-28T21:51:46.864045+0000","last_clean":"2023-11-28T21:51:46.864045+0000","last_became_active":"2023-11-28T21:51:44.950437+0000","last_became_peered":"2023-11-28T21:51:44.950437+0000","last_unstale":"2023-11-28T21:51:46.864045+0000","last_undegraded":"2023-11-28T21:51:46.864045+0000","last_fullsized":"2023-11-28T21:51:46.864045+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:49.752643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863885+0000","last_change":"2023-11-28T21:51:45.877417+0000","last_active":"2023-11-28T21:51:46.863885+0000","last_peered":"2023-11-28T21:51:46.863885+0000","last_clean":"2023-11-28T21:51:46.863885+0000","last_became_active":"2023-11-28T21:51:45.877131+0000","last_became_peered":"2023-11-28T21:51:45.877131+0000","last_unstale":"2023-11-28T21:51:46.863885+0000","last_undegraded":"2023-11-28T21:51:46.863885+0000","last_fullsized":"2023-11-28T21:51:46.863885+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:02.659275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857789+0000","last_change":"2023-11-28T21:51:44.940553+0000","last_active":"2023-11-28T21:51:46.857789+0000","last_peered":"2023-11-28T21:51:46.857789+0000","last_clean":"2023-11-28T21:51:46.857789+0000","last_became_active":"2023-11-28T21:51:44.940201+0000","last_became_peered":"2023-11-28T21:51:44.940201+0000","last_unstale":"2023-11-28T21:51:46.857789+0000","last_undegraded":"2023-11-28T21:51:46.857789+0000","last_fullsized":"2023-11-28T21:51:46.857789+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:20:08.464784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846825+0000","last_change":"2023-11-28T21:51:45.881560+0000","last_active":"2023-11-28T21:51:46.846825+0000","last_peered":"2023-11-28T21:51:46.846825+0000","last_clean":"2023-11-28T21:51:46.846825+0000","last_became_active":"2023-11-28T21:51:45.881195+0000","last_became_peered":"2023-11-28T21:51:45.881195+0000","last_unstale":"2023-11-28T21:51:46.846825+0000","last_undegraded":"2023-11-28T21:51:46.846825+0000","last_fullsized":"2023-11-28T21:51:46.846825+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:10:21.497544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846953+0000","last_change":"2023-11-28T21:51:44.949737+0000","last_active":"2023-11-28T21:51:46.846953+0000","last_peered":"2023-11-28T21:51:46.846953+0000","last_clean":"2023-11-28T21:51:46.846953+0000","last_became_active":"2023-11-28T21:51:44.948114+0000","last_became_peered":"2023-11-28T21:51:44.948114+0000","last_unstale":"2023-11-28T21:51:46.846953+0000","last_undegraded":"2023-11-28T21:51:46.846953+0000","last_fullsized":"2023-11-28T21:51:46.846953+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:49:35.804765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849359+0000","last_change":"2023-11-28T21:51:45.870645+0000","last_active":"2023-11-28T21:51:46.849359+0000","last_peered":"2023-11-28T21:51:46.849359+0000","last_clean":"2023-11-28T21:51:46.849359+0000","last_became_active":"2023-11-28T21:51:45.870358+0000","last_became_peered":"2023-11-28T21:51:45.870358+0000","last_unstale":"2023-11-28T21:51:46.849359+0000","last_undegraded":"2023-11-28T21:51:46.849359+0000","last_fullsized":"2023-11-28T21:51:46.849359+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:00:18.324325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860981+0000","last_change":"2023-11-28T21:51:44.954725+0000","last_active":"2023-11-28T21:51:46.860981+0000","last_peered":"2023-11-28T21:51:46.860981+0000","last_clean":"2023-11-28T21:51:46.860981+0000","last_became_active":"2023-11-28T21:51:44.954428+0000","last_became_peered":"2023-11-28T21:51:44.954428+0000","last_unstale":"2023-11-28T21:51:46.860981+0000","last_undegraded":"2023-11-28T21:51:46.860981+0000","last_fullsized":"2023-11-28T21:51:46.860981+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:16:39.896912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861331+0000","last_change":"2023-11-28T21:51:45.886350+0000","last_active":"2023-11-28T21:51:46.861331+0000","last_peered":"2023-11-28T21:51:46.861331+0000","last_clean":"2023-11-28T21:51:46.861331+0000","last_became_active":"2023-11-28T21:51:45.886058+0000","last_became_peered":"2023-11-28T21:51:45.886058+0000","last_unstale":"2023-11-28T21:51:46.861331+0000","last_undegraded":"2023-11-28T21:51:46.861331+0000","last_fullsized":"2023-11-28T21:51:46.861331+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:57:01.404286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844792+0000","last_change":"2023-11-28T21:51:44.948872+0000","last_active":"2023-11-28T21:51:46.844792+0000","last_peered":"2023-11-28T21:51:46.844792+0000","last_clean":"2023-11-28T21:51:46.844792+0000","last_became_active":"2023-11-28T21:51:44.948590+0000","last_became_peered":"2023-11-28T21:51:44.948590+0000","last_unstale":"2023-11-28T21:51:46.844792+0000","last_undegraded":"2023-11-28T21:51:46.844792+0000","last_fullsized":"2023-11-28T21:51:46.844792+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:14:20.212267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857046+0000","last_change":"2023-11-28T21:51:45.872047+0000","last_active":"2023-11-28T21:51:46.857046+0000","last_peered":"2023-11-28T21:51:46.857046+0000","last_clean":"2023-11-28T21:51:46.857046+0000","last_became_active":"2023-11-28T21:51:45.871687+0000","last_became_peered":"2023-11-28T21:51:45.871687+0000","last_unstale":"2023-11-28T21:51:46.857046+0000","last_undegraded":"2023-11-28T21:51:46.857046+0000","last_fullsized":"2023-11-28T21:51:46.857046+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:40:33.540462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860093+0000","last_change":"2023-11-28T21:51:44.939343+0000","last_active":"2023-11-28T21:51:46.860093+0000","last_peered":"2023-11-28T21:51:46.860093+0000","last_clean":"2023-11-28T21:51:46.860093+0000","last_became_active":"2023-11-28T21:51:44.939069+0000","last_became_peered":"2023-11-28T21:51:44.939069+0000","last_unstale":"2023-11-28T21:51:46.860093+0000","last_undegraded":"2023-11-28T21:51:46.860093+0000","last_fullsized":"2023-11-28T21:51:46.860093+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:18:17.848127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849874+0000","last_change":"2023-11-28T21:51:45.884679+0000","last_active":"2023-11-28T21:51:46.849874+0000","last_peered":"2023-11-28T21:51:46.849874+0000","last_clean":"2023-11-28T21:51:46.849874+0000","last_became_active":"2023-11-28T21:51:45.883978+0000","last_became_peered":"2023-11-28T21:51:45.883978+0000","last_unstale":"2023-11-28T21:51:46.849874+0000","last_undegraded":"2023-11-28T21:51:46.849874+0000","last_fullsized":"2023-11-28T21:51:46.849874+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:19.779511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859826+0000","last_change":"2023-11-28T21:51:44.954533+0000","last_active":"2023-11-28T21:51:46.859826+0000","last_peered":"2023-11-28T21:51:46.859826+0000","last_clean":"2023-11-28T21:51:46.859826+0000","last_became_active":"2023-11-28T21:51:44.953590+0000","last_became_peered":"2023-11-28T21:51:44.953590+0000","last_unstale":"2023-11-28T21:51:46.859826+0000","last_undegraded":"2023-11-28T21:51:46.859826+0000","last_fullsized":"2023-11-28T21:51:46.859826+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:24:44.899820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.850421+0000","last_change":"2023-11-28T21:51:45.875713+0000","last_active":"2023-11-28T21:51:46.850421+0000","last_peered":"2023-11-28T21:51:46.850421+0000","last_clean":"2023-11-28T21:51:46.850421+0000","last_became_active":"2023-11-28T21:51:45.875428+0000","last_became_peered":"2023-11-28T21:51:45.875428+0000","last_unstale":"2023-11-28T21:51:46.850421+0000","last_undegraded":"2023-11-28T21:51:46.850421+0000","last_fullsized":"2023-11-28T21:51:46.850421+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:53.969541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"184'6","reported_seq":18,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859333+0000","last_change":"2023-11-28T21:51:44.947806+0000","last_active":"2023-11-28T21:51:46.859333+0000","last_peered":"2023-11-28T21:51:46.859333+0000","last_clean":"2023-11-28T21:51:46.859333+0000","last_became_active":"2023-11-28T21:51:44.947398+0000","last_became_peered":"2023-11-28T21:51:44.947398+0000","last_unstale":"2023-11-28T21:51:46.859333+0000","last_undegraded":"2023-11-28T21:51:46.859333+0000","last_fullsized":"2023-11-28T21:51:46.859333+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:06.132079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"172'4778","reported_seq":5465,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-11-28T21:51:45.865629+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T21:51:45.865629+0000","last_peered":"2023-11-28T21:51:45.865629+0000","last_clean":"2023-11-28T21:51:45.865629+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T21:51:45.865629+0000","last_undegraded":"2023-11-28T21:51:45.865629+0000","last_fullsized":"2023-11-28T21:51:45.865629+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1573,"num_read_kb":3215,"num_write":451,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1573,"num_read_kb":3215,"num_write":451,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4778,"ondisk_log_size":4778,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575776,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56220,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315620,"statfs":{"total":96636764160,"available":96579194880,"internally_reserved":0,"allocated":2895872,"data_stored":1765746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.256,"15min":1.212},"min":{"1min":0.92500000000000004,"5min":0.76800000000000002,"15min":0.68600000000000005},"max":{"1min":1.6799999999999999,"5min":1.865,"15min":1.865},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.238,"15min":1.242},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.79600000000000004},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.954},"last":1.25}]},{"osd":1,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2070000000000001,"15min":1.244},"min":{"1min":0.94699999999999995,"5min":0.79400000000000004,"15min":0.66800000000000004},"max":{"1min":1.6259999999999999,"5min":1.6859999999999999,"15min":2.1880000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2909999999999999,"15min":1.288},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.61899999999999999},"max":{"1min":1.6910000000000001,"5min":1.718,"15min":2.4279999999999999},"last":1.1910000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3420000000000001,"15min":1.325},"min":{"1min":0.97699999999999998,"5min":0.90100000000000002,"15min":0.73499999999999999},"max":{"1min":1.732,"5min":1.754,"15min":1.893},"last":1.29},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.377,"15min":1.369},"min":{"1min":1.034,"5min":0.98799999999999999,"15min":0.95499999999999996},"max":{"1min":1.8280000000000001,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.1479999999999999}]},{"osd":3,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.321,"15min":1.3200000000000001},"min":{"1min":1.008,"5min":0.90400000000000003,"15min":0.76000000000000001},"max":{"1min":1.845,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":1.573},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.496,"15min":1.4730000000000001},"min":{"1min":1.2649999999999999,"5min":0.94699999999999995,"15min":0.872},"max":{"1min":1.7150000000000001,"5min":2.3769999999999998,"15min":2.3769999999999998},"last":1.5229999999999999}]},{"osd":4,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.349,"15min":1.3029999999999999},"min":{"1min":0.98599999999999999,"5min":0.88400000000000001,"15min":0.76500000000000001},"max":{"1min":1.796,"5min":1.8460000000000001,"15min":2.1240000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.321,"15min":1.347},"min":{"1min":1.002,"5min":0.88100000000000001,"15min":0.81299999999999994},"max":{"1min":1.748,"5min":1.895,"15min":1.9470000000000001},"last":1.635}]},{"osd":5,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4750000000000001,"15min":1.456},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.79200000000000004},"max":{"1min":1.722,"5min":1.976,"15min":2.008},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.5580000000000001,"15min":1.5089999999999999},"min":{"1min":1.3220000000000001,"5min":1.0389999999999999,"15min":0.89700000000000002},"max":{"1min":1.9670000000000001,"5min":2.246,"15min":2.246},"last":1.5069999999999999}]},{"osd":6,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.6160000000000001,"15min":1.597},"min":{"1min":1.3120000000000001,"5min":1.2250000000000001,"15min":0.94399999999999995},"max":{"1min":1.998,"5min":2.1640000000000001,"15min":2.3029999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.506,"5min":1.4870000000000001,"15min":1.508},"min":{"1min":1.1579999999999999,"5min":1.0009999999999999,"15min":0.94399999999999995},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":2.2170000000000001},"last":1.7010000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575776,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37600,"kb_used_data":2260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334240,"statfs":{"total":96636764160,"available":96598261760,"internally_reserved":0,"allocated":2314240,"data_stored":1182074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.129,"15min":1.1339999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.597,"5min":1.6180000000000001,"15min":1.8740000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.149,"5min":1.165,"15min":1.159},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5760000000000001,"5min":1.702,"15min":1.702},"last":0.84499999999999997}]},{"osd":1,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.224,"15min":1.2609999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7,"5min":1.7989999999999999,"15min":1.9550000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.427,"15min":1.4419999999999999},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.796,"5min":1.8169999999999999,"15min":2.2109999999999999},"last":1.1000000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.294,"15min":1.2789999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":2.012},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4059999999999999,"15min":1.4199999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":2.4420000000000002},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.3100000000000001,"15min":1.331},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.581,"5min":1.9419999999999999,"15min":2.0779999999999998},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3340000000000001,"15min":1.3240000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.7889999999999999,"5min":1.8600000000000001,"15min":2.1499999999999999},"last":0.97999999999999998}]},{"osd":4,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2490000000000001,"15min":1.256},"min":{"1min":0.90700000000000003,"5min":0.80200000000000005,"15min":0.77900000000000003},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":2.726},"last":1.506},{"interface":"front","average":{"1min":1.337,"5min":1.3420000000000001,"15min":1.349},"min":{"1min":0.95599999999999996,"5min":0.82499999999999996,"15min":0.80100000000000005},"max":{"1min":1.764,"5min":2.004,"15min":3.069},"last":1.1479999999999999}]},{"osd":5,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.4019999999999999,"15min":1.401},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.83699999999999997},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.319},"last":1.173},{"interface":"front","average":{"1min":1.53,"5min":1.5549999999999999,"15min":1.5680000000000001},"min":{"1min":1.1339999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.994,"5min":1.994,"15min":2.5680000000000001},"last":1.5800000000000001}]},{"osd":7,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.5109999999999999,"15min":1.5329999999999999},"min":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":0.99099999999999999},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.96},"last":1.343},{"interface":"front","average":{"1min":1.45,"5min":1.504,"15min":1.516},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":1.99,"5min":1.99,"15min":2.3439999999999999},"last":1.663}]}]},{"osd":5,"up_from":13,"seq":55834575776,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59812,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312028,"statfs":{"total":96636764160,"available":96575516672,"internally_reserved":0,"allocated":2310144,"data_stored":1182004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.198,"15min":1.2},"min":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.65400000000000003},"max":{"1min":1.411,"5min":1.6919999999999999,"15min":1.8879999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.177,"15min":1.1990000000000001},"min":{"1min":0.879,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.542,"5min":1.653,"15min":1.8620000000000001},"last":0.92600000000000005}]},{"osd":1,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2050000000000001,"15min":1.218},"min":{"1min":0.89400000000000002,"5min":0.81699999999999995,"15min":0.752},"max":{"1min":1.5580000000000001,"5min":1.724,"15min":1.843},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.242,"15min":1.2769999999999999},"min":{"1min":0.79500000000000004,"5min":0.76400000000000001,"15min":0.74399999999999999},"max":{"1min":1.5569999999999999,"5min":1.8520000000000001,"15min":2.109},"last":1.5309999999999999}]},{"osd":2,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.3169999999999999,"15min":1.325},"min":{"1min":0.91000000000000003,"5min":0.72999999999999998,"15min":0.70199999999999996},"max":{"1min":1.7030000000000001,"5min":1.7450000000000001,"15min":1.9379999999999999},"last":1.657},{"interface":"front","average":{"1min":1.393,"5min":1.3400000000000001,"15min":1.337},"min":{"1min":1.113,"5min":0.876,"15min":0.83699999999999997},"max":{"1min":1.5629999999999999,"5min":1.766,"15min":1.9990000000000001},"last":1.373}]},{"osd":3,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4159999999999999,"15min":1.389},"min":{"1min":1.0940000000000001,"5min":0.92400000000000004,"15min":0.754},"max":{"1min":1.627,"5min":1.954,"15min":2.1030000000000002},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.3320000000000001,"15min":1.329},"min":{"1min":1.101,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.643,"5min":1.71,"15min":2.0630000000000002},"last":1.206}]},{"osd":4,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.3779999999999999,"15min":1.349},"min":{"1min":1.111,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.776,"5min":1.8169999999999999,"15min":1.9399999999999999},"last":1.319},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4890000000000001,"15min":1.458},"min":{"1min":0.94499999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.833,"5min":1.9059999999999999,"15min":2.2040000000000002},"last":1.728}]},{"osd":6,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.3720000000000001,"15min":1.381},"min":{"1min":1.032,"5min":0.752,"15min":0.752},"max":{"1min":1.7050000000000001,"5min":2.0110000000000001,"15min":2.0699999999999998},"last":1.593},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.524,"15min":1.526},"min":{"1min":1.3200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.8520000000000001,"5min":1.905,"15min":2.1560000000000001},"last":1.4419999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.431,"15min":1.4530000000000001},"min":{"1min":1.0960000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.7330000000000001,"5min":1.9670000000000001,"15min":3.0110000000000001},"last":1.78},{"interface":"front","average":{"1min":1.587,"5min":1.5269999999999999,"15min":1.5289999999999999},"min":{"1min":1.379,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.7430000000000001,"5min":1.9790000000000001,"15min":2.2010000000000001},"last":1.8080000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575777,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29272,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342568,"statfs":{"total":96636764160,"available":96606789632,"internally_reserved":0,"allocated":2310144,"data_stored":1176968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.109,"15min":1.1439999999999999},"min":{"1min":0.78000000000000003,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.5169999999999999,"5min":1.9379999999999999,"15min":1.9550000000000001},"last":1.262},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.1659999999999999,"15min":1.165},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.76300000000000001},"max":{"1min":1.679,"5min":1.679,"15min":1.9570000000000001},"last":0.83399999999999996}]},{"osd":1,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.363,"15min":1.351},"min":{"1min":1.0509999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.6719999999999999,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.389,"15min":1.393},"min":{"1min":1.0920000000000001,"5min":0.82699999999999996,"15min":0.79900000000000004},"max":{"1min":1.752,"5min":1.8320000000000001,"15min":1.9119999999999999},"last":1.0680000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2450000000000001,"15min":1.27},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.86399999999999999},"max":{"1min":1.4910000000000001,"5min":1.788,"15min":1.9570000000000001},"last":1.198},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.98499999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.6990000000000001,"5min":1.7809999999999999,"15min":1.905},"last":1.508}]},{"osd":3,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.3009999999999999,"15min":1.2969999999999999},"min":{"1min":0.94699999999999995,"5min":0.876,"15min":0.73699999999999999},"max":{"1min":1.671,"5min":1.7090000000000001,"15min":1.7769999999999999},"last":1.742},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.415,"15min":1.4139999999999999},"min":{"1min":1.157,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.923},"last":1.5960000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.306,"15min":1.2889999999999999},"min":{"1min":0.98399999999999999,"5min":0.749,"15min":0.749},"max":{"1min":1.7210000000000001,"5min":1.873,"15min":1.891},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.343,"15min":1.3220000000000001},"min":{"1min":0.96099999999999997,"5min":0.81499999999999995,"15min":0.79100000000000004},"max":{"1min":1.7849999999999999,"5min":1.8089999999999999,"15min":1.9339999999999999},"last":1.954}]},{"osd":6,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.464,"15min":1.4359999999999999},"min":{"1min":0.92400000000000004,"5min":0.86399999999999999,"15min":0.85399999999999998},"max":{"1min":1.8029999999999999,"5min":1.974,"15min":2.1099999999999999},"last":1.9990000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.587,"15min":1.579},"min":{"1min":1.2709999999999999,"5min":1.246,"15min":0.86499999999999999},"max":{"1min":1.8680000000000001,"5min":2.0049999999999999,"15min":2.23},"last":1.8120000000000001}]},{"osd":7,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.4850000000000001,"15min":1.4750000000000001},"min":{"1min":1.0820000000000001,"5min":1.0069999999999999,"15min":0.94399999999999995},"max":{"1min":1.8360000000000001,"5min":1.921,"15min":2.085},"last":2.056},{"interface":"front","average":{"1min":1.625,"5min":1.589,"15min":1.5660000000000001},"min":{"1min":1.1819999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":1.925,"5min":2.1640000000000001,"15min":2.2210000000000001},"last":1.8759999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575777,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51484,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320356,"statfs":{"total":96636764160,"available":96584044544,"internally_reserved":0,"allocated":2306048,"data_stored":1175972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.002,"15min":0.98099999999999998},"min":{"1min":0.68300000000000005,"5min":0.67400000000000004,"15min":0.60399999999999998},"max":{"1min":1.492,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.105,"15min":1.115},"min":{"1min":0.84299999999999997,"5min":0.67900000000000005,"15min":0.66500000000000004},"max":{"1min":1.369,"5min":1.895,"15min":1.9399999999999999},"last":1.2210000000000001}]},{"osd":1,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.036,"15min":1.0229999999999999},"min":{"1min":0.75700000000000001,"5min":0.753,"15min":0.72999999999999998},"max":{"1min":1.474,"5min":1.7390000000000001,"15min":1.8069999999999999},"last":1.079},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.284,"15min":1.262},"min":{"1min":0.89200000000000002,"5min":0.82899999999999996,"15min":0.68999999999999995},"max":{"1min":1.694,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.2929999999999999}]},{"osd":2,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3069999999999999,"15min":1.3129999999999999},"min":{"1min":0.88800000000000001,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.663,"5min":1.839,"15min":1.8560000000000001},"last":1.262},{"interface":"front","average":{"1min":1.48,"5min":1.4770000000000001,"15min":1.4319999999999999},"min":{"1min":1.1439999999999999,"5min":0.92400000000000004,"15min":0.751},"max":{"1min":1.778,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.129}]},{"osd":4,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.355,"15min":1.387},"min":{"1min":0.996,"5min":0.94799999999999995,"15min":0.85299999999999998},"max":{"1min":1.673,"5min":2.2000000000000002,"15min":2.246},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.456,"15min":1.46},"min":{"1min":1.111,"5min":1.0589999999999999,"15min":1.0509999999999999},"max":{"1min":1.835,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.323}]},{"osd":5,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.607,"15min":1.577},"min":{"1min":1.3440000000000001,"5min":1.1259999999999999,"15min":1.0229999999999999},"max":{"1min":2.0960000000000001,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.51,"15min":1.4890000000000001},"min":{"1min":1.2529999999999999,"5min":1.109,"15min":0.94699999999999995},"max":{"1min":1.7310000000000001,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.518}]},{"osd":6,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.569,"15min":1.5620000000000001},"min":{"1min":1.117,"5min":1.117,"15min":1.111},"max":{"1min":1.9810000000000001,"5min":2.149,"15min":2.153},"last":1.371},{"interface":"front","average":{"1min":1.7490000000000001,"5min":1.706,"15min":1.677},"min":{"1min":1.431,"5min":1.27,"15min":1.099},"max":{"1min":2.1619999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.919}]},{"osd":7,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.6319999999999999,"15min":1.617},"min":{"1min":1.232,"5min":1.214,"15min":0.995},"max":{"1min":2.0499999999999998,"5min":2.5,"15min":2.5},"last":1.996},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.72,"15min":1.716},"min":{"1min":1.2889999999999999,"5min":1.23,"15min":1.23},"max":{"1min":2.0859999999999999,"5min":2.339,"15min":2.4489999999999998},"last":1.585}]}]},{"osd":2,"up_from":13,"seq":55834575777,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47336,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324504,"statfs":{"total":96636764160,"available":96588292096,"internally_reserved":0,"allocated":2318336,"data_stored":1182116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":0.97599999999999998,"15min":0.96899999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.369},{"interface":"front","average":{"1min":1.006,"5min":1.141,"15min":1.1439999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.6559999999999999,"5min":1.764,"15min":1.8160000000000001},"last":0.79600000000000004}]},{"osd":1,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.98399999999999999,"15min":1.0049999999999999},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0379999999999998},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.222,"15min":1.2430000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.8380000000000001},"last":0.89200000000000002}]},{"osd":3,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.1990000000000001,"15min":1.222},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.7679999999999998,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.133},{"interface":"front","average":{"1min":1.292,"5min":1.3109999999999999,"15min":1.329},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.9199999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.5189999999999999}]},{"osd":4,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.456,"15min":1.4670000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":3.202,"5min":3.202,"15min":3.202},"last":1.9119999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.498,"15min":1.502},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.4530000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.508,"15min":1.522},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":0.99299999999999999},"max":{"1min":3.1269999999999998,"5min":3.1269999999999998,"15min":3.1269999999999998},"last":1.8109999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.5800000000000001,"15min":1.542},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.3149999999999999,"5min":3.3149999999999999,"15min":3.3149999999999999},"last":1.831}]},{"osd":6,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.583,"15min":1.591},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.661,"5min":1.633,"15min":1.6279999999999999},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.706}]},{"osd":7,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.621,"15min":1.633},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":4.2990000000000004,"5min":4.2990000000000004,"15min":4.2990000000000004},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.679,"15min":1.7},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.8940000000000001,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":1.649}]}]},{"osd":1,"up_from":13,"seq":55834575777,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45912,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325928,"statfs":{"total":96636764160,"available":96589750272,"internally_reserved":0,"allocated":2301952,"data_stored":1175426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.91700000000000004,"15min":0.91800000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.127,"15min":1.1240000000000001},"min":{"1min":0.76000000000000001,"5min":0.753,"15min":0.69799999999999995},"max":{"1min":2.9489999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.88200000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1819999999999999,"15min":1.1499999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77400000000000002},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.103},{"interface":"front","average":{"1min":1.254,"5min":1.258,"15min":1.2649999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.921,"5min":1.921,"15min":2.0750000000000002},"last":1.4530000000000001}]},{"osd":3,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4039999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.889,"5min":1.9790000000000001,"15min":1.9810000000000001},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.1910000000000001,"15min":1.1839999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":0.95299999999999996}]},{"osd":4,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.4279999999999999,"15min":1.415},"min":{"1min":1.1299999999999999,"5min":1.0209999999999999,"15min":0.90900000000000003},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.23},{"interface":"front","average":{"1min":1.534,"5min":1.4570000000000001,"15min":1.4450000000000001},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.84199999999999997},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.4239999999999999}]},{"osd":5,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.482,"15min":1.48},"min":{"1min":1.008,"5min":1.008,"15min":0.97399999999999998},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.694},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.569,"15min":1.5700000000000001},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.0110000000000001,"5min":2.0979999999999999,"15min":2.25},"last":1.8700000000000001}]},{"osd":6,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.516,"15min":1.5029999999999999},"min":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.0429999999999999},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.302},{"interface":"front","average":{"1min":1.712,"5min":1.605,"15min":1.591},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":3.1520000000000001,"5min":3.1520000000000001,"15min":3.1520000000000001},"last":1.5820000000000001}]},{"osd":7,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.6439999999999999,"15min":1.6559999999999999},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.2589999999999999,"5min":2.2629999999999999,"15min":2.3439999999999999},"last":1.9990000000000001},{"interface":"front","average":{"1min":1.734,"5min":1.6299999999999999,"15min":1.607},"min":{"1min":1.258,"5min":1.151,"15min":1.1080000000000001},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.2589999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575777,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29868,"kb_used_data":2848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341972,"statfs":{"total":96636764160,"available":96606179328,"internally_reserved":0,"allocated":2916352,"data_stored":1768042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.93899999999999995,"15min":0.93999999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.72,"5min":1.72,"15min":1.724},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1719999999999999,"15min":1.1639999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.7879999999999998,"5min":2.7879999999999998,"15min":2.7879999999999998},"last":0.75900000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.1339999999999999,"15min":1.1359999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.5449999999999999,"5min":1.6339999999999999,"15min":1.8959999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.246,"15min":1.2569999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.7929999999999999,"5min":1.8819999999999999,"15min":2.0409999999999999},"last":0.67200000000000004}]},{"osd":3,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.1240000000000001,"15min":1.1319999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.4750000000000001,"5min":1.7110000000000001,"15min":2.0649999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.401,"15min":1.3640000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.1899999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.88100000000000001}]},{"osd":4,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4570000000000001,"15min":1.46},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.302,"5min":1.446,"15min":1.4139999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.6459999999999999,"5min":1.869,"15min":2.0600000000000001},"last":1.089}]},{"osd":5,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.45,"15min":1.4550000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.77,"5min":2.153,"15min":2.153},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5229999999999999,"15min":1.5289999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.159}]},{"osd":6,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.613,"15min":1.6060000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5940000000000001,"15min":1.599},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":1.0149999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.6819999999999999,"15min":1.667},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.589,"15min":1.5800000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.371,"5min":2.371,"15min":2.371},"last":1.181}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T21:51:50.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T21:51:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.759+0000 7f5fcc988640 1 Processor -- start 2023-11-28T21:51:50.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.760+0000 7f5fcc988640 1 -- start start 2023-11-28T21:51:50.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.761+0000 7f5fcc988640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5fc8155780 0x7f5fc80abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.761+0000 7f5fcc988640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5fc80ae540 0x7f5fc80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.761+0000 7f5fcc988640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 0x7f5fc815ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.761+0000 7f5fcc988640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc80ceac0 con 0x7f5fc80ae540 2023-11-28T21:51:50.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.761+0000 7f5fcc988640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc815c3c0 con 0x7f5fc8155780 2023-11-28T21:51:50.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.761+0000 7f5fcc988640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8155b60 con 0x7f5fc80ce630 2023-11-28T21:51:50.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.762+0000 7f5fbe59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5fc80ae540 0x7f5fc80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.762+0000 7f5fbe59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5fc80ae540 0x7f5fc80ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34588/0 (socket says 172.21.15.45:34588) 2023-11-28T21:51:50.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.762+0000 7f5fbe59c640 1 -- 172.21.15.45:0/2590121061 learned_addr learned my addr 172.21.15.45:0/2590121061 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:50.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.762+0000 7f5fbe59c640 1 -- 172.21.15.45:0/2590121061 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 msgr2=0x7f5fc815ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:50.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.762+0000 7f5fbe59c640 1 --2- 172.21.15.45:0/2590121061 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 0x7f5fc815ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.762+0000 7f5fbe59c640 1 -- 172.21.15.45:0/2590121061 >> v2:172.21.15.190:3300/0 conn(0x7f5fc8155780 msgr2=0x7f5fc80abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:50.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.762+0000 7f5fbe59c640 1 --2- 172.21.15.45:0/2590121061 >> v2:172.21.15.190:3300/0 conn(0x7f5fc8155780 0x7f5fc80abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.762+0000 7f5fbe59c640 1 -- 172.21.15.45:0/2590121061 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc815ba90 con 0x7f5fc80ae540 2023-11-28T21:51:50.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.763+0000 7f5fbe59c640 1 --2- 172.21.15.45:0/2590121061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc80ae540 0x7f5fc80ce0f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f5fc806e6d0 tx=0x7f5fb002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43b81bbac99cc79 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:50.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.764+0000 7f5fc659c640 1 -- 172.21.15.45:0/2590121061 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0031880 con 0x7f5fc80ae540 2023-11-28T21:51:50.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.764+0000 7f5fc659c640 1 -- 172.21.15.45:0/2590121061 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb00319e0 con 0x7f5fc80ae540 2023-11-28T21:51:50.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.764+0000 7f5fc659c640 1 -- 172.21.15.45:0/2590121061 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0031d70 con 0x7f5fc80ae540 2023-11-28T21:51:50.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.764+0000 7f5fcc988640 1 -- 172.21.15.45:0/2590121061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc80ae540 msgr2=0x7f5fc80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.764+0000 7f5fcc988640 1 --2- 172.21.15.45:0/2590121061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc80ae540 0x7f5fc80ce0f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f5fc806e6d0 tx=0x7f5fb002b920 comp rx=0 tx=0).stop 2023-11-28T21:51:50.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.765+0000 7f5fcc988640 1 -- 172.21.15.45:0/2590121061 shutdown_connections 2023-11-28T21:51:50.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.765+0000 7f5fcc988640 1 --2- 172.21.15.45:0/2590121061 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 0x7f5fc815ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.765+0000 7f5fcc988640 1 --2- 172.21.15.45:0/2590121061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc80ae540 0x7f5fc80ce0f0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.765+0000 7f5fcc988640 1 --2- 172.21.15.45:0/2590121061 >> v2:172.21.15.190:3300/0 conn(0x7f5fc8155780 0x7f5fc80abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.765+0000 7f5fcc988640 1 -- 172.21.15.45:0/2590121061 >> 172.21.15.45:0/2590121061 conn(0x7f5fc806ab30 msgr2=0x7f5fc80b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:50.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.765+0000 7f5fcc988640 1 -- 172.21.15.45:0/2590121061 shutdown_connections 2023-11-28T21:51:50.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.765+0000 7f5fcc988640 1 -- 172.21.15.45:0/2590121061 wait complete. 2023-11-28T21:51:50.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.766+0000 7f5fcc988640 1 Processor -- start 2023-11-28T21:51:50.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.767+0000 7f5fcc988640 1 -- start start 2023-11-28T21:51:50.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fcc988640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc80ae540 0x7f5fc81443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fcc988640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 0x7f5fc815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fcc988640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc8155780 0x7f5fc815a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fcc988640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc80cdd70 con 0x7f5fc8155780 2023-11-28T21:51:50.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fcc988640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8147c30 con 0x7f5fc80ae540 2023-11-28T21:51:50.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fcc988640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8144920 con 0x7f5fc80ce630 2023-11-28T21:51:50.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fc759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc8155780 0x7f5fc815a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fc759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc8155780 0x7f5fc815a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34598/0 (socket says 172.21.15.45:34598) 2023-11-28T21:51:50.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fc759e640 1 -- 172.21.15.45:0/1142051253 learned_addr learned my addr 172.21.15.45:0/1142051253 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:50.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fc6d9d640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc80ae540 0x7f5fc81443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.768+0000 7f5fbe59c640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 0x7f5fc815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fc759e640 1 -- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 msgr2=0x7f5fc815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fc759e640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 0x7f5fc815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fc759e640 1 -- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc80ae540 msgr2=0x7f5fc81443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fc759e640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc80ae540 0x7f5fc81443e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fc759e640 1 -- 172.21.15.45:0/1142051253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00094d0 con 0x7f5fc8155780 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fbe59c640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 0x7f5fc815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fc759e640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc8155780 0x7f5fc815a620 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f5fc000b410 tx=0x7f5fc000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fbffff640 1 -- 172.21.15.45:0/1142051253 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0011830 con 0x7f5fc8155780 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fbffff640 1 -- 172.21.15.45:0/1142051253 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc00119b0 con 0x7f5fc8155780 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fbffff640 1 -- 172.21.15.45:0/1142051253 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0011d60 con 0x7f5fc8155780 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.769+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc8144da0 con 0x7f5fc8155780 2023-11-28T21:51:50.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.770+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc8145360 con 0x7f5fc8155780 2023-11-28T21:51:50.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.771+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc804e920 con 0x7f5fc8155780 2023-11-28T21:51:50.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.772+0000 7f5fbffff640 1 -- 172.21.15.45:0/1142051253 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5fc001a400 con 0x7f5fc8155780 2023-11-28T21:51:50.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.773+0000 7f5fbffff640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f5fa806af60 0x7f5fa806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:50.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.773+0000 7f5fbffff640 1 -- 172.21.15.45:0/1142051253 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5fc00899b0 con 0x7f5fc8155780 2023-11-28T21:51:50.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.774+0000 7f5fc6d9d640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f5fa806af60 0x7f5fa806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:50.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.775+0000 7f5fc6d9d640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f5fa806af60 0x7f5fa806d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009f20 tx=0x7f5fb80094f0 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:50.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.778+0000 7f5fbffff640 1 -- 172.21.15.45:0/1142051253 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5fc0054880 con 0x7f5fc8155780 2023-11-28T21:51:50.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.971+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 --> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5fc815b740 con 0x7f5fa806af60 2023-11-28T21:51:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:50.994+0000 7f5fbffff640 1 -- 172.21.15.45:0/1142051253 <== mgr.7741 v2:172.21.15.190:6832/2040970137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286758 (secure 0 0 0) 0x7f5fb80096a0 con 0x7f5fa806af60 2023-11-28T21:51:51.000 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:51.001 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T21:51:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f5fa806af60 msgr2=0x7f5fa806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f5fa806af60 0x7f5fa806d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009f20 tx=0x7f5fb80094f0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc8155780 msgr2=0x7f5fc815a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc8155780 0x7f5fc815a620 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f5fc000b410 tx=0x7f5fc000b8d0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 shutdown_connections 2023-11-28T21:51:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f5fa806af60 0x7f5fa806d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fc8155780 0x7f5fc815a620 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5fc80ce630 0x7f5fc815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 --2- 172.21.15.45:0/1142051253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc80ae540 0x7f5fc81443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 >> 172.21.15.45:0/1142051253 conn(0x7f5fc806ab30 msgr2=0x7f5fc815d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:51.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 shutdown_connections 2023-11-28T21:51:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.001+0000 7f5fcc988640 1 -- 172.21.15.45:0/1142051253 wait complete. 2023-11-28T21:51:51.022 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-11-28T21:51:50.421129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1573,"num_read_kb":3215,"num_write":495,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4816,"ondisk_log_size":4816,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":357504,"kb_used_data":19212,"kb_used_omap":12,"kb_used_meta":338163,"kb_avail":754617216,"statfs":{"total":773094113280,"available":772728029184,"internally_reserved":0,"allocated":19673088,"data_stored":10608348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":346279472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002198"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862016+0000","last_change":"2023-11-28T21:51:45.878079+0000","last_active":"2023-11-28T21:51:46.862016+0000","last_peered":"2023-11-28T21:51:46.862016+0000","last_clean":"2023-11-28T21:51:46.862016+0000","last_became_active":"2023-11-28T21:51:45.877139+0000","last_became_peered":"2023-11-28T21:51:45.877139+0000","last_unstale":"2023-11-28T21:51:46.862016+0000","last_undegraded":"2023-11-28T21:51:46.862016+0000","last_fullsized":"2023-11-28T21:51:46.862016+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:03.595603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.858117+0000","last_change":"2023-11-28T21:51:45.870384+0000","last_active":"2023-11-28T21:51:46.858117+0000","last_peered":"2023-11-28T21:51:46.858117+0000","last_clean":"2023-11-28T21:51:46.858117+0000","last_became_active":"2023-11-28T21:51:45.869986+0000","last_became_peered":"2023-11-28T21:51:45.869986+0000","last_unstale":"2023-11-28T21:51:46.858117+0000","last_undegraded":"2023-11-28T21:51:46.858117+0000","last_fullsized":"2023-11-28T21:51:46.858117+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:01:01.721880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857617+0000","last_change":"2023-11-28T21:51:45.877935+0000","last_active":"2023-11-28T21:51:46.857617+0000","last_peered":"2023-11-28T21:51:46.857617+0000","last_clean":"2023-11-28T21:51:46.857617+0000","last_became_active":"2023-11-28T21:51:45.875509+0000","last_became_peered":"2023-11-28T21:51:45.875509+0000","last_unstale":"2023-11-28T21:51:46.857617+0000","last_undegraded":"2023-11-28T21:51:46.857617+0000","last_fullsized":"2023-11-28T21:51:46.857617+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:39.313237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862974+0000","last_change":"2023-11-28T21:51:45.872782+0000","last_active":"2023-11-28T21:51:46.862974+0000","last_peered":"2023-11-28T21:51:46.862974+0000","last_clean":"2023-11-28T21:51:46.862974+0000","last_became_active":"2023-11-28T21:51:45.872417+0000","last_became_peered":"2023-11-28T21:51:45.872417+0000","last_unstale":"2023-11-28T21:51:46.862974+0000","last_undegraded":"2023-11-28T21:51:46.862974+0000","last_fullsized":"2023-11-28T21:51:46.862974+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:49:15.783536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846342+0000","last_change":"2023-11-28T21:51:45.879034+0000","last_active":"2023-11-28T21:51:46.846342+0000","last_peered":"2023-11-28T21:51:46.846342+0000","last_clean":"2023-11-28T21:51:46.846342+0000","last_became_active":"2023-11-28T21:51:45.877738+0000","last_became_peered":"2023-11-28T21:51:45.877738+0000","last_unstale":"2023-11-28T21:51:46.846342+0000","last_undegraded":"2023-11-28T21:51:46.846342+0000","last_fullsized":"2023-11-28T21:51:46.846342+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:10.716408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846256+0000","last_change":"2023-11-28T21:51:45.876805+0000","last_active":"2023-11-28T21:51:46.846256+0000","last_peered":"2023-11-28T21:51:46.846256+0000","last_clean":"2023-11-28T21:51:46.846256+0000","last_became_active":"2023-11-28T21:51:45.876460+0000","last_became_peered":"2023-11-28T21:51:45.876460+0000","last_unstale":"2023-11-28T21:51:46.846256+0000","last_undegraded":"2023-11-28T21:51:46.846256+0000","last_fullsized":"2023-11-28T21:51:46.846256+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:24:12.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859424+0000","last_change":"2023-11-28T21:51:45.868813+0000","last_active":"2023-11-28T21:51:46.859424+0000","last_peered":"2023-11-28T21:51:46.859424+0000","last_clean":"2023-11-28T21:51:46.859424+0000","last_became_active":"2023-11-28T21:51:45.866088+0000","last_became_peered":"2023-11-28T21:51:45.866088+0000","last_unstale":"2023-11-28T21:51:46.859424+0000","last_undegraded":"2023-11-28T21:51:46.859424+0000","last_fullsized":"2023-11-28T21:51:46.859424+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:36:09.187494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.848924+0000","last_change":"2023-11-28T21:51:45.878796+0000","last_active":"2023-11-28T21:51:46.848924+0000","last_peered":"2023-11-28T21:51:46.848924+0000","last_clean":"2023-11-28T21:51:46.848924+0000","last_became_active":"2023-11-28T21:51:45.878494+0000","last_became_peered":"2023-11-28T21:51:45.878494+0000","last_unstale":"2023-11-28T21:51:46.848924+0000","last_undegraded":"2023-11-28T21:51:46.848924+0000","last_fullsized":"2023-11-28T21:51:46.848924+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:15:10.394505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846629+0000","last_change":"2023-11-28T21:51:45.885567+0000","last_active":"2023-11-28T21:51:46.846629+0000","last_peered":"2023-11-28T21:51:46.846629+0000","last_clean":"2023-11-28T21:51:46.846629+0000","last_became_active":"2023-11-28T21:51:45.885114+0000","last_became_peered":"2023-11-28T21:51:45.885114+0000","last_unstale":"2023-11-28T21:51:46.846629+0000","last_undegraded":"2023-11-28T21:51:46.846629+0000","last_fullsized":"2023-11-28T21:51:46.846629+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:03.743376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.847273+0000","last_change":"2023-11-28T21:51:45.877926+0000","last_active":"2023-11-28T21:51:46.847273+0000","last_peered":"2023-11-28T21:51:46.847273+0000","last_clean":"2023-11-28T21:51:46.847273+0000","last_became_active":"2023-11-28T21:51:45.877630+0000","last_became_peered":"2023-11-28T21:51:45.877630+0000","last_unstale":"2023-11-28T21:51:46.847273+0000","last_undegraded":"2023-11-28T21:51:46.847273+0000","last_fullsized":"2023-11-28T21:51:46.847273+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:17:41.920511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844924+0000","last_change":"2023-11-28T21:51:45.880525+0000","last_active":"2023-11-28T21:51:46.844924+0000","last_peered":"2023-11-28T21:51:46.844924+0000","last_clean":"2023-11-28T21:51:46.844924+0000","last_became_active":"2023-11-28T21:51:45.878656+0000","last_became_peered":"2023-11-28T21:51:45.878656+0000","last_unstale":"2023-11-28T21:51:46.844924+0000","last_undegraded":"2023-11-28T21:51:46.844924+0000","last_fullsized":"2023-11-28T21:51:46.844924+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:27:53.152730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863298+0000","last_change":"2023-11-28T21:51:45.882768+0000","last_active":"2023-11-28T21:51:46.863298+0000","last_peered":"2023-11-28T21:51:46.863298+0000","last_clean":"2023-11-28T21:51:46.863298+0000","last_became_active":"2023-11-28T21:51:45.882489+0000","last_became_peered":"2023-11-28T21:51:45.882489+0000","last_unstale":"2023-11-28T21:51:46.863298+0000","last_undegraded":"2023-11-28T21:51:46.863298+0000","last_fullsized":"2023-11-28T21:51:46.863298+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:53:10.392014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846222+0000","last_change":"2023-11-28T21:51:45.882283+0000","last_active":"2023-11-28T21:51:46.846222+0000","last_peered":"2023-11-28T21:51:46.846222+0000","last_clean":"2023-11-28T21:51:46.846222+0000","last_became_active":"2023-11-28T21:51:45.879568+0000","last_became_peered":"2023-11-28T21:51:45.879568+0000","last_unstale":"2023-11-28T21:51:46.846222+0000","last_undegraded":"2023-11-28T21:51:46.846222+0000","last_fullsized":"2023-11-28T21:51:46.846222+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:43:46.949024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860926+0000","last_change":"2023-11-28T21:51:45.881935+0000","last_active":"2023-11-28T21:51:46.860926+0000","last_peered":"2023-11-28T21:51:46.860926+0000","last_clean":"2023-11-28T21:51:46.860926+0000","last_became_active":"2023-11-28T21:51:45.881106+0000","last_became_peered":"2023-11-28T21:51:45.881106+0000","last_unstale":"2023-11-28T21:51:46.860926+0000","last_undegraded":"2023-11-28T21:51:46.860926+0000","last_fullsized":"2023-11-28T21:51:46.860926+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:02:30.064451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.856893+0000","last_change":"2023-11-28T21:51:45.876791+0000","last_active":"2023-11-28T21:51:46.856893+0000","last_peered":"2023-11-28T21:51:46.856893+0000","last_clean":"2023-11-28T21:51:46.856893+0000","last_became_active":"2023-11-28T21:51:45.876142+0000","last_became_peered":"2023-11-28T21:51:45.876142+0000","last_unstale":"2023-11-28T21:51:46.856893+0000","last_undegraded":"2023-11-28T21:51:46.856893+0000","last_fullsized":"2023-11-28T21:51:46.856893+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:58:40.619635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845753+0000","last_change":"2023-11-28T21:51:45.882304+0000","last_active":"2023-11-28T21:51:46.845753+0000","last_peered":"2023-11-28T21:51:46.845753+0000","last_clean":"2023-11-28T21:51:46.845753+0000","last_became_active":"2023-11-28T21:51:45.880898+0000","last_became_peered":"2023-11-28T21:51:45.880898+0000","last_unstale":"2023-11-28T21:51:46.845753+0000","last_undegraded":"2023-11-28T21:51:46.845753+0000","last_fullsized":"2023-11-28T21:51:46.845753+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:08:42.682615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862341+0000","last_change":"2023-11-28T21:51:45.873499+0000","last_active":"2023-11-28T21:51:46.862341+0000","last_peered":"2023-11-28T21:51:46.862341+0000","last_clean":"2023-11-28T21:51:46.862341+0000","last_became_active":"2023-11-28T21:51:45.873112+0000","last_became_peered":"2023-11-28T21:51:45.873112+0000","last_unstale":"2023-11-28T21:51:46.862341+0000","last_undegraded":"2023-11-28T21:51:46.862341+0000","last_fullsized":"2023-11-28T21:51:46.862341+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:01.025010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845003+0000","last_change":"2023-11-28T21:51:45.876388+0000","last_active":"2023-11-28T21:51:46.845003+0000","last_peered":"2023-11-28T21:51:46.845003+0000","last_clean":"2023-11-28T21:51:46.845003+0000","last_became_active":"2023-11-28T21:51:45.874079+0000","last_became_peered":"2023-11-28T21:51:45.874079+0000","last_unstale":"2023-11-28T21:51:46.845003+0000","last_undegraded":"2023-11-28T21:51:46.845003+0000","last_fullsized":"2023-11-28T21:51:46.845003+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:25:23.311571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862320+0000","last_change":"2023-11-28T21:51:45.878326+0000","last_active":"2023-11-28T21:51:46.862320+0000","last_peered":"2023-11-28T21:51:46.862320+0000","last_clean":"2023-11-28T21:51:46.862320+0000","last_became_active":"2023-11-28T21:51:45.877889+0000","last_became_peered":"2023-11-28T21:51:45.877889+0000","last_unstale":"2023-11-28T21:51:46.862320+0000","last_undegraded":"2023-11-28T21:51:46.862320+0000","last_fullsized":"2023-11-28T21:51:46.862320+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:58.310908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859339+0000","last_change":"2023-11-28T21:51:45.871842+0000","last_active":"2023-11-28T21:51:46.859339+0000","last_peered":"2023-11-28T21:51:46.859339+0000","last_clean":"2023-11-28T21:51:46.859339+0000","last_became_active":"2023-11-28T21:51:45.871422+0000","last_became_peered":"2023-11-28T21:51:45.871422+0000","last_unstale":"2023-11-28T21:51:46.859339+0000","last_undegraded":"2023-11-28T21:51:46.859339+0000","last_fullsized":"2023-11-28T21:51:46.859339+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:49:36.819922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846200+0000","last_change":"2023-11-28T21:51:45.883170+0000","last_active":"2023-11-28T21:51:46.846200+0000","last_peered":"2023-11-28T21:51:46.846200+0000","last_clean":"2023-11-28T21:51:46.846200+0000","last_became_active":"2023-11-28T21:51:45.881790+0000","last_became_peered":"2023-11-28T21:51:45.881790+0000","last_unstale":"2023-11-28T21:51:46.846200+0000","last_undegraded":"2023-11-28T21:51:46.846200+0000","last_fullsized":"2023-11-28T21:51:46.846200+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:58.310908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860259+0000","last_change":"2023-11-28T21:51:45.865242+0000","last_active":"2023-11-28T21:51:46.860259+0000","last_peered":"2023-11-28T21:51:46.860259+0000","last_clean":"2023-11-28T21:51:46.860259+0000","last_became_active":"2023-11-28T21:51:45.864937+0000","last_became_peered":"2023-11-28T21:51:45.864937+0000","last_unstale":"2023-11-28T21:51:46.860259+0000","last_undegraded":"2023-11-28T21:51:46.860259+0000","last_fullsized":"2023-11-28T21:51:46.860259+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:15:57.392428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849030+0000","last_change":"2023-11-28T21:51:45.871786+0000","last_active":"2023-11-28T21:51:46.849030+0000","last_peered":"2023-11-28T21:51:46.849030+0000","last_clean":"2023-11-28T21:51:46.849030+0000","last_became_active":"2023-11-28T21:51:45.871490+0000","last_became_peered":"2023-11-28T21:51:45.871490+0000","last_unstale":"2023-11-28T21:51:46.849030+0000","last_undegraded":"2023-11-28T21:51:46.849030+0000","last_fullsized":"2023-11-28T21:51:46.849030+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:33:53.274607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844413+0000","last_change":"2023-11-28T21:51:45.876606+0000","last_active":"2023-11-28T21:51:46.844413+0000","last_peered":"2023-11-28T21:51:46.844413+0000","last_clean":"2023-11-28T21:51:46.844413+0000","last_became_active":"2023-11-28T21:51:45.876297+0000","last_became_peered":"2023-11-28T21:51:45.876297+0000","last_unstale":"2023-11-28T21:51:46.844413+0000","last_undegraded":"2023-11-28T21:51:46.844413+0000","last_fullsized":"2023-11-28T21:51:46.844413+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:12.261849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862027+0000","last_change":"2023-11-28T21:51:45.868438+0000","last_active":"2023-11-28T21:51:46.862027+0000","last_peered":"2023-11-28T21:51:46.862027+0000","last_clean":"2023-11-28T21:51:46.862027+0000","last_became_active":"2023-11-28T21:51:45.868111+0000","last_became_peered":"2023-11-28T21:51:45.868111+0000","last_unstale":"2023-11-28T21:51:46.862027+0000","last_undegraded":"2023-11-28T21:51:46.862027+0000","last_fullsized":"2023-11-28T21:51:46.862027+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:52:57.310627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863299+0000","last_change":"2023-11-28T21:51:45.872229+0000","last_active":"2023-11-28T21:51:46.863299+0000","last_peered":"2023-11-28T21:51:46.863299+0000","last_clean":"2023-11-28T21:51:46.863299+0000","last_became_active":"2023-11-28T21:51:45.869659+0000","last_became_peered":"2023-11-28T21:51:45.869659+0000","last_unstale":"2023-11-28T21:51:46.863299+0000","last_undegraded":"2023-11-28T21:51:46.863299+0000","last_fullsized":"2023-11-28T21:51:46.863299+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:20.412397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846994+0000","last_change":"2023-11-28T21:51:45.881027+0000","last_active":"2023-11-28T21:51:46.846994+0000","last_peered":"2023-11-28T21:51:46.846994+0000","last_clean":"2023-11-28T21:51:46.846994+0000","last_became_active":"2023-11-28T21:51:45.878851+0000","last_became_peered":"2023-11-28T21:51:45.878851+0000","last_unstale":"2023-11-28T21:51:46.846994+0000","last_undegraded":"2023-11-28T21:51:46.846994+0000","last_fullsized":"2023-11-28T21:51:46.846994+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:43:11.146254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844656+0000","last_change":"2023-11-28T21:51:45.867114+0000","last_active":"2023-11-28T21:51:46.844656+0000","last_peered":"2023-11-28T21:51:46.844656+0000","last_clean":"2023-11-28T21:51:46.844656+0000","last_became_active":"2023-11-28T21:51:45.866735+0000","last_became_peered":"2023-11-28T21:51:45.866735+0000","last_unstale":"2023-11-28T21:51:46.844656+0000","last_undegraded":"2023-11-28T21:51:46.844656+0000","last_fullsized":"2023-11-28T21:51:46.844656+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:07:20.731285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849871+0000","last_change":"2023-11-28T21:51:45.885398+0000","last_active":"2023-11-28T21:51:46.849871+0000","last_peered":"2023-11-28T21:51:46.849871+0000","last_clean":"2023-11-28T21:51:46.849871+0000","last_became_active":"2023-11-28T21:51:45.885081+0000","last_became_peered":"2023-11-28T21:51:45.885081+0000","last_unstale":"2023-11-28T21:51:46.849871+0000","last_undegraded":"2023-11-28T21:51:46.849871+0000","last_fullsized":"2023-11-28T21:51:46.849871+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:38:43.699130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844662+0000","last_change":"2023-11-28T21:51:45.869331+0000","last_active":"2023-11-28T21:51:46.844662+0000","last_peered":"2023-11-28T21:51:46.844662+0000","last_clean":"2023-11-28T21:51:46.844662+0000","last_became_active":"2023-11-28T21:51:45.868975+0000","last_became_peered":"2023-11-28T21:51:45.868975+0000","last_unstale":"2023-11-28T21:51:46.844662+0000","last_undegraded":"2023-11-28T21:51:46.844662+0000","last_fullsized":"2023-11-28T21:51:46.844662+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:19.332000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845926+0000","last_change":"2023-11-28T21:51:45.880541+0000","last_active":"2023-11-28T21:51:46.845926+0000","last_peered":"2023-11-28T21:51:46.845926+0000","last_clean":"2023-11-28T21:51:46.845926+0000","last_became_active":"2023-11-28T21:51:45.880252+0000","last_became_peered":"2023-11-28T21:51:45.880252+0000","last_unstale":"2023-11-28T21:51:46.845926+0000","last_undegraded":"2023-11-28T21:51:46.845926+0000","last_fullsized":"2023-11-28T21:51:46.845926+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:08:53.846283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.848997+0000","last_change":"2023-11-28T21:51:45.887436+0000","last_active":"2023-11-28T21:51:46.848997+0000","last_peered":"2023-11-28T21:51:46.848997+0000","last_clean":"2023-11-28T21:51:46.848997+0000","last_became_active":"2023-11-28T21:51:45.887077+0000","last_became_peered":"2023-11-28T21:51:45.887077+0000","last_unstale":"2023-11-28T21:51:46.848997+0000","last_undegraded":"2023-11-28T21:51:46.848997+0000","last_fullsized":"2023-11-28T21:51:46.848997+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:19:19.212771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857692+0000","last_change":"2023-11-28T21:51:45.866602+0000","last_active":"2023-11-28T21:51:46.857692+0000","last_peered":"2023-11-28T21:51:46.857692+0000","last_clean":"2023-11-28T21:51:46.857692+0000","last_became_active":"2023-11-28T21:51:45.866022+0000","last_became_peered":"2023-11-28T21:51:45.866022+0000","last_unstale":"2023-11-28T21:51:46.857692+0000","last_undegraded":"2023-11-28T21:51:46.857692+0000","last_fullsized":"2023-11-28T21:51:46.857692+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:44.823306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"184'4","reported_seq":16,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846964+0000","last_change":"2023-11-28T21:51:44.938778+0000","last_active":"2023-11-28T21:51:46.846964+0000","last_peered":"2023-11-28T21:51:46.846964+0000","last_clean":"2023-11-28T21:51:46.846964+0000","last_became_active":"2023-11-28T21:51:44.938490+0000","last_became_peered":"2023-11-28T21:51:44.938490+0000","last_unstale":"2023-11-28T21:51:46.846964+0000","last_undegraded":"2023-11-28T21:51:46.846964+0000","last_fullsized":"2023-11-28T21:51:46.846964+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:24:28.548759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857953+0000","last_change":"2023-11-28T21:51:45.874870+0000","last_active":"2023-11-28T21:51:46.857953+0000","last_peered":"2023-11-28T21:51:46.857953+0000","last_clean":"2023-11-28T21:51:46.857953+0000","last_became_active":"2023-11-28T21:51:45.874694+0000","last_became_peered":"2023-11-28T21:51:45.874694+0000","last_unstale":"2023-11-28T21:51:46.857953+0000","last_undegraded":"2023-11-28T21:51:46.857953+0000","last_fullsized":"2023-11-28T21:51:46.857953+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:57.433051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860824+0000","last_change":"2023-11-28T21:51:44.956071+0000","last_active":"2023-11-28T21:51:46.860824+0000","last_peered":"2023-11-28T21:51:46.860824+0000","last_clean":"2023-11-28T21:51:46.860824+0000","last_became_active":"2023-11-28T21:51:44.955513+0000","last_became_peered":"2023-11-28T21:51:44.955513+0000","last_unstale":"2023-11-28T21:51:46.860824+0000","last_undegraded":"2023-11-28T21:51:46.860824+0000","last_fullsized":"2023-11-28T21:51:46.860824+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:32.173902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.850381+0000","last_change":"2023-11-28T21:51:45.875314+0000","last_active":"2023-11-28T21:51:46.850381+0000","last_peered":"2023-11-28T21:51:46.850381+0000","last_clean":"2023-11-28T21:51:46.850381+0000","last_became_active":"2023-11-28T21:51:45.874679+0000","last_became_peered":"2023-11-28T21:51:45.874679+0000","last_unstale":"2023-11-28T21:51:46.850381+0000","last_undegraded":"2023-11-28T21:51:46.850381+0000","last_fullsized":"2023-11-28T21:51:46.850381+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:02:05.357903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862940+0000","last_change":"2023-11-28T21:51:44.956398+0000","last_active":"2023-11-28T21:51:46.862940+0000","last_peered":"2023-11-28T21:51:46.862940+0000","last_clean":"2023-11-28T21:51:46.862940+0000","last_became_active":"2023-11-28T21:51:44.954997+0000","last_became_peered":"2023-11-28T21:51:44.954997+0000","last_unstale":"2023-11-28T21:51:46.862940+0000","last_undegraded":"2023-11-28T21:51:46.862940+0000","last_fullsized":"2023-11-28T21:51:46.862940+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:49:35.804765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860962+0000","last_change":"2023-11-28T21:51:45.875181+0000","last_active":"2023-11-28T21:51:46.860962+0000","last_peered":"2023-11-28T21:51:46.860962+0000","last_clean":"2023-11-28T21:51:46.860962+0000","last_became_active":"2023-11-28T21:51:45.874246+0000","last_became_peered":"2023-11-28T21:51:45.874246+0000","last_unstale":"2023-11-28T21:51:46.860962+0000","last_undegraded":"2023-11-28T21:51:46.860962+0000","last_fullsized":"2023-11-28T21:51:46.860962+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:08:42.682615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857527+0000","last_change":"2023-11-28T21:51:44.940748+0000","last_active":"2023-11-28T21:51:46.857527+0000","last_peered":"2023-11-28T21:51:46.857527+0000","last_clean":"2023-11-28T21:51:46.857527+0000","last_became_active":"2023-11-28T21:51:44.940456+0000","last_became_peered":"2023-11-28T21:51:44.940456+0000","last_unstale":"2023-11-28T21:51:46.857527+0000","last_undegraded":"2023-11-28T21:51:46.857527+0000","last_fullsized":"2023-11-28T21:51:46.857527+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:53:09.376856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.858645+0000","last_change":"2023-11-28T21:51:45.865601+0000","last_active":"2023-11-28T21:51:46.858645+0000","last_peered":"2023-11-28T21:51:46.858645+0000","last_clean":"2023-11-28T21:51:46.858645+0000","last_became_active":"2023-11-28T21:51:45.864793+0000","last_became_peered":"2023-11-28T21:51:45.864793+0000","last_unstale":"2023-11-28T21:51:46.858645+0000","last_undegraded":"2023-11-28T21:51:46.858645+0000","last_fullsized":"2023-11-28T21:51:46.858645+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:32:54.787053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.850130+0000","last_change":"2023-11-28T21:51:44.956999+0000","last_active":"2023-11-28T21:51:46.850130+0000","last_peered":"2023-11-28T21:51:46.850130+0000","last_clean":"2023-11-28T21:51:46.850130+0000","last_became_active":"2023-11-28T21:51:44.956355+0000","last_became_peered":"2023-11-28T21:51:44.956355+0000","last_unstale":"2023-11-28T21:51:46.850130+0000","last_undegraded":"2023-11-28T21:51:46.850130+0000","last_fullsized":"2023-11-28T21:51:46.850130+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:29:51.530533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860975+0000","last_change":"2023-11-28T21:51:45.875060+0000","last_active":"2023-11-28T21:51:46.860975+0000","last_peered":"2023-11-28T21:51:46.860975+0000","last_clean":"2023-11-28T21:51:46.860975+0000","last_became_active":"2023-11-28T21:51:45.873876+0000","last_became_peered":"2023-11-28T21:51:45.873876+0000","last_unstale":"2023-11-28T21:51:46.860975+0000","last_undegraded":"2023-11-28T21:51:46.860975+0000","last_fullsized":"2023-11-28T21:51:46.860975+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:08:00.930788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846700+0000","last_change":"2023-11-28T21:51:44.945722+0000","last_active":"2023-11-28T21:51:46.846700+0000","last_peered":"2023-11-28T21:51:46.846700+0000","last_clean":"2023-11-28T21:51:46.846700+0000","last_became_active":"2023-11-28T21:51:44.945393+0000","last_became_peered":"2023-11-28T21:51:44.945393+0000","last_unstale":"2023-11-28T21:51:46.846700+0000","last_undegraded":"2023-11-28T21:51:46.846700+0000","last_fullsized":"2023-11-28T21:51:46.846700+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:45:47.899500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857241+0000","last_change":"2023-11-28T21:51:45.867180+0000","last_active":"2023-11-28T21:51:46.857241+0000","last_peered":"2023-11-28T21:51:46.857241+0000","last_clean":"2023-11-28T21:51:46.857241+0000","last_became_active":"2023-11-28T21:51:45.866844+0000","last_became_peered":"2023-11-28T21:51:45.866844+0000","last_unstale":"2023-11-28T21:51:46.857241+0000","last_undegraded":"2023-11-28T21:51:46.857241+0000","last_fullsized":"2023-11-28T21:51:46.857241+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:53:06.355914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859846+0000","last_change":"2023-11-28T21:51:44.935312+0000","last_active":"2023-11-28T21:51:46.859846+0000","last_peered":"2023-11-28T21:51:46.859846+0000","last_clean":"2023-11-28T21:51:46.859846+0000","last_became_active":"2023-11-28T21:51:44.935014+0000","last_became_peered":"2023-11-28T21:51:44.935014+0000","last_unstale":"2023-11-28T21:51:46.859846+0000","last_undegraded":"2023-11-28T21:51:46.859846+0000","last_fullsized":"2023-11-28T21:51:46.859846+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:34.679453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860755+0000","last_change":"2023-11-28T21:51:45.879934+0000","last_active":"2023-11-28T21:51:46.860755+0000","last_peered":"2023-11-28T21:51:46.860755+0000","last_clean":"2023-11-28T21:51:46.860755+0000","last_became_active":"2023-11-28T21:51:45.879584+0000","last_became_peered":"2023-11-28T21:51:45.879584+0000","last_unstale":"2023-11-28T21:51:46.860755+0000","last_undegraded":"2023-11-28T21:51:46.860755+0000","last_fullsized":"2023-11-28T21:51:46.860755+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:49.171516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845825+0000","last_change":"2023-11-28T21:51:44.949413+0000","last_active":"2023-11-28T21:51:46.845825+0000","last_peered":"2023-11-28T21:51:46.845825+0000","last_clean":"2023-11-28T21:51:46.845825+0000","last_became_active":"2023-11-28T21:51:44.948658+0000","last_became_peered":"2023-11-28T21:51:44.948658+0000","last_unstale":"2023-11-28T21:51:46.845825+0000","last_undegraded":"2023-11-28T21:51:46.845825+0000","last_fullsized":"2023-11-28T21:51:46.845825+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:37.048471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.850120+0000","last_change":"2023-11-28T21:51:45.873238+0000","last_active":"2023-11-28T21:51:46.850120+0000","last_peered":"2023-11-28T21:51:46.850120+0000","last_clean":"2023-11-28T21:51:46.850120+0000","last_became_active":"2023-11-28T21:51:45.872945+0000","last_became_peered":"2023-11-28T21:51:45.872945+0000","last_unstale":"2023-11-28T21:51:46.850120+0000","last_undegraded":"2023-11-28T21:51:46.850120+0000","last_fullsized":"2023-11-28T21:51:46.850120+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:22:54.646623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861339+0000","last_change":"2023-11-28T21:51:44.955272+0000","last_active":"2023-11-28T21:51:46.861339+0000","last_peered":"2023-11-28T21:51:46.861339+0000","last_clean":"2023-11-28T21:51:46.861339+0000","last_became_active":"2023-11-28T21:51:44.954729+0000","last_became_peered":"2023-11-28T21:51:44.954729+0000","last_unstale":"2023-11-28T21:51:46.861339+0000","last_undegraded":"2023-11-28T21:51:46.861339+0000","last_fullsized":"2023-11-28T21:51:46.861339+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:37.048471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845127+0000","last_change":"2023-11-28T21:51:45.870661+0000","last_active":"2023-11-28T21:51:46.845127+0000","last_peered":"2023-11-28T21:51:46.845127+0000","last_clean":"2023-11-28T21:51:46.845127+0000","last_became_active":"2023-11-28T21:51:45.870347+0000","last_became_peered":"2023-11-28T21:51:45.870347+0000","last_unstale":"2023-11-28T21:51:46.845127+0000","last_undegraded":"2023-11-28T21:51:46.845127+0000","last_fullsized":"2023-11-28T21:51:46.845127+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:30:24.222242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863898+0000","last_change":"2023-11-28T21:51:44.942670+0000","last_active":"2023-11-28T21:51:46.863898+0000","last_peered":"2023-11-28T21:51:46.863898+0000","last_clean":"2023-11-28T21:51:46.863898+0000","last_became_active":"2023-11-28T21:51:44.940785+0000","last_became_peered":"2023-11-28T21:51:44.940785+0000","last_unstale":"2023-11-28T21:51:46.863898+0000","last_undegraded":"2023-11-28T21:51:46.863898+0000","last_fullsized":"2023-11-28T21:51:46.863898+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:14:34.926484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845032+0000","last_change":"2023-11-28T21:51:45.876091+0000","last_active":"2023-11-28T21:51:46.845032+0000","last_peered":"2023-11-28T21:51:46.845032+0000","last_clean":"2023-11-28T21:51:46.845032+0000","last_became_active":"2023-11-28T21:51:45.873605+0000","last_became_peered":"2023-11-28T21:51:45.873605+0000","last_unstale":"2023-11-28T21:51:46.845032+0000","last_undegraded":"2023-11-28T21:51:46.845032+0000","last_fullsized":"2023-11-28T21:51:46.845032+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:07:10.605728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"184'3","reported_seq":15,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.845206+0000","last_change":"2023-11-28T21:51:44.936696+0000","last_active":"2023-11-28T21:51:46.845206+0000","last_peered":"2023-11-28T21:51:46.845206+0000","last_clean":"2023-11-28T21:51:46.845206+0000","last_became_active":"2023-11-28T21:51:44.936426+0000","last_became_peered":"2023-11-28T21:51:44.936426+0000","last_unstale":"2023-11-28T21:51:46.845206+0000","last_undegraded":"2023-11-28T21:51:46.845206+0000","last_fullsized":"2023-11-28T21:51:46.845206+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:32:12.144221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849282+0000","last_change":"2023-11-28T21:51:45.874326+0000","last_active":"2023-11-28T21:51:46.849282+0000","last_peered":"2023-11-28T21:51:46.849282+0000","last_clean":"2023-11-28T21:51:46.849282+0000","last_became_active":"2023-11-28T21:51:45.873633+0000","last_became_peered":"2023-11-28T21:51:45.873633+0000","last_unstale":"2023-11-28T21:51:46.849282+0000","last_undegraded":"2023-11-28T21:51:46.849282+0000","last_fullsized":"2023-11-28T21:51:46.849282+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:58:35.716414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846427+0000","last_change":"2023-11-28T21:51:44.937471+0000","last_active":"2023-11-28T21:51:46.846427+0000","last_peered":"2023-11-28T21:51:46.846427+0000","last_clean":"2023-11-28T21:51:46.846427+0000","last_became_active":"2023-11-28T21:51:44.937096+0000","last_became_peered":"2023-11-28T21:51:44.937096+0000","last_unstale":"2023-11-28T21:51:46.846427+0000","last_undegraded":"2023-11-28T21:51:46.846427+0000","last_fullsized":"2023-11-28T21:51:46.846427+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:55:44.808450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857190+0000","last_change":"2023-11-28T21:51:45.876970+0000","last_active":"2023-11-28T21:51:46.857190+0000","last_peered":"2023-11-28T21:51:46.857190+0000","last_clean":"2023-11-28T21:51:46.857190+0000","last_became_active":"2023-11-28T21:51:45.876518+0000","last_became_peered":"2023-11-28T21:51:45.876518+0000","last_unstale":"2023-11-28T21:51:46.857190+0000","last_undegraded":"2023-11-28T21:51:46.857190+0000","last_fullsized":"2023-11-28T21:51:46.857190+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:37:21.095095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861342+0000","last_change":"2023-11-28T21:51:44.952532+0000","last_active":"2023-11-28T21:51:46.861342+0000","last_peered":"2023-11-28T21:51:46.861342+0000","last_clean":"2023-11-28T21:51:46.861342+0000","last_became_active":"2023-11-28T21:51:44.952240+0000","last_became_peered":"2023-11-28T21:51:44.952240+0000","last_unstale":"2023-11-28T21:51:46.861342+0000","last_undegraded":"2023-11-28T21:51:46.861342+0000","last_fullsized":"2023-11-28T21:51:46.861342+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:09:33.914911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846693+0000","last_change":"2023-11-28T21:51:45.873066+0000","last_active":"2023-11-28T21:51:46.846693+0000","last_peered":"2023-11-28T21:51:46.846693+0000","last_clean":"2023-11-28T21:51:46.846693+0000","last_became_active":"2023-11-28T21:51:45.872695+0000","last_became_peered":"2023-11-28T21:51:45.872695+0000","last_unstale":"2023-11-28T21:51:46.846693+0000","last_undegraded":"2023-11-28T21:51:46.846693+0000","last_fullsized":"2023-11-28T21:51:46.846693+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:48:48.044068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861580+0000","last_change":"2023-11-28T21:51:44.945752+0000","last_active":"2023-11-28T21:51:46.861580+0000","last_peered":"2023-11-28T21:51:46.861580+0000","last_clean":"2023-11-28T21:51:46.861580+0000","last_became_active":"2023-11-28T21:51:44.945423+0000","last_became_peered":"2023-11-28T21:51:44.945423+0000","last_unstale":"2023-11-28T21:51:46.861580+0000","last_undegraded":"2023-11-28T21:51:46.861580+0000","last_fullsized":"2023-11-28T21:51:46.861580+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:40:54.257461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857318+0000","last_change":"2023-11-28T21:51:45.883636+0000","last_active":"2023-11-28T21:51:46.857318+0000","last_peered":"2023-11-28T21:51:46.857318+0000","last_clean":"2023-11-28T21:51:46.857318+0000","last_became_active":"2023-11-28T21:51:45.883228+0000","last_became_peered":"2023-11-28T21:51:45.883228+0000","last_unstale":"2023-11-28T21:51:46.857318+0000","last_undegraded":"2023-11-28T21:51:46.857318+0000","last_fullsized":"2023-11-28T21:51:46.857318+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:07:43.308822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"184'8","reported_seq":20,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.858867+0000","last_change":"2023-11-28T21:51:44.951036+0000","last_active":"2023-11-28T21:51:46.858867+0000","last_peered":"2023-11-28T21:51:46.858867+0000","last_clean":"2023-11-28T21:51:46.858867+0000","last_became_active":"2023-11-28T21:51:44.950584+0000","last_became_peered":"2023-11-28T21:51:44.950584+0000","last_unstale":"2023-11-28T21:51:46.858867+0000","last_undegraded":"2023-11-28T21:51:46.858867+0000","last_fullsized":"2023-11-28T21:51:46.858867+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:57:36.241338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861016+0000","last_change":"2023-11-28T21:51:45.883146+0000","last_active":"2023-11-28T21:51:46.861016+0000","last_peered":"2023-11-28T21:51:46.861016+0000","last_clean":"2023-11-28T21:51:46.861016+0000","last_became_active":"2023-11-28T21:51:45.880601+0000","last_became_peered":"2023-11-28T21:51:45.880601+0000","last_unstale":"2023-11-28T21:51:46.861016+0000","last_undegraded":"2023-11-28T21:51:46.861016+0000","last_fullsized":"2023-11-28T21:51:46.861016+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:08:53.846283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844729+0000","last_change":"2023-11-28T21:51:44.951766+0000","last_active":"2023-11-28T21:51:46.844729+0000","last_peered":"2023-11-28T21:51:46.844729+0000","last_clean":"2023-11-28T21:51:46.844729+0000","last_became_active":"2023-11-28T21:51:44.951154+0000","last_became_peered":"2023-11-28T21:51:44.951154+0000","last_unstale":"2023-11-28T21:51:46.844729+0000","last_undegraded":"2023-11-28T21:51:46.844729+0000","last_fullsized":"2023-11-28T21:51:46.844729+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:35:09.211879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862697+0000","last_change":"2023-11-28T21:51:45.878486+0000","last_active":"2023-11-28T21:51:46.862697+0000","last_peered":"2023-11-28T21:51:46.862697+0000","last_clean":"2023-11-28T21:51:46.862697+0000","last_became_active":"2023-11-28T21:51:45.878177+0000","last_became_peered":"2023-11-28T21:51:45.878177+0000","last_unstale":"2023-11-28T21:51:46.862697+0000","last_undegraded":"2023-11-28T21:51:46.862697+0000","last_fullsized":"2023-11-28T21:51:46.862697+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:43:46.949024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861068+0000","last_change":"2023-11-28T21:51:44.956246+0000","last_active":"2023-11-28T21:51:46.861068+0000","last_peered":"2023-11-28T21:51:46.861068+0000","last_clean":"2023-11-28T21:51:46.861068+0000","last_became_active":"2023-11-28T21:51:44.955819+0000","last_became_peered":"2023-11-28T21:51:44.955819+0000","last_unstale":"2023-11-28T21:51:46.861068+0000","last_undegraded":"2023-11-28T21:51:46.861068+0000","last_fullsized":"2023-11-28T21:51:46.861068+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:45:47.899500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863681+0000","last_change":"2023-11-28T21:51:45.880930+0000","last_active":"2023-11-28T21:51:46.863681+0000","last_peered":"2023-11-28T21:51:46.863681+0000","last_clean":"2023-11-28T21:51:46.863681+0000","last_became_active":"2023-11-28T21:51:45.880664+0000","last_became_peered":"2023-11-28T21:51:45.880664+0000","last_unstale":"2023-11-28T21:51:46.863681+0000","last_undegraded":"2023-11-28T21:51:46.863681+0000","last_fullsized":"2023-11-28T21:51:46.863681+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:29:01.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"184'1","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857709+0000","last_change":"2023-11-28T21:51:44.954935+0000","last_active":"2023-11-28T21:51:46.857709+0000","last_peered":"2023-11-28T21:51:46.857709+0000","last_clean":"2023-11-28T21:51:46.857709+0000","last_became_active":"2023-11-28T21:51:44.954537+0000","last_became_peered":"2023-11-28T21:51:44.954537+0000","last_unstale":"2023-11-28T21:51:46.857709+0000","last_undegraded":"2023-11-28T21:51:46.857709+0000","last_fullsized":"2023-11-28T21:51:46.857709+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:21:49.107429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862380+0000","last_change":"2023-11-28T21:51:45.885482+0000","last_active":"2023-11-28T21:51:46.862380+0000","last_peered":"2023-11-28T21:51:46.862380+0000","last_clean":"2023-11-28T21:51:46.862380+0000","last_became_active":"2023-11-28T21:51:45.882481+0000","last_became_peered":"2023-11-28T21:51:45.882481+0000","last_unstale":"2023-11-28T21:51:46.862380+0000","last_undegraded":"2023-11-28T21:51:46.862380+0000","last_fullsized":"2023-11-28T21:51:46.862380+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:58:01.699459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863177+0000","last_change":"2023-11-28T21:51:44.952419+0000","last_active":"2023-11-28T21:51:46.863177+0000","last_peered":"2023-11-28T21:51:46.863177+0000","last_clean":"2023-11-28T21:51:46.863177+0000","last_became_active":"2023-11-28T21:51:44.952131+0000","last_became_peered":"2023-11-28T21:51:44.952131+0000","last_unstale":"2023-11-28T21:51:46.863177+0000","last_undegraded":"2023-11-28T21:51:46.863177+0000","last_fullsized":"2023-11-28T21:51:46.863177+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:59:13.462136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859583+0000","last_change":"2023-11-28T21:51:45.881750+0000","last_active":"2023-11-28T21:51:46.859583+0000","last_peered":"2023-11-28T21:51:46.859583+0000","last_clean":"2023-11-28T21:51:46.859583+0000","last_became_active":"2023-11-28T21:51:45.881456+0000","last_became_peered":"2023-11-28T21:51:45.881456+0000","last_unstale":"2023-11-28T21:51:46.859583+0000","last_undegraded":"2023-11-28T21:51:46.859583+0000","last_fullsized":"2023-11-28T21:51:46.859583+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:24:29.563916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859091+0000","last_change":"2023-11-28T21:51:44.948846+0000","last_active":"2023-11-28T21:51:46.859091+0000","last_peered":"2023-11-28T21:51:46.859091+0000","last_clean":"2023-11-28T21:51:46.859091+0000","last_became_active":"2023-11-28T21:51:44.948425+0000","last_became_peered":"2023-11-28T21:51:44.948425+0000","last_unstale":"2023-11-28T21:51:46.859091+0000","last_undegraded":"2023-11-28T21:51:46.859091+0000","last_fullsized":"2023-11-28T21:51:46.859091+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:32.173902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857062+0000","last_change":"2023-11-28T21:51:45.868425+0000","last_active":"2023-11-28T21:51:46.857062+0000","last_peered":"2023-11-28T21:51:46.857062+0000","last_clean":"2023-11-28T21:51:46.857062+0000","last_became_active":"2023-11-28T21:51:45.868064+0000","last_became_peered":"2023-11-28T21:51:45.868064+0000","last_unstale":"2023-11-28T21:51:46.857062+0000","last_undegraded":"2023-11-28T21:51:46.857062+0000","last_fullsized":"2023-11-28T21:51:46.857062+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:26:19.218196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849502+0000","last_change":"2023-11-28T21:51:44.947351+0000","last_active":"2023-11-28T21:51:46.849502+0000","last_peered":"2023-11-28T21:51:46.849502+0000","last_clean":"2023-11-28T21:51:46.849502+0000","last_became_active":"2023-11-28T21:51:44.947075+0000","last_became_peered":"2023-11-28T21:51:44.947075+0000","last_unstale":"2023-11-28T21:51:46.849502+0000","last_undegraded":"2023-11-28T21:51:46.849502+0000","last_fullsized":"2023-11-28T21:51:46.849502+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:36:35.800471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859960+0000","last_change":"2023-11-28T21:51:45.880030+0000","last_active":"2023-11-28T21:51:46.859960+0000","last_peered":"2023-11-28T21:51:46.859960+0000","last_clean":"2023-11-28T21:51:46.859960+0000","last_became_active":"2023-11-28T21:51:45.879668+0000","last_became_peered":"2023-11-28T21:51:45.879668+0000","last_unstale":"2023-11-28T21:51:46.859960+0000","last_undegraded":"2023-11-28T21:51:46.859960+0000","last_fullsized":"2023-11-28T21:51:46.859960+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:55:45.823607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.858781+0000","last_change":"2023-11-28T21:51:44.945354+0000","last_active":"2023-11-28T21:51:46.858781+0000","last_peered":"2023-11-28T21:51:46.858781+0000","last_clean":"2023-11-28T21:51:46.858781+0000","last_became_active":"2023-11-28T21:51:44.944812+0000","last_became_peered":"2023-11-28T21:51:44.944812+0000","last_unstale":"2023-11-28T21:51:46.858781+0000","last_undegraded":"2023-11-28T21:51:46.858781+0000","last_fullsized":"2023-11-28T21:51:46.858781+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:24:44.899820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857187+0000","last_change":"2023-11-28T21:51:45.877419+0000","last_active":"2023-11-28T21:51:46.857187+0000","last_peered":"2023-11-28T21:51:46.857187+0000","last_clean":"2023-11-28T21:51:46.857187+0000","last_became_active":"2023-11-28T21:51:45.873727+0000","last_became_peered":"2023-11-28T21:51:45.873727+0000","last_unstale":"2023-11-28T21:51:46.857187+0000","last_undegraded":"2023-11-28T21:51:46.857187+0000","last_fullsized":"2023-11-28T21:51:46.857187+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:22:24.545177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849540+0000","last_change":"2023-11-28T21:51:44.940523+0000","last_active":"2023-11-28T21:51:46.849540+0000","last_peered":"2023-11-28T21:51:46.849540+0000","last_clean":"2023-11-28T21:51:46.849540+0000","last_became_active":"2023-11-28T21:51:44.940236+0000","last_became_peered":"2023-11-28T21:51:44.940236+0000","last_unstale":"2023-11-28T21:51:46.849540+0000","last_undegraded":"2023-11-28T21:51:46.849540+0000","last_fullsized":"2023-11-28T21:51:46.849540+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:24:54.137809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846729+0000","last_change":"2023-11-28T21:51:45.879426+0000","last_active":"2023-11-28T21:51:46.846729+0000","last_peered":"2023-11-28T21:51:46.846729+0000","last_clean":"2023-11-28T21:51:46.846729+0000","last_became_active":"2023-11-28T21:51:45.878959+0000","last_became_peered":"2023-11-28T21:51:45.878959+0000","last_unstale":"2023-11-28T21:51:46.846729+0000","last_undegraded":"2023-11-28T21:51:46.846729+0000","last_fullsized":"2023-11-28T21:51:46.846729+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:58:26.996916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"184'2","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862435+0000","last_change":"2023-11-28T21:51:44.953351+0000","last_active":"2023-11-28T21:51:46.862435+0000","last_peered":"2023-11-28T21:51:46.862435+0000","last_clean":"2023-11-28T21:51:46.862435+0000","last_became_active":"2023-11-28T21:51:44.953094+0000","last_became_peered":"2023-11-28T21:51:44.953094+0000","last_unstale":"2023-11-28T21:51:46.862435+0000","last_undegraded":"2023-11-28T21:51:46.862435+0000","last_fullsized":"2023-11-28T21:51:46.862435+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:14:15.482556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.862465+0000","last_change":"2023-11-28T21:51:45.885000+0000","last_active":"2023-11-28T21:51:46.862465+0000","last_peered":"2023-11-28T21:51:46.862465+0000","last_clean":"2023-11-28T21:51:46.862465+0000","last_became_active":"2023-11-28T21:51:45.884699+0000","last_became_peered":"2023-11-28T21:51:45.884699+0000","last_unstale":"2023-11-28T21:51:46.862465+0000","last_undegraded":"2023-11-28T21:51:46.862465+0000","last_fullsized":"2023-11-28T21:51:46.862465+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:37:12.676367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.864045+0000","last_change":"2023-11-28T21:51:44.952870+0000","last_active":"2023-11-28T21:51:46.864045+0000","last_peered":"2023-11-28T21:51:46.864045+0000","last_clean":"2023-11-28T21:51:46.864045+0000","last_became_active":"2023-11-28T21:51:44.950437+0000","last_became_peered":"2023-11-28T21:51:44.950437+0000","last_unstale":"2023-11-28T21:51:46.864045+0000","last_undegraded":"2023-11-28T21:51:46.864045+0000","last_fullsized":"2023-11-28T21:51:46.864045+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:49.752643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.863885+0000","last_change":"2023-11-28T21:51:45.877417+0000","last_active":"2023-11-28T21:51:46.863885+0000","last_peered":"2023-11-28T21:51:46.863885+0000","last_clean":"2023-11-28T21:51:46.863885+0000","last_became_active":"2023-11-28T21:51:45.877131+0000","last_became_peered":"2023-11-28T21:51:45.877131+0000","last_unstale":"2023-11-28T21:51:46.863885+0000","last_undegraded":"2023-11-28T21:51:46.863885+0000","last_fullsized":"2023-11-28T21:51:46.863885+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:02.659275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857789+0000","last_change":"2023-11-28T21:51:44.940553+0000","last_active":"2023-11-28T21:51:46.857789+0000","last_peered":"2023-11-28T21:51:46.857789+0000","last_clean":"2023-11-28T21:51:46.857789+0000","last_became_active":"2023-11-28T21:51:44.940201+0000","last_became_peered":"2023-11-28T21:51:44.940201+0000","last_unstale":"2023-11-28T21:51:46.857789+0000","last_undegraded":"2023-11-28T21:51:46.857789+0000","last_fullsized":"2023-11-28T21:51:46.857789+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:20:08.464784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846825+0000","last_change":"2023-11-28T21:51:45.881560+0000","last_active":"2023-11-28T21:51:46.846825+0000","last_peered":"2023-11-28T21:51:46.846825+0000","last_clean":"2023-11-28T21:51:46.846825+0000","last_became_active":"2023-11-28T21:51:45.881195+0000","last_became_peered":"2023-11-28T21:51:45.881195+0000","last_unstale":"2023-11-28T21:51:46.846825+0000","last_undegraded":"2023-11-28T21:51:46.846825+0000","last_fullsized":"2023-11-28T21:51:46.846825+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:10:21.497544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.846953+0000","last_change":"2023-11-28T21:51:44.949737+0000","last_active":"2023-11-28T21:51:46.846953+0000","last_peered":"2023-11-28T21:51:46.846953+0000","last_clean":"2023-11-28T21:51:46.846953+0000","last_became_active":"2023-11-28T21:51:44.948114+0000","last_became_peered":"2023-11-28T21:51:44.948114+0000","last_unstale":"2023-11-28T21:51:46.846953+0000","last_undegraded":"2023-11-28T21:51:46.846953+0000","last_fullsized":"2023-11-28T21:51:46.846953+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:49:35.804765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849359+0000","last_change":"2023-11-28T21:51:45.870645+0000","last_active":"2023-11-28T21:51:46.849359+0000","last_peered":"2023-11-28T21:51:46.849359+0000","last_clean":"2023-11-28T21:51:46.849359+0000","last_became_active":"2023-11-28T21:51:45.870358+0000","last_became_peered":"2023-11-28T21:51:45.870358+0000","last_unstale":"2023-11-28T21:51:46.849359+0000","last_undegraded":"2023-11-28T21:51:46.849359+0000","last_fullsized":"2023-11-28T21:51:46.849359+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:00:18.324325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860981+0000","last_change":"2023-11-28T21:51:44.954725+0000","last_active":"2023-11-28T21:51:46.860981+0000","last_peered":"2023-11-28T21:51:46.860981+0000","last_clean":"2023-11-28T21:51:46.860981+0000","last_became_active":"2023-11-28T21:51:44.954428+0000","last_became_peered":"2023-11-28T21:51:44.954428+0000","last_unstale":"2023-11-28T21:51:46.860981+0000","last_undegraded":"2023-11-28T21:51:46.860981+0000","last_fullsized":"2023-11-28T21:51:46.860981+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:16:39.896912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.861331+0000","last_change":"2023-11-28T21:51:45.886350+0000","last_active":"2023-11-28T21:51:46.861331+0000","last_peered":"2023-11-28T21:51:46.861331+0000","last_clean":"2023-11-28T21:51:46.861331+0000","last_became_active":"2023-11-28T21:51:45.886058+0000","last_became_peered":"2023-11-28T21:51:45.886058+0000","last_unstale":"2023-11-28T21:51:46.861331+0000","last_undegraded":"2023-11-28T21:51:46.861331+0000","last_fullsized":"2023-11-28T21:51:46.861331+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:57:01.404286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.844792+0000","last_change":"2023-11-28T21:51:44.948872+0000","last_active":"2023-11-28T21:51:46.844792+0000","last_peered":"2023-11-28T21:51:46.844792+0000","last_clean":"2023-11-28T21:51:46.844792+0000","last_became_active":"2023-11-28T21:51:44.948590+0000","last_became_peered":"2023-11-28T21:51:44.948590+0000","last_unstale":"2023-11-28T21:51:46.844792+0000","last_undegraded":"2023-11-28T21:51:46.844792+0000","last_fullsized":"2023-11-28T21:51:46.844792+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:14:20.212267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.857046+0000","last_change":"2023-11-28T21:51:45.872047+0000","last_active":"2023-11-28T21:51:46.857046+0000","last_peered":"2023-11-28T21:51:46.857046+0000","last_clean":"2023-11-28T21:51:46.857046+0000","last_became_active":"2023-11-28T21:51:45.871687+0000","last_became_peered":"2023-11-28T21:51:45.871687+0000","last_unstale":"2023-11-28T21:51:46.857046+0000","last_undegraded":"2023-11-28T21:51:46.857046+0000","last_fullsized":"2023-11-28T21:51:46.857046+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:40:33.540462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.860093+0000","last_change":"2023-11-28T21:51:44.939343+0000","last_active":"2023-11-28T21:51:46.860093+0000","last_peered":"2023-11-28T21:51:46.860093+0000","last_clean":"2023-11-28T21:51:46.860093+0000","last_became_active":"2023-11-28T21:51:44.939069+0000","last_became_peered":"2023-11-28T21:51:44.939069+0000","last_unstale":"2023-11-28T21:51:46.860093+0000","last_undegraded":"2023-11-28T21:51:46.860093+0000","last_fullsized":"2023-11-28T21:51:46.860093+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:18:17.848127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.849874+0000","last_change":"2023-11-28T21:51:45.884679+0000","last_active":"2023-11-28T21:51:46.849874+0000","last_peered":"2023-11-28T21:51:46.849874+0000","last_clean":"2023-11-28T21:51:46.849874+0000","last_became_active":"2023-11-28T21:51:45.883978+0000","last_became_peered":"2023-11-28T21:51:45.883978+0000","last_unstale":"2023-11-28T21:51:46.849874+0000","last_undegraded":"2023-11-28T21:51:46.849874+0000","last_fullsized":"2023-11-28T21:51:46.849874+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:19.779511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859826+0000","last_change":"2023-11-28T21:51:44.954533+0000","last_active":"2023-11-28T21:51:46.859826+0000","last_peered":"2023-11-28T21:51:46.859826+0000","last_clean":"2023-11-28T21:51:46.859826+0000","last_became_active":"2023-11-28T21:51:44.953590+0000","last_became_peered":"2023-11-28T21:51:44.953590+0000","last_unstale":"2023-11-28T21:51:46.859826+0000","last_undegraded":"2023-11-28T21:51:46.859826+0000","last_fullsized":"2023-11-28T21:51:46.859826+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:24:44.899820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.850421+0000","last_change":"2023-11-28T21:51:45.875713+0000","last_active":"2023-11-28T21:51:46.850421+0000","last_peered":"2023-11-28T21:51:46.850421+0000","last_clean":"2023-11-28T21:51:46.850421+0000","last_became_active":"2023-11-28T21:51:45.875428+0000","last_became_peered":"2023-11-28T21:51:45.875428+0000","last_unstale":"2023-11-28T21:51:46.850421+0000","last_undegraded":"2023-11-28T21:51:46.850421+0000","last_fullsized":"2023-11-28T21:51:46.850421+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:44.905107+0000","last_clean_scrub_stamp":"2023-11-28T21:51:44.905107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:53.969541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"184'6","reported_seq":18,"reported_epoch":185,"state":"active+clean","last_fresh":"2023-11-28T21:51:46.859333+0000","last_change":"2023-11-28T21:51:44.947806+0000","last_active":"2023-11-28T21:51:46.859333+0000","last_peered":"2023-11-28T21:51:46.859333+0000","last_clean":"2023-11-28T21:51:46.859333+0000","last_became_active":"2023-11-28T21:51:44.947398+0000","last_became_peered":"2023-11-28T21:51:44.947398+0000","last_unstale":"2023-11-28T21:51:46.859333+0000","last_undegraded":"2023-11-28T21:51:46.859333+0000","last_fullsized":"2023-11-28T21:51:46.859333+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:51:43.889950+0000","last_clean_scrub_stamp":"2023-11-28T21:51:43.889950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:16:06.132079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"172'4778","reported_seq":5465,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-11-28T21:51:45.865629+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T21:51:45.865629+0000","last_peered":"2023-11-28T21:51:45.865629+0000","last_clean":"2023-11-28T21:51:45.865629+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T21:51:45.865629+0000","last_undegraded":"2023-11-28T21:51:45.865629+0000","last_fullsized":"2023-11-28T21:51:45.865629+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1573,"num_read_kb":3215,"num_write":451,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1573,"num_read_kb":3215,"num_write":451,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4778,"ondisk_log_size":4778,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575776,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56220,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315620,"statfs":{"total":96636764160,"available":96579194880,"internally_reserved":0,"allocated":2895872,"data_stored":1765746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.256,"15min":1.212},"min":{"1min":0.92500000000000004,"5min":0.76800000000000002,"15min":0.68600000000000005},"max":{"1min":1.6799999999999999,"5min":1.865,"15min":1.865},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.238,"15min":1.242},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.79600000000000004},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.954},"last":1.25}]},{"osd":1,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2070000000000001,"15min":1.244},"min":{"1min":0.94699999999999995,"5min":0.79400000000000004,"15min":0.66800000000000004},"max":{"1min":1.6259999999999999,"5min":1.6859999999999999,"15min":2.1880000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2909999999999999,"15min":1.288},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.61899999999999999},"max":{"1min":1.6910000000000001,"5min":1.718,"15min":2.4279999999999999},"last":1.1910000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3420000000000001,"15min":1.325},"min":{"1min":0.97699999999999998,"5min":0.90100000000000002,"15min":0.73499999999999999},"max":{"1min":1.732,"5min":1.754,"15min":1.893},"last":1.29},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.377,"15min":1.369},"min":{"1min":1.034,"5min":0.98799999999999999,"15min":0.95499999999999996},"max":{"1min":1.8280000000000001,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.1479999999999999}]},{"osd":3,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.321,"15min":1.3200000000000001},"min":{"1min":1.008,"5min":0.90400000000000003,"15min":0.76000000000000001},"max":{"1min":1.845,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":1.573},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.496,"15min":1.4730000000000001},"min":{"1min":1.2649999999999999,"5min":0.94699999999999995,"15min":0.872},"max":{"1min":1.7150000000000001,"5min":2.3769999999999998,"15min":2.3769999999999998},"last":1.5229999999999999}]},{"osd":4,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.349,"15min":1.3029999999999999},"min":{"1min":0.98599999999999999,"5min":0.88400000000000001,"15min":0.76500000000000001},"max":{"1min":1.796,"5min":1.8460000000000001,"15min":2.1240000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.321,"15min":1.347},"min":{"1min":1.002,"5min":0.88100000000000001,"15min":0.81299999999999994},"max":{"1min":1.748,"5min":1.895,"15min":1.9470000000000001},"last":1.635}]},{"osd":5,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4750000000000001,"15min":1.456},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.79200000000000004},"max":{"1min":1.722,"5min":1.976,"15min":2.008},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.5580000000000001,"15min":1.5089999999999999},"min":{"1min":1.3220000000000001,"5min":1.0389999999999999,"15min":0.89700000000000002},"max":{"1min":1.9670000000000001,"5min":2.246,"15min":2.246},"last":1.5069999999999999}]},{"osd":6,"last update":"Tue Nov 28 21:51:14 2023","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.6160000000000001,"15min":1.597},"min":{"1min":1.3120000000000001,"5min":1.2250000000000001,"15min":0.94399999999999995},"max":{"1min":1.998,"5min":2.1640000000000001,"15min":2.3029999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.506,"5min":1.4870000000000001,"15min":1.508},"min":{"1min":1.1579999999999999,"5min":1.0009999999999999,"15min":0.94399999999999995},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":2.2170000000000001},"last":1.7010000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575776,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37600,"kb_used_data":2260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334240,"statfs":{"total":96636764160,"available":96598261760,"internally_reserved":0,"allocated":2314240,"data_stored":1182074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.129,"15min":1.1339999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.597,"5min":1.6180000000000001,"15min":1.8740000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.149,"5min":1.165,"15min":1.159},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5760000000000001,"5min":1.702,"15min":1.702},"last":0.84499999999999997}]},{"osd":1,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.224,"15min":1.2609999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7,"5min":1.7989999999999999,"15min":1.9550000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.427,"15min":1.4419999999999999},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.796,"5min":1.8169999999999999,"15min":2.2109999999999999},"last":1.1000000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.294,"15min":1.2789999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":2.012},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4059999999999999,"15min":1.4199999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":2.4420000000000002},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.3100000000000001,"15min":1.331},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.581,"5min":1.9419999999999999,"15min":2.0779999999999998},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3340000000000001,"15min":1.3240000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.7889999999999999,"5min":1.8600000000000001,"15min":2.1499999999999999},"last":0.97999999999999998}]},{"osd":4,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2490000000000001,"15min":1.256},"min":{"1min":0.90700000000000003,"5min":0.80200000000000005,"15min":0.77900000000000003},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":2.726},"last":1.506},{"interface":"front","average":{"1min":1.337,"5min":1.3420000000000001,"15min":1.349},"min":{"1min":0.95599999999999996,"5min":0.82499999999999996,"15min":0.80100000000000005},"max":{"1min":1.764,"5min":2.004,"15min":3.069},"last":1.1479999999999999}]},{"osd":5,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.4019999999999999,"15min":1.401},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.83699999999999997},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.319},"last":1.173},{"interface":"front","average":{"1min":1.53,"5min":1.5549999999999999,"15min":1.5680000000000001},"min":{"1min":1.1339999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.994,"5min":1.994,"15min":2.5680000000000001},"last":1.5800000000000001}]},{"osd":7,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.5109999999999999,"15min":1.5329999999999999},"min":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":0.99099999999999999},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.96},"last":1.343},{"interface":"front","average":{"1min":1.45,"5min":1.504,"15min":1.516},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":1.99,"5min":1.99,"15min":2.3439999999999999},"last":1.663}]}]},{"osd":5,"up_from":13,"seq":55834575776,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59812,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312028,"statfs":{"total":96636764160,"available":96575516672,"internally_reserved":0,"allocated":2310144,"data_stored":1182004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.198,"15min":1.2},"min":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.65400000000000003},"max":{"1min":1.411,"5min":1.6919999999999999,"15min":1.8879999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.177,"15min":1.1990000000000001},"min":{"1min":0.879,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.542,"5min":1.653,"15min":1.8620000000000001},"last":0.92600000000000005}]},{"osd":1,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2050000000000001,"15min":1.218},"min":{"1min":0.89400000000000002,"5min":0.81699999999999995,"15min":0.752},"max":{"1min":1.5580000000000001,"5min":1.724,"15min":1.843},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.242,"15min":1.2769999999999999},"min":{"1min":0.79500000000000004,"5min":0.76400000000000001,"15min":0.74399999999999999},"max":{"1min":1.5569999999999999,"5min":1.8520000000000001,"15min":2.109},"last":1.5309999999999999}]},{"osd":2,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.3169999999999999,"15min":1.325},"min":{"1min":0.91000000000000003,"5min":0.72999999999999998,"15min":0.70199999999999996},"max":{"1min":1.7030000000000001,"5min":1.7450000000000001,"15min":1.9379999999999999},"last":1.657},{"interface":"front","average":{"1min":1.393,"5min":1.3400000000000001,"15min":1.337},"min":{"1min":1.113,"5min":0.876,"15min":0.83699999999999997},"max":{"1min":1.5629999999999999,"5min":1.766,"15min":1.9990000000000001},"last":1.373}]},{"osd":3,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4159999999999999,"15min":1.389},"min":{"1min":1.0940000000000001,"5min":0.92400000000000004,"15min":0.754},"max":{"1min":1.627,"5min":1.954,"15min":2.1030000000000002},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.3320000000000001,"15min":1.329},"min":{"1min":1.101,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.643,"5min":1.71,"15min":2.0630000000000002},"last":1.206}]},{"osd":4,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.3779999999999999,"15min":1.349},"min":{"1min":1.111,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.776,"5min":1.8169999999999999,"15min":1.9399999999999999},"last":1.319},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4890000000000001,"15min":1.458},"min":{"1min":0.94499999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.833,"5min":1.9059999999999999,"15min":2.2040000000000002},"last":1.728}]},{"osd":6,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.3720000000000001,"15min":1.381},"min":{"1min":1.032,"5min":0.752,"15min":0.752},"max":{"1min":1.7050000000000001,"5min":2.0110000000000001,"15min":2.0699999999999998},"last":1.593},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.524,"15min":1.526},"min":{"1min":1.3200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.8520000000000001,"5min":1.905,"15min":2.1560000000000001},"last":1.4419999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:50:56 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.431,"15min":1.4530000000000001},"min":{"1min":1.0960000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.7330000000000001,"5min":1.9670000000000001,"15min":3.0110000000000001},"last":1.78},{"interface":"front","average":{"1min":1.587,"5min":1.5269999999999999,"15min":1.5289999999999999},"min":{"1min":1.379,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.7430000000000001,"5min":1.9790000000000001,"15min":2.2010000000000001},"last":1.8080000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575777,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29272,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342568,"statfs":{"total":96636764160,"available":96606789632,"internally_reserved":0,"allocated":2310144,"data_stored":1176968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.109,"15min":1.1439999999999999},"min":{"1min":0.78000000000000003,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.5169999999999999,"5min":1.9379999999999999,"15min":1.9550000000000001},"last":1.262},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.1659999999999999,"15min":1.165},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.76300000000000001},"max":{"1min":1.679,"5min":1.679,"15min":1.9570000000000001},"last":0.83399999999999996}]},{"osd":1,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.363,"15min":1.351},"min":{"1min":1.0509999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.6719999999999999,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.389,"15min":1.393},"min":{"1min":1.0920000000000001,"5min":0.82699999999999996,"15min":0.79900000000000004},"max":{"1min":1.752,"5min":1.8320000000000001,"15min":1.9119999999999999},"last":1.0680000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2450000000000001,"15min":1.27},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.86399999999999999},"max":{"1min":1.4910000000000001,"5min":1.788,"15min":1.9570000000000001},"last":1.198},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.98499999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.6990000000000001,"5min":1.7809999999999999,"15min":1.905},"last":1.508}]},{"osd":3,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.3009999999999999,"15min":1.2969999999999999},"min":{"1min":0.94699999999999995,"5min":0.876,"15min":0.73699999999999999},"max":{"1min":1.671,"5min":1.7090000000000001,"15min":1.7769999999999999},"last":1.742},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.415,"15min":1.4139999999999999},"min":{"1min":1.157,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.923},"last":1.5960000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.306,"15min":1.2889999999999999},"min":{"1min":0.98399999999999999,"5min":0.749,"15min":0.749},"max":{"1min":1.7210000000000001,"5min":1.873,"15min":1.891},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.343,"15min":1.3220000000000001},"min":{"1min":0.96099999999999997,"5min":0.81499999999999995,"15min":0.79100000000000004},"max":{"1min":1.7849999999999999,"5min":1.8089999999999999,"15min":1.9339999999999999},"last":1.954}]},{"osd":6,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.464,"15min":1.4359999999999999},"min":{"1min":0.92400000000000004,"5min":0.86399999999999999,"15min":0.85399999999999998},"max":{"1min":1.8029999999999999,"5min":1.974,"15min":2.1099999999999999},"last":1.9990000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.587,"15min":1.579},"min":{"1min":1.2709999999999999,"5min":1.246,"15min":0.86499999999999999},"max":{"1min":1.8680000000000001,"5min":2.0049999999999999,"15min":2.23},"last":1.8120000000000001}]},{"osd":7,"last update":"Tue Nov 28 21:51:07 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.4850000000000001,"15min":1.4750000000000001},"min":{"1min":1.0820000000000001,"5min":1.0069999999999999,"15min":0.94399999999999995},"max":{"1min":1.8360000000000001,"5min":1.921,"15min":2.085},"last":2.056},{"interface":"front","average":{"1min":1.625,"5min":1.589,"15min":1.5660000000000001},"min":{"1min":1.1819999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":1.925,"5min":2.1640000000000001,"15min":2.2210000000000001},"last":1.8759999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575777,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51484,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320356,"statfs":{"total":96636764160,"available":96584044544,"internally_reserved":0,"allocated":2306048,"data_stored":1175972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.002,"15min":0.98099999999999998},"min":{"1min":0.68300000000000005,"5min":0.67400000000000004,"15min":0.60399999999999998},"max":{"1min":1.492,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.105,"15min":1.115},"min":{"1min":0.84299999999999997,"5min":0.67900000000000005,"15min":0.66500000000000004},"max":{"1min":1.369,"5min":1.895,"15min":1.9399999999999999},"last":1.2210000000000001}]},{"osd":1,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.036,"15min":1.0229999999999999},"min":{"1min":0.75700000000000001,"5min":0.753,"15min":0.72999999999999998},"max":{"1min":1.474,"5min":1.7390000000000001,"15min":1.8069999999999999},"last":1.079},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.284,"15min":1.262},"min":{"1min":0.89200000000000002,"5min":0.82899999999999996,"15min":0.68999999999999995},"max":{"1min":1.694,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.2929999999999999}]},{"osd":2,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3069999999999999,"15min":1.3129999999999999},"min":{"1min":0.88800000000000001,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.663,"5min":1.839,"15min":1.8560000000000001},"last":1.262},{"interface":"front","average":{"1min":1.48,"5min":1.4770000000000001,"15min":1.4319999999999999},"min":{"1min":1.1439999999999999,"5min":0.92400000000000004,"15min":0.751},"max":{"1min":1.778,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.129}]},{"osd":4,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.355,"15min":1.387},"min":{"1min":0.996,"5min":0.94799999999999995,"15min":0.85299999999999998},"max":{"1min":1.673,"5min":2.2000000000000002,"15min":2.246},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.456,"15min":1.46},"min":{"1min":1.111,"5min":1.0589999999999999,"15min":1.0509999999999999},"max":{"1min":1.835,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.323}]},{"osd":5,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.607,"15min":1.577},"min":{"1min":1.3440000000000001,"5min":1.1259999999999999,"15min":1.0229999999999999},"max":{"1min":2.0960000000000001,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.51,"15min":1.4890000000000001},"min":{"1min":1.2529999999999999,"5min":1.109,"15min":0.94699999999999995},"max":{"1min":1.7310000000000001,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.518}]},{"osd":6,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.569,"15min":1.5620000000000001},"min":{"1min":1.117,"5min":1.117,"15min":1.111},"max":{"1min":1.9810000000000001,"5min":2.149,"15min":2.153},"last":1.371},{"interface":"front","average":{"1min":1.7490000000000001,"5min":1.706,"15min":1.677},"min":{"1min":1.431,"5min":1.27,"15min":1.099},"max":{"1min":2.1619999999999999,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":1.919}]},{"osd":7,"last update":"Tue Nov 28 21:51:10 2023","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.6319999999999999,"15min":1.617},"min":{"1min":1.232,"5min":1.214,"15min":0.995},"max":{"1min":2.0499999999999998,"5min":2.5,"15min":2.5},"last":1.996},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.72,"15min":1.716},"min":{"1min":1.2889999999999999,"5min":1.23,"15min":1.23},"max":{"1min":2.0859999999999999,"5min":2.339,"15min":2.4489999999999998},"last":1.585}]}]},{"osd":2,"up_from":13,"seq":55834575777,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47336,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324504,"statfs":{"total":96636764160,"available":96588292096,"internally_reserved":0,"allocated":2318336,"data_stored":1182116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":0.97599999999999998,"15min":0.96899999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.369},{"interface":"front","average":{"1min":1.006,"5min":1.141,"15min":1.1439999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.6559999999999999,"5min":1.764,"15min":1.8160000000000001},"last":0.79600000000000004}]},{"osd":1,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.98399999999999999,"15min":1.0049999999999999},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0379999999999998},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.222,"15min":1.2430000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.8380000000000001},"last":0.89200000000000002}]},{"osd":3,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.1990000000000001,"15min":1.222},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.7679999999999998,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.133},{"interface":"front","average":{"1min":1.292,"5min":1.3109999999999999,"15min":1.329},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.9199999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.5189999999999999}]},{"osd":4,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.456,"15min":1.4670000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":3.202,"5min":3.202,"15min":3.202},"last":1.9119999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.498,"15min":1.502},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.4530000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.508,"15min":1.522},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":0.99299999999999999},"max":{"1min":3.1269999999999998,"5min":3.1269999999999998,"15min":3.1269999999999998},"last":1.8109999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.5800000000000001,"15min":1.542},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.3149999999999999,"5min":3.3149999999999999,"15min":3.3149999999999999},"last":1.831}]},{"osd":6,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.583,"15min":1.591},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.661,"5min":1.633,"15min":1.6279999999999999},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.706}]},{"osd":7,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.621,"15min":1.633},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":4.2990000000000004,"5min":4.2990000000000004,"15min":4.2990000000000004},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.679,"15min":1.7},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.8940000000000001,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":1.649}]}]},{"osd":1,"up_from":13,"seq":55834575777,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45912,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325928,"statfs":{"total":96636764160,"available":96589750272,"internally_reserved":0,"allocated":2301952,"data_stored":1175426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.91700000000000004,"15min":0.91800000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.127,"15min":1.1240000000000001},"min":{"1min":0.76000000000000001,"5min":0.753,"15min":0.69799999999999995},"max":{"1min":2.9489999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.88200000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1819999999999999,"15min":1.1499999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77400000000000002},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.103},{"interface":"front","average":{"1min":1.254,"5min":1.258,"15min":1.2649999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.921,"5min":1.921,"15min":2.0750000000000002},"last":1.4530000000000001}]},{"osd":3,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4039999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.889,"5min":1.9790000000000001,"15min":1.9810000000000001},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.1910000000000001,"15min":1.1839999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":0.95299999999999996}]},{"osd":4,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.4279999999999999,"15min":1.415},"min":{"1min":1.1299999999999999,"5min":1.0209999999999999,"15min":0.90900000000000003},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.23},{"interface":"front","average":{"1min":1.534,"5min":1.4570000000000001,"15min":1.4450000000000001},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.84199999999999997},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.4239999999999999}]},{"osd":5,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.482,"15min":1.48},"min":{"1min":1.008,"5min":1.008,"15min":0.97399999999999998},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.694},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.569,"15min":1.5700000000000001},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.0110000000000001,"5min":2.0979999999999999,"15min":2.25},"last":1.8700000000000001}]},{"osd":6,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.516,"15min":1.5029999999999999},"min":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.0429999999999999},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.302},{"interface":"front","average":{"1min":1.712,"5min":1.605,"15min":1.591},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":3.1520000000000001,"5min":3.1520000000000001,"15min":3.1520000000000001},"last":1.5820000000000001}]},{"osd":7,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.6439999999999999,"15min":1.6559999999999999},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.2589999999999999,"5min":2.2629999999999999,"15min":2.3439999999999999},"last":1.9990000000000001},{"interface":"front","average":{"1min":1.734,"5min":1.6299999999999999,"15min":1.607},"min":{"1min":1.258,"5min":1.151,"15min":1.1080000000000001},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.2589999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575777,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29868,"kb_used_data":2848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341972,"statfs":{"total":96636764160,"available":96606179328,"internally_reserved":0,"allocated":2916352,"data_stored":1768042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.93899999999999995,"15min":0.93999999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.72,"5min":1.72,"15min":1.724},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1719999999999999,"15min":1.1639999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.7879999999999998,"5min":2.7879999999999998,"15min":2.7879999999999998},"last":0.75900000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.1339999999999999,"15min":1.1359999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.5449999999999999,"5min":1.6339999999999999,"15min":1.8959999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.246,"15min":1.2569999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.7929999999999999,"5min":1.8819999999999999,"15min":2.0409999999999999},"last":0.67200000000000004}]},{"osd":3,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.1240000000000001,"15min":1.1319999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.4750000000000001,"5min":1.7110000000000001,"15min":2.0649999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.401,"15min":1.3640000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.1899999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.88100000000000001}]},{"osd":4,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4570000000000001,"15min":1.46},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.302,"5min":1.446,"15min":1.4139999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.6459999999999999,"5min":1.869,"15min":2.0600000000000001},"last":1.089}]},{"osd":5,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.45,"15min":1.4550000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.77,"5min":2.153,"15min":2.153},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5229999999999999,"15min":1.5289999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.159}]},{"osd":6,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.613,"15min":1.6060000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5940000000000001,"15min":1.599},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":1.0149999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:51:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.6819999999999999,"15min":1.667},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.589,"15min":1.5800000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.371,"5min":2.371,"15min":2.371},"last":1.181}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T21:51:51.027 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T21:51:51.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T21:51:51.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.307+0000 7f4bb183d640 1 Processor -- start 2023-11-28T21:51:51.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.307+0000 7f4bb183d640 1 -- start start 2023-11-28T21:51:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.309+0000 7f4bb183d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4bac15e840 0x7f4bac06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.309+0000 7f4bb183d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4bac06ac10 0x7f4bac0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.309+0000 7f4bb183d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bac15ecd0 con 0x7f4bac15f720 2023-11-28T21:51:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.309+0000 7f4bb183d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bac071ce0 con 0x7f4bac06ac10 2023-11-28T21:51:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.309+0000 7f4bb183d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bac15f290 con 0x7f4bac15e840 2023-11-28T21:51:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.309+0000 7f4baa7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4bac15f720 0x7f4bac06a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60932/0 (socket says 172.21.15.45:60932) 2023-11-28T21:51:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.309+0000 7f4baa7fc640 1 -- 172.21.15.45:0/55320049 learned_addr learned my addr 172.21.15.45:0/55320049 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.309+0000 7f4bab7fe640 1 --2- 172.21.15.45:0/55320049 >> v2:172.21.15.45:3300/0 conn(0x7f4bac06ac10 0x7f4bac0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.310+0000 7f4baaffd640 1 --2- 172.21.15.45:0/55320049 >> v2:172.21.15.190:3300/0 conn(0x7f4bac15e840 0x7f4bac06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.310+0000 7f4ba9ffb640 1 -- 172.21.15.45:0/55320049 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677364233 0 0) 0x7f4bac15ecd0 con 0x7f4bac15f720 2023-11-28T21:51:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.310+0000 7f4ba9ffb640 1 -- 172.21.15.45:0/55320049 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b94003520 con 0x7f4bac15f720 2023-11-28T21:51:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.310+0000 7f4bab7fe640 1 -- 172.21.15.45:0/55320049 >> v2:172.21.15.190:3300/0 conn(0x7f4bac15e840 msgr2=0x7f4bac06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.310+0000 7f4bab7fe640 1 --2- 172.21.15.45:0/55320049 >> v2:172.21.15.190:3300/0 conn(0x7f4bac15e840 0x7f4bac06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.310+0000 7f4bab7fe640 1 -- 172.21.15.45:0/55320049 >> v1:172.21.15.45:6789/0 conn(0x7f4bac15f720 legacy=0x7f4bac06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.310+0000 7f4bab7fe640 1 -- 172.21.15.45:0/55320049 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac15fb60 con 0x7f4bac06ac10 2023-11-28T21:51:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.312+0000 7f4bab7fe640 1 --2- 172.21.15.45:0/55320049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac06ac10 0x7f4bac0717a0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0cb5c0 tx=0x7f4ba402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fdfce94bd909b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.312+0000 7f4ba9ffb640 1 -- 172.21.15.45:0/55320049 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba402cc80 con 0x7f4bac06ac10 2023-11-28T21:51:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.312+0000 7f4ba9ffb640 1 -- 172.21.15.45:0/55320049 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba402cde0 con 0x7f4bac06ac10 2023-11-28T21:51:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.312+0000 7f4ba9ffb640 1 -- 172.21.15.45:0/55320049 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4034660 con 0x7f4bac06ac10 2023-11-28T21:51:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.313+0000 7f4bb183d640 1 -- 172.21.15.45:0/55320049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac06ac10 msgr2=0x7f4bac0717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.313+0000 7f4bb183d640 1 --2- 172.21.15.45:0/55320049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac06ac10 0x7f4bac0717a0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0cb5c0 tx=0x7f4ba402bed0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.313+0000 7f4bb183d640 1 -- 172.21.15.45:0/55320049 shutdown_connections 2023-11-28T21:51:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.313+0000 7f4bb183d640 1 --2- 172.21.15.45:0/55320049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac06ac10 0x7f4bac0717a0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.313+0000 7f4bb183d640 1 --2- 172.21.15.45:0/55320049 >> v2:172.21.15.190:3300/0 conn(0x7f4bac15e840 0x7f4bac06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.313+0000 7f4bb183d640 1 -- 172.21.15.45:0/55320049 >> 172.21.15.45:0/55320049 conn(0x7f4bac0ced60 msgr2=0x7f4bac0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.313+0000 7f4bb183d640 1 -- 172.21.15.45:0/55320049 shutdown_connections 2023-11-28T21:51:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.313+0000 7f4bb183d640 1 -- 172.21.15.45:0/55320049 wait complete. 2023-11-28T21:51:51.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.315+0000 7f4bb183d640 1 Processor -- start 2023-11-28T21:51:51.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.315+0000 7f4bb183d640 1 -- start start 2023-11-28T21:51:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bb183d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bac06ac10 0x7f4bac15d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bb183d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bac15e840 0x7f4bac0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bb183d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac15f720 0x7f4bac0b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bb183d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac0c7890 con 0x7f4bac15f720 2023-11-28T21:51:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bb183d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac0c7b30 con 0x7f4bac15e840 2023-11-28T21:51:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bb183d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac0c7e10 con 0x7f4bac06ac10 2023-11-28T21:51:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bab7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac15f720 0x7f4bac0b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bab7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac15f720 0x7f4bac0b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34616/0 (socket says 172.21.15.45:34616) 2023-11-28T21:51:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bab7fe640 1 -- 172.21.15.45:0/2012792147 learned_addr learned my addr 172.21.15.45:0/2012792147 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4baa7fc640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bac15e840 0x7f4bac0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4baaffd640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bac06ac10 0x7f4bac15d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bab7fe640 1 -- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bac06ac10 msgr2=0x7f4bac15d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bab7fe640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bac06ac10 0x7f4bac15d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bab7fe640 1 -- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bac15e840 msgr2=0x7f4bac0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.316+0000 7f4bab7fe640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bac15e840 0x7f4bac0b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.317+0000 7f4bab7fe640 1 -- 172.21.15.45:0/2012792147 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba4009a70 con 0x7f4bac15f720 2023-11-28T21:51:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.317+0000 7f4bab7fe640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac15f720 0x7f4bac0b3d60 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009e80 tx=0x7f4ba4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.317+0000 7f4b93fff640 1 -- 172.21.15.45:0/2012792147 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba403e610 con 0x7f4bac15f720 2023-11-28T21:51:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.317+0000 7f4b93fff640 1 -- 172.21.15.45:0/2012792147 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba403e770 con 0x7f4bac15f720 2023-11-28T21:51:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.317+0000 7f4b93fff640 1 -- 172.21.15.45:0/2012792147 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba403eb60 con 0x7f4bac15f720 2023-11-28T21:51:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.317+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bac0c80f0 con 0x7f4bac15f720 2023-11-28T21:51:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.317+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bac0c86b0 con 0x7f4bac15f720 2023-11-28T21:51:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.318+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bac0733d0 con 0x7f4bac15f720 2023-11-28T21:51:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.319+0000 7f4b93fff640 1 -- 172.21.15.45:0/2012792147 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4ba403b050 con 0x7f4bac15f720 2023-11-28T21:51:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.320+0000 7f4b93fff640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4b9806af60 0x7f4b9806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.320+0000 7f4b93fff640 1 -- 172.21.15.45:0/2012792147 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4ba40ac0d0 con 0x7f4bac15f720 2023-11-28T21:51:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.321+0000 7f4baaffd640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4b9806af60 0x7f4b9806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.322+0000 7f4baaffd640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4b9806af60 0x7f4b9806d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c009a10 tx=0x7f4b9c008040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.325+0000 7f4b93fff640 1 -- 172.21.15.45:0/2012792147 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ba4076fb0 con 0x7f4bac15f720 2023-11-28T21:51:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.595+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4bac0ce2b0 con 0x7f4bac15f720 2023-11-28T21:51:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.600+0000 7f4b93fff640 1 -- 172.21.15.45:0/2012792147 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4ba404e0b0 con 0x7f4bac15f720 2023-11-28T21:51:51.602 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T21:51:51.602 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T21:51:51.602 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T21:51:51.602 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T21:51:51.602 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T21:51:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4b9806af60 msgr2=0x7f4b9806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4b9806af60 0x7f4b9806d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c009a10 tx=0x7f4b9c008040 comp rx=0 tx=0).stop 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac15f720 msgr2=0x7f4bac0b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac15f720 0x7f4bac0b3d60 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009e80 tx=0x7f4ba4004a40 comp rx=0 tx=0).stop 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 shutdown_connections 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4b9806af60 0x7f4b9806d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac15f720 0x7f4bac0b3d60 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bac15e840 0x7f4bac0b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 --2- 172.21.15.45:0/2012792147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bac06ac10 0x7f4bac15d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 >> 172.21.15.45:0/2012792147 conn(0x7f4bac0ced60 msgr2=0x7f4bac0700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 shutdown_connections 2023-11-28T21:51:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.603+0000 7f4bb183d640 1 -- 172.21.15.45:0/2012792147 wait complete. 2023-11-28T21:51:51.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:51:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.846+0000 7f938f59e640 1 Processor -- start 2023-11-28T21:51:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.847+0000 7f938f59e640 1 -- start start 2023-11-28T21:51:51.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.848+0000 7f938f59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9390072200 0x7f93900b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.848+0000 7f938f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390146610 0x7f93900b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.848+0000 7f938f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9390146b60 0x7f9390160e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.849+0000 7f938f59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93900725e0 con 0x7f9390072200 2023-11-28T21:51:51.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.849+0000 7f938f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93900b4310 con 0x7f9390146b60 2023-11-28T21:51:51.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.849+0000 7f938f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939006d8d0 con 0x7f9390146610 2023-11-28T21:51:51.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.849+0000 7f938e59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9390072200 0x7f93900b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.849+0000 7f938e59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9390072200 0x7f93900b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34626/0 (socket says 172.21.15.45:34626) 2023-11-28T21:51:51.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.849+0000 7f938e59c640 1 -- 172.21.15.45:0/2712140912 learned_addr learned my addr 172.21.15.45:0/2712140912 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:51.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.849+0000 7f9387fff640 1 --2- 172.21.15.45:0/2712140912 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390146610 0x7f93900b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.849+0000 7f938ed9d640 1 --2- 172.21.15.45:0/2712140912 >> v2:172.21.15.190:3300/0 conn(0x7f9390146b60 0x7f9390160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.850+0000 7f938e59c640 1 -- 172.21.15.45:0/2712140912 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390146610 msgr2=0x7f93900b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:51.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.850+0000 7f938e59c640 1 --2- 172.21.15.45:0/2712140912 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390146610 0x7f93900b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.850+0000 7f938e59c640 1 -- 172.21.15.45:0/2712140912 >> v2:172.21.15.190:3300/0 conn(0x7f9390146b60 msgr2=0x7f9390160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:51.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.850+0000 7f938e59c640 1 --2- 172.21.15.45:0/2712140912 >> v2:172.21.15.190:3300/0 conn(0x7f9390146b60 0x7f9390160e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.850+0000 7f938e59c640 1 -- 172.21.15.45:0/2712140912 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939006ca20 con 0x7f9390072200 2023-11-28T21:51:51.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.850+0000 7f9387fff640 1 --2- 172.21.15.45:0/2712140912 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390146610 0x7f93900b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:51.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.851+0000 7f938e59c640 1 --2- 172.21.15.45:0/2712140912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390072200 0x7f93900b1d90 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f93900b3710 tx=0x7f938002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b9493364069466 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:51.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.851+0000 7f938dd9b640 1 -- 172.21.15.45:0/2712140912 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9380031870 con 0x7f9390072200 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938dd9b640 1 -- 172.21.15.45:0/2712140912 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93800319d0 con 0x7f9390072200 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938dd9b640 1 -- 172.21.15.45:0/2712140912 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9380031d60 con 0x7f9390072200 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938f59e640 1 -- 172.21.15.45:0/2712140912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390072200 msgr2=0x7f93900b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938f59e640 1 --2- 172.21.15.45:0/2712140912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390072200 0x7f93900b1d90 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f93900b3710 tx=0x7f938002b910 comp rx=0 tx=0).stop 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938f59e640 1 -- 172.21.15.45:0/2712140912 shutdown_connections 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938f59e640 1 --2- 172.21.15.45:0/2712140912 >> v2:172.21.15.190:3300/0 conn(0x7f9390146b60 0x7f9390160e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938f59e640 1 --2- 172.21.15.45:0/2712140912 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390146610 0x7f93900b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938f59e640 1 --2- 172.21.15.45:0/2712140912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390072200 0x7f93900b1d90 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938f59e640 1 -- 172.21.15.45:0/2712140912 >> 172.21.15.45:0/2712140912 conn(0x7f939006ab30 msgr2=0x7f93900b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:51.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938f59e640 1 -- 172.21.15.45:0/2712140912 shutdown_connections 2023-11-28T21:51:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.852+0000 7f938f59e640 1 -- 172.21.15.45:0/2712140912 wait complete. 2023-11-28T21:51:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.854+0000 7f938f59e640 1 Processor -- start 2023-11-28T21:51:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.854+0000 7f938f59e640 1 -- start start 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.855+0000 7f938f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390072200 0x7f93900c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.855+0000 7f938f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9390146610 0x7f939009b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.855+0000 7f938f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390146b60 0x7f93901395f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939009cee0 con 0x7f9390146b60 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939009be40 con 0x7f9390146610 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939009c120 con 0x7f9390072200 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390146b60 0x7f93901395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390146b60 0x7f93901395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34638/0 (socket says 172.21.15.45:34638) 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938ed9d640 1 -- 172.21.15.45:0/2922411725 learned_addr learned my addr 172.21.15.45:0/2922411725 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938e59c640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390072200 0x7f93900c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f9387fff640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9390146610 0x7f939009b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938ed9d640 1 -- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390072200 msgr2=0x7f93900c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938ed9d640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390072200 0x7f93900c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938ed9d640 1 -- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9390146610 msgr2=0x7f939009b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938ed9d640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9390146610 0x7f939009b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.856+0000 7f938ed9d640 1 -- 172.21.15.45:0/2922411725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93800094d0 con 0x7f9390146b60 2023-11-28T21:51:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.857+0000 7f938ed9d640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390146b60 0x7f93901395f0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9388007e60 tx=0x7f938800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.857+0000 7f9386ffd640 1 -- 172.21.15.45:0/2922411725 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9388014420 con 0x7f9390146b60 2023-11-28T21:51:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.857+0000 7f9386ffd640 1 -- 172.21.15.45:0/2922411725 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93880145a0 con 0x7f9390146b60 2023-11-28T21:51:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.857+0000 7f9386ffd640 1 -- 172.21.15.45:0/2922411725 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9388014950 con 0x7f9390146b60 2023-11-28T21:51:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.857+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939009c400 con 0x7f9390146b60 2023-11-28T21:51:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.857+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939009c9c0 con 0x7f9390146b60 2023-11-28T21:51:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.858+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93901f4280 con 0x7f9390146b60 2023-11-28T21:51:51.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.859+0000 7f9386ffd640 1 -- 172.21.15.45:0/2922411725 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9388014ab0 con 0x7f9390146b60 2023-11-28T21:51:51.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.860+0000 7f9386ffd640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f937006af90 0x7f937006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:51.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.860+0000 7f9386ffd640 1 -- 172.21.15.45:0/2922411725 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f938808a1f0 con 0x7f9390146b60 2023-11-28T21:51:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.861+0000 7f938e59c640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f937006af90 0x7f937006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:51.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.862+0000 7f938e59c640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f937006af90 0x7f937006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9380036000 tx=0x7f938003c040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:51.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:51.865+0000 7f9386ffd640 1 -- 172.21.15.45:0/2922411725 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93901f4280 con 0x7f9390146b60 2023-11-28T21:51:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.105+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93901f4280 con 0x7f9390146b60 2023-11-28T21:51:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.105+0000 7f9386ffd640 1 -- 172.21.15.45:0/2922411725 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4908 (secure 0 0 0) 0x7f93901f4280 con 0x7f9390146b60 2023-11-28T21:51:52.107 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:51:52.107 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7130,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/500262407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":500262407},{"type":"v1","addr":"172.21.15.190:6835","nonce":500262407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/195287870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":195287870},{"type":"v1","addr":"172.21.15.190:6837","nonce":195287870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":9657,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/440031651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":440031651},{"type":"v1","addr":"172.21.15.45:6833","nonce":440031651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:51:45.400395+0000","modified":"2023-11-28T21:51:48.467537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9660":{"gid":9660,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/99455191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":99455191},{"type":"v1","addr":"172.21.15.45:6835","nonce":99455191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-28T21:51:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 84 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f937006af90 msgr2=0x7f937006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f937006af90 0x7f937006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9380036000 tx=0x7f938003c040 comp rx=0 tx=0).stop 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390146b60 msgr2=0x7f93901395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390146b60 0x7f93901395f0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9388007e60 tx=0x7f938800e4b0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 shutdown_connections 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f937006af90 0x7f937006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9390146b60 0x7f93901395f0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9390146610 0x7f939009b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 --2- 172.21.15.45:0/2922411725 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9390072200 0x7f93900c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 >> 172.21.15.45:0/2922411725 conn(0x7f939006ab30 msgr2=0x7f93900aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:52.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.108+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 shutdown_connections 2023-11-28T21:51:52.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.109+0000 7f938f59e640 1 -- 172.21.15.45:0/2922411725 wait complete. 2023-11-28T21:51:52.532 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T21:51:45.400395+0000', 'modified': '2023-11-28T21:51:48.467537+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9660': {'gid': 9660, 'name': 'a', 'rank': 0, 'incarnation': 80, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/99455191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6834', 'nonce': 99455191}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 99455191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T21:51:52.532 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T21:51:52.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-28T21:51:52.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.766+0000 7f4f4cdf1640 1 Processor -- start 2023-11-28T21:51:52.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.766+0000 7f4f4cdf1640 1 -- start start 2023-11-28T21:51:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.768+0000 7f4f4cdf1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f4815d320 0x7f4f480b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.768+0000 7f4f4cdf1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4f48066900 0x7f4f480adb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.768+0000 7f4f4cdf1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f480af850 con 0x7f4f480b4a00 2023-11-28T21:51:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.768+0000 7f4f4cdf1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f480b51c0 con 0x7f4f48066900 2023-11-28T21:51:52.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.768+0000 7f4f4cdf1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f480b4ee0 con 0x7f4f4815d320 2023-11-28T21:51:52.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.768+0000 7f4f45d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4f480b4a00 0x7f4f480661f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60936/0 (socket says 172.21.15.45:60936) 2023-11-28T21:51:52.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.768+0000 7f4f45d9b640 1 -- 172.21.15.45:0/1199711913 learned_addr learned my addr 172.21.15.45:0/1199711913 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f46d9d640 1 --2- 172.21.15.45:0/1199711913 >> v2:172.21.15.190:3300/0 conn(0x7f4f48066900 0x7f4f480adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4659c640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f4815d320 0x7f4f480b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116751297 0 0) 0x7f4f480af850 con 0x7f4f480b4a00 2023-11-28T21:51:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f34003520 con 0x7f4f480b4a00 2023-11-28T21:51:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3610523530 0 0) 0x7f4f34003520 con 0x7f4f480b4a00 2023-11-28T21:51:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f4815d320 msgr2=0x7f4f480b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f4815d320 0x7f4f480b2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 >> v2:172.21.15.190:3300/0 conn(0x7f4f48066900 msgr2=0x7f4f480adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 --2- 172.21.15.45:0/1199711913 >> v2:172.21.15.190:3300/0 conn(0x7f4f48066900 0x7f4f480adb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4806b8b0 con 0x7f4f480b4a00 2023-11-28T21:51:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4f30001670 con 0x7f4f480b4a00 2023-11-28T21:51:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 >> v1:172.21.15.45:6789/0 conn(0x7f4f480b4a00 legacy=0x7f4f480661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f34003970 0x7f4f34003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f340047c0 0x7f4f34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f4815d320 0x7f4f34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f480b51c0 con 0x7f4f34003970 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f480b4ee0 con 0x7f4f340047c0 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.769+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3400b980 con 0x7f4f4815d320 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.770+0000 7f4f4659c640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f34003970 0x7f4f34003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.770+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f340047c0 0x7f4f34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.770+0000 7f4f4659c640 1 -- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f4815d320 msgr2=0x7f4f34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.770+0000 7f4f4659c640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f4815d320 0x7f4f34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.770+0000 7f4f4659c640 1 -- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f340047c0 msgr2=0x7f4f34004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.770+0000 7f4f4659c640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f340047c0 0x7f4f34004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.770+0000 7f4f4659c640 1 -- 172.21.15.45:0/1199711913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f34003520 con 0x7f4f34003970 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.770+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f340047c0 0x7f4f34004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.771+0000 7f4f4659c640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f34003970 0x7f4f34003d90 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800e170 tx=0x7f4f340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd7b52d1813c89d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.771+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3802cc20 con 0x7f4f34003970 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.772+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3800f040 con 0x7f4f34003970 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.772+0000 7f4f4559a640 1 -- 172.21.15.45:0/1199711913 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38034440 con 0x7f4f34003970 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.772+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1199711913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f34003970 msgr2=0x7f4f34003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.772+0000 7f4f4cdf1640 1 --2- 172.21.15.45:0/1199711913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f34003970 0x7f4f34003d90 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800e170 tx=0x7f4f340057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.772+0000 7f4f46d9d640 1 -- 172.21.15.45:0/1199711913 reap_dead start 2023-11-28T21:51:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.772+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1199711913 shutdown_connections 2023-11-28T21:51:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.772+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1199711913 >> 172.21.15.45:0/1199711913 conn(0x7f4f480cfed0 msgr2=0x7f4f480d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.772+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1199711913 shutdown_connections 2023-11-28T21:51:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.772+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1199711913 wait complete. 2023-11-28T21:51:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.774+0000 7f4f4cdf1640 1 Processor -- start 2023-11-28T21:51:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.774+0000 7f4f4cdf1640 1 -- start start 2023-11-28T21:51:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.775+0000 7f4f4cdf1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f34003970 0x7f4f4815b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.775+0000 7f4f4cdf1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f481521e0 0x7f4f48152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.775+0000 7f4f4cdf1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f48156ba0 0x7f4f48157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.775+0000 7f4f4cdf1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f48157560 con 0x7f4f481521e0 2023-11-28T21:51:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.775+0000 7f4f4cdf1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f48152c80 con 0x7f4f48156ba0 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.775+0000 7f4f4cdf1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f48152f60 con 0x7f4f34003970 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f45d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f481521e0 0x7f4f48152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f45d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f481521e0 0x7f4f48152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34654/0 (socket says 172.21.15.45:34654) 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f45d9b640 1 -- 172.21.15.45:0/1684858922 learned_addr learned my addr 172.21.15.45:0/1684858922 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f4659c640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f34003970 0x7f4f4815b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f45d9b640 1 -- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f34003970 msgr2=0x7f4f4815b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f34003970 0x7f4f4815b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f45d9b640 1 -- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f48156ba0 msgr2=0x7f4f48157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f46d9d640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f48156ba0 0x7f4f48157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f48156ba0 0x7f4f48157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f45d9b640 1 -- 172.21.15.45:0/1684858922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f48153240 con 0x7f4f481521e0 2023-11-28T21:51:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f46d9d640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f48156ba0 0x7f4f48157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.776+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f481521e0 0x7f4f48152660 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000d8b0 tx=0x7f4f3000dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.777+0000 7f4f3effd640 1 -- 172.21.15.45:0/1684858922 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f300074e0 con 0x7f4f481521e0 2023-11-28T21:51:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.777+0000 7f4f3effd640 1 -- 172.21.15.45:0/1684858922 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f30007640 con 0x7f4f481521e0 2023-11-28T21:51:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.777+0000 7f4f3effd640 1 -- 172.21.15.45:0/1684858922 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f300049e0 con 0x7f4f481521e0 2023-11-28T21:51:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.777+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1684858922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f48153520 con 0x7f4f481521e0 2023-11-28T21:51:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.777+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1684858922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f48150fc0 con 0x7f4f481521e0 2023-11-28T21:51:52.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.780+0000 7f4f3effd640 1 -- 172.21.15.45:0/1684858922 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4f30004b40 con 0x7f4f481521e0 2023-11-28T21:51:52.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.781+0000 7f4f3effd640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4f2806af90 0x7f4f2806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.781+0000 7f4f3effd640 1 -- 172.21.15.45:0/1684858922 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4f30089520 con 0x7f4f481521e0 2023-11-28T21:51:52.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.782+0000 7f4f4f986740 1 Processor -- start 2023-11-28T21:51:52.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.782+0000 7f4f4659c640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4f2806af90 0x7f4f2806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.782+0000 7f4f4f986740 1 -- start start 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 0x5576b759e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b759d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b759d5e0 0x5576b75e1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f4f986740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576b74fc880 con 0x5576b7598ba0 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f4f986740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576b750bae0 con 0x5576b759dea0 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f4f986740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576b75091d0 con 0x5576b759d5e0 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f4659c640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4f2806af90 0x7f4f2806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4f38036000 tx=0x7f4f3802e000 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f45d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b759d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f45d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b759d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34656/0 (socket says 172.21.15.45:34656) 2023-11-28T21:51:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f45d9b640 1 -- 172.21.15.45:0/2530103473 learned_addr learned my addr 172.21.15.45:0/2530103473 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f4659c640 1 --2- 172.21.15.45:0/2530103473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 0x5576b759e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f45d9b640 1 -- 172.21.15.45:0/2530103473 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b759d5e0 msgr2=0x5576b75e1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/2530103473 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b759d5e0 0x5576b75e1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f45d9b640 1 -- 172.21.15.45:0/2530103473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 msgr2=0x5576b759e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/2530103473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 0x5576b759e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.783+0000 7f4f45d9b640 1 -- 172.21.15.45:0/2530103473 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576b74cc860 con 0x5576b7598ba0 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.784+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/2530103473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b759d0a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f4f30087560 tx=0x7f4f300586b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23294a69cb04265c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.784+0000 7f4f27fff640 1 -- 172.21.15.45:0/2530103473 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3005e0d0 con 0x5576b7598ba0 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.784+0000 7f4f27fff640 1 -- 172.21.15.45:0/2530103473 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f30060170 con 0x5576b7598ba0 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.784+0000 7f4f27fff640 1 -- 172.21.15.45:0/2530103473 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30054580 con 0x5576b7598ba0 2023-11-28T21:51:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.784+0000 7f4f4f986740 1 -- 172.21.15.45:0/2530103473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 msgr2=0x5576b759d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.784+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2530103473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b759d0a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f4f30087560 tx=0x7f4f300586b0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.785+0000 7f4f4f986740 1 -- 172.21.15.45:0/2530103473 shutdown_connections 2023-11-28T21:51:52.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.785+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2530103473 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b759d5e0 0x5576b75e1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.785+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2530103473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b759d0a0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.785+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2530103473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 0x5576b759e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.785+0000 7f4f4f986740 1 -- 172.21.15.45:0/2530103473 >> 172.21.15.45:0/2530103473 conn(0x5576b755fc70 msgr2=0x5576b75e3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:52.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.785+0000 7f4f4f986740 1 -- 172.21.15.45:0/2530103473 shutdown_connections 2023-11-28T21:51:52.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.785+0000 7f4f4f986740 1 -- 172.21.15.45:0/2530103473 wait complete. 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.785+0000 7f4f4f986740 1 -- start start 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.786+0000 7f4f4f986740 1 Processor -- start 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 0x5576b75c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 0x5576b75c8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759dea0 0x5576b75e1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f4f986740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576b7504ac0 con 0x5576b759dea0 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f4f986740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576b74a12f0 con 0x5576b759d5e0 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f4f986740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576b74bec70 con 0x5576b7598ba0 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f4f986740 10 client.0 _ll_register_callbacks cb 0x5576b74c33b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T21:51:52.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f46d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759dea0 0x5576b75e1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f46d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759dea0 0x5576b75e1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34666/0 (socket says 172.21.15.45:34666) 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f46d9d640 1 -- 172.21.15.45:0/2897970143 learned_addr learned my addr 172.21.15.45:0/2897970143 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 0x5576b75c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f46d9d640 1 -- 172.21.15.45:0/2897970143 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 msgr2=0x5576b75c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f46d9d640 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 0x5576b75c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f46d9d640 1 -- 172.21.15.45:0/2897970143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 msgr2=0x5576b75c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f46d9d640 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 0x5576b75c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.787+0000 7f4f46d9d640 1 -- 172.21.15.45:0/2897970143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f48150fc0 con 0x5576b759dea0 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.788+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 0x5576b75c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.788+0000 7f4f46d9d640 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759dea0 0x5576b75e1a90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f4f40004700 tx=0x7f4f40010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d959649812544b5f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:52.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.788+0000 7f4f257fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.788+0000 7f4f257fa640 1 -- 172.21.15.45:0/2897970143 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40004220 con 0x5576b759dea0 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.788+0000 7f4f4f986740 1 -- 172.21.15.45:0/2897970143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5576b74fdaa0 con 0x5576b759dea0 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.788+0000 7f4f257fa640 1 -- 172.21.15.45:0/2897970143 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4f40004380 con 0x5576b759dea0 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.788+0000 7f4f257fa640 1 -- 172.21.15.45:0/2897970143 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f40003610 con 0x5576b759dea0 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.788+0000 7f4f4f986740 10 client.9789 fetch_fsmap learned FSMap version 84 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.788+0000 7f4f4f986740 1 -- 172.21.15.45:0/2897970143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x5576b7503140 con 0x5576b759dea0 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.789+0000 7f4f257fa640 1 -- 172.21.15.45:0/2897970143 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5576b74fdaa0 con 0x5576b759dea0 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.789+0000 7f4f257fa640 1 -- 172.21.15.45:0/2897970143 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 84) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4f40003770 con 0x5576b759dea0 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.789+0000 7f4f4f986740 10 client.9789 fetch_fsmap finished waiting for FSMap version 84 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.789+0000 7f4f4f986740 10 client.9789 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T21:51:52.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.789+0000 7f4f4f986740 20 client.9789 populate_metadata read hostname 'smithi045' 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.789+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] conn(0x5576b75ca540 0x5576b75e90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.789+0000 7f4f4f986740 4 client.9789 mds_command: new command op to 9660 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.789+0000 7f4f4f986740 1 -- 172.21.15.45:0/2897970143 --> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5576b74fd7c0 con 0x5576b75ca540 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.789+0000 7f4f4659c640 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] conn(0x5576b75ca540 0x5576b75e90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.790+0000 7f4f4659c640 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] conn(0x5576b75ca540 0x5576b75e90b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f21be84343ef787e server_cookie=3afc0a29060eaaa7 in_seq=0 out_seq=0 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.790+0000 7f4f257fa640 10 client.9789 ms_handle_connect on v2:172.21.15.45:6834/99455191 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f257fa640 1 -- 172.21.15.45:0/2897970143 <== mds.0 v2:172.21.15.45:6834/99455191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5576b74fd7c0 con 0x5576b75ca540 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f257fa640 10 client.9789 handle_command_reply: tid=0 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 client.9789 shutdown 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 -- 172.21.15.45:0/2897970143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759dea0 msgr2=0x5576b75e1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759dea0 0x5576b75e1a90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f4f40004700 tx=0x7f4f40010680 comp rx=0 tx=0).stop 2023-11-28T21:51:52.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 -- 172.21.15.45:0/2897970143 shutdown_connections 2023-11-28T21:51:52.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759dea0 0x5576b75e1a90 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] conn(0x5576b75ca540 0x5576b75e90b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 0x5576b75c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f257fa640 0 client.9789 ms_handle_reset on v2:172.21.15.45:6834/99455191 2023-11-28T21:51:52.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 --2- 172.21.15.45:0/2897970143 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 0x5576b75c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 -- 172.21.15.45:0/2897970143 >> 172.21.15.45:0/2897970143 conn(0x5576b755fc70 msgr2=0x5576b75e1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:52.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 -- 172.21.15.45:0/2897970143 shutdown_connections 2023-11-28T21:51:52.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 1 -- 172.21.15.45:0/2897970143 wait complete. 2023-11-28T21:51:52.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.791+0000 7f4f4f986740 20 client.9789 trim_cache size 0 max 16384 2023-11-28T21:51:52.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.809+0000 7f4f4f986740 1 Processor -- start 2023-11-28T21:51:52.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.809+0000 7f4f4f986740 1 -- start start 2023-11-28T21:51:52.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.810+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 0x5576b759e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.810+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759d5e0 0x5576b7597d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.810+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 0x5576b75eaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.810+0000 7f4f4f986740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576b7267f20 con 0x5576b759d5e0 2023-11-28T21:51:52.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.810+0000 7f4f4f986740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576b75c9540 con 0x5576b759dea0 2023-11-28T21:51:52.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.810+0000 7f4f4f986740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576b759e7c0 con 0x5576b7598ba0 2023-11-28T21:51:52.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.810+0000 7f4f45d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759d5e0 0x5576b7597d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.810+0000 7f4f45d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759d5e0 0x5576b7597d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34674/0 (socket says 172.21.15.45:34674) 2023-11-28T21:51:52.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.810+0000 7f4f45d9b640 1 -- 172.21.15.45:0/122510283 learned_addr learned my addr 172.21.15.45:0/122510283 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:52.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f46d9d640 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 0x5576b75eaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f4659c640 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 0x5576b759e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f45d9b640 1 -- 172.21.15.45:0/122510283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 msgr2=0x5576b759e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 0x5576b759e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f45d9b640 1 -- 172.21.15.45:0/122510283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 msgr2=0x5576b75eaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 0x5576b75eaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f45d9b640 1 -- 172.21.15.45:0/122510283 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576b74a0b80 con 0x5576b759d5e0 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f4659c640 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 0x5576b759e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759d5e0 0x5576b7597d70 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f4f30057e00 tx=0x7f4f30089050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8bcc7fa01325c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f3cff9640 1 -- 172.21.15.45:0/122510283 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3005e040 con 0x5576b759d5e0 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f3cff9640 1 -- 172.21.15.45:0/122510283 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f30055fb0 con 0x5576b759d5e0 2023-11-28T21:51:52.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.811+0000 7f4f3cff9640 1 -- 172.21.15.45:0/122510283 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30054580 con 0x5576b759d5e0 2023-11-28T21:51:52.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.812+0000 7f4f4f986740 1 -- 172.21.15.45:0/122510283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759d5e0 msgr2=0x5576b7597d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.812+0000 7f4f4f986740 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759d5e0 0x5576b7597d70 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f4f30057e00 tx=0x7f4f30089050 comp rx=0 tx=0).stop 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.812+0000 7f4f4f986740 1 -- 172.21.15.45:0/122510283 shutdown_connections 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.812+0000 7f4f4f986740 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759dea0 0x5576b75eaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.812+0000 7f4f4f986740 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b759d5e0 0x5576b7597d70 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.812+0000 7f4f4f986740 1 --2- 172.21.15.45:0/122510283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b7598ba0 0x5576b759e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.812+0000 7f4f4f986740 1 -- 172.21.15.45:0/122510283 >> 172.21.15.45:0/122510283 conn(0x5576b755fc70 msgr2=0x5576b759c790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.812+0000 7f4f4f986740 1 -- 172.21.15.45:0/122510283 shutdown_connections 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.812+0000 7f4f4f986740 1 -- 172.21.15.45:0/122510283 wait complete. 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.813+0000 7f4f4f986740 1 -- start start 2023-11-28T21:51:52.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.813+0000 7f4f4f986740 1 Processor -- start 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b75be4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 0x5576b75df850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4f986740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b759dea0 0x5576b75ead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4f986740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576b75cb900 con 0x5576b7598ba0 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4f986740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576b75bedd0 con 0x5576b759d5e0 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4f986740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576b75e7c00 con 0x5576b759dea0 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4f986740 10 client.0 _ll_register_callbacks cb 0x7f4f40009320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b75be4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b75be4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34676/0 (socket says 172.21.15.45:34676) 2023-11-28T21:51:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.814+0000 7f4f4659c640 1 -- 172.21.15.45:0/697476432 learned_addr learned my addr 172.21.15.45:0/697476432 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f46d9d640 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b759dea0 0x5576b75ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f4659c640 1 -- 172.21.15.45:0/697476432 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b759dea0 msgr2=0x5576b75ead60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 0x5576b75df850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f4659c640 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b759dea0 0x5576b75ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f4659c640 1 -- 172.21.15.45:0/697476432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 msgr2=0x5576b75df850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f4659c640 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 0x5576b75df850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f4659c640 1 -- 172.21.15.45:0/697476432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576b74cc860 con 0x5576b7598ba0 2023-11-28T21:51:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f4659c640 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b75be4d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f4f38037040 tx=0x7f4f38004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54b4f86917c7a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f257fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f257fa640 1 -- 172.21.15.45:0/697476432 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38046040 con 0x5576b7598ba0 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f257fa640 1 -- 172.21.15.45:0/697476432 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4f3804b310 con 0x5576b7598ba0 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.815+0000 7f4f4f986740 1 -- 172.21.15.45:0/697476432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f30060170 con 0x5576b7598ba0 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f257fa640 1 -- 172.21.15.45:0/697476432 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f38034e70 con 0x5576b7598ba0 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f4f986740 10 client.9795 fetch_fsmap learned FSMap version 84 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f4f986740 1 -- 172.21.15.45:0/697476432 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x5576b74c6170 con 0x5576b7598ba0 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f257fa640 1 -- 172.21.15.45:0/697476432 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f30060170 con 0x5576b7598ba0 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f257fa640 1 -- 172.21.15.45:0/697476432 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 84) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4f38051030 con 0x5576b7598ba0 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f4f986740 10 client.9795 fetch_fsmap finished waiting for FSMap version 84 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f4f986740 10 client.9795 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f4f986740 20 client.9795 populate_metadata read hostname 'smithi045' 2023-11-28T21:51:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f4f986740 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] conn(0x5576b75eb2a0 0x5576b75f6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:52.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f4f986740 4 client.9795 mds_command: new command op to 9660 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T21:51:52.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f4f986740 1 -- 172.21.15.45:0/697476432 --> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5576b72056c0 con 0x5576b75eb2a0 2023-11-28T21:51:52.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.816+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] conn(0x5576b75eb2a0 0x5576b75f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:52.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.817+0000 7f4f45d9b640 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] conn(0x5576b75eb2a0 0x5576b75f6090 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39d66ebaf2a671ae server_cookie=71081367ce48627 in_seq=0 out_seq=0 2023-11-28T21:51:52.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.817+0000 7f4f257fa640 10 client.9795 ms_handle_connect on v2:172.21.15.45:6834/99455191 2023-11-28T21:51:52.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.817+0000 7f4f257fa640 1 -- 172.21.15.45:0/697476432 <== mds.0 v2:172.21.15.45:6834/99455191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x5576b72056c0 con 0x5576b75eb2a0 2023-11-28T21:51:52.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.817+0000 7f4f257fa640 10 client.9795 handle_command_reply: tid=0 2023-11-28T21:51:52.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.817+0000 7f4f4f986740 1 client.9795 shutdown 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.817+0000 7f4f4f986740 1 -- 172.21.15.45:0/697476432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 msgr2=0x5576b75be4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.817+0000 7f4f4f986740 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b75be4d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f4f38037040 tx=0x7f4f38004ab0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f4f986740 1 -- 172.21.15.45:0/697476432 shutdown_connections 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f4f986740 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5576b759dea0 0x5576b75ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f4f986740 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.45:6834/99455191,v1:172.21.15.45:6835/99455191] conn(0x5576b75eb2a0 0x5576b75f6090 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f4f986740 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5576b759d5e0 0x5576b75df850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f4f986740 1 --2- 172.21.15.45:0/697476432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576b7598ba0 0x5576b75be4d0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f4f986740 1 -- 172.21.15.45:0/697476432 >> 172.21.15.45:0/697476432 conn(0x5576b755fc70 msgr2=0x5576b75c7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f257fa640 0 client.9795 ms_handle_reset on v2:172.21.15.45:6834/99455191 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f4f986740 1 -- 172.21.15.45:0/697476432 shutdown_connections 2023-11-28T21:51:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f4f986740 1 -- 172.21.15.45:0/697476432 wait complete. 2023-11-28T21:51:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.818+0000 7f4f4f986740 20 client.9795 trim_cache size 0 max 16384 2023-11-28T21:51:52.825 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T21:51:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T21:51:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T21:51:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 9660, 2023-11-28T21:51:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T21:51:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T21:51:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T21:51:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.876860486, 2023-11-28T21:51:52.826 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 84, 2023-11-28T21:51:52.826 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 185, 2023-11-28T21:51:52.826 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 185, 2023-11-28T21:51:52.826 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 13.188731079 2023-11-28T21:51:52.826 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4f2806af90 msgr2=0x7f4f2806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4f2806af90 0x7f4f2806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4f38036000 tx=0x7f4f3802e000 comp rx=0 tx=0).stop 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1684858922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f481521e0 msgr2=0x7f4f48152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f481521e0 0x7f4f48152660 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000d8b0 tx=0x7f4f3000dd70 comp rx=0 tx=0).stop 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1684858922 shutdown_connections 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f4f2806af90 0x7f4f2806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f48156ba0 0x7f4f48157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f481521e0 0x7f4f48152660 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 --2- 172.21.15.45:0/1684858922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f34003970 0x7f4f4815b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1684858922 >> 172.21.15.45:0/1684858922 conn(0x7f4f480cfed0 msgr2=0x7f4f480b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:52.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1684858922 shutdown_connections 2023-11-28T21:51:52.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:52.819+0000 7f4f4cdf1640 1 -- 172.21.15.45:0/1684858922 wait complete. 2023-11-28T21:51:52.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T21:51:53.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.049+0000 7f965258f640 1 Processor -- start 2023-11-28T21:51:53.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.049+0000 7f965258f640 1 -- start start 2023-11-28T21:51:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.050+0000 7f965258f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f964c0b3870 0x7f964c15db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.050+0000 7f965258f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c15e680 con 0x7f964c06d880 2023-11-28T21:51:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.050+0000 7f965258f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c15e960 con 0x7f964c0721e0 2023-11-28T21:51:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.050+0000 7f965258f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964c15a460 con 0x7f964c0b3870 2023-11-28T21:51:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.051+0000 7f9651d8e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f964c06d880 0x7f964c0ac840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60942/0 (socket says 172.21.15.45:60942) 2023-11-28T21:51:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.051+0000 7f9651d8e640 1 -- 172.21.15.45:0/2912017664 learned_addr learned my addr 172.21.15.45:0/2912017664 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.051+0000 7f965158d640 1 --2- 172.21.15.45:0/2912017664 >> v2:172.21.15.190:3300/0 conn(0x7f964c0b3870 0x7f964c15db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.051+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971297549 0 0) 0x7f964c15e680 con 0x7f964c06d880 2023-11-28T21:51:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.051+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9638003520 con 0x7f964c06d880 2023-11-28T21:51:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 359654574 0 0) 0x7f9638003520 con 0x7f964c06d880 2023-11-28T21:51:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 >> v2:172.21.15.190:3300/0 conn(0x7f964c0b3870 msgr2=0x7f964c15db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 --2- 172.21.15.45:0/2912017664 >> v2:172.21.15.190:3300/0 conn(0x7f964c0b3870 0x7f964c15db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 >> v1:172.21.15.190:6789/0 conn(0x7f964c0721e0 legacy=0x7f964c06d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c071700 con 0x7f964c06d880 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f96440019f0 con 0x7f964c06d880 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 >> v1:172.21.15.45:6789/0 conn(0x7f964c06d880 legacy=0x7f964c0ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 --2- 172.21.15.45:0/2912017664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96380039c0 0x7f9638003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 --2- 172.21.15.45:0/2912017664 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96380047c0 0x7f9638004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 --2- 172.21.15.45:0/2912017664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9638005180 0x7f963800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963800c050 con 0x7f96380039c0 2023-11-28T21:51:53.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963800e3a0 con 0x7f9638005180 2023-11-28T21:51:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963800e560 con 0x7f96380047c0 2023-11-28T21:51:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f965158d640 1 --2- 172.21.15.45:0/2912017664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96380039c0 0x7f9638003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f965158d640 1 -- 172.21.15.45:0/2912017664 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96380047c0 msgr2=0x7f9638004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f965158d640 1 --2- 172.21.15.45:0/2912017664 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96380047c0 0x7f9638004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f965158d640 1 -- 172.21.15.45:0/2912017664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9638005180 msgr2=0x7f963800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f965158d640 1 --2- 172.21.15.45:0/2912017664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9638005180 0x7f963800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f965158d640 1 -- 172.21.15.45:0/2912017664 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638003520 con 0x7f96380039c0 2023-11-28T21:51:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.052+0000 7f9651d8e640 1 -- 172.21.15.45:0/2912017664 reap_dead start 2023-11-28T21:51:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f965158d640 1 --2- 172.21.15.45:0/2912017664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96380039c0 0x7f9638003dc0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f9648009ac0 tx=0x7f96380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c92fd491a1b49b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964802c710 con 0x7f96380039c0 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964802c870 con 0x7f96380039c0 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f963ffff640 1 -- 172.21.15.45:0/2912017664 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964802cc00 con 0x7f96380039c0 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f965258f640 1 -- 172.21.15.45:0/2912017664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96380039c0 msgr2=0x7f9638003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f965258f640 1 --2- 172.21.15.45:0/2912017664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96380039c0 0x7f9638003dc0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f9648009ac0 tx=0x7f96380057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f965258f640 1 -- 172.21.15.45:0/2912017664 shutdown_connections 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f965258f640 1 --2- 172.21.15.45:0/2912017664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96380039c0 0x7f9638003dc0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f965258f640 1 -- 172.21.15.45:0/2912017664 >> 172.21.15.45:0/2912017664 conn(0x7f964c06ab30 msgr2=0x7f964c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.054+0000 7f965258f640 1 -- 172.21.15.45:0/2912017664 shutdown_connections 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.055+0000 7f965258f640 1 -- 172.21.15.45:0/2912017664 wait complete. 2023-11-28T21:51:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.056+0000 7f965258f640 1 Processor -- start 2023-11-28T21:51:53.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.056+0000 7f965258f640 1 -- start start 2023-11-28T21:51:53.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.057+0000 7f965258f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96380039c0 0x7f964c146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:53.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.057+0000 7f965258f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f964c156630 0x7f964c144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.057+0000 7f965258f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f964c145450 0x7f964c0cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.057+0000 7f965258f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c145e30 con 0x7f964c156630 2023-11-28T21:51:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.057+0000 7f965258f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c146110 con 0x7f964c145450 2023-11-28T21:51:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.057+0000 7f965258f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c1463f0 con 0x7f96380039c0 2023-11-28T21:51:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9650d8c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f964c156630 0x7f964c144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9650d8c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f964c156630 0x7f964c144d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34682/0 (socket says 172.21.15.45:34682) 2023-11-28T21:51:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9650d8c640 1 -- 172.21.15.45:0/1526251160 learned_addr learned my addr 172.21.15.45:0/1526251160 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f965158d640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96380039c0 0x7f964c146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9651d8e640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f964c145450 0x7f964c0cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9650d8c640 1 -- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96380039c0 msgr2=0x7f964c146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9650d8c640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96380039c0 0x7f964c146ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9650d8c640 1 -- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f964c145450 msgr2=0x7f964c0cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9650d8c640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f964c145450 0x7f964c0cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9650d8c640 1 -- 172.21.15.45:0/1526251160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c1466d0 con 0x7f964c156630 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.058+0000 7f9651d8e640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f964c145450 0x7f964c0cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.059+0000 7f9650d8c640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f964c156630 0x7f964c144d80 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f964000c3c0 tx=0x7f964000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.059+0000 7f963dffb640 1 -- 172.21.15.45:0/1526251160 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640011790 con 0x7f964c156630 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.059+0000 7f963dffb640 1 -- 172.21.15.45:0/1526251160 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96400118f0 con 0x7f964c156630 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.059+0000 7f963dffb640 1 -- 172.21.15.45:0/1526251160 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640011ce0 con 0x7f964c156630 2023-11-28T21:51:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.059+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964c1469b0 con 0x7f964c156630 2023-11-28T21:51:53.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.059+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964c145ae0 con 0x7f964c156630 2023-11-28T21:51:53.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.060+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c146ba0 con 0x7f964c156630 2023-11-28T21:51:53.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.061+0000 7f963dffb640 1 -- 172.21.15.45:0/1526251160 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f964001c460 con 0x7f964c156630 2023-11-28T21:51:53.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.063+0000 7f963dffb640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f963006af60 0x7f963006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:53.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.063+0000 7f963dffb640 1 -- 172.21.15.45:0/1526251160 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f96400897d0 con 0x7f964c156630 2023-11-28T21:51:53.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.063+0000 7f965158d640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f963006af60 0x7f963006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:53.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.065+0000 7f965158d640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f963006af60 0x7f963006d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9648036000 tx=0x7f9648030000 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:53.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.067+0000 7f963dffb640 1 -- 172.21.15.45:0/1526251160 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96400546a0 con 0x7f964c156630 2023-11-28T21:51:53.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.306+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f964c09d0f0 con 0x7f964c156630 2023-11-28T21:51:53.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.307+0000 7f963dffb640 1 -- 172.21.15.45:0/1526251160 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v84) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9640057cc0 con 0x7f964c156630 2023-11-28T21:51:53.308 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T21:51:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f963006af60 msgr2=0x7f963006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f963006af60 0x7f963006d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9648036000 tx=0x7f9648030000 comp rx=0 tx=0).stop 2023-11-28T21:51:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f964c156630 msgr2=0x7f964c144d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f964c156630 0x7f964c144d80 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f964000c3c0 tx=0x7f964000c7c0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 shutdown_connections 2023-11-28T21:51:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f963006af60 0x7f963006d420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f964c145450 0x7f964c0cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f964c156630 0x7f964c144d80 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 --2- 172.21.15.45:0/1526251160 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96380039c0 0x7f964c146ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:53.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 >> 172.21.15.45:0/1526251160 conn(0x7f964c06ab30 msgr2=0x7f964c06c900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:53.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 shutdown_connections 2023-11-28T21:51:53.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:53.310+0000 7f965258f640 1 -- 172.21.15.45:0/1526251160 wait complete. 2023-11-28T21:51:53.332 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T21:51:53.333 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T21:51:53.333 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T21:51:53.333 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T21:51:53.333 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T21:51:53.333 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T21:51:53.333 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T21:51:53.334 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T21:51:53.334 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T21:51:53.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:53.334 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T21:51:53.390 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T21:51:53.390 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T21:51:53.390 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T21:51:53.390 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:51:53.390 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T21:51:53.390 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:51:53.391 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T21:51:53.391 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T21:51:53.391 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T21:51:53.391 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T21:51:53.391 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T21:51:53.391 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T21:51:53.391 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T21:51:53.392 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T21:51:53.392 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T21:51:53.392 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T21:51:53.392 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 37629sec preferred_lft 37629sec 2023-11-28T21:51:53.392 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T21:51:53.392 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:51:53.392 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T21:51:53.393 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T21:51:53.394 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T21:51:53.394 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:51:53.394 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T21:51:53.394 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T21:51:53.394 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T21:51:53.394 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:51:53.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:53.395 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T21:51:53.489 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T21:51:53.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:53.494 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T21:51:53.551 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T21:51:53.551 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T21:51:53.551 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T21:51:53.551 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T21:51:53.551 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T21:51:53.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:53.552 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T21:51:53.552 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T21:51:53.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T21:51:53.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T21:51:53.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T21:51:53.552 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T21:51:53.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:53.749 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T21:51:53.804 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T21:51:53.804 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T21:51:53.805 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T21:51:53.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:53.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T21:51:53.916 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T21:51:53.916 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T21:51:53.972 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T21:51:53.972 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T21:51:53.973 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T21:51:54.027 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T21:51:54.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:54.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T21:51:54.177 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T21:51:54.178 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T21:51:54.178 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T21:51:54.201 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T21:51:54.201 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T21:51:54.201 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T21:51:54.201 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T21:51:54.201 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T21:51:54.227 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T21:51:54.227 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T21:51:54.227 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T21:51:54.227 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T21:51:54.227 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T21:51:54.228 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T21:51:54.228 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T21:51:54.228 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T21:51:54.230 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T21:51:54.230 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T21:51:54.230 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T21:51:54.230 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T21:51:54.230 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T21:51:54.234 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T21:51:54.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:54.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T21:51:54.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:54.322 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T21:51:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:client7193 2023-11-28T21:51:54.391 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:51:54.391 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:51:54.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T21:51:54.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.670+0000 7fa700461640 1 Processor -- start 2023-11-28T21:51:54.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.671+0000 7fa700461640 1 -- start start 2023-11-28T21:51:54.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.672+0000 7fa700461640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa6f815b820 0x7fa6f806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:54.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.672+0000 7fa700461640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f80cf020 0x7fa6f80cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:54.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.672+0000 7fa700461640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f815fa80 con 0x7fa6f815edc0 2023-11-28T21:51:54.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.672+0000 7fa700461640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f815fd60 con 0x7fa6f815b820 2023-11-28T21:51:54.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.672+0000 7fa700461640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f80cf9f0 con 0x7fa6f80cf020 2023-11-28T21:51:54.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.673+0000 7fa6fe9fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa6f815edc0 0x7fa6f815f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36372/0 (socket says 172.21.15.45:36372) 2023-11-28T21:51:54.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.673+0000 7fa6fe9fe640 1 -- 172.21.15.45:0/521245543 learned_addr learned my addr 172.21.15.45:0/521245543 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:51:54.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.673+0000 7fa6fe1fd640 1 --2- 172.21.15.45:0/521245543 >> v2:172.21.15.190:3300/0 conn(0x7fa6f815b820 0x7fa6f806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:54.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.673+0000 7fa6fd9fc640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f80cf020 0x7fa6f80cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.673+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424788793 0 0) 0x7fa6f815fa80 con 0x7fa6f815edc0 2023-11-28T21:51:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.673+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e0003520 con 0x7fa6f815edc0 2023-11-28T21:51:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251413612 0 0) 0x7fa6e0003520 con 0x7fa6f815edc0 2023-11-28T21:51:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f80cf020 msgr2=0x7fa6f80cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f80cf020 0x7fa6f80cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 >> v2:172.21.15.190:3300/0 conn(0x7fa6f815b820 msgr2=0x7fa6f806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 --2- 172.21.15.45:0/521245543 >> v2:172.21.15.190:3300/0 conn(0x7fa6f815b820 0x7fa6f806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f806cf40 con 0x7fa6f815edc0 2023-11-28T21:51:54.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fa6f0001670 con 0x7fa6f815edc0 2023-11-28T21:51:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 >> v1:172.21.15.45:6789/0 conn(0x7fa6f815edc0 legacy=0x7fa6f815f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6e0003970 0x7fa6e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6e00047c0 0x7fa6e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f80cf020 0x7fa6e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f815fd60 con 0x7fa6e0003970 2023-11-28T21:51:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f80cf9f0 con 0x7fa6e00047c0 2023-11-28T21:51:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e000ba00 con 0x7fa6f80cf020 2023-11-28T21:51:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.674+0000 7fa6fe1fd640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6e0003970 0x7fa6e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.675+0000 7fa6fd9fc640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6e00047c0 0x7fa6e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:54.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.675+0000 7fa6fe1fd640 1 -- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f80cf020 msgr2=0x7fa6e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:54.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.675+0000 7fa6fe1fd640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f80cf020 0x7fa6e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.675+0000 7fa6fe1fd640 1 -- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6e00047c0 msgr2=0x7fa6e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:54.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.675+0000 7fa6fe1fd640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6e00047c0 0x7fa6e0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.675+0000 7fa6fe1fd640 1 -- 172.21.15.45:0/521245543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e0003520 con 0x7fa6e0003970 2023-11-28T21:51:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.676+0000 7fa6fe1fd640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6e0003970 0x7fa6e0003d90 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009fc0 tx=0x7fa6e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f52b87f82889e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.676+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e802cc60 con 0x7fa6e0003970 2023-11-28T21:51:54.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.676+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e800f040 con 0x7fa6e0003970 2023-11-28T21:51:54.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.676+0000 7fa6fd1fb640 1 -- 172.21.15.45:0/521245543 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8034440 con 0x7fa6e0003970 2023-11-28T21:51:54.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.677+0000 7fa700461640 1 -- 172.21.15.45:0/521245543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6e0003970 msgr2=0x7fa6e0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:54.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.677+0000 7fa700461640 1 --2- 172.21.15.45:0/521245543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6e0003970 0x7fa6e0003d90 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009fc0 tx=0x7fa6e00057f0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.677+0000 7fa6fe9fe640 1 -- 172.21.15.45:0/521245543 reap_dead start 2023-11-28T21:51:54.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.677+0000 7fa700461640 1 -- 172.21.15.45:0/521245543 shutdown_connections 2023-11-28T21:51:54.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.677+0000 7fa700461640 1 -- 172.21.15.45:0/521245543 >> 172.21.15.45:0/521245543 conn(0x7fa6f806a6e0 msgr2=0x7fa6f806aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:54.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.677+0000 7fa700461640 1 -- 172.21.15.45:0/521245543 shutdown_connections 2023-11-28T21:51:54.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.677+0000 7fa700461640 1 -- 172.21.15.45:0/521245543 wait complete. 2023-11-28T21:51:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.679+0000 7fa700461640 1 Processor -- start 2023-11-28T21:51:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.679+0000 7fa700461640 1 -- start start 2023-11-28T21:51:54.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.680+0000 7fa700461640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6e0003970 0x7fa6f814c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.680+0000 7fa700461640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f814d770 0x7fa6f814a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.680+0000 7fa700461640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6f814aaa0 0x7fa6f80b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.680+0000 7fa700461640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f80cad20 con 0x7fa6f814aaa0 2023-11-28T21:51:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.680+0000 7fa700461640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f814c890 con 0x7fa6f814d770 2023-11-28T21:51:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.680+0000 7fa700461640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f814a770 con 0x7fa6e0003970 2023-11-28T21:51:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.680+0000 7fa6fe9fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6f814aaa0 0x7fa6f80b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.680+0000 7fa6fe9fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6f814aaa0 0x7fa6f80b0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39116/0 (socket says 172.21.15.45:39116) 2023-11-28T21:51:54.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.680+0000 7fa6fe9fe640 1 -- 172.21.15.45:0/1992023256 learned_addr learned my addr 172.21.15.45:0/1992023256 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6fd9fc640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f814d770 0x7fa6f814a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6fe1fd640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6e0003970 0x7fa6f814c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6fe9fe640 1 -- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6e0003970 msgr2=0x7fa6f814c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6fe9fe640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6e0003970 0x7fa6f814c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6fe9fe640 1 -- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f814d770 msgr2=0x7fa6f814a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6fe9fe640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f814d770 0x7fa6f814a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6fe9fe640 1 -- 172.21.15.45:0/1992023256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f814b2d0 con 0x7fa6f814aaa0 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6fd9fc640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f814d770 0x7fa6f814a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6fe9fe640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6f814aaa0 0x7fa6f80b0180 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fa6f809cf90 tx=0x7fa6f000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:54.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6f6ffd640 1 -- 172.21.15.45:0/1992023256 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00074e0 con 0x7fa6f814aaa0 2023-11-28T21:51:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6f6ffd640 1 -- 172.21.15.45:0/1992023256 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f0007680 con 0x7fa6f814aaa0 2023-11-28T21:51:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa6f6ffd640 1 -- 172.21.15.45:0/1992023256 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0004a00 con 0x7fa6f814aaa0 2023-11-28T21:51:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.681+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f814b5b0 con 0x7fa6f814aaa0 2023-11-28T21:51:54.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.682+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f814bb70 con 0x7fa6f814aaa0 2023-11-28T21:51:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.683+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f8073440 con 0x7fa6f814aaa0 2023-11-28T21:51:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.684+0000 7fa6f6ffd640 1 -- 172.21.15.45:0/1992023256 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa6f0004b60 con 0x7fa6f814aaa0 2023-11-28T21:51:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.685+0000 7fa6f6ffd640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fa6e406af60 0x7fa6e406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.685+0000 7fa6f6ffd640 1 -- 172.21.15.45:0/1992023256 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa6f0088b60 con 0x7fa6f814aaa0 2023-11-28T21:51:54.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.685+0000 7fa6fe1fd640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fa6e406af60 0x7fa6e406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:54.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.687+0000 7fa6fe1fd640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fa6e406af60 0x7fa6e406d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8036000 tx=0x7fa6e802e000 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:54.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.689+0000 7fa6f6ffd640 1 -- 172.21.15.45:0/1992023256 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6f0053a30 con 0x7fa6f814aaa0 2023-11-28T21:51:54.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.909+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 --> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6f806d500 con 0x7fa6e406af60 2023-11-28T21:51:54.914 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.913+0000 7fa6f6ffd640 1 -- 172.21.15.45:0/1992023256 <== mgr.7741 v2:172.21.15.190:6832/2040970137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa6e802cdb0 con 0x7fa6e406af60 2023-11-28T21:51:54.914 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T21:51:54.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:51:54.915 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T21:51:54.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:51:54.915 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T21:51:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fa6e406af60 msgr2=0x7fa6e406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fa6e406af60 0x7fa6e406d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8036000 tx=0x7fa6e802e000 comp rx=0 tx=0).stop 2023-11-28T21:51:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6f814aaa0 msgr2=0x7fa6f80b0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6f814aaa0 0x7fa6f80b0180 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fa6f809cf90 tx=0x7fa6f000c960 comp rx=0 tx=0).stop 2023-11-28T21:51:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 shutdown_connections 2023-11-28T21:51:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fa6e406af60 0x7fa6e406d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6f814aaa0 0x7fa6f80b0180 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f814d770 0x7fa6f814a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 --2- 172.21.15.45:0/1992023256 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6e0003970 0x7fa6f814c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.915+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 >> 172.21.15.45:0/1992023256 conn(0x7fa6f806a6e0 msgr2=0x7fa6f815c910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.916+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 shutdown_connections 2023-11-28T21:51:54.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:54.916+0000 7fa700461640 1 -- 172.21.15.45:0/1992023256 wait complete. 2023-11-28T21:51:54.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T21:51:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.175+0000 7fe09a410640 1 Processor -- start 2023-11-28T21:51:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.175+0000 7fe09a410640 1 -- start start 2023-11-28T21:51:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.177+0000 7fe09a410640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe0940ce600 0x7fe09406a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.177+0000 7fe09a410640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 0x7fe0940c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.177+0000 7fe09a410640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09406e030 con 0x7fe09406dc50 2023-11-28T21:51:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.177+0000 7fe09a410640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09406aed0 con 0x7fe0940ce600 2023-11-28T21:51:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.177+0000 7fe09a410640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940ccf60 con 0x7fe09406aa40 2023-11-28T21:51:55.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.177+0000 7fe0937fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe0940ce600 0x7fe09406a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:55.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.177+0000 7fe0937fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe0940ce600 0x7fe09406a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39124/0 (socket says 172.21.15.45:39124) 2023-11-28T21:51:55.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.177+0000 7fe0937fe640 1 -- 172.21.15.45:0/2237518536 learned_addr learned my addr 172.21.15.45:0/2237518536 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:55.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.177+0000 7fe0989ad640 1 --2- 172.21.15.45:0/2237518536 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 0x7fe0940c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:55.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.178+0000 7fe0937fe640 1 -- 172.21.15.45:0/2237518536 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 msgr2=0x7fe0940c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.178+0000 7fe0937fe640 1 --2- 172.21.15.45:0/2237518536 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 0x7fe0940c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.178+0000 7fe0937fe640 1 -- 172.21.15.45:0/2237518536 >> v1:172.21.15.190:6789/0 conn(0x7fe09406dc50 legacy=0x7fe0940cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.178+0000 7fe0937fe640 1 -- 172.21.15.45:0/2237518536 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0940cc930 con 0x7fe0940ce600 2023-11-28T21:51:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.179+0000 7fe0937fe640 1 --2- 172.21.15.45:0/2237518536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0940ce600 0x7fe09406a500 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fe088009870 tx=0x7fe08802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d87a3b0b1f9a694 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.179+0000 7fe092ffd640 1 -- 172.21.15.45:0/2237518536 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088030840 con 0x7fe0940ce600 2023-11-28T21:51:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.179+0000 7fe092ffd640 1 -- 172.21.15.45:0/2237518536 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0880309a0 con 0x7fe0940ce600 2023-11-28T21:51:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.179+0000 7fe092ffd640 1 -- 172.21.15.45:0/2237518536 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088030d30 con 0x7fe0940ce600 2023-11-28T21:51:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.180+0000 7fe09a410640 1 -- 172.21.15.45:0/2237518536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0940ce600 msgr2=0x7fe09406a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.180+0000 7fe09a410640 1 --2- 172.21.15.45:0/2237518536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0940ce600 0x7fe09406a500 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fe088009870 tx=0x7fe08802b980 comp rx=0 tx=0).stop 2023-11-28T21:51:55.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.180+0000 7fe09a410640 1 -- 172.21.15.45:0/2237518536 shutdown_connections 2023-11-28T21:51:55.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.180+0000 7fe09a410640 1 --2- 172.21.15.45:0/2237518536 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 0x7fe0940c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:55.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.180+0000 7fe09a410640 1 --2- 172.21.15.45:0/2237518536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0940ce600 0x7fe09406a500 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:55.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.180+0000 7fe09a410640 1 -- 172.21.15.45:0/2237518536 >> 172.21.15.45:0/2237518536 conn(0x7fe0940ced60 msgr2=0x7fe0940cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:55.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.180+0000 7fe09a410640 1 -- 172.21.15.45:0/2237518536 shutdown_connections 2023-11-28T21:51:55.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.180+0000 7fe09a410640 1 -- 172.21.15.45:0/2237518536 wait complete. 2023-11-28T21:51:55.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.182+0000 7fe09a410640 1 Processor -- start 2023-11-28T21:51:55.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.182+0000 7fe09a410640 1 -- start start 2023-11-28T21:51:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.183+0000 7fe09a410640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 0x7fe094155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.183+0000 7fe09a410640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe09406dc50 0x7fe0940c2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.183+0000 7fe09a410640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940ce600 0x7fe0940b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.183+0000 7fe09a410640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940b3de0 con 0x7fe09406dc50 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.183+0000 7fe09a410640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940b31d0 con 0x7fe0940ce600 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.183+0000 7fe09a410640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940b34b0 con 0x7fe09406aa40 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe09406dc50 0x7fe0940c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe09406dc50 0x7fe0940c2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39136/0 (socket says 172.21.15.45:39136) 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0937fe640 1 -- 172.21.15.45:0/96256855 learned_addr learned my addr 172.21.15.45:0/96256855 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0989ad640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940ce600 0x7fe0940b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe093fff640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 0x7fe094155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0937fe640 1 -- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 msgr2=0x7fe094155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0937fe640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 0x7fe094155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0937fe640 1 -- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940ce600 msgr2=0x7fe0940b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0937fe640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940ce600 0x7fe0940b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0937fe640 1 -- 172.21.15.45:0/96256855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0880094d0 con 0x7fe09406dc50 2023-11-28T21:51:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.184+0000 7fe0989ad640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940ce600 0x7fe0940b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.185+0000 7fe0937fe640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe09406dc50 0x7fe0940c2890 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fe088000c00 tx=0x7fe088004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.185+0000 7fe090ff9640 1 -- 172.21.15.45:0/96256855 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088039a90 con 0x7fe09406dc50 2023-11-28T21:51:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.185+0000 7fe090ff9640 1 -- 172.21.15.45:0/96256855 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe088039bf0 con 0x7fe09406dc50 2023-11-28T21:51:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.185+0000 7fe090ff9640 1 -- 172.21.15.45:0/96256855 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088043410 con 0x7fe09406dc50 2023-11-28T21:51:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.185+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0940b3790 con 0x7fe09406dc50 2023-11-28T21:51:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.185+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0941558a0 con 0x7fe09406dc50 2023-11-28T21:51:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.187+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0940b3f50 con 0x7fe09406dc50 2023-11-28T21:51:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.188+0000 7fe090ff9640 1 -- 172.21.15.45:0/96256855 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 90) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe08804c3b0 con 0x7fe09406dc50 2023-11-28T21:51:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.188+0000 7fe090ff9640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fe08006afb0 0x7fe08006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.189+0000 7fe090ff9640 1 -- 172.21.15.45:0/96256855 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe0880abf50 con 0x7fe09406dc50 2023-11-28T21:51:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.189+0000 7fe093fff640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fe08006afb0 0x7fe08006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.190+0000 7fe093fff640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fe08006afb0 0x7fe08006d470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe084000f60 tx=0x7fe084009040 comp rx=0 tx=0).ready entity=mgr.7741 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.193+0000 7fe090ff9640 1 -- 172.21.15.45:0/96256855 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe088076df0 con 0x7fe09406dc50 2023-11-28T21:51:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:55.440+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe0940733d0 con 0x7fe09406dc50 2023-11-28T21:51:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.006+0000 7fe090ff9640 1 -- 172.21.15.45:0/96256855 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v91) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe0880340a0 con 0x7fe09406dc50 2023-11-28T21:51:56.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fe08006afb0 msgr2=0x7fe08006d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fe08006afb0 0x7fe08006d470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe084000f60 tx=0x7fe084009040 comp rx=0 tx=0).stop 2023-11-28T21:51:56.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe09406dc50 msgr2=0x7fe0940c2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe09406dc50 0x7fe0940c2890 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fe088000c00 tx=0x7fe088004730 comp rx=0 tx=0).stop 2023-11-28T21:51:56.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 shutdown_connections 2023-11-28T21:51:56.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7fe08006afb0 0x7fe08006d470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940ce600 0x7fe0940b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe09406dc50 0x7fe0940c2890 secure :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fe088000c00 tx=0x7fe088004730 comp rx=0 tx=0).stop 2023-11-28T21:51:56.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 --2- 172.21.15.45:0/96256855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe09406aa40 0x7fe094155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.010+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 >> 172.21.15.45:0/96256855 conn(0x7fe0940ced60 msgr2=0x7fe094160ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.013+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 shutdown_connections 2023-11-28T21:51:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.013+0000 7fe09a410640 1 -- 172.21.15.45:0/96256855 wait complete. 2023-11-28T21:51:56.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T21:51:56.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.261+0000 7f2b1ddde640 1 Processor -- start 2023-11-28T21:51:56.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.261+0000 7f2b1ddde640 1 -- start start 2023-11-28T21:51:56.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.262+0000 7f2b1ddde640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2b1806e120 0x7f2b1806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.262+0000 7f2b1ddde640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2b1815d6e0 0x7f2b1806e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.262+0000 7f2b1ddde640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 0x7f2b180661f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.262+0000 7f2b1ddde640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1806ea40 con 0x7f2b1806e120 2023-11-28T21:51:56.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.262+0000 7f2b1ddde640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1806d680 con 0x7f2b1815d6e0 2023-11-28T21:51:56.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.262+0000 7f2b1ddde640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180cf1a0 con 0x7f2b180c3510 2023-11-28T21:51:56.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b177fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2b1806e120 0x7f2b1806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:56.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b177fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2b1806e120 0x7f2b1806d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39172/0 (socket says 172.21.15.45:39172) 2023-11-28T21:51:56.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b177fe640 1 -- 172.21.15.45:0/3620846261 learned_addr learned my addr 172.21.15.45:0/3620846261 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b17fff640 1 --2- 172.21.15.45:0/3620846261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 0x7f2b180661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b16ffd640 1 --2- 172.21.15.45:0/3620846261 >> v2:172.21.15.190:3300/0 conn(0x7f2b1815d6e0 0x7f2b1806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b177fe640 1 -- 172.21.15.45:0/3620846261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 msgr2=0x7f2b180661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b177fe640 1 --2- 172.21.15.45:0/3620846261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 0x7f2b180661f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b177fe640 1 -- 172.21.15.45:0/3620846261 >> v2:172.21.15.190:3300/0 conn(0x7f2b1815d6e0 msgr2=0x7f2b1806e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:56.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b177fe640 1 --2- 172.21.15.45:0/3620846261 >> v2:172.21.15.190:3300/0 conn(0x7f2b1815d6e0 0x7f2b1806e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b177fe640 1 -- 172.21.15.45:0/3620846261 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b180cf480 con 0x7f2b1806e120 2023-11-28T21:51:56.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.263+0000 7f2b16ffd640 1 --2- 172.21.15.45:0/3620846261 >> v2:172.21.15.190:3300/0 conn(0x7f2b1815d6e0 0x7f2b1806e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:56.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.265+0000 7f2b177fe640 1 --2- 172.21.15.45:0/3620846261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1806e120 0x7f2b1806d8f0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f2b1806f1f0 tx=0x7f2b0802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1761cee64f261bc4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:56.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.265+0000 7f2b167fc640 1 -- 172.21.15.45:0/3620846261 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0802c9e0 con 0x7f2b1806e120 2023-11-28T21:51:56.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.265+0000 7f2b167fc640 1 -- 172.21.15.45:0/3620846261 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0802cb40 con 0x7f2b1806e120 2023-11-28T21:51:56.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.265+0000 7f2b167fc640 1 -- 172.21.15.45:0/3620846261 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b080343f0 con 0x7f2b1806e120 2023-11-28T21:51:56.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.266+0000 7f2b1ddde640 1 -- 172.21.15.45:0/3620846261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1806e120 msgr2=0x7f2b1806d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.266+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/3620846261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1806e120 0x7f2b1806d8f0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f2b1806f1f0 tx=0x7f2b0802be40 comp rx=0 tx=0).stop 2023-11-28T21:51:56.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.266+0000 7f2b1ddde640 1 -- 172.21.15.45:0/3620846261 shutdown_connections 2023-11-28T21:51:56.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.266+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/3620846261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 0x7f2b180661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.266+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/3620846261 >> v2:172.21.15.190:3300/0 conn(0x7f2b1815d6e0 0x7f2b1806e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.266+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/3620846261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1806e120 0x7f2b1806d8f0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.266+0000 7f2b1ddde640 1 -- 172.21.15.45:0/3620846261 >> 172.21.15.45:0/3620846261 conn(0x7f2b180cc900 msgr2=0x7f2b180ccd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:56.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.266+0000 7f2b1ddde640 1 -- 172.21.15.45:0/3620846261 shutdown_connections 2023-11-28T21:51:56.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.266+0000 7f2b1ddde640 1 -- 172.21.15.45:0/3620846261 wait complete. 2023-11-28T21:51:56.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.268+0000 7f2b1ddde640 1 Processor -- start 2023-11-28T21:51:56.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.268+0000 7f2b1ddde640 1 -- start start 2023-11-28T21:51:56.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b1ddde640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b1806e120 0x7f2b181597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b1ddde640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 0x7f2b180c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b1ddde640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b180c6590 0x7f2b180b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b1ddde640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180c5b50 con 0x7f2b180c6590 2023-11-28T21:51:56.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b1ddde640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180c4af0 con 0x7f2b1806e120 2023-11-28T21:51:56.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b1ddde640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b18159d10 con 0x7f2b180c3510 2023-11-28T21:51:56.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b17fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b180c6590 0x7f2b180b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:56.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b17fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b180c6590 0x7f2b180b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39180/0 (socket says 172.21.15.45:39180) 2023-11-28T21:51:56.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b17fff640 1 -- 172.21.15.45:0/2898947945 learned_addr learned my addr 172.21.15.45:0/2898947945 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:56.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.269+0000 7f2b16ffd640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 0x7f2b180c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:56.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.270+0000 7f2b177fe640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b1806e120 0x7f2b181597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:56.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.270+0000 7f2b17fff640 1 -- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 msgr2=0x7f2b180c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.270+0000 7f2b17fff640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 0x7f2b180c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.270+0000 7f2b17fff640 1 -- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b1806e120 msgr2=0x7f2b181597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.270+0000 7f2b17fff640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b1806e120 0x7f2b181597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.270+0000 7f2b17fff640 1 -- 172.21.15.45:0/2898947945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b08009b30 con 0x7f2b180c6590 2023-11-28T21:51:56.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.270+0000 7f2b17fff640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b180c6590 0x7f2b180b4110 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000e7d0 tx=0x7f2b1000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:56.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.271+0000 7f2b0ffff640 1 -- 172.21.15.45:0/2898947945 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1000eee0 con 0x7f2b180c6590 2023-11-28T21:51:56.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.271+0000 7f2b0ffff640 1 -- 172.21.15.45:0/2898947945 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1000c940 con 0x7f2b180c6590 2023-11-28T21:51:56.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.271+0000 7f2b0ffff640 1 -- 172.21.15.45:0/2898947945 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1000ccf0 con 0x7f2b180c6590 2023-11-28T21:51:56.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.271+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b180c4d90 con 0x7f2b180c6590 2023-11-28T21:51:56.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.271+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b180c5350 con 0x7f2b180c6590 2023-11-28T21:51:56.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.272+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b180733d0 con 0x7f2b180c6590 2023-11-28T21:51:56.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.273+0000 7f2b0ffff640 1 -- 172.21.15.45:0/2898947945 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 91) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f2b10013470 con 0x7f2b180c6590 2023-11-28T21:51:56.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.274+0000 7f2b0ffff640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2af806afb0 0x7f2af806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.274+0000 7f2b0ffff640 1 -- 172.21.15.45:0/2898947945 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2b10089df0 con 0x7f2b180c6590 2023-11-28T21:51:56.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.274+0000 7f2b177fe640 1 -- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2af806afb0 msgr2=0x7f2af806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2040970137 2023-11-28T21:51:56.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.274+0000 7f2b177fe640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2af806afb0 0x7f2af806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:56.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.279+0000 7f2b0ffff640 1 -- 172.21.15.45:0/2898947945 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b10054cd0 con 0x7f2b180c6590 2023-11-28T21:51:56.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.475+0000 7f2b177fe640 1 -- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2af806afb0 msgr2=0x7f2af806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2040970137 2023-11-28T21:51:56.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.475+0000 7f2b177fe640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2af806afb0 0x7f2af806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:51:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.477+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2b18072080 con 0x7f2b180c6590 2023-11-28T21:51:56.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.481+0000 7f2b0ffff640 1 -- 172.21.15.45:0/2898947945 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2b100170d0 con 0x7f2b180c6590 2023-11-28T21:51:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2af806afb0 msgr2=0x7f2af806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2af806afb0 0x7f2af806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b180c6590 msgr2=0x7f2b180b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b180c6590 0x7f2b180b4110 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000e7d0 tx=0x7f2b1000ec90 comp rx=0 tx=0).stop 2023-11-28T21:51:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 shutdown_connections 2023-11-28T21:51:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2af806afb0 0x7f2af806d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b180c6590 0x7f2b180b4110 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b180c3510 0x7f2b180c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 --2- 172.21.15.45:0/2898947945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b1806e120 0x7f2b181597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 >> 172.21.15.45:0/2898947945 conn(0x7f2b180cc900 msgr2=0x7f2b1806af90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 shutdown_connections 2023-11-28T21:51:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.485+0000 7f2b1ddde640 1 -- 172.21.15.45:0/2898947945 wait complete. 2023-11-28T21:51:56.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T21:51:56.521 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:56.520+0000 7f39c0e2e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T21:51:56.556 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:56.555+0000 7f697222e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T21:51:56.573 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:56.573+0000 7f39c0e2e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T21:51:56.615 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:56.614+0000 7f697222e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T21:51:56.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.712+0000 7f2ec1ce0640 1 Processor -- start 2023-11-28T21:51:56.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.713+0000 7f2ec1ce0640 1 -- start start 2023-11-28T21:51:56.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.714+0000 7f2ec1ce0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ebc099a40 0x7f2ebc06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.714+0000 7f2ec1ce0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc0d1fc0 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.714+0000 7f2ec1ce0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc06a910 con 0x7f2ebc0aff90 2023-11-28T21:51:56.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.714+0000 7f2ec1ce0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ebc0cb8a0 con 0x7f2ebc099a40 2023-11-28T21:51:56.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.714+0000 7f2ebb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ebc099a40 0x7f2ebc06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:56.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.715+0000 7f2ebb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2ebc099a40 0x7f2ebc06b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39188/0 (socket says 172.21.15.45:39188) 2023-11-28T21:51:56.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.715+0000 7f2ebb7fe640 1 -- 172.21.15.45:0/1566133816 learned_addr learned my addr 172.21.15.45:0/1566133816 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:56.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.715+0000 7f2ebb7fe640 1 -- 172.21.15.45:0/1566133816 >> v1:172.21.15.190:6789/0 conn(0x7f2ebc0aff90 legacy=0x7f2ebc099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:56.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.715+0000 7f2ebb7fe640 1 -- 172.21.15.45:0/1566133816 >> v1:172.21.15.45:6789/0 conn(0x7f2ebc0cbd30 legacy=0x7f2ebc0cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.715+0000 7f2ebb7fe640 1 -- 172.21.15.45:0/1566133816 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc06a630 con 0x7f2ebc099a40 2023-11-28T21:51:56.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.716+0000 7f2ebb7fe640 1 --2- 172.21.15.45:0/1566133816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc099a40 0x7f2ebc06b4d0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009870 tx=0x7f2eac02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815650168df0fa37 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:56.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.716+0000 7f2ebaffd640 1 -- 172.21.15.45:0/1566133816 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac031860 con 0x7f2ebc099a40 2023-11-28T21:51:56.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.716+0000 7f2ebaffd640 1 -- 172.21.15.45:0/1566133816 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eac0319c0 con 0x7f2ebc099a40 2023-11-28T21:51:56.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.716+0000 7f2ebaffd640 1 -- 172.21.15.45:0/1566133816 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac031d50 con 0x7f2ebc099a40 2023-11-28T21:51:56.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.717+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/1566133816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc099a40 msgr2=0x7f2ebc06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.717+0000 7f2ec1ce0640 1 --2- 172.21.15.45:0/1566133816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc099a40 0x7f2ebc06b4d0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009870 tx=0x7f2eac02b900 comp rx=0 tx=0).stop 2023-11-28T21:51:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.717+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/1566133816 shutdown_connections 2023-11-28T21:51:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.717+0000 7f2ec1ce0640 1 --2- 172.21.15.45:0/1566133816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc099a40 0x7f2ebc06b4d0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.717+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/1566133816 >> 172.21.15.45:0/1566133816 conn(0x7f2ebc0d1020 msgr2=0x7f2ebc0d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.717+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/1566133816 shutdown_connections 2023-11-28T21:51:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.717+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/1566133816 wait complete. 2023-11-28T21:51:56.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.720+0000 7f2ec1ce0640 1 Processor -- start 2023-11-28T21:51:56.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.720+0000 7f2ec1ce0640 1 -- start start 2023-11-28T21:51:56.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ec1ce0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2ebc099a40 0x7f2ebc14f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ec1ce0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc0aff90 0x7f2ebc0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ec1ce0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc0cbd30 0x7f2ebc14c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ec1ce0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc14fcc0 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ec1ce0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0c96e0 con 0x7f2ebc0aff90 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ec1ce0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0c99c0 con 0x7f2ebc099a40 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ebbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc0cbd30 0x7f2ebc14c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ebbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc0cbd30 0x7f2ebc14c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39192/0 (socket says 172.21.15.45:39192) 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ebbfff640 1 -- 172.21.15.45:0/2850909832 learned_addr learned my addr 172.21.15.45:0/2850909832 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.722+0000 7f2ebb7fe640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2ebc099a40 0x7f2ebc14f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2ebbfff640 1 -- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2ebc099a40 msgr2=0x7f2ebc14f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2ebbfff640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2ebc099a40 0x7f2ebc14f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2ebbfff640 1 -- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc0aff90 msgr2=0x7f2ebc0b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2ebbfff640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc0aff90 0x7f2ebc0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2ebbfff640 1 -- 172.21.15.45:0/2850909832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eac0094d0 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2ebbfff640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc0cbd30 0x7f2ebc14c0d0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400c6b0 tx=0x7f2eb400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2eb8ff9640 1 -- 172.21.15.45:0/2850909832 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4003b70 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2eb8ff9640 1 -- 172.21.15.45:0/2850909832 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb4003cf0 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2eb8ff9640 1 -- 172.21.15.45:0/2850909832 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4003ea0 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc0c9ca0 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.723+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc0ca260 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.724+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc0733d0 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.725+0000 7f2eb8ff9640 1 -- 172.21.15.45:0/2850909832 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 91) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f2eb4005310 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.726+0000 7f2eb8ff9640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2e9c06afb0 0x7f2e9c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:56.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.726+0000 7f2eb8ff9640 1 -- 172.21.15.45:0/2850909832 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2eb4089790 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.726+0000 7f2ebb7fe640 1 -- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2e9c06afb0 msgr2=0x7f2e9c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2040970137 2023-11-28T21:51:56.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.727+0000 7f2ebb7fe640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2e9c06afb0 0x7f2e9c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:56.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.731+0000 7f2eb8ff9640 1 -- 172.21.15.45:0/2850909832 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2eb40544a0 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.923+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f2ebc0d0180 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.927+0000 7f2eb8ff9640 1 -- 172.21.15.45:0/2850909832 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2eb4057ac0 con 0x7f2ebc0cbd30 2023-11-28T21:51:56.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.928+0000 7f2ebb7fe640 1 -- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2e9c06afb0 msgr2=0x7f2e9c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2040970137 2023-11-28T21:51:56.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.928+0000 7f2ebb7fe640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2e9c06afb0 0x7f2e9c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:51:56.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2e9c06afb0 msgr2=0x7f2e9c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2e9c06afb0 0x7f2e9c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc0cbd30 msgr2=0x7f2ebc14c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc0cbd30 0x7f2ebc14c0d0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400c6b0 tx=0x7f2eb400cb70 comp rx=0 tx=0).stop 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 shutdown_connections 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f2e9c06afb0 0x7f2e9c06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ebc0cbd30 0x7f2ebc14c0d0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc0aff90 0x7f2ebc0b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 --2- 172.21.15.45:0/2850909832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2ebc099a40 0x7f2ebc14f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 >> 172.21.15.45:0/2850909832 conn(0x7f2ebc0d1020 msgr2=0x7f2ebc06e980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:56.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 shutdown_connections 2023-11-28T21:51:56.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:56.930+0000 7f2ec1ce0640 1 -- 172.21.15.45:0/2850909832 wait complete. 2023-11-28T21:51:56.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:51:56.951 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-11-28T21:51:57.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2023-11-28T21:51:57.023 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.022+0000 7f39c0e2e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T21:51:57.076 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.075+0000 7f39c0e2e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T21:51:57.096 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.095+0000 7f697222e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T21:51:57.125 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.124+0000 7f39c0e2e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T21:51:57.151 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.150+0000 7f697222e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T21:51:57.186 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.185+0000 7f39c0e2e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T21:51:57.205 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.204+0000 7f697222e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T21:51:57.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.217+0000 7f950f59e640 1 Processor -- start 2023-11-28T21:51:57.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.217+0000 7f950f59e640 1 -- start start 2023-11-28T21:51:57.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.219+0000 7f950f59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f951015d580 0x7f95100cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:57.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.219+0000 7f950f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f95100d1e40 0x7f95100ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:57.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.219+0000 7f950f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95100d0490 con 0x7f95100b2500 2023-11-28T21:51:57.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.219+0000 7f950f59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95100d1360 con 0x7f951015d580 2023-11-28T21:51:57.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.219+0000 7f950f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95100d1640 con 0x7f95100d1e40 2023-11-28T21:51:57.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.219+0000 7f950e59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f951015d580 0x7f95100cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:57.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.219+0000 7f950e59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f951015d580 0x7f95100cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39200/0 (socket says 172.21.15.45:39200) 2023-11-28T21:51:57.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.219+0000 7f950e59c640 1 -- 172.21.15.45:0/3412753267 learned_addr learned my addr 172.21.15.45:0/3412753267 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:57.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.219+0000 7f950ed9d640 1 --1- 172.21.15.45:0/3412753267 >> v1:172.21.15.45:6789/0 conn(0x7f95100b2500 0x7f95100b4bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36400/0 (socket says 172.21.15.45:36400) 2023-11-28T21:51:57.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.220+0000 7f950e59c640 1 -- 172.21.15.45:0/3412753267 >> v2:172.21.15.190:3300/0 conn(0x7f95100d1e40 msgr2=0x7f95100ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:51:57.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.220+0000 7f9507fff640 1 --2- 172.21.15.45:0/3412753267 >> v2:172.21.15.190:3300/0 conn(0x7f95100d1e40 0x7f95100ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:57.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.220+0000 7f950e59c640 1 --2- 172.21.15.45:0/3412753267 >> v2:172.21.15.190:3300/0 conn(0x7f95100d1e40 0x7f95100ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:57.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.220+0000 7f950e59c640 1 -- 172.21.15.45:0/3412753267 >> v1:172.21.15.45:6789/0 conn(0x7f95100b2500 legacy=0x7f95100b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:57.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.220+0000 7f950e59c640 1 -- 172.21.15.45:0/3412753267 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95100d1920 con 0x7f951015d580 2023-11-28T21:51:57.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.220+0000 7f9507fff640 1 --2- 172.21.15.45:0/3412753267 >> v2:172.21.15.190:3300/0 conn(0x7f95100d1e40 0x7f95100ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:51:57.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.221+0000 7f950e59c640 1 --2- 172.21.15.45:0/3412753267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 0x7f95100cfb00 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f951008c500 tx=0x7f950002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7188f225e87e23 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:57.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.221+0000 7f950dd9b640 1 -- 172.21.15.45:0/3412753267 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500031880 con 0x7f951015d580 2023-11-28T21:51:57.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.221+0000 7f950dd9b640 1 -- 172.21.15.45:0/3412753267 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95000319e0 con 0x7f951015d580 2023-11-28T21:51:57.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.221+0000 7f950dd9b640 1 -- 172.21.15.45:0/3412753267 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500031d70 con 0x7f951015d580 2023-11-28T21:51:57.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.222+0000 7f950f59e640 1 -- 172.21.15.45:0/3412753267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 msgr2=0x7f95100cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:57.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.222+0000 7f950f59e640 1 --2- 172.21.15.45:0/3412753267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 0x7f95100cfb00 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f951008c500 tx=0x7f950002b920 comp rx=0 tx=0).stop 2023-11-28T21:51:57.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.222+0000 7f950f59e640 1 -- 172.21.15.45:0/3412753267 shutdown_connections 2023-11-28T21:51:57.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.222+0000 7f950f59e640 1 --2- 172.21.15.45:0/3412753267 >> v2:172.21.15.190:3300/0 conn(0x7f95100d1e40 0x7f95100ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:57.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.222+0000 7f950f59e640 1 --2- 172.21.15.45:0/3412753267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 0x7f95100cfb00 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:57.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.222+0000 7f950f59e640 1 -- 172.21.15.45:0/3412753267 >> 172.21.15.45:0/3412753267 conn(0x7f951006a500 msgr2=0x7f951006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:51:57.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.222+0000 7f950f59e640 1 -- 172.21.15.45:0/3412753267 shutdown_connections 2023-11-28T21:51:57.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.222+0000 7f950f59e640 1 -- 172.21.15.45:0/3412753267 wait complete. 2023-11-28T21:51:57.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.224+0000 7f950f59e640 1 Processor -- start 2023-11-28T21:51:57.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.224+0000 7f950f59e640 1 -- start start 2023-11-28T21:51:57.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.225+0000 7f950f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95100b2500 0x7f9510150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.225+0000 7f950f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95100d1e40 0x7f951015ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.225+0000 7f950f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 0x7f951015f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.225+0000 7f950f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9510150d40 con 0x7f951015d580 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.225+0000 7f950f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951015fc50 con 0x7f95100d1e40 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.225+0000 7f950f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951015ff30 con 0x7f95100b2500 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.225+0000 7f950ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 0x7f951015f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.225+0000 7f950ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 0x7f951015f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39204/0 (socket says 172.21.15.45:39204) 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.225+0000 7f950ed9d640 1 -- 172.21.15.45:0/3375905961 learned_addr learned my addr 172.21.15.45:0/3375905961 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f950e59c640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95100b2500 0x7f9510150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f9507fff640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95100d1e40 0x7f951015ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f950ed9d640 1 -- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95100b2500 msgr2=0x7f9510150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:57.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f950ed9d640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95100b2500 0x7f9510150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f950ed9d640 1 -- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95100d1e40 msgr2=0x7f951015ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f950ed9d640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95100d1e40 0x7f951015ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f950ed9d640 1 -- 172.21.15.45:0/3375905961 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95000094d0 con 0x7f951015d580 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f950e59c640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95100b2500 0x7f9510150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f950ed9d640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 0x7f951015f280 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f950800c730 tx=0x7f950800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950800ce40 con 0x7f951015d580 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950800a5e0 con 0x7f951015d580 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950800a990 con 0x7f951015d580 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.226+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9510160210 con 0x7f951015d580 2023-11-28T21:51:57.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.227+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9510151810 con 0x7f951015d580 2023-11-28T21:51:57.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.228+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9510071b30 con 0x7f951015d580 2023-11-28T21:51:57.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.229+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 91) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f950800aaf0 con 0x7f951015d580 2023-11-28T21:51:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.230+0000 7f9506ffd640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 0x7f94f006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:51:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.231+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9508088fe0 con 0x7f951015d580 2023-11-28T21:51:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.231+0000 7f950e59c640 1 -- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 msgr2=0x7f94f006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2040970137 2023-11-28T21:51:57.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.231+0000 7f950e59c640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 0x7f94f006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:51:57.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.234+0000 7f39c0e2e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T21:51:57.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.235+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9508053e50 con 0x7f951015d580 2023-11-28T21:51:57.271 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.269+0000 7f697222e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T21:51:57.323 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.322+0000 7f697222e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T21:51:57.346 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.345+0000 7f39c0e2e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T21:51:57.408 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.407+0000 7f39c0e2e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T21:51:57.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.432+0000 7f950e59c640 1 -- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 msgr2=0x7f94f006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2040970137 2023-11-28T21:51:57.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.432+0000 7f950e59c640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 0x7f94f006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:51:57.441 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.440+0000 7f697222e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T21:51:57.456 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.455+0000 7f39c0e2e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T21:51:57.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.455+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 --> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f95100d0250 con 0x7f94f006afb0 2023-11-28T21:51:57.506 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.505+0000 7f697222e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T21:51:57.514 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.513+0000 7f39c0e2e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T21:51:57.558 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.557+0000 7f697222e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T21:51:57.563 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.562+0000 7f39c0e2e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T21:51:57.618 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.616+0000 7f697222e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T21:51:57.670 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:57.669+0000 7f697222e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T21:51:57.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.833+0000 7f950e59c640 1 -- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 msgr2=0x7f94f006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2040970137 2023-11-28T21:51:57.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:57.833+0000 7f950e59c640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 0x7f94f006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T21:51:57.943 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:57.941+0000 7f39c0e2e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T21:51:58.070 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:58.069+0000 7f697222e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T21:51:58.100 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:58.099+0000 7f39c0e2e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T21:51:58.209 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:58.208+0000 7f39c0e2e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T21:51:58.236 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:58.235+0000 7f697222e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T21:51:58.259 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:58.258+0000 7f39c0e2e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T21:51:58.350 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:58.348+0000 7f697222e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T21:51:58.366 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:58.365+0000 7f39c0e2e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:51:58.402 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:58.400+0000 7f697222e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T21:51:58.472 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:58.471+0000 7f39c0e2e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T21:51:58.515 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:58.513+0000 7f697222e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:51:58.627 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:58.625+0000 7f697222e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T21:51:58.627 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:58.626+0000 7f39c0e2e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:51:58.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:58.634+0000 7f950e59c640 1 -- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 msgr2=0x7f94f006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/2040970137 2023-11-28T21:51:58.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:51:58.634+0000 7f950e59c640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 0x7f94f006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T21:51:58.676 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:58.675+0000 7f39c0e2e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T21:51:58.791 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:58.789+0000 7f697222e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:51:58.795 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:58.793+0000 7f39c0e2e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T21:51:58.842 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:58.841+0000 7f697222e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T21:51:58.967 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:58.966+0000 7f697222e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T21:51:59.597 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:59.596+0000 7f39c0e2e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T21:51:59.855 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:51:59.854+0000 7f697222e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T21:51:59.894 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:59.893+0000 7f39c0e2e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T21:51:59.951 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:51:59.950+0000 7f39c0e2e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T21:52:00.065 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.064+0000 7f39c0e2e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T21:52:00.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:00.090+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 92) v1 ==== 88959+0+0 (secure 0 0 0) 0x7f9508054220 con 0x7f951015d580 2023-11-28T21:52:00.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:00.090+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 msgr2=0x7f94f006d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:00.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:00.090+0000 7f9506ffd640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/2040970137,v1:172.21.15.190:6833/2040970137] conn(0x7f94f006afb0 0x7f94f006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:00.178 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:00.177+0000 7f697222e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.233+0000 7f39401c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.233+0000 7f39401c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.233+0000 7f39401c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.233+0000 7f39401c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.233+0000 7f39401c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.234+0000 7f393d1c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.234+0000 7f393d1c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.234+0000 7f393d1c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.234+0000 7f393d1c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:00.234+0000 7f393d1c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:00.242 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:00.241+0000 7f697222e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T21:52:00.361 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:00.360+0000 7f697222e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T21:52:01.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.097+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 93) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f9508087320 con 0x7f951015d580 2023-11-28T21:52:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.098+0000 7f9506ffd640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f94f0035850 0x7f94f00753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.098+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 --> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f94f0035c30 con 0x7f94f0035850 2023-11-28T21:52:01.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.105+0000 7f950e59c640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f94f0035850 0x7f94f00753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:01.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.109+0000 7f950e59c640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f94f0035850 0x7f94f00753e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9500036000 tx=0x7f950003c040 comp rx=0 tx=0).ready entity=mgr.8023 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:01.253 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.252+0000 7f9506ffd640 1 -- 172.21.15.45:0/3375905961 <== mgr.8023 v2:172.21.15.190:6832/1285100946 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f94f0035c30 con 0x7f94f0035850 2023-11-28T21:52:01.253 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1 2023-11-28T21:52:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.255+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f94f0035850 msgr2=0x7f94f00753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.255+0000 7f950f59e640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f94f0035850 0x7f94f00753e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9500036000 tx=0x7f950003c040 comp rx=0 tx=0).stop 2023-11-28T21:52:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.255+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 msgr2=0x7f951015f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:01.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.255+0000 7f950f59e640 1 --2- 172.21.15.45:0/3375905961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f951015d580 0x7f951015f280 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f950800c730 tx=0x7f950800cbf0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.255+0000 7f950ed9d640 1 -- 172.21.15.45:0/3375905961 reap_dead start 2023-11-28T21:52:01.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.255+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 shutdown_connections 2023-11-28T21:52:01.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.255+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 >> 172.21.15.45:0/3375905961 conn(0x7f951006a500 msgr2=0x7f951006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:01.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.256+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 shutdown_connections 2023-11-28T21:52:01.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.256+0000 7f950f59e640 1 -- 172.21.15.45:0/3375905961 wait complete. 2023-11-28T21:52:01.278 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-11-28T21:52:01.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-11-28T21:52:01.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.505+0000 7fe106ac2640 1 Processor -- start 2023-11-28T21:52:01.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.506+0000 7fe106ac2640 1 -- start start 2023-11-28T21:52:01.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.507+0000 7fe106ac2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe10015e840 0x7fe10006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.507+0000 7fe106ac2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe10015f720 0x7fe1000717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.507+0000 7fe106ac2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10015ecd0 con 0x7fe10006a500 2023-11-28T21:52:01.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.507+0000 7fe106ac2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe100071ce0 con 0x7fe10015e840 2023-11-28T21:52:01.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.507+0000 7fe106ac2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10015f290 con 0x7fe10015f720 2023-11-28T21:52:01.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe10505f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe10006a500 0x7fe1000cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36404/0 (socket says 172.21.15.45:36404) 2023-11-28T21:52:01.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe10505f640 1 -- 172.21.15.45:0/423254674 learned_addr learned my addr 172.21.15.45:0/423254674 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:01.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0fffff640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe10015f720 0x7fe1000717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:01.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe10485e640 1 --2- 172.21.15.45:0/423254674 >> v2:172.21.15.190:3300/0 conn(0x7fe10015e840 0x7fe10006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:01.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855590504 0 0) 0x7fe10015ecd0 con 0x7fe10006a500 2023-11-28T21:52:01.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ec003520 con 0x7fe10006a500 2023-11-28T21:52:01.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3473611378 0 0) 0x7fe0ec003520 con 0x7fe10006a500 2023-11-28T21:52:01.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe10015f720 msgr2=0x7fe1000717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:01.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0ff7fe640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe10015f720 0x7fe1000717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 >> v2:172.21.15.190:3300/0 conn(0x7fe10015e840 msgr2=0x7fe10006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:01.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0ff7fe640 1 --2- 172.21.15.45:0/423254674 >> v2:172.21.15.190:3300/0 conn(0x7fe10015e840 0x7fe10006c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10015fb60 con 0x7fe10006a500 2023-11-28T21:52:01.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.508+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fe0f8001670 con 0x7fe10006a500 2023-11-28T21:52:01.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 >> v1:172.21.15.45:6789/0 conn(0x7fe10006a500 legacy=0x7fe1000cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe0ff7fe640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe10015f720 0x7fe0ec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe0ff7fe640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0ec003ef0 0x7fe0ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe0ff7fe640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ec004d00 0x7fe0ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10015f290 con 0x7fe0ec004d00 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec00b980 con 0x7fe0ec003ef0 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec00bac0 con 0x7fe10015f720 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe10505f640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ec004d00 0x7fe0ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe10485e640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe10015f720 0x7fe0ec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe10505f640 1 -- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe10015f720 msgr2=0x7fe0ec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe10505f640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe10015f720 0x7fe0ec003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe0fffff640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0ec003ef0 0x7fe0ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe10505f640 1 -- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0ec003ef0 msgr2=0x7fe0ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:01.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe10505f640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0ec003ef0 0x7fe0ec0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.509+0000 7fe10505f640 1 -- 172.21.15.45:0/423254674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec003520 con 0x7fe0ec004d00 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.511+0000 7fe10505f640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ec004d00 0x7fe0ec005180 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800c240 tx=0x7fe0ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5a6d0197ebf7ba server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.511+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f8005ad0 con 0x7fe0ec004d00 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.511+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f8005c30 con 0x7fe0ec004d00 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.511+0000 7fe0ff7fe640 1 -- 172.21.15.45:0/423254674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f802cae0 con 0x7fe0ec004d00 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.511+0000 7fe106ac2640 1 -- 172.21.15.45:0/423254674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ec004d00 msgr2=0x7fe0ec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.511+0000 7fe106ac2640 1 --2- 172.21.15.45:0/423254674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0ec004d00 0x7fe0ec005180 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800c240 tx=0x7fe0ec0057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.511+0000 7fe10505f640 1 -- 172.21.15.45:0/423254674 reap_dead start 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.511+0000 7fe106ac2640 1 -- 172.21.15.45:0/423254674 shutdown_connections 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.511+0000 7fe106ac2640 1 -- 172.21.15.45:0/423254674 >> 172.21.15.45:0/423254674 conn(0x7fe1000ced60 msgr2=0x7fe1000cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.512+0000 7fe106ac2640 1 -- 172.21.15.45:0/423254674 shutdown_connections 2023-11-28T21:52:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.512+0000 7fe106ac2640 1 -- 172.21.15.45:0/423254674 wait complete. 2023-11-28T21:52:01.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.513+0000 7fe106ac2640 1 Processor -- start 2023-11-28T21:52:01.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.514+0000 7fe106ac2640 1 -- start start 2023-11-28T21:52:01.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.514+0000 7fe106ac2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0ec004d00 0x7fe10014ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe106ac2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1000c5e70 0x7fe1000b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe106ac2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1001f0690 0x7fe1000b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe106ac2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000c63e0 con 0x7fe1000c5e70 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe106ac2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000b4300 con 0x7fe1001f0690 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe106ac2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe100153210 con 0x7fe0ec004d00 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe0fffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1000c5e70 0x7fe1000b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe0fffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1000c5e70 0x7fe1000b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39226/0 (socket says 172.21.15.45:39226) 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe0fffff640 1 -- 172.21.15.45:0/3211464232 learned_addr learned my addr 172.21.15.45:0/3211464232 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe0fffff640 1 -- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0ec004d00 msgr2=0x7fe10014ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe0fffff640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0ec004d00 0x7fe10014ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe0fffff640 1 -- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1001f0690 msgr2=0x7fe1000b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe0fffff640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1001f0690 0x7fe1000b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.515+0000 7fe0fffff640 1 -- 172.21.15.45:0/3211464232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1000b2ac0 con 0x7fe1000c5e70 2023-11-28T21:52:01.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.516+0000 7fe0fffff640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1000c5e70 0x7fe1000b3880 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8010860 tx=0x7fe0e8010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:01.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.516+0000 7fe0fd7fa640 1 -- 172.21.15.45:0/3211464232 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8009610 con 0x7fe1000c5e70 2023-11-28T21:52:01.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.516+0000 7fe0fd7fa640 1 -- 172.21.15.45:0/3211464232 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e800eae0 con 0x7fe1000c5e70 2023-11-28T21:52:01.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.516+0000 7fe0fd7fa640 1 -- 172.21.15.45:0/3211464232 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e800ee70 con 0x7fe1000c5e70 2023-11-28T21:52:01.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.516+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1000b2da0 con 0x7fe1000c5e70 2023-11-28T21:52:01.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.516+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1000b3360 con 0x7fe1000c5e70 2023-11-28T21:52:01.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.517+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1000733d0 con 0x7fe1000c5e70 2023-11-28T21:52:01.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.518+0000 7fe0fd7fa640 1 -- 172.21.15.45:0/3211464232 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 93) v1 ==== 89086+0+0 (secure 0 0 0) 0x7fe0e801c400 con 0x7fe1000c5e70 2023-11-28T21:52:01.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.519+0000 7fe0fd7fa640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fe0e006ace0 0x7fe0e006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:01.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.519+0000 7fe0fd7fa640 1 -- 172.21.15.45:0/3211464232 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe0e8089b70 con 0x7fe1000c5e70 2023-11-28T21:52:01.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.520+0000 7fe10485e640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fe0e006ace0 0x7fe0e006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:01.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.521+0000 7fe10485e640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fe0e006ace0 0x7fe0e006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009fd0 tx=0x7fe0f0009750 comp rx=0 tx=0).ready entity=mgr.8023 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.524+0000 7fe0fd7fa640 1 -- 172.21.15.45:0/3211464232 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0e8054c80 con 0x7fe1000c5e70 2023-11-28T21:52:01.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.739+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 --> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe10015ef20 con 0x7fe0e006ace0 2023-11-28T21:52:01.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:52:01.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:52:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.892+0000 7fe0fd7fa640 1 -- 172.21.15.45:0/3211464232 <== mgr.8023 v2:172.21.15.190:6832/1285100946 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe0f000f050 con 0x7fe0e006ace0 2023-11-28T21:52:01.893 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path snap_test_dir1 2023-11-28T21:52:01.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fe0e006ace0 msgr2=0x7fe0e006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:01.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fe0e006ace0 0x7fe0e006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009fd0 tx=0x7fe0f0009750 comp rx=0 tx=0).stop 2023-11-28T21:52:01.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1000c5e70 msgr2=0x7fe1000b3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:01.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1000c5e70 0x7fe1000b3880 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8010860 tx=0x7fe0e8010d20 comp rx=0 tx=0).stop 2023-11-28T21:52:01.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 shutdown_connections 2023-11-28T21:52:01.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1001f0690 0x7fe1000b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fe0e006ace0 0x7fe0e006d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1000c5e70 0x7fe1000b3880 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 --2- 172.21.15.45:0/3211464232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0ec004d00 0x7fe10014ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:01.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 >> 172.21.15.45:0/3211464232 conn(0x7fe1000ced60 msgr2=0x7fe1000ac4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:01.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 shutdown_connections 2023-11-28T21:52:01.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:01.895+0000 7fe106ac2640 1 -- 172.21.15.45:0/3211464232 wait complete. 2023-11-28T21:52:01.916 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-11-28T21:52:01.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-11-28T21:52:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.175+0000 7f9d0b407640 1 Processor -- start 2023-11-28T21:52:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.175+0000 7f9d0b407640 1 -- start start 2023-11-28T21:52:02.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.176+0000 7f9d0b407640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.176+0000 7f9d0b407640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9d0415f720 0x7f9d040717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.177+0000 7f9d0b407640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0415ecd0 con 0x7f9d0406a500 2023-11-28T21:52:02.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.177+0000 7f9d0b407640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d04071ce0 con 0x7f9d0415f720 2023-11-28T21:52:02.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.177+0000 7f9d0b407640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0415f290 con 0x7f9d0415e840 2023-11-28T21:52:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.177+0000 7f9d091a3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.177+0000 7f9d091a3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0406c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:35558/0 (socket says 172.21.15.45:35558) 2023-11-28T21:52:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.177+0000 7f9d091a3640 1 -- 172.21.15.45:0/726841856 learned_addr learned my addr 172.21.15.45:0/726841856 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.178+0000 7f9d089a2640 1 --2- 172.21.15.45:0/726841856 >> v2:172.21.15.190:3300/0 conn(0x7f9d0415f720 0x7f9d040717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.178+0000 7f9d091a3640 1 -- 172.21.15.45:0/726841856 >> v2:172.21.15.190:3300/0 conn(0x7f9d0415f720 msgr2=0x7f9d040717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.178+0000 7f9d091a3640 1 --2- 172.21.15.45:0/726841856 >> v2:172.21.15.190:3300/0 conn(0x7f9d0415f720 0x7f9d040717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.178+0000 7f9d091a3640 1 -- 172.21.15.45:0/726841856 >> v1:172.21.15.190:6789/0 conn(0x7f9d0406a500 legacy=0x7f9d040cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.178+0000 7f9d091a3640 1 -- 172.21.15.45:0/726841856 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0415fb60 con 0x7f9d0415e840 2023-11-28T21:52:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.180+0000 7f9d091a3640 1 --2- 172.21.15.45:0/726841856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0406c350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9d000098a0 tx=0x7f9d0002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bccad80e5f34ae89 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:02.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.180+0000 7f9cfbfff640 1 -- 172.21.15.45:0/726841856 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00030840 con 0x7f9d0415e840 2023-11-28T21:52:02.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.180+0000 7f9cfbfff640 1 -- 172.21.15.45:0/726841856 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d000309a0 con 0x7f9d0415e840 2023-11-28T21:52:02.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.180+0000 7f9cfbfff640 1 -- 172.21.15.45:0/726841856 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00030d30 con 0x7f9d0415e840 2023-11-28T21:52:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.181+0000 7f9d0b407640 1 -- 172.21.15.45:0/726841856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 msgr2=0x7f9d0406c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.181+0000 7f9d0b407640 1 --2- 172.21.15.45:0/726841856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0406c350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9d000098a0 tx=0x7f9d0002b980 comp rx=0 tx=0).stop 2023-11-28T21:52:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.181+0000 7f9d0b407640 1 -- 172.21.15.45:0/726841856 shutdown_connections 2023-11-28T21:52:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.181+0000 7f9d0b407640 1 --2- 172.21.15.45:0/726841856 >> v2:172.21.15.190:3300/0 conn(0x7f9d0415f720 0x7f9d040717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.181+0000 7f9d0b407640 1 --2- 172.21.15.45:0/726841856 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0406c350 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.181+0000 7f9d0b407640 1 -- 172.21.15.45:0/726841856 >> 172.21.15.45:0/726841856 conn(0x7f9d040ced60 msgr2=0x7f9d040cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.181+0000 7f9d0b407640 1 -- 172.21.15.45:0/726841856 shutdown_connections 2023-11-28T21:52:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.181+0000 7f9d0b407640 1 -- 172.21.15.45:0/726841856 wait complete. 2023-11-28T21:52:02.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.183+0000 7f9d0b407640 1 Processor -- start 2023-11-28T21:52:02.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.183+0000 7f9d0b407640 1 -- start start 2023-11-28T21:52:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.184+0000 7f9d0b407640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d0406a500 0x7f9d04099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.184+0000 7f9d0b407640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0409b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.184+0000 7f9d0b407640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d0415f720 0x7f9d040b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.184+0000 7f9d0b407640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0409ad30 con 0x7f9d0406a500 2023-11-28T21:52:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.184+0000 7f9d0b407640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0409a120 con 0x7f9d0415f720 2023-11-28T21:52:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.184+0000 7f9d0b407640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0409a400 con 0x7f9d0415e840 2023-11-28T21:52:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d091a3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d0406a500 0x7f9d04099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d091a3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d0406a500 0x7f9d04099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39234/0 (socket says 172.21.15.45:39234) 2023-11-28T21:52:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d091a3640 1 -- 172.21.15.45:0/1022325496 learned_addr learned my addr 172.21.15.45:0/1022325496 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d089a2640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0409b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d099a4640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d0415f720 0x7f9d040b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d091a3640 1 -- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 msgr2=0x7f9d0409b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d091a3640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0409b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d091a3640 1 -- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d0415f720 msgr2=0x7f9d040b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d091a3640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d0415f720 0x7f9d040b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d091a3640 1 -- 172.21.15.45:0/1022325496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d000094d0 con 0x7f9d0406a500 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d099a4640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d0415f720 0x7f9d040b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.185+0000 7f9d089a2640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d0415e840 0x7f9d0409b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.186+0000 7f9d091a3640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d0406a500 0x7f9d04099a40 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f9d000097c0 tx=0x7f9d00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.186+0000 7f9cf9ffb640 1 -- 172.21.15.45:0/1022325496 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00047070 con 0x7f9d0406a500 2023-11-28T21:52:02.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.186+0000 7f9cf9ffb640 1 -- 172.21.15.45:0/1022325496 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d00039a90 con 0x7f9d0406a500 2023-11-28T21:52:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.186+0000 7f9cf9ffb640 1 -- 172.21.15.45:0/1022325496 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00039e20 con 0x7f9d0406a500 2023-11-28T21:52:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.186+0000 7f9d0b407640 1 -- 172.21.15.45:0/1022325496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0409a6e0 con 0x7f9d0406a500 2023-11-28T21:52:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.186+0000 7f9d0b407640 1 -- 172.21.15.45:0/1022325496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d04157e40 con 0x7f9d0406a500 2023-11-28T21:52:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.187+0000 7f9d0b407640 1 -- 172.21.15.45:0/1022325496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d040733d0 con 0x7f9d0406a500 2023-11-28T21:52:02.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.188+0000 7f9cf9ffb640 1 -- 172.21.15.45:0/1022325496 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 94) v1 ==== 89351+0+0 (secure 0 0 0) 0x7f9d0004c410 con 0x7f9d0406a500 2023-11-28T21:52:02.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.189+0000 7f9cf9ffb640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f9ce806afb0 0x7f9ce806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.189+0000 7f9cf9ffb640 1 -- 172.21.15.45:0/1022325496 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f9d0003e070 con 0x7f9d0406a500 2023-11-28T21:52:02.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.190+0000 7f9d089a2640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f9ce806afb0 0x7f9ce806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.191+0000 7f9d089a2640 1 --2- 172.21.15.45:0/1022325496 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f9ce806afb0 0x7f9ce806d470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9d040c5ee0 tx=0x7f9cf4008040 comp rx=0 tx=0).ready entity=mgr.8023 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:02.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.194+0000 7f9cf9ffb640 1 -- 172.21.15.45:0/1022325496 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d00076b50 con 0x7f9d0406a500 2023-11-28T21:52:02.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.416+0000 7f9d0b407640 1 -- 172.21.15.45:0/1022325496 --> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d0415ef20 con 0x7f9ce806afb0 2023-11-28T21:52:02.433 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:02.432+0000 7f39471d7640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-11-28T21:52:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.432+0000 7f9cf9ffb640 1 -- 172.21.15.45:0/1022325496 <== mgr.8023 v2:172.21.15.190:6832/1285100946 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f9cf4009760 con 0x7f9ce806afb0 2023-11-28T21:52:02.434 INFO:teuthology.orchestra.run.smithi045.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-11-28T21:52:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T21:52:02.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:02.449 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-11-28T21:52:02.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T21:52:02.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.737+0000 7f0a57fff640 1 Processor -- start 2023-11-28T21:52:02.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.737+0000 7f0a57fff640 1 -- start start 2023-11-28T21:52:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.739+0000 7f0a57fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0a5806e060 0x7f0a58071f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.739+0000 7f0a57fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a580ae4b0 0x7f0a580b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.739+0000 7f0a57fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a58162bb0 con 0x7f0a58161770 2023-11-28T21:52:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.739+0000 7f0a57fff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a58161fa0 con 0x7f0a5806e060 2023-11-28T21:52:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.739+0000 7f0a57fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58162280 con 0x7f0a580ae4b0 2023-11-28T21:52:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.740+0000 7f0a577fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a580ae4b0 0x7f0a580b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.740+0000 7f0a56ffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0a58161770 0x7f0a5806d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:50296/0 (socket says 172.21.15.45:50296) 2023-11-28T21:52:02.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.740+0000 7f0a56ffd640 1 -- 172.21.15.45:0/2025530651 learned_addr learned my addr 172.21.15.45:0/2025530651 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:02.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.740+0000 7f0a567fc640 1 --2- 172.21.15.45:0/2025530651 >> v2:172.21.15.190:3300/0 conn(0x7f0a5806e060 0x7f0a58071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.741+0000 7f0a577fe640 1 -- 172.21.15.45:0/2025530651 >> v2:172.21.15.190:3300/0 conn(0x7f0a5806e060 msgr2=0x7f0a58071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:02.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.741+0000 7f0a577fe640 1 --2- 172.21.15.45:0/2025530651 >> v2:172.21.15.190:3300/0 conn(0x7f0a5806e060 0x7f0a58071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.741+0000 7f0a577fe640 1 -- 172.21.15.45:0/2025530651 >> v1:172.21.15.190:6789/0 conn(0x7f0a58161770 legacy=0x7f0a5806d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.741+0000 7f0a577fe640 1 -- 172.21.15.45:0/2025530651 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a58162560 con 0x7f0a580ae4b0 2023-11-28T21:52:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.741+0000 7f0a55ffb640 1 -- 172.21.15.45:0/2025530651 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751580279 0 0) 0x7f0a58162bb0 con 0x7f0a58161770 2023-11-28T21:52:02.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.743+0000 7f0a577fe640 1 --2- 172.21.15.45:0/2025530651 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a580ae4b0 0x7f0a580b1d00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009d90 tx=0x7f0a5002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42a3b383086d46c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:02.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.743+0000 7f0a55ffb640 1 -- 172.21.15.45:0/2025530651 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5002cc50 con 0x7f0a580ae4b0 2023-11-28T21:52:02.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.743+0000 7f0a55ffb640 1 -- 172.21.15.45:0/2025530651 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5002cdb0 con 0x7f0a580ae4b0 2023-11-28T21:52:02.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.743+0000 7f0a55ffb640 1 -- 172.21.15.45:0/2025530651 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50034660 con 0x7f0a580ae4b0 2023-11-28T21:52:02.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.744+0000 7f0a57fff640 1 -- 172.21.15.45:0/2025530651 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a580ae4b0 msgr2=0x7f0a580b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.744+0000 7f0a57fff640 1 --2- 172.21.15.45:0/2025530651 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a580ae4b0 0x7f0a580b1d00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009d90 tx=0x7f0a5002bea0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.744+0000 7f0a57fff640 1 -- 172.21.15.45:0/2025530651 shutdown_connections 2023-11-28T21:52:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.744+0000 7f0a57fff640 1 --2- 172.21.15.45:0/2025530651 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a580ae4b0 0x7f0a580b1d00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.744+0000 7f0a57fff640 1 --2- 172.21.15.45:0/2025530651 >> v2:172.21.15.190:3300/0 conn(0x7f0a5806e060 0x7f0a58071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.744+0000 7f0a57fff640 1 -- 172.21.15.45:0/2025530651 >> 172.21.15.45:0/2025530651 conn(0x7f0a580cecd0 msgr2=0x7f0a580cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.744+0000 7f0a57fff640 1 -- 172.21.15.45:0/2025530651 shutdown_connections 2023-11-28T21:52:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.744+0000 7f0a57fff640 1 -- 172.21.15.45:0/2025530651 wait complete. 2023-11-28T21:52:02.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.747+0000 7f0a57fff640 1 Processor -- start 2023-11-28T21:52:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.747+0000 7f0a57fff640 1 -- start start 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.748+0000 7f0a57fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a5806e060 0x7f0a5814f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.748+0000 7f0a57fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a580ae4b0 0x7f0a58160120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.748+0000 7f0a57fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a58161770 0x7f0a580b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.748+0000 7f0a57fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5814d180 con 0x7f0a580ae4b0 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.748+0000 7f0a57fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5814c870 con 0x7f0a58161770 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.748+0000 7f0a57fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58160b00 con 0x7f0a5806e060 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.748+0000 7f0a567fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a580ae4b0 0x7f0a58160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.748+0000 7f0a567fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a580ae4b0 0x7f0a58160120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39238/0 (socket says 172.21.15.45:39238) 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.749+0000 7f0a567fc640 1 -- 172.21.15.45:0/1880193125 learned_addr learned my addr 172.21.15.45:0/1880193125 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.749+0000 7f0a56ffd640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a5806e060 0x7f0a5814f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.749+0000 7f0a567fc640 1 -- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a5806e060 msgr2=0x7f0a5814f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.749+0000 7f0a567fc640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a5806e060 0x7f0a5814f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.749+0000 7f0a577fe640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a58161770 0x7f0a580b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.749+0000 7f0a567fc640 1 -- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a58161770 msgr2=0x7f0a580b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.749+0000 7f0a567fc640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a58161770 0x7f0a580b2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.749+0000 7f0a567fc640 1 -- 172.21.15.45:0/1880193125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a50009a70 con 0x7f0a580ae4b0 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.749+0000 7f0a567fc640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a580ae4b0 0x7f0a58160120 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00e930 tx=0x7f0a4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.750+0000 7f0a3f7fe640 1 -- 172.21.15.45:0/1880193125 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c009690 con 0x7f0a580ae4b0 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.750+0000 7f0a3f7fe640 1 -- 172.21.15.45:0/1880193125 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4c00ca00 con 0x7f0a580ae4b0 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.750+0000 7f0a3f7fe640 1 -- 172.21.15.45:0/1880193125 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c00cdb0 con 0x7f0a580ae4b0 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.750+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5814cb30 con 0x7f0a580ae4b0 2023-11-28T21:52:02.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.750+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5814f6e0 con 0x7f0a580ae4b0 2023-11-28T21:52:02.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.751+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a58165450 con 0x7f0a580ae4b0 2023-11-28T21:52:02.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.752+0000 7f0a3f7fe640 1 -- 172.21.15.45:0/1880193125 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 94) v1 ==== 89351+0+0 (secure 0 0 0) 0x7f0a4c013470 con 0x7f0a580ae4b0 2023-11-28T21:52:02.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.753+0000 7f0a3f7fe640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f0a4406afb0 0x7f0a4406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:02.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.753+0000 7f0a3f7fe640 1 -- 172.21.15.45:0/1880193125 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f0a4c089d40 con 0x7f0a580ae4b0 2023-11-28T21:52:02.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.754+0000 7f0a56ffd640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f0a4406afb0 0x7f0a4406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:02.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.755+0000 7f0a56ffd640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f0a4406afb0 0x7f0a4406d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a48000f60 tx=0x7f0a48007d00 comp rx=0 tx=0).ready entity=mgr.8023 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:02.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:02.758+0000 7f0a3f7fe640 1 -- 172.21.15.45:0/1880193125 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a4c054ad0 con 0x7f0a580ae4b0 2023-11-28T21:52:03.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.008+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0a5809d0f0 con 0x7f0a580ae4b0 2023-11-28T21:52:03.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.139+0000 7f0a3f7fe640 1 -- 172.21.15.45:0/1880193125 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v95) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0a4c0170a0 con 0x7f0a580ae4b0 2023-11-28T21:52:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.142+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f0a4406afb0 msgr2=0x7f0a4406d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.142+0000 7f0a57fff640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f0a4406afb0 0x7f0a4406d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a48000f60 tx=0x7f0a48007d00 comp rx=0 tx=0).stop 2023-11-28T21:52:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.142+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a580ae4b0 msgr2=0x7f0a58160120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:03.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.142+0000 7f0a57fff640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a580ae4b0 0x7f0a58160120 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00e930 tx=0x7f0a4c00edf0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.143+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 shutdown_connections 2023-11-28T21:52:03.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.143+0000 7f0a57fff640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a58161770 0x7f0a580b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.143+0000 7f0a57fff640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f0a4406afb0 0x7f0a4406d470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.143+0000 7f0a57fff640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a580ae4b0 0x7f0a58160120 secure :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00e930 tx=0x7f0a4c00edf0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.143+0000 7f0a57fff640 1 --2- 172.21.15.45:0/1880193125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0a5806e060 0x7f0a5814f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.143+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 >> 172.21.15.45:0/1880193125 conn(0x7f0a580cecd0 msgr2=0x7f0a58164330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.143+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 shutdown_connections 2023-11-28T21:52:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.144+0000 7f0a57fff640 1 -- 172.21.15.45:0/1880193125 wait complete. 2023-11-28T21:52:03.165 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T21:52:03.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T21:52:03.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T21:52:03.355 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T21:52:03.355 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:52:03.355 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:03.408 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:52:03.408 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:03.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:52:03.672 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:03.670+0000 7fc35142c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T21:52:03.712 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:03.711+0000 7f09db42b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T21:52:03.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.713+0000 7fa10f2cf640 1 Processor -- start 2023-11-28T21:52:03.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.714+0000 7fa10f2cf640 1 -- start start 2023-11-28T21:52:03.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.715+0000 7fa10f2cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1080cf2b0 0x7fa108168270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.715+0000 7fa10f2cf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa108066fc0 con 0x7fa10815d8d0 2023-11-28T21:52:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.715+0000 7fa10f2cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1080ca520 con 0x7fa10806ef00 2023-11-28T21:52:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.715+0000 7fa10f2cf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa10806a9f0 con 0x7fa1080cf2b0 2023-11-28T21:52:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.715+0000 7fa10d06b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa10815d8d0 0x7fa1080661f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36420/0 (socket says 172.21.15.45:36420) 2023-11-28T21:52:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.715+0000 7fa10d86c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1080cf2b0 0x7fa108168270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.715+0000 7fa10d06b640 1 -- 172.21.15.45:0/3028367505 learned_addr learned my addr 172.21.15.45:0/3028367505 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:03.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.716+0000 7fa10d86c640 1 -- 172.21.15.45:0/3028367505 >> v1:172.21.15.190:6789/0 conn(0x7fa10806ef00 legacy=0x7fa1080ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:03.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.716+0000 7fa10d86c640 1 -- 172.21.15.45:0/3028367505 >> v1:172.21.15.45:6789/0 conn(0x7fa10815d8d0 legacy=0x7fa1080661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:03.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.716+0000 7fa10d86c640 1 -- 172.21.15.45:0/3028367505 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa108066a00 con 0x7fa1080cf2b0 2023-11-28T21:52:03.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.716+0000 7fa0fffff640 1 -- 172.21.15.45:0/3028367505 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417583438 0 0) 0x7fa108066fc0 con 0x7fa10815d8d0 2023-11-28T21:52:03.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.717+0000 7fa10d86c640 1 --2- 172.21.15.45:0/3028367505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1080cf2b0 0x7fa108168270 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fa100009d90 tx=0x7fa10002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d474e6c73be63696 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.718+0000 7fa0fffff640 1 -- 172.21.15.45:0/3028367505 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10002cd00 con 0x7fa1080cf2b0 2023-11-28T21:52:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.718+0000 7fa0fffff640 1 -- 172.21.15.45:0/3028367505 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10002ce60 con 0x7fa1080cf2b0 2023-11-28T21:52:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.718+0000 7fa0fffff640 1 -- 172.21.15.45:0/3028367505 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa100034660 con 0x7fa1080cf2b0 2023-11-28T21:52:03.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.718+0000 7fa10f2cf640 1 -- 172.21.15.45:0/3028367505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1080cf2b0 msgr2=0x7fa108168270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:03.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.718+0000 7fa10f2cf640 1 --2- 172.21.15.45:0/3028367505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1080cf2b0 0x7fa108168270 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fa100009d90 tx=0x7fa10002bf50 comp rx=0 tx=0).stop 2023-11-28T21:52:03.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.719+0000 7fa10f2cf640 1 -- 172.21.15.45:0/3028367505 shutdown_connections 2023-11-28T21:52:03.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.719+0000 7fa10f2cf640 1 --2- 172.21.15.45:0/3028367505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1080cf2b0 0x7fa108168270 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.719+0000 7fa10f2cf640 1 -- 172.21.15.45:0/3028367505 >> 172.21.15.45:0/3028367505 conn(0x7fa1080cc950 msgr2=0x7fa1080ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:03.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.719+0000 7fa10f2cf640 1 -- 172.21.15.45:0/3028367505 shutdown_connections 2023-11-28T21:52:03.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.719+0000 7fa10f2cf640 1 -- 172.21.15.45:0/3028367505 wait complete. 2023-11-28T21:52:03.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.720+0000 7fa10f2cf640 1 Processor -- start 2023-11-28T21:52:03.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.721+0000 7fa10f2cf640 1 -- start start 2023-11-28T21:52:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.721+0000 7fa10f2cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa10806ef00 0x7fa1081f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10f2cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1080cf2b0 0x7fa1080c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10f2cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa10809e7b0 0x7fa1080ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10f2cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1080af330 con 0x7fa10806ef00 2023-11-28T21:52:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10f2cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1080c9e30 con 0x7fa10809e7b0 2023-11-28T21:52:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10f2cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1081f7280 con 0x7fa1080cf2b0 2023-11-28T21:52:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10d06b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa10806ef00 0x7fa1081f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:03.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10d06b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa10806ef00 0x7fa1081f6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39264/0 (socket says 172.21.15.45:39264) 2023-11-28T21:52:03.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10d06b640 1 -- 172.21.15.45:0/158222153 learned_addr learned my addr 172.21.15.45:0/158222153 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10d86c640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa10809e7b0 0x7fa1080ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10c86a640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1080cf2b0 0x7fa1080c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10d06b640 1 -- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1080cf2b0 msgr2=0x7fa1080c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10d06b640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1080cf2b0 0x7fa1080c98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10d06b640 1 -- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa10809e7b0 msgr2=0x7fa1080ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10d06b640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa10809e7b0 0x7fa1080ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10d06b640 1 -- 172.21.15.45:0/158222153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100009a70 con 0x7fa10806ef00 2023-11-28T21:52:03.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.722+0000 7fa10c86a640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1080cf2b0 0x7fa1080c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.723+0000 7fa10d06b640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa10806ef00 0x7fa1081f6350 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fa10400c4d0 tx=0x7fa10400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.723+0000 7fa0fdffb640 1 -- 172.21.15.45:0/158222153 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1040074e0 con 0x7fa10806ef00 2023-11-28T21:52:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.723+0000 7fa0fdffb640 1 -- 172.21.15.45:0/158222153 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104007660 con 0x7fa10806ef00 2023-11-28T21:52:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.723+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1081f69e0 con 0x7fa10806ef00 2023-11-28T21:52:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.723+0000 7fa0fdffb640 1 -- 172.21.15.45:0/158222153 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104004a20 con 0x7fa10806ef00 2023-11-28T21:52:03.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.723+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1081f6fa0 con 0x7fa10806ef00 2023-11-28T21:52:03.727 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:03.724+0000 7fc35142c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T21:52:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.725+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1080733d0 con 0x7fa10806ef00 2023-11-28T21:52:03.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.726+0000 7fa0fdffb640 1 -- 172.21.15.45:0/158222153 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 95) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fa104004b80 con 0x7fa10806ef00 2023-11-28T21:52:03.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.726+0000 7fa0fdffb640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fa0ec06af60 0x7fa0ec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:03.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.727+0000 7fa0fdffb640 1 -- 172.21.15.45:0/158222153 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fa104088f20 con 0x7fa10806ef00 2023-11-28T21:52:03.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.727+0000 7fa10c86a640 1 -- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fa0ec06af60 msgr2=0x7fa0ec06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:03.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.727+0000 7fa10c86a640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fa0ec06af60 0x7fa0ec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:03.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.731+0000 7fa0fdffb640 1 -- 172.21.15.45:0/158222153 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa104053c90 con 0x7fa10806ef00 2023-11-28T21:52:03.774 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:03.773+0000 7f09db42b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T21:52:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.927+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1080cecd0 con 0x7fa10806ef00 2023-11-28T21:52:03.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.928+0000 7fa10c86a640 1 -- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fa0ec06af60 msgr2=0x7fa0ec06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:03.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.928+0000 7fa10c86a640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fa0ec06af60 0x7fa0ec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:52:03.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.928+0000 7fa0fdffb640 1 -- 172.21.15.45:0/158222153 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+17264 (secure 0 0 0) 0x7fa104018020 con 0x7fa10806ef00 2023-11-28T21:52:03.930 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:03.931 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":186,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:52:00.068817+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:51:43.335999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-11-28T21:51:44.431458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/2404812260":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2400910006":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6833/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2240536749":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6832/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2615023161":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/4027676180":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/508215298":"2023-11-29T21:52:00.068699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:52:03.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fa0ec06af60 msgr2=0x7fa0ec06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:03.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fa0ec06af60 0x7fa0ec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa10806ef00 msgr2=0x7fa1081f6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:03.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa10806ef00 0x7fa1081f6350 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fa10400c4d0 tx=0x7fa10400c990 comp rx=0 tx=0).stop 2023-11-28T21:52:03.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 shutdown_connections 2023-11-28T21:52:03.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa10809e7b0 0x7fa1080ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1080cf2b0 0x7fa1080c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fa0ec06af60 0x7fa0ec06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 --2- 172.21.15.45:0/158222153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa10806ef00 0x7fa1081f6350 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:03.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.931+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 >> 172.21.15.45:0/158222153 conn(0x7fa1080cc950 msgr2=0x7fa1080cb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:03.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.932+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 shutdown_connections 2023-11-28T21:52:03.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:03.932+0000 7fa10f2cf640 1 -- 172.21.15.45:0/158222153 wait complete. 2023-11-28T21:52:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T21:51:43.335999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T21:51:44.431458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T21:52:03.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T21:52:04.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.150+0000 7f2f6f5b7640 1 Processor -- start 2023-11-28T21:52:04.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.151+0000 7f2f6f5b7640 1 -- start start 2023-11-28T21:52:04.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.152+0000 7f2f6f5b7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f680aec60 0x7f2f6815c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.152+0000 7f2f6f5b7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6815c5d0 con 0x7f2f6806cbe0 2023-11-28T21:52:04.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.152+0000 7f2f6f5b7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6806d160 con 0x7f2f680cdf90 2023-11-28T21:52:04.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.152+0000 7f2f6f5b7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6806d440 con 0x7f2f680aec60 2023-11-28T21:52:04.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.152+0000 7f2f6cb52640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2f6806cbe0 0x7f2f6806e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50316/0 (socket says 172.21.15.45:50316) 2023-11-28T21:52:04.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.152+0000 7f2f6cb52640 1 -- 172.21.15.45:0/3133805262 learned_addr learned my addr 172.21.15.45:0/3133805262 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:04.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f6d353640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f680aec60 0x7f2f6815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968649504 0 0) 0x7f2f6815c5d0 con 0x7f2f6806cbe0 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f54003520 con 0x7f2f6806cbe0 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 458828282 0 0) 0x7f2f54003520 con 0x7f2f6806cbe0 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f680aec60 msgr2=0x7f2f6815c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f680aec60 0x7f2f6815c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 >> v1:172.21.15.190:6789/0 conn(0x7f2f680cdf90 legacy=0x7f2f680ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6806d720 con 0x7f2f6806cbe0 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f2f58001670 con 0x7f2f6806cbe0 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 >> v1:172.21.15.45:6789/0 conn(0x7f2f6806cbe0 legacy=0x7f2f6806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f680aec60 0x7f2f540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f54003ee0 0x7f2f540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f54004d00 0x7f2f54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6806d440 con 0x7f2f54004d00 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.153+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5400ba30 con 0x7f2f54003ee0 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5400bbf0 con 0x7f2f680aec60 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6db54640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f54004d00 0x7f2f54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6d353640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f680aec60 0x7f2f540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6cb52640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f54003ee0 0x7f2f540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6db54640 1 -- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f680aec60 msgr2=0x7f2f540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6db54640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f680aec60 0x7f2f540039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6db54640 1 -- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f54003ee0 msgr2=0x7f2f540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6db54640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f54003ee0 0x7f2f540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6db54640 1 -- 172.21.15.45:0/3133805262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f54003520 con 0x7f2f54004d00 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6d353640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f680aec60 0x7f2f540039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:04.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.154+0000 7f2f6cb52640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f54003ee0 0x7f2f540047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:04.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.155+0000 7f2f6db54640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f54004d00 0x7f2f54005180 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f2f60001f60 tx=0x7f2f540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b82d8d7e8e3fe2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.156+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6002d6f0 con 0x7f2f54004d00 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.156+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6002d870 con 0x7f2f54004d00 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.156+0000 7f2f5ffff640 1 -- 172.21.15.45:0/3133805262 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6002dc20 con 0x7f2f54004d00 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.156+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/3133805262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f54004d00 msgr2=0x7f2f54005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.156+0000 7f2f6f5b7640 1 --2- 172.21.15.45:0/3133805262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f54004d00 0x7f2f54005180 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f2f60001f60 tx=0x7f2f540057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.156+0000 7f2f6db54640 1 -- 172.21.15.45:0/3133805262 reap_dead start 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.156+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/3133805262 shutdown_connections 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.156+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/3133805262 >> 172.21.15.45:0/3133805262 conn(0x7f2f6806a500 msgr2=0x7f2f6806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.157+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/3133805262 shutdown_connections 2023-11-28T21:52:04.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.157+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/3133805262 wait complete. 2023-11-28T21:52:04.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.158+0000 7f2f6f5b7640 1 Processor -- start 2023-11-28T21:52:04.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.159+0000 7f2f6f5b7640 1 -- start start 2023-11-28T21:52:04.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.159+0000 7f2f6f5b7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f54004d00 0x7f2f68158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6f5b7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f68143000 0x7f2f681f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6f5b7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f681f6720 0x7f2f681e58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6f5b7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f681f6ba0 con 0x7f2f68143000 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6f5b7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f68151820 con 0x7f2f681f6720 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6f5b7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f680b3620 con 0x7f2f54004d00 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6cb52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f68143000 0x7f2f681f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6cb52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f68143000 0x7f2f681f61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40162/0 (socket says 172.21.15.45:40162) 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6cb52640 1 -- 172.21.15.45:0/1385466132 learned_addr learned my addr 172.21.15.45:0/1385466132 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6db54640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f681f6720 0x7f2f681e58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6cb52640 1 -- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f54004d00 msgr2=0x7f2f68158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6cb52640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f54004d00 0x7f2f68158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6cb52640 1 -- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f681f6720 msgr2=0x7f2f681e58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6cb52640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f681f6720 0x7f2f681e58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.160+0000 7f2f6cb52640 1 -- 172.21.15.45:0/1385466132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f680b3900 con 0x7f2f68143000 2023-11-28T21:52:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.161+0000 7f2f6cb52640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f68143000 0x7f2f681f61e0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800d810 tx=0x7f2f5800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.161+0000 7f2f5dffb640 1 -- 172.21.15.45:0/1385466132 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f580074e0 con 0x7f2f68143000 2023-11-28T21:52:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.161+0000 7f2f5dffb640 1 -- 172.21.15.45:0/1385466132 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f58007640 con 0x7f2f68143000 2023-11-28T21:52:04.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.161+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f680b3be0 con 0x7f2f68143000 2023-11-28T21:52:04.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.161+0000 7f2f5dffb640 1 -- 172.21.15.45:0/1385466132 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f580049e0 con 0x7f2f68143000 2023-11-28T21:52:04.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.161+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f68143480 con 0x7f2f68143000 2023-11-28T21:52:04.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.162+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f680b3760 con 0x7f2f68143000 2023-11-28T21:52:04.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.163+0000 7f2f5dffb640 1 -- 172.21.15.45:0/1385466132 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 95) v1 ==== 89334+0+0 (secure 0 0 0) 0x7f2f58004b40 con 0x7f2f68143000 2023-11-28T21:52:04.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.164+0000 7f2f5dffb640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f2f4c06af90 0x7f2f4c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.164+0000 7f2f5dffb640 1 -- 172.21.15.45:0/1385466132 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f2f580897d0 con 0x7f2f68143000 2023-11-28T21:52:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.164+0000 7f2f6d353640 1 -- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f2f4c06af90 msgr2=0x7f2f4c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:04.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.164+0000 7f2f6d353640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f2f4c06af90 0x7f2f4c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:04.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.172+0000 7f2f5dffb640 1 -- 172.21.15.45:0/1385466132 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f680b3760 con 0x7f2f68143000 2023-11-28T21:52:04.176 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.175+0000 7fc35142c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T21:52:04.228 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.227+0000 7fc35142c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T21:52:04.259 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.258+0000 7f09db42b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T21:52:04.277 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.276+0000 7fc35142c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T21:52:04.314 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.313+0000 7f09db42b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T21:52:04.337 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.336+0000 7fc35142c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T21:52:04.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.365+0000 7f2f6d353640 1 -- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f2f4c06af90 msgr2=0x7f2f4c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:04.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.365+0000 7f2f6d353640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f2f4c06af90 0x7f2f4c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:52:04.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.365+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f680b3760 con 0x7f2f68143000 2023-11-28T21:52:04.367 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.366+0000 7f09db42b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T21:52:04.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.366+0000 7f2f5dffb640 1 -- 172.21.15.45:0/1385466132 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v186) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f58019020 con 0x7f2f68143000 2023-11-28T21:52:04.368 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T21:52:04.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f2f4c06af90 msgr2=0x7f2f4c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:04.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f2f4c06af90 0x7f2f4c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f68143000 msgr2=0x7f2f681f61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f68143000 0x7f2f681f61e0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800d810 tx=0x7f2f5800dcd0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 shutdown_connections 2023-11-28T21:52:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f681f6720 0x7f2f681e58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7f2f4c06af90 0x7f2f4c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f68143000 0x7f2f681f61e0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 --2- 172.21.15.45:0/1385466132 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2f54004d00 0x7f2f68158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 >> 172.21.15.45:0/1385466132 conn(0x7f2f6806a500 msgr2=0x7f2f6806aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 shutdown_connections 2023-11-28T21:52:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.369+0000 7f2f6f5b7640 1 -- 172.21.15.45:0/1385466132 wait complete. 2023-11-28T21:52:04.386 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.385+0000 7fc35142c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T21:52:04.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T21:52:04.433 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.432+0000 7f09db42b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T21:52:04.487 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.485+0000 7f09db42b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T21:52:04.498 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.497+0000 7fc35142c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T21:52:04.560 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.558+0000 7fc35142c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T21:52:04.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.596+0000 7fcdf329b640 1 Processor -- start 2023-11-28T21:52:04.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.597+0000 7fcdf329b640 1 -- start start 2023-11-28T21:52:04.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.598+0000 7fcdf329b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcdec15e840 0x7fcdec06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.598+0000 7fcdf329b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 0x7fcdec0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.598+0000 7fcdf329b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcdec06a500 0x7fcdec06a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.598+0000 7fcdf329b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec071ce0 con 0x7fcdec06a500 2023-11-28T21:52:04.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.598+0000 7fcdf329b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec06aed0 con 0x7fcdec15e840 2023-11-28T21:52:04.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.598+0000 7fcdf329b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec15ecd0 con 0x7fcdec15f720 2023-11-28T21:52:04.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.598+0000 7fcdf1838640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcdec06a500 0x7fcdec06a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.599+0000 7fcdf1838640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcdec06a500 0x7fcdec06a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40170/0 (socket says 172.21.15.45:40170) 2023-11-28T21:52:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.599+0000 7fcdf1838640 1 -- 172.21.15.45:0/1213787893 learned_addr learned my addr 172.21.15.45:0/1213787893 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.599+0000 7fcdf0836640 1 --2- 172.21.15.45:0/1213787893 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 0x7fcdec0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.599+0000 7fcdf1037640 1 --2- 172.21.15.45:0/1213787893 >> v2:172.21.15.190:3300/0 conn(0x7fcdec15e840 0x7fcdec06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.599+0000 7fcdf1838640 1 -- 172.21.15.45:0/1213787893 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 msgr2=0x7fcdec0717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.599+0000 7fcdf1838640 1 --2- 172.21.15.45:0/1213787893 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 0x7fcdec0717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.599+0000 7fcdf1838640 1 -- 172.21.15.45:0/1213787893 >> v2:172.21.15.190:3300/0 conn(0x7fcdec15e840 msgr2=0x7fcdec06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.599+0000 7fcdf1838640 1 --2- 172.21.15.45:0/1213787893 >> v2:172.21.15.190:3300/0 conn(0x7fcdec15e840 0x7fcdec06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.599+0000 7fcdf1838640 1 -- 172.21.15.45:0/1213787893 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec15f290 con 0x7fcdec06a500 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.600+0000 7fcdf1838640 1 --2- 172.21.15.45:0/1213787893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 0x7fcdec06a990 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009870 tx=0x7fcde402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd71b58cf2c977dd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.600+0000 7fcdebfff640 1 -- 172.21.15.45:0/1213787893 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde4031640 con 0x7fcdec06a500 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdebfff640 1 -- 172.21.15.45:0/1213787893 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde40317a0 con 0x7fcdec06a500 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdebfff640 1 -- 172.21.15.45:0/1213787893 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde4031b30 con 0x7fcdec06a500 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdf329b640 1 -- 172.21.15.45:0/1213787893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 msgr2=0x7fcdec06a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdf329b640 1 --2- 172.21.15.45:0/1213787893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 0x7fcdec06a990 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009870 tx=0x7fcde402b8f0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdf329b640 1 -- 172.21.15.45:0/1213787893 shutdown_connections 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdf329b640 1 --2- 172.21.15.45:0/1213787893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 0x7fcdec06a990 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdf329b640 1 --2- 172.21.15.45:0/1213787893 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 0x7fcdec0717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdf329b640 1 --2- 172.21.15.45:0/1213787893 >> v2:172.21.15.190:3300/0 conn(0x7fcdec15e840 0x7fcdec06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdf329b640 1 -- 172.21.15.45:0/1213787893 >> 172.21.15.45:0/1213787893 conn(0x7fcdec0ced60 msgr2=0x7fcdec0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdf329b640 1 -- 172.21.15.45:0/1213787893 shutdown_connections 2023-11-28T21:52:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.601+0000 7fcdf329b640 1 -- 172.21.15.45:0/1213787893 wait complete. 2023-11-28T21:52:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.603+0000 7fcdf329b640 1 Processor -- start 2023-11-28T21:52:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.603+0000 7fcdf329b640 1 -- start start 2023-11-28T21:52:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.604+0000 7fcdf329b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 0x7fcdec0ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.604+0000 7fcdf329b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdec15e840 0x7fcdec0c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.604+0000 7fcdf329b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 0x7fcdec0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.604+0000 7fcdf329b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec0cae70 con 0x7fcdec06a500 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.604+0000 7fcdf329b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec0c9890 con 0x7fcdec15e840 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.604+0000 7fcdf329b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec0c9b70 con 0x7fcdec15f720 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.604+0000 7fcdf1037640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 0x7fcdec0ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.604+0000 7fcdf1037640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 0x7fcdec0ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40182/0 (socket says 172.21.15.45:40182) 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.604+0000 7fcdf1037640 1 -- 172.21.15.45:0/3506015495 learned_addr learned my addr 172.21.15.45:0/3506015495 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.605+0000 7fcdf0836640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdec15e840 0x7fcdec0c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.605+0000 7fcdf1037640 1 -- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 msgr2=0x7fcdec0c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.605+0000 7fcdf1838640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 0x7fcdec0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.605+0000 7fcdf1037640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 0x7fcdec0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.605+0000 7fcdf1037640 1 -- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdec15e840 msgr2=0x7fcdec0c2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.605+0000 7fcdf1037640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdec15e840 0x7fcdec0c2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.605+0000 7fcdf1037640 1 -- 172.21.15.45:0/3506015495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde40094d0 con 0x7fcdec06a500 2023-11-28T21:52:04.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.605+0000 7fcdf0836640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdec15e840 0x7fcdec0c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:04.609 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.605+0000 7f09db42b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T21:52:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.605+0000 7fcdf1037640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 0x7fcdec0ca930 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fcde0007e60 tx=0x7fcde000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.606+0000 7fcde9ffb640 1 -- 172.21.15.45:0/3506015495 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0014620 con 0x7fcdec06a500 2023-11-28T21:52:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.606+0000 7fcde9ffb640 1 -- 172.21.15.45:0/3506015495 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde000ee90 con 0x7fcdec06a500 2023-11-28T21:52:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.606+0000 7fcde9ffb640 1 -- 172.21.15.45:0/3506015495 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde00149c0 con 0x7fcdec06a500 2023-11-28T21:52:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.606+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec0c9e50 con 0x7fcdec06a500 2023-11-28T21:52:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.606+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec0ca410 con 0x7fcdec06a500 2023-11-28T21:52:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.607+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec15eee0 con 0x7fcdec06a500 2023-11-28T21:52:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.607+0000 7fcde9ffb640 1 -- 172.21.15.45:0/3506015495 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 95) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fcde0014b20 con 0x7fcdec06a500 2023-11-28T21:52:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.608+0000 7fcde9ffb640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fcdd006af60 0x7fcdd006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:04.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.608+0000 7fcde9ffb640 1 -- 172.21.15.45:0/3506015495 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fcde008a460 con 0x7fcdec06a500 2023-11-28T21:52:04.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.608+0000 7fcdf0836640 1 -- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fcdd006af60 msgr2=0x7fcdd006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:04.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.608+0000 7fcdf0836640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fcdd006af60 0x7fcdd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:04.612 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.608+0000 7fc35142c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T21:52:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.612+0000 7fcde9ffb640 1 -- 172.21.15.45:0/3506015495 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcde00550a0 con 0x7fcdec06a500 2023-11-28T21:52:04.668 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.667+0000 7fc35142c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T21:52:04.675 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.673+0000 7f09db42b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T21:52:04.718 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:04.717+0000 7fc35142c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T21:52:04.728 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.726+0000 7f09db42b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T21:52:04.789 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.786+0000 7f09db42b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T21:52:04.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.809+0000 7fcdf0836640 1 -- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fcdd006af60 msgr2=0x7fcdd006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:04.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.809+0000 7fcdf0836640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fcdd006af60 0x7fcdd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:52:04.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.811+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcdec0ca5a0 con 0x7fcdec06a500 2023-11-28T21:52:04.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.812+0000 7fcde9ffb640 1 -- 172.21.15.45:0/3506015495 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v186) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcde00586c0 con 0x7fcdec06a500 2023-11-28T21:52:04.814 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T21:52:04.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fcdd006af60 msgr2=0x7fcdd006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:04.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fcdd006af60 0x7fcdd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 msgr2=0x7fcdec0ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:04.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 0x7fcdec0ca930 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fcde0007e60 tx=0x7fcde000e4b0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 shutdown_connections 2023-11-28T21:52:04.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdec15f720 0x7fcdec0c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdec15e840 0x7fcdec0c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fcdd006af60 0x7fcdd006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 --2- 172.21.15.45:0/3506015495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdec06a500 0x7fcdec0ca930 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:04.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 >> 172.21.15.45:0/3506015495 conn(0x7fcdec0ced60 msgr2=0x7fcdec158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:04.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 shutdown_connections 2023-11-28T21:52:04.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:04.815+0000 7fcdf329b640 1 -- 172.21.15.45:0/3506015495 wait complete. 2023-11-28T21:52:04.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T21:52:04.843 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:04.841+0000 7f09db42b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T21:52:05.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.039+0000 7fd3d326d640 1 Processor -- start 2023-11-28T21:52:05.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.039+0000 7fd3d326d640 1 -- start start 2023-11-28T21:52:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.041+0000 7fd3d326d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 0x7fd3cc0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.041+0000 7fd3d326d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd3cc0b1d40 0x7fd3cc0d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.041+0000 7fd3d326d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc0b90a0 con 0x7fd3cc0b2290 2023-11-28T21:52:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.041+0000 7fd3d326d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3cc0d1920 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.041+0000 7fd3d326d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0d2dc0 con 0x7fd3cc0d0040 2023-11-28T21:52:05.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d2a6c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd3cc0b1d40 0x7fd3cc0d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d226b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 0x7fd3cc0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d2a6c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd3cc0b1d40 0x7fd3cc0d3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:36208/0 (socket says 172.21.15.45:36208) 2023-11-28T21:52:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d2a6c640 1 -- 172.21.15.45:0/279553042 learned_addr learned my addr 172.21.15.45:0/279553042 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d1a6a640 1 --1- 172.21.15.45:0/279553042 >> v1:172.21.15.190:6789/0 conn(0x7fd3cc0b2290 0x7fd3cc0d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:49506/0 (socket says 172.21.15.45:49506) 2023-11-28T21:52:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d2a6c640 1 -- 172.21.15.45:0/279553042 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 msgr2=0x7fd3cc0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d2a6c640 1 --2- 172.21.15.45:0/279553042 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 0x7fd3cc0d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d2a6c640 1 -- 172.21.15.45:0/279553042 >> v1:172.21.15.190:6789/0 conn(0x7fd3cc0b2290 legacy=0x7fd3cc0d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d2a6c640 1 -- 172.21.15.45:0/279553042 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc0d0960 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.042+0000 7fd3d226b640 1 --2- 172.21.15.45:0/279553042 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 0x7fd3cc0d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T21:52:05.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.044+0000 7fd3d2a6c640 1 --2- 172.21.15.45:0/279553042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b1d40 0x7fd3cc0d3750 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4009e50 tx=0x7fd3c402bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b5d5dd270019d22 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:05.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.045+0000 7fd3d1269640 1 -- 172.21.15.45:0/279553042 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c402cb90 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.045+0000 7fd3d1269640 1 -- 172.21.15.45:0/279553042 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c402ccf0 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.045+0000 7fd3d1269640 1 -- 172.21.15.45:0/279553042 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4034610 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.045+0000 7fd3d326d640 1 -- 172.21.15.45:0/279553042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b1d40 msgr2=0x7fd3cc0d3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.045+0000 7fd3d326d640 1 --2- 172.21.15.45:0/279553042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b1d40 0x7fd3cc0d3750 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4009e50 tx=0x7fd3c402bff0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.045+0000 7fd3d326d640 1 -- 172.21.15.45:0/279553042 shutdown_connections 2023-11-28T21:52:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.045+0000 7fd3d326d640 1 --2- 172.21.15.45:0/279553042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b1d40 0x7fd3cc0d3750 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.045+0000 7fd3d326d640 1 --2- 172.21.15.45:0/279553042 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 0x7fd3cc0d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.045+0000 7fd3d326d640 1 -- 172.21.15.45:0/279553042 >> 172.21.15.45:0/279553042 conn(0x7fd3cc070f40 msgr2=0x7fd3cc071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.046+0000 7fd3d326d640 1 -- 172.21.15.45:0/279553042 shutdown_connections 2023-11-28T21:52:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.046+0000 7fd3d326d640 1 -- 172.21.15.45:0/279553042 wait complete. 2023-11-28T21:52:05.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.048+0000 7fd3d326d640 1 Processor -- start 2023-11-28T21:52:05.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.048+0000 7fd3d326d640 1 -- start start 2023-11-28T21:52:05.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.049+0000 7fd3d326d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3cc0b1d40 0x7fd3cc0caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.049+0000 7fd3d326d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b2290 0x7fd3cc14b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.049+0000 7fd3d326d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 0x7fd3cc1e54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.049+0000 7fd3d326d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc14bb20 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.049+0000 7fd3d326d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0cb0f0 con 0x7fd3cc0b2290 2023-11-28T21:52:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.049+0000 7fd3d326d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0cb3d0 con 0x7fd3cc0d0040 2023-11-28T21:52:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.049+0000 7fd3d226b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3cc0b1d40 0x7fd3cc0caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.049+0000 7fd3d226b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3cc0b1d40 0x7fd3cc0caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40186/0 (socket says 172.21.15.45:40186) 2023-11-28T21:52:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.049+0000 7fd3d226b640 1 -- 172.21.15.45:0/1225276341 learned_addr learned my addr 172.21.15.45:0/1225276341 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d226b640 1 -- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 msgr2=0x7fd3cc1e54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d226b640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 0x7fd3cc1e54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d1a6a640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b2290 0x7fd3cc14b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d226b640 1 -- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b2290 msgr2=0x7fd3cc14b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d226b640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b2290 0x7fd3cc14b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d226b640 1 -- 172.21.15.45:0/1225276341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c8009400 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d1a6a640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b2290 0x7fd3cc14b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d226b640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3cc0b1d40 0x7fd3cc0caa60 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800ebe0 tx=0x7fd3c800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3c2ffd640 1 -- 172.21.15.45:0/1225276341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8009760 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3c2ffd640 1 -- 172.21.15.45:0/1225276341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c800cc30 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3c2ffd640 1 -- 172.21.15.45:0/1225276341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c800eee0 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c4009b30 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.050+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3cc0cb990 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.052+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3cc071b30 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.053+0000 7fd3c2ffd640 1 -- 172.21.15.45:0/1225276341 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 95) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fd3c8013470 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.053+0000 7fd3c2ffd640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd3b006af60 0x7fd3b006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.054+0000 7fd3c2ffd640 1 -- 172.21.15.45:0/1225276341 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd3c8089de0 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.054+0000 7fd3d1a6a640 1 -- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd3b006af60 msgr2=0x7fd3b006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.054+0000 7fd3d1a6a640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd3b006af60 0x7fd3b006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:05.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.058+0000 7fd3c2ffd640 1 -- 172.21.15.45:0/1225276341 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3c8054ac0 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.102 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:05.100+0000 7fc35142c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T21:52:05.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.251+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd3cc04e920 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.252+0000 7fd3c2ffd640 1 -- 172.21.15.45:0/1225276341 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v186) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd3c80170a0 con 0x7fd3cc0b1d40 2023-11-28T21:52:05.255 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:05.252+0000 7f09db42b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T21:52:05.255 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T21:52:05.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.254+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd3b006af60 msgr2=0x7fd3b006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:05.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.254+0000 7fd3d326d640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd3b006af60 0x7fd3b006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.254+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3cc0b1d40 msgr2=0x7fd3cc0caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.254+0000 7fd3d326d640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3cc0b1d40 0x7fd3cc0caa60 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800ebe0 tx=0x7fd3c800c520 comp rx=0 tx=0).stop 2023-11-28T21:52:05.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.255+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 shutdown_connections 2023-11-28T21:52:05.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.255+0000 7fd3d326d640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3cc0d0040 0x7fd3cc1e54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.255+0000 7fd3d326d640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3cc0b2290 0x7fd3cc14b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.255+0000 7fd3d326d640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd3b006af60 0x7fd3b006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.255+0000 7fd3d326d640 1 --2- 172.21.15.45:0/1225276341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3cc0b1d40 0x7fd3cc0caa60 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.255+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 >> 172.21.15.45:0/1225276341 conn(0x7fd3cc070f40 msgr2=0x7fd3cc0cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:05.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.255+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 shutdown_connections 2023-11-28T21:52:05.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.255+0000 7fd3d326d640 1 -- 172.21.15.45:0/1225276341 wait complete. 2023-11-28T21:52:05.262 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:05.261+0000 7fc35142c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T21:52:05.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T21:52:05.370 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:05.369+0000 7fc35142c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T21:52:05.419 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:05.418+0000 7fc35142c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T21:52:05.427 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:05.424+0000 7f09db42b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T21:52:05.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.483+0000 7fbc4d7ae640 1 Processor -- start 2023-11-28T21:52:05.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.484+0000 7fbc4d7ae640 1 -- start start 2023-11-28T21:52:05.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.485+0000 7fbc4d7ae640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbc48160e70 0x7fbc480cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.485+0000 7fbc4d7ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc480ce3a0 0x7fbc480d19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.485+0000 7fbc4d7ae640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc480d14d0 con 0x7fbc4815d310 2023-11-28T21:52:05.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.485+0000 7fbc4d7ae640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc480ce810 con 0x7fbc48160e70 2023-11-28T21:52:05.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.485+0000 7fbc4d7ae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc481613c0 con 0x7fbc480ce3a0 2023-11-28T21:52:05.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.485+0000 7fbc3ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbc48160e70 0x7fbc480cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:05.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.486+0000 7fbc3ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbc48160e70 0x7fbc480cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40194/0 (socket says 172.21.15.45:40194) 2023-11-28T21:52:05.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.486+0000 7fbc3ffff640 1 -- 172.21.15.45:0/2842613539 learned_addr learned my addr 172.21.15.45:0/2842613539 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:05.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.486+0000 7fbc4cfad640 1 --2- 172.21.15.45:0/2842613539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc480ce3a0 0x7fbc480d19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:05.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.486+0000 7fbc3ffff640 1 -- 172.21.15.45:0/2842613539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc480ce3a0 msgr2=0x7fbc480d19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:05.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.486+0000 7fbc3ffff640 1 --2- 172.21.15.45:0/2842613539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc480ce3a0 0x7fbc480d19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.486+0000 7fbc3ffff640 1 -- 172.21.15.45:0/2842613539 >> v1:172.21.15.190:6789/0 conn(0x7fbc4815d310 legacy=0x7fbc480b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.486+0000 7fbc3ffff640 1 -- 172.21.15.45:0/2842613539 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc481616a0 con 0x7fbc48160e70 2023-11-28T21:52:05.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.487+0000 7fbc3ffff640 1 --2- 172.21.15.45:0/2842613539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48160e70 0x7fbc480cde60 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009870 tx=0x7fbc3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224257c35a16cdce server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:05.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc477fe640 1 -- 172.21.15.45:0/2842613539 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30031640 con 0x7fbc48160e70 2023-11-28T21:52:05.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc477fe640 1 -- 172.21.15.45:0/2842613539 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc300317a0 con 0x7fbc48160e70 2023-11-28T21:52:05.490 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc477fe640 1 -- 172.21.15.45:0/2842613539 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30031b30 con 0x7fbc48160e70 2023-11-28T21:52:05.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/2842613539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48160e70 msgr2=0x7fbc480cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc4d7ae640 1 --2- 172.21.15.45:0/2842613539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48160e70 0x7fbc480cde60 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009870 tx=0x7fbc3002b8f0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/2842613539 shutdown_connections 2023-11-28T21:52:05.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc4d7ae640 1 --2- 172.21.15.45:0/2842613539 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc480ce3a0 0x7fbc480d19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc4d7ae640 1 --2- 172.21.15.45:0/2842613539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc48160e70 0x7fbc480cde60 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/2842613539 >> 172.21.15.45:0/2842613539 conn(0x7fbc4806a500 msgr2=0x7fbc4806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:05.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/2842613539 shutdown_connections 2023-11-28T21:52:05.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.488+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/2842613539 wait complete. 2023-11-28T21:52:05.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.490+0000 7fbc4d7ae640 1 Processor -- start 2023-11-28T21:52:05.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.490+0000 7fbc4d7ae640 1 -- start start 2023-11-28T21:52:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.491+0000 7fbc4d7ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc480ce3a0 0x7fbc481f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.491+0000 7fbc4d7ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4815d310 0x7fbc48147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.491+0000 7fbc4d7ae640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc48160e70 0x7fbc481f7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.491+0000 7fbc4d7ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc480b44a0 con 0x7fbc480ce3a0 2023-11-28T21:52:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.491+0000 7fbc4d7ae640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc481479b0 con 0x7fbc48160e70 2023-11-28T21:52:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.491+0000 7fbc4d7ae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc480cbd80 con 0x7fbc4815d310 2023-11-28T21:52:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc47fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc480ce3a0 0x7fbc481f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc47fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc480ce3a0 0x7fbc481f5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40204/0 (socket says 172.21.15.45:40204) 2023-11-28T21:52:05.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc47fff640 1 -- 172.21.15.45:0/1980782012 learned_addr learned my addr 172.21.15.45:0/1980782012 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc4cfad640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc48160e70 0x7fbc481f7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc3ffff640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4815d310 0x7fbc48147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc47fff640 1 -- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4815d310 msgr2=0x7fbc48147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc47fff640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4815d310 0x7fbc48147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc47fff640 1 -- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc48160e70 msgr2=0x7fbc481f7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc47fff640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc48160e70 0x7fbc481f7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.492+0000 7fbc47fff640 1 -- 172.21.15.45:0/1980782012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc300094d0 con 0x7fbc480ce3a0 2023-11-28T21:52:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.493+0000 7fbc47fff640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc480ce3a0 0x7fbc481f5fb0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fbc3800c450 tx=0x7fbc3800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.493+0000 7fbc457fa640 1 -- 172.21.15.45:0/1980782012 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc380052d0 con 0x7fbc480ce3a0 2023-11-28T21:52:05.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.493+0000 7fbc457fa640 1 -- 172.21.15.45:0/1980782012 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc38005430 con 0x7fbc480ce3a0 2023-11-28T21:52:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.493+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc481559d0 con 0x7fbc480ce3a0 2023-11-28T21:52:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.493+0000 7fbc457fa640 1 -- 172.21.15.45:0/1980782012 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc38003830 con 0x7fbc480ce3a0 2023-11-28T21:52:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.493+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc48147e90 con 0x7fbc480ce3a0 2023-11-28T21:52:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.494+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc48071b30 con 0x7fbc480ce3a0 2023-11-28T21:52:05.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.495+0000 7fbc457fa640 1 -- 172.21.15.45:0/1980782012 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 95) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fbc38003990 con 0x7fbc480ce3a0 2023-11-28T21:52:05.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.496+0000 7fbc457fa640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fbc2806af60 0x7fbc2806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:05.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.496+0000 7fbc457fa640 1 -- 172.21.15.45:0/1980782012 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fbc38088f90 con 0x7fbc480ce3a0 2023-11-28T21:52:05.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.496+0000 7fbc3ffff640 1 -- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fbc2806af60 msgr2=0x7fbc2806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:05.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.496+0000 7fbc3ffff640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fbc2806af60 0x7fbc2806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:05.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.500+0000 7fbc457fa640 1 -- 172.21.15.45:0/1980782012 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc380bbe90 con 0x7fbc480ce3a0 2023-11-28T21:52:05.529 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:05.528+0000 7fc35142c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:52:05.544 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:05.542+0000 7f09db42b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T21:52:05.598 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:05.596+0000 7f09db42b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T21:52:05.637 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:05.636+0000 7fc35142c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T21:52:05.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.697+0000 7fbc3ffff640 1 -- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fbc2806af60 msgr2=0x7fbc2806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:05.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.697+0000 7fbc3ffff640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fbc2806af60 0x7fbc2806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:52:05.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.709+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbc480cd670 con 0x7fbc480ce3a0 2023-11-28T21:52:05.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.710+0000 7fbc457fa640 1 -- 172.21.15.45:0/1980782012 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v186) v1 ==== 80+0+477 (secure 0 0 0) 0x7fbc38017080 con 0x7fbc480ce3a0 2023-11-28T21:52:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2404812260 2023-11-29T21:52:00.068699+0000 2023-11-28T21:52:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2400910006 2023-11-29T21:52:00.068699+0000 2023-11-28T21:52:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/2040970137 2023-11-29T21:52:00.068699+0000 2023-11-28T21:52:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2240536749 2023-11-29T21:52:00.068699+0000 2023-11-28T21:52:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/2040970137 2023-11-29T21:52:00.068699+0000 2023-11-28T21:52:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2615023161 2023-11-29T21:52:00.068699+0000 2023-11-28T21:52:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/4027676180 2023-11-29T21:52:00.068699+0000 2023-11-28T21:52:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/508215298 2023-11-29T21:52:00.068699+0000 2023-11-28T21:52:05.713 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2023-11-28T21:52:05.715 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:05.713+0000 7f09db42b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:52:05.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fbc2806af60 msgr2=0x7fbc2806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fbc2806af60 0x7fbc2806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc480ce3a0 msgr2=0x7fbc481f5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc480ce3a0 0x7fbc481f5fb0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fbc3800c450 tx=0x7fbc3800c910 comp rx=0 tx=0).stop 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 shutdown_connections 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc48160e70 0x7fbc481f7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc4815d310 0x7fbc48147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fbc2806af60 0x7fbc2806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 --2- 172.21.15.45:0/1980782012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc480ce3a0 0x7fbc481f5fb0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 >> 172.21.15.45:0/1980782012 conn(0x7fbc4806a500 msgr2=0x7fbc480cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 shutdown_connections 2023-11-28T21:52:05.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:05.713+0000 7fbc4d7ae640 1 -- 172.21.15.45:0/1980782012 wait complete. 2023-11-28T21:52:05.741 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T21:52:05.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:05.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T21:52:05.795 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:05.793+0000 7fc35142c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:52:05.838 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:05.835+0000 7f09db42b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T21:52:05.844 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:05.843+0000 7fc35142c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T21:52:05.852 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T21:52:05.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:05.852 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T21:52:05.897 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T21:52:05.897 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:05.951 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T21:52:05.952 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:52:05.952 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:05.970 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:05.969+0000 7fc35142c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T21:52:06.004 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T21:52:06.004 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T21:52:06.004 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:06.010 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:06.008+0000 7f09db42b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:52:06.058 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T21:52:06.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:52:06.067 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:06.060+0000 7f09db42b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T21:52:06.193 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:06.190+0000 7f09db42b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T21:52:06.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.271+0000 7ff52bfff640 1 Processor -- start 2023-11-28T21:52:06.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.271+0000 7ff52bfff640 1 -- start start 2023-11-28T21:52:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.273+0000 7ff52bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff52c15d550 0x7ff52c0cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.273+0000 7ff52bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff52c0d0100 0x7ff52c0d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.273+0000 7ff52bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d1ab0 0x7ff52c070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.273+0000 7ff52bfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52c0d0590 con 0x7ff52c0d0100 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.273+0000 7ff52bfff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52c0d2230 con 0x7ff52c15d550 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.273+0000 7ff52bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c0d2510 con 0x7ff52c0d1ab0 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff523fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff52c0d0100 0x7ff52c0d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff523fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff52c0d0100 0x7ff52c0d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40212/0 (socket says 172.21.15.45:40212) 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff523fff640 1 -- 172.21.15.45:0/2505745942 learned_addr learned my addr 172.21.15.45:0/2505745942 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff52affd640 1 --2- 172.21.15.45:0/2505745942 >> v2:172.21.15.190:3300/0 conn(0x7ff52c15d550 0x7ff52c0cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff52b7fe640 1 --2- 172.21.15.45:0/2505745942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d1ab0 0x7ff52c070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff523fff640 1 -- 172.21.15.45:0/2505745942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d1ab0 msgr2=0x7ff52c070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff523fff640 1 --2- 172.21.15.45:0/2505745942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d1ab0 0x7ff52c070b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff523fff640 1 -- 172.21.15.45:0/2505745942 >> v2:172.21.15.190:3300/0 conn(0x7ff52c15d550 msgr2=0x7ff52c0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:06.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff523fff640 1 --2- 172.21.15.45:0/2505745942 >> v2:172.21.15.190:3300/0 conn(0x7ff52c15d550 0x7ff52c0cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.274+0000 7ff523fff640 1 -- 172.21.15.45:0/2505745942 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c0d27f0 con 0x7ff52c0d0100 2023-11-28T21:52:06.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.275+0000 7ff523fff640 1 --2- 172.21.15.45:0/2505745942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c0d0100 0x7ff52c0d1570 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0b5dc0 tx=0x7ff51402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38dccd576d4911 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:06.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.276+0000 7ff52a7fc640 1 -- 172.21.15.45:0/2505745942 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514031870 con 0x7ff52c0d0100 2023-11-28T21:52:06.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.276+0000 7ff52a7fc640 1 -- 172.21.15.45:0/2505745942 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5140319d0 con 0x7ff52c0d0100 2023-11-28T21:52:06.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.276+0000 7ff52a7fc640 1 -- 172.21.15.45:0/2505745942 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514031d60 con 0x7ff52c0d0100 2023-11-28T21:52:06.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.276+0000 7ff52bfff640 1 -- 172.21.15.45:0/2505745942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c0d0100 msgr2=0x7ff52c0d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.276+0000 7ff52bfff640 1 --2- 172.21.15.45:0/2505745942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c0d0100 0x7ff52c0d1570 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0b5dc0 tx=0x7ff51402b910 comp rx=0 tx=0).stop 2023-11-28T21:52:06.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.277+0000 7ff52bfff640 1 -- 172.21.15.45:0/2505745942 shutdown_connections 2023-11-28T21:52:06.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.277+0000 7ff52bfff640 1 --2- 172.21.15.45:0/2505745942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d1ab0 0x7ff52c070b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.277+0000 7ff52bfff640 1 --2- 172.21.15.45:0/2505745942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c0d0100 0x7ff52c0d1570 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.277+0000 7ff52bfff640 1 --2- 172.21.15.45:0/2505745942 >> v2:172.21.15.190:3300/0 conn(0x7ff52c15d550 0x7ff52c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.277+0000 7ff52bfff640 1 -- 172.21.15.45:0/2505745942 >> 172.21.15.45:0/2505745942 conn(0x7ff52c06ac70 msgr2=0x7ff52c0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:06.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.277+0000 7ff52bfff640 1 -- 172.21.15.45:0/2505745942 shutdown_connections 2023-11-28T21:52:06.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.277+0000 7ff52bfff640 1 -- 172.21.15.45:0/2505745942 wait complete. 2023-11-28T21:52:06.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.278+0000 7ff52bfff640 1 Processor -- start 2023-11-28T21:52:06.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.279+0000 7ff52bfff640 1 -- start start 2023-11-28T21:52:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.279+0000 7ff52bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d0100 0x7ff52c1e8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.279+0000 7ff52bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c0d1ab0 0x7ff52c0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c15d550 0x7ff52c0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52bfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1e5c60 con 0x7ff52c15d550 2023-11-28T21:52:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52bfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c0cd600 con 0x7ff52c0d1ab0 2023-11-28T21:52:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c0cd8e0 con 0x7ff52c0d0100 2023-11-28T21:52:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c15d550 0x7ff52c0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c15d550 0x7ff52c0ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40218/0 (socket says 172.21.15.45:40218) 2023-11-28T21:52:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52b7fe640 1 -- 172.21.15.45:0/1142780198 learned_addr learned my addr 172.21.15.45:0/1142780198 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff523fff640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c0d1ab0 0x7ff52c0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52affd640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d0100 0x7ff52c1e8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52b7fe640 1 -- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d0100 msgr2=0x7ff52c1e8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52b7fe640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d0100 0x7ff52c1e8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52b7fe640 1 -- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c0d1ab0 msgr2=0x7ff52c0cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52b7fe640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c0d1ab0 0x7ff52c0cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.280+0000 7ff52b7fe640 1 -- 172.21.15.45:0/1142780198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140094d0 con 0x7ff52c15d550 2023-11-28T21:52:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.281+0000 7ff52b7fe640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c15d550 0x7ff52c0ccd80 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7ff524007e60 tx=0x7ff52400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.281+0000 7ff528ff9640 1 -- 172.21.15.45:0/1142780198 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff524014620 con 0x7ff52c15d550 2023-11-28T21:52:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.281+0000 7ff528ff9640 1 -- 172.21.15.45:0/1142780198 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52400ee90 con 0x7ff52c15d550 2023-11-28T21:52:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.281+0000 7ff528ff9640 1 -- 172.21.15.45:0/1142780198 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5240149e0 con 0x7ff52c15d550 2023-11-28T21:52:06.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.281+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52c1e5eb0 con 0x7ff52c15d550 2023-11-28T21:52:06.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.281+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52c0cd2c0 con 0x7ff52c15d550 2023-11-28T21:52:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.282+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52c1e63d0 con 0x7ff52c15d550 2023-11-28T21:52:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.283+0000 7ff528ff9640 1 -- 172.21.15.45:0/1142780198 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 95) v1 ==== 89334+0+0 (secure 0 0 0) 0x7ff524014b40 con 0x7ff52c15d550 2023-11-28T21:52:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.284+0000 7ff528ff9640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7ff50c06af90 0x7ff50c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.284+0000 7ff528ff9640 1 -- 172.21.15.45:0/1142780198 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff52408a350 con 0x7ff52c15d550 2023-11-28T21:52:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.284+0000 7ff52affd640 1 -- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7ff50c06af90 msgr2=0x7ff50c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.284+0000 7ff52affd640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7ff50c06af90 0x7ff50c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:06.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.289+0000 7ff528ff9640 1 -- 172.21.15.45:0/1142780198 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff52c1e63d0 con 0x7ff52c15d550 2023-11-28T21:52:06.487 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.485+0000 7ff52affd640 1 -- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7ff50c06af90 msgr2=0x7ff50c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:06.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.485+0000 7ff52affd640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7ff50c06af90 0x7ff50c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:52:06.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.520+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff52c1e63d0 con 0x7ff52c15d550 2023-11-28T21:52:06.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.521+0000 7ff528ff9640 1 -- 172.21.15.45:0/1142780198 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4908 (secure 0 0 0) 0x7ff52c1e63d0 con 0x7ff52c15d550 2023-11-28T21:52:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:06.524 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7130,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/500262407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":500262407},{"type":"v1","addr":"172.21.15.190:6835","nonce":500262407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/195287870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":195287870},{"type":"v1","addr":"172.21.15.190:6837","nonce":195287870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":9657,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/440031651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":440031651},{"type":"v1","addr":"172.21.15.45:6833","nonce":440031651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:51:45.400395+0000","modified":"2023-11-28T21:51:48.467537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9660":{"gid":9660,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/99455191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":99455191},{"type":"v1","addr":"172.21.15.45:6835","nonce":99455191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-28T21:52:06.524 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 84 2023-11-28T21:52:06.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7ff50c06af90 msgr2=0x7ff50c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:06.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7ff50c06af90 0x7ff50c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c15d550 msgr2=0x7ff52c0ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c15d550 0x7ff52c0ccd80 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7ff524007e60 tx=0x7ff52400e4b0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 shutdown_connections 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7ff50c06af90 0x7ff50c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff52c15d550 0x7ff52c0ccd80 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c0d1ab0 0x7ff52c0cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 --2- 172.21.15.45:0/1142780198 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff52c0d0100 0x7ff52c1e8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 >> 172.21.15.45:0/1142780198 conn(0x7ff52c06ac70 msgr2=0x7ff52c0d2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 shutdown_connections 2023-11-28T21:52:06.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.524+0000 7ff52bfff640 1 -- 172.21.15.45:0/1142780198 wait complete. 2023-11-28T21:52:06.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:52:06.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.729+0000 7fd7b98d7640 1 Processor -- start 2023-11-28T21:52:06.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.729+0000 7fd7b98d7640 1 -- start start 2023-11-28T21:52:06.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.731+0000 7fd7b98d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7b40b35b0 0x7fd7b415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.731+0000 7fd7b98d7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b415bda0 con 0x7fd7b415b850 2023-11-28T21:52:06.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.731+0000 7fd7b98d7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b415c080 con 0x7fd7b40b1340 2023-11-28T21:52:06.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.731+0000 7fd7b98d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b415ee40 con 0x7fd7b40b35b0 2023-11-28T21:52:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.731+0000 7fd7abfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd7b415b850 0x7fd7b40b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50322/0 (socket says 172.21.15.45:50322) 2023-11-28T21:52:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.731+0000 7fd7abfff640 1 -- 172.21.15.45:0/3528453429 learned_addr learned my addr 172.21.15.45:0/3528453429 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.731+0000 7fd7b2ffd640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7b40b35b0 0x7fd7b415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504243160 0 0) 0x7fd7b415bda0 con 0x7fd7b415b850 2023-11-28T21:52:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a0003520 con 0x7fd7b415b850 2023-11-28T21:52:06.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578031539 0 0) 0x7fd7b415c080 con 0x7fd7b40b1340 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b415bda0 con 0x7fd7b40b1340 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474065823 0 0) 0x7fd7a0003520 con 0x7fd7b415b850 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7b40b35b0 msgr2=0x7fd7b415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7b40b35b0 0x7fd7b415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 >> v1:172.21.15.190:6789/0 conn(0x7fd7b40b1340 legacy=0x7fd7b415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b415c360 con 0x7fd7b415b850 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd79c001670 con 0x7fd7b415b850 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 >> v1:172.21.15.45:6789/0 conn(0x7fd7b415b850 legacy=0x7fd7b40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a0003b30 0x7fd7a0003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7b40b35b0 0x7fd7a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7a0004d00 0x7fd7a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b415ee40 con 0x7fd7a0004d00 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a000ba30 con 0x7fd7a0003b30 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.732+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a000bbf0 con 0x7fd7b40b35b0 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.733+0000 7fd7b37fe640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7a0004d00 0x7fd7a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.733+0000 7fd7b2ffd640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a0003b30 0x7fd7a0003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.733+0000 7fd7abfff640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7b40b35b0 0x7fd7a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.733+0000 7fd7b37fe640 1 -- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7b40b35b0 msgr2=0x7fd7a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.733+0000 7fd7b37fe640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7b40b35b0 0x7fd7a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.733+0000 7fd7b37fe640 1 -- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a0003b30 msgr2=0x7fd7a0003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.733+0000 7fd7b37fe640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a0003b30 0x7fd7a0003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.733+0000 7fd7b37fe640 1 -- 172.21.15.45:0/3528453429 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0003520 con 0x7fd7a0004d00 2023-11-28T21:52:06.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.733+0000 7fd7abfff640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7b40b35b0 0x7fd7a00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:06.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.734+0000 7fd7b37fe640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7a0004d00 0x7fd7a0005180 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac001fd0 tx=0x7fd7a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145b9577bff5f339 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.734+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac003b70 con 0x7fd7a0004d00 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.734+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ac003cd0 con 0x7fd7a0004d00 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.734+0000 7fd7b27fc640 1 -- 172.21.15.45:0/3528453429 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0052f0 con 0x7fd7a0004d00 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.735+0000 7fd7b98d7640 1 -- 172.21.15.45:0/3528453429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7a0004d00 msgr2=0x7fd7a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.735+0000 7fd7b98d7640 1 --2- 172.21.15.45:0/3528453429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7a0004d00 0x7fd7a0005180 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac001fd0 tx=0x7fd7a00057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.735+0000 7fd7b37fe640 1 -- 172.21.15.45:0/3528453429 reap_dead start 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.735+0000 7fd7b98d7640 1 -- 172.21.15.45:0/3528453429 shutdown_connections 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.735+0000 7fd7b98d7640 1 -- 172.21.15.45:0/3528453429 >> 172.21.15.45:0/3528453429 conn(0x7fd7b406a500 msgr2=0x7fd7b406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.735+0000 7fd7b98d7640 1 -- 172.21.15.45:0/3528453429 shutdown_connections 2023-11-28T21:52:06.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.735+0000 7fd7b98d7640 1 -- 172.21.15.45:0/3528453429 wait complete. 2023-11-28T21:52:06.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.737+0000 7fd7b98d7640 1 Processor -- start 2023-11-28T21:52:06.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.737+0000 7fd7b98d7640 1 -- start start 2023-11-28T21:52:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.738+0000 7fd7b98d7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0004d00 0x7fd7b40a8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.738+0000 7fd7b98d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b40ad7f0 0x7fd7b40adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.738+0000 7fd7b98d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7b413e670 0x7fd7b413eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.738+0000 7fd7b98d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b413ffb0 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.738+0000 7fd7b98d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b40ae1b0 con 0x7fd7b413e670 2023-11-28T21:52:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.738+0000 7fd7b98d7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b413f3a0 con 0x7fd7a0004d00 2023-11-28T21:52:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.738+0000 7fd7abfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b40ad7f0 0x7fd7b40adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.738+0000 7fd7abfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b40ad7f0 0x7fd7b40adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40244/0 (socket says 172.21.15.45:40244) 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.738+0000 7fd7abfff640 1 -- 172.21.15.45:0/1199378540 learned_addr learned my addr 172.21.15.45:0/1199378540 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.739+0000 7fd7b2ffd640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0004d00 0x7fd7b40a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.739+0000 7fd7b37fe640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7b413e670 0x7fd7b413eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.739+0000 7fd7abfff640 1 -- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0004d00 msgr2=0x7fd7b40a8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.739+0000 7fd7abfff640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0004d00 0x7fd7b40a8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.739+0000 7fd7abfff640 1 -- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7b413e670 msgr2=0x7fd7b413eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.739+0000 7fd7abfff640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7b413e670 0x7fd7b413eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.739+0000 7fd7abfff640 1 -- 172.21.15.45:0/1199378540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b413f680 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.739+0000 7fd7b37fe640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7b413e670 0x7fd7b413eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.739+0000 7fd7abfff640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b40ad7f0 0x7fd7b40adc70 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00d780 tx=0x7fd79c00dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.740+0000 7fd7b0ff9640 1 -- 172.21.15.45:0/1199378540 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0074e0 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.740+0000 7fd7b0ff9640 1 -- 172.21.15.45:0/1199378540 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c00de90 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.740+0000 7fd7b0ff9640 1 -- 172.21.15.45:0/1199378540 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79c004a00 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.740+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b413f960 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.740+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b413f030 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.741+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b4140160 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.743+0000 7fd7b0ff9640 1 -- 172.21.15.45:0/1199378540 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 95) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fd79c004b60 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.745+0000 7fd7b0ff9640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd79406af60 0x7fd79406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.745+0000 7fd7b0ff9640 1 -- 172.21.15.45:0/1199378540 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd79c089480 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.745+0000 7fd7b2ffd640 1 -- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd79406af60 msgr2=0x7fd79406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:06.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.745+0000 7fd7b2ffd640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd79406af60 0x7fd79406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:06.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.749+0000 7fd7b0ff9640 1 -- 172.21.15.45:0/1199378540 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7b4140160 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.812 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:06.811+0000 7fc35142c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T21:52:06.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.944+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7b40733d0 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.946+0000 7fd7b2ffd640 1 -- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd79406af60 msgr2=0x7fd79406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.946+0000 7fd7b2ffd640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd79406af60 0x7fd79406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:52:06.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.946+0000 7fd7b0ff9640 1 -- 172.21.15.45:0/1199378540 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+17264 (secure 0 0 0) 0x7fd79c054040 con 0x7fd7b40ad7f0 2023-11-28T21:52:06.949 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:06.949 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":186,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:52:00.068817+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:51:43.335999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-11-28T21:51:44.431458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/2404812260":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2400910006":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6833/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2240536749":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6832/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2615023161":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/4027676180":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/508215298":"2023-11-29T21:52:00.068699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:52:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd79406af60 msgr2=0x7fd79406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd79406af60 0x7fd79406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b40ad7f0 msgr2=0x7fd7b40adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b40ad7f0 0x7fd7b40adc70 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00d780 tx=0x7fd79c00dc40 comp rx=0 tx=0).stop 2023-11-28T21:52:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 shutdown_connections 2023-11-28T21:52:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7b413e670 0x7fd7b413eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fd79406af60 0x7fd79406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7b40ad7f0 0x7fd7b40adc70 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 --2- 172.21.15.45:0/1199378540 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0004d00 0x7fd7b40a8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 >> 172.21.15.45:0/1199378540 conn(0x7fd7b406a500 msgr2=0x7fd7b415d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 shutdown_connections 2023-11-28T21:52:06.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:06.949+0000 7fd7b98d7640 1 -- 172.21.15.45:0/1199378540 wait complete. 2023-11-28T21:52:06.968 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T21:51:43.335999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T21:51:44.431458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T21:52:06.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T21:52:07.048 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:07.046+0000 7f09db42b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T21:52:07.112 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.111+0000 7fc35142c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T21:52:07.170 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.168+0000 7fc35142c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T21:52:07.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.172+0000 7fdaf4bae640 1 Processor -- start 2023-11-28T21:52:07.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.172+0000 7fdaf4bae640 1 -- start start 2023-11-28T21:52:07.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.174+0000 7fdaf4bae640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdaf015f050 0x7fdaf01625a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.174+0000 7fdaf4bae640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdaf00b4600 0x7fdaf00b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.174+0000 7fdaf4bae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf006b510 0x7fdaf006b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.174+0000 7fdaf4bae640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf0162ae0 con 0x7fdaf015f050 2023-11-28T21:52:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.174+0000 7fdaf4bae640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf006bea0 con 0x7fdaf00b4600 2023-11-28T21:52:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.174+0000 7fdaf4bae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00b3e60 con 0x7fdaf006b510 2023-11-28T21:52:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdaef7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdaf015f050 0x7fdaf01625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdaef7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdaf015f050 0x7fdaf01625a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40256/0 (socket says 172.21.15.45:40256) 2023-11-28T21:52:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdaef7fe640 1 -- 172.21.15.45:0/3327731290 learned_addr learned my addr 172.21.15.45:0/3327731290 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdae7fff640 1 --2- 172.21.15.45:0/3327731290 >> v2:172.21.15.190:3300/0 conn(0x7fdaf00b4600 0x7fdaf00b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdaeffff640 1 --2- 172.21.15.45:0/3327731290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf006b510 0x7fdaf006b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdaef7fe640 1 -- 172.21.15.45:0/3327731290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf006b510 msgr2=0x7fdaf006b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdaef7fe640 1 --2- 172.21.15.45:0/3327731290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf006b510 0x7fdaf006b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdaef7fe640 1 -- 172.21.15.45:0/3327731290 >> v2:172.21.15.190:3300/0 conn(0x7fdaf00b4600 msgr2=0x7fdaf00b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:07.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdaef7fe640 1 --2- 172.21.15.45:0/3327731290 >> v2:172.21.15.190:3300/0 conn(0x7fdaf00b4600 0x7fdaf00b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdaef7fe640 1 -- 172.21.15.45:0/3327731290 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf00b4b80 con 0x7fdaf015f050 2023-11-28T21:52:07.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.175+0000 7fdae7fff640 1 --2- 172.21.15.45:0/3327731290 >> v2:172.21.15.190:3300/0 conn(0x7fdaf00b4600 0x7fdaf00b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:07.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.176+0000 7fdaef7fe640 1 --2- 172.21.15.45:0/3327731290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf015f050 0x7fdaf01625a0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00726c0 tx=0x7fdae002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9e777c9cf75665 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:07.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.177+0000 7fdaeeffd640 1 -- 172.21.15.45:0/3327731290 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0031870 con 0x7fdaf015f050 2023-11-28T21:52:07.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.177+0000 7fdaeeffd640 1 -- 172.21.15.45:0/3327731290 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae00319d0 con 0x7fdaf015f050 2023-11-28T21:52:07.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.177+0000 7fdaeeffd640 1 -- 172.21.15.45:0/3327731290 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0031d60 con 0x7fdaf015f050 2023-11-28T21:52:07.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.177+0000 7fdaf4bae640 1 -- 172.21.15.45:0/3327731290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf015f050 msgr2=0x7fdaf01625a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:07.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.177+0000 7fdaf4bae640 1 --2- 172.21.15.45:0/3327731290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf015f050 0x7fdaf01625a0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00726c0 tx=0x7fdae002b910 comp rx=0 tx=0).stop 2023-11-28T21:52:07.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.178+0000 7fdaf4bae640 1 -- 172.21.15.45:0/3327731290 shutdown_connections 2023-11-28T21:52:07.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.178+0000 7fdaf4bae640 1 --2- 172.21.15.45:0/3327731290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf006b510 0x7fdaf006b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.178+0000 7fdaf4bae640 1 --2- 172.21.15.45:0/3327731290 >> v2:172.21.15.190:3300/0 conn(0x7fdaf00b4600 0x7fdaf00b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.178+0000 7fdaf4bae640 1 --2- 172.21.15.45:0/3327731290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf015f050 0x7fdaf01625a0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.178+0000 7fdaf4bae640 1 -- 172.21.15.45:0/3327731290 >> 172.21.15.45:0/3327731290 conn(0x7fdaf0071240 msgr2=0x7fdaf0071650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:07.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.178+0000 7fdaf4bae640 1 -- 172.21.15.45:0/3327731290 shutdown_connections 2023-11-28T21:52:07.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.178+0000 7fdaf4bae640 1 -- 172.21.15.45:0/3327731290 wait complete. 2023-11-28T21:52:07.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.179+0000 7fdaf4bae640 1 Processor -- start 2023-11-28T21:52:07.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.180+0000 7fdaf4bae640 1 -- start start 2023-11-28T21:52:07.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.180+0000 7fdaf4bae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf006b510 0x7fdaf006f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.180+0000 7fdaf4bae640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaf00b4600 0x7fdaf009a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.180+0000 7fdaf4bae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf015f050 0x7fdaf015b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaf4bae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf009ade0 con 0x7fdaf006b510 2023-11-28T21:52:07.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaf4bae640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf009b040 con 0x7fdaf00b4600 2023-11-28T21:52:07.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaf4bae640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf009b320 con 0x7fdaf015f050 2023-11-28T21:52:07.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf006b510 0x7fdaf006f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf006b510 0x7fdaf006f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40262/0 (socket says 172.21.15.45:40262) 2023-11-28T21:52:07.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaef7fe640 1 -- 172.21.15.45:0/54356294 learned_addr learned my addr 172.21.15.45:0/54356294 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:07.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdae7fff640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaf00b4600 0x7fdaf009a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaeffff640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf015f050 0x7fdaf015b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaef7fe640 1 -- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf015f050 msgr2=0x7fdaf015b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaef7fe640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf015f050 0x7fdaf015b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaef7fe640 1 -- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaf00b4600 msgr2=0x7fdaf009a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaef7fe640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaf00b4600 0x7fdaf009a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaef7fe640 1 -- 172.21.15.45:0/54356294 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae00094d0 con 0x7fdaf006b510 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.181+0000 7fdaeffff640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf015f050 0x7fdaf015b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.182+0000 7fdaef7fe640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf006b510 0x7fdaf006f5d0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fdae002b950 tx=0x7fdae00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.182+0000 7fdaecff9640 1 -- 172.21.15.45:0/54356294 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae003ad30 con 0x7fdaf006b510 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.182+0000 7fdaecff9640 1 -- 172.21.15.45:0/54356294 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae00443f0 con 0x7fdaf006b510 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.182+0000 7fdaecff9640 1 -- 172.21.15.45:0/54356294 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae00447a0 con 0x7fdaf006b510 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.182+0000 7fdaf4bae640 1 -- 172.21.15.45:0/54356294 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf006d0e0 con 0x7fdaf006b510 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.182+0000 7fdaf4bae640 1 -- 172.21.15.45:0/54356294 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf006d6a0 con 0x7fdaf006b510 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.183+0000 7fdaf4bae640 1 -- 172.21.15.45:0/54356294 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf00b1850 con 0x7fdaf006b510 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.184+0000 7fdaecff9640 1 -- 172.21.15.45:0/54356294 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 95) v1 ==== 89334+0+0 (secure 0 0 0) 0x7fdae004d450 con 0x7fdaf006b510 2023-11-28T21:52:07.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.185+0000 7fdaecff9640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fdad006af60 0x7fdad006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.185+0000 7fdaecff9640 1 -- 172.21.15.45:0/54356294 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fdae002d040 con 0x7fdaf006b510 2023-11-28T21:52:07.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.186+0000 7fdae7fff640 1 -- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fdad006af60 msgr2=0x7fdad006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/1285100946 2023-11-28T21:52:07.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.186+0000 7fdae7fff640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fdad006af60 0x7fdad006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:07.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.189+0000 7fdaecff9640 1 -- 172.21.15.45:0/54356294 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdae0076ca0 con 0x7fdaf006b510 2023-11-28T21:52:07.284 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.283+0000 7fc35142c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T21:52:07.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.306+0000 7fdaecff9640 1 -- 172.21.15.45:0/54356294 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 96) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fdae00af030 con 0x7fdaf006b510 2023-11-28T21:52:07.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.307+0000 7fdaecff9640 1 -- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fdad006af60 msgr2=0x7fdad006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:07.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.307+0000 7fdaecff9640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fdad006af60 0x7fdad006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.371 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d16a3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.371 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d16a3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.371 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d16a3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.371 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d16a3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.371 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d16a3640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.371 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d5eac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.371 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d5eac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.371 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d5eac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.372 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d5eac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.372 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:07.370+0000 7fc2d5eac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:07.380 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:07.378+0000 7f09db42b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T21:52:07.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.396+0000 7fdaf4bae640 1 -- 172.21.15.45:0/54356294 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdaf009e100 con 0x7fdaf006b510 2023-11-28T21:52:07.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.397+0000 7fdaecff9640 1 -- 172.21.15.45:0/54356294 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v187) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdae0090dc0 con 0x7fdaf006b510 2023-11-28T21:52:07.399 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.399+0000 7fdaf4bae640 1 -- 172.21.15.45:0/54356294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf006b510 msgr2=0x7fdaf006f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.399+0000 7fdaf4bae640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf006b510 0x7fdaf006f5d0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fdae002b950 tx=0x7fdae00046c0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.399+0000 7fdaf4bae640 1 -- 172.21.15.45:0/54356294 shutdown_connections 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.399+0000 7fdaf4bae640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdaf015f050 0x7fdaf015b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.399+0000 7fdaf4bae640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaf00b4600 0x7fdaf009a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.399+0000 7fdaf4bae640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.190:6832/1285100946,v1:172.21.15.190:6833/1285100946] conn(0x7fdad006af60 0x7fdad006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.399+0000 7fdaf4bae640 1 --2- 172.21.15.45:0/54356294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf006b510 0x7fdaf006f5d0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.399+0000 7fdaf4bae640 1 -- 172.21.15.45:0/54356294 >> 172.21.15.45:0/54356294 conn(0x7fdaf0071240 msgr2=0x7fdaf0071620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.399+0000 7fdaf4bae640 1 -- 172.21.15.45:0/54356294 shutdown_connections 2023-11-28T21:52:07.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.400+0000 7fdaf4bae640 1 -- 172.21.15.45:0/54356294 wait complete. 2023-11-28T21:52:07.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T21:52:07.442 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:07.440+0000 7f09db42b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T21:52:07.563 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:07.561+0000 7f09db42b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T21:52:07.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.615+0000 7f3c95eef640 1 Processor -- start 2023-11-28T21:52:07.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.615+0000 7f3c95eef640 1 -- start start 2023-11-28T21:52:07.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.616+0000 7f3c95eef640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3c90159720 0x7f3c900b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.616+0000 7f3c95eef640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3c90071b90 0x7f3c90071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.616+0000 7f3c95eef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c900cc120 con 0x7f3c9006dfa0 2023-11-28T21:52:07.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c95eef640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9006e430 con 0x7f3c90071b90 2023-11-28T21:52:07.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c95eef640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c900724e0 con 0x7f3c90159720 2023-11-28T21:52:07.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c8ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3c9006dfa0 0x7f3c900701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50332/0 (socket says 172.21.15.45:50332) 2023-11-28T21:52:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c8ffff640 1 -- 172.21.15.45:0/1338948021 learned_addr learned my addr 172.21.15.45:0/1338948021 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c956ee640 1 --2- 172.21.15.45:0/1338948021 >> v2:172.21.15.45:3300/0 conn(0x7f3c90071b90 0x7f3c90071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c94eed640 1 --2- 172.21.15.45:0/1338948021 >> v2:172.21.15.190:3300/0 conn(0x7f3c90159720 0x7f3c900b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c956ee640 1 -- 172.21.15.45:0/1338948021 >> v2:172.21.15.190:3300/0 conn(0x7f3c90159720 msgr2=0x7f3c900b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c956ee640 1 --2- 172.21.15.45:0/1338948021 >> v2:172.21.15.190:3300/0 conn(0x7f3c90159720 0x7f3c900b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c956ee640 1 -- 172.21.15.45:0/1338948021 >> v1:172.21.15.45:6789/0 conn(0x7f3c9006dfa0 legacy=0x7f3c900701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c956ee640 1 -- 172.21.15.45:0/1338948021 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c900b2420 con 0x7f3c90071b90 2023-11-28T21:52:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c94eed640 1 --2- 172.21.15.45:0/1338948021 >> v2:172.21.15.190:3300/0 conn(0x7f3c90159720 0x7f3c900b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.617+0000 7f3c8f7fe640 1 -- 172.21.15.45:0/1338948021 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686313298 0 0) 0x7f3c7c001110 con 0x7f3c9006dfa0 2023-11-28T21:52:07.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.618+0000 7f3c956ee640 1 --2- 172.21.15.45:0/1338948021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c90071b90 0x7f3c90071fa0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f3c900cc2c0 tx=0x7f3c8802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111dddafbb133670 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:07.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c8f7fe640 1 -- 172.21.15.45:0/1338948021 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8802cc70 con 0x7f3c90071b90 2023-11-28T21:52:07.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c8f7fe640 1 -- 172.21.15.45:0/1338948021 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8802cdd0 con 0x7f3c90071b90 2023-11-28T21:52:07.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c8f7fe640 1 -- 172.21.15.45:0/1338948021 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88034660 con 0x7f3c90071b90 2023-11-28T21:52:07.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c95eef640 1 -- 172.21.15.45:0/1338948021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c90071b90 msgr2=0x7f3c90071fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c95eef640 1 --2- 172.21.15.45:0/1338948021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c90071b90 0x7f3c90071fa0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f3c900cc2c0 tx=0x7f3c8802bec0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c95eef640 1 -- 172.21.15.45:0/1338948021 shutdown_connections 2023-11-28T21:52:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c95eef640 1 --2- 172.21.15.45:0/1338948021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c90071b90 0x7f3c90071fa0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c95eef640 1 --2- 172.21.15.45:0/1338948021 >> v2:172.21.15.190:3300/0 conn(0x7f3c90159720 0x7f3c900b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c95eef640 1 -- 172.21.15.45:0/1338948021 >> 172.21.15.45:0/1338948021 conn(0x7f3c900b1050 msgr2=0x7f3c900cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c95eef640 1 -- 172.21.15.45:0/1338948021 shutdown_connections 2023-11-28T21:52:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.619+0000 7f3c95eef640 1 -- 172.21.15.45:0/1338948021 wait complete. 2023-11-28T21:52:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.621+0000 7f3c95eef640 1 Processor -- start 2023-11-28T21:52:07.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.621+0000 7f3c95eef640 1 -- start start 2023-11-28T21:52:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.622+0000 7f3c95eef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9006dfa0 0x7f3c90158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.622+0000 7f3c95eef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c90071b90 0x7f3c900cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.622+0000 7f3c95eef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c90159720 0x7f3c900cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.622+0000 7f3c95eef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90158da0 con 0x7f3c9006dfa0 2023-11-28T21:52:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.622+0000 7f3c95eef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90147b50 con 0x7f3c90159720 2023-11-28T21:52:07.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.622+0000 7f3c95eef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90159f40 con 0x7f3c90071b90 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.622+0000 7f3c94eed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9006dfa0 0x7f3c90158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.622+0000 7f3c94eed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9006dfa0 0x7f3c90158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40284/0 (socket says 172.21.15.45:40284) 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.622+0000 7f3c94eed640 1 -- 172.21.15.45:0/497427233 learned_addr learned my addr 172.21.15.45:0/497427233 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c8ffff640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c90071b90 0x7f3c900cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c956ee640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c90159720 0x7f3c900cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c94eed640 1 -- 172.21.15.45:0/497427233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c90071b90 msgr2=0x7f3c900cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c94eed640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c90071b90 0x7f3c900cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c94eed640 1 -- 172.21.15.45:0/497427233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c90159720 msgr2=0x7f3c900cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c94eed640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c90159720 0x7f3c900cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c94eed640 1 -- 172.21.15.45:0/497427233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c88009a70 con 0x7f3c9006dfa0 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c8ffff640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c90071b90 0x7f3c900cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:07.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c94eed640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9006dfa0 0x7f3c90158860 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400c7f0 tx=0x7f3c8400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c8d7fa640 1 -- 172.21.15.45:0/497427233 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c84007820 con 0x7f3c9006dfa0 2023-11-28T21:52:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c8d7fa640 1 -- 172.21.15.45:0/497427233 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c84007980 con 0x7f3c9006dfa0 2023-11-28T21:52:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c8d7fa640 1 -- 172.21.15.45:0/497427233 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c84007cf0 con 0x7f3c9006dfa0 2023-11-28T21:52:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c95eef640 1 -- 172.21.15.45:0/497427233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9015a220 con 0x7f3c9006dfa0 2023-11-28T21:52:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.623+0000 7f3c95eef640 1 -- 172.21.15.45:0/497427233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c90148080 con 0x7f3c9006dfa0 2023-11-28T21:52:07.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.625+0000 7f3c95eef640 1 -- 172.21.15.45:0/497427233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9009d0f0 con 0x7f3c9006dfa0 2023-11-28T21:52:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.625+0000 7f3c8d7fa640 1 -- 172.21.15.45:0/497427233 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 96) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f3c840113f0 con 0x7f3c9006dfa0 2023-11-28T21:52:07.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.626+0000 7f3c8d7fa640 1 -- 172.21.15.45:0/497427233 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f3c840898d0 con 0x7f3c9006dfa0 2023-11-28T21:52:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.631+0000 7f3c8d7fa640 1 -- 172.21.15.45:0/497427233 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c840548d0 con 0x7f3c9006dfa0 2023-11-28T21:52:07.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.823+0000 7f3c95eef640 1 -- 172.21.15.45:0/497427233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3c90158f40 con 0x7f3c9006dfa0 2023-11-28T21:52:07.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.824+0000 7f3c8d7fa640 1 -- 172.21.15.45:0/497427233 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v187) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3c84057ef0 con 0x7f3c9006dfa0 2023-11-28T21:52:07.826 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T21:52:07.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.827+0000 7f3c95eef640 1 -- 172.21.15.45:0/497427233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9006dfa0 msgr2=0x7f3c90158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:07.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.827+0000 7f3c95eef640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9006dfa0 0x7f3c90158860 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400c7f0 tx=0x7f3c8400ccb0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.827+0000 7f3c95eef640 1 -- 172.21.15.45:0/497427233 shutdown_connections 2023-11-28T21:52:07.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.827+0000 7f3c95eef640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c90159720 0x7f3c900cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.827+0000 7f3c95eef640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c90071b90 0x7f3c900cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.827+0000 7f3c95eef640 1 --2- 172.21.15.45:0/497427233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9006dfa0 0x7f3c90158860 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:07.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.827+0000 7f3c95eef640 1 -- 172.21.15.45:0/497427233 >> 172.21.15.45:0/497427233 conn(0x7f3c900b1050 msgr2=0x7f3c9006bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:07.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.827+0000 7f3c95eef640 1 -- 172.21.15.45:0/497427233 shutdown_connections 2023-11-28T21:52:07.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:07.827+0000 7f3c95eef640 1 -- 172.21.15.45:0/497427233 wait complete. 2023-11-28T21:52:07.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T21:52:08.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.128+0000 7fd09bfff640 1 Processor -- start 2023-11-28T21:52:08.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.128+0000 7fd09bfff640 1 -- start start 2023-11-28T21:52:08.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.130+0000 7fd09bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd09c15b770 0x7fd09c06b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.130+0000 7fd09bfff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c06beb0 con 0x7fd09c15ac80 2023-11-28T21:52:08.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.130+0000 7fd09bfff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c06d9a0 con 0x7fd09c06b510 2023-11-28T21:52:08.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.130+0000 7fd09bfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c06f0f0 con 0x7fd09c15b770 2023-11-28T21:52:08.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.130+0000 7fd09affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd09c15ac80 0x7fd09c06d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50334/0 (socket says 172.21.15.45:50334) 2023-11-28T21:52:08.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.130+0000 7fd09affd640 1 -- 172.21.15.45:0/587531259 learned_addr learned my addr 172.21.15.45:0/587531259 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:08.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.131+0000 7fd09b7fe640 1 --2- 172.21.15.45:0/587531259 >> v2:172.21.15.45:3300/0 conn(0x7fd09c15b770 0x7fd09c06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.131+0000 7fd09a7fc640 1 -- 172.21.15.45:0/587531259 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910188048 0 0) 0x7fd09c06beb0 con 0x7fd09c15ac80 2023-11-28T21:52:08.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.131+0000 7fd09a7fc640 1 -- 172.21.15.45:0/587531259 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd088003520 con 0x7fd09c15ac80 2023-11-28T21:52:08.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.131+0000 7fd09b7fe640 1 -- 172.21.15.45:0/587531259 >> v1:172.21.15.190:6789/0 conn(0x7fd09c06b510 legacy=0x7fd09c15b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.131+0000 7fd09b7fe640 1 -- 172.21.15.45:0/587531259 >> v1:172.21.15.45:6789/0 conn(0x7fd09c15ac80 legacy=0x7fd09c06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.131+0000 7fd09b7fe640 1 -- 172.21.15.45:0/587531259 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c06f3d0 con 0x7fd09c15b770 2023-11-28T21:52:08.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.133+0000 7fd09b7fe640 1 --2- 172.21.15.45:0/587531259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15b770 0x7fd09c06b970 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fd0940097a0 tx=0x7fd09402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6759e160de5139a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:08.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.133+0000 7fd09a7fc640 1 -- 172.21.15.45:0/587531259 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094030610 con 0x7fd09c15b770 2023-11-28T21:52:08.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.133+0000 7fd09a7fc640 1 -- 172.21.15.45:0/587531259 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd094030770 con 0x7fd09c15b770 2023-11-28T21:52:08.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.133+0000 7fd09a7fc640 1 -- 172.21.15.45:0/587531259 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094030b00 con 0x7fd09c15b770 2023-11-28T21:52:08.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.134+0000 7fd09bfff640 1 -- 172.21.15.45:0/587531259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15b770 msgr2=0x7fd09c06b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.134+0000 7fd09bfff640 1 --2- 172.21.15.45:0/587531259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15b770 0x7fd09c06b970 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fd0940097a0 tx=0x7fd09402b960 comp rx=0 tx=0).stop 2023-11-28T21:52:08.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.134+0000 7fd09bfff640 1 -- 172.21.15.45:0/587531259 shutdown_connections 2023-11-28T21:52:08.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.134+0000 7fd09bfff640 1 --2- 172.21.15.45:0/587531259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15b770 0x7fd09c06b970 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.134+0000 7fd09bfff640 1 -- 172.21.15.45:0/587531259 >> 172.21.15.45:0/587531259 conn(0x7fd09c071290 msgr2=0x7fd09c0716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:08.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.134+0000 7fd09bfff640 1 -- 172.21.15.45:0/587531259 shutdown_connections 2023-11-28T21:52:08.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.134+0000 7fd09bfff640 1 -- 172.21.15.45:0/587531259 wait complete. 2023-11-28T21:52:08.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.135+0000 7fd09bfff640 1 Processor -- start 2023-11-28T21:52:08.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.136+0000 7fd09bfff640 1 -- start start 2023-11-28T21:52:08.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.136+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c06b510 0x7fd09c159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.136+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15ac80 0x7fd09c153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09c15b770 0x7fd09c1506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd09bfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c159a90 con 0x7fd09c15ac80 2023-11-28T21:52:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd09bfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c158790 con 0x7fd09c15b770 2023-11-28T21:52:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd09bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c158a70 con 0x7fd09c06b510 2023-11-28T21:52:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd0927fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15ac80 0x7fd09c153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd0927fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15ac80 0x7fd09c153f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40304/0 (socket says 172.21.15.45:40304) 2023-11-28T21:52:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd0927fc640 1 -- 172.21.15.45:0/3078373424 learned_addr learned my addr 172.21.15.45:0/3078373424 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd09affd640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c06b510 0x7fd09c159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd09b7fe640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09c15b770 0x7fd09c1506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd0927fc640 1 -- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c06b510 msgr2=0x7fd09c159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd0927fc640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c06b510 0x7fd09c159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd0927fc640 1 -- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09c15b770 msgr2=0x7fd09c1506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd0927fc640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09c15b770 0x7fd09c1506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd0927fc640 1 -- 172.21.15.45:0/3078373424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd094009400 con 0x7fd09c15ac80 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.137+0000 7fd09b7fe640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09c15b770 0x7fd09c1506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.138+0000 7fd0927fc640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15ac80 0x7fd09c153f50 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fd08400c400 tx=0x7fd08400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.138+0000 7fd093fff640 1 -- 172.21.15.45:0/3078373424 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0840052d0 con 0x7fd09c15ac80 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.138+0000 7fd093fff640 1 -- 172.21.15.45:0/3078373424 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd084005450 con 0x7fd09c15ac80 2023-11-28T21:52:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.138+0000 7fd093fff640 1 -- 172.21.15.45:0/3078373424 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084007260 con 0x7fd09c15ac80 2023-11-28T21:52:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.138+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c158d50 con 0x7fd09c15ac80 2023-11-28T21:52:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.138+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c158440 con 0x7fd09c15ac80 2023-11-28T21:52:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.139+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c159270 con 0x7fd09c15ac80 2023-11-28T21:52:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.140+0000 7fd093fff640 1 -- 172.21.15.45:0/3078373424 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 96) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fd0840073c0 con 0x7fd09c15ac80 2023-11-28T21:52:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.141+0000 7fd093fff640 1 -- 172.21.15.45:0/3078373424 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fd0840889c0 con 0x7fd09c15ac80 2023-11-28T21:52:08.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.145+0000 7fd093fff640 1 -- 172.21.15.45:0/3078373424 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd09c159270 con 0x7fd09c15ac80 2023-11-28T21:52:08.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.310+0000 7fd093fff640 1 -- 172.21.15.45:0/3078373424 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 97) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fd084053470 con 0x7fd09c15ac80 2023-11-28T21:52:08.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.311+0000 7fd093fff640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd07c070b80 0x7fd07c072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.322+0000 7fd09affd640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd07c070b80 0x7fd07c072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.333+0000 7fd09affd640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd07c070b80 0x7fd07c072f70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd08c001560 tx=0x7fd08c009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:08.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.341+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd09c159270 con 0x7fd09c15ac80 2023-11-28T21:52:08.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.342+0000 7fd093fff640 1 -- 172.21.15.45:0/3078373424 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v187) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd09c159270 con 0x7fd09c15ac80 2023-11-28T21:52:08.346 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T21:52:08.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd07c070b80 msgr2=0x7fd07c072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd07c070b80 0x7fd07c072f70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd08c001560 tx=0x7fd08c009040 comp rx=0 tx=0).stop 2023-11-28T21:52:08.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15ac80 msgr2=0x7fd09c153f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15ac80 0x7fd09c153f50 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fd08400c400 tx=0x7fd08400c8c0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 shutdown_connections 2023-11-28T21:52:08.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd07c070b80 0x7fd07c072f70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09c15b770 0x7fd09c1506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c15ac80 0x7fd09c153f50 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3078373424 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c06b510 0x7fd09c159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.345+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 >> 172.21.15.45:0/3078373424 conn(0x7fd09c071290 msgr2=0x7fd09c0aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:08.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.346+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 shutdown_connections 2023-11-28T21:52:08.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.346+0000 7fd09bfff640 1 -- 172.21.15.45:0/3078373424 wait complete. 2023-11-28T21:52:08.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:52:08.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.583+0000 7feac8d97640 1 Processor -- start 2023-11-28T21:52:08.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.583+0000 7feac8d97640 1 -- start start 2023-11-28T21:52:08.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.585+0000 7feac8d97640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feac415b850 0x7feac40b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.585+0000 7feac8d97640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac415bda0 con 0x7feac40b1fa0 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.585+0000 7feac8d97640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac415c080 con 0x7feac40b35b0 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.585+0000 7feac8d97640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac415ee40 con 0x7feac415b850 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.585+0000 7feac2d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7feac40b1fa0 0x7feac415f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50348/0 (socket says 172.21.15.45:50348) 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.585+0000 7feac2d9d640 1 -- 172.21.15.45:0/99393159 learned_addr learned my addr 172.21.15.45:0/99393159 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881635304 0 0) 0x7feac415bda0 con 0x7feac40b1fa0 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feabbfff640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feac415b850 0x7feac40b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab0003520 con 0x7feac40b1fa0 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2538048558 0 0) 0x7feab0003520 con 0x7feac40b1fa0 2023-11-28T21:52:08.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feac415b850 msgr2=0x7feac40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feac415b850 0x7feac40b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 >> v1:172.21.15.190:6789/0 conn(0x7feac40b35b0 legacy=0x7feac415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac415c360 con 0x7feac40b1fa0 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7feabc0019f0 con 0x7feac40b1fa0 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 >> v1:172.21.15.45:6789/0 conn(0x7feac40b1fa0 legacy=0x7feac415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feab00039a0 0x7feab0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feab0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0005180 0x7feab000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac415ee40 con 0x7feab00047c0 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab000c0b0 con 0x7feab0005180 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.586+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab000c270 con 0x7feab00039a0 2023-11-28T21:52:08.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.587+0000 7feabbfff640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feab0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.587+0000 7feac259c640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feab00039a0 0x7feab0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.587+0000 7feac2d9d640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0005180 0x7feab000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.587+0000 7feabbfff640 1 -- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feab00039a0 msgr2=0x7feab0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.587+0000 7feabbfff640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feab00039a0 0x7feab0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.587+0000 7feabbfff640 1 -- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0005180 msgr2=0x7feab000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.587+0000 7feabbfff640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0005180 0x7feab000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.587+0000 7feabbfff640 1 -- 172.21.15.45:0/99393159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab0003520 con 0x7feab00047c0 2023-11-28T21:52:08.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.587+0000 7feac2d9d640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0005180 0x7feab000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:08.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.588+0000 7feabbfff640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feab0004c40 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7feac40cd380 tx=0x7feab00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14c21f5b5663b1c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:08.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac035660 con 0x7feab00047c0 2023-11-28T21:52:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac0357c0 con 0x7feab00047c0 2023-11-28T21:52:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac1d9b640 1 -- 172.21.15.45:0/99393159 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac035b70 con 0x7feab00047c0 2023-11-28T21:52:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac8d97640 1 -- 172.21.15.45:0/99393159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 msgr2=0x7feab0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac8d97640 1 --2- 172.21.15.45:0/99393159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feab0004c40 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7feac40cd380 tx=0x7feab00057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac2d9d640 1 -- 172.21.15.45:0/99393159 reap_dead start 2023-11-28T21:52:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac8d97640 1 -- 172.21.15.45:0/99393159 shutdown_connections 2023-11-28T21:52:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac8d97640 1 -- 172.21.15.45:0/99393159 >> 172.21.15.45:0/99393159 conn(0x7feac406a500 msgr2=0x7feac406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac8d97640 1 -- 172.21.15.45:0/99393159 shutdown_connections 2023-11-28T21:52:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.589+0000 7feac8d97640 1 -- 172.21.15.45:0/99393159 wait complete. 2023-11-28T21:52:08.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.591+0000 7feac8d97640 1 Processor -- start 2023-11-28T21:52:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.591+0000 7feac8d97640 1 -- start start 2023-11-28T21:52:08.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.592+0000 7feac8d97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feac41e5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.592+0000 7feac8d97640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feac41f4bf0 0x7feac4154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.592+0000 7feac8d97640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41f5070 0x7feac4154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.592+0000 7feac8d97640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41586e0 con 0x7feab00047c0 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.592+0000 7feac8d97640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41e5650 con 0x7feac41f5070 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.592+0000 7feac8d97640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac41589a0 con 0x7feac41f4bf0 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feac41e5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feac41e5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40330/0 (socket says 172.21.15.45:40330) 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac259c640 1 -- 172.21.15.45:0/2999058059 learned_addr learned my addr 172.21.15.45:0/2999058059 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac2d9d640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41f5070 0x7feac4154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feabbfff640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feac41f4bf0 0x7feac4154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac259c640 1 -- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feac41f4bf0 msgr2=0x7feac4154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac259c640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feac41f4bf0 0x7feac4154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac259c640 1 -- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41f5070 msgr2=0x7feac4154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac259c640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41f5070 0x7feac4154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac259c640 1 -- 172.21.15.45:0/2999058059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac4158c80 con 0x7feab00047c0 2023-11-28T21:52:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feac259c640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feac41e5d30 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7feab400e4b0 tx=0x7feab400e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.593+0000 7feabaffd640 1 -- 172.21.15.45:0/2999058059 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab40052e0 con 0x7feab00047c0 2023-11-28T21:52:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.594+0000 7feabaffd640 1 -- 172.21.15.45:0/2999058059 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab4005440 con 0x7feab00047c0 2023-11-28T21:52:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.594+0000 7feabaffd640 1 -- 172.21.15.45:0/2999058059 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab4003830 con 0x7feab00047c0 2023-11-28T21:52:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.594+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac41591a0 con 0x7feab00047c0 2023-11-28T21:52:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.594+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac4159630 con 0x7feab00047c0 2023-11-28T21:52:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.595+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac4158540 con 0x7feab00047c0 2023-11-28T21:52:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.596+0000 7feabaffd640 1 -- 172.21.15.45:0/2999058059 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 97) v1 ==== 89069+0+0 (secure 0 0 0) 0x7feab4003990 con 0x7feab00047c0 2023-11-28T21:52:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.597+0000 7feabaffd640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7feaa406ac90 0x7feaa406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.597+0000 7feabaffd640 1 -- 172.21.15.45:0/2999058059 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6745+0+0 (secure 0 0 0) 0x7feab40894a0 con 0x7feab00047c0 2023-11-28T21:52:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.598+0000 7feabbfff640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7feaa406ac90 0x7feaa406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.599+0000 7feabbfff640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7feaa406ac90 0x7feaa406d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feab00057a0 tx=0x7feaac037040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:08.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.601+0000 7feabaffd640 1 -- 172.21.15.45:0/2999058059 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feac4158540 con 0x7feab00047c0 2023-11-28T21:52:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.834+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feac4158540 con 0x7feab00047c0 2023-11-28T21:52:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.835+0000 7feabaffd640 1 -- 172.21.15.45:0/2999058059 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4908 (secure 0 0 0) 0x7feac4158540 con 0x7feab00047c0 2023-11-28T21:52:08.838 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:08.838 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7130,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/500262407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":500262407},{"type":"v1","addr":"172.21.15.190:6835","nonce":500262407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/195287870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":195287870},{"type":"v1","addr":"172.21.15.190:6837","nonce":195287870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":9657,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/440031651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":440031651},{"type":"v1","addr":"172.21.15.45:6833","nonce":440031651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:51:45.400395+0000","modified":"2023-11-28T21:51:48.467537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9660":{"gid":9660,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/99455191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":99455191},{"type":"v1","addr":"172.21.15.45:6835","nonce":99455191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-28T21:52:08.838 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 84 2023-11-28T21:52:08.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7feaa406ac90 msgr2=0x7feaa406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7feaa406ac90 0x7feaa406d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feab00057a0 tx=0x7feaac037040 comp rx=0 tx=0).stop 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 msgr2=0x7feac41e5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feac41e5d30 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7feab400e4b0 tx=0x7feab400e970 comp rx=0 tx=0).stop 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 shutdown_connections 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7feaa406ac90 0x7feaa406d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41f5070 0x7feac4154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feac41f4bf0 0x7feac4154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 --2- 172.21.15.45:0/2999058059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab00047c0 0x7feac41e5d30 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 >> 172.21.15.45:0/2999058059 conn(0x7feac406a500 msgr2=0x7feac406c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 shutdown_connections 2023-11-28T21:52:08.841 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:08.838+0000 7feac8d97640 1 -- 172.21.15.45:0/2999058059 wait complete. 2023-11-28T21:52:08.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:52:09.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.085+0000 7f057de67640 1 Processor -- start 2023-11-28T21:52:09.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.085+0000 7f057de67640 1 -- start start 2023-11-28T21:52:09.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.087+0000 7f057de67640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0578155780 0x7f05780abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.087+0000 7f057de67640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ce800 0x7f05780b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.087+0000 7f057de67640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057815c3c0 con 0x7f05780ae540 2023-11-28T21:52:09.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.087+0000 7f057de67640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0578155b60 con 0x7f0578155780 2023-11-28T21:52:09.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.087+0000 7f057de67640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057815ba90 con 0x7f05780ce800 2023-11-28T21:52:09.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.088+0000 7f057ce65640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0578155780 0x7f05780abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.088+0000 7f057d666640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ce800 0x7f05780b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.088+0000 7f057ce65640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0578155780 0x7f05780abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:36324/0 (socket says 172.21.15.45:36324) 2023-11-28T21:52:09.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.088+0000 7f057ce65640 1 -- 172.21.15.45:0/3557518296 learned_addr learned my addr 172.21.15.45:0/3557518296 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:09.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.088+0000 7f057d666640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ce800 0x7f05780b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:59930/0 (socket says 172.21.15.45:59930) 2023-11-28T21:52:09.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.088+0000 7f057d666640 1 -- 172.21.15.45:0/3557518296 >> v2:172.21.15.190:3300/0 conn(0x7f0578155780 msgr2=0x7f05780abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:09.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.088+0000 7f057d666640 1 --2- 172.21.15.45:0/3557518296 >> v2:172.21.15.190:3300/0 conn(0x7f0578155780 0x7f05780abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.088+0000 7f057d666640 1 -- 172.21.15.45:0/3557518296 >> v1:172.21.15.190:6789/0 conn(0x7f05780ae540 legacy=0x7f05780ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.088+0000 7f057d666640 1 -- 172.21.15.45:0/3557518296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057815bd70 con 0x7f05780ce800 2023-11-28T21:52:09.091 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.090+0000 7f057d666640 1 --2- 172.21.15.45:0/3557518296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ce800 0x7f05780b20a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0570009d90 tx=0x7f057002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e60553af3772b0c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:09.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.090+0000 7f05777fe640 1 -- 172.21.15.45:0/3557518296 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057002cc60 con 0x7f05780ce800 2023-11-28T21:52:09.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.090+0000 7f05777fe640 1 -- 172.21.15.45:0/3557518296 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057002cdc0 con 0x7f05780ce800 2023-11-28T21:52:09.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.090+0000 7f05777fe640 1 -- 172.21.15.45:0/3557518296 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0570034660 con 0x7f05780ce800 2023-11-28T21:52:09.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.091+0000 7f057de67640 1 -- 172.21.15.45:0/3557518296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ce800 msgr2=0x7f05780b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.091+0000 7f057de67640 1 --2- 172.21.15.45:0/3557518296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ce800 0x7f05780b20a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0570009d90 tx=0x7f057002beb0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.091+0000 7f057de67640 1 -- 172.21.15.45:0/3557518296 shutdown_connections 2023-11-28T21:52:09.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.091+0000 7f057de67640 1 --2- 172.21.15.45:0/3557518296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ce800 0x7f05780b20a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.091+0000 7f057de67640 1 --2- 172.21.15.45:0/3557518296 >> v2:172.21.15.190:3300/0 conn(0x7f0578155780 0x7f05780abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.091+0000 7f057de67640 1 -- 172.21.15.45:0/3557518296 >> 172.21.15.45:0/3557518296 conn(0x7f057806ab30 msgr2=0x7f05780b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:09.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.091+0000 7f057de67640 1 -- 172.21.15.45:0/3557518296 shutdown_connections 2023-11-28T21:52:09.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.091+0000 7f057de67640 1 -- 172.21.15.45:0/3557518296 wait complete. 2023-11-28T21:52:09.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.093+0000 7f057de67640 1 Processor -- start 2023-11-28T21:52:09.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.093+0000 7f057de67640 1 -- start start 2023-11-28T21:52:09.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.094+0000 7f057de67640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ae540 0x7f057809c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.094+0000 7f057de67640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05780ce800 0x7f0578071b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.094+0000 7f057de67640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0578155780 0x7f057815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.094+0000 7f057de67640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057809a470 con 0x7f05780ce800 2023-11-28T21:52:09.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.094+0000 7f057de67640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057809aa30 con 0x7f0578155780 2023-11-28T21:52:09.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.094+0000 7f057de67640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057809c9f0 con 0x7f05780ae540 2023-11-28T21:52:09.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.094+0000 7f0577fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05780ce800 0x7f0578071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.094+0000 7f0577fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05780ce800 0x7f0578071b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40334/0 (socket says 172.21.15.45:40334) 2023-11-28T21:52:09.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.094+0000 7f0577fff640 1 -- 172.21.15.45:0/2665852488 learned_addr learned my addr 172.21.15.45:0/2665852488 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:09.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f057d666640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0578155780 0x7f057815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f057ce65640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ae540 0x7f057809c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f0577fff640 1 -- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ae540 msgr2=0x7f057809c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f0577fff640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ae540 0x7f057809c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f0577fff640 1 -- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0578155780 msgr2=0x7f057815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f0577fff640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0578155780 0x7f057815a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f0577fff640 1 -- 172.21.15.45:0/2665852488 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570009a70 con 0x7f05780ce800 2023-11-28T21:52:09.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f057ce65640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ae540 0x7f057809c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:09.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f0577fff640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05780ce800 0x7f0578071b40 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f056400c430 tx=0x7f056400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f05757fa640 1 -- 172.21.15.45:0/2665852488 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05640052d0 con 0x7f05780ce800 2023-11-28T21:52:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f05757fa640 1 -- 172.21.15.45:0/2665852488 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0564005450 con 0x7f05780ce800 2023-11-28T21:52:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.095+0000 7f05757fa640 1 -- 172.21.15.45:0/2665852488 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564007260 con 0x7f05780ce800 2023-11-28T21:52:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.096+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0578099a40 con 0x7f05780ce800 2023-11-28T21:52:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.096+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057809a000 con 0x7f05780ce800 2023-11-28T21:52:09.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.097+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0578151f90 con 0x7f05780ce800 2023-11-28T21:52:09.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.098+0000 7f05757fa640 1 -- 172.21.15.45:0/2665852488 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 97) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f05640073c0 con 0x7f05780ce800 2023-11-28T21:52:09.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.099+0000 7f05757fa640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f055c06ac90 0x7f055c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.099+0000 7f05757fa640 1 -- 172.21.15.45:0/2665852488 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f0564088bb0 con 0x7f05780ce800 2023-11-28T21:52:09.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.100+0000 7f057ce65640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f055c06ac90 0x7f055c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.101+0000 7f057ce65640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f055c06ac90 0x7f055c06d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f056c009a20 tx=0x7f056c008040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:09.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.103+0000 7f05757fa640 1 -- 172.21.15.45:0/2665852488 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0564053a40 con 0x7f05780ce800 2023-11-28T21:52:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.298+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f057804e920 con 0x7f05780ce800 2023-11-28T21:52:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.302+0000 7f05757fa640 1 -- 172.21.15.45:0/2665852488 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+17771 (secure 0 0 0) 0x7f0564057060 con 0x7f05780ce800 2023-11-28T21:52:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:09.305 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":187,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:52:07.287056+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:51:43.335999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-11-28T21:51:44.431458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3757577966":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/788198125":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/3142102971":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/1232828858":"2023-11-29T21:52:07.286885+0000","172.21.15.190:6833/1285100946":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/158019673":"2023-11-29T21:52:07.286885+0000","172.21.15.190:6832/1285100946":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/2404812260":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/102436699":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/2400910006":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6833/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2240536749":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6832/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2615023161":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/4027676180":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/508215298":"2023-11-29T21:52:00.068699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:52:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f055c06ac90 msgr2=0x7f055c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f055c06ac90 0x7f055c06d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f056c009a20 tx=0x7f056c008040 comp rx=0 tx=0).stop 2023-11-28T21:52:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05780ce800 msgr2=0x7f0578071b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05780ce800 0x7f0578071b40 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f056400c430 tx=0x7f056400c8f0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 shutdown_connections 2023-11-28T21:52:09.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f055c06ac90 0x7f055c06d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0578155780 0x7f057815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05780ce800 0x7f0578071b40 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 --2- 172.21.15.45:0/2665852488 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f05780ae540 0x7f057809c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 >> 172.21.15.45:0/2665852488 conn(0x7f057806ab30 msgr2=0x7f05780b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:09.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 shutdown_connections 2023-11-28T21:52:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.305+0000 7f057de67640 1 -- 172.21.15.45:0/2665852488 wait complete. 2023-11-28T21:52:09.332 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T21:52:09.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:52:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.553+0000 7f8eaacc3640 1 Processor -- start 2023-11-28T21:52:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.554+0000 7f8eaacc3640 1 -- start start 2023-11-28T21:52:09.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.555+0000 7f8eaacc3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8ea4155780 0x7f8ea40abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.555+0000 7f8eaacc3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ea40ae540 0x7f8ea40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.555+0000 7f8eaacc3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 0x7f8ea415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.555+0000 7f8eaacc3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea40ceac0 con 0x7f8ea40ae540 2023-11-28T21:52:09.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.555+0000 7f8eaacc3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea415c3c0 con 0x7f8ea4155780 2023-11-28T21:52:09.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.555+0000 7f8eaacc3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea4155b60 con 0x7f8ea40ce630 2023-11-28T21:52:09.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.555+0000 7f8ea94c0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ea40ae540 0x7f8ea40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8ea94c0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ea40ae540 0x7f8ea40ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40338/0 (socket says 172.21.15.45:40338) 2023-11-28T21:52:09.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8ea94c0640 1 -- 172.21.15.45:0/101097408 learned_addr learned my addr 172.21.15.45:0/101097408 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:09.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8ea9cc1640 1 --2- 172.21.15.45:0/101097408 >> v2:172.21.15.190:3300/0 conn(0x7f8ea4155780 0x7f8ea40abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8eaa4c2640 1 --2- 172.21.15.45:0/101097408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 0x7f8ea415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8ea94c0640 1 -- 172.21.15.45:0/101097408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 msgr2=0x7f8ea415ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:09.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8ea94c0640 1 --2- 172.21.15.45:0/101097408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 0x7f8ea415ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8ea94c0640 1 -- 172.21.15.45:0/101097408 >> v2:172.21.15.190:3300/0 conn(0x7f8ea4155780 msgr2=0x7f8ea40abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:09.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8ea94c0640 1 --2- 172.21.15.45:0/101097408 >> v2:172.21.15.190:3300/0 conn(0x7f8ea4155780 0x7f8ea40abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8ea94c0640 1 -- 172.21.15.45:0/101097408 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea415ba90 con 0x7f8ea40ae540 2023-11-28T21:52:09.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.556+0000 7f8ea9cc1640 1 --2- 172.21.15.45:0/101097408 >> v2:172.21.15.190:3300/0 conn(0x7f8ea4155780 0x7f8ea40abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:09.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.557+0000 7f8ea94c0640 1 --2- 172.21.15.45:0/101097408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 0x7f8ea40ce0f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f8ea406e6d0 tx=0x7f8e9402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c69a340452ebfb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:09.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.557+0000 7f8ea8cbf640 1 -- 172.21.15.45:0/101097408 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94031860 con 0x7f8ea40ae540 2023-11-28T21:52:09.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.557+0000 7f8ea8cbf640 1 -- 172.21.15.45:0/101097408 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e940319c0 con 0x7f8ea40ae540 2023-11-28T21:52:09.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.557+0000 7f8ea8cbf640 1 -- 172.21.15.45:0/101097408 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94031d50 con 0x7f8ea40ae540 2023-11-28T21:52:09.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.558+0000 7f8eaacc3640 1 -- 172.21.15.45:0/101097408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 msgr2=0x7f8ea40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.558+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/101097408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 0x7f8ea40ce0f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f8ea406e6d0 tx=0x7f8e9402b900 comp rx=0 tx=0).stop 2023-11-28T21:52:09.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.558+0000 7f8eaacc3640 1 -- 172.21.15.45:0/101097408 shutdown_connections 2023-11-28T21:52:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.558+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/101097408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 0x7f8ea415ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.558+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/101097408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 0x7f8ea40ce0f0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.558+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/101097408 >> v2:172.21.15.190:3300/0 conn(0x7f8ea4155780 0x7f8ea40abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.558+0000 7f8eaacc3640 1 -- 172.21.15.45:0/101097408 >> 172.21.15.45:0/101097408 conn(0x7f8ea406ab30 msgr2=0x7f8ea40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.558+0000 7f8eaacc3640 1 -- 172.21.15.45:0/101097408 shutdown_connections 2023-11-28T21:52:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.558+0000 7f8eaacc3640 1 -- 172.21.15.45:0/101097408 wait complete. 2023-11-28T21:52:09.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.560+0000 7f8eaacc3640 1 Processor -- start 2023-11-28T21:52:09.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.560+0000 7f8eaacc3640 1 -- start start 2023-11-28T21:52:09.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.561+0000 7f8eaacc3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 0x7f8ea406d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.561+0000 7f8eaacc3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 0x7f8ea413d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.561+0000 7f8eaacc3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea4155780 0x7f8ea413d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.561+0000 7f8eaacc3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea414c630 con 0x7f8ea40ae540 2023-11-28T21:52:09.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.561+0000 7f8eaacc3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea413dd80 con 0x7f8ea4155780 2023-11-28T21:52:09.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.561+0000 7f8eaacc3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea414d280 con 0x7f8ea40ce630 2023-11-28T21:52:09.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea9cc1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 0x7f8ea406d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea9cc1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 0x7f8ea406d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40344/0 (socket says 172.21.15.45:40344) 2023-11-28T21:52:09.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea9cc1640 1 -- 172.21.15.45:0/1025147766 learned_addr learned my addr 172.21.15.45:0/1025147766 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:09.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea94c0640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 0x7f8ea413d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8eaa4c2640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea4155780 0x7f8ea413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea9cc1640 1 -- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 msgr2=0x7f8ea413d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea9cc1640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 0x7f8ea413d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea9cc1640 1 -- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea4155780 msgr2=0x7f8ea413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea9cc1640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea4155780 0x7f8ea413d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea9cc1640 1 -- 172.21.15.45:0/1025147766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e940094d0 con 0x7f8ea40ae540 2023-11-28T21:52:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.562+0000 7f8ea9cc1640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 0x7f8ea406d030 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f8ea000c7f0 tx=0x7f8ea000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.563+0000 7f8e9a7fc640 1 -- 172.21.15.45:0/1025147766 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0007a30 con 0x7f8ea40ae540 2023-11-28T21:52:09.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.563+0000 7f8e9a7fc640 1 -- 172.21.15.45:0/1025147766 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea0007b90 con 0x7f8ea40ae540 2023-11-28T21:52:09.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.563+0000 7f8e9a7fc640 1 -- 172.21.15.45:0/1025147766 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea000ce40 con 0x7f8ea40ae540 2023-11-28T21:52:09.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.563+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea414d560 con 0x7f8ea40ae540 2023-11-28T21:52:09.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.563+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea414db20 con 0x7f8ea40ae540 2023-11-28T21:52:09.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.564+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea404e920 con 0x7f8ea40ae540 2023-11-28T21:52:09.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.565+0000 7f8e9a7fc640 1 -- 172.21.15.45:0/1025147766 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 98) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f8ea00113f0 con 0x7f8ea40ae540 2023-11-28T21:52:09.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.566+0000 7f8e9a7fc640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8e8806af20 0x7f8e8806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:09.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.566+0000 7f8e9a7fc640 1 -- 172.21.15.45:0/1025147766 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f8ea0089cd0 con 0x7f8ea40ae540 2023-11-28T21:52:09.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.567+0000 7f8ea94c0640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8e8806af20 0x7f8e8806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:09.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.568+0000 7f8ea94c0640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8e8806af20 0x7f8e8806d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e94036000 tx=0x7f8e9403c040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:09.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.570+0000 7f8e9a7fc640 1 -- 172.21.15.45:0/1025147766 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ea0054880 con 0x7f8ea40ae540 2023-11-28T21:52:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.800+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ea4073eb0 con 0x7f8ea40ae540 2023-11-28T21:52:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.801+0000 7f8e9a7fc640 1 -- 172.21.15.45:0/1025147766 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4908 (secure 0 0 0) 0x7f8ea0057ea0 con 0x7f8ea40ae540 2023-11-28T21:52:09.804 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:09.804 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7130,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/500262407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":500262407},{"type":"v1","addr":"172.21.15.190:6835","nonce":500262407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/195287870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":195287870},{"type":"v1","addr":"172.21.15.190:6837","nonce":195287870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":9657,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/440031651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":440031651},{"type":"v1","addr":"172.21.15.45:6833","nonce":440031651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:51:45.400395+0000","modified":"2023-11-28T21:51:48.467537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9660":{"gid":9660,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/99455191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":99455191},{"type":"v1","addr":"172.21.15.45:6835","nonce":99455191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-28T21:52:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 84 2023-11-28T21:52:09.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8e8806af20 msgr2=0x7f8e8806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8e8806af20 0x7f8e8806d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e94036000 tx=0x7f8e9403c040 comp rx=0 tx=0).stop 2023-11-28T21:52:09.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 msgr2=0x7f8ea406d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:09.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 0x7f8ea406d030 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f8ea000c7f0 tx=0x7f8ea000cbf0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 shutdown_connections 2023-11-28T21:52:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8e8806af20 0x7f8e8806d3e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea4155780 0x7f8ea413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ea40ce630 0x7f8ea413d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 --2- 172.21.15.45:0/1025147766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ea40ae540 0x7f8ea406d030 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 >> 172.21.15.45:0/1025147766 conn(0x7f8ea406ab30 msgr2=0x7f8ea415c870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 shutdown_connections 2023-11-28T21:52:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:09.804+0000 7f8eaacc3640 1 -- 172.21.15.45:0/1025147766 wait complete. 2023-11-28T21:52:09.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:52:10.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.052+0000 7f58769b7640 1 Processor -- start 2023-11-28T21:52:10.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.052+0000 7f58769b7640 1 -- start start 2023-11-28T21:52:10.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.054+0000 7f58769b7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f587015b850 0x7f58700b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.054+0000 7f58769b7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587015bda0 con 0x7f58700b35b0 2023-11-28T21:52:10.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.054+0000 7f58769b7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587015c080 con 0x7f58700b1fa0 2023-11-28T21:52:10.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.054+0000 7f58769b7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f587015ee40 con 0x7f587015b850 2023-11-28T21:52:10.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.055+0000 7f586ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f58700b35b0 0x7f587015ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50356/0 (socket says 172.21.15.45:50356) 2023-11-28T21:52:10.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.055+0000 7f586ffff640 1 -- 172.21.15.45:0/2493527091 learned_addr learned my addr 172.21.15.45:0/2493527091 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:10.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.055+0000 7f586f7fe640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f587015b850 0x7f58700b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.055+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797412984 0 0) 0x7f587015bda0 con 0x7f58700b35b0 2023-11-28T21:52:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.055+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5858003520 con 0x7f58700b35b0 2023-11-28T21:52:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1537670712 0 0) 0x7f5858003520 con 0x7f58700b35b0 2023-11-28T21:52:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f587015b850 msgr2=0x7f58700b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f587015b850 0x7f58700b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 >> v1:172.21.15.190:6789/0 conn(0x7f58700b1fa0 legacy=0x7f587015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587015c360 con 0x7f58700b35b0 2023-11-28T21:52:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5860001670 con 0x7f58700b35b0 2023-11-28T21:52:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 >> v1:172.21.15.45:6789/0 conn(0x7f58700b35b0 legacy=0x7f587015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5858003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58580047c0 0x7f5858004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f587015b850 0x7f58580051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587015ee40 con 0x7f58580039a0 2023-11-28T21:52:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585800ba90 con 0x7f58580047c0 2023-11-28T21:52:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585800bc50 con 0x7f587015b850 2023-11-28T21:52:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.056+0000 7f586ffff640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5858003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.057+0000 7f586f7fe640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58580047c0 0x7f5858004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.057+0000 7f5874f54640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f587015b850 0x7f58580051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.057+0000 7f586ffff640 1 -- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f587015b850 msgr2=0x7f58580051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.057+0000 7f586ffff640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f587015b850 0x7f58580051a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.057+0000 7f586ffff640 1 -- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58580047c0 msgr2=0x7f5858004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.057+0000 7f586ffff640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58580047c0 0x7f5858004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.057+0000 7f586ffff640 1 -- 172.21.15.45:0/2493527091 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5858003520 con 0x7f58580039a0 2023-11-28T21:52:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.057+0000 7f586f7fe640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58580047c0 0x7f5858004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.058+0000 7f586ffff640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5858003dc0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f586000c0e0 tx=0x7f58580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a38862081255bb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.058+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860005ad0 con 0x7f58580039a0 2023-11-28T21:52:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.058+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5860005c30 con 0x7f58580039a0 2023-11-28T21:52:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.058+0000 7f586effd640 1 -- 172.21.15.45:0/2493527091 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586002c810 con 0x7f58580039a0 2023-11-28T21:52:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.059+0000 7f58769b7640 1 -- 172.21.15.45:0/2493527091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 msgr2=0x7f5858003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.059+0000 7f58769b7640 1 --2- 172.21.15.45:0/2493527091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5858003dc0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f586000c0e0 tx=0x7f58580057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.059+0000 7f5874f54640 1 -- 172.21.15.45:0/2493527091 reap_dead start 2023-11-28T21:52:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.059+0000 7f58769b7640 1 -- 172.21.15.45:0/2493527091 shutdown_connections 2023-11-28T21:52:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.059+0000 7f58769b7640 1 -- 172.21.15.45:0/2493527091 >> 172.21.15.45:0/2493527091 conn(0x7f587006a500 msgr2=0x7f587006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.059+0000 7f58769b7640 1 -- 172.21.15.45:0/2493527091 shutdown_connections 2023-11-28T21:52:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.059+0000 7f58769b7640 1 -- 172.21.15.45:0/2493527091 wait complete. 2023-11-28T21:52:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.061+0000 7f58769b7640 1 Processor -- start 2023-11-28T21:52:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.061+0000 7f58769b7640 1 -- start start 2023-11-28T21:52:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.062+0000 7f58769b7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5870142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.062+0000 7f58769b7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58700ad030 0x7f58700ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.062+0000 7f58769b7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58700cdae0 0x7f58700cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.062+0000 7f58769b7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58700ce4a0 con 0x7f58580039a0 2023-11-28T21:52:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.062+0000 7f58769b7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58701f2cd0 con 0x7f58700ad030 2023-11-28T21:52:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.062+0000 7f58769b7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58700706a0 con 0x7f58700cdae0 2023-11-28T21:52:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.062+0000 7f586ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5870142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.062+0000 7f586ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5870142ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40358/0 (socket says 172.21.15.45:40358) 2023-11-28T21:52:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.062+0000 7f586ffff640 1 -- 172.21.15.45:0/3571306340 learned_addr learned my addr 172.21.15.45:0/3571306340 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586ffff640 1 -- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58700cdae0 msgr2=0x7f58700cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586ffff640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58700cdae0 0x7f58700cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586ffff640 1 -- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58700ad030 msgr2=0x7f58700ad4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586f7fe640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58700ad030 0x7f58700ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586ffff640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58700ad030 0x7f58700ad4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586ffff640 1 -- 172.21.15.45:0/3571306340 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5870070980 con 0x7f58580039a0 2023-11-28T21:52:10.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586ffff640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5870142ba0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f586000c0e0 tx=0x7f5860001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:10.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586cff9640 1 -- 172.21.15.45:0/3571306340 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586003c040 con 0x7f58580039a0 2023-11-28T21:52:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586cff9640 1 -- 172.21.15.45:0/3571306340 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586002dcb0 con 0x7f58580039a0 2023-11-28T21:52:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f586cff9640 1 -- 172.21.15.45:0/3571306340 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860007270 con 0x7f58580039a0 2023-11-28T21:52:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.063+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5870070c60 con 0x7f58580039a0 2023-11-28T21:52:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.064+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5870070330 con 0x7f58580039a0 2023-11-28T21:52:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.065+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58700733d0 con 0x7f58580039a0 2023-11-28T21:52:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.066+0000 7f586cff9640 1 -- 172.21.15.45:0/3571306340 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 98) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f5860032070 con 0x7f58580039a0 2023-11-28T21:52:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.067+0000 7f586cff9640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f585c06af20 0x7f585c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.067+0000 7f586cff9640 1 -- 172.21.15.45:0/3571306340 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f58600ab4e0 con 0x7f58580039a0 2023-11-28T21:52:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.067+0000 7f586f7fe640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f585c06af20 0x7f585c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.068+0000 7f586f7fe640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f585c06af20 0x7f585c06d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f587015a810 tx=0x7f586400b040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:10.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.071+0000 7f586cff9640 1 -- 172.21.15.45:0/3571306340 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58600761a0 con 0x7f58580039a0 2023-11-28T21:52:10.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.302+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58700cd3b0 con 0x7f58580039a0 2023-11-28T21:52:10.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.303+0000 7f586cff9640 1 -- 172.21.15.45:0/3571306340 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4908 (secure 0 0 0) 0x7f58600797c0 con 0x7f58580039a0 2023-11-28T21:52:10.306 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:10.306 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7130,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/500262407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":500262407},{"type":"v1","addr":"172.21.15.190:6835","nonce":500262407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/195287870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":195287870},{"type":"v1","addr":"172.21.15.190:6837","nonce":195287870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":9657,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/440031651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":440031651},{"type":"v1","addr":"172.21.15.45:6833","nonce":440031651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:51:45.400395+0000","modified":"2023-11-28T21:51:48.467537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9660":{"gid":9660,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/99455191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":99455191},{"type":"v1","addr":"172.21.15.45:6835","nonce":99455191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-11-28T21:52:10.306 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 84 2023-11-28T21:52:10.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f585c06af20 msgr2=0x7f585c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f585c06af20 0x7f585c06d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f587015a810 tx=0x7f586400b040 comp rx=0 tx=0).stop 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 msgr2=0x7f5870142ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5870142ba0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f586000c0e0 tx=0x7f5860001930 comp rx=0 tx=0).stop 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 shutdown_connections 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f585c06af20 0x7f585c06d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58700cdae0 0x7f58700cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58700ad030 0x7f58700ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 --2- 172.21.15.45:0/3571306340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58580039a0 0x7f5870142ba0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 >> 172.21.15.45:0/3571306340 conn(0x7f587006a500 msgr2=0x7f58700b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 shutdown_connections 2023-11-28T21:52:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.306+0000 7f58769b7640 1 -- 172.21.15.45:0/3571306340 wait complete. 2023-11-28T21:52:10.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:52:10.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.553+0000 7feff701a640 1 Processor -- start 2023-11-28T21:52:10.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.554+0000 7feff701a640 1 -- start start 2023-11-28T21:52:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.555+0000 7feff701a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feff00b3870 0x7feff015db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.555+0000 7feff701a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feff00721e0 0x7feff015e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.555+0000 7feff701a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff015e680 con 0x7feff006d170 2023-11-28T21:52:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.555+0000 7feff701a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff015a460 con 0x7feff00b3870 2023-11-28T21:52:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.555+0000 7feff701a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff015e960 con 0x7feff00721e0 2023-11-28T21:52:10.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.555+0000 7feff6018640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feff00b3870 0x7feff015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.556+0000 7feff6018640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feff00b3870 0x7feff015db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40364/0 (socket says 172.21.15.45:40364) 2023-11-28T21:52:10.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.556+0000 7feff6018640 1 -- 172.21.15.45:0/1080739882 learned_addr learned my addr 172.21.15.45:0/1080739882 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:10.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.556+0000 7feff6018640 1 -- 172.21.15.45:0/1080739882 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feff00721e0 msgr2=0x7feff015e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:10.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.556+0000 7feff6018640 1 --2- 172.21.15.45:0/1080739882 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feff00721e0 0x7feff015e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.556+0000 7feff6018640 1 -- 172.21.15.45:0/1080739882 >> v1:172.21.15.190:6789/0 conn(0x7feff006d170 legacy=0x7feff00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:10.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.556+0000 7feff6018640 1 -- 172.21.15.45:0/1080739882 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff0071700 con 0x7feff00b3870 2023-11-28T21:52:10.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.557+0000 7feff6018640 1 --2- 172.21.15.45:0/1080739882 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 0x7feff015db10 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fefec009870 tx=0x7fefec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c5fc5b235fdc2b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:10.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.557+0000 7feff5016640 1 -- 172.21.15.45:0/1080739882 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec031640 con 0x7feff00b3870 2023-11-28T21:52:10.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.557+0000 7feff5016640 1 -- 172.21.15.45:0/1080739882 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec0317a0 con 0x7feff00b3870 2023-11-28T21:52:10.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.557+0000 7feff5016640 1 -- 172.21.15.45:0/1080739882 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec031b30 con 0x7feff00b3870 2023-11-28T21:52:10.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.558+0000 7feff701a640 1 -- 172.21.15.45:0/1080739882 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 msgr2=0x7feff015db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.558+0000 7feff701a640 1 --2- 172.21.15.45:0/1080739882 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 0x7feff015db10 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fefec009870 tx=0x7fefec02b8f0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.558+0000 7feff701a640 1 -- 172.21.15.45:0/1080739882 shutdown_connections 2023-11-28T21:52:10.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.558+0000 7feff701a640 1 --2- 172.21.15.45:0/1080739882 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feff00721e0 0x7feff015e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.558+0000 7feff701a640 1 --2- 172.21.15.45:0/1080739882 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 0x7feff015db10 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.558+0000 7feff701a640 1 -- 172.21.15.45:0/1080739882 >> 172.21.15.45:0/1080739882 conn(0x7feff006ab30 msgr2=0x7feff00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:10.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.558+0000 7feff701a640 1 -- 172.21.15.45:0/1080739882 shutdown_connections 2023-11-28T21:52:10.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.558+0000 7feff701a640 1 -- 172.21.15.45:0/1080739882 wait complete. 2023-11-28T21:52:10.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.560+0000 7feff701a640 1 Processor -- start 2023-11-28T21:52:10.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.560+0000 7feff701a640 1 -- start start 2023-11-28T21:52:10.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.561+0000 7feff701a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feff006d170 0x7feff0156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.561+0000 7feff701a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff00721e0 0x7feff009bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.561+0000 7feff701a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 0x7feff00c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.561+0000 7feff701a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff0099f40 con 0x7feff00b3870 2023-11-28T21:52:10.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.561+0000 7feff701a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff0157410 con 0x7feff00721e0 2023-11-28T21:52:10.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.561+0000 7feff701a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff01576f0 con 0x7feff006d170 2023-11-28T21:52:10.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.561+0000 7feff6819640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 0x7feff00c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff6819640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 0x7feff00c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40376/0 (socket says 172.21.15.45:40376) 2023-11-28T21:52:10.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff6819640 1 -- 172.21.15.45:0/984395631 learned_addr learned my addr 172.21.15.45:0/984395631 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:10.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff6819640 1 -- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feff006d170 msgr2=0x7feff0156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff6018640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feff006d170 0x7feff0156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff6819640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feff006d170 0x7feff0156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff5817640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff00721e0 0x7feff009bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff6819640 1 -- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff00721e0 msgr2=0x7feff009bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff6819640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff00721e0 0x7feff009bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff6819640 1 -- 172.21.15.45:0/984395631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec0094d0 con 0x7feff00b3870 2023-11-28T21:52:10.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.562+0000 7feff6819640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 0x7feff00c3d90 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fefe800c390 tx=0x7fefe800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:10.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.563+0000 7fefe6ffd640 1 -- 172.21.15.45:0/984395631 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe80035f0 con 0x7feff00b3870 2023-11-28T21:52:10.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.563+0000 7fefe6ffd640 1 -- 172.21.15.45:0/984395631 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe8003750 con 0x7feff00b3870 2023-11-28T21:52:10.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.563+0000 7fefe6ffd640 1 -- 172.21.15.45:0/984395631 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe8003b00 con 0x7feff00b3870 2023-11-28T21:52:10.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.563+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff01579d0 con 0x7feff00b3870 2023-11-28T21:52:10.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.563+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff009c430 con 0x7feff00b3870 2023-11-28T21:52:10.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.564+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff009a110 con 0x7feff00b3870 2023-11-28T21:52:10.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.565+0000 7fefe6ffd640 1 -- 172.21.15.45:0/984395631 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 98) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fefe8003c60 con 0x7feff00b3870 2023-11-28T21:52:10.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.566+0000 7fefe6ffd640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fefd406aed0 0x7fefd406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:10.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.566+0000 7fefe6ffd640 1 -- 172.21.15.45:0/984395631 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fefe8088cd0 con 0x7feff00b3870 2023-11-28T21:52:10.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.567+0000 7feff6018640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fefd406aed0 0x7fefd406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:10.571 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.568+0000 7feff6018640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fefd406aed0 0x7fefd406d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fefec036000 tx=0x7fefec03c040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:10.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.570+0000 7fefe6ffd640 1 -- 172.21.15.45:0/984395631 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fefe8053890 con 0x7feff00b3870 2023-11-28T21:52:10.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.764+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feff00ac420 con 0x7feff00b3870 2023-11-28T21:52:10.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.767+0000 7fefe6ffd640 1 -- 172.21.15.45:0/984395631 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+17771 (secure 0 0 0) 0x7fefe8056eb0 con 0x7feff00b3870 2023-11-28T21:52:10.770 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:10.770 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":187,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:52:07.287056+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:51:43.335999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-11-28T21:51:44.431458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3757577966":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/788198125":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/3142102971":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/1232828858":"2023-11-29T21:52:07.286885+0000","172.21.15.190:6833/1285100946":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/158019673":"2023-11-29T21:52:07.286885+0000","172.21.15.190:6832/1285100946":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/2404812260":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/102436699":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/2400910006":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6833/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2240536749":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6832/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2615023161":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/4027676180":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/508215298":"2023-11-29T21:52:00.068699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:52:10.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.769+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fefd406aed0 msgr2=0x7fefd406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.769+0000 7feff701a640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fefd406aed0 0x7fefd406d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fefec036000 tx=0x7fefec03c040 comp rx=0 tx=0).stop 2023-11-28T21:52:10.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.769+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 msgr2=0x7feff00c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:10.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.769+0000 7feff701a640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 0x7feff00c3d90 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fefe800c390 tx=0x7fefe800c850 comp rx=0 tx=0).stop 2023-11-28T21:52:10.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.770+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 shutdown_connections 2023-11-28T21:52:10.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.770+0000 7feff701a640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fefd406aed0 0x7fefd406d390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.770+0000 7feff701a640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feff00b3870 0x7feff00c3d90 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.770+0000 7feff701a640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff00721e0 0x7feff009bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.770+0000 7feff701a640 1 --2- 172.21.15.45:0/984395631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feff006d170 0x7feff0156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:10.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.770+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 >> 172.21.15.45:0/984395631 conn(0x7feff006ab30 msgr2=0x7feff00b0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:10.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.770+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 shutdown_connections 2023-11-28T21:52:10.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:10.770+0000 7feff701a640 1 -- 172.21.15.45:0/984395631 wait complete. 2023-11-28T21:52:10.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T21:52:11.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.017+0000 7f17b1b54640 1 Processor -- start 2023-11-28T21:52:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.018+0000 7f17b1b54640 1 -- start start 2023-11-28T21:52:11.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.020+0000 7f17b1b54640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17ac15d220 0x7f17ac0b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.020+0000 7f17b1b54640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f17ac0cf610 0x7f17ac0704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.020+0000 7f17b1b54640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac15d600 con 0x7f17ac06c1d0 2023-11-28T21:52:11.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.020+0000 7f17b1b54640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ac06c6b0 con 0x7f17ac15d220 2023-11-28T21:52:11.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.020+0000 7f17b1b54640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ac06c990 con 0x7f17ac0cf610 2023-11-28T21:52:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.020+0000 7f17b0b52640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17ac15d220 0x7f17ac0b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.021+0000 7f17b0b52640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f17ac15d220 0x7f17ac0b3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40392/0 (socket says 172.21.15.45:40392) 2023-11-28T21:52:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.021+0000 7f17b0b52640 1 -- 172.21.15.45:0/415956597 learned_addr learned my addr 172.21.15.45:0/415956597 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.021+0000 7f17b1353640 1 --2- 172.21.15.45:0/415956597 >> v2:172.21.15.190:3300/0 conn(0x7f17ac0cf610 0x7f17ac0704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.021+0000 7f17b0b52640 1 -- 172.21.15.45:0/415956597 >> v2:172.21.15.190:3300/0 conn(0x7f17ac0cf610 msgr2=0x7f17ac0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.021+0000 7f17b0b52640 1 --2- 172.21.15.45:0/415956597 >> v2:172.21.15.190:3300/0 conn(0x7f17ac0cf610 0x7f17ac0704e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.021+0000 7f17b0b52640 1 -- 172.21.15.45:0/415956597 >> v1:172.21.15.190:6789/0 conn(0x7f17ac06c1d0 legacy=0x7f17ac0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.021+0000 7f17b0b52640 1 -- 172.21.15.45:0/415956597 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac06cc70 con 0x7f17ac15d220 2023-11-28T21:52:11.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.023+0000 7f17b0b52640 1 --2- 172.21.15.45:0/415956597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac15d220 0x7f17ac0b3cf0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f17ac06df00 tx=0x7f17a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c506f1210168f121 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.024+0000 7f17ab7fe640 1 -- 172.21.15.45:0/415956597 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0031870 con 0x7f17ac15d220 2023-11-28T21:52:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.024+0000 7f17ab7fe640 1 -- 172.21.15.45:0/415956597 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a00319d0 con 0x7f17ac15d220 2023-11-28T21:52:11.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.024+0000 7f17ab7fe640 1 -- 172.21.15.45:0/415956597 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0031d60 con 0x7f17ac15d220 2023-11-28T21:52:11.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.024+0000 7f17b1b54640 1 -- 172.21.15.45:0/415956597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac15d220 msgr2=0x7f17ac0b3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.024+0000 7f17b1b54640 1 --2- 172.21.15.45:0/415956597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac15d220 0x7f17ac0b3cf0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f17ac06df00 tx=0x7f17a002b910 comp rx=0 tx=0).stop 2023-11-28T21:52:11.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.025+0000 7f17b1b54640 1 -- 172.21.15.45:0/415956597 shutdown_connections 2023-11-28T21:52:11.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.025+0000 7f17b1b54640 1 --2- 172.21.15.45:0/415956597 >> v2:172.21.15.190:3300/0 conn(0x7f17ac0cf610 0x7f17ac0704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.025+0000 7f17b1b54640 1 --2- 172.21.15.45:0/415956597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac15d220 0x7f17ac0b3cf0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.025+0000 7f17b1b54640 1 -- 172.21.15.45:0/415956597 >> 172.21.15.45:0/415956597 conn(0x7f17ac06a710 msgr2=0x7f17ac06ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:11.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.025+0000 7f17b1b54640 1 -- 172.21.15.45:0/415956597 shutdown_connections 2023-11-28T21:52:11.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.025+0000 7f17b1b54640 1 -- 172.21.15.45:0/415956597 wait complete. 2023-11-28T21:52:11.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.026+0000 7f17b1b54640 1 Processor -- start 2023-11-28T21:52:11.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.027+0000 7f17b1b54640 1 -- start start 2023-11-28T21:52:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.027+0000 7f17b1b54640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac06c1d0 0x7f17ac154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b1b54640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17ac0cf610 0x7f17ac0ad660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b1b54640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17ac15d220 0x7f17ac1462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b1b54640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac15b660 con 0x7f17ac06c1d0 2023-11-28T21:52:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b1b54640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1549c0 con 0x7f17ac15d220 2023-11-28T21:52:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b1b54640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac155390 con 0x7f17ac0cf610 2023-11-28T21:52:11.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b0b52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac06c1d0 0x7f17ac154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b0b52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac06c1d0 0x7f17ac154e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40406/0 (socket says 172.21.15.45:40406) 2023-11-28T21:52:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b0b52640 1 -- 172.21.15.45:0/4260873736 learned_addr learned my addr 172.21.15.45:0/4260873736 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17abfff640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17ac0cf610 0x7f17ac0ad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b0b52640 1 -- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17ac0cf610 msgr2=0x7f17ac0ad660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b0b52640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17ac0cf610 0x7f17ac0ad660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b0b52640 1 -- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17ac15d220 msgr2=0x7f17ac1462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b0b52640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17ac15d220 0x7f17ac1462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.028+0000 7f17b0b52640 1 -- 172.21.15.45:0/4260873736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a00094d0 con 0x7f17ac06c1d0 2023-11-28T21:52:11.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.029+0000 7f17b0b52640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac06c1d0 0x7f17ac154e50 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f17a002b950 tx=0x7f17a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.029+0000 7f17a97fa640 1 -- 172.21.15.45:0/4260873736 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a003abe0 con 0x7f17ac06c1d0 2023-11-28T21:52:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.029+0000 7f17a97fa640 1 -- 172.21.15.45:0/4260873736 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a003ad60 con 0x7f17ac06c1d0 2023-11-28T21:52:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.029+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ac15b380 con 0x7f17ac06c1d0 2023-11-28T21:52:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.029+0000 7f17a97fa640 1 -- 172.21.15.45:0/4260873736 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0044630 con 0x7f17ac06c1d0 2023-11-28T21:52:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.029+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ac154650 con 0x7f17ac06c1d0 2023-11-28T21:52:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.030+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ac1541d0 con 0x7f17ac06c1d0 2023-11-28T21:52:11.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.031+0000 7f17a97fa640 1 -- 172.21.15.45:0/4260873736 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 98) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f17a0044790 con 0x7f17ac06c1d0 2023-11-28T21:52:11.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.032+0000 7f17a97fa640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f179006aed0 0x7f179006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.032+0000 7f17a97fa640 1 -- 172.21.15.45:0/4260873736 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f17a002d040 con 0x7f17ac06c1d0 2023-11-28T21:52:11.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.033+0000 7f17abfff640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f179006aed0 0x7f179006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.034+0000 7f17abfff640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f179006aed0 0x7f179006d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f17ac154180 tx=0x7f1798009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:11.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.037+0000 7f17a97fa640 1 -- 172.21.15.45:0/4260873736 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17ac1541d0 con 0x7f17ac06c1d0 2023-11-28T21:52:11.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.269+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f17ac1541d0 con 0x7f17ac06c1d0 2023-11-28T21:52:11.273 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T21:52:11.271+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T21:52:11.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.274+0000 7f17a97fa640 1 -- 172.21.15.45:0/4260873736 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v85) v1 ==== 161+0+0 (secure 0 0 0) 0x7f17ac1541d0 con 0x7f17ac06c1d0 2023-11-28T21:52:11.277 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T21:52:11.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f179006aed0 msgr2=0x7f179006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f179006aed0 0x7f179006d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f17ac154180 tx=0x7f1798009040 comp rx=0 tx=0).stop 2023-11-28T21:52:11.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac06c1d0 msgr2=0x7f17ac154e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac06c1d0 0x7f17ac154e50 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f17a002b950 tx=0x7f17a00046c0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 shutdown_connections 2023-11-28T21:52:11.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f179006aed0 0x7f179006d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17ac15d220 0x7f17ac1462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17ac0cf610 0x7f17ac0ad660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 --2- 172.21.15.45:0/4260873736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17ac06c1d0 0x7f17ac154e50 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 >> 172.21.15.45:0/4260873736 conn(0x7f17ac06a710 msgr2=0x7f17ac06aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:11.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.277+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 shutdown_connections 2023-11-28T21:52:11.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.278+0000 7f17b1b54640 1 -- 172.21.15.45:0/4260873736 wait complete. 2023-11-28T21:52:11.299 INFO:tasks.ceph.mds.a.smithi045.stderr: -3748> 2023-11-28T21:51:50.343+0000 7f3792864640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T21:52:11.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T21:52:11.368 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T21:52:11.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.505+0000 7f68c759e640 1 Processor -- start 2023-11-28T21:52:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.505+0000 7f68c759e640 1 -- start start 2023-11-28T21:52:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.506+0000 7f68c759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c80abd70 0x7f68c80ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.506+0000 7f68c759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c80ac270 con 0x7f68c8071fd0 2023-11-28T21:52:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.506+0000 7f68c759e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c815b830 con 0x7f68c815b080 2023-11-28T21:52:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.507+0000 7f68c759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c815bb10 con 0x7f68c80abd70 2023-11-28T21:52:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.507+0000 7f68bffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f68c8071fd0 0x7f68c80702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50368/0 (socket says 172.21.15.45:50368) 2023-11-28T21:52:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.507+0000 7f68bffff640 1 -- 172.21.15.45:0/2932392868 learned_addr learned my addr 172.21.15.45:0/2932392868 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.507+0000 7f68c6d9d640 1 --2- 172.21.15.45:0/2932392868 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c80abd70 0x7f68c80ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.507+0000 7f68c5d9b640 1 -- 172.21.15.45:0/2932392868 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670717196 0 0) 0x7f68c80ac270 con 0x7f68c8071fd0 2023-11-28T21:52:11.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.507+0000 7f68c5d9b640 1 -- 172.21.15.45:0/2932392868 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b4003520 con 0x7f68c8071fd0 2023-11-28T21:52:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.508+0000 7f68c6d9d640 1 -- 172.21.15.45:0/2932392868 >> v1:172.21.15.190:6789/0 conn(0x7f68c815b080 legacy=0x7f68c80718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.508+0000 7f68c5d9b640 1 -- 172.21.15.45:0/2932392868 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727650966 0 0) 0x7f68b4003520 con 0x7f68c8071fd0 2023-11-28T21:52:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.508+0000 7f68c6d9d640 1 -- 172.21.15.45:0/2932392868 >> v1:172.21.15.45:6789/0 conn(0x7f68c8071fd0 legacy=0x7f68c80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.508+0000 7f68c6d9d640 1 -- 172.21.15.45:0/2932392868 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c815bdf0 con 0x7f68c80abd70 2023-11-28T21:52:11.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.509+0000 7f68c6d9d640 1 --2- 172.21.15.45:0/2932392868 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c80abd70 0x7f68c80ae370 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009dc0 tx=0x7f68c002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=278255589d59a0e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:11.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.510+0000 7f68c5d9b640 1 -- 172.21.15.45:0/2932392868 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0001690 con 0x7f68c80abd70 2023-11-28T21:52:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.510+0000 7f68c5d9b640 1 -- 172.21.15.45:0/2932392868 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c002c910 con 0x7f68c80abd70 2023-11-28T21:52:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.510+0000 7f68c5d9b640 1 -- 172.21.15.45:0/2932392868 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c002cca0 con 0x7f68c80abd70 2023-11-28T21:52:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.511+0000 7f68c759e640 1 -- 172.21.15.45:0/2932392868 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c80abd70 msgr2=0x7f68c80ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.511+0000 7f68c759e640 1 --2- 172.21.15.45:0/2932392868 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c80abd70 0x7f68c80ae370 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009dc0 tx=0x7f68c002bde0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.511+0000 7f68c759e640 1 -- 172.21.15.45:0/2932392868 shutdown_connections 2023-11-28T21:52:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.511+0000 7f68c759e640 1 --2- 172.21.15.45:0/2932392868 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c80abd70 0x7f68c80ae370 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.511+0000 7f68c759e640 1 -- 172.21.15.45:0/2932392868 >> 172.21.15.45:0/2932392868 conn(0x7f68c80cecd0 msgr2=0x7f68c80cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.511+0000 7f68c759e640 1 -- 172.21.15.45:0/2932392868 shutdown_connections 2023-11-28T21:52:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.511+0000 7f68c759e640 1 -- 172.21.15.45:0/2932392868 wait complete. 2023-11-28T21:52:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.513+0000 7f68c759e640 1 Processor -- start 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.513+0000 7f68c759e640 1 -- start start 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68c759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8071fd0 0x7f68c8070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68c759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c80abd70 0x7f68c81513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68c759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c815b080 0x7f68c8151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68c759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c80b1fa0 con 0x7f68c80abd70 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68c759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c8151e60 con 0x7f68c8071fd0 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68c759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c8157ae0 con 0x7f68c815b080 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68bffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c80abd70 0x7f68c81513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68bffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c80abd70 0x7f68c81513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40438/0 (socket says 172.21.15.45:40438) 2023-11-28T21:52:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68bffff640 1 -- 172.21.15.45:0/3351667015 learned_addr learned my addr 172.21.15.45:0/3351667015 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68c659c640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8071fd0 0x7f68c8070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.514+0000 7f68c6d9d640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c815b080 0x7f68c8151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68bffff640 1 -- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c815b080 msgr2=0x7f68c8151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68bffff640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c815b080 0x7f68c8151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68bffff640 1 -- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8071fd0 msgr2=0x7f68c8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68bffff640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8071fd0 0x7f68c8070430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68bffff640 1 -- 172.21.15.45:0/3351667015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c0009a70 con 0x7f68c80abd70 2023-11-28T21:52:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68c659c640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8071fd0 0x7f68c8070430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68bffff640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c80abd70 0x7f68c81513e0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f68b000c780 tx=0x7f68b000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68beffd640 1 -- 172.21.15.45:0/3351667015 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b00072d0 con 0x7f68c80abd70 2023-11-28T21:52:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68beffd640 1 -- 172.21.15.45:0/3351667015 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b0007430 con 0x7f68c80abd70 2023-11-28T21:52:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68beffd640 1 -- 172.21.15.45:0/3351667015 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b00047e0 con 0x7f68c80abd70 2023-11-28T21:52:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c8157240 con 0x7f68c80abd70 2023-11-28T21:52:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.515+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c8157800 con 0x7f68c80abd70 2023-11-28T21:52:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.517+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c80b20e0 con 0x7f68c80abd70 2023-11-28T21:52:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.518+0000 7f68beffd640 1 -- 172.21.15.45:0/3351667015 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 98) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f68b0004940 con 0x7f68c80abd70 2023-11-28T21:52:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.519+0000 7f68beffd640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f68a806aed0 0x7f68a806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.519+0000 7f68beffd640 1 -- 172.21.15.45:0/3351667015 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f68b0088d70 con 0x7f68c80abd70 2023-11-28T21:52:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.519+0000 7f68c659c640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f68a806aed0 0x7f68a806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.521+0000 7f68c659c640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f68a806aed0 0x7f68a806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f68b8000f60 tx=0x7f68b8009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:11.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.523+0000 7f68beffd640 1 -- 172.21.15.45:0/3351667015 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68c80b20e0 con 0x7f68c80abd70 2023-11-28T21:52:11.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:11.762+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f68c80b20e0 con 0x7f68c80abd70 2023-11-28T21:52:12.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.290+0000 7f68beffd640 1 -- 172.21.15.45:0/3351667015 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f68b0053da0 con 0x7f68c80abd70 2023-11-28T21:52:12.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.331+0000 7f68beffd640 1 -- 172.21.15.45:0/3351667015 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v86) v1 ==== 104+0+0 (secure 0 0 0) 0x7f68b0073b90 con 0x7f68c80abd70 2023-11-28T21:52:12.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f68a806aed0 msgr2=0x7f68a806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:12.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f68a806aed0 0x7f68a806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f68b8000f60 tx=0x7f68b8009040 comp rx=0 tx=0).stop 2023-11-28T21:52:12.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c80abd70 msgr2=0x7f68c81513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:12.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c80abd70 0x7f68c81513e0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f68b000c780 tx=0x7f68b000cc40 comp rx=0 tx=0).stop 2023-11-28T21:52:12.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 shutdown_connections 2023-11-28T21:52:12.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f68a806aed0 0x7f68a806d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68c815b080 0x7f68c8151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c80abd70 0x7f68c81513e0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 --2- 172.21.15.45:0/3351667015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68c8071fd0 0x7f68c8070430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.334+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 >> 172.21.15.45:0/3351667015 conn(0x7f68c80cecd0 msgr2=0x7f68c815c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:12.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.335+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 shutdown_connections 2023-11-28T21:52:12.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.335+0000 7f68c759e640 1 -- 172.21.15.45:0/3351667015 wait complete. 2023-11-28T21:52:12.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T21:52:12.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.575+0000 7fc7b8e71640 1 Processor -- start 2023-11-28T21:52:12.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.575+0000 7fc7b8e71640 1 -- start start 2023-11-28T21:52:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.577+0000 7fc7b8e71640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b40af2c0 0x7fc7b406b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.577+0000 7fc7b8e71640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc7b4154420 0x7fc7b406ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.577+0000 7fc7b8e71640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b40ad920 con 0x7fc7b4154040 2023-11-28T21:52:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.577+0000 7fc7b8e71640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b4160e30 con 0x7fc7b4154420 2023-11-28T21:52:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.577+0000 7fc7b8e71640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4161110 con 0x7fc7b40af2c0 2023-11-28T21:52:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.577+0000 7fc7b3fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc7b4154420 0x7fc7b406ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:12.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.578+0000 7fc7b2ffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b40af2c0 0x7fc7b406b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:12.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.578+0000 7fc7b3fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc7b4154420 0x7fc7b406ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:36478/0 (socket says 172.21.15.45:36478) 2023-11-28T21:52:12.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.578+0000 7fc7b3fff640 1 -- 172.21.15.45:0/737420902 learned_addr learned my addr 172.21.15.45:0/737420902 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:12.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.578+0000 7fc7b3fff640 1 -- 172.21.15.45:0/737420902 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b40af2c0 msgr2=0x7fc7b406b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:12.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.578+0000 7fc7b3fff640 1 --2- 172.21.15.45:0/737420902 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b40af2c0 0x7fc7b406b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.578+0000 7fc7b3fff640 1 -- 172.21.15.45:0/737420902 >> v1:172.21.15.190:6789/0 conn(0x7fc7b4154040 legacy=0x7fc7b40acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.578+0000 7fc7b3fff640 1 -- 172.21.15.45:0/737420902 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b41613f0 con 0x7fc7b4154420 2023-11-28T21:52:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.580+0000 7fc7b3fff640 1 --2- 172.21.15.45:0/737420902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 0x7fc7b406ba50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009870 tx=0x7fc7ac02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2983139e13261af server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.580+0000 7fc7b27fc640 1 -- 172.21.15.45:0/737420902 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac031640 con 0x7fc7b4154420 2023-11-28T21:52:12.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.580+0000 7fc7b27fc640 1 -- 172.21.15.45:0/737420902 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac0317a0 con 0x7fc7b4154420 2023-11-28T21:52:12.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.580+0000 7fc7b27fc640 1 -- 172.21.15.45:0/737420902 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac031b30 con 0x7fc7b4154420 2023-11-28T21:52:12.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.581+0000 7fc7b8e71640 1 -- 172.21.15.45:0/737420902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 msgr2=0x7fc7b406ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:12.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.581+0000 7fc7b8e71640 1 --2- 172.21.15.45:0/737420902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 0x7fc7b406ba50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009870 tx=0x7fc7ac02b8f0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.581+0000 7fc7b8e71640 1 -- 172.21.15.45:0/737420902 shutdown_connections 2023-11-28T21:52:12.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.581+0000 7fc7b8e71640 1 --2- 172.21.15.45:0/737420902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 0x7fc7b406ba50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.581+0000 7fc7b8e71640 1 --2- 172.21.15.45:0/737420902 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b40af2c0 0x7fc7b406b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.581+0000 7fc7b8e71640 1 -- 172.21.15.45:0/737420902 >> 172.21.15.45:0/737420902 conn(0x7fc7b4071220 msgr2=0x7fc7b4071630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:12.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.581+0000 7fc7b8e71640 1 -- 172.21.15.45:0/737420902 shutdown_connections 2023-11-28T21:52:12.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.581+0000 7fc7b8e71640 1 -- 172.21.15.45:0/737420902 wait complete. 2023-11-28T21:52:12.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.584+0000 7fc7b8e71640 1 Processor -- start 2023-11-28T21:52:12.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.584+0000 7fc7b8e71640 1 -- start start 2023-11-28T21:52:12.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.585+0000 7fc7b8e71640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7b40af2c0 0x7fc7b413f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:12.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.585+0000 7fc7b8e71640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b4154040 0x7fc7b41e7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:12.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.585+0000 7fc7b8e71640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 0x7fc7b4140000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:12.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.585+0000 7fc7b8e71640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4142ef0 con 0x7fc7b40af2c0 2023-11-28T21:52:12.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.585+0000 7fc7b8e71640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b413fd20 con 0x7fc7b4154420 2023-11-28T21:52:12.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.585+0000 7fc7b8e71640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b41e6ce0 con 0x7fc7b4154040 2023-11-28T21:52:12.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.586+0000 7fc7b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7b40af2c0 0x7fc7b413f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:12.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.586+0000 7fc7b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7b40af2c0 0x7fc7b413f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40440/0 (socket says 172.21.15.45:40440) 2023-11-28T21:52:12.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.586+0000 7fc7b37fe640 1 -- 172.21.15.45:0/3635708723 learned_addr learned my addr 172.21.15.45:0/3635708723 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:12.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.586+0000 7fc7b3fff640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 0x7fc7b4140000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:12.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.586+0000 7fc7b2ffd640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b4154040 0x7fc7b41e7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:12.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7b37fe640 1 -- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b4154040 msgr2=0x7fc7b41e7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:12.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7b37fe640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b4154040 0x7fc7b41e7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7b37fe640 1 -- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 msgr2=0x7fc7b4140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:12.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7b37fe640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 0x7fc7b4140000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:12.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7b37fe640 1 -- 172.21.15.45:0/3635708723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0094d0 con 0x7fc7b40af2c0 2023-11-28T21:52:12.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7b3fff640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 0x7fc7b4140000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:12.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7b2ffd640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b4154040 0x7fc7b41e7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7b37fe640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7b40af2c0 0x7fc7b413f7e0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400c730 tx=0x7fc7a400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7abfff640 1 -- 172.21.15.45:0/3635708723 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4006d50 con 0x7fc7b40af2c0 2023-11-28T21:52:12.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7abfff640 1 -- 172.21.15.45:0/3635708723 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a400ce40 con 0x7fc7b40af2c0 2023-11-28T21:52:12.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b41e6fc0 con 0x7fc7b40af2c0 2023-11-28T21:52:12.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.587+0000 7fc7abfff640 1 -- 172.21.15.45:0/3635708723 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4003bb0 con 0x7fc7b40af2c0 2023-11-28T21:52:12.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.588+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b41e7580 con 0x7fc7b40af2c0 2023-11-28T21:52:12.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.589+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b409e100 con 0x7fc7b40af2c0 2023-11-28T21:52:12.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.590+0000 7fc7abfff640 1 -- 172.21.15.45:0/3635708723 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc7a4003d10 con 0x7fc7b40af2c0 2023-11-28T21:52:12.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.591+0000 7fc7abfff640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fc79406af60 0x7fc79406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:12.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.591+0000 7fc7abfff640 1 -- 172.21.15.45:0/3635708723 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fc7a4089710 con 0x7fc7b40af2c0 2023-11-28T21:52:12.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.592+0000 7fc7b2ffd640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fc79406af60 0x7fc79406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:12.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.593+0000 7fc7b2ffd640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fc79406af60 0x7fc79406d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc7b415ce20 tx=0x7fc79c008040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:12.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.596+0000 7fc7abfff640 1 -- 172.21.15.45:0/3635708723 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7a4057d70 con 0x7fc7b40af2c0 2023-11-28T21:52:12.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:12.793+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc7b404e920 con 0x7fc7b40af2c0 2023-11-28T21:52:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.344+0000 7fc7abfff640 1 -- 172.21.15.45:0/3635708723 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v189) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc7a4057900 con 0x7fc7b40af2c0 2023-11-28T21:52:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T21:52:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.349+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fc79406af60 msgr2=0x7fc79406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.349+0000 7fc7b8e71640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fc79406af60 0x7fc79406d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc7b415ce20 tx=0x7fc79c008040 comp rx=0 tx=0).stop 2023-11-28T21:52:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.349+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7b40af2c0 msgr2=0x7fc7b413f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:13.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.349+0000 7fc7b8e71640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7b40af2c0 0x7fc7b413f7e0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400c730 tx=0x7fc7a400cbf0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.351+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 shutdown_connections 2023-11-28T21:52:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.351+0000 7fc7b8e71640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fc79406af60 0x7fc79406d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.351+0000 7fc7b8e71640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b4154420 0x7fc7b4140000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.351+0000 7fc7b8e71640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7b4154040 0x7fc7b41e7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.351+0000 7fc7b8e71640 1 --2- 172.21.15.45:0/3635708723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7b40af2c0 0x7fc7b413f7e0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.351+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 >> 172.21.15.45:0/3635708723 conn(0x7fc7b4071220 msgr2=0x7fc7b4153220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.352+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 shutdown_connections 2023-11-28T21:52:13.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.352+0000 7fc7b8e71640 1 -- 172.21.15.45:0/3635708723 wait complete. 2023-11-28T21:52:13.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T21:52:13.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.597+0000 7f7490f9e640 1 Processor -- start 2023-11-28T21:52:13.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.597+0000 7f7490f9e640 1 -- start start 2023-11-28T21:52:13.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.598+0000 7f7490f9e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f748c15b190 0x7f748c06de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:13.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.598+0000 7f7490f9e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c06e3a0 con 0x7f748c06c090 2023-11-28T21:52:13.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.598+0000 7f7490f9e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c0c9d20 con 0x7f748c070030 2023-11-28T21:52:13.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.598+0000 7f7490f9e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748c15b570 con 0x7f748c15b190 2023-11-28T21:52:13.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.598+0000 7f748bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f748c06c090 0x7f748c0ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50380/0 (socket says 172.21.15.45:50380) 2023-11-28T21:52:13.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.598+0000 7f748bfff640 1 -- 172.21.15.45:0/1712513523 learned_addr learned my addr 172.21.15.45:0/1712513523 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:13.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.599+0000 7f748b7fe640 1 --2- 172.21.15.45:0/1712513523 >> v2:172.21.15.190:3300/0 conn(0x7f748c15b190 0x7f748c06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:13.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.599+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727283265 0 0) 0x7f7484001490 con 0x7f748c06c090 2023-11-28T21:52:13.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.599+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7478003520 con 0x7f748c06c090 2023-11-28T21:52:13.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.599+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1536950390 0 0) 0x7f7478003520 con 0x7f748c06c090 2023-11-28T21:52:13.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.599+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 >> v2:172.21.15.190:3300/0 conn(0x7f748c15b190 msgr2=0x7f748c06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:13.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.599+0000 7f748affd640 1 --2- 172.21.15.45:0/1712513523 >> v2:172.21.15.190:3300/0 conn(0x7f748c15b190 0x7f748c06de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 >> v1:172.21.15.190:6789/0 conn(0x7f748c070030 legacy=0x7f748c0b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:13.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c071c50 con 0x7f748c06c090 2023-11-28T21:52:13.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f7484001b80 con 0x7f748c06c090 2023-11-28T21:52:13.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 >> v1:172.21.15.45:6789/0 conn(0x7f748c06c090 legacy=0x7f748c0ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:13.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74780039c0 0x7f7478003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:13.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74780047c0 0x7f7478004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:13.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7478005180 0x7f747800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:13.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c15b570 con 0x7f74780039c0 2023-11-28T21:52:13.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747800c080 con 0x7f7478005180 2023-11-28T21:52:13.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747800c240 con 0x7f74780047c0 2023-11-28T21:52:13.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.600+0000 7f748b7fe640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74780039c0 0x7f7478003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:13.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.601+0000 7f748bfff640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7478005180 0x7f747800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:13.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.601+0000 7f748b7fe640 1 -- 172.21.15.45:0/1712513523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74780047c0 msgr2=0x7f7478004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:13.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.601+0000 7f748b7fe640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74780047c0 0x7f7478004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.601+0000 7f748b7fe640 1 -- 172.21.15.45:0/1712513523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7478005180 msgr2=0x7f747800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:13.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.601+0000 7f748b7fe640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7478005180 0x7f747800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.601+0000 7f748b7fe640 1 -- 172.21.15.45:0/1712513523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7478003520 con 0x7f74780039c0 2023-11-28T21:52:13.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.601+0000 7f748bfff640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7478005180 0x7f747800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:13.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.601+0000 7f748bfff640 1 -- 172.21.15.45:0/1712513523 reap_dead start 2023-11-28T21:52:13.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.602+0000 7f748b7fe640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74780039c0 0x7f7478003de0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f747c009ac0 tx=0x7f74780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81efb45938bfca4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:13.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.602+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c02c6b0 con 0x7f74780039c0 2023-11-28T21:52:13.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.602+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747c02c810 con 0x7f74780039c0 2023-11-28T21:52:13.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.602+0000 7f748affd640 1 -- 172.21.15.45:0/1712513523 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c02cba0 con 0x7f74780039c0 2023-11-28T21:52:13.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.603+0000 7f7490f9e640 1 -- 172.21.15.45:0/1712513523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74780039c0 msgr2=0x7f7478003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:13.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.603+0000 7f7490f9e640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74780039c0 0x7f7478003de0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f747c009ac0 tx=0x7f74780057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.603+0000 7f7490f9e640 1 -- 172.21.15.45:0/1712513523 shutdown_connections 2023-11-28T21:52:13.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.603+0000 7f7490f9e640 1 --2- 172.21.15.45:0/1712513523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74780039c0 0x7f7478003de0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.603+0000 7f7490f9e640 1 -- 172.21.15.45:0/1712513523 >> 172.21.15.45:0/1712513523 conn(0x7f748c0cec30 msgr2=0x7f748c0cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:13.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.603+0000 7f7490f9e640 1 -- 172.21.15.45:0/1712513523 shutdown_connections 2023-11-28T21:52:13.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.603+0000 7f7490f9e640 1 -- 172.21.15.45:0/1712513523 wait complete. 2023-11-28T21:52:13.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.605+0000 7f7490f9e640 1 Processor -- start 2023-11-28T21:52:13.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.605+0000 7f7490f9e640 1 -- start start 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.606+0000 7f7490f9e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74780039c0 0x7f748c0c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.606+0000 7f7490f9e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f748c0cd210 0x7f748c14b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.606+0000 7f7490f9e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f748c0c33e0 0x7f748c0c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.606+0000 7f7490f9e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0c3da0 con 0x7f748c0c33e0 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.606+0000 7f7490f9e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0cd980 con 0x7f748c0cd210 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.606+0000 7f7490f9e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0cdc60 con 0x7f74780039c0 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f748c0c33e0 0x7f748c0c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f748c0c33e0 0x7f748c0c3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40448/0 (socket says 172.21.15.45:40448) 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748bfff640 1 -- 172.21.15.45:0/2143420398 learned_addr learned my addr 172.21.15.45:0/2143420398 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f7483fff640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f748c0cd210 0x7f748c14b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:13.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748b7fe640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74780039c0 0x7f748c0c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748bfff640 1 -- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74780039c0 msgr2=0x7f748c0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748bfff640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74780039c0 0x7f748c0c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748bfff640 1 -- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f748c0cd210 msgr2=0x7f748c14b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748bfff640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f748c0cd210 0x7f748c14b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748bfff640 1 -- 172.21.15.45:0/2143420398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c0cdf40 con 0x7f748c0c33e0 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748b7fe640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74780039c0 0x7f748c0c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.607+0000 7f748bfff640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f748c0c33e0 0x7f748c0c3860 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f748c0ae6c0 tx=0x7f748400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.608+0000 7f7488ff9640 1 -- 172.21.15.45:0/2143420398 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484004d60 con 0x7f748c0c33e0 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.608+0000 7f7488ff9640 1 -- 172.21.15.45:0/2143420398 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748400bb80 con 0x7f748c0c33e0 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.608+0000 7f7488ff9640 1 -- 172.21.15.45:0/2143420398 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484007070 con 0x7f748c0c33e0 2023-11-28T21:52:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.608+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748c0c7630 con 0x7f748c0c33e0 2023-11-28T21:52:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.608+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748c14b560 con 0x7f748c0c33e0 2023-11-28T21:52:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.609+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748c0ce180 con 0x7f748c0c33e0 2023-11-28T21:52:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.610+0000 7f7488ff9640 1 -- 172.21.15.45:0/2143420398 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f74840071d0 con 0x7f748c0c33e0 2023-11-28T21:52:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.611+0000 7f7488ff9640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f746c06af60 0x7f746c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.611+0000 7f7488ff9640 1 -- 172.21.15.45:0/2143420398 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f7484089e90 con 0x7f748c0c33e0 2023-11-28T21:52:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.612+0000 7f748b7fe640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f746c06af60 0x7f746c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.613+0000 7f748b7fe640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f746c06af60 0x7f746c06d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f747c036000 tx=0x7f747c030000 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:13.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.615+0000 7f7488ff9640 1 -- 172.21.15.45:0/2143420398 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f74840549d0 con 0x7f748c0c33e0 2023-11-28T21:52:13.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:13.810+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f748c06e3a0 con 0x7f748c0c33e0 2023-11-28T21:52:14.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.356+0000 7f7488ff9640 1 -- 172.21.15.45:0/2143420398 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v190) v1 ==== 169+0+0 (secure 0 0 0) 0x7f748c06e3a0 con 0x7f748c0c33e0 2023-11-28T21:52:14.359 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T21:52:14.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.361+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f746c06af60 msgr2=0x7f746c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:14.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.361+0000 7f7490f9e640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f746c06af60 0x7f746c06d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f747c036000 tx=0x7f747c030000 comp rx=0 tx=0).stop 2023-11-28T21:52:14.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.362+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f748c0c33e0 msgr2=0x7f748c0c3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:14.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.362+0000 7f7490f9e640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f748c0c33e0 0x7f748c0c3860 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f748c0ae6c0 tx=0x7f748400b4a0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.366+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 shutdown_connections 2023-11-28T21:52:14.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.366+0000 7f7490f9e640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f746c06af60 0x7f746c06d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.366+0000 7f7490f9e640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f748c0c33e0 0x7f748c0c3860 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.366+0000 7f7490f9e640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f748c0cd210 0x7f748c14b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.366+0000 7f7490f9e640 1 --2- 172.21.15.45:0/2143420398 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f74780039c0 0x7f748c0c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.366+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 >> 172.21.15.45:0/2143420398 conn(0x7f748c0cec30 msgr2=0x7f748c0afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:14.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.367+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 shutdown_connections 2023-11-28T21:52:14.370 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.367+0000 7f7490f9e640 1 -- 172.21.15.45:0/2143420398 wait complete. 2023-11-28T21:52:14.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T21:52:14.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.604+0000 7f14d8a10640 1 Processor -- start 2023-11-28T21:52:14.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.604+0000 7f14d8a10640 1 -- start start 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.606+0000 7f14d8a10640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f14d406db90 0x7f14d40cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.606+0000 7f14d8a10640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f14d406df70 0x7f14d40b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.606+0000 7f14d8a10640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d406d9e0 con 0x7f14d415d3c0 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.606+0000 7f14d8a10640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d40b2a40 con 0x7f14d406df70 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.606+0000 7f14d8a10640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d406e400 con 0x7f14d406db90 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.606+0000 7f14d2d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f14d406df70 0x7f14d40b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.606+0000 7f14d2d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f14d406df70 0x7f14d40b31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53508/0 (socket says 172.21.15.45:53508) 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.606+0000 7f14d2d9d640 1 -- 172.21.15.45:0/3027175629 learned_addr learned my addr 172.21.15.45:0/3027175629 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.607+0000 7f14d2d9d640 1 -- 172.21.15.45:0/3027175629 >> v2:172.21.15.190:3300/0 conn(0x7f14d406db90 msgr2=0x7f14d40cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.607+0000 7f14d2d9d640 1 --2- 172.21.15.45:0/3027175629 >> v2:172.21.15.190:3300/0 conn(0x7f14d406db90 0x7f14d40cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.607+0000 7f14d2d9d640 1 -- 172.21.15.45:0/3027175629 >> v1:172.21.15.190:6789/0 conn(0x7f14d415d3c0 legacy=0x7f14d406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:14.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.607+0000 7f14d2d9d640 1 -- 172.21.15.45:0/3027175629 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d415d960 con 0x7f14d406df70 2023-11-28T21:52:14.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.608+0000 7f14d2d9d640 1 --2- 172.21.15.45:0/3027175629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d406df70 0x7f14d40b31c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f14d40d07a0 tx=0x7f14cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c73ea2aaa5ed8c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:14.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.608+0000 7f14d1d9b640 1 -- 172.21.15.45:0/3027175629 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc031870 con 0x7f14d406df70 2023-11-28T21:52:14.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.608+0000 7f14d1d9b640 1 -- 172.21.15.45:0/3027175629 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc0319d0 con 0x7f14d406df70 2023-11-28T21:52:14.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.608+0000 7f14d1d9b640 1 -- 172.21.15.45:0/3027175629 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc031d60 con 0x7f14d406df70 2023-11-28T21:52:14.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.609+0000 7f14d8a10640 1 -- 172.21.15.45:0/3027175629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d406df70 msgr2=0x7f14d40b31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:14.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.609+0000 7f14d8a10640 1 --2- 172.21.15.45:0/3027175629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d406df70 0x7f14d40b31c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f14d40d07a0 tx=0x7f14cc02b910 comp rx=0 tx=0).stop 2023-11-28T21:52:14.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.609+0000 7f14d8a10640 1 -- 172.21.15.45:0/3027175629 shutdown_connections 2023-11-28T21:52:14.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.609+0000 7f14d8a10640 1 --2- 172.21.15.45:0/3027175629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d406df70 0x7f14d40b31c0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.609+0000 7f14d8a10640 1 --2- 172.21.15.45:0/3027175629 >> v2:172.21.15.190:3300/0 conn(0x7f14d406db90 0x7f14d40cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.609+0000 7f14d8a10640 1 -- 172.21.15.45:0/3027175629 >> 172.21.15.45:0/3027175629 conn(0x7f14d406a500 msgr2=0x7f14d406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:14.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.609+0000 7f14d8a10640 1 -- 172.21.15.45:0/3027175629 shutdown_connections 2023-11-28T21:52:14.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.609+0000 7f14d8a10640 1 -- 172.21.15.45:0/3027175629 wait complete. 2023-11-28T21:52:14.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.611+0000 7f14d8a10640 1 Processor -- start 2023-11-28T21:52:14.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.611+0000 7f14d8a10640 1 -- start start 2023-11-28T21:52:14.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d8a10640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14d406db90 0x7f14d414f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:14.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d8a10640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14d406df70 0x7f14d414cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:14.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d8a10640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d415d3c0 0x7f14d41588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:14.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d8a10640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d414fb20 con 0x7f14d415d3c0 2023-11-28T21:52:14.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d8a10640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d414acf0 con 0x7f14d406db90 2023-11-28T21:52:14.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d8a10640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d4149820 con 0x7f14d406df70 2023-11-28T21:52:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d415d3c0 0x7f14d41588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d415d3c0 0x7f14d41588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53510/0 (socket says 172.21.15.45:53510) 2023-11-28T21:52:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d2d9d640 1 -- 172.21.15.45:0/862885171 learned_addr learned my addr 172.21.15.45:0/862885171 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14d259c640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14d406db90 0x7f14d414f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.612+0000 7f14cbfff640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14d406df70 0x7f14d414cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.613+0000 7f14d2d9d640 1 -- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14d406df70 msgr2=0x7f14d414cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.613+0000 7f14d2d9d640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14d406df70 0x7f14d414cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.613+0000 7f14d2d9d640 1 -- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14d406db90 msgr2=0x7f14d414f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:14.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.613+0000 7f14d2d9d640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14d406db90 0x7f14d414f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.613+0000 7f14d2d9d640 1 -- 172.21.15.45:0/862885171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc0094d0 con 0x7f14d415d3c0 2023-11-28T21:52:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.613+0000 7f14d2d9d640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d415d3c0 0x7f14d41588a0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f14cc02b950 tx=0x7f14cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.613+0000 7f14caffd640 1 -- 172.21.15.45:0/862885171 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc03abe0 con 0x7f14d415d3c0 2023-11-28T21:52:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.614+0000 7f14caffd640 1 -- 172.21.15.45:0/862885171 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc03ad60 con 0x7f14d415d3c0 2023-11-28T21:52:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.614+0000 7f14caffd640 1 -- 172.21.15.45:0/862885171 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc044630 con 0x7f14d415d3c0 2023-11-28T21:52:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.614+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d414fd50 con 0x7f14d415d3c0 2023-11-28T21:52:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.614+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d4150310 con 0x7f14d415d3c0 2023-11-28T21:52:14.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.615+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d406b490 con 0x7f14d415d3c0 2023-11-28T21:52:14.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.616+0000 7f14caffd640 1 -- 172.21.15.45:0/862885171 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f14cc044790 con 0x7f14d415d3c0 2023-11-28T21:52:14.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.617+0000 7f14caffd640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f14b406afb0 0x7f14b406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:14.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.617+0000 7f14caffd640 1 -- 172.21.15.45:0/862885171 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f14cc02d040 con 0x7f14d415d3c0 2023-11-28T21:52:14.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.618+0000 7f14d259c640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f14b406afb0 0x7f14b406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:14.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.618+0000 7f14d259c640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f14b406afb0 0x7f14b406d470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f14c400a400 tx=0x7f14c400d040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:14.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.621+0000 7f14caffd640 1 -- 172.21.15.45:0/862885171 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14cc076ea0 con 0x7f14d415d3c0 2023-11-28T21:52:14.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.818+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f14d40733d0 con 0x7f14d415d3c0 2023-11-28T21:52:14.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.823+0000 7f14caffd640 1 -- 172.21.15.45:0/862885171 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v61) v1 ==== 120+0+0 (secure 0 0 0) 0x7f14cc07a4c0 con 0x7f14d415d3c0 2023-11-28T21:52:14.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f14b406afb0 msgr2=0x7f14b406d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f14b406afb0 0x7f14b406d470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f14c400a400 tx=0x7f14c400d040 comp rx=0 tx=0).stop 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d415d3c0 msgr2=0x7f14d41588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d415d3c0 0x7f14d41588a0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f14cc02b950 tx=0x7f14cc0046c0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 shutdown_connections 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f14b406afb0 0x7f14b406d470 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14d415d3c0 0x7f14d41588a0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14d406df70 0x7f14d414cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 --2- 172.21.15.45:0/862885171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14d406db90 0x7f14d414f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 >> 172.21.15.45:0/862885171 conn(0x7f14d406a500 msgr2=0x7f14d415c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 shutdown_connections 2023-11-28T21:52:14.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:14.826+0000 7f14d8a10640 1 -- 172.21.15.45:0/862885171 wait complete. 2023-11-28T21:52:14.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T21:52:15.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.071+0000 7f772759e640 1 Processor -- start 2023-11-28T21:52:15.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.071+0000 7f772759e640 1 -- start start 2023-11-28T21:52:15.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.073+0000 7f772759e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f772815b080 0x7f77280ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.073+0000 7f772759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280ac270 con 0x7f77280abd70 2023-11-28T21:52:15.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.073+0000 7f772759e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772815b830 con 0x7f7728071fd0 2023-11-28T21:52:15.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.073+0000 7f772759e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f772815bb10 con 0x7f772815b080 2023-11-28T21:52:15.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.073+0000 7f7725d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f77280abd70 0x7f77280718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42370/0 (socket says 172.21.15.45:42370) 2023-11-28T21:52:15.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.073+0000 7f7725d9b640 1 -- 172.21.15.45:0/299583678 learned_addr learned my addr 172.21.15.45:0/299583678 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772659c640 1 --2- 172.21.15.45:0/299583678 >> v2:172.21.15.190:3300/0 conn(0x7f772815b080 0x7f77280ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743598780 0 0) 0x7f77280ac270 con 0x7f77280abd70 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7710003520 con 0x7f77280abd70 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3385628975 0 0) 0x7f7710003520 con 0x7f77280abd70 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 >> v2:172.21.15.190:3300/0 conn(0x7f772815b080 msgr2=0x7f77280ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772559a640 1 --2- 172.21.15.45:0/299583678 >> v2:172.21.15.190:3300/0 conn(0x7f772815b080 0x7f77280ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 >> v1:172.21.15.190:6789/0 conn(0x7f7728071fd0 legacy=0x7f77280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772815bdf0 con 0x7f77280abd70 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f771c001670 con 0x7f77280abd70 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.074+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 >> v1:172.21.15.45:6789/0 conn(0x7f77280abd70 legacy=0x7f77280718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f772559a640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77100039c0 0x7f7710003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f772559a640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77100047c0 0x7f7710004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f772559a640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100051d0 0x7f771000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772815bb10 con 0x7f77100039c0 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771000c100 con 0x7f77100047c0 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771000c2c0 con 0x7f77100051d0 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f772659c640 1 --2- 172.21.15.45:0/299583678 >> v2:172.21.15.190:3300/0 conn(0x7f772815b080 0x7f77280ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f772659c640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77100039c0 0x7f7710003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f7725d9b640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77100047c0 0x7f7710004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.075+0000 7f7726d9d640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100051d0 0x7f771000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.076+0000 7f772659c640 1 -- 172.21.15.45:0/299583678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100051d0 msgr2=0x7f771000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.076+0000 7f772659c640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100051d0 0x7f771000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.076+0000 7f772659c640 1 -- 172.21.15.45:0/299583678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77100047c0 msgr2=0x7f7710004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.076+0000 7f772659c640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77100047c0 0x7f7710004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.076+0000 7f772659c640 1 -- 172.21.15.45:0/299583678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710003520 con 0x7f77100039c0 2023-11-28T21:52:15.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.076+0000 7f7725d9b640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77100047c0 0x7f7710004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:15.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.076+0000 7f7726d9d640 1 -- 172.21.15.45:0/299583678 reap_dead start 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.077+0000 7f772659c640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77100039c0 0x7f7710003dc0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f7718006fd0 tx=0x7f77100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c37f24a29293c2b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.077+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77180305f0 con 0x7f77100039c0 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.077+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7718030750 con 0x7f77100039c0 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.077+0000 7f772559a640 1 -- 172.21.15.45:0/299583678 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718030ac0 con 0x7f77100039c0 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.078+0000 7f772759e640 1 -- 172.21.15.45:0/299583678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77100039c0 msgr2=0x7f7710003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.078+0000 7f772759e640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77100039c0 0x7f7710003dc0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f7718006fd0 tx=0x7f77100057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.078+0000 7f772759e640 1 -- 172.21.15.45:0/299583678 shutdown_connections 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.078+0000 7f772759e640 1 --2- 172.21.15.45:0/299583678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77100039c0 0x7f7710003dc0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.078+0000 7f772759e640 1 -- 172.21.15.45:0/299583678 >> 172.21.15.45:0/299583678 conn(0x7f77280cecd0 msgr2=0x7f77280cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.078+0000 7f772759e640 1 -- 172.21.15.45:0/299583678 shutdown_connections 2023-11-28T21:52:15.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.078+0000 7f772759e640 1 -- 172.21.15.45:0/299583678 wait complete. 2023-11-28T21:52:15.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.080+0000 7f772759e640 1 Processor -- start 2023-11-28T21:52:15.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.080+0000 7f772759e640 1 -- start start 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.081+0000 7f772759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100039c0 0x7f7728153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.081+0000 7f772759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77281f3130 0x7f772814f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.081+0000 7f772759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f772814ffa0 0x7f77281518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.081+0000 7f772759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7728154090 con 0x7f77281f3130 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.081+0000 7f772759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77280b2a60 con 0x7f772814ffa0 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.081+0000 7f772759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7728153e40 con 0x7f77100039c0 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7725d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77281f3130 0x7f772814f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7725d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77281f3130 0x7f772814f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53526/0 (socket says 172.21.15.45:53526) 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7725d9b640 1 -- 172.21.15.45:0/1683536495 learned_addr learned my addr 172.21.15.45:0/1683536495 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f772659c640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100039c0 0x7f7728153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7726d9d640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f772814ffa0 0x7f77281518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7725d9b640 1 -- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100039c0 msgr2=0x7f7728153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7725d9b640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100039c0 0x7f7728153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7725d9b640 1 -- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f772814ffa0 msgr2=0x7f77281518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7725d9b640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f772814ffa0 0x7f77281518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7725d9b640 1 -- 172.21.15.45:0/1683536495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280b20c0 con 0x7f77281f3130 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f772659c640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100039c0 0x7f7728153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.082+0000 7f7725d9b640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77281f3130 0x7f772814f7f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f771c001b00 tx=0x7f771c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.083+0000 7f7716ffd640 1 -- 172.21.15.45:0/1683536495 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c00ee60 con 0x7f77281f3130 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.083+0000 7f7716ffd640 1 -- 172.21.15.45:0/1683536495 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771c00c870 con 0x7f77281f3130 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.083+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77280b23a0 con 0x7f77281f3130 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.083+0000 7f7716ffd640 1 -- 172.21.15.45:0/1683536495 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c00cc00 con 0x7f77281f3130 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.083+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77281507c0 con 0x7f77281f3130 2023-11-28T21:52:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.084+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77280b2ba0 con 0x7f77281f3130 2023-11-28T21:52:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.085+0000 7f7716ffd640 1 -- 172.21.15.45:0/1683536495 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f771c01d460 con 0x7f77281f3130 2023-11-28T21:52:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.086+0000 7f7716ffd640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f770806af60 0x7f770806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.086+0000 7f7716ffd640 1 -- 172.21.15.45:0/1683536495 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f771c0897c0 con 0x7f77281f3130 2023-11-28T21:52:15.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.087+0000 7f772659c640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f770806af60 0x7f770806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.088+0000 7f772659c640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f770806af60 0x7f770806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7718036000 tx=0x7f7718031000 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:15.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.091+0000 7f7716ffd640 1 -- 172.21.15.45:0/1683536495 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77280b2ba0 con 0x7f77281f3130 2023-11-28T21:52:15.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.288+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f77280b2ba0 con 0x7f77281f3130 2023-11-28T21:52:15.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.293+0000 7f7716ffd640 1 -- 172.21.15.45:0/1683536495 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f77280b2ba0 con 0x7f77281f3130 2023-11-28T21:52:15.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f770806af60 msgr2=0x7f770806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f770806af60 0x7f770806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7718036000 tx=0x7f7718031000 comp rx=0 tx=0).stop 2023-11-28T21:52:15.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77281f3130 msgr2=0x7f772814f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77281f3130 0x7f772814f7f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f771c001b00 tx=0x7f771c00ec10 comp rx=0 tx=0).stop 2023-11-28T21:52:15.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 shutdown_connections 2023-11-28T21:52:15.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f770806af60 0x7f770806d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f772814ffa0 0x7f77281518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77281f3130 0x7f772814f7f0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 --2- 172.21.15.45:0/1683536495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f77100039c0 0x7f7728153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 >> 172.21.15.45:0/1683536495 conn(0x7f77280cecd0 msgr2=0x7f77280af2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:15.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 shutdown_connections 2023-11-28T21:52:15.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.296+0000 7f772759e640 1 -- 172.21.15.45:0/1683536495 wait complete. 2023-11-28T21:52:15.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T21:52:15.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.532+0000 7f381ce09640 1 Processor -- start 2023-11-28T21:52:15.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.532+0000 7f381ce09640 1 -- start start 2023-11-28T21:52:15.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.533+0000 7f381ce09640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f38180cfd20 0x7f38180d18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.533+0000 7f381ce09640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38180b2930 con 0x7f38180d1e10 2023-11-28T21:52:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.533+0000 7f381ce09640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38180b2650 con 0x7f38180cf940 2023-11-28T21:52:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.533+0000 7f381ce09640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38180ce240 con 0x7f38180cfd20 2023-11-28T21:52:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.533+0000 7f3817fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f38180d1e10 0x7f38180d22a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42386/0 (socket says 172.21.15.45:42386) 2023-11-28T21:52:15.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.533+0000 7f3817fff640 1 -- 172.21.15.45:0/808372575 learned_addr learned my addr 172.21.15.45:0/808372575 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:15.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f3816ffd640 1 --2- 172.21.15.45:0/808372575 >> v2:172.21.15.190:3300/0 conn(0x7f38180cfd20 0x7f38180d18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790067021 0 0) 0x7f38180b2930 con 0x7f38180d1e10 2023-11-28T21:52:15.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3804003520 con 0x7f38180d1e10 2023-11-28T21:52:15.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2439225520 0 0) 0x7f3804003520 con 0x7f38180d1e10 2023-11-28T21:52:15.538 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 >> v2:172.21.15.190:3300/0 conn(0x7f38180cfd20 msgr2=0x7f38180d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f38167fc640 1 --2- 172.21.15.45:0/808372575 >> v2:172.21.15.190:3300/0 conn(0x7f38180cfd20 0x7f38180d18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 >> v1:172.21.15.190:6789/0 conn(0x7f38180cf940 legacy=0x7f3818161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38180ce520 con 0x7f38180d1e10 2023-11-28T21:52:15.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3810001670 con 0x7f38180d1e10 2023-11-28T21:52:15.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.534+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 >> v1:172.21.15.45:6789/0 conn(0x7f38180d1e10 legacy=0x7f38180d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f38167fc640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38040039c0 0x7f3804003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f38167fc640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38040047c0 0x7f3804004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f38167fc640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38040052a0 0x7f380400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38180ce240 con 0x7f38040047c0 2023-11-28T21:52:15.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380400c1a0 con 0x7f38040052a0 2023-11-28T21:52:15.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380400c360 con 0x7f38040039c0 2023-11-28T21:52:15.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f3816ffd640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38040047c0 0x7f3804004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f38177fe640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38040039c0 0x7f3804003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f3816ffd640 1 -- 172.21.15.45:0/808372575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38040039c0 msgr2=0x7f3804003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f3816ffd640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38040039c0 0x7f3804003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f3816ffd640 1 -- 172.21.15.45:0/808372575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38040052a0 msgr2=0x7f380400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f3816ffd640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38040052a0 0x7f380400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f3816ffd640 1 -- 172.21.15.45:0/808372575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3804003520 con 0x7f38040047c0 2023-11-28T21:52:15.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.535+0000 7f3817fff640 1 -- 172.21.15.45:0/808372575 reap_dead start 2023-11-28T21:52:15.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.537+0000 7f3816ffd640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38040047c0 0x7f3804004c40 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f3800009a90 tx=0x7f38040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe778f52bcef9d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:15.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.537+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380002c6b0 con 0x7f38040047c0 2023-11-28T21:52:15.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.537+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380002c810 con 0x7f38040047c0 2023-11-28T21:52:15.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.537+0000 7f38167fc640 1 -- 172.21.15.45:0/808372575 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380002cba0 con 0x7f38040047c0 2023-11-28T21:52:15.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.537+0000 7f381ce09640 1 -- 172.21.15.45:0/808372575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38040047c0 msgr2=0x7f3804004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.537+0000 7f381ce09640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38040047c0 0x7f3804004c40 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f3800009a90 tx=0x7f38040057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.538+0000 7f381ce09640 1 -- 172.21.15.45:0/808372575 shutdown_connections 2023-11-28T21:52:15.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.538+0000 7f381ce09640 1 --2- 172.21.15.45:0/808372575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38040047c0 0x7f3804004c40 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.538+0000 7f381ce09640 1 -- 172.21.15.45:0/808372575 >> 172.21.15.45:0/808372575 conn(0x7f381806a500 msgr2=0x7f381806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:15.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.538+0000 7f381ce09640 1 -- 172.21.15.45:0/808372575 shutdown_connections 2023-11-28T21:52:15.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.538+0000 7f381ce09640 1 -- 172.21.15.45:0/808372575 wait complete. 2023-11-28T21:52:15.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.539+0000 7f381ce09640 1 Processor -- start 2023-11-28T21:52:15.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.540+0000 7f381ce09640 1 -- start start 2023-11-28T21:52:15.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f381ce09640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38040047c0 0x7f3818145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f381ce09640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f381815faa0 0x7f3818146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f381ce09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f381815ff20 0x7f38181467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f381ce09640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381815efe0 con 0x7f381815ff20 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f381ce09640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38180ca4f0 con 0x7f381815faa0 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f381ce09640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38180ca7d0 con 0x7f38040047c0 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f3817fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f381815ff20 0x7f38181467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f3817fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f381815ff20 0x7f38181467c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53538/0 (socket says 172.21.15.45:53538) 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f3817fff640 1 -- 172.21.15.45:0/3838855596 learned_addr learned my addr 172.21.15.45:0/3838855596 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f38177fe640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38040047c0 0x7f3818145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.541+0000 7f3816ffd640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f381815faa0 0x7f3818146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f3817fff640 1 -- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38040047c0 msgr2=0x7f3818145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f3817fff640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38040047c0 0x7f3818145090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f3817fff640 1 -- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f381815faa0 msgr2=0x7f3818146280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f3817fff640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f381815faa0 0x7f3818146280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f3817fff640 1 -- 172.21.15.45:0/3838855596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381815f2a0 con 0x7f381815ff20 2023-11-28T21:52:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f3817fff640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f381815ff20 0x7f38181467c0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f38180b9200 tx=0x7f381000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f380ffff640 1 -- 172.21.15.45:0/3838855596 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38100074e0 con 0x7f381815ff20 2023-11-28T21:52:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f380ffff640 1 -- 172.21.15.45:0/3838855596 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3810007640 con 0x7f381815ff20 2023-11-28T21:52:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f380ffff640 1 -- 172.21.15.45:0/3838855596 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810004a20 con 0x7f381815ff20 2023-11-28T21:52:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381815f580 con 0x7f381815ff20 2023-11-28T21:52:15.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.542+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38180ca180 con 0x7f381815ff20 2023-11-28T21:52:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.544+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38180ca990 con 0x7f381815ff20 2023-11-28T21:52:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.545+0000 7f380ffff640 1 -- 172.21.15.45:0/3838855596 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3810004b80 con 0x7f381815ff20 2023-11-28T21:52:15.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.546+0000 7f380ffff640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f37f806af90 0x7f37f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.546+0000 7f380ffff640 1 -- 172.21.15.45:0/3838855596 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f3810088df0 con 0x7f381815ff20 2023-11-28T21:52:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.546+0000 7f38177fe640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f37f806af90 0x7f37f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.547+0000 7f38177fe640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f37f806af90 0x7f37f806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3818068190 tx=0x7f3808009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:15.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.550+0000 7f380ffff640 1 -- 172.21.15.45:0/3838855596 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3810053be0 con 0x7f381815ff20 2023-11-28T21:52:15.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.742+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f3818071b30 con 0x7f381815ff20 2023-11-28T21:52:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.750+0000 7f380ffff640 1 -- 172.21.15.45:0/3838855596 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v63) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3810057200 con 0x7f381815ff20 2023-11-28T21:52:15.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f37f806af90 msgr2=0x7f37f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f37f806af90 0x7f37f806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3818068190 tx=0x7f3808009040 comp rx=0 tx=0).stop 2023-11-28T21:52:15.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f381815ff20 msgr2=0x7f38181467c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f381815ff20 0x7f38181467c0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f38180b9200 tx=0x7f381000c760 comp rx=0 tx=0).stop 2023-11-28T21:52:15.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 shutdown_connections 2023-11-28T21:52:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f37f806af90 0x7f37f806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f381815ff20 0x7f38181467c0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f381815faa0 0x7f3818146280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 --2- 172.21.15.45:0/3838855596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38040047c0 0x7f3818145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 >> 172.21.15.45:0/3838855596 conn(0x7f381806a500 msgr2=0x7f38180b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 shutdown_connections 2023-11-28T21:52:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.753+0000 7f381ce09640 1 -- 172.21.15.45:0/3838855596 wait complete. 2023-11-28T21:52:15.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-11-28T21:52:15.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.978+0000 7f0c1b3ed640 1 Processor -- start 2023-11-28T21:52:15.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.978+0000 7f0c1b3ed640 1 -- start start 2023-11-28T21:52:15.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.980+0000 7f0c1b3ed640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0c140cc2a0 0x7f0c1415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.980+0000 7f0c1b3ed640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c1415be80 0x7f0c1415c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.980+0000 7f0c1b3ed640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c1415c670 con 0x7f0c14141600 2023-11-28T21:52:15.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.980+0000 7f0c1b3ed640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c140cda40 con 0x7f0c140cc2a0 2023-11-28T21:52:15.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.980+0000 7f0c1b3ed640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140ccc80 con 0x7f0c1415be80 2023-11-28T21:52:15.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.980+0000 7f0c1a3eb640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0c14141600 0x7f0c140cbb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42390/0 (socket says 172.21.15.45:42390) 2023-11-28T21:52:15.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.980+0000 7f0c1a3eb640 1 -- 172.21.15.45:0/1462209632 learned_addr learned my addr 172.21.15.45:0/1462209632 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:15.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c1abec640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c1415be80 0x7f0c1415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c19bea640 1 --2- 172.21.15.45:0/1462209632 >> v2:172.21.15.190:3300/0 conn(0x7f0c140cc2a0 0x7f0c1415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023440939 0 0) 0x7f0c1415c670 con 0x7f0c14141600 2023-11-28T21:52:15.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c00003520 con 0x7f0c14141600 2023-11-28T21:52:15.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4241571211 0 0) 0x7f0c00003520 con 0x7f0c14141600 2023-11-28T21:52:15.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c1415be80 msgr2=0x7f0c1415c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c193e9640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c1415be80 0x7f0c1415c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 >> v2:172.21.15.190:3300/0 conn(0x7f0c140cc2a0 msgr2=0x7f0c1415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c193e9640 1 --2- 172.21.15.45:0/1462209632 >> v2:172.21.15.190:3300/0 conn(0x7f0c140cc2a0 0x7f0c1415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c140ccf60 con 0x7f0c14141600 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.981+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f0c10001670 con 0x7f0c14141600 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 >> v1:172.21.15.45:6789/0 conn(0x7f0c14141600 legacy=0x7f0c140cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c193e9640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00003970 0x7f0c00003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c193e9640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c1415be80 0x7f0c000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c193e9640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c00004d00 0x7f0c00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140cda40 con 0x7f0c00004d00 2023-11-28T21:52:15.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140ccc80 con 0x7f0c00003970 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0000b980 con 0x7f0c1415be80 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c1abec640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c00004d00 0x7f0c00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c1a3eb640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00003970 0x7f0c00003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c19bea640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c1415be80 0x7f0c000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c1abec640 1 -- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c1415be80 msgr2=0x7f0c000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c1abec640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c1415be80 0x7f0c000047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c1abec640 1 -- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00003970 msgr2=0x7f0c00003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c1abec640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00003970 0x7f0c00003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c1abec640 1 -- 172.21.15.45:0/1462209632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00003520 con 0x7f0c00004d00 2023-11-28T21:52:15.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.982+0000 7f0c19bea640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c1415be80 0x7f0c000047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.984+0000 7f0c1abec640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c00004d00 0x7f0c00005180 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00e170 tx=0x7f0c000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38701eefb03e7e91 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.984+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c02cc20 con 0x7f0c00004d00 2023-11-28T21:52:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.984+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c00f040 con 0x7f0c00004d00 2023-11-28T21:52:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.984+0000 7f0c193e9640 1 -- 172.21.15.45:0/1462209632 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c034440 con 0x7f0c00004d00 2023-11-28T21:52:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.984+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/1462209632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c00004d00 msgr2=0x7f0c00005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.984+0000 7f0c1b3ed640 1 --2- 172.21.15.45:0/1462209632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c00004d00 0x7f0c00005180 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00e170 tx=0x7f0c000057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.984+0000 7f0c1abec640 1 -- 172.21.15.45:0/1462209632 reap_dead start 2023-11-28T21:52:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.985+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/1462209632 shutdown_connections 2023-11-28T21:52:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.985+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/1462209632 >> 172.21.15.45:0/1462209632 conn(0x7f0c140b0e70 msgr2=0x7f0c140b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.985+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/1462209632 shutdown_connections 2023-11-28T21:52:15.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.985+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/1462209632 wait complete. 2023-11-28T21:52:15.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.987+0000 7f0c1b3ed640 1 Processor -- start 2023-11-28T21:52:15.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.987+0000 7f0c1b3ed640 1 -- start start 2023-11-28T21:52:15.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.988+0000 7f0c1b3ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00004d00 0x7f0c1415a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.988+0000 7f0c1b3ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c1415cfe0 0x7f0c1415d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.988+0000 7f0c1b3ed640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c140c8180 0x7f0c140c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.988+0000 7f0c1b3ed640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1412fe40 con 0x7f0c1415cfe0 2023-11-28T21:52:15.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.988+0000 7f0c1b3ed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1415d980 con 0x7f0c00004d00 2023-11-28T21:52:15.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.988+0000 7f0c1b3ed640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140c8b40 con 0x7f0c140c8180 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.988+0000 7f0c19bea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c1415cfe0 0x7f0c1415d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.988+0000 7f0c19bea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c1415cfe0 0x7f0c1415d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53564/0 (socket says 172.21.15.45:53564) 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.988+0000 7f0c19bea640 1 -- 172.21.15.45:0/3800962604 learned_addr learned my addr 172.21.15.45:0/3800962604 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c1a3eb640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00004d00 0x7f0c1415a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c1abec640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c140c8180 0x7f0c140c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c19bea640 1 -- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c140c8180 msgr2=0x7f0c140c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c19bea640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c140c8180 0x7f0c140c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c19bea640 1 -- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00004d00 msgr2=0x7f0c1415a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c19bea640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00004d00 0x7f0c1415a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c19bea640 1 -- 172.21.15.45:0/3800962604 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1412f1c0 con 0x7f0c1415cfe0 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c1abec640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c140c8180 0x7f0c140c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:15.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c1a3eb640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00004d00 0x7f0c1415a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c19bea640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c1415cfe0 0x7f0c1415d440 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f0c040109a0 tx=0x7f0c04010e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c0affd640 1 -- 172.21.15.45:0/3800962604 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0400ec20 con 0x7f0c1415cfe0 2023-11-28T21:52:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c0affd640 1 -- 172.21.15.45:0/3800962604 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0400ed80 con 0x7f0c1415cfe0 2023-11-28T21:52:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c0affd640 1 -- 172.21.15.45:0/3800962604 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0401c620 con 0x7f0c1415cfe0 2023-11-28T21:52:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.989+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1412f4a0 con 0x7f0c1415cfe0 2023-11-28T21:52:15.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.990+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1412fa60 con 0x7f0c1415cfe0 2023-11-28T21:52:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.991+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1404e920 con 0x7f0c1415cfe0 2023-11-28T21:52:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.992+0000 7f0c0affd640 1 -- 172.21.15.45:0/3800962604 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f0c0401c780 con 0x7f0c1415cfe0 2023-11-28T21:52:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.993+0000 7f0c0affd640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f0bf806af60 0x7f0bf806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.993+0000 7f0c0affd640 1 -- 172.21.15.45:0/3800962604 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f0c04089f90 con 0x7f0c1415cfe0 2023-11-28T21:52:15.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.993+0000 7f0c1a3eb640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f0bf806af60 0x7f0bf806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:15.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.995+0000 7f0c1a3eb640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f0bf806af60 0x7f0bf806d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0c100017a0 tx=0x7f0c10009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:16.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:15.998+0000 7f0c0affd640 1 -- 172.21.15.45:0/3800962604 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c04054e30 con 0x7f0c1415cfe0 2023-11-28T21:52:16.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.272+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f0c140ce890 con 0x7f0c1415cfe0 2023-11-28T21:52:16.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.361+0000 7f0c0affd640 1 -- 172.21.15.45:0/3800962604 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2580) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0c04058450 con 0x7f0c1415cfe0 2023-11-28T21:52:16.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f0bf806af60 msgr2=0x7f0bf806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f0bf806af60 0x7f0bf806d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0c100017a0 tx=0x7f0c10009040 comp rx=0 tx=0).stop 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c1415cfe0 msgr2=0x7f0c1415d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c1415cfe0 0x7f0c1415d440 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f0c040109a0 tx=0x7f0c04010e60 comp rx=0 tx=0).stop 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 shutdown_connections 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f0bf806af60 0x7f0bf806d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c140c8180 0x7f0c140c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c1415cfe0 0x7f0c1415d440 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 --2- 172.21.15.45:0/3800962604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c00004d00 0x7f0c1415a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 >> 172.21.15.45:0/3800962604 conn(0x7f0c140b0e70 msgr2=0x7f0c140b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 shutdown_connections 2023-11-28T21:52:16.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.365+0000 7f0c1b3ed640 1 -- 172.21.15.45:0/3800962604 wait complete. 2023-11-28T21:52:16.387 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-11-28T21:52:16.387 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-11-28T21:52:16.387 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T21:52:16.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:52:16.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.599+0000 7f5cc759e640 1 Processor -- start 2023-11-28T21:52:16.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.600+0000 7f5cc759e640 1 -- start start 2023-11-28T21:52:16.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.602+0000 7f5cc759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc80ae540 0x7f5cc80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:16.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.602+0000 7f5cc759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc815c3c0 con 0x7f5cc8155780 2023-11-28T21:52:16.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.602+0000 7f5cc759e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc8155b60 con 0x7f5cc80ce630 2023-11-28T21:52:16.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.602+0000 7f5cc759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80ceac0 con 0x7f5cc80ae540 2023-11-28T21:52:16.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.602+0000 7f5cc659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5cc8155780 0x7f5cc80abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42406/0 (socket says 172.21.15.45:42406) 2023-11-28T21:52:16.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.602+0000 7f5cc659c640 1 -- 172.21.15.45:0/2856073772 learned_addr learned my addr 172.21.15.45:0/2856073772 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:16.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.602+0000 7f5cbffff640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc80ae540 0x7f5cc80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:16.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872729241 0 0) 0x7f5cc815c3c0 con 0x7f5cc8155780 2023-11-28T21:52:16.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb4003520 con 0x7f5cc8155780 2023-11-28T21:52:16.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3102772466 0 0) 0x7f5cb4003520 con 0x7f5cc8155780 2023-11-28T21:52:16.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc80ae540 msgr2=0x7f5cc80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc80ae540 0x7f5cc80ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 >> v1:172.21.15.190:6789/0 conn(0x7f5cc80ce630 legacy=0x7f5cc80b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc815ba90 con 0x7f5cc8155780 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5cb8001670 con 0x7f5cc8155780 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 >> v1:172.21.15.45:6789/0 conn(0x7f5cc8155780 legacy=0x7f5cc80abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc80ae540 0x7f5cb40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb4003ee0 0x7f5cb40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cb4004d00 0x7f5cb4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80ceac0 con 0x7f5cb4003ee0 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb400b9e0 con 0x7f5cb4004d00 2023-11-28T21:52:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.603+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb400bba0 con 0x7f5cc80ae540 2023-11-28T21:52:16.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.604+0000 7f5cbffff640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb4003ee0 0x7f5cb40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:16.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.604+0000 7f5cc659c640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc80ae540 0x7f5cb40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:16.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.604+0000 7f5cbffff640 1 -- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc80ae540 msgr2=0x7f5cb40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:16.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.604+0000 7f5cbffff640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc80ae540 0x7f5cb40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.604+0000 7f5cbffff640 1 -- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cb4004d00 msgr2=0x7f5cb4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:16.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.604+0000 7f5cbffff640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cb4004d00 0x7f5cb4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.604+0000 7f5cbffff640 1 -- 172.21.15.45:0/2856073772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb4003520 con 0x7f5cb4003ee0 2023-11-28T21:52:16.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.605+0000 7f5cbffff640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb4003ee0 0x7f5cb40047c0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009ac0 tx=0x7f5cb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac2918d0cbb41a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.606+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb002c600 con 0x7f5cb4003ee0 2023-11-28T21:52:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.606+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb002c760 con 0x7f5cb4003ee0 2023-11-28T21:52:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.606+0000 7f5cc5d9b640 1 -- 172.21.15.45:0/2856073772 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb002caf0 con 0x7f5cb4003ee0 2023-11-28T21:52:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.606+0000 7f5cc759e640 1 -- 172.21.15.45:0/2856073772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb4003ee0 msgr2=0x7f5cb40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.606+0000 7f5cc759e640 1 --2- 172.21.15.45:0/2856073772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cb4003ee0 0x7f5cb40047c0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009ac0 tx=0x7f5cb40057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.606+0000 7f5cc6d9d640 1 -- 172.21.15.45:0/2856073772 reap_dead start 2023-11-28T21:52:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.606+0000 7f5cc759e640 1 -- 172.21.15.45:0/2856073772 shutdown_connections 2023-11-28T21:52:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.606+0000 7f5cc759e640 1 -- 172.21.15.45:0/2856073772 >> 172.21.15.45:0/2856073772 conn(0x7f5cc806ab30 msgr2=0x7f5cc80b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.607+0000 7f5cc759e640 1 -- 172.21.15.45:0/2856073772 shutdown_connections 2023-11-28T21:52:16.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.607+0000 7f5cc759e640 1 -- 172.21.15.45:0/2856073772 wait complete. 2023-11-28T21:52:16.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.608+0000 7f5cc759e640 1 Processor -- start 2023-11-28T21:52:16.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.608+0000 7f5cc759e640 1 -- start start 2023-11-28T21:52:16.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.609+0000 7f5cc759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cb4003ee0 0x7f5cc814a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.609+0000 7f5cc759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc8150b10 0x7f5cc80aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.609+0000 7f5cc759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc81512b0 0x7f5cc8151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.609+0000 7f5cc759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80afff0 con 0x7f5cc81512b0 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.609+0000 7f5cc759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc8151c80 con 0x7f5cb4003ee0 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.609+0000 7f5cc759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80b0410 con 0x7f5cc8150b10 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cc6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc81512b0 0x7f5cc8151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cc6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc81512b0 0x7f5cc8151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53596/0 (socket says 172.21.15.45:53596) 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cc6d9d640 1 -- 172.21.15.45:0/271376131 learned_addr learned my addr 172.21.15.45:0/271376131 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cc659c640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cb4003ee0 0x7f5cc814a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:16.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cbffff640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc8150b10 0x7f5cc80aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cc6d9d640 1 -- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc8150b10 msgr2=0x7f5cc80aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cc6d9d640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc8150b10 0x7f5cc80aea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cc6d9d640 1 -- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cb4003ee0 msgr2=0x7f5cc814a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cc6d9d640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cb4003ee0 0x7f5cc814a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.610+0000 7f5cc6d9d640 1 -- 172.21.15.45:0/271376131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc80af370 con 0x7f5cc81512b0 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.611+0000 7f5cc6d9d640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc81512b0 0x7f5cc8151740 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f5cc806e4d0 tx=0x7f5cc000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.611+0000 7f5cbeffd640 1 -- 172.21.15.45:0/271376131 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0005d40 con 0x7f5cc81512b0 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.611+0000 7f5cbeffd640 1 -- 172.21.15.45:0/271376131 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc000a610 con 0x7f5cc81512b0 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.611+0000 7f5cbeffd640 1 -- 172.21.15.45:0/271376131 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0005ea0 con 0x7f5cc81512b0 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.611+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc80af650 con 0x7f5cc81512b0 2023-11-28T21:52:16.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.611+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc80afc10 con 0x7f5cc81512b0 2023-11-28T21:52:16.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.613+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc80b0130 con 0x7f5cc81512b0 2023-11-28T21:52:16.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.613+0000 7f5cbeffd640 1 -- 172.21.15.45:0/271376131 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5cc000a9c0 con 0x7f5cc81512b0 2023-11-28T21:52:16.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.614+0000 7f5cbeffd640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5ca806af60 0x7f5ca806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:16.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.615+0000 7f5cbeffd640 1 -- 172.21.15.45:0/271376131 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f5cc0088ec0 con 0x7f5cc81512b0 2023-11-28T21:52:16.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.615+0000 7f5cc659c640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5ca806af60 0x7f5ca806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:16.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.616+0000 7f5cc659c640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5ca806af60 0x7f5ca806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8000f60 tx=0x7f5cb8000ed0 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:16.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.619+0000 7f5cbeffd640 1 -- 172.21.15.45:0/271376131 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5cc80b0130 con 0x7f5cc81512b0 2023-11-28T21:52:16.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.804+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cc80b0130 con 0x7f5cc81512b0 2023-11-28T21:52:16.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.806+0000 7f5cbeffd640 1 -- 172.21.15.45:0/271376131 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+14245 (secure 0 0 0) 0x7f5cc80b0130 con 0x7f5cc81512b0 2023-11-28T21:52:16.809 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:16.809 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":190,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:52:14.346093+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6834/99455191":"2023-11-29T21:52:11.271867+0000","172.21.15.45:6835/99455191":"2023-11-29T21:52:11.271867+0000","172.21.15.190:0/3757577966":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/788198125":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/3142102971":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/1232828858":"2023-11-29T21:52:07.286885+0000","172.21.15.190:6833/1285100946":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/158019673":"2023-11-29T21:52:07.286885+0000","172.21.15.190:6832/1285100946":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/2404812260":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/102436699":"2023-11-29T21:52:07.286885+0000","172.21.15.190:0/2400910006":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6833/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2240536749":"2023-11-29T21:52:00.068699+0000","172.21.15.190:6832/2040970137":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/2615023161":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/4027676180":"2023-11-29T21:52:00.068699+0000","172.21.15.190:0/508215298":"2023-11-29T21:52:00.068699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:52:16.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5ca806af60 msgr2=0x7f5ca806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:16.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5ca806af60 0x7f5ca806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8000f60 tx=0x7f5cb8000ed0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc81512b0 msgr2=0x7f5cc8151740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:16.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc81512b0 0x7f5cc8151740 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f5cc806e4d0 tx=0x7f5cc000cb90 comp rx=0 tx=0).stop 2023-11-28T21:52:16.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 shutdown_connections 2023-11-28T21:52:16.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5ca806af60 0x7f5ca806d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc81512b0 0x7f5cc8151740 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cc8150b10 0x7f5cc80aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 --2- 172.21.15.45:0/271376131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cb4003ee0 0x7f5cc814a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 >> 172.21.15.45:0/271376131 conn(0x7f5cc806ab30 msgr2=0x7f5cc80ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 shutdown_connections 2023-11-28T21:52:16.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:16.809+0000 7f5cc759e640 1 -- 172.21.15.45:0/271376131 wait complete. 2023-11-28T21:52:16.831 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T21:52:16.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T21:52:17.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.054+0000 7f0695a91640 1 Processor -- start 2023-11-28T21:52:17.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.054+0000 7f0695a91640 1 -- start start 2023-11-28T21:52:17.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.056+0000 7f0695a91640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f069015b270 0x7f069006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.056+0000 7f0695a91640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069006d410 con 0x7f06900b3040 2023-11-28T21:52:17.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.056+0000 7f0695a91640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069006d6f0 con 0x7f06900b3420 2023-11-28T21:52:17.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.056+0000 7f0695a91640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06900ceaa0 con 0x7f069015b270 2023-11-28T21:52:17.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.057+0000 7f0694a8f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f06900b3040 0x7f069015f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42422/0 (socket says 172.21.15.45:42422) 2023-11-28T21:52:17.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.057+0000 7f0694a8f640 1 -- 172.21.15.45:0/4043492303 learned_addr learned my addr 172.21.15.45:0/4043492303 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:17.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.057+0000 7f068ffff640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f069015b270 0x7f069006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.057+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006602169 0 0) 0x7f069006d410 con 0x7f06900b3040 2023-11-28T21:52:17.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.057+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0680003520 con 0x7f06900b3040 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.057+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4286435836 0 0) 0x7f0680003520 con 0x7f06900b3040 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f069015b270 msgr2=0x7f069006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f069015b270 0x7f069006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 >> v1:172.21.15.190:6789/0 conn(0x7f06900b3420 legacy=0x7f06900ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069006d9d0 con 0x7f06900b3040 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f0684001670 con 0x7f06900b3040 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 >> v1:172.21.15.45:6789/0 conn(0x7f06900b3040 legacy=0x7f069015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06800039a0 0x7f0680003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06800047c0 0x7f0680004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f069015b270 0x7f0680005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068000bb10 con 0x7f06800039a0 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068000bc80 con 0x7f06800047c0 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068000be40 con 0x7f069015b270 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f0694a8f640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06800039a0 0x7f0680003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f068ffff640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06800047c0 0x7f0680004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f0694a8f640 1 -- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f069015b270 msgr2=0x7f0680005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.058+0000 7f0695290640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f069015b270 0x7f0680005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.059+0000 7f0694a8f640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f069015b270 0x7f0680005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.059+0000 7f0694a8f640 1 -- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06800047c0 msgr2=0x7f0680004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.059+0000 7f0694a8f640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06800047c0 0x7f0680004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.059+0000 7f0694a8f640 1 -- 172.21.15.45:0/4043492303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0680003520 con 0x7f06800039a0 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.059+0000 7f0695290640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f069015b270 0x7f0680005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:17.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.059+0000 7f068ffff640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06800047c0 0x7f0680004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.060+0000 7f0694a8f640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06800039a0 0x7f0680003dc0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f068400c240 tx=0x7f06800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe7af90bfd37562 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.060+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684005ad0 con 0x7f06800039a0 2023-11-28T21:52:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.060+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0684005c30 con 0x7f06800039a0 2023-11-28T21:52:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.060+0000 7f068f7fe640 1 -- 172.21.15.45:0/4043492303 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068402c8b0 con 0x7f06800039a0 2023-11-28T21:52:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.061+0000 7f0695a91640 1 -- 172.21.15.45:0/4043492303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06800039a0 msgr2=0x7f0680003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.061+0000 7f0695a91640 1 --2- 172.21.15.45:0/4043492303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06800039a0 0x7f0680003dc0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f068400c240 tx=0x7f06800057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.061+0000 7f0695290640 1 -- 172.21.15.45:0/4043492303 reap_dead start 2023-11-28T21:52:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.061+0000 7f0695a91640 1 -- 172.21.15.45:0/4043492303 shutdown_connections 2023-11-28T21:52:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.061+0000 7f0695a91640 1 -- 172.21.15.45:0/4043492303 >> 172.21.15.45:0/4043492303 conn(0x7f06900b1050 msgr2=0x7f06900cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:17.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.061+0000 7f0695a91640 1 -- 172.21.15.45:0/4043492303 shutdown_connections 2023-11-28T21:52:17.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.061+0000 7f0695a91640 1 -- 172.21.15.45:0/4043492303 wait complete. 2023-11-28T21:52:17.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.063+0000 7f0695a91640 1 Processor -- start 2023-11-28T21:52:17.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.063+0000 7f0695a91640 1 -- start start 2023-11-28T21:52:17.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f0695a91640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06800039a0 0x7f069009c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f0695a91640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0690140270 0x7f0690157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f0695a91640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0690158dd0 0x7f069015c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f0695a91640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0690157d30 con 0x7f0690140270 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f0695a91640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0690158010 con 0x7f0690158dd0 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f0695a91640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06901582f0 con 0x7f06800039a0 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f068ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0690140270 0x7f0690157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f068ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0690140270 0x7f0690157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53612/0 (socket says 172.21.15.45:53612) 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f068ffff640 1 -- 172.21.15.45:0/2340313466 learned_addr learned my addr 172.21.15.45:0/2340313466 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f0694a8f640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06800039a0 0x7f069009c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.064+0000 7f0695290640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0690158dd0 0x7f069015c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f068ffff640 1 -- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06800039a0 msgr2=0x7f069009c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f068ffff640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06800039a0 0x7f069009c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f068ffff640 1 -- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0690158dd0 msgr2=0x7f069015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f068ffff640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0690158dd0 0x7f069015c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f068ffff640 1 -- 172.21.15.45:0/2340313466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06901585d0 con 0x7f0690140270 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f0695290640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0690158dd0 0x7f069015c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f0694a8f640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06800039a0 0x7f069009c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f068ffff640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0690140270 0x7f0690157500 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f067c010880 tx=0x7f067c010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f068d7fa640 1 -- 172.21.15.45:0/2340313466 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c00eb60 con 0x7f0690140270 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f068d7fa640 1 -- 172.21.15.45:0/2340313466 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c00ecc0 con 0x7f0690140270 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f068d7fa640 1 -- 172.21.15.45:0/2340313466 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c01c400 con 0x7f0690140270 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06901588b0 con 0x7f0690140270 2023-11-28T21:52:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.065+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0690157a40 con 0x7f0690140270 2023-11-28T21:52:17.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.067+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069009d0f0 con 0x7f0690140270 2023-11-28T21:52:17.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.068+0000 7f068d7fa640 1 -- 172.21.15.45:0/2340313466 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f067c01c560 con 0x7f0690140270 2023-11-28T21:52:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.069+0000 7f068d7fa640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f067406af60 0x7f067406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.069+0000 7f068d7fa640 1 -- 172.21.15.45:0/2340313466 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f067c089ef0 con 0x7f0690140270 2023-11-28T21:52:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.069+0000 7f0694a8f640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f067406af60 0x7f067406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.071+0000 7f0694a8f640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f067406af60 0x7f067406d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0684034000 tx=0x7f0684009030 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:17.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.073+0000 7f068d7fa640 1 -- 172.21.15.45:0/2340313466 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f067c054d90 con 0x7f0690140270 2023-11-28T21:52:17.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.265+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f069004e920 con 0x7f0690140270 2023-11-28T21:52:17.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.266+0000 7f068d7fa640 1 -- 172.21.15.45:0/2340313466 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v190) v1 ==== 93+0+10 (secure 0 0 0) 0x7f067c0583b0 con 0x7f0690140270 2023-11-28T21:52:17.268 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T21:52:17.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f067406af60 msgr2=0x7f067406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f067406af60 0x7f067406d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0684034000 tx=0x7f0684009030 comp rx=0 tx=0).stop 2023-11-28T21:52:17.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0690140270 msgr2=0x7f0690157500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0690140270 0x7f0690157500 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f067c010880 tx=0x7f067c010d40 comp rx=0 tx=0).stop 2023-11-28T21:52:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 shutdown_connections 2023-11-28T21:52:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f067406af60 0x7f067406d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0690158dd0 0x7f069015c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0690140270 0x7f0690157500 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 --2- 172.21.15.45:0/2340313466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06800039a0 0x7f069009c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 >> 172.21.15.45:0/2340313466 conn(0x7f06900b1050 msgr2=0x7f06900cd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 shutdown_connections 2023-11-28T21:52:17.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.269+0000 7f0695a91640 1 -- 172.21.15.45:0/2340313466 wait complete. 2023-11-28T21:52:17.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-11-28T21:52:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.517+0000 7fe31759e640 1 Processor -- start 2023-11-28T21:52:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.517+0000 7fe31759e640 1 -- start start 2023-11-28T21:52:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.519+0000 7fe31759e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe3180ae540 0x7fe3180ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.519+0000 7fe31759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 0x7fe3180b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.519+0000 7fe31759e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31815c3c0 con 0x7fe318155780 2023-11-28T21:52:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.519+0000 7fe31759e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3180ceac0 con 0x7fe3180ae540 2023-11-28T21:52:17.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.519+0000 7fe31759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe318155b60 con 0x7fe3180ce630 2023-11-28T21:52:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.520+0000 7fe315d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe3180ae540 0x7fe3180ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.520+0000 7fe316d9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 0x7fe3180b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.520+0000 7fe316d9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 0x7fe3180b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:55784/0 (socket says 172.21.15.45:55784) 2023-11-28T21:52:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.520+0000 7fe316d9d640 1 -- 172.21.15.45:0/3726188054 learned_addr learned my addr 172.21.15.45:0/3726188054 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.520+0000 7fe315d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe3180ae540 0x7fe3180ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:41766/0 (socket says 172.21.15.45:41766) 2023-11-28T21:52:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.520+0000 7fe31659c640 1 --1- 172.21.15.45:0/3726188054 >> v1:172.21.15.190:6789/0 conn(0x7fe318155780 0x7fe3180abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:34848/0 (socket says 172.21.15.45:34848) 2023-11-28T21:52:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.521+0000 7fe315d9b640 1 -- 172.21.15.45:0/3726188054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 msgr2=0x7fe3180b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.521+0000 7fe315d9b640 1 --2- 172.21.15.45:0/3726188054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 0x7fe3180b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.521+0000 7fe315d9b640 1 -- 172.21.15.45:0/3726188054 >> v1:172.21.15.190:6789/0 conn(0x7fe318155780 legacy=0x7fe3180abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.521+0000 7fe315d9b640 1 -- 172.21.15.45:0/3726188054 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31815ba90 con 0x7fe3180ae540 2023-11-28T21:52:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.522+0000 7fe315d9b640 1 --2- 172.21.15.45:0/3726188054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3180ae540 0x7fe3180ce0f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0097a0 tx=0x7fe30c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85379bd4f6d35ec3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.523+0000 7fe31559a640 1 -- 172.21.15.45:0/3726188054 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c02d040 con 0x7fe3180ae540 2023-11-28T21:52:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.523+0000 7fe31559a640 1 -- 172.21.15.45:0/3726188054 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c031810 con 0x7fe3180ae540 2023-11-28T21:52:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.523+0000 7fe31559a640 1 -- 172.21.15.45:0/3726188054 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c031ba0 con 0x7fe3180ae540 2023-11-28T21:52:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.524+0000 7fe31759e640 1 -- 172.21.15.45:0/3726188054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3180ae540 msgr2=0x7fe3180ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.524+0000 7fe31759e640 1 --2- 172.21.15.45:0/3726188054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3180ae540 0x7fe3180ce0f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0097a0 tx=0x7fe30c02b8b0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.524+0000 7fe31759e640 1 -- 172.21.15.45:0/3726188054 shutdown_connections 2023-11-28T21:52:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.524+0000 7fe31759e640 1 --2- 172.21.15.45:0/3726188054 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 0x7fe3180b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.524+0000 7fe31759e640 1 --2- 172.21.15.45:0/3726188054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3180ae540 0x7fe3180ce0f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.524+0000 7fe31759e640 1 -- 172.21.15.45:0/3726188054 >> 172.21.15.45:0/3726188054 conn(0x7fe31806ab30 msgr2=0x7fe3180b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.524+0000 7fe31759e640 1 -- 172.21.15.45:0/3726188054 shutdown_connections 2023-11-28T21:52:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.524+0000 7fe31759e640 1 -- 172.21.15.45:0/3726188054 wait complete. 2023-11-28T21:52:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.526+0000 7fe31759e640 1 Processor -- start 2023-11-28T21:52:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.526+0000 7fe31759e640 1 -- start start 2023-11-28T21:52:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.528+0000 7fe31759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3180ae540 0x7fe318071c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.528+0000 7fe31759e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 0x7fe31813ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.528+0000 7fe31759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe318155780 0x7fe318149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.528+0000 7fe31759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318072160 con 0x7fe3180ae540 2023-11-28T21:52:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.528+0000 7fe31759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318141770 con 0x7fe318155780 2023-11-28T21:52:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.528+0000 7fe31759e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe318141fd0 con 0x7fe3180ce630 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.528+0000 7fe31659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3180ae540 0x7fe318071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.528+0000 7fe31659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3180ae540 0x7fe318071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53620/0 (socket says 172.21.15.45:53620) 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.528+0000 7fe31659c640 1 -- 172.21.15.45:0/1745295652 learned_addr learned my addr 172.21.15.45:0/1745295652 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe31659c640 1 -- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 msgr2=0x7fe31813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe31659c640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 0x7fe31813ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe31659c640 1 -- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe318155780 msgr2=0x7fe318149d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe316d9d640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe318155780 0x7fe318149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe31659c640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe318155780 0x7fe318149d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe31659c640 1 -- 172.21.15.45:0/1745295652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c009400 con 0x7fe3180ae540 2023-11-28T21:52:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe31659c640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3180ae540 0x7fe318071c20 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c3e0 tx=0x7fe30800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe306ffd640 1 -- 172.21.15.45:0/1745295652 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3080074e0 con 0x7fe3180ae540 2023-11-28T21:52:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe306ffd640 1 -- 172.21.15.45:0/1745295652 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe308007640 con 0x7fe3180ae540 2023-11-28T21:52:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe306ffd640 1 -- 172.21.15.45:0/1745295652 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3080049e0 con 0x7fe3180ae540 2023-11-28T21:52:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe318141a10 con 0x7fe3180ae540 2023-11-28T21:52:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.529+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3180b0cc0 con 0x7fe3180ae540 2023-11-28T21:52:17.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.530+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe318147170 con 0x7fe3180ae540 2023-11-28T21:52:17.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.532+0000 7fe306ffd640 1 -- 172.21.15.45:0/1745295652 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe308004b40 con 0x7fe3180ae540 2023-11-28T21:52:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.533+0000 7fe306ffd640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fe2f806af90 0x7fe2f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.533+0000 7fe306ffd640 1 -- 172.21.15.45:0/1745295652 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fe308089300 con 0x7fe3180ae540 2023-11-28T21:52:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.534+0000 7fe315d9b640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fe2f806af90 0x7fe2f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:17.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.535+0000 7fe315d9b640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fe2f806af90 0x7fe2f806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe30c036000 tx=0x7fe30c030000 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:17.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.539+0000 7fe306ffd640 1 -- 172.21.15.45:0/1745295652 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3080540f0 con 0x7fe3180ae540 2023-11-28T21:52:17.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:17.808+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fe31804e920 con 0x7fe3180ae540 2023-11-28T21:52:18.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.373+0000 7fe306ffd640 1 -- 172.21.15.45:0/1745295652 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2582) v1 ==== 168+0+0 (secure 0 0 0) 0x7fe308057710 con 0x7fe3180ae540 2023-11-28T21:52:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fe2f806af90 msgr2=0x7fe2f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fe2f806af90 0x7fe2f806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe30c036000 tx=0x7fe30c030000 comp rx=0 tx=0).stop 2023-11-28T21:52:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3180ae540 msgr2=0x7fe318071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3180ae540 0x7fe318071c20 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c3e0 tx=0x7fe30800c8a0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 shutdown_connections 2023-11-28T21:52:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fe2f806af90 0x7fe2f806d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe318155780 0x7fe318149d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3180ce630 0x7fe31813ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 --2- 172.21.15.45:0/1745295652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3180ae540 0x7fe318071c20 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.377+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 >> 172.21.15.45:0/1745295652 conn(0x7fe31806ab30 msgr2=0x7fe31814edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:18.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.378+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 shutdown_connections 2023-11-28T21:52:18.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.378+0000 7fe31759e640 1 -- 172.21.15.45:0/1745295652 wait complete. 2023-11-28T21:52:18.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T21:52:18.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.640+0000 7f468abfb640 1 Processor -- start 2023-11-28T21:52:18.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.640+0000 7f468abfb640 1 -- start start 2023-11-28T21:52:18.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.642+0000 7f468abfb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f468406ac10 0x7f468406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:18.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.642+0000 7f468abfb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f468415f720 0x7f46840717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:18.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.642+0000 7f468abfb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468415ecd0 con 0x7f468415e840 2023-11-28T21:52:18.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.642+0000 7f468abfb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4684071ce0 con 0x7f468415f720 2023-11-28T21:52:18.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.642+0000 7f468abfb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468415fbd0 con 0x7f468406ac10 2023-11-28T21:52:18.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.643+0000 7f4689198640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f468415f720 0x7f46840717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:18.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.643+0000 7f4689198640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f468415f720 0x7f46840717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53634/0 (socket says 172.21.15.45:53634) 2023-11-28T21:52:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.643+0000 7f4689198640 1 -- 172.21.15.45:0/2592994275 learned_addr learned my addr 172.21.15.45:0/2592994275 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.643+0000 7f4683fff640 1 --2- 172.21.15.45:0/2592994275 >> v2:172.21.15.190:3300/0 conn(0x7f468406ac10 0x7f468406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.643+0000 7f4689198640 1 -- 172.21.15.45:0/2592994275 >> v2:172.21.15.190:3300/0 conn(0x7f468406ac10 msgr2=0x7f468406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:18.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.643+0000 7f4689198640 1 --2- 172.21.15.45:0/2592994275 >> v2:172.21.15.190:3300/0 conn(0x7f468406ac10 0x7f468406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.643+0000 7f4689198640 1 -- 172.21.15.45:0/2592994275 >> v1:172.21.15.190:6789/0 conn(0x7f468415e840 legacy=0x7f468406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:18.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.644+0000 7f4689198640 1 -- 172.21.15.45:0/2592994275 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468415f290 con 0x7f468415f720 2023-11-28T21:52:18.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.645+0000 7f4689198640 1 --2- 172.21.15.45:0/2592994275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415f720 0x7f46840717a0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f468406e6d0 tx=0x7f467c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb91c5f4d0736c01 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:18.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.645+0000 7f46837fe640 1 -- 172.21.15.45:0/2592994275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c02cb20 con 0x7f468415f720 2023-11-28T21:52:18.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.645+0000 7f46837fe640 1 -- 172.21.15.45:0/2592994275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467c02cc80 con 0x7f468415f720 2023-11-28T21:52:18.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.645+0000 7f46837fe640 1 -- 172.21.15.45:0/2592994275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c0343f0 con 0x7f468415f720 2023-11-28T21:52:18.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.646+0000 7f468abfb640 1 -- 172.21.15.45:0/2592994275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415f720 msgr2=0x7f46840717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:18.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.646+0000 7f468abfb640 1 --2- 172.21.15.45:0/2592994275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415f720 0x7f46840717a0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f468406e6d0 tx=0x7f467c02bf80 comp rx=0 tx=0).stop 2023-11-28T21:52:18.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.646+0000 7f468abfb640 1 -- 172.21.15.45:0/2592994275 shutdown_connections 2023-11-28T21:52:18.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.646+0000 7f468abfb640 1 --2- 172.21.15.45:0/2592994275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415f720 0x7f46840717a0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.646+0000 7f468abfb640 1 --2- 172.21.15.45:0/2592994275 >> v2:172.21.15.190:3300/0 conn(0x7f468406ac10 0x7f468406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.646+0000 7f468abfb640 1 -- 172.21.15.45:0/2592994275 >> 172.21.15.45:0/2592994275 conn(0x7f46840ced60 msgr2=0x7f46840cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:18.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.646+0000 7f468abfb640 1 -- 172.21.15.45:0/2592994275 shutdown_connections 2023-11-28T21:52:18.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.646+0000 7f468abfb640 1 -- 172.21.15.45:0/2592994275 wait complete. 2023-11-28T21:52:18.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.648+0000 7f468abfb640 1 Processor -- start 2023-11-28T21:52:18.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.648+0000 7f468abfb640 1 -- start start 2023-11-28T21:52:18.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.649+0000 7f468abfb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f468406ac10 0x7f46840b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:18.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.649+0000 7f468abfb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415e840 0x7f46840c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.649+0000 7f468abfb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f468415f720 0x7f4684149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.649+0000 7f468abfb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46840c7f20 con 0x7f468415e840 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.649+0000 7f468abfb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46840c8200 con 0x7f468406ac10 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.649+0000 7f468abfb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46840c84e0 con 0x7f468415f720 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4683fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415e840 0x7f46840c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4683fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415e840 0x7f46840c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53636/0 (socket says 172.21.15.45:53636) 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4683fff640 1 -- 172.21.15.45:0/691520523 learned_addr learned my addr 172.21.15.45:0/691520523 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4688997640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f468406ac10 0x7f46840b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4689198640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f468415f720 0x7f4684149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4683fff640 1 -- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f468415f720 msgr2=0x7f4684149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:18.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4683fff640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f468415f720 0x7f4684149bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4683fff640 1 -- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f468406ac10 msgr2=0x7f46840b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:18.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4683fff640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f468406ac10 0x7f46840b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4683fff640 1 -- 172.21.15.45:0/691520523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c009b30 con 0x7f468415e840 2023-11-28T21:52:18.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.650+0000 7f4683fff640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415e840 0x7f46840c5e30 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f467000d800 tx=0x7f467000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:18.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.651+0000 7f46817fa640 1 -- 172.21.15.45:0/691520523 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467000b6c0 con 0x7f468415e840 2023-11-28T21:52:18.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.651+0000 7f46817fa640 1 -- 172.21.15.45:0/691520523 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467000b820 con 0x7f468415e840 2023-11-28T21:52:18.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.651+0000 7f46817fa640 1 -- 172.21.15.45:0/691520523 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467000bbb0 con 0x7f468415e840 2023-11-28T21:52:18.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.651+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46840c87c0 con 0x7f468415e840 2023-11-28T21:52:18.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.651+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46840c7b10 con 0x7f468415e840 2023-11-28T21:52:18.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.652+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46840c7ca0 con 0x7f468415e840 2023-11-28T21:52:18.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.653+0000 7f46817fa640 1 -- 172.21.15.45:0/691520523 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f467000bd10 con 0x7f468415e840 2023-11-28T21:52:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.654+0000 7f46817fa640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f466806af90 0x7f466806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:18.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.654+0000 7f46817fa640 1 -- 172.21.15.45:0/691520523 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f4670089e50 con 0x7f468415e840 2023-11-28T21:52:18.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.655+0000 7f4688997640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f466806af90 0x7f466806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.656+0000 7f4688997640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f466806af90 0x7f466806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4678001640 tx=0x7f4678009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:18.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.659+0000 7f46817fa640 1 -- 172.21.15.45:0/691520523 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4670054c40 con 0x7f468415e840 2023-11-28T21:52:18.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.854+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f46840733d0 con 0x7f468415e840 2023-11-28T21:52:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.860+0000 7f46817fa640 1 -- 172.21.15.45:0/691520523 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4670058260 con 0x7f468415e840 2023-11-28T21:52:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.863+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f466806af90 msgr2=0x7f466806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.863+0000 7f468abfb640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f466806af90 0x7f466806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4678001640 tx=0x7f4678009040 comp rx=0 tx=0).stop 2023-11-28T21:52:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.863+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415e840 msgr2=0x7f46840c5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.863+0000 7f468abfb640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415e840 0x7f46840c5e30 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f467000d800 tx=0x7f467000dcc0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.863+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 shutdown_connections 2023-11-28T21:52:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.863+0000 7f468abfb640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f466806af90 0x7f466806d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.863+0000 7f468abfb640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f468415f720 0x7f4684149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.863+0000 7f468abfb640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468415e840 0x7f46840c5e30 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.863+0000 7f468abfb640 1 --2- 172.21.15.45:0/691520523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f468406ac10 0x7f46840b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.864+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 >> 172.21.15.45:0/691520523 conn(0x7f46840ced60 msgr2=0x7f468409ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.864+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 shutdown_connections 2023-11-28T21:52:18.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:18.864+0000 7f468abfb640 1 -- 172.21.15.45:0/691520523 wait complete. 2023-11-28T21:52:18.886 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T21:52:18.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T21:52:19.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T21:52:19.130 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T21:52:19.130 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:52:19.130 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:19.190 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T21:52:19.190 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T21:52:19.191 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:19.250 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T21:52:19.250 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:52:19.251 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:19.314 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T21:52:19.314 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T21:52:19.314 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:19.372 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T21:52:19.372 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T21:52:19.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:52:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.610+0000 7f8fe28c3640 1 Processor -- start 2023-11-28T21:52:19.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.610+0000 7f8fe28c3640 1 -- start start 2023-11-28T21:52:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.612+0000 7f8fe28c3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8fdc0b24a0 0x7f8fdc0703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.612+0000 7f8fe28c3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8fdc146610 0x7f8fdc070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.612+0000 7f8fe28c3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fdc0b4310 con 0x7f8fdc072200 2023-11-28T21:52:19.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.612+0000 7f8fe28c3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fdc0725e0 con 0x7f8fdc0b24a0 2023-11-28T21:52:19.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.612+0000 7f8fe28c3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fdc06d8d0 con 0x7f8fdc146610 2023-11-28T21:52:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.612+0000 7f8fe10c0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8fdc0b24a0 0x7f8fdc0703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.613+0000 7f8fe10c0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8fdc0b24a0 0x7f8fdc0703d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53640/0 (socket says 172.21.15.45:53640) 2023-11-28T21:52:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.613+0000 7f8fe10c0640 1 -- 172.21.15.45:0/3773614129 learned_addr learned my addr 172.21.15.45:0/3773614129 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.613+0000 7f8fe20c2640 1 --2- 172.21.15.45:0/3773614129 >> v2:172.21.15.190:3300/0 conn(0x7f8fdc146610 0x7f8fdc070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.613+0000 7f8fe10c0640 1 -- 172.21.15.45:0/3773614129 >> v2:172.21.15.190:3300/0 conn(0x7f8fdc146610 msgr2=0x7f8fdc070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.613+0000 7f8fe10c0640 1 --2- 172.21.15.45:0/3773614129 >> v2:172.21.15.190:3300/0 conn(0x7f8fdc146610 0x7f8fdc070910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.613+0000 7f8fe10c0640 1 -- 172.21.15.45:0/3773614129 >> v1:172.21.15.190:6789/0 conn(0x7f8fdc072200 legacy=0x7f8fdc0b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.613+0000 7f8fe10c0640 1 -- 172.21.15.45:0/3773614129 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc06ca20 con 0x7f8fdc0b24a0 2023-11-28T21:52:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.615+0000 7f8fe10c0640 1 --2- 172.21.15.45:0/3773614129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc0b24a0 0x7f8fdc0703d0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc06b360 tx=0x7f8fcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba196e974ef44a60 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.615+0000 7f8fe08bf640 1 -- 172.21.15.45:0/3773614129 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc031870 con 0x7f8fdc0b24a0 2023-11-28T21:52:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.615+0000 7f8fe08bf640 1 -- 172.21.15.45:0/3773614129 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc0319d0 con 0x7f8fdc0b24a0 2023-11-28T21:52:19.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.615+0000 7f8fe08bf640 1 -- 172.21.15.45:0/3773614129 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc031d60 con 0x7f8fdc0b24a0 2023-11-28T21:52:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.616+0000 7f8fe28c3640 1 -- 172.21.15.45:0/3773614129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc0b24a0 msgr2=0x7f8fdc0703d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.616+0000 7f8fe28c3640 1 --2- 172.21.15.45:0/3773614129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc0b24a0 0x7f8fdc0703d0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc06b360 tx=0x7f8fcc02b910 comp rx=0 tx=0).stop 2023-11-28T21:52:19.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.616+0000 7f8fe28c3640 1 -- 172.21.15.45:0/3773614129 shutdown_connections 2023-11-28T21:52:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.616+0000 7f8fe28c3640 1 --2- 172.21.15.45:0/3773614129 >> v2:172.21.15.190:3300/0 conn(0x7f8fdc146610 0x7f8fdc070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.616+0000 7f8fe28c3640 1 --2- 172.21.15.45:0/3773614129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc0b24a0 0x7f8fdc0703d0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.616+0000 7f8fe28c3640 1 -- 172.21.15.45:0/3773614129 >> 172.21.15.45:0/3773614129 conn(0x7f8fdc06ab30 msgr2=0x7f8fdc0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.616+0000 7f8fe28c3640 1 -- 172.21.15.45:0/3773614129 shutdown_connections 2023-11-28T21:52:19.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.616+0000 7f8fe28c3640 1 -- 172.21.15.45:0/3773614129 wait complete. 2023-11-28T21:52:19.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.618+0000 7f8fe28c3640 1 Processor -- start 2023-11-28T21:52:19.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.618+0000 7f8fe28c3640 1 -- start start 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.619+0000 7f8fe28c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc072200 0x7f8fdc136300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.619+0000 7f8fe28c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fdc0b24a0 0x7f8fdc15e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.619+0000 7f8fe28c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fdc146610 0x7f8fdc161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.619+0000 7f8fe28c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc1347a0 con 0x7f8fdc072200 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.619+0000 7f8fe28c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc136840 con 0x7f8fdc146610 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.619+0000 7f8fe28c3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc133b90 con 0x7f8fdc0b24a0 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.619+0000 7f8fe18c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc072200 0x7f8fdc136300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.619+0000 7f8fe18c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc072200 0x7f8fdc136300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53656/0 (socket says 172.21.15.45:53656) 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.619+0000 7f8fe18c1640 1 -- 172.21.15.45:0/963564859 learned_addr learned my addr 172.21.15.45:0/963564859 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fe18c1640 1 -- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fdc0b24a0 msgr2=0x7f8fdc15e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:19.625 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fe18c1640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fdc0b24a0 0x7f8fdc15e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fe18c1640 1 -- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fdc146610 msgr2=0x7f8fdc161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fe18c1640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fdc146610 0x7f8fdc161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fe18c1640 1 -- 172.21.15.45:0/963564859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0094d0 con 0x7f8fdc072200 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fe18c1640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc072200 0x7f8fdc136300 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800c460 tx=0x7f8fd800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fd27fc640 1 -- 172.21.15.45:0/963564859 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8012830 con 0x7f8fdc072200 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fd27fc640 1 -- 172.21.15.45:0/963564859 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd80129d0 con 0x7f8fdc072200 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fd27fc640 1 -- 172.21.15.45:0/963564859 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8012d80 con 0x7f8fdc072200 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fdc133e70 con 0x7f8fdc072200 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.620+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fdc134430 con 0x7f8fdc072200 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.622+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fdc09d0f0 con 0x7f8fdc072200 2023-11-28T21:52:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.622+0000 7f8fd27fc640 1 -- 172.21.15.45:0/963564859 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8fd801a3b0 con 0x7f8fdc072200 2023-11-28T21:52:19.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.623+0000 7f8fd27fc640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8fc006af60 0x7f8fc006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:19.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.624+0000 7f8fd27fc640 1 -- 172.21.15.45:0/963564859 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f8fd8089540 con 0x7f8fdc072200 2023-11-28T21:52:19.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.624+0000 7f8fe10c0640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8fc006af60 0x7f8fc006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:19.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.625+0000 7f8fe10c0640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8fc006af60 0x7f8fc006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc036000 tx=0x7f8fcc030000 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:19.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.628+0000 7f8fd27fc640 1 -- 172.21.15.45:0/963564859 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8fd80543c0 con 0x7f8fdc072200 2023-11-28T21:52:19.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.860+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fdc04e920 con 0x7f8fdc072200 2023-11-28T21:52:19.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.860+0000 7f8fd27fc640 1 -- 172.21.15.45:0/963564859 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+3697 (secure 0 0 0) 0x7f8fd80579e0 con 0x7f8fdc072200 2023-11-28T21:52:19.864 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:19.864 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":86,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7130,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/500262407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":500262407},{"type":"v1","addr":"172.21.15.190:6835","nonce":500262407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/195287870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":195287870},{"type":"v1","addr":"172.21.15.190:6837","nonce":195287870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":9657,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/440031651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":440031651},{"type":"v1","addr":"172.21.15.45:6833","nonce":440031651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":10029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2929688603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":2929688603},{"type":"v1","addr":"172.21.15.45:6835","nonce":2929688603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[]} 2023-11-28T21:52:19.865 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 86 2023-11-28T21:52:19.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8fc006af60 msgr2=0x7f8fc006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:19.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8fc006af60 0x7f8fc006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc036000 tx=0x7f8fcc030000 comp rx=0 tx=0).stop 2023-11-28T21:52:19.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc072200 msgr2=0x7f8fdc136300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:19.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc072200 0x7f8fdc136300 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800c460 tx=0x7f8fd800c920 comp rx=0 tx=0).stop 2023-11-28T21:52:19.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 shutdown_connections 2023-11-28T21:52:19.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f8fc006af60 0x7f8fc006d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:19.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fdc146610 0x7f8fdc161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:19.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fdc0b24a0 0x7f8fdc15e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:19.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 --2- 172.21.15.45:0/963564859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fdc072200 0x7f8fdc136300 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:19.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.863+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 >> 172.21.15.45:0/963564859 conn(0x7f8fdc06ab30 msgr2=0x7f8fdc0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:19.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.864+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 shutdown_connections 2023-11-28T21:52:19.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:19.864+0000 7f8fe28c3640 1 -- 172.21.15.45:0/963564859 wait complete. 2023-11-28T21:52:19.884 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T21:52:19.884 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T21:52:19.885 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T21:52:19.885 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T21:52:19.885 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T21:52:19.885 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T21:52:19.886 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T21:52:19.886 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T21:52:19.886 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T21:52:19.886 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T21:52:19.886 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T21:52:19.887 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T21:52:19.887 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T21:52:19.887 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T21:52:19.887 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T21:52:19.888 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T21:52:19.903 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T21:52:19.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T21:52:19.905 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T21:52:19.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T21:52:19.906 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T21:52:19.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T21:52:19.919 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T21:52:19.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T21:52:19.954 INFO:tasks.ceph.mds.c:Started 2023-11-28T21:52:19.954 DEBUG:teuthology.parallel:result is None 2023-11-28T21:52:19.962 INFO:tasks.ceph.mds.a:Started 2023-11-28T21:52:19.962 DEBUG:teuthology.parallel:result is None 2023-11-28T21:52:19.968 INFO:tasks.ceph.mds.b:Started 2023-11-28T21:52:19.968 DEBUG:teuthology.parallel:result is None 2023-11-28T21:52:19.975 INFO:tasks.ceph.mds.d:Started 2023-11-28T21:52:19.976 DEBUG:teuthology.parallel:result is None 2023-11-28T21:52:19.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T21:52:20.118 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T21:52:20.119 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T21:52:20.130 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T21:52:20.135 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T21:52:20.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.169+0000 7fac63a48640 1 Processor -- start 2023-11-28T21:52:20.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.170+0000 7fac63a48640 1 -- start start 2023-11-28T21:52:20.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac63a48640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fac5c15b820 0x7fac5c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:20.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac63a48640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fac5c0cf020 0x7fac5c0cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:20.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac63a48640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c15fa80 con 0x7fac5c15edc0 2023-11-28T21:52:20.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac63a48640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c15fd60 con 0x7fac5c15b820 2023-11-28T21:52:20.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac63a48640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c0cf9f0 con 0x7fac5c0cf020 2023-11-28T21:52:20.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac617e4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fac5c15b820 0x7fac5c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:20.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac61fe5640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fac5c15edc0 0x7fac5c15f1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42438/0 (socket says 172.21.15.45:42438) 2023-11-28T21:52:20.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac61fe5640 1 -- 172.21.15.45:0/833174566 learned_addr learned my addr 172.21.15.45:0/833174566 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:20.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac617e4640 1 --2- 172.21.15.45:0/833174566 >> v2:172.21.15.45:3300/0 conn(0x7fac5c15b820 0x7fac5c06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53684/0 (socket says 172.21.15.45:53684) 2023-11-28T21:52:20.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.171+0000 7fac60fe3640 1 --2- 172.21.15.45:0/833174566 >> v2:172.21.15.190:3300/0 conn(0x7fac5c0cf020 0x7fac5c0cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:20.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.172+0000 7fac617e4640 1 -- 172.21.15.45:0/833174566 >> v2:172.21.15.190:3300/0 conn(0x7fac5c0cf020 msgr2=0x7fac5c0cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:20.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.172+0000 7fac617e4640 1 --2- 172.21.15.45:0/833174566 >> v2:172.21.15.190:3300/0 conn(0x7fac5c0cf020 0x7fac5c0cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:20.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.172+0000 7fac617e4640 1 -- 172.21.15.45:0/833174566 >> v1:172.21.15.45:6789/0 conn(0x7fac5c15edc0 legacy=0x7fac5c15f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:20.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.172+0000 7fac617e4640 1 -- 172.21.15.45:0/833174566 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c06cf40 con 0x7fac5c15b820 2023-11-28T21:52:20.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.172+0000 7fac53fff640 1 -- 172.21.15.45:0/833174566 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925691945 0 0) 0x7fac5c15fa80 con 0x7fac5c15edc0 2023-11-28T21:52:20.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.173+0000 7fac617e4640 1 --2- 172.21.15.45:0/833174566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 0x7fac5c06da20 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0cfec0 tx=0x7fac5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea469359d5f8a4b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:20.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.173+0000 7fac53fff640 1 -- 172.21.15.45:0/833174566 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58031830 con 0x7fac5c15b820 2023-11-28T21:52:20.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.173+0000 7fac53fff640 1 -- 172.21.15.45:0/833174566 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac58031990 con 0x7fac5c15b820 2023-11-28T21:52:20.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.173+0000 7fac53fff640 1 -- 172.21.15.45:0/833174566 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58031d20 con 0x7fac5c15b820 2023-11-28T21:52:20.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.174+0000 7fac63a48640 1 -- 172.21.15.45:0/833174566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 msgr2=0x7fac5c06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:20.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.174+0000 7fac63a48640 1 --2- 172.21.15.45:0/833174566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 0x7fac5c06da20 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0cfec0 tx=0x7fac5802b8d0 comp rx=0 tx=0).stop 2023-11-28T21:52:20.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.174+0000 7fac63a48640 1 -- 172.21.15.45:0/833174566 shutdown_connections 2023-11-28T21:52:20.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.174+0000 7fac63a48640 1 --2- 172.21.15.45:0/833174566 >> v2:172.21.15.190:3300/0 conn(0x7fac5c0cf020 0x7fac5c0cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:20.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.174+0000 7fac63a48640 1 --2- 172.21.15.45:0/833174566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 0x7fac5c06da20 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:20.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.174+0000 7fac63a48640 1 -- 172.21.15.45:0/833174566 >> 172.21.15.45:0/833174566 conn(0x7fac5c06a6e0 msgr2=0x7fac5c06aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:20.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.174+0000 7fac63a48640 1 -- 172.21.15.45:0/833174566 shutdown_connections 2023-11-28T21:52:20.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.174+0000 7fac63a48640 1 -- 172.21.15.45:0/833174566 wait complete. 2023-11-28T21:52:20.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.176+0000 7fac63a48640 1 Processor -- start 2023-11-28T21:52:20.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.176+0000 7fac63a48640 1 -- start start 2023-11-28T21:52:20.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac63a48640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac5c0cf020 0x7fac5c14fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:20.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac63a48640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 0x7fac5c0b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:20.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac63a48640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac5c15edc0 0x7fac5c0b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:20.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac63a48640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c1f64e0 con 0x7fac5c15b820 2023-11-28T21:52:20.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac63a48640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c09c590 con 0x7fac5c0cf020 2023-11-28T21:52:20.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac63a48640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0b3c40 con 0x7fac5c15edc0 2023-11-28T21:52:20.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac60fe3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 0x7fac5c0b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:20.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac60fe3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 0x7fac5c0b30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53698/0 (socket says 172.21.15.45:53698) 2023-11-28T21:52:20.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac60fe3640 1 -- 172.21.15.45:0/1784738774 learned_addr learned my addr 172.21.15.45:0/1784738774 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:20.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac617e4640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac5c0cf020 0x7fac5c14fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:20.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.177+0000 7fac61fe5640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac5c15edc0 0x7fac5c0b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:20.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac60fe3640 1 -- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac5c15edc0 msgr2=0x7fac5c0b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:20.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac60fe3640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac5c15edc0 0x7fac5c0b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:20.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac60fe3640 1 -- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac5c0cf020 msgr2=0x7fac5c14fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:20.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac60fe3640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac5c0cf020 0x7fac5c14fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:20.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac60fe3640 1 -- 172.21.15.45:0/1784738774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58009400 con 0x7fac5c15b820 2023-11-28T21:52:20.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac60fe3640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 0x7fac5c0b30d0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00e9b0 tx=0x7fac4c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:20.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac51ffb640 1 -- 172.21.15.45:0/1784738774 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c009670 con 0x7fac5c15b820 2023-11-28T21:52:20.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac51ffb640 1 -- 172.21.15.45:0/1784738774 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c00c9a0 con 0x7fac5c15b820 2023-11-28T21:52:20.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac51ffb640 1 -- 172.21.15.45:0/1784738774 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00cd50 con 0x7fac5c15b820 2023-11-28T21:52:20.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c1f6730 con 0x7fac5c15b820 2023-11-28T21:52:20.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.178+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c1f6cf0 con 0x7fac5c15b820 2023-11-28T21:52:20.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.179+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c073440 con 0x7fac5c15b820 2023-11-28T21:52:20.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.180+0000 7fac51ffb640 1 -- 172.21.15.45:0/1784738774 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fac4c013470 con 0x7fac5c15b820 2023-11-28T21:52:20.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.181+0000 7fac51ffb640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fac4006af60 0x7fac4006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:20.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.181+0000 7fac51ffb640 1 -- 172.21.15.45:0/1784738774 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fac4c089d50 con 0x7fac5c15b820 2023-11-28T21:52:20.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.182+0000 7fac617e4640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fac4006af60 0x7fac4006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:20.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.182+0000 7fac617e4640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fac4006af60 0x7fac4006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac58036000 tx=0x7fac5803c040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:20.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.186+0000 7fac51ffb640 1 -- 172.21.15.45:0/1784738774 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac4c08d030 con 0x7fac5c15b820 2023-11-28T21:52:20.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:20.393+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fac5c06c8c0 con 0x7fac5c15b820 2023-11-28T21:52:21.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.409+0000 7fac51ffb640 1 -- 172.21.15.45:0/1784738774 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v195) v1 ==== 97+0+0 (secure 0 0 0) 0x7fac4c054d20 con 0x7fac5c15b820 2023-11-28T21:52:21.412 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T21:52:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fac4006af60 msgr2=0x7fac4006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:21.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fac4006af60 0x7fac4006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac58036000 tx=0x7fac5803c040 comp rx=0 tx=0).stop 2023-11-28T21:52:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 msgr2=0x7fac5c0b30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 0x7fac5c0b30d0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00e9b0 tx=0x7fac4c00ee70 comp rx=0 tx=0).stop 2023-11-28T21:52:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 shutdown_connections 2023-11-28T21:52:21.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fac4006af60 0x7fac4006d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac5c15edc0 0x7fac5c0b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac5c15b820 0x7fac5c0b30d0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 --2- 172.21.15.45:0/1784738774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac5c0cf020 0x7fac5c14fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.412+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 >> 172.21.15.45:0/1784738774 conn(0x7fac5c06a6e0 msgr2=0x7fac5c0ccee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:21.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.413+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 shutdown_connections 2023-11-28T21:52:21.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.413+0000 7fac63a48640 1 -- 172.21.15.45:0/1784738774 wait complete. 2023-11-28T21:52:21.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T21:52:21.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.657+0000 7f35b3402640 1 Processor -- start 2023-11-28T21:52:21.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.657+0000 7f35b3402640 1 -- start start 2023-11-28T21:52:21.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.659+0000 7f35b3402640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f35ac141600 0x7f35ac0cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:21.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.659+0000 7f35b3402640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f35ac0cc0d0 0x7f35ac15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:21.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.659+0000 7f35b3402640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 0x7f35ac15ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:21.662 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.659+0000 7f35b3402640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac0cc580 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.659+0000 7f35b3402640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac15c670 con 0x7f35ac141600 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.659+0000 7f35b3402640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac0cda40 con 0x7f35ac15be80 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b1bff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f35ac0cc0d0 0x7f35ac15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b1bff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f35ac0cc0d0 0x7f35ac15b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53702/0 (socket says 172.21.15.45:53702) 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b1bff640 1 -- 172.21.15.45:0/2247406380 learned_addr learned my addr 172.21.15.45:0/2247406380 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b2400640 1 --2- 172.21.15.45:0/2247406380 >> v2:172.21.15.190:3300/0 conn(0x7f35ac141600 0x7f35ac0cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b2c01640 1 --2- 172.21.15.45:0/2247406380 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 0x7f35ac15ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b1bff640 1 -- 172.21.15.45:0/2247406380 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 msgr2=0x7f35ac15ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b1bff640 1 --2- 172.21.15.45:0/2247406380 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 0x7f35ac15ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b1bff640 1 -- 172.21.15.45:0/2247406380 >> v2:172.21.15.190:3300/0 conn(0x7f35ac141600 msgr2=0x7f35ac0cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b1bff640 1 --2- 172.21.15.45:0/2247406380 >> v2:172.21.15.190:3300/0 conn(0x7f35ac141600 0x7f35ac0cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.663 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.660+0000 7f35b1bff640 1 -- 172.21.15.45:0/2247406380 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac0ccc80 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.662+0000 7f35b1bff640 1 --2- 172.21.15.45:0/2247406380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 0x7f35ac15b940 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0cc820 tx=0x7f359c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67bd89627a18d0a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:21.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.662+0000 7f35b13fe640 1 -- 172.21.15.45:0/2247406380 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c031870 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.662+0000 7f35b13fe640 1 -- 172.21.15.45:0/2247406380 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359c0319d0 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.662+0000 7f35b13fe640 1 -- 172.21.15.45:0/2247406380 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c031d60 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.663+0000 7f35b3402640 1 -- 172.21.15.45:0/2247406380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 msgr2=0x7f35ac15b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:21.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.663+0000 7f35b3402640 1 --2- 172.21.15.45:0/2247406380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 0x7f35ac15b940 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0cc820 tx=0x7f359c02b910 comp rx=0 tx=0).stop 2023-11-28T21:52:21.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.663+0000 7f35b3402640 1 -- 172.21.15.45:0/2247406380 shutdown_connections 2023-11-28T21:52:21.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.663+0000 7f35b3402640 1 --2- 172.21.15.45:0/2247406380 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 0x7f35ac15ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.663+0000 7f35b3402640 1 --2- 172.21.15.45:0/2247406380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 0x7f35ac15b940 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.663+0000 7f35b3402640 1 --2- 172.21.15.45:0/2247406380 >> v2:172.21.15.190:3300/0 conn(0x7f35ac141600 0x7f35ac0cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.663+0000 7f35b3402640 1 -- 172.21.15.45:0/2247406380 >> 172.21.15.45:0/2247406380 conn(0x7f35ac0b0e70 msgr2=0x7f35ac0b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:21.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.663+0000 7f35b3402640 1 -- 172.21.15.45:0/2247406380 shutdown_connections 2023-11-28T21:52:21.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.663+0000 7f35b3402640 1 -- 172.21.15.45:0/2247406380 wait complete. 2023-11-28T21:52:21.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.665+0000 7f35b3402640 1 Processor -- start 2023-11-28T21:52:21.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.665+0000 7f35b3402640 1 -- start start 2023-11-28T21:52:21.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.666+0000 7f35b3402640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 0x7f35ac15a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:21.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.666+0000 7f35b3402640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35ac141600 0x7f35ac1e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:21.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.666+0000 7f35b3402640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 0x7f35ac13cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:21.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.666+0000 7f35b3402640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac1e4ad0 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.666+0000 7f35b3402640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac1e4db0 con 0x7f35ac141600 2023-11-28T21:52:21.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.666+0000 7f35b3402640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac1e5090 con 0x7f35ac15be80 2023-11-28T21:52:21.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.666+0000 7f35b2400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 0x7f35ac15a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:21.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.666+0000 7f35b2400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 0x7f35ac15a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53710/0 (socket says 172.21.15.45:53710) 2023-11-28T21:52:21.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.666+0000 7f35b2400640 1 -- 172.21.15.45:0/635172533 learned_addr learned my addr 172.21.15.45:0/635172533 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:21.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35b2c01640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 0x7f35ac13cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:21.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35b1bff640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35ac141600 0x7f35ac1e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:21.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35b2400640 1 -- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 msgr2=0x7f35ac13cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:21.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35b2400640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 0x7f35ac13cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35b2400640 1 -- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35ac141600 msgr2=0x7f35ac1e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35b2400640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35ac141600 0x7f35ac1e5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35b2400640 1 -- 172.21.15.45:0/635172533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c0094d0 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35b2c01640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 0x7f35ac13cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35b2400640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 0x7f35ac15a500 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f35a800d800 tx=0x7f35a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.667+0000 7f35a2ffd640 1 -- 172.21.15.45:0/635172533 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a80096a0 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.668+0000 7f35a2ffd640 1 -- 172.21.15.45:0/635172533 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8009800 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.668+0000 7f35a2ffd640 1 -- 172.21.15.45:0/635172533 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8009b90 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.668+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ac1e5370 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.668+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ac1e5930 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.669+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ac09c500 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.670+0000 7f35a2ffd640 1 -- 172.21.15.45:0/635172533 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f35a8009cf0 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.671+0000 7f35a2ffd640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f359006af60 0x7f359006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:21.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.671+0000 7f35a2ffd640 1 -- 172.21.15.45:0/635172533 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f35a8089a30 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.672+0000 7f35b1bff640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f359006af60 0x7f359006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.673+0000 7f35b1bff640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f359006af60 0x7f359006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f359c036000 tx=0x7f359c03c040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.676+0000 7f35a2ffd640 1 -- 172.21.15.45:0/635172533 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35a8054bb0 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.940+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f35ac1e7b60 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.942+0000 7f35a2ffd640 1 -- 172.21.15.45:0/635172533 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f35a800b440 con 0x7f35ac0cc0d0 2023-11-28T21:52:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:21.944 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T21:52:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T21:52:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T21:52:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T21:52:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:52:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:52:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:52:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T21:52:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T21:52:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:52:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:52:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:52:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T21:52:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:52:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T21:52:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:52:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:52:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T21:52:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T21:52:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T21:52:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T21:52:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T21:52:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T21:52:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T21:52:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T21:52:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T21:52:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T21:52:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T21:52:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T21:52:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T21:52:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T21:52:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T21:52:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T21:52:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T21:52:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T21:52:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T21:52:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T21:52:21.962 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T21:52:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f359006af60 msgr2=0x7f359006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f359006af60 0x7f359006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f359c036000 tx=0x7f359c03c040 comp rx=0 tx=0).stop 2023-11-28T21:52:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 msgr2=0x7f35ac15a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 0x7f35ac15a500 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f35a800d800 tx=0x7f35a800dcc0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 shutdown_connections 2023-11-28T21:52:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f359006af60 0x7f359006d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35ac15be80 0x7f35ac13cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35ac141600 0x7f35ac1e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 --2- 172.21.15.45:0/635172533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35ac0cc0d0 0x7f35ac15a500 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:21.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 >> 172.21.15.45:0/635172533 conn(0x7f35ac0b0e70 msgr2=0x7f35ac0b1250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:21.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 shutdown_connections 2023-11-28T21:52:21.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:21.944+0000 7f35b3402640 1 -- 172.21.15.45:0/635172533 wait complete. 2023-11-28T21:52:21.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:52:22.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.190+0000 7efe7de3d640 1 Processor -- start 2023-11-28T21:52:22.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.190+0000 7efe7de3d640 1 -- start start 2023-11-28T21:52:22.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.192+0000 7efe7de3d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780b1340 0x7efe7815f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.192+0000 7efe7de3d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7815bda0 con 0x7efe780b35b0 2023-11-28T21:52:22.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.192+0000 7efe7de3d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7815c080 con 0x7efe7815b850 2023-11-28T21:52:22.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.192+0000 7efe7de3d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7815ee40 con 0x7efe780b1340 2023-11-28T21:52:22.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.192+0000 7efe777fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7efe780b35b0 0x7efe7815ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42440/0 (socket says 172.21.15.45:42440) 2023-11-28T21:52:22.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.192+0000 7efe777fe640 1 -- 172.21.15.45:0/507864873 learned_addr learned my addr 172.21.15.45:0/507864873 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:22.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.192+0000 7efe77fff640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780b1340 0x7efe7815f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117195514 0 0) 0x7efe7815bda0 con 0x7efe780b35b0 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe64003520 con 0x7efe780b35b0 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681961871 0 0) 0x7efe7815c080 con 0x7efe7815b850 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe7815bda0 con 0x7efe7815b850 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 524328852 0 0) 0x7efe64003520 con 0x7efe780b35b0 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780b1340 msgr2=0x7efe7815f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780b1340 0x7efe7815f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 >> v1:172.21.15.190:6789/0 conn(0x7efe7815b850 legacy=0x7efe780b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7815c360 con 0x7efe780b35b0 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7efe68001670 con 0x7efe780b35b0 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 >> v1:172.21.15.45:6789/0 conn(0x7efe780b35b0 legacy=0x7efe7815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe64003b30 0x7efe64003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780b1340 0x7efe640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe64004d00 0x7efe64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7815ee40 con 0x7efe64003b30 2023-11-28T21:52:22.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6400bac0 con 0x7efe64004d00 2023-11-28T21:52:22.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.193+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6400bc80 con 0x7efe780b1340 2023-11-28T21:52:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.194+0000 7efe777fe640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe64003b30 0x7efe64003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.194+0000 7efe77fff640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe64004d00 0x7efe64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.194+0000 7efe6ffff640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780b1340 0x7efe640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.194+0000 7efe777fe640 1 -- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780b1340 msgr2=0x7efe640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.194+0000 7efe777fe640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780b1340 0x7efe640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.194+0000 7efe777fe640 1 -- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe64004d00 msgr2=0x7efe64005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.194+0000 7efe777fe640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe64004d00 0x7efe64005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.194+0000 7efe777fe640 1 -- 172.21.15.45:0/507864873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe64003520 con 0x7efe64003b30 2023-11-28T21:52:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.196+0000 7efe777fe640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe64003b30 0x7efe64003f30 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7efe68007f90 tx=0x7efe640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e63a63366db1845 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.196+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe68005ad0 con 0x7efe64003b30 2023-11-28T21:52:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.196+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe68005c30 con 0x7efe64003b30 2023-11-28T21:52:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.196+0000 7efe76ffd640 1 -- 172.21.15.45:0/507864873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe68007070 con 0x7efe64003b30 2023-11-28T21:52:22.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.197+0000 7efe7de3d640 1 -- 172.21.15.45:0/507864873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe64003b30 msgr2=0x7efe64003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.197+0000 7efe7de3d640 1 --2- 172.21.15.45:0/507864873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe64003b30 0x7efe64003f30 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7efe68007f90 tx=0x7efe640057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.197+0000 7efe77fff640 1 -- 172.21.15.45:0/507864873 reap_dead start 2023-11-28T21:52:22.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.197+0000 7efe7de3d640 1 -- 172.21.15.45:0/507864873 shutdown_connections 2023-11-28T21:52:22.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.197+0000 7efe7de3d640 1 -- 172.21.15.45:0/507864873 >> 172.21.15.45:0/507864873 conn(0x7efe7806a500 msgr2=0x7efe7806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:22.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.197+0000 7efe7de3d640 1 -- 172.21.15.45:0/507864873 shutdown_connections 2023-11-28T21:52:22.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.197+0000 7efe7de3d640 1 -- 172.21.15.45:0/507864873 wait complete. 2023-11-28T21:52:22.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.199+0000 7efe7de3d640 1 Processor -- start 2023-11-28T21:52:22.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.199+0000 7efe7de3d640 1 -- start start 2023-11-28T21:52:22.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.200+0000 7efe7de3d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780cc0c0 0x7efe780a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.200+0000 7efe7de3d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe780cc4a0 0x7efe780aab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.200+0000 7efe7de3d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe781380c0 0x7efe781f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.200+0000 7efe7de3d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe780a9ad0 con 0x7efe781380c0 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.200+0000 7efe7de3d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe780a9db0 con 0x7efe780cc4a0 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.200+0000 7efe7de3d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe780aa090 con 0x7efe780cc0c0 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.200+0000 7efe77fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe781380c0 0x7efe781f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.200+0000 7efe77fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe781380c0 0x7efe781f6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53726/0 (socket says 172.21.15.45:53726) 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.200+0000 7efe77fff640 1 -- 172.21.15.45:0/2612796155 learned_addr learned my addr 172.21.15.45:0/2612796155 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe777fe640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780cc0c0 0x7efe780a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe6ffff640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe780cc4a0 0x7efe780aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe77fff640 1 -- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780cc0c0 msgr2=0x7efe780a94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe77fff640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780cc0c0 0x7efe780a94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe77fff640 1 -- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe780cc4a0 msgr2=0x7efe780aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe77fff640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe780cc4a0 0x7efe780aab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe77fff640 1 -- 172.21.15.45:0/2612796155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe780aa370 con 0x7efe781380c0 2023-11-28T21:52:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe77fff640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe781380c0 0x7efe781f6570 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7efe7806e440 tx=0x7efe700115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe74ff9640 1 -- 172.21.15.45:0/2612796155 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7000e070 con 0x7efe781380c0 2023-11-28T21:52:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe74ff9640 1 -- 172.21.15.45:0/2612796155 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe70011e10 con 0x7efe781380c0 2023-11-28T21:52:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe74ff9640 1 -- 172.21.15.45:0/2612796155 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70003990 con 0x7efe781380c0 2023-11-28T21:52:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.201+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe780aa650 con 0x7efe781380c0 2023-11-28T21:52:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.202+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe781f6ab0 con 0x7efe781380c0 2023-11-28T21:52:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.203+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe780b4a00 con 0x7efe781380c0 2023-11-28T21:52:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.204+0000 7efe74ff9640 1 -- 172.21.15.45:0/2612796155 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7efe70003af0 con 0x7efe781380c0 2023-11-28T21:52:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.205+0000 7efe74ff9640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7efe5806af60 0x7efe5806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.205+0000 7efe74ff9640 1 -- 172.21.15.45:0/2612796155 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5205+0+0 (secure 0 0 0) 0x7efe70089fc0 con 0x7efe781380c0 2023-11-28T21:52:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.206+0000 7efe777fe640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7efe5806af60 0x7efe5806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.207+0000 7efe777fe640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7efe5806af60 0x7efe5806d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe68034000 tx=0x7efe68009030 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:22.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.209+0000 7efe74ff9640 1 -- 172.21.15.45:0/2612796155 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe70054fa0 con 0x7efe781380c0 2023-11-28T21:52:22.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.408+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe780aa840 con 0x7efe781380c0 2023-11-28T21:52:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.410+0000 7efe74ff9640 1 -- 172.21.15.45:0/2612796155 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+13104 (secure 0 0 0) 0x7efe700585c0 con 0x7efe781380c0 2023-11-28T21:52:22.412 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:22.412 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":195,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:52:21.406622+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:52:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7efe5806af60 msgr2=0x7efe5806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7efe5806af60 0x7efe5806d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe68034000 tx=0x7efe68009030 comp rx=0 tx=0).stop 2023-11-28T21:52:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe781380c0 msgr2=0x7efe781f6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe781380c0 0x7efe781f6570 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7efe7806e440 tx=0x7efe700115a0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 shutdown_connections 2023-11-28T21:52:22.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7efe5806af60 0x7efe5806d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe781380c0 0x7efe781f6570 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe780cc4a0 0x7efe780aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 --2- 172.21.15.45:0/2612796155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe780cc0c0 0x7efe780a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 >> 172.21.15.45:0/2612796155 conn(0x7efe7806a500 msgr2=0x7efe780aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:22.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 shutdown_connections 2023-11-28T21:52:22.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.414+0000 7efe7de3d640 1 -- 172.21.15.45:0/2612796155 wait complete. 2023-11-28T21:52:22.438 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T21:52:22.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T21:52:22.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.667+0000 7fa9df573640 1 Processor -- start 2023-11-28T21:52:22.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.667+0000 7fa9df573640 1 -- start start 2023-11-28T21:52:22.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.669+0000 7fa9df573640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80cddf0 0x7fa9d80ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.669+0000 7fa9df573640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d80ce780 con 0x7fa9d815d230 2023-11-28T21:52:22.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.669+0000 7fa9df573640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d80d11d0 con 0x7fa9d80b2640 2023-11-28T21:52:22.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.669+0000 7fa9df573640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80b68f0 con 0x7fa9d80cddf0 2023-11-28T21:52:22.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.669+0000 7fa9de571640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa9d815d230 0x7fa9d80b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42452/0 (socket says 172.21.15.45:42452) 2023-11-28T21:52:22.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.669+0000 7fa9de571640 1 -- 172.21.15.45:0/25255593 learned_addr learned my addr 172.21.15.45:0/25255593 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:22.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9ded72640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80cddf0 0x7fa9d80ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823794179 0 0) 0x7fa9d4001110 con 0x7fa9d815d230 2023-11-28T21:52:22.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c4003520 con 0x7fa9d815d230 2023-11-28T21:52:22.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3119762769 0 0) 0x7fa9c4003520 con 0x7fa9d815d230 2023-11-28T21:52:22.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80cddf0 msgr2=0x7fa9d80ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:22.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80cddf0 0x7fa9d80ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 >> v1:172.21.15.190:6789/0 conn(0x7fa9d80b2640 legacy=0x7fa9d80b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d80d1490 con 0x7fa9d815d230 2023-11-28T21:52:22.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fa9d4001800 con 0x7fa9d815d230 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9ded72640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80cddf0 0x7fa9d80ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 >> v1:172.21.15.45:6789/0 conn(0x7fa9d815d230 legacy=0x7fa9d80b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.670+0000 7fa9dd56f640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c40039a0 0x7fa9c4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9dd56f640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40047c0 0x7fa9c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9dd56f640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80cddf0 0x7fa9c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c400bba0 con 0x7fa9c40039a0 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c400bce0 con 0x7fa9c40047c0 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c400be50 con 0x7fa9d80cddf0 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9de571640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c40039a0 0x7fa9c4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9ded72640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80cddf0 0x7fa9c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9ddd70640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40047c0 0x7fa9c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9de571640 1 -- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80cddf0 msgr2=0x7fa9c40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:22.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9de571640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80cddf0 0x7fa9c40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9de571640 1 -- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40047c0 msgr2=0x7fa9c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:22.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9de571640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40047c0 0x7fa9c4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.671+0000 7fa9de571640 1 -- 172.21.15.45:0/25255593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c4003520 con 0x7fa9c40039a0 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.673+0000 7fa9de571640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c40039a0 0x7fa9c4003dc0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4007fc0 tx=0x7fa9c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad8c2bfce53064a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.673+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4005ad0 con 0x7fa9c40039a0 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.673+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d4005c50 con 0x7fa9c40039a0 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.673+0000 7fa9dd56f640 1 -- 172.21.15.45:0/25255593 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d402ca20 con 0x7fa9c40039a0 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.673+0000 7fa9df573640 1 -- 172.21.15.45:0/25255593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c40039a0 msgr2=0x7fa9c4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.673+0000 7fa9df573640 1 --2- 172.21.15.45:0/25255593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9c40039a0 0x7fa9c4003dc0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4007fc0 tx=0x7fa9c40057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.673+0000 7fa9ded72640 1 -- 172.21.15.45:0/25255593 reap_dead start 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.674+0000 7fa9df573640 1 -- 172.21.15.45:0/25255593 shutdown_connections 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.674+0000 7fa9df573640 1 -- 172.21.15.45:0/25255593 >> 172.21.15.45:0/25255593 conn(0x7fa9d8070f40 msgr2=0x7fa9d8071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.674+0000 7fa9df573640 1 -- 172.21.15.45:0/25255593 shutdown_connections 2023-11-28T21:52:22.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.674+0000 7fa9df573640 1 -- 172.21.15.45:0/25255593 wait complete. 2023-11-28T21:52:22.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.676+0000 7fa9df573640 1 Processor -- start 2023-11-28T21:52:22.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.676+0000 7fa9df573640 1 -- start start 2023-11-28T21:52:22.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9df573640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40039a0 0x7fa9d8153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9df573640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9d81560e0 0x7fa9d8150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9df573640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80b3080 0x7fa9d80b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9df573640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80b3a40 con 0x7fa9d81560e0 2023-11-28T21:52:22.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9df573640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d815dee0 con 0x7fa9c40039a0 2023-11-28T21:52:22.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9df573640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d815f630 con 0x7fa9d80b3080 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9ddd70640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9d81560e0 0x7fa9d8150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9ddd70640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9d81560e0 0x7fa9d8150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53754/0 (socket says 172.21.15.45:53754) 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9ddd70640 1 -- 172.21.15.45:0/4144511166 learned_addr learned my addr 172.21.15.45:0/4144511166 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9ded72640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80b3080 0x7fa9d80b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9de571640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40039a0 0x7fa9d8153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9ddd70640 1 -- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80b3080 msgr2=0x7fa9d80b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9ddd70640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80b3080 0x7fa9d80b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9ddd70640 1 -- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40039a0 msgr2=0x7fa9d8153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.677+0000 7fa9ddd70640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40039a0 0x7fa9d8153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.678+0000 7fa9ddd70640 1 -- 172.21.15.45:0/4144511166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d815f300 con 0x7fa9d81560e0 2023-11-28T21:52:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.678+0000 7fa9ded72640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80b3080 0x7fa9d80b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.678+0000 7fa9de571640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40039a0 0x7fa9d8153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.678+0000 7fa9ddd70640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9d81560e0 0x7fa9d8150a70 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800e580 tx=0x7fa9c800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.678+0000 7fa9ceffd640 1 -- 172.21.15.45:0/4144511166 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c80052e0 con 0x7fa9d81560e0 2023-11-28T21:52:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.678+0000 7fa9ceffd640 1 -- 172.21.15.45:0/4144511166 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c8005440 con 0x7fa9d81560e0 2023-11-28T21:52:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.678+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d815d920 con 0x7fa9d81560e0 2023-11-28T21:52:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.678+0000 7fa9ceffd640 1 -- 172.21.15.45:0/4144511166 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8003810 con 0x7fa9d81560e0 2023-11-28T21:52:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.678+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d81f6fe0 con 0x7fa9d81560e0 2023-11-28T21:52:22.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.679+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d8071b30 con 0x7fa9d81560e0 2023-11-28T21:52:22.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.681+0000 7fa9ceffd640 1 -- 172.21.15.45:0/4144511166 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa9c8003970 con 0x7fa9d81560e0 2023-11-28T21:52:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.682+0000 7fa9ceffd640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fa9bc06af60 0x7fa9bc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.682+0000 7fa9ceffd640 1 -- 172.21.15.45:0/4144511166 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fa9c8089390 con 0x7fa9d81560e0 2023-11-28T21:52:22.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.682+0000 7fa9de571640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fa9bc06af60 0x7fa9bc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:22.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.684+0000 7fa9de571640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fa9bc06af60 0x7fa9bc06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4034000 tx=0x7fa9d4009030 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:22.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.686+0000 7fa9ceffd640 1 -- 172.21.15.45:0/4144511166 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9c8054490 con 0x7fa9d81560e0 2023-11-28T21:52:22.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.879+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9d80c9740 con 0x7fa9d81560e0 2023-11-28T21:52:22.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.880+0000 7fa9ceffd640 1 -- 172.21.15.45:0/4144511166 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v195) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9c8057ab0 con 0x7fa9d81560e0 2023-11-28T21:52:22.882 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T21:52:22.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.882+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fa9bc06af60 msgr2=0x7fa9bc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.882+0000 7fa9df573640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fa9bc06af60 0x7fa9bc06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4034000 tx=0x7fa9d4009030 comp rx=0 tx=0).stop 2023-11-28T21:52:22.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.882+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9d81560e0 msgr2=0x7fa9d8150a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:22.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.882+0000 7fa9df573640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9d81560e0 0x7fa9d8150a70 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800e580 tx=0x7fa9c800ea40 comp rx=0 tx=0).stop 2023-11-28T21:52:22.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.882+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 shutdown_connections 2023-11-28T21:52:22.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.882+0000 7fa9df573640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fa9bc06af60 0x7fa9bc06d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.882+0000 7fa9df573640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9d80b3080 0x7fa9d80b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.883+0000 7fa9df573640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9d81560e0 0x7fa9d8150a70 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.883+0000 7fa9df573640 1 --2- 172.21.15.45:0/4144511166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9c40039a0 0x7fa9d8153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:22.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.883+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 >> 172.21.15.45:0/4144511166 conn(0x7fa9d8070f40 msgr2=0x7fa9d80b6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:22.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.883+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 shutdown_connections 2023-11-28T21:52:22.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:22.883+0000 7fa9df573640 1 -- 172.21.15.45:0/4144511166 wait complete. 2023-11-28T21:52:22.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T21:52:23.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.133+0000 7ff4a0924640 1 Processor -- start 2023-11-28T21:52:23.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.133+0000 7ff4a0924640 1 -- start start 2023-11-28T21:52:23.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.135+0000 7ff4a0924640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff49c0cfed0 0x7ff49c0cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.135+0000 7ff4a0924640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c0d1250 con 0x7ff49c0ce010 2023-11-28T21:52:23.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.135+0000 7ff4a0924640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c0ce6b0 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.135+0000 7ff4a0924640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49c0b6c70 con 0x7ff49c0cfed0 2023-11-28T21:52:23.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.135+0000 7ff493fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff49c0cfed0 0x7ff49c0cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.135+0000 7ff493fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff49c0cfed0 0x7ff49c0cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53768/0 (socket says 172.21.15.45:53768) 2023-11-28T21:52:23.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.135+0000 7ff493fff640 1 -- 172.21.15.45:0/873737087 learned_addr learned my addr 172.21.15.45:0/873737087 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:23.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.135+0000 7ff493fff640 1 -- 172.21.15.45:0/873737087 >> v1:172.21.15.190:6789/0 conn(0x7ff49c0cfaf0 legacy=0x7ff49c0b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:23.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.135+0000 7ff493fff640 1 -- 172.21.15.45:0/873737087 >> v1:172.21.15.45:6789/0 conn(0x7ff49c0ce010 legacy=0x7ff49c15cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:23.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.136+0000 7ff493fff640 1 -- 172.21.15.45:0/873737087 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c0ce990 con 0x7ff49c0cfed0 2023-11-28T21:52:23.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.137+0000 7ff493fff640 1 --2- 172.21.15.45:0/873737087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfed0 0x7ff49c0cdad0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff484009870 tx=0x7ff48402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59b8ad1b702b03e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:23.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.137+0000 7ff49a59c640 1 -- 172.21.15.45:0/873737087 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff484031870 con 0x7ff49c0cfed0 2023-11-28T21:52:23.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.137+0000 7ff49a59c640 1 -- 172.21.15.45:0/873737087 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4840319d0 con 0x7ff49c0cfed0 2023-11-28T21:52:23.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.137+0000 7ff49a59c640 1 -- 172.21.15.45:0/873737087 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff484031d60 con 0x7ff49c0cfed0 2023-11-28T21:52:23.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.138+0000 7ff4a0924640 1 -- 172.21.15.45:0/873737087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfed0 msgr2=0x7ff49c0cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:23.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.138+0000 7ff4a0924640 1 --2- 172.21.15.45:0/873737087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfed0 0x7ff49c0cdad0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff484009870 tx=0x7ff48402b910 comp rx=0 tx=0).stop 2023-11-28T21:52:23.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.138+0000 7ff4a0924640 1 -- 172.21.15.45:0/873737087 shutdown_connections 2023-11-28T21:52:23.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.138+0000 7ff4a0924640 1 --2- 172.21.15.45:0/873737087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfed0 0x7ff49c0cdad0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.138+0000 7ff4a0924640 1 -- 172.21.15.45:0/873737087 >> 172.21.15.45:0/873737087 conn(0x7ff49c0b4c40 msgr2=0x7ff49c0b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:23.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.138+0000 7ff4a0924640 1 -- 172.21.15.45:0/873737087 shutdown_connections 2023-11-28T21:52:23.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.138+0000 7ff4a0924640 1 -- 172.21.15.45:0/873737087 wait complete. 2023-11-28T21:52:23.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.140+0000 7ff4a0924640 1 Processor -- start 2023-11-28T21:52:23.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.140+0000 7ff4a0924640 1 -- start start 2023-11-28T21:52:23.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff4a0924640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff49c0ce010 0x7ff49c1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff4a0924640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfaf0 0x7ff49c0c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff4a0924640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff49c0cfed0 0x7ff49c0c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff4a0924640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c0c8b90 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff4a0924640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c0c8e70 con 0x7ff49c0ce010 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff4a0924640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c0c6e00 con 0x7ff49c0cfed0 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff493fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfaf0 0x7ff49c0c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff493fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfaf0 0x7ff49c0c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53780/0 (socket says 172.21.15.45:53780) 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff493fff640 1 -- 172.21.15.45:0/1722936755 learned_addr learned my addr 172.21.15.45:0/1722936755 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff493fff640 1 -- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff49c0cfed0 msgr2=0x7ff49c0c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff493fff640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff49c0cfed0 0x7ff49c0c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff493fff640 1 -- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff49c0ce010 msgr2=0x7ff49c1582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:23.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff493fff640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff49c0ce010 0x7ff49c1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.141+0000 7ff493fff640 1 -- 172.21.15.45:0/1722936755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4840094d0 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.142+0000 7ff493fff640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfaf0 0x7ff49c0c8340 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7ff484009870 tx=0x7ff4840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:23.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.142+0000 7ff4937fe640 1 -- 172.21.15.45:0/1722936755 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff484031820 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.142+0000 7ff4937fe640 1 -- 172.21.15.45:0/1722936755 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4840443f0 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.142+0000 7ff4937fe640 1 -- 172.21.15.45:0/1722936755 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4840447a0 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.142+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49c0c70e0 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.142+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c0c8880 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.143+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c0c6c60 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.144+0000 7ff4937fe640 1 -- 172.21.15.45:0/1722936755 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff48404d450 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.145+0000 7ff4937fe640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7ff47c06af60 0x7ff47c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.145+0000 7ff4937fe640 1 -- 172.21.15.45:0/1722936755 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5205+0+0 (secure 0 0 0) 0x7ff48402d040 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.146+0000 7ff49ad9d640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7ff47c06af60 0x7ff47c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.147+0000 7ff49ad9d640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7ff47c06af60 0x7ff47c06d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff48c003cc0 tx=0x7ff48c0079b0 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:23.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.150+0000 7ff4937fe640 1 -- 172.21.15.45:0/1722936755 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff49c0c6c60 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.351+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff49c0c7600 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.352+0000 7ff4937fe640 1 -- 172.21.15.45:0/1722936755 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v195) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff49c0c7600 con 0x7ff49c0cfaf0 2023-11-28T21:52:23.354 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:23.354 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T21:52:23.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:23.354 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T21:52:23.355 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T21:52:23.355 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:23.355 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T21:52:23.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.354+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7ff47c06af60 msgr2=0x7ff47c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:23.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.354+0000 7ff4a0924640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7ff47c06af60 0x7ff47c06d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff48c003cc0 tx=0x7ff48c0079b0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.354+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfaf0 msgr2=0x7ff49c0c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:23.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.354+0000 7ff4a0924640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfaf0 0x7ff49c0c8340 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7ff484009870 tx=0x7ff4840046c0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.355+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 shutdown_connections 2023-11-28T21:52:23.357 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.355+0000 7ff4a0924640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7ff47c06af60 0x7ff47c06d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.355+0000 7ff4a0924640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff49c0cfed0 0x7ff49c0c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.355+0000 7ff4a0924640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49c0cfaf0 0x7ff49c0c8340 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.355+0000 7ff4a0924640 1 --2- 172.21.15.45:0/1722936755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff49c0ce010 0x7ff49c1582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.355+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 >> 172.21.15.45:0/1722936755 conn(0x7ff49c0b4c40 msgr2=0x7ff49c0b6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.355+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 shutdown_connections 2023-11-28T21:52:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.355+0000 7ff4a0924640 1 -- 172.21.15.45:0/1722936755 wait complete. 2023-11-28T21:52:23.378 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T21:52:23.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T21:52:23.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.603+0000 7fd34ec4f640 1 Processor -- start 2023-11-28T21:52:23.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.604+0000 7fd34ec4f640 1 -- start start 2023-11-28T21:52:23.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.605+0000 7fd34ec4f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd34815d3c0 0x7fd3480cda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.605+0000 7fd34ec4f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd34815d7a0 0x7fd3480b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.605+0000 7fd34ec4f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd348161090 0x7fd3480cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.605+0000 7fd34ec4f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3480b6920 con 0x7fd34815d3c0 2023-11-28T21:52:23.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.605+0000 7fd34ec4f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3480d2e70 con 0x7fd34815d7a0 2023-11-28T21:52:23.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.605+0000 7fd34ec4f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480ce110 con 0x7fd348161090 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd34c9eb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd34815d3c0 0x7fd3480cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd34c9eb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd34815d3c0 0x7fd3480cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53786/0 (socket says 172.21.15.45:53786) 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd34c9eb640 1 -- 172.21.15.45:0/2452676069 learned_addr learned my addr 172.21.15.45:0/2452676069 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd347fff640 1 --2- 172.21.15.45:0/2452676069 >> v2:172.21.15.190:3300/0 conn(0x7fd34815d7a0 0x7fd3480b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd34d1ec640 1 --2- 172.21.15.45:0/2452676069 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd348161090 0x7fd3480cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd34c9eb640 1 -- 172.21.15.45:0/2452676069 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd348161090 msgr2=0x7fd3480cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd34c9eb640 1 --2- 172.21.15.45:0/2452676069 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd348161090 0x7fd3480cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd34c9eb640 1 -- 172.21.15.45:0/2452676069 >> v2:172.21.15.190:3300/0 conn(0x7fd34815d7a0 msgr2=0x7fd3480b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd34c9eb640 1 --2- 172.21.15.45:0/2452676069 >> v2:172.21.15.190:3300/0 conn(0x7fd34815d7a0 0x7fd3480b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.606+0000 7fd34c9eb640 1 -- 172.21.15.45:0/2452676069 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480ce3f0 con 0x7fd34815d3c0 2023-11-28T21:52:23.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.607+0000 7fd34c9eb640 1 --2- 172.21.15.45:0/2452676069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 0x7fd3480cda10 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fd3480b61b0 tx=0x7fd33802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2edf7cf1e9be746 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:23.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.608+0000 7fd3477fe640 1 -- 172.21.15.45:0/2452676069 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33802c9e0 con 0x7fd34815d3c0 2023-11-28T21:52:23.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.608+0000 7fd3477fe640 1 -- 172.21.15.45:0/2452676069 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33802cb40 con 0x7fd34815d3c0 2023-11-28T21:52:23.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.608+0000 7fd3477fe640 1 -- 172.21.15.45:0/2452676069 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3380343f0 con 0x7fd34815d3c0 2023-11-28T21:52:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.608+0000 7fd34ec4f640 1 -- 172.21.15.45:0/2452676069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 msgr2=0x7fd3480cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.608+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/2452676069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 0x7fd3480cda10 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fd3480b61b0 tx=0x7fd33802be40 comp rx=0 tx=0).stop 2023-11-28T21:52:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.609+0000 7fd34ec4f640 1 -- 172.21.15.45:0/2452676069 shutdown_connections 2023-11-28T21:52:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.609+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/2452676069 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd348161090 0x7fd3480cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.609+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/2452676069 >> v2:172.21.15.190:3300/0 conn(0x7fd34815d7a0 0x7fd3480b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.609+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/2452676069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 0x7fd3480cda10 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.609+0000 7fd34ec4f640 1 -- 172.21.15.45:0/2452676069 >> 172.21.15.45:0/2452676069 conn(0x7fd3480b4ca0 msgr2=0x7fd3480b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:23.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.609+0000 7fd34ec4f640 1 -- 172.21.15.45:0/2452676069 shutdown_connections 2023-11-28T21:52:23.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.609+0000 7fd34ec4f640 1 -- 172.21.15.45:0/2452676069 wait complete. 2023-11-28T21:52:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.611+0000 7fd34ec4f640 1 Processor -- start 2023-11-28T21:52:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.611+0000 7fd34ec4f640 1 -- start start 2023-11-28T21:52:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.612+0000 7fd34ec4f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 0x7fd3481f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.612+0000 7fd34ec4f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd348161090 0x7fd34814ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.612+0000 7fd34ec4f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3481f66a0 0x7fd3481f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.612+0000 7fd34ec4f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34814f120 con 0x7fd34815d3c0 2023-11-28T21:52:23.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.612+0000 7fd34ec4f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34814e190 con 0x7fd348161090 2023-11-28T21:52:23.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.612+0000 7fd34ec4f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34814be80 con 0x7fd3481f66a0 2023-11-28T21:52:23.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.612+0000 7fd34c9eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 0x7fd3481f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.612+0000 7fd34c9eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 0x7fd3481f2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53792/0 (socket says 172.21.15.45:53792) 2023-11-28T21:52:23.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.612+0000 7fd34c9eb640 1 -- 172.21.15.45:0/1815873310 learned_addr learned my addr 172.21.15.45:0/1815873310 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.613+0000 7fd34d1ec640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3481f66a0 0x7fd3481f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.613+0000 7fd347fff640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd348161090 0x7fd34814ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.613+0000 7fd34c9eb640 1 -- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3481f66a0 msgr2=0x7fd3481f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.613+0000 7fd34c9eb640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3481f66a0 0x7fd3481f6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.613+0000 7fd34c9eb640 1 -- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd348161090 msgr2=0x7fd34814ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.613+0000 7fd34c9eb640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd348161090 0x7fd34814ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.613+0000 7fd34c9eb640 1 -- 172.21.15.45:0/1815873310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd338009b30 con 0x7fd34815d3c0 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.613+0000 7fd347fff640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd348161090 0x7fd34814ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.613+0000 7fd34c9eb640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 0x7fd3481f2ad0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fd338000f80 tx=0x7fd338004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.614+0000 7fd3457fa640 1 -- 172.21.15.45:0/1815873310 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd338009510 con 0x7fd34815d3c0 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.614+0000 7fd3457fa640 1 -- 172.21.15.45:0/1815873310 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33803e450 con 0x7fd34815d3c0 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.614+0000 7fd3457fa640 1 -- 172.21.15.45:0/1815873310 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33803e800 con 0x7fd34815d3c0 2023-11-28T21:52:23.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.614+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3481f7140 con 0x7fd34815d3c0 2023-11-28T21:52:23.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.614+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34814e3e0 con 0x7fd34815d3c0 2023-11-28T21:52:23.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.615+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3480733d0 con 0x7fd34815d3c0 2023-11-28T21:52:23.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.616+0000 7fd3457fa640 1 -- 172.21.15.45:0/1815873310 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd33803e960 con 0x7fd34815d3c0 2023-11-28T21:52:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.617+0000 7fd3457fa640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd32806af60 0x7fd32806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.618+0000 7fd3457fa640 1 -- 172.21.15.45:0/1815873310 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fd3380ac410 con 0x7fd34815d3c0 2023-11-28T21:52:23.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.618+0000 7fd347fff640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd32806af60 0x7fd32806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:23.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.619+0000 7fd347fff640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd32806af60 0x7fd32806d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd3480b36b0 tx=0x7fd330008040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:23.624 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.622+0000 7fd3457fa640 1 -- 172.21.15.45:0/1815873310 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd338077500 con 0x7fd34815d3c0 2023-11-28T21:52:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:23.817+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd3480d0ec0 con 0x7fd34815d3c0 2023-11-28T21:52:24.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.440+0000 7fd3457fa640 1 -- 172.21.15.45:0/1815873310 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v196) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd33807ab20 con 0x7fd34815d3c0 2023-11-28T21:52:24.443 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T21:52:24.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.444+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd32806af60 msgr2=0x7fd32806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:24.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.444+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd32806af60 0x7fd32806d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd3480b36b0 tx=0x7fd330008040 comp rx=0 tx=0).stop 2023-11-28T21:52:24.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 msgr2=0x7fd3481f2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:24.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 0x7fd3481f2ad0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fd338000f80 tx=0x7fd338004a40 comp rx=0 tx=0).stop 2023-11-28T21:52:24.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 shutdown_connections 2023-11-28T21:52:24.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd32806af60 0x7fd32806d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3481f66a0 0x7fd3481f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd348161090 0x7fd34814ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 --2- 172.21.15.45:0/1815873310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd34815d3c0 0x7fd3481f2ad0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 >> 172.21.15.45:0/1815873310 conn(0x7fd3480b4ca0 msgr2=0x7fd34815c340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:24.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 shutdown_connections 2023-11-28T21:52:24.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.445+0000 7fd34ec4f640 1 -- 172.21.15.45:0/1815873310 wait complete. 2023-11-28T21:52:24.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T21:52:24.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.686+0000 7f607b59e640 1 Processor -- start 2023-11-28T21:52:24.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.686+0000 7f607b59e640 1 -- start start 2023-11-28T21:52:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.687+0000 7f607b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 0x7f607c0cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.687+0000 7f607b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f607c0cdd30 0x7f607c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.687+0000 7f607b59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f607c0b1ba0 0x7f607c0cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.687+0000 7f607b59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607c0b1240 con 0x7f607c0b1ba0 2023-11-28T21:52:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.687+0000 7f607b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607c0ce7c0 con 0x7f607c0cdd30 2023-11-28T21:52:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.687+0000 7f607b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607c0ce200 con 0x7f607c15afd0 2023-11-28T21:52:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f607ad9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f607c0b1ba0 0x7f607c0cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:24.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f607ad9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f607c0b1ba0 0x7f607c0cf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50570/0 (socket says 172.21.15.45:50570) 2023-11-28T21:52:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f607ad9d640 1 -- 172.21.15.45:0/2835661085 learned_addr learned my addr 172.21.15.45:0/2835661085 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f607a59c640 1 --2- 172.21.15.45:0/2835661085 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 0x7f607c0cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f6079d9b640 1 --2- 172.21.15.45:0/2835661085 >> v2:172.21.15.190:3300/0 conn(0x7f607c0cdd30 0x7f607c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:24.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f607ad9d640 1 -- 172.21.15.45:0/2835661085 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 msgr2=0x7f607c0cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f607ad9d640 1 --2- 172.21.15.45:0/2835661085 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 0x7f607c0cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f607ad9d640 1 -- 172.21.15.45:0/2835661085 >> v2:172.21.15.190:3300/0 conn(0x7f607c0cdd30 msgr2=0x7f607c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f607ad9d640 1 --2- 172.21.15.45:0/2835661085 >> v2:172.21.15.190:3300/0 conn(0x7f607c0cdd30 0x7f607c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.688+0000 7f607ad9d640 1 -- 172.21.15.45:0/2835661085 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c0ce4e0 con 0x7f607c0b1ba0 2023-11-28T21:52:24.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.690+0000 7f607ad9d640 1 --2- 172.21.15.45:0/2835661085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0b1ba0 0x7f607c0cf4c0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f607c0cea80 tx=0x7f607402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c0e2f229e77dd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:24.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.690+0000 7f607959a640 1 -- 172.21.15.45:0/2835661085 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074031870 con 0x7f607c0b1ba0 2023-11-28T21:52:24.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.690+0000 7f607959a640 1 -- 172.21.15.45:0/2835661085 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60740319d0 con 0x7f607c0b1ba0 2023-11-28T21:52:24.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.690+0000 7f607959a640 1 -- 172.21.15.45:0/2835661085 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074031d60 con 0x7f607c0b1ba0 2023-11-28T21:52:24.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.691+0000 7f607b59e640 1 -- 172.21.15.45:0/2835661085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0b1ba0 msgr2=0x7f607c0cf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.691+0000 7f607b59e640 1 --2- 172.21.15.45:0/2835661085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0b1ba0 0x7f607c0cf4c0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f607c0cea80 tx=0x7f607402b910 comp rx=0 tx=0).stop 2023-11-28T21:52:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.691+0000 7f607b59e640 1 -- 172.21.15.45:0/2835661085 shutdown_connections 2023-11-28T21:52:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.691+0000 7f607b59e640 1 --2- 172.21.15.45:0/2835661085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0b1ba0 0x7f607c0cf4c0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.691+0000 7f607b59e640 1 --2- 172.21.15.45:0/2835661085 >> v2:172.21.15.190:3300/0 conn(0x7f607c0cdd30 0x7f607c0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.691+0000 7f607b59e640 1 --2- 172.21.15.45:0/2835661085 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 0x7f607c0cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.691+0000 7f607b59e640 1 -- 172.21.15.45:0/2835661085 >> 172.21.15.45:0/2835661085 conn(0x7f607c099a40 msgr2=0x7f607c099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.691+0000 7f607b59e640 1 -- 172.21.15.45:0/2835661085 shutdown_connections 2023-11-28T21:52:24.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.691+0000 7f607b59e640 1 -- 172.21.15.45:0/2835661085 wait complete. 2023-11-28T21:52:24.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.693+0000 7f607b59e640 1 Processor -- start 2023-11-28T21:52:24.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.693+0000 7f607b59e640 1 -- start start 2023-11-28T21:52:24.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.694+0000 7f607b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f607c0b1ba0 0x7f607c156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:24.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.694+0000 7f607b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0cdd30 0x7f607c0ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.694+0000 7f607b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 0x7f607c0b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.694+0000 7f607b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c0b5c60 con 0x7f607c0cdd30 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.694+0000 7f607b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c1f3e00 con 0x7f607c0b1ba0 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.694+0000 7f607b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607c0b5ee0 con 0x7f607c15afd0 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f6079d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0cdd30 0x7f607c0ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f6079d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0cdd30 0x7f607c0ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50586/0 (socket says 172.21.15.45:50586) 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f6079d9b640 1 -- 172.21.15.45:0/4068069873 learned_addr learned my addr 172.21.15.45:0/4068069873 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f607ad9d640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 0x7f607c0b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f607a59c640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f607c0b1ba0 0x7f607c156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f6079d9b640 1 -- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 msgr2=0x7f607c0b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:24.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f6079d9b640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 0x7f607c0b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f6079d9b640 1 -- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f607c0b1ba0 msgr2=0x7f607c156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:24.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f6079d9b640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f607c0b1ba0 0x7f607c156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:24.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.695+0000 7f6079d9b640 1 -- 172.21.15.45:0/4068069873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60740094d0 con 0x7f607c0cdd30 2023-11-28T21:52:24.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.696+0000 7f6079d9b640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0cdd30 0x7f607c0ca930 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f606400d7d0 tx=0x7f606400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:24.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.696+0000 7f6072ffd640 1 -- 172.21.15.45:0/4068069873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60640096a0 con 0x7f607c0cdd30 2023-11-28T21:52:24.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.696+0000 7f6072ffd640 1 -- 172.21.15.45:0/4068069873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6064009800 con 0x7f607c0cdd30 2023-11-28T21:52:24.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.696+0000 7f6072ffd640 1 -- 172.21.15.45:0/4068069873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064009b90 con 0x7f607c0cdd30 2023-11-28T21:52:24.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.696+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607c0b61c0 con 0x7f607c0cdd30 2023-11-28T21:52:24.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.696+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607c0b6780 con 0x7f607c0cdd30 2023-11-28T21:52:24.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.697+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607c066ca0 con 0x7f607c0cdd30 2023-11-28T21:52:24.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.700+0000 7f6072ffd640 1 -- 172.21.15.45:0/4068069873 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6064009cf0 con 0x7f607c0cdd30 2023-11-28T21:52:24.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.700+0000 7f6072ffd640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f605c06af60 0x7f605c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:24.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.701+0000 7f6072ffd640 1 -- 172.21.15.45:0/4068069873 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f606408a660 con 0x7f607c0cdd30 2023-11-28T21:52:24.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.701+0000 7f607a59c640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f605c06af60 0x7f605c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:24.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.702+0000 7f607a59c640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f605c06af60 0x7f605c06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f606c009f30 tx=0x7f606c00b040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:24.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.705+0000 7f6072ffd640 1 -- 172.21.15.45:0/4068069873 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6064055760 con 0x7f607c0cdd30 2023-11-28T21:52:24.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:24.898+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f607c0733d0 con 0x7f607c0cdd30 2023-11-28T21:52:25.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.447+0000 7f6072ffd640 1 -- 172.21.15.45:0/4068069873 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v197) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6064058d80 con 0x7f607c0cdd30 2023-11-28T21:52:25.450 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T21:52:25.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f605c06af60 msgr2=0x7f605c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:25.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f605c06af60 0x7f605c06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f606c009f30 tx=0x7f606c00b040 comp rx=0 tx=0).stop 2023-11-28T21:52:25.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0cdd30 msgr2=0x7f607c0ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:25.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0cdd30 0x7f607c0ca930 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f606400d7d0 tx=0x7f606400dc90 comp rx=0 tx=0).stop 2023-11-28T21:52:25.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 shutdown_connections 2023-11-28T21:52:25.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f605c06af60 0x7f605c06d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:25.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f607c15afd0 0x7f607c0b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:25.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f607c0cdd30 0x7f607c0ca930 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:25.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 --2- 172.21.15.45:0/4068069873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f607c0b1ba0 0x7f607c156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:25.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 >> 172.21.15.45:0/4068069873 conn(0x7f607c099a40 msgr2=0x7f607c0cb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:25.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 shutdown_connections 2023-11-28T21:52:25.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.453+0000 7f607b59e640 1 -- 172.21.15.45:0/4068069873 wait complete. 2023-11-28T21:52:25.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T21:52:25.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.682+0000 7f2ca1961640 1 Processor -- start 2023-11-28T21:52:25.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.682+0000 7f2ca1961640 1 -- start start 2023-11-28T21:52:25.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.684+0000 7f2ca1961640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c9c06c350 0x7f2c9c15f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:25.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.684+0000 7f2ca1961640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9c15ecd0 con 0x7f2c9c15e840 2023-11-28T21:52:25.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.684+0000 7f2ca1961640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9c15f290 con 0x7f2c9c06ac10 2023-11-28T21:52:25.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.684+0000 7f2ca1961640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9c072020 con 0x7f2c9c06c350 2023-11-28T21:52:25.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.684+0000 7f2c9b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c9c06c350 0x7f2c9c15f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:25.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.684+0000 7f2c9b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2c9c06c350 0x7f2c9c15f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50596/0 (socket says 172.21.15.45:50596) 2023-11-28T21:52:25.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.684+0000 7f2c9b7fe640 1 -- 172.21.15.45:0/4224540560 learned_addr learned my addr 172.21.15.45:0/4224540560 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:25.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.685+0000 7f2c9b7fe640 1 -- 172.21.15.45:0/4224540560 >> v1:172.21.15.190:6789/0 conn(0x7f2c9c06ac10 legacy=0x7f2c9c0cc930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T21:52:25.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.685+0000 7f2c9b7fe640 1 -- 172.21.15.45:0/4224540560 >> v1:172.21.15.45:6789/0 conn(0x7f2c9c15e840 legacy=0x7f2c9c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:25.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.685+0000 7f2c9b7fe640 1 -- 172.21.15.45:0/4224540560 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c072300 con 0x7f2c9c06c350 2023-11-28T21:52:25.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.686+0000 7f2c9b7fe640 1 --2- 172.21.15.45:0/4224540560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c06c350 0x7f2c9c15f720 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009e50 tx=0x7f2c9402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66cdc7e705046f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:25.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.686+0000 7f2c99ffb640 1 -- 172.21.15.45:0/4224540560 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9402cb10 con 0x7f2c9c06c350 2023-11-28T21:52:25.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.686+0000 7f2c99ffb640 1 -- 172.21.15.45:0/4224540560 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9402cc70 con 0x7f2c9c06c350 2023-11-28T21:52:25.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.686+0000 7f2c99ffb640 1 -- 172.21.15.45:0/4224540560 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c940343f0 con 0x7f2c9c06c350 2023-11-28T21:52:25.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.687+0000 7f2ca1961640 1 -- 172.21.15.45:0/4224540560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c06c350 msgr2=0x7f2c9c15f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:25.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.687+0000 7f2ca1961640 1 --2- 172.21.15.45:0/4224540560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c06c350 0x7f2c9c15f720 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009e50 tx=0x7f2c9402bf70 comp rx=0 tx=0).stop 2023-11-28T21:52:25.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.687+0000 7f2ca1961640 1 -- 172.21.15.45:0/4224540560 shutdown_connections 2023-11-28T21:52:25.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.687+0000 7f2ca1961640 1 --2- 172.21.15.45:0/4224540560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c06c350 0x7f2c9c15f720 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:25.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.687+0000 7f2ca1961640 1 -- 172.21.15.45:0/4224540560 >> 172.21.15.45:0/4224540560 conn(0x7f2c9c0ced60 msgr2=0x7f2c9c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:25.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.687+0000 7f2ca1961640 1 -- 172.21.15.45:0/4224540560 shutdown_connections 2023-11-28T21:52:25.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.687+0000 7f2ca1961640 1 -- 172.21.15.45:0/4224540560 wait complete. 2023-11-28T21:52:25.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.689+0000 7f2ca1961640 1 Processor -- start 2023-11-28T21:52:25.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.689+0000 7f2ca1961640 1 -- start start 2023-11-28T21:52:25.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.690+0000 7f2ca1961640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9c06ac10 0x7f2c9c0c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:25.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.690+0000 7f2ca1961640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c9c06c350 0x7f2c9c0c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.690+0000 7f2ca1961640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c15e840 0x7f2c9c0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.690+0000 7f2ca1961640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c1f1b30 con 0x7f2c9c15e840 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.690+0000 7f2ca1961640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0c9000 con 0x7f2c9c06c350 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.690+0000 7f2ca1961640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c1f1200 con 0x7f2c9c06ac10 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c15e840 0x7f2c9c0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c15e840 0x7f2c9c0c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50610/0 (socket says 172.21.15.45:50610) 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9b7fe640 1 -- 172.21.15.45:0/1402580757 learned_addr learned my addr 172.21.15.45:0/1402580757 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9a7fc640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c9c06c350 0x7f2c9c0c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9affd640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9c06ac10 0x7f2c9c0c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9b7fe640 1 -- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9c06ac10 msgr2=0x7f2c9c0c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9b7fe640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9c06ac10 0x7f2c9c0c6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:25.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9b7fe640 1 -- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c9c06c350 msgr2=0x7f2c9c0c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:25.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9b7fe640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c9c06c350 0x7f2c9c0c8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:25.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.691+0000 7f2c9b7fe640 1 -- 172.21.15.45:0/1402580757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c94009b30 con 0x7f2c9c15e840 2023-11-28T21:52:25.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.692+0000 7f2c9b7fe640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c15e840 0x7f2c9c0c8980 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f2c94036040 tx=0x7f2c94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:25.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.692+0000 7f2c83fff640 1 -- 172.21.15.45:0/1402580757 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9403e450 con 0x7f2c9c15e840 2023-11-28T21:52:25.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.692+0000 7f2c83fff640 1 -- 172.21.15.45:0/1402580757 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9403e5d0 con 0x7f2c9c15e840 2023-11-28T21:52:25.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.692+0000 7f2c83fff640 1 -- 172.21.15.45:0/1402580757 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9403e980 con 0x7f2c9c15e840 2023-11-28T21:52:25.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.692+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9c1f14e0 con 0x7f2c9c15e840 2023-11-28T21:52:25.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.692+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c15a630 con 0x7f2c9c15e840 2023-11-28T21:52:25.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.693+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c1f1ce0 con 0x7f2c9c15e840 2023-11-28T21:52:25.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.695+0000 7f2c83fff640 1 -- 172.21.15.45:0/1402580757 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2c9403eae0 con 0x7f2c9c15e840 2023-11-28T21:52:25.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.695+0000 7f2c83fff640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f2c8806afb0 0x7f2c8806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:25.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.696+0000 7f2c83fff640 1 -- 172.21.15.45:0/1402580757 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f2c940ac5c0 con 0x7f2c9c15e840 2023-11-28T21:52:25.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.696+0000 7f2c9affd640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f2c8806afb0 0x7f2c8806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:25.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.697+0000 7f2c9affd640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f2c8806afb0 0x7f2c8806d470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c007fd0 tx=0x7f2c8c009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:25.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.700+0000 7f2c83fff640 1 -- 172.21.15.45:0/1402580757 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c9c1f1ce0 con 0x7f2c9c15e840 2023-11-28T21:52:25.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:25.936+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2c9c0733d0 con 0x7f2c9c15e840 2023-11-28T21:52:26.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.855+0000 7f2c83fff640 1 -- 172.21.15.45:0/1402580757 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v88) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2c94077590 con 0x7f2c9c15e840 2023-11-28T21:52:26.857 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 23 and data pool 24 2023-11-28T21:52:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.859+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f2c8806afb0 msgr2=0x7f2c8806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.859+0000 7f2ca1961640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f2c8806afb0 0x7f2c8806d470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c007fd0 tx=0x7f2c8c009040 comp rx=0 tx=0).stop 2023-11-28T21:52:26.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.859+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c15e840 msgr2=0x7f2c9c0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.859+0000 7f2ca1961640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c15e840 0x7f2c9c0c8980 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f2c94036040 tx=0x7f2c94004a40 comp rx=0 tx=0).stop 2023-11-28T21:52:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.860+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 shutdown_connections 2023-11-28T21:52:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.860+0000 7f2ca1961640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f2c8806afb0 0x7f2c8806d470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.860+0000 7f2ca1961640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c9c15e840 0x7f2c9c0c8980 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.860+0000 7f2ca1961640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c9c06c350 0x7f2c9c0c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.860+0000 7f2ca1961640 1 --2- 172.21.15.45:0/1402580757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c9c06ac10 0x7f2c9c0c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.860+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 >> 172.21.15.45:0/1402580757 conn(0x7f2c9c0ced60 msgr2=0x7f2c9c0ac450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.860+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 shutdown_connections 2023-11-28T21:52:26.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:26.860+0000 7f2ca1961640 1 -- 172.21.15.45:0/1402580757 wait complete. 2023-11-28T21:52:26.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:52:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.080+0000 7f23761a5640 1 Processor -- start 2023-11-28T21:52:27.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.080+0000 7f23761a5640 1 -- start start 2023-11-28T21:52:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.082+0000 7f23761a5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f23700b35b0 0x7f237015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.082+0000 7f23761a5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f237015b850 0x7f23700b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.082+0000 7f23761a5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237015bda0 con 0x7f23700b1fa0 2023-11-28T21:52:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.082+0000 7f23761a5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237015ee40 con 0x7f23700b35b0 2023-11-28T21:52:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.082+0000 7f23761a5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237015c080 con 0x7f237015b850 2023-11-28T21:52:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f2367fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f237015b850 0x7f23700b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f236f7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f23700b35b0 0x7f237015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f236f7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f23700b35b0 0x7f237015ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:39144/0 (socket says 172.21.15.45:39144) 2023-11-28T21:52:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f236f7fe640 1 -- 172.21.15.45:0/3898619423 learned_addr learned my addr 172.21.15.45:0/3898619423 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f2367fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f237015b850 0x7f23700b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:53476/0 (socket says 172.21.15.45:53476) 2023-11-28T21:52:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f236ffff640 1 --1- 172.21.15.45:0/3898619423 >> v1:172.21.15.190:6789/0 conn(0x7f23700b1fa0 0x7f237015f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:32968/0 (socket says 172.21.15.45:32968) 2023-11-28T21:52:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f236f7fe640 1 -- 172.21.15.45:0/3898619423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f237015b850 msgr2=0x7f23700b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f236f7fe640 1 --2- 172.21.15.45:0/3898619423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f237015b850 0x7f23700b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f236f7fe640 1 -- 172.21.15.45:0/3898619423 >> v1:172.21.15.190:6789/0 conn(0x7f23700b1fa0 legacy=0x7f237015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.083+0000 7f236f7fe640 1 -- 172.21.15.45:0/3898619423 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237015c360 con 0x7f23700b35b0 2023-11-28T21:52:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.085+0000 7f236f7fe640 1 --2- 172.21.15.45:0/3898619423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23700b35b0 0x7f237015ca10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f23600098a0 tx=0x7f236002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85877fd76f988f4e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.085+0000 7f236effd640 1 -- 172.21.15.45:0/3898619423 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360030840 con 0x7f23700b35b0 2023-11-28T21:52:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.085+0000 7f236effd640 1 -- 172.21.15.45:0/3898619423 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23600309a0 con 0x7f23700b35b0 2023-11-28T21:52:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.085+0000 7f236effd640 1 -- 172.21.15.45:0/3898619423 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360030d30 con 0x7f23700b35b0 2023-11-28T21:52:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.086+0000 7f23761a5640 1 -- 172.21.15.45:0/3898619423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23700b35b0 msgr2=0x7f237015ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.086+0000 7f23761a5640 1 --2- 172.21.15.45:0/3898619423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23700b35b0 0x7f237015ca10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f23600098a0 tx=0x7f236002b980 comp rx=0 tx=0).stop 2023-11-28T21:52:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.086+0000 7f23761a5640 1 -- 172.21.15.45:0/3898619423 shutdown_connections 2023-11-28T21:52:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.086+0000 7f23761a5640 1 --2- 172.21.15.45:0/3898619423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f237015b850 0x7f23700b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.086+0000 7f23761a5640 1 --2- 172.21.15.45:0/3898619423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23700b35b0 0x7f237015ca10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.086+0000 7f23761a5640 1 -- 172.21.15.45:0/3898619423 >> 172.21.15.45:0/3898619423 conn(0x7f237006a500 msgr2=0x7f237006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:27.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.086+0000 7f23761a5640 1 -- 172.21.15.45:0/3898619423 shutdown_connections 2023-11-28T21:52:27.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.086+0000 7f23761a5640 1 -- 172.21.15.45:0/3898619423 wait complete. 2023-11-28T21:52:27.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.088+0000 7f23761a5640 1 Processor -- start 2023-11-28T21:52:27.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.088+0000 7f23761a5640 1 -- start start 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.089+0000 7f23761a5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23700b1fa0 0x7f23701f0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.089+0000 7f23761a5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23700b35b0 0x7f2370134e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.089+0000 7f23761a5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f237015b850 0x7f23700ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.089+0000 7f23761a5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23701353a0 con 0x7f23700b1fa0 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.089+0000 7f23761a5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23701555f0 con 0x7f237015b850 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.089+0000 7f23761a5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23700ad210 con 0x7f23700b35b0 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.089+0000 7f236f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23700b1fa0 0x7f23701f0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.089+0000 7f236f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23700b1fa0 0x7f23701f0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50622/0 (socket says 172.21.15.45:50622) 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.089+0000 7f236f7fe640 1 -- 172.21.15.45:0/4123321062 learned_addr learned my addr 172.21.15.45:0/4123321062 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f2367fff640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23700b35b0 0x7f2370134e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236ffff640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f237015b850 0x7f23700ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236f7fe640 1 -- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23700b35b0 msgr2=0x7f2370134e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236f7fe640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23700b35b0 0x7f2370134e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236f7fe640 1 -- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f237015b850 msgr2=0x7f23700ac9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236f7fe640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f237015b850 0x7f23700ac9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236f7fe640 1 -- 172.21.15.45:0/4123321062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23600094d0 con 0x7f23700b1fa0 2023-11-28T21:52:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236ffff640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f237015b850 0x7f23700ac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:27.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236f7fe640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23700b1fa0 0x7f23701f0920 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f2360000c00 tx=0x7f2360004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:27.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236cff9640 1 -- 172.21.15.45:0/4123321062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360039c90 con 0x7f23700b1fa0 2023-11-28T21:52:27.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236cff9640 1 -- 172.21.15.45:0/4123321062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2360039e10 con 0x7f23700b1fa0 2023-11-28T21:52:27.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f236cff9640 1 -- 172.21.15.45:0/4123321062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360043640 con 0x7f23700b1fa0 2023-11-28T21:52:27.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.090+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23700ad4f0 con 0x7f23700b1fa0 2023-11-28T21:52:27.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.091+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23700adab0 con 0x7f23700b1fa0 2023-11-28T21:52:27.094 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.092+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23700733d0 con 0x7f23700b1fa0 2023-11-28T21:52:27.095 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.093+0000 7f236cff9640 1 -- 172.21.15.45:0/4123321062 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f23600437a0 con 0x7f23700b1fa0 2023-11-28T21:52:27.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.094+0000 7f236cff9640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f235006af60 0x7f235006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.094+0000 7f236cff9640 1 -- 172.21.15.45:0/4123321062 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f236003e070 con 0x7f23700b1fa0 2023-11-28T21:52:27.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.095+0000 7f2367fff640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f235006af60 0x7f235006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.098 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.096+0000 7f2367fff640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f235006af60 0x7f235006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2370138370 tx=0x7f2358008040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:27.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.099+0000 7f236cff9640 1 -- 172.21.15.45:0/4123321062 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2360077710 con 0x7f23700b1fa0 2023-11-28T21:52:27.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.298+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23700cd3b0 con 0x7f23700b1fa0 2023-11-28T21:52:27.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.301+0000 7f236cff9640 1 -- 172.21.15.45:0/4123321062 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+16761 (secure 0 0 0) 0x7f2360034090 con 0x7f23700b1fa0 2023-11-28T21:52:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:27.304 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":198,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:52:25.938416+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:52:23.878130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-11-28T21:52:24.956954+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:52:27.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.303+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f235006af60 msgr2=0x7f235006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.303+0000 7f23761a5640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f235006af60 0x7f235006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2370138370 tx=0x7f2358008040 comp rx=0 tx=0).stop 2023-11-28T21:52:27.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.303+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23700b1fa0 msgr2=0x7f23701f0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.303+0000 7f23761a5640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23700b1fa0 0x7f23701f0920 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f2360000c00 tx=0x7f2360004730 comp rx=0 tx=0).stop 2023-11-28T21:52:27.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.303+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 shutdown_connections 2023-11-28T21:52:27.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.303+0000 7f23761a5640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f235006af60 0x7f235006d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.303+0000 7f23761a5640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f237015b850 0x7f23700ac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.303+0000 7f23761a5640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f23700b35b0 0x7f2370134e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.303+0000 7f23761a5640 1 --2- 172.21.15.45:0/4123321062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23700b1fa0 0x7f23701f0920 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.304+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 >> 172.21.15.45:0/4123321062 conn(0x7f237006a500 msgr2=0x7f23701447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:27.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.304+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 shutdown_connections 2023-11-28T21:52:27.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.304+0000 7f23761a5640 1 -- 172.21.15.45:0/4123321062 wait complete. 2023-11-28T21:52:27.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:52:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.544+0000 7f5d9b5a9640 1 Processor -- start 2023-11-28T21:52:27.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.544+0000 7f5d9b5a9640 1 -- start start 2023-11-28T21:52:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.546+0000 7f5d9b5a9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5d940b1340 0x7f5d9415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.546+0000 7f5d9b5a9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d9415bda0 con 0x7f5d940b35b0 2023-11-28T21:52:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.546+0000 7f5d9b5a9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d9415c080 con 0x7f5d9415b850 2023-11-28T21:52:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.546+0000 7f5d9b5a9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d9415ee40 con 0x7f5d940b1340 2023-11-28T21:52:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.546+0000 7f5d99345640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5d940b35b0 0x7f5d9415ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54256/0 (socket says 172.21.15.45:54256) 2023-11-28T21:52:27.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.546+0000 7f5d99345640 1 -- 172.21.15.45:0/2541423373 learned_addr learned my addr 172.21.15.45:0/2541423373 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562160122 0 0) 0x7f5d9415bda0 con 0x7f5d940b35b0 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d80003520 con 0x7f5d940b35b0 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d99b46640 1 --2- 172.21.15.45:0/2541423373 >> v2:172.21.15.190:3300/0 conn(0x7f5d940b1340 0x7f5d9415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3163390177 0 0) 0x7f5d80003520 con 0x7f5d940b35b0 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 >> v2:172.21.15.190:3300/0 conn(0x7f5d940b1340 msgr2=0x7f5d9415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 --2- 172.21.15.45:0/2541423373 >> v2:172.21.15.190:3300/0 conn(0x7f5d940b1340 0x7f5d9415f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 >> v1:172.21.15.190:6789/0 conn(0x7f5d9415b850 legacy=0x7f5d940b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9415c360 con 0x7f5d940b35b0 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5d90001670 con 0x7f5d940b35b0 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 >> v1:172.21.15.45:6789/0 conn(0x7f5d940b35b0 legacy=0x7f5d9415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d80003970 0x7f5d80003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d800047c0 0x7f5d80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80005180 0x7f5d8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9415ee40 con 0x7f5d80005180 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8000c020 con 0x7f5d800047c0 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.547+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8000c1e0 con 0x7f5d80003970 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.548+0000 7f5d99b46640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80005180 0x7f5d8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.548+0000 7f5d99345640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d80003970 0x7f5d80003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.548+0000 7f5d98b44640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d800047c0 0x7f5d80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.548+0000 7f5d99b46640 1 -- 172.21.15.45:0/2541423373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d80003970 msgr2=0x7f5d80003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.548+0000 7f5d99b46640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d80003970 0x7f5d80003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.548+0000 7f5d99b46640 1 -- 172.21.15.45:0/2541423373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d800047c0 msgr2=0x7f5d80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.548+0000 7f5d99b46640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d800047c0 0x7f5d80004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.548+0000 7f5d99b46640 1 -- 172.21.15.45:0/2541423373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d80003520 con 0x7f5d80005180 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.549+0000 7f5d99b46640 1 -- 172.21.15.45:0/2541423373 reap_dead start 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.549+0000 7f5d99b46640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80005180 0x7f5d8000b980 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009f90 tx=0x7f5d800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9168fa1640542eb4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.549+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c02cc20 con 0x7f5d80005180 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.550+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c00f040 con 0x7f5d80005180 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.550+0000 7f5d8bfff640 1 -- 172.21.15.45:0/2541423373 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c034440 con 0x7f5d80005180 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.550+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/2541423373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80005180 msgr2=0x7f5d8000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.550+0000 7f5d9b5a9640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80005180 0x7f5d8000b980 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009f90 tx=0x7f5d800057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.550+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/2541423373 shutdown_connections 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.550+0000 7f5d9b5a9640 1 --2- 172.21.15.45:0/2541423373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80005180 0x7f5d8000b980 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.550+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/2541423373 >> 172.21.15.45:0/2541423373 conn(0x7f5d9406a500 msgr2=0x7f5d9406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:27.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.550+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/2541423373 shutdown_connections 2023-11-28T21:52:27.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.550+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/2541423373 wait complete. 2023-11-28T21:52:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.552+0000 7f5d9b5a9640 1 Processor -- start 2023-11-28T21:52:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.552+0000 7f5d9b5a9640 1 -- start start 2023-11-28T21:52:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.553+0000 7f5d9b5a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d80005180 0x7f5d941427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.553+0000 7f5d9b5a9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d940cc2c0 0x7f5d941e5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.553+0000 7f5d9b5a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d941e68a0 0x7f5d940a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.553+0000 7f5d9b5a9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940ab930 con 0x7f5d941e68a0 2023-11-28T21:52:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.553+0000 7f5d9b5a9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d941e6410 con 0x7f5d80005180 2023-11-28T21:52:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.553+0000 7f5d9b5a9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d94142d30 con 0x7f5d940cc2c0 2023-11-28T21:52:27.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d99b46640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d941e68a0 0x7f5d940a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d99b46640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d941e68a0 0x7f5d940a9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50642/0 (socket says 172.21.15.45:50642) 2023-11-28T21:52:27.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d99b46640 1 -- 172.21.15.45:0/596988793 learned_addr learned my addr 172.21.15.45:0/596988793 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:27.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d99345640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d80005180 0x7f5d941427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d98b44640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d940cc2c0 0x7f5d941e5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d99b46640 1 -- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d940cc2c0 msgr2=0x7f5d941e5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d99b46640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d940cc2c0 0x7f5d941e5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d99b46640 1 -- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d80005180 msgr2=0x7f5d941427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d99b46640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d80005180 0x7f5d941427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d99b46640 1 -- 172.21.15.45:0/596988793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d941e5560 con 0x7f5d941e68a0 2023-11-28T21:52:27.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.554+0000 7f5d98b44640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d940cc2c0 0x7f5d941e5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:27.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.555+0000 7f5d99b46640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d941e68a0 0x7f5d940a9d30 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00e230 tx=0x7f5d8c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:27.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.555+0000 7f5d89ffb640 1 -- 172.21.15.45:0/596988793 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c02cea0 con 0x7f5d941e68a0 2023-11-28T21:52:27.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.555+0000 7f5d89ffb640 1 -- 172.21.15.45:0/596988793 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c03d400 con 0x7f5d941e68a0 2023-11-28T21:52:27.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.555+0000 7f5d89ffb640 1 -- 172.21.15.45:0/596988793 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c03d7b0 con 0x7f5d941e68a0 2023-11-28T21:52:27.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.555+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d941e5800 con 0x7f5d941e68a0 2023-11-28T21:52:27.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.555+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d941e5dc0 con 0x7f5d941e68a0 2023-11-28T21:52:27.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.556+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d940733d0 con 0x7f5d941e68a0 2023-11-28T21:52:27.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.557+0000 7f5d89ffb640 1 -- 172.21.15.45:0/596988793 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5d8c03b040 con 0x7f5d941e68a0 2023-11-28T21:52:27.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.558+0000 7f5d89ffb640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5d7806af90 0x7f5d7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:27.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.558+0000 7f5d89ffb640 1 -- 172.21.15.45:0/596988793 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5d8c0ab540 con 0x7f5d941e68a0 2023-11-28T21:52:27.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.559+0000 7f5d99345640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5d7806af90 0x7f5d7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:27.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.560+0000 7f5d99345640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5d7806af90 0x7f5d7806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5d90001050 tx=0x7f5d90007040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.563+0000 7f5d89ffb640 1 -- 172.21.15.45:0/596988793 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d8c076390 con 0x7f5d941e68a0 2023-11-28T21:52:27.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.761+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d941e6550 con 0x7f5d941e68a0 2023-11-28T21:52:27.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.763+0000 7f5d89ffb640 1 -- 172.21.15.45:0/596988793 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+16748 (secure 0 0 0) 0x7f5d8c0799b0 con 0x7f5d941e68a0 2023-11-28T21:52:27.765 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:27.765 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":199,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:52:27.475937+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:52:23.878130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-11-28T21:52:24.956954+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:52:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5d7806af90 msgr2=0x7f5d7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5d7806af90 0x7f5d7806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5d90001050 tx=0x7f5d90007040 comp rx=0 tx=0).stop 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d941e68a0 msgr2=0x7f5d940a9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d941e68a0 0x7f5d940a9d30 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00e230 tx=0x7f5d8c009750 comp rx=0 tx=0).stop 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 shutdown_connections 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f5d7806af90 0x7f5d7806d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d941e68a0 0x7f5d940a9d30 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d940cc2c0 0x7f5d941e5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 --2- 172.21.15.45:0/596988793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d80005180 0x7f5d941427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 >> 172.21.15.45:0/596988793 conn(0x7f5d9406a500 msgr2=0x7f5d941447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 shutdown_connections 2023-11-28T21:52:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:27.766+0000 7f5d9b5a9640 1 -- 172.21.15.45:0/596988793 wait complete. 2023-11-28T21:52:27.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T21:52:28.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.016+0000 7fba417fc640 1 Processor -- start 2023-11-28T21:52:28.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.016+0000 7fba417fc640 1 -- start start 2023-11-28T21:52:28.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.017+0000 7fba417fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c15e840 0x7fba3c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:28.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.017+0000 7fba417fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba3c06ac10 0x7fba3c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:28.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.017+0000 7fba417fc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3c15ecd0 con 0x7fba3c15f720 2023-11-28T21:52:28.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.018+0000 7fba417fc640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3c071ce0 con 0x7fba3c06ac10 2023-11-28T21:52:28.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.018+0000 7fba417fc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c15f290 con 0x7fba3c15e840 2023-11-28T21:52:28.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.018+0000 7fba3b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba3c06ac10 0x7fba3c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:28.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.018+0000 7fba3b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fba3c06ac10 0x7fba3c0717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50646/0 (socket says 172.21.15.45:50646) 2023-11-28T21:52:28.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.018+0000 7fba3b7fe640 1 -- 172.21.15.45:0/281587362 learned_addr learned my addr 172.21.15.45:0/281587362 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:28.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.018+0000 7fba3affd640 1 --2- 172.21.15.45:0/281587362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c15e840 0x7fba3c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:28.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.018+0000 7fba3b7fe640 1 -- 172.21.15.45:0/281587362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c15e840 msgr2=0x7fba3c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:28.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.018+0000 7fba3b7fe640 1 --2- 172.21.15.45:0/281587362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c15e840 0x7fba3c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.019+0000 7fba3b7fe640 1 -- 172.21.15.45:0/281587362 >> v1:172.21.15.190:6789/0 conn(0x7fba3c15f720 legacy=0x7fba3c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:28.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.019+0000 7fba3b7fe640 1 -- 172.21.15.45:0/281587362 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c15fb60 con 0x7fba3c06ac10 2023-11-28T21:52:28.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.019+0000 7fba3affd640 1 --2- 172.21.15.45:0/281587362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c15e840 0x7fba3c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:28.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.020+0000 7fba3b7fe640 1 --2- 172.21.15.45:0/281587362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c06ac10 0x7fba3c0717a0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fba34009870 tx=0x7fba3402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8358d49e6ec7a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:28.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.020+0000 7fba39ffb640 1 -- 172.21.15.45:0/281587362 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34031640 con 0x7fba3c06ac10 2023-11-28T21:52:28.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.020+0000 7fba39ffb640 1 -- 172.21.15.45:0/281587362 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba340317a0 con 0x7fba3c06ac10 2023-11-28T21:52:28.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.020+0000 7fba39ffb640 1 -- 172.21.15.45:0/281587362 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34031b30 con 0x7fba3c06ac10 2023-11-28T21:52:28.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.021+0000 7fba417fc640 1 -- 172.21.15.45:0/281587362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c06ac10 msgr2=0x7fba3c0717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:28.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.021+0000 7fba417fc640 1 --2- 172.21.15.45:0/281587362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c06ac10 0x7fba3c0717a0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fba34009870 tx=0x7fba3402b8f0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.021+0000 7fba417fc640 1 -- 172.21.15.45:0/281587362 shutdown_connections 2023-11-28T21:52:28.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.021+0000 7fba417fc640 1 --2- 172.21.15.45:0/281587362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c06ac10 0x7fba3c0717a0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.021+0000 7fba417fc640 1 --2- 172.21.15.45:0/281587362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c15e840 0x7fba3c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.021+0000 7fba417fc640 1 -- 172.21.15.45:0/281587362 >> 172.21.15.45:0/281587362 conn(0x7fba3c0ced60 msgr2=0x7fba3c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:28.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.021+0000 7fba417fc640 1 -- 172.21.15.45:0/281587362 shutdown_connections 2023-11-28T21:52:28.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.021+0000 7fba417fc640 1 -- 172.21.15.45:0/281587362 wait complete. 2023-11-28T21:52:28.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.023+0000 7fba417fc640 1 Processor -- start 2023-11-28T21:52:28.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.023+0000 7fba417fc640 1 -- start start 2023-11-28T21:52:28.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba417fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c06ac10 0x7fba3c0cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba417fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c15e840 0x7fba3c15a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba417fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba3c15f720 0x7fba3c0c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba417fc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c15a860 con 0x7fba3c15e840 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba417fc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0c9fe0 con 0x7fba3c15f720 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba417fc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0ca2c0 con 0x7fba3c06ac10 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba3a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c15e840 0x7fba3c15a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba3a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c15e840 0x7fba3c15a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50660/0 (socket says 172.21.15.45:50660) 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba3a7fc640 1 -- 172.21.15.45:0/1742865399 learned_addr learned my addr 172.21.15.45:0/1742865399 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba3a7fc640 1 -- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c06ac10 msgr2=0x7fba3c0cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:28.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba3a7fc640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c06ac10 0x7fba3c0cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba3a7fc640 1 -- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba3c15f720 msgr2=0x7fba3c0c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:28.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba3a7fc640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba3c15f720 0x7fba3c0c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.024+0000 7fba3a7fc640 1 -- 172.21.15.45:0/1742865399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba340094d0 con 0x7fba3c15e840 2023-11-28T21:52:28.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.025+0000 7fba3a7fc640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c15e840 0x7fba3c15a320 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fba2400b3f0 tx=0x7fba2400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:28.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.025+0000 7fba337fe640 1 -- 172.21.15.45:0/1742865399 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24011830 con 0x7fba3c15e840 2023-11-28T21:52:28.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.025+0000 7fba337fe640 1 -- 172.21.15.45:0/1742865399 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba24011990 con 0x7fba3c15e840 2023-11-28T21:52:28.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.025+0000 7fba337fe640 1 -- 172.21.15.45:0/1742865399 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24011d40 con 0x7fba3c15e840 2023-11-28T21:52:28.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.025+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c0ca5a0 con 0x7fba3c15e840 2023-11-28T21:52:28.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.025+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c0cab60 con 0x7fba3c15e840 2023-11-28T21:52:28.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.026+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c0cad50 con 0x7fba3c15e840 2023-11-28T21:52:28.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.028+0000 7fba337fe640 1 -- 172.21.15.45:0/1742865399 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fba2401a400 con 0x7fba3c15e840 2023-11-28T21:52:28.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.028+0000 7fba337fe640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fba1c06af60 0x7fba1c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:28.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.029+0000 7fba337fe640 1 -- 172.21.15.45:0/1742865399 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fba240897d0 con 0x7fba3c15e840 2023-11-28T21:52:28.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.029+0000 7fba3affd640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fba1c06af60 0x7fba1c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:28.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.030+0000 7fba3affd640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fba1c06af60 0x7fba1c06d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba3c06aa90 tx=0x7fba2c0083d0 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:28.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.033+0000 7fba337fe640 1 -- 172.21.15.45:0/1742865399 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba24054570 con 0x7fba3c15e840 2023-11-28T21:52:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.271+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fba3c0733d0 con 0x7fba3c15e840 2023-11-28T21:52:28.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.868+0000 7fba337fe640 1 -- 172.21.15.45:0/1742865399 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v91) v1 ==== 118+0+0 (secure 0 0 0) 0x7fba24057b90 con 0x7fba3c15e840 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fba1c06af60 msgr2=0x7fba1c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fba1c06af60 0x7fba1c06d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba3c06aa90 tx=0x7fba2c0083d0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c15e840 msgr2=0x7fba3c15a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c15e840 0x7fba3c15a320 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fba2400b3f0 tx=0x7fba2400b8b0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 shutdown_connections 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fba1c06af60 0x7fba1c06d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba3c15f720 0x7fba3c0c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba3c15e840 0x7fba3c15a320 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 --2- 172.21.15.45:0/1742865399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fba3c06ac10 0x7fba3c0cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 >> 172.21.15.45:0/1742865399 conn(0x7fba3c0ced60 msgr2=0x7fba3c0ace10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 shutdown_connections 2023-11-28T21:52:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:28.872+0000 7fba417fc640 1 -- 172.21.15.45:0/1742865399 wait complete. 2023-11-28T21:52:28.897 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T21:52:28.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T21:52:29.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.124+0000 7f00dd954640 1 Processor -- start 2023-11-28T21:52:29.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.124+0000 7f00dd954640 1 -- start start 2023-11-28T21:52:29.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.126+0000 7f00dd954640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f00d80b2430 0x7f00d80ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:29.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.126+0000 7f00dd954640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f00d8071ac0 0x7f00d80cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:29.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.126+0000 7f00dd954640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d809ceb0 con 0x7f00d8159fc0 2023-11-28T21:52:29.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.126+0000 7f00dd954640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d8071450 con 0x7f00d8071ac0 2023-11-28T21:52:29.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.126+0000 7f00dd954640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d80b2cb0 con 0x7f00d80b2430 2023-11-28T21:52:29.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.126+0000 7f00dc952640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f00d8159fc0 0x7f00d80b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54266/0 (socket says 172.21.15.45:54266) 2023-11-28T21:52:29.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.126+0000 7f00dc952640 1 -- 172.21.15.45:0/3292388584 learned_addr learned my addr 172.21.15.45:0/3292388584 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:29.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.126+0000 7f00dd153640 1 --2- 172.21.15.45:0/3292388584 >> v2:172.21.15.45:3300/0 conn(0x7f00d8071ac0 0x7f00d80cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:29.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.127+0000 7f00d7fff640 1 --2- 172.21.15.45:0/3292388584 >> v2:172.21.15.190:3300/0 conn(0x7f00d80b2430 0x7f00d80ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:29.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.127+0000 7f00dd153640 1 -- 172.21.15.45:0/3292388584 >> v2:172.21.15.190:3300/0 conn(0x7f00d80b2430 msgr2=0x7f00d80ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:29.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.127+0000 7f00dd153640 1 --2- 172.21.15.45:0/3292388584 >> v2:172.21.15.190:3300/0 conn(0x7f00d80b2430 0x7f00d80ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:29.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.127+0000 7f00dd153640 1 -- 172.21.15.45:0/3292388584 >> v1:172.21.15.45:6789/0 conn(0x7f00d8159fc0 legacy=0x7f00d80b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:29.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.127+0000 7f00dd153640 1 -- 172.21.15.45:0/3292388584 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80b29d0 con 0x7f00d8071ac0 2023-11-28T21:52:29.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.128+0000 7f00dd153640 1 --2- 172.21.15.45:0/3292388584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 0x7f00d80cd0d0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f00d806b630 tx=0x7f00d002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3fcd656e9027cc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:29.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.128+0000 7f00d77fe640 1 -- 172.21.15.45:0/3292388584 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d002cbe0 con 0x7f00d8071ac0 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.128+0000 7f00d77fe640 1 -- 172.21.15.45:0/3292388584 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d002cd40 con 0x7f00d8071ac0 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.128+0000 7f00d77fe640 1 -- 172.21.15.45:0/3292388584 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d0034660 con 0x7f00d8071ac0 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.129+0000 7f00dd954640 1 -- 172.21.15.45:0/3292388584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 msgr2=0x7f00d80cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.129+0000 7f00dd954640 1 --2- 172.21.15.45:0/3292388584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 0x7f00d80cd0d0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f00d806b630 tx=0x7f00d002be30 comp rx=0 tx=0).stop 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.129+0000 7f00dd954640 1 -- 172.21.15.45:0/3292388584 shutdown_connections 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.129+0000 7f00dd954640 1 --2- 172.21.15.45:0/3292388584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 0x7f00d80cd0d0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.129+0000 7f00dd954640 1 --2- 172.21.15.45:0/3292388584 >> v2:172.21.15.190:3300/0 conn(0x7f00d80b2430 0x7f00d80ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.129+0000 7f00dd954640 1 -- 172.21.15.45:0/3292388584 >> 172.21.15.45:0/3292388584 conn(0x7f00d80b1050 msgr2=0x7f00d80cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.129+0000 7f00dd954640 1 -- 172.21.15.45:0/3292388584 shutdown_connections 2023-11-28T21:52:29.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.129+0000 7f00dd954640 1 -- 172.21.15.45:0/3292388584 wait complete. 2023-11-28T21:52:29.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.131+0000 7f00dd954640 1 Processor -- start 2023-11-28T21:52:29.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.131+0000 7f00dd954640 1 -- start start 2023-11-28T21:52:29.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.132+0000 7f00dd954640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 0x7f00d814dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:29.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.132+0000 7f00dd954640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00d80b2430 0x7f00d8151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:29.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.132+0000 7f00dd954640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00d8159fc0 0x7f00d814b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:29.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.132+0000 7f00dd954640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d814e130 con 0x7f00d8071ac0 2023-11-28T21:52:29.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.132+0000 7f00dd954640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d8152840 con 0x7f00d8159fc0 2023-11-28T21:52:29.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.132+0000 7f00dd954640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d8152b20 con 0x7f00d80b2430 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.132+0000 7f00dc952640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 0x7f00d814dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.132+0000 7f00dc952640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 0x7f00d814dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50684/0 (socket says 172.21.15.45:50684) 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.132+0000 7f00dc952640 1 -- 172.21.15.45:0/1010368276 learned_addr learned my addr 172.21.15.45:0/1010368276 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00dd153640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00d8159fc0 0x7f00d814b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00d7fff640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00d80b2430 0x7f00d8151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00dc952640 1 -- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00d80b2430 msgr2=0x7f00d8151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00dc952640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00d80b2430 0x7f00d8151ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00dc952640 1 -- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00d8159fc0 msgr2=0x7f00d814b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00dc952640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00d8159fc0 0x7f00d814b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00dc952640 1 -- 172.21.15.45:0/1010368276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d0009a70 con 0x7f00d8071ac0 2023-11-28T21:52:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00dc952640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 0x7f00d814dbf0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00c480 tx=0x7f00cc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:29.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00d57fa640 1 -- 172.21.15.45:0/1010368276 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc005ce0 con 0x7f00d8071ac0 2023-11-28T21:52:29.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00d57fa640 1 -- 172.21.15.45:0/1010368276 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc005e40 con 0x7f00d8071ac0 2023-11-28T21:52:29.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d8152e00 con 0x7f00d8071ac0 2023-11-28T21:52:29.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.133+0000 7f00d57fa640 1 -- 172.21.15.45:0/1010368276 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc0072f0 con 0x7f00d8071ac0 2023-11-28T21:52:29.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.134+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d81346d0 con 0x7f00d8071ac0 2023-11-28T21:52:29.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.135+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d8072460 con 0x7f00d8071ac0 2023-11-28T21:52:29.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.136+0000 7f00d57fa640 1 -- 172.21.15.45:0/1010368276 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f00cc007510 con 0x7f00d8071ac0 2023-11-28T21:52:29.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.136+0000 7f00d57fa640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f00bc06af60 0x7f00bc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:29.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.137+0000 7f00d57fa640 1 -- 172.21.15.45:0/1010368276 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f00cc088d80 con 0x7f00d8071ac0 2023-11-28T21:52:29.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.137+0000 7f00d7fff640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f00bc06af60 0x7f00bc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:29.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.138+0000 7f00d7fff640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f00bc06af60 0x7f00bc06d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f00d814e900 tx=0x7f00c4008040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:29.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.141+0000 7f00d57fa640 1 -- 172.21.15.45:0/1010368276 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00cc053d00 con 0x7f00d8071ac0 2023-11-28T21:52:29.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.370+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f00d804e920 con 0x7f00d8071ac0 2023-11-28T21:52:29.883 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.881+0000 7f00d57fa640 1 -- 172.21.15.45:0/1010368276 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v92) v1 ==== 122+0+0 (secure 0 0 0) 0x7f00cc017020 con 0x7f00d8071ac0 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f00bc06af60 msgr2=0x7f00bc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f00bc06af60 0x7f00bc06d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f00d814e900 tx=0x7f00c4008040 comp rx=0 tx=0).stop 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 msgr2=0x7f00d814dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 0x7f00d814dbf0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00c480 tx=0x7f00cc00c940 comp rx=0 tx=0).stop 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 shutdown_connections 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f00bc06af60 0x7f00bc06d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00d8159fc0 0x7f00d814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f00d80b2430 0x7f00d8151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 --2- 172.21.15.45:0/1010368276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00d8071ac0 0x7f00d814dbf0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:29.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.884+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 >> 172.21.15.45:0/1010368276 conn(0x7f00d80b1050 msgr2=0x7f00d80ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:29.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.885+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 shutdown_connections 2023-11-28T21:52:29.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:29.885+0000 7f00dd954640 1 -- 172.21.15.45:0/1010368276 wait complete. 2023-11-28T21:52:29.908 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T21:52:29.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:52:30.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.140+0000 7fee6d2b4640 1 Processor -- start 2023-11-28T21:52:30.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.140+0000 7fee6d2b4640 1 -- start start 2023-11-28T21:52:30.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.141+0000 7fee6d2b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee6815d550 0x7fee680cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.141+0000 7fee6d2b4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fee680d0100 0x7fee680d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.141+0000 7fee6d2b4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee680d0560 con 0x7fee680d1ab0 2023-11-28T21:52:30.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.141+0000 7fee6d2b4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee680d2230 con 0x7fee680d0100 2023-11-28T21:52:30.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.141+0000 7fee6d2b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee680d2510 con 0x7fee6815d550 2023-11-28T21:52:30.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.142+0000 7fee67fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee6815d550 0x7fee680cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.142+0000 7fee67fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee6815d550 0x7fee680cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:53540/0 (socket says 172.21.15.45:53540) 2023-11-28T21:52:30.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.142+0000 7fee67fff640 1 -- 172.21.15.45:0/3704506188 learned_addr learned my addr 172.21.15.45:0/3704506188 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:30.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.142+0000 7fee5f7fe640 1 --2- 172.21.15.45:0/3704506188 >> v2:172.21.15.190:3300/0 conn(0x7fee680d0100 0x7fee680d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.143+0000 7fee5f7fe640 1 -- 172.21.15.45:0/3704506188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee6815d550 msgr2=0x7fee680cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:30.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.143+0000 7fee5f7fe640 1 --2- 172.21.15.45:0/3704506188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee6815d550 0x7fee680cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.143+0000 7fee5f7fe640 1 -- 172.21.15.45:0/3704506188 >> v1:172.21.15.190:6789/0 conn(0x7fee680d1ab0 legacy=0x7fee68070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.143+0000 7fee5f7fe640 1 -- 172.21.15.45:0/3704506188 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee680d27f0 con 0x7fee680d0100 2023-11-28T21:52:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.143+0000 7fee677fe640 1 -- 172.21.15.45:0/3704506188 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049369774 0 0) 0x7fee680d0560 con 0x7fee680d1ab0 2023-11-28T21:52:30.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.145+0000 7fee5f7fe640 1 --2- 172.21.15.45:0/3704506188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee680d0100 0x7fee680d1570 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fee500097a0 tx=0x7fee5002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67e01768b2c2f080 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:30.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee677fe640 1 -- 172.21.15.45:0/3704506188 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50031830 con 0x7fee680d0100 2023-11-28T21:52:30.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee677fe640 1 -- 172.21.15.45:0/3704506188 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee50031990 con 0x7fee680d0100 2023-11-28T21:52:30.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee677fe640 1 -- 172.21.15.45:0/3704506188 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50031d20 con 0x7fee680d0100 2023-11-28T21:52:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee6d2b4640 1 -- 172.21.15.45:0/3704506188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee680d0100 msgr2=0x7fee680d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee6d2b4640 1 --2- 172.21.15.45:0/3704506188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee680d0100 0x7fee680d1570 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fee500097a0 tx=0x7fee5002b8d0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee6d2b4640 1 -- 172.21.15.45:0/3704506188 shutdown_connections 2023-11-28T21:52:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee6d2b4640 1 --2- 172.21.15.45:0/3704506188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee680d0100 0x7fee680d1570 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee6d2b4640 1 --2- 172.21.15.45:0/3704506188 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee6815d550 0x7fee680cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee6d2b4640 1 -- 172.21.15.45:0/3704506188 >> 172.21.15.45:0/3704506188 conn(0x7fee6806ac70 msgr2=0x7fee680b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.146+0000 7fee6d2b4640 1 -- 172.21.15.45:0/3704506188 shutdown_connections 2023-11-28T21:52:30.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.147+0000 7fee6d2b4640 1 -- 172.21.15.45:0/3704506188 wait complete. 2023-11-28T21:52:30.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.148+0000 7fee6d2b4640 1 Processor -- start 2023-11-28T21:52:30.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.148+0000 7fee6d2b4640 1 -- start start 2023-11-28T21:52:30.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.149+0000 7fee6d2b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee680d0100 0x7fee681f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.149+0000 7fee6d2b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee680d1ab0 0x7fee680cc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.149+0000 7fee6d2b4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6815d550 0x7fee680cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.149+0000 7fee6d2b4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee681f6720 con 0x7fee680d0100 2023-11-28T21:52:30.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.149+0000 7fee6d2b4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680cb5e0 con 0x7fee6815d550 2023-11-28T21:52:30.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.149+0000 7fee6d2b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee680c78e0 con 0x7fee680d1ab0 2023-11-28T21:52:30.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee67fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee680d0100 0x7fee681f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee67fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee680d0100 0x7fee681f61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50688/0 (socket says 172.21.15.45:50688) 2023-11-28T21:52:30.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee67fff640 1 -- 172.21.15.45:0/2488384386 learned_addr learned my addr 172.21.15.45:0/2488384386 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:30.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee6cab3640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6815d550 0x7fee680cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee5f7fe640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee680d1ab0 0x7fee680cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee67fff640 1 -- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee680d1ab0 msgr2=0x7fee680cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee67fff640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee680d1ab0 0x7fee680cc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee67fff640 1 -- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6815d550 msgr2=0x7fee680cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee67fff640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6815d550 0x7fee680cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.150+0000 7fee67fff640 1 -- 172.21.15.45:0/2488384386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50009400 con 0x7fee680d0100 2023-11-28T21:52:30.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.151+0000 7fee67fff640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee680d0100 0x7fee681f61e0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fee5800efc0 tx=0x7fee5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:30.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.151+0000 7fee657fa640 1 -- 172.21.15.45:0/2488384386 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58009b60 con 0x7fee680d0100 2023-11-28T21:52:30.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.151+0000 7fee657fa640 1 -- 172.21.15.45:0/2488384386 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5800f040 con 0x7fee680d0100 2023-11-28T21:52:30.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.151+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee680c7bc0 con 0x7fee680d0100 2023-11-28T21:52:30.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.151+0000 7fee657fa640 1 -- 172.21.15.45:0/2488384386 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee580092e0 con 0x7fee680d0100 2023-11-28T21:52:30.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.151+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee680c8180 con 0x7fee680d0100 2023-11-28T21:52:30.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.152+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6815c3f0 con 0x7fee680d0100 2023-11-28T21:52:30.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.153+0000 7fee657fa640 1 -- 172.21.15.45:0/2488384386 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fee58014480 con 0x7fee680d0100 2023-11-28T21:52:30.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.154+0000 7fee657fa640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fee4806af60 0x7fee4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.154+0000 7fee657fa640 1 -- 172.21.15.45:0/2488384386 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fee5808a1b0 con 0x7fee680d0100 2023-11-28T21:52:30.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.155+0000 7fee5f7fe640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fee4806af60 0x7fee4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.156+0000 7fee5f7fe640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fee4806af60 0x7fee4806d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee50036000 tx=0x7fee50008030 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:30.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.159+0000 7fee657fa640 1 -- 172.21.15.45:0/2488384386 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee58054e50 con 0x7fee680d0100 2023-11-28T21:52:30.394 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.392+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee6804e920 con 0x7fee680d0100 2023-11-28T21:52:30.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.394+0000 7fee657fa640 1 -- 172.21.15.45:0/2488384386 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4927 (secure 0 0 0) 0x7fee58058470 con 0x7fee680d0100 2023-11-28T21:52:30.396 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:30.396 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3312844962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3312844962},{"type":"v1","addr":"172.21.15.190:6835","nonce":3312844962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":8296,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1302790439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1302790439},{"type":"v1","addr":"172.21.15.190:6837","nonce":1302790439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10104,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2082498185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2082498185},{"type":"v1","addr":"172.21.15.45:6834","nonce":2082498185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:52:25.939261+0000","modified":"2023-11-28T21:52:29.372705+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10110":{"gid":10110,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3063278837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3063278837},{"type":"v1","addr":"172.21.15.45:6835","nonce":3063278837}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-28T21:52:30.396 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 92 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fee4806af60 msgr2=0x7fee4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fee4806af60 0x7fee4806d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee50036000 tx=0x7fee50008030 comp rx=0 tx=0).stop 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee680d0100 msgr2=0x7fee681f61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee680d0100 0x7fee681f61e0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fee5800efc0 tx=0x7fee5800c490 comp rx=0 tx=0).stop 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 shutdown_connections 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fee4806af60 0x7fee4806d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6815d550 0x7fee680cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee680d1ab0 0x7fee680cc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 --2- 172.21.15.45:0/2488384386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee680d0100 0x7fee681f61e0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.396+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 >> 172.21.15.45:0/2488384386 conn(0x7fee6806ac70 msgr2=0x7fee6815e410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:30.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.397+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 shutdown_connections 2023-11-28T21:52:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.397+0000 7fee6d2b4640 1 -- 172.21.15.45:0/2488384386 wait complete. 2023-11-28T21:52:30.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T21:52:30.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.640+0000 7fef9ffff640 1 Processor -- start 2023-11-28T21:52:30.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.640+0000 7fef9ffff640 1 -- start start 2023-11-28T21:52:30.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.642+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa014b090 0x7fefa00b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.642+0000 7fef9ffff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fefa00ae590 0x7fefa00b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.642+0000 7fef9ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fefa006c060 0x7fefa0165250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.642+0000 7fef9ffff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa00abf10 con 0x7fefa006c060 2023-11-28T21:52:30.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.642+0000 7fef9ffff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa006db90 con 0x7fefa00ae590 2023-11-28T21:52:30.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.642+0000 7fef9ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa006d2f0 con 0x7fefa014b090 2023-11-28T21:52:30.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9f7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fefa006c060 0x7fefa0165250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9f7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fefa006c060 0x7fefa0165250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50698/0 (socket says 172.21.15.45:50698) 2023-11-28T21:52:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9f7fe640 1 -- 172.21.15.45:0/588568616 learned_addr learned my addr 172.21.15.45:0/588568616 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9e7fc640 1 --2- 172.21.15.45:0/588568616 >> v2:172.21.15.190:3300/0 conn(0x7fefa00ae590 0x7fefa00b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9effd640 1 --2- 172.21.15.45:0/588568616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa014b090 0x7fefa00b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9f7fe640 1 -- 172.21.15.45:0/588568616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa014b090 msgr2=0x7fefa00b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9f7fe640 1 --2- 172.21.15.45:0/588568616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa014b090 0x7fefa00b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9f7fe640 1 -- 172.21.15.45:0/588568616 >> v2:172.21.15.190:3300/0 conn(0x7fefa00ae590 msgr2=0x7fefa00b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9f7fe640 1 --2- 172.21.15.45:0/588568616 >> v2:172.21.15.190:3300/0 conn(0x7fefa00ae590 0x7fefa00b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.643+0000 7fef9f7fe640 1 -- 172.21.15.45:0/588568616 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa006d5d0 con 0x7fefa006c060 2023-11-28T21:52:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.644+0000 7fef9f7fe640 1 --2- 172.21.15.45:0/588568616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa006c060 0x7fefa0165250 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fefa006b410 tx=0x7fef9802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93eba47a578829d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.645+0000 7fef9dffb640 1 -- 172.21.15.45:0/588568616 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9802ca70 con 0x7fefa006c060 2023-11-28T21:52:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.645+0000 7fef9dffb640 1 -- 172.21.15.45:0/588568616 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9802cbd0 con 0x7fefa006c060 2023-11-28T21:52:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.645+0000 7fef9dffb640 1 -- 172.21.15.45:0/588568616 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef980343f0 con 0x7fefa006c060 2023-11-28T21:52:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.645+0000 7fef9ffff640 1 -- 172.21.15.45:0/588568616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa006c060 msgr2=0x7fefa0165250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.645+0000 7fef9ffff640 1 --2- 172.21.15.45:0/588568616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa006c060 0x7fefa0165250 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fefa006b410 tx=0x7fef9802bed0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.646+0000 7fef9ffff640 1 -- 172.21.15.45:0/588568616 shutdown_connections 2023-11-28T21:52:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.646+0000 7fef9ffff640 1 --2- 172.21.15.45:0/588568616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa006c060 0x7fefa0165250 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.646+0000 7fef9ffff640 1 --2- 172.21.15.45:0/588568616 >> v2:172.21.15.190:3300/0 conn(0x7fefa00ae590 0x7fefa00b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.646+0000 7fef9ffff640 1 --2- 172.21.15.45:0/588568616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa014b090 0x7fefa00b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.646+0000 7fef9ffff640 1 -- 172.21.15.45:0/588568616 >> 172.21.15.45:0/588568616 conn(0x7fefa006ab30 msgr2=0x7fefa00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.646+0000 7fef9ffff640 1 -- 172.21.15.45:0/588568616 shutdown_connections 2023-11-28T21:52:30.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.646+0000 7fef9ffff640 1 -- 172.21.15.45:0/588568616 wait complete. 2023-11-28T21:52:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.647+0000 7fef9ffff640 1 Processor -- start 2023-11-28T21:52:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.648+0000 7fef9ffff640 1 -- start start 2023-11-28T21:52:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefa006c060 0x7fefa0146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa00ae590 0x7fefa01ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa014b090 0x7fefa0149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9ffff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa01630b0 con 0x7fefa014b090 2023-11-28T21:52:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9ffff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa01ed620 con 0x7fefa006c060 2023-11-28T21:52:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa01ed900 con 0x7fefa00ae590 2023-11-28T21:52:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa014b090 0x7fefa0149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa014b090 0x7fefa0149070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50712/0 (socket says 172.21.15.45:50712) 2023-11-28T21:52:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9f7fe640 1 -- 172.21.15.45:0/1250512596 learned_addr learned my addr 172.21.15.45:0/1250512596 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9effd640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefa006c060 0x7fefa0146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9e7fc640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa00ae590 0x7fefa01ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9f7fe640 1 -- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa00ae590 msgr2=0x7fefa01ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9f7fe640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa00ae590 0x7fefa01ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9f7fe640 1 -- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefa006c060 msgr2=0x7fefa0146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9f7fe640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefa006c060 0x7fefa0146020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.649+0000 7fef9f7fe640 1 -- 172.21.15.45:0/1250512596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef98009b30 con 0x7fefa014b090 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.650+0000 7fef9f7fe640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa014b090 0x7fefa0149070 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fef98036040 tx=0x7fef98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.650+0000 7fef877fe640 1 -- 172.21.15.45:0/1250512596 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef98009510 con 0x7fefa014b090 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.650+0000 7fef877fe640 1 -- 172.21.15.45:0/1250512596 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9803e450 con 0x7fefa014b090 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.650+0000 7fef877fe640 1 -- 172.21.15.45:0/1250512596 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9803e800 con 0x7fefa014b090 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.650+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa01edbe0 con 0x7fefa014b090 2023-11-28T21:52:30.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.650+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa0146560 con 0x7fefa014b090 2023-11-28T21:52:30.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.652+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa0163260 con 0x7fefa014b090 2023-11-28T21:52:30.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.653+0000 7fef877fe640 1 -- 172.21.15.45:0/1250512596 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fef9803e960 con 0x7fefa014b090 2023-11-28T21:52:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.654+0000 7fef877fe640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fef8c06af60 0x7fef8c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.654+0000 7fef877fe640 1 -- 172.21.15.45:0/1250512596 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fef980ac7d0 con 0x7fefa014b090 2023-11-28T21:52:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.654+0000 7fef9effd640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fef8c06af60 0x7fef8c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:30.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.656+0000 7fef9effd640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fef8c06af60 0x7fef8c06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef90006fd0 tx=0x7fef90008040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:30.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.658+0000 7fef877fe640 1 -- 172.21.15.45:0/1250512596 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fefa0163260 con 0x7fefa014b090 2023-11-28T21:52:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.854+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefa01ede20 con 0x7fefa014b090 2023-11-28T21:52:30.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.859+0000 7fef877fe640 1 -- 172.21.15.45:0/1250512596 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+16748 (secure 0 0 0) 0x7fef980774e0 con 0x7fefa014b090 2023-11-28T21:52:30.861 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:30.862 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":199,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T21:52:27.475937+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:52:23.878130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-11-28T21:52:24.956954+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T21:52:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fef8c06af60 msgr2=0x7fef8c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fef8c06af60 0x7fef8c06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef90006fd0 tx=0x7fef90008040 comp rx=0 tx=0).stop 2023-11-28T21:52:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa014b090 msgr2=0x7fefa0149070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa014b090 0x7fefa0149070 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fef98036040 tx=0x7fef98004a40 comp rx=0 tx=0).stop 2023-11-28T21:52:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 shutdown_connections 2023-11-28T21:52:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fef8c06af60 0x7fef8c06d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa014b090 0x7fefa0149070 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fefa00ae590 0x7fefa01ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 --2- 172.21.15.45:0/1250512596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefa006c060 0x7fefa0146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 >> 172.21.15.45:0/1250512596 conn(0x7fefa006ab30 msgr2=0x7fefa00af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 shutdown_connections 2023-11-28T21:52:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:30.862+0000 7fef9ffff640 1 -- 172.21.15.45:0/1250512596 wait complete. 2023-11-28T21:52:30.881 INFO:tasks.ceph.mds.a.smithi045.stderr:2023-11-28T21:52:30.879+0000 7f573707c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T21:52:30.886 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T21:52:30.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T21:52:31.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.095+0000 7f29c51b3640 1 Processor -- start 2023-11-28T21:52:31.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.095+0000 7f29c51b3640 1 -- start start 2023-11-28T21:52:31.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.096+0000 7f29c51b3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f29c00ae540 0x7f29c00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.096+0000 7f29c51b3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 0x7f29c00b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.096+0000 7f29c51b3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c015c3c0 con 0x7f29c0155780 2023-11-28T21:52:31.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.097+0000 7f29c51b3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c00ceac0 con 0x7f29c00ae540 2023-11-28T21:52:31.099 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.097+0000 7f29c51b3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0155b60 con 0x7f29c00ce630 2023-11-28T21:52:31.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.097+0000 7f29c49b2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 0x7f29c00b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.100 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.097+0000 7f29bffff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f29c0155780 0x7f29c00abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:33002/0 (socket says 172.21.15.45:33002) 2023-11-28T21:52:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.098+0000 7f29bffff640 1 -- 172.21.15.45:0/4294403367 learned_addr learned my addr 172.21.15.45:0/4294403367 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.097+0000 7f29bf7fe640 1 --2- 172.21.15.45:0/4294403367 >> v2:172.21.15.190:3300/0 conn(0x7f29c00ae540 0x7f29c00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.098+0000 7f29c49b2640 1 -- 172.21.15.45:0/4294403367 >> v2:172.21.15.190:3300/0 conn(0x7f29c00ae540 msgr2=0x7f29c00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.098+0000 7f29c49b2640 1 --2- 172.21.15.45:0/4294403367 >> v2:172.21.15.190:3300/0 conn(0x7f29c00ae540 0x7f29c00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.098+0000 7f29c49b2640 1 -- 172.21.15.45:0/4294403367 >> v1:172.21.15.190:6789/0 conn(0x7f29c0155780 legacy=0x7f29c00abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.101 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.098+0000 7f29c49b2640 1 -- 172.21.15.45:0/4294403367 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c015ba90 con 0x7f29c00ce630 2023-11-28T21:52:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.100+0000 7f29c49b2640 1 --2- 172.21.15.45:0/4294403367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 0x7f29c00b20a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f29713536e931b9c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:31.102 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.100+0000 7f29beffd640 1 -- 172.21.15.45:0/4294403367 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b802cc50 con 0x7f29c00ce630 2023-11-28T21:52:31.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.100+0000 7f29beffd640 1 -- 172.21.15.45:0/4294403367 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b802cdb0 con 0x7f29c00ce630 2023-11-28T21:52:31.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.100+0000 7f29beffd640 1 -- 172.21.15.45:0/4294403367 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8034660 con 0x7f29c00ce630 2023-11-28T21:52:31.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.101+0000 7f29c51b3640 1 -- 172.21.15.45:0/4294403367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 msgr2=0x7f29c00b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.101+0000 7f29c51b3640 1 --2- 172.21.15.45:0/4294403367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 0x7f29c00b20a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b802bea0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.101+0000 7f29c51b3640 1 -- 172.21.15.45:0/4294403367 shutdown_connections 2023-11-28T21:52:31.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.101+0000 7f29c51b3640 1 --2- 172.21.15.45:0/4294403367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 0x7f29c00b20a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.101+0000 7f29c51b3640 1 --2- 172.21.15.45:0/4294403367 >> v2:172.21.15.190:3300/0 conn(0x7f29c00ae540 0x7f29c00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.101+0000 7f29c51b3640 1 -- 172.21.15.45:0/4294403367 >> 172.21.15.45:0/4294403367 conn(0x7f29c006ab30 msgr2=0x7f29c00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:31.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.101+0000 7f29c51b3640 1 -- 172.21.15.45:0/4294403367 shutdown_connections 2023-11-28T21:52:31.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.101+0000 7f29c51b3640 1 -- 172.21.15.45:0/4294403367 wait complete. 2023-11-28T21:52:31.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.103+0000 7f29c51b3640 1 Processor -- start 2023-11-28T21:52:31.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.103+0000 7f29c51b3640 1 -- start start 2023-11-28T21:52:31.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.104+0000 7f29c51b3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29c00ae540 0x7f29c015d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.104+0000 7f29c51b3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 0x7f29c015d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.104+0000 7f29c51b3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29c0155780 0x7f29c00aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.104+0000 7f29c51b3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c014f130 con 0x7f29c0155780 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.104+0000 7f29c51b3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c00af910 con 0x7f29c00ae540 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.104+0000 7f29c51b3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c013f930 con 0x7f29c00ce630 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29c49b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29c0155780 0x7f29c00aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29c49b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29c0155780 0x7f29c00aebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50718/0 (socket says 172.21.15.45:50718) 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29c49b2640 1 -- 172.21.15.45:0/3677840518 learned_addr learned my addr 172.21.15.45:0/3677840518 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29bffff640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29c00ae540 0x7f29c015d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29bf7fe640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 0x7f29c015d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29c49b2640 1 -- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 msgr2=0x7f29c015d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29c49b2640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 0x7f29c015d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29c49b2640 1 -- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29c00ae540 msgr2=0x7f29c015d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29c49b2640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29c00ae540 0x7f29c015d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29c49b2640 1 -- 172.21.15.45:0/3677840518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b8009a70 con 0x7f29c0155780 2023-11-28T21:52:31.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.105+0000 7f29bffff640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29c00ae540 0x7f29c015d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:31.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.106+0000 7f29c49b2640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29c0155780 0x7f29c00aebc0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009eb0 tx=0x7f29b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:31.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.106+0000 7f29bcff9640 1 -- 172.21.15.45:0/3677840518 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b803e610 con 0x7f29c0155780 2023-11-28T21:52:31.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.106+0000 7f29bcff9640 1 -- 172.21.15.45:0/3677840518 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b803e770 con 0x7f29c0155780 2023-11-28T21:52:31.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.106+0000 7f29bcff9640 1 -- 172.21.15.45:0/3677840518 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b803eb20 con 0x7f29c0155780 2023-11-28T21:52:31.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.106+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c013fc10 con 0x7f29c0155780 2023-11-28T21:52:31.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.106+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c013f160 con 0x7f29c0155780 2023-11-28T21:52:31.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.107+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c013f790 con 0x7f29c0155780 2023-11-28T21:52:31.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.108+0000 7f29bcff9640 1 -- 172.21.15.45:0/3677840518 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f29b803b050 con 0x7f29c0155780 2023-11-28T21:52:31.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.110+0000 7f29bcff9640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f29ac06af90 0x7f29ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.110+0000 7f29bcff9640 1 -- 172.21.15.45:0/3677840518 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f29b80ac1e0 con 0x7f29c0155780 2023-11-28T21:52:31.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.111+0000 7f29bffff640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f29ac06af90 0x7f29ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.112+0000 7f29bffff640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f29ac06af90 0x7f29ac06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f29c014f030 tx=0x7f29b0009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:31.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.114+0000 7f29bcff9640 1 -- 172.21.15.45:0/3677840518 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29c013f790 con 0x7f29c0155780 2023-11-28T21:52:31.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.305+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 --> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29c015b740 con 0x7f29ac06af90 2023-11-28T21:52:31.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.333+0000 7f29bcff9640 1 -- 172.21.15.45:0/3677840518 <== mgr.8101 v2:172.21.15.190:6832/3252186332 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287394 (secure 0 0 0) 0x7f29b0005890 con 0x7f29ac06af90 2023-11-28T21:52:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:31.341 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T21:52:31.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f29ac06af90 msgr2=0x7f29ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f29ac06af90 0x7f29ac06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f29c014f030 tx=0x7f29b0009040 comp rx=0 tx=0).stop 2023-11-28T21:52:31.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29c0155780 msgr2=0x7f29c00aebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29c0155780 0x7f29c00aebc0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009eb0 tx=0x7f29b8004a40 comp rx=0 tx=0).stop 2023-11-28T21:52:31.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 shutdown_connections 2023-11-28T21:52:31.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f29ac06af90 0x7f29ac06d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29c0155780 0x7f29c00aebc0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29c00ce630 0x7f29c015d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 --2- 172.21.15.45:0/3677840518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29c00ae540 0x7f29c015d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 >> 172.21.15.45:0/3677840518 conn(0x7f29c006ab30 msgr2=0x7f29c00b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:31.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 shutdown_connections 2023-11-28T21:52:31.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.340+0000 7f29c51b3640 1 -- 172.21.15.45:0/3677840518 wait complete. 2023-11-28T21:52:31.362 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-11-28T21:52:29.325551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1765,"num_read_kb":3627,"num_write":535,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4838,"ondisk_log_size":4838,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":346412,"kb_used_data":20608,"kb_used_omap":12,"kb_used_meta":325683,"kb_avail":754628308,"statfs":{"total":773094113280,"available":772739387392,"internally_reserved":0,"allocated":21102592,"data_stored":11340044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":333499952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.880314"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486322+0000","last_change":"2023-11-28T21:52:26.510407+0000","last_active":"2023-11-28T21:52:27.486322+0000","last_peered":"2023-11-28T21:52:27.486322+0000","last_clean":"2023-11-28T21:52:27.486322+0000","last_became_active":"2023-11-28T21:52:26.510101+0000","last_became_peered":"2023-11-28T21:52:26.510101+0000","last_unstale":"2023-11-28T21:52:27.486322+0000","last_undegraded":"2023-11-28T21:52:27.486322+0000","last_fullsized":"2023-11-28T21:52:27.486322+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:47:55.083635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184301+0000","last_change":"2023-11-28T21:52:26.513977+0000","last_active":"2023-11-28T21:52:28.184301+0000","last_peered":"2023-11-28T21:52:28.184301+0000","last_clean":"2023-11-28T21:52:28.184301+0000","last_became_active":"2023-11-28T21:52:26.513648+0000","last_became_peered":"2023-11-28T21:52:26.513648+0000","last_unstale":"2023-11-28T21:52:28.184301+0000","last_undegraded":"2023-11-28T21:52:28.184301+0000","last_fullsized":"2023-11-28T21:52:28.184301+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:00:56.918491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487018+0000","last_change":"2023-11-28T21:52:26.505270+0000","last_active":"2023-11-28T21:52:27.487018+0000","last_peered":"2023-11-28T21:52:27.487018+0000","last_clean":"2023-11-28T21:52:27.487018+0000","last_became_active":"2023-11-28T21:52:26.502326+0000","last_became_peered":"2023-11-28T21:52:26.502326+0000","last_unstale":"2023-11-28T21:52:27.487018+0000","last_undegraded":"2023-11-28T21:52:27.487018+0000","last_fullsized":"2023-11-28T21:52:27.487018+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:08:04.148069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487001+0000","last_change":"2023-11-28T21:52:26.505989+0000","last_active":"2023-11-28T21:52:27.487001+0000","last_peered":"2023-11-28T21:52:27.487001+0000","last_clean":"2023-11-28T21:52:27.487001+0000","last_became_active":"2023-11-28T21:52:26.505661+0000","last_became_peered":"2023-11-28T21:52:26.505661+0000","last_unstale":"2023-11-28T21:52:27.487001+0000","last_undegraded":"2023-11-28T21:52:27.487001+0000","last_fullsized":"2023-11-28T21:52:27.487001+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:15.466076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.499977+0000","last_change":"2023-11-28T21:52:26.491413+0000","last_active":"2023-11-28T21:52:26.499977+0000","last_peered":"2023-11-28T21:52:26.499977+0000","last_clean":"2023-11-28T21:52:26.499977+0000","last_became_active":"2023-11-28T21:52:26.490641+0000","last_became_peered":"2023-11-28T21:52:26.490641+0000","last_unstale":"2023-11-28T21:52:26.499977+0000","last_undegraded":"2023-11-28T21:52:26.499977+0000","last_fullsized":"2023-11-28T21:52:26.499977+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:17:13.113631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183596+0000","last_change":"2023-11-28T21:52:26.519572+0000","last_active":"2023-11-28T21:52:28.183596+0000","last_peered":"2023-11-28T21:52:28.183596+0000","last_clean":"2023-11-28T21:52:28.183596+0000","last_became_active":"2023-11-28T21:52:26.519245+0000","last_became_peered":"2023-11-28T21:52:26.519245+0000","last_unstale":"2023-11-28T21:52:28.183596+0000","last_undegraded":"2023-11-28T21:52:28.183596+0000","last_fullsized":"2023-11-28T21:52:28.183596+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:02.275716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485571+0000","last_change":"2023-11-28T21:52:26.519773+0000","last_active":"2023-11-28T21:52:27.485571+0000","last_peered":"2023-11-28T21:52:27.485571+0000","last_clean":"2023-11-28T21:52:27.485571+0000","last_became_active":"2023-11-28T21:52:26.516103+0000","last_became_peered":"2023-11-28T21:52:26.516103+0000","last_unstale":"2023-11-28T21:52:27.485571+0000","last_undegraded":"2023-11-28T21:52:27.485571+0000","last_fullsized":"2023-11-28T21:52:27.485571+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:24:05.710382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.484285+0000","last_change":"2023-11-28T21:52:26.515904+0000","last_active":"2023-11-28T21:52:27.484285+0000","last_peered":"2023-11-28T21:52:27.484285+0000","last_clean":"2023-11-28T21:52:27.484285+0000","last_became_active":"2023-11-28T21:52:26.515601+0000","last_became_peered":"2023-11-28T21:52:26.515601+0000","last_unstale":"2023-11-28T21:52:27.484285+0000","last_undegraded":"2023-11-28T21:52:27.484285+0000","last_fullsized":"2023-11-28T21:52:27.484285+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:48.543300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486160+0000","last_change":"2023-11-28T21:52:26.521039+0000","last_active":"2023-11-28T21:52:27.486160+0000","last_peered":"2023-11-28T21:52:27.486160+0000","last_clean":"2023-11-28T21:52:27.486160+0000","last_became_active":"2023-11-28T21:52:26.520757+0000","last_became_peered":"2023-11-28T21:52:26.520757+0000","last_unstale":"2023-11-28T21:52:27.486160+0000","last_undegraded":"2023-11-28T21:52:27.486160+0000","last_fullsized":"2023-11-28T21:52:27.486160+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:42:15.642917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485596+0000","last_change":"2023-11-28T21:52:26.518812+0000","last_active":"2023-11-28T21:52:27.485596+0000","last_peered":"2023-11-28T21:52:27.485596+0000","last_clean":"2023-11-28T21:52:27.485596+0000","last_became_active":"2023-11-28T21:52:26.518497+0000","last_became_peered":"2023-11-28T21:52:26.518497+0000","last_unstale":"2023-11-28T21:52:27.485596+0000","last_undegraded":"2023-11-28T21:52:27.485596+0000","last_fullsized":"2023-11-28T21:52:27.485596+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:49:08.650335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.510935+0000","last_change":"2023-11-28T21:52:26.509089+0000","last_active":"2023-11-28T21:52:26.510935+0000","last_peered":"2023-11-28T21:52:26.510935+0000","last_clean":"2023-11-28T21:52:26.510935+0000","last_became_active":"2023-11-28T21:52:26.508754+0000","last_became_peered":"2023-11-28T21:52:26.508754+0000","last_unstale":"2023-11-28T21:52:26.510935+0000","last_undegraded":"2023-11-28T21:52:26.510935+0000","last_fullsized":"2023-11-28T21:52:26.510935+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:44:27.485031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184059+0000","last_change":"2023-11-28T21:52:26.520316+0000","last_active":"2023-11-28T21:52:28.184059+0000","last_peered":"2023-11-28T21:52:28.184059+0000","last_clean":"2023-11-28T21:52:28.184059+0000","last_became_active":"2023-11-28T21:52:26.520050+0000","last_became_peered":"2023-11-28T21:52:26.520050+0000","last_unstale":"2023-11-28T21:52:28.184059+0000","last_undegraded":"2023-11-28T21:52:28.184059+0000","last_fullsized":"2023-11-28T21:52:28.184059+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:42:20.326034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485260+0000","last_change":"2023-11-28T21:52:26.521339+0000","last_active":"2023-11-28T21:52:27.485260+0000","last_peered":"2023-11-28T21:52:27.485260+0000","last_clean":"2023-11-28T21:52:27.485260+0000","last_became_active":"2023-11-28T21:52:26.520883+0000","last_became_peered":"2023-11-28T21:52:26.520883+0000","last_unstale":"2023-11-28T21:52:27.485260+0000","last_undegraded":"2023-11-28T21:52:27.485260+0000","last_fullsized":"2023-11-28T21:52:27.485260+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:03:37.991850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.513972+0000","last_change":"2023-11-28T21:52:26.510793+0000","last_active":"2023-11-28T21:52:26.513972+0000","last_peered":"2023-11-28T21:52:26.513972+0000","last_clean":"2023-11-28T21:52:26.513972+0000","last_became_active":"2023-11-28T21:52:26.510199+0000","last_became_peered":"2023-11-28T21:52:26.510199+0000","last_unstale":"2023-11-28T21:52:26.513972+0000","last_undegraded":"2023-11-28T21:52:26.513972+0000","last_fullsized":"2023-11-28T21:52:26.513972+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:58:13.097139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.519293+0000","last_change":"2023-11-28T21:52:26.516342+0000","last_active":"2023-11-28T21:52:26.519293+0000","last_peered":"2023-11-28T21:52:26.519293+0000","last_clean":"2023-11-28T21:52:26.519293+0000","last_became_active":"2023-11-28T21:52:26.515842+0000","last_became_peered":"2023-11-28T21:52:26.515842+0000","last_unstale":"2023-11-28T21:52:26.519293+0000","last_undegraded":"2023-11-28T21:52:26.519293+0000","last_fullsized":"2023-11-28T21:52:26.519293+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:59:07.532923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485713+0000","last_change":"2023-11-28T21:52:26.522202+0000","last_active":"2023-11-28T21:52:27.485713+0000","last_peered":"2023-11-28T21:52:27.485713+0000","last_clean":"2023-11-28T21:52:27.485713+0000","last_became_active":"2023-11-28T21:52:26.521878+0000","last_became_peered":"2023-11-28T21:52:26.521878+0000","last_unstale":"2023-11-28T21:52:27.485713+0000","last_undegraded":"2023-11-28T21:52:27.485713+0000","last_fullsized":"2023-11-28T21:52:27.485713+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:43:47.292224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486528+0000","last_change":"2023-11-28T21:52:26.525215+0000","last_active":"2023-11-28T21:52:27.486528+0000","last_peered":"2023-11-28T21:52:27.486528+0000","last_clean":"2023-11-28T21:52:27.486528+0000","last_became_active":"2023-11-28T21:52:26.524957+0000","last_became_peered":"2023-11-28T21:52:26.524957+0000","last_unstale":"2023-11-28T21:52:27.486528+0000","last_undegraded":"2023-11-28T21:52:27.486528+0000","last_fullsized":"2023-11-28T21:52:27.486528+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:03:05.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207716+0000","last_change":"2023-11-28T21:52:26.517710+0000","last_active":"2023-11-28T21:52:28.207716+0000","last_peered":"2023-11-28T21:52:28.207716+0000","last_clean":"2023-11-28T21:52:28.207716+0000","last_became_active":"2023-11-28T21:52:26.514743+0000","last_became_peered":"2023-11-28T21:52:26.514743+0000","last_unstale":"2023-11-28T21:52:28.207716+0000","last_undegraded":"2023-11-28T21:52:28.207716+0000","last_fullsized":"2023-11-28T21:52:28.207716+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:09:30.212157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.518982+0000","last_change":"2023-11-28T21:52:26.513687+0000","last_active":"2023-11-28T21:52:26.518982+0000","last_peered":"2023-11-28T21:52:26.518982+0000","last_clean":"2023-11-28T21:52:26.518982+0000","last_became_active":"2023-11-28T21:52:26.513289+0000","last_became_peered":"2023-11-28T21:52:26.513289+0000","last_unstale":"2023-11-28T21:52:26.518982+0000","last_undegraded":"2023-11-28T21:52:26.518982+0000","last_fullsized":"2023-11-28T21:52:26.518982+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.382289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.521722+0000","last_change":"2023-11-28T21:52:26.516595+0000","last_active":"2023-11-28T21:52:26.521722+0000","last_peered":"2023-11-28T21:52:26.521722+0000","last_clean":"2023-11-28T21:52:26.521722+0000","last_became_active":"2023-11-28T21:52:26.516297+0000","last_became_peered":"2023-11-28T21:52:26.516297+0000","last_unstale":"2023-11-28T21:52:26.521722+0000","last_undegraded":"2023-11-28T21:52:26.521722+0000","last_fullsized":"2023-11-28T21:52:26.521722+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:31.572761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485394+0000","last_change":"2023-11-28T21:52:26.520600+0000","last_active":"2023-11-28T21:52:27.485394+0000","last_peered":"2023-11-28T21:52:27.485394+0000","last_clean":"2023-11-28T21:52:27.485394+0000","last_became_active":"2023-11-28T21:52:26.519585+0000","last_became_peered":"2023-11-28T21:52:26.519585+0000","last_unstale":"2023-11-28T21:52:27.485394+0000","last_undegraded":"2023-11-28T21:52:27.485394+0000","last_fullsized":"2023-11-28T21:52:27.485394+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:09:14.897540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183745+0000","last_change":"2023-11-28T21:52:26.521889+0000","last_active":"2023-11-28T21:52:28.183745+0000","last_peered":"2023-11-28T21:52:28.183745+0000","last_clean":"2023-11-28T21:52:28.183745+0000","last_became_active":"2023-11-28T21:52:26.521559+0000","last_became_peered":"2023-11-28T21:52:26.521559+0000","last_unstale":"2023-11-28T21:52:28.183745+0000","last_undegraded":"2023-11-28T21:52:28.183745+0000","last_fullsized":"2023-11-28T21:52:28.183745+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:44:01.487785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.517165+0000","last_change":"2023-11-28T21:52:26.514329+0000","last_active":"2023-11-28T21:52:26.517165+0000","last_peered":"2023-11-28T21:52:26.517165+0000","last_clean":"2023-11-28T21:52:26.517165+0000","last_became_active":"2023-11-28T21:52:26.514036+0000","last_became_peered":"2023-11-28T21:52:26.514036+0000","last_unstale":"2023-11-28T21:52:26.517165+0000","last_undegraded":"2023-11-28T21:52:26.517165+0000","last_fullsized":"2023-11-28T21:52:26.517165+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:59.225483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485898+0000","last_change":"2023-11-28T21:52:26.505996+0000","last_active":"2023-11-28T21:52:27.485898+0000","last_peered":"2023-11-28T21:52:27.485898+0000","last_clean":"2023-11-28T21:52:27.485898+0000","last_became_active":"2023-11-28T21:52:26.503830+0000","last_became_peered":"2023-11-28T21:52:26.503830+0000","last_unstale":"2023-11-28T21:52:27.485898+0000","last_undegraded":"2023-11-28T21:52:27.485898+0000","last_fullsized":"2023-11-28T21:52:27.485898+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:59:55.013300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487958+0000","last_change":"2023-11-28T21:52:26.510032+0000","last_active":"2023-11-28T21:52:27.487958+0000","last_peered":"2023-11-28T21:52:27.487958+0000","last_clean":"2023-11-28T21:52:27.487958+0000","last_became_active":"2023-11-28T21:52:26.509760+0000","last_became_peered":"2023-11-28T21:52:26.509760+0000","last_unstale":"2023-11-28T21:52:27.487958+0000","last_undegraded":"2023-11-28T21:52:27.487958+0000","last_fullsized":"2023-11-28T21:52:27.487958+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:08:44.370603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.514605+0000","last_change":"2023-11-28T21:52:26.513177+0000","last_active":"2023-11-28T21:52:26.514605+0000","last_peered":"2023-11-28T21:52:26.514605+0000","last_clean":"2023-11-28T21:52:26.514605+0000","last_became_active":"2023-11-28T21:52:26.512875+0000","last_became_peered":"2023-11-28T21:52:26.512875+0000","last_unstale":"2023-11-28T21:52:26.514605+0000","last_undegraded":"2023-11-28T21:52:26.514605+0000","last_fullsized":"2023-11-28T21:52:26.514605+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:19:04.216039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184190+0000","last_change":"2023-11-28T21:52:26.513597+0000","last_active":"2023-11-28T21:52:28.184190+0000","last_peered":"2023-11-28T21:52:28.184190+0000","last_clean":"2023-11-28T21:52:28.184190+0000","last_became_active":"2023-11-28T21:52:26.512309+0000","last_became_peered":"2023-11-28T21:52:26.512309+0000","last_unstale":"2023-11-28T21:52:28.184190+0000","last_undegraded":"2023-11-28T21:52:28.184190+0000","last_fullsized":"2023-11-28T21:52:28.184190+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:36:37.484249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485650+0000","last_change":"2023-11-28T21:52:26.509363+0000","last_active":"2023-11-28T21:52:27.485650+0000","last_peered":"2023-11-28T21:52:27.485650+0000","last_clean":"2023-11-28T21:52:27.485650+0000","last_became_active":"2023-11-28T21:52:26.508938+0000","last_became_peered":"2023-11-28T21:52:26.508938+0000","last_unstale":"2023-11-28T21:52:27.485650+0000","last_undegraded":"2023-11-28T21:52:27.485650+0000","last_fullsized":"2023-11-28T21:52:27.485650+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:24:00.060172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183780+0000","last_change":"2023-11-28T21:52:26.517758+0000","last_active":"2023-11-28T21:52:28.183780+0000","last_peered":"2023-11-28T21:52:28.183780+0000","last_clean":"2023-11-28T21:52:28.183780+0000","last_became_active":"2023-11-28T21:52:26.516743+0000","last_became_peered":"2023-11-28T21:52:26.516743+0000","last_unstale":"2023-11-28T21:52:28.183780+0000","last_undegraded":"2023-11-28T21:52:28.183780+0000","last_fullsized":"2023-11-28T21:52:28.183780+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:39:34.345308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207626+0000","last_change":"2023-11-28T21:52:26.519251+0000","last_active":"2023-11-28T21:52:28.207626+0000","last_peered":"2023-11-28T21:52:28.207626+0000","last_clean":"2023-11-28T21:52:28.207626+0000","last_became_active":"2023-11-28T21:52:26.518981+0000","last_became_peered":"2023-11-28T21:52:26.518981+0000","last_unstale":"2023-11-28T21:52:28.207626+0000","last_undegraded":"2023-11-28T21:52:28.207626+0000","last_fullsized":"2023-11-28T21:52:28.207626+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:20:19.844626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.519311+0000","last_change":"2023-11-28T21:52:26.517234+0000","last_active":"2023-11-28T21:52:26.519311+0000","last_peered":"2023-11-28T21:52:26.519311+0000","last_clean":"2023-11-28T21:52:26.519311+0000","last_became_active":"2023-11-28T21:52:26.516221+0000","last_became_peered":"2023-11-28T21:52:26.516221+0000","last_unstale":"2023-11-28T21:52:26.519311+0000","last_undegraded":"2023-11-28T21:52:26.519311+0000","last_fullsized":"2023-11-28T21:52:26.519311+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:44.279383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207648+0000","last_change":"2023-11-28T21:52:26.511641+0000","last_active":"2023-11-28T21:52:28.207648+0000","last_peered":"2023-11-28T21:52:28.207648+0000","last_clean":"2023-11-28T21:52:28.207648+0000","last_became_active":"2023-11-28T21:52:26.511217+0000","last_became_peered":"2023-11-28T21:52:26.511217+0000","last_unstale":"2023-11-28T21:52:28.207648+0000","last_undegraded":"2023-11-28T21:52:28.207648+0000","last_fullsized":"2023-11-28T21:52:28.207648+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:47:32.633158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183580+0000","last_change":"2023-11-28T21:52:26.519361+0000","last_active":"2023-11-28T21:52:28.183580+0000","last_peered":"2023-11-28T21:52:28.183580+0000","last_clean":"2023-11-28T21:52:28.183580+0000","last_became_active":"2023-11-28T21:52:26.517188+0000","last_became_peered":"2023-11-28T21:52:26.517188+0000","last_unstale":"2023-11-28T21:52:28.183580+0000","last_undegraded":"2023-11-28T21:52:28.183580+0000","last_fullsized":"2023-11-28T21:52:28.183580+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:20:15.681537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"198'4","reported_seq":16,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486909+0000","last_change":"2023-11-28T21:52:25.488943+0000","last_active":"2023-11-28T21:52:27.486909+0000","last_peered":"2023-11-28T21:52:27.486909+0000","last_clean":"2023-11-28T21:52:27.486909+0000","last_became_active":"2023-11-28T21:52:25.488099+0000","last_became_peered":"2023-11-28T21:52:25.488099+0000","last_unstale":"2023-11-28T21:52:27.486909+0000","last_undegraded":"2023-11-28T21:52:27.486909+0000","last_fullsized":"2023-11-28T21:52:27.486909+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:12:55.202306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.484916+0000","last_change":"2023-11-28T21:52:26.523883+0000","last_active":"2023-11-28T21:52:27.484916+0000","last_peered":"2023-11-28T21:52:27.484916+0000","last_clean":"2023-11-28T21:52:27.484916+0000","last_became_active":"2023-11-28T21:52:26.523520+0000","last_became_peered":"2023-11-28T21:52:26.523520+0000","last_unstale":"2023-11-28T21:52:27.484916+0000","last_undegraded":"2023-11-28T21:52:27.484916+0000","last_fullsized":"2023-11-28T21:52:27.484916+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:11:38.067985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"198'1","reported_seq":12,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.878156+0000","last_change":"2023-11-28T21:52:25.493330+0000","last_active":"2023-11-28T21:52:26.878156+0000","last_peered":"2023-11-28T21:52:26.878156+0000","last_clean":"2023-11-28T21:52:26.878156+0000","last_became_active":"2023-11-28T21:52:25.492500+0000","last_became_peered":"2023-11-28T21:52:25.492500+0000","last_unstale":"2023-11-28T21:52:26.878156+0000","last_undegraded":"2023-11-28T21:52:26.878156+0000","last_fullsized":"2023-11-28T21:52:26.878156+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:35:00.649429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.513950+0000","last_change":"2023-11-28T21:52:26.510550+0000","last_active":"2023-11-28T21:52:26.513950+0000","last_peered":"2023-11-28T21:52:26.513950+0000","last_clean":"2023-11-28T21:52:26.513950+0000","last_became_active":"2023-11-28T21:52:26.509134+0000","last_became_peered":"2023-11-28T21:52:26.509134+0000","last_unstale":"2023-11-28T21:52:26.513950+0000","last_undegraded":"2023-11-28T21:52:26.513950+0000","last_fullsized":"2023-11-28T21:52:26.513950+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:13:58.316790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.878842+0000","last_change":"2023-11-28T21:52:25.481412+0000","last_active":"2023-11-28T21:52:26.878842+0000","last_peered":"2023-11-28T21:52:26.878842+0000","last_clean":"2023-11-28T21:52:26.878842+0000","last_became_active":"2023-11-28T21:52:25.480824+0000","last_became_peered":"2023-11-28T21:52:25.480824+0000","last_unstale":"2023-11-28T21:52:26.878842+0000","last_undegraded":"2023-11-28T21:52:26.878842+0000","last_fullsized":"2023-11-28T21:52:26.878842+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:24:52.055854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183813+0000","last_change":"2023-11-28T21:52:26.512942+0000","last_active":"2023-11-28T21:52:28.183813+0000","last_peered":"2023-11-28T21:52:28.183813+0000","last_clean":"2023-11-28T21:52:28.183813+0000","last_became_active":"2023-11-28T21:52:26.512485+0000","last_became_peered":"2023-11-28T21:52:26.512485+0000","last_unstale":"2023-11-28T21:52:28.183813+0000","last_undegraded":"2023-11-28T21:52:28.183813+0000","last_fullsized":"2023-11-28T21:52:28.183813+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:49:41.492838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.876824+0000","last_change":"2023-11-28T21:52:25.482865+0000","last_active":"2023-11-28T21:52:26.876824+0000","last_peered":"2023-11-28T21:52:26.876824+0000","last_clean":"2023-11-28T21:52:26.876824+0000","last_became_active":"2023-11-28T21:52:25.478880+0000","last_became_peered":"2023-11-28T21:52:25.478880+0000","last_unstale":"2023-11-28T21:52:26.876824+0000","last_undegraded":"2023-11-28T21:52:26.876824+0000","last_fullsized":"2023-11-28T21:52:26.876824+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:00:55.384722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486034+0000","last_change":"2023-11-28T21:52:26.524889+0000","last_active":"2023-11-28T21:52:27.486034+0000","last_peered":"2023-11-28T21:52:27.486034+0000","last_clean":"2023-11-28T21:52:27.486034+0000","last_became_active":"2023-11-28T21:52:26.522092+0000","last_became_peered":"2023-11-28T21:52:26.522092+0000","last_unstale":"2023-11-28T21:52:27.486034+0000","last_undegraded":"2023-11-28T21:52:27.486034+0000","last_fullsized":"2023-11-28T21:52:27.486034+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:19:12.902407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"198'1","reported_seq":13,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485489+0000","last_change":"2023-11-28T21:52:25.491662+0000","last_active":"2023-11-28T21:52:27.485489+0000","last_peered":"2023-11-28T21:52:27.485489+0000","last_clean":"2023-11-28T21:52:27.485489+0000","last_became_active":"2023-11-28T21:52:25.491377+0000","last_became_peered":"2023-11-28T21:52:25.491377+0000","last_unstale":"2023-11-28T21:52:27.485489+0000","last_undegraded":"2023-11-28T21:52:27.485489+0000","last_fullsized":"2023-11-28T21:52:27.485489+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:42.178330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.521340+0000","last_change":"2023-11-28T21:52:26.519897+0000","last_active":"2023-11-28T21:52:26.521340+0000","last_peered":"2023-11-28T21:52:26.521340+0000","last_clean":"2023-11-28T21:52:26.521340+0000","last_became_active":"2023-11-28T21:52:26.519697+0000","last_became_peered":"2023-11-28T21:52:26.519697+0000","last_unstale":"2023-11-28T21:52:26.521340+0000","last_undegraded":"2023-11-28T21:52:26.521340+0000","last_fullsized":"2023-11-28T21:52:26.521340+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:42:00.315518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"198'1","reported_seq":12,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.877508+0000","last_change":"2023-11-28T21:52:25.480618+0000","last_active":"2023-11-28T21:52:26.877508+0000","last_peered":"2023-11-28T21:52:26.877508+0000","last_clean":"2023-11-28T21:52:26.877508+0000","last_became_active":"2023-11-28T21:52:25.480314+0000","last_became_peered":"2023-11-28T21:52:25.480314+0000","last_unstale":"2023-11-28T21:52:26.877508+0000","last_undegraded":"2023-11-28T21:52:26.877508+0000","last_fullsized":"2023-11-28T21:52:26.877508+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:50.458339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.521758+0000","last_change":"2023-11-28T21:52:26.516396+0000","last_active":"2023-11-28T21:52:26.521758+0000","last_peered":"2023-11-28T21:52:26.521758+0000","last_clean":"2023-11-28T21:52:26.521758+0000","last_became_active":"2023-11-28T21:52:26.516076+0000","last_became_peered":"2023-11-28T21:52:26.516076+0000","last_unstale":"2023-11-28T21:52:26.521758+0000","last_undegraded":"2023-11-28T21:52:26.521758+0000","last_fullsized":"2023-11-28T21:52:26.521758+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:23:35.182629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.503487+0000","last_change":"2023-11-28T21:52:25.476455+0000","last_active":"2023-11-28T21:52:26.503487+0000","last_peered":"2023-11-28T21:52:26.503487+0000","last_clean":"2023-11-28T21:52:26.503487+0000","last_became_active":"2023-11-28T21:52:25.474241+0000","last_became_peered":"2023-11-28T21:52:25.474241+0000","last_unstale":"2023-11-28T21:52:26.503487+0000","last_undegraded":"2023-11-28T21:52:26.503487+0000","last_fullsized":"2023-11-28T21:52:26.503487+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:01.742868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486721+0000","last_change":"2023-11-28T21:52:26.510076+0000","last_active":"2023-11-28T21:52:27.486721+0000","last_peered":"2023-11-28T21:52:27.486721+0000","last_clean":"2023-11-28T21:52:27.486721+0000","last_became_active":"2023-11-28T21:52:26.509775+0000","last_became_peered":"2023-11-28T21:52:26.509775+0000","last_unstale":"2023-11-28T21:52:27.486721+0000","last_undegraded":"2023-11-28T21:52:27.486721+0000","last_fullsized":"2023-11-28T21:52:27.486721+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:14:57.033720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.484954+0000","last_change":"2023-11-28T21:52:25.486150+0000","last_active":"2023-11-28T21:52:27.484954+0000","last_peered":"2023-11-28T21:52:27.484954+0000","last_clean":"2023-11-28T21:52:27.484954+0000","last_became_active":"2023-11-28T21:52:25.485521+0000","last_became_peered":"2023-11-28T21:52:25.485521+0000","last_unstale":"2023-11-28T21:52:27.484954+0000","last_undegraded":"2023-11-28T21:52:27.484954+0000","last_fullsized":"2023-11-28T21:52:27.484954+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:16:36.911483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183399+0000","last_change":"2023-11-28T21:52:26.516376+0000","last_active":"2023-11-28T21:52:28.183399+0000","last_peered":"2023-11-28T21:52:28.183399+0000","last_clean":"2023-11-28T21:52:28.183399+0000","last_became_active":"2023-11-28T21:52:26.513230+0000","last_became_peered":"2023-11-28T21:52:26.513230+0000","last_unstale":"2023-11-28T21:52:28.183399+0000","last_undegraded":"2023-11-28T21:52:28.183399+0000","last_fullsized":"2023-11-28T21:52:28.183399+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:59.443868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487856+0000","last_change":"2023-11-28T21:52:25.493028+0000","last_active":"2023-11-28T21:52:27.487856+0000","last_peered":"2023-11-28T21:52:27.487856+0000","last_clean":"2023-11-28T21:52:27.487856+0000","last_became_active":"2023-11-28T21:52:25.492740+0000","last_became_peered":"2023-11-28T21:52:25.492740+0000","last_unstale":"2023-11-28T21:52:27.487856+0000","last_undegraded":"2023-11-28T21:52:27.487856+0000","last_fullsized":"2023-11-28T21:52:27.487856+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:30.234133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485573+0000","last_change":"2023-11-28T21:52:26.512257+0000","last_active":"2023-11-28T21:52:27.485573+0000","last_peered":"2023-11-28T21:52:27.485573+0000","last_clean":"2023-11-28T21:52:27.485573+0000","last_became_active":"2023-11-28T21:52:26.511978+0000","last_became_peered":"2023-11-28T21:52:26.511978+0000","last_unstale":"2023-11-28T21:52:27.485573+0000","last_undegraded":"2023-11-28T21:52:27.485573+0000","last_fullsized":"2023-11-28T21:52:27.485573+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:33:10.943660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"198'3","reported_seq":15,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487146+0000","last_change":"2023-11-28T21:52:25.486002+0000","last_active":"2023-11-28T21:52:27.487146+0000","last_peered":"2023-11-28T21:52:27.487146+0000","last_clean":"2023-11-28T21:52:27.487146+0000","last_became_active":"2023-11-28T21:52:25.485695+0000","last_became_peered":"2023-11-28T21:52:25.485695+0000","last_unstale":"2023-11-28T21:52:27.487146+0000","last_undegraded":"2023-11-28T21:52:27.487146+0000","last_fullsized":"2023-11-28T21:52:27.487146+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:35:12.856054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.514754+0000","last_change":"2023-11-28T21:52:26.506641+0000","last_active":"2023-11-28T21:52:26.514754+0000","last_peered":"2023-11-28T21:52:26.514754+0000","last_clean":"2023-11-28T21:52:26.514754+0000","last_became_active":"2023-11-28T21:52:26.506332+0000","last_became_peered":"2023-11-28T21:52:26.506332+0000","last_unstale":"2023-11-28T21:52:26.514754+0000","last_undegraded":"2023-11-28T21:52:26.514754+0000","last_fullsized":"2023-11-28T21:52:26.514754+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:24.235136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"198'6","reported_seq":18,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485875+0000","last_change":"2023-11-28T21:52:25.484959+0000","last_active":"2023-11-28T21:52:27.485875+0000","last_peered":"2023-11-28T21:52:27.485875+0000","last_clean":"2023-11-28T21:52:27.485875+0000","last_became_active":"2023-11-28T21:52:25.484694+0000","last_became_peered":"2023-11-28T21:52:25.484694+0000","last_unstale":"2023-11-28T21:52:27.485875+0000","last_undegraded":"2023-11-28T21:52:27.485875+0000","last_fullsized":"2023-11-28T21:52:27.485875+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:02.708444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207502+0000","last_change":"2023-11-28T21:52:26.512230+0000","last_active":"2023-11-28T21:52:28.207502+0000","last_peered":"2023-11-28T21:52:28.207502+0000","last_clean":"2023-11-28T21:52:28.207502+0000","last_became_active":"2023-11-28T21:52:26.511946+0000","last_became_peered":"2023-11-28T21:52:26.511946+0000","last_unstale":"2023-11-28T21:52:28.207502+0000","last_undegraded":"2023-11-28T21:52:28.207502+0000","last_fullsized":"2023-11-28T21:52:28.207502+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:29:53.726741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.880802+0000","last_change":"2023-11-28T21:52:25.478593+0000","last_active":"2023-11-28T21:52:26.880802+0000","last_peered":"2023-11-28T21:52:26.880802+0000","last_clean":"2023-11-28T21:52:26.880802+0000","last_became_active":"2023-11-28T21:52:25.478285+0000","last_became_peered":"2023-11-28T21:52:25.478285+0000","last_unstale":"2023-11-28T21:52:26.880802+0000","last_undegraded":"2023-11-28T21:52:26.880802+0000","last_fullsized":"2023-11-28T21:52:26.880802+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:48:00.340285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486885+0000","last_change":"2023-11-28T21:52:26.519286+0000","last_active":"2023-11-28T21:52:27.486885+0000","last_peered":"2023-11-28T21:52:27.486885+0000","last_clean":"2023-11-28T21:52:27.486885+0000","last_became_active":"2023-11-28T21:52:26.519010+0000","last_became_peered":"2023-11-28T21:52:26.519010+0000","last_unstale":"2023-11-28T21:52:27.486885+0000","last_undegraded":"2023-11-28T21:52:27.486885+0000","last_fullsized":"2023-11-28T21:52:27.486885+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:25:05.253795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485878+0000","last_change":"2023-11-28T21:52:25.481104+0000","last_active":"2023-11-28T21:52:27.485878+0000","last_peered":"2023-11-28T21:52:27.485878+0000","last_clean":"2023-11-28T21:52:27.485878+0000","last_became_active":"2023-11-28T21:52:25.480483+0000","last_became_peered":"2023-11-28T21:52:25.480483+0000","last_unstale":"2023-11-28T21:52:27.485878+0000","last_undegraded":"2023-11-28T21:52:27.485878+0000","last_fullsized":"2023-11-28T21:52:27.485878+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:45:13.059071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.519101+0000","last_change":"2023-11-28T21:52:26.515183+0000","last_active":"2023-11-28T21:52:26.519101+0000","last_peered":"2023-11-28T21:52:26.519101+0000","last_clean":"2023-11-28T21:52:26.519101+0000","last_became_active":"2023-11-28T21:52:26.514881+0000","last_became_peered":"2023-11-28T21:52:26.514881+0000","last_unstale":"2023-11-28T21:52:26.519101+0000","last_undegraded":"2023-11-28T21:52:26.519101+0000","last_fullsized":"2023-11-28T21:52:26.519101+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:18:22.456517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.501256+0000","last_change":"2023-11-28T21:52:25.484996+0000","last_active":"2023-11-28T21:52:26.501256+0000","last_peered":"2023-11-28T21:52:26.501256+0000","last_clean":"2023-11-28T21:52:26.501256+0000","last_became_active":"2023-11-28T21:52:25.484488+0000","last_became_peered":"2023-11-28T21:52:25.484488+0000","last_unstale":"2023-11-28T21:52:26.501256+0000","last_undegraded":"2023-11-28T21:52:26.501256+0000","last_fullsized":"2023-11-28T21:52:26.501256+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:43:50.665309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.521043+0000","last_change":"2023-11-28T21:52:26.517778+0000","last_active":"2023-11-28T21:52:26.521043+0000","last_peered":"2023-11-28T21:52:26.521043+0000","last_clean":"2023-11-28T21:52:26.521043+0000","last_became_active":"2023-11-28T21:52:26.513758+0000","last_became_peered":"2023-11-28T21:52:26.513758+0000","last_unstale":"2023-11-28T21:52:26.521043+0000","last_undegraded":"2023-11-28T21:52:26.521043+0000","last_fullsized":"2023-11-28T21:52:26.521043+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:02:45.893909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"198'8","reported_seq":20,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485456+0000","last_change":"2023-11-28T21:52:25.489778+0000","last_active":"2023-11-28T21:52:27.485456+0000","last_peered":"2023-11-28T21:52:27.485456+0000","last_clean":"2023-11-28T21:52:27.485456+0000","last_became_active":"2023-11-28T21:52:25.489502+0000","last_became_peered":"2023-11-28T21:52:25.489502+0000","last_unstale":"2023-11-28T21:52:27.485456+0000","last_undegraded":"2023-11-28T21:52:27.485456+0000","last_fullsized":"2023-11-28T21:52:27.485456+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:28.690571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.484293+0000","last_change":"2023-11-28T21:52:26.511251+0000","last_active":"2023-11-28T21:52:27.484293+0000","last_peered":"2023-11-28T21:52:27.484293+0000","last_clean":"2023-11-28T21:52:27.484293+0000","last_became_active":"2023-11-28T21:52:26.510239+0000","last_became_peered":"2023-11-28T21:52:26.510239+0000","last_unstale":"2023-11-28T21:52:27.484293+0000","last_undegraded":"2023-11-28T21:52:27.484293+0000","last_fullsized":"2023-11-28T21:52:27.484293+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:11:40.364708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"198'1","reported_seq":13,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.488015+0000","last_change":"2023-11-28T21:52:25.488326+0000","last_active":"2023-11-28T21:52:27.488015+0000","last_peered":"2023-11-28T21:52:27.488015+0000","last_clean":"2023-11-28T21:52:27.488015+0000","last_became_active":"2023-11-28T21:52:25.487639+0000","last_became_peered":"2023-11-28T21:52:25.487639+0000","last_unstale":"2023-11-28T21:52:27.488015+0000","last_undegraded":"2023-11-28T21:52:27.488015+0000","last_fullsized":"2023-11-28T21:52:27.488015+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:03.190529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207650+0000","last_change":"2023-11-28T21:52:26.514585+0000","last_active":"2023-11-28T21:52:28.207650+0000","last_peered":"2023-11-28T21:52:28.207650+0000","last_clean":"2023-11-28T21:52:28.207650+0000","last_became_active":"2023-11-28T21:52:26.514227+0000","last_became_peered":"2023-11-28T21:52:26.514227+0000","last_unstale":"2023-11-28T21:52:28.207650+0000","last_undegraded":"2023-11-28T21:52:28.207650+0000","last_fullsized":"2023-11-28T21:52:28.207650+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:45:32.433696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"198'2","reported_seq":14,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486594+0000","last_change":"2023-11-28T21:52:25.489907+0000","last_active":"2023-11-28T21:52:27.486594+0000","last_peered":"2023-11-28T21:52:27.486594+0000","last_clean":"2023-11-28T21:52:27.486594+0000","last_became_active":"2023-11-28T21:52:25.489563+0000","last_became_peered":"2023-11-28T21:52:25.489563+0000","last_unstale":"2023-11-28T21:52:27.486594+0000","last_undegraded":"2023-11-28T21:52:27.486594+0000","last_fullsized":"2023-11-28T21:52:27.486594+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:43.114658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.512403+0000","last_change":"2023-11-28T21:52:26.508055+0000","last_active":"2023-11-28T21:52:26.512403+0000","last_peered":"2023-11-28T21:52:26.512403+0000","last_clean":"2023-11-28T21:52:26.512403+0000","last_became_active":"2023-11-28T21:52:26.507776+0000","last_became_peered":"2023-11-28T21:52:26.507776+0000","last_unstale":"2023-11-28T21:52:26.512403+0000","last_undegraded":"2023-11-28T21:52:26.512403+0000","last_fullsized":"2023-11-28T21:52:26.512403+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:38.846915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"198'1","reported_seq":12,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.877496+0000","last_change":"2023-11-28T21:52:25.487555+0000","last_active":"2023-11-28T21:52:26.877496+0000","last_peered":"2023-11-28T21:52:26.877496+0000","last_clean":"2023-11-28T21:52:26.877496+0000","last_became_active":"2023-11-28T21:52:25.486193+0000","last_became_peered":"2023-11-28T21:52:25.486193+0000","last_unstale":"2023-11-28T21:52:26.877496+0000","last_undegraded":"2023-11-28T21:52:26.877496+0000","last_fullsized":"2023-11-28T21:52:26.877496+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:46.492484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.510688+0000","last_change":"2023-11-28T21:52:26.506531+0000","last_active":"2023-11-28T21:52:26.510688+0000","last_peered":"2023-11-28T21:52:26.510688+0000","last_clean":"2023-11-28T21:52:26.510688+0000","last_became_active":"2023-11-28T21:52:26.506225+0000","last_became_peered":"2023-11-28T21:52:26.506225+0000","last_unstale":"2023-11-28T21:52:26.510688+0000","last_undegraded":"2023-11-28T21:52:26.510688+0000","last_fullsized":"2023-11-28T21:52:26.510688+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:09:23.218622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486386+0000","last_change":"2023-11-28T21:52:25.491913+0000","last_active":"2023-11-28T21:52:27.486386+0000","last_peered":"2023-11-28T21:52:27.486386+0000","last_clean":"2023-11-28T21:52:27.486386+0000","last_became_active":"2023-11-28T21:52:25.491634+0000","last_became_peered":"2023-11-28T21:52:25.491634+0000","last_unstale":"2023-11-28T21:52:27.486386+0000","last_undegraded":"2023-11-28T21:52:27.486386+0000","last_fullsized":"2023-11-28T21:52:27.486386+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:59.931452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485781+0000","last_change":"2023-11-28T21:52:26.509367+0000","last_active":"2023-11-28T21:52:27.485781+0000","last_peered":"2023-11-28T21:52:27.485781+0000","last_clean":"2023-11-28T21:52:27.485781+0000","last_became_active":"2023-11-28T21:52:26.509080+0000","last_became_peered":"2023-11-28T21:52:26.509080+0000","last_unstale":"2023-11-28T21:52:27.485781+0000","last_undegraded":"2023-11-28T21:52:27.485781+0000","last_fullsized":"2023-11-28T21:52:27.485781+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:15:02.553545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.878218+0000","last_change":"2023-11-28T21:52:25.477958+0000","last_active":"2023-11-28T21:52:26.878218+0000","last_peered":"2023-11-28T21:52:26.878218+0000","last_clean":"2023-11-28T21:52:26.878218+0000","last_became_active":"2023-11-28T21:52:25.477643+0000","last_became_peered":"2023-11-28T21:52:25.477643+0000","last_unstale":"2023-11-28T21:52:26.878218+0000","last_undegraded":"2023-11-28T21:52:26.878218+0000","last_fullsized":"2023-11-28T21:52:26.878218+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:01.258764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.523144+0000","last_change":"2023-11-28T21:52:26.517396+0000","last_active":"2023-11-28T21:52:26.523144+0000","last_peered":"2023-11-28T21:52:26.523144+0000","last_clean":"2023-11-28T21:52:26.523144+0000","last_became_active":"2023-11-28T21:52:26.515493+0000","last_became_peered":"2023-11-28T21:52:26.515493+0000","last_unstale":"2023-11-28T21:52:26.523144+0000","last_undegraded":"2023-11-28T21:52:26.523144+0000","last_fullsized":"2023-11-28T21:52:26.523144+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:37:09.751545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.499419+0000","last_change":"2023-11-28T21:52:25.480399+0000","last_active":"2023-11-28T21:52:26.499419+0000","last_peered":"2023-11-28T21:52:26.499419+0000","last_clean":"2023-11-28T21:52:26.499419+0000","last_became_active":"2023-11-28T21:52:25.480075+0000","last_became_peered":"2023-11-28T21:52:25.480075+0000","last_unstale":"2023-11-28T21:52:26.499419+0000","last_undegraded":"2023-11-28T21:52:26.499419+0000","last_fullsized":"2023-11-28T21:52:26.499419+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:50.235463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.516766+0000","last_change":"2023-11-28T21:52:26.513268+0000","last_active":"2023-11-28T21:52:26.516766+0000","last_peered":"2023-11-28T21:52:26.516766+0000","last_clean":"2023-11-28T21:52:26.516766+0000","last_became_active":"2023-11-28T21:52:26.512501+0000","last_became_peered":"2023-11-28T21:52:26.512501+0000","last_unstale":"2023-11-28T21:52:26.516766+0000","last_undegraded":"2023-11-28T21:52:26.516766+0000","last_fullsized":"2023-11-28T21:52:26.516766+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:49:57.772440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.503486+0000","last_change":"2023-11-28T21:52:25.482730+0000","last_active":"2023-11-28T21:52:26.503486+0000","last_peered":"2023-11-28T21:52:26.503486+0000","last_clean":"2023-11-28T21:52:26.503486+0000","last_became_active":"2023-11-28T21:52:25.480128+0000","last_became_peered":"2023-11-28T21:52:25.480128+0000","last_unstale":"2023-11-28T21:52:26.503486+0000","last_undegraded":"2023-11-28T21:52:26.503486+0000","last_fullsized":"2023-11-28T21:52:26.503486+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:45:19.558564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184485+0000","last_change":"2023-11-28T21:52:26.516709+0000","last_active":"2023-11-28T21:52:28.184485+0000","last_peered":"2023-11-28T21:52:28.184485+0000","last_clean":"2023-11-28T21:52:28.184485+0000","last_became_active":"2023-11-28T21:52:26.516415+0000","last_became_peered":"2023-11-28T21:52:26.516415+0000","last_unstale":"2023-11-28T21:52:28.184485+0000","last_undegraded":"2023-11-28T21:52:28.184485+0000","last_fullsized":"2023-11-28T21:52:28.184485+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:03.725396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183689+0000","last_change":"2023-11-28T21:52:25.481985+0000","last_active":"2023-11-28T21:52:28.183689+0000","last_peered":"2023-11-28T21:52:28.183689+0000","last_clean":"2023-11-28T21:52:28.183689+0000","last_became_active":"2023-11-28T21:52:25.478699+0000","last_became_peered":"2023-11-28T21:52:25.478699+0000","last_unstale":"2023-11-28T21:52:28.183689+0000","last_undegraded":"2023-11-28T21:52:28.183689+0000","last_fullsized":"2023-11-28T21:52:28.183689+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:04:08.070617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.519051+0000","last_change":"2023-11-28T21:52:26.515074+0000","last_active":"2023-11-28T21:52:26.519051+0000","last_peered":"2023-11-28T21:52:26.519051+0000","last_clean":"2023-11-28T21:52:26.519051+0000","last_became_active":"2023-11-28T21:52:26.514800+0000","last_became_peered":"2023-11-28T21:52:26.514800+0000","last_unstale":"2023-11-28T21:52:26.519051+0000","last_undegraded":"2023-11-28T21:52:26.519051+0000","last_fullsized":"2023-11-28T21:52:26.519051+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:49:28.580075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.878280+0000","last_change":"2023-11-28T21:52:25.478699+0000","last_active":"2023-11-28T21:52:26.878280+0000","last_peered":"2023-11-28T21:52:26.878280+0000","last_clean":"2023-11-28T21:52:26.878280+0000","last_became_active":"2023-11-28T21:52:25.478402+0000","last_became_peered":"2023-11-28T21:52:25.478402+0000","last_unstale":"2023-11-28T21:52:26.878280+0000","last_undegraded":"2023-11-28T21:52:26.878280+0000","last_fullsized":"2023-11-28T21:52:26.878280+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:04:08.070617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485939+0000","last_change":"2023-11-28T21:52:26.508747+0000","last_active":"2023-11-28T21:52:27.485939+0000","last_peered":"2023-11-28T21:52:27.485939+0000","last_clean":"2023-11-28T21:52:27.485939+0000","last_became_active":"2023-11-28T21:52:26.508377+0000","last_became_peered":"2023-11-28T21:52:26.508377+0000","last_unstale":"2023-11-28T21:52:27.485939+0000","last_undegraded":"2023-11-28T21:52:27.485939+0000","last_fullsized":"2023-11-28T21:52:27.485939+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:46:00.409740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487651+0000","last_change":"2023-11-28T21:52:25.490303+0000","last_active":"2023-11-28T21:52:27.487651+0000","last_peered":"2023-11-28T21:52:27.487651+0000","last_clean":"2023-11-28T21:52:27.487651+0000","last_became_active":"2023-11-28T21:52:25.490022+0000","last_became_peered":"2023-11-28T21:52:25.490022+0000","last_unstale":"2023-11-28T21:52:27.487651+0000","last_undegraded":"2023-11-28T21:52:27.487651+0000","last_fullsized":"2023-11-28T21:52:27.487651+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:51:09.218840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.514884+0000","last_change":"2023-11-28T21:52:26.512789+0000","last_active":"2023-11-28T21:52:26.514884+0000","last_peered":"2023-11-28T21:52:26.514884+0000","last_clean":"2023-11-28T21:52:26.514884+0000","last_became_active":"2023-11-28T21:52:26.512236+0000","last_became_peered":"2023-11-28T21:52:26.512236+0000","last_unstale":"2023-11-28T21:52:26.514884+0000","last_undegraded":"2023-11-28T21:52:26.514884+0000","last_fullsized":"2023-11-28T21:52:26.514884+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:00:58.860332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486159+0000","last_change":"2023-11-28T21:52:25.469187+0000","last_active":"2023-11-28T21:52:27.486159+0000","last_peered":"2023-11-28T21:52:27.486159+0000","last_clean":"2023-11-28T21:52:27.486159+0000","last_became_active":"2023-11-28T21:52:25.468729+0000","last_became_peered":"2023-11-28T21:52:25.468729+0000","last_unstale":"2023-11-28T21:52:27.486159+0000","last_undegraded":"2023-11-28T21:52:27.486159+0000","last_fullsized":"2023-11-28T21:52:27.486159+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:22:19.583099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485172+0000","last_change":"2023-11-28T21:52:26.524747+0000","last_active":"2023-11-28T21:52:27.485172+0000","last_peered":"2023-11-28T21:52:27.485172+0000","last_clean":"2023-11-28T21:52:27.485172+0000","last_became_active":"2023-11-28T21:52:26.516902+0000","last_became_peered":"2023-11-28T21:52:26.516902+0000","last_unstale":"2023-11-28T21:52:27.485172+0000","last_undegraded":"2023-11-28T21:52:27.485172+0000","last_fullsized":"2023-11-28T21:52:27.485172+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:18:06.006228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.505662+0000","last_change":"2023-11-28T21:52:25.483481+0000","last_active":"2023-11-28T21:52:26.505662+0000","last_peered":"2023-11-28T21:52:26.505662+0000","last_clean":"2023-11-28T21:52:26.505662+0000","last_became_active":"2023-11-28T21:52:25.483183+0000","last_became_peered":"2023-11-28T21:52:25.483183+0000","last_unstale":"2023-11-28T21:52:26.505662+0000","last_undegraded":"2023-11-28T21:52:26.505662+0000","last_fullsized":"2023-11-28T21:52:26.505662+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:21:36.169741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487278+0000","last_change":"2023-11-28T21:52:26.493659+0000","last_active":"2023-11-28T21:52:27.487278+0000","last_peered":"2023-11-28T21:52:27.487278+0000","last_clean":"2023-11-28T21:52:27.487278+0000","last_became_active":"2023-11-28T21:52:26.492831+0000","last_became_peered":"2023-11-28T21:52:26.492831+0000","last_unstale":"2023-11-28T21:52:27.487278+0000","last_undegraded":"2023-11-28T21:52:27.487278+0000","last_fullsized":"2023-11-28T21:52:27.487278+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:41:35.808625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486028+0000","last_change":"2023-11-28T21:52:25.491316+0000","last_active":"2023-11-28T21:52:27.486028+0000","last_peered":"2023-11-28T21:52:27.486028+0000","last_clean":"2023-11-28T21:52:27.486028+0000","last_became_active":"2023-11-28T21:52:25.491087+0000","last_became_peered":"2023-11-28T21:52:25.491087+0000","last_unstale":"2023-11-28T21:52:27.486028+0000","last_undegraded":"2023-11-28T21:52:27.486028+0000","last_fullsized":"2023-11-28T21:52:27.486028+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:28.380972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184263+0000","last_change":"2023-11-28T21:52:26.513417+0000","last_active":"2023-11-28T21:52:28.184263+0000","last_peered":"2023-11-28T21:52:28.184263+0000","last_clean":"2023-11-28T21:52:28.184263+0000","last_became_active":"2023-11-28T21:52:26.513142+0000","last_became_peered":"2023-11-28T21:52:26.513142+0000","last_unstale":"2023-11-28T21:52:28.184263+0000","last_undegraded":"2023-11-28T21:52:28.184263+0000","last_fullsized":"2023-11-28T21:52:28.184263+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:56:23.584627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486110+0000","last_change":"2023-11-28T21:52:25.476811+0000","last_active":"2023-11-28T21:52:27.486110+0000","last_peered":"2023-11-28T21:52:27.486110+0000","last_clean":"2023-11-28T21:52:27.486110+0000","last_became_active":"2023-11-28T21:52:25.475907+0000","last_became_peered":"2023-11-28T21:52:25.475907+0000","last_unstale":"2023-11-28T21:52:27.486110+0000","last_undegraded":"2023-11-28T21:52:27.486110+0000","last_fullsized":"2023-11-28T21:52:27.486110+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:59.807724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486386+0000","last_change":"2023-11-28T21:52:26.523095+0000","last_active":"2023-11-28T21:52:27.486386+0000","last_peered":"2023-11-28T21:52:27.486386+0000","last_clean":"2023-11-28T21:52:27.486386+0000","last_became_active":"2023-11-28T21:52:26.522659+0000","last_became_peered":"2023-11-28T21:52:26.522659+0000","last_unstale":"2023-11-28T21:52:27.486386+0000","last_undegraded":"2023-11-28T21:52:27.486386+0000","last_fullsized":"2023-11-28T21:52:27.486386+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:47:36.977837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.501641+0000","last_change":"2023-11-28T21:52:25.480460+0000","last_active":"2023-11-28T21:52:26.501641+0000","last_peered":"2023-11-28T21:52:26.501641+0000","last_clean":"2023-11-28T21:52:26.501641+0000","last_became_active":"2023-11-28T21:52:25.480101+0000","last_became_peered":"2023-11-28T21:52:25.480101+0000","last_unstale":"2023-11-28T21:52:26.501641+0000","last_undegraded":"2023-11-28T21:52:26.501641+0000","last_fullsized":"2023-11-28T21:52:26.501641+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:36:41.581693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.517627+0000","last_change":"2023-11-28T21:52:26.513466+0000","last_active":"2023-11-28T21:52:26.517627+0000","last_peered":"2023-11-28T21:52:26.517627+0000","last_clean":"2023-11-28T21:52:26.517627+0000","last_became_active":"2023-11-28T21:52:26.513162+0000","last_became_peered":"2023-11-28T21:52:26.513162+0000","last_unstale":"2023-11-28T21:52:26.517627+0000","last_undegraded":"2023-11-28T21:52:26.517627+0000","last_fullsized":"2023-11-28T21:52:26.517627+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:34.505548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.481111+0000","last_change":"2023-11-28T21:52:25.472451+0000","last_active":"2023-11-28T21:52:26.481111+0000","last_peered":"2023-11-28T21:52:26.481111+0000","last_clean":"2023-11-28T21:52:26.481111+0000","last_became_active":"2023-11-28T21:52:25.472162+0000","last_became_peered":"2023-11-28T21:52:25.472162+0000","last_unstale":"2023-11-28T21:52:26.481111+0000","last_undegraded":"2023-11-28T21:52:26.481111+0000","last_fullsized":"2023-11-28T21:52:26.481111+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:40:56.289968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183516+0000","last_change":"2023-11-28T21:52:26.516714+0000","last_active":"2023-11-28T21:52:28.183516+0000","last_peered":"2023-11-28T21:52:28.183516+0000","last_clean":"2023-11-28T21:52:28.183516+0000","last_became_active":"2023-11-28T21:52:26.516340+0000","last_became_peered":"2023-11-28T21:52:26.516340+0000","last_unstale":"2023-11-28T21:52:28.183516+0000","last_undegraded":"2023-11-28T21:52:28.183516+0000","last_fullsized":"2023-11-28T21:52:28.183516+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:07:06.713616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487554+0000","last_change":"2023-11-28T21:52:25.489532+0000","last_active":"2023-11-28T21:52:27.487554+0000","last_peered":"2023-11-28T21:52:27.487554+0000","last_clean":"2023-11-28T21:52:27.487554+0000","last_became_active":"2023-11-28T21:52:25.489237+0000","last_became_peered":"2023-11-28T21:52:25.489237+0000","last_unstale":"2023-11-28T21:52:27.487554+0000","last_undegraded":"2023-11-28T21:52:27.487554+0000","last_fullsized":"2023-11-28T21:52:27.487554+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:23:05.185207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"187'4800","reported_seq":5555,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.497586+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T21:52:26.497586+0000","last_peered":"2023-11-28T21:52:26.497586+0000","last_clean":"2023-11-28T21:52:26.497586+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T21:52:26.497586+0000","last_undegraded":"2023-11-28T21:52:26.497586+0000","last_fullsized":"2023-11-28T21:52:26.497586+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4800,"log_dups_size":0,"ondisk_log_size":4800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1765,"num_read_kb":3627,"num_write":491,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1765,"num_read_kb":3627,"num_write":491,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4800,"ondisk_log_size":4800,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575783,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56388,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315452,"statfs":{"total":96636764160,"available":96579022848,"internally_reserved":0,"allocated":3072000,"data_stored":1858017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.24,"15min":1.2110000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.641,"5min":1.865,"15min":1.865},"last":1.524},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.177,"15min":1.2330000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6140000000000001,"5min":1.8140000000000001,"15min":1.954},"last":1.4590000000000001}]},{"osd":1,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1819999999999999,"15min":1.232},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.641,"5min":1.6659999999999999,"15min":2.1880000000000002},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.2809999999999999,"15min":1.28},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.853,"5min":1.853,"15min":2.4279999999999999},"last":1.651}]},{"osd":2,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5229999999999999,"5min":1.754,"15min":1.893},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.349,"15min":1.359},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.5600000000000001,"5min":1.8280000000000001,"15min":2.0339999999999998},"last":1.7050000000000001}]},{"osd":3,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3,"15min":1.3129999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.643,"5min":1.845,"15min":2.1000000000000001},"last":1.008},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4630000000000001,"15min":1.464},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.768,"5min":1.8660000000000001,"15min":2.3769999999999998},"last":1.3480000000000001}]},{"osd":4,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.347,"15min":1.304},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.909},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3300000000000001,"15min":1.341},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.704,"5min":1.895,"15min":1.9470000000000001},"last":1.3839999999999999}]},{"osd":5,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.496,"15min":1.4570000000000001},"min":{"1min":1.099,"5min":0.94899999999999995,"15min":0.79200000000000004},"max":{"1min":2.794,"5min":2.794,"15min":2.794},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.538,"15min":1.5029999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.028,"5min":2.028,"15min":2.246},"last":1.8049999999999999}]},{"osd":6,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.599,"15min":1.5900000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.3029999999999999},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4830000000000001,"15min":1.4990000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.8160000000000001,"5min":1.8859999999999999,"15min":2.2170000000000001},"last":1.2450000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575783,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41920,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329920,"statfs":{"total":96636764160,"available":96593838080,"internally_reserved":0,"allocated":2478080,"data_stored":1267649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.1060000000000001,"15min":1.1259999999999999},"min":{"1min":0.63100000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3420000000000001,"5min":1.597,"15min":1.8740000000000001},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.173,"15min":1.159},"min":{"1min":0.84099999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":2.0840000000000001}]},{"osd":1,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.204,"15min":1.2569999999999999},"min":{"1min":0.72499999999999998,"5min":0.443,"15min":0.443},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.9550000000000001},"last":1.484},{"interface":"front","average":{"1min":1.286,"5min":1.3979999999999999,"15min":1.4279999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.976,"5min":1.976,"15min":2.2109999999999999},"last":1.976}]},{"osd":2,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2729999999999999,"15min":1.274},"min":{"1min":0.72799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.647,"5min":1.8009999999999999,"15min":2.012},"last":1.202},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.405,"15min":1.4199999999999999},"min":{"1min":0.91800000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.804,"5min":1.804,"15min":2.4420000000000002},"last":1.79}]},{"osd":3,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3100000000000001,"15min":1.327},"min":{"1min":0.72199999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0779999999999998},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.304,"15min":1.323},"min":{"1min":0.81899999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.508,"5min":1.8600000000000001,"15min":2.1499999999999999},"last":0.91000000000000003}]},{"osd":4,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.2869999999999999,"15min":1.276},"min":{"1min":0.89800000000000002,"5min":0.80200000000000005,"15min":0.77900000000000003},"max":{"1min":1.873,"5min":1.873,"15min":2.726},"last":1.677},{"interface":"front","average":{"1min":1.494,"5min":1.377,"15min":1.3600000000000001},"min":{"1min":1.0249999999999999,"5min":0.82499999999999996,"15min":0.80100000000000005},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":3.069},"last":1.8400000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.415,"15min":1.399},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.9179999999999999,"5min":2.0529999999999999,"15min":2.319},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.5649999999999999,"15min":1.5800000000000001},"min":{"1min":1.165,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.0289999999999999,"5min":2.0289999999999999,"15min":2.5680000000000001},"last":2.0289999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.55,"15min":1.5409999999999999},"min":{"1min":1.262,"5min":1.0800000000000001,"15min":0.99099999999999999},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.96},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.5349999999999999,"15min":1.528},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.9610000000000001,"5min":1.99,"15min":2.3439999999999999},"last":1.5409999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59976,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311864,"statfs":{"total":96636764160,"available":96575348736,"internally_reserved":0,"allocated":2486272,"data_stored":1267773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.65400000000000003},"max":{"1min":3.4430000000000001,"5min":3.4430000000000001,"15min":3.4430000000000001},"last":3.323},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.204,"15min":1.21},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":4.8070000000000004,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":12.429}]},{"osd":1,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1990000000000001,"15min":1.2190000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.591,"5min":1.724,"15min":1.843},"last":8.2080000000000002},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.2689999999999999,"15min":1.288},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":9.3949999999999996}]},{"osd":2,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3009999999999999,"15min":1.3280000000000001},"min":{"1min":0.74099999999999999,"5min":0.72999999999999998,"15min":0.70199999999999996},"max":{"1min":3.1989999999999998,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":13.605},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.341,"15min":1.3340000000000001},"min":{"1min":0.91200000000000003,"5min":0.876,"15min":0.83699999999999997},"max":{"1min":1.7609999999999999,"5min":1.766,"15min":1.9990000000000001},"last":6.8849999999999998}]},{"osd":3,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.407,"15min":1.391},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.754},"max":{"1min":1.6970000000000001,"5min":1.954,"15min":2.1030000000000002},"last":7.008},{"interface":"front","average":{"1min":1.399,"5min":1.3500000000000001,"15min":1.3340000000000001},"min":{"1min":0.97599999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":0.97899999999999998}]},{"osd":4,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.417,"15min":1.365},"min":{"1min":0.91300000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":3.1400000000000001,"5min":3.1400000000000001,"15min":3.1400000000000001},"last":3.004},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.4990000000000001,"15min":1.4670000000000001},"min":{"1min":1.1859999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":9.4480000000000004}]},{"osd":6,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.403,"15min":1.3859999999999999},"min":{"1min":0.873,"5min":0.752,"15min":0.752},"max":{"1min":1.9710000000000001,"5min":2.0110000000000001,"15min":2.0699999999999998},"last":9.2189999999999994},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.5509999999999999,"15min":1.5429999999999999},"min":{"1min":1.2230000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":1.0780000000000001}]},{"osd":7,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.4590000000000001,"15min":1.4650000000000001},"min":{"1min":1.1799999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":3.157,"5min":3.157,"15min":3.157},"last":3.3889999999999998},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.542,"15min":1.5309999999999999},"min":{"1min":1.208,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.9059999999999999,"5min":1.9790000000000001,"15min":2.2010000000000001},"last":6.9370000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575784,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29460,"kb_used_data":2436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342380,"statfs":{"total":96636764160,"available":96606597120,"internally_reserved":0,"allocated":2494464,"data_stored":1268413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0600000000000001,"15min":1.1259999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.2929999999999999,"5min":1.9379999999999999,"15min":1.9550000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1539999999999999,"15min":1.151},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.647,"5min":1.679,"15min":1.732},"last":0.65500000000000003}]},{"osd":1,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.353,"15min":1.3500000000000001},"min":{"1min":0.92500000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.6830000000000001,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.349,"15min":1.3799999999999999},"min":{"1min":0.95499999999999996,"5min":0.82699999999999996,"15min":0.79900000000000004},"max":{"1min":1.718,"5min":1.8320000000000001,"15min":1.9119999999999999},"last":0.73699999999999999}]},{"osd":2,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.216,"15min":1.258},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.506,"5min":1.788,"15min":1.9570000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2809999999999999,"15min":1.3080000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.5820000000000001,"5min":1.706,"15min":1.905},"last":0.61499999999999999}]},{"osd":3,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.306,"15min":1.2989999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7769999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.407,"15min":1.4119999999999999},"min":{"1min":1.0469999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.923},"last":0.75600000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3280000000000001,"15min":1.302},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.363,"15min":1.3320000000000001},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.79100000000000004},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.48799999999999999}]},{"osd":6,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.476,"15min":1.448},"min":{"1min":1.085,"5min":0.89900000000000002,"15min":0.85399999999999998},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.1099999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.585},"min":{"1min":1.105,"5min":1.105,"15min":0.86499999999999999},"max":{"1min":2.113,"5min":2.113,"15min":2.23},"last":0.67400000000000004}]},{"osd":7,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4890000000000001,"15min":1.482},"min":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":2.056,"5min":2.056,"15min":2.085},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.615,"5min":1.6020000000000001,"15min":1.577},"min":{"1min":1.3089999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.0640000000000001,"5min":2.1640000000000001,"15min":2.2210000000000001},"last":0.63600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575784,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51680,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320160,"statfs":{"total":96636764160,"available":96583843840,"internally_reserved":0,"allocated":2510848,"data_stored":1276513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.99299999999999999,"15min":0.98099999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.2829999999999999,"5min":3.2829999999999999,"15min":3.2829999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.877,"5min":1.0529999999999999,"15min":1.0960000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.296,"5min":1.673,"15min":1.9399999999999999},"last":0.70699999999999996}]},{"osd":1,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.014,"15min":1.0209999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.087,"5min":1.2390000000000001,"15min":1.248},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.7999999999999998,"5min":2.7999999999999998,"15min":2.7999999999999998},"last":0.747}]},{"osd":2,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.2430000000000001,"15min":1.296},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":3.6299999999999999,"5min":3.6299999999999999,"15min":3.6299999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.4199999999999999,"15min":1.423},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":0.82499999999999996}]},{"osd":4,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.3160000000000001,"15min":1.379},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.968,"5min":1.968,"15min":2.246},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.387,"5min":1.431,"15min":1.46},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.9119999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.873}]},{"osd":5,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.5529999999999999,"15min":1.5660000000000001},"min":{"1min":1.004,"5min":1.004,"15min":1.004},"max":{"1min":2.927,"5min":2.927,"15min":2.927},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.468,"15min":1.484},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.121,"5min":2.1699999999999999,"15min":2.3820000000000001},"last":1.371}]},{"osd":6,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.54,"15min":1.5549999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.7410000000000001,"5min":2.7410000000000001,"15min":2.7410000000000001},"last":1.452},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.655,"15min":1.6679999999999999},"min":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"max":{"1min":3.0569999999999999,"5min":3.0569999999999999,"15min":3.0569999999999999},"last":1.296}]},{"osd":7,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.573,"15min":1.6060000000000001},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.5},"last":1.319},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.6599999999999999,"15min":1.7070000000000001},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.8519999999999999,"5min":2.8519999999999999,"15min":2.8519999999999999},"last":1.397}]}]},{"osd":2,"up_from":13,"seq":55834575784,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47492,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324348,"statfs":{"total":96636764160,"available":96588132352,"internally_reserved":0,"allocated":2478080,"data_stored":1267649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":0.97599999999999998,"15min":0.96899999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.006,"5min":1.141,"15min":1.1439999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.6559999999999999,"5min":1.764,"15min":1.8160000000000001},"last":1.3620000000000001}]},{"osd":1,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.98399999999999999,"15min":1.0049999999999999},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0379999999999998},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.222,"15min":1.2430000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.8380000000000001},"last":1.3240000000000001}]},{"osd":3,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.1990000000000001,"15min":1.222},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.7679999999999998,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.292,"5min":1.3109999999999999,"15min":1.329},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.9199999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.827}]},{"osd":4,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.456,"15min":1.4670000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":3.202,"5min":3.202,"15min":3.202},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.498,"15min":1.502},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.6220000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.508,"15min":1.522},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":0.99299999999999999},"max":{"1min":3.1269999999999998,"5min":3.1269999999999998,"15min":3.1269999999999998},"last":1.579},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.5800000000000001,"15min":1.542},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.3149999999999999,"5min":3.3149999999999999,"15min":3.3149999999999999},"last":1.976}]},{"osd":6,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.583,"15min":1.591},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.782},{"interface":"front","average":{"1min":1.661,"5min":1.633,"15min":1.6279999999999999},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.9039999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.621,"15min":1.633},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":4.2990000000000004,"5min":4.2990000000000004,"15min":4.2990000000000004},"last":1.7310000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.679,"15min":1.7},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.8940000000000001,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":1.867}]}]},{"osd":1,"up_from":13,"seq":55834575784,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29464,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342376,"statfs":{"total":96636764160,"available":96606593024,"internally_reserved":0,"allocated":2502656,"data_stored":1275909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.91700000000000004,"15min":0.91800000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.147,"5min":1.127,"15min":1.1240000000000001},"min":{"1min":0.76000000000000001,"5min":0.753,"15min":0.69799999999999995},"max":{"1min":2.9489999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.63600000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1819999999999999,"15min":1.1499999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77400000000000002},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.254,"5min":1.258,"15min":1.2649999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.921,"5min":1.921,"15min":2.0750000000000002},"last":1.2330000000000001}]},{"osd":3,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4039999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.889,"5min":1.9790000000000001,"15min":1.9810000000000001},"last":1.49},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.1910000000000001,"15min":1.1839999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":1.708}]},{"osd":4,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.4279999999999999,"15min":1.415},"min":{"1min":1.1299999999999999,"5min":1.0209999999999999,"15min":0.90900000000000003},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.4570000000000001,"15min":1.4450000000000001},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.84199999999999997},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.0189999999999999}]},{"osd":5,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.482,"15min":1.48},"min":{"1min":1.008,"5min":1.008,"15min":0.97399999999999998},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.129},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.569,"15min":1.5700000000000001},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.0110000000000001,"5min":2.0979999999999999,"15min":2.25},"last":1.591}]},{"osd":6,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.516,"15min":1.5029999999999999},"min":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.0429999999999999},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.518},{"interface":"front","average":{"1min":1.712,"5min":1.605,"15min":1.591},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":3.1520000000000001,"5min":3.1520000000000001,"15min":3.1520000000000001},"last":1.417}]},{"osd":7,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.6439999999999999,"15min":1.6559999999999999},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.2589999999999999,"5min":2.2629999999999999,"15min":2.3439999999999999},"last":1.669},{"interface":"front","average":{"1min":1.734,"5min":1.6299999999999999,"15min":1.607},"min":{"1min":1.258,"5min":1.151,"15min":1.1080000000000001},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.4570000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575784,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30032,"kb_used_data":3008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341808,"statfs":{"total":96636764160,"available":96606011392,"internally_reserved":0,"allocated":3080192,"data_stored":1858121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.92900000000000005,"15min":0.94399999999999995},"min":{"1min":0.48299999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.492,"5min":1.72,"15min":1.724},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.115,"15min":1.1459999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2829999999999999,"5min":2.7879999999999998,"15min":2.7879999999999998},"last":0.52700000000000002}]},{"osd":2,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.113,"15min":1.1339999999999999},"min":{"1min":0.621,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8959999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.1990000000000001,"15min":1.2470000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.5649999999999999,"5min":1.8819999999999999,"15min":2.0409999999999999},"last":0.69099999999999995}]},{"osd":3,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.1040000000000001,"15min":1.1180000000000001},"min":{"1min":0.57499999999999996,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.4650000000000001,"5min":1.7110000000000001,"15min":1.829},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.363,"15min":1.355},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.782,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.4299999999999999,"15min":1.4610000000000001},"min":{"1min":0.94299999999999995,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.7529999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":1.034,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.762,"5min":1.869,"15min":2.0600000000000001},"last":1.323}]},{"osd":5,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.46,"15min":1.4510000000000001},"min":{"1min":0.96499999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.8009999999999999,"5min":2.153,"15min":2.153},"last":1.012},{"interface":"front","average":{"1min":1.534,"5min":1.5029999999999999,"15min":1.536},"min":{"1min":1.159,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.9790000000000001,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.282}]},{"osd":6,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.5820000000000001,"15min":1.6020000000000001},"min":{"1min":1.1379999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.0099999999999998,"5min":2.319,"15min":2.319},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.546,"5min":1.591,"15min":1.6000000000000001},"min":{"1min":1.0149999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.052,"5min":2.73,"15min":2.73},"last":1.3029999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.657,"15min":1.6619999999999999},"min":{"1min":0.90300000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1699999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.5860000000000001,"15min":1.583},"min":{"1min":0.90600000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0649999999999999,"5min":2.371,"15min":2.371},"last":1.228}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T21:52:31.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T21:52:31.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.633+0000 7fd9bf59e640 1 Processor -- start 2023-11-28T21:52:31.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.634+0000 7fd9bf59e640 1 -- start start 2023-11-28T21:52:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.635+0000 7fd9bf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c015d550 0x7fd9c00cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.635+0000 7fd9bf59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c00d0560 con 0x7fd9c00d1c80 2023-11-28T21:52:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.635+0000 7fd9bf59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c00d2230 con 0x7fd9c00d0100 2023-11-28T21:52:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.635+0000 7fd9bf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00d2510 con 0x7fd9c015d550 2023-11-28T21:52:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.635+0000 7fd9bed9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd9c00d1c80 0x7fd9c00b6f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54278/0 (socket says 172.21.15.45:54278) 2023-11-28T21:52:31.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.635+0000 7fd9bed9d640 1 -- 172.21.15.45:0/1631737006 learned_addr learned my addr 172.21.15.45:0/1631737006 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:31.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9be59c640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c015d550 0x7fd9c00cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665578520 0 0) 0x7fd9c00d0560 con 0x7fd9c00d1c80 2023-11-28T21:52:31.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9ac003520 con 0x7fd9c00d1c80 2023-11-28T21:52:31.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861069242 0 0) 0x7fd9ac003520 con 0x7fd9c00d1c80 2023-11-28T21:52:31.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c015d550 msgr2=0x7fd9c00cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:31.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9bdd9b640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c015d550 0x7fd9c00cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 >> v1:172.21.15.190:6789/0 conn(0x7fd9c00d0100 legacy=0x7fd9c00d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c00d27f0 con 0x7fd9c00d1c80 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd9b80019f0 con 0x7fd9c00d1c80 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.636+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 >> v1:172.21.15.45:6789/0 conn(0x7fd9c00d1c80 legacy=0x7fd9c00b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bdd9b640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0039a0 0x7fd9ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bdd9b640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c015d550 0x7fd9ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bdd9b640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac004d00 0x7fd9ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00d2510 con 0x7fd9ac004d00 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac00ba30 con 0x7fd9ac0039a0 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac00bbf0 con 0x7fd9c015d550 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bed9d640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac004d00 0x7fd9ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9b7fff640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c015d550 0x7fd9ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bed9d640 1 -- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c015d550 msgr2=0x7fd9ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:31.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bed9d640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c015d550 0x7fd9ac0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bed9d640 1 -- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0039a0 msgr2=0x7fd9ac003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:31.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9be59c640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0039a0 0x7fd9ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bed9d640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0039a0 0x7fd9ac003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9bed9d640 1 -- 172.21.15.45:0/1631737006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac003520 con 0x7fd9ac004d00 2023-11-28T21:52:31.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.637+0000 7fd9be59c640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac0039a0 0x7fd9ac003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:31.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.638+0000 7fd9bed9d640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac004d00 0x7fd9ac005180 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800a800 tx=0x7fd9ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c57cc225ed87143 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:31.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.639+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b80072d0 con 0x7fd9ac004d00 2023-11-28T21:52:31.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.639+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b802f070 con 0x7fd9ac004d00 2023-11-28T21:52:31.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.639+0000 7fd9bdd9b640 1 -- 172.21.15.45:0/1631737006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b80076b0 con 0x7fd9ac004d00 2023-11-28T21:52:31.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.639+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1631737006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac004d00 msgr2=0x7fd9ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.639+0000 7fd9bf59e640 1 --2- 172.21.15.45:0/1631737006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9ac004d00 0x7fd9ac005180 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800a800 tx=0x7fd9ac0057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.639+0000 7fd9bed9d640 1 -- 172.21.15.45:0/1631737006 reap_dead start 2023-11-28T21:52:31.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.639+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1631737006 shutdown_connections 2023-11-28T21:52:31.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.639+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1631737006 >> 172.21.15.45:0/1631737006 conn(0x7fd9c006ac70 msgr2=0x7fd9c00b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:31.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.640+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1631737006 shutdown_connections 2023-11-28T21:52:31.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.640+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1631737006 wait complete. 2023-11-28T21:52:31.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.641+0000 7fd9bf59e640 1 Processor -- start 2023-11-28T21:52:31.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.641+0000 7fd9bf59e640 1 -- start start 2023-11-28T21:52:31.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.642+0000 7fd9bf59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac004d00 0x7fd9c0159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.642+0000 7fd9bf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c0152210 0x7fd9c0155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.642+0000 7fd9bf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c0153f00 0x7fd9c0156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.642+0000 7fd9bf59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00c8fe0 con 0x7fd9c0152210 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.642+0000 7fd9bf59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c0156950 con 0x7fd9ac004d00 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.642+0000 7fd9bf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00cb7d0 con 0x7fd9c0153f00 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c0152210 0x7fd9c0155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c0152210 0x7fd9c0155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50742/0 (socket says 172.21.15.45:50742) 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9b7fff640 1 -- 172.21.15.45:0/1336930839 learned_addr learned my addr 172.21.15.45:0/1336930839 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9be59c640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac004d00 0x7fd9c0159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9b7fff640 1 -- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c0153f00 msgr2=0x7fd9c0156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9b7fff640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c0153f00 0x7fd9c0156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9b7fff640 1 -- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac004d00 msgr2=0x7fd9c0159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9b7fff640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac004d00 0x7fd9c0159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9b7fff640 1 -- 172.21.15.45:0/1336930839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0154a30 con 0x7fd9c0152210 2023-11-28T21:52:31.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9be59c640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac004d00 0x7fd9c0159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:31.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.643+0000 7fd9b7fff640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c0152210 0x7fd9c0155ed0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800d5a0 tx=0x7fd9a800da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:31.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.644+0000 7fd9b6ffd640 1 -- 172.21.15.45:0/1336930839 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8005480 con 0x7fd9c0152210 2023-11-28T21:52:31.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.644+0000 7fd9b6ffd640 1 -- 172.21.15.45:0/1336930839 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a8003610 con 0x7fd9c0152210 2023-11-28T21:52:31.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.644+0000 7fd9b6ffd640 1 -- 172.21.15.45:0/1336930839 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8003980 con 0x7fd9c0152210 2023-11-28T21:52:31.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.644+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c0154470 con 0x7fd9c0152210 2023-11-28T21:52:31.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.644+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c00cba70 con 0x7fd9c0152210 2023-11-28T21:52:31.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.645+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c00cbc30 con 0x7fd9c0152210 2023-11-28T21:52:31.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.646+0000 7fd9b6ffd640 1 -- 172.21.15.45:0/1336930839 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd9a8003ae0 con 0x7fd9c0152210 2023-11-28T21:52:31.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.647+0000 7fd9b6ffd640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd9a006af90 0x7fd9a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:31.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.647+0000 7fd9b6ffd640 1 -- 172.21.15.45:0/1336930839 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd9a8089760 con 0x7fd9c0152210 2023-11-28T21:52:31.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.648+0000 7fd9be59c640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd9a006af90 0x7fd9a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:31.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.649+0000 7fd9be59c640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd9a006af90 0x7fd9a006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009aa0 tx=0x7fd9b0008040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:31.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.651+0000 7fd9b6ffd640 1 -- 172.21.15.45:0/1336930839 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9a8054490 con 0x7fd9c0152210 2023-11-28T21:52:31.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.841+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 --> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9c015d930 con 0x7fd9a006af90 2023-11-28T21:52:31.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.864+0000 7fd9b6ffd640 1 -- 172.21.15.45:0/1336930839 <== mgr.8101 v2:172.21.15.190:6832/3252186332 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287394 (secure 0 0 0) 0x7fd9b00119a0 con 0x7fd9a006af90 2023-11-28T21:52:31.871 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:31.873 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd9a006af90 msgr2=0x7fd9a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd9a006af90 0x7fd9a006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009aa0 tx=0x7fd9b0008040 comp rx=0 tx=0).stop 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c0152210 msgr2=0x7fd9c0155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c0152210 0x7fd9c0155ed0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800d5a0 tx=0x7fd9a800da60 comp rx=0 tx=0).stop 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 shutdown_connections 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fd9a006af90 0x7fd9a006d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd9c0153f00 0x7fd9c0156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c0152210 0x7fd9c0155ed0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 --2- 172.21.15.45:0/1336930839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9ac004d00 0x7fd9c0159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 >> 172.21.15.45:0/1336930839 conn(0x7fd9c006ac70 msgr2=0x7fd9c015de20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:31.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 shutdown_connections 2023-11-28T21:52:31.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:31.870+0000 7fd9bf59e640 1 -- 172.21.15.45:0/1336930839 wait complete. 2023-11-28T21:52:31.895 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-11-28T21:52:31.332779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1765,"num_read_kb":3627,"num_write":535,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4838,"ondisk_log_size":4838,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":346412,"kb_used_data":20608,"kb_used_omap":12,"kb_used_meta":325683,"kb_avail":754628308,"statfs":{"total":773094113280,"available":772739387392,"internally_reserved":0,"allocated":21102592,"data_stored":11340044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":333499952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.008524"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486322+0000","last_change":"2023-11-28T21:52:26.510407+0000","last_active":"2023-11-28T21:52:27.486322+0000","last_peered":"2023-11-28T21:52:27.486322+0000","last_clean":"2023-11-28T21:52:27.486322+0000","last_became_active":"2023-11-28T21:52:26.510101+0000","last_became_peered":"2023-11-28T21:52:26.510101+0000","last_unstale":"2023-11-28T21:52:27.486322+0000","last_undegraded":"2023-11-28T21:52:27.486322+0000","last_fullsized":"2023-11-28T21:52:27.486322+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:47:55.083635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184301+0000","last_change":"2023-11-28T21:52:26.513977+0000","last_active":"2023-11-28T21:52:28.184301+0000","last_peered":"2023-11-28T21:52:28.184301+0000","last_clean":"2023-11-28T21:52:28.184301+0000","last_became_active":"2023-11-28T21:52:26.513648+0000","last_became_peered":"2023-11-28T21:52:26.513648+0000","last_unstale":"2023-11-28T21:52:28.184301+0000","last_undegraded":"2023-11-28T21:52:28.184301+0000","last_fullsized":"2023-11-28T21:52:28.184301+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:00:56.918491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487018+0000","last_change":"2023-11-28T21:52:26.505270+0000","last_active":"2023-11-28T21:52:27.487018+0000","last_peered":"2023-11-28T21:52:27.487018+0000","last_clean":"2023-11-28T21:52:27.487018+0000","last_became_active":"2023-11-28T21:52:26.502326+0000","last_became_peered":"2023-11-28T21:52:26.502326+0000","last_unstale":"2023-11-28T21:52:27.487018+0000","last_undegraded":"2023-11-28T21:52:27.487018+0000","last_fullsized":"2023-11-28T21:52:27.487018+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:08:04.148069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487001+0000","last_change":"2023-11-28T21:52:26.505989+0000","last_active":"2023-11-28T21:52:27.487001+0000","last_peered":"2023-11-28T21:52:27.487001+0000","last_clean":"2023-11-28T21:52:27.487001+0000","last_became_active":"2023-11-28T21:52:26.505661+0000","last_became_peered":"2023-11-28T21:52:26.505661+0000","last_unstale":"2023-11-28T21:52:27.487001+0000","last_undegraded":"2023-11-28T21:52:27.487001+0000","last_fullsized":"2023-11-28T21:52:27.487001+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:10:15.466076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.499977+0000","last_change":"2023-11-28T21:52:26.491413+0000","last_active":"2023-11-28T21:52:26.499977+0000","last_peered":"2023-11-28T21:52:26.499977+0000","last_clean":"2023-11-28T21:52:26.499977+0000","last_became_active":"2023-11-28T21:52:26.490641+0000","last_became_peered":"2023-11-28T21:52:26.490641+0000","last_unstale":"2023-11-28T21:52:26.499977+0000","last_undegraded":"2023-11-28T21:52:26.499977+0000","last_fullsized":"2023-11-28T21:52:26.499977+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:17:13.113631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183596+0000","last_change":"2023-11-28T21:52:26.519572+0000","last_active":"2023-11-28T21:52:28.183596+0000","last_peered":"2023-11-28T21:52:28.183596+0000","last_clean":"2023-11-28T21:52:28.183596+0000","last_became_active":"2023-11-28T21:52:26.519245+0000","last_became_peered":"2023-11-28T21:52:26.519245+0000","last_unstale":"2023-11-28T21:52:28.183596+0000","last_undegraded":"2023-11-28T21:52:28.183596+0000","last_fullsized":"2023-11-28T21:52:28.183596+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:02.275716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485571+0000","last_change":"2023-11-28T21:52:26.519773+0000","last_active":"2023-11-28T21:52:27.485571+0000","last_peered":"2023-11-28T21:52:27.485571+0000","last_clean":"2023-11-28T21:52:27.485571+0000","last_became_active":"2023-11-28T21:52:26.516103+0000","last_became_peered":"2023-11-28T21:52:26.516103+0000","last_unstale":"2023-11-28T21:52:27.485571+0000","last_undegraded":"2023-11-28T21:52:27.485571+0000","last_fullsized":"2023-11-28T21:52:27.485571+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:24:05.710382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.484285+0000","last_change":"2023-11-28T21:52:26.515904+0000","last_active":"2023-11-28T21:52:27.484285+0000","last_peered":"2023-11-28T21:52:27.484285+0000","last_clean":"2023-11-28T21:52:27.484285+0000","last_became_active":"2023-11-28T21:52:26.515601+0000","last_became_peered":"2023-11-28T21:52:26.515601+0000","last_unstale":"2023-11-28T21:52:27.484285+0000","last_undegraded":"2023-11-28T21:52:27.484285+0000","last_fullsized":"2023-11-28T21:52:27.484285+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:48.543300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486160+0000","last_change":"2023-11-28T21:52:26.521039+0000","last_active":"2023-11-28T21:52:27.486160+0000","last_peered":"2023-11-28T21:52:27.486160+0000","last_clean":"2023-11-28T21:52:27.486160+0000","last_became_active":"2023-11-28T21:52:26.520757+0000","last_became_peered":"2023-11-28T21:52:26.520757+0000","last_unstale":"2023-11-28T21:52:27.486160+0000","last_undegraded":"2023-11-28T21:52:27.486160+0000","last_fullsized":"2023-11-28T21:52:27.486160+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:42:15.642917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485596+0000","last_change":"2023-11-28T21:52:26.518812+0000","last_active":"2023-11-28T21:52:27.485596+0000","last_peered":"2023-11-28T21:52:27.485596+0000","last_clean":"2023-11-28T21:52:27.485596+0000","last_became_active":"2023-11-28T21:52:26.518497+0000","last_became_peered":"2023-11-28T21:52:26.518497+0000","last_unstale":"2023-11-28T21:52:27.485596+0000","last_undegraded":"2023-11-28T21:52:27.485596+0000","last_fullsized":"2023-11-28T21:52:27.485596+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:49:08.650335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.510935+0000","last_change":"2023-11-28T21:52:26.509089+0000","last_active":"2023-11-28T21:52:26.510935+0000","last_peered":"2023-11-28T21:52:26.510935+0000","last_clean":"2023-11-28T21:52:26.510935+0000","last_became_active":"2023-11-28T21:52:26.508754+0000","last_became_peered":"2023-11-28T21:52:26.508754+0000","last_unstale":"2023-11-28T21:52:26.510935+0000","last_undegraded":"2023-11-28T21:52:26.510935+0000","last_fullsized":"2023-11-28T21:52:26.510935+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:44:27.485031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184059+0000","last_change":"2023-11-28T21:52:26.520316+0000","last_active":"2023-11-28T21:52:28.184059+0000","last_peered":"2023-11-28T21:52:28.184059+0000","last_clean":"2023-11-28T21:52:28.184059+0000","last_became_active":"2023-11-28T21:52:26.520050+0000","last_became_peered":"2023-11-28T21:52:26.520050+0000","last_unstale":"2023-11-28T21:52:28.184059+0000","last_undegraded":"2023-11-28T21:52:28.184059+0000","last_fullsized":"2023-11-28T21:52:28.184059+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:42:20.326034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485260+0000","last_change":"2023-11-28T21:52:26.521339+0000","last_active":"2023-11-28T21:52:27.485260+0000","last_peered":"2023-11-28T21:52:27.485260+0000","last_clean":"2023-11-28T21:52:27.485260+0000","last_became_active":"2023-11-28T21:52:26.520883+0000","last_became_peered":"2023-11-28T21:52:26.520883+0000","last_unstale":"2023-11-28T21:52:27.485260+0000","last_undegraded":"2023-11-28T21:52:27.485260+0000","last_fullsized":"2023-11-28T21:52:27.485260+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:03:37.991850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.513972+0000","last_change":"2023-11-28T21:52:26.510793+0000","last_active":"2023-11-28T21:52:26.513972+0000","last_peered":"2023-11-28T21:52:26.513972+0000","last_clean":"2023-11-28T21:52:26.513972+0000","last_became_active":"2023-11-28T21:52:26.510199+0000","last_became_peered":"2023-11-28T21:52:26.510199+0000","last_unstale":"2023-11-28T21:52:26.513972+0000","last_undegraded":"2023-11-28T21:52:26.513972+0000","last_fullsized":"2023-11-28T21:52:26.513972+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:58:13.097139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.519293+0000","last_change":"2023-11-28T21:52:26.516342+0000","last_active":"2023-11-28T21:52:26.519293+0000","last_peered":"2023-11-28T21:52:26.519293+0000","last_clean":"2023-11-28T21:52:26.519293+0000","last_became_active":"2023-11-28T21:52:26.515842+0000","last_became_peered":"2023-11-28T21:52:26.515842+0000","last_unstale":"2023-11-28T21:52:26.519293+0000","last_undegraded":"2023-11-28T21:52:26.519293+0000","last_fullsized":"2023-11-28T21:52:26.519293+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:59:07.532923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485713+0000","last_change":"2023-11-28T21:52:26.522202+0000","last_active":"2023-11-28T21:52:27.485713+0000","last_peered":"2023-11-28T21:52:27.485713+0000","last_clean":"2023-11-28T21:52:27.485713+0000","last_became_active":"2023-11-28T21:52:26.521878+0000","last_became_peered":"2023-11-28T21:52:26.521878+0000","last_unstale":"2023-11-28T21:52:27.485713+0000","last_undegraded":"2023-11-28T21:52:27.485713+0000","last_fullsized":"2023-11-28T21:52:27.485713+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:43:47.292224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486528+0000","last_change":"2023-11-28T21:52:26.525215+0000","last_active":"2023-11-28T21:52:27.486528+0000","last_peered":"2023-11-28T21:52:27.486528+0000","last_clean":"2023-11-28T21:52:27.486528+0000","last_became_active":"2023-11-28T21:52:26.524957+0000","last_became_peered":"2023-11-28T21:52:26.524957+0000","last_unstale":"2023-11-28T21:52:27.486528+0000","last_undegraded":"2023-11-28T21:52:27.486528+0000","last_fullsized":"2023-11-28T21:52:27.486528+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:03:05.180763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207716+0000","last_change":"2023-11-28T21:52:26.517710+0000","last_active":"2023-11-28T21:52:28.207716+0000","last_peered":"2023-11-28T21:52:28.207716+0000","last_clean":"2023-11-28T21:52:28.207716+0000","last_became_active":"2023-11-28T21:52:26.514743+0000","last_became_peered":"2023-11-28T21:52:26.514743+0000","last_unstale":"2023-11-28T21:52:28.207716+0000","last_undegraded":"2023-11-28T21:52:28.207716+0000","last_fullsized":"2023-11-28T21:52:28.207716+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:09:30.212157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.518982+0000","last_change":"2023-11-28T21:52:26.513687+0000","last_active":"2023-11-28T21:52:26.518982+0000","last_peered":"2023-11-28T21:52:26.518982+0000","last_clean":"2023-11-28T21:52:26.518982+0000","last_became_active":"2023-11-28T21:52:26.513289+0000","last_became_peered":"2023-11-28T21:52:26.513289+0000","last_unstale":"2023-11-28T21:52:26.518982+0000","last_undegraded":"2023-11-28T21:52:26.518982+0000","last_fullsized":"2023-11-28T21:52:26.518982+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.382289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.521722+0000","last_change":"2023-11-28T21:52:26.516595+0000","last_active":"2023-11-28T21:52:26.521722+0000","last_peered":"2023-11-28T21:52:26.521722+0000","last_clean":"2023-11-28T21:52:26.521722+0000","last_became_active":"2023-11-28T21:52:26.516297+0000","last_became_peered":"2023-11-28T21:52:26.516297+0000","last_unstale":"2023-11-28T21:52:26.521722+0000","last_undegraded":"2023-11-28T21:52:26.521722+0000","last_fullsized":"2023-11-28T21:52:26.521722+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:31.572761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485394+0000","last_change":"2023-11-28T21:52:26.520600+0000","last_active":"2023-11-28T21:52:27.485394+0000","last_peered":"2023-11-28T21:52:27.485394+0000","last_clean":"2023-11-28T21:52:27.485394+0000","last_became_active":"2023-11-28T21:52:26.519585+0000","last_became_peered":"2023-11-28T21:52:26.519585+0000","last_unstale":"2023-11-28T21:52:27.485394+0000","last_undegraded":"2023-11-28T21:52:27.485394+0000","last_fullsized":"2023-11-28T21:52:27.485394+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:09:14.897540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183745+0000","last_change":"2023-11-28T21:52:26.521889+0000","last_active":"2023-11-28T21:52:28.183745+0000","last_peered":"2023-11-28T21:52:28.183745+0000","last_clean":"2023-11-28T21:52:28.183745+0000","last_became_active":"2023-11-28T21:52:26.521559+0000","last_became_peered":"2023-11-28T21:52:26.521559+0000","last_unstale":"2023-11-28T21:52:28.183745+0000","last_undegraded":"2023-11-28T21:52:28.183745+0000","last_fullsized":"2023-11-28T21:52:28.183745+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:44:01.487785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.517165+0000","last_change":"2023-11-28T21:52:26.514329+0000","last_active":"2023-11-28T21:52:26.517165+0000","last_peered":"2023-11-28T21:52:26.517165+0000","last_clean":"2023-11-28T21:52:26.517165+0000","last_became_active":"2023-11-28T21:52:26.514036+0000","last_became_peered":"2023-11-28T21:52:26.514036+0000","last_unstale":"2023-11-28T21:52:26.517165+0000","last_undegraded":"2023-11-28T21:52:26.517165+0000","last_fullsized":"2023-11-28T21:52:26.517165+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:59.225483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485898+0000","last_change":"2023-11-28T21:52:26.505996+0000","last_active":"2023-11-28T21:52:27.485898+0000","last_peered":"2023-11-28T21:52:27.485898+0000","last_clean":"2023-11-28T21:52:27.485898+0000","last_became_active":"2023-11-28T21:52:26.503830+0000","last_became_peered":"2023-11-28T21:52:26.503830+0000","last_unstale":"2023-11-28T21:52:27.485898+0000","last_undegraded":"2023-11-28T21:52:27.485898+0000","last_fullsized":"2023-11-28T21:52:27.485898+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:59:55.013300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487958+0000","last_change":"2023-11-28T21:52:26.510032+0000","last_active":"2023-11-28T21:52:27.487958+0000","last_peered":"2023-11-28T21:52:27.487958+0000","last_clean":"2023-11-28T21:52:27.487958+0000","last_became_active":"2023-11-28T21:52:26.509760+0000","last_became_peered":"2023-11-28T21:52:26.509760+0000","last_unstale":"2023-11-28T21:52:27.487958+0000","last_undegraded":"2023-11-28T21:52:27.487958+0000","last_fullsized":"2023-11-28T21:52:27.487958+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:08:44.370603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.514605+0000","last_change":"2023-11-28T21:52:26.513177+0000","last_active":"2023-11-28T21:52:26.514605+0000","last_peered":"2023-11-28T21:52:26.514605+0000","last_clean":"2023-11-28T21:52:26.514605+0000","last_became_active":"2023-11-28T21:52:26.512875+0000","last_became_peered":"2023-11-28T21:52:26.512875+0000","last_unstale":"2023-11-28T21:52:26.514605+0000","last_undegraded":"2023-11-28T21:52:26.514605+0000","last_fullsized":"2023-11-28T21:52:26.514605+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:19:04.216039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184190+0000","last_change":"2023-11-28T21:52:26.513597+0000","last_active":"2023-11-28T21:52:28.184190+0000","last_peered":"2023-11-28T21:52:28.184190+0000","last_clean":"2023-11-28T21:52:28.184190+0000","last_became_active":"2023-11-28T21:52:26.512309+0000","last_became_peered":"2023-11-28T21:52:26.512309+0000","last_unstale":"2023-11-28T21:52:28.184190+0000","last_undegraded":"2023-11-28T21:52:28.184190+0000","last_fullsized":"2023-11-28T21:52:28.184190+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:36:37.484249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485650+0000","last_change":"2023-11-28T21:52:26.509363+0000","last_active":"2023-11-28T21:52:27.485650+0000","last_peered":"2023-11-28T21:52:27.485650+0000","last_clean":"2023-11-28T21:52:27.485650+0000","last_became_active":"2023-11-28T21:52:26.508938+0000","last_became_peered":"2023-11-28T21:52:26.508938+0000","last_unstale":"2023-11-28T21:52:27.485650+0000","last_undegraded":"2023-11-28T21:52:27.485650+0000","last_fullsized":"2023-11-28T21:52:27.485650+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:24:00.060172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183780+0000","last_change":"2023-11-28T21:52:26.517758+0000","last_active":"2023-11-28T21:52:28.183780+0000","last_peered":"2023-11-28T21:52:28.183780+0000","last_clean":"2023-11-28T21:52:28.183780+0000","last_became_active":"2023-11-28T21:52:26.516743+0000","last_became_peered":"2023-11-28T21:52:26.516743+0000","last_unstale":"2023-11-28T21:52:28.183780+0000","last_undegraded":"2023-11-28T21:52:28.183780+0000","last_fullsized":"2023-11-28T21:52:28.183780+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:39:34.345308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207626+0000","last_change":"2023-11-28T21:52:26.519251+0000","last_active":"2023-11-28T21:52:28.207626+0000","last_peered":"2023-11-28T21:52:28.207626+0000","last_clean":"2023-11-28T21:52:28.207626+0000","last_became_active":"2023-11-28T21:52:26.518981+0000","last_became_peered":"2023-11-28T21:52:26.518981+0000","last_unstale":"2023-11-28T21:52:28.207626+0000","last_undegraded":"2023-11-28T21:52:28.207626+0000","last_fullsized":"2023-11-28T21:52:28.207626+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:20:19.844626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.519311+0000","last_change":"2023-11-28T21:52:26.517234+0000","last_active":"2023-11-28T21:52:26.519311+0000","last_peered":"2023-11-28T21:52:26.519311+0000","last_clean":"2023-11-28T21:52:26.519311+0000","last_became_active":"2023-11-28T21:52:26.516221+0000","last_became_peered":"2023-11-28T21:52:26.516221+0000","last_unstale":"2023-11-28T21:52:26.519311+0000","last_undegraded":"2023-11-28T21:52:26.519311+0000","last_fullsized":"2023-11-28T21:52:26.519311+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:44.279383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207648+0000","last_change":"2023-11-28T21:52:26.511641+0000","last_active":"2023-11-28T21:52:28.207648+0000","last_peered":"2023-11-28T21:52:28.207648+0000","last_clean":"2023-11-28T21:52:28.207648+0000","last_became_active":"2023-11-28T21:52:26.511217+0000","last_became_peered":"2023-11-28T21:52:26.511217+0000","last_unstale":"2023-11-28T21:52:28.207648+0000","last_undegraded":"2023-11-28T21:52:28.207648+0000","last_fullsized":"2023-11-28T21:52:28.207648+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:47:32.633158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183580+0000","last_change":"2023-11-28T21:52:26.519361+0000","last_active":"2023-11-28T21:52:28.183580+0000","last_peered":"2023-11-28T21:52:28.183580+0000","last_clean":"2023-11-28T21:52:28.183580+0000","last_became_active":"2023-11-28T21:52:26.517188+0000","last_became_peered":"2023-11-28T21:52:26.517188+0000","last_unstale":"2023-11-28T21:52:28.183580+0000","last_undegraded":"2023-11-28T21:52:28.183580+0000","last_fullsized":"2023-11-28T21:52:28.183580+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:20:15.681537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"198'4","reported_seq":16,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486909+0000","last_change":"2023-11-28T21:52:25.488943+0000","last_active":"2023-11-28T21:52:27.486909+0000","last_peered":"2023-11-28T21:52:27.486909+0000","last_clean":"2023-11-28T21:52:27.486909+0000","last_became_active":"2023-11-28T21:52:25.488099+0000","last_became_peered":"2023-11-28T21:52:25.488099+0000","last_unstale":"2023-11-28T21:52:27.486909+0000","last_undegraded":"2023-11-28T21:52:27.486909+0000","last_fullsized":"2023-11-28T21:52:27.486909+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:12:55.202306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.484916+0000","last_change":"2023-11-28T21:52:26.523883+0000","last_active":"2023-11-28T21:52:27.484916+0000","last_peered":"2023-11-28T21:52:27.484916+0000","last_clean":"2023-11-28T21:52:27.484916+0000","last_became_active":"2023-11-28T21:52:26.523520+0000","last_became_peered":"2023-11-28T21:52:26.523520+0000","last_unstale":"2023-11-28T21:52:27.484916+0000","last_undegraded":"2023-11-28T21:52:27.484916+0000","last_fullsized":"2023-11-28T21:52:27.484916+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:11:38.067985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"198'1","reported_seq":12,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.878156+0000","last_change":"2023-11-28T21:52:25.493330+0000","last_active":"2023-11-28T21:52:26.878156+0000","last_peered":"2023-11-28T21:52:26.878156+0000","last_clean":"2023-11-28T21:52:26.878156+0000","last_became_active":"2023-11-28T21:52:25.492500+0000","last_became_peered":"2023-11-28T21:52:25.492500+0000","last_unstale":"2023-11-28T21:52:26.878156+0000","last_undegraded":"2023-11-28T21:52:26.878156+0000","last_fullsized":"2023-11-28T21:52:26.878156+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:35:00.649429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.513950+0000","last_change":"2023-11-28T21:52:26.510550+0000","last_active":"2023-11-28T21:52:26.513950+0000","last_peered":"2023-11-28T21:52:26.513950+0000","last_clean":"2023-11-28T21:52:26.513950+0000","last_became_active":"2023-11-28T21:52:26.509134+0000","last_became_peered":"2023-11-28T21:52:26.509134+0000","last_unstale":"2023-11-28T21:52:26.513950+0000","last_undegraded":"2023-11-28T21:52:26.513950+0000","last_fullsized":"2023-11-28T21:52:26.513950+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:13:58.316790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.878842+0000","last_change":"2023-11-28T21:52:25.481412+0000","last_active":"2023-11-28T21:52:26.878842+0000","last_peered":"2023-11-28T21:52:26.878842+0000","last_clean":"2023-11-28T21:52:26.878842+0000","last_became_active":"2023-11-28T21:52:25.480824+0000","last_became_peered":"2023-11-28T21:52:25.480824+0000","last_unstale":"2023-11-28T21:52:26.878842+0000","last_undegraded":"2023-11-28T21:52:26.878842+0000","last_fullsized":"2023-11-28T21:52:26.878842+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:24:52.055854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183813+0000","last_change":"2023-11-28T21:52:26.512942+0000","last_active":"2023-11-28T21:52:28.183813+0000","last_peered":"2023-11-28T21:52:28.183813+0000","last_clean":"2023-11-28T21:52:28.183813+0000","last_became_active":"2023-11-28T21:52:26.512485+0000","last_became_peered":"2023-11-28T21:52:26.512485+0000","last_unstale":"2023-11-28T21:52:28.183813+0000","last_undegraded":"2023-11-28T21:52:28.183813+0000","last_fullsized":"2023-11-28T21:52:28.183813+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:49:41.492838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.876824+0000","last_change":"2023-11-28T21:52:25.482865+0000","last_active":"2023-11-28T21:52:26.876824+0000","last_peered":"2023-11-28T21:52:26.876824+0000","last_clean":"2023-11-28T21:52:26.876824+0000","last_became_active":"2023-11-28T21:52:25.478880+0000","last_became_peered":"2023-11-28T21:52:25.478880+0000","last_unstale":"2023-11-28T21:52:26.876824+0000","last_undegraded":"2023-11-28T21:52:26.876824+0000","last_fullsized":"2023-11-28T21:52:26.876824+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:00:55.384722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486034+0000","last_change":"2023-11-28T21:52:26.524889+0000","last_active":"2023-11-28T21:52:27.486034+0000","last_peered":"2023-11-28T21:52:27.486034+0000","last_clean":"2023-11-28T21:52:27.486034+0000","last_became_active":"2023-11-28T21:52:26.522092+0000","last_became_peered":"2023-11-28T21:52:26.522092+0000","last_unstale":"2023-11-28T21:52:27.486034+0000","last_undegraded":"2023-11-28T21:52:27.486034+0000","last_fullsized":"2023-11-28T21:52:27.486034+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:19:12.902407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"198'1","reported_seq":13,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485489+0000","last_change":"2023-11-28T21:52:25.491662+0000","last_active":"2023-11-28T21:52:27.485489+0000","last_peered":"2023-11-28T21:52:27.485489+0000","last_clean":"2023-11-28T21:52:27.485489+0000","last_became_active":"2023-11-28T21:52:25.491377+0000","last_became_peered":"2023-11-28T21:52:25.491377+0000","last_unstale":"2023-11-28T21:52:27.485489+0000","last_undegraded":"2023-11-28T21:52:27.485489+0000","last_fullsized":"2023-11-28T21:52:27.485489+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:42.178330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.521340+0000","last_change":"2023-11-28T21:52:26.519897+0000","last_active":"2023-11-28T21:52:26.521340+0000","last_peered":"2023-11-28T21:52:26.521340+0000","last_clean":"2023-11-28T21:52:26.521340+0000","last_became_active":"2023-11-28T21:52:26.519697+0000","last_became_peered":"2023-11-28T21:52:26.519697+0000","last_unstale":"2023-11-28T21:52:26.521340+0000","last_undegraded":"2023-11-28T21:52:26.521340+0000","last_fullsized":"2023-11-28T21:52:26.521340+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:42:00.315518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"198'1","reported_seq":12,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.877508+0000","last_change":"2023-11-28T21:52:25.480618+0000","last_active":"2023-11-28T21:52:26.877508+0000","last_peered":"2023-11-28T21:52:26.877508+0000","last_clean":"2023-11-28T21:52:26.877508+0000","last_became_active":"2023-11-28T21:52:25.480314+0000","last_became_peered":"2023-11-28T21:52:25.480314+0000","last_unstale":"2023-11-28T21:52:26.877508+0000","last_undegraded":"2023-11-28T21:52:26.877508+0000","last_fullsized":"2023-11-28T21:52:26.877508+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:50.458339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.521758+0000","last_change":"2023-11-28T21:52:26.516396+0000","last_active":"2023-11-28T21:52:26.521758+0000","last_peered":"2023-11-28T21:52:26.521758+0000","last_clean":"2023-11-28T21:52:26.521758+0000","last_became_active":"2023-11-28T21:52:26.516076+0000","last_became_peered":"2023-11-28T21:52:26.516076+0000","last_unstale":"2023-11-28T21:52:26.521758+0000","last_undegraded":"2023-11-28T21:52:26.521758+0000","last_fullsized":"2023-11-28T21:52:26.521758+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:23:35.182629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.503487+0000","last_change":"2023-11-28T21:52:25.476455+0000","last_active":"2023-11-28T21:52:26.503487+0000","last_peered":"2023-11-28T21:52:26.503487+0000","last_clean":"2023-11-28T21:52:26.503487+0000","last_became_active":"2023-11-28T21:52:25.474241+0000","last_became_peered":"2023-11-28T21:52:25.474241+0000","last_unstale":"2023-11-28T21:52:26.503487+0000","last_undegraded":"2023-11-28T21:52:26.503487+0000","last_fullsized":"2023-11-28T21:52:26.503487+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:01.742868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486721+0000","last_change":"2023-11-28T21:52:26.510076+0000","last_active":"2023-11-28T21:52:27.486721+0000","last_peered":"2023-11-28T21:52:27.486721+0000","last_clean":"2023-11-28T21:52:27.486721+0000","last_became_active":"2023-11-28T21:52:26.509775+0000","last_became_peered":"2023-11-28T21:52:26.509775+0000","last_unstale":"2023-11-28T21:52:27.486721+0000","last_undegraded":"2023-11-28T21:52:27.486721+0000","last_fullsized":"2023-11-28T21:52:27.486721+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:14:57.033720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.484954+0000","last_change":"2023-11-28T21:52:25.486150+0000","last_active":"2023-11-28T21:52:27.484954+0000","last_peered":"2023-11-28T21:52:27.484954+0000","last_clean":"2023-11-28T21:52:27.484954+0000","last_became_active":"2023-11-28T21:52:25.485521+0000","last_became_peered":"2023-11-28T21:52:25.485521+0000","last_unstale":"2023-11-28T21:52:27.484954+0000","last_undegraded":"2023-11-28T21:52:27.484954+0000","last_fullsized":"2023-11-28T21:52:27.484954+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:16:36.911483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183399+0000","last_change":"2023-11-28T21:52:26.516376+0000","last_active":"2023-11-28T21:52:28.183399+0000","last_peered":"2023-11-28T21:52:28.183399+0000","last_clean":"2023-11-28T21:52:28.183399+0000","last_became_active":"2023-11-28T21:52:26.513230+0000","last_became_peered":"2023-11-28T21:52:26.513230+0000","last_unstale":"2023-11-28T21:52:28.183399+0000","last_undegraded":"2023-11-28T21:52:28.183399+0000","last_fullsized":"2023-11-28T21:52:28.183399+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:59.443868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487856+0000","last_change":"2023-11-28T21:52:25.493028+0000","last_active":"2023-11-28T21:52:27.487856+0000","last_peered":"2023-11-28T21:52:27.487856+0000","last_clean":"2023-11-28T21:52:27.487856+0000","last_became_active":"2023-11-28T21:52:25.492740+0000","last_became_peered":"2023-11-28T21:52:25.492740+0000","last_unstale":"2023-11-28T21:52:27.487856+0000","last_undegraded":"2023-11-28T21:52:27.487856+0000","last_fullsized":"2023-11-28T21:52:27.487856+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:57:30.234133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485573+0000","last_change":"2023-11-28T21:52:26.512257+0000","last_active":"2023-11-28T21:52:27.485573+0000","last_peered":"2023-11-28T21:52:27.485573+0000","last_clean":"2023-11-28T21:52:27.485573+0000","last_became_active":"2023-11-28T21:52:26.511978+0000","last_became_peered":"2023-11-28T21:52:26.511978+0000","last_unstale":"2023-11-28T21:52:27.485573+0000","last_undegraded":"2023-11-28T21:52:27.485573+0000","last_fullsized":"2023-11-28T21:52:27.485573+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:33:10.943660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"198'3","reported_seq":15,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487146+0000","last_change":"2023-11-28T21:52:25.486002+0000","last_active":"2023-11-28T21:52:27.487146+0000","last_peered":"2023-11-28T21:52:27.487146+0000","last_clean":"2023-11-28T21:52:27.487146+0000","last_became_active":"2023-11-28T21:52:25.485695+0000","last_became_peered":"2023-11-28T21:52:25.485695+0000","last_unstale":"2023-11-28T21:52:27.487146+0000","last_undegraded":"2023-11-28T21:52:27.487146+0000","last_fullsized":"2023-11-28T21:52:27.487146+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:35:12.856054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.514754+0000","last_change":"2023-11-28T21:52:26.506641+0000","last_active":"2023-11-28T21:52:26.514754+0000","last_peered":"2023-11-28T21:52:26.514754+0000","last_clean":"2023-11-28T21:52:26.514754+0000","last_became_active":"2023-11-28T21:52:26.506332+0000","last_became_peered":"2023-11-28T21:52:26.506332+0000","last_unstale":"2023-11-28T21:52:26.514754+0000","last_undegraded":"2023-11-28T21:52:26.514754+0000","last_fullsized":"2023-11-28T21:52:26.514754+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:24.235136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"198'6","reported_seq":18,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485875+0000","last_change":"2023-11-28T21:52:25.484959+0000","last_active":"2023-11-28T21:52:27.485875+0000","last_peered":"2023-11-28T21:52:27.485875+0000","last_clean":"2023-11-28T21:52:27.485875+0000","last_became_active":"2023-11-28T21:52:25.484694+0000","last_became_peered":"2023-11-28T21:52:25.484694+0000","last_unstale":"2023-11-28T21:52:27.485875+0000","last_undegraded":"2023-11-28T21:52:27.485875+0000","last_fullsized":"2023-11-28T21:52:27.485875+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:02.708444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207502+0000","last_change":"2023-11-28T21:52:26.512230+0000","last_active":"2023-11-28T21:52:28.207502+0000","last_peered":"2023-11-28T21:52:28.207502+0000","last_clean":"2023-11-28T21:52:28.207502+0000","last_became_active":"2023-11-28T21:52:26.511946+0000","last_became_peered":"2023-11-28T21:52:26.511946+0000","last_unstale":"2023-11-28T21:52:28.207502+0000","last_undegraded":"2023-11-28T21:52:28.207502+0000","last_fullsized":"2023-11-28T21:52:28.207502+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:29:53.726741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.880802+0000","last_change":"2023-11-28T21:52:25.478593+0000","last_active":"2023-11-28T21:52:26.880802+0000","last_peered":"2023-11-28T21:52:26.880802+0000","last_clean":"2023-11-28T21:52:26.880802+0000","last_became_active":"2023-11-28T21:52:25.478285+0000","last_became_peered":"2023-11-28T21:52:25.478285+0000","last_unstale":"2023-11-28T21:52:26.880802+0000","last_undegraded":"2023-11-28T21:52:26.880802+0000","last_fullsized":"2023-11-28T21:52:26.880802+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:48:00.340285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486885+0000","last_change":"2023-11-28T21:52:26.519286+0000","last_active":"2023-11-28T21:52:27.486885+0000","last_peered":"2023-11-28T21:52:27.486885+0000","last_clean":"2023-11-28T21:52:27.486885+0000","last_became_active":"2023-11-28T21:52:26.519010+0000","last_became_peered":"2023-11-28T21:52:26.519010+0000","last_unstale":"2023-11-28T21:52:27.486885+0000","last_undegraded":"2023-11-28T21:52:27.486885+0000","last_fullsized":"2023-11-28T21:52:27.486885+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:25:05.253795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485878+0000","last_change":"2023-11-28T21:52:25.481104+0000","last_active":"2023-11-28T21:52:27.485878+0000","last_peered":"2023-11-28T21:52:27.485878+0000","last_clean":"2023-11-28T21:52:27.485878+0000","last_became_active":"2023-11-28T21:52:25.480483+0000","last_became_peered":"2023-11-28T21:52:25.480483+0000","last_unstale":"2023-11-28T21:52:27.485878+0000","last_undegraded":"2023-11-28T21:52:27.485878+0000","last_fullsized":"2023-11-28T21:52:27.485878+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:45:13.059071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.519101+0000","last_change":"2023-11-28T21:52:26.515183+0000","last_active":"2023-11-28T21:52:26.519101+0000","last_peered":"2023-11-28T21:52:26.519101+0000","last_clean":"2023-11-28T21:52:26.519101+0000","last_became_active":"2023-11-28T21:52:26.514881+0000","last_became_peered":"2023-11-28T21:52:26.514881+0000","last_unstale":"2023-11-28T21:52:26.519101+0000","last_undegraded":"2023-11-28T21:52:26.519101+0000","last_fullsized":"2023-11-28T21:52:26.519101+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:18:22.456517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.501256+0000","last_change":"2023-11-28T21:52:25.484996+0000","last_active":"2023-11-28T21:52:26.501256+0000","last_peered":"2023-11-28T21:52:26.501256+0000","last_clean":"2023-11-28T21:52:26.501256+0000","last_became_active":"2023-11-28T21:52:25.484488+0000","last_became_peered":"2023-11-28T21:52:25.484488+0000","last_unstale":"2023-11-28T21:52:26.501256+0000","last_undegraded":"2023-11-28T21:52:26.501256+0000","last_fullsized":"2023-11-28T21:52:26.501256+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:43:50.665309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.521043+0000","last_change":"2023-11-28T21:52:26.517778+0000","last_active":"2023-11-28T21:52:26.521043+0000","last_peered":"2023-11-28T21:52:26.521043+0000","last_clean":"2023-11-28T21:52:26.521043+0000","last_became_active":"2023-11-28T21:52:26.513758+0000","last_became_peered":"2023-11-28T21:52:26.513758+0000","last_unstale":"2023-11-28T21:52:26.521043+0000","last_undegraded":"2023-11-28T21:52:26.521043+0000","last_fullsized":"2023-11-28T21:52:26.521043+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:02:45.893909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"198'8","reported_seq":20,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485456+0000","last_change":"2023-11-28T21:52:25.489778+0000","last_active":"2023-11-28T21:52:27.485456+0000","last_peered":"2023-11-28T21:52:27.485456+0000","last_clean":"2023-11-28T21:52:27.485456+0000","last_became_active":"2023-11-28T21:52:25.489502+0000","last_became_peered":"2023-11-28T21:52:25.489502+0000","last_unstale":"2023-11-28T21:52:27.485456+0000","last_undegraded":"2023-11-28T21:52:27.485456+0000","last_fullsized":"2023-11-28T21:52:27.485456+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:28.690571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.484293+0000","last_change":"2023-11-28T21:52:26.511251+0000","last_active":"2023-11-28T21:52:27.484293+0000","last_peered":"2023-11-28T21:52:27.484293+0000","last_clean":"2023-11-28T21:52:27.484293+0000","last_became_active":"2023-11-28T21:52:26.510239+0000","last_became_peered":"2023-11-28T21:52:26.510239+0000","last_unstale":"2023-11-28T21:52:27.484293+0000","last_undegraded":"2023-11-28T21:52:27.484293+0000","last_fullsized":"2023-11-28T21:52:27.484293+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:11:40.364708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"198'1","reported_seq":13,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.488015+0000","last_change":"2023-11-28T21:52:25.488326+0000","last_active":"2023-11-28T21:52:27.488015+0000","last_peered":"2023-11-28T21:52:27.488015+0000","last_clean":"2023-11-28T21:52:27.488015+0000","last_became_active":"2023-11-28T21:52:25.487639+0000","last_became_peered":"2023-11-28T21:52:25.487639+0000","last_unstale":"2023-11-28T21:52:27.488015+0000","last_undegraded":"2023-11-28T21:52:27.488015+0000","last_fullsized":"2023-11-28T21:52:27.488015+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:03.190529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.207650+0000","last_change":"2023-11-28T21:52:26.514585+0000","last_active":"2023-11-28T21:52:28.207650+0000","last_peered":"2023-11-28T21:52:28.207650+0000","last_clean":"2023-11-28T21:52:28.207650+0000","last_became_active":"2023-11-28T21:52:26.514227+0000","last_became_peered":"2023-11-28T21:52:26.514227+0000","last_unstale":"2023-11-28T21:52:28.207650+0000","last_undegraded":"2023-11-28T21:52:28.207650+0000","last_fullsized":"2023-11-28T21:52:28.207650+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:45:32.433696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"198'2","reported_seq":14,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486594+0000","last_change":"2023-11-28T21:52:25.489907+0000","last_active":"2023-11-28T21:52:27.486594+0000","last_peered":"2023-11-28T21:52:27.486594+0000","last_clean":"2023-11-28T21:52:27.486594+0000","last_became_active":"2023-11-28T21:52:25.489563+0000","last_became_peered":"2023-11-28T21:52:25.489563+0000","last_unstale":"2023-11-28T21:52:27.486594+0000","last_undegraded":"2023-11-28T21:52:27.486594+0000","last_fullsized":"2023-11-28T21:52:27.486594+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:43.114658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.512403+0000","last_change":"2023-11-28T21:52:26.508055+0000","last_active":"2023-11-28T21:52:26.512403+0000","last_peered":"2023-11-28T21:52:26.512403+0000","last_clean":"2023-11-28T21:52:26.512403+0000","last_became_active":"2023-11-28T21:52:26.507776+0000","last_became_peered":"2023-11-28T21:52:26.507776+0000","last_unstale":"2023-11-28T21:52:26.512403+0000","last_undegraded":"2023-11-28T21:52:26.512403+0000","last_fullsized":"2023-11-28T21:52:26.512403+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:20:38.846915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"198'1","reported_seq":12,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.877496+0000","last_change":"2023-11-28T21:52:25.487555+0000","last_active":"2023-11-28T21:52:26.877496+0000","last_peered":"2023-11-28T21:52:26.877496+0000","last_clean":"2023-11-28T21:52:26.877496+0000","last_became_active":"2023-11-28T21:52:25.486193+0000","last_became_peered":"2023-11-28T21:52:25.486193+0000","last_unstale":"2023-11-28T21:52:26.877496+0000","last_undegraded":"2023-11-28T21:52:26.877496+0000","last_fullsized":"2023-11-28T21:52:26.877496+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:12:46.492484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.510688+0000","last_change":"2023-11-28T21:52:26.506531+0000","last_active":"2023-11-28T21:52:26.510688+0000","last_peered":"2023-11-28T21:52:26.510688+0000","last_clean":"2023-11-28T21:52:26.510688+0000","last_became_active":"2023-11-28T21:52:26.506225+0000","last_became_peered":"2023-11-28T21:52:26.506225+0000","last_unstale":"2023-11-28T21:52:26.510688+0000","last_undegraded":"2023-11-28T21:52:26.510688+0000","last_fullsized":"2023-11-28T21:52:26.510688+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:09:23.218622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486386+0000","last_change":"2023-11-28T21:52:25.491913+0000","last_active":"2023-11-28T21:52:27.486386+0000","last_peered":"2023-11-28T21:52:27.486386+0000","last_clean":"2023-11-28T21:52:27.486386+0000","last_became_active":"2023-11-28T21:52:25.491634+0000","last_became_peered":"2023-11-28T21:52:25.491634+0000","last_unstale":"2023-11-28T21:52:27.486386+0000","last_undegraded":"2023-11-28T21:52:27.486386+0000","last_fullsized":"2023-11-28T21:52:27.486386+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:59.931452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485781+0000","last_change":"2023-11-28T21:52:26.509367+0000","last_active":"2023-11-28T21:52:27.485781+0000","last_peered":"2023-11-28T21:52:27.485781+0000","last_clean":"2023-11-28T21:52:27.485781+0000","last_became_active":"2023-11-28T21:52:26.509080+0000","last_became_peered":"2023-11-28T21:52:26.509080+0000","last_unstale":"2023-11-28T21:52:27.485781+0000","last_undegraded":"2023-11-28T21:52:27.485781+0000","last_fullsized":"2023-11-28T21:52:27.485781+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:15:02.553545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.878218+0000","last_change":"2023-11-28T21:52:25.477958+0000","last_active":"2023-11-28T21:52:26.878218+0000","last_peered":"2023-11-28T21:52:26.878218+0000","last_clean":"2023-11-28T21:52:26.878218+0000","last_became_active":"2023-11-28T21:52:25.477643+0000","last_became_peered":"2023-11-28T21:52:25.477643+0000","last_unstale":"2023-11-28T21:52:26.878218+0000","last_undegraded":"2023-11-28T21:52:26.878218+0000","last_fullsized":"2023-11-28T21:52:26.878218+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:01.258764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.523144+0000","last_change":"2023-11-28T21:52:26.517396+0000","last_active":"2023-11-28T21:52:26.523144+0000","last_peered":"2023-11-28T21:52:26.523144+0000","last_clean":"2023-11-28T21:52:26.523144+0000","last_became_active":"2023-11-28T21:52:26.515493+0000","last_became_peered":"2023-11-28T21:52:26.515493+0000","last_unstale":"2023-11-28T21:52:26.523144+0000","last_undegraded":"2023-11-28T21:52:26.523144+0000","last_fullsized":"2023-11-28T21:52:26.523144+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:37:09.751545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.499419+0000","last_change":"2023-11-28T21:52:25.480399+0000","last_active":"2023-11-28T21:52:26.499419+0000","last_peered":"2023-11-28T21:52:26.499419+0000","last_clean":"2023-11-28T21:52:26.499419+0000","last_became_active":"2023-11-28T21:52:25.480075+0000","last_became_peered":"2023-11-28T21:52:25.480075+0000","last_unstale":"2023-11-28T21:52:26.499419+0000","last_undegraded":"2023-11-28T21:52:26.499419+0000","last_fullsized":"2023-11-28T21:52:26.499419+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:50.235463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.516766+0000","last_change":"2023-11-28T21:52:26.513268+0000","last_active":"2023-11-28T21:52:26.516766+0000","last_peered":"2023-11-28T21:52:26.516766+0000","last_clean":"2023-11-28T21:52:26.516766+0000","last_became_active":"2023-11-28T21:52:26.512501+0000","last_became_peered":"2023-11-28T21:52:26.512501+0000","last_unstale":"2023-11-28T21:52:26.516766+0000","last_undegraded":"2023-11-28T21:52:26.516766+0000","last_fullsized":"2023-11-28T21:52:26.516766+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:49:57.772440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.503486+0000","last_change":"2023-11-28T21:52:25.482730+0000","last_active":"2023-11-28T21:52:26.503486+0000","last_peered":"2023-11-28T21:52:26.503486+0000","last_clean":"2023-11-28T21:52:26.503486+0000","last_became_active":"2023-11-28T21:52:25.480128+0000","last_became_peered":"2023-11-28T21:52:25.480128+0000","last_unstale":"2023-11-28T21:52:26.503486+0000","last_undegraded":"2023-11-28T21:52:26.503486+0000","last_fullsized":"2023-11-28T21:52:26.503486+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:45:19.558564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184485+0000","last_change":"2023-11-28T21:52:26.516709+0000","last_active":"2023-11-28T21:52:28.184485+0000","last_peered":"2023-11-28T21:52:28.184485+0000","last_clean":"2023-11-28T21:52:28.184485+0000","last_became_active":"2023-11-28T21:52:26.516415+0000","last_became_peered":"2023-11-28T21:52:26.516415+0000","last_unstale":"2023-11-28T21:52:28.184485+0000","last_undegraded":"2023-11-28T21:52:28.184485+0000","last_fullsized":"2023-11-28T21:52:28.184485+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:03.725396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183689+0000","last_change":"2023-11-28T21:52:25.481985+0000","last_active":"2023-11-28T21:52:28.183689+0000","last_peered":"2023-11-28T21:52:28.183689+0000","last_clean":"2023-11-28T21:52:28.183689+0000","last_became_active":"2023-11-28T21:52:25.478699+0000","last_became_peered":"2023-11-28T21:52:25.478699+0000","last_unstale":"2023-11-28T21:52:28.183689+0000","last_undegraded":"2023-11-28T21:52:28.183689+0000","last_fullsized":"2023-11-28T21:52:28.183689+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:04:08.070617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.519051+0000","last_change":"2023-11-28T21:52:26.515074+0000","last_active":"2023-11-28T21:52:26.519051+0000","last_peered":"2023-11-28T21:52:26.519051+0000","last_clean":"2023-11-28T21:52:26.519051+0000","last_became_active":"2023-11-28T21:52:26.514800+0000","last_became_peered":"2023-11-28T21:52:26.514800+0000","last_unstale":"2023-11-28T21:52:26.519051+0000","last_undegraded":"2023-11-28T21:52:26.519051+0000","last_fullsized":"2023-11-28T21:52:26.519051+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:49:28.580075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"198'2","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.878280+0000","last_change":"2023-11-28T21:52:25.478699+0000","last_active":"2023-11-28T21:52:26.878280+0000","last_peered":"2023-11-28T21:52:26.878280+0000","last_clean":"2023-11-28T21:52:26.878280+0000","last_became_active":"2023-11-28T21:52:25.478402+0000","last_became_peered":"2023-11-28T21:52:25.478402+0000","last_unstale":"2023-11-28T21:52:26.878280+0000","last_undegraded":"2023-11-28T21:52:26.878280+0000","last_fullsized":"2023-11-28T21:52:26.878280+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:04:08.070617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485939+0000","last_change":"2023-11-28T21:52:26.508747+0000","last_active":"2023-11-28T21:52:27.485939+0000","last_peered":"2023-11-28T21:52:27.485939+0000","last_clean":"2023-11-28T21:52:27.485939+0000","last_became_active":"2023-11-28T21:52:26.508377+0000","last_became_peered":"2023-11-28T21:52:26.508377+0000","last_unstale":"2023-11-28T21:52:27.485939+0000","last_undegraded":"2023-11-28T21:52:27.485939+0000","last_fullsized":"2023-11-28T21:52:27.485939+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:46:00.409740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487651+0000","last_change":"2023-11-28T21:52:25.490303+0000","last_active":"2023-11-28T21:52:27.487651+0000","last_peered":"2023-11-28T21:52:27.487651+0000","last_clean":"2023-11-28T21:52:27.487651+0000","last_became_active":"2023-11-28T21:52:25.490022+0000","last_became_peered":"2023-11-28T21:52:25.490022+0000","last_unstale":"2023-11-28T21:52:27.487651+0000","last_undegraded":"2023-11-28T21:52:27.487651+0000","last_fullsized":"2023-11-28T21:52:27.487651+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:51:09.218840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.514884+0000","last_change":"2023-11-28T21:52:26.512789+0000","last_active":"2023-11-28T21:52:26.514884+0000","last_peered":"2023-11-28T21:52:26.514884+0000","last_clean":"2023-11-28T21:52:26.514884+0000","last_became_active":"2023-11-28T21:52:26.512236+0000","last_became_peered":"2023-11-28T21:52:26.512236+0000","last_unstale":"2023-11-28T21:52:26.514884+0000","last_undegraded":"2023-11-28T21:52:26.514884+0000","last_fullsized":"2023-11-28T21:52:26.514884+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:00:58.860332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486159+0000","last_change":"2023-11-28T21:52:25.469187+0000","last_active":"2023-11-28T21:52:27.486159+0000","last_peered":"2023-11-28T21:52:27.486159+0000","last_clean":"2023-11-28T21:52:27.486159+0000","last_became_active":"2023-11-28T21:52:25.468729+0000","last_became_peered":"2023-11-28T21:52:25.468729+0000","last_unstale":"2023-11-28T21:52:27.486159+0000","last_undegraded":"2023-11-28T21:52:27.486159+0000","last_fullsized":"2023-11-28T21:52:27.486159+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:22:19.583099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.485172+0000","last_change":"2023-11-28T21:52:26.524747+0000","last_active":"2023-11-28T21:52:27.485172+0000","last_peered":"2023-11-28T21:52:27.485172+0000","last_clean":"2023-11-28T21:52:27.485172+0000","last_became_active":"2023-11-28T21:52:26.516902+0000","last_became_peered":"2023-11-28T21:52:26.516902+0000","last_unstale":"2023-11-28T21:52:27.485172+0000","last_undegraded":"2023-11-28T21:52:27.485172+0000","last_fullsized":"2023-11-28T21:52:27.485172+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:18:06.006228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.505662+0000","last_change":"2023-11-28T21:52:25.483481+0000","last_active":"2023-11-28T21:52:26.505662+0000","last_peered":"2023-11-28T21:52:26.505662+0000","last_clean":"2023-11-28T21:52:26.505662+0000","last_became_active":"2023-11-28T21:52:25.483183+0000","last_became_peered":"2023-11-28T21:52:25.483183+0000","last_unstale":"2023-11-28T21:52:26.505662+0000","last_undegraded":"2023-11-28T21:52:26.505662+0000","last_fullsized":"2023-11-28T21:52:26.505662+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:21:36.169741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487278+0000","last_change":"2023-11-28T21:52:26.493659+0000","last_active":"2023-11-28T21:52:27.487278+0000","last_peered":"2023-11-28T21:52:27.487278+0000","last_clean":"2023-11-28T21:52:27.487278+0000","last_became_active":"2023-11-28T21:52:26.492831+0000","last_became_peered":"2023-11-28T21:52:26.492831+0000","last_unstale":"2023-11-28T21:52:27.487278+0000","last_undegraded":"2023-11-28T21:52:27.487278+0000","last_fullsized":"2023-11-28T21:52:27.487278+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:41:35.808625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486028+0000","last_change":"2023-11-28T21:52:25.491316+0000","last_active":"2023-11-28T21:52:27.486028+0000","last_peered":"2023-11-28T21:52:27.486028+0000","last_clean":"2023-11-28T21:52:27.486028+0000","last_became_active":"2023-11-28T21:52:25.491087+0000","last_became_peered":"2023-11-28T21:52:25.491087+0000","last_unstale":"2023-11-28T21:52:27.486028+0000","last_undegraded":"2023-11-28T21:52:27.486028+0000","last_fullsized":"2023-11-28T21:52:27.486028+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:28.380972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.184263+0000","last_change":"2023-11-28T21:52:26.513417+0000","last_active":"2023-11-28T21:52:28.184263+0000","last_peered":"2023-11-28T21:52:28.184263+0000","last_clean":"2023-11-28T21:52:28.184263+0000","last_became_active":"2023-11-28T21:52:26.513142+0000","last_became_peered":"2023-11-28T21:52:26.513142+0000","last_unstale":"2023-11-28T21:52:28.184263+0000","last_undegraded":"2023-11-28T21:52:28.184263+0000","last_fullsized":"2023-11-28T21:52:28.184263+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:56:23.584627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486110+0000","last_change":"2023-11-28T21:52:25.476811+0000","last_active":"2023-11-28T21:52:27.486110+0000","last_peered":"2023-11-28T21:52:27.486110+0000","last_clean":"2023-11-28T21:52:27.486110+0000","last_became_active":"2023-11-28T21:52:25.475907+0000","last_became_peered":"2023-11-28T21:52:25.475907+0000","last_unstale":"2023-11-28T21:52:27.486110+0000","last_undegraded":"2023-11-28T21:52:27.486110+0000","last_fullsized":"2023-11-28T21:52:27.486110+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:59.807724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.486386+0000","last_change":"2023-11-28T21:52:26.523095+0000","last_active":"2023-11-28T21:52:27.486386+0000","last_peered":"2023-11-28T21:52:27.486386+0000","last_clean":"2023-11-28T21:52:27.486386+0000","last_became_active":"2023-11-28T21:52:26.522659+0000","last_became_peered":"2023-11-28T21:52:26.522659+0000","last_unstale":"2023-11-28T21:52:27.486386+0000","last_undegraded":"2023-11-28T21:52:27.486386+0000","last_fullsized":"2023-11-28T21:52:27.486386+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:47:36.977837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.501641+0000","last_change":"2023-11-28T21:52:25.480460+0000","last_active":"2023-11-28T21:52:26.501641+0000","last_peered":"2023-11-28T21:52:26.501641+0000","last_clean":"2023-11-28T21:52:26.501641+0000","last_became_active":"2023-11-28T21:52:25.480101+0000","last_became_peered":"2023-11-28T21:52:25.480101+0000","last_unstale":"2023-11-28T21:52:26.501641+0000","last_undegraded":"2023-11-28T21:52:26.501641+0000","last_fullsized":"2023-11-28T21:52:26.501641+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:36:41.581693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.517627+0000","last_change":"2023-11-28T21:52:26.513466+0000","last_active":"2023-11-28T21:52:26.517627+0000","last_peered":"2023-11-28T21:52:26.517627+0000","last_clean":"2023-11-28T21:52:26.517627+0000","last_became_active":"2023-11-28T21:52:26.513162+0000","last_became_peered":"2023-11-28T21:52:26.513162+0000","last_unstale":"2023-11-28T21:52:26.517627+0000","last_undegraded":"2023-11-28T21:52:26.517627+0000","last_fullsized":"2023-11-28T21:52:26.517627+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:34.505548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.481111+0000","last_change":"2023-11-28T21:52:25.472451+0000","last_active":"2023-11-28T21:52:26.481111+0000","last_peered":"2023-11-28T21:52:26.481111+0000","last_clean":"2023-11-28T21:52:26.481111+0000","last_became_active":"2023-11-28T21:52:25.472162+0000","last_became_peered":"2023-11-28T21:52:25.472162+0000","last_unstale":"2023-11-28T21:52:26.481111+0000","last_undegraded":"2023-11-28T21:52:26.481111+0000","last_fullsized":"2023-11-28T21:52:26.481111+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:40:56.289968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:28.183516+0000","last_change":"2023-11-28T21:52:26.516714+0000","last_active":"2023-11-28T21:52:28.183516+0000","last_peered":"2023-11-28T21:52:28.183516+0000","last_clean":"2023-11-28T21:52:28.183516+0000","last_became_active":"2023-11-28T21:52:26.516340+0000","last_became_peered":"2023-11-28T21:52:26.516340+0000","last_unstale":"2023-11-28T21:52:28.183516+0000","last_undegraded":"2023-11-28T21:52:28.183516+0000","last_fullsized":"2023-11-28T21:52:28.183516+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:25.441114+0000","last_clean_scrub_stamp":"2023-11-28T21:52:25.441114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:07:06.713616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-11-28T21:52:27.487554+0000","last_change":"2023-11-28T21:52:25.489532+0000","last_active":"2023-11-28T21:52:27.487554+0000","last_peered":"2023-11-28T21:52:27.487554+0000","last_clean":"2023-11-28T21:52:27.487554+0000","last_became_active":"2023-11-28T21:52:25.489237+0000","last_became_peered":"2023-11-28T21:52:25.489237+0000","last_unstale":"2023-11-28T21:52:27.487554+0000","last_undegraded":"2023-11-28T21:52:27.487554+0000","last_fullsized":"2023-11-28T21:52:27.487554+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T21:52:24.424162+0000","last_clean_scrub_stamp":"2023-11-28T21:52:24.424162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:23:05.185207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"187'4800","reported_seq":5555,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-11-28T21:52:26.497586+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T21:52:26.497586+0000","last_peered":"2023-11-28T21:52:26.497586+0000","last_clean":"2023-11-28T21:52:26.497586+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T21:52:26.497586+0000","last_undegraded":"2023-11-28T21:52:26.497586+0000","last_fullsized":"2023-11-28T21:52:26.497586+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4800,"log_dups_size":0,"ondisk_log_size":4800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1765,"num_read_kb":3627,"num_write":491,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1765,"num_read_kb":3627,"num_write":491,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4800,"ondisk_log_size":4800,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575783,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56388,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315452,"statfs":{"total":96636764160,"available":96579022848,"internally_reserved":0,"allocated":3072000,"data_stored":1858017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.24,"15min":1.2110000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.641,"5min":1.865,"15min":1.865},"last":1.524},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.177,"15min":1.2330000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6140000000000001,"5min":1.8140000000000001,"15min":1.954},"last":1.4590000000000001}]},{"osd":1,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1819999999999999,"15min":1.232},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.641,"5min":1.6659999999999999,"15min":2.1880000000000002},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.2809999999999999,"15min":1.28},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.853,"5min":1.853,"15min":2.4279999999999999},"last":1.651}]},{"osd":2,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5229999999999999,"5min":1.754,"15min":1.893},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.349,"15min":1.359},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.5600000000000001,"5min":1.8280000000000001,"15min":2.0339999999999998},"last":1.7050000000000001}]},{"osd":3,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3,"15min":1.3129999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.643,"5min":1.845,"15min":2.1000000000000001},"last":1.008},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4630000000000001,"15min":1.464},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.768,"5min":1.8660000000000001,"15min":2.3769999999999998},"last":1.3480000000000001}]},{"osd":4,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.347,"15min":1.304},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.909},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3300000000000001,"15min":1.341},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.704,"5min":1.895,"15min":1.9470000000000001},"last":1.3839999999999999}]},{"osd":5,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.496,"15min":1.4570000000000001},"min":{"1min":1.099,"5min":0.94899999999999995,"15min":0.79200000000000004},"max":{"1min":2.794,"5min":2.794,"15min":2.794},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.538,"15min":1.5029999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.028,"5min":2.028,"15min":2.246},"last":1.8049999999999999}]},{"osd":6,"last update":"Tue Nov 28 21:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.599,"15min":1.5900000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.3029999999999999},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4830000000000001,"15min":1.4990000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.8160000000000001,"5min":1.8859999999999999,"15min":2.2170000000000001},"last":1.2450000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575783,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41920,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329920,"statfs":{"total":96636764160,"available":96593838080,"internally_reserved":0,"allocated":2478080,"data_stored":1267649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.1060000000000001,"15min":1.1259999999999999},"min":{"1min":0.63100000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3420000000000001,"5min":1.597,"15min":1.8740000000000001},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.173,"15min":1.159},"min":{"1min":0.84099999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":2.0840000000000001}]},{"osd":1,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.204,"15min":1.2569999999999999},"min":{"1min":0.72499999999999998,"5min":0.443,"15min":0.443},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.9550000000000001},"last":1.484},{"interface":"front","average":{"1min":1.286,"5min":1.3979999999999999,"15min":1.4279999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.976,"5min":1.976,"15min":2.2109999999999999},"last":1.976}]},{"osd":2,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2729999999999999,"15min":1.274},"min":{"1min":0.72799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.647,"5min":1.8009999999999999,"15min":2.012},"last":1.202},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.405,"15min":1.4199999999999999},"min":{"1min":0.91800000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.804,"5min":1.804,"15min":2.4420000000000002},"last":1.79}]},{"osd":3,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3100000000000001,"15min":1.327},"min":{"1min":0.72199999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0779999999999998},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.304,"15min":1.323},"min":{"1min":0.81899999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.508,"5min":1.8600000000000001,"15min":2.1499999999999999},"last":0.91000000000000003}]},{"osd":4,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.2869999999999999,"15min":1.276},"min":{"1min":0.89800000000000002,"5min":0.80200000000000005,"15min":0.77900000000000003},"max":{"1min":1.873,"5min":1.873,"15min":2.726},"last":1.677},{"interface":"front","average":{"1min":1.494,"5min":1.377,"15min":1.3600000000000001},"min":{"1min":1.0249999999999999,"5min":0.82499999999999996,"15min":0.80100000000000005},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":3.069},"last":1.8400000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.415,"15min":1.399},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.9179999999999999,"5min":2.0529999999999999,"15min":2.319},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.5649999999999999,"15min":1.5800000000000001},"min":{"1min":1.165,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.0289999999999999,"5min":2.0289999999999999,"15min":2.5680000000000001},"last":2.0289999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.55,"15min":1.5409999999999999},"min":{"1min":1.262,"5min":1.0800000000000001,"15min":0.99099999999999999},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.96},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.5349999999999999,"15min":1.528},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.9610000000000001,"5min":1.99,"15min":2.3439999999999999},"last":1.5409999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59976,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311864,"statfs":{"total":96636764160,"available":96575348736,"internally_reserved":0,"allocated":2486272,"data_stored":1267773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.65400000000000003},"max":{"1min":3.4430000000000001,"5min":3.4430000000000001,"15min":3.4430000000000001},"last":3.323},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.204,"15min":1.21},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":4.8070000000000004,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":12.429}]},{"osd":1,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1990000000000001,"15min":1.2190000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.591,"5min":1.724,"15min":1.843},"last":8.2080000000000002},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.2689999999999999,"15min":1.288},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":9.3949999999999996}]},{"osd":2,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3009999999999999,"15min":1.3280000000000001},"min":{"1min":0.74099999999999999,"5min":0.72999999999999998,"15min":0.70199999999999996},"max":{"1min":3.1989999999999998,"5min":3.1989999999999998,"15min":3.1989999999999998},"last":13.605},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.341,"15min":1.3340000000000001},"min":{"1min":0.91200000000000003,"5min":0.876,"15min":0.83699999999999997},"max":{"1min":1.7609999999999999,"5min":1.766,"15min":1.9990000000000001},"last":6.8849999999999998}]},{"osd":3,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.407,"15min":1.391},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.754},"max":{"1min":1.6970000000000001,"5min":1.954,"15min":2.1030000000000002},"last":7.008},{"interface":"front","average":{"1min":1.399,"5min":1.3500000000000001,"15min":1.3340000000000001},"min":{"1min":0.97599999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":0.97899999999999998}]},{"osd":4,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.417,"15min":1.365},"min":{"1min":0.91300000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":3.1400000000000001,"5min":3.1400000000000001,"15min":3.1400000000000001},"last":3.004},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.4990000000000001,"15min":1.4670000000000001},"min":{"1min":1.1859999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":9.4480000000000004}]},{"osd":6,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.403,"15min":1.3859999999999999},"min":{"1min":0.873,"5min":0.752,"15min":0.752},"max":{"1min":1.9710000000000001,"5min":2.0110000000000001,"15min":2.0699999999999998},"last":9.2189999999999994},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.5509999999999999,"15min":1.5429999999999999},"min":{"1min":1.2230000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":1.0780000000000001}]},{"osd":7,"last update":"Tue Nov 28 21:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.4590000000000001,"15min":1.4650000000000001},"min":{"1min":1.1799999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":3.157,"5min":3.157,"15min":3.157},"last":3.3889999999999998},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.542,"15min":1.5309999999999999},"min":{"1min":1.208,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.9059999999999999,"5min":1.9790000000000001,"15min":2.2010000000000001},"last":6.9370000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575784,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29460,"kb_used_data":2436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342380,"statfs":{"total":96636764160,"available":96606597120,"internally_reserved":0,"allocated":2494464,"data_stored":1268413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0600000000000001,"15min":1.1259999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.2929999999999999,"5min":1.9379999999999999,"15min":1.9550000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1539999999999999,"15min":1.151},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.647,"5min":1.679,"15min":1.732},"last":0.65500000000000003}]},{"osd":1,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.353,"15min":1.3500000000000001},"min":{"1min":0.92500000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.6830000000000001,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.349,"15min":1.3799999999999999},"min":{"1min":0.95499999999999996,"5min":0.82699999999999996,"15min":0.79900000000000004},"max":{"1min":1.718,"5min":1.8320000000000001,"15min":1.9119999999999999},"last":0.73699999999999999}]},{"osd":2,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.216,"15min":1.258},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.506,"5min":1.788,"15min":1.9570000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2809999999999999,"15min":1.3080000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.5820000000000001,"5min":1.706,"15min":1.905},"last":0.61499999999999999}]},{"osd":3,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.306,"15min":1.2989999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7769999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.407,"15min":1.4119999999999999},"min":{"1min":1.0469999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.923},"last":0.75600000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3280000000000001,"15min":1.302},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.363,"15min":1.3320000000000001},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.79100000000000004},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.48799999999999999}]},{"osd":6,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.476,"15min":1.448},"min":{"1min":1.085,"5min":0.89900000000000002,"15min":0.85399999999999998},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.1099999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.585},"min":{"1min":1.105,"5min":1.105,"15min":0.86499999999999999},"max":{"1min":2.113,"5min":2.113,"15min":2.23},"last":0.67400000000000004}]},{"osd":7,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4890000000000001,"15min":1.482},"min":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":2.056,"5min":2.056,"15min":2.085},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.615,"5min":1.6020000000000001,"15min":1.577},"min":{"1min":1.3089999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.0640000000000001,"5min":2.1640000000000001,"15min":2.2210000000000001},"last":0.63600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575784,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51680,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320160,"statfs":{"total":96636764160,"available":96583843840,"internally_reserved":0,"allocated":2510848,"data_stored":1276513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.99299999999999999,"15min":0.98099999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.2829999999999999,"5min":3.2829999999999999,"15min":3.2829999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.877,"5min":1.0529999999999999,"15min":1.0960000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.296,"5min":1.673,"15min":1.9399999999999999},"last":0.70699999999999996}]},{"osd":1,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.014,"15min":1.0209999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.087,"5min":1.2390000000000001,"15min":1.248},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.7999999999999998,"5min":2.7999999999999998,"15min":2.7999999999999998},"last":0.747}]},{"osd":2,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.2430000000000001,"15min":1.296},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":3.6299999999999999,"5min":3.6299999999999999,"15min":3.6299999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.4199999999999999,"15min":1.423},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":0.82499999999999996}]},{"osd":4,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.3160000000000001,"15min":1.379},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.968,"5min":1.968,"15min":2.246},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.387,"5min":1.431,"15min":1.46},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.9119999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.873}]},{"osd":5,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.5529999999999999,"15min":1.5660000000000001},"min":{"1min":1.004,"5min":1.004,"15min":1.004},"max":{"1min":2.927,"5min":2.927,"15min":2.927},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.468,"15min":1.484},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.121,"5min":2.1699999999999999,"15min":2.3820000000000001},"last":1.371}]},{"osd":6,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.54,"15min":1.5549999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.7410000000000001,"5min":2.7410000000000001,"15min":2.7410000000000001},"last":1.452},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.655,"15min":1.6679999999999999},"min":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"max":{"1min":3.0569999999999999,"5min":3.0569999999999999,"15min":3.0569999999999999},"last":1.296}]},{"osd":7,"last update":"Tue Nov 28 21:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.573,"15min":1.6060000000000001},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.5},"last":1.319},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.6599999999999999,"15min":1.7070000000000001},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.8519999999999999,"5min":2.8519999999999999,"15min":2.8519999999999999},"last":1.397}]}]},{"osd":2,"up_from":13,"seq":55834575784,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47492,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324348,"statfs":{"total":96636764160,"available":96588132352,"internally_reserved":0,"allocated":2478080,"data_stored":1267649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":0.97599999999999998,"15min":0.96899999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.006,"5min":1.141,"15min":1.1439999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.6559999999999999,"5min":1.764,"15min":1.8160000000000001},"last":1.3620000000000001}]},{"osd":1,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.98399999999999999,"15min":1.0049999999999999},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0379999999999998},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.222,"15min":1.2430000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.8380000000000001},"last":1.3240000000000001}]},{"osd":3,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.1990000000000001,"15min":1.222},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.7679999999999998,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.292,"5min":1.3109999999999999,"15min":1.329},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.9199999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.827}]},{"osd":4,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.456,"15min":1.4670000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":3.202,"5min":3.202,"15min":3.202},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.498,"15min":1.502},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.6220000000000001}]},{"osd":5,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.508,"15min":1.522},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":0.99299999999999999},"max":{"1min":3.1269999999999998,"5min":3.1269999999999998,"15min":3.1269999999999998},"last":1.579},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.5800000000000001,"15min":1.542},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.3149999999999999,"5min":3.3149999999999999,"15min":3.3149999999999999},"last":1.976}]},{"osd":6,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.583,"15min":1.591},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.782},{"interface":"front","average":{"1min":1.661,"5min":1.633,"15min":1.6279999999999999},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.9039999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:51:41 2023","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.621,"15min":1.633},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":4.2990000000000004,"5min":4.2990000000000004,"15min":4.2990000000000004},"last":1.7310000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.679,"15min":1.7},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.8940000000000001,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":1.867}]}]},{"osd":1,"up_from":13,"seq":55834575784,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29464,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342376,"statfs":{"total":96636764160,"available":96606593024,"internally_reserved":0,"allocated":2502656,"data_stored":1275909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.91700000000000004,"15min":0.91800000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.147,"5min":1.127,"15min":1.1240000000000001},"min":{"1min":0.76000000000000001,"5min":0.753,"15min":0.69799999999999995},"max":{"1min":2.9489999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.63600000000000001}]},{"osd":2,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1819999999999999,"15min":1.1499999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77400000000000002},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.254,"5min":1.258,"15min":1.2649999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.921,"5min":1.921,"15min":2.0750000000000002},"last":1.2330000000000001}]},{"osd":3,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4039999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.889,"5min":1.9790000000000001,"15min":1.9810000000000001},"last":1.49},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.1910000000000001,"15min":1.1839999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":1.708}]},{"osd":4,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.4279999999999999,"15min":1.415},"min":{"1min":1.1299999999999999,"5min":1.0209999999999999,"15min":0.90900000000000003},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.4570000000000001,"15min":1.4450000000000001},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.84199999999999997},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.0189999999999999}]},{"osd":5,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.482,"15min":1.48},"min":{"1min":1.008,"5min":1.008,"15min":0.97399999999999998},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.129},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.569,"15min":1.5700000000000001},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.0110000000000001,"5min":2.0979999999999999,"15min":2.25},"last":1.591}]},{"osd":6,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.516,"15min":1.5029999999999999},"min":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.0429999999999999},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.518},{"interface":"front","average":{"1min":1.712,"5min":1.605,"15min":1.591},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":3.1520000000000001,"5min":3.1520000000000001,"15min":3.1520000000000001},"last":1.417}]},{"osd":7,"last update":"Tue Nov 28 21:51:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.6439999999999999,"15min":1.6559999999999999},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.2589999999999999,"5min":2.2629999999999999,"15min":2.3439999999999999},"last":1.669},{"interface":"front","average":{"1min":1.734,"5min":1.6299999999999999,"15min":1.607},"min":{"1min":1.258,"5min":1.151,"15min":1.1080000000000001},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.4570000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575784,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30032,"kb_used_data":3008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341808,"statfs":{"total":96636764160,"available":96606011392,"internally_reserved":0,"allocated":3080192,"data_stored":1858121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.92900000000000005,"15min":0.94399999999999995},"min":{"1min":0.48299999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.492,"5min":1.72,"15min":1.724},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.115,"15min":1.1459999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2829999999999999,"5min":2.7879999999999998,"15min":2.7879999999999998},"last":0.52700000000000002}]},{"osd":2,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.113,"15min":1.1339999999999999},"min":{"1min":0.621,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8959999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.1990000000000001,"15min":1.2470000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.5649999999999999,"5min":1.8819999999999999,"15min":2.0409999999999999},"last":0.69099999999999995}]},{"osd":3,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.1040000000000001,"15min":1.1180000000000001},"min":{"1min":0.57499999999999996,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.4650000000000001,"5min":1.7110000000000001,"15min":1.829},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.363,"15min":1.355},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.782,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.4299999999999999,"15min":1.4610000000000001},"min":{"1min":0.94299999999999995,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.7529999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":1.034,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.762,"5min":1.869,"15min":2.0600000000000001},"last":1.323}]},{"osd":5,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.46,"15min":1.4510000000000001},"min":{"1min":0.96499999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.8009999999999999,"5min":2.153,"15min":2.153},"last":1.012},{"interface":"front","average":{"1min":1.534,"5min":1.5029999999999999,"15min":1.536},"min":{"1min":1.159,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.9790000000000001,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.282}]},{"osd":6,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.5820000000000001,"15min":1.6020000000000001},"min":{"1min":1.1379999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.0099999999999998,"5min":2.319,"15min":2.319},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.546,"5min":1.591,"15min":1.6000000000000001},"min":{"1min":1.0149999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.052,"5min":2.73,"15min":2.73},"last":1.3029999999999999}]},{"osd":7,"last update":"Tue Nov 28 21:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.657,"15min":1.6619999999999999},"min":{"1min":0.90300000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1699999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.5860000000000001,"15min":1.583},"min":{"1min":0.90600000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0649999999999999,"5min":2.371,"15min":2.371},"last":1.228}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T21:52:31.900 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T21:52:31.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T21:52:31.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:52:32.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:52:32.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.160+0000 7f7cca378640 1 Processor -- start 2023-11-28T21:52:32.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.160+0000 7f7cca378640 1 -- start start 2023-11-28T21:52:32.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.162+0000 7f7cca378640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7cc406dc50 0x7f7cc40cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.162+0000 7f7cca378640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7cc40ce600 0x7f7cc406a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.162+0000 7f7cca378640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc406e030 con 0x7f7cc406aa40 2023-11-28T21:52:32.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.162+0000 7f7cca378640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc406aed0 con 0x7f7cc40ce600 2023-11-28T21:52:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.162+0000 7f7cca378640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40ccf60 con 0x7f7cc406dc50 2023-11-28T21:52:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.163+0000 7f7cbb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7cc40ce600 0x7f7cc406a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.163+0000 7f7cbb7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7cc40ce600 0x7f7cc406a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50756/0 (socket says 172.21.15.45:50756) 2023-11-28T21:52:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.163+0000 7f7cbb7fe640 1 -- 172.21.15.45:0/2103064276 learned_addr learned my addr 172.21.15.45:0/2103064276 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.163+0000 7f7cc3fff640 1 --2- 172.21.15.45:0/2103064276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7cc406dc50 0x7f7cc40cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.163+0000 7f7cbb7fe640 1 -- 172.21.15.45:0/2103064276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7cc406dc50 msgr2=0x7f7cc40cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.163+0000 7f7cbb7fe640 1 --2- 172.21.15.45:0/2103064276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7cc406dc50 0x7f7cc40cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.163+0000 7f7cbb7fe640 1 -- 172.21.15.45:0/2103064276 >> v1:172.21.15.190:6789/0 conn(0x7f7cc406aa40 legacy=0x7f7cc40c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.163+0000 7f7cbb7fe640 1 -- 172.21.15.45:0/2103064276 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc40cd930 con 0x7f7cc40ce600 2023-11-28T21:52:32.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.164+0000 7f7cbb7fe640 1 --2- 172.21.15.45:0/2103064276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc40ce600 0x7f7cc406a500 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009870 tx=0x7f7cac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570f4a6efd83b0fc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:32.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.165+0000 7f7cc37fe640 1 -- 172.21.15.45:0/2103064276 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac031640 con 0x7f7cc40ce600 2023-11-28T21:52:32.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.165+0000 7f7cc37fe640 1 -- 172.21.15.45:0/2103064276 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac0317a0 con 0x7f7cc40ce600 2023-11-28T21:52:32.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.165+0000 7f7cc37fe640 1 -- 172.21.15.45:0/2103064276 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac031b30 con 0x7f7cc40ce600 2023-11-28T21:52:32.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.165+0000 7f7cca378640 1 -- 172.21.15.45:0/2103064276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc40ce600 msgr2=0x7f7cc406a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.165+0000 7f7cca378640 1 --2- 172.21.15.45:0/2103064276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc40ce600 0x7f7cc406a500 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009870 tx=0x7f7cac02b8f0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.166+0000 7f7cca378640 1 -- 172.21.15.45:0/2103064276 shutdown_connections 2023-11-28T21:52:32.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.166+0000 7f7cca378640 1 --2- 172.21.15.45:0/2103064276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc40ce600 0x7f7cc406a500 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.166+0000 7f7cca378640 1 --2- 172.21.15.45:0/2103064276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7cc406dc50 0x7f7cc40cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.166+0000 7f7cca378640 1 -- 172.21.15.45:0/2103064276 >> 172.21.15.45:0/2103064276 conn(0x7f7cc40ced60 msgr2=0x7f7cc40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:32.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.166+0000 7f7cca378640 1 -- 172.21.15.45:0/2103064276 shutdown_connections 2023-11-28T21:52:32.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.166+0000 7f7cca378640 1 -- 172.21.15.45:0/2103064276 wait complete. 2023-11-28T21:52:32.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.167+0000 7f7cca378640 1 Processor -- start 2023-11-28T21:52:32.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.168+0000 7f7cca378640 1 -- start start 2023-11-28T21:52:32.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cca378640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc406aa40 0x7f7cc415f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cca378640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7cc406dc50 0x7f7cc40c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cca378640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cc40ce600 0x7f7cc40c2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cca378640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40c3f40 con 0x7f7cc406aa40 2023-11-28T21:52:32.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cca378640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40c32c0 con 0x7f7cc40ce600 2023-11-28T21:52:32.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cca378640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40c35a0 con 0x7f7cc406dc50 2023-11-28T21:52:32.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cc3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc406aa40 0x7f7cc415f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:32.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cc3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc406aa40 0x7f7cc415f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50764/0 (socket says 172.21.15.45:50764) 2023-11-28T21:52:32.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cc3fff640 1 -- 172.21.15.45:0/902001709 learned_addr learned my addr 172.21.15.45:0/902001709 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cc3fff640 1 -- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7cc406dc50 msgr2=0x7f7cc40c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cc3fff640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7cc406dc50 0x7f7cc40c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cc3fff640 1 -- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cc40ce600 msgr2=0x7f7cc40c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cc3fff640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cc40ce600 0x7f7cc40c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.169+0000 7f7cc3fff640 1 -- 172.21.15.45:0/902001709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0094d0 con 0x7f7cc406aa40 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.170+0000 7f7cc3fff640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc406aa40 0x7f7cc415f250 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400d800 tx=0x7f7cb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.170+0000 7f7cc17fa640 1 -- 172.21.15.45:0/902001709 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb400b6c0 con 0x7f7cc406aa40 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.170+0000 7f7cc17fa640 1 -- 172.21.15.45:0/902001709 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb400b820 con 0x7f7cc406aa40 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.170+0000 7f7cc17fa640 1 -- 172.21.15.45:0/902001709 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb400bbb0 con 0x7f7cc406aa40 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.170+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc40c3880 con 0x7f7cc406aa40 2023-11-28T21:52:32.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.170+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc415f790 con 0x7f7cc406aa40 2023-11-28T21:52:32.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.171+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc40733d0 con 0x7f7cc406aa40 2023-11-28T21:52:32.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.172+0000 7f7cc17fa640 1 -- 172.21.15.45:0/902001709 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7cb400bd10 con 0x7f7cc406aa40 2023-11-28T21:52:32.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.173+0000 7f7cc17fa640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f7ca406af90 0x7f7ca406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.174+0000 7f7cc17fa640 1 -- 172.21.15.45:0/902001709 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7cb4089f70 con 0x7f7cc406aa40 2023-11-28T21:52:32.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.174+0000 7f7cbb7fe640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f7ca406af90 0x7f7ca406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:32.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.175+0000 7f7cbb7fe640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f7ca406af90 0x7f7ca406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7cac036000 tx=0x7f7cac030000 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:32.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.178+0000 7f7cc17fa640 1 -- 172.21.15.45:0/902001709 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7cb4054c80 con 0x7f7cc406aa40 2023-11-28T21:52:32.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.450+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7cc40caf40 con 0x7f7cc406aa40 2023-11-28T21:52:32.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.456+0000 7f7cc17fa640 1 -- 172.21.15.45:0/902001709 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v43) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7cb40582a0 con 0x7f7cc406aa40 2023-11-28T21:52:32.458 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T21:52:32.458 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T21:52:32.459 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T21:52:32.459 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T21:52:32.459 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T21:52:32.459 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T21:52:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.458+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f7ca406af90 msgr2=0x7f7ca406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.458+0000 7f7cca378640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f7ca406af90 0x7f7ca406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7cac036000 tx=0x7f7cac030000 comp rx=0 tx=0).stop 2023-11-28T21:52:32.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc406aa40 msgr2=0x7f7cc415f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc406aa40 0x7f7cc415f250 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400d800 tx=0x7f7cb400dcc0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 shutdown_connections 2023-11-28T21:52:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f7ca406af90 0x7f7ca406d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cc40ce600 0x7f7cc40c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7cc406dc50 0x7f7cc40c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 --2- 172.21.15.45:0/902001709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cc406aa40 0x7f7cc415f250 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 >> 172.21.15.45:0/902001709 conn(0x7f7cc40ced60 msgr2=0x7f7cc40cdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 shutdown_connections 2023-11-28T21:52:32.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.459+0000 7f7cca378640 1 -- 172.21.15.45:0/902001709 wait complete. 2023-11-28T21:52:32.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T21:52:32.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.706+0000 7f3d8359e640 1 Processor -- start 2023-11-28T21:52:32.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.707+0000 7f3d8359e640 1 -- start start 2023-11-28T21:52:32.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.708+0000 7f3d8359e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3d84155780 0x7f3d840abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.708+0000 7f3d8359e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3d840ce800 0x7f3d840b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.708+0000 7f3d8359e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8415c3c0 con 0x7f3d840ae540 2023-11-28T21:52:32.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.708+0000 7f3d8359e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d84155b60 con 0x7f3d840ce800 2023-11-28T21:52:32.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.708+0000 7f3d8359e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8415ba90 con 0x7f3d84155780 2023-11-28T21:52:32.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.709+0000 7f3d82d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3d840ce800 0x7f3d840b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:32.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.709+0000 7f3d82d9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3d840ce800 0x7f3d840b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50774/0 (socket says 172.21.15.45:50774) 2023-11-28T21:52:32.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.709+0000 7f3d82d9d640 1 -- 172.21.15.45:0/1658057244 learned_addr learned my addr 172.21.15.45:0/1658057244 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:32.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.709+0000 7f3d8259c640 1 --2- 172.21.15.45:0/1658057244 >> v2:172.21.15.190:3300/0 conn(0x7f3d84155780 0x7f3d840abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:32.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.709+0000 7f3d82d9d640 1 -- 172.21.15.45:0/1658057244 >> v2:172.21.15.190:3300/0 conn(0x7f3d84155780 msgr2=0x7f3d840abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:32.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.709+0000 7f3d82d9d640 1 --2- 172.21.15.45:0/1658057244 >> v2:172.21.15.190:3300/0 conn(0x7f3d84155780 0x7f3d840abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.709+0000 7f3d82d9d640 1 -- 172.21.15.45:0/1658057244 >> v1:172.21.15.190:6789/0 conn(0x7f3d840ae540 legacy=0x7f3d840ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:32.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.709+0000 7f3d82d9d640 1 -- 172.21.15.45:0/1658057244 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8415bd70 con 0x7f3d840ce800 2023-11-28T21:52:32.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.710+0000 7f3d82d9d640 1 --2- 172.21.15.45:0/1658057244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d840ce800 0x7f3d840b20a0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f3d8406e6d0 tx=0x7f3d7c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba01ce96313f86fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:32.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.711+0000 7f3d81d9b640 1 -- 172.21.15.45:0/1658057244 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c02ca80 con 0x7f3d840ce800 2023-11-28T21:52:32.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.711+0000 7f3d81d9b640 1 -- 172.21.15.45:0/1658057244 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c02cbe0 con 0x7f3d840ce800 2023-11-28T21:52:32.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.711+0000 7f3d81d9b640 1 -- 172.21.15.45:0/1658057244 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c0343f0 con 0x7f3d840ce800 2023-11-28T21:52:32.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.712+0000 7f3d8359e640 1 -- 172.21.15.45:0/1658057244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d840ce800 msgr2=0x7f3d840b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.712+0000 7f3d8359e640 1 --2- 172.21.15.45:0/1658057244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d840ce800 0x7f3d840b20a0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f3d8406e6d0 tx=0x7f3d7c02bee0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.712+0000 7f3d8359e640 1 -- 172.21.15.45:0/1658057244 shutdown_connections 2023-11-28T21:52:32.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.712+0000 7f3d8359e640 1 --2- 172.21.15.45:0/1658057244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d840ce800 0x7f3d840b20a0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.712+0000 7f3d8359e640 1 --2- 172.21.15.45:0/1658057244 >> v2:172.21.15.190:3300/0 conn(0x7f3d84155780 0x7f3d840abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.712+0000 7f3d8359e640 1 -- 172.21.15.45:0/1658057244 >> 172.21.15.45:0/1658057244 conn(0x7f3d8406ab30 msgr2=0x7f3d840b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:32.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.712+0000 7f3d8359e640 1 -- 172.21.15.45:0/1658057244 shutdown_connections 2023-11-28T21:52:32.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.712+0000 7f3d8359e640 1 -- 172.21.15.45:0/1658057244 wait complete. 2023-11-28T21:52:32.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.714+0000 7f3d8359e640 1 Processor -- start 2023-11-28T21:52:32.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.714+0000 7f3d8359e640 1 -- start start 2023-11-28T21:52:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d8359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d840ae540 0x7f3d841431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d8359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d840ce800 0x7f3d840aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d8359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84155780 0x7f3d840af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d8359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d840ce020 con 0x7f3d84155780 2023-11-28T21:52:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d8359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d84146e50 con 0x7f3d840ae540 2023-11-28T21:52:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d8359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d841436e0 con 0x7f3d840ce800 2023-11-28T21:52:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d82d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84155780 0x7f3d840af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:32.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d82d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84155780 0x7f3d840af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50784/0 (socket says 172.21.15.45:50784) 2023-11-28T21:52:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d82d9d640 1 -- 172.21.15.45:0/1152762546 learned_addr learned my addr 172.21.15.45:0/1152762546 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.715+0000 7f3d8259c640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d840ae540 0x7f3d841431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d82d9d640 1 -- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d840ce800 msgr2=0x7f3d840aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d82d9d640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d840ce800 0x7f3d840aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d82d9d640 1 -- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d840ae540 msgr2=0x7f3d841431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d82d9d640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d840ae540 0x7f3d841431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d82d9d640 1 -- 172.21.15.45:0/1152762546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c009b30 con 0x7f3d84155780 2023-11-28T21:52:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d8259c640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d840ae540 0x7f3d841431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d82d9d640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84155780 0x7f3d840af320 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009f40 tx=0x7f3d7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d7affd640 1 -- 172.21.15.45:0/1152762546 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c009510 con 0x7f3d84155780 2023-11-28T21:52:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d7affd640 1 -- 172.21.15.45:0/1152762546 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c03e450 con 0x7f3d84155780 2023-11-28T21:52:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d7affd640 1 -- 172.21.15.45:0/1152762546 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c03e7e0 con 0x7f3d84155780 2023-11-28T21:52:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d84145010 con 0x7f3d84155780 2023-11-28T21:52:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.716+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d841455d0 con 0x7f3d84155780 2023-11-28T21:52:32.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.718+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d841457f0 con 0x7f3d84155780 2023-11-28T21:52:32.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.718+0000 7f3d7affd640 1 -- 172.21.15.45:0/1152762546 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3d7c02cd50 con 0x7f3d84155780 2023-11-28T21:52:32.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.719+0000 7f3d7affd640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3d6406af60 0x7f3d6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.720+0000 7f3d7affd640 1 -- 172.21.15.45:0/1152762546 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3d7c0ac510 con 0x7f3d84155780 2023-11-28T21:52:32.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.720+0000 7f3d8259c640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3d6406af60 0x7f3d6406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:32.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.721+0000 7f3d8259c640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3d6406af60 0x7f3d6406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d74006fd0 tx=0x7f3d74008040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:32.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.724+0000 7f3d7affd640 1 -- 172.21.15.45:0/1152762546 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d7c0772b0 con 0x7f3d84155780 2023-11-28T21:52:32.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.961+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8404e920 con 0x7f3d84155780 2023-11-28T21:52:32.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.962+0000 7f3d7affd640 1 -- 172.21.15.45:0/1152762546 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f3d7c07a8d0 con 0x7f3d84155780 2023-11-28T21:52:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T21:52:32.966 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3312844962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3312844962},{"type":"v1","addr":"172.21.15.190:6835","nonce":3312844962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":8296,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1302790439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1302790439},{"type":"v1","addr":"172.21.15.190:6837","nonce":1302790439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10104,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2082498185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2082498185},{"type":"v1","addr":"172.21.15.45:6834","nonce":2082498185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:52:25.939261+0000","modified":"2023-11-28T21:52:29.372705+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10110":{"gid":10110,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3063278837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3063278837},{"type":"v1","addr":"172.21.15.45:6835","nonce":3063278837}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-28T21:52:32.966 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 92 2023-11-28T21:52:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3d6406af60 msgr2=0x7f3d6406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3d6406af60 0x7f3d6406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d74006fd0 tx=0x7f3d74008040 comp rx=0 tx=0).stop 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84155780 msgr2=0x7f3d840af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84155780 0x7f3d840af320 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009f40 tx=0x7f3d7c004a40 comp rx=0 tx=0).stop 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 shutdown_connections 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3d6406af60 0x7f3d6406d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84155780 0x7f3d840af320 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d840ce800 0x7f3d840aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 --2- 172.21.15.45:0/1152762546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d840ae540 0x7f3d841431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 >> 172.21.15.45:0/1152762546 conn(0x7f3d8406ab30 msgr2=0x7f3d840b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 shutdown_connections 2023-11-28T21:52:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:32.965+0000 7f3d8359e640 1 -- 172.21.15.45:0/1152762546 wait complete. 2023-11-28T21:52:33.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T21:52:25.939261+0000', 'modified': '2023-11-28T21:52:29.372705+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10110}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10110': {'gid': 10110, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6835/3063278837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6833', 'nonce': 3063278837}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 3063278837}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T21:52:33.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T21:52:33.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-28T21:52:33.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.627+0000 7fcdd9343640 1 Processor -- start 2023-11-28T21:52:33.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.627+0000 7fcdd9343640 1 -- start start 2023-11-28T21:52:33.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.629+0000 7fcdd9343640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcdd415b820 0x7fcdd406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.629+0000 7fcdd9343640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcdd415f4d0 0x7fcdd40cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.629+0000 7fcdd9343640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd415fa80 con 0x7fcdd40cf020 2023-11-28T21:52:33.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.629+0000 7fcdd9343640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd415fd60 con 0x7fcdd415f4d0 2023-11-28T21:52:33.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.629+0000 7fcdd9343640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd40cf9f0 con 0x7fcdd415b820 2023-11-28T21:52:33.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.630+0000 7fcdd37fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcdd415f4d0 0x7fcdd40cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.630+0000 7fcdd37fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcdd415f4d0 0x7fcdd40cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50790/0 (socket says 172.21.15.45:50790) 2023-11-28T21:52:33.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.630+0000 7fcdd37fe640 1 -- 172.21.15.45:0/3112985277 learned_addr learned my addr 172.21.15.45:0/3112985277 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:33.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.630+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/3112985277 >> v2:172.21.15.190:3300/0 conn(0x7fcdd415b820 0x7fcdd406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.630+0000 7fcdd37fe640 1 -- 172.21.15.45:0/3112985277 >> v2:172.21.15.190:3300/0 conn(0x7fcdd415b820 msgr2=0x7fcdd406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:33.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.630+0000 7fcdd37fe640 1 --2- 172.21.15.45:0/3112985277 >> v2:172.21.15.190:3300/0 conn(0x7fcdd415b820 0x7fcdd406da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.630+0000 7fcdd37fe640 1 -- 172.21.15.45:0/3112985277 >> v1:172.21.15.45:6789/0 conn(0x7fcdd40cf020 legacy=0x7fcdd415edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.630+0000 7fcdd27fc640 1 -- 172.21.15.45:0/3112985277 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986577561 0 0) 0x7fcdd415fa80 con 0x7fcdd40cf020 2023-11-28T21:52:33.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.630+0000 7fcdd37fe640 1 -- 172.21.15.45:0/3112985277 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd406cf40 con 0x7fcdd415f4d0 2023-11-28T21:52:33.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.632+0000 7fcdd37fe640 1 --2- 172.21.15.45:0/3112985277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd415f4d0 0x7fcdd40cf4b0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40b40e0 tx=0x7fcdcc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9093b1955e92b25 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.632+0000 7fcdd27fc640 1 -- 172.21.15.45:0/3112985277 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc030860 con 0x7fcdd415f4d0 2023-11-28T21:52:33.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.632+0000 7fcdd27fc640 1 -- 172.21.15.45:0/3112985277 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc0309c0 con 0x7fcdd415f4d0 2023-11-28T21:52:33.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.632+0000 7fcdd27fc640 1 -- 172.21.15.45:0/3112985277 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc030d50 con 0x7fcdd415f4d0 2023-11-28T21:52:33.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.632+0000 7fcdd9343640 1 -- 172.21.15.45:0/3112985277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd415f4d0 msgr2=0x7fcdd40cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.632+0000 7fcdd9343640 1 --2- 172.21.15.45:0/3112985277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd415f4d0 0x7fcdd40cf4b0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40b40e0 tx=0x7fcdcc02b9a0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.633+0000 7fcdd9343640 1 -- 172.21.15.45:0/3112985277 shutdown_connections 2023-11-28T21:52:33.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.633+0000 7fcdd9343640 1 --2- 172.21.15.45:0/3112985277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd415f4d0 0x7fcdd40cf4b0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.633+0000 7fcdd9343640 1 --2- 172.21.15.45:0/3112985277 >> v2:172.21.15.190:3300/0 conn(0x7fcdd415b820 0x7fcdd406da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.633+0000 7fcdd9343640 1 -- 172.21.15.45:0/3112985277 >> 172.21.15.45:0/3112985277 conn(0x7fcdd406a6e0 msgr2=0x7fcdd406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:33.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.633+0000 7fcdd9343640 1 -- 172.21.15.45:0/3112985277 shutdown_connections 2023-11-28T21:52:33.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.633+0000 7fcdd9343640 1 -- 172.21.15.45:0/3112985277 wait complete. 2023-11-28T21:52:33.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.634+0000 7fcdd9343640 1 Processor -- start 2023-11-28T21:52:33.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.635+0000 7fcdd9343640 1 -- start start 2023-11-28T21:52:33.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd9343640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd40cf020 0x7fcdd414ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd9343640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdd415b820 0x7fcdd40c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd9343640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdd415f4d0 0x7fcdd41521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd9343640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4151d50 con 0x7fcdd40cf020 2023-11-28T21:52:33.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd9343640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4151180 con 0x7fcdd415b820 2023-11-28T21:52:33.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd9343640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4151420 con 0x7fcdd415f4d0 2023-11-28T21:52:33.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd40cf020 0x7fcdd414ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd40cf020 0x7fcdd414ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50802/0 (socket says 172.21.15.45:50802) 2023-11-28T21:52:33.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/1513480163 learned_addr learned my addr 172.21.15.45:0/1513480163 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:33.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdd415f4d0 msgr2=0x7fcdd41521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdd415f4d0 0x7fcdd41521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdd415b820 msgr2=0x7fcdd40c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdd415b820 0x7fcdd40c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.636+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/1513480163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc0094d0 con 0x7fcdd40cf020 2023-11-28T21:52:33.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.637+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd40cf020 0x7fcdd414ea80 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4007ef0 tx=0x7fcdc400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.637+0000 7fcdcbfff640 1 -- 172.21.15.45:0/1513480163 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4014620 con 0x7fcdd40cf020 2023-11-28T21:52:33.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.637+0000 7fcdcbfff640 1 -- 172.21.15.45:0/1513480163 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc4014780 con 0x7fcdd40cf020 2023-11-28T21:52:33.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.637+0000 7fcdcbfff640 1 -- 172.21.15.45:0/1513480163 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4014b10 con 0x7fcdd40cf020 2023-11-28T21:52:33.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.637+0000 7fcdd9343640 1 -- 172.21.15.45:0/1513480163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd4151700 con 0x7fcdd40cf020 2023-11-28T21:52:33.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.637+0000 7fcdd9343640 1 -- 172.21.15.45:0/1513480163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd414efc0 con 0x7fcdd40cf020 2023-11-28T21:52:33.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.639+0000 7fcdcbfff640 1 -- 172.21.15.45:0/1513480163 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fcdc4014c70 con 0x7fcdd40cf020 2023-11-28T21:52:33.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.640+0000 7fcdcbfff640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fcdb406af90 0x7fcdb406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.640+0000 7fcdcbfff640 1 -- 172.21.15.45:0/1513480163 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcdc408a2d0 con 0x7fcdd40cf020 2023-11-28T21:52:33.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.641+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fcdb406af90 0x7fcdb406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.641+0000 7fcddbfbc740 1 Processor -- start 2023-11-28T21:52:33.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.642+0000 7fcddbfbc740 1 -- start start 2023-11-28T21:52:33.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.642+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fcdb406af90 0x7fcdb406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40cae80 tx=0x7fcdbc007910 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e82260280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e822a3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e821be880 con 0x557e8225aba0 2023-11-28T21:52:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e821cdae0 con 0x557e8225f5e0 2023-11-28T21:52:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x557e821cb1d0 con 0x557e8225fea0 2023-11-28T21:52:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcdca7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcdca7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50814/0 (socket says 172.21.15.45:50814) 2023-11-28T21:52:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcdca7fc640 1 -- 172.21.15.45:0/3558464086 learned_addr learned my addr 172.21.15.45:0/3558464086 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/3558464086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e82260280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.643+0000 7fcdd37fe640 1 --2- 172.21.15.45:0/3558464086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e822a3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.644+0000 7fcdca7fc640 1 -- 172.21.15.45:0/3558464086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 msgr2=0x557e82260280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:33.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.644+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/3558464086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e82260280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.644+0000 7fcdca7fc640 1 -- 172.21.15.45:0/3558464086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 msgr2=0x557e822a3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:33.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.644+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/3558464086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e822a3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.644+0000 7fcdca7fc640 1 -- 172.21.15.45:0/3558464086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e8218e860 con 0x557e8225aba0 2023-11-28T21:52:33.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.644+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/3558464086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225f0a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc005520 tx=0x7fcdbc00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287a34da3c62126 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.644+0000 7fcdc8ff9640 1 -- 172.21.15.45:0/3558464086 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc01b070 con 0x557e8225aba0 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.644+0000 7fcdc8ff9640 1 -- 172.21.15.45:0/3558464086 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc012650 con 0x557e8225aba0 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.644+0000 7fcdc8ff9640 1 -- 172.21.15.45:0/3558464086 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0129c0 con 0x557e8225aba0 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.645+0000 7fcddbfbc740 1 -- 172.21.15.45:0/3558464086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 msgr2=0x557e8225f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.645+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/3558464086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225f0a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc005520 tx=0x7fcdbc00f000 comp rx=0 tx=0).stop 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.645+0000 7fcddbfbc740 1 -- 172.21.15.45:0/3558464086 shutdown_connections 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.645+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/3558464086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e822a3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.645+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/3558464086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225f0a0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.645+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/3558464086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e82260280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.645+0000 7fcddbfbc740 1 -- 172.21.15.45:0/3558464086 >> 172.21.15.45:0/3558464086 conn(0x557e82221c70 msgr2=0x557e822a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.645+0000 7fcddbfbc740 1 -- 172.21.15.45:0/3558464086 shutdown_connections 2023-11-28T21:52:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.645+0000 7fcddbfbc740 1 -- 172.21.15.45:0/3558464086 wait complete. 2023-11-28T21:52:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.646+0000 7fcddbfbc740 1 -- start start 2023-11-28T21:52:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.646+0000 7fcddbfbc740 1 Processor -- start 2023-11-28T21:52:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.647+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82273ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.647+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e822ac920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.647+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e82289320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.647+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e821c6ac0 con 0x557e8225aba0 2023-11-28T21:52:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.647+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e821632f0 con 0x557e8225f5e0 2023-11-28T21:52:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.647+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x557e82180c70 con 0x557e8225fea0 2023-11-28T21:52:33.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.647+0000 7fcddbfbc740 10 client.0 _ll_register_callbacks cb 0x557e821853b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82273ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82273ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50824/0 (socket says 172.21.15.45:50824) 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/2001985569 learned_addr learned my addr 172.21.15.45:0/2001985569 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e822ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd37fe640 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e82289320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/2001985569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 msgr2=0x557e82289320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e82289320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/2001985569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 msgr2=0x557e822ac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e822ac920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/2001985569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc005170 con 0x557e8225aba0 2023-11-28T21:52:33.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.648+0000 7fcdd37fe640 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e82289320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82273ce0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e9b0 tx=0x7fcdc40594d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94aeb9156339e4c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcdb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/2001985569 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc405cf20 con 0x557e8225aba0 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/2001985569 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcdc40610d0 con 0x557e8225aba0 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcddbfbc740 1 -- 172.21.15.45:0/2001985569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e821bfaa0 con 0x557e8225aba0 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/2001985569 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdc40882a0 con 0x557e8225aba0 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcddbfbc740 10 client.10245 fetch_fsmap learned FSMap version 92 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcddbfbc740 1 -- 172.21.15.45:0/2001985569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x557e821c5140 con 0x557e8225aba0 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/2001985569 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557e821bfaa0 con 0x557e8225aba0 2023-11-28T21:52:33.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/2001985569 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 92) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcdc40554e0 con 0x557e8225aba0 2023-11-28T21:52:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcddbfbc740 10 client.10245 fetch_fsmap finished waiting for FSMap version 92 2023-11-28T21:52:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.649+0000 7fcddbfbc740 10 client.10245 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T21:52:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.650+0000 7fcddbfbc740 20 client.10245 populate_metadata read hostname 'smithi045' 2023-11-28T21:52:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.650+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] conn(0x557e822882a0 0x557e822c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.650+0000 7fcddbfbc740 4 client.10245 mds_command: new command op to 10110 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T21:52:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.650+0000 7fcddbfbc740 1 -- 172.21.15.45:0/2001985569 --> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e821bf7c0 con 0x557e822882a0 2023-11-28T21:52:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.650+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] conn(0x557e822882a0 0x557e822c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.650+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] conn(0x557e822882a0 0x557e822c2f70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55d30ee3cfa35f3 server_cookie=399282f01d5ba960 in_seq=0 out_seq=0 2023-11-28T21:52:33.656 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.650+0000 7fcdb1ffb640 10 client.10245 ms_handle_connect on v2:172.21.15.45:6833/3063278837 2023-11-28T21:52:33.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.651+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/2001985569 <== mds.0 v2:172.21.15.45:6833/3063278837 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557e821bf7c0 con 0x557e822882a0 2023-11-28T21:52:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.651+0000 7fcdb1ffb640 10 client.10245 handle_command_reply: tid=0 2023-11-28T21:52:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.651+0000 7fcddbfbc740 1 client.10245 shutdown 2023-11-28T21:52:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.651+0000 7fcddbfbc740 1 -- 172.21.15.45:0/2001985569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 msgr2=0x557e82273ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.651+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82273ce0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e9b0 tx=0x7fcdc40594d0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcddbfbc740 1 -- 172.21.15.45:0/2001985569 shutdown_connections 2023-11-28T21:52:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] conn(0x557e822882a0 0x557e822c2f70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e82289320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e822ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/2001985569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82273ce0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcddbfbc740 1 -- 172.21.15.45:0/2001985569 >> 172.21.15.45:0/2001985569 conn(0x557e82221c70 msgr2=0x557e82259b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcdb1ffb640 0 client.10245 ms_handle_reset on v2:172.21.15.45:6833/3063278837 2023-11-28T21:52:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcddbfbc740 1 -- 172.21.15.45:0/2001985569 shutdown_connections 2023-11-28T21:52:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcddbfbc740 1 -- 172.21.15.45:0/2001985569 wait complete. 2023-11-28T21:52:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.652+0000 7fcddbfbc740 20 client.10245 trim_cache size 0 max 16384 2023-11-28T21:52:33.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.669+0000 7fcddbfbc740 1 Processor -- start 2023-11-28T21:52:33.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.669+0000 7fcddbfbc740 1 -- start start 2023-11-28T21:52:33.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.670+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82270450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.670+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e82270990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.670+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e822a8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.670+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e81f29f20 con 0x557e8225aba0 2023-11-28T21:52:33.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.670+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8225a370 con 0x557e8225f5e0 2023-11-28T21:52:33.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.670+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x557e82256310 con 0x557e8225fea0 2023-11-28T21:52:33.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82270450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82270450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50836/0 (socket says 172.21.15.45:50836) 2023-11-28T21:52:33.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/677697336 learned_addr learned my addr 172.21.15.45:0/677697336 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:33.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e82270990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.674 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 msgr2=0x557e822a8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:33.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd37fe640 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e822a8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e822a8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 msgr2=0x557e82270990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:33.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e82270990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/677697336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e82162b80 con 0x557e8225aba0 2023-11-28T21:52:33.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e82270990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:33.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.671+0000 7fcdd37fe640 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e822a8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:33.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.672+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82270450 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4058d60 tx=0x7fcdc408a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254414571c170fac server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.672+0000 7fcdc97fa640 1 -- 172.21.15.45:0/677697336 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc405cf20 con 0x557e8225aba0 2023-11-28T21:52:33.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.672+0000 7fcdc97fa640 1 -- 172.21.15.45:0/677697336 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc40610d0 con 0x557e8225aba0 2023-11-28T21:52:33.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.672+0000 7fcdc97fa640 1 -- 172.21.15.45:0/677697336 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc40882a0 con 0x557e8225aba0 2023-11-28T21:52:33.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.672+0000 7fcddbfbc740 1 -- 172.21.15.45:0/677697336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 msgr2=0x557e82270450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.672+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82270450 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4058d60 tx=0x7fcdc408a000 comp rx=0 tx=0).stop 2023-11-28T21:52:33.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.673+0000 7fcddbfbc740 1 -- 172.21.15.45:0/677697336 shutdown_connections 2023-11-28T21:52:33.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.673+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e822a8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.673+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e82270990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.673+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/677697336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e82270450 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.673+0000 7fcddbfbc740 1 -- 172.21.15.45:0/677697336 >> 172.21.15.45:0/677697336 conn(0x557e82221c70 msgr2=0x557e82285b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:33.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.673+0000 7fcddbfbc740 1 -- 172.21.15.45:0/677697336 shutdown_connections 2023-11-28T21:52:33.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.673+0000 7fcddbfbc740 1 -- 172.21.15.45:0/677697336 wait complete. 2023-11-28T21:52:33.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.673+0000 7fcddbfbc740 1 -- start start 2023-11-28T21:52:33.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.674+0000 7fcddbfbc740 1 Processor -- start 2023-11-28T21:52:33.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e8230f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcddbfbc740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e822a9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e822b7a90 con 0x557e8225aba0 2023-11-28T21:52:33.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e822603f0 con 0x557e8225f5e0 2023-11-28T21:52:33.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcddbfbc740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x557e8230fdb0 con 0x557e8225fea0 2023-11-28T21:52:33.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcddbfbc740 10 client.0 _ll_register_callbacks cb 0x7fcdc40587a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T21:52:33.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50842/0 (socket says 172.21.15.45:50842) 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/350878158 learned_addr learned my addr 172.21.15.45:0/350878158 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/350878158 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 msgr2=0x557e822a9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e822a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/350878158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 msgr2=0x557e8230f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e8230f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.675+0000 7fcdd2ffd640 1 -- 172.21.15.45:0/350878158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e821c5140 con 0x557e8225aba0 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcdd2ffd640 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225a270 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e9b0 tx=0x7fcdc4016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8bbb80ef576b29c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcdb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/350878158 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc405cf20 con 0x557e8225aba0 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/350878158 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcdc4060e40 con 0x557e8225aba0 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcddbfbc740 1 -- 172.21.15.45:0/350878158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcdbc012650 con 0x557e8225aba0 2023-11-28T21:52:33.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/350878158 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdc406cb20 con 0x557e8225aba0 2023-11-28T21:52:33.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcddbfbc740 10 client.10251 fetch_fsmap learned FSMap version 92 2023-11-28T21:52:33.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcddbfbc740 1 -- 172.21.15.45:0/350878158 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x557e82188170 con 0x557e8225aba0 2023-11-28T21:52:33.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/350878158 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc012650 con 0x557e8225aba0 2023-11-28T21:52:33.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.676+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/350878158 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 92) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcdc407d070 con 0x557e8225aba0 2023-11-28T21:52:33.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.677+0000 7fcddbfbc740 10 client.10251 fetch_fsmap finished waiting for FSMap version 92 2023-11-28T21:52:33.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.677+0000 7fcddbfbc740 10 client.10251 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T21:52:33.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.677+0000 7fcddbfbc740 20 client.10251 populate_metadata read hostname 'smithi045' 2023-11-28T21:52:33.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.677+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] conn(0x557e822ada10 0x557e82313450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.677+0000 7fcddbfbc740 4 client.10251 mds_command: new command op to 10110 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T21:52:33.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.677+0000 7fcddbfbc740 1 -- 172.21.15.45:0/350878158 --> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557e81ec76c0 con 0x557e822ada10 2023-11-28T21:52:33.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.677+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] conn(0x557e822ada10 0x557e82313450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.677+0000 7fcdca7fc640 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] conn(0x557e822ada10 0x557e82313450 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91cf6df44b3c4749 server_cookie=12ad4edf95757fea in_seq=0 out_seq=0 2023-11-28T21:52:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.677+0000 7fcdb1ffb640 10 client.10251 ms_handle_connect on v2:172.21.15.45:6833/3063278837 2023-11-28T21:52:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcdb1ffb640 1 -- 172.21.15.45:0/350878158 <== mds.0 v2:172.21.15.45:6833/3063278837 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x557e81ec76c0 con 0x557e822ada10 2023-11-28T21:52:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcdb1ffb640 10 client.10251 handle_command_reply: tid=0 2023-11-28T21:52:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 client.10251 shutdown 2023-11-28T21:52:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 -- 172.21.15.45:0/350878158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 msgr2=0x557e8225a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225a270 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e9b0 tx=0x7fcdc4016040 comp rx=0 tx=0).stop 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 -- 172.21.15.45:0/350878158 shutdown_connections 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.45:6833/3063278837,v1:172.21.15.45:6835/3063278837] conn(0x557e822ada10 0x557e82313450 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x557e8225fea0 0x557e822a9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557e8225f5e0 0x557e8230f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 --2- 172.21.15.45:0/350878158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e8225aba0 0x557e8225a270 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcdb1ffb640 0 client.10251 ms_handle_reset on v2:172.21.15.45:6833/3063278837 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 -- 172.21.15.45:0/350878158 >> 172.21.15.45:0/350878158 conn(0x557e82221c70 msgr2=0x557e822a4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 -- 172.21.15.45:0/350878158 shutdown_connections 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.678+0000 7fcddbfbc740 1 -- 172.21.15.45:0/350878158 wait complete. 2023-11-28T21:52:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcddbfbc740 20 client.10251 trim_cache size 0 max 16384 2023-11-28T21:52:33.687 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T21:52:33.687 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T21:52:33.687 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T21:52:33.687 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 10110, 2023-11-28T21:52:33.687 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T21:52:33.687 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T21:52:33.688 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T21:52:33.688 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.8215334199999997, 2023-11-28T21:52:33.688 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 92, 2023-11-28T21:52:33.688 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 199, 2023-11-28T21:52:33.688 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 199, 2023-11-28T21:52:33.688 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 13.554468497 2023-11-28T21:52:33.688 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T21:52:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 -- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fcdb406af90 msgr2=0x7fcdb406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fcdb406af90 0x7fcdb406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40cae80 tx=0x7fcdbc007910 comp rx=0 tx=0).stop 2023-11-28T21:52:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 -- 172.21.15.45:0/1513480163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd40cf020 msgr2=0x7fcdd414ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd40cf020 0x7fcdd414ea80 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4007ef0 tx=0x7fcdc400e4b0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 -- 172.21.15.45:0/1513480163 shutdown_connections 2023-11-28T21:52:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fcdb406af90 0x7fcdb406d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcdd415f4d0 0x7fcdd41521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdd415b820 0x7fcdd40c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 --2- 172.21.15.45:0/1513480163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd40cf020 0x7fcdd414ea80 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 -- 172.21.15.45:0/1513480163 >> 172.21.15.45:0/1513480163 conn(0x7fcdd406a6e0 msgr2=0x7fcdd40cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 -- 172.21.15.45:0/1513480163 shutdown_connections 2023-11-28T21:52:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.679+0000 7fcdd9343640 1 -- 172.21.15.45:0/1513480163 wait complete. 2023-11-28T21:52:33.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T21:52:33.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.914+0000 7f3fb924f640 1 Processor -- start 2023-11-28T21:52:33.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.915+0000 7f3fb924f640 1 -- start start 2023-11-28T21:52:33.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.917+0000 7f3fb924f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3fb40b10b0 0x7f3fb40718a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.917+0000 7f3fb924f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb40ce710 con 0x7f3fb40b0c30 2023-11-28T21:52:33.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.917+0000 7f3fb924f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb40ae2c0 con 0x7f3fb415d320 2023-11-28T21:52:33.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.917+0000 7f3fb924f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb40ae5a0 con 0x7f3fb40b10b0 2023-11-28T21:52:33.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.917+0000 7f3fb259c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3fb40b0c30 0x7f3fb4071190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51284/0 (socket says 172.21.15.45:51284) 2023-11-28T21:52:33.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.917+0000 7f3fb259c640 1 -- 172.21.15.45:0/148170180 learned_addr learned my addr 172.21.15.45:0/148170180 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.918+0000 7f3fb359e640 1 --2- 172.21.15.45:0/148170180 >> v2:172.21.15.45:3300/0 conn(0x7f3fb40b10b0 0x7f3fb40718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.918+0000 7f3fb1d9b640 1 -- 172.21.15.45:0/148170180 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043960518 0 0) 0x7f3fb40ce710 con 0x7f3fb40b0c30 2023-11-28T21:52:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.918+0000 7f3fb1d9b640 1 -- 172.21.15.45:0/148170180 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f9c003520 con 0x7f3fb40b0c30 2023-11-28T21:52:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.918+0000 7f3fb1d9b640 1 -- 172.21.15.45:0/148170180 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272577737 0 0) 0x7f3fb40ae2c0 con 0x7f3fb415d320 2023-11-28T21:52:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.918+0000 7f3fb359e640 1 -- 172.21.15.45:0/148170180 >> v1:172.21.15.190:6789/0 conn(0x7f3fb415d320 legacy=0x7f3fb40ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.918+0000 7f3fb359e640 1 -- 172.21.15.45:0/148170180 >> v1:172.21.15.45:6789/0 conn(0x7f3fb40b0c30 legacy=0x7f3fb4071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.918+0000 7f3fb359e640 1 -- 172.21.15.45:0/148170180 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb41611f0 con 0x7f3fb40b10b0 2023-11-28T21:52:33.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.920+0000 7f3fb359e640 1 --2- 172.21.15.45:0/148170180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b10b0 0x7f3fb40718a0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009d90 tx=0x7f3fac02bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf3b246b2ae5a79 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.920+0000 7f3fb1d9b640 1 -- 172.21.15.45:0/148170180 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8001670 con 0x7f3fb40b10b0 2023-11-28T21:52:33.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.920+0000 7f3fb1d9b640 1 -- 172.21.15.45:0/148170180 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac02c860 con 0x7f3fb40b10b0 2023-11-28T21:52:33.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.920+0000 7f3fb1d9b640 1 -- 172.21.15.45:0/148170180 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac02cbf0 con 0x7f3fb40b10b0 2023-11-28T21:52:33.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.921+0000 7f3fb924f640 1 -- 172.21.15.45:0/148170180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b10b0 msgr2=0x7f3fb40718a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.921+0000 7f3fb924f640 1 --2- 172.21.15.45:0/148170180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b10b0 0x7f3fb40718a0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009d90 tx=0x7f3fac02bd30 comp rx=0 tx=0).stop 2023-11-28T21:52:33.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.921+0000 7f3fb924f640 1 -- 172.21.15.45:0/148170180 shutdown_connections 2023-11-28T21:52:33.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.921+0000 7f3fb924f640 1 --2- 172.21.15.45:0/148170180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b10b0 0x7f3fb40718a0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.921+0000 7f3fb924f640 1 -- 172.21.15.45:0/148170180 >> 172.21.15.45:0/148170180 conn(0x7f3fb406a500 msgr2=0x7f3fb406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:33.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.921+0000 7f3fb924f640 1 -- 172.21.15.45:0/148170180 shutdown_connections 2023-11-28T21:52:33.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.921+0000 7f3fb924f640 1 -- 172.21.15.45:0/148170180 wait complete. 2023-11-28T21:52:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.923+0000 7f3fb924f640 1 Processor -- start 2023-11-28T21:52:33.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.923+0000 7f3fb924f640 1 -- start start 2023-11-28T21:52:33.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.924+0000 7f3fb924f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b0c30 0x7f3fb4147db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.924+0000 7f3fb924f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb40b10b0 0x7f3fb4144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.924+0000 7f3fb924f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3fb415d320 0x7f3fb4146920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.924+0000 7f3fb924f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4146490 con 0x7f3fb40b0c30 2023-11-28T21:52:33.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.924+0000 7f3fb924f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb41482f0 con 0x7f3fb40b10b0 2023-11-28T21:52:33.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.924+0000 7f3fb924f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4145020 con 0x7f3fb415d320 2023-11-28T21:52:33.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.924+0000 7f3fb2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b0c30 0x7f3fb4147db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b0c30 0x7f3fb4147db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34008/0 (socket says 172.21.15.45:34008) 2023-11-28T21:52:33.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb2d9d640 1 -- 172.21.15.45:0/1665860796 learned_addr learned my addr 172.21.15.45:0/1665860796 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb259c640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb40b10b0 0x7f3fb4144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb359e640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3fb415d320 0x7f3fb4146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb2d9d640 1 -- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3fb415d320 msgr2=0x7f3fb4146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb2d9d640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3fb415d320 0x7f3fb4146920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb2d9d640 1 -- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb40b10b0 msgr2=0x7f3fb4144ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb2d9d640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb40b10b0 0x7f3fb4144ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb2d9d640 1 -- 172.21.15.45:0/1665860796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac009a70 con 0x7f3fb40b0c30 2023-11-28T21:52:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb359e640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3fb415d320 0x7f3fb4146920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.925+0000 7f3fb2d9d640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b0c30 0x7f3fb4147db0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400c4e0 tx=0x7f3fa400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.926+0000 7f3f9b7fe640 1 -- 172.21.15.45:0/1665860796 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4005480 con 0x7f3fb40b0c30 2023-11-28T21:52:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.926+0000 7f3f9b7fe640 1 -- 172.21.15.45:0/1665860796 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa40035f0 con 0x7f3fb40b0c30 2023-11-28T21:52:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.926+0000 7f3f9b7fe640 1 -- 172.21.15.45:0/1665860796 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa40039c0 con 0x7f3fb40b0c30 2023-11-28T21:52:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.926+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb4145880 con 0x7f3fb40b0c30 2023-11-28T21:52:33.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.926+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb4145e40 con 0x7f3fb40b0c30 2023-11-28T21:52:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.927+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb40733d0 con 0x7f3fb40b0c30 2023-11-28T21:52:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.928+0000 7f3f9b7fe640 1 -- 172.21.15.45:0/1665860796 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3fa4003b20 con 0x7f3fb40b0c30 2023-11-28T21:52:33.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.929+0000 7f3f9b7fe640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3fa006af90 0x7f3fa006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:33.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.929+0000 7f3f9b7fe640 1 -- 172.21.15.45:0/1665860796 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3fa4088dc0 con 0x7f3fb40b0c30 2023-11-28T21:52:33.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.929+0000 7f3fb259c640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3fa006af90 0x7f3fa006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:33.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.930+0000 7f3fb259c640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3fa006af90 0x7f3fa006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80017a0 tx=0x7f3fa8009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:33.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:33.933+0000 7f3f9b7fe640 1 -- 172.21.15.45:0/1665860796 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3fa4053c10 con 0x7f3fb40b0c30 2023-11-28T21:52:34.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.166+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3fb4159550 con 0x7f3fb40b0c30 2023-11-28T21:52:34.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.167+0000 7f3f9b7fe640 1 -- 172.21.15.45:0/1665860796 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v92) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3fa4017070 con 0x7f3fb40b0c30 2023-11-28T21:52:34.169 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T21:52:34.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3fa006af90 msgr2=0x7f3fa006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3fa006af90 0x7f3fa006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80017a0 tx=0x7f3fa8009040 comp rx=0 tx=0).stop 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b0c30 msgr2=0x7f3fb4147db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b0c30 0x7f3fb4147db0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400c4e0 tx=0x7f3fa400c9a0 comp rx=0 tx=0).stop 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 shutdown_connections 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f3fa006af90 0x7f3fa006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3fb415d320 0x7f3fb4146920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fb40b10b0 0x7f3fb4144ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 --2- 172.21.15.45:0/1665860796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fb40b0c30 0x7f3fb4147db0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 >> 172.21.15.45:0/1665860796 conn(0x7f3fb406a500 msgr2=0x7f3fb40ad470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:34.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 shutdown_connections 2023-11-28T21:52:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:34.169+0000 7f3fb924f640 1 -- 172.21.15.45:0/1665860796 wait complete. 2023-11-28T21:52:34.190 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T21:52:34.190 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T21:52:34.190 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T21:52:34.190 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T21:52:34.191 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T21:52:34.191 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T21:52:34.191 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T21:52:34.191 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T21:52:34.191 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T21:52:34.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:34.191 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T21:52:34.248 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T21:52:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 37588sec preferred_lft 37588sec 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T21:52:34.250 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T21:52:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T21:52:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T21:52:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:52:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T21:52:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:52:34.251 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T21:52:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T21:52:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T21:52:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:52:34.252 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T21:52:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T21:52:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T21:52:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T21:52:34.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:34.253 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T21:52:34.347 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T21:52:34.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:34.352 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T21:52:34.411 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T21:52:34.411 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T21:52:34.411 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T21:52:34.411 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T21:52:34.411 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T21:52:34.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:34.412 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T21:52:34.412 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T21:52:34.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T21:52:34.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T21:52:34.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T21:52:34.412 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T21:52:34.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:34.615 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T21:52:34.670 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T21:52:34.670 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T21:52:34.671 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T21:52:34.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:34.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T21:52:34.775 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T21:52:34.775 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T21:52:34.835 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T21:52:34.836 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T21:52:34.836 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T21:52:34.892 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T21:52:34.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:34.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T21:52:35.046 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T21:52:35.046 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T21:52:35.047 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T21:52:35.069 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T21:52:35.069 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T21:52:35.069 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T21:52:35.069 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T21:52:35.070 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T21:52:35.095 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T21:52:35.095 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T21:52:35.095 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T21:52:35.095 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T21:52:35.095 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T21:52:35.096 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T21:52:35.096 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T21:52:35.096 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T21:52:35.097 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T21:52:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T21:52:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T21:52:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T21:52:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T21:52:35.102 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T21:52:35.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:35.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T21:52:35.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:35.202 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T21:52:35.274 INFO:teuthology.orchestra.run.smithi045.stdout:client10263 2023-11-28T21:52:35.274 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T21:52:35.274 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T21:52:35.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T21:52:35.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.560+0000 7fca91878640 1 Processor -- start 2023-11-28T21:52:35.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.560+0000 7fca91878640 1 -- start start 2023-11-28T21:52:35.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.562+0000 7fca91878640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fca8c15d310 0x7fca8c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:35.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.562+0000 7fca91878640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fca8c160e70 0x7fca8c0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:35.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.562+0000 7fca91878640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca8c0ce7d0 con 0x7fca8c0ce3a0 2023-11-28T21:52:35.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.562+0000 7fca91878640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c0d14d0 con 0x7fca8c15d310 2023-11-28T21:52:35.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.562+0000 7fca91878640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c1613c0 con 0x7fca8c160e70 2023-11-28T21:52:35.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.562+0000 7fca91077640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fca8c0ce3a0 0x7fca8c0b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51286/0 (socket says 172.21.15.45:51286) 2023-11-28T21:52:35.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.562+0000 7fca91077640 1 -- 172.21.15.45:0/3092401483 learned_addr learned my addr 172.21.15.45:0/3092401483 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:35.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.562+0000 7fca90876640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fca8c15d310 0x7fca8c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:35.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.563+0000 7fca8bfff640 1 --2- 172.21.15.45:0/3092401483 >> v2:172.21.15.190:3300/0 conn(0x7fca8c160e70 0x7fca8c0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:35.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.563+0000 7fca8b7fe640 1 -- 172.21.15.45:0/3092401483 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419789054 0 0) 0x7fca8c0ce7d0 con 0x7fca8c0ce3a0 2023-11-28T21:52:35.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.563+0000 7fca8b7fe640 1 -- 172.21.15.45:0/3092401483 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca7c003520 con 0x7fca8c0ce3a0 2023-11-28T21:52:35.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.563+0000 7fca90876640 1 -- 172.21.15.45:0/3092401483 >> v2:172.21.15.190:3300/0 conn(0x7fca8c160e70 msgr2=0x7fca8c0cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:35.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.563+0000 7fca90876640 1 --2- 172.21.15.45:0/3092401483 >> v2:172.21.15.190:3300/0 conn(0x7fca8c160e70 0x7fca8c0cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.563+0000 7fca90876640 1 -- 172.21.15.45:0/3092401483 >> v1:172.21.15.45:6789/0 conn(0x7fca8c0ce3a0 legacy=0x7fca8c0b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:35.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.563+0000 7fca90876640 1 -- 172.21.15.45:0/3092401483 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c1616a0 con 0x7fca8c15d310 2023-11-28T21:52:35.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.563+0000 7fca8bfff640 1 --2- 172.21.15.45:0/3092401483 >> v2:172.21.15.190:3300/0 conn(0x7fca8c160e70 0x7fca8c0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:35.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.564+0000 7fca90876640 1 --2- 172.21.15.45:0/3092401483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 0x7fca8c0b8a20 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0d4130 tx=0x7fca8002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9229559966fb9a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:35.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca8b7fe640 1 -- 172.21.15.45:0/3092401483 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8002e070 con 0x7fca8c15d310 2023-11-28T21:52:35.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca8b7fe640 1 -- 172.21.15.45:0/3092401483 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca800315a0 con 0x7fca8c15d310 2023-11-28T21:52:35.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca8b7fe640 1 -- 172.21.15.45:0/3092401483 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca80031930 con 0x7fca8c15d310 2023-11-28T21:52:35.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca91878640 1 -- 172.21.15.45:0/3092401483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 msgr2=0x7fca8c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:35.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca91878640 1 --2- 172.21.15.45:0/3092401483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 0x7fca8c0b8a20 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0d4130 tx=0x7fca8002b840 comp rx=0 tx=0).stop 2023-11-28T21:52:35.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca91878640 1 -- 172.21.15.45:0/3092401483 shutdown_connections 2023-11-28T21:52:35.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca91878640 1 --2- 172.21.15.45:0/3092401483 >> v2:172.21.15.190:3300/0 conn(0x7fca8c160e70 0x7fca8c0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca91878640 1 --2- 172.21.15.45:0/3092401483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 0x7fca8c0b8a20 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca91878640 1 -- 172.21.15.45:0/3092401483 >> 172.21.15.45:0/3092401483 conn(0x7fca8c06a500 msgr2=0x7fca8c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:35.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca91878640 1 -- 172.21.15.45:0/3092401483 shutdown_connections 2023-11-28T21:52:35.570 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.565+0000 7fca91878640 1 -- 172.21.15.45:0/3092401483 wait complete. 2023-11-28T21:52:35.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.567+0000 7fca91878640 1 Processor -- start 2023-11-28T21:52:35.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.567+0000 7fca91878640 1 -- start start 2023-11-28T21:52:35.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.568+0000 7fca91878640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca8c0ce3a0 0x7fca8c0c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:35.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.568+0000 7fca91878640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 0x7fca8c0c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.568+0000 7fca91878640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca8c160e70 0x7fca8c15f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.568+0000 7fca91878640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0c6670 con 0x7fca8c15d310 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.568+0000 7fca91878640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0ca210 con 0x7fca8c0ce3a0 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.568+0000 7fca91878640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0ca7d0 con 0x7fca8c160e70 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.568+0000 7fca8bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 0x7fca8c0c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca8bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 0x7fca8c0c6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34014/0 (socket says 172.21.15.45:34014) 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca8bfff640 1 -- 172.21.15.45:0/3450909648 learned_addr learned my addr 172.21.15.45:0/3450909648 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca91077640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca8c160e70 0x7fca8c15f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca8bfff640 1 -- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca8c160e70 msgr2=0x7fca8c15f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca8bfff640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca8c160e70 0x7fca8c15f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca8bfff640 1 -- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca8c0ce3a0 msgr2=0x7fca8c0c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca8bfff640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca8c0ce3a0 0x7fca8c0c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca8bfff640 1 -- 172.21.15.45:0/3450909648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca80009400 con 0x7fca8c15d310 2023-11-28T21:52:35.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca91077640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca8c160e70 0x7fca8c15f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:35.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.569+0000 7fca8bfff640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 0x7fca8c0c6130 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fca7800e7e0 tx=0x7fca7800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:35.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.570+0000 7fca897fa640 1 -- 172.21.15.45:0/3450909648 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78004030 con 0x7fca8c15d310 2023-11-28T21:52:35.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.570+0000 7fca897fa640 1 -- 172.21.15.45:0/3450909648 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca78004190 con 0x7fca8c15d310 2023-11-28T21:52:35.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.570+0000 7fca897fa640 1 -- 172.21.15.45:0/3450909648 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78004520 con 0x7fca8c15d310 2023-11-28T21:52:35.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.570+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c1603d0 con 0x7fca8c15d310 2023-11-28T21:52:35.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.570+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c0ca460 con 0x7fca8c15d310 2023-11-28T21:52:35.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.571+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c071b30 con 0x7fca8c15d310 2023-11-28T21:52:35.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.572+0000 7fca897fa640 1 -- 172.21.15.45:0/3450909648 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fca7800c6d0 con 0x7fca8c15d310 2023-11-28T21:52:35.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.573+0000 7fca897fa640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fca7006af60 0x7fca7006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:35.575 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.573+0000 7fca897fa640 1 -- 172.21.15.45:0/3450909648 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fca780896e0 con 0x7fca8c15d310 2023-11-28T21:52:35.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.574+0000 7fca90876640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fca7006af60 0x7fca7006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:35.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.575+0000 7fca90876640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fca7006af60 0x7fca7006d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca80036000 tx=0x7fca80030000 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:35.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.578+0000 7fca897fa640 1 -- 172.21.15.45:0/3450909648 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca78054370 con 0x7fca8c15d310 2023-11-28T21:52:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.797+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 --> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fca8c0d11f0 con 0x7fca7006af60 2023-11-28T21:52:35.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.802+0000 7fca897fa640 1 -- 172.21.15.45:0/3450909648 <== mgr.8101 v2:172.21.15.190:6832/3252186332 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fca8c0d11f0 con 0x7fca7006af60 2023-11-28T21:52:35.804 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T21:52:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T21:52:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T21:52:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T21:52:35.804 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T21:52:35.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fca7006af60 msgr2=0x7fca7006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:35.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fca7006af60 0x7fca7006d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca80036000 tx=0x7fca80030000 comp rx=0 tx=0).stop 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 msgr2=0x7fca8c0c6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 0x7fca8c0c6130 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fca7800e7e0 tx=0x7fca7800eca0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 shutdown_connections 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fca7006af60 0x7fca7006d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca8c160e70 0x7fca8c15f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c15d310 0x7fca8c0c6130 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 --2- 172.21.15.45:0/3450909648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca8c0ce3a0 0x7fca8c0c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 >> 172.21.15.45:0/3450909648 conn(0x7fca8c06a500 msgr2=0x7fca8c0b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 shutdown_connections 2023-11-28T21:52:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:35.804+0000 7fca91878640 1 -- 172.21.15.45:0/3450909648 wait complete. 2023-11-28T21:52:35.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T21:52:36.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.048+0000 7faf63fff640 1 Processor -- start 2023-11-28T21:52:36.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.048+0000 7faf63fff640 1 -- start start 2023-11-28T21:52:36.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.050+0000 7faf63fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf640b3490 0x7faf6415e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:36.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.050+0000 7faf63fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6406dbe0 con 0x7faf640b2410 2023-11-28T21:52:36.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.050+0000 7faf63fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf640b3870 con 0x7faf64071f20 2023-11-28T21:52:36.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.050+0000 7faf63fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6406e110 con 0x7faf640b3490 2023-11-28T21:52:36.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.051+0000 7faf637fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faf640b2410 0x7faf64160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51290/0 (socket says 172.21.15.45:51290) 2023-11-28T21:52:36.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.051+0000 7faf637fe640 1 -- 172.21.15.45:0/3568199588 learned_addr learned my addr 172.21.15.45:0/3568199588 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:36.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.051+0000 7faf62ffd640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf640b3490 0x7faf6415e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:36.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.051+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654707088 0 0) 0x7faf6406dbe0 con 0x7faf640b2410 2023-11-28T21:52:36.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.051+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf50003520 con 0x7faf640b2410 2023-11-28T21:52:36.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4062123346 0 0) 0x7faf50003520 con 0x7faf640b2410 2023-11-28T21:52:36.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf640b3490 msgr2=0x7faf6415e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:36.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf640b3490 0x7faf6415e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 >> v1:172.21.15.190:6789/0 conn(0x7faf64071f20 legacy=0x7faf640b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:36.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf640b2880 con 0x7faf640b2410 2023-11-28T21:52:36.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7faf5c001670 con 0x7faf640b2410 2023-11-28T21:52:36.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 >> v1:172.21.15.45:6789/0 conn(0x7faf640b2410 legacy=0x7faf64160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:36.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf640b3490 0x7faf500039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:36.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf50003ee0 0x7faf500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:36.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:36.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6406e110 con 0x7faf50004d00 2023-11-28T21:52:36.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5000bac0 con 0x7faf50003ee0 2023-11-28T21:52:36.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf5000bc80 con 0x7faf640b3490 2023-11-28T21:52:36.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.052+0000 7faf637fe640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:36.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.053+0000 7faf5a7fc640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf50003ee0 0x7faf500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:36.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.053+0000 7faf62ffd640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf640b3490 0x7faf500039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:36.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.053+0000 7faf637fe640 1 -- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf640b3490 msgr2=0x7faf500039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:36.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.053+0000 7faf637fe640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf640b3490 0x7faf500039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.053+0000 7faf637fe640 1 -- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf50003ee0 msgr2=0x7faf500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:36.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.053+0000 7faf637fe640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf50003ee0 0x7faf500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.053+0000 7faf637fe640 1 -- 172.21.15.45:0/3568199588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf50003520 con 0x7faf50004d00 2023-11-28T21:52:36.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.053+0000 7faf62ffd640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf640b3490 0x7faf500039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:36.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.054+0000 7faf637fe640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf50005180 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00c190 tx=0x7faf500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214b60c5324f6d65 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:36.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.054+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c005ad0 con 0x7faf50004d00 2023-11-28T21:52:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.054+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c005c30 con 0x7faf50004d00 2023-11-28T21:52:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.054+0000 7faf627fc640 1 -- 172.21.15.45:0/3568199588 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c02c810 con 0x7faf50004d00 2023-11-28T21:52:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.055+0000 7faf63fff640 1 -- 172.21.15.45:0/3568199588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 msgr2=0x7faf50005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.055+0000 7faf63fff640 1 --2- 172.21.15.45:0/3568199588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf50005180 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00c190 tx=0x7faf500057f0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.055+0000 7faf637fe640 1 -- 172.21.15.45:0/3568199588 reap_dead start 2023-11-28T21:52:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.055+0000 7faf63fff640 1 -- 172.21.15.45:0/3568199588 shutdown_connections 2023-11-28T21:52:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.055+0000 7faf63fff640 1 -- 172.21.15.45:0/3568199588 >> 172.21.15.45:0/3568199588 conn(0x7faf640cecd0 msgr2=0x7faf640cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.055+0000 7faf63fff640 1 -- 172.21.15.45:0/3568199588 shutdown_connections 2023-11-28T21:52:36.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.055+0000 7faf63fff640 1 -- 172.21.15.45:0/3568199588 wait complete. 2023-11-28T21:52:36.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.057+0000 7faf63fff640 1 Processor -- start 2023-11-28T21:52:36.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.057+0000 7faf63fff640 1 -- start start 2023-11-28T21:52:36.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.058+0000 7faf63fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf6415c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:36.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.058+0000 7faf63fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf64148860 0x7faf640acf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:36.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.058+0000 7faf63fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf640ad4b0 0x7faf6415ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:36.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.058+0000 7faf63fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf640ad960 con 0x7faf50004d00 2023-11-28T21:52:36.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.058+0000 7faf63fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf64148ce0 con 0x7faf640ad4b0 2023-11-28T21:52:36.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.058+0000 7faf63fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6415b280 con 0x7faf64148860 2023-11-28T21:52:36.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.058+0000 7faf62ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf6415c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:36.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.058+0000 7faf62ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf6415c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34030/0 (socket says 172.21.15.45:34030) 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.058+0000 7faf62ffd640 1 -- 172.21.15.45:0/648716671 learned_addr learned my addr 172.21.15.45:0/648716671 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf637fe640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf640ad4b0 0x7faf6415ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf62ffd640 1 -- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf64148860 msgr2=0x7faf640acf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf5a7fc640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf64148860 0x7faf640acf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf62ffd640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf64148860 0x7faf640acf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf62ffd640 1 -- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf640ad4b0 msgr2=0x7faf6415ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf62ffd640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf640ad4b0 0x7faf6415ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf62ffd640 1 -- 172.21.15.45:0/648716671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6415b560 con 0x7faf50004d00 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf5a7fc640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf64148860 0x7faf640acf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:36.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf637fe640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf640ad4b0 0x7faf6415ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T21:52:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.059+0000 7faf62ffd640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf6415c320 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7faf540116d0 tx=0x7faf54011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.060+0000 7faf5bfff640 1 -- 172.21.15.45:0/648716671 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5400f9c0 con 0x7faf50004d00 2023-11-28T21:52:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.060+0000 7faf5bfff640 1 -- 172.21.15.45:0/648716671 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5400fb20 con 0x7faf50004d00 2023-11-28T21:52:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.060+0000 7faf5bfff640 1 -- 172.21.15.45:0/648716671 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5400fed0 con 0x7faf50004d00 2023-11-28T21:52:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.060+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6415b840 con 0x7faf50004d00 2023-11-28T21:52:36.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.060+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6415be00 con 0x7faf50004d00 2023-11-28T21:52:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.061+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6415a190 con 0x7faf50004d00 2023-11-28T21:52:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.062+0000 7faf5bfff640 1 -- 172.21.15.45:0/648716671 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 99) v1 ==== 89333+0+0 (secure 0 0 0) 0x7faf54015460 con 0x7faf50004d00 2023-11-28T21:52:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.063+0000 7faf5bfff640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7faf4406af90 0x7faf4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.063+0000 7faf5bfff640 1 -- 172.21.15.45:0/648716671 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7faf5408a1e0 con 0x7faf50004d00 2023-11-28T21:52:36.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.064+0000 7faf5a7fc640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7faf4406af90 0x7faf4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:36.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.065+0000 7faf5a7fc640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7faf4406af90 0x7faf4406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faf4c000f80 tx=0x7faf4c009040 comp rx=0 tx=0).ready entity=mgr.8101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:36.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.068+0000 7faf5bfff640 1 -- 172.21.15.45:0/648716671 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf54055010 con 0x7faf50004d00 2023-11-28T21:52:36.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.313+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7faf6409d0f0 con 0x7faf50004d00 2023-11-28T21:52:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.923+0000 7faf5bfff640 1 -- 172.21.15.45:0/648716671 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v100) v1 ==== 92+0+0 (secure 0 0 0) 0x7faf54058630 con 0x7faf50004d00 2023-11-28T21:52:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.926+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7faf4406af90 msgr2=0x7faf4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.926+0000 7faf63fff640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7faf4406af90 0x7faf4406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faf4c000f80 tx=0x7faf4c009040 comp rx=0 tx=0).stop 2023-11-28T21:52:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.926+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 msgr2=0x7faf6415c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.926+0000 7faf63fff640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf6415c320 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7faf540116d0 tx=0x7faf54011b90 comp rx=0 tx=0).stop 2023-11-28T21:52:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.927+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 shutdown_connections 2023-11-28T21:52:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.927+0000 7faf63fff640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7faf4406af90 0x7faf4406d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.927+0000 7faf63fff640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf640ad4b0 0x7faf6415ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.927+0000 7faf63fff640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf64148860 0x7faf640acf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.927+0000 7faf63fff640 1 --2- 172.21.15.45:0/648716671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf50004d00 0x7faf6415c320 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.927+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 >> 172.21.15.45:0/648716671 conn(0x7faf640cecd0 msgr2=0x7faf6406cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.927+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 shutdown_connections 2023-11-28T21:52:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:36.927+0000 7faf63fff640 1 -- 172.21.15.45:0/648716671 wait complete. 2023-11-28T21:52:36.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T21:52:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.158+0000 7fbd511ff640 1 Processor -- start 2023-11-28T21:52:37.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.158+0000 7fbd511ff640 1 -- start start 2023-11-28T21:52:37.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.159+0000 7fbd511ff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b2290 0x7fbd4c0d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.159+0000 7fbd511ff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbd4c0b1d40 0x7fbd4c0d22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.160+0000 7fbd511ff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4c0b90a0 con 0x7fbd4c0d0040 2023-11-28T21:52:37.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.160+0000 7fbd511ff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4c0d1920 con 0x7fbd4c0b1d40 2023-11-28T21:52:37.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.160+0000 7fbd511ff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c0b8ae0 con 0x7fbd4c0b2290 2023-11-28T21:52:37.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.160+0000 7fbd509fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbd4c0b1d40 0x7fbd4c0d22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.160+0000 7fbd4bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbd4c0d0040 0x7fbd4c0d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51292/0 (socket says 172.21.15.45:51292) 2023-11-28T21:52:37.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.160+0000 7fbd4bfff640 1 -- 172.21.15.45:0/591997929 learned_addr learned my addr 172.21.15.45:0/591997929 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:37.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.160+0000 7fbd43fff640 1 --2- 172.21.15.45:0/591997929 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b2290 0x7fbd4c0d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.160+0000 7fbd4b7fe640 1 -- 172.21.15.45:0/591997929 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377648622 0 0) 0x7fbd4c0b90a0 con 0x7fbd4c0d0040 2023-11-28T21:52:37.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.160+0000 7fbd4b7fe640 1 -- 172.21.15.45:0/591997929 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd38003520 con 0x7fbd4c0d0040 2023-11-28T21:52:37.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.161+0000 7fbd509fe640 1 -- 172.21.15.45:0/591997929 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b2290 msgr2=0x7fbd4c0d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:37.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.161+0000 7fbd509fe640 1 --2- 172.21.15.45:0/591997929 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b2290 0x7fbd4c0d1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.161+0000 7fbd509fe640 1 -- 172.21.15.45:0/591997929 >> v1:172.21.15.45:6789/0 conn(0x7fbd4c0d0040 legacy=0x7fbd4c0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.161+0000 7fbd509fe640 1 -- 172.21.15.45:0/591997929 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c0b8dc0 con 0x7fbd4c0b1d40 2023-11-28T21:52:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.162+0000 7fbd509fe640 1 --2- 172.21.15.45:0/591997929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0b1d40 0x7fbd4c0d22f0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009d90 tx=0x7fbd4402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8e3273e4ed46a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.162+0000 7fbd4b7fe640 1 -- 172.21.15.45:0/591997929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4402cce0 con 0x7fbd4c0b1d40 2023-11-28T21:52:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.162+0000 7fbd4b7fe640 1 -- 172.21.15.45:0/591997929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4402ce40 con 0x7fbd4c0b1d40 2023-11-28T21:52:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.162+0000 7fbd4b7fe640 1 -- 172.21.15.45:0/591997929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44034660 con 0x7fbd4c0b1d40 2023-11-28T21:52:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.163+0000 7fbd511ff640 1 -- 172.21.15.45:0/591997929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0b1d40 msgr2=0x7fbd4c0d22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.163+0000 7fbd511ff640 1 --2- 172.21.15.45:0/591997929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0b1d40 0x7fbd4c0d22f0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009d90 tx=0x7fbd4402bf30 comp rx=0 tx=0).stop 2023-11-28T21:52:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.163+0000 7fbd511ff640 1 -- 172.21.15.45:0/591997929 shutdown_connections 2023-11-28T21:52:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.163+0000 7fbd511ff640 1 --2- 172.21.15.45:0/591997929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0b1d40 0x7fbd4c0d22f0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.163+0000 7fbd511ff640 1 --2- 172.21.15.45:0/591997929 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b2290 0x7fbd4c0d1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.163+0000 7fbd511ff640 1 -- 172.21.15.45:0/591997929 >> 172.21.15.45:0/591997929 conn(0x7fbd4c070f40 msgr2=0x7fbd4c071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.163+0000 7fbd511ff640 1 -- 172.21.15.45:0/591997929 shutdown_connections 2023-11-28T21:52:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.163+0000 7fbd511ff640 1 -- 172.21.15.45:0/591997929 wait complete. 2023-11-28T21:52:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.165+0000 7fbd511ff640 1 Processor -- start 2023-11-28T21:52:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.165+0000 7fbd511ff640 1 -- start start 2023-11-28T21:52:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.166+0000 7fbd511ff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b1d40 0x7fbd4c150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.166+0000 7fbd511ff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd4c0b2290 0x7fbd4c1f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.166+0000 7fbd511ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0d0040 0x7fbd4c0b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.166+0000 7fbd511ff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c1f72a0 con 0x7fbd4c0d0040 2023-11-28T21:52:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.166+0000 7fbd511ff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c150ad0 con 0x7fbd4c0b2290 2023-11-28T21:52:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.166+0000 7fbd511ff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c0c7f00 con 0x7fbd4c0b1d40 2023-11-28T21:52:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.166+0000 7fbd509fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0d0040 0x7fbd4c0b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.166+0000 7fbd509fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0d0040 0x7fbd4c0b6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34074/0 (socket says 172.21.15.45:34074) 2023-11-28T21:52:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.166+0000 7fbd509fe640 1 -- 172.21.15.45:0/2941251081 learned_addr learned my addr 172.21.15.45:0/2941251081 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.167+0000 7fbd4bfff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b1d40 0x7fbd4c150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.167+0000 7fbd43fff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd4c0b2290 0x7fbd4c1f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.167+0000 7fbd509fe640 1 -- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b1d40 msgr2=0x7fbd4c150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.167+0000 7fbd509fe640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b1d40 0x7fbd4c150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.167+0000 7fbd509fe640 1 -- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd4c0b2290 msgr2=0x7fbd4c1f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.167+0000 7fbd509fe640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd4c0b2290 0x7fbd4c1f6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.167+0000 7fbd509fe640 1 -- 172.21.15.45:0/2941251081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd44009a70 con 0x7fbd4c0d0040 2023-11-28T21:52:37.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.167+0000 7fbd4bfff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b1d40 0x7fbd4c150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T21:52:37.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.167+0000 7fbd509fe640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0d0040 0x7fbd4c0b6e80 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009eb0 tx=0x7fbd44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:37.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.168+0000 7fbd497fa640 1 -- 172.21.15.45:0/2941251081 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4403e610 con 0x7fbd4c0d0040 2023-11-28T21:52:37.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.168+0000 7fbd497fa640 1 -- 172.21.15.45:0/2941251081 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4403e770 con 0x7fbd4c0d0040 2023-11-28T21:52:37.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.168+0000 7fbd497fa640 1 -- 172.21.15.45:0/2941251081 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4403eb20 con 0x7fbd4c0d0040 2023-11-28T21:52:37.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.168+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4c1f6ce0 con 0x7fbd4c0d0040 2023-11-28T21:52:37.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.168+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4c0c7b90 con 0x7fbd4c0d0040 2023-11-28T21:52:37.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.169+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4c0c79f0 con 0x7fbd4c0d0040 2023-11-28T21:52:37.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.170+0000 7fbd497fa640 1 -- 172.21.15.45:0/2941251081 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 100) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fbd4403b050 con 0x7fbd4c0d0040 2023-11-28T21:52:37.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.171+0000 7fbd497fa640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fbd2c06afb0 0x7fbd2c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.172+0000 7fbd497fa640 1 -- 172.21.15.45:0/2941251081 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbd440ac240 con 0x7fbd4c0d0040 2023-11-28T21:52:37.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.172+0000 7fbd4bfff640 1 -- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fbd2c06afb0 msgr2=0x7fbd2c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3252186332 2023-11-28T21:52:37.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.172+0000 7fbd4bfff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fbd2c06afb0 0x7fbd2c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:37.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.176+0000 7fbd497fa640 1 -- 172.21.15.45:0/2941251081 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd4c0c79f0 con 0x7fbd4c0d0040 2023-11-28T21:52:37.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.372+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fbd4c0c79f0 con 0x7fbd4c0d0040 2023-11-28T21:52:37.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.373+0000 7fbd4bfff640 1 -- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fbd2c06afb0 msgr2=0x7fbd2c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3252186332 2023-11-28T21:52:37.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.373+0000 7fbd4bfff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fbd2c06afb0 0x7fbd2c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:52:37.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.379+0000 7fbd497fa640 1 -- 172.21.15.45:0/2941251081 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbd4c0c79f0 con 0x7fbd4c0d0040 2023-11-28T21:52:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fbd2c06afb0 msgr2=0x7fbd2c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fbd2c06afb0 0x7fbd2c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0d0040 msgr2=0x7fbd4c0b6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0d0040 0x7fbd4c0b6e80 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009eb0 tx=0x7fbd44004a40 comp rx=0 tx=0).stop 2023-11-28T21:52:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 shutdown_connections 2023-11-28T21:52:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fbd2c06afb0 0x7fbd2c06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd4c0d0040 0x7fbd4c0b6e80 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd4c0b2290 0x7fbd4c1f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 --2- 172.21.15.45:0/2941251081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd4c0b1d40 0x7fbd4c150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 >> 172.21.15.45:0/2941251081 conn(0x7fbd4c070f40 msgr2=0x7fbd4c0d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:37.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 shutdown_connections 2023-11-28T21:52:37.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.383+0000 7fbd511ff640 1 -- 172.21.15.45:0/2941251081 wait complete. 2023-11-28T21:52:37.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T21:52:37.454 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:37.452+0000 7fed7f230040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T21:52:37.495 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:37.493+0000 7fa0d424a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T21:52:37.506 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:37.505+0000 7fed7f230040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T21:52:37.552 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:37.549+0000 7fa0d424a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T21:52:37.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.606+0000 7fdd2e239640 1 Processor -- start 2023-11-28T21:52:37.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.606+0000 7fdd2e239640 1 -- start start 2023-11-28T21:52:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.608+0000 7fdd2e239640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd280d0040 0x7fdd280d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.608+0000 7fdd2e239640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdd280b1d40 0x7fdd280d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.608+0000 7fdd2e239640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd280b90a0 con 0x7fdd280b2290 2023-11-28T21:52:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.608+0000 7fdd2e239640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd280d1920 con 0x7fdd280b1d40 2023-11-28T21:52:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.608+0000 7fdd2e239640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd280d2dc0 con 0x7fdd280d0040 2023-11-28T21:52:37.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.608+0000 7fdd2da38640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fdd280b1d40 0x7fdd280d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.608+0000 7fdd2d237640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd280d0040 0x7fdd280d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.608+0000 7fdd2ca36640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fdd280b2290 0x7fdd280d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:59028/0 (socket says 172.21.15.45:59028) 2023-11-28T21:52:37.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.608+0000 7fdd2ca36640 1 -- 172.21.15.45:0/2527202250 learned_addr learned my addr 172.21.15.45:0/2527202250 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:37.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.609+0000 7fdd2da38640 1 -- 172.21.15.45:0/2527202250 >> v2:172.21.15.190:3300/0 conn(0x7fdd280d0040 msgr2=0x7fdd280d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:37.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.609+0000 7fdd2da38640 1 --2- 172.21.15.45:0/2527202250 >> v2:172.21.15.190:3300/0 conn(0x7fdd280d0040 0x7fdd280d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.609+0000 7fdd1ffff640 1 -- 172.21.15.45:0/2527202250 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704111458 0 0) 0x7fdd280b90a0 con 0x7fdd280b2290 2023-11-28T21:52:37.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.609+0000 7fdd2da38640 1 -- 172.21.15.45:0/2527202250 >> v1:172.21.15.190:6789/0 conn(0x7fdd280b2290 legacy=0x7fdd280d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.609+0000 7fdd2da38640 1 -- 172.21.15.45:0/2527202250 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd280d0960 con 0x7fdd280b1d40 2023-11-28T21:52:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.610+0000 7fdd2da38640 1 --2- 172.21.15.45:0/2527202250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b1d40 0x7fdd280d3750 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fdd280b53c0 tx=0x7fdd2002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a0ad5ad929c045 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.611+0000 7fdd1ffff640 1 -- 172.21.15.45:0/2527202250 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20030610 con 0x7fdd280b1d40 2023-11-28T21:52:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.611+0000 7fdd1ffff640 1 -- 172.21.15.45:0/2527202250 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd20030770 con 0x7fdd280b1d40 2023-11-28T21:52:37.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.611+0000 7fdd1ffff640 1 -- 172.21.15.45:0/2527202250 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20030b00 con 0x7fdd280b1d40 2023-11-28T21:52:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.611+0000 7fdd2e239640 1 -- 172.21.15.45:0/2527202250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b1d40 msgr2=0x7fdd280d3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.611+0000 7fdd2e239640 1 --2- 172.21.15.45:0/2527202250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b1d40 0x7fdd280d3750 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fdd280b53c0 tx=0x7fdd2002b960 comp rx=0 tx=0).stop 2023-11-28T21:52:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.612+0000 7fdd2e239640 1 -- 172.21.15.45:0/2527202250 shutdown_connections 2023-11-28T21:52:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.612+0000 7fdd2e239640 1 --2- 172.21.15.45:0/2527202250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b1d40 0x7fdd280d3750 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.612+0000 7fdd2e239640 1 --2- 172.21.15.45:0/2527202250 >> v2:172.21.15.190:3300/0 conn(0x7fdd280d0040 0x7fdd280d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.612+0000 7fdd2e239640 1 -- 172.21.15.45:0/2527202250 >> 172.21.15.45:0/2527202250 conn(0x7fdd28070f40 msgr2=0x7fdd28071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.612+0000 7fdd2e239640 1 -- 172.21.15.45:0/2527202250 shutdown_connections 2023-11-28T21:52:37.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.612+0000 7fdd2e239640 1 -- 172.21.15.45:0/2527202250 wait complete. 2023-11-28T21:52:37.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.613+0000 7fdd2e239640 1 Processor -- start 2023-11-28T21:52:37.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.614+0000 7fdd2e239640 1 -- start start 2023-11-28T21:52:37.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2e239640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd280b1d40 0x7fdd280c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2e239640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b2290 0x7fdd2815a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2e239640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd280d0040 0x7fdd281e2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2e239640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280c8100 con 0x7fdd280b2290 2023-11-28T21:52:37.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2e239640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280c6b20 con 0x7fdd280d0040 2023-11-28T21:52:37.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2e239640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280c6e00 con 0x7fdd280b1d40 2023-11-28T21:52:37.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2ca36640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b2290 0x7fdd2815a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2ca36640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b2290 0x7fdd2815a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34086/0 (socket says 172.21.15.45:34086) 2023-11-28T21:52:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2ca36640 1 -- 172.21.15.45:0/1733469617 learned_addr learned my addr 172.21.15.45:0/1733469617 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2d237640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd280b1d40 0x7fdd280c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2da38640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd280d0040 0x7fdd281e2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.615+0000 7fdd2ca36640 1 -- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd280b1d40 msgr2=0x7fdd280c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd2ca36640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd280b1d40 0x7fdd280c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd2ca36640 1 -- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd280d0040 msgr2=0x7fdd281e2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd2ca36640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd280d0040 0x7fdd281e2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd2ca36640 1 -- 172.21.15.45:0/1733469617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20009400 con 0x7fdd280b2290 2023-11-28T21:52:37.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd2ca36640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b2290 0x7fdd2815a7c0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800c4d0 tx=0x7fdd1800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:37.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd1dffb640 1 -- 172.21.15.45:0/1733469617 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd180074e0 con 0x7fdd280b2290 2023-11-28T21:52:37.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd1dffb640 1 -- 172.21.15.45:0/1733469617 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd18007660 con 0x7fdd280b2290 2023-11-28T21:52:37.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd1dffb640 1 -- 172.21.15.45:0/1733469617 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18004a20 con 0x7fdd280b2290 2023-11-28T21:52:37.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd280c70e0 con 0x7fdd280b2290 2023-11-28T21:52:37.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.616+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd280c76a0 con 0x7fdd280b2290 2023-11-28T21:52:37.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.617+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd280c7880 con 0x7fdd280b2290 2023-11-28T21:52:37.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.618+0000 7fdd1dffb640 1 -- 172.21.15.45:0/1733469617 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 100) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fdd18004b80 con 0x7fdd280b2290 2023-11-28T21:52:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.619+0000 7fdd1dffb640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fdd0c06afb0 0x7fdd0c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.619+0000 7fdd1dffb640 1 -- 172.21.15.45:0/1733469617 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd18088b40 con 0x7fdd280b2290 2023-11-28T21:52:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.620+0000 7fdd2d237640 1 -- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fdd0c06afb0 msgr2=0x7fdd0c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3252186332 2023-11-28T21:52:37.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.620+0000 7fdd2d237640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fdd0c06afb0 0x7fdd0c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:37.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.624+0000 7fdd1dffb640 1 -- 172.21.15.45:0/1733469617 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd18053980 con 0x7fdd280b2290 2023-11-28T21:52:37.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.815+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fdd28071b30 con 0x7fdd280b2290 2023-11-28T21:52:37.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.820+0000 7fdd1dffb640 1 -- 172.21.15.45:0/1733469617 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7fdd18018020 con 0x7fdd280b2290 2023-11-28T21:52:37.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.821+0000 7fdd2d237640 1 -- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fdd0c06afb0 msgr2=0x7fdd0c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3252186332 2023-11-28T21:52:37.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.821+0000 7fdd2d237640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fdd0c06afb0 0x7fdd0c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:52:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.822+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fdd0c06afb0 msgr2=0x7fdd0c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.822+0000 7fdd2e239640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fdd0c06afb0 0x7fdd0c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.822+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b2290 msgr2=0x7fdd2815a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.822+0000 7fdd2e239640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b2290 0x7fdd2815a7c0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800c4d0 tx=0x7fdd1800c990 comp rx=0 tx=0).stop 2023-11-28T21:52:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.822+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 shutdown_connections 2023-11-28T21:52:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.822+0000 7fdd2e239640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7fdd0c06afb0 0x7fdd0c06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.822+0000 7fdd2e239640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd280d0040 0x7fdd281e2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.822+0000 7fdd2e239640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd280b2290 0x7fdd2815a7c0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.822+0000 7fdd2e239640 1 --2- 172.21.15.45:0/1733469617 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd280b1d40 0x7fdd280c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.823+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 >> 172.21.15.45:0/1733469617 conn(0x7fdd28070f40 msgr2=0x7fdd280d3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.823+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 shutdown_connections 2023-11-28T21:52:37.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:37.823+0000 7fdd2e239640 1 -- 172.21.15.45:0/1733469617 wait complete. 2023-11-28T21:52:37.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:37.844 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-11-28T21:52:37.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T21:52:37.903 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-11-28T21:52:37.955 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:37.955+0000 7fed7f230040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T21:52:37.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2023-11-28T21:52:38.008 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.007+0000 7fed7f230040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T21:52:38.034 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.031+0000 7fa0d424a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T21:52:38.059 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.057+0000 7fed7f230040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T21:52:38.089 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.086+0000 7fa0d424a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T21:52:38.119 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.118+0000 7fed7f230040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T21:52:38.142 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.140+0000 7fa0d424a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T21:52:38.168 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.167+0000 7fed7f230040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T21:52:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.166+0000 7f445e351640 1 Processor -- start 2023-11-28T21:52:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.167+0000 7f445e351640 1 -- start start 2023-11-28T21:52:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.169+0000 7f445e351640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4458071df0 0x7f44580aed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.169+0000 7f445e351640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f44580b3050 0x7f445815d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.169+0000 7f445e351640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580ae9b0 con 0x7f445806b510 2023-11-28T21:52:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.169+0000 7f445e351640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4458072250 con 0x7f44580b3050 2023-11-28T21:52:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.169+0000 7f445e351640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44580af2b0 con 0x7f4458071df0 2023-11-28T21:52:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.169+0000 7f4457fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f445806b510 0x7f445806b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:59034/0 (socket says 172.21.15.45:59034) 2023-11-28T21:52:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.169+0000 7f445c8ee640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f44580b3050 0x7f445815d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.169+0000 7f4457fff640 1 -- 172.21.15.45:0/1948700648 learned_addr learned my addr 172.21.15.45:0/1948700648 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T21:52:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.169+0000 7f44577fe640 1 --2- 172.21.15.45:0/1948700648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4458071df0 0x7f44580aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.170+0000 7f4456ffd640 1 -- 172.21.15.45:0/1948700648 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994002091 0 0) 0x7f44580ae9b0 con 0x7f445806b510 2023-11-28T21:52:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.170+0000 7f4456ffd640 1 -- 172.21.15.45:0/1948700648 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4440003520 con 0x7f445806b510 2023-11-28T21:52:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.170+0000 7f445c8ee640 1 -- 172.21.15.45:0/1948700648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4458071df0 msgr2=0x7f44580aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.170+0000 7f445c8ee640 1 --2- 172.21.15.45:0/1948700648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4458071df0 0x7f44580aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.170+0000 7f445c8ee640 1 -- 172.21.15.45:0/1948700648 >> v1:172.21.15.190:6789/0 conn(0x7f445806b510 legacy=0x7f445806b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.170+0000 7f445c8ee640 1 -- 172.21.15.45:0/1948700648 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4458161db0 con 0x7f44580b3050 2023-11-28T21:52:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.171+0000 7f445c8ee640 1 --2- 172.21.15.45:0/1948700648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44580b3050 0x7f445815d8e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4450009d90 tx=0x7f445002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3e5a3f04348dedd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.172+0000 7f4456ffd640 1 -- 172.21.15.45:0/1948700648 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445002cbc0 con 0x7f44580b3050 2023-11-28T21:52:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.172+0000 7f4456ffd640 1 -- 172.21.15.45:0/1948700648 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445002cd20 con 0x7f44580b3050 2023-11-28T21:52:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.172+0000 7f4456ffd640 1 -- 172.21.15.45:0/1948700648 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450034660 con 0x7f44580b3050 2023-11-28T21:52:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.172+0000 7f445e351640 1 -- 172.21.15.45:0/1948700648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44580b3050 msgr2=0x7f445815d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.172+0000 7f445e351640 1 --2- 172.21.15.45:0/1948700648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44580b3050 0x7f445815d8e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4450009d90 tx=0x7f445002be10 comp rx=0 tx=0).stop 2023-11-28T21:52:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.172+0000 7f445e351640 1 -- 172.21.15.45:0/1948700648 shutdown_connections 2023-11-28T21:52:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.172+0000 7f445e351640 1 --2- 172.21.15.45:0/1948700648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44580b3050 0x7f445815d8e0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.172+0000 7f445e351640 1 --2- 172.21.15.45:0/1948700648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4458071df0 0x7f44580aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:38.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.172+0000 7f445e351640 1 -- 172.21.15.45:0/1948700648 >> 172.21.15.45:0/1948700648 conn(0x7f44580712d0 msgr2=0x7f44580716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:38.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.173+0000 7f445e351640 1 -- 172.21.15.45:0/1948700648 shutdown_connections 2023-11-28T21:52:38.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.173+0000 7f445e351640 1 -- 172.21.15.45:0/1948700648 wait complete. 2023-11-28T21:52:38.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.174+0000 7f445e351640 1 Processor -- start 2023-11-28T21:52:38.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.174+0000 7f445e351640 1 -- start start 2023-11-28T21:52:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.175+0000 7f445e351640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f445806b510 0x7f44581f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.175+0000 7f445e351640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458071df0 0x7f44580cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.175+0000 7f445e351640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44580b3050 0x7f4458160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.175+0000 7f445e351640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44580cbaf0 con 0x7f4458071df0 2023-11-28T21:52:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.175+0000 7f445e351640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44581573e0 con 0x7f445806b510 2023-11-28T21:52:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.175+0000 7f445e351640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44580cdaa0 con 0x7f44580b3050 2023-11-28T21:52:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f44577fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458071df0 0x7f44580cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f44577fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458071df0 0x7f44580cb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34098/0 (socket says 172.21.15.45:34098) 2023-11-28T21:52:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f44577fe640 1 -- 172.21.15.45:0/3612178071 learned_addr learned my addr 172.21.15.45:0/3612178071 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f4457fff640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f445806b510 0x7f44581f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f445c8ee640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44580b3050 0x7f4458160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f44577fe640 1 -- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44580b3050 msgr2=0x7f4458160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f44577fe640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44580b3050 0x7f4458160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f44577fe640 1 -- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f445806b510 msgr2=0x7f44581f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f44577fe640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f445806b510 0x7f44581f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.176+0000 7f44577fe640 1 -- 172.21.15.45:0/3612178071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450009a70 con 0x7f4458071df0 2023-11-28T21:52:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.177+0000 7f44577fe640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458071df0 0x7f44580cb5b0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e9e0 tx=0x7f444c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.177+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c009670 con 0x7f4458071df0 2023-11-28T21:52:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.177+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c00c9d0 con 0x7f4458071df0 2023-11-28T21:52:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.177+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44580cdd80 con 0x7f4458071df0 2023-11-28T21:52:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.177+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c00cd80 con 0x7f4458071df0 2023-11-28T21:52:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.177+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44580ce340 con 0x7f4458071df0 2023-11-28T21:52:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.178+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445815b370 con 0x7f4458071df0 2023-11-28T21:52:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.179+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 100) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f444c013470 con 0x7f4458071df0 2023-11-28T21:52:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.180+0000 7f4454ff9640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 0x7f444406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.181+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f444c089b30 con 0x7f4458071df0 2023-11-28T21:52:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.181+0000 7f4457fff640 1 -- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 msgr2=0x7f444406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3252186332 2023-11-28T21:52:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.181+0000 7f4457fff640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 0x7f444406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T21:52:38.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.185+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f445815b370 con 0x7f4458071df0 2023-11-28T21:52:38.209 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.207+0000 7fa0d424a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T21:52:38.261 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.258+0000 7fa0d424a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T21:52:38.282 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.280+0000 7fed7f230040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T21:52:38.345 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.343+0000 7fed7f230040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T21:52:38.378 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.376+0000 7fa0d424a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T21:52:38.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.382+0000 7f4457fff640 1 -- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 msgr2=0x7f444406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3252186332 2023-11-28T21:52:38.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.382+0000 7f4457fff640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 0x7f444406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T21:52:38.394 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.393+0000 7fed7f230040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T21:52:38.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.402+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 --> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f44580b5140 con 0x7f444406afb0 2023-11-28T21:52:38.444 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.442+0000 7fa0d424a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T21:52:38.452 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.451+0000 7fed7f230040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T21:52:38.496 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.493+0000 7fa0d424a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T21:52:38.502 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.501+0000 7fed7f230040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T21:52:38.555 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.554+0000 7fa0d424a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T21:52:38.607 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:38.606+0000 7fa0d424a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T21:52:38.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.784+0000 7f4457fff640 1 -- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 msgr2=0x7f444406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3252186332 2023-11-28T21:52:38.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:38.784+0000 7f4457fff640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 0x7f444406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T21:52:38.886 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:38.885+0000 7fed7f230040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T21:52:39.007 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:39.006+0000 7fa0d424a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T21:52:39.045 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:39.044+0000 7fed7f230040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T21:52:39.153 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:39.152+0000 7fed7f230040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T21:52:39.174 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:39.173+0000 7fa0d424a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T21:52:39.203 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:39.203+0000 7fed7f230040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T21:52:39.287 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:39.286+0000 7fa0d424a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T21:52:39.312 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:39.311+0000 7fed7f230040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:52:39.340 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:39.338+0000 7fa0d424a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T21:52:39.419 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:39.418+0000 7fed7f230040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T21:52:39.452 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:39.451+0000 7fa0d424a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:52:39.564 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:39.563+0000 7fa0d424a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T21:52:39.576 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:39.575+0000 7fed7f230040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:52:39.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:39.585+0000 7f4457fff640 1 -- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 msgr2=0x7f444406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3252186332 2023-11-28T21:52:39.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:39.585+0000 7f4457fff640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 0x7f444406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T21:52:39.625 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:39.624+0000 7fed7f230040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T21:52:39.729 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:39.727+0000 7fa0d424a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T21:52:39.744 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:39.742+0000 7fed7f230040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T21:52:39.780 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:39.779+0000 7fa0d424a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T21:52:39.906 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:39.905+0000 7fa0d424a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T21:52:40.568 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:40.567+0000 7fed7f230040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T21:52:40.791 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:40.789+0000 7fa0d424a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T21:52:40.863 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:40.862+0000 7fed7f230040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T21:52:40.922 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:40.921+0000 7fed7f230040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T21:52:41.036 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.034+0000 7fed7f230040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T21:52:41.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:41.057+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 101) v1 ==== 88959+0+0 (secure 0 0 0) 0x7f444c087ef0 con 0x7f4458071df0 2023-11-28T21:52:41.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:41.058+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 msgr2=0x7f444406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T21:52:41.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:41.058+0000 7f4454ff9640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3252186332,v1:172.21.15.190:6833/3252186332] conn(0x7f444406afb0 0x7f444406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:41.110 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:41.109+0000 7fa0d424a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T21:52:41.180 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:41.179+0000 7fa0d424a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.241+0000 7fecfddc1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.241+0000 7fecfddc1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.241+0000 7fecfddc1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.241+0000 7fecfddc1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.241+0000 7fecfddc1640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.242+0000 7fecf9db9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.242+0000 7fecf9db9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.242+0000 7fecf9db9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.242+0000 7fecf9db9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.243 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:52:41.242+0000 7fecf9db9640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:52:41.308 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T21:52:41.307+0000 7fa0d424a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T21:52:42.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.067+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 102) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f444c054d40 con 0x7f4458071df0 2023-11-28T21:52:42.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.068+0000 7f4454ff9640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f4444035850 0x7f44440753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:42.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.068+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 --> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4444035c30 con 0x7f4444035850 2023-11-28T21:52:42.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.072+0000 7f4457fff640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f4444035850 0x7f44440753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:42.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.082+0000 7f4457fff640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f4444035850 0x7f44440753e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f44480011e0 tx=0x7f4448009040 comp rx=0 tx=0).ready entity=mgr.8398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:42.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.241+0000 7f4454ff9640 1 -- 172.21.15.45:0/3612178071 <== mgr.8398 v2:172.21.15.190:6832/3833863335 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f4444035c30 con 0x7f4444035850 2023-11-28T21:52:42.243 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-11-28T21:52:42.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.245+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f4444035850 msgr2=0x7f44440753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:42.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.245+0000 7f445e351640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f4444035850 0x7f44440753e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f44480011e0 tx=0x7f4448009040 comp rx=0 tx=0).stop 2023-11-28T21:52:42.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.245+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458071df0 msgr2=0x7f44580cb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:42.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.245+0000 7f445e351640 1 --2- 172.21.15.45:0/3612178071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4458071df0 0x7f44580cb5b0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e9e0 tx=0x7f444c00eea0 comp rx=0 tx=0).stop 2023-11-28T21:52:42.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.245+0000 7f445c8ee640 1 -- 172.21.15.45:0/3612178071 reap_dead start 2023-11-28T21:52:42.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.245+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 shutdown_connections 2023-11-28T21:52:42.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.245+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 >> 172.21.15.45:0/3612178071 conn(0x7f44580712d0 msgr2=0x7f44580ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:42.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.246+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 shutdown_connections 2023-11-28T21:52:42.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.246+0000 7f445e351640 1 -- 172.21.15.45:0/3612178071 wait complete. 2023-11-28T21:52:42.269 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-11-28T21:52:42.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2023-11-28T21:52:42.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.495+0000 7f8126b3c640 1 Processor -- start 2023-11-28T21:52:42.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.495+0000 7f8126b3c640 1 -- start start 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.497+0000 7f8126b3c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f81200adcb0 0x7f812006ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.497+0000 7f8126b3c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812006f710 con 0x7f812006f2c0 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.497+0000 7f8126b3c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812006b1b0 con 0x7f812015d230 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.497+0000 7f8126b3c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81200b4b20 con 0x7f81200adcb0 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.497+0000 7f8125339640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f81200adcb0 0x7f812006ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.497+0000 7f8125339640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f81200adcb0 0x7f812006ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34100/0 (socket says 172.21.15.45:34100) 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.497+0000 7f8125339640 1 -- 172.21.15.45:0/4231178112 learned_addr learned my addr 172.21.15.45:0/4231178112 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.498+0000 7f8125339640 1 -- 172.21.15.45:0/4231178112 >> v1:172.21.15.190:6789/0 conn(0x7f812015d230 legacy=0x7f81200b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.498+0000 7f8125339640 1 -- 172.21.15.45:0/4231178112 >> v1:172.21.15.45:6789/0 conn(0x7f812006f2c0 legacy=0x7f812006b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T21:52:42.499 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.498+0000 7f8125339640 1 -- 172.21.15.45:0/4231178112 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812006a5a0 con 0x7f81200adcb0 2023-11-28T21:52:42.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.499+0000 7f8125339640 1 --2- 172.21.15.45:0/4231178112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 0x7f812006ed80 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f8110009870 tx=0x7f811002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a732a871c0d97e00 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:42.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.499+0000 7f8124b38640 1 -- 172.21.15.45:0/4231178112 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8110030860 con 0x7f81200adcb0 2023-11-28T21:52:42.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.499+0000 7f8124b38640 1 -- 172.21.15.45:0/4231178112 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81100309c0 con 0x7f81200adcb0 2023-11-28T21:52:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.499+0000 7f8124b38640 1 -- 172.21.15.45:0/4231178112 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8110030d50 con 0x7f81200adcb0 2023-11-28T21:52:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.500+0000 7f8126b3c640 1 -- 172.21.15.45:0/4231178112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 msgr2=0x7f812006ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.500+0000 7f8126b3c640 1 --2- 172.21.15.45:0/4231178112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 0x7f812006ed80 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f8110009870 tx=0x7f811002b9a0 comp rx=0 tx=0).stop 2023-11-28T21:52:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.500+0000 7f8126b3c640 1 -- 172.21.15.45:0/4231178112 shutdown_connections 2023-11-28T21:52:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.500+0000 7f8126b3c640 1 --2- 172.21.15.45:0/4231178112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 0x7f812006ed80 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.500+0000 7f8126b3c640 1 -- 172.21.15.45:0/4231178112 >> 172.21.15.45:0/4231178112 conn(0x7f8120099a40 msgr2=0x7f8120099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.500+0000 7f8126b3c640 1 -- 172.21.15.45:0/4231178112 shutdown_connections 2023-11-28T21:52:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.500+0000 7f8126b3c640 1 -- 172.21.15.45:0/4231178112 wait complete. 2023-11-28T21:52:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.502+0000 7f8126b3c640 1 Processor -- start 2023-11-28T21:52:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.502+0000 7f8126b3c640 1 -- start start 2023-11-28T21:52:42.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.503+0000 7f8126b3c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f812006f2c0 0x7f8120154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:42.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.503+0000 7f8126b3c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 0x7f81200cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:42.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.503+0000 7f8126b3c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812015d230 0x7f812015ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:42.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.503+0000 7f8126b3c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8120150bd0 con 0x7f81200adcb0 2023-11-28T21:52:42.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.503+0000 7f8126b3c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81201561f0 con 0x7f812015d230 2023-11-28T21:52:42.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.503+0000 7f8126b3c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81201564d0 con 0x7f812006f2c0 2023-11-28T21:52:42.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.503+0000 7f8125339640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 0x7f81200cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:42.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.503+0000 7f8125339640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 0x7f81200cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34114/0 (socket says 172.21.15.45:34114) 2023-11-28T21:52:42.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.503+0000 7f8125339640 1 -- 172.21.15.45:0/1106101228 learned_addr learned my addr 172.21.15.45:0/1106101228 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T21:52:42.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.504+0000 7f8125339640 1 -- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f812006f2c0 msgr2=0x7f8120154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:42.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.504+0000 7f8125b3a640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f812006f2c0 0x7f8120154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:42.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.504+0000 7f812633b640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812015d230 0x7f812015ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:42.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.504+0000 7f8125339640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f812006f2c0 0x7f8120154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:42.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.504+0000 7f8125339640 1 -- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812015d230 msgr2=0x7f812015ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:42.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.504+0000 7f8125339640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812015d230 0x7f812015ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:42.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.504+0000 7f8125339640 1 -- 172.21.15.45:0/1106101228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81100094d0 con 0x7f81200adcb0 2023-11-28T21:52:42.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.504+0000 7f8125339640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 0x7f81200cce60 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f8110009870 tx=0x7f8110004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:42.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.504+0000 7f81167fc640 1 -- 172.21.15.45:0/1106101228 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8110030810 con 0x7f81200adcb0 2023-11-28T21:52:42.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.505+0000 7f81167fc640 1 -- 172.21.15.45:0/1106101228 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8110043400 con 0x7f81200adcb0 2023-11-28T21:52:42.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.505+0000 7f81167fc640 1 -- 172.21.15.45:0/1106101228 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81100437b0 con 0x7f81200adcb0 2023-11-28T21:52:42.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.505+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81201567b0 con 0x7f81200adcb0 2023-11-28T21:52:42.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.505+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8120154a40 con 0x7f81200adcb0 2023-11-28T21:52:42.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.506+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8120154c30 con 0x7f81200adcb0 2023-11-28T21:52:42.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.507+0000 7f81167fc640 1 -- 172.21.15.45:0/1106101228 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 102) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f8110043910 con 0x7f81200adcb0 2023-11-28T21:52:42.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.508+0000 7f81167fc640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f810406ace0 0x7f810406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T21:52:42.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.508+0000 7f81167fc640 1 -- 172.21.15.45:0/1106101228 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f811003e070 con 0x7f81200adcb0 2023-11-28T21:52:42.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.509+0000 7f8125b3a640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f810406ace0 0x7f810406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T21:52:42.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.510+0000 7f8125b3a640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f810406ace0 0x7f810406d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f811c003d90 tx=0x7f811c009040 comp rx=0 tx=0).ready entity=mgr.8398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T21:52:42.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.513+0000 7f81167fc640 1 -- 172.21.15.45:0/1106101228 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8120154c30 con 0x7f81200adcb0 2023-11-28T21:52:42.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.733+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 --> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f812006ae40 con 0x7f810406ace0 2023-11-28T21:52:42.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.849+0000 7f81167fc640 1 -- 172.21.15.45:0/1106101228 <== mgr.8398 v2:172.21.15.190:6832/3833863335 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f811c003980 con 0x7f810406ace0 2023-11-28T21:52:42.850 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-11-28T21:52:42.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f810406ace0 msgr2=0x7f810406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:42.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f810406ace0 0x7f810406d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f811c003d90 tx=0x7f811c009040 comp rx=0 tx=0).stop 2023-11-28T21:52:42.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 msgr2=0x7f81200cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T21:52:42.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 0x7f81200cce60 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f8110009870 tx=0x7f8110004730 comp rx=0 tx=0).stop 2023-11-28T21:52:42.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 shutdown_connections 2023-11-28T21:52:42.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f810406ace0 0x7f810406d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:42.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812015d230 0x7f812015ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:42.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81200adcb0 0x7f81200cce60 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:42.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 --2- 172.21.15.45:0/1106101228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f812006f2c0 0x7f8120154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T21:52:42.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 >> 172.21.15.45:0/1106101228 conn(0x7f8120099a40 msgr2=0x7f81200b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T21:52:42.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.852+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 shutdown_connections 2023-11-28T21:52:42.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T21:52:42.853+0000 7f8126b3c640 1 -- 172.21.15.45:0/1106101228 wait complete. 2023-11-28T21:52:42.874 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-11-28T21:53:00.879 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:53:00.877+0000 7feced5a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:53:00.879 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:53:00.877+0000 7feced5a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:53:00.880 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:53:00.877+0000 7feced5a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:53:00.880 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:53:00.877+0000 7feced5a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:53:00.880 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T21:53:00.877+0000 7feced5a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T21:53:02.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:53:02.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:53:32.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:53:32.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:54:02.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:54:02.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:54:32.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:54:32.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:55:02.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:55:02.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:55:32.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:55:32.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:56:03.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:56:03.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:56:33.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:56:33.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:57:03.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:57:03.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:57:33.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:57:33.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:58:03.666 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:58:03.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:58:33.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:58:33.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:59:04.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:59:04.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:59:34.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T21:59:34.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:00:04.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:00:04.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:00:35.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:00:35.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:01:05.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:01:05.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:01:35.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:01:35.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:02:05.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:02:05.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:02:35.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:02:39.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:03:09.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:03:09.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:03:15.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:03:15.875 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-11-28T22:03:15.944 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_53_00_UTC 2023-11-28T22:03:15.944 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_54_00_UTC 2023-11-28T22:03:15.944 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_55_00_UTC 2023-11-28T22:03:15.944 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_56_00_UTC 2023-11-28T22:03:15.944 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_57_00_UTC 2023-11-28T22:03:15.944 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_58_00_UTC 2023-11-28T22:03:15.944 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_59_00_UTC 2023-11-28T22:03:15.945 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_00_00_UTC 2023-11-28T22:03:15.945 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_01_00_UTC 2023-11-28T22:03:15.945 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_02_00_UTC 2023-11-28T22:03:15.945 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_03_00_UTC 2023-11-28T22:03:15.946 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-21_53_00_UTC', 'scheduled-2023-11-28-21_54_00_UTC', 'scheduled-2023-11-28-21_55_00_UTC', 'scheduled-2023-11-28-21_56_00_UTC', 'scheduled-2023-11-28-21_57_00_UTC', 'scheduled-2023-11-28-21_58_00_UTC', 'scheduled-2023-11-28-21_59_00_UTC', 'scheduled-2023-11-28-22_00_00_UTC', 'scheduled-2023-11-28-22_01_00_UTC', 'scheduled-2023-11-28-22_02_00_UTC', 'scheduled-2023-11-28-22_03_00_UTC'] 2023-11-28T22:03:15.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-11-28T22:03:16.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.188+0000 7f5199f6e640 1 Processor -- start 2023-11-28T22:03:16.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.188+0000 7f5199f6e640 1 -- start start 2023-11-28T22:03:16.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.190+0000 7f5199f6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51940712c0 0x7f51940ce500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.190+0000 7f5199f6e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5194161b70 con 0x7f51940c1cd0 2023-11-28T22:03:16.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.190+0000 7f5199f6e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51940724a0 con 0x7f51940ca4d0 2023-11-28T22:03:16.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.190+0000 7f5199f6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51940cabb0 con 0x7f51940712c0 2023-11-28T22:03:16.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.190+0000 7f51937fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f51940c1cd0 0x7f51940c20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55202/0 (socket says 172.21.15.45:55202) 2023-11-28T22:03:16.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.190+0000 7f51937fe640 1 -- 172.21.15.45:0/3786159715 learned_addr learned my addr 172.21.15.45:0/3786159715 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:03:16.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f5192ffd640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51940712c0 0x7f51940ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440922330 0 0) 0x7f5184001110 con 0x7f51940c1cd0 2023-11-28T22:03:16.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f517c003520 con 0x7f51940c1cd0 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 717869785 0 0) 0x7f517c003520 con 0x7f51940c1cd0 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51940712c0 msgr2=0x7f51940ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51940712c0 0x7f51940ce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 >> v1:172.21.15.190:6789/0 conn(0x7f51940ca4d0 legacy=0x7f5194165290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51940ca8f0 con 0x7f51940c1cd0 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5184001800 con 0x7f51940c1cd0 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 >> v1:172.21.15.45:6789/0 conn(0x7f51940c1cd0 legacy=0x7f51940c20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f517c0039a0 0x7f517c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.191+0000 7f51927fc640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51940712c0 0x7f517c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51927fc640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f517c004d00 0x7f517c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940cabb0 con 0x7f517c0039a0 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c00ba90 con 0x7f517c004d00 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c00bc50 con 0x7f51940712c0 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51937fe640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f517c0039a0 0x7f517c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f5192ffd640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51940712c0 0x7f517c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51937fe640 1 -- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51940712c0 msgr2=0x7f517c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51937fe640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51940712c0 0x7f517c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51937fe640 1 -- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f517c004d00 msgr2=0x7f517c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51937fe640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f517c004d00 0x7f517c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f51937fe640 1 -- 172.21.15.45:0/3786159715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c003520 con 0x7f517c0039a0 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.192+0000 7f5192ffd640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51940712c0 0x7f517c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:03:16.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.193+0000 7f51937fe640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f517c0039a0 0x7f517c003dc0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f5184007fa0 tx=0x7f517c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9bf21165f31d5e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:03:16.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5184005ad0 con 0x7f517c0039a0 2023-11-28T22:03:16.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5184005c30 con 0x7f517c0039a0 2023-11-28T22:03:16.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f51927fc640 1 -- 172.21.15.45:0/3786159715 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518402c790 con 0x7f517c0039a0 2023-11-28T22:03:16.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f5199f6e640 1 -- 172.21.15.45:0/3786159715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f517c0039a0 msgr2=0x7f517c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f5199f6e640 1 --2- 172.21.15.45:0/3786159715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f517c0039a0 0x7f517c003dc0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f5184007fa0 tx=0x7f517c0057f0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f5193fff640 1 -- 172.21.15.45:0/3786159715 reap_dead start 2023-11-28T22:03:16.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f5199f6e640 1 -- 172.21.15.45:0/3786159715 shutdown_connections 2023-11-28T22:03:16.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f5199f6e640 1 -- 172.21.15.45:0/3786159715 >> 172.21.15.45:0/3786159715 conn(0x7f51940cb780 msgr2=0x7f51940cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:03:16.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f5199f6e640 1 -- 172.21.15.45:0/3786159715 shutdown_connections 2023-11-28T22:03:16.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.194+0000 7f5199f6e640 1 -- 172.21.15.45:0/3786159715 wait complete. 2023-11-28T22:03:16.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.196+0000 7f5199f6e640 1 Processor -- start 2023-11-28T22:03:16.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.197+0000 7f5199f6e640 1 -- start start 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5199f6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f517c0039a0 0x7f51940b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5199f6e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51940c8670 0x7f519415eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5199f6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519415f0d0 0x7f519414ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5199f6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940c3470 con 0x7f519415f0d0 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5199f6e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940c4480 con 0x7f51940c8670 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5199f6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51940c36c0 con 0x7f517c0039a0 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5193fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519415f0d0 0x7f519414ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5193fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519415f0d0 0x7f519414ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50242/0 (socket says 172.21.15.45:50242) 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5193fff640 1 -- 172.21.15.45:0/1623402443 learned_addr learned my addr 172.21.15.45:0/1623402443 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f5192ffd640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51940c8670 0x7f519415eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.198+0000 7f51937fe640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f517c0039a0 0x7f51940b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f5193fff640 1 -- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f517c0039a0 msgr2=0x7f51940b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f5193fff640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f517c0039a0 0x7f51940b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f5193fff640 1 -- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51940c8670 msgr2=0x7f519415eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f5193fff640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51940c8670 0x7f519415eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f5193fff640 1 -- 172.21.15.45:0/1623402443 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51940c39a0 con 0x7f519415f0d0 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f5193fff640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519415f0d0 0x7f519414ef10 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f51940cb6d0 tx=0x7f518c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f517bfff640 1 -- 172.21.15.45:0/1623402443 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c007070 con 0x7f519415f0d0 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f517bfff640 1 -- 172.21.15.45:0/1623402443 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c0071d0 con 0x7f519415f0d0 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f517bfff640 1 -- 172.21.15.45:0/1623402443 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c007580 con 0x7f519415f0d0 2023-11-28T22:03:16.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51940c3c80 con 0x7f519415f0d0 2023-11-28T22:03:16.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.199+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51940c8af0 con 0x7f519415f0d0 2023-11-28T22:03:16.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.201+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51940cb070 con 0x7f519415f0d0 2023-11-28T22:03:16.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.201+0000 7f517bfff640 1 -- 172.21.15.45:0/1623402443 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 106) v1 ==== 89362+0+0 (secure 0 0 0) 0x7f518c0076e0 con 0x7f519415f0d0 2023-11-28T22:03:16.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.202+0000 7f517bfff640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f518006afe0 0x7f518006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.202+0000 7f517bfff640 1 -- 172.21.15.45:0/1623402443 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f518c088a80 con 0x7f519415f0d0 2023-11-28T22:03:16.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.203+0000 7f51937fe640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f518006afe0 0x7f518006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.204+0000 7f51937fe640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f518006afe0 0x7f518006d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5184034000 tx=0x7f5184009030 comp rx=0 tx=0).ready entity=mgr.8398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:03:16.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.208+0000 7f517bfff640 1 -- 172.21.15.45:0/1623402443 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f518c053750 con 0x7f519415f0d0 2023-11-28T22:03:16.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.424+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 --> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f519406a980 con 0x7f518006afe0 2023-11-28T22:03:16.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.440+0000 7f517bfff640 1 -- 172.21.15.45:0/1623402443 <== mgr.8398 v2:172.21.15.190:6832/3833863335 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f5184001540 con 0x7f518006afe0 2023-11-28T22:03:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:03:16.442 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T21:52:42", "first": "2023-11-28T21:53:00", "last": "2023-11-28T22:03:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-11-28T22:03:16.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f518006afe0 msgr2=0x7f518006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f518006afe0 0x7f518006d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5184034000 tx=0x7f5184009030 comp rx=0 tx=0).stop 2023-11-28T22:03:16.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519415f0d0 msgr2=0x7f519414ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519415f0d0 0x7f519414ef10 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f51940cb6d0 tx=0x7f518c00c960 comp rx=0 tx=0).stop 2023-11-28T22:03:16.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 shutdown_connections 2023-11-28T22:03:16.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f518006afe0 0x7f518006d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519415f0d0 0x7f519414ef10 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51940c8670 0x7f519415eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 --2- 172.21.15.45:0/1623402443 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f517c0039a0 0x7f51940b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.443+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 >> 172.21.15.45:0/1623402443 conn(0x7f51940cb780 msgr2=0x7f519406de60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:03:16.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.444+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 shutdown_connections 2023-11-28T22:03:16.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.444+0000 7f5199f6e640 1 -- 172.21.15.45:0/1623402443 wait complete. 2023-11-28T22:03:16.465 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T21:52:42", "first": "2023-11-28T21:53:00", "last": "2023-11-28T22:03:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-11-28T22:03:16.465 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T21:52:42', 'first': '2023-11-28T21:53:00', 'last': '2023-11-28T22:03:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-11-28T22:03:16.466 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-11-28T22:03:16.466 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-11-28T22:03:16.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-28T22:03:16.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.698+0000 7f02d30d3640 1 Processor -- start 2023-11-28T22:03:16.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.698+0000 7f02d30d3640 1 -- start start 2023-11-28T22:03:16.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.700+0000 7f02d30d3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f02cc15b850 0x7f02cc0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.700+0000 7f02d30d3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc0b1fa0 0x7f02cc0b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.700+0000 7f02d30d3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02cc15bda0 con 0x7f02cc0b35b0 2023-11-28T22:03:16.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.700+0000 7f02d30d3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02cc15ee40 con 0x7f02cc15b850 2023-11-28T22:03:16.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.700+0000 7f02d30d3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc15c080 con 0x7f02cc0b1fa0 2023-11-28T22:03:16.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.700+0000 7f02d0e6f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f02cc0b35b0 0x7f02cc15ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55212/0 (socket says 172.21.15.45:55212) 2023-11-28T22:03:16.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.700+0000 7f02d0e6f640 1 -- 172.21.15.45:0/2637187383 learned_addr learned my addr 172.21.15.45:0/2637187383 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:03:16.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cbfff640 1 --2- 172.21.15.45:0/2637187383 >> v2:172.21.15.190:3300/0 conn(0x7f02cc15b850 0x7f02cc0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02d1670640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc0b1fa0 0x7f02cc0b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766212244 0 0) 0x7f02cc15bda0 con 0x7f02cc0b35b0 2023-11-28T22:03:16.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b4003520 con 0x7f02cc0b35b0 2023-11-28T22:03:16.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447846056 0 0) 0x7f02b4003520 con 0x7f02cc0b35b0 2023-11-28T22:03:16.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc0b1fa0 msgr2=0x7f02cc0b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:03:16.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc0b1fa0 0x7f02cc0b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 >> v2:172.21.15.190:3300/0 conn(0x7f02cc15b850 msgr2=0x7f02cc0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:03:16.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 --2- 172.21.15.45:0/2637187383 >> v2:172.21.15.190:3300/0 conn(0x7f02cc15b850 0x7f02cc0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc15c360 con 0x7f02cc0b35b0 2023-11-28T22:03:16.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f02bc0019f0 con 0x7f02cc0b35b0 2023-11-28T22:03:16.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 >> v1:172.21.15.45:6789/0 conn(0x7f02cc0b35b0 legacy=0x7f02cc15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc0b1fa0 0x7f02b4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02b4003f40 0x7f02b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.701+0000 7f02cb7fe640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b4004d00 0x7f02b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc15ee40 con 0x7f02b4004d00 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc15c080 con 0x7f02b4003f40 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b400b980 con 0x7f02cc0b1fa0 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02d1670640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b4004d00 0x7f02b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02d0e6f640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc0b1fa0 0x7f02b4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02cbfff640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02b4003f40 0x7f02b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02d1670640 1 -- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc0b1fa0 msgr2=0x7f02b4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02d1670640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc0b1fa0 0x7f02b4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02d1670640 1 -- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02b4003f40 msgr2=0x7f02b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:03:16.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02d1670640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02b4003f40 0x7f02b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.702+0000 7f02d1670640 1 -- 172.21.15.45:0/2637187383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b4003520 con 0x7f02b4004d00 2023-11-28T22:03:16.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.703+0000 7f02d1670640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b4004d00 0x7f02b4005180 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009ac0 tx=0x7f02b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f832c0d3375bc85d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:03:16.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.704+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c402c680 con 0x7f02b4004d00 2023-11-28T22:03:16.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.704+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c402c7e0 con 0x7f02b4004d00 2023-11-28T22:03:16.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.704+0000 7f02cb7fe640 1 -- 172.21.15.45:0/2637187383 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c402cb90 con 0x7f02b4004d00 2023-11-28T22:03:16.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.704+0000 7f02d30d3640 1 -- 172.21.15.45:0/2637187383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b4004d00 msgr2=0x7f02b4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.704+0000 7f02d30d3640 1 --2- 172.21.15.45:0/2637187383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b4004d00 0x7f02b4005180 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009ac0 tx=0x7f02b40057f0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.704+0000 7f02d1670640 1 -- 172.21.15.45:0/2637187383 reap_dead start 2023-11-28T22:03:16.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.704+0000 7f02d30d3640 1 -- 172.21.15.45:0/2637187383 shutdown_connections 2023-11-28T22:03:16.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.704+0000 7f02d30d3640 1 -- 172.21.15.45:0/2637187383 >> 172.21.15.45:0/2637187383 conn(0x7f02cc06a500 msgr2=0x7f02cc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:03:16.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.705+0000 7f02d30d3640 1 -- 172.21.15.45:0/2637187383 shutdown_connections 2023-11-28T22:03:16.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.705+0000 7f02d30d3640 1 -- 172.21.15.45:0/2637187383 wait complete. 2023-11-28T22:03:16.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.707+0000 7f02d30d3640 1 Processor -- start 2023-11-28T22:03:16.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.707+0000 7f02d30d3640 1 -- start start 2023-11-28T22:03:16.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.708+0000 7f02d30d3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02b4004d00 0x7f02cc1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.708+0000 7f02d30d3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc138340 0x7f02cc0c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.708+0000 7f02d30d3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0c4450 0x7f02cc1f6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.708+0000 7f02d30d3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0c4900 con 0x7f02cc0c4450 2023-11-28T22:03:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.708+0000 7f02d30d3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc137600 con 0x7f02b4004d00 2023-11-28T22:03:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.708+0000 7f02d30d3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc1f7640 con 0x7f02cc138340 2023-11-28T22:03:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.708+0000 7f02d1670640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0c4450 0x7f02cc1f6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.708+0000 7f02d1670640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0c4450 0x7f02cc1f6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50254/0 (socket says 172.21.15.45:50254) 2023-11-28T22:03:16.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.708+0000 7f02d1670640 1 -- 172.21.15.45:0/1951489151 learned_addr learned my addr 172.21.15.45:0/1951489151 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.709+0000 7f02d0e6f640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02b4004d00 0x7f02cc1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.709+0000 7f02cbfff640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc138340 0x7f02cc0c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.709+0000 7f02d1670640 1 -- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc138340 msgr2=0x7f02cc0c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.709+0000 7f02d1670640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc138340 0x7f02cc0c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.709+0000 7f02d1670640 1 -- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02b4004d00 msgr2=0x7f02cc1551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.709+0000 7f02d1670640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02b4004d00 0x7f02cc1551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.709+0000 7f02d1670640 1 -- 172.21.15.45:0/1951489151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc137860 con 0x7f02cc0c4450 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.709+0000 7f02d1670640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0c4450 0x7f02cc1f6980 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f02c4037040 tx=0x7f02c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.710+0000 7f02c97fa640 1 -- 172.21.15.45:0/1951489151 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4034880 con 0x7f02cc0c4450 2023-11-28T22:03:16.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.710+0000 7f02c97fa640 1 -- 172.21.15.45:0/1951489151 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c4034a00 con 0x7f02cc0c4450 2023-11-28T22:03:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.710+0000 7f02c97fa640 1 -- 172.21.15.45:0/1951489151 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4034db0 con 0x7f02cc0c4450 2023-11-28T22:03:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.710+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02cc137b40 con 0x7f02cc0c4450 2023-11-28T22:03:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.710+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02cc1556e0 con 0x7f02cc0c4450 2023-11-28T22:03:16.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.711+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02cc138060 con 0x7f02cc0c4450 2023-11-28T22:03:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.712+0000 7f02c97fa640 1 -- 172.21.15.45:0/1951489151 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 106) v1 ==== 89362+0+0 (secure 0 0 0) 0x7f02c403e440 con 0x7f02cc0c4450 2023-11-28T22:03:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.713+0000 7f02c97fa640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f02b806afb0 0x7f02b806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:03:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.714+0000 7f02c97fa640 1 -- 172.21.15.45:0/1951489151 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f02c4000e40 con 0x7f02cc0c4450 2023-11-28T22:03:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.714+0000 7f02d0e6f640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f02b806afb0 0x7f02b806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:03:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.715+0000 7f02d0e6f640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f02b806afb0 0x7f02b806d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f02bc002230 tx=0x7f02bc009040 comp rx=0 tx=0).ready entity=mgr.8398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:03:16.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.718+0000 7f02c97fa640 1 -- 172.21.15.45:0/1951489151 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02cc138060 con 0x7f02cc0c4450 2023-11-28T22:03:16.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.963+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f02cc138060 con 0x7f02cc0c4450 2023-11-28T22:03:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.973+0000 7f02c97fa640 1 -- 172.21.15.45:0/1951489151 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v106) v1 ==== 74+0+171250 (secure 0 0 0) 0x7f02cc138060 con 0x7f02cc0c4450 2023-11-28T22:03:16.979 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:03:16.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f02b806afb0 msgr2=0x7f02b806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f02b806afb0 0x7f02b806d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f02bc002230 tx=0x7f02bc009040 comp rx=0 tx=0).stop 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0c4450 msgr2=0x7f02cc1f6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0c4450 0x7f02cc1f6980 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f02c4037040 tx=0x7f02c4004730 comp rx=0 tx=0).stop 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 shutdown_connections 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:6832/3833863335,v1:172.21.15.190:6833/3833863335] conn(0x7f02b806afb0 0x7f02b806d470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0c4450 0x7f02cc1f6980 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f02cc138340 0x7f02cc0c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 --2- 172.21.15.45:0/1951489151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02b4004d00 0x7f02cc1551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 >> 172.21.15.45:0/1951489151 conn(0x7f02cc06a500 msgr2=0x7f02cc06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 shutdown_connections 2023-11-28T22:03:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:03:16.978+0000 7f02d30d3640 1 -- 172.21.15.45:0/1951489151 wait complete. 2023-11-28T22:03:16.999 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":106,"active_gid":8398,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":3833863335},{"type":"v1","addr":"172.21.15.190:6833","nonce":3833863335}]},"active_addr":"172.21.15.190:6833/3833863335","active_change":"2023-11-28T21:52:41.039955+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":10290,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":200,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3522054455}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3848257557}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3938773910}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3880574704}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":2706131286}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":2512261826}]}]} 2023-11-28T22:03:17.003 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2023-11-28T22:03:17.003 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-28T22:03:17.003 INFO:tasks.ceph.mgr.x:Stopping old one... 2023-11-28T22:03:17.004 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-28T22:03:17.004 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:03:17.078 INFO:tasks.ceph.mgr.x:Stopped 2023-11-28T22:03:17.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-28T22:03:17.087 INFO:tasks.ceph.mgr.x:Started 2023-11-28T22:03:17.669 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:17.667+0000 7f9c5942e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T22:03:17.722 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:17.719+0000 7f9c5942e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T22:03:18.162 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.160+0000 7f9c5942e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T22:03:18.214 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.212+0000 7f9c5942e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T22:03:18.263 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.261+0000 7f9c5942e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T22:03:18.323 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.321+0000 7f9c5942e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T22:03:18.371 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.369+0000 7f9c5942e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T22:03:18.482 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.480+0000 7f9c5942e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T22:03:18.544 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.542+0000 7f9c5942e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T22:03:18.592 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.590+0000 7f9c5942e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T22:03:18.648 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.646+0000 7f9c5942e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T22:03:18.698 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:18.696+0000 7f9c5942e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T22:03:19.075 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:19.073+0000 7f9c5942e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T22:03:19.234 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:19.231+0000 7f9c5942e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T22:03:19.340 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:19.338+0000 7f9c5942e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T22:03:19.389 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:19.387+0000 7f9c5942e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T22:03:19.496 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:19.494+0000 7f9c5942e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:03:19.602 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:19.599+0000 7f9c5942e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T22:03:19.758 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:19.755+0000 7f9c5942e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:03:19.806 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:19.804+0000 7f9c5942e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T22:03:19.924 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:19.922+0000 7f9c5942e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T22:03:20.773 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:20.770+0000 7f9c5942e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T22:03:21.086 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.083+0000 7f9c5942e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T22:03:21.143 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.141+0000 7f9c5942e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T22:03:21.256 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.254+0000 7f9c5942e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd4fc2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd4fc2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd4fc2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd7fc8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd4fc2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd4fc2640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd7fc8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd7fc8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd7fc8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:21.579 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:03:21.576+0000 7f9bd7fc8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:03:39.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:03:39.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:04:00.576 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:04:00.575+0000 7f9bdffd8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:04:00.577 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:04:00.575+0000 7f9bdffd8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:04:00.577 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:04:00.575+0000 7f9bdffd8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:04:00.577 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:04:00.575+0000 7f9bdffd8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:04:00.577 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:04:00.575+0000 7f9bdffd8640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:04:09.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:04:09.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:04:40.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:04:40.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:05:10.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:05:20.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:05:50.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:05:50.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:06:20.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:06:20.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:06:50.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:06:50.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:07:20.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:07:20.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:07:51.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:07:51.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:08:17.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2023-11-28T22:08:17.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.330+0000 7fa0c3101640 1 Processor -- start 2023-11-28T22:08:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.330+0000 7fa0c3101640 1 -- start start 2023-11-28T22:08:17.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.332+0000 7fa0c3101640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0bc15e840 0x7fa0bc06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:17.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.332+0000 7fa0c3101640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa0bc15f720 0x7fa0bc0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.332+0000 7fa0c3101640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc06a500 0x7fa0bc06a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.332+0000 7fa0c3101640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc071ce0 con 0x7fa0bc15e840 2023-11-28T22:08:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.332+0000 7fa0c3101640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc06aed0 con 0x7fa0bc15f720 2023-11-28T22:08:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.332+0000 7fa0c3101640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc15ecd0 con 0x7fa0bc06a500 2023-11-28T22:08:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.332+0000 7fa0c0e9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0bc15e840 0x7fa0bc06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.332+0000 7fa0c0e9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa0bc15e840 0x7fa0bc06c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42536/0 (socket says 172.21.15.45:42536) 2023-11-28T22:08:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.332+0000 7fa0c0e9d640 1 -- 172.21.15.45:0/3073680346 learned_addr learned my addr 172.21.15.45:0/3073680346 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.333+0000 7fa0b3fff640 1 --2- 172.21.15.45:0/3073680346 >> v2:172.21.15.190:3300/0 conn(0x7fa0bc15f720 0x7fa0bc0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.333+0000 7fa0c169e640 1 --2- 172.21.15.45:0/3073680346 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc06a500 0x7fa0bc06a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.333+0000 7fa0c0e9d640 1 -- 172.21.15.45:0/3073680346 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc06a500 msgr2=0x7fa0bc06a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.333+0000 7fa0c0e9d640 1 --2- 172.21.15.45:0/3073680346 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc06a500 0x7fa0bc06a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.333+0000 7fa0c0e9d640 1 -- 172.21.15.45:0/3073680346 >> v2:172.21.15.190:3300/0 conn(0x7fa0bc15f720 msgr2=0x7fa0bc0717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.333+0000 7fa0c0e9d640 1 --2- 172.21.15.45:0/3073680346 >> v2:172.21.15.190:3300/0 conn(0x7fa0bc15f720 0x7fa0bc0717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.333+0000 7fa0c0e9d640 1 -- 172.21.15.45:0/3073680346 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc15f290 con 0x7fa0bc15e840 2023-11-28T22:08:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.334+0000 7fa0c0e9d640 1 --2- 172.21.15.45:0/3073680346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15e840 0x7fa0bc06c350 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc06e6d0 tx=0x7fa0b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205d069075403a1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.337+0000 7fa0b37fe640 1 -- 172.21.15.45:0/3073680346 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8031640 con 0x7fa0bc15e840 2023-11-28T22:08:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.337+0000 7fa0b37fe640 1 -- 172.21.15.45:0/3073680346 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b80317a0 con 0x7fa0bc15e840 2023-11-28T22:08:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.337+0000 7fa0b37fe640 1 -- 172.21.15.45:0/3073680346 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8031b30 con 0x7fa0bc15e840 2023-11-28T22:08:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.338+0000 7fa0c3101640 1 -- 172.21.15.45:0/3073680346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15e840 msgr2=0x7fa0bc06c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.338+0000 7fa0c3101640 1 --2- 172.21.15.45:0/3073680346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15e840 0x7fa0bc06c350 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc06e6d0 tx=0x7fa0b802b8f0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.338+0000 7fa0c3101640 1 -- 172.21.15.45:0/3073680346 shutdown_connections 2023-11-28T22:08:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.338+0000 7fa0c3101640 1 --2- 172.21.15.45:0/3073680346 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc06a500 0x7fa0bc06a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.338+0000 7fa0c3101640 1 --2- 172.21.15.45:0/3073680346 >> v2:172.21.15.190:3300/0 conn(0x7fa0bc15f720 0x7fa0bc0717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.338+0000 7fa0c3101640 1 --2- 172.21.15.45:0/3073680346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15e840 0x7fa0bc06c350 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.338+0000 7fa0c3101640 1 -- 172.21.15.45:0/3073680346 >> 172.21.15.45:0/3073680346 conn(0x7fa0bc0ced60 msgr2=0x7fa0bc0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.338+0000 7fa0c3101640 1 -- 172.21.15.45:0/3073680346 shutdown_connections 2023-11-28T22:08:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.338+0000 7fa0c3101640 1 -- 172.21.15.45:0/3073680346 wait complete. 2023-11-28T22:08:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.341+0000 7fa0c3101640 1 Processor -- start 2023-11-28T22:08:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.341+0000 7fa0c3101640 1 -- start start 2023-11-28T22:08:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.342+0000 7fa0c3101640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0bc06a500 0x7fa0bc1f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.342+0000 7fa0c3101640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc15e840 0x7fa0bc0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.342+0000 7fa0c3101640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f720 0x7fa0bc0c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.342+0000 7fa0c3101640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc1f3540 con 0x7fa0bc15f720 2023-11-28T22:08:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.342+0000 7fa0c3101640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc09ae20 con 0x7fa0bc06a500 2023-11-28T22:08:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.342+0000 7fa0c3101640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc09b100 con 0x7fa0bc15e840 2023-11-28T22:08:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0c169e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f720 0x7fa0bc0c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0c169e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f720 0x7fa0bc0c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42540/0 (socket says 172.21.15.45:42540) 2023-11-28T22:08:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0c169e640 1 -- 172.21.15.45:0/159313347 learned_addr learned my addr 172.21.15.45:0/159313347 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0c0e9d640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0bc06a500 0x7fa0bc1f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0b3fff640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc15e840 0x7fa0bc0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0c169e640 1 -- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc15e840 msgr2=0x7fa0bc0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0c169e640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc15e840 0x7fa0bc0b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0c169e640 1 -- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0bc06a500 msgr2=0x7fa0bc1f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0c169e640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0bc06a500 0x7fa0bc1f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.343+0000 7fa0c169e640 1 -- 172.21.15.45:0/159313347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b80094d0 con 0x7fa0bc15f720 2023-11-28T22:08:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.344+0000 7fa0c169e640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f720 0x7fa0bc0c5250 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4007d10 tx=0x7fa0b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.344+0000 7fa0b17fa640 1 -- 172.21.15.45:0/159313347 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4014420 con 0x7fa0bc15f720 2023-11-28T22:08:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.344+0000 7fa0b17fa640 1 -- 172.21.15.45:0/159313347 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b40145a0 con 0x7fa0bc15f720 2023-11-28T22:08:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.344+0000 7fa0b17fa640 1 -- 172.21.15.45:0/159313347 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4014950 con 0x7fa0bc15f720 2023-11-28T22:08:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.344+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc09b3e0 con 0x7fa0bc15f720 2023-11-28T22:08:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.344+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc09b9a0 con 0x7fa0bc15f720 2023-11-28T22:08:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.346+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc09bb90 con 0x7fa0bc15f720 2023-11-28T22:08:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.347+0000 7fa0b17fa640 1 -- 172.21.15.45:0/159313347 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 112) v1 ==== 89362+0+0 (secure 0 0 0) 0x7fa0b4014ab0 con 0x7fa0bc15f720 2023-11-28T22:08:17.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.347+0000 7fa0b17fa640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa0a806afb0 0x7fa0a806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:17.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.348+0000 7fa0b17fa640 1 -- 172.21.15.45:0/159313347 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fa0b408a440 con 0x7fa0bc15f720 2023-11-28T22:08:17.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.348+0000 7fa0c0e9d640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa0a806afb0 0x7fa0a806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.350+0000 7fa0c0e9d640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa0a806afb0 0x7fa0a806d470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8036000 tx=0x7fa0b803c040 comp rx=0 tx=0).ready entity=mgr.8488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:17.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.353+0000 7fa0b17fa640 1 -- 172.21.15.45:0/159313347 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0b4054fe0 con 0x7fa0bc15f720 2023-11-28T22:08:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.575+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 --> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0bc072300 con 0x7fa0a806afb0 2023-11-28T22:08:17.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.721+0000 7fa0b17fa640 1 -- 172.21.15.45:0/159313347 <== mgr.8488 v2:172.21.15.190:6832/697528389 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fa0b8031990 con 0x7fa0a806afb0 2023-11-28T22:08:17.723 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-11-28T22:08:17.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa0a806afb0 msgr2=0x7fa0a806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:17.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa0a806afb0 0x7fa0a806d470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8036000 tx=0x7fa0b803c040 comp rx=0 tx=0).stop 2023-11-28T22:08:17.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f720 msgr2=0x7fa0bc0c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:17.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f720 0x7fa0bc0c5250 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4007d10 tx=0x7fa0b400e4b0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 shutdown_connections 2023-11-28T22:08:17.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa0a806afb0 0x7fa0a806d470 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0bc15f720 0x7fa0bc0c5250 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0bc15e840 0x7fa0bc0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 --2- 172.21.15.45:0/159313347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0bc06a500 0x7fa0bc1f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:17.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.725+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 >> 172.21.15.45:0/159313347 conn(0x7fa0bc0ced60 msgr2=0x7fa0bc0701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:17.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.726+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 shutdown_connections 2023-11-28T22:08:17.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:17.726+0000 7fa0c3101640 1 -- 172.21.15.45:0/159313347 wait complete. 2023-11-28T22:08:17.750 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-11-28T22:08:17.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:17.750 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-11-28T22:08:17.807 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_53_00_UTC 2023-11-28T22:08:17.808 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_54_00_UTC 2023-11-28T22:08:17.808 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_55_00_UTC 2023-11-28T22:08:17.808 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_56_00_UTC 2023-11-28T22:08:17.808 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_57_00_UTC 2023-11-28T22:08:17.808 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_58_00_UTC 2023-11-28T22:08:17.808 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_59_00_UTC 2023-11-28T22:08:17.808 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_00_00_UTC 2023-11-28T22:08:17.808 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_01_00_UTC 2023-11-28T22:08:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_02_00_UTC 2023-11-28T22:08:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_03_00_UTC 2023-11-28T22:08:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_04_00_UTC 2023-11-28T22:08:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_05_00_UTC 2023-11-28T22:08:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_06_00_UTC 2023-11-28T22:08:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_07_00_UTC 2023-11-28T22:08:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_08_00_UTC 2023-11-28T22:08:17.810 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-11-28-21_53_00_UTC', 'scheduled-2023-11-28-21_54_00_UTC', 'scheduled-2023-11-28-21_55_00_UTC', 'scheduled-2023-11-28-21_56_00_UTC', 'scheduled-2023-11-28-21_57_00_UTC', 'scheduled-2023-11-28-21_58_00_UTC', 'scheduled-2023-11-28-21_59_00_UTC', 'scheduled-2023-11-28-22_00_00_UTC', 'scheduled-2023-11-28-22_01_00_UTC', 'scheduled-2023-11-28-22_02_00_UTC', 'scheduled-2023-11-28-22_03_00_UTC', 'scheduled-2023-11-28-22_04_00_UTC', 'scheduled-2023-11-28-22_05_00_UTC', 'scheduled-2023-11-28-22_06_00_UTC', 'scheduled-2023-11-28-22_07_00_UTC', 'scheduled-2023-11-28-22_08_00_UTC'] 2023-11-28T22:08:17.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-11-28T22:08:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.034+0000 7f8771ac3640 1 Processor -- start 2023-11-28T22:08:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.035+0000 7f8771ac3640 1 -- start start 2023-11-28T22:08:18.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.036+0000 7f8771ac3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c15b080 0x7f876c0ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.036+0000 7f8771ac3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f876c0abd70 0x7f876c0718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.036+0000 7f8771ac3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876c0ac270 con 0x7f876c071e00 2023-11-28T22:08:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.036+0000 7f8771ac3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876c0722b0 con 0x7f876c0abd70 2023-11-28T22:08:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.036+0000 7f8771ac3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876c15b830 con 0x7f876c15b080 2023-11-28T22:08:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.036+0000 7f87712c2640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f876c071e00 0x7f876c15eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36882/0 (socket says 172.21.15.45:36882) 2023-11-28T22:08:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.036+0000 7f87712c2640 1 -- 172.21.15.45:0/1137582217 learned_addr learned my addr 172.21.15.45:0/1137582217 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f8770ac1640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c15b080 0x7f876c0ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876bfff640 1 --2- 172.21.15.45:0/1137582217 >> v2:172.21.15.190:3300/0 conn(0x7f876c0abd70 0x7f876c0718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380326307 0 0) 0x7f876c0ac270 con 0x7f876c071e00 2023-11-28T22:08:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f875c003520 con 0x7f876c071e00 2023-11-28T22:08:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3582955662 0 0) 0x7f875c003520 con 0x7f876c071e00 2023-11-28T22:08:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c15b080 msgr2=0x7f876c0ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c15b080 0x7f876c0ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 >> v2:172.21.15.190:3300/0 conn(0x7f876c0abd70 msgr2=0x7f876c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 --2- 172.21.15.45:0/1137582217 >> v2:172.21.15.190:3300/0 conn(0x7f876c0abd70 0x7f876c0718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c15bb10 con 0x7f876c071e00 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f87640019f0 con 0x7f876c071e00 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.037+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 >> v1:172.21.15.45:6789/0 conn(0x7f876c071e00 legacy=0x7f876c15eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f876b7fe640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c15b080 0x7f875c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f876b7fe640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f875c003f40 0x7f875c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f876b7fe640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f875c004d00 0x7f875c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876c0722b0 con 0x7f875c004d00 2023-11-28T22:08:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876c15b830 con 0x7f875c003f40 2023-11-28T22:08:18.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875c00b980 con 0x7f876c15b080 2023-11-28T22:08:18.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f87712c2640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f875c004d00 0x7f875c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f8770ac1640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c15b080 0x7f875c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f876bfff640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f875c003f40 0x7f875c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f87712c2640 1 -- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c15b080 msgr2=0x7f875c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:18.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f87712c2640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c15b080 0x7f875c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f87712c2640 1 -- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f875c003f40 msgr2=0x7f875c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:18.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f87712c2640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f875c003f40 0x7f875c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.038+0000 7f87712c2640 1 -- 172.21.15.45:0/1137582217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c003520 con 0x7f875c004d00 2023-11-28T22:08:18.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.040+0000 7f87712c2640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f875c004d00 0x7f875c005180 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f876400a7c0 tx=0x7f875c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a7c4999c1ccc84 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:18.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.040+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764034630 con 0x7f875c004d00 2023-11-28T22:08:18.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.040+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876402f070 con 0x7f875c004d00 2023-11-28T22:08:18.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.040+0000 7f876b7fe640 1 -- 172.21.15.45:0/1137582217 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764034a30 con 0x7f875c004d00 2023-11-28T22:08:18.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.040+0000 7f8771ac3640 1 -- 172.21.15.45:0/1137582217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f875c004d00 msgr2=0x7f875c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.040+0000 7f8771ac3640 1 --2- 172.21.15.45:0/1137582217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f875c004d00 0x7f875c005180 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f876400a7c0 tx=0x7f875c0057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.040+0000 7f87712c2640 1 -- 172.21.15.45:0/1137582217 reap_dead start 2023-11-28T22:08:18.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.041+0000 7f8771ac3640 1 -- 172.21.15.45:0/1137582217 shutdown_connections 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.041+0000 7f8771ac3640 1 -- 172.21.15.45:0/1137582217 >> 172.21.15.45:0/1137582217 conn(0x7f876c0cecd0 msgr2=0x7f876c0cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.041+0000 7f8771ac3640 1 -- 172.21.15.45:0/1137582217 shutdown_connections 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.041+0000 7f8771ac3640 1 -- 172.21.15.45:0/1137582217 wait complete. 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.042+0000 7f8771ac3640 1 Processor -- start 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.043+0000 7f8771ac3640 1 -- start start 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f8771ac3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f875c004d00 0x7f876c14de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f8771ac3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876c1533f0 0x7f876c153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f8771ac3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c151ae0 0x7f876c151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f8771ac3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876c0aedf0 con 0x7f876c1533f0 2023-11-28T22:08:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f8771ac3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876c153db0 con 0x7f875c004d00 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f8771ac3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876c1524a0 con 0x7f876c151ae0 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f876bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876c1533f0 0x7f876c153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f876bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876c1533f0 0x7f876c153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42562/0 (socket says 172.21.15.45:42562) 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f876bfff640 1 -- 172.21.15.45:0/714702399 learned_addr learned my addr 172.21.15.45:0/714702399 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f87712c2640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c151ae0 0x7f876c151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f8770ac1640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f875c004d00 0x7f876c14de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f876bfff640 1 -- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c151ae0 msgr2=0x7f876c151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f876bfff640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c151ae0 0x7f876c151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f876bfff640 1 -- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f875c004d00 msgr2=0x7f876c14de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f876bfff640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f875c004d00 0x7f876c14de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.044+0000 7f876bfff640 1 -- 172.21.15.45:0/714702399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c14e340 con 0x7f876c1533f0 2023-11-28T22:08:18.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.045+0000 7f876bfff640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876c1533f0 0x7f876c153870 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f8758011700 tx=0x7f8758011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:18.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.045+0000 7f87697fa640 1 -- 172.21.15.45:0/714702399 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875800f960 con 0x7f876c1533f0 2023-11-28T22:08:18.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.045+0000 7f87697fa640 1 -- 172.21.15.45:0/714702399 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875800fac0 con 0x7f876c1533f0 2023-11-28T22:08:18.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.045+0000 7f87697fa640 1 -- 172.21.15.45:0/714702399 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875800fe30 con 0x7f876c1533f0 2023-11-28T22:08:18.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.045+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f876c1f32b0 con 0x7f876c1533f0 2023-11-28T22:08:18.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.045+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876c1529a0 con 0x7f876c1533f0 2023-11-28T22:08:18.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.046+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f876c0aef60 con 0x7f876c1533f0 2023-11-28T22:08:18.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.047+0000 7f87697fa640 1 -- 172.21.15.45:0/714702399 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 112) v1 ==== 89362+0+0 (secure 0 0 0) 0x7f8758015460 con 0x7f876c1533f0 2023-11-28T22:08:18.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.048+0000 7f87697fa640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f875006afe0 0x7f875006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.048+0000 7f87697fa640 1 -- 172.21.15.45:0/714702399 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f875808a190 con 0x7f876c1533f0 2023-11-28T22:08:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.049+0000 7f8770ac1640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f875006afe0 0x7f875006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.050+0000 7f8770ac1640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f875006afe0 0x7f875006d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8760009950 tx=0x7f8760008040 comp rx=0 tx=0).ready entity=mgr.8488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.053+0000 7f87697fa640 1 -- 172.21.15.45:0/714702399 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f876c0aef60 con 0x7f876c1533f0 2023-11-28T22:08:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.273+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 --> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f876c09cd80 con 0x7f875006afe0 2023-11-28T22:08:18.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.288+0000 7f87697fa640 1 -- 172.21.15.45:0/714702399 <== mgr.8488 v2:172.21.15.190:6832/697528389 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+361 (secure 0 0 0) 0x7f8760012950 con 0x7f875006afe0 2023-11-28T22:08:18.290 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:18.290 INFO:teuthology.orchestra.run.smithi045.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T21:52:42", "first": "2023-11-28T21:53:00", "last": "2023-11-28T22:08:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f875006afe0 msgr2=0x7f875006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f875006afe0 0x7f875006d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8760009950 tx=0x7f8760008040 comp rx=0 tx=0).stop 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876c1533f0 msgr2=0x7f876c153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876c1533f0 0x7f876c153870 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f8758011700 tx=0x7f8758011bc0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 shutdown_connections 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c151ae0 0x7f876c151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f875006afe0 0x7f875006d4a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876c1533f0 0x7f876c153870 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 --2- 172.21.15.45:0/714702399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f875c004d00 0x7f876c14de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 >> 172.21.15.45:0/714702399 conn(0x7f876c0cecd0 msgr2=0x7f876c15c310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:18.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.291+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 shutdown_connections 2023-11-28T22:08:18.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.292+0000 7f8771ac3640 1 -- 172.21.15.45:0/714702399 wait complete. 2023-11-28T22:08:18.312 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-11-28T00:00:00", "created": "2023-11-28T21:52:42", "first": "2023-11-28T21:53:00", "last": "2023-11-28T22:08:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-11-28T22:08:18.313 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-11-28T00:00:00', 'created': '2023-11-28T21:52:42', 'first': '2023-11-28T21:53:00', 'last': '2023-11-28T22:08:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-11-28T22:08:18.313 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-11-28T22:08:18.313 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-11-28T22:08:18.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2023-11-28T22:08:18.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.532+0000 7fbca101a640 1 Processor -- start 2023-11-28T22:08:18.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.533+0000 7fbca101a640 1 -- start start 2023-11-28T22:08:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.534+0000 7fbca101a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15cf10 0x7fbc9c0d1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.534+0000 7fbca101a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbc9c0ce000 0x7fbc9c0d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.534+0000 7fbca101a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c0d02b0 con 0x7fbc9c15d2f0 2023-11-28T22:08:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.534+0000 7fbca101a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c0d1720 con 0x7fbc9c0ce000 2023-11-28T22:08:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.534+0000 7fbca101a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0d1a00 con 0x7fbc9c15cf10 2023-11-28T22:08:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.535+0000 7fbc9b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbc9c15d2f0 0x7fbc9c0cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36886/0 (socket says 172.21.15.45:36886) 2023-11-28T22:08:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.535+0000 7fbc9b7fe640 1 -- 172.21.15.45:0/2610937445 learned_addr learned my addr 172.21.15.45:0/2610937445 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.535+0000 7fbca0819640 1 --2- 172.21.15.45:0/2610937445 >> v2:172.21.15.45:3300/0 conn(0x7fbc9c0ce000 0x7fbc9c0d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.535+0000 7fbc9bfff640 1 --2- 172.21.15.45:0/2610937445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15cf10 0x7fbc9c0d1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.535+0000 7fbc9affd640 1 -- 172.21.15.45:0/2610937445 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130223294 0 0) 0x7fbc9c0d02b0 con 0x7fbc9c15d2f0 2023-11-28T22:08:18.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.535+0000 7fbca0819640 1 -- 172.21.15.45:0/2610937445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15cf10 msgr2=0x7fbc9c0d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:18.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.535+0000 7fbca0819640 1 --2- 172.21.15.45:0/2610937445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15cf10 0x7fbc9c0d1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.535+0000 7fbca0819640 1 -- 172.21.15.45:0/2610937445 >> v1:172.21.15.45:6789/0 conn(0x7fbc9c15d2f0 legacy=0x7fbc9c0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.535+0000 7fbca0819640 1 -- 172.21.15.45:0/2610937445 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c0d1ce0 con 0x7fbc9c0ce000 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.536+0000 7fbca0819640 1 --2- 172.21.15.45:0/2610937445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 0x7fbc9c0d2990 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fbc940097a0 tx=0x7fbc9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab25e319dc4447f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbc9affd640 1 -- 172.21.15.45:0/2610937445 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94031830 con 0x7fbc9c0ce000 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbc9affd640 1 -- 172.21.15.45:0/2610937445 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc94031990 con 0x7fbc9c0ce000 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbc9affd640 1 -- 172.21.15.45:0/2610937445 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94031d20 con 0x7fbc9c0ce000 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbca101a640 1 -- 172.21.15.45:0/2610937445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 msgr2=0x7fbc9c0d2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbca101a640 1 --2- 172.21.15.45:0/2610937445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 0x7fbc9c0d2990 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fbc940097a0 tx=0x7fbc9402b8d0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbca101a640 1 -- 172.21.15.45:0/2610937445 shutdown_connections 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbca101a640 1 --2- 172.21.15.45:0/2610937445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 0x7fbc9c0d2990 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbca101a640 1 --2- 172.21.15.45:0/2610937445 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15cf10 0x7fbc9c0d1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbca101a640 1 -- 172.21.15.45:0/2610937445 >> 172.21.15.45:0/2610937445 conn(0x7fbc9c070f40 msgr2=0x7fbc9c071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:18.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbca101a640 1 -- 172.21.15.45:0/2610937445 shutdown_connections 2023-11-28T22:08:18.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.537+0000 7fbca101a640 1 -- 172.21.15.45:0/2610937445 wait complete. 2023-11-28T22:08:18.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.539+0000 7fbca101a640 1 Processor -- start 2023-11-28T22:08:18.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.540+0000 7fbca101a640 1 -- start start 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.540+0000 7fbca101a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 0x7fbc9c0cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.540+0000 7fbca101a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc9c15cf10 0x7fbc9c157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbca101a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15d2f0 0x7fbc9c0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbca101a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0cb710 con 0x7fbc9c0ce000 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbca101a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0c7210 con 0x7fbc9c15cf10 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbca101a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0b2c90 con 0x7fbc9c15d2f0 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 0x7fbc9c0cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 0x7fbc9c0cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42580/0 (socket says 172.21.15.45:42580) 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbc9bfff640 1 -- 172.21.15.45:0/2300093696 learned_addr learned my addr 172.21.15.45:0/2300093696 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbca0819640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15d2f0 0x7fbc9c0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbc9b7fe640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc9c15cf10 0x7fbc9c157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbc9bfff640 1 -- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15d2f0 msgr2=0x7fbc9c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbc9bfff640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15d2f0 0x7fbc9c0cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbc9bfff640 1 -- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc9c15cf10 msgr2=0x7fbc9c157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbc9bfff640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc9c15cf10 0x7fbc9c157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.541+0000 7fbc9bfff640 1 -- 172.21.15.45:0/2300093696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc94009400 con 0x7fbc9c0ce000 2023-11-28T22:08:18.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.542+0000 7fbc9bfff640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 0x7fbc9c0cab70 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c007ef0 tx=0x7fbc8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:18.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.542+0000 7fbc98ff9640 1 -- 172.21.15.45:0/2300093696 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c014620 con 0x7fbc9c0ce000 2023-11-28T22:08:18.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.542+0000 7fbc98ff9640 1 -- 172.21.15.45:0/2300093696 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8c014780 con 0x7fbc9c0ce000 2023-11-28T22:08:18.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.542+0000 7fbc98ff9640 1 -- 172.21.15.45:0/2300093696 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c014b30 con 0x7fbc9c0ce000 2023-11-28T22:08:18.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.542+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c0cb150 con 0x7fbc9c0ce000 2023-11-28T22:08:18.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.542+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c0b2f30 con 0x7fbc9c0ce000 2023-11-28T22:08:18.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.543+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c0b2af0 con 0x7fbc9c0ce000 2023-11-28T22:08:18.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.544+0000 7fbc98ff9640 1 -- 172.21.15.45:0/2300093696 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 112) v1 ==== 89362+0+0 (secure 0 0 0) 0x7fbc8c014c90 con 0x7fbc9c0ce000 2023-11-28T22:08:18.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.545+0000 7fbc98ff9640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fbc8806afb0 0x7fbc8806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:18.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.545+0000 7fbc98ff9640 1 -- 172.21.15.45:0/2300093696 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fbc8c08a690 con 0x7fbc9c0ce000 2023-11-28T22:08:18.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.546+0000 7fbc9b7fe640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fbc8806afb0 0x7fbc8806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:18.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.547+0000 7fbc9b7fe640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fbc8806afb0 0x7fbc8806d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbc90001360 tx=0x7fbc90009040 comp rx=0 tx=0).ready entity=mgr.8488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:18.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.550+0000 7fbc98ff9640 1 -- 172.21.15.45:0/2300093696 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc9c0b2af0 con 0x7fbc9c0ce000 2023-11-28T22:08:18.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.771+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 --> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc9c160c60 con 0x7fbc8806afb0 2023-11-28T22:08:18.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.926+0000 7fbc98ff9640 1 -- 172.21.15.45:0/2300093696 <== mgr.8488 v2:172.21.15.190:6832/697528389 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7fbc90001c30 con 0x7fbc8806afb0 2023-11-28T22:08:18.928 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fbc8806afb0 msgr2=0x7fbc8806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fbc8806afb0 0x7fbc8806d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbc90001360 tx=0x7fbc90009040 comp rx=0 tx=0).stop 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 msgr2=0x7fbc9c0cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 0x7fbc9c0cab70 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c007ef0 tx=0x7fbc8c00e4b0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 shutdown_connections 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc9c15d2f0 0x7fbc9c0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc9c15cf10 0x7fbc9c157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fbc8806afb0 0x7fbc8806d470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 --2- 172.21.15.45:0/2300093696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc9c0ce000 0x7fbc9c0cab70 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 >> 172.21.15.45:0/2300093696 conn(0x7fbc9c070f40 msgr2=0x7fbc9c0b8060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:18.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 shutdown_connections 2023-11-28T22:08:18.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:18.930+0000 7fbca101a640 1 -- 172.21.15.45:0/2300093696 wait complete. 2023-11-28T22:08:18.952 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-11-28T22:08:18.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:18.953 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-11-28T22:08:19.008 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_53_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_54_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_55_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_56_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_57_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_58_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-21_59_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_00_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_01_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_02_00_UTC 2023-11-28T22:08:19.009 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_03_00_UTC 2023-11-28T22:08:19.010 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_04_00_UTC 2023-11-28T22:08:19.010 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_05_00_UTC 2023-11-28T22:08:19.010 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_06_00_UTC 2023-11-28T22:08:19.010 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_07_00_UTC 2023-11-28T22:08:19.010 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_08_00_UTC 2023-11-28T22:08:19.011 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_53_00_UTC 2023-11-28T22:08:19.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.011 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_53_00_UTC) 2023-11-28T22:08:19.078 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_54_00_UTC 2023-11-28T22:08:19.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.078 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_54_00_UTC) 2023-11-28T22:08:19.147 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_55_00_UTC 2023-11-28T22:08:19.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.148 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_55_00_UTC) 2023-11-28T22:08:19.215 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_56_00_UTC 2023-11-28T22:08:19.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.215 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_56_00_UTC) 2023-11-28T22:08:19.286 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_57_00_UTC 2023-11-28T22:08:19.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.286 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_57_00_UTC) 2023-11-28T22:08:19.352 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_58_00_UTC 2023-11-28T22:08:19.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.352 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_58_00_UTC) 2023-11-28T22:08:19.416 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_59_00_UTC 2023-11-28T22:08:19.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.417 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-21_59_00_UTC) 2023-11-28T22:08:19.483 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_00_00_UTC 2023-11-28T22:08:19.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.483 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_00_00_UTC) 2023-11-28T22:08:19.547 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_01_00_UTC 2023-11-28T22:08:19.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.548 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_01_00_UTC) 2023-11-28T22:08:19.613 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_02_00_UTC 2023-11-28T22:08:19.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.613 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_02_00_UTC) 2023-11-28T22:08:19.677 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_03_00_UTC 2023-11-28T22:08:19.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.678 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_03_00_UTC) 2023-11-28T22:08:19.741 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_04_00_UTC 2023-11-28T22:08:19.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.742 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_04_00_UTC) 2023-11-28T22:08:19.811 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_05_00_UTC 2023-11-28T22:08:19.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.811 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_05_00_UTC) 2023-11-28T22:08:19.883 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_06_00_UTC 2023-11-28T22:08:19.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.883 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_06_00_UTC) 2023-11-28T22:08:19.952 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_07_00_UTC 2023-11-28T22:08:19.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:19.953 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_07_00_UTC) 2023-11-28T22:08:20.021 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_08_00_UTC 2023-11-28T22:08:20.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:20.022 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-11-28-22_08_00_UTC) 2023-11-28T22:08:20.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:20.089 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-11-28T22:08:20.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T22:08:20.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.382+0000 7f854bfff640 1 Processor -- start 2023-11-28T22:08:20.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.382+0000 7f854bfff640 1 -- start start 2023-11-28T22:08:20.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.384+0000 7f854bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f854c0cf940 0x7f854c161800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:20.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.384+0000 7f854bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f854c0cfd20 0x7f854c0b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:20.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.384+0000 7f854bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0b5070 0x7f854c0b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:20.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.384+0000 7f854bfff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854c0b2930 con 0x7f854c0cfd20 2023-11-28T22:08:20.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.384+0000 7f854bfff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854c161d40 con 0x7f854c0cf940 2023-11-28T22:08:20.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.384+0000 7f854bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c0b2650 con 0x7f854c0b5070 2023-11-28T22:08:20.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.385+0000 7f854a7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f854c0cfd20 0x7f854c0b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:20.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.385+0000 7f854a7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f854c0cfd20 0x7f854c0b4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42584/0 (socket says 172.21.15.45:42584) 2023-11-28T22:08:20.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.385+0000 7f854a7fc640 1 -- 172.21.15.45:0/323520673 learned_addr learned my addr 172.21.15.45:0/323520673 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:20.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.385+0000 7f854b7fe640 1 --2- 172.21.15.45:0/323520673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0b5070 0x7f854c0b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:20.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.385+0000 7f854a7fc640 1 -- 172.21.15.45:0/323520673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0b5070 msgr2=0x7f854c0b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:20.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.385+0000 7f854a7fc640 1 --2- 172.21.15.45:0/323520673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0b5070 0x7f854c0b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:20.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.385+0000 7f854a7fc640 1 -- 172.21.15.45:0/323520673 >> v2:172.21.15.190:3300/0 conn(0x7f854c0cf940 msgr2=0x7f854c161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:20.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.385+0000 7f854a7fc640 1 --2- 172.21.15.45:0/323520673 >> v2:172.21.15.190:3300/0 conn(0x7f854c0cf940 0x7f854c161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:20.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.385+0000 7f854a7fc640 1 -- 172.21.15.45:0/323520673 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c0ce240 con 0x7f854c0cfd20 2023-11-28T22:08:20.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.386+0000 7f854b7fe640 1 --2- 172.21.15.45:0/323520673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0b5070 0x7f854c0b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:20.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.387+0000 7f854a7fc640 1 --2- 172.21.15.45:0/323520673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cfd20 0x7f854c0b4b30 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f854c08b720 tx=0x7f854002b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f7024fbf96f9e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:20.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.388+0000 7f8549ffb640 1 -- 172.21.15.45:0/323520673 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540031760 con 0x7f854c0cfd20 2023-11-28T22:08:20.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.388+0000 7f8549ffb640 1 -- 172.21.15.45:0/323520673 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85400318c0 con 0x7f854c0cfd20 2023-11-28T22:08:20.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.388+0000 7f8549ffb640 1 -- 172.21.15.45:0/323520673 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540031c50 con 0x7f854c0cfd20 2023-11-28T22:08:20.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.389+0000 7f854bfff640 1 -- 172.21.15.45:0/323520673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cfd20 msgr2=0x7f854c0b4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:20.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.389+0000 7f854bfff640 1 --2- 172.21.15.45:0/323520673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cfd20 0x7f854c0b4b30 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f854c08b720 tx=0x7f854002b800 comp rx=0 tx=0).stop 2023-11-28T22:08:20.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.390+0000 7f854bfff640 1 -- 172.21.15.45:0/323520673 shutdown_connections 2023-11-28T22:08:20.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.390+0000 7f854bfff640 1 --2- 172.21.15.45:0/323520673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0b5070 0x7f854c0b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:20.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.390+0000 7f854bfff640 1 --2- 172.21.15.45:0/323520673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cfd20 0x7f854c0b4b30 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:20.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.390+0000 7f854bfff640 1 --2- 172.21.15.45:0/323520673 >> v2:172.21.15.190:3300/0 conn(0x7f854c0cf940 0x7f854c161800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:20.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.390+0000 7f854bfff640 1 -- 172.21.15.45:0/323520673 >> 172.21.15.45:0/323520673 conn(0x7f854c06a500 msgr2=0x7f854c06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:20.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.390+0000 7f854bfff640 1 -- 172.21.15.45:0/323520673 shutdown_connections 2023-11-28T22:08:20.392 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.390+0000 7f854bfff640 1 -- 172.21.15.45:0/323520673 wait complete. 2023-11-28T22:08:20.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.394+0000 7f854bfff640 1 Processor -- start 2023-11-28T22:08:20.395 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.394+0000 7f854bfff640 1 -- start start 2023-11-28T22:08:20.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f854c0b5070 0x7f854c1514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:20.397 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cf940 0x7f854c15eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:20.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0cfd20 0x7f854c0cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:20.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854bfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c15e6c0 con 0x7f854c0cf940 2023-11-28T22:08:20.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854bfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c1519e0 con 0x7f854c0b5070 2023-11-28T22:08:20.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c15da40 con 0x7f854c0cfd20 2023-11-28T22:08:20.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cf940 0x7f854c15eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:20.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cf940 0x7f854c15eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42600/0 (socket says 172.21.15.45:42600) 2023-11-28T22:08:20.398 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854a7fc640 1 -- 172.21.15.45:0/3823475407 learned_addr learned my addr 172.21.15.45:0/3823475407 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:20.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.396+0000 7f854b7fe640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0cfd20 0x7f854c0cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:20.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.397+0000 7f854a7fc640 1 -- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0cfd20 msgr2=0x7f854c0cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:20.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.397+0000 7f854a7fc640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0cfd20 0x7f854c0cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:20.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.397+0000 7f854a7fc640 1 -- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f854c0b5070 msgr2=0x7f854c1514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:20.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.397+0000 7f854a7fc640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f854c0b5070 0x7f854c1514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:20.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.397+0000 7f854a7fc640 1 -- 172.21.15.45:0/3823475407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85400094d0 con 0x7f854c0cf940 2023-11-28T22:08:20.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.398+0000 7f854a7fc640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cf940 0x7f854c15eae0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f85400090b0 tx=0x7f85400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:20.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.398+0000 7f8533fff640 1 -- 172.21.15.45:0/3823475407 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854003b800 con 0x7f854c0cf940 2023-11-28T22:08:20.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.398+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c15dd20 con 0x7f854c0cf940 2023-11-28T22:08:20.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.399+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c15e2e0 con 0x7f854c0cf940 2023-11-28T22:08:20.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.399+0000 7f8533fff640 1 -- 172.21.15.45:0/3823475407 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854003b960 con 0x7f854c0cf940 2023-11-28T22:08:20.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.399+0000 7f8533fff640 1 -- 172.21.15.45:0/3823475407 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854003bd10 con 0x7f854c0cf940 2023-11-28T22:08:20.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.401+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852c005250 con 0x7f854c0cf940 2023-11-28T22:08:20.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.403+0000 7f8533fff640 1 -- 172.21.15.45:0/3823475407 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 112) v1 ==== 89362+0+0 (secure 0 0 0) 0x7f854004e450 con 0x7f854c0cf940 2023-11-28T22:08:20.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.405+0000 7f8533fff640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f853806b000 0x7f853806d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:20.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.406+0000 7f8533fff640 1 -- 172.21.15.45:0/3823475407 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6833+0+0 (secure 0 0 0) 0x7f854002f050 con 0x7f854c0cf940 2023-11-28T22:08:20.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.407+0000 7f854affd640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f853806b000 0x7f853806d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:20.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.410+0000 7f854affd640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f853806b000 0x7f853806d4c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f853c009670 tx=0x7f853c006bf0 comp rx=0 tx=0).ready entity=mgr.8488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:20.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.414+0000 7f8533fff640 1 -- 172.21.15.45:0/3823475407 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8540076520 con 0x7f854c0cf940 2023-11-28T22:08:20.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:20.729+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f852c005a80 con 0x7f854c0cf940 2023-11-28T22:08:21.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:08:21.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:08:21.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.365+0000 7f8533fff640 1 -- 172.21.15.45:0/3823475407 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v113) v1 ==== 93+0+0 (secure 0 0 0) 0x7f8540079b40 con 0x7f854c0cf940 2023-11-28T22:08:21.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.373+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f853806b000 msgr2=0x7f853806d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:21.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.373+0000 7f854bfff640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f853806b000 0x7f853806d4c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f853c009670 tx=0x7f853c006bf0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.373+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cf940 msgr2=0x7f854c15eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:21.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.373+0000 7f854bfff640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cf940 0x7f854c15eae0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f85400090b0 tx=0x7f85400046c0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.376+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 shutdown_connections 2023-11-28T22:08:21.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.376+0000 7f854bfff640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f854c0cfd20 0x7f854c0cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.376+0000 7f854bfff640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f853806b000 0x7f853806d4c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.376+0000 7f854bfff640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f854c0cf940 0x7f854c15eae0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.376+0000 7f854bfff640 1 --2- 172.21.15.45:0/3823475407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f854c0b5070 0x7f854c1514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.376+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 >> 172.21.15.45:0/3823475407 conn(0x7f854c06a500 msgr2=0x7f854c06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:21.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.378+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 shutdown_connections 2023-11-28T22:08:21.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.378+0000 7f854bfff640 1 -- 172.21.15.45:0/3823475407 wait complete. 2023-11-28T22:08:21.401 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T22:08:21.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T22:08:21.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T22:08:21.595 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T22:08:21.595 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:08:21.595 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:21.651 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:08:21.651 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:21.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:08:21.923 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:21.921+0000 7f46a5c46040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T22:08:21.935 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:21.934+0000 7f4ee7830040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T22:08:21.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.948+0000 7f980340e640 1 Processor -- start 2023-11-28T22:08:21.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.948+0000 7f980340e640 1 -- start start 2023-11-28T22:08:21.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.950+0000 7f980340e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f97fc15e570 0x7f97fc0ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:21.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.950+0000 7f980340e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97fc0ad760 0x7f97fc0b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:21.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.950+0000 7f980340e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc0b2650 con 0x7f97fc15e180 2023-11-28T22:08:21.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.950+0000 7f980340e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97fc0adde0 con 0x7f97fc0ad760 2023-11-28T22:08:21.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.950+0000 7f980340e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97fc0adbf0 con 0x7f97fc15e570 2023-11-28T22:08:21.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.950+0000 7f98019ab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97fc0ad760 0x7f97fc0b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:21.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.950+0000 7f98019ab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97fc0ad760 0x7f97fc0b2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42628/0 (socket says 172.21.15.45:42628) 2023-11-28T22:08:21.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.950+0000 7f98019ab640 1 -- 172.21.15.45:0/980411316 learned_addr learned my addr 172.21.15.45:0/980411316 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:21.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.950+0000 7f98009a9640 1 --2- 172.21.15.45:0/980411316 >> v2:172.21.15.190:3300/0 conn(0x7f97fc15e570 0x7f97fc0ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:21.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.951+0000 7f98019ab640 1 -- 172.21.15.45:0/980411316 >> v2:172.21.15.190:3300/0 conn(0x7f97fc15e570 msgr2=0x7f97fc0ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:21.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.951+0000 7f98019ab640 1 --2- 172.21.15.45:0/980411316 >> v2:172.21.15.190:3300/0 conn(0x7f97fc15e570 0x7f97fc0ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.951+0000 7f98019ab640 1 -- 172.21.15.45:0/980411316 >> v1:172.21.15.190:6789/0 conn(0x7f97fc15e180 legacy=0x7f97fc070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:21.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.951+0000 7f98019ab640 1 -- 172.21.15.45:0/980411316 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc0713d0 con 0x7f97fc0ad760 2023-11-28T22:08:21.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.952+0000 7f98019ab640 1 --2- 172.21.15.45:0/980411316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc0ad760 0x7f97fc0b2ae0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009870 tx=0x7f97f402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ffdd8bb2556d43 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:21.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.952+0000 7f97f3fff640 1 -- 172.21.15.45:0/980411316 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f40317e0 con 0x7f97fc0ad760 2023-11-28T22:08:21.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.952+0000 7f97f3fff640 1 -- 172.21.15.45:0/980411316 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f4031940 con 0x7f97fc0ad760 2023-11-28T22:08:21.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.952+0000 7f97f3fff640 1 -- 172.21.15.45:0/980411316 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4031cd0 con 0x7f97fc0ad760 2023-11-28T22:08:21.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.953+0000 7f980340e640 1 -- 172.21.15.45:0/980411316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc0ad760 msgr2=0x7f97fc0b2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:21.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.953+0000 7f980340e640 1 --2- 172.21.15.45:0/980411316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc0ad760 0x7f97fc0b2ae0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009870 tx=0x7f97f402b880 comp rx=0 tx=0).stop 2023-11-28T22:08:21.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.953+0000 7f980340e640 1 -- 172.21.15.45:0/980411316 shutdown_connections 2023-11-28T22:08:21.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.953+0000 7f980340e640 1 --2- 172.21.15.45:0/980411316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc0ad760 0x7f97fc0b2ae0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.953+0000 7f980340e640 1 --2- 172.21.15.45:0/980411316 >> v2:172.21.15.190:3300/0 conn(0x7f97fc15e570 0x7f97fc0ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.953+0000 7f980340e640 1 -- 172.21.15.45:0/980411316 >> 172.21.15.45:0/980411316 conn(0x7f97fc0b1350 msgr2=0x7f97fc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:21.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.953+0000 7f980340e640 1 -- 172.21.15.45:0/980411316 shutdown_connections 2023-11-28T22:08:21.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.953+0000 7f980340e640 1 -- 172.21.15.45:0/980411316 wait complete. 2023-11-28T22:08:21.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.955+0000 7f980340e640 1 Processor -- start 2023-11-28T22:08:21.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.955+0000 7f980340e640 1 -- start start 2023-11-28T22:08:21.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f980340e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97fc0ad760 0x7f97fc144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:21.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f980340e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97fc15e570 0x7f97fc0cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:21.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f980340e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc151580 0x7f97fc06dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:21.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f980340e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc144e70 con 0x7f97fc151580 2023-11-28T22:08:21.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f980340e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1445f0 con 0x7f97fc0ad760 2023-11-28T22:08:21.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f980340e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1529f0 con 0x7f97fc15e570 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f98019ab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc151580 0x7f97fc06dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f98019ab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc151580 0x7f97fc06dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42632/0 (socket says 172.21.15.45:42632) 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f98019ab640 1 -- 172.21.15.45:0/2095421571 learned_addr learned my addr 172.21.15.45:0/2095421571 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f98011aa640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97fc0ad760 0x7f97fc144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.956+0000 7f98009a9640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97fc15e570 0x7f97fc0cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f98019ab640 1 -- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97fc15e570 msgr2=0x7f97fc0cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f98019ab640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97fc15e570 0x7f97fc0cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f98019ab640 1 -- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97fc0ad760 msgr2=0x7f97fc144930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f98019ab640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97fc0ad760 0x7f97fc144930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f98019ab640 1 -- 172.21.15.45:0/2095421571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f40094d0 con 0x7f97fc151580 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f98009a9640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97fc15e570 0x7f97fc0cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:21.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f98019ab640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc151580 0x7f97fc06dd60 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f97f402b8c0 tx=0x7f97f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:21.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f97f1ffb640 1 -- 172.21.15.45:0/2095421571 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4032880 con 0x7f97fc151580 2023-11-28T22:08:21.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f97f1ffb640 1 -- 172.21.15.45:0/2095421571 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f4032a00 con 0x7f97fc151580 2023-11-28T22:08:21.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f97f1ffb640 1 -- 172.21.15.45:0/2095421571 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4032db0 con 0x7f97fc151580 2023-11-28T22:08:21.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97fc152cd0 con 0x7f97fc151580 2023-11-28T22:08:21.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.957+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc153290 con 0x7f97fc151580 2023-11-28T22:08:21.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.959+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc0733d0 con 0x7f97fc151580 2023-11-28T22:08:21.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.959+0000 7f97f1ffb640 1 -- 172.21.15.45:0/2095421571 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 113) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f97f404d3f0 con 0x7f97fc151580 2023-11-28T22:08:21.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.960+0000 7f97f1ffb640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f97e006af60 0x7f97e006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:21.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.960+0000 7f97f1ffb640 1 -- 172.21.15.45:0/2095421571 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f97f40aba10 con 0x7f97fc151580 2023-11-28T22:08:21.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.961+0000 7f98011aa640 1 -- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f97e006af60 msgr2=0x7f97e006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:21.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.961+0000 7f98011aa640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f97e006af60 0x7f97e006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:21.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:21.965+0000 7f97f1ffb640 1 -- 172.21.15.45:0/2095421571 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f97f4076580 con 0x7f97fc151580 2023-11-28T22:08:21.980 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:21.978+0000 7f46a5c46040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T22:08:21.993 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:21.992+0000 7f4ee7830040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T22:08:22.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.162+0000 7f98011aa640 1 -- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f97e006af60 msgr2=0x7f97e006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:22.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.162+0000 7f98011aa640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f97e006af60 0x7f97e006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:08:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.168+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97fc073e00 con 0x7f97fc151580 2023-11-28T22:08:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.169+0000 7f97f1ffb640 1 -- 172.21.15.45:0/2095421571 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v203) v1 ==== 74+0+17905 (secure 0 0 0) 0x7f97f4079ba0 con 0x7f97fc151580 2023-11-28T22:08:22.171 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:22.171 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":203,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:08:21.350175+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:52:23.878130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-11-28T21:52:24.956954+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":203,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3880574704":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6833/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6832/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3938773910":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2706131286":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3522054455":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2512261826":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3252656094":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2538549380":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3848257557":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6833/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3811510364":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2138187339":"2023-11-29T21:52:41.039626+0000","172.21.15.190:6832/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/624508208":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2094131491":"2023-11-29T21:52:41.039626+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:08:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.172+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f97e006af60 msgr2=0x7f97e006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.172+0000 7f980340e640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f97e006af60 0x7f97e006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.172+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc151580 msgr2=0x7f97fc06dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.172+0000 7f980340e640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc151580 0x7f97fc06dd60 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f97f402b8c0 tx=0x7f97f40046c0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.173+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 shutdown_connections 2023-11-28T22:08:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.173+0000 7f980340e640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f97e006af60 0x7f97e006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.173+0000 7f980340e640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97fc151580 0x7f97fc06dd60 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.173+0000 7f980340e640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97fc15e570 0x7f97fc0cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.173+0000 7f980340e640 1 --2- 172.21.15.45:0/2095421571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97fc0ad760 0x7f97fc144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.173+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 >> 172.21.15.45:0/2095421571 conn(0x7f97fc0b1350 msgr2=0x7f97fc15ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:22.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.173+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 shutdown_connections 2023-11-28T22:08:22.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.173+0000 7f980340e640 1 -- 172.21.15.45:0/2095421571 wait complete. 2023-11-28T22:08:22.191 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T21:52:23.878130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '198', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T21:52:24.956954+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '203', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 203, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T22:08:22.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T22:08:22.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.398+0000 7fa483e9b640 1 Processor -- start 2023-11-28T22:08:22.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.398+0000 7fa483e9b640 1 -- start start 2023-11-28T22:08:22.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.399+0000 7fa483e9b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 0x7fa47c0b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.399+0000 7fa483e9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa47c0ce7f0 0x7fa47c0d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.399+0000 7fa483e9b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c160e80 con 0x7fa47c0d24f0 2023-11-28T22:08:22.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.399+0000 7fa483e9b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47c0d1220 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.399+0000 7fa483e9b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c0d2e70 con 0x7fa47c160aa0 2023-11-28T22:08:22.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.400+0000 7fa482438640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa47c0ce7f0 0x7fa47c0d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.400+0000 7fa481436640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa47c0d24f0 0x7fa47c0ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36888/0 (socket says 172.21.15.45:36888) 2023-11-28T22:08:22.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.400+0000 7fa481436640 1 -- 172.21.15.45:0/2191751392 learned_addr learned my addr 172.21.15.45:0/2191751392 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:22.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.400+0000 7fa481c37640 1 --2- 172.21.15.45:0/2191751392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 0x7fa47c0b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.400+0000 7fa482438640 1 -- 172.21.15.45:0/2191751392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 msgr2=0x7fa47c0b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:22.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.400+0000 7fa482438640 1 --2- 172.21.15.45:0/2191751392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 0x7fa47c0b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.400+0000 7fa482438640 1 -- 172.21.15.45:0/2191751392 >> v1:172.21.15.45:6789/0 conn(0x7fa47c0d24f0 legacy=0x7fa47c0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.400+0000 7fa480c35640 1 -- 172.21.15.45:0/2191751392 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119461400 0 0) 0x7fa47c160e80 con 0x7fa47c0d24f0 2023-11-28T22:08:22.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.401+0000 7fa482438640 1 -- 172.21.15.45:0/2191751392 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c0b6960 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.402+0000 7fa482438640 1 --2- 172.21.15.45:0/2191751392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 0x7fa47c0d2930 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fa474009d90 tx=0x7fa47402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17406f0bb17396f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:22.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.402+0000 7fa480c35640 1 -- 172.21.15.45:0/2191751392 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47402cce0 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.402+0000 7fa480c35640 1 -- 172.21.15.45:0/2191751392 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47402ce40 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.402+0000 7fa480c35640 1 -- 172.21.15.45:0/2191751392 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa474034660 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.402+0000 7fa483e9b640 1 -- 172.21.15.45:0/2191751392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 msgr2=0x7fa47c0d2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.402+0000 7fa483e9b640 1 --2- 172.21.15.45:0/2191751392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 0x7fa47c0d2930 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fa474009d90 tx=0x7fa47402bf30 comp rx=0 tx=0).stop 2023-11-28T22:08:22.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.403+0000 7fa483e9b640 1 -- 172.21.15.45:0/2191751392 shutdown_connections 2023-11-28T22:08:22.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.403+0000 7fa483e9b640 1 --2- 172.21.15.45:0/2191751392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 0x7fa47c0d2930 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.403+0000 7fa483e9b640 1 --2- 172.21.15.45:0/2191751392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 0x7fa47c0b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.403+0000 7fa483e9b640 1 -- 172.21.15.45:0/2191751392 >> 172.21.15.45:0/2191751392 conn(0x7fa47c070b20 msgr2=0x7fa47c070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:22.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.403+0000 7fa483e9b640 1 -- 172.21.15.45:0/2191751392 shutdown_connections 2023-11-28T22:08:22.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.403+0000 7fa483e9b640 1 -- 172.21.15.45:0/2191751392 wait complete. 2023-11-28T22:08:22.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.404+0000 7fa483e9b640 1 Processor -- start 2023-11-28T22:08:22.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.404+0000 7fa483e9b640 1 -- start start 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.405+0000 7fa483e9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 0x7fa47c15af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.405+0000 7fa483e9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa47c0d24f0 0x7fa47c0b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.405+0000 7fa483e9b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 0x7fa47c0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.405+0000 7fa483e9b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c15b4d0 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.405+0000 7fa483e9b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c0b2cf0 con 0x7fa47c0d24f0 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.405+0000 7fa483e9b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c0c9850 con 0x7fa47c160aa0 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa481c37640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 0x7fa47c15af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa481c37640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 0x7fa47c15af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42646/0 (socket says 172.21.15.45:42646) 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa481c37640 1 -- 172.21.15.45:0/980877353 learned_addr learned my addr 172.21.15.45:0/980877353 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa482438640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 0x7fa47c0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa481436640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa47c0d24f0 0x7fa47c0b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa481c37640 1 -- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 msgr2=0x7fa47c0c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa481c37640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 0x7fa47c0c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa481c37640 1 -- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa47c0d24f0 msgr2=0x7fa47c0b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa481c37640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa47c0d24f0 0x7fa47c0b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.406+0000 7fa481c37640 1 -- 172.21.15.45:0/980877353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474009a70 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.407+0000 7fa481c37640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 0x7fa47c15af90 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e870 tx=0x7fa47800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.407+0000 7fa4727fc640 1 -- 172.21.15.45:0/980877353 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4780094f0 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.407+0000 7fa4727fc640 1 -- 172.21.15.45:0/980877353 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47800c940 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.407+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47c0b2f90 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.407+0000 7fa4727fc640 1 -- 172.21.15.45:0/980877353 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47800ccf0 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.407+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47c1fae80 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.408+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47c0733d0 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.409+0000 7fa4727fc640 1 -- 172.21.15.45:0/980877353 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 113) v1 ==== 89345+0+0 (secure 0 0 0) 0x7fa478013470 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.410+0000 7fa4727fc640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa46006af60 0x7fa46006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.410+0000 7fa4727fc640 1 -- 172.21.15.45:0/980877353 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6817+0+0 (secure 0 0 0) 0x7fa478089c90 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.411+0000 7fa481436640 1 -- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa46006af60 msgr2=0x7fa46006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:22.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.411+0000 7fa481436640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa46006af60 0x7fa46006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.415+0000 7fa4727fc640 1 -- 172.21.15.45:0/980877353 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa478054860 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.438 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.435+0000 7f46a5c46040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T22:08:22.484 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:22.483+0000 7f4ee7830040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T22:08:22.488 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.486+0000 7f46a5c46040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T22:08:22.538 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.535+0000 7f46a5c46040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T22:08:22.539 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:22.537+0000 7f4ee7830040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T22:08:22.591 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:22.590+0000 7f4ee7830040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T22:08:22.598 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.596+0000 7f46a5c46040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T22:08:22.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.608+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa47c0d1000 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.609+0000 7fa4727fc640 1 -- 172.21.15.45:0/980877353 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v203) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa4780210a0 con 0x7fa47c0ce7f0 2023-11-28T22:08:22.610 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa46006af60 msgr2=0x7fa46006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa46006af60 0x7fa46006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 msgr2=0x7fa47c15af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 0x7fa47c15af90 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e870 tx=0x7fa47800ed30 comp rx=0 tx=0).stop 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 shutdown_connections 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c160aa0 0x7fa47c0c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa47c0d24f0 0x7fa47c0b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa46006af60 0x7fa46006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 --2- 172.21.15.45:0/980877353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa47c0ce7f0 0x7fa47c15af90 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 >> 172.21.15.45:0/980877353 conn(0x7fa47c070b20 msgr2=0x7fa47c15e190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:22.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 shutdown_connections 2023-11-28T22:08:22.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.612+0000 7fa483e9b640 1 -- 172.21.15.45:0/980877353 wait complete. 2023-11-28T22:08:22.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T22:08:22.646 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.644+0000 7f46a5c46040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T22:08:22.657 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:22.656+0000 7f4ee7830040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T22:08:22.710 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:22.708+0000 7f4ee7830040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T22:08:22.756 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.754+0000 7f46a5c46040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T22:08:22.818 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.815+0000 7f46a5c46040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T22:08:22.829 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:22.827+0000 7f4ee7830040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T22:08:22.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.841+0000 7f3a4d891640 1 Processor -- start 2023-11-28T22:08:22.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.842+0000 7f3a4d891640 1 -- start start 2023-11-28T22:08:22.844 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.843+0000 7f3a4d891640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a480b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.843+0000 7f3a4d891640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3a48071ac0 0x7f3a480b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.843+0000 7f3a4d891640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a480b27a0 con 0x7f3a48072010 2023-11-28T22:08:22.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.843+0000 7f3a4d891640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a48071450 con 0x7f3a48071ac0 2023-11-28T22:08:22.845 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.843+0000 7f3a4d891640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4809ceb0 con 0x7f3a48159fc0 2023-11-28T22:08:22.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.844+0000 7f3a4c88f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a480b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.844+0000 7f3a47fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3a48071ac0 0x7f3a480b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.844+0000 7f3a47fff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3a48071ac0 0x7f3a480b2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:59728/0 (socket says 172.21.15.45:59728) 2023-11-28T22:08:22.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.844+0000 7f3a4c88f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a480b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:53440/0 (socket says 172.21.15.45:53440) 2023-11-28T22:08:22.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.844+0000 7f3a47fff640 1 -- 172.21.15.45:0/4086740572 learned_addr learned my addr 172.21.15.45:0/4086740572 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:22.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.845+0000 7f3a47fff640 1 -- 172.21.15.45:0/4086740572 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 msgr2=0x7f3a480b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:22.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.845+0000 7f3a47fff640 1 --2- 172.21.15.45:0/4086740572 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a480b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.845+0000 7f3a47fff640 1 -- 172.21.15.45:0/4086740572 >> v1:172.21.15.190:6789/0 conn(0x7f3a48072010 legacy=0x7f3a4815dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.845+0000 7f3a47fff640 1 -- 172.21.15.45:0/4086740572 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a480b2cb0 con 0x7f3a48071ac0 2023-11-28T22:08:22.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.847+0000 7f3a47fff640 1 --2- 172.21.15.45:0/4086740572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 0x7f3a480b2260 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009870 tx=0x7f3a3402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5128a1c740bdc60 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:22.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.848+0000 7f3a477fe640 1 -- 172.21.15.45:0/4086740572 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34031640 con 0x7f3a48071ac0 2023-11-28T22:08:22.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.848+0000 7f3a477fe640 1 -- 172.21.15.45:0/4086740572 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a340317a0 con 0x7f3a48071ac0 2023-11-28T22:08:22.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.848+0000 7f3a477fe640 1 -- 172.21.15.45:0/4086740572 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34031b30 con 0x7f3a48071ac0 2023-11-28T22:08:22.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.848+0000 7f3a4d891640 1 -- 172.21.15.45:0/4086740572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 msgr2=0x7f3a480b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.848+0000 7f3a4d891640 1 --2- 172.21.15.45:0/4086740572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 0x7f3a480b2260 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009870 tx=0x7f3a3402b8f0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.848+0000 7f3a4d891640 1 -- 172.21.15.45:0/4086740572 shutdown_connections 2023-11-28T22:08:22.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.848+0000 7f3a4d891640 1 --2- 172.21.15.45:0/4086740572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 0x7f3a480b2260 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.848+0000 7f3a4d891640 1 --2- 172.21.15.45:0/4086740572 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a480b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.848+0000 7f3a4d891640 1 -- 172.21.15.45:0/4086740572 >> 172.21.15.45:0/4086740572 conn(0x7f3a480b1050 msgr2=0x7f3a480cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:22.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.849+0000 7f3a4d891640 1 -- 172.21.15.45:0/4086740572 shutdown_connections 2023-11-28T22:08:22.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.849+0000 7f3a4d891640 1 -- 172.21.15.45:0/4086740572 wait complete. 2023-11-28T22:08:22.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.850+0000 7f3a4d891640 1 Processor -- start 2023-11-28T22:08:22.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.851+0000 7f3a4d891640 1 -- start start 2023-11-28T22:08:22.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a4d891640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 0x7f3a4809c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a4d891640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a48072010 0x7f3a480cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a4d891640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a4809a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a4d891640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4809b230 con 0x7f3a48072010 2023-11-28T22:08:22.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a4d891640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4809b510 con 0x7f3a48071ac0 2023-11-28T22:08:22.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a4d891640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4809b7f0 con 0x7f3a48159fc0 2023-11-28T22:08:22.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a47fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a48072010 0x7f3a480cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a47fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a48072010 0x7f3a480cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42662/0 (socket says 172.21.15.45:42662) 2023-11-28T22:08:22.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a47fff640 1 -- 172.21.15.45:0/3659803117 learned_addr learned my addr 172.21.15.45:0/3659803117 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:22.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a4c88f640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 0x7f3a4809c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.852+0000 7f3a4d090640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a4809a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:22.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a47fff640 1 -- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 msgr2=0x7f3a4809a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a47fff640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a4809a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a47fff640 1 -- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 msgr2=0x7f3a4809c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:22.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a47fff640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 0x7f3a4809c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:22.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a47fff640 1 -- 172.21.15.45:0/3659803117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a340094d0 con 0x7f3a48072010 2023-11-28T22:08:22.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a4d090640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a4809a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:08:22.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a4c88f640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 0x7f3a4809c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:08:22.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a47fff640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a48072010 0x7f3a480cb7a0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009870 tx=0x7f3a340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:22.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a457fa640 1 -- 172.21.15.45:0/3659803117 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3403a930 con 0x7f3a48072010 2023-11-28T22:08:22.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a457fa640 1 -- 172.21.15.45:0/3659803117 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3403aa90 con 0x7f3a48072010 2023-11-28T22:08:22.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a457fa640 1 -- 172.21.15.45:0/3659803117 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3403ae40 con 0x7f3a48072010 2023-11-28T22:08:22.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.853+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4809bad0 con 0x7f3a48072010 2023-11-28T22:08:22.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.854+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4809c810 con 0x7f3a48072010 2023-11-28T22:08:22.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.855+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4804e920 con 0x7f3a48072010 2023-11-28T22:08:22.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.856+0000 7f3a457fa640 1 -- 172.21.15.45:0/3659803117 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 113) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f3a3402f050 con 0x7f3a48072010 2023-11-28T22:08:22.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.857+0000 7f3a457fa640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3a2c06af60 0x7f3a2c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:22.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.857+0000 7f3a457fa640 1 -- 172.21.15.45:0/3659803117 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f3a340abff0 con 0x7f3a48072010 2023-11-28T22:08:22.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.857+0000 7f3a4c88f640 1 -- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3a2c06af60 msgr2=0x7f3a2c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:22.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.857+0000 7f3a4c88f640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3a2c06af60 0x7f3a2c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:22.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:22.860+0000 7f3a457fa640 1 -- 172.21.15.45:0/3659803117 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a34076b40 con 0x7f3a48072010 2023-11-28T22:08:22.868 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.866+0000 7f46a5c46040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T22:08:22.898 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:22.897+0000 7f4ee7830040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T22:08:22.925 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.922+0000 7f46a5c46040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T22:08:22.951 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:22.949+0000 7f4ee7830040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T22:08:22.974 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:22.972+0000 7f46a5c46040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T22:08:23.010 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:23.009+0000 7f4ee7830040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T22:08:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.052+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3a480cc490 con 0x7f3a48072010 2023-11-28T22:08:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.053+0000 7f3a457fa640 1 -- 172.21.15.45:0/3659803117 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v203) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3a3407a160 con 0x7f3a48072010 2023-11-28T22:08:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T22:08:23.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3a2c06af60 msgr2=0x7f3a2c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3a2c06af60 0x7f3a2c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a48072010 msgr2=0x7f3a480cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a48072010 0x7f3a480cb7a0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009870 tx=0x7f3a340046c0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 shutdown_connections 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3a48159fc0 0x7f3a4809a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3a2c06af60 0x7f3a2c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a48072010 0x7f3a480cb7a0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 --2- 172.21.15.45:0/3659803117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a48071ac0 0x7f3a4809c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 >> 172.21.15.45:0/3659803117 conn(0x7f3a480b1050 msgr2=0x7f3a4806d530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 shutdown_connections 2023-11-28T22:08:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.056+0000 7f3a4d891640 1 -- 172.21.15.45:0/3659803117 wait complete. 2023-11-28T22:08:23.064 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:23.063+0000 7f4ee7830040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T22:08:23.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T22:08:23.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.331+0000 7f5933351640 1 Processor -- start 2023-11-28T22:08:23.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.331+0000 7f5933351640 1 -- start start 2023-11-28T22:08:23.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.333+0000 7f5933351640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c15d3c0 0x7f592c0cda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.333+0000 7f5933351640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592c0d2e70 con 0x7f592c15d7a0 2023-11-28T22:08:23.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.333+0000 7f5933351640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592c0ce110 con 0x7f592c0d02d0 2023-11-28T22:08:23.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.333+0000 7f5933351640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0ce3f0 con 0x7f592c15d3c0 2023-11-28T22:08:23.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.333+0000 7f59308ec640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f592c15d7a0 0x7f592c0cfbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36900/0 (socket says 172.21.15.45:36900) 2023-11-28T22:08:23.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.333+0000 7f59308ec640 1 -- 172.21.15.45:0/1681059404 learned_addr learned my addr 172.21.15.45:0/1681059404 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:23.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f59310ed640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c15d3c0 0x7f592c0cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297363113 0 0) 0x7f592c0d2e70 con 0x7f592c15d7a0 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5918003520 con 0x7f592c15d7a0 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991819550 0 0) 0x7f5918003520 con 0x7f592c15d7a0 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c15d3c0 msgr2=0x7f592c0cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c15d3c0 0x7f592c0cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 >> v1:172.21.15.190:6789/0 conn(0x7f592c0d02d0 legacy=0x7f592c0d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c0ce6d0 con 0x7f592c15d7a0 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f591c001670 con 0x7f592c15d7a0 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 >> v1:172.21.15.45:6789/0 conn(0x7f592c15d7a0 legacy=0x7f592c0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.334+0000 7f5923fff640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f5918003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f5923fff640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59180047c0 0x7f5918004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f5923fff640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c15d3c0 0x7f59180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0ce3f0 con 0x7f59180039a0 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591800bab0 con 0x7f59180047c0 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591800bc70 con 0x7f592c15d3c0 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f59310ed640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f5918003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f59318ee640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c15d3c0 0x7f59180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f59308ec640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59180047c0 0x7f5918004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f59310ed640 1 -- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c15d3c0 msgr2=0x7f59180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f59310ed640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c15d3c0 0x7f59180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f59310ed640 1 -- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59180047c0 msgr2=0x7f5918004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f59310ed640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59180047c0 0x7f5918004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.335+0000 7f59310ed640 1 -- 172.21.15.45:0/1681059404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918003520 con 0x7f59180039a0 2023-11-28T22:08:23.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.336+0000 7f59310ed640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f5918003dc0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f5928009ac0 tx=0x7f59180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c32212c0d78e25b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:23.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.337+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592802c6d0 con 0x7f59180039a0 2023-11-28T22:08:23.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.337+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592802c830 con 0x7f59180039a0 2023-11-28T22:08:23.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.337+0000 7f5923fff640 1 -- 172.21.15.45:0/1681059404 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592802cbc0 con 0x7f59180039a0 2023-11-28T22:08:23.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.337+0000 7f5933351640 1 -- 172.21.15.45:0/1681059404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 msgr2=0x7f5918003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.337+0000 7f5933351640 1 --2- 172.21.15.45:0/1681059404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f5918003dc0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f5928009ac0 tx=0x7f59180057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.337+0000 7f59318ee640 1 -- 172.21.15.45:0/1681059404 reap_dead start 2023-11-28T22:08:23.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.338+0000 7f5933351640 1 -- 172.21.15.45:0/1681059404 shutdown_connections 2023-11-28T22:08:23.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.338+0000 7f5933351640 1 -- 172.21.15.45:0/1681059404 >> 172.21.15.45:0/1681059404 conn(0x7f592c0b4ca0 msgr2=0x7f592c0b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:23.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.338+0000 7f5933351640 1 -- 172.21.15.45:0/1681059404 shutdown_connections 2023-11-28T22:08:23.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.338+0000 7f5933351640 1 -- 172.21.15.45:0/1681059404 wait complete. 2023-11-28T22:08:23.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.340+0000 7f5933351640 1 Processor -- start 2023-11-28T22:08:23.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.340+0000 7f5933351640 1 -- start start 2023-11-28T22:08:23.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f5933351640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f592c152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f5933351640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c0cac90 0x7f592c153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f5933351640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f592c0cb400 0x7f592c1f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f5933351640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c156060 con 0x7f59180039a0 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f5933351640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c155450 con 0x7f592c0cb400 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f5933351640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592c155730 con 0x7f592c0cac90 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59310ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f592c152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59310ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f592c152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42676/0 (socket says 172.21.15.45:42676) 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59310ed640 1 -- 172.21.15.45:0/2265119799 learned_addr learned my addr 172.21.15.45:0/2265119799 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59308ec640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c0cac90 0x7f592c153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59318ee640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f592c0cb400 0x7f592c1f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59310ed640 1 -- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c0cac90 msgr2=0x7f592c153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59310ed640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c0cac90 0x7f592c153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59310ed640 1 -- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f592c0cb400 msgr2=0x7f592c1f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59310ed640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f592c0cb400 0x7f592c1f6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59310ed640 1 -- 172.21.15.45:0/2265119799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c155a10 con 0x7f59180039a0 2023-11-28T22:08:23.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.341+0000 7f59318ee640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f592c0cb400 0x7f592c1f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:23.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.342+0000 7f59310ed640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f592c152df0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f5928037040 tx=0x7f5928004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:23.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.342+0000 7f5921ffb640 1 -- 172.21.15.45:0/2265119799 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928003810 con 0x7f59180039a0 2023-11-28T22:08:23.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.342+0000 7f5921ffb640 1 -- 172.21.15.45:0/2265119799 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5928003970 con 0x7f59180039a0 2023-11-28T22:08:23.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.342+0000 7f5921ffb640 1 -- 172.21.15.45:0/2265119799 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928003d20 con 0x7f59180039a0 2023-11-28T22:08:23.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.342+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592c156060 con 0x7f59180039a0 2023-11-28T22:08:23.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.342+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592c154ec0 con 0x7f59180039a0 2023-11-28T22:08:23.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.343+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592c0733d0 con 0x7f59180039a0 2023-11-28T22:08:23.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.345+0000 7f5921ffb640 1 -- 172.21.15.45:0/2265119799 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 113) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f5928051480 con 0x7f59180039a0 2023-11-28T22:08:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.346+0000 7f5921ffb640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f591006af60 0x7f591006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.346+0000 7f5921ffb640 1 -- 172.21.15.45:0/2265119799 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f59280ac2a0 con 0x7f59180039a0 2023-11-28T22:08:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.346+0000 7f59308ec640 1 -- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f591006af60 msgr2=0x7f591006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.346+0000 7f59308ec640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f591006af60 0x7f591006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:23.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.353+0000 7f5921ffb640 1 -- 172.21.15.45:0/2265119799 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5928076e70 con 0x7f59180039a0 2023-11-28T22:08:23.361 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:23.356+0000 7f46a5c46040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T22:08:23.476 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:23.475+0000 7f4ee7830040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T22:08:23.518 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:23.516+0000 7f46a5c46040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T22:08:23.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.544+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f592c0d0ec0 con 0x7f59180039a0 2023-11-28T22:08:23.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.545+0000 7f5921ffb640 1 -- 172.21.15.45:0/2265119799 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v203) v1 ==== 100+0+11 (secure 0 0 0) 0x7f592804c020 con 0x7f59180039a0 2023-11-28T22:08:23.546 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T22:08:23.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f59308ec640 1 -- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f591006af60 msgr2=0x7f591006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:23.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f59308ec640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f591006af60 0x7f591006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:08:23.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f591006af60 msgr2=0x7f591006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:23.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f591006af60 0x7f591006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 msgr2=0x7f592c152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f592c152df0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f5928037040 tx=0x7f5928004730 comp rx=0 tx=0).stop 2023-11-28T22:08:23.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 shutdown_connections 2023-11-28T22:08:23.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f592c0cb400 0x7f592c1f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f592c0cac90 0x7f592c153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f591006af60 0x7f591006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 --2- 172.21.15.45:0/2265119799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59180039a0 0x7f592c152df0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.547+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 >> 172.21.15.45:0/2265119799 conn(0x7f592c0b4ca0 msgr2=0x7f592c0b6490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:23.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.548+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 shutdown_connections 2023-11-28T22:08:23.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.548+0000 7f5933351640 1 -- 172.21.15.45:0/2265119799 wait complete. 2023-11-28T22:08:23.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T22:08:23.624 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:23.622+0000 7f46a5c46040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T22:08:23.647 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:23.645+0000 7f4ee7830040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T22:08:23.674 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:23.671+0000 7f46a5c46040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T22:08:23.763 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:23.762+0000 7f4ee7830040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T22:08:23.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.772+0000 7fa814a17640 1 Processor -- start 2023-11-28T22:08:23.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.772+0000 7fa814a17640 1 -- start start 2023-11-28T22:08:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.774+0000 7fa814a17640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8100cf730 0x7fa81006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.774+0000 7fa814a17640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa81015edc0 0x7fa81015f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.774+0000 7fa814a17640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81015fa80 con 0x7fa81015b820 2023-11-28T22:08:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.774+0000 7fa814a17640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81015fd60 con 0x7fa81015edc0 2023-11-28T22:08:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.774+0000 7fa814a17640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81006cf40 con 0x7fa8100cf730 2023-11-28T22:08:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.774+0000 7fa80ed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa81015edc0 0x7fa81015f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.774+0000 7fa80e59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa81015b820 0x7fa8100cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36904/0 (socket says 172.21.15.45:36904) 2023-11-28T22:08:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.774+0000 7fa80ed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa81015edc0 0x7fa81015f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42684/0 (socket says 172.21.15.45:42684) 2023-11-28T22:08:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.775+0000 7fa80e59c640 1 -- 172.21.15.45:0/228714324 learned_addr learned my addr 172.21.15.45:0/228714324 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:23.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.775+0000 7fa807fff640 1 --2- 172.21.15.45:0/228714324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8100cf730 0x7fa81006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.775+0000 7fa80ed9d640 1 -- 172.21.15.45:0/228714324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8100cf730 msgr2=0x7fa81006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:23.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.775+0000 7fa80ed9d640 1 --2- 172.21.15.45:0/228714324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8100cf730 0x7fa81006da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.775+0000 7fa80ed9d640 1 -- 172.21.15.45:0/228714324 >> v1:172.21.15.45:6789/0 conn(0x7fa81015b820 legacy=0x7fa8100cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.775+0000 7fa80ed9d640 1 -- 172.21.15.45:0/228714324 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81006d220 con 0x7fa81015edc0 2023-11-28T22:08:23.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.775+0000 7fa80dd9b640 1 -- 172.21.15.45:0/228714324 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040194605 0 0) 0x7fa81015fa80 con 0x7fa81015b820 2023-11-28T22:08:23.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.776+0000 7fa80ed9d640 1 --2- 172.21.15.45:0/228714324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81015edc0 0x7fa81015f1f0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fa808009d90 tx=0x7fa80802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d31d3062f7ae83 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:23.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.776+0000 7fa80dd9b640 1 -- 172.21.15.45:0/228714324 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80802cc50 con 0x7fa81015edc0 2023-11-28T22:08:23.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.776+0000 7fa80dd9b640 1 -- 172.21.15.45:0/228714324 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80802cdb0 con 0x7fa81015edc0 2023-11-28T22:08:23.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.776+0000 7fa80dd9b640 1 -- 172.21.15.45:0/228714324 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808034660 con 0x7fa81015edc0 2023-11-28T22:08:23.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.777+0000 7fa814a17640 1 -- 172.21.15.45:0/228714324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81015edc0 msgr2=0x7fa81015f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.777+0000 7fa814a17640 1 --2- 172.21.15.45:0/228714324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81015edc0 0x7fa81015f1f0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fa808009d90 tx=0x7fa80802bea0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.777+0000 7fa814a17640 1 -- 172.21.15.45:0/228714324 shutdown_connections 2023-11-28T22:08:23.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.777+0000 7fa814a17640 1 --2- 172.21.15.45:0/228714324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81015edc0 0x7fa81015f1f0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.777+0000 7fa814a17640 1 --2- 172.21.15.45:0/228714324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8100cf730 0x7fa81006da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.777+0000 7fa814a17640 1 -- 172.21.15.45:0/228714324 >> 172.21.15.45:0/228714324 conn(0x7fa81006a6e0 msgr2=0x7fa81006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:23.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.777+0000 7fa814a17640 1 -- 172.21.15.45:0/228714324 shutdown_connections 2023-11-28T22:08:23.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.777+0000 7fa814a17640 1 -- 172.21.15.45:0/228714324 wait complete. 2023-11-28T22:08:23.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.779+0000 7fa814a17640 1 Processor -- start 2023-11-28T22:08:23.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.779+0000 7fa814a17640 1 -- start start 2023-11-28T22:08:23.782 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:23.779+0000 7f46a5c46040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:08:23.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.780+0000 7fa814a17640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8100cf730 0x7fa81015dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.780+0000 7fa814a17640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa81015b820 0x7fa8101f6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.780+0000 7fa814a17640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa81015edc0 0x7fa8100c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.780+0000 7fa814a17640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81014f520 con 0x7fa8100cf730 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.780+0000 7fa814a17640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8101f6ed0 con 0x7fa81015b820 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.780+0000 7fa814a17640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81014e8a0 con 0x7fa81015edc0 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.780+0000 7fa80e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8100cf730 0x7fa81015dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa80e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8100cf730 0x7fa81015dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42686/0 (socket says 172.21.15.45:42686) 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa80e59c640 1 -- 172.21.15.45:0/1049125907 learned_addr learned my addr 172.21.15.45:0/1049125907 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa807fff640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa81015b820 0x7fa8101f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa80e59c640 1 -- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa81015edc0 msgr2=0x7fa8100c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa80ed9d640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa81015edc0 0x7fa8100c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa80e59c640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa81015edc0 0x7fa8100c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa80e59c640 1 -- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa81015b820 msgr2=0x7fa8101f6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:23.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa80e59c640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa81015b820 0x7fa8101f6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:23.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa80e59c640 1 -- 172.21.15.45:0/1049125907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa808009a70 con 0x7fa8100cf730 2023-11-28T22:08:23.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.781+0000 7fa80e59c640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8100cf730 0x7fa81015dc70 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fa80000c4d0 tx=0x7fa80000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:23.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.782+0000 7fa806ffd640 1 -- 172.21.15.45:0/1049125907 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8000074e0 con 0x7fa8100cf730 2023-11-28T22:08:23.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.782+0000 7fa806ffd640 1 -- 172.21.15.45:0/1049125907 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa800007660 con 0x7fa8100cf730 2023-11-28T22:08:23.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.782+0000 7fa806ffd640 1 -- 172.21.15.45:0/1049125907 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800004a20 con 0x7fa8100cf730 2023-11-28T22:08:23.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.782+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81014eb80 con 0x7fa8100cf730 2023-11-28T22:08:23.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.782+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81014f140 con 0x7fa8100cf730 2023-11-28T22:08:23.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.783+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81014f660 con 0x7fa8100cf730 2023-11-28T22:08:23.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.784+0000 7fa806ffd640 1 -- 172.21.15.45:0/1049125907 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 113) v1 ==== 89345+0+0 (secure 0 0 0) 0x7fa80001b360 con 0x7fa8100cf730 2023-11-28T22:08:23.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.785+0000 7fa806ffd640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa7f006af60 0x7fa7f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:23.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.785+0000 7fa806ffd640 1 -- 172.21.15.45:0/1049125907 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6817+0+0 (secure 0 0 0) 0x7fa800089100 con 0x7fa8100cf730 2023-11-28T22:08:23.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.786+0000 7fa807fff640 1 -- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa7f006af60 msgr2=0x7fa7f006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:23.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.786+0000 7fa807fff640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa7f006af60 0x7fa7f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:23.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.792+0000 7fa806ffd640 1 -- 172.21.15.45:0/1049125907 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa800053cb0 con 0x7fa8100cf730 2023-11-28T22:08:23.821 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:23.820+0000 7f4ee7830040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T22:08:23.889 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:23.887+0000 7f46a5c46040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T22:08:23.936 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:23.934+0000 7f4ee7830040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:08:23.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.987+0000 7fa807fff640 1 -- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa7f006af60 msgr2=0x7fa7f006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:23.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:23.987+0000 7fa807fff640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa7f006af60 0x7fa7f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:08:24.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.001+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa810073440 con 0x7fa8100cf730 2023-11-28T22:08:24.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.001+0000 7fa806ffd640 1 -- 172.21.15.45:0/1049125907 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v203) v1 ==== 81+0+955 (secure 0 0 0) 0x7fa800018070 con 0x7fa8100cf730 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3880574704 2023-11-29T22:03:21.258883+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/3833863335 2023-11-29T22:03:21.258883+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/3833863335 2023-11-29T22:03:21.258883+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3938773910 2023-11-29T22:03:21.258883+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2706131286 2023-11-29T22:03:21.258883+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3522054455 2023-11-29T22:03:21.258883+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2512261826 2023-11-29T22:03:21.258883+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3252656094 2023-11-29T21:52:41.039626+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2538549380 2023-11-29T21:52:41.039626+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3848257557 2023-11-29T22:03:21.258883+0000 2023-11-28T22:08:24.003 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/3252186332 2023-11-29T21:52:41.039626+0000 2023-11-28T22:08:24.004 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3811510364 2023-11-29T21:52:41.039626+0000 2023-11-28T22:08:24.004 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2138187339 2023-11-29T21:52:41.039626+0000 2023-11-28T22:08:24.004 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/3252186332 2023-11-29T21:52:41.039626+0000 2023-11-28T22:08:24.004 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/624508208 2023-11-29T21:52:41.039626+0000 2023-11-28T22:08:24.004 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/2094131491 2023-11-29T21:52:41.039626+0000 2023-11-28T22:08:24.004 INFO:teuthology.orchestra.run.smithi045.stderr:listed 16 entries 2023-11-28T22:08:24.005 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa7f006af60 msgr2=0x7fa7f006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa7f006af60 0x7fa7f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8100cf730 msgr2=0x7fa81015dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8100cf730 0x7fa81015dc70 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fa80000c4d0 tx=0x7fa80000c990 comp rx=0 tx=0).stop 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 shutdown_connections 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa81015edc0 0x7fa8100c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa81015b820 0x7fa8101f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7fa7f006af60 0x7fa7f006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 --2- 172.21.15.45:0/1049125907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8100cf730 0x7fa81015dc70 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 >> 172.21.15.45:0/1049125907 conn(0x7fa81006a6e0 msgr2=0x7fa81015aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 shutdown_connections 2023-11-28T22:08:24.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.004+0000 7fa814a17640 1 -- 172.21.15.45:0/1049125907 wait complete. 2023-11-28T22:08:24.023 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T22:08:24.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:24.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T22:08:24.044 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:24.041+0000 7f46a5c46040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:08:24.051 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:24.050+0000 7f4ee7830040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T22:08:24.092 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:24.090+0000 7f46a5c46040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T22:08:24.111 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T22:08:24.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:24.111 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T22:08:24.204 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T22:08:24.204 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:24.213 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:24.208+0000 7f46a5c46040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T22:08:24.220 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:24.219+0000 7f4ee7830040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:08:24.259 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T22:08:24.259 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:08:24.259 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:24.273 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:24.272+0000 7f4ee7830040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T22:08:24.315 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T22:08:24.315 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T22:08:24.316 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:24.369 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T22:08:24.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:08:24.402 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:24.401+0000 7f4ee7830040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T22:08:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.587+0000 7f3b30db5640 1 Processor -- start 2023-11-28T22:08:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.588+0000 7f3b30db5640 1 -- start start 2023-11-28T22:08:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.589+0000 7f3b30db5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b7f60 0x7f3b2c1681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.589+0000 7f3b30db5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0d2230 con 0x7f3b2c0d02d0 2023-11-28T22:08:24.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.589+0000 7f3b30db5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0d2510 con 0x7f3b2c15d550 2023-11-28T22:08:24.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.589+0000 7f3b30db5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c0d27f0 con 0x7f3b2c0b7f60 2023-11-28T22:08:24.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.590+0000 7f3b2affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3b2c0d02d0 0x7f3b2c0d1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37602/0 (socket says 172.21.15.45:37602) 2023-11-28T22:08:24.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.590+0000 7f3b2affd640 1 -- 172.21.15.45:0/754647319 learned_addr learned my addr 172.21.15.45:0/754647319 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:24.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.590+0000 7f3b2bfff640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b7f60 0x7f3b2c1681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:24.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.590+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956152443 0 0) 0x7f3b2c0d2230 con 0x7f3b2c0d02d0 2023-11-28T22:08:24.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.590+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b18003520 con 0x7f3b2c0d02d0 2023-11-28T22:08:24.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.590+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 988741638 0 0) 0x7f3b18003520 con 0x7f3b2c0d02d0 2023-11-28T22:08:24.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b7f60 msgr2=0x7f3b2c1681d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b7f60 0x7f3b2c1681d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 >> v1:172.21.15.190:6789/0 conn(0x7f3b2c15d550 legacy=0x7f3b2c0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0b6a30 con 0x7f3b2c0d02d0 2023-11-28T22:08:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3b14001670 con 0x7f3b2c0d02d0 2023-11-28T22:08:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 >> v1:172.21.15.45:6789/0 conn(0x7f3b2c0d02d0 legacy=0x7f3b2c0d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b7f60 0x7f3b180039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b18004d00 0x7f3b18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c0d27f0 con 0x7f3b18003f70 2023-11-28T22:08:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1800bac0 con 0x7f3b18004d00 2023-11-28T22:08:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1800bc80 con 0x7f3b2c0b7f60 2023-11-28T22:08:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2affd640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2bfff640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b18004d00 0x7f3b18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2b7fe640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b7f60 0x7f3b180039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2affd640 1 -- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b7f60 msgr2=0x7f3b180039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2affd640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c0b7f60 0x7f3b180039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2affd640 1 -- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b18004d00 msgr2=0x7f3b18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:24.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2affd640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b18004d00 0x7f3b18005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.591+0000 7f3b2affd640 1 -- 172.21.15.45:0/754647319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b18003520 con 0x7f3b18003f70 2023-11-28T22:08:24.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.593+0000 7f3b2affd640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b180047c0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400c0e0 tx=0x7f3b180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79df31a4712533b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:24.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.593+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14005ad0 con 0x7f3b18003f70 2023-11-28T22:08:24.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.593+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14005c30 con 0x7f3b18003f70 2023-11-28T22:08:24.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.593+0000 7f3b2a7fc640 1 -- 172.21.15.45:0/754647319 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b140072b0 con 0x7f3b18003f70 2023-11-28T22:08:24.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.593+0000 7f3b30db5640 1 -- 172.21.15.45:0/754647319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 msgr2=0x7f3b180047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:24.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.593+0000 7f3b30db5640 1 --2- 172.21.15.45:0/754647319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b180047c0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400c0e0 tx=0x7f3b180057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.593+0000 7f3b2bfff640 1 -- 172.21.15.45:0/754647319 reap_dead start 2023-11-28T22:08:24.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.594+0000 7f3b30db5640 1 -- 172.21.15.45:0/754647319 shutdown_connections 2023-11-28T22:08:24.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.594+0000 7f3b30db5640 1 -- 172.21.15.45:0/754647319 >> 172.21.15.45:0/754647319 conn(0x7f3b2c06ac70 msgr2=0x7f3b2c0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:24.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.594+0000 7f3b30db5640 1 -- 172.21.15.45:0/754647319 shutdown_connections 2023-11-28T22:08:24.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.594+0000 7f3b30db5640 1 -- 172.21.15.45:0/754647319 wait complete. 2023-11-28T22:08:24.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.595+0000 7f3b30db5640 1 Processor -- start 2023-11-28T22:08:24.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.596+0000 7f3b30db5640 1 -- start start 2023-11-28T22:08:24.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.596+0000 7f3b30db5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b2c156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:24.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.596+0000 7f3b30db5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c1574c0 0x7f3b2c153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b30db5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c154120 0x7f3b2c1545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b30db5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c157c30 con 0x7f3b18003f70 2023-11-28T22:08:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b30db5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c157f10 con 0x7f3b2c1574c0 2023-11-28T22:08:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b30db5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c1581f0 con 0x7f3b2c154120 2023-11-28T22:08:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b2c156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b2c156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49698/0 (socket says 172.21.15.45:49698) 2023-11-28T22:08:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2b7fe640 1 -- 172.21.15.45:0/703034103 learned_addr learned my addr 172.21.15.45:0/703034103 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:24.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2bfff640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c154120 0x7f3b2c1545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:24.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2b7fe640 1 -- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c154120 msgr2=0x7f3b2c1545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:24.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2affd640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c1574c0 0x7f3b2c153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:24.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2b7fe640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c154120 0x7f3b2c1545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2b7fe640 1 -- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c1574c0 msgr2=0x7f3b2c153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:24.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2b7fe640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c1574c0 0x7f3b2c153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2b7fe640 1 -- 172.21.15.45:0/703034103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c1584d0 con 0x7f3b18003f70 2023-11-28T22:08:24.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.597+0000 7f3b2bfff640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c154120 0x7f3b2c1545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:24.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.598+0000 7f3b2b7fe640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b2c156e50 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00e8c0 tx=0x7f3b1c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:24.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.598+0000 7f3b23fff640 1 -- 172.21.15.45:0/703034103 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c003b80 con 0x7f3b18003f70 2023-11-28T22:08:24.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.598+0000 7f3b23fff640 1 -- 172.21.15.45:0/703034103 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c003ce0 con 0x7f3b18003f70 2023-11-28T22:08:24.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.598+0000 7f3b23fff640 1 -- 172.21.15.45:0/703034103 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0052f0 con 0x7f3b18003f70 2023-11-28T22:08:24.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.598+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c1587b0 con 0x7f3b18003f70 2023-11-28T22:08:24.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.598+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c153760 con 0x7f3b18003f70 2023-11-28T22:08:24.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.599+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c0c6b50 con 0x7f3b18003f70 2023-11-28T22:08:24.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.600+0000 7f3b23fff640 1 -- 172.21.15.45:0/703034103 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 113) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f3b1c005450 con 0x7f3b18003f70 2023-11-28T22:08:24.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.601+0000 7f3b23fff640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3b0c06af90 0x7f3b0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:24.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.601+0000 7f3b23fff640 1 -- 172.21.15.45:0/703034103 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f3b1c089e10 con 0x7f3b18003f70 2023-11-28T22:08:24.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.601+0000 7f3b2affd640 1 -- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3b0c06af90 msgr2=0x7f3b0c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:24.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.601+0000 7f3b2affd640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3b0c06af90 0x7f3b0c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:24.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.606+0000 7f3b23fff640 1 -- 172.21.15.45:0/703034103 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b1c08d030 con 0x7f3b18003f70 2023-11-28T22:08:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.802+0000 7f3b2affd640 1 -- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3b0c06af90 msgr2=0x7f3b0c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:24.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.802+0000 7f3b2affd640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3b0c06af90 0x7f3b0c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:08:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.841+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b2c0cd530 con 0x7f3b18003f70 2023-11-28T22:08:24.843 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.842+0000 7f3b23fff640 1 -- 172.21.15.45:0/703034103 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f3b1c0548b0 con 0x7f3b18003f70 2023-11-28T22:08:24.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:24.844 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3312844962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3312844962},{"type":"v1","addr":"172.21.15.190:6835","nonce":3312844962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":8296,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1302790439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1302790439},{"type":"v1","addr":"172.21.15.190:6837","nonce":1302790439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10104,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2082498185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2082498185},{"type":"v1","addr":"172.21.15.45:6834","nonce":2082498185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:52:25.939261+0000","modified":"2023-11-28T21:52:29.372705+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10110":{"gid":10110,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3063278837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3063278837},{"type":"v1","addr":"172.21.15.45:6835","nonce":3063278837}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-28T22:08:24.844 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 92 2023-11-28T22:08:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3b0c06af90 msgr2=0x7f3b0c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3b0c06af90 0x7f3b0c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 msgr2=0x7f3b2c156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b2c156e50 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00e8c0 tx=0x7f3b1c00ed80 comp rx=0 tx=0).stop 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 shutdown_connections 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3b2c154120 0x7f3b2c1545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b2c1574c0 0x7f3b2c153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f3b0c06af90 0x7f3b0c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 --2- 172.21.15.45:0/703034103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b18003f70 0x7f3b2c156e50 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 >> 172.21.15.45:0/703034103 conn(0x7f3b2c06ac70 msgr2=0x7f3b2c15ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 shutdown_connections 2023-11-28T22:08:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:24.845+0000 7f3b30db5640 1 -- 172.21.15.45:0/703034103 wait complete. 2023-11-28T22:08:24.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:08:25.061 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.059+0000 7f46a5c46040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T22:08:25.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.111+0000 7f1eb2217640 1 Processor -- start 2023-11-28T22:08:25.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.111+0000 7f1eb2217640 1 -- start start 2023-11-28T22:08:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.113+0000 7f1eb2217640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1eac0b1340 0x7f1eac15f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.113+0000 7f1eb2217640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eac15bda0 con 0x7f1eac15b850 2023-11-28T22:08:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.113+0000 7f1eb2217640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eac15c080 con 0x7f1eac0b35b0 2023-11-28T22:08:25.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.113+0000 7f1eb2217640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac15ee40 con 0x7f1eac0b1340 2023-11-28T22:08:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.113+0000 7f1eabfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1eac0b1340 0x7f1eac15f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.114+0000 7f1eabfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1eac0b1340 0x7f1eac15f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49710/0 (socket says 172.21.15.45:49710) 2023-11-28T22:08:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.114+0000 7f1eabfff640 1 -- 172.21.15.45:0/1978846087 learned_addr learned my addr 172.21.15.45:0/1978846087 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.114+0000 7f1ea3fff640 1 --1- 172.21.15.45:0/1978846087 >> v1:172.21.15.45:6789/0 conn(0x7f1eac15b850 0x7f1eac0b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37616/0 (socket says 172.21.15.45:37616) 2023-11-28T22:08:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.114+0000 7f1eabfff640 1 -- 172.21.15.45:0/1978846087 >> v1:172.21.15.190:6789/0 conn(0x7f1eac0b35b0 legacy=0x7f1eac15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.114+0000 7f1eabfff640 1 -- 172.21.15.45:0/1978846087 >> v1:172.21.15.45:6789/0 conn(0x7f1eac15b850 legacy=0x7f1eac0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.114+0000 7f1eabfff640 1 -- 172.21.15.45:0/1978846087 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac15c360 con 0x7f1eac0b1340 2023-11-28T22:08:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.115+0000 7f1eabfff640 1 --2- 172.21.15.45:0/1978846087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac0b1340 0x7f1eac15f290 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009e50 tx=0x7f1ea402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb0f956fac0494a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.115+0000 7f1eaaffd640 1 -- 172.21.15.45:0/1978846087 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea402cb20 con 0x7f1eac0b1340 2023-11-28T22:08:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.115+0000 7f1eaaffd640 1 -- 172.21.15.45:0/1978846087 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea402cc80 con 0x7f1eac0b1340 2023-11-28T22:08:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.115+0000 7f1eaaffd640 1 -- 172.21.15.45:0/1978846087 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea40343f0 con 0x7f1eac0b1340 2023-11-28T22:08:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.116+0000 7f1eb2217640 1 -- 172.21.15.45:0/1978846087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac0b1340 msgr2=0x7f1eac15f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.116+0000 7f1eb2217640 1 --2- 172.21.15.45:0/1978846087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac0b1340 0x7f1eac15f290 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009e50 tx=0x7f1ea402bf80 comp rx=0 tx=0).stop 2023-11-28T22:08:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.116+0000 7f1eb2217640 1 -- 172.21.15.45:0/1978846087 shutdown_connections 2023-11-28T22:08:25.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.116+0000 7f1eb2217640 1 --2- 172.21.15.45:0/1978846087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac0b1340 0x7f1eac15f290 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.116+0000 7f1eb2217640 1 -- 172.21.15.45:0/1978846087 >> 172.21.15.45:0/1978846087 conn(0x7f1eac06a500 msgr2=0x7f1eac06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:25.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.116+0000 7f1eb2217640 1 -- 172.21.15.45:0/1978846087 shutdown_connections 2023-11-28T22:08:25.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.116+0000 7f1eb2217640 1 -- 172.21.15.45:0/1978846087 wait complete. 2023-11-28T22:08:25.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.118+0000 7f1eb2217640 1 Processor -- start 2023-11-28T22:08:25.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.118+0000 7f1eb2217640 1 -- start start 2023-11-28T22:08:25.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.119+0000 7f1eb2217640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1eac0b1340 0x7f1eac13ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.119+0000 7f1eb2217640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eac0b35b0 0x7f1eac15a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.119+0000 7f1eb2217640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac15b850 0x7f1eac15a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.119+0000 7f1eb2217640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac071460 con 0x7f1eac15b850 2023-11-28T22:08:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.119+0000 7f1eb2217640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac071740 con 0x7f1eac0b35b0 2023-11-28T22:08:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.119+0000 7f1eb2217640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac071a20 con 0x7f1eac0b1340 2023-11-28T22:08:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.119+0000 7f1eabfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac15b850 0x7f1eac15a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:25.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.119+0000 7f1eabfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac15b850 0x7f1eac15a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49720/0 (socket says 172.21.15.45:49720) 2023-11-28T22:08:25.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.119+0000 7f1eabfff640 1 -- 172.21.15.45:0/3357679259 learned_addr learned my addr 172.21.15.45:0/3357679259 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1eab7fe640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1eac0b1340 0x7f1eac13ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1eabfff640 1 -- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1eac0b1340 msgr2=0x7f1eac13ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1eabfff640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1eac0b1340 0x7f1eac13ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1ea3fff640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eac0b35b0 0x7f1eac15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1eabfff640 1 -- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eac0b35b0 msgr2=0x7f1eac15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1eabfff640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eac0b35b0 0x7f1eac15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1eabfff640 1 -- 172.21.15.45:0/3357679259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4009b30 con 0x7f1eac15b850 2023-11-28T22:08:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1ea3fff640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eac0b35b0 0x7f1eac15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1eabfff640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac15b850 0x7f1eac15a890 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4036040 tx=0x7f1ea4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:25.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1ea8ff9640 1 -- 172.21.15.45:0/3357679259 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4009510 con 0x7f1eac15b850 2023-11-28T22:08:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1ea8ff9640 1 -- 172.21.15.45:0/3357679259 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea4007e50 con 0x7f1eac15b850 2023-11-28T22:08:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1ea8ff9640 1 -- 172.21.15.45:0/3357679259 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea403e690 con 0x7f1eac15b850 2023-11-28T22:08:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.120+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac071d00 con 0x7f1eac15b850 2023-11-28T22:08:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.121+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac13b190 con 0x7f1eac15b850 2023-11-28T22:08:25.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.122+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eac072220 con 0x7f1eac15b850 2023-11-28T22:08:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.123+0000 7f1ea8ff9640 1 -- 172.21.15.45:0/3357679259 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 113) v1 ==== 89345+0+0 (secure 0 0 0) 0x7f1ea403e7f0 con 0x7f1eac15b850 2023-11-28T22:08:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.124+0000 7f1ea8ff9640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f1e8c06af60 0x7f1e8c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.124+0000 7f1ea8ff9640 1 -- 172.21.15.45:0/3357679259 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f1ea40ac920 con 0x7f1eac15b850 2023-11-28T22:08:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.124+0000 7f1eab7fe640 1 -- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f1e8c06af60 msgr2=0x7f1e8c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:25.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.124+0000 7f1eab7fe640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f1e8c06af60 0x7f1e8c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:25.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.132+0000 7f1ea8ff9640 1 -- 172.21.15.45:0/3357679259 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1eac072220 con 0x7f1eac15b850 2023-11-28T22:08:25.274 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:25.273+0000 7f4ee7830040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T22:08:25.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.325+0000 7f1eab7fe640 1 -- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f1e8c06af60 msgr2=0x7f1e8c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/697528389 2023-11-28T22:08:25.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.325+0000 7f1eab7fe640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f1e8c06af60 0x7f1e8c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:08:25.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.328+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1eac072220 con 0x7f1eac15b850 2023-11-28T22:08:25.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.331+0000 7f1ea8ff9640 1 -- 172.21.15.45:0/3357679259 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v203) v1 ==== 74+0+17905 (secure 0 0 0) 0x7f1eac072220 con 0x7f1eac15b850 2023-11-28T22:08:25.332 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:25.333 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":203,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:08:21.350175+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:52:23.878130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-11-28T21:52:24.956954+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":203,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3880574704":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6833/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6832/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3938773910":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2706131286":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3522054455":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2512261826":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3252656094":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2538549380":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3848257557":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6833/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3811510364":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2138187339":"2023-11-29T21:52:41.039626+0000","172.21.15.190:6832/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/624508208":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2094131491":"2023-11-29T21:52:41.039626+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:08:25.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f1e8c06af60 msgr2=0x7f1e8c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f1e8c06af60 0x7f1e8c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac15b850 msgr2=0x7f1eac15a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac15b850 0x7f1eac15a890 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4036040 tx=0x7f1ea4004a40 comp rx=0 tx=0).stop 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 shutdown_connections 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:6832/697528389,v1:172.21.15.190:6833/697528389] conn(0x7f1e8c06af60 0x7f1e8c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1eac15b850 0x7f1eac15a890 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eac0b35b0 0x7f1eac15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 --2- 172.21.15.45:0/3357679259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1eac0b1340 0x7f1eac13ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 >> 172.21.15.45:0/3357679259 conn(0x7f1eac06a500 msgr2=0x7f1eac06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 shutdown_connections 2023-11-28T22:08:25.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.334+0000 7f1eb2217640 1 -- 172.21.15.45:0/3357679259 wait complete. 2023-11-28T22:08:25.354 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T21:52:23.878130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '198', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T21:52:24.956954+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '203', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 203, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T22:08:25.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T22:08:25.356 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.354+0000 7f46a5c46040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T22:08:25.414 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.412+0000 7f46a5c46040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T22:08:25.530 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.527+0000 7f46a5c46040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T22:08:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.581+0000 7fb0c063d640 1 Processor -- start 2023-11-28T22:08:25.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.581+0000 7fb0c063d640 1 -- start start 2023-11-28T22:08:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0c063d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb0b8143f50 0x7fb0b815f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0c063d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb0b80b3280 0x7fb0b80c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0c063d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b80c7e60 con 0x7fb0b806f280 2023-11-28T22:08:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0c063d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b8144330 con 0x7fb0b80b3280 2023-11-28T22:08:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0c063d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b815ed30 con 0x7fb0b8143f50 2023-11-28T22:08:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0be3d9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb0b8143f50 0x7fb0b815f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0bebda640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb0b80b3280 0x7fb0b80c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0be3d9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb0b8143f50 0x7fb0b815f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:54898/0 (socket says 172.21.15.45:54898) 2023-11-28T22:08:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0be3d9640 1 -- 172.21.15.45:0/2910269731 learned_addr learned my addr 172.21.15.45:0/2910269731 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0be3d9640 1 -- 172.21.15.45:0/2910269731 >> v2:172.21.15.45:3300/0 conn(0x7fb0b80b3280 msgr2=0x7fb0b80c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.584+0000 7fb0be3d9640 1 --2- 172.21.15.45:0/2910269731 >> v2:172.21.15.45:3300/0 conn(0x7fb0b80b3280 0x7fb0b80c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.585+0000 7fb0be3d9640 1 -- 172.21.15.45:0/2910269731 >> v1:172.21.15.45:6789/0 conn(0x7fb0b806f280 legacy=0x7fb0b80b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.585+0000 7fb0be3d9640 1 -- 172.21.15.45:0/2910269731 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b80cef40 con 0x7fb0b8143f50 2023-11-28T22:08:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.585+0000 7fb0bebda640 1 --2- 172.21.15.45:0/2910269731 >> v2:172.21.15.45:3300/0 conn(0x7fb0b80b3280 0x7fb0b80c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T22:08:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.586+0000 7fb0be3d9640 1 --2- 172.21.15.45:0/2910269731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b8143f50 0x7fb0b815f7a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb0b806d410 tx=0x7fb0b402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a405c672de3aaa08 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.587+0000 7fb0bd3d7640 1 -- 172.21.15.45:0/2910269731 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4030860 con 0x7fb0b8143f50 2023-11-28T22:08:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.587+0000 7fb0bd3d7640 1 -- 172.21.15.45:0/2910269731 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b40309c0 con 0x7fb0b8143f50 2023-11-28T22:08:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.587+0000 7fb0bd3d7640 1 -- 172.21.15.45:0/2910269731 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4030d50 con 0x7fb0b8143f50 2023-11-28T22:08:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.587+0000 7fb0c063d640 1 -- 172.21.15.45:0/2910269731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b8143f50 msgr2=0x7fb0b815f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.587+0000 7fb0c063d640 1 --2- 172.21.15.45:0/2910269731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b8143f50 0x7fb0b815f7a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb0b806d410 tx=0x7fb0b402b9a0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.588+0000 7fb0c063d640 1 -- 172.21.15.45:0/2910269731 shutdown_connections 2023-11-28T22:08:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.588+0000 7fb0c063d640 1 --2- 172.21.15.45:0/2910269731 >> v2:172.21.15.45:3300/0 conn(0x7fb0b80b3280 0x7fb0b80c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.588+0000 7fb0c063d640 1 --2- 172.21.15.45:0/2910269731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b8143f50 0x7fb0b815f7a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.588+0000 7fb0c063d640 1 -- 172.21.15.45:0/2910269731 >> 172.21.15.45:0/2910269731 conn(0x7fb0b8066820 msgr2=0x7fb0b806c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.588+0000 7fb0c063d640 1 -- 172.21.15.45:0/2910269731 shutdown_connections 2023-11-28T22:08:25.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.588+0000 7fb0c063d640 1 -- 172.21.15.45:0/2910269731 wait complete. 2023-11-28T22:08:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.590+0000 7fb0c063d640 1 Processor -- start 2023-11-28T22:08:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.591+0000 7fb0c063d640 1 -- start start 2023-11-28T22:08:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.592+0000 7fb0c063d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b806f280 0x7fb0b813ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.592+0000 7fb0c063d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b80b3280 0x7fb0b81430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.592+0000 7fb0c063d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b8143f50 0x7fb0b80b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.592+0000 7fb0c063d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b80b5600 con 0x7fb0b806f280 2023-11-28T22:08:25.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.592+0000 7fb0c063d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b80b58e0 con 0x7fb0b8143f50 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.592+0000 7fb0c063d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b80b5bc0 con 0x7fb0b80b3280 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0be3d9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b806f280 0x7fb0b813ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0be3d9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b806f280 0x7fb0b813ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49752/0 (socket says 172.21.15.45:49752) 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0be3d9640 1 -- 172.21.15.45:0/452377017 learned_addr learned my addr 172.21.15.45:0/452377017 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0bebda640 1 --2- 172.21.15.45:0/452377017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b8143f50 0x7fb0b80b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0bdbd8640 1 --2- 172.21.15.45:0/452377017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b80b3280 0x7fb0b81430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0be3d9640 1 -- 172.21.15.45:0/452377017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b80b3280 msgr2=0x7fb0b81430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0be3d9640 1 --2- 172.21.15.45:0/452377017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b80b3280 0x7fb0b81430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0be3d9640 1 -- 172.21.15.45:0/452377017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b8143f50 msgr2=0x7fb0b80b66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0be3d9640 1 --2- 172.21.15.45:0/452377017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b8143f50 0x7fb0b80b66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0be3d9640 1 -- 172.21.15.45:0/452377017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0009a70 con 0x7fb0b806f280 2023-11-28T22:08:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.593+0000 7fb0be3d9640 1 --2- 172.21.15.45:0/452377017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b806f280 0x7fb0b813ee70 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fb0b402b9e0 tx=0x7fb0b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.594+0000 7fb0aeffd640 1 -- 172.21.15.45:0/452377017 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4039a90 con 0x7fb0b806f280 2023-11-28T22:08:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.594+0000 7fb0aeffd640 1 -- 172.21.15.45:0/452377017 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b4039c30 con 0x7fb0b806f280 2023-11-28T22:08:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.594+0000 7fb0c063d640 1 -- 172.21.15.45:0/452377017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b40094d0 con 0x7fb0b806f280 2023-11-28T22:08:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.594+0000 7fb0aeffd640 1 -- 172.21.15.45:0/452377017 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4043410 con 0x7fb0b806f280 2023-11-28T22:08:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.594+0000 7fb0c063d640 1 -- 172.21.15.45:0/452377017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b80b6180 con 0x7fb0b806f280 2023-11-28T22:08:25.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.595+0000 7fb0c063d640 1 -- 172.21.15.45:0/452377017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b80b6370 con 0x7fb0b806f280 2023-11-28T22:08:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.597+0000 7fb0aeffd640 1 -- 172.21.15.45:0/452377017 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 114) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fb0b4043570 con 0x7fb0b806f280 2023-11-28T22:08:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.598+0000 7fb0aeffd640 1 -- 172.21.15.45:0/452377017 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7117+0+0 (secure 0 0 0) 0x7fb0b40ab870 con 0x7fb0b806f280 2023-11-28T22:08:25.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.602+0000 7fb0aeffd640 1 -- 172.21.15.45:0/452377017 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0b40768d0 con 0x7fb0b806f280 2023-11-28T22:08:25.611 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:25.610+0000 7f4ee7830040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T22:08:25.627 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.625+0000 7f462aeba640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.627 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.625+0000 7f462aeba640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.627 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.625+0000 7f462aeba640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.627 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.625+0000 7f462aeba640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.627 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.625+0000 7f462aeba640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.628 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.626+0000 7f4627eb4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.628 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.626+0000 7f4627eb4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.628 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.626+0000 7f4627eb4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.628 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.626+0000 7f4627eb4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.628 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:25.626+0000 7f4627eb4640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:25.676 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:25.674+0000 7f4ee7830040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T22:08:25.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.791+0000 7fb0c063d640 1 -- 172.21.15.45:0/452377017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0b80733d0 con 0x7fb0b806f280 2023-11-28T22:08:25.794 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.792+0000 7fb0aeffd640 1 -- 172.21.15.45:0/452377017 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v204) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb0b40480a0 con 0x7fb0b806f280 2023-11-28T22:08:25.794 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T22:08:25.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.795+0000 7fb0c063d640 1 -- 172.21.15.45:0/452377017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b806f280 msgr2=0x7fb0b813ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:25.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.795+0000 7fb0c063d640 1 --2- 172.21.15.45:0/452377017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b806f280 0x7fb0b813ee70 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fb0b402b9e0 tx=0x7fb0b4004730 comp rx=0 tx=0).stop 2023-11-28T22:08:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.795+0000 7fb0c063d640 1 -- 172.21.15.45:0/452377017 shutdown_connections 2023-11-28T22:08:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.795+0000 7fb0c063d640 1 --2- 172.21.15.45:0/452377017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b8143f50 0x7fb0b80b66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.795+0000 7fb0c063d640 1 --2- 172.21.15.45:0/452377017 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b80b3280 0x7fb0b81430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.795+0000 7fb0c063d640 1 --2- 172.21.15.45:0/452377017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b806f280 0x7fb0b813ee70 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.795+0000 7fb0c063d640 1 -- 172.21.15.45:0/452377017 >> 172.21.15.45:0/452377017 conn(0x7fb0b8066820 msgr2=0x7fb0b806c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.795+0000 7fb0c063d640 1 -- 172.21.15.45:0/452377017 shutdown_connections 2023-11-28T22:08:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:25.795+0000 7fb0c063d640 1 -- 172.21.15.45:0/452377017 wait complete. 2023-11-28T22:08:25.798 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:25.797+0000 7f4ee7830040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T22:08:25.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T22:08:26.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.018+0000 7fb0b62b4640 1 Processor -- start 2023-11-28T22:08:26.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.018+0000 7fb0b62b4640 1 -- start start 2023-11-28T22:08:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.019+0000 7fb0b62b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 0x7fb0b00cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.019+0000 7fb0b62b4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb0b00ce600 0x7fb0b006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.019+0000 7fb0b62b4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb0b006aa40 0x7fb0b00c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.019+0000 7fb0b62b4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b006aed0 con 0x7fb0b00ce600 2023-11-28T22:08:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.019+0000 7fb0b62b4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b006e030 con 0x7fb0b006aa40 2023-11-28T22:08:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.019+0000 7fb0b62b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00ccf60 con 0x7fb0b006dc50 2023-11-28T22:08:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0af7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb0b00ce600 0x7fb0b006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0af7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb0b00ce600 0x7fb0b006a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49766/0 (socket says 172.21.15.45:49766) 2023-11-28T22:08:26.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0af7fe640 1 -- 172.21.15.45:0/3924325275 learned_addr learned my addr 172.21.15.45:0/3924325275 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0affff640 1 --2- 172.21.15.45:0/3924325275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 0x7fb0b00cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0b4851640 1 --2- 172.21.15.45:0/3924325275 >> v2:172.21.15.190:3300/0 conn(0x7fb0b006aa40 0x7fb0b00c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0af7fe640 1 -- 172.21.15.45:0/3924325275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 msgr2=0x7fb0b00cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0af7fe640 1 --2- 172.21.15.45:0/3924325275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 0x7fb0b00cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0af7fe640 1 -- 172.21.15.45:0/3924325275 >> v2:172.21.15.190:3300/0 conn(0x7fb0b006aa40 msgr2=0x7fb0b00c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0af7fe640 1 --2- 172.21.15.45:0/3924325275 >> v2:172.21.15.190:3300/0 conn(0x7fb0b006aa40 0x7fb0b00c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.020+0000 7fb0af7fe640 1 -- 172.21.15.45:0/3924325275 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b00cd930 con 0x7fb0b00ce600 2023-11-28T22:08:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.022+0000 7fb0af7fe640 1 --2- 172.21.15.45:0/3924325275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b00ce600 0x7fb0b006a500 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00c2860 tx=0x7fb0a402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4015d9cbb4f2c37 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.022+0000 7fb0aeffd640 1 -- 172.21.15.45:0/3924325275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4031860 con 0x7fb0b00ce600 2023-11-28T22:08:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.022+0000 7fb0aeffd640 1 -- 172.21.15.45:0/3924325275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a40319c0 con 0x7fb0b00ce600 2023-11-28T22:08:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.022+0000 7fb0aeffd640 1 -- 172.21.15.45:0/3924325275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4031d50 con 0x7fb0b00ce600 2023-11-28T22:08:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.023+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3924325275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b00ce600 msgr2=0x7fb0b006a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.023+0000 7fb0b62b4640 1 --2- 172.21.15.45:0/3924325275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b00ce600 0x7fb0b006a500 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00c2860 tx=0x7fb0a402b900 comp rx=0 tx=0).stop 2023-11-28T22:08:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.023+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3924325275 shutdown_connections 2023-11-28T22:08:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.023+0000 7fb0b62b4640 1 --2- 172.21.15.45:0/3924325275 >> v2:172.21.15.190:3300/0 conn(0x7fb0b006aa40 0x7fb0b00c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.023+0000 7fb0b62b4640 1 --2- 172.21.15.45:0/3924325275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b00ce600 0x7fb0b006a500 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.023+0000 7fb0b62b4640 1 --2- 172.21.15.45:0/3924325275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 0x7fb0b00cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.023+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3924325275 >> 172.21.15.45:0/3924325275 conn(0x7fb0b00ced60 msgr2=0x7fb0b00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:26.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.023+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3924325275 shutdown_connections 2023-11-28T22:08:26.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.023+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3924325275 wait complete. 2023-11-28T22:08:26.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.025+0000 7fb0b62b4640 1 Processor -- start 2023-11-28T22:08:26.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.025+0000 7fb0b62b4640 1 -- start start 2023-11-28T22:08:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.026+0000 7fb0b62b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b006aa40 0x7fb0b01f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.026+0000 7fb0b62b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 0x7fb0b00b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.026+0000 7fb0b62b4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b00ce600 0x7fb0b00b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.026+0000 7fb0b62b4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b0099c90 con 0x7fb0b006aa40 2023-11-28T22:08:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.026+0000 7fb0b62b4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00c5e30 con 0x7fb0b00ce600 2023-11-28T22:08:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.026+0000 7fb0b62b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b01f3ea0 con 0x7fb0b006dc50 2023-11-28T22:08:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0affff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b006aa40 0x7fb0b01f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0affff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b006aa40 0x7fb0b01f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49776/0 (socket says 172.21.15.45:49776) 2023-11-28T22:08:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0affff640 1 -- 172.21.15.45:0/3323599051 learned_addr learned my addr 172.21.15.45:0/3323599051 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0af7fe640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 0x7fb0b00b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0b4851640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b00ce600 0x7fb0b00b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0affff640 1 -- 172.21.15.45:0/3323599051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 msgr2=0x7fb0b00b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0affff640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 0x7fb0b00b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0affff640 1 -- 172.21.15.45:0/3323599051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b00ce600 msgr2=0x7fb0b00b37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0affff640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b00ce600 0x7fb0b00b37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0affff640 1 -- 172.21.15.45:0/3323599051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a40094d0 con 0x7fb0b006aa40 2023-11-28T22:08:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.027+0000 7fb0af7fe640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 0x7fb0b00b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:08:26.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.028+0000 7fb0affff640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b006aa40 0x7fb0b01f5f50 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000e9b0 tx=0x7fb0a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:26.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.028+0000 7fb0acff9640 1 -- 172.21.15.45:0/3323599051 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0009670 con 0x7fb0b006aa40 2023-11-28T22:08:26.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.028+0000 7fb0acff9640 1 -- 172.21.15.45:0/3323599051 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a000c9a0 con 0x7fb0b006aa40 2023-11-28T22:08:26.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.028+0000 7fb0acff9640 1 -- 172.21.15.45:0/3323599051 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a000cd50 con 0x7fb0b006aa40 2023-11-28T22:08:26.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.028+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3323599051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b01f4180 con 0x7fb0b006aa40 2023-11-28T22:08:26.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.028+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3323599051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b00c6360 con 0x7fb0b006aa40 2023-11-28T22:08:26.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.030+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3323599051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b01f4370 con 0x7fb0b006aa40 2023-11-28T22:08:26.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.030+0000 7fb0acff9640 1 -- 172.21.15.45:0/3323599051 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 114) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fb0a0013470 con 0x7fb0b006aa40 2023-11-28T22:08:26.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.031+0000 7fb0acff9640 1 -- 172.21.15.45:0/3323599051 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7117+0+0 (secure 0 0 0) 0x7fb0a0089820 con 0x7fb0b006aa40 2023-11-28T22:08:26.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.036+0000 7fb0acff9640 1 -- 172.21.15.45:0/3323599051 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0a00547a0 con 0x7fb0b006aa40 2023-11-28T22:08:26.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.227+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3323599051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0b00733d0 con 0x7fb0b006aa40 2023-11-28T22:08:26.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.228+0000 7fb0acff9640 1 -- 172.21.15.45:0/3323599051 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v204) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0a0057dc0 con 0x7fb0b006aa40 2023-11-28T22:08:26.229 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T22:08:26.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.230+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3323599051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b006aa40 msgr2=0x7fb0b01f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.230+0000 7fb0b62b4640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b006aa40 0x7fb0b01f5f50 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000e9b0 tx=0x7fb0a000ee70 comp rx=0 tx=0).stop 2023-11-28T22:08:26.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.231+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3323599051 shutdown_connections 2023-11-28T22:08:26.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.231+0000 7fb0b62b4640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0b00ce600 0x7fb0b00b37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.231+0000 7fb0b62b4640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0b006dc50 0x7fb0b00b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.231+0000 7fb0b62b4640 1 --2- 172.21.15.45:0/3323599051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0b006aa40 0x7fb0b01f5f50 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.231+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3323599051 >> 172.21.15.45:0/3323599051 conn(0x7fb0b00ced60 msgr2=0x7fb0b00b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:26.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.231+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3323599051 shutdown_connections 2023-11-28T22:08:26.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.231+0000 7fb0b62b4640 1 -- 172.21.15.45:0/3323599051 wait complete. 2023-11-28T22:08:26.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T22:08:26.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.483+0000 7f4a63aec640 1 Processor -- start 2023-11-28T22:08:26.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.484+0000 7f4a63aec640 1 -- start start 2023-11-28T22:08:26.491 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.485+0000 7f4a63aec640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a5c06c350 0x7f4a5c15f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.485+0000 7f4a63aec640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c15ecd0 con 0x7f4a5c15e840 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.486+0000 7f4a63aec640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c15f290 con 0x7f4a5c06ac10 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.486+0000 7f4a63aec640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c072020 con 0x7f4a5c06c350 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.486+0000 7f4a61888640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4a5c15e840 0x7f4a5c06a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37638/0 (socket says 172.21.15.45:37638) 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.486+0000 7f4a61888640 1 -- 172.21.15.45:0/875691523 learned_addr learned my addr 172.21.15.45:0/875691523 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.486+0000 7f4a62089640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a5c06c350 0x7f4a5c15f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.486+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522586913 0 0) 0x7f4a5c15ecd0 con 0x7f4a5c15e840 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.486+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a44003520 con 0x7f4a5c15e840 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 389214002 0 0) 0x7f4a44003520 con 0x7f4a5c15e840 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a5c06c350 msgr2=0x7f4a5c15f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:26.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a5c06c350 0x7f4a5c15f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 >> v1:172.21.15.190:6789/0 conn(0x7f4a5c06ac10 legacy=0x7f4a5c0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c072300 con 0x7f4a5c15e840 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4a4c0019f0 con 0x7f4a5c15e840 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 >> v1:172.21.15.45:6789/0 conn(0x7f4a5c15e840 legacy=0x7f4a5c06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a440039a0 0x7f4a44003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a440047c0 0x7f4a44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a5c06c350 0x7f4a44005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c072020 con 0x7f4a440047c0 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4400ba90 con 0x7f4a440039a0 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4400bc50 con 0x7f4a5c06c350 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.487+0000 7f4a61087640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a440047c0 0x7f4a44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.488+0000 7f4a62089640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a5c06c350 0x7f4a44005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.488+0000 7f4a61087640 1 -- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a5c06c350 msgr2=0x7f4a44005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:26.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.488+0000 7f4a61087640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a5c06c350 0x7f4a44005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.488+0000 7f4a61087640 1 -- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a440039a0 msgr2=0x7f4a44003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:26.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.488+0000 7f4a61087640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a440039a0 0x7f4a44003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.488+0000 7f4a61087640 1 -- 172.21.15.45:0/875691523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a44003520 con 0x7f4a440047c0 2023-11-28T22:08:26.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.489+0000 7f4a61087640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a440047c0 0x7f4a44004c40 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f4a50001720 tx=0x7f4a440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef946fa0362e0e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:26.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.489+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a500052d0 con 0x7f4a440047c0 2023-11-28T22:08:26.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.489+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5002e050 con 0x7f4a440047c0 2023-11-28T22:08:26.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.489+0000 7f4a60886640 1 -- 172.21.15.45:0/875691523 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50003830 con 0x7f4a440047c0 2023-11-28T22:08:26.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.490+0000 7f4a63aec640 1 -- 172.21.15.45:0/875691523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a440047c0 msgr2=0x7f4a44004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.490+0000 7f4a63aec640 1 --2- 172.21.15.45:0/875691523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a440047c0 0x7f4a44004c40 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f4a50001720 tx=0x7f4a440057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.490+0000 7f4a62089640 1 -- 172.21.15.45:0/875691523 reap_dead start 2023-11-28T22:08:26.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.490+0000 7f4a63aec640 1 -- 172.21.15.45:0/875691523 shutdown_connections 2023-11-28T22:08:26.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.490+0000 7f4a63aec640 1 -- 172.21.15.45:0/875691523 >> 172.21.15.45:0/875691523 conn(0x7f4a5c0ced60 msgr2=0x7f4a5c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:26.497 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.490+0000 7f4a63aec640 1 -- 172.21.15.45:0/875691523 shutdown_connections 2023-11-28T22:08:26.498 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.490+0000 7f4a63aec640 1 -- 172.21.15.45:0/875691523 wait complete. 2023-11-28T22:08:26.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.492+0000 7f4a63aec640 1 Processor -- start 2023-11-28T22:08:26.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.492+0000 7f4a63aec640 1 -- start start 2023-11-28T22:08:26.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.493+0000 7f4a63aec640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a440047c0 0x7f4a5c1555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.493+0000 7f4a63aec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a5c155c30 0x7f4a5c0b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.493+0000 7f4a63aec640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0c3660 0x7f4a5c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.493+0000 7f4a63aec640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c1564d0 con 0x7f4a5c155c30 2023-11-28T22:08:26.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.493+0000 7f4a63aec640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c0b2560 con 0x7f4a5c0c3660 2023-11-28T22:08:26.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.493+0000 7f4a63aec640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c0b2820 con 0x7f4a440047c0 2023-11-28T22:08:26.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.493+0000 7f4a61087640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a5c155c30 0x7f4a5c0b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.493+0000 7f4a61087640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a5c155c30 0x7f4a5c0b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49782/0 (socket says 172.21.15.45:49782) 2023-11-28T22:08:26.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.493+0000 7f4a61087640 1 -- 172.21.15.45:0/3868476712 learned_addr learned my addr 172.21.15.45:0/3868476712 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:26.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a62089640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0c3660 0x7f4a5c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a61087640 1 -- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a440047c0 msgr2=0x7f4a5c1555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a61087640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a440047c0 0x7f4a5c1555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a61087640 1 -- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0c3660 msgr2=0x7f4a5c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a61087640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0c3660 0x7f4a5c0c3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a61087640 1 -- 172.21.15.45:0/3868476712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c0b2b00 con 0x7f4a5c155c30 2023-11-28T22:08:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a61087640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a5c155c30 0x7f4a5c0b4150 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f4a50008fd0 tx=0x7f4a50001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a5a7fc640 1 -- 172.21.15.45:0/3868476712 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50007640 con 0x7f4a5c155c30 2023-11-28T22:08:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a5a7fc640 1 -- 172.21.15.45:0/3868476712 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a500069e0 con 0x7f4a5c155c30 2023-11-28T22:08:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a5a7fc640 1 -- 172.21.15.45:0/3868476712 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50006d90 con 0x7f4a5c155c30 2023-11-28T22:08:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.494+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c1f3830 con 0x7f4a5c155c30 2023-11-28T22:08:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.495+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c1f34e0 con 0x7f4a5c155c30 2023-11-28T22:08:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.496+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c0b2960 con 0x7f4a5c155c30 2023-11-28T22:08:26.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.497+0000 7f4a5a7fc640 1 -- 172.21.15.45:0/3868476712 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 114) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f4a500035f0 con 0x7f4a5c155c30 2023-11-28T22:08:26.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.498+0000 7f4a5a7fc640 1 -- 172.21.15.45:0/3868476712 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7117+0+0 (secure 0 0 0) 0x7f4a500ab370 con 0x7f4a5c155c30 2023-11-28T22:08:26.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.502+0000 7f4a5a7fc640 1 -- 172.21.15.45:0/3868476712 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a5c0b2960 con 0x7f4a5c155c30 2023-11-28T22:08:26.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.559+0000 7f4a5a7fc640 1 -- 172.21.15.45:0/3868476712 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 115) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f4a50075da0 con 0x7f4a5c155c30 2023-11-28T22:08:26.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.559+0000 7f4a5a7fc640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4a48070e50 0x7f4a48073240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.569 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.568+0000 7f4a61888640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4a48070e50 0x7f4a48073240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.573+0000 7f4a61888640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4a48070e50 0x7f4a48073240 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c001e90 tx=0x7f4a4c001880 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:26.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.706+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4a5c0b2960 con 0x7f4a5c155c30 2023-11-28T22:08:26.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.707+0000 7f4a5a7fc640 1 -- 172.21.15.45:0/3868476712 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v204) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4a5c0b2960 con 0x7f4a5c155c30 2023-11-28T22:08:26.709 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T22:08:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4a48070e50 msgr2=0x7f4a48073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4a48070e50 0x7f4a48073240 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c001e90 tx=0x7f4a4c001880 comp rx=0 tx=0).stop 2023-11-28T22:08:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a5c155c30 msgr2=0x7f4a5c0b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a5c155c30 0x7f4a5c0b4150 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f4a50008fd0 tx=0x7f4a50001120 comp rx=0 tx=0).stop 2023-11-28T22:08:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 shutdown_connections 2023-11-28T22:08:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a5c0c3660 0x7f4a5c0c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4a48070e50 0x7f4a48073240 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a5c155c30 0x7f4a5c0b4150 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 --2- 172.21.15.45:0/3868476712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a440047c0 0x7f4a5c1555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 >> 172.21.15.45:0/3868476712 conn(0x7f4a5c0ced60 msgr2=0x7f4a5c0ad2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 shutdown_connections 2023-11-28T22:08:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.710+0000 7f4a63aec640 1 -- 172.21.15.45:0/3868476712 wait complete. 2023-11-28T22:08:26.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:08:26.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.952+0000 7f1a2b83d640 1 Processor -- start 2023-11-28T22:08:26.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.952+0000 7f1a2b83d640 1 -- start start 2023-11-28T22:08:26.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.954+0000 7f1a2b83d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a240b35b0 0x7f1a2415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.954+0000 7f1a2b83d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2415bda0 con 0x7f1a240b1340 2023-11-28T22:08:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.954+0000 7f1a2b83d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2415c080 con 0x7f1a2415b850 2023-11-28T22:08:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.954+0000 7f1a2b83d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2415ee40 con 0x7f1a240b35b0 2023-11-28T22:08:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.954+0000 7f1a29dda640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1a240b1340 0x7f1a2415f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37654/0 (socket says 172.21.15.45:37654) 2023-11-28T22:08:26.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.954+0000 7f1a29dda640 1 -- 172.21.15.45:0/4089532337 learned_addr learned my addr 172.21.15.45:0/4089532337 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a295d9640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a240b35b0 0x7f1a2415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762965958 0 0) 0x7f1a2415bda0 con 0x7f1a240b1340 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a10003520 con 0x7f1a240b1340 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 790932647 0 0) 0x7f1a10003520 con 0x7f1a240b1340 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a240b35b0 msgr2=0x7f1a2415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a240b35b0 0x7f1a2415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 >> v1:172.21.15.190:6789/0 conn(0x7f1a2415b850 legacy=0x7f1a240b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2415c360 con 0x7f1a240b1340 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f1a1c0019f0 con 0x7f1a240b1340 2023-11-28T22:08:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 >> v1:172.21.15.45:6789/0 conn(0x7f1a240b1340 legacy=0x7f1a2415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.955+0000 7f1a1bfff640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a1bfff640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a240b35b0 0x7f1a100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a1bfff640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a10004d00 0x7f1a10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2415ee40 con 0x7f1a100039a0 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1000ba60 con 0x7f1a10004d00 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1000bc20 con 0x7f1a240b35b0 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a295d9640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a28dd8640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a240b35b0 0x7f1a100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a295d9640 1 -- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a240b35b0 msgr2=0x7f1a100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a29dda640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a10004d00 0x7f1a10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a295d9640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a240b35b0 0x7f1a100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a295d9640 1 -- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a10004d00 msgr2=0x7f1a10005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a295d9640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a10004d00 0x7f1a10005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a295d9640 1 -- 172.21.15.45:0/4089532337 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10003520 con 0x7f1a100039a0 2023-11-28T22:08:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a28dd8640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a240b35b0 0x7f1a100047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.956+0000 7f1a29dda640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a10004d00 0x7f1a10005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.958+0000 7f1a295d9640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a10003dc0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009a90 tx=0x7f1a100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ed5fd8e2aa19ec server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.958+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2002c680 con 0x7f1a100039a0 2023-11-28T22:08:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.958+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2002c7e0 con 0x7f1a100039a0 2023-11-28T22:08:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.958+0000 7f1a1bfff640 1 -- 172.21.15.45:0/4089532337 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2002cb70 con 0x7f1a100039a0 2023-11-28T22:08:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.958+0000 7f1a2b83d640 1 -- 172.21.15.45:0/4089532337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 msgr2=0x7f1a10003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.958+0000 7f1a2b83d640 1 --2- 172.21.15.45:0/4089532337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a10003dc0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009a90 tx=0x7f1a100057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.958+0000 7f1a29dda640 1 -- 172.21.15.45:0/4089532337 reap_dead start 2023-11-28T22:08:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.959+0000 7f1a2b83d640 1 -- 172.21.15.45:0/4089532337 shutdown_connections 2023-11-28T22:08:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.959+0000 7f1a2b83d640 1 -- 172.21.15.45:0/4089532337 >> 172.21.15.45:0/4089532337 conn(0x7f1a2406a500 msgr2=0x7f1a2406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.959+0000 7f1a2b83d640 1 -- 172.21.15.45:0/4089532337 shutdown_connections 2023-11-28T22:08:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.959+0000 7f1a2b83d640 1 -- 172.21.15.45:0/4089532337 wait complete. 2023-11-28T22:08:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.960+0000 7f1a2b83d640 1 Processor -- start 2023-11-28T22:08:26.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.961+0000 7f1a2b83d640 1 -- start start 2023-11-28T22:08:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a2b83d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a24140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a2b83d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a2413e460 0x7f1a240af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a2b83d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a2413d7e0 0x7f1a2413dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a2b83d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a24157fd0 con 0x7f1a100039a0 2023-11-28T22:08:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a2b83d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2413e1a0 con 0x7f1a2413e460 2023-11-28T22:08:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a2b83d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a24158270 con 0x7f1a2413d7e0 2023-11-28T22:08:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a295d9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a24140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a295d9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a24140cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49810/0 (socket says 172.21.15.45:49810) 2023-11-28T22:08:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a295d9640 1 -- 172.21.15.45:0/3944186472 learned_addr learned my addr 172.21.15.45:0/3944186472 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a295d9640 1 -- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a2413d7e0 msgr2=0x7f1a2413dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a295d9640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a2413d7e0 0x7f1a2413dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a295d9640 1 -- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a2413e460 msgr2=0x7f1a240af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a295d9640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a2413e460 0x7f1a240af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.962+0000 7f1a295d9640 1 -- 172.21.15.45:0/3944186472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a24158550 con 0x7f1a100039a0 2023-11-28T22:08:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.963+0000 7f1a295d9640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a24140cb0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f1a20000c00 tx=0x7f1a20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.963+0000 7f1a19ffb640 1 -- 172.21.15.45:0/3944186472 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20003810 con 0x7f1a100039a0 2023-11-28T22:08:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.963+0000 7f1a19ffb640 1 -- 172.21.15.45:0/3944186472 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a20003990 con 0x7f1a100039a0 2023-11-28T22:08:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.963+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a24158830 con 0x7f1a100039a0 2023-11-28T22:08:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.963+0000 7f1a19ffb640 1 -- 172.21.15.45:0/3944186472 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20003d40 con 0x7f1a100039a0 2023-11-28T22:08:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.963+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a24158b10 con 0x7f1a100039a0 2023-11-28T22:08:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.964+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a240733d0 con 0x7f1a100039a0 2023-11-28T22:08:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.965+0000 7f1a19ffb640 1 -- 172.21.15.45:0/3944186472 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 115) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f1a2003b070 con 0x7f1a100039a0 2023-11-28T22:08:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.966+0000 7f1a19ffb640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1a0806acc0 0x7f1a0806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.966+0000 7f1a19ffb640 1 -- 172.21.15.45:0/3944186472 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7117+0+0 (secure 0 0 0) 0x7f1a200ab590 con 0x7f1a100039a0 2023-11-28T22:08:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.967+0000 7f1a28dd8640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1a0806acc0 0x7f1a0806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.968+0000 7f1a28dd8640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1a0806acc0 0x7f1a0806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a24144470 tx=0x7f1a14009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:26.971+0000 7f1a19ffb640 1 -- 172.21.15.45:0/3944186472 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a20076500 con 0x7f1a100039a0 2023-11-28T22:08:27.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.199+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a240cd3b0 con 0x7f1a100039a0 2023-11-28T22:08:27.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.201+0000 7f1a19ffb640 1 -- 172.21.15.45:0/3944186472 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f1a2004c090 con 0x7f1a100039a0 2023-11-28T22:08:27.203 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:27.204 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3312844962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3312844962},{"type":"v1","addr":"172.21.15.190:6835","nonce":3312844962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":8296,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1302790439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1302790439},{"type":"v1","addr":"172.21.15.190:6837","nonce":1302790439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10104,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2082498185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2082498185},{"type":"v1","addr":"172.21.15.45:6834","nonce":2082498185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:52:25.939261+0000","modified":"2023-11-28T21:52:29.372705+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10110":{"gid":10110,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3063278837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3063278837},{"type":"v1","addr":"172.21.15.45:6835","nonce":3063278837}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-28T22:08:27.204 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 92 2023-11-28T22:08:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.203+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1a0806acc0 msgr2=0x7f1a0806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1a0806acc0 0x7f1a0806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a24144470 tx=0x7f1a14009040 comp rx=0 tx=0).stop 2023-11-28T22:08:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 msgr2=0x7f1a24140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a24140cb0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f1a20000c00 tx=0x7f1a20004730 comp rx=0 tx=0).stop 2023-11-28T22:08:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 shutdown_connections 2023-11-28T22:08:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1a2413d7e0 0x7f1a2413dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a2413e460 0x7f1a240af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1a0806acc0 0x7f1a0806d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 --2- 172.21.15.45:0/3944186472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a100039a0 0x7f1a24140cb0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 >> 172.21.15.45:0/3944186472 conn(0x7f1a2406a500 msgr2=0x7f1a240cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:27.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 shutdown_connections 2023-11-28T22:08:27.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.204+0000 7f1a2b83d640 1 -- 172.21.15.45:0/3944186472 wait complete. 2023-11-28T22:08:27.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:08:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.452+0000 7f524d1e3640 1 Processor -- start 2023-11-28T22:08:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.453+0000 7f524d1e3640 1 -- start start 2023-11-28T22:08:27.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.454+0000 7f524d1e3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52480b35b0 0x7f524815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.454+0000 7f524d1e3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f524815b850 0x7f52480b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.454+0000 7f524d1e3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f524815bda0 con 0x7f52480b1fa0 2023-11-28T22:08:27.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.454+0000 7f524d1e3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524815ee40 con 0x7f524815b850 2023-11-28T22:08:27.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.454+0000 7f524d1e3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524815c080 con 0x7f52480b35b0 2023-11-28T22:08:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.455+0000 7f523ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f524815b850 0x7f52480b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.455+0000 7f523ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f524815b850 0x7f52480b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49822/0 (socket says 172.21.15.45:49822) 2023-11-28T22:08:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.455+0000 7f523ffff640 1 -- 172.21.15.45:0/177375372 learned_addr learned my addr 172.21.15.45:0/177375372 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.455+0000 7f5246d9d640 1 --2- 172.21.15.45:0/177375372 >> v2:172.21.15.190:3300/0 conn(0x7f52480b35b0 0x7f524815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.455+0000 7f523ffff640 1 -- 172.21.15.45:0/177375372 >> v2:172.21.15.190:3300/0 conn(0x7f52480b35b0 msgr2=0x7f524815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.455+0000 7f523ffff640 1 --2- 172.21.15.45:0/177375372 >> v2:172.21.15.190:3300/0 conn(0x7f52480b35b0 0x7f524815ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.455+0000 7f523ffff640 1 -- 172.21.15.45:0/177375372 >> v1:172.21.15.190:6789/0 conn(0x7f52480b1fa0 legacy=0x7f524815f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.455+0000 7f523ffff640 1 -- 172.21.15.45:0/177375372 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524815c360 con 0x7f524815b850 2023-11-28T22:08:27.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.457+0000 7f523ffff640 1 --2- 172.21.15.45:0/177375372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f524815b850 0x7f52480b0c30 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f524806e7f0 tx=0x7f523002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900610aebc4c7d37 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:27.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.457+0000 7f524659c640 1 -- 172.21.15.45:0/177375372 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5230031870 con 0x7f524815b850 2023-11-28T22:08:27.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.457+0000 7f524659c640 1 -- 172.21.15.45:0/177375372 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52300319d0 con 0x7f524815b850 2023-11-28T22:08:27.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.457+0000 7f524659c640 1 -- 172.21.15.45:0/177375372 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5230031d60 con 0x7f524815b850 2023-11-28T22:08:27.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.458+0000 7f524d1e3640 1 -- 172.21.15.45:0/177375372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f524815b850 msgr2=0x7f52480b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.458+0000 7f524d1e3640 1 --2- 172.21.15.45:0/177375372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f524815b850 0x7f52480b0c30 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f524806e7f0 tx=0x7f523002b910 comp rx=0 tx=0).stop 2023-11-28T22:08:27.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.458+0000 7f524d1e3640 1 -- 172.21.15.45:0/177375372 shutdown_connections 2023-11-28T22:08:27.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.458+0000 7f524d1e3640 1 --2- 172.21.15.45:0/177375372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f524815b850 0x7f52480b0c30 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.458+0000 7f524d1e3640 1 --2- 172.21.15.45:0/177375372 >> v2:172.21.15.190:3300/0 conn(0x7f52480b35b0 0x7f524815ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.458+0000 7f524d1e3640 1 -- 172.21.15.45:0/177375372 >> 172.21.15.45:0/177375372 conn(0x7f524806a500 msgr2=0x7f524806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:27.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.458+0000 7f524d1e3640 1 -- 172.21.15.45:0/177375372 shutdown_connections 2023-11-28T22:08:27.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.458+0000 7f524d1e3640 1 -- 172.21.15.45:0/177375372 wait complete. 2023-11-28T22:08:27.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.460+0000 7f524d1e3640 1 Processor -- start 2023-11-28T22:08:27.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.460+0000 7f524d1e3640 1 -- start start 2023-11-28T22:08:27.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.461+0000 7f524d1e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52480b1fa0 0x7f52481f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.461+0000 7f524d1e3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52480b35b0 0x7f52480c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.461+0000 7f524d1e3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f524815b850 0x7f52480cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.461+0000 7f524d1e3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524815b370 con 0x7f52480b1fa0 2023-11-28T22:08:27.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.461+0000 7f524d1e3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480c68d0 con 0x7f52480b35b0 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.461+0000 7f524d1e3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5248135460 con 0x7f524815b850 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.461+0000 7f5246d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52480b1fa0 0x7f52481f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.461+0000 7f5246d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52480b1fa0 0x7f52481f2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49824/0 (socket says 172.21.15.45:49824) 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.461+0000 7f5246d9d640 1 -- 172.21.15.45:0/711884956 learned_addr learned my addr 172.21.15.45:0/711884956 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.462+0000 7f523ffff640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52480b35b0 0x7f52480c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.462+0000 7f5246d9d640 1 -- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f524815b850 msgr2=0x7f52480cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.462+0000 7f5246d9d640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f524815b850 0x7f52480cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.462+0000 7f5246d9d640 1 -- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52480b35b0 msgr2=0x7f52480c5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.462+0000 7f5246d9d640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52480b35b0 0x7f52480c5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.462+0000 7f5246d9d640 1 -- 172.21.15.45:0/711884956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52300094d0 con 0x7f52480b1fa0 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.462+0000 7f523ffff640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52480b35b0 0x7f52480c5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.462+0000 7f5246d9d640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52480b1fa0 0x7f52481f2850 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f523800d800 tx=0x7f523800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:27.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.462+0000 7f523f7fe640 1 -- 172.21.15.45:0/711884956 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523800b6c0 con 0x7f52480b1fa0 2023-11-28T22:08:27.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.463+0000 7f523f7fe640 1 -- 172.21.15.45:0/711884956 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523800b820 con 0x7f52480b1fa0 2023-11-28T22:08:27.466 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.463+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5248135740 con 0x7f52480b1fa0 2023-11-28T22:08:27.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.463+0000 7f523f7fe640 1 -- 172.21.15.45:0/711884956 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523800bbb0 con 0x7f52480b1fa0 2023-11-28T22:08:27.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.463+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5248135ad0 con 0x7f52480b1fa0 2023-11-28T22:08:27.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.464+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52480733d0 con 0x7f52480b1fa0 2023-11-28T22:08:27.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.465+0000 7f523f7fe640 1 -- 172.21.15.45:0/711884956 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 115) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f523800bd10 con 0x7f52480b1fa0 2023-11-28T22:08:27.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.466+0000 7f523f7fe640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f522806ac90 0x7f522806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.466+0000 7f523f7fe640 1 -- 172.21.15.45:0/711884956 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7117+0+0 (secure 0 0 0) 0x7f5238089d50 con 0x7f52480b1fa0 2023-11-28T22:08:27.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.466+0000 7f523ffff640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f522806ac90 0x7f522806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.468+0000 7f523ffff640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f522806ac90 0x7f522806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5230036000 tx=0x7f523003c040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:27.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.470+0000 7f523f7fe640 1 -- 172.21.15.45:0/711884956 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5238054bc0 con 0x7f52480b1fa0 2023-11-28T22:08:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.570+0000 7f523f7fe640 1 -- 172.21.15.45:0/711884956 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 116) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f523808f020 con 0x7f52480b1fa0 2023-11-28T22:08:27.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.681+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52480b16b0 con 0x7f52480b1fa0 2023-11-28T22:08:27.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.683+0000 7f523f7fe640 1 -- 172.21.15.45:0/711884956 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+18342 (secure 0 0 0) 0x7f5238086940 con 0x7f52480b1fa0 2023-11-28T22:08:27.686 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:27.686 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":204,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:08:25.533337+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:52:23.878130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-11-28T21:52:24.956954+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":203,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/667031807":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/454828214":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/2397783727":"2023-11-29T22:08:25.533236+0000","172.21.15.190:6833/697528389":"2023-11-29T22:08:25.533236+0000","172.21.15.190:6832/697528389":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/2743963182":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/3942686825":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/3880574704":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3087092286":"2023-11-29T22:08:25.533236+0000","172.21.15.190:6833/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6832/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3938773910":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2706131286":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3522054455":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2512261826":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3252656094":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2538549380":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3848257557":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6833/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3811510364":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2138187339":"2023-11-29T21:52:41.039626+0000","172.21.15.190:6832/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/624508208":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2094131491":"2023-11-29T21:52:41.039626+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f522806ac90 msgr2=0x7f522806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f522806ac90 0x7f522806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5230036000 tx=0x7f523003c040 comp rx=0 tx=0).stop 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52480b1fa0 msgr2=0x7f52481f2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52480b1fa0 0x7f52481f2850 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f523800d800 tx=0x7f523800dcc0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 shutdown_connections 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f524815b850 0x7f52480cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52480b35b0 0x7f52480c5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f522806ac90 0x7f522806d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 --2- 172.21.15.45:0/711884956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52480b1fa0 0x7f52481f2850 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 >> 172.21.15.45:0/711884956 conn(0x7f524806a500 msgr2=0x7f52480adfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:27.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 shutdown_connections 2023-11-28T22:08:27.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.686+0000 7f524d1e3640 1 -- 172.21.15.45:0/711884956 wait complete. 2023-11-28T22:08:27.710 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T22:08:27.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:08:27.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.937+0000 7f38b0b75640 1 Processor -- start 2023-11-28T22:08:27.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.938+0000 7f38b0b75640 1 -- start start 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.939+0000 7f38b0b75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 0x7f38ac0abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.939+0000 7f38b0b75640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f38ac06d640 0x7f38ac0b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.939+0000 7f38b0b75640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ac0708d0 con 0x7f38ac0b2830 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.939+0000 7f38b0b75640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ac0b31b0 con 0x7f38ac06d640 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.939+0000 7f38b0b75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac06c390 con 0x7f38ac159650 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.940+0000 7f38abfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f38ac06d640 0x7f38ac0b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.940+0000 7f38abfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f38ac06d640 0x7f38ac0b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49840/0 (socket says 172.21.15.45:49840) 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.940+0000 7f38abfff640 1 -- 172.21.15.45:0/736658532 learned_addr learned my addr 172.21.15.45:0/736658532 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.940+0000 7f38ab7fe640 1 --2- 172.21.15.45:0/736658532 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 0x7f38ac0abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.940+0000 7f38abfff640 1 -- 172.21.15.45:0/736658532 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 msgr2=0x7f38ac0abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:27.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.940+0000 7f38abfff640 1 --2- 172.21.15.45:0/736658532 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 0x7f38ac0abfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.940+0000 7f38abfff640 1 -- 172.21.15.45:0/736658532 >> v1:172.21.15.190:6789/0 conn(0x7f38ac0b2830 legacy=0x7f38ac06cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:27.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.940+0000 7f38abfff640 1 -- 172.21.15.45:0/736658532 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac0ac520 con 0x7f38ac06d640 2023-11-28T22:08:27.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.942+0000 7f38abfff640 1 --2- 172.21.15.45:0/736658532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac06d640 0x7f38ac0b2c70 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f38a40098a0 tx=0x7f38a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6efb6c683d284825 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:27.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.942+0000 7f38aa7fc640 1 -- 172.21.15.45:0/736658532 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4031640 con 0x7f38ac06d640 2023-11-28T22:08:27.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.942+0000 7f38aa7fc640 1 -- 172.21.15.45:0/736658532 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a40317a0 con 0x7f38ac06d640 2023-11-28T22:08:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.942+0000 7f38aa7fc640 1 -- 172.21.15.45:0/736658532 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4031b30 con 0x7f38ac06d640 2023-11-28T22:08:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.943+0000 7f38b0b75640 1 -- 172.21.15.45:0/736658532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac06d640 msgr2=0x7f38ac0b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.943+0000 7f38b0b75640 1 --2- 172.21.15.45:0/736658532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac06d640 0x7f38ac0b2c70 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f38a40098a0 tx=0x7f38a402b8f0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.943+0000 7f38b0b75640 1 -- 172.21.15.45:0/736658532 shutdown_connections 2023-11-28T22:08:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.943+0000 7f38b0b75640 1 --2- 172.21.15.45:0/736658532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac06d640 0x7f38ac0b2c70 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.943+0000 7f38b0b75640 1 --2- 172.21.15.45:0/736658532 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 0x7f38ac0abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.943+0000 7f38b0b75640 1 -- 172.21.15.45:0/736658532 >> 172.21.15.45:0/736658532 conn(0x7f38ac06ab30 msgr2=0x7f38ac0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.943+0000 7f38b0b75640 1 -- 172.21.15.45:0/736658532 shutdown_connections 2023-11-28T22:08:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.943+0000 7f38b0b75640 1 -- 172.21.15.45:0/736658532 wait complete. 2023-11-28T22:08:27.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.945+0000 7f38b0b75640 1 Processor -- start 2023-11-28T22:08:27.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.945+0000 7f38b0b75640 1 -- start start 2023-11-28T22:08:27.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38b0b75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac06d640 0x7f38ac15d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38b0b75640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac0b2830 0x7f38ac14dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38b0b75640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 0x7f38ac14e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38b0b75640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac15cb70 con 0x7f38ac0b2830 2023-11-28T22:08:27.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38b0b75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac15bf60 con 0x7f38ac06d640 2023-11-28T22:08:27.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38b0b75640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac15c240 con 0x7f38ac159650 2023-11-28T22:08:27.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38aaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac0b2830 0x7f38ac14dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38aaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac0b2830 0x7f38ac14dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49842/0 (socket says 172.21.15.45:49842) 2023-11-28T22:08:27.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38aaffd640 1 -- 172.21.15.45:0/1950708758 learned_addr learned my addr 172.21.15.45:0/1950708758 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:27.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38ab7fe640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac06d640 0x7f38ac15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38abfff640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 0x7f38ac14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38aaffd640 1 -- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 msgr2=0x7f38ac14e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38aaffd640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 0x7f38ac14e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38aaffd640 1 -- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac06d640 msgr2=0x7f38ac15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:27.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38aaffd640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac06d640 0x7f38ac15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:27.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.946+0000 7f38aaffd640 1 -- 172.21.15.45:0/1950708758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a40094d0 con 0x7f38ac0b2830 2023-11-28T22:08:27.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.947+0000 7f38aaffd640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac0b2830 0x7f38ac14dc20 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f389400c7c0 tx=0x7f389400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:27.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.947+0000 7f38a3fff640 1 -- 172.21.15.45:0/1950708758 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389400aa50 con 0x7f38ac0b2830 2023-11-28T22:08:27.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.947+0000 7f38a3fff640 1 -- 172.21.15.45:0/1950708758 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389400abb0 con 0x7f38ac0b2830 2023-11-28T22:08:27.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.947+0000 7f38a3fff640 1 -- 172.21.15.45:0/1950708758 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894012420 con 0x7f38ac0b2830 2023-11-28T22:08:27.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.947+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ac15c520 con 0x7f38ac0b2830 2023-11-28T22:08:27.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.947+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ac146b30 con 0x7f38ac0b2830 2023-11-28T22:08:27.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.948+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ac15cce0 con 0x7f38ac0b2830 2023-11-28T22:08:27.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.949+0000 7f38a3fff640 1 -- 172.21.15.45:0/1950708758 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 116) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f3894012580 con 0x7f38ac0b2830 2023-11-28T22:08:27.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.951+0000 7f38a3fff640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f388c06aed0 0x7f388c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:27.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.951+0000 7f38a3fff640 1 -- 172.21.15.45:0/1950708758 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7117+0+0 (secure 0 0 0) 0x7f3894089b90 con 0x7f38ac0b2830 2023-11-28T22:08:27.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.952+0000 7f38ab7fe640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f388c06aed0 0x7f388c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:27.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.953+0000 7f38ab7fe640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f388c06aed0 0x7f388c06d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f389c00a970 tx=0x7f389c008040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:27.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:27.956+0000 7f38a3fff640 1 -- 172.21.15.45:0/1950708758 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3894054660 con 0x7f38ac0b2830 2023-11-28T22:08:28.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.191+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38ac04e920 con 0x7f38ac0b2830 2023-11-28T22:08:28.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.192+0000 7f38a3fff640 1 -- 172.21.15.45:0/1950708758 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f3894016020 con 0x7f38ac0b2830 2023-11-28T22:08:28.195 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:28.195 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3312844962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3312844962},{"type":"v1","addr":"172.21.15.190:6835","nonce":3312844962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":8296,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1302790439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1302790439},{"type":"v1","addr":"172.21.15.190:6837","nonce":1302790439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10104,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2082498185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2082498185},{"type":"v1","addr":"172.21.15.45:6834","nonce":2082498185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:52:25.939261+0000","modified":"2023-11-28T21:52:29.372705+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10110":{"gid":10110,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3063278837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3063278837},{"type":"v1","addr":"172.21.15.45:6835","nonce":3063278837}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-28T22:08:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 92 2023-11-28T22:08:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f388c06aed0 msgr2=0x7f388c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f388c06aed0 0x7f388c06d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f389c00a970 tx=0x7f389c008040 comp rx=0 tx=0).stop 2023-11-28T22:08:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac0b2830 msgr2=0x7f38ac14dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac0b2830 0x7f38ac14dc20 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f389400c7c0 tx=0x7f389400cc80 comp rx=0 tx=0).stop 2023-11-28T22:08:28.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 shutdown_connections 2023-11-28T22:08:28.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38ac159650 0x7f38ac14e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f388c06aed0 0x7f388c06d390 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38ac0b2830 0x7f38ac14dc20 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 --2- 172.21.15.45:0/1950708758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38ac06d640 0x7f38ac15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 >> 172.21.15.45:0/1950708758 conn(0x7f38ac06ab30 msgr2=0x7f38ac0ac740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 shutdown_connections 2023-11-28T22:08:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.195+0000 7f38b0b75640 1 -- 172.21.15.45:0/1950708758 wait complete. 2023-11-28T22:08:28.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:08:28.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.435+0000 7f5aa25c9640 1 Processor -- start 2023-11-28T22:08:28.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.435+0000 7f5aa25c9640 1 -- start start 2023-11-28T22:08:28.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.437+0000 7f5aa25c9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a9c0d02d0 0x7f5a9c0d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.437+0000 7f5aa25c9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5a9c0d1ab0 0x7f5a9c0b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.437+0000 7f5aa25c9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9c0d2230 con 0x7f5a9c15d550 2023-11-28T22:08:28.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.437+0000 7f5aa25c9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9c0d2510 con 0x7f5a9c0d02d0 2023-11-28T22:08:28.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.437+0000 7f5aa25c9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9c0d27f0 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.437+0000 7f5aa0dc6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a9c0d02d0 0x7f5a9c0d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.437+0000 7f5aa0dc6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a9c0d02d0 0x7f5a9c0d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49844/0 (socket says 172.21.15.45:49844) 2023-11-28T22:08:28.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.437+0000 7f5aa0dc6640 1 -- 172.21.15.45:0/1585051931 learned_addr learned my addr 172.21.15.45:0/1585051931 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.438+0000 7f5aa1dc8640 1 --2- 172.21.15.45:0/1585051931 >> v2:172.21.15.190:3300/0 conn(0x7f5a9c0d1ab0 0x7f5a9c0b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.438+0000 7f5aa0dc6640 1 -- 172.21.15.45:0/1585051931 >> v2:172.21.15.190:3300/0 conn(0x7f5a9c0d1ab0 msgr2=0x7f5a9c0b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.438+0000 7f5aa0dc6640 1 --2- 172.21.15.45:0/1585051931 >> v2:172.21.15.190:3300/0 conn(0x7f5a9c0d1ab0 0x7f5a9c0b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.438+0000 7f5aa0dc6640 1 -- 172.21.15.45:0/1585051931 >> v1:172.21.15.190:6789/0 conn(0x7f5a9c15d550 legacy=0x7f5a9c0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.438+0000 7f5aa0dc6640 1 -- 172.21.15.45:0/1585051931 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c0b6a30 con 0x7f5a9c0d02d0 2023-11-28T22:08:28.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.439+0000 7f5aa0dc6640 1 --2- 172.21.15.45:0/1585051931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d02d0 0x7f5a9c0d1570 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c1610c0 tx=0x7f5a8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ef9e22d73056aa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:28.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.439+0000 7f5a93fff640 1 -- 172.21.15.45:0/1585051931 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c031870 con 0x7f5a9c0d02d0 2023-11-28T22:08:28.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.439+0000 7f5a93fff640 1 -- 172.21.15.45:0/1585051931 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8c0319d0 con 0x7f5a9c0d02d0 2023-11-28T22:08:28.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.439+0000 7f5a93fff640 1 -- 172.21.15.45:0/1585051931 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c031d60 con 0x7f5a9c0d02d0 2023-11-28T22:08:28.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.440+0000 7f5aa25c9640 1 -- 172.21.15.45:0/1585051931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d02d0 msgr2=0x7f5a9c0d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.440+0000 7f5aa25c9640 1 --2- 172.21.15.45:0/1585051931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d02d0 0x7f5a9c0d1570 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c1610c0 tx=0x7f5a8c02b910 comp rx=0 tx=0).stop 2023-11-28T22:08:28.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.440+0000 7f5aa25c9640 1 -- 172.21.15.45:0/1585051931 shutdown_connections 2023-11-28T22:08:28.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.440+0000 7f5aa25c9640 1 --2- 172.21.15.45:0/1585051931 >> v2:172.21.15.190:3300/0 conn(0x7f5a9c0d1ab0 0x7f5a9c0b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.440+0000 7f5aa25c9640 1 --2- 172.21.15.45:0/1585051931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d02d0 0x7f5a9c0d1570 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.440+0000 7f5aa25c9640 1 -- 172.21.15.45:0/1585051931 >> 172.21.15.45:0/1585051931 conn(0x7f5a9c06ac70 msgr2=0x7f5a9c0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:28.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.440+0000 7f5aa25c9640 1 -- 172.21.15.45:0/1585051931 shutdown_connections 2023-11-28T22:08:28.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.440+0000 7f5aa25c9640 1 -- 172.21.15.45:0/1585051931 wait complete. 2023-11-28T22:08:28.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.442+0000 7f5aa25c9640 1 Processor -- start 2023-11-28T22:08:28.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.442+0000 7f5aa25c9640 1 -- start start 2023-11-28T22:08:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.443+0000 7f5aa25c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a9c0d02d0 0x7f5a9c0a9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.443+0000 7f5aa25c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d1ab0 0x7f5a9c0ac610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.443+0000 7f5aa25c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a9c15d550 0x7f5a9c0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.443+0000 7f5aa25c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c1f42c0 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.443+0000 7f5aa25c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c1f45a0 con 0x7f5a9c15d550 2023-11-28T22:08:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.443+0000 7f5aa25c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c1f4880 con 0x7f5a9c0d02d0 2023-11-28T22:08:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa0dc6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d1ab0 0x7f5a9c0ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa0dc6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d1ab0 0x7f5a9c0ac610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49848/0 (socket says 172.21.15.45:49848) 2023-11-28T22:08:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa0dc6640 1 -- 172.21.15.45:0/3190049688 learned_addr learned my addr 172.21.15.45:0/3190049688 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:28.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa15c7640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a9c0d02d0 0x7f5a9c0a9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa1dc8640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a9c15d550 0x7f5a9c0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa0dc6640 1 -- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a9c0d02d0 msgr2=0x7f5a9c0a9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa0dc6640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a9c0d02d0 0x7f5a9c0a9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa0dc6640 1 -- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a9c15d550 msgr2=0x7f5a9c0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa0dc6640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a9c15d550 0x7f5a9c0cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.444+0000 7f5aa0dc6640 1 -- 172.21.15.45:0/3190049688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c0094d0 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.445+0000 7f5aa0dc6640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d1ab0 0x7f5a9c0ac610 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0090b0 tx=0x7f5a8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.445+0000 7f5a91ffb640 1 -- 172.21.15.45:0/3190049688 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c03ad20 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.445+0000 7f5a91ffb640 1 -- 172.21.15.45:0/3190049688 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8c0443f0 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.445+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c1f4b60 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.445+0000 7f5a91ffb640 1 -- 172.21.15.45:0/3190049688 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c03aea0 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.445+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9c1f3f00 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.446+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9c1f5080 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.447+0000 7f5a91ffb640 1 -- 172.21.15.45:0/3190049688 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 116) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f5a8c04d450 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.448+0000 7f5a91ffb640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a8006aed0 0x7f5a8006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.448+0000 7f5a91ffb640 1 -- 172.21.15.45:0/3190049688 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7117+0+0 (secure 0 0 0) 0x7f5a8c02d040 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.449+0000 7f5aa15c7640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a8006aed0 0x7f5a8006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.450+0000 7f5aa15c7640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a8006aed0 0x7f5a8006d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a98006fd0 tx=0x7f5a98008040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.452+0000 7f5a91ffb640 1 -- 172.21.15.45:0/3190049688 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a9c1f5080 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.690+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a9c1f5080 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.691+0000 7f5a91ffb640 1 -- 172.21.15.45:0/3190049688 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4927 (secure 0 0 0) 0x7f5a9c1f5080 con 0x7f5a9c0d1ab0 2023-11-28T22:08:28.694 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:28.694 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3312844962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3312844962},{"type":"v1","addr":"172.21.15.190:6835","nonce":3312844962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":8296,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1302790439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1302790439},{"type":"v1","addr":"172.21.15.190:6837","nonce":1302790439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10104,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2082498185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2082498185},{"type":"v1","addr":"172.21.15.45:6834","nonce":2082498185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T21:52:25.939261+0000","modified":"2023-11-28T21:52:29.372705+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10110":{"gid":10110,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/3063278837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":3063278837},{"type":"v1","addr":"172.21.15.45:6835","nonce":3063278837}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-11-28T22:08:28.695 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 92 2023-11-28T22:08:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.694+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a8006aed0 msgr2=0x7f5a8006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.694+0000 7f5aa25c9640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a8006aed0 0x7f5a8006d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a98006fd0 tx=0x7f5a98008040 comp rx=0 tx=0).stop 2023-11-28T22:08:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.694+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d1ab0 msgr2=0x7f5a9c0ac610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.694+0000 7f5aa25c9640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d1ab0 0x7f5a9c0ac610 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0090b0 tx=0x7f5a8c0046c0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.694+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 shutdown_connections 2023-11-28T22:08:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.695+0000 7f5aa25c9640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a9c15d550 0x7f5a9c0cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.695+0000 7f5aa25c9640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a8006aed0 0x7f5a8006d390 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.695+0000 7f5aa25c9640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a9c0d1ab0 0x7f5a9c0ac610 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.695+0000 7f5aa25c9640 1 --2- 172.21.15.45:0/3190049688 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a9c0d02d0 0x7f5a9c0a9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.695+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 >> 172.21.15.45:0/3190049688 conn(0x7f5a9c06ac70 msgr2=0x7f5a9c0ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.695+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 shutdown_connections 2023-11-28T22:08:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.695+0000 7f5aa25c9640 1 -- 172.21.15.45:0/3190049688 wait complete. 2023-11-28T22:08:28.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:08:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.944+0000 7fd09bfff640 1 Processor -- start 2023-11-28T22:08:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.944+0000 7fd09bfff640 1 -- start start 2023-11-28T22:08:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.945+0000 7fd09bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd09c15d550 0x7fd09c0cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.945+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0d0100 0x7fd09c0d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.945+0000 7fd09bfff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c0d0560 con 0x7fd09c0d1ab0 2023-11-28T22:08:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.946+0000 7fd09bfff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c0d2230 con 0x7fd09c15d550 2023-11-28T22:08:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.946+0000 7fd09bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0d2510 con 0x7fd09c0d0100 2023-11-28T22:08:28.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.946+0000 7fd09b7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd09c0d1ab0 0x7fd09c070b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37670/0 (socket says 172.21.15.45:37670) 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.946+0000 7fd09b7fe640 1 -- 172.21.15.45:0/3234080914 learned_addr learned my addr 172.21.15.45:0/3234080914 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.946+0000 7fd09a7fc640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0d0100 0x7fd09c0d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.946+0000 7fd09affd640 1 --2- 172.21.15.45:0/3234080914 >> v2:172.21.15.190:3300/0 conn(0x7fd09c15d550 0x7fd09c0cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.946+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357709392 0 0) 0x7fd09c0d0560 con 0x7fd09c0d1ab0 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd088003520 con 0x7fd09c0d1ab0 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960138946 0 0) 0x7fd088003520 con 0x7fd09c0d1ab0 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0d0100 msgr2=0x7fd09c0d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0d0100 0x7fd09c0d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 >> v2:172.21.15.190:3300/0 conn(0x7fd09c15d550 msgr2=0x7fd09c0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 --2- 172.21.15.45:0/3234080914 >> v2:172.21.15.190:3300/0 conn(0x7fd09c15d550 0x7fd09c0cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c0d27f0 con 0x7fd09c0d1ab0 2023-11-28T22:08:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd094001670 con 0x7fd09c0d1ab0 2023-11-28T22:08:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 >> v1:172.21.15.45:6789/0 conn(0x7fd09c0d1ab0 legacy=0x7fd09c070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd088003970 0x7fd088003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0880047c0 0x7fd088004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0d0100 0x7fd088005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0d2230 con 0x7fd088003970 2023-11-28T22:08:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0d2510 con 0x7fd0880047c0 2023-11-28T22:08:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.947+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08800b980 con 0x7fd09c0d0100 2023-11-28T22:08:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.948+0000 7fd09affd640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd088003970 0x7fd088003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.948+0000 7fd09a7fc640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0880047c0 0x7fd088004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.948+0000 7fd09affd640 1 -- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0d0100 msgr2=0x7fd088005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.948+0000 7fd09affd640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0d0100 0x7fd088005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.948+0000 7fd09affd640 1 -- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0880047c0 msgr2=0x7fd088004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.948+0000 7fd09affd640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0880047c0 0x7fd088004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.948+0000 7fd09affd640 1 -- 172.21.15.45:0/3234080914 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd088003520 con 0x7fd088003970 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.949+0000 7fd09affd640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd088003970 0x7fd088003d90 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009fc0 tx=0x7fd0880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d5012a652d27ef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.949+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c02cc60 con 0x7fd088003970 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.949+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c00f040 con 0x7fd088003970 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.949+0000 7fd099ffb640 1 -- 172.21.15.45:0/3234080914 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c034440 con 0x7fd088003970 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.950+0000 7fd09bfff640 1 -- 172.21.15.45:0/3234080914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd088003970 msgr2=0x7fd088003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.950+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3234080914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd088003970 0x7fd088003d90 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009fc0 tx=0x7fd0880057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.950+0000 7fd09b7fe640 1 -- 172.21.15.45:0/3234080914 reap_dead start 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.950+0000 7fd09bfff640 1 -- 172.21.15.45:0/3234080914 shutdown_connections 2023-11-28T22:08:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.950+0000 7fd09bfff640 1 -- 172.21.15.45:0/3234080914 >> 172.21.15.45:0/3234080914 conn(0x7fd09c06ac70 msgr2=0x7fd09c0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.950+0000 7fd09bfff640 1 -- 172.21.15.45:0/3234080914 shutdown_connections 2023-11-28T22:08:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.950+0000 7fd09bfff640 1 -- 172.21.15.45:0/3234080914 wait complete. 2023-11-28T22:08:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.952+0000 7fd09bfff640 1 Processor -- start 2023-11-28T22:08:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.952+0000 7fd09bfff640 1 -- start start 2023-11-28T22:08:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.953+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd088003970 0x7fd09c0b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.953+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0a9b20 0x7fd09c0a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.953+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0cbcb0 0x7fd09c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.953+0000 7fd09bfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0cccf0 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.953+0000 7fd09bfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0b49b0 con 0x7fd088003970 2023-11-28T22:08:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.953+0000 7fd09bfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0aa4e0 con 0x7fd09c0a9b20 2023-11-28T22:08:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.954+0000 7fd09b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0cbcb0 0x7fd09c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.954+0000 7fd09b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0cbcb0 0x7fd09c0cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49856/0 (socket says 172.21.15.45:49856) 2023-11-28T22:08:28.956 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.954+0000 7fd09b7fe640 1 -- 172.21.15.45:0/3112354582 learned_addr learned my addr 172.21.15.45:0/3112354582 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:28.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.954+0000 7fd09b7fe640 1 -- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0a9b20 msgr2=0x7fd09c0a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.954+0000 7fd09b7fe640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0a9b20 0x7fd09c0a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.954+0000 7fd09b7fe640 1 -- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd088003970 msgr2=0x7fd09c0b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:28.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.954+0000 7fd09b7fe640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd088003970 0x7fd09c0b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.954+0000 7fd09b7fe640 1 -- 172.21.15.45:0/3112354582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c0c7da0 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.954+0000 7fd09b7fe640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0cbcb0 0x7fd09c0cc130 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fd09c071bc0 tx=0x7fd09400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.955+0000 7fd0937fe640 1 -- 172.21.15.45:0/3112354582 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094007440 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.955+0000 7fd0937fe640 1 -- 172.21.15.45:0/3112354582 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0940075e0 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.955+0000 7fd0937fe640 1 -- 172.21.15.45:0/3112354582 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094004a20 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.955+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c0cca10 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.955+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c0c7a30 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.956+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c0c75b0 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.957+0000 7fd0937fe640 1 -- 172.21.15.45:0/3112354582 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 116) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fd094004b80 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.958+0000 7fd0937fe640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd07c06aed0 0x7fd07c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.958+0000 7fd0937fe640 1 -- 172.21.15.45:0/3112354582 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7117+0+0 (secure 0 0 0) 0x7fd094089160 con 0x7fd09c0cbcb0 2023-11-28T22:08:28.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.959+0000 7fd09affd640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd07c06aed0 0x7fd07c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.960+0000 7fd09affd640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd07c06aed0 0x7fd07c06d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd08c036000 tx=0x7fd08c02e070 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:28.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:28.962+0000 7fd0937fe640 1 -- 172.21.15.45:0/3112354582 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd09c0c75b0 con 0x7fd09c0cbcb0 2023-11-28T22:08:29.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.160+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd09c0c75b0 con 0x7fd09c0cbcb0 2023-11-28T22:08:29.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.163+0000 7fd0937fe640 1 -- 172.21.15.45:0/3112354582 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+18342 (secure 0 0 0) 0x7fd09c0c75b0 con 0x7fd09c0cbcb0 2023-11-28T22:08:29.165 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:29.165 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":204,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:08:25.533337+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-11-28T21:52:23.878130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-11-28T21:52:24.956954+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":203,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/667031807":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/454828214":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/2397783727":"2023-11-29T22:08:25.533236+0000","172.21.15.190:6833/697528389":"2023-11-29T22:08:25.533236+0000","172.21.15.190:6832/697528389":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/2743963182":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/3942686825":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/3880574704":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3087092286":"2023-11-29T22:08:25.533236+0000","172.21.15.190:6833/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6832/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3938773910":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2706131286":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3522054455":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2512261826":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3252656094":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2538549380":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3848257557":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6833/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3811510364":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2138187339":"2023-11-29T21:52:41.039626+0000","172.21.15.190:6832/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/624508208":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2094131491":"2023-11-29T21:52:41.039626+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:08:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.165+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd07c06aed0 msgr2=0x7fd07c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.165+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd07c06aed0 0x7fd07c06d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd08c036000 tx=0x7fd08c02e070 comp rx=0 tx=0).stop 2023-11-28T22:08:29.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.165+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0cbcb0 msgr2=0x7fd09c0cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.165+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0cbcb0 0x7fd09c0cc130 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fd09c071bc0 tx=0x7fd09400c960 comp rx=0 tx=0).stop 2023-11-28T22:08:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.165+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 shutdown_connections 2023-11-28T22:08:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.166+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd07c06aed0 0x7fd07c06d390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.166+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0cbcb0 0x7fd09c0cc130 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.166+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd09c0a9b20 0x7fd09c0a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.166+0000 7fd09bfff640 1 --2- 172.21.15.45:0/3112354582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd088003970 0x7fd09c0b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.166+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 >> 172.21.15.45:0/3112354582 conn(0x7fd09c06ac70 msgr2=0x7fd09c15e310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.166+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 shutdown_connections 2023-11-28T22:08:29.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.166+0000 7fd09bfff640 1 -- 172.21.15.45:0/3112354582 wait complete. 2023-11-28T22:08:29.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T22:08:29.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.408+0000 7fb0959a5640 1 Processor -- start 2023-11-28T22:08:29.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.408+0000 7fb0959a5640 1 -- start start 2023-11-28T22:08:29.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.410+0000 7fb0959a5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb0900cf730 0x7fb09006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.410+0000 7fb0959a5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb09015edc0 0x7fb09015f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.410+0000 7fb0959a5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09015fa80 con 0x7fb09015b820 2023-11-28T22:08:29.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.410+0000 7fb0959a5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09015fd60 con 0x7fb0900cf730 2023-11-28T22:08:29.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.410+0000 7fb0959a5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09006cf40 con 0x7fb09015edc0 2023-11-28T22:08:29.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.410+0000 7fb08e7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb0900cf730 0x7fb09006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.410+0000 7fb08e7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb0900cf730 0x7fb09006da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49868/0 (socket says 172.21.15.45:49868) 2023-11-28T22:08:29.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.410+0000 7fb08e7fc640 1 -- 172.21.15.45:0/1102632545 learned_addr learned my addr 172.21.15.45:0/1102632545 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:29.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.411+0000 7fb08f7fe640 1 --2- 172.21.15.45:0/1102632545 >> v2:172.21.15.190:3300/0 conn(0x7fb09015edc0 0x7fb09015f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.411+0000 7fb08e7fc640 1 -- 172.21.15.45:0/1102632545 >> v2:172.21.15.190:3300/0 conn(0x7fb09015edc0 msgr2=0x7fb09015f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:29.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.411+0000 7fb08e7fc640 1 --2- 172.21.15.45:0/1102632545 >> v2:172.21.15.190:3300/0 conn(0x7fb09015edc0 0x7fb09015f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.411+0000 7fb08e7fc640 1 -- 172.21.15.45:0/1102632545 >> v1:172.21.15.190:6789/0 conn(0x7fb09015b820 legacy=0x7fb0900cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:29.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.411+0000 7fb08e7fc640 1 -- 172.21.15.45:0/1102632545 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09006d220 con 0x7fb0900cf730 2023-11-28T22:08:29.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.412+0000 7fb08e7fc640 1 --2- 172.21.15.45:0/1102632545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0900cf730 0x7fb09006da20 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fb0900cb720 tx=0x7fb08402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b04e2a5ef70f14 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:29.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.412+0000 7fb08dffb640 1 -- 172.21.15.45:0/1102632545 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08402ca80 con 0x7fb0900cf730 2023-11-28T22:08:29.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.412+0000 7fb08dffb640 1 -- 172.21.15.45:0/1102632545 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08402cbe0 con 0x7fb0900cf730 2023-11-28T22:08:29.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.412+0000 7fb08dffb640 1 -- 172.21.15.45:0/1102632545 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0840343f0 con 0x7fb0900cf730 2023-11-28T22:08:29.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.413+0000 7fb0959a5640 1 -- 172.21.15.45:0/1102632545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0900cf730 msgr2=0x7fb09006da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.413+0000 7fb0959a5640 1 --2- 172.21.15.45:0/1102632545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0900cf730 0x7fb09006da20 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fb0900cb720 tx=0x7fb08402bee0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.413+0000 7fb0959a5640 1 -- 172.21.15.45:0/1102632545 shutdown_connections 2023-11-28T22:08:29.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.413+0000 7fb0959a5640 1 --2- 172.21.15.45:0/1102632545 >> v2:172.21.15.190:3300/0 conn(0x7fb09015edc0 0x7fb09015f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.413+0000 7fb0959a5640 1 --2- 172.21.15.45:0/1102632545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0900cf730 0x7fb09006da20 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.413+0000 7fb0959a5640 1 -- 172.21.15.45:0/1102632545 >> 172.21.15.45:0/1102632545 conn(0x7fb09006a6e0 msgr2=0x7fb09006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:29.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.413+0000 7fb0959a5640 1 -- 172.21.15.45:0/1102632545 shutdown_connections 2023-11-28T22:08:29.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.413+0000 7fb0959a5640 1 -- 172.21.15.45:0/1102632545 wait complete. 2023-11-28T22:08:29.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.415+0000 7fb0959a5640 1 Processor -- start 2023-11-28T22:08:29.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.415+0000 7fb0959a5640 1 -- start start 2023-11-28T22:08:29.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.416+0000 7fb0959a5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0900cf730 0x7fb0900b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.416+0000 7fb0959a5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb09015b820 0x7fb0900c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.416+0000 7fb0959a5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09015edc0 0x7fb09014d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.416+0000 7fb0959a5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900ca3a0 con 0x7fb09015edc0 2023-11-28T22:08:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.416+0000 7fb0959a5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900ca660 con 0x7fb0900cf730 2023-11-28T22:08:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.416+0000 7fb0959a5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900ca940 con 0x7fb09015b820 2023-11-28T22:08:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.416+0000 7fb08f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09015edc0 0x7fb09014d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09015edc0 0x7fb09014d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49872/0 (socket says 172.21.15.45:49872) 2023-11-28T22:08:29.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08f7fe640 1 -- 172.21.15.45:0/1967618654 learned_addr learned my addr 172.21.15.45:0/1967618654 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08effd640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0900cf730 0x7fb0900b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08e7fc640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb09015b820 0x7fb0900c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08f7fe640 1 -- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb09015b820 msgr2=0x7fb0900c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08f7fe640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb09015b820 0x7fb0900c9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08f7fe640 1 -- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0900cf730 msgr2=0x7fb0900b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08f7fe640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0900cf730 0x7fb0900b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08f7fe640 1 -- 172.21.15.45:0/1967618654 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb084009b30 con 0x7fb09015edc0 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08e7fc640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb09015b820 0x7fb0900c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.417+0000 7fb08f7fe640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09015edc0 0x7fb09014d260 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fb08000c550 tx=0x7fb08000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.418+0000 7fb077fff640 1 -- 172.21.15.45:0/1967618654 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080004220 con 0x7fb09015edc0 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.418+0000 7fb077fff640 1 -- 172.21.15.45:0/1967618654 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb080004380 con 0x7fb09015edc0 2023-11-28T22:08:29.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.418+0000 7fb077fff640 1 -- 172.21.15.45:0/1967618654 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080003830 con 0x7fb09015edc0 2023-11-28T22:08:29.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.418+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0900cac20 con 0x7fb09015edc0 2023-11-28T22:08:29.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.418+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09014e410 con 0x7fb09015edc0 2023-11-28T22:08:29.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.419+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09014e600 con 0x7fb09015edc0 2023-11-28T22:08:29.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.420+0000 7fb077fff640 1 -- 172.21.15.45:0/1967618654 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 116) v1 ==== 89277+0+0 (secure 0 0 0) 0x7fb080003990 con 0x7fb09015edc0 2023-11-28T22:08:29.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.421+0000 7fb077fff640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb07c06aed0 0x7fb07c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.422+0000 7fb077fff640 1 -- 172.21.15.45:0/1967618654 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7117+0+0 (secure 0 0 0) 0x7fb080089ac0 con 0x7fb09015edc0 2023-11-28T22:08:29.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.422+0000 7fb08effd640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb07c06aed0 0x7fb07c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.423+0000 7fb08effd640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb07c06aed0 0x7fb07c06d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb0901f66f0 tx=0x7fb088009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:29.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.426+0000 7fb077fff640 1 -- 172.21.15.45:0/1967618654 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0800543f0 con 0x7fb09015edc0 2023-11-28T22:08:29.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.594+0000 7fb077fff640 1 -- 172.21.15.45:0/1967618654 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb0800547c0 con 0x7fb09015edc0 2023-11-28T22:08:29.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.668+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb090073440 con 0x7fb09015edc0 2023-11-28T22:08:29.671 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T22:08:29.669+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T22:08:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.672+0000 7fb077fff640 1 -- 172.21.15.45:0/1967618654 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v93) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb080071ed0 con 0x7fb09015edc0 2023-11-28T22:08:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T22:08:29.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.676+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb07c06aed0 msgr2=0x7fb07c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.676+0000 7fb0959a5640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb07c06aed0 0x7fb07c06d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb0901f66f0 tx=0x7fb088009040 comp rx=0 tx=0).stop 2023-11-28T22:08:29.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.676+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09015edc0 msgr2=0x7fb09014d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.676+0000 7fb0959a5640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09015edc0 0x7fb09014d260 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fb08000c550 tx=0x7fb08000ca10 comp rx=0 tx=0).stop 2023-11-28T22:08:29.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.677+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 shutdown_connections 2023-11-28T22:08:29.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.677+0000 7fb0959a5640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb07c06aed0 0x7fb07c06d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.677+0000 7fb0959a5640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09015edc0 0x7fb09014d260 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.677+0000 7fb0959a5640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb09015b820 0x7fb0900c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.677+0000 7fb0959a5640 1 --2- 172.21.15.45:0/1967618654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0900cf730 0x7fb0900b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.677+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 >> 172.21.15.45:0/1967618654 conn(0x7fb09006a6e0 msgr2=0x7fb0900ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:29.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.677+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 shutdown_connections 2023-11-28T22:08:29.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.677+0000 7fb0959a5640 1 -- 172.21.15.45:0/1967618654 wait complete. 2023-11-28T22:08:29.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T22:08:29.760 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T22:08:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.910+0000 7f358e02e640 1 Processor -- start 2023-11-28T22:08:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.911+0000 7f358e02e640 1 -- start start 2023-11-28T22:08:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.912+0000 7f358e02e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3588071f20 0x7f35880b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.912+0000 7f358e02e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f35880b2240 0x7f3588160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.912+0000 7f358e02e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358806dbe0 con 0x7f35880b3490 2023-11-28T22:08:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.912+0000 7f358e02e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f358806e110 con 0x7f3588071f20 2023-11-28T22:08:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.912+0000 7f358e02e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f358815d840 con 0x7f35880b2240 2023-11-28T22:08:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.913+0000 7f358d02c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f35880b3490 0x7f358815e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37680/0 (socket says 172.21.15.45:37680) 2023-11-28T22:08:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.913+0000 7f358d02c640 1 -- 172.21.15.45:0/3740768006 learned_addr learned my addr 172.21.15.45:0/3740768006 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:29.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.913+0000 7f358c82b640 1 --2- 172.21.15.45:0/3740768006 >> v2:172.21.15.45:3300/0 conn(0x7f3588071f20 0x7f35880b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.913+0000 7f358d82d640 1 --2- 172.21.15.45:0/3740768006 >> v2:172.21.15.190:3300/0 conn(0x7f35880b2240 0x7f3588160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.914+0000 7f357ffff640 1 -- 172.21.15.45:0/3740768006 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943237646 0 0) 0x7f358806dbe0 con 0x7f35880b3490 2023-11-28T22:08:29.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.914+0000 7f357ffff640 1 -- 172.21.15.45:0/3740768006 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3574003520 con 0x7f35880b3490 2023-11-28T22:08:29.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.914+0000 7f358c82b640 1 -- 172.21.15.45:0/3740768006 >> v2:172.21.15.190:3300/0 conn(0x7f35880b2240 msgr2=0x7f3588160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:29.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.914+0000 7f358c82b640 1 --2- 172.21.15.45:0/3740768006 >> v2:172.21.15.190:3300/0 conn(0x7f35880b2240 0x7f3588160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.914+0000 7f358c82b640 1 -- 172.21.15.45:0/3740768006 >> v1:172.21.15.45:6789/0 conn(0x7f35880b3490 legacy=0x7f358815e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.914+0000 7f358c82b640 1 -- 172.21.15.45:0/3740768006 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35880b3870 con 0x7f3588071f20 2023-11-28T22:08:29.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.915+0000 7f358c82b640 1 --2- 172.21.15.45:0/3740768006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3588071f20 0x7f35880b1d00 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f35880ae920 tx=0x7f357802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70557bc23745fdc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:29.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.915+0000 7f357ffff640 1 -- 172.21.15.45:0/3740768006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578031830 con 0x7f3588071f20 2023-11-28T22:08:29.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.915+0000 7f357ffff640 1 -- 172.21.15.45:0/3740768006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3578031990 con 0x7f3588071f20 2023-11-28T22:08:29.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.915+0000 7f357ffff640 1 -- 172.21.15.45:0/3740768006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578031d20 con 0x7f3588071f20 2023-11-28T22:08:29.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.916+0000 7f358e02e640 1 -- 172.21.15.45:0/3740768006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3588071f20 msgr2=0x7f35880b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.916+0000 7f358e02e640 1 --2- 172.21.15.45:0/3740768006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3588071f20 0x7f35880b1d00 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f35880ae920 tx=0x7f357802b8d0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.916+0000 7f358e02e640 1 -- 172.21.15.45:0/3740768006 shutdown_connections 2023-11-28T22:08:29.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.916+0000 7f358e02e640 1 --2- 172.21.15.45:0/3740768006 >> v2:172.21.15.190:3300/0 conn(0x7f35880b2240 0x7f3588160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.916+0000 7f358e02e640 1 --2- 172.21.15.45:0/3740768006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3588071f20 0x7f35880b1d00 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.916+0000 7f358e02e640 1 -- 172.21.15.45:0/3740768006 >> 172.21.15.45:0/3740768006 conn(0x7f35880cecd0 msgr2=0x7f35880cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:29.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.916+0000 7f358e02e640 1 -- 172.21.15.45:0/3740768006 shutdown_connections 2023-11-28T22:08:29.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.916+0000 7f358e02e640 1 -- 172.21.15.45:0/3740768006 wait complete. 2023-11-28T22:08:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.918+0000 7f358e02e640 1 Processor -- start 2023-11-28T22:08:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.918+0000 7f358e02e640 1 -- start start 2023-11-28T22:08:29.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.919+0000 7f358e02e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3588071f20 0x7f358809b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.919+0000 7f358e02e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35880b2240 0x7f35881547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.919+0000 7f358e02e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35880b3490 0x7f35880ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.919+0000 7f358e02e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358809b520 con 0x7f35880b2240 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.919+0000 7f358e02e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358809a8a0 con 0x7f3588071f20 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.919+0000 7f358e02e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358809ab80 con 0x7f35880b3490 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.919+0000 7f358c82b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35880b2240 0x7f35881547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.919+0000 7f358c82b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35880b2240 0x7f35881547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49916/0 (socket says 172.21.15.45:49916) 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.919+0000 7f358c82b640 1 -- 172.21.15.45:0/303363714 learned_addr learned my addr 172.21.15.45:0/303363714 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358d02c640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3588071f20 0x7f358809b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358c82b640 1 -- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35880b3490 msgr2=0x7f35880ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358d82d640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35880b3490 0x7f35880ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358c82b640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35880b3490 0x7f35880ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358c82b640 1 -- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3588071f20 msgr2=0x7f358809b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:29.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358c82b640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3588071f20 0x7f358809b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:29.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358c82b640 1 -- 172.21.15.45:0/303363714 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3578009400 con 0x7f35880b2240 2023-11-28T22:08:29.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358d02c640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3588071f20 0x7f358809b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:29.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358d82d640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35880b3490 0x7f35880ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:29.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358c82b640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35880b2240 0x7f35881547b0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f3578009890 tx=0x7f35780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:29.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f357dffb640 1 -- 172.21.15.45:0/303363714 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357803a630 con 0x7f35880b2240 2023-11-28T22:08:29.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f357dffb640 1 -- 172.21.15.45:0/303363714 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357803a7b0 con 0x7f35880b2240 2023-11-28T22:08:29.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f357dffb640 1 -- 172.21.15.45:0/303363714 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357803ab60 con 0x7f35880b2240 2023-11-28T22:08:29.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.920+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358809ae60 con 0x7f35880b2240 2023-11-28T22:08:29.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.921+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3588154cf0 con 0x7f35880b2240 2023-11-28T22:08:29.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.922+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358809b660 con 0x7f35880b2240 2023-11-28T22:08:29.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.923+0000 7f357dffb640 1 -- 172.21.15.45:0/303363714 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f357803acc0 con 0x7f35880b2240 2023-11-28T22:08:29.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.923+0000 7f357dffb640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f356c06af60 0x7f356c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:29.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.924+0000 7f357dffb640 1 -- 172.21.15.45:0/303363714 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 7203+0+0 (secure 0 0 0) 0x7f35780aba90 con 0x7f35880b2240 2023-11-28T22:08:29.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.924+0000 7f358d02c640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f356c06af60 0x7f356c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:29.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.925+0000 7f358d02c640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f356c06af60 0x7f356c06d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f35840011e0 tx=0x7f3584009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:29.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:29.928+0000 7f357dffb640 1 -- 172.21.15.45:0/303363714 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f358809b660 con 0x7f35880b2240 2023-11-28T22:08:30.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.161+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f358809b660 con 0x7f35880b2240 2023-11-28T22:08:30.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.676+0000 7f357dffb640 1 -- 172.21.15.45:0/303363714 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v94) v1 ==== 104+0+0 (secure 0 0 0) 0x7f358809b660 con 0x7f35880b2240 2023-11-28T22:08:30.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f356c06af60 msgr2=0x7f356c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:30.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f356c06af60 0x7f356c06d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f35840011e0 tx=0x7f3584009040 comp rx=0 tx=0).stop 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35880b2240 msgr2=0x7f35881547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35880b2240 0x7f35881547b0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f3578009890 tx=0x7f35780046c0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 shutdown_connections 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35880b3490 0x7f35880ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f356c06af60 0x7f356c06d420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35880b2240 0x7f35881547b0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 --2- 172.21.15.45:0/303363714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3588071f20 0x7f358809b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 >> 172.21.15.45:0/303363714 conn(0x7f35880cecd0 msgr2=0x7f3588070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 shutdown_connections 2023-11-28T22:08:30.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.680+0000 7f358e02e640 1 -- 172.21.15.45:0/303363714 wait complete. 2023-11-28T22:08:30.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T22:08:30.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.936+0000 7fb8e7968640 1 Processor -- start 2023-11-28T22:08:30.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.936+0000 7fb8e7968640 1 -- start start 2023-11-28T22:08:30.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.938+0000 7fb8e7968640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb8e006ac10 0x7fb8e006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:30.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.938+0000 7fb8e7968640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8e015f720 0x7fb8e00717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:30.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.938+0000 7fb8e7968640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e015ecd0 con 0x7fb8e015e840 2023-11-28T22:08:30.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.938+0000 7fb8e7968640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e0071ce0 con 0x7fb8e015f720 2023-11-28T22:08:30.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.938+0000 7fb8e7968640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e015fbd0 con 0x7fb8e006ac10 2023-11-28T22:08:30.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.938+0000 7fb8e5f05640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8e015f720 0x7fb8e00717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:30.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.939+0000 7fb8e5f05640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8e015f720 0x7fb8e00717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49930/0 (socket says 172.21.15.45:49930) 2023-11-28T22:08:30.941 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.939+0000 7fb8e5f05640 1 -- 172.21.15.45:0/4134950768 learned_addr learned my addr 172.21.15.45:0/4134950768 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:30.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.939+0000 7fb8e4f03640 1 --2- 172.21.15.45:0/4134950768 >> v2:172.21.15.190:3300/0 conn(0x7fb8e006ac10 0x7fb8e006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:30.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.939+0000 7fb8e5f05640 1 -- 172.21.15.45:0/4134950768 >> v2:172.21.15.190:3300/0 conn(0x7fb8e006ac10 msgr2=0x7fb8e006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:30.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.939+0000 7fb8e5f05640 1 --2- 172.21.15.45:0/4134950768 >> v2:172.21.15.190:3300/0 conn(0x7fb8e006ac10 0x7fb8e006c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.939+0000 7fb8e5f05640 1 -- 172.21.15.45:0/4134950768 >> v1:172.21.15.190:6789/0 conn(0x7fb8e015e840 legacy=0x7fb8e006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:30.942 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.939+0000 7fb8e5f05640 1 -- 172.21.15.45:0/4134950768 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e015f290 con 0x7fb8e015f720 2023-11-28T22:08:30.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.940+0000 7fb8e5f05640 1 --2- 172.21.15.45:0/4134950768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e015f720 0x7fb8e00717a0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fb8e006e6d0 tx=0x7fb8d802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a8b683622a6568 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:30.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.941+0000 7fb8d7fff640 1 -- 172.21.15.45:0/4134950768 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d802ca80 con 0x7fb8e015f720 2023-11-28T22:08:30.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.941+0000 7fb8d7fff640 1 -- 172.21.15.45:0/4134950768 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d802cbe0 con 0x7fb8e015f720 2023-11-28T22:08:30.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.941+0000 7fb8d7fff640 1 -- 172.21.15.45:0/4134950768 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d80343f0 con 0x7fb8e015f720 2023-11-28T22:08:30.943 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.941+0000 7fb8e7968640 1 -- 172.21.15.45:0/4134950768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e015f720 msgr2=0x7fb8e00717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:30.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.941+0000 7fb8e7968640 1 --2- 172.21.15.45:0/4134950768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e015f720 0x7fb8e00717a0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fb8e006e6d0 tx=0x7fb8d802bee0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.941+0000 7fb8e7968640 1 -- 172.21.15.45:0/4134950768 shutdown_connections 2023-11-28T22:08:30.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.941+0000 7fb8e7968640 1 --2- 172.21.15.45:0/4134950768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e015f720 0x7fb8e00717a0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.941+0000 7fb8e7968640 1 --2- 172.21.15.45:0/4134950768 >> v2:172.21.15.190:3300/0 conn(0x7fb8e006ac10 0x7fb8e006c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.941+0000 7fb8e7968640 1 -- 172.21.15.45:0/4134950768 >> 172.21.15.45:0/4134950768 conn(0x7fb8e00ced60 msgr2=0x7fb8e00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:30.944 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.942+0000 7fb8e7968640 1 -- 172.21.15.45:0/4134950768 shutdown_connections 2023-11-28T22:08:30.945 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.942+0000 7fb8e7968640 1 -- 172.21.15.45:0/4134950768 wait complete. 2023-11-28T22:08:30.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.943+0000 7fb8e7968640 1 Processor -- start 2023-11-28T22:08:30.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.943+0000 7fb8e7968640 1 -- start start 2023-11-28T22:08:30.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.944+0000 7fb8e7968640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e006ac10 0x7fb8e00c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:30.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.944+0000 7fb8e7968640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8e015e840 0x7fb8e00afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:30.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.944+0000 7fb8e7968640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8e015f720 0x7fb8e00aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:30.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.944+0000 7fb8e7968640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01515f0 con 0x7fb8e006ac10 2023-11-28T22:08:30.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.944+0000 7fb8e7968640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e0151890 con 0x7fb8e015e840 2023-11-28T22:08:30.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.944+0000 7fb8e7968640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e0151b70 con 0x7fb8e015f720 2023-11-28T22:08:30.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.945+0000 7fb8e5704640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e006ac10 0x7fb8e00c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:30.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.945+0000 7fb8e5704640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e006ac10 0x7fb8e00c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49932/0 (socket says 172.21.15.45:49932) 2023-11-28T22:08:30.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.945+0000 7fb8e5704640 1 -- 172.21.15.45:0/532524887 learned_addr learned my addr 172.21.15.45:0/532524887 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:30.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.945+0000 7fb8e5704640 1 -- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8e015f720 msgr2=0x7fb8e00aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:30.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.945+0000 7fb8e5704640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8e015f720 0x7fb8e00aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.945+0000 7fb8e5704640 1 -- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8e015e840 msgr2=0x7fb8e00afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.945+0000 7fb8e5704640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8e015e840 0x7fb8e00afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.945+0000 7fb8e5704640 1 -- 172.21.15.45:0/532524887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d8009b30 con 0x7fb8e006ac10 2023-11-28T22:08:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.945+0000 7fb8e5704640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e006ac10 0x7fb8e00c63f0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00c430 tx=0x7fb8dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.946+0000 7fb8d5ffb640 1 -- 172.21.15.45:0/532524887 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc011790 con 0x7fb8e006ac10 2023-11-28T22:08:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.946+0000 7fb8d5ffb640 1 -- 172.21.15.45:0/532524887 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc0118f0 con 0x7fb8e006ac10 2023-11-28T22:08:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.946+0000 7fb8d5ffb640 1 -- 172.21.15.45:0/532524887 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc011ce0 con 0x7fb8e006ac10 2023-11-28T22:08:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.946+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e0151e50 con 0x7fb8e006ac10 2023-11-28T22:08:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.946+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e00c6930 con 0x7fb8e006ac10 2023-11-28T22:08:30.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.947+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e0152370 con 0x7fb8e006ac10 2023-11-28T22:08:30.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.948+0000 7fb8d5ffb640 1 -- 172.21.15.45:0/532524887 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb8dc01c460 con 0x7fb8e006ac10 2023-11-28T22:08:30.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.949+0000 7fb8d5ffb640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb8c406af60 0x7fb8c406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:30.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.949+0000 7fb8d5ffb640 1 -- 172.21.15.45:0/532524887 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 7203+0+0 (secure 0 0 0) 0x7fb8dc089d80 con 0x7fb8e006ac10 2023-11-28T22:08:30.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.950+0000 7fb8e4f03640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb8c406af60 0x7fb8c406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:30.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.951+0000 7fb8e4f03640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb8c406af60 0x7fb8c406d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb8e015df30 tx=0x7fb8d0008040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:30.953+0000 7fb8d5ffb640 1 -- 172.21.15.45:0/532524887 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8e0152370 con 0x7fb8e006ac10 2023-11-28T22:08:31.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.147+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb8e0152370 con 0x7fb8e006ac10 2023-11-28T22:08:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.259+0000 7fb8d5ffb640 1 -- 172.21.15.45:0/532524887 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v206) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb8e0152370 con 0x7fb8e006ac10 2023-11-28T22:08:31.261 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T22:08:31.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.262+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb8c406af60 msgr2=0x7fb8c406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:31.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.262+0000 7fb8e7968640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb8c406af60 0x7fb8c406d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb8e015df30 tx=0x7fb8d0008040 comp rx=0 tx=0).stop 2023-11-28T22:08:31.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.262+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e006ac10 msgr2=0x7fb8e00c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:31.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.263+0000 7fb8e7968640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e006ac10 0x7fb8e00c63f0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00c430 tx=0x7fb8dc00c8f0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.263+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 shutdown_connections 2023-11-28T22:08:31.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.263+0000 7fb8e7968640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8e015f720 0x7fb8e00aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.263+0000 7fb8e7968640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8e015e840 0x7fb8e00afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.263+0000 7fb8e7968640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fb8c406af60 0x7fb8c406d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.263+0000 7fb8e7968640 1 --2- 172.21.15.45:0/532524887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8e006ac10 0x7fb8e00c63f0 secure :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00c430 tx=0x7fb8dc00c8f0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.264+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 >> 172.21.15.45:0/532524887 conn(0x7fb8e00ced60 msgr2=0x7fb8e00cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:31.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.265+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 shutdown_connections 2023-11-28T22:08:31.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.265+0000 7fb8e7968640 1 -- 172.21.15.45:0/532524887 wait complete. 2023-11-28T22:08:31.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T22:08:31.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.501+0000 7fd866ee0640 1 Processor -- start 2023-11-28T22:08:31.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.501+0000 7fd866ee0640 1 -- start start 2023-11-28T22:08:31.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.503+0000 7fd866ee0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd8600c1d30 0x7fd8600ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:31.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.503+0000 7fd866ee0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8601653a0 0x7fd8600c2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.503+0000 7fd866ee0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86015f7e0 con 0x7fd8600712c0 2023-11-28T22:08:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.503+0000 7fd866ee0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8600c2650 con 0x7fd8600c1d30 2023-11-28T22:08:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.503+0000 7fd866ee0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601658a0 con 0x7fd8601653a0 2023-11-28T22:08:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.503+0000 7fd86547d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd8600712c0 0x7fd860162290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37688/0 (socket says 172.21.15.45:37688) 2023-11-28T22:08:31.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.503+0000 7fd86547d640 1 -- 172.21.15.45:0/4219941055 learned_addr learned my addr 172.21.15.45:0/4219941055 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85ffff640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8601653a0 0x7fd8600c2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd864c7c640 1 --2- 172.21.15.45:0/4219941055 >> v2:172.21.15.190:3300/0 conn(0x7fd8600c1d30 0x7fd8600ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747354201 0 0) 0x7fd86015f7e0 con 0x7fd8600712c0 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd848003520 con 0x7fd8600712c0 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3401638953 0 0) 0x7fd848003520 con 0x7fd8600712c0 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8601653a0 msgr2=0x7fd8600c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8601653a0 0x7fd8600c2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 >> v2:172.21.15.190:3300/0 conn(0x7fd8600c1d30 msgr2=0x7fd8600ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 --2- 172.21.15.45:0/4219941055 >> v2:172.21.15.190:3300/0 conn(0x7fd8600c1d30 0x7fd8600ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860165b80 con 0x7fd8600712c0 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd858001670 con 0x7fd8600712c0 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.504+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 >> v1:172.21.15.45:6789/0 conn(0x7fd8600712c0 legacy=0x7fd860162290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85f7fe640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd848003970 0x7fd848003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85f7fe640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8480047c0 0x7fd848004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85f7fe640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8601653a0 0x7fd848005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600c2650 con 0x7fd8480047c0 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601658a0 con 0x7fd848003970 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84800ba00 con 0x7fd8601653a0 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd864c7c640 1 --2- 172.21.15.45:0/4219941055 >> v2:172.21.15.190:3300/0 conn(0x7fd8600c1d30 0x7fd8600ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85ffff640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8480047c0 0x7fd848004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85ffff640 1 -- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8601653a0 msgr2=0x7fd848005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85ffff640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8601653a0 0x7fd848005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85ffff640 1 -- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd848003970 msgr2=0x7fd848003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd864c7c640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd848003970 0x7fd848003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85ffff640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd848003970 0x7fd848003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.505+0000 7fd85ffff640 1 -- 172.21.15.45:0/4219941055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd848003520 con 0x7fd8480047c0 2023-11-28T22:08:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.507+0000 7fd85ffff640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8480047c0 0x7fd848004c40 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fd854009ac0 tx=0x7fd8480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e501d1c224199d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.507+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85402c710 con 0x7fd8480047c0 2023-11-28T22:08:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.507+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85402c870 con 0x7fd8480047c0 2023-11-28T22:08:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.507+0000 7fd85f7fe640 1 -- 172.21.15.45:0/4219941055 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85402cc00 con 0x7fd8480047c0 2023-11-28T22:08:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.507+0000 7fd866ee0640 1 -- 172.21.15.45:0/4219941055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8480047c0 msgr2=0x7fd848004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.507+0000 7fd866ee0640 1 --2- 172.21.15.45:0/4219941055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8480047c0 0x7fd848004c40 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fd854009ac0 tx=0x7fd8480057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.507+0000 7fd86547d640 1 -- 172.21.15.45:0/4219941055 reap_dead start 2023-11-28T22:08:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.507+0000 7fd866ee0640 1 -- 172.21.15.45:0/4219941055 shutdown_connections 2023-11-28T22:08:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.507+0000 7fd866ee0640 1 -- 172.21.15.45:0/4219941055 >> 172.21.15.45:0/4219941055 conn(0x7fd8600ced60 msgr2=0x7fd8600cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:31.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.508+0000 7fd866ee0640 1 -- 172.21.15.45:0/4219941055 shutdown_connections 2023-11-28T22:08:31.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.508+0000 7fd866ee0640 1 -- 172.21.15.45:0/4219941055 wait complete. 2023-11-28T22:08:31.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.509+0000 7fd866ee0640 1 Processor -- start 2023-11-28T22:08:31.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.510+0000 7fd866ee0640 1 -- start start 2023-11-28T22:08:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.510+0000 7fd866ee0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8480047c0 0x7fd8600abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:31.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.510+0000 7fd866ee0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8600ae5a0 0x7fd8600ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:31.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.510+0000 7fd866ee0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd860160370 0x7fd86015d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:31.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.510+0000 7fd866ee0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600adc30 con 0x7fd8600ae5a0 2023-11-28T22:08:31.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.510+0000 7fd866ee0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600acd70 con 0x7fd8480047c0 2023-11-28T22:08:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.510+0000 7fd866ee0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601595e0 con 0x7fd860160370 2023-11-28T22:08:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd85ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8600ae5a0 0x7fd8600ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd85ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8600ae5a0 0x7fd8600ad300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49962/0 (socket says 172.21.15.45:49962) 2023-11-28T22:08:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd85ffff640 1 -- 172.21.15.45:0/2826610630 learned_addr learned my addr 172.21.15.45:0/2826610630 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd86547d640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd860160370 0x7fd86015d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd864c7c640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8480047c0 0x7fd8600abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd85ffff640 1 -- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd860160370 msgr2=0x7fd86015d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd85ffff640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd860160370 0x7fd86015d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd85ffff640 1 -- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8480047c0 msgr2=0x7fd8600abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd85ffff640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8480047c0 0x7fd8600abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd85ffff640 1 -- 172.21.15.45:0/2826610630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8601598c0 con 0x7fd8600ae5a0 2023-11-28T22:08:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.511+0000 7fd864c7c640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8480047c0 0x7fd8600abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:31.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.512+0000 7fd85ffff640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8600ae5a0 0x7fd8600ad300 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fd85402c2b0 tx=0x7fd854004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.512+0000 7fd85d7fa640 1 -- 172.21.15.45:0/2826610630 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd854003810 con 0x7fd8600ae5a0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.512+0000 7fd85d7fa640 1 -- 172.21.15.45:0/2826610630 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85402de90 con 0x7fd8600ae5a0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.512+0000 7fd85d7fa640 1 -- 172.21.15.45:0/2826610630 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd854003bb0 con 0x7fd8600ae5a0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.512+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd860159ba0 con 0x7fd8600ae5a0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.512+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8600ad840 con 0x7fd8600ae5a0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.513+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860159720 con 0x7fd8600ae5a0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.514+0000 7fd85d7fa640 1 -- 172.21.15.45:0/2826610630 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd854003d10 con 0x7fd8600ae5a0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.515+0000 7fd85d7fa640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd84c06af90 0x7fd84c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.515+0000 7fd85d7fa640 1 -- 172.21.15.45:0/2826610630 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6765+0+0 (secure 0 0 0) 0x7fd8540abc90 con 0x7fd8600ae5a0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.516+0000 7fd864c7c640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd84c06af90 0x7fd84c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.517+0000 7fd864c7c640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd84c06af90 0x7fd84c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8500094c0 tx=0x7fd85000b040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:31.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.520+0000 7fd85d7fa640 1 -- 172.21.15.45:0/2826610630 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd860159720 con 0x7fd8600ae5a0 2023-11-28T22:08:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:31.715+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd860159720 con 0x7fd8600ae5a0 2023-11-28T22:08:32.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.603+0000 7fd85d7fa640 1 -- 172.21.15.45:0/2826610630 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v207) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd860159720 con 0x7fd8600ae5a0 2023-11-28T22:08:32.606 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T22:08:32.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.607+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd84c06af90 msgr2=0x7fd84c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:32.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.607+0000 7fd866ee0640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd84c06af90 0x7fd84c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8500094c0 tx=0x7fd85000b040 comp rx=0 tx=0).stop 2023-11-28T22:08:32.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.607+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8600ae5a0 msgr2=0x7fd8600ad300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:32.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.607+0000 7fd866ee0640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8600ae5a0 0x7fd8600ad300 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fd85402c2b0 tx=0x7fd854004730 comp rx=0 tx=0).stop 2023-11-28T22:08:32.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.609+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 shutdown_connections 2023-11-28T22:08:32.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.609+0000 7fd866ee0640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd860160370 0x7fd86015d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.609+0000 7fd866ee0640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd84c06af90 0x7fd84c06d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.609+0000 7fd866ee0640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8600ae5a0 0x7fd8600ad300 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.609+0000 7fd866ee0640 1 --2- 172.21.15.45:0/2826610630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8480047c0 0x7fd8600abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.609+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 >> 172.21.15.45:0/2826610630 conn(0x7fd8600ced60 msgr2=0x7fd8600cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:32.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.611+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 shutdown_connections 2023-11-28T22:08:32.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.611+0000 7fd866ee0640 1 -- 172.21.15.45:0/2826610630 wait complete. 2023-11-28T22:08:32.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T22:08:32.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.851+0000 7f1f15197640 1 Processor -- start 2023-11-28T22:08:32.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.851+0000 7f1f15197640 1 -- start start 2023-11-28T22:08:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.852+0000 7f1f15197640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1f1006dc50 0x7f1f100cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.852+0000 7f1f15197640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f1006ac10 0x7f1f100c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.852+0000 7f1f15197640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1006e030 con 0x7f1f100ce600 2023-11-28T22:08:32.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.852+0000 7f1f15197640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f100ccf60 con 0x7f1f1006dc50 2023-11-28T22:08:32.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.852+0000 7f1f15197640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f100cd930 con 0x7f1f1006ac10 2023-11-28T22:08:32.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.853+0000 7f1f0e59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1f100ce600 0x7f1f1006a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37696/0 (socket says 172.21.15.45:37696) 2023-11-28T22:08:32.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.853+0000 7f1f0e59c640 1 -- 172.21.15.45:0/1023009444 learned_addr learned my addr 172.21.15.45:0/1023009444 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:32.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.853+0000 7f1f0ed9d640 1 --2- 172.21.15.45:0/1023009444 >> v2:172.21.15.45:3300/0 conn(0x7f1f1006dc50 0x7f1f100cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:32.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.853+0000 7f1f0f59e640 1 --2- 172.21.15.45:0/1023009444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f1006ac10 0x7f1f100c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.853+0000 7f1f0ed9d640 1 -- 172.21.15.45:0/1023009444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f1006ac10 msgr2=0x7f1f100c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.853+0000 7f1f0ed9d640 1 --2- 172.21.15.45:0/1023009444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f1006ac10 0x7f1f100c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.853+0000 7f1f0ed9d640 1 -- 172.21.15.45:0/1023009444 >> v1:172.21.15.45:6789/0 conn(0x7f1f100ce600 legacy=0x7f1f1006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.853+0000 7f1f0ed9d640 1 -- 172.21.15.45:0/1023009444 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f100cc930 con 0x7f1f1006dc50 2023-11-28T22:08:32.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.855+0000 7f1f0ed9d640 1 --2- 172.21.15.45:0/1023009444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006dc50 0x7f1f100cd3f0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f1f000097a0 tx=0x7f1f0002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f211e944a38f45bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.855+0000 7f1f0dd9b640 1 -- 172.21.15.45:0/1023009444 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0002d040 con 0x7f1f1006dc50 2023-11-28T22:08:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.855+0000 7f1f0dd9b640 1 -- 172.21.15.45:0/1023009444 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f00031810 con 0x7f1f1006dc50 2023-11-28T22:08:32.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.855+0000 7f1f0dd9b640 1 -- 172.21.15.45:0/1023009444 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00031ba0 con 0x7f1f1006dc50 2023-11-28T22:08:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.855+0000 7f1f15197640 1 -- 172.21.15.45:0/1023009444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006dc50 msgr2=0x7f1f100cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.855+0000 7f1f15197640 1 --2- 172.21.15.45:0/1023009444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006dc50 0x7f1f100cd3f0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f1f000097a0 tx=0x7f1f0002b8b0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.856+0000 7f1f15197640 1 -- 172.21.15.45:0/1023009444 shutdown_connections 2023-11-28T22:08:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.856+0000 7f1f15197640 1 --2- 172.21.15.45:0/1023009444 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f1006ac10 0x7f1f100c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.856+0000 7f1f15197640 1 --2- 172.21.15.45:0/1023009444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006dc50 0x7f1f100cd3f0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.856+0000 7f1f15197640 1 -- 172.21.15.45:0/1023009444 >> 172.21.15.45:0/1023009444 conn(0x7f1f100ced60 msgr2=0x7f1f100cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.856+0000 7f1f15197640 1 -- 172.21.15.45:0/1023009444 shutdown_connections 2023-11-28T22:08:32.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.856+0000 7f1f15197640 1 -- 172.21.15.45:0/1023009444 wait complete. 2023-11-28T22:08:32.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.857+0000 7f1f15197640 1 Processor -- start 2023-11-28T22:08:32.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.858+0000 7f1f15197640 1 -- start start 2023-11-28T22:08:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f15197640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006ac10 0x7f1f101502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f15197640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f1006dc50 0x7f1f100c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f15197640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f100ce600 0x7f1f100c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f15197640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f100c5e30 con 0x7f1f1006ac10 2023-11-28T22:08:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f15197640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f101507e0 con 0x7f1f1006dc50 2023-11-28T22:08:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f15197640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f100c6010 con 0x7f1f100ce600 2023-11-28T22:08:32.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f0ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006ac10 0x7f1f101502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:32.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f0ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006ac10 0x7f1f101502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49974/0 (socket says 172.21.15.45:49974) 2023-11-28T22:08:32.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f0ed9d640 1 -- 172.21.15.45:0/3692195727 learned_addr learned my addr 172.21.15.45:0/3692195727 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f0f59e640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f100ce600 0x7f1f100c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f0e59c640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f1006dc50 0x7f1f100c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f0ed9d640 1 -- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f100ce600 msgr2=0x7f1f100c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f0ed9d640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f100ce600 0x7f1f100c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f0ed9d640 1 -- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f1006dc50 msgr2=0x7f1f100c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:32.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.859+0000 7f1f0ed9d640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f1006dc50 0x7f1f100c4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.860+0000 7f1f0ed9d640 1 -- 172.21.15.45:0/3692195727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00009400 con 0x7f1f1006ac10 2023-11-28T22:08:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.860+0000 7f1f0f59e640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f100ce600 0x7f1f100c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.860+0000 7f1f0ed9d640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006ac10 0x7f1f101502a0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f1f000096f0 tx=0x7f1f000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.860+0000 7f1ef77fe640 1 -- 172.21.15.45:0/3692195727 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0003ad60 con 0x7f1f1006ac10 2023-11-28T22:08:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.860+0000 7f1ef77fe640 1 -- 172.21.15.45:0/3692195727 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f00044400 con 0x7f1f1006ac10 2023-11-28T22:08:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.860+0000 7f1ef77fe640 1 -- 172.21.15.45:0/3692195727 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f000447b0 con 0x7f1f1006ac10 2023-11-28T22:08:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.860+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f100c62f0 con 0x7f1f1006ac10 2023-11-28T22:08:32.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.860+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f100c68b0 con 0x7f1f1006ac10 2023-11-28T22:08:32.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.862+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f100733d0 con 0x7f1f1006ac10 2023-11-28T22:08:32.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.863+0000 7f1ef77fe640 1 -- 172.21.15.45:0/3692195727 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1f0004d440 con 0x7f1f1006ac10 2023-11-28T22:08:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.864+0000 7f1ef77fe640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1efc06af60 0x7f1efc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.864+0000 7f1ef77fe640 1 -- 172.21.15.45:0/3692195727 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7f1f000abfc0 con 0x7f1f1006ac10 2023-11-28T22:08:32.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.865+0000 7f1f0e59c640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1efc06af60 0x7f1efc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:32.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.866+0000 7f1f0e59c640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1efc06af60 0x7f1efc06d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1f100ce490 tx=0x7f1f04009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:32.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:32.869+0000 7f1ef77fe640 1 -- 172.21.15.45:0/3692195727 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f00076ab0 con 0x7f1f1006ac10 2023-11-28T22:08:33.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.076+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f1f100caf40 con 0x7f1f1006ac10 2023-11-28T22:08:33.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.082+0000 7f1ef77fe640 1 -- 172.21.15.45:0/3692195727 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v67) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1f0007a0d0 con 0x7f1f1006ac10 2023-11-28T22:08:33.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1efc06af60 msgr2=0x7f1efc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1efc06af60 0x7f1efc06d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1f100ce490 tx=0x7f1f04009040 comp rx=0 tx=0).stop 2023-11-28T22:08:33.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006ac10 msgr2=0x7f1f101502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006ac10 0x7f1f101502a0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f1f000096f0 tx=0x7f1f000046c0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 shutdown_connections 2023-11-28T22:08:33.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f100ce600 0x7f1f100c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f1006dc50 0x7f1f100c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f1efc06af60 0x7f1efc06d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.089 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 --2- 172.21.15.45:0/3692195727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f1006ac10 0x7f1f101502a0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 >> 172.21.15.45:0/3692195727 conn(0x7f1f100ced60 msgr2=0x7f1f10160ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:33.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.085+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 shutdown_connections 2023-11-28T22:08:33.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.086+0000 7f1f15197640 1 -- 172.21.15.45:0/3692195727 wait complete. 2023-11-28T22:08:33.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T22:08:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.329+0000 7fd15e388640 1 Processor -- start 2023-11-28T22:08:33.332 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.330+0000 7fd15e388640 1 -- start start 2023-11-28T22:08:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.331+0000 7fd15e388640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd158071100 0x7fd15815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.332+0000 7fd15e388640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15815ea30 con 0x7fd1580ad900 2023-11-28T22:08:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.332+0000 7fd15e388640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15815de20 con 0x7fd1580ad520 2023-11-28T22:08:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.332+0000 7fd15e388640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15815e100 con 0x7fd158071100 2023-11-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.332+0000 7fd15cb85640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd1580ad900 0x7fd1580709f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37712/0 (socket says 172.21.15.45:37712) 2023-11-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.332+0000 7fd15cb85640 1 -- 172.21.15.45:0/2168163448 learned_addr learned my addr 172.21.15.45:0/2168163448 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.332+0000 7fd15db87640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd158071100 0x7fd15815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.332+0000 7fd15db87640 1 -- 172.21.15.45:0/2168163448 >> v1:172.21.15.190:6789/0 conn(0x7fd1580ad520 legacy=0x7fd1580ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.332+0000 7fd15db87640 1 -- 172.21.15.45:0/2168163448 >> v1:172.21.15.45:6789/0 conn(0x7fd1580ad900 legacy=0x7fd1580709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.333+0000 7fd15db87640 1 -- 172.21.15.45:0/2168163448 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15815e3e0 con 0x7fd158071100 2023-11-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.333+0000 7fd14ffff640 1 -- 172.21.15.45:0/2168163448 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969669323 0 0) 0x7fd15815ea30 con 0x7fd1580ad900 2023-11-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.334+0000 7fd15db87640 1 --2- 172.21.15.45:0/2168163448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd158071100 0x7fd15815d7d0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fd150009d90 tx=0x7fd15002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5a492b918e3c02 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.334+0000 7fd14ffff640 1 -- 172.21.15.45:0/2168163448 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15002cc80 con 0x7fd158071100 2023-11-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.334+0000 7fd14ffff640 1 -- 172.21.15.45:0/2168163448 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15002cde0 con 0x7fd158071100 2023-11-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.334+0000 7fd14ffff640 1 -- 172.21.15.45:0/2168163448 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150034660 con 0x7fd158071100 2023-11-28T22:08:33.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.335+0000 7fd15e388640 1 -- 172.21.15.45:0/2168163448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd158071100 msgr2=0x7fd15815d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.335+0000 7fd15e388640 1 --2- 172.21.15.45:0/2168163448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd158071100 0x7fd15815d7d0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fd150009d90 tx=0x7fd15002bed0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.335+0000 7fd15e388640 1 -- 172.21.15.45:0/2168163448 shutdown_connections 2023-11-28T22:08:33.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.335+0000 7fd15e388640 1 --2- 172.21.15.45:0/2168163448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd158071100 0x7fd15815d7d0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.335+0000 7fd15e388640 1 -- 172.21.15.45:0/2168163448 >> 172.21.15.45:0/2168163448 conn(0x7fd1580b17a0 msgr2=0x7fd1580cf4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:33.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.335+0000 7fd15e388640 1 -- 172.21.15.45:0/2168163448 shutdown_connections 2023-11-28T22:08:33.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.335+0000 7fd15e388640 1 -- 172.21.15.45:0/2168163448 wait complete. 2023-11-28T22:08:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.337+0000 7fd15e388640 1 Processor -- start 2023-11-28T22:08:33.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.337+0000 7fd15e388640 1 -- start start 2023-11-28T22:08:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.338+0000 7fd15e388640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd158071100 0x7fd158146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.338+0000 7fd15e388640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1580ad520 0x7fd15806ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.338+0000 7fd15e388640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1580ad900 0x7fd15806cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.338+0000 7fd15e388640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580b3c30 con 0x7fd1580ad900 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.338+0000 7fd15e388640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd158146970 con 0x7fd1580ad520 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.338+0000 7fd15e388640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580b3080 con 0x7fd158071100 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.338+0000 7fd15db87640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1580ad900 0x7fd15806cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.338+0000 7fd15db87640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1580ad900 0x7fd15806cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49992/0 (socket says 172.21.15.45:49992) 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.338+0000 7fd15db87640 1 -- 172.21.15.45:0/2408040086 learned_addr learned my addr 172.21.15.45:0/2408040086 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd15cb85640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1580ad520 0x7fd15806ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd15d386640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd158071100 0x7fd158146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd15db87640 1 -- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd158071100 msgr2=0x7fd158146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd15db87640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd158071100 0x7fd158146430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd15db87640 1 -- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1580ad520 msgr2=0x7fd15806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd15db87640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1580ad520 0x7fd15806ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd15db87640 1 -- 172.21.15.45:0/2408040086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd150009a70 con 0x7fd1580ad900 2023-11-28T22:08:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd15cb85640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1580ad520 0x7fd15806ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:33.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd15db87640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1580ad900 0x7fd15806cf80 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fd150037040 tx=0x7fd150004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.339+0000 7fd14dffb640 1 -- 172.21.15.45:0/2408040086 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15003e610 con 0x7fd1580ad900 2023-11-28T22:08:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.340+0000 7fd14dffb640 1 -- 172.21.15.45:0/2408040086 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15003e790 con 0x7fd1580ad900 2023-11-28T22:08:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.340+0000 7fd14dffb640 1 -- 172.21.15.45:0/2408040086 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15003eb40 con 0x7fd1580ad900 2023-11-28T22:08:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.340+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1580b40c0 con 0x7fd1580ad900 2023-11-28T22:08:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.340+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1580b35e0 con 0x7fd1580ad900 2023-11-28T22:08:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.341+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1580b3de0 con 0x7fd1580ad900 2023-11-28T22:08:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.342+0000 7fd14dffb640 1 -- 172.21.15.45:0/2408040086 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd15003b050 con 0x7fd1580ad900 2023-11-28T22:08:33.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.343+0000 7fd14dffb640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd13c06af60 0x7fd13c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.343+0000 7fd14dffb640 1 -- 172.21.15.45:0/2408040086 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7fd1500ac520 con 0x7fd1580ad900 2023-11-28T22:08:33.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.343+0000 7fd15d386640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd13c06af60 0x7fd13c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.344+0000 7fd15d386640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd13c06af60 0x7fd13c06d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd154000f60 tx=0x7fd154009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:33.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.347+0000 7fd14dffb640 1 -- 172.21.15.45:0/2408040086 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1580b3de0 con 0x7fd1580ad900 2023-11-28T22:08:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.542+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd1580b3de0 con 0x7fd1580ad900 2023-11-28T22:08:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.548+0000 7fd14dffb640 1 -- 172.21.15.45:0/2408040086 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd1580b3de0 con 0x7fd1580ad900 2023-11-28T22:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd13c06af60 msgr2=0x7fd13c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd13c06af60 0x7fd13c06d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd154000f60 tx=0x7fd154009040 comp rx=0 tx=0).stop 2023-11-28T22:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1580ad900 msgr2=0x7fd15806cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1580ad900 0x7fd15806cf80 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fd150037040 tx=0x7fd150004a40 comp rx=0 tx=0).stop 2023-11-28T22:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 shutdown_connections 2023-11-28T22:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd13c06af60 0x7fd13c06d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1580ad900 0x7fd15806cf80 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1580ad520 0x7fd15806ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 --2- 172.21.15.45:0/2408040086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd158071100 0x7fd158146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 >> 172.21.15.45:0/2408040086 conn(0x7fd1580b17a0 msgr2=0x7fd15815b280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 shutdown_connections 2023-11-28T22:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.551+0000 7fd15e388640 1 -- 172.21.15.45:0/2408040086 wait complete. 2023-11-28T22:08:33.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T22:08:33.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.800+0000 7f4e2982d640 1 Processor -- start 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.800+0000 7f4e2982d640 1 -- start start 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.802+0000 7f4e2982d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e2406c350 0x7f4e2415f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.802+0000 7f4e2982d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2415ecd0 con 0x7f4e2415e840 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.802+0000 7f4e2982d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2415f290 con 0x7f4e2406ac10 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.802+0000 7f4e2982d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e24072020 con 0x7f4e2406c350 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.802+0000 7f4e22ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4e2415e840 0x7f4e2406a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37718/0 (socket says 172.21.15.45:37718) 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.802+0000 7f4e22ffd640 1 -- 172.21.15.45:0/961162493 learned_addr learned my addr 172.21.15.45:0/961162493 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.802+0000 7f4e237fe640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e2406c350 0x7f4e2415f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.802+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451275178 0 0) 0x7f4e2415ecd0 con 0x7f4e2415e840 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.802+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c003520 con 0x7f4e2415e840 2023-11-28T22:08:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3843403343 0 0) 0x7f4e0c003520 con 0x7f4e2415e840 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e2406c350 msgr2=0x7f4e2415f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e2406c350 0x7f4e2415f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 >> v1:172.21.15.190:6789/0 conn(0x7f4e2406ac10 legacy=0x7f4e240cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24072300 con 0x7f4e2415e840 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4e140019f0 con 0x7f4e2415e840 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 >> v1:172.21.15.45:6789/0 conn(0x7f4e2415e840 legacy=0x7f4e2406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e0c0039a0 0x7f4e0c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e2406c350 0x7f4e0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e0c004d00 0x7f4e0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e24072020 con 0x7f4e0c0039a0 2023-11-28T22:08:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c00b9b0 con 0x7f4e0c004d00 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c00bb70 con 0x7f4e2406c350 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.803+0000 7f4e22ffd640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e0c0039a0 0x7f4e0c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.804+0000 7f4e237fe640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e0c004d00 0x7f4e0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.804+0000 7f4e22ffd640 1 -- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e2406c350 msgr2=0x7f4e0c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.804+0000 7f4e22ffd640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e2406c350 0x7f4e0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.804+0000 7f4e22ffd640 1 -- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e0c004d00 msgr2=0x7f4e0c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.804+0000 7f4e22ffd640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e0c004d00 0x7f4e0c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.804+0000 7f4e22ffd640 1 -- 172.21.15.45:0/961162493 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c003520 con 0x7f4e0c0039a0 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.804+0000 7f4e237fe640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e0c004d00 0x7f4e0c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.805+0000 7f4e22ffd640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e0c0039a0 0x7f4e0c003d80 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400a830 tx=0x7f4e0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247ea57b5b4a6f49 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.805+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14034630 con 0x7f4e0c0039a0 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.805+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1402f070 con 0x7f4e0c0039a0 2023-11-28T22:08:33.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.805+0000 7f4e21ffb640 1 -- 172.21.15.45:0/961162493 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14034a70 con 0x7f4e0c0039a0 2023-11-28T22:08:33.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.806+0000 7f4e2982d640 1 -- 172.21.15.45:0/961162493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e0c0039a0 msgr2=0x7f4e0c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.806+0000 7f4e2982d640 1 --2- 172.21.15.45:0/961162493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e0c0039a0 0x7f4e0c003d80 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400a830 tx=0x7f4e0c0057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.806+0000 7f4e237fe640 1 -- 172.21.15.45:0/961162493 reap_dead start 2023-11-28T22:08:33.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.806+0000 7f4e2982d640 1 -- 172.21.15.45:0/961162493 shutdown_connections 2023-11-28T22:08:33.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.806+0000 7f4e2982d640 1 -- 172.21.15.45:0/961162493 >> 172.21.15.45:0/961162493 conn(0x7f4e240ced60 msgr2=0x7f4e240cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:33.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.806+0000 7f4e2982d640 1 -- 172.21.15.45:0/961162493 shutdown_connections 2023-11-28T22:08:33.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.806+0000 7f4e2982d640 1 -- 172.21.15.45:0/961162493 wait complete. 2023-11-28T22:08:33.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.808+0000 7f4e2982d640 1 Processor -- start 2023-11-28T22:08:33.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.808+0000 7f4e2982d640 1 -- start start 2023-11-28T22:08:33.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.809+0000 7f4e2982d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e0c0039a0 0x7f4e240b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.809+0000 7f4e2982d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e241f6340 0x7f4e240b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.809+0000 7f4e2982d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e240b37e0 0x7f4e240c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.809+0000 7f4e2982d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e241f7270 con 0x7f4e241f6340 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.809+0000 7f4e2982d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e241f6cb0 con 0x7f4e240b37e0 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.809+0000 7f4e2982d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e241f6f90 con 0x7f4e0c0039a0 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.809+0000 7f4e227fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e241f6340 0x7f4e240b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.809+0000 7f4e227fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e241f6340 0x7f4e240b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50008/0 (socket says 172.21.15.45:50008) 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.809+0000 7f4e227fc640 1 -- 172.21.15.45:0/1076966763 learned_addr learned my addr 172.21.15.45:0/1076966763 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e22ffd640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e0c0039a0 0x7f4e240b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e237fe640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e240b37e0 0x7f4e240c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e227fc640 1 -- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e0c0039a0 msgr2=0x7f4e240b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e227fc640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e0c0039a0 0x7f4e240b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e227fc640 1 -- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e240b37e0 msgr2=0x7f4e240c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:33.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e227fc640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e240b37e0 0x7f4e240c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:33.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e227fc640 1 -- 172.21.15.45:0/1076966763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24158be0 con 0x7f4e241f6340 2023-11-28T22:08:33.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e227fc640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e241f6340 0x7f4e240b32a0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800c780 tx=0x7f4e1800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:33.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e0bfff640 1 -- 172.21.15.45:0/1076966763 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18005270 con 0x7f4e241f6340 2023-11-28T22:08:33.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e0bfff640 1 -- 172.21.15.45:0/1076966763 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e180053d0 con 0x7f4e241f6340 2023-11-28T22:08:33.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.810+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e24158ec0 con 0x7f4e241f6340 2023-11-28T22:08:33.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.811+0000 7f4e0bfff640 1 -- 172.21.15.45:0/1076966763 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1800a880 con 0x7f4e241f6340 2023-11-28T22:08:33.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.811+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e240b1a90 con 0x7f4e241f6340 2023-11-28T22:08:33.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.812+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e240733d0 con 0x7f4e241f6340 2023-11-28T22:08:33.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.813+0000 7f4e0bfff640 1 -- 172.21.15.45:0/1076966763 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4e1800a9e0 con 0x7f4e241f6340 2023-11-28T22:08:33.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.814+0000 7f4e0bfff640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:33.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.814+0000 7f4e0bfff640 1 -- 172.21.15.45:0/1076966763 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7f4e18088f50 con 0x7f4e241f6340 2023-11-28T22:08:33.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.814+0000 7f4e22ffd640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:33.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.816+0000 7f4e22ffd640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4e1006af90 0x7f4e1006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e14034000 tx=0x7f4e1402f000 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:33.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:33.818+0000 7f4e0bfff640 1 -- 172.21.15.45:0/1076966763 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e18053d10 con 0x7f4e241f6340 2023-11-28T22:08:34.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.014+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f4e240ce2b0 con 0x7f4e241f6340 2023-11-28T22:08:34.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.020+0000 7f4e0bfff640 1 -- 172.21.15.45:0/1076966763 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v69) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4e18017070 con 0x7f4e241f6340 2023-11-28T22:08:34.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.022+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4e1006af90 msgr2=0x7f4e1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.022+0000 7f4e2982d640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4e1006af90 0x7f4e1006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e14034000 tx=0x7f4e1402f000 comp rx=0 tx=0).stop 2023-11-28T22:08:34.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e241f6340 msgr2=0x7f4e240b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e241f6340 0x7f4e240b32a0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800c780 tx=0x7f4e1800cc40 comp rx=0 tx=0).stop 2023-11-28T22:08:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 shutdown_connections 2023-11-28T22:08:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e240b37e0 0x7f4e240c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e241f6340 0x7f4e240b32a0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 --2- 172.21.15.45:0/1076966763 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e0c0039a0 0x7f4e240b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 >> 172.21.15.45:0/1076966763 conn(0x7f4e240ced60 msgr2=0x7f4e240adfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 shutdown_connections 2023-11-28T22:08:34.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.023+0000 7f4e2982d640 1 -- 172.21.15.45:0/1076966763 wait complete. 2023-11-28T22:08:34.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-11-28T22:08:34.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.261+0000 7f5cec871640 1 Processor -- start 2023-11-28T22:08:34.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.262+0000 7f5cec871640 1 -- start start 2023-11-28T22:08:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.263+0000 7f5cec871640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5ce8154350 0x7f5ce80cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.263+0000 7f5cec871640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80cf520 con 0x7f5ce81614a0 2023-11-28T22:08:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.263+0000 7f5cec871640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80cd220 con 0x7f5ce806d5d0 2023-11-28T22:08:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.263+0000 7f5cec871640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce80cd500 con 0x7f5ce8154350 2023-11-28T22:08:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.264+0000 7f5ce7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5ce81614a0 0x7f5ce80b48c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51912/0 (socket says 172.21.15.45:51912) 2023-11-28T22:08:34.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.264+0000 7f5ce7fff640 1 -- 172.21.15.45:0/2350011545 learned_addr learned my addr 172.21.15.45:0/2350011545 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.264+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618403511 0 0) 0x7f5ce80cf520 con 0x7f5ce81614a0 2023-11-28T22:08:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.264+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd4003520 con 0x7f5ce81614a0 2023-11-28T22:08:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.264+0000 7f5ce77fe640 1 --2- 172.21.15.45:0/2350011545 >> v2:172.21.15.190:3300/0 conn(0x7f5ce8154350 0x7f5ce80cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1120977408 0 0) 0x7f5cd4003520 con 0x7f5ce81614a0 2023-11-28T22:08:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 >> v2:172.21.15.190:3300/0 conn(0x7f5ce8154350 msgr2=0x7f5ce80cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 --2- 172.21.15.45:0/2350011545 >> v2:172.21.15.190:3300/0 conn(0x7f5ce8154350 0x7f5ce80cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 >> v1:172.21.15.190:6789/0 conn(0x7f5ce806d5d0 legacy=0x7f5ce8160d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce80cd7e0 con 0x7f5ce81614a0 2023-11-28T22:08:34.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f5ce0001670 con 0x7f5ce81614a0 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 >> v1:172.21.15.45:6789/0 conn(0x7f5ce81614a0 legacy=0x7f5ce80b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd4003970 0x7f5cd4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd40047c0 0x7f5cd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cd40052a0 0x7f5cd400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80cd500 con 0x7f5cd4003970 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd400c170 con 0x7f5cd40047c0 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd400c330 con 0x7f5cd40052a0 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce77fe640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd4003970 0x7f5cd4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.265+0000 7f5ce6ffd640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd40047c0 0x7f5cd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.266+0000 7f5ce77fe640 1 -- 172.21.15.45:0/2350011545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cd40052a0 msgr2=0x7f5cd400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.266+0000 7f5ce77fe640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5cd40052a0 0x7f5cd400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.266+0000 7f5ce77fe640 1 -- 172.21.15.45:0/2350011545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd40047c0 msgr2=0x7f5cd4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:34.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.266+0000 7f5ce77fe640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd40047c0 0x7f5cd4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.266+0000 7f5ce77fe640 1 -- 172.21.15.45:0/2350011545 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4003520 con 0x7f5cd4003970 2023-11-28T22:08:34.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.266+0000 7f5ce6ffd640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd40047c0 0x7f5cd4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:34.270 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.266+0000 7f5ce7fff640 1 -- 172.21.15.45:0/2350011545 reap_dead start 2023-11-28T22:08:34.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.267+0000 7f5ce77fe640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd4003970 0x7f5cd4003d90 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8007330 tx=0x7f5cd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea462e09a1dbedb0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:34.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.267+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd802cc70 con 0x7f5cd4003970 2023-11-28T22:08:34.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.267+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd802cdd0 con 0x7f5cd4003970 2023-11-28T22:08:34.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.267+0000 7f5ce67fc640 1 -- 172.21.15.45:0/2350011545 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8034660 con 0x7f5cd4003970 2023-11-28T22:08:34.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.268+0000 7f5cec871640 1 -- 172.21.15.45:0/2350011545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd4003970 msgr2=0x7f5cd4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.268+0000 7f5cec871640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd4003970 0x7f5cd4003d90 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8007330 tx=0x7f5cd40057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.268+0000 7f5cec871640 1 -- 172.21.15.45:0/2350011545 shutdown_connections 2023-11-28T22:08:34.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.268+0000 7f5cec871640 1 --2- 172.21.15.45:0/2350011545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd4003970 0x7f5cd4003d90 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.268+0000 7f5cec871640 1 -- 172.21.15.45:0/2350011545 >> 172.21.15.45:0/2350011545 conn(0x7f5ce806a500 msgr2=0x7f5ce806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:34.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.268+0000 7f5cec871640 1 -- 172.21.15.45:0/2350011545 shutdown_connections 2023-11-28T22:08:34.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.268+0000 7f5cec871640 1 -- 172.21.15.45:0/2350011545 wait complete. 2023-11-28T22:08:34.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.270+0000 7f5cec871640 1 Processor -- start 2023-11-28T22:08:34.272 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.270+0000 7f5cec871640 1 -- start start 2023-11-28T22:08:34.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.271+0000 7f5cec871640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd4003970 0x7f5ce8146e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.271+0000 7f5cec871640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce8156cf0 0x7f5ce81596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.271+0000 7f5cec871640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ce8159bf0 0x7f5ce81551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.273 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.271+0000 7f5cec871640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8147360 con 0x7f5ce8156cf0 2023-11-28T22:08:34.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.271+0000 7f5cec871640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce815a070 con 0x7f5cd4003970 2023-11-28T22:08:34.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.271+0000 7f5cec871640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8156150 con 0x7f5ce8159bf0 2023-11-28T22:08:34.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.271+0000 7f5ce6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce8156cf0 0x7f5ce81596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.271+0000 7f5ce6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce8156cf0 0x7f5ce81596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37766/0 (socket says 172.21.15.45:37766) 2023-11-28T22:08:34.274 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.271+0000 7f5ce6ffd640 1 -- 172.21.15.45:0/2909747712 learned_addr learned my addr 172.21.15.45:0/2909747712 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:34.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5ce77fe640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd4003970 0x7f5ce8146e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5ce7fff640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ce8159bf0 0x7f5ce81551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5ce6ffd640 1 -- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ce8159bf0 msgr2=0x7f5ce81551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.275 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5ce6ffd640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ce8159bf0 0x7f5ce81551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5ce6ffd640 1 -- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd4003970 msgr2=0x7f5ce8146e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5ce6ffd640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd4003970 0x7f5ce8146e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5ce6ffd640 1 -- 172.21.15.45:0/2909747712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce8155e70 con 0x7f5ce8156cf0 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5ce7fff640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ce8159bf0 0x7f5ce81551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5ce6ffd640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce8156cf0 0x7f5ce81596b0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000d770 tx=0x7f5cd000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5cdffff640 1 -- 172.21.15.45:0/2909747712 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0005270 con 0x7f5ce8156cf0 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5cdffff640 1 -- 172.21.15.45:0/2909747712 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd00053d0 con 0x7f5ce8156cf0 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5cdffff640 1 -- 172.21.15.45:0/2909747712 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd000b8d0 con 0x7f5ce8156cf0 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.272+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce815b8a0 con 0x7f5ce8156cf0 2023-11-28T22:08:34.276 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.273+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce815be60 con 0x7f5ce8156cf0 2023-11-28T22:08:34.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.274+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce815b700 con 0x7f5ce8156cf0 2023-11-28T22:08:34.277 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.275+0000 7f5cdffff640 1 -- 172.21.15.45:0/2909747712 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5cd000ba30 con 0x7f5ce8156cf0 2023-11-28T22:08:34.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.276+0000 7f5cdffff640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5cc806af60 0x7f5cc806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.276+0000 7f5cdffff640 1 -- 172.21.15.45:0/2909747712 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7f5cd0089a10 con 0x7f5ce8156cf0 2023-11-28T22:08:34.278 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.276+0000 7f5ce77fe640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5cc806af60 0x7f5cc806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.278+0000 7f5ce77fe640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5cc806af60 0x7f5cc806d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8036000 tx=0x7f5cd8030000 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:34.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.280+0000 7f5cdffff640 1 -- 172.21.15.45:0/2909747712 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ce815b700 con 0x7f5ce8156cf0 2023-11-28T22:08:34.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.545+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f5ce815b700 con 0x7f5ce8156cf0 2023-11-28T22:08:34.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.608+0000 7f5cdffff640 1 -- 172.21.15.45:0/2909747712 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3084) v1 ==== 165+0+0 (secure 0 0 0) 0x7f5ce815b700 con 0x7f5ce8156cf0 2023-11-28T22:08:34.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5cc806af60 msgr2=0x7f5cc806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.614 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5cc806af60 0x7f5cc806d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8036000 tx=0x7f5cd8030000 comp rx=0 tx=0).stop 2023-11-28T22:08:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce8156cf0 msgr2=0x7f5ce81596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce8156cf0 0x7f5ce81596b0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000d770 tx=0x7f5cd000dc30 comp rx=0 tx=0).stop 2023-11-28T22:08:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 shutdown_connections 2023-11-28T22:08:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ce8159bf0 0x7f5ce81551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5cc806af60 0x7f5cc806d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ce8156cf0 0x7f5ce81596b0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 --2- 172.21.15.45:0/2909747712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cd4003970 0x7f5ce8146e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 >> 172.21.15.45:0/2909747712 conn(0x7f5ce806a500 msgr2=0x7f5ce815f400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 shutdown_connections 2023-11-28T22:08:34.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.612+0000 7f5cec871640 1 -- 172.21.15.45:0/2909747712 wait complete. 2023-11-28T22:08:34.634 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-11-28T22:08:34.635 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-11-28T22:08:34.635 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T22:08:34.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:08:34.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.862+0000 7f5076c5d640 1 Processor -- start 2023-11-28T22:08:34.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.862+0000 7f5076c5d640 1 -- start start 2023-11-28T22:08:34.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.864+0000 7f5076c5d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f50700b35b0 0x7f507015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.864+0000 7f5076c5d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f50700b1340 0x7f507015f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.864+0000 7f5076c5d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507015bda0 con 0x7f507015b850 2023-11-28T22:08:34.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.864+0000 7f5076c5d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507015ee40 con 0x7f50700b1340 2023-11-28T22:08:34.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.864+0000 7f5076c5d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507015c080 con 0x7f50700b35b0 2023-11-28T22:08:34.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.864+0000 7f50751fa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f50700b1340 0x7f507015f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.865+0000 7f50751fa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f50700b1340 0x7f507015f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37768/0 (socket says 172.21.15.45:37768) 2023-11-28T22:08:34.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.865+0000 7f50751fa640 1 -- 172.21.15.45:0/1300056476 learned_addr learned my addr 172.21.15.45:0/1300056476 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:34.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.865+0000 7f50749f9640 1 --2- 172.21.15.45:0/1300056476 >> v2:172.21.15.190:3300/0 conn(0x7f50700b35b0 0x7f507015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.865+0000 7f50751fa640 1 -- 172.21.15.45:0/1300056476 >> v2:172.21.15.190:3300/0 conn(0x7f50700b35b0 msgr2=0x7f507015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:34.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.865+0000 7f50751fa640 1 --2- 172.21.15.45:0/1300056476 >> v2:172.21.15.190:3300/0 conn(0x7f50700b35b0 0x7f507015ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.865+0000 7f50751fa640 1 -- 172.21.15.45:0/1300056476 >> v1:172.21.15.190:6789/0 conn(0x7f507015b850 legacy=0x7f50700b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:34.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.865+0000 7f50751fa640 1 -- 172.21.15.45:0/1300056476 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507015c360 con 0x7f50700b1340 2023-11-28T22:08:34.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.866+0000 7f50751fa640 1 --2- 172.21.15.45:0/1300056476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 0x7f507015f290 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f507006e7f0 tx=0x7f506802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68a4bece8d7f81f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:34.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f506f7fe640 1 -- 172.21.15.45:0/1300056476 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506802ca80 con 0x7f50700b1340 2023-11-28T22:08:34.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f506f7fe640 1 -- 172.21.15.45:0/1300056476 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506802cbe0 con 0x7f50700b1340 2023-11-28T22:08:34.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f506f7fe640 1 -- 172.21.15.45:0/1300056476 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50680343f0 con 0x7f50700b1340 2023-11-28T22:08:34.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f5076c5d640 1 -- 172.21.15.45:0/1300056476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 msgr2=0x7f507015f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f5076c5d640 1 --2- 172.21.15.45:0/1300056476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 0x7f507015f290 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f507006e7f0 tx=0x7f506802bee0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f5076c5d640 1 -- 172.21.15.45:0/1300056476 shutdown_connections 2023-11-28T22:08:34.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f5076c5d640 1 --2- 172.21.15.45:0/1300056476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 0x7f507015f290 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f5076c5d640 1 --2- 172.21.15.45:0/1300056476 >> v2:172.21.15.190:3300/0 conn(0x7f50700b35b0 0x7f507015ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f5076c5d640 1 -- 172.21.15.45:0/1300056476 >> 172.21.15.45:0/1300056476 conn(0x7f507006a500 msgr2=0x7f507006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:34.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.867+0000 7f5076c5d640 1 -- 172.21.15.45:0/1300056476 shutdown_connections 2023-11-28T22:08:34.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.868+0000 7f5076c5d640 1 -- 172.21.15.45:0/1300056476 wait complete. 2023-11-28T22:08:34.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.869+0000 7f5076c5d640 1 Processor -- start 2023-11-28T22:08:34.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.869+0000 7f5076c5d640 1 -- start start 2023-11-28T22:08:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.870+0000 7f5076c5d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 0x7f50701f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.870+0000 7f5076c5d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50700b35b0 0x7f5070142b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.870+0000 7f5076c5d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f507015b850 0x7f507015a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.870+0000 7f5076c5d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50701f46a0 con 0x7f50700b1340 2023-11-28T22:08:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.870+0000 7f5076c5d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50701436d0 con 0x7f50700b35b0 2023-11-28T22:08:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.870+0000 7f5076c5d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50700706b0 con 0x7f507015b850 2023-11-28T22:08:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50749f9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 0x7f50701f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50749f9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 0x7f50701f4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37772/0 (socket says 172.21.15.45:37772) 2023-11-28T22:08:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50749f9640 1 -- 172.21.15.45:0/328357802 learned_addr learned my addr 172.21.15.45:0/328357802 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:34.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50751fa640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f507015b850 0x7f507015a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f506ffff640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50700b35b0 0x7f5070142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50749f9640 1 -- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f507015b850 msgr2=0x7f507015a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50749f9640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f507015b850 0x7f507015a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50749f9640 1 -- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50700b35b0 msgr2=0x7f5070142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50749f9640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50700b35b0 0x7f5070142b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50749f9640 1 -- 172.21.15.45:0/328357802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5068009b30 con 0x7f50700b1340 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f50751fa640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f507015b850 0x7f507015a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.871+0000 7f506ffff640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50700b35b0 0x7f5070142b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.872+0000 7f50749f9640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 0x7f50701f4160 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f506400c7f0 tx=0x7f506400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.872+0000 7f506d7fa640 1 -- 172.21.15.45:0/328357802 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064004220 con 0x7f50700b1340 2023-11-28T22:08:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.872+0000 7f506d7fa640 1 -- 172.21.15.45:0/328357802 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50640043a0 con 0x7f50700b1340 2023-11-28T22:08:34.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.872+0000 7f506d7fa640 1 -- 172.21.15.45:0/328357802 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064007860 con 0x7f50700b1340 2023-11-28T22:08:34.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.872+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50701f71e0 con 0x7f50700b1340 2023-11-28T22:08:34.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.872+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50701f77a0 con 0x7f50700b1340 2023-11-28T22:08:34.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.873+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50700b1fa0 con 0x7f50700b1340 2023-11-28T22:08:34.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.874+0000 7f506d7fa640 1 -- 172.21.15.45:0/328357802 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5064007bd0 con 0x7f50700b1340 2023-11-28T22:08:34.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.875+0000 7f506d7fa640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f505406af60 0x7f505406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:34.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.875+0000 7f506d7fa640 1 -- 172.21.15.45:0/328357802 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7f5064089b30 con 0x7f50700b1340 2023-11-28T22:08:34.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.876+0000 7f506ffff640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f505406af60 0x7f505406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:34.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.877+0000 7f506ffff640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f505406af60 0x7f505406d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f50701f7d90 tx=0x7f505c008040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:34.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:34.880+0000 7f506d7fa640 1 -- 172.21.15.45:0/328357802 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50640546b0 con 0x7f50700b1340 2023-11-28T22:08:35.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.075+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50700733d0 con 0x7f50700b1340 2023-11-28T22:08:35.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.077+0000 7f506d7fa640 1 -- 172.21.15.45:0/328357802 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+14758 (secure 0 0 0) 0x7f506401e020 con 0x7f50700b1340 2023-11-28T22:08:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:35.080 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":207,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:08:32.594160+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6835/3063278837":"2023-11-29T22:08:29.670224+0000","172.21.15.45:6833/3063278837":"2023-11-29T22:08:29.670224+0000","172.21.15.190:0/667031807":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/454828214":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/2397783727":"2023-11-29T22:08:25.533236+0000","172.21.15.190:6833/697528389":"2023-11-29T22:08:25.533236+0000","172.21.15.190:6832/697528389":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/2743963182":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/3942686825":"2023-11-29T22:08:25.533236+0000","172.21.15.190:0/3880574704":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3087092286":"2023-11-29T22:08:25.533236+0000","172.21.15.190:6833/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6832/3833863335":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3938773910":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2706131286":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3522054455":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/2512261826":"2023-11-29T22:03:21.258883+0000","172.21.15.190:0/3252656094":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2538549380":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3848257557":"2023-11-29T22:03:21.258883+0000","172.21.15.190:6833/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/3811510364":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2138187339":"2023-11-29T21:52:41.039626+0000","172.21.15.190:6832/3252186332":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/624508208":"2023-11-29T21:52:41.039626+0000","172.21.15.190:0/2094131491":"2023-11-29T21:52:41.039626+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:08:35.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.079+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f505406af60 msgr2=0x7f505406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.079+0000 7f5076c5d640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f505406af60 0x7f505406d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f50701f7d90 tx=0x7f505c008040 comp rx=0 tx=0).stop 2023-11-28T22:08:35.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.079+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 msgr2=0x7f50701f4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.079+0000 7f5076c5d640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 0x7f50701f4160 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f506400c7f0 tx=0x7f506400ccb0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.079+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 shutdown_connections 2023-11-28T22:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.079+0000 7f5076c5d640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f507015b850 0x7f507015a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.079+0000 7f5076c5d640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50700b35b0 0x7f5070142b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.079+0000 7f5076c5d640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f505406af60 0x7f505406d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.079+0000 7f5076c5d640 1 --2- 172.21.15.45:0/328357802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50700b1340 0x7f50701f4160 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.080+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 >> 172.21.15.45:0/328357802 conn(0x7f507006a500 msgr2=0x7f507015d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.080+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 shutdown_connections 2023-11-28T22:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.080+0000 7f5076c5d640 1 -- 172.21.15.45:0/328357802 wait complete. 2023-11-28T22:08:35.103 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T22:08:35.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T22:08:35.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.334+0000 7f633f59e640 1 Processor -- start 2023-11-28T22:08:35.337 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.334+0000 7f633f59e640 1 -- start start 2023-11-28T22:08:35.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.336+0000 7f633f59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f63400b3040 0x7f634006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.336+0000 7f633f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f634015b270 0x7f634015f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.336+0000 7f633f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634015faf0 con 0x7f63400b3420 2023-11-28T22:08:35.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.336+0000 7f633f59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63400ceaa0 con 0x7f63400b3040 2023-11-28T22:08:35.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.336+0000 7f633f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634006d410 con 0x7f634015b270 2023-11-28T22:08:35.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.336+0000 7f633ed9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f63400b3420 0x7f63400b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51924/0 (socket says 172.21.15.45:51924) 2023-11-28T22:08:35.339 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.336+0000 7f633ed9d640 1 -- 172.21.15.45:0/1475623851 learned_addr learned my addr 172.21.15.45:0/1475623851 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:35.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633dd9b640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f634015b270 0x7f634015f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901413843 0 0) 0x7f634015faf0 con 0x7f63400b3420 2023-11-28T22:08:35.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6328003520 con 0x7f63400b3420 2023-11-28T22:08:35.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633e59c640 1 --2- 172.21.15.45:0/1475623851 >> v2:172.21.15.190:3300/0 conn(0x7f63400b3040 0x7f634006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1935195641 0 0) 0x7f6328003520 con 0x7f63400b3420 2023-11-28T22:08:35.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f634015b270 msgr2=0x7f634015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:35.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f634015b270 0x7f634015f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 >> v2:172.21.15.190:3300/0 conn(0x7f63400b3040 msgr2=0x7f634006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:35.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 --2- 172.21.15.45:0/1475623851 >> v2:172.21.15.190:3300/0 conn(0x7f63400b3040 0x7f634006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634006d6f0 con 0x7f63400b3420 2023-11-28T22:08:35.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f63380019f0 con 0x7f63400b3420 2023-11-28T22:08:35.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 >> v1:172.21.15.45:6789/0 conn(0x7f63400b3420 legacy=0x7f63400b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6328003970 0x7f6328003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63280047c0 0x7f6328004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f634015b270 0x7f6328005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634006d410 con 0x7f6328003970 2023-11-28T22:08:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632800ba30 con 0x7f63280047c0 2023-11-28T22:08:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.337+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632800bba0 con 0x7f634015b270 2023-11-28T22:08:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.338+0000 7f633e59c640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6328003970 0x7f6328003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.338+0000 7f633e59c640 1 -- 172.21.15.45:0/1475623851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f634015b270 msgr2=0x7f6328005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:35.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.338+0000 7f633e59c640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f634015b270 0x7f6328005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.338+0000 7f633e59c640 1 -- 172.21.15.45:0/1475623851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63280047c0 msgr2=0x7f6328004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:35.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.338+0000 7f633e59c640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63280047c0 0x7f6328004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.338+0000 7f633e59c640 1 -- 172.21.15.45:0/1475623851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6328003520 con 0x7f6328003970 2023-11-28T22:08:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.339+0000 7f633e59c640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6328003970 0x7f6328003d90 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f6330009a90 tx=0x7f63280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00536ecaca03b51 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:35.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633002c6b0 con 0x7f6328003970 2023-11-28T22:08:35.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633002c810 con 0x7f6328003970 2023-11-28T22:08:35.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633d59a640 1 -- 172.21.15.45:0/1475623851 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633002cba0 con 0x7f6328003970 2023-11-28T22:08:35.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633f59e640 1 -- 172.21.15.45:0/1475623851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6328003970 msgr2=0x7f6328003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633f59e640 1 --2- 172.21.15.45:0/1475623851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6328003970 0x7f6328003d90 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f6330009a90 tx=0x7f63280057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633ed9d640 1 -- 172.21.15.45:0/1475623851 reap_dead start 2023-11-28T22:08:35.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633f59e640 1 -- 172.21.15.45:0/1475623851 shutdown_connections 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633f59e640 1 -- 172.21.15.45:0/1475623851 >> 172.21.15.45:0/1475623851 conn(0x7f63400b1050 msgr2=0x7f63400cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633f59e640 1 -- 172.21.15.45:0/1475623851 shutdown_connections 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.340+0000 7f633f59e640 1 -- 172.21.15.45:0/1475623851 wait complete. 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.342+0000 7f633f59e640 1 Processor -- start 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.342+0000 7f633f59e640 1 -- start start 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.343+0000 7f633f59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6328003970 0x7f6340144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.343+0000 7f633f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63401484b0 0x7f6340099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.343+0000 7f633f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6340099f80 0x7f6340156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.343+0000 7f633f59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6340148020 con 0x7f63401484b0 2023-11-28T22:08:35.351 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.343+0000 7f633f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634009a890 con 0x7f6340099f80 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.343+0000 7f633f59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634009a5b0 con 0x7f6328003970 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63401484b0 0x7f6340099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63401484b0 0x7f6340099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37778/0 (socket says 172.21.15.45:37778) 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633dd9b640 1 -- 172.21.15.45:0/1755519534 learned_addr learned my addr 172.21.15.45:0/1755519534 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633dd9b640 1 -- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6328003970 msgr2=0x7f6340144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633e59c640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6328003970 0x7f6340144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633dd9b640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6328003970 0x7f6340144ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633dd9b640 1 -- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6340099f80 msgr2=0x7f6340156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633dd9b640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6340099f80 0x7f6340156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633dd9b640 1 -- 172.21.15.45:0/1755519534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63401479d0 con 0x7f63401484b0 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.344+0000 7f633dd9b640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63401484b0 0x7f6340099a40 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f633400e7c0 tx=0x7f633400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:35.352 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.345+0000 7f632effd640 1 -- 172.21.15.45:0/1755519534 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633400eed0 con 0x7f63401484b0 2023-11-28T22:08:35.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.345+0000 7f632effd640 1 -- 172.21.15.45:0/1755519534 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633400c8c0 con 0x7f63401484b0 2023-11-28T22:08:35.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.345+0000 7f632effd640 1 -- 172.21.15.45:0/1755519534 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633400cc70 con 0x7f63401484b0 2023-11-28T22:08:35.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.345+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6340147cb0 con 0x7f63401484b0 2023-11-28T22:08:35.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.345+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6340145420 con 0x7f63401484b0 2023-11-28T22:08:35.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.346+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63401481d0 con 0x7f63401484b0 2023-11-28T22:08:35.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.347+0000 7f632effd640 1 -- 172.21.15.45:0/1755519534 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6334013470 con 0x7f63401484b0 2023-11-28T22:08:35.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.348+0000 7f632effd640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f632006af60 0x7f632006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.348+0000 7f632effd640 1 -- 172.21.15.45:0/1755519534 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7f6334089c50 con 0x7f63401484b0 2023-11-28T22:08:35.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.349+0000 7f633e59c640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f632006af60 0x7f632006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.350+0000 7f633e59c640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f632006af60 0x7f632006d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6330036000 tx=0x7f6330030000 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:35.354 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.352+0000 7f632effd640 1 -- 172.21.15.45:0/1755519534 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63401481d0 con 0x7f63401484b0 2023-11-28T22:08:35.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.546+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f634009d0f0 con 0x7f63401484b0 2023-11-28T22:08:35.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.547+0000 7f632effd640 1 -- 172.21.15.45:0/1755519534 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v207) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63340210a0 con 0x7f63401484b0 2023-11-28T22:08:35.549 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T22:08:35.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f632006af60 msgr2=0x7f632006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f632006af60 0x7f632006d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6330036000 tx=0x7f6330030000 comp rx=0 tx=0).stop 2023-11-28T22:08:35.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63401484b0 msgr2=0x7f6340099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63401484b0 0x7f6340099a40 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f633400e7c0 tx=0x7f633400ec80 comp rx=0 tx=0).stop 2023-11-28T22:08:35.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 shutdown_connections 2023-11-28T22:08:35.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6340099f80 0x7f6340156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f632006af60 0x7f632006d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63401484b0 0x7f6340099a40 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 --2- 172.21.15.45:0/1755519534 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6328003970 0x7f6340144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.549+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 >> 172.21.15.45:0/1755519534 conn(0x7f63400b1050 msgr2=0x7f63400ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:35.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.550+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 shutdown_connections 2023-11-28T22:08:35.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.550+0000 7f633f59e640 1 -- 172.21.15.45:0/1755519534 wait complete. 2023-11-28T22:08:35.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-11-28T22:08:35.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.794+0000 7f4109b1d640 1 Processor -- start 2023-11-28T22:08:35.797 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.794+0000 7f4109b1d640 1 -- start start 2023-11-28T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.796+0000 7f4109b1d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f410415d320 0x7f41040ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.796+0000 7f4109b1d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41040b1340 0x7f4104071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.796+0000 7f4109b1d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41040ce710 con 0x7f41040add40 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.796+0000 7f4109b1d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41040ae5a0 con 0x7f41040b1340 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.796+0000 7f4109b1d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4104160fe0 con 0x7f410415d320 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.797+0000 7f4103fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41040b1340 0x7f4104071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.797+0000 7f4103fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f41040b1340 0x7f4104071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37788/0 (socket says 172.21.15.45:37788) 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.797+0000 7f4103fff640 1 -- 172.21.15.45:0/3833228428 learned_addr learned my addr 172.21.15.45:0/3833228428 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.797+0000 7f4103fff640 1 -- 172.21.15.45:0/3833228428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f410415d320 msgr2=0x7f41040ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.797+0000 7f4103fff640 1 --2- 172.21.15.45:0/3833228428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f410415d320 0x7f41040ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.797+0000 7f4103fff640 1 -- 172.21.15.45:0/3833228428 >> v1:172.21.15.190:6789/0 conn(0x7f41040add40 legacy=0x7f41040b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:35.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.797+0000 7f4103fff640 1 -- 172.21.15.45:0/3833228428 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41040ae2c0 con 0x7f41040b1340 2023-11-28T22:08:35.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.798+0000 7f4103fff640 1 --2- 172.21.15.45:0/3833228428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 0x7f4104071190 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f41040ae1c0 tx=0x7f40fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571ec0fc6ba65e73 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:35.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4102ffd640 1 -- 172.21.15.45:0/3833228428 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc031640 con 0x7f41040b1340 2023-11-28T22:08:35.802 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4102ffd640 1 -- 172.21.15.45:0/3833228428 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc0317a0 con 0x7f41040b1340 2023-11-28T22:08:35.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4102ffd640 1 -- 172.21.15.45:0/3833228428 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc031b30 con 0x7f41040b1340 2023-11-28T22:08:35.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4109b1d640 1 -- 172.21.15.45:0/3833228428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 msgr2=0x7f4104071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4109b1d640 1 --2- 172.21.15.45:0/3833228428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 0x7f4104071190 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f41040ae1c0 tx=0x7f40fc02b8f0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4109b1d640 1 -- 172.21.15.45:0/3833228428 shutdown_connections 2023-11-28T22:08:35.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4109b1d640 1 --2- 172.21.15.45:0/3833228428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 0x7f4104071190 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4109b1d640 1 --2- 172.21.15.45:0/3833228428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f410415d320 0x7f41040ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4109b1d640 1 -- 172.21.15.45:0/3833228428 >> 172.21.15.45:0/3833228428 conn(0x7f410406a500 msgr2=0x7f410406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:35.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4109b1d640 1 -- 172.21.15.45:0/3833228428 shutdown_connections 2023-11-28T22:08:35.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.799+0000 7f4109b1d640 1 -- 172.21.15.45:0/3833228428 wait complete. 2023-11-28T22:08:35.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.801+0000 7f4109b1d640 1 Processor -- start 2023-11-28T22:08:35.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.801+0000 7f4109b1d640 1 -- start start 2023-11-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.802+0000 7f4109b1d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41040add40 0x7f41040b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.802+0000 7f4109b1d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 0x7f41041465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.802+0000 7f4109b1d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f410415d320 0x7f41040ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.802+0000 7f4109b1d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41041472c0 con 0x7f41040b1340 2023-11-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.802+0000 7f4109b1d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41040b10c0 con 0x7f410415d320 2023-11-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.802+0000 7f4109b1d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4104159480 con 0x7f41040add40 2023-11-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f40faffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 0x7f41041465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f40faffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 0x7f41041465d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37796/0 (socket says 172.21.15.45:37796) 2023-11-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f40faffd640 1 -- 172.21.15.45:0/159863379 learned_addr learned my addr 172.21.15.45:0/159863379 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f41037fe640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41040add40 0x7f41040b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f40faffd640 1 -- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41040add40 msgr2=0x7f41040b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f40faffd640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41040add40 0x7f41040b0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f40faffd640 1 -- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f410415d320 msgr2=0x7f41040ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:35.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f40faffd640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f410415d320 0x7f41040ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:35.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f40faffd640 1 -- 172.21.15.45:0/159863379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40fc0094d0 con 0x7f41040b1340 2023-11-28T22:08:35.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.803+0000 7f40faffd640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 0x7f41041465d0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00c7a0 tx=0x7f40ec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:35.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.804+0000 7f4100ff9640 1 -- 172.21.15.45:0/159863379 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec003d80 con 0x7f41040b1340 2023-11-28T22:08:35.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.804+0000 7f4100ff9640 1 -- 172.21.15.45:0/159863379 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec005070 con 0x7f41040b1340 2023-11-28T22:08:35.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.804+0000 7f4100ff9640 1 -- 172.21.15.45:0/159863379 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec005460 con 0x7f41040b1340 2023-11-28T22:08:35.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.804+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4104159760 con 0x7f41040b1340 2023-11-28T22:08:35.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.804+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4104159d20 con 0x7f41040b1340 2023-11-28T22:08:35.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.805+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4104157200 con 0x7f41040b1340 2023-11-28T22:08:35.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.806+0000 7f4100ff9640 1 -- 172.21.15.45:0/159863379 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f40ec0055c0 con 0x7f41040b1340 2023-11-28T22:08:35.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.807+0000 7f4100ff9640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f40e406af90 0x7f40e406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:35.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.807+0000 7f4100ff9640 1 -- 172.21.15.45:0/159863379 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7f40ec088e30 con 0x7f41040b1340 2023-11-28T22:08:35.810 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.808+0000 7f41037fe640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f40e406af90 0x7f40e406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:35.811 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.809+0000 7f41037fe640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f40e406af90 0x7f40e406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f40f400a770 tx=0x7f40f4008040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:35.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:35.811+0000 7f4100ff9640 1 -- 172.21.15.45:0/159863379 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40ec053b50 con 0x7f41040b1340 2023-11-28T22:08:36.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.079+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f41040733d0 con 0x7f41040b1340 2023-11-28T22:08:36.623 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.621+0000 7f4100ff9640 1 -- 172.21.15.45:0/159863379 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3086) v1 ==== 166+0+0 (secure 0 0 0) 0x7f40ec057170 con 0x7f41040b1340 2023-11-28T22:08:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f40e406af90 msgr2=0x7f40e406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f40e406af90 0x7f40e406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f40f400a770 tx=0x7f40f4008040 comp rx=0 tx=0).stop 2023-11-28T22:08:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 msgr2=0x7f41041465d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 0x7f41041465d0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00c7a0 tx=0x7f40ec00cc60 comp rx=0 tx=0).stop 2023-11-28T22:08:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 shutdown_connections 2023-11-28T22:08:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f410415d320 0x7f41040ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f40e406af90 0x7f40e406d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41040b1340 0x7f41041465d0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 --2- 172.21.15.45:0/159863379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f41040add40 0x7f41040b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.624+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 >> 172.21.15.45:0/159863379 conn(0x7f410406a500 msgr2=0x7f4104155b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:36.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.625+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 shutdown_connections 2023-11-28T22:08:36.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.625+0000 7f4109b1d640 1 -- 172.21.15.45:0/159863379 wait complete. 2023-11-28T22:08:36.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T22:08:36.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.871+0000 7ff5a8fc1640 1 Processor -- start 2023-11-28T22:08:36.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.871+0000 7ff5a8fc1640 1 -- start start 2023-11-28T22:08:36.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.873+0000 7ff5a8fc1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff5a415ae10 0x7ff5a415d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:36.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.873+0000 7ff5a8fc1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 0x7ff5a40cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:36.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.873+0000 7ff5a8fc1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a415b1f0 con 0x7ff5a406a710 2023-11-28T22:08:36.875 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.873+0000 7ff5a8fc1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a40b26c0 con 0x7ff5a415ae10 2023-11-28T22:08:36.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.873+0000 7ff5a8fc1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40cf050 con 0x7ff5a40cde40 2023-11-28T22:08:36.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.873+0000 7ff5a37fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff5a415ae10 0x7ff5a415d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:36.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.873+0000 7ff5a2ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff5a406a710 0x7ff5a40cd730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51936/0 (socket says 172.21.15.45:51936) 2023-11-28T22:08:36.876 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.873+0000 7ff5a2ffd640 1 -- 172.21.15.45:0/1168959771 learned_addr learned my addr 172.21.15.45:0/1168959771 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.874+0000 7ff5a37fe640 1 -- 172.21.15.45:0/1168959771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 msgr2=0x7ff5a40cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.874+0000 7ff5a3fff640 1 --2- 172.21.15.45:0/1168959771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 0x7ff5a40cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.874+0000 7ff5a37fe640 1 --2- 172.21.15.45:0/1168959771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 0x7ff5a40cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.874+0000 7ff5a27fc640 1 -- 172.21.15.45:0/1168959771 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110630298 0 0) 0x7ff5a415b1f0 con 0x7ff5a406a710 2023-11-28T22:08:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.874+0000 7ff5a37fe640 1 -- 172.21.15.45:0/1168959771 >> v1:172.21.15.45:6789/0 conn(0x7ff5a406a710 legacy=0x7ff5a40cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.874+0000 7ff5a37fe640 1 -- 172.21.15.45:0/1168959771 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a406ae80 con 0x7ff5a415ae10 2023-11-28T22:08:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.874+0000 7ff5a3fff640 1 --2- 172.21.15.45:0/1168959771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 0x7ff5a40cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:36.877 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.875+0000 7ff5a37fe640 1 --2- 172.21.15.45:0/1168959771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a415ae10 0x7ff5a415d7d0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7ff5940097a0 tx=0x7ff59402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477e7c38e4226369 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:36.878 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.875+0000 7ff5a27fc640 1 -- 172.21.15.45:0/1168959771 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59402d040 con 0x7ff5a415ae10 2023-11-28T22:08:36.879 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.875+0000 7ff5a27fc640 1 -- 172.21.15.45:0/1168959771 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff594031810 con 0x7ff5a415ae10 2023-11-28T22:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.875+0000 7ff5a27fc640 1 -- 172.21.15.45:0/1168959771 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594031ba0 con 0x7ff5a415ae10 2023-11-28T22:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.876+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/1168959771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a415ae10 msgr2=0x7ff5a415d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.876+0000 7ff5a8fc1640 1 --2- 172.21.15.45:0/1168959771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a415ae10 0x7ff5a415d7d0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7ff5940097a0 tx=0x7ff59402b8b0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.876+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/1168959771 shutdown_connections 2023-11-28T22:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.876+0000 7ff5a8fc1640 1 --2- 172.21.15.45:0/1168959771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 0x7ff5a40cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.876+0000 7ff5a8fc1640 1 --2- 172.21.15.45:0/1168959771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a415ae10 0x7ff5a415d7d0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.876+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/1168959771 >> 172.21.15.45:0/1168959771 conn(0x7ff5a40d0d20 msgr2=0x7ff5a40d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.876+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/1168959771 shutdown_connections 2023-11-28T22:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.876+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/1168959771 wait complete. 2023-11-28T22:08:36.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.878+0000 7ff5a8fc1640 1 Processor -- start 2023-11-28T22:08:36.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.878+0000 7ff5a8fc1640 1 -- start start 2023-11-28T22:08:36.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.879+0000 7ff5a8fc1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a406a710 0x7ff5a4148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:36.884 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.879+0000 7ff5a8fc1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 0x7ff5a4160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.879+0000 7ff5a8fc1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5a415ae10 0x7ff5a4160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.879+0000 7ff5a8fc1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4161520 con 0x7ff5a406a710 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.879+0000 7ff5a8fc1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40c6290 con 0x7ff5a415ae10 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.879+0000 7ff5a8fc1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40c64e0 con 0x7ff5a40cde40 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a406a710 0x7ff5a4148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a406a710 0x7ff5a4148f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37820/0 (socket says 172.21.15.45:37820) 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a37fe640 1 -- 172.21.15.45:0/4055043166 learned_addr learned my addr 172.21.15.45:0/4055043166 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a2ffd640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 0x7ff5a4160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a37fe640 1 -- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 msgr2=0x7ff5a4160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a37fe640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 0x7ff5a4160aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a37fe640 1 -- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5a415ae10 msgr2=0x7ff5a4160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a37fe640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5a415ae10 0x7ff5a4160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:36.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a37fe640 1 -- 172.21.15.45:0/4055043166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594009400 con 0x7ff5a406a710 2023-11-28T22:08:36.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.880+0000 7ff5a37fe640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a406a710 0x7ff5a4148f80 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7ff5940098c0 tx=0x7ff5940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:36.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.881+0000 7ff58bfff640 1 -- 172.21.15.45:0/4055043166 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59403ad60 con 0x7ff5a406a710 2023-11-28T22:08:36.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.881+0000 7ff58bfff640 1 -- 172.21.15.45:0/4055043166 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5940443f0 con 0x7ff5a406a710 2023-11-28T22:08:36.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.881+0000 7ff58bfff640 1 -- 172.21.15.45:0/4055043166 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5940447a0 con 0x7ff5a406a710 2023-11-28T22:08:36.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.881+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a40c67c0 con 0x7ff5a406a710 2023-11-28T22:08:36.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.881+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a40c6d80 con 0x7ff5a406a710 2023-11-28T22:08:36.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.882+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a404e840 con 0x7ff5a406a710 2023-11-28T22:08:36.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.883+0000 7ff58bfff640 1 -- 172.21.15.45:0/4055043166 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff59404d450 con 0x7ff5a406a710 2023-11-28T22:08:36.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.884+0000 7ff58bfff640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff59006af60 0x7ff59006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:36.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.884+0000 7ff58bfff640 1 -- 172.21.15.45:0/4055043166 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7ff5940abff0 con 0x7ff5a406a710 2023-11-28T22:08:36.887 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.885+0000 7ff5a2ffd640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff59006af60 0x7ff59006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:36.888 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.886+0000 7ff5a2ffd640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff59006af60 0x7ff59006d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40b21e0 tx=0x7ff598009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:36.890 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:36.888+0000 7ff58bfff640 1 -- 172.21.15.45:0/4055043166 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff594076b70 con 0x7ff5a406a710 2023-11-28T22:08:37.087 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.084+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff5a40cc510 con 0x7ff5a406a710 2023-11-28T22:08:37.092 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.090+0000 7ff58bfff640 1 -- 172.21.15.45:0/4055043166 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff594035020 con 0x7ff5a406a710 2023-11-28T22:08:37.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff59006af60 msgr2=0x7ff59006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:37.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff59006af60 0x7ff59006d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40b21e0 tx=0x7ff598009040 comp rx=0 tx=0).stop 2023-11-28T22:08:37.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a406a710 msgr2=0x7ff5a4148f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:37.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a406a710 0x7ff5a4148f80 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7ff5940098c0 tx=0x7ff5940046c0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 shutdown_connections 2023-11-28T22:08:37.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5a415ae10 0x7ff5a4160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5a40cde40 0x7ff5a4160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.096 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff59006af60 0x7ff59006d420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 --2- 172.21.15.45:0/4055043166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5a406a710 0x7ff5a4148f80 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.093+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 >> 172.21.15.45:0/4055043166 conn(0x7ff5a40d0d20 msgr2=0x7ff5a40cd730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:37.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.094+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 shutdown_connections 2023-11-28T22:08:37.097 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.094+0000 7ff5a8fc1640 1 -- 172.21.15.45:0/4055043166 wait complete. 2023-11-28T22:08:37.115 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T22:08:37.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T22:08:37.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T22:08:37.365 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T22:08:37.365 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:08:37.366 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:37.431 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T22:08:37.432 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T22:08:37.432 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:37.496 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T22:08:37.496 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:08:37.496 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:37.557 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T22:08:37.557 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T22:08:37.557 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:37.617 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T22:08:37.617 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T22:08:37.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:08:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.850+0000 7f5a28b9e640 1 Processor -- start 2023-11-28T22:08:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.851+0000 7f5a28b9e640 1 -- start start 2023-11-28T22:08:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.852+0000 7f5a28b9e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a240ae540 0x7f5a240ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.852+0000 7f5a28b9e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5a240ce630 0x7f5a240b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.852+0000 7f5a28b9e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2415c3c0 con 0x7f5a24155780 2023-11-28T22:08:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.852+0000 7f5a28b9e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a240ceac0 con 0x7f5a240ae540 2023-11-28T22:08:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.852+0000 7f5a28b9e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a24155b60 con 0x7f5a240ce630 2023-11-28T22:08:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.853+0000 7f5a1bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a240ae540 0x7f5a240ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.853+0000 7f5a1bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a240ae540 0x7f5a240ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37834/0 (socket says 172.21.15.45:37834) 2023-11-28T22:08:37.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.853+0000 7f5a1bfff640 1 -- 172.21.15.45:0/1784228114 learned_addr learned my addr 172.21.15.45:0/1784228114 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.853+0000 7f5a1bfff640 1 -- 172.21.15.45:0/1784228114 >> v2:172.21.15.190:3300/0 conn(0x7f5a240ce630 msgr2=0x7f5a240b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.853+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1784228114 >> v2:172.21.15.190:3300/0 conn(0x7f5a240ce630 0x7f5a240b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.853+0000 7f5a1bfff640 1 -- 172.21.15.45:0/1784228114 >> v1:172.21.15.190:6789/0 conn(0x7f5a24155780 legacy=0x7f5a240abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.853+0000 7f5a1bfff640 1 -- 172.21.15.45:0/1784228114 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2415ba90 con 0x7f5a240ae540 2023-11-28T22:08:37.856 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.854+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1784228114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 0x7f5a240ce0f0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f5a240b1690 tx=0x7f5a0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1898a566b8c3dfa8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:37.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.855+0000 7f5a22ffd640 1 -- 172.21.15.45:0/1784228114 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c031870 con 0x7f5a240ae540 2023-11-28T22:08:37.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.855+0000 7f5a22ffd640 1 -- 172.21.15.45:0/1784228114 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0319d0 con 0x7f5a240ae540 2023-11-28T22:08:37.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.855+0000 7f5a22ffd640 1 -- 172.21.15.45:0/1784228114 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c031d60 con 0x7f5a240ae540 2023-11-28T22:08:37.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.855+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1784228114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 msgr2=0x7f5a240ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:37.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.855+0000 7f5a28b9e640 1 --2- 172.21.15.45:0/1784228114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 0x7f5a240ce0f0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f5a240b1690 tx=0x7f5a0c02b910 comp rx=0 tx=0).stop 2023-11-28T22:08:37.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.856+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1784228114 shutdown_connections 2023-11-28T22:08:37.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.856+0000 7f5a28b9e640 1 --2- 172.21.15.45:0/1784228114 >> v2:172.21.15.190:3300/0 conn(0x7f5a240ce630 0x7f5a240b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.856+0000 7f5a28b9e640 1 --2- 172.21.15.45:0/1784228114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 0x7f5a240ce0f0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.856+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1784228114 >> 172.21.15.45:0/1784228114 conn(0x7f5a2406ab30 msgr2=0x7f5a240b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:37.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.856+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1784228114 shutdown_connections 2023-11-28T22:08:37.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.856+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1784228114 wait complete. 2023-11-28T22:08:37.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.857+0000 7f5a28b9e640 1 Processor -- start 2023-11-28T22:08:37.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.858+0000 7f5a28b9e640 1 -- start start 2023-11-28T22:08:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a28b9e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 0x7f5a240aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a28b9e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a240ce630 0x7f5a2409bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a28b9e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a24155780 0x7f5a2414a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a28b9e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2414d3f0 con 0x7f5a240ae540 2023-11-28T22:08:37.861 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a28b9e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a241f7720 con 0x7f5a24155780 2023-11-28T22:08:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a28b9e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2409b8f0 con 0x7f5a240ce630 2023-11-28T22:08:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 0x7f5a240aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 0x7f5a240aea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37842/0 (socket says 172.21.15.45:37842) 2023-11-28T22:08:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a237fe640 1 -- 172.21.15.45:0/1463704480 learned_addr learned my addr 172.21.15.45:0/1463704480 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a240ce630 0x7f5a2409bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a237fe640 1 -- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a240ce630 msgr2=0x7f5a2409bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a23fff640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a24155780 0x7f5a2414a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a237fe640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a240ce630 0x7f5a2409bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a237fe640 1 -- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a24155780 msgr2=0x7f5a2414a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a237fe640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a24155780 0x7f5a2414a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a237fe640 1 -- 172.21.15.45:0/1463704480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c0094d0 con 0x7f5a240ae540 2023-11-28T22:08:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.859+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a240ce630 0x7f5a2409bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.860+0000 7f5a237fe640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 0x7f5a240aea00 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400b3e0 tx=0x7f5a1400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.860+0000 7f5a20ff9640 1 -- 172.21.15.45:0/1463704480 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14011830 con 0x7f5a240ae540 2023-11-28T22:08:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.860+0000 7f5a20ff9640 1 -- 172.21.15.45:0/1463704480 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a140119b0 con 0x7f5a240ae540 2023-11-28T22:08:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.860+0000 7f5a20ff9640 1 -- 172.21.15.45:0/1463704480 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14011d60 con 0x7f5a240ae540 2023-11-28T22:08:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.860+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2414cab0 con 0x7f5a240ae540 2023-11-28T22:08:37.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.860+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2414d010 con 0x7f5a240ae540 2023-11-28T22:08:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.861+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2414d530 con 0x7f5a240ae540 2023-11-28T22:08:37.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.862+0000 7f5a20ff9640 1 -- 172.21.15.45:0/1463704480 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5a1401a400 con 0x7f5a240ae540 2023-11-28T22:08:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.864+0000 7f5a20ff9640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a0406af60 0x7f5a0406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.864+0000 7f5a20ff9640 1 -- 172.21.15.45:0/1463704480 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7f5a14089b80 con 0x7f5a240ae540 2023-11-28T22:08:37.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.864+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a0406af60 0x7f5a0406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:37.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.866+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a0406af60 0x7f5a0406d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c036000 tx=0x7f5a0c03c040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:37.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:37.868+0000 7f5a20ff9640 1 -- 172.21.15.45:0/1463704480 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a2414d530 con 0x7f5a240ae540 2023-11-28T22:08:38.103 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.101+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a2414d530 con 0x7f5a240ae540 2023-11-28T22:08:38.104 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.102+0000 7f5a20ff9640 1 -- 172.21.15.45:0/1463704480 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+3707 (secure 0 0 0) 0x7f5a2414d530 con 0x7f5a240ae540 2023-11-28T22:08:38.104 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:38.104 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":94,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3312844962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3312844962},{"type":"v1","addr":"172.21.15.190:6835","nonce":3312844962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":8296,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1302790439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1302790439},{"type":"v1","addr":"172.21.15.190:6837","nonce":1302790439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10104,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6834/2082498185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2082498185},{"type":"v1","addr":"172.21.15.45:6834","nonce":2082498185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10542,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/2125978492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6833","nonce":2125978492},{"type":"v1","addr":"172.21.15.45:6835","nonce":2125978492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[]} 2023-11-28T22:08:38.105 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 94 2023-11-28T22:08:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.104+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a0406af60 msgr2=0x7f5a0406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.104+0000 7f5a28b9e640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a0406af60 0x7f5a0406d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c036000 tx=0x7f5a0c03c040 comp rx=0 tx=0).stop 2023-11-28T22:08:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 msgr2=0x7f5a240aea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 0x7f5a240aea00 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400b3e0 tx=0x7f5a1400b8a0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 shutdown_connections 2023-11-28T22:08:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a24155780 0x7f5a2414a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a240ce630 0x7f5a2409bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f5a0406af60 0x7f5a0406d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 --2- 172.21.15.45:0/1463704480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a240ae540 0x7f5a240aea00 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 >> 172.21.15.45:0/1463704480 conn(0x7f5a2406ab30 msgr2=0x7f5a240ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 shutdown_connections 2023-11-28T22:08:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.105+0000 7f5a28b9e640 1 -- 172.21.15.45:0/1463704480 wait complete. 2023-11-28T22:08:38.126 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T22:08:38.126 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T22:08:38.126 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T22:08:38.126 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:08:38.127 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T22:08:38.127 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T22:08:38.127 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T22:08:38.127 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:08:38.127 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T22:08:38.127 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T22:08:38.127 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T22:08:38.128 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:08:38.128 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T22:08:38.128 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T22:08:38.128 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T22:08:38.128 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:08:38.147 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T22:08:38.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T22:08:38.148 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T22:08:38.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T22:08:38.150 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T22:08:38.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T22:08:38.155 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T22:08:38.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T22:08:38.196 INFO:tasks.ceph.mds.c:Started 2023-11-28T22:08:38.196 DEBUG:teuthology.parallel:result is None 2023-11-28T22:08:38.203 INFO:tasks.ceph.mds.a:Started 2023-11-28T22:08:38.203 DEBUG:teuthology.parallel:result is None 2023-11-28T22:08:38.204 INFO:tasks.ceph.mds.d:Started 2023-11-28T22:08:38.204 DEBUG:teuthology.parallel:result is None 2023-11-28T22:08:38.211 INFO:tasks.ceph.mds.b:Started 2023-11-28T22:08:38.211 DEBUG:teuthology.parallel:result is None 2023-11-28T22:08:38.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T22:08:38.363 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2023-11-28T22:08:38.364 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T22:08:38.365 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at 2023-11-28T22:08:38.372 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2023-11-28T22:08:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.406+0000 7fd3b9f68640 1 Processor -- start 2023-11-28T22:08:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.406+0000 7fd3b9f68640 1 -- start start 2023-11-28T22:08:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.407+0000 7fd3b9f68640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd3b4143a60 0x7fd3b406c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.407+0000 7fd3b9f68640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd3b40cf820 0x7fd3b40cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.407+0000 7fd3b9f68640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40cd1f0 con 0x7fd3b406c820 2023-11-28T22:08:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.407+0000 7fd3b9f68640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b406cda0 con 0x7fd3b40cf820 2023-11-28T22:08:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.407+0000 7fd3b9f68640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b406d080 con 0x7fd3b4143a60 2023-11-28T22:08:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.408+0000 7fd3b3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd3b40cf820 0x7fd3b40cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:38.410 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.408+0000 7fd3b3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd3b40cf820 0x7fd3b40cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37888/0 (socket says 172.21.15.45:37888) 2023-11-28T22:08:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.408+0000 7fd3b3fff640 1 -- 172.21.15.45:0/2307608301 learned_addr learned my addr 172.21.15.45:0/2307608301 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.408+0000 7fd3b37fe640 1 --2- 172.21.15.45:0/2307608301 >> v2:172.21.15.190:3300/0 conn(0x7fd3b4143a60 0x7fd3b406c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.408+0000 7fd3b3fff640 1 -- 172.21.15.45:0/2307608301 >> v2:172.21.15.190:3300/0 conn(0x7fd3b4143a60 msgr2=0x7fd3b406c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.408+0000 7fd3b3fff640 1 --2- 172.21.15.45:0/2307608301 >> v2:172.21.15.190:3300/0 conn(0x7fd3b4143a60 0x7fd3b406c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.408+0000 7fd3b3fff640 1 -- 172.21.15.45:0/2307608301 >> v1:172.21.15.190:6789/0 conn(0x7fd3b406c820 legacy=0x7fd3b40cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:38.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.408+0000 7fd3b3fff640 1 -- 172.21.15.45:0/2307608301 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b40cce80 con 0x7fd3b40cf820 2023-11-28T22:08:38.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.409+0000 7fd3b3fff640 1 --2- 172.21.15.45:0/2307608301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b40cf820 0x7fd3b40cd680 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fd3b406e790 tx=0x7fd3ac02e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0584c060893e5e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:38.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.410+0000 7fd3b2ffd640 1 -- 172.21.15.45:0/2307608301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00b0d0 con 0x7fd3b40cf820 2023-11-28T22:08:38.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.410+0000 7fd3b2ffd640 1 -- 172.21.15.45:0/2307608301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ac00b230 con 0x7fd3b40cf820 2023-11-28T22:08:38.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.410+0000 7fd3b2ffd640 1 -- 172.21.15.45:0/2307608301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac009600 con 0x7fd3b40cf820 2023-11-28T22:08:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.410+0000 7fd3b9f68640 1 -- 172.21.15.45:0/2307608301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b40cf820 msgr2=0x7fd3b40cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.410+0000 7fd3b9f68640 1 --2- 172.21.15.45:0/2307608301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b40cf820 0x7fd3b40cd680 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fd3b406e790 tx=0x7fd3ac02e180 comp rx=0 tx=0).stop 2023-11-28T22:08:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.410+0000 7fd3b9f68640 1 -- 172.21.15.45:0/2307608301 shutdown_connections 2023-11-28T22:08:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.410+0000 7fd3b9f68640 1 --2- 172.21.15.45:0/2307608301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b40cf820 0x7fd3b40cd680 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.410+0000 7fd3b9f68640 1 --2- 172.21.15.45:0/2307608301 >> v2:172.21.15.190:3300/0 conn(0x7fd3b4143a60 0x7fd3b406c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.411+0000 7fd3b9f68640 1 -- 172.21.15.45:0/2307608301 >> 172.21.15.45:0/2307608301 conn(0x7fd3b406a6e0 msgr2=0x7fd3b406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.411+0000 7fd3b9f68640 1 -- 172.21.15.45:0/2307608301 shutdown_connections 2023-11-28T22:08:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.411+0000 7fd3b9f68640 1 -- 172.21.15.45:0/2307608301 wait complete. 2023-11-28T22:08:38.414 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.412+0000 7fd3b9f68640 1 Processor -- start 2023-11-28T22:08:38.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.413+0000 7fd3b9f68640 1 -- start start 2023-11-28T22:08:38.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b9f68640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b406c820 0x7fd3b4159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:38.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b9f68640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3b40cf820 0x7fd3b4137de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b9f68640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3b4143a60 0x7fd3b41ee920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b9f68640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4135bf0 con 0x7fd3b406c820 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b9f68640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4137be0 con 0x7fd3b40cf820 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b9f68640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40af980 con 0x7fd3b4143a60 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b406c820 0x7fd3b4159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b406c820 0x7fd3b4159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37904/0 (socket says 172.21.15.45:37904) 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b37fe640 1 -- 172.21.15.45:0/3098729543 learned_addr learned my addr 172.21.15.45:0/3098729543 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3abfff640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3b40cf820 0x7fd3b4137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b3fff640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3b4143a60 0x7fd3b41ee920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b37fe640 1 -- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3b4143a60 msgr2=0x7fd3b41ee920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b37fe640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3b4143a60 0x7fd3b41ee920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b37fe640 1 -- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3b40cf820 msgr2=0x7fd3b4137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:38.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b37fe640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3b40cf820 0x7fd3b4137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:38.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b37fe640 1 -- 172.21.15.45:0/3098729543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac00bd40 con 0x7fd3b406c820 2023-11-28T22:08:38.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3b3fff640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3b4143a60 0x7fd3b41ee920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:08:38.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.414+0000 7fd3abfff640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3b40cf820 0x7fd3b4137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:08:38.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.415+0000 7fd3b37fe640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b406c820 0x7fd3b4159b90 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400e8a0 tx=0x7fd3a400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:38.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.415+0000 7fd3b0ff9640 1 -- 172.21.15.45:0/3098729543 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40094f0 con 0x7fd3b406c820 2023-11-28T22:08:38.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.415+0000 7fd3b0ff9640 1 -- 172.21.15.45:0/3098729543 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a400c940 con 0x7fd3b406c820 2023-11-28T22:08:38.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.415+0000 7fd3b0ff9640 1 -- 172.21.15.45:0/3098729543 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a400ccf0 con 0x7fd3b406c820 2023-11-28T22:08:38.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.415+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b4135de0 con 0x7fd3b406c820 2023-11-28T22:08:38.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.415+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b41363a0 con 0x7fd3b406c820 2023-11-28T22:08:38.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.417+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b4073440 con 0x7fd3b406c820 2023-11-28T22:08:38.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.417+0000 7fd3b0ff9640 1 -- 172.21.15.45:0/3098729543 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd3a4013470 con 0x7fd3b406c820 2023-11-28T22:08:38.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.418+0000 7fd3b0ff9640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd39406af60 0x7fd39406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:38.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.418+0000 7fd3b0ff9640 1 -- 172.21.15.45:0/3098729543 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6323+0+0 (secure 0 0 0) 0x7fd3a4089a50 con 0x7fd3b406c820 2023-11-28T22:08:38.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.419+0000 7fd3abfff640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd39406af60 0x7fd39406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:38.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.420+0000 7fd3abfff640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd39406af60 0x7fd39406d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd39c007fd0 tx=0x7fd39c009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:38.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.423+0000 7fd3b0ff9640 1 -- 172.21.15.45:0/3098729543 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3a4054810 con 0x7fd3b406c820 2023-11-28T22:08:38.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:38.627+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd3b406db90 con 0x7fd3b406c820 2023-11-28T22:08:39.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.116+0000 7fd3b0ff9640 1 -- 172.21.15.45:0/3098729543 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v209) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd3a4021080 con 0x7fd3b406c820 2023-11-28T22:08:39.119 INFO:teuthology.orchestra.run.smithi045.stderr: removed all blocklist entries 2023-11-28T22:08:39.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd39406af60 msgr2=0x7fd39406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd39406af60 0x7fd39406d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd39c007fd0 tx=0x7fd39c009040 comp rx=0 tx=0).stop 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b406c820 msgr2=0x7fd3b4159b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b406c820 0x7fd3b4159b90 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400e8a0 tx=0x7fd3a400ed60 comp rx=0 tx=0).stop 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 shutdown_connections 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd3b4143a60 0x7fd3b41ee920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3b40cf820 0x7fd3b4137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd39406af60 0x7fd39406d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 --2- 172.21.15.45:0/3098729543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3b406c820 0x7fd3b4159b90 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 >> 172.21.15.45:0/3098729543 conn(0x7fd3b406a6e0 msgr2=0x7fd3b40cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:39.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 shutdown_connections 2023-11-28T22:08:39.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.120+0000 7fd3b9f68640 1 -- 172.21.15.45:0/3098729543 wait complete. 2023-11-28T22:08:39.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T22:08:39.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.365+0000 7f071d6ef640 1 Processor -- start 2023-11-28T22:08:39.369 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.366+0000 7f071d6ef640 1 -- start start 2023-11-28T22:08:39.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.367+0000 7f071d6ef640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f071815c050 0x7f0718166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.368+0000 7f071d6ef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f071815c670 con 0x7f0718141600 2023-11-28T22:08:39.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.368+0000 7f071d6ef640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07180cda40 con 0x7f07180cc2a0 2023-11-28T22:08:39.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.368+0000 7f071d6ef640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07180ccc80 con 0x7f071815c050 2023-11-28T22:08:39.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.368+0000 7f0717fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0718141600 0x7f07180cbb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51956/0 (socket says 172.21.15.45:51956) 2023-11-28T22:08:39.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.368+0000 7f0717fff640 1 -- 172.21.15.45:0/1353108809 learned_addr learned my addr 172.21.15.45:0/1353108809 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:39.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.368+0000 7f071ceee640 1 --2- 172.21.15.45:0/1353108809 >> v2:172.21.15.190:3300/0 conn(0x7f071815c050 0x7f0718166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.368+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472954382 0 0) 0x7f071815c670 con 0x7f0718141600 2023-11-28T22:08:39.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.368+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0704003520 con 0x7f0718141600 2023-11-28T22:08:39.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 483997741 0 0) 0x7f0704003520 con 0x7f0718141600 2023-11-28T22:08:39.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 >> v2:172.21.15.190:3300/0 conn(0x7f071815c050 msgr2=0x7f0718166020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:39.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 --2- 172.21.15.45:0/1353108809 >> v2:172.21.15.190:3300/0 conn(0x7f071815c050 0x7f0718166020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 >> v1:172.21.15.190:6789/0 conn(0x7f07180cc2a0 legacy=0x7f071815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07180ccf60 con 0x7f0718141600 2023-11-28T22:08:39.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f0708001670 con 0x7f0718141600 2023-11-28T22:08:39.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 >> v1:172.21.15.45:6789/0 conn(0x7f0718141600 legacy=0x7f07180cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0704003970 0x7f0704003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07040047c0 0x7f0704004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0704005180 0x7f070400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07180ccc80 con 0x7f0704005180 2023-11-28T22:08:39.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070400c020 con 0x7f0704003970 2023-11-28T22:08:39.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070400c1e0 con 0x7f07040047c0 2023-11-28T22:08:39.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f071ceee640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0704005180 0x7f070400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.369+0000 7f0717fff640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0704003970 0x7f0704003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.370+0000 7f070f7fe640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07040047c0 0x7f0704004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.370+0000 7f071ceee640 1 -- 172.21.15.45:0/1353108809 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07040047c0 msgr2=0x7f0704004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:39.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.370+0000 7f071ceee640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07040047c0 0x7f0704004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.370+0000 7f071ceee640 1 -- 172.21.15.45:0/1353108809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0704003970 msgr2=0x7f0704003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:39.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.370+0000 7f071ceee640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0704003970 0x7f0704003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.370+0000 7f071ceee640 1 -- 172.21.15.45:0/1353108809 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0704003520 con 0x7f0704005180 2023-11-28T22:08:39.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.370+0000 7f0717fff640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0704003970 0x7f0704003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:08:39.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.371+0000 7f071ceee640 1 -- 172.21.15.45:0/1353108809 reap_dead start 2023-11-28T22:08:39.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.371+0000 7f071ceee640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0704005180 0x7f070400b980 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f0710009fc0 tx=0x7f07040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876a3f9783433ebd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:39.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.371+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071002cc60 con 0x7f0704005180 2023-11-28T22:08:39.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.371+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071000f040 con 0x7f0704005180 2023-11-28T22:08:39.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.371+0000 7f07177fe640 1 -- 172.21.15.45:0/1353108809 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710003990 con 0x7f0704005180 2023-11-28T22:08:39.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.372+0000 7f071d6ef640 1 -- 172.21.15.45:0/1353108809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0704005180 msgr2=0x7f070400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.372+0000 7f071d6ef640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0704005180 0x7f070400b980 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f0710009fc0 tx=0x7f07040057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.372+0000 7f071d6ef640 1 -- 172.21.15.45:0/1353108809 shutdown_connections 2023-11-28T22:08:39.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.372+0000 7f071d6ef640 1 --2- 172.21.15.45:0/1353108809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0704005180 0x7f070400b980 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.372+0000 7f071d6ef640 1 -- 172.21.15.45:0/1353108809 >> 172.21.15.45:0/1353108809 conn(0x7f07180b0e70 msgr2=0x7f07180b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:39.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.372+0000 7f071d6ef640 1 -- 172.21.15.45:0/1353108809 shutdown_connections 2023-11-28T22:08:39.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.372+0000 7f071d6ef640 1 -- 172.21.15.45:0/1353108809 wait complete. 2023-11-28T22:08:39.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.375+0000 7f071d6ef640 1 Processor -- start 2023-11-28T22:08:39.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.375+0000 7f071d6ef640 1 -- start start 2023-11-28T22:08:39.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.376+0000 7f071d6ef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0704005180 0x7f07180ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.376+0000 7f071d6ef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07180c80a0 0x7f0718158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.376+0000 7f071d6ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f071815d250 0x7f071815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.376+0000 7f071d6ef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071815dc10 con 0x7f071815d250 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.376+0000 7f071d6ef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0718134ef0 con 0x7f07180c80a0 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.376+0000 7f071d6ef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07180c8520 con 0x7f0704005180 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f071ceee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f071815d250 0x7f071815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f071ceee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f071815d250 0x7f071815d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37922/0 (socket says 172.21.15.45:37922) 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f071ceee640 1 -- 172.21.15.45:0/3793558834 learned_addr learned my addr 172.21.15.45:0/3793558834 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f0717fff640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0704005180 0x7f07180ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f070f7fe640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07180c80a0 0x7f0718158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f071ceee640 1 -- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0704005180 msgr2=0x7f07180ca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f071ceee640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0704005180 0x7f07180ca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f071ceee640 1 -- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07180c80a0 msgr2=0x7f0718158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f071ceee640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07180c80a0 0x7f0718158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f071ceee640 1 -- 172.21.15.45:0/3793558834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07180cacf0 con 0x7f071815d250 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.377+0000 7f070f7fe640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07180c80a0 0x7f0718158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.378+0000 7f071ceee640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f071815d250 0x7f071815d6d0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f071000e200 tx=0x7f0710009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.378+0000 7f07157fa640 1 -- 172.21.15.45:0/3793558834 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710003da0 con 0x7f071815d250 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.378+0000 7f07157fa640 1 -- 172.21.15.45:0/3793558834 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0710035980 con 0x7f071815d250 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.378+0000 7f07157fa640 1 -- 172.21.15.45:0/3793558834 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710035d30 con 0x7f071815d250 2023-11-28T22:08:39.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.378+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0718070260 con 0x7f071815d250 2023-11-28T22:08:39.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.378+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0718070820 con 0x7f071815d250 2023-11-28T22:08:39.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.379+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071804e920 con 0x7f071815d250 2023-11-28T22:08:39.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.380+0000 7f07157fa640 1 -- 172.21.15.45:0/3793558834 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f0710004030 con 0x7f071815d250 2023-11-28T22:08:39.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.381+0000 7f07157fa640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f06f806af60 0x7f06f806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.381+0000 7f07157fa640 1 -- 172.21.15.45:0/3793558834 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f07100ac240 con 0x7f071815d250 2023-11-28T22:08:39.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.382+0000 7f0717fff640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f06f806af60 0x7f06f806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.383+0000 7f0717fff640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f06f806af60 0x7f06f806d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0708000ff0 tx=0x7f0708009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:39.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.386+0000 7f07157fa640 1 -- 172.21.15.45:0/3793558834 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0710077450 con 0x7f071815d250 2023-11-28T22:08:39.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.649+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f07180ce890 con 0x7f071815d250 2023-11-28T22:08:39.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.650+0000 7f07157fa640 1 -- 172.21.15.45:0/3793558834 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v47) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f071007aa70 con 0x7f071815d250 2023-11-28T22:08:39.653 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:39.653 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:08:39.653 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_dump": [ 2023-11-28T22:08:39.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.653 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.a", 2023-11-28T22:08:39.653 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZltyqDKhAAP7QUUi9L29v5j+4mISer0w==", 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.b", 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZldjwBExAAcbV/mc0wsb4ucQCB8nBoBQ==", 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.c", 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZllGcPARAAdb5iFHZfxcwG/no293bQhw==", 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T22:08:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T22:08:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mds.d", 2023-11-28T22:08:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlIyHHIxAAh66A+3W2+YdNerbpXLIFBw==", 2023-11-28T22:08:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T22:08:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T22:08:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T22:08:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.0", 2023-11-28T22:08:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB0TmZlYEacFhAAgQudZV6mdM5VN3dJLDSypA==", 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.1", 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB3TmZl85EhMBAA0xtha9MvdTz2a9w8TsTj6Q==", 2023-11-28T22:08:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.2", 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB7TmZlU5YeDhAA/S+XazPhapQH5gxK40iAdw==", 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.3", 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQB+TmZlDRd1KBAA1CeLDSqw9bMPDVxnRrQPng==", 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.4", 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCFTmZlP2fuNRAAgWE0IujHmOwFIh1lPaOtCg==", 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.5", 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCJTmZlNvG/BBAAIrEdbdS0TQWpc//AY4F6+Q==", 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.6", 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCMTmZlHUINHRAAmV5iDbMAlfJb3sAb61URmw==", 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T22:08:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "osd.7", 2023-11-28T22:08:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQCPTmZlBXu/NRAA8xOEukCtYZ//n/iAL2LADQ==", 2023-11-28T22:08:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow profile osd", 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile osd", 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.0", 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w==", 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow r", 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T22:08:39.664 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.1", 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlntFRDRAAWUn04QpHKQLNuinxkHe8uw==", 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.666 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.2", 2023-11-28T22:08:39.666 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlggICAxAAYhDCQX3OYUNhBK12jPRpJA==", 2023-11-28T22:08:39.666 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.666 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T22:08:39.666 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T22:08:39.666 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T22:08:39.666 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T22:08:39.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.3", 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQByTmZlVDPCFhAA7uvV1Oham33JsRrePwh0Qg==", 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow", 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow r", 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow rw", 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow rwx" 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "client.admin", 2023-11-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBvTmZl9KEmGxAAtna1kWvxfAE4NwRrOJKUGg==", 2023-11-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": "allow *", 2023-11-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow *", 2023-11-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.x", 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlrTXSHRAANTyDOg28QGpAf1AUW39C6w==", 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: "entity": "mgr.y", 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "AQBwTmZlhdhnERAAtxjmfJFBwSops0KTha7B9w==", 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": { 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": "allow *", 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": "allow profile mgr", 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": "allow *" 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-11-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:08:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.652+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f06f806af60 msgr2=0x7f06f806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.652+0000 7f071d6ef640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f06f806af60 0x7f06f806d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0708000ff0 tx=0x7f0708009040 comp rx=0 tx=0).stop 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.652+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f071815d250 msgr2=0x7f071815d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.652+0000 7f071d6ef640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f071815d250 0x7f071815d6d0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f071000e200 tx=0x7f0710009750 comp rx=0 tx=0).stop 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.652+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 shutdown_connections 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.653+0000 7f071d6ef640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f06f806af60 0x7f06f806d420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.653+0000 7f071d6ef640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f071815d250 0x7f071815d6d0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.653+0000 7f071d6ef640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07180c80a0 0x7f0718158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.653+0000 7f071d6ef640 1 --2- 172.21.15.45:0/3793558834 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0704005180 0x7f07180ca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.653+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 >> 172.21.15.45:0/3793558834 conn(0x7f07180b0e70 msgr2=0x7f07180cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.653+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 shutdown_connections 2023-11-28T22:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.653+0000 7f071d6ef640 1 -- 172.21.15.45:0/3793558834 wait complete. 2023-11-28T22:08:39.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:08:39.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.901+0000 7f654aa67640 1 Processor -- start 2023-11-28T22:08:39.903 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.901+0000 7f654aa67640 1 -- start start 2023-11-28T22:08:39.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.903+0000 7f654aa67640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65440721e0 0x7f654406d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.903+0000 7f654aa67640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f654406d6b0 0x7f65440ac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.903+0000 7f654aa67640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654415e680 con 0x7f65440b3870 2023-11-28T22:08:39.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.903+0000 7f654aa67640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654415a460 con 0x7f654406d6b0 2023-11-28T22:08:39.905 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.903+0000 7f654aa67640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654406db40 con 0x7f65440721e0 2023-11-28T22:08:39.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.903+0000 7f654a266640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f654406d6b0 0x7f65440ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.903+0000 7f654a266640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f654406d6b0 0x7f65440ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37926/0 (socket says 172.21.15.45:37926) 2023-11-28T22:08:39.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.903+0000 7f654a266640 1 -- 172.21.15.45:0/362343335 learned_addr learned my addr 172.21.15.45:0/362343335 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:39.906 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.903+0000 7f6549264640 1 --2- 172.21.15.45:0/362343335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65440721e0 0x7f654406d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.904+0000 7f654a266640 1 -- 172.21.15.45:0/362343335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65440721e0 msgr2=0x7f654406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:39.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.904+0000 7f654a266640 1 --2- 172.21.15.45:0/362343335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65440721e0 0x7f654406d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.904+0000 7f654a266640 1 -- 172.21.15.45:0/362343335 >> v1:172.21.15.45:6789/0 conn(0x7f65440b3870 legacy=0x7f654415db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.904+0000 7f654a266640 1 -- 172.21.15.45:0/362343335 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654415e960 con 0x7f654406d6b0 2023-11-28T22:08:39.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.905+0000 7f654a266640 1 --2- 172.21.15.45:0/362343335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f654406d6b0 0x7f65440ac840 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f653c009870 tx=0x7f653c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba82c24c533c1c1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:39.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.905+0000 7f6548a63640 1 -- 172.21.15.45:0/362343335 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c030840 con 0x7f654406d6b0 2023-11-28T22:08:39.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.905+0000 7f6548a63640 1 -- 172.21.15.45:0/362343335 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c0309a0 con 0x7f654406d6b0 2023-11-28T22:08:39.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.905+0000 7f6548a63640 1 -- 172.21.15.45:0/362343335 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c030d30 con 0x7f654406d6b0 2023-11-28T22:08:39.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.906+0000 7f654aa67640 1 -- 172.21.15.45:0/362343335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f654406d6b0 msgr2=0x7f65440ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.906+0000 7f654aa67640 1 --2- 172.21.15.45:0/362343335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f654406d6b0 0x7f65440ac840 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f653c009870 tx=0x7f653c02b980 comp rx=0 tx=0).stop 2023-11-28T22:08:39.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.906+0000 7f654aa67640 1 -- 172.21.15.45:0/362343335 shutdown_connections 2023-11-28T22:08:39.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.906+0000 7f654aa67640 1 --2- 172.21.15.45:0/362343335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f654406d6b0 0x7f65440ac840 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.906+0000 7f654aa67640 1 --2- 172.21.15.45:0/362343335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65440721e0 0x7f654406d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.906+0000 7f654aa67640 1 -- 172.21.15.45:0/362343335 >> 172.21.15.45:0/362343335 conn(0x7f654406ab30 msgr2=0x7f65440b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:39.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.906+0000 7f654aa67640 1 -- 172.21.15.45:0/362343335 shutdown_connections 2023-11-28T22:08:39.909 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.906+0000 7f654aa67640 1 -- 172.21.15.45:0/362343335 wait complete. 2023-11-28T22:08:39.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.908+0000 7f654aa67640 1 Processor -- start 2023-11-28T22:08:39.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.908+0000 7f654aa67640 1 -- start start 2023-11-28T22:08:39.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.909+0000 7f654aa67640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f654406d6b0 0x7f65440b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.909+0000 7f654aa67640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65440721e0 0x7f6544099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.909+0000 7f654aa67640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65440b3870 0x7f6544152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.909+0000 7f654aa67640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654409ad50 con 0x7f65440b3870 2023-11-28T22:08:39.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.909+0000 7f654aa67640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654409a140 con 0x7f65440721e0 2023-11-28T22:08:39.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.909+0000 7f654aa67640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654409a420 con 0x7f654406d6b0 2023-11-28T22:08:39.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.909+0000 7f654a266640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65440b3870 0x7f6544152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.909+0000 7f654a266640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65440b3870 0x7f6544152200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37934/0 (socket says 172.21.15.45:37934) 2023-11-28T22:08:39.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.909+0000 7f654a266640 1 -- 172.21.15.45:0/2311206334 learned_addr learned my addr 172.21.15.45:0/2311206334 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:39.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f654a266640 1 -- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f654406d6b0 msgr2=0x7f65440b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f6549a65640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f654406d6b0 0x7f65440b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f6549264640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65440721e0 0x7f6544099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.915 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f654a266640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f654406d6b0 0x7f65440b0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f654a266640 1 -- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65440721e0 msgr2=0x7f6544099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:39.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f654a266640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65440721e0 0x7f6544099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:39.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f654a266640 1 -- 172.21.15.45:0/2311206334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c0094d0 con 0x7f65440b3870 2023-11-28T22:08:39.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f6549a65640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f654406d6b0 0x7f65440b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:39.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f654a266640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65440b3870 0x7f6544152200 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f653c009870 tx=0x7f653c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:39.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f653a7fc640 1 -- 172.21.15.45:0/2311206334 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c039c90 con 0x7f65440b3870 2023-11-28T22:08:39.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f653a7fc640 1 -- 172.21.15.45:0/2311206334 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c039e10 con 0x7f65440b3870 2023-11-28T22:08:39.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f653a7fc640 1 -- 172.21.15.45:0/2311206334 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c043640 con 0x7f65440b3870 2023-11-28T22:08:39.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.910+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654409a700 con 0x7f65440b3870 2023-11-28T22:08:39.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.911+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6544152740 con 0x7f65440b3870 2023-11-28T22:08:39.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.912+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654409a560 con 0x7f65440b3870 2023-11-28T22:08:39.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.913+0000 7f653a7fc640 1 -- 172.21.15.45:0/2311206334 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f653c0437a0 con 0x7f65440b3870 2023-11-28T22:08:39.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.914+0000 7f653a7fc640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f652806af60 0x7f652806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:39.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.914+0000 7f653a7fc640 1 -- 172.21.15.45:0/2311206334 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f653c03e070 con 0x7f65440b3870 2023-11-28T22:08:39.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.914+0000 7f6549a65640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f652806af60 0x7f652806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:39.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.916+0000 7f6549a65640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f652806af60 0x7f652806d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65440b3ec0 tx=0x7f6540009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:39.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:39.918+0000 7f653a7fc640 1 -- 172.21.15.45:0/2311206334 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f654409a560 con 0x7f65440b3870 2023-11-28T22:08:40.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.116+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f654409a560 con 0x7f65440b3870 2023-11-28T22:08:40.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.117+0000 7f653a7fc640 1 -- 172.21.15.45:0/2311206334 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v211) v1 ==== 74+0+13104 (secure 0 0 0) 0x7f654409a560 con 0x7f65440b3870 2023-11-28T22:08:40.120 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:40.121 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":211,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:08:39.129133+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:08:40.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.121+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f652806af60 msgr2=0x7f652806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.124 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.121+0000 7f654aa67640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f652806af60 0x7f652806d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65440b3ec0 tx=0x7f6540009040 comp rx=0 tx=0).stop 2023-11-28T22:08:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.121+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65440b3870 msgr2=0x7f6544152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.121+0000 7f654aa67640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65440b3870 0x7f6544152200 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f653c009870 tx=0x7f653c004730 comp rx=0 tx=0).stop 2023-11-28T22:08:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.122+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 shutdown_connections 2023-11-28T22:08:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.122+0000 7f654aa67640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f652806af60 0x7f652806d420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.122+0000 7f654aa67640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65440b3870 0x7f6544152200 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.122+0000 7f654aa67640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65440721e0 0x7f6544099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.122+0000 7f654aa67640 1 --2- 172.21.15.45:0/2311206334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f654406d6b0 0x7f65440b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.122+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 >> 172.21.15.45:0/2311206334 conn(0x7f654406ab30 msgr2=0x7f65440ad330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.122+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 shutdown_connections 2023-11-28T22:08:40.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.122+0000 7f654aa67640 1 -- 172.21.15.45:0/2311206334 wait complete. 2023-11-28T22:08:40.144 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T22:08:40.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T22:08:40.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.356+0000 7fdcdd29b640 1 Processor -- start 2023-11-28T22:08:40.359 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.356+0000 7fdcdd29b640 1 -- start start 2023-11-28T22:08:40.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.358+0000 7fdcdd29b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdcd806dc50 0x7fdcd80cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.358+0000 7fdcdd29b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcd80ce600 0x7fdcd806a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.358+0000 7fdcdd29b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd806e030 con 0x7fdcd806aa40 2023-11-28T22:08:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.358+0000 7fdcdd29b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd806aed0 con 0x7fdcd806dc50 2023-11-28T22:08:40.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.358+0000 7fdcdd29b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80ccf60 con 0x7fdcd80ce600 2023-11-28T22:08:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.358+0000 7fdcd77fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdcd806aa40 0x7fdcd80c93b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51980/0 (socket says 172.21.15.45:51980) 2023-11-28T22:08:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.358+0000 7fdcd77fe640 1 -- 172.21.15.45:0/1351782702 learned_addr learned my addr 172.21.15.45:0/1351782702 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd67fc640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcd80ce600 0x7fdcd806a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/1351782702 >> v2:172.21.15.190:3300/0 conn(0x7fdcd806dc50 0x7fdcd80cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108504443 0 0) 0x7fdcd0001110 con 0x7fdcd806aa40 2023-11-28T22:08:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc0003520 con 0x7fdcd806aa40 2023-11-28T22:08:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3561971049 0 0) 0x7fdcc0003520 con 0x7fdcd806aa40 2023-11-28T22:08:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcd80ce600 msgr2=0x7fdcd806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd5ffb640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcd80ce600 0x7fdcd806a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 >> v2:172.21.15.190:3300/0 conn(0x7fdcd806dc50 msgr2=0x7fdcd80cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd5ffb640 1 --2- 172.21.15.45:0/1351782702 >> v2:172.21.15.190:3300/0 conn(0x7fdcd806dc50 0x7fdcd80cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80cd930 con 0x7fdcd806aa40 2023-11-28T22:08:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.359+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fdcd0001800 con 0x7fdcd806aa40 2023-11-28T22:08:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 >> v1:172.21.15.45:6789/0 conn(0x7fdcd806aa40 legacy=0x7fdcd80c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd5ffb640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcd80ce600 0x7fdcc0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd5ffb640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcc0003ef0 0x7fdcc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd5ffb640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc0004d00 0x7fdcc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd806aed0 con 0x7fdcc0004d00 2023-11-28T22:08:40.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80ccf60 con 0x7fdcc0003ef0 2023-11-28T22:08:40.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc000b980 con 0x7fdcd80ce600 2023-11-28T22:08:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc0004d00 0x7fdcc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcd80ce600 0x7fdcc0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd77fe640 1 -- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcd80ce600 msgr2=0x7fdcc0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcd80ce600 0x7fdcc0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd77fe640 1 -- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcc0003ef0 msgr2=0x7fdcc00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:40.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcc0003ef0 0x7fdcc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.360+0000 7fdcd77fe640 1 -- 172.21.15.45:0/1351782702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc0003520 con 0x7fdcc0004d00 2023-11-28T22:08:40.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.361+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc0004d00 0x7fdcc0005180 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000c240 tx=0x7fdcc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a88d81cb225ac41 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.362+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0005ad0 con 0x7fdcc0004d00 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.362+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd0005c30 con 0x7fdcc0004d00 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.362+0000 7fdcd5ffb640 1 -- 172.21.15.45:0/1351782702 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd002cae0 con 0x7fdcc0004d00 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.363+0000 7fdcdd29b640 1 -- 172.21.15.45:0/1351782702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc0004d00 msgr2=0x7fdcc0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.363+0000 7fdcdd29b640 1 --2- 172.21.15.45:0/1351782702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcc0004d00 0x7fdcc0005180 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000c240 tx=0x7fdcc00057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.363+0000 7fdcd77fe640 1 -- 172.21.15.45:0/1351782702 reap_dead start 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.363+0000 7fdcdd29b640 1 -- 172.21.15.45:0/1351782702 shutdown_connections 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.363+0000 7fdcdd29b640 1 -- 172.21.15.45:0/1351782702 >> 172.21.15.45:0/1351782702 conn(0x7fdcd80ced60 msgr2=0x7fdcd80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.363+0000 7fdcdd29b640 1 -- 172.21.15.45:0/1351782702 shutdown_connections 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.363+0000 7fdcdd29b640 1 -- 172.21.15.45:0/1351782702 wait complete. 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.365+0000 7fdcdd29b640 1 Processor -- start 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.365+0000 7fdcdd29b640 1 -- start start 2023-11-28T22:08:40.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.366+0000 7fdcdd29b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcc0004d00 0x7fdcd80b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.366+0000 7fdcdd29b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd80b41e0 0x7fdcd815a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.366+0000 7fdcdd29b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd81596c0 0x7fdcd806c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.366+0000 7fdcdd29b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd815ae80 con 0x7fdcd81596c0 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.366+0000 7fdcdd29b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80b3140 con 0x7fdcd80b41e0 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.366+0000 7fdcdd29b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80b3420 con 0x7fdcc0004d00 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.366+0000 7fdcd77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd81596c0 0x7fdcd806c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.366+0000 7fdcd77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd81596c0 0x7fdcd806c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37940/0 (socket says 172.21.15.45:37940) 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.366+0000 7fdcd77fe640 1 -- 172.21.15.45:0/3802600933 learned_addr learned my addr 172.21.15.45:0/3802600933 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.367+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcc0004d00 0x7fdcd80b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.367+0000 7fdcd67fc640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd80b41e0 0x7fdcd815a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.367+0000 7fdcd77fe640 1 -- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcc0004d00 msgr2=0x7fdcd80b2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.367+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcc0004d00 0x7fdcd80b2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.367+0000 7fdcd77fe640 1 -- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd80b41e0 msgr2=0x7fdcd815a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.367+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd80b41e0 0x7fdcd815a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.367+0000 7fdcd77fe640 1 -- 172.21.15.45:0/3802600933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80b3700 con 0x7fdcd81596c0 2023-11-28T22:08:40.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.367+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd81596c0 0x7fdcd806c1c0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000c240 tx=0x7fdcd80cced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:40.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.367+0000 7fdcbf7fe640 1 -- 172.21.15.45:0/3802600933 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0044070 con 0x7fdcd81596c0 2023-11-28T22:08:40.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.368+0000 7fdcbf7fe640 1 -- 172.21.15.45:0/3802600933 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd003a050 con 0x7fdcd81596c0 2023-11-28T22:08:40.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.368+0000 7fdcbf7fe640 1 -- 172.21.15.45:0/3802600933 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0001110 con 0x7fdcd81596c0 2023-11-28T22:08:40.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.368+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd80b39e0 con 0x7fdcd81596c0 2023-11-28T22:08:40.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.368+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd8159b50 con 0x7fdcd81596c0 2023-11-28T22:08:40.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.369+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd80b3f00 con 0x7fdcd81596c0 2023-11-28T22:08:40.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.370+0000 7fdcbf7fe640 1 -- 172.21.15.45:0/3802600933 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fdcd0032070 con 0x7fdcd81596c0 2023-11-28T22:08:40.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.371+0000 7fdcbf7fe640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fdcc406af90 0x7fdcc406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.371+0000 7fdcbf7fe640 1 -- 172.21.15.45:0/3802600933 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fdcd00aabe0 con 0x7fdcd81596c0 2023-11-28T22:08:40.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.372+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fdcc406af90 0x7fdcc406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.373+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fdcc406af90 0x7fdcc406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8007520 tx=0x7fdcc800b040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:40.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.375+0000 7fdcbf7fe640 1 -- 172.21.15.45:0/3802600933 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdcd80b3f00 con 0x7fdcd81596c0 2023-11-28T22:08:40.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.569+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdcd806e030 con 0x7fdcd81596c0 2023-11-28T22:08:40.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.570+0000 7fdcbf7fe640 1 -- 172.21.15.45:0/3802600933 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v211) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdcd806e030 con 0x7fdcd81596c0 2023-11-28T22:08:40.574 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T22:08:40.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fdcc406af90 msgr2=0x7fdcc406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fdcc406af90 0x7fdcc406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8007520 tx=0x7fdcc800b040 comp rx=0 tx=0).stop 2023-11-28T22:08:40.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd81596c0 msgr2=0x7fdcd806c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd81596c0 0x7fdcd806c1c0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000c240 tx=0x7fdcd80cced0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 shutdown_connections 2023-11-28T22:08:40.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fdcc406af90 0x7fdcc406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd81596c0 0x7fdcd806c1c0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd80b41e0 0x7fdcd815a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 --2- 172.21.15.45:0/3802600933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdcc0004d00 0x7fdcd80b2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 >> 172.21.15.45:0/3802600933 conn(0x7fdcd80ced60 msgr2=0x7fdcd80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:40.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.573+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 shutdown_connections 2023-11-28T22:08:40.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.574+0000 7fdcdd29b640 1 -- 172.21.15.45:0/3802600933 wait complete. 2023-11-28T22:08:40.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T22:08:40.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.816+0000 7f235e0df640 1 Processor -- start 2023-11-28T22:08:40.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.816+0000 7f235e0df640 1 -- start start 2023-11-28T22:08:40.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.817+0000 7f235e0df640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f235815d550 0x7f23580cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.818+0000 7f235e0df640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23580d2230 con 0x7f23580d1c80 2023-11-28T22:08:40.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.818+0000 7f235e0df640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23580d2510 con 0x7f23580d0100 2023-11-28T22:08:40.820 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.818+0000 7f235e0df640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23580d0590 con 0x7f235815d550 2023-11-28T22:08:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.818+0000 7f235d8de640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f23580d1c80 0x7f23580b6f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51992/0 (socket says 172.21.15.45:51992) 2023-11-28T22:08:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.818+0000 7f235d8de640 1 -- 172.21.15.45:0/2462555430 learned_addr learned my addr 172.21.15.45:0/2462555430 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:40.822 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.818+0000 7f235d0dd640 1 --2- 172.21.15.45:0/2462555430 >> v2:172.21.15.190:3300/0 conn(0x7f235815d550 0x7f23580cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.818+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141964852 0 0) 0x7f23580d2230 con 0x7f23580d1c80 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2344003520 con 0x7f23580d1c80 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4017409060 0 0) 0x7f2344003520 con 0x7f23580d1c80 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 >> v2:172.21.15.190:3300/0 conn(0x7f235815d550 msgr2=0x7f23580cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 --2- 172.21.15.45:0/2462555430 >> v2:172.21.15.190:3300/0 conn(0x7f235815d550 0x7f23580cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 >> v1:172.21.15.190:6789/0 conn(0x7f23580d0100 legacy=0x7f23580d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580d27f0 con 0x7f23580d1c80 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f2350001670 con 0x7f23580d1c80 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 >> v1:172.21.15.45:6789/0 conn(0x7f23580d1c80 legacy=0x7f23580b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2344003970 0x7f2344003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23440047c0 0x7f2344004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2344005180 0x7f234400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.823 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580d0590 con 0x7f2344005180 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234400c020 con 0x7f23440047c0 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.819+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234400c1e0 con 0x7f2344003970 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.820+0000 7f235d8de640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2344005180 0x7f234400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.820+0000 7f235d0dd640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2344003970 0x7f2344003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.820+0000 7f235c8dc640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23440047c0 0x7f2344004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.820+0000 7f235d8de640 1 -- 172.21.15.45:0/2462555430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2344003970 msgr2=0x7f2344003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.820+0000 7f235d8de640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2344003970 0x7f2344003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.820+0000 7f235d8de640 1 -- 172.21.15.45:0/2462555430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23440047c0 msgr2=0x7f2344004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.820+0000 7f235d8de640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23440047c0 0x7f2344004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.820+0000 7f235d8de640 1 -- 172.21.15.45:0/2462555430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2344003520 con 0x7f2344005180 2023-11-28T22:08:40.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.821+0000 7f235d8de640 1 -- 172.21.15.45:0/2462555430 reap_dead start 2023-11-28T22:08:40.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.821+0000 7f235d8de640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2344005180 0x7f234400b980 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f235000c240 tx=0x7f23440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21cfe27b2726165 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:40.825 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.821+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2350005ad0 con 0x7f2344005180 2023-11-28T22:08:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.822+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2350005c70 con 0x7f2344005180 2023-11-28T22:08:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.822+0000 7f234ffff640 1 -- 172.21.15.45:0/2462555430 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235002c8b0 con 0x7f2344005180 2023-11-28T22:08:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.822+0000 7f235e0df640 1 -- 172.21.15.45:0/2462555430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2344005180 msgr2=0x7f234400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.822+0000 7f235e0df640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2344005180 0x7f234400b980 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f235000c240 tx=0x7f23440057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.822+0000 7f235e0df640 1 -- 172.21.15.45:0/2462555430 shutdown_connections 2023-11-28T22:08:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.822+0000 7f235e0df640 1 --2- 172.21.15.45:0/2462555430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2344005180 0x7f234400b980 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.822+0000 7f235e0df640 1 -- 172.21.15.45:0/2462555430 >> 172.21.15.45:0/2462555430 conn(0x7f235806ac70 msgr2=0x7f23580b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.822+0000 7f235e0df640 1 -- 172.21.15.45:0/2462555430 shutdown_connections 2023-11-28T22:08:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.822+0000 7f235e0df640 1 -- 172.21.15.45:0/2462555430 wait complete. 2023-11-28T22:08:40.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.824+0000 7f235e0df640 1 Processor -- start 2023-11-28T22:08:40.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.824+0000 7f235e0df640 1 -- start start 2023-11-28T22:08:40.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.825+0000 7f235e0df640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2344005180 0x7f2358154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.825+0000 7f235e0df640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358156e50 0x7f23581593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.825+0000 7f235e0df640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23581572d0 0x7f23581598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.825+0000 7f235e0df640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580c97e0 con 0x7f23581572d0 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.825+0000 7f235e0df640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580c8970 con 0x7f2358156e50 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.825+0000 7f235e0df640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2358159e20 con 0x7f2344005180 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d8de640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23581572d0 0x7f23581598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d8de640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23581572d0 0x7f23581598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37948/0 (socket says 172.21.15.45:37948) 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d8de640 1 -- 172.21.15.45:0/2898892455 learned_addr learned my addr 172.21.15.45:0/2898892455 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d0dd640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2344005180 0x7f2358154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235c8dc640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358156e50 0x7f23581593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d8de640 1 -- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2344005180 msgr2=0x7f2358154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d8de640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2344005180 0x7f2358154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d8de640 1 -- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358156e50 msgr2=0x7f23581593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:40.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d8de640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358156e50 0x7f23581593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d8de640 1 -- 172.21.15.45:0/2898892455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580c8b60 con 0x7f23581572d0 2023-11-28T22:08:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.826+0000 7f235d8de640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23581572d0 0x7f23581598e0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f235002c520 tx=0x7f2350001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.827+0000 7f234dffb640 1 -- 172.21.15.45:0/2898892455 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23500049e0 con 0x7f23581572d0 2023-11-28T22:08:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.827+0000 7f234dffb640 1 -- 172.21.15.45:0/2898892455 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2350004b40 con 0x7f23581572d0 2023-11-28T22:08:40.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.827+0000 7f234dffb640 1 -- 172.21.15.45:0/2898892455 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235002de80 con 0x7f23581572d0 2023-11-28T22:08:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.827+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23580c8e40 con 0x7f23581572d0 2023-11-28T22:08:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.827+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23580c9400 con 0x7f23581572d0 2023-11-28T22:08:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.829+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23580c9920 con 0x7f23581572d0 2023-11-28T22:08:40.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.830+0000 7f234dffb640 1 -- 172.21.15.45:0/2898892455 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2350032070 con 0x7f23581572d0 2023-11-28T22:08:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.830+0000 7f234dffb640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f233c06af60 0x7f233c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.831+0000 7f234dffb640 1 -- 172.21.15.45:0/2898892455 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f235003c070 con 0x7f23581572d0 2023-11-28T22:08:40.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.831+0000 7f235d0dd640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f233c06af60 0x7f233c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:40.835 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.832+0000 7f235d0dd640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f233c06af60 0x7f233c06d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2354009fd0 tx=0x7f235400b040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:40.839 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:40.837+0000 7f234dffb640 1 -- 172.21.15.45:0/2898892455 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23580c9920 con 0x7f23581572d0 2023-11-28T22:08:41.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.040+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f23580c9920 con 0x7f23581572d0 2023-11-28T22:08:41.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.041+0000 7f234dffb640 1 -- 172.21.15.45:0/2898892455 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v211) v1 ==== 84+0+65 (secure 0 0 0) 0x7f23580c9920 con 0x7f23581572d0 2023-11-28T22:08:41.043 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:41.044 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T22:08:41.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:41.044 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2023-11-28T22:08:41.044 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2023-11-28T22:08:41.044 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:41.044 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T22:08:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.043+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f233c06af60 msgr2=0x7f233c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.043+0000 7f235e0df640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f233c06af60 0x7f233c06d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2354009fd0 tx=0x7f235400b040 comp rx=0 tx=0).stop 2023-11-28T22:08:41.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.043+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23581572d0 msgr2=0x7f23581598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.043+0000 7f235e0df640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23581572d0 0x7f23581598e0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f235002c520 tx=0x7f2350001a00 comp rx=0 tx=0).stop 2023-11-28T22:08:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.044+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 shutdown_connections 2023-11-28T22:08:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.044+0000 7f235e0df640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f233c06af60 0x7f233c06d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.044+0000 7f235e0df640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23581572d0 0x7f23581598e0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.044+0000 7f235e0df640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358156e50 0x7f23581593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.044+0000 7f235e0df640 1 --2- 172.21.15.45:0/2898892455 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2344005180 0x7f2358154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.044+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 >> 172.21.15.45:0/2898892455 conn(0x7f235806ac70 msgr2=0x7f235815dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.044+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 shutdown_connections 2023-11-28T22:08:41.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.044+0000 7f235e0df640 1 -- 172.21.15.45:0/2898892455 wait complete. 2023-11-28T22:08:41.066 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T22:08:41.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T22:08:41.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.292+0000 7f34a6b39640 1 Processor -- start 2023-11-28T22:08:41.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.292+0000 7f34a6b39640 1 -- start start 2023-11-28T22:08:41.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.294+0000 7f34a6b39640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34a0159780 0x7f34a015d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:41.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.294+0000 7f34a6b39640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f34a00ae030 0x7f34a015d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:41.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.294+0000 7f34a6b39640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34a015de10 0x7f34a00701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:41.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.294+0000 7f34a6b39640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a00ce330 con 0x7f34a0159780 2023-11-28T22:08:41.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.294+0000 7f34a6b39640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a00cd6b0 con 0x7f34a00ae030 2023-11-28T22:08:41.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.294+0000 7f34a6b39640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a00cd990 con 0x7f34a015de10 2023-11-28T22:08:41.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f34a48d5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34a0159780 0x7f34a015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:41.299 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f34a48d5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34a0159780 0x7f34a015d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37956/0 (socket says 172.21.15.45:37956) 2023-11-28T22:08:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f34a48d5640 1 -- 172.21.15.45:0/412884115 learned_addr learned my addr 172.21.15.45:0/412884115 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f349ffff640 1 --2- 172.21.15.45:0/412884115 >> v2:172.21.15.190:3300/0 conn(0x7f34a00ae030 0x7f34a015d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f34a50d6640 1 --2- 172.21.15.45:0/412884115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34a015de10 0x7f34a00701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f34a48d5640 1 -- 172.21.15.45:0/412884115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34a015de10 msgr2=0x7f34a00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f34a48d5640 1 --2- 172.21.15.45:0/412884115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34a015de10 0x7f34a00701c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f34a48d5640 1 -- 172.21.15.45:0/412884115 >> v2:172.21.15.190:3300/0 conn(0x7f34a00ae030 msgr2=0x7f34a015d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:41.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f34a48d5640 1 --2- 172.21.15.45:0/412884115 >> v2:172.21.15.190:3300/0 conn(0x7f34a00ae030 0x7f34a015d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.295+0000 7f34a48d5640 1 -- 172.21.15.45:0/412884115 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00cdc70 con 0x7f34a0159780 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.297+0000 7f34a48d5640 1 --2- 172.21.15.45:0/412884115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a0159780 0x7f34a015d390 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f34a015d2d0 tx=0x7f349402db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a8cb4000cfef3a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.297+0000 7f349f7fe640 1 -- 172.21.15.45:0/412884115 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349402f8e0 con 0x7f34a0159780 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.297+0000 7f349f7fe640 1 -- 172.21.15.45:0/412884115 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349402fa40 con 0x7f34a0159780 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.297+0000 7f349f7fe640 1 -- 172.21.15.45:0/412884115 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349402fdd0 con 0x7f34a0159780 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.297+0000 7f34a6b39640 1 -- 172.21.15.45:0/412884115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a0159780 msgr2=0x7f34a015d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.297+0000 7f34a6b39640 1 --2- 172.21.15.45:0/412884115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a0159780 0x7f34a015d390 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f34a015d2d0 tx=0x7f349402db20 comp rx=0 tx=0).stop 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.298+0000 7f34a6b39640 1 -- 172.21.15.45:0/412884115 shutdown_connections 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.298+0000 7f34a6b39640 1 --2- 172.21.15.45:0/412884115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34a015de10 0x7f34a00701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.298+0000 7f34a6b39640 1 --2- 172.21.15.45:0/412884115 >> v2:172.21.15.190:3300/0 conn(0x7f34a00ae030 0x7f34a015d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.298+0000 7f34a6b39640 1 --2- 172.21.15.45:0/412884115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a0159780 0x7f34a015d390 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.298+0000 7f34a6b39640 1 -- 172.21.15.45:0/412884115 >> 172.21.15.45:0/412884115 conn(0x7f34a00ced60 msgr2=0x7f34a00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.298+0000 7f34a6b39640 1 -- 172.21.15.45:0/412884115 shutdown_connections 2023-11-28T22:08:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.298+0000 7f34a6b39640 1 -- 172.21.15.45:0/412884115 wait complete. 2023-11-28T22:08:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.299+0000 7f34a6b39640 1 Processor -- start 2023-11-28T22:08:41.303 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.300+0000 7f34a6b39640 1 -- start start 2023-11-28T22:08:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.300+0000 7f34a6b39640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34a00ae030 0x7f34a01efe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.300+0000 7f34a6b39640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a0159780 0x7f34a015c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a6b39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a015de10 0x7f34a00af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a6b39640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a015cc00 con 0x7f34a015de10 2023-11-28T22:08:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a6b39640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a00c7280 con 0x7f34a0159780 2023-11-28T22:08:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a6b39640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a00c7560 con 0x7f34a00ae030 2023-11-28T22:08:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a50d6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a015de10 0x7f34a00af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:41.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a50d6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a015de10 0x7f34a00af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37964/0 (socket says 172.21.15.45:37964) 2023-11-28T22:08:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a50d6640 1 -- 172.21.15.45:0/4173287124 learned_addr learned my addr 172.21.15.45:0/4173287124 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f349ffff640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a0159780 0x7f34a015c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a50d6640 1 -- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34a00ae030 msgr2=0x7f34a01efe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a50d6640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34a00ae030 0x7f34a01efe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a50d6640 1 -- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a0159780 msgr2=0x7f34a015c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a50d6640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a0159780 0x7f34a015c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f34a50d6640 1 -- 172.21.15.45:0/4173287124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34940094d0 con 0x7f34a015de10 2023-11-28T22:08:41.306 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.301+0000 7f349ffff640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a0159780 0x7f34a015c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.302+0000 7f34a50d6640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a015de10 0x7f34a00af550 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f3498007d90 tx=0x7f349800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.302+0000 7f349d7fa640 1 -- 172.21.15.45:0/4173287124 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3498014420 con 0x7f34a015de10 2023-11-28T22:08:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.302+0000 7f349d7fa640 1 -- 172.21.15.45:0/4173287124 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34980145a0 con 0x7f34a015de10 2023-11-28T22:08:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.302+0000 7f349d7fa640 1 -- 172.21.15.45:0/4173287124 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3498014950 con 0x7f34a015de10 2023-11-28T22:08:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.302+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a00c7840 con 0x7f34a015de10 2023-11-28T22:08:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.302+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a009b680 con 0x7f34a015de10 2023-11-28T22:08:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.303+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a00733d0 con 0x7f34a015de10 2023-11-28T22:08:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.304+0000 7f349d7fa640 1 -- 172.21.15.45:0/4173287124 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3498014ab0 con 0x7f34a015de10 2023-11-28T22:08:41.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.306+0000 7f349d7fa640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f348406af90 0x7f348406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:41.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.306+0000 7f349d7fa640 1 -- 172.21.15.45:0/4173287124 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f3498089eb0 con 0x7f34a015de10 2023-11-28T22:08:41.308 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.307+0000 7f34a48d5640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f348406af90 0x7f348406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:41.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.308+0000 7f34a48d5640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f348406af90 0x7f348406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3494038000 tx=0x7f3494032000 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:41.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.311+0000 7f349d7fa640 1 -- 172.21.15.45:0/4173287124 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3498055030 con 0x7f34a015de10 2023-11-28T22:08:41.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:41.509+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f34a00b4ce0 con 0x7f34a015de10 2023-11-28T22:08:42.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.278+0000 7f349d7fa640 1 -- 172.21.15.45:0/4173287124 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v212) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3498058650 con 0x7f34a015de10 2023-11-28T22:08:42.281 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2023-11-28T22:08:42.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.283+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f348406af90 msgr2=0x7f348406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:42.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.283+0000 7f34a6b39640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f348406af90 0x7f348406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3494038000 tx=0x7f3494032000 comp rx=0 tx=0).stop 2023-11-28T22:08:42.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.283+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a015de10 msgr2=0x7f34a00af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:42.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.283+0000 7f34a6b39640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a015de10 0x7f34a00af550 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f3498007d90 tx=0x7f349800e4b0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.284+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 shutdown_connections 2023-11-28T22:08:42.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.284+0000 7f34a6b39640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f348406af90 0x7f348406d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.284+0000 7f34a6b39640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a015de10 0x7f34a00af550 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.284+0000 7f34a6b39640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34a0159780 0x7f34a015c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.284+0000 7f34a6b39640 1 --2- 172.21.15.45:0/4173287124 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34a00ae030 0x7f34a01efe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.284+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 >> 172.21.15.45:0/4173287124 conn(0x7f34a00ced60 msgr2=0x7f34a015d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:42.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.285+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 shutdown_connections 2023-11-28T22:08:42.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.285+0000 7f34a6b39640 1 -- 172.21.15.45:0/4173287124 wait complete. 2023-11-28T22:08:42.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T22:08:42.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.538+0000 7feb16f1d640 1 Processor -- start 2023-11-28T22:08:42.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.538+0000 7feb16f1d640 1 -- start start 2023-11-28T22:08:42.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.540+0000 7feb16f1d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 0x7feb100cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:42.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.540+0000 7feb16f1d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb1006ac10 0x7feb100c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:42.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.540+0000 7feb16f1d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1006e030 con 0x7feb100ce600 2023-11-28T22:08:42.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.540+0000 7feb16f1d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb100ccf60 con 0x7feb1006ac10 2023-11-28T22:08:42.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.540+0000 7feb16f1d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb100cd930 con 0x7feb1006dc50 2023-11-28T22:08:42.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.540+0000 7feb154ba640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb1006ac10 0x7feb100c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:42.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.540+0000 7feb154ba640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb1006ac10 0x7feb100c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37978/0 (socket says 172.21.15.45:37978) 2023-11-28T22:08:42.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.540+0000 7feb154ba640 1 -- 172.21.15.45:0/1971060368 learned_addr learned my addr 172.21.15.45:0/1971060368 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:42.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.541+0000 7feb14cb9640 1 --2- 172.21.15.45:0/1971060368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 0x7feb100cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:42.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.541+0000 7feb154ba640 1 -- 172.21.15.45:0/1971060368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 msgr2=0x7feb100cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:42.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.541+0000 7feb154ba640 1 --2- 172.21.15.45:0/1971060368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 0x7feb100cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.541+0000 7feb154ba640 1 -- 172.21.15.45:0/1971060368 >> v1:172.21.15.190:6789/0 conn(0x7feb100ce600 legacy=0x7feb1006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:42.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.541+0000 7feb154ba640 1 -- 172.21.15.45:0/1971060368 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100cc930 con 0x7feb1006ac10 2023-11-28T22:08:42.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.542+0000 7feb154ba640 1 --2- 172.21.15.45:0/1971060368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 0x7feb100c93b0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7feb08009e50 tx=0x7feb0802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3180ab42a6d53ecc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:42.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.543+0000 7feb0f7fe640 1 -- 172.21.15.45:0/1971060368 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0802cb00 con 0x7feb1006ac10 2023-11-28T22:08:42.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.543+0000 7feb0f7fe640 1 -- 172.21.15.45:0/1971060368 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0802cc60 con 0x7feb1006ac10 2023-11-28T22:08:42.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.543+0000 7feb0f7fe640 1 -- 172.21.15.45:0/1971060368 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080343f0 con 0x7feb1006ac10 2023-11-28T22:08:42.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.543+0000 7feb16f1d640 1 -- 172.21.15.45:0/1971060368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 msgr2=0x7feb100c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:42.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.543+0000 7feb16f1d640 1 --2- 172.21.15.45:0/1971060368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 0x7feb100c93b0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7feb08009e50 tx=0x7feb0802bf60 comp rx=0 tx=0).stop 2023-11-28T22:08:42.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.543+0000 7feb16f1d640 1 -- 172.21.15.45:0/1971060368 shutdown_connections 2023-11-28T22:08:42.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.543+0000 7feb16f1d640 1 --2- 172.21.15.45:0/1971060368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 0x7feb100c93b0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.543+0000 7feb16f1d640 1 --2- 172.21.15.45:0/1971060368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 0x7feb100cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.543+0000 7feb16f1d640 1 -- 172.21.15.45:0/1971060368 >> 172.21.15.45:0/1971060368 conn(0x7feb100ced60 msgr2=0x7feb100cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:42.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.544+0000 7feb16f1d640 1 -- 172.21.15.45:0/1971060368 shutdown_connections 2023-11-28T22:08:42.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.544+0000 7feb16f1d640 1 -- 172.21.15.45:0/1971060368 wait complete. 2023-11-28T22:08:42.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.545+0000 7feb16f1d640 1 Processor -- start 2023-11-28T22:08:42.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.545+0000 7feb16f1d640 1 -- start start 2023-11-28T22:08:42.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.546+0000 7feb16f1d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 0x7feb100c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:42.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.546+0000 7feb16f1d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 0x7feb100b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:42.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.546+0000 7feb16f1d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100ce600 0x7feb100b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:42.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.546+0000 7feb16f1d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb100c5690 con 0x7feb1006ac10 2023-11-28T22:08:42.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.546+0000 7feb16f1d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1015db10 con 0x7feb100ce600 2023-11-28T22:08:42.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.546+0000 7feb16f1d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1015cf90 con 0x7feb1006dc50 2023-11-28T22:08:42.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb14cb9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 0x7feb100c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:42.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb14cb9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 0x7feb100c5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37980/0 (socket says 172.21.15.45:37980) 2023-11-28T22:08:42.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb14cb9640 1 -- 172.21.15.45:0/250916751 learned_addr learned my addr 172.21.15.45:0/250916751 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:42.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb14cb9640 1 -- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 msgr2=0x7feb100b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:42.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb154ba640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100ce600 0x7feb100b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:42.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb0ffff640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 0x7feb100b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:42.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb14cb9640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 0x7feb100b19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb14cb9640 1 -- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100ce600 msgr2=0x7feb100b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:42.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb14cb9640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100ce600 0x7feb100b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:42.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb14cb9640 1 -- 172.21.15.45:0/250916751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb08009b30 con 0x7feb1006ac10 2023-11-28T22:08:42.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb154ba640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100ce600 0x7feb100b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:42.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.547+0000 7feb0ffff640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 0x7feb100b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:42.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.548+0000 7feb14cb9640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 0x7feb100c5150 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7feb0400d600 tx=0x7feb0400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:42.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.548+0000 7feb0d7fa640 1 -- 172.21.15.45:0/250916751 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb040125f0 con 0x7feb1006ac10 2023-11-28T22:08:42.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.548+0000 7feb0d7fa640 1 -- 172.21.15.45:0/250916751 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb04012770 con 0x7feb1006ac10 2023-11-28T22:08:42.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.548+0000 7feb0d7fa640 1 -- 172.21.15.45:0/250916751 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04012b20 con 0x7feb1006ac10 2023-11-28T22:08:42.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.548+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1015d270 con 0x7feb1006ac10 2023-11-28T22:08:42.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.548+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1015d830 con 0x7feb1006ac10 2023-11-28T22:08:42.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.549+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb10070c40 con 0x7feb1006ac10 2023-11-28T22:08:42.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.551+0000 7feb0d7fa640 1 -- 172.21.15.45:0/250916751 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7feb04012c80 con 0x7feb1006ac10 2023-11-28T22:08:42.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.551+0000 7feb0d7fa640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7feaf406afb0 0x7feaf406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:42.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.552+0000 7feb0d7fa640 1 -- 172.21.15.45:0/250916751 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 5575+0+0 (secure 0 0 0) 0x7feb0408a570 con 0x7feb1006ac10 2023-11-28T22:08:42.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.552+0000 7feb0ffff640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7feaf406afb0 0x7feaf406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:42.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.554+0000 7feb0ffff640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7feaf406afb0 0x7feaf406d470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feb100adf60 tx=0x7feafc009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:42.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.556+0000 7feb0d7fa640 1 -- 172.21.15.45:0/250916751 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb04055700 con 0x7feb1006ac10 2023-11-28T22:08:42.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:42.753+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7feb100733d0 con 0x7feb1006ac10 2023-11-28T22:08:43.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.286+0000 7feb0d7fa640 1 -- 172.21.15.45:0/250916751 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v213) v1 ==== 170+0+0 (secure 0 0 0) 0x7feb04058d20 con 0x7feb1006ac10 2023-11-28T22:08:43.289 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2023-11-28T22:08:43.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7feaf406afb0 msgr2=0x7feaf406d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:43.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7feaf406afb0 0x7feaf406d470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feb100adf60 tx=0x7feafc009040 comp rx=0 tx=0).stop 2023-11-28T22:08:43.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 msgr2=0x7feb100c5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:43.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 0x7feb100c5150 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7feb0400d600 tx=0x7feb0400dac0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 shutdown_connections 2023-11-28T22:08:43.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb100ce600 0x7feb100b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb1006dc50 0x7feb100b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7feaf406afb0 0x7feaf406d470 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 --2- 172.21.15.45:0/250916751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb1006ac10 0x7feb100c5150 secure :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7feb0400d600 tx=0x7feb0400dac0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.291+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 >> 172.21.15.45:0/250916751 conn(0x7feb100ced60 msgr2=0x7feb1006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:43.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.292+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 shutdown_connections 2023-11-28T22:08:43.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.292+0000 7feb16f1d640 1 -- 172.21.15.45:0/250916751 wait complete. 2023-11-28T22:08:43.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T22:08:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.526+0000 7f6b59c71640 1 Processor -- start 2023-11-28T22:08:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.526+0000 7f6b59c71640 1 -- start start 2023-11-28T22:08:43.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.528+0000 7f6b59c71640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b540b2430 0x7f6b540ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:43.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.528+0000 7f6b59c71640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5409ceb0 con 0x7f6b54159fc0 2023-11-28T22:08:43.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.528+0000 7f6b59c71640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b540b2cb0 con 0x7f6b54071ac0 2023-11-28T22:08:43.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.528+0000 7f6b59c71640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b54071450 con 0x7f6b540b2430 2023-11-28T22:08:43.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.528+0000 7f6b58c6f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6b54159fc0 0x7f6b540b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52006/0 (socket says 172.21.15.45:52006) 2023-11-28T22:08:43.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.528+0000 7f6b58c6f640 1 -- 172.21.15.45:0/1092390722 learned_addr learned my addr 172.21.15.45:0/1092390722 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b53fff640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b540b2430 0x7f6b540ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165869021 0 0) 0x7f6b5409ceb0 con 0x7f6b54159fc0 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b44003520 con 0x7f6b54159fc0 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1272882247 0 0) 0x7f6b44003520 con 0x7f6b54159fc0 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b540b2430 msgr2=0x7f6b540ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b540b2430 0x7f6b540ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 >> v1:172.21.15.190:6789/0 conn(0x7f6b54071ac0 legacy=0x7f6b540cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b540b29d0 con 0x7f6b54159fc0 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f6b48001670 con 0x7f6b54159fc0 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 >> v1:172.21.15.45:6789/0 conn(0x7f6b54159fc0 legacy=0x7f6b540b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b44003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b540b2430 0x7f6b440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b44004d00 0x7f6b44005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b54071450 con 0x7f6b440039a0 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4400bab0 con 0x7f6b44004d00 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.529+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4400de80 con 0x7f6b540b2430 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.530+0000 7f6b58c6f640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b44003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.530+0000 7f6b53fff640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b540b2430 0x7f6b440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.530+0000 7f6b59470640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b44004d00 0x7f6b44005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.530+0000 7f6b58c6f640 1 -- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b540b2430 msgr2=0x7f6b440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.530+0000 7f6b58c6f640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b540b2430 0x7f6b440047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.530+0000 7f6b58c6f640 1 -- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b44004d00 msgr2=0x7f6b44005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.530+0000 7f6b58c6f640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b44004d00 0x7f6b44005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.530+0000 7f6b58c6f640 1 -- 172.21.15.45:0/1092390722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b44003520 con 0x7f6b440039a0 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.531+0000 7f6b58c6f640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b44003dc0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800c240 tx=0x7f6b440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d4c1610c75eac1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.532+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48005ad0 con 0x7f6b440039a0 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.532+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b48005c30 con 0x7f6b440039a0 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.532+0000 7f6b537fe640 1 -- 172.21.15.45:0/1092390722 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4802c810 con 0x7f6b440039a0 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.532+0000 7f6b59c71640 1 -- 172.21.15.45:0/1092390722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 msgr2=0x7f6b44003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.532+0000 7f6b59c71640 1 --2- 172.21.15.45:0/1092390722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b44003dc0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800c240 tx=0x7f6b440057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.532+0000 7f6b59470640 1 -- 172.21.15.45:0/1092390722 reap_dead start 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.532+0000 7f6b59c71640 1 -- 172.21.15.45:0/1092390722 shutdown_connections 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.532+0000 7f6b59c71640 1 -- 172.21.15.45:0/1092390722 >> 172.21.15.45:0/1092390722 conn(0x7f6b540b1050 msgr2=0x7f6b540cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.532+0000 7f6b59c71640 1 -- 172.21.15.45:0/1092390722 shutdown_connections 2023-11-28T22:08:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.533+0000 7f6b59c71640 1 -- 172.21.15.45:0/1092390722 wait complete. 2023-11-28T22:08:43.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.534+0000 7f6b59c71640 1 Processor -- start 2023-11-28T22:08:43.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.535+0000 7f6b59c71640 1 -- start start 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.535+0000 7f6b59c71640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b54154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b59c71640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b5415d200 0x7f6b540ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b59c71640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b5415c3a0 0x7f6b5406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b59c71640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5415b0a0 con 0x7f6b440039a0 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b59c71640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5415b340 con 0x7f6b5415c3a0 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b59c71640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5415b5e0 con 0x7f6b5415d200 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b58c6f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b54154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b58c6f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b54154970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38002/0 (socket says 172.21.15.45:38002) 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b58c6f640 1 -- 172.21.15.45:0/484710244 learned_addr learned my addr 172.21.15.45:0/484710244 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:43.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b59470640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b5415c3a0 0x7f6b5406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b53fff640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b5415d200 0x7f6b540ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b58c6f640 1 -- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b5415d200 msgr2=0x7f6b540ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b58c6f640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b5415d200 0x7f6b540ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b58c6f640 1 -- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b5415c3a0 msgr2=0x7f6b5406d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.536+0000 7f6b58c6f640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b5415c3a0 0x7f6b5406d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.537+0000 7f6b58c6f640 1 -- 172.21.15.45:0/484710244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5415b8c0 con 0x7f6b440039a0 2023-11-28T22:08:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.537+0000 7f6b53fff640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b5415d200 0x7f6b540ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.537+0000 7f6b58c6f640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b54154970 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f6b48001930 tx=0x7f6b48001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.537+0000 7f6b517fa640 1 -- 172.21.15.45:0/484710244 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4803c040 con 0x7f6b440039a0 2023-11-28T22:08:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.537+0000 7f6b517fa640 1 -- 172.21.15.45:0/484710244 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4802cc20 con 0x7f6b440039a0 2023-11-28T22:08:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.537+0000 7f6b517fa640 1 -- 172.21.15.45:0/484710244 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48007070 con 0x7f6b440039a0 2023-11-28T22:08:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.537+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5415bba0 con 0x7f6b440039a0 2023-11-28T22:08:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.537+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b54154eb0 con 0x7f6b440039a0 2023-11-28T22:08:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.539+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5409cc80 con 0x7f6b440039a0 2023-11-28T22:08:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.541+0000 7f6b517fa640 1 -- 172.21.15.45:0/484710244 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6b48032070 con 0x7f6b440039a0 2023-11-28T22:08:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.541+0000 7f6b517fa640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f6b3806af60 0x7f6b3806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:43.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.542+0000 7f6b517fa640 1 -- 172.21.15.45:0/484710244 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f6b480aa630 con 0x7f6b440039a0 2023-11-28T22:08:43.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.542+0000 7f6b53fff640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f6b3806af60 0x7f6b3806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:43.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.543+0000 7f6b53fff640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f6b3806af60 0x7f6b3806d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6b40006ce0 tx=0x7f6b40008040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:43.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.546+0000 7f6b517fa640 1 -- 172.21.15.45:0/484710244 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b480754b0 con 0x7f6b440039a0 2023-11-28T22:08:43.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:43.774+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6b5404e920 con 0x7f6b440039a0 2023-11-28T22:08:44.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.348+0000 7f6b517fa640 1 -- 172.21.15.45:0/484710244 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v97) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6b48078ad0 con 0x7f6b440039a0 2023-11-28T22:08:44.355 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 25 and data pool 26 2023-11-28T22:08:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.358+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f6b3806af60 msgr2=0x7f6b3806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.358+0000 7f6b59c71640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f6b3806af60 0x7f6b3806d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6b40006ce0 tx=0x7f6b40008040 comp rx=0 tx=0).stop 2023-11-28T22:08:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.358+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 msgr2=0x7f6b54154970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.358+0000 7f6b59c71640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b54154970 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f6b48001930 tx=0x7f6b48001a00 comp rx=0 tx=0).stop 2023-11-28T22:08:44.364 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.362+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 shutdown_connections 2023-11-28T22:08:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.362+0000 7f6b59c71640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b5415c3a0 0x7f6b5406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.362+0000 7f6b59c71640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b5415d200 0x7f6b540ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.362+0000 7f6b59c71640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f6b3806af60 0x7f6b3806d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.362+0000 7f6b59c71640 1 --2- 172.21.15.45:0/484710244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b440039a0 0x7f6b54154970 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.362+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 >> 172.21.15.45:0/484710244 conn(0x7f6b540b1050 msgr2=0x7f6b540ad120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:44.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.364+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 shutdown_connections 2023-11-28T22:08:44.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.364+0000 7f6b59c71640 1 -- 172.21.15.45:0/484710244 wait complete. 2023-11-28T22:08:44.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:08:44.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.575+0000 7fa36010f640 1 Processor -- start 2023-11-28T22:08:44.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.575+0000 7fa36010f640 1 -- start start 2023-11-28T22:08:44.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.577+0000 7fa36010f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa35815d320 0x7fa3580ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:44.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.577+0000 7fa36010f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580ce710 con 0x7fa3580add40 2023-11-28T22:08:44.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.577+0000 7fa36010f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580ae2c0 con 0x7fa3580b1340 2023-11-28T22:08:44.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.577+0000 7fa36010f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3580ae5a0 con 0x7fa35815d320 2023-11-28T22:08:44.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.577+0000 7fa35d6aa640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa3580add40 0x7fa3580b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48204/0 (socket says 172.21.15.45:48204) 2023-11-28T22:08:44.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.577+0000 7fa35d6aa640 1 -- 172.21.15.45:0/2914210169 learned_addr learned my addr 172.21.15.45:0/2914210169 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35deab640 1 --2- 172.21.15.45:0/2914210169 >> v2:172.21.15.190:3300/0 conn(0x7fa35815d320 0x7fa3580ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150888083 0 0) 0x7fa3580ce710 con 0x7fa3580add40 2023-11-28T22:08:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa344003520 con 0x7fa3580add40 2023-11-28T22:08:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1744007710 0 0) 0x7fa344003520 con 0x7fa3580add40 2023-11-28T22:08:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 >> v2:172.21.15.190:3300/0 conn(0x7fa35815d320 msgr2=0x7fa3580ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 --2- 172.21.15.45:0/2914210169 >> v2:172.21.15.190:3300/0 conn(0x7fa35815d320 0x7fa3580ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 >> v1:172.21.15.190:6789/0 conn(0x7fa3580b1340 legacy=0x7fa358071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3581611f0 con 0x7fa3580add40 2023-11-28T22:08:44.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fa348001670 con 0x7fa3580add40 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 >> v1:172.21.15.45:6789/0 conn(0x7fa3580add40 legacy=0x7fa3580b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa344003970 0x7fa344003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440047c0 0x7fa344004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa344005180 0x7fa34400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34400bff0 con 0x7fa344003970 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34400c160 con 0x7fa3440047c0 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.578+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34400c320 con 0x7fa344005180 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35deab640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa344003970 0x7fa344003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35d6aa640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440047c0 0x7fa344004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35deab640 1 -- 172.21.15.45:0/2914210169 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa344005180 msgr2=0x7fa34400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35e6ac640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa344005180 0x7fa34400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35deab640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa344005180 0x7fa34400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35deab640 1 -- 172.21.15.45:0/2914210169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440047c0 msgr2=0x7fa344004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35deab640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440047c0 0x7fa344004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35deab640 1 -- 172.21.15.45:0/2914210169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa344003520 con 0x7fa344003970 2023-11-28T22:08:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35e6ac640 1 -- 172.21.15.45:0/2914210169 reap_dead start 2023-11-28T22:08:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.579+0000 7fa35d6aa640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440047c0 0x7fa344004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.580+0000 7fa35deab640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa344003970 0x7fa344003d90 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fa354009fc0 tx=0x7fa3440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625ffb598d5f2186 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35402cc20 con 0x7fa344003970 2023-11-28T22:08:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35400f040 con 0x7fa344003970 2023-11-28T22:08:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa35cea9640 1 -- 172.21.15.45:0/2914210169 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa354034440 con 0x7fa344003970 2023-11-28T22:08:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa36010f640 1 -- 172.21.15.45:0/2914210169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa344003970 msgr2=0x7fa344003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa36010f640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa344003970 0x7fa344003d90 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fa354009fc0 tx=0x7fa3440057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa36010f640 1 -- 172.21.15.45:0/2914210169 shutdown_connections 2023-11-28T22:08:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa36010f640 1 --2- 172.21.15.45:0/2914210169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa344003970 0x7fa344003d90 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa36010f640 1 -- 172.21.15.45:0/2914210169 >> 172.21.15.45:0/2914210169 conn(0x7fa35806a500 msgr2=0x7fa35806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:44.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa36010f640 1 -- 172.21.15.45:0/2914210169 shutdown_connections 2023-11-28T22:08:44.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.581+0000 7fa36010f640 1 -- 172.21.15.45:0/2914210169 wait complete. 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.583+0000 7fa36010f640 1 Processor -- start 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.583+0000 7fa36010f640 1 -- start start 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.584+0000 7fa36010f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa344003970 0x7fa358151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.584+0000 7fa36010f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa358152630 0x7fa3581e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.584+0000 7fa36010f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3581e7390 0x7fa3581e77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.584+0000 7fa36010f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa358152ce0 con 0x7fa3581e7390 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.584+0000 7fa36010f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa358152250 con 0x7fa344003970 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.584+0000 7fa36010f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580b49d0 con 0x7fa358152630 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35e6ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3581e7390 0x7fa3581e77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35e6ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3581e7390 0x7fa3581e77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59520/0 (socket says 172.21.15.45:59520) 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35e6ac640 1 -- 172.21.15.45:0/2115426502 learned_addr learned my addr 172.21.15.45:0/2115426502 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35deab640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa344003970 0x7fa358151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35d6aa640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa358152630 0x7fa3581e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35e6ac640 1 -- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa358152630 msgr2=0x7fa3581e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35e6ac640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa358152630 0x7fa3581e6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35e6ac640 1 -- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa344003970 msgr2=0x7fa358151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35e6ac640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa344003970 0x7fa358151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35e6ac640 1 -- 172.21.15.45:0/2115426502 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3580b4cb0 con 0x7fa3581e7390 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.585+0000 7fa35e6ac640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3581e7390 0x7fa3581e77f0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fa35806e600 tx=0x7fa35000d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.586+0000 7fa34e7fc640 1 -- 172.21.15.45:0/2115426502 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35000b070 con 0x7fa3581e7390 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.586+0000 7fa34e7fc640 1 -- 172.21.15.45:0/2115426502 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3500054e0 con 0x7fa3581e7390 2023-11-28T22:08:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.586+0000 7fa34e7fc640 1 -- 172.21.15.45:0/2115426502 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350003810 con 0x7fa3581e7390 2023-11-28T22:08:44.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.586+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa358152f30 con 0x7fa3581e7390 2023-11-28T22:08:44.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.586+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3580b34a0 con 0x7fa3581e7390 2023-11-28T22:08:44.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.587+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3580733d0 con 0x7fa3581e7390 2023-11-28T22:08:44.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.588+0000 7fa34e7fc640 1 -- 172.21.15.45:0/2115426502 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa350003970 con 0x7fa3581e7390 2023-11-28T22:08:44.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.589+0000 7fa34e7fc640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fa33c06af60 0x7fa33c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:44.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.589+0000 7fa34e7fc640 1 -- 172.21.15.45:0/2115426502 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa350089090 con 0x7fa3581e7390 2023-11-28T22:08:44.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.590+0000 7fa35deab640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fa33c06af60 0x7fa33c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:44.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.591+0000 7fa35deab640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fa33c06af60 0x7fa33c06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa354036000 tx=0x7fa35402e000 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:44.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.593+0000 7fa34e7fc640 1 -- 172.21.15.45:0/2115426502 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa350053d20 con 0x7fa3581e7390 2023-11-28T22:08:44.789 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.787+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa358152ac0 con 0x7fa3581e7390 2023-11-28T22:08:44.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.790+0000 7fa34e7fc640 1 -- 172.21.15.45:0/2115426502 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+16765 (secure 0 0 0) 0x7fa350018090 con 0x7fa3581e7390 2023-11-28T22:08:44.793 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:44.793 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":214,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:08:43.776257+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-11-28T22:08:41.574328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-11-28T22:08:42.812303+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fa33c06af60 msgr2=0x7fa33c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fa33c06af60 0x7fa33c06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa354036000 tx=0x7fa35402e000 comp rx=0 tx=0).stop 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3581e7390 msgr2=0x7fa3581e77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3581e7390 0x7fa3581e77f0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fa35806e600 tx=0x7fa35000d890 comp rx=0 tx=0).stop 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 shutdown_connections 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fa33c06af60 0x7fa33c06d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3581e7390 0x7fa3581e77f0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa358152630 0x7fa3581e6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 --2- 172.21.15.45:0/2115426502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa344003970 0x7fa358151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 >> 172.21.15.45:0/2115426502 conn(0x7fa35806a500 msgr2=0x7fa35815a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 shutdown_connections 2023-11-28T22:08:44.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:44.793+0000 7fa36010f640 1 -- 172.21.15.45:0/2115426502 wait complete. 2023-11-28T22:08:44.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:08:45.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.030+0000 7f1235bf0640 1 Processor -- start 2023-11-28T22:08:45.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.030+0000 7f1235bf0640 1 -- start start 2023-11-28T22:08:45.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.032+0000 7f1235bf0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1230155780 0x7f12300abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.032+0000 7f1235bf0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f12300ce800 0x7f12300b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.032+0000 7f1235bf0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123015c3c0 con 0x7f12300ae540 2023-11-28T22:08:45.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.032+0000 7f1235bf0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1230155b60 con 0x7f12300ce800 2023-11-28T22:08:45.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.032+0000 7f1235bf0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123015ba90 con 0x7f1230155780 2023-11-28T22:08:45.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.033+0000 7f122ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f12300ae540 0x7f12300ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48220/0 (socket says 172.21.15.45:48220) 2023-11-28T22:08:45.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.033+0000 7f122ffff640 1 -- 172.21.15.45:0/3793565986 learned_addr learned my addr 172.21.15.45:0/3793565986 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:45.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.033+0000 7f1234bee640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1230155780 0x7f12300abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.033+0000 7f12353ef640 1 --2- 172.21.15.45:0/3793565986 >> v2:172.21.15.190:3300/0 conn(0x7f12300ce800 0x7f12300b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.033+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651117383 0 0) 0x7f123015c3c0 con 0x7f12300ae540 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.033+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1220003520 con 0x7f12300ae540 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3251727241 0 0) 0x7f1220003520 con 0x7f12300ae540 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1230155780 msgr2=0x7f12300abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1230155780 0x7f12300abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 >> v2:172.21.15.190:3300/0 conn(0x7f12300ce800 msgr2=0x7f12300b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 --2- 172.21.15.45:0/3793565986 >> v2:172.21.15.190:3300/0 conn(0x7f12300ce800 0x7f12300b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123015bd70 con 0x7f12300ae540 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f121c001670 con 0x7f12300ae540 2023-11-28T22:08:45.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 >> v1:172.21.15.45:6789/0 conn(0x7f12300ae540 legacy=0x7f12300ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1220003970 0x7f1220003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12200047c0 0x7f1220004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1230155780 0x7f1220005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1230155b60 con 0x7f12200047c0 2023-11-28T22:08:45.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123015ba90 con 0x7f1220003970 2023-11-28T22:08:45.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122000b980 con 0x7f1230155780 2023-11-28T22:08:45.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.034+0000 7f122ffff640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12200047c0 0x7f1220004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.035+0000 7f1234bee640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1220003970 0x7f1220003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.035+0000 7f12353ef640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1230155780 0x7f1220005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.035+0000 7f122ffff640 1 -- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1230155780 msgr2=0x7f1220005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.035+0000 7f122ffff640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1230155780 0x7f1220005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.035+0000 7f122ffff640 1 -- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1220003970 msgr2=0x7f1220003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.035+0000 7f122ffff640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1220003970 0x7f1220003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.035+0000 7f122ffff640 1 -- 172.21.15.45:0/3793565986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1220003520 con 0x7f12200047c0 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.036+0000 7f122ffff640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12200047c0 0x7f1220004c40 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f121c00c110 tx=0x7f12200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddaa09801619ace server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c005ad0 con 0x7f12200047c0 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c005c50 con 0x7f12200047c0 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f122f7fe640 1 -- 172.21.15.45:0/3793565986 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c02c810 con 0x7f12200047c0 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f1235bf0640 1 -- 172.21.15.45:0/3793565986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12200047c0 msgr2=0x7f1220004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f1235bf0640 1 --2- 172.21.15.45:0/3793565986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12200047c0 0x7f1220004c40 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f121c00c110 tx=0x7f12200057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f12353ef640 1 -- 172.21.15.45:0/3793565986 reap_dead start 2023-11-28T22:08:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f1235bf0640 1 -- 172.21.15.45:0/3793565986 shutdown_connections 2023-11-28T22:08:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f1235bf0640 1 -- 172.21.15.45:0/3793565986 >> 172.21.15.45:0/3793565986 conn(0x7f123006ab30 msgr2=0x7f12300b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f1235bf0640 1 -- 172.21.15.45:0/3793565986 shutdown_connections 2023-11-28T22:08:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.037+0000 7f1235bf0640 1 -- 172.21.15.45:0/3793565986 wait complete. 2023-11-28T22:08:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.039+0000 7f1235bf0640 1 Processor -- start 2023-11-28T22:08:45.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.039+0000 7f1235bf0640 1 -- start start 2023-11-28T22:08:45.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.040+0000 7f1235bf0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12200047c0 0x7f12301e4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.040+0000 7f1235bf0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12301e5ae0 0x7f12301538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.040+0000 7f1235bf0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1230156850 0x7f1230156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.040+0000 7f1235bf0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1230157240 con 0x7f12301e5ae0 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.040+0000 7f1235bf0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12301e49f0 con 0x7f1230156850 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.040+0000 7f1235bf0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12301e4cd0 con 0x7f12200047c0 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f122ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12301e5ae0 0x7f12301538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f122ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12301e5ae0 0x7f12301538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59534/0 (socket says 172.21.15.45:59534) 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f122ffff640 1 -- 172.21.15.45:0/160645383 learned_addr learned my addr 172.21.15.45:0/160645383 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f1234bee640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12200047c0 0x7f12301e4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f12353ef640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1230156850 0x7f1230156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f122ffff640 1 -- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12200047c0 msgr2=0x7f12301e4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f122ffff640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12200047c0 0x7f12301e4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f122ffff640 1 -- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1230156850 msgr2=0x7f1230156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f122ffff640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1230156850 0x7f1230156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f122ffff640 1 -- 172.21.15.45:0/160645383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12301e4fb0 con 0x7f12301e5ae0 2023-11-28T22:08:45.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.041+0000 7f122ffff640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12301e5ae0 0x7f12301538b0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f121c000fd0 tx=0x7f121c001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:45.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.042+0000 7f122d7fa640 1 -- 172.21.15.45:0/160645383 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c02cc20 con 0x7f12301e5ae0 2023-11-28T22:08:45.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.042+0000 7f122d7fa640 1 -- 172.21.15.45:0/160645383 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c02cd80 con 0x7f12301e5ae0 2023-11-28T22:08:45.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.042+0000 7f122d7fa640 1 -- 172.21.15.45:0/160645383 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c007270 con 0x7f12301e5ae0 2023-11-28T22:08:45.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.042+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1230158be0 con 0x7f12301e5ae0 2023-11-28T22:08:45.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.042+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12301e46d0 con 0x7f12301e5ae0 2023-11-28T22:08:45.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.043+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123015d060 con 0x7f12301e5ae0 2023-11-28T22:08:45.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.044+0000 7f122d7fa640 1 -- 172.21.15.45:0/160645383 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f121c032070 con 0x7f12301e5ae0 2023-11-28T22:08:45.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.046+0000 7f122d7fa640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f121406af90 0x7f121406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.046+0000 7f122d7fa640 1 -- 172.21.15.45:0/160645383 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f121c03c040 con 0x7f12301e5ae0 2023-11-28T22:08:45.049 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.047+0000 7f1234bee640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f121406af90 0x7f121406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.048+0000 7f1234bee640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f121406af90 0x7f121406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1224009930 tx=0x7f1224008040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:45.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.050+0000 7f122d7fa640 1 -- 172.21.15.45:0/160645383 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f121c075870 con 0x7f12301e5ae0 2023-11-28T22:08:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.236+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f123004e920 con 0x7f12301e5ae0 2023-11-28T22:08:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.237+0000 7f122d7fa640 1 -- 172.21.15.45:0/160645383 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+16765 (secure 0 0 0) 0x7f121c045020 con 0x7f12301e5ae0 2023-11-28T22:08:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:45.240 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":214,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:08:43.776257+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-11-28T22:08:41.574328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-11-28T22:08:42.812303+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f121406af90 msgr2=0x7f121406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f121406af90 0x7f121406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1224009930 tx=0x7f1224008040 comp rx=0 tx=0).stop 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12301e5ae0 msgr2=0x7f12301538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12301e5ae0 0x7f12301538b0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f121c000fd0 tx=0x7f121c001930 comp rx=0 tx=0).stop 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 shutdown_connections 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1230156850 0x7f1230156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f121406af90 0x7f121406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12301e5ae0 0x7f12301538b0 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 --2- 172.21.15.45:0/160645383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12200047c0 0x7f12301e4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 >> 172.21.15.45:0/160645383 conn(0x7f123006ab30 msgr2=0x7f12300af030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 shutdown_connections 2023-11-28T22:08:45.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.240+0000 7f1235bf0640 1 -- 172.21.15.45:0/160645383 wait complete. 2023-11-28T22:08:45.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T22:08:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.465+0000 7f64f0c7e640 1 Processor -- start 2023-11-28T22:08:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.465+0000 7f64f0c7e640 1 -- start start 2023-11-28T22:08:45.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.466+0000 7f64f0c7e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64ec0cdca0 0x7f64ec06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.467+0000 7f64f0c7e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ec06d410 con 0x7f64ec15b270 2023-11-28T22:08:45.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.467+0000 7f64f0c7e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ec06d6f0 con 0x7f64ec0b3040 2023-11-28T22:08:45.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.467+0000 7f64f0c7e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ec0ceaa0 con 0x7f64ec0cdca0 2023-11-28T22:08:45.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.467+0000 7f64ebfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64ec0cdca0 0x7f64ec06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.467+0000 7f64ebfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64ec0cdca0 0x7f64ec06cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59544/0 (socket says 172.21.15.45:59544) 2023-11-28T22:08:45.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.467+0000 7f64ebfff640 1 -- 172.21.15.45:0/2609388547 learned_addr learned my addr 172.21.15.45:0/2609388547 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:45.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.467+0000 7f64ebfff640 1 -- 172.21.15.45:0/2609388547 >> v1:172.21.15.190:6789/0 conn(0x7f64ec0b3040 legacy=0x7f64ec15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:45.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.467+0000 7f64ebfff640 1 -- 172.21.15.45:0/2609388547 >> v1:172.21.15.45:6789/0 conn(0x7f64ec15b270 legacy=0x7f64ec0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.470 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.467+0000 7f64ebfff640 1 -- 172.21.15.45:0/2609388547 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec06d9d0 con 0x7f64ec0cdca0 2023-11-28T22:08:45.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.468+0000 7f64ebfff640 1 --2- 172.21.15.45:0/2609388547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 0x7f64ec06cda0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009870 tx=0x7f64e402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5218a24e562e7100 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:45.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.469+0000 7f64ea7fc640 1 -- 172.21.15.45:0/2609388547 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e40317e0 con 0x7f64ec0cdca0 2023-11-28T22:08:45.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.469+0000 7f64ea7fc640 1 -- 172.21.15.45:0/2609388547 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e4031940 con 0x7f64ec0cdca0 2023-11-28T22:08:45.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.469+0000 7f64ea7fc640 1 -- 172.21.15.45:0/2609388547 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4031cd0 con 0x7f64ec0cdca0 2023-11-28T22:08:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.469+0000 7f64f0c7e640 1 -- 172.21.15.45:0/2609388547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 msgr2=0x7f64ec06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.469+0000 7f64f0c7e640 1 --2- 172.21.15.45:0/2609388547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 0x7f64ec06cda0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009870 tx=0x7f64e402b880 comp rx=0 tx=0).stop 2023-11-28T22:08:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.470+0000 7f64f0c7e640 1 -- 172.21.15.45:0/2609388547 shutdown_connections 2023-11-28T22:08:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.470+0000 7f64f0c7e640 1 --2- 172.21.15.45:0/2609388547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 0x7f64ec06cda0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.470+0000 7f64f0c7e640 1 -- 172.21.15.45:0/2609388547 >> 172.21.15.45:0/2609388547 conn(0x7f64ec0b1050 msgr2=0x7f64ec0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.470+0000 7f64f0c7e640 1 -- 172.21.15.45:0/2609388547 shutdown_connections 2023-11-28T22:08:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.470+0000 7f64f0c7e640 1 -- 172.21.15.45:0/2609388547 wait complete. 2023-11-28T22:08:45.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.472+0000 7f64f0c7e640 1 Processor -- start 2023-11-28T22:08:45.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.472+0000 7f64f0c7e640 1 -- start start 2023-11-28T22:08:45.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.473+0000 7f64f0c7e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64ec0b3040 0x7f64ec09c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64f0c7e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 0x7f64ec140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64f0c7e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ec15b270 0x7f64ec15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64f0c7e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec1422c0 con 0x7f64ec0cdca0 2023-11-28T22:08:45.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64f0c7e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec1425a0 con 0x7f64ec15b270 2023-11-28T22:08:45.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64f0c7e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec142880 con 0x7f64ec0b3040 2023-11-28T22:08:45.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 0x7f64ec140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 0x7f64ec140e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59552/0 (socket says 172.21.15.45:59552) 2023-11-28T22:08:45.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eaffd640 1 -- 172.21.15.45:0/899319512 learned_addr learned my addr 172.21.15.45:0/899319512 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:45.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eb7fe640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64ec0b3040 0x7f64ec09c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eaffd640 1 -- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64ec0b3040 msgr2=0x7f64ec09c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eaffd640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64ec0b3040 0x7f64ec09c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64ebfff640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ec15b270 0x7f64ec15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eaffd640 1 -- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ec15b270 msgr2=0x7f64ec15b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eaffd640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ec15b270 0x7f64ec15b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eaffd640 1 -- 172.21.15.45:0/899319512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e40094d0 con 0x7f64ec0cdca0 2023-11-28T22:08:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.474+0000 7f64eb7fe640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64ec0b3040 0x7f64ec09c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.475+0000 7f64eaffd640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 0x7f64ec140e30 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c3c0 tx=0x7f64e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.475+0000 7f64d3fff640 1 -- 172.21.15.45:0/899319512 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e00052d0 con 0x7f64ec0cdca0 2023-11-28T22:08:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.475+0000 7f64d3fff640 1 -- 172.21.15.45:0/899319512 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e0005450 con 0x7f64ec0cdca0 2023-11-28T22:08:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.475+0000 7f64d3fff640 1 -- 172.21.15.45:0/899319512 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0003830 con 0x7f64ec0cdca0 2023-11-28T22:08:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.475+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ec142b60 con 0x7f64ec0cdca0 2023-11-28T22:08:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.475+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ec141370 con 0x7f64ec0cdca0 2023-11-28T22:08:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.476+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ec142d50 con 0x7f64ec0cdca0 2023-11-28T22:08:45.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.477+0000 7f64d3fff640 1 -- 172.21.15.45:0/899319512 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f64e0003990 con 0x7f64ec0cdca0 2023-11-28T22:08:45.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.478+0000 7f64d3fff640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f64d806af60 0x7f64d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:45.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.478+0000 7f64d3fff640 1 -- 172.21.15.45:0/899319512 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f64e0088e90 con 0x7f64ec0cdca0 2023-11-28T22:08:45.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.479+0000 7f64eb7fe640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f64d806af60 0x7f64d806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:45.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.480+0000 7f64eb7fe640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f64d806af60 0x7f64d806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f64ec1f4570 tx=0x7f64dc009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:45.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.483+0000 7f64d3fff640 1 -- 172.21.15.45:0/899319512 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64e0053c30 con 0x7f64ec0cdca0 2023-11-28T22:08:45.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:45.720+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f64ec142400 con 0x7f64ec0cdca0 2023-11-28T22:08:46.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.372+0000 7f64d3fff640 1 -- 172.21.15.45:0/899319512 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v100) v1 ==== 118+0+0 (secure 0 0 0) 0x7f64ec142400 con 0x7f64ec0cdca0 2023-11-28T22:08:46.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f64d806af60 msgr2=0x7f64d806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:46.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f64d806af60 0x7f64d806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f64ec1f4570 tx=0x7f64dc009040 comp rx=0 tx=0).stop 2023-11-28T22:08:46.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 msgr2=0x7f64ec140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:46.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 0x7f64ec140e30 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c3c0 tx=0x7f64e000c880 comp rx=0 tx=0).stop 2023-11-28T22:08:46.379 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 shutdown_connections 2023-11-28T22:08:46.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ec15b270 0x7f64ec15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f64d806af60 0x7f64d806d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.380 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64ec0cdca0 0x7f64ec140e30 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 --2- 172.21.15.45:0/899319512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f64ec0b3040 0x7f64ec09c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 >> 172.21.15.45:0/899319512 conn(0x7f64ec0b1050 msgr2=0x7f64ec0aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 shutdown_connections 2023-11-28T22:08:46.381 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.376+0000 7f64f0c7e640 1 -- 172.21.15.45:0/899319512 wait complete. 2023-11-28T22:08:46.400 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T22:08:46.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T22:08:46.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.628+0000 7ff723505640 1 Processor -- start 2023-11-28T22:08:46.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.628+0000 7ff723505640 1 -- start start 2023-11-28T22:08:46.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.630+0000 7ff723505640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff71c0d24f0 0x7ff71c0d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:46.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.630+0000 7ff723505640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c160e80 con 0x7ff71c160aa0 2023-11-28T22:08:46.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.630+0000 7ff723505640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c0d1220 con 0x7ff71c0ce0e0 2023-11-28T22:08:46.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.630+0000 7ff723505640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71c0d2e70 con 0x7ff71c0d24f0 2023-11-28T22:08:46.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.630+0000 7ff7212a1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff71c160aa0 0x7ff71c0b6420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48238/0 (socket says 172.21.15.45:48238) 2023-11-28T22:08:46.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.630+0000 7ff7212a1640 1 -- 172.21.15.45:0/3165253922 learned_addr learned my addr 172.21.15.45:0/3165253922 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:46.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff720aa0640 1 --2- 172.21.15.45:0/3165253922 >> v2:172.21.15.190:3300/0 conn(0x7ff71c0d24f0 0x7ff71c0d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:46.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381787332 0 0) 0x7ff71c160e80 con 0x7ff71c160aa0 2023-11-28T22:08:46.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff708003520 con 0x7ff71c160aa0 2023-11-28T22:08:46.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2330368779 0 0) 0x7ff708003520 con 0x7ff71c160aa0 2023-11-28T22:08:46.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 >> v2:172.21.15.190:3300/0 conn(0x7ff71c0d24f0 msgr2=0x7ff71c0d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:46.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 --2- 172.21.15.45:0/3165253922 >> v2:172.21.15.190:3300/0 conn(0x7ff71c0d24f0 0x7ff71c0d2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 >> v1:172.21.15.190:6789/0 conn(0x7ff71c0ce0e0 legacy=0x7ff71c0ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:46.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c0d1470 con 0x7ff71c160aa0 2023-11-28T22:08:46.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ff718001670 con 0x7ff71c160aa0 2023-11-28T22:08:46.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 >> v1:172.21.15.45:6789/0 conn(0x7ff71c160aa0 legacy=0x7ff71c0b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:46.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7080039c0 0x7ff708003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:46.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7080047c0 0x7ff708004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:46.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff708005180 0x7ff70800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0d2e70 con 0x7ff7080047c0 2023-11-28T22:08:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70800c060 con 0x7ff708005180 2023-11-28T22:08:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.631+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70800c220 con 0x7ff7080039c0 2023-11-28T22:08:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff720aa0640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7080047c0 0x7ff708004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:46.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff7212a1640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7080039c0 0x7ff708003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff721aa2640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff708005180 0x7ff70800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff720aa0640 1 -- 172.21.15.45:0/3165253922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7080039c0 msgr2=0x7ff708003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff720aa0640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7080039c0 0x7ff708003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff720aa0640 1 -- 172.21.15.45:0/3165253922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff708005180 msgr2=0x7ff70800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:46.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff720aa0640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff708005180 0x7ff70800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff720aa0640 1 -- 172.21.15.45:0/3165253922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708003520 con 0x7ff7080047c0 2023-11-28T22:08:46.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff721aa2640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff708005180 0x7ff70800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:46.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.632+0000 7ff721aa2640 1 -- 172.21.15.45:0/3165253922 reap_dead start 2023-11-28T22:08:46.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.633+0000 7ff720aa0640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7080047c0 0x7ff708004c40 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009ac0 tx=0x7ff7080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e5cced984d8adf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:46.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c02c6b0 con 0x7ff7080047c0 2023-11-28T22:08:46.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70c02c810 con 0x7ff7080047c0 2023-11-28T22:08:46.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff713fff640 1 -- 172.21.15.45:0/3165253922 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c02cba0 con 0x7ff7080047c0 2023-11-28T22:08:46.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff723505640 1 -- 172.21.15.45:0/3165253922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7080047c0 msgr2=0x7ff708004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:46.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff723505640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7080047c0 0x7ff708004c40 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009ac0 tx=0x7ff7080057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff723505640 1 -- 172.21.15.45:0/3165253922 shutdown_connections 2023-11-28T22:08:46.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff723505640 1 --2- 172.21.15.45:0/3165253922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7080047c0 0x7ff708004c40 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff723505640 1 -- 172.21.15.45:0/3165253922 >> 172.21.15.45:0/3165253922 conn(0x7ff71c070b20 msgr2=0x7ff71c070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:46.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff723505640 1 -- 172.21.15.45:0/3165253922 shutdown_connections 2023-11-28T22:08:46.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.634+0000 7ff723505640 1 -- 172.21.15.45:0/3165253922 wait complete. 2023-11-28T22:08:46.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.636+0000 7ff723505640 1 Processor -- start 2023-11-28T22:08:46.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.637+0000 7ff723505640 1 -- start start 2023-11-28T22:08:46.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff723505640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7080047c0 0x7ff71c1f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff723505640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c1f07b0 0x7ff71c0c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff723505640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c0b5040 0x7ff71c0b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff723505640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c14df00 con 0x7ff71c1f07b0 2023-11-28T22:08:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff723505640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0c9550 con 0x7ff71c0b5040 2023-11-28T22:08:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff723505640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c1f0c30 con 0x7ff7080047c0 2023-11-28T22:08:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff720aa0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c1f07b0 0x7ff71c0c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff720aa0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c1f07b0 0x7ff71c0c80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59564/0 (socket says 172.21.15.45:59564) 2023-11-28T22:08:46.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff720aa0640 1 -- 172.21.15.45:0/1110377895 learned_addr learned my addr 172.21.15.45:0/1110377895 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff7212a1640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7080047c0 0x7ff71c1f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff721aa2640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c0b5040 0x7ff71c0b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff720aa0640 1 -- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7080047c0 msgr2=0x7ff71c1f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:46.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff720aa0640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7080047c0 0x7ff71c1f3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff720aa0640 1 -- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c0b5040 msgr2=0x7ff71c0b54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff720aa0640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c0b5040 0x7ff71c0b54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.638+0000 7ff720aa0640 1 -- 172.21.15.45:0/1110377895 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c0b5a00 con 0x7ff71c1f07b0 2023-11-28T22:08:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.639+0000 7ff720aa0640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c1f07b0 0x7ff71c0c80d0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7ff70c037040 tx=0x7ff70c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:46.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.639+0000 7ff711ffb640 1 -- 172.21.15.45:0/1110377895 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c003810 con 0x7ff71c1f07b0 2023-11-28T22:08:46.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.639+0000 7ff711ffb640 1 -- 172.21.15.45:0/1110377895 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70c003970 con 0x7ff71c1f07b0 2023-11-28T22:08:46.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.639+0000 7ff711ffb640 1 -- 172.21.15.45:0/1110377895 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c003d00 con 0x7ff71c1f07b0 2023-11-28T22:08:46.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.639+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c0c8940 con 0x7ff71c1f07b0 2023-11-28T22:08:46.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.639+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c0c8f00 con 0x7ff71c1f07b0 2023-11-28T22:08:46.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.641+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c066f40 con 0x7ff71c1f07b0 2023-11-28T22:08:46.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.641+0000 7ff711ffb640 1 -- 172.21.15.45:0/1110377895 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff70c034440 con 0x7ff71c1f07b0 2023-11-28T22:08:46.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.642+0000 7ff711ffb640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff70006af90 0x7ff70006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:46.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.642+0000 7ff711ffb640 1 -- 172.21.15.45:0/1110377895 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff70c0ab860 con 0x7ff71c1f07b0 2023-11-28T22:08:46.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.643+0000 7ff7212a1640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff70006af90 0x7ff70006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:46.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.644+0000 7ff7212a1640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff70006af90 0x7ff70006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff71c069a70 tx=0x7ff718000ed0 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:46.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.648+0000 7ff711ffb640 1 -- 172.21.15.45:0/1110377895 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff70c076600 con 0x7ff71c1f07b0 2023-11-28T22:08:46.880 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:46.878+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff71c14dee0 con 0x7ff71c1f07b0 2023-11-28T22:08:47.378 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.375+0000 7ff711ffb640 1 -- 172.21.15.45:0/1110377895 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v101) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff70c079c20 con 0x7ff71c1f07b0 2023-11-28T22:08:47.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.379+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff70006af90 msgr2=0x7ff70006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:47.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.379+0000 7ff723505640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff70006af90 0x7ff70006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff71c069a70 tx=0x7ff718000ed0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.382 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.379+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c1f07b0 msgr2=0x7ff71c0c80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.379+0000 7ff723505640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c1f07b0 0x7ff71c0c80d0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7ff70c037040 tx=0x7ff70c004730 comp rx=0 tx=0).stop 2023-11-28T22:08:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.380+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 shutdown_connections 2023-11-28T22:08:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.380+0000 7ff723505640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c0b5040 0x7ff71c0b54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.380+0000 7ff723505640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff70006af90 0x7ff70006d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.380+0000 7ff723505640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c1f07b0 0x7ff71c0c80d0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.380+0000 7ff723505640 1 --2- 172.21.15.45:0/1110377895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7080047c0 0x7ff71c1f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.380+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 >> 172.21.15.45:0/1110377895 conn(0x7ff71c070b20 msgr2=0x7ff71c0b64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.380+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 shutdown_connections 2023-11-28T22:08:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.380+0000 7ff723505640 1 -- 172.21.15.45:0/1110377895 wait complete. 2023-11-28T22:08:47.404 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T22:08:47.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:08:47.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.628+0000 7fd64ffff640 1 Processor -- start 2023-11-28T22:08:47.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.629+0000 7fd64ffff640 1 -- start start 2023-11-28T22:08:47.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.630+0000 7fd64ffff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd6500721e0 0x7fd65006d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:47.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.630+0000 7fd64ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd65006d6b0 0x7fd6500ac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:47.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.630+0000 7fd64ffff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65015e680 con 0x7fd6500b3870 2023-11-28T22:08:47.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.630+0000 7fd64ffff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd65015a460 con 0x7fd65006d6b0 2023-11-28T22:08:47.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.630+0000 7fd64ffff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd65006db40 con 0x7fd6500721e0 2023-11-28T22:08:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.631+0000 7fd64f7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd65006d6b0 0x7fd6500ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.631+0000 7fd64f7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd65006d6b0 0x7fd6500ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59576/0 (socket says 172.21.15.45:59576) 2023-11-28T22:08:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.631+0000 7fd64f7fe640 1 -- 172.21.15.45:0/519814100 learned_addr learned my addr 172.21.15.45:0/519814100 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.631+0000 7fd64f7fe640 1 -- 172.21.15.45:0/519814100 >> v2:172.21.15.190:3300/0 conn(0x7fd6500721e0 msgr2=0x7fd65006d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.631+0000 7fd64e7fc640 1 --2- 172.21.15.45:0/519814100 >> v2:172.21.15.190:3300/0 conn(0x7fd6500721e0 0x7fd65006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.631+0000 7fd64f7fe640 1 --2- 172.21.15.45:0/519814100 >> v2:172.21.15.190:3300/0 conn(0x7fd6500721e0 0x7fd65006d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.631+0000 7fd64f7fe640 1 -- 172.21.15.45:0/519814100 >> v1:172.21.15.45:6789/0 conn(0x7fd6500b3870 legacy=0x7fd65015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:47.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.631+0000 7fd64f7fe640 1 -- 172.21.15.45:0/519814100 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65015e960 con 0x7fd65006d6b0 2023-11-28T22:08:47.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.632+0000 7fd64f7fe640 1 --2- 172.21.15.45:0/519814100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd65006d6b0 0x7fd6500ac840 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fd65006e2d0 tx=0x7fd64802b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4801b4a3b01c960 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:47.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.633+0000 7fd64dffb640 1 -- 172.21.15.45:0/519814100 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648030880 con 0x7fd65006d6b0 2023-11-28T22:08:47.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.633+0000 7fd64dffb640 1 -- 172.21.15.45:0/519814100 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6480309e0 con 0x7fd65006d6b0 2023-11-28T22:08:47.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.633+0000 7fd64dffb640 1 -- 172.21.15.45:0/519814100 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648030d70 con 0x7fd65006d6b0 2023-11-28T22:08:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.633+0000 7fd64ffff640 1 -- 172.21.15.45:0/519814100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd65006d6b0 msgr2=0x7fd6500ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.633+0000 7fd64ffff640 1 --2- 172.21.15.45:0/519814100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd65006d6b0 0x7fd6500ac840 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fd65006e2d0 tx=0x7fd64802b9c0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.634+0000 7fd64ffff640 1 -- 172.21.15.45:0/519814100 shutdown_connections 2023-11-28T22:08:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.634+0000 7fd64ffff640 1 --2- 172.21.15.45:0/519814100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd65006d6b0 0x7fd6500ac840 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.634+0000 7fd64ffff640 1 --2- 172.21.15.45:0/519814100 >> v2:172.21.15.190:3300/0 conn(0x7fd6500721e0 0x7fd65006d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.634+0000 7fd64ffff640 1 -- 172.21.15.45:0/519814100 >> 172.21.15.45:0/519814100 conn(0x7fd65006ab30 msgr2=0x7fd6500b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.634+0000 7fd64ffff640 1 -- 172.21.15.45:0/519814100 shutdown_connections 2023-11-28T22:08:47.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.634+0000 7fd64ffff640 1 -- 172.21.15.45:0/519814100 wait complete. 2023-11-28T22:08:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.635+0000 7fd64ffff640 1 Processor -- start 2023-11-28T22:08:47.638 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.636+0000 7fd64ffff640 1 -- start start 2023-11-28T22:08:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd65006d6b0 0x7fd6501526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64ffff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6500721e0 0x7fd65014b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6500b3870 0x7fd6500cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64ffff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65009ac10 con 0x7fd6500b3870 2023-11-28T22:08:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64ffff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65014bdc0 con 0x7fd65006d6b0 2023-11-28T22:08:47.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64ffff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd650099f90 con 0x7fd6500721e0 2023-11-28T22:08:47.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6500b3870 0x7fd6500cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:47.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6500b3870 0x7fd6500cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59580/0 (socket says 172.21.15.45:59580) 2023-11-28T22:08:47.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64f7fe640 1 -- 172.21.15.45:0/1576241430 learned_addr learned my addr 172.21.15.45:0/1576241430 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64f7fe640 1 -- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6500721e0 msgr2=0x7fd65014b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64f7fe640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6500721e0 0x7fd65014b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64f7fe640 1 -- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd65006d6b0 msgr2=0x7fd6501526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64f7fe640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd65006d6b0 0x7fd6501526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.637+0000 7fd64f7fe640 1 -- 172.21.15.45:0/1576241430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6480094d0 con 0x7fd6500b3870 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.638+0000 7fd64f7fe640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6500b3870 0x7fd6500cf040 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fd64802ba00 tx=0x7fd648004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.638+0000 7fd6477fe640 1 -- 172.21.15.45:0/1576241430 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648030810 con 0x7fd6500b3870 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.638+0000 7fd6477fe640 1 -- 172.21.15.45:0/1576241430 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd648039d90 con 0x7fd6500b3870 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.638+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65009a270 con 0x7fd6500b3870 2023-11-28T22:08:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.638+0000 7fd6477fe640 1 -- 172.21.15.45:0/1576241430 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648043650 con 0x7fd6500b3870 2023-11-28T22:08:47.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.638+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65009a830 con 0x7fd6500b3870 2023-11-28T22:08:47.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.639+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65009d0f0 con 0x7fd6500b3870 2023-11-28T22:08:47.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.640+0000 7fd6477fe640 1 -- 172.21.15.45:0/1576241430 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd64802f3d0 con 0x7fd6500b3870 2023-11-28T22:08:47.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.641+0000 7fd6477fe640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd63006af60 0x7fd63006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:47.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.641+0000 7fd6477fe640 1 -- 172.21.15.45:0/1576241430 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd6480ac340 con 0x7fd6500b3870 2023-11-28T22:08:47.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.642+0000 7fd64effd640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd63006af60 0x7fd63006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:47.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.643+0000 7fd64effd640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd63006af60 0x7fd63006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd640000f60 tx=0x7fd6400079b0 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:47.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.646+0000 7fd6477fe640 1 -- 172.21.15.45:0/1576241430 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd648076fd0 con 0x7fd6500b3870 2023-11-28T22:08:47.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.879+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd65004e920 con 0x7fd6500b3870 2023-11-28T22:08:47.882 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.880+0000 7fd6477fe640 1 -- 172.21.15.45:0/1576241430 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4928 (secure 0 0 0) 0x7fd64807a5f0 con 0x7fd6500b3870 2023-11-28T22:08:47.882 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:47.882 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8692,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3311227176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3311227176},{"type":"v1","addr":"172.21.15.190:6835","nonce":3311227176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8698,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3492680695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3492680695},{"type":"v1","addr":"172.21.15.45:6835","nonce":3492680695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":8701,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3937197634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3937197634},{"type":"v1","addr":"172.21.15.190:6837","nonce":3937197634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T22:08:43.777075+0000","modified":"2023-11-28T22:08:46.880928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10650":{"gid":10650,"name":"c","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6833/3094566588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3094566588},{"type":"v1","addr":"172.21.15.45:6833","nonce":3094566588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-28T22:08:47.883 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 101 2023-11-28T22:08:47.885 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd63006af60 msgr2=0x7fd63006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd63006af60 0x7fd63006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd640000f60 tx=0x7fd6400079b0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6500b3870 msgr2=0x7fd6500cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6500b3870 0x7fd6500cf040 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fd64802ba00 tx=0x7fd648004730 comp rx=0 tx=0).stop 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 shutdown_connections 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fd63006af60 0x7fd63006d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6500b3870 0x7fd6500cf040 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd6500721e0 0x7fd65014b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 --2- 172.21.15.45:0/1576241430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd65006d6b0 0x7fd6501526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 >> 172.21.15.45:0/1576241430 conn(0x7fd65006ab30 msgr2=0x7fd6500b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 shutdown_connections 2023-11-28T22:08:47.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:47.883+0000 7fd64ffff640 1 -- 172.21.15.45:0/1576241430 wait complete. 2023-11-28T22:08:47.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:08:48.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.130+0000 7ff8953c9640 1 Processor -- start 2023-11-28T22:08:48.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.130+0000 7ff8953c9640 1 -- start start 2023-11-28T22:08:48.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff8953c9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff8900b35b0 0x7ff89015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff8953c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 0x7ff89015f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff8953c9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89015bda0 con 0x7ff89015b850 2023-11-28T22:08:48.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff8953c9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff89015ee40 con 0x7ff8900b35b0 2023-11-28T22:08:48.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff8953c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89015c080 con 0x7ff8900b1340 2023-11-28T22:08:48.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff88effd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff8900b35b0 0x7ff89015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff88effd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff8900b35b0 0x7ff89015ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59584/0 (socket says 172.21.15.45:59584) 2023-11-28T22:08:48.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff88effd640 1 -- 172.21.15.45:0/491853863 learned_addr learned my addr 172.21.15.45:0/491853863 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:48.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff88effd640 1 -- 172.21.15.45:0/491853863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 msgr2=0x7ff89015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:48.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff88effd640 1 --2- 172.21.15.45:0/491853863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 0x7ff89015f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.132+0000 7ff88effd640 1 -- 172.21.15.45:0/491853863 >> v1:172.21.15.190:6789/0 conn(0x7ff89015b850 legacy=0x7ff8900b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:48.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.133+0000 7ff88effd640 1 -- 172.21.15.45:0/491853863 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89015c360 con 0x7ff8900b35b0 2023-11-28T22:08:48.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.134+0000 7ff88effd640 1 --2- 172.21.15.45:0/491853863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 0x7ff89015ca10 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7ff880009870 tx=0x7ff88002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdabc09fd73af1b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:48.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.134+0000 7ff88e7fc640 1 -- 172.21.15.45:0/491853863 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff880031640 con 0x7ff8900b35b0 2023-11-28T22:08:48.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.134+0000 7ff88e7fc640 1 -- 172.21.15.45:0/491853863 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8800317a0 con 0x7ff8900b35b0 2023-11-28T22:08:48.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.134+0000 7ff88e7fc640 1 -- 172.21.15.45:0/491853863 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff880031b30 con 0x7ff8900b35b0 2023-11-28T22:08:48.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.135+0000 7ff8953c9640 1 -- 172.21.15.45:0/491853863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 msgr2=0x7ff89015ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.135+0000 7ff8953c9640 1 --2- 172.21.15.45:0/491853863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 0x7ff89015ca10 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7ff880009870 tx=0x7ff88002b8f0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.135+0000 7ff8953c9640 1 -- 172.21.15.45:0/491853863 shutdown_connections 2023-11-28T22:08:48.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.135+0000 7ff8953c9640 1 --2- 172.21.15.45:0/491853863 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 0x7ff89015f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.135+0000 7ff8953c9640 1 --2- 172.21.15.45:0/491853863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 0x7ff89015ca10 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.135+0000 7ff8953c9640 1 -- 172.21.15.45:0/491853863 >> 172.21.15.45:0/491853863 conn(0x7ff89006a500 msgr2=0x7ff89006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:48.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.135+0000 7ff8953c9640 1 -- 172.21.15.45:0/491853863 shutdown_connections 2023-11-28T22:08:48.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.135+0000 7ff8953c9640 1 -- 172.21.15.45:0/491853863 wait complete. 2023-11-28T22:08:48.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.137+0000 7ff8953c9640 1 Processor -- start 2023-11-28T22:08:48.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.137+0000 7ff8953c9640 1 -- start start 2023-11-28T22:08:48.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff8953c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 0x7ff890158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff8953c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 0x7ff89013c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff8953c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89015b850 0x7ff890154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff8953c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89013d8c0 con 0x7ff8900b35b0 2023-11-28T22:08:48.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff8953c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89013cf20 con 0x7ff89015b850 2023-11-28T22:08:48.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff8953c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89013d200 con 0x7ff8900b1340 2023-11-28T22:08:48.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff887fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 0x7ff89013c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff887fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 0x7ff89013c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59594/0 (socket says 172.21.15.45:59594) 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff887fff640 1 -- 172.21.15.45:0/721016520 learned_addr learned my addr 172.21.15.45:0/721016520 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff88effd640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 0x7ff890158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff88f7fe640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89015b850 0x7ff890154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff887fff640 1 -- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 msgr2=0x7ff890158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff887fff640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 0x7ff890158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff887fff640 1 -- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89015b850 msgr2=0x7ff890154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.138+0000 7ff887fff640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89015b850 0x7ff890154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.139+0000 7ff887fff640 1 -- 172.21.15.45:0/721016520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8800094d0 con 0x7ff8900b35b0 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.139+0000 7ff88effd640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 0x7ff890158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:48.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.139+0000 7ff887fff640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 0x7ff89013c110 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff87800b3c0 tx=0x7ff87800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:48.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.139+0000 7ff8877fe640 1 -- 172.21.15.45:0/721016520 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878011830 con 0x7ff8900b35b0 2023-11-28T22:08:48.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.139+0000 7ff8877fe640 1 -- 172.21.15.45:0/721016520 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff878011990 con 0x7ff8900b35b0 2023-11-28T22:08:48.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.139+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89013d4e0 con 0x7ff8900b35b0 2023-11-28T22:08:48.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.139+0000 7ff8877fe640 1 -- 172.21.15.45:0/721016520 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878011d60 con 0x7ff8900b35b0 2023-11-28T22:08:48.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.139+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff890154e70 con 0x7ff8900b35b0 2023-11-28T22:08:48.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.140+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89013da00 con 0x7ff8900b35b0 2023-11-28T22:08:48.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.141+0000 7ff8877fe640 1 -- 172.21.15.45:0/721016520 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff87801b460 con 0x7ff8900b35b0 2023-11-28T22:08:48.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.142+0000 7ff8877fe640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff87006af60 0x7ff87006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.142+0000 7ff8877fe640 1 -- 172.21.15.45:0/721016520 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff878089710 con 0x7ff8900b35b0 2023-11-28T22:08:48.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.143+0000 7ff88effd640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff87006af60 0x7ff87006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.144+0000 7ff88effd640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff87006af60 0x7ff87006d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff880036000 tx=0x7ff88003c040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:48.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.147+0000 7ff8877fe640 1 -- 172.21.15.45:0/721016520 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff89013da00 con 0x7ff8900b35b0 2023-11-28T22:08:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.338+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff89013da00 con 0x7ff8900b35b0 2023-11-28T22:08:48.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.341+0000 7ff8877fe640 1 -- 172.21.15.45:0/721016520 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v215) v1 ==== 74+0+16752 (secure 0 0 0) 0x7ff89013da00 con 0x7ff8900b35b0 2023-11-28T22:08:48.344 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:48.344 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":215,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:08:45.315905+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-11-28T22:08:41.574328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-11-28T22:08:42.812303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"215","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:08:48.346 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff87006af60 msgr2=0x7ff87006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff87006af60 0x7ff87006d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff880036000 tx=0x7ff88003c040 comp rx=0 tx=0).stop 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 msgr2=0x7ff89013c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 0x7ff89013c110 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff87800b3c0 tx=0x7ff87800b880 comp rx=0 tx=0).stop 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 shutdown_connections 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89015b850 0x7ff890154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff87006af60 0x7ff87006d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8900b35b0 0x7ff89013c110 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 --2- 172.21.15.45:0/721016520 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff8900b1340 0x7ff890158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 >> 172.21.15.45:0/721016520 conn(0x7ff89006a500 msgr2=0x7ff8900aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:48.347 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 shutdown_connections 2023-11-28T22:08:48.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.344+0000 7ff8953c9640 1 -- 172.21.15.45:0/721016520 wait complete. 2023-11-28T22:08:48.364 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T22:08:48.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T22:08:48.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.585+0000 7f6646d81640 1 Processor -- start 2023-11-28T22:08:48.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.586+0000 7f6646d81640 1 -- start start 2023-11-28T22:08:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.587+0000 7f6646d81640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6640155780 0x7f66400abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.587+0000 7f6646d81640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664015c3c0 con 0x7f66400ce800 2023-11-28T22:08:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.587+0000 7f6646d81640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6640155b60 con 0x7f66400ae540 2023-11-28T22:08:48.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.587+0000 7f6646d81640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664015ba90 con 0x7f6640155780 2023-11-28T22:08:48.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.587+0000 7f6646580640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f66400ce800 0x7f66400b20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48254/0 (socket says 172.21.15.45:48254) 2023-11-28T22:08:48.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.587+0000 7f6646580640 1 -- 172.21.15.45:0/4228134023 learned_addr learned my addr 172.21.15.45:0/4228134023 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6645d7f640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6640155780 0x7f66400abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533975826 0 0) 0x7f664015c3c0 con 0x7f66400ce800 2023-11-28T22:08:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f662c003520 con 0x7f66400ce800 2023-11-28T22:08:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724788476 0 0) 0x7f662c003520 con 0x7f66400ce800 2023-11-28T22:08:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6640155780 msgr2=0x7f66400abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6640155780 0x7f66400abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 >> v1:172.21.15.190:6789/0 conn(0x7f66400ae540 legacy=0x7f66400ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664015bd70 con 0x7f66400ce800 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f66380019f0 con 0x7f66400ce800 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 >> v1:172.21.15.45:6789/0 conn(0x7f66400ce800 legacy=0x7f66400b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f662c0039a0 0x7f662c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f662c0047c0 0x7f662c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f662c005180 0x7f662c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664015ba90 con 0x7f662c005180 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c00c050 con 0x7f662c0039a0 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662c00c210 con 0x7f662c0047c0 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.588+0000 7f6645d7f640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6640155780 0x7f66400abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.589+0000 7f6646580640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f662c005180 0x7f662c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.589+0000 7f664557e640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f662c0047c0 0x7f662c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.589+0000 7f6645d7f640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f662c0039a0 0x7f662c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.589+0000 7f6646580640 1 -- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f662c0047c0 msgr2=0x7f662c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.589+0000 7f6646580640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f662c0047c0 0x7f662c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.589+0000 7f6646580640 1 -- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f662c0039a0 msgr2=0x7f662c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.589+0000 7f6646580640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f662c0039a0 0x7f662c003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.589+0000 7f6646580640 1 -- 172.21.15.45:0/4228134023 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c003520 con 0x7f662c005180 2023-11-28T22:08:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.590+0000 7f6646580640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f662c005180 0x7f662c00b980 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f663800a830 tx=0x7f662c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47b885d22806c53 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.591+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380072d0 con 0x7f662c005180 2023-11-28T22:08:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.591+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663802f070 con 0x7f662c005180 2023-11-28T22:08:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.591+0000 7f6644d7d640 1 -- 172.21.15.45:0/4228134023 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380076b0 con 0x7f662c005180 2023-11-28T22:08:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.591+0000 7f6646d81640 1 -- 172.21.15.45:0/4228134023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f662c005180 msgr2=0x7f662c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.591+0000 7f6646d81640 1 --2- 172.21.15.45:0/4228134023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f662c005180 0x7f662c00b980 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f663800a830 tx=0x7f662c0057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.591+0000 7f6646580640 1 -- 172.21.15.45:0/4228134023 reap_dead start 2023-11-28T22:08:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.591+0000 7f6646d81640 1 -- 172.21.15.45:0/4228134023 shutdown_connections 2023-11-28T22:08:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.591+0000 7f6646d81640 1 -- 172.21.15.45:0/4228134023 >> 172.21.15.45:0/4228134023 conn(0x7f664006ab30 msgr2=0x7f66400b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.591+0000 7f6646d81640 1 -- 172.21.15.45:0/4228134023 shutdown_connections 2023-11-28T22:08:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.592+0000 7f6646d81640 1 -- 172.21.15.45:0/4228134023 wait complete. 2023-11-28T22:08:48.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.593+0000 7f6646d81640 1 Processor -- start 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.593+0000 7f6646d81640 1 -- start start 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.594+0000 7f6646d81640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f662c005180 0x7f6640144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.594+0000 7f6646d81640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640156860 0x7f66400ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.594+0000 7f6646d81640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6640156ce0 0x7f66400adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.594+0000 7f6646d81640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640151100 con 0x7f6640156860 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.594+0000 7f6646d81640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640155df0 con 0x7f662c005180 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.594+0000 7f6646d81640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664009c5e0 con 0x7f6640156ce0 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.595+0000 7f664557e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640156860 0x7f66400ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.595+0000 7f664557e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640156860 0x7f66400ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59610/0 (socket says 172.21.15.45:59610) 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.595+0000 7f664557e640 1 -- 172.21.15.45:0/3147955618 learned_addr learned my addr 172.21.15.45:0/3147955618 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:48.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.595+0000 7f664557e640 1 -- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6640156ce0 msgr2=0x7f66400adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.595+0000 7f664557e640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6640156ce0 0x7f66400adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.595+0000 7f6645d7f640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f662c005180 0x7f6640144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.595+0000 7f664557e640 1 -- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f662c005180 msgr2=0x7f6640144930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.595+0000 7f664557e640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f662c005180 0x7f6640144930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.595+0000 7f664557e640 1 -- 172.21.15.45:0/3147955618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640156060 con 0x7f6640156860 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.596+0000 7f664557e640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640156860 0x7f66400ad9a0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f663000e500 tx=0x7f663000e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.596+0000 7f66367fc640 1 -- 172.21.15.45:0/3147955618 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66300052e0 con 0x7f6640156860 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.596+0000 7f66367fc640 1 -- 172.21.15.45:0/3147955618 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6630005440 con 0x7f6640156860 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.596+0000 7f66367fc640 1 -- 172.21.15.45:0/3147955618 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630003810 con 0x7f6640156860 2023-11-28T22:08:48.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.596+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6640156340 con 0x7f6640156860 2023-11-28T22:08:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.596+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664009b9d0 con 0x7f6640156860 2023-11-28T22:08:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.597+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66400b0fc0 con 0x7f6640156860 2023-11-28T22:08:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.598+0000 7f66367fc640 1 -- 172.21.15.45:0/3147955618 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6630003970 con 0x7f6640156860 2023-11-28T22:08:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.599+0000 7f66367fc640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f662406af60 0x7f662406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.599+0000 7f66367fc640 1 -- 172.21.15.45:0/3147955618 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6630089750 con 0x7f6640156860 2023-11-28T22:08:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.600+0000 7f6645d7f640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f662406af60 0x7f662406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.601+0000 7f6645d7f640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f662406af60 0x7f662406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f663c006fd0 tx=0x7f663c008040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.603+0000 7f66367fc640 1 -- 172.21.15.45:0/3147955618 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6630054470 con 0x7f6640156860 2023-11-28T22:08:48.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.793+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 --> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f664015b740 con 0x7f662406af60 2023-11-28T22:08:48.824 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.822+0000 7f66367fc640 1 -- 172.21.15.45:0/3147955618 <== mgr.7643 v2:172.21.15.190:6832/397715021 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286773 (secure 0 0 0) 0x7f663c00c040 con 0x7f662406af60 2023-11-28T22:08:48.826 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:48.830 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T22:08:48.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f662406af60 msgr2=0x7f662406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f662406af60 0x7f662406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f663c006fd0 tx=0x7f663c008040 comp rx=0 tx=0).stop 2023-11-28T22:08:48.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640156860 msgr2=0x7f66400ad9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:48.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640156860 0x7f66400ad9a0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f663000e500 tx=0x7f663000e9c0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 shutdown_connections 2023-11-28T22:08:48.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6640156ce0 0x7f66400adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f662406af60 0x7f662406d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640156860 0x7f66400ad9a0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 --2- 172.21.15.45:0/3147955618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f662c005180 0x7f6640144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:48.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 >> 172.21.15.45:0/3147955618 conn(0x7f664006ab30 msgr2=0x7f664015d640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:48.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 shutdown_connections 2023-11-28T22:08:48.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:48.829+0000 7f6646d81640 1 -- 172.21.15.45:0/3147955618 wait complete. 2023-11-28T22:08:48.854 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-28T22:08:47.579917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2053,"num_read_kb":4245,"num_write":595,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4871,"ondisk_log_size":4871,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436928,"kb_used_data":22268,"kb_used_omap":12,"kb_used_meta":414515,"kb_avail":754537792,"statfs":{"total":773094113280,"available":772646699008,"internally_reserved":0,"allocated":22802432,"data_stored":12244444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":424463920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.298035"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321246+0000","last_change":"2023-11-28T22:08:44.354200+0000","last_active":"2023-11-28T22:08:45.321246+0000","last_peered":"2023-11-28T22:08:45.321246+0000","last_clean":"2023-11-28T22:08:45.321246+0000","last_became_active":"2023-11-28T22:08:44.353849+0000","last_became_peered":"2023-11-28T22:08:44.353849+0000","last_unstale":"2023-11-28T22:08:45.321246+0000","last_undegraded":"2023-11-28T22:08:45.321246+0000","last_fullsized":"2023-11-28T22:08:45.321246+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:18:22.337269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320895+0000","last_change":"2023-11-28T22:08:44.344177+0000","last_active":"2023-11-28T22:08:45.320895+0000","last_peered":"2023-11-28T22:08:45.320895+0000","last_clean":"2023-11-28T22:08:45.320895+0000","last_became_active":"2023-11-28T22:08:44.343880+0000","last_became_peered":"2023-11-28T22:08:44.343880+0000","last_unstale":"2023-11-28T22:08:45.320895+0000","last_undegraded":"2023-11-28T22:08:45.320895+0000","last_fullsized":"2023-11-28T22:08:45.320895+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:32:01.808458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657888+0000","last_change":"2023-11-28T22:08:44.345774+0000","last_active":"2023-11-28T22:08:45.657888+0000","last_peered":"2023-11-28T22:08:45.657888+0000","last_clean":"2023-11-28T22:08:45.657888+0000","last_became_active":"2023-11-28T22:08:44.345409+0000","last_became_peered":"2023-11-28T22:08:44.345409+0000","last_unstale":"2023-11-28T22:08:45.657888+0000","last_undegraded":"2023-11-28T22:08:45.657888+0000","last_fullsized":"2023-11-28T22:08:45.657888+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:19:09.037444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657253+0000","last_change":"2023-11-28T22:08:44.347680+0000","last_active":"2023-11-28T22:08:45.657253+0000","last_peered":"2023-11-28T22:08:45.657253+0000","last_clean":"2023-11-28T22:08:45.657253+0000","last_became_active":"2023-11-28T22:08:44.347366+0000","last_became_peered":"2023-11-28T22:08:44.347366+0000","last_unstale":"2023-11-28T22:08:45.657253+0000","last_undegraded":"2023-11-28T22:08:45.657253+0000","last_fullsized":"2023-11-28T22:08:45.657253+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:30.647873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323554+0000","last_change":"2023-11-28T22:08:44.346142+0000","last_active":"2023-11-28T22:08:45.323554+0000","last_peered":"2023-11-28T22:08:45.323554+0000","last_clean":"2023-11-28T22:08:45.323554+0000","last_became_active":"2023-11-28T22:08:44.345821+0000","last_became_peered":"2023-11-28T22:08:44.345821+0000","last_unstale":"2023-11-28T22:08:45.323554+0000","last_undegraded":"2023-11-28T22:08:45.323554+0000","last_fullsized":"2023-11-28T22:08:45.323554+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:53.184945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323519+0000","last_change":"2023-11-28T22:08:44.364735+0000","last_active":"2023-11-28T22:08:45.323519+0000","last_peered":"2023-11-28T22:08:45.323519+0000","last_clean":"2023-11-28T22:08:45.323519+0000","last_became_active":"2023-11-28T22:08:44.364393+0000","last_became_peered":"2023-11-28T22:08:44.364393+0000","last_unstale":"2023-11-28T22:08:45.323519+0000","last_undegraded":"2023-11-28T22:08:45.323519+0000","last_fullsized":"2023-11-28T22:08:45.323519+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:18:18.813791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655708+0000","last_change":"2023-11-28T22:08:44.355251+0000","last_active":"2023-11-28T22:08:45.655708+0000","last_peered":"2023-11-28T22:08:45.655708+0000","last_clean":"2023-11-28T22:08:45.655708+0000","last_became_active":"2023-11-28T22:08:44.354974+0000","last_became_peered":"2023-11-28T22:08:44.354974+0000","last_unstale":"2023-11-28T22:08:45.655708+0000","last_undegraded":"2023-11-28T22:08:45.655708+0000","last_fullsized":"2023-11-28T22:08:45.655708+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:28.164258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656953+0000","last_change":"2023-11-28T22:08:44.355044+0000","last_active":"2023-11-28T22:08:45.656953+0000","last_peered":"2023-11-28T22:08:45.656953+0000","last_clean":"2023-11-28T22:08:45.656953+0000","last_became_active":"2023-11-28T22:08:44.354012+0000","last_became_peered":"2023-11-28T22:08:44.354012+0000","last_unstale":"2023-11-28T22:08:45.656953+0000","last_undegraded":"2023-11-28T22:08:45.656953+0000","last_fullsized":"2023-11-28T22:08:45.656953+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:53.107310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321159+0000","last_change":"2023-11-28T22:08:44.348375+0000","last_active":"2023-11-28T22:08:45.321159+0000","last_peered":"2023-11-28T22:08:45.321159+0000","last_clean":"2023-11-28T22:08:45.321159+0000","last_became_active":"2023-11-28T22:08:44.348037+0000","last_became_peered":"2023-11-28T22:08:44.348037+0000","last_unstale":"2023-11-28T22:08:45.321159+0000","last_undegraded":"2023-11-28T22:08:45.321159+0000","last_fullsized":"2023-11-28T22:08:45.321159+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:27.751749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324009+0000","last_change":"2023-11-28T22:08:44.345972+0000","last_active":"2023-11-28T22:08:45.324009+0000","last_peered":"2023-11-28T22:08:45.324009+0000","last_clean":"2023-11-28T22:08:45.324009+0000","last_became_active":"2023-11-28T22:08:44.345670+0000","last_became_peered":"2023-11-28T22:08:44.345670+0000","last_unstale":"2023-11-28T22:08:45.324009+0000","last_undegraded":"2023-11-28T22:08:45.324009+0000","last_fullsized":"2023-11-28T22:08:45.324009+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:11.921386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659320+0000","last_change":"2023-11-28T22:08:44.346368+0000","last_active":"2023-11-28T22:08:45.659320+0000","last_peered":"2023-11-28T22:08:45.659320+0000","last_clean":"2023-11-28T22:08:45.659320+0000","last_became_active":"2023-11-28T22:08:44.345928+0000","last_became_peered":"2023-11-28T22:08:44.345928+0000","last_unstale":"2023-11-28T22:08:45.659320+0000","last_undegraded":"2023-11-28T22:08:45.659320+0000","last_fullsized":"2023-11-28T22:08:45.659320+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:50.077471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657115+0000","last_change":"2023-11-28T22:08:44.356964+0000","last_active":"2023-11-28T22:08:45.657115+0000","last_peered":"2023-11-28T22:08:45.657115+0000","last_clean":"2023-11-28T22:08:45.657115+0000","last_became_active":"2023-11-28T22:08:44.355131+0000","last_became_peered":"2023-11-28T22:08:44.355131+0000","last_unstale":"2023-11-28T22:08:45.657115+0000","last_undegraded":"2023-11-28T22:08:45.657115+0000","last_fullsized":"2023-11-28T22:08:45.657115+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:53.107310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327442+0000","last_change":"2023-11-28T22:08:44.343174+0000","last_active":"2023-11-28T22:08:45.327442+0000","last_peered":"2023-11-28T22:08:45.327442+0000","last_clean":"2023-11-28T22:08:45.327442+0000","last_became_active":"2023-11-28T22:08:44.342846+0000","last_became_peered":"2023-11-28T22:08:44.342846+0000","last_unstale":"2023-11-28T22:08:45.327442+0000","last_undegraded":"2023-11-28T22:08:45.327442+0000","last_fullsized":"2023-11-28T22:08:45.327442+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:37:02.928782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655664+0000","last_change":"2023-11-28T22:08:44.347680+0000","last_active":"2023-11-28T22:08:45.655664+0000","last_peered":"2023-11-28T22:08:45.655664+0000","last_clean":"2023-11-28T22:08:45.655664+0000","last_became_active":"2023-11-28T22:08:44.347316+0000","last_became_peered":"2023-11-28T22:08:44.347316+0000","last_unstale":"2023-11-28T22:08:45.655664+0000","last_undegraded":"2023-11-28T22:08:45.655664+0000","last_fullsized":"2023-11-28T22:08:45.655664+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:49:11.197502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320642+0000","last_change":"2023-11-28T22:08:44.335536+0000","last_active":"2023-11-28T22:08:45.320642+0000","last_peered":"2023-11-28T22:08:45.320642+0000","last_clean":"2023-11-28T22:08:45.320642+0000","last_became_active":"2023-11-28T22:08:44.334171+0000","last_became_peered":"2023-11-28T22:08:44.334171+0000","last_unstale":"2023-11-28T22:08:45.320642+0000","last_undegraded":"2023-11-28T22:08:45.320642+0000","last_fullsized":"2023-11-28T22:08:45.320642+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:50.056281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323426+0000","last_change":"2023-11-28T22:08:44.353608+0000","last_active":"2023-11-28T22:08:45.323426+0000","last_peered":"2023-11-28T22:08:45.323426+0000","last_clean":"2023-11-28T22:08:45.323426+0000","last_became_active":"2023-11-28T22:08:44.348573+0000","last_became_peered":"2023-11-28T22:08:44.348573+0000","last_unstale":"2023-11-28T22:08:45.323426+0000","last_undegraded":"2023-11-28T22:08:45.323426+0000","last_fullsized":"2023-11-28T22:08:45.323426+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:50:51.787300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324378+0000","last_change":"2023-11-28T22:08:44.356606+0000","last_active":"2023-11-28T22:08:45.324378+0000","last_peered":"2023-11-28T22:08:45.324378+0000","last_clean":"2023-11-28T22:08:45.324378+0000","last_became_active":"2023-11-28T22:08:44.356302+0000","last_became_peered":"2023-11-28T22:08:44.356302+0000","last_unstale":"2023-11-28T22:08:45.324378+0000","last_undegraded":"2023-11-28T22:08:45.324378+0000","last_fullsized":"2023-11-28T22:08:45.324378+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:08:42.041519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328178+0000","last_change":"2023-11-28T22:08:44.355189+0000","last_active":"2023-11-28T22:08:45.328178+0000","last_peered":"2023-11-28T22:08:45.328178+0000","last_clean":"2023-11-28T22:08:45.328178+0000","last_became_active":"2023-11-28T22:08:44.354425+0000","last_became_peered":"2023-11-28T22:08:44.354425+0000","last_unstale":"2023-11-28T22:08:45.328178+0000","last_undegraded":"2023-11-28T22:08:45.328178+0000","last_fullsized":"2023-11-28T22:08:45.328178+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:41:57.181203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327885+0000","last_change":"2023-11-28T22:08:44.353039+0000","last_active":"2023-11-28T22:08:45.327885+0000","last_peered":"2023-11-28T22:08:45.327885+0000","last_clean":"2023-11-28T22:08:45.327885+0000","last_became_active":"2023-11-28T22:08:44.351399+0000","last_became_peered":"2023-11-28T22:08:44.351399+0000","last_unstale":"2023-11-28T22:08:45.327885+0000","last_undegraded":"2023-11-28T22:08:45.327885+0000","last_fullsized":"2023-11-28T22:08:45.327885+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:19:44.782055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323726+0000","last_change":"2023-11-28T22:08:44.356856+0000","last_active":"2023-11-28T22:08:45.323726+0000","last_peered":"2023-11-28T22:08:45.323726+0000","last_clean":"2023-11-28T22:08:45.323726+0000","last_became_active":"2023-11-28T22:08:44.356587+0000","last_became_peered":"2023-11-28T22:08:44.356587+0000","last_unstale":"2023-11-28T22:08:45.323726+0000","last_undegraded":"2023-11-28T22:08:45.323726+0000","last_fullsized":"2023-11-28T22:08:45.323726+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:32:19.991433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321209+0000","last_change":"2023-11-28T22:08:44.349209+0000","last_active":"2023-11-28T22:08:45.321209+0000","last_peered":"2023-11-28T22:08:45.321209+0000","last_clean":"2023-11-28T22:08:45.321209+0000","last_became_active":"2023-11-28T22:08:44.348049+0000","last_became_peered":"2023-11-28T22:08:44.348049+0000","last_unstale":"2023-11-28T22:08:45.321209+0000","last_undegraded":"2023-11-28T22:08:45.321209+0000","last_fullsized":"2023-11-28T22:08:45.321209+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:34.168754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320662+0000","last_change":"2023-11-28T22:08:44.352696+0000","last_active":"2023-11-28T22:08:45.320662+0000","last_peered":"2023-11-28T22:08:45.320662+0000","last_clean":"2023-11-28T22:08:45.320662+0000","last_became_active":"2023-11-28T22:08:44.348105+0000","last_became_peered":"2023-11-28T22:08:44.348105+0000","last_unstale":"2023-11-28T22:08:45.320662+0000","last_undegraded":"2023-11-28T22:08:45.320662+0000","last_fullsized":"2023-11-28T22:08:45.320662+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:38:05.221760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656951+0000","last_change":"2023-11-28T22:08:44.363713+0000","last_active":"2023-11-28T22:08:45.656951+0000","last_peered":"2023-11-28T22:08:45.656951+0000","last_clean":"2023-11-28T22:08:45.656951+0000","last_became_active":"2023-11-28T22:08:44.363425+0000","last_became_peered":"2023-11-28T22:08:44.363425+0000","last_unstale":"2023-11-28T22:08:45.656951+0000","last_undegraded":"2023-11-28T22:08:45.656951+0000","last_fullsized":"2023-11-28T22:08:45.656951+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:30.647873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320452+0000","last_change":"2023-11-28T22:08:44.354641+0000","last_active":"2023-11-28T22:08:45.320452+0000","last_peered":"2023-11-28T22:08:45.320452+0000","last_clean":"2023-11-28T22:08:45.320452+0000","last_became_active":"2023-11-28T22:08:44.354317+0000","last_became_peered":"2023-11-28T22:08:44.354317+0000","last_unstale":"2023-11-28T22:08:45.320452+0000","last_undegraded":"2023-11-28T22:08:45.320452+0000","last_fullsized":"2023-11-28T22:08:45.320452+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:39:41.245262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656571+0000","last_change":"2023-11-28T22:08:44.361169+0000","last_active":"2023-11-28T22:08:45.656571+0000","last_peered":"2023-11-28T22:08:45.656571+0000","last_clean":"2023-11-28T22:08:45.656571+0000","last_became_active":"2023-11-28T22:08:44.360790+0000","last_became_peered":"2023-11-28T22:08:44.360790+0000","last_unstale":"2023-11-28T22:08:45.656571+0000","last_undegraded":"2023-11-28T22:08:45.656571+0000","last_fullsized":"2023-11-28T22:08:45.656571+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:17.115567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660339+0000","last_change":"2023-11-28T22:08:44.349408+0000","last_active":"2023-11-28T22:08:45.660339+0000","last_peered":"2023-11-28T22:08:45.660339+0000","last_clean":"2023-11-28T22:08:45.660339+0000","last_became_active":"2023-11-28T22:08:44.348642+0000","last_became_peered":"2023-11-28T22:08:44.348642+0000","last_unstale":"2023-11-28T22:08:45.660339+0000","last_undegraded":"2023-11-28T22:08:45.660339+0000","last_fullsized":"2023-11-28T22:08:45.660339+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:22.784612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659606+0000","last_change":"2023-11-28T22:08:44.355649+0000","last_active":"2023-11-28T22:08:45.659606+0000","last_peered":"2023-11-28T22:08:45.659606+0000","last_clean":"2023-11-28T22:08:45.659606+0000","last_became_active":"2023-11-28T22:08:44.353617+0000","last_became_peered":"2023-11-28T22:08:44.353617+0000","last_unstale":"2023-11-28T22:08:45.659606+0000","last_undegraded":"2023-11-28T22:08:45.659606+0000","last_fullsized":"2023-11-28T22:08:45.659606+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:40:06.791697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323763+0000","last_change":"2023-11-28T22:08:44.347981+0000","last_active":"2023-11-28T22:08:45.323763+0000","last_peered":"2023-11-28T22:08:45.323763+0000","last_clean":"2023-11-28T22:08:45.323763+0000","last_became_active":"2023-11-28T22:08:44.346763+0000","last_became_peered":"2023-11-28T22:08:44.346763+0000","last_unstale":"2023-11-28T22:08:45.323763+0000","last_undegraded":"2023-11-28T22:08:45.323763+0000","last_fullsized":"2023-11-28T22:08:45.323763+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:05:02.093775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658425+0000","last_change":"2023-11-28T22:08:44.344267+0000","last_active":"2023-11-28T22:08:45.658425+0000","last_peered":"2023-11-28T22:08:45.658425+0000","last_clean":"2023-11-28T22:08:45.658425+0000","last_became_active":"2023-11-28T22:08:44.343864+0000","last_became_peered":"2023-11-28T22:08:44.343864+0000","last_unstale":"2023-11-28T22:08:45.658425+0000","last_undegraded":"2023-11-28T22:08:45.658425+0000","last_fullsized":"2023-11-28T22:08:45.658425+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:44.277097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657849+0000","last_change":"2023-11-28T22:08:44.366000+0000","last_active":"2023-11-28T22:08:45.657849+0000","last_peered":"2023-11-28T22:08:45.657849+0000","last_clean":"2023-11-28T22:08:45.657849+0000","last_became_active":"2023-11-28T22:08:44.365688+0000","last_became_peered":"2023-11-28T22:08:44.365688+0000","last_unstale":"2023-11-28T22:08:45.657849+0000","last_undegraded":"2023-11-28T22:08:45.657849+0000","last_fullsized":"2023-11-28T22:08:45.657849+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:50:01.770791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655353+0000","last_change":"2023-11-28T22:08:44.348577+0000","last_active":"2023-11-28T22:08:45.655353+0000","last_peered":"2023-11-28T22:08:45.655353+0000","last_clean":"2023-11-28T22:08:45.655353+0000","last_became_active":"2023-11-28T22:08:44.347988+0000","last_became_peered":"2023-11-28T22:08:44.347988+0000","last_unstale":"2023-11-28T22:08:45.655353+0000","last_undegraded":"2023-11-28T22:08:45.655353+0000","last_fullsized":"2023-11-28T22:08:45.655353+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:08:01.770933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323562+0000","last_change":"2023-11-28T22:08:44.349028+0000","last_active":"2023-11-28T22:08:45.323562+0000","last_peered":"2023-11-28T22:08:45.323562+0000","last_clean":"2023-11-28T22:08:45.323562+0000","last_became_active":"2023-11-28T22:08:44.348218+0000","last_became_peered":"2023-11-28T22:08:44.348218+0000","last_unstale":"2023-11-28T22:08:45.323562+0000","last_undegraded":"2023-11-28T22:08:45.323562+0000","last_fullsized":"2023-11-28T22:08:45.323562+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:47.357945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659158+0000","last_change":"2023-11-28T22:08:44.354508+0000","last_active":"2023-11-28T22:08:45.659158+0000","last_peered":"2023-11-28T22:08:45.659158+0000","last_clean":"2023-11-28T22:08:45.659158+0000","last_became_active":"2023-11-28T22:08:44.354220+0000","last_became_peered":"2023-11-28T22:08:44.354220+0000","last_unstale":"2023-11-28T22:08:45.659158+0000","last_undegraded":"2023-11-28T22:08:45.659158+0000","last_fullsized":"2023-11-28T22:08:45.659158+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:53:26.407432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"214'4","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657789+0000","last_change":"2023-11-28T22:08:43.320868+0000","last_active":"2023-11-28T22:08:45.657789+0000","last_peered":"2023-11-28T22:08:45.657789+0000","last_clean":"2023-11-28T22:08:45.657789+0000","last_became_active":"2023-11-28T22:08:43.320600+0000","last_became_peered":"2023-11-28T22:08:43.320600+0000","last_unstale":"2023-11-28T22:08:45.657789+0000","last_undegraded":"2023-11-28T22:08:45.657789+0000","last_fullsized":"2023-11-28T22:08:45.657789+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:18:59.336441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656509+0000","last_change":"2023-11-28T22:08:44.357609+0000","last_active":"2023-11-28T22:08:45.656509+0000","last_peered":"2023-11-28T22:08:45.656509+0000","last_clean":"2023-11-28T22:08:45.656509+0000","last_became_active":"2023-11-28T22:08:44.357345+0000","last_became_peered":"2023-11-28T22:08:44.357345+0000","last_unstale":"2023-11-28T22:08:45.656509+0000","last_undegraded":"2023-11-28T22:08:45.656509+0000","last_fullsized":"2023-11-28T22:08:45.656509+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:57.579829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324118+0000","last_change":"2023-11-28T22:08:43.313174+0000","last_active":"2023-11-28T22:08:45.324118+0000","last_peered":"2023-11-28T22:08:45.324118+0000","last_clean":"2023-11-28T22:08:45.324118+0000","last_became_active":"2023-11-28T22:08:43.312907+0000","last_became_peered":"2023-11-28T22:08:43.312907+0000","last_unstale":"2023-11-28T22:08:45.324118+0000","last_undegraded":"2023-11-28T22:08:45.324118+0000","last_fullsized":"2023-11-28T22:08:45.324118+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:42:51.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660528+0000","last_change":"2023-11-28T22:08:44.355843+0000","last_active":"2023-11-28T22:08:45.660528+0000","last_peered":"2023-11-28T22:08:45.660528+0000","last_clean":"2023-11-28T22:08:45.660528+0000","last_became_active":"2023-11-28T22:08:44.355577+0000","last_became_peered":"2023-11-28T22:08:44.355577+0000","last_unstale":"2023-11-28T22:08:45.660528+0000","last_undegraded":"2023-11-28T22:08:45.660528+0000","last_fullsized":"2023-11-28T22:08:45.660528+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:30:20.094815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659565+0000","last_change":"2023-11-28T22:08:43.314834+0000","last_active":"2023-11-28T22:08:45.659565+0000","last_peered":"2023-11-28T22:08:45.659565+0000","last_clean":"2023-11-28T22:08:45.659565+0000","last_became_active":"2023-11-28T22:08:43.313415+0000","last_became_peered":"2023-11-28T22:08:43.313415+0000","last_unstale":"2023-11-28T22:08:45.659565+0000","last_undegraded":"2023-11-28T22:08:45.659565+0000","last_fullsized":"2023-11-28T22:08:45.659565+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:52.151763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657855+0000","last_change":"2023-11-28T22:08:44.346534+0000","last_active":"2023-11-28T22:08:45.657855+0000","last_peered":"2023-11-28T22:08:45.657855+0000","last_clean":"2023-11-28T22:08:45.657855+0000","last_became_active":"2023-11-28T22:08:44.346153+0000","last_became_peered":"2023-11-28T22:08:44.346153+0000","last_unstale":"2023-11-28T22:08:45.657855+0000","last_undegraded":"2023-11-28T22:08:45.657855+0000","last_fullsized":"2023-11-28T22:08:45.657855+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:35:32.804621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328210+0000","last_change":"2023-11-28T22:08:43.319845+0000","last_active":"2023-11-28T22:08:45.328210+0000","last_peered":"2023-11-28T22:08:45.328210+0000","last_clean":"2023-11-28T22:08:45.328210+0000","last_became_active":"2023-11-28T22:08:43.319549+0000","last_became_peered":"2023-11-28T22:08:43.319549+0000","last_unstale":"2023-11-28T22:08:45.328210+0000","last_undegraded":"2023-11-28T22:08:45.328210+0000","last_fullsized":"2023-11-28T22:08:45.328210+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:06.730371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327023+0000","last_change":"2023-11-28T22:08:44.361496+0000","last_active":"2023-11-28T22:08:45.327023+0000","last_peered":"2023-11-28T22:08:45.327023+0000","last_clean":"2023-11-28T22:08:45.327023+0000","last_became_active":"2023-11-28T22:08:44.361177+0000","last_became_peered":"2023-11-28T22:08:44.361177+0000","last_unstale":"2023-11-28T22:08:45.327023+0000","last_undegraded":"2023-11-28T22:08:45.327023+0000","last_fullsized":"2023-11-28T22:08:45.327023+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:59.637363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323660+0000","last_change":"2023-11-28T22:08:43.324359+0000","last_active":"2023-11-28T22:08:45.323660+0000","last_peered":"2023-11-28T22:08:45.323660+0000","last_clean":"2023-11-28T22:08:45.323660+0000","last_became_active":"2023-11-28T22:08:43.323933+0000","last_became_peered":"2023-11-28T22:08:43.323933+0000","last_unstale":"2023-11-28T22:08:45.323660+0000","last_undegraded":"2023-11-28T22:08:45.323660+0000","last_fullsized":"2023-11-28T22:08:45.323660+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:40.625533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320810+0000","last_change":"2023-11-28T22:08:44.345155+0000","last_active":"2023-11-28T22:08:45.320810+0000","last_peered":"2023-11-28T22:08:45.320810+0000","last_clean":"2023-11-28T22:08:45.320810+0000","last_became_active":"2023-11-28T22:08:44.344846+0000","last_became_peered":"2023-11-28T22:08:44.344846+0000","last_unstale":"2023-11-28T22:08:45.320810+0000","last_undegraded":"2023-11-28T22:08:45.320810+0000","last_fullsized":"2023-11-28T22:08:45.320810+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:09:55.289797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656876+0000","last_change":"2023-11-28T22:08:43.327207+0000","last_active":"2023-11-28T22:08:45.656876+0000","last_peered":"2023-11-28T22:08:45.656876+0000","last_clean":"2023-11-28T22:08:45.656876+0000","last_became_active":"2023-11-28T22:08:43.326921+0000","last_became_peered":"2023-11-28T22:08:43.326921+0000","last_unstale":"2023-11-28T22:08:45.656876+0000","last_undegraded":"2023-11-28T22:08:45.656876+0000","last_fullsized":"2023-11-28T22:08:45.656876+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:19:35.409181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320745+0000","last_change":"2023-11-28T22:08:44.352117+0000","last_active":"2023-11-28T22:08:45.320745+0000","last_peered":"2023-11-28T22:08:45.320745+0000","last_clean":"2023-11-28T22:08:45.320745+0000","last_became_active":"2023-11-28T22:08:44.343825+0000","last_became_peered":"2023-11-28T22:08:44.343825+0000","last_unstale":"2023-11-28T22:08:45.320745+0000","last_undegraded":"2023-11-28T22:08:45.320745+0000","last_fullsized":"2023-11-28T22:08:45.320745+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:52:29.867964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327928+0000","last_change":"2023-11-28T22:08:43.320287+0000","last_active":"2023-11-28T22:08:45.327928+0000","last_peered":"2023-11-28T22:08:45.327928+0000","last_clean":"2023-11-28T22:08:45.327928+0000","last_became_active":"2023-11-28T22:08:43.318097+0000","last_became_peered":"2023-11-28T22:08:43.318097+0000","last_unstale":"2023-11-28T22:08:45.327928+0000","last_undegraded":"2023-11-28T22:08:45.327928+0000","last_fullsized":"2023-11-28T22:08:45.327928+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:54:37.447100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659190+0000","last_change":"2023-11-28T22:08:44.356005+0000","last_active":"2023-11-28T22:08:45.659190+0000","last_peered":"2023-11-28T22:08:45.659190+0000","last_clean":"2023-11-28T22:08:45.659190+0000","last_became_active":"2023-11-28T22:08:44.355694+0000","last_became_peered":"2023-11-28T22:08:44.355694+0000","last_unstale":"2023-11-28T22:08:45.659190+0000","last_undegraded":"2023-11-28T22:08:45.659190+0000","last_fullsized":"2023-11-28T22:08:45.659190+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:48:40.533350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324274+0000","last_change":"2023-11-28T22:08:43.312754+0000","last_active":"2023-11-28T22:08:45.324274+0000","last_peered":"2023-11-28T22:08:45.324274+0000","last_clean":"2023-11-28T22:08:45.324274+0000","last_became_active":"2023-11-28T22:08:43.310889+0000","last_became_peered":"2023-11-28T22:08:43.310889+0000","last_unstale":"2023-11-28T22:08:45.324274+0000","last_undegraded":"2023-11-28T22:08:45.324274+0000","last_fullsized":"2023-11-28T22:08:45.324274+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:44:57.568897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323945+0000","last_change":"2023-11-28T22:08:44.358994+0000","last_active":"2023-11-28T22:08:45.323945+0000","last_peered":"2023-11-28T22:08:45.323945+0000","last_clean":"2023-11-28T22:08:45.323945+0000","last_became_active":"2023-11-28T22:08:44.354736+0000","last_became_peered":"2023-11-28T22:08:44.354736+0000","last_unstale":"2023-11-28T22:08:45.323945+0000","last_undegraded":"2023-11-28T22:08:45.323945+0000","last_fullsized":"2023-11-28T22:08:45.323945+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:54.673324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321107+0000","last_change":"2023-11-28T22:08:43.316292+0000","last_active":"2023-11-28T22:08:45.321107+0000","last_peered":"2023-11-28T22:08:45.321107+0000","last_clean":"2023-11-28T22:08:45.321107+0000","last_became_active":"2023-11-28T22:08:43.316037+0000","last_became_peered":"2023-11-28T22:08:43.316037+0000","last_unstale":"2023-11-28T22:08:45.321107+0000","last_undegraded":"2023-11-28T22:08:45.321107+0000","last_fullsized":"2023-11-28T22:08:45.321107+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:43:03.567062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328079+0000","last_change":"2023-11-28T22:08:44.343326+0000","last_active":"2023-11-28T22:08:45.328079+0000","last_peered":"2023-11-28T22:08:45.328079+0000","last_clean":"2023-11-28T22:08:45.328079+0000","last_became_active":"2023-11-28T22:08:44.342964+0000","last_became_peered":"2023-11-28T22:08:44.342964+0000","last_unstale":"2023-11-28T22:08:45.328079+0000","last_undegraded":"2023-11-28T22:08:45.328079+0000","last_fullsized":"2023-11-28T22:08:45.328079+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:13:14.601881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"214'3","reported_seq":15,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657385+0000","last_change":"2023-11-28T22:08:43.326251+0000","last_active":"2023-11-28T22:08:45.657385+0000","last_peered":"2023-11-28T22:08:45.657385+0000","last_clean":"2023-11-28T22:08:45.657385+0000","last_became_active":"2023-11-28T22:08:43.325985+0000","last_became_peered":"2023-11-28T22:08:43.325985+0000","last_unstale":"2023-11-28T22:08:45.657385+0000","last_undegraded":"2023-11-28T22:08:45.657385+0000","last_fullsized":"2023-11-28T22:08:45.657385+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:09:39.056123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321347+0000","last_change":"2023-11-28T22:08:44.346762+0000","last_active":"2023-11-28T22:08:45.321347+0000","last_peered":"2023-11-28T22:08:45.321347+0000","last_clean":"2023-11-28T22:08:45.321347+0000","last_became_active":"2023-11-28T22:08:44.346480+0000","last_became_peered":"2023-11-28T22:08:44.346480+0000","last_unstale":"2023-11-28T22:08:45.321347+0000","last_undegraded":"2023-11-28T22:08:45.321347+0000","last_fullsized":"2023-11-28T22:08:45.321347+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:48:56.037544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"214'6","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328484+0000","last_change":"2023-11-28T22:08:43.314229+0000","last_active":"2023-11-28T22:08:45.328484+0000","last_peered":"2023-11-28T22:08:45.328484+0000","last_clean":"2023-11-28T22:08:45.328484+0000","last_became_active":"2023-11-28T22:08:43.313936+0000","last_became_peered":"2023-11-28T22:08:43.313936+0000","last_unstale":"2023-11-28T22:08:45.328484+0000","last_undegraded":"2023-11-28T22:08:45.328484+0000","last_fullsized":"2023-11-28T22:08:45.328484+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:17:21.971727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655041+0000","last_change":"2023-11-28T22:08:44.355681+0000","last_active":"2023-11-28T22:08:45.655041+0000","last_peered":"2023-11-28T22:08:45.655041+0000","last_clean":"2023-11-28T22:08:45.655041+0000","last_became_active":"2023-11-28T22:08:44.355415+0000","last_became_peered":"2023-11-28T22:08:44.355415+0000","last_unstale":"2023-11-28T22:08:45.655041+0000","last_undegraded":"2023-11-28T22:08:45.655041+0000","last_fullsized":"2023-11-28T22:08:45.655041+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:51:35.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328372+0000","last_change":"2023-11-28T22:08:43.317099+0000","last_active":"2023-11-28T22:08:45.328372+0000","last_peered":"2023-11-28T22:08:45.328372+0000","last_clean":"2023-11-28T22:08:45.328372+0000","last_became_active":"2023-11-28T22:08:43.316674+0000","last_became_peered":"2023-11-28T22:08:43.316674+0000","last_unstale":"2023-11-28T22:08:45.328372+0000","last_undegraded":"2023-11-28T22:08:45.328372+0000","last_fullsized":"2023-11-28T22:08:45.328372+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:02:19.708415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323411+0000","last_change":"2023-11-28T22:08:44.350936+0000","last_active":"2023-11-28T22:08:45.323411+0000","last_peered":"2023-11-28T22:08:45.323411+0000","last_clean":"2023-11-28T22:08:45.323411+0000","last_became_active":"2023-11-28T22:08:44.350659+0000","last_became_peered":"2023-11-28T22:08:44.350659+0000","last_unstale":"2023-11-28T22:08:45.323411+0000","last_undegraded":"2023-11-28T22:08:45.323411+0000","last_fullsized":"2023-11-28T22:08:45.323411+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:43:30.287811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320832+0000","last_change":"2023-11-28T22:08:43.328483+0000","last_active":"2023-11-28T22:08:45.320832+0000","last_peered":"2023-11-28T22:08:45.320832+0000","last_clean":"2023-11-28T22:08:45.320832+0000","last_became_active":"2023-11-28T22:08:43.328124+0000","last_became_peered":"2023-11-28T22:08:43.328124+0000","last_unstale":"2023-11-28T22:08:45.320832+0000","last_undegraded":"2023-11-28T22:08:45.320832+0000","last_fullsized":"2023-11-28T22:08:45.320832+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:52.662588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327868+0000","last_change":"2023-11-28T22:08:44.352432+0000","last_active":"2023-11-28T22:08:45.327868+0000","last_peered":"2023-11-28T22:08:45.327868+0000","last_clean":"2023-11-28T22:08:45.327868+0000","last_became_active":"2023-11-28T22:08:44.352169+0000","last_became_peered":"2023-11-28T22:08:44.352169+0000","last_unstale":"2023-11-28T22:08:45.327868+0000","last_undegraded":"2023-11-28T22:08:45.327868+0000","last_fullsized":"2023-11-28T22:08:45.327868+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:08:42.150626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656776+0000","last_change":"2023-11-28T22:08:43.312914+0000","last_active":"2023-11-28T22:08:45.656776+0000","last_peered":"2023-11-28T22:08:45.656776+0000","last_clean":"2023-11-28T22:08:45.656776+0000","last_became_active":"2023-11-28T22:08:43.312634+0000","last_became_peered":"2023-11-28T22:08:43.312634+0000","last_unstale":"2023-11-28T22:08:45.656776+0000","last_undegraded":"2023-11-28T22:08:45.656776+0000","last_fullsized":"2023-11-28T22:08:45.656776+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:29:26.298257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327887+0000","last_change":"2023-11-28T22:08:44.346948+0000","last_active":"2023-11-28T22:08:45.327887+0000","last_peered":"2023-11-28T22:08:45.327887+0000","last_clean":"2023-11-28T22:08:45.327887+0000","last_became_active":"2023-11-28T22:08:44.346649+0000","last_became_peered":"2023-11-28T22:08:44.346649+0000","last_unstale":"2023-11-28T22:08:45.327887+0000","last_undegraded":"2023-11-28T22:08:45.327887+0000","last_fullsized":"2023-11-28T22:08:45.327887+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:18:25.673593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"214'8","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657438+0000","last_change":"2023-11-28T22:08:43.315699+0000","last_active":"2023-11-28T22:08:45.657438+0000","last_peered":"2023-11-28T22:08:45.657438+0000","last_clean":"2023-11-28T22:08:45.657438+0000","last_became_active":"2023-11-28T22:08:43.315402+0000","last_became_peered":"2023-11-28T22:08:43.315402+0000","last_unstale":"2023-11-28T22:08:45.657438+0000","last_undegraded":"2023-11-28T22:08:45.657438+0000","last_fullsized":"2023-11-28T22:08:45.657438+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:36:57.829619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655569+0000","last_change":"2023-11-28T22:08:44.361918+0000","last_active":"2023-11-28T22:08:45.655569+0000","last_peered":"2023-11-28T22:08:45.655569+0000","last_clean":"2023-11-28T22:08:45.655569+0000","last_became_active":"2023-11-28T22:08:44.353476+0000","last_became_peered":"2023-11-28T22:08:44.353476+0000","last_unstale":"2023-11-28T22:08:45.655569+0000","last_undegraded":"2023-11-28T22:08:45.655569+0000","last_fullsized":"2023-11-28T22:08:45.655569+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:18:32.123119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658240+0000","last_change":"2023-11-28T22:08:43.318071+0000","last_active":"2023-11-28T22:08:45.658240+0000","last_peered":"2023-11-28T22:08:45.658240+0000","last_clean":"2023-11-28T22:08:45.658240+0000","last_became_active":"2023-11-28T22:08:43.317724+0000","last_became_peered":"2023-11-28T22:08:43.317724+0000","last_unstale":"2023-11-28T22:08:45.658240+0000","last_undegraded":"2023-11-28T22:08:45.658240+0000","last_fullsized":"2023-11-28T22:08:45.658240+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:12.910031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323970+0000","last_change":"2023-11-28T22:08:44.358763+0000","last_active":"2023-11-28T22:08:45.323970+0000","last_peered":"2023-11-28T22:08:45.323970+0000","last_clean":"2023-11-28T22:08:45.323970+0000","last_became_active":"2023-11-28T22:08:44.357079+0000","last_became_peered":"2023-11-28T22:08:44.357079+0000","last_unstale":"2023-11-28T22:08:45.323970+0000","last_undegraded":"2023-11-28T22:08:45.323970+0000","last_fullsized":"2023-11-28T22:08:45.323970+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:52.612283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657340+0000","last_change":"2023-11-28T22:08:43.314738+0000","last_active":"2023-11-28T22:08:45.657340+0000","last_peered":"2023-11-28T22:08:45.657340+0000","last_clean":"2023-11-28T22:08:45.657340+0000","last_became_active":"2023-11-28T22:08:43.313266+0000","last_became_peered":"2023-11-28T22:08:43.313266+0000","last_unstale":"2023-11-28T22:08:45.657340+0000","last_undegraded":"2023-11-28T22:08:45.657340+0000","last_fullsized":"2023-11-28T22:08:45.657340+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:47:14.899794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660725+0000","last_change":"2023-11-28T22:08:44.352654+0000","last_active":"2023-11-28T22:08:45.660725+0000","last_peered":"2023-11-28T22:08:45.660725+0000","last_clean":"2023-11-28T22:08:45.660725+0000","last_became_active":"2023-11-28T22:08:44.352364+0000","last_became_peered":"2023-11-28T22:08:44.352364+0000","last_unstale":"2023-11-28T22:08:45.660725+0000","last_undegraded":"2023-11-28T22:08:45.660725+0000","last_fullsized":"2023-11-28T22:08:45.660725+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:08:17.786769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658048+0000","last_change":"2023-11-28T22:08:43.323810+0000","last_active":"2023-11-28T22:08:45.658048+0000","last_peered":"2023-11-28T22:08:45.658048+0000","last_clean":"2023-11-28T22:08:45.658048+0000","last_became_active":"2023-11-28T22:08:43.323502+0000","last_became_peered":"2023-11-28T22:08:43.323502+0000","last_unstale":"2023-11-28T22:08:45.658048+0000","last_undegraded":"2023-11-28T22:08:45.658048+0000","last_fullsized":"2023-11-28T22:08:45.658048+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:32.402507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321313+0000","last_change":"2023-11-28T22:08:44.351275+0000","last_active":"2023-11-28T22:08:45.321313+0000","last_peered":"2023-11-28T22:08:45.321313+0000","last_clean":"2023-11-28T22:08:45.321313+0000","last_became_active":"2023-11-28T22:08:44.351017+0000","last_became_peered":"2023-11-28T22:08:44.351017+0000","last_unstale":"2023-11-28T22:08:45.321313+0000","last_undegraded":"2023-11-28T22:08:45.321313+0000","last_fullsized":"2023-11-28T22:08:45.321313+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:34:51.472372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658763+0000","last_change":"2023-11-28T22:08:43.323254+0000","last_active":"2023-11-28T22:08:45.658763+0000","last_peered":"2023-11-28T22:08:45.658763+0000","last_clean":"2023-11-28T22:08:45.658763+0000","last_became_active":"2023-11-28T22:08:43.322941+0000","last_became_peered":"2023-11-28T22:08:43.322941+0000","last_unstale":"2023-11-28T22:08:45.658763+0000","last_undegraded":"2023-11-28T22:08:45.658763+0000","last_fullsized":"2023-11-28T22:08:45.658763+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:10:42.253649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320508+0000","last_change":"2023-11-28T22:08:44.356477+0000","last_active":"2023-11-28T22:08:45.320508+0000","last_peered":"2023-11-28T22:08:45.320508+0000","last_clean":"2023-11-28T22:08:45.320508+0000","last_became_active":"2023-11-28T22:08:44.356189+0000","last_became_peered":"2023-11-28T22:08:44.356189+0000","last_unstale":"2023-11-28T22:08:45.320508+0000","last_undegraded":"2023-11-28T22:08:45.320508+0000","last_fullsized":"2023-11-28T22:08:45.320508+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:41:42.249628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328352+0000","last_change":"2023-11-28T22:08:43.325165+0000","last_active":"2023-11-28T22:08:45.328352+0000","last_peered":"2023-11-28T22:08:45.328352+0000","last_clean":"2023-11-28T22:08:45.328352+0000","last_became_active":"2023-11-28T22:08:43.323690+0000","last_became_peered":"2023-11-28T22:08:43.323690+0000","last_unstale":"2023-11-28T22:08:45.328352+0000","last_undegraded":"2023-11-28T22:08:45.328352+0000","last_fullsized":"2023-11-28T22:08:45.328352+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:11:27.455178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656534+0000","last_change":"2023-11-28T22:08:44.356291+0000","last_active":"2023-11-28T22:08:45.656534+0000","last_peered":"2023-11-28T22:08:45.656534+0000","last_clean":"2023-11-28T22:08:45.656534+0000","last_became_active":"2023-11-28T22:08:44.356022+0000","last_became_peered":"2023-11-28T22:08:44.356022+0000","last_unstale":"2023-11-28T22:08:45.656534+0000","last_undegraded":"2023-11-28T22:08:45.656534+0000","last_fullsized":"2023-11-28T22:08:45.656534+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:22.349395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656783+0000","last_change":"2023-11-28T22:08:43.322751+0000","last_active":"2023-11-28T22:08:45.656783+0000","last_peered":"2023-11-28T22:08:45.656783+0000","last_clean":"2023-11-28T22:08:45.656783+0000","last_became_active":"2023-11-28T22:08:43.322469+0000","last_became_peered":"2023-11-28T22:08:43.322469+0000","last_unstale":"2023-11-28T22:08:45.656783+0000","last_undegraded":"2023-11-28T22:08:45.656783+0000","last_fullsized":"2023-11-28T22:08:45.656783+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:09:20.278851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656687+0000","last_change":"2023-11-28T22:08:44.351594+0000","last_active":"2023-11-28T22:08:45.656687+0000","last_peered":"2023-11-28T22:08:45.656687+0000","last_clean":"2023-11-28T22:08:45.656687+0000","last_became_active":"2023-11-28T22:08:44.351295+0000","last_became_peered":"2023-11-28T22:08:44.351295+0000","last_unstale":"2023-11-28T22:08:45.656687+0000","last_undegraded":"2023-11-28T22:08:45.656687+0000","last_fullsized":"2023-11-28T22:08:45.656687+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:15.900529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655895+0000","last_change":"2023-11-28T22:08:43.314065+0000","last_active":"2023-11-28T22:08:45.655895+0000","last_peered":"2023-11-28T22:08:45.655895+0000","last_clean":"2023-11-28T22:08:45.655895+0000","last_became_active":"2023-11-28T22:08:43.313794+0000","last_became_peered":"2023-11-28T22:08:43.313794+0000","last_unstale":"2023-11-28T22:08:45.655895+0000","last_undegraded":"2023-11-28T22:08:45.655895+0000","last_fullsized":"2023-11-28T22:08:45.655895+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:30:54.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323661+0000","last_change":"2023-11-28T22:08:44.359633+0000","last_active":"2023-11-28T22:08:45.323661+0000","last_peered":"2023-11-28T22:08:45.323661+0000","last_clean":"2023-11-28T22:08:45.323661+0000","last_became_active":"2023-11-28T22:08:44.359354+0000","last_became_peered":"2023-11-28T22:08:44.359354+0000","last_unstale":"2023-11-28T22:08:45.323661+0000","last_undegraded":"2023-11-28T22:08:45.323661+0000","last_fullsized":"2023-11-28T22:08:45.323661+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:19:15.603608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660146+0000","last_change":"2023-11-28T22:08:43.322476+0000","last_active":"2023-11-28T22:08:45.660146+0000","last_peered":"2023-11-28T22:08:45.660146+0000","last_clean":"2023-11-28T22:08:45.660146+0000","last_became_active":"2023-11-28T22:08:43.322162+0000","last_became_peered":"2023-11-28T22:08:43.322162+0000","last_unstale":"2023-11-28T22:08:45.660146+0000","last_undegraded":"2023-11-28T22:08:45.660146+0000","last_fullsized":"2023-11-28T22:08:45.660146+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:39:21.240030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655924+0000","last_change":"2023-11-28T22:08:44.349297+0000","last_active":"2023-11-28T22:08:45.655924+0000","last_peered":"2023-11-28T22:08:45.655924+0000","last_clean":"2023-11-28T22:08:45.655924+0000","last_became_active":"2023-11-28T22:08:44.349006+0000","last_became_peered":"2023-11-28T22:08:44.349006+0000","last_unstale":"2023-11-28T22:08:45.655924+0000","last_undegraded":"2023-11-28T22:08:45.655924+0000","last_fullsized":"2023-11-28T22:08:45.655924+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:22.351455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657913+0000","last_change":"2023-11-28T22:08:43.322116+0000","last_active":"2023-11-28T22:08:45.657913+0000","last_peered":"2023-11-28T22:08:45.657913+0000","last_clean":"2023-11-28T22:08:45.657913+0000","last_became_active":"2023-11-28T22:08:43.321844+0000","last_became_peered":"2023-11-28T22:08:43.321844+0000","last_unstale":"2023-11-28T22:08:45.657913+0000","last_undegraded":"2023-11-28T22:08:45.657913+0000","last_fullsized":"2023-11-28T22:08:45.657913+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:40:43.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320853+0000","last_change":"2023-11-28T22:08:44.348881+0000","last_active":"2023-11-28T22:08:45.320853+0000","last_peered":"2023-11-28T22:08:45.320853+0000","last_clean":"2023-11-28T22:08:45.320853+0000","last_became_active":"2023-11-28T22:08:44.348144+0000","last_became_peered":"2023-11-28T22:08:44.348144+0000","last_unstale":"2023-11-28T22:08:45.320853+0000","last_undegraded":"2023-11-28T22:08:45.320853+0000","last_fullsized":"2023-11-28T22:08:45.320853+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:03:16.357248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655951+0000","last_change":"2023-11-28T22:08:43.313470+0000","last_active":"2023-11-28T22:08:45.655951+0000","last_peered":"2023-11-28T22:08:45.655951+0000","last_clean":"2023-11-28T22:08:45.655951+0000","last_became_active":"2023-11-28T22:08:43.313187+0000","last_became_peered":"2023-11-28T22:08:43.313187+0000","last_unstale":"2023-11-28T22:08:45.655951+0000","last_undegraded":"2023-11-28T22:08:45.655951+0000","last_fullsized":"2023-11-28T22:08:45.655951+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:24.779529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659677+0000","last_change":"2023-11-28T22:08:44.361687+0000","last_active":"2023-11-28T22:08:45.659677+0000","last_peered":"2023-11-28T22:08:45.659677+0000","last_clean":"2023-11-28T22:08:45.659677+0000","last_became_active":"2023-11-28T22:08:44.361110+0000","last_became_peered":"2023-11-28T22:08:44.361110+0000","last_unstale":"2023-11-28T22:08:45.659677+0000","last_undegraded":"2023-11-28T22:08:45.659677+0000","last_fullsized":"2023-11-28T22:08:45.659677+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:34:44.898544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324047+0000","last_change":"2023-11-28T22:08:43.321163+0000","last_active":"2023-11-28T22:08:45.324047+0000","last_peered":"2023-11-28T22:08:45.324047+0000","last_clean":"2023-11-28T22:08:45.324047+0000","last_became_active":"2023-11-28T22:08:43.320863+0000","last_became_peered":"2023-11-28T22:08:43.320863+0000","last_unstale":"2023-11-28T22:08:45.324047+0000","last_undegraded":"2023-11-28T22:08:45.324047+0000","last_fullsized":"2023-11-28T22:08:45.324047+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:21:48.835996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657011+0000","last_change":"2023-11-28T22:08:44.351869+0000","last_active":"2023-11-28T22:08:45.657011+0000","last_peered":"2023-11-28T22:08:45.657011+0000","last_clean":"2023-11-28T22:08:45.657011+0000","last_became_active":"2023-11-28T22:08:44.351607+0000","last_became_peered":"2023-11-28T22:08:44.351607+0000","last_unstale":"2023-11-28T22:08:45.657011+0000","last_undegraded":"2023-11-28T22:08:45.657011+0000","last_fullsized":"2023-11-28T22:08:45.657011+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:56:57.236240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323776+0000","last_change":"2023-11-28T22:08:43.324318+0000","last_active":"2023-11-28T22:08:45.323776+0000","last_peered":"2023-11-28T22:08:45.323776+0000","last_clean":"2023-11-28T22:08:45.323776+0000","last_became_active":"2023-11-28T22:08:43.324043+0000","last_became_peered":"2023-11-28T22:08:43.324043+0000","last_unstale":"2023-11-28T22:08:45.323776+0000","last_undegraded":"2023-11-28T22:08:45.323776+0000","last_fullsized":"2023-11-28T22:08:45.323776+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:05:31.778068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657395+0000","last_change":"2023-11-28T22:08:44.348238+0000","last_active":"2023-11-28T22:08:45.657395+0000","last_peered":"2023-11-28T22:08:45.657395+0000","last_clean":"2023-11-28T22:08:45.657395+0000","last_became_active":"2023-11-28T22:08:44.347958+0000","last_became_peered":"2023-11-28T22:08:44.347958+0000","last_unstale":"2023-11-28T22:08:45.657395+0000","last_undegraded":"2023-11-28T22:08:45.657395+0000","last_fullsized":"2023-11-28T22:08:45.657395+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:56.089290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657243+0000","last_change":"2023-11-28T22:08:43.324700+0000","last_active":"2023-11-28T22:08:45.657243+0000","last_peered":"2023-11-28T22:08:45.657243+0000","last_clean":"2023-11-28T22:08:45.657243+0000","last_became_active":"2023-11-28T22:08:43.324404+0000","last_became_peered":"2023-11-28T22:08:43.324404+0000","last_unstale":"2023-11-28T22:08:45.657243+0000","last_undegraded":"2023-11-28T22:08:45.657243+0000","last_fullsized":"2023-11-28T22:08:45.657243+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:40:43.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327669+0000","last_change":"2023-11-28T22:08:44.351073+0000","last_active":"2023-11-28T22:08:45.327669+0000","last_peered":"2023-11-28T22:08:45.327669+0000","last_clean":"2023-11-28T22:08:45.327669+0000","last_became_active":"2023-11-28T22:08:44.350680+0000","last_became_peered":"2023-11-28T22:08:44.350680+0000","last_unstale":"2023-11-28T22:08:45.327669+0000","last_undegraded":"2023-11-28T22:08:45.327669+0000","last_fullsized":"2023-11-28T22:08:45.327669+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:42:09.371217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656552+0000","last_change":"2023-11-28T22:08:43.326545+0000","last_active":"2023-11-28T22:08:45.656552+0000","last_peered":"2023-11-28T22:08:45.656552+0000","last_clean":"2023-11-28T22:08:45.656552+0000","last_became_active":"2023-11-28T22:08:43.326140+0000","last_became_peered":"2023-11-28T22:08:43.326140+0000","last_unstale":"2023-11-28T22:08:45.656552+0000","last_undegraded":"2023-11-28T22:08:45.656552+0000","last_fullsized":"2023-11-28T22:08:45.656552+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:35.471277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320802+0000","last_change":"2023-11-28T22:08:44.352885+0000","last_active":"2023-11-28T22:08:45.320802+0000","last_peered":"2023-11-28T22:08:45.320802+0000","last_clean":"2023-11-28T22:08:45.320802+0000","last_became_active":"2023-11-28T22:08:44.352393+0000","last_became_peered":"2023-11-28T22:08:44.352393+0000","last_unstale":"2023-11-28T22:08:45.320802+0000","last_undegraded":"2023-11-28T22:08:45.320802+0000","last_fullsized":"2023-11-28T22:08:45.320802+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:18:55.181191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655821+0000","last_change":"2023-11-28T22:08:43.317721+0000","last_active":"2023-11-28T22:08:45.655821+0000","last_peered":"2023-11-28T22:08:45.655821+0000","last_clean":"2023-11-28T22:08:45.655821+0000","last_became_active":"2023-11-28T22:08:43.317449+0000","last_became_peered":"2023-11-28T22:08:43.317449+0000","last_unstale":"2023-11-28T22:08:45.655821+0000","last_undegraded":"2023-11-28T22:08:45.655821+0000","last_fullsized":"2023-11-28T22:08:45.655821+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:19:15.001388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656989+0000","last_change":"2023-11-28T22:08:44.363832+0000","last_active":"2023-11-28T22:08:45.656989+0000","last_peered":"2023-11-28T22:08:45.656989+0000","last_clean":"2023-11-28T22:08:45.656989+0000","last_became_active":"2023-11-28T22:08:44.363530+0000","last_became_peered":"2023-11-28T22:08:44.363530+0000","last_unstale":"2023-11-28T22:08:45.656989+0000","last_undegraded":"2023-11-28T22:08:45.656989+0000","last_fullsized":"2023-11-28T22:08:45.656989+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:36.147972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660165+0000","last_change":"2023-11-28T22:08:43.322473+0000","last_active":"2023-11-28T22:08:45.660165+0000","last_peered":"2023-11-28T22:08:45.660165+0000","last_clean":"2023-11-28T22:08:45.660165+0000","last_became_active":"2023-11-28T22:08:43.321812+0000","last_became_peered":"2023-11-28T22:08:43.321812+0000","last_unstale":"2023-11-28T22:08:45.660165+0000","last_undegraded":"2023-11-28T22:08:45.660165+0000","last_fullsized":"2023-11-28T22:08:45.660165+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:47:17.834687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658888+0000","last_change":"2023-11-28T22:08:44.361774+0000","last_active":"2023-11-28T22:08:45.658888+0000","last_peered":"2023-11-28T22:08:45.658888+0000","last_clean":"2023-11-28T22:08:45.658888+0000","last_became_active":"2023-11-28T22:08:44.361437+0000","last_became_peered":"2023-11-28T22:08:44.361437+0000","last_unstale":"2023-11-28T22:08:45.658888+0000","last_undegraded":"2023-11-28T22:08:45.658888+0000","last_fullsized":"2023-11-28T22:08:45.658888+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:19.102529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655769+0000","last_change":"2023-11-28T22:08:43.317927+0000","last_active":"2023-11-28T22:08:45.655769+0000","last_peered":"2023-11-28T22:08:45.655769+0000","last_clean":"2023-11-28T22:08:45.655769+0000","last_became_active":"2023-11-28T22:08:43.316095+0000","last_became_peered":"2023-11-28T22:08:43.316095+0000","last_unstale":"2023-11-28T22:08:45.655769+0000","last_undegraded":"2023-11-28T22:08:45.655769+0000","last_fullsized":"2023-11-28T22:08:45.655769+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:35.471277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"204'4833","reported_seq":5694,"reported_epoch":214,"state":"active+clean","last_fresh":"2023-11-28T22:08:44.339674+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T22:08:44.339674+0000","last_peered":"2023-11-28T22:08:44.339674+0000","last_clean":"2023-11-28T22:08:44.339674+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T22:08:44.339674+0000","last_undegraded":"2023-11-28T22:08:44.339674+0000","last_fullsized":"2023-11-28T22:08:44.339674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4833,"log_dups_size":0,"ondisk_log_size":4833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2053,"num_read_kb":4245,"num_write":551,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2053,"num_read_kb":4245,"num_write":551,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4833,"ondisk_log_size":4833,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575976,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74644,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297196,"statfs":{"total":96636764160,"available":96560328704,"internally_reserved":0,"allocated":3276800,"data_stored":1966912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.228,"15min":1.218},"min":{"1min":0.98099999999999998,"5min":0.76300000000000001,"15min":0.57599999999999996},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9330000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.252,"15min":1.266},"min":{"1min":0.93999999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.6379999999999999,"5min":2.0190000000000001,"15min":2.6819999999999999},"last":0.82999999999999996}]},{"osd":1,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.282,"15min":1.2589999999999999},"min":{"1min":0.92800000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.7869999999999999,"5min":2.694,"15min":2.694},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.244,"15min":1.29},"min":{"1min":0.90400000000000003,"5min":0.84199999999999997,"15min":0.71099999999999997},"max":{"1min":1.756,"5min":1.756,"15min":3.0430000000000001},"last":1.21}]},{"osd":2,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.2969999999999999,"15min":1.3169999999999999},"min":{"1min":0.95599999999999996,"5min":0.94199999999999995,"15min":0.86599999999999999},"max":{"1min":1.671,"5min":1.6910000000000001,"15min":2.7480000000000002},"last":1.349},{"interface":"front","average":{"1min":1.373,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":1.0009999999999999,"5min":0.84899999999999998,"15min":0.79200000000000004},"max":{"1min":1.833,"5min":1.9019999999999999,"15min":2.2400000000000002},"last":0.70099999999999996}]},{"osd":3,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3440000000000001,"15min":1.351},"min":{"1min":0.97099999999999997,"5min":0.89100000000000001,"15min":0.82499999999999996},"max":{"1min":1.7250000000000001,"5min":1.7989999999999999,"15min":1.8740000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.585,"5min":1.48,"15min":1.4630000000000001},"min":{"1min":1.2050000000000001,"5min":1.1020000000000001,"15min":0.93300000000000005},"max":{"1min":2.258,"5min":2.258,"15min":2.9329999999999998},"last":1.484}]},{"osd":4,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3180000000000001,"15min":1.3340000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.81299999999999994},"max":{"1min":1.7889999999999999,"5min":1.9930000000000001,"15min":2.6930000000000001},"last":1.054},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.367,"15min":1.3740000000000001},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.823,"5min":1.905,"15min":2.371},"last":0.88500000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.4390000000000001,"15min":1.454},"min":{"1min":0.98899999999999999,"5min":0.871,"15min":0.77500000000000002},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":3.2229999999999999},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4990000000000001,"15min":1.5169999999999999},"min":{"1min":1.03,"5min":0.92100000000000004,"15min":0.90100000000000002},"max":{"1min":1.859,"5min":2.0760000000000001,"15min":2.508},"last":1.2909999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.599,"15min":1.6220000000000001},"min":{"1min":1.038,"5min":1.038,"15min":0.98999999999999999},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.6110000000000002},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.554,"15min":1.5349999999999999},"min":{"1min":1.2030000000000001,"5min":1.113,"15min":0.76800000000000002},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2869999999999999},"last":1.2549999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575976,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42136,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329704,"statfs":{"total":96636764160,"available":96593616896,"internally_reserved":0,"allocated":2699264,"data_stored":1383262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.135,"15min":1.125},"min":{"1min":0.78500000000000003,"5min":0.78000000000000003,"15min":0.68300000000000005},"max":{"1min":1.5409999999999999,"5min":1.6930000000000001,"15min":1.772},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1040000000000001,"15min":1.1379999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.4730000000000001,"5min":1.653,"15min":1.974},"last":1.113}]},{"osd":1,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.276,"15min":1.2989999999999999},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.71599999999999997},"max":{"1min":1.603,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4570000000000001,"15min":1.4419999999999999},"min":{"1min":0.999,"5min":0.89300000000000002,"15min":0.84199999999999997},"max":{"1min":1.659,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":0.98999999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.369,"15min":1.353},"min":{"1min":0.94799999999999995,"5min":0.85399999999999998,"15min":0.80000000000000004},"max":{"1min":1.6679999999999999,"5min":2.0779999999999998,"15min":2.4809999999999999},"last":1.304},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4139999999999999,"15min":1.4359999999999999},"min":{"1min":1.1990000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.835,"5min":2.1829999999999998,"15min":2.6269999999999998},"last":1.212}]},{"osd":3,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.3720000000000001,"15min":1.3540000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.88500000000000001},"max":{"1min":1.794,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.492},{"interface":"front","average":{"1min":1.331,"5min":1.3160000000000001,"15min":1.339},"min":{"1min":1.0249999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.629,"5min":1.7430000000000001,"15min":1.8859999999999999},"last":1.46}]},{"osd":4,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2729999999999999,"15min":1.2949999999999999},"min":{"1min":1.056,"5min":0.81599999999999995,"15min":0.79800000000000004},"max":{"1min":1.7310000000000001,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.071},{"interface":"front","average":{"1min":1.357,"5min":1.397,"15min":1.3779999999999999},"min":{"1min":1.034,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.875,"5min":2.02,"15min":2.02},"last":1.431}]},{"osd":5,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.421,"15min":1.3959999999999999},"min":{"1min":0.94999999999999996,"5min":0.86499999999999999,"15min":0.84199999999999997},"max":{"1min":1.7170000000000001,"5min":1.9990000000000001,"15min":2.081},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.573,"15min":1.5940000000000001},"min":{"1min":1.1719999999999999,"5min":1.077,"15min":1.0149999999999999},"max":{"1min":1.9059999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":1.3660000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.5449999999999999,"15min":1.538},"min":{"1min":1.127,"5min":1.0649999999999999,"15min":0.98899999999999999},"max":{"1min":1.9419999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.258},{"interface":"front","average":{"1min":1.456,"5min":1.4950000000000001,"15min":1.5249999999999999},"min":{"1min":1.006,"5min":1.0029999999999999,"15min":0.80000000000000004},"max":{"1min":1.8009999999999999,"5min":2.1150000000000002,"15min":2.2130000000000001},"last":1.532}]}]},{"osd":5,"up_from":13,"seq":55834575976,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60196,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311644,"statfs":{"total":96636764160,"available":96575123456,"internally_reserved":0,"allocated":2711552,"data_stored":1384826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.153,"15min":1.179},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.48499999999999999},"max":{"1min":1.9039999999999999,"5min":1.9039999999999999,"15min":1.9550000000000001},"last":1.153},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.216,"15min":1.2070000000000001},"min":{"1min":0.82599999999999996,"5min":0.79200000000000004,"15min":0.65700000000000003},"max":{"1min":1.6679999999999999,"5min":1.889,"15min":2.1709999999999998},"last":1.2669999999999999}]},{"osd":1,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.196,"15min":1.214},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6499999999999999,"5min":1.706,"15min":1.8979999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.292,"15min":1.2929999999999999},"min":{"1min":1.024,"5min":0.86199999999999999,"15min":0.54400000000000004},"max":{"1min":1.7350000000000001,"5min":1.7509999999999999,"15min":1.8859999999999999},"last":1.0620000000000001}]},{"osd":2,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3240000000000001,"15min":1.3400000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.60699999999999998},"max":{"1min":1.768,"5min":1.768,"15min":1.974},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.381,"15min":1.3660000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.76,"5min":2.0289999999999999,"15min":2.105},"last":1.492}]},{"osd":3,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.446,"15min":1.4179999999999999},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.68899999999999995},"max":{"1min":1.6850000000000001,"5min":1.988,"15min":1.988},"last":1.421},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3140000000000001,"15min":1.341},"min":{"1min":0.97699999999999998,"5min":0.94599999999999995,"15min":0.87},"max":{"1min":1.752,"5min":1.8779999999999999,"15min":1.9330000000000001},"last":1.1220000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.3799999999999999,"15min":1.377},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.72599999999999998},"max":{"1min":1.744,"5min":1.833,"15min":1.833},"last":1.212},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.46,"15min":1.492},"min":{"1min":0.86399999999999999,"5min":0.83999999999999997,"15min":0.76200000000000001},"max":{"1min":1.8089999999999999,"5min":1.95,"15min":2.0249999999999999},"last":1.333}]},{"osd":6,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.4390000000000001,"15min":1.4119999999999999},"min":{"1min":1.0209999999999999,"5min":0.84099999999999997,"15min":0.80000000000000004},"max":{"1min":1.833,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.518,"15min":1.5449999999999999},"min":{"1min":1.024,"5min":0.97199999999999998,"15min":0.94399999999999995},"max":{"1min":1.865,"5min":1.982,"15min":1.982},"last":1.476}]},{"osd":7,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4690000000000001,"15min":1.4830000000000001},"min":{"1min":1.262,"5min":1.0580000000000001,"15min":0.76100000000000001},"max":{"1min":1.7370000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.238},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.575,"15min":1.575},"min":{"1min":1.1679999999999999,"5min":0.877,"15min":0.877},"max":{"1min":1.9450000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.6200000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575977,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47720,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324120,"statfs":{"total":96636764160,"available":96587898880,"internally_reserved":0,"allocated":2711552,"data_stored":1384838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.099,"15min":1.1359999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.573,"5min":1.6839999999999999,"15min":1.8149999999999999},"last":1.133},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1699999999999999,"15min":1.163},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.742},"last":1.4970000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.367,"15min":1.3779999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.873},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.375,"15min":1.409},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.984,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.77}]},{"osd":2,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.258,"15min":1.268},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3300000000000001,"15min":1.351},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6519999999999999,"5min":1.915,"15min":1.944},"last":1.927}]},{"osd":3,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6579999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.016},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.403,"15min":1.4039999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.9099999999999999,"5min":1.929,"15min":2.036},"last":1.6899999999999999}]},{"osd":5,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.343,"15min":1.3300000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5740000000000001,"5min":2.0030000000000001,"15min":2.0609999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.355,"15min":1.3580000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7669999999999999,"5min":1.9139999999999999,"15min":2.1659999999999999},"last":1.738}]},{"osd":6,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.4470000000000001,"15min":1.4359999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.61499999999999999},"max":{"1min":1.8460000000000001,"5min":1.9630000000000001,"15min":2.097},"last":2.004},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.23},"last":2.1120000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4810000000000001,"15min":1.492},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.869,"5min":2.0190000000000001,"15min":2.3340000000000001},"last":1.8049999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5760000000000001,"15min":1.583},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.986,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":2.1659999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575977,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56032,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315808,"statfs":{"total":96636764160,"available":96579387392,"internally_reserved":0,"allocated":2703360,"data_stored":1383352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.97699999999999998,"15min":0.95599999999999996},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.58399999999999996},"max":{"1min":1.2949999999999999,"5min":1.786,"15min":2.0950000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.167,"5min":1.155,"15min":1.1339999999999999},"min":{"1min":0.83299999999999996,"5min":0.74399999999999999,"15min":0.625},"max":{"1min":1.835,"5min":1.835,"15min":2.3879999999999999},"last":0.60699999999999998}]},{"osd":1,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0449999999999999,"15min":1.034},"min":{"1min":0.78200000000000003,"5min":0.66700000000000004,"15min":0.56899999999999995},"max":{"1min":1.512,"5min":1.6299999999999999,"15min":1.698},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2649999999999999,"15min":1.262},"min":{"1min":0.89800000000000002,"5min":0.83499999999999996,"15min":0.66300000000000003},"max":{"1min":1.6479999999999999,"5min":1.827,"15min":2.6019999999999999},"last":0.995}]},{"osd":2,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.321,"15min":1.3160000000000001},"min":{"1min":0.95799999999999996,"5min":0.81999999999999995,"15min":0.58099999999999996},"max":{"1min":1.621,"5min":2.105,"15min":2.327},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.411,"15min":1.407},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.702,"5min":1.9390000000000001,"15min":2.218},"last":1.0589999999999999}]},{"osd":4,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.3839999999999999,"15min":1.3919999999999999},"min":{"1min":1.016,"5min":0.94499999999999995,"15min":0.82899999999999996},"max":{"1min":1.6040000000000001,"5min":1.948,"15min":2.181},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.488,"15min":1.4890000000000001},"min":{"1min":1.115,"5min":1.109,"15min":0.80000000000000004},"max":{"1min":1.736,"5min":2.0569999999999999,"15min":2.1949999999999998},"last":1.4099999999999999}]},{"osd":5,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.5389999999999999,"15min":1.5489999999999999},"min":{"1min":1.069,"5min":1.038,"15min":0.97699999999999998},"max":{"1min":1.9470000000000001,"5min":2.3109999999999999,"15min":2.4329999999999998},"last":1.111},{"interface":"front","average":{"1min":1.49,"5min":1.5109999999999999,"15min":1.508},"min":{"1min":1.151,"5min":1.0269999999999999,"15min":0.98999999999999999},"max":{"1min":1.984,"5min":1.984,"15min":2.2639999999999998},"last":1.224}]},{"osd":6,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.5880000000000001,"15min":1.575},"min":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":0.878},"max":{"1min":2.048,"5min":2.1230000000000002,"15min":2.331},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.681,"15min":1.6659999999999999},"min":{"1min":1.196,"5min":1.087,"15min":1.087},"max":{"1min":1.913,"5min":2.5070000000000001,"15min":2.5329999999999999},"last":1.1499999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.663,"15min":1.6339999999999999},"min":{"1min":1.0940000000000001,"5min":1.056,"15min":1.036},"max":{"1min":1.9890000000000001,"5min":2.0950000000000002,"15min":2.3380000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.657,"5min":1.7150000000000001,"15min":1.6990000000000001},"min":{"1min":1.3240000000000001,"5min":1.1699999999999999,"15min":0.95899999999999996},"max":{"1min":2.1800000000000002,"5min":2.1800000000000002,"15min":2.4670000000000001},"last":1.4710000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575977,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51872,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319968,"statfs":{"total":96636764160,"available":96583647232,"internally_reserved":0,"allocated":2699264,"data_stored":1383774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.97799999999999998,"15min":0.97599999999999998},"min":{"1min":0.71899999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.5589999999999999,"5min":1.6120000000000001,"15min":1.7190000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1419999999999999,"15min":1.157},"min":{"1min":0.81000000000000005,"5min":0.76200000000000001,"15min":0.49199999999999999},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9099999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.036,"15min":1.036},"min":{"1min":0.78900000000000003,"5min":0.76300000000000001,"15min":0.56000000000000005},"max":{"1min":2.097,"5min":2.097,"15min":2.097},"last":0.62},{"interface":"front","average":{"1min":1.175,"5min":1.232,"15min":1.2390000000000001},"min":{"1min":0.89800000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.6000000000000001,"5min":1.625,"15min":1.8149999999999999},"last":0.59599999999999997}]},{"osd":3,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2490000000000001,"15min":1.2729999999999999},"min":{"1min":0.95099999999999996,"5min":0.871,"15min":0.754},"max":{"1min":1.72,"5min":1.72,"15min":1.9330000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.365,"15min":1.3480000000000001},"min":{"1min":0.99099999999999999,"5min":0.92300000000000004,"15min":0.89700000000000002},"max":{"1min":1.849,"5min":1.849,"15min":2.2210000000000001},"last":0.55500000000000005}]},{"osd":4,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.48,"15min":1.4790000000000001},"min":{"1min":1.127,"5min":0.995,"15min":0.92300000000000004},"max":{"1min":1.956,"5min":1.956,"15min":2.0640000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.468,"15min":1.476},"min":{"1min":1.1419999999999999,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":1.8640000000000001,"5min":1.8779999999999999,"15min":2.173},"last":0.90300000000000002}]},{"osd":5,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.4970000000000001,"15min":1.5069999999999999},"min":{"1min":1.1890000000000001,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.893,"5min":2.222,"15min":2.3359999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5940000000000001,"15min":1.577},"min":{"1min":1.2150000000000001,"5min":1.107,"15min":0.86099999999999999},"max":{"1min":1.875,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.81100000000000005}]},{"osd":6,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.6140000000000001,"15min":1.615},"min":{"1min":1.2150000000000001,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.9710000000000001,"5min":2.387,"15min":2.387},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.669,"5min":1.6759999999999999,"15min":1.6659999999999999},"min":{"1min":1.2949999999999999,"5min":1.177,"15min":1.129},"max":{"1min":1.992,"5min":2.3940000000000001,"15min":2.4020000000000001},"last":1.075}]},{"osd":7,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.615,"15min":1.6240000000000001},"min":{"1min":1.1839999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.966,"5min":2.1899999999999999,"15min":2.4260000000000002},"last":1.153},{"interface":"front","average":{"1min":1.649,"5min":1.6879999999999999,"15min":1.716},"min":{"1min":1.194,"5min":1.0469999999999999,"15min":1.0469999999999999},"max":{"1min":2.04,"5min":2.2599999999999998,"15min":2.2759999999999998},"last":0.79200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575977,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51864,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319976,"statfs":{"total":96636764160,"available":96583655424,"internally_reserved":0,"allocated":2695168,"data_stored":1383192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.745,"5min":0.498,"15min":0.498},"max":{"1min":1.5449999999999999,"5min":1.669,"15min":1.8959999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1799999999999999,"15min":1.171},"min":{"1min":0.76500000000000001,"5min":0.629,"15min":0.629},"max":{"1min":1.9279999999999999,"5min":1.9410000000000001,"15min":2.3900000000000001},"last":1.3959999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.1879999999999999,"15min":1.1819999999999999},"min":{"1min":0.88200000000000001,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.984,"5min":1.984,"15min":2.1589999999999998},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2709999999999999,"15min":1.2829999999999999},"min":{"1min":0.85599999999999998,"5min":0.81499999999999995,"15min":0.77800000000000002},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.80000000000000004}]},{"osd":3,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.3999999999999999,"15min":1.4179999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.81699999999999995},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2320000000000002},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.218,"5min":1.1910000000000001,"15min":1.2110000000000001},"min":{"1min":0.81599999999999995,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7150000000000001,"5min":1.734,"15min":2.4359999999999999},"last":1.216}]},{"osd":4,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.419,"15min":1.4359999999999999},"min":{"1min":1.113,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":1.9510000000000001,"5min":2.2050000000000001,"15min":2.3559999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.431,"15min":1.46},"min":{"1min":1.0880000000000001,"5min":0.95099999999999996,"15min":0.83599999999999997},"max":{"1min":1.7569999999999999,"5min":2.2759999999999998,"15min":2.3079999999999998},"last":0.82899999999999996}]},{"osd":5,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.478,"15min":1.5069999999999999},"min":{"1min":1.1359999999999999,"5min":0.98899999999999999,"15min":0.77800000000000002},"max":{"1min":1.8300000000000001,"5min":2.0310000000000001,"15min":2.347},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.609,"15min":1.6180000000000001},"min":{"1min":1.248,"5min":1.121,"15min":0.95699999999999996},"max":{"1min":2.3420000000000001,"5min":2.3580000000000001,"15min":2.448},"last":1.2769999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5109999999999999,"15min":1.5289999999999999},"min":{"1min":1.218,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.875,"5min":2.1160000000000001,"15min":2.5249999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5760000000000001,"15min":1.603},"min":{"1min":1.2030000000000001,"5min":1.1319999999999999,"15min":1.093},"max":{"1min":2.0619999999999998,"5min":2.1389999999999998,"15min":2.4550000000000001},"last":1.252}]},{"osd":7,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.6779999999999999,"15min":1.706},"min":{"1min":1.4079999999999999,"5min":1.3620000000000001,"15min":0.83999999999999997},"max":{"1min":2.1150000000000002,"5min":2.5150000000000001,"15min":2.6110000000000002},"last":1.125},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.575,"15min":1.613},"min":{"1min":1.2609999999999999,"5min":1.056,"15min":1.056},"max":{"1min":2.1389999999999998,"5min":2.2730000000000001,"15min":2.8460000000000001},"last":1.4279999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575977,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52464,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319376,"statfs":{"total":96636764160,"available":96583041024,"internally_reserved":0,"allocated":3305472,"data_stored":1974288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.93200000000000005,"15min":0.92900000000000005},"min":{"1min":0.75800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4850000000000001,"5min":1.621,"15min":1.722},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1659999999999999,"15min":1.2},"min":{"1min":0.91300000000000003,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4610000000000001,"5min":2.0350000000000001,"15min":2.0569999999999999},"last":0.82599999999999996}]},{"osd":2,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1719999999999999,"15min":1.1870000000000001},"min":{"1min":0.90700000000000003,"5min":0.77600000000000002,"15min":0.76100000000000001},"max":{"1min":1.6279999999999999,"5min":1.835,"15min":2.1709999999999998},"last":1.159},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2669999999999999,"15min":1.252},"min":{"1min":0.872,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.71,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.085}]},{"osd":3,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.218,"15min":1.173},"min":{"1min":0.86799999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8009999999999999,"5min":1.9139999999999999,"15min":1.986},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.403,"15min":1.383},"min":{"1min":1.012,"5min":0.99399999999999999,"15min":0.89400000000000002},"max":{"1min":1.746,"5min":1.7949999999999999,"15min":2.4409999999999998},"last":1.369}]},{"osd":4,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.4850000000000001,"15min":1.48},"min":{"1min":1.077,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":1.98,"5min":1.98,"15min":2.1949999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.431,"15min":1.4330000000000001},"min":{"1min":1.04,"5min":1.04,"15min":0.96399999999999997},"max":{"1min":1.756,"5min":2.149,"15min":2.254},"last":1.1020000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.504,"15min":1.496},"min":{"1min":1.2050000000000001,"5min":0.997,"15min":0.997},"max":{"1min":2.0409999999999999,"5min":2.048,"15min":2.1520000000000001},"last":1.329},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5900000000000001,"15min":1.5640000000000001},"min":{"1min":1.284,"5min":1.1619999999999999,"15min":1.093},"max":{"1min":2.173,"5min":2.3050000000000002,"15min":2.3359999999999999},"last":1.6870000000000001}]},{"osd":6,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.657,"15min":1.633},"min":{"1min":1.357,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":2.0030000000000001,"5min":2.21,"15min":2.246},"last":1.754},{"interface":"front","average":{"1min":1.623,"5min":1.617,"15min":1.6180000000000001},"min":{"1min":1.1040000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":2.1200000000000001,"5min":2.54,"15min":2.54},"last":1.518}]},{"osd":7,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":1.6910000000000001,"15min":1.6870000000000001},"min":{"1min":1.427,"5min":1.0549999999999999,"15min":1.0549999999999999},"max":{"1min":2.206,"5min":2.2360000000000002,"15min":2.528},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.641,"15min":1.6339999999999999},"min":{"1min":1.167,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.2650000000000001,"5min":2.4089999999999998,"15min":2.4420000000000002},"last":1.407}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T22:08:48.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T22:08:49.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.126+0000 7f58ab59e640 1 Processor -- start 2023-11-28T22:08:49.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.127+0000 7f58ab59e640 1 -- start start 2023-11-28T22:08:49.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.128+0000 7f58ab59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f58ac072200 0x7f58ac0b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.128+0000 7f58ab59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58ac146610 0x7f58ac0b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58ab59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac146b60 0x7f58ac160e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58ab59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac0725e0 con 0x7f58ac146610 2023-11-28T22:08:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58ab59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac0b4310 con 0x7f58ac072200 2023-11-28T22:08:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58ab59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac06d8d0 con 0x7f58ac146b60 2023-11-28T22:08:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58a3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58ac146610 0x7f58ac0b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58a3fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58ac146610 0x7f58ac0b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59626/0 (socket says 172.21.15.45:59626) 2023-11-28T22:08:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58a3fff640 1 -- 172.21.15.45:0/1468065472 learned_addr learned my addr 172.21.15.45:0/1468065472 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:49.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58aa59c640 1 --2- 172.21.15.45:0/1468065472 >> v2:172.21.15.190:3300/0 conn(0x7f58ac072200 0x7f58ac0b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58aad9d640 1 --2- 172.21.15.45:0/1468065472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac146b60 0x7f58ac160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58a3fff640 1 -- 172.21.15.45:0/1468065472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac146b60 msgr2=0x7f58ac160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58a3fff640 1 --2- 172.21.15.45:0/1468065472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac146b60 0x7f58ac160e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58a3fff640 1 -- 172.21.15.45:0/1468065472 >> v2:172.21.15.190:3300/0 conn(0x7f58ac072200 msgr2=0x7f58ac0b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58a3fff640 1 --2- 172.21.15.45:0/1468065472 >> v2:172.21.15.190:3300/0 conn(0x7f58ac072200 0x7f58ac0b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.129+0000 7f58a3fff640 1 -- 172.21.15.45:0/1468065472 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac06ca20 con 0x7f58ac146610 2023-11-28T22:08:49.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.131+0000 7f58a3fff640 1 --2- 172.21.15.45:0/1468065472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146610 0x7f58ac0b22d0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0ae9d0 tx=0x7f589402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2668ab10a9cb8ef7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.131+0000 7f58a9d9b640 1 -- 172.21.15.45:0/1468065472 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894031870 con 0x7f58ac146610 2023-11-28T22:08:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.131+0000 7f58a9d9b640 1 -- 172.21.15.45:0/1468065472 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58940319d0 con 0x7f58ac146610 2023-11-28T22:08:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.131+0000 7f58a9d9b640 1 -- 172.21.15.45:0/1468065472 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894031d60 con 0x7f58ac146610 2023-11-28T22:08:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.131+0000 7f58ab59e640 1 -- 172.21.15.45:0/1468065472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146610 msgr2=0x7f58ac0b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.131+0000 7f58ab59e640 1 --2- 172.21.15.45:0/1468065472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146610 0x7f58ac0b22d0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0ae9d0 tx=0x7f589402b910 comp rx=0 tx=0).stop 2023-11-28T22:08:49.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.132+0000 7f58ab59e640 1 -- 172.21.15.45:0/1468065472 shutdown_connections 2023-11-28T22:08:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.132+0000 7f58ab59e640 1 --2- 172.21.15.45:0/1468065472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac146b60 0x7f58ac160e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.132+0000 7f58ab59e640 1 --2- 172.21.15.45:0/1468065472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146610 0x7f58ac0b22d0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.132+0000 7f58ab59e640 1 --2- 172.21.15.45:0/1468065472 >> v2:172.21.15.190:3300/0 conn(0x7f58ac072200 0x7f58ac0b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.132+0000 7f58ab59e640 1 -- 172.21.15.45:0/1468065472 >> 172.21.15.45:0/1468065472 conn(0x7f58ac06ab30 msgr2=0x7f58ac0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.132+0000 7f58ab59e640 1 -- 172.21.15.45:0/1468065472 shutdown_connections 2023-11-28T22:08:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.132+0000 7f58ab59e640 1 -- 172.21.15.45:0/1468065472 wait complete. 2023-11-28T22:08:49.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.133+0000 7f58ab59e640 1 Processor -- start 2023-11-28T22:08:49.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.134+0000 7f58ab59e640 1 -- start start 2023-11-28T22:08:49.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.134+0000 7f58ab59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac072200 0x7f58ac1400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.134+0000 7f58ab59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58ac146610 0x7f58ac13e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58ab59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146b60 0x7f58ac09bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58ab59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac13dcc0 con 0x7f58ac146b60 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58ab59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac13e220 con 0x7f58ac146610 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58ab59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac140610 con 0x7f58ac072200 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58aad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146b60 0x7f58ac09bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58aad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146b60 0x7f58ac09bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59632/0 (socket says 172.21.15.45:59632) 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58aad9d640 1 -- 172.21.15.45:0/2616404341 learned_addr learned my addr 172.21.15.45:0/2616404341 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58a3fff640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58ac146610 0x7f58ac13e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58aa59c640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac072200 0x7f58ac1400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58aad9d640 1 -- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac072200 msgr2=0x7f58ac1400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58aad9d640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac072200 0x7f58ac1400d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58aad9d640 1 -- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58ac146610 msgr2=0x7f58ac13e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58aad9d640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58ac146610 0x7f58ac13e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58aad9d640 1 -- 172.21.15.45:0/2616404341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58940094d0 con 0x7f58ac146b60 2023-11-28T22:08:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.135+0000 7f58a3fff640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58ac146610 0x7f58ac13e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:49.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.136+0000 7f58aad9d640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146b60 0x7f58ac09bf90 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f58a400cda0 tx=0x7f58a400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:49.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.136+0000 7f58a2ffd640 1 -- 172.21.15.45:0/2616404341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4003b90 con 0x7f58ac146b60 2023-11-28T22:08:49.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.136+0000 7f58a2ffd640 1 -- 172.21.15.45:0/2616404341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a4003cf0 con 0x7f58ac146b60 2023-11-28T22:08:49.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.136+0000 7f58a2ffd640 1 -- 172.21.15.45:0/2616404341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4004240 con 0x7f58ac146b60 2023-11-28T22:08:49.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.136+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac099c70 con 0x7f58ac146b60 2023-11-28T22:08:49.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.136+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac13deb0 con 0x7f58ac146b60 2023-11-28T22:08:49.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.138+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac09a190 con 0x7f58ac146b60 2023-11-28T22:08:49.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.138+0000 7f58a2ffd640 1 -- 172.21.15.45:0/2616404341 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f58a40043a0 con 0x7f58ac146b60 2023-11-28T22:08:49.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.139+0000 7f58a2ffd640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f588c06af90 0x7f588c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.139+0000 7f58a2ffd640 1 -- 172.21.15.45:0/2616404341 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f58a4012070 con 0x7f58ac146b60 2023-11-28T22:08:49.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.140+0000 7f58aa59c640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f588c06af90 0x7f588c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.141+0000 7f58aa59c640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f588c06af90 0x7f588c06d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f589c006fd0 tx=0x7f589c008040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:49.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.143+0000 7f58a2ffd640 1 -- 172.21.15.45:0/2616404341 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58ac09a190 con 0x7f58ac146b60 2023-11-28T22:08:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.339+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 --> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58ac06d280 con 0x7f588c06af90 2023-11-28T22:08:49.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.363+0000 7f58a2ffd640 1 -- 172.21.15.45:0/2616404341 <== mgr.7643 v2:172.21.15.190:6832/397715021 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286773 (secure 0 0 0) 0x7f589c00e470 con 0x7f588c06af90 2023-11-28T22:08:49.369 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:49.371 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T22:08:49.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f588c06af90 msgr2=0x7f588c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f588c06af90 0x7f588c06d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f589c006fd0 tx=0x7f589c008040 comp rx=0 tx=0).stop 2023-11-28T22:08:49.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146b60 msgr2=0x7f58ac09bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146b60 0x7f58ac09bf90 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f58a400cda0 tx=0x7f58a400a660 comp rx=0 tx=0).stop 2023-11-28T22:08:49.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 shutdown_connections 2023-11-28T22:08:49.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f588c06af90 0x7f588c06d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ac146b60 0x7f58ac09bf90 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58ac146610 0x7f58ac13e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 --2- 172.21.15.45:0/2616404341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58ac072200 0x7f58ac1400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 >> 172.21.15.45:0/2616404341 conn(0x7f58ac06ab30 msgr2=0x7f58ac160d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:49.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 shutdown_connections 2023-11-28T22:08:49.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.369+0000 7f58ab59e640 1 -- 172.21.15.45:0/2616404341 wait complete. 2023-11-28T22:08:49.394 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-28T22:08:47.579917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2053,"num_read_kb":4245,"num_write":595,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4871,"ondisk_log_size":4871,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436928,"kb_used_data":22268,"kb_used_omap":12,"kb_used_meta":414515,"kb_avail":754537792,"statfs":{"total":773094113280,"available":772646699008,"internally_reserved":0,"allocated":22802432,"data_stored":12244444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":424463920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.298035"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321246+0000","last_change":"2023-11-28T22:08:44.354200+0000","last_active":"2023-11-28T22:08:45.321246+0000","last_peered":"2023-11-28T22:08:45.321246+0000","last_clean":"2023-11-28T22:08:45.321246+0000","last_became_active":"2023-11-28T22:08:44.353849+0000","last_became_peered":"2023-11-28T22:08:44.353849+0000","last_unstale":"2023-11-28T22:08:45.321246+0000","last_undegraded":"2023-11-28T22:08:45.321246+0000","last_fullsized":"2023-11-28T22:08:45.321246+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:18:22.337269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320895+0000","last_change":"2023-11-28T22:08:44.344177+0000","last_active":"2023-11-28T22:08:45.320895+0000","last_peered":"2023-11-28T22:08:45.320895+0000","last_clean":"2023-11-28T22:08:45.320895+0000","last_became_active":"2023-11-28T22:08:44.343880+0000","last_became_peered":"2023-11-28T22:08:44.343880+0000","last_unstale":"2023-11-28T22:08:45.320895+0000","last_undegraded":"2023-11-28T22:08:45.320895+0000","last_fullsized":"2023-11-28T22:08:45.320895+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:32:01.808458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657888+0000","last_change":"2023-11-28T22:08:44.345774+0000","last_active":"2023-11-28T22:08:45.657888+0000","last_peered":"2023-11-28T22:08:45.657888+0000","last_clean":"2023-11-28T22:08:45.657888+0000","last_became_active":"2023-11-28T22:08:44.345409+0000","last_became_peered":"2023-11-28T22:08:44.345409+0000","last_unstale":"2023-11-28T22:08:45.657888+0000","last_undegraded":"2023-11-28T22:08:45.657888+0000","last_fullsized":"2023-11-28T22:08:45.657888+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:19:09.037444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657253+0000","last_change":"2023-11-28T22:08:44.347680+0000","last_active":"2023-11-28T22:08:45.657253+0000","last_peered":"2023-11-28T22:08:45.657253+0000","last_clean":"2023-11-28T22:08:45.657253+0000","last_became_active":"2023-11-28T22:08:44.347366+0000","last_became_peered":"2023-11-28T22:08:44.347366+0000","last_unstale":"2023-11-28T22:08:45.657253+0000","last_undegraded":"2023-11-28T22:08:45.657253+0000","last_fullsized":"2023-11-28T22:08:45.657253+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:30.647873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323554+0000","last_change":"2023-11-28T22:08:44.346142+0000","last_active":"2023-11-28T22:08:45.323554+0000","last_peered":"2023-11-28T22:08:45.323554+0000","last_clean":"2023-11-28T22:08:45.323554+0000","last_became_active":"2023-11-28T22:08:44.345821+0000","last_became_peered":"2023-11-28T22:08:44.345821+0000","last_unstale":"2023-11-28T22:08:45.323554+0000","last_undegraded":"2023-11-28T22:08:45.323554+0000","last_fullsized":"2023-11-28T22:08:45.323554+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:22:53.184945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323519+0000","last_change":"2023-11-28T22:08:44.364735+0000","last_active":"2023-11-28T22:08:45.323519+0000","last_peered":"2023-11-28T22:08:45.323519+0000","last_clean":"2023-11-28T22:08:45.323519+0000","last_became_active":"2023-11-28T22:08:44.364393+0000","last_became_peered":"2023-11-28T22:08:44.364393+0000","last_unstale":"2023-11-28T22:08:45.323519+0000","last_undegraded":"2023-11-28T22:08:45.323519+0000","last_fullsized":"2023-11-28T22:08:45.323519+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:18:18.813791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655708+0000","last_change":"2023-11-28T22:08:44.355251+0000","last_active":"2023-11-28T22:08:45.655708+0000","last_peered":"2023-11-28T22:08:45.655708+0000","last_clean":"2023-11-28T22:08:45.655708+0000","last_became_active":"2023-11-28T22:08:44.354974+0000","last_became_peered":"2023-11-28T22:08:44.354974+0000","last_unstale":"2023-11-28T22:08:45.655708+0000","last_undegraded":"2023-11-28T22:08:45.655708+0000","last_fullsized":"2023-11-28T22:08:45.655708+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:28.164258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656953+0000","last_change":"2023-11-28T22:08:44.355044+0000","last_active":"2023-11-28T22:08:45.656953+0000","last_peered":"2023-11-28T22:08:45.656953+0000","last_clean":"2023-11-28T22:08:45.656953+0000","last_became_active":"2023-11-28T22:08:44.354012+0000","last_became_peered":"2023-11-28T22:08:44.354012+0000","last_unstale":"2023-11-28T22:08:45.656953+0000","last_undegraded":"2023-11-28T22:08:45.656953+0000","last_fullsized":"2023-11-28T22:08:45.656953+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:53.107310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321159+0000","last_change":"2023-11-28T22:08:44.348375+0000","last_active":"2023-11-28T22:08:45.321159+0000","last_peered":"2023-11-28T22:08:45.321159+0000","last_clean":"2023-11-28T22:08:45.321159+0000","last_became_active":"2023-11-28T22:08:44.348037+0000","last_became_peered":"2023-11-28T22:08:44.348037+0000","last_unstale":"2023-11-28T22:08:45.321159+0000","last_undegraded":"2023-11-28T22:08:45.321159+0000","last_fullsized":"2023-11-28T22:08:45.321159+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:27.751749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324009+0000","last_change":"2023-11-28T22:08:44.345972+0000","last_active":"2023-11-28T22:08:45.324009+0000","last_peered":"2023-11-28T22:08:45.324009+0000","last_clean":"2023-11-28T22:08:45.324009+0000","last_became_active":"2023-11-28T22:08:44.345670+0000","last_became_peered":"2023-11-28T22:08:44.345670+0000","last_unstale":"2023-11-28T22:08:45.324009+0000","last_undegraded":"2023-11-28T22:08:45.324009+0000","last_fullsized":"2023-11-28T22:08:45.324009+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:41:11.921386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659320+0000","last_change":"2023-11-28T22:08:44.346368+0000","last_active":"2023-11-28T22:08:45.659320+0000","last_peered":"2023-11-28T22:08:45.659320+0000","last_clean":"2023-11-28T22:08:45.659320+0000","last_became_active":"2023-11-28T22:08:44.345928+0000","last_became_peered":"2023-11-28T22:08:44.345928+0000","last_unstale":"2023-11-28T22:08:45.659320+0000","last_undegraded":"2023-11-28T22:08:45.659320+0000","last_fullsized":"2023-11-28T22:08:45.659320+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:26:50.077471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657115+0000","last_change":"2023-11-28T22:08:44.356964+0000","last_active":"2023-11-28T22:08:45.657115+0000","last_peered":"2023-11-28T22:08:45.657115+0000","last_clean":"2023-11-28T22:08:45.657115+0000","last_became_active":"2023-11-28T22:08:44.355131+0000","last_became_peered":"2023-11-28T22:08:44.355131+0000","last_unstale":"2023-11-28T22:08:45.657115+0000","last_undegraded":"2023-11-28T22:08:45.657115+0000","last_fullsized":"2023-11-28T22:08:45.657115+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:44:53.107310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327442+0000","last_change":"2023-11-28T22:08:44.343174+0000","last_active":"2023-11-28T22:08:45.327442+0000","last_peered":"2023-11-28T22:08:45.327442+0000","last_clean":"2023-11-28T22:08:45.327442+0000","last_became_active":"2023-11-28T22:08:44.342846+0000","last_became_peered":"2023-11-28T22:08:44.342846+0000","last_unstale":"2023-11-28T22:08:45.327442+0000","last_undegraded":"2023-11-28T22:08:45.327442+0000","last_fullsized":"2023-11-28T22:08:45.327442+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:37:02.928782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655664+0000","last_change":"2023-11-28T22:08:44.347680+0000","last_active":"2023-11-28T22:08:45.655664+0000","last_peered":"2023-11-28T22:08:45.655664+0000","last_clean":"2023-11-28T22:08:45.655664+0000","last_became_active":"2023-11-28T22:08:44.347316+0000","last_became_peered":"2023-11-28T22:08:44.347316+0000","last_unstale":"2023-11-28T22:08:45.655664+0000","last_undegraded":"2023-11-28T22:08:45.655664+0000","last_fullsized":"2023-11-28T22:08:45.655664+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:49:11.197502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320642+0000","last_change":"2023-11-28T22:08:44.335536+0000","last_active":"2023-11-28T22:08:45.320642+0000","last_peered":"2023-11-28T22:08:45.320642+0000","last_clean":"2023-11-28T22:08:45.320642+0000","last_became_active":"2023-11-28T22:08:44.334171+0000","last_became_peered":"2023-11-28T22:08:44.334171+0000","last_unstale":"2023-11-28T22:08:45.320642+0000","last_undegraded":"2023-11-28T22:08:45.320642+0000","last_fullsized":"2023-11-28T22:08:45.320642+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:16:50.056281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323426+0000","last_change":"2023-11-28T22:08:44.353608+0000","last_active":"2023-11-28T22:08:45.323426+0000","last_peered":"2023-11-28T22:08:45.323426+0000","last_clean":"2023-11-28T22:08:45.323426+0000","last_became_active":"2023-11-28T22:08:44.348573+0000","last_became_peered":"2023-11-28T22:08:44.348573+0000","last_unstale":"2023-11-28T22:08:45.323426+0000","last_undegraded":"2023-11-28T22:08:45.323426+0000","last_fullsized":"2023-11-28T22:08:45.323426+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:50:51.787300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324378+0000","last_change":"2023-11-28T22:08:44.356606+0000","last_active":"2023-11-28T22:08:45.324378+0000","last_peered":"2023-11-28T22:08:45.324378+0000","last_clean":"2023-11-28T22:08:45.324378+0000","last_became_active":"2023-11-28T22:08:44.356302+0000","last_became_peered":"2023-11-28T22:08:44.356302+0000","last_unstale":"2023-11-28T22:08:45.324378+0000","last_undegraded":"2023-11-28T22:08:45.324378+0000","last_fullsized":"2023-11-28T22:08:45.324378+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:08:42.041519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328178+0000","last_change":"2023-11-28T22:08:44.355189+0000","last_active":"2023-11-28T22:08:45.328178+0000","last_peered":"2023-11-28T22:08:45.328178+0000","last_clean":"2023-11-28T22:08:45.328178+0000","last_became_active":"2023-11-28T22:08:44.354425+0000","last_became_peered":"2023-11-28T22:08:44.354425+0000","last_unstale":"2023-11-28T22:08:45.328178+0000","last_undegraded":"2023-11-28T22:08:45.328178+0000","last_fullsized":"2023-11-28T22:08:45.328178+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:41:57.181203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327885+0000","last_change":"2023-11-28T22:08:44.353039+0000","last_active":"2023-11-28T22:08:45.327885+0000","last_peered":"2023-11-28T22:08:45.327885+0000","last_clean":"2023-11-28T22:08:45.327885+0000","last_became_active":"2023-11-28T22:08:44.351399+0000","last_became_peered":"2023-11-28T22:08:44.351399+0000","last_unstale":"2023-11-28T22:08:45.327885+0000","last_undegraded":"2023-11-28T22:08:45.327885+0000","last_fullsized":"2023-11-28T22:08:45.327885+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:19:44.782055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323726+0000","last_change":"2023-11-28T22:08:44.356856+0000","last_active":"2023-11-28T22:08:45.323726+0000","last_peered":"2023-11-28T22:08:45.323726+0000","last_clean":"2023-11-28T22:08:45.323726+0000","last_became_active":"2023-11-28T22:08:44.356587+0000","last_became_peered":"2023-11-28T22:08:44.356587+0000","last_unstale":"2023-11-28T22:08:45.323726+0000","last_undegraded":"2023-11-28T22:08:45.323726+0000","last_fullsized":"2023-11-28T22:08:45.323726+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:32:19.991433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321209+0000","last_change":"2023-11-28T22:08:44.349209+0000","last_active":"2023-11-28T22:08:45.321209+0000","last_peered":"2023-11-28T22:08:45.321209+0000","last_clean":"2023-11-28T22:08:45.321209+0000","last_became_active":"2023-11-28T22:08:44.348049+0000","last_became_peered":"2023-11-28T22:08:44.348049+0000","last_unstale":"2023-11-28T22:08:45.321209+0000","last_undegraded":"2023-11-28T22:08:45.321209+0000","last_fullsized":"2023-11-28T22:08:45.321209+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:34.168754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320662+0000","last_change":"2023-11-28T22:08:44.352696+0000","last_active":"2023-11-28T22:08:45.320662+0000","last_peered":"2023-11-28T22:08:45.320662+0000","last_clean":"2023-11-28T22:08:45.320662+0000","last_became_active":"2023-11-28T22:08:44.348105+0000","last_became_peered":"2023-11-28T22:08:44.348105+0000","last_unstale":"2023-11-28T22:08:45.320662+0000","last_undegraded":"2023-11-28T22:08:45.320662+0000","last_fullsized":"2023-11-28T22:08:45.320662+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:38:05.221760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656951+0000","last_change":"2023-11-28T22:08:44.363713+0000","last_active":"2023-11-28T22:08:45.656951+0000","last_peered":"2023-11-28T22:08:45.656951+0000","last_clean":"2023-11-28T22:08:45.656951+0000","last_became_active":"2023-11-28T22:08:44.363425+0000","last_became_peered":"2023-11-28T22:08:44.363425+0000","last_unstale":"2023-11-28T22:08:45.656951+0000","last_undegraded":"2023-11-28T22:08:45.656951+0000","last_fullsized":"2023-11-28T22:08:45.656951+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:30.647873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320452+0000","last_change":"2023-11-28T22:08:44.354641+0000","last_active":"2023-11-28T22:08:45.320452+0000","last_peered":"2023-11-28T22:08:45.320452+0000","last_clean":"2023-11-28T22:08:45.320452+0000","last_became_active":"2023-11-28T22:08:44.354317+0000","last_became_peered":"2023-11-28T22:08:44.354317+0000","last_unstale":"2023-11-28T22:08:45.320452+0000","last_undegraded":"2023-11-28T22:08:45.320452+0000","last_fullsized":"2023-11-28T22:08:45.320452+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:39:41.245262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656571+0000","last_change":"2023-11-28T22:08:44.361169+0000","last_active":"2023-11-28T22:08:45.656571+0000","last_peered":"2023-11-28T22:08:45.656571+0000","last_clean":"2023-11-28T22:08:45.656571+0000","last_became_active":"2023-11-28T22:08:44.360790+0000","last_became_peered":"2023-11-28T22:08:44.360790+0000","last_unstale":"2023-11-28T22:08:45.656571+0000","last_undegraded":"2023-11-28T22:08:45.656571+0000","last_fullsized":"2023-11-28T22:08:45.656571+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:17.115567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660339+0000","last_change":"2023-11-28T22:08:44.349408+0000","last_active":"2023-11-28T22:08:45.660339+0000","last_peered":"2023-11-28T22:08:45.660339+0000","last_clean":"2023-11-28T22:08:45.660339+0000","last_became_active":"2023-11-28T22:08:44.348642+0000","last_became_peered":"2023-11-28T22:08:44.348642+0000","last_unstale":"2023-11-28T22:08:45.660339+0000","last_undegraded":"2023-11-28T22:08:45.660339+0000","last_fullsized":"2023-11-28T22:08:45.660339+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:22.784612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659606+0000","last_change":"2023-11-28T22:08:44.355649+0000","last_active":"2023-11-28T22:08:45.659606+0000","last_peered":"2023-11-28T22:08:45.659606+0000","last_clean":"2023-11-28T22:08:45.659606+0000","last_became_active":"2023-11-28T22:08:44.353617+0000","last_became_peered":"2023-11-28T22:08:44.353617+0000","last_unstale":"2023-11-28T22:08:45.659606+0000","last_undegraded":"2023-11-28T22:08:45.659606+0000","last_fullsized":"2023-11-28T22:08:45.659606+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:40:06.791697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323763+0000","last_change":"2023-11-28T22:08:44.347981+0000","last_active":"2023-11-28T22:08:45.323763+0000","last_peered":"2023-11-28T22:08:45.323763+0000","last_clean":"2023-11-28T22:08:45.323763+0000","last_became_active":"2023-11-28T22:08:44.346763+0000","last_became_peered":"2023-11-28T22:08:44.346763+0000","last_unstale":"2023-11-28T22:08:45.323763+0000","last_undegraded":"2023-11-28T22:08:45.323763+0000","last_fullsized":"2023-11-28T22:08:45.323763+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:05:02.093775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658425+0000","last_change":"2023-11-28T22:08:44.344267+0000","last_active":"2023-11-28T22:08:45.658425+0000","last_peered":"2023-11-28T22:08:45.658425+0000","last_clean":"2023-11-28T22:08:45.658425+0000","last_became_active":"2023-11-28T22:08:44.343864+0000","last_became_peered":"2023-11-28T22:08:44.343864+0000","last_unstale":"2023-11-28T22:08:45.658425+0000","last_undegraded":"2023-11-28T22:08:45.658425+0000","last_fullsized":"2023-11-28T22:08:45.658425+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:05:44.277097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657849+0000","last_change":"2023-11-28T22:08:44.366000+0000","last_active":"2023-11-28T22:08:45.657849+0000","last_peered":"2023-11-28T22:08:45.657849+0000","last_clean":"2023-11-28T22:08:45.657849+0000","last_became_active":"2023-11-28T22:08:44.365688+0000","last_became_peered":"2023-11-28T22:08:44.365688+0000","last_unstale":"2023-11-28T22:08:45.657849+0000","last_undegraded":"2023-11-28T22:08:45.657849+0000","last_fullsized":"2023-11-28T22:08:45.657849+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:50:01.770791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655353+0000","last_change":"2023-11-28T22:08:44.348577+0000","last_active":"2023-11-28T22:08:45.655353+0000","last_peered":"2023-11-28T22:08:45.655353+0000","last_clean":"2023-11-28T22:08:45.655353+0000","last_became_active":"2023-11-28T22:08:44.347988+0000","last_became_peered":"2023-11-28T22:08:44.347988+0000","last_unstale":"2023-11-28T22:08:45.655353+0000","last_undegraded":"2023-11-28T22:08:45.655353+0000","last_fullsized":"2023-11-28T22:08:45.655353+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:08:01.770933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323562+0000","last_change":"2023-11-28T22:08:44.349028+0000","last_active":"2023-11-28T22:08:45.323562+0000","last_peered":"2023-11-28T22:08:45.323562+0000","last_clean":"2023-11-28T22:08:45.323562+0000","last_became_active":"2023-11-28T22:08:44.348218+0000","last_became_peered":"2023-11-28T22:08:44.348218+0000","last_unstale":"2023-11-28T22:08:45.323562+0000","last_undegraded":"2023-11-28T22:08:45.323562+0000","last_fullsized":"2023-11-28T22:08:45.323562+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:47.357945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659158+0000","last_change":"2023-11-28T22:08:44.354508+0000","last_active":"2023-11-28T22:08:45.659158+0000","last_peered":"2023-11-28T22:08:45.659158+0000","last_clean":"2023-11-28T22:08:45.659158+0000","last_became_active":"2023-11-28T22:08:44.354220+0000","last_became_peered":"2023-11-28T22:08:44.354220+0000","last_unstale":"2023-11-28T22:08:45.659158+0000","last_undegraded":"2023-11-28T22:08:45.659158+0000","last_fullsized":"2023-11-28T22:08:45.659158+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:53:26.407432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"214'4","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657789+0000","last_change":"2023-11-28T22:08:43.320868+0000","last_active":"2023-11-28T22:08:45.657789+0000","last_peered":"2023-11-28T22:08:45.657789+0000","last_clean":"2023-11-28T22:08:45.657789+0000","last_became_active":"2023-11-28T22:08:43.320600+0000","last_became_peered":"2023-11-28T22:08:43.320600+0000","last_unstale":"2023-11-28T22:08:45.657789+0000","last_undegraded":"2023-11-28T22:08:45.657789+0000","last_fullsized":"2023-11-28T22:08:45.657789+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:18:59.336441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656509+0000","last_change":"2023-11-28T22:08:44.357609+0000","last_active":"2023-11-28T22:08:45.656509+0000","last_peered":"2023-11-28T22:08:45.656509+0000","last_clean":"2023-11-28T22:08:45.656509+0000","last_became_active":"2023-11-28T22:08:44.357345+0000","last_became_peered":"2023-11-28T22:08:44.357345+0000","last_unstale":"2023-11-28T22:08:45.656509+0000","last_undegraded":"2023-11-28T22:08:45.656509+0000","last_fullsized":"2023-11-28T22:08:45.656509+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:31:57.579829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324118+0000","last_change":"2023-11-28T22:08:43.313174+0000","last_active":"2023-11-28T22:08:45.324118+0000","last_peered":"2023-11-28T22:08:45.324118+0000","last_clean":"2023-11-28T22:08:45.324118+0000","last_became_active":"2023-11-28T22:08:43.312907+0000","last_became_peered":"2023-11-28T22:08:43.312907+0000","last_unstale":"2023-11-28T22:08:45.324118+0000","last_undegraded":"2023-11-28T22:08:45.324118+0000","last_fullsized":"2023-11-28T22:08:45.324118+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:42:51.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660528+0000","last_change":"2023-11-28T22:08:44.355843+0000","last_active":"2023-11-28T22:08:45.660528+0000","last_peered":"2023-11-28T22:08:45.660528+0000","last_clean":"2023-11-28T22:08:45.660528+0000","last_became_active":"2023-11-28T22:08:44.355577+0000","last_became_peered":"2023-11-28T22:08:44.355577+0000","last_unstale":"2023-11-28T22:08:45.660528+0000","last_undegraded":"2023-11-28T22:08:45.660528+0000","last_fullsized":"2023-11-28T22:08:45.660528+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:30:20.094815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659565+0000","last_change":"2023-11-28T22:08:43.314834+0000","last_active":"2023-11-28T22:08:45.659565+0000","last_peered":"2023-11-28T22:08:45.659565+0000","last_clean":"2023-11-28T22:08:45.659565+0000","last_became_active":"2023-11-28T22:08:43.313415+0000","last_became_peered":"2023-11-28T22:08:43.313415+0000","last_unstale":"2023-11-28T22:08:45.659565+0000","last_undegraded":"2023-11-28T22:08:45.659565+0000","last_fullsized":"2023-11-28T22:08:45.659565+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:53:52.151763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657855+0000","last_change":"2023-11-28T22:08:44.346534+0000","last_active":"2023-11-28T22:08:45.657855+0000","last_peered":"2023-11-28T22:08:45.657855+0000","last_clean":"2023-11-28T22:08:45.657855+0000","last_became_active":"2023-11-28T22:08:44.346153+0000","last_became_peered":"2023-11-28T22:08:44.346153+0000","last_unstale":"2023-11-28T22:08:45.657855+0000","last_undegraded":"2023-11-28T22:08:45.657855+0000","last_fullsized":"2023-11-28T22:08:45.657855+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:35:32.804621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328210+0000","last_change":"2023-11-28T22:08:43.319845+0000","last_active":"2023-11-28T22:08:45.328210+0000","last_peered":"2023-11-28T22:08:45.328210+0000","last_clean":"2023-11-28T22:08:45.328210+0000","last_became_active":"2023-11-28T22:08:43.319549+0000","last_became_peered":"2023-11-28T22:08:43.319549+0000","last_unstale":"2023-11-28T22:08:45.328210+0000","last_undegraded":"2023-11-28T22:08:45.328210+0000","last_fullsized":"2023-11-28T22:08:45.328210+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:06.730371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327023+0000","last_change":"2023-11-28T22:08:44.361496+0000","last_active":"2023-11-28T22:08:45.327023+0000","last_peered":"2023-11-28T22:08:45.327023+0000","last_clean":"2023-11-28T22:08:45.327023+0000","last_became_active":"2023-11-28T22:08:44.361177+0000","last_became_peered":"2023-11-28T22:08:44.361177+0000","last_unstale":"2023-11-28T22:08:45.327023+0000","last_undegraded":"2023-11-28T22:08:45.327023+0000","last_fullsized":"2023-11-28T22:08:45.327023+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:59.637363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323660+0000","last_change":"2023-11-28T22:08:43.324359+0000","last_active":"2023-11-28T22:08:45.323660+0000","last_peered":"2023-11-28T22:08:45.323660+0000","last_clean":"2023-11-28T22:08:45.323660+0000","last_became_active":"2023-11-28T22:08:43.323933+0000","last_became_peered":"2023-11-28T22:08:43.323933+0000","last_unstale":"2023-11-28T22:08:45.323660+0000","last_undegraded":"2023-11-28T22:08:45.323660+0000","last_fullsized":"2023-11-28T22:08:45.323660+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:40.625533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320810+0000","last_change":"2023-11-28T22:08:44.345155+0000","last_active":"2023-11-28T22:08:45.320810+0000","last_peered":"2023-11-28T22:08:45.320810+0000","last_clean":"2023-11-28T22:08:45.320810+0000","last_became_active":"2023-11-28T22:08:44.344846+0000","last_became_peered":"2023-11-28T22:08:44.344846+0000","last_unstale":"2023-11-28T22:08:45.320810+0000","last_undegraded":"2023-11-28T22:08:45.320810+0000","last_fullsized":"2023-11-28T22:08:45.320810+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:09:55.289797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656876+0000","last_change":"2023-11-28T22:08:43.327207+0000","last_active":"2023-11-28T22:08:45.656876+0000","last_peered":"2023-11-28T22:08:45.656876+0000","last_clean":"2023-11-28T22:08:45.656876+0000","last_became_active":"2023-11-28T22:08:43.326921+0000","last_became_peered":"2023-11-28T22:08:43.326921+0000","last_unstale":"2023-11-28T22:08:45.656876+0000","last_undegraded":"2023-11-28T22:08:45.656876+0000","last_fullsized":"2023-11-28T22:08:45.656876+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:19:35.409181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320745+0000","last_change":"2023-11-28T22:08:44.352117+0000","last_active":"2023-11-28T22:08:45.320745+0000","last_peered":"2023-11-28T22:08:45.320745+0000","last_clean":"2023-11-28T22:08:45.320745+0000","last_became_active":"2023-11-28T22:08:44.343825+0000","last_became_peered":"2023-11-28T22:08:44.343825+0000","last_unstale":"2023-11-28T22:08:45.320745+0000","last_undegraded":"2023-11-28T22:08:45.320745+0000","last_fullsized":"2023-11-28T22:08:45.320745+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:52:29.867964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327928+0000","last_change":"2023-11-28T22:08:43.320287+0000","last_active":"2023-11-28T22:08:45.327928+0000","last_peered":"2023-11-28T22:08:45.327928+0000","last_clean":"2023-11-28T22:08:45.327928+0000","last_became_active":"2023-11-28T22:08:43.318097+0000","last_became_peered":"2023-11-28T22:08:43.318097+0000","last_unstale":"2023-11-28T22:08:45.327928+0000","last_undegraded":"2023-11-28T22:08:45.327928+0000","last_fullsized":"2023-11-28T22:08:45.327928+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:54:37.447100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659190+0000","last_change":"2023-11-28T22:08:44.356005+0000","last_active":"2023-11-28T22:08:45.659190+0000","last_peered":"2023-11-28T22:08:45.659190+0000","last_clean":"2023-11-28T22:08:45.659190+0000","last_became_active":"2023-11-28T22:08:44.355694+0000","last_became_peered":"2023-11-28T22:08:44.355694+0000","last_unstale":"2023-11-28T22:08:45.659190+0000","last_undegraded":"2023-11-28T22:08:45.659190+0000","last_fullsized":"2023-11-28T22:08:45.659190+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:48:40.533350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324274+0000","last_change":"2023-11-28T22:08:43.312754+0000","last_active":"2023-11-28T22:08:45.324274+0000","last_peered":"2023-11-28T22:08:45.324274+0000","last_clean":"2023-11-28T22:08:45.324274+0000","last_became_active":"2023-11-28T22:08:43.310889+0000","last_became_peered":"2023-11-28T22:08:43.310889+0000","last_unstale":"2023-11-28T22:08:45.324274+0000","last_undegraded":"2023-11-28T22:08:45.324274+0000","last_fullsized":"2023-11-28T22:08:45.324274+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:44:57.568897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323945+0000","last_change":"2023-11-28T22:08:44.358994+0000","last_active":"2023-11-28T22:08:45.323945+0000","last_peered":"2023-11-28T22:08:45.323945+0000","last_clean":"2023-11-28T22:08:45.323945+0000","last_became_active":"2023-11-28T22:08:44.354736+0000","last_became_peered":"2023-11-28T22:08:44.354736+0000","last_unstale":"2023-11-28T22:08:45.323945+0000","last_undegraded":"2023-11-28T22:08:45.323945+0000","last_fullsized":"2023-11-28T22:08:45.323945+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:13:54.673324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321107+0000","last_change":"2023-11-28T22:08:43.316292+0000","last_active":"2023-11-28T22:08:45.321107+0000","last_peered":"2023-11-28T22:08:45.321107+0000","last_clean":"2023-11-28T22:08:45.321107+0000","last_became_active":"2023-11-28T22:08:43.316037+0000","last_became_peered":"2023-11-28T22:08:43.316037+0000","last_unstale":"2023-11-28T22:08:45.321107+0000","last_undegraded":"2023-11-28T22:08:45.321107+0000","last_fullsized":"2023-11-28T22:08:45.321107+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:43:03.567062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328079+0000","last_change":"2023-11-28T22:08:44.343326+0000","last_active":"2023-11-28T22:08:45.328079+0000","last_peered":"2023-11-28T22:08:45.328079+0000","last_clean":"2023-11-28T22:08:45.328079+0000","last_became_active":"2023-11-28T22:08:44.342964+0000","last_became_peered":"2023-11-28T22:08:44.342964+0000","last_unstale":"2023-11-28T22:08:45.328079+0000","last_undegraded":"2023-11-28T22:08:45.328079+0000","last_fullsized":"2023-11-28T22:08:45.328079+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:13:14.601881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"214'3","reported_seq":15,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657385+0000","last_change":"2023-11-28T22:08:43.326251+0000","last_active":"2023-11-28T22:08:45.657385+0000","last_peered":"2023-11-28T22:08:45.657385+0000","last_clean":"2023-11-28T22:08:45.657385+0000","last_became_active":"2023-11-28T22:08:43.325985+0000","last_became_peered":"2023-11-28T22:08:43.325985+0000","last_unstale":"2023-11-28T22:08:45.657385+0000","last_undegraded":"2023-11-28T22:08:45.657385+0000","last_fullsized":"2023-11-28T22:08:45.657385+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:09:39.056123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321347+0000","last_change":"2023-11-28T22:08:44.346762+0000","last_active":"2023-11-28T22:08:45.321347+0000","last_peered":"2023-11-28T22:08:45.321347+0000","last_clean":"2023-11-28T22:08:45.321347+0000","last_became_active":"2023-11-28T22:08:44.346480+0000","last_became_peered":"2023-11-28T22:08:44.346480+0000","last_unstale":"2023-11-28T22:08:45.321347+0000","last_undegraded":"2023-11-28T22:08:45.321347+0000","last_fullsized":"2023-11-28T22:08:45.321347+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:48:56.037544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"214'6","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328484+0000","last_change":"2023-11-28T22:08:43.314229+0000","last_active":"2023-11-28T22:08:45.328484+0000","last_peered":"2023-11-28T22:08:45.328484+0000","last_clean":"2023-11-28T22:08:45.328484+0000","last_became_active":"2023-11-28T22:08:43.313936+0000","last_became_peered":"2023-11-28T22:08:43.313936+0000","last_unstale":"2023-11-28T22:08:45.328484+0000","last_undegraded":"2023-11-28T22:08:45.328484+0000","last_fullsized":"2023-11-28T22:08:45.328484+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:17:21.971727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655041+0000","last_change":"2023-11-28T22:08:44.355681+0000","last_active":"2023-11-28T22:08:45.655041+0000","last_peered":"2023-11-28T22:08:45.655041+0000","last_clean":"2023-11-28T22:08:45.655041+0000","last_became_active":"2023-11-28T22:08:44.355415+0000","last_became_peered":"2023-11-28T22:08:44.355415+0000","last_unstale":"2023-11-28T22:08:45.655041+0000","last_undegraded":"2023-11-28T22:08:45.655041+0000","last_fullsized":"2023-11-28T22:08:45.655041+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:51:35.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328372+0000","last_change":"2023-11-28T22:08:43.317099+0000","last_active":"2023-11-28T22:08:45.328372+0000","last_peered":"2023-11-28T22:08:45.328372+0000","last_clean":"2023-11-28T22:08:45.328372+0000","last_became_active":"2023-11-28T22:08:43.316674+0000","last_became_peered":"2023-11-28T22:08:43.316674+0000","last_unstale":"2023-11-28T22:08:45.328372+0000","last_undegraded":"2023-11-28T22:08:45.328372+0000","last_fullsized":"2023-11-28T22:08:45.328372+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:02:19.708415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323411+0000","last_change":"2023-11-28T22:08:44.350936+0000","last_active":"2023-11-28T22:08:45.323411+0000","last_peered":"2023-11-28T22:08:45.323411+0000","last_clean":"2023-11-28T22:08:45.323411+0000","last_became_active":"2023-11-28T22:08:44.350659+0000","last_became_peered":"2023-11-28T22:08:44.350659+0000","last_unstale":"2023-11-28T22:08:45.323411+0000","last_undegraded":"2023-11-28T22:08:45.323411+0000","last_fullsized":"2023-11-28T22:08:45.323411+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:43:30.287811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320832+0000","last_change":"2023-11-28T22:08:43.328483+0000","last_active":"2023-11-28T22:08:45.320832+0000","last_peered":"2023-11-28T22:08:45.320832+0000","last_clean":"2023-11-28T22:08:45.320832+0000","last_became_active":"2023-11-28T22:08:43.328124+0000","last_became_peered":"2023-11-28T22:08:43.328124+0000","last_unstale":"2023-11-28T22:08:45.320832+0000","last_undegraded":"2023-11-28T22:08:45.320832+0000","last_fullsized":"2023-11-28T22:08:45.320832+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:52.662588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327868+0000","last_change":"2023-11-28T22:08:44.352432+0000","last_active":"2023-11-28T22:08:45.327868+0000","last_peered":"2023-11-28T22:08:45.327868+0000","last_clean":"2023-11-28T22:08:45.327868+0000","last_became_active":"2023-11-28T22:08:44.352169+0000","last_became_peered":"2023-11-28T22:08:44.352169+0000","last_unstale":"2023-11-28T22:08:45.327868+0000","last_undegraded":"2023-11-28T22:08:45.327868+0000","last_fullsized":"2023-11-28T22:08:45.327868+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:08:42.150626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656776+0000","last_change":"2023-11-28T22:08:43.312914+0000","last_active":"2023-11-28T22:08:45.656776+0000","last_peered":"2023-11-28T22:08:45.656776+0000","last_clean":"2023-11-28T22:08:45.656776+0000","last_became_active":"2023-11-28T22:08:43.312634+0000","last_became_peered":"2023-11-28T22:08:43.312634+0000","last_unstale":"2023-11-28T22:08:45.656776+0000","last_undegraded":"2023-11-28T22:08:45.656776+0000","last_fullsized":"2023-11-28T22:08:45.656776+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:29:26.298257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327887+0000","last_change":"2023-11-28T22:08:44.346948+0000","last_active":"2023-11-28T22:08:45.327887+0000","last_peered":"2023-11-28T22:08:45.327887+0000","last_clean":"2023-11-28T22:08:45.327887+0000","last_became_active":"2023-11-28T22:08:44.346649+0000","last_became_peered":"2023-11-28T22:08:44.346649+0000","last_unstale":"2023-11-28T22:08:45.327887+0000","last_undegraded":"2023-11-28T22:08:45.327887+0000","last_fullsized":"2023-11-28T22:08:45.327887+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:18:25.673593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"214'8","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657438+0000","last_change":"2023-11-28T22:08:43.315699+0000","last_active":"2023-11-28T22:08:45.657438+0000","last_peered":"2023-11-28T22:08:45.657438+0000","last_clean":"2023-11-28T22:08:45.657438+0000","last_became_active":"2023-11-28T22:08:43.315402+0000","last_became_peered":"2023-11-28T22:08:43.315402+0000","last_unstale":"2023-11-28T22:08:45.657438+0000","last_undegraded":"2023-11-28T22:08:45.657438+0000","last_fullsized":"2023-11-28T22:08:45.657438+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:36:57.829619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655569+0000","last_change":"2023-11-28T22:08:44.361918+0000","last_active":"2023-11-28T22:08:45.655569+0000","last_peered":"2023-11-28T22:08:45.655569+0000","last_clean":"2023-11-28T22:08:45.655569+0000","last_became_active":"2023-11-28T22:08:44.353476+0000","last_became_peered":"2023-11-28T22:08:44.353476+0000","last_unstale":"2023-11-28T22:08:45.655569+0000","last_undegraded":"2023-11-28T22:08:45.655569+0000","last_fullsized":"2023-11-28T22:08:45.655569+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:18:32.123119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658240+0000","last_change":"2023-11-28T22:08:43.318071+0000","last_active":"2023-11-28T22:08:45.658240+0000","last_peered":"2023-11-28T22:08:45.658240+0000","last_clean":"2023-11-28T22:08:45.658240+0000","last_became_active":"2023-11-28T22:08:43.317724+0000","last_became_peered":"2023-11-28T22:08:43.317724+0000","last_unstale":"2023-11-28T22:08:45.658240+0000","last_undegraded":"2023-11-28T22:08:45.658240+0000","last_fullsized":"2023-11-28T22:08:45.658240+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:28:12.910031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323970+0000","last_change":"2023-11-28T22:08:44.358763+0000","last_active":"2023-11-28T22:08:45.323970+0000","last_peered":"2023-11-28T22:08:45.323970+0000","last_clean":"2023-11-28T22:08:45.323970+0000","last_became_active":"2023-11-28T22:08:44.357079+0000","last_became_peered":"2023-11-28T22:08:44.357079+0000","last_unstale":"2023-11-28T22:08:45.323970+0000","last_undegraded":"2023-11-28T22:08:45.323970+0000","last_fullsized":"2023-11-28T22:08:45.323970+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:52.612283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657340+0000","last_change":"2023-11-28T22:08:43.314738+0000","last_active":"2023-11-28T22:08:45.657340+0000","last_peered":"2023-11-28T22:08:45.657340+0000","last_clean":"2023-11-28T22:08:45.657340+0000","last_became_active":"2023-11-28T22:08:43.313266+0000","last_became_peered":"2023-11-28T22:08:43.313266+0000","last_unstale":"2023-11-28T22:08:45.657340+0000","last_undegraded":"2023-11-28T22:08:45.657340+0000","last_fullsized":"2023-11-28T22:08:45.657340+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:47:14.899794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660725+0000","last_change":"2023-11-28T22:08:44.352654+0000","last_active":"2023-11-28T22:08:45.660725+0000","last_peered":"2023-11-28T22:08:45.660725+0000","last_clean":"2023-11-28T22:08:45.660725+0000","last_became_active":"2023-11-28T22:08:44.352364+0000","last_became_peered":"2023-11-28T22:08:44.352364+0000","last_unstale":"2023-11-28T22:08:45.660725+0000","last_undegraded":"2023-11-28T22:08:45.660725+0000","last_fullsized":"2023-11-28T22:08:45.660725+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:08:17.786769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"214'1","reported_seq":13,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658048+0000","last_change":"2023-11-28T22:08:43.323810+0000","last_active":"2023-11-28T22:08:45.658048+0000","last_peered":"2023-11-28T22:08:45.658048+0000","last_clean":"2023-11-28T22:08:45.658048+0000","last_became_active":"2023-11-28T22:08:43.323502+0000","last_became_peered":"2023-11-28T22:08:43.323502+0000","last_unstale":"2023-11-28T22:08:45.658048+0000","last_undegraded":"2023-11-28T22:08:45.658048+0000","last_fullsized":"2023-11-28T22:08:45.658048+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:32.402507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.321313+0000","last_change":"2023-11-28T22:08:44.351275+0000","last_active":"2023-11-28T22:08:45.321313+0000","last_peered":"2023-11-28T22:08:45.321313+0000","last_clean":"2023-11-28T22:08:45.321313+0000","last_became_active":"2023-11-28T22:08:44.351017+0000","last_became_peered":"2023-11-28T22:08:44.351017+0000","last_unstale":"2023-11-28T22:08:45.321313+0000","last_undegraded":"2023-11-28T22:08:45.321313+0000","last_fullsized":"2023-11-28T22:08:45.321313+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:34:51.472372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658763+0000","last_change":"2023-11-28T22:08:43.323254+0000","last_active":"2023-11-28T22:08:45.658763+0000","last_peered":"2023-11-28T22:08:45.658763+0000","last_clean":"2023-11-28T22:08:45.658763+0000","last_became_active":"2023-11-28T22:08:43.322941+0000","last_became_peered":"2023-11-28T22:08:43.322941+0000","last_unstale":"2023-11-28T22:08:45.658763+0000","last_undegraded":"2023-11-28T22:08:45.658763+0000","last_fullsized":"2023-11-28T22:08:45.658763+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:10:42.253649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320508+0000","last_change":"2023-11-28T22:08:44.356477+0000","last_active":"2023-11-28T22:08:45.320508+0000","last_peered":"2023-11-28T22:08:45.320508+0000","last_clean":"2023-11-28T22:08:45.320508+0000","last_became_active":"2023-11-28T22:08:44.356189+0000","last_became_peered":"2023-11-28T22:08:44.356189+0000","last_unstale":"2023-11-28T22:08:45.320508+0000","last_undegraded":"2023-11-28T22:08:45.320508+0000","last_fullsized":"2023-11-28T22:08:45.320508+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T07:41:42.249628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.328352+0000","last_change":"2023-11-28T22:08:43.325165+0000","last_active":"2023-11-28T22:08:45.328352+0000","last_peered":"2023-11-28T22:08:45.328352+0000","last_clean":"2023-11-28T22:08:45.328352+0000","last_became_active":"2023-11-28T22:08:43.323690+0000","last_became_peered":"2023-11-28T22:08:43.323690+0000","last_unstale":"2023-11-28T22:08:45.328352+0000","last_undegraded":"2023-11-28T22:08:45.328352+0000","last_fullsized":"2023-11-28T22:08:45.328352+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:11:27.455178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656534+0000","last_change":"2023-11-28T22:08:44.356291+0000","last_active":"2023-11-28T22:08:45.656534+0000","last_peered":"2023-11-28T22:08:45.656534+0000","last_clean":"2023-11-28T22:08:45.656534+0000","last_became_active":"2023-11-28T22:08:44.356022+0000","last_became_peered":"2023-11-28T22:08:44.356022+0000","last_unstale":"2023-11-28T22:08:45.656534+0000","last_undegraded":"2023-11-28T22:08:45.656534+0000","last_fullsized":"2023-11-28T22:08:45.656534+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:22.349395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656783+0000","last_change":"2023-11-28T22:08:43.322751+0000","last_active":"2023-11-28T22:08:45.656783+0000","last_peered":"2023-11-28T22:08:45.656783+0000","last_clean":"2023-11-28T22:08:45.656783+0000","last_became_active":"2023-11-28T22:08:43.322469+0000","last_became_peered":"2023-11-28T22:08:43.322469+0000","last_unstale":"2023-11-28T22:08:45.656783+0000","last_undegraded":"2023-11-28T22:08:45.656783+0000","last_fullsized":"2023-11-28T22:08:45.656783+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:09:20.278851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656687+0000","last_change":"2023-11-28T22:08:44.351594+0000","last_active":"2023-11-28T22:08:45.656687+0000","last_peered":"2023-11-28T22:08:45.656687+0000","last_clean":"2023-11-28T22:08:45.656687+0000","last_became_active":"2023-11-28T22:08:44.351295+0000","last_became_peered":"2023-11-28T22:08:44.351295+0000","last_unstale":"2023-11-28T22:08:45.656687+0000","last_undegraded":"2023-11-28T22:08:45.656687+0000","last_fullsized":"2023-11-28T22:08:45.656687+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:49:15.900529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655895+0000","last_change":"2023-11-28T22:08:43.314065+0000","last_active":"2023-11-28T22:08:45.655895+0000","last_peered":"2023-11-28T22:08:45.655895+0000","last_clean":"2023-11-28T22:08:45.655895+0000","last_became_active":"2023-11-28T22:08:43.313794+0000","last_became_peered":"2023-11-28T22:08:43.313794+0000","last_unstale":"2023-11-28T22:08:45.655895+0000","last_undegraded":"2023-11-28T22:08:45.655895+0000","last_fullsized":"2023-11-28T22:08:45.655895+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:30:54.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323661+0000","last_change":"2023-11-28T22:08:44.359633+0000","last_active":"2023-11-28T22:08:45.323661+0000","last_peered":"2023-11-28T22:08:45.323661+0000","last_clean":"2023-11-28T22:08:45.323661+0000","last_became_active":"2023-11-28T22:08:44.359354+0000","last_became_peered":"2023-11-28T22:08:44.359354+0000","last_unstale":"2023-11-28T22:08:45.323661+0000","last_undegraded":"2023-11-28T22:08:45.323661+0000","last_fullsized":"2023-11-28T22:08:45.323661+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:19:15.603608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660146+0000","last_change":"2023-11-28T22:08:43.322476+0000","last_active":"2023-11-28T22:08:45.660146+0000","last_peered":"2023-11-28T22:08:45.660146+0000","last_clean":"2023-11-28T22:08:45.660146+0000","last_became_active":"2023-11-28T22:08:43.322162+0000","last_became_peered":"2023-11-28T22:08:43.322162+0000","last_unstale":"2023-11-28T22:08:45.660146+0000","last_undegraded":"2023-11-28T22:08:45.660146+0000","last_fullsized":"2023-11-28T22:08:45.660146+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:39:21.240030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655924+0000","last_change":"2023-11-28T22:08:44.349297+0000","last_active":"2023-11-28T22:08:45.655924+0000","last_peered":"2023-11-28T22:08:45.655924+0000","last_clean":"2023-11-28T22:08:45.655924+0000","last_became_active":"2023-11-28T22:08:44.349006+0000","last_became_peered":"2023-11-28T22:08:44.349006+0000","last_unstale":"2023-11-28T22:08:45.655924+0000","last_undegraded":"2023-11-28T22:08:45.655924+0000","last_fullsized":"2023-11-28T22:08:45.655924+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:54:22.351455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"214'2","reported_seq":14,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657913+0000","last_change":"2023-11-28T22:08:43.322116+0000","last_active":"2023-11-28T22:08:45.657913+0000","last_peered":"2023-11-28T22:08:45.657913+0000","last_clean":"2023-11-28T22:08:45.657913+0000","last_became_active":"2023-11-28T22:08:43.321844+0000","last_became_peered":"2023-11-28T22:08:43.321844+0000","last_unstale":"2023-11-28T22:08:45.657913+0000","last_undegraded":"2023-11-28T22:08:45.657913+0000","last_fullsized":"2023-11-28T22:08:45.657913+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:40:43.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320853+0000","last_change":"2023-11-28T22:08:44.348881+0000","last_active":"2023-11-28T22:08:45.320853+0000","last_peered":"2023-11-28T22:08:45.320853+0000","last_clean":"2023-11-28T22:08:45.320853+0000","last_became_active":"2023-11-28T22:08:44.348144+0000","last_became_peered":"2023-11-28T22:08:44.348144+0000","last_unstale":"2023-11-28T22:08:45.320853+0000","last_undegraded":"2023-11-28T22:08:45.320853+0000","last_fullsized":"2023-11-28T22:08:45.320853+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:03:16.357248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655951+0000","last_change":"2023-11-28T22:08:43.313470+0000","last_active":"2023-11-28T22:08:45.655951+0000","last_peered":"2023-11-28T22:08:45.655951+0000","last_clean":"2023-11-28T22:08:45.655951+0000","last_became_active":"2023-11-28T22:08:43.313187+0000","last_became_peered":"2023-11-28T22:08:43.313187+0000","last_unstale":"2023-11-28T22:08:45.655951+0000","last_undegraded":"2023-11-28T22:08:45.655951+0000","last_fullsized":"2023-11-28T22:08:45.655951+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:24.779529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.659677+0000","last_change":"2023-11-28T22:08:44.361687+0000","last_active":"2023-11-28T22:08:45.659677+0000","last_peered":"2023-11-28T22:08:45.659677+0000","last_clean":"2023-11-28T22:08:45.659677+0000","last_became_active":"2023-11-28T22:08:44.361110+0000","last_became_peered":"2023-11-28T22:08:44.361110+0000","last_unstale":"2023-11-28T22:08:45.659677+0000","last_undegraded":"2023-11-28T22:08:45.659677+0000","last_fullsized":"2023-11-28T22:08:45.659677+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:34:44.898544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.324047+0000","last_change":"2023-11-28T22:08:43.321163+0000","last_active":"2023-11-28T22:08:45.324047+0000","last_peered":"2023-11-28T22:08:45.324047+0000","last_clean":"2023-11-28T22:08:45.324047+0000","last_became_active":"2023-11-28T22:08:43.320863+0000","last_became_peered":"2023-11-28T22:08:43.320863+0000","last_unstale":"2023-11-28T22:08:45.324047+0000","last_undegraded":"2023-11-28T22:08:45.324047+0000","last_fullsized":"2023-11-28T22:08:45.324047+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:21:48.835996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657011+0000","last_change":"2023-11-28T22:08:44.351869+0000","last_active":"2023-11-28T22:08:45.657011+0000","last_peered":"2023-11-28T22:08:45.657011+0000","last_clean":"2023-11-28T22:08:45.657011+0000","last_became_active":"2023-11-28T22:08:44.351607+0000","last_became_peered":"2023-11-28T22:08:44.351607+0000","last_unstale":"2023-11-28T22:08:45.657011+0000","last_undegraded":"2023-11-28T22:08:45.657011+0000","last_fullsized":"2023-11-28T22:08:45.657011+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:56:57.236240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.323776+0000","last_change":"2023-11-28T22:08:43.324318+0000","last_active":"2023-11-28T22:08:45.323776+0000","last_peered":"2023-11-28T22:08:45.323776+0000","last_clean":"2023-11-28T22:08:45.323776+0000","last_became_active":"2023-11-28T22:08:43.324043+0000","last_became_peered":"2023-11-28T22:08:43.324043+0000","last_unstale":"2023-11-28T22:08:45.323776+0000","last_undegraded":"2023-11-28T22:08:45.323776+0000","last_fullsized":"2023-11-28T22:08:45.323776+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:05:31.778068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657395+0000","last_change":"2023-11-28T22:08:44.348238+0000","last_active":"2023-11-28T22:08:45.657395+0000","last_peered":"2023-11-28T22:08:45.657395+0000","last_clean":"2023-11-28T22:08:45.657395+0000","last_became_active":"2023-11-28T22:08:44.347958+0000","last_became_peered":"2023-11-28T22:08:44.347958+0000","last_unstale":"2023-11-28T22:08:45.657395+0000","last_undegraded":"2023-11-28T22:08:45.657395+0000","last_fullsized":"2023-11-28T22:08:45.657395+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:26:56.089290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.657243+0000","last_change":"2023-11-28T22:08:43.324700+0000","last_active":"2023-11-28T22:08:45.657243+0000","last_peered":"2023-11-28T22:08:45.657243+0000","last_clean":"2023-11-28T22:08:45.657243+0000","last_became_active":"2023-11-28T22:08:43.324404+0000","last_became_peered":"2023-11-28T22:08:43.324404+0000","last_unstale":"2023-11-28T22:08:45.657243+0000","last_undegraded":"2023-11-28T22:08:45.657243+0000","last_fullsized":"2023-11-28T22:08:45.657243+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:40:43.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.327669+0000","last_change":"2023-11-28T22:08:44.351073+0000","last_active":"2023-11-28T22:08:45.327669+0000","last_peered":"2023-11-28T22:08:45.327669+0000","last_clean":"2023-11-28T22:08:45.327669+0000","last_became_active":"2023-11-28T22:08:44.350680+0000","last_became_peered":"2023-11-28T22:08:44.350680+0000","last_unstale":"2023-11-28T22:08:45.327669+0000","last_undegraded":"2023-11-28T22:08:45.327669+0000","last_fullsized":"2023-11-28T22:08:45.327669+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:42:09.371217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656552+0000","last_change":"2023-11-28T22:08:43.326545+0000","last_active":"2023-11-28T22:08:45.656552+0000","last_peered":"2023-11-28T22:08:45.656552+0000","last_clean":"2023-11-28T22:08:45.656552+0000","last_became_active":"2023-11-28T22:08:43.326140+0000","last_became_peered":"2023-11-28T22:08:43.326140+0000","last_unstale":"2023-11-28T22:08:45.656552+0000","last_undegraded":"2023-11-28T22:08:45.656552+0000","last_fullsized":"2023-11-28T22:08:45.656552+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:35.471277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.320802+0000","last_change":"2023-11-28T22:08:44.352885+0000","last_active":"2023-11-28T22:08:45.320802+0000","last_peered":"2023-11-28T22:08:45.320802+0000","last_clean":"2023-11-28T22:08:45.320802+0000","last_became_active":"2023-11-28T22:08:44.352393+0000","last_became_peered":"2023-11-28T22:08:44.352393+0000","last_unstale":"2023-11-28T22:08:45.320802+0000","last_undegraded":"2023-11-28T22:08:45.320802+0000","last_fullsized":"2023-11-28T22:08:45.320802+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T08:18:55.181191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655821+0000","last_change":"2023-11-28T22:08:43.317721+0000","last_active":"2023-11-28T22:08:45.655821+0000","last_peered":"2023-11-28T22:08:45.655821+0000","last_clean":"2023-11-28T22:08:45.655821+0000","last_became_active":"2023-11-28T22:08:43.317449+0000","last_became_peered":"2023-11-28T22:08:43.317449+0000","last_unstale":"2023-11-28T22:08:45.655821+0000","last_undegraded":"2023-11-28T22:08:45.655821+0000","last_fullsized":"2023-11-28T22:08:45.655821+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:19:15.001388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.656989+0000","last_change":"2023-11-28T22:08:44.363832+0000","last_active":"2023-11-28T22:08:45.656989+0000","last_peered":"2023-11-28T22:08:45.656989+0000","last_clean":"2023-11-28T22:08:45.656989+0000","last_became_active":"2023-11-28T22:08:44.363530+0000","last_became_peered":"2023-11-28T22:08:44.363530+0000","last_unstale":"2023-11-28T22:08:45.656989+0000","last_undegraded":"2023-11-28T22:08:45.656989+0000","last_fullsized":"2023-11-28T22:08:45.656989+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:36.147972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.660165+0000","last_change":"2023-11-28T22:08:43.322473+0000","last_active":"2023-11-28T22:08:45.660165+0000","last_peered":"2023-11-28T22:08:45.660165+0000","last_clean":"2023-11-28T22:08:45.660165+0000","last_became_active":"2023-11-28T22:08:43.321812+0000","last_became_peered":"2023-11-28T22:08:43.321812+0000","last_unstale":"2023-11-28T22:08:45.660165+0000","last_undegraded":"2023-11-28T22:08:45.660165+0000","last_fullsized":"2023-11-28T22:08:45.660165+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:47:17.834687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":11,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.658888+0000","last_change":"2023-11-28T22:08:44.361774+0000","last_active":"2023-11-28T22:08:45.658888+0000","last_peered":"2023-11-28T22:08:45.658888+0000","last_clean":"2023-11-28T22:08:45.658888+0000","last_became_active":"2023-11-28T22:08:44.361437+0000","last_became_peered":"2023-11-28T22:08:44.361437+0000","last_unstale":"2023-11-28T22:08:45.658888+0000","last_undegraded":"2023-11-28T22:08:45.658888+0000","last_fullsized":"2023-11-28T22:08:45.658888+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:43.279679+0000","last_clean_scrub_stamp":"2023-11-28T22:08:43.279679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:39:19.102529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-11-28T22:08:45.655769+0000","last_change":"2023-11-28T22:08:43.317927+0000","last_active":"2023-11-28T22:08:45.655769+0000","last_peered":"2023-11-28T22:08:45.655769+0000","last_clean":"2023-11-28T22:08:45.655769+0000","last_became_active":"2023-11-28T22:08:43.316095+0000","last_became_peered":"2023-11-28T22:08:43.316095+0000","last_unstale":"2023-11-28T22:08:45.655769+0000","last_undegraded":"2023-11-28T22:08:45.655769+0000","last_fullsized":"2023-11-28T22:08:45.655769+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T22:08:42.266286+0000","last_clean_scrub_stamp":"2023-11-28T22:08:42.266286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T06:14:35.471277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"204'4833","reported_seq":5694,"reported_epoch":214,"state":"active+clean","last_fresh":"2023-11-28T22:08:44.339674+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T22:08:44.339674+0000","last_peered":"2023-11-28T22:08:44.339674+0000","last_clean":"2023-11-28T22:08:44.339674+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T22:08:44.339674+0000","last_undegraded":"2023-11-28T22:08:44.339674+0000","last_fullsized":"2023-11-28T22:08:44.339674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4833,"log_dups_size":0,"ondisk_log_size":4833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2053,"num_read_kb":4245,"num_write":551,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2053,"num_read_kb":4245,"num_write":551,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4833,"ondisk_log_size":4833,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575976,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74644,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297196,"statfs":{"total":96636764160,"available":96560328704,"internally_reserved":0,"allocated":3276800,"data_stored":1966912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.228,"15min":1.218},"min":{"1min":0.98099999999999998,"5min":0.76300000000000001,"15min":0.57599999999999996},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9330000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.252,"15min":1.266},"min":{"1min":0.93999999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.6379999999999999,"5min":2.0190000000000001,"15min":2.6819999999999999},"last":0.82999999999999996}]},{"osd":1,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.282,"15min":1.2589999999999999},"min":{"1min":0.92800000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.7869999999999999,"5min":2.694,"15min":2.694},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.244,"15min":1.29},"min":{"1min":0.90400000000000003,"5min":0.84199999999999997,"15min":0.71099999999999997},"max":{"1min":1.756,"5min":1.756,"15min":3.0430000000000001},"last":1.21}]},{"osd":2,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.2969999999999999,"15min":1.3169999999999999},"min":{"1min":0.95599999999999996,"5min":0.94199999999999995,"15min":0.86599999999999999},"max":{"1min":1.671,"5min":1.6910000000000001,"15min":2.7480000000000002},"last":1.349},{"interface":"front","average":{"1min":1.373,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":1.0009999999999999,"5min":0.84899999999999998,"15min":0.79200000000000004},"max":{"1min":1.833,"5min":1.9019999999999999,"15min":2.2400000000000002},"last":0.70099999999999996}]},{"osd":3,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3440000000000001,"15min":1.351},"min":{"1min":0.97099999999999997,"5min":0.89100000000000001,"15min":0.82499999999999996},"max":{"1min":1.7250000000000001,"5min":1.7989999999999999,"15min":1.8740000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.585,"5min":1.48,"15min":1.4630000000000001},"min":{"1min":1.2050000000000001,"5min":1.1020000000000001,"15min":0.93300000000000005},"max":{"1min":2.258,"5min":2.258,"15min":2.9329999999999998},"last":1.484}]},{"osd":4,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3180000000000001,"15min":1.3340000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.81299999999999994},"max":{"1min":1.7889999999999999,"5min":1.9930000000000001,"15min":2.6930000000000001},"last":1.054},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.367,"15min":1.3740000000000001},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.823,"5min":1.905,"15min":2.371},"last":0.88500000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.4390000000000001,"15min":1.454},"min":{"1min":0.98899999999999999,"5min":0.871,"15min":0.77500000000000002},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":3.2229999999999999},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4990000000000001,"15min":1.5169999999999999},"min":{"1min":1.03,"5min":0.92100000000000004,"15min":0.90100000000000002},"max":{"1min":1.859,"5min":2.0760000000000001,"15min":2.508},"last":1.2909999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:07:51 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.599,"15min":1.6220000000000001},"min":{"1min":1.038,"5min":1.038,"15min":0.98999999999999999},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.6110000000000002},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.554,"15min":1.5349999999999999},"min":{"1min":1.2030000000000001,"5min":1.113,"15min":0.76800000000000002},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2869999999999999},"last":1.2549999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575976,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42136,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329704,"statfs":{"total":96636764160,"available":96593616896,"internally_reserved":0,"allocated":2699264,"data_stored":1383262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.135,"15min":1.125},"min":{"1min":0.78500000000000003,"5min":0.78000000000000003,"15min":0.68300000000000005},"max":{"1min":1.5409999999999999,"5min":1.6930000000000001,"15min":1.772},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1040000000000001,"15min":1.1379999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.4730000000000001,"5min":1.653,"15min":1.974},"last":1.113}]},{"osd":1,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.276,"15min":1.2989999999999999},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.71599999999999997},"max":{"1min":1.603,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4570000000000001,"15min":1.4419999999999999},"min":{"1min":0.999,"5min":0.89300000000000002,"15min":0.84199999999999997},"max":{"1min":1.659,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":0.98999999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.369,"15min":1.353},"min":{"1min":0.94799999999999995,"5min":0.85399999999999998,"15min":0.80000000000000004},"max":{"1min":1.6679999999999999,"5min":2.0779999999999998,"15min":2.4809999999999999},"last":1.304},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4139999999999999,"15min":1.4359999999999999},"min":{"1min":1.1990000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.835,"5min":2.1829999999999998,"15min":2.6269999999999998},"last":1.212}]},{"osd":3,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.3720000000000001,"15min":1.3540000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.88500000000000001},"max":{"1min":1.794,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.492},{"interface":"front","average":{"1min":1.331,"5min":1.3160000000000001,"15min":1.339},"min":{"1min":1.0249999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.629,"5min":1.7430000000000001,"15min":1.8859999999999999},"last":1.46}]},{"osd":4,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2729999999999999,"15min":1.2949999999999999},"min":{"1min":1.056,"5min":0.81599999999999995,"15min":0.79800000000000004},"max":{"1min":1.7310000000000001,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.071},{"interface":"front","average":{"1min":1.357,"5min":1.397,"15min":1.3779999999999999},"min":{"1min":1.034,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.875,"5min":2.02,"15min":2.02},"last":1.431}]},{"osd":5,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.421,"15min":1.3959999999999999},"min":{"1min":0.94999999999999996,"5min":0.86499999999999999,"15min":0.84199999999999997},"max":{"1min":1.7170000000000001,"5min":1.9990000000000001,"15min":2.081},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.573,"15min":1.5940000000000001},"min":{"1min":1.1719999999999999,"5min":1.077,"15min":1.0149999999999999},"max":{"1min":1.9059999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":1.3660000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:07:52 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.5449999999999999,"15min":1.538},"min":{"1min":1.127,"5min":1.0649999999999999,"15min":0.98899999999999999},"max":{"1min":1.9419999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.258},{"interface":"front","average":{"1min":1.456,"5min":1.4950000000000001,"15min":1.5249999999999999},"min":{"1min":1.006,"5min":1.0029999999999999,"15min":0.80000000000000004},"max":{"1min":1.8009999999999999,"5min":2.1150000000000002,"15min":2.2130000000000001},"last":1.532}]}]},{"osd":5,"up_from":13,"seq":55834575976,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60196,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311644,"statfs":{"total":96636764160,"available":96575123456,"internally_reserved":0,"allocated":2711552,"data_stored":1384826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.153,"15min":1.179},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.48499999999999999},"max":{"1min":1.9039999999999999,"5min":1.9039999999999999,"15min":1.9550000000000001},"last":1.153},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.216,"15min":1.2070000000000001},"min":{"1min":0.82599999999999996,"5min":0.79200000000000004,"15min":0.65700000000000003},"max":{"1min":1.6679999999999999,"5min":1.889,"15min":2.1709999999999998},"last":1.2669999999999999}]},{"osd":1,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.196,"15min":1.214},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6499999999999999,"5min":1.706,"15min":1.8979999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.292,"15min":1.2929999999999999},"min":{"1min":1.024,"5min":0.86199999999999999,"15min":0.54400000000000004},"max":{"1min":1.7350000000000001,"5min":1.7509999999999999,"15min":1.8859999999999999},"last":1.0620000000000001}]},{"osd":2,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3240000000000001,"15min":1.3400000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.60699999999999998},"max":{"1min":1.768,"5min":1.768,"15min":1.974},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.381,"15min":1.3660000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.76,"5min":2.0289999999999999,"15min":2.105},"last":1.492}]},{"osd":3,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.446,"15min":1.4179999999999999},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.68899999999999995},"max":{"1min":1.6850000000000001,"5min":1.988,"15min":1.988},"last":1.421},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3140000000000001,"15min":1.341},"min":{"1min":0.97699999999999998,"5min":0.94599999999999995,"15min":0.87},"max":{"1min":1.752,"5min":1.8779999999999999,"15min":1.9330000000000001},"last":1.1220000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.3799999999999999,"15min":1.377},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.72599999999999998},"max":{"1min":1.744,"5min":1.833,"15min":1.833},"last":1.212},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.46,"15min":1.492},"min":{"1min":0.86399999999999999,"5min":0.83999999999999997,"15min":0.76200000000000001},"max":{"1min":1.8089999999999999,"5min":1.95,"15min":2.0249999999999999},"last":1.333}]},{"osd":6,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.4390000000000001,"15min":1.4119999999999999},"min":{"1min":1.0209999999999999,"5min":0.84099999999999997,"15min":0.80000000000000004},"max":{"1min":1.833,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.518,"15min":1.5449999999999999},"min":{"1min":1.024,"5min":0.97199999999999998,"15min":0.94399999999999995},"max":{"1min":1.865,"5min":1.982,"15min":1.982},"last":1.476}]},{"osd":7,"last update":"Tue Nov 28 22:08:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4690000000000001,"15min":1.4830000000000001},"min":{"1min":1.262,"5min":1.0580000000000001,"15min":0.76100000000000001},"max":{"1min":1.7370000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.238},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.575,"15min":1.575},"min":{"1min":1.1679999999999999,"5min":0.877,"15min":0.877},"max":{"1min":1.9450000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.6200000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575977,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47720,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324120,"statfs":{"total":96636764160,"available":96587898880,"internally_reserved":0,"allocated":2711552,"data_stored":1384838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.099,"15min":1.1359999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.573,"5min":1.6839999999999999,"15min":1.8149999999999999},"last":1.133},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1699999999999999,"15min":1.163},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.742},"last":1.4970000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.367,"15min":1.3779999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.873},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.375,"15min":1.409},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.984,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.77}]},{"osd":2,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.258,"15min":1.268},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3300000000000001,"15min":1.351},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6519999999999999,"5min":1.915,"15min":1.944},"last":1.927}]},{"osd":3,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6579999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.016},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.403,"15min":1.4039999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.9099999999999999,"5min":1.929,"15min":2.036},"last":1.6899999999999999}]},{"osd":5,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.343,"15min":1.3300000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5740000000000001,"5min":2.0030000000000001,"15min":2.0609999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.355,"15min":1.3580000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7669999999999999,"5min":1.9139999999999999,"15min":2.1659999999999999},"last":1.738}]},{"osd":6,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.4470000000000001,"15min":1.4359999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.61499999999999999},"max":{"1min":1.8460000000000001,"5min":1.9630000000000001,"15min":2.097},"last":2.004},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.23},"last":2.1120000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:08:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4810000000000001,"15min":1.492},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.869,"5min":2.0190000000000001,"15min":2.3340000000000001},"last":1.8049999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5760000000000001,"15min":1.583},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.986,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":2.1659999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575977,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56032,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315808,"statfs":{"total":96636764160,"available":96579387392,"internally_reserved":0,"allocated":2703360,"data_stored":1383352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.97699999999999998,"15min":0.95599999999999996},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.58399999999999996},"max":{"1min":1.2949999999999999,"5min":1.786,"15min":2.0950000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.167,"5min":1.155,"15min":1.1339999999999999},"min":{"1min":0.83299999999999996,"5min":0.74399999999999999,"15min":0.625},"max":{"1min":1.835,"5min":1.835,"15min":2.3879999999999999},"last":0.60699999999999998}]},{"osd":1,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0449999999999999,"15min":1.034},"min":{"1min":0.78200000000000003,"5min":0.66700000000000004,"15min":0.56899999999999995},"max":{"1min":1.512,"5min":1.6299999999999999,"15min":1.698},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2649999999999999,"15min":1.262},"min":{"1min":0.89800000000000002,"5min":0.83499999999999996,"15min":0.66300000000000003},"max":{"1min":1.6479999999999999,"5min":1.827,"15min":2.6019999999999999},"last":0.995}]},{"osd":2,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.321,"15min":1.3160000000000001},"min":{"1min":0.95799999999999996,"5min":0.81999999999999995,"15min":0.58099999999999996},"max":{"1min":1.621,"5min":2.105,"15min":2.327},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.411,"15min":1.407},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.702,"5min":1.9390000000000001,"15min":2.218},"last":1.0589999999999999}]},{"osd":4,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.3839999999999999,"15min":1.3919999999999999},"min":{"1min":1.016,"5min":0.94499999999999995,"15min":0.82899999999999996},"max":{"1min":1.6040000000000001,"5min":1.948,"15min":2.181},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.488,"15min":1.4890000000000001},"min":{"1min":1.115,"5min":1.109,"15min":0.80000000000000004},"max":{"1min":1.736,"5min":2.0569999999999999,"15min":2.1949999999999998},"last":1.4099999999999999}]},{"osd":5,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.5389999999999999,"15min":1.5489999999999999},"min":{"1min":1.069,"5min":1.038,"15min":0.97699999999999998},"max":{"1min":1.9470000000000001,"5min":2.3109999999999999,"15min":2.4329999999999998},"last":1.111},{"interface":"front","average":{"1min":1.49,"5min":1.5109999999999999,"15min":1.508},"min":{"1min":1.151,"5min":1.0269999999999999,"15min":0.98999999999999999},"max":{"1min":1.984,"5min":1.984,"15min":2.2639999999999998},"last":1.224}]},{"osd":6,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.5880000000000001,"15min":1.575},"min":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":0.878},"max":{"1min":2.048,"5min":2.1230000000000002,"15min":2.331},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.681,"15min":1.6659999999999999},"min":{"1min":1.196,"5min":1.087,"15min":1.087},"max":{"1min":1.913,"5min":2.5070000000000001,"15min":2.5329999999999999},"last":1.1499999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:07:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.663,"15min":1.6339999999999999},"min":{"1min":1.0940000000000001,"5min":1.056,"15min":1.036},"max":{"1min":1.9890000000000001,"5min":2.0950000000000002,"15min":2.3380000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.657,"5min":1.7150000000000001,"15min":1.6990000000000001},"min":{"1min":1.3240000000000001,"5min":1.1699999999999999,"15min":0.95899999999999996},"max":{"1min":2.1800000000000002,"5min":2.1800000000000002,"15min":2.4670000000000001},"last":1.4710000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575977,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51872,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319968,"statfs":{"total":96636764160,"available":96583647232,"internally_reserved":0,"allocated":2699264,"data_stored":1383774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.97799999999999998,"15min":0.97599999999999998},"min":{"1min":0.71899999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.5589999999999999,"5min":1.6120000000000001,"15min":1.7190000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1419999999999999,"15min":1.157},"min":{"1min":0.81000000000000005,"5min":0.76200000000000001,"15min":0.49199999999999999},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9099999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.036,"15min":1.036},"min":{"1min":0.78900000000000003,"5min":0.76300000000000001,"15min":0.56000000000000005},"max":{"1min":2.097,"5min":2.097,"15min":2.097},"last":0.62},{"interface":"front","average":{"1min":1.175,"5min":1.232,"15min":1.2390000000000001},"min":{"1min":0.89800000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.6000000000000001,"5min":1.625,"15min":1.8149999999999999},"last":0.59599999999999997}]},{"osd":3,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2490000000000001,"15min":1.2729999999999999},"min":{"1min":0.95099999999999996,"5min":0.871,"15min":0.754},"max":{"1min":1.72,"5min":1.72,"15min":1.9330000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.365,"15min":1.3480000000000001},"min":{"1min":0.99099999999999999,"5min":0.92300000000000004,"15min":0.89700000000000002},"max":{"1min":1.849,"5min":1.849,"15min":2.2210000000000001},"last":0.55500000000000005}]},{"osd":4,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.48,"15min":1.4790000000000001},"min":{"1min":1.127,"5min":0.995,"15min":0.92300000000000004},"max":{"1min":1.956,"5min":1.956,"15min":2.0640000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.468,"15min":1.476},"min":{"1min":1.1419999999999999,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":1.8640000000000001,"5min":1.8779999999999999,"15min":2.173},"last":0.90300000000000002}]},{"osd":5,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.4970000000000001,"15min":1.5069999999999999},"min":{"1min":1.1890000000000001,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.893,"5min":2.222,"15min":2.3359999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5940000000000001,"15min":1.577},"min":{"1min":1.2150000000000001,"5min":1.107,"15min":0.86099999999999999},"max":{"1min":1.875,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.81100000000000005}]},{"osd":6,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.6140000000000001,"15min":1.615},"min":{"1min":1.2150000000000001,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.9710000000000001,"5min":2.387,"15min":2.387},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.669,"5min":1.6759999999999999,"15min":1.6659999999999999},"min":{"1min":1.2949999999999999,"5min":1.177,"15min":1.129},"max":{"1min":1.992,"5min":2.3940000000000001,"15min":2.4020000000000001},"last":1.075}]},{"osd":7,"last update":"Tue Nov 28 22:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.615,"15min":1.6240000000000001},"min":{"1min":1.1839999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.966,"5min":2.1899999999999999,"15min":2.4260000000000002},"last":1.153},{"interface":"front","average":{"1min":1.649,"5min":1.6879999999999999,"15min":1.716},"min":{"1min":1.194,"5min":1.0469999999999999,"15min":1.0469999999999999},"max":{"1min":2.04,"5min":2.2599999999999998,"15min":2.2759999999999998},"last":0.79200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575977,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51864,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319976,"statfs":{"total":96636764160,"available":96583655424,"internally_reserved":0,"allocated":2695168,"data_stored":1383192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.745,"5min":0.498,"15min":0.498},"max":{"1min":1.5449999999999999,"5min":1.669,"15min":1.8959999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1799999999999999,"15min":1.171},"min":{"1min":0.76500000000000001,"5min":0.629,"15min":0.629},"max":{"1min":1.9279999999999999,"5min":1.9410000000000001,"15min":2.3900000000000001},"last":1.3959999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.1879999999999999,"15min":1.1819999999999999},"min":{"1min":0.88200000000000001,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.984,"5min":1.984,"15min":2.1589999999999998},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2709999999999999,"15min":1.2829999999999999},"min":{"1min":0.85599999999999998,"5min":0.81499999999999995,"15min":0.77800000000000002},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.80000000000000004}]},{"osd":3,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.3999999999999999,"15min":1.4179999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.81699999999999995},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2320000000000002},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.218,"5min":1.1910000000000001,"15min":1.2110000000000001},"min":{"1min":0.81599999999999995,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7150000000000001,"5min":1.734,"15min":2.4359999999999999},"last":1.216}]},{"osd":4,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.419,"15min":1.4359999999999999},"min":{"1min":1.113,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":1.9510000000000001,"5min":2.2050000000000001,"15min":2.3559999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.431,"15min":1.46},"min":{"1min":1.0880000000000001,"5min":0.95099999999999996,"15min":0.83599999999999997},"max":{"1min":1.7569999999999999,"5min":2.2759999999999998,"15min":2.3079999999999998},"last":0.82899999999999996}]},{"osd":5,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.478,"15min":1.5069999999999999},"min":{"1min":1.1359999999999999,"5min":0.98899999999999999,"15min":0.77800000000000002},"max":{"1min":1.8300000000000001,"5min":2.0310000000000001,"15min":2.347},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.609,"15min":1.6180000000000001},"min":{"1min":1.248,"5min":1.121,"15min":0.95699999999999996},"max":{"1min":2.3420000000000001,"5min":2.3580000000000001,"15min":2.448},"last":1.2769999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5109999999999999,"15min":1.5289999999999999},"min":{"1min":1.218,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.875,"5min":2.1160000000000001,"15min":2.5249999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5760000000000001,"15min":1.603},"min":{"1min":1.2030000000000001,"5min":1.1319999999999999,"15min":1.093},"max":{"1min":2.0619999999999998,"5min":2.1389999999999998,"15min":2.4550000000000001},"last":1.252}]},{"osd":7,"last update":"Tue Nov 28 22:08:05 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.6779999999999999,"15min":1.706},"min":{"1min":1.4079999999999999,"5min":1.3620000000000001,"15min":0.83999999999999997},"max":{"1min":2.1150000000000002,"5min":2.5150000000000001,"15min":2.6110000000000002},"last":1.125},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.575,"15min":1.613},"min":{"1min":1.2609999999999999,"5min":1.056,"15min":1.056},"max":{"1min":2.1389999999999998,"5min":2.2730000000000001,"15min":2.8460000000000001},"last":1.4279999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575977,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52464,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319376,"statfs":{"total":96636764160,"available":96583041024,"internally_reserved":0,"allocated":3305472,"data_stored":1974288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.93200000000000005,"15min":0.92900000000000005},"min":{"1min":0.75800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4850000000000001,"5min":1.621,"15min":1.722},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1659999999999999,"15min":1.2},"min":{"1min":0.91300000000000003,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4610000000000001,"5min":2.0350000000000001,"15min":2.0569999999999999},"last":0.82599999999999996}]},{"osd":2,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1719999999999999,"15min":1.1870000000000001},"min":{"1min":0.90700000000000003,"5min":0.77600000000000002,"15min":0.76100000000000001},"max":{"1min":1.6279999999999999,"5min":1.835,"15min":2.1709999999999998},"last":1.159},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2669999999999999,"15min":1.252},"min":{"1min":0.872,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.71,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.085}]},{"osd":3,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.218,"15min":1.173},"min":{"1min":0.86799999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8009999999999999,"5min":1.9139999999999999,"15min":1.986},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.403,"15min":1.383},"min":{"1min":1.012,"5min":0.99399999999999999,"15min":0.89400000000000002},"max":{"1min":1.746,"5min":1.7949999999999999,"15min":2.4409999999999998},"last":1.369}]},{"osd":4,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.4850000000000001,"15min":1.48},"min":{"1min":1.077,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":1.98,"5min":1.98,"15min":2.1949999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.431,"15min":1.4330000000000001},"min":{"1min":1.04,"5min":1.04,"15min":0.96399999999999997},"max":{"1min":1.756,"5min":2.149,"15min":2.254},"last":1.1020000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.504,"15min":1.496},"min":{"1min":1.2050000000000001,"5min":0.997,"15min":0.997},"max":{"1min":2.0409999999999999,"5min":2.048,"15min":2.1520000000000001},"last":1.329},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5900000000000001,"15min":1.5640000000000001},"min":{"1min":1.284,"5min":1.1619999999999999,"15min":1.093},"max":{"1min":2.173,"5min":2.3050000000000002,"15min":2.3359999999999999},"last":1.6870000000000001}]},{"osd":6,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.657,"15min":1.633},"min":{"1min":1.357,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":2.0030000000000001,"5min":2.21,"15min":2.246},"last":1.754},{"interface":"front","average":{"1min":1.623,"5min":1.617,"15min":1.6180000000000001},"min":{"1min":1.1040000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":2.1200000000000001,"5min":2.54,"15min":2.54},"last":1.518}]},{"osd":7,"last update":"Tue Nov 28 22:07:58 2023","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":1.6910000000000001,"15min":1.6870000000000001},"min":{"1min":1.427,"5min":1.0549999999999999,"15min":1.0549999999999999},"max":{"1min":2.206,"5min":2.2360000000000002,"15min":2.528},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.641,"15min":1.6339999999999999},"min":{"1min":1.167,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.2650000000000001,"5min":2.4089999999999998,"15min":2.4420000000000002},"last":1.407}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T22:08:49.399 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T22:08:49.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T22:08:49.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.677+0000 7ff0250bb640 1 Processor -- start 2023-11-28T22:08:49.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.677+0000 7ff0250bb640 1 -- start start 2023-11-28T22:08:49.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.679+0000 7ff0250bb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff020160aa0 0x7ff0200b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.679+0000 7ff0250bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0200ce7f0 0x7ff0200d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.679+0000 7ff0250bb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff020160e80 con 0x7ff0200d24f0 2023-11-28T22:08:49.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.679+0000 7ff0250bb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0200d1220 con 0x7ff020160aa0 2023-11-28T22:08:49.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.679+0000 7ff0250bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0200d2e70 con 0x7ff0200ce7f0 2023-11-28T22:08:49.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.679+0000 7ff01659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff0200d24f0 0x7ff0200ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48258/0 (socket says 172.21.15.45:48258) 2023-11-28T22:08:49.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.679+0000 7ff01659c640 1 -- 172.21.15.45:0/3651263151 learned_addr learned my addr 172.21.15.45:0/3651263151 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01ed9d640 1 --2- 172.21.15.45:0/3651263151 >> v2:172.21.15.190:3300/0 conn(0x7ff020160aa0 0x7ff0200b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01f59e640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0200ce7f0 0x7ff0200d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449591206 0 0) 0x7ff020160e80 con 0x7ff0200d24f0 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff00c003520 con 0x7ff0200d24f0 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194902040 0 0) 0x7ff00c003520 con 0x7ff0200d24f0 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0200ce7f0 msgr2=0x7ff0200d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0200ce7f0 0x7ff0200d2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 >> v2:172.21.15.190:3300/0 conn(0x7ff020160aa0 msgr2=0x7ff0200b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 --2- 172.21.15.45:0/3651263151 >> v2:172.21.15.190:3300/0 conn(0x7ff020160aa0 0x7ff0200b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0200b6960 con 0x7ff0200d24f0 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ff008001670 con 0x7ff0200d24f0 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 >> v1:172.21.15.45:6789/0 conn(0x7ff0200d24f0 legacy=0x7ff0200ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c003970 0x7ff00c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.680+0000 7ff01e59c640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0047c0 0x7ff00c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01e59c640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0200ce7f0 0x7ff00c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0200d1220 con 0x7ff00c0047c0 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0200d2e70 con 0x7ff00c003970 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c00b9b0 con 0x7ff0200ce7f0 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01659c640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0047c0 0x7ff00c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01f59e640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0200ce7f0 0x7ff00c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01ed9d640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c003970 0x7ff00c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01659c640 1 -- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0200ce7f0 msgr2=0x7ff00c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:49.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01659c640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0200ce7f0 0x7ff00c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01659c640 1 -- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c003970 msgr2=0x7ff00c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:49.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01659c640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c003970 0x7ff00c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.681+0000 7ff01659c640 1 -- 172.21.15.45:0/3651263151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c003520 con 0x7ff00c0047c0 2023-11-28T22:08:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.682+0000 7ff01659c640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0047c0 0x7ff00c004c40 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7ff00800c240 tx=0x7ff00c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e66371bb7c0b9ef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.683+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00802cb00 con 0x7ff00c0047c0 2023-11-28T22:08:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.683+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00802cc60 con 0x7ff00c0047c0 2023-11-28T22:08:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.683+0000 7ff01e59c640 1 -- 172.21.15.45:0/3651263151 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0080058d0 con 0x7ff00c0047c0 2023-11-28T22:08:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.683+0000 7ff0250bb640 1 -- 172.21.15.45:0/3651263151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0047c0 msgr2=0x7ff00c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.683+0000 7ff0250bb640 1 --2- 172.21.15.45:0/3651263151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff00c0047c0 0x7ff00c004c40 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7ff00800c240 tx=0x7ff00c0057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.683+0000 7ff01f59e640 1 -- 172.21.15.45:0/3651263151 reap_dead start 2023-11-28T22:08:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.683+0000 7ff0250bb640 1 -- 172.21.15.45:0/3651263151 shutdown_connections 2023-11-28T22:08:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.683+0000 7ff0250bb640 1 -- 172.21.15.45:0/3651263151 >> 172.21.15.45:0/3651263151 conn(0x7ff020070b20 msgr2=0x7ff020070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:49.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.684+0000 7ff0250bb640 1 -- 172.21.15.45:0/3651263151 shutdown_connections 2023-11-28T22:08:49.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.684+0000 7ff0250bb640 1 -- 172.21.15.45:0/3651263151 wait complete. 2023-11-28T22:08:49.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.685+0000 7ff0250bb640 1 Processor -- start 2023-11-28T22:08:49.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.686+0000 7ff0250bb640 1 -- start start 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff0250bb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c0047c0 0x7ff02015b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff0250bb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02015f1c0 0x7ff0201f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff0250bb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0201f3100 0x7ff0200ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff0250bb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0201f3ff0 con 0x7ff0201f3100 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff0250bb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0201f74f0 con 0x7ff00c0047c0 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff0250bb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0201f3a30 con 0x7ff02015f1c0 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0201f3100 0x7ff0200ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0201f3100 0x7ff0200ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59638/0 (socket says 172.21.15.45:59638) 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01f59e640 1 -- 172.21.15.45:0/111095241 learned_addr learned my addr 172.21.15.45:0/111095241 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01659c640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02015f1c0 0x7ff0201f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01ed9d640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c0047c0 0x7ff02015b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01f59e640 1 -- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02015f1c0 msgr2=0x7ff0201f6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01f59e640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02015f1c0 0x7ff0201f6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01f59e640 1 -- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c0047c0 msgr2=0x7ff02015b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01f59e640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c0047c0 0x7ff02015b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01f59e640 1 -- 172.21.15.45:0/111095241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0201f3d10 con 0x7ff0201f3100 2023-11-28T22:08:49.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01659c640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02015f1c0 0x7ff0201f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:49.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.687+0000 7ff01ed9d640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c0047c0 0x7ff02015b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:49.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.688+0000 7ff01f59e640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0201f3100 0x7ff0200ca2e0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7ff02006c5b0 tx=0x7ff018011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:49.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.688+0000 7ff017fff640 1 -- 172.21.15.45:0/111095241 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018004500 con 0x7ff0201f3100 2023-11-28T22:08:49.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.688+0000 7ff017fff640 1 -- 172.21.15.45:0/111095241 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff018003610 con 0x7ff0201f3100 2023-11-28T22:08:49.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.688+0000 7ff017fff640 1 -- 172.21.15.45:0/111095241 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0180039e0 con 0x7ff0201f3100 2023-11-28T22:08:49.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.688+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0201f67b0 con 0x7ff0201f3100 2023-11-28T22:08:49.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.688+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0201f3590 con 0x7ff0201f3100 2023-11-28T22:08:49.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.690+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0200b9220 con 0x7ff0201f3100 2023-11-28T22:08:49.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.691+0000 7ff017fff640 1 -- 172.21.15.45:0/111095241 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff018003b40 con 0x7ff0201f3100 2023-11-28T22:08:49.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.692+0000 7ff017fff640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff00006af60 0x7ff00006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:49.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.692+0000 7ff017fff640 1 -- 172.21.15.45:0/111095241 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff018089970 con 0x7ff0201f3100 2023-11-28T22:08:49.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.693+0000 7ff01ed9d640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff00006af60 0x7ff00006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:49.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.694+0000 7ff01ed9d640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff00006af60 0x7ff00006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff010007440 tx=0x7ff01000b040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:49.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.697+0000 7ff017fff640 1 -- 172.21.15.45:0/111095241 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0180547a0 con 0x7ff0201f3100 2023-11-28T22:08:49.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.957+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff0201f6cd0 con 0x7ff0201f3100 2023-11-28T22:08:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.962+0000 7ff017fff640 1 -- 172.21.15.45:0/111095241 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v48) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff0201f6cd0 con 0x7ff0201f3100 2023-11-28T22:08:49.965 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-11-28T22:08:49.965 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBxTmZlTznpMRAALGESJPilH4nGMGP1jodi1w== 2023-11-28T22:08:49.965 INFO:teuthology.orchestra.run.smithi045.stdout: caps mds = "allow" 2023-11-28T22:08:49.965 INFO:teuthology.orchestra.run.smithi045.stdout: caps mon = "allow r" 2023-11-28T22:08:49.965 INFO:teuthology.orchestra.run.smithi045.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T22:08:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:updated caps for client.0 2023-11-28T22:08:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff00006af60 msgr2=0x7ff00006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff00006af60 0x7ff00006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff010007440 tx=0x7ff01000b040 comp rx=0 tx=0).stop 2023-11-28T22:08:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0201f3100 msgr2=0x7ff0200ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0201f3100 0x7ff0200ca2e0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7ff02006c5b0 tx=0x7ff018011920 comp rx=0 tx=0).stop 2023-11-28T22:08:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 shutdown_connections 2023-11-28T22:08:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7ff00006af60 0x7ff00006d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0201f3100 0x7ff0200ca2e0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff02015f1c0 0x7ff0201f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 --2- 172.21.15.45:0/111095241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff00c0047c0 0x7ff02015b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 >> 172.21.15.45:0/111095241 conn(0x7ff020070b20 msgr2=0x7ff0200b8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 shutdown_connections 2023-11-28T22:08:49.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:49.966+0000 7ff0250bb640 1 -- 172.21.15.45:0/111095241 wait complete. 2023-11-28T22:08:49.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:08:50.131 INFO:tasks.ceph.mds.c.smithi045.stderr:2023-11-28T22:08:50.128+0000 7f7ab7337640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T22:08:50.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.200+0000 7f6638935640 1 Processor -- start 2023-11-28T22:08:50.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.200+0000 7f6638935640 1 -- start start 2023-11-28T22:08:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.201+0000 7f6638935640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66340b7f60 0x7f66341681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.201+0000 7f6638935640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66340d2230 con 0x7f66340d02d0 2023-11-28T22:08:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.201+0000 7f6638935640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66340d2510 con 0x7f663415d550 2023-11-28T22:08:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.201+0000 7f6638935640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66340d27f0 con 0x7f66340b7f60 2023-11-28T22:08:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.202+0000 7f662bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f66340d02d0 0x7f66340d1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48268/0 (socket says 172.21.15.45:48268) 2023-11-28T22:08:50.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.202+0000 7f662bfff640 1 -- 172.21.15.45:0/3224328128 learned_addr learned my addr 172.21.15.45:0/3224328128 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.202+0000 7f6633fff640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66340b7f60 0x7f66341681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.202+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570093917 0 0) 0x7f66340d2230 con 0x7f66340d02d0 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.202+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6620003520 con 0x7f66340d02d0 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1016249251 0 0) 0x7f6620003520 con 0x7f66340d02d0 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66340b7f60 msgr2=0x7f66341681d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66340b7f60 0x7f66341681d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 >> v1:172.21.15.190:6789/0 conn(0x7f663415d550 legacy=0x7f66340cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66340b6a30 con 0x7f66340d02d0 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f661c001670 con 0x7f66340d02d0 2023-11-28T22:08:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 >> v1:172.21.15.45:6789/0 conn(0x7f66340d02d0 legacy=0x7f66340d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66200039a0 0x7f6620003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66340b7f60 0x7f66200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6620004d00 0x7f6620005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66340d27f0 con 0x7f66200039a0 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662000ba30 con 0x7f6620004d00 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662000bbf0 con 0x7f66340b7f60 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f66337fe640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66200039a0 0x7f6620003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f66337fe640 1 -- 172.21.15.45:0/3224328128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66340b7f60 msgr2=0x7f66200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f66337fe640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66340b7f60 0x7f66200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f66337fe640 1 -- 172.21.15.45:0/3224328128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6620004d00 msgr2=0x7f6620005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f66337fe640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6620004d00 0x7f6620005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.203+0000 7f66337fe640 1 -- 172.21.15.45:0/3224328128 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6620003520 con 0x7f66200039a0 2023-11-28T22:08:50.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.205+0000 7f66337fe640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66200039a0 0x7f6620003dc0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f6624001720 tx=0x7f66200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9f557eacd01c65 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.205+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66240052d0 con 0x7f66200039a0 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.205+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662402e050 con 0x7f66200039a0 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.205+0000 7f6632ffd640 1 -- 172.21.15.45:0/3224328128 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66240035f0 con 0x7f66200039a0 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.205+0000 7f6638935640 1 -- 172.21.15.45:0/3224328128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66200039a0 msgr2=0x7f6620003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.205+0000 7f6638935640 1 --2- 172.21.15.45:0/3224328128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66200039a0 0x7f6620003dc0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f6624001720 tx=0x7f66200057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.205+0000 7f6633fff640 1 -- 172.21.15.45:0/3224328128 reap_dead start 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.206+0000 7f6638935640 1 -- 172.21.15.45:0/3224328128 shutdown_connections 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.206+0000 7f6638935640 1 -- 172.21.15.45:0/3224328128 >> 172.21.15.45:0/3224328128 conn(0x7f663406ac70 msgr2=0x7f66340b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.206+0000 7f6638935640 1 -- 172.21.15.45:0/3224328128 shutdown_connections 2023-11-28T22:08:50.209 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.206+0000 7f6638935640 1 -- 172.21.15.45:0/3224328128 wait complete. 2023-11-28T22:08:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.207+0000 7f6638935640 1 Processor -- start 2023-11-28T22:08:50.210 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.208+0000 7f6638935640 1 -- start start 2023-11-28T22:08:50.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f6638935640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66200039a0 0x7f66341f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f6638935640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66340cb2f0 0x7f66340cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f6638935640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f663414e890 0x7f66340ac450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f6638935640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66340cc3c0 con 0x7f66340cb2f0 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f6638935640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66341521f0 con 0x7f66200039a0 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f6638935640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66340cb7b0 con 0x7f663414e890 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f662bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66340cb2f0 0x7f66340cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f662bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66340cb2f0 0x7f66340cc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59656/0 (socket says 172.21.15.45:59656) 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f662bfff640 1 -- 172.21.15.45:0/3620375058 learned_addr learned my addr 172.21.15.45:0/3620375058 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f66337fe640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66200039a0 0x7f66341f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f662bfff640 1 -- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f663414e890 msgr2=0x7f66340ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:50.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f662bfff640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f663414e890 0x7f66340ac450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f662bfff640 1 -- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66200039a0 msgr2=0x7f66341f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f662bfff640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66200039a0 0x7f66341f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.209+0000 7f662bfff640 1 -- 172.21.15.45:0/3620375058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66340cba90 con 0x7f66340cb2f0 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.210+0000 7f662bfff640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66340cb2f0 0x7f66340cc850 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f661c00c730 tx=0x7f661c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.210+0000 7f6630ff9640 1 -- 172.21.15.45:0/3620375058 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c0072d0 con 0x7f66340cb2f0 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.210+0000 7f6630ff9640 1 -- 172.21.15.45:0/3620375058 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c007450 con 0x7f66340cb2f0 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.210+0000 7f6630ff9640 1 -- 172.21.15.45:0/3620375058 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c0047e0 con 0x7f66340cb2f0 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.210+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66340cbd70 con 0x7f66340cb2f0 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.210+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663414ed20 con 0x7f66340cb2f0 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.211+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66340cc530 con 0x7f66340cb2f0 2023-11-28T22:08:50.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.212+0000 7f6630ff9640 1 -- 172.21.15.45:0/3620375058 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f661c004b50 con 0x7f66340cb2f0 2023-11-28T22:08:50.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.213+0000 7f6630ff9640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f661406af60 0x7f661406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:50.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.213+0000 7f6630ff9640 1 -- 172.21.15.45:0/3620375058 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f661c088fe0 con 0x7f66340cb2f0 2023-11-28T22:08:50.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.214+0000 7f66337fe640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f661406af60 0x7f661406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:50.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.215+0000 7f66337fe640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f661406af60 0x7f661406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6624033000 tx=0x7f6624009030 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:50.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.217+0000 7f6630ff9640 1 -- 172.21.15.45:0/3620375058 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f661c053d80 con 0x7f66340cb2f0 2023-11-28T22:08:50.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.450+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f663404e920 con 0x7f66340cb2f0 2023-11-28T22:08:50.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.452+0000 7f6630ff9640 1 -- 172.21.15.45:0/3620375058 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4928 (secure 0 0 0) 0x7f661c018070 con 0x7f66340cb2f0 2023-11-28T22:08:50.455 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:08:50.455 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8692,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3311227176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3311227176},{"type":"v1","addr":"172.21.15.190:6835","nonce":3311227176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8698,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3492680695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3492680695},{"type":"v1","addr":"172.21.15.45:6835","nonce":3492680695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":8701,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3937197634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3937197634},{"type":"v1","addr":"172.21.15.190:6837","nonce":3937197634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T22:08:43.777075+0000","modified":"2023-11-28T22:08:46.880928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10650":{"gid":10650,"name":"c","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6833/3094566588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3094566588},{"type":"v1","addr":"172.21.15.45:6833","nonce":3094566588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-28T22:08:50.455 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 101 2023-11-28T22:08:50.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f661406af60 msgr2=0x7f661406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:50.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f661406af60 0x7f661406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6624033000 tx=0x7f6624009030 comp rx=0 tx=0).stop 2023-11-28T22:08:50.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66340cb2f0 msgr2=0x7f66340cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:50.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66340cb2f0 0x7f66340cc850 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f661c00c730 tx=0x7f661c00cbf0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 shutdown_connections 2023-11-28T22:08:50.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f663414e890 0x7f66340ac450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f661406af60 0x7f661406d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66340cb2f0 0x7f66340cc850 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 --2- 172.21.15.45:0/3620375058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66200039a0 0x7f66341f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 >> 172.21.15.45:0/3620375058 conn(0x7f663406ac70 msgr2=0x7f6634070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 shutdown_connections 2023-11-28T22:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:50.455+0000 7f6638935640 1 -- 172.21.15.45:0/3620375058 wait complete. 2023-11-28T22:08:50.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T22:08:43.777075+0000', 'modified': '2023-11-28T22:08:46.880928+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10650}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10650': {'gid': 10650, 'name': 'c', 'rank': 0, 'incarnation': 97, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6833/3094566588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 3094566588}, {'type': 'v1', 'addr': '172.21.15.45:6833', 'nonce': 3094566588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T22:08:50.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T22:08:50.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T22:08:51.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.115+0000 7fed553cf640 1 Processor -- start 2023-11-28T22:08:51.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.115+0000 7fed553cf640 1 -- start start 2023-11-28T22:08:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.117+0000 7fed553cf640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fed50154040 0x7fed50154420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.117+0000 7fed553cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed5015a400 0x7fed5006e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.117+0000 7fed553cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5015ab70 con 0x7fed5006d7e0 2023-11-28T22:08:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.117+0000 7fed553cf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5015b410 con 0x7fed5015a400 2023-11-28T22:08:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.117+0000 7fed553cf640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed50154960 con 0x7fed50154040 2023-11-28T22:08:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.117+0000 7fed54bce640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed5015a400 0x7fed5006e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.117+0000 7fed54bce640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed5015a400 0x7fed5006e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59658/0 (socket says 172.21.15.45:59658) 2023-11-28T22:08:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.117+0000 7fed54bce640 1 -- 172.21.15.45:0/3953734493 learned_addr learned my addr 172.21.15.45:0/3953734493 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.118+0000 7fed4ffff640 1 --2- 172.21.15.45:0/3953734493 >> v2:172.21.15.190:3300/0 conn(0x7fed50154040 0x7fed50154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.118+0000 7fed54bce640 1 -- 172.21.15.45:0/3953734493 >> v2:172.21.15.190:3300/0 conn(0x7fed50154040 msgr2=0x7fed50154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.118+0000 7fed54bce640 1 --2- 172.21.15.45:0/3953734493 >> v2:172.21.15.190:3300/0 conn(0x7fed50154040 0x7fed50154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.118+0000 7fed54bce640 1 -- 172.21.15.45:0/3953734493 >> v1:172.21.15.190:6789/0 conn(0x7fed5006d7e0 legacy=0x7fed500ad6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.118+0000 7fed54bce640 1 -- 172.21.15.45:0/3953734493 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5015ae50 con 0x7fed5015a400 2023-11-28T22:08:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.119+0000 7fed54bce640 1 --2- 172.21.15.45:0/3953734493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5015a400 0x7fed5006e830 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fed5006d180 tx=0x7fed4802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3cddeb4327c37d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.119+0000 7fed4effd640 1 -- 172.21.15.45:0/3953734493 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed480317e0 con 0x7fed5015a400 2023-11-28T22:08:51.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.119+0000 7fed4effd640 1 -- 172.21.15.45:0/3953734493 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed48031940 con 0x7fed5015a400 2023-11-28T22:08:51.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.119+0000 7fed4effd640 1 -- 172.21.15.45:0/3953734493 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48031cd0 con 0x7fed5015a400 2023-11-28T22:08:51.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.120+0000 7fed553cf640 1 -- 172.21.15.45:0/3953734493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5015a400 msgr2=0x7fed5006e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.120+0000 7fed553cf640 1 --2- 172.21.15.45:0/3953734493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5015a400 0x7fed5006e830 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fed5006d180 tx=0x7fed4802b880 comp rx=0 tx=0).stop 2023-11-28T22:08:51.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.120+0000 7fed553cf640 1 -- 172.21.15.45:0/3953734493 shutdown_connections 2023-11-28T22:08:51.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.120+0000 7fed553cf640 1 --2- 172.21.15.45:0/3953734493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5015a400 0x7fed5006e830 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.120+0000 7fed553cf640 1 --2- 172.21.15.45:0/3953734493 >> v2:172.21.15.190:3300/0 conn(0x7fed50154040 0x7fed50154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.120+0000 7fed553cf640 1 -- 172.21.15.45:0/3953734493 >> 172.21.15.45:0/3953734493 conn(0x7fed500cfdd0 msgr2=0x7fed500d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:51.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.120+0000 7fed553cf640 1 -- 172.21.15.45:0/3953734493 shutdown_connections 2023-11-28T22:08:51.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.120+0000 7fed553cf640 1 -- 172.21.15.45:0/3953734493 wait complete. 2023-11-28T22:08:51.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.123+0000 7fed553cf640 1 Processor -- start 2023-11-28T22:08:51.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.123+0000 7fed553cf640 1 -- start start 2023-11-28T22:08:51.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.124+0000 7fed553cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5006d7e0 0x7fed501487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.124+0000 7fed553cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed50154040 0x7fed50150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.124+0000 7fed553cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed5015a400 0x7fed50149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed553cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed50149620 con 0x7fed5006d7e0 2023-11-28T22:08:51.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed553cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed50150b00 con 0x7fed50154040 2023-11-28T22:08:51.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed553cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed50150de0 con 0x7fed5015a400 2023-11-28T22:08:51.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5006d7e0 0x7fed501487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5006d7e0 0x7fed501487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59666/0 (socket says 172.21.15.45:59666) 2023-11-28T22:08:51.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4ffff640 1 -- 172.21.15.45:0/1236055543 learned_addr learned my addr 172.21.15.45:0/1236055543 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:51.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed50154040 0x7fed50150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4ffff640 1 -- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed5015a400 msgr2=0x7fed50149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4ffff640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed5015a400 0x7fed50149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4ffff640 1 -- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed50154040 msgr2=0x7fed50150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4ffff640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed50154040 0x7fed50150190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4ffff640 1 -- 172.21.15.45:0/1236055543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed480094d0 con 0x7fed5006d7e0 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.125+0000 7fed4ffff640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5006d7e0 0x7fed501487e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fed40007ef0 tx=0x7fed4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.126+0000 7fed4cff9640 1 -- 172.21.15.45:0/1236055543 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40014620 con 0x7fed5006d7e0 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.126+0000 7fed4cff9640 1 -- 172.21.15.45:0/1236055543 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed40014780 con 0x7fed5006d7e0 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.126+0000 7fed4cff9640 1 -- 172.21.15.45:0/1236055543 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40014b30 con 0x7fed5006d7e0 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.126+0000 7fed553cf640 1 -- 172.21.15.45:0/1236055543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed50149240 con 0x7fed5006d7e0 2023-11-28T22:08:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.126+0000 7fed553cf640 1 -- 172.21.15.45:0/1236055543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed501510c0 con 0x7fed5006d7e0 2023-11-28T22:08:51.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.128+0000 7fed4cff9640 1 -- 172.21.15.45:0/1236055543 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fed40014c90 con 0x7fed5006d7e0 2023-11-28T22:08:51.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.129+0000 7fed4cff9640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fed3c06af60 0x7fed3c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.129+0000 7fed4cff9640 1 -- 172.21.15.45:0/1236055543 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fed4008a2e0 con 0x7fed5006d7e0 2023-11-28T22:08:51.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.130+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fed3c06af60 0x7fed3c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.130+0000 7fed591c9740 1 Processor -- start 2023-11-28T22:08:51.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.130+0000 7fed591c9740 1 -- start start 2023-11-28T22:08:51.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed591c9740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 0x55dc26a69450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed591c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a63d70 0x55dc26a68270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fed3c06af60 0x7fed3c06d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fed5015b7e0 tx=0x7fed44009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed591c9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26aac730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed591c9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc269c7880 con 0x55dc26a687b0 2023-11-28T22:08:51.134 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed591c9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc269d6ae0 con 0x55dc26a63d70 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed591c9740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc269d41d0 con 0x55dc26a69070 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed54bce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26aac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed54bce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26aac730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59672/0 (socket says 172.21.15.45:59672) 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.131+0000 7fed54bce640 1 -- 172.21.15.45:0/2169004289 learned_addr learned my addr 172.21.15.45:0/2169004289 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.132+0000 7fed4ffff640 1 --2- 172.21.15.45:0/2169004289 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 0x55dc26a69450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.132+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/2169004289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a63d70 0x55dc26a68270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.132+0000 7fed54bce640 1 -- 172.21.15.45:0/2169004289 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 msgr2=0x55dc26a69450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.132+0000 7fed54bce640 1 --2- 172.21.15.45:0/2169004289 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 0x55dc26a69450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.132+0000 7fed54bce640 1 -- 172.21.15.45:0/2169004289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a63d70 msgr2=0x55dc26a68270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.132+0000 7fed54bce640 1 --2- 172.21.15.45:0/2169004289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a63d70 0x55dc26a68270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.132+0000 7fed54bce640 1 -- 172.21.15.45:0/2169004289 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc26997860 con 0x55dc26a687b0 2023-11-28T22:08:51.135 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.132+0000 7fed54bce640 1 --2- 172.21.15.45:0/2169004289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26aac730 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fed48009960 tx=0x7fed4802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaacf346567e2224 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.132+0000 7fed35ffb640 1 -- 172.21.15.45:0/2169004289 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48031ab0 con 0x55dc26a687b0 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed35ffb640 1 -- 172.21.15.45:0/2169004289 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed48031c10 con 0x55dc26a687b0 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed35ffb640 1 -- 172.21.15.45:0/2169004289 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48032470 con 0x55dc26a687b0 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed591c9740 1 -- 172.21.15.45:0/2169004289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 msgr2=0x55dc26aac730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed591c9740 1 --2- 172.21.15.45:0/2169004289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26aac730 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fed48009960 tx=0x7fed4802e000 comp rx=0 tx=0).stop 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed591c9740 1 -- 172.21.15.45:0/2169004289 shutdown_connections 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed591c9740 1 --2- 172.21.15.45:0/2169004289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26aac730 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed591c9740 1 --2- 172.21.15.45:0/2169004289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a63d70 0x55dc26a68270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed591c9740 1 --2- 172.21.15.45:0/2169004289 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 0x55dc26a69450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed591c9740 1 -- 172.21.15.45:0/2169004289 >> 172.21.15.45:0/2169004289 conn(0x55dc26a2ac70 msgr2=0x55dc26aaeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed591c9740 1 -- 172.21.15.45:0/2169004289 shutdown_connections 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.133+0000 7fed591c9740 1 -- 172.21.15.45:0/2169004289 wait complete. 2023-11-28T22:08:51.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.134+0000 7fed591c9740 1 -- start start 2023-11-28T22:08:51.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.134+0000 7fed591c9740 1 Processor -- start 2023-11-28T22:08:51.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.135+0000 7fed591c9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a63d70 0x55dc26ab5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.135+0000 7fed591c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 0x55dc26ab6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.135+0000 7fed591c9740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 0x55dc26ab3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.135+0000 7fed591c9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc269cfac0 con 0x55dc26a63d70 2023-11-28T22:08:51.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.135+0000 7fed591c9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc2696c2f0 con 0x55dc26a687b0 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.135+0000 7fed591c9740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc26989c70 con 0x55dc26a69070 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.135+0000 7fed591c9740 10 client.0 _ll_register_callbacks cb 0x55dc2698e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a63d70 0x55dc26ab5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a63d70 0x55dc26ab5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59680/0 (socket says 172.21.15.45:59680) 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4ffff640 1 -- 172.21.15.45:0/211789210 learned_addr learned my addr 172.21.15.45:0/211789210 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 0x55dc26ab6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed54bce640 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 0x55dc26ab3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4ffff640 1 -- 172.21.15.45:0/211789210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 msgr2=0x55dc26ab3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4ffff640 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 0x55dc26ab3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4ffff640 1 -- 172.21.15.45:0/211789210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 msgr2=0x55dc26ab6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4ffff640 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 0x55dc26ab6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4ffff640 1 -- 172.21.15.45:0/211789210 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5015ae50 con 0x55dc26a63d70 2023-11-28T22:08:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 0x55dc26ab6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed4ffff640 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a63d70 0x55dc26ab5d00 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fed4000e9b0 tx=0x7fed40058820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7b719752cfb91c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.136+0000 7fed2effd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed2effd640 1 -- 172.21.15.45:0/211789210 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed400610d0 con 0x55dc26a63d70 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed2effd640 1 -- 172.21.15.45:0/211789210 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fed400554e0 con 0x55dc26a63d70 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed591c9740 1 -- 172.21.15.45:0/211789210 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dc269c8aa0 con 0x55dc26a63d70 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed2effd640 1 -- 172.21.15.45:0/211789210 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed400556a0 con 0x55dc26a63d70 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed591c9740 10 client.10779 fetch_fsmap learned FSMap version 101 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed591c9740 1 -- 172.21.15.45:0/211789210 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x55dc269ce140 con 0x55dc26a63d70 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed2effd640 1 -- 172.21.15.45:0/211789210 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed40074020 con 0x55dc26a63d70 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed2effd640 1 -- 172.21.15.45:0/211789210 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 101) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fed40059030 con 0x55dc26a63d70 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed591c9740 10 client.10779 fetch_fsmap finished waiting for FSMap version 101 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed591c9740 10 client.10779 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed591c9740 20 client.10779 populate_metadata read hostname 'smithi045' 2023-11-28T22:08:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed591c9740 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] conn(0x55dc26a8b950 0x55dc26acbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed591c9740 4 client.10779 mds_command: new command op to 10650 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T22:08:51.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.137+0000 7fed591c9740 1 -- 172.21.15.45:0/211789210 --> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dc269c87c0 con 0x55dc26a8b950 2023-11-28T22:08:51.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.138+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] conn(0x55dc26a8b950 0x55dc26acbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.138+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] conn(0x55dc26a8b950 0x55dc26acbd70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efc1aff16b093860 server_cookie=4c6ec4d6c6bffe80 in_seq=0 out_seq=0 2023-11-28T22:08:51.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.138+0000 7fed2effd640 10 client.10779 ms_handle_connect on v2:172.21.15.45:6832/3094566588 2023-11-28T22:08:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed2effd640 1 -- 172.21.15.45:0/211789210 <== mds.0 v2:172.21.15.45:6832/3094566588 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dc269c87c0 con 0x55dc26a8b950 2023-11-28T22:08:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed2effd640 10 client.10779 handle_command_reply: tid=0 2023-11-28T22:08:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed591c9740 1 client.10779 shutdown 2023-11-28T22:08:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed591c9740 1 -- 172.21.15.45:0/211789210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a63d70 msgr2=0x55dc26ab5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed591c9740 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a63d70 0x55dc26ab5d00 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fed4000e9b0 tx=0x7fed40058820 comp rx=0 tx=0).stop 2023-11-28T22:08:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed591c9740 1 -- 172.21.15.45:0/211789210 shutdown_connections 2023-11-28T22:08:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed591c9740 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] conn(0x55dc26a8b950 0x55dc26acbd70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed591c9740 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a69070 0x55dc26ab3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed591c9740 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 0x55dc26ab6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed2effd640 0 client.10779 ms_handle_reset on v2:172.21.15.45:6832/3094566588 2023-11-28T22:08:51.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed591c9740 1 --2- 172.21.15.45:0/211789210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a63d70 0x55dc26ab5d00 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.139+0000 7fed591c9740 1 -- 172.21.15.45:0/211789210 >> 172.21.15.45:0/211789210 conn(0x55dc26a2ac70 msgr2=0x55dc26a62dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:51.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.140+0000 7fed591c9740 1 -- 172.21.15.45:0/211789210 shutdown_connections 2023-11-28T22:08:51.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.140+0000 7fed591c9740 1 -- 172.21.15.45:0/211789210 wait complete. 2023-11-28T22:08:51.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.140+0000 7fed591c9740 20 client.10779 trim_cache size 0 max 16384 2023-11-28T22:08:51.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.156+0000 7fed591c9740 1 Processor -- start 2023-11-28T22:08:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.156+0000 7fed591c9740 1 -- start start 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.157+0000 7fed591c9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26a62dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.157+0000 7fed591c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a69070 0x55dc26a63310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.157+0000 7fed591c9740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a8ab40 0x55dc26ab3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.157+0000 7fed591c9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc26732f20 con 0x55dc26a687b0 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.157+0000 7fed591c9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc26a662a0 con 0x55dc26a69070 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.157+0000 7fed591c9740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc26a8a2b0 con 0x55dc26a8ab40 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.157+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26a62dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.157+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26a62dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59682/0 (socket says 172.21.15.45:59682) 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.157+0000 7fed4ffff640 1 -- 172.21.15.45:0/4182918074 learned_addr learned my addr 172.21.15.45:0/4182918074 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.158+0000 7fed4ffff640 1 -- 172.21.15.45:0/4182918074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a8ab40 msgr2=0x55dc26ab3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.158+0000 7fed4ffff640 1 --2- 172.21.15.45:0/4182918074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a8ab40 0x55dc26ab3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.158+0000 7fed4ffff640 1 -- 172.21.15.45:0/4182918074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a69070 msgr2=0x55dc26a63310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.158+0000 7fed4ffff640 1 --2- 172.21.15.45:0/4182918074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a69070 0x55dc26a63310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.158+0000 7fed4ffff640 1 -- 172.21.15.45:0/4182918074 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc2696bb80 con 0x55dc26a687b0 2023-11-28T22:08:51.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.158+0000 7fed4ffff640 1 --2- 172.21.15.45:0/4182918074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26a62dd0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fed40058d60 tx=0x7fed4001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e889aa13a9fc558 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.158+0000 7fed367fc640 1 -- 172.21.15.45:0/4182918074 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed400610d0 con 0x55dc26a687b0 2023-11-28T22:08:51.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.158+0000 7fed367fc640 1 -- 172.21.15.45:0/4182918074 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed40074070 con 0x55dc26a687b0 2023-11-28T22:08:51.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.158+0000 7fed367fc640 1 -- 172.21.15.45:0/4182918074 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40058910 con 0x55dc26a687b0 2023-11-28T22:08:51.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.159+0000 7fed591c9740 1 -- 172.21.15.45:0/4182918074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 msgr2=0x55dc26a62dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.159+0000 7fed591c9740 1 --2- 172.21.15.45:0/4182918074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26a62dd0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fed40058d60 tx=0x7fed4001f000 comp rx=0 tx=0).stop 2023-11-28T22:08:51.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.159+0000 7fed591c9740 1 -- 172.21.15.45:0/4182918074 shutdown_connections 2023-11-28T22:08:51.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.159+0000 7fed591c9740 1 --2- 172.21.15.45:0/4182918074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a8ab40 0x55dc26ab3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.159+0000 7fed591c9740 1 --2- 172.21.15.45:0/4182918074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a69070 0x55dc26a63310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.159+0000 7fed591c9740 1 --2- 172.21.15.45:0/4182918074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a687b0 0x55dc26a62dd0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.159+0000 7fed591c9740 1 -- 172.21.15.45:0/4182918074 >> 172.21.15.45:0/4182918074 conn(0x55dc26a2ac70 msgr2=0x55dc26a5f550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.159+0000 7fed591c9740 1 -- 172.21.15.45:0/4182918074 shutdown_connections 2023-11-28T22:08:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.159+0000 7fed591c9740 1 -- 172.21.15.45:0/4182918074 wait complete. 2023-11-28T22:08:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.160+0000 7fed591c9740 1 -- start start 2023-11-28T22:08:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.160+0000 7fed591c9740 1 Processor -- start 2023-11-28T22:08:51.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed591c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 0x55dc26ad0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed591c9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a69070 0x55dc26b1b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed591c9740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a8ab40 0x55dc26b1bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed591c9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc26ab06a0 con 0x55dc26a69070 2023-11-28T22:08:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed591c9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc26a88630 con 0x55dc26a687b0 2023-11-28T22:08:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed591c9740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc26ab0b90 con 0x55dc26a8ab40 2023-11-28T22:08:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed591c9740 10 client.0 _ll_register_callbacks cb 0x7fed4000ef30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T22:08:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed4f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a69070 0x55dc26b1b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed4f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a69070 0x55dc26b1b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59696/0 (socket says 172.21.15.45:59696) 2023-11-28T22:08:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed4f7fe640 1 -- 172.21.15.45:0/3620910853 learned_addr learned my addr 172.21.15.45:0/3620910853 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed4f7fe640 1 -- 172.21.15.45:0/3620910853 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a8ab40 msgr2=0x55dc26b1bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a8ab40 0x55dc26b1bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed4f7fe640 1 -- 172.21.15.45:0/3620910853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 msgr2=0x55dc26ad0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 0x55dc26ad0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.161+0000 7fed4f7fe640 1 -- 172.21.15.45:0/3620910853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc269ce140 con 0x55dc26a69070 2023-11-28T22:08:51.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.162+0000 7fed4f7fe640 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a69070 0x55dc26b1b9f0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fed500af0e0 tx=0x7fed44003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b9e45ed19b1706 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.162+0000 7fed2effd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2023-11-28T22:08:51.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.162+0000 7fed2effd640 1 -- 172.21.15.45:0/3620910853 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44023330 con 0x55dc26a69070 2023-11-28T22:08:51.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.162+0000 7fed2effd640 1 -- 172.21.15.45:0/3620910853 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fed44023490 con 0x55dc26a69070 2023-11-28T22:08:51.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.162+0000 7fed591c9740 1 -- 172.21.15.45:0/3620910853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed48031c10 con 0x55dc26a69070 2023-11-28T22:08:51.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.162+0000 7fed2effd640 1 -- 172.21.15.45:0/3620910853 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed44023690 con 0x55dc26a69070 2023-11-28T22:08:51.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.162+0000 7fed591c9740 10 client.10785 fetch_fsmap learned FSMap version 101 2023-11-28T22:08:51.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.162+0000 7fed591c9740 1 -- 172.21.15.45:0/3620910853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x55dc26991170 con 0x55dc26a69070 2023-11-28T22:08:51.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.163+0000 7fed2effd640 1 -- 172.21.15.45:0/3620910853 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed48031c10 con 0x55dc26a69070 2023-11-28T22:08:51.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.163+0000 7fed2effd640 1 -- 172.21.15.45:0/3620910853 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 101) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fed4400d890 con 0x55dc26a69070 2023-11-28T22:08:51.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.163+0000 7fed591c9740 10 client.10785 fetch_fsmap finished waiting for FSMap version 101 2023-11-28T22:08:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.163+0000 7fed591c9740 10 client.10785 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T22:08:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.163+0000 7fed591c9740 20 client.10785 populate_metadata read hostname 'smithi045' 2023-11-28T22:08:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.163+0000 7fed591c9740 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] conn(0x55dc26a878d0 0x55dc26a87cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.163+0000 7fed591c9740 4 client.10785 mds_command: new command op to 10650 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T22:08:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.163+0000 7fed591c9740 1 -- 172.21.15.45:0/3620910853 --> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55dc266d06c0 con 0x55dc26a878d0 2023-11-28T22:08:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.163+0000 7fed4ffff640 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] conn(0x55dc26a878d0 0x55dc26a87cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed4ffff640 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] conn(0x55dc26a878d0 0x55dc26a87cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d6ee336a054eddf server_cookie=d6aa92b45664325c in_seq=0 out_seq=0 2023-11-28T22:08:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed2effd640 10 client.10785 ms_handle_connect on v2:172.21.15.45:6832/3094566588 2023-11-28T22:08:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed2effd640 1 -- 172.21.15.45:0/3620910853 <== mds.0 v2:172.21.15.45:6832/3094566588 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x55dc266d06c0 con 0x55dc26a878d0 2023-11-28T22:08:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed2effd640 10 client.10785 handle_command_reply: tid=0 2023-11-28T22:08:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 client.10785 shutdown 2023-11-28T22:08:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 -- 172.21.15.45:0/3620910853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a69070 msgr2=0x55dc26b1b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a69070 0x55dc26b1b9f0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fed500af0e0 tx=0x7fed44003760 comp rx=0 tx=0).stop 2023-11-28T22:08:51.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 -- 172.21.15.45:0/3620910853 shutdown_connections 2023-11-28T22:08:51.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.45:6832/3094566588,v1:172.21.15.45:6833/3094566588] conn(0x55dc26a878d0 0x55dc26a87cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55dc26a8ab40 0x55dc26b1bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc26a69070 0x55dc26b1b9f0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 --2- 172.21.15.45:0/3620910853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dc26a687b0 0x55dc26ad0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed2effd640 0 client.10785 ms_handle_reset on v2:172.21.15.45:6832/3094566588 2023-11-28T22:08:51.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 -- 172.21.15.45:0/3620910853 >> 172.21.15.45:0/3620910853 conn(0x55dc26a2ac70 msgr2=0x55dc26aaeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:51.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 -- 172.21.15.45:0/3620910853 shutdown_connections 2023-11-28T22:08:51.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 1 -- 172.21.15.45:0/3620910853 wait complete. 2023-11-28T22:08:51.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.164+0000 7fed591c9740 20 client.10785 trim_cache size 0 max 16384 2023-11-28T22:08:51.177 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:08:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "ef8eac8d-57a5-4d94-ab39-848e2a7bfeef", 2023-11-28T22:08:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2023-11-28T22:08:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 10650, 2023-11-28T22:08:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2023-11-28T22:08:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2023-11-28T22:08:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2023-11-28T22:08:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 6.8140828559999997, 2023-11-28T22:08:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 101, 2023-11-28T22:08:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 215, 2023-11-28T22:08:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 215, 2023-11-28T22:08:51.179 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 12.798383498 2023-11-28T22:08:51.179 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:08:51.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 -- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fed3c06af60 msgr2=0x7fed3c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fed3c06af60 0x7fed3c06d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fed5015b7e0 tx=0x7fed44009040 comp rx=0 tx=0).stop 2023-11-28T22:08:51.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 -- 172.21.15.45:0/1236055543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5006d7e0 msgr2=0x7fed501487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5006d7e0 0x7fed501487e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fed40007ef0 tx=0x7fed4000e4b0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 -- 172.21.15.45:0/1236055543 shutdown_connections 2023-11-28T22:08:51.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fed5015a400 0x7fed50149a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed50154040 0x7fed50150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fed3c06af60 0x7fed3c06d420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 --2- 172.21.15.45:0/1236055543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed5006d7e0 0x7fed501487e0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 -- 172.21.15.45:0/1236055543 >> 172.21.15.45:0/1236055543 conn(0x7fed500cfdd0 msgr2=0x7fed500ae940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:51.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 -- 172.21.15.45:0/1236055543 shutdown_connections 2023-11-28T22:08:51.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.165+0000 7fed553cf640 1 -- 172.21.15.45:0/1236055543 wait complete. 2023-11-28T22:08:51.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T22:08:51.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:08:51.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:08:51.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.406+0000 7fcc8cd88640 1 Processor -- start 2023-11-28T22:08:51.409 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.407+0000 7fcc8cd88640 1 -- start start 2023-11-28T22:08:51.411 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.408+0000 7fcc8cd88640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcc880b3120 0x7fcc880b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.408+0000 7fcc8cd88640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc8806b510 0x7fcc8806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.408+0000 7fcc8cd88640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc880b3d50 con 0x7fcc8815a1f0 2023-11-28T22:08:51.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.408+0000 7fcc8cd88640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc880af7a0 con 0x7fcc880b3120 2023-11-28T22:08:51.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.408+0000 7fcc8cd88640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8806bee0 con 0x7fcc8806b510 2023-11-28T22:08:51.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.408+0000 7fcc877fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcc8815a1f0 0x7fcc880ae7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48272/0 (socket says 172.21.15.45:48272) 2023-11-28T22:08:51.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.408+0000 7fcc877fe640 1 -- 172.21.15.45:0/2826370236 learned_addr learned my addr 172.21.15.45:0/2826370236 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:51.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.408+0000 7fcc7ffff640 1 --2- 172.21.15.45:0/2826370236 >> v2:172.21.15.45:3300/0 conn(0x7fcc880b3120 0x7fcc880b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.409+0000 7fcc87fff640 1 --2- 172.21.15.45:0/2826370236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc8806b510 0x7fcc8806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.409+0000 7fcc7ffff640 1 -- 172.21.15.45:0/2826370236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc8806b510 msgr2=0x7fcc8806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:51.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.409+0000 7fcc7ffff640 1 --2- 172.21.15.45:0/2826370236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc8806b510 0x7fcc8806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.409+0000 7fcc7ffff640 1 -- 172.21.15.45:0/2826370236 >> v1:172.21.15.45:6789/0 conn(0x7fcc8815a1f0 legacy=0x7fcc880ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.409+0000 7fcc7ffff640 1 -- 172.21.15.45:0/2826370236 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc880ad790 con 0x7fcc880b3120 2023-11-28T22:08:51.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.410+0000 7fcc7ffff640 1 --2- 172.21.15.45:0/2826370236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc880b3120 0x7fcc880b3670 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fcc700097a0 tx=0x7fcc7002dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42be7fbcc66fad0c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.410+0000 7fcc86ffd640 1 -- 172.21.15.45:0/2826370236 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7002f880 con 0x7fcc880b3120 2023-11-28T22:08:51.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.410+0000 7fcc86ffd640 1 -- 172.21.15.45:0/2826370236 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7002f9e0 con 0x7fcc880b3120 2023-11-28T22:08:51.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.411+0000 7fcc86ffd640 1 -- 172.21.15.45:0/2826370236 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7002fd70 con 0x7fcc880b3120 2023-11-28T22:08:51.417 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.411+0000 7fcc8cd88640 1 -- 172.21.15.45:0/2826370236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc880b3120 msgr2=0x7fcc880b3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.411+0000 7fcc8cd88640 1 --2- 172.21.15.45:0/2826370236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc880b3120 0x7fcc880b3670 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fcc700097a0 tx=0x7fcc7002dac0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.411+0000 7fcc8cd88640 1 -- 172.21.15.45:0/2826370236 shutdown_connections 2023-11-28T22:08:51.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.411+0000 7fcc8cd88640 1 --2- 172.21.15.45:0/2826370236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc8806b510 0x7fcc8806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.411+0000 7fcc8cd88640 1 --2- 172.21.15.45:0/2826370236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc880b3120 0x7fcc880b3670 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.411+0000 7fcc8cd88640 1 -- 172.21.15.45:0/2826370236 >> 172.21.15.45:0/2826370236 conn(0x7fcc880712d0 msgr2=0x7fcc880716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:51.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.411+0000 7fcc8cd88640 1 -- 172.21.15.45:0/2826370236 shutdown_connections 2023-11-28T22:08:51.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.412+0000 7fcc8cd88640 1 -- 172.21.15.45:0/2826370236 wait complete. 2023-11-28T22:08:51.418 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.413+0000 7fcc8cd88640 1 Processor -- start 2023-11-28T22:08:51.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.414+0000 7fcc8cd88640 1 -- start start 2023-11-28T22:08:51.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc8cd88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8806b510 0x7fcc88154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc8cd88640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc880b3120 0x7fcc88148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.421 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc8cd88640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc88151c40 0x7fcc880afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc8cd88640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc88151260 con 0x7fcc8806b510 2023-11-28T22:08:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc8cd88640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc881551b0 con 0x7fcc880b3120 2023-11-28T22:08:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc8cd88640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc881503b0 con 0x7fcc88151c40 2023-11-28T22:08:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc877fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8806b510 0x7fcc88154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc877fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8806b510 0x7fcc88154c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59706/0 (socket says 172.21.15.45:59706) 2023-11-28T22:08:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc877fe640 1 -- 172.21.15.45:0/3601890869 learned_addr learned my addr 172.21.15.45:0/3601890869 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc7ffff640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc880b3120 0x7fcc88148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc87fff640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc88151c40 0x7fcc880afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc877fe640 1 -- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc88151c40 msgr2=0x7fcc880afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc877fe640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc88151c40 0x7fcc880afa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc877fe640 1 -- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc880b3120 msgr2=0x7fcc88148970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc877fe640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc880b3120 0x7fcc88148970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.415+0000 7fcc877fe640 1 -- 172.21.15.45:0/3601890869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc70009400 con 0x7fcc8806b510 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.416+0000 7fcc877fe640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8806b510 0x7fcc88154c70 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c480 tx=0x7fcc7800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.416+0000 7fcc84ff9640 1 -- 172.21.15.45:0/3601890869 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78005ce0 con 0x7fcc8806b510 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.416+0000 7fcc84ff9640 1 -- 172.21.15.45:0/3601890869 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78005e40 con 0x7fcc8806b510 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.416+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc88148eb0 con 0x7fcc8806b510 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.416+0000 7fcc84ff9640 1 -- 172.21.15.45:0/3601890869 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc780072f0 con 0x7fcc8806b510 2023-11-28T22:08:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.416+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc88150930 con 0x7fcc8806b510 2023-11-28T22:08:51.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.417+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc880cf350 con 0x7fcc8806b510 2023-11-28T22:08:51.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.418+0000 7fcc84ff9640 1 -- 172.21.15.45:0/3601890869 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fcc78007450 con 0x7fcc8806b510 2023-11-28T22:08:51.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.419+0000 7fcc84ff9640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fcc6806af90 0x7fcc6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:51.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.419+0000 7fcc84ff9640 1 -- 172.21.15.45:0/3601890869 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcc78088c40 con 0x7fcc8806b510 2023-11-28T22:08:51.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.420+0000 7fcc7ffff640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fcc6806af90 0x7fcc6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:51.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.421+0000 7fcc7ffff640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fcc6806af90 0x7fcc6806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcc70038000 tx=0x7fcc7003b040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:51.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.424+0000 7fcc84ff9640 1 -- 172.21.15.45:0/3601890869 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc78053b20 con 0x7fcc8806b510 2023-11-28T22:08:51.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.664+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcc880b3d50 con 0x7fcc8806b510 2023-11-28T22:08:51.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.665+0000 7fcc84ff9640 1 -- 172.21.15.45:0/3601890869 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v101) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcc880b3d50 con 0x7fcc8806b510 2023-11-28T22:08:51.668 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T22:08:51.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fcc6806af90 msgr2=0x7fcc6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fcc6806af90 0x7fcc6806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcc70038000 tx=0x7fcc7003b040 comp rx=0 tx=0).stop 2023-11-28T22:08:51.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8806b510 msgr2=0x7fcc88154c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:51.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8806b510 0x7fcc88154c70 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c480 tx=0x7fcc7800c940 comp rx=0 tx=0).stop 2023-11-28T22:08:51.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 shutdown_connections 2023-11-28T22:08:51.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcc88151c40 0x7fcc880afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc880b3120 0x7fcc88148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fcc6806af90 0x7fcc6806d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 --2- 172.21.15.45:0/3601890869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc8806b510 0x7fcc88154c70 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:51.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 >> 172.21.15.45:0/3601890869 conn(0x7fcc880712d0 msgr2=0x7fcc880b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:51.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 shutdown_connections 2023-11-28T22:08:51.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:51.668+0000 7fcc8cd88640 1 -- 172.21.15.45:0/3601890869 wait complete. 2023-11-28T22:08:51.690 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T22:08:51.690 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T22:08:51.690 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-11-28T22:08:51.691 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T22:08:51.691 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T22:08:51.691 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T22:08:51.691 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T22:08:51.691 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T22:08:51.691 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T22:08:51.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:51.691 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-11-28T22:08:51.746 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T22:08:51.746 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T22:08:51.746 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T22:08:51.746 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T22:08:51.746 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-11-28T22:08:51.746 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T22:08:51.746 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 36611sec preferred_lft 36611sec 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-11-28T22:08:51.747 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::3090:dfff:fe65:1ee6/64 scope link 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T22:08:51.748 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 6a:6c:75:a3:8f:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T22:08:51.749 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::686c:75ff:fea3:8fbb/64 scope link 2023-11-28T22:08:51.749 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T22:08:51.749 INFO:teuthology.orchestra.run.smithi045.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T22:08:51.749 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether c6:1d:65:1f:74:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T22:08:51.749 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::c41d:65ff:fe1f:74f7/64 scope link 2023-11-28T22:08:51.749 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-11-28T22:08:51.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:51.750 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T22:08:51.847 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-11-28T22:08:51.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:51.850 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-11-28T22:08:51.910 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-11-28T22:08:51.910 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T22:08:51.910 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-28T22:08:51.910 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-28T22:08:51.910 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T22:08:51.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:51.911 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-11-28T22:08:51.911 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-11-28T22:08:51.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-28T22:08:51.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-28T22:08:51.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-28T22:08:51.911 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-11-28T22:08:52.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:52.116 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T22:08:52.173 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T22:08:52.174 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T22:08:52.174 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T22:08:52.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:52.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-11-28T22:08:52.279 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T22:08:52.279 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T22:08:52.338 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T22:08:52.338 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T22:08:52.339 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T22:08:52.396 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T22:08:52.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:52.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T22:08:52.544 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T22:08:52.545 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T22:08:52.545 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T22:08:52.568 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T22:08:52.568 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T22:08:52.568 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T22:08:52.568 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T22:08:52.569 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T22:08:52.594 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T22:08:52.594 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T22:08:52.594 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-11-28T22:08:52.594 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.190:3300,172.21.15.190:3301" 2023-11-28T22:08:52.595 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T22:08:52.595 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790" 2023-11-28T22:08:52.595 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2023-11-28T22:08:52.595 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ef8eac8d-57a5-4d94-ab39-848e2a7bfeef" will pass to kernel 2023-11-28T22:08:52.596 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-11-28T22:08:52.596 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-11-28T22:08:52.596 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-11-28T22:08:52.596 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-11-28T22:08:52.598 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-11-28T22:08:52.599 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T22:08:52.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:52.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T22:08:52.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:52.694 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T22:08:52.764 INFO:teuthology.orchestra.run.smithi045.stdout:client10797 2023-11-28T22:08:52.765 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:08:52.765 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:08:52.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-28T22:08:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.050+0000 7f4c56d39640 1 Processor -- start 2023-11-28T22:08:53.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.050+0000 7f4c56d39640 1 -- start start 2023-11-28T22:08:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.051+0000 7f4c56d39640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4c50143a60 0x7f4c5006c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.052+0000 7f4c56d39640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500cf820 0x7f4c500cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.052+0000 7f4c56d39640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c500cd1f0 con 0x7f4c5006c820 2023-11-28T22:08:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.052+0000 7f4c56d39640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c5006cda0 con 0x7f4c50143a60 2023-11-28T22:08:53.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.052+0000 7f4c56d39640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5006d080 con 0x7f4c500cf820 2023-11-28T22:08:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.052+0000 7f4c4ffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4c5006c820 0x7f4c500cf110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48278/0 (socket says 172.21.15.45:48278) 2023-11-28T22:08:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.052+0000 7f4c4ffff640 1 -- 172.21.15.45:0/2221042353 learned_addr learned my addr 172.21.15.45:0/2221042353 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.052+0000 7f4c552d6640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500cf820 0x7f4c500cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.052+0000 7f4c54ad5640 1 --2- 172.21.15.45:0/2221042353 >> v2:172.21.15.190:3300/0 conn(0x7f4c50143a60 0x7f4c5006c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769309403 0 0) 0x7f4c500cd1f0 con 0x7f4c5006c820 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c38003520 con 0x7f4c5006c820 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1274861703 0 0) 0x7f4c38003520 con 0x7f4c5006c820 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500cf820 msgr2=0x7f4c500cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500cf820 0x7f4c500cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 >> v2:172.21.15.190:3300/0 conn(0x7f4c50143a60 msgr2=0x7f4c5006c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 --2- 172.21.15.45:0/2221042353 >> v2:172.21.15.190:3300/0 conn(0x7f4c50143a60 0x7f4c5006c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c500cce80 con 0x7f4c5006c820 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4c40001670 con 0x7f4c5006c820 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 >> v1:172.21.15.45:6789/0 conn(0x7f4c5006c820 legacy=0x7f4c500cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c38003970 0x7f4c38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500cf820 0x7f4c380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5006cda0 con 0x7f4c38004d00 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5006d080 con 0x7f4c38003970 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.053+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3800db90 con 0x7f4c500cf820 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.054+0000 7f4c552d6640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.054+0000 7f4c54ad5640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c38003970 0x7f4c38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.054+0000 7f4c4ffff640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500cf820 0x7f4c380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.054+0000 7f4c552d6640 1 -- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500cf820 msgr2=0x7f4c380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.054+0000 7f4c552d6640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500cf820 0x7f4c380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.054+0000 7f4c552d6640 1 -- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c38003970 msgr2=0x7f4c38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.054+0000 7f4c552d6640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c38003970 0x7f4c38003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.058 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.054+0000 7f4c552d6640 1 -- 172.21.15.45:0/2221042353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38003520 con 0x7f4c38004d00 2023-11-28T22:08:53.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.055+0000 7f4c552d6640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c38005180 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f4c48009fc0 tx=0x7f4c380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c89d671c8ad91d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:53.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.056+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4802cbd0 con 0x7f4c38004d00 2023-11-28T22:08:53.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.056+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4800f040 con 0x7f4c38004d00 2023-11-28T22:08:53.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.056+0000 7f4c4f7fe640 1 -- 172.21.15.45:0/2221042353 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48034440 con 0x7f4c38004d00 2023-11-28T22:08:53.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.056+0000 7f4c56d39640 1 -- 172.21.15.45:0/2221042353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 msgr2=0x7f4c38005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.056+0000 7f4c56d39640 1 --2- 172.21.15.45:0/2221042353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c38005180 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f4c48009fc0 tx=0x7f4c380057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.056+0000 7f4c552d6640 1 -- 172.21.15.45:0/2221042353 reap_dead start 2023-11-28T22:08:53.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.057+0000 7f4c56d39640 1 -- 172.21.15.45:0/2221042353 shutdown_connections 2023-11-28T22:08:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.057+0000 7f4c56d39640 1 -- 172.21.15.45:0/2221042353 >> 172.21.15.45:0/2221042353 conn(0x7f4c5006a6e0 msgr2=0x7f4c5006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.057+0000 7f4c56d39640 1 -- 172.21.15.45:0/2221042353 shutdown_connections 2023-11-28T22:08:53.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.057+0000 7f4c56d39640 1 -- 172.21.15.45:0/2221042353 wait complete. 2023-11-28T22:08:53.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.059+0000 7f4c56d39640 1 Processor -- start 2023-11-28T22:08:53.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.059+0000 7f4c56d39640 1 -- start start 2023-11-28T22:08:53.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.060+0000 7f4c56d39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c5013fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.060+0000 7f4c56d39640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c500c4480 0x7f4c500c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.060+0000 7f4c56d39640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500adc80 0x7f4c500ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.060+0000 7f4c56d39640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500c4e40 con 0x7f4c38004d00 2023-11-28T22:08:53.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.060+0000 7f4c56d39640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500ae990 con 0x7f4c500c4480 2023-11-28T22:08:53.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.060+0000 7f4c56d39640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500aec70 con 0x7f4c500adc80 2023-11-28T22:08:53.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.060+0000 7f4c54ad5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c5013fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.060+0000 7f4c54ad5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c5013fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59732/0 (socket says 172.21.15.45:59732) 2023-11-28T22:08:53.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.060+0000 7f4c54ad5640 1 -- 172.21.15.45:0/983853469 learned_addr learned my addr 172.21.15.45:0/983853469 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:53.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.061+0000 7f4c4ffff640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c500c4480 0x7f4c500c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.061+0000 7f4c552d6640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500adc80 0x7f4c500ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.061+0000 7f4c54ad5640 1 -- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500adc80 msgr2=0x7f4c500ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.061+0000 7f4c54ad5640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500adc80 0x7f4c500ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.061+0000 7f4c54ad5640 1 -- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c500c4480 msgr2=0x7f4c500c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.061+0000 7f4c54ad5640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c500c4480 0x7f4c500c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.061+0000 7f4c54ad5640 1 -- 172.21.15.45:0/983853469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c500aef50 con 0x7f4c38004d00 2023-11-28T22:08:53.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.061+0000 7f4c54ad5640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c5013fa80 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f4c44011600 tx=0x7f4c44011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:53.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.062+0000 7f4c4d7fa640 1 -- 172.21.15.45:0/983853469 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4400f920 con 0x7f4c38004d00 2023-11-28T22:08:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.062+0000 7f4c4d7fa640 1 -- 172.21.15.45:0/983853469 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4400fa80 con 0x7f4c38004d00 2023-11-28T22:08:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.062+0000 7f4c4d7fa640 1 -- 172.21.15.45:0/983853469 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4400fdf0 con 0x7f4c38004d00 2023-11-28T22:08:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.062+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c500af230 con 0x7f4c38004d00 2023-11-28T22:08:53.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.062+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5013ffc0 con 0x7f4c38004d00 2023-11-28T22:08:53.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.063+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c50073440 con 0x7f4c38004d00 2023-11-28T22:08:53.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.064+0000 7f4c4d7fa640 1 -- 172.21.15.45:0/983853469 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4c44015460 con 0x7f4c38004d00 2023-11-28T22:08:53.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.065+0000 7f4c4d7fa640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4c3c06af90 0x7f4c3c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.065+0000 7f4c4d7fa640 1 -- 172.21.15.45:0/983853469 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4c44089e20 con 0x7f4c38004d00 2023-11-28T22:08:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.066+0000 7f4c4ffff640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4c3c06af90 0x7f4c3c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.067+0000 7f4c4ffff640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4c3c06af90 0x7f4c3c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4c40001eb0 tx=0x7f4c40009040 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:53.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.070+0000 7f4c4d7fa640 1 -- 172.21.15.45:0/983853469 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c44054c50 con 0x7f4c38004d00 2023-11-28T22:08:53.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.291+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 --> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c50070b10 con 0x7f4c3c06af90 2023-11-28T22:08:53.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.296+0000 7f4c4d7fa640 1 -- 172.21.15.45:0/983853469 <== mgr.7643 v2:172.21.15.190:6832/397715021 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4c50070b10 con 0x7f4c3c06af90 2023-11-28T22:08:53.298 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2023-11-28T22:08:53.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2023-11-28T22:08:53.298 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephfs" 2023-11-28T22:08:53.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-11-28T22:08:53.298 INFO:teuthology.orchestra.run.smithi045.stdout:] 2023-11-28T22:08:53.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4c3c06af90 msgr2=0x7f4c3c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4c3c06af90 0x7f4c3c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4c40001eb0 tx=0x7f4c40009040 comp rx=0 tx=0).stop 2023-11-28T22:08:53.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 msgr2=0x7f4c5013fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.300 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c5013fa80 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f4c44011600 tx=0x7f4c44011ac0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.301 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 shutdown_connections 2023-11-28T22:08:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4c500adc80 0x7f4c500ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c500c4480 0x7f4c500c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f4c3c06af90 0x7f4c3c06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 --2- 172.21.15.45:0/983853469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c38004d00 0x7f4c5013fa80 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 >> 172.21.15.45:0/983853469 conn(0x7f4c5006a6e0 msgr2=0x7f4c500cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 shutdown_connections 2023-11-28T22:08:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.298+0000 7f4c56d39640 1 -- 172.21.15.45:0/983853469 wait complete. 2023-11-28T22:08:53.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-11-28T22:08:53.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.549+0000 7f8be92b4640 1 Processor -- start 2023-11-28T22:08:53.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.550+0000 7f8be92b4640 1 -- start start 2023-11-28T22:08:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8be92b4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8be40cef10 0x7f8be415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8be92b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8be40cd590 0x7f8be40cda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8be92b4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be415e520 con 0x7f8be415dc00 2023-11-28T22:08:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8be92b4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be406e790 con 0x7f8be40cef10 2023-11-28T22:08:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8be92b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be406ea70 con 0x7f8be40cd590 2023-11-28T22:08:53.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8be3fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8be415dc00 0x7f8be40ae9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48280/0 (socket says 172.21.15.45:48280) 2023-11-28T22:08:53.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8be3fff640 1 -- 172.21.15.45:0/1480899580 learned_addr learned my addr 172.21.15.45:0/1480899580 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:53.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8bdb7fe640 1 --2- 172.21.15.45:0/1480899580 >> v2:172.21.15.190:3300/0 conn(0x7f8be40cef10 0x7f8be415dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885404512 0 0) 0x7f8be415e520 con 0x7f8be415dc00 2023-11-28T22:08:53.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.552+0000 7f8be8ab3640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8be40cd590 0x7f8be40cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bd0003520 con 0x7f8be415dc00 2023-11-28T22:08:53.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1835094991 0 0) 0x7f8bd0003520 con 0x7f8be415dc00 2023-11-28T22:08:53.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8be40cd590 msgr2=0x7f8be40cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8be40cd590 0x7f8be40cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 >> v2:172.21.15.190:3300/0 conn(0x7f8be40cef10 msgr2=0x7f8be415dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 --2- 172.21.15.45:0/1480899580 >> v2:172.21.15.190:3300/0 conn(0x7f8be40cef10 0x7f8be415dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be406ed50 con 0x7f8be415dc00 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f8bd4001670 con 0x7f8be415dc00 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 >> v1:172.21.15.45:6789/0 conn(0x7f8be415dc00 legacy=0x7f8be40ae9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bd00039a0 0x7f8bd0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8be40cd590 0x7f8bd00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd0004d00 0x7f8bd0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be406e790 con 0x7f8bd0004d00 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be406ea70 con 0x7f8bd00039a0 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd000ba80 con 0x7f8be40cd590 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.553+0000 7f8be8ab3640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd0004d00 0x7f8bd0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.554+0000 7f8bdb7fe640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8be40cd590 0x7f8bd00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.554+0000 7f8be3fff640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bd00039a0 0x7f8bd0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.554+0000 7f8be8ab3640 1 -- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8be40cd590 msgr2=0x7f8bd00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:53.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.554+0000 7f8be8ab3640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8be40cd590 0x7f8bd00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.554+0000 7f8be8ab3640 1 -- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bd00039a0 msgr2=0x7f8bd0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:53.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.554+0000 7f8be8ab3640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bd00039a0 0x7f8bd0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.554+0000 7f8be8ab3640 1 -- 172.21.15.45:0/1480899580 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd0003520 con 0x7f8bd0004d00 2023-11-28T22:08:53.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.555+0000 7f8be8ab3640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd0004d00 0x7f8bd0005180 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc007360 tx=0x7f8bd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd92a96c19508b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:53.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.555+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc02ccd0 con 0x7f8bd0004d00 2023-11-28T22:08:53.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.556+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bdc02ce30 con 0x7f8bd0004d00 2023-11-28T22:08:53.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.556+0000 7f8be37fe640 1 -- 172.21.15.45:0/1480899580 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc034660 con 0x7f8bd0004d00 2023-11-28T22:08:53.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.556+0000 7f8be92b4640 1 -- 172.21.15.45:0/1480899580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd0004d00 msgr2=0x7f8bd0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.556+0000 7f8be92b4640 1 --2- 172.21.15.45:0/1480899580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd0004d00 0x7f8bd0005180 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc007360 tx=0x7f8bd00057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.556+0000 7f8be8ab3640 1 -- 172.21.15.45:0/1480899580 reap_dead start 2023-11-28T22:08:53.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.556+0000 7f8be92b4640 1 -- 172.21.15.45:0/1480899580 shutdown_connections 2023-11-28T22:08:53.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.556+0000 7f8be92b4640 1 -- 172.21.15.45:0/1480899580 >> 172.21.15.45:0/1480899580 conn(0x7f8be40cbf20 msgr2=0x7f8be40cc330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:53.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.556+0000 7f8be92b4640 1 -- 172.21.15.45:0/1480899580 shutdown_connections 2023-11-28T22:08:53.560 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.556+0000 7f8be92b4640 1 -- 172.21.15.45:0/1480899580 wait complete. 2023-11-28T22:08:53.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.558+0000 7f8be92b4640 1 Processor -- start 2023-11-28T22:08:53.561 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.558+0000 7f8be92b4640 1 -- start start 2023-11-28T22:08:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.559+0000 7f8be92b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0004d00 0x7f8be41516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.559+0000 7f8be92b4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8be4148bc0 0x7f8be4152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.559+0000 7f8be92b4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8be41532f0 0x7f8be4144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.559+0000 7f8be92b4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be4152990 con 0x7f8be4148bc0 2023-11-28T22:08:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.559+0000 7f8be92b4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be40b5390 con 0x7f8be41532f0 2023-11-28T22:08:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.559+0000 7f8be92b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be4151d10 con 0x7f8bd0004d00 2023-11-28T22:08:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8bdb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8be4148bc0 0x7f8be4152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8bdb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8be4148bc0 0x7f8be4152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59756/0 (socket says 172.21.15.45:59756) 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8bdb7fe640 1 -- 172.21.15.45:0/1423138197 learned_addr learned my addr 172.21.15.45:0/1423138197 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8be8ab3640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8be41532f0 0x7f8be4144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8be3fff640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0004d00 0x7f8be41516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8bdb7fe640 1 -- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0004d00 msgr2=0x7f8be41516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8bdb7fe640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0004d00 0x7f8be41516a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8bdb7fe640 1 -- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8be41532f0 msgr2=0x7f8be4144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8bdb7fe640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8be41532f0 0x7f8be4144890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8bdb7fe640 1 -- 172.21.15.45:0/1423138197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be4151510 con 0x7f8be4148bc0 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8be8ab3640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8be41532f0 0x7f8be4144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.560+0000 7f8bdb7fe640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8be4148bc0 0x7f8be4152db0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00f650 tx=0x7f8bcc00fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.561+0000 7f8be17fa640 1 -- 172.21.15.45:0/1423138197 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc004020 con 0x7f8be4148bc0 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.561+0000 7f8be17fa640 1 -- 172.21.15.45:0/1423138197 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc0041a0 con 0x7f8be4148bc0 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.561+0000 7f8be17fa640 1 -- 172.21.15.45:0/1423138197 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc00d740 con 0x7f8be4148bc0 2023-11-28T22:08:53.564 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.561+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be4151ff0 con 0x7f8be4148bc0 2023-11-28T22:08:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.561+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be41525b0 con 0x7f8be4148bc0 2023-11-28T22:08:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.562+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be4152ad0 con 0x7f8be4148bc0 2023-11-28T22:08:53.565 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.563+0000 7f8be17fa640 1 -- 172.21.15.45:0/1423138197 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 117) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8bcc00d8a0 con 0x7f8be4148bc0 2023-11-28T22:08:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.564+0000 7f8be17fa640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f8bc406af90 0x7f8bc406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:53.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.564+0000 7f8be17fa640 1 -- 172.21.15.45:0/1423138197 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8bcc089ec0 con 0x7f8be4148bc0 2023-11-28T22:08:53.567 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.565+0000 7f8be3fff640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f8bc406af90 0x7f8bc406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:53.568 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.566+0000 7f8be3fff640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f8bc406af90 0x7f8bc406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4001eb0 tx=0x7f8bd4001da0 comp rx=0 tx=0).ready entity=mgr.7643 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:53.572 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.568+0000 7f8be17fa640 1 -- 172.21.15.45:0/1423138197 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8be4152ad0 con 0x7f8be4148bc0 2023-11-28T22:08:53.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:53.811+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f8be4152ad0 con 0x7f8be4148bc0 2023-11-28T22:08:54.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.430+0000 7f8be17fa640 1 -- 172.21.15.45:0/1423138197 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v118) v1 ==== 92+0+0 (secure 0 0 0) 0x7f8be4152ad0 con 0x7f8be4148bc0 2023-11-28T22:08:54.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.433+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f8bc406af90 msgr2=0x7f8bc406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:54.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.433+0000 7f8be92b4640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f8bc406af90 0x7f8bc406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4001eb0 tx=0x7f8bd4001da0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.433+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8be4148bc0 msgr2=0x7f8be4152db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:54.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.433+0000 7f8be92b4640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8be4148bc0 0x7f8be4152db0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00f650 tx=0x7f8bcc00fb10 comp rx=0 tx=0).stop 2023-11-28T22:08:54.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.434+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 shutdown_connections 2023-11-28T22:08:54.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.434+0000 7f8be92b4640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8be41532f0 0x7f8be4144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.434+0000 7f8be92b4640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f8bc406af90 0x7f8bc406d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.434+0000 7f8be92b4640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8be4148bc0 0x7f8be4152db0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.434+0000 7f8be92b4640 1 --2- 172.21.15.45:0/1423138197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8bd0004d00 0x7f8be41516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.434+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 >> 172.21.15.45:0/1423138197 conn(0x7f8be40cbf20 msgr2=0x7f8be40c87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:54.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.435+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 shutdown_connections 2023-11-28T22:08:54.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.435+0000 7f8be92b4640 1 -- 172.21.15.45:0/1423138197 wait complete. 2023-11-28T22:08:54.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-11-28T22:08:54.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.695+0000 7f48fefbe640 1 Processor -- start 2023-11-28T22:08:54.698 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.695+0000 7f48fefbe640 1 -- start start 2023-11-28T22:08:54.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.697+0000 7f48fefbe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f815e840 0x7f48f806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:54.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.697+0000 7f48fefbe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f48f806ac10 0x7f48f80717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:54.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.697+0000 7f48fefbe640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f815ecd0 con 0x7f48f815f720 2023-11-28T22:08:54.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.697+0000 7f48fefbe640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f8071ce0 con 0x7f48f806ac10 2023-11-28T22:08:54.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.697+0000 7f48fefbe640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f815f290 con 0x7f48f815e840 2023-11-28T22:08:54.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.697+0000 7f48fcd5a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f815e840 0x7f48f806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:54.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.697+0000 7f48fd55b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f48f806ac10 0x7f48f80717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.698+0000 7f48fcd5a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f815e840 0x7f48f806c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:52320/0 (socket says 172.21.15.45:52320) 2023-11-28T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.698+0000 7f48fcd5a640 1 -- 172.21.15.45:0/2847829505 learned_addr learned my addr 172.21.15.45:0/2847829505 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.698+0000 7f48fcd5a640 1 -- 172.21.15.45:0/2847829505 >> v2:172.21.15.190:3300/0 conn(0x7f48f806ac10 msgr2=0x7f48f80717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.698+0000 7f48fcd5a640 1 --2- 172.21.15.45:0/2847829505 >> v2:172.21.15.190:3300/0 conn(0x7f48f806ac10 0x7f48f80717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.698+0000 7f48fcd5a640 1 -- 172.21.15.45:0/2847829505 >> v1:172.21.15.190:6789/0 conn(0x7f48f815f720 legacy=0x7f48f806a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.698+0000 7f48fcd5a640 1 -- 172.21.15.45:0/2847829505 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f815fb60 con 0x7f48f815e840 2023-11-28T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.699+0000 7f48fcd5a640 1 --2- 172.21.15.45:0/2847829505 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f815e840 0x7f48f806c350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f48e4009870 tx=0x7f48e402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd86038bdc907a10 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:54.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.700+0000 7f48f77fe640 1 -- 172.21.15.45:0/2847829505 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e4030840 con 0x7f48f815e840 2023-11-28T22:08:54.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.700+0000 7f48f77fe640 1 -- 172.21.15.45:0/2847829505 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e40309a0 con 0x7f48f815e840 2023-11-28T22:08:54.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.700+0000 7f48f77fe640 1 -- 172.21.15.45:0/2847829505 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e4030d30 con 0x7f48f815e840 2023-11-28T22:08:54.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.700+0000 7f48fefbe640 1 -- 172.21.15.45:0/2847829505 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f815e840 msgr2=0x7f48f806c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:54.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.700+0000 7f48fefbe640 1 --2- 172.21.15.45:0/2847829505 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f815e840 0x7f48f806c350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f48e4009870 tx=0x7f48e402b980 comp rx=0 tx=0).stop 2023-11-28T22:08:54.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.701+0000 7f48fefbe640 1 -- 172.21.15.45:0/2847829505 shutdown_connections 2023-11-28T22:08:54.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.701+0000 7f48fefbe640 1 --2- 172.21.15.45:0/2847829505 >> v2:172.21.15.190:3300/0 conn(0x7f48f806ac10 0x7f48f80717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.703 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.701+0000 7f48fefbe640 1 --2- 172.21.15.45:0/2847829505 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f815e840 0x7f48f806c350 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.701+0000 7f48fefbe640 1 -- 172.21.15.45:0/2847829505 >> 172.21.15.45:0/2847829505 conn(0x7f48f80ced60 msgr2=0x7f48f80cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:54.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.701+0000 7f48fefbe640 1 -- 172.21.15.45:0/2847829505 shutdown_connections 2023-11-28T22:08:54.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.701+0000 7f48fefbe640 1 -- 172.21.15.45:0/2847829505 wait complete. 2023-11-28T22:08:54.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.702+0000 7f48fefbe640 1 Processor -- start 2023-11-28T22:08:54.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.703+0000 7f48fefbe640 1 -- start start 2023-11-28T22:08:54.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fefbe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f806ac10 0x7f48f8159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:54.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fefbe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48f815e840 0x7f48f80c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:54.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fefbe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48f815f720 0x7f48f80c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:54.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fefbe640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f8157b70 con 0x7f48f815f720 2023-11-28T22:08:54.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fefbe640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f80c7bf0 con 0x7f48f815e840 2023-11-28T22:08:54.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fefbe640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f8159e60 con 0x7f48f806ac10 2023-11-28T22:08:54.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fd55b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48f815f720 0x7f48f80c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fd55b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48f815f720 0x7f48f80c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50096/0 (socket says 172.21.15.45:50096) 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fd55b640 1 -- 172.21.15.45:0/891701836 learned_addr learned my addr 172.21.15.45:0/891701836 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fcd5a640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f806ac10 0x7f48f8159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48f7fff640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48f815e840 0x7f48f80c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fd55b640 1 -- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f806ac10 msgr2=0x7f48f8159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fd55b640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f806ac10 0x7f48f8159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fd55b640 1 -- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48f815e840 msgr2=0x7f48f80c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fd55b640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48f815e840 0x7f48f80c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.704+0000 7f48fd55b640 1 -- 172.21.15.45:0/891701836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e40094d0 con 0x7f48f815f720 2023-11-28T22:08:54.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.705+0000 7f48fd55b640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48f815f720 0x7f48f80c44b0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f48f000efc0 tx=0x7f48f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:54.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.705+0000 7f48f57fa640 1 -- 172.21.15.45:0/891701836 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0009b10 con 0x7f48f815f720 2023-11-28T22:08:54.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.705+0000 7f48f57fa640 1 -- 172.21.15.45:0/891701836 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f000f040 con 0x7f48f815f720 2023-11-28T22:08:54.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.705+0000 7f48f57fa640 1 -- 172.21.15.45:0/891701836 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f00146c0 con 0x7f48f815f720 2023-11-28T22:08:54.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.705+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f80c7070 con 0x7f48f815f720 2023-11-28T22:08:54.709 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.705+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f80c7630 con 0x7f48f815f720 2023-11-28T22:08:54.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.707+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f8157cb0 con 0x7f48f815f720 2023-11-28T22:08:54.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.708+0000 7f48f57fa640 1 -- 172.21.15.45:0/891701836 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 118) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f48f0014820 con 0x7f48f815f720 2023-11-28T22:08:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.709+0000 7f48f57fa640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f48dc06afe0 0x7f48dc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.709+0000 7f48f57fa640 1 -- 172.21.15.45:0/891701836 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f48f008a550 con 0x7f48f815f720 2023-11-28T22:08:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.709+0000 7f48fcd5a640 1 -- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f48dc06afe0 msgr2=0x7f48dc06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/397715021 2023-11-28T22:08:54.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.709+0000 7f48fcd5a640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f48dc06afe0 0x7f48dc06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:54.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.717+0000 7f48f57fa640 1 -- 172.21.15.45:0/891701836 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48f00551e0 con 0x7f48f815f720 2023-11-28T22:08:54.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.910+0000 7f48fcd5a640 1 -- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f48dc06afe0 msgr2=0x7f48dc06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/397715021 2023-11-28T22:08:54.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.910+0000 7f48fcd5a640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f48dc06afe0 0x7f48dc06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:08:54.917 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.915+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f48f80b4ca0 con 0x7f48f815f720 2023-11-28T22:08:54.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.919+0000 7f48f57fa640 1 -- 172.21.15.45:0/891701836 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f48f0058800 con 0x7f48f815f720 2023-11-28T22:08:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f48dc06afe0 msgr2=0x7f48dc06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f48dc06afe0 0x7f48dc06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48f815f720 msgr2=0x7f48f80c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48f815f720 0x7f48f80c44b0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f48f000efc0 tx=0x7f48f000c490 comp rx=0 tx=0).stop 2023-11-28T22:08:54.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 shutdown_connections 2023-11-28T22:08:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7f48dc06afe0 0x7f48dc06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48f815f720 0x7f48f80c44b0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48f815e840 0x7f48f80c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 --2- 172.21.15.45:0/891701836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48f806ac10 0x7f48f8159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 >> 172.21.15.45:0/891701836 conn(0x7f48f80ced60 msgr2=0x7f48f8070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 shutdown_connections 2023-11-28T22:08:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:54.922+0000 7f48fefbe640 1 -- 172.21.15.45:0/891701836 wait complete. 2023-11-28T22:08:54.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-11-28T22:08:54.986 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:54.984+0000 7f6e68c2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T22:08:55.025 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:55.022+0000 7ff8c782b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T22:08:55.038 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.035+0000 7f6e68c2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T22:08:55.081 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:55.079+0000 7ff8c782b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T22:08:55.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.155+0000 7fea80e56640 1 Processor -- start 2023-11-28T22:08:55.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.155+0000 7fea80e56640 1 -- start start 2023-11-28T22:08:55.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.157+0000 7fea80e56640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fea7c0d24f0 0x7fea7c0d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.157+0000 7fea80e56640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 0x7fea7c0ce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.157+0000 7fea80e56640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c160e80 con 0x7fea7c160aa0 2023-11-28T22:08:55.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.157+0000 7fea80e56640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea7c0d1220 con 0x7fea7c0d24f0 2023-11-28T22:08:55.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.157+0000 7fea80e56640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7c0d2e70 con 0x7fea7c0ce0e0 2023-11-28T22:08:55.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.157+0000 7fea7a59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fea7c160aa0 0x7fea7c0b6420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58626/0 (socket says 172.21.15.45:58626) 2023-11-28T22:08:55.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.157+0000 7fea7a59c640 1 -- 172.21.15.45:0/1645011958 learned_addr learned my addr 172.21.15.45:0/1645011958 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:55.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.157+0000 7fea73fff640 1 --2- 172.21.15.45:0/1645011958 >> v2:172.21.15.45:3300/0 conn(0x7fea7c0d24f0 0x7fea7c0d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.157+0000 7fea7ad9d640 1 --2- 172.21.15.45:0/1645011958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 0x7fea7c0ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.158+0000 7fea73fff640 1 -- 172.21.15.45:0/1645011958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 msgr2=0x7fea7c0ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:55.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.158+0000 7fea73fff640 1 --2- 172.21.15.45:0/1645011958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 0x7fea7c0ce570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.158+0000 7fea73fff640 1 -- 172.21.15.45:0/1645011958 >> v1:172.21.15.45:6789/0 conn(0x7fea7c160aa0 legacy=0x7fea7c0b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.158+0000 7fea73fff640 1 -- 172.21.15.45:0/1645011958 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c0ceab0 con 0x7fea7c0d24f0 2023-11-28T22:08:55.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.159+0000 7fea73fff640 1 --2- 172.21.15.45:0/1645011958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 0x7fea7c0d2930 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fea640097a0 tx=0x7fea6402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbf03e164353a73 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.159+0000 7fea79d9b640 1 -- 172.21.15.45:0/1645011958 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64030800 con 0x7fea7c0d24f0 2023-11-28T22:08:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.159+0000 7fea79d9b640 1 -- 172.21.15.45:0/1645011958 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea64030960 con 0x7fea7c0d24f0 2023-11-28T22:08:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.159+0000 7fea79d9b640 1 -- 172.21.15.45:0/1645011958 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64030cf0 con 0x7fea7c0d24f0 2023-11-28T22:08:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.160+0000 7fea80e56640 1 -- 172.21.15.45:0/1645011958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 msgr2=0x7fea7c0d2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.160+0000 7fea80e56640 1 --2- 172.21.15.45:0/1645011958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 0x7fea7c0d2930 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fea640097a0 tx=0x7fea6402b940 comp rx=0 tx=0).stop 2023-11-28T22:08:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.160+0000 7fea80e56640 1 -- 172.21.15.45:0/1645011958 shutdown_connections 2023-11-28T22:08:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.160+0000 7fea80e56640 1 --2- 172.21.15.45:0/1645011958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 0x7fea7c0ce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.160+0000 7fea80e56640 1 --2- 172.21.15.45:0/1645011958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 0x7fea7c0d2930 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.163 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.160+0000 7fea80e56640 1 -- 172.21.15.45:0/1645011958 >> 172.21.15.45:0/1645011958 conn(0x7fea7c070b20 msgr2=0x7fea7c070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:55.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.160+0000 7fea80e56640 1 -- 172.21.15.45:0/1645011958 shutdown_connections 2023-11-28T22:08:55.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.160+0000 7fea80e56640 1 -- 172.21.15.45:0/1645011958 wait complete. 2023-11-28T22:08:55.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.162+0000 7fea80e56640 1 Processor -- start 2023-11-28T22:08:55.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.162+0000 7fea80e56640 1 -- start start 2023-11-28T22:08:55.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.163+0000 7fea80e56640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 0x7fea7c1f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.163+0000 7fea80e56640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 0x7fea7c0cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.163+0000 7fea80e56640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea7c160aa0 0x7fea7c0ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.163+0000 7fea80e56640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7c1f3f60 con 0x7fea7c0d24f0 2023-11-28T22:08:55.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.163+0000 7fea80e56640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7c0b2e10 con 0x7fea7c160aa0 2023-11-28T22:08:55.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.163+0000 7fea80e56640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7c0b30f0 con 0x7fea7c0ce0e0 2023-11-28T22:08:55.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.163+0000 7fea73fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 0x7fea7c0cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.163+0000 7fea73fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 0x7fea7c0cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50112/0 (socket says 172.21.15.45:50112) 2023-11-28T22:08:55.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.163+0000 7fea73fff640 1 -- 172.21.15.45:0/1586613678 learned_addr learned my addr 172.21.15.45:0/1586613678 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea7ad9d640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea7c160aa0 0x7fea7c0ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea7a59c640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 0x7fea7c1f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea73fff640 1 -- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 msgr2=0x7fea7c1f3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea73fff640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 0x7fea7c1f3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea73fff640 1 -- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea7c160aa0 msgr2=0x7fea7c0ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea73fff640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea7c160aa0 0x7fea7c0ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea73fff640 1 -- 172.21.15.45:0/1586613678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea64009400 con 0x7fea7c0d24f0 2023-11-28T22:08:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea7a59c640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 0x7fea7c1f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea73fff640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 0x7fea7c0cc890 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fea6402bf50 tx=0x7fea64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea72ffd640 1 -- 172.21.15.45:0/1586613678 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6403e040 con 0x7fea7c0d24f0 2023-11-28T22:08:55.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea72ffd640 1 -- 172.21.15.45:0/1586613678 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea64039cb0 con 0x7fea7c0d24f0 2023-11-28T22:08:55.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea72ffd640 1 -- 172.21.15.45:0/1586613678 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64044630 con 0x7fea7c0d24f0 2023-11-28T22:08:55.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.164+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea7c0b33d0 con 0x7fea7c0d24f0 2023-11-28T22:08:55.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.165+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7c0b3990 con 0x7fea7c0d24f0 2023-11-28T22:08:55.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.166+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7c0b3bd0 con 0x7fea7c0d24f0 2023-11-28T22:08:55.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.167+0000 7fea72ffd640 1 -- 172.21.15.45:0/1586613678 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 118) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fea64043390 con 0x7fea7c0d24f0 2023-11-28T22:08:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.168+0000 7fea72ffd640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fea5c06afb0 0x7fea5c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.168+0000 7fea72ffd640 1 -- 172.21.15.45:0/1586613678 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fea640abda0 con 0x7fea7c0d24f0 2023-11-28T22:08:55.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.168+0000 7fea7a59c640 1 -- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fea5c06afb0 msgr2=0x7fea5c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/397715021 2023-11-28T22:08:55.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.168+0000 7fea7a59c640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fea5c06afb0 0x7fea5c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.173+0000 7fea72ffd640 1 -- 172.21.15.45:0/1586613678 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea7c0b3bd0 con 0x7fea7c0d24f0 2023-11-28T22:08:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.364+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fea7c0b3bd0 con 0x7fea7c0d24f0 2023-11-28T22:08:55.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.369+0000 7fea7a59c640 1 -- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fea5c06afb0 msgr2=0x7fea5c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/397715021 2023-11-28T22:08:55.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.369+0000 7fea7a59c640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fea5c06afb0 0x7fea5c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:08:55.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.369+0000 7fea72ffd640 1 -- 172.21.15.45:0/1586613678 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7fea7c0b3bd0 con 0x7fea7c0d24f0 2023-11-28T22:08:55.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fea5c06afb0 msgr2=0x7fea5c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:55.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fea5c06afb0 0x7fea5c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 msgr2=0x7fea7c0cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 0x7fea7c0cc890 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fea6402bf50 tx=0x7fea64004730 comp rx=0 tx=0).stop 2023-11-28T22:08:55.375 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 shutdown_connections 2023-11-28T22:08:55.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea7c160aa0 0x7fea7c0ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fea5c06afb0 0x7fea5c06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea7c0d24f0 0x7fea7c0cc890 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 --2- 172.21.15.45:0/1586613678 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea7c0ce0e0 0x7fea7c1f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 >> 172.21.15.45:0/1586613678 conn(0x7fea7c070b20 msgr2=0x7fea7c0d3870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:55.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 shutdown_connections 2023-11-28T22:08:55.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.372+0000 7fea80e56640 1 -- 172.21.15.45:0/1586613678 wait complete. 2023-11-28T22:08:55.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:08:55.394 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-11-28T22:08:55.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-11-28T22:08:55.486 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.483+0000 7f6e68c2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T22:08:55.537 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.535+0000 7f6e68c2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T22:08:55.562 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:55.560+0000 7ff8c782b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T22:08:55.587 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.585+0000 7f6e68c2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T22:08:55.618 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:55.615+0000 7ff8c782b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T22:08:55.647 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.645+0000 7f6e68c2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T22:08:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.667+0000 7fc4def24640 1 Processor -- start 2023-11-28T22:08:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.668+0000 7fc4def24640 1 -- start start 2023-11-28T22:08:55.671 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:55.668+0000 7ff8c782b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.669+0000 7fc4def24640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d80d28a0 0x7fc4d80b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.669+0000 7fc4def24640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d80d1800 con 0x7fc4d8160980 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.670+0000 7fc4def24640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d80d1ae0 con 0x7fc4d80d2ce0 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.670+0000 7fc4def24640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d80d1dc0 con 0x7fc4d80d28a0 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.670+0000 7fc4dccc0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc4d8160980 0x7fc4d80b8ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58634/0 (socket says 172.21.15.45:58634) 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.670+0000 7fc4dccc0640 1 -- 172.21.15.45:0/936714628 learned_addr learned my addr 172.21.15.45:0/936714628 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.670+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d80d28a0 0x7fc4d80b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.670+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224438861 0 0) 0x7fc4d80d1800 con 0x7fc4d8160980 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.670+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c4003520 con 0x7fc4d8160980 2023-11-28T22:08:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889274331 0 0) 0x7fc4c4003520 con 0x7fc4d8160980 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d80d28a0 msgr2=0x7fc4d80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d80d28a0 0x7fc4d80b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 >> v1:172.21.15.190:6789/0 conn(0x7fc4d80d2ce0 legacy=0x7fc4d80b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d80d20a0 con 0x7fc4d8160980 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fc4c8001670 con 0x7fc4d8160980 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 >> v1:172.21.15.45:6789/0 conn(0x7fc4d8160980 legacy=0x7fc4d80b8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4c40039a0 0x7fc4c4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 0x7fc4c4004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d80d28a0 0x7fc4c4005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d80d1dc0 con 0x7fc4c4004350 2023-11-28T22:08:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c400b9b0 con 0x7fc4c40039a0 2023-11-28T22:08:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c400bb70 con 0x7fc4d80d28a0 2023-11-28T22:08:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 0x7fc4c4004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.671+0000 7fc4dccc0640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4c40039a0 0x7fc4c4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.672+0000 7fc4dd4c1640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d80d28a0 0x7fc4c4005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.672+0000 7fc4d7fff640 1 -- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d80d28a0 msgr2=0x7fc4c4005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.672+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d80d28a0 0x7fc4c4005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.672+0000 7fc4d7fff640 1 -- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4c40039a0 msgr2=0x7fc4c4003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:08:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.672+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4c40039a0 0x7fc4c4003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.672+0000 7fc4d7fff640 1 -- 172.21.15.45:0/936714628 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c4003520 con 0x7fc4c4004350 2023-11-28T22:08:55.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.672+0000 7fc4dd4c1640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d80d28a0 0x7fc4c4005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.673+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 0x7fc4c4004820 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009fc0 tx=0x7fc4c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb351d438b886c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.673+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c002cc20 con 0x7fc4c4004350 2023-11-28T22:08:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.673+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c000f040 con 0x7fc4c4004350 2023-11-28T22:08:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.673+0000 7fc4d77fe640 1 -- 172.21.15.45:0/936714628 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0034440 con 0x7fc4c4004350 2023-11-28T22:08:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.674+0000 7fc4def24640 1 -- 172.21.15.45:0/936714628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 msgr2=0x7fc4c4004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.674+0000 7fc4def24640 1 --2- 172.21.15.45:0/936714628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 0x7fc4c4004820 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009fc0 tx=0x7fc4c40057f0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.674+0000 7fc4dd4c1640 1 -- 172.21.15.45:0/936714628 reap_dead start 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.674+0000 7fc4def24640 1 -- 172.21.15.45:0/936714628 shutdown_connections 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.674+0000 7fc4def24640 1 -- 172.21.15.45:0/936714628 >> 172.21.15.45:0/936714628 conn(0x7fc4d80cfd00 msgr2=0x7fc4d80d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.674+0000 7fc4def24640 1 -- 172.21.15.45:0/936714628 shutdown_connections 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.674+0000 7fc4def24640 1 -- 172.21.15.45:0/936714628 wait complete. 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.676+0000 7fc4def24640 1 Processor -- start 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.676+0000 7fc4def24640 1 -- start start 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.677+0000 7fc4def24640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 0x7fc4d80c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.677+0000 7fc4def24640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4d81e8070 0x7fc4d81e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.677+0000 7fc4def24640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d81e6990 0x7fc4d81e6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.677+0000 7fc4def24640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81e8a30 con 0x7fc4c4004350 2023-11-28T22:08:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.677+0000 7fc4def24640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d814b880 con 0x7fc4d81e8070 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.677+0000 7fc4def24640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d80c8bc0 con 0x7fc4d81e6990 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.677+0000 7fc4dccc0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 0x7fc4d80c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.677+0000 7fc4dccc0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 0x7fc4d80c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50126/0 (socket says 172.21.15.45:50126) 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.677+0000 7fc4dccc0640 1 -- 172.21.15.45:0/59967894 learned_addr learned my addr 172.21.15.45:0/59967894 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4d81e8070 0x7fc4d81e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4dd4c1640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d81e6990 0x7fc4d81e6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4dccc0640 1 -- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d81e6990 msgr2=0x7fc4d81e6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4dccc0640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d81e6990 0x7fc4d81e6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4dccc0640 1 -- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4d81e8070 msgr2=0x7fc4d81e84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4dccc0640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4d81e8070 0x7fc4d81e84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4dccc0640 1 -- 172.21.15.45:0/59967894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d81e7710 con 0x7fc4c4004350 2023-11-28T22:08:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4d81e8070 0x7fc4d81e84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:08:55.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4dd4c1640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d81e6990 0x7fc4d81e6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:08:55.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4dccc0640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 0x7fc4d80c8680 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8007290 tx=0x7fc4c8007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:55.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.678+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c800fa00 con 0x7fc4c4004350 2023-11-28T22:08:55.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.679+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c800fb80 con 0x7fc4c4004350 2023-11-28T22:08:55.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.679+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c80133f0 con 0x7fc4c4004350 2023-11-28T22:08:55.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.679+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d814af50 con 0x7fc4c4004350 2023-11-28T22:08:55.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.679+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d814b510 con 0x7fc4c4004350 2023-11-28T22:08:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.680+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d80733d0 con 0x7fc4c4004350 2023-11-28T22:08:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.681+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 118) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fc4c8013550 con 0x7fc4c4004350 2023-11-28T22:08:55.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.682+0000 7fc4d57fa640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 0x7fc4b806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:55.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.682+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc4c8089d90 con 0x7fc4c4004350 2023-11-28T22:08:55.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.682+0000 7fc4d7fff640 1 -- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 msgr2=0x7fc4b806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/397715021 2023-11-28T22:08:55.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.683+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 0x7fc4b806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:08:55.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.687+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4c8054b30 con 0x7fc4c4004350 2023-11-28T22:08:55.697 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.695+0000 7f6e68c2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T22:08:55.738 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:55.735+0000 7ff8c782b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T22:08:55.790 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:55.788+0000 7ff8c782b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T22:08:55.807 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.805+0000 7f6e68c2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T22:08:55.869 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.867+0000 7f6e68c2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T22:08:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.884+0000 7fc4d7fff640 1 -- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 msgr2=0x7fc4b806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/397715021 2023-11-28T22:08:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.884+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 0x7fc4b806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:08:55.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:55.900+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 --> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4d815f810 con 0x7fc4b806afb0 2023-11-28T22:08:55.907 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:55.904+0000 7ff8c782b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T22:08:55.918 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.916+0000 7f6e68c2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T22:08:55.973 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:55.970+0000 7ff8c782b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T22:08:55.976 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:55.974+0000 7f6e68c2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T22:08:56.026 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:56.023+0000 7ff8c782b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T22:08:56.026 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:56.024+0000 7f6e68c2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T22:08:56.085 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:56.083+0000 7ff8c782b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T22:08:56.138 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:56.135+0000 7ff8c782b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T22:08:56.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:56.285+0000 7fc4d7fff640 1 -- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 msgr2=0x7fc4b806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/397715021 2023-11-28T22:08:56.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:56.285+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 0x7fc4b806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T22:08:56.410 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:56.407+0000 7f6e68c2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T22:08:56.540 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:56.537+0000 7ff8c782b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T22:08:56.568 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:56.566+0000 7f6e68c2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T22:08:56.676 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:56.673+0000 7f6e68c2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T22:08:56.708 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:56.706+0000 7ff8c782b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T22:08:56.727 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:56.724+0000 7f6e68c2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T22:08:56.821 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:56.818+0000 7ff8c782b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T22:08:56.834 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:56.832+0000 7f6e68c2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:08:56.873 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:56.871+0000 7ff8c782b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T22:08:56.942 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:56.939+0000 7f6e68c2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T22:08:56.986 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:56.984+0000 7ff8c782b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:08:57.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:57.086+0000 7fc4d7fff640 1 -- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 msgr2=0x7fc4b806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/397715021 2023-11-28T22:08:57.090 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:57.086+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 0x7fc4b806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T22:08:57.097 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:57.095+0000 7f6e68c2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:08:57.099 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:57.096+0000 7ff8c782b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T22:08:57.146 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:57.143+0000 7f6e68c2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T22:08:57.265 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:57.263+0000 7f6e68c2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T22:08:57.266 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:57.263+0000 7ff8c782b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:08:57.317 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:57.314+0000 7ff8c782b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T22:08:57.444 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:57.441+0000 7ff8c782b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T22:08:58.095 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.092+0000 7f6e68c2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T22:08:58.330 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:58.327+0000 7ff8c782b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T22:08:58.391 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.389+0000 7f6e68c2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T22:08:58.448 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.446+0000 7f6e68c2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T22:08:58.562 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.559+0000 7f6e68c2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T22:08:58.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:58.581+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 119) v1 ==== 88959+0+0 (secure 0 0 0) 0x7fc4c8054f00 con 0x7fc4c4004350 2023-11-28T22:08:58.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:58.582+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 msgr2=0x7fc4b806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:08:58.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:58.582+0000 7fc4d57fa640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/397715021,v1:172.21.15.190:6833/397715021] conn(0x7fc4b806afb0 0x7fc4b806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:08:58.654 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:58.652+0000 7ff8c782b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T22:08:58.725 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:58.722+0000 7ff8c782b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T22:08:58.766 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.763+0000 7f6de7fac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.766 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.763+0000 7f6de7fac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.766 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.763+0000 7f6de7fac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.766 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.763+0000 7f6de7fac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.766 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.763+0000 7f6de7fac640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.766 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.764+0000 7f6de4fa6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.767 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.764+0000 7f6de4fa6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.767 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.764+0000 7f6de4fa6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.767 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.764+0000 7f6de4fa6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.767 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:08:58.764+0000 7f6de4fa6640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:08:58.847 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:08:58.844+0000 7ff8c782b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T22:08:59.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.588+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 120) v1 ==== 89086+0+0 (secure 0 0 0) 0x7fc4c8070310 con 0x7fc4c4004350 2023-11-28T22:08:59.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.589+0000 7fc4d57fa640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc4b8035850 0x7fc4b80753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:08:59.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.589+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 --> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4d815f810 con 0x7fc4b8035850 2023-11-28T22:08:59.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.597+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc4b8035850 0x7fc4b80753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:08:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.606+0000 7fc4d7fff640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc4b8035850 0x7fc4b80753e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0036000 tx=0x7fc4c002e000 comp rx=0 tx=0).ready entity=mgr.8797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:08:59.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.762+0000 7fc4d57fa640 1 -- 172.21.15.45:0/59967894 <== mgr.8797 v2:172.21.15.190:6832/3154132268 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc4d815f810 con 0x7fc4b8035850 2023-11-28T22:08:59.765 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path snap_test_dir1 2023-11-28T22:08:59.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.765+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc4b8035850 msgr2=0x7fc4b80753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:59.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.765+0000 7fc4def24640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc4b8035850 0x7fc4b80753e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0036000 tx=0x7fc4c002e000 comp rx=0 tx=0).stop 2023-11-28T22:08:59.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.765+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 msgr2=0x7fc4d80c8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:08:59.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.765+0000 7fc4def24640 1 --2- 172.21.15.45:0/59967894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4c4004350 0x7fc4d80c8680 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8007290 tx=0x7fc4c8007750 comp rx=0 tx=0).stop 2023-11-28T22:08:59.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.765+0000 7fc4dd4c1640 1 -- 172.21.15.45:0/59967894 reap_dead start 2023-11-28T22:08:59.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.765+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 shutdown_connections 2023-11-28T22:08:59.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.765+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 >> 172.21.15.45:0/59967894 conn(0x7fc4d80cfd00 msgr2=0x7fc4d815e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:08:59.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.765+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 shutdown_connections 2023-11-28T22:08:59.768 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:08:59.766+0000 7fc4def24640 1 -- 172.21.15.45:0/59967894 wait complete. 2023-11-28T22:08:59.791 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-11-28T22:08:59.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-11-28T22:09:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.008+0000 7f5b19d2e640 1 Processor -- start 2023-11-28T22:09:00.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.008+0000 7f5b19d2e640 1 -- start start 2023-11-28T22:09:00.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.010+0000 7f5b19d2e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b1415b190 0x7f5b1406de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.010+0000 7f5b19d2e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b14070030 0x7f5b140b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.010+0000 7f5b19d2e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1406e3a0 con 0x7f5b1406c090 2023-11-28T22:09:00.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.010+0000 7f5b19d2e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1415b570 con 0x7f5b1415b190 2023-11-28T22:09:00.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.010+0000 7f5b19d2e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140c9d20 con 0x7f5b14070030 2023-11-28T22:09:00.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.010+0000 7f5b18d2c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b1415b190 0x7f5b1406de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:09:00.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.010+0000 7f5b18d2c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b1415b190 0x7f5b1406de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50142/0 (socket says 172.21.15.45:50142) 2023-11-28T22:09:00.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.010+0000 7f5b18d2c640 1 -- 172.21.15.45:0/4246863261 learned_addr learned my addr 172.21.15.45:0/4246863261 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:09:00.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.011+0000 7f5b13fff640 1 --2- 172.21.15.45:0/4246863261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b14070030 0x7f5b140b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:09:00.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.011+0000 7f5b18d2c640 1 -- 172.21.15.45:0/4246863261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b14070030 msgr2=0x7f5b140b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:09:00.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.011+0000 7f5b18d2c640 1 --2- 172.21.15.45:0/4246863261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b14070030 0x7f5b140b1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.011+0000 7f5b18d2c640 1 -- 172.21.15.45:0/4246863261 >> v1:172.21.15.45:6789/0 conn(0x7f5b1406c090 legacy=0x7f5b140ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.011+0000 7f5b18d2c640 1 -- 172.21.15.45:0/4246863261 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b14071c50 con 0x7f5b1415b190 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.012+0000 7f5b18d2c640 1 --2- 172.21.15.45:0/4246863261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1415b190 0x7f5b1406de60 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009870 tx=0x7f5b0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf6bab280ddab3c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.012+0000 7f5b137fe640 1 -- 172.21.15.45:0/4246863261 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08030840 con 0x7f5b1415b190 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.012+0000 7f5b137fe640 1 -- 172.21.15.45:0/4246863261 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b080309a0 con 0x7f5b1415b190 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.012+0000 7f5b137fe640 1 -- 172.21.15.45:0/4246863261 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08030d30 con 0x7f5b1415b190 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.013+0000 7f5b19d2e640 1 -- 172.21.15.45:0/4246863261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1415b190 msgr2=0x7f5b1406de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.013+0000 7f5b19d2e640 1 --2- 172.21.15.45:0/4246863261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1415b190 0x7f5b1406de60 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009870 tx=0x7f5b0802b980 comp rx=0 tx=0).stop 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.013+0000 7f5b19d2e640 1 -- 172.21.15.45:0/4246863261 shutdown_connections 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.013+0000 7f5b19d2e640 1 --2- 172.21.15.45:0/4246863261 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b14070030 0x7f5b140b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.013+0000 7f5b19d2e640 1 --2- 172.21.15.45:0/4246863261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b1415b190 0x7f5b1406de60 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.016 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.013+0000 7f5b19d2e640 1 -- 172.21.15.45:0/4246863261 >> 172.21.15.45:0/4246863261 conn(0x7f5b140cec30 msgr2=0x7f5b140cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:09:00.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.013+0000 7f5b19d2e640 1 -- 172.21.15.45:0/4246863261 shutdown_connections 2023-11-28T22:09:00.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.013+0000 7f5b19d2e640 1 -- 172.21.15.45:0/4246863261 wait complete. 2023-11-28T22:09:00.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.015+0000 7f5b19d2e640 1 Processor -- start 2023-11-28T22:09:00.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.015+0000 7f5b19d2e640 1 -- start start 2023-11-28T22:09:00.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.016+0000 7f5b19d2e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b1406c090 0x7f5b140c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.016+0000 7f5b19d2e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14070030 0x7f5b140c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.016+0000 7f5b19d2e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b1415b190 0x7f5b14152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.016+0000 7f5b19d2e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140c7f20 con 0x7f5b14070030 2023-11-28T22:09:00.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.016+0000 7f5b19d2e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140c4f40 con 0x7f5b1415b190 2023-11-28T22:09:00.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.016+0000 7f5b19d2e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140c8150 con 0x7f5b1406c090 2023-11-28T22:09:00.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.016+0000 7f5b13fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14070030 0x7f5b140c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:09:00.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.016+0000 7f5b13fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14070030 0x7f5b140c60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50150/0 (socket says 172.21.15.45:50150) 2023-11-28T22:09:00.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.016+0000 7f5b13fff640 1 -- 172.21.15.45:0/3438791159 learned_addr learned my addr 172.21.15.45:0/3438791159 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:09:00.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b13fff640 1 -- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b1406c090 msgr2=0x7f5b140c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b13fff640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b1406c090 0x7f5b140c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b13fff640 1 -- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b1415b190 msgr2=0x7f5b14152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b13fff640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b1415b190 0x7f5b14152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b13fff640 1 -- 172.21.15.45:0/3438791159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b080094d0 con 0x7f5b14070030 2023-11-28T22:09:00.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b13fff640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14070030 0x7f5b140c60c0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000d800 tx=0x7f5b0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:09:00.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b117fa640 1 -- 172.21.15.45:0/3438791159 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0000b6c0 con 0x7f5b14070030 2023-11-28T22:09:00.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b117fa640 1 -- 172.21.15.45:0/3438791159 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0000b820 con 0x7f5b14070030 2023-11-28T22:09:00.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b117fa640 1 -- 172.21.15.45:0/3438791159 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0000bbb0 con 0x7f5b14070030 2023-11-28T22:09:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b140c8430 con 0x7f5b14070030 2023-11-28T22:09:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.017+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b140c4bf0 con 0x7f5b14070030 2023-11-28T22:09:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.018+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b140733d0 con 0x7f5b14070030 2023-11-28T22:09:00.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.019+0000 7f5b117fa640 1 -- 172.21.15.45:0/3438791159 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 120) v1 ==== 89086+0+0 (secure 0 0 0) 0x7f5b0000bd10 con 0x7f5b14070030 2023-11-28T22:09:00.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.020+0000 7f5b117fa640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f5af806ad10 0x7f5af806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.020+0000 7f5b117fa640 1 -- 172.21.15.45:0/3438791159 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f5b00089b50 con 0x7f5b14070030 2023-11-28T22:09:00.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.021+0000 7f5b18d2c640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f5af806ad10 0x7f5af806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:09:00.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.022+0000 7f5b18d2c640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f5af806ad10 0x7f5af806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b08036000 tx=0x7f5b0803b040 comp rx=0 tx=0).ready entity=mgr.8797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:09:00.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.025+0000 7f5b117fa640 1 -- 172.21.15.45:0/3438791159 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b00054bb0 con 0x7f5b14070030 2023-11-28T22:09:00.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.242+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 --> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b140721d0 con 0x7f5af806ad10 2023-11-28T22:09:00.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.382+0000 7f5b117fa640 1 -- 172.21.15.45:0/3438791159 <== mgr.8797 v2:172.21.15.190:6832/3154132268 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f5b08030b10 con 0x7f5af806ad10 2023-11-28T22:09:00.385 INFO:teuthology.orchestra.run.smithi045.stdout:Retention added to path snap_test_dir1 2023-11-28T22:09:00.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f5af806ad10 msgr2=0x7f5af806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f5af806ad10 0x7f5af806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b08036000 tx=0x7f5b0803b040 comp rx=0 tx=0).stop 2023-11-28T22:09:00.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14070030 msgr2=0x7f5b140c60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14070030 0x7f5b140c60c0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000d800 tx=0x7f5b0000dcc0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 shutdown_connections 2023-11-28T22:09:00.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f5af806ad10 0x7f5af806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b1415b190 0x7f5b14152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14070030 0x7f5b140c60c0 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 --2- 172.21.15.45:0/3438791159 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5b1406c090 0x7f5b140c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 >> 172.21.15.45:0/3438791159 conn(0x7f5b140cec30 msgr2=0x7f5b140af640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:09:00.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 shutdown_connections 2023-11-28T22:09:00.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.387+0000 7f5b19d2e640 1 -- 172.21.15.45:0/3438791159 wait complete. 2023-11-28T22:09:00.411 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-11-28T22:09:00.412 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-11-28-22_10_00 in ~60s... 2023-11-28T22:09:00.412 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-11-28T22:09:00.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-11-28T22:09:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.638+0000 7f2eb7fff640 1 Processor -- start 2023-11-28T22:09:00.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.639+0000 7f2eb7fff640 1 -- start start 2023-11-28T22:09:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.640+0000 7f2eb7fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 0x7f2eb8161800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.640+0000 7f2eb7fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2eb80cfd20 0x7f2eb80b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.640+0000 7f2eb7fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb80b2930 con 0x7f2eb80b5070 2023-11-28T22:09:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.640+0000 7f2eb7fff640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb8161d40 con 0x7f2eb80cfd20 2023-11-28T22:09:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.640+0000 7f2eb7fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80b2650 con 0x7f2eb80cf940 2023-11-28T22:09:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.641+0000 7f2eae7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2eb80cfd20 0x7f2eb80b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:09:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.641+0000 7f2eb77fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2eb80b5070 0x7f2eb80d18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58654/0 (socket says 172.21.15.45:58654) 2023-11-28T22:09:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.641+0000 7f2eb77fe640 1 -- 172.21.15.45:0/1219799779 learned_addr learned my addr 172.21.15.45:0/1219799779 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:09:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.641+0000 7f2eb6ffd640 1 --2- 172.21.15.45:0/1219799779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 0x7f2eb8161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:09:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.641+0000 7f2eae7fc640 1 -- 172.21.15.45:0/1219799779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 msgr2=0x7f2eb8161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:09:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.641+0000 7f2eae7fc640 1 --2- 172.21.15.45:0/1219799779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 0x7f2eb8161800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.641+0000 7f2eae7fc640 1 -- 172.21.15.45:0/1219799779 >> v1:172.21.15.45:6789/0 conn(0x7f2eb80b5070 legacy=0x7f2eb80d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.641+0000 7f2eae7fc640 1 -- 172.21.15.45:0/1219799779 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb80ce240 con 0x7f2eb80cfd20 2023-11-28T22:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.642+0000 7f2eae7fc640 1 --2- 172.21.15.45:0/1219799779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 0x7f2eb80b4b30 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00097a0 tx=0x7f2ea002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4067385b6fe711f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.643+0000 7f2eb67fc640 1 -- 172.21.15.45:0/1219799779 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00316f0 con 0x7f2eb80cfd20 2023-11-28T22:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.643+0000 7f2eb67fc640 1 -- 172.21.15.45:0/1219799779 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0031850 con 0x7f2eb80cfd20 2023-11-28T22:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.643+0000 7f2eb67fc640 1 -- 172.21.15.45:0/1219799779 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0031be0 con 0x7f2eb80cfd20 2023-11-28T22:09:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.644+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1219799779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 msgr2=0x7f2eb80b4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.644+0000 7f2eb7fff640 1 --2- 172.21.15.45:0/1219799779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 0x7f2eb80b4b30 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00097a0 tx=0x7f2ea002b790 comp rx=0 tx=0).stop 2023-11-28T22:09:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.644+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1219799779 shutdown_connections 2023-11-28T22:09:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.644+0000 7f2eb7fff640 1 --2- 172.21.15.45:0/1219799779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 0x7f2eb80b4b30 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.644+0000 7f2eb7fff640 1 --2- 172.21.15.45:0/1219799779 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 0x7f2eb8161800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.644+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1219799779 >> 172.21.15.45:0/1219799779 conn(0x7f2eb806a500 msgr2=0x7f2eb806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:09:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.644+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1219799779 shutdown_connections 2023-11-28T22:09:00.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.644+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1219799779 wait complete. 2023-11-28T22:09:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.646+0000 7f2eb7fff640 1 Processor -- start 2023-11-28T22:09:00.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.646+0000 7f2eb7fff640 1 -- start start 2023-11-28T22:09:00.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.647+0000 7f2eb7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2eb80b5070 0x7f2eb80cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.647+0000 7f2eb7fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 0x7f2eb815d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.647+0000 7f2eb7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 0x7f2eb81e8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.647+0000 7f2eb7fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81e7cd0 con 0x7f2eb80cfd20 2023-11-28T22:09:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.647+0000 7f2eb7fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81e7fb0 con 0x7f2eb80b5070 2023-11-28T22:09:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.647+0000 7f2eb7fff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81e8290 con 0x7f2eb80cf940 2023-11-28T22:09:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.647+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 0x7f2eb81e8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:09:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.647+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 0x7f2eb81e8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50164/0 (socket says 172.21.15.45:50164) 2023-11-28T22:09:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.647+0000 7f2eb77fe640 1 -- 172.21.15.45:0/1054615036 learned_addr learned my addr 172.21.15.45:0/1054615036 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:09:00.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eae7fc640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 0x7f2eb815d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:09:00.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eb77fe640 1 -- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 msgr2=0x7f2eb815d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eb77fe640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 0x7f2eb815d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eb77fe640 1 -- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2eb80b5070 msgr2=0x7f2eb80cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eb77fe640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2eb80b5070 0x7f2eb80cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eb77fe640 1 -- 172.21.15.45:0/1054615036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea0009400 con 0x7f2eb80cfd20 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eb77fe640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 0x7f2eb81e8a70 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000caa0 tx=0x7f2eb000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eaffff640 1 -- 172.21.15.45:0/1054615036 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000ab40 con 0x7f2eb80cfd20 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eaffff640 1 -- 172.21.15.45:0/1054615036 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb000aca0 con 0x7f2eb80cfd20 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.648+0000 7f2eaffff640 1 -- 172.21.15.45:0/1054615036 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0012600 con 0x7f2eb80cfd20 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.649+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb81e8570 con 0x7f2eb80cfd20 2023-11-28T22:09:00.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.649+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb80cbe90 con 0x7f2eb80cfd20 2023-11-28T22:09:00.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.650+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb8071b30 con 0x7f2eb80cfd20 2023-11-28T22:09:00.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.651+0000 7f2eaffff640 1 -- 172.21.15.45:0/1054615036 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 121) v1 ==== 89351+0+0 (secure 0 0 0) 0x7f2eb0007070 con 0x7f2eb80cfd20 2023-11-28T22:09:00.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.652+0000 7f2eaffff640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f2e9c06afb0 0x7f2e9c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:09:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.652+0000 7f2eaffff640 1 -- 172.21.15.45:0/1054615036 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f2eb00897c0 con 0x7f2eb80cfd20 2023-11-28T22:09:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.653+0000 7f2eb6ffd640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f2e9c06afb0 0x7f2e9c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:09:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.655+0000 7f2eb6ffd640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f2e9c06afb0 0x7f2e9c06d470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8006fd0 tx=0x7f2ea8008040 comp rx=0 tx=0).ready entity=mgr.8797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:09:00.661 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.658+0000 7f2eaffff640 1 -- 172.21.15.45:0/1054615036 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2eb00545e0 con 0x7f2eb80cfd20 2023-11-28T22:09:00.881 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.878+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 --> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2eb80b2320 con 0x7f2e9c06afb0 2023-11-28T22:09:00.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.894+0000 7f2eaffff640 1 -- 172.21.15.45:0/1054615036 <== mgr.8797 v2:172.21.15.190:6832/3154132268 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f2ea800be20 con 0x7f2e9c06afb0 2023-11-28T22:09:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:09:00.896 INFO:teuthology.orchestra.run.smithi045.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-11-28T22:09:00.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f2e9c06afb0 msgr2=0x7f2e9c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f2e9c06afb0 0x7f2e9c06d470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8006fd0 tx=0x7f2ea8008040 comp rx=0 tx=0).stop 2023-11-28T22:09:00.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 msgr2=0x7f2eb81e8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:09:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 0x7f2eb81e8a70 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000caa0 tx=0x7f2eb000cf60 comp rx=0 tx=0).stop 2023-11-28T22:09:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 shutdown_connections 2023-11-28T22:09:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f2e9c06afb0 0x7f2e9c06d470 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eb80cfd20 0x7f2eb81e8a70 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2eb80cf940 0x7f2eb815d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 --2- 172.21.15.45:0/1054615036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2eb80b5070 0x7f2eb80cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:09:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.896+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 >> 172.21.15.45:0/1054615036 conn(0x7f2eb806a500 msgr2=0x7f2eb80b68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:09:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.897+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 shutdown_connections 2023-11-28T22:09:00.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:09:00.897+0000 7f2eb7fff640 1 -- 172.21.15.45:0/1054615036 wait complete. 2023-11-28T22:09:00.918 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-11-28T22:09:00.918 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-11-28T22:09:00.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:00.918 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:00.975 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:00.975 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:00.975 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:01.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:01.976 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:02.043 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:02.043 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:02.043 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:03.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:03.044 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:03.110 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:03.111 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:03.111 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:04.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:04.112 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:04.178 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:04.179 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:04.179 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:05.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:05.181 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:05.249 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:05.249 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:05.249 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:06.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:06.250 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:06.316 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:06.317 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:06.317 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:07.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:07.318 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:07.384 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:07.384 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:07.384 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:08.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:08.385 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:08.449 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:08.450 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:08.450 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:09.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:09.452 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:09.518 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:09.518 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:09.518 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:10.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:10.519 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:10.587 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:10.587 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:10.587 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:11.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:11.588 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:11.656 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:11.656 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:11.656 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:12.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:12.658 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:12.724 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:12.725 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:12.725 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:13.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:13.726 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:13.791 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:13.791 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:13.791 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:14.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:14.792 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:14.860 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:14.860 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:14.860 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:15.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:15.861 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:15.929 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:15.929 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:15.929 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:16.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:16.931 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:16.998 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:16.999 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:16.999 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:18.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:18.000 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:18.068 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:18.068 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:18.068 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:19.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:19.069 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:19.137 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:19.137 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:19.137 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:20.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:20.138 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:20.205 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:20.205 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:20.205 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:21.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:21.206 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:21.271 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:21.272 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:21.272 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:21.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:09:21.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:09:22.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:22.273 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:22.340 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:22.340 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:22.340 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:23.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:23.341 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:23.410 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:23.411 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:23.411 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:24.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:24.413 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:24.478 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:24.478 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:24.479 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:25.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:25.479 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:25.546 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:25.547 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:25.547 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:26.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:26.547 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:26.614 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:26.614 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:26.614 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:27.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:27.615 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:27.684 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:27.684 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:27.684 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:28.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:28.685 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:28.754 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:28.755 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:28.755 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:29.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:29.756 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:29.821 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:29.821 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:29.821 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:30.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:30.823 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:30.887 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:30.887 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:30.887 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:31.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:31.888 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:31.953 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:31.954 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:31.954 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:32.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:32.954 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:33.023 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:33.024 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:33.024 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:34.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:34.025 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:34.093 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:34.094 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:34.094 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:35.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:35.095 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:35.162 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:35.162 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:35.162 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:36.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:36.163 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:36.230 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:36.230 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:36.230 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:37.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:37.231 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:37.298 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:37.298 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:37.298 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:38.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:38.300 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:38.366 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:38.366 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:38.366 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:39.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:39.367 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:39.433 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:39.433 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:39.433 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:40.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:40.434 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:40.500 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:40.500 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:40.500 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:41.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:41.501 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:41.569 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:41.569 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:41.569 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:42.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:42.570 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:42.634 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:42.634 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:42.634 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:43.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:43.635 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:43.702 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:43.702 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:43.703 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:44.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:44.703 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:44.771 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:44.771 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:44.771 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:45.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:45.772 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:45.840 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:45.841 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:45.841 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:46.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:46.842 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:46.910 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:46.910 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:46.910 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:47.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:47.911 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:47.978 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:47.978 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:47.978 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:48.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:48.979 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:49.048 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:49.048 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:49.049 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:50.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:50.050 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:50.118 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:50.118 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:50.118 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:51.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:51.119 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:51.185 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:51.186 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:51.186 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:51.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:09:51.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:09:52.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:52.187 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:52.254 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:52.255 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:52.255 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:53.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:53.256 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:53.323 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:53.324 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:53.324 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:54.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:54.324 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:54.391 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:54.391 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:54.391 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:55.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:55.392 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:55.458 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:55.458 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:55.458 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:56.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:56.459 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:56.526 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:56.526 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:56.526 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:57.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:09:57.527 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:09:57.592 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-11-28T22:09:57.592 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-11-28T22:09:57.592 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:09:58.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:05.457 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-11-28T22:10:05.459 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:00.407+0000 7f6dd6f8a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:05.460 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:00.407+0000 7f6dd6f8a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:05.460 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:00.407+0000 7f6dd6f8a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:05.460 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:00.407+0000 7f6dd6f8a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:05.460 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:00.407+0000 7f6dd6f8a640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:scheduled-2023-11-28-22_10_00_UTC 2023-11-28T22:10:05.523 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-11-28-22_10_00_UTC'} 2023-11-28T22:10:05.523 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-11-28-22_10_00_UTC'} 2023-11-28T22:10:05.523 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-11-28T22:10:05.523 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-11-28-22_10_00_UTC'] 2023-11-28T22:10:05.524 DEBUG:tasks.cephfs.test_snap_schedules:exec=1701209340.4119997, now = 1701209405.5240366, timo = 60 2023-11-28T22:10:05.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-11-28T22:10:05.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.756+0000 7fd13ac39640 1 Processor -- start 2023-11-28T22:10:05.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.756+0000 7fd13ac39640 1 -- start start 2023-11-28T22:10:05.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.758+0000 7fd13ac39640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1340ac7f0 0x7fd1340710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:05.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.758+0000 7fd13ac39640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1340addf0 con 0x7fd13406dbc0 2023-11-28T22:10:05.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.758+0000 7fd13ac39640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd134070c20 con 0x7fd1340ad7e0 2023-11-28T22:10:05.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.758+0000 7fd13ac39640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1340acc80 con 0x7fd1340ac7f0 2023-11-28T22:10:05.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.758+0000 7fd139c37640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd13406dbc0 0x7fd13406dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57424/0 (socket says 172.21.15.45:57424) 2023-11-28T22:10:05.760 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.758+0000 7fd139c37640 1 -- 172.21.15.45:0/1024477191 learned_addr learned my addr 172.21.15.45:0/1024477191 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:05.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd13a438640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1340ac7f0 0x7fd1340710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022559933 0 0) 0x7fd1340addf0 con 0x7fd13406dbc0 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd120003520 con 0x7fd13406dbc0 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 743890854 0 0) 0x7fd120003520 con 0x7fd13406dbc0 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1340ac7f0 msgr2=0x7fd1340710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1340ac7f0 0x7fd1340710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 >> v1:172.21.15.190:6789/0 conn(0x7fd1340ad7e0 legacy=0x7fd1340b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1340702f0 con 0x7fd13406dbc0 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fd130001670 con 0x7fd13406dbc0 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 >> v1:172.21.15.45:6789/0 conn(0x7fd13406dbc0 legacy=0x7fd13406dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1340ac7f0 0x7fd1200039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.759+0000 7fd138c35640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd138c35640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd120004d00 0x7fd120005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12000bb40 con 0x7fd120003f70 2023-11-28T22:10:05.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12000bcb0 con 0x7fd120004d00 2023-11-28T22:10:05.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12000be70 con 0x7fd1340ac7f0 2023-11-28T22:10:05.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd139436640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:05.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd139436640 1 -- 172.21.15.45:0/1024477191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1340ac7f0 msgr2=0x7fd1200039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:05.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd139436640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1340ac7f0 0x7fd1200039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:05.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd139436640 1 -- 172.21.15.45:0/1024477191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd120004d00 msgr2=0x7fd120005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:05.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd139436640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd120004d00 0x7fd120005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:05.763 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.760+0000 7fd139436640 1 -- 172.21.15.45:0/1024477191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd120003520 con 0x7fd120003f70 2023-11-28T22:10:05.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.761+0000 7fd139436640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1200047c0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fd124001750 tx=0x7fd1200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6e4cdb133800cc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:05.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1240052d0 con 0x7fd120003f70 2023-11-28T22:10:05.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12402e050 con 0x7fd120003f70 2023-11-28T22:10:05.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd138c35640 1 -- 172.21.15.45:0/1024477191 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124003830 con 0x7fd120003f70 2023-11-28T22:10:05.764 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd13ac39640 1 -- 172.21.15.45:0/1024477191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 msgr2=0x7fd1200047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:05.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd13ac39640 1 --2- 172.21.15.45:0/1024477191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1200047c0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fd124001750 tx=0x7fd1200057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:05.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd13a438640 1 -- 172.21.15.45:0/1024477191 reap_dead start 2023-11-28T22:10:05.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd13ac39640 1 -- 172.21.15.45:0/1024477191 shutdown_connections 2023-11-28T22:10:05.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd13ac39640 1 -- 172.21.15.45:0/1024477191 >> 172.21.15.45:0/1024477191 conn(0x7fd1340cecd0 msgr2=0x7fd1340cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:05.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd13ac39640 1 -- 172.21.15.45:0/1024477191 shutdown_connections 2023-11-28T22:10:05.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.762+0000 7fd13ac39640 1 -- 172.21.15.45:0/1024477191 wait complete. 2023-11-28T22:10:05.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.764+0000 7fd13ac39640 1 Processor -- start 2023-11-28T22:10:05.765 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.765+0000 7fd13ac39640 1 -- start start 2023-11-28T22:10:05.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.765+0000 7fd13ac39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1340afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:05.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd13ac39640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd13409c3d0 0x7fd1341563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:05.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd13ac39640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1341370e0 0x7fd134137560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:05.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd13ac39640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd134137aa0 con 0x7fd120003f70 2023-11-28T22:10:05.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd13ac39640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd134163dd0 con 0x7fd13409c3d0 2023-11-28T22:10:05.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd13ac39640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13409c8b0 con 0x7fd1341370e0 2023-11-28T22:10:05.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd139c37640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1340afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:05.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd139c37640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1340afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49548/0 (socket says 172.21.15.45:49548) 2023-11-28T22:10:05.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd139c37640 1 -- 172.21.15.45:0/3645955518 learned_addr learned my addr 172.21.15.45:0/3645955518 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:05.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd139436640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd13409c3d0 0x7fd1341563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:05.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd139c37640 1 -- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1341370e0 msgr2=0x7fd134137560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:05.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd139c37640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1341370e0 0x7fd134137560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:05.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd139c37640 1 -- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd13409c3d0 msgr2=0x7fd1341563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:05.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd139c37640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd13409c3d0 0x7fd1341563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:05.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.766+0000 7fd139c37640 1 -- 172.21.15.45:0/3645955518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1340adbc0 con 0x7fd120003f70 2023-11-28T22:10:05.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.767+0000 7fd139c37640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1340afaf0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fd13000c6b0 tx=0x7fd13000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:05.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.767+0000 7fd12a7fc640 1 -- 172.21.15.45:0/3645955518 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1300072d0 con 0x7fd120003f70 2023-11-28T22:10:05.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.767+0000 7fd12a7fc640 1 -- 172.21.15.45:0/3645955518 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd130007450 con 0x7fd120003f70 2023-11-28T22:10:05.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.767+0000 7fd12a7fc640 1 -- 172.21.15.45:0/3645955518 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1300047e0 con 0x7fd120003f70 2023-11-28T22:10:05.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.767+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13409b830 con 0x7fd120003f70 2023-11-28T22:10:05.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.767+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13409bdf0 con 0x7fd120003f70 2023-11-28T22:10:05.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.768+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13409d0f0 con 0x7fd120003f70 2023-11-28T22:10:05.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.769+0000 7fd12a7fc640 1 -- 172.21.15.45:0/3645955518 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 123) v1 ==== 89464+0+0 (secure 0 0 0) 0x7fd130004960 con 0x7fd120003f70 2023-11-28T22:10:05.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.771+0000 7fd12a7fc640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fd11806b100 0x7fd11806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:05.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.771+0000 7fd12a7fc640 1 -- 172.21.15.45:0/3645955518 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6488+0+0 (secure 0 0 0) 0x7fd130089150 con 0x7fd120003f70 2023-11-28T22:10:05.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.771+0000 7fd139436640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fd11806b100 0x7fd11806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:05.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.773+0000 7fd139436640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fd11806b100 0x7fd11806d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd124033000 tx=0x7fd124009030 comp rx=0 tx=0).ready entity=mgr.8797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:05.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:05.776+0000 7fd12a7fc640 1 -- 172.21.15.45:0/3645955518 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd130053bc0 con 0x7fd120003f70 2023-11-28T22:10:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.019+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd1340b07e0 con 0x7fd120003f70 2023-11-28T22:10:06.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.550+0000 7fd12a7fc640 1 -- 172.21.15.45:0/3645955518 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v124) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd1300571e0 con 0x7fd120003f70 2023-11-28T22:10:06.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.554+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fd11806b100 msgr2=0x7fd11806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:06.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.554+0000 7fd13ac39640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fd11806b100 0x7fd11806d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd124033000 tx=0x7fd124009030 comp rx=0 tx=0).stop 2023-11-28T22:10:06.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.554+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 msgr2=0x7fd1340afaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:06.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.554+0000 7fd13ac39640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1340afaf0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fd13000c6b0 tx=0x7fd13000cb70 comp rx=0 tx=0).stop 2023-11-28T22:10:06.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.555+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 shutdown_connections 2023-11-28T22:10:06.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.555+0000 7fd13ac39640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fd11806b100 0x7fd11806d5c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:06.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.555+0000 7fd13ac39640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1341370e0 0x7fd134137560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:06.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.555+0000 7fd13ac39640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd13409c3d0 0x7fd1341563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:06.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.555+0000 7fd13ac39640 1 --2- 172.21.15.45:0/3645955518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd120003f70 0x7fd1340afaf0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:06.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.555+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 >> 172.21.15.45:0/3645955518 conn(0x7fd1340cecd0 msgr2=0x7fd1340b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:06.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.556+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 shutdown_connections 2023-11-28T22:10:06.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:06.556+0000 7fd13ac39640 1 -- 172.21.15.45:0/3645955518 wait complete. 2023-11-28T22:10:06.578 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T22:10:06.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T22:10:06.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T22:10:06.791 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T22:10:06.791 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:10:06.791 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:10:06.848 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:10:06.848 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:10:06.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:10:07.079 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:07.077+0000 7f338fc32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T22:10:07.107 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:07.106+0000 7ff865378040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T22:10:07.132 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:07.130+0000 7f338fc32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T22:10:07.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.145+0000 7f83abfff640 1 Processor -- start 2023-11-28T22:10:07.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.145+0000 7f83abfff640 1 -- start start 2023-11-28T22:10:07.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.146+0000 7f83abfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac0d02d0 0x7f83ac0d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.146+0000 7f83abfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f83ac0d1ab0 0x7f83ac0b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.147+0000 7f83abfff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac0d2230 con 0x7f83ac15d550 2023-11-28T22:10:07.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.147+0000 7f83abfff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac0d2510 con 0x7f83ac0d1ab0 2023-11-28T22:10:07.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.147+0000 7f83abfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0d27f0 con 0x7f83ac0d02d0 2023-11-28T22:10:07.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.147+0000 7f83aaffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f83ac15d550 0x7f83ac0cfbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57440/0 (socket says 172.21.15.45:57440) 2023-11-28T22:10:07.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.147+0000 7f83aaffd640 1 -- 172.21.15.45:0/4073026469 learned_addr learned my addr 172.21.15.45:0/4073026469 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.147+0000 7f83ab7fe640 1 --2- 172.21.15.45:0/4073026469 >> v2:172.21.15.190:3300/0 conn(0x7f83ac0d1ab0 0x7f83ac0b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.147+0000 7f83a3fff640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac0d02d0 0x7f83ac0d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.147+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198104000 0 0) 0x7f83ac0d2230 con 0x7f83ac15d550 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8398003520 con 0x7f83ac15d550 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1139911336 0 0) 0x7f8398003520 con 0x7f83ac15d550 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac0d02d0 msgr2=0x7f83ac0d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac0d02d0 0x7f83ac0d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 >> v2:172.21.15.190:3300/0 conn(0x7f83ac0d1ab0 msgr2=0x7f83ac0b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 --2- 172.21.15.45:0/4073026469 >> v2:172.21.15.190:3300/0 conn(0x7f83ac0d1ab0 0x7f83ac0b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac0b6a30 con 0x7f83ac15d550 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f839c0019f0 con 0x7f83ac15d550 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 >> v1:172.21.15.45:6789/0 conn(0x7f83ac15d550 legacy=0x7f83ac0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f8398003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83980047c0 0x7f8398004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac0d02d0 0x7f8398005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0d2510 con 0x7f8398003970 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0d27f0 con 0x7f83980047c0 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839800b980 con 0x7f83ac0d02d0 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.148+0000 7f83aaffd640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f8398003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.149+0000 7f83a3fff640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83980047c0 0x7f8398004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.149+0000 7f83ab7fe640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac0d02d0 0x7f8398005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.149+0000 7f83aaffd640 1 -- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac0d02d0 msgr2=0x7f8398005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:07.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.149+0000 7f83aaffd640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac0d02d0 0x7f8398005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.149+0000 7f83aaffd640 1 -- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83980047c0 msgr2=0x7f8398004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:07.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.149+0000 7f83aaffd640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83980047c0 0x7f8398004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.149+0000 7f83aaffd640 1 -- 172.21.15.45:0/4073026469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8398003520 con 0x7f8398003970 2023-11-28T22:10:07.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.149+0000 7f83a3fff640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83980047c0 0x7f8398004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:10:07.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.150+0000 7f83aaffd640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f8398003d90 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f839c00a790 tx=0x7f83980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d6087d8e36cb61b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:07.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.150+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c034630 con 0x7f8398003970 2023-11-28T22:10:07.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.150+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c02f070 con 0x7f8398003970 2023-11-28T22:10:07.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.150+0000 7f83aa7fc640 1 -- 172.21.15.45:0/4073026469 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c0349f0 con 0x7f8398003970 2023-11-28T22:10:07.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.151+0000 7f83abfff640 1 -- 172.21.15.45:0/4073026469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 msgr2=0x7f8398003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.151+0000 7f83abfff640 1 --2- 172.21.15.45:0/4073026469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f8398003d90 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f839c00a790 tx=0x7f83980057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.151+0000 7f83ab7fe640 1 -- 172.21.15.45:0/4073026469 reap_dead start 2023-11-28T22:10:07.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.151+0000 7f83abfff640 1 -- 172.21.15.45:0/4073026469 shutdown_connections 2023-11-28T22:10:07.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.151+0000 7f83abfff640 1 -- 172.21.15.45:0/4073026469 >> 172.21.15.45:0/4073026469 conn(0x7f83ac06ac70 msgr2=0x7f83ac0b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:07.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.151+0000 7f83abfff640 1 -- 172.21.15.45:0/4073026469 shutdown_connections 2023-11-28T22:10:07.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.151+0000 7f83abfff640 1 -- 172.21.15.45:0/4073026469 wait complete. 2023-11-28T22:10:07.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.153+0000 7f83abfff640 1 Processor -- start 2023-11-28T22:10:07.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.153+0000 7f83abfff640 1 -- start start 2023-11-28T22:10:07.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.154+0000 7f83abfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f83ac156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.154+0000 7f83abfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac15a000 0x7f83ac15f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.154+0000 7f83abfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83ac0b4510 0x7f83ac15f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.154+0000 7f83abfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac158090 con 0x7f8398003970 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.154+0000 7f83abfff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac1f2b10 con 0x7f83ac0b4510 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.154+0000 7f83abfff640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0c9c20 con 0x7f83ac15a000 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.154+0000 7f83aaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f83ac156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.154+0000 7f83aaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f83ac156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49590/0 (socket says 172.21.15.45:49590) 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.154+0000 7f83aaffd640 1 -- 172.21.15.45:0/752434567 learned_addr learned my addr 172.21.15.45:0/752434567 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83ab7fe640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83ac0b4510 0x7f83ac15f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83a3fff640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac15a000 0x7f83ac15f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83aaffd640 1 -- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac15a000 msgr2=0x7f83ac15f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83aaffd640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac15a000 0x7f83ac15f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83aaffd640 1 -- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83ac0b4510 msgr2=0x7f83ac15f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83aaffd640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83ac0b4510 0x7f83ac15f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83aaffd640 1 -- 172.21.15.45:0/752434567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac1577f0 con 0x7f8398003970 2023-11-28T22:10:07.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83aaffd640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f83ac156730 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f839c02cad0 tx=0x7f839c001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:07.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83a37fe640 1 -- 172.21.15.45:0/752434567 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c034630 con 0x7f8398003970 2023-11-28T22:10:07.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83a37fe640 1 -- 172.21.15.45:0/752434567 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c034790 con 0x7f8398003970 2023-11-28T22:10:07.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83a37fe640 1 -- 172.21.15.45:0/752434567 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c034b40 con 0x7f8398003970 2023-11-28T22:10:07.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.155+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac157ad0 con 0x7f8398003970 2023-11-28T22:10:07.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.156+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac0c7e90 con 0x7f8398003970 2023-11-28T22:10:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.157+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac04e920 con 0x7f8398003970 2023-11-28T22:10:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.157+0000 7f83a37fe640 1 -- 172.21.15.45:0/752434567 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 124) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f839c031030 con 0x7f8398003970 2023-11-28T22:10:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.159+0000 7f83a37fe640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f838c06b080 0x7f838c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.159+0000 7f83a37fe640 1 -- 172.21.15.45:0/752434567 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f839c0ac470 con 0x7f8398003970 2023-11-28T22:10:07.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.159+0000 7f83a3fff640 1 -- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f838c06b080 msgr2=0x7f838c06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:07.161 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.159+0000 7f83a3fff640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f838c06b080 0x7f838c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:10:07.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.164+0000 7f83a37fe640 1 -- 172.21.15.45:0/752434567 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f839c076dd0 con 0x7f8398003970 2023-11-28T22:10:07.167 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:07.165+0000 7ff865378040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T22:10:07.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.360+0000 7f83a3fff640 1 -- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f838c06b080 msgr2=0x7f838c06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:07.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.360+0000 7f83a3fff640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f838c06b080 0x7f838c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:10:07.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.364+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83ac04e920 con 0x7f8398003970 2023-11-28T22:10:07.368 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.367+0000 7f83a37fe640 1 -- 172.21.15.45:0/752434567 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) v1 ==== 74+0+17399 (secure 0 0 0) 0x7f839c0a9410 con 0x7f8398003970 2023-11-28T22:10:07.369 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:07.369 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":218,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:10:00.447350+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-11-28T22:08:41.574328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-11-28T22:08:42.812303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"215","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3310710469":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1697878697":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1107658380":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/4107859064":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/3311741793":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1352690774":"2023-11-29T22:08:58.564601+0000","172.21.15.190:6833/397715021":"2023-11-29T22:08:58.564601+0000","172.21.15.190:6832/397715021":"2023-11-29T22:08:58.564601+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:10:07.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f838c06b080 msgr2=0x7f838c06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:10:07.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f838c06b080 0x7f838c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 msgr2=0x7f83ac156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f83ac156730 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f839c02cad0 tx=0x7f839c001cb0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 shutdown_connections 2023-11-28T22:10:07.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f838c06b080 0x7f838c06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83ac0b4510 0x7f83ac15f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83ac15a000 0x7f83ac15f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 --2- 172.21.15.45:0/752434567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8398003970 0x7f83ac156730 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.372 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 >> 172.21.15.45:0/752434567 conn(0x7f83ac06ac70 msgr2=0x7f83ac0b5020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:07.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 shutdown_connections 2023-11-28T22:10:07.373 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.370+0000 7f83abfff640 1 -- 172.21.15.45:0/752434567 wait complete. 2023-11-28T22:10:07.392 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T22:08:41.574328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '214', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T22:08:42.812303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '215', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T22:10:07.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T22:10:07.581 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:07.579+0000 7f338fc32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T22:10:07.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.604+0000 7f0c04dc9640 1 Processor -- start 2023-11-28T22:10:07.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.604+0000 7f0c04dc9640 1 -- start start 2023-11-28T22:10:07.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.606+0000 7f0c04dc9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0c000d0040 0x7f0c000d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.606+0000 7f0c04dc9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c000b90a0 con 0x7f0c000b2290 2023-11-28T22:10:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.606+0000 7f0c04dc9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c000d1920 con 0x7f0c000b1d40 2023-11-28T22:10:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.606+0000 7f0c04dc9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c000d3e60 con 0x7f0c000d0040 2023-11-28T22:10:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.606+0000 7f0bff7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0c000d0040 0x7f0c000d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.606+0000 7f0bff7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0c000d0040 0x7f0c000d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49596/0 (socket says 172.21.15.45:49596) 2023-11-28T22:10:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.606+0000 7f0bff7fe640 1 -- 172.21.15.45:0/2754831006 learned_addr learned my addr 172.21.15.45:0/2754831006 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:07.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.606+0000 7f0bf7fff640 1 --1- 172.21.15.45:0/2754831006 >> v1:172.21.15.45:6789/0 conn(0x7f0c000b2290 0x7f0c000d3040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57450/0 (socket says 172.21.15.45:57450) 2023-11-28T22:10:07.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.607+0000 7f0bff7fe640 1 -- 172.21.15.45:0/2754831006 >> v1:172.21.15.190:6789/0 conn(0x7f0c000b1d40 legacy=0x7f0c000d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:07.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.607+0000 7f0bff7fe640 1 -- 172.21.15.45:0/2754831006 >> v1:172.21.15.45:6789/0 conn(0x7f0c000b2290 legacy=0x7f0c000d3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.607+0000 7f0bff7fe640 1 -- 172.21.15.45:0/2754831006 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000d2dc0 con 0x7f0c000d0040 2023-11-28T22:10:07.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.607+0000 7f0bfeffd640 1 -- 172.21.15.45:0/2754831006 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768391972 0 0) 0x7f0c000b90a0 con 0x7f0c000b2290 2023-11-28T22:10:07.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.608+0000 7f0bff7fe640 1 --2- 172.21.15.45:0/2754831006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000d0040 0x7f0c000d0420 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009870 tx=0x7f0bf002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6cc097fdff2b66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:07.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.608+0000 7f0bfeffd640 1 -- 172.21.15.45:0/2754831006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf002f840 con 0x7f0c000d0040 2023-11-28T22:10:07.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.608+0000 7f0bfeffd640 1 -- 172.21.15.45:0/2754831006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf002f9a0 con 0x7f0c000d0040 2023-11-28T22:10:07.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.608+0000 7f0bfeffd640 1 -- 172.21.15.45:0/2754831006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf002fd30 con 0x7f0c000d0040 2023-11-28T22:10:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.609+0000 7f0c04dc9640 1 -- 172.21.15.45:0/2754831006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000d0040 msgr2=0x7f0c000d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.609+0000 7f0c04dc9640 1 --2- 172.21.15.45:0/2754831006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000d0040 0x7f0c000d0420 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009870 tx=0x7f0bf002ba20 comp rx=0 tx=0).stop 2023-11-28T22:10:07.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.609+0000 7f0c04dc9640 1 -- 172.21.15.45:0/2754831006 shutdown_connections 2023-11-28T22:10:07.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.609+0000 7f0c04dc9640 1 --2- 172.21.15.45:0/2754831006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000d0040 0x7f0c000d0420 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.609+0000 7f0c04dc9640 1 -- 172.21.15.45:0/2754831006 >> 172.21.15.45:0/2754831006 conn(0x7f0c00070f40 msgr2=0x7f0c00071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:07.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.609+0000 7f0c04dc9640 1 -- 172.21.15.45:0/2754831006 shutdown_connections 2023-11-28T22:10:07.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.609+0000 7f0c04dc9640 1 -- 172.21.15.45:0/2754831006 wait complete. 2023-11-28T22:10:07.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.611+0000 7f0c04dc9640 1 Processor -- start 2023-11-28T22:10:07.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.611+0000 7f0c04dc9640 1 -- start start 2023-11-28T22:10:07.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.612+0000 7f0c04dc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000b1d40 0x7f0c000c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.612+0000 7f0c04dc9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c000b2290 0x7f0c00159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.612+0000 7f0c04dc9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c000d0040 0x7f0c001f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.612+0000 7f0c04dc9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000c85b0 con 0x7f0c000b1d40 2023-11-28T22:10:07.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.612+0000 7f0c04dc9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000c68e0 con 0x7f0c000d0040 2023-11-28T22:10:07.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.612+0000 7f0c04dc9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c00156de0 con 0x7f0c000b2290 2023-11-28T22:10:07.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.612+0000 7f0bff7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000b1d40 0x7f0c000c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:07.617 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.612+0000 7f0bff7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000b1d40 0x7f0c000c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49602/0 (socket says 172.21.15.45:49602) 2023-11-28T22:10:07.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.612+0000 7f0bff7fe640 1 -- 172.21.15.45:0/1844852155 learned_addr learned my addr 172.21.15.45:0/1844852155 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:07.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.613+0000 7f0bff7fe640 1 -- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c000b2290 msgr2=0x7f0c00159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.613+0000 7f0bff7fe640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c000b2290 0x7f0c00159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.613+0000 7f0bff7fe640 1 -- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c000d0040 msgr2=0x7f0c001f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.613+0000 7f0bff7fe640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c000d0040 0x7f0c001f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.613+0000 7f0bff7fe640 1 -- 172.21.15.45:0/1844852155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf00094d0 con 0x7f0c000b1d40 2023-11-28T22:10:07.618 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.613+0000 7f0bff7fe640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000b1d40 0x7f0c000c7380 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009870 tx=0x7f0bf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:07.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.614+0000 7f0bfcff9640 1 -- 172.21.15.45:0/1844852155 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0037d10 con 0x7f0c000b1d40 2023-11-28T22:10:07.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.614+0000 7f0bfcff9640 1 -- 172.21.15.45:0/1844852155 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf0037e70 con 0x7f0c000b1d40 2023-11-28T22:10:07.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.614+0000 7f0bfcff9640 1 -- 172.21.15.45:0/1844852155 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0041620 con 0x7f0c000b1d40 2023-11-28T22:10:07.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.614+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0015a4b0 con 0x7f0c000b1d40 2023-11-28T22:10:07.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.614+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c000c6b80 con 0x7f0c000b1d40 2023-11-28T22:10:07.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.615+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c000c6740 con 0x7f0c000b1d40 2023-11-28T22:10:07.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.616+0000 7f0bfcff9640 1 -- 172.21.15.45:0/1844852155 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 124) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f0bf0041780 con 0x7f0c000b1d40 2023-11-28T22:10:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.617+0000 7f0bfcff9640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f0be006b080 0x7f0be006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.617+0000 7f0bfcff9640 1 -- 172.21.15.45:0/1844852155 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f0bf003c070 con 0x7f0c000b1d40 2023-11-28T22:10:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.617+0000 7f0bf7fff640 1 -- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f0be006b080 msgr2=0x7f0be006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:07.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.617+0000 7f0bf7fff640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f0be006b080 0x7f0be006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:10:07.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.621+0000 7f0bfcff9640 1 -- 172.21.15.45:0/1844852155 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c000c6740 con 0x7f0c000b1d40 2023-11-28T22:10:07.635 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:07.632+0000 7f338fc32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T22:10:07.647 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:07.646+0000 7ff865378040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T22:10:07.684 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:07.682+0000 7f338fc32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T22:10:07.703 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:07.701+0000 7ff865378040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T22:10:07.745 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:07.742+0000 7f338fc32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T22:10:07.755 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:07.754+0000 7ff865378040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T22:10:07.793 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:07.791+0000 7f338fc32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T22:10:07.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.811+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c000c6740 con 0x7f0c000b1d40 2023-11-28T22:10:07.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.812+0000 7f0bfcff9640 1 -- 172.21.15.45:0/1844852155 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v218) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c000c6740 con 0x7f0c000b1d40 2023-11-28T22:10:07.813 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T22:10:07.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.814+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f0be006b080 msgr2=0x7f0be006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:10:07.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.814+0000 7f0c04dc9640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f0be006b080 0x7f0be006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000b1d40 msgr2=0x7f0c000c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:07.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000b1d40 0x7f0c000c7380 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009870 tx=0x7f0bf0004730 comp rx=0 tx=0).stop 2023-11-28T22:10:07.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 shutdown_connections 2023-11-28T22:10:07.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f0be006b080 0x7f0be006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c000d0040 0x7f0c001f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c000b2290 0x7f0c00159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 --2- 172.21.15.45:0/1844852155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c000b1d40 0x7f0c000c7380 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:07.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 >> 172.21.15.45:0/1844852155 conn(0x7f0c00070f40 msgr2=0x7f0c000b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:07.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 shutdown_connections 2023-11-28T22:10:07.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:07.815+0000 7f0c04dc9640 1 -- 172.21.15.45:0/1844852155 wait complete. 2023-11-28T22:10:07.820 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:07.819+0000 7ff865378040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T22:10:07.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T22:10:07.873 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:07.872+0000 7ff865378040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T22:10:07.903 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:07.901+0000 7f338fc32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T22:10:07.965 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:07.963+0000 7f338fc32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T22:10:07.992 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:07.991+0000 7ff865378040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T22:10:08.013 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:08.011+0000 7f338fc32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T22:10:08.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.049+0000 7f24fb8a6640 1 Processor -- start 2023-11-28T22:10:08.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.049+0000 7f24fb8a6640 1 -- start start 2023-11-28T22:10:08.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.051+0000 7f24fb8a6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f24f406a500 0x7f24f406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.051+0000 7f24fb8a6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f24f415ddd0 0x7f24f415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.051+0000 7f24fb8a6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f406ca60 con 0x7f24f4157c10 2023-11-28T22:10:08.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.051+0000 7f24fb8a6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f415eb40 con 0x7f24f406a500 2023-11-28T22:10:08.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.051+0000 7f24fb8a6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f4157ff0 con 0x7f24f415ddd0 2023-11-28T22:10:08.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.051+0000 7f24f8e41640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f24f406a500 0x7f24f406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.053 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.051+0000 7f24f8e41640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f24f406a500 0x7f24f406a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49610/0 (socket says 172.21.15.45:49610) 2023-11-28T22:10:08.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.051+0000 7f24f8e41640 1 -- 172.21.15.45:0/3430677123 learned_addr learned my addr 172.21.15.45:0/3430677123 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:08.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.052+0000 7f24f9e43640 1 --2- 172.21.15.45:0/3430677123 >> v2:172.21.15.190:3300/0 conn(0x7f24f415ddd0 0x7f24f415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.052+0000 7f24f8e41640 1 -- 172.21.15.45:0/3430677123 >> v2:172.21.15.190:3300/0 conn(0x7f24f415ddd0 msgr2=0x7f24f415e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:08.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.052+0000 7f24f8e41640 1 --2- 172.21.15.45:0/3430677123 >> v2:172.21.15.190:3300/0 conn(0x7f24f415ddd0 0x7f24f415e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.052+0000 7f24f8e41640 1 -- 172.21.15.45:0/3430677123 >> v1:172.21.15.190:6789/0 conn(0x7f24f4157c10 legacy=0x7f24f406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:08.054 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.052+0000 7f24f8e41640 1 -- 172.21.15.45:0/3430677123 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f406aea0 con 0x7f24f406a500 2023-11-28T22:10:08.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.053+0000 7f24f8e41640 1 --2- 172.21.15.45:0/3430677123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 0x7f24f406a960 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f24f40c94f0 tx=0x7f24e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c704eb030e01b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:08.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.053+0000 7f24ebfff640 1 -- 172.21.15.45:0/3430677123 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4030860 con 0x7f24f406a500 2023-11-28T22:10:08.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.053+0000 7f24ebfff640 1 -- 172.21.15.45:0/3430677123 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e40309c0 con 0x7f24f406a500 2023-11-28T22:10:08.055 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.053+0000 7f24ebfff640 1 -- 172.21.15.45:0/3430677123 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4030d50 con 0x7f24f406a500 2023-11-28T22:10:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.054+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3430677123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 msgr2=0x7f24f406a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.054+0000 7f24fb8a6640 1 --2- 172.21.15.45:0/3430677123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 0x7f24f406a960 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f24f40c94f0 tx=0x7f24e402b9a0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.054+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3430677123 shutdown_connections 2023-11-28T22:10:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.054+0000 7f24fb8a6640 1 --2- 172.21.15.45:0/3430677123 >> v2:172.21.15.190:3300/0 conn(0x7f24f415ddd0 0x7f24f415e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.054+0000 7f24fb8a6640 1 --2- 172.21.15.45:0/3430677123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 0x7f24f406a960 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.054+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3430677123 >> 172.21.15.45:0/3430677123 conn(0x7f24f40ced60 msgr2=0x7f24f40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.054+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3430677123 shutdown_connections 2023-11-28T22:10:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.054+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3430677123 wait complete. 2023-11-28T22:10:08.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.056+0000 7f24fb8a6640 1 Processor -- start 2023-11-28T22:10:08.057 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.056+0000 7f24fb8a6640 1 -- start start 2023-11-28T22:10:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.057+0000 7f24fb8a6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 0x7f24f414bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.057+0000 7f24fb8a6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24f4157c10 0x7f24f415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.057+0000 7f24fb8a6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f415ddd0 0x7f24f414c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.057+0000 7f24fb8a6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f414c470 con 0x7f24f406a500 2023-11-28T22:10:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.057+0000 7f24fb8a6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f415c570 con 0x7f24f415ddd0 2023-11-28T22:10:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.057+0000 7f24fb8a6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f40acfd0 con 0x7f24f4157c10 2023-11-28T22:10:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.057+0000 7f24f9642640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 0x7f24f414bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.057+0000 7f24f9642640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 0x7f24f414bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49624/0 (socket says 172.21.15.45:49624) 2023-11-28T22:10:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.057+0000 7f24f9642640 1 -- 172.21.15.45:0/3349829885 learned_addr learned my addr 172.21.15.45:0/3349829885 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.058+0000 7f24f9e43640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f415ddd0 0x7f24f414c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.058+0000 7f24f8e41640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24f4157c10 0x7f24f415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.058+0000 7f24f9642640 1 -- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24f4157c10 msgr2=0x7f24f415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.058+0000 7f24f9642640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24f4157c10 0x7f24f415ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.058+0000 7f24f9642640 1 -- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f415ddd0 msgr2=0x7f24f414c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.058+0000 7f24f9642640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f415ddd0 0x7f24f414c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.058+0000 7f24f9642640 1 -- 172.21.15.45:0/3349829885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e40094d0 con 0x7f24f406a500 2023-11-28T22:10:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.058+0000 7f24f9642640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 0x7f24f414bf30 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f24f000c450 tx=0x7f24f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:08.061 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:08.058+0000 7ff865378040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T22:10:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.058+0000 7f24e9ffb640 1 -- 172.21.15.45:0/3349829885 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f00052d0 con 0x7f24f406a500 2023-11-28T22:10:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.059+0000 7f24e9ffb640 1 -- 172.21.15.45:0/3349829885 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f0005430 con 0x7f24f406a500 2023-11-28T22:10:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.059+0000 7f24e9ffb640 1 -- 172.21.15.45:0/3349829885 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0012870 con 0x7f24f406a500 2023-11-28T22:10:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.059+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f40ad2b0 con 0x7f24f406a500 2023-11-28T22:10:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.059+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f415c200 con 0x7f24f406a500 2023-11-28T22:10:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.060+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f415dc40 con 0x7f24f406a500 2023-11-28T22:10:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.061+0000 7f24e9ffb640 1 -- 172.21.15.45:0/3349829885 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 124) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f24f00129d0 con 0x7f24f406a500 2023-11-28T22:10:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.062+0000 7f24e9ffb640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f24d806b080 0x7f24d806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.062+0000 7f24e9ffb640 1 -- 172.21.15.45:0/3349829885 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f24f00898b0 con 0x7f24f406a500 2023-11-28T22:10:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.062+0000 7f24f8e41640 1 -- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f24d806b080 msgr2=0x7f24d806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.062+0000 7f24f8e41640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f24d806b080 0x7f24d806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:10:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.070+0000 7f24e9ffb640 1 -- 172.21.15.45:0/3349829885 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24f00542e0 con 0x7f24f406a500 2023-11-28T22:10:08.072 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:08.070+0000 7f338fc32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T22:10:08.115 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:08.114+0000 7ff865378040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T22:10:08.122 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:08.119+0000 7f338fc32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T22:10:08.175 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:08.174+0000 7ff865378040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T22:10:08.228 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:08.226+0000 7ff865378040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T22:10:08.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.263+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f24f406a960 con 0x7f24f406a500 2023-11-28T22:10:08.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.263+0000 7f24f8e41640 1 -- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f24d806b080 msgr2=0x7f24d806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:08.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.263+0000 7f24f8e41640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f24d806b080 0x7f24d806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:10:08.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.264+0000 7f24e9ffb640 1 -- 172.21.15.45:0/3349829885 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v218) v1 ==== 104+0+11 (secure 0 0 0) 0x7f24f0057900 con 0x7f24f406a500 2023-11-28T22:10:08.265 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T22:10:08.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f24d806b080 msgr2=0x7f24d806d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:10:08.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f24d806b080 0x7f24d806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 msgr2=0x7f24f414bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:08.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 0x7f24f414bf30 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f24f000c450 tx=0x7f24f000c910 comp rx=0 tx=0).stop 2023-11-28T22:10:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 shutdown_connections 2023-11-28T22:10:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f24d806b080 0x7f24d806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f415ddd0 0x7f24f414c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24f4157c10 0x7f24f415ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 --2- 172.21.15.45:0/3349829885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f406a500 0x7f24f414bf30 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 >> 172.21.15.45:0/3349829885 conn(0x7f24f40ced60 msgr2=0x7f24f40aee50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 shutdown_connections 2023-11-28T22:10:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.266+0000 7f24fb8a6640 1 -- 172.21.15.45:0/3349829885 wait complete. 2023-11-28T22:10:08.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T22:10:08.502 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:08.499+0000 7f338fc32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T22:10:08.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.542+0000 7ff45d7d0640 1 Processor -- start 2023-11-28T22:10:08.544 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.542+0000 7ff45d7d0640 1 -- start start 2023-11-28T22:10:08.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.544+0000 7ff45d7d0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff458160aa0 0x7ff4580b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.544+0000 7ff45d7d0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580d24f0 0x7ff4580d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.544+0000 7ff45d7d0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff4580ce0e0 0x7ff4580ce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.544+0000 7ff45d7d0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff458160e80 con 0x7ff4580ce0e0 2023-11-28T22:10:08.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.544+0000 7ff45d7d0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4580d1220 con 0x7ff458160aa0 2023-11-28T22:10:08.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.544+0000 7ff45d7d0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580d2e70 con 0x7ff4580d24f0 2023-11-28T22:10:08.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff4577fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff4580ce0e0 0x7ff4580ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff4577fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff4580ce0e0 0x7ff4580ce570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49626/0 (socket says 172.21.15.45:49626) 2023-11-28T22:10:08.546 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff4577fe640 1 -- 172.21.15.45:0/3156109511 learned_addr learned my addr 172.21.15.45:0/3156109511 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:08.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff4567fc640 1 --2- 172.21.15.45:0/3156109511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580d24f0 0x7ff4580d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff456ffd640 1 --2- 172.21.15.45:0/3156109511 >> v2:172.21.15.190:3300/0 conn(0x7ff458160aa0 0x7ff4580b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff4577fe640 1 -- 172.21.15.45:0/3156109511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580d24f0 msgr2=0x7ff4580d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:08.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff4577fe640 1 --2- 172.21.15.45:0/3156109511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580d24f0 0x7ff4580d2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff4577fe640 1 -- 172.21.15.45:0/3156109511 >> v2:172.21.15.190:3300/0 conn(0x7ff458160aa0 msgr2=0x7ff4580b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:08.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff4577fe640 1 --2- 172.21.15.45:0/3156109511 >> v2:172.21.15.190:3300/0 conn(0x7ff458160aa0 0x7ff4580b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff4577fe640 1 -- 172.21.15.45:0/3156109511 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4580b6960 con 0x7ff4580ce0e0 2023-11-28T22:10:08.547 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.545+0000 7ff456ffd640 1 --2- 172.21.15.45:0/3156109511 >> v2:172.21.15.190:3300/0 conn(0x7ff458160aa0 0x7ff4580b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:10:08.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.547+0000 7ff4577fe640 1 --2- 172.21.15.45:0/3156109511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580ce0e0 0x7ff4580ce570 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7ff4580d42e0 tx=0x7ff45002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2ff4c4ae1f0898 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:08.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.547+0000 7ff455ffb640 1 -- 172.21.15.45:0/3156109511 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45002cb00 con 0x7ff4580ce0e0 2023-11-28T22:10:08.548 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.547+0000 7ff455ffb640 1 -- 172.21.15.45:0/3156109511 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45002cc60 con 0x7ff4580ce0e0 2023-11-28T22:10:08.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.547+0000 7ff455ffb640 1 -- 172.21.15.45:0/3156109511 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4500343f0 con 0x7ff4580ce0e0 2023-11-28T22:10:08.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.548+0000 7ff45d7d0640 1 -- 172.21.15.45:0/3156109511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580ce0e0 msgr2=0x7ff4580ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:08.549 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.548+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/3156109511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580ce0e0 0x7ff4580ce570 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7ff4580d42e0 tx=0x7ff45002bf60 comp rx=0 tx=0).stop 2023-11-28T22:10:08.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.548+0000 7ff45d7d0640 1 -- 172.21.15.45:0/3156109511 shutdown_connections 2023-11-28T22:10:08.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.548+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/3156109511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580ce0e0 0x7ff4580ce570 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.548+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/3156109511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580d24f0 0x7ff4580d2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.548+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/3156109511 >> v2:172.21.15.190:3300/0 conn(0x7ff458160aa0 0x7ff4580b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.548+0000 7ff45d7d0640 1 -- 172.21.15.45:0/3156109511 >> 172.21.15.45:0/3156109511 conn(0x7ff458070b20 msgr2=0x7ff458070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:08.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.548+0000 7ff45d7d0640 1 -- 172.21.15.45:0/3156109511 shutdown_connections 2023-11-28T22:10:08.550 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.548+0000 7ff45d7d0640 1 -- 172.21.15.45:0/3156109511 wait complete. 2023-11-28T22:10:08.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.550+0000 7ff45d7d0640 1 Processor -- start 2023-11-28T22:10:08.551 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.550+0000 7ff45d7d0640 1 -- start start 2023-11-28T22:10:08.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.551+0000 7ff45d7d0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580ce0e0 0x7ff4581f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.551+0000 7ff45d7d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580d24f0 0x7ff4580b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.551+0000 7ff45d7d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff458160aa0 0x7ff458156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.551+0000 7ff45d7d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580c8aa0 con 0x7ff4580d24f0 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.551+0000 7ff45d7d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580c8d80 con 0x7ff458160aa0 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.551+0000 7ff45d7d0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff458157020 con 0x7ff4580ce0e0 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.551+0000 7ff4567fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580d24f0 0x7ff4580b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4567fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580d24f0 0x7ff4580b4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49628/0 (socket says 172.21.15.45:49628) 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4567fc640 1 -- 172.21.15.45:0/1305420806 learned_addr learned my addr 172.21.15.45:0/1305420806 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff456ffd640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580ce0e0 0x7ff4581f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4577fe640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff458160aa0 0x7ff458156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4567fc640 1 -- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580ce0e0 msgr2=0x7ff4581f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4567fc640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580ce0e0 0x7ff4581f41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4567fc640 1 -- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff458160aa0 msgr2=0x7ff458156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4567fc640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff458160aa0 0x7ff458156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4567fc640 1 -- 172.21.15.45:0/1305420806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450009b30 con 0x7ff4580d24f0 2023-11-28T22:10:08.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4577fe640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff458160aa0 0x7ff458156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:08.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.552+0000 7ff4567fc640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580d24f0 0x7ff4580b4f50 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00d800 tx=0x7ff44c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:08.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.553+0000 7ff43f7fe640 1 -- 172.21.15.45:0/1305420806 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0096a0 con 0x7ff4580d24f0 2023-11-28T22:10:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.553+0000 7ff43f7fe640 1 -- 172.21.15.45:0/1305420806 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c009800 con 0x7ff4580d24f0 2023-11-28T22:10:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.553+0000 7ff43f7fe640 1 -- 172.21.15.45:0/1305420806 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c009b90 con 0x7ff4580d24f0 2023-11-28T22:10:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.553+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff458157300 con 0x7ff4580d24f0 2023-11-28T22:10:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.553+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4581f46f0 con 0x7ff4580d24f0 2023-11-28T22:10:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.554+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4580cb830 con 0x7ff4580d24f0 2023-11-28T22:10:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.555+0000 7ff43f7fe640 1 -- 172.21.15.45:0/1305420806 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 124) v1 ==== 89447+0+0 (secure 0 0 0) 0x7ff44c009cf0 con 0x7ff4580d24f0 2023-11-28T22:10:08.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.556+0000 7ff43f7fe640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff44406b080 0x7ff44406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.557 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.556+0000 7ff43f7fe640 1 -- 172.21.15.45:0/1305420806 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6488+0+0 (secure 0 0 0) 0x7ff44c08a230 con 0x7ff4580d24f0 2023-11-28T22:10:08.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.556+0000 7ff456ffd640 1 -- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff44406b080 msgr2=0x7ff44406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:08.558 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.556+0000 7ff456ffd640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff44406b080 0x7ff44406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:10:08.566 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.565+0000 7ff43f7fe640 1 -- 172.21.15.45:0/1305420806 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff44c054c60 con 0x7ff4580d24f0 2023-11-28T22:10:08.638 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:08.636+0000 7ff865378040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T22:10:08.662 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:08.659+0000 7f338fc32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T22:10:08.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.756+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff4580733d0 con 0x7ff4580d24f0 2023-11-28T22:10:08.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.757+0000 7ff43f7fe640 1 -- 172.21.15.45:0/1305420806 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v218) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff44c00b440 con 0x7ff4580d24f0 2023-11-28T22:10:08.758 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T22:10:08.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.758+0000 7ff456ffd640 1 -- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff44406b080 msgr2=0x7ff44406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:08.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.758+0000 7ff456ffd640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff44406b080 0x7ff44406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:10:08.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff44406b080 msgr2=0x7ff44406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:10:08.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff44406b080 0x7ff44406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580d24f0 msgr2=0x7ff4580b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:08.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580d24f0 0x7ff4580b4f50 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00d800 tx=0x7ff44c00dcc0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 shutdown_connections 2023-11-28T22:10:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff44406b080 0x7ff44406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff458160aa0 0x7ff458156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4580d24f0 0x7ff4580b4f50 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 --2- 172.21.15.45:0/1305420806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4580ce0e0 0x7ff4581f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.759+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 >> 172.21.15.45:0/1305420806 conn(0x7ff458070b20 msgr2=0x7ff4580ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.760+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 shutdown_connections 2023-11-28T22:10:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.760+0000 7ff45d7d0640 1 -- 172.21.15.45:0/1305420806 wait complete. 2023-11-28T22:10:08.768 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:08.766+0000 7f338fc32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T22:10:08.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T22:10:08.809 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:08.808+0000 7ff865378040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T22:10:08.817 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:08.815+0000 7f338fc32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T22:10:08.924 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:08.921+0000 7f338fc32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:10:08.924 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:08.922+0000 7ff865378040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T22:10:08.977 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:08.976+0000 7ff865378040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T22:10:08.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.986+0000 7ff15c143640 1 Processor -- start 2023-11-28T22:10:08.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.986+0000 7ff15c143640 1 -- start start 2023-11-28T22:10:08.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.988+0000 7ff15c143640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 0x7ff15406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.988+0000 7ff15c143640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1540cf020 0x7ff1540cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.988+0000 7ff15c143640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff15415edc0 0x7ff15415f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:08.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.988+0000 7ff15c143640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15415fd60 con 0x7ff1540cf020 2023-11-28T22:10:08.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.988+0000 7ff15c143640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1540cf9c0 con 0x7ff15415edc0 2023-11-28T22:10:08.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.988+0000 7ff15c143640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15415fa80 con 0x7ff15415b820 2023-11-28T22:10:08.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff1596de640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1540cf020 0x7ff1540cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff1596de640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff1540cf020 0x7ff1540cf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49642/0 (socket says 172.21.15.45:49642) 2023-11-28T22:10:08.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff1596de640 1 -- 172.21.15.45:0/719093962 learned_addr learned my addr 172.21.15.45:0/719093962 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:08.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff159edf640 1 --2- 172.21.15.45:0/719093962 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 0x7ff15406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff15a6e0640 1 --2- 172.21.15.45:0/719093962 >> v2:172.21.15.190:3300/0 conn(0x7ff15415edc0 0x7ff15415f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:08.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff1596de640 1 -- 172.21.15.45:0/719093962 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 msgr2=0x7ff15406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:08.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff1596de640 1 --2- 172.21.15.45:0/719093962 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 0x7ff15406da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff1596de640 1 -- 172.21.15.45:0/719093962 >> v2:172.21.15.190:3300/0 conn(0x7ff15415edc0 msgr2=0x7ff15415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:08.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff1596de640 1 --2- 172.21.15.45:0/719093962 >> v2:172.21.15.190:3300/0 conn(0x7ff15415edc0 0x7ff15415f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.989+0000 7ff1596de640 1 -- 172.21.15.45:0/719093962 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15406cf40 con 0x7ff1540cf020 2023-11-28T22:10:08.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.990+0000 7ff1596de640 1 --2- 172.21.15.45:0/719093962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1540cf020 0x7ff1540cf480 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7ff15406e770 tx=0x7ff14402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4459d3aab948bb7e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:08.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.990+0000 7ff158edd640 1 -- 172.21.15.45:0/719093962 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144031870 con 0x7ff1540cf020 2023-11-28T22:10:08.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff158edd640 1 -- 172.21.15.45:0/719093962 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1440319d0 con 0x7ff1540cf020 2023-11-28T22:10:08.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff158edd640 1 -- 172.21.15.45:0/719093962 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144031d60 con 0x7ff1540cf020 2023-11-28T22:10:08.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff15c143640 1 -- 172.21.15.45:0/719093962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1540cf020 msgr2=0x7ff1540cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:08.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff15c143640 1 --2- 172.21.15.45:0/719093962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1540cf020 0x7ff1540cf480 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7ff15406e770 tx=0x7ff14402b910 comp rx=0 tx=0).stop 2023-11-28T22:10:08.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff15c143640 1 -- 172.21.15.45:0/719093962 shutdown_connections 2023-11-28T22:10:08.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff15c143640 1 --2- 172.21.15.45:0/719093962 >> v2:172.21.15.190:3300/0 conn(0x7ff15415edc0 0x7ff15415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff15c143640 1 --2- 172.21.15.45:0/719093962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1540cf020 0x7ff1540cf480 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff15c143640 1 --2- 172.21.15.45:0/719093962 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 0x7ff15406da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:08.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff15c143640 1 -- 172.21.15.45:0/719093962 >> 172.21.15.45:0/719093962 conn(0x7ff15406a6e0 msgr2=0x7ff15406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:08.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff15c143640 1 -- 172.21.15.45:0/719093962 shutdown_connections 2023-11-28T22:10:08.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.991+0000 7ff15c143640 1 -- 172.21.15.45:0/719093962 wait complete. 2023-11-28T22:10:08.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.993+0000 7ff15c143640 1 Processor -- start 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.993+0000 7ff15c143640 1 -- start start 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.994+0000 7ff15c143640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1540cf020 0x7ff1541fa940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.994+0000 7ff15c143640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 0x7ff1540b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.994+0000 7ff15c143640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15415edc0 0x7ff15414f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.994+0000 7ff15c143640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541fb850 con 0x7ff15415edc0 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.994+0000 7ff15c143640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541fafb0 con 0x7ff1540cf020 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.994+0000 7ff15c143640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541fb290 con 0x7ff15415b820 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.994+0000 7ff15a6e0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15415edc0 0x7ff15414f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff15a6e0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15415edc0 0x7ff15414f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49654/0 (socket says 172.21.15.45:49654) 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff15a6e0640 1 -- 172.21.15.45:0/48627174 learned_addr learned my addr 172.21.15.45:0/48627174 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff159edf640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1540cf020 0x7ff1541fa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff1596de640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 0x7ff1540b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:09.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff15a6e0640 1 -- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 msgr2=0x7ff1540b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff15a6e0640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 0x7ff1540b36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff15a6e0640 1 -- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1540cf020 msgr2=0x7ff1541fa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff15a6e0640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1540cf020 0x7ff1541fa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff15a6e0640 1 -- 172.21.15.45:0/48627174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1440094d0 con 0x7ff15415edc0 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff159edf640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1540cf020 0x7ff1541fa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.995+0000 7ff15a6e0640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15415edc0 0x7ff15414f9d0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00d5d0 tx=0x7ff14c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.996+0000 7ff14a7fc640 1 -- 172.21.15.45:0/48627174 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c0125f0 con 0x7ff15415edc0 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.996+0000 7ff14a7fc640 1 -- 172.21.15.45:0/48627174 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c012770 con 0x7ff15415edc0 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.996+0000 7ff14a7fc640 1 -- 172.21.15.45:0/48627174 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c012b20 con 0x7ff15415edc0 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.996+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1541fb570 con 0x7ff15415edc0 2023-11-28T22:10:09.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.996+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15414ff10 con 0x7ff15415edc0 2023-11-28T22:10:09.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.997+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1541f76a0 con 0x7ff15415edc0 2023-11-28T22:10:09.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:08.999+0000 7ff14a7fc640 1 -- 172.21.15.45:0/48627174 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 124) v1 ==== 89447+0+0 (secure 0 0 0) 0x7ff14c012c80 con 0x7ff15415edc0 2023-11-28T22:10:09.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.000+0000 7ff14a7fc640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff13806b0b0 0x7ff13806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.000+0000 7ff14a7fc640 1 -- 172.21.15.45:0/48627174 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6488+0+0 (secure 0 0 0) 0x7ff14c08a200 con 0x7ff15415edc0 2023-11-28T22:10:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.000+0000 7ff159edf640 1 -- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff13806b0b0 msgr2=0x7ff13806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:09.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.000+0000 7ff159edf640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff13806b0b0 0x7ff13806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:10:09.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.008+0000 7ff14a7fc640 1 -- 172.21.15.45:0/48627174 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1541f76a0 con 0x7ff15415edc0 2023-11-28T22:10:09.031 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:09.029+0000 7f338fc32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T22:10:09.097 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:09.096+0000 7ff865378040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:10:09.187 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:09.184+0000 7f338fc32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:10:09.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.201+0000 7ff159edf640 1 -- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff13806b0b0 msgr2=0x7ff13806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:09.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.201+0000 7ff159edf640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff13806b0b0 0x7ff13806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:10:09.210 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:09.208+0000 7ff865378040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T22:10:09.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.214+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff1541f76a0 con 0x7ff15415edc0 2023-11-28T22:10:09.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.215+0000 7ff14a7fc640 1 -- 172.21.15.45:0/48627174 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v218) v1 ==== 80+0+476 (secure 0 0 0) 0x7ff1541f76a0 con 0x7ff15415edc0 2023-11-28T22:10:09.216 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3310710469 2023-11-29T22:08:58.564601+0000 2023-11-28T22:10:09.217 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1697878697 2023-11-29T22:08:58.564601+0000 2023-11-28T22:10:09.217 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1107658380 2023-11-29T22:08:58.564601+0000 2023-11-28T22:10:09.217 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/4107859064 2023-11-29T22:08:58.564601+0000 2023-11-28T22:10:09.217 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/3311741793 2023-11-29T22:08:58.564601+0000 2023-11-28T22:10:09.217 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:0/1352690774 2023-11-29T22:08:58.564601+0000 2023-11-28T22:10:09.217 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6833/397715021 2023-11-29T22:08:58.564601+0000 2023-11-28T22:10:09.217 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.190:6832/397715021 2023-11-29T22:08:58.564601+0000 2023-11-28T22:10:09.217 INFO:teuthology.orchestra.run.smithi045.stderr:listed 8 entries 2023-11-28T22:10:09.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff13806b0b0 msgr2=0x7ff13806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:10:09.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff13806b0b0 0x7ff13806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15415edc0 msgr2=0x7ff15414f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:09.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15415edc0 0x7ff15414f9d0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00d5d0 tx=0x7ff14c00da90 comp rx=0 tx=0).stop 2023-11-28T22:10:09.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 shutdown_connections 2023-11-28T22:10:09.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7ff13806b0b0 0x7ff13806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff15415edc0 0x7ff15414f9d0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff15415b820 0x7ff1540b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 --2- 172.21.15.45:0/48627174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1540cf020 0x7ff1541fa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.219+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 >> 172.21.15.45:0/48627174 conn(0x7ff15406a6e0 msgr2=0x7ff15406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:09.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.220+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 shutdown_connections 2023-11-28T22:10:09.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.220+0000 7ff15c143640 1 -- 172.21.15.45:0/48627174 wait complete. 2023-11-28T22:10:09.235 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:09.233+0000 7f338fc32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T22:10:09.240 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T22:10:09.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:09.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T22:10:09.354 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:09.352+0000 7f338fc32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T22:10:09.375 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T22:10:09.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:09.375 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T22:10:09.382 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:09.379+0000 7ff865378040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T22:10:09.422 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T22:10:09.422 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:10:09.433 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:09.432+0000 7ff865378040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T22:10:09.476 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T22:10:09.476 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:10:09.477 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:10:09.532 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T22:10:09.533 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T22:10:09.533 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:10:09.560 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:09.559+0000 7ff865378040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T22:10:09.587 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T22:10:09.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:10:09.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.798+0000 7fc09e6e1640 1 Processor -- start 2023-11-28T22:10:09.800 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.798+0000 7fc09e6e1640 1 -- start start 2023-11-28T22:10:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.800+0000 7fc09e6e1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 0x7fc0980703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.800+0000 7fc09e6e1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc098146610 0x7fc098070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.800+0000 7fc09e6e1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0980b4310 con 0x7fc098072200 2023-11-28T22:10:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.800+0000 7fc09e6e1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0980725e0 con 0x7fc098146610 2023-11-28T22:10:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.800+0000 7fc09e6e1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09806d8d0 con 0x7fc0980b24a0 2023-11-28T22:10:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.801+0000 7fc09dee0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc098146610 0x7fc098070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.801+0000 7fc09dee0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc098146610 0x7fc098070910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:42950/0 (socket says 172.21.15.45:42950) 2023-11-28T22:10:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.801+0000 7fc09dee0640 1 -- 172.21.15.45:0/3135592437 learned_addr learned my addr 172.21.15.45:0/3135592437 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.801+0000 7fc09cede640 1 --2- 172.21.15.45:0/3135592437 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 0x7fc0980703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.801+0000 7fc09dee0640 1 -- 172.21.15.45:0/3135592437 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 msgr2=0x7fc0980703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.801+0000 7fc09dee0640 1 --2- 172.21.15.45:0/3135592437 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 0x7fc0980703d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.801+0000 7fc09dee0640 1 -- 172.21.15.45:0/3135592437 >> v1:172.21.15.190:6789/0 conn(0x7fc098072200 legacy=0x7fc0980b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.801+0000 7fc09dee0640 1 -- 172.21.15.45:0/3135592437 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09806ca20 con 0x7fc098146610 2023-11-28T22:10:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.801+0000 7fc09cede640 1 --2- 172.21.15.45:0/3135592437 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 0x7fc0980703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T22:10:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.803+0000 7fc09dee0640 1 --2- 172.21.15.45:0/3135592437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc098146610 0x7fc098070910 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc090009870 tx=0x7fc09002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e482e9c88b7ce9fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.803+0000 7fc08ffff640 1 -- 172.21.15.45:0/3135592437 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090030840 con 0x7fc098146610 2023-11-28T22:10:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.803+0000 7fc08ffff640 1 -- 172.21.15.45:0/3135592437 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0900309a0 con 0x7fc098146610 2023-11-28T22:10:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.803+0000 7fc08ffff640 1 -- 172.21.15.45:0/3135592437 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090030d30 con 0x7fc098146610 2023-11-28T22:10:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.804+0000 7fc09e6e1640 1 -- 172.21.15.45:0/3135592437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc098146610 msgr2=0x7fc098070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.804+0000 7fc09e6e1640 1 --2- 172.21.15.45:0/3135592437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc098146610 0x7fc098070910 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc090009870 tx=0x7fc09002b980 comp rx=0 tx=0).stop 2023-11-28T22:10:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.804+0000 7fc09e6e1640 1 -- 172.21.15.45:0/3135592437 shutdown_connections 2023-11-28T22:10:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.804+0000 7fc09e6e1640 1 --2- 172.21.15.45:0/3135592437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc098146610 0x7fc098070910 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.804+0000 7fc09e6e1640 1 --2- 172.21.15.45:0/3135592437 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 0x7fc0980703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.804+0000 7fc09e6e1640 1 -- 172.21.15.45:0/3135592437 >> 172.21.15.45:0/3135592437 conn(0x7fc09806ab30 msgr2=0x7fc0980b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.804+0000 7fc09e6e1640 1 -- 172.21.15.45:0/3135592437 shutdown_connections 2023-11-28T22:10:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.804+0000 7fc09e6e1640 1 -- 172.21.15.45:0/3135592437 wait complete. 2023-11-28T22:10:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.806+0000 7fc09e6e1640 1 Processor -- start 2023-11-28T22:10:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.806+0000 7fc09e6e1640 1 -- start start 2023-11-28T22:10:09.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09e6e1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc098072200 0x7fc098134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09e6e1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 0x7fc09815c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09e6e1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc098146610 0x7fc0981331f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09e6e1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0981347c0 con 0x7fc098146610 2023-11-28T22:10:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09e6e1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09815cde0 con 0x7fc098072200 2023-11-28T22:10:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09e6e1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc098133b40 con 0x7fc0980b24a0 2023-11-28T22:10:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09dee0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc098146610 0x7fc0981331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09dee0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc098146610 0x7fc0981331f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49670/0 (socket says 172.21.15.45:49670) 2023-11-28T22:10:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09dee0640 1 -- 172.21.15.45:0/114288903 learned_addr learned my addr 172.21.15.45:0/114288903 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09cede640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 0x7fc09815c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.807+0000 7fc09d6df640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc098072200 0x7fc098134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc09dee0640 1 -- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 msgr2=0x7fc09815c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc09dee0640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 0x7fc09815c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc09dee0640 1 -- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc098072200 msgr2=0x7fc098134be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc09dee0640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc098072200 0x7fc098134be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc09dee0640 1 -- 172.21.15.45:0/114288903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088009400 con 0x7fc098146610 2023-11-28T22:10:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc09dee0640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc098146610 0x7fc0981331f0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fc0900090b0 tx=0x7fc0900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc08dffb640 1 -- 172.21.15.45:0/114288903 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09003e070 con 0x7fc098146610 2023-11-28T22:10:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc08dffb640 1 -- 172.21.15.45:0/114288903 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc090039a10 con 0x7fc098146610 2023-11-28T22:10:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc08dffb640 1 -- 172.21.15.45:0/114288903 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090039dc0 con 0x7fc098146610 2023-11-28T22:10:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.808+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0900094d0 con 0x7fc098146610 2023-11-28T22:10:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.809+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc098134100 con 0x7fc098146610 2023-11-28T22:10:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.810+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09809d0f0 con 0x7fc098146610 2023-11-28T22:10:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.810+0000 7fc08dffb640 1 -- 172.21.15.45:0/114288903 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 124) v1 ==== 89447+0+0 (secure 0 0 0) 0x7fc09004b400 con 0x7fc098146610 2023-11-28T22:10:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.811+0000 7fc08dffb640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc07c06b080 0x7fc07c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.811+0000 7fc08dffb640 1 -- 172.21.15.45:0/114288903 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6488+0+0 (secure 0 0 0) 0x7fc0900ac260 con 0x7fc098146610 2023-11-28T22:10:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.811+0000 7fc09d6df640 1 -- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc07c06b080 msgr2=0x7fc07c06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:09.819 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.811+0000 7fc09d6df640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc07c06b080 0x7fc07c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:10:09.821 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:09.819+0000 7fc08dffb640 1 -- 172.21.15.45:0/114288903 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc090076c30 con 0x7fc098146610 2023-11-28T22:10:10.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.012+0000 7fc09d6df640 1 -- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc07c06b080 msgr2=0x7fc07c06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:10.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.012+0000 7fc09d6df640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc07c06b080 0x7fc07c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:10:10.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.046+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc09804e920 con 0x7fc098146610 2023-11-28T22:10:10.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.047+0000 7fc08dffb640 1 -- 172.21.15.45:0/114288903 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4928 (secure 0 0 0) 0x7fc09007a250 con 0x7fc098146610 2023-11-28T22:10:10.048 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:10.049 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8692,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3311227176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3311227176},{"type":"v1","addr":"172.21.15.190:6835","nonce":3311227176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8698,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3492680695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3492680695},{"type":"v1","addr":"172.21.15.45:6835","nonce":3492680695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":8701,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3937197634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3937197634},{"type":"v1","addr":"172.21.15.190:6837","nonce":3937197634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T22:08:43.777075+0000","modified":"2023-11-28T22:08:46.880928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10650":{"gid":10650,"name":"c","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6833/3094566588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3094566588},{"type":"v1","addr":"172.21.15.45:6833","nonce":3094566588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-28T22:10:10.049 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 101 2023-11-28T22:10:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc07c06b080 msgr2=0x7fc07c06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:10:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc07c06b080 0x7fc07c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc098146610 msgr2=0x7fc0981331f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc098146610 0x7fc0981331f0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fc0900090b0 tx=0x7fc0900046c0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 shutdown_connections 2023-11-28T22:10:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7fc07c06b080 0x7fc07c06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc098146610 0x7fc0981331f0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.051 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc0980b24a0 0x7fc09815c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 --2- 172.21.15.45:0/114288903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc098072200 0x7fc098134be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.049+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 >> 172.21.15.45:0/114288903 conn(0x7fc09806ab30 msgr2=0x7fc0980b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:10.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.050+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 shutdown_connections 2023-11-28T22:10:10.052 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.050+0000 7fc09e6e1640 1 -- 172.21.15.45:0/114288903 wait complete. 2023-11-28T22:10:10.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:10:10.198 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.195+0000 7f338fc32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.277+0000 7f1bbc831640 1 Processor -- start 2023-11-28T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.277+0000 7f1bbc831640 1 -- start start 2023-11-28T22:10:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.278+0000 7f1bbc831640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb81438b0 0x7f1bb80b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.278+0000 7f1bbc831640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1bb80b24e0 0x7f1bb80b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.278+0000 7f1bbc831640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb815dcc0 con 0x7f1bb80b1170 2023-11-28T22:10:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.278+0000 7f1bbc831640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb815e840 con 0x7f1bb80b24e0 2023-11-28T22:10:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.278+0000 7f1bbc831640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80b15b0 con 0x7f1bb81438b0 2023-11-28T22:10:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.279+0000 7f1bb5d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1bb80b24e0 0x7f1bb80b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.279+0000 7f1bb659c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb81438b0 0x7f1bb80b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.279+0000 7f1bb6d9d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f1bb80b1170 0x7f1bb80ccdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:47526/0 (socket says 172.21.15.45:47526) 2023-11-28T22:10:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.279+0000 7f1bb6d9d640 1 -- 172.21.15.45:0/2098970712 learned_addr learned my addr 172.21.15.45:0/2098970712 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:10.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.280+0000 7f1bb559a640 1 -- 172.21.15.45:0/2098970712 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595499238 0 0) 0x7f1bb815dcc0 con 0x7f1bb80b1170 2023-11-28T22:10:10.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.280+0000 7f1bb559a640 1 -- 172.21.15.45:0/2098970712 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba0003520 con 0x7f1bb80b1170 2023-11-28T22:10:10.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.280+0000 7f1bb5d9b640 1 -- 172.21.15.45:0/2098970712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb81438b0 msgr2=0x7f1bb80b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:10.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.280+0000 7f1bb5d9b640 1 --2- 172.21.15.45:0/2098970712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb81438b0 0x7f1bb80b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.280+0000 7f1bb5d9b640 1 -- 172.21.15.45:0/2098970712 >> v1:172.21.15.190:6789/0 conn(0x7f1bb80b1170 legacy=0x7f1bb80ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.280+0000 7f1bb5d9b640 1 -- 172.21.15.45:0/2098970712 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb815df10 con 0x7f1bb80b24e0 2023-11-28T22:10:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.280+0000 7f1bb659c640 1 --2- 172.21.15.45:0/2098970712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb81438b0 0x7f1bb80b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T22:10:10.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.281+0000 7f1bb5d9b640 1 --2- 172.21.15.45:0/2098970712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80b24e0 0x7f1bb80b0c30 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f1bac0097a0 tx=0x7f1bac02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75211164477734f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:10.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bb559a640 1 -- 172.21.15.45:0/2098970712 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac0316f0 con 0x7f1bb80b24e0 2023-11-28T22:10:10.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bb559a640 1 -- 172.21.15.45:0/2098970712 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac031850 con 0x7f1bb80b24e0 2023-11-28T22:10:10.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bb559a640 1 -- 172.21.15.45:0/2098970712 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac031be0 con 0x7f1bb80b24e0 2023-11-28T22:10:10.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bbc831640 1 -- 172.21.15.45:0/2098970712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80b24e0 msgr2=0x7f1bb80b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bbc831640 1 --2- 172.21.15.45:0/2098970712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80b24e0 0x7f1bb80b0c30 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f1bac0097a0 tx=0x7f1bac02b790 comp rx=0 tx=0).stop 2023-11-28T22:10:10.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bbc831640 1 -- 172.21.15.45:0/2098970712 shutdown_connections 2023-11-28T22:10:10.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bbc831640 1 --2- 172.21.15.45:0/2098970712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80b24e0 0x7f1bb80b0c30 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bbc831640 1 --2- 172.21.15.45:0/2098970712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb81438b0 0x7f1bb80b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bbc831640 1 -- 172.21.15.45:0/2098970712 >> 172.21.15.45:0/2098970712 conn(0x7f1bb806a500 msgr2=0x7f1bb806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:10.291 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.282+0000 7f1bbc831640 1 -- 172.21.15.45:0/2098970712 shutdown_connections 2023-11-28T22:10:10.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.283+0000 7f1bbc831640 1 -- 172.21.15.45:0/2098970712 wait complete. 2023-11-28T22:10:10.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.284+0000 7f1bbc831640 1 Processor -- start 2023-11-28T22:10:10.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.284+0000 7f1bbc831640 1 -- start start 2023-11-28T22:10:10.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.285+0000 7f1bbc831640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb80b1170 0x7f1bb81e43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.285+0000 7f1bbc831640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80b24e0 0x7f1bb80c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.285+0000 7f1bbc831640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb81438b0 0x7f1bb8140310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.285+0000 7f1bbc831640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb81e49d0 con 0x7f1bb81438b0 2023-11-28T22:10:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.285+0000 7f1bbc831640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb81e4cb0 con 0x7f1bb80b24e0 2023-11-28T22:10:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.285+0000 7f1bbc831640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb81e4f90 con 0x7f1bb80b1170 2023-11-28T22:10:10.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb81438b0 0x7f1bb8140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb81438b0 0x7f1bb8140310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49682/0 (socket says 172.21.15.45:49682) 2023-11-28T22:10:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb6d9d640 1 -- 172.21.15.45:0/1684955327 learned_addr learned my addr 172.21.15.45:0/1684955327 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb5d9b640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80b24e0 0x7f1bb80c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb659c640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb80b1170 0x7f1bb81e43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.294 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb6d9d640 1 -- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb80b1170 msgr2=0x7f1bb81e43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb6d9d640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb80b1170 0x7f1bb81e43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb6d9d640 1 -- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80b24e0 msgr2=0x7f1bb80c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb6d9d640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80b24e0 0x7f1bb80c9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb6d9d640 1 -- 172.21.15.45:0/1684955327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba8009400 con 0x7f1bb81438b0 2023-11-28T22:10:10.295 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.286+0000 7f1bb6d9d640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb81438b0 0x7f1bb8140310 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000caa0 tx=0x7f1bb000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:10.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.287+0000 7f1ba6ffd640 1 -- 172.21.15.45:0/1684955327 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000ac00 con 0x7f1bb81438b0 2023-11-28T22:10:10.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.287+0000 7f1ba6ffd640 1 -- 172.21.15.45:0/1684955327 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb000ad60 con 0x7f1bb81438b0 2023-11-28T22:10:10.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.287+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac009400 con 0x7f1bb81438b0 2023-11-28T22:10:10.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.287+0000 7f1ba6ffd640 1 -- 172.21.15.45:0/1684955327 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0012620 con 0x7f1bb81438b0 2023-11-28T22:10:10.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.287+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb81e5550 con 0x7f1bb81438b0 2023-11-28T22:10:10.296 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.288+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb81e5740 con 0x7f1bb81438b0 2023-11-28T22:10:10.297 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.289+0000 7f1ba6ffd640 1 -- 172.21.15.45:0/1684955327 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 124) v1 ==== 89447+0+0 (secure 0 0 0) 0x7f1bb0007070 con 0x7f1bb81438b0 2023-11-28T22:10:10.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.289+0000 7f1ba6ffd640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f1b9806b0b0 0x7f1b9806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.290+0000 7f1ba6ffd640 1 -- 172.21.15.45:0/1684955327 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f1bb0089e40 con 0x7f1bb81438b0 2023-11-28T22:10:10.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.290+0000 7f1bb659c640 1 -- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f1b9806b0b0 msgr2=0x7f1b9806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:10.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.290+0000 7f1bb659c640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f1b9806b0b0 0x7f1b9806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T22:10:10.298 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.294+0000 7f1ba6ffd640 1 -- 172.21.15.45:0/1684955327 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bb0054750 con 0x7f1bb81438b0 2023-11-28T22:10:10.428 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:10.427+0000 7ff865378040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T22:10:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.487+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bb80733d0 con 0x7f1bb81438b0 2023-11-28T22:10:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.491+0000 7f1ba6ffd640 1 -- 172.21.15.45:0/1684955327 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) v1 ==== 74+0+17399 (secure 0 0 0) 0x7f1bb0017070 con 0x7f1bb81438b0 2023-11-28T22:10:10.493 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:10.493 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":218,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:10:00.447350+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-11-28T22:08:41.574328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-11-28T22:08:42.812303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"215","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3310710469":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1697878697":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1107658380":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/4107859064":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/3311741793":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1352690774":"2023-11-29T22:08:58.564601+0000","172.21.15.190:6833/397715021":"2023-11-29T22:08:58.564601+0000","172.21.15.190:6832/397715021":"2023-11-29T22:08:58.564601+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:10:10.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.491+0000 7f1bb659c640 1 -- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f1b9806b0b0 msgr2=0x7f1b9806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:6832/3154132268 2023-11-28T22:10:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.491+0000 7f1bb659c640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f1b9806b0b0 0x7f1b9806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T22:10:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f1b9806b0b0 msgr2=0x7f1b9806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T22:10:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f1b9806b0b0 0x7f1b9806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb81438b0 msgr2=0x7f1bb8140310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb81438b0 0x7f1bb8140310 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000caa0 tx=0x7f1bb000cf60 comp rx=0 tx=0).stop 2023-11-28T22:10:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 shutdown_connections 2023-11-28T22:10:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:6832/3154132268,v1:172.21.15.190:6833/3154132268] conn(0x7f1b9806b0b0 0x7f1b9806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb81438b0 0x7f1bb8140310 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80b24e0 0x7f1bb80c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 --2- 172.21.15.45:0/1684955327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bb80b1170 0x7f1bb81e43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.493+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 >> 172.21.15.45:0/1684955327 conn(0x7f1bb806a500 msgr2=0x7f1bb80cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:10.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.494+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 shutdown_connections 2023-11-28T22:10:10.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.494+0000 7f1bbc831640 1 -- 172.21.15.45:0/1684955327 wait complete. 2023-11-28T22:10:10.497 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.493+0000 7f338fc32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T22:10:10.515 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T20:33:47.196691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T22:08:41.574328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '214', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T22:08:42.812303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '215', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T22:10:10.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T22:10:10.551 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.549+0000 7f338fc32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T22:10:10.666 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.664+0000 7f338fc32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T22:10:10.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.738+0000 7f1425d00640 1 Processor -- start 2023-11-28T22:10:10.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.738+0000 7f1425d00640 1 -- start start 2023-11-28T22:10:10.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f1425d00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1420099a40 0x7f142006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f1425d00640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f14200aff90 0x7f1420099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f1425d00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14200d1fc0 con 0x7f14200cf260 2023-11-28T22:10:10.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f1425d00640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142006a910 con 0x7f14200aff90 2023-11-28T22:10:10.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f1425d00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200cb8a0 con 0x7f1420099a40 2023-11-28T22:10:10.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f141effd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f14200aff90 0x7f1420099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1420099a40 0x7f142006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1420099a40 0x7f142006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.45:33968/0 (socket says 172.21.15.45:33968) 2023-11-28T22:10:10.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f141f7fe640 1 -- 172.21.15.45:0/3158237341 learned_addr learned my addr 172.21.15.45:0/3158237341 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:10.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.740+0000 7f141ffff640 1 --1- 172.21.15.45:0/3158237341 >> v1:172.21.15.190:6789/0 conn(0x7f14200cf260 0x7f14200cf6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:47536/0 (socket says 172.21.15.45:47536) 2023-11-28T22:10:10.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.741+0000 7f141f7fe640 1 -- 172.21.15.45:0/3158237341 >> v2:172.21.15.45:3300/0 conn(0x7f14200aff90 msgr2=0x7f1420099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:10.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.741+0000 7f141f7fe640 1 --2- 172.21.15.45:0/3158237341 >> v2:172.21.15.45:3300/0 conn(0x7f14200aff90 0x7f1420099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.741+0000 7f141f7fe640 1 -- 172.21.15.45:0/3158237341 >> v1:172.21.15.190:6789/0 conn(0x7f14200cf260 legacy=0x7f14200cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.741+0000 7f141f7fe640 1 -- 172.21.15.45:0/3158237341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142009a360 con 0x7f1420099a40 2023-11-28T22:10:10.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.741+0000 7f141effd640 1 --2- 172.21.15.45:0/3158237341 >> v2:172.21.15.45:3300/0 conn(0x7f14200aff90 0x7f1420099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T22:10:10.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.742+0000 7f141f7fe640 1 --2- 172.21.15.45:0/3158237341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1420099a40 0x7f142006b4d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1410009e80 tx=0x7f141002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bed70c8c3598570f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:10.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f141e7fc640 1 -- 172.21.15.45:0/3158237341 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141002ca50 con 0x7f1420099a40 2023-11-28T22:10:10.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f141e7fc640 1 -- 172.21.15.45:0/3158237341 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141002cbb0 con 0x7f1420099a40 2023-11-28T22:10:10.745 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f141e7fc640 1 -- 172.21.15.45:0/3158237341 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14100343f0 con 0x7f1420099a40 2023-11-28T22:10:10.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f1425d00640 1 -- 172.21.15.45:0/3158237341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1420099a40 msgr2=0x7f142006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.746 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f1425d00640 1 --2- 172.21.15.45:0/3158237341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1420099a40 0x7f142006b4d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1410009e80 tx=0x7f141002beb0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f1425d00640 1 -- 172.21.15.45:0/3158237341 shutdown_connections 2023-11-28T22:10:10.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f1425d00640 1 --2- 172.21.15.45:0/3158237341 >> v2:172.21.15.45:3300/0 conn(0x7f14200aff90 0x7f1420099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f1425d00640 1 --2- 172.21.15.45:0/3158237341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1420099a40 0x7f142006b4d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f1425d00640 1 -- 172.21.15.45:0/3158237341 >> 172.21.15.45:0/3158237341 conn(0x7f14200d1020 msgr2=0x7f14200d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:10.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.743+0000 7f1425d00640 1 -- 172.21.15.45:0/3158237341 shutdown_connections 2023-11-28T22:10:10.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.744+0000 7f1425d00640 1 -- 172.21.15.45:0/3158237341 wait complete. 2023-11-28T22:10:10.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.745+0000 7f1425d00640 1 Processor -- start 2023-11-28T22:10:10.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.745+0000 7f1425d00640 1 -- start start 2023-11-28T22:10:10.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.746+0000 7f1425d00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1420099a40 0x7f1420151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.746+0000 7f1425d00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14200aff90 0x7f14200b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.746+0000 7f1425d00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14200cf260 0x7f14200c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:10.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.746+0000 7f1425d00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200b4a30 con 0x7f14200aff90 2023-11-28T22:10:10.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.746+0000 7f1425d00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14201518d0 con 0x7f1420099a40 2023-11-28T22:10:10.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.746+0000 7f1425d00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200b3e20 con 0x7f14200cf260 2023-11-28T22:10:10.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.746+0000 7f141effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14200aff90 0x7f14200b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14200aff90 0x7f14200b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49696/0 (socket says 172.21.15.45:49696) 2023-11-28T22:10:10.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141effd640 1 -- 172.21.15.45:0/2225255870 learned_addr learned my addr 172.21.15.45:0/2225255870 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:10.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141ffff640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14200cf260 0x7f14200c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141f7fe640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1420099a40 0x7f1420151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:10.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141effd640 1 -- 172.21.15.45:0/2225255870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14200cf260 msgr2=0x7f14200c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141effd640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14200cf260 0x7f14200c6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141effd640 1 -- 172.21.15.45:0/2225255870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1420099a40 msgr2=0x7f1420151390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141effd640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1420099a40 0x7f1420151390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141effd640 1 -- 172.21.15.45:0/2225255870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1408009400 con 0x7f14200aff90 2023-11-28T22:10:10.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141ffff640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14200cf260 0x7f14200c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:10:10.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f141effd640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14200aff90 0x7f14200b4ec0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f140800e960 tx=0x7f140800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:10.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f1417fff640 1 -- 172.21.15.45:0/2225255870 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140800ca00 con 0x7f14200aff90 2023-11-28T22:10:10.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.747+0000 7f1417fff640 1 -- 172.21.15.45:0/2225255870 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140800cb60 con 0x7f14200aff90 2023-11-28T22:10:10.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.748+0000 7f1425d00640 1 -- 172.21.15.45:0/2225255870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1410009b30 con 0x7f14200aff90 2023-11-28T22:10:10.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.748+0000 7f1417fff640 1 -- 172.21.15.45:0/2225255870 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1408013470 con 0x7f14200aff90 2023-11-28T22:10:10.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.748+0000 7f1425d00640 1 -- 172.21.15.45:0/2225255870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14200b43e0 con 0x7f14200aff90 2023-11-28T22:10:10.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.749+0000 7f1425d00640 1 -- 172.21.15.45:0/2225255870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14200733d0 con 0x7f14200aff90 2023-11-28T22:10:10.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.750+0000 7f1417fff640 1 -- 172.21.15.45:0/2225255870 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 125) v1 ==== 88942+0+0 (secure 0 0 0) 0x7f14080135d0 con 0x7f14200aff90 2023-11-28T22:10:10.755 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:10.751+0000 7ff865378040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T22:10:10.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.751+0000 7f1417fff640 1 -- 172.21.15.45:0/2225255870 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6918+0+0 (secure 0 0 0) 0x7f1408089860 con 0x7f14200aff90 2023-11-28T22:10:10.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.756+0000 7f1417fff640 1 -- 172.21.15.45:0/2225255870 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14080547b0 con 0x7f14200aff90 2023-11-28T22:10:10.759 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.756+0000 7f331178c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.760 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.756+0000 7f331178c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.760 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.756+0000 7f331178c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.760 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.756+0000 7f331178c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.760 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.756+0000 7f331178c640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.760 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.757+0000 7f3313f91640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.760 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.757+0000 7f3313f91640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.760 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.757+0000 7f3313f91640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.761 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.757+0000 7f3313f91640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.761 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-28T22:10:10.757+0000 7f3313f91640 -1 client.0 error registering admin socket command: (17) File exists 2023-11-28T22:10:10.814 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:10.813+0000 7ff865378040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T22:10:10.934 INFO:tasks.ceph.mgr.y.smithi045.stderr:2023-11-28T22:10:10.933+0000 7ff865378040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T22:10:10.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.948+0000 7f1425d00640 1 -- 172.21.15.45:0/2225255870 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f14200ce2f0 con 0x7f14200aff90 2023-11-28T22:10:10.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.949+0000 7f1417fff640 1 -- 172.21.15.45:0/2225255870 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1408057dd0 con 0x7f14200aff90 2023-11-28T22:10:10.951 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-11-28T22:10:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.951+0000 7f1425d00640 1 -- 172.21.15.45:0/2225255870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14200aff90 msgr2=0x7f14200b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.951+0000 7f1425d00640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14200aff90 0x7f14200b4ec0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f140800e960 tx=0x7f140800ee20 comp rx=0 tx=0).stop 2023-11-28T22:10:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.952+0000 7f1425d00640 1 -- 172.21.15.45:0/2225255870 shutdown_connections 2023-11-28T22:10:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.952+0000 7f1425d00640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14200cf260 0x7f14200c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.952+0000 7f1425d00640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14200aff90 0x7f14200b4ec0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.952+0000 7f1425d00640 1 --2- 172.21.15.45:0/2225255870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1420099a40 0x7f1420151390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.952+0000 7f1425d00640 1 -- 172.21.15.45:0/2225255870 >> 172.21.15.45:0/2225255870 conn(0x7f14200d1020 msgr2=0x7f14200b23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.952+0000 7f1425d00640 1 -- 172.21.15.45:0/2225255870 shutdown_connections 2023-11-28T22:10:10.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:10.952+0000 7f1425d00640 1 -- 172.21.15.45:0/2225255870 wait complete. 2023-11-28T22:10:10.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T22:10:11.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.184+0000 7fd8284c2640 1 Processor -- start 2023-11-28T22:10:11.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.184+0000 7fd8284c2640 1 -- start start 2023-11-28T22:10:11.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.186+0000 7fd8284c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82015b120 0x7fd8200ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.186+0000 7fd8284c2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd82006ac10 0x7fd82015ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.186+0000 7fd8284c2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200ce5c0 con 0x7fd820071ef0 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.186+0000 7fd8284c2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8200abd70 con 0x7fd82006ac10 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.186+0000 7fd8284c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8200cd2c0 con 0x7fd82015b120 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.186+0000 7fd825a5d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd820071ef0 0x7fd82006a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57454/0 (socket says 172.21.15.45:57454) 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.186+0000 7fd825a5d640 1 -- 172.21.15.45:0/1268230459 learned_addr learned my addr 172.21.15.45:0/1268230459 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.186+0000 7fd826a5f640 1 --2- 172.21.15.45:0/1268230459 >> v2:172.21.15.45:3300/0 conn(0x7fd82006ac10 0x7fd82015ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.187+0000 7fd82625e640 1 --2- 172.21.15.45:0/1268230459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82015b120 0x7fd8200ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.187+0000 7fd826a5f640 1 -- 172.21.15.45:0/1268230459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82015b120 msgr2=0x7fd8200ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.187+0000 7fd826a5f640 1 --2- 172.21.15.45:0/1268230459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82015b120 0x7fd8200ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.187+0000 7fd826a5f640 1 -- 172.21.15.45:0/1268230459 >> v1:172.21.15.45:6789/0 conn(0x7fd820071ef0 legacy=0x7fd82006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.187+0000 7fd826a5f640 1 -- 172.21.15.45:0/1268230459 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8200cc9d0 con 0x7fd82006ac10 2023-11-28T22:10:11.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.187+0000 7fd82525c640 1 -- 172.21.15.45:0/1268230459 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433583510 0 0) 0x7fd810001110 con 0x7fd820071ef0 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.188+0000 7fd826a5f640 1 --2- 172.21.15.45:0/1268230459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82006ac10 0x7fd82015ec60 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fd818009d90 tx=0x7fd81802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72015a77a8865 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd82525c640 1 -- 172.21.15.45:0/1268230459 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81802cce0 con 0x7fd82006ac10 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd82525c640 1 -- 172.21.15.45:0/1268230459 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81802ce40 con 0x7fd82006ac10 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd82525c640 1 -- 172.21.15.45:0/1268230459 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818034660 con 0x7fd82006ac10 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd8284c2640 1 -- 172.21.15.45:0/1268230459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82006ac10 msgr2=0x7fd82015ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd8284c2640 1 --2- 172.21.15.45:0/1268230459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82006ac10 0x7fd82015ec60 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fd818009d90 tx=0x7fd81802bf30 comp rx=0 tx=0).stop 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd8284c2640 1 -- 172.21.15.45:0/1268230459 shutdown_connections 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd8284c2640 1 --2- 172.21.15.45:0/1268230459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82006ac10 0x7fd82015ec60 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd8284c2640 1 --2- 172.21.15.45:0/1268230459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82015b120 0x7fd8200ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd8284c2640 1 -- 172.21.15.45:0/1268230459 >> 172.21.15.45:0/1268230459 conn(0x7fd8200ced60 msgr2=0x7fd8200cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:11.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd8284c2640 1 -- 172.21.15.45:0/1268230459 shutdown_connections 2023-11-28T22:10:11.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.189+0000 7fd8284c2640 1 -- 172.21.15.45:0/1268230459 wait complete. 2023-11-28T22:10:11.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.191+0000 7fd8284c2640 1 Processor -- start 2023-11-28T22:10:11.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.192+0000 7fd8284c2640 1 -- start start 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.192+0000 7fd8284c2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82006ac10 0x7fd82015d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd8284c2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd820071ef0 0x7fd8200c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd8284c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82015b120 0x7fd8200af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd8284c2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8200c7980 con 0x7fd82015b120 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd8284c2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8200c8220 con 0x7fd820071ef0 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd8284c2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82015d810 con 0x7fd82006ac10 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd826a5f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82015b120 0x7fd8200af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd826a5f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82015b120 0x7fd8200af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49716/0 (socket says 172.21.15.45:49716) 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd826a5f640 1 -- 172.21.15.45:0/2317219109 learned_addr learned my addr 172.21.15.45:0/2317219109 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd82625e640 1 --2- 172.21.15.45:0/2317219109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82006ac10 0x7fd82015d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd825a5d640 1 --2- 172.21.15.45:0/2317219109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd820071ef0 0x7fd8200c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd826a5f640 1 -- 172.21.15.45:0/2317219109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82006ac10 msgr2=0x7fd82015d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd826a5f640 1 --2- 172.21.15.45:0/2317219109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82006ac10 0x7fd82015d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd826a5f640 1 -- 172.21.15.45:0/2317219109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd820071ef0 msgr2=0x7fd8200c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.193+0000 7fd826a5f640 1 --2- 172.21.15.45:0/2317219109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd820071ef0 0x7fd8200c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.194+0000 7fd826a5f640 1 -- 172.21.15.45:0/2317219109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd818009a70 con 0x7fd82015b120 2023-11-28T22:10:11.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.194+0000 7fd826a5f640 1 --2- 172.21.15.45:0/2317219109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82015b120 0x7fd8200af9c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fd818009eb0 tx=0x7fd818004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:11.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.194+0000 7fd816ffd640 1 -- 172.21.15.45:0/2317219109 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81803d600 con 0x7fd82015b120 2023-11-28T22:10:11.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.194+0000 7fd816ffd640 1 -- 172.21.15.45:0/2317219109 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81803d780 con 0x7fd82015b120 2023-11-28T22:10:11.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.194+0000 7fd816ffd640 1 -- 172.21.15.45:0/2317219109 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81803db30 con 0x7fd82015b120 2023-11-28T22:10:11.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.194+0000 7fd8284c2640 1 -- 172.21.15.45:0/2317219109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8200c6cb0 con 0x7fd82015b120 2023-11-28T22:10:11.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.194+0000 7fd8284c2640 1 -- 172.21.15.45:0/2317219109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8200c7eb0 con 0x7fd82015b120 2023-11-28T22:10:11.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.196+0000 7fd8284c2640 1 -- 172.21.15.45:0/2317219109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8200733d0 con 0x7fd82015b120 2023-11-28T22:10:11.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.196+0000 7fd816ffd640 1 -- 172.21.15.45:0/2317219109 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 125) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fd81803b040 con 0x7fd82015b120 2023-11-28T22:10:11.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.198+0000 7fd816ffd640 1 -- 172.21.15.45:0/2317219109 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6918+0+0 (secure 0 0 0) 0x7fd8180abb90 con 0x7fd82015b120 2023-11-28T22:10:11.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.203+0000 7fd816ffd640 1 -- 172.21.15.45:0/2317219109 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd818076ae0 con 0x7fd82015b120 2023-11-28T22:10:11.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.399+0000 7fd8284c2640 1 -- 172.21.15.45:0/2317219109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd8200c8360 con 0x7fd82015b120 2023-11-28T22:10:11.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.400+0000 7fd816ffd640 1 -- 172.21.15.45:0/2317219109 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v219) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd818039020 con 0x7fd82015b120 2023-11-28T22:10:11.401 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-11-28T22:10:11.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.402+0000 7fd8284c2640 1 -- 172.21.15.45:0/2317219109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82015b120 msgr2=0x7fd8200af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.402+0000 7fd8284c2640 1 --2- 172.21.15.45:0/2317219109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82015b120 0x7fd8200af9c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fd818009eb0 tx=0x7fd818004a40 comp rx=0 tx=0).stop 2023-11-28T22:10:11.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.403+0000 7fd8284c2640 1 -- 172.21.15.45:0/2317219109 shutdown_connections 2023-11-28T22:10:11.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.403+0000 7fd8284c2640 1 --2- 172.21.15.45:0/2317219109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82015b120 0x7fd8200af9c0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.403+0000 7fd8284c2640 1 --2- 172.21.15.45:0/2317219109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd820071ef0 0x7fd8200c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.403+0000 7fd8284c2640 1 --2- 172.21.15.45:0/2317219109 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd82006ac10 0x7fd82015d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.403+0000 7fd8284c2640 1 -- 172.21.15.45:0/2317219109 >> 172.21.15.45:0/2317219109 conn(0x7fd8200ced60 msgr2=0x7fd8200c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.403+0000 7fd8284c2640 1 -- 172.21.15.45:0/2317219109 shutdown_connections 2023-11-28T22:10:11.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.403+0000 7fd8284c2640 1 -- 172.21.15.45:0/2317219109 wait complete. 2023-11-28T22:10:11.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.639+0000 7fb25d810640 1 Processor -- start 2023-11-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.639+0000 7fb25d810640 1 -- start start 2023-11-28T22:10:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.640+0000 7fb25d810640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb258157c10 0x7fb25806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.640+0000 7fb25d810640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb25806a500 0x7fb25806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.641+0000 7fb25d810640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25806ca60 con 0x7fb25815ddd0 2023-11-28T22:10:11.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.641+0000 7fb25d810640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb25815eb40 con 0x7fb25806a500 2023-11-28T22:10:11.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.641+0000 7fb25d810640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb258157ff0 con 0x7fb258157c10 2023-11-28T22:10:11.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.641+0000 7fb2577fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb25815ddd0 0x7fb25815e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57466/0 (socket says 172.21.15.45:57466) 2023-11-28T22:10:11.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.641+0000 7fb2577fe640 1 -- 172.21.15.45:0/3401590954 learned_addr learned my addr 172.21.15.45:0/3401590954 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:11.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.641+0000 7fb256ffd640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb258157c10 0x7fb25806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.641+0000 7fb2567fc640 1 --2- 172.21.15.45:0/3401590954 >> v2:172.21.15.190:3300/0 conn(0x7fb25806a500 0x7fb25806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245603299 0 0) 0x7fb25806ca60 con 0x7fb25815ddd0 2023-11-28T22:10:11.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb240005730 con 0x7fb25815ddd0 2023-11-28T22:10:11.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3391912343 0 0) 0x7fb240005730 con 0x7fb25815ddd0 2023-11-28T22:10:11.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb258157c10 msgr2=0x7fb25806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:11.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb258157c10 0x7fb25806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 >> v2:172.21.15.190:3300/0 conn(0x7fb25806a500 msgr2=0x7fb25806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:11.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 --2- 172.21.15.45:0/3401590954 >> v2:172.21.15.190:3300/0 conn(0x7fb25806a500 0x7fb25806a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25806aea0 con 0x7fb25815ddd0 2023-11-28T22:10:11.646 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb2500019f0 con 0x7fb25815ddd0 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 >> v1:172.21.15.45:6789/0 conn(0x7fb25815ddd0 legacy=0x7fb25815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb240005b80 0x7fb240005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2400069d0 0x7fb240006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb258157c10 0x7fb240007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25815eb40 con 0x7fb240005b80 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb258157ff0 con 0x7fb2400069d0 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24000db90 con 0x7fb258157c10 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.642+0000 7fb256ffd640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb240005b80 0x7fb240005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.643+0000 7fb2567fc640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2400069d0 0x7fb240006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.643+0000 7fb256ffd640 1 -- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb258157c10 msgr2=0x7fb240007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.643+0000 7fb256ffd640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb258157c10 0x7fb240007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.643+0000 7fb256ffd640 1 -- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2400069d0 msgr2=0x7fb240006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:11.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.643+0000 7fb256ffd640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2400069d0 0x7fb240006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.643+0000 7fb256ffd640 1 -- 172.21.15.45:0/3401590954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb240005730 con 0x7fb240005b80 2023-11-28T22:10:11.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.644+0000 7fb256ffd640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb240005b80 0x7fb240005fa0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fb248009ac0 tx=0x7fb240007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae321aaffe299a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:11.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.644+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24802c6d0 con 0x7fb240005b80 2023-11-28T22:10:11.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.644+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24802c830 con 0x7fb240005b80 2023-11-28T22:10:11.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.644+0000 7fb255ffb640 1 -- 172.21.15.45:0/3401590954 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24802cbc0 con 0x7fb240005b80 2023-11-28T22:10:11.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.645+0000 7fb25d810640 1 -- 172.21.15.45:0/3401590954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb240005b80 msgr2=0x7fb240005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.650 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.645+0000 7fb25d810640 1 --2- 172.21.15.45:0/3401590954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb240005b80 0x7fb240005fa0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fb248009ac0 tx=0x7fb240007a00 comp rx=0 tx=0).stop 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.645+0000 7fb2577fe640 1 -- 172.21.15.45:0/3401590954 reap_dead start 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.645+0000 7fb25d810640 1 -- 172.21.15.45:0/3401590954 shutdown_connections 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.645+0000 7fb25d810640 1 -- 172.21.15.45:0/3401590954 >> 172.21.15.45:0/3401590954 conn(0x7fb2580ced60 msgr2=0x7fb2580cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.645+0000 7fb25d810640 1 -- 172.21.15.45:0/3401590954 shutdown_connections 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.645+0000 7fb25d810640 1 -- 172.21.15.45:0/3401590954 wait complete. 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.647+0000 7fb25d810640 1 Processor -- start 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.647+0000 7fb25d810640 1 -- start start 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.648+0000 7fb25d810640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb240005b80 0x7fb2580ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.648+0000 7fb25d810640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580abd70 0x7fb2581e3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.648+0000 7fb25d810640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb258153fe0 0x7fb258154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.648+0000 7fb25d810640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2581549a0 con 0x7fb2580abd70 2023-11-28T22:10:11.651 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.648+0000 7fb25d810640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580ace70 con 0x7fb258153fe0 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.648+0000 7fb25d810640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580ac260 con 0x7fb240005b80 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.648+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580abd70 0x7fb2581e3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580abd70 0x7fb2581e3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49724/0 (socket says 172.21.15.45:49724) 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2567fc640 1 -- 172.21.15.45:0/417483579 learned_addr learned my addr 172.21.15.45:0/417483579 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2577fe640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb258153fe0 0x7fb258154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb256ffd640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb240005b80 0x7fb2580ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2567fc640 1 -- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb240005b80 msgr2=0x7fb2580ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2567fc640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb240005b80 0x7fb2580ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2567fc640 1 -- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb258153fe0 msgr2=0x7fb258154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2567fc640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb258153fe0 0x7fb258154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2567fc640 1 -- 172.21.15.45:0/417483579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2580ac540 con 0x7fb2580abd70 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2577fe640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb258153fe0 0x7fb258154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:11.652 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.649+0000 7fb2567fc640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580abd70 0x7fb2581e3990 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0108e0 tx=0x7fb24c010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:11.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.650+0000 7fb23ffff640 1 -- 172.21.15.45:0/417483579 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c00eb60 con 0x7fb2580abd70 2023-11-28T22:10:11.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.650+0000 7fb23ffff640 1 -- 172.21.15.45:0/417483579 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24c00ecc0 con 0x7fb2580abd70 2023-11-28T22:10:11.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.650+0000 7fb23ffff640 1 -- 172.21.15.45:0/417483579 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c01c400 con 0x7fb2580abd70 2023-11-28T22:10:11.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.650+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2580ac820 con 0x7fb2580abd70 2023-11-28T22:10:11.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.650+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2580ce7a0 con 0x7fb2580abd70 2023-11-28T22:10:11.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.651+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2580733d0 con 0x7fb2580abd70 2023-11-28T22:10:11.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.652+0000 7fb23ffff640 1 -- 172.21.15.45:0/417483579 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 125) v1 ==== 88942+0+0 (secure 0 0 0) 0x7fb24c01c560 con 0x7fb2580abd70 2023-11-28T22:10:11.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.653+0000 7fb23ffff640 1 -- 172.21.15.45:0/417483579 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6918+0+0 (secure 0 0 0) 0x7fb24c089c20 con 0x7fb2580abd70 2023-11-28T22:10:11.660 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.659+0000 7fb23ffff640 1 -- 172.21.15.45:0/417483579 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb24c054c20 con 0x7fb2580abd70 2023-11-28T22:10:11.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.690+0000 7fb23ffff640 1 -- 172.21.15.45:0/417483579 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 126) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fb24c0546d0 con 0x7fb2580abd70 2023-11-28T22:10:11.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.690+0000 7fb23ffff640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb244070da0 0x7fb244073190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:11.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.692+0000 7fb256ffd640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb244070da0 0x7fb244073190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:11.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.698+0000 7fb256ffd640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb244070da0 0x7fb244073190 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb248036000 tx=0x7fb248030000 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:11.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.865+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb2580ae600 con 0x7fb2580abd70 2023-11-28T22:10:11.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.866+0000 7fb23ffff640 1 -- 172.21.15.45:0/417483579 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v219) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb24c086a90 con 0x7fb2580abd70 2023-11-28T22:10:11.867 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2023-11-28T22:10:11.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb244070da0 msgr2=0x7fb244073190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.870 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb244070da0 0x7fb244073190 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb248036000 tx=0x7fb248030000 comp rx=0 tx=0).stop 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580abd70 msgr2=0x7fb2581e3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580abd70 0x7fb2581e3990 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0108e0 tx=0x7fb24c010da0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 shutdown_connections 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb244070da0 0x7fb244073190 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb258153fe0 0x7fb258154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580abd70 0x7fb2581e3990 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 --2- 172.21.15.45:0/417483579 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb240005b80 0x7fb2580ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 >> 172.21.15.45:0/417483579 conn(0x7fb2580ced60 msgr2=0x7fb25806da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 shutdown_connections 2023-11-28T22:10:11.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:11.869+0000 7fb25d810640 1 -- 172.21.15.45:0/417483579 wait complete. 2023-11-28T22:10:11.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:10:12.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.103+0000 7feba5010640 1 Processor -- start 2023-11-28T22:10:12.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.104+0000 7feba5010640 1 -- start start 2023-11-28T22:10:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.105+0000 7feba5010640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feba014b090 0x7feba00b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.105+0000 7feba5010640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feba00ad3b0 0x7feba00b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.105+0000 7feba5010640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba00abf10 con 0x7feba00ae590 2023-11-28T22:10:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.105+0000 7feba5010640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba006db90 con 0x7feba00ad3b0 2023-11-28T22:10:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.105+0000 7feba5010640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba006d2f0 con 0x7feba014b090 2023-11-28T22:10:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.106+0000 7feba480f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feba00ad3b0 0x7feba00b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.106+0000 7feba480f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feba00ad3b0 0x7feba00b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49740/0 (socket says 172.21.15.45:49740) 2023-11-28T22:10:12.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.106+0000 7feba480f640 1 -- 172.21.15.45:0/1372632386 learned_addr learned my addr 172.21.15.45:0/1372632386 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:12.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.106+0000 7feba480f640 1 -- 172.21.15.45:0/1372632386 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feba014b090 msgr2=0x7feba00b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:12.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.106+0000 7feba480f640 1 --2- 172.21.15.45:0/1372632386 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feba014b090 0x7feba00b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.106+0000 7feba480f640 1 -- 172.21.15.45:0/1372632386 >> v1:172.21.15.45:6789/0 conn(0x7feba00ae590 legacy=0x7feba0165250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.106+0000 7feba480f640 1 -- 172.21.15.45:0/1372632386 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba006d5d0 con 0x7feba00ad3b0 2023-11-28T22:10:12.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.107+0000 7feba480f640 1 --2- 172.21.15.45:0/1372632386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ad3b0 0x7feba00b22d0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7feb98009e80 tx=0x7feb9802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f9cdde7a8faa24 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:12.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.107+0000 7feb9f7fe640 1 -- 172.21.15.45:0/1372632386 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9802cae0 con 0x7feba00ad3b0 2023-11-28T22:10:12.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feb9f7fe640 1 -- 172.21.15.45:0/1372632386 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9802cc40 con 0x7feba00ad3b0 2023-11-28T22:10:12.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feb9f7fe640 1 -- 172.21.15.45:0/1372632386 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb980343f0 con 0x7feba00ad3b0 2023-11-28T22:10:12.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feba5010640 1 -- 172.21.15.45:0/1372632386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ad3b0 msgr2=0x7feba00b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feba5010640 1 --2- 172.21.15.45:0/1372632386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ad3b0 0x7feba00b22d0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7feb98009e80 tx=0x7feb9802bf40 comp rx=0 tx=0).stop 2023-11-28T22:10:12.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feba5010640 1 -- 172.21.15.45:0/1372632386 shutdown_connections 2023-11-28T22:10:12.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feba5010640 1 --2- 172.21.15.45:0/1372632386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ad3b0 0x7feba00b22d0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feba5010640 1 --2- 172.21.15.45:0/1372632386 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feba014b090 0x7feba00b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feba5010640 1 -- 172.21.15.45:0/1372632386 >> 172.21.15.45:0/1372632386 conn(0x7feba006ab30 msgr2=0x7feba00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:12.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feba5010640 1 -- 172.21.15.45:0/1372632386 shutdown_connections 2023-11-28T22:10:12.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.108+0000 7feba5010640 1 -- 172.21.15.45:0/1372632386 wait complete. 2023-11-28T22:10:12.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.110+0000 7feba5010640 1 Processor -- start 2023-11-28T22:10:12.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.110+0000 7feba5010640 1 -- start start 2023-11-28T22:10:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.111+0000 7feba5010640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba00ad3b0 0x7feba01eb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.111+0000 7feba5010640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ae590 0x7feba0143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.111+0000 7feba5010640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feba014b090 0x7feba01ebfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.111+0000 7feba5010640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba0143db0 con 0x7feba00ae590 2023-11-28T22:10:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.111+0000 7feba5010640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba0146a50 con 0x7feba00ad3b0 2023-11-28T22:10:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.111+0000 7feba5010640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba0143fc0 con 0x7feba014b090 2023-11-28T22:10:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.111+0000 7feb97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ae590 0x7feba0143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.111+0000 7feb97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ae590 0x7feba0143870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49750/0 (socket says 172.21.15.45:49750) 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feb97fff640 1 -- 172.21.15.45:0/2749177960 learned_addr learned my addr 172.21.15.45:0/2749177960 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feba480f640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feba014b090 0x7feba01ebfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feb9ffff640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba00ad3b0 0x7feba01eb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feb97fff640 1 -- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feba014b090 msgr2=0x7feba01ebfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feb97fff640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feba014b090 0x7feba01ebfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feb97fff640 1 -- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba00ad3b0 msgr2=0x7feba01eb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feb97fff640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba00ad3b0 0x7feba01eb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feb97fff640 1 -- 172.21.15.45:0/2749177960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98009b30 con 0x7feba00ae590 2023-11-28T22:10:12.115 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feb9ffff640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba00ad3b0 0x7feba01eb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:12.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.112+0000 7feb97fff640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ae590 0x7feba0143870 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c400 tx=0x7feb8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:12.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.113+0000 7feb9d7fa640 1 -- 172.21.15.45:0/2749177960 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb880052d0 con 0x7feba00ae590 2023-11-28T22:10:12.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.113+0000 7feb9d7fa640 1 -- 172.21.15.45:0/2749177960 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb88005430 con 0x7feba00ae590 2023-11-28T22:10:12.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.113+0000 7feb9d7fa640 1 -- 172.21.15.45:0/2749177960 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb88003830 con 0x7feba00ae590 2023-11-28T22:10:12.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.113+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba0146120 con 0x7feba00ae590 2023-11-28T22:10:12.116 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.113+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba01466e0 con 0x7feba00ae590 2023-11-28T22:10:12.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.114+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba009d0f0 con 0x7feba00ae590 2023-11-28T22:10:12.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.115+0000 7feb9d7fa640 1 -- 172.21.15.45:0/2749177960 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 126) v1 ==== 89069+0+0 (secure 0 0 0) 0x7feb88003990 con 0x7feba00ae590 2023-11-28T22:10:12.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.115+0000 7feb9d7fa640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feb8006ac90 0x7feb8006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.117 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.116+0000 7feb9d7fa640 1 -- 172.21.15.45:0/2749177960 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6918+0+0 (secure 0 0 0) 0x7feb88088d20 con 0x7feba00ae590 2023-11-28T22:10:12.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.116+0000 7feb9ffff640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feb8006ac90 0x7feb8006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:12.118 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.117+0000 7feb9ffff640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feb8006ac90 0x7feb8006d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feba009b8e0 tx=0x7feb90008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:12.121 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.120+0000 7feb9d7fa640 1 -- 172.21.15.45:0/2749177960 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb88053c50 con 0x7feba00ae590 2023-11-28T22:10:12.361 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.359+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feba01ebe00 con 0x7feba00ae590 2023-11-28T22:10:12.362 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.360+0000 7feb9d7fa640 1 -- 172.21.15.45:0/2749177960 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4928 (secure 0 0 0) 0x7feb88017090 con 0x7feba00ae590 2023-11-28T22:10:12.362 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:12.362 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8692,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3311227176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3311227176},{"type":"v1","addr":"172.21.15.190:6835","nonce":3311227176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8698,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3492680695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3492680695},{"type":"v1","addr":"172.21.15.45:6835","nonce":3492680695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":8701,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3937197634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3937197634},{"type":"v1","addr":"172.21.15.190:6837","nonce":3937197634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T22:08:43.777075+0000","modified":"2023-11-28T22:08:46.880928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10650":{"gid":10650,"name":"c","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6833/3094566588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3094566588},{"type":"v1","addr":"172.21.15.45:6833","nonce":3094566588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-28T22:10:12.363 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 101 2023-11-28T22:10:12.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feb8006ac90 msgr2=0x7feb8006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feb8006ac90 0x7feb8006d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feba009b8e0 tx=0x7feb90008040 comp rx=0 tx=0).stop 2023-11-28T22:10:12.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ae590 msgr2=0x7feba0143870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ae590 0x7feba0143870 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c400 tx=0x7feb8800c8c0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 shutdown_connections 2023-11-28T22:10:12.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feba014b090 0x7feba01ebfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.365 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba00ae590 0x7feba0143870 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feb8006ac90 0x7feb8006d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 --2- 172.21.15.45:0/2749177960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feba00ad3b0 0x7feba01eb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.363+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 >> 172.21.15.45:0/2749177960 conn(0x7feba006ab30 msgr2=0x7feba00aed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.364+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 shutdown_connections 2023-11-28T22:10:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.364+0000 7feba5010640 1 -- 172.21.15.45:0/2749177960 wait complete. 2023-11-28T22:10:12.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.601+0000 7f62213db640 1 Processor -- start 2023-11-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.601+0000 7f62213db640 1 -- start start 2023-11-28T22:10:12.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.602+0000 7f62213db640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f621c0b24a0 0x7f621c0703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f62213db640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f621c146610 0x7f621c070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f62213db640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f621c0b4310 con 0x7f621c072200 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f62213db640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f621c0725e0 con 0x7f621c146610 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f62213db640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f621c06d8d0 con 0x7f621c0b24a0 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f6220bda640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f621c146610 0x7f621c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f6220bda640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f621c146610 0x7f621c070910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49766/0 (socket says 172.21.15.45:49766) 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f6220bda640 1 -- 172.21.15.45:0/2567708960 learned_addr learned my addr 172.21.15.45:0/2567708960 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f6213fff640 1 --2- 172.21.15.45:0/2567708960 >> v2:172.21.15.190:3300/0 conn(0x7f621c0b24a0 0x7f621c0703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f6220bda640 1 -- 172.21.15.45:0/2567708960 >> v2:172.21.15.190:3300/0 conn(0x7f621c0b24a0 msgr2=0x7f621c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f6220bda640 1 --2- 172.21.15.45:0/2567708960 >> v2:172.21.15.190:3300/0 conn(0x7f621c0b24a0 0x7f621c0703d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f6220bda640 1 -- 172.21.15.45:0/2567708960 >> v1:172.21.15.190:6789/0 conn(0x7f621c072200 legacy=0x7f621c0b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:12.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.603+0000 7f6220bda640 1 -- 172.21.15.45:0/2567708960 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c06ca20 con 0x7f621c146610 2023-11-28T22:10:12.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.604+0000 7f6220bda640 1 --2- 172.21.15.45:0/2567708960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c146610 0x7f621c070910 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f621c09d0c0 tx=0x7f621402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8253abc47faf30 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:12.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.605+0000 7f621b7fe640 1 -- 172.21.15.45:0/2567708960 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621402ca80 con 0x7f621c146610 2023-11-28T22:10:12.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.605+0000 7f621b7fe640 1 -- 172.21.15.45:0/2567708960 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621402cbe0 con 0x7f621c146610 2023-11-28T22:10:12.606 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.605+0000 7f621b7fe640 1 -- 172.21.15.45:0/2567708960 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62140343f0 con 0x7f621c146610 2023-11-28T22:10:12.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.605+0000 7f62213db640 1 -- 172.21.15.45:0/2567708960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c146610 msgr2=0x7f621c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.605+0000 7f62213db640 1 --2- 172.21.15.45:0/2567708960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c146610 0x7f621c070910 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f621c09d0c0 tx=0x7f621402bee0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.606+0000 7f62213db640 1 -- 172.21.15.45:0/2567708960 shutdown_connections 2023-11-28T22:10:12.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.606+0000 7f62213db640 1 --2- 172.21.15.45:0/2567708960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c146610 0x7f621c070910 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.606+0000 7f62213db640 1 --2- 172.21.15.45:0/2567708960 >> v2:172.21.15.190:3300/0 conn(0x7f621c0b24a0 0x7f621c0703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.606+0000 7f62213db640 1 -- 172.21.15.45:0/2567708960 >> 172.21.15.45:0/2567708960 conn(0x7f621c06ab30 msgr2=0x7f621c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:12.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.606+0000 7f62213db640 1 -- 172.21.15.45:0/2567708960 shutdown_connections 2023-11-28T22:10:12.608 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.606+0000 7f62213db640 1 -- 172.21.15.45:0/2567708960 wait complete. 2023-11-28T22:10:12.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.608+0000 7f62213db640 1 Processor -- start 2023-11-28T22:10:12.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.608+0000 7f62213db640 1 -- start start 2023-11-28T22:10:12.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f62213db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c072200 0x7f621c15eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f62213db640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f621c0b24a0 0x7f621c1601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f62213db640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f621c146610 0x7f621c1371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f62213db640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c15f400 con 0x7f621c072200 2023-11-28T22:10:12.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f62213db640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c141c60 con 0x7f621c146610 2023-11-28T22:10:12.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f62213db640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621c15de20 con 0x7f621c0b24a0 2023-11-28T22:10:12.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f621bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c072200 0x7f621c15eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:12.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f621bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c072200 0x7f621c15eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49780/0 (socket says 172.21.15.45:49780) 2023-11-28T22:10:12.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f621bfff640 1 -- 172.21.15.45:0/1529470379 learned_addr learned my addr 172.21.15.45:0/1529470379 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f621bfff640 1 -- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f621c0b24a0 msgr2=0x7f621c1601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f621bfff640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f621c0b24a0 0x7f621c1601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f621bfff640 1 -- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f621c146610 msgr2=0x7f621c1371c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f621bfff640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f621c146610 0x7f621c1371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.609+0000 7f621bfff640 1 -- 172.21.15.45:0/1529470379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6214009b30 con 0x7f621c072200 2023-11-28T22:10:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.610+0000 7f621bfff640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c072200 0x7f621c15eec0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f620c00c430 tx=0x7f620c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.610+0000 7f62197fa640 1 -- 172.21.15.45:0/1529470379 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c011790 con 0x7f621c072200 2023-11-28T22:10:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.610+0000 7f62197fa640 1 -- 172.21.15.45:0/1529470379 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620c0118f0 con 0x7f621c072200 2023-11-28T22:10:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.610+0000 7f62197fa640 1 -- 172.21.15.45:0/1529470379 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c011ca0 con 0x7f621c072200 2023-11-28T22:10:12.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.610+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f621c15e100 con 0x7f621c072200 2023-11-28T22:10:12.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.610+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621c15e6c0 con 0x7f621c072200 2023-11-28T22:10:12.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.612+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621c09d0f0 con 0x7f621c072200 2023-11-28T22:10:12.613 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.613+0000 7f62197fa640 1 -- 172.21.15.45:0/1529470379 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 126) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f620c01c460 con 0x7f621c072200 2023-11-28T22:10:12.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.613+0000 7f62197fa640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f61fc06acc0 0x7f61fc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:12.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.614+0000 7f62197fa640 1 -- 172.21.15.45:0/1529470379 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6918+0+0 (secure 0 0 0) 0x7f620c089660 con 0x7f621c072200 2023-11-28T22:10:12.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.614+0000 7f6213fff640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f61fc06acc0 0x7f61fc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:12.616 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.615+0000 7f6213fff640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f61fc06acc0 0x7f61fc06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f621c142340 tx=0x7f6204008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:12.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.618+0000 7f62197fa640 1 -- 172.21.15.45:0/1529470379 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f620c054590 con 0x7f621c072200 2023-11-28T22:10:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.705+0000 7f62197fa640 1 -- 172.21.15.45:0/1529470379 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 127) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f620c054960 con 0x7f621c072200 2023-11-28T22:10:12.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.826+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f621c04e920 con 0x7f621c072200 2023-11-28T22:10:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.827+0000 7f62197fa640 1 -- 172.21.15.45:0/1529470379 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+18034 (secure 0 0 0) 0x7f620c087510 con 0x7f621c072200 2023-11-28T22:10:12.829 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:12.829 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":219,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:10:10.669328+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-11-28T22:08:41.574328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-11-28T22:08:42.812303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"215","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3465043952":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/1255156042":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/964161599":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/1168814133":"2023-11-29T22:10:10.669273+0000","172.21.15.190:6832/3154132268":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/3277311498":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/3310710469":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1697878697":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/2564313259":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/1107658380":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/4107859064":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/3311741793":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1352690774":"2023-11-29T22:08:58.564601+0000","172.21.15.190:6833/3154132268":"2023-11-29T22:10:10.669273+0000","172.21.15.190:6833/397715021":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1385562856":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/3670565434":"2023-11-29T22:10:10.669273+0000","172.21.15.190:6832/397715021":"2023-11-29T22:08:58.564601+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:10:12.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f61fc06acc0 msgr2=0x7f61fc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f61fc06acc0 0x7f61fc06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f621c142340 tx=0x7f6204008040 comp rx=0 tx=0).stop 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c072200 msgr2=0x7f621c15eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c072200 0x7f621c15eec0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f620c00c430 tx=0x7f620c00c8f0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 shutdown_connections 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f61fc06acc0 0x7f61fc06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f621c146610 0x7f621c1371c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f621c0b24a0 0x7f621c1601a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 --2- 172.21.15.45:0/1529470379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c072200 0x7f621c15eec0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 >> 172.21.15.45:0/1529470379 conn(0x7f621c06ab30 msgr2=0x7f621c0abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 shutdown_connections 2023-11-28T22:10:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:12.830+0000 7f62213db640 1 -- 172.21.15.45:0/1529470379 wait complete. 2023-11-28T22:10:12.852 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T22:10:12.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:10:13.064 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.063+0000 7f3d974f1640 1 Processor -- start 2023-11-28T22:10:13.065 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.063+0000 7f3d974f1640 1 -- start start 2023-11-28T22:10:13.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.065+0000 7f3d974f1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3d900721e0 0x7f3d9006d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.066 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.065+0000 7f3d974f1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9006d6b0 0x7f3d900ac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.065+0000 7f3d974f1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9015e680 con 0x7f3d900b3870 2023-11-28T22:10:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.065+0000 7f3d974f1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9015a460 con 0x7f3d900721e0 2023-11-28T22:10:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.065+0000 7f3d974f1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9006db40 con 0x7f3d9006d6b0 2023-11-28T22:10:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.065+0000 7f3d95cee640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3d900721e0 0x7f3d9006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.065+0000 7f3d95cee640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3d900721e0 0x7f3d9006d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49788/0 (socket says 172.21.15.45:49788) 2023-11-28T22:10:13.067 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.065+0000 7f3d95cee640 1 -- 172.21.15.45:0/1601381657 learned_addr learned my addr 172.21.15.45:0/1601381657 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:13.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.066+0000 7f3d96cf0640 1 --2- 172.21.15.45:0/1601381657 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9006d6b0 0x7f3d900ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.066+0000 7f3d95cee640 1 -- 172.21.15.45:0/1601381657 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9006d6b0 msgr2=0x7f3d900ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.066+0000 7f3d95cee640 1 --2- 172.21.15.45:0/1601381657 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9006d6b0 0x7f3d900ac840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.066+0000 7f3d95cee640 1 -- 172.21.15.45:0/1601381657 >> v1:172.21.15.45:6789/0 conn(0x7f3d900b3870 legacy=0x7f3d9015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.066+0000 7f3d95cee640 1 -- 172.21.15.45:0/1601381657 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9015e960 con 0x7f3d900721e0 2023-11-28T22:10:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.066+0000 7f3d96cf0640 1 --2- 172.21.15.45:0/1601381657 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9006d6b0 0x7f3d900ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.067+0000 7f3d95cee640 1 --2- 172.21.15.45:0/1601381657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d900721e0 0x7f3d9006d170 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009870 tx=0x7f3d8002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9349ed75e7e468bc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.067+0000 7f3d954ed640 1 -- 172.21.15.45:0/1601381657 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80030840 con 0x7f3d900721e0 2023-11-28T22:10:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.067+0000 7f3d954ed640 1 -- 172.21.15.45:0/1601381657 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d800309a0 con 0x7f3d900721e0 2023-11-28T22:10:13.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.067+0000 7f3d954ed640 1 -- 172.21.15.45:0/1601381657 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80030d30 con 0x7f3d900721e0 2023-11-28T22:10:13.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.068+0000 7f3d974f1640 1 -- 172.21.15.45:0/1601381657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d900721e0 msgr2=0x7f3d9006d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.068+0000 7f3d974f1640 1 --2- 172.21.15.45:0/1601381657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d900721e0 0x7f3d9006d170 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009870 tx=0x7f3d8002b980 comp rx=0 tx=0).stop 2023-11-28T22:10:13.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.068+0000 7f3d974f1640 1 -- 172.21.15.45:0/1601381657 shutdown_connections 2023-11-28T22:10:13.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.068+0000 7f3d974f1640 1 --2- 172.21.15.45:0/1601381657 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d9006d6b0 0x7f3d900ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.068+0000 7f3d974f1640 1 --2- 172.21.15.45:0/1601381657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d900721e0 0x7f3d9006d170 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.068+0000 7f3d974f1640 1 -- 172.21.15.45:0/1601381657 >> 172.21.15.45:0/1601381657 conn(0x7f3d9006ab30 msgr2=0x7f3d900b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:13.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.068+0000 7f3d974f1640 1 -- 172.21.15.45:0/1601381657 shutdown_connections 2023-11-28T22:10:13.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.068+0000 7f3d974f1640 1 -- 172.21.15.45:0/1601381657 wait complete. 2023-11-28T22:10:13.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.070+0000 7f3d974f1640 1 Processor -- start 2023-11-28T22:10:13.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.070+0000 7f3d974f1640 1 -- start start 2023-11-28T22:10:13.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.071+0000 7f3d974f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9006d6b0 0x7f3d90099dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.071+0000 7f3d974f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d900721e0 0x7f3d901460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.071+0000 7f3d974f1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d900b3870 0x7f3d900cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.071+0000 7f3d974f1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90143380 con 0x7f3d9006d6b0 2023-11-28T22:10:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.071+0000 7f3d974f1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90143610 con 0x7f3d900721e0 2023-11-28T22:10:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.071+0000 7f3d974f1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d901438f0 con 0x7f3d900b3870 2023-11-28T22:10:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.071+0000 7f3d964ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9006d6b0 0x7f3d90099dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.071+0000 7f3d964ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9006d6b0 0x7f3d90099dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49792/0 (socket says 172.21.15.45:49792) 2023-11-28T22:10:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.072+0000 7f3d964ef640 1 -- 172.21.15.45:0/3399558199 learned_addr learned my addr 172.21.15.45:0/3399558199 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:13.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.072+0000 7f3d95cee640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d900721e0 0x7f3d901460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.072+0000 7f3d964ef640 1 -- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d900b3870 msgr2=0x7f3d900cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.072+0000 7f3d964ef640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d900b3870 0x7f3d900cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.072+0000 7f3d964ef640 1 -- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d900721e0 msgr2=0x7f3d901460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.072+0000 7f3d964ef640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d900721e0 0x7f3d901460b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.072+0000 7f3d964ef640 1 -- 172.21.15.45:0/3399558199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d800094d0 con 0x7f3d9006d6b0 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.072+0000 7f3d95cee640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d900721e0 0x7f3d901460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.072+0000 7f3d964ef640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9006d6b0 0x7f3d90099dc0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00c400 tx=0x7f3d8c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.073+0000 7f3d86ffd640 1 -- 172.21.15.45:0/3399558199 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c0052d0 con 0x7f3d9006d6b0 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.073+0000 7f3d86ffd640 1 -- 172.21.15.45:0/3399558199 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c005430 con 0x7f3d9006d6b0 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.073+0000 7f3d86ffd640 1 -- 172.21.15.45:0/3399558199 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c003830 con 0x7f3d9006d6b0 2023-11-28T22:10:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.073+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d901465f0 con 0x7f3d9006d6b0 2023-11-28T22:10:13.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.073+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d900c81b0 con 0x7f3d9006d6b0 2023-11-28T22:10:13.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.074+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d900ac420 con 0x7f3d9006d6b0 2023-11-28T22:10:13.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.075+0000 7f3d86ffd640 1 -- 172.21.15.45:0/3399558199 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 127) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f3d8c003990 con 0x7f3d9006d6b0 2023-11-28T22:10:13.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.076+0000 7f3d86ffd640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3d7406aed0 0x7f3d7406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.076+0000 7f3d86ffd640 1 -- 172.21.15.45:0/3399558199 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6918+0+0 (secure 0 0 0) 0x7f3d8c0890e0 con 0x7f3d9006d6b0 2023-11-28T22:10:13.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.077+0000 7f3d95cee640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3d7406aed0 0x7f3d7406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.078+0000 7f3d95cee640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3d7406aed0 0x7f3d7406d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d80036000 tx=0x7f3d8003b040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:13.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.081+0000 7f3d86ffd640 1 -- 172.21.15.45:0/3399558199 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d8c053c50 con 0x7f3d9006d6b0 2023-11-28T22:10:13.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.322+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d9009d0f0 con 0x7f3d9006d6b0 2023-11-28T22:10:13.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.323+0000 7f3d86ffd640 1 -- 172.21.15.45:0/3399558199 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4928 (secure 0 0 0) 0x7f3d8c017090 con 0x7f3d9006d6b0 2023-11-28T22:10:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:13.325 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8692,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3311227176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3311227176},{"type":"v1","addr":"172.21.15.190:6835","nonce":3311227176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8698,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3492680695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3492680695},{"type":"v1","addr":"172.21.15.45:6835","nonce":3492680695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":8701,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3937197634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3937197634},{"type":"v1","addr":"172.21.15.190:6837","nonce":3937197634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T22:08:43.777075+0000","modified":"2023-11-28T22:08:46.880928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10650":{"gid":10650,"name":"c","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6833/3094566588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3094566588},{"type":"v1","addr":"172.21.15.45:6833","nonce":3094566588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-28T22:10:13.325 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 101 2023-11-28T22:10:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.326+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3d7406aed0 msgr2=0x7f3d7406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.326+0000 7f3d974f1640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3d7406aed0 0x7f3d7406d390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d80036000 tx=0x7f3d8003b040 comp rx=0 tx=0).stop 2023-11-28T22:10:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.326+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9006d6b0 msgr2=0x7f3d90099dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.326+0000 7f3d974f1640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9006d6b0 0x7f3d90099dc0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00c400 tx=0x7f3d8c00c8c0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.327+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 shutdown_connections 2023-11-28T22:10:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.327+0000 7f3d974f1640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d900b3870 0x7f3d900cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.327+0000 7f3d974f1640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3d7406aed0 0x7f3d7406d390 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.327+0000 7f3d974f1640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d900721e0 0x7f3d901460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.327+0000 7f3d974f1640 1 --2- 172.21.15.45:0/3399558199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d9006d6b0 0x7f3d90099dc0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.327+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 >> 172.21.15.45:0/3399558199 conn(0x7f3d9006ab30 msgr2=0x7f3d901573a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.327+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 shutdown_connections 2023-11-28T22:10:13.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.327+0000 7f3d974f1640 1 -- 172.21.15.45:0/3399558199 wait complete. 2023-11-28T22:10:13.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:10:13.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.572+0000 7faf7f6cf640 1 Processor -- start 2023-11-28T22:10:13.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.573+0000 7faf7f6cf640 1 -- start start 2023-11-28T22:10:13.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.574+0000 7faf7f6cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7815bd10 0x7faf7815c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.574+0000 7faf7f6cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7faf78140f60 0x7faf7813f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.574+0000 7faf7f6cf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf78070660 con 0x7faf7806fd60 2023-11-28T22:10:13.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.574+0000 7faf7f6cf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7815c630 con 0x7faf78140f60 2023-11-28T22:10:13.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.574+0000 7faf7f6cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7813e540 con 0x7faf7815bd10 2023-11-28T22:10:13.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.575+0000 7faf7cc6a640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faf7806fd60 0x7faf78140850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57494/0 (socket says 172.21.15.45:57494) 2023-11-28T22:10:13.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.575+0000 7faf7cc6a640 1 -- 172.21.15.45:0/1185986683 learned_addr learned my addr 172.21.15.45:0/1185986683 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:13.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.575+0000 7faf7dc6c640 1 --2- 172.21.15.45:0/1185986683 >> v2:172.21.15.45:3300/0 conn(0x7faf78140f60 0x7faf7813f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.575+0000 7faf7d46b640 1 --2- 172.21.15.45:0/1185986683 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7815bd10 0x7faf7815c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.575+0000 7faf7dc6c640 1 -- 172.21.15.45:0/1185986683 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7815bd10 msgr2=0x7faf7815c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:13.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.575+0000 7faf7dc6c640 1 --2- 172.21.15.45:0/1185986683 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7815bd10 0x7faf7815c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.575+0000 7faf7dc6c640 1 -- 172.21.15.45:0/1185986683 >> v1:172.21.15.45:6789/0 conn(0x7faf7806fd60 legacy=0x7faf78140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.575+0000 7faf7dc6c640 1 -- 172.21.15.45:0/1185986683 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf78070940 con 0x7faf78140f60 2023-11-28T22:10:13.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.576+0000 7faf7dc6c640 1 --2- 172.21.15.45:0/1185986683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf78140f60 0x7faf7813f060 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7faf700097a0 tx=0x7faf7002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23fb8e58208ed5a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:13.578 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.577+0000 7faf6ffff640 1 -- 172.21.15.45:0/1185986683 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7002d040 con 0x7faf78140f60 2023-11-28T22:10:13.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.577+0000 7faf6ffff640 1 -- 172.21.15.45:0/1185986683 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf70031810 con 0x7faf78140f60 2023-11-28T22:10:13.579 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.577+0000 7faf6ffff640 1 -- 172.21.15.45:0/1185986683 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70031ba0 con 0x7faf78140f60 2023-11-28T22:10:13.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.577+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1185986683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf78140f60 msgr2=0x7faf7813f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.577+0000 7faf7f6cf640 1 --2- 172.21.15.45:0/1185986683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf78140f60 0x7faf7813f060 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7faf700097a0 tx=0x7faf7002b8b0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.578+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1185986683 shutdown_connections 2023-11-28T22:10:13.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.578+0000 7faf7f6cf640 1 --2- 172.21.15.45:0/1185986683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf78140f60 0x7faf7813f060 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.578+0000 7faf7f6cf640 1 --2- 172.21.15.45:0/1185986683 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7815bd10 0x7faf7815c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.578+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1185986683 >> 172.21.15.45:0/1185986683 conn(0x7faf7806e7c0 msgr2=0x7faf7806ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:13.580 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.578+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1185986683 shutdown_connections 2023-11-28T22:10:13.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.578+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1185986683 wait complete. 2023-11-28T22:10:13.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.579+0000 7faf7f6cf640 1 Processor -- start 2023-11-28T22:10:13.581 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.580+0000 7faf7f6cf640 1 -- start start 2023-11-28T22:10:13.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7f6cf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7806fd60 0x7faf781f2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7f6cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf78140f60 0x7faf7815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7f6cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf7815bd10 0x7faf7815a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.582 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7f6cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7806a910 con 0x7faf7815bd10 2023-11-28T22:10:13.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7f6cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7813a900 con 0x7faf78140f60 2023-11-28T22:10:13.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7f6cf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7813abe0 con 0x7faf7806fd60 2023-11-28T22:10:13.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7dc6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf7815bd10 0x7faf7815a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7dc6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf7815bd10 0x7faf7815a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49814/0 (socket says 172.21.15.45:49814) 2023-11-28T22:10:13.583 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7dc6c640 1 -- 172.21.15.45:0/1962423974 learned_addr learned my addr 172.21.15.45:0/1962423974 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:13.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7cc6a640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf78140f60 0x7faf7815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.584 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7d46b640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7806fd60 0x7faf781f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7dc6c640 1 -- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7806fd60 msgr2=0x7faf781f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.581+0000 7faf7dc6c640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7806fd60 0x7faf781f2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.582+0000 7faf7dc6c640 1 -- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf78140f60 msgr2=0x7faf7815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.582+0000 7faf7dc6c640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf78140f60 0x7faf7815a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.582+0000 7faf7dc6c640 1 -- 172.21.15.45:0/1962423974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf70009400 con 0x7faf7815bd10 2023-11-28T22:10:13.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.582+0000 7faf7dc6c640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf7815bd10 0x7faf7815a680 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7faf7002b8f0 tx=0x7faf700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:13.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.582+0000 7faf6dffb640 1 -- 172.21.15.45:0/1962423974 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7003ace0 con 0x7faf7815bd10 2023-11-28T22:10:13.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.582+0000 7faf6dffb640 1 -- 172.21.15.45:0/1962423974 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7003ae60 con 0x7faf7815bd10 2023-11-28T22:10:13.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.582+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7806a5a0 con 0x7faf7815bd10 2023-11-28T22:10:13.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.582+0000 7faf6dffb640 1 -- 172.21.15.45:0/1962423974 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70044630 con 0x7faf7815bd10 2023-11-28T22:10:13.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.582+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7813aec0 con 0x7faf7815bd10 2023-11-28T22:10:13.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.584+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7806aa80 con 0x7faf7815bd10 2023-11-28T22:10:13.586 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.585+0000 7faf6dffb640 1 -- 172.21.15.45:0/1962423974 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 127) v1 ==== 89277+0+0 (secure 0 0 0) 0x7faf70044790 con 0x7faf7815bd10 2023-11-28T22:10:13.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.586+0000 7faf6dffb640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7faf5c06aed0 0x7faf5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:13.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.586+0000 7faf6dffb640 1 -- 172.21.15.45:0/1962423974 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6918+0+0 (secure 0 0 0) 0x7faf700ac060 con 0x7faf7815bd10 2023-11-28T22:10:13.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.587+0000 7faf7d46b640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7faf5c06aed0 0x7faf5c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:13.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.588+0000 7faf7d46b640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7faf5c06aed0 0x7faf5c06d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf74009e90 tx=0x7faf7400b040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:13.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.590+0000 7faf6dffb640 1 -- 172.21.15.45:0/1962423974 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf700dee90 con 0x7faf7815bd10 2023-11-28T22:10:13.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.826+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf7806a730 con 0x7faf7815bd10 2023-11-28T22:10:13.828 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.827+0000 7faf6dffb640 1 -- 172.21.15.45:0/1962423974 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4928 (secure 0 0 0) 0x7faf70035020 con 0x7faf7815bd10 2023-11-28T22:10:13.829 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:13.829 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8692,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3311227176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3311227176},{"type":"v1","addr":"172.21.15.190:6835","nonce":3311227176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8698,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3492680695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3492680695},{"type":"v1","addr":"172.21.15.45:6835","nonce":3492680695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":8701,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3937197634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3937197634},{"type":"v1","addr":"172.21.15.190:6837","nonce":3937197634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T22:08:43.777075+0000","modified":"2023-11-28T22:08:46.880928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10650":{"gid":10650,"name":"c","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6833/3094566588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3094566588},{"type":"v1","addr":"172.21.15.45:6833","nonce":3094566588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-11-28T22:10:13.829 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 101 2023-11-28T22:10:13.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7faf5c06aed0 msgr2=0x7faf5c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7faf5c06aed0 0x7faf5c06d390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf74009e90 tx=0x7faf7400b040 comp rx=0 tx=0).stop 2023-11-28T22:10:13.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf7815bd10 msgr2=0x7faf7815a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:13.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf7815bd10 0x7faf7815a680 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7faf7002b8f0 tx=0x7faf700046c0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 shutdown_connections 2023-11-28T22:10:13.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf7815bd10 0x7faf7815a680 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7faf5c06aed0 0x7faf5c06d390 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf78140f60 0x7faf7815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 --2- 172.21.15.45:0/1962423974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf7806fd60 0x7faf781f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:13.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 >> 172.21.15.45:0/1962423974 conn(0x7faf7806e7c0 msgr2=0x7faf780b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:13.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 shutdown_connections 2023-11-28T22:10:13.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:13.830+0000 7faf7f6cf640 1 -- 172.21.15.45:0/1962423974 wait complete. 2023-11-28T22:10:13.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:10:14.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.069+0000 7feeed48c640 1 Processor -- start 2023-11-28T22:10:14.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.069+0000 7feeed48c640 1 -- start start 2023-11-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.070+0000 7feeed48c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 0x7feee8099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.070+0000 7feeed48c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feee80cf010 0x7feee80cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.071+0000 7feeed48c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee80c4ac0 con 0x7feee80b3610 2023-11-28T22:10:14.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.071+0000 7feeed48c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee80c5080 con 0x7feee80cf010 2023-11-28T22:10:14.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.071+0000 7feeed48c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee809a360 con 0x7feee8099a40 2023-11-28T22:10:14.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.071+0000 7feee77fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feee80cf010 0x7feee80cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.071+0000 7feee77fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feee80cf010 0x7feee80cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34520/0 (socket says 172.21.15.45:34520) 2023-11-28T22:10:14.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.071+0000 7feee77fe640 1 -- 172.21.15.45:0/2950523459 learned_addr learned my addr 172.21.15.45:0/2950523459 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:14.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.071+0000 7feee7fff640 1 --2- 172.21.15.45:0/2950523459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 0x7feee8099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.072+0000 7feee77fe640 1 -- 172.21.15.45:0/2950523459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 msgr2=0x7feee8099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:14.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.072+0000 7feee77fe640 1 --2- 172.21.15.45:0/2950523459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 0x7feee8099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.072+0000 7feee77fe640 1 -- 172.21.15.45:0/2950523459 >> v1:172.21.15.190:6789/0 conn(0x7feee80b3610 legacy=0x7feee80cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:14.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.072+0000 7feee77fe640 1 -- 172.21.15.45:0/2950523459 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee809a610 con 0x7feee80cf010 2023-11-28T22:10:14.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.073+0000 7feee77fe640 1 --2- 172.21.15.45:0/2950523459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 0x7feee80cba10 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7feed0009870 tx=0x7feed002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbf3f1f0a95bfd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:14.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feee6ffd640 1 -- 172.21.15.45:0/2950523459 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0031640 con 0x7feee80cf010 2023-11-28T22:10:14.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feee6ffd640 1 -- 172.21.15.45:0/2950523459 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed00317a0 con 0x7feee80cf010 2023-11-28T22:10:14.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feee6ffd640 1 -- 172.21.15.45:0/2950523459 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0031b30 con 0x7feee80cf010 2023-11-28T22:10:14.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feeed48c640 1 -- 172.21.15.45:0/2950523459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 msgr2=0x7feee80cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feeed48c640 1 --2- 172.21.15.45:0/2950523459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 0x7feee80cba10 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7feed0009870 tx=0x7feed002b8f0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feeed48c640 1 -- 172.21.15.45:0/2950523459 shutdown_connections 2023-11-28T22:10:14.076 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feeed48c640 1 --2- 172.21.15.45:0/2950523459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 0x7feee80cba10 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feeed48c640 1 --2- 172.21.15.45:0/2950523459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 0x7feee8099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feeed48c640 1 -- 172.21.15.45:0/2950523459 >> 172.21.15.45:0/2950523459 conn(0x7feee80b2b20 msgr2=0x7feee80b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:14.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feeed48c640 1 -- 172.21.15.45:0/2950523459 shutdown_connections 2023-11-28T22:10:14.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.074+0000 7feeed48c640 1 -- 172.21.15.45:0/2950523459 wait complete. 2023-11-28T22:10:14.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.076+0000 7feeed48c640 1 Processor -- start 2023-11-28T22:10:14.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.076+0000 7feeed48c640 1 -- start start 2023-11-28T22:10:14.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.077+0000 7feeed48c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 0x7feee80b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.077+0000 7feeed48c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee80b3610 0x7feee81494b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.077+0000 7feeed48c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 0x7feee806b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeed48c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee80b5320 con 0x7feee80cf010 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeed48c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee80b4260 con 0x7feee80b3610 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeed48c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee80b4540 con 0x7feee8099a40 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeecc8b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 0x7feee806b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeecc8b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 0x7feee806b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34528/0 (socket says 172.21.15.45:34528) 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeecc8b640 1 -- 172.21.15.45:0/651169453 learned_addr learned my addr 172.21.15.45:0/651169453 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feee7fff640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 0x7feee80b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feee77fe640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee80b3610 0x7feee81494b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeecc8b640 1 -- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 msgr2=0x7feee80b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeecc8b640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 0x7feee80b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeecc8b640 1 -- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee80b3610 msgr2=0x7feee81494b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeecc8b640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee80b3610 0x7feee81494b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feeecc8b640 1 -- 172.21.15.45:0/651169453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed00094d0 con 0x7feee80cf010 2023-11-28T22:10:14.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.078+0000 7feee77fe640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee80b3610 0x7feee81494b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:14.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.079+0000 7feeecc8b640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 0x7feee806b5a0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7feee000c410 tx=0x7feee000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:14.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.079+0000 7feee4ff9640 1 -- 172.21.15.45:0/651169453 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0011790 con 0x7feee80cf010 2023-11-28T22:10:14.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.079+0000 7feee4ff9640 1 -- 172.21.15.45:0/651169453 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee00118f0 con 0x7feee80cf010 2023-11-28T22:10:14.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.079+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee80b4820 con 0x7feee80cf010 2023-11-28T22:10:14.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.079+0000 7feee4ff9640 1 -- 172.21.15.45:0/651169453 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0011ce0 con 0x7feee80cf010 2023-11-28T22:10:14.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.079+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee80cb280 con 0x7feee80cf010 2023-11-28T22:10:14.082 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.080+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee80b5490 con 0x7feee80cf010 2023-11-28T22:10:14.083 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.081+0000 7feee4ff9640 1 -- 172.21.15.45:0/651169453 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 127) v1 ==== 89277+0+0 (secure 0 0 0) 0x7feee001c460 con 0x7feee80cf010 2023-11-28T22:10:14.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.083+0000 7feee4ff9640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feec806aed0 0x7feec806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.083+0000 7feee4ff9640 1 -- 172.21.15.45:0/651169453 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6918+0+0 (secure 0 0 0) 0x7feee0089790 con 0x7feee80cf010 2023-11-28T22:10:14.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.084+0000 7feee7fff640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feec806aed0 0x7feec806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.086 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.085+0000 7feee7fff640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feec806aed0 0x7feec806d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feed80070b0 tx=0x7feed8007040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:14.088 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.087+0000 7feee4ff9640 1 -- 172.21.15.45:0/651169453 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feee0054390 con 0x7feee80cf010 2023-11-28T22:10:14.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.278+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feee809f110 con 0x7feee80cf010 2023-11-28T22:10:14.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.281+0000 7feee4ff9640 1 -- 172.21.15.45:0/651169453 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+18034 (secure 0 0 0) 0x7feee00579b0 con 0x7feee80cf010 2023-11-28T22:10:14.283 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:14.283 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":219,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:10:10.669328+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-11-28T22:08:41.574328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-11-28T22:08:42.812303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"215","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/3465043952":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/1255156042":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/964161599":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/1168814133":"2023-11-29T22:10:10.669273+0000","172.21.15.190:6832/3154132268":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/3277311498":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/3310710469":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1697878697":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/2564313259":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/1107658380":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/4107859064":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/3311741793":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1352690774":"2023-11-29T22:08:58.564601+0000","172.21.15.190:6833/3154132268":"2023-11-29T22:10:10.669273+0000","172.21.15.190:6833/397715021":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1385562856":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/3670565434":"2023-11-29T22:10:10.669273+0000","172.21.15.190:6832/397715021":"2023-11-29T22:08:58.564601+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:10:14.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feec806aed0 msgr2=0x7feec806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feec806aed0 0x7feec806d390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feed80070b0 tx=0x7feed8007040 comp rx=0 tx=0).stop 2023-11-28T22:10:14.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 msgr2=0x7feee806b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 0x7feee806b5a0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7feee000c410 tx=0x7feee000c8d0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 shutdown_connections 2023-11-28T22:10:14.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feee80cf010 0x7feee806b5a0 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feec806aed0 0x7feec806d390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feee80b3610 0x7feee81494b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 --2- 172.21.15.45:0/651169453 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feee8099a40 0x7feee80b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.283+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 >> 172.21.15.45:0/651169453 conn(0x7feee80b2b20 msgr2=0x7feee806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:14.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.284+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 shutdown_connections 2023-11-28T22:10:14.287 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.284+0000 7feeed48c640 1 -- 172.21.15.45:0/651169453 wait complete. 2023-11-28T22:10:14.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T22:10:14.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.503+0000 7f14faf16640 1 Processor -- start 2023-11-28T22:10:14.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.503+0000 7f14faf16640 1 -- start start 2023-11-28T22:10:14.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.505+0000 7f14faf16640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f14f40cf610 0x7f14f40ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.505+0000 7f14faf16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40b2fb0 0x7f14f415eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.505+0000 7f14faf16640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f40abd70 con 0x7f14f415b070 2023-11-28T22:10:14.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.505+0000 7f14faf16640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f415b450 con 0x7f14f40cf610 2023-11-28T22:10:14.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.505+0000 7f14faf16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40b3ed0 con 0x7f14f40b2fb0 2023-11-28T22:10:14.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.506+0000 7f14f9713640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f14f40cf610 0x7f14f40ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.506+0000 7f14f9713640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f14f40cf610 0x7f14f40ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:39002/0 (socket says 172.21.15.45:39002) 2023-11-28T22:10:14.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.506+0000 7f14f9713640 1 -- 172.21.15.45:0/2525008707 learned_addr learned my addr 172.21.15.45:0/2525008707 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:14.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.506+0000 7f14fa715640 1 --2- 172.21.15.45:0/2525008707 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40b2fb0 0x7f14f415eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.506+0000 7f14f9713640 1 -- 172.21.15.45:0/2525008707 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40b2fb0 msgr2=0x7f14f415eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:14.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.506+0000 7f14f9713640 1 --2- 172.21.15.45:0/2525008707 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40b2fb0 0x7f14f415eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.506+0000 7f14f9713640 1 -- 172.21.15.45:0/2525008707 >> v1:172.21.15.190:6789/0 conn(0x7f14f415b070 legacy=0x7f14f40cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.506+0000 7f14f9713640 1 -- 172.21.15.45:0/2525008707 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40abfc0 con 0x7f14f40cf610 2023-11-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.508+0000 7f14f9713640 1 --2- 172.21.15.45:0/2525008707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14f40cf610 0x7f14f40ae370 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009870 tx=0x7f14e402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e5b36a33371f84 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.508+0000 7f14f8f12640 1 -- 172.21.15.45:0/2525008707 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4031640 con 0x7f14f40cf610 2023-11-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.508+0000 7f14f8f12640 1 -- 172.21.15.45:0/2525008707 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e40317a0 con 0x7f14f40cf610 2023-11-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.508+0000 7f14f8f12640 1 -- 172.21.15.45:0/2525008707 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4031b30 con 0x7f14f40cf610 2023-11-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.509+0000 7f14faf16640 1 -- 172.21.15.45:0/2525008707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14f40cf610 msgr2=0x7f14f40ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.509+0000 7f14faf16640 1 --2- 172.21.15.45:0/2525008707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14f40cf610 0x7f14f40ae370 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009870 tx=0x7f14e402b8f0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.509+0000 7f14faf16640 1 -- 172.21.15.45:0/2525008707 shutdown_connections 2023-11-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.509+0000 7f14faf16640 1 --2- 172.21.15.45:0/2525008707 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40b2fb0 0x7f14f415eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.509+0000 7f14faf16640 1 --2- 172.21.15.45:0/2525008707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14f40cf610 0x7f14f40ae370 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.509+0000 7f14faf16640 1 -- 172.21.15.45:0/2525008707 >> 172.21.15.45:0/2525008707 conn(0x7f14f406a710 msgr2=0x7f14f406ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.509+0000 7f14faf16640 1 -- 172.21.15.45:0/2525008707 shutdown_connections 2023-11-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.509+0000 7f14faf16640 1 -- 172.21.15.45:0/2525008707 wait complete. 2023-11-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.511+0000 7f14faf16640 1 Processor -- start 2023-11-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.511+0000 7f14faf16640 1 -- start start 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.512+0000 7f14faf16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14f40b2fb0 0x7f14f41436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.512+0000 7f14faf16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40cf610 0x7f14f4158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.512+0000 7f14faf16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14f415b070 0x7f14f4099f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.512+0000 7f14faf16640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4159770 con 0x7f14f415b070 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.512+0000 7f14faf16640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4159a50 con 0x7f14f40b2fb0 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.512+0000 7f14faf16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4159d30 con 0x7f14f40cf610 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.512+0000 7f14fa715640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14f415b070 0x7f14f4099f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.512+0000 7f14fa715640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14f415b070 0x7f14f4099f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34530/0 (socket says 172.21.15.45:34530) 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.512+0000 7f14fa715640 1 -- 172.21.15.45:0/186377127 learned_addr learned my addr 172.21.15.45:0/186377127 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.513+0000 7f14f9f14640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14f40b2fb0 0x7f14f41436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.513+0000 7f14f9713640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40cf610 0x7f14f4158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.513+0000 7f14fa715640 1 -- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40cf610 msgr2=0x7f14f4158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.513+0000 7f14fa715640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40cf610 0x7f14f4158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.513+0000 7f14fa715640 1 -- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14f40b2fb0 msgr2=0x7f14f41436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.513+0000 7f14fa715640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14f40b2fb0 0x7f14f41436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.513+0000 7f14fa715640 1 -- 172.21.15.45:0/186377127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e40094d0 con 0x7f14f415b070 2023-11-28T22:10:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.513+0000 7f14fa715640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14f415b070 0x7f14f4099f70 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00efc0 tx=0x7f14ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.513+0000 7f14ea7fc640 1 -- 172.21.15.45:0/186377127 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec009ba0 con 0x7f14f415b070 2023-11-28T22:10:14.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.514+0000 7f14ea7fc640 1 -- 172.21.15.45:0/186377127 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec00f040 con 0x7f14f415b070 2023-11-28T22:10:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.514+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f415a010 con 0x7f14f415b070 2023-11-28T22:10:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.514+0000 7f14ea7fc640 1 -- 172.21.15.45:0/186377127 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0146c0 con 0x7f14f415b070 2023-11-28T22:10:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.514+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f415a810 con 0x7f14f415b070 2023-11-28T22:10:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.515+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f415a530 con 0x7f14f415b070 2023-11-28T22:10:14.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.516+0000 7f14ea7fc640 1 -- 172.21.15.45:0/186377127 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 127) v1 ==== 89277+0+0 (secure 0 0 0) 0x7f14ec014820 con 0x7f14f415b070 2023-11-28T22:10:14.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.517+0000 7f14ea7fc640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14d806aed0 0x7f14d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.517+0000 7f14ea7fc640 1 -- 172.21.15.45:0/186377127 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6918+0+0 (secure 0 0 0) 0x7f14ec08a7e0 con 0x7f14f415b070 2023-11-28T22:10:14.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.517+0000 7f14f9f14640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14d806aed0 0x7f14d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.519+0000 7f14f9f14640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14d806aed0 0x7f14d806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f14f0000f60 tx=0x7f14f0009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:14.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.521+0000 7f14ea7fc640 1 -- 172.21.15.45:0/186377127 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14f415a530 con 0x7f14f415b070 2023-11-28T22:10:14.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.727+0000 7f14ea7fc640 1 -- 172.21.15.45:0/186377127 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f14ec0555a0 con 0x7f14f415b070 2023-11-28T22:10:14.761 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.760+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f14f415a530 con 0x7f14f415b070 2023-11-28T22:10:14.762 INFO:tasks.ceph.mon.a.smithi045.stderr:2023-11-28T22:10:14.761+0000 7fdd924d5640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T22:10:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.765+0000 7f14ea7fc640 1 -- 172.21.15.45:0/186377127 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v102) v1 ==== 161+0+0 (secure 0 0 0) 0x7f14f415a530 con 0x7f14f415b070 2023-11-28T22:10:14.767 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T22:10:14.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.768+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14d806aed0 msgr2=0x7f14d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.768+0000 7f14faf16640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14d806aed0 0x7f14d806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f14f0000f60 tx=0x7f14f0009040 comp rx=0 tx=0).stop 2023-11-28T22:10:14.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.768+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14f415b070 msgr2=0x7f14f4099f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.768+0000 7f14faf16640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14f415b070 0x7f14f4099f70 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00efc0 tx=0x7f14ec00c490 comp rx=0 tx=0).stop 2023-11-28T22:10:14.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.769+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 shutdown_connections 2023-11-28T22:10:14.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.769+0000 7f14faf16640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14f415b070 0x7f14f4099f70 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.769+0000 7f14faf16640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f14f40cf610 0x7f14f4158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.769+0000 7f14faf16640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14d806aed0 0x7f14d806d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.769+0000 7f14faf16640 1 --2- 172.21.15.45:0/186377127 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14f40b2fb0 0x7f14f41436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.769+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 >> 172.21.15.45:0/186377127 conn(0x7f14f406a710 msgr2=0x7f14f40ae900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:14.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.769+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 shutdown_connections 2023-11-28T22:10:14.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.770+0000 7f14faf16640 1 -- 172.21.15.45:0/186377127 wait complete. 2023-11-28T22:10:14.790 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T22:10:14.876 INFO:tasks.ceph.mds.c.smithi045.stdout:starting mds.c at 2023-11-28T22:10:14.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.985+0000 7f3c70b16640 1 Processor -- start 2023-11-28T22:10:14.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.985+0000 7f3c70b16640 1 -- start start 2023-11-28T22:10:14.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.987+0000 7f3c70b16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c6c15b080 0x7f3c6c0ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.987+0000 7f3c70b16640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3c6c071fd0 0x7f3c6c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.987+0000 7f3c70b16640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6c0ac270 con 0x7f3c6c0abd70 2023-11-28T22:10:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.987+0000 7f3c70b16640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6c15b830 con 0x7f3c6c071fd0 2023-11-28T22:10:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.987+0000 7f3c70b16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c15bb10 con 0x7f3c6c15b080 2023-11-28T22:10:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.987+0000 7f3c6affd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3c6c0abd70 0x7f3c6c0718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36500/0 (socket says 172.21.15.45:36500) 2023-11-28T22:10:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.987+0000 7f3c6affd640 1 -- 172.21.15.45:0/3818908084 learned_addr learned my addr 172.21.15.45:0/3818908084 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.987+0000 7f3c6bfff640 1 --2- 172.21.15.45:0/3818908084 >> v2:172.21.15.45:3300/0 conn(0x7f3c6c071fd0 0x7f3c6c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.987+0000 7f3c6b7fe640 1 --2- 172.21.15.45:0/3818908084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c6c15b080 0x7f3c6c0ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.988+0000 7f3c6a7fc640 1 -- 172.21.15.45:0/3818908084 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195409840 0 0) 0x7f3c6c0ac270 con 0x7f3c6c0abd70 2023-11-28T22:10:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.988+0000 7f3c6a7fc640 1 -- 172.21.15.45:0/3818908084 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c54003520 con 0x7f3c6c0abd70 2023-11-28T22:10:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.988+0000 7f3c6bfff640 1 -- 172.21.15.45:0/3818908084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c6c15b080 msgr2=0x7f3c6c0ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.988+0000 7f3c6bfff640 1 --2- 172.21.15.45:0/3818908084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c6c15b080 0x7f3c6c0ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.988+0000 7f3c6bfff640 1 -- 172.21.15.45:0/3818908084 >> v1:172.21.15.45:6789/0 conn(0x7f3c6c0abd70 legacy=0x7f3c6c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.988+0000 7f3c6bfff640 1 -- 172.21.15.45:0/3818908084 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c15bdf0 con 0x7f3c6c071fd0 2023-11-28T22:10:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.989+0000 7f3c6bfff640 1 --2- 172.21.15.45:0/3818908084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 0x7f3c6c0702c0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009d90 tx=0x7f3c6402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e8c28e52d8e6ce server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.989+0000 7f3c6a7fc640 1 -- 172.21.15.45:0/3818908084 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6402cc60 con 0x7f3c6c071fd0 2023-11-28T22:10:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.989+0000 7f3c6a7fc640 1 -- 172.21.15.45:0/3818908084 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6402cdc0 con 0x7f3c6c071fd0 2023-11-28T22:10:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.989+0000 7f3c6a7fc640 1 -- 172.21.15.45:0/3818908084 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c64034660 con 0x7f3c6c071fd0 2023-11-28T22:10:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.990+0000 7f3c70b16640 1 -- 172.21.15.45:0/3818908084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 msgr2=0x7f3c6c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.990+0000 7f3c70b16640 1 --2- 172.21.15.45:0/3818908084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 0x7f3c6c0702c0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009d90 tx=0x7f3c6402beb0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.990+0000 7f3c70b16640 1 -- 172.21.15.45:0/3818908084 shutdown_connections 2023-11-28T22:10:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.990+0000 7f3c70b16640 1 --2- 172.21.15.45:0/3818908084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 0x7f3c6c0702c0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.990+0000 7f3c70b16640 1 --2- 172.21.15.45:0/3818908084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c6c15b080 0x7f3c6c0ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.990+0000 7f3c70b16640 1 -- 172.21.15.45:0/3818908084 >> 172.21.15.45:0/3818908084 conn(0x7f3c6c0cecd0 msgr2=0x7f3c6c0cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.990+0000 7f3c70b16640 1 -- 172.21.15.45:0/3818908084 shutdown_connections 2023-11-28T22:10:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.990+0000 7f3c70b16640 1 -- 172.21.15.45:0/3818908084 wait complete. 2023-11-28T22:10:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.991+0000 7f3c70b16640 1 Processor -- start 2023-11-28T22:10:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.992+0000 7f3c70b16640 1 -- start start 2023-11-28T22:10:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.992+0000 7f3c70b16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 0x7f3c6c15c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.992+0000 7f3c70b16640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c6c0abd70 0x7f3c6c1446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.992+0000 7f3c70b16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c15b080 0x7f3c6c09c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.992+0000 7f3c70b16640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c144bf0 con 0x7f3c6c071fd0 2023-11-28T22:10:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.992+0000 7f3c70b16640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c15cc70 con 0x7f3c6c15b080 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.992+0000 7f3c70b16640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c15cf50 con 0x7f3c6c0abd70 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.993+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 0x7f3c6c15c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.993+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 0x7f3c6c15c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34576/0 (socket says 172.21.15.45:34576) 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.993+0000 7f3c6b7fe640 1 -- 172.21.15.45:0/331468102 learned_addr learned my addr 172.21.15.45:0/331468102 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.993+0000 7f3c6b7fe640 1 -- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c6c0abd70 msgr2=0x7f3c6c1446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.993+0000 7f3c6b7fe640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c6c0abd70 0x7f3c6c1446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.993+0000 7f3c6b7fe640 1 -- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c15b080 msgr2=0x7f3c6c09c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.993+0000 7f3c6b7fe640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c15b080 0x7f3c6c09c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.993+0000 7f3c6b7fe640 1 -- 172.21.15.45:0/331468102 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c64009a70 con 0x7f3c6c071fd0 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.993+0000 7f3c6b7fe640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 0x7f3c6c15c690 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00d5f0 tx=0x7f3c5c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.994+0000 7f3c53fff640 1 -- 172.21.15.45:0/331468102 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c0125f0 con 0x7f3c6c071fd0 2023-11-28T22:10:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.994+0000 7f3c53fff640 1 -- 172.21.15.45:0/331468102 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c012750 con 0x7f3c6c071fd0 2023-11-28T22:10:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.994+0000 7f3c53fff640 1 -- 172.21.15.45:0/331468102 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c012b00 con 0x7f3c6c071fd0 2023-11-28T22:10:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.994+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c15d230 con 0x7f3c6c071fd0 2023-11-28T22:10:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.994+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c15d7f0 con 0x7f3c6c071fd0 2023-11-28T22:10:14.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.995+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6c15d9b0 con 0x7f3c6c071fd0 2023-11-28T22:10:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.996+0000 7f3c53fff640 1 -- 172.21.15.45:0/331468102 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3c5c012c60 con 0x7f3c6c071fd0 2023-11-28T22:10:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.997+0000 7f3c53fff640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3c5806af60 0x7f3c5806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.997+0000 7f3c53fff640 1 -- 172.21.15.45:0/331468102 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 7004+0+0 (secure 0 0 0) 0x7f3c5c08a1a0 con 0x7f3c6c071fd0 2023-11-28T22:10:14.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.998+0000 7f3c6affd640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3c5806af60 0x7f3c5806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:14.999+0000 7f3c6affd640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3c5806af60 0x7f3c5806d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c144ec0 tx=0x7f3c60007930 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:15.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.001+0000 7f3c53fff640 1 -- 172.21.15.45:0/331468102 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c5c054c20 con 0x7f3c6c071fd0 2023-11-28T22:10:15.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.222+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3c6c09d0f0 con 0x7f3c6c071fd0 2023-11-28T22:10:15.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.768+0000 7f3c53fff640 1 -- 172.21.15.45:0/331468102 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v103) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3c5c058240 con 0x7f3c6c071fd0 2023-11-28T22:10:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.771+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3c5806af60 msgr2=0x7f3c5806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3c5806af60 0x7f3c5806d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c144ec0 tx=0x7f3c60007930 comp rx=0 tx=0).stop 2023-11-28T22:10:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 msgr2=0x7f3c6c15c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 0x7f3c6c15c690 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00d5f0 tx=0x7f3c5c00dab0 comp rx=0 tx=0).stop 2023-11-28T22:10:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 shutdown_connections 2023-11-28T22:10:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3c5806af60 0x7f3c5806d420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c15b080 0x7f3c6c09c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3c6c0abd70 0x7f3c6c1446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 --2- 172.21.15.45:0/331468102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c071fd0 0x7f3c6c15c690 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 >> 172.21.15.45:0/331468102 conn(0x7f3c6c0cecd0 msgr2=0x7f3c6c0ae940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 shutdown_connections 2023-11-28T22:10:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:15.772+0000 7f3c70b16640 1 -- 172.21.15.45:0/331468102 wait complete. 2023-11-28T22:10:15.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T22:10:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.013+0000 7f86d3267640 1 Processor -- start 2023-11-28T22:10:16.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.013+0000 7f86d3267640 1 -- start start 2023-11-28T22:10:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.015+0000 7f86d3267640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f86cc0cd580 0x7f86cc0b3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.015+0000 7f86d3267640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f86cc0b3a80 0x7f86cc0b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.015+0000 7f86d3267640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86cc0b3ec0 con 0x7f86cc160ff0 2023-11-28T22:10:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.015+0000 7f86d3267640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86cc0cd9e0 con 0x7f86cc0b3a80 2023-11-28T22:10:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.015+0000 7f86d3267640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86cc146d90 con 0x7f86cc0cd580 2023-11-28T22:10:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.015+0000 7f86d2a66640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f86cc0b3a80 0x7f86cc0b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:16.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.015+0000 7f86d2a66640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f86cc0b3a80 0x7f86cc0b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34588/0 (socket says 172.21.15.45:34588) 2023-11-28T22:10:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.015+0000 7f86d2a66640 1 -- 172.21.15.45:0/4166180886 learned_addr learned my addr 172.21.15.45:0/4166180886 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.016+0000 7f86d1a64640 1 --2- 172.21.15.45:0/4166180886 >> v2:172.21.15.190:3300/0 conn(0x7f86cc0cd580 0x7f86cc0b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.016+0000 7f86d2a66640 1 -- 172.21.15.45:0/4166180886 >> v2:172.21.15.190:3300/0 conn(0x7f86cc0cd580 msgr2=0x7f86cc0b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.016+0000 7f86d2a66640 1 --2- 172.21.15.45:0/4166180886 >> v2:172.21.15.190:3300/0 conn(0x7f86cc0cd580 0x7f86cc0b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.016+0000 7f86d2a66640 1 -- 172.21.15.45:0/4166180886 >> v1:172.21.15.190:6789/0 conn(0x7f86cc160ff0 legacy=0x7f86cc1613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.016+0000 7f86d2a66640 1 -- 172.21.15.45:0/4166180886 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc0cdbd0 con 0x7f86cc0b3a80 2023-11-28T22:10:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.017+0000 7f86d2a66640 1 --2- 172.21.15.45:0/4166180886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 0x7f86cc0b1d20 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0b4050 tx=0x7f86c402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5adc53368246550d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.017+0000 7f86d1263640 1 -- 172.21.15.45:0/4166180886 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c402c9e0 con 0x7f86cc0b3a80 2023-11-28T22:10:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.017+0000 7f86d1263640 1 -- 172.21.15.45:0/4166180886 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c402cb40 con 0x7f86cc0b3a80 2023-11-28T22:10:16.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.017+0000 7f86d1263640 1 -- 172.21.15.45:0/4166180886 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c40343f0 con 0x7f86cc0b3a80 2023-11-28T22:10:16.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.018+0000 7f86d3267640 1 -- 172.21.15.45:0/4166180886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 msgr2=0x7f86cc0b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:16.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.018+0000 7f86d3267640 1 --2- 172.21.15.45:0/4166180886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 0x7f86cc0b1d20 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0b4050 tx=0x7f86c402be40 comp rx=0 tx=0).stop 2023-11-28T22:10:16.020 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.018+0000 7f86d3267640 1 -- 172.21.15.45:0/4166180886 shutdown_connections 2023-11-28T22:10:16.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.018+0000 7f86d3267640 1 --2- 172.21.15.45:0/4166180886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 0x7f86cc0b1d20 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.018+0000 7f86d3267640 1 --2- 172.21.15.45:0/4166180886 >> v2:172.21.15.190:3300/0 conn(0x7f86cc0cd580 0x7f86cc0b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.018+0000 7f86d3267640 1 -- 172.21.15.45:0/4166180886 >> 172.21.15.45:0/4166180886 conn(0x7f86cc0cec70 msgr2=0x7f86cc0cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:16.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.018+0000 7f86d3267640 1 -- 172.21.15.45:0/4166180886 shutdown_connections 2023-11-28T22:10:16.021 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.019+0000 7f86d3267640 1 -- 172.21.15.45:0/4166180886 wait complete. 2023-11-28T22:10:16.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.021+0000 7f86d3267640 1 Processor -- start 2023-11-28T22:10:16.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.021+0000 7f86d3267640 1 -- start start 2023-11-28T22:10:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.022+0000 7f86d3267640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 0x7f86cc0ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.022+0000 7f86d3267640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86cc0cd580 0x7f86cc15bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:16.024 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.022+0000 7f86d3267640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86cc160ff0 0x7f86cc15f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:16.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.022+0000 7f86d3267640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0abea0 con 0x7f86cc0b3a80 2023-11-28T22:10:16.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.022+0000 7f86d3267640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc15fde0 con 0x7f86cc160ff0 2023-11-28T22:10:16.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.022+0000 7f86d3267640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0ac140 con 0x7f86cc0cd580 2023-11-28T22:10:16.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d2265640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 0x7f86cc0ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:16.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d2265640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 0x7f86cc0ac940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34596/0 (socket says 172.21.15.45:34596) 2023-11-28T22:10:16.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d2265640 1 -- 172.21.15.45:0/2957429559 learned_addr learned my addr 172.21.15.45:0/2957429559 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:16.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d1a64640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86cc0cd580 0x7f86cc15bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:16.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d2a66640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86cc160ff0 0x7f86cc15f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:16.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d2265640 1 -- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86cc0cd580 msgr2=0x7f86cc15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:16.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d2265640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86cc0cd580 0x7f86cc15bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d2265640 1 -- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86cc160ff0 msgr2=0x7f86cc15f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:16.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d2265640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86cc160ff0 0x7f86cc15f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.023+0000 7f86d2265640 1 -- 172.21.15.45:0/2957429559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4009b30 con 0x7f86cc0b3a80 2023-11-28T22:10:16.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.024+0000 7f86d2265640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 0x7f86cc0ac940 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b3e0 tx=0x7f86c800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:16.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.024+0000 7f86c2ffd640 1 -- 172.21.15.45:0/2957429559 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8011830 con 0x7f86cc0b3a80 2023-11-28T22:10:16.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.024+0000 7f86c2ffd640 1 -- 172.21.15.45:0/2957429559 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c80119b0 con 0x7f86cc0b3a80 2023-11-28T22:10:16.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.024+0000 7f86c2ffd640 1 -- 172.21.15.45:0/2957429559 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8011d60 con 0x7f86cc0b3a80 2023-11-28T22:10:16.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.024+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86cc0ac420 con 0x7f86cc0b3a80 2023-11-28T22:10:16.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.024+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86cc1600c0 con 0x7f86cc0b3a80 2023-11-28T22:10:16.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.025+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86cc09d0f0 con 0x7f86cc0b3a80 2023-11-28T22:10:16.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.026+0000 7f86c2ffd640 1 -- 172.21.15.45:0/2957429559 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f86c801a400 con 0x7f86cc0b3a80 2023-11-28T22:10:16.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.027+0000 7f86c2ffd640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f86b006af60 0x7f86b006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:16.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.027+0000 7f86c2ffd640 1 -- 172.21.15.45:0/2957429559 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 7004+0+0 (secure 0 0 0) 0x7f86c8089d80 con 0x7f86cc0b3a80 2023-11-28T22:10:16.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.028+0000 7f86d1a64640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f86b006af60 0x7f86b006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:16.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.029+0000 7f86d1a64640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f86b006af60 0x7f86b006d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f86cc072610 tx=0x7f86bc008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:16.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.033+0000 7f86c2ffd640 1 -- 172.21.15.45:0/2957429559 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86c80548a0 con 0x7f86cc0b3a80 2023-11-28T22:10:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.225+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f86cc04e920 con 0x7f86cc0b3a80 2023-11-28T22:10:16.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.772+0000 7f86c2ffd640 1 -- 172.21.15.45:0/2957429559 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v221) v1 ==== 181+0+0 (secure 0 0 0) 0x7f86c8015020 con 0x7f86cc0b3a80 2023-11-28T22:10:16.774 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2023-11-28T22:10:16.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.777+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f86b006af60 msgr2=0x7f86b006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:16.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.777+0000 7f86d3267640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f86b006af60 0x7f86b006d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f86cc072610 tx=0x7f86bc008040 comp rx=0 tx=0).stop 2023-11-28T22:10:16.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.777+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 msgr2=0x7f86cc0ac940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:16.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.777+0000 7f86d3267640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 0x7f86cc0ac940 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b3e0 tx=0x7f86c800b8a0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.778+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 shutdown_connections 2023-11-28T22:10:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.778+0000 7f86d3267640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f86b006af60 0x7f86b006d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.778+0000 7f86d3267640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86cc160ff0 0x7f86cc15f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.778+0000 7f86d3267640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86cc0cd580 0x7f86cc15bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.778+0000 7f86d3267640 1 --2- 172.21.15.45:0/2957429559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86cc0b3a80 0x7f86cc0ac940 secure :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b3e0 tx=0x7f86c800b8a0 comp rx=0 tx=0).stop 2023-11-28T22:10:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.778+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 >> 172.21.15.45:0/2957429559 conn(0x7f86cc0cec70 msgr2=0x7f86cc0b29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.778+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 shutdown_connections 2023-11-28T22:10:16.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:16.779+0000 7f86d3267640 1 -- 172.21.15.45:0/2957429559 wait complete. 2023-11-28T22:10:16.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T22:10:17.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.027+0000 7f8ce8a32640 1 Processor -- start 2023-11-28T22:10:17.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.028+0000 7f8ce8a32640 1 -- start start 2023-11-28T22:10:17.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.029+0000 7f8ce8a32640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ce4157c10 0x7f8ce406da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:17.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.029+0000 7f8ce8a32640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8ce406a500 0x7f8ce406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:17.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.029+0000 7f8ce8a32640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ce415ddd0 0x7f8ce415e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:17.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.029+0000 7f8ce8a32640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce406ca60 con 0x7f8ce4157c10 2023-11-28T22:10:17.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce8a32640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce406aea0 con 0x7f8ce406a500 2023-11-28T22:10:17.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce8a32640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce415eb40 con 0x7f8ce415ddd0 2023-11-28T22:10:17.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ce4157c10 0x7f8ce406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:17.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce259c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ce4157c10 0x7f8ce406da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34600/0 (socket says 172.21.15.45:34600) 2023-11-28T22:10:17.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce259c640 1 -- 172.21.15.45:0/3897160816 learned_addr learned my addr 172.21.15.45:0/3897160816 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:17.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce1d9b640 1 --2- 172.21.15.45:0/3897160816 >> v2:172.21.15.190:3300/0 conn(0x7f8ce406a500 0x7f8ce406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:17.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce259c640 1 -- 172.21.15.45:0/3897160816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ce415ddd0 msgr2=0x7f8ce415e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:17.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce259c640 1 --2- 172.21.15.45:0/3897160816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ce415ddd0 0x7f8ce415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce259c640 1 -- 172.21.15.45:0/3897160816 >> v2:172.21.15.190:3300/0 conn(0x7f8ce406a500 msgr2=0x7f8ce406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:17.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce259c640 1 --2- 172.21.15.45:0/3897160816 >> v2:172.21.15.190:3300/0 conn(0x7f8ce406a500 0x7f8ce406a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce259c640 1 -- 172.21.15.45:0/3897160816 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce4157ff0 con 0x7f8ce4157c10 2023-11-28T22:10:17.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.030+0000 7f8ce1d9b640 1 --2- 172.21.15.45:0/3897160816 >> v2:172.21.15.190:3300/0 conn(0x7f8ce406a500 0x7f8ce406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:17.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.031+0000 7f8ce259c640 1 --2- 172.21.15.45:0/3897160816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce4157c10 0x7f8ce406da50 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40cfe40 tx=0x7f8cd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f80e45f501fa68b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:17.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.032+0000 7f8ce159a640 1 -- 172.21.15.45:0/3897160816 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd402ca70 con 0x7f8ce4157c10 2023-11-28T22:10:17.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.032+0000 7f8ce159a640 1 -- 172.21.15.45:0/3897160816 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd402cbd0 con 0x7f8ce4157c10 2023-11-28T22:10:17.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.032+0000 7f8ce159a640 1 -- 172.21.15.45:0/3897160816 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd40343f0 con 0x7f8ce4157c10 2023-11-28T22:10:17.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.033+0000 7f8ce8a32640 1 -- 172.21.15.45:0/3897160816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce4157c10 msgr2=0x7f8ce406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:17.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.033+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/3897160816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce4157c10 0x7f8ce406da50 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40cfe40 tx=0x7f8cd402bed0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.033+0000 7f8ce8a32640 1 -- 172.21.15.45:0/3897160816 shutdown_connections 2023-11-28T22:10:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.033+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/3897160816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ce415ddd0 0x7f8ce415e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.033+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/3897160816 >> v2:172.21.15.190:3300/0 conn(0x7f8ce406a500 0x7f8ce406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.033+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/3897160816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce4157c10 0x7f8ce406da50 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.033+0000 7f8ce8a32640 1 -- 172.21.15.45:0/3897160816 >> 172.21.15.45:0/3897160816 conn(0x7f8ce40ced60 msgr2=0x7f8ce40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.033+0000 7f8ce8a32640 1 -- 172.21.15.45:0/3897160816 shutdown_connections 2023-11-28T22:10:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.033+0000 7f8ce8a32640 1 -- 172.21.15.45:0/3897160816 wait complete. 2023-11-28T22:10:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.035+0000 7f8ce8a32640 1 Processor -- start 2023-11-28T22:10:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.035+0000 7f8ce8a32640 1 -- start start 2023-11-28T22:10:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.036+0000 7f8ce8a32640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce406a500 0x7f8ce4152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.036+0000 7f8ce8a32640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce4157c10 0x7f8ce4159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:17.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.036+0000 7f8ce8a32640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ce415ddd0 0x7f8ce4153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:17.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.036+0000 7f8ce8a32640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce41530c0 con 0x7f8ce406a500 2023-11-28T22:10:17.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.036+0000 7f8ce8a32640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce40c3160 con 0x7f8ce4157c10 2023-11-28T22:10:17.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.036+0000 7f8ce8a32640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce40c2930 con 0x7f8ce415ddd0 2023-11-28T22:10:17.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.036+0000 7f8ce259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce406a500 0x7f8ce4152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:17.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.036+0000 7f8ce259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce406a500 0x7f8ce4152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34608/0 (socket says 172.21.15.45:34608) 2023-11-28T22:10:17.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.036+0000 7f8ce259c640 1 -- 172.21.15.45:0/2044994439 learned_addr learned my addr 172.21.15.45:0/2044994439 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce1d9b640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce4157c10 0x7f8ce4159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce259c640 1 -- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ce415ddd0 msgr2=0x7f8ce4153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce2d9d640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ce415ddd0 0x7f8ce4153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce259c640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ce415ddd0 0x7f8ce4153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce259c640 1 -- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce4157c10 msgr2=0x7f8ce4159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce259c640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce4157c10 0x7f8ce4159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce259c640 1 -- 172.21.15.45:0/2044994439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd4009b30 con 0x7f8ce406a500 2023-11-28T22:10:17.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce1d9b640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce4157c10 0x7f8ce4159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:17.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce259c640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce406a500 0x7f8ce4152b80 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4036040 tx=0x7f8cd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:17.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8cdaffd640 1 -- 172.21.15.45:0/2044994439 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd403e450 con 0x7f8ce406a500 2023-11-28T22:10:17.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8cdaffd640 1 -- 172.21.15.45:0/2044994439 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd403e5f0 con 0x7f8ce406a500 2023-11-28T22:10:17.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8cdaffd640 1 -- 172.21.15.45:0/2044994439 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd403e9a0 con 0x7f8ce406a500 2023-11-28T22:10:17.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.037+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce40c2ba0 con 0x7f8ce406a500 2023-11-28T22:10:17.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.038+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce40c48c0 con 0x7f8ce406a500 2023-11-28T22:10:17.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.039+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce40733d0 con 0x7f8ce406a500 2023-11-28T22:10:17.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.040+0000 7f8cdaffd640 1 -- 172.21.15.45:0/2044994439 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8cd403eb00 con 0x7f8ce406a500 2023-11-28T22:10:17.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.041+0000 7f8cdaffd640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f8cc406af60 0x7f8cc406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:17.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.041+0000 7f8cdaffd640 1 -- 172.21.15.45:0/2044994439 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f8cd40acaa0 con 0x7f8ce406a500 2023-11-28T22:10:17.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.042+0000 7f8ce1d9b640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f8cc406af60 0x7f8cc406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:17.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.043+0000 7f8ce1d9b640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f8cc406af60 0x7f8cc406d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc006fd0 tx=0x7f8ccc008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:17.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.046+0000 7f8cdaffd640 1 -- 172.21.15.45:0/2044994439 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8cd4077760 con 0x7f8ce406a500 2023-11-28T22:10:17.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.257+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8ce4070210 con 0x7f8ce406a500 2023-11-28T22:10:17.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.784+0000 7f8cdaffd640 1 -- 172.21.15.45:0/2044994439 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v222) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8cd403edb0 con 0x7f8ce406a500 2023-11-28T22:10:17.786 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2023-11-28T22:10:17.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.790+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f8cc406af60 msgr2=0x7f8cc406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:17.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.790+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f8cc406af60 0x7f8cc406d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc006fd0 tx=0x7f8ccc008040 comp rx=0 tx=0).stop 2023-11-28T22:10:17.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.790+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce406a500 msgr2=0x7f8ce4152b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:17.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.790+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce406a500 0x7f8ce4152b80 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4036040 tx=0x7f8cd4004a40 comp rx=0 tx=0).stop 2023-11-28T22:10:17.792 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.791+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 shutdown_connections 2023-11-28T22:10:17.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.791+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ce415ddd0 0x7f8ce4153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.791+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f8cc406af60 0x7f8cc406d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.791+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce4157c10 0x7f8ce4159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.791+0000 7f8ce8a32640 1 --2- 172.21.15.45:0/2044994439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ce406a500 0x7f8ce4152b80 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:17.793 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.791+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 >> 172.21.15.45:0/2044994439 conn(0x7f8ce40ced60 msgr2=0x7f8ce40709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:17.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.794+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 shutdown_connections 2023-11-28T22:10:17.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:17.794+0000 7f8ce8a32640 1 -- 172.21.15.45:0/2044994439 wait complete. 2023-11-28T22:10:17.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-11-28T22:10:18.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.027+0000 7f609dae1640 1 Processor -- start 2023-11-28T22:10:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.027+0000 7f609dae1640 1 -- start start 2023-11-28T22:10:18.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.029+0000 7f609dae1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f60980abd70 0x7f60980ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.029+0000 7f609dae1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60980ac270 con 0x7f609815b080 2023-11-28T22:10:18.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.029+0000 7f609dae1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609815b830 con 0x7f6098071fd0 2023-11-28T22:10:18.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.029+0000 7f609dae1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f609815bb10 con 0x7f60980abd70 2023-11-28T22:10:18.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.029+0000 7f609cadf640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f609815b080 0x7f60980718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36506/0 (socket says 172.21.15.45:36506) 2023-11-28T22:10:18.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.029+0000 7f609cadf640 1 -- 172.21.15.45:0/1500015905 learned_addr learned my addr 172.21.15.45:0/1500015905 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:18.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f609d2e0640 1 --2- 172.21.15.45:0/1500015905 >> v2:172.21.15.190:3300/0 conn(0x7f60980abd70 0x7f60980ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419471801 0 0) 0x7f60980ac270 con 0x7f609815b080 2023-11-28T22:10:18.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6088003520 con 0x7f609815b080 2023-11-28T22:10:18.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2745539145 0 0) 0x7f6088003520 con 0x7f609815b080 2023-11-28T22:10:18.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 >> v2:172.21.15.190:3300/0 conn(0x7f60980abd70 msgr2=0x7f60980ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:18.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 --2- 172.21.15.45:0/1500015905 >> v2:172.21.15.190:3300/0 conn(0x7f60980abd70 0x7f60980ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 >> v1:172.21.15.190:6789/0 conn(0x7f6098071fd0 legacy=0x7f60980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:18.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609815bdf0 con 0x7f609815b080 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f608c001670 con 0x7f609815b080 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 >> v1:172.21.15.45:6789/0 conn(0x7f609815b080 legacy=0x7f60980718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6088003970 0x7f6088003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.030+0000 7f60977fe640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60880047c0 0x7f6088004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f60977fe640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6088005180 0x7f608800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609815bb10 con 0x7f6088003970 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608800c020 con 0x7f60880047c0 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608800c1e0 con 0x7f6088005180 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f609cadf640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6088003970 0x7f6088003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f6097fff640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60880047c0 0x7f6088004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f609cadf640 1 -- 172.21.15.45:0/1500015905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6088005180 msgr2=0x7f608800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f609cadf640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6088005180 0x7f608800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f609cadf640 1 -- 172.21.15.45:0/1500015905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60880047c0 msgr2=0x7f6088004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:18.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f609cadf640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60880047c0 0x7f6088004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f609d2e0640 1 -- 172.21.15.45:0/1500015905 reap_dead start 2023-11-28T22:10:18.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f609cadf640 1 -- 172.21.15.45:0/1500015905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088003520 con 0x7f6088003970 2023-11-28T22:10:18.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.031+0000 7f6097fff640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60880047c0 0x7f6088004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.032+0000 7f609cadf640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6088003970 0x7f6088003d90 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f608c00c240 tx=0x7f60880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb2530fe00efcd4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.033+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c005ad0 con 0x7f6088003970 2023-11-28T22:10:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.033+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c005c30 con 0x7f6088003970 2023-11-28T22:10:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.033+0000 7f60977fe640 1 -- 172.21.15.45:0/1500015905 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c02cab0 con 0x7f6088003970 2023-11-28T22:10:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.033+0000 7f609dae1640 1 -- 172.21.15.45:0/1500015905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6088003970 msgr2=0x7f6088003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.033+0000 7f609dae1640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6088003970 0x7f6088003d90 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f608c00c240 tx=0x7f60880057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.033+0000 7f609dae1640 1 -- 172.21.15.45:0/1500015905 shutdown_connections 2023-11-28T22:10:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.033+0000 7f609dae1640 1 --2- 172.21.15.45:0/1500015905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6088003970 0x7f6088003d90 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.033+0000 7f609dae1640 1 -- 172.21.15.45:0/1500015905 >> 172.21.15.45:0/1500015905 conn(0x7f60980cecd0 msgr2=0x7f60980cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:18.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.034+0000 7f609dae1640 1 -- 172.21.15.45:0/1500015905 shutdown_connections 2023-11-28T22:10:18.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.034+0000 7f609dae1640 1 -- 172.21.15.45:0/1500015905 wait complete. 2023-11-28T22:10:18.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.035+0000 7f609dae1640 1 Processor -- start 2023-11-28T22:10:18.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.035+0000 7f609dae1640 1 -- start start 2023-11-28T22:10:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.036+0000 7f609dae1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6088003970 0x7f609815d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.036+0000 7f609dae1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f609815dcd0 0x7f609809aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.036+0000 7f609dae1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609809afd0 0x7f609809b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.036+0000 7f609dae1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60981f43e0 con 0x7f609815dcd0 2023-11-28T22:10:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.036+0000 7f609dae1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609815e150 con 0x7f609809afd0 2023-11-28T22:10:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.036+0000 7f609dae1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60981f4f20 con 0x7f6088003970 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.037+0000 7f609d2e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609809afd0 0x7f609809b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.037+0000 7f6097fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f609815dcd0 0x7f609809aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.037+0000 7f609d2e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609809afd0 0x7f609809b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:39108/0 (socket says 172.21.15.45:39108) 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.037+0000 7f609d2e0640 1 -- 172.21.15.45:0/799572944 learned_addr learned my addr 172.21.15.45:0/799572944 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.037+0000 7f6097fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f609815dcd0 0x7f609809aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34630/0 (socket says 172.21.15.45:34630) 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.037+0000 7f609cadf640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6088003970 0x7f609815d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f6097fff640 1 -- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6088003970 msgr2=0x7f609815d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f6097fff640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6088003970 0x7f609815d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f6097fff640 1 -- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609809afd0 msgr2=0x7f609809b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f6097fff640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609809afd0 0x7f609809b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f6097fff640 1 -- 172.21.15.45:0/799572944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60981f45f0 con 0x7f609815dcd0 2023-11-28T22:10:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f6097fff640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f609815dcd0 0x7f609809aa90 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f609802cf10 tx=0x7f608400d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f60957fa640 1 -- 172.21.15.45:0/799572944 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608400b070 con 0x7f609815dcd0 2023-11-28T22:10:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f60957fa640 1 -- 172.21.15.45:0/799572944 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60840054e0 con 0x7f609815dcd0 2023-11-28T22:10:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f60957fa640 1 -- 172.21.15.45:0/799572944 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084003830 con 0x7f609815dcd0 2023-11-28T22:10:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.038+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60981f48d0 con 0x7f609815dcd0 2023-11-28T22:10:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.039+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6098142e90 con 0x7f609815dcd0 2023-11-28T22:10:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.040+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609809d0f0 con 0x7f609815dcd0 2023-11-28T22:10:18.042 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.041+0000 7f60957fa640 1 -- 172.21.15.45:0/799572944 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6084003990 con 0x7f609815dcd0 2023-11-28T22:10:18.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.042+0000 7f60957fa640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f607c06af60 0x7f607c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.042+0000 7f60957fa640 1 -- 172.21.15.45:0/799572944 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f6084089090 con 0x7f609815dcd0 2023-11-28T22:10:18.043 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.042+0000 7f609cadf640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f607c06af60 0x7f607c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.044+0000 7f609cadf640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f607c06af60 0x7f607c06d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f608c034000 tx=0x7f608c009030 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:18.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.046+0000 7f60957fa640 1 -- 172.21.15.45:0/799572944 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6084053e60 con 0x7f609815dcd0 2023-11-28T22:10:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.249+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f609804e920 con 0x7f609815dcd0 2023-11-28T22:10:18.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.255+0000 7f60957fa640 1 -- 172.21.15.45:0/799572944 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v73) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6084057480 con 0x7f609815dcd0 2023-11-28T22:10:18.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f607c06af60 msgr2=0x7f607c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.259 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f607c06af60 0x7f607c06d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f608c034000 tx=0x7f608c009030 comp rx=0 tx=0).stop 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f609815dcd0 msgr2=0x7f609809aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f609815dcd0 0x7f609809aa90 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f609802cf10 tx=0x7f608400d890 comp rx=0 tx=0).stop 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 shutdown_connections 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f607c06af60 0x7f607c06d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f609809afd0 0x7f609809b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f609815dcd0 0x7f609809aa90 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 --2- 172.21.15.45:0/799572944 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6088003970 0x7f609815d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 >> 172.21.15.45:0/799572944 conn(0x7f60980cecd0 msgr2=0x7f6098070280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 shutdown_connections 2023-11-28T22:10:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.258+0000 7f609dae1640 1 -- 172.21.15.45:0/799572944 wait complete. 2023-11-28T22:10:18.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T22:10:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.505+0000 7fdd33fd7640 1 Processor -- start 2023-11-28T22:10:18.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.505+0000 7fdd33fd7640 1 -- start start 2023-11-28T22:10:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.507+0000 7fdd33fd7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c0ce160 0x7fdd2c0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.507+0000 7fdd33fd7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd2c06ead0 0x7fdd2c15e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.507+0000 7fdd33fd7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2c15ecc0 con 0x7fdd2c0cc2b0 2023-11-28T22:10:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.507+0000 7fdd33fd7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2c06f270 con 0x7fdd2c06ead0 2023-11-28T22:10:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.507+0000 7fdd33fd7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c06d5d0 con 0x7fdd2c0ce160 2023-11-28T22:10:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.507+0000 7fdd32574640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd2c06ead0 0x7fdd2c15e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.507+0000 7fdd31572640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c0ce160 0x7fdd2c0ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.508+0000 7fdd32574640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd2c06ead0 0x7fdd2c15e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:39120/0 (socket says 172.21.15.45:39120) 2023-11-28T22:10:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.508+0000 7fdd32574640 1 -- 172.21.15.45:0/2793992122 learned_addr learned my addr 172.21.15.45:0/2793992122 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.508+0000 7fdd32574640 1 -- 172.21.15.45:0/2793992122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c0ce160 msgr2=0x7fdd2c0ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.508+0000 7fdd32574640 1 --2- 172.21.15.45:0/2793992122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c0ce160 0x7fdd2c0ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.508+0000 7fdd32574640 1 -- 172.21.15.45:0/2793992122 >> v1:172.21.15.190:6789/0 conn(0x7fdd2c0cc2b0 legacy=0x7fdd2c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.508+0000 7fdd32574640 1 -- 172.21.15.45:0/2793992122 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c0cc690 con 0x7fdd2c06ead0 2023-11-28T22:10:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.510+0000 7fdd32574640 1 --2- 172.21.15.45:0/2793992122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd2c06ead0 0x7fdd2c15e780 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009e50 tx=0x7fdd2402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42f2c13f1310647b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.510+0000 7fdd30d71640 1 -- 172.21.15.45:0/2793992122 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2402cb00 con 0x7fdd2c06ead0 2023-11-28T22:10:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.510+0000 7fdd30d71640 1 -- 172.21.15.45:0/2793992122 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2402cc60 con 0x7fdd2c06ead0 2023-11-28T22:10:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.510+0000 7fdd30d71640 1 -- 172.21.15.45:0/2793992122 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd240343f0 con 0x7fdd2c06ead0 2023-11-28T22:10:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.511+0000 7fdd33fd7640 1 -- 172.21.15.45:0/2793992122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd2c06ead0 msgr2=0x7fdd2c15e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.511+0000 7fdd33fd7640 1 --2- 172.21.15.45:0/2793992122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd2c06ead0 0x7fdd2c15e780 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009e50 tx=0x7fdd2402bf60 comp rx=0 tx=0).stop 2023-11-28T22:10:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.511+0000 7fdd33fd7640 1 -- 172.21.15.45:0/2793992122 shutdown_connections 2023-11-28T22:10:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.511+0000 7fdd33fd7640 1 --2- 172.21.15.45:0/2793992122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd2c06ead0 0x7fdd2c15e780 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.511+0000 7fdd33fd7640 1 --2- 172.21.15.45:0/2793992122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c0ce160 0x7fdd2c0ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.511+0000 7fdd33fd7640 1 -- 172.21.15.45:0/2793992122 >> 172.21.15.45:0/2793992122 conn(0x7fdd2c0cc900 msgr2=0x7fdd2c0ccd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.511+0000 7fdd33fd7640 1 -- 172.21.15.45:0/2793992122 shutdown_connections 2023-11-28T22:10:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.511+0000 7fdd33fd7640 1 -- 172.21.15.45:0/2793992122 wait complete. 2023-11-28T22:10:18.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.513+0000 7fdd33fd7640 1 Processor -- start 2023-11-28T22:10:18.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.513+0000 7fdd33fd7640 1 -- start start 2023-11-28T22:10:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.514+0000 7fdd33fd7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c06ead0 0x7fdd2c0b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.514+0000 7fdd33fd7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd2c0cc2b0 0x7fdd2c14c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.514+0000 7fdd33fd7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd2c0ce160 0x7fdd2c0c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.514+0000 7fdd33fd7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c15a330 con 0x7fdd2c0ce160 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.514+0000 7fdd33fd7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0b3200 con 0x7fdd2c0cc2b0 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.514+0000 7fdd33fd7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0b34e0 con 0x7fdd2c06ead0 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd32574640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd2c0ce160 0x7fdd2c0c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd32574640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd2c0ce160 0x7fdd2c0c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34646/0 (socket says 172.21.15.45:34646) 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd32574640 1 -- 172.21.15.45:0/3241308756 learned_addr learned my addr 172.21.15.45:0/3241308756 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd31572640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd2c0cc2b0 0x7fdd2c14c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd31d73640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c06ead0 0x7fdd2c0b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd32574640 1 -- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c06ead0 msgr2=0x7fdd2c0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd32574640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c06ead0 0x7fdd2c0b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd32574640 1 -- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd2c0cc2b0 msgr2=0x7fdd2c14c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd32574640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd2c0cc2b0 0x7fdd2c14c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.515+0000 7fdd32574640 1 -- 172.21.15.45:0/3241308756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24009b30 con 0x7fdd2c0ce160 2023-11-28T22:10:18.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.516+0000 7fdd32574640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd2c0ce160 0x7fdd2c0c8de0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009e50 tx=0x7fdd24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:18.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.516+0000 7fdd227fc640 1 -- 172.21.15.45:0/3241308756 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2403e450 con 0x7fdd2c0ce160 2023-11-28T22:10:18.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.516+0000 7fdd227fc640 1 -- 172.21.15.45:0/3241308756 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2403e5b0 con 0x7fdd2c0ce160 2023-11-28T22:10:18.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.516+0000 7fdd227fc640 1 -- 172.21.15.45:0/3241308756 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2403e960 con 0x7fdd2c0ce160 2023-11-28T22:10:18.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.516+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c0b37c0 con 0x7fdd2c0ce160 2023-11-28T22:10:18.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.516+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c0b3d80 con 0x7fdd2c0ce160 2023-11-28T22:10:18.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.518+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c0733d0 con 0x7fdd2c0ce160 2023-11-28T22:10:18.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.518+0000 7fdd227fc640 1 -- 172.21.15.45:0/3241308756 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fdd2403eac0 con 0x7fdd2c0ce160 2023-11-28T22:10:18.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.519+0000 7fdd227fc640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fdd1006af60 0x7fdd1006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.519+0000 7fdd227fc640 1 -- 172.21.15.45:0/3241308756 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fdd240ac520 con 0x7fdd2c0ce160 2023-11-28T22:10:18.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.520+0000 7fdd31d73640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fdd1006af60 0x7fdd1006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.521+0000 7fdd31d73640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fdd1006af60 0x7fdd1006d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd28000f60 tx=0x7fdd28009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:18.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.523+0000 7fdd227fc640 1 -- 172.21.15.45:0/3241308756 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd24077350 con 0x7fdd2c0ce160 2023-11-28T22:10:18.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.721+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdd2c0cd4f0 con 0x7fdd2c0ce160 2023-11-28T22:10:18.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.728+0000 7fdd227fc640 1 -- 172.21.15.45:0/3241308756 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdd2407a970 con 0x7fdd2c0ce160 2023-11-28T22:10:18.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.730+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fdd1006af60 msgr2=0x7fdd1006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fdd1006af60 0x7fdd1006d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd28000f60 tx=0x7fdd28009040 comp rx=0 tx=0).stop 2023-11-28T22:10:18.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd2c0ce160 msgr2=0x7fdd2c0c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd2c0ce160 0x7fdd2c0c8de0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009e50 tx=0x7fdd24004a40 comp rx=0 tx=0).stop 2023-11-28T22:10:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 shutdown_connections 2023-11-28T22:10:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd2c0ce160 0x7fdd2c0c8de0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fdd1006af60 0x7fdd1006d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd2c0cc2b0 0x7fdd2c14c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 --2- 172.21.15.45:0/3241308756 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd2c06ead0 0x7fdd2c0b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 >> 172.21.15.45:0/3241308756 conn(0x7fdd2c0cc900 msgr2=0x7fdd2c071dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 shutdown_connections 2023-11-28T22:10:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.731+0000 7fdd33fd7640 1 -- 172.21.15.45:0/3241308756 wait complete. 2023-11-28T22:10:18.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-11-28T22:10:18.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.985+0000 7fa50c915640 1 Processor -- start 2023-11-28T22:10:18.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.985+0000 7fa50c915640 1 -- start start 2023-11-28T22:10:18.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.987+0000 7fa50c915640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa5080ad7e0 0x7fa5080ac7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.987+0000 7fa50c915640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa5080b1d00 0x7fa5080710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.987+0000 7fa50c915640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5080addf0 con 0x7fa50806dbc0 2023-11-28T22:10:18.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.987+0000 7fa50c915640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa508070c20 con 0x7fa5080ad7e0 2023-11-28T22:10:18.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.987+0000 7fa50c915640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5080702f0 con 0x7fa5080b1d00 2023-11-28T22:10:18.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.987+0000 7fa506d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa50806dbc0 0x7fa50806dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36508/0 (socket says 172.21.15.45:36508) 2023-11-28T22:10:18.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.987+0000 7fa506d9d640 1 -- 172.21.15.45:0/3302870611 learned_addr learned my addr 172.21.15.45:0/3302870611 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:18.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.987+0000 7fa50659c640 1 --2- 172.21.15.45:0/3302870611 >> v2:172.21.15.45:3300/0 conn(0x7fa5080ad7e0 0x7fa5080ac7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.989 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.988+0000 7fa50759e640 1 --2- 172.21.15.45:0/3302870611 >> v2:172.21.15.190:3300/0 conn(0x7fa5080b1d00 0x7fa5080710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.988+0000 7fa50659c640 1 -- 172.21.15.45:0/3302870611 >> v2:172.21.15.190:3300/0 conn(0x7fa5080b1d00 msgr2=0x7fa5080710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:18.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.988+0000 7fa50659c640 1 --2- 172.21.15.45:0/3302870611 >> v2:172.21.15.190:3300/0 conn(0x7fa5080b1d00 0x7fa5080710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.988+0000 7fa50659c640 1 -- 172.21.15.45:0/3302870611 >> v1:172.21.15.45:6789/0 conn(0x7fa50806dbc0 legacy=0x7fa50806dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.988+0000 7fa50659c640 1 -- 172.21.15.45:0/3302870611 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5080705d0 con 0x7fa5080ad7e0 2023-11-28T22:10:18.990 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.989+0000 7fa50659c640 1 --2- 172.21.15.45:0/3302870611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080ad7e0 0x7fa5080ac7f0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fa5080ae920 tx=0x7fa4fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2014075f85cbe12 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:18.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.989+0000 7fa505d9b640 1 -- 172.21.15.45:0/3302870611 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc02e070 con 0x7fa5080ad7e0 2023-11-28T22:10:18.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.989+0000 7fa505d9b640 1 -- 172.21.15.45:0/3302870611 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc0315a0 con 0x7fa5080ad7e0 2023-11-28T22:10:18.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.989+0000 7fa505d9b640 1 -- 172.21.15.45:0/3302870611 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc031930 con 0x7fa5080ad7e0 2023-11-28T22:10:18.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.990+0000 7fa50c915640 1 -- 172.21.15.45:0/3302870611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080ad7e0 msgr2=0x7fa5080ac7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.990+0000 7fa50c915640 1 --2- 172.21.15.45:0/3302870611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080ad7e0 0x7fa5080ac7f0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fa5080ae920 tx=0x7fa4fc02b840 comp rx=0 tx=0).stop 2023-11-28T22:10:18.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.990+0000 7fa50c915640 1 -- 172.21.15.45:0/3302870611 shutdown_connections 2023-11-28T22:10:18.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.990+0000 7fa50c915640 1 --2- 172.21.15.45:0/3302870611 >> v2:172.21.15.190:3300/0 conn(0x7fa5080b1d00 0x7fa5080710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.990+0000 7fa50c915640 1 --2- 172.21.15.45:0/3302870611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080ad7e0 0x7fa5080ac7f0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.990+0000 7fa50c915640 1 -- 172.21.15.45:0/3302870611 >> 172.21.15.45:0/3302870611 conn(0x7fa5080cecd0 msgr2=0x7fa5080cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:18.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.990+0000 7fa50c915640 1 -- 172.21.15.45:0/3302870611 shutdown_connections 2023-11-28T22:10:18.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.990+0000 7fa50c915640 1 -- 172.21.15.45:0/3302870611 wait complete. 2023-11-28T22:10:18.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.992+0000 7fa50c915640 1 Processor -- start 2023-11-28T22:10:18.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.992+0000 7fa50c915640 1 -- start start 2023-11-28T22:10:18.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.993+0000 7fa50c915640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa50806dbc0 0x7fa508159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.993+0000 7fa50c915640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ad7e0 0x7fa5081416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.993+0000 7fa50c915640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080b1d00 0x7fa508141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:18.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.993+0000 7fa50c915640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5081421d0 con 0x7fa5080b1d00 2023-11-28T22:10:18.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.993+0000 7fa50c915640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5081424b0 con 0x7fa5080ad7e0 2023-11-28T22:10:18.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.993+0000 7fa50c915640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa508142790 con 0x7fa50806dbc0 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.993+0000 7fa50759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080b1d00 0x7fa508141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.993+0000 7fa50759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080b1d00 0x7fa508141be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34672/0 (socket says 172.21.15.45:34672) 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.993+0000 7fa50759e640 1 -- 172.21.15.45:0/2085825065 learned_addr learned my addr 172.21.15.45:0/2085825065 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa50659c640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ad7e0 0x7fa5081416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa50759e640 1 -- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa50806dbc0 msgr2=0x7fa508159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa50759e640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa50806dbc0 0x7fa508159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa50759e640 1 -- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ad7e0 msgr2=0x7fa5081416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa50759e640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ad7e0 0x7fa5081416a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa50759e640 1 -- 172.21.15.45:0/2085825065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc009400 con 0x7fa5080b1d00 2023-11-28T22:10:18.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa50759e640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080b1d00 0x7fa508141be0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fa500007e60 tx=0x7fa50000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:18.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa4ef7fe640 1 -- 172.21.15.45:0/2085825065 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500014420 con 0x7fa5080b1d00 2023-11-28T22:10:18.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa4ef7fe640 1 -- 172.21.15.45:0/2085825065 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5000145a0 con 0x7fa5080b1d00 2023-11-28T22:10:18.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa4ef7fe640 1 -- 172.21.15.45:0/2085825065 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500014950 con 0x7fa5080b1d00 2023-11-28T22:10:18.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.994+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa508142a70 con 0x7fa5080b1d00 2023-11-28T22:10:18.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.995+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5080ce500 con 0x7fa5080b1d00 2023-11-28T22:10:18.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.996+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa508142f90 con 0x7fa5080b1d00 2023-11-28T22:10:18.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.997+0000 7fa4ef7fe640 1 -- 172.21.15.45:0/2085825065 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa500014ab0 con 0x7fa5080b1d00 2023-11-28T22:10:18.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.998+0000 7fa4ef7fe640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fa4f406af60 0x7fa4f406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:19.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.998+0000 7fa4ef7fe640 1 -- 172.21.15.45:0/2085825065 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fa50008a2f0 con 0x7fa5080b1d00 2023-11-28T22:10:19.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:18.999+0000 7fa506d9d640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fa4f406af60 0x7fa4f406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:19.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.000+0000 7fa506d9d640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fa4f406af60 0x7fa4f406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa50815af40 tx=0x7fa4f8007a40 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:19.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.003+0000 7fa4ef7fe640 1 -- 172.21.15.45:0/2085825065 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa508142f90 con 0x7fa5080b1d00 2023-11-28T22:10:19.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.197+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fa508142f90 con 0x7fa5080b1d00 2023-11-28T22:10:19.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.203+0000 7fa4ef7fe640 1 -- 172.21.15.45:0/2085825065 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v75) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa508142f90 con 0x7fa5080b1d00 2023-11-28T22:10:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.205+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fa4f406af60 msgr2=0x7fa4f406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.205+0000 7fa50c915640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fa4f406af60 0x7fa4f406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa50815af40 tx=0x7fa4f8007a40 comp rx=0 tx=0).stop 2023-11-28T22:10:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.205+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080b1d00 msgr2=0x7fa508141be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.205+0000 7fa50c915640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080b1d00 0x7fa508141be0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fa500007e60 tx=0x7fa50000e4b0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.206+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 shutdown_connections 2023-11-28T22:10:19.207 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.206+0000 7fa50c915640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5080b1d00 0x7fa508141be0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.206+0000 7fa50c915640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fa4f406af60 0x7fa4f406d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.206+0000 7fa50c915640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ad7e0 0x7fa5081416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.206+0000 7fa50c915640 1 --2- 172.21.15.45:0/2085825065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa50806dbc0 0x7fa508159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.206+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 >> 172.21.15.45:0/2085825065 conn(0x7fa5080cecd0 msgr2=0x7fa5080acd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.206+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 shutdown_connections 2023-11-28T22:10:19.208 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.206+0000 7fa50c915640 1 -- 172.21.15.45:0/2085825065 wait complete. 2023-11-28T22:10:19.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-11-28T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.426+0000 7fbe84963640 1 Processor -- start 2023-11-28T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.426+0000 7fbe84963640 1 -- start start 2023-11-28T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.428+0000 7fbe84963640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbe800b3870 0x7fbe8015db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.428+0000 7fbe84963640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbe8006d880 0x7fbe8015e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.428+0000 7fbe84963640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8015e680 con 0x7fbe800721e0 2023-11-28T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.428+0000 7fbe84963640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8015a460 con 0x7fbe800b3870 2023-11-28T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.428+0000 7fbe84963640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8015e960 con 0x7fbe8006d880 2023-11-28T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.428+0000 7fbe7effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbe800721e0 0x7fbe8006d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36514/0 (socket says 172.21.15.45:36514) 2023-11-28T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.428+0000 7fbe7effd640 1 -- 172.21.15.45:0/2420970513 learned_addr learned my addr 172.21.15.45:0/2420970513 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.428+0000 7fbe7f7fe640 1 --2- 172.21.15.45:0/2420970513 >> v2:172.21.15.45:3300/0 conn(0x7fbe800b3870 0x7fbe8015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.429+0000 7fbe7ffff640 1 --2- 172.21.15.45:0/2420970513 >> v2:172.21.15.190:3300/0 conn(0x7fbe8006d880 0x7fbe8015e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.429+0000 7fbe7f7fe640 1 -- 172.21.15.45:0/2420970513 >> v2:172.21.15.190:3300/0 conn(0x7fbe8006d880 msgr2=0x7fbe8015e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.429+0000 7fbe7f7fe640 1 --2- 172.21.15.45:0/2420970513 >> v2:172.21.15.190:3300/0 conn(0x7fbe8006d880 0x7fbe8015e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.429+0000 7fbe7f7fe640 1 -- 172.21.15.45:0/2420970513 >> v1:172.21.15.45:6789/0 conn(0x7fbe800721e0 legacy=0x7fbe8006d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.429+0000 7fbe7f7fe640 1 -- 172.21.15.45:0/2420970513 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe80071700 con 0x7fbe800b3870 2023-11-28T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.429+0000 7fbe7e7fc640 1 -- 172.21.15.45:0/2420970513 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17749712 0 0) 0x7fbe8015e680 con 0x7fbe800721e0 2023-11-28T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.430+0000 7fbe7f7fe640 1 --2- 172.21.15.45:0/2420970513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe800b3870 0x7fbe8015db10 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fbe800cecd0 tx=0x7fbe7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16585af6ac1708f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.430+0000 7fbe7e7fc640 1 -- 172.21.15.45:0/2420970513 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70031830 con 0x7fbe800b3870 2023-11-28T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.430+0000 7fbe7e7fc640 1 -- 172.21.15.45:0/2420970513 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70031990 con 0x7fbe800b3870 2023-11-28T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.430+0000 7fbe7e7fc640 1 -- 172.21.15.45:0/2420970513 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70031d20 con 0x7fbe800b3870 2023-11-28T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.431+0000 7fbe84963640 1 -- 172.21.15.45:0/2420970513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe800b3870 msgr2=0x7fbe8015db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.431+0000 7fbe84963640 1 --2- 172.21.15.45:0/2420970513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe800b3870 0x7fbe8015db10 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fbe800cecd0 tx=0x7fbe7002b8d0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.431+0000 7fbe84963640 1 -- 172.21.15.45:0/2420970513 shutdown_connections 2023-11-28T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.431+0000 7fbe84963640 1 --2- 172.21.15.45:0/2420970513 >> v2:172.21.15.190:3300/0 conn(0x7fbe8006d880 0x7fbe8015e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.431+0000 7fbe84963640 1 --2- 172.21.15.45:0/2420970513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe800b3870 0x7fbe8015db10 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.431+0000 7fbe84963640 1 -- 172.21.15.45:0/2420970513 >> 172.21.15.45:0/2420970513 conn(0x7fbe8006ab30 msgr2=0x7fbe800b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.431+0000 7fbe84963640 1 -- 172.21.15.45:0/2420970513 shutdown_connections 2023-11-28T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.431+0000 7fbe84963640 1 -- 172.21.15.45:0/2420970513 wait complete. 2023-11-28T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.433+0000 7fbe84963640 1 Processor -- start 2023-11-28T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.433+0000 7fbe84963640 1 -- start start 2023-11-28T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.434+0000 7fbe84963640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe8006d880 0x7fbe8009bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.434+0000 7fbe84963640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe800721e0 0x7fbe80155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.434+0000 7fbe84963640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe800b3870 0x7fbe800b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe84963640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8009aae0 con 0x7fbe8006d880 2023-11-28T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe84963640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe80099ed0 con 0x7fbe800b3870 2023-11-28T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe84963640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8009a1b0 con 0x7fbe800721e0 2023-11-28T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe8006d880 0x7fbe8009bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe8006d880 0x7fbe8009bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34684/0 (socket says 172.21.15.45:34684) 2023-11-28T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7f7fe640 1 -- 172.21.15.45:0/2027195909 learned_addr learned my addr 172.21.15.45:0/2027195909 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7ffff640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe800b3870 0x7fbe800b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7effd640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe800721e0 0x7fbe80155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7f7fe640 1 -- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe800721e0 msgr2=0x7fbe80155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7f7fe640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe800721e0 0x7fbe80155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7f7fe640 1 -- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe800b3870 msgr2=0x7fbe800b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7f7fe640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe800b3870 0x7fbe800b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7f7fe640 1 -- 172.21.15.45:0/2027195909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70009400 con 0x7fbe8006d880 2023-11-28T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.435+0000 7fbe7ffff640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe800b3870 0x7fbe800b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.436+0000 7fbe7f7fe640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe8006d880 0x7fbe8009bce0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fbe70000c00 tx=0x7fbe700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.436+0000 7fbe67fff640 1 -- 172.21.15.45:0/2027195909 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70004230 con 0x7fbe8006d880 2023-11-28T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.436+0000 7fbe67fff640 1 -- 172.21.15.45:0/2027195909 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70004390 con 0x7fbe8006d880 2023-11-28T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.436+0000 7fbe67fff640 1 -- 172.21.15.45:0/2027195909 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7003a870 con 0x7fbe8006d880 2023-11-28T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.436+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8009a490 con 0x7fbe8006d880 2023-11-28T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.436+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe8006c6c0 con 0x7fbe8006d880 2023-11-28T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.438+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8009d0f0 con 0x7fbe8006d880 2023-11-28T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.438+0000 7fbe67fff640 1 -- 172.21.15.45:0/2027195909 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbe7003a9d0 con 0x7fbe8006d880 2023-11-28T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.439+0000 7fbe67fff640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbe6c06af90 0x7fbe6c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.439+0000 7fbe67fff640 1 -- 172.21.15.45:0/2027195909 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fbe700abbe0 con 0x7fbe8006d880 2023-11-28T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.440+0000 7fbe7effd640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbe6c06af90 0x7fbe6c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.441+0000 7fbe7effd640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbe6c06af90 0x7fbe6c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbe74007fd0 tx=0x7fbe74009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:19.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.444+0000 7fbe67fff640 1 -- 172.21.15.45:0/2027195909 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe70076900 con 0x7fbe8006d880 2023-11-28T22:10:19.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.712+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fbe8004e920 con 0x7fbe8006d880 2023-11-28T22:10:19.791 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.790+0000 7fbe67fff640 1 -- 172.21.15.45:0/2027195909 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3156) v1 ==== 163+0+0 (secure 0 0 0) 0x7fbe70035020 con 0x7fbe8006d880 2023-11-28T22:10:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbe6c06af90 msgr2=0x7fbe6c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbe6c06af90 0x7fbe6c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbe74007fd0 tx=0x7fbe74009040 comp rx=0 tx=0).stop 2023-11-28T22:10:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe8006d880 msgr2=0x7fbe8009bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe8006d880 0x7fbe8009bce0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fbe70000c00 tx=0x7fbe700046c0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 shutdown_connections 2023-11-28T22:10:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbe6c06af90 0x7fbe6c06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe800b3870 0x7fbe800b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbe800721e0 0x7fbe80155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.795 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 --2- 172.21.15.45:0/2027195909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe8006d880 0x7fbe8009bce0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.793+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 >> 172.21.15.45:0/2027195909 conn(0x7fbe8006ab30 msgr2=0x7fbe800b0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.794+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 shutdown_connections 2023-11-28T22:10:19.796 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:19.794+0000 7fbe84963640 1 -- 172.21.15.45:0/2027195909 wait complete. 2023-11-28T22:10:19.850 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... FAIL 2023-11-28T22:10:19.850 INFO:tasks.cephfs_test_runner: 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner:FAIL: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs/test_snap_schedules.py", line 337, in test_snap_schedule_with_retention 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner: self.verify(TestSnapSchedules.TEST_DIRECTORY, to_wait) 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs/test_snap_schedules.py", line 138, in verify 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner: res = cbk(list(added)) 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs/test_snap_schedules.py", line 333, in verify_added 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner: self.check_scheduled_snapshot(exec_time, timo_1) 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs/test_snap_schedules.py", line 36, in check_scheduled_snapshot 2023-11-28T22:10:19.851 INFO:tasks.cephfs_test_runner: self.assertTrue((delta <= timo + 5) and (delta >= timo - 5)) 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner:AssertionError: False is not true 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner: 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner:Ran 11 tests in 5718.954s 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner: 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner: 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner:FAIL: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T22:10:19.852 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs/test_snap_schedules.py", line 337, in test_snap_schedule_with_retention 2023-11-28T22:10:19.853 INFO:tasks.cephfs_test_runner: self.verify(TestSnapSchedules.TEST_DIRECTORY, to_wait) 2023-11-28T22:10:19.853 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs/test_snap_schedules.py", line 138, in verify 2023-11-28T22:10:19.853 INFO:tasks.cephfs_test_runner: res = cbk(list(added)) 2023-11-28T22:10:19.853 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs/test_snap_schedules.py", line 333, in verify_added 2023-11-28T22:10:19.853 INFO:tasks.cephfs_test_runner: self.check_scheduled_snapshot(exec_time, timo_1) 2023-11-28T22:10:19.853 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs/test_snap_schedules.py", line 36, in check_scheduled_snapshot 2023-11-28T22:10:19.853 INFO:tasks.cephfs_test_runner: self.assertTrue((delta <= timo + 5) and (delta >= timo - 5)) 2023-11-28T22:10:19.853 INFO:tasks.cephfs_test_runner:AssertionError: False is not true 2023-11-28T22:10:19.853 INFO:tasks.cephfs_test_runner: 2023-11-28T22:10:19.853 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T22:10:19.961 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5143c8dfba6c4029833be10f60c36a78 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_43dd94c190073cf16cff1e711dd7f5a7b1cbc883/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-11-28T22:10:19.962 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-11-28T22:10:19.972 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-11-28T22:10:20.016 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-28T22:10:20.016 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:10:20.016 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:10:20.079 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T22:10:20.080 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-11-28T22:10:20.080 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:10:20.133 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T22:10:20.134 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T22:10:20.134 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:10:20.195 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T22:10:20.195 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-28T22:10:20.195 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T22:10:20.251 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054304,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T22:10:20.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:20.251 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T22:10:20.317 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T22:10:20.317 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T22:10:20.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:20.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T22:10:20.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:20.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T22:10:20.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:20.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2023-11-28T22:10:20.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:20.672 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-11-28T22:10:20.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:20.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2023-11-28T22:10:20.869 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2023-11-28T22:10:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T22:10:20.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:20.873 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2023-11-28T22:10:20.934 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-28T22:10:20.934 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-28T22:10:20.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:20.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T22:10:21.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:21.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T22:10:21.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:21.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link delete ceph-brx 2023-11-28T22:10:21.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:21.242 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2023-11-28T22:10:21.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T22:10:21.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link delete ceph-brx 2023-11-28T22:10:21.374 INFO:teuthology.orchestra.run.smithi190.stderr:Cannot find device "ceph-brx" 2023-11-28T22:10:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T22:10:21.378 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-28T22:10:21.388 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T22:10:21.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T22:10:21.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.625+0000 7f356991d640 1 Processor -- start 2023-11-28T22:10:21.627 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.625+0000 7f356991d640 1 -- start start 2023-11-28T22:10:21.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.627+0000 7f356991d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f35640b1340 0x7f356415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:21.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.627+0000 7f356991d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356415bda0 con 0x7f35640b35b0 2023-11-28T22:10:21.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.627+0000 7f356991d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356415c080 con 0x7f356415b850 2023-11-28T22:10:21.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.627+0000 7f356991d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356415ee40 con 0x7f35640b1340 2023-11-28T22:10:21.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.628+0000 7f3562ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f35640b35b0 0x7f356415ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36524/0 (socket says 172.21.15.45:36524) 2023-11-28T22:10:21.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.628+0000 7f3562ffd640 1 -- 172.21.15.45:0/3953977025 learned_addr learned my addr 172.21.15.45:0/3953977025 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:21.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.628+0000 7f35637fe640 1 --2- 172.21.15.45:0/3953977025 >> v2:172.21.15.190:3300/0 conn(0x7f35640b1340 0x7f356415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:21.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.628+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944384683 0 0) 0x7f356415bda0 con 0x7f35640b35b0 2023-11-28T22:10:21.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.628+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550003520 con 0x7f35640b35b0 2023-11-28T22:10:21.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 207925963 0 0) 0x7f3550003520 con 0x7f35640b35b0 2023-11-28T22:10:21.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 >> v2:172.21.15.190:3300/0 conn(0x7f35640b1340 msgr2=0x7f356415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:21.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 --2- 172.21.15.45:0/3953977025 >> v2:172.21.15.190:3300/0 conn(0x7f35640b1340 0x7f356415f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 >> v1:172.21.15.190:6789/0 conn(0x7f356415b850 legacy=0x7f35640b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:21.632 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356415c360 con 0x7f35640b35b0 2023-11-28T22:10:21.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f35540019f0 con 0x7f35640b35b0 2023-11-28T22:10:21.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 >> v1:172.21.15.45:6789/0 conn(0x7f35640b35b0 legacy=0x7f356415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:21.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35500039c0 0x7f3550003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:21.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35500047c0 0x7f3550004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:21.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3550005180 0x7f355000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:21.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356415ee40 con 0x7f3550005180 2023-11-28T22:10:21.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355000c080 con 0x7f35500047c0 2023-11-28T22:10:21.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355000c240 con 0x7f35500039c0 2023-11-28T22:10:21.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.629+0000 7f35637fe640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3550005180 0x7f355000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:21.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.630+0000 7f35627fc640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35500047c0 0x7f3550004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:21.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.630+0000 7f3562ffd640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35500039c0 0x7f3550003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:21.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.630+0000 7f35637fe640 1 -- 172.21.15.45:0/3953977025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35500039c0 msgr2=0x7f3550003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:21.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.630+0000 7f35637fe640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35500039c0 0x7f3550003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.630+0000 7f35637fe640 1 -- 172.21.15.45:0/3953977025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35500047c0 msgr2=0x7f3550004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:21.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.630+0000 7f35637fe640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35500047c0 0x7f3550004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.630+0000 7f35637fe640 1 -- 172.21.15.45:0/3953977025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3550003520 con 0x7f3550005180 2023-11-28T22:10:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.631+0000 7f35637fe640 1 -- 172.21.15.45:0/3953977025 reap_dead start 2023-11-28T22:10:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.631+0000 7f35637fe640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3550005180 0x7f355000b980 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f355c009a90 tx=0x7f35500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45037413b766d586 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.632+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c02c6b0 con 0x7f3550005180 2023-11-28T22:10:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.632+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c02c830 con 0x7f3550005180 2023-11-28T22:10:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.632+0000 7f3561ffb640 1 -- 172.21.15.45:0/3953977025 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c02cbe0 con 0x7f3550005180 2023-11-28T22:10:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.632+0000 7f356991d640 1 -- 172.21.15.45:0/3953977025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3550005180 msgr2=0x7f355000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:21.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.632+0000 7f356991d640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3550005180 0x7f355000b980 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f355c009a90 tx=0x7f35500057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.633+0000 7f356991d640 1 -- 172.21.15.45:0/3953977025 shutdown_connections 2023-11-28T22:10:21.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.633+0000 7f356991d640 1 --2- 172.21.15.45:0/3953977025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3550005180 0x7f355000b980 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.633+0000 7f356991d640 1 -- 172.21.15.45:0/3953977025 >> 172.21.15.45:0/3953977025 conn(0x7f356406a500 msgr2=0x7f356406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:21.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.633+0000 7f356991d640 1 -- 172.21.15.45:0/3953977025 shutdown_connections 2023-11-28T22:10:21.637 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.633+0000 7f356991d640 1 -- 172.21.15.45:0/3953977025 wait complete. 2023-11-28T22:10:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.635+0000 7f356991d640 1 Processor -- start 2023-11-28T22:10:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.635+0000 7f356991d640 1 -- start start 2023-11-28T22:10:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.636+0000 7f356991d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3550005180 0x7f35641e6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.636+0000 7f356991d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35641f6ea0 0x7f35640705d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f356991d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35641e76b0 0x7f3564070b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f356991d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35641f61e0 con 0x7f35641f6ea0 2023-11-28T22:10:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f356991d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35641e70b0 con 0x7f3550005180 2023-11-28T22:10:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f356991d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35641f63c0 con 0x7f35641e76b0 2023-11-28T22:10:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f35627fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35641f6ea0 0x7f35640705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f35627fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35641f6ea0 0x7f35640705d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34700/0 (socket says 172.21.15.45:34700) 2023-11-28T22:10:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f35627fc640 1 -- 172.21.15.45:0/1441105190 learned_addr learned my addr 172.21.15.45:0/1441105190 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f3562ffd640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3550005180 0x7f35641e6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f35637fe640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35641e76b0 0x7f3564070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f35627fc640 1 -- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35641e76b0 msgr2=0x7f3564070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f35627fc640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35641e76b0 0x7f3564070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f35627fc640 1 -- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3550005180 msgr2=0x7f35641e6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f35627fc640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3550005180 0x7f35641e6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.637+0000 7f35627fc640 1 -- 172.21.15.45:0/1441105190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35641f66a0 con 0x7f35641f6ea0 2023-11-28T22:10:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.638+0000 7f35627fc640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35641f6ea0 0x7f35640705d0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f354c00f5c0 tx=0x7f354c00fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.638+0000 7f355bfff640 1 -- 172.21.15.45:0/1441105190 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0052e0 con 0x7f35641f6ea0 2023-11-28T22:10:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.638+0000 7f355bfff640 1 -- 172.21.15.45:0/1441105190 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c005440 con 0x7f35641f6ea0 2023-11-28T22:10:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.638+0000 7f355bfff640 1 -- 172.21.15.45:0/1441105190 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c003810 con 0x7f35641f6ea0 2023-11-28T22:10:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.638+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35641f6980 con 0x7f35641f6ea0 2023-11-28T22:10:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.638+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3564142e70 con 0x7f35641f6ea0 2023-11-28T22:10:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.640+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35641f6b40 con 0x7f35641f6ea0 2023-11-28T22:10:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.641+0000 7f355bfff640 1 -- 172.21.15.45:0/1441105190 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f354c003970 con 0x7f35641f6ea0 2023-11-28T22:10:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.641+0000 7f355bfff640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f354406af60 0x7f354406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.642+0000 7f355bfff640 1 -- 172.21.15.45:0/1441105190 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f354c0897b0 con 0x7f35641f6ea0 2023-11-28T22:10:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.643+0000 7f3562ffd640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f354406af60 0x7f354406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.644+0000 7f3562ffd640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f354406af60 0x7f354406d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3554002230 tx=0x7f3554009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:21.647 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.646+0000 7f355bfff640 1 -- 172.21.15.45:0/1441105190 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f354c0545e0 con 0x7f35641f6ea0 2023-11-28T22:10:21.840 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.839+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35641f6530 con 0x7f354406af60 2023-11-28T22:10:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.846+0000 7f355bfff640 1 -- 172.21.15.45:0/1441105190 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47842 (secure 0 0 0) 0x7f35641f6530 con 0x7f354406af60 2023-11-28T22:10:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:21.849 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T22:10:21.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f354406af60 msgr2=0x7f354406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f354406af60 0x7f354406d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3554002230 tx=0x7f3554009040 comp rx=0 tx=0).stop 2023-11-28T22:10:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35641f6ea0 msgr2=0x7f35640705d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35641f6ea0 0x7f35640705d0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f354c00f5c0 tx=0x7f354c00fa80 comp rx=0 tx=0).stop 2023-11-28T22:10:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 shutdown_connections 2023-11-28T22:10:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35641e76b0 0x7f3564070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35641f6ea0 0x7f35640705d0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f354406af60 0x7f354406d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 --2- 172.21.15.45:0/1441105190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3550005180 0x7f35641e6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:21.852 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 >> 172.21.15.45:0/1441105190 conn(0x7f356406a500 msgr2=0x7f35641447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:21.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 shutdown_connections 2023-11-28T22:10:21.854 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:21.849+0000 7f356991d640 1 -- 172.21.15.45:0/1441105190 wait complete. 2023-11-28T22:10:21.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:10:21.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:10:21.872 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-11-28T22:10:20.712512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4855,"ondisk_log_size":4855,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":442808,"kb_used_data":24028,"kb_used_omap":12,"kb_used_meta":418675,"kb_avail":754531912,"statfs":{"total":773094113280,"available":772640677888,"internally_reserved":0,"allocated":24604672,"data_stored":13841470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":428723760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667486,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-220,"num_read_kb":-74,"num_write":-251,"num_write_kb":-1666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002147"},"pg_stats":[{"pgid":"1.0","version":"219'4855","reported_seq":5778,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-28T22:10:16.280484+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T22:10:16.280484+0000","last_peered":"2023-11-28T22:10:16.280484+0000","last_clean":"2023-11-28T22:10:16.280484+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T22:10:16.280484+0000","last_undegraded":"2023-11-28T22:10:16.280484+0000","last_fullsized":"2023-11-28T22:10:16.280484+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4855,"ondisk_log_size":4855,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575993,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74692,"kb_used_data":3260,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297148,"statfs":{"total":96636764160,"available":96560279552,"internally_reserved":0,"allocated":3338240,"data_stored":2001532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2410000000000001,"15min":1.214},"min":{"1min":0.83799999999999997,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6499999999999999,"5min":1.9019999999999999,"15min":1.9330000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.218,"5min":1.2150000000000001,"15min":1.26},"min":{"1min":0.75800000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5720000000000001,"5min":2.0190000000000001,"15min":2.6819999999999999},"last":0.99199999999999999}]},{"osd":1,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.248,"15min":1.2569999999999999},"min":{"1min":0.88600000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6419999999999999,"5min":2.694,"15min":2.694},"last":1.153},{"interface":"front","average":{"1min":1.371,"5min":1.2749999999999999,"15min":1.29},"min":{"1min":0.93799999999999994,"5min":0.84199999999999997,"15min":0.80900000000000005},"max":{"1min":1.722,"5min":1.756,"15min":3.0430000000000001},"last":1.409}]},{"osd":2,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2809999999999999,"15min":1.3180000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.643,"5min":1.754,"15min":2.7480000000000002},"last":1.181},{"interface":"front","average":{"1min":1.409,"5min":1.371,"15min":1.3999999999999999},"min":{"1min":0.93700000000000006,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.8009999999999999,"5min":1.833,"15min":2.2400000000000002},"last":1.2290000000000001}]},{"osd":3,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3240000000000001,"15min":1.345},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.7729999999999999,"5min":1.7969999999999999,"15min":1.7989999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.444,"15min":1.464},"min":{"1min":1.048,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.784,"5min":2.258,"15min":2.9329999999999998},"last":1.2589999999999999}]},{"osd":4,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2929999999999999,"15min":1.3280000000000001},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.81299999999999994},"max":{"1min":1.7969999999999999,"5min":1.8100000000000001,"15min":2.6930000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.333,"15min":1.365},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8580000000000001,"5min":1.905,"15min":2.371},"last":1.05}]},{"osd":5,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.77500000000000002},"max":{"1min":1.728,"5min":1.9750000000000001,"15min":3.2229999999999999},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.468,"5min":1.48,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.992,"5min":2.0760000000000001,"15min":2.508},"last":1.3100000000000001}]},{"osd":6,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.579,"15min":1.613},"min":{"1min":1.1739999999999999,"5min":1.038,"15min":1.038},"max":{"1min":2.0329999999999999,"5min":2.0720000000000001,"15min":2.4409999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.5089999999999999,"15min":1.532},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.8500000000000001,"5min":2.2010000000000001,"15min":2.2869999999999999},"last":1.284}]}]},{"osd":6,"up_from":13,"seq":55834575993,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42520,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329320,"statfs":{"total":96636764160,"available":96593223680,"internally_reserved":0,"allocated":3092480,"data_stored":1746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1259999999999999,"15min":1.129},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.73699999999999999},"max":{"1min":1.4299999999999999,"5min":1.6679999999999999,"15min":1.772},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.0940000000000001,"15min":1.1299999999999999},"min":{"1min":0.872,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.718,"5min":1.718,"15min":1.7509999999999999},"last":1.635}]},{"osd":1,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.292,"15min":1.298},"min":{"1min":1.0509999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.667,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4239999999999999,"15min":1.4350000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.89300000000000002},"max":{"1min":1.6619999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.4930000000000001}]},{"osd":2,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3440000000000001,"15min":1.3420000000000001},"min":{"1min":0.99199999999999999,"5min":0.89900000000000002,"15min":0.80000000000000004},"max":{"1min":1.579,"5min":2.0779999999999998,"15min":2.4809999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.448,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.145,"5min":0.871,"15min":0.85199999999999998},"max":{"1min":1.8009999999999999,"5min":2.1829999999999998,"15min":2.6269999999999998},"last":0.79600000000000004}]},{"osd":3,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.387,"15min":1.361},"min":{"1min":1.018,"5min":0.92000000000000004,"15min":0.88500000000000001},"max":{"1min":1.7929999999999999,"5min":1.827,"15min":1.9330000000000001},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3120000000000001,"15min":1.331},"min":{"1min":1.008,"5min":0.94599999999999995,"15min":0.86099999999999999},"max":{"1min":1.6359999999999999,"5min":1.7190000000000001,"15min":1.8859999999999999},"last":1.27}]},{"osd":4,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.327,"15min":1.304},"min":{"1min":0.86399999999999999,"5min":0.81599999999999995,"15min":0.81000000000000005},"max":{"1min":1.9219999999999999,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.385},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.3879999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.86199999999999999},"max":{"1min":1.917,"5min":1.917,"15min":2.02},"last":1.8220000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.409,"15min":1.3999999999999999},"min":{"1min":0.97299999999999998,"5min":0.94999999999999996,"15min":0.84199999999999997},"max":{"1min":1.772,"5min":1.9990000000000001,"15min":2.081},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.5980000000000001,"15min":1.5960000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0270000000000001,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":1.79}]},{"osd":7,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.544,"15min":1.5369999999999999},"min":{"1min":1.077,"5min":1.0649999999999999,"15min":0.98899999999999999},"max":{"1min":1.9870000000000001,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.573,"5min":1.504,"15min":1.526},"min":{"1min":1.252,"5min":1.0029999999999999,"15min":0.879},"max":{"1min":2.024,"5min":2.024,"15min":2.2130000000000001},"last":1.444}]}]},{"osd":5,"up_from":13,"seq":55834575993,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60264,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311576,"statfs":{"total":96636764160,"available":96575053824,"internally_reserved":0,"allocated":2781184,"data_stored":1429252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1539999999999999,"15min":1.173},"min":{"1min":0.71699999999999997,"5min":0.65700000000000003,"15min":0.48499999999999999},"max":{"1min":1.6120000000000001,"5min":1.9039999999999999,"15min":1.9550000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2170000000000001,"15min":1.2},"min":{"1min":0.89600000000000002,"5min":0.80600000000000005,"15min":0.65700000000000003},"max":{"1min":1.8069999999999999,"5min":1.889,"15min":1.889},"last":1.3100000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1990000000000001,"15min":1.212},"min":{"1min":0.77200000000000002,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.494,"5min":1.706,"15min":1.8979999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2789999999999999,"15min":1.286},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.54400000000000004},"max":{"1min":1.7370000000000001,"5min":1.7509999999999999,"15min":1.8859999999999999},"last":0.96999999999999997}]},{"osd":2,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.3100000000000001,"15min":1.3260000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.60699999999999998},"max":{"1min":1.677,"5min":1.768,"15min":1.867},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.361,"15min":1.3560000000000001},"min":{"1min":0.81799999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.7410000000000001,"5min":1.8899999999999999,"15min":2.044},"last":1.204}]},{"osd":3,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4490000000000001,"15min":1.4139999999999999},"min":{"1min":1.1599999999999999,"5min":0.96999999999999997,"15min":0.68899999999999995},"max":{"1min":1.5720000000000001,"5min":1.8180000000000001,"15min":1.988},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.3069999999999999,"15min":1.3380000000000001},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.87},"max":{"1min":1.595,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.492}]},{"osd":4,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3720000000000001,"15min":1.377},"min":{"1min":0.873,"5min":0.84899999999999998,"15min":0.72599999999999998},"max":{"1min":1.802,"5min":1.833,"15min":1.833},"last":1.788},{"interface":"front","average":{"1min":1.48,"5min":1.4550000000000001,"15min":1.494},"min":{"1min":1.103,"5min":0.83999999999999997,"15min":0.76200000000000001},"max":{"1min":1.792,"5min":1.8440000000000001,"15min":2.0249999999999999},"last":1.758}]},{"osd":6,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4379999999999999,"15min":1.415},"min":{"1min":0.88800000000000001,"5min":0.84099999999999997,"15min":0.80000000000000004},"max":{"1min":1.885,"5min":1.899,"15min":2.0960000000000001},"last":1.728},{"interface":"front","average":{"1min":1.528,"5min":1.5069999999999999,"15min":1.544},"min":{"1min":1.1619999999999999,"5min":0.97199999999999998,"15min":0.94399999999999995},"max":{"1min":1.7849999999999999,"5min":1.982,"15min":1.982},"last":1.9239999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.46,"15min":1.484},"min":{"1min":1.069,"5min":1.0580000000000001,"15min":0.76100000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.1880000000000002},"last":1.8759999999999999},{"interface":"front","average":{"1min":1.589,"5min":1.5660000000000001,"15min":1.5740000000000001},"min":{"1min":1.214,"5min":0.877,"15min":0.877},"max":{"1min":1.905,"5min":1.9790000000000001,"15min":2.1970000000000001},"last":1.9770000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51944,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319896,"statfs":{"total":96636764160,"available":96583573504,"internally_reserved":0,"allocated":2781184,"data_stored":1429264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.0900000000000001,"15min":1.131},"min":{"1min":0.79500000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.8149999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1619999999999999,"15min":1.1579999999999999},"min":{"1min":0.82199999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5940000000000001,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.90900000000000003}]},{"osd":1,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.3720000000000001,"15min":1.3839999999999999},"min":{"1min":0.88,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.873,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.383,"15min":1.4119999999999999},"min":{"1min":1.109,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.77,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.23}]},{"osd":2,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.266,"15min":1.266},"min":{"1min":0.95199999999999996,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.7509999999999999,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.3280000000000001,"15min":1.3500000000000001},"min":{"1min":1.034,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.927,"5min":1.927,"15min":1.944},"last":1.3300000000000001}]},{"osd":3,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3220000000000001,"15min":1.3080000000000001},"min":{"1min":1.002,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8120000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.421,"15min":1.4099999999999999},"min":{"1min":1.1399999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.8169999999999999,"5min":1.929,"15min":2.036},"last":0.96099999999999997}]},{"osd":5,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3600000000000001,"15min":1.3280000000000001},"min":{"1min":0.91900000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.8,"5min":2.0030000000000001,"15min":2.0609999999999999},"last":1.427},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3700000000000001,"15min":1.359},"min":{"1min":0.92000000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7769999999999999,"5min":1.9139999999999999,"15min":2.1659999999999999},"last":1.0209999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.492,"15min":1.446},"min":{"1min":1.01,"5min":0.70899999999999996,"15min":0.61499999999999999},"max":{"1min":2.004,"5min":2.004,"15min":2.097},"last":1.181},{"interface":"front","average":{"1min":1.661,"5min":1.615,"15min":1.601},"min":{"1min":1.296,"5min":0.75,"15min":0.75},"max":{"1min":2.1120000000000001,"5min":2.1150000000000002,"15min":2.23},"last":1.4910000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.5,"15min":1.4950000000000001},"min":{"1min":1.2290000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8680000000000001,"5min":2.0190000000000001,"15min":2.3340000000000001},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.611,"15min":1.5920000000000001},"min":{"1min":1.3759999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1659999999999999,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.28}]}]},{"osd":3,"up_from":13,"seq":55834575994,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56400,"kb_used_data":3012,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315440,"statfs":{"total":96636764160,"available":96579010560,"internally_reserved":0,"allocated":3084288,"data_stored":1739556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.94299999999999995,"15min":0.95599999999999996},"min":{"1min":0.72199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4530000000000001,"5min":2.0760000000000001,"15min":2.0950000000000002},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.135,"15min":1.1339999999999999},"min":{"1min":0.51400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.3879999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.012,"15min":1.0289999999999999},"min":{"1min":0.70799999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.677,"5min":1.677,"15min":1.698},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.214,"15min":1.256},"min":{"1min":0.77600000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.718,"5min":1.956,"15min":2.6019999999999999},"last":1.167}]},{"osd":2,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2629999999999999,"15min":1.3},"min":{"1min":0.86399999999999999,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5529999999999999,"5min":2.105,"15min":2.327},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.369,"15min":1.397},"min":{"1min":0.70999999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.782,"5min":1.9119999999999999,"15min":2.218},"last":1.3740000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.325,"15min":1.379},"min":{"1min":1.0029999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.706,"5min":1.948,"15min":2.0379999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.474,"5min":1.446,"15min":1.482},"min":{"1min":1.0189999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.165},"last":1.2290000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.5329999999999999,"15min":1.55},"min":{"1min":1.331,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.98,"5min":2.2989999999999999,"15min":2.4329999999999998},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.498,"15min":1.5109999999999999},"min":{"1min":1.0760000000000001,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":1.8240000000000001,"5min":2.1389999999999998,"15min":2.2639999999999998},"last":0.96599999999999997}]},{"osd":6,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5580000000000001,"15min":1.5720000000000001},"min":{"1min":1.1990000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.9830000000000001,"5min":2.0640000000000001,"15min":2.2109999999999999},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.637,"15min":1.6659999999999999},"min":{"1min":1.425,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":2.0779999999999998,"5min":2.379,"15min":2.5329999999999999},"last":1.5960000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.629,"15min":1.6359999999999999},"min":{"1min":1.1579999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0310000000000001,"5min":2.2599999999999998,"15min":2.266},"last":1.145},{"interface":"front","average":{"1min":1.738,"5min":1.6919999999999999,"15min":1.694},"min":{"1min":1.3500000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.129,"5min":2.1800000000000002,"15min":2.3029999999999999},"last":1.4379999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575994,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52232,"kb_used_data":3004,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319608,"statfs":{"total":96636764160,"available":96583278592,"internally_reserved":0,"allocated":3076096,"data_stored":1739426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.98699999999999999,"15min":0.97799999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.113,"15min":1.1459999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.0960000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.0049999999999999,"15min":1.0169999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.161,"5min":2.097,"15min":2.097},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.1659999999999999,"15min":1.2210000000000001},"min":{"1min":0.59599999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.331,"5min":1.625,"15min":1.8149999999999999},"last":1.2130000000000001}]},{"osd":3,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.206,"15min":1.25},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.625,"5min":1.72,"15min":1.9330000000000001},"last":1.024},{"interface":"front","average":{"1min":1.123,"5min":1.337,"15min":1.331},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.575,"5min":1.849,"15min":2.2210000000000001},"last":0.65800000000000003}]},{"osd":4,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.4530000000000001,"15min":1.4670000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6799999999999999,"5min":1.956,"15min":2.0640000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4330000000000001,"15min":1.464},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.8480000000000001,"5min":1.8779999999999999,"15min":2.173},"last":4.4619999999999997}]},{"osd":5,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4830000000000001,"15min":1.5049999999999999},"min":{"1min":1.0229999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.792},{"interface":"front","average":{"1min":1.444,"5min":1.552,"15min":1.5700000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.976,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.4339999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.5569999999999999,"15min":1.605},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8029999999999999,"5min":2.387,"15min":2.387},"last":3.1659999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.635,"15min":1.653},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.8620000000000001,"5min":2.1640000000000001,"15min":2.4020000000000001},"last":1.538}]},{"osd":7,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.589,"15min":1.615},"min":{"1min":1.153,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.702},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.6359999999999999,"15min":1.698},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.8520000000000001,"5min":2.2599999999999998,"15min":2.2690000000000001},"last":3.7789999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52240,"kb_used_data":3012,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319600,"statfs":{"total":96636764160,"available":96583270400,"internally_reserved":0,"allocated":3084288,"data_stored":1746134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92300000000000004,"15min":0.92000000000000004},"min":{"1min":0.55000000000000004,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.417,"5min":1.6240000000000001,"15min":1.8839999999999999},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.153,"15min":1.1579999999999999},"min":{"1min":0.64800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.546,"5min":1.9410000000000001,"15min":2.3900000000000001},"last":1.01}]},{"osd":2,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.147,"15min":1.1599999999999999},"min":{"1min":0.623,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.464,"5min":1.984,"15min":1.9890000000000001},"last":1.038},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2669999999999999,"15min":1.2729999999999999},"min":{"1min":0.58299999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.77,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.363,"15min":1.3999999999999999},"min":{"1min":0.71499999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.722,"5min":2.2040000000000002,"15min":2.2320000000000002},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.79000000000000004,"5min":0.57799999999999996,"15min":0.46800000000000003},"max":{"1min":1.641,"5min":1.786,"15min":2.4359999999999999},"last":0.93600000000000005}]},{"osd":4,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4019999999999999,"15min":1.427},"min":{"1min":1.089,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3559999999999999},"last":1.375},{"interface":"front","average":{"1min":1.399,"5min":1.4099999999999999,"15min":1.4379999999999999},"min":{"1min":0.99199999999999999,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.3079999999999998},"last":2.1680000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4750000000000001,"15min":1.4930000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.77800000000000002},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":2.032},{"interface":"front","average":{"1min":1.544,"5min":1.5880000000000001,"15min":1.6120000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.925,"5min":2.3580000000000001,"15min":2.448},"last":1.7829999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.522,"15min":1.516},"min":{"1min":1.028,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.5249999999999999},"last":2.0800000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.579,"15min":1.593},"min":{"1min":1.175,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.246,"5min":2.246,"15min":2.4550000000000001},"last":1.9179999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.665,"15min":1.6950000000000001},"min":{"1min":1.131,"5min":1.125,"15min":1.125},"max":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.571,"5min":1.5740000000000001,"15min":1.591},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":2.0369999999999999,"5min":2.2730000000000001,"15min":2.8460000000000001},"last":2.2360000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575994,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52516,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319324,"statfs":{"total":96636764160,"available":96582987776,"internally_reserved":0,"allocated":3366912,"data_stored":2009550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.93799999999999994,"15min":0.93200000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.728,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.1439999999999999,"15min":1.1830000000000001},"min":{"1min":0.751,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.2869999999999999},"last":1.272}]},{"osd":2,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1279999999999999,"15min":1.1779999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.1709999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.367,"5min":1.256,"15min":1.244},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":1.458}]},{"osd":3,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.1830000000000001,"15min":1.179},"min":{"1min":0.80100000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.655,"5min":1.8009999999999999,"15min":1.986},"last":1.077},{"interface":"front","average":{"1min":1.393,"5min":1.387,"15min":1.3819999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":1.347}]},{"osd":4,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.484,"15min":1.4750000000000001},"min":{"1min":1.016,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.395,"5min":2.395,"15min":2.395},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.411,"15min":1.4330000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.8109999999999999,"5min":2.149,"15min":2.254},"last":1.679}]},{"osd":5,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.48,"15min":1.4870000000000001},"min":{"1min":0.94999999999999996,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.8959999999999999,"5min":2.0819999999999999,"15min":2.1520000000000001},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.585,"15min":1.5640000000000001},"min":{"1min":1.1970000000000001,"5min":1.05,"15min":1.05},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.3359999999999999},"last":1.633}]},{"osd":6,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.675,"15min":1.639},"min":{"1min":1.202,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.722},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6259999999999999,"15min":1.611},"min":{"1min":0.97399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.756}]},{"osd":7,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.6990000000000001,"15min":1.6859999999999999},"min":{"1min":1.034,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6060000000000001,"15min":1.625},"min":{"1min":1.177,"5min":0.97899999999999998,"15min":0.97899999999999998},"max":{"1min":1.984,"5min":2.4089999999999998,"15min":2.4180000000000001},"last":1.5549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T22:10:21.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.124+0000 7fb60a86d640 1 Processor -- start 2023-11-28T22:10:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.124+0000 7fb60a86d640 1 -- start start 2023-11-28T22:10:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.126+0000 7fb60a86d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb604072200 0x7fb6040b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.126+0000 7fb60a86d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb6040713e0 0x7fb6040b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.126+0000 7fb60a86d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6040b4310 con 0x7fb604146610 2023-11-28T22:10:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.126+0000 7fb60a86d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6040725e0 con 0x7fb6040713e0 2023-11-28T22:10:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.126+0000 7fb60a86d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60406d8d0 con 0x7fb604072200 2023-11-28T22:10:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.126+0000 7fb60906a640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb604146610 0x7fb604160e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36534/0 (socket says 172.21.15.45:36534) 2023-11-28T22:10:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.126+0000 7fb60906a640 1 -- 172.21.15.45:0/3295976101 learned_addr learned my addr 172.21.15.45:0/3295976101 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.126+0000 7fb60a06c640 1 --2- 172.21.15.45:0/3295976101 >> v2:172.21.15.45:3300/0 conn(0x7fb6040713e0 0x7fb6040b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.127+0000 7fb60986b640 1 --2- 172.21.15.45:0/3295976101 >> v2:172.21.15.190:3300/0 conn(0x7fb604072200 0x7fb6040b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.127+0000 7fb60a06c640 1 -- 172.21.15.45:0/3295976101 >> v2:172.21.15.190:3300/0 conn(0x7fb604072200 msgr2=0x7fb6040b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.127+0000 7fb60a06c640 1 --2- 172.21.15.45:0/3295976101 >> v2:172.21.15.190:3300/0 conn(0x7fb604072200 0x7fb6040b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.127+0000 7fb60a06c640 1 -- 172.21.15.45:0/3295976101 >> v1:172.21.15.45:6789/0 conn(0x7fb604146610 legacy=0x7fb604160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.132 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.127+0000 7fb60a06c640 1 -- 172.21.15.45:0/3295976101 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60406ca20 con 0x7fb6040713e0 2023-11-28T22:10:22.133 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.127+0000 7fb608869640 1 -- 172.21.15.45:0/3295976101 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212847100 0 0) 0x7fb5f4001110 con 0x7fb604146610 2023-11-28T22:10:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.128+0000 7fb60a06c640 1 --2- 172.21.15.45:0/3295976101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6040713e0 0x7fb6040b22d0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fb6040ae9d0 tx=0x7fb5fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0cedec757ceeca9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.129+0000 7fb608869640 1 -- 172.21.15.45:0/3295976101 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc02cc80 con 0x7fb6040713e0 2023-11-28T22:10:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.129+0000 7fb608869640 1 -- 172.21.15.45:0/3295976101 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc02cde0 con 0x7fb6040713e0 2023-11-28T22:10:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.129+0000 7fb608869640 1 -- 172.21.15.45:0/3295976101 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc034660 con 0x7fb6040713e0 2023-11-28T22:10:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.129+0000 7fb60a86d640 1 -- 172.21.15.45:0/3295976101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6040713e0 msgr2=0x7fb6040b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.129+0000 7fb60a86d640 1 --2- 172.21.15.45:0/3295976101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6040713e0 0x7fb6040b22d0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fb6040ae9d0 tx=0x7fb5fc02bed0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.129+0000 7fb60a86d640 1 -- 172.21.15.45:0/3295976101 shutdown_connections 2023-11-28T22:10:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.129+0000 7fb60a86d640 1 --2- 172.21.15.45:0/3295976101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6040713e0 0x7fb6040b22d0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.129+0000 7fb60a86d640 1 --2- 172.21.15.45:0/3295976101 >> v2:172.21.15.190:3300/0 conn(0x7fb604072200 0x7fb6040b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.129+0000 7fb60a86d640 1 -- 172.21.15.45:0/3295976101 >> 172.21.15.45:0/3295976101 conn(0x7fb60406ab30 msgr2=0x7fb6040b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.130+0000 7fb60a86d640 1 -- 172.21.15.45:0/3295976101 shutdown_connections 2023-11-28T22:10:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.130+0000 7fb60a86d640 1 -- 172.21.15.45:0/3295976101 wait complete. 2023-11-28T22:10:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.131+0000 7fb60a86d640 1 Processor -- start 2023-11-28T22:10:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.132+0000 7fb60a86d640 1 -- start start 2023-11-28T22:10:22.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.132+0000 7fb60a86d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6040713e0 0x7fb6041426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.132+0000 7fb60a86d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb604072200 0x7fb6040acde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60a86d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb604146610 0x7fb60415c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60a86d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6040ad320 con 0x7fb604072200 2023-11-28T22:10:22.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60a86d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6040ae5f0 con 0x7fb604146610 2023-11-28T22:10:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60a86d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6040ad550 con 0x7fb6040713e0 2023-11-28T22:10:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60906a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb604072200 0x7fb6040acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60906a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb604072200 0x7fb6040acde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34726/0 (socket says 172.21.15.45:34726) 2023-11-28T22:10:22.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60906a640 1 -- 172.21.15.45:0/926802050 learned_addr learned my addr 172.21.15.45:0/926802050 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:22.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60a06c640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb604146610 0x7fb60415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60986b640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6040713e0 0x7fb6041426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60906a640 1 -- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6040713e0 msgr2=0x7fb6041426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60906a640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6040713e0 0x7fb6041426b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60906a640 1 -- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb604146610 msgr2=0x7fb60415c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60906a640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb604146610 0x7fb60415c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60906a640 1 -- 172.21.15.45:0/926802050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc009a70 con 0x7fb604072200 2023-11-28T22:10:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.133+0000 7fb60a06c640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb604146610 0x7fb60415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:22.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.134+0000 7fb60906a640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb604072200 0x7fb6040acde0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c430 tx=0x7fb5f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.134+0000 7fb5fa7fc640 1 -- 172.21.15.45:0/926802050 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f40074e0 con 0x7fb604072200 2023-11-28T22:10:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.134+0000 7fb5fa7fc640 1 -- 172.21.15.45:0/926802050 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f4007660 con 0x7fb604072200 2023-11-28T22:10:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.134+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6040ad830 con 0x7fb604072200 2023-11-28T22:10:22.145 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.134+0000 7fb5fa7fc640 1 -- 172.21.15.45:0/926802050 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4004a20 con 0x7fb604072200 2023-11-28T22:10:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.134+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6040addf0 con 0x7fb604072200 2023-11-28T22:10:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.135+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb604142bf0 con 0x7fb604072200 2023-11-28T22:10:22.147 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.137+0000 7fb5fa7fc640 1 -- 172.21.15.45:0/926802050 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb5f4004b80 con 0x7fb604072200 2023-11-28T22:10:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.138+0000 7fb5fa7fc640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb5e806af90 0x7fb5e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.138+0000 7fb5fa7fc640 1 -- 172.21.15.45:0/926802050 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fb5f40896b0 con 0x7fb604072200 2023-11-28T22:10:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.139+0000 7fb60986b640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb5e806af90 0x7fb5e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.148 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.140+0000 7fb60986b640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb5e806af90 0x7fb5e806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb600006fd0 tx=0x7fb600008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:22.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.143+0000 7fb5fa7fc640 1 -- 172.21.15.45:0/926802050 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5f40543b0 con 0x7fb604072200 2023-11-28T22:10:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.337+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb60406d280 con 0x7fb5e806af90 2023-11-28T22:10:22.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.343+0000 7fb5fa7fc640 1 -- 172.21.15.45:0/926802050 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47842 (secure 0 0 0) 0x7fb600009d60 con 0x7fb5e806af90 2023-11-28T22:10:22.346 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:22.346 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T22:10:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb5e806af90 msgr2=0x7fb5e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb5e806af90 0x7fb5e806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb600006fd0 tx=0x7fb600008040 comp rx=0 tx=0).stop 2023-11-28T22:10:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb604072200 msgr2=0x7fb6040acde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb604072200 0x7fb6040acde0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c430 tx=0x7fb5f400c8f0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 shutdown_connections 2023-11-28T22:10:22.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb5e806af90 0x7fb5e806d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb604146610 0x7fb60415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb604072200 0x7fb6040acde0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 --2- 172.21.15.45:0/926802050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6040713e0 0x7fb6041426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 >> 172.21.15.45:0/926802050 conn(0x7fb60406ab30 msgr2=0x7fb6040b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:22.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.346+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 shutdown_connections 2023-11-28T22:10:22.350 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.347+0000 7fb60a86d640 1 -- 172.21.15.45:0/926802050 wait complete. 2023-11-28T22:10:22.368 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-11-28T22:10:20.712512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4855,"ondisk_log_size":4855,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":442808,"kb_used_data":24028,"kb_used_omap":12,"kb_used_meta":418675,"kb_avail":754531912,"statfs":{"total":773094113280,"available":772640677888,"internally_reserved":0,"allocated":24604672,"data_stored":13841470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":428723760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667486,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-220,"num_read_kb":-74,"num_write":-251,"num_write_kb":-1666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002147"},"pg_stats":[{"pgid":"1.0","version":"219'4855","reported_seq":5778,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-11-28T22:10:16.280484+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T22:10:16.280484+0000","last_peered":"2023-11-28T22:10:16.280484+0000","last_clean":"2023-11-28T22:10:16.280484+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T22:10:16.280484+0000","last_undegraded":"2023-11-28T22:10:16.280484+0000","last_fullsized":"2023-11-28T22:10:16.280484+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4855,"ondisk_log_size":4855,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575993,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74692,"kb_used_data":3260,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297148,"statfs":{"total":96636764160,"available":96560279552,"internally_reserved":0,"allocated":3338240,"data_stored":2001532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2410000000000001,"15min":1.214},"min":{"1min":0.83799999999999997,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6499999999999999,"5min":1.9019999999999999,"15min":1.9330000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.218,"5min":1.2150000000000001,"15min":1.26},"min":{"1min":0.75800000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5720000000000001,"5min":2.0190000000000001,"15min":2.6819999999999999},"last":0.99199999999999999}]},{"osd":1,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.248,"15min":1.2569999999999999},"min":{"1min":0.88600000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6419999999999999,"5min":2.694,"15min":2.694},"last":1.153},{"interface":"front","average":{"1min":1.371,"5min":1.2749999999999999,"15min":1.29},"min":{"1min":0.93799999999999994,"5min":0.84199999999999997,"15min":0.80900000000000005},"max":{"1min":1.722,"5min":1.756,"15min":3.0430000000000001},"last":1.409}]},{"osd":2,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2809999999999999,"15min":1.3180000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.643,"5min":1.754,"15min":2.7480000000000002},"last":1.181},{"interface":"front","average":{"1min":1.409,"5min":1.371,"15min":1.3999999999999999},"min":{"1min":0.93700000000000006,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.8009999999999999,"5min":1.833,"15min":2.2400000000000002},"last":1.2290000000000001}]},{"osd":3,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3240000000000001,"15min":1.345},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.7729999999999999,"5min":1.7969999999999999,"15min":1.7989999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.444,"15min":1.464},"min":{"1min":1.048,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.784,"5min":2.258,"15min":2.9329999999999998},"last":1.2589999999999999}]},{"osd":4,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2929999999999999,"15min":1.3280000000000001},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.81299999999999994},"max":{"1min":1.7969999999999999,"5min":1.8100000000000001,"15min":2.6930000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.333,"15min":1.365},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8580000000000001,"5min":1.905,"15min":2.371},"last":1.05}]},{"osd":5,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.77500000000000002},"max":{"1min":1.728,"5min":1.9750000000000001,"15min":3.2229999999999999},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.468,"5min":1.48,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.992,"5min":2.0760000000000001,"15min":2.508},"last":1.3100000000000001}]},{"osd":6,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.579,"15min":1.613},"min":{"1min":1.1739999999999999,"5min":1.038,"15min":1.038},"max":{"1min":2.0329999999999999,"5min":2.0720000000000001,"15min":2.4409999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.5089999999999999,"15min":1.532},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.8500000000000001,"5min":2.2010000000000001,"15min":2.2869999999999999},"last":1.284}]}]},{"osd":6,"up_from":13,"seq":55834575993,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42520,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329320,"statfs":{"total":96636764160,"available":96593223680,"internally_reserved":0,"allocated":3092480,"data_stored":1746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1259999999999999,"15min":1.129},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.73699999999999999},"max":{"1min":1.4299999999999999,"5min":1.6679999999999999,"15min":1.772},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.0940000000000001,"15min":1.1299999999999999},"min":{"1min":0.872,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.718,"5min":1.718,"15min":1.7509999999999999},"last":1.635}]},{"osd":1,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.292,"15min":1.298},"min":{"1min":1.0509999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.667,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4239999999999999,"15min":1.4350000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.89300000000000002},"max":{"1min":1.6619999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.4930000000000001}]},{"osd":2,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3440000000000001,"15min":1.3420000000000001},"min":{"1min":0.99199999999999999,"5min":0.89900000000000002,"15min":0.80000000000000004},"max":{"1min":1.579,"5min":2.0779999999999998,"15min":2.4809999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.448,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.145,"5min":0.871,"15min":0.85199999999999998},"max":{"1min":1.8009999999999999,"5min":2.1829999999999998,"15min":2.6269999999999998},"last":0.79600000000000004}]},{"osd":3,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.387,"15min":1.361},"min":{"1min":1.018,"5min":0.92000000000000004,"15min":0.88500000000000001},"max":{"1min":1.7929999999999999,"5min":1.827,"15min":1.9330000000000001},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3120000000000001,"15min":1.331},"min":{"1min":1.008,"5min":0.94599999999999995,"15min":0.86099999999999999},"max":{"1min":1.6359999999999999,"5min":1.7190000000000001,"15min":1.8859999999999999},"last":1.27}]},{"osd":4,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.327,"15min":1.304},"min":{"1min":0.86399999999999999,"5min":0.81599999999999995,"15min":0.81000000000000005},"max":{"1min":1.9219999999999999,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.385},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.3879999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.86199999999999999},"max":{"1min":1.917,"5min":1.917,"15min":2.02},"last":1.8220000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.409,"15min":1.3999999999999999},"min":{"1min":0.97299999999999998,"5min":0.94999999999999996,"15min":0.84199999999999997},"max":{"1min":1.772,"5min":1.9990000000000001,"15min":2.081},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.5980000000000001,"15min":1.5960000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0270000000000001,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":1.79}]},{"osd":7,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.544,"15min":1.5369999999999999},"min":{"1min":1.077,"5min":1.0649999999999999,"15min":0.98899999999999999},"max":{"1min":1.9870000000000001,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.573,"5min":1.504,"15min":1.526},"min":{"1min":1.252,"5min":1.0029999999999999,"15min":0.879},"max":{"1min":2.024,"5min":2.024,"15min":2.2130000000000001},"last":1.444}]}]},{"osd":5,"up_from":13,"seq":55834575993,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60264,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311576,"statfs":{"total":96636764160,"available":96575053824,"internally_reserved":0,"allocated":2781184,"data_stored":1429252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1539999999999999,"15min":1.173},"min":{"1min":0.71699999999999997,"5min":0.65700000000000003,"15min":0.48499999999999999},"max":{"1min":1.6120000000000001,"5min":1.9039999999999999,"15min":1.9550000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2170000000000001,"15min":1.2},"min":{"1min":0.89600000000000002,"5min":0.80600000000000005,"15min":0.65700000000000003},"max":{"1min":1.8069999999999999,"5min":1.889,"15min":1.889},"last":1.3100000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1990000000000001,"15min":1.212},"min":{"1min":0.77200000000000002,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.494,"5min":1.706,"15min":1.8979999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2789999999999999,"15min":1.286},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.54400000000000004},"max":{"1min":1.7370000000000001,"5min":1.7509999999999999,"15min":1.8859999999999999},"last":0.96999999999999997}]},{"osd":2,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.3100000000000001,"15min":1.3260000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.60699999999999998},"max":{"1min":1.677,"5min":1.768,"15min":1.867},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.361,"15min":1.3560000000000001},"min":{"1min":0.81799999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.7410000000000001,"5min":1.8899999999999999,"15min":2.044},"last":1.204}]},{"osd":3,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4490000000000001,"15min":1.4139999999999999},"min":{"1min":1.1599999999999999,"5min":0.96999999999999997,"15min":0.68899999999999995},"max":{"1min":1.5720000000000001,"5min":1.8180000000000001,"15min":1.988},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.3069999999999999,"15min":1.3380000000000001},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.87},"max":{"1min":1.595,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.492}]},{"osd":4,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3720000000000001,"15min":1.377},"min":{"1min":0.873,"5min":0.84899999999999998,"15min":0.72599999999999998},"max":{"1min":1.802,"5min":1.833,"15min":1.833},"last":1.788},{"interface":"front","average":{"1min":1.48,"5min":1.4550000000000001,"15min":1.494},"min":{"1min":1.103,"5min":0.83999999999999997,"15min":0.76200000000000001},"max":{"1min":1.792,"5min":1.8440000000000001,"15min":2.0249999999999999},"last":1.758}]},{"osd":6,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4379999999999999,"15min":1.415},"min":{"1min":0.88800000000000001,"5min":0.84099999999999997,"15min":0.80000000000000004},"max":{"1min":1.885,"5min":1.899,"15min":2.0960000000000001},"last":1.728},{"interface":"front","average":{"1min":1.528,"5min":1.5069999999999999,"15min":1.544},"min":{"1min":1.1619999999999999,"5min":0.97199999999999998,"15min":0.94399999999999995},"max":{"1min":1.7849999999999999,"5min":1.982,"15min":1.982},"last":1.9239999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.46,"15min":1.484},"min":{"1min":1.069,"5min":1.0580000000000001,"15min":0.76100000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.1880000000000002},"last":1.8759999999999999},{"interface":"front","average":{"1min":1.589,"5min":1.5660000000000001,"15min":1.5740000000000001},"min":{"1min":1.214,"5min":0.877,"15min":0.877},"max":{"1min":1.905,"5min":1.9790000000000001,"15min":2.1970000000000001},"last":1.9770000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51944,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319896,"statfs":{"total":96636764160,"available":96583573504,"internally_reserved":0,"allocated":2781184,"data_stored":1429264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.0900000000000001,"15min":1.131},"min":{"1min":0.79500000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.8149999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1619999999999999,"15min":1.1579999999999999},"min":{"1min":0.82199999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5940000000000001,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.90900000000000003}]},{"osd":1,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.3720000000000001,"15min":1.3839999999999999},"min":{"1min":0.88,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.873,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.383,"15min":1.4119999999999999},"min":{"1min":1.109,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.77,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.23}]},{"osd":2,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.266,"15min":1.266},"min":{"1min":0.95199999999999996,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.7509999999999999,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.3280000000000001,"15min":1.3500000000000001},"min":{"1min":1.034,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.927,"5min":1.927,"15min":1.944},"last":1.3300000000000001}]},{"osd":3,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3220000000000001,"15min":1.3080000000000001},"min":{"1min":1.002,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8120000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.421,"15min":1.4099999999999999},"min":{"1min":1.1399999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.8169999999999999,"5min":1.929,"15min":2.036},"last":0.96099999999999997}]},{"osd":5,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3600000000000001,"15min":1.3280000000000001},"min":{"1min":0.91900000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.8,"5min":2.0030000000000001,"15min":2.0609999999999999},"last":1.427},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3700000000000001,"15min":1.359},"min":{"1min":0.92000000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7769999999999999,"5min":1.9139999999999999,"15min":2.1659999999999999},"last":1.0209999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.492,"15min":1.446},"min":{"1min":1.01,"5min":0.70899999999999996,"15min":0.61499999999999999},"max":{"1min":2.004,"5min":2.004,"15min":2.097},"last":1.181},{"interface":"front","average":{"1min":1.661,"5min":1.615,"15min":1.601},"min":{"1min":1.296,"5min":0.75,"15min":0.75},"max":{"1min":2.1120000000000001,"5min":2.1150000000000002,"15min":2.23},"last":1.4910000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.5,"15min":1.4950000000000001},"min":{"1min":1.2290000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8680000000000001,"5min":2.0190000000000001,"15min":2.3340000000000001},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.611,"15min":1.5920000000000001},"min":{"1min":1.3759999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1659999999999999,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.28}]}]},{"osd":3,"up_from":13,"seq":55834575994,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56400,"kb_used_data":3012,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315440,"statfs":{"total":96636764160,"available":96579010560,"internally_reserved":0,"allocated":3084288,"data_stored":1739556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.94299999999999995,"15min":0.95599999999999996},"min":{"1min":0.72199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4530000000000001,"5min":2.0760000000000001,"15min":2.0950000000000002},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.135,"15min":1.1339999999999999},"min":{"1min":0.51400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.3879999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.012,"15min":1.0289999999999999},"min":{"1min":0.70799999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.677,"5min":1.677,"15min":1.698},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.214,"15min":1.256},"min":{"1min":0.77600000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.718,"5min":1.956,"15min":2.6019999999999999},"last":1.167}]},{"osd":2,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2629999999999999,"15min":1.3},"min":{"1min":0.86399999999999999,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5529999999999999,"5min":2.105,"15min":2.327},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.369,"15min":1.397},"min":{"1min":0.70999999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.782,"5min":1.9119999999999999,"15min":2.218},"last":1.3740000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.325,"15min":1.379},"min":{"1min":1.0029999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.706,"5min":1.948,"15min":2.0379999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.474,"5min":1.446,"15min":1.482},"min":{"1min":1.0189999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.165},"last":1.2290000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.5329999999999999,"15min":1.55},"min":{"1min":1.331,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.98,"5min":2.2989999999999999,"15min":2.4329999999999998},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.498,"15min":1.5109999999999999},"min":{"1min":1.0760000000000001,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":1.8240000000000001,"5min":2.1389999999999998,"15min":2.2639999999999998},"last":0.96599999999999997}]},{"osd":6,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5580000000000001,"15min":1.5720000000000001},"min":{"1min":1.1990000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.9830000000000001,"5min":2.0640000000000001,"15min":2.2109999999999999},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.637,"15min":1.6659999999999999},"min":{"1min":1.425,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":2.0779999999999998,"5min":2.379,"15min":2.5329999999999999},"last":1.5960000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.629,"15min":1.6359999999999999},"min":{"1min":1.1579999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0310000000000001,"5min":2.2599999999999998,"15min":2.266},"last":1.145},{"interface":"front","average":{"1min":1.738,"5min":1.6919999999999999,"15min":1.694},"min":{"1min":1.3500000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.129,"5min":2.1800000000000002,"15min":2.3029999999999999},"last":1.4379999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575994,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52232,"kb_used_data":3004,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319608,"statfs":{"total":96636764160,"available":96583278592,"internally_reserved":0,"allocated":3076096,"data_stored":1739426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.98699999999999999,"15min":0.97799999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.113,"15min":1.1459999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.0960000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.0049999999999999,"15min":1.0169999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.161,"5min":2.097,"15min":2.097},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.1659999999999999,"15min":1.2210000000000001},"min":{"1min":0.59599999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.331,"5min":1.625,"15min":1.8149999999999999},"last":1.2130000000000001}]},{"osd":3,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.206,"15min":1.25},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.625,"5min":1.72,"15min":1.9330000000000001},"last":1.024},{"interface":"front","average":{"1min":1.123,"5min":1.337,"15min":1.331},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.575,"5min":1.849,"15min":2.2210000000000001},"last":0.65800000000000003}]},{"osd":4,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.4530000000000001,"15min":1.4670000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6799999999999999,"5min":1.956,"15min":2.0640000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4330000000000001,"15min":1.464},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.8480000000000001,"5min":1.8779999999999999,"15min":2.173},"last":4.4619999999999997}]},{"osd":5,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4830000000000001,"15min":1.5049999999999999},"min":{"1min":1.0229999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.792},{"interface":"front","average":{"1min":1.444,"5min":1.552,"15min":1.5700000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.976,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.4339999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.5569999999999999,"15min":1.605},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8029999999999999,"5min":2.387,"15min":2.387},"last":3.1659999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.635,"15min":1.653},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.8620000000000001,"5min":2.1640000000000001,"15min":2.4020000000000001},"last":1.538}]},{"osd":7,"last update":"Tue Nov 28 22:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.589,"15min":1.615},"min":{"1min":1.153,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.702},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.6359999999999999,"15min":1.698},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.8520000000000001,"5min":2.2599999999999998,"15min":2.2690000000000001},"last":3.7789999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52240,"kb_used_data":3012,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319600,"statfs":{"total":96636764160,"available":96583270400,"internally_reserved":0,"allocated":3084288,"data_stored":1746134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92300000000000004,"15min":0.92000000000000004},"min":{"1min":0.55000000000000004,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.417,"5min":1.6240000000000001,"15min":1.8839999999999999},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.153,"15min":1.1579999999999999},"min":{"1min":0.64800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.546,"5min":1.9410000000000001,"15min":2.3900000000000001},"last":1.01}]},{"osd":2,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.147,"15min":1.1599999999999999},"min":{"1min":0.623,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.464,"5min":1.984,"15min":1.9890000000000001},"last":1.038},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2669999999999999,"15min":1.2729999999999999},"min":{"1min":0.58299999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.77,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.363,"15min":1.3999999999999999},"min":{"1min":0.71499999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.722,"5min":2.2040000000000002,"15min":2.2320000000000002},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.79000000000000004,"5min":0.57799999999999996,"15min":0.46800000000000003},"max":{"1min":1.641,"5min":1.786,"15min":2.4359999999999999},"last":0.93600000000000005}]},{"osd":4,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4019999999999999,"15min":1.427},"min":{"1min":1.089,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3559999999999999},"last":1.375},{"interface":"front","average":{"1min":1.399,"5min":1.4099999999999999,"15min":1.4379999999999999},"min":{"1min":0.99199999999999999,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.3079999999999998},"last":2.1680000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4750000000000001,"15min":1.4930000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.77800000000000002},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":2.032},{"interface":"front","average":{"1min":1.544,"5min":1.5880000000000001,"15min":1.6120000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.925,"5min":2.3580000000000001,"15min":2.448},"last":1.7829999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.522,"15min":1.516},"min":{"1min":1.028,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.5249999999999999},"last":2.0800000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.579,"15min":1.593},"min":{"1min":1.175,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.246,"5min":2.246,"15min":2.4550000000000001},"last":1.9179999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.665,"15min":1.6950000000000001},"min":{"1min":1.131,"5min":1.125,"15min":1.125},"max":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.571,"5min":1.5740000000000001,"15min":1.591},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":2.0369999999999999,"5min":2.2730000000000001,"15min":2.8460000000000001},"last":2.2360000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575994,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52516,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319324,"statfs":{"total":96636764160,"available":96582987776,"internally_reserved":0,"allocated":3366912,"data_stored":2009550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.93799999999999994,"15min":0.93200000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.728,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.1439999999999999,"15min":1.1830000000000001},"min":{"1min":0.751,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.2869999999999999},"last":1.272}]},{"osd":2,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1279999999999999,"15min":1.1779999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.1709999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.367,"5min":1.256,"15min":1.244},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":1.458}]},{"osd":3,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.1830000000000001,"15min":1.179},"min":{"1min":0.80100000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.655,"5min":1.8009999999999999,"15min":1.986},"last":1.077},{"interface":"front","average":{"1min":1.393,"5min":1.387,"15min":1.3819999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":1.347}]},{"osd":4,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.484,"15min":1.4750000000000001},"min":{"1min":1.016,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.395,"5min":2.395,"15min":2.395},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.411,"15min":1.4330000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.8109999999999999,"5min":2.149,"15min":2.254},"last":1.679}]},{"osd":5,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.48,"15min":1.4870000000000001},"min":{"1min":0.94999999999999996,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.8959999999999999,"5min":2.0819999999999999,"15min":2.1520000000000001},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.585,"15min":1.5640000000000001},"min":{"1min":1.1970000000000001,"5min":1.05,"15min":1.05},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.3359999999999999},"last":1.633}]},{"osd":6,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.675,"15min":1.639},"min":{"1min":1.202,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.722},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6259999999999999,"15min":1.611},"min":{"1min":0.97399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.756}]},{"osd":7,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.6990000000000001,"15min":1.6859999999999999},"min":{"1min":1.034,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6060000000000001,"15min":1.625},"min":{"1min":1.177,"5min":0.97899999999999998,"15min":0.97899999999999998},"max":{"1min":1.984,"5min":2.4089999999999998,"15min":2.4180000000000001},"last":1.5549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T22:10:22.369 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T22:10:22.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T22:10:22.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.584+0000 7f75af0ad640 1 Processor -- start 2023-11-28T22:10:22.585 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.584+0000 7f75af0ad640 1 -- start start 2023-11-28T22:10:22.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.585+0000 7f75af0ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a815d550 0x7f75a80cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.585+0000 7f75af0ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f75a80d0100 0x7f75a80d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.585+0000 7f75af0ad640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a80d0560 con 0x7f75a80d1ab0 2023-11-28T22:10:22.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.585+0000 7f75af0ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a80d2230 con 0x7f75a80d0100 2023-11-28T22:10:22.587 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.585+0000 7f75af0ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a80d2510 con 0x7f75a815d550 2023-11-28T22:10:22.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.586+0000 7f75ae8ac640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f75a80d1ab0 0x7f75a8070b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36538/0 (socket says 172.21.15.45:36538) 2023-11-28T22:10:22.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.586+0000 7f75ae8ac640 1 -- 172.21.15.45:0/1647457252 learned_addr learned my addr 172.21.15.45:0/1647457252 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:22.588 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.586+0000 7f75ad8aa640 1 --2- 172.21.15.45:0/1647457252 >> v2:172.21.15.45:3300/0 conn(0x7f75a80d0100 0x7f75a80d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.586+0000 7f75ae0ab640 1 --2- 172.21.15.45:0/1647457252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a815d550 0x7f75a80cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.586+0000 7f75ad0a9640 1 -- 172.21.15.45:0/1647457252 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813066124 0 0) 0x7f75a80d0560 con 0x7f75a80d1ab0 2023-11-28T22:10:22.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.587+0000 7f75ad0a9640 1 -- 172.21.15.45:0/1647457252 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7594003520 con 0x7f75a80d1ab0 2023-11-28T22:10:22.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.587+0000 7f75ad8aa640 1 -- 172.21.15.45:0/1647457252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a815d550 msgr2=0x7f75a80cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:22.589 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.587+0000 7f75ad8aa640 1 --2- 172.21.15.45:0/1647457252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a815d550 0x7f75a80cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.587+0000 7f75ad8aa640 1 -- 172.21.15.45:0/1647457252 >> v1:172.21.15.45:6789/0 conn(0x7f75a80d1ab0 legacy=0x7f75a8070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.587+0000 7f75ad8aa640 1 -- 172.21.15.45:0/1647457252 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a80d27f0 con 0x7f75a80d0100 2023-11-28T22:10:22.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.587+0000 7f75ae0ab640 1 --2- 172.21.15.45:0/1647457252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a815d550 0x7f75a80cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:10:22.590 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.588+0000 7f75ad8aa640 1 --2- 172.21.15.45:0/1647457252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 0x7f75a80d1570 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f75980097a0 tx=0x7f759802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8161848bcd060314 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:22.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.588+0000 7f75ad0a9640 1 -- 172.21.15.45:0/1647457252 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759802d040 con 0x7f75a80d0100 2023-11-28T22:10:22.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.588+0000 7f75ad0a9640 1 -- 172.21.15.45:0/1647457252 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7598031810 con 0x7f75a80d0100 2023-11-28T22:10:22.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.588+0000 7f75ad0a9640 1 -- 172.21.15.45:0/1647457252 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598031b80 con 0x7f75a80d0100 2023-11-28T22:10:22.591 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.589+0000 7f75af0ad640 1 -- 172.21.15.45:0/1647457252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 msgr2=0x7f75a80d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.589+0000 7f75af0ad640 1 --2- 172.21.15.45:0/1647457252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 0x7f75a80d1570 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f75980097a0 tx=0x7f759802b8b0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.589+0000 7f75af0ad640 1 -- 172.21.15.45:0/1647457252 shutdown_connections 2023-11-28T22:10:22.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.589+0000 7f75af0ad640 1 --2- 172.21.15.45:0/1647457252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 0x7f75a80d1570 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.589+0000 7f75af0ad640 1 --2- 172.21.15.45:0/1647457252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a815d550 0x7f75a80cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.592 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.589+0000 7f75af0ad640 1 -- 172.21.15.45:0/1647457252 >> 172.21.15.45:0/1647457252 conn(0x7f75a806ac70 msgr2=0x7f75a80b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:22.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.589+0000 7f75af0ad640 1 -- 172.21.15.45:0/1647457252 shutdown_connections 2023-11-28T22:10:22.593 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.589+0000 7f75af0ad640 1 -- 172.21.15.45:0/1647457252 wait complete. 2023-11-28T22:10:22.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.591+0000 7f75af0ad640 1 Processor -- start 2023-11-28T22:10:22.595 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.591+0000 7f75af0ad640 1 -- start start 2023-11-28T22:10:22.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.592+0000 7f75af0ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 0x7f75a80c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.592+0000 7f75af0ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a80d1ab0 0x7f75a80cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.592+0000 7f75af0ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75a815d550 0x7f75a81f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.592+0000 7f75af0ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a80c9700 con 0x7f75a80d0100 2023-11-28T22:10:22.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.592+0000 7f75af0ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a8154da0 con 0x7f75a815d550 2023-11-28T22:10:22.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.592+0000 7f75af0ad640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a8154a70 con 0x7f75a80d1ab0 2023-11-28T22:10:22.596 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.592+0000 7f75ae0ab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 0x7f75a80c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.592+0000 7f75ae0ab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 0x7f75a80c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34746/0 (socket says 172.21.15.45:34746) 2023-11-28T22:10:22.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.592+0000 7f75ae0ab640 1 -- 172.21.15.45:0/833965625 learned_addr learned my addr 172.21.15.45:0/833965625 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:22.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75ae0ab640 1 -- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a80d1ab0 msgr2=0x7f75a80cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75ae8ac640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75a815d550 0x7f75a81f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75ad8aa640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a80d1ab0 0x7f75a80cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75ae0ab640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a80d1ab0 0x7f75a80cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75ae0ab640 1 -- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75a815d550 msgr2=0x7f75a81f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75ae0ab640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75a815d550 0x7f75a81f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75ae0ab640 1 -- 172.21.15.45:0/833965625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7598009400 con 0x7f75a80d0100 2023-11-28T22:10:22.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75ae8ac640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75a815d550 0x7f75a81f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:22.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75ae0ab640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 0x7f75a80c8760 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f75a400d7d0 tx=0x7f75a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:22.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f759effd640 1 -- 172.21.15.45:0/833965625 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4004030 con 0x7f75a80d0100 2023-11-28T22:10:22.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f759effd640 1 -- 172.21.15.45:0/833965625 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a4004190 con 0x7f75a80d0100 2023-11-28T22:10:22.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f759effd640 1 -- 172.21.15.45:0/833965625 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4004520 con 0x7f75a80d0100 2023-11-28T22:10:22.598 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.593+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a80c8e60 con 0x7f75a80d0100 2023-11-28T22:10:22.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.594+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a80c9420 con 0x7f75a80d0100 2023-11-28T22:10:22.599 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.595+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a8067610 con 0x7f75a80d0100 2023-11-28T22:10:22.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.596+0000 7f759effd640 1 -- 172.21.15.45:0/833965625 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f75a40096a0 con 0x7f75a80d0100 2023-11-28T22:10:22.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.597+0000 7f759effd640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f758c06af60 0x7f758c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:22.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.597+0000 7f759effd640 1 -- 172.21.15.45:0/833965625 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f75a4089ab0 con 0x7f75a80d0100 2023-11-28T22:10:22.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.598+0000 7f75ad8aa640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f758c06af60 0x7f758c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:22.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.599+0000 7f75ad8aa640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f758c06af60 0x7f758c06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7598036000 tx=0x7f759803c040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:22.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.601+0000 7f759effd640 1 -- 172.21.15.45:0/833965625 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75a40548e0 con 0x7f75a80d0100 2023-11-28T22:10:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.796+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f75a80b5df0 con 0x7f758c06af60 2023-11-28T22:10:22.804 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.803+0000 7f759effd640 1 -- 172.21.15.45:0/833965625 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47973 (secure 0 0 0) 0x7f75a80b5df0 con 0x7f758c06af60 2023-11-28T22:10:22.805 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:22.805 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T22:10:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.805+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f758c06af60 msgr2=0x7f758c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.805+0000 7f75af0ad640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f758c06af60 0x7f758c06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7598036000 tx=0x7f759803c040 comp rx=0 tx=0).stop 2023-11-28T22:10:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.805+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 msgr2=0x7f75a80c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.805+0000 7f75af0ad640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 0x7f75a80c8760 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f75a400d7d0 tx=0x7f75a400dc90 comp rx=0 tx=0).stop 2023-11-28T22:10:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.806+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 shutdown_connections 2023-11-28T22:10:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.806+0000 7f75af0ad640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f758c06af60 0x7f758c06d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.806+0000 7f75af0ad640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75a815d550 0x7f75a81f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.806+0000 7f75af0ad640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f75a80d1ab0 0x7f75a80cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.806+0000 7f75af0ad640 1 --2- 172.21.15.45:0/833965625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a80d0100 0x7f75a80c8760 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:22.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.806+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 >> 172.21.15.45:0/833965625 conn(0x7f75a806ac70 msgr2=0x7f75a80b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:22.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.806+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 shutdown_connections 2023-11-28T22:10:22.809 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:22.806+0000 7f75af0ad640 1 -- 172.21.15.45:0/833965625 wait complete. 2023-11-28T22:10:22.825 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-11-28T22:10:22.713024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4855,"ondisk_log_size":4855,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":441756,"kb_used_data":22952,"kb_used_omap":12,"kb_used_meta":418675,"kb_avail":754532964,"statfs":{"total":773094113280,"available":772641755136,"internally_reserved":0,"allocated":23502848,"data_stored":12639968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":428723760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667486,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-220,"num_read_kb":-74,"num_write":-251,"num_write_kb":-1666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.944231"},"pg_stats":[{"pgid":"1.0","version":"219'4855","reported_seq":5780,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-11-28T22:10:17.209528+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T22:10:17.209528+0000","last_peered":"2023-11-28T22:10:17.209528+0000","last_clean":"2023-11-28T22:10:17.209528+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T22:10:17.209528+0000","last_undegraded":"2023-11-28T22:10:17.209528+0000","last_fullsized":"2023-11-28T22:10:17.209528+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4855,"ondisk_log_size":4855,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575994,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74740,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297100,"statfs":{"total":96636764160,"available":96560230400,"internally_reserved":0,"allocated":3383296,"data_stored":2022772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2410000000000001,"15min":1.214},"min":{"1min":0.83799999999999997,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6499999999999999,"5min":1.9019999999999999,"15min":1.9330000000000001},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.218,"5min":1.2150000000000001,"15min":1.26},"min":{"1min":0.75800000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5720000000000001,"5min":2.0190000000000001,"15min":2.6819999999999999},"last":1.105}]},{"osd":1,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.248,"15min":1.2569999999999999},"min":{"1min":0.88600000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6419999999999999,"5min":2.694,"15min":2.694},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.371,"5min":1.2749999999999999,"15min":1.29},"min":{"1min":0.93799999999999994,"5min":0.84199999999999997,"15min":0.80900000000000005},"max":{"1min":1.722,"5min":1.756,"15min":3.0430000000000001},"last":1.0169999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2809999999999999,"15min":1.3180000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.643,"5min":1.754,"15min":2.7480000000000002},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.371,"15min":1.3999999999999999},"min":{"1min":0.93700000000000006,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.8009999999999999,"5min":1.833,"15min":2.2400000000000002},"last":0.90300000000000002}]},{"osd":3,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3240000000000001,"15min":1.345},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.7729999999999999,"5min":1.7969999999999999,"15min":1.7989999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.444,"15min":1.464},"min":{"1min":1.048,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.784,"5min":2.258,"15min":2.9329999999999998},"last":1.0880000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2929999999999999,"15min":1.3280000000000001},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.81299999999999994},"max":{"1min":1.7969999999999999,"5min":1.8100000000000001,"15min":2.6930000000000001},"last":0.621},{"interface":"front","average":{"1min":1.323,"5min":1.333,"15min":1.365},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8580000000000001,"5min":1.905,"15min":2.371},"last":0.77200000000000002}]},{"osd":5,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.77500000000000002},"max":{"1min":1.728,"5min":1.9750000000000001,"15min":3.2229999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.468,"5min":1.48,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.992,"5min":2.0760000000000001,"15min":2.508},"last":0.83099999999999996}]},{"osd":6,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.579,"15min":1.613},"min":{"1min":1.1739999999999999,"5min":1.038,"15min":1.038},"max":{"1min":2.0329999999999999,"5min":2.0720000000000001,"15min":2.4409999999999998},"last":0.996},{"interface":"front","average":{"1min":1.444,"5min":1.5089999999999999,"15min":1.532},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.8500000000000001,"5min":2.2010000000000001,"15min":2.2869999999999999},"last":0.88200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":42228,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1259999999999999,"15min":1.129},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.73699999999999999},"max":{"1min":1.4299999999999999,"5min":1.6679999999999999,"15min":1.772},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.137,"5min":1.0940000000000001,"15min":1.1299999999999999},"min":{"1min":0.872,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.718,"5min":1.718,"15min":1.7509999999999999},"last":1.714}]},{"osd":1,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.292,"15min":1.298},"min":{"1min":1.0509999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.667,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4239999999999999,"15min":1.4350000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.89300000000000002},"max":{"1min":1.6619999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.9019999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3440000000000001,"15min":1.3420000000000001},"min":{"1min":0.99199999999999999,"5min":0.89900000000000002,"15min":0.80000000000000004},"max":{"1min":1.579,"5min":2.0779999999999998,"15min":2.4809999999999999},"last":1},{"interface":"front","average":{"1min":1.448,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.145,"5min":0.871,"15min":0.85199999999999998},"max":{"1min":1.8009999999999999,"5min":2.1829999999999998,"15min":2.6269999999999998},"last":1.1839999999999999}]},{"osd":3,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.387,"15min":1.361},"min":{"1min":1.018,"5min":0.92000000000000004,"15min":0.88500000000000001},"max":{"1min":1.7929999999999999,"5min":1.827,"15min":1.9330000000000001},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3120000000000001,"15min":1.331},"min":{"1min":1.008,"5min":0.94599999999999995,"15min":0.86099999999999999},"max":{"1min":1.6359999999999999,"5min":1.7190000000000001,"15min":1.8859999999999999},"last":0.875}]},{"osd":4,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.327,"15min":1.304},"min":{"1min":0.86399999999999999,"5min":0.81599999999999995,"15min":0.81000000000000005},"max":{"1min":1.9219999999999999,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.3879999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.86199999999999999},"max":{"1min":1.917,"5min":1.917,"15min":2.02},"last":1.4970000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.409,"15min":1.3999999999999999},"min":{"1min":0.97299999999999998,"5min":0.94999999999999996,"15min":0.84199999999999997},"max":{"1min":1.772,"5min":1.9990000000000001,"15min":2.081},"last":1.149},{"interface":"front","average":{"1min":1.619,"5min":1.5980000000000001,"15min":1.5960000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0270000000000001,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":1.3380000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.544,"15min":1.5369999999999999},"min":{"1min":1.077,"5min":1.0649999999999999,"15min":0.98899999999999999},"max":{"1min":1.9870000000000001,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.8089999999999999},{"interface":"front","average":{"1min":1.573,"5min":1.504,"15min":1.526},"min":{"1min":1.252,"5min":1.0029999999999999,"15min":0.879},"max":{"1min":2.024,"5min":2.024,"15min":2.2130000000000001},"last":1.2809999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60272,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311568,"statfs":{"total":96636764160,"available":96575045632,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1539999999999999,"15min":1.173},"min":{"1min":0.71699999999999997,"5min":0.65700000000000003,"15min":0.48499999999999999},"max":{"1min":1.6120000000000001,"5min":1.9039999999999999,"15min":1.9550000000000001},"last":0.747},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2170000000000001,"15min":1.2},"min":{"1min":0.89600000000000002,"5min":0.80600000000000005,"15min":0.65700000000000003},"max":{"1min":1.8069999999999999,"5min":1.889,"15min":1.889},"last":1.0529999999999999}]},{"osd":1,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1990000000000001,"15min":1.212},"min":{"1min":0.77200000000000002,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.494,"5min":1.706,"15min":1.8979999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2789999999999999,"15min":1.286},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.54400000000000004},"max":{"1min":1.7370000000000001,"5min":1.7509999999999999,"15min":1.8859999999999999},"last":0.88300000000000001}]},{"osd":2,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.3100000000000001,"15min":1.3260000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.60699999999999998},"max":{"1min":1.677,"5min":1.768,"15min":1.867},"last":1.006},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.361,"15min":1.3560000000000001},"min":{"1min":0.81799999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.7410000000000001,"5min":1.8899999999999999,"15min":2.044},"last":1.165}]},{"osd":3,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4490000000000001,"15min":1.4139999999999999},"min":{"1min":1.1599999999999999,"5min":0.96999999999999997,"15min":0.68899999999999995},"max":{"1min":1.5720000000000001,"5min":1.8180000000000001,"15min":1.988},"last":1.304},{"interface":"front","average":{"1min":1.341,"5min":1.3069999999999999,"15min":1.3380000000000001},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.87},"max":{"1min":1.595,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.95699999999999996}]},{"osd":4,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3720000000000001,"15min":1.377},"min":{"1min":0.873,"5min":0.84899999999999998,"15min":0.72599999999999998},"max":{"1min":1.802,"5min":1.833,"15min":1.833},"last":1.333},{"interface":"front","average":{"1min":1.48,"5min":1.4550000000000001,"15min":1.494},"min":{"1min":1.103,"5min":0.83999999999999997,"15min":0.76200000000000001},"max":{"1min":1.792,"5min":1.8440000000000001,"15min":2.0249999999999999},"last":1.236}]},{"osd":6,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4379999999999999,"15min":1.415},"min":{"1min":0.88800000000000001,"5min":0.84099999999999997,"15min":0.80000000000000004},"max":{"1min":1.885,"5min":1.899,"15min":2.0960000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.5069999999999999,"15min":1.544},"min":{"1min":1.1619999999999999,"5min":0.97199999999999998,"15min":0.94399999999999995},"max":{"1min":1.7849999999999999,"5min":1.982,"15min":1.982},"last":1.4610000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.46,"15min":1.484},"min":{"1min":1.069,"5min":1.0580000000000001,"15min":0.76100000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.1880000000000002},"last":1.123},{"interface":"front","average":{"1min":1.589,"5min":1.5660000000000001,"15min":1.5740000000000001},"min":{"1min":1.214,"5min":0.877,"15min":0.877},"max":{"1min":1.905,"5min":1.9790000000000001,"15min":2.1970000000000001},"last":1.405}]}]},{"osd":4,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51952,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319888,"statfs":{"total":96636764160,"available":96583565312,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.0900000000000001,"15min":1.131},"min":{"1min":0.79500000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.8149999999999999},"last":2.109},{"interface":"front","average":{"1min":1.173,"5min":1.1619999999999999,"15min":1.1579999999999999},"min":{"1min":0.82199999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5940000000000001,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.3300000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.3720000000000001,"15min":1.3839999999999999},"min":{"1min":0.88,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.873,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.45},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.383,"15min":1.4119999999999999},"min":{"1min":1.109,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.77,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":2.2349999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.266,"15min":1.266},"min":{"1min":0.95199999999999996,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.7509999999999999,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.242},{"interface":"front","average":{"1min":1.353,"5min":1.3280000000000001,"15min":1.3500000000000001},"min":{"1min":1.034,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.927,"5min":1.927,"15min":1.944},"last":2.335}]},{"osd":3,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3220000000000001,"15min":1.3080000000000001},"min":{"1min":1.002,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8120000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.421,"15min":1.4099999999999999},"min":{"1min":1.1399999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.8169999999999999,"5min":1.929,"15min":2.036},"last":1.859}]},{"osd":5,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3600000000000001,"15min":1.3280000000000001},"min":{"1min":0.91900000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.8,"5min":2.0030000000000001,"15min":2.0609999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3700000000000001,"15min":1.359},"min":{"1min":0.92000000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7769999999999999,"5min":1.9139999999999999,"15min":2.1659999999999999},"last":0.96699999999999997}]},{"osd":6,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.492,"15min":1.446},"min":{"1min":1.01,"5min":0.70899999999999996,"15min":0.61499999999999999},"max":{"1min":2.004,"5min":2.004,"15min":2.097},"last":2.0489999999999999},{"interface":"front","average":{"1min":1.661,"5min":1.615,"15min":1.601},"min":{"1min":1.296,"5min":0.75,"15min":0.75},"max":{"1min":2.1120000000000001,"5min":2.1150000000000002,"15min":2.23},"last":1.6899999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.5,"15min":1.4950000000000001},"min":{"1min":1.2290000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8680000000000001,"5min":2.0190000000000001,"15min":2.3340000000000001},"last":1.583},{"interface":"front","average":{"1min":1.681,"5min":1.611,"15min":1.5920000000000001},"min":{"1min":1.3759999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1659999999999999,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.9490000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56120,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315720,"statfs":{"total":96636764160,"available":96579297280,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.94299999999999995,"15min":0.95599999999999996},"min":{"1min":0.72199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4530000000000001,"5min":2.0760000000000001,"15min":2.0950000000000002},"last":1.177},{"interface":"front","average":{"1min":1.163,"5min":1.135,"15min":1.1339999999999999},"min":{"1min":0.51400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.3879999999999999},"last":0.92300000000000004}]},{"osd":1,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.012,"15min":1.0289999999999999},"min":{"1min":0.70799999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.677,"5min":1.677,"15min":1.698},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.214,"15min":1.256},"min":{"1min":0.77600000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.718,"5min":1.956,"15min":2.6019999999999999},"last":0.81999999999999995}]},{"osd":2,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2629999999999999,"15min":1.3},"min":{"1min":0.86399999999999999,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5529999999999999,"5min":2.105,"15min":2.327},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.369,"15min":1.397},"min":{"1min":0.70999999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.782,"5min":1.9119999999999999,"15min":2.218},"last":1.7390000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.325,"15min":1.379},"min":{"1min":1.0029999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.706,"5min":1.948,"15min":2.0379999999999998},"last":1.123},{"interface":"front","average":{"1min":1.474,"5min":1.446,"15min":1.482},"min":{"1min":1.0189999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.165},"last":1.5740000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.5329999999999999,"15min":1.55},"min":{"1min":1.331,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.98,"5min":2.2989999999999999,"15min":2.4329999999999998},"last":1.292},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.498,"15min":1.5109999999999999},"min":{"1min":1.0760000000000001,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":1.8240000000000001,"5min":2.1389999999999998,"15min":2.2639999999999998},"last":1.619}]},{"osd":6,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5580000000000001,"15min":1.5720000000000001},"min":{"1min":1.1990000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.9830000000000001,"5min":2.0640000000000001,"15min":2.2109999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.637,"15min":1.6659999999999999},"min":{"1min":1.425,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":2.0779999999999998,"5min":2.379,"15min":2.5329999999999999},"last":1.5249999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.629,"15min":1.6359999999999999},"min":{"1min":1.1579999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0310000000000001,"5min":2.2599999999999998,"15min":2.266},"last":1.839},{"interface":"front","average":{"1min":1.738,"5min":1.6919999999999999,"15min":1.694},"min":{"1min":1.3500000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.129,"5min":2.1800000000000002,"15min":2.3029999999999999},"last":1.8029999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51956,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319884,"statfs":{"total":96636764160,"available":96583561216,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.96699999999999997,"15min":0.97399999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2350000000000001,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.095,"15min":1.141},"min":{"1min":0.496,"5min":0.495,"15min":0.495},"max":{"1min":1.6399999999999999,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.496}]},{"osd":1,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.98799999999999999,"15min":1.0069999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.3400000000000001,"5min":2.097,"15min":2.097},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.157,"5min":1.153,"15min":1.2190000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.5249999999999999,"5min":1.625,"15min":1.8149999999999999},"last":0.46200000000000002}]},{"osd":3,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1919999999999999,"15min":1.24},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.925,"5min":1.925,"15min":1.9330000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.3069999999999999,"15min":1.3149999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.597,"5min":1.849,"15min":2.2210000000000001},"last":0.52500000000000002}]},{"osd":4,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.425,"15min":1.4610000000000001},"min":{"1min":0.96799999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.835,"5min":1.956,"15min":2.0640000000000001},"last":1.024},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4570000000000001,"15min":1.47},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":4.4619999999999997,"5min":4.4619999999999997,"15min":4.4619999999999997},"last":1.115}]},{"osd":5,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.49,"15min":1.5049999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.877,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.403,"5min":1.5269999999999999,"15min":1.5589999999999999},"min":{"1min":0.80000000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.7669999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.80000000000000004}]},{"osd":6,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.552,"15min":1.599},"min":{"1min":0.84299999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.6220000000000001,"15min":1.645},"min":{"1min":0.82899999999999996,"5min":0.751,"15min":0.751},"max":{"1min":1.9610000000000001,"5min":2.1640000000000001,"15min":2.4020000000000001},"last":0.82899999999999996}]},{"osd":7,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.5860000000000001,"15min":1.6100000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.9650000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.089},{"interface":"front","average":{"1min":1.651,"5min":1.6359999999999999,"15min":1.6930000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":3.7789999999999999,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":0.78300000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575995,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51952,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319888,"statfs":{"total":96636764160,"available":96583565312,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92300000000000004,"15min":0.92000000000000004},"min":{"1min":0.55000000000000004,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.417,"5min":1.6240000000000001,"15min":1.8839999999999999},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.153,"15min":1.1579999999999999},"min":{"1min":0.64800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.546,"5min":1.9410000000000001,"15min":2.3900000000000001},"last":1.01}]},{"osd":2,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.147,"15min":1.1599999999999999},"min":{"1min":0.623,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.464,"5min":1.984,"15min":1.9890000000000001},"last":1.038},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2669999999999999,"15min":1.2729999999999999},"min":{"1min":0.58299999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.77,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.363,"15min":1.3999999999999999},"min":{"1min":0.71499999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.722,"5min":2.2040000000000002,"15min":2.2320000000000002},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.79000000000000004,"5min":0.57799999999999996,"15min":0.46800000000000003},"max":{"1min":1.641,"5min":1.786,"15min":2.4359999999999999},"last":0.93600000000000005}]},{"osd":4,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4019999999999999,"15min":1.427},"min":{"1min":1.089,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3559999999999999},"last":1.375},{"interface":"front","average":{"1min":1.399,"5min":1.4099999999999999,"15min":1.4379999999999999},"min":{"1min":0.99199999999999999,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.3079999999999998},"last":2.1680000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4750000000000001,"15min":1.4930000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.77800000000000002},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":2.032},{"interface":"front","average":{"1min":1.544,"5min":1.5880000000000001,"15min":1.6120000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.925,"5min":2.3580000000000001,"15min":2.448},"last":1.7829999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.522,"15min":1.516},"min":{"1min":1.028,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.5249999999999999},"last":2.0800000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.579,"15min":1.593},"min":{"1min":1.175,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.246,"5min":2.246,"15min":2.4550000000000001},"last":1.9179999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.665,"15min":1.6950000000000001},"min":{"1min":1.131,"5min":1.125,"15min":1.125},"max":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.571,"5min":1.5740000000000001,"15min":1.591},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":2.0369999999999999,"5min":2.2730000000000001,"15min":2.8460000000000001},"last":2.2360000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52536,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319304,"statfs":{"total":96636764160,"available":96582967296,"internally_reserved":0,"allocated":3383296,"data_stored":2022772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.93799999999999994,"15min":0.93200000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.728,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.1439999999999999,"15min":1.1830000000000001},"min":{"1min":0.751,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.2869999999999999},"last":0.96399999999999997}]},{"osd":2,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1279999999999999,"15min":1.1779999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.1709999999999998},"last":1.016},{"interface":"front","average":{"1min":1.367,"5min":1.256,"15min":1.244},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":1.3460000000000001}]},{"osd":3,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.1830000000000001,"15min":1.179},"min":{"1min":0.80100000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.655,"5min":1.8009999999999999,"15min":1.986},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.387,"15min":1.3819999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":1.2969999999999999}]},{"osd":4,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.484,"15min":1.4750000000000001},"min":{"1min":1.016,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.395,"5min":2.395,"15min":2.395},"last":1.706},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.411,"15min":1.4330000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.8109999999999999,"5min":2.149,"15min":2.254},"last":1.5009999999999999}]},{"osd":5,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.48,"15min":1.4870000000000001},"min":{"1min":0.94999999999999996,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.8959999999999999,"5min":2.0819999999999999,"15min":2.1520000000000001},"last":1.405},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.585,"15min":1.5640000000000001},"min":{"1min":1.1970000000000001,"5min":1.05,"15min":1.05},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.3359999999999999},"last":1.6499999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.675,"15min":1.639},"min":{"1min":1.202,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.78},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6259999999999999,"15min":1.611},"min":{"1min":0.97399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.927}]},{"osd":7,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.6990000000000001,"15min":1.6859999999999999},"min":{"1min":1.034,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":1.8560000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6060000000000001,"15min":1.625},"min":{"1min":1.177,"5min":0.97899999999999998,"15min":0.97899999999999998},"max":{"1min":1.984,"5min":2.4089999999999998,"15min":2.4180000000000001},"last":1.581}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T22:10:22.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T22:10:23.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.022+0000 7fde265fb640 1 Processor -- start 2023-11-28T22:10:23.023 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.022+0000 7fde265fb640 1 -- start start 2023-11-28T22:10:23.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.023+0000 7fde265fb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fde200d02d0 0x7fde200d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:23.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.023+0000 7fde265fb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde200d1ab0 0x7fde200b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:23.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.023+0000 7fde265fb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde200d2230 con 0x7fde2015d550 2023-11-28T22:10:23.025 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.023+0000 7fde265fb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde200d2510 con 0x7fde200d02d0 2023-11-28T22:10:23.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.023+0000 7fde265fb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde200d27f0 con 0x7fde200d1ab0 2023-11-28T22:10:23.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.024+0000 7fde24df8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fde200d02d0 0x7fde200d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:23.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.024+0000 7fde24df8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fde200d02d0 0x7fde200d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34756/0 (socket says 172.21.15.45:34756) 2023-11-28T22:10:23.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.024+0000 7fde24df8640 1 -- 172.21.15.45:0/426666768 learned_addr learned my addr 172.21.15.45:0/426666768 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:23.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.024+0000 7fde25dfa640 1 --2- 172.21.15.45:0/426666768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde200d1ab0 0x7fde200b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:23.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.024+0000 7fde24df8640 1 -- 172.21.15.45:0/426666768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde200d1ab0 msgr2=0x7fde200b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:23.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.024+0000 7fde24df8640 1 --2- 172.21.15.45:0/426666768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde200d1ab0 0x7fde200b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.024+0000 7fde24df8640 1 -- 172.21.15.45:0/426666768 >> v1:172.21.15.190:6789/0 conn(0x7fde2015d550 legacy=0x7fde200cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:23.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.024+0000 7fde24df8640 1 -- 172.21.15.45:0/426666768 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde200b6a30 con 0x7fde200d02d0 2023-11-28T22:10:23.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.026+0000 7fde24df8640 1 --2- 172.21.15.45:0/426666768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 0x7fde200d1570 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fde10009870 tx=0x7fde1002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d746e2a00078cf8f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:23.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.026+0000 7fde17fff640 1 -- 172.21.15.45:0/426666768 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10031640 con 0x7fde200d02d0 2023-11-28T22:10:23.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.026+0000 7fde17fff640 1 -- 172.21.15.45:0/426666768 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde100317a0 con 0x7fde200d02d0 2023-11-28T22:10:23.028 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.026+0000 7fde17fff640 1 -- 172.21.15.45:0/426666768 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10031b30 con 0x7fde200d02d0 2023-11-28T22:10:23.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.027+0000 7fde265fb640 1 -- 172.21.15.45:0/426666768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 msgr2=0x7fde200d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:23.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.027+0000 7fde265fb640 1 --2- 172.21.15.45:0/426666768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 0x7fde200d1570 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fde10009870 tx=0x7fde1002b8f0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.027+0000 7fde265fb640 1 -- 172.21.15.45:0/426666768 shutdown_connections 2023-11-28T22:10:23.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.027+0000 7fde265fb640 1 --2- 172.21.15.45:0/426666768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde200d1ab0 0x7fde200b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.027+0000 7fde265fb640 1 --2- 172.21.15.45:0/426666768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 0x7fde200d1570 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.027+0000 7fde265fb640 1 -- 172.21.15.45:0/426666768 >> 172.21.15.45:0/426666768 conn(0x7fde2006ac70 msgr2=0x7fde200b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:23.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.027+0000 7fde265fb640 1 -- 172.21.15.45:0/426666768 shutdown_connections 2023-11-28T22:10:23.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.027+0000 7fde265fb640 1 -- 172.21.15.45:0/426666768 wait complete. 2023-11-28T22:10:23.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.029+0000 7fde265fb640 1 Processor -- start 2023-11-28T22:10:23.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.029+0000 7fde265fb640 1 -- start start 2023-11-28T22:10:23.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde265fb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 0x7fde201e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:23.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde265fb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde200d1ab0 0x7fde20158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:23.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde265fb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2015d550 0x7fde20159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:23.031 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde265fb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde201e64d0 con 0x7fde200d02d0 2023-11-28T22:10:23.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde265fb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde200ca2a0 con 0x7fde2015d550 2023-11-28T22:10:23.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde265fb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde200ca580 con 0x7fde200d1ab0 2023-11-28T22:10:23.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde255f9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 0x7fde201e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:23.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde255f9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 0x7fde201e5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34758/0 (socket says 172.21.15.45:34758) 2023-11-28T22:10:23.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde255f9640 1 -- 172.21.15.45:0/2418236209 learned_addr learned my addr 172.21.15.45:0/2418236209 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:23.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde25dfa640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2015d550 0x7fde20159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:23.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde255f9640 1 -- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde200d1ab0 msgr2=0x7fde20158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:23.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde255f9640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde200d1ab0 0x7fde20158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde255f9640 1 -- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2015d550 msgr2=0x7fde20159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:23.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde255f9640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2015d550 0x7fde20159bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.030+0000 7fde255f9640 1 -- 172.21.15.45:0/2418236209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde100094d0 con 0x7fde200d02d0 2023-11-28T22:10:23.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.031+0000 7fde255f9640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 0x7fde201e5f90 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00b3f0 tx=0x7fde1c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:23.034 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.031+0000 7fde15ffb640 1 -- 172.21.15.45:0/2418236209 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c011830 con 0x7fde200d02d0 2023-11-28T22:10:23.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.031+0000 7fde15ffb640 1 -- 172.21.15.45:0/2418236209 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1c011990 con 0x7fde200d02d0 2023-11-28T22:10:23.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.031+0000 7fde15ffb640 1 -- 172.21.15.45:0/2418236209 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c011d40 con 0x7fde200d02d0 2023-11-28T22:10:23.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.031+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde200ca860 con 0x7fde200d02d0 2023-11-28T22:10:23.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.031+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde200cae20 con 0x7fde200d02d0 2023-11-28T22:10:23.035 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.032+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2004e920 con 0x7fde200d02d0 2023-11-28T22:10:23.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.033+0000 7fde15ffb640 1 -- 172.21.15.45:0/2418236209 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fde1c01a400 con 0x7fde200d02d0 2023-11-28T22:10:23.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.034+0000 7fde15ffb640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fde0406af60 0x7fde0406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:23.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.034+0000 7fde15ffb640 1 -- 172.21.15.45:0/2418236209 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fde1c089750 con 0x7fde200d02d0 2023-11-28T22:10:23.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.035+0000 7fde24df8640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fde0406af60 0x7fde0406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:23.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.036+0000 7fde24df8640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fde0406af60 0x7fde0406d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde10036000 tx=0x7fde10030000 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:23.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.039+0000 7fde15ffb640 1 -- 172.21.15.45:0/2418236209 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde1c054570 con 0x7fde200d02d0 2023-11-28T22:10:23.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.232+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde200cffb0 con 0x7fde200d02d0 2023-11-28T22:10:23.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.233+0000 7fde15ffb640 1 -- 172.21.15.45:0/2418236209 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+14376 (secure 0 0 0) 0x7fde1c057b90 con 0x7fde200d02d0 2023-11-28T22:10:23.244 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:23.244 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":222,"fsid":"ef8eac8d-57a5-4d94-ab39-848e2a7bfeef","created":"2023-11-28T20:33:33.919032+0000","modified":"2023-11-28T22:10:17.772910+0000","last_up_change":"2023-11-28T20:33:45.955837+0000","last_in_change":"2023-11-28T20:33:37.443264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T20:33:47.196691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83d93ee1-0e46-4bac-b972-2e148e343c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6817","nonce":2634303336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6819","nonce":2634303336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6823","nonce":2634303336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2634303336},{"type":"v1","addr":"172.21.15.45:6821","nonce":2634303336}]},"public_addr":"172.21.15.45:6817/2634303336","cluster_addr":"172.21.15.45:6819/2634303336","heartbeat_back_addr":"172.21.15.45:6823/2634303336","heartbeat_front_addr":"172.21.15.45:6821/2634303336","state":["exists","up"]},{"osd":1,"uuid":"4eaa39fb-6b40-4a4b-ad83-0a3427a81b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6801","nonce":901684499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6803","nonce":901684499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6807","nonce":901684499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":901684499},{"type":"v1","addr":"172.21.15.45:6805","nonce":901684499}]},"public_addr":"172.21.15.45:6801/901684499","cluster_addr":"172.21.15.45:6803/901684499","heartbeat_back_addr":"172.21.15.45:6807/901684499","heartbeat_front_addr":"172.21.15.45:6805/901684499","state":["exists","up"]},{"osd":2,"uuid":"92ec50eb-f647-4f17-9fd4-7437537ce096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6825","nonce":539709697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6827","nonce":539709697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6831","nonce":539709697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":539709697},{"type":"v1","addr":"172.21.15.45:6829","nonce":539709697}]},"public_addr":"172.21.15.45:6825/539709697","cluster_addr":"172.21.15.45:6827/539709697","heartbeat_back_addr":"172.21.15.45:6831/539709697","heartbeat_front_addr":"172.21.15.45:6829/539709697","state":["exists","up"]},{"osd":3,"uuid":"52955585-272d-45b1-839d-e89bad4ed76a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6809","nonce":1024370250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6811","nonce":1024370250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6815","nonce":1024370250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1024370250},{"type":"v1","addr":"172.21.15.45:6813","nonce":1024370250}]},"public_addr":"172.21.15.45:6809/1024370250","cluster_addr":"172.21.15.45:6811/1024370250","heartbeat_back_addr":"172.21.15.45:6815/1024370250","heartbeat_front_addr":"172.21.15.45:6813/1024370250","state":["exists","up"]},{"osd":4,"uuid":"429ee08c-9684-47e8-afab-16671b7045fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6801","nonce":1794504124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6803","nonce":1794504124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6807","nonce":1794504124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1794504124},{"type":"v1","addr":"172.21.15.190:6805","nonce":1794504124}]},"public_addr":"172.21.15.190:6801/1794504124","cluster_addr":"172.21.15.190:6803/1794504124","heartbeat_back_addr":"172.21.15.190:6807/1794504124","heartbeat_front_addr":"172.21.15.190:6805/1794504124","state":["exists","up"]},{"osd":5,"uuid":"65bce9bc-01b3-40a1-aa05-9a0ad17e5de2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6809","nonce":3522971559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6811","nonce":3522971559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6815","nonce":3522971559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3522971559},{"type":"v1","addr":"172.21.15.190:6813","nonce":3522971559}]},"public_addr":"172.21.15.190:6809/3522971559","cluster_addr":"172.21.15.190:6811/3522971559","heartbeat_back_addr":"172.21.15.190:6815/3522971559","heartbeat_front_addr":"172.21.15.190:6813/3522971559","state":["exists","up"]},{"osd":6,"uuid":"d414aa0f-ad7f-44dd-9072-6d3f07d1e44d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6822","nonce":2089844775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6827","nonce":2089844775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6831","nonce":2089844775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2089844775},{"type":"v1","addr":"172.21.15.190:6829","nonce":2089844775}]},"public_addr":"172.21.15.190:6822/2089844775","cluster_addr":"172.21.15.190:6827/2089844775","heartbeat_back_addr":"172.21.15.190:6831/2089844775","heartbeat_front_addr":"172.21.15.190:6829/2089844775","state":["exists","up"]},{"osd":7,"uuid":"5cfcd895-a0df-4b21-b568-5aa243d732e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6817","nonce":3686540929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6819","nonce":3686540929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6825","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6826","nonce":3686540929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3686540929},{"type":"v1","addr":"172.21.15.190:6823","nonce":3686540929}]},"public_addr":"172.21.15.190:6817/3686540929","cluster_addr":"172.21.15.190:6819/3686540929","heartbeat_back_addr":"172.21.15.190:6826/3686540929","heartbeat_front_addr":"172.21.15.190:6823/3686540929","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.784675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.953296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.792745+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.786939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.569906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.721856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.809289+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T20:33:42.811209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6832/3094566588":"2023-11-29T22:10:14.761253+0000","172.21.15.190:0/3465043952":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/1255156042":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/964161599":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/1168814133":"2023-11-29T22:10:10.669273+0000","172.21.15.190:6832/3154132268":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/3277311498":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/3310710469":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1697878697":"2023-11-29T22:08:58.564601+0000","172.21.15.45:6833/3094566588":"2023-11-29T22:10:14.761253+0000","172.21.15.190:0/2564313259":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/1107658380":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/4107859064":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/3311741793":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1352690774":"2023-11-29T22:08:58.564601+0000","172.21.15.190:6833/3154132268":"2023-11-29T22:10:10.669273+0000","172.21.15.190:6833/397715021":"2023-11-29T22:08:58.564601+0000","172.21.15.190:0/1385562856":"2023-11-29T22:10:10.669273+0000","172.21.15.190:0/3670565434":"2023-11-29T22:10:10.669273+0000","172.21.15.190:6832/397715021":"2023-11-29T22:08:58.564601+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T22:10:23.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fde0406af60 msgr2=0x7fde0406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:23.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fde0406af60 0x7fde0406d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde10036000 tx=0x7fde10030000 comp rx=0 tx=0).stop 2023-11-28T22:10:23.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 msgr2=0x7fde201e5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:23.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 0x7fde201e5f90 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00b3f0 tx=0x7fde1c00b8b0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 shutdown_connections 2023-11-28T22:10:23.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fde0406af60 0x7fde0406d420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2015d550 0x7fde20159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde200d1ab0 0x7fde20158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 --2- 172.21.15.45:0/2418236209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde200d02d0 0x7fde201e5f90 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:23.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 >> 172.21.15.45:0/2418236209 conn(0x7fde2006ac70 msgr2=0x7fde200d2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:23.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 shutdown_connections 2023-11-28T22:10:23.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:23.236+0000 7fde265fb640 1 -- 172.21.15.45:0/2418236209 wait complete. 2023-11-28T22:10:24.259 INFO:tasks.ceph:Scrubbing osd.0 2023-11-28T22:10:24.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-28T22:10:24.502 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.501+0000 7ff7219c9640 1 Processor -- start 2023-11-28T22:10:24.503 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.501+0000 7ff7219c9640 1 -- start start 2023-11-28T22:10:24.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.503+0000 7ff7219c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff71c099a40 0x7ff71c06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.503+0000 7ff7219c9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff71c0aff90 0x7ff71c099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.504 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.503+0000 7ff7219c9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c0d1fc0 con 0x7ff71c0cf260 2023-11-28T22:10:24.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.503+0000 7ff7219c9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71c06a910 con 0x7ff71c0aff90 2023-11-28T22:10:24.505 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.503+0000 7ff7219c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0cb8a0 con 0x7ff71c099a40 2023-11-28T22:10:24.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.503+0000 7ff71a7fc640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff71c0aff90 0x7ff71c099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.503+0000 7ff71a7fc640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff71c0aff90 0x7ff71c099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:51786/0 (socket says 172.21.15.45:51786) 2023-11-28T22:10:24.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.503+0000 7ff71a7fc640 1 -- 172.21.15.45:0/2905883901 learned_addr learned my addr 172.21.15.45:0/2905883901 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:24.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.503+0000 7ff71affd640 1 --2- 172.21.15.45:0/2905883901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff71c099a40 0x7ff71c06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.504+0000 7ff71a7fc640 1 -- 172.21.15.45:0/2905883901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff71c099a40 msgr2=0x7ff71c06b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:24.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.504+0000 7ff71a7fc640 1 --2- 172.21.15.45:0/2905883901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff71c099a40 0x7ff71c06b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.504+0000 7ff71a7fc640 1 -- 172.21.15.45:0/2905883901 >> v1:172.21.15.190:6789/0 conn(0x7ff71c0cf260 legacy=0x7ff71c0cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.504+0000 7ff71a7fc640 1 -- 172.21.15.45:0/2905883901 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c09a360 con 0x7ff71c0aff90 2023-11-28T22:10:24.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.505+0000 7ff71a7fc640 1 --2- 172.21.15.45:0/2905883901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c0aff90 0x7ff71c099e20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff704009e80 tx=0x7ff70402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=762135bda41b42b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:24.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.506+0000 7ff719ffb640 1 -- 172.21.15.45:0/2905883901 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70402ca50 con 0x7ff71c0aff90 2023-11-28T22:10:24.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.506+0000 7ff719ffb640 1 -- 172.21.15.45:0/2905883901 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70402cbb0 con 0x7ff71c0aff90 2023-11-28T22:10:24.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.506+0000 7ff719ffb640 1 -- 172.21.15.45:0/2905883901 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7040343f0 con 0x7ff71c0aff90 2023-11-28T22:10:24.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.506+0000 7ff7219c9640 1 -- 172.21.15.45:0/2905883901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c0aff90 msgr2=0x7ff71c099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.506+0000 7ff7219c9640 1 --2- 172.21.15.45:0/2905883901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c0aff90 0x7ff71c099e20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff704009e80 tx=0x7ff70402beb0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.506+0000 7ff7219c9640 1 -- 172.21.15.45:0/2905883901 shutdown_connections 2023-11-28T22:10:24.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.506+0000 7ff7219c9640 1 --2- 172.21.15.45:0/2905883901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c0aff90 0x7ff71c099e20 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.506+0000 7ff7219c9640 1 --2- 172.21.15.45:0/2905883901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff71c099a40 0x7ff71c06b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.506+0000 7ff7219c9640 1 -- 172.21.15.45:0/2905883901 >> 172.21.15.45:0/2905883901 conn(0x7ff71c0d1020 msgr2=0x7ff71c0d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:24.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.507+0000 7ff7219c9640 1 -- 172.21.15.45:0/2905883901 shutdown_connections 2023-11-28T22:10:24.510 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.507+0000 7ff7219c9640 1 -- 172.21.15.45:0/2905883901 wait complete. 2023-11-28T22:10:24.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.508+0000 7ff7219c9640 1 Processor -- start 2023-11-28T22:10:24.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.508+0000 7ff7219c9640 1 -- start start 2023-11-28T22:10:24.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.509+0000 7ff7219c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c099a40 0x7ff71c0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.509+0000 7ff7219c9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff71c0aff90 0x7ff71c1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.509+0000 7ff7219c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c0cf260 0x7ff71c0ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.509+0000 7ff7219c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0b5ad0 con 0x7ff71c0cf260 2023-11-28T22:10:24.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.509+0000 7ff7219c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c157bf0 con 0x7ff71c099a40 2023-11-28T22:10:24.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.509+0000 7ff7219c9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0b4e50 con 0x7ff71c0aff90 2023-11-28T22:10:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c0cf260 0x7ff71c0ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c0cf260 0x7ff71c0ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43822/0 (socket says 172.21.15.45:43822) 2023-11-28T22:10:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71b7fe640 1 -- 172.21.15.45:0/97063020 learned_addr learned my addr 172.21.15.45:0/97063020 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71affd640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c099a40 0x7ff71c0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71a7fc640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff71c0aff90 0x7ff71c1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71b7fe640 1 -- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff71c0aff90 msgr2=0x7ff71c1576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71b7fe640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff71c0aff90 0x7ff71c1576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71b7fe640 1 -- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c099a40 msgr2=0x7ff71c0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71b7fe640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff71c099a40 0x7ff71c0b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.510+0000 7ff71b7fe640 1 -- 172.21.15.45:0/97063020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff704009b30 con 0x7ff71c0cf260 2023-11-28T22:10:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.511+0000 7ff71b7fe640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c0cf260 0x7ff71c0ca340 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7ff71400c390 tx=0x7ff71400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.511+0000 7ff713fff640 1 -- 172.21.15.45:0/97063020 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7140052d0 con 0x7ff71c0cf260 2023-11-28T22:10:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.511+0000 7ff713fff640 1 -- 172.21.15.45:0/97063020 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff714005430 con 0x7ff71c0cf260 2023-11-28T22:10:24.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.511+0000 7ff713fff640 1 -- 172.21.15.45:0/97063020 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff714003810 con 0x7ff71c0cf260 2023-11-28T22:10:24.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.511+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c0b5130 con 0x7ff71c0cf260 2023-11-28T22:10:24.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.511+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c0b56f0 con 0x7ff71c0cf260 2023-11-28T22:10:24.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.512+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff71c0d05a0 con 0x7ff71c0cf260 2023-11-28T22:10:24.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.513+0000 7ff713fff640 1 -- 172.21.15.45:0/97063020 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff714003970 con 0x7ff71c0cf260 2023-11-28T22:10:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.514+0000 7ff713fff640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff6fc06af60 0x7ff6fc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.514+0000 7ff713fff640 1 -- 172.21.15.45:0/97063020 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7ff714088e30 con 0x7ff71c0cf260 2023-11-28T22:10:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.515+0000 7ff713fff640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ff6fc0718f0 0x7ff6fc073d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.515+0000 7ff713fff640 1 -- 172.21.15.45:0/97063020 --> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6fc074420 con 0x7ff6fc0718f0 2023-11-28T22:10:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.515+0000 7ff713fff640 1 -- 172.21.15.45:0/97063020 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=222) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff71408b410 con 0x7ff71c0cf260 2023-11-28T22:10:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.515+0000 7ff71affd640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff6fc06af60 0x7ff6fc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.515+0000 7ff71a7fc640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ff6fc0718f0 0x7ff6fc073d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.516+0000 7ff71a7fc640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ff6fc0718f0 0x7ff6fc073d50 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:24.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.516+0000 7ff71affd640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff6fc06af60 0x7ff6fc06d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff70c006fd0 tx=0x7ff70c008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:24.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.516+0000 7ff713fff640 1 -- 172.21.15.45:0/97063020 <== osd.0 v2:172.21.15.45:6816/2634303336 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff6fc074420 con 0x7ff6fc0718f0 2023-11-28T22:10:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.531+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 --> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff71c0d1ce0 con 0x7ff6fc0718f0 2023-11-28T22:10:24.532 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.531+0000 7ff713fff640 1 -- 172.21.15.45:0/97063020 <== osd.0 v2:172.21.15.45:6816/2634303336 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff70402ca20 con 0x7ff6fc0718f0 2023-11-28T22:10:24.533 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:10:24.533 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-28T22:10:24.533 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:10:24.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.531+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ff6fc0718f0 msgr2=0x7ff6fc073d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.531+0000 7ff7219c9640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.45:6816/2634303336,v1:172.21.15.45:6817/2634303336] conn(0x7ff6fc0718f0 0x7ff6fc073d50 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.534 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.532+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff6fc06af60 msgr2=0x7ff6fc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.532+0000 7ff7219c9640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff6fc06af60 0x7ff6fc06d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff70c006fd0 tx=0x7ff70c008040 comp rx=0 tx=0).stop 2023-11-28T22:10:24.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.532+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c0cf260 msgr2=0x7ff71c0ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.532+0000 7ff7219c9640 1 --2- 172.21.15.45:0/97063020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c0cf260 0x7ff71c0ca340 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7ff71400c390 tx=0x7ff71400c850 comp rx=0 tx=0).stop 2023-11-28T22:10:24.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.532+0000 7ff71b7fe640 1 -- 172.21.15.45:0/97063020 reap_dead start 2023-11-28T22:10:24.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.532+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 shutdown_connections 2023-11-28T22:10:24.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.532+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 >> 172.21.15.45:0/97063020 conn(0x7ff71c0d1020 msgr2=0x7ff71c0aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:24.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.532+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 shutdown_connections 2023-11-28T22:10:24.535 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.532+0000 7ff7219c9640 1 -- 172.21.15.45:0/97063020 wait complete. 2023-11-28T22:10:24.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-11-28T22:10:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.768+0000 7fcb08b4e640 1 Processor -- start 2023-11-28T22:10:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.769+0000 7fcb08b4e640 1 -- start start 2023-11-28T22:10:24.771 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.770+0000 7fcb08b4e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcb040d0190 0x7fcb04167f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.770+0000 7fcb08b4e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb040d2810 con 0x7fcb0415d520 2023-11-28T22:10:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.770+0000 7fcb08b4e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb040b71b0 con 0x7fcb0415d140 2023-11-28T22:10:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.770+0000 7fcb08b4e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb040d1c00 con 0x7fcb040d0190 2023-11-28T22:10:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.770+0000 7fcb02ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcb0415d520 0x7fcb040cfa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37266/0 (socket says 172.21.15.45:37266) 2023-11-28T22:10:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.771+0000 7fcb02ffd640 1 -- 172.21.15.45:0/660849104 learned_addr learned my addr 172.21.15.45:0/660849104 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.771+0000 7fcb03fff640 1 --2- 172.21.15.45:0/660849104 >> v2:172.21.15.190:3300/0 conn(0x7fcb040d0190 0x7fcb04167f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.771+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592516177 0 0) 0x7fcb040d2810 con 0x7fcb0415d520 2023-11-28T22:10:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.771+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf0003520 con 0x7fcb0415d520 2023-11-28T22:10:24.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.771+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1902708417 0 0) 0x7fcaf0003520 con 0x7fcb0415d520 2023-11-28T22:10:24.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.771+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 >> v2:172.21.15.190:3300/0 conn(0x7fcb040d0190 msgr2=0x7fcb04167f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:24.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.771+0000 7fcb027fc640 1 --2- 172.21.15.45:0/660849104 >> v2:172.21.15.190:3300/0 conn(0x7fcb040d0190 0x7fcb04167f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.771+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 >> v1:172.21.15.190:6789/0 conn(0x7fcb0415d140 legacy=0x7fcb040d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.771+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb040d1ee0 con 0x7fcb0415d520 2023-11-28T22:10:24.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fcaec001670 con 0x7fcb0415d520 2023-11-28T22:10:24.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 >> v1:172.21.15.45:6789/0 conn(0x7fcb0415d520 legacy=0x7fcb040cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb027fc640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcaf0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb027fc640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf00047c0 0x7fcaf0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb027fc640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf0005180 0x7fcaf000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb040d1c00 con 0x7fcaf00039c0 2023-11-28T22:10:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf000c060 con 0x7fcaf0005180 2023-11-28T22:10:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf000c220 con 0x7fcaf00047c0 2023-11-28T22:10:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb037fe640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcaf0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb02ffd640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf00047c0 0x7fcaf0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb037fe640 1 -- 172.21.15.45:0/660849104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf00047c0 msgr2=0x7fcaf0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb03fff640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf0005180 0x7fcaf000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb037fe640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcaf00047c0 0x7fcaf0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb037fe640 1 -- 172.21.15.45:0/660849104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf0005180 msgr2=0x7fcaf000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb037fe640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf0005180 0x7fcaf000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb03fff640 1 -- 172.21.15.45:0/660849104 reap_dead start 2023-11-28T22:10:24.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.772+0000 7fcb037fe640 1 -- 172.21.15.45:0/660849104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf0003520 con 0x7fcaf00039c0 2023-11-28T22:10:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.774+0000 7fcb037fe640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcaf0003de0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4001e50 tx=0x7fcaf00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffad9b57bee34ca server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.774+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4003b70 con 0x7fcaf00039c0 2023-11-28T22:10:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.774+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf4003cd0 con 0x7fcaf00039c0 2023-11-28T22:10:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.774+0000 7fcb027fc640 1 -- 172.21.15.45:0/660849104 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4005310 con 0x7fcaf00039c0 2023-11-28T22:10:24.779 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.774+0000 7fcb08b4e640 1 -- 172.21.15.45:0/660849104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 msgr2=0x7fcaf0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.774+0000 7fcb08b4e640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcaf0003de0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4001e50 tx=0x7fcaf00057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.775+0000 7fcb08b4e640 1 -- 172.21.15.45:0/660849104 shutdown_connections 2023-11-28T22:10:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.775+0000 7fcb08b4e640 1 --2- 172.21.15.45:0/660849104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcaf0003de0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.775+0000 7fcb08b4e640 1 -- 172.21.15.45:0/660849104 >> 172.21.15.45:0/660849104 conn(0x7fcb0406a500 msgr2=0x7fcb0406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.775+0000 7fcb08b4e640 1 -- 172.21.15.45:0/660849104 shutdown_connections 2023-11-28T22:10:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.775+0000 7fcb08b4e640 1 -- 172.21.15.45:0/660849104 wait complete. 2023-11-28T22:10:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.776+0000 7fcb08b4e640 1 Processor -- start 2023-11-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.777+0000 7fcb08b4e640 1 -- start start 2023-11-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.777+0000 7fcb08b4e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcb0415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.777+0000 7fcb08b4e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb0414be20 0x7fcb0415fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb08b4e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb041602c0 0x7fcb04154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb08b4e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb04152540 con 0x7fcaf00039c0 2023-11-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb08b4e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb04156850 con 0x7fcb041602c0 2023-11-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb08b4e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb04156290 con 0x7fcb0414be20 2023-11-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb037fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcb0415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb037fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcb0415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43848/0 (socket says 172.21.15.45:43848) 2023-11-28T22:10:24.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb037fe640 1 -- 172.21.15.45:0/799277125 learned_addr learned my addr 172.21.15.45:0/799277125 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:24.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb03fff640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb041602c0 0x7fcb04154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb02ffd640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb0414be20 0x7fcb0415fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb037fe640 1 -- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb0414be20 msgr2=0x7fcb0415fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb037fe640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb0414be20 0x7fcb0415fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb037fe640 1 -- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb041602c0 msgr2=0x7fcb04154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb037fe640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb041602c0 0x7fcb04154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb037fe640 1 -- 172.21.15.45:0/799277125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb04156570 con 0x7fcaf00039c0 2023-11-28T22:10:24.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.778+0000 7fcb03fff640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb041602c0 0x7fcb04154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:24.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.779+0000 7fcb037fe640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcb0415e2f0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fcaf402bef0 tx=0x7fcaf40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:24.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.779+0000 7fcafbfff640 1 -- 172.21.15.45:0/799277125 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4044670 con 0x7fcaf00039c0 2023-11-28T22:10:24.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.779+0000 7fcafbfff640 1 -- 172.21.15.45:0/799277125 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf40447d0 con 0x7fcaf00039c0 2023-11-28T22:10:24.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.779+0000 7fcafbfff640 1 -- 172.21.15.45:0/799277125 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4044b40 con 0x7fcaf00039c0 2023-11-28T22:10:24.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.779+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb040b3840 con 0x7fcaf00039c0 2023-11-28T22:10:24.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.779+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb04160740 con 0x7fcaf00039c0 2023-11-28T22:10:24.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.780+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb040cef10 con 0x7fcaf00039c0 2023-11-28T22:10:24.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.781+0000 7fcafbfff640 1 -- 172.21.15.45:0/799277125 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fcaf4041070 con 0x7fcaf00039c0 2023-11-28T22:10:24.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.782+0000 7fcafbfff640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcae406af60 0x7fcae406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:24.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.782+0000 7fcafbfff640 1 -- 172.21.15.45:0/799277125 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fcaf40ac540 con 0x7fcaf00039c0 2023-11-28T22:10:24.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.783+0000 7fcb02ffd640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcae406af60 0x7fcae406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:24.786 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.784+0000 7fcb02ffd640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcae406af60 0x7fcae406d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcaec000ff0 tx=0x7fcaec009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:24.788 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.787+0000 7fcafbfff640 1 -- 172.21.15.45:0/799277125 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcaf4077240 con 0x7fcaf00039c0 2023-11-28T22:10:24.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.978+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb040b8f40 con 0x7fcae406af60 2023-11-28T22:10:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.979+0000 7fcafbfff640 1 -- 172.21.15.45:0/799277125 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcaec00d040 con 0x7fcae406af60 2023-11-28T22:10:24.981 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 0 to deep-scrub 2023-11-28T22:10:24.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.981+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcae406af60 msgr2=0x7fcae406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcae406af60 0x7fcae406d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcaec000ff0 tx=0x7fcaec009040 comp rx=0 tx=0).stop 2023-11-28T22:10:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 msgr2=0x7fcb0415e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcb0415e2f0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fcaf402bef0 tx=0x7fcaf40014a0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 shutdown_connections 2023-11-28T22:10:24.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcae406af60 0x7fcae406d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb041602c0 0x7fcb04154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb0414be20 0x7fcb0415fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 --2- 172.21.15.45:0/799277125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaf00039c0 0x7fcb0415e2f0 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 >> 172.21.15.45:0/799277125 conn(0x7fcb0406a500 msgr2=0x7fcb040ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 shutdown_connections 2023-11-28T22:10:24.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:24.982+0000 7fcb08b4e640 1 -- 172.21.15.45:0/799277125 wait complete. 2023-11-28T22:10:25.003 INFO:tasks.ceph:Scrubbing osd.1 2023-11-28T22:10:25.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-28T22:10:25.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.230+0000 7f3e9b59e640 1 Processor -- start 2023-11-28T22:10:25.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.230+0000 7f3e9b59e640 1 -- start start 2023-11-28T22:10:25.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.232+0000 7f3e9b59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3e9c070030 0x7f3e9c06de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.232+0000 7f3e9b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3e9c06c090 0x7f3e9c06c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.232+0000 7f3e9b59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c06e3a0 con 0x7f3e9c15b190 2023-11-28T22:10:25.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.232+0000 7f3e9b59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9c06ca60 con 0x7f3e9c070030 2023-11-28T22:10:25.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.232+0000 7f3e9b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9c15b570 con 0x7f3e9c06c090 2023-11-28T22:10:25.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.232+0000 7f3e99d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3e9c070030 0x7f3e9c06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.232+0000 7f3e9a59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3e9c15b190 0x7f3e9c0b1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37274/0 (socket says 172.21.15.45:37274) 2023-11-28T22:10:25.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.232+0000 7f3e9a59c640 1 -- 172.21.15.45:0/4202546434 learned_addr learned my addr 172.21.15.45:0/4202546434 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:25.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.233+0000 7f3e9ad9d640 1 --2- 172.21.15.45:0/4202546434 >> v2:172.21.15.190:3300/0 conn(0x7f3e9c06c090 0x7f3e9c06c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.233+0000 7f3e99d9b640 1 -- 172.21.15.45:0/4202546434 >> v2:172.21.15.190:3300/0 conn(0x7f3e9c06c090 msgr2=0x7f3e9c06c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:25.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.233+0000 7f3e99d9b640 1 --2- 172.21.15.45:0/4202546434 >> v2:172.21.15.190:3300/0 conn(0x7f3e9c06c090 0x7f3e9c06c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.233+0000 7f3e99d9b640 1 -- 172.21.15.45:0/4202546434 >> v1:172.21.15.45:6789/0 conn(0x7f3e9c15b190 legacy=0x7f3e9c0b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.233+0000 7f3e9959a640 1 -- 172.21.15.45:0/4202546434 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267320254 0 0) 0x7f3e8c001110 con 0x7f3e9c15b190 2023-11-28T22:10:25.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.233+0000 7f3e99d9b640 1 -- 172.21.15.45:0/4202546434 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c0c9d20 con 0x7f3e9c070030 2023-11-28T22:10:25.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.233+0000 7f3e9ad9d640 1 --2- 172.21.15.45:0/4202546434 >> v2:172.21.15.190:3300/0 conn(0x7f3e9c06c090 0x7f3e9c06c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:25.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.234+0000 7f3e99d9b640 1 --2- 172.21.15.45:0/4202546434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 0x7f3e9c06de60 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c0cea10 tx=0x7f3e9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec35a887bcec569 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.236 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.234+0000 7f3e9959a640 1 -- 172.21.15.45:0/4202546434 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90031830 con 0x7f3e9c070030 2023-11-28T22:10:25.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.234+0000 7f3e9959a640 1 -- 172.21.15.45:0/4202546434 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e90031990 con 0x7f3e9c070030 2023-11-28T22:10:25.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.234+0000 7f3e9959a640 1 -- 172.21.15.45:0/4202546434 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90031d20 con 0x7f3e9c070030 2023-11-28T22:10:25.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.235+0000 7f3e9b59e640 1 -- 172.21.15.45:0/4202546434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 msgr2=0x7f3e9c06de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.235+0000 7f3e9b59e640 1 --2- 172.21.15.45:0/4202546434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 0x7f3e9c06de60 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c0cea10 tx=0x7f3e9002b8d0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.235+0000 7f3e9b59e640 1 -- 172.21.15.45:0/4202546434 shutdown_connections 2023-11-28T22:10:25.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.235+0000 7f3e9b59e640 1 --2- 172.21.15.45:0/4202546434 >> v2:172.21.15.190:3300/0 conn(0x7f3e9c06c090 0x7f3e9c06c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.235+0000 7f3e9b59e640 1 --2- 172.21.15.45:0/4202546434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 0x7f3e9c06de60 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.235+0000 7f3e9b59e640 1 -- 172.21.15.45:0/4202546434 >> 172.21.15.45:0/4202546434 conn(0x7f3e9c0cec30 msgr2=0x7f3e9c0cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:25.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.235+0000 7f3e9b59e640 1 -- 172.21.15.45:0/4202546434 shutdown_connections 2023-11-28T22:10:25.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.235+0000 7f3e9b59e640 1 -- 172.21.15.45:0/4202546434 wait complete. 2023-11-28T22:10:25.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.237+0000 7f3e9b59e640 1 Processor -- start 2023-11-28T22:10:25.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.237+0000 7f3e9b59e640 1 -- start start 2023-11-28T22:10:25.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.238+0000 7f3e9b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e9c06c090 0x7f3e9c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.238+0000 7f3e9b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 0x7f3e9c0c3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.238+0000 7f3e9b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e9c15b190 0x7f3e9c15d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.240 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.238+0000 7f3e9b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c0c3690 con 0x7f3e9c070030 2023-11-28T22:10:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.238+0000 7f3e9b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c0c38c0 con 0x7f3e9c06c090 2023-11-28T22:10:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.238+0000 7f3e9b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c0c3ba0 con 0x7f3e9c15b190 2023-11-28T22:10:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.238+0000 7f3e99d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 0x7f3e9c0c3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e99d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 0x7f3e9c0c3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43870/0 (socket says 172.21.15.45:43870) 2023-11-28T22:10:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e99d9b640 1 -- 172.21.15.45:0/75896314 learned_addr learned my addr 172.21.15.45:0/75896314 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:25.241 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e9a59c640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e9c06c090 0x7f3e9c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.242 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e99d9b640 1 -- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e9c15b190 msgr2=0x7f3e9c15d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e9ad9d640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e9c15b190 0x7f3e9c15d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e99d9b640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3e9c15b190 0x7f3e9c15d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e99d9b640 1 -- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e9c06c090 msgr2=0x7f3e9c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e99d9b640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e9c06c090 0x7f3e9c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e99d9b640 1 -- 172.21.15.45:0/75896314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e90009400 con 0x7f3e9c070030 2023-11-28T22:10:25.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.239+0000 7f3e99d9b640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 0x7f3e9c0c3150 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009890 tx=0x7f3e900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.240+0000 7f3e8affd640 1 -- 172.21.15.45:0/75896314 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9003ba70 con 0x7f3e9c070030 2023-11-28T22:10:25.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.240+0000 7f3e8affd640 1 -- 172.21.15.45:0/75896314 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9003bbf0 con 0x7f3e9c070030 2023-11-28T22:10:25.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.240+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c0c3e80 con 0x7f3e9c070030 2023-11-28T22:10:25.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.240+0000 7f3e8affd640 1 -- 172.21.15.45:0/75896314 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900453f0 con 0x7f3e9c070030 2023-11-28T22:10:25.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.240+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c0c4440 con 0x7f3e9c070030 2023-11-28T22:10:25.244 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.241+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e9c071ef0 con 0x7f3e9c070030 2023-11-28T22:10:25.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.242+0000 7f3e8affd640 1 -- 172.21.15.45:0/75896314 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3e90045550 con 0x7f3e9c070030 2023-11-28T22:10:25.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.243+0000 7f3e8affd640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3e7c06af60 0x7f3e7c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.245 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.243+0000 7f3e8affd640 1 -- 172.21.15.45:0/75896314 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f3e900abe70 con 0x7f3e9c070030 2023-11-28T22:10:25.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.243+0000 7f3e8affd640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7f3e7c0718f0 0x7f3e7c073d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.243+0000 7f3e8affd640 1 -- 172.21.15.45:0/75896314 --> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e7c074420 con 0x7f3e7c0718f0 2023-11-28T22:10:25.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.243+0000 7f3e8affd640 1 -- 172.21.15.45:0/75896314 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=222) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e900ae450 con 0x7f3e9c070030 2023-11-28T22:10:25.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.243+0000 7f3e9ad9d640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7f3e7c0718f0 0x7f3e7c073d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.244+0000 7f3e9a59c640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3e7c06af60 0x7f3e7c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.246 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.244+0000 7f3e9ad9d640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7f3e7c0718f0 0x7f3e7c073d50 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.245+0000 7f3e9a59c640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3e7c06af60 0x7f3e7c06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c000f60 tx=0x7f3e8c007c00 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.247 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.245+0000 7f3e8affd640 1 -- 172.21.15.45:0/75896314 <== osd.1 v2:172.21.15.45:6800/901684499 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f3e7c074420 con 0x7f3e7c0718f0 2023-11-28T22:10:25.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.259+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 --> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3e9c071c50 con 0x7f3e7c0718f0 2023-11-28T22:10:25.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.260+0000 7f3e8affd640 1 -- 172.21.15.45:0/75896314 <== osd.1 v2:172.21.15.45:6800/901684499 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3e9400d490 con 0x7f3e7c0718f0 2023-11-28T22:10:25.262 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:10:25.262 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-28T22:10:25.262 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:10:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7f3e7c0718f0 msgr2=0x7f3e7c073d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.45:6800/901684499,v1:172.21.15.45:6801/901684499] conn(0x7f3e7c0718f0 0x7f3e7c073d50 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3e7c06af60 msgr2=0x7f3e7c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f3e7c06af60 0x7f3e7c06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c000f60 tx=0x7f3e8c007c00 comp rx=0 tx=0).stop 2023-11-28T22:10:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 msgr2=0x7f3e9c0c3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 --2- 172.21.15.45:0/75896314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e9c070030 0x7f3e9c0c3150 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009890 tx=0x7f3e900046c0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9ad9d640 1 -- 172.21.15.45:0/75896314 reap_dead start 2023-11-28T22:10:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 shutdown_connections 2023-11-28T22:10:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 >> 172.21.15.45:0/75896314 conn(0x7f3e9c0cec30 msgr2=0x7f3e9c0cbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:25.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 shutdown_connections 2023-11-28T22:10:25.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.261+0000 7f3e9b59e640 1 -- 172.21.15.45:0/75896314 wait complete. 2023-11-28T22:10:25.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-11-28T22:10:25.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.505+0000 7fcb1b59e640 1 Processor -- start 2023-11-28T22:10:25.507 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.505+0000 7fcb1b59e640 1 -- start start 2023-11-28T22:10:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.507+0000 7fcb1b59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcb1c15c050 0x7fcb1c166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.507+0000 7fcb1b59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c15c670 con 0x7fcb1c141600 2023-11-28T22:10:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.507+0000 7fcb1b59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c0cda40 con 0x7fcb1c0cc2a0 2023-11-28T22:10:25.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.507+0000 7fcb1b59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1c0ccc80 con 0x7fcb1c15c050 2023-11-28T22:10:25.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.507+0000 7fcb1a59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcb1c141600 0x7fcb1c0cbb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37290/0 (socket says 172.21.15.45:37290) 2023-11-28T22:10:25.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.507+0000 7fcb1a59c640 1 -- 172.21.15.45:0/2452785146 learned_addr learned my addr 172.21.15.45:0/2452785146 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.507+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985902470 0 0) 0x7fcb1c15c670 con 0x7fcb1c141600 2023-11-28T22:10:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb08003520 con 0x7fcb1c141600 2023-11-28T22:10:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb1ad9d640 1 --2- 172.21.15.45:0/2452785146 >> v2:172.21.15.190:3300/0 conn(0x7fcb1c15c050 0x7fcb1c166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1159996483 0 0) 0x7fcb08003520 con 0x7fcb1c141600 2023-11-28T22:10:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 >> v2:172.21.15.190:3300/0 conn(0x7fcb1c15c050 msgr2=0x7fcb1c166020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 --2- 172.21.15.45:0/2452785146 >> v2:172.21.15.190:3300/0 conn(0x7fcb1c15c050 0x7fcb1c166020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 >> v1:172.21.15.190:6789/0 conn(0x7fcb1c0cc2a0 legacy=0x7fcb1c15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c0ccf60 con 0x7fcb1c141600 2023-11-28T22:10:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fcb0c001670 con 0x7fcb1c141600 2023-11-28T22:10:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 >> v1:172.21.15.45:6789/0 conn(0x7fcb1c141600 legacy=0x7fcb1c0cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb08003970 0x7fcb08003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb080047c0 0x7fcb08004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb0800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c0ccc80 con 0x7fcb08005180 2023-11-28T22:10:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0800c0b0 con 0x7fcb080047c0 2023-11-28T22:10:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.508+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0800c270 con 0x7fcb08003970 2023-11-28T22:10:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.509+0000 7fcb1ad9d640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb0800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.509+0000 7fcb13fff640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb080047c0 0x7fcb08004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.509+0000 7fcb1a59c640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb08003970 0x7fcb08003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.509+0000 7fcb1ad9d640 1 -- 172.21.15.45:0/2452785146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb08003970 msgr2=0x7fcb08003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:25.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.509+0000 7fcb1ad9d640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb08003970 0x7fcb08003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.509+0000 7fcb1ad9d640 1 -- 172.21.15.45:0/2452785146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb080047c0 msgr2=0x7fcb08004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:25.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.509+0000 7fcb1ad9d640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb080047c0 0x7fcb08004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.514 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.509+0000 7fcb1ad9d640 1 -- 172.21.15.45:0/2452785146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb08003520 con 0x7fcb08005180 2023-11-28T22:10:25.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.510+0000 7fcb1ad9d640 1 -- 172.21.15.45:0/2452785146 reap_dead start 2023-11-28T22:10:25.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.510+0000 7fcb1ad9d640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb0800b980 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fcb14007360 tx=0x7fcb080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e3b57ee7b4d3e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1402ccf0 con 0x7fcb08005180 2023-11-28T22:10:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1402ce50 con 0x7fcb08005180 2023-11-28T22:10:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb19d9b640 1 -- 172.21.15.45:0/2452785146 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb14003bd0 con 0x7fcb08005180 2023-11-28T22:10:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2452785146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 msgr2=0x7fcb0800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb1b59e640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb0800b980 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fcb14007360 tx=0x7fcb080057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2452785146 shutdown_connections 2023-11-28T22:10:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb1b59e640 1 --2- 172.21.15.45:0/2452785146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb0800b980 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2452785146 >> 172.21.15.45:0/2452785146 conn(0x7fcb1c0b0e70 msgr2=0x7fcb1c0b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:25.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2452785146 shutdown_connections 2023-11-28T22:10:25.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.511+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2452785146 wait complete. 2023-11-28T22:10:25.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.513+0000 7fcb1b59e640 1 Processor -- start 2023-11-28T22:10:25.517 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.514+0000 7fcb1b59e640 1 -- start start 2023-11-28T22:10:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb1c1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1b59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb1c0acdf0 0x7fcb1c0ad250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1b59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb1c09a680 0x7fcb1c09ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c09b040 con 0x7fcb08005180 2023-11-28T22:10:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1b59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c158730 con 0x7fcb1c09a680 2023-11-28T22:10:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1b59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c0b3a30 con 0x7fcb1c0acdf0 2023-11-28T22:10:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb1c1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb1c1581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43892/0 (socket says 172.21.15.45:43892) 2023-11-28T22:10:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1a59c640 1 -- 172.21.15.45:0/2066885137 learned_addr learned my addr 172.21.15.45:0/2066885137 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1ad9d640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb1c09a680 0x7fcb1c09ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb13fff640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb1c0acdf0 0x7fcb1c0ad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1a59c640 1 -- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb1c0acdf0 msgr2=0x7fcb1c0ad250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1a59c640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb1c0acdf0 0x7fcb1c0ad250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1a59c640 1 -- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb1c09a680 msgr2=0x7fcb1c09ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1a59c640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb1c09a680 0x7fcb1c09ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.515+0000 7fcb1a59c640 1 -- 172.21.15.45:0/2066885137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c0b2b40 con 0x7fcb08005180 2023-11-28T22:10:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.516+0000 7fcb1ad9d640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb1c09a680 0x7fcb1c09ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.516+0000 7fcb1a59c640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb1c1581f0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c001da0 tx=0x7fcb0c0084f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.516+0000 7fcb12ffd640 1 -- 172.21.15.45:0/2066885137 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c00eae0 con 0x7fcb08005180 2023-11-28T22:10:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.516+0000 7fcb12ffd640 1 -- 172.21.15.45:0/2066885137 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0c00ec60 con 0x7fcb08005180 2023-11-28T22:10:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.516+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c0b2e20 con 0x7fcb08005180 2023-11-28T22:10:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.516+0000 7fcb12ffd640 1 -- 172.21.15.45:0/2066885137 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c008740 con 0x7fcb08005180 2023-11-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.516+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c0b33e0 con 0x7fcb08005180 2023-11-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.518+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c06d710 con 0x7fcb08005180 2023-11-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.518+0000 7fcb12ffd640 1 -- 172.21.15.45:0/2066885137 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fcb0c01d460 con 0x7fcb08005180 2023-11-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.519+0000 7fcb12ffd640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcafc06af60 0x7fcafc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.519+0000 7fcb12ffd640 1 -- 172.21.15.45:0/2066885137 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fcb0c089cd0 con 0x7fcb08005180 2023-11-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.520+0000 7fcb13fff640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcafc06af60 0x7fcafc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.521+0000 7fcb13fff640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcafc06af60 0x7fcafc06d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c0b20c0 tx=0x7fcb04009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.524+0000 7fcb12ffd640 1 -- 172.21.15.45:0/2066885137 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb0c0583f0 con 0x7fcb08005180 2023-11-28T22:10:25.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.721+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb1c1419e0 con 0x7fcafc06af60 2023-11-28T22:10:25.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.722+0000 7fcb12ffd640 1 -- 172.21.15.45:0/2066885137 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcb0400d050 con 0x7fcafc06af60 2023-11-28T22:10:25.724 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 1 to deep-scrub 2023-11-28T22:10:25.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcafc06af60 msgr2=0x7fcafc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcafc06af60 0x7fcafc06d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c0b20c0 tx=0x7fcb04009040 comp rx=0 tx=0).stop 2023-11-28T22:10:25.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 msgr2=0x7fcb1c1581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb1c1581f0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c001da0 tx=0x7fcb0c0084f0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 shutdown_connections 2023-11-28T22:10:25.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcafc06af60 0x7fcafc06d420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb1c09a680 0x7fcb1c09ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb1c0acdf0 0x7fcb1c0ad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 --2- 172.21.15.45:0/2066885137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb08005180 0x7fcb1c1581f0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 >> 172.21.15.45:0/2066885137 conn(0x7fcb1c0b0e70 msgr2=0x7fcb1c0cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:25.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.725+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 shutdown_connections 2023-11-28T22:10:25.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.726+0000 7fcb1b59e640 1 -- 172.21.15.45:0/2066885137 wait complete. 2023-11-28T22:10:25.747 INFO:tasks.ceph:Scrubbing osd.2 2023-11-28T22:10:25.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-28T22:10:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.978+0000 7ff5da3be640 1 Processor -- start 2023-11-28T22:10:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.978+0000 7ff5da3be640 1 -- start start 2023-11-28T22:10:25.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.980+0000 7ff5da3be640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d406ac10 0x7ff5d40c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.980+0000 7ff5da3be640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d406e030 con 0x7ff5d40ce600 2023-11-28T22:10:25.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.980+0000 7ff5da3be640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d40ccf60 con 0x7ff5d406dc50 2023-11-28T22:10:25.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.980+0000 7ff5da3be640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40cc930 con 0x7ff5d406ac10 2023-11-28T22:10:25.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.980+0000 7ff5cbfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff5d40ce600 0x7ff5d406a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37298/0 (socket says 172.21.15.45:37298) 2023-11-28T22:10:25.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.980+0000 7ff5cbfff640 1 -- 172.21.15.45:0/171712663 learned_addr learned my addr 172.21.15.45:0/171712663 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.981+0000 7ff5d895b640 1 --2- 172.21.15.45:0/171712663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d406ac10 0x7ff5d40c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.981+0000 7ff5d37fe640 1 -- 172.21.15.45:0/171712663 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58926720 0 0) 0x7ff5bc001110 con 0x7ff5d40ce600 2023-11-28T22:10:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.981+0000 7ff5d37fe640 1 -- 172.21.15.45:0/171712663 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c0003520 con 0x7ff5d40ce600 2023-11-28T22:10:25.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.981+0000 7ff5d895b640 1 -- 172.21.15.45:0/171712663 >> v1:172.21.15.190:6789/0 conn(0x7ff5d406dc50 legacy=0x7ff5d40cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.981+0000 7ff5d895b640 1 -- 172.21.15.45:0/171712663 >> v1:172.21.15.45:6789/0 conn(0x7ff5d40ce600 legacy=0x7ff5d406a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.982+0000 7ff5d895b640 1 -- 172.21.15.45:0/171712663 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d40ae4d0 con 0x7ff5d406ac10 2023-11-28T22:10:25.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.983+0000 7ff5d895b640 1 --2- 172.21.15.45:0/171712663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d406ac10 0x7ff5d40c93b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009dc0 tx=0x7ff5cc02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40e80223fc2a11ef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.983+0000 7ff5d37fe640 1 -- 172.21.15.45:0/171712663 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc02ccc0 con 0x7ff5d406ac10 2023-11-28T22:10:25.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.983+0000 7ff5d37fe640 1 -- 172.21.15.45:0/171712663 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5cc02ce20 con 0x7ff5d406ac10 2023-11-28T22:10:25.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.983+0000 7ff5d37fe640 1 -- 172.21.15.45:0/171712663 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc034640 con 0x7ff5d406ac10 2023-11-28T22:10:25.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.984+0000 7ff5da3be640 1 -- 172.21.15.45:0/171712663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d406ac10 msgr2=0x7ff5d40c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.984+0000 7ff5da3be640 1 --2- 172.21.15.45:0/171712663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d406ac10 0x7ff5d40c93b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009dc0 tx=0x7ff5cc02bf10 comp rx=0 tx=0).stop 2023-11-28T22:10:25.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.984+0000 7ff5da3be640 1 -- 172.21.15.45:0/171712663 shutdown_connections 2023-11-28T22:10:25.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.984+0000 7ff5da3be640 1 --2- 172.21.15.45:0/171712663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d406ac10 0x7ff5d40c93b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.984+0000 7ff5da3be640 1 -- 172.21.15.45:0/171712663 >> 172.21.15.45:0/171712663 conn(0x7ff5d40ced60 msgr2=0x7ff5d40cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:25.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.984+0000 7ff5da3be640 1 -- 172.21.15.45:0/171712663 shutdown_connections 2023-11-28T22:10:25.987 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.984+0000 7ff5da3be640 1 -- 172.21.15.45:0/171712663 wait complete. 2023-11-28T22:10:25.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.986+0000 7ff5da3be640 1 Processor -- start 2023-11-28T22:10:25.988 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.986+0000 7ff5da3be640 1 -- start start 2023-11-28T22:10:25.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.987+0000 7ff5da3be640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5d406ac10 0x7ff5d415c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5da3be640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5d406dc50 0x7ff5d40c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.991 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5da3be640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d40ce600 0x7ff5d40c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5da3be640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40b3c40 con 0x7ff5d406dc50 2023-11-28T22:10:25.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5da3be640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40b3f20 con 0x7ff5d406ac10 2023-11-28T22:10:25.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5da3be640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40b4200 con 0x7ff5d40ce600 2023-11-28T22:10:25.992 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5d406dc50 0x7ff5d40c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5d406dc50 0x7ff5d40c6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43898/0 (socket says 172.21.15.45:43898) 2023-11-28T22:10:25.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5cbfff640 1 -- 172.21.15.45:0/1648246172 learned_addr learned my addr 172.21.15.45:0/1648246172 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:25.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5d3fff640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5d406ac10 0x7ff5d415c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.993 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5d895b640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d40ce600 0x7ff5d40c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5cbfff640 1 -- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d40ce600 msgr2=0x7ff5d40c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5cbfff640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d40ce600 0x7ff5d40c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5cbfff640 1 -- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5d406ac10 msgr2=0x7ff5d415c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:25.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.988+0000 7ff5cbfff640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5d406ac10 0x7ff5d415c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:25.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.989+0000 7ff5cbfff640 1 -- 172.21.15.45:0/1648246172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc009a70 con 0x7ff5d406dc50 2023-11-28T22:10:25.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.989+0000 7ff5d895b640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff5d40ce600 0x7ff5d40c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:25.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.989+0000 7ff5cbfff640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5d406dc50 0x7ff5d40c6c90 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00c420 tx=0x7ff5bc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.989+0000 7ff5d17fa640 1 -- 172.21.15.45:0/1648246172 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc005ce0 con 0x7ff5d406dc50 2023-11-28T22:10:25.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.989+0000 7ff5d17fa640 1 -- 172.21.15.45:0/1648246172 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc005e40 con 0x7ff5d406dc50 2023-11-28T22:10:25.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.989+0000 7ff5d17fa640 1 -- 172.21.15.45:0/1648246172 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0072f0 con 0x7ff5d406dc50 2023-11-28T22:10:25.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.989+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d40b44e0 con 0x7ff5d406dc50 2023-11-28T22:10:25.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.989+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d415c810 con 0x7ff5d406dc50 2023-11-28T22:10:25.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.990+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5d40ccbf0 con 0x7ff5d406dc50 2023-11-28T22:10:25.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.991+0000 7ff5d17fa640 1 -- 172.21.15.45:0/1648246172 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff5bc007510 con 0x7ff5d406dc50 2023-11-28T22:10:25.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.993+0000 7ff5d17fa640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff5b406af60 0x7ff5b406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.993+0000 7ff5d17fa640 1 -- 172.21.15.45:0/1648246172 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7ff5bc088e10 con 0x7ff5d406dc50 2023-11-28T22:10:25.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.993+0000 7ff5d17fa640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7ff5b40718f0 0x7ff5b4073d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:25.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.993+0000 7ff5d17fa640 1 -- 172.21.15.45:0/1648246172 --> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5b4074420 con 0x7ff5b40718f0 2023-11-28T22:10:25.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.993+0000 7ff5d17fa640 1 -- 172.21.15.45:0/1648246172 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=222) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5bc08b3f0 con 0x7ff5d406dc50 2023-11-28T22:10:25.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.993+0000 7ff5d3fff640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff5b406af60 0x7ff5b406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.993+0000 7ff5d895b640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7ff5b40718f0 0x7ff5b4073d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:25.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.994+0000 7ff5d895b640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7ff5b40718f0 0x7ff5b4073d50 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.995+0000 7ff5d3fff640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff5b406af60 0x7ff5b406d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4001640 tx=0x7ff5c4009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:25.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:25.995+0000 7ff5d17fa640 1 -- 172.21.15.45:0/1648246172 <== osd.2 v2:172.21.15.45:6824/539709697 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff5b4074420 con 0x7ff5b40718f0 2023-11-28T22:10:26.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.010+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 --> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff5d4070880 con 0x7ff5b40718f0 2023-11-28T22:10:26.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.010+0000 7ff5d17fa640 1 -- 172.21.15.45:0/1648246172 <== osd.2 v2:172.21.15.45:6824/539709697 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff5cc003a90 con 0x7ff5b40718f0 2023-11-28T22:10:26.012 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:10:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-28T22:10:26.012 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:10:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.010+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7ff5b40718f0 msgr2=0x7ff5b4073d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.010+0000 7ff5da3be640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.45:6824/539709697,v1:172.21.15.45:6825/539709697] conn(0x7ff5b40718f0 0x7ff5b4073d50 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.011+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff5b406af60 msgr2=0x7ff5b406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.011+0000 7ff5da3be640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7ff5b406af60 0x7ff5b406d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4001640 tx=0x7ff5c4009040 comp rx=0 tx=0).stop 2023-11-28T22:10:26.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.011+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5d406dc50 msgr2=0x7ff5d40c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.011+0000 7ff5da3be640 1 --2- 172.21.15.45:0/1648246172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5d406dc50 0x7ff5d40c6c90 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00c420 tx=0x7ff5bc00c8e0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.011+0000 7ff5d895b640 1 -- 172.21.15.45:0/1648246172 reap_dead start 2023-11-28T22:10:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.011+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 shutdown_connections 2023-11-28T22:10:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.011+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 >> 172.21.15.45:0/1648246172 conn(0x7ff5d40ced60 msgr2=0x7ff5d40cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.011+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 shutdown_connections 2023-11-28T22:10:26.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.011+0000 7ff5da3be640 1 -- 172.21.15.45:0/1648246172 wait complete. 2023-11-28T22:10:26.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-11-28T22:10:26.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.253+0000 7fcd7c97c640 1 Processor -- start 2023-11-28T22:10:26.254 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.253+0000 7fcd7c97c640 1 -- start start 2023-11-28T22:10:26.255 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.254+0000 7fcd7c97c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd7815b290 0x7fcd780d18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.254+0000 7fcd7c97c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcd780cec80 0x7fcd780cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.254+0000 7fcd7c97c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd780cfd30 con 0x7fcd780cf8e0 2023-11-28T22:10:26.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.254+0000 7fcd7c97c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd780ccd10 con 0x7fcd780cec80 2023-11-28T22:10:26.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.254+0000 7fcd7c97c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd780b0840 con 0x7fcd7815b290 2023-11-28T22:10:26.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.255+0000 7fcd77fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcd780cf8e0 0x7fcd780cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37312/0 (socket says 172.21.15.45:37312) 2023-11-28T22:10:26.256 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.255+0000 7fcd77fff640 1 -- 172.21.15.45:0/2053183092 learned_addr learned my addr 172.21.15.45:0/2053183092 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:26.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.255+0000 7fcd777fe640 1 --2- 172.21.15.45:0/2053183092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd7815b290 0x7fcd780d18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.255+0000 7fcd76ffd640 1 --2- 172.21.15.45:0/2053183092 >> v2:172.21.15.190:3300/0 conn(0x7fcd780cec80 0x7fcd780cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.255+0000 7fcd767fc640 1 -- 172.21.15.45:0/2053183092 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988143047 0 0) 0x7fcd70001110 con 0x7fcd780cf8e0 2023-11-28T22:10:26.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.255+0000 7fcd767fc640 1 -- 172.21.15.45:0/2053183092 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd64003520 con 0x7fcd780cf8e0 2023-11-28T22:10:26.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.256+0000 7fcd777fe640 1 -- 172.21.15.45:0/2053183092 >> v2:172.21.15.190:3300/0 conn(0x7fcd780cec80 msgr2=0x7fcd780cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:26.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.256+0000 7fcd777fe640 1 --2- 172.21.15.45:0/2053183092 >> v2:172.21.15.190:3300/0 conn(0x7fcd780cec80 0x7fcd780cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.256+0000 7fcd777fe640 1 -- 172.21.15.45:0/2053183092 >> v1:172.21.15.45:6789/0 conn(0x7fcd780cf8e0 legacy=0x7fcd780cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.256+0000 7fcd767fc640 1 -- 172.21.15.45:0/2053183092 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2854273787 0 0) 0x7fcd64003520 con 0x7fcd780cf8e0 2023-11-28T22:10:26.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.256+0000 7fcd777fe640 1 -- 172.21.15.45:0/2053183092 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd780cc750 con 0x7fcd7815b290 2023-11-28T22:10:26.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.257+0000 7fcd777fe640 1 --2- 172.21.15.45:0/2053183092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd7815b290 0x7fcd780d18e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcd680097a0 tx=0x7fcd6802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91951a326183caf9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd767fc640 1 -- 172.21.15.45:0/2053183092 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70001800 con 0x7fcd7815b290 2023-11-28T22:10:26.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd767fc640 1 -- 172.21.15.45:0/2053183092 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6802e050 con 0x7fcd7815b290 2023-11-28T22:10:26.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd767fc640 1 -- 172.21.15.45:0/2053183092 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68030710 con 0x7fcd7815b290 2023-11-28T22:10:26.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd7c97c640 1 -- 172.21.15.45:0/2053183092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd7815b290 msgr2=0x7fcd780d18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd7c97c640 1 --2- 172.21.15.45:0/2053183092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd7815b290 0x7fcd780d18e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcd680097a0 tx=0x7fcd6802b790 comp rx=0 tx=0).stop 2023-11-28T22:10:26.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd7c97c640 1 -- 172.21.15.45:0/2053183092 shutdown_connections 2023-11-28T22:10:26.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd7c97c640 1 --2- 172.21.15.45:0/2053183092 >> v2:172.21.15.190:3300/0 conn(0x7fcd780cec80 0x7fcd780cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd7c97c640 1 --2- 172.21.15.45:0/2053183092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd7815b290 0x7fcd780d18e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd7c97c640 1 -- 172.21.15.45:0/2053183092 >> 172.21.15.45:0/2053183092 conn(0x7fcd780d0dc0 msgr2=0x7fcd780d11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:26.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd7c97c640 1 -- 172.21.15.45:0/2053183092 shutdown_connections 2023-11-28T22:10:26.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.258+0000 7fcd7c97c640 1 -- 172.21.15.45:0/2053183092 wait complete. 2023-11-28T22:10:26.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.260+0000 7fcd7c97c640 1 Processor -- start 2023-11-28T22:10:26.262 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.260+0000 7fcd7c97c640 1 -- start start 2023-11-28T22:10:26.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.261+0000 7fcd7c97c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd780cec80 0x7fcd78154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.261+0000 7fcd7c97c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd780cf8e0 0x7fcd780c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.261+0000 7fcd7c97c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd7815b290 0x7fcd780c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.261+0000 7fcd7c97c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7815caa0 con 0x7fcd7815b290 2023-11-28T22:10:26.264 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.261+0000 7fcd7c97c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd781557e0 con 0x7fcd780cf8e0 2023-11-28T22:10:26.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.261+0000 7fcd7c97c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78155ac0 con 0x7fcd780cec80 2023-11-28T22:10:26.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd77fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd7815b290 0x7fcd780c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd77fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd7815b290 0x7fcd780c6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43906/0 (socket says 172.21.15.45:43906) 2023-11-28T22:10:26.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd77fff640 1 -- 172.21.15.45:0/3198115511 learned_addr learned my addr 172.21.15.45:0/3198115511 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:26.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd777fe640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd780cec80 0x7fcd78154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd77fff640 1 -- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd780cec80 msgr2=0x7fcd78154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.265 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd76ffd640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd780cf8e0 0x7fcd780c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd77fff640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd780cec80 0x7fcd78154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd77fff640 1 -- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd780cf8e0 msgr2=0x7fcd780c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd77fff640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd780cf8e0 0x7fcd780c8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd77fff640 1 -- 172.21.15.45:0/3198115511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68009400 con 0x7fcd7815b290 2023-11-28T22:10:26.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd76ffd640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd780cf8e0 0x7fcd780c8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:26.266 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.262+0000 7fcd77fff640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd7815b290 0x7fcd780c6280 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fcd70007fd0 tx=0x7fcd7000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.263+0000 7fcd6ffff640 1 -- 172.21.15.45:0/3198115511 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70005ce0 con 0x7fcd7815b290 2023-11-28T22:10:26.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.263+0000 7fcd6ffff640 1 -- 172.21.15.45:0/3198115511 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd70005e60 con 0x7fcd7815b290 2023-11-28T22:10:26.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.263+0000 7fcd6ffff640 1 -- 172.21.15.45:0/3198115511 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd700072b0 con 0x7fcd7815b290 2023-11-28T22:10:26.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.263+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd78155da0 con 0x7fcd7815b290 2023-11-28T22:10:26.267 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.263+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd78155480 con 0x7fcd7815b290 2023-11-28T22:10:26.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.264+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7815cbe0 con 0x7fcd7815b290 2023-11-28T22:10:26.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.265+0000 7fcd6ffff640 1 -- 172.21.15.45:0/3198115511 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fcd70007410 con 0x7fcd7815b290 2023-11-28T22:10:26.268 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.266+0000 7fcd6ffff640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcd5806af60 0x7fcd5806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.266+0000 7fcd6ffff640 1 -- 172.21.15.45:0/3198115511 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fcd700894b0 con 0x7fcd7815b290 2023-11-28T22:10:26.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.267+0000 7fcd777fe640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcd5806af60 0x7fcd5806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.269 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.268+0000 7fcd777fe640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcd5806af60 0x7fcd5806d420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcd68034000 tx=0x7fcd6802f000 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.271 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.270+0000 7fcd6ffff640 1 -- 172.21.15.45:0/3198115511 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd7815cbe0 con 0x7fcd7815b290 2023-11-28T22:10:26.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.467+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd7815b670 con 0x7fcd5806af60 2023-11-28T22:10:26.469 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.468+0000 7fcd6ffff640 1 -- 172.21.15.45:0/3198115511 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcd680043e0 con 0x7fcd5806af60 2023-11-28T22:10:26.469 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 2 to deep-scrub 2023-11-28T22:10:26.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.470+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcd5806af60 msgr2=0x7fcd5806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.470+0000 7fcd7c97c640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcd5806af60 0x7fcd5806d420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcd68034000 tx=0x7fcd6802f000 comp rx=0 tx=0).stop 2023-11-28T22:10:26.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.470+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd7815b290 msgr2=0x7fcd780c6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.470+0000 7fcd7c97c640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd7815b290 0x7fcd780c6280 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fcd70007fd0 tx=0x7fcd7000c9b0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.471+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 shutdown_connections 2023-11-28T22:10:26.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.471+0000 7fcd7c97c640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd7815b290 0x7fcd780c6280 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.471+0000 7fcd7c97c640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fcd5806af60 0x7fcd5806d420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.471+0000 7fcd7c97c640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd780cf8e0 0x7fcd780c8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.471+0000 7fcd7c97c640 1 --2- 172.21.15.45:0/3198115511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd780cec80 0x7fcd78154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.471+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 >> 172.21.15.45:0/3198115511 conn(0x7fcd780d0dc0 msgr2=0x7fcd780afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:26.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.471+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 shutdown_connections 2023-11-28T22:10:26.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.471+0000 7fcd7c97c640 1 -- 172.21.15.45:0/3198115511 wait complete. 2023-11-28T22:10:26.490 INFO:tasks.ceph:Scrubbing osd.3 2023-11-28T22:10:26.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-11-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.709+0000 7f19bf59e640 1 Processor -- start 2023-11-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.709+0000 7f19bf59e640 1 -- start start 2023-11-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.711+0000 7f19bf59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f19c015d3b0 0x7f19c00d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.711+0000 7f19bf59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f19c00d26e0 0x7f19c00cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.711+0000 7f19bf59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c00ce7f0 con 0x7f19c00ce340 2023-11-28T22:10:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.711+0000 7f19bf59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c015d790 con 0x7f19c015d3b0 2023-11-28T22:10:26.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.711+0000 7f19bf59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c00d2bf0 con 0x7f19c00d26e0 2023-11-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.711+0000 7f19bed9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f19c00ce340 0x7f19c00cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37318/0 (socket says 172.21.15.45:37318) 2023-11-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.711+0000 7f19bed9d640 1 -- 172.21.15.45:0/174795592 learned_addr learned my addr 172.21.15.45:0/174795592 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.711+0000 7f19be59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f19c015d3b0 0x7f19c00d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.712+0000 7f19bdd9b640 1 --2- 172.21.15.45:0/174795592 >> v2:172.21.15.190:3300/0 conn(0x7f19c00d26e0 0x7f19c00cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.712+0000 7f19bd59a640 1 -- 172.21.15.45:0/174795592 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390397916 0 0) 0x7f19c00ce7f0 con 0x7f19c00ce340 2023-11-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.712+0000 7f19be59c640 1 -- 172.21.15.45:0/174795592 >> v2:172.21.15.190:3300/0 conn(0x7f19c00d26e0 msgr2=0x7f19c00cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.712+0000 7f19be59c640 1 --2- 172.21.15.45:0/174795592 >> v2:172.21.15.190:3300/0 conn(0x7f19c00d26e0 0x7f19c00cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.712+0000 7f19be59c640 1 -- 172.21.15.45:0/174795592 >> v1:172.21.15.45:6789/0 conn(0x7f19c00ce340 legacy=0x7f19c00cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.712+0000 7f19be59c640 1 -- 172.21.15.45:0/174795592 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c00d2ed0 con 0x7f19c015d3b0 2023-11-28T22:10:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.713+0000 7f19be59c640 1 --2- 172.21.15.45:0/174795592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c015d3b0 0x7f19c00d21a0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f19c00b5a60 tx=0x7f19b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a644fe8f5f7701 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.713+0000 7f19bd59a640 1 -- 172.21.15.45:0/174795592 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0031830 con 0x7f19c015d3b0 2023-11-28T22:10:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bd59a640 1 -- 172.21.15.45:0/174795592 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b0031990 con 0x7f19c015d3b0 2023-11-28T22:10:26.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bd59a640 1 -- 172.21.15.45:0/174795592 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0031d20 con 0x7f19c015d3b0 2023-11-28T22:10:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bf59e640 1 -- 172.21.15.45:0/174795592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c015d3b0 msgr2=0x7f19c00d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bf59e640 1 --2- 172.21.15.45:0/174795592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c015d3b0 0x7f19c00d21a0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f19c00b5a60 tx=0x7f19b002b8d0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bf59e640 1 -- 172.21.15.45:0/174795592 shutdown_connections 2023-11-28T22:10:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bf59e640 1 --2- 172.21.15.45:0/174795592 >> v2:172.21.15.190:3300/0 conn(0x7f19c00d26e0 0x7f19c00cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bf59e640 1 --2- 172.21.15.45:0/174795592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c015d3b0 0x7f19c00d21a0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bf59e640 1 -- 172.21.15.45:0/174795592 >> 172.21.15.45:0/174795592 conn(0x7f19c0070f40 msgr2=0x7f19c0071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:26.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bf59e640 1 -- 172.21.15.45:0/174795592 shutdown_connections 2023-11-28T22:10:26.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.714+0000 7f19bf59e640 1 -- 172.21.15.45:0/174795592 wait complete. 2023-11-28T22:10:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.716+0000 7f19bf59e640 1 Processor -- start 2023-11-28T22:10:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.716+0000 7f19bf59e640 1 -- start start 2023-11-28T22:10:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.717+0000 7f19bf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c00ce340 0x7f19c01601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.717+0000 7f19bf59e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19c00d26e0 0x7f19c014a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.717+0000 7f19bf59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c015d3b0 0x7f19c00b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.717+0000 7f19bf59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00c8df0 con 0x7f19c00ce340 2023-11-28T22:10:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.717+0000 7f19bf59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c015fd30 con 0x7f19c015d3b0 2023-11-28T22:10:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.717+0000 7f19bf59e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c015f120 con 0x7f19c00d26e0 2023-11-28T22:10:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.718+0000 7f19be59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c00ce340 0x7f19c01601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.718+0000 7f19be59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c00ce340 0x7f19c01601c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43922/0 (socket says 172.21.15.45:43922) 2023-11-28T22:10:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.718+0000 7f19be59c640 1 -- 172.21.15.45:0/3889057825 learned_addr learned my addr 172.21.15.45:0/3889057825 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:26.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.718+0000 7f19be59c640 1 -- 172.21.15.45:0/3889057825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19c00d26e0 msgr2=0x7f19c014a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.718+0000 7f19be59c640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19c00d26e0 0x7f19c014a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.718+0000 7f19be59c640 1 -- 172.21.15.45:0/3889057825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c015d3b0 msgr2=0x7f19c00b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.718+0000 7f19be59c640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19c015d3b0 0x7f19c00b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.718+0000 7f19be59c640 1 -- 172.21.15.45:0/3889057825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b0009400 con 0x7f19c00ce340 2023-11-28T22:10:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.718+0000 7f19be59c640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c00ce340 0x7f19c01601c0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f19b0000c00 tx=0x7f19b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.719+0000 7f19aeffd640 1 -- 172.21.15.45:0/3889057825 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b003aa60 con 0x7f19c00ce340 2023-11-28T22:10:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.719+0000 7f19aeffd640 1 -- 172.21.15.45:0/3889057825 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b003abc0 con 0x7f19c00ce340 2023-11-28T22:10:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.719+0000 7f19aeffd640 1 -- 172.21.15.45:0/3889057825 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b00443f0 con 0x7f19c00ce340 2023-11-28T22:10:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.719+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c015f400 con 0x7f19c00ce340 2023-11-28T22:10:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.719+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c015f9c0 con 0x7f19c00ce340 2023-11-28T22:10:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.720+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f19c00d3770 con 0x7f19c00ce340 2023-11-28T22:10:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.721+0000 7f19aeffd640 1 -- 172.21.15.45:0/3889057825 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f19b0044550 con 0x7f19c00ce340 2023-11-28T22:10:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.722+0000 7f19aeffd640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f19a006af60 0x7f19a006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.722+0000 7f19aeffd640 1 -- 172.21.15.45:0/3889057825 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f19b00abef0 con 0x7f19c00ce340 2023-11-28T22:10:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.722+0000 7f19aeffd640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f19a00718f0 0x7f19a0073d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.722+0000 7f19aeffd640 1 -- 172.21.15.45:0/3889057825 --> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19a0074420 con 0x7f19a00718f0 2023-11-28T22:10:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.722+0000 7f19aeffd640 1 -- 172.21.15.45:0/3889057825 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=222) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19b00ae4d0 con 0x7f19c00ce340 2023-11-28T22:10:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.723+0000 7f19bed9d640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f19a00718f0 0x7f19a0073d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.723+0000 7f19bdd9b640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f19a006af60 0x7f19a006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.723+0000 7f19bed9d640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f19a00718f0 0x7f19a0073d50 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.724+0000 7f19bdd9b640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f19a006af60 0x7f19a006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f19c00b3d10 tx=0x7f19b4008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.724+0000 7f19aeffd640 1 -- 172.21.15.45:0/3889057825 <== osd.3 v2:172.21.15.45:6808/1024370250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f19a0074420 con 0x7f19a00718f0 2023-11-28T22:10:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.739+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 --> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f19c00d3490 con 0x7f19a00718f0 2023-11-28T22:10:26.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.739+0000 7f19aeffd640 1 -- 172.21.15.45:0/3889057825 <== osd.3 v2:172.21.15.45:6808/1024370250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f19b8001e10 con 0x7f19a00718f0 2023-11-28T22:10:26.741 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:10:26.741 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-28T22:10:26.741 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:10:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.739+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f19a00718f0 msgr2=0x7f19a0073d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.739+0000 7f19bf59e640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.45:6808/1024370250,v1:172.21.15.45:6809/1024370250] conn(0x7f19a00718f0 0x7f19a0073d50 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.740+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f19a006af60 msgr2=0x7f19a006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.740+0000 7f19bf59e640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f19a006af60 0x7f19a006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f19c00b3d10 tx=0x7f19b4008040 comp rx=0 tx=0).stop 2023-11-28T22:10:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.740+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c00ce340 msgr2=0x7f19c01601c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.740+0000 7f19bf59e640 1 --2- 172.21.15.45:0/3889057825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19c00ce340 0x7f19c01601c0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f19b0000c00 tx=0x7f19b00046c0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.740+0000 7f19bed9d640 1 -- 172.21.15.45:0/3889057825 reap_dead start 2023-11-28T22:10:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.740+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 shutdown_connections 2023-11-28T22:10:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.740+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 >> 172.21.15.45:0/3889057825 conn(0x7f19c0070f40 msgr2=0x7f19c00cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:26.743 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.740+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 shutdown_connections 2023-11-28T22:10:26.744 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.740+0000 7f19bf59e640 1 -- 172.21.15.45:0/3889057825 wait complete. 2023-11-28T22:10:26.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-11-28T22:10:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.962+0000 7fb34359e640 1 Processor -- start 2023-11-28T22:10:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.962+0000 7fb34359e640 1 -- start start 2023-11-28T22:10:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.963+0000 7fb34359e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb3441596f0 0x7fb3440b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.963+0000 7fb34359e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440ccc10 con 0x7fb344071ce0 2023-11-28T22:10:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.963+0000 7fb34359e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440b34c0 con 0x7fb344071850 2023-11-28T22:10:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.963+0000 7fb34359e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3440ccf80 con 0x7fb3441596f0 2023-11-28T22:10:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.963+0000 7fb33bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb344071ce0 0x7fb344071140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37332/0 (socket says 172.21.15.45:37332) 2023-11-28T22:10:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb33bfff640 1 -- 172.21.15.45:0/2334247588 learned_addr learned my addr 172.21.15.45:0/2334247588 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb34259c640 1 --2- 172.21.15.45:0/2334247588 >> v2:172.21.15.190:3300/0 conn(0x7fb3441596f0 0x7fb3440b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957556005 0 0) 0x7fb3440ccc10 con 0x7fb344071ce0 2023-11-28T22:10:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb330003420 con 0x7fb344071ce0 2023-11-28T22:10:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266855082 0 0) 0x7fb330003420 con 0x7fb344071ce0 2023-11-28T22:10:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 >> v2:172.21.15.190:3300/0 conn(0x7fb3441596f0 msgr2=0x7fb3440b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 --2- 172.21.15.45:0/2334247588 >> v2:172.21.15.190:3300/0 conn(0x7fb3441596f0 0x7fb3440b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 >> v1:172.21.15.190:6789/0 conn(0x7fb344071850 legacy=0x7fb3440cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb344159ad0 con 0x7fb344071ce0 2023-11-28T22:10:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb32c001670 con 0x7fb344071ce0 2023-11-28T22:10:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 >> v1:172.21.15.45:6789/0 conn(0x7fb344071ce0 legacy=0x7fb344071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.964+0000 7fb341d9b640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb330004a50 0x7fb330004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb341d9b640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb330003660 0x7fb330003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb341d9b640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330004270 0x7fb33000fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440ccf80 con 0x7fb330003660 2023-11-28T22:10:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3300104d0 con 0x7fb330004270 2023-11-28T22:10:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb330010690 con 0x7fb330004a50 2023-11-28T22:10:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb33bfff640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb330003660 0x7fb330003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb34259c640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb330004a50 0x7fb330004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb342d9d640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330004270 0x7fb33000fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb33bfff640 1 -- 172.21.15.45:0/2334247588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb330004a50 msgr2=0x7fb330004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb33bfff640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb330004a50 0x7fb330004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb33bfff640 1 -- 172.21.15.45:0/2334247588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330004270 msgr2=0x7fb33000fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb33bfff640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330004270 0x7fb33000fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb33bfff640 1 -- 172.21.15.45:0/2334247588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb330003420 con 0x7fb330003660 2023-11-28T22:10:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.965+0000 7fb342d9d640 1 -- 172.21.15.45:0/2334247588 reap_dead start 2023-11-28T22:10:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.967+0000 7fb33bfff640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb330003660 0x7fb330003ae0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00c210 tx=0x7fb3300046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4cd8a277128699 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.967+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c005ad0 con 0x7fb330003660 2023-11-28T22:10:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.967+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c005c30 con 0x7fb330003660 2023-11-28T22:10:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.967+0000 7fb341d9b640 1 -- 172.21.15.45:0/2334247588 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c02cad0 con 0x7fb330003660 2023-11-28T22:10:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.967+0000 7fb34359e640 1 -- 172.21.15.45:0/2334247588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb330003660 msgr2=0x7fb330003ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.967+0000 7fb34359e640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb330003660 0x7fb330003ae0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00c210 tx=0x7fb3300046e0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.968+0000 7fb34359e640 1 -- 172.21.15.45:0/2334247588 shutdown_connections 2023-11-28T22:10:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.968+0000 7fb34359e640 1 --2- 172.21.15.45:0/2334247588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb330003660 0x7fb330003ae0 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.968+0000 7fb34359e640 1 -- 172.21.15.45:0/2334247588 >> 172.21.15.45:0/2334247588 conn(0x7fb3440b0e70 msgr2=0x7fb3440b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.968+0000 7fb34359e640 1 -- 172.21.15.45:0/2334247588 shutdown_connections 2023-11-28T22:10:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.968+0000 7fb34359e640 1 -- 172.21.15.45:0/2334247588 wait complete. 2023-11-28T22:10:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.970+0000 7fb34359e640 1 Processor -- start 2023-11-28T22:10:26.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.970+0000 7fb34359e640 1 -- start start 2023-11-28T22:10:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.971+0000 7fb34359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330003660 0x7fb3441516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.971+0000 7fb34359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb344071240 0x7fb3441ee450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.971+0000 7fb34359e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb344152fd0 0x7fb344153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.971+0000 7fb34359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3441ee990 con 0x7fb344071240 2023-11-28T22:10:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.971+0000 7fb34359e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34406d350 con 0x7fb330003660 2023-11-28T22:10:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.971+0000 7fb34359e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34406d630 con 0x7fb344152fd0 2023-11-28T22:10:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.971+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb344071240 0x7fb3441ee450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.971+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb344071240 0x7fb3441ee450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43934/0 (socket says 172.21.15.45:43934) 2023-11-28T22:10:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.971+0000 7fb33bfff640 1 -- 172.21.15.45:0/2784164569 learned_addr learned my addr 172.21.15.45:0/2784164569 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb34259c640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330003660 0x7fb3441516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb342d9d640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb344152fd0 0x7fb344153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb33bfff640 1 -- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb344152fd0 msgr2=0x7fb344153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb33bfff640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb344152fd0 0x7fb344153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb33bfff640 1 -- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330003660 msgr2=0x7fb3441516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb33bfff640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330003660 0x7fb3441516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb33bfff640 1 -- 172.21.15.45:0/2784164569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb344150ed0 con 0x7fb344071240 2023-11-28T22:10:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb34259c640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330003660 0x7fb3441516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb33bfff640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb344071240 0x7fb3441ee450 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fb32c001930 tx=0x7fb32c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb33affd640 1 -- 172.21.15.45:0/2784164569 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c03c040 con 0x7fb344071240 2023-11-28T22:10:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb33affd640 1 -- 172.21.15.45:0/2784164569 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c02cca0 con 0x7fb344071240 2023-11-28T22:10:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb33affd640 1 -- 172.21.15.45:0/2784164569 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c007290 con 0x7fb344071240 2023-11-28T22:10:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.972+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3441511b0 con 0x7fb344071240 2023-11-28T22:10:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.973+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34406d000 con 0x7fb344071240 2023-11-28T22:10:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.974+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb344151340 con 0x7fb344071240 2023-11-28T22:10:26.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.975+0000 7fb33affd640 1 -- 172.21.15.45:0/2784164569 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb32c032070 con 0x7fb344071240 2023-11-28T22:10:26.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.975+0000 7fb33affd640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb32406af90 0x7fb32406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:26.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.976+0000 7fb33affd640 1 -- 172.21.15.45:0/2784164569 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fb32c0aade0 con 0x7fb344071240 2023-11-28T22:10:26.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.976+0000 7fb34259c640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb32406af90 0x7fb32406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:26.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.977+0000 7fb34259c640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb32406af90 0x7fb32406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb334009a80 tx=0x7fb334008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:26.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:26.980+0000 7fb33affd640 1 -- 172.21.15.45:0/2784164569 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb32c075b70 con 0x7fb344071240 2023-11-28T22:10:27.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.179+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb34406d7a0 con 0x7fb32406af90 2023-11-28T22:10:27.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.180+0000 7fb33affd640 1 -- 172.21.15.45:0/2784164569 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb34406d7a0 con 0x7fb32406af90 2023-11-28T22:10:27.181 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 3 to deep-scrub 2023-11-28T22:10:27.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb32406af90 msgr2=0x7fb32406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb32406af90 0x7fb32406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb334009a80 tx=0x7fb334008040 comp rx=0 tx=0).stop 2023-11-28T22:10:27.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb344071240 msgr2=0x7fb3441ee450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb344071240 0x7fb3441ee450 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fb32c001930 tx=0x7fb32c02e070 comp rx=0 tx=0).stop 2023-11-28T22:10:27.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 shutdown_connections 2023-11-28T22:10:27.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb344152fd0 0x7fb344153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb344071240 0x7fb3441ee450 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb32406af90 0x7fb32406d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.185 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 --2- 172.21.15.45:0/2784164569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb330003660 0x7fb3441516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 >> 172.21.15.45:0/2784164569 conn(0x7fb3440b0e70 msgr2=0x7fb3440afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:27.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 shutdown_connections 2023-11-28T22:10:27.186 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.183+0000 7fb34359e640 1 -- 172.21.15.45:0/2784164569 wait complete. 2023-11-28T22:10:27.203 INFO:tasks.ceph:Scrubbing osd.4 2023-11-28T22:10:27.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-11-28T22:10:27.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.438+0000 7fba01c52640 1 Processor -- start 2023-11-28T22:10:27.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.438+0000 7fba01c52640 1 -- start start 2023-11-28T22:10:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.439+0000 7fba01c52640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9fc06b4d0 0x7fb9fc0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.439+0000 7fba01c52640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc0d1fc0 con 0x7fb9fc099a40 2023-11-28T22:10:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.439+0000 7fba01c52640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc06a910 con 0x7fb9fc0aff90 2023-11-28T22:10:27.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.439+0000 7fba01c52640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc0cb8a0 con 0x7fb9fc06b4d0 2023-11-28T22:10:27.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.440+0000 7fb9fb7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb9fc099a40 0x7fb9fc099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37346/0 (socket says 172.21.15.45:37346) 2023-11-28T22:10:27.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.440+0000 7fb9fb7fe640 1 -- 172.21.15.45:0/2116525275 learned_addr learned my addr 172.21.15.45:0/2116525275 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:27.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.440+0000 7fb9fbfff640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9fc06b4d0 0x7fb9fc0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.440+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990950705 0 0) 0x7fb9fc0d1fc0 con 0x7fb9fc099a40 2023-11-28T22:10:27.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e8003520 con 0x7fb9fc099a40 2023-11-28T22:10:27.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822861986 0 0) 0x7fb9e8003520 con 0x7fb9fc099a40 2023-11-28T22:10:27.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9fc06b4d0 msgr2=0x7fb9fc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:27.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9fc06b4d0 0x7fb9fc0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 >> v1:172.21.15.190:6789/0 conn(0x7fb9fc0aff90 legacy=0x7fb9fc0cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc06a630 con 0x7fb9fc099a40 2023-11-28T22:10:27.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fb9ec001670 con 0x7fb9fc099a40 2023-11-28T22:10:27.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 >> v1:172.21.15.45:6789/0 conn(0x7fb9fc099a40 legacy=0x7fb9fc099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9fc06b4d0 0x7fb9e80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8003f70 0x7fb9e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9e8004d00 0x7fb9e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc0cb8a0 con 0x7fb9e8003f70 2023-11-28T22:10:27.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e800bb70 con 0x7fb9e8004d00 2023-11-28T22:10:27.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.441+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e800bd30 con 0x7fb9fc06b4d0 2023-11-28T22:10:27.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.442+0000 7fb9faffd640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8003f70 0x7fb9e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.442+0000 7fb9fb7fe640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9fc06b4d0 0x7fb9e80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.442+0000 7fb9fbfff640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9e8004d00 0x7fb9e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.442+0000 7fb9faffd640 1 -- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9fc06b4d0 msgr2=0x7fb9e80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.442+0000 7fb9faffd640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9fc06b4d0 0x7fb9e80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.442+0000 7fb9faffd640 1 -- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9e8004d00 msgr2=0x7fb9e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:27.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.442+0000 7fb9faffd640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9e8004d00 0x7fb9e8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.442+0000 7fb9faffd640 1 -- 172.21.15.45:0/2116525275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e8003520 con 0x7fb9e8003f70 2023-11-28T22:10:27.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.444+0000 7fb9faffd640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8003f70 0x7fb9e80047c0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4001e50 tx=0x7fb9e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922aaaed95cf52a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:27.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.444+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4003b70 con 0x7fb9e8003f70 2023-11-28T22:10:27.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.444+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e4003cd0 con 0x7fb9e8003f70 2023-11-28T22:10:27.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.444+0000 7fb9fa7fc640 1 -- 172.21.15.45:0/2116525275 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4005310 con 0x7fb9e8003f70 2023-11-28T22:10:27.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.445+0000 7fba01c52640 1 -- 172.21.15.45:0/2116525275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8003f70 msgr2=0x7fb9e80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.445+0000 7fba01c52640 1 --2- 172.21.15.45:0/2116525275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e8003f70 0x7fb9e80047c0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4001e50 tx=0x7fb9e80057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.445+0000 7fb9fbfff640 1 -- 172.21.15.45:0/2116525275 reap_dead start 2023-11-28T22:10:27.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.445+0000 7fba01c52640 1 -- 172.21.15.45:0/2116525275 shutdown_connections 2023-11-28T22:10:27.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.445+0000 7fba01c52640 1 -- 172.21.15.45:0/2116525275 >> 172.21.15.45:0/2116525275 conn(0x7fb9fc0d1020 msgr2=0x7fb9fc0d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:27.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.445+0000 7fba01c52640 1 -- 172.21.15.45:0/2116525275 shutdown_connections 2023-11-28T22:10:27.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.445+0000 7fba01c52640 1 -- 172.21.15.45:0/2116525275 wait complete. 2023-11-28T22:10:27.451 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.447+0000 7fba01c52640 1 Processor -- start 2023-11-28T22:10:27.452 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.447+0000 7fba01c52640 1 -- start start 2023-11-28T22:10:27.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.448+0000 7fba01c52640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9e8003f70 0x7fb9fc1f2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.448+0000 7fba01c52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc15e970 0x7fb9fc1522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.448+0000 7fba01c52640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9fc06e710 0x7fb9fc06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.448+0000 7fba01c52640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc06f0d0 con 0x7fb9fc15e970 2023-11-28T22:10:27.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.448+0000 7fba01c52640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc0b4d50 con 0x7fb9fc06e710 2023-11-28T22:10:27.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.448+0000 7fba01c52640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc1f2a80 con 0x7fb9e8003f70 2023-11-28T22:10:27.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9faffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc15e970 0x7fb9fc1522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9faffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc15e970 0x7fb9fc1522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43946/0 (socket says 172.21.15.45:43946) 2023-11-28T22:10:27.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9faffd640 1 -- 172.21.15.45:0/3901278521 learned_addr learned my addr 172.21.15.45:0/3901278521 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:27.454 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9fbfff640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9fc06e710 0x7fb9fc06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9fb7fe640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9e8003f70 0x7fb9fc1f2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9faffd640 1 -- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9e8003f70 msgr2=0x7fb9fc1f2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9faffd640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9e8003f70 0x7fb9fc1f2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9faffd640 1 -- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9fc06e710 msgr2=0x7fb9fc06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9faffd640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9fc06e710 0x7fb9fc06eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9faffd640 1 -- 172.21.15.45:0/3901278521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc15eec0 con 0x7fb9fc15e970 2023-11-28T22:10:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.449+0000 7fb9fbfff640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb9fc06e710 0x7fb9fc06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:27.455 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.450+0000 7fb9faffd640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc15e970 0x7fb9fc1522c0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4001f70 tx=0x7fb9e40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:27.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.450+0000 7fb9f3fff640 1 -- 172.21.15.45:0/3901278521 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e403e070 con 0x7fb9fc15e970 2023-11-28T22:10:27.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.450+0000 7fb9f3fff640 1 -- 172.21.15.45:0/3901278521 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e4006b50 con 0x7fb9fc15e970 2023-11-28T22:10:27.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.450+0000 7fb9f3fff640 1 -- 172.21.15.45:0/3901278521 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4007030 con 0x7fb9fc15e970 2023-11-28T22:10:27.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.450+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9fc15f1a0 con 0x7fb9fc15e970 2023-11-28T22:10:27.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.450+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9fc15f760 con 0x7fb9fc15e970 2023-11-28T22:10:27.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.451+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb9fc0af970 con 0x7fb9fc15e970 2023-11-28T22:10:27.456 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.452+0000 7fb9f3fff640 1 -- 172.21.15.45:0/3901278521 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb9e4007190 con 0x7fb9fc15e970 2023-11-28T22:10:27.457 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.453+0000 7fb9f3fff640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb9dc06af60 0x7fb9dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.453+0000 7fb9f3fff640 1 -- 172.21.15.45:0/3901278521 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fb9e40ab5b0 con 0x7fb9fc15e970 2023-11-28T22:10:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.453+0000 7fb9f3fff640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fb9dc0718f0 0x7fb9dc073d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.453+0000 7fb9f3fff640 1 -- 172.21.15.45:0/3901278521 --> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9dc074420 con 0x7fb9dc0718f0 2023-11-28T22:10:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.453+0000 7fb9f3fff640 1 -- 172.21.15.45:0/3901278521 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=222) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9e40adb90 con 0x7fb9fc15e970 2023-11-28T22:10:27.458 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.454+0000 7fb9fb7fe640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb9dc06af60 0x7fb9dc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.454+0000 7fb9fbfff640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fb9dc0718f0 0x7fb9dc073d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.455+0000 7fb9fb7fe640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb9dc06af60 0x7fb9dc06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec001eb0 tx=0x7fb9ec009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:27.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.456+0000 7fb9fbfff640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fb9dc0718f0 0x7fb9dc073d50 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:27.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.458+0000 7fb9f3fff640 1 -- 172.21.15.45:0/3901278521 <== osd.4 v2:172.21.15.190:6800/1794504124 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb9dc074420 con 0x7fb9dc0718f0 2023-11-28T22:10:27.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.473+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 --> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb9fc15d7a0 con 0x7fb9dc0718f0 2023-11-28T22:10:27.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.474+0000 7fb9f3fff640 1 -- 172.21.15.45:0/3901278521 <== osd.4 v2:172.21.15.190:6800/1794504124 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb9f4005990 con 0x7fb9dc0718f0 2023-11-28T22:10:27.475 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:10:27.475 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-28T22:10:27.475 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:10:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.474+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fb9dc0718f0 msgr2=0x7fb9dc073d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.474+0000 7fba01c52640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6800/1794504124,v1:172.21.15.190:6801/1794504124] conn(0x7fb9dc0718f0 0x7fb9dc073d50 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.475+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb9dc06af60 msgr2=0x7fb9dc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.475+0000 7fba01c52640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb9dc06af60 0x7fb9dc06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec001eb0 tx=0x7fb9ec009040 comp rx=0 tx=0).stop 2023-11-28T22:10:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.475+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc15e970 msgr2=0x7fb9fc1522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.475+0000 7fba01c52640 1 --2- 172.21.15.45:0/3901278521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc15e970 0x7fb9fc1522c0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4001f70 tx=0x7fb9e40014a0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.475+0000 7fb9fbfff640 1 -- 172.21.15.45:0/3901278521 reap_dead start 2023-11-28T22:10:27.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.475+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 shutdown_connections 2023-11-28T22:10:27.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.475+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 >> 172.21.15.45:0/3901278521 conn(0x7fb9fc0d1020 msgr2=0x7fb9fc0b2380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:27.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.475+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 shutdown_connections 2023-11-28T22:10:27.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.475+0000 7fba01c52640 1 -- 172.21.15.45:0/3901278521 wait complete. 2023-11-28T22:10:27.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-11-28T22:10:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.721+0000 7f4df0de2640 1 Processor -- start 2023-11-28T22:10:27.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.721+0000 7f4df0de2640 1 -- start start 2023-11-28T22:10:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.722+0000 7f4df0de2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0d0190 0x7f4dec167f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.722+0000 7f4df0de2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dec0d2810 con 0x7f4dec15d140 2023-11-28T22:10:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.722+0000 7f4df0de2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dec0b71b0 con 0x7f4dec15d520 2023-11-28T22:10:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.722+0000 7f4df0de2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec0d1c00 con 0x7f4dec0d0190 2023-11-28T22:10:27.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.722+0000 7f4deb7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4dec15d140 0x7f4dec0d1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37350/0 (socket says 172.21.15.45:37350) 2023-11-28T22:10:27.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.722+0000 7f4deb7fe640 1 -- 172.21.15.45:0/123344305 learned_addr learned my addr 172.21.15.45:0/123344305 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4debfff640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0d0190 0x7f4dec167f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750416544 0 0) 0x7f4dec0d2810 con 0x7f4dec15d140 2023-11-28T22:10:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd8003520 con 0x7f4dec15d140 2023-11-28T22:10:27.728 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2770760335 0 0) 0x7f4dd8003520 con 0x7f4dec15d140 2023-11-28T22:10:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0d0190 msgr2=0x7f4dec167f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0d0190 0x7f4dec167f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 >> v1:172.21.15.190:6789/0 conn(0x7f4dec15d520 legacy=0x7f4dec0cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec0d1ee0 con 0x7f4dec15d140 2023-11-28T22:10:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f4ddc0019f0 con 0x7f4dec15d140 2023-11-28T22:10:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 >> v1:172.21.15.45:6789/0 conn(0x7f4dec15d140 legacy=0x7f4dec0d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80039a0 0x7f4dd8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.723+0000 7f4dea7fc640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0d0190 0x7f4dd80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4dea7fc640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd8004d00 0x7f4dd8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec0d1c00 con 0x7f4dd80039a0 2023-11-28T22:10:27.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd800baf0 con 0x7f4dd8004d00 2023-11-28T22:10:27.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd800bcb0 con 0x7f4dec0d0190 2023-11-28T22:10:27.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4deb7fe640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80039a0 0x7f4dd8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4debfff640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd8004d00 0x7f4dd8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4deb7fe640 1 -- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0d0190 msgr2=0x7f4dd80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:27.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4deaffd640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0d0190 0x7f4dd80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4deb7fe640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0d0190 0x7f4dd80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4deb7fe640 1 -- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd8004d00 msgr2=0x7f4dd8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:27.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4deb7fe640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd8004d00 0x7f4dd8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4deb7fe640 1 -- 172.21.15.45:0/123344305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd8003520 con 0x7f4dd80039a0 2023-11-28T22:10:27.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.724+0000 7f4debfff640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd8004d00 0x7f4dd8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:27.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.725+0000 7f4deb7fe640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80039a0 0x7f4dd8003dc0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00a800 tx=0x7f4dd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a34e48e84415600 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:27.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.726+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc0072d0 con 0x7f4dd80039a0 2023-11-28T22:10:27.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.726+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc02f070 con 0x7f4dd80039a0 2023-11-28T22:10:27.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.726+0000 7f4dea7fc640 1 -- 172.21.15.45:0/123344305 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc007690 con 0x7f4dd80039a0 2023-11-28T22:10:27.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.726+0000 7f4df0de2640 1 -- 172.21.15.45:0/123344305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80039a0 msgr2=0x7f4dd8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.726+0000 7f4df0de2640 1 --2- 172.21.15.45:0/123344305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dd80039a0 0x7f4dd8003dc0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00a800 tx=0x7f4dd80057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.726+0000 7f4debfff640 1 -- 172.21.15.45:0/123344305 reap_dead start 2023-11-28T22:10:27.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.726+0000 7f4df0de2640 1 -- 172.21.15.45:0/123344305 shutdown_connections 2023-11-28T22:10:27.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.726+0000 7f4df0de2640 1 -- 172.21.15.45:0/123344305 >> 172.21.15.45:0/123344305 conn(0x7f4dec06a500 msgr2=0x7f4dec06a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:27.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.727+0000 7f4df0de2640 1 -- 172.21.15.45:0/123344305 shutdown_connections 2023-11-28T22:10:27.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.727+0000 7f4df0de2640 1 -- 172.21.15.45:0/123344305 wait complete. 2023-11-28T22:10:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.729+0000 7f4df0de2640 1 Processor -- start 2023-11-28T22:10:27.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.729+0000 7f4df0de2640 1 -- start start 2023-11-28T22:10:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4df0de2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd80039a0 0x7f4dec15e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4df0de2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0b25b0 0x7f4dec0cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4df0de2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dec157710 0x7f4dec0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4df0de2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec148650 con 0x7f4dec157710 2023-11-28T22:10:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4df0de2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec15e9d0 con 0x7f4dd80039a0 2023-11-28T22:10:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4df0de2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec15ecb0 con 0x7f4dec0b25b0 2023-11-28T22:10:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4debfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dec157710 0x7f4dec0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4debfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dec157710 0x7f4dec0ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43968/0 (socket says 172.21.15.45:43968) 2023-11-28T22:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4debfff640 1 -- 172.21.15.45:0/1068541185 learned_addr learned my addr 172.21.15.45:0/1068541185 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.730+0000 7f4deb7fe640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd80039a0 0x7f4dec15e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4debfff640 1 -- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0b25b0 msgr2=0x7f4dec0cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4deaffd640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0b25b0 0x7f4dec0cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4debfff640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0b25b0 0x7f4dec0cc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4debfff640 1 -- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd80039a0 msgr2=0x7f4dec15e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4debfff640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd80039a0 0x7f4dec15e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4debfff640 1 -- 172.21.15.45:0/1068541185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec15ef90 con 0x7f4dec157710 2023-11-28T22:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4deb7fe640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd80039a0 0x7f4dec15e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4debfff640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dec157710 0x7f4dec0ccbb0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0b5c00 tx=0x7f4de4010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4de3fff640 1 -- 172.21.15.45:0/1068541185 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de400e8a0 con 0x7f4dec157710 2023-11-28T22:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4de3fff640 1 -- 172.21.15.45:0/1068541185 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de400ea00 con 0x7f4dec157710 2023-11-28T22:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dec15f270 con 0x7f4dec157710 2023-11-28T22:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4de3fff640 1 -- 172.21.15.45:0/1068541185 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de400ed90 con 0x7f4dec157710 2023-11-28T22:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.731+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dec14a680 con 0x7f4dec157710 2023-11-28T22:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.733+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dec15c520 con 0x7f4dec157710 2023-11-28T22:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.734+0000 7f4de3fff640 1 -- 172.21.15.45:0/1068541185 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4de4020410 con 0x7f4dec157710 2023-11-28T22:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.735+0000 7f4de3fff640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f4dcc06af90 0x7f4dcc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.735+0000 7f4de3fff640 1 -- 172.21.15.45:0/1068541185 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f4de4089100 con 0x7f4dec157710 2023-11-28T22:10:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.735+0000 7f4deb7fe640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f4dcc06af90 0x7f4dcc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.737+0000 7f4deb7fe640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f4dcc06af90 0x7f4dcc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc034000 tx=0x7f4ddc007000 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.739+0000 7f4de3fff640 1 -- 172.21.15.45:0/1068541185 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4de4053f40 con 0x7f4dec157710 2023-11-28T22:10:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.931+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dec0b8f40 con 0x7f4dcc06af90 2023-11-28T22:10:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.932+0000 7f4de3fff640 1 -- 172.21.15.45:0/1068541185 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4ddc02ce90 con 0x7f4dcc06af90 2023-11-28T22:10:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 4 to deep-scrub 2023-11-28T22:10:27.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f4dcc06af90 msgr2=0x7f4dcc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f4dcc06af90 0x7f4dcc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc034000 tx=0x7f4ddc007000 comp rx=0 tx=0).stop 2023-11-28T22:10:27.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dec157710 msgr2=0x7f4dec0ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:27.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dec157710 0x7f4dec0ccbb0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0b5c00 tx=0x7f4de4010c20 comp rx=0 tx=0).stop 2023-11-28T22:10:27.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 shutdown_connections 2023-11-28T22:10:27.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4dec157710 0x7f4dec0ccbb0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4dec0b25b0 0x7f4dec0cc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f4dcc06af90 0x7f4dcc06d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 --2- 172.21.15.45:0/1068541185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dd80039a0 0x7f4dec15e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:27.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.934+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 >> 172.21.15.45:0/1068541185 conn(0x7f4dec06a500 msgr2=0x7f4dec0d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:27.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.935+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 shutdown_connections 2023-11-28T22:10:27.939 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:27.935+0000 7f4df0de2640 1 -- 172.21.15.45:0/1068541185 wait complete. 2023-11-28T22:10:27.956 INFO:tasks.ceph:Scrubbing osd.5 2023-11-28T22:10:27.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-11-28T22:10:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.185+0000 7f9ce9459640 1 Processor -- start 2023-11-28T22:10:28.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.186+0000 7f9ce9459640 1 -- start start 2023-11-28T22:10:28.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.187+0000 7f9ce9459640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ce406c6e0 0x7f9ce40ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.187+0000 7f9ce9459640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9ce4161ba0 0x7f9ce4161ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.187+0000 7f9ce9459640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce40cea90 con 0x7f9ce406c300 2023-11-28T22:10:28.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.187+0000 7f9ce9459640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce40cc800 con 0x7f9ce4161ba0 2023-11-28T22:10:28.189 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.187+0000 7f9ce9459640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40ccac0 con 0x7f9ce406c6e0 2023-11-28T22:10:28.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.188+0000 7f9ce8c58640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9ce4161ba0 0x7f9ce4161ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.188+0000 7f9ce8c58640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9ce4161ba0 0x7f9ce4161ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:51982/0 (socket says 172.21.15.45:51982) 2023-11-28T22:10:28.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.188+0000 7f9ce8c58640 1 -- 172.21.15.45:0/3767783823 learned_addr learned my addr 172.21.15.45:0/3767783823 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:28.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.188+0000 7f9ce37fe640 1 --2- 172.21.15.45:0/3767783823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ce406c6e0 0x7f9ce40ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.190 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.189+0000 7f9ce8c58640 1 -- 172.21.15.45:0/3767783823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ce406c6e0 msgr2=0x7f9ce40ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.189+0000 7f9ce8c58640 1 --2- 172.21.15.45:0/3767783823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ce406c6e0 0x7f9ce40ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.189+0000 7f9ce8c58640 1 -- 172.21.15.45:0/3767783823 >> v1:172.21.15.190:6789/0 conn(0x7f9ce406c300 legacy=0x7f9ce40cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.189+0000 7f9ce8c58640 1 -- 172.21.15.45:0/3767783823 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce40ccda0 con 0x7f9ce4161ba0 2023-11-28T22:10:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.190+0000 7f9ce8c58640 1 --2- 172.21.15.45:0/3767783823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ce4161ba0 0x7f9ce4161ff0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009870 tx=0x7f9cdc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb8b20d2680601e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.191+0000 7f9ce2ffd640 1 -- 172.21.15.45:0/3767783823 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc030840 con 0x7f9ce4161ba0 2023-11-28T22:10:28.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce2ffd640 1 -- 172.21.15.45:0/3767783823 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc0309a0 con 0x7f9ce4161ba0 2023-11-28T22:10:28.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce2ffd640 1 -- 172.21.15.45:0/3767783823 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc030d30 con 0x7f9ce4161ba0 2023-11-28T22:10:28.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce9459640 1 -- 172.21.15.45:0/3767783823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ce4161ba0 msgr2=0x7f9ce4161ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce9459640 1 --2- 172.21.15.45:0/3767783823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ce4161ba0 0x7f9ce4161ff0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009870 tx=0x7f9cdc02b980 comp rx=0 tx=0).stop 2023-11-28T22:10:28.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce9459640 1 -- 172.21.15.45:0/3767783823 shutdown_connections 2023-11-28T22:10:28.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce9459640 1 --2- 172.21.15.45:0/3767783823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ce4161ba0 0x7f9ce4161ff0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce9459640 1 --2- 172.21.15.45:0/3767783823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ce406c6e0 0x7f9ce40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce9459640 1 -- 172.21.15.45:0/3767783823 >> 172.21.15.45:0/3767783823 conn(0x7f9ce40cec30 msgr2=0x7f9ce40cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce9459640 1 -- 172.21.15.45:0/3767783823 shutdown_connections 2023-11-28T22:10:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.192+0000 7f9ce9459640 1 -- 172.21.15.45:0/3767783823 wait complete. 2023-11-28T22:10:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.194+0000 7f9ce9459640 1 Processor -- start 2023-11-28T22:10:28.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.194+0000 7f9ce9459640 1 -- start start 2023-11-28T22:10:28.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.195+0000 7f9ce9459640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ce406c300 0x7f9ce4164ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.195+0000 7f9ce9459640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ce406c6e0 0x7f9ce415b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.195+0000 7f9ce9459640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce4161ba0 0x7f9ce4146760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.195+0000 7f9ce9459640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4166120 con 0x7f9ce4161ba0 2023-11-28T22:10:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.195+0000 7f9ce9459640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce415ba30 con 0x7f9ce406c300 2023-11-28T22:10:28.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.195+0000 7f9ce9459640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce415bd10 con 0x7f9ce406c6e0 2023-11-28T22:10:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.196+0000 7f9ce8c58640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce4161ba0 0x7f9ce4146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.196+0000 7f9ce3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ce406c300 0x7f9ce4164ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.196+0000 7f9ce3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ce406c300 0x7f9ce4164ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.45:51990/0 (socket says 172.21.15.45:51990) 2023-11-28T22:10:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.196+0000 7f9ce3fff640 1 -- 172.21.15.45:0/3338183908 learned_addr learned my addr 172.21.15.45:0/3338183908 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.196+0000 7f9ce8c58640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce4161ba0 0x7f9ce4146760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43978/0 (socket says 172.21.15.45:43978) 2023-11-28T22:10:28.199 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.196+0000 7f9ce37fe640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ce406c6e0 0x7f9ce415b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce8c58640 1 -- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ce406c6e0 msgr2=0x7f9ce415b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce8c58640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ce406c6e0 0x7f9ce415b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce8c58640 1 -- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ce406c300 msgr2=0x7f9ce4164ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce8c58640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ce406c300 0x7f9ce4164ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce8c58640 1 -- 172.21.15.45:0/3338183908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc0094d0 con 0x7f9ce4161ba0 2023-11-28T22:10:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce8c58640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce4161ba0 0x7f9ce4146760 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc033000 tx=0x7f9cdc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce0ff9640 1 -- 172.21.15.45:0/3338183908 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc039c10 con 0x7f9ce4161ba0 2023-11-28T22:10:28.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce0ff9640 1 -- 172.21.15.45:0/3338183908 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc039d90 con 0x7f9ce4161ba0 2023-11-28T22:10:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce415bff0 con 0x7f9ce4161ba0 2023-11-28T22:10:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.197+0000 7f9ce0ff9640 1 -- 172.21.15.45:0/3338183908 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc043640 con 0x7f9ce4161ba0 2023-11-28T22:10:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.198+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce415c5b0 con 0x7f9ce4161ba0 2023-11-28T22:10:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.198+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ce40cd640 con 0x7f9ce4161ba0 2023-11-28T22:10:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.200+0000 7f9ce0ff9640 1 -- 172.21.15.45:0/3338183908 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9cdc0437a0 con 0x7f9ce4161ba0 2023-11-28T22:10:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.201+0000 7f9ce0ff9640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f9cd006af60 0x7f9cd006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.201+0000 7f9ce0ff9640 1 -- 172.21.15.45:0/3338183908 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f9cdc03e070 con 0x7f9ce4161ba0 2023-11-28T22:10:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.201+0000 7f9ce0ff9640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f9cd00718f0 0x7f9cd0073d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.201+0000 7f9ce0ff9640 1 -- 172.21.15.45:0/3338183908 --> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9cd0074420 con 0x7f9cd00718f0 2023-11-28T22:10:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.201+0000 7f9ce0ff9640 1 -- 172.21.15.45:0/3338183908 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=222) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cdc0ad490 con 0x7f9ce4161ba0 2023-11-28T22:10:28.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.202+0000 7f9ce3fff640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f9cd006af60 0x7f9cd006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.202+0000 7f9ce37fe640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f9cd00718f0 0x7f9cd0073d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.203+0000 7f9ce3fff640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f9cd006af60 0x7f9cd006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4007fd0 tx=0x7f9cd4009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.203+0000 7f9ce37fe640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f9cd00718f0 0x7f9cd0073d50 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.205+0000 7f9ce0ff9640 1 -- 172.21.15.45:0/3338183908 <== osd.5 v2:172.21.15.190:6808/3522971559 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9cd0074420 con 0x7f9cd00718f0 2023-11-28T22:10:28.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.220+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 --> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9ce40cd360 con 0x7f9cd00718f0 2023-11-28T22:10:28.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.221+0000 7f9ce0ff9640 1 -- 172.21.15.45:0/3338183908 <== osd.5 v2:172.21.15.190:6808/3522971559 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9cd8009d50 con 0x7f9cd00718f0 2023-11-28T22:10:28.223 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:10:28.223 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-28T22:10:28.223 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:10:28.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.221+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f9cd00718f0 msgr2=0x7f9cd0073d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.221+0000 7f9ce9459640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6808/3522971559,v1:172.21.15.190:6809/3522971559] conn(0x7f9cd00718f0 0x7f9cd0073d50 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.222+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f9cd006af60 msgr2=0x7f9cd006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.222+0000 7f9ce9459640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f9cd006af60 0x7f9cd006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4007fd0 tx=0x7f9cd4009040 comp rx=0 tx=0).stop 2023-11-28T22:10:28.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.222+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce4161ba0 msgr2=0x7f9ce4146760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.222+0000 7f9ce9459640 1 --2- 172.21.15.45:0/3338183908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ce4161ba0 0x7f9ce4146760 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc033000 tx=0x7f9cdc004730 comp rx=0 tx=0).stop 2023-11-28T22:10:28.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.222+0000 7f9ce8c58640 1 -- 172.21.15.45:0/3338183908 reap_dead start 2023-11-28T22:10:28.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.222+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 shutdown_connections 2023-11-28T22:10:28.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.222+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 >> 172.21.15.45:0/3338183908 conn(0x7f9ce40cec30 msgr2=0x7f9ce40707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:28.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.222+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 shutdown_connections 2023-11-28T22:10:28.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.222+0000 7f9ce9459640 1 -- 172.21.15.45:0/3338183908 wait complete. 2023-11-28T22:10:28.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-11-28T22:10:28.471 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.470+0000 7fb536c90640 1 Processor -- start 2023-11-28T22:10:28.472 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.470+0000 7fb536c90640 1 -- start start 2023-11-28T22:10:28.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.472+0000 7fb536c90640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb5301438b0 0x7fb5300b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.472+0000 7fb536c90640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5300b1340 0x7fb5300b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.473 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.472+0000 7fb536c90640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53015dcc0 con 0x7fb5300b24e0 2023-11-28T22:10:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.472+0000 7fb536c90640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53015e840 con 0x7fb5300b1340 2023-11-28T22:10:28.474 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.472+0000 7fb536c90640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53015df10 con 0x7fb5301438b0 2023-11-28T22:10:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.472+0000 7fb53522d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb5300b1340 0x7fb5300b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.472+0000 7fb527fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb5300b24e0 0x7fb5300b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37358/0 (socket says 172.21.15.45:37358) 2023-11-28T22:10:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.472+0000 7fb527fff640 1 -- 172.21.15.45:0/1766808379 learned_addr learned my addr 172.21.15.45:0/1766808379 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.473+0000 7fb534a2c640 1 --2- 172.21.15.45:0/1766808379 >> v2:172.21.15.190:3300/0 conn(0x7fb5301438b0 0x7fb5300b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.473+0000 7fb53522d640 1 -- 172.21.15.45:0/1766808379 >> v2:172.21.15.190:3300/0 conn(0x7fb5301438b0 msgr2=0x7fb5300b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.473+0000 7fb53522d640 1 --2- 172.21.15.45:0/1766808379 >> v2:172.21.15.190:3300/0 conn(0x7fb5301438b0 0x7fb5300b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.475 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.473+0000 7fb53522d640 1 -- 172.21.15.45:0/1766808379 >> v1:172.21.15.45:6789/0 conn(0x7fb5300b24e0 legacy=0x7fb5300b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.473+0000 7fb53522d640 1 -- 172.21.15.45:0/1766808379 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53015e1f0 con 0x7fb5300b1340 2023-11-28T22:10:28.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.473+0000 7fb534a2c640 1 --2- 172.21.15.45:0/1766808379 >> v2:172.21.15.190:3300/0 conn(0x7fb5301438b0 0x7fb5300b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.474+0000 7fb53522d640 1 --2- 172.21.15.45:0/1766808379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 0x7fb5300b3d60 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7fb5300cccc0 tx=0x7fb52802bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa7470b8db83879 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb5277fe640 1 -- 172.21.15.45:0/1766808379 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52802cb50 con 0x7fb5300b1340 2023-11-28T22:10:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb5277fe640 1 -- 172.21.15.45:0/1766808379 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52802ccb0 con 0x7fb5300b1340 2023-11-28T22:10:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb5277fe640 1 -- 172.21.15.45:0/1766808379 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528034660 con 0x7fb5300b1340 2023-11-28T22:10:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb536c90640 1 -- 172.21.15.45:0/1766808379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 msgr2=0x7fb5300b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb536c90640 1 --2- 172.21.15.45:0/1766808379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 0x7fb5300b3d60 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7fb5300cccc0 tx=0x7fb52802bda0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb536c90640 1 -- 172.21.15.45:0/1766808379 shutdown_connections 2023-11-28T22:10:28.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb536c90640 1 --2- 172.21.15.45:0/1766808379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 0x7fb5300b3d60 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb536c90640 1 --2- 172.21.15.45:0/1766808379 >> v2:172.21.15.190:3300/0 conn(0x7fb5301438b0 0x7fb5300b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb536c90640 1 -- 172.21.15.45:0/1766808379 >> 172.21.15.45:0/1766808379 conn(0x7fb53006a500 msgr2=0x7fb53006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:28.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb536c90640 1 -- 172.21.15.45:0/1766808379 shutdown_connections 2023-11-28T22:10:28.478 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.475+0000 7fb536c90640 1 -- 172.21.15.45:0/1766808379 wait complete. 2023-11-28T22:10:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.477+0000 7fb536c90640 1 Processor -- start 2023-11-28T22:10:28.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.477+0000 7fb536c90640 1 -- start start 2023-11-28T22:10:28.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.478+0000 7fb536c90640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 0x7fb530142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.478+0000 7fb536c90640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5300b24e0 0x7fb530157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.478+0000 7fb536c90640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5301438b0 0x7fb5300cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.478+0000 7fb536c90640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb530158120 con 0x7fb5300b1340 2023-11-28T22:10:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.478+0000 7fb536c90640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53013f5b0 con 0x7fb5301438b0 2023-11-28T22:10:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.478+0000 7fb536c90640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53013fe50 con 0x7fb5300b24e0 2023-11-28T22:10:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.478+0000 7fb534a2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 0x7fb530142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.478+0000 7fb534a2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 0x7fb530142840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44002/0 (socket says 172.21.15.45:44002) 2023-11-28T22:10:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.478+0000 7fb534a2c640 1 -- 172.21.15.45:0/2586350737 learned_addr learned my addr 172.21.15.45:0/2586350737 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.479+0000 7fb534a2c640 1 -- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5300b24e0 msgr2=0x7fb530157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.479+0000 7fb534a2c640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5300b24e0 0x7fb530157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.479+0000 7fb534a2c640 1 -- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5301438b0 msgr2=0x7fb5300cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.479+0000 7fb534a2c640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5301438b0 0x7fb5300cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.479+0000 7fb534a2c640 1 -- 172.21.15.45:0/2586350737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb528009a70 con 0x7fb5300b1340 2023-11-28T22:10:28.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.479+0000 7fb534a2c640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 0x7fb530142840 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00c820 tx=0x7fb52c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.479+0000 7fb5257fa640 1 -- 172.21.15.45:0/2586350737 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c007a30 con 0x7fb5300b1340 2023-11-28T22:10:28.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.480+0000 7fb5257fa640 1 -- 172.21.15.45:0/2586350737 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52c007b90 con 0x7fb5300b1340 2023-11-28T22:10:28.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.480+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5300cc0c0 con 0x7fb5300b1340 2023-11-28T22:10:28.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.480+0000 7fb5257fa640 1 -- 172.21.15.45:0/2586350737 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c00ce70 con 0x7fb5300b1340 2023-11-28T22:10:28.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.480+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53013f800 con 0x7fb5300b1340 2023-11-28T22:10:28.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.481+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5300733d0 con 0x7fb5300b1340 2023-11-28T22:10:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.482+0000 7fb5257fa640 1 -- 172.21.15.45:0/2586350737 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb52c0113f0 con 0x7fb5300b1340 2023-11-28T22:10:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.483+0000 7fb5257fa640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb51c06af60 0x7fb51c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.484 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.483+0000 7fb5257fa640 1 -- 172.21.15.45:0/2586350737 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fb52c089bc0 con 0x7fb5300b1340 2023-11-28T22:10:28.485 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.484+0000 7fb527fff640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb51c06af60 0x7fb51c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.485+0000 7fb527fff640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb51c06af60 0x7fb51c06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb5301f4880 tx=0x7fb520007a40 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.488 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.487+0000 7fb5257fa640 1 -- 172.21.15.45:0/2586350737 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb52c0548d0 con 0x7fb5300b1340 2023-11-28T22:10:28.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.684+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fb530143c90 con 0x7fb51c06af60 2023-11-28T22:10:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.685+0000 7fb5257fa640 1 -- 172.21.15.45:0/2586350737 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb5200058b0 con 0x7fb51c06af60 2023-11-28T22:10:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 5 to deep-scrub 2023-11-28T22:10:28.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb51c06af60 msgr2=0x7fb51c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb51c06af60 0x7fb51c06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb5301f4880 tx=0x7fb520007a40 comp rx=0 tx=0).stop 2023-11-28T22:10:28.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 msgr2=0x7fb530142840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 0x7fb530142840 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00c820 tx=0x7fb52c00cc20 comp rx=0 tx=0).stop 2023-11-28T22:10:28.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 shutdown_connections 2023-11-28T22:10:28.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fb51c06af60 0x7fb51c06d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5301438b0 0x7fb5300cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5300b24e0 0x7fb530157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 --2- 172.21.15.45:0/2586350737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5300b1340 0x7fb530142840 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.687+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 >> 172.21.15.45:0/2586350737 conn(0x7fb53006a500 msgr2=0x7fb530071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:28.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.688+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 shutdown_connections 2023-11-28T22:10:28.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.688+0000 7fb536c90640 1 -- 172.21.15.45:0/2586350737 wait complete. 2023-11-28T22:10:28.710 INFO:tasks.ceph:Scrubbing osd.6 2023-11-28T22:10:28.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-11-28T22:10:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.944+0000 7f980ef47640 1 Processor -- start 2023-11-28T22:10:28.946 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.945+0000 7f980ef47640 1 -- start start 2023-11-28T22:10:28.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.946+0000 7f980ef47640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cdd00 0x7f98080ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.947 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.946+0000 7f980ef47640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98080b6960 con 0x7f980815d120 2023-11-28T22:10:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.946+0000 7f980ef47640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98080b6c40 con 0x7f98080b8c20 2023-11-28T22:10:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.946+0000 7f980ef47640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98080b9090 con 0x7f98080cdd00 2023-11-28T22:10:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.946+0000 7f980df45640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f980815d120 0x7f98080b60a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37362/0 (socket says 172.21.15.45:37362) 2023-11-28T22:10:28.948 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.946+0000 7f980df45640 1 -- 172.21.15.45:0/1110547293 learned_addr learned my addr 172.21.15.45:0/1110547293 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980e746640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cdd00 0x7f98080ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333344475 0 0) 0x7f98080b6960 con 0x7f980815d120 2023-11-28T22:10:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f4003520 con 0x7f980815d120 2023-11-28T22:10:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 654210741 0 0) 0x7f97f4003520 con 0x7f980815d120 2023-11-28T22:10:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cdd00 msgr2=0x7f98080ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:28.950 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cdd00 0x7f98080ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 >> v1:172.21.15.190:6789/0 conn(0x7f98080b8c20 legacy=0x7f98080d1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98080b6f20 con 0x7f980815d120 2023-11-28T22:10:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f9804001670 con 0x7f980815d120 2023-11-28T22:10:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 >> v1:172.21.15.45:6789/0 conn(0x7f980815d120 legacy=0x7f98080b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.947+0000 7f980cf43640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cdd00 0x7f97f40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980cf43640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f4003f20 0x7f97f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980cf43640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4004d00 0x7f97f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98080b9090 con 0x7f97f4003f20 2023-11-28T22:10:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f400baf0 con 0x7f97f4004d00 2023-11-28T22:10:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f400bcb0 con 0x7f98080cdd00 2023-11-28T22:10:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980d744640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f4003f20 0x7f97f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980df45640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cdd00 0x7f97f40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980e746640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4004d00 0x7f97f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980d744640 1 -- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cdd00 msgr2=0x7f97f40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980d744640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cdd00 0x7f97f40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980d744640 1 -- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4004d00 msgr2=0x7f97f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:28.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980d744640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4004d00 0x7f97f4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.953 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.948+0000 7f980d744640 1 -- 172.21.15.45:0/1110547293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f4003520 con 0x7f97f4003f20 2023-11-28T22:10:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.950+0000 7f980d744640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f4003f20 0x7f97f40047c0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f97f8001750 tx=0x7f97f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14618feed0db3ea5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.950+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f80052d0 con 0x7f97f4003f20 2023-11-28T22:10:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.950+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f802e050 con 0x7f97f4003f20 2023-11-28T22:10:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.950+0000 7f980cf43640 1 -- 172.21.15.45:0/1110547293 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8003830 con 0x7f97f4003f20 2023-11-28T22:10:28.954 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.950+0000 7f980ef47640 1 -- 172.21.15.45:0/1110547293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f4003f20 msgr2=0x7f97f40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.950+0000 7f980ef47640 1 --2- 172.21.15.45:0/1110547293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f4003f20 0x7f97f40047c0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f97f8001750 tx=0x7f97f40057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.951+0000 7f980e746640 1 -- 172.21.15.45:0/1110547293 reap_dead start 2023-11-28T22:10:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.951+0000 7f980ef47640 1 -- 172.21.15.45:0/1110547293 shutdown_connections 2023-11-28T22:10:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.951+0000 7f980ef47640 1 -- 172.21.15.45:0/1110547293 >> 172.21.15.45:0/1110547293 conn(0x7f9808070f40 msgr2=0x7f9808071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.951+0000 7f980ef47640 1 -- 172.21.15.45:0/1110547293 shutdown_connections 2023-11-28T22:10:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.951+0000 7f980ef47640 1 -- 172.21.15.45:0/1110547293 wait complete. 2023-11-28T22:10:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.952+0000 7f980ef47640 1 Processor -- start 2023-11-28T22:10:28.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.953+0000 7f980ef47640 1 -- start start 2023-11-28T22:10:28.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980ef47640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4003f20 0x7f98081f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.957 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980ef47640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98081f59d0 0x7f98080cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980ef47640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cbd20 0x7f98080ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980ef47640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98080d1780 con 0x7f98081f59d0 2023-11-28T22:10:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980ef47640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98080cc1a0 con 0x7f97f4003f20 2023-11-28T22:10:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980ef47640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98080cb1f0 con 0x7f98080cbd20 2023-11-28T22:10:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980d744640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98081f59d0 0x7f98080cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980d744640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98081f59d0 0x7f98080cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44018/0 (socket says 172.21.15.45:44018) 2023-11-28T22:10:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980d744640 1 -- 172.21.15.45:0/3353537228 learned_addr learned my addr 172.21.15.45:0/3353537228 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980df45640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4003f20 0x7f98081f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980e746640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cbd20 0x7f98080ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980d744640 1 -- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cbd20 msgr2=0x7f98080ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.954+0000 7f980d744640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cbd20 0x7f98080ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f980d744640 1 -- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4003f20 msgr2=0x7f98081f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f980d744640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4003f20 0x7f98081f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f980d744640 1 -- 172.21.15.45:0/3353537228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98081f5e30 con 0x7f98081f59d0 2023-11-28T22:10:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f980df45640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4003f20 0x7f98081f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f980e746640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98080cbd20 0x7f98080ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f980d744640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98081f59d0 0x7f98080cb7e0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f97f8008fd0 tx=0x7f97f8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f97fe7fc640 1 -- 172.21.15.45:0/3353537228 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f80076c0 con 0x7f98081f59d0 2023-11-28T22:10:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f97fe7fc640 1 -- 172.21.15.45:0/3353537228 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f8007a30 con 0x7f98081f59d0 2023-11-28T22:10:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f97fe7fc640 1 -- 172.21.15.45:0/3353537228 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8006be0 con 0x7f98081f59d0 2023-11-28T22:10:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980815d820 con 0x7f98081f59d0 2023-11-28T22:10:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.955+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980815dde0 con 0x7f98081f59d0 2023-11-28T22:10:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.956+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f98080b74e0 con 0x7f98081f59d0 2023-11-28T22:10:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.957+0000 7f97fe7fc640 1 -- 172.21.15.45:0/3353537228 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f97f8006d40 con 0x7f98081f59d0 2023-11-28T22:10:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.958+0000 7f97fe7fc640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f97ec06af60 0x7f97ec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.958+0000 7f97fe7fc640 1 -- 172.21.15.45:0/3353537228 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f97f80ab710 con 0x7f98081f59d0 2023-11-28T22:10:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.959+0000 7f97fe7fc640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f97ec0718f0 0x7f97ec073d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.959+0000 7f97fe7fc640 1 -- 172.21.15.45:0/3353537228 --> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97ec074420 con 0x7f97ec0718f0 2023-11-28T22:10:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.959+0000 7f97fe7fc640 1 -- 172.21.15.45:0/3353537228 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=222) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97f80adcf0 con 0x7f98081f59d0 2023-11-28T22:10:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.959+0000 7f980df45640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f97ec06af60 0x7f97ec06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.959+0000 7f980e746640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f97ec0718f0 0x7f97ec073d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.960+0000 7f980df45640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f97ec06af60 0x7f97ec06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9804001050 tx=0x7f9804001d40 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.961+0000 7f980e746640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f97ec0718f0 0x7f97ec073d50 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:28.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.963+0000 7f97fe7fc640 1 -- 172.21.15.45:0/3353537228 <== osd.6 v2:172.21.15.190:6820/2089844775 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f97ec074420 con 0x7f97ec0718f0 2023-11-28T22:10:28.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.978+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 --> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f98080ce150 con 0x7f97ec0718f0 2023-11-28T22:10:28.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.979+0000 7f97fe7fc640 1 -- 172.21.15.45:0/3353537228 <== osd.6 v2:172.21.15.190:6820/2089844775 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9800009ce0 con 0x7f97ec0718f0 2023-11-28T22:10:28.980 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:10:28.981 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-28T22:10:28.981 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:10:28.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.979+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f97ec0718f0 msgr2=0x7f97ec073d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.979+0000 7f980ef47640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6820/2089844775,v1:172.21.15.190:6822/2089844775] conn(0x7f97ec0718f0 0x7f97ec073d50 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:28.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.980+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f97ec06af60 msgr2=0x7f97ec06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.980+0000 7f980ef47640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f97ec06af60 0x7f97ec06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9804001050 tx=0x7f9804001d40 comp rx=0 tx=0).stop 2023-11-28T22:10:28.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.980+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98081f59d0 msgr2=0x7f98080cb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:28.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.980+0000 7f980ef47640 1 --2- 172.21.15.45:0/3353537228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98081f59d0 0x7f98080cb7e0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f97f8008fd0 tx=0x7f97f8001120 comp rx=0 tx=0).stop 2023-11-28T22:10:28.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.980+0000 7f980e746640 1 -- 172.21.15.45:0/3353537228 reap_dead start 2023-11-28T22:10:28.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.980+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 shutdown_connections 2023-11-28T22:10:28.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.980+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 >> 172.21.15.45:0/3353537228 conn(0x7f9808070f40 msgr2=0x7f98080d2590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:28.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.980+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 shutdown_connections 2023-11-28T22:10:28.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:28.980+0000 7f980ef47640 1 -- 172.21.15.45:0/3353537228 wait complete. 2023-11-28T22:10:28.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-11-28T22:10:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.212+0000 7fd5b9f1d640 1 Processor -- start 2023-11-28T22:10:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.212+0000 7fd5b9f1d640 1 -- start start 2023-11-28T22:10:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.214+0000 7fd5b9f1d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd5b40b35b0 0x7fd5b415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.214+0000 7fd5b9f1d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5b415b850 0x7fd5b40b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.214+0000 7fd5b9f1d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b415bda0 con 0x7fd5b40b1fa0 2023-11-28T22:10:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.214+0000 7fd5b9f1d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b415ee40 con 0x7fd5b40b35b0 2023-11-28T22:10:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.214+0000 7fd5b9f1d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b415c080 con 0x7fd5b415b850 2023-11-28T22:10:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.215+0000 7fd5b37fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd5b40b35b0 0x7fd5b415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.215+0000 7fd5b37fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd5b40b35b0 0x7fd5b415ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44022/0 (socket says 172.21.15.45:44022) 2023-11-28T22:10:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.215+0000 7fd5b37fe640 1 -- 172.21.15.45:0/2479178416 learned_addr learned my addr 172.21.15.45:0/2479178416 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.215+0000 7fd5b2ffd640 1 --2- 172.21.15.45:0/2479178416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5b415b850 0x7fd5b40b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.216+0000 7fd5b37fe640 1 -- 172.21.15.45:0/2479178416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5b415b850 msgr2=0x7fd5b40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.216+0000 7fd5b37fe640 1 --2- 172.21.15.45:0/2479178416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5b415b850 0x7fd5b40b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.216+0000 7fd5b37fe640 1 -- 172.21.15.45:0/2479178416 >> v1:172.21.15.190:6789/0 conn(0x7fd5b40b1fa0 legacy=0x7fd5b415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.216+0000 7fd5b37fe640 1 -- 172.21.15.45:0/2479178416 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b415c360 con 0x7fd5b40b35b0 2023-11-28T22:10:29.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.217+0000 7fd5b37fe640 1 --2- 172.21.15.45:0/2479178416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 0x7fd5b415ca10 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098a0 tx=0x7fd5a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8762d122b3cfdc9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.218+0000 7fd5b27fc640 1 -- 172.21.15.45:0/2479178416 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4030840 con 0x7fd5b40b35b0 2023-11-28T22:10:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.218+0000 7fd5b27fc640 1 -- 172.21.15.45:0/2479178416 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a40309a0 con 0x7fd5b40b35b0 2023-11-28T22:10:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.218+0000 7fd5b27fc640 1 -- 172.21.15.45:0/2479178416 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4030d30 con 0x7fd5b40b35b0 2023-11-28T22:10:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.218+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/2479178416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 msgr2=0x7fd5b415ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.218+0000 7fd5b9f1d640 1 --2- 172.21.15.45:0/2479178416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 0x7fd5b415ca10 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098a0 tx=0x7fd5a402b980 comp rx=0 tx=0).stop 2023-11-28T22:10:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.219+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/2479178416 shutdown_connections 2023-11-28T22:10:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.219+0000 7fd5b9f1d640 1 --2- 172.21.15.45:0/2479178416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5b415b850 0x7fd5b40b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.219+0000 7fd5b9f1d640 1 --2- 172.21.15.45:0/2479178416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 0x7fd5b415ca10 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.219+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/2479178416 >> 172.21.15.45:0/2479178416 conn(0x7fd5b406a500 msgr2=0x7fd5b406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.219+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/2479178416 shutdown_connections 2023-11-28T22:10:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.219+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/2479178416 wait complete. 2023-11-28T22:10:29.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.221+0000 7fd5b9f1d640 1 Processor -- start 2023-11-28T22:10:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.221+0000 7fd5b9f1d640 1 -- start start 2023-11-28T22:10:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b9f1d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5b40b1fa0 0x7fd5b40abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b9f1d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 0x7fd5b40aa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b9f1d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5b415b850 0x7fd5b40c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b9f1d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41f44e0 con 0x7fd5b40b35b0 2023-11-28T22:10:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b9f1d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41f47c0 con 0x7fd5b40b1fa0 2023-11-28T22:10:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b9f1d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41f4aa0 con 0x7fd5b415b850 2023-11-28T22:10:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 0x7fd5b40aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 0x7fd5b40aa730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44030/0 (socket says 172.21.15.45:44030) 2023-11-28T22:10:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b2ffd640 1 -- 172.21.15.45:0/3197589367 learned_addr learned my addr 172.21.15.45:0/3197589367 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.222+0000 7fd5b37fe640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5b40b1fa0 0x7fd5b40abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd5b2ffd640 1 -- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5b415b850 msgr2=0x7fd5b40c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd5b2ffd640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5b415b850 0x7fd5b40c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd5b2ffd640 1 -- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5b40b1fa0 msgr2=0x7fd5b40abd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd5b2ffd640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5b40b1fa0 0x7fd5b40abd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd5b2ffd640 1 -- 172.21.15.45:0/3197589367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a40094d0 con 0x7fd5b40b35b0 2023-11-28T22:10:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd5b2ffd640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 0x7fd5b40aa730 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800e840 tx=0x7fd5a800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd59bfff640 1 -- 172.21.15.45:0/3197589367 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a80094f0 con 0x7fd5b40b35b0 2023-11-28T22:10:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd59bfff640 1 -- 172.21.15.45:0/3197589367 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a800c940 con 0x7fd5b40b35b0 2023-11-28T22:10:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd59bfff640 1 -- 172.21.15.45:0/3197589367 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a800ccd0 con 0x7fd5b40b35b0 2023-11-28T22:10:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b41f4d80 con 0x7fd5b40b35b0 2023-11-28T22:10:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.223+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b40aac70 con 0x7fd5b40b35b0 2023-11-28T22:10:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.224+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b40733d0 con 0x7fd5b40b35b0 2023-11-28T22:10:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.225+0000 7fd59bfff640 1 -- 172.21.15.45:0/3197589367 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd5a8013470 con 0x7fd5b40b35b0 2023-11-28T22:10:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.226+0000 7fd59bfff640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fd5a006af60 0x7fd5a006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.232 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.227+0000 7fd59bfff640 1 -- 172.21.15.45:0/3197589367 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fd5a80899d0 con 0x7fd5b40b35b0 2023-11-28T22:10:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.227+0000 7fd5b37fe640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fd5a006af60 0x7fd5a006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.228+0000 7fd5b37fe640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fd5a006af60 0x7fd5a006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4036000 tx=0x7fd5a4008030 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.233 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.231+0000 7fd59bfff640 1 -- 172.21.15.45:0/3197589367 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5a8054800 con 0x7fd5b40b35b0 2023-11-28T22:10:29.430 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.428+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5b40aae90 con 0x7fd5a006af60 2023-11-28T22:10:29.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.429+0000 7fd59bfff640 1 -- 172.21.15.45:0/3197589367 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd5b40aae90 con 0x7fd5a006af60 2023-11-28T22:10:29.431 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 6 to deep-scrub 2023-11-28T22:10:29.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fd5a006af60 msgr2=0x7fd5a006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fd5a006af60 0x7fd5a006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4036000 tx=0x7fd5a4008030 comp rx=0 tx=0).stop 2023-11-28T22:10:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 msgr2=0x7fd5b40aa730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 0x7fd5b40aa730 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800e840 tx=0x7fd5a800ed00 comp rx=0 tx=0).stop 2023-11-28T22:10:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 shutdown_connections 2023-11-28T22:10:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5b415b850 0x7fd5b40c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b40b35b0 0x7fd5b40aa730 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fd5a006af60 0x7fd5a006d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 --2- 172.21.15.45:0/3197589367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5b40b1fa0 0x7fd5b40abd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 >> 172.21.15.45:0/3197589367 conn(0x7fd5b406a500 msgr2=0x7fd5b40aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:29.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.432+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 shutdown_connections 2023-11-28T22:10:29.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.433+0000 7fd5b9f1d640 1 -- 172.21.15.45:0/3197589367 wait complete. 2023-11-28T22:10:29.453 INFO:tasks.ceph:Scrubbing osd.7 2023-11-28T22:10:29.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-11-28T22:10:29.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.686+0000 7f78d646d640 1 Processor -- start 2023-11-28T22:10:29.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.686+0000 7f78d646d640 1 -- start start 2023-11-28T22:10:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.688+0000 7f78d646d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f78d015b190 0x7f78d006de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.688+0000 7f78d646d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f78d006c090 0x7f78d006c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.688+0000 7f78d646d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d006e3a0 con 0x7f78d0070030 2023-11-28T22:10:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.688+0000 7f78d646d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d006ca60 con 0x7f78d006c090 2023-11-28T22:10:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.688+0000 7f78d646d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d015b570 con 0x7f78d015b190 2023-11-28T22:10:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.688+0000 7f78d5c6c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f78d006c090 0x7f78d006c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.688+0000 7f78d5c6c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f78d006c090 0x7f78d006c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44044/0 (socket says 172.21.15.45:44044) 2023-11-28T22:10:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.688+0000 7f78d5c6c640 1 -- 172.21.15.45:0/2611509224 learned_addr learned my addr 172.21.15.45:0/2611509224 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.689+0000 7f78d546b640 1 --2- 172.21.15.45:0/2611509224 >> v2:172.21.15.190:3300/0 conn(0x7f78d015b190 0x7f78d006de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.689+0000 7f78d5c6c640 1 -- 172.21.15.45:0/2611509224 >> v2:172.21.15.190:3300/0 conn(0x7f78d015b190 msgr2=0x7f78d006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:29.691 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.689+0000 7f78d5c6c640 1 --2- 172.21.15.45:0/2611509224 >> v2:172.21.15.190:3300/0 conn(0x7f78d015b190 0x7f78d006de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.689+0000 7f78d5c6c640 1 -- 172.21.15.45:0/2611509224 >> v1:172.21.15.190:6789/0 conn(0x7f78d0070030 legacy=0x7f78d00b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:29.692 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.689+0000 7f78d5c6c640 1 -- 172.21.15.45:0/2611509224 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d00c9d20 con 0x7f78d006c090 2023-11-28T22:10:29.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.690+0000 7f78d5c6c640 1 --2- 172.21.15.45:0/2611509224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d006c090 0x7f78d006c520 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f78d006bee0 tx=0x7f78c802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616facc04fafdbb0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.691+0000 7f78c7fff640 1 -- 172.21.15.45:0/2611509224 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c802ca80 con 0x7f78d006c090 2023-11-28T22:10:29.693 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.691+0000 7f78c7fff640 1 -- 172.21.15.45:0/2611509224 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c802cbe0 con 0x7f78d006c090 2023-11-28T22:10:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.691+0000 7f78c7fff640 1 -- 172.21.15.45:0/2611509224 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c80343f0 con 0x7f78d006c090 2023-11-28T22:10:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.691+0000 7f78d646d640 1 -- 172.21.15.45:0/2611509224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d006c090 msgr2=0x7f78d006c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.691+0000 7f78d646d640 1 --2- 172.21.15.45:0/2611509224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d006c090 0x7f78d006c520 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f78d006bee0 tx=0x7f78c802bee0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.692+0000 7f78d646d640 1 -- 172.21.15.45:0/2611509224 shutdown_connections 2023-11-28T22:10:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.692+0000 7f78d646d640 1 --2- 172.21.15.45:0/2611509224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d006c090 0x7f78d006c520 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.692+0000 7f78d646d640 1 --2- 172.21.15.45:0/2611509224 >> v2:172.21.15.190:3300/0 conn(0x7f78d015b190 0x7f78d006de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.692+0000 7f78d646d640 1 -- 172.21.15.45:0/2611509224 >> 172.21.15.45:0/2611509224 conn(0x7f78d00cec30 msgr2=0x7f78d00cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.692+0000 7f78d646d640 1 -- 172.21.15.45:0/2611509224 shutdown_connections 2023-11-28T22:10:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.692+0000 7f78d646d640 1 -- 172.21.15.45:0/2611509224 wait complete. 2023-11-28T22:10:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.694+0000 7f78d646d640 1 Processor -- start 2023-11-28T22:10:29.695 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.694+0000 7f78d646d640 1 -- start start 2023-11-28T22:10:29.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d646d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78d006c090 0x7f78d00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d646d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78d0070030 0x7f78d014ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d646d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d015b190 0x7f78d015b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d646d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d01504f0 con 0x7f78d015b190 2023-11-28T22:10:29.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d646d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d015de50 con 0x7f78d006c090 2023-11-28T22:10:29.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d646d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d015e990 con 0x7f78d0070030 2023-11-28T22:10:29.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d5c6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d015b190 0x7f78d015b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d5c6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d015b190 0x7f78d015b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44058/0 (socket says 172.21.15.45:44058) 2023-11-28T22:10:29.697 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d5c6c640 1 -- 172.21.15.45:0/2896966684 learned_addr learned my addr 172.21.15.45:0/2896966684 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:29.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d546b640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78d006c090 0x7f78d00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d5c6c640 1 -- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78d0070030 msgr2=0x7f78d014ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d5c6c640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78d0070030 0x7f78d014ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.695+0000 7f78d5c6c640 1 -- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78d006c090 msgr2=0x7f78d00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.699 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.696+0000 7f78d5c6c640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78d006c090 0x7f78d00b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.696+0000 7f78d5c6c640 1 -- 172.21.15.45:0/2896966684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8009b30 con 0x7f78d015b190 2023-11-28T22:10:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.696+0000 7f78d5c6c640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d015b190 0x7f78d015b6b0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f78c8036040 tx=0x7f78c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.696+0000 7f78c5ffb640 1 -- 172.21.15.45:0/2896966684 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8009510 con 0x7f78d015b190 2023-11-28T22:10:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.696+0000 7f78c5ffb640 1 -- 172.21.15.45:0/2896966684 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c803e450 con 0x7f78d015b190 2023-11-28T22:10:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.696+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d015e0f0 con 0x7f78d015b190 2023-11-28T22:10:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.696+0000 7f78c5ffb640 1 -- 172.21.15.45:0/2896966684 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c803e7e0 con 0x7f78d015b190 2023-11-28T22:10:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.696+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d015e6b0 con 0x7f78d015b190 2023-11-28T22:10:29.700 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.697+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78d0071ef0 con 0x7f78d015b190 2023-11-28T22:10:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.699+0000 7f78c5ffb640 1 -- 172.21.15.45:0/2896966684 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f78c802cd50 con 0x7f78d015b190 2023-11-28T22:10:29.701 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.700+0000 7f78c5ffb640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f78b406af60 0x7f78b406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.700+0000 7f78c5ffb640 1 -- 172.21.15.45:0/2896966684 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f78c80ac2d0 con 0x7f78d015b190 2023-11-28T22:10:29.702 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.700+0000 7f78c5ffb640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f78b40718f0 0x7f78b4073d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.704 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.700+0000 7f78c5ffb640 1 -- 172.21.15.45:0/2896966684 --> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78b4074420 con 0x7f78b40718f0 2023-11-28T22:10:29.705 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.700+0000 7f78c5ffb640 1 -- 172.21.15.45:0/2896966684 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=222) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78c80ae990 con 0x7f78d015b190 2023-11-28T22:10:29.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.701+0000 7f78d4c6a640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f78b40718f0 0x7f78b4073d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.701+0000 7f78d546b640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f78b406af60 0x7f78b406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.702+0000 7f78d4c6a640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f78b40718f0 0x7f78b4073d50 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.702+0000 7f78d546b640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f78b406af60 0x7f78b406d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78cc006fd0 tx=0x7f78cc008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.706 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.704+0000 7f78c5ffb640 1 -- 172.21.15.45:0/2896966684 <== osd.7 v2:172.21.15.190:6816/3686540929 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f78b4074420 con 0x7f78b40718f0 2023-11-28T22:10:29.721 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.720+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 --> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f78d0071c50 con 0x7f78b40718f0 2023-11-28T22:10:29.722 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.721+0000 7f78c5ffb640 1 -- 172.21.15.45:0/2896966684 <== osd.7 v2:172.21.15.190:6816/3686540929 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f78c00050b0 con 0x7f78b40718f0 2023-11-28T22:10:29.722 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-11-28T22:10:29.723 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-28T22:10:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-11-28T22:10:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.721+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f78b40718f0 msgr2=0x7f78b4073d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.721+0000 7f78d646d640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6816/3686540929,v1:172.21.15.190:6817/3686540929] conn(0x7f78b40718f0 0x7f78b4073d50 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.722+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f78b406af60 msgr2=0x7f78b406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.722+0000 7f78d646d640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f78b406af60 0x7f78b406d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78cc006fd0 tx=0x7f78cc008040 comp rx=0 tx=0).stop 2023-11-28T22:10:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.722+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d015b190 msgr2=0x7f78d015b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.722+0000 7f78d646d640 1 --2- 172.21.15.45:0/2896966684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78d015b190 0x7f78d015b6b0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f78c8036040 tx=0x7f78c8004a40 comp rx=0 tx=0).stop 2023-11-28T22:10:29.724 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.722+0000 7f78d5c6c640 1 -- 172.21.15.45:0/2896966684 reap_dead start 2023-11-28T22:10:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.722+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 shutdown_connections 2023-11-28T22:10:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.722+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 >> 172.21.15.45:0/2896966684 conn(0x7f78d00cec30 msgr2=0x7f78d00cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.722+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 shutdown_connections 2023-11-28T22:10:29.725 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.722+0000 7f78d646d640 1 -- 172.21.15.45:0/2896966684 wait complete. 2023-11-28T22:10:29.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-11-28T22:10:29.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.965+0000 7f3846825640 1 Processor -- start 2023-11-28T22:10:29.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.965+0000 7f3846825640 1 -- start start 2023-11-28T22:10:29.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.966+0000 7f3846825640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f38400d29b0 0x7f38400b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.966+0000 7f3846825640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 0x7f38400cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.966+0000 7f3846825640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400d0370 con 0x7f384015d350 2023-11-28T22:10:29.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.966+0000 7f3846825640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38400d1910 con 0x7f38400d29b0 2023-11-28T22:10:29.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.966+0000 7f3846825640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400d1bf0 con 0x7f38400cf9c0 2023-11-28T22:10:29.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.967+0000 7f3845823640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f384015d350 0x7f38400d10c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37378/0 (socket says 172.21.15.45:37378) 2023-11-28T22:10:29.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.967+0000 7f3845823640 1 -- 172.21.15.45:0/2736182053 learned_addr learned my addr 172.21.15.45:0/2736182053 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.967+0000 7f3846024640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 0x7f38400cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.967+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300940329 0 0) 0x7f3830001490 con 0x7f384015d350 2023-11-28T22:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.967+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3828003520 con 0x7f384015d350 2023-11-28T22:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.967+0000 7f3845022640 1 --2- 172.21.15.45:0/2736182053 >> v2:172.21.15.190:3300/0 conn(0x7f38400d29b0 0x7f38400b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2167740219 0 0) 0x7f3828003520 con 0x7f384015d350 2023-11-28T22:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 msgr2=0x7f38400cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 0x7f38400cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 >> v2:172.21.15.190:3300/0 conn(0x7f38400d29b0 msgr2=0x7f38400b8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 --2- 172.21.15.45:0/2736182053 >> v2:172.21.15.190:3300/0 conn(0x7f38400d29b0 0x7f38400b8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400d1ed0 con 0x7f384015d350 2023-11-28T22:10:29.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7f3830001b80 con 0x7f384015d350 2023-11-28T22:10:29.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3846024640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 0x7f38400cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T22:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 >> v1:172.21.15.45:6789/0 conn(0x7f384015d350 legacy=0x7f38400d10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828003970 0x7f3828003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 0x7f38280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f3828005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38400d1910 con 0x7f3828004d00 2023-11-28T22:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38400d1bf0 con 0x7f3828003970 2023-11-28T22:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382800ba30 con 0x7f38400cf9c0 2023-11-28T22:10:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.968+0000 7f3846024640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f3828005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.969+0000 7f3845823640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828003970 0x7f3828003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.969+0000 7f3845022640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 0x7f38280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.969+0000 7f3846024640 1 -- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 msgr2=0x7f38280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.969+0000 7f3846024640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 0x7f38280047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.969+0000 7f3846024640 1 -- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828003970 msgr2=0x7f3828003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.969+0000 7f3846024640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828003970 0x7f3828003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.969+0000 7f3846024640 1 -- 172.21.15.45:0/2736182053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828003520 con 0x7f3828004d00 2023-11-28T22:10:29.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.969+0000 7f3845823640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3828003970 0x7f3828003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:29.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.969+0000 7f3845022640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38400cf9c0 0x7f38280047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:29.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.970+0000 7f3846024640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f3828005180 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f38380098a0 tx=0x7f38280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33ba623b544e563 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.970+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838004230 con 0x7f3828004d00 2023-11-28T22:10:29.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.970+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3838004390 con 0x7f3828004d00 2023-11-28T22:10:29.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.970+0000 7f3844821640 1 -- 172.21.15.45:0/2736182053 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838003830 con 0x7f3828004d00 2023-11-28T22:10:29.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.971+0000 7f3846825640 1 -- 172.21.15.45:0/2736182053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 msgr2=0x7f3828005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.977 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.971+0000 7f3846825640 1 --2- 172.21.15.45:0/2736182053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f3828005180 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f38380098a0 tx=0x7f38280057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.971+0000 7f3846024640 1 -- 172.21.15.45:0/2736182053 reap_dead start 2023-11-28T22:10:29.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.971+0000 7f3846825640 1 -- 172.21.15.45:0/2736182053 shutdown_connections 2023-11-28T22:10:29.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.971+0000 7f3846825640 1 -- 172.21.15.45:0/2736182053 >> 172.21.15.45:0/2736182053 conn(0x7f3840070f40 msgr2=0x7f3840071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:29.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.971+0000 7f3846825640 1 -- 172.21.15.45:0/2736182053 shutdown_connections 2023-11-28T22:10:29.978 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.971+0000 7f3846825640 1 -- 172.21.15.45:0/2736182053 wait complete. 2023-11-28T22:10:29.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.973+0000 7f3846825640 1 Processor -- start 2023-11-28T22:10:29.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.973+0000 7f3846825640 1 -- start start 2023-11-28T22:10:29.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.974+0000 7f3846825640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f38401562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.981 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.974+0000 7f3846825640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3840156970 0x7f3840154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.974+0000 7f3846825640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38401522a0 0x7f38400cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.974+0000 7f3846825640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840154770 con 0x7f3828004d00 2023-11-28T22:10:29.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.974+0000 7f3846825640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840154980 con 0x7f38401522a0 2023-11-28T22:10:29.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.974+0000 7f3846825640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400c7cd0 con 0x7f3840156970 2023-11-28T22:10:29.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.974+0000 7f3845823640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f38401562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.974+0000 7f3845823640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f38401562b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44076/0 (socket says 172.21.15.45:44076) 2023-11-28T22:10:29.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.974+0000 7f3845823640 1 -- 172.21.15.45:0/349734832 learned_addr learned my addr 172.21.15.45:0/349734832 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3845022640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3840156970 0x7f3840154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3846024640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38401522a0 0x7f38400cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3845823640 1 -- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3840156970 msgr2=0x7f3840154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3845823640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3840156970 0x7f3840154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3845823640 1 -- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38401522a0 msgr2=0x7f38400cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3845823640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38401522a0 0x7f38400cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3845823640 1 -- 172.21.15.45:0/349734832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400c7fb0 con 0x7f3828004d00 2023-11-28T22:10:29.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3845022640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3840156970 0x7f3840154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3845823640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f38401562b0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f3830001c10 tx=0x7f383000b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f383e7fc640 1 -- 172.21.15.45:0/349734832 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830004d60 con 0x7f3828004d00 2023-11-28T22:10:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f383e7fc640 1 -- 172.21.15.45:0/349734832 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3830007150 con 0x7f3828004d00 2023-11-28T22:10:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f383e7fc640 1 -- 172.21.15.45:0/349734832 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38300074c0 con 0x7f3828004d00 2023-11-28T22:10:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38400c8290 con 0x7f3828004d00 2023-11-28T22:10:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.975+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3840152720 con 0x7f3828004d00 2023-11-28T22:10:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.976+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3840071b30 con 0x7f3828004d00 2023-11-28T22:10:29.984 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.978+0000 7f383e7fc640 1 -- 172.21.15.45:0/349734832 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3830007620 con 0x7f3828004d00 2023-11-28T22:10:29.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.978+0000 7f383e7fc640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f382c06af60 0x7f382c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:29.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.979+0000 7f383e7fc640 1 -- 172.21.15.45:0/349734832 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f3830089950 con 0x7f3828004d00 2023-11-28T22:10:29.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.979+0000 7f3845022640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f382c06af60 0x7f382c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:29.985 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.980+0000 7f3845022640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f382c06af60 0x7f382c06d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f38401598a0 tx=0x7f3834008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:29.986 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:29.983+0000 7f383e7fc640 1 -- 172.21.15.45:0/349734832 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3830054700 con 0x7f3828004d00 2023-11-28T22:10:30.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.178+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f3840152940 con 0x7f382c06af60 2023-11-28T22:10:30.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.179+0000 7f383e7fc640 1 -- 172.21.15.45:0/349734832 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3840152940 con 0x7f382c06af60 2023-11-28T22:10:30.180 INFO:teuthology.orchestra.run.smithi045.stderr:instructed osd(s) 7 to deep-scrub 2023-11-28T22:10:30.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f382c06af60 msgr2=0x7f382c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f382c06af60 0x7f382c06d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f38401598a0 tx=0x7f3834008040 comp rx=0 tx=0).stop 2023-11-28T22:10:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 msgr2=0x7f38401562b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f38401562b0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f3830001c10 tx=0x7f383000b560 comp rx=0 tx=0).stop 2023-11-28T22:10:30.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 shutdown_connections 2023-11-28T22:10:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f382c06af60 0x7f382c06d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38401522a0 0x7f38400cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3840156970 0x7f3840154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 --2- 172.21.15.45:0/349734832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3828004d00 0x7f38401562b0 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.181+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 >> 172.21.15.45:0/349734832 conn(0x7f3840070f40 msgr2=0x7f38400d2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.182+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 shutdown_connections 2023-11-28T22:10:30.184 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.182+0000 7f3846825640 1 -- 172.21.15.45:0/349734832 wait complete. 2023-11-28T22:10:30.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T22:10:30.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.428+0000 7f1518b14640 1 Processor -- start 2023-11-28T22:10:30.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.428+0000 7f1518b14640 1 -- start start 2023-11-28T22:10:30.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1518b14640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1514155780 0x7f15140abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:30.432 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1518b14640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15140ae540 0x7f15140ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1518b14640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15140ce630 0x7f151415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1518b14640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15140ceac0 con 0x7f15140ce630 2023-11-28T22:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1518b14640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151415c3c0 con 0x7f1514155780 2023-11-28T22:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1518b14640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1514155b60 con 0x7f15140ae540 2023-11-28T22:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1513fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15140ce630 0x7f151415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1513fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f15140ce630 0x7f151415ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44090/0 (socket says 172.21.15.45:44090) 2023-11-28T22:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1513fff640 1 -- 172.21.15.45:0/1899313282 learned_addr learned my addr 172.21.15.45:0/1899313282 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.430+0000 7f1513fff640 1 -- 172.21.15.45:0/1899313282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15140ae540 msgr2=0x7f15140ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.431+0000 7f1513fff640 1 --2- 172.21.15.45:0/1899313282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15140ae540 0x7f15140ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.431+0000 7f1513fff640 1 -- 172.21.15.45:0/1899313282 >> v2:172.21.15.190:3300/0 conn(0x7f1514155780 msgr2=0x7f15140abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.431+0000 7f1513fff640 1 --2- 172.21.15.45:0/1899313282 >> v2:172.21.15.190:3300/0 conn(0x7f1514155780 0x7f15140abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.431+0000 7f1513fff640 1 -- 172.21.15.45:0/1899313282 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151415ba90 con 0x7f15140ce630 2023-11-28T22:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.432+0000 7f1513fff640 1 --2- 172.21.15.45:0/1899313282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ce630 0x7f151415ed30 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f150c009e50 tx=0x7f150c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c12c4e2582d817b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.432+0000 7f1512ffd640 1 -- 172.21.15.45:0/1899313282 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c02ca80 con 0x7f15140ce630 2023-11-28T22:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.432+0000 7f1512ffd640 1 -- 172.21.15.45:0/1899313282 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c02cbe0 con 0x7f15140ce630 2023-11-28T22:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.432+0000 7f1512ffd640 1 -- 172.21.15.45:0/1899313282 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c0343f0 con 0x7f15140ce630 2023-11-28T22:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.433+0000 7f1518b14640 1 -- 172.21.15.45:0/1899313282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ce630 msgr2=0x7f151415ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.433+0000 7f1518b14640 1 --2- 172.21.15.45:0/1899313282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ce630 0x7f151415ed30 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f150c009e50 tx=0x7f150c02bee0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.433+0000 7f1518b14640 1 -- 172.21.15.45:0/1899313282 shutdown_connections 2023-11-28T22:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.433+0000 7f1518b14640 1 --2- 172.21.15.45:0/1899313282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ce630 0x7f151415ed30 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.433+0000 7f1518b14640 1 --2- 172.21.15.45:0/1899313282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15140ae540 0x7f15140ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.433+0000 7f1518b14640 1 --2- 172.21.15.45:0/1899313282 >> v2:172.21.15.190:3300/0 conn(0x7f1514155780 0x7f15140abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.433+0000 7f1518b14640 1 -- 172.21.15.45:0/1899313282 >> 172.21.15.45:0/1899313282 conn(0x7f151406ab30 msgr2=0x7f15140b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.433+0000 7f1518b14640 1 -- 172.21.15.45:0/1899313282 shutdown_connections 2023-11-28T22:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.433+0000 7f1518b14640 1 -- 172.21.15.45:0/1899313282 wait complete. 2023-11-28T22:10:30.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.435+0000 7f1518b14640 1 Processor -- start 2023-11-28T22:10:30.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.435+0000 7f1518b14640 1 -- start start 2023-11-28T22:10:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.436+0000 7f1518b14640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ae540 0x7f1514151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.436+0000 7f1518b14640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15140ce630 0x7f15141f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.436+0000 7f1518b14640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1514155780 0x7f151415a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.436+0000 7f1518b14640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15140705d0 con 0x7f15140ae540 2023-11-28T22:10:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.436+0000 7f1518b14640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15141f67a0 con 0x7f15140ce630 2023-11-28T22:10:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.436+0000 7f1518b14640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15141f69d0 con 0x7f1514155780 2023-11-28T22:10:30.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.436+0000 7f15137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ae540 0x7f1514151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:30.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.436+0000 7f15137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ae540 0x7f1514151a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44106/0 (socket says 172.21.15.45:44106) 2023-11-28T22:10:30.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.436+0000 7f15137fe640 1 -- 172.21.15.45:0/2422576391 learned_addr learned my addr 172.21.15.45:0/2422576391 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:30.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.437+0000 7f150bfff640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15140ce630 0x7f15141f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:30.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.437+0000 7f1513fff640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1514155780 0x7f151415a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:30.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.437+0000 7f15137fe640 1 -- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1514155780 msgr2=0x7f151415a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:30.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.437+0000 7f15137fe640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1514155780 0x7f151415a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.437+0000 7f15137fe640 1 -- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15140ce630 msgr2=0x7f15141f6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:30.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.437+0000 7f15137fe640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15140ce630 0x7f15141f6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.437+0000 7f15137fe640 1 -- 172.21.15.45:0/2422576391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c009b30 con 0x7f15140ae540 2023-11-28T22:10:30.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.437+0000 7f150bfff640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15140ce630 0x7f15141f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:30.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.437+0000 7f15137fe640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ae540 0x7f1514151a90 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f150400b4a0 tx=0x7f150400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:30.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.438+0000 7f1510ff9640 1 -- 172.21.15.45:0/2422576391 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504010610 con 0x7f15140ae540 2023-11-28T22:10:30.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.438+0000 7f1510ff9640 1 -- 172.21.15.45:0/2422576391 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1504010770 con 0x7f15140ae540 2023-11-28T22:10:30.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.438+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15141f6cb0 con 0x7f15140ae540 2023-11-28T22:10:30.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.438+0000 7f1510ff9640 1 -- 172.21.15.45:0/2422576391 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504010b20 con 0x7f15140ae540 2023-11-28T22:10:30.443 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.438+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151415a930 con 0x7f15140ae540 2023-11-28T22:10:30.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.439+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1514070710 con 0x7f15140ae540 2023-11-28T22:10:30.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.440+0000 7f1510ff9640 1 -- 172.21.15.45:0/2422576391 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1504010c80 con 0x7f15140ae540 2023-11-28T22:10:30.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.441+0000 7f1510ff9640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14f406af60 0x7f14f406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:30.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.441+0000 7f1510ff9640 1 -- 172.21.15.45:0/2422576391 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f1504089530 con 0x7f15140ae540 2023-11-28T22:10:30.444 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.442+0000 7f150bfff640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14f406af60 0x7f14f406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:30.445 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.443+0000 7f150bfff640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14f406af60 0x7f14f406d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1514070270 tx=0x7f14fc008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:30.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.446+0000 7f1510ff9640 1 -- 172.21.15.45:0/2422576391 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1514070710 con 0x7f15140ae540 2023-11-28T22:10:30.641 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.639+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1514070370 con 0x7f14f406af60 2023-11-28T22:10:30.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.645+0000 7f1510ff9640 1 -- 172.21.15.45:0/2422576391 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47844 (secure 0 0 0) 0x7f14fc00ae70 con 0x7f14f406af60 2023-11-28T22:10:30.648 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:30.649 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T22:10:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14f406af60 msgr2=0x7f14f406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14f406af60 0x7f14f406d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1514070270 tx=0x7f14fc008040 comp rx=0 tx=0).stop 2023-11-28T22:10:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ae540 msgr2=0x7f1514151a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ae540 0x7f1514151a90 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f150400b4a0 tx=0x7f150400b960 comp rx=0 tx=0).stop 2023-11-28T22:10:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 shutdown_connections 2023-11-28T22:10:30.653 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1514155780 0x7f151415a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f14f406af60 0x7f14f406d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15140ce630 0x7f15141f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 --2- 172.21.15.45:0/2422576391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15140ae540 0x7f1514151a90 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.648+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 >> 172.21.15.45:0/2422576391 conn(0x7f151406ab30 msgr2=0x7f15140b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.649+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 shutdown_connections 2023-11-28T22:10:30.654 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:30.649+0000 7f1518b14640 1 -- 172.21.15.45:0/2422576391 wait complete. 2023-11-28T22:10:30.669 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-11-28T22:10:28.714553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4855,"ondisk_log_size":4855,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":441756,"kb_used_data":22952,"kb_used_omap":12,"kb_used_meta":418675,"kb_avail":754532964,"statfs":{"total":773094113280,"available":772641755136,"internally_reserved":0,"allocated":23502848,"data_stored":12639968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":428723760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.927007"},"pg_stats":[{"pgid":"1.0","version":"219'4855","reported_seq":5780,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-11-28T22:10:17.209528+0000","last_change":"2023-11-28T20:33:49.677688+0000","last_active":"2023-11-28T22:10:17.209528+0000","last_peered":"2023-11-28T22:10:17.209528+0000","last_clean":"2023-11-28T22:10:17.209528+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T22:10:17.209528+0000","last_undegraded":"2023-11-28T22:10:17.209528+0000","last_fullsized":"2023-11-28T22:10:17.209528+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T20:33:47.962083+0000","last_clean_scrub_stamp":"2023-11-28T20:33:47.962083+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:27:42.126860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4855,"ondisk_log_size":4855,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575995,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74740,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297100,"statfs":{"total":96636764160,"available":96560230400,"internally_reserved":0,"allocated":3383296,"data_stored":2022772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2410000000000001,"15min":1.214},"min":{"1min":0.83799999999999997,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6499999999999999,"5min":1.9019999999999999,"15min":1.9330000000000001},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.218,"5min":1.2150000000000001,"15min":1.26},"min":{"1min":0.75800000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5720000000000001,"5min":2.0190000000000001,"15min":2.6819999999999999},"last":1.105}]},{"osd":1,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.248,"15min":1.2569999999999999},"min":{"1min":0.88600000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6419999999999999,"5min":2.694,"15min":2.694},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.371,"5min":1.2749999999999999,"15min":1.29},"min":{"1min":0.93799999999999994,"5min":0.84199999999999997,"15min":0.80900000000000005},"max":{"1min":1.722,"5min":1.756,"15min":3.0430000000000001},"last":1.0169999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2809999999999999,"15min":1.3180000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.643,"5min":1.754,"15min":2.7480000000000002},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.371,"15min":1.3999999999999999},"min":{"1min":0.93700000000000006,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.8009999999999999,"5min":1.833,"15min":2.2400000000000002},"last":0.90300000000000002}]},{"osd":3,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3240000000000001,"15min":1.345},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.7729999999999999,"5min":1.7969999999999999,"15min":1.7989999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.444,"15min":1.464},"min":{"1min":1.048,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.784,"5min":2.258,"15min":2.9329999999999998},"last":1.0880000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2929999999999999,"15min":1.3280000000000001},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.81299999999999994},"max":{"1min":1.7969999999999999,"5min":1.8100000000000001,"15min":2.6930000000000001},"last":0.621},{"interface":"front","average":{"1min":1.323,"5min":1.333,"15min":1.365},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8580000000000001,"5min":1.905,"15min":2.371},"last":0.77200000000000002}]},{"osd":5,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.77500000000000002},"max":{"1min":1.728,"5min":1.9750000000000001,"15min":3.2229999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.468,"5min":1.48,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.992,"5min":2.0760000000000001,"15min":2.508},"last":0.83099999999999996}]},{"osd":6,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.579,"15min":1.613},"min":{"1min":1.1739999999999999,"5min":1.038,"15min":1.038},"max":{"1min":2.0329999999999999,"5min":2.0720000000000001,"15min":2.4409999999999998},"last":0.996},{"interface":"front","average":{"1min":1.444,"5min":1.5089999999999999,"15min":1.532},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.8500000000000001,"5min":2.2010000000000001,"15min":2.2869999999999999},"last":0.88200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":42228,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1259999999999999,"15min":1.129},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.73699999999999999},"max":{"1min":1.4299999999999999,"5min":1.6679999999999999,"15min":1.772},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.0940000000000001,"15min":1.1299999999999999},"min":{"1min":0.872,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.718,"5min":1.718,"15min":1.7509999999999999},"last":0.77800000000000002}]},{"osd":1,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.292,"15min":1.298},"min":{"1min":1.0509999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.667,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4239999999999999,"15min":1.4350000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.89300000000000002},"max":{"1min":1.6619999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.24}]},{"osd":2,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3440000000000001,"15min":1.3420000000000001},"min":{"1min":0.99199999999999999,"5min":0.89900000000000002,"15min":0.80000000000000004},"max":{"1min":1.579,"5min":2.0779999999999998,"15min":2.4809999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.145,"5min":0.871,"15min":0.85199999999999998},"max":{"1min":1.8009999999999999,"5min":2.1829999999999998,"15min":2.6269999999999998},"last":1.175}]},{"osd":3,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.387,"15min":1.361},"min":{"1min":1.018,"5min":0.92000000000000004,"15min":0.88500000000000001},"max":{"1min":1.7929999999999999,"5min":1.827,"15min":1.9330000000000001},"last":1.357},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3120000000000001,"15min":1.331},"min":{"1min":1.008,"5min":0.94599999999999995,"15min":0.86099999999999999},"max":{"1min":1.6359999999999999,"5min":1.7190000000000001,"15min":1.8859999999999999},"last":1.296}]},{"osd":4,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.327,"15min":1.304},"min":{"1min":0.86399999999999999,"5min":0.81599999999999995,"15min":0.81000000000000005},"max":{"1min":1.9219999999999999,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.3879999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.86199999999999999},"max":{"1min":1.917,"5min":1.917,"15min":2.02},"last":1.429}]},{"osd":5,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.409,"15min":1.3999999999999999},"min":{"1min":0.97299999999999998,"5min":0.94999999999999996,"15min":0.84199999999999997},"max":{"1min":1.772,"5min":1.9990000000000001,"15min":2.081},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.619,"5min":1.5980000000000001,"15min":1.5960000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0270000000000001,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":1.7569999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.544,"15min":1.5369999999999999},"min":{"1min":1.077,"5min":1.0649999999999999,"15min":0.98899999999999999},"max":{"1min":1.9870000000000001,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.875},{"interface":"front","average":{"1min":1.573,"5min":1.504,"15min":1.526},"min":{"1min":1.252,"5min":1.0029999999999999,"15min":0.879},"max":{"1min":2.024,"5min":2.024,"15min":2.2130000000000001},"last":1.554}]}]},{"osd":5,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60272,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311568,"statfs":{"total":96636764160,"available":96575045632,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1539999999999999,"15min":1.173},"min":{"1min":0.71699999999999997,"5min":0.65700000000000003,"15min":0.48499999999999999},"max":{"1min":1.6120000000000001,"5min":1.9039999999999999,"15min":1.9550000000000001},"last":1.498},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2170000000000001,"15min":1.2},"min":{"1min":0.89600000000000002,"5min":0.80600000000000005,"15min":0.65700000000000003},"max":{"1min":1.8069999999999999,"5min":1.889,"15min":1.889},"last":1.2130000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1990000000000001,"15min":1.212},"min":{"1min":0.77200000000000002,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.494,"5min":1.706,"15min":1.8979999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2789999999999999,"15min":1.286},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.54400000000000004},"max":{"1min":1.7370000000000001,"5min":1.7509999999999999,"15min":1.8859999999999999},"last":1.3899999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.3100000000000001,"15min":1.3260000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.60699999999999998},"max":{"1min":1.677,"5min":1.768,"15min":1.867},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.361,"15min":1.3560000000000001},"min":{"1min":0.81799999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.7410000000000001,"5min":1.8899999999999999,"15min":2.044},"last":1.7589999999999999}]},{"osd":3,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4490000000000001,"15min":1.4139999999999999},"min":{"1min":1.1599999999999999,"5min":0.96999999999999997,"15min":0.68899999999999995},"max":{"1min":1.5720000000000001,"5min":1.8180000000000001,"15min":1.988},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.3069999999999999,"15min":1.3380000000000001},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.87},"max":{"1min":1.595,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.4419999999999999}]},{"osd":4,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3720000000000001,"15min":1.377},"min":{"1min":0.873,"5min":0.84899999999999998,"15min":0.72599999999999998},"max":{"1min":1.802,"5min":1.833,"15min":1.833},"last":1.175},{"interface":"front","average":{"1min":1.48,"5min":1.4550000000000001,"15min":1.494},"min":{"1min":1.103,"5min":0.83999999999999997,"15min":0.76200000000000001},"max":{"1min":1.792,"5min":1.8440000000000001,"15min":2.0249999999999999},"last":1.7150000000000001}]},{"osd":6,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4379999999999999,"15min":1.415},"min":{"1min":0.88800000000000001,"5min":0.84099999999999997,"15min":0.80000000000000004},"max":{"1min":1.885,"5min":1.899,"15min":2.0960000000000001},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.5069999999999999,"15min":1.544},"min":{"1min":1.1619999999999999,"5min":0.97199999999999998,"15min":0.94399999999999995},"max":{"1min":1.7849999999999999,"5min":1.982,"15min":1.982},"last":1.6000000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:34 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.46,"15min":1.484},"min":{"1min":1.069,"5min":1.0580000000000001,"15min":0.76100000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.1880000000000002},"last":1.637},{"interface":"front","average":{"1min":1.589,"5min":1.5660000000000001,"15min":1.5740000000000001},"min":{"1min":1.214,"5min":0.877,"15min":0.877},"max":{"1min":1.905,"5min":1.9790000000000001,"15min":2.1970000000000001},"last":1.573}]}]},{"osd":4,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51952,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319888,"statfs":{"total":96636764160,"available":96583565312,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.0900000000000001,"15min":1.131},"min":{"1min":0.79500000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.8149999999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1619999999999999,"15min":1.1579999999999999},"min":{"1min":0.82199999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5940000000000001,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.0580000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.3720000000000001,"15min":1.3839999999999999},"min":{"1min":0.88,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.873,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.383,"15min":1.4119999999999999},"min":{"1min":1.109,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.77,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.998}]},{"osd":2,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.266,"15min":1.266},"min":{"1min":0.95199999999999996,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.7509999999999999,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.3280000000000001,"15min":1.3500000000000001},"min":{"1min":1.034,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.927,"5min":1.927,"15min":1.944},"last":1.444}]},{"osd":3,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3220000000000001,"15min":1.3080000000000001},"min":{"1min":1.002,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8120000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.421,"15min":1.4099999999999999},"min":{"1min":1.1399999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.8169999999999999,"5min":1.929,"15min":2.036},"last":1.4730000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3600000000000001,"15min":1.3280000000000001},"min":{"1min":0.91900000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.8,"5min":2.0030000000000001,"15min":2.0609999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3700000000000001,"15min":1.359},"min":{"1min":0.92000000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7769999999999999,"5min":1.9139999999999999,"15min":2.1659999999999999},"last":1.284}]},{"osd":6,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.492,"15min":1.446},"min":{"1min":1.01,"5min":0.70899999999999996,"15min":0.61499999999999999},"max":{"1min":2.004,"5min":2.004,"15min":2.097},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.661,"5min":1.615,"15min":1.601},"min":{"1min":1.296,"5min":0.75,"15min":0.75},"max":{"1min":2.1120000000000001,"5min":2.1150000000000002,"15min":2.23},"last":1.3560000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.5,"15min":1.4950000000000001},"min":{"1min":1.2290000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8680000000000001,"5min":2.0190000000000001,"15min":2.3340000000000001},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.611,"15min":1.5920000000000001},"min":{"1min":1.3759999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1659999999999999,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.5920000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56120,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315720,"statfs":{"total":96636764160,"available":96579297280,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.94299999999999995,"15min":0.95599999999999996},"min":{"1min":0.72199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4530000000000001,"5min":2.0760000000000001,"15min":2.0950000000000002},"last":1.034},{"interface":"front","average":{"1min":1.163,"5min":1.135,"15min":1.1339999999999999},"min":{"1min":0.51400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.3879999999999999},"last":1.4339999999999999}]},{"osd":1,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.012,"15min":1.0289999999999999},"min":{"1min":0.70799999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.677,"5min":1.677,"15min":1.698},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.214,"15min":1.256},"min":{"1min":0.77600000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.718,"5min":1.956,"15min":2.6019999999999999},"last":1.5049999999999999}]},{"osd":2,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2629999999999999,"15min":1.3},"min":{"1min":0.86399999999999999,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5529999999999999,"5min":2.105,"15min":2.327},"last":1.089},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.369,"15min":1.397},"min":{"1min":0.70999999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.782,"5min":1.9119999999999999,"15min":2.218},"last":0.73299999999999998}]},{"osd":4,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.325,"15min":1.379},"min":{"1min":1.0029999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.706,"5min":1.948,"15min":2.0379999999999998},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.474,"5min":1.446,"15min":1.482},"min":{"1min":1.0189999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.165},"last":1.3240000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.5329999999999999,"15min":1.55},"min":{"1min":1.331,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.98,"5min":2.2989999999999999,"15min":2.4329999999999998},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.498,"15min":1.5109999999999999},"min":{"1min":1.0760000000000001,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":1.8240000000000001,"5min":2.1389999999999998,"15min":2.2639999999999998},"last":1.23}]},{"osd":6,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5580000000000001,"15min":1.5720000000000001},"min":{"1min":1.1990000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.9830000000000001,"5min":2.0640000000000001,"15min":2.2109999999999999},"last":1.8700000000000001},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.637,"15min":1.6659999999999999},"min":{"1min":1.425,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":2.0779999999999998,"5min":2.379,"15min":2.5329999999999999},"last":1.8280000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.629,"15min":1.6359999999999999},"min":{"1min":1.1579999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0310000000000001,"5min":2.2599999999999998,"15min":2.266},"last":1.552},{"interface":"front","average":{"1min":1.738,"5min":1.6919999999999999,"15min":1.694},"min":{"1min":1.3500000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.129,"5min":2.1800000000000002,"15min":2.3029999999999999},"last":1.6819999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51956,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319884,"statfs":{"total":96636764160,"available":96583561216,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.96699999999999997,"15min":0.97399999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2350000000000001,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.095,"15min":1.141},"min":{"1min":0.496,"5min":0.495,"15min":0.495},"max":{"1min":1.6399999999999999,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.343}]},{"osd":1,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.98799999999999999,"15min":1.0069999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.3400000000000001,"5min":2.097,"15min":2.097},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.157,"5min":1.153,"15min":1.2190000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.5249999999999999,"5min":1.625,"15min":1.8149999999999999},"last":0.70699999999999996}]},{"osd":3,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1919999999999999,"15min":1.24},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.925,"5min":1.925,"15min":1.9330000000000001},"last":1.272},{"interface":"front","average":{"1min":1.163,"5min":1.3069999999999999,"15min":1.3149999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.597,"5min":1.849,"15min":2.2210000000000001},"last":0.82099999999999995}]},{"osd":4,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.425,"15min":1.4610000000000001},"min":{"1min":0.96799999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.835,"5min":1.956,"15min":2.0640000000000001},"last":1.234},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4570000000000001,"15min":1.47},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":4.4619999999999997,"5min":4.4619999999999997,"15min":4.4619999999999997},"last":1.133}]},{"osd":5,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.49,"15min":1.5049999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.877,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.403,"5min":1.5269999999999999,"15min":1.5589999999999999},"min":{"1min":0.80000000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.7669999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.89500000000000002}]},{"osd":6,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.552,"15min":1.599},"min":{"1min":0.84299999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":1.409},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.6220000000000001,"15min":1.645},"min":{"1min":0.82899999999999996,"5min":0.751,"15min":0.751},"max":{"1min":1.9610000000000001,"5min":2.1640000000000001,"15min":2.4020000000000001},"last":0.94199999999999995}]},{"osd":7,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.5860000000000001,"15min":1.6100000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.9650000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.069},{"interface":"front","average":{"1min":1.651,"5min":1.6359999999999999,"15min":1.6930000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":3.7789999999999999,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.4650000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51952,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319888,"statfs":{"total":96636764160,"available":96583565312,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92300000000000004,"15min":0.92000000000000004},"min":{"1min":0.55000000000000004,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.417,"5min":1.6240000000000001,"15min":1.8839999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.153,"15min":1.1579999999999999},"min":{"1min":0.64800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.546,"5min":1.9410000000000001,"15min":2.3900000000000001},"last":1.006}]},{"osd":2,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.147,"15min":1.1599999999999999},"min":{"1min":0.623,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.464,"5min":1.984,"15min":1.9890000000000001},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2669999999999999,"15min":1.2729999999999999},"min":{"1min":0.58299999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.77,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":1.113}]},{"osd":3,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.363,"15min":1.3999999999999999},"min":{"1min":0.71499999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.722,"5min":2.2040000000000002,"15min":2.2320000000000002},"last":1.242},{"interface":"front","average":{"1min":1.175,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.79000000000000004,"5min":0.57799999999999996,"15min":0.46800000000000003},"max":{"1min":1.641,"5min":1.786,"15min":2.4359999999999999},"last":0.88500000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4019999999999999,"15min":1.427},"min":{"1min":1.089,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3559999999999999},"last":1.298},{"interface":"front","average":{"1min":1.399,"5min":1.4099999999999999,"15min":1.4379999999999999},"min":{"1min":0.99199999999999999,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.3079999999999998},"last":1.798}]},{"osd":5,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4750000000000001,"15min":1.4930000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.77800000000000002},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.919},{"interface":"front","average":{"1min":1.544,"5min":1.5880000000000001,"15min":1.6120000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.925,"5min":2.3580000000000001,"15min":2.448},"last":1.8779999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.522,"15min":1.516},"min":{"1min":1.028,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.5249999999999999},"last":1.643},{"interface":"front","average":{"1min":1.609,"5min":1.579,"15min":1.593},"min":{"1min":1.175,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.246,"5min":2.246,"15min":2.4550000000000001},"last":2.1040000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.665,"15min":1.6950000000000001},"min":{"1min":1.131,"5min":1.125,"15min":1.125},"max":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":1.97},{"interface":"front","average":{"1min":1.571,"5min":1.5740000000000001,"15min":1.591},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":2.0369999999999999,"5min":2.2730000000000001,"15min":2.8460000000000001},"last":1.7110000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575996,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52536,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319304,"statfs":{"total":96636764160,"available":96582967296,"internally_reserved":0,"allocated":3383296,"data_stored":2022772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.93799999999999994,"15min":0.93200000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.728,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.1439999999999999,"15min":1.1830000000000001},"min":{"1min":0.751,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.2869999999999999},"last":1.288}]},{"osd":2,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1279999999999999,"15min":1.1779999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.1709999999999998},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.256,"15min":1.244},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":1.7090000000000001}]},{"osd":3,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.1830000000000001,"15min":1.179},"min":{"1min":0.80100000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.655,"5min":1.8009999999999999,"15min":1.986},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.387,"15min":1.3819999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":1.516}]},{"osd":4,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.484,"15min":1.4750000000000001},"min":{"1min":1.016,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.395,"5min":2.395,"15min":2.395},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.411,"15min":1.4330000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.8109999999999999,"5min":2.149,"15min":2.254},"last":1.675}]},{"osd":5,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.48,"15min":1.4870000000000001},"min":{"1min":0.94999999999999996,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.8959999999999999,"5min":2.0819999999999999,"15min":2.1520000000000001},"last":1.754},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.585,"15min":1.5640000000000001},"min":{"1min":1.1970000000000001,"5min":1.05,"15min":1.05},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.3359999999999999},"last":1.7969999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.675,"15min":1.639},"min":{"1min":1.202,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.9359999999999999},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6259999999999999,"15min":1.611},"min":{"1min":0.97399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.8540000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.6990000000000001,"15min":1.6859999999999999},"min":{"1min":1.034,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.6060000000000001,"15min":1.625},"min":{"1min":1.177,"5min":0.97899999999999998,"15min":0.97899999999999998},"max":{"1min":1.984,"5min":2.4089999999999998,"15min":2.4180000000000001},"last":1.4510000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T22:10:30.673 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-11-28T20:33:47.962083+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=20, tm_min=33, tm_sec=47, tm_wday=1, tm_yday=332, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=22, tm_min=10, tm_sec=23, tm_wday=1, tm_yday=332, tm_isdst=0) 2023-11-28T22:10:30.673 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-28T22:10:50.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T22:10:50.918 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.916+0000 7ff01803d640 1 Processor -- start 2023-11-28T22:10:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.916+0000 7ff01803d640 1 -- start start 2023-11-28T22:10:50.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.918+0000 7ff01803d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0101438b0 0x7ff0100b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:50.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.918+0000 7ff01803d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01015dcc0 con 0x7ff0100b24e0 2023-11-28T22:10:50.920 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.918+0000 7ff01803d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01015e840 con 0x7ff0100b1340 2023-11-28T22:10:50.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.918+0000 7ff01803d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01015df10 con 0x7ff0101438b0 2023-11-28T22:10:50.922 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.918+0000 7ff0155d8640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff0100b24e0 0x7ff0100b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58848/0 (socket says 172.21.15.45:58848) 2023-11-28T22:10:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.918+0000 7ff0155d8640 1 -- 172.21.15.45:0/1591569658 learned_addr learned my addr 172.21.15.45:0/1591569658 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff015dd9640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0101438b0 0x7ff0100b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417196477 0 0) 0x7ff01015dcc0 con 0x7ff0100b24e0 2023-11-28T22:10:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feffc003520 con 0x7ff0100b24e0 2023-11-28T22:10:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3597593007 0 0) 0x7feffc003520 con 0x7ff0100b24e0 2023-11-28T22:10:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0101438b0 msgr2=0x7ff0100b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0101438b0 0x7ff0100b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 >> v1:172.21.15.190:6789/0 conn(0x7ff0100b1340 legacy=0x7ff0100b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01015e1f0 con 0x7ff0100b24e0 2023-11-28T22:10:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7ff000001670 con 0x7ff0100b24e0 2023-11-28T22:10:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 >> v1:172.21.15.45:6789/0 conn(0x7ff0100b24e0 legacy=0x7ff0100b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0101438b0 0x7feffc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.919+0000 7ff014dd7640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7feffc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff014dd7640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc004d00 0x7feffc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01015df10 con 0x7feffc003f20 2023-11-28T22:10:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc00bbc0 con 0x7feffc004d00 2023-11-28T22:10:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc00df90 con 0x7ff0101438b0 2023-11-28T22:10:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff0155d8640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7feffc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff015dd9640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0101438b0 0x7feffc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff0165da640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc004d00 0x7feffc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff0155d8640 1 -- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0101438b0 msgr2=0x7feffc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff0155d8640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff0101438b0 0x7feffc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff0155d8640 1 -- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc004d00 msgr2=0x7feffc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff0155d8640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc004d00 0x7feffc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff0155d8640 1 -- 172.21.15.45:0/1591569658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc003520 con 0x7feffc003f20 2023-11-28T22:10:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.920+0000 7ff0165da640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc004d00 0x7feffc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:50.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.921+0000 7ff0155d8640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7feffc0047c0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7ff00000c210 tx=0x7feffc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683055f759f7f183 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff000005ad0 con 0x7feffc003f20 2023-11-28T22:10:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff000005c30 con 0x7feffc003f20 2023-11-28T22:10:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff014dd7640 1 -- 172.21.15.45:0/1591569658 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00002cad0 con 0x7feffc003f20 2023-11-28T22:10:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff01803d640 1 -- 172.21.15.45:0/1591569658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 msgr2=0x7feffc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff01803d640 1 --2- 172.21.15.45:0/1591569658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7feffc0047c0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7ff00000c210 tx=0x7feffc0057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff0165da640 1 -- 172.21.15.45:0/1591569658 reap_dead start 2023-11-28T22:10:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff01803d640 1 -- 172.21.15.45:0/1591569658 shutdown_connections 2023-11-28T22:10:50.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff01803d640 1 -- 172.21.15.45:0/1591569658 >> 172.21.15.45:0/1591569658 conn(0x7ff01006a500 msgr2=0x7ff01006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:50.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff01803d640 1 -- 172.21.15.45:0/1591569658 shutdown_connections 2023-11-28T22:10:50.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.922+0000 7ff01803d640 1 -- 172.21.15.45:0/1591569658 wait complete. 2023-11-28T22:10:50.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.924+0000 7ff01803d640 1 Processor -- start 2023-11-28T22:10:50.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.924+0000 7ff01803d640 1 -- start start 2023-11-28T22:10:50.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.925+0000 7ff01803d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7ff0100aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:50.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.925+0000 7ff01803d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff010159800 0x7ff010158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.925+0000 7ff01803d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff01015d9a0 0x7ff0101e3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.925+0000 7ff01803d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff010159c90 con 0x7feffc003f20 2023-11-28T22:10:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.925+0000 7ff01803d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff010159520 con 0x7ff010159800 2023-11-28T22:10:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.925+0000 7ff01803d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0100cc5e0 con 0x7ff01015d9a0 2023-11-28T22:10:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff015dd9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7ff0100aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff015dd9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7ff0100aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55978/0 (socket says 172.21.15.45:55978) 2023-11-28T22:10:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff015dd9640 1 -- 172.21.15.45:0/3640622392 learned_addr learned my addr 172.21.15.45:0/3640622392 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff0165da640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff01015d9a0 0x7ff0101e3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:50.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff0155d8640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff010159800 0x7ff010158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:50.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff015dd9640 1 -- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff01015d9a0 msgr2=0x7ff0101e3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff015dd9640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff01015d9a0 0x7ff0101e3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff015dd9640 1 -- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff010159800 msgr2=0x7ff010158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff015dd9640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff010159800 0x7ff010158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff015dd9640 1 -- 172.21.15.45:0/3640622392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0101e3fb0 con 0x7feffc003f20 2023-11-28T22:10:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff0165da640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff01015d9a0 0x7ff0101e3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T22:10:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.926+0000 7ff015dd9640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7ff0100aec10 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7ff00c011600 tx=0x7ff00c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:50.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.927+0000 7ff0067fc640 1 -- 172.21.15.45:0/3640622392 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c00f920 con 0x7feffc003f20 2023-11-28T22:10:50.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.927+0000 7ff0067fc640 1 -- 172.21.15.45:0/3640622392 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c00fa80 con 0x7feffc003f20 2023-11-28T22:10:50.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.927+0000 7ff0067fc640 1 -- 172.21.15.45:0/3640622392 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c00fdf0 con 0x7feffc003f20 2023-11-28T22:10:50.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.927+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0100b4d70 con 0x7feffc003f20 2023-11-28T22:10:50.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.927+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0101ef890 con 0x7feffc003f20 2023-11-28T22:10:50.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.928+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01015d5e0 con 0x7feffc003f20 2023-11-28T22:10:50.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.929+0000 7ff0067fc640 1 -- 172.21.15.45:0/3640622392 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff00c015460 con 0x7feffc003f20 2023-11-28T22:10:50.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.930+0000 7ff0067fc640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feff406af60 0x7feff406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:50.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.930+0000 7ff0067fc640 1 -- 172.21.15.45:0/3640622392 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7ff00c089d50 con 0x7feffc003f20 2023-11-28T22:10:50.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.931+0000 7ff0155d8640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feff406af60 0x7feff406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:50.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.932+0000 7ff0155d8640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feff406af60 0x7feff406d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff000034000 tx=0x7ff000009030 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:50.938 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:50.935+0000 7ff0067fc640 1 -- 172.21.15.45:0/3640622392 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff00c08d030 con 0x7feffc003f20 2023-11-28T22:10:51.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.127+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 --> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff010159690 con 0x7feff406af60 2023-11-28T22:10:51.136 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.133+0000 7ff0067fc640 1 -- 172.21.15.45:0/3640622392 <== mgr.8884 v2:172.21.15.190:6832/334700924 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47659 (secure 0 0 0) 0x7ff010159690 con 0x7feff406af60 2023-11-28T22:10:51.137 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:51.138 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-11-28T22:10:51.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feff406af60 msgr2=0x7feff406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feff406af60 0x7feff406d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff000034000 tx=0x7ff000009030 comp rx=0 tx=0).stop 2023-11-28T22:10:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 msgr2=0x7ff0100aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.141 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7ff0100aec10 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7ff00c011600 tx=0x7ff00c011ac0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 shutdown_connections 2023-11-28T22:10:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff01015d9a0 0x7ff0101e3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7feff406af60 0x7feff406d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.142 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff010159800 0x7ff010158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 --2- 172.21.15.45:0/3640622392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feffc003f20 0x7ff0100aec10 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.143 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 >> 172.21.15.45:0/3640622392 conn(0x7ff01006a500 msgr2=0x7ff01013e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 shutdown_connections 2023-11-28T22:10:51.144 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.136+0000 7ff01803d640 1 -- 172.21.15.45:0/3640622392 wait complete. 2023-11-28T22:10:51.159 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-11-28T22:10:50.720937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4856,"ondisk_log_size":4856,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":441744,"kb_used_data":22952,"kb_used_omap":12,"kb_used_meta":418675,"kb_avail":754532976,"statfs":{"total":773094113280,"available":772641767424,"internally_reserved":0,"allocated":23502848,"data_stored":12639968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":428723760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003413"},"pg_stats":[{"pgid":"1.0","version":"222'4856","reported_seq":5790,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-11-28T22:10:30.979008+0000","last_change":"2023-11-28T22:10:30.979081+0000","last_active":"2023-11-28T22:10:30.979008+0000","last_peered":"2023-11-28T22:10:30.979008+0000","last_clean":"2023-11-28T22:10:30.979008+0000","last_became_active":"2023-11-28T20:33:49.676668+0000","last_became_peered":"2023-11-28T20:33:49.676668+0000","last_unstale":"2023-11-28T22:10:30.979008+0000","last_undegraded":"2023-11-28T22:10:30.979008+0000","last_fullsized":"2023-11-28T22:10:30.979008+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'4856","last_scrub_stamp":"2023-11-28T22:10:30.978856+0000","last_deep_scrub":"222'4856","last_deep_scrub_stamp":"2023-11-28T22:10:30.978856+0000","last_clean_scrub_stamp":"2023-11-28T22:10:30.978856+0000","objects_scrubbed":2,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T09:58:10.112426+0000","scrub_duration":0.014139554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2245,"num_read_kb":4657,"num_write":591,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4856,"ondisk_log_size":4856,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575999,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74736,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297104,"statfs":{"total":96636764160,"available":96560234496,"internally_reserved":0,"allocated":3383296,"data_stored":2022772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2410000000000001,"15min":1.214},"min":{"1min":0.83799999999999997,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6499999999999999,"5min":1.9019999999999999,"15min":1.9330000000000001},"last":1.113},{"interface":"front","average":{"1min":1.218,"5min":1.2150000000000001,"15min":1.26},"min":{"1min":0.75800000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5720000000000001,"5min":2.0190000000000001,"15min":2.6819999999999999},"last":1.286}]},{"osd":1,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.248,"15min":1.2569999999999999},"min":{"1min":0.88600000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6419999999999999,"5min":2.694,"15min":2.694},"last":1.371},{"interface":"front","average":{"1min":1.371,"5min":1.2749999999999999,"15min":1.29},"min":{"1min":0.93799999999999994,"5min":0.84199999999999997,"15min":0.80900000000000005},"max":{"1min":1.722,"5min":1.756,"15min":3.0430000000000001},"last":1.48}]},{"osd":2,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2809999999999999,"15min":1.3180000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.643,"5min":1.754,"15min":2.7480000000000002},"last":1.804},{"interface":"front","average":{"1min":1.409,"5min":1.371,"15min":1.3999999999999999},"min":{"1min":0.93700000000000006,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.8009999999999999,"5min":1.833,"15min":2.2400000000000002},"last":1.5249999999999999}]},{"osd":3,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3240000000000001,"15min":1.345},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.7729999999999999,"5min":1.7969999999999999,"15min":1.7989999999999999},"last":1.74},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.444,"15min":1.464},"min":{"1min":1.048,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.784,"5min":2.258,"15min":2.9329999999999998},"last":1.7030000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2929999999999999,"15min":1.3280000000000001},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.81299999999999994},"max":{"1min":1.7969999999999999,"5min":1.8100000000000001,"15min":2.6930000000000001},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.333,"15min":1.365},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8580000000000001,"5min":1.905,"15min":2.371},"last":1.5720000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.77500000000000002},"max":{"1min":1.728,"5min":1.9750000000000001,"15min":3.2229999999999999},"last":1.8759999999999999},{"interface":"front","average":{"1min":1.468,"5min":1.48,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.992,"5min":2.0760000000000001,"15min":2.508},"last":1.677}]},{"osd":6,"last update":"Tue Nov 28 22:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.579,"15min":1.613},"min":{"1min":1.1739999999999999,"5min":1.038,"15min":1.038},"max":{"1min":2.0329999999999999,"5min":2.0720000000000001,"15min":2.4409999999999998},"last":1.768},{"interface":"front","average":{"1min":1.444,"5min":1.5089999999999999,"15min":1.532},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.8500000000000001,"5min":2.2010000000000001,"15min":2.2869999999999999},"last":1.841}]}]},{"osd":6,"up_from":13,"seq":55834575999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":42228,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1259999999999999,"15min":1.129},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.73699999999999999},"max":{"1min":1.4299999999999999,"5min":1.6679999999999999,"15min":1.772},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.0940000000000001,"15min":1.1299999999999999},"min":{"1min":0.872,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.718,"5min":1.718,"15min":1.7509999999999999},"last":1.5509999999999999}]},{"osd":1,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.292,"15min":1.298},"min":{"1min":1.0509999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.667,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.077},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4239999999999999,"15min":1.4350000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.89300000000000002},"max":{"1min":1.6619999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.4450000000000001}]},{"osd":2,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3440000000000001,"15min":1.3420000000000001},"min":{"1min":0.99199999999999999,"5min":0.89900000000000002,"15min":0.80000000000000004},"max":{"1min":1.579,"5min":2.0779999999999998,"15min":2.4809999999999999},"last":1.274},{"interface":"front","average":{"1min":1.448,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.145,"5min":0.871,"15min":0.85199999999999998},"max":{"1min":1.8009999999999999,"5min":2.1829999999999998,"15min":2.6269999999999998},"last":1.3939999999999999}]},{"osd":3,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.387,"15min":1.361},"min":{"1min":1.018,"5min":0.92000000000000004,"15min":0.88500000000000001},"max":{"1min":1.7929999999999999,"5min":1.827,"15min":1.9330000000000001},"last":1.345},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3120000000000001,"15min":1.331},"min":{"1min":1.008,"5min":0.94599999999999995,"15min":0.86099999999999999},"max":{"1min":1.6359999999999999,"5min":1.7190000000000001,"15min":1.8859999999999999},"last":0.92400000000000004}]},{"osd":4,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.327,"15min":1.304},"min":{"1min":0.86399999999999999,"5min":0.81599999999999995,"15min":0.81000000000000005},"max":{"1min":1.9219999999999999,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.3879999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.86199999999999999},"max":{"1min":1.917,"5min":1.917,"15min":2.02},"last":1.2170000000000001}]},{"osd":5,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.409,"15min":1.3999999999999999},"min":{"1min":0.97299999999999998,"5min":0.94999999999999996,"15min":0.84199999999999997},"max":{"1min":1.772,"5min":1.9990000000000001,"15min":2.081},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.619,"5min":1.5980000000000001,"15min":1.5960000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0270000000000001,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":1.488}]},{"osd":7,"last update":"Tue Nov 28 22:09:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.544,"15min":1.5369999999999999},"min":{"1min":1.077,"5min":1.0649999999999999,"15min":0.98899999999999999},"max":{"1min":1.9870000000000001,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.633},{"interface":"front","average":{"1min":1.573,"5min":1.504,"15min":1.526},"min":{"1min":1.252,"5min":1.0029999999999999,"15min":0.879},"max":{"1min":2.024,"5min":2.024,"15min":2.2130000000000001},"last":1.135}]}]},{"osd":5,"up_from":13,"seq":55834575999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60272,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311568,"statfs":{"total":96636764160,"available":96575045632,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1499999999999999,"15min":1.1739999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.48499999999999999},"max":{"1min":1.754,"5min":1.9039999999999999,"15min":1.9550000000000001},"last":1.581},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.206,"15min":1.1890000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.7110000000000001,"5min":1.889,"15min":1.889},"last":1.1100000000000001}]},{"osd":1,"last update":"Tue Nov 28 22:10:35 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.177,"15min":1.21},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.381,"5min":1.706,"15min":1.8979999999999999},"last":1.298},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.252,"15min":1.276},"min":{"1min":0.84199999999999997,"5min":0.57699999999999996,"15min":0.54400000000000004},"max":{"1min":1.873,"5min":1.873,"15min":1.8859999999999999},"last":1.3460000000000001}]},{"osd":2,"last update":"Tue Nov 28 22:10:35 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.29,"15min":1.3160000000000001},"min":{"1min":0.89600000000000002,"5min":0.76200000000000001,"15min":0.60699999999999998},"max":{"1min":1.587,"5min":1.768,"15min":1.7969999999999999},"last":1.212},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.321,"15min":1.347},"min":{"1min":0.89400000000000002,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.7589999999999999,"5min":1.77,"15min":2.0289999999999999},"last":1.3819999999999999}]},{"osd":3,"last update":"Tue Nov 28 22:10:35 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.407,"15min":1.4059999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.68899999999999995},"max":{"1min":1.7789999999999999,"5min":1.8180000000000001,"15min":1.988},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.3049999999999999,"15min":1.331},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.671,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.488}]},{"osd":4,"last update":"Tue Nov 28 22:10:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3799999999999999,"15min":1.3759999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.72599999999999998},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":1.631},{"interface":"front","average":{"1min":1.419,"5min":1.4530000000000001,"15min":1.49},"min":{"1min":0.93000000000000005,"5min":0.86399999999999999,"15min":0.83899999999999997},"max":{"1min":1.944,"5min":1.944,"15min":2.0249999999999999},"last":0.84299999999999997}]},{"osd":6,"last update":"Tue Nov 28 22:10:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4470000000000001,"15min":1.415},"min":{"1min":0.99199999999999999,"5min":0.88800000000000001,"15min":0.80000000000000004},"max":{"1min":1.728,"5min":1.885,"15min":2.0960000000000001},"last":1.264},{"interface":"front","average":{"1min":1.524,"5min":1.5169999999999999,"15min":1.5429999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.9239999999999999,"5min":1.982,"15min":1.982},"last":1.718}]},{"osd":7,"last update":"Tue Nov 28 22:10:35 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.4650000000000001,"15min":1.4830000000000001},"min":{"1min":1.0649999999999999,"5min":1.0580000000000001,"15min":0.76100000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.1880000000000002},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.5640000000000001,"15min":1.5680000000000001},"min":{"1min":1.139,"5min":1.139,"15min":0.877},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":2.1970000000000001},"last":1.5409999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51952,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319888,"statfs":{"total":96636764160,"available":96583565312,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0900000000000001,"15min":1.123},"min":{"1min":0.58899999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.095,"5min":1.1539999999999999,"15min":1.155},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.492,"5min":2.492,"15min":2.492},"last":0.94899999999999995}]},{"osd":1,"last update":"Tue Nov 28 22:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3740000000000001,"15min":1.3879999999999999},"min":{"1min":0.69599999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6859999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.3740000000000001,"15min":1.4039999999999999},"min":{"1min":0.81000000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.333}]},{"osd":2,"last update":"Tue Nov 28 22:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.2909999999999999,"15min":1.272},"min":{"1min":0.86299999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.3360000000000001,"15min":1.3500000000000001},"min":{"1min":0.95399999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.216}]},{"osd":3,"last update":"Tue Nov 28 22:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3280000000000001,"15min":1.3100000000000001},"min":{"1min":0.79200000000000004,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.3799999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.452,"15min":1.415},"min":{"1min":0.96099999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.524}]},{"osd":5,"last update":"Tue Nov 28 22:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.367,"15min":1.329},"min":{"1min":0.82499999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.909,"5min":2.0030000000000001,"15min":2.0609999999999999},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.381,"15min":1.359},"min":{"1min":0.96699999999999997,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3399999999999999},"last":1.1830000000000001}]},{"osd":6,"last update":"Tue Nov 28 22:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.508,"15min":1.458},"min":{"1min":0.98799999999999999,"5min":0.70899999999999996,"15min":0.61499999999999999},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6299999999999999,"15min":1.6020000000000001},"min":{"1min":1.0009999999999999,"5min":0.75,"15min":0.75},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.23},"last":1.5640000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.504,"15min":1.4950000000000001},"min":{"1min":1.254,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.448,"5min":2.448,"15min":2.448},"last":1.373},{"interface":"front","average":{"1min":1.643,"5min":1.6299999999999999,"15min":1.599},"min":{"1min":1.123,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.282}]}]},{"osd":3,"up_from":13,"seq":55834576000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56120,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315720,"statfs":{"total":96636764160,"available":96579297280,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.94299999999999995,"15min":0.95599999999999996},"min":{"1min":0.72199999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4530000000000001,"5min":2.0760000000000001,"15min":2.0950000000000002},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.135,"15min":1.1339999999999999},"min":{"1min":0.51400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.3879999999999999},"last":1.238}]},{"osd":1,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.012,"15min":1.0289999999999999},"min":{"1min":0.70799999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.677,"5min":1.677,"15min":1.698},"last":1.294},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.214,"15min":1.256},"min":{"1min":0.77600000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.718,"5min":1.956,"15min":2.6019999999999999},"last":1.3220000000000001}]},{"osd":2,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2629999999999999,"15min":1.3},"min":{"1min":0.86399999999999999,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5529999999999999,"5min":2.105,"15min":2.327},"last":1.161},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.369,"15min":1.397},"min":{"1min":0.70999999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.782,"5min":1.9119999999999999,"15min":2.218},"last":1.8049999999999999}]},{"osd":4,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.325,"15min":1.379},"min":{"1min":1.0029999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.706,"5min":1.948,"15min":2.0379999999999998},"last":1.409},{"interface":"front","average":{"1min":1.474,"5min":1.446,"15min":1.482},"min":{"1min":1.0189999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.165},"last":1.5449999999999999}]},{"osd":5,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.5329999999999999,"15min":1.55},"min":{"1min":1.331,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.98,"5min":2.2989999999999999,"15min":2.4329999999999998},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.498,"15min":1.5109999999999999},"min":{"1min":1.0760000000000001,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":1.8240000000000001,"5min":2.1389999999999998,"15min":2.2639999999999998},"last":1.6040000000000001}]},{"osd":6,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5580000000000001,"15min":1.5720000000000001},"min":{"1min":1.1990000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.9830000000000001,"5min":2.0640000000000001,"15min":2.2109999999999999},"last":1.7350000000000001},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.637,"15min":1.6659999999999999},"min":{"1min":1.425,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":2.0779999999999998,"5min":2.379,"15min":2.5329999999999999},"last":2.1600000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:09:53 2023","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.629,"15min":1.6359999999999999},"min":{"1min":1.1579999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0310000000000001,"5min":2.2599999999999998,"15min":2.266},"last":1.766},{"interface":"front","average":{"1min":1.738,"5min":1.6919999999999999,"15min":1.694},"min":{"1min":1.3500000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.129,"5min":2.1800000000000002,"15min":2.3029999999999999},"last":2.0739999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51956,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319884,"statfs":{"total":96636764160,"available":96583561216,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.96699999999999997,"15min":0.97399999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2350000000000001,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.095,"15min":1.141},"min":{"1min":0.496,"5min":0.495,"15min":0.495},"max":{"1min":1.6399999999999999,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.2949999999999999}]},{"osd":1,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.98799999999999999,"15min":1.0069999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.3400000000000001,"5min":2.097,"15min":2.097},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.157,"5min":1.153,"15min":1.2190000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.5249999999999999,"5min":1.625,"15min":1.8149999999999999},"last":1.1339999999999999}]},{"osd":3,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1919999999999999,"15min":1.24},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.925,"5min":1.925,"15min":1.9330000000000001},"last":1.196},{"interface":"front","average":{"1min":1.163,"5min":1.3069999999999999,"15min":1.3149999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.597,"5min":1.849,"15min":2.2210000000000001},"last":1.226}]},{"osd":4,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.425,"15min":1.4610000000000001},"min":{"1min":0.96799999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.835,"5min":1.956,"15min":2.0640000000000001},"last":1.321},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4570000000000001,"15min":1.47},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":4.4619999999999997,"5min":4.4619999999999997,"15min":4.4619999999999997},"last":1.5449999999999999}]},{"osd":5,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.49,"15min":1.5049999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.877,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.361},{"interface":"front","average":{"1min":1.403,"5min":1.5269999999999999,"15min":1.5589999999999999},"min":{"1min":0.80000000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.7669999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.411}]},{"osd":6,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.552,"15min":1.599},"min":{"1min":0.84299999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.6220000000000001,"15min":1.645},"min":{"1min":0.82899999999999996,"5min":0.751,"15min":0.751},"max":{"1min":1.9610000000000001,"5min":2.1640000000000001,"15min":2.4020000000000001},"last":1.5209999999999999}]},{"osd":7,"last update":"Tue Nov 28 22:10:19 2023","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.5860000000000001,"15min":1.6100000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.9650000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.698},{"interface":"front","average":{"1min":1.651,"5min":1.6359999999999999,"15min":1.6930000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":3.7789999999999999,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.75}]}]},{"osd":1,"up_from":13,"seq":55834576000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51952,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319888,"statfs":{"total":96636764160,"available":96583565312,"internally_reserved":0,"allocated":2789376,"data_stored":1432404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92300000000000004,"15min":0.92000000000000004},"min":{"1min":0.55000000000000004,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.417,"5min":1.6240000000000001,"15min":1.8839999999999999},"last":1.143},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.153,"15min":1.1579999999999999},"min":{"1min":0.64800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.546,"5min":1.9410000000000001,"15min":2.3900000000000001},"last":1.3420000000000001}]},{"osd":2,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.147,"15min":1.1599999999999999},"min":{"1min":0.623,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.464,"5min":1.984,"15min":1.9890000000000001},"last":1.23},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2669999999999999,"15min":1.2729999999999999},"min":{"1min":0.58299999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.77,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.997}]},{"osd":3,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.363,"15min":1.3999999999999999},"min":{"1min":0.71499999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.722,"5min":2.2040000000000002,"15min":2.2320000000000002},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.79000000000000004,"5min":0.57799999999999996,"15min":0.46800000000000003},"max":{"1min":1.641,"5min":1.786,"15min":2.4359999999999999},"last":1.079}]},{"osd":4,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4019999999999999,"15min":1.427},"min":{"1min":1.089,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3559999999999999},"last":1.286},{"interface":"front","average":{"1min":1.399,"5min":1.4099999999999999,"15min":1.4379999999999999},"min":{"1min":0.99199999999999999,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.3079999999999998},"last":1.5129999999999999}]},{"osd":5,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4750000000000001,"15min":1.4930000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.77800000000000002},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.544,"5min":1.5880000000000001,"15min":1.6120000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.925,"5min":2.3580000000000001,"15min":2.448},"last":1.5349999999999999}]},{"osd":6,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.522,"15min":1.516},"min":{"1min":1.028,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.5249999999999999},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.609,"5min":1.579,"15min":1.593},"min":{"1min":1.175,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.246,"5min":2.246,"15min":2.4550000000000001},"last":1.377}]},{"osd":7,"last update":"Tue Nov 28 22:10:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.665,"15min":1.6950000000000001},"min":{"1min":1.131,"5min":1.125,"15min":1.125},"max":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":1.466},{"interface":"front","average":{"1min":1.571,"5min":1.5740000000000001,"15min":1.591},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":2.0369999999999999,"5min":2.2730000000000001,"15min":2.8460000000000001},"last":1.421}]}]},{"osd":0,"up_from":13,"seq":55834576000,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52528,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319312,"statfs":{"total":96636764160,"available":96582975488,"internally_reserved":0,"allocated":3383296,"data_stored":2022772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.93799999999999994,"15min":0.93200000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.728,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.1439999999999999,"15min":1.1830000000000001},"min":{"1min":0.751,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.2869999999999999},"last":0.877}]},{"osd":2,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1279999999999999,"15min":1.1779999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.1709999999999998},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.367,"5min":1.256,"15min":1.244},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":3.4279999999999999,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":1.3839999999999999}]},{"osd":3,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.1830000000000001,"15min":1.179},"min":{"1min":0.80100000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.655,"5min":1.8009999999999999,"15min":1.986},"last":1.079},{"interface":"front","average":{"1min":1.393,"5min":1.387,"15min":1.3819999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":1.5800000000000001}]},{"osd":4,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.484,"15min":1.4750000000000001},"min":{"1min":1.016,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.395,"5min":2.395,"15min":2.395},"last":1.272},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.411,"15min":1.4330000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.8109999999999999,"5min":2.149,"15min":2.254},"last":1.51}]},{"osd":5,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.48,"15min":1.4870000000000001},"min":{"1min":0.94999999999999996,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.8959999999999999,"5min":2.0819999999999999,"15min":2.1520000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.585,"15min":1.5640000000000001},"min":{"1min":1.1970000000000001,"5min":1.05,"15min":1.05},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.3359999999999999},"last":1.448}]},{"osd":6,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.675,"15min":1.639},"min":{"1min":1.202,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.306},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6259999999999999,"15min":1.611},"min":{"1min":0.97399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.5580000000000001}]},{"osd":7,"last update":"Tue Nov 28 22:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.6990000000000001,"15min":1.6859999999999999},"min":{"1min":1.034,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":1.488},{"interface":"front","average":{"1min":1.671,"5min":1.6060000000000001,"15min":1.625},"min":{"1min":1.177,"5min":0.97899999999999998,"15min":0.97899999999999998},"max":{"1min":1.984,"5min":2.4089999999999998,"15min":2.4180000000000001},"last":1.343}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T22:10:51.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-28T22:10:51.383 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.381+0000 7f669d906640 1 Processor -- start 2023-11-28T22:10:51.384 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.382+0000 7f669d906640 1 -- start start 2023-11-28T22:10:51.385 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.383+0000 7f669d906640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 0x7f66980b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.383+0000 7f669d906640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6698071ac0 0x7f66980b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.383+0000 7f669d906640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66980b27a0 con 0x7f6698072010 2023-11-28T22:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.383+0000 7f669d906640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6698071450 con 0x7f6698071ac0 2023-11-28T22:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.383+0000 7f669d906640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669809ceb0 con 0x7f6698159fc0 2023-11-28T22:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.383+0000 7f6697fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6698071ac0 0x7f66980b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.383+0000 7f6697fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6698071ac0 0x7f66980b2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55988/0 (socket says 172.21.15.45:55988) 2023-11-28T22:10:51.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.383+0000 7f6697fff640 1 -- 172.21.15.45:0/2953259940 learned_addr learned my addr 172.21.15.45:0/2953259940 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:51.387 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.384+0000 7f669c904640 1 --2- 172.21.15.45:0/2953259940 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 0x7f66980b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.384+0000 7f6697fff640 1 -- 172.21.15.45:0/2953259940 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 msgr2=0x7f66980b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.384+0000 7f6697fff640 1 --2- 172.21.15.45:0/2953259940 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 0x7f66980b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.384+0000 7f6697fff640 1 -- 172.21.15.45:0/2953259940 >> v1:172.21.15.190:6789/0 conn(0x7f6698072010 legacy=0x7f669815dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.384+0000 7f6697fff640 1 -- 172.21.15.45:0/2953259940 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66980b2cb0 con 0x7f6698071ac0 2023-11-28T22:10:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.385+0000 7f6697fff640 1 --2- 172.21.15.45:0/2953259940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 0x7f66980b2260 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f668400bb10 tx=0x7f668402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c226c171f6faef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.385+0000 7f66977fe640 1 -- 172.21.15.45:0/2953259940 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668402ede0 con 0x7f6698071ac0 2023-11-28T22:10:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f66977fe640 1 -- 172.21.15.45:0/2953259940 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6684036440 con 0x7f6698071ac0 2023-11-28T22:10:51.389 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f66977fe640 1 -- 172.21.15.45:0/2953259940 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66840367d0 con 0x7f6698071ac0 2023-11-28T22:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f669d906640 1 -- 172.21.15.45:0/2953259940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 msgr2=0x7f66980b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f669d906640 1 --2- 172.21.15.45:0/2953259940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 0x7f66980b2260 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f668400bb10 tx=0x7f668402e030 comp rx=0 tx=0).stop 2023-11-28T22:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f669d906640 1 -- 172.21.15.45:0/2953259940 shutdown_connections 2023-11-28T22:10:51.391 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f669d906640 1 --2- 172.21.15.45:0/2953259940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 0x7f66980b2260 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f669d906640 1 --2- 172.21.15.45:0/2953259940 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 0x7f66980b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f669d906640 1 -- 172.21.15.45:0/2953259940 >> 172.21.15.45:0/2953259940 conn(0x7f66980b1050 msgr2=0x7f66980cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f669d906640 1 -- 172.21.15.45:0/2953259940 shutdown_connections 2023-11-28T22:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.386+0000 7f669d906640 1 -- 172.21.15.45:0/2953259940 wait complete. 2023-11-28T22:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.388+0000 7f669d906640 1 Processor -- start 2023-11-28T22:10:51.396 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.389+0000 7f669d906640 1 -- start start 2023-11-28T22:10:51.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.389+0000 7f669d906640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 0x7f6698099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.389+0000 7f669d906640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6698072010 0x7f669814e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669d906640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 0x7f66980ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.399 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669d906640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669809a290 con 0x7f6698071ac0 2023-11-28T22:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669d906640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669809a570 con 0x7f6698072010 2023-11-28T22:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669d906640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669809a850 con 0x7f6698159fc0 2023-11-28T22:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669c904640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 0x7f6698099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669c904640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 0x7f6698099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56002/0 (socket says 172.21.15.45:56002) 2023-11-28T22:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669c904640 1 -- 172.21.15.45:0/429812934 learned_addr learned my addr 172.21.15.45:0/429812934 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669d105640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 0x7f66980ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669c904640 1 -- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 msgr2=0x7f66980ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.400 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669c904640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 0x7f66980ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669c904640 1 -- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6698072010 msgr2=0x7f669814e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f6697fff640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6698072010 0x7f669814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669c904640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6698072010 0x7f669814e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.390+0000 7f669c904640 1 -- 172.21.15.45:0/429812934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668400b770 con 0x7f6698071ac0 2023-11-28T22:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.391+0000 7f669c904640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 0x7f6698099a40 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f668c00d5f0 tx=0x7f668c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.391+0000 7f66957fa640 1 -- 172.21.15.45:0/429812934 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c0125f0 con 0x7f6698071ac0 2023-11-28T22:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.391+0000 7f66957fa640 1 -- 172.21.15.45:0/429812934 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668c012750 con 0x7f6698071ac0 2023-11-28T22:10:51.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.391+0000 7f66957fa640 1 -- 172.21.15.45:0/429812934 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c012b00 con 0x7f6698071ac0 2023-11-28T22:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.391+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669809ab30 con 0x7f6698071ac0 2023-11-28T22:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.391+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669814ec70 con 0x7f6698071ac0 2023-11-28T22:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.393+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669809b1a0 con 0x7f6698071ac0 2023-11-28T22:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.393+0000 7f66957fa640 1 -- 172.21.15.45:0/429812934 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f668c012c60 con 0x7f6698071ac0 2023-11-28T22:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.394+0000 7f66957fa640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f667c06af60 0x7f667c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.395+0000 7f66957fa640 1 -- 172.21.15.45:0/429812934 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f668c089de0 con 0x7f6698071ac0 2023-11-28T22:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.395+0000 7f6697fff640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f667c06af60 0x7f667c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.396+0000 7f6697fff640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f667c06af60 0x7f667c06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6684038000 tx=0x7f6684008040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.399+0000 7f66957fa640 1 -- 172.21.15.45:0/429812934 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f668c054c00 con 0x7f6698071ac0 2023-11-28T22:10:51.597 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.595+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f669804e920 con 0x7f6698071ac0 2023-11-28T22:10:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.603+0000 7f66957fa640 1 -- 172.21.15.45:0/429812934 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v76)=0 v76) v1 ==== 123+0+0 (secure 0 0 0) 0x7f668c058220 con 0x7f6698071ac0 2023-11-28T22:10:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.605+0000 7f66957fa640 1 -- 172.21.15.45:0/429812934 <== mon.0 v2:172.21.15.45:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f668c08d030 con 0x7f6698071ac0 2023-11-28T22:10:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f667c06af60 msgr2=0x7f667c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f667c06af60 0x7f667c06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6684038000 tx=0x7f6684008040 comp rx=0 tx=0).stop 2023-11-28T22:10:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 msgr2=0x7f6698099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 0x7f6698099a40 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f668c00d5f0 tx=0x7f668c00dab0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 shutdown_connections 2023-11-28T22:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6698159fc0 0x7f66980ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7f667c06af60 0x7f667c06d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6698072010 0x7f669814e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 --2- 172.21.15.45:0/429812934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6698071ac0 0x7f6698099a40 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.607+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 >> 172.21.15.45:0/429812934 conn(0x7f66980b1050 msgr2=0x7f66980acc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.608+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 shutdown_connections 2023-11-28T22:10:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.608+0000 7f669d906640 1 -- 172.21.15.45:0/429812934 wait complete. 2023-11-28T22:10:51.627 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T22:10:51.627 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T22:10:51.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.850+0000 7fbb7b5d0640 1 Processor -- start 2023-11-28T22:10:51.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.850+0000 7fbb7b5d0640 1 -- start start 2023-11-28T22:10:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.852+0000 7fbb7b5d0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740b0c30 0x7fbb740b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.852+0000 7fbb7b5d0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740ce710 con 0x7fbb7415d320 2023-11-28T22:10:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.852+0000 7fbb7b5d0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740ae2c0 con 0x7fbb74071190 2023-11-28T22:10:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.852+0000 7fbb7b5d0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740ae5a0 con 0x7fbb740b0c30 2023-11-28T22:10:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.853+0000 7fbb7936c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbb7415d320 0x7fbb740ad800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58854/0 (socket says 172.21.15.45:58854) 2023-11-28T22:10:51.855 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.853+0000 7fbb7936c640 1 -- 172.21.15.45:0/2196874082 learned_addr learned my addr 172.21.15.45:0/2196874082 (peer_addr_for_me v1:172.21.15.45:0/0) 2023-11-28T22:10:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.853+0000 7fbb78b6b640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740b0c30 0x7fbb740b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.853+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253308106 0 0) 0x7fbb740ce710 con 0x7fbb7415d320 2023-11-28T22:10:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.853+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb60003520 con 0x7fbb7415d320 2023-11-28T22:10:51.857 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2717337133 0 0) 0x7fbb60003520 con 0x7fbb7415d320 2023-11-28T22:10:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740b0c30 msgr2=0x7fbb740b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740b0c30 0x7fbb740b10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 >> v1:172.21.15.190:6789/0 conn(0x7fbb74071190 legacy=0x7fbb74071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb741611f0 con 0x7fbb7415d320 2023-11-28T22:10:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2770878141 0 0) 0x7fbb700019f0 con 0x7fbb7415d320 2023-11-28T22:10:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 >> v1:172.21.15.45:6789/0 conn(0x7fbb7415d320 legacy=0x7fbb740ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.858 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740b0c30 0x7fbb600039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb60003f70 0x7fbb600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb60004d00 0x7fbb60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6000ba90 con 0x7fbb60003f70 2023-11-28T22:10:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6000bc00 con 0x7fbb60004d00 2023-11-28T22:10:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6000bdc0 con 0x7fbb740b0c30 2023-11-28T22:10:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb78b6b640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb60003f70 0x7fbb600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb79b6d640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb60004d00 0x7fbb60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.854+0000 7fbb78b6b640 1 -- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740b0c30 msgr2=0x7fbb600039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.855+0000 7fbb78b6b640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb740b0c30 0x7fbb600039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.855+0000 7fbb78b6b640 1 -- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb60004d00 msgr2=0x7fbb60005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T22:10:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.855+0000 7fbb78b6b640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb60004d00 0x7fbb60005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.855+0000 7fbb78b6b640 1 -- 172.21.15.45:0/2196874082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb60003520 con 0x7fbb60003f70 2023-11-28T22:10:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.856+0000 7fbb78b6b640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb60003f70 0x7fbb600047c0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009ac0 tx=0x7fbb600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971db0d66015480e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:51.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.856+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6402c630 con 0x7fbb60003f70 2023-11-28T22:10:51.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.856+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbb6402c790 con 0x7fbb60003f70 2023-11-28T22:10:51.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.856+0000 7fbb6bfff640 1 -- 172.21.15.45:0/2196874082 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6402cb40 con 0x7fbb60003f70 2023-11-28T22:10:51.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.856+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/2196874082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb60003f70 msgr2=0x7fbb600047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.856+0000 7fbb7b5d0640 1 --2- 172.21.15.45:0/2196874082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb60003f70 0x7fbb600047c0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009ac0 tx=0x7fbb600057f0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.856+0000 7fbb79b6d640 1 -- 172.21.15.45:0/2196874082 reap_dead start 2023-11-28T22:10:51.863 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.857+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/2196874082 shutdown_connections 2023-11-28T22:10:51.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.857+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/2196874082 >> 172.21.15.45:0/2196874082 conn(0x7fbb7406a500 msgr2=0x7fbb7406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:51.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.857+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/2196874082 shutdown_connections 2023-11-28T22:10:51.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.857+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/2196874082 wait complete. 2023-11-28T22:10:51.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.858+0000 7fbb7b5d0640 1 Processor -- start 2023-11-28T22:10:51.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.859+0000 7fbb7b5d0640 1 -- start start 2023-11-28T22:10:51.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb7b5d0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb60003f70 0x7fbb7414ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb7b5d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb7415d700 0x7fbb7415db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb7b5d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415f890 0x7fbb7415fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb7b5d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74160250 con 0x7fbb7415f890 2023-11-28T22:10:51.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb7b5d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7415f310 con 0x7fbb7415d700 2023-11-28T22:10:51.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb7b5d0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7415ee80 con 0x7fbb60003f70 2023-11-28T22:10:51.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb79b6d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415f890 0x7fbb7415fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb79b6d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415f890 0x7fbb7415fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56022/0 (socket says 172.21.15.45:56022) 2023-11-28T22:10:51.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb79b6d640 1 -- 172.21.15.45:0/1264428423 learned_addr learned my addr 172.21.15.45:0/1264428423 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-11-28T22:10:51.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb78b6b640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb7415d700 0x7fbb7415db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb7936c640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb60003f70 0x7fbb7414ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb79b6d640 1 -- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb60003f70 msgr2=0x7fbb7414ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb79b6d640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb60003f70 0x7fbb7414ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb79b6d640 1 -- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb7415d700 msgr2=0x7fbb7415db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:51.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb79b6d640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb7415d700 0x7fbb7415db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:51.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.860+0000 7fbb79b6d640 1 -- 172.21.15.45:0/1264428423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740ae090 con 0x7fbb7415f890 2023-11-28T22:10:51.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.861+0000 7fbb79b6d640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415f890 0x7fbb7415fd10 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fbb740ceda0 tx=0x7fbb6c00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:51.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.861+0000 7fbb69ffb640 1 -- 172.21.15.45:0/1264428423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00b070 con 0x7fbb7415f890 2023-11-28T22:10:51.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.861+0000 7fbb69ffb640 1 -- 172.21.15.45:0/1264428423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbb6c0054e0 con 0x7fbb7415f890 2023-11-28T22:10:51.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.861+0000 7fbb69ffb640 1 -- 172.21.15.45:0/1264428423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c003830 con 0x7fbb7415f890 2023-11-28T22:10:51.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.861+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7415e270 con 0x7fbb7415f890 2023-11-28T22:10:51.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.861+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7415e830 con 0x7fbb7415f890 2023-11-28T22:10:51.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.862+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb740733d0 con 0x7fbb7415f890 2023-11-28T22:10:51.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.863+0000 7fbb69ffb640 1 -- 172.21.15.45:0/1264428423 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 128) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbb6c003990 con 0x7fbb7415f890 2023-11-28T22:10:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.864+0000 7fbb69ffb640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbb5806af60 0x7fbb5806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T22:10:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.864+0000 7fbb69ffb640 1 -- 172.21.15.45:0/1264428423 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fbb6c089280 con 0x7fbb7415f890 2023-11-28T22:10:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.865+0000 7fbb7936c640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbb5806af60 0x7fbb5806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T22:10:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.866+0000 7fbb7936c640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbb5806af60 0x7fbb5806d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbb70001f70 tx=0x7fbb70009040 comp rx=0 tx=0).ready entity=mgr.8884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T22:10:51.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:51.869+0000 7fbb69ffb640 1 -- 172.21.15.45:0/1264428423 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb6c053de0 con 0x7fbb7415f890 2023-11-28T22:10:52.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:10:52.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:10:52.105 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.102+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb740ad250 con 0x7fbb7415f890 2023-11-28T22:10:52.106 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.103+0000 7fbb69ffb640 1 -- 172.21.15.45:0/1264428423 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3708 (secure 0 0 0) 0x7fbb6c018090 con 0x7fbb7415f890 2023-11-28T22:10:52.106 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-11-28T22:10:52.106 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":103,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8692,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/3311227176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":3311227176},{"type":"v1","addr":"172.21.15.190:6835","nonce":3311227176}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8698,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/3492680695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":3492680695},{"type":"v1","addr":"172.21.15.45:6835","nonce":3492680695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":8701,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3937197634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3937197634},{"type":"v1","addr":"172.21.15.190:6837","nonce":3937197634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":11022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/4222434392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":4222434392},{"type":"v1","addr":"172.21.15.45:6833","nonce":4222434392}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[]} 2023-11-28T22:10:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 103 2023-11-28T22:10:52.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.105+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbb5806af60 msgr2=0x7fbb5806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:52.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.105+0000 7fbb7b5d0640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbb5806af60 0x7fbb5806d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbb70001f70 tx=0x7fbb70009040 comp rx=0 tx=0).stop 2023-11-28T22:10:52.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.105+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415f890 msgr2=0x7fbb7415fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T22:10:52.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.105+0000 7fbb7b5d0640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415f890 0x7fbb7415fd10 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fbb740ceda0 tx=0x7fbb6c00d890 comp rx=0 tx=0).stop 2023-11-28T22:10:52.109 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.106+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 shutdown_connections 2023-11-28T22:10:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.106+0000 7fbb7b5d0640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb7415f890 0x7fbb7415fd10 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.106+0000 7fbb7b5d0640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:6832/334700924,v1:172.21.15.190:6833/334700924] conn(0x7fbb5806af60 0x7fbb5806d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.106+0000 7fbb7b5d0640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb7415d700 0x7fbb7415db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.106+0000 7fbb7b5d0640 1 --2- 172.21.15.45:0/1264428423 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb60003f70 0x7fbb7414ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T22:10:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.106+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 >> 172.21.15.45:0/1264428423 conn(0x7fbb7406a500 msgr2=0x7fbb7406c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T22:10:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.106+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 shutdown_connections 2023-11-28T22:10:52.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-11-28T22:10:52.106+0000 7fbb7b5d0640 1 -- 172.21.15.45:0/1264428423 wait complete. 2023-11-28T22:10:52.125 DEBUG:tasks.cephfs.filesystem:already dead... 2023-11-28T22:10:52.125 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-28T22:10:52.125 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T22:10:52.126 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.143 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T22:10:52.143 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T22:10:52.144 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.168 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T22:10:52.168 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T22:10:52.168 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.196 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T22:10:52.196 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T22:10:52.197 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.235 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T22:10:52.235 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-28T22:10:52.236 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-28T22:10:52.236 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.282 INFO:tasks.ceph.osd.0:Stopped 2023-11-28T22:10:52.282 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-28T22:10:52.282 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.373 INFO:tasks.ceph.osd.1:Stopped 2023-11-28T22:10:52.374 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-28T22:10:52.374 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.443 INFO:tasks.ceph.osd.2:Stopped 2023-11-28T22:10:52.443 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-28T22:10:52.443 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.510 INFO:tasks.ceph.osd.3:Stopped 2023-11-28T22:10:52.511 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-28T22:10:52.511 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.598 INFO:tasks.ceph.osd.4:Stopped 2023-11-28T22:10:52.598 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-28T22:10:52.598 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.686 INFO:tasks.ceph.osd.5:Stopped 2023-11-28T22:10:52.686 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-28T22:10:52.686 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:52.774 INFO:tasks.ceph.osd.6:Stopped 2023-11-28T22:10:52.775 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-28T22:10:52.775 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:53.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-11-28T22:10:53.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-11-28T22:10:53.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-11-28T22:10:58.877 INFO:tasks.ceph.osd.7:Stopped 2023-11-28T22:10:58.878 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-28T22:10:58.878 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-28T22:10:58.878 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:10:58.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-11-28T22:10:58.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-11-28T22:10:58.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-11-28T22:10:58.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-11-28T22:11:04.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2023-11-28T22:11:04.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-11-28T22:11:04.980 INFO:tasks.ceph.mgr.y:Stopped 2023-11-28T22:11:04.981 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-28T22:11:04.981 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:11:05.055 INFO:tasks.ceph.mgr.x:Stopped 2023-11-28T22:11:05.056 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-28T22:11:05.056 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-28T22:11:05.056 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:11:05.138 INFO:tasks.ceph.mon.a:Stopped 2023-11-28T22:11:05.138 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-28T22:11:05.139 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:11:05.178 INFO:tasks.ceph.mon.b:Stopped 2023-11-28T22:11:05.179 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-28T22:11:05.179 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T22:11:09.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-11-28T22:11:09.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2023-11-28T22:11:11.280 INFO:tasks.ceph.mon.c:Stopped 2023-11-28T22:11:11.281 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-28T22:11:11.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-28T22:11:11.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T22:11:11.402 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-28T22:11:11.651 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T22:11:11.652 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-28T22:11:11.888 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T22:11:11.889 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-28T22:11:12.032 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T22:11:12.033 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-28T22:11:12.215 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T22:11:12.216 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-28T22:11:12.579 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T22:11:12.580 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-28T22:11:12.752 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T22:11:12.753 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-28T22:11:12.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T22:11:12.926 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-28T22:11:13.099 INFO:tasks.ceph:Archiving mon data... 2023-11-28T22:11:13.101 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/data/mon.a.tgz 2023-11-28T22:11:13.101 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2023-11-28T22:11:13.165 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.YyaCzwvqIN 2023-11-28T22:11:13.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.YyaCzwvqIN 2023-11-28T22:11:13.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.YyaCzwvqIN 2023-11-28T22:11:13.938 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.YyaCzwvqIN is 3MB 2023-11-28T22:11:14.109 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.YyaCzwvqIN 2023-11-28T22:11:14.175 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/data/mon.b.tgz 2023-11-28T22:11:14.175 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2023-11-28T22:11:14.237 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.gZXMgJcVqN 2023-11-28T22:11:14.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.gZXMgJcVqN 2023-11-28T22:11:14.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-11-28T22:11:15.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.gZXMgJcVqN 2023-11-28T22:11:15.239 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.gZXMgJcVqN is 3MB 2023-11-28T22:11:15.401 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.gZXMgJcVqN 2023-11-28T22:11:15.462 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/data/mon.c.tgz 2023-11-28T22:11:15.462 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2023-11-28T22:11:15.514 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.uVg5orwgh6 2023-11-28T22:11:15.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.uVg5orwgh6 2023-11-28T22:11:16.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.uVg5orwgh6 2023-11-28T22:11:16.243 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.uVg5orwgh6 is 3MB 2023-11-28T22:11:16.399 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.uVg5orwgh6 2023-11-28T22:11:16.461 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-28T22:11:16.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T22:11:16.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T22:11:16.572 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-28T22:11:16.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T22:11:16.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T22:11:16.684 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T22:11:16.685 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T22:11:16.698 INFO:tasks.ceph:Archiving crash dumps... 2023-11-28T22:11:16.700 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/remote/smithi045/crash 2023-11-28T22:11:16.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T22:11:16.778 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/remote/smithi190/crash 2023-11-28T22:11:16.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T22:11:16.879 INFO:tasks.ceph:Shutting down logrotate 2023-11-28T22:11:16.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:11:16.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T22:11:17.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T22:11:17.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T22:11:17.233 INFO:tasks.ceph:Compressing logs... 2023-11-28T22:11:17.233 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T22:11:17.244 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T22:11:17.340 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.94619.log 2023-11-28T22:11:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144380.log 2023-11-28T22:11:17.341 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-11-28T22:11:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.144094.log 2023-11-28T22:11:17.343 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.1.94619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.94619.log.gz 2023-11-28T22:11:17.343 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89186.log 2023-11-28T22:11:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.144380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128770.log 2023-11-28T22:11:17.345 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.144380.log.gz 2023-11-28T22:11:17.345 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89146.log 2023-11-28T22:11:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.144094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103461.log 2023-11-28T22:11:17.347 INFO:teuthology.orchestra.run.smithi045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.144094.log.gz 2023-11-28T22:11:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95098.log 2023-11-28T22:11:17.347 INFO:teuthology.orchestra.run.smithi045.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.128770.log.gz 2023-11-28T22:11:17.348 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.89186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.95020.log 2023-11-28T22:11:17.349 INFO:teuthology.orchestra.run.smithi190.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89186.log.gz 2023-11-28T22:11:17.349 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.89146.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-28T22:11:17.349 INFO:teuthology.orchestra.run.smithi190.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.89146.log.gz 2023-11-28T22:11:17.349 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.3.95020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.95020.log.gz 2023-11-28T22:11:17.349 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89066.log 2023-11-28T22:11:17.350 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-11-28T22:11:17.351 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.89066.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.89066.log.gz 2023-11-28T22:11:17.351 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-28T22:11:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103461.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103461.log.gz 2023-11-28T22:11:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136396.log 2023-11-28T22:11:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94708.log 2023-11-28T22:11:17.352 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95098.log.gz 2023-11-28T22:11:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128037.log 2023-11-28T22:11:17.352 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.136396.log.gz 2023-11-28T22:11:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94708.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94708.log.gz 2023-11-28T22:11:17.353 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147470.log 2023-11-28T22:11:17.355 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89026.log 2023-11-28T22:11:17.355 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-11-28T22:11:17.356 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.89026.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-11-28T22:11:17.356 INFO:teuthology.orchestra.run.smithi190.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89026.log.gz 2023-11-28T22:11:17.356 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103999.log 2023-11-28T22:11:17.356 INFO:teuthology.orchestra.run.smithi045.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.128037.log.gz 2023-11-28T22:11:17.356 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123831.log 2023-11-28T22:11:17.356 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.147470.log.gz 2023-11-28T22:11:17.356 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98097.log 2023-11-28T22:11:17.357 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103999.log.gz 2023-11-28T22:11:17.358 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88946.log 2023-11-28T22:11:17.359 INFO:teuthology.orchestra.run.smithi190.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-28T22:11:17.360 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138218.log 2023-11-28T22:11:17.360 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98097.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.123831.log.gz 2023-11-28T22:11:17.360 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98097.log.gz 2023-11-28T22:11:17.360 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118446.log 2023-11-28T22:11:17.360 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138218.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138218.log.gz 2023-11-28T22:11:17.360 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139811.log 2023-11-28T22:11:17.362 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88986.log 2023-11-28T22:11:17.362 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.88946.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-28T22:11:17.362 INFO:teuthology.orchestra.run.smithi190.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88946.log.gz 2023-11-28T22:11:17.362 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.88986.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.88986.log.gz 2023-11-28T22:11:17.363 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.3.95016.log 2023-11-28T22:11:17.363 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92779.log 2023-11-28T22:11:17.363 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118446.log.gz 2023-11-28T22:11:17.364 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145738.log 2023-11-28T22:11:17.364 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.139811.log.gz 2023-11-28T22:11:17.364 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.92779.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92779.log.gz 2023-11-28T22:11:17.364 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93125.log 2023-11-28T22:11:17.365 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.94628.log 2023-11-28T22:11:17.365 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.3.95016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.95016.log.gz 2023-11-28T22:11:17.366 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.145738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145492.log 2023-11-28T22:11:17.366 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.145738.log.gz 2023-11-28T22:11:17.367 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142456.log 2023-11-28T22:11:17.367 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.93125.log.gz 2023-11-28T22:11:17.367 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.145492.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145492.log.gz 2023-11-28T22:11:17.367 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109855.log 2023-11-28T22:11:17.368 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92821.log 2023-11-28T22:11:17.369 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142456.log.gz 2023-11-28T22:11:17.370 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89106.log 2023-11-28T22:11:17.371 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.1.94628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.94628.log.gz 2023-11-28T22:11:17.371 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.89106.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89106.log.gz 2023-11-28T22:11:17.371 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-28T22:11:17.371 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88906.log 2023-11-28T22:11:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102919.log 2023-11-28T22:11:17.374 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109855.log.gz 2023-11-28T22:11:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.92821.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92821.log.gz 2023-11-28T22:11:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-28T22:11:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94498.log 2023-11-28T22:11:17.374 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.102919.log.gz 2023-11-28T22:11:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130014.log 2023-11-28T22:11:17.375 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94498.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.94498.log.gz 2023-11-28T22:11:17.375 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126753.log 2023-11-28T22:11:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110392.log 2023-11-28T22:11:17.376 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.130014.log.gz 2023-11-28T22:11:17.377 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.88906.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88906.log.gz 2023-11-28T22:11:17.377 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-28T22:11:17.377 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-11-28T22:11:17.378 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126753.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.126753.log.gz 2023-11-28T22:11:17.378 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101955.log 2023-11-28T22:11:17.379 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130280.log 2023-11-28T22:11:17.379 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110392.log.gz 2023-11-28T22:11:17.380 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101955.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101955.log.gz 2023-11-28T22:11:17.380 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100458.log 2023-11-28T22:11:17.381 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mds.d.log: 93.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-11-28T22:11:17.382 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144475.log 2023-11-28T22:11:17.382 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130280.log.gz 2023-11-28T22:11:17.382 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100458.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.100458.log.gz 2023-11-28T22:11:17.383 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139745.log 2023-11-28T22:11:17.383 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.144475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138405.log 2023-11-28T22:11:17.383 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.144475.log.gz 2023-11-28T22:11:17.384 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-28T22:11:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139745.log: gzip -5 --verbose -- 91.1% /var/log/ceph/ceph-client.admin.107943.log -- replaced with /var/log/ceph/ceph-client.admin.139745.log.gz 2023-11-28T22:11:17.384 INFO:teuthology.orchestra.run.smithi045.stderr: 2023-11-28T22:11:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138405.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138405.log.gz 2023-11-28T22:11:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102435.log 2023-11-28T22:11:17.387 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.107943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139703.log 2023-11-28T22:11:17.388 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107943.log.gz 2023-11-28T22:11:17.388 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mon.c.log: 94.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-28T22:11:17.389 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130968.log 2023-11-28T22:11:17.389 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102435.log.gz 2023-11-28T22:11:17.391 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100056.log 2023-11-28T22:11:17.391 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139703.log.gz 2023-11-28T22:11:17.391 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130968.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.130968.log.gz 2023-11-28T22:11:17.391 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137924.log 2023-11-28T22:11:17.392 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146913.log 2023-11-28T22:11:17.393 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100056.log.gz 2023-11-28T22:11:17.394 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140869.log 2023-11-28T22:11:17.394 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.137924.log.gz 2023-11-28T22:11:17.395 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108655.log 2023-11-28T22:11:17.395 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.146913.log.gz 2023-11-28T22:11:17.396 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.140869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136216.log 2023-11-28T22:11:17.397 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.140869.log.gz 2023-11-28T22:11:17.397 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117920.log 2023-11-28T22:11:17.398 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108655.log.gz 2023-11-28T22:11:17.399 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96862.log 2023-11-28T22:11:17.399 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.136216.log.gz 2023-11-28T22:11:17.400 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126575.log 2023-11-28T22:11:17.400 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.117920.log.gz 2023-11-28T22:11:17.401 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146053.log 2023-11-28T22:11:17.401 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96862.log.gz 2023-11-28T22:11:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126575.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.126575.log.gz -5 2023-11-28T22:11:17.402 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.125395.log 2023-11-28T22:11:17.404 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139166.log 2023-11-28T22:11:17.404 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.146053.log.gz 2023-11-28T22:11:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135537.log 2023-11-28T22:11:17.405 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125395.log.gz 2023-11-28T22:11:17.406 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147008.log 2023-11-28T22:11:17.406 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.139166.log.gz 2023-11-28T22:11:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135537.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.135537.log.gz 2023-11-28T22:11:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135579.log 2023-11-28T22:11:17.408 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121170.log 2023-11-28T22:11:17.408 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.147008.log.gz 2023-11-28T22:11:17.409 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99714.log 2023-11-28T22:11:17.409 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135579.log.gz 2023-11-28T22:11:17.411 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103284.log 2023-11-28T22:11:17.411 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.121170.log.gz 2023-11-28T22:11:17.412 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99714.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.128541.log 2023-11-28T22:11:17.412 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.99714.log.gz 2023-11-28T22:11:17.413 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97152.log 2023-11-28T22:11:17.413 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.103284.log.gz 2023-11-28T22:11:17.414 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111158.log 2023-11-28T22:11:17.414 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.128541.log.gz 2023-11-28T22:11:17.415 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.97152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.118173.log 2023-11-28T22:11:17.416 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97152.log.gz 2023-11-28T22:11:17.417 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.111158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145654.log 2023-11-28T22:11:17.417 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111158.log.gz 2023-11-28T22:11:17.418 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.118173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.118173.log.gz 2023-11-28T22:11:17.418 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118498.log 2023-11-28T22:11:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.145654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125708.log 2023-11-28T22:11:17.421 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.145654.log.gz 2023-11-28T22:11:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113286.log 2023-11-28T22:11:17.422 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.118498.log.gz 2023-11-28T22:11:17.422 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143777.log 2023-11-28T22:11:17.422 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125708.log.gz 2023-11-28T22:11:17.424 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113286.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113286.log.gz -5 2023-11-28T22:11:17.424 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.113893.log 2023-11-28T22:11:17.424 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143777.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.143777.log.gz 2023-11-28T22:11:17.424 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135164.log 2023-11-28T22:11:17.425 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116749.log 2023-11-28T22:11:17.425 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113893.log.gz 2023-11-28T22:11:17.426 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127423.log 2023-11-28T22:11:17.426 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135164.log.gz 2023-11-28T22:11:17.427 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120500.log 2023-11-28T22:11:17.428 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116749.log.gz 2023-11-28T22:11:17.428 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127423.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.127423.log.gz 2023-11-28T22:11:17.428 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.123271.log 2023-11-28T22:11:17.429 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145874.log 2023-11-28T22:11:17.430 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120500.log.gz 2023-11-28T22:11:17.431 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117092.log 2023-11-28T22:11:17.431 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123271.log.gz 2023-11-28T22:11:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.145874.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145874.log.gz 2023-11-28T22:11:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128499.log 2023-11-28T22:11:17.433 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117092.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117092.log.gz 2023-11-28T22:11:17.433 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108340.log 2023-11-28T22:11:17.435 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95404.log 2023-11-28T22:11:17.435 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128499.log.gz 2023-11-28T22:11:17.436 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129726.log 2023-11-28T22:11:17.436 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108340.log.gz 2023-11-28T22:11:17.437 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95404.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95404.log.gz 2023-11-28T22:11:17.438 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94314.log 2023-11-28T22:11:17.438 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126044.log 2023-11-28T22:11:17.438 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.129726.log.gz 2023-11-28T22:11:17.439 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145780.log 2023-11-28T22:11:17.440 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94314.log.gz 2023-11-28T22:11:17.441 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135122.log 2023-11-28T22:11:17.441 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.126044.log.gz 2023-11-28T22:11:17.442 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.145780.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.145780.log.gz 2023-11-28T22:11:17.442 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125038.log 2023-11-28T22:11:17.446 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115786.log 2023-11-28T22:11:17.446 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135122.log.gz 2023-11-28T22:11:17.446 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141074.log 2023-11-28T22:11:17.446 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125038.log.gz 2023-11-28T22:11:17.447 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115786.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115786.log.gz -5 2023-11-28T22:11:17.447 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.117734.log 2023-11-28T22:11:17.448 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119455.log 2023-11-28T22:11:17.448 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.141074.log.gz 2023-11-28T22:11:17.448 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102123.log 2023-11-28T22:11:17.448 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117734.log.gz 2023-11-28T22:11:17.449 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119455.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119455.log.gz 2023-11-28T22:11:17.449 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123705.log 2023-11-28T22:11:17.450 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142750.log 2023-11-28T22:11:17.450 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102123.log.gz 2023-11-28T22:11:17.451 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116371.log 2023-11-28T22:11:17.452 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.123705.log.gz 2023-11-28T22:11:17.453 INFO:teuthology.orchestra.run.smithi190.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-11-28T22:11:17.454 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100374.log 2023-11-28T22:11:17.454 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142750.log.gz 2023-11-28T22:11:17.454 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116371.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.116371.log.gz 2023-11-28T22:11:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116707.log 2023-11-28T22:11:17.455 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100374.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.133425.log 2023-11-28T22:11:17.455 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.100374.log.gz 2023-11-28T22:11:17.456 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110961.log 2023-11-28T22:11:17.457 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116707.log.gz 2023-11-28T22:11:17.457 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.133425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112615.log 2023-11-28T22:11:17.458 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.133425.log.gz 2023-11-28T22:11:17.458 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110961.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110961.log.gz 2023-11-28T22:11:17.459 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117776.log 2023-11-28T22:11:17.460 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.112615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121493.log 2023-11-28T22:11:17.460 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112615.log.gz 2023-11-28T22:11:17.461 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140171.log 2023-11-28T22:11:17.461 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117776.log.gz 2023-11-28T22:11:17.462 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109770.log 2023-11-28T22:11:17.462 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121493.log.gz 2023-11-28T22:11:17.463 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.140171.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.140171.log.gz 2023-11-28T22:11:17.463 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117012.log 2023-11-28T22:11:17.465 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147596.log 2023-11-28T22:11:17.465 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109770.log.gz 2023-11-28T22:11:17.466 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95014.log 2023-11-28T22:11:17.466 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.117012.log.gz 2023-11-28T22:11:17.467 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134869.log 2023-11-28T22:11:17.467 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147596.log.gz 2023-11-28T22:11:17.468 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95014.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95014.log.gz 2023-11-28T22:11:17.468 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99552.log 2023-11-28T22:11:17.469 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146095.log 2023-11-28T22:11:17.470 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.134869.log.gz 2023-11-28T22:11:17.470 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115828.log 2023-11-28T22:11:17.471 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.99552.log.gz 2023-11-28T22:11:17.471 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146095.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146095.log.gz 2023-11-28T22:11:17.472 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108781.log 2023-11-28T22:11:17.473 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100098.log 2023-11-28T22:11:17.473 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115828.log.gz 2023-11-28T22:11:17.474 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129210.log 2023-11-28T22:11:17.474 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108781.log.gz 2023-11-28T22:11:17.475 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100098.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.100098.log.gz 2023-11-28T22:11:17.475 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119226.log 2023-11-28T22:11:17.476 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115079.log 2023-11-28T22:11:17.477 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.129210.log.gz 2023-11-28T22:11:17.478 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124912.log 2023-11-28T22:11:17.478 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119226.log.gz 2023-11-28T22:11:17.479 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115079.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115079.log.gz 2023-11-28T22:11:17.479 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88919.log 2023-11-28T22:11:17.480 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118701.log 2023-11-28T22:11:17.480 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.124912.log.gz 2023-11-28T22:11:17.481 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.88919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142017.log 2023-11-28T22:11:17.481 INFO:teuthology.orchestra.run.smithi045.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.88919.log.gz 2023-11-28T22:11:17.483 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118701.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118701.log.gz 2023-11-28T22:11:17.483 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123789.log 2023-11-28T22:11:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129066.log 2023-11-28T22:11:17.484 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142017.log.gz 2023-11-28T22:11:17.485 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129631.log 2023-11-28T22:11:17.485 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.123789.log.gz 2023-11-28T22:11:17.486 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129066.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.129066.log.gz 2023-11-28T22:11:17.486 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100222.log 2023-11-28T22:11:17.487 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137389.log 2023-11-28T22:11:17.488 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129631.log.gz 2023-11-28T22:11:17.489 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138447.log 2023-11-28T22:11:17.489 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100222.log.gz 2023-11-28T22:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137389.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.137389.log.gz 2023-11-28T22:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118931.log 2023-11-28T22:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101647.log 2023-11-28T22:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.138447.log.gz 2023-11-28T22:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118931.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118931.log.gz 2023-11-28T22:11:17.493 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139347.log 2023-11-28T22:11:17.494 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95739.log 2023-11-28T22:11:17.495 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101647.log.gz 2023-11-28T22:11:17.496 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124446.log 2023-11-28T22:11:17.496 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.139347.log.gz 2023-11-28T22:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95739.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95739.log.gz 2023-11-28T22:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146618.log 2023-11-28T22:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135038.log 2023-11-28T22:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124446.log.gz 2023-11-28T22:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146618.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.146618.log.gz 2023-11-28T22:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129806.log 2023-11-28T22:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135747.log 2023-11-28T22:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135038.log.gz 2023-11-28T22:11:17.502 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131072.log 2023-11-28T22:11:17.503 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129806.log.gz 2023-11-28T22:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135747.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.135747.log.gz 2023-11-28T22:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111003.log 2023-11-28T22:11:17.505 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.131072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128160.log 2023-11-28T22:11:17.505 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.131072.log.gz 2023-11-28T22:11:17.506 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.111003.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.111003.log.gz 2023-11-28T22:11:17.506 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96946.log 2023-11-28T22:11:17.508 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125876.log 2023-11-28T22:11:17.508 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.128160.log.gz 2023-11-28T22:11:17.509 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123621.log 2023-11-28T22:11:17.510 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96946.log.gz 2023-11-28T22:11:17.510 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125876.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.125876.log.gz 2023-11-28T22:11:17.511 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129972.log 2023-11-28T22:11:17.512 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138573.log 2023-11-28T22:11:17.512 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.123621.log.gz 2023-11-28T22:11:17.513 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129972.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.129972.log.gz 2023-11-28T22:11:17.513 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129505.log 2023-11-28T22:11:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99930.log 2023-11-28T22:11:17.515 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.138573.log.gz 2023-11-28T22:11:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129505.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.129505.log.gz 2023-11-28T22:11:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92557.log 2023-11-28T22:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99930.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99930.log.gz 2023-11-28T22:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143525.log 2023-11-28T22:11:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.92557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129421.log 2023-11-28T22:11:17.519 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92557.log.gz 2023-11-28T22:11:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143525.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143525.log.gz 2023-11-28T22:11:17.520 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126921.log 2023-11-28T22:11:17.522 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147302.log 2023-11-28T22:11:17.522 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.129421.log.gz 2023-11-28T22:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126921.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.126921.log.gz 2023-11-28T22:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146408.log 2023-11-28T22:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147302.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147302.log.gz 2023-11-28T22:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110290.log 2023-11-28T22:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.121051.log 2023-11-28T22:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146408.log.gz 2023-11-28T22:11:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110290.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110290.log.gz 2023-11-28T22:11:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121535.log 2023-11-28T22:11:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.121051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.121051.log.gz 2023-11-28T22:11:17.528 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95140.log 2023-11-28T22:11:17.530 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109304.log 2023-11-28T22:11:17.530 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.121535.log.gz 2023-11-28T22:11:17.530 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95140.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95140.log.gz 2023-11-28T22:11:17.531 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108823.log 2023-11-28T22:11:17.532 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101065.log 2023-11-28T22:11:17.532 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109304.log.gz 2023-11-28T22:11:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108823.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.108823.log.gz 2023-11-28T22:11:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138489.log 2023-11-28T22:11:17.535 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117860.log 2023-11-28T22:11:17.535 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.101065.log.gz 2023-11-28T22:11:17.536 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134954.log 2023-11-28T22:11:17.537 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.138489.log.gz 2023-11-28T22:11:17.537 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117860.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117860.log.gz 2023-11-28T22:11:17.538 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125311.log 2023-11-28T22:11:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130322.log 2023-11-28T22:11:17.539 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.134954.log.gz 2023-11-28T22:11:17.540 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125311.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125311.log.gz 2023-11-28T22:11:17.540 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143089.log 2023-11-28T22:11:17.541 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108221.log 2023-11-28T22:11:17.541 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.130322.log.gz 2023-11-28T22:11:17.542 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99510.log 2023-11-28T22:11:17.543 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.143089.log.gz 2023-11-28T22:11:17.543 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122552.log 2023-11-28T22:11:17.544 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108221.log.gz 2023-11-28T22:11:17.545 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99510.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99510.log.gz 2023-11-28T22:11:17.545 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121696.log 2023-11-28T22:11:17.546 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96568.log 2023-11-28T22:11:17.546 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.122552.log.gz 2023-11-28T22:11:17.547 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138363.log 2023-11-28T22:11:17.547 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121696.log.gz 2023-11-28T22:11:17.548 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96568.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96568.log.gz 2023-11-28T22:11:17.549 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139040.log 2023-11-28T22:11:17.550 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124274.log 2023-11-28T22:11:17.550 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138363.log.gz 2023-11-28T22:11:17.551 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96052.log 2023-11-28T22:11:17.551 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.139040.log.gz 2023-11-28T22:11:17.552 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124274.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124274.log.gz 2023-11-28T22:11:17.552 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119370.log 2023-11-28T22:11:17.553 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120234.log 2023-11-28T22:11:17.554 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96052.log.gz 2023-11-28T22:11:17.555 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116665.log 2023-11-28T22:11:17.555 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.119370.log.gz 2023-11-28T22:11:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120234.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.120234.log.gz 2023-11-28T22:11:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119184.log 2023-11-28T22:11:17.557 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125479.log 2023-11-28T22:11:17.557 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116665.log.gz 2023-11-28T22:11:17.558 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126837.log 2023-11-28T22:11:17.558 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119184.log.gz 2023-11-28T22:11:17.559 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125479.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.125479.log.gz 2023-11-28T22:11:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99972.log 2023-11-28T22:11:17.561 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119497.log 2023-11-28T22:11:17.561 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126837.log.gz 2023-11-28T22:11:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117986.log 2023-11-28T22:11:17.562 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.99972.log.gz 2023-11-28T22:11:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119497.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119497.log.gz 2023-11-28T22:11:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103957.log 2023-11-28T22:11:17.564 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123579.log 2023-11-28T22:11:17.565 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.117986.log.gz 2023-11-28T22:11:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93213.log 2023-11-28T22:11:17.566 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.103957.log.gz 2023-11-28T22:11:17.567 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123579.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.123579.log.gz 2023-11-28T22:11:17.567 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119268.log 2023-11-28T22:11:17.568 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125666.log 2023-11-28T22:11:17.568 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93213.log.gz 2023-11-28T22:11:17.569 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95224.log 2023-11-28T22:11:17.569 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119268.log.gz 2023-11-28T22:11:17.570 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125666.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.125666.log.gz 2023-11-28T22:11:17.571 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127837.log 2023-11-28T22:11:17.571 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100014.log 2023-11-28T22:11:17.572 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95224.log.gz 2023-11-28T22:11:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143735.log 2023-11-28T22:11:17.573 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127837.log.gz 2023-11-28T22:11:17.574 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100014.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.100014.log.gz 2023-11-28T22:11:17.574 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113159.log 2023-11-28T22:11:17.575 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123409.log 2023-11-28T22:11:17.576 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.143735.log.gz 2023-11-28T22:11:17.576 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135621.log 2023-11-28T22:11:17.577 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113159.log.gz 2023-11-28T22:11:17.578 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123409.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.123409.log.gz 2023-11-28T22:11:17.578 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143439.log 2023-11-28T22:11:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99630.log 2023-11-28T22:11:17.579 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135621.log.gz 2023-11-28T22:11:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147386.log 2023-11-28T22:11:17.580 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.143439.log.gz 2023-11-28T22:11:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99630.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.99630.log.gz 2023-11-28T22:11:17.582 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137608.log 2023-11-28T22:11:17.583 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121960.log 2023-11-28T22:11:17.583 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.147386.log.gz 2023-11-28T22:11:17.584 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94230.log 2023-11-28T22:11:17.584 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.137608.log.gz 2023-11-28T22:11:17.585 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121960.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.121960.log.gz 2023-11-28T22:11:17.585 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139124.log 2023-11-28T22:11:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142834.log 2023-11-28T22:11:17.587 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.94230.log.gz 2023-11-28T22:11:17.588 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139124.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.139124.log.gz 2023-11-28T22:11:17.588 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147554.log 2023-11-28T22:11:17.589 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116539.log 2023-11-28T22:11:17.589 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.142834.log.gz 2023-11-28T22:11:17.590 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109897.log 2023-11-28T22:11:17.590 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.147554.log.gz 2023-11-28T22:11:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130194.log 2023-11-28T22:11:17.591 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116539.log.gz 2023-11-28T22:11:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109897.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109897.log.gz 2023-11-28T22:11:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142101.log 2023-11-28T22:11:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96274.log 2023-11-28T22:11:17.594 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130194.log.gz 2023-11-28T22:11:17.595 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102477.log 2023-11-28T22:11:17.595 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142101.log.gz 2023-11-28T22:11:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96274.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.96274.log.gz 2023-11-28T22:11:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125269.log 2023-11-28T22:11:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146357.log 2023-11-28T22:11:17.598 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125269.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.102477.log.gz 2023-11-28T22:11:17.598 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140229.log 2023-11-28T22:11:17.599 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.125269.log.gz 2023-11-28T22:11:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146357.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146357.log.gz 2023-11-28T22:11:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113809.log 2023-11-28T22:11:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.140229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114462.log 2023-11-28T22:11:17.601 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.140229.log.gz 2023-11-28T22:11:17.602 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109220.log 2023-11-28T22:11:17.602 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113809.log.gz 2023-11-28T22:11:17.603 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114462.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.114462.log.gz 2023-11-28T22:11:17.603 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122678.log 2023-11-28T22:11:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143131.log 2023-11-28T22:11:17.605 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109220.log.gz 2023-11-28T22:11:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136172.log 2023-11-28T22:11:17.606 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122678.log.gz 2023-11-28T22:11:17.607 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143131.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.143131.log.gz 2023-11-28T22:11:17.607 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134996.log 2023-11-28T22:11:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121654.log 2023-11-28T22:11:17.608 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.136172.log.gz 2023-11-28T22:11:17.609 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109812.log 2023-11-28T22:11:17.609 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134996.log.gz 2023-11-28T22:11:17.610 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121654.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.121654.log.gz 2023-11-28T22:11:17.611 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100500.log 2023-11-28T22:11:17.612 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121738.log 2023-11-28T22:11:17.612 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109812.log.gz 2023-11-28T22:11:17.613 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118785.log 2023-11-28T22:11:17.613 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100500.log.gz 2023-11-28T22:11:17.614 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121738.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121738.log.gz 2023-11-28T22:11:17.614 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113598.log 2023-11-28T22:11:17.615 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125918.log 2023-11-28T22:11:17.615 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118785.log.gz 2023-11-28T22:11:17.616 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.102730.log 2023-11-28T22:11:17.617 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.113598.log.gz 2023-11-28T22:11:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125918.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125918.log.gz 2023-11-28T22:11:17.618 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120584.log 2023-11-28T22:11:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.102730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.102730.log.gz 2023-11-28T22:11:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120132.log 2023-11-28T22:11:17.620 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109983.log 2023-11-28T22:11:17.620 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.120584.log.gz 2023-11-28T22:11:17.621 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122804.log 2023-11-28T22:11:17.621 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120132.log.gz 2023-11-28T22:11:17.623 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109983.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.109983.log.gz 2023-11-28T22:11:17.623 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120090.log 2023-11-28T22:11:17.624 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114506.log 2023-11-28T22:11:17.624 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122804.log.gz 2023-11-28T22:11:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98391.log 2023-11-28T22:11:17.625 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120090.log.gz 2023-11-28T22:11:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114506.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114506.log.gz 2023-11-28T22:11:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109514.log 2023-11-28T22:11:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95727.log 2023-11-28T22:11:17.628 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98391.log.gz 2023-11-28T22:11:17.628 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139559.log 2023-11-28T22:11:17.629 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109514.log.gz 2023-11-28T22:11:17.630 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95727.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95727.log.gz 2023-11-28T22:11:17.630 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141744.log 2023-11-28T22:11:17.631 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119310.log 2023-11-28T22:11:17.631 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.139559.log.gz 2023-11-28T22:11:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134597.log 2023-11-28T22:11:17.632 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.141744.log.gz 2023-11-28T22:11:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119310.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119310.log.gz 2023-11-28T22:11:17.633 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104503.log 2023-11-28T22:11:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98307.log 2023-11-28T22:11:17.635 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.134597.log.gz 2023-11-28T22:11:17.636 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118659.log 2023-11-28T22:11:17.636 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104503.log.gz 2023-11-28T22:11:17.637 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98307.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98307.log.gz 2023-11-28T22:11:17.637 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94440.log 2023-11-28T22:11:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118659.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118659.log.gz 2023-11-28T22:11:17.638 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115744.log 2023-11-28T22:11:17.639 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99189.log 2023-11-28T22:11:17.640 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.94440.log.gz 2023-11-28T22:11:17.641 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146534.log 2023-11-28T22:11:17.641 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115744.log.gz 2023-11-28T22:11:17.642 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99189.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99189.log.gz 2023-11-28T22:11:17.642 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115455.log 2023-11-28T22:11:17.643 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100416.log 2023-11-28T22:11:17.643 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146534.log.gz 2023-11-28T22:11:17.644 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126403.log 2023-11-28T22:11:17.644 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115455.log.gz 2023-11-28T22:11:17.645 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100416.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100416.log.gz 2023-11-28T22:11:17.646 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98811.log 2023-11-28T22:11:17.647 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125353.log 2023-11-28T22:11:17.647 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126403.log.gz 2023-11-28T22:11:17.648 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138998.log 2023-11-28T22:11:17.648 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98811.log.gz 2023-11-28T22:11:17.649 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125353.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.125353.log.gz 2023-11-28T22:11:17.649 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124954.log 2023-11-28T22:11:17.650 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134639.log 2023-11-28T22:11:17.651 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.138998.log.gz 2023-11-28T22:11:17.654 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100331.log 2023-11-28T22:11:17.654 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124954.log.gz 2023-11-28T22:11:17.654 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134639.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.134639.log.gz -5 2023-11-28T22:11:17.654 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.122305.log 2023-11-28T22:11:17.655 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109939.log 2023-11-28T22:11:17.655 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100331.log.gz 2023-11-28T22:11:17.656 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.115270.log 2023-11-28T22:11:17.656 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.122305.log.gz 2023-11-28T22:11:17.656 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109939.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109939.log.gz 2023-11-28T22:11:17.656 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96526.log 2023-11-28T22:11:17.658 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.115270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123315.log 2023-11-28T22:11:17.658 INFO:teuthology.orchestra.run.smithi045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.115270.log.gz 2023-11-28T22:11:17.659 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96400.log 2023-11-28T22:11:17.659 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96526.log.gz 2023-11-28T22:11:17.660 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94666.log 2023-11-28T22:11:17.660 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.123315.log.gz 2023-11-28T22:11:17.661 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96400.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96400.log.gz 2023-11-28T22:11:17.661 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102811.log 2023-11-28T22:11:17.662 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95741.log 2023-11-28T22:11:17.663 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.94666.log.gz 2023-11-28T22:11:17.664 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117174.log 2023-11-28T22:11:17.664 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102811.log.gz 2023-11-28T22:11:17.665 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96736.log 2023-11-28T22:11:17.665 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95741.log.gz 2023-11-28T22:11:17.666 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117174.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.117174.log.gz 2023-11-28T22:11:17.666 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147680.log 2023-11-28T22:11:17.667 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128940.log 2023-11-28T22:11:17.667 INFO:teuthology.orchestra.run.smithi045.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.96736.log.gz 2023-11-28T22:11:17.668 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137692.log 2023-11-28T22:11:17.668 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147680.log.gz 2023-11-28T22:11:17.669 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128940.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.128940.log.gz 2023-11-28T22:11:17.670 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141974.log 2023-11-28T22:11:17.671 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113683.log 2023-11-28T22:11:17.671 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.137692.log.gz 2023-11-28T22:11:17.672 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129379.log 2023-11-28T22:11:17.672 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141974.log.gz 2023-11-28T22:11:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113683.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113683.log.gz 2023-11-28T22:11:17.673 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103663.log 2023-11-28T22:11:17.674 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122425.log 2023-11-28T22:11:17.675 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.129379.log.gz 2023-11-28T22:11:17.675 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103663.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103663.log.gz 2023-11-28T22:11:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120322.log 2023-11-28T22:11:17.677 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127047.log 2023-11-28T22:11:17.677 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.122425.log.gz 2023-11-28T22:11:17.678 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125132.log 2023-11-28T22:11:17.679 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120322.log.gz 2023-11-28T22:11:17.680 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128625.log 2023-11-28T22:11:17.680 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.127047.log.gz 2023-11-28T22:11:17.681 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134912.log 2023-11-28T22:11:17.681 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.125132.log.gz 2023-11-28T22:11:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128625.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.128625.log.gz 2023-11-28T22:11:17.682 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113538.log 2023-11-28T22:11:17.684 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119581.log 2023-11-28T22:11:17.684 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.134912.log.gz 2023-11-28T22:11:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96075.log 2023-11-28T22:11:17.685 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113538.log.gz 2023-11-28T22:11:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119581.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119581.log.gz 2023-11-28T22:11:17.686 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137276.log 2023-11-28T22:11:17.687 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122846.log 2023-11-28T22:11:17.688 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.96075.log.gz 2023-11-28T22:11:17.688 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137276.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.137276.log.gz 2023-11-28T22:11:17.689 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99756.log 2023-11-28T22:11:17.690 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117300.log 2023-11-28T22:11:17.691 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122846.log.gz 2023-11-28T22:11:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134436.log 2023-11-28T22:11:17.692 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.99756.log.gz 2023-11-28T22:11:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117300.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117300.log.gz 2023-11-28T22:11:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124208.log 2023-11-28T22:11:17.694 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96078.log 2023-11-28T22:11:17.694 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134436.log.gz 2023-11-28T22:11:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124208.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124208.log.gz 2023-11-28T22:11:17.695 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98685.log 2023-11-28T22:11:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137734.log 2023-11-28T22:11:17.697 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98685.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96078.log.gz 2023-11-28T22:11:17.698 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98685.log.gz 2023-11-28T22:11:17.698 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99342.log 2023-11-28T22:11:17.699 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106031.log 2023-11-28T22:11:17.700 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.137734.log.gz 2023-11-28T22:11:17.700 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99342.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99342.log.gz 2023-11-28T22:11:17.701 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131177.log 2023-11-28T22:11:17.702 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.106031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122179.log 2023-11-28T22:11:17.703 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106031.log.gz 2023-11-28T22:11:17.704 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.131177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142059.log 2023-11-28T22:11:17.704 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.131177.log.gz 2023-11-28T22:11:17.705 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122179.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122179.log.gz 2023-11-28T22:11:17.705 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125792.log 2023-11-28T22:11:17.706 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105134.log 2023-11-28T22:11:17.707 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142059.log.gz 2023-11-28T22:11:17.707 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125792.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125792.log.gz 2023-11-28T22:11:17.708 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116006.log 2023-11-28T22:11:17.709 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.105134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123187.log 2023-11-28T22:11:17.709 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105134.log.gz 2023-11-28T22:11:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116006.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.116006.log.gz 2023-11-28T22:11:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104839.log 2023-11-28T22:11:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116227.log 2023-11-28T22:11:17.712 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.123187.log.gz 2023-11-28T22:11:17.713 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104839.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104839.log.gz 2023-11-28T22:11:17.713 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144321.log 2023-11-28T22:11:17.714 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110584.log 2023-11-28T22:11:17.715 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.144321.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.116227.log.gz 2023-11-28T22:11:17.715 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.144321.log.gz 2023-11-28T22:11:17.716 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142143.log 2023-11-28T22:11:17.717 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94972.log 2023-11-28T22:11:17.717 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110584.log.gz 2023-11-28T22:11:17.718 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142143.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142143.log.gz 2023-11-28T22:11:17.718 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93423.log 2023-11-28T22:11:17.720 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126795.log 2023-11-28T22:11:17.720 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94972.log.gz 2023-11-28T22:11:17.721 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93423.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93423.log.gz 2023-11-28T22:11:17.722 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126963.log 2023-11-28T22:11:17.722 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137966.log 2023-11-28T22:11:17.724 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.126795.log.gz 2023-11-28T22:11:17.724 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126963.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126963.log.gz 2023-11-28T22:11:17.724 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101605.log 2023-11-28T22:11:17.725 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137966.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-28T22:11:17.726 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.137966.log.gz 2023-11-28T22:11:17.727 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101605.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.101605.log.gz 2023-11-28T22:11:17.727 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.116185.log 2023-11-28T22:11:17.728 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146179.log 2023-11-28T22:11:17.729 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101310.log 2023-11-28T22:11:17.729 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116185.log.gz 2023-11-28T22:11:17.732 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101107.log 2023-11-28T22:11:17.732 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146179.log.gz 2023-11-28T22:11:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101310.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.101310.log.gz 2023-11-28T22:11:17.734 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134681.log 2023-11-28T22:11:17.734 INFO:teuthology.orchestra.run.smithi045.stderr: 89.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-28T22:11:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114420.log 2023-11-28T22:11:17.735 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101107.log.gz 2023-11-28T22:11:17.735 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108949.log 2023-11-28T22:11:17.735 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.134681.log.gz 2023-11-28T22:11:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127531.log 2023-11-28T22:11:17.737 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114420.log.gz 2023-11-28T22:11:17.738 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108949.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108949.log.gz 2023-11-28T22:11:17.738 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139659.log 2023-11-28T22:11:17.739 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126319.log 2023-11-28T22:11:17.739 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127531.log.gz 2023-11-28T22:11:17.740 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126667.log 2023-11-28T22:11:17.741 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.139659.log.gz 2023-11-28T22:11:17.742 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93573.log 2023-11-28T22:11:17.742 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.126319.log.gz 2023-11-28T22:11:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126667.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.126667.log.gz 2023-11-28T22:11:17.743 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100542.log 2023-11-28T22:11:17.745 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143481.log 2023-11-28T22:11:17.745 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93573.log.gz 2023-11-28T22:11:17.746 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105311.log 2023-11-28T22:11:17.747 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100542.log.gz 2023-11-28T22:11:17.747 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143481.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.143481.log.gz -5 2023-11-28T22:11:17.747 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.146263.log 2023-11-28T22:11:17.748 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.105311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114592.log 2023-11-28T22:11:17.748 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105311.log.gz 2023-11-28T22:11:17.749 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110853.log 2023-11-28T22:11:17.749 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.146263.log.gz 2023-11-28T22:11:17.750 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120542.log 2023-11-28T22:11:17.751 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.114592.log.gz 2023-11-28T22:11:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110853.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110853.log.gz 2023-11-28T22:11:17.752 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93075.log 2023-11-28T22:11:17.753 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95731.log 2023-11-28T22:11:17.753 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120542.log.gz 2023-11-28T22:11:17.754 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93075.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.93075.log.gz 2023-11-28T22:11:17.754 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139433.log 2023-11-28T22:11:17.756 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95726.log 2023-11-28T22:11:17.756 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95731.log.gz 2023-11-28T22:11:17.757 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110518.log 2023-11-28T22:11:17.757 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.139433.log.gz 2023-11-28T22:11:17.758 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104898.log 2023-11-28T22:11:17.758 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95726.log.gz 2023-11-28T22:11:17.759 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146810.log 2023-11-28T22:11:17.761 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104898.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.110518.log.gz 2023-11-28T22:11:17.761 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104898.log.gz 2023-11-28T22:11:17.761 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130682.log 2023-11-28T22:11:17.762 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115389.log 2023-11-28T22:11:17.763 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.146810.log.gz 2023-11-28T22:11:17.763 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135495.log 2023-11-28T22:11:17.763 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.130682.log.gz 2023-11-28T22:11:17.765 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142624.log 2023-11-28T22:11:17.766 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.115389.log.gz 2023-11-28T22:11:17.766 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138092.log 2023-11-28T22:11:17.767 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135495.log.gz 2023-11-28T22:11:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142624.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142624.log.gz 2023-11-28T22:11:17.767 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122941.log 2023-11-28T22:11:17.768 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136130.log 2023-11-28T22:11:17.768 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138092.log.gz 2023-11-28T22:11:17.769 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140121.log 88.3% 2023-11-28T22:11:17.769 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.122941.log.gz 2023-11-28T22:11:17.771 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108991.log 2023-11-28T22:11:17.771 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.136130.log.gz 2023-11-28T22:11:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.140121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103166.log 2023-11-28T22:11:17.772 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.140121.log.gz 2023-11-28T22:11:17.773 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108991.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108991.log.gz 2023-11-28T22:11:17.774 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104755.log 2023-11-28T22:11:17.774 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137447.log 2023-11-28T22:11:17.775 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103166.log.gz 2023-11-28T22:11:17.776 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104755.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104755.log.gz 2023-11-28T22:11:17.776 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96988.log 2023-11-28T22:11:17.777 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96610.log 2023-11-28T22:11:17.777 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.137447.log.gz 2023-11-28T22:11:17.778 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118974.log 2023-11-28T22:11:17.780 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96610.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.96988.log.gz 2023-11-28T22:11:17.780 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96610.log.gz 2023-11-28T22:11:17.780 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116455.log 2023-11-28T22:11:17.781 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118974.log: gzip -5 --verbose -- 87.0% /var/log/ceph/ceph-client.admin.136606.log -- replaced with /var/log/ceph/ceph-client.admin.118974.log.gz 2023-11-28T22:11:17.781 INFO:teuthology.orchestra.run.smithi045.stderr: 2023-11-28T22:11:17.782 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138615.log 2023-11-28T22:11:17.783 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.116455.log.gz 2023-11-28T22:11:17.784 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136606.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136606.log.gz 2023-11-28T22:11:17.784 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122221.log 2023-11-28T22:11:17.785 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123663.log 2023-11-28T22:11:17.785 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.138615.log.gz 2023-11-28T22:11:17.786 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122221.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122221.log.gz 2023-11-28T22:11:17.786 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94272.log 2023-11-28T22:11:17.787 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105252.log 2023-11-28T22:11:17.788 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.123663.log.gz 2023-11-28T22:11:17.789 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94272.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.94272.log.gz 2023-11-28T22:11:17.789 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121236.log 2023-11-28T22:11:17.790 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.105252.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105252.log.gz 2023-11-28T22:11:17.790 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128078.log 2023-11-28T22:11:17.791 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99147.log 2023-11-28T22:11:17.792 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.121236.log.gz 2023-11-28T22:11:17.793 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143009.log 2023-11-28T22:11:17.793 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.128078.log.gz 2023-11-28T22:11:17.794 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99147.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99147.log.gz 2023-11-28T22:11:17.794 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136732.log 2023-11-28T22:11:17.795 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143009.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.143009.log.gz 2023-11-28T22:11:17.795 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102165.log 2023-11-28T22:11:17.796 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107151.log 2023-11-28T22:11:17.797 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.136732.log.gz 2023-11-28T22:11:17.798 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102165.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102165.log.gz 2023-11-28T22:11:17.798 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147218.log 2023-11-28T22:11:17.799 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.107151.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.107151.log.gz -5 2023-11-28T22:11:17.799 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.0.124088.log 2023-11-28T22:11:17.800 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147218.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.147218.log.gz 2023-11-28T22:11:17.800 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125834.log 2023-11-28T22:11:17.801 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.124088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.124088.log.gz 2023-11-28T22:11:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93297.log 2023-11-28T22:11:17.803 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125834.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125834.log.gz 2023-11-28T22:11:17.803 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.115266.log 2023-11-28T22:11:17.804 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93297.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93297.log.gz 2023-11-28T22:11:17.804 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144171.log 2023-11-28T22:11:17.805 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.115266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.115266.log.gz 2023-11-28T22:11:17.805 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138699.log 2023-11-28T22:11:17.807 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.144171.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.144171.log.gz 2023-11-28T22:11:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118617.log 2023-11-28T22:11:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138699.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.138699.log.gz 2023-11-28T22:11:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94582.log 2023-11-28T22:11:17.809 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99426.log 2023-11-28T22:11:17.809 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.118617.log.gz 2023-11-28T22:11:17.810 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123495.log 2023-11-28T22:11:17.810 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.94582.log.gz 2023-11-28T22:11:17.811 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123103.log 2023-11-28T22:11:17.812 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99426.log.gz 2023-11-28T22:11:17.813 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140305.log 2023-11-28T22:11:17.813 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.123495.log.gz 2023-11-28T22:11:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123103.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.123103.log.gz 2023-11-28T22:11:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122720.log 2023-11-28T22:11:17.815 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.140305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135351.log 2023-11-28T22:11:17.815 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.140305.log.gz 2023-11-28T22:11:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.118177.log 2023-11-28T22:11:17.817 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.122720.log.gz 2023-11-28T22:11:17.817 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104797.log 2023-11-28T22:11:17.818 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135351.log.gz 2023-11-28T22:11:17.818 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.118177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.118177.log.gz 2023-11-28T22:11:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120860.log 2023-11-28T22:11:17.820 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104797.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104797.log.gz 2023-11-28T22:11:17.820 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147260.log 2023-11-28T22:11:17.821 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139305.log 2023-11-28T22:11:17.821 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.120860.log.gz 2023-11-28T22:11:17.822 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92976.log 2023-11-28T22:11:17.822 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.147260.log.gz 2023-11-28T22:11:17.823 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117566.log 2023-11-28T22:11:17.823 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139305.log.gz 2023-11-28T22:11:17.824 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.92976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98727.log 2023-11-28T22:11:17.824 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92976.log.gz 2023-11-28T22:11:17.825 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117566.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.117566.log.gz -5 2023-11-28T22:11:17.826 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.123021.log 2023-11-28T22:11:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.110771.log 2023-11-28T22:11:17.827 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98727.log.gz 2023-11-28T22:11:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123021.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.123021.log.gz 2023-11-28T22:11:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115497.log 2023-11-28T22:11:17.829 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.110771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.110771.log.gz 2023-11-28T22:11:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143651.log 2023-11-28T22:11:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115497.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.115497.log.gz 2023-11-28T22:11:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96484.log 2023-11-28T22:11:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108508.log 2023-11-28T22:11:17.832 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.143651.log.gz 2023-11-28T22:11:17.833 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96484.log: gzip 89.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.96484.log.gz 2023-11-28T22:11:17.833 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.114334.log 2023-11-28T22:11:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100836.log 2023-11-28T22:11:17.835 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108508.log.gz 2023-11-28T22:11:17.835 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114334.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114334.log.gz 2023-11-28T22:11:17.836 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109093.log 2023-11-28T22:11:17.837 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126625.log 2023-11-28T22:11:17.837 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100836.log.gz 2023-11-28T22:11:17.838 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.124084.log 2023-11-28T22:11:17.838 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109093.log.gz 2023-11-28T22:11:17.839 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136046.log 2023-11-28T22:11:17.839 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.126625.log.gz 2023-11-28T22:11:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.124084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.124084.log.gz 2023-11-28T22:11:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142540.log 2023-11-28T22:11:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142227.log 2023-11-28T22:11:17.842 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.136046.log.gz 2023-11-28T22:11:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142540.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.142540.log.gz 2023-11-28T22:11:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119791.log 2023-11-28T22:11:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138916.log 2023-11-28T22:11:17.844 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142227.log.gz 2023-11-28T22:11:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137109.log 2023-11-28T22:11:17.845 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119791.log.gz 2023-11-28T22:11:17.846 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118254.log 2023-11-28T22:11:17.847 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.138916.log.gz 2023-11-28T22:11:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98265.log 2023-11-28T22:11:17.848 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137109.log.gz 2023-11-28T22:11:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118254.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118254.log.gz 2023-11-28T22:11:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95401.log 2023-11-28T22:11:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124658.log 2023-11-28T22:11:17.850 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98265.log.gz 2023-11-28T22:11:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144433.log 2023-11-28T22:11:17.851 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95401.log.gz 2023-11-28T22:11:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105016.log 2023-11-28T22:11:17.852 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124658.log.gz 2023-11-28T22:11:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.144433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130056.log 2023-11-28T22:11:17.854 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.144433.log.gz 2023-11-28T22:11:17.858 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.105016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114761.log 2023-11-28T22:11:17.858 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105016.log.gz 2023-11-28T22:11:17.858 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119886.log 2023-11-28T22:11:17.858 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.130056.log.gz 2023-11-28T22:11:17.859 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114761.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.114761.log.gz 2023-11-28T22:11:17.859 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119966.log 2023-11-28T22:11:17.860 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135308.log 2023-11-28T22:11:17.860 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.119886.log.gz 2023-11-28T22:11:17.860 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119966.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119966.log.gz 2023-11-28T22:11:17.861 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98349.log 2023-11-28T22:11:17.861 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135308.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.135308.log.gz 2023-11-28T22:11:17.861 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105193.log 2023-11-28T22:11:17.862 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114719.log 2023-11-28T22:11:17.862 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98349.log.gz 2023-11-28T22:11:17.863 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.105193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127719.log 2023-11-28T22:11:17.863 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105193.log.gz 2023-11-28T22:11:17.864 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137217.log 2023-11-28T22:11:17.864 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114719.log.gz 2023-11-28T22:11:17.865 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95182.log 2023-11-28T22:11:17.865 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127719.log.gz 2023-11-28T22:11:17.866 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137217.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-28T22:11:17.866 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.137217.log.gz 2023-11-28T22:11:17.867 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134723.log 2023-11-28T22:11:17.867 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95182.log.gz 2023-11-28T22:11:17.868 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147050.log 2023-11-28T22:11:17.869 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134723.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.134723.log.gz 2023-11-28T22:11:17.870 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129150.log 2023-11-28T22:11:17.871 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103402.log 2023-11-28T22:11:17.871 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147050.log.gz 2023-11-28T22:11:17.872 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129150.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129150.log.gz 2023-11-28T22:11:17.873 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128982.log 2023-11-28T22:11:17.874 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122053.log 2023-11-28T22:11:17.874 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103402.log.gz 2023-11-28T22:11:17.875 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128982.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.128982.log.gz 2023-11-28T22:11:17.875 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134817.log 2023-11-28T22:11:17.877 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147176.log 2023-11-28T22:11:17.877 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.122053.log.gz 2023-11-28T22:11:17.878 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138657.log 2023-11-28T22:11:17.878 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134817.log.gz 2023-11-28T22:11:17.879 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130574.log 2023-11-28T22:11:17.880 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.147176.log.gz 2023-11-28T22:11:17.880 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138657.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.138657.log.gz 2023-11-28T22:11:17.881 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131010.log 2023-11-28T22:11:17.882 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.97475.log 2023-11-28T22:11:17.882 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.130574.log.gz 2023-11-28T22:11:17.883 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.131010.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.131010.log.gz 2023-11-28T22:11:17.884 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115013.log 2023-11-28T22:11:17.885 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.97475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.97475.log.gz 2023-11-28T22:11:17.885 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143819.log 2023-11-28T22:11:17.886 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92867.log 2023-11-28T22:11:17.887 INFO:teuthology.orchestra.run.smithi045.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.115013.log.gz 2023-11-28T22:11:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142666.log 2023-11-28T22:11:17.888 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.92867.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.143819.log.gz 2023-11-28T22:11:17.889 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.92867.log.gz 2023-11-28T22:11:17.890 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147134.log 2023-11-28T22:11:17.891 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118346.log 2023-11-28T22:11:17.891 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142666.log.gz 2023-11-28T22:11:17.892 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147134.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147134.log.gz 2023-11-28T22:11:17.892 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119665.log 2023-11-28T22:11:17.894 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142353.log 2023-11-28T22:11:17.894 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.118346.log.gz 2023-11-28T22:11:17.895 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119665.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119665.log.gz 2023-11-28T22:11:17.895 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107677.log 2023-11-28T22:11:17.896 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95402.log 2023-11-28T22:11:17.897 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.142353.log.gz 2023-11-28T22:11:17.898 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.107677.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107677.log.gz 2023-11-28T22:11:17.898 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97604.log 2023-11-28T22:11:17.899 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94930.log 2023-11-28T22:11:17.900 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95402.log.gz 2023-11-28T22:11:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.97604.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97604.log.gz 2023-11-28T22:11:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130236.log 2023-11-28T22:11:17.902 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98559.log 2023-11-28T22:11:17.903 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.94930.log.gz 2023-11-28T22:11:17.904 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119413.log 2023-11-28T22:11:17.904 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.130236.log.gz 2023-11-28T22:11:17.905 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98559.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.98559.log.gz 2023-11-28T22:11:17.905 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95389.log 2023-11-28T22:11:17.907 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146011.log 2023-11-28T22:11:17.907 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119413.log.gz 2023-11-28T22:11:17.908 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95389.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95389.log.gz 2023-11-28T22:11:17.908 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139211.log 2023-11-28T22:11:17.910 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96073.log 2023-11-28T22:11:17.911 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.146011.log.gz 2023-11-28T22:11:17.911 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139211.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.139211.log.gz 2023-11-28T22:11:17.912 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137175.log 2023-11-28T22:11:17.913 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110248.log 2023-11-28T22:11:17.913 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96073.log.gz 2023-11-28T22:11:17.913 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137175.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.137175.log.gz 2023-11-28T22:11:17.914 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129295.log 2023-11-28T22:11:17.915 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113935.log 2023-11-28T22:11:17.916 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110248.log.gz 2023-11-28T22:11:17.916 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129295.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.129295.log.gz 2023-11-28T22:11:17.917 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131274.log 2023-11-28T22:11:17.918 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109346.log 2023-11-28T22:11:17.918 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.113935.log.gz 2023-11-28T22:11:17.919 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.131274.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.131274.log.gz 2023-11-28T22:11:17.919 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136480.log 2023-11-28T22:11:17.921 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111116.log 2023-11-28T22:11:17.921 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109346.log.gz 2023-11-28T22:11:17.922 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136480.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.136480.log.gz 2023-11-28T22:11:17.923 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.82536.log 2023-11-28T22:11:17.924 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.111116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98181.log 2023-11-28T22:11:17.924 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111116.log.gz 2023-11-28T22:11:17.925 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.tmp-client.admin.82536.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.82536.log.gz 2023-11-28T22:11:17.925 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98643.log 2023-11-28T22:11:17.927 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113202.log 2023-11-28T22:11:17.927 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98181.log.gz 2023-11-28T22:11:17.928 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98643.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.98643.log.gz 2023-11-28T22:11:17.928 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94540.log 2023-11-28T22:11:17.929 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137329.log 2023-11-28T22:11:17.930 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.113202.log.gz 2023-11-28T22:11:17.931 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94540.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94540.log.gz 2023-11-28T22:11:17.931 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99672.log 2023-11-28T22:11:17.932 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102393.log 2023-11-28T22:11:17.933 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.137329.log.gz 2023-11-28T22:11:17.934 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99672.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99672.log.gz 2023-11-28T22:11:17.934 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100962.log 2023-11-28T22:11:17.935 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102393.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.102393.log.gz 2023-11-28T22:11:17.935 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105429.log 2023-11-28T22:11:17.936 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100962.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.100962.log.gz 2023-11-28T22:11:17.937 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96068.log 2023-11-28T22:11:17.938 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.105429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139391.log 2023-11-28T22:11:17.938 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.105429.log.gz 2023-11-28T22:11:17.939 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96068.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96068.log.gz 2023-11-28T22:11:17.939 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119707.log 2023-11-28T22:11:17.941 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109609.log 2023-11-28T22:11:17.941 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.139391.log.gz 2023-11-28T22:11:17.942 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119707.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119707.log.gz 2023-11-28T22:11:17.942 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119016.log 2023-11-28T22:11:17.943 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98139.log 2023-11-28T22:11:17.944 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109609.log.gz 2023-11-28T22:11:17.944 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119016.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119016.log.gz 2023-11-28T22:11:17.945 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139601.log 2023-11-28T22:11:17.947 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98139.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98139.log.gz 2023-11-28T22:11:17.947 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147092.log 2023-11-28T22:11:17.948 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139601.log: gzip -5 --verbose -- 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139601.log.gz /var/log/ceph/ceph-client.admin.118296.log 2023-11-28T22:11:17.948 INFO:teuthology.orchestra.run.smithi045.stderr: 2023-11-28T22:11:17.950 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147092.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.147092.log.gz 2023-11-28T22:11:17.950 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106591.log 2023-11-28T22:11:17.950 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118296.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.118296.log.gz 2023-11-28T22:11:17.951 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102081.log 2023-11-28T22:11:17.952 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.106591.log: gzip -5 87.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.106591.log.gz 2023-11-28T22:11:17.952 INFO:teuthology.orchestra.run.smithi045.stderr: /var/log/ceph/ceph-client.admin.135206.log 2023-11-28T22:11:17.953 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102081.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.102081.log.gz 2023-11-28T22:11:17.953 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98517.log 2023-11-28T22:11:17.955 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102039.log 2023-11-28T22:11:17.955 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.135206.log.gz 2023-11-28T22:11:17.956 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98517.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98517.log.gz 2023-11-28T22:11:17.956 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100752.log 2023-11-28T22:11:17.957 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93023.log 2023-11-28T22:11:17.957 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102039.log.gz 2023-11-28T22:11:17.958 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100752.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100752.log.gz 2023-11-28T22:11:17.958 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122762.log 2023-11-28T22:11:17.959 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102291.log 2023-11-28T22:11:17.960 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.93023.log.gz 2023-11-28T22:11:17.961 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122762.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122762.log.gz 2023-11-28T22:11:17.961 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104545.log 2023-11-28T22:11:17.962 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103915.log 2023-11-28T22:11:17.962 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102291.log.gz 2023-11-28T22:11:17.963 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104545.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104545.log.gz 2023-11-28T22:11:17.963 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136004.log 2023-11-28T22:11:17.965 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125437.log 2023-11-28T22:11:17.965 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103915.log.gz 2023-11-28T22:11:17.966 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136004.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136004.log.gz 2023-11-28T22:11:17.966 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136564.log 2023-11-28T22:11:17.967 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143567.log 2023-11-28T22:11:17.967 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.125437.log.gz 2023-11-28T22:11:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136564.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.136564.log.gz 2023-11-28T22:11:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101437.log 2023-11-28T22:11:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127955.log 2023-11-28T22:11:17.970 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.143567.log.gz 2023-11-28T22:11:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101437.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.101437.log.gz 2023-11-28T22:11:17.971 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116413.log 2023-11-28T22:11:17.972 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96062.log 2023-11-28T22:11:17.972 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.127955.log.gz 2023-11-28T22:11:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116413.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116413.log.gz 2023-11-28T22:11:17.973 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136522.log 2023-11-28T22:11:17.974 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101731.log 2023-11-28T22:11:17.974 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96062.log.gz 2023-11-28T22:11:17.975 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136522.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.136522.log.gz 2023-11-28T22:11:17.976 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102249.log 2023-11-28T22:11:17.977 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106311.log 2023-11-28T22:11:17.977 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101731.log.gz 2023-11-28T22:11:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102249.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102249.log.gz 2023-11-28T22:11:17.978 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109033.log 2023-11-28T22:11:17.980 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.106311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113454.log 2023-11-28T22:11:17.980 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106311.log.gz 2023-11-28T22:11:17.981 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109033.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.109033.log.gz 2023-11-28T22:11:17.981 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110027.log 2023-11-28T22:11:17.983 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119623.log 2023-11-28T22:11:17.983 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113454.log.gz 2023-11-28T22:11:17.984 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110027.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110027.log.gz 2023-11-28T22:11:17.984 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138260.log 2023-11-28T22:11:17.985 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116791.log 2023-11-28T22:11:17.985 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119623.log.gz 2023-11-28T22:11:17.986 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138260.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138260.log.gz 2023-11-28T22:11:17.986 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135705.log 2023-11-28T22:11:17.988 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117608.log 2023-11-28T22:11:17.988 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116791.log.gz 2023-11-28T22:11:17.989 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135705.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135705.log.gz 2023-11-28T22:11:17.989 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122095.log 2023-11-28T22:11:17.990 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115870.log 2023-11-28T22:11:17.990 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.117608.log.gz 2023-11-28T22:11:17.991 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122095.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122095.log.gz 2023-11-28T22:11:17.992 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147638.log 2023-11-28T22:11:17.993 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116101.log 2023-11-28T22:11:17.993 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115870.log.gz 2023-11-28T22:11:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147638.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.147638.log.gz 2023-11-28T22:11:17.994 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108466.log 2023-11-28T22:11:17.995 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95734.log 2023-11-28T22:11:17.996 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.116101.log.gz 2023-11-28T22:11:17.996 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108466.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108466.log.gz 2023-11-28T22:11:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93465.log 2023-11-28T22:11:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115912.log 2023-11-28T22:11:17.998 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95734.log.gz 2023-11-28T22:11:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93465.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.93465.log.gz 2023-11-28T22:11:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117342.log 2023-11-28T22:11:18.000 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136648.log 2023-11-28T22:11:18.001 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115912.log.gz 2023-11-28T22:11:18.001 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117342.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117342.log.gz 2023-11-28T22:11:18.002 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-28T22:11:18.004 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136648.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.136648.log.gz 2023-11-28T22:11:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100710.log 2023-11-28T22:11:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117522.log 2023-11-28T22:11:18.005 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100710.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100710.log.gz 2023-11-28T22:11:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143303.log 2023-11-28T22:11:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138531.log 2023-11-28T22:11:18.008 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.117522.log.gz 2023-11-28T22:11:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125960.log 2023-11-28T22:11:18.009 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143303.log.gz 2023-11-28T22:11:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138531.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138531.log.gz 2023-11-28T22:11:18.011 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98055.log 2023-11-28T22:11:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109689.log 2023-11-28T22:11:18.012 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.125960.log.gz 2023-11-28T22:11:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98055.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98055.log.gz 2023-11-28T22:11:18.014 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93167.log 2023-11-28T22:11:18.015 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109689.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109689.log.gz 2023-11-28T22:11:18.015 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109178.log 2023-11-28T22:11:18.016 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92734.log 2023-11-28T22:11:18.017 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.93167.log.gz 2023-11-28T22:11:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109178.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.109178.log.gz 2023-11-28T22:11:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116875.log 2023-11-28T22:11:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.92734.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.92734.log.gz 2023-11-28T22:11:18.020 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147344.log 2023-11-28T22:11:18.021 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101149.log 2023-11-28T22:11:18.021 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116875.log.gz 2023-11-28T22:11:18.023 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147344.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.147344.log.gz 2023-11-28T22:11:18.023 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147512.log 2023-11-28T22:11:18.024 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101149.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.101149.log.gz 2023-11-28T22:11:18.024 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135663.log 2023-11-28T22:11:18.025 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126361.log 2023-11-28T22:11:18.026 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.147512.log.gz 2023-11-28T22:11:18.027 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135663.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.135663.log.gzgzip 2023-11-28T22:11:18.027 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.129589.log 2023-11-28T22:11:18.028 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126361.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.126361.log.gz 2023-11-28T22:11:18.029 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141828.log 2023-11-28T22:11:18.030 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105471.log 2023-11-28T22:11:18.030 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.129589.log.gz 2023-11-28T22:11:18.031 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141828.log: gzip 88.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.141828.log.gz -- 2023-11-28T22:11:18.032 INFO:teuthology.orchestra.run.smithi045.stderr: /var/log/ceph/ceph-client.admin.115347.log 2023-11-28T22:11:18.033 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.105471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143175.log 2023-11-28T22:11:18.033 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105471.log.gz 2023-11-28T22:11:18.034 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115347.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.115347.log.gz 2023-11-28T22:11:18.035 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127381.log 2023-11-28T22:11:18.036 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123537.log 2023-11-28T22:11:18.036 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.143175.log.gz 2023-11-28T22:11:18.037 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127381.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127381.log.gzgzip 2023-11-28T22:11:18.038 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.122137.log 2023-11-28T22:11:18.039 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101395.log 2023-11-28T22:11:18.039 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123537.log.gz 2023-11-28T22:11:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122137.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.122137.log.gz 2023-11-28T22:11:18.041 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135080.log 2023-11-28T22:11:18.042 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100584.log 2023-11-28T22:11:18.042 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101395.log.gz 2023-11-28T22:11:18.044 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135080.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135080.log.gz 2023-11-28T22:11:18.044 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142929.log 2023-11-28T22:11:18.045 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100584.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.100584.log.gz 2023-11-28T22:11:18.045 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147764.log 2023-11-28T22:11:18.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88817.log 2023-11-28T22:11:18.047 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142929.log.gz 2023-11-28T22:11:18.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93255.log 2023-11-28T22:11:18.049 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147764.log.gz 2023-11-28T22:11:18.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.88817.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88817.log.gz 2023-11-28T22:11:18.050 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95736.log 2023-11-28T22:11:18.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117650.log 2023-11-28T22:11:18.051 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93255.log.gz 2023-11-28T22:11:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142708.log 2023-11-28T22:11:18.052 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95736.log.gz 2023-11-28T22:11:18.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117650.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117650.log.gz 2023-11-28T22:11:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135411.log 2023-11-28T22:11:18.055 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109388.log 2023-11-28T22:11:18.055 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142708.log.gz 2023-11-28T22:11:18.056 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124166.log 2023-11-28T22:11:18.057 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.135411.log.gz 2023-11-28T22:11:18.058 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109388.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109388.log.gz 2023-11-28T22:11:18.058 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102853.log 2023-11-28T22:11:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130406.log 2023-11-28T22:11:18.062 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124166.log.gz 2023-11-28T22:11:18.062 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102853.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102853.log.gz 2023-11-28T22:11:18.062 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131118.log 2023-11-28T22:11:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130406.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.130406.log.gz 2023-11-28T22:11:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113244.log 2023-11-28T22:11:18.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.131118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143397.log 2023-11-28T22:11:18.064 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.131118.log.gz 2023-11-28T22:11:18.065 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116581.log 2023-11-28T22:11:18.065 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113244.log.gz 2023-11-28T22:11:18.067 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126002.log 2023-11-28T22:11:18.067 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143397.log.gz 2023-11-28T22:11:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123747.log 2023-11-28T22:11:18.068 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116581.log.gz 2023-11-28T22:11:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117386.log 2023-11-28T22:11:18.070 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.126002.log.gz 2023-11-28T22:11:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123747.log: gzip 88.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.123747.log.gz -- 2023-11-28T22:11:18.071 INFO:teuthology.orchestra.run.smithi045.stderr: /var/log/ceph/ceph-client.0.121047.log 2023-11-28T22:11:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117386.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117386.log.gz 2023-11-28T22:11:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139082.log 2023-11-28T22:11:18.073 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.121047.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.0.121047.log.gz 2023-11-28T22:11:18.074 INFO:teuthology.orchestra.run.smithi045.stderr: /var/log/ceph/ceph-client.admin.99468.log 2023-11-28T22:11:18.075 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139082.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.139082.log.gz 2023-11-28T22:11:18.075 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126445.log 2023-11-28T22:11:18.077 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138741.log 2023-11-28T22:11:18.077 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99468.log.gz 2023-11-28T22:11:18.078 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126445.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.126445.log.gz 2023-11-28T22:11:18.078 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146450.log 2023-11-28T22:11:18.079 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130144.log 2023-11-28T22:11:18.080 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138741.log.gz 2023-11-28T22:11:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146450.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.146450.log.gz 2023-11-28T22:11:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136352.log 2023-11-28T22:11:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103343.log 2023-11-28T22:11:18.083 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.130144.log.gz 2023-11-28T22:11:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136352.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.136352.log.gz 2023-11-28T22:11:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116497.log 2023-11-28T22:11:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147428.log 2023-11-28T22:11:18.086 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.103343.log.gz 2023-11-28T22:11:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125623.log 2023-11-28T22:11:18.087 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116497.log.gz 2023-11-28T22:11:18.089 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100140.log 2023-11-28T22:11:18.089 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147428.log.gz 2023-11-28T22:11:18.090 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125623.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.125623.log.gz 2023-11-28T22:11:18.090 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.110077.log 2023-11-28T22:11:18.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140358.log 2023-11-28T22:11:18.092 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100140.log.gz 2023-11-28T22:11:18.093 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141660.log 2023-11-28T22:11:18.093 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110077.log.gz 2023-11-28T22:11:18.094 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.140358.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.140358.log.gz 2023-11-28T22:11:18.095 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.140002.log 2023-11-28T22:11:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101869.log 2023-11-28T22:11:18.096 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.141660.log.gz 2023-11-28T22:11:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.140002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.140002.log.gz 2023-11-28T22:11:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101563.log 2023-11-28T22:11:18.099 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101022.log 2023-11-28T22:11:18.099 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.101869.log.gz 2023-11-28T22:11:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101563.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101563.log.gz 2023-11-28T22:11:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146870.log 2023-11-28T22:11:18.102 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130490.log 2023-11-28T22:11:18.102 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.101022.log.gz 2023-11-28T22:11:18.103 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146870.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146870.log.gz -5 2023-11-28T22:11:18.103 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.94005.log 2023-11-28T22:11:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104671.log 2023-11-28T22:11:18.105 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.130490.log.gz 2023-11-28T22:11:18.106 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137650.log 2023-11-28T22:11:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96316.log 2023-11-28T22:11:18.108 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.104671.log.gz 2023-11-28T22:11:18.108 INFO:teuthology.orchestra.run.smithi045.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.94005.log.gz 2023-11-28T22:11:18.109 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95742.log 2023-11-28T22:11:18.109 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.137650.log.gz 2023-11-28T22:11:18.110 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96316.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96316.log.gz 2023-11-28T22:11:18.111 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110119.log 2023-11-28T22:11:18.112 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95742.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95742.log.gz 2023-11-28T22:11:18.112 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116833.log 2023-11-28T22:11:18.113 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121916.log 2023-11-28T22:11:18.114 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110119.log.gz 2023-11-28T22:11:18.115 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116833.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.116833.log.gz 2023-11-28T22:11:18.115 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122636.log 2023-11-28T22:11:18.116 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121916.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.121916.log.gz 2023-11-28T22:11:18.117 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96358.log 2023-11-28T22:11:18.118 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122636.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.122636.log.gz 2023-11-28T22:11:18.118 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147724.log 2023-11-28T22:11:18.119 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102961.log 2023-11-28T22:11:18.120 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96358.log.gz 2023-11-28T22:11:18.121 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135248.log 2023-11-28T22:11:18.121 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147724.log.gz 2023-11-28T22:11:18.122 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102961.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102961.log.gz 2023-11-28T22:11:18.122 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143693.log 2023-11-28T22:11:18.123 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.136989.log 2023-11-28T22:11:18.124 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135248.log.gz 2023-11-28T22:11:18.125 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143693.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.143693.log.gz 2023-11-28T22:11:18.125 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121373.log 2023-11-28T22:11:18.126 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.136989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.136989.log.gz 2023-11-28T22:11:18.127 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99798.log 2023-11-28T22:11:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121373.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.121373.log.gz 2023-11-28T22:11:18.128 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121822.log 2023-11-28T22:11:18.129 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129887.log 2023-11-28T22:11:18.130 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99798.log.gz 2023-11-28T22:11:18.131 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121822.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.121822.log.gzgzip 2023-11-28T22:11:18.131 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.130100.log 2023-11-28T22:11:18.132 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129887.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.129887.log.gz 2023-11-28T22:11:18.133 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.114550.log 2023-11-28T22:11:18.134 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101819.log 2023-11-28T22:11:18.134 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.130100.log.gz 2023-11-28T22:11:18.135 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114550.log: gzip -5 88.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.114550.log.gz /var/log/ceph/ceph-client.admin.133383.log 2023-11-28T22:11:18.135 INFO:teuthology.orchestra.run.smithi045.stderr: 2023-11-28T22:11:18.137 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112845.log 2023-11-28T22:11:18.137 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101819.log.gz 2023-11-28T22:11:18.138 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.133383.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.133383.log.gz -5 2023-11-28T22:11:18.138 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.126139.log 2023-11-28T22:11:18.139 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.112845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108907.log 2023-11-28T22:11:18.140 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112845.log.gz 2023-11-28T22:11:18.141 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103705.log 2023-11-28T22:11:18.141 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126139.log.gz 2023-11-28T22:11:18.143 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108907.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108907.log.gz 2023-11-28T22:11:18.143 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139475.log 2023-11-28T22:11:18.144 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109136.log 2023-11-28T22:11:18.144 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103705.log.gz 2023-11-28T22:11:18.145 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139475.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.139475.log.gz 2023-11-28T22:11:18.145 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101775.log 2023-11-28T22:11:18.146 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144273.log 2023-11-28T22:11:18.147 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109136.log.gz 2023-11-28T22:11:18.148 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101775.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101775.log.gz 2023-11-28T22:11:18.148 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119749.log 2023-11-28T22:11:18.149 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.144273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141702.log 2023-11-28T22:11:18.150 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.144273.log.gz 2023-11-28T22:11:18.151 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119749.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.119749.log.gz 2023-11-28T22:11:18.151 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103789.log 2023-11-28T22:11:18.152 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99063.log 2023-11-28T22:11:18.152 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.141702.log.gz 2023-11-28T22:11:18.153 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103789.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103789.log.gz 2023-11-28T22:11:18.154 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135842.log 2023-11-28T22:11:18.155 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98769.log 2023-11-28T22:11:18.155 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99063.log.gz 2023-11-28T22:11:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135842.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.135842.log.gz 2023-11-28T22:11:18.156 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110205.log 2023-11-28T22:11:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109472.log 2023-11-28T22:11:18.158 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98769.log.gz 2023-11-28T22:11:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110205.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110205.log.gz 2023-11-28T22:11:18.159 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127660.log 2023-11-28T22:11:18.160 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141175.log 2023-11-28T22:11:18.161 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109472.log.gz 2023-11-28T22:11:18.161 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127660.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.127660.log.gz 2023-11-28T22:11:18.162 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101229.log 2023-11-28T22:11:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141175.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.141175.log.gz 2023-11-28T22:11:18.163 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138320.log 2023-11-28T22:11:18.164 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94846.log 2023-11-28T22:11:18.165 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101229.log.gz 2023-11-28T22:11:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138320.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.138320.log.gz 2023-11-28T22:11:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115625.log 2023-11-28T22:11:18.167 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94846.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94846.log.gz 2023-11-28T22:11:18.168 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124751.log 2023-11-28T22:11:18.169 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121278.log 2023-11-28T22:11:18.169 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115625.log.gz 2023-11-28T22:11:18.170 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124751.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.124751.log.gz 2023-11-28T22:11:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100289.log 2023-11-28T22:11:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121278.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.121278.log.gz 2023-11-28T22:11:18.172 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142498.log 2023-11-28T22:11:18.173 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117818.log 2023-11-28T22:11:18.173 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100289.log.gz 2023-11-28T22:11:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142498.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142498.log.gz 2023-11-28T22:11:18.175 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135922.log 2023-11-28T22:11:18.176 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117818.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.117818.log.gz 2023-11-28T22:11:18.177 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104629.log 2023-11-28T22:11:18.178 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135922.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.98475.log -- replaced with /var/log/ceph/ceph-client.admin.135922.log.gz 2023-11-28T22:11:18.178 INFO:teuthology.orchestra.run.smithi045.stderr: 2023-11-28T22:11:18.180 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113107.log 2023-11-28T22:11:18.180 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104629.log.gz 2023-11-28T22:11:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98475.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98475.log.gz 2023-11-28T22:11:18.181 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-11-28T22:11:18.182 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124375.log 2023-11-28T22:11:18.183 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.113107.log.gz 2023-11-28T22:11:18.184 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141922.log 2023-11-28T22:11:18.186 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112929.log 2023-11-28T22:11:18.186 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.124375.log.gz 2023-11-28T22:11:18.187 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113496.log 2023-11-28T22:11:18.188 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.141922.log.gz 2023-11-28T22:11:18.189 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.112929.log: gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.112929.log.gz --verbose 2023-11-28T22:11:18.189 INFO:teuthology.orchestra.run.smithi045.stderr: -- /var/log/ceph/ceph-client.admin.123145.log 2023-11-28T22:11:18.190 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146137.log 2023-11-28T22:11:18.190 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113496.log.gz 2023-11-28T22:11:18.192 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101997.log 2023-11-28T22:11:18.192 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.123145.log.gz 2023-11-28T22:11:18.194 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146137.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146137.log.gz 2023-11-28T22:11:18.194 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101352.log 2023-11-28T22:11:18.195 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120192.log 2023-11-28T22:11:18.195 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101997.log.gz 2023-11-28T22:11:18.196 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126219.log 2023-11-28T22:11:18.197 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.101352.log.gz 2023-11-28T22:11:18.198 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137566.log 2023-11-28T22:11:18.198 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120192.log.gz 2023-11-28T22:11:18.199 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96820.log 2023-11-28T22:11:18.199 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.126219.log.gz 2023-11-28T22:11:18.201 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92653.log 2023-11-28T22:11:18.201 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.137566.log.gz 2023-11-28T22:11:18.203 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134555.log 2023-11-28T22:11:18.203 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96820.log.gz 2023-11-28T22:11:18.204 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.92653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126711.log 2023-11-28T22:11:18.204 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92653.log.gz 2023-11-28T22:11:18.205 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.134555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143347.log 2023-11-28T22:11:18.205 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.134555.log.gz 2023-11-28T22:11:18.207 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145969.log 2023-11-28T22:11:18.207 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.126711.log.gz 2023-11-28T22:11:18.208 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123359.log 2023-11-28T22:11:18.208 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.143347.log.gz 2023-11-28T22:11:18.210 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.145969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124870.log 2023-11-28T22:11:18.210 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.145969.log.gz 2023-11-28T22:11:18.212 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114845.log 2023-11-28T22:11:18.212 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123359.log.gz 2023-11-28T22:11:18.213 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120414.log 2023-11-28T22:11:18.213 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.124870.log.gz 2023-11-28T22:11:18.214 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125183.log 2023-11-28T22:11:18.214 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114845.log.gz 2023-11-28T22:11:18.215 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108602.log 2023-11-28T22:11:18.216 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120414.log.gz 2023-11-28T22:11:18.217 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127996.log 2023-11-28T22:11:18.217 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.125183.log.gz 2023-11-28T22:11:18.219 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113641.log 2023-11-28T22:11:18.219 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108602.log.gz 2023-11-28T22:11:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103037.log 2023-11-28T22:11:18.220 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.127996.log.gz 2023-11-28T22:11:18.221 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127778.log 2023-11-28T22:11:18.222 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113641.log.gz 2023-11-28T22:11:18.223 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94356.log 2023-11-28T22:11:18.224 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.103037.log.gz 2023-11-28T22:11:18.224 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127590.log 2023-11-28T22:11:18.225 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.127778.log.gz 2023-11-28T22:11:18.226 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94356.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.94356.log.gz 2023-11-28T22:11:18.226 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119100.log 2023-11-28T22:11:18.228 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128719.log 2023-11-28T22:11:18.228 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.127590.log.gz 2023-11-28T22:11:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138836.log 2023-11-28T22:11:18.230 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.119100.log.gz 2023-11-28T22:11:18.231 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127896.log 2023-11-28T22:11:18.231 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128719.log.gz 2023-11-28T22:11:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.102734.log 2023-11-28T22:11:18.232 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138836.log.gz 2023-11-28T22:11:18.234 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103621.log 2023-11-28T22:11:18.234 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127896.log.gz 2023-11-28T22:11:18.235 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.102734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.102734.log.gz 2023-11-28T22:11:18.235 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95400.log 2023-11-28T22:11:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127489.log 2023-11-28T22:11:18.237 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103621.log.gz 2023-11-28T22:11:18.238 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100878.log 2023-11-28T22:11:18.238 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95400.log.gz 2023-11-28T22:11:18.239 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128119.log 2023-11-28T22:11:18.240 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.127489.log.gz 2023-11-28T22:11:18.241 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114677.log 2023-11-28T22:11:18.241 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100878.log.gz 2023-11-28T22:11:18.243 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120668.log 2023-11-28T22:11:18.243 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.128119.log.gz 2023-11-28T22:11:18.244 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138134.log 2023-11-28T22:11:18.244 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.114677.log.gz 2023-11-28T22:11:18.246 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103873.log 2023-11-28T22:11:18.246 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120668.log.gz 2023-11-28T22:11:18.247 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98853.log 2023-11-28T22:11:18.247 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.138134.log.gz 2023-11-28T22:11:18.248 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120048.log 2023-11-28T22:11:18.249 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103873.log.gz 2023-11-28T22:11:18.250 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146305.log 2023-11-28T22:11:18.250 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98853.log.gz 2023-11-28T22:11:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120752.log 2023-11-28T22:11:18.252 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120048.log.gz 2023-11-28T22:11:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114037.log 2023-11-28T22:11:18.253 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146305.log.gz 2023-11-28T22:11:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117692.log 2023-11-28T22:11:18.254 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120752.log.gz 2023-11-28T22:11:18.256 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114971.log 2023-11-28T22:11:18.256 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114037.log.gz 2023-11-28T22:11:18.257 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125227.log 2023-11-28T22:11:18.258 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117692.log.gz 2023-11-28T22:11:18.259 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108865.log 2023-11-28T22:11:18.259 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114971.log.gz 2023-11-28T22:11:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125227.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.125227.log.gz 2023-11-28T22:11:18.260 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.118743.log 2023-11-28T22:11:18.262 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108865.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108865.log.gz 2023-11-28T22:11:18.262 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137828.log 2023-11-28T22:11:18.266 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104587.log 2023-11-28T22:11:18.266 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.118743.log.gz 2023-11-28T22:11:18.266 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96904.log 2023-11-28T22:11:18.266 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.137828.log.gz 2023-11-28T22:11:18.267 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120458.log 2023-11-28T22:11:18.267 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104587.log.gz 2023-11-28T22:11:18.268 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111062.log 2023-11-28T22:11:18.268 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96904.log.gz 2023-11-28T22:11:18.269 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127113.log 2023-11-28T22:11:18.269 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.120458.log.gz 2023-11-28T22:11:18.270 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.111062.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.111062.log.gz -5 2023-11-28T22:11:18.271 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.94888.log 2023-11-28T22:11:18.272 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103520.log 2023-11-28T22:11:18.272 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127113.log.gz 2023-11-28T22:11:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126879.log 2023-11-28T22:11:18.274 INFO:teuthology.orchestra.run.smithi045.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94888.log.gz 2023-11-28T22:11:18.275 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108739.log 2023-11-28T22:11:18.275 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103520.log.gz 2023-11-28T22:11:18.276 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100920.log 2023-11-28T22:11:18.277 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126879.log.gz 2023-11-28T22:11:18.278 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145612.log 2023-11-28T22:11:18.278 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108739.log.gz 2023-11-28T22:11:18.280 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146492.log 2023-11-28T22:11:18.280 INFO:teuthology.orchestra.run.smithi045.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.100920.log.gz 2023-11-28T22:11:18.281 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.145612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108424.log 2023-11-28T22:11:18.281 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.145612.log.gz 2023-11-28T22:11:18.283 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146492.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.146492.log.gz 2023-11-28T22:11:18.283 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114929.log 2023-11-28T22:11:18.285 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108424.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108424.log.gz 2023-11-28T22:11:18.285 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96067.log 2023-11-28T22:11:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114929.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114929.log.gz 2023-11-28T22:11:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128814.log 2023-11-28T22:11:18.288 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136088.log 2023-11-28T22:11:18.288 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96067.log.gz 2023-11-28T22:11:18.289 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.127300.log 2023-11-28T22:11:18.289 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.128814.log.gz 2023-11-28T22:11:18.291 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136088.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.136088.log.gz 2023-11-28T22:11:18.291 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129108.log 2023-11-28T22:11:18.292 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.127300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.127300.log.gz 2023-11-28T22:11:18.292 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120278.log 2023-11-28T22:11:18.294 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105075.log 2023-11-28T22:11:18.294 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129108.log.gz 2023-11-28T22:11:18.295 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120278.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-28T22:11:18.295 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.120278.log.gz 2023-11-28T22:11:18.296 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.105075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103831.log 2023-11-28T22:11:18.297 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105075.log.gz 2023-11-28T22:11:18.298 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128583.log 2023-11-28T22:11:18.300 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122510.log 2023-11-28T22:11:18.300 INFO:teuthology.orchestra.run.smithi045.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103831.log.gz 2023-11-28T22:11:18.301 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128583.log: 93.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-11-28T22:11:18.302 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139517.log 2023-11-28T22:11:18.302 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128583.log.gz 2023-11-28T22:11:18.303 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129253.log 2023-11-28T22:11:18.303 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.122510.log.gz 2023-11-28T22:11:18.304 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139517.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.139517.log.gz 2023-11-28T22:11:18.304 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105370.log 2023-11-28T22:11:18.305 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144213.log 2023-11-28T22:11:18.306 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.129253.log.gz 2023-11-28T22:11:18.306 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.105370.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105370.log.gz 2023-11-28T22:11:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113328.log 2023-11-28T22:11:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.144213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113851.log 2023-11-28T22:11:18.308 INFO:teuthology.orchestra.run.smithi045.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.144213.log.gz 2023-11-28T22:11:18.308 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113328.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113328.log.gz 2023-11-28T22:11:18.309 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.130887.log 2023-11-28T22:11:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116329.log 2023-11-28T22:11:18.310 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113851.log.gz 2023-11-28T22:11:18.311 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.130887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.130887.log.gz 2023-11-28T22:11:18.311 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141523.log 2023-11-28T22:11:18.312 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137067.log 2023-11-28T22:11:18.313 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116329.log.gz 2023-11-28T22:11:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141523.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.141523.log.gz 2023-11-28T22:11:18.314 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94624.log 2023-11-28T22:11:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.136985.log 2023-11-28T22:11:18.316 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137067.log.gz 2023-11-28T22:11:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94624.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.94624.log.gz 2023-11-28T22:11:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.97908.log 2023-11-28T22:11:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.136985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141133.log 2023-11-28T22:11:18.318 INFO:teuthology.orchestra.run.smithi045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.136985.log.gz 2023-11-28T22:11:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.2.97908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.97908.log.gz 2023-11-28T22:11:18.319 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92920.log 2023-11-28T22:11:18.320 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102207.log 2023-11-28T22:11:18.320 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.141133.log.gz 2023-11-28T22:11:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.92920.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116285.log 2023-11-28T22:11:18.321 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92920.log.gz 2023-11-28T22:11:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102543.log 2023-11-28T22:11:18.323 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.102207.log.gz 2023-11-28T22:11:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116285.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116285.log.gz 2023-11-28T22:11:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136438.log 2023-11-28T22:11:18.325 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138008.log 2023-11-28T22:11:18.325 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102543.log.gz 2023-11-28T22:11:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136438.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136438.log.gz 2023-11-28T22:11:18.327 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127005.log 2023-11-28T22:11:18.329 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138008.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.138008.log.gz 2023-11-28T22:11:18.329 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110434.log 2023-11-28T22:11:18.330 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.127005.log: gzip -5 --verbose 88.0% -- -- replaced with /var/log/ceph/ceph-client.admin.127005.log.gz /var/log/ceph/ceph-client.admin.119142.log 2023-11-28T22:11:18.330 INFO:teuthology.orchestra.run.smithi045.stderr: 2023-11-28T22:11:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99105.log 2023-11-28T22:11:18.332 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110434.log.gz 2023-11-28T22:11:18.333 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125563.log 2023-11-28T22:11:18.333 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119142.log.gz 2023-11-28T22:11:18.334 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99105.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99105.log.gz 2023-11-28T22:11:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142269.log 2023-11-28T22:11:18.336 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147806.log 2023-11-28T22:11:18.336 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.125563.log.gz 2023-11-28T22:11:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142269.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142269.log.gz 2023-11-28T22:11:18.337 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140827.log 2023-11-28T22:11:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.147806.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147806.log.gz 2023-11-28T22:11:18.339 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103107.log 2023-11-28T22:11:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.140827.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.140827.log.gz 2023-11-28T22:11:18.340 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98895.log 2023-11-28T22:11:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103107.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103107.log.gz 2023-11-28T22:11:18.342 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141786.log 2023-11-28T22:11:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98895.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98895.log.gz 2023-11-28T22:11:18.343 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123897.log 2023-11-28T22:11:18.345 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.141786.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.141786.log.gz 2023-11-28T22:11:18.345 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108697.log 2023-11-28T22:11:18.346 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123897.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.123897.log.gz 2023-11-28T22:11:18.346 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120372.log 2023-11-28T22:11:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108697.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108697.log.gz 2023-11-28T22:11:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130364.log 2023-11-28T22:11:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120372.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.120372.log.gz 2023-11-28T22:11:18.350 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123229.log 2023-11-28T22:11:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130364.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.130364.log.gz 2023-11-28T22:11:18.351 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101689.log 2023-11-28T22:11:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123229.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123229.log.gz 2023-11-28T22:11:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96442.log 2023-11-28T22:11:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110895.log 2023-11-28T22:11:18.354 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101689.log.gz 2023-11-28T22:11:18.357 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143609.log 2023-11-28T22:11:18.357 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96442.log.gz 2023-11-28T22:11:18.357 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110895.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110895.log.gz 2023-11-28T22:11:18.357 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145696.log 2023-11-28T22:11:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114292.log 2023-11-28T22:11:18.359 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143609.log.gz 2023-11-28T22:11:18.359 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.145696.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145696.log.gz 2023-11-28T22:11:18.360 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-28T22:11:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107414.log 2023-11-28T22:11:18.362 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114292.log.gz 2023-11-28T22:11:18.363 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116143.log 2023-11-28T22:11:18.364 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.107414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.144090.log 2023-11-28T22:11:18.364 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107414.log.gz 2023-11-28T22:11:18.365 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98223.log 2023-11-28T22:11:18.366 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116143.log.gz 2023-11-28T22:11:18.366 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.144090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.144090.log.gz 2023-11-28T22:11:18.367 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95405.log 2023-11-28T22:11:18.368 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120710.log 2023-11-28T22:11:18.368 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98223.log.gz 2023-11-28T22:11:18.371 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101911.log 2023-11-28T22:11:18.371 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95405.log.gz 2023-11-28T22:11:18.372 INFO:teuthology.orchestra.run.smithi045.stderr: 89.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-28T22:11:18.372 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98601.log 2023-11-28T22:11:18.372 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.120710.log.gz 2023-11-28T22:11:18.373 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101911.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.101911.log.gz 2023-11-28T22:11:18.373 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-28T22:11:18.375 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.130891.log 2023-11-28T22:11:18.375 INFO:teuthology.orchestra.run.smithi045.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.98601.log.gz 2023-11-28T22:11:18.376 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129337.log 2023-11-28T22:11:18.376 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.130891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.130891.log.gz 2023-11-28T22:11:18.377 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116052.log 2023-11-28T22:11:18.378 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137871.log 2023-11-28T22:11:18.379 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.129337.log.gz 2023-11-28T22:11:18.380 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116052.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116052.log.gz 2023-11-28T22:11:18.381 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117258.log 2023-11-28T22:11:18.381 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.137871.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.137871.log.gz 2023-11-28T22:11:18.382 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.110775.log 2023-11-28T22:11:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142311.log 2023-11-28T22:11:18.384 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117258.log.gz 2023-11-28T22:11:18.384 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.110775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.110775.log.gz 2023-11-28T22:11:18.385 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126531.log 2023-11-28T22:11:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142582.log 2023-11-28T22:11:18.387 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142311.log.gz 2023-11-28T22:11:18.387 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126531.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.126531.log.gz 2023-11-28T22:11:18.388 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112971.log 2023-11-28T22:11:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142582.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.142582.log.gz 2023-11-28T22:11:18.390 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114212.log 2023-11-28T22:11:18.391 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.112971.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112971.log.gz 2023-11-28T22:11:18.391 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-11-28T22:11:18.393 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114212.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114212.log.gz 2023-11-28T22:11:18.393 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121451.log 2023-11-28T22:11:18.395 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122468.log 2023-11-28T22:11:18.396 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121451.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.121451.log.gz 2023-11-28T22:11:18.396 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121320.log 2023-11-28T22:11:18.399 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122468.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122468.log.gz 2023-11-28T22:11:18.400 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104957.log 2023-11-28T22:11:18.400 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121320.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.121320.log.gz 2023-11-28T22:11:18.403 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93507.log 2023-11-28T22:11:18.403 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104957.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104957.log.gz 2023-11-28T22:11:18.403 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128457.log 2023-11-28T22:11:18.404 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93507.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.93507.log.gz 2023-11-28T22:11:18.405 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.128457.log: /var/log/ceph/ceph-client.admin.125521.log 2023-11-28T22:11:18.405 INFO:teuthology.orchestra.run.smithi045.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.128457.log.gz 2023-11-28T22:11:18.406 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96652.log 2023-11-28T22:11:18.408 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125521.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.125521.log.gz 2023-11-28T22:11:18.408 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114634.log 2023-11-28T22:11:18.411 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96652.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96652.log.gz 2023-11-28T22:11:18.411 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136690.log 2023-11-28T22:11:18.413 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114634.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114634.log.gz 2023-11-28T22:11:18.413 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116623.log 2023-11-28T22:11:18.417 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136690.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.136690.log.gz 2023-11-28T22:11:18.417 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113977.log 2023-11-28T22:11:18.417 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116623.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.116623.log.gz 2023-11-28T22:11:18.417 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100794.log 2023-11-28T22:11:18.424 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113977.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113977.log.gz 2023-11-28T22:11:18.424 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116917.log 2023-11-28T22:11:18.425 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100794.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100794.log.gz 2023-11-28T22:11:18.425 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101479.log 2023-11-28T22:11:18.425 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.116917.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116917.log.gz 2023-11-28T22:11:18.425 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113725.log 2023-11-28T22:11:18.426 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101479.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.101479.log.gz 2023-11-28T22:11:18.429 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124316.log 2023-11-28T22:11:18.429 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113725.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113725.log.gz 2023-11-28T22:11:18.429 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102351.log 2023-11-28T22:11:18.429 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124316.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.124316.log.gz 2023-11-28T22:11:18.429 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99021.log 2023-11-28T22:11:18.431 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.102351.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.102351.log.gz 2023-11-28T22:11:18.431 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118879.log 2023-11-28T22:11:18.432 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99021.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99021.log.gz 2023-11-28T22:11:18.433 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122263.log 2023-11-28T22:11:18.435 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118879.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118879.log.gz 2023-11-28T22:11:18.435 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106871.log 2023-11-28T22:11:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122263.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122263.log.gz 2023-11-28T22:11:18.437 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136260.log 2023-11-28T22:11:18.439 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.106871.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.106871.log.gz 2023-11-28T22:11:18.439 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115556.log 2023-11-28T22:11:18.440 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136260.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.136260.log.gz 2023-11-28T22:11:18.441 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129547.log 2023-11-28T22:11:18.442 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.115556.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115556.log.gz 2023-11-28T22:11:18.444 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.129547.log: -- /var/log/ceph/ceph-client.admin.117216.log 2023-11-28T22:11:18.444 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.129547.log.gz 2023-11-28T22:11:18.445 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139255.log 2023-11-28T22:11:18.446 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117216.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.117216.log.gz 2023-11-28T22:11:18.447 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117430.log 2023-11-28T22:11:18.448 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.139255.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.139255.log.gz 2023-11-28T22:11:18.449 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103747.log 2023-11-28T22:11:18.450 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117430.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.117430.log.gz 2023-11-28T22:11:18.451 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120626.log 2023-11-28T22:11:18.452 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103747.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.103747.log.gz 2023-11-28T22:11:18.453 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93381.log 2023-11-28T22:11:18.454 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120626.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.120626.log.gz 2023-11-28T22:11:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95406.log 2023-11-28T22:11:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93381.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.93381.log.gz 2023-11-28T22:11:18.457 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112726.log 2023-11-28T22:11:18.459 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95406.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95406.log.gz 2023-11-28T22:11:18.460 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110163.log 2023-11-28T22:11:18.460 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.112726.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.112726.log.gz 2023-11-28T22:11:18.461 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96694.log 2023-11-28T22:11:18.463 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110163.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110163.log.gz 2023-11-28T22:11:18.464 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113767.log 2023-11-28T22:11:18.464 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96694.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96694.log.gz 2023-11-28T22:11:18.466 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128898.log 2023-11-28T22:11:18.468 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113767.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113767.log.gz 2023-11-28T22:11:18.468 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96077.log 2023-11-28T22:11:18.469 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128898.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.128898.log.gz 2023-11-28T22:11:18.470 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138050.log 2023-11-28T22:11:18.477 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96077.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96077.log.gz 2023-11-28T22:11:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100668.log 2023-11-28T22:11:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138050.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.138050.log.gz 2023-11-28T22:11:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98433.log 2023-11-28T22:11:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100668.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100668.log.gz 2023-11-28T22:11:18.478 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142413.log 2023-11-28T22:11:18.479 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98433.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98433.log.gz 2023-11-28T22:11:18.479 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130448.log 2023-11-28T22:11:18.480 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142413.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.142413.log.gz 2023-11-28T22:11:18.481 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113412.log 2023-11-28T22:11:18.481 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130448.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130448.log.gz 2023-11-28T22:11:18.482 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113370.log 2023-11-28T22:11:18.483 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113412.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113412.log.gz 2023-11-28T22:11:18.484 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104461.log 2023-11-28T22:11:18.484 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113370.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113370.log.gz 2023-11-28T22:11:18.485 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113013.log 2023-11-28T22:11:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104461.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.104461.log.gz 2023-11-28T22:11:18.488 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109262.log 2023-11-28T22:11:18.488 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.113013.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113013.log.gz 2023-11-28T22:11:18.489 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110476.log 2023-11-28T22:11:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109262.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.109262.log.gz 2023-11-28T22:11:18.492 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114887.log 2023-11-28T22:11:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110476.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.110476.log.gz 2023-11-28T22:11:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131231.log 2023-11-28T22:11:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114887.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.114887.log.gz 2023-11-28T22:11:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114132.log 2023-11-28T22:11:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.131231.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131231.log.gz 2023-11-28T22:11:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145926.log 2023-11-28T22:11:18.499 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114132.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114132.log.gz 2023-11-28T22:11:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118404.log 2023-11-28T22:11:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.145926.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.145926.log.gz 2023-11-28T22:11:18.502 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95056.log 2023-11-28T22:11:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.118404.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118404.log.gz 2023-11-28T22:11:18.504 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.95056.log: /var/log/ceph/ceph-client.admin.110332.log 2023-11-28T22:11:18.505 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95056.log.gz 2023-11-28T22:11:18.505 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94804.log 2023-11-28T22:11:18.507 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.110332.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110332.log.gz 2023-11-28T22:11:18.508 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114803.log 2023-11-28T22:11:18.509 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.94804.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94804.log.gz 2023-11-28T22:11:18.510 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99840.log 2023-11-28T22:11:18.511 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114803.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.114803.log.gz 2023-11-28T22:11:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124996.log 2023-11-28T22:11:18.513 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99840.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99840.log.gz 2023-11-28T22:11:18.514 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142792.log 2023-11-28T22:11:18.515 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.124996.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124996.log.gz 2023-11-28T22:11:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136798.log 2023-11-28T22:11:18.517 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142792.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142792.log.gz 2023-11-28T22:11:18.518 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95387.log 2023-11-28T22:11:18.519 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136798.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.136798.log.gz 2023-11-28T22:11:18.520 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121128.log 2023-11-28T22:11:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.95387.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95387.log.gz 2023-11-28T22:11:18.522 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122594.log 2023-11-28T22:11:18.523 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121128.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121128.log.gz 2023-11-28T22:11:18.524 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.139998.log 2023-11-28T22:11:18.525 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122594.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122594.log.gz 2023-11-28T22:11:18.526 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129463.log 2023-11-28T22:11:18.527 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.139998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.139998.log.gz 2023-11-28T22:11:18.527 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103579.log 2023-11-28T22:11:18.529 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129463.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.129463.log.gz 2023-11-28T22:11:18.530 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129024.log 2023-11-28T22:11:18.531 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103579.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.103579.log.gz 2023-11-28T22:11:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143885.log 2023-11-28T22:11:18.532 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129024.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.129024.log.gz 2023-11-28T22:11:18.533 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136310.log 2023-11-28T22:11:18.535 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143885.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.143885.log.gz 2023-11-28T22:11:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93339.log 2023-11-28T22:11:18.536 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.136310.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.136310.log.gz 2023-11-28T22:11:18.537 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117480.log 2023-11-28T22:11:18.539 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.93339.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.93339.log.gz 2023-11-28T22:11:18.540 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119058.log 2023-11-28T22:11:18.540 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.117480.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117480.log.gz 2023-11-28T22:11:18.541 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108382.log 2023-11-28T22:11:18.543 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119058.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119058.log.gz 2023-11-28T22:11:18.544 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142185.log 2023-11-28T22:11:18.544 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.108382.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108382.log.gz 2023-11-28T22:11:18.545 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99384.log 2023-11-28T22:11:18.547 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.142185.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.142185.log.gz 2023-11-28T22:11:18.548 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.97904.log 2023-11-28T22:11:18.548 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99384.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99384.log.gz 2023-11-28T22:11:18.549 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103225.log 2023-11-28T22:11:18.551 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.2.97904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.97904.log.gz 2023-11-28T22:11:18.551 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101521.log 2023-11-28T22:11:18.553 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.103225.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103225.log.gz 2023-11-28T22:11:18.554 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.101521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122011.log 2023-11-28T22:11:18.554 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101521.log.gz 2023-11-28T22:11:18.556 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140079.log 2023-11-28T22:11:18.557 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.122011.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122011.log.gz 2023-11-28T22:11:18.558 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120794.log 2023-11-28T22:11:18.559 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.140079.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.140079.log.gz 2023-11-28T22:11:18.560 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122347.log 2023-11-28T22:11:18.562 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.120794.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.120794.log.gz 2023-11-28T22:11:18.563 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/ceph-client.admin.122347.log: --verbose -- /var/log/ceph/ceph-client.admin.143217.log 2023-11-28T22:11:18.563 INFO:teuthology.orchestra.run.smithi045.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.122347.log.gz 2023-11-28T22:11:18.564 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.97466.log 2023-11-28T22:11:18.566 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143217.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.143217.log.gz 2023-11-28T22:11:18.567 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143259.log 2023-11-28T22:11:18.567 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.97466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.97466.log.gz 2023-11-28T22:11:18.568 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99231.log 2023-11-28T22:11:18.570 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.143259.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.143259.log.gz 2023-11-28T22:11:18.570 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135453.log 2023-11-28T22:11:18.571 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.99231.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99231.log.gz 2023-11-28T22:11:18.572 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96778.log 2023-11-28T22:11:18.573 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.135453.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.135453.log.gz 2023-11-28T22:11:18.574 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146221.log 2023-11-28T22:11:18.576 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.96778.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96778.log.gz 2023-11-28T22:11:18.577 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130616.log 2023-11-28T22:11:18.577 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146221.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146221.log.gz 2023-11-28T22:11:18.578 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94398.log 2023-11-28T22:11:18.580 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130616.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.130616.log.gz 2023-11-28T22:11:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ceph-client.admin.94398.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.125750.log 2023-11-28T22:11:18.582 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.94398.log.gz 2023-11-28T22:11:18.583 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123451.log 2023-11-28T22:11:18.584 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.125750.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.125750.log.gz 2023-11-28T22:11:18.585 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-11-28T22:11:18.586 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.123451.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123451.log.gz 2023-11-28T22:11:18.587 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112887.log 2023-11-28T22:11:18.590 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.112887.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112887.log.gz 2023-11-28T22:11:18.591 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100626.log 2023-11-28T22:11:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.100626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88551.log 2023-11-28T22:11:18.595 INFO:teuthology.orchestra.run.smithi045.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100626.log.gz 2023-11-28T22:11:18.596 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130532.log 2023-11-28T22:11:18.599 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.88551.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88551.log.gz 2023-11-28T22:11:18.601 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.130532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138176.log 2023-11-28T22:11:18.603 INFO:teuthology.orchestra.run.smithi045.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.130532.log.gz 2023-11-28T22:11:18.604 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121780.log 2023-11-28T22:11:18.606 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.138176.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138176.log.gz 2023-11-28T22:11:18.608 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.121780.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121780.log.gz 2023-11-28T22:11:18.609 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126487.log 2023-11-28T22:11:18.611 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129929.log 2023-11-28T22:11:18.613 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.126487.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98979.log 2023-11-28T22:11:18.613 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.126487.log.gz 2023-11-28T22:11:18.615 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.129929.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.129929.log.gz 2023-11-28T22:11:18.618 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.98979.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98979.log.gz 2023-11-28T22:11:18.619 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119539.log 2023-11-28T22:11:18.622 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.119539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109430.log 2023-11-28T22:11:18.623 INFO:teuthology.orchestra.run.smithi045.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.119539.log.gz 2023-11-28T22:11:18.625 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105751.log 2023-11-28T22:11:18.629 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.109430.log: /var/log/ceph/ceph-client.admin.105751.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104713.log 2023-11-28T22:11:18.630 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.109430.log.gz 2023-11-28T22:11:18.631 INFO:teuthology.orchestra.run.smithi045.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105751.log.gz 2023-11-28T22:11:18.633 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.104713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.127304.log 2023-11-28T22:11:18.637 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.127304.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104713.log.gz 2023-11-28T22:11:18.637 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114378.log 2023-11-28T22:11:18.638 INFO:teuthology.orchestra.run.smithi045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.127304.log.gz 2023-11-28T22:11:18.640 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128320.log 2023-11-28T22:11:18.643 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98937.log 2023-11-28T22:11:18.643 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.114378.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114378.log.gz 2023-11-28T22:11:18.644 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.128320.log.gz 2023-11-28T22:11:18.646 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146576.log 2023-11-28T22:11:18.649 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.146576.log: /var/log/ceph/ceph-client.admin.98937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146958.log 2023-11-28T22:11:18.652 INFO:teuthology.orchestra.run.smithi045.stderr: 88.2% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98937.log.gz 2023-11-28T22:11:18.652 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.146576.log.gz 2023-11-28T22:11:18.654 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128856.log 2023-11-28T22:11:18.658 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.128856.log: /var/log/ceph/ceph-client.admin.146958.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146958.log.gz 2023-11-28T22:11:18.659 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128856.log.gz 2023-11-28T22:11:19.102 INFO:teuthology.orchestra.run.smithi045.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-11-28T22:11:19.509 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-28T22:11:19.541 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-28T22:11:19.593 INFO:teuthology.orchestra.run.smithi190.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-28T22:11:20.298 INFO:teuthology.orchestra.run.smithi045.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-11-28T22:11:21.018 INFO:teuthology.orchestra.run.smithi045.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-28T22:11:21.879 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-28T22:11:21.930 INFO:teuthology.orchestra.run.smithi190.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-11-28T22:11:22.390 INFO:teuthology.orchestra.run.smithi190.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-28T22:11:22.408 INFO:teuthology.orchestra.run.smithi045.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-28T22:11:23.411 INFO:teuthology.orchestra.run.smithi045.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-11-28T22:11:23.869 INFO:teuthology.orchestra.run.smithi045.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-28T22:11:24.102 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-11-28T22:11:24.106 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-11-28T22:11:24.106 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m6.806s 2023-11-28T22:11:24.107 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m25.503s 2023-11-28T22:11:24.107 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m2.137s 2023-11-28T22:11:25.463 INFO:teuthology.orchestra.run.smithi045.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-28T22:11:25.468 INFO:teuthology.orchestra.run.smithi045.stderr: 2023-11-28T22:11:25.468 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m8.173s 2023-11-28T22:11:25.468 INFO:teuthology.orchestra.run.smithi045.stderr:user 0m27.177s 2023-11-28T22:11:25.469 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m4.519s 2023-11-28T22:11:25.470 INFO:tasks.ceph:Archiving logs... 2023-11-28T22:11:25.470 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/remote/smithi045/log 2023-11-28T22:11:25.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T22:11:29.781 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/remote/smithi190/log 2023-11-28T22:11:29.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T22:11:33.245 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-28T22:11:33.255 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-28T22:11:33.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T22:11:33.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T22:11:33.421 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-28T22:11:33.431 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-28T22:11:33.431 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T22:11:33.441 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T22:11:33.491 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2023-11-28T22:11:33.499 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2023-11-28T22:11:33.501 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T22:11:33.501 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2023-11-28T22:11:33.501 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 69 +50us[ +42us] +/- 84ms 2023-11-28T22:11:33.501 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -1202us[-1210us] +/- 62ms 2023-11-28T22:11:33.502 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 +1485us[+1485us] +/- 82ms 2023-11-28T22:11:33.502 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T22:11:33.510 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T22:11:33.510 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2023-11-28T22:11:33.510 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +100us[ +100us] +/- 85ms 2023-11-28T22:11:33.511 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 8 377 81 -1122us[-1169us] +/- 61ms 2023-11-28T22:11:33.511 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +1525us[+1525us] +/- 82ms 2023-11-28T22:11:33.511 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T22:11:33.513 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-28T22:11:33.523 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-28T22:11:33.524 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-28T22:11:33.558 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T22:11:33.750 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T22:11:34.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T22:11:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T22:11:34.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T22:11:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T22:11:34.256 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-28T22:11:34.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-28T22:11:34.325 INFO:teuthology.task.internal:Duration was 6717.960561 seconds 2023-11-28T22:11:34.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-28T22:11:34.367 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-28T22:11:34.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T22:11:34.377 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T22:11:34.542 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T22:11:34.550 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T22:11:35.046 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-28T22:11:35.046 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2023-11-28T22:11:35.047 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T22:11:35.130 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2023-11-28T22:11:35.131 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T22:11:35.207 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-28T22:11:35.207 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T22:11:35.216 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T22:11:35.321 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-28T22:11:35.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T22:11:35.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T22:11:35.593 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-28T22:11:35.604 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-28T22:11:35.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T22:11:35.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T22:11:35.815 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-28T22:11:35.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T22:11:35.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T22:11:35.948 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2023-11-28T22:11:35.948 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2023-11-28T22:11:36.005 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T22:11:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T22:11:36.062 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T22:11:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T22:11:36.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-28T22:11:36.131 INFO:teuthology.task.internal:Transferring archived files... 2023-11-28T22:11:36.131 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/remote/smithi045 2023-11-28T22:11:36.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T22:11:36.249 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637/remote/smithi190 2023-11-28T22:11:36.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T22:11:36.360 INFO:teuthology.task.internal:Removing archive directory... 2023-11-28T22:11:36.360 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-28T22:11:36.371 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-28T22:11:36.428 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-28T22:11:36.440 INFO:teuthology.task.internal:Not uploading archives. 2023-11-28T22:11:36.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-28T22:11:36.485 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-28T22:11:36.485 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T22:11:36.494 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T22:11:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 22:11 /home/ubuntu/cephtest 2023-11-28T22:11:36.552 INFO:teuthology.orchestra.run.smithi190.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 22:11 /home/ubuntu/cephtest 2023-11-28T22:11:36.555 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-28T22:11:36.566 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-28T22:11:36.651 INFO:teuthology.nuke:Checking targets against current locks 2023-11-28T22:11:36.678 DEBUG:teuthology.nuke:shortname: smithi045 2023-11-28T22:11:36.678 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T22:11:36.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 20:09:45.812081', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhPbRY6n/z7r1pF/uD1tRgmCEGcsmUX170GbKygNpTf9+0WWSUXUT7sNh8sneEl4yMj4WqUfMlgxQbC129zAy0='} 2023-11-28T22:11:36.709 DEBUG:teuthology.nuke:shortname: smithi190 2023-11-28T22:11:36.710 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T22:11:36.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:29:14-fs-wip-vshankar-testing-20231127.102914-testing-default-smithi/7469637', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 20:09:45.813540', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/yvpBetb1Oc/9+gPKWVCU/1lcSer5fSZJTIBiZbE7JIHVrtJbCVMGxerg9tkEPspZTIz+fk48ivQh6MPM1vyc='} 2023-11-28T22:11:36.759 INFO:teuthology.orchestra.console.smithi045:Power off 2023-11-28T22:11:36.760 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T22:11:36.786 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2023-11-28T22:11:36.787 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T22:11:36.804 INFO:teuthology.orchestra.console.smithi190:Power off 2023-11-28T22:11:36.804 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T22:11:36.814 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2023-11-28T22:11:36.827 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2023-11-28T22:11:36.828 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T22:11:36.851 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-11-28T22:11:40.816 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T22:11:40.852 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T22:11:40.927 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2023-11-28T22:11:40.963 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-11-28T22:11:44.929 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T22:11:44.964 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T22:11:45.040 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2023-11-28T22:11:45.075 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2023-11-28T22:11:45.141 INFO:teuthology.orchestra.console.smithi045:Power off completed 2023-11-28T22:11:45.176 INFO:teuthology.orchestra.console.smithi190:Power off completed 2023-11-28T22:11:45.282 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 client.1-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6717.960561275482 failure_reason: 'Test failure: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules)' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5143c8dfba6c4029833be10f60c36a78 status: fail success: false 2023-11-28T22:11:45.282 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T22:11:45.362 INFO:teuthology.run:FAIL